2024-02-09T17:01:56.687 INFO:root:teuthology version: 0.0.1.dev256+gd9fdb22 2024-02-09T17:01:56.687 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/quincy backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/radosbench} --owner scheduled_lflores@teuthology --archive /home/teuthworker/archive/lflores-2024-02-09_16:40:54-rados-wip-yuri2-testing-2024-02-06-1154-distro-default-smithi/7554109 --name lflores-2024-02-09_16:40:54-rados-wip-yuri2-testing-2024-02-06-1154-distro-default-smithi --verbose -- /home/teuthworker/archive/lflores-2024-02-09_16:40:54-rados-wip-yuri2-testing-2024-02-06-1154-distro-default-smithi/7554109/orig.config.yaml 2024-02-09T17:01:56.730 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-09T17:01:56.812 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/lflores-2024-02-09_16:40:54-rados-wip-yuri2-testing-2024-02-06-1154-distro-default-smithi/7554109 branch: wip-yuri2-testing-2024-02-06-1154 description: rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/quincy backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/radosbench} email: null first_in_suite: false job_id: '7554109' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: lflores-2024-02-09_16:40:54-rados-wip-yuri2-testing-2024-02-06-1154-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 30 - volumes: count: 4 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri2-testing-2024-02-06-1154 ceph: conf: client.2: debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms inject socket failures: 2500 ms tcp read timeout: 5 osd_pool_default_min_size: 2 osd_pool_default_size: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 paxos service trim min: 10 osd: bluestore zero block detection: true debug ms: 1 debug osd: 20 osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.3 osd debug verify cached snaps: true osd debug verify missing on start: true osd max backfills: 3 osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(MON_DOWN\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN sha1: 720625d38f7f95e6fa31efce87133247d9d28517 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 720625d38f7f95e6fa31efce87133247d9d28517 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: min_in: 4 workunit: branch: wip-tracker-64343 sha1: 9dda454f26f61ca0da16ee8ddca7ba78eb9abae7 owner: scheduled_lflores@teuthology priority: 50 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - - mon.c - osd.8 - osd.9 - osd.10 - osd.11 - client.2 seed: 7797 sha1: 720625d38f7f95e6fa31efce87133247d9d28517 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-tracker-64343 suite_path: /home/teuthworker/src/github.com_ljflores_ceph_9dda454f26f61ca0da16ee8ddca7ba78eb9abae7/qa suite_relpath: qa suite_repo: https://github.com/ljflores/ceph.git suite_sha1: 9dda454f26f61ca0da16ee8ddca7ba78eb9abae7 targets: smithi099.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIOkSp3bbjLr6kK3awzep5s9kTS9xb+WUQrOAcagKtk1e1gLsEWitevL0IBcheKc8IcHpQ+27M1mIOSfzGr3Kiw= smithi162.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI+8p+P8LpC8YqiPlB7Q3hXsXHduf9OHu6bXQLE5hd2P8TMdB8p3b7uHVbNyvQw90d/9uZR2JZTLsaLqV4/nXlU= smithi179.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJWV+jDbi2q9qIMcY5bjzRFollCWSTk/cUhTWsadUrHWx8+sh/V62MYj07tPHtu72R/hYsR6LsLW2QbHTQDMJ+8= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: branch: quincy exclude_packages: - ceph-mgr-dashboard - ceph-mgr-diskprediction-local - ceph-mgr-rook - ceph-mgr-cephadm - ceph-base-debuginfo - ceph-common-debuginfo - ceph-immutable-object-cache-debuginfo - ceph-radosgw-debuginfo - ceph-test-debuginfo - ceph-base-debuginfo - ceph-mgr-debuginfo - ceph-mds-debuginfo - ceph-mon-debuginfo - ceph-osd-debuginfo - ceph-fuse-debuginfo - librados-devel-debuginfo - libcephfs2-debuginfo - librados2-debuginfo - librbd1-debuginfo - python3-cephfs-debuginfo - python3-rados-debuginfo - python3-rbd-debuginfo - python3-rgw-debuginfo - rbd-fuse-debuginfo - rbd-mirror-debuginfo - rbd-nbd-debuginfo - ceph-volume - cephadm: conf: mon: auth allow insecure global id reclaim: true - thrashosds: chance_pgnum_grow: 1 chance_pgpnum_fix: 1 timeout: 1200 - full_sequential: - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: d9fdb2209e15b39d9f061fd85399f352ce0f0894 timestamp: 2024-02-09_16:40:54 tube: smithi user: lflores verbose: true verify_ceph_hash: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-02-09T17:01:56.812 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ljflores_ceph_9dda454f26f61ca0da16ee8ddca7ba78eb9abae7/qa; will attempt to use it 2024-02-09T17:01:56.812 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ljflores_ceph_9dda454f26f61ca0da16ee8ddca7ba78eb9abae7/qa/tasks 2024-02-09T17:01:56.813 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-09T17:01:56.814 INFO:teuthology.task.internal:Saving configuration 2024-02-09T17:01:56.830 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-09T17:01:56.836 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-09T17:01:56.860 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi099.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2024-02-09_16:40:54-rados-wip-yuri2-testing-2024-02-06-1154-distro-default-smithi/7554109', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-09 16:56:38.887306', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIOkSp3bbjLr6kK3awzep5s9kTS9xb+WUQrOAcagKtk1e1gLsEWitevL0IBcheKc8IcHpQ+27M1mIOSfzGr3Kiw='} 2024-02-09T17:01:56.878 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi162.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2024-02-09_16:40:54-rados-wip-yuri2-testing-2024-02-06-1154-distro-default-smithi/7554109', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-09 16:56:38.886029', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI+8p+P8LpC8YqiPlB7Q3hXsXHduf9OHu6bXQLE5hd2P8TMdB8p3b7uHVbNyvQw90d/9uZR2JZTLsaLqV4/nXlU='} 2024-02-09T17:01:56.900 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi179.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2024-02-09_16:40:54-rados-wip-yuri2-testing-2024-02-06-1154-distro-default-smithi/7554109', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-09 16:56:38.888299', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJWV+jDbi2q9qIMcY5bjzRFollCWSTk/cUhTWsadUrHWx8+sh/V62MYj07tPHtu72R/hYsR6LsLW2QbHTQDMJ+8='} 2024-02-09T17:01:56.901 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-09T17:01:56.907 INFO:teuthology.task.internal:roles: ubuntu@smithi099.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-02-09T17:01:56.907 INFO:teuthology.task.internal:roles: ubuntu@smithi162.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2024-02-09T17:01:56.907 INFO:teuthology.task.internal:roles: ubuntu@smithi179.front.sepia.ceph.com - ['mon.c', 'osd.8', 'osd.9', 'osd.10', 'osd.11', 'client.2'] 2024-02-09T17:01:56.907 INFO:teuthology.run_tasks:Running task console_log... 2024-02-09T17:01:57.008 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb944e8cf70>, signals=[15]) 2024-02-09T17:01:57.009 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-09T17:01:57.016 INFO:teuthology.task.internal:Opening connections... 2024-02-09T17:01:57.016 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi099.front.sepia.ceph.com 2024-02-09T17:01:57.018 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T17:01:57.095 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi162.front.sepia.ceph.com 2024-02-09T17:01:57.096 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T17:01:57.170 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi179.front.sepia.ceph.com 2024-02-09T17:01:57.171 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T17:01:57.245 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-09T17:01:57.252 DEBUG:teuthology.orchestra.run.smithi099:> uname -m 2024-02-09T17:01:57.274 INFO:teuthology.orchestra.run.smithi099.stdout:x86_64 2024-02-09T17:01:57.275 DEBUG:teuthology.orchestra.run.smithi099:> cat /etc/os-release 2024-02-09T17:01:57.335 INFO:teuthology.orchestra.run.smithi099.stdout:NAME="CentOS Stream" 2024-02-09T17:01:57.335 INFO:teuthology.orchestra.run.smithi099.stdout:VERSION="8" 2024-02-09T17:01:57.335 INFO:teuthology.orchestra.run.smithi099.stdout:ID="centos" 2024-02-09T17:01:57.336 INFO:teuthology.orchestra.run.smithi099.stdout:ID_LIKE="rhel fedora" 2024-02-09T17:01:57.336 INFO:teuthology.orchestra.run.smithi099.stdout:VERSION_ID="8" 2024-02-09T17:01:57.336 INFO:teuthology.orchestra.run.smithi099.stdout:PLATFORM_ID="platform:el8" 2024-02-09T17:01:57.336 INFO:teuthology.orchestra.run.smithi099.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-09T17:01:57.336 INFO:teuthology.orchestra.run.smithi099.stdout:ANSI_COLOR="0;31" 2024-02-09T17:01:57.336 INFO:teuthology.orchestra.run.smithi099.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-09T17:01:57.336 INFO:teuthology.orchestra.run.smithi099.stdout:HOME_URL="https://centos.org/" 2024-02-09T17:01:57.336 INFO:teuthology.orchestra.run.smithi099.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-09T17:01:57.336 INFO:teuthology.orchestra.run.smithi099.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-09T17:01:57.336 INFO:teuthology.orchestra.run.smithi099.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-09T17:01:57.337 INFO:teuthology.lock.ops:Updating smithi099.front.sepia.ceph.com on lock server 2024-02-09T17:01:57.361 DEBUG:teuthology.orchestra.run.smithi162:> uname -m 2024-02-09T17:01:57.383 INFO:teuthology.orchestra.run.smithi162.stdout:x86_64 2024-02-09T17:01:57.383 DEBUG:teuthology.orchestra.run.smithi162:> cat /etc/os-release 2024-02-09T17:01:57.444 INFO:teuthology.orchestra.run.smithi162.stdout:NAME="CentOS Stream" 2024-02-09T17:01:57.444 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION="8" 2024-02-09T17:01:57.444 INFO:teuthology.orchestra.run.smithi162.stdout:ID="centos" 2024-02-09T17:01:57.445 INFO:teuthology.orchestra.run.smithi162.stdout:ID_LIKE="rhel fedora" 2024-02-09T17:01:57.445 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION_ID="8" 2024-02-09T17:01:57.445 INFO:teuthology.orchestra.run.smithi162.stdout:PLATFORM_ID="platform:el8" 2024-02-09T17:01:57.445 INFO:teuthology.orchestra.run.smithi162.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-09T17:01:57.445 INFO:teuthology.orchestra.run.smithi162.stdout:ANSI_COLOR="0;31" 2024-02-09T17:01:57.445 INFO:teuthology.orchestra.run.smithi162.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-09T17:01:57.445 INFO:teuthology.orchestra.run.smithi162.stdout:HOME_URL="https://centos.org/" 2024-02-09T17:01:57.445 INFO:teuthology.orchestra.run.smithi162.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-09T17:01:57.445 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-09T17:01:57.445 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-09T17:01:57.446 INFO:teuthology.lock.ops:Updating smithi162.front.sepia.ceph.com on lock server 2024-02-09T17:01:57.469 DEBUG:teuthology.orchestra.run.smithi179:> uname -m 2024-02-09T17:01:57.493 INFO:teuthology.orchestra.run.smithi179.stdout:x86_64 2024-02-09T17:01:57.493 DEBUG:teuthology.orchestra.run.smithi179:> cat /etc/os-release 2024-02-09T17:01:57.554 INFO:teuthology.orchestra.run.smithi179.stdout:NAME="CentOS Stream" 2024-02-09T17:01:57.554 INFO:teuthology.orchestra.run.smithi179.stdout:VERSION="8" 2024-02-09T17:01:57.554 INFO:teuthology.orchestra.run.smithi179.stdout:ID="centos" 2024-02-09T17:01:57.554 INFO:teuthology.orchestra.run.smithi179.stdout:ID_LIKE="rhel fedora" 2024-02-09T17:01:57.554 INFO:teuthology.orchestra.run.smithi179.stdout:VERSION_ID="8" 2024-02-09T17:01:57.554 INFO:teuthology.orchestra.run.smithi179.stdout:PLATFORM_ID="platform:el8" 2024-02-09T17:01:57.554 INFO:teuthology.orchestra.run.smithi179.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-09T17:01:57.554 INFO:teuthology.orchestra.run.smithi179.stdout:ANSI_COLOR="0;31" 2024-02-09T17:01:57.555 INFO:teuthology.orchestra.run.smithi179.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-09T17:01:57.555 INFO:teuthology.orchestra.run.smithi179.stdout:HOME_URL="https://centos.org/" 2024-02-09T17:01:57.555 INFO:teuthology.orchestra.run.smithi179.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-09T17:01:57.555 INFO:teuthology.orchestra.run.smithi179.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-09T17:01:57.555 INFO:teuthology.orchestra.run.smithi179.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-09T17:01:57.555 INFO:teuthology.lock.ops:Updating smithi179.front.sepia.ceph.com on lock server 2024-02-09T17:01:57.577 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-09T17:01:57.587 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-09T17:01:57.593 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-09T17:01:57.593 DEBUG:teuthology.orchestra.run.smithi099:> test '!' -e /home/ubuntu/cephtest 2024-02-09T17:01:57.595 DEBUG:teuthology.orchestra.run.smithi162:> test '!' -e /home/ubuntu/cephtest 2024-02-09T17:01:57.597 DEBUG:teuthology.orchestra.run.smithi179:> test '!' -e /home/ubuntu/cephtest 2024-02-09T17:01:57.617 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-09T17:01:57.622 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-09T17:01:57.623 DEBUG:teuthology.orchestra.run.smithi099:> test -z $(ls -A /var/lib/ceph) 2024-02-09T17:01:57.657 DEBUG:teuthology.orchestra.run.smithi162:> test -z $(ls -A /var/lib/ceph) 2024-02-09T17:01:57.659 DEBUG:teuthology.orchestra.run.smithi179:> test -z $(ls -A /var/lib/ceph) 2024-02-09T17:01:57.690 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-09T17:01:57.767 INFO:teuthology.run_tasks:Running task kernel... 2024-02-09T17:01:57.781 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-09T17:01:57.781 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-09T17:01:57.781 DEBUG:teuthology.orchestra.run.smithi099:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-09T17:01:57.782 DEBUG:teuthology.orchestra.run.smithi162:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-09T17:01:57.782 DEBUG:teuthology.orchestra.run.smithi179:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-09T17:01:57.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-09T17:01:57.802 DEBUG:teuthology.orchestra.run.smithi162:> uname -r 2024-02-09T17:01:57.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-09T17:01:57.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-09T17:01:57.804 DEBUG:teuthology.orchestra.run.smithi099:> uname -r 2024-02-09T17:01:57.804 DEBUG:teuthology.orchestra.run.smithi179:> uname -r 2024-02-09T17:01:57.863 INFO:teuthology.orchestra.run.smithi162.stdout:4.18.0-489.el8.x86_64 2024-02-09T17:01:57.863 INFO:teuthology.task.kernel:Running kernel on smithi162: 4.18.0-489.el8.x86_64 2024-02-09T17:01:57.863 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum install -y kernel 2024-02-09T17:01:57.864 INFO:teuthology.orchestra.run.smithi179.stdout:4.18.0-489.el8.x86_64 2024-02-09T17:01:57.864 INFO:teuthology.task.kernel:Running kernel on smithi179: 4.18.0-489.el8.x86_64 2024-02-09T17:01:57.864 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum install -y kernel 2024-02-09T17:01:57.865 INFO:teuthology.orchestra.run.smithi099.stdout:4.18.0-489.el8.x86_64 2024-02-09T17:01:57.866 INFO:teuthology.task.kernel:Running kernel on smithi099: 4.18.0-489.el8.x86_64 2024-02-09T17:01:57.866 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum install -y kernel 2024-02-09T17:02:02.201 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 8 - AppStream 16 MB/s | 27 MB 00:01 2024-02-09T17:02:02.239 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 8 - AppStream 15 MB/s | 27 MB 00:01 2024-02-09T17:02:02.290 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 8 - AppStream 14 MB/s | 27 MB 00:01 2024-02-09T17:02:08.089 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 8 - BaseOS 9.5 MB/s | 10 MB 00:01 2024-02-09T17:02:08.227 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 8 - BaseOS 8.1 MB/s | 10 MB 00:01 2024-02-09T17:02:08.306 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 8 - BaseOS 8.1 MB/s | 10 MB 00:01 2024-02-09T17:02:09.939 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 8 - Extras 72 kB/s | 18 kB 00:00 2024-02-09T17:02:10.164 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 8 - Extras 72 kB/s | 18 kB 00:00 2024-02-09T17:02:10.243 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 8 - Extras 43 kB/s | 18 kB 00:00 2024-02-09T17:02:10.431 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 8 - Extras common packages 18 kB/s | 6.9 kB 00:00 2024-02-09T17:02:10.648 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 8 - Extras common packages 18 kB/s | 6.9 kB 00:00 2024-02-09T17:02:10.708 INFO:teuthology.orchestra.run.smithi099.stdout:Copr repo for python3-asyncssh owned by ceph 19 kB/s | 3.5 kB 00:00 2024-02-09T17:02:10.916 INFO:teuthology.orchestra.run.smithi162.stdout:Copr repo for python3-asyncssh owned by ceph 20 kB/s | 3.5 kB 00:00 2024-02-09T17:02:10.919 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 8 - Extras common packages 12 kB/s | 6.9 kB 00:00 2024-02-09T17:02:11.171 INFO:teuthology.orchestra.run.smithi179.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2024-02-09T17:02:11.986 INFO:teuthology.orchestra.run.smithi162.stdout:Extra Packages for Enterprise Linux 16 MB/s | 16 MB 00:00 2024-02-09T17:02:12.335 INFO:teuthology.orchestra.run.smithi099.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2024-02-09T17:02:13.091 INFO:teuthology.orchestra.run.smithi179.stdout:Extra Packages for Enterprise Linux 8.8 MB/s | 16 MB 00:01 2024-02-09T17:02:15.684 INFO:teuthology.orchestra.run.smithi162.stdout:lab-extras 415 kB/s | 24 kB 00:00 2024-02-09T17:02:16.075 INFO:teuthology.orchestra.run.smithi099.stdout:lab-extras 472 kB/s | 24 kB 00:00 2024-02-09T17:02:16.855 INFO:teuthology.orchestra.run.smithi179.stdout:lab-extras 538 kB/s | 24 kB 00:00 2024-02-09T17:02:18.680 INFO:teuthology.orchestra.run.smithi162.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-09T17:02:18.680 INFO:teuthology.orchestra.run.smithi162.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-09T17:02:18.795 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-09T17:02:18.795 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-09T17:02:18.795 INFO:teuthology.orchestra.run.smithi162.stdout: Package Architecture Version Repository Size 2024-02-09T17:02:18.795 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-09T17:02:18.796 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-02-09T17:02:18.796 INFO:teuthology.orchestra.run.smithi162.stdout: kernel x86_64 4.18.0-536.el8 baseos 10 M 2024-02-09T17:02:18.796 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-02-09T17:02:18.796 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-core x86_64 4.18.0-536.el8 baseos 44 M 2024-02-09T17:02:18.796 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-modules x86_64 4.18.0-536.el8 baseos 36 M 2024-02-09T17:02:18.796 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:02:18.796 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-02-09T17:02:18.796 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-09T17:02:18.796 INFO:teuthology.orchestra.run.smithi162.stdout:Install 3 Packages 2024-02-09T17:02:18.796 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:02:18.797 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 90 M 2024-02-09T17:02:18.797 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 96 M 2024-02-09T17:02:18.797 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-02-09T17:02:19.030 INFO:teuthology.orchestra.run.smithi099.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-09T17:02:19.030 INFO:teuthology.orchestra.run.smithi099.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-09T17:02:19.136 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-02-09T17:02:19.136 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-02-09T17:02:19.136 INFO:teuthology.orchestra.run.smithi099.stdout: Package Architecture Version Repository Size 2024-02-09T17:02:19.137 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-02-09T17:02:19.137 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-02-09T17:02:19.137 INFO:teuthology.orchestra.run.smithi099.stdout: kernel x86_64 4.18.0-536.el8 baseos 10 M 2024-02-09T17:02:19.137 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2024-02-09T17:02:19.137 INFO:teuthology.orchestra.run.smithi099.stdout: kernel-core x86_64 4.18.0-536.el8 baseos 44 M 2024-02-09T17:02:19.137 INFO:teuthology.orchestra.run.smithi099.stdout: kernel-modules x86_64 4.18.0-536.el8 baseos 36 M 2024-02-09T17:02:19.137 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:02:19.137 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-02-09T17:02:19.137 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-02-09T17:02:19.137 INFO:teuthology.orchestra.run.smithi099.stdout:Install 3 Packages 2024-02-09T17:02:19.137 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:02:19.138 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 90 M 2024-02-09T17:02:19.138 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 96 M 2024-02-09T17:02:19.138 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-02-09T17:02:19.727 INFO:teuthology.orchestra.run.smithi179.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-09T17:02:19.727 INFO:teuthology.orchestra.run.smithi179.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-09T17:02:19.845 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-02-09T17:02:19.845 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-02-09T17:02:19.846 INFO:teuthology.orchestra.run.smithi179.stdout: Package Architecture Version Repository Size 2024-02-09T17:02:19.846 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-02-09T17:02:19.846 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-02-09T17:02:19.846 INFO:teuthology.orchestra.run.smithi179.stdout: kernel x86_64 4.18.0-536.el8 baseos 10 M 2024-02-09T17:02:19.846 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-02-09T17:02:19.846 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-core x86_64 4.18.0-536.el8 baseos 44 M 2024-02-09T17:02:19.846 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-modules x86_64 4.18.0-536.el8 baseos 36 M 2024-02-09T17:02:19.846 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:02:19.846 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-02-09T17:02:19.846 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-02-09T17:02:19.846 INFO:teuthology.orchestra.run.smithi179.stdout:Install 3 Packages 2024-02-09T17:02:19.846 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:02:19.847 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 90 M 2024-02-09T17:02:19.847 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 96 M 2024-02-09T17:02:19.847 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-02-09T17:02:20.209 INFO:teuthology.orchestra.run.smithi162.stdout:(1/3): kernel-4.18.0-536.el8.x86_64.rpm 8.4 MB/s | 10 MB 00:01 2024-02-09T17:02:20.926 INFO:teuthology.orchestra.run.smithi162.stdout:(2/3): kernel-modules-4.18.0-536.el8.x86_64.rpm 19 MB/s | 36 MB 00:01 2024-02-09T17:02:21.048 INFO:teuthology.orchestra.run.smithi179.stdout:(1/3): kernel-4.18.0-536.el8.x86_64.rpm 8.8 MB/s | 10 MB 00:01 2024-02-09T17:02:21.651 INFO:teuthology.orchestra.run.smithi099.stdout:(1/3): kernel-modules-4.18.0-536.el8.x86_64.rpm 15 MB/s | 36 MB 00:02 2024-02-09T17:02:21.959 INFO:teuthology.orchestra.run.smithi099.stdout:(2/3): kernel-4.18.0-536.el8.x86_64.rpm 3.7 MB/s | 10 MB 00:02 2024-02-09T17:02:22.701 INFO:teuthology.orchestra.run.smithi099.stdout:(3/3): kernel-core-4.18.0-536.el8.x86_64.rpm 12 MB/s | 44 MB 00:03 2024-02-09T17:02:22.701 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-02-09T17:02:22.702 INFO:teuthology.orchestra.run.smithi099.stdout:Total 25 MB/s | 90 MB 00:03 2024-02-09T17:02:23.384 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-02-09T17:02:23.815 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-02-09T17:02:23.815 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-02-09T17:02:23.990 INFO:teuthology.orchestra.run.smithi179.stdout:(2/3): kernel-core-4.18.0-536.el8.x86_64.rpm 11 MB/s | 44 MB 00:04 2024-02-09T17:02:24.467 INFO:teuthology.orchestra.run.smithi162.stdout:(3/3): kernel-core-4.18.0-536.el8.x86_64.rpm 8.0 MB/s | 44 MB 00:05 2024-02-09T17:02:24.467 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-02-09T17:02:24.468 INFO:teuthology.orchestra.run.smithi162.stdout:Total 16 MB/s | 90 MB 00:05 2024-02-09T17:02:25.040 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-02-09T17:02:25.053 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-02-09T17:02:25.150 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-02-09T17:02:25.564 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-02-09T17:02:25.564 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-02-09T17:02:26.007 INFO:teuthology.orchestra.run.smithi179.stdout:(3/3): kernel-modules-4.18.0-536.el8.x86_64.rpm 5.9 MB/s | 36 MB 00:06 2024-02-09T17:02:26.007 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-02-09T17:02:26.007 INFO:teuthology.orchestra.run.smithi179.stdout:Total 15 MB/s | 90 MB 00:06 2024-02-09T17:02:26.691 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-02-09T17:02:26.827 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-02-09T17:02:26.852 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-02-09T17:02:27.105 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-02-09T17:02:27.105 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-02-09T17:02:27.813 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-02-09T17:02:28.280 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : kernel-core-4.18.0-536.el8.x86_64 1/3 2024-02-09T17:02:28.335 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-02-09T17:02:28.358 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-02-09T17:02:29.639 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-02-09T17:02:30.224 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : kernel-core-4.18.0-536.el8.x86_64 1/3 2024-02-09T17:02:31.107 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-02-09T17:02:31.605 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : kernel-core-4.18.0-536.el8.x86_64 1/3 2024-02-09T17:02:31.840 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: kernel-core-4.18.0-536.el8.x86_64 1/3 2024-02-09T17:02:32.238 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : kernel-modules-4.18.0-536.el8.x86_64 2/3 2024-02-09T17:02:33.777 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-core-4.18.0-536.el8.x86_64 1/3 2024-02-09T17:02:34.173 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : kernel-modules-4.18.0-536.el8.x86_64 2/3 2024-02-09T17:02:35.175 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: kernel-core-4.18.0-536.el8.x86_64 1/3 2024-02-09T17:02:35.563 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : kernel-modules-4.18.0-536.el8.x86_64 2/3 2024-02-09T17:02:35.690 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: kernel-modules-4.18.0-536.el8.x86_64 2/3 2024-02-09T17:02:35.977 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : kernel-4.18.0-536.el8.x86_64 3/3 2024-02-09T17:02:37.600 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-modules-4.18.0-536.el8.x86_64 2/3 2024-02-09T17:02:37.878 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : kernel-4.18.0-536.el8.x86_64 3/3 2024-02-09T17:02:38.991 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: kernel-modules-4.18.0-536.el8.x86_64 2/3 2024-02-09T17:02:39.285 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : kernel-4.18.0-536.el8.x86_64 3/3 2024-02-09T17:02:57.876 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: kernel-core-4.18.0-536.el8.x86_64 3/3 2024-02-09T17:02:57.899 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: kernel-modules-4.18.0-536.el8.x86_64 3/3 2024-02-09T17:02:58.243 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: kernel-4.18.0-536.el8.x86_64 3/3 2024-02-09T17:02:58.243 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : kernel-4.18.0-536.el8.x86_64 1/3 2024-02-09T17:02:58.243 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : kernel-core-4.18.0-536.el8.x86_64 2/3 2024-02-09T17:02:58.773 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : kernel-modules-4.18.0-536.el8.x86_64 3/3 2024-02-09T17:02:58.773 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:02:58.773 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-02-09T17:02:58.773 INFO:teuthology.orchestra.run.smithi099.stdout: kernel-4.18.0-536.el8.x86_64 kernel-core-4.18.0-536.el8.x86_64 2024-02-09T17:02:58.773 INFO:teuthology.orchestra.run.smithi099.stdout: kernel-modules-4.18.0-536.el8.x86_64 2024-02-09T17:02:58.774 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:02:58.774 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-02-09T17:02:58.906 DEBUG:teuthology.orchestra.run.smithi099:> rpm -q kernel | sort -rV | head -n 1 2024-02-09T17:02:59.161 INFO:teuthology.orchestra.run.smithi099.stdout:kernel-4.18.0-536.el8.x86_64 2024-02-09T17:02:59.161 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-536.el8.x86_64 2024-02-09T17:02:59.161 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-536.el8.x86_64 2024-02-09T17:02:59.161 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-09T17:02:59.161 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-09T17:02:59.161 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-02-09T17:02:59.162 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi099.front.sepia.ceph.com, path=None, version=distro) 2024-02-09T17:02:59.162 DEBUG:teuthology.orchestra.run.smithi099:> rpm -q kernel | sort -rV | head -n 1 2024-02-09T17:02:59.371 INFO:teuthology.orchestra.run.smithi099.stdout:kernel-4.18.0-536.el8.x86_64 2024-02-09T17:02:59.371 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-536.el8.x86_64 2024-02-09T17:02:59.371 DEBUG:teuthology.orchestra.run.smithi099:> sudo rpm -qi grub2-tools 2024-02-09T17:02:59.405 INFO:teuthology.orchestra.run.smithi099.stdout:Name : grub2-tools 2024-02-09T17:02:59.405 INFO:teuthology.orchestra.run.smithi099.stdout:Epoch : 1 2024-02-09T17:02:59.405 INFO:teuthology.orchestra.run.smithi099.stdout:Version : 2.02 2024-02-09T17:02:59.405 INFO:teuthology.orchestra.run.smithi099.stdout:Release : 129.el8 2024-02-09T17:02:59.405 INFO:teuthology.orchestra.run.smithi099.stdout:Architecture: x86_64 2024-02-09T17:02:59.405 INFO:teuthology.orchestra.run.smithi099.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-02-09T17:02:59.405 INFO:teuthology.orchestra.run.smithi099.stdout:Group : System Environment/Base 2024-02-09T17:02:59.405 INFO:teuthology.orchestra.run.smithi099.stdout:Size : 9265586 2024-02-09T17:02:59.405 INFO:teuthology.orchestra.run.smithi099.stdout:License : GPLv3+ 2024-02-09T17:02:59.405 INFO:teuthology.orchestra.run.smithi099.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-02-09T17:02:59.405 INFO:teuthology.orchestra.run.smithi099.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-02-09T17:02:59.405 INFO:teuthology.orchestra.run.smithi099.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-02-09T17:02:59.406 INFO:teuthology.orchestra.run.smithi099.stdout:Build Host : kbuilder.bsys.centos.org 2024-02-09T17:02:59.406 INFO:teuthology.orchestra.run.smithi099.stdout:Relocations : (not relocatable) 2024-02-09T17:02:59.406 INFO:teuthology.orchestra.run.smithi099.stdout:Packager : CentOS BuildSystem 2024-02-09T17:02:59.406 INFO:teuthology.orchestra.run.smithi099.stdout:Vendor : CentOS 2024-02-09T17:02:59.406 INFO:teuthology.orchestra.run.smithi099.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-09T17:02:59.406 INFO:teuthology.orchestra.run.smithi099.stdout:Summary : Support tools for GRUB. 2024-02-09T17:02:59.406 INFO:teuthology.orchestra.run.smithi099.stdout:Description : 2024-02-09T17:02:59.406 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:02:59.406 INFO:teuthology.orchestra.run.smithi099.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-09T17:02:59.406 INFO:teuthology.orchestra.run.smithi099.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-09T17:02:59.406 INFO:teuthology.orchestra.run.smithi099.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-09T17:02:59.406 INFO:teuthology.orchestra.run.smithi099.stdout:hardware devices. 2024-02-09T17:02:59.406 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:02:59.406 INFO:teuthology.orchestra.run.smithi099.stdout:This subpackage provides tools for support of all platforms. 2024-02-09T17:02:59.407 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-09T17:02:59.407 INFO:teuthology.task.kernel:Updating grub on smithi099 to boot 4.18.0-536.el8.x86_64 2024-02-09T17:02:59.407 DEBUG:teuthology.orchestra.run.smithi099:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-09T17:02:59.669 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-core-4.18.0-536.el8.x86_64 3/3 2024-02-09T17:02:59.692 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-modules-4.18.0-536.el8.x86_64 3/3 2024-02-09T17:03:00.140 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-4.18.0-536.el8.x86_64 3/3 2024-02-09T17:03:00.141 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : kernel-4.18.0-536.el8.x86_64 1/3 2024-02-09T17:03:00.141 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : kernel-core-4.18.0-536.el8.x86_64 2/3 2024-02-09T17:03:00.711 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : kernel-modules-4.18.0-536.el8.x86_64 3/3 2024-02-09T17:03:00.711 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:03:00.712 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-02-09T17:03:00.712 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-4.18.0-536.el8.x86_64 kernel-core-4.18.0-536.el8.x86_64 2024-02-09T17:03:00.712 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-modules-4.18.0-536.el8.x86_64 2024-02-09T17:03:00.712 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:03:00.712 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-09T17:03:00.868 DEBUG:teuthology.orchestra.run.smithi162:> rpm -q kernel | sort -rV | head -n 1 2024-02-09T17:03:01.080 INFO:teuthology.orchestra.run.smithi099.stderr:Generating grub configuration file ... 2024-02-09T17:03:01.086 INFO:teuthology.orchestra.run.smithi162.stdout:kernel-4.18.0-536.el8.x86_64 2024-02-09T17:03:01.086 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-536.el8.x86_64 2024-02-09T17:03:01.086 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-536.el8.x86_64 2024-02-09T17:03:01.086 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-09T17:03:01.086 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-09T17:03:01.086 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-02-09T17:03:01.086 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi162.front.sepia.ceph.com, path=None, version=distro) 2024-02-09T17:03:01.087 DEBUG:teuthology.orchestra.run.smithi162:> rpm -q kernel | sort -rV | head -n 1 2024-02-09T17:03:01.300 INFO:teuthology.orchestra.run.smithi162.stdout:kernel-4.18.0-536.el8.x86_64 2024-02-09T17:03:01.301 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-536.el8.x86_64 2024-02-09T17:03:01.301 DEBUG:teuthology.orchestra.run.smithi162:> sudo rpm -qi grub2-tools 2024-02-09T17:03:01.334 INFO:teuthology.orchestra.run.smithi162.stdout:Name : grub2-tools 2024-02-09T17:03:01.334 INFO:teuthology.orchestra.run.smithi162.stdout:Epoch : 1 2024-02-09T17:03:01.334 INFO:teuthology.orchestra.run.smithi162.stdout:Version : 2.02 2024-02-09T17:03:01.335 INFO:teuthology.orchestra.run.smithi162.stdout:Release : 129.el8 2024-02-09T17:03:01.335 INFO:teuthology.orchestra.run.smithi162.stdout:Architecture: x86_64 2024-02-09T17:03:01.335 INFO:teuthology.orchestra.run.smithi162.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-02-09T17:03:01.335 INFO:teuthology.orchestra.run.smithi162.stdout:Group : System Environment/Base 2024-02-09T17:03:01.335 INFO:teuthology.orchestra.run.smithi162.stdout:Size : 9265586 2024-02-09T17:03:01.335 INFO:teuthology.orchestra.run.smithi162.stdout:License : GPLv3+ 2024-02-09T17:03:01.335 INFO:teuthology.orchestra.run.smithi162.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-02-09T17:03:01.335 INFO:teuthology.orchestra.run.smithi162.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-02-09T17:03:01.335 INFO:teuthology.orchestra.run.smithi162.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-02-09T17:03:01.335 INFO:teuthology.orchestra.run.smithi162.stdout:Build Host : kbuilder.bsys.centos.org 2024-02-09T17:03:01.335 INFO:teuthology.orchestra.run.smithi162.stdout:Relocations : (not relocatable) 2024-02-09T17:03:01.335 INFO:teuthology.orchestra.run.smithi162.stdout:Packager : CentOS BuildSystem 2024-02-09T17:03:01.335 INFO:teuthology.orchestra.run.smithi162.stdout:Vendor : CentOS 2024-02-09T17:03:01.335 INFO:teuthology.orchestra.run.smithi162.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-09T17:03:01.335 INFO:teuthology.orchestra.run.smithi162.stdout:Summary : Support tools for GRUB. 2024-02-09T17:03:01.336 INFO:teuthology.orchestra.run.smithi162.stdout:Description : 2024-02-09T17:03:01.336 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:03:01.336 INFO:teuthology.orchestra.run.smithi162.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-09T17:03:01.336 INFO:teuthology.orchestra.run.smithi162.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-09T17:03:01.336 INFO:teuthology.orchestra.run.smithi162.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-09T17:03:01.336 INFO:teuthology.orchestra.run.smithi162.stdout:hardware devices. 2024-02-09T17:03:01.336 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:03:01.336 INFO:teuthology.orchestra.run.smithi162.stdout:This subpackage provides tools for support of all platforms. 2024-02-09T17:03:01.337 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-09T17:03:01.337 INFO:teuthology.task.kernel:Updating grub on smithi162 to boot 4.18.0-536.el8.x86_64 2024-02-09T17:03:01.337 DEBUG:teuthology.orchestra.run.smithi162:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-09T17:03:01.365 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: kernel-core-4.18.0-536.el8.x86_64 3/3 2024-02-09T17:03:01.389 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: kernel-modules-4.18.0-536.el8.x86_64 3/3 2024-02-09T17:03:01.830 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: kernel-4.18.0-536.el8.x86_64 3/3 2024-02-09T17:03:01.830 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : kernel-4.18.0-536.el8.x86_64 1/3 2024-02-09T17:03:01.830 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : kernel-core-4.18.0-536.el8.x86_64 2/3 2024-02-09T17:03:02.320 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : kernel-modules-4.18.0-536.el8.x86_64 3/3 2024-02-09T17:03:02.320 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:03:02.320 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-02-09T17:03:02.320 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-4.18.0-536.el8.x86_64 kernel-core-4.18.0-536.el8.x86_64 2024-02-09T17:03:02.321 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-modules-4.18.0-536.el8.x86_64 2024-02-09T17:03:02.321 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:03:02.321 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-02-09T17:03:02.487 DEBUG:teuthology.orchestra.run.smithi179:> rpm -q kernel | sort -rV | head -n 1 2024-02-09T17:03:02.703 INFO:teuthology.orchestra.run.smithi179.stdout:kernel-4.18.0-536.el8.x86_64 2024-02-09T17:03:02.703 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-536.el8.x86_64 2024-02-09T17:03:02.703 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-536.el8.x86_64 2024-02-09T17:03:02.703 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-09T17:03:02.703 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-09T17:03:02.704 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-02-09T17:03:02.704 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi179.front.sepia.ceph.com, path=None, version=distro) 2024-02-09T17:03:02.704 DEBUG:teuthology.orchestra.run.smithi179:> rpm -q kernel | sort -rV | head -n 1 2024-02-09T17:03:02.741 INFO:teuthology.orchestra.run.smithi162.stderr:Generating grub configuration file ... 2024-02-09T17:03:02.915 INFO:teuthology.orchestra.run.smithi179.stdout:kernel-4.18.0-536.el8.x86_64 2024-02-09T17:03:02.915 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-536.el8.x86_64 2024-02-09T17:03:02.915 DEBUG:teuthology.orchestra.run.smithi179:> sudo rpm -qi grub2-tools 2024-02-09T17:03:02.949 INFO:teuthology.orchestra.run.smithi179.stdout:Name : grub2-tools 2024-02-09T17:03:02.949 INFO:teuthology.orchestra.run.smithi179.stdout:Epoch : 1 2024-02-09T17:03:02.949 INFO:teuthology.orchestra.run.smithi179.stdout:Version : 2.02 2024-02-09T17:03:02.949 INFO:teuthology.orchestra.run.smithi179.stdout:Release : 129.el8 2024-02-09T17:03:02.949 INFO:teuthology.orchestra.run.smithi179.stdout:Architecture: x86_64 2024-02-09T17:03:02.949 INFO:teuthology.orchestra.run.smithi179.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-02-09T17:03:02.949 INFO:teuthology.orchestra.run.smithi179.stdout:Group : System Environment/Base 2024-02-09T17:03:02.949 INFO:teuthology.orchestra.run.smithi179.stdout:Size : 9265586 2024-02-09T17:03:02.949 INFO:teuthology.orchestra.run.smithi179.stdout:License : GPLv3+ 2024-02-09T17:03:02.949 INFO:teuthology.orchestra.run.smithi179.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-02-09T17:03:02.950 INFO:teuthology.orchestra.run.smithi179.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-02-09T17:03:02.950 INFO:teuthology.orchestra.run.smithi179.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-02-09T17:03:02.950 INFO:teuthology.orchestra.run.smithi179.stdout:Build Host : kbuilder.bsys.centos.org 2024-02-09T17:03:02.950 INFO:teuthology.orchestra.run.smithi179.stdout:Relocations : (not relocatable) 2024-02-09T17:03:02.950 INFO:teuthology.orchestra.run.smithi179.stdout:Packager : CentOS BuildSystem 2024-02-09T17:03:02.950 INFO:teuthology.orchestra.run.smithi179.stdout:Vendor : CentOS 2024-02-09T17:03:02.950 INFO:teuthology.orchestra.run.smithi179.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-09T17:03:02.950 INFO:teuthology.orchestra.run.smithi179.stdout:Summary : Support tools for GRUB. 2024-02-09T17:03:02.950 INFO:teuthology.orchestra.run.smithi179.stdout:Description : 2024-02-09T17:03:02.950 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:03:02.950 INFO:teuthology.orchestra.run.smithi179.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-09T17:03:02.950 INFO:teuthology.orchestra.run.smithi179.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-09T17:03:02.950 INFO:teuthology.orchestra.run.smithi179.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-09T17:03:02.950 INFO:teuthology.orchestra.run.smithi179.stdout:hardware devices. 2024-02-09T17:03:02.951 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:03:02.951 INFO:teuthology.orchestra.run.smithi179.stdout:This subpackage provides tools for support of all platforms. 2024-02-09T17:03:02.951 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-09T17:03:02.952 INFO:teuthology.task.kernel:Updating grub on smithi179 to boot 4.18.0-536.el8.x86_64 2024-02-09T17:03:02.952 DEBUG:teuthology.orchestra.run.smithi179:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-09T17:03:03.298 INFO:teuthology.orchestra.run.smithi099.stderr:done 2024-02-09T17:03:03.299 DEBUG:teuthology.orchestra.run.smithi099:> mktemp 2024-02-09T17:03:03.324 INFO:teuthology.orchestra.run.smithi099.stdout:/tmp/tmp.FJRGw2gxxE 2024-02-09T17:03:03.324 DEBUG:teuthology.orchestra.run.smithi099:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.FJRGw2gxxE 2024-02-09T17:03:03.407 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 0666 /tmp/tmp.FJRGw2gxxE 2024-02-09T17:03:03.556 DEBUG:teuthology.orchestra.remote:smithi099:/tmp/tmp.FJRGw2gxxE is 6KB 2024-02-09T17:03:03.604 DEBUG:teuthology.orchestra.run.smithi099:> rm -fr /tmp/tmp.FJRGw2gxxE 2024-02-09T17:03:03.621 DEBUG:teuthology.orchestra.run.smithi099:> sudo /bin/ls /boot/loader/entries || true 2024-02-09T17:03:03.686 INFO:teuthology.orchestra.run.smithi099.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-02-09T17:03:03.687 INFO:teuthology.orchestra.run.smithi099.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-02-09T17:03:03.687 INFO:teuthology.orchestra.run.smithi099.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-02-09T17:03:03.687 INFO:teuthology.orchestra.run.smithi099.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-536.el8.x86_64.conf 2024-02-09T17:03:03.688 DEBUG:teuthology.orchestra.run.smithi099:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-536.el8.x86_64 2024-02-09T17:03:03.847 DEBUG:teuthology.orchestra.run.smithi099:> sudo shutdown -r now 2024-02-09T17:03:04.458 INFO:teuthology.orchestra.run.smithi179.stderr:Generating grub configuration file ... 2024-02-09T17:03:05.141 INFO:teuthology.orchestra.run.smithi162.stderr:done 2024-02-09T17:03:05.143 DEBUG:teuthology.orchestra.run.smithi162:> mktemp 2024-02-09T17:03:05.166 INFO:teuthology.orchestra.run.smithi162.stdout:/tmp/tmp.qZE3gD9nQu 2024-02-09T17:03:05.166 DEBUG:teuthology.orchestra.run.smithi162:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.qZE3gD9nQu 2024-02-09T17:03:05.248 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 0666 /tmp/tmp.qZE3gD9nQu 2024-02-09T17:03:05.395 DEBUG:teuthology.orchestra.remote:smithi162:/tmp/tmp.qZE3gD9nQu is 6KB 2024-02-09T17:03:05.444 DEBUG:teuthology.orchestra.run.smithi162:> rm -fr /tmp/tmp.qZE3gD9nQu 2024-02-09T17:03:05.461 DEBUG:teuthology.orchestra.run.smithi162:> sudo /bin/ls /boot/loader/entries || true 2024-02-09T17:03:05.526 INFO:teuthology.orchestra.run.smithi162.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-02-09T17:03:05.526 INFO:teuthology.orchestra.run.smithi162.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-02-09T17:03:05.526 INFO:teuthology.orchestra.run.smithi162.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-02-09T17:03:05.526 INFO:teuthology.orchestra.run.smithi162.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-536.el8.x86_64.conf 2024-02-09T17:03:05.527 DEBUG:teuthology.orchestra.run.smithi162:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-536.el8.x86_64 2024-02-09T17:03:05.656 DEBUG:teuthology.orchestra.run.smithi162:> sudo shutdown -r now 2024-02-09T17:03:06.756 INFO:teuthology.orchestra.run.smithi179.stderr:done 2024-02-09T17:03:06.757 DEBUG:teuthology.orchestra.run.smithi179:> mktemp 2024-02-09T17:03:06.791 INFO:teuthology.orchestra.run.smithi179.stdout:/tmp/tmp.uSOmVn7rxU 2024-02-09T17:03:06.791 DEBUG:teuthology.orchestra.run.smithi179:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.uSOmVn7rxU 2024-02-09T17:03:06.864 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod 0666 /tmp/tmp.uSOmVn7rxU 2024-02-09T17:03:07.027 DEBUG:teuthology.orchestra.remote:smithi179:/tmp/tmp.uSOmVn7rxU is 6KB 2024-02-09T17:03:07.074 DEBUG:teuthology.orchestra.run.smithi179:> rm -fr /tmp/tmp.uSOmVn7rxU 2024-02-09T17:03:07.092 DEBUG:teuthology.orchestra.run.smithi179:> sudo /bin/ls /boot/loader/entries || true 2024-02-09T17:03:07.164 INFO:teuthology.orchestra.run.smithi179.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-02-09T17:03:07.164 INFO:teuthology.orchestra.run.smithi179.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-02-09T17:03:07.164 INFO:teuthology.orchestra.run.smithi179.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-02-09T17:03:07.164 INFO:teuthology.orchestra.run.smithi179.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-536.el8.x86_64.conf 2024-02-09T17:03:07.166 DEBUG:teuthology.orchestra.run.smithi179:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-536.el8.x86_64 2024-02-09T17:03:07.312 DEBUG:teuthology.orchestra.run.smithi179:> sudo shutdown -r now 2024-02-09T17:03:33.877 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-02-09T17:03:33.878 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi099.front.sepia.ceph.com' 2024-02-09T17:03:33.879 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T17:03:35.661 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-02-09T17:03:35.661 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-02-09T17:03:35.662 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T17:03:37.316 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-02-09T17:03:37.316 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-02-09T17:03:37.317 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T17:03:52.380 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.99 2024-02-09T17:03:54.172 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.162 2024-02-09T17:03:55.708 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.179 2024-02-09T17:04:01.386 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi099.front.sepia.ceph.com' 2024-02-09T17:04:01.387 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T17:04:03.173 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-02-09T17:04:03.173 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T17:04:04.448 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.99 2024-02-09T17:04:04.710 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-02-09T17:04:04.710 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T17:04:06.236 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.162 2024-02-09T17:04:07.772 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.179 2024-02-09T17:04:16.454 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi099.front.sepia.ceph.com' 2024-02-09T17:04:16.455 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T17:04:18.239 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-02-09T17:04:18.239 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T17:04:19.774 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-02-09T17:04:19.775 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T17:04:38.204 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.179 2024-02-09T17:04:48.444 DEBUG:teuthology.orchestra.run.smithi099:> true 2024-02-09T17:04:49.085 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi099.front.sepia.ceph.com' 2024-02-09T17:04:49.085 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-536.el8.x86_64"... 2024-02-09T17:04:49.085 DEBUG:teuthology.orchestra.run.smithi099:> uname -r 2024-02-09T17:04:49.101 INFO:teuthology.orchestra.run.smithi099.stdout:4.18.0-536.el8.x86_64 2024-02-09T17:04:49.101 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-536.el8.x86_64 vs 4.18.0-536.el8.x86_64 2024-02-09T17:04:49.101 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-09T17:04:49.102 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-09T17:04:50.102 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-02-09T17:04:50.103 DEBUG:teuthology.orchestra.run.smithi099:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-09T17:04:50.174 INFO:teuthology.orchestra.run.smithi099.stdout:ttyS1 2024-02-09T17:04:50.192 DEBUG:teuthology.parallel:result is None 2024-02-09T17:04:53.205 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-02-09T17:04:53.206 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T17:04:54.509 DEBUG:teuthology.orchestra.run.smithi179:> true 2024-02-09T17:04:55.021 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-02-09T17:04:55.022 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "4.18.0-536.el8.x86_64"... 2024-02-09T17:04:55.022 DEBUG:teuthology.orchestra.run.smithi179:> uname -r 2024-02-09T17:04:55.039 INFO:teuthology.orchestra.run.smithi179.stdout:4.18.0-536.el8.x86_64 2024-02-09T17:04:55.039 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-536.el8.x86_64 vs 4.18.0-536.el8.x86_64 2024-02-09T17:04:55.039 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-09T17:04:55.040 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-09T17:04:56.040 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-02-09T17:04:56.040 DEBUG:teuthology.orchestra.run.smithi179:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-09T17:04:56.113 INFO:teuthology.orchestra.run.smithi179.stdout:ttyS1 2024-02-09T17:04:56.127 DEBUG:teuthology.parallel:result is None 2024-02-09T17:05:18.241 DEBUG:teuthology.orchestra.remote:timed out 2024-02-09T17:05:33.243 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-02-09T17:05:33.244 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T17:05:33.607 DEBUG:teuthology.orchestra.run.smithi162:> true 2024-02-09T17:05:33.894 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-02-09T17:05:33.894 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-536.el8.x86_64"... 2024-02-09T17:05:33.894 DEBUG:teuthology.orchestra.run.smithi162:> uname -r 2024-02-09T17:05:33.913 INFO:teuthology.orchestra.run.smithi162.stdout:4.18.0-536.el8.x86_64 2024-02-09T17:05:33.913 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-536.el8.x86_64 vs 4.18.0-536.el8.x86_64 2024-02-09T17:05:33.913 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-09T17:05:33.914 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-09T17:05:34.914 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-02-09T17:05:34.915 DEBUG:teuthology.orchestra.run.smithi162:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-09T17:05:34.986 INFO:teuthology.orchestra.run.smithi162.stdout:ttyS1 2024-02-09T17:05:35.002 DEBUG:teuthology.parallel:result is None 2024-02-09T17:05:35.002 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-09T17:05:35.010 INFO:teuthology.task.internal:Creating test directory... 2024-02-09T17:05:35.011 DEBUG:teuthology.orchestra.run.smithi099:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-09T17:05:35.014 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-09T17:05:35.016 DEBUG:teuthology.orchestra.run.smithi179:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-09T17:05:35.039 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-09T17:05:35.045 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-09T17:05:35.051 INFO:teuthology.task.internal:Creating archive directory... 2024-02-09T17:05:35.052 DEBUG:teuthology.orchestra.run.smithi099:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-09T17:05:35.076 DEBUG:teuthology.orchestra.run.smithi162:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-09T17:05:35.078 DEBUG:teuthology.orchestra.run.smithi179:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-09T17:05:35.117 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-09T17:05:35.122 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-09T17:05:35.123 DEBUG:teuthology.orchestra.run.smithi099:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-09T17:05:35.148 DEBUG:teuthology.orchestra.run.smithi162:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-09T17:05:35.153 DEBUG:teuthology.orchestra.run.smithi179:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-09T17:05:35.179 INFO:teuthology.orchestra.run.smithi099.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-09T17:05:35.187 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-09T17:05:35.189 INFO:teuthology.orchestra.run.smithi099.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-09T17:05:35.196 INFO:teuthology.orchestra.run.smithi179.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-09T17:05:35.198 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-09T17:05:35.208 INFO:teuthology.orchestra.run.smithi179.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-09T17:05:35.209 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-09T17:05:35.216 INFO:teuthology.task.internal:Configuring sudo... 2024-02-09T17:05:35.216 DEBUG:teuthology.orchestra.run.smithi099:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-09T17:05:35.233 DEBUG:teuthology.orchestra.run.smithi162:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-09T17:05:35.242 DEBUG:teuthology.orchestra.run.smithi179:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-09T17:05:35.280 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-09T17:05:35.289 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-09T17:05:35.289 DEBUG:teuthology.orchestra.run.smithi099:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-09T17:05:35.303 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-09T17:05:35.311 DEBUG:teuthology.orchestra.run.smithi179:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-09T17:05:35.339 DEBUG:teuthology.orchestra.run.smithi099:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-09T17:05:35.404 DEBUG:teuthology.orchestra.run.smithi099:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-09T17:05:35.454 DEBUG:teuthology.orchestra.run.smithi099:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-09T17:05:35.539 DEBUG:teuthology.orchestra.run.smithi099:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-09T17:05:35.581 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-02-09T17:05:35.581 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-09T17:05:35.648 DEBUG:teuthology.orchestra.run.smithi162:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-09T17:05:35.699 DEBUG:teuthology.orchestra.run.smithi162:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-09T17:05:35.751 DEBUG:teuthology.orchestra.run.smithi162:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-09T17:05:35.838 DEBUG:teuthology.orchestra.run.smithi162:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-09T17:05:35.883 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-02-09T17:05:35.883 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-09T17:05:35.952 DEBUG:teuthology.orchestra.run.smithi179:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-09T17:05:35.999 DEBUG:teuthology.orchestra.run.smithi179:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-09T17:05:36.057 DEBUG:teuthology.orchestra.run.smithi179:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-09T17:05:36.145 DEBUG:teuthology.orchestra.run.smithi179:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-09T17:05:36.190 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-02-09T17:05:36.190 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-09T17:05:36.259 DEBUG:teuthology.orchestra.run.smithi099:> sudo service rsyslog restart 2024-02-09T17:05:36.262 DEBUG:teuthology.orchestra.run.smithi162:> sudo service rsyslog restart 2024-02-09T17:05:36.264 DEBUG:teuthology.orchestra.run.smithi179:> sudo service rsyslog restart 2024-02-09T17:05:36.325 INFO:teuthology.orchestra.run.smithi162.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-09T17:05:36.329 INFO:teuthology.orchestra.run.smithi099.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-09T17:05:36.376 INFO:teuthology.orchestra.run.smithi179.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-09T17:05:36.763 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-09T17:05:36.771 INFO:teuthology.task.internal:Starting timer... 2024-02-09T17:05:36.771 INFO:teuthology.run_tasks:Running task pcp... 2024-02-09T17:05:36.781 INFO:teuthology.run_tasks:Running task selinux... 2024-02-09T17:05:36.788 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-02-09T17:05:36.789 DEBUG:teuthology.orchestra.run.smithi099:> sudo service auditd rotate 2024-02-09T17:05:36.900 INFO:teuthology.orchestra.run.smithi099.stdout:Rotating logs: 2024-02-09T17:05:36.901 DEBUG:teuthology.orchestra.run.smithi162:> sudo service auditd rotate 2024-02-09T17:05:36.965 INFO:teuthology.orchestra.run.smithi162.stdout:Rotating logs: 2024-02-09T17:05:36.967 DEBUG:teuthology.orchestra.run.smithi179:> sudo service auditd rotate 2024-02-09T17:05:37.035 INFO:teuthology.orchestra.run.smithi179.stdout:Rotating logs: 2024-02-09T17:05:37.037 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-09T17:05:37.037 DEBUG:teuthology.orchestra.run.smithi099:> /usr/sbin/getenforce 2024-02-09T17:05:37.064 INFO:teuthology.orchestra.run.smithi099.stdout:Permissive 2024-02-09T17:05:37.064 DEBUG:teuthology.orchestra.run.smithi162:> /usr/sbin/getenforce 2024-02-09T17:05:37.112 INFO:teuthology.orchestra.run.smithi162.stdout:Permissive 2024-02-09T17:05:37.112 DEBUG:teuthology.orchestra.run.smithi179:> /usr/sbin/getenforce 2024-02-09T17:05:37.159 INFO:teuthology.orchestra.run.smithi179.stdout:Permissive 2024-02-09T17:05:37.160 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi099.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi162.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi179.front.sepia.ceph.com': 'permissive'} 2024-02-09T17:05:37.160 DEBUG:teuthology.orchestra.run.smithi099:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-09T17:05:37.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-09T17:05:37.188 DEBUG:teuthology.orchestra.run.smithi162:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-09T17:05:37.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-09T17:05:37.215 DEBUG:teuthology.orchestra.run.smithi179:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-09T17:05:37.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-09T17:05:37.242 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-09T17:05:37.243 DEBUG:teuthology.orchestra.run.smithi099:> sudo /usr/sbin/setenforce permissive 2024-02-09T17:05:37.269 DEBUG:teuthology.orchestra.run.smithi162:> sudo /usr/sbin/setenforce permissive 2024-02-09T17:05:37.297 DEBUG:teuthology.orchestra.run.smithi179:> sudo /usr/sbin/setenforce permissive 2024-02-09T17:05:37.324 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-09T17:05:37.334 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-09T17:05:37.353 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-09T17:05:37.354 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi099.front.sepia.ceph.com,smithi162.front.sepia.ceph.com,smithi179.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-09T17:11:23.459 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi099.front.sepia.ceph.com'), Remote(name='ubuntu@smithi162.front.sepia.ceph.com'), Remote(name='ubuntu@smithi179.front.sepia.ceph.com')] 2024-02-09T17:11:23.460 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi099.front.sepia.ceph.com' 2024-02-09T17:11:23.462 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T17:11:23.538 DEBUG:teuthology.orchestra.run.smithi099:> true 2024-02-09T17:11:23.607 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi099.front.sepia.ceph.com' 2024-02-09T17:11:23.607 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-02-09T17:11:23.608 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T17:11:23.681 DEBUG:teuthology.orchestra.run.smithi162:> true 2024-02-09T17:11:23.709 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-02-09T17:11:23.709 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-02-09T17:11:23.710 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T17:11:23.788 DEBUG:teuthology.orchestra.run.smithi179:> true 2024-02-09T17:11:23.855 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-02-09T17:11:23.856 INFO:teuthology.run_tasks:Running task clock... 2024-02-09T17:11:23.867 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-09T17:11:23.867 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-09T17:11:23.868 DEBUG:teuthology.orchestra.run.smithi099:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-09T17:11:23.870 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-09T17:11:23.870 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-09T17:11:23.873 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-09T17:11:23.873 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-09T17:11:23.901 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-09T17:11:23.904 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-09T17:11:23.915 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-09T17:11:23.919 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-09T17:11:23.929 INFO:teuthology.orchestra.run.smithi179.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-09T17:11:23.942 INFO:teuthology.orchestra.run.smithi179.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-09T17:11:23.991 INFO:teuthology.orchestra.run.smithi099.stderr:sudo: ntpd: command not found 2024-02-09T17:11:23.998 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: ntpd: command not found 2024-02-09T17:11:24.002 INFO:teuthology.orchestra.run.smithi099.stdout:506 Cannot talk to daemon 2024-02-09T17:11:24.009 INFO:teuthology.orchestra.run.smithi162.stdout:506 Cannot talk to daemon 2024-02-09T17:11:24.009 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: ntpd: command not found 2024-02-09T17:11:24.016 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-09T17:11:24.020 INFO:teuthology.orchestra.run.smithi179.stdout:506 Cannot talk to daemon 2024-02-09T17:11:24.023 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-09T17:11:24.029 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-09T17:11:24.033 INFO:teuthology.orchestra.run.smithi179.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-09T17:11:24.036 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-09T17:11:24.047 INFO:teuthology.orchestra.run.smithi179.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-09T17:11:24.091 INFO:teuthology.orchestra.run.smithi099.stderr:bash: ntpq: command not found 2024-02-09T17:11:24.096 INFO:teuthology.orchestra.run.smithi099.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-09T17:11:24.096 INFO:teuthology.orchestra.run.smithi099.stdout:=============================================================================== 2024-02-09T17:11:24.096 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-09T17:11:24.096 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-09T17:11:24.096 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-09T17:11:24.096 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-09T17:11:24.109 INFO:teuthology.orchestra.run.smithi162.stderr:bash: ntpq: command not found 2024-02-09T17:11:24.111 INFO:teuthology.orchestra.run.smithi179.stderr:bash: ntpq: command not found 2024-02-09T17:11:24.112 INFO:teuthology.orchestra.run.smithi162.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-09T17:11:24.113 INFO:teuthology.orchestra.run.smithi162.stdout:=============================================================================== 2024-02-09T17:11:24.113 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-09T17:11:24.113 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-09T17:11:24.113 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-09T17:11:24.113 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-09T17:11:24.114 INFO:teuthology.orchestra.run.smithi179.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-09T17:11:24.114 INFO:teuthology.orchestra.run.smithi179.stdout:=============================================================================== 2024-02-09T17:11:24.115 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-09T17:11:24.115 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-09T17:11:24.115 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-09T17:11:24.115 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-09T17:11:24.116 INFO:teuthology.run_tasks:Running task pexec... 2024-02-09T17:11:24.124 INFO:teuthology.task.pexec:Executing custom commands... 2024-02-09T17:11:24.125 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi099.front.sepia.ceph.com 2024-02-09T17:11:24.125 DEBUG:teuthology.orchestra.run.smithi099:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-09T17:11:24.125 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi162.front.sepia.ceph.com 2024-02-09T17:11:24.125 DEBUG:teuthology.orchestra.run.smithi162:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-09T17:11:24.126 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi179.front.sepia.ceph.com 2024-02-09T17:11:24.126 DEBUG:teuthology.orchestra.run.smithi179:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-09T17:11:24.923 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:22 ago on Fri 09 Feb 2024 05:10:02 PM UTC. 2024-02-09T17:11:24.943 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:23 ago on Fri 09 Feb 2024 05:10:01 PM UTC. 2024-02-09T17:11:24.945 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:22 ago on Fri 09 Feb 2024 05:10:02 PM UTC. 2024-02-09T17:11:26.840 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-02-09T17:11:26.841 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-02-09T17:11:26.842 INFO:teuthology.orchestra.run.smithi179.stdout: Package Architecture Version Repository Size 2024-02-09T17:11:26.842 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-02-09T17:11:26.842 INFO:teuthology.orchestra.run.smithi179.stdout:Resetting modules: 2024-02-09T17:11:26.842 INFO:teuthology.orchestra.run.smithi179.stdout: container-tools 2024-02-09T17:11:26.842 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:11:26.842 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-02-09T17:11:26.842 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-02-09T17:11:26.842 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:11:26.891 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-02-09T17:11:26.892 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-02-09T17:11:26.893 INFO:teuthology.orchestra.run.smithi099.stdout: Package Architecture Version Repository Size 2024-02-09T17:11:26.893 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-02-09T17:11:26.893 INFO:teuthology.orchestra.run.smithi099.stdout:Resetting modules: 2024-02-09T17:11:26.893 INFO:teuthology.orchestra.run.smithi099.stdout: container-tools 2024-02-09T17:11:26.893 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:11:26.893 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-02-09T17:11:26.893 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-02-09T17:11:26.893 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:11:26.914 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-09T17:11:26.915 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-09T17:11:26.915 INFO:teuthology.orchestra.run.smithi162.stdout: Package Architecture Version Repository Size 2024-02-09T17:11:26.915 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-09T17:11:26.915 INFO:teuthology.orchestra.run.smithi162.stdout:Resetting modules: 2024-02-09T17:11:26.915 INFO:teuthology.orchestra.run.smithi162.stdout: container-tools 2024-02-09T17:11:26.915 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:11:26.916 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-02-09T17:11:26.916 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-09T17:11:26.916 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:11:27.100 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-02-09T17:11:27.142 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-02-09T17:11:27.161 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-09T17:11:27.944 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:25 ago on Fri 09 Feb 2024 05:10:02 PM UTC. 2024-02-09T17:11:27.997 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:25 ago on Fri 09 Feb 2024 05:10:02 PM UTC. 2024-02-09T17:11:28.017 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:26 ago on Fri 09 Feb 2024 05:10:01 PM UTC. 2024-02-09T17:11:29.893 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-02-09T17:11:29.928 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-02-09T17:11:29.928 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-02-09T17:11:29.928 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-02-09T17:11:29.928 INFO:teuthology.orchestra.run.smithi179.stdout:Installing group/module packages: 2024-02-09T17:11:29.928 INFO:teuthology.orchestra.run.smithi179.stdout: aardvark-dns x86_64 2:1.9.0-1.module_el8+804+f131391c appstream 1.0 M 2024-02-09T17:11:29.928 INFO:teuthology.orchestra.run.smithi179.stdout: buildah x86_64 1:1.34.0-1.module_el8+804+f131391c appstream 9.7 M 2024-02-09T17:11:29.928 INFO:teuthology.orchestra.run.smithi179.stdout: cockpit-podman noarch 82-1.module_el8+804+f131391c appstream 759 k 2024-02-09T17:11:29.928 INFO:teuthology.orchestra.run.smithi179.stdout: crun x86_64 1.12-1.module_el8+804+f131391c appstream 248 k 2024-02-09T17:11:29.928 INFO:teuthology.orchestra.run.smithi179.stdout: netavark x86_64 2:1.9.0-1.module_el8+804+f131391c appstream 4.0 M 2024-02-09T17:11:29.928 INFO:teuthology.orchestra.run.smithi179.stdout: python3-podman noarch 4.8.2-1.module_el8+804+f131391c appstream 179 k 2024-02-09T17:11:29.928 INFO:teuthology.orchestra.run.smithi179.stdout: skopeo x86_64 2:1.14.0-2.module_el8+804+f131391c appstream 8.8 M 2024-02-09T17:11:29.929 INFO:teuthology.orchestra.run.smithi179.stdout: toolbox x86_64 0.0.99.4-8.module_el8+804+f131391c appstream 2.5 M 2024-02-09T17:11:29.929 INFO:teuthology.orchestra.run.smithi179.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-02-09T17:11:29.929 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-02-09T17:11:29.929 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-02-09T17:11:29.929 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-02-09T17:11:29.929 INFO:teuthology.orchestra.run.smithi179.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-02-09T17:11:29.929 INFO:teuthology.orchestra.run.smithi179.stdout:Installing weak dependencies: 2024-02-09T17:11:29.929 INFO:teuthology.orchestra.run.smithi179.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-02-09T17:11:29.929 INFO:teuthology.orchestra.run.smithi179.stdout:Installing module profiles: 2024-02-09T17:11:29.929 INFO:teuthology.orchestra.run.smithi179.stdout: container-tools/common 2024-02-09T17:11:29.929 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:11:29.929 INFO:teuthology.orchestra.run.smithi179.stdout:Enabling module streams: 2024-02-09T17:11:29.929 INFO:teuthology.orchestra.run.smithi179.stdout: container-tools rhel8 2024-02-09T17:11:29.929 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:11:29.929 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-02-09T17:11:29.929 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-02-09T17:11:29.930 INFO:teuthology.orchestra.run.smithi179.stdout:Install 13 Packages 2024-02-09T17:11:29.930 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:11:29.930 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 27 M 2024-02-09T17:11:29.930 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 91 M 2024-02-09T17:11:29.930 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-02-09T17:11:29.965 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-02-09T17:11:29.985 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-09T17:11:29.999 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-02-09T17:11:29.999 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2024-02-09T17:11:29.999 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-02-09T17:11:30.000 INFO:teuthology.orchestra.run.smithi099.stdout:Installing group/module packages: 2024-02-09T17:11:30.000 INFO:teuthology.orchestra.run.smithi099.stdout: aardvark-dns x86_64 2:1.9.0-1.module_el8+804+f131391c appstream 1.0 M 2024-02-09T17:11:30.000 INFO:teuthology.orchestra.run.smithi099.stdout: buildah x86_64 1:1.34.0-1.module_el8+804+f131391c appstream 9.7 M 2024-02-09T17:11:30.000 INFO:teuthology.orchestra.run.smithi099.stdout: cockpit-podman noarch 82-1.module_el8+804+f131391c appstream 759 k 2024-02-09T17:11:30.000 INFO:teuthology.orchestra.run.smithi099.stdout: crun x86_64 1.12-1.module_el8+804+f131391c appstream 248 k 2024-02-09T17:11:30.000 INFO:teuthology.orchestra.run.smithi099.stdout: netavark x86_64 2:1.9.0-1.module_el8+804+f131391c appstream 4.0 M 2024-02-09T17:11:30.000 INFO:teuthology.orchestra.run.smithi099.stdout: python3-podman noarch 4.8.2-1.module_el8+804+f131391c appstream 179 k 2024-02-09T17:11:30.000 INFO:teuthology.orchestra.run.smithi099.stdout: skopeo x86_64 2:1.14.0-2.module_el8+804+f131391c appstream 8.8 M 2024-02-09T17:11:30.000 INFO:teuthology.orchestra.run.smithi099.stdout: toolbox x86_64 0.0.99.4-8.module_el8+804+f131391c appstream 2.5 M 2024-02-09T17:11:30.000 INFO:teuthology.orchestra.run.smithi099.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-02-09T17:11:30.000 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2024-02-09T17:11:30.000 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-02-09T17:11:30.000 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-02-09T17:11:30.001 INFO:teuthology.orchestra.run.smithi099.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-02-09T17:11:30.001 INFO:teuthology.orchestra.run.smithi099.stdout:Installing weak dependencies: 2024-02-09T17:11:30.001 INFO:teuthology.orchestra.run.smithi099.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-02-09T17:11:30.001 INFO:teuthology.orchestra.run.smithi099.stdout:Installing module profiles: 2024-02-09T17:11:30.001 INFO:teuthology.orchestra.run.smithi099.stdout: container-tools/common 2024-02-09T17:11:30.001 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:11:30.001 INFO:teuthology.orchestra.run.smithi099.stdout:Enabling module streams: 2024-02-09T17:11:30.001 INFO:teuthology.orchestra.run.smithi099.stdout: container-tools rhel8 2024-02-09T17:11:30.001 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:11:30.001 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-02-09T17:11:30.001 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-02-09T17:11:30.001 INFO:teuthology.orchestra.run.smithi099.stdout:Install 13 Packages 2024-02-09T17:11:30.001 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:11:30.002 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 27 M 2024-02-09T17:11:30.002 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 91 M 2024-02-09T17:11:30.002 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-02-09T17:11:30.020 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-09T17:11:30.020 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-02-09T17:11:30.020 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-09T17:11:30.020 INFO:teuthology.orchestra.run.smithi162.stdout:Installing group/module packages: 2024-02-09T17:11:30.020 INFO:teuthology.orchestra.run.smithi162.stdout: aardvark-dns x86_64 2:1.9.0-1.module_el8+804+f131391c appstream 1.0 M 2024-02-09T17:11:30.020 INFO:teuthology.orchestra.run.smithi162.stdout: buildah x86_64 1:1.34.0-1.module_el8+804+f131391c appstream 9.7 M 2024-02-09T17:11:30.020 INFO:teuthology.orchestra.run.smithi162.stdout: cockpit-podman noarch 82-1.module_el8+804+f131391c appstream 759 k 2024-02-09T17:11:30.020 INFO:teuthology.orchestra.run.smithi162.stdout: crun x86_64 1.12-1.module_el8+804+f131391c appstream 248 k 2024-02-09T17:11:30.020 INFO:teuthology.orchestra.run.smithi162.stdout: netavark x86_64 2:1.9.0-1.module_el8+804+f131391c appstream 4.0 M 2024-02-09T17:11:30.020 INFO:teuthology.orchestra.run.smithi162.stdout: python3-podman noarch 4.8.2-1.module_el8+804+f131391c appstream 179 k 2024-02-09T17:11:30.021 INFO:teuthology.orchestra.run.smithi162.stdout: skopeo x86_64 2:1.14.0-2.module_el8+804+f131391c appstream 8.8 M 2024-02-09T17:11:30.021 INFO:teuthology.orchestra.run.smithi162.stdout: toolbox x86_64 0.0.99.4-8.module_el8+804+f131391c appstream 2.5 M 2024-02-09T17:11:30.021 INFO:teuthology.orchestra.run.smithi162.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-02-09T17:11:30.021 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-02-09T17:11:30.021 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-02-09T17:11:30.021 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-02-09T17:11:30.021 INFO:teuthology.orchestra.run.smithi162.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-02-09T17:11:30.021 INFO:teuthology.orchestra.run.smithi162.stdout:Installing weak dependencies: 2024-02-09T17:11:30.021 INFO:teuthology.orchestra.run.smithi162.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-02-09T17:11:30.021 INFO:teuthology.orchestra.run.smithi162.stdout:Installing module profiles: 2024-02-09T17:11:30.021 INFO:teuthology.orchestra.run.smithi162.stdout: container-tools/common 2024-02-09T17:11:30.021 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:11:30.021 INFO:teuthology.orchestra.run.smithi162.stdout:Enabling module streams: 2024-02-09T17:11:30.021 INFO:teuthology.orchestra.run.smithi162.stdout: container-tools rhel8 2024-02-09T17:11:30.021 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:11:30.022 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-02-09T17:11:30.022 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-09T17:11:30.022 INFO:teuthology.orchestra.run.smithi162.stdout:Install 13 Packages 2024-02-09T17:11:30.022 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:11:30.022 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 27 M 2024-02-09T17:11:30.022 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 91 M 2024-02-09T17:11:30.022 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-02-09T17:11:30.607 INFO:teuthology.orchestra.run.smithi099.stdout:(1/13): cockpit-podman-82-1.module_el8+804+f131 1.3 MB/s | 759 kB 00:00 2024-02-09T17:11:30.687 INFO:teuthology.orchestra.run.smithi179.stdout:(1/13): cockpit-podman-82-1.module_el8+804+f131 1.3 MB/s | 759 kB 00:00 2024-02-09T17:11:30.699 INFO:teuthology.orchestra.run.smithi099.stdout:(2/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 413 kB/s | 38 kB 00:00 2024-02-09T17:11:30.732 INFO:teuthology.orchestra.run.smithi099.stdout:(3/13): aardvark-dns-1.9.0-1.module_el8+804+f13 1.5 MB/s | 1.0 MB 00:00 2024-02-09T17:11:30.745 INFO:teuthology.orchestra.run.smithi179.stdout:(2/13): aardvark-dns-1.9.0-1.module_el8+804+f13 1.6 MB/s | 1.0 MB 00:00 2024-02-09T17:11:30.786 INFO:teuthology.orchestra.run.smithi162.stdout:(1/13): cockpit-podman-82-1.module_el8+804+f131 1.3 MB/s | 759 kB 00:00 2024-02-09T17:11:30.787 INFO:teuthology.orchestra.run.smithi179.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 377 kB/s | 38 kB 00:00 2024-02-09T17:11:30.808 INFO:teuthology.orchestra.run.smithi099.stdout:(4/13): crun-1.12-1.module_el8+804+f131391c.x86 2.2 MB/s | 248 kB 00:00 2024-02-09T17:11:30.846 INFO:teuthology.orchestra.run.smithi179.stdout:(4/13): crun-1.12-1.module_el8+804+f131391c.x86 2.4 MB/s | 248 kB 00:00 2024-02-09T17:11:30.886 INFO:teuthology.orchestra.run.smithi162.stdout:(2/13): aardvark-dns-1.9.0-1.module_el8+804+f13 1.5 MB/s | 1.0 MB 00:00 2024-02-09T17:11:30.908 INFO:teuthology.orchestra.run.smithi099.stdout:(5/13): python3-podman-4.8.2-1.module_el8+804+f 1.8 MB/s | 179 kB 00:00 2024-02-09T17:11:30.911 INFO:teuthology.orchestra.run.smithi162.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 302 kB/s | 38 kB 00:00 2024-02-09T17:11:30.947 INFO:teuthology.orchestra.run.smithi179.stdout:(5/13): python3-podman-4.8.2-1.module_el8+804+f 1.8 MB/s | 179 kB 00:00 2024-02-09T17:11:30.986 INFO:teuthology.orchestra.run.smithi162.stdout:(4/13): crun-1.12-1.module_el8+804+f131391c.x86 2.4 MB/s | 248 kB 00:00 2024-02-09T17:11:31.047 INFO:teuthology.orchestra.run.smithi179.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 248 kB/s | 25 kB 00:00 2024-02-09T17:11:31.059 INFO:teuthology.orchestra.run.smithi099.stdout:(6/13): buildah-1.34.0-1.module_el8+804+f131391 9.5 MB/s | 9.7 MB 00:01 2024-02-09T17:11:31.079 INFO:teuthology.orchestra.run.smithi162.stdout:(5/13): python3-podman-4.8.2-1.module_el8+804+f 1.9 MB/s | 179 kB 00:00 2024-02-09T17:11:31.084 INFO:teuthology.orchestra.run.smithi099.stdout:(7/13): python3-pytoml-0.1.14-5.git7dea353.el8. 141 kB/s | 25 kB 00:00 2024-02-09T17:11:31.139 INFO:teuthology.orchestra.run.smithi179.stdout:(7/13): netavark-1.9.0-1.module_el8+804+f131391 11 MB/s | 4.0 MB 00:00 2024-02-09T17:11:31.151 INFO:teuthology.orchestra.run.smithi099.stdout:(8/13): netavark-1.9.0-1.module_el8+804+f131391 9.5 MB/s | 4.0 MB 00:00 2024-02-09T17:11:31.164 INFO:teuthology.orchestra.run.smithi179.stdout:(8/13): python3-pyxdg-0.25-16.el8.noarch.rpm 803 kB/s | 94 kB 00:00 2024-02-09T17:11:31.176 INFO:teuthology.orchestra.run.smithi099.stdout:(9/13): python3-pyxdg-0.25-16.el8.noarch.rpm 802 kB/s | 94 kB 00:00 2024-02-09T17:11:31.221 INFO:teuthology.orchestra.run.smithi162.stdout:(6/13): buildah-1.34.0-1.module_el8+804+f131391 9.6 MB/s | 9.7 MB 00:01 2024-02-09T17:11:31.246 INFO:teuthology.orchestra.run.smithi162.stdout:(7/13): python3-pytoml-0.1.14-5.git7dea353.el8. 148 kB/s | 25 kB 00:00 2024-02-09T17:11:31.273 INFO:teuthology.orchestra.run.smithi179.stdout:(9/13): buildah-1.34.0-1.module_el8+804+f131391 8.4 MB/s | 9.7 MB 00:01 2024-02-09T17:11:31.276 INFO:teuthology.orchestra.run.smithi099.stdout:(10/13): toolbox-0.0.99.4-8.module_el8+804+f131 20 MB/s | 2.5 MB 00:00 2024-02-09T17:11:31.304 INFO:teuthology.orchestra.run.smithi162.stdout:(8/13): netavark-1.9.0-1.module_el8+804+f131391 10 MB/s | 4.0 MB 00:00 2024-02-09T17:11:31.321 INFO:teuthology.orchestra.run.smithi099.stdout:(11/13): udica-0.2.6-20.module_el8+804+f131391c 364 kB/s | 53 kB 00:00 2024-02-09T17:11:31.329 INFO:teuthology.orchestra.run.smithi162.stdout:(9/13): python3-pyxdg-0.25-16.el8.noarch.rpm 864 kB/s | 94 kB 00:00 2024-02-09T17:11:31.373 INFO:teuthology.orchestra.run.smithi179.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 528 kB/s | 53 kB 00:00 2024-02-09T17:11:31.377 INFO:teuthology.orchestra.run.smithi099.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 409 kB/s | 41 kB 00:00 2024-02-09T17:11:31.423 INFO:teuthology.orchestra.run.smithi179.stdout:(11/13): toolbox-0.0.99.4-8.module_el8+804+f131 9.7 MB/s | 2.5 MB 00:00 2024-02-09T17:11:31.438 INFO:teuthology.orchestra.run.smithi162.stdout:(10/13): toolbox-0.0.99.4-8.module_el8+804+f131 19 MB/s | 2.5 MB 00:00 2024-02-09T17:11:31.455 INFO:teuthology.orchestra.run.smithi162.stdout:(11/13): udica-0.2.6-20.module_el8+804+f131391c 420 kB/s | 53 kB 00:00 2024-02-09T17:11:31.473 INFO:teuthology.orchestra.run.smithi179.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 409 kB/s | 41 kB 00:00 2024-02-09T17:11:31.531 INFO:teuthology.orchestra.run.smithi162.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 446 kB/s | 41 kB 00:00 2024-02-09T17:11:31.599 INFO:teuthology.orchestra.run.smithi179.stdout:(13/13): skopeo-1.14.0-2.module_el8+804+f131391 19 MB/s | 8.8 MB 00:00 2024-02-09T17:11:31.599 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-02-09T17:11:31.600 INFO:teuthology.orchestra.run.smithi179.stdout:Total 16 MB/s | 27 MB 00:01 2024-02-09T17:11:31.644 INFO:teuthology.orchestra.run.smithi099.stdout:(13/13): skopeo-1.14.0-2.module_el8+804+f131391 16 MB/s | 8.8 MB 00:00 2024-02-09T17:11:31.644 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-02-09T17:11:31.644 INFO:teuthology.orchestra.run.smithi099.stdout:Total 17 MB/s | 27 MB 00:01 2024-02-09T17:11:31.781 INFO:teuthology.orchestra.run.smithi162.stdout:(13/13): skopeo-1.14.0-2.module_el8+804+f131391 16 MB/s | 8.8 MB 00:00 2024-02-09T17:11:31.781 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-02-09T17:11:31.781 INFO:teuthology.orchestra.run.smithi162.stdout:Total 16 MB/s | 27 MB 00:01 2024-02-09T17:11:31.872 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-02-09T17:11:31.892 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-02-09T17:11:31.893 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-02-09T17:11:31.917 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-02-09T17:11:31.938 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-02-09T17:11:31.939 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-02-09T17:11:32.054 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-02-09T17:11:32.075 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-02-09T17:11:32.075 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-02-09T17:11:32.147 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-02-09T17:11:32.150 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-02-09T17:11:32.203 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-02-09T17:11:32.207 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-02-09T17:11:32.327 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-02-09T17:11:32.331 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-02-09T17:11:32.821 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-02-09T17:11:32.894 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-02-09T17:11:32.987 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-02-09T17:11:33.054 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-02-09T17:11:33.095 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-02-09T17:11:33.173 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-02-09T17:11:33.241 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-02-09T17:11:33.290 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-02-09T17:11:33.335 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-02-09T17:11:33.468 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2024-02-09T17:11:33.483 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2024-02-09T17:11:33.533 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2024-02-09T17:11:33.708 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-09T17:11:33.711 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-09T17:11:33.770 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-09T17:11:34.271 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86 5/13 2024-02-09T17:11:34.281 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86 5/13 2024-02-09T17:11:34.300 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86 5/13 2024-02-09T17:11:35.248 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 6/13 2024-02-09T17:11:35.263 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 6/13 2024-02-09T17:11:35.264 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 6/13 2024-02-09T17:11:35.494 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 7/13 2024-02-09T17:11:35.505 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 7/13 2024-02-09T17:11:35.515 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 7/13 2024-02-09T17:11:35.698 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : crun-1.12-1.module_el8+804+f131391c.x86_64 8/13 2024-02-09T17:11:35.701 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : crun-1.12-1.module_el8+804+f131391c.x86_64 8/13 2024-02-09T17:11:35.744 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : crun-1.12-1.module_el8+804+f131391c.x86_64 8/13 2024-02-09T17:11:35.943 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-podman-4.8.2-1.module_el8+804+f131391c.noa 9/13 2024-02-09T17:11:35.947 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-podman-4.8.2-1.module_el8+804+f131391c.noa 9/13 2024-02-09T17:11:35.987 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-podman-4.8.2-1.module_el8+804+f131391c.noa 9/13 2024-02-09T17:11:36.336 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 10/13 2024-02-09T17:11:36.359 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 10/13 2024-02-09T17:11:36.395 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 10/13 2024-02-09T17:11:37.280 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 11/13 2024-02-09T17:11:37.310 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 11/13 2024-02-09T17:11:37.314 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 11/13 2024-02-09T17:11:37.477 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 12/13 2024-02-09T17:11:37.564 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 12/13 2024-02-09T17:11:37.582 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 12/13 2024-02-09T17:11:37.671 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : cockpit-podman-82-1.module_el8+804+f131391c.noarch 13/13 2024-02-09T17:11:37.765 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : cockpit-podman-82-1.module_el8+804+f131391c.noarch 13/13 2024-02-09T17:11:37.767 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : cockpit-podman-82-1.module_el8+804+f131391c.noarch 13/13 2024-02-09T17:11:38.110 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: cockpit-podman-82-1.module_el8+804+f131391c.noarch 13/13 2024-02-09T17:11:38.110 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86 1/13 2024-02-09T17:11:38.110 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 2/13 2024-02-09T17:11:38.110 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : cockpit-podman-82-1.module_el8+804+f131391c.noarch 3/13 2024-02-09T17:11:38.111 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-09T17:11:38.111 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : crun-1.12-1.module_el8+804+f131391c.x86_64 5/13 2024-02-09T17:11:38.111 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 6/13 2024-02-09T17:11:38.111 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-podman-4.8.2-1.module_el8+804+f131391c.noa 7/13 2024-02-09T17:11:38.111 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-02-09T17:11:38.111 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-02-09T17:11:38.111 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 10/13 2024-02-09T17:11:38.111 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 11/13 2024-02-09T17:11:38.111 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-09T17:11:38.189 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: cockpit-podman-82-1.module_el8+804+f131391c.noarch 13/13 2024-02-09T17:11:38.189 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86 1/13 2024-02-09T17:11:38.190 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 2/13 2024-02-09T17:11:38.190 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : cockpit-podman-82-1.module_el8+804+f131391c.noarch 3/13 2024-02-09T17:11:38.190 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-09T17:11:38.190 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : crun-1.12-1.module_el8+804+f131391c.x86_64 5/13 2024-02-09T17:11:38.190 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 6/13 2024-02-09T17:11:38.190 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-podman-4.8.2-1.module_el8+804+f131391c.noa 7/13 2024-02-09T17:11:38.190 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-02-09T17:11:38.190 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-02-09T17:11:38.190 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 10/13 2024-02-09T17:11:38.190 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 11/13 2024-02-09T17:11:38.190 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-09T17:11:38.205 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: cockpit-podman-82-1.module_el8+804+f131391c.noarch 13/13 2024-02-09T17:11:38.207 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86 1/13 2024-02-09T17:11:38.207 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 2/13 2024-02-09T17:11:38.207 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : cockpit-podman-82-1.module_el8+804+f131391c.noarch 3/13 2024-02-09T17:11:38.207 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-09T17:11:38.207 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : crun-1.12-1.module_el8+804+f131391c.x86_64 5/13 2024-02-09T17:11:38.207 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 6/13 2024-02-09T17:11:38.207 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-podman-4.8.2-1.module_el8+804+f131391c.noa 7/13 2024-02-09T17:11:38.207 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-02-09T17:11:38.207 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-02-09T17:11:38.208 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 10/13 2024-02-09T17:11:38.208 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 11/13 2024-02-09T17:11:38.208 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-09T17:11:38.623 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-02-09T17:11:38.623 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:11:38.623 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-02-09T17:11:38.623 INFO:teuthology.orchestra.run.smithi099.stdout: aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86_64 2024-02-09T17:11:38.624 INFO:teuthology.orchestra.run.smithi099.stdout: buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 2024-02-09T17:11:38.624 INFO:teuthology.orchestra.run.smithi099.stdout: cockpit-podman-82-1.module_el8+804+f131391c.noarch 2024-02-09T17:11:38.624 INFO:teuthology.orchestra.run.smithi099.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-02-09T17:11:38.624 INFO:teuthology.orchestra.run.smithi099.stdout: crun-1.12-1.module_el8+804+f131391c.x86_64 2024-02-09T17:11:38.624 INFO:teuthology.orchestra.run.smithi099.stdout: netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 2024-02-09T17:11:38.624 INFO:teuthology.orchestra.run.smithi099.stdout: python3-podman-4.8.2-1.module_el8+804+f131391c.noarch 2024-02-09T17:11:38.624 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-09T17:11:38.624 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-09T17:11:38.624 INFO:teuthology.orchestra.run.smithi099.stdout: skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 2024-02-09T17:11:38.624 INFO:teuthology.orchestra.run.smithi099.stdout: toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 2024-02-09T17:11:38.624 INFO:teuthology.orchestra.run.smithi099.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-02-09T17:11:38.624 INFO:teuthology.orchestra.run.smithi099.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-09T17:11:38.624 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:11:38.625 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-02-09T17:11:38.668 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-02-09T17:11:38.668 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:11:38.668 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-02-09T17:11:38.668 INFO:teuthology.orchestra.run.smithi162.stdout: aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86_64 2024-02-09T17:11:38.668 INFO:teuthology.orchestra.run.smithi162.stdout: buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 2024-02-09T17:11:38.668 INFO:teuthology.orchestra.run.smithi162.stdout: cockpit-podman-82-1.module_el8+804+f131391c.noarch 2024-02-09T17:11:38.669 INFO:teuthology.orchestra.run.smithi162.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-02-09T17:11:38.669 INFO:teuthology.orchestra.run.smithi162.stdout: crun-1.12-1.module_el8+804+f131391c.x86_64 2024-02-09T17:11:38.669 INFO:teuthology.orchestra.run.smithi162.stdout: netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 2024-02-09T17:11:38.669 INFO:teuthology.orchestra.run.smithi162.stdout: python3-podman-4.8.2-1.module_el8+804+f131391c.noarch 2024-02-09T17:11:38.669 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-09T17:11:38.669 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-09T17:11:38.669 INFO:teuthology.orchestra.run.smithi162.stdout: skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 2024-02-09T17:11:38.669 INFO:teuthology.orchestra.run.smithi162.stdout: toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 2024-02-09T17:11:38.669 INFO:teuthology.orchestra.run.smithi162.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-02-09T17:11:38.669 INFO:teuthology.orchestra.run.smithi162.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-09T17:11:38.669 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:11:38.669 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-09T17:11:38.713 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-02-09T17:11:38.714 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:11:38.714 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-02-09T17:11:38.714 INFO:teuthology.orchestra.run.smithi179.stdout: aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86_64 2024-02-09T17:11:38.714 INFO:teuthology.orchestra.run.smithi179.stdout: buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 2024-02-09T17:11:38.714 INFO:teuthology.orchestra.run.smithi179.stdout: cockpit-podman-82-1.module_el8+804+f131391c.noarch 2024-02-09T17:11:38.714 INFO:teuthology.orchestra.run.smithi179.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-02-09T17:11:38.714 INFO:teuthology.orchestra.run.smithi179.stdout: crun-1.12-1.module_el8+804+f131391c.x86_64 2024-02-09T17:11:38.714 INFO:teuthology.orchestra.run.smithi179.stdout: netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 2024-02-09T17:11:38.714 INFO:teuthology.orchestra.run.smithi179.stdout: python3-podman-4.8.2-1.module_el8+804+f131391c.noarch 2024-02-09T17:11:38.714 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-09T17:11:38.714 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-09T17:11:38.714 INFO:teuthology.orchestra.run.smithi179.stdout: skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 2024-02-09T17:11:38.714 INFO:teuthology.orchestra.run.smithi179.stdout: toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 2024-02-09T17:11:38.714 INFO:teuthology.orchestra.run.smithi179.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-02-09T17:11:38.714 INFO:teuthology.orchestra.run.smithi179.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-09T17:11:38.715 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:11:38.715 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-02-09T17:11:38.854 DEBUG:teuthology.parallel:result is None 2024-02-09T17:11:38.945 DEBUG:teuthology.parallel:result is None 2024-02-09T17:11:39.043 DEBUG:teuthology.parallel:result is None 2024-02-09T17:11:39.043 INFO:teuthology.run_tasks:Running task install... 2024-02-09T17:11:39.052 DEBUG:teuthology.task.install:project ceph 2024-02-09T17:11:39.052 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '720625d38f7f95e6fa31efce87133247d9d28517'}} 2024-02-09T17:11:39.053 DEBUG:teuthology.task.install:config {'branch': 'quincy', 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'flavor': 'default', 'sha1': '720625d38f7f95e6fa31efce87133247d9d28517'} 2024-02-09T17:11:39.053 INFO:teuthology.task.install:Using flavor: default 2024-02-09T17:11:39.058 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-09T17:11:39.058 INFO:teuthology.task.install:extra packages: [] 2024-02-09T17:11:39.058 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '720625d38f7f95e6fa31efce87133247d9d28517', 'tag': None, 'wait_for_package': False} 2024-02-09T17:11:39.059 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-09T17:11:39.059 INFO:teuthology.packaging:ref: None 2024-02-09T17:11:39.059 INFO:teuthology.packaging:tag: None 2024-02-09T17:11:39.059 INFO:teuthology.packaging:branch: quincy 2024-02-09T17:11:39.059 INFO:teuthology.packaging:sha1: 720625d38f7f95e6fa31efce87133247d9d28517 2024-02-09T17:11:39.059 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=quincy 2024-02-09T17:11:39.061 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '720625d38f7f95e6fa31efce87133247d9d28517', 'tag': None, 'wait_for_package': False} 2024-02-09T17:11:39.061 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-09T17:11:39.062 INFO:teuthology.packaging:ref: None 2024-02-09T17:11:39.062 INFO:teuthology.packaging:tag: None 2024-02-09T17:11:39.062 INFO:teuthology.packaging:branch: quincy 2024-02-09T17:11:39.062 INFO:teuthology.packaging:sha1: 720625d38f7f95e6fa31efce87133247d9d28517 2024-02-09T17:11:39.062 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=quincy 2024-02-09T17:11:39.063 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '720625d38f7f95e6fa31efce87133247d9d28517', 'tag': None, 'wait_for_package': False} 2024-02-09T17:11:39.063 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-09T17:11:39.063 INFO:teuthology.packaging:ref: None 2024-02-09T17:11:39.063 INFO:teuthology.packaging:tag: None 2024-02-09T17:11:39.063 INFO:teuthology.packaging:branch: quincy 2024-02-09T17:11:39.063 INFO:teuthology.packaging:sha1: 720625d38f7f95e6fa31efce87133247d9d28517 2024-02-09T17:11:39.063 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=quincy 2024-02-09T17:11:39.328 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/quincy/89a83b9871e934592f31c3d4eaa9e0dac74770e7/centos/8/flavors/default/ 2024-02-09T17:11:39.328 INFO:teuthology.task.install.rpm:Package version is 17.2.7-499.g89a83b98 2024-02-09T17:11:39.376 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/quincy/89a83b9871e934592f31c3d4eaa9e0dac74770e7/centos/8/flavors/default/ 2024-02-09T17:11:39.376 INFO:teuthology.task.install.rpm:Package version is 17.2.7-499.g89a83b98 2024-02-09T17:11:39.380 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/quincy/89a83b9871e934592f31c3d4eaa9e0dac74770e7/centos/8/flavors/default/ 2024-02-09T17:11:39.380 INFO:teuthology.task.install.rpm:Package version is 17.2.7-499.g89a83b98 2024-02-09T17:11:39.473 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/quincy/89a83b9871e934592f31c3d4eaa9e0dac74770e7/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/quincy/89a83b9871e934592f31c3d4eaa9e0dac74770e7/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/quincy/89a83b9871e934592f31c3d4eaa9e0dac74770e7/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-09T17:11:39.473 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-02-09T17:11:39.474 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-09T17:11:39.502 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-09T17:11:39.502 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-09T17:11:39.502 INFO:teuthology.packaging:ref: None 2024-02-09T17:11:39.502 INFO:teuthology.packaging:tag: None 2024-02-09T17:11:39.502 INFO:teuthology.packaging:branch: quincy 2024-02-09T17:11:39.502 INFO:teuthology.packaging:sha1: 720625d38f7f95e6fa31efce87133247d9d28517 2024-02-09T17:11:39.502 DEBUG:teuthology.orchestra.run.smithi179:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-09T17:11:39.509 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/quincy/89a83b9871e934592f31c3d4eaa9e0dac74770e7/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/quincy/89a83b9871e934592f31c3d4eaa9e0dac74770e7/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/quincy/89a83b9871e934592f31c3d4eaa9e0dac74770e7/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-09T17:11:39.509 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-02-09T17:11:39.509 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-09T17:11:39.538 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-09T17:11:39.539 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-09T17:11:39.539 INFO:teuthology.packaging:ref: None 2024-02-09T17:11:39.539 INFO:teuthology.packaging:tag: None 2024-02-09T17:11:39.539 INFO:teuthology.packaging:branch: quincy 2024-02-09T17:11:39.539 INFO:teuthology.packaging:sha1: 720625d38f7f95e6fa31efce87133247d9d28517 2024-02-09T17:11:39.539 DEBUG:teuthology.orchestra.run.smithi162:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-09T17:11:39.539 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/quincy/89a83b9871e934592f31c3d4eaa9e0dac74770e7/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/quincy/89a83b9871e934592f31c3d4eaa9e0dac74770e7/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/quincy/89a83b9871e934592f31c3d4eaa9e0dac74770e7/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-09T17:11:39.539 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-02-09T17:11:39.539 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-09T17:11:39.566 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-09T17:11:39.567 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-09T17:11:39.567 INFO:teuthology.packaging:ref: None 2024-02-09T17:11:39.567 INFO:teuthology.packaging:tag: None 2024-02-09T17:11:39.567 INFO:teuthology.packaging:branch: quincy 2024-02-09T17:11:39.567 INFO:teuthology.packaging:sha1: 720625d38f7f95e6fa31efce87133247d9d28517 2024-02-09T17:11:39.567 DEBUG:teuthology.orchestra.run.smithi099:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-09T17:11:39.570 DEBUG:teuthology.orchestra.run.smithi179:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-09T17:11:39.607 DEBUG:teuthology.orchestra.run.smithi162:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-09T17:11:39.633 DEBUG:teuthology.orchestra.run.smithi099:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-09T17:11:39.647 DEBUG:teuthology.orchestra.run.smithi179:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-09T17:11:39.684 DEBUG:teuthology.orchestra.run.smithi162:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-09T17:11:39.709 DEBUG:teuthology.orchestra.run.smithi099:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-09T17:11:39.714 INFO:teuthology.orchestra.run.smithi179.stdout:check_obsoletes = 1 2024-02-09T17:11:39.715 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum clean all 2024-02-09T17:11:39.751 INFO:teuthology.orchestra.run.smithi162.stdout:check_obsoletes = 1 2024-02-09T17:11:39.752 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum clean all 2024-02-09T17:11:39.776 INFO:teuthology.orchestra.run.smithi099.stdout:check_obsoletes = 1 2024-02-09T17:11:39.777 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum clean all 2024-02-09T17:11:39.976 INFO:teuthology.orchestra.run.smithi179.stdout:76 files removed 2024-02-09T17:11:40.004 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-radosgw 2024-02-09T17:11:40.013 INFO:teuthology.orchestra.run.smithi162.stdout:76 files removed 2024-02-09T17:11:40.039 INFO:teuthology.orchestra.run.smithi099.stdout:76 files removed 2024-02-09T17:11:40.041 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-radosgw 2024-02-09T17:11:40.067 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-radosgw 2024-02-09T17:11:40.863 INFO:teuthology.orchestra.run.smithi099.stdout:ceph packages for x86_64 202 kB/s | 78 kB 00:00 2024-02-09T17:11:40.880 INFO:teuthology.orchestra.run.smithi162.stdout:ceph packages for x86_64 181 kB/s | 78 kB 00:00 2024-02-09T17:11:40.900 INFO:teuthology.orchestra.run.smithi179.stdout:ceph packages for x86_64 160 kB/s | 78 kB 00:00 2024-02-09T17:11:41.189 INFO:teuthology.orchestra.run.smithi162.stdout:ceph noarch packages 34 kB/s | 9.1 kB 00:00 2024-02-09T17:11:41.195 INFO:teuthology.orchestra.run.smithi099.stdout:ceph noarch packages 31 kB/s | 9.1 kB 00:00 2024-02-09T17:11:41.201 INFO:teuthology.orchestra.run.smithi179.stdout:ceph noarch packages 35 kB/s | 9.1 kB 00:00 2024-02-09T17:11:41.479 INFO:teuthology.orchestra.run.smithi099.stdout:ceph source packages 5.7 kB/s | 1.5 kB 00:00 2024-02-09T17:11:41.481 INFO:teuthology.orchestra.run.smithi162.stdout:ceph source packages 5.5 kB/s | 1.5 kB 00:00 2024-02-09T17:11:41.502 INFO:teuthology.orchestra.run.smithi179.stdout:ceph source packages 5.3 kB/s | 1.5 kB 00:00 2024-02-09T17:11:43.230 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 8 - AppStream 16 MB/s | 27 MB 00:01 2024-02-09T17:11:43.394 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 8 - AppStream 14 MB/s | 27 MB 00:01 2024-02-09T17:11:43.546 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 8 - AppStream 13 MB/s | 27 MB 00:02 2024-02-09T17:11:48.930 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 8 - BaseOS 12 MB/s | 10 MB 00:00 2024-02-09T17:11:49.058 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 8 - BaseOS 14 MB/s | 10 MB 00:00 2024-02-09T17:11:49.395 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 8 - BaseOS 9.9 MB/s | 10 MB 00:01 2024-02-09T17:11:50.905 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 8 - Extras 44 kB/s | 18 kB 00:00 2024-02-09T17:11:50.933 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 8 - Extras 62 kB/s | 18 kB 00:00 2024-02-09T17:11:51.296 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 8 - Extras 73 kB/s | 18 kB 00:00 2024-02-09T17:11:51.556 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 8 - Extras common packages 13 kB/s | 6.9 kB 00:00 2024-02-09T17:11:51.583 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 8 - Extras common packages 13 kB/s | 6.9 kB 00:00 2024-02-09T17:11:51.771 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 8 - Extras common packages 19 kB/s | 6.9 kB 00:00 2024-02-09T17:11:52.476 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 8 - PowerTools 5.9 MB/s | 4.8 MB 00:00 2024-02-09T17:11:52.500 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 8 - PowerTools 5.6 MB/s | 4.8 MB 00:00 2024-02-09T17:11:53.439 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 8 - PowerTools 3.0 MB/s | 4.8 MB 00:01 2024-02-09T17:11:53.536 INFO:teuthology.orchestra.run.smithi162.stdout:Copr repo for python3-asyncssh owned by ceph 22 kB/s | 3.5 kB 00:00 2024-02-09T17:11:53.550 INFO:teuthology.orchestra.run.smithi099.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2024-02-09T17:11:54.490 INFO:teuthology.orchestra.run.smithi179.stdout:Copr repo for python3-asyncssh owned by ceph 24 kB/s | 3.5 kB 00:00 2024-02-09T17:11:54.622 INFO:teuthology.orchestra.run.smithi162.stdout:Extra Packages for Enterprise Linux 16 MB/s | 16 MB 00:00 2024-02-09T17:11:54.826 INFO:teuthology.orchestra.run.smithi099.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2024-02-09T17:11:55.200 INFO:teuthology.orchestra.run.smithi179.stdout:Extra Packages for Enterprise Linux 26 MB/s | 16 MB 00:00 2024-02-09T17:11:58.575 INFO:teuthology.orchestra.run.smithi099.stdout:lab-extras 577 kB/s | 24 kB 00:00 2024-02-09T17:11:58.687 INFO:teuthology.orchestra.run.smithi162.stdout:lab-extras 105 kB/s | 24 kB 00:00 2024-02-09T17:11:58.950 INFO:teuthology.orchestra.run.smithi179.stdout:lab-extras 417 kB/s | 24 kB 00:00 2024-02-09T17:12:02.047 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-02-09T17:12:02.050 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-02-09T17:12:02.050 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2024-02-09T17:12:02.050 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-02-09T17:12:02.050 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-02-09T17:12:02.050 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-radosgw x86_64 2:17.2.7-499.g89a83b98.el8 ceph 17 M 2024-02-09T17:12:02.050 INFO:teuthology.orchestra.run.smithi099.stdout:Upgrading: 2024-02-09T17:12:02.050 INFO:teuthology.orchestra.run.smithi099.stdout: librados2 x86_64 2:17.2.7-499.g89a83b98.el8 ceph 3.5 M 2024-02-09T17:12:02.050 INFO:teuthology.orchestra.run.smithi099.stdout: librbd1 x86_64 2:17.2.7-499.g89a83b98.el8 ceph 3.6 M 2024-02-09T17:12:02.051 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2024-02-09T17:12:02.051 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-base x86_64 2:17.2.7-499.g89a83b98.el8 ceph 5.2 M 2024-02-09T17:12:02.051 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-common x86_64 2:17.2.7-499.g89a83b98.el8 ceph 23 M 2024-02-09T17:12:02.051 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-selinux x86_64 2:17.2.7-499.g89a83b98.el8 ceph 24 k 2024-02-09T17:12:02.051 INFO:teuthology.orchestra.run.smithi099.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-02-09T17:12:02.051 INFO:teuthology.orchestra.run.smithi099.stdout: libcephfs2 x86_64 2:17.2.7-499.g89a83b98.el8 ceph 798 k 2024-02-09T17:12:02.051 INFO:teuthology.orchestra.run.smithi099.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-02-09T17:12:02.051 INFO:teuthology.orchestra.run.smithi099.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-02-09T17:12:02.051 INFO:teuthology.orchestra.run.smithi099.stdout: libradosstriper1 x86_64 2:17.2.7-499.g89a83b98.el8 ceph 507 k 2024-02-09T17:12:02.051 INFO:teuthology.orchestra.run.smithi099.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-02-09T17:12:02.051 INFO:teuthology.orchestra.run.smithi099.stdout: librgw2 x86_64 2:17.2.7-499.g89a83b98.el8 ceph 5.6 M 2024-02-09T17:12:02.051 INFO:teuthology.orchestra.run.smithi099.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-02-09T17:12:02.051 INFO:teuthology.orchestra.run.smithi099.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-02-09T17:12:02.052 INFO:teuthology.orchestra.run.smithi099.stdout: python3-ceph-argparse x86_64 2:17.2.7-499.g89a83b98.el8 ceph 45 k 2024-02-09T17:12:02.052 INFO:teuthology.orchestra.run.smithi099.stdout: python3-ceph-common x86_64 2:17.2.7-499.g89a83b98.el8 ceph 115 k 2024-02-09T17:12:02.052 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cephfs x86_64 2:17.2.7-499.g89a83b98.el8 ceph 190 k 2024-02-09T17:12:02.052 INFO:teuthology.orchestra.run.smithi099.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-02-09T17:12:02.052 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rados x86_64 2:17.2.7-499.g89a83b98.el8 ceph 372 k 2024-02-09T17:12:02.052 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rbd x86_64 2:17.2.7-499.g89a83b98.el8 ceph 358 k 2024-02-09T17:12:02.052 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rgw x86_64 2:17.2.7-499.g89a83b98.el8 ceph 107 k 2024-02-09T17:12:02.052 INFO:teuthology.orchestra.run.smithi099.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-02-09T17:12:02.052 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:12:02.052 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-02-09T17:12:02.052 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-02-09T17:12:02.052 INFO:teuthology.orchestra.run.smithi099.stdout:Install 21 Packages 2024-02-09T17:12:02.052 INFO:teuthology.orchestra.run.smithi099.stdout:Upgrade 2 Packages 2024-02-09T17:12:02.053 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:12:02.053 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 64 M 2024-02-09T17:12:02.053 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-02-09T17:12:02.182 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-09T17:12:02.185 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-09T17:12:02.185 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-02-09T17:12:02.185 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-09T17:12:02.185 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-02-09T17:12:02.185 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-radosgw x86_64 2:17.2.7-499.g89a83b98.el8 ceph 17 M 2024-02-09T17:12:02.185 INFO:teuthology.orchestra.run.smithi162.stdout:Upgrading: 2024-02-09T17:12:02.185 INFO:teuthology.orchestra.run.smithi162.stdout: librados2 x86_64 2:17.2.7-499.g89a83b98.el8 ceph 3.5 M 2024-02-09T17:12:02.185 INFO:teuthology.orchestra.run.smithi162.stdout: librbd1 x86_64 2:17.2.7-499.g89a83b98.el8 ceph 3.6 M 2024-02-09T17:12:02.185 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-02-09T17:12:02.186 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-base x86_64 2:17.2.7-499.g89a83b98.el8 ceph 5.2 M 2024-02-09T17:12:02.186 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-common x86_64 2:17.2.7-499.g89a83b98.el8 ceph 23 M 2024-02-09T17:12:02.186 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-selinux x86_64 2:17.2.7-499.g89a83b98.el8 ceph 24 k 2024-02-09T17:12:02.186 INFO:teuthology.orchestra.run.smithi162.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-02-09T17:12:02.186 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs2 x86_64 2:17.2.7-499.g89a83b98.el8 ceph 798 k 2024-02-09T17:12:02.186 INFO:teuthology.orchestra.run.smithi162.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-02-09T17:12:02.187 INFO:teuthology.orchestra.run.smithi162.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-02-09T17:12:02.187 INFO:teuthology.orchestra.run.smithi162.stdout: libradosstriper1 x86_64 2:17.2.7-499.g89a83b98.el8 ceph 507 k 2024-02-09T17:12:02.187 INFO:teuthology.orchestra.run.smithi162.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-02-09T17:12:02.187 INFO:teuthology.orchestra.run.smithi162.stdout: librgw2 x86_64 2:17.2.7-499.g89a83b98.el8 ceph 5.6 M 2024-02-09T17:12:02.187 INFO:teuthology.orchestra.run.smithi162.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-02-09T17:12:02.187 INFO:teuthology.orchestra.run.smithi162.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-02-09T17:12:02.188 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-argparse x86_64 2:17.2.7-499.g89a83b98.el8 ceph 45 k 2024-02-09T17:12:02.188 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-common x86_64 2:17.2.7-499.g89a83b98.el8 ceph 115 k 2024-02-09T17:12:02.188 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cephfs x86_64 2:17.2.7-499.g89a83b98.el8 ceph 190 k 2024-02-09T17:12:02.188 INFO:teuthology.orchestra.run.smithi162.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-02-09T17:12:02.188 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rados x86_64 2:17.2.7-499.g89a83b98.el8 ceph 372 k 2024-02-09T17:12:02.188 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rbd x86_64 2:17.2.7-499.g89a83b98.el8 ceph 358 k 2024-02-09T17:12:02.188 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rgw x86_64 2:17.2.7-499.g89a83b98.el8 ceph 107 k 2024-02-09T17:12:02.189 INFO:teuthology.orchestra.run.smithi162.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-02-09T17:12:02.189 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:12:02.189 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-02-09T17:12:02.189 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-09T17:12:02.189 INFO:teuthology.orchestra.run.smithi162.stdout:Install 21 Packages 2024-02-09T17:12:02.189 INFO:teuthology.orchestra.run.smithi162.stdout:Upgrade 2 Packages 2024-02-09T17:12:02.189 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:12:02.190 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 64 M 2024-02-09T17:12:02.190 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-02-09T17:12:02.405 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-02-09T17:12:02.408 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-02-09T17:12:02.408 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-02-09T17:12:02.408 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-02-09T17:12:02.408 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-02-09T17:12:02.408 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-radosgw x86_64 2:17.2.7-499.g89a83b98.el8 ceph 17 M 2024-02-09T17:12:02.408 INFO:teuthology.orchestra.run.smithi179.stdout:Upgrading: 2024-02-09T17:12:02.408 INFO:teuthology.orchestra.run.smithi179.stdout: librados2 x86_64 2:17.2.7-499.g89a83b98.el8 ceph 3.5 M 2024-02-09T17:12:02.408 INFO:teuthology.orchestra.run.smithi179.stdout: librbd1 x86_64 2:17.2.7-499.g89a83b98.el8 ceph 3.6 M 2024-02-09T17:12:02.408 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-02-09T17:12:02.408 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-base x86_64 2:17.2.7-499.g89a83b98.el8 ceph 5.2 M 2024-02-09T17:12:02.409 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-common x86_64 2:17.2.7-499.g89a83b98.el8 ceph 23 M 2024-02-09T17:12:02.409 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-selinux x86_64 2:17.2.7-499.g89a83b98.el8 ceph 24 k 2024-02-09T17:12:02.409 INFO:teuthology.orchestra.run.smithi179.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-02-09T17:12:02.409 INFO:teuthology.orchestra.run.smithi179.stdout: libcephfs2 x86_64 2:17.2.7-499.g89a83b98.el8 ceph 798 k 2024-02-09T17:12:02.409 INFO:teuthology.orchestra.run.smithi179.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-02-09T17:12:02.409 INFO:teuthology.orchestra.run.smithi179.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-02-09T17:12:02.409 INFO:teuthology.orchestra.run.smithi179.stdout: libradosstriper1 x86_64 2:17.2.7-499.g89a83b98.el8 ceph 507 k 2024-02-09T17:12:02.409 INFO:teuthology.orchestra.run.smithi179.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-02-09T17:12:02.409 INFO:teuthology.orchestra.run.smithi179.stdout: librgw2 x86_64 2:17.2.7-499.g89a83b98.el8 ceph 5.6 M 2024-02-09T17:12:02.409 INFO:teuthology.orchestra.run.smithi179.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-02-09T17:12:02.409 INFO:teuthology.orchestra.run.smithi179.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-02-09T17:12:02.409 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ceph-argparse x86_64 2:17.2.7-499.g89a83b98.el8 ceph 45 k 2024-02-09T17:12:02.409 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ceph-common x86_64 2:17.2.7-499.g89a83b98.el8 ceph 115 k 2024-02-09T17:12:02.409 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cephfs x86_64 2:17.2.7-499.g89a83b98.el8 ceph 190 k 2024-02-09T17:12:02.410 INFO:teuthology.orchestra.run.smithi179.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-02-09T17:12:02.410 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rados x86_64 2:17.2.7-499.g89a83b98.el8 ceph 372 k 2024-02-09T17:12:02.410 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rbd x86_64 2:17.2.7-499.g89a83b98.el8 ceph 358 k 2024-02-09T17:12:02.410 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rgw x86_64 2:17.2.7-499.g89a83b98.el8 ceph 107 k 2024-02-09T17:12:02.410 INFO:teuthology.orchestra.run.smithi179.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-02-09T17:12:02.410 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:12:02.410 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-02-09T17:12:02.410 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-02-09T17:12:02.410 INFO:teuthology.orchestra.run.smithi179.stdout:Install 21 Packages 2024-02-09T17:12:02.410 INFO:teuthology.orchestra.run.smithi179.stdout:Upgrade 2 Packages 2024-02-09T17:12:02.410 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:12:02.411 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 64 M 2024-02-09T17:12:02.411 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-02-09T17:12:05.071 INFO:teuthology.orchestra.run.smithi162.stdout:(1/23): ceph-base-17.2.7-499.g89a83b98.el8.x86_ 3.7 MB/s | 5.2 MB 00:01 2024-02-09T17:12:05.179 INFO:teuthology.orchestra.run.smithi162.stdout:(2/23): ceph-selinux-17.2.7-499.g89a83b98.el8.x 225 kB/s | 24 kB 00:00 2024-02-09T17:12:05.284 INFO:teuthology.orchestra.run.smithi179.stdout:(1/23): ceph-base-17.2.7-499.g89a83b98.el8.x86_ 3.5 MB/s | 5.2 MB 00:01 2024-02-09T17:12:05.368 INFO:teuthology.orchestra.run.smithi179.stdout:(2/23): ceph-selinux-17.2.7-499.g89a83b98.el8.x 293 kB/s | 24 kB 00:00 2024-02-09T17:12:05.546 INFO:teuthology.orchestra.run.smithi162.stdout:(3/23): libcephfs2-17.2.7-499.g89a83b98.el8.x86 2.1 MB/s | 798 kB 00:00 2024-02-09T17:12:05.780 INFO:teuthology.orchestra.run.smithi162.stdout:(4/23): libradosstriper1-17.2.7-499.g89a83b98.e 2.1 MB/s | 507 kB 00:00 2024-02-09T17:12:05.785 INFO:teuthology.orchestra.run.smithi179.stdout:(3/23): libcephfs2-17.2.7-499.g89a83b98.el8.x86 1.9 MB/s | 798 kB 00:00 2024-02-09T17:12:05.964 INFO:teuthology.orchestra.run.smithi162.stdout:(5/23): ceph-radosgw-17.2.7-499.g89a83b98.el8.x 7.3 MB/s | 17 MB 00:02 2024-02-09T17:12:06.002 INFO:teuthology.orchestra.run.smithi179.stdout:(4/23): libradosstriper1-17.2.7-499.g89a83b98.e 2.3 MB/s | 507 kB 00:00 2024-02-09T17:12:06.047 INFO:teuthology.orchestra.run.smithi162.stdout:(6/23): python3-ceph-argparse-17.2.7-499.g89a83 539 kB/s | 45 kB 00:00 2024-02-09T17:12:06.147 INFO:teuthology.orchestra.run.smithi162.stdout:(7/23): python3-ceph-common-17.2.7-499.g89a83b9 1.1 MB/s | 115 kB 00:00 2024-02-09T17:12:06.256 INFO:teuthology.orchestra.run.smithi162.stdout:(8/23): python3-cephfs-17.2.7-499.g89a83b98.el8 1.7 MB/s | 190 kB 00:00 2024-02-09T17:12:06.383 INFO:teuthology.orchestra.run.smithi162.stdout:(9/23): python3-rados-17.2.7-499.g89a83b98.el8. 2.9 MB/s | 372 kB 00:00 2024-02-09T17:12:06.498 INFO:teuthology.orchestra.run.smithi162.stdout:(10/23): python3-rbd-17.2.7-499.g89a83b98.el8.x 3.0 MB/s | 358 kB 00:00 2024-02-09T17:12:06.657 INFO:teuthology.orchestra.run.smithi162.stdout:(11/23): python3-rgw-17.2.7-499.g89a83b98.el8.x 674 kB/s | 107 kB 00:00 2024-02-09T17:12:06.875 INFO:teuthology.orchestra.run.smithi099.stdout:(1/23): ceph-base-17.2.7-499.g89a83b98.el8.x86_ 2.5 MB/s | 5.2 MB 00:02 2024-02-09T17:12:06.959 INFO:teuthology.orchestra.run.smithi099.stdout:(2/23): ceph-selinux-17.2.7-499.g89a83b98.el8.x 293 kB/s | 24 kB 00:00 2024-02-09T17:12:07.199 INFO:teuthology.orchestra.run.smithi162.stdout:(12/23): librdkafka-1.6.1-1.el8.x86_64.rpm 1.2 MB/s | 677 kB 00:00 2024-02-09T17:12:07.276 INFO:teuthology.orchestra.run.smithi099.stdout:(3/23): libcephfs2-17.2.7-499.g89a83b98.el8.x86 2.5 MB/s | 798 kB 00:00 2024-02-09T17:12:07.349 INFO:teuthology.orchestra.run.smithi162.stdout:(13/23): librgw2-17.2.7-499.g89a83b98.el8.x86_6 3.6 MB/s | 5.6 MB 00:01 2024-02-09T17:12:07.374 INFO:teuthology.orchestra.run.smithi162.stdout:(14/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.4 MB/s | 259 kB 00:00 2024-02-09T17:12:07.492 INFO:teuthology.orchestra.run.smithi099.stdout:(4/23): libradosstriper1-17.2.7-499.g89a83b98.e 2.3 MB/s | 507 kB 00:00 2024-02-09T17:12:07.543 INFO:teuthology.orchestra.run.smithi179.stdout:(5/23): librgw2-17.2.7-499.g89a83b98.el8.x86_64 3.6 MB/s | 5.6 MB 00:01 2024-02-09T17:12:07.575 INFO:teuthology.orchestra.run.smithi162.stdout:(15/23): librabbitmq-0.9.0-5.el8.x86_64.rpm 235 kB/s | 47 kB 00:00 2024-02-09T17:12:07.627 INFO:teuthology.orchestra.run.smithi179.stdout:(6/23): python3-ceph-argparse-17.2.7-499.g89a83 537 kB/s | 45 kB 00:00 2024-02-09T17:12:07.692 INFO:teuthology.orchestra.run.smithi162.stdout:(16/23): python3-prettytable-0.7.2-14.el8.noarc 128 kB/s | 44 kB 00:00 2024-02-09T17:12:07.710 INFO:teuthology.orchestra.run.smithi179.stdout:(7/23): python3-ceph-common-17.2.7-499.g89a83b9 1.4 MB/s | 115 kB 00:00 2024-02-09T17:12:07.767 INFO:teuthology.orchestra.run.smithi162.stdout:(17/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.6 MB/s | 306 kB 00:00 2024-02-09T17:12:07.803 INFO:teuthology.orchestra.run.smithi179.stdout:(8/23): python3-cephfs-17.2.7-499.g89a83b98.el8 2.0 MB/s | 190 kB 00:00 2024-02-09T17:12:07.817 INFO:teuthology.orchestra.run.smithi162.stdout:(18/23): libunwind-1.3.1-3.el8.x86_64.rpm 1.5 MB/s | 75 kB 00:00 2024-02-09T17:12:07.859 INFO:teuthology.orchestra.run.smithi162.stdout:(19/23): liboath-2.6.2-3.el8.x86_64.rpm 352 kB/s | 59 kB 00:00 2024-02-09T17:12:07.935 INFO:teuthology.orchestra.run.smithi162.stdout:(20/23): thrift-0.13.0-2.el8.x86_64.rpm 15 MB/s | 1.7 MB 00:00 2024-02-09T17:12:07.987 INFO:teuthology.orchestra.run.smithi179.stdout:(9/23): ceph-radosgw-17.2.7-499.g89a83b98.el8.x 4.0 MB/s | 17 MB 00:04 2024-02-09T17:12:08.028 INFO:teuthology.orchestra.run.smithi179.stdout:(10/23): python3-rados-17.2.7-499.g89a83b98.el8 1.6 MB/s | 372 kB 00:00 2024-02-09T17:12:08.187 INFO:teuthology.orchestra.run.smithi179.stdout:(11/23): python3-rgw-17.2.7-499.g89a83b98.el8.x 673 kB/s | 107 kB 00:00 2024-02-09T17:12:08.212 INFO:teuthology.orchestra.run.smithi179.stdout:(12/23): python3-rbd-17.2.7-499.g89a83b98.el8.x 1.6 MB/s | 358 kB 00:00 2024-02-09T17:12:08.735 INFO:teuthology.orchestra.run.smithi162.stdout:(21/23): librados2-17.2.7-499.g89a83b98.el8.x86 4.0 MB/s | 3.5 MB 00:00 2024-02-09T17:12:08.746 INFO:teuthology.orchestra.run.smithi179.stdout:(13/23): ceph-common-17.2.7-499.g89a83b98.el8.x 4.8 MB/s | 23 MB 00:04 2024-02-09T17:12:08.771 INFO:teuthology.orchestra.run.smithi179.stdout:(14/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 465 kB/s | 259 kB 00:00 2024-02-09T17:12:08.793 INFO:teuthology.orchestra.run.smithi162.stdout:(22/23): librbd1-17.2.7-499.g89a83b98.el8.x86_6 4.2 MB/s | 3.6 MB 00:00 2024-02-09T17:12:08.896 INFO:teuthology.orchestra.run.smithi179.stdout:(15/23): librdkafka-1.6.1-1.el8.x86_64.rpm 954 kB/s | 677 kB 00:00 2024-02-09T17:12:08.927 INFO:teuthology.orchestra.run.smithi099.stdout:(5/23): librgw2-17.2.7-499.g89a83b98.el8.x86_64 3.9 MB/s | 5.6 MB 00:01 2024-02-09T17:12:08.947 INFO:teuthology.orchestra.run.smithi179.stdout:(16/23): librabbitmq-0.9.0-5.el8.x86_64.rpm 268 kB/s | 47 kB 00:00 2024-02-09T17:12:08.985 INFO:teuthology.orchestra.run.smithi099.stdout:(6/23): python3-ceph-argparse-17.2.7-499.g89a83 770 kB/s | 45 kB 00:00 2024-02-09T17:12:09.036 INFO:teuthology.orchestra.run.smithi099.stdout:(7/23): python3-ceph-common-17.2.7-499.g89a83b9 2.2 MB/s | 115 kB 00:00 2024-02-09T17:12:09.064 INFO:teuthology.orchestra.run.smithi179.stdout:(17/23): python3-prettytable-0.7.2-14.el8.noarc 138 kB/s | 44 kB 00:00 2024-02-09T17:12:09.077 INFO:teuthology.orchestra.run.smithi099.stdout:(8/23): python3-cephfs-17.2.7-499.g89a83b98.el8 4.5 MB/s | 190 kB 00:00 2024-02-09T17:12:09.178 INFO:teuthology.orchestra.run.smithi099.stdout:(9/23): python3-rados-17.2.7-499.g89a83b98.el8. 3.6 MB/s | 372 kB 00:00 2024-02-09T17:12:09.278 INFO:teuthology.orchestra.run.smithi099.stdout:(10/23): python3-rbd-17.2.7-499.g89a83b98.el8.x 3.5 MB/s | 358 kB 00:00 2024-02-09T17:12:09.337 INFO:teuthology.orchestra.run.smithi099.stdout:(11/23): python3-rgw-17.2.7-499.g89a83b98.el8.x 1.8 MB/s | 107 kB 00:00 2024-02-09T17:12:09.464 INFO:teuthology.orchestra.run.smithi179.stdout:(18/23): liboath-2.6.2-3.el8.x86_64.rpm 114 kB/s | 59 kB 00:00 2024-02-09T17:12:09.539 INFO:teuthology.orchestra.run.smithi179.stdout:(19/23): gperftools-libs-2.7-9.el8.x86_64.rpm 476 kB/s | 306 kB 00:00 2024-02-09T17:12:09.565 INFO:teuthology.orchestra.run.smithi179.stdout:(20/23): libunwind-1.3.1-3.el8.x86_64.rpm 150 kB/s | 75 kB 00:00 2024-02-09T17:12:09.771 INFO:teuthology.orchestra.run.smithi099.stdout:(12/23): ceph-common-17.2.7-499.g89a83b98.el8.x 4.7 MB/s | 23 MB 00:04 2024-02-09T17:12:09.794 INFO:teuthology.orchestra.run.smithi162.stdout:(23/23): ceph-common-17.2.7-499.g89a83b98.el8.x 3.8 MB/s | 23 MB 00:06 2024-02-09T17:12:09.798 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-02-09T17:12:09.798 INFO:teuthology.orchestra.run.smithi162.stdout:Total 8.4 MB/s | 64 MB 00:07 2024-02-09T17:12:09.840 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-02-09T17:12:09.887 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-02-09T17:12:09.887 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-02-09T17:12:09.954 INFO:teuthology.orchestra.run.smithi099.stdout:(13/23): librdkafka-1.6.1-1.el8.x86_64.rpm 1.1 MB/s | 677 kB 00:00 2024-02-09T17:12:09.982 INFO:teuthology.orchestra.run.smithi179.stdout:(21/23): librados2-17.2.7-499.g89a83b98.el8.x86 7.8 MB/s | 3.5 MB 00:00 2024-02-09T17:12:10.015 INFO:teuthology.orchestra.run.smithi179.stdout:(22/23): thrift-0.13.0-2.el8.x86_64.rpm 3.1 MB/s | 1.7 MB 00:00 2024-02-09T17:12:10.054 INFO:teuthology.orchestra.run.smithi099.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 440 kB/s | 44 kB 00:00 2024-02-09T17:12:10.074 INFO:teuthology.orchestra.run.smithi179.stdout:(23/23): librbd1-17.2.7-499.g89a83b98.el8.x86_6 7.1 MB/s | 3.6 MB 00:00 2024-02-09T17:12:10.078 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-02-09T17:12:10.078 INFO:teuthology.orchestra.run.smithi179.stdout:Total 8.3 MB/s | 64 MB 00:07 2024-02-09T17:12:10.120 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-02-09T17:12:10.165 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-02-09T17:12:10.165 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-02-09T17:12:10.246 INFO:teuthology.orchestra.run.smithi099.stdout:(15/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 546 kB/s | 259 kB 00:00 2024-02-09T17:12:10.251 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-02-09T17:12:10.254 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-02-09T17:12:10.271 INFO:teuthology.orchestra.run.smithi099.stdout:(16/23): librabbitmq-0.9.0-5.el8.x86_64.rpm 217 kB/s | 47 kB 00:00 2024-02-09T17:12:10.372 INFO:teuthology.orchestra.run.smithi099.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 589 kB/s | 59 kB 00:00 2024-02-09T17:12:10.564 INFO:teuthology.orchestra.run.smithi099.stdout:(18/23): ceph-radosgw-17.2.7-499.g89a83b98.el8. 2.9 MB/s | 17 MB 00:05 2024-02-09T17:12:10.589 INFO:teuthology.orchestra.run.smithi099.stdout:(19/23): libunwind-1.3.1-3.el8.x86_64.rpm 346 kB/s | 75 kB 00:00 2024-02-09T17:12:10.613 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-02-09T17:12:10.614 INFO:teuthology.orchestra.run.smithi099.stdout:(20/23): gperftools-libs-2.7-9.el8.x86_64.rpm 833 kB/s | 306 kB 00:00 2024-02-09T17:12:10.640 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-02-09T17:12:10.789 INFO:teuthology.orchestra.run.smithi099.stdout:(21/23): thrift-0.13.0-2.el8.x86_64.rpm 7.7 MB/s | 1.7 MB 00:00 2024-02-09T17:12:11.223 INFO:teuthology.orchestra.run.smithi099.stdout:(22/23): librbd1-17.2.7-499.g89a83b98.el8.x86_6 5.9 MB/s | 3.6 MB 00:00 2024-02-09T17:12:11.315 INFO:teuthology.orchestra.run.smithi099.stdout:(23/23): librados2-17.2.7-499.g89a83b98.el8.x86 4.8 MB/s | 3.5 MB 00:00 2024-02-09T17:12:11.317 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-02-09T17:12:11.317 INFO:teuthology.orchestra.run.smithi099.stdout:Total 6.9 MB/s | 64 MB 00:09 2024-02-09T17:12:11.357 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-02-09T17:12:11.402 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-02-09T17:12:11.402 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-02-09T17:12:11.697 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-02-09T17:12:11.763 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-02-09T17:12:11.766 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-02-09T17:12:11.893 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-02-09T17:12:11.907 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2024-02-09T17:12:12.085 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-02-09T17:12:12.152 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2024-02-09T17:12:12.344 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-02-09T17:12:12.451 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-02-09T17:12:12.627 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : librados2-2:17.2.7-499.g89a83b98.el8.x86_64 3/25 2024-02-09T17:12:12.723 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-02-09T17:12:12.727 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librados2-2:17.2.7-499.g89a83b98.el8.x86_64 3/25 2024-02-09T17:12:12.854 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-02-09T17:12:12.886 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : librados2-2:17.2.7-499.g89a83b98.el8.x86_64 3/25 2024-02-09T17:12:12.983 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librados2-2:17.2.7-499.g89a83b98.el8.x86_64 3/25 2024-02-09T17:12:12.984 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rados-2:17.2.7-499.g89a83b98.el8.x86_64 4/25 2024-02-09T17:12:13.028 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2024-02-09T17:12:13.185 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephfs2-2:17.2.7-499.g89a83b98.el8.x86_64 5/25 2024-02-09T17:12:13.220 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-rados-2:17.2.7-499.g89a83b98.el8.x86_64 4/25 2024-02-09T17:12:13.239 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-02-09T17:12:13.402 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libcephfs2-2:17.2.7-499.g89a83b98.el8.x86_64 5/25 2024-02-09T17:12:13.579 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libcephfs2-2:17.2.7-499.g89a83b98.el8.x86_64 5/25 2024-02-09T17:12:13.620 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-02-09T17:12:13.743 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : librbd1-2:17.2.7-499.g89a83b98.el8.x86_64 6/25 2024-02-09T17:12:13.800 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: libcephfs2-2:17.2.7-499.g89a83b98.el8.x86_64 5/25 2024-02-09T17:12:13.814 INFO:teuthology.orchestra.run.smithi099.stdout: Upgrading : librados2-2:17.2.7-499.g89a83b98.el8.x86_64 3/25 2024-02-09T17:12:13.897 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librados2-2:17.2.7-499.g89a83b98.el8.x86_64 3/25 2024-02-09T17:12:13.956 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librbd1-2:17.2.7-499.g89a83b98.el8.x86_64 6/25 2024-02-09T17:12:13.977 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : librbd1-2:17.2.7-499.g89a83b98.el8.x86_64 6/25 2024-02-09T17:12:14.145 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-rados-2:17.2.7-499.g89a83b98.el8.x86_64 4/25 2024-02-09T17:12:14.156 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2024-02-09T17:12:14.197 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librbd1-2:17.2.7-499.g89a83b98.el8.x86_64 6/25 2024-02-09T17:12:14.306 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libcephfs2-2:17.2.7-499.g89a83b98.el8.x86_64 5/25 2024-02-09T17:12:14.344 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 8/25 2024-02-09T17:12:14.415 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2024-02-09T17:12:14.636 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 8/25 2024-02-09T17:12:14.726 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: libcephfs2-2:17.2.7-499.g89a83b98.el8.x86_64 5/25 2024-02-09T17:12:14.905 INFO:teuthology.orchestra.run.smithi099.stdout: Upgrading : librbd1-2:17.2.7-499.g89a83b98.el8.x86_64 6/25 2024-02-09T17:12:15.031 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 9/25 2024-02-09T17:12:15.117 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librbd1-2:17.2.7-499.g89a83b98.el8.x86_64 6/25 2024-02-09T17:12:15.176 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librgw2-2:17.2.7-499.g89a83b98.el8.x86_64 10/25 2024-02-09T17:12:15.245 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librgw2-2:17.2.7-499.g89a83b98.el8.x86_64 10/25 2024-02-09T17:12:15.309 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2024-02-09T17:12:15.315 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 9/25 2024-02-09T17:12:15.411 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-ceph-argparse-2:17.2.7-499.g89a83b98.el8.x 11/25 2024-02-09T17:12:15.469 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : librgw2-2:17.2.7-499.g89a83b98.el8.x86_64 10/25 2024-02-09T17:12:15.497 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 8/25 2024-02-09T17:12:15.559 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librgw2-2:17.2.7-499.g89a83b98.el8.x86_64 10/25 2024-02-09T17:12:15.591 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cephfs-2:17.2.7-499.g89a83b98.el8.x86_64 12/25 2024-02-09T17:12:15.704 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-ceph-argparse-2:17.2.7-499.g89a83b98.el8.x 11/25 2024-02-09T17:12:15.811 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rgw-2:17.2.7-499.g89a83b98.el8.x86_64 13/25 2024-02-09T17:12:15.889 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cephfs-2:17.2.7-499.g89a83b98.el8.x86_64 12/25 2024-02-09T17:12:16.079 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-rgw-2:17.2.7-499.g89a83b98.el8.x86_64 13/25 2024-02-09T17:12:16.079 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rbd-2:17.2.7-499.g89a83b98.el8.x86_64 14/25 2024-02-09T17:12:16.193 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 9/25 2024-02-09T17:12:16.234 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libradosstriper1-2:17.2.7-499.g89a83b98.el8.x86_64 15/25 2024-02-09T17:12:16.285 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libradosstriper1-2:17.2.7-499.g89a83b98.el8.x86_64 15/25 2024-02-09T17:12:16.355 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-rbd-2:17.2.7-499.g89a83b98.el8.x86_64 14/25 2024-02-09T17:12:16.363 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : librgw2-2:17.2.7-499.g89a83b98.el8.x86_64 10/25 2024-02-09T17:12:16.432 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librgw2-2:17.2.7-499.g89a83b98.el8.x86_64 10/25 2024-02-09T17:12:16.494 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libradosstriper1-2:17.2.7-499.g89a83b98.el8.x86_64 15/25 2024-02-09T17:12:16.501 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2024-02-09T17:12:16.565 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: libradosstriper1-2:17.2.7-499.g89a83b98.el8.x86_64 15/25 2024-02-09T17:12:16.582 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-ceph-argparse-2:17.2.7-499.g89a83b98.el8.x 11/25 2024-02-09T17:12:16.692 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2024-02-09T17:12:16.770 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cephfs-2:17.2.7-499.g89a83b98.el8.x86_64 12/25 2024-02-09T17:12:16.802 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2024-02-09T17:12:16.884 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-02-09T17:12:16.985 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2024-02-09T17:12:16.989 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-rgw-2:17.2.7-499.g89a83b98.el8.x86_64 13/25 2024-02-09T17:12:17.109 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-ceph-common-2:17.2.7-499.g89a83b98.el8.x86 19/25 2024-02-09T17:12:17.168 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-02-09T17:12:17.207 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-rbd-2:17.2.7-499.g89a83b98.el8.x86_64 14/25 2024-02-09T17:12:17.346 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libradosstriper1-2:17.2.7-499.g89a83b98.el8.x86_64 15/25 2024-02-09T17:12:17.369 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-ceph-common-2:17.2.7-499.g89a83b98.el8.x86 19/25 2024-02-09T17:12:17.404 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: libradosstriper1-2:17.2.7-499.g89a83b98.el8.x86_64 15/25 2024-02-09T17:12:17.638 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2024-02-09T17:12:17.804 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2024-02-09T17:12:17.995 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-02-09T17:12:18.188 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-ceph-common-2:17.2.7-499.g89a83b98.el8.x86 19/25 2024-02-09T17:12:19.780 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-common-2:17.2.7-499.g89a83b98.el8.x86_64 20/25 2024-02-09T17:12:20.026 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-common-2:17.2.7-499.g89a83b98.el8.x86_64 20/25 2024-02-09T17:12:20.045 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-common-2:17.2.7-499.g89a83b98.el8.x86_64 20/25 2024-02-09T17:12:20.294 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-common-2:17.2.7-499.g89a83b98.el8.x86_64 20/25 2024-02-09T17:12:20.630 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-common-2:17.2.7-499.g89a83b98.el8.x86_64 20/25 2024-02-09T17:12:20.875 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-base-2:17.2.7-499.g89a83b98.el8.x86_64 21/25 2024-02-09T17:12:20.884 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-common-2:17.2.7-499.g89a83b98.el8.x86_64 20/25 2024-02-09T17:12:20.979 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-base-2:17.2.7-499.g89a83b98.el8.x86_64 21/25 2024-02-09T17:12:21.112 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-common-2:17.2.7-499.g89a83b98.el8.x86_64 20/25 2024-02-09T17:12:21.154 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-base-2:17.2.7-499.g89a83b98.el8.x86_64 21/25 2024-02-09T17:12:21.329 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-common-2:17.2.7-499.g89a83b98.el8.x86_64 20/25 2024-02-09T17:12:21.450 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-selinux-2:17.2.7-499.g89a83b98.el8.x86_64 22/25 2024-02-09T17:12:21.520 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-base-2:17.2.7-499.g89a83b98.el8.x86_64 21/25 2024-02-09T17:12:21.913 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-common-2:17.2.7-499.g89a83b98.el8.x86_64 20/25 2024-02-09T17:12:22.010 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-selinux-2:17.2.7-499.g89a83b98.el8.x86_64 22/25 2024-02-09T17:12:22.205 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-base-2:17.2.7-499.g89a83b98.el8.x86_64 21/25 2024-02-09T17:12:22.307 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-base-2:17.2.7-499.g89a83b98.el8.x86_64 21/25 2024-02-09T17:12:22.462 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-selinux-2:17.2.7-499.g89a83b98.el8.x86_64 22/25 2024-02-09T17:12:40.561 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-selinux-2:17.2.7-499.g89a83b98.el8.x86_64 22/25 2024-02-09T17:12:40.561 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /sys 2024-02-09T17:12:40.561 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /proc 2024-02-09T17:12:40.561 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /dev 2024-02-09T17:12:40.561 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /run 2024-02-09T17:12:40.561 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /mnt 2024-02-09T17:12:40.562 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /var/tmp 2024-02-09T17:12:40.562 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /home 2024-02-09T17:12:40.562 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /tmp 2024-02-09T17:12:40.562 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /dev 2024-02-09T17:12:40.562 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:12:41.410 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-selinux-2:17.2.7-499.g89a83b98.el8.x86_64 22/25 2024-02-09T17:12:41.410 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /sys 2024-02-09T17:12:41.410 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /proc 2024-02-09T17:12:41.410 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /dev 2024-02-09T17:12:41.410 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /run 2024-02-09T17:12:41.410 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /mnt 2024-02-09T17:12:41.410 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /var/tmp 2024-02-09T17:12:41.410 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /home 2024-02-09T17:12:41.410 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /tmp 2024-02-09T17:12:41.410 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /dev 2024-02-09T17:12:41.410 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:12:41.747 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-selinux-2:17.2.7-499.g89a83b98.el8.x86_64 22/25 2024-02-09T17:12:41.748 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /sys 2024-02-09T17:12:41.748 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /proc 2024-02-09T17:12:41.748 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /dev 2024-02-09T17:12:41.748 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /run 2024-02-09T17:12:41.748 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /mnt 2024-02-09T17:12:41.748 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /var/tmp 2024-02-09T17:12:41.748 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /home 2024-02-09T17:12:41.748 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /tmp 2024-02-09T17:12:41.748 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /dev 2024-02-09T17:12:41.748 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:12:42.295 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-radosgw-2:17.2.7-499.g89a83b98.el8.x86_64 23/25 2024-02-09T17:12:42.390 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-499.g89a83b98.el8.x86_64 23/25 2024-02-09T17:12:42.390 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-02-09T17:12:42.605 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-02-09T17:12:42.606 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-02-09T17:12:43.151 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-radosgw-2:17.2.7-499.g89a83b98.el8.x86_64 23/25 2024-02-09T17:12:43.213 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-499.g89a83b98.el8.x86_64 23/25 2024-02-09T17:12:43.213 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-02-09T17:12:43.275 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-02-09T17:12:43.276 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-base-2:17.2.7-499.g89a83b98.el8.x86_64 1/25 2024-02-09T17:12:43.276 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-common-2:17.2.7-499.g89a83b98.el8.x86_64 2/25 2024-02-09T17:12:43.276 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-radosgw-2:17.2.7-499.g89a83b98.el8.x86_64 3/25 2024-02-09T17:12:43.276 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-selinux-2:17.2.7-499.g89a83b98.el8.x86_64 4/25 2024-02-09T17:12:43.276 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephfs2-2:17.2.7-499.g89a83b98.el8.x86_64 5/25 2024-02-09T17:12:43.276 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libradosstriper1-2:17.2.7-499.g89a83b98.el8.x86_64 6/25 2024-02-09T17:12:43.276 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librgw2-2:17.2.7-499.g89a83b98.el8.x86_64 7/25 2024-02-09T17:12:43.276 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-ceph-argparse-2:17.2.7-499.g89a83b98.el8.x 8/25 2024-02-09T17:12:43.276 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-ceph-common-2:17.2.7-499.g89a83b98.el8.x86 9/25 2024-02-09T17:12:43.276 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cephfs-2:17.2.7-499.g89a83b98.el8.x86_64 10/25 2024-02-09T17:12:43.276 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rados-2:17.2.7-499.g89a83b98.el8.x86_64 11/25 2024-02-09T17:12:43.276 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rbd-2:17.2.7-499.g89a83b98.el8.x86_64 12/25 2024-02-09T17:12:43.276 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rgw-2:17.2.7-499.g89a83b98.el8.x86_64 13/25 2024-02-09T17:12:43.277 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-02-09T17:12:43.277 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-02-09T17:12:43.277 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-02-09T17:12:43.277 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 17/25 2024-02-09T17:12:43.277 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-02-09T17:12:43.277 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2024-02-09T17:12:43.277 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2024-02-09T17:12:43.277 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2024-02-09T17:12:43.277 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados2-2:17.2.7-499.g89a83b98.el8.x86_64 22/25 2024-02-09T17:12:43.277 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-02-09T17:12:43.277 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librbd1-2:17.2.7-499.g89a83b98.el8.x86_64 24/25 2024-02-09T17:12:43.442 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-02-09T17:12:43.442 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-02-09T17:12:43.484 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-radosgw-2:17.2.7-499.g89a83b98.el8.x86_64 23/25 2024-02-09T17:12:43.558 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-499.g89a83b98.el8.x86_64 23/25 2024-02-09T17:12:43.558 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-02-09T17:12:43.759 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-02-09T17:12:43.759 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-02-09T17:12:44.142 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-02-09T17:12:44.142 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-base-2:17.2.7-499.g89a83b98.el8.x86_64 1/25 2024-02-09T17:12:44.142 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-common-2:17.2.7-499.g89a83b98.el8.x86_64 2/25 2024-02-09T17:12:44.142 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-radosgw-2:17.2.7-499.g89a83b98.el8.x86_64 3/25 2024-02-09T17:12:44.142 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-selinux-2:17.2.7-499.g89a83b98.el8.x86_64 4/25 2024-02-09T17:12:44.142 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libcephfs2-2:17.2.7-499.g89a83b98.el8.x86_64 5/25 2024-02-09T17:12:44.143 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libradosstriper1-2:17.2.7-499.g89a83b98.el8.x86_64 6/25 2024-02-09T17:12:44.143 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librgw2-2:17.2.7-499.g89a83b98.el8.x86_64 7/25 2024-02-09T17:12:44.143 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-ceph-argparse-2:17.2.7-499.g89a83b98.el8.x 8/25 2024-02-09T17:12:44.143 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-ceph-common-2:17.2.7-499.g89a83b98.el8.x86 9/25 2024-02-09T17:12:44.143 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cephfs-2:17.2.7-499.g89a83b98.el8.x86_64 10/25 2024-02-09T17:12:44.143 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-rados-2:17.2.7-499.g89a83b98.el8.x86_64 11/25 2024-02-09T17:12:44.143 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-rbd-2:17.2.7-499.g89a83b98.el8.x86_64 12/25 2024-02-09T17:12:44.143 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-rgw-2:17.2.7-499.g89a83b98.el8.x86_64 13/25 2024-02-09T17:12:44.143 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-02-09T17:12:44.143 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-02-09T17:12:44.143 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-02-09T17:12:44.143 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 17/25 2024-02-09T17:12:44.143 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-02-09T17:12:44.143 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2024-02-09T17:12:44.143 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2024-02-09T17:12:44.144 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2024-02-09T17:12:44.144 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librados2-2:17.2.7-499.g89a83b98.el8.x86_64 22/25 2024-02-09T17:12:44.144 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-02-09T17:12:44.144 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librbd1-2:17.2.7-499.g89a83b98.el8.x86_64 24/25 2024-02-09T17:12:44.167 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-02-09T17:12:44.168 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:12:44.168 INFO:teuthology.orchestra.run.smithi162.stdout:Upgraded: 2024-02-09T17:12:44.168 INFO:teuthology.orchestra.run.smithi162.stdout: librados2-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:12:44.168 INFO:teuthology.orchestra.run.smithi162.stdout: librbd1-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:12:44.168 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-02-09T17:12:44.168 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-base-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:12:44.168 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-common-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:12:44.168 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-radosgw-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:12:44.168 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-selinux-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:12:44.168 INFO:teuthology.orchestra.run.smithi162.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-02-09T17:12:44.168 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs2-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:12:44.168 INFO:teuthology.orchestra.run.smithi162.stdout: liboath-2.6.2-3.el8.x86_64 2024-02-09T17:12:44.169 INFO:teuthology.orchestra.run.smithi162.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-02-09T17:12:44.169 INFO:teuthology.orchestra.run.smithi162.stdout: libradosstriper1-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:12:44.169 INFO:teuthology.orchestra.run.smithi162.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-02-09T17:12:44.169 INFO:teuthology.orchestra.run.smithi162.stdout: librgw2-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:12:44.169 INFO:teuthology.orchestra.run.smithi162.stdout: libunwind-1.3.1-3.el8.x86_64 2024-02-09T17:12:44.169 INFO:teuthology.orchestra.run.smithi162.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-02-09T17:12:44.169 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-argparse-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:12:44.169 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-common-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:12:44.169 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cephfs-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:12:44.169 INFO:teuthology.orchestra.run.smithi162.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-02-09T17:12:44.169 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rados-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:12:44.169 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rbd-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:12:44.169 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rgw-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:12:44.169 INFO:teuthology.orchestra.run.smithi162.stdout: thrift-0.13.0-2.el8.x86_64 2024-02-09T17:12:44.169 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:12:44.170 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-09T17:12:44.472 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-02-09T17:12:44.473 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-base-2:17.2.7-499.g89a83b98.el8.x86_64 1/25 2024-02-09T17:12:44.473 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-common-2:17.2.7-499.g89a83b98.el8.x86_64 2/25 2024-02-09T17:12:44.473 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-radosgw-2:17.2.7-499.g89a83b98.el8.x86_64 3/25 2024-02-09T17:12:44.473 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-selinux-2:17.2.7-499.g89a83b98.el8.x86_64 4/25 2024-02-09T17:12:44.473 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libcephfs2-2:17.2.7-499.g89a83b98.el8.x86_64 5/25 2024-02-09T17:12:44.473 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libradosstriper1-2:17.2.7-499.g89a83b98.el8.x86_64 6/25 2024-02-09T17:12:44.473 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librgw2-2:17.2.7-499.g89a83b98.el8.x86_64 7/25 2024-02-09T17:12:44.473 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-ceph-argparse-2:17.2.7-499.g89a83b98.el8.x 8/25 2024-02-09T17:12:44.473 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-ceph-common-2:17.2.7-499.g89a83b98.el8.x86 9/25 2024-02-09T17:12:44.473 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cephfs-2:17.2.7-499.g89a83b98.el8.x86_64 10/25 2024-02-09T17:12:44.473 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-rados-2:17.2.7-499.g89a83b98.el8.x86_64 11/25 2024-02-09T17:12:44.473 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-rbd-2:17.2.7-499.g89a83b98.el8.x86_64 12/25 2024-02-09T17:12:44.473 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-rgw-2:17.2.7-499.g89a83b98.el8.x86_64 13/25 2024-02-09T17:12:44.473 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-02-09T17:12:44.474 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-02-09T17:12:44.474 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-02-09T17:12:44.474 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 17/25 2024-02-09T17:12:44.474 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-02-09T17:12:44.474 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2024-02-09T17:12:44.474 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2024-02-09T17:12:44.474 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2024-02-09T17:12:44.474 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librados2-2:17.2.7-499.g89a83b98.el8.x86_64 22/25 2024-02-09T17:12:44.474 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-02-09T17:12:44.474 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librbd1-2:17.2.7-499.g89a83b98.el8.x86_64 24/25 2024-02-09T17:12:44.475 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-test 2024-02-09T17:12:44.993 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-02-09T17:12:44.993 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:12:44.994 INFO:teuthology.orchestra.run.smithi179.stdout:Upgraded: 2024-02-09T17:12:44.994 INFO:teuthology.orchestra.run.smithi179.stdout: librados2-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:12:44.994 INFO:teuthology.orchestra.run.smithi179.stdout: librbd1-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:12:44.994 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-02-09T17:12:44.994 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-base-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:12:44.994 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-common-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:12:44.994 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-radosgw-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:12:44.994 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-selinux-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:12:44.994 INFO:teuthology.orchestra.run.smithi179.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-02-09T17:12:44.994 INFO:teuthology.orchestra.run.smithi179.stdout: libcephfs2-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:12:44.994 INFO:teuthology.orchestra.run.smithi179.stdout: liboath-2.6.2-3.el8.x86_64 2024-02-09T17:12:44.994 INFO:teuthology.orchestra.run.smithi179.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-02-09T17:12:44.994 INFO:teuthology.orchestra.run.smithi179.stdout: libradosstriper1-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:12:44.995 INFO:teuthology.orchestra.run.smithi179.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-02-09T17:12:44.995 INFO:teuthology.orchestra.run.smithi179.stdout: librgw2-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:12:44.995 INFO:teuthology.orchestra.run.smithi179.stdout: libunwind-1.3.1-3.el8.x86_64 2024-02-09T17:12:44.995 INFO:teuthology.orchestra.run.smithi179.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-02-09T17:12:44.995 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ceph-argparse-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:12:44.995 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ceph-common-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:12:44.995 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cephfs-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:12:44.995 INFO:teuthology.orchestra.run.smithi179.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-02-09T17:12:44.995 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rados-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:12:44.995 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rbd-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:12:44.995 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rgw-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:12:44.995 INFO:teuthology.orchestra.run.smithi179.stdout: thrift-0.13.0-2.el8.x86_64 2024-02-09T17:12:44.995 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:12:44.995 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-02-09T17:12:45.234 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:47 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:12:45.283 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-test 2024-02-09T17:12:45.324 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-02-09T17:12:45.325 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:12:45.325 INFO:teuthology.orchestra.run.smithi099.stdout:Upgraded: 2024-02-09T17:12:45.325 INFO:teuthology.orchestra.run.smithi099.stdout: librados2-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:12:45.325 INFO:teuthology.orchestra.run.smithi099.stdout: librbd1-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:12:45.325 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-02-09T17:12:45.325 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-base-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:12:45.325 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-common-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:12:45.325 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-radosgw-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:12:45.325 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-selinux-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:12:45.325 INFO:teuthology.orchestra.run.smithi099.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-02-09T17:12:45.325 INFO:teuthology.orchestra.run.smithi099.stdout: libcephfs2-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:12:45.325 INFO:teuthology.orchestra.run.smithi099.stdout: liboath-2.6.2-3.el8.x86_64 2024-02-09T17:12:45.325 INFO:teuthology.orchestra.run.smithi099.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-02-09T17:12:45.325 INFO:teuthology.orchestra.run.smithi099.stdout: libradosstriper1-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:12:45.326 INFO:teuthology.orchestra.run.smithi099.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-02-09T17:12:45.326 INFO:teuthology.orchestra.run.smithi099.stdout: librgw2-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:12:45.326 INFO:teuthology.orchestra.run.smithi099.stdout: libunwind-1.3.1-3.el8.x86_64 2024-02-09T17:12:45.326 INFO:teuthology.orchestra.run.smithi099.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-02-09T17:12:45.326 INFO:teuthology.orchestra.run.smithi099.stdout: python3-ceph-argparse-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:12:45.326 INFO:teuthology.orchestra.run.smithi099.stdout: python3-ceph-common-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:12:45.326 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cephfs-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:12:45.326 INFO:teuthology.orchestra.run.smithi099.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-02-09T17:12:45.326 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rados-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:12:45.326 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rbd-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:12:45.326 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rgw-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:12:45.326 INFO:teuthology.orchestra.run.smithi099.stdout: thrift-0.13.0-2.el8.x86_64 2024-02-09T17:12:45.326 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:12:45.326 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-02-09T17:12:45.645 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-test 2024-02-09T17:12:46.039 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:00:48 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:12:46.401 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:00:48 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:12:46.772 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-09T17:12:46.773 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-09T17:12:46.773 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-02-09T17:12:46.773 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-09T17:12:46.773 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-02-09T17:12:46.773 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-test x86_64 2:17.2.7-499.g89a83b98.el8 ceph 52 M 2024-02-09T17:12:46.773 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-02-09T17:12:46.773 INFO:teuthology.orchestra.run.smithi162.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-02-09T17:12:46.773 INFO:teuthology.orchestra.run.smithi162.stdout: libcephsqlite x86_64 2:17.2.7-499.g89a83b98.el8 ceph 178 k 2024-02-09T17:12:46.773 INFO:teuthology.orchestra.run.smithi162.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-02-09T17:12:46.773 INFO:teuthology.orchestra.run.smithi162.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-02-09T17:12:46.773 INFO:teuthology.orchestra.run.smithi162.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-02-09T17:12:46.774 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:12:46.774 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-02-09T17:12:46.774 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-09T17:12:46.774 INFO:teuthology.orchestra.run.smithi162.stdout:Install 6 Packages 2024-02-09T17:12:46.774 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:12:46.774 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 53 M 2024-02-09T17:12:46.775 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 234 M 2024-02-09T17:12:46.775 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-02-09T17:12:47.047 INFO:teuthology.orchestra.run.smithi162.stdout:(1/6): libcephsqlite-17.2.7-499.g89a83b98.el8.x 771 kB/s | 178 kB 00:00 2024-02-09T17:12:47.214 INFO:teuthology.orchestra.run.smithi162.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 510 kB/s | 203 kB 00:00 2024-02-09T17:12:47.389 INFO:teuthology.orchestra.run.smithi162.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 1.8 MB/s | 323 kB 00:00 2024-02-09T17:12:47.440 INFO:teuthology.orchestra.run.smithi162.stdout:(4/6): oniguruma-6.8.2-3.el8.x86_64.rpm 480 kB/s | 188 kB 00:00 2024-02-09T17:12:47.490 INFO:teuthology.orchestra.run.smithi162.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 704 kB/s | 71 kB 00:00 2024-02-09T17:12:47.549 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-02-09T17:12:47.550 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-02-09T17:12:47.550 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-02-09T17:12:47.551 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-02-09T17:12:47.551 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-02-09T17:12:47.551 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-test x86_64 2:17.2.7-499.g89a83b98.el8 ceph 52 M 2024-02-09T17:12:47.551 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-02-09T17:12:47.551 INFO:teuthology.orchestra.run.smithi179.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-02-09T17:12:47.551 INFO:teuthology.orchestra.run.smithi179.stdout: libcephsqlite x86_64 2:17.2.7-499.g89a83b98.el8 ceph 178 k 2024-02-09T17:12:47.551 INFO:teuthology.orchestra.run.smithi179.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-02-09T17:12:47.551 INFO:teuthology.orchestra.run.smithi179.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-02-09T17:12:47.551 INFO:teuthology.orchestra.run.smithi179.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-02-09T17:12:47.551 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:12:47.551 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-02-09T17:12:47.551 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-02-09T17:12:47.551 INFO:teuthology.orchestra.run.smithi179.stdout:Install 6 Packages 2024-02-09T17:12:47.552 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:12:47.552 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 53 M 2024-02-09T17:12:47.552 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 234 M 2024-02-09T17:12:47.552 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-02-09T17:12:47.829 INFO:teuthology.orchestra.run.smithi179.stdout:(1/6): libcephsqlite-17.2.7-499.g89a83b98.el8.x 761 kB/s | 178 kB 00:00 2024-02-09T17:12:47.903 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-02-09T17:12:47.905 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-02-09T17:12:47.905 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2024-02-09T17:12:47.905 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-02-09T17:12:47.905 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-02-09T17:12:47.905 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-test x86_64 2:17.2.7-499.g89a83b98.el8 ceph 52 M 2024-02-09T17:12:47.905 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2024-02-09T17:12:47.905 INFO:teuthology.orchestra.run.smithi099.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-02-09T17:12:47.905 INFO:teuthology.orchestra.run.smithi099.stdout: libcephsqlite x86_64 2:17.2.7-499.g89a83b98.el8 ceph 178 k 2024-02-09T17:12:47.905 INFO:teuthology.orchestra.run.smithi099.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-02-09T17:12:47.905 INFO:teuthology.orchestra.run.smithi099.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-02-09T17:12:47.905 INFO:teuthology.orchestra.run.smithi099.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-02-09T17:12:47.905 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:12:47.905 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-02-09T17:12:47.905 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-02-09T17:12:47.906 INFO:teuthology.orchestra.run.smithi099.stdout:Install 6 Packages 2024-02-09T17:12:47.906 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:12:47.906 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 53 M 2024-02-09T17:12:47.906 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 234 M 2024-02-09T17:12:47.906 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-02-09T17:12:47.996 INFO:teuthology.orchestra.run.smithi179.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 506 kB/s | 203 kB 00:00 2024-02-09T17:12:48.171 INFO:teuthology.orchestra.run.smithi179.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 1.8 MB/s | 323 kB 00:00 2024-02-09T17:12:48.221 INFO:teuthology.orchestra.run.smithi179.stdout:(4/6): oniguruma-6.8.2-3.el8.x86_64.rpm 479 kB/s | 188 kB 00:00 2024-02-09T17:12:48.271 INFO:teuthology.orchestra.run.smithi179.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 706 kB/s | 71 kB 00:00 2024-02-09T17:12:48.423 INFO:teuthology.orchestra.run.smithi099.stdout:(1/6): libcephsqlite-17.2.7-499.g89a83b98.el8.x 540 kB/s | 178 kB 00:00 2024-02-09T17:12:48.482 INFO:teuthology.orchestra.run.smithi099.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 522 kB/s | 203 kB 00:00 2024-02-09T17:12:48.666 INFO:teuthology.orchestra.run.smithi099.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 1.7 MB/s | 323 kB 00:00 2024-02-09T17:12:48.758 INFO:teuthology.orchestra.run.smithi099.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 769 kB/s | 71 kB 00:00 2024-02-09T17:12:48.816 INFO:teuthology.orchestra.run.smithi099.stdout:(5/6): oniguruma-6.8.2-3.el8.x86_64.rpm 479 kB/s | 188 kB 00:00 2024-02-09T17:12:50.756 INFO:teuthology.orchestra.run.smithi162.stdout:(6/6): ceph-test-17.2.7-499.g89a83b98.el8.x86_6 13 MB/s | 52 MB 00:03 2024-02-09T17:12:50.759 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-02-09T17:12:50.759 INFO:teuthology.orchestra.run.smithi162.stdout:Total 13 MB/s | 53 MB 00:03 2024-02-09T17:12:50.800 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-02-09T17:12:50.821 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-02-09T17:12:50.821 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-02-09T17:12:51.109 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-02-09T17:12:51.112 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-02-09T17:12:51.602 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-02-09T17:12:51.817 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-02-09T17:12:52.028 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-02-09T17:12:52.176 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-09T17:12:52.260 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-09T17:12:52.458 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-02-09T17:12:52.642 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephsqlite-2:17.2.7-499.g89a83b98.el8.x86_64 5/6 2024-02-09T17:12:53.382 INFO:teuthology.orchestra.run.smithi099.stdout:(6/6): ceph-test-17.2.7-499.g89a83b98.el8.x86_6 9.8 MB/s | 52 MB 00:05 2024-02-09T17:12:53.384 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-02-09T17:12:53.384 INFO:teuthology.orchestra.run.smithi099.stdout:Total 9.6 MB/s | 53 MB 00:05 2024-02-09T17:12:53.425 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-02-09T17:12:53.446 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-02-09T17:12:53.446 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-02-09T17:12:53.732 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-02-09T17:12:53.735 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-02-09T17:12:53.897 INFO:teuthology.orchestra.run.smithi179.stdout:(6/6): ceph-test-17.2.7-499.g89a83b98.el8.x86_6 8.2 MB/s | 52 MB 00:06 2024-02-09T17:12:53.898 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-02-09T17:12:53.899 INFO:teuthology.orchestra.run.smithi179.stdout:Total 8.3 MB/s | 53 MB 00:06 2024-02-09T17:12:53.941 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-02-09T17:12:53.962 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-02-09T17:12:53.962 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-02-09T17:12:54.208 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-02-09T17:12:54.248 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-02-09T17:12:54.251 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-02-09T17:12:54.441 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-02-09T17:12:54.615 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-02-09T17:12:54.738 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-09T17:12:54.751 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-02-09T17:12:54.820 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-09T17:12:54.966 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-02-09T17:12:55.037 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-02-09T17:12:55.179 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libcephsqlite-2:17.2.7-499.g89a83b98.el8.x86_64 5/6 2024-02-09T17:12:55.190 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-02-09T17:12:55.330 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-09T17:12:55.392 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-09T17:12:55.603 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-02-09T17:12:55.787 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libcephsqlite-2:17.2.7-499.g89a83b98.el8.x86_64 5/6 2024-02-09T17:12:57.630 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libcephsqlite-2:17.2.7-499.g89a83b98.el8.x86_64 5/6 2024-02-09T17:12:57.882 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-test-2:17.2.7-499.g89a83b98.el8.x86_64 6/6 2024-02-09T17:12:58.153 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-test-2:17.2.7-499.g89a83b98.el8.x86_64 6/6 2024-02-09T17:12:58.153 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-test-2:17.2.7-499.g89a83b98.el8.x86_64 1/6 2024-02-09T17:12:58.153 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephsqlite-2:17.2.7-499.g89a83b98.el8.x86_64 2/6 2024-02-09T17:12:58.153 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-02-09T17:12:58.154 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-02-09T17:12:58.154 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-02-09T17:12:58.586 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-02-09T17:12:58.586 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:12:58.586 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-02-09T17:12:58.586 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-test-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:12:58.586 INFO:teuthology.orchestra.run.smithi162.stdout: jq-1.6-8.el8.x86_64 2024-02-09T17:12:58.587 INFO:teuthology.orchestra.run.smithi162.stdout: libcephsqlite-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:12:58.587 INFO:teuthology.orchestra.run.smithi162.stdout: oniguruma-6.8.2-3.el8.x86_64 2024-02-09T17:12:58.587 INFO:teuthology.orchestra.run.smithi162.stdout: socat-1.7.4.1-1.el8.x86_64 2024-02-09T17:12:58.587 INFO:teuthology.orchestra.run.smithi162.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-02-09T17:12:58.587 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:12:58.587 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-09T17:12:58.827 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph 2024-02-09T17:12:59.607 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:01 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:13:00.154 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: libcephsqlite-2:17.2.7-499.g89a83b98.el8.x86_64 5/6 2024-02-09T17:13:00.402 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-test-2:17.2.7-499.g89a83b98.el8.x86_64 6/6 2024-02-09T17:13:00.676 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-test-2:17.2.7-499.g89a83b98.el8.x86_64 6/6 2024-02-09T17:13:00.676 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-test-2:17.2.7-499.g89a83b98.el8.x86_64 1/6 2024-02-09T17:13:00.676 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libcephsqlite-2:17.2.7-499.g89a83b98.el8.x86_64 2/6 2024-02-09T17:13:00.676 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-02-09T17:13:00.676 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-02-09T17:13:00.676 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-02-09T17:13:00.758 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: libcephsqlite-2:17.2.7-499.g89a83b98.el8.x86_64 5/6 2024-02-09T17:13:01.035 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-test-2:17.2.7-499.g89a83b98.el8.x86_64 6/6 2024-02-09T17:13:01.114 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-09T17:13:01.118 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-09T17:13:01.118 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-02-09T17:13:01.118 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-09T17:13:01.118 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-02-09T17:13:01.118 INFO:teuthology.orchestra.run.smithi162.stdout: ceph x86_64 2:17.2.7-499.g89a83b98.el8 ceph 6.4 k 2024-02-09T17:13:01.118 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-02-09T17:13:01.118 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mds x86_64 2:17.2.7-499.g89a83b98.el8 ceph 2.3 M 2024-02-09T17:13:01.118 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr x86_64 2:17.2.7-499.g89a83b98.el8 ceph 1.6 M 2024-02-09T17:13:01.119 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-modules-core noarch 2:17.2.7-499.g89a83b98.el8 ceph-noarch 239 k 2024-02-09T17:13:01.119 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mon x86_64 2:17.2.7-499.g89a83b98.el8 ceph 3.9 M 2024-02-09T17:13:01.119 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-osd x86_64 2:17.2.7-499.g89a83b98.el8 ceph 17 M 2024-02-09T17:13:01.119 INFO:teuthology.orchestra.run.smithi162.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-02-09T17:13:01.119 INFO:teuthology.orchestra.run.smithi162.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-02-09T17:13:01.119 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-02-09T17:13:01.119 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-02-09T17:13:01.119 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-02-09T17:13:01.119 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-02-09T17:13:01.119 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-02-09T17:13:01.119 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-02-09T17:13:01.119 INFO:teuthology.orchestra.run.smithi162.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-02-09T17:13:01.120 INFO:teuthology.orchestra.run.smithi162.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-02-09T17:13:01.120 INFO:teuthology.orchestra.run.smithi162.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-02-09T17:13:01.120 INFO:teuthology.orchestra.run.smithi162.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-02-09T17:13:01.120 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-02-09T17:13:01.120 INFO:teuthology.orchestra.run.smithi162.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-02-09T17:13:01.120 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-02-09T17:13:01.120 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-02-09T17:13:01.120 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-02-09T17:13:01.120 INFO:teuthology.orchestra.run.smithi162.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-02-09T17:13:01.120 INFO:teuthology.orchestra.run.smithi162.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-02-09T17:13:01.120 INFO:teuthology.orchestra.run.smithi162.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-02-09T17:13:01.120 INFO:teuthology.orchestra.run.smithi162.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-02-09T17:13:01.120 INFO:teuthology.orchestra.run.smithi162.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-02-09T17:13:01.121 INFO:teuthology.orchestra.run.smithi162.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-02-09T17:13:01.121 INFO:teuthology.orchestra.run.smithi162.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-02-09T17:13:01.121 INFO:teuthology.orchestra.run.smithi162.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-02-09T17:13:01.121 INFO:teuthology.orchestra.run.smithi162.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-02-09T17:13:01.121 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:13:01.121 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-02-09T17:13:01.121 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-09T17:13:01.121 INFO:teuthology.orchestra.run.smithi162.stdout:Install 32 Packages 2024-02-09T17:13:01.121 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:13:01.122 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 28 M 2024-02-09T17:13:01.122 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 110 M 2024-02-09T17:13:01.122 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-02-09T17:13:01.132 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-02-09T17:13:01.133 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:13:01.133 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-02-09T17:13:01.133 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-test-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:13:01.133 INFO:teuthology.orchestra.run.smithi099.stdout: jq-1.6-8.el8.x86_64 2024-02-09T17:13:01.133 INFO:teuthology.orchestra.run.smithi099.stdout: libcephsqlite-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:13:01.133 INFO:teuthology.orchestra.run.smithi099.stdout: oniguruma-6.8.2-3.el8.x86_64 2024-02-09T17:13:01.133 INFO:teuthology.orchestra.run.smithi099.stdout: socat-1.7.4.1-1.el8.x86_64 2024-02-09T17:13:01.134 INFO:teuthology.orchestra.run.smithi099.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-02-09T17:13:01.134 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:13:01.134 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-02-09T17:13:01.309 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-test-2:17.2.7-499.g89a83b98.el8.x86_64 6/6 2024-02-09T17:13:01.309 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-test-2:17.2.7-499.g89a83b98.el8.x86_64 1/6 2024-02-09T17:13:01.309 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libcephsqlite-2:17.2.7-499.g89a83b98.el8.x86_64 2/6 2024-02-09T17:13:01.309 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-02-09T17:13:01.309 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-02-09T17:13:01.310 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-02-09T17:13:01.333 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph 2024-02-09T17:13:01.624 INFO:teuthology.orchestra.run.smithi162.stdout:(1/32): ceph-17.2.7-499.g89a83b98.el8.x86_64.rp 44 kB/s | 6.4 kB 00:00 2024-02-09T17:13:01.773 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-02-09T17:13:01.773 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:13:01.773 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-02-09T17:13:01.773 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-test-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:13:01.773 INFO:teuthology.orchestra.run.smithi179.stdout: jq-1.6-8.el8.x86_64 2024-02-09T17:13:01.773 INFO:teuthology.orchestra.run.smithi179.stdout: libcephsqlite-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:13:01.773 INFO:teuthology.orchestra.run.smithi179.stdout: oniguruma-6.8.2-3.el8.x86_64 2024-02-09T17:13:01.773 INFO:teuthology.orchestra.run.smithi179.stdout: socat-1.7.4.1-1.el8.x86_64 2024-02-09T17:13:01.773 INFO:teuthology.orchestra.run.smithi179.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-02-09T17:13:01.773 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:13:01.773 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-02-09T17:13:01.841 INFO:teuthology.orchestra.run.smithi162.stdout:(2/32): ceph-mds-17.2.7-499.g89a83b98.el8.x86_6 6.4 MB/s | 2.3 MB 00:00 2024-02-09T17:13:01.883 INFO:teuthology.orchestra.run.smithi162.stdout:(3/32): ceph-mgr-17.2.7-499.g89a83b98.el8.x86_6 3.9 MB/s | 1.6 MB 00:00 2024-02-09T17:13:02.034 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph 2024-02-09T17:13:02.086 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:04 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:13:02.142 INFO:teuthology.orchestra.run.smithi162.stdout:(4/32): ceph-mgr-modules-core-17.2.7-499.g89a83 924 kB/s | 239 kB 00:00 2024-02-09T17:13:02.676 INFO:teuthology.orchestra.run.smithi162.stdout:(5/32): ceph-osd-17.2.7-499.g89a83b98.el8.x86_6 20 MB/s | 17 MB 00:00 2024-02-09T17:13:02.701 INFO:teuthology.orchestra.run.smithi162.stdout:(6/32): python3-mako-1.0.6-14.el8.noarch.rpm 281 kB/s | 157 kB 00:00 2024-02-09T17:13:02.801 INFO:teuthology.orchestra.run.smithi162.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 1.0 MB/s | 103 kB 00:00 2024-02-09T17:13:02.802 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:04 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:13:02.893 INFO:teuthology.orchestra.run.smithi162.stdout:(8/32): ceph-mon-17.2.7-499.g89a83b98.el8.x86_6 3.1 MB/s | 3.9 MB 00:01 2024-02-09T17:13:02.910 INFO:teuthology.orchestra.run.smithi162.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 524 kB/s | 57 kB 00:00 2024-02-09T17:13:02.927 INFO:teuthology.orchestra.run.smithi162.stdout:(10/32): python3-markupsafe-0.23-19.el8.x86_64. 154 kB/s | 39 kB 00:00 2024-02-09T17:13:03.194 INFO:teuthology.orchestra.run.smithi162.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 877 kB/s | 249 kB 00:00 2024-02-09T17:13:03.236 INFO:teuthology.orchestra.run.smithi162.stdout:(12/32): python3-cryptography-3.2.1-7.el8.x86_6 2.0 MB/s | 641 kB 00:00 2024-02-09T17:13:03.261 INFO:teuthology.orchestra.run.smithi162.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 1.6 MB/s | 109 kB 00:00 2024-02-09T17:13:03.378 INFO:teuthology.orchestra.run.smithi162.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 307 kB/s | 44 kB 00:00 2024-02-09T17:13:03.420 INFO:teuthology.orchestra.run.smithi162.stdout:(15/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.1 MB/s | 185 kB 00:00 2024-02-09T17:13:03.445 INFO:teuthology.orchestra.run.smithi162.stdout:(16/32): python3-werkzeug-0.12.2-4.el8.noarch.r 829 kB/s | 457 kB 00:00 2024-02-09T17:13:03.513 INFO:teuthology.orchestra.run.smithi162.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.3 MB/s | 173 kB 00:00 2024-02-09T17:13:03.537 INFO:teuthology.orchestra.run.smithi162.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 121 kB/s | 11 kB 00:00 2024-02-09T17:13:03.572 INFO:teuthology.orchestra.run.smithi162.stdout:(19/32): python3-cherrypy-18.4.0-1.el8.noarch.r 2.5 MB/s | 384 kB 00:00 2024-02-09T17:13:03.586 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-02-09T17:13:03.590 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-02-09T17:13:03.590 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2024-02-09T17:13:03.590 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-02-09T17:13:03.590 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-02-09T17:13:03.590 INFO:teuthology.orchestra.run.smithi099.stdout: ceph x86_64 2:17.2.7-499.g89a83b98.el8 ceph 6.4 k 2024-02-09T17:13:03.590 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2024-02-09T17:13:03.591 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mds x86_64 2:17.2.7-499.g89a83b98.el8 ceph 2.3 M 2024-02-09T17:13:03.591 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr x86_64 2:17.2.7-499.g89a83b98.el8 ceph 1.6 M 2024-02-09T17:13:03.591 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-modules-core noarch 2:17.2.7-499.g89a83b98.el8 ceph-noarch 239 k 2024-02-09T17:13:03.591 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mon x86_64 2:17.2.7-499.g89a83b98.el8 ceph 3.9 M 2024-02-09T17:13:03.591 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-osd x86_64 2:17.2.7-499.g89a83b98.el8 ceph 17 M 2024-02-09T17:13:03.591 INFO:teuthology.orchestra.run.smithi099.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-02-09T17:13:03.591 INFO:teuthology.orchestra.run.smithi099.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-02-09T17:13:03.591 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-02-09T17:13:03.591 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-02-09T17:13:03.592 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-02-09T17:13:03.592 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-02-09T17:13:03.592 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-02-09T17:13:03.592 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-02-09T17:13:03.592 INFO:teuthology.orchestra.run.smithi099.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-02-09T17:13:03.592 INFO:teuthology.orchestra.run.smithi099.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-02-09T17:13:03.593 INFO:teuthology.orchestra.run.smithi099.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-02-09T17:13:03.593 INFO:teuthology.orchestra.run.smithi099.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-02-09T17:13:03.593 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-02-09T17:13:03.593 INFO:teuthology.orchestra.run.smithi099.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-02-09T17:13:03.593 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-02-09T17:13:03.593 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-02-09T17:13:03.593 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-02-09T17:13:03.594 INFO:teuthology.orchestra.run.smithi099.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-02-09T17:13:03.594 INFO:teuthology.orchestra.run.smithi099.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-02-09T17:13:03.594 INFO:teuthology.orchestra.run.smithi099.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-02-09T17:13:03.594 INFO:teuthology.orchestra.run.smithi099.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-02-09T17:13:03.594 INFO:teuthology.orchestra.run.smithi099.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-02-09T17:13:03.594 INFO:teuthology.orchestra.run.smithi099.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-02-09T17:13:03.594 INFO:teuthology.orchestra.run.smithi099.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-02-09T17:13:03.594 INFO:teuthology.orchestra.run.smithi099.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-02-09T17:13:03.594 INFO:teuthology.orchestra.run.smithi099.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-02-09T17:13:03.595 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:13:03.595 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-02-09T17:13:03.595 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-02-09T17:13:03.595 INFO:teuthology.orchestra.run.smithi099.stdout:Install 32 Packages 2024-02-09T17:13:03.595 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:13:03.596 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 28 M 2024-02-09T17:13:03.596 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 110 M 2024-02-09T17:13:03.596 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-02-09T17:13:03.597 INFO:teuthology.orchestra.run.smithi162.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 210 kB/s | 18 kB 00:00 2024-02-09T17:13:03.622 INFO:teuthology.orchestra.run.smithi162.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 580 kB/s | 49 kB 00:00 2024-02-09T17:13:03.647 INFO:teuthology.orchestra.run.smithi162.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 778 kB/s | 59 kB 00:00 2024-02-09T17:13:03.672 INFO:teuthology.orchestra.run.smithi162.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 311 kB/s | 16 kB 00:00 2024-02-09T17:13:03.698 INFO:teuthology.orchestra.run.smithi162.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 370 kB/s | 19 kB 00:00 2024-02-09T17:13:03.732 INFO:teuthology.orchestra.run.smithi162.stdout:(25/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.1 MB/s | 283 kB 00:00 2024-02-09T17:13:03.757 INFO:teuthology.orchestra.run.smithi162.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 284 kB/s | 24 kB 00:00 2024-02-09T17:13:03.782 INFO:teuthology.orchestra.run.smithi162.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 349 kB/s | 29 kB 00:00 2024-02-09T17:13:03.808 INFO:teuthology.orchestra.run.smithi162.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 364 kB/s | 27 kB 00:00 2024-02-09T17:13:03.858 INFO:teuthology.orchestra.run.smithi099.stdout:(1/32): ceph-17.2.7-499.g89a83b98.el8.x86_64.rp 42 kB/s | 6.4 kB 00:00 2024-02-09T17:13:03.858 INFO:teuthology.orchestra.run.smithi162.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.7 MB/s | 85 kB 00:00 2024-02-09T17:13:03.883 INFO:teuthology.orchestra.run.smithi162.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 3.3 MB/s | 251 kB 00:00 2024-02-09T17:13:03.908 INFO:teuthology.orchestra.run.smithi162.stdout:(31/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.7 MB/s | 255 kB 00:00 2024-02-09T17:13:03.933 INFO:teuthology.orchestra.run.smithi162.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 305 kB/s | 23 kB 00:00 2024-02-09T17:13:03.935 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-02-09T17:13:03.935 INFO:teuthology.orchestra.run.smithi162.stdout:Total 10 MB/s | 28 MB 00:02 2024-02-09T17:13:04.017 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-02-09T17:13:04.055 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-02-09T17:13:04.055 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-02-09T17:13:04.216 INFO:teuthology.orchestra.run.smithi099.stdout:(2/32): ceph-mon-17.2.7-499.g89a83b98.el8.x86_6 11 MB/s | 3.9 MB 00:00 2024-02-09T17:13:04.258 INFO:teuthology.orchestra.run.smithi099.stdout:(3/32): ceph-mgr-17.2.7-499.g89a83b98.el8.x86_6 2.9 MB/s | 1.6 MB 00:00 2024-02-09T17:13:04.321 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-02-09T17:13:04.324 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-02-09T17:13:04.324 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-02-09T17:13:04.325 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-02-09T17:13:04.325 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-02-09T17:13:04.325 INFO:teuthology.orchestra.run.smithi179.stdout: ceph x86_64 2:17.2.7-499.g89a83b98.el8 ceph 6.4 k 2024-02-09T17:13:04.325 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-02-09T17:13:04.325 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mds x86_64 2:17.2.7-499.g89a83b98.el8 ceph 2.3 M 2024-02-09T17:13:04.325 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr x86_64 2:17.2.7-499.g89a83b98.el8 ceph 1.6 M 2024-02-09T17:13:04.325 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-modules-core noarch 2:17.2.7-499.g89a83b98.el8 ceph-noarch 239 k 2024-02-09T17:13:04.325 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mon x86_64 2:17.2.7-499.g89a83b98.el8 ceph 3.9 M 2024-02-09T17:13:04.325 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-osd x86_64 2:17.2.7-499.g89a83b98.el8 ceph 17 M 2024-02-09T17:13:04.326 INFO:teuthology.orchestra.run.smithi179.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-02-09T17:13:04.326 INFO:teuthology.orchestra.run.smithi179.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-02-09T17:13:04.326 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-02-09T17:13:04.326 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-02-09T17:13:04.326 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-02-09T17:13:04.326 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-02-09T17:13:04.326 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-02-09T17:13:04.326 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-02-09T17:13:04.327 INFO:teuthology.orchestra.run.smithi179.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-02-09T17:13:04.327 INFO:teuthology.orchestra.run.smithi179.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-02-09T17:13:04.327 INFO:teuthology.orchestra.run.smithi179.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-02-09T17:13:04.327 INFO:teuthology.orchestra.run.smithi179.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-02-09T17:13:04.327 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-02-09T17:13:04.327 INFO:teuthology.orchestra.run.smithi179.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-02-09T17:13:04.327 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-02-09T17:13:04.327 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-02-09T17:13:04.327 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-02-09T17:13:04.327 INFO:teuthology.orchestra.run.smithi179.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-02-09T17:13:04.328 INFO:teuthology.orchestra.run.smithi179.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-02-09T17:13:04.328 INFO:teuthology.orchestra.run.smithi179.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-02-09T17:13:04.328 INFO:teuthology.orchestra.run.smithi179.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-02-09T17:13:04.328 INFO:teuthology.orchestra.run.smithi179.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-02-09T17:13:04.328 INFO:teuthology.orchestra.run.smithi179.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-02-09T17:13:04.328 INFO:teuthology.orchestra.run.smithi179.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-02-09T17:13:04.328 INFO:teuthology.orchestra.run.smithi179.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-02-09T17:13:04.328 INFO:teuthology.orchestra.run.smithi179.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-02-09T17:13:04.329 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:13:04.329 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-02-09T17:13:04.329 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-02-09T17:13:04.329 INFO:teuthology.orchestra.run.smithi179.stdout:Install 32 Packages 2024-02-09T17:13:04.329 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:13:04.329 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 28 M 2024-02-09T17:13:04.330 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 110 M 2024-02-09T17:13:04.330 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-02-09T17:13:04.360 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-02-09T17:13:04.363 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-02-09T17:13:04.417 INFO:teuthology.orchestra.run.smithi099.stdout:(4/32): ceph-mgr-modules-core-17.2.7-499.g89a83 1.5 MB/s | 239 kB 00:00 2024-02-09T17:13:04.817 INFO:teuthology.orchestra.run.smithi099.stdout:(5/32): python3-mako-1.0.6-14.el8.noarch.rpm 392 kB/s | 157 kB 00:00 2024-02-09T17:13:04.901 INFO:teuthology.orchestra.run.smithi099.stdout:(6/32): ceph-mds-17.2.7-499.g89a83b98.el8.x86_6 1.9 MB/s | 2.3 MB 00:01 2024-02-09T17:13:05.015 INFO:teuthology.orchestra.run.smithi179.stdout:(1/32): ceph-17.2.7-499.g89a83b98.el8.x86_64.rp 48 kB/s | 6.4 kB 00:00 2024-02-09T17:13:05.068 INFO:teuthology.orchestra.run.smithi099.stdout:(7/32): ceph-osd-17.2.7-499.g89a83b98.el8.x86_6 20 MB/s | 17 MB 00:00 2024-02-09T17:13:05.085 INFO:teuthology.orchestra.run.smithi099.stdout:(8/32): python3-markupsafe-0.23-19.el8.x86_64.r 145 kB/s | 39 kB 00:00 2024-02-09T17:13:05.201 INFO:teuthology.orchestra.run.smithi099.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 491 kB/s | 57 kB 00:00 2024-02-09T17:13:05.240 INFO:teuthology.orchestra.run.smithi179.stdout:(2/32): ceph-mgr-17.2.7-499.g89a83b98.el8.x86_6 4.4 MB/s | 1.6 MB 00:00 2024-02-09T17:13:05.291 INFO:teuthology.orchestra.run.smithi179.stdout:(3/32): ceph-mds-17.2.7-499.g89a83b98.el8.x86_6 5.7 MB/s | 2.3 MB 00:00 2024-02-09T17:13:05.390 INFO:teuthology.orchestra.run.smithi179.stdout:(4/32): ceph-mgr-modules-core-17.2.7-499.g89a83 2.3 MB/s | 239 kB 00:00 2024-02-09T17:13:05.410 INFO:teuthology.orchestra.run.smithi099.stdout:(10/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 203 kB/s | 103 kB 00:00 2024-02-09T17:13:05.459 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-02-09T17:13:05.477 INFO:teuthology.orchestra.run.smithi099.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 902 kB/s | 249 kB 00:00 2024-02-09T17:13:05.544 INFO:teuthology.orchestra.run.smithi099.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 1.6 MB/s | 109 kB 00:00 2024-02-09T17:13:05.678 INFO:teuthology.orchestra.run.smithi099.stdout:(13/32): python3-werkzeug-0.12.2-4.el8.noarch.r 772 kB/s | 457 kB 00:00 2024-02-09T17:13:05.687 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-02-09T17:13:05.741 INFO:teuthology.orchestra.run.smithi179.stdout:(5/32): ceph-mon-17.2.7-499.g89a83b98.el8.x86_6 5.4 MB/s | 3.9 MB 00:00 2024-02-09T17:13:05.770 INFO:teuthology.orchestra.run.smithi099.stdout:(14/32): python3-cryptography-3.2.1-7.el8.x86_6 1.7 MB/s | 641 kB 00:00 2024-02-09T17:13:05.783 INFO:teuthology.orchestra.run.smithi179.stdout:(6/32): python3-mako-1.0.6-14.el8.noarch.rpm 400 kB/s | 157 kB 00:00 2024-02-09T17:13:05.787 INFO:teuthology.orchestra.run.smithi099.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 180 kB/s | 44 kB 00:00 2024-02-09T17:13:05.883 INFO:teuthology.orchestra.run.smithi179.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 1.0 MB/s | 103 kB 00:00 2024-02-09T17:13:05.912 INFO:teuthology.orchestra.run.smithi099.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.2 MB/s | 173 kB 00:00 2024-02-09T17:13:05.930 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-02-09T17:13:05.946 INFO:teuthology.orchestra.run.smithi099.stdout:(17/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 693 kB/s | 185 kB 00:00 2024-02-09T17:13:05.979 INFO:teuthology.orchestra.run.smithi099.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 166 kB/s | 11 kB 00:00 2024-02-09T17:13:05.983 INFO:teuthology.orchestra.run.smithi179.stdout:(8/32): python3-pytz-2017.2-11.el8.noarch.rpm 570 kB/s | 57 kB 00:00 2024-02-09T17:13:06.009 INFO:teuthology.orchestra.run.smithi179.stdout:(9/32): python3-markupsafe-0.23-19.el8.x86_64.r 145 kB/s | 39 kB 00:00 2024-02-09T17:13:06.021 INFO:teuthology.orchestra.run.smithi099.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 233 kB/s | 18 kB 00:00 2024-02-09T17:13:06.063 INFO:teuthology.orchestra.run.smithi099.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 584 kB/s | 49 kB 00:00 2024-02-09T17:13:06.088 INFO:teuthology.orchestra.run.smithi099.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 878 kB/s | 59 kB 00:00 2024-02-09T17:13:06.113 INFO:teuthology.orchestra.run.smithi099.stdout:(22/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.2 MB/s | 384 kB 00:00 2024-02-09T17:13:06.155 INFO:teuthology.orchestra.run.smithi099.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 233 kB/s | 16 kB 00:00 2024-02-09T17:13:06.156 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-02-09T17:13:06.184 INFO:teuthology.orchestra.run.smithi179.stdout:(10/32): ceph-osd-17.2.7-499.g89a83b98.el8.x86_ 18 MB/s | 17 MB 00:00 2024-02-09T17:13:06.189 INFO:teuthology.orchestra.run.smithi099.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 249 kB/s | 19 kB 00:00 2024-02-09T17:13:06.210 INFO:teuthology.orchestra.run.smithi179.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.0 MB/s | 457 kB 00:00 2024-02-09T17:13:06.214 INFO:teuthology.orchestra.run.smithi099.stdout:(25/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.8 MB/s | 283 kB 00:00 2024-02-09T17:13:06.240 INFO:teuthology.orchestra.run.smithi099.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 285 kB/s | 24 kB 00:00 2024-02-09T17:13:06.273 INFO:teuthology.orchestra.run.smithi099.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 349 kB/s | 29 kB 00:00 2024-02-09T17:13:06.298 INFO:teuthology.orchestra.run.smithi099.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 329 kB/s | 27 kB 00:00 2024-02-09T17:13:06.365 INFO:teuthology.orchestra.run.smithi099.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.7 MB/s | 251 kB 00:00 2024-02-09T17:13:06.391 INFO:teuthology.orchestra.run.smithi099.stdout:(30/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.7 MB/s | 255 kB 00:00 2024-02-09T17:13:06.410 INFO:teuthology.orchestra.run.smithi179.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 621 kB/s | 249 kB 00:00 2024-02-09T17:13:06.412 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-02-09T17:13:06.416 INFO:teuthology.orchestra.run.smithi099.stdout:(31/32): python3-webtest-2.0.33-1.el8.noarch.rp 728 kB/s | 85 kB 00:00 2024-02-09T17:13:06.435 INFO:teuthology.orchestra.run.smithi179.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 484 kB/s | 109 kB 00:00 2024-02-09T17:13:06.449 INFO:teuthology.orchestra.run.smithi099.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 275 kB/s | 23 kB 00:00 2024-02-09T17:13:06.451 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-02-09T17:13:06.451 INFO:teuthology.orchestra.run.smithi099.stdout:Total 10 MB/s | 28 MB 00:02 2024-02-09T17:13:06.477 INFO:teuthology.orchestra.run.smithi179.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 653 kB/s | 44 kB 00:00 2024-02-09T17:13:06.532 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-02-09T17:13:06.536 INFO:teuthology.orchestra.run.smithi179.stdout:(15/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.8 MB/s | 185 kB 00:00 2024-02-09T17:13:06.561 INFO:teuthology.orchestra.run.smithi179.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 2.0 MB/s | 173 kB 00:00 2024-02-09T17:13:06.569 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-02-09T17:13:06.569 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-02-09T17:13:06.586 INFO:teuthology.orchestra.run.smithi179.stdout:(17/32): python3-cryptography-3.2.1-7.el8.x86_6 1.6 MB/s | 641 kB 00:00 2024-02-09T17:13:06.593 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-02-09T17:13:06.612 INFO:teuthology.orchestra.run.smithi179.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 220 kB/s | 11 kB 00:00 2024-02-09T17:13:06.637 INFO:teuthology.orchestra.run.smithi179.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 349 kB/s | 18 kB 00:00 2024-02-09T17:13:06.670 INFO:teuthology.orchestra.run.smithi179.stdout:(20/32): python3-cherrypy-18.4.0-1.el8.noarch.r 2.8 MB/s | 384 kB 00:00 2024-02-09T17:13:06.696 INFO:teuthology.orchestra.run.smithi179.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 582 kB/s | 49 kB 00:00 2024-02-09T17:13:06.721 INFO:teuthology.orchestra.run.smithi179.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 703 kB/s | 59 kB 00:00 2024-02-09T17:13:06.746 INFO:teuthology.orchestra.run.smithi179.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 3.7 MB/s | 283 kB 00:00 2024-02-09T17:13:06.771 INFO:teuthology.orchestra.run.smithi179.stdout:(24/32): python3-portend-2.6-1.el8.noarch.rpm 310 kB/s | 16 kB 00:00 2024-02-09T17:13:06.774 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-02-09T17:13:06.797 INFO:teuthology.orchestra.run.smithi179.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 248 kB/s | 19 kB 00:00 2024-02-09T17:13:06.822 INFO:teuthology.orchestra.run.smithi179.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 317 kB/s | 24 kB 00:00 2024-02-09T17:13:06.847 INFO:teuthology.orchestra.run.smithi179.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 388 kB/s | 29 kB 00:00 2024-02-09T17:13:06.864 INFO:teuthology.orchestra.run.smithi179.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 411 kB/s | 27 kB 00:00 2024-02-09T17:13:06.870 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-02-09T17:13:06.873 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-02-09T17:13:06.889 INFO:teuthology.orchestra.run.smithi179.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 3.7 MB/s | 255 kB 00:00 2024-02-09T17:13:06.915 INFO:teuthology.orchestra.run.smithi179.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.7 MB/s | 85 kB 00:00 2024-02-09T17:13:06.932 INFO:teuthology.orchestra.run.smithi179.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 548 kB/s | 23 kB 00:00 2024-02-09T17:13:06.957 INFO:teuthology.orchestra.run.smithi179.stdout:(32/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.3 MB/s | 251 kB 00:00 2024-02-09T17:13:06.958 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-02-09T17:13:06.959 INFO:teuthology.orchestra.run.smithi179.stdout:Total 11 MB/s | 28 MB 00:02 2024-02-09T17:13:06.975 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-02-09T17:13:07.042 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-02-09T17:13:07.078 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-02-09T17:13:07.078 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-02-09T17:13:07.186 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-02-09T17:13:07.401 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-02-09T17:13:07.404 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-02-09T17:13:07.414 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-02-09T17:13:07.659 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-02-09T17:13:07.888 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-02-09T17:13:07.964 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-02-09T17:13:08.145 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-02-09T17:13:08.190 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-02-09T17:13:08.426 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-02-09T17:13:08.526 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-02-09T17:13:08.545 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-02-09T17:13:08.651 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-02-09T17:13:08.766 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-02-09T17:13:08.809 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-02-09T17:13:08.874 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-02-09T17:13:08.993 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-02-09T17:13:09.013 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-02-09T17:13:09.072 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-02-09T17:13:09.218 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-02-09T17:13:09.251 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-02-09T17:13:09.331 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-02-09T17:13:09.420 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-02-09T17:13:09.459 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-02-09T17:13:09.573 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-09T17:13:09.589 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-02-09T17:13:09.665 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-02-09T17:13:09.829 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-02-09T17:13:09.862 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-02-09T17:13:09.907 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-02-09T17:13:10.033 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-02-09T17:13:10.089 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-02-09T17:13:10.130 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-02-09T17:13:10.246 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-02-09T17:13:10.283 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-02-09T17:13:10.299 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-02-09T17:13:10.493 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-02-09T17:13:10.510 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-02-09T17:13:10.519 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-02-09T17:13:10.738 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-02-09T17:13:10.762 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-02-09T17:13:10.895 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-02-09T17:13:10.967 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-02-09T17:13:11.008 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-02-09T17:13:11.153 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-02-09T17:13:11.224 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-02-09T17:13:11.249 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-02-09T17:13:11.341 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-02-09T17:13:11.557 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-02-09T17:13:11.612 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-02-09T17:13:11.640 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-02-09T17:13:11.877 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-09T17:13:11.885 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-02-09T17:13:11.913 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-02-09T17:13:12.110 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-02-09T17:13:12.190 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-02-09T17:13:12.268 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-499.g89a83b98.el8.n 27/32 2024-02-09T17:13:12.437 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-02-09T17:13:12.488 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-2:17.2.7-499.g89a83b98.el8.x86_64 28/32 2024-02-09T17:13:12.565 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-02-09T17:13:12.636 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-09T17:13:12.729 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-02-09T17:13:12.923 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-02-09T17:13:12.978 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-02-09T17:13:13.213 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-02-09T17:13:13.229 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-02-09T17:13:13.387 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-02-09T17:13:13.471 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-02-09T17:13:13.607 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-02-09T17:13:13.706 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-02-09T17:13:13.842 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-02-09T17:13:13.971 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-2:17.2.7-499.g89a83b98.el8.x86_64 28/32 2024-02-09T17:13:14.024 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-02-09T17:13:14.114 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-02-09T17:13:14.163 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-osd-2:17.2.7-499.g89a83b98.el8.x86_64 29/32 2024-02-09T17:13:14.314 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-02-09T17:13:14.336 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-02-09T17:13:14.556 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-osd-2:17.2.7-499.g89a83b98.el8.x86_64 29/32 2024-02-09T17:13:14.672 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-02-09T17:13:14.722 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-499.g89a83b98.el8.n 27/32 2024-02-09T17:13:14.755 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mon-2:17.2.7-499.g89a83b98.el8.x86_64 30/32 2024-02-09T17:13:14.943 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-2:17.2.7-499.g89a83b98.el8.x86_64 28/32 2024-02-09T17:13:14.970 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mon-2:17.2.7-499.g89a83b98.el8.x86_64 30/32 2024-02-09T17:13:14.990 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-02-09T17:13:15.171 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mds-2:17.2.7-499.g89a83b98.el8.x86_64 31/32 2024-02-09T17:13:15.188 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mds-2:17.2.7-499.g89a83b98.el8.x86_64 31/32 2024-02-09T17:13:15.372 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-499.g89a83b98.el8.n 27/32 2024-02-09T17:13:15.556 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-2:17.2.7-499.g89a83b98.el8.x86_64 32/32 2024-02-09T17:13:15.588 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-2:17.2.7-499.g89a83b98.el8.x86_64 28/32 2024-02-09T17:13:16.329 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-2:17.2.7-499.g89a83b98.el8.x86_64 32/32 2024-02-09T17:13:16.329 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-2:17.2.7-499.g89a83b98.el8.x86_64 1/32 2024-02-09T17:13:16.329 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mds-2:17.2.7-499.g89a83b98.el8.x86_64 2/32 2024-02-09T17:13:16.329 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-2:17.2.7-499.g89a83b98.el8.x86_64 3/32 2024-02-09T17:13:16.329 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mon-2:17.2.7-499.g89a83b98.el8.x86_64 4/32 2024-02-09T17:13:16.329 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-osd-2:17.2.7-499.g89a83b98.el8.x86_64 5/32 2024-02-09T17:13:16.329 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-499.g89a83b98.el8.n 6/32 2024-02-09T17:13:16.329 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-02-09T17:13:16.329 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-02-09T17:13:16.329 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-02-09T17:13:16.329 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-02-09T17:13:16.330 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-02-09T17:13:16.330 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-02-09T17:13:16.330 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-02-09T17:13:16.330 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-02-09T17:13:16.330 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-02-09T17:13:16.330 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-02-09T17:13:16.330 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-09T17:13:16.330 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-02-09T17:13:16.330 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-02-09T17:13:16.330 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-02-09T17:13:16.330 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-02-09T17:13:16.330 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-02-09T17:13:16.330 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-02-09T17:13:16.330 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-02-09T17:13:16.331 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-02-09T17:13:16.331 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-02-09T17:13:16.331 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-02-09T17:13:16.331 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-02-09T17:13:16.331 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-02-09T17:13:16.331 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-02-09T17:13:16.331 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-02-09T17:13:16.420 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mgr-2:17.2.7-499.g89a83b98.el8.x86_64 28/32 2024-02-09T17:13:16.626 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-osd-2:17.2.7-499.g89a83b98.el8.x86_64 29/32 2024-02-09T17:13:17.026 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-osd-2:17.2.7-499.g89a83b98.el8.x86_64 29/32 2024-02-09T17:13:17.089 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mgr-2:17.2.7-499.g89a83b98.el8.x86_64 28/32 2024-02-09T17:13:17.166 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-02-09T17:13:17.166 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:13:17.166 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-02-09T17:13:17.166 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:13:17.166 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mds-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:13:17.166 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:13:17.166 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-modules-core-2:17.2.7-499.g89a83b98.el8.noarch 2024-02-09T17:13:17.166 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mon-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:13:17.167 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-osd-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:13:17.167 INFO:teuthology.orchestra.run.smithi162.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-02-09T17:13:17.167 INFO:teuthology.orchestra.run.smithi162.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-02-09T17:13:17.167 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-02-09T17:13:17.167 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-02-09T17:13:17.167 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-02-09T17:13:17.167 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-02-09T17:13:17.167 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-6.2-6.el8.noarch 2024-02-09T17:13:17.167 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-02-09T17:13:17.167 INFO:teuthology.orchestra.run.smithi162.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-02-09T17:13:17.167 INFO:teuthology.orchestra.run.smithi162.stdout: python3-mako-1.0.6-14.el8.noarch 2024-02-09T17:13:17.167 INFO:teuthology.orchestra.run.smithi162.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-02-09T17:13:17.167 INFO:teuthology.orchestra.run.smithi162.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-02-09T17:13:17.167 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-02-09T17:13:17.168 INFO:teuthology.orchestra.run.smithi162.stdout: python3-portend-2.6-1.el8.noarch 2024-02-09T17:13:17.168 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-02-09T17:13:17.168 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pycparser-2.14-14.el8.noarch 2024-02-09T17:13:17.168 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pytz-2017.2-11.el8.noarch 2024-02-09T17:13:17.168 INFO:teuthology.orchestra.run.smithi162.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-02-09T17:13:17.168 INFO:teuthology.orchestra.run.smithi162.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-02-09T17:13:17.168 INFO:teuthology.orchestra.run.smithi162.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-02-09T17:13:17.168 INFO:teuthology.orchestra.run.smithi162.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-02-09T17:13:17.168 INFO:teuthology.orchestra.run.smithi162.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-02-09T17:13:17.168 INFO:teuthology.orchestra.run.smithi162.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-02-09T17:13:17.168 INFO:teuthology.orchestra.run.smithi162.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-02-09T17:13:17.168 INFO:teuthology.orchestra.run.smithi162.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-02-09T17:13:17.168 INFO:teuthology.orchestra.run.smithi162.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-02-09T17:13:17.169 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:13:17.169 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-09T17:13:17.288 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-osd-2:17.2.7-499.g89a83b98.el8.x86_64 29/32 2024-02-09T17:13:17.432 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-base 2024-02-09T17:13:17.458 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mon-2:17.2.7-499.g89a83b98.el8.x86_64 30/32 2024-02-09T17:13:17.675 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mon-2:17.2.7-499.g89a83b98.el8.x86_64 30/32 2024-02-09T17:13:17.690 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-osd-2:17.2.7-499.g89a83b98.el8.x86_64 29/32 2024-02-09T17:13:17.849 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mds-2:17.2.7-499.g89a83b98.el8.x86_64 31/32 2024-02-09T17:13:17.866 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mds-2:17.2.7-499.g89a83b98.el8.x86_64 31/32 2024-02-09T17:13:17.885 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mon-2:17.2.7-499.g89a83b98.el8.x86_64 30/32 2024-02-09T17:13:17.982 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-2:17.2.7-499.g89a83b98.el8.x86_64 32/32 2024-02-09T17:13:18.101 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mon-2:17.2.7-499.g89a83b98.el8.x86_64 30/32 2024-02-09T17:13:18.196 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:20 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:13:18.285 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mds-2:17.2.7-499.g89a83b98.el8.x86_64 31/32 2024-02-09T17:13:18.303 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mds-2:17.2.7-499.g89a83b98.el8.x86_64 31/32 2024-02-09T17:13:18.388 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-2:17.2.7-499.g89a83b98.el8.x86_64 32/32 2024-02-09T17:13:18.389 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-2:17.2.7-499.g89a83b98.el8.x86_64 1/32 2024-02-09T17:13:18.389 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mds-2:17.2.7-499.g89a83b98.el8.x86_64 2/32 2024-02-09T17:13:18.389 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-2:17.2.7-499.g89a83b98.el8.x86_64 3/32 2024-02-09T17:13:18.389 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mon-2:17.2.7-499.g89a83b98.el8.x86_64 4/32 2024-02-09T17:13:18.389 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-osd-2:17.2.7-499.g89a83b98.el8.x86_64 5/32 2024-02-09T17:13:18.389 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-499.g89a83b98.el8.n 6/32 2024-02-09T17:13:18.389 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-02-09T17:13:18.389 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-02-09T17:13:18.389 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-02-09T17:13:18.389 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-02-09T17:13:18.390 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-02-09T17:13:18.390 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-02-09T17:13:18.390 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-02-09T17:13:18.390 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-02-09T17:13:18.390 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-02-09T17:13:18.390 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-02-09T17:13:18.390 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-09T17:13:18.390 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-02-09T17:13:18.390 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-02-09T17:13:18.390 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-02-09T17:13:18.391 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-02-09T17:13:18.391 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-02-09T17:13:18.391 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-02-09T17:13:18.391 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-02-09T17:13:18.391 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-02-09T17:13:18.391 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-02-09T17:13:18.391 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-02-09T17:13:18.391 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-02-09T17:13:18.391 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-02-09T17:13:18.391 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-02-09T17:13:18.392 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-02-09T17:13:18.426 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-2:17.2.7-499.g89a83b98.el8.x86_64 32/32 2024-02-09T17:13:18.836 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-2:17.2.7-499.g89a83b98.el8.x86_64 32/32 2024-02-09T17:13:18.836 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-2:17.2.7-499.g89a83b98.el8.x86_64 1/32 2024-02-09T17:13:18.836 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mds-2:17.2.7-499.g89a83b98.el8.x86_64 2/32 2024-02-09T17:13:18.836 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-2:17.2.7-499.g89a83b98.el8.x86_64 3/32 2024-02-09T17:13:18.836 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mon-2:17.2.7-499.g89a83b98.el8.x86_64 4/32 2024-02-09T17:13:18.836 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-osd-2:17.2.7-499.g89a83b98.el8.x86_64 5/32 2024-02-09T17:13:18.837 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-499.g89a83b98.el8.n 6/32 2024-02-09T17:13:18.837 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-02-09T17:13:18.837 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-02-09T17:13:18.837 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-02-09T17:13:18.837 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-02-09T17:13:18.837 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-02-09T17:13:18.837 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-02-09T17:13:18.837 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-02-09T17:13:18.837 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-02-09T17:13:18.837 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-02-09T17:13:18.837 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-02-09T17:13:18.837 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-09T17:13:18.837 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-02-09T17:13:18.837 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-02-09T17:13:18.837 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-02-09T17:13:18.838 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-02-09T17:13:18.838 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-02-09T17:13:18.838 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-02-09T17:13:18.838 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-02-09T17:13:18.838 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-02-09T17:13:18.838 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-02-09T17:13:18.838 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-02-09T17:13:18.838 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-02-09T17:13:18.838 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-02-09T17:13:18.838 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-02-09T17:13:18.838 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-02-09T17:13:19.259 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-02-09T17:13:19.259 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:13:19.259 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-02-09T17:13:19.259 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:13:19.260 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mds-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:13:19.260 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:13:19.260 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-modules-core-2:17.2.7-499.g89a83b98.el8.noarch 2024-02-09T17:13:19.260 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mon-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:13:19.260 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-osd-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:13:19.260 INFO:teuthology.orchestra.run.smithi099.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-02-09T17:13:19.260 INFO:teuthology.orchestra.run.smithi099.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-02-09T17:13:19.260 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-02-09T17:13:19.260 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-02-09T17:13:19.260 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-02-09T17:13:19.260 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-02-09T17:13:19.260 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco-6.2-6.el8.noarch 2024-02-09T17:13:19.260 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-02-09T17:13:19.260 INFO:teuthology.orchestra.run.smithi099.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-02-09T17:13:19.261 INFO:teuthology.orchestra.run.smithi099.stdout: python3-mako-1.0.6-14.el8.noarch 2024-02-09T17:13:19.261 INFO:teuthology.orchestra.run.smithi099.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-02-09T17:13:19.261 INFO:teuthology.orchestra.run.smithi099.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-02-09T17:13:19.261 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-02-09T17:13:19.261 INFO:teuthology.orchestra.run.smithi099.stdout: python3-portend-2.6-1.el8.noarch 2024-02-09T17:13:19.261 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-02-09T17:13:19.261 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pycparser-2.14-14.el8.noarch 2024-02-09T17:13:19.261 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pytz-2017.2-11.el8.noarch 2024-02-09T17:13:19.261 INFO:teuthology.orchestra.run.smithi099.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-02-09T17:13:19.261 INFO:teuthology.orchestra.run.smithi099.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-02-09T17:13:19.261 INFO:teuthology.orchestra.run.smithi099.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-02-09T17:13:19.261 INFO:teuthology.orchestra.run.smithi099.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-02-09T17:13:19.261 INFO:teuthology.orchestra.run.smithi099.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-02-09T17:13:19.261 INFO:teuthology.orchestra.run.smithi099.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-02-09T17:13:19.261 INFO:teuthology.orchestra.run.smithi099.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-02-09T17:13:19.262 INFO:teuthology.orchestra.run.smithi099.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-02-09T17:13:19.262 INFO:teuthology.orchestra.run.smithi099.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-02-09T17:13:19.262 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:13:19.262 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-02-09T17:13:19.535 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-base 2024-02-09T17:13:19.632 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-02-09T17:13:19.632 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:13:19.632 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-02-09T17:13:19.632 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:13:19.632 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mds-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:13:19.632 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:13:19.632 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-modules-core-2:17.2.7-499.g89a83b98.el8.noarch 2024-02-09T17:13:19.632 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mon-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:13:19.632 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-osd-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:13:19.632 INFO:teuthology.orchestra.run.smithi179.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-02-09T17:13:19.632 INFO:teuthology.orchestra.run.smithi179.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-02-09T17:13:19.633 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-02-09T17:13:19.633 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-02-09T17:13:19.633 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-02-09T17:13:19.633 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-02-09T17:13:19.633 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-6.2-6.el8.noarch 2024-02-09T17:13:19.633 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-02-09T17:13:19.633 INFO:teuthology.orchestra.run.smithi179.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-02-09T17:13:19.633 INFO:teuthology.orchestra.run.smithi179.stdout: python3-mako-1.0.6-14.el8.noarch 2024-02-09T17:13:19.633 INFO:teuthology.orchestra.run.smithi179.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-02-09T17:13:19.633 INFO:teuthology.orchestra.run.smithi179.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-02-09T17:13:19.633 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-02-09T17:13:19.633 INFO:teuthology.orchestra.run.smithi179.stdout: python3-portend-2.6-1.el8.noarch 2024-02-09T17:13:19.633 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-02-09T17:13:19.633 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pycparser-2.14-14.el8.noarch 2024-02-09T17:13:19.633 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pytz-2017.2-11.el8.noarch 2024-02-09T17:13:19.634 INFO:teuthology.orchestra.run.smithi179.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-02-09T17:13:19.634 INFO:teuthology.orchestra.run.smithi179.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-02-09T17:13:19.634 INFO:teuthology.orchestra.run.smithi179.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-02-09T17:13:19.634 INFO:teuthology.orchestra.run.smithi179.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-02-09T17:13:19.634 INFO:teuthology.orchestra.run.smithi179.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-02-09T17:13:19.634 INFO:teuthology.orchestra.run.smithi179.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-02-09T17:13:19.634 INFO:teuthology.orchestra.run.smithi179.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-02-09T17:13:19.634 INFO:teuthology.orchestra.run.smithi179.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-02-09T17:13:19.634 INFO:teuthology.orchestra.run.smithi179.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-02-09T17:13:19.634 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:13:19.634 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-02-09T17:13:19.659 INFO:teuthology.orchestra.run.smithi162.stdout:Package ceph-base-2:17.2.7-499.g89a83b98.el8.x86_64 is already installed. 2024-02-09T17:13:19.705 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-09T17:13:19.706 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-02-09T17:13:19.706 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-09T17:13:19.834 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install cephadm 2024-02-09T17:13:19.914 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-base 2024-02-09T17:13:20.294 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:22 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:13:20.588 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:22 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:13:20.681 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:22 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:13:21.768 INFO:teuthology.orchestra.run.smithi099.stdout:Package ceph-base-2:17.2.7-499.g89a83b98.el8.x86_64 is already installed. 2024-02-09T17:13:21.815 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-02-09T17:13:21.816 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2024-02-09T17:13:21.816 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-02-09T17:13:21.933 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install cephadm 2024-02-09T17:13:22.099 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-09T17:13:22.099 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-09T17:13:22.099 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-02-09T17:13:22.100 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-09T17:13:22.100 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-02-09T17:13:22.100 INFO:teuthology.orchestra.run.smithi162.stdout: cephadm noarch 2:17.2.7-499.g89a83b98.el8 ceph-noarch 80 k 2024-02-09T17:13:22.100 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:13:22.100 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-02-09T17:13:22.100 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-09T17:13:22.100 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-02-09T17:13:22.100 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:13:22.101 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 80 k 2024-02-09T17:13:22.101 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 364 k 2024-02-09T17:13:22.101 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-02-09T17:13:22.138 INFO:teuthology.orchestra.run.smithi179.stdout:Package ceph-base-2:17.2.7-499.g89a83b98.el8.x86_64 is already installed. 2024-02-09T17:13:22.185 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-02-09T17:13:22.185 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2024-02-09T17:13:22.185 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-02-09T17:13:22.293 INFO:teuthology.orchestra.run.smithi162.stdout:cephadm-17.2.7-499.g89a83b98.el8.noarch.rpm 418 kB/s | 80 kB 00:00 2024-02-09T17:13:22.294 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-02-09T17:13:22.294 INFO:teuthology.orchestra.run.smithi162.stdout:Total 415 kB/s | 80 kB 00:00 2024-02-09T17:13:22.294 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-02-09T17:13:22.297 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install cephadm 2024-02-09T17:13:22.302 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-02-09T17:13:22.302 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-02-09T17:13:22.304 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-02-09T17:13:22.307 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-02-09T17:13:22.346 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-02-09T17:13:22.691 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:24 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:13:22.801 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: cephadm-2:17.2.7-499.g89a83b98.el8.noarch 1/1 2024-02-09T17:13:22.965 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : cephadm-2:17.2.7-499.g89a83b98.el8.noarch 1/1 2024-02-09T17:13:23.051 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:25 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:13:23.195 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: cephadm-2:17.2.7-499.g89a83b98.el8.noarch 1/1 2024-02-09T17:13:23.586 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : cephadm-2:17.2.7-499.g89a83b98.el8.noarch 1/1 2024-02-09T17:13:23.587 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:13:23.587 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-02-09T17:13:23.587 INFO:teuthology.orchestra.run.smithi162.stdout: cephadm-2:17.2.7-499.g89a83b98.el8.noarch 2024-02-09T17:13:23.587 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:13:23.587 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-09T17:13:23.839 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-immutable-object-cache 2024-02-09T17:13:24.204 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-02-09T17:13:24.205 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-02-09T17:13:24.205 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2024-02-09T17:13:24.205 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-02-09T17:13:24.206 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-02-09T17:13:24.206 INFO:teuthology.orchestra.run.smithi099.stdout: cephadm noarch 2:17.2.7-499.g89a83b98.el8 ceph-noarch 80 k 2024-02-09T17:13:24.206 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:13:24.206 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-02-09T17:13:24.206 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-02-09T17:13:24.206 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2024-02-09T17:13:24.206 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:13:24.206 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 80 k 2024-02-09T17:13:24.206 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 364 k 2024-02-09T17:13:24.206 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-02-09T17:13:24.404 INFO:teuthology.orchestra.run.smithi099.stdout:cephadm-17.2.7-499.g89a83b98.el8.noarch.rpm 406 kB/s | 80 kB 00:00 2024-02-09T17:13:24.405 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-02-09T17:13:24.405 INFO:teuthology.orchestra.run.smithi099.stdout:Total 403 kB/s | 80 kB 00:00 2024-02-09T17:13:24.406 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-02-09T17:13:24.413 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-02-09T17:13:24.413 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-02-09T17:13:24.415 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-02-09T17:13:24.418 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-02-09T17:13:24.464 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-02-09T17:13:24.563 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-02-09T17:13:24.564 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-02-09T17:13:24.564 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-02-09T17:13:24.564 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-02-09T17:13:24.564 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-02-09T17:13:24.564 INFO:teuthology.orchestra.run.smithi179.stdout: cephadm noarch 2:17.2.7-499.g89a83b98.el8 ceph-noarch 80 k 2024-02-09T17:13:24.565 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:13:24.565 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-02-09T17:13:24.565 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-02-09T17:13:24.565 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-02-09T17:13:24.565 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:13:24.565 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 80 k 2024-02-09T17:13:24.565 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 364 k 2024-02-09T17:13:24.565 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-02-09T17:13:24.611 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:26 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:13:24.756 INFO:teuthology.orchestra.run.smithi179.stdout:cephadm-17.2.7-499.g89a83b98.el8.noarch.rpm 421 kB/s | 80 kB 00:00 2024-02-09T17:13:24.756 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-02-09T17:13:24.757 INFO:teuthology.orchestra.run.smithi179.stdout:Total 419 kB/s | 80 kB 00:00 2024-02-09T17:13:24.757 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-02-09T17:13:24.764 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-02-09T17:13:24.764 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-02-09T17:13:24.766 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-02-09T17:13:24.769 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-02-09T17:13:24.810 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-02-09T17:13:24.978 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: cephadm-2:17.2.7-499.g89a83b98.el8.noarch 1/1 2024-02-09T17:13:25.153 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : cephadm-2:17.2.7-499.g89a83b98.el8.noarch 1/1 2024-02-09T17:13:25.327 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: cephadm-2:17.2.7-499.g89a83b98.el8.noarch 1/1 2024-02-09T17:13:25.382 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: cephadm-2:17.2.7-499.g89a83b98.el8.noarch 1/1 2024-02-09T17:13:25.491 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : cephadm-2:17.2.7-499.g89a83b98.el8.noarch 1/1 2024-02-09T17:13:25.722 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: cephadm-2:17.2.7-499.g89a83b98.el8.noarch 1/1 2024-02-09T17:13:25.744 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : cephadm-2:17.2.7-499.g89a83b98.el8.noarch 1/1 2024-02-09T17:13:25.745 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:13:25.745 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-02-09T17:13:25.745 INFO:teuthology.orchestra.run.smithi099.stdout: cephadm-2:17.2.7-499.g89a83b98.el8.noarch 2024-02-09T17:13:25.745 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:13:25.745 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-02-09T17:13:25.998 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-immutable-object-cache 2024-02-09T17:13:26.099 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : cephadm-2:17.2.7-499.g89a83b98.el8.noarch 1/1 2024-02-09T17:13:26.099 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:13:26.099 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-02-09T17:13:26.099 INFO:teuthology.orchestra.run.smithi179.stdout: cephadm-2:17.2.7-499.g89a83b98.el8.noarch 2024-02-09T17:13:26.100 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:13:26.100 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-02-09T17:13:26.129 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-09T17:13:26.130 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-09T17:13:26.130 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-02-09T17:13:26.130 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-09T17:13:26.130 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-02-09T17:13:26.130 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-499.g89a83b98.el8 ceph 161 k 2024-02-09T17:13:26.130 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:13:26.131 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-02-09T17:13:26.131 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-09T17:13:26.131 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-02-09T17:13:26.131 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:13:26.131 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 161 k 2024-02-09T17:13:26.131 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 490 k 2024-02-09T17:13:26.131 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-02-09T17:13:26.303 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-immutable-object-cache 2024-02-09T17:13:26.704 INFO:teuthology.orchestra.run.smithi162.stdout:ceph-immutable-object-cache-17.2.7-499.g89a83b9 281 kB/s | 161 kB 00:00 2024-02-09T17:13:26.705 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-02-09T17:13:26.705 INFO:teuthology.orchestra.run.smithi162.stdout:Total 281 kB/s | 161 kB 00:00 2024-02-09T17:13:26.705 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-02-09T17:13:26.718 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-02-09T17:13:26.718 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-02-09T17:13:26.761 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:28 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:13:26.771 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-02-09T17:13:26.775 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-02-09T17:13:27.063 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:29 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:13:27.565 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-02-09T17:13:28.275 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-02-09T17:13:28.275 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-02-09T17:13:28.275 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2024-02-09T17:13:28.275 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-02-09T17:13:28.275 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-02-09T17:13:28.275 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-499.g89a83b98.el8 ceph 161 k 2024-02-09T17:13:28.275 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:13:28.276 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-02-09T17:13:28.276 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-02-09T17:13:28.276 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2024-02-09T17:13:28.276 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:13:28.276 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 161 k 2024-02-09T17:13:28.276 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 490 k 2024-02-09T17:13:28.277 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-02-09T17:13:28.401 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-499.g89a83b98.e 1/1 2024-02-09T17:13:28.588 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-02-09T17:13:28.588 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-02-09T17:13:28.588 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repo Size 2024-02-09T17:13:28.588 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-02-09T17:13:28.588 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-02-09T17:13:28.588 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-499.g89a83b98.el8 ceph 161 k 2024-02-09T17:13:28.589 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:13:28.589 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-02-09T17:13:28.589 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-02-09T17:13:28.589 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-02-09T17:13:28.589 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:13:28.589 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 161 k 2024-02-09T17:13:28.590 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 490 k 2024-02-09T17:13:28.590 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-02-09T17:13:28.726 INFO:teuthology.orchestra.run.smithi099.stdout:ceph-immutable-object-cache-17.2.7-499.g89a83b9 359 kB/s | 161 kB 00:00 2024-02-09T17:13:28.726 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-02-09T17:13:28.727 INFO:teuthology.orchestra.run.smithi099.stdout:Total 358 kB/s | 161 kB 00:00 2024-02-09T17:13:28.727 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-02-09T17:13:28.738 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-02-09T17:13:28.738 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-02-09T17:13:28.787 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-02-09T17:13:28.790 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-02-09T17:13:28.794 INFO:teuthology.orchestra.run.smithi179.stdout:ceph-immutable-object-cache-17.2.7-499.g89a83b9 786 kB/s | 161 kB 00:00 2024-02-09T17:13:28.795 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-02-09T17:13:28.795 INFO:teuthology.orchestra.run.smithi179.stdout:Total 782 kB/s | 161 kB 00:00 2024-02-09T17:13:28.796 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-02-09T17:13:28.806 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-02-09T17:13:28.806 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-02-09T17:13:28.831 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-499.g89a83b98.e 1/1 2024-02-09T17:13:28.853 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-02-09T17:13:28.857 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-02-09T17:13:28.902 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-02-09T17:13:28.965 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-02-09T17:13:29.114 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-499.g89a83b98.e 1/1 2024-02-09T17:13:29.221 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-499.g89a83b98.e 1/1 2024-02-09T17:13:29.221 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:13:29.221 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-02-09T17:13:29.221 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-immutable-object-cache-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:13:29.222 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:13:29.222 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-09T17:13:29.233 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-499.g89a83b98.e 1/1 2024-02-09T17:13:29.425 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-mgr 2024-02-09T17:13:29.525 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-499.g89a83b98.e 1/1 2024-02-09T17:13:29.656 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-499.g89a83b98.e 1/1 2024-02-09T17:13:29.899 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-499.g89a83b98.e 1/1 2024-02-09T17:13:29.900 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:13:29.900 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-02-09T17:13:29.900 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-immutable-object-cache-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:13:29.900 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:13:29.900 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-02-09T17:13:30.066 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-499.g89a83b98.e 1/1 2024-02-09T17:13:30.066 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:13:30.067 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-02-09T17:13:30.067 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-immutable-object-cache-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:13:30.067 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:13:30.067 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-02-09T17:13:30.157 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-mgr 2024-02-09T17:13:30.197 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:32 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:13:30.286 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-mgr 2024-02-09T17:13:30.926 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:32 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:13:31.052 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:33 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:13:31.667 INFO:teuthology.orchestra.run.smithi162.stdout:Package ceph-mgr-2:17.2.7-499.g89a83b98.el8.x86_64 is already installed. 2024-02-09T17:13:31.713 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-09T17:13:31.715 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-02-09T17:13:31.715 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-09T17:13:31.821 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-fuse 2024-02-09T17:13:32.375 INFO:teuthology.orchestra.run.smithi099.stdout:Package ceph-mgr-2:17.2.7-499.g89a83b98.el8.x86_64 is already installed. 2024-02-09T17:13:32.421 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-02-09T17:13:32.422 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2024-02-09T17:13:32.422 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-02-09T17:13:32.504 INFO:teuthology.orchestra.run.smithi179.stdout:Package ceph-mgr-2:17.2.7-499.g89a83b98.el8.x86_64 is already installed. 2024-02-09T17:13:32.525 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-fuse 2024-02-09T17:13:32.550 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-02-09T17:13:32.551 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2024-02-09T17:13:32.551 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-02-09T17:13:32.593 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:34 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:13:32.647 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-fuse 2024-02-09T17:13:33.283 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:35 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:13:33.409 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:35 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:13:34.102 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-09T17:13:34.103 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-09T17:13:34.103 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-02-09T17:13:34.103 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-09T17:13:34.103 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-02-09T17:13:34.103 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-fuse x86_64 2:17.2.7-499.g89a83b98.el8 ceph 876 k 2024-02-09T17:13:34.103 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-02-09T17:13:34.103 INFO:teuthology.orchestra.run.smithi162.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-02-09T17:13:34.104 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:13:34.104 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-02-09T17:13:34.104 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-09T17:13:34.104 INFO:teuthology.orchestra.run.smithi162.stdout:Install 2 Packages 2024-02-09T17:13:34.104 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:13:34.104 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 959 k 2024-02-09T17:13:34.104 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 3.0 M 2024-02-09T17:13:34.105 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-02-09T17:13:34.352 INFO:teuthology.orchestra.run.smithi162.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 403 kB/s | 83 kB 00:00 2024-02-09T17:13:34.527 INFO:teuthology.orchestra.run.smithi162.stdout:(2/2): ceph-fuse-17.2.7-499.g89a83b98.el8.x86_6 2.2 MB/s | 876 kB 00:00 2024-02-09T17:13:34.528 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-02-09T17:13:34.528 INFO:teuthology.orchestra.run.smithi162.stdout:Total 2.2 MB/s | 959 kB 00:00 2024-02-09T17:13:34.538 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-02-09T17:13:34.558 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-02-09T17:13:34.558 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-02-09T17:13:34.642 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-02-09T17:13:34.645 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-02-09T17:13:34.794 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-02-09T17:13:34.795 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-02-09T17:13:34.795 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2024-02-09T17:13:34.795 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-02-09T17:13:34.795 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-02-09T17:13:34.795 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-fuse x86_64 2:17.2.7-499.g89a83b98.el8 ceph 876 k 2024-02-09T17:13:34.795 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2024-02-09T17:13:34.795 INFO:teuthology.orchestra.run.smithi099.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-02-09T17:13:34.795 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:13:34.795 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-02-09T17:13:34.795 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-02-09T17:13:34.795 INFO:teuthology.orchestra.run.smithi099.stdout:Install 2 Packages 2024-02-09T17:13:34.796 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:13:34.796 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 959 k 2024-02-09T17:13:34.796 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 3.0 M 2024-02-09T17:13:34.796 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-02-09T17:13:34.814 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-02-09T17:13:34.928 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-02-09T17:13:34.928 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-02-09T17:13:34.928 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-02-09T17:13:34.928 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-02-09T17:13:34.929 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-02-09T17:13:34.929 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-fuse x86_64 2:17.2.7-499.g89a83b98.el8 ceph 876 k 2024-02-09T17:13:34.929 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-02-09T17:13:34.929 INFO:teuthology.orchestra.run.smithi179.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-02-09T17:13:34.929 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:13:34.929 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-02-09T17:13:34.929 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-02-09T17:13:34.929 INFO:teuthology.orchestra.run.smithi179.stdout:Install 2 Packages 2024-02-09T17:13:34.929 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:13:34.930 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 959 k 2024-02-09T17:13:34.930 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 3.0 M 2024-02-09T17:13:34.930 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-02-09T17:13:35.100 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-02-09T17:13:35.187 INFO:teuthology.orchestra.run.smithi099.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 406 kB/s | 83 kB 00:00 2024-02-09T17:13:35.299 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-fuse-2:17.2.7-499.g89a83b98.el8.x86_64 2/2 2024-02-09T17:13:35.306 INFO:teuthology.orchestra.run.smithi179.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 436 kB/s | 83 kB 00:00 2024-02-09T17:13:35.353 INFO:teuthology.orchestra.run.smithi099.stdout:(2/2): ceph-fuse-17.2.7-499.g89a83b98.el8.x86_6 2.3 MB/s | 876 kB 00:00 2024-02-09T17:13:35.354 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-02-09T17:13:35.354 INFO:teuthology.orchestra.run.smithi099.stdout:Total 1.7 MB/s | 959 kB 00:00 2024-02-09T17:13:35.364 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-02-09T17:13:35.385 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-02-09T17:13:35.385 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-02-09T17:13:35.470 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-02-09T17:13:35.473 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-02-09T17:13:35.516 INFO:teuthology.orchestra.run.smithi179.stdout:(2/2): ceph-fuse-17.2.7-499.g89a83b98.el8.x86_6 2.1 MB/s | 876 kB 00:00 2024-02-09T17:13:35.517 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-02-09T17:13:35.517 INFO:teuthology.orchestra.run.smithi179.stdout:Total 1.6 MB/s | 959 kB 00:00 2024-02-09T17:13:35.527 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-02-09T17:13:35.547 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-02-09T17:13:35.547 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-02-09T17:13:35.630 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-02-09T17:13:35.633 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-02-09T17:13:35.639 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-02-09T17:13:35.689 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-fuse-2:17.2.7-499.g89a83b98.el8.x86_64 2/2 2024-02-09T17:13:35.690 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-fuse-2:17.2.7-499.g89a83b98.el8.x86_64 1/2 2024-02-09T17:13:35.802 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-02-09T17:13:35.946 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-02-09T17:13:36.073 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-02-09T17:13:36.076 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-02-09T17:13:36.076 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:13:36.077 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-02-09T17:13:36.077 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-fuse-2:17.2.7-499.g89a83b98.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-02-09T17:13:36.077 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:13:36.077 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-09T17:13:36.136 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-fuse-2:17.2.7-499.g89a83b98.el8.x86_64 2/2 2024-02-09T17:13:36.273 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-fuse-2:17.2.7-499.g89a83b98.el8.x86_64 2/2 2024-02-09T17:13:36.344 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install librados-devel 2024-02-09T17:13:36.537 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-fuse-2:17.2.7-499.g89a83b98.el8.x86_64 2/2 2024-02-09T17:13:36.538 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-fuse-2:17.2.7-499.g89a83b98.el8.x86_64 1/2 2024-02-09T17:13:36.661 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-fuse-2:17.2.7-499.g89a83b98.el8.x86_64 2/2 2024-02-09T17:13:36.661 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-fuse-2:17.2.7-499.g89a83b98.el8.x86_64 1/2 2024-02-09T17:13:36.899 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-02-09T17:13:36.899 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:13:36.899 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-02-09T17:13:36.899 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-fuse-2:17.2.7-499.g89a83b98.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-02-09T17:13:36.899 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:13:36.899 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-02-09T17:13:37.033 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-02-09T17:13:37.033 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:13:37.033 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-02-09T17:13:37.033 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-fuse-2:17.2.7-499.g89a83b98.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-02-09T17:13:37.033 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:13:37.033 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-02-09T17:13:37.107 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:39 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:13:37.130 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install librados-devel 2024-02-09T17:13:37.247 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install librados-devel 2024-02-09T17:13:37.892 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:39 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:13:38.013 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:40 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:13:38.629 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-09T17:13:38.629 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-09T17:13:38.629 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-02-09T17:13:38.630 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-09T17:13:38.630 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-02-09T17:13:38.630 INFO:teuthology.orchestra.run.smithi162.stdout: librados-devel x86_64 2:17.2.7-499.g89a83b98.el8 ceph 130 k 2024-02-09T17:13:38.630 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:13:38.630 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-02-09T17:13:38.630 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-09T17:13:38.630 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-02-09T17:13:38.630 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:13:38.631 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 130 k 2024-02-09T17:13:38.631 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 464 k 2024-02-09T17:13:38.631 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-02-09T17:13:38.841 INFO:teuthology.orchestra.run.smithi162.stdout:librados-devel-17.2.7-499.g89a83b98.el8.x86_64. 617 kB/s | 130 kB 00:00 2024-02-09T17:13:38.842 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-02-09T17:13:38.842 INFO:teuthology.orchestra.run.smithi162.stdout:Total 613 kB/s | 130 kB 00:00 2024-02-09T17:13:38.842 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-02-09T17:13:38.848 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-02-09T17:13:38.848 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-02-09T17:13:38.894 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-02-09T17:13:38.897 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-02-09T17:13:39.005 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-02-09T17:13:39.205 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librados-devel-2:17.2.7-499.g89a83b98.el8.x86_64 1/1 2024-02-09T17:13:39.403 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-02-09T17:13:39.403 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-02-09T17:13:39.403 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2024-02-09T17:13:39.403 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-02-09T17:13:39.403 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-02-09T17:13:39.404 INFO:teuthology.orchestra.run.smithi099.stdout: librados-devel x86_64 2:17.2.7-499.g89a83b98.el8 ceph 130 k 2024-02-09T17:13:39.404 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:13:39.404 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-02-09T17:13:39.404 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-02-09T17:13:39.404 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2024-02-09T17:13:39.404 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:13:39.405 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 130 k 2024-02-09T17:13:39.405 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 464 k 2024-02-09T17:13:39.405 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-02-09T17:13:39.497 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librados-devel-2:17.2.7-499.g89a83b98.el8.x86_64 1/1 2024-02-09T17:13:39.530 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-02-09T17:13:39.531 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-02-09T17:13:39.531 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repo Size 2024-02-09T17:13:39.531 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-02-09T17:13:39.531 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-02-09T17:13:39.531 INFO:teuthology.orchestra.run.smithi179.stdout: librados-devel x86_64 2:17.2.7-499.g89a83b98.el8 ceph 130 k 2024-02-09T17:13:39.531 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:13:39.531 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-02-09T17:13:39.531 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-02-09T17:13:39.531 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-02-09T17:13:39.531 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:13:39.532 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 130 k 2024-02-09T17:13:39.532 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 464 k 2024-02-09T17:13:39.532 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-02-09T17:13:39.615 INFO:teuthology.orchestra.run.smithi099.stdout:librados-devel-17.2.7-499.g89a83b98.el8.x86_64. 618 kB/s | 130 kB 00:00 2024-02-09T17:13:39.616 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-02-09T17:13:39.616 INFO:teuthology.orchestra.run.smithi099.stdout:Total 613 kB/s | 130 kB 00:00 2024-02-09T17:13:39.616 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-02-09T17:13:39.621 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-02-09T17:13:39.621 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-02-09T17:13:39.668 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-02-09T17:13:39.671 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-02-09T17:13:39.736 INFO:teuthology.orchestra.run.smithi179.stdout:librados-devel-17.2.7-499.g89a83b98.el8.x86_64. 637 kB/s | 130 kB 00:00 2024-02-09T17:13:39.736 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-02-09T17:13:39.737 INFO:teuthology.orchestra.run.smithi179.stdout:Total 633 kB/s | 130 kB 00:00 2024-02-09T17:13:39.737 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-02-09T17:13:39.742 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-02-09T17:13:39.742 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-02-09T17:13:39.790 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-02-09T17:13:39.793 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-02-09T17:13:39.797 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-02-09T17:13:39.867 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados-devel-2:17.2.7-499.g89a83b98.el8.x86_64 1/1 2024-02-09T17:13:39.867 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:13:39.867 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-02-09T17:13:39.867 INFO:teuthology.orchestra.run.smithi162.stdout: librados-devel-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:13:39.868 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:13:39.868 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-09T17:13:39.911 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-02-09T17:13:39.976 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : librados-devel-2:17.2.7-499.g89a83b98.el8.x86_64 1/1 2024-02-09T17:13:40.096 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install libcephfs2 2024-02-09T17:13:40.129 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : librados-devel-2:17.2.7-499.g89a83b98.el8.x86_64 1/1 2024-02-09T17:13:40.266 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librados-devel-2:17.2.7-499.g89a83b98.el8.x86_64 1/1 2024-02-09T17:13:40.431 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librados-devel-2:17.2.7-499.g89a83b98.el8.x86_64 1/1 2024-02-09T17:13:40.639 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librados-devel-2:17.2.7-499.g89a83b98.el8.x86_64 1/1 2024-02-09T17:13:40.639 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:13:40.639 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-02-09T17:13:40.639 INFO:teuthology.orchestra.run.smithi099.stdout: librados-devel-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:13:40.639 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:13:40.639 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-02-09T17:13:40.798 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librados-devel-2:17.2.7-499.g89a83b98.el8.x86_64 1/1 2024-02-09T17:13:40.798 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:13:40.799 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-02-09T17:13:40.799 INFO:teuthology.orchestra.run.smithi179.stdout: librados-devel-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:13:40.799 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:13:40.799 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-02-09T17:13:40.832 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install libcephfs2 2024-02-09T17:13:40.866 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:42 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:13:41.041 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install libcephfs2 2024-02-09T17:13:41.596 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:43 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:13:41.807 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:43 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:13:42.317 INFO:teuthology.orchestra.run.smithi162.stdout:Package libcephfs2-2:17.2.7-499.g89a83b98.el8.x86_64 is already installed. 2024-02-09T17:13:42.363 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-09T17:13:42.365 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-02-09T17:13:42.365 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-09T17:13:42.477 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install libcephfs-devel 2024-02-09T17:13:43.049 INFO:teuthology.orchestra.run.smithi099.stdout:Package libcephfs2-2:17.2.7-499.g89a83b98.el8.x86_64 is already installed. 2024-02-09T17:13:43.095 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-02-09T17:13:43.097 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2024-02-09T17:13:43.097 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-02-09T17:13:43.189 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install libcephfs-devel 2024-02-09T17:13:43.231 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:45 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:13:43.278 INFO:teuthology.orchestra.run.smithi179.stdout:Package libcephfs2-2:17.2.7-499.g89a83b98.el8.x86_64 is already installed. 2024-02-09T17:13:43.325 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-02-09T17:13:43.326 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2024-02-09T17:13:43.326 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-02-09T17:13:43.423 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install libcephfs-devel 2024-02-09T17:13:43.935 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:45 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:13:44.177 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:46 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:13:44.738 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-09T17:13:44.739 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-09T17:13:44.739 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-02-09T17:13:44.739 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-09T17:13:44.739 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-02-09T17:13:44.739 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs-devel x86_64 2:17.2.7-499.g89a83b98.el8 ceph 25 k 2024-02-09T17:13:44.739 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:13:44.739 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-02-09T17:13:44.739 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-09T17:13:44.739 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-02-09T17:13:44.740 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:13:44.740 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 25 k 2024-02-09T17:13:44.740 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 109 k 2024-02-09T17:13:44.740 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-02-09T17:13:44.905 INFO:teuthology.orchestra.run.smithi162.stdout:libcephfs-devel-17.2.7-499.g89a83b98.el8.x86_64 150 kB/s | 25 kB 00:00 2024-02-09T17:13:44.906 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-02-09T17:13:44.906 INFO:teuthology.orchestra.run.smithi162.stdout:Total 149 kB/s | 25 kB 00:00 2024-02-09T17:13:44.906 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-02-09T17:13:44.910 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-02-09T17:13:44.910 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-02-09T17:13:44.912 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-02-09T17:13:44.914 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-02-09T17:13:44.965 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-02-09T17:13:45.129 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephfs-devel-2:17.2.7-499.g89a83b98.el8.x86_64 1/1 2024-02-09T17:13:45.375 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-499.g89a83b98.el8.x86_64 1/1 2024-02-09T17:13:45.448 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-02-09T17:13:45.449 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-02-09T17:13:45.449 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2024-02-09T17:13:45.449 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-02-09T17:13:45.449 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-02-09T17:13:45.449 INFO:teuthology.orchestra.run.smithi099.stdout: libcephfs-devel x86_64 2:17.2.7-499.g89a83b98.el8 ceph 25 k 2024-02-09T17:13:45.449 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:13:45.449 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-02-09T17:13:45.449 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-02-09T17:13:45.449 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2024-02-09T17:13:45.450 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:13:45.450 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 25 k 2024-02-09T17:13:45.450 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 109 k 2024-02-09T17:13:45.450 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-02-09T17:13:45.628 INFO:teuthology.orchestra.run.smithi099.stdout:libcephfs-devel-17.2.7-499.g89a83b98.el8.x86_64 140 kB/s | 25 kB 00:00 2024-02-09T17:13:45.629 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-02-09T17:13:45.629 INFO:teuthology.orchestra.run.smithi099.stdout:Total 139 kB/s | 25 kB 00:00 2024-02-09T17:13:45.629 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-02-09T17:13:45.632 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-02-09T17:13:45.632 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-02-09T17:13:45.634 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-02-09T17:13:45.637 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-02-09T17:13:45.683 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-02-09T17:13:45.697 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-02-09T17:13:45.697 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-02-09T17:13:45.697 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repo Size 2024-02-09T17:13:45.698 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-02-09T17:13:45.698 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-02-09T17:13:45.698 INFO:teuthology.orchestra.run.smithi179.stdout: libcephfs-devel x86_64 2:17.2.7-499.g89a83b98.el8 ceph 25 k 2024-02-09T17:13:45.698 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:13:45.698 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-02-09T17:13:45.698 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-02-09T17:13:45.698 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-02-09T17:13:45.698 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:13:45.699 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 25 k 2024-02-09T17:13:45.699 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 109 k 2024-02-09T17:13:45.699 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-02-09T17:13:45.721 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephfs-devel-2:17.2.7-499.g89a83b98.el8.x86_64 1/1 2024-02-09T17:13:45.721 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:13:45.721 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-02-09T17:13:45.721 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs-devel-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:13:45.721 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:13:45.721 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-09T17:13:45.836 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install librados2 2024-02-09T17:13:45.866 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libcephfs-devel-2:17.2.7-499.g89a83b98.el8.x86_64 1/1 2024-02-09T17:13:45.872 INFO:teuthology.orchestra.run.smithi179.stdout:libcephfs-devel-17.2.7-499.g89a83b98.el8.x86_64 143 kB/s | 25 kB 00:00 2024-02-09T17:13:45.873 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-02-09T17:13:45.873 INFO:teuthology.orchestra.run.smithi179.stdout:Total 142 kB/s | 25 kB 00:00 2024-02-09T17:13:45.874 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-02-09T17:13:45.877 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-02-09T17:13:45.877 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-02-09T17:13:45.879 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-02-09T17:13:45.882 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-02-09T17:13:45.947 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-02-09T17:13:46.110 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-499.g89a83b98.el8.x86_64 1/1 2024-02-09T17:13:46.112 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libcephfs-devel-2:17.2.7-499.g89a83b98.el8.x86_64 1/1 2024-02-09T17:13:46.364 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-499.g89a83b98.el8.x86_64 1/1 2024-02-09T17:13:46.453 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libcephfs-devel-2:17.2.7-499.g89a83b98.el8.x86_64 1/1 2024-02-09T17:13:46.453 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:13:46.453 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-02-09T17:13:46.454 INFO:teuthology.orchestra.run.smithi099.stdout: libcephfs-devel-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:13:46.454 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:13:46.454 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-02-09T17:13:46.567 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install librados2 2024-02-09T17:13:46.583 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:48 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:13:46.696 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libcephfs-devel-2:17.2.7-499.g89a83b98.el8.x86_64 1/1 2024-02-09T17:13:46.696 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:13:46.696 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-02-09T17:13:46.696 INFO:teuthology.orchestra.run.smithi179.stdout: libcephfs-devel-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:13:46.696 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:13:46.697 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-02-09T17:13:46.802 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install librados2 2024-02-09T17:13:47.311 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:49 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:13:47.551 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:49 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:13:48.047 INFO:teuthology.orchestra.run.smithi162.stdout:Package librados2-2:17.2.7-499.g89a83b98.el8.x86_64 is already installed. 2024-02-09T17:13:48.093 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-09T17:13:48.095 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-02-09T17:13:48.095 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-09T17:13:48.194 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install librbd1 2024-02-09T17:13:48.781 INFO:teuthology.orchestra.run.smithi099.stdout:Package librados2-2:17.2.7-499.g89a83b98.el8.x86_64 is already installed. 2024-02-09T17:13:48.827 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-02-09T17:13:48.829 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2024-02-09T17:13:48.829 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-02-09T17:13:48.939 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install librbd1 2024-02-09T17:13:48.945 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:50 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:13:49.014 INFO:teuthology.orchestra.run.smithi179.stdout:Package librados2-2:17.2.7-499.g89a83b98.el8.x86_64 is already installed. 2024-02-09T17:13:49.060 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-02-09T17:13:49.061 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2024-02-09T17:13:49.061 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-02-09T17:13:49.168 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install librbd1 2024-02-09T17:13:49.682 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:51 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:13:49.918 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:51 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:13:50.412 INFO:teuthology.orchestra.run.smithi162.stdout:Package librbd1-2:17.2.7-499.g89a83b98.el8.x86_64 is already installed. 2024-02-09T17:13:50.458 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-09T17:13:50.459 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-02-09T17:13:50.459 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-09T17:13:50.565 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install python3-rados 2024-02-09T17:13:51.137 INFO:teuthology.orchestra.run.smithi099.stdout:Package librbd1-2:17.2.7-499.g89a83b98.el8.x86_64 is already installed. 2024-02-09T17:13:51.183 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-02-09T17:13:51.185 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2024-02-09T17:13:51.185 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-02-09T17:13:51.293 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install python3-rados 2024-02-09T17:13:51.316 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:53 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:13:51.375 INFO:teuthology.orchestra.run.smithi179.stdout:Package librbd1-2:17.2.7-499.g89a83b98.el8.x86_64 is already installed. 2024-02-09T17:13:51.422 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-02-09T17:13:51.424 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2024-02-09T17:13:51.424 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-02-09T17:13:51.526 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install python3-rados 2024-02-09T17:13:52.044 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:54 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:13:52.274 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:54 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:13:52.789 INFO:teuthology.orchestra.run.smithi162.stdout:Package python3-rados-2:17.2.7-499.g89a83b98.el8.x86_64 is already installed. 2024-02-09T17:13:52.836 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-09T17:13:52.837 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-02-09T17:13:52.837 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-09T17:13:52.943 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install python3-rgw 2024-02-09T17:13:53.524 INFO:teuthology.orchestra.run.smithi099.stdout:Package python3-rados-2:17.2.7-499.g89a83b98.el8.x86_64 is already installed. 2024-02-09T17:13:53.570 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-02-09T17:13:53.571 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2024-02-09T17:13:53.572 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-02-09T17:13:53.677 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install python3-rgw 2024-02-09T17:13:53.699 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:55 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:13:53.726 INFO:teuthology.orchestra.run.smithi179.stdout:Package python3-rados-2:17.2.7-499.g89a83b98.el8.x86_64 is already installed. 2024-02-09T17:13:53.771 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-02-09T17:13:53.772 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2024-02-09T17:13:53.773 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-02-09T17:13:53.869 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install python3-rgw 2024-02-09T17:13:54.422 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:56 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:13:54.618 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:56 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:13:55.175 INFO:teuthology.orchestra.run.smithi162.stdout:Package python3-rgw-2:17.2.7-499.g89a83b98.el8.x86_64 is already installed. 2024-02-09T17:13:55.221 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-09T17:13:55.223 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-02-09T17:13:55.223 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-09T17:13:55.337 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install python3-cephfs 2024-02-09T17:13:55.890 INFO:teuthology.orchestra.run.smithi099.stdout:Package python3-rgw-2:17.2.7-499.g89a83b98.el8.x86_64 is already installed. 2024-02-09T17:13:55.936 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-02-09T17:13:55.937 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2024-02-09T17:13:55.938 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-02-09T17:13:56.042 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install python3-cephfs 2024-02-09T17:13:56.085 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:58 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:13:56.088 INFO:teuthology.orchestra.run.smithi179.stdout:Package python3-rgw-2:17.2.7-499.g89a83b98.el8.x86_64 is already installed. 2024-02-09T17:13:56.134 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-02-09T17:13:56.135 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2024-02-09T17:13:56.136 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-02-09T17:13:56.240 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install python3-cephfs 2024-02-09T17:13:56.791 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:58 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:13:56.985 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:58 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:13:57.588 INFO:teuthology.orchestra.run.smithi162.stdout:Package python3-cephfs-2:17.2.7-499.g89a83b98.el8.x86_64 is already installed. 2024-02-09T17:13:57.634 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-09T17:13:57.636 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-02-09T17:13:57.636 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-09T17:13:57.748 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install python3-rbd 2024-02-09T17:13:58.262 INFO:teuthology.orchestra.run.smithi099.stdout:Package python3-cephfs-2:17.2.7-499.g89a83b98.el8.x86_64 is already installed. 2024-02-09T17:13:58.308 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-02-09T17:13:58.309 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2024-02-09T17:13:58.310 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-02-09T17:13:58.425 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install python3-rbd 2024-02-09T17:13:58.452 INFO:teuthology.orchestra.run.smithi179.stdout:Package python3-cephfs-2:17.2.7-499.g89a83b98.el8.x86_64 is already installed. 2024-02-09T17:13:58.499 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-02-09T17:13:58.500 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2024-02-09T17:13:58.500 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-02-09T17:13:58.504 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:00 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:13:58.617 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install python3-rbd 2024-02-09T17:13:59.191 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:02:01 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:13:59.369 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:02:01 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:13:59.969 INFO:teuthology.orchestra.run.smithi162.stdout:Package python3-rbd-2:17.2.7-499.g89a83b98.el8.x86_64 is already installed. 2024-02-09T17:14:00.016 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-09T17:14:00.017 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-02-09T17:14:00.017 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-09T17:14:00.122 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install rbd-fuse 2024-02-09T17:14:00.636 INFO:teuthology.orchestra.run.smithi099.stdout:Package python3-rbd-2:17.2.7-499.g89a83b98.el8.x86_64 is already installed. 2024-02-09T17:14:00.681 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-02-09T17:14:00.683 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2024-02-09T17:14:00.683 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-02-09T17:14:00.790 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install rbd-fuse 2024-02-09T17:14:00.827 INFO:teuthology.orchestra.run.smithi179.stdout:Package python3-rbd-2:17.2.7-499.g89a83b98.el8.x86_64 is already installed. 2024-02-09T17:14:00.872 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:02 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:14:00.875 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-02-09T17:14:00.876 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2024-02-09T17:14:00.876 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-02-09T17:14:00.982 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install rbd-fuse 2024-02-09T17:14:01.535 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:02:03 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:14:01.731 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:02:03 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:14:02.391 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-09T17:14:02.392 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-09T17:14:02.392 INFO:teuthology.orchestra.run.smithi162.stdout: Package Architecture Version Repository Size 2024-02-09T17:14:02.392 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-09T17:14:02.392 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-02-09T17:14:02.392 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-fuse x86_64 2:17.2.7-499.g89a83b98.el8 ceph 90 k 2024-02-09T17:14:02.392 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:14:02.392 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-02-09T17:14:02.393 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-09T17:14:02.393 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-02-09T17:14:02.393 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:14:02.393 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 90 k 2024-02-09T17:14:02.394 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 264 k 2024-02-09T17:14:02.394 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-02-09T17:14:02.701 INFO:teuthology.orchestra.run.smithi162.stdout:rbd-fuse-17.2.7-499.g89a83b98.el8.x86_64.rpm 292 kB/s | 90 kB 00:00 2024-02-09T17:14:02.702 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-02-09T17:14:02.702 INFO:teuthology.orchestra.run.smithi162.stdout:Total 291 kB/s | 90 kB 00:00 2024-02-09T17:14:02.702 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-02-09T17:14:02.712 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-02-09T17:14:02.712 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-02-09T17:14:02.759 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-02-09T17:14:02.762 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-02-09T17:14:02.861 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-02-09T17:14:03.039 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-02-09T17:14:03.040 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-02-09T17:14:03.040 INFO:teuthology.orchestra.run.smithi099.stdout: Package Architecture Version Repository Size 2024-02-09T17:14:03.040 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-02-09T17:14:03.040 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-02-09T17:14:03.040 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-fuse x86_64 2:17.2.7-499.g89a83b98.el8 ceph 90 k 2024-02-09T17:14:03.040 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:14:03.040 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-02-09T17:14:03.040 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-02-09T17:14:03.040 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2024-02-09T17:14:03.040 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:14:03.041 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 90 k 2024-02-09T17:14:03.041 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 264 k 2024-02-09T17:14:03.041 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-02-09T17:14:03.058 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-fuse-2:17.2.7-499.g89a83b98.el8.x86_64 1/1 2024-02-09T17:14:03.220 INFO:teuthology.orchestra.run.smithi099.stdout:rbd-fuse-17.2.7-499.g89a83b98.el8.x86_64.rpm 506 kB/s | 90 kB 00:00 2024-02-09T17:14:03.220 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-02-09T17:14:03.220 INFO:teuthology.orchestra.run.smithi099.stdout:Total 502 kB/s | 90 kB 00:00 2024-02-09T17:14:03.221 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-02-09T17:14:03.230 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-02-09T17:14:03.230 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-02-09T17:14:03.231 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-02-09T17:14:03.232 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-02-09T17:14:03.232 INFO:teuthology.orchestra.run.smithi179.stdout: Package Architecture Version Repository Size 2024-02-09T17:14:03.232 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-02-09T17:14:03.232 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-02-09T17:14:03.232 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-fuse x86_64 2:17.2.7-499.g89a83b98.el8 ceph 90 k 2024-02-09T17:14:03.232 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:14:03.233 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-02-09T17:14:03.233 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-02-09T17:14:03.233 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-02-09T17:14:03.233 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:14:03.233 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 90 k 2024-02-09T17:14:03.233 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 264 k 2024-02-09T17:14:03.233 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-02-09T17:14:03.277 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-02-09T17:14:03.280 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-02-09T17:14:03.348 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: rbd-fuse-2:17.2.7-499.g89a83b98.el8.x86_64 1/1 2024-02-09T17:14:03.378 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-02-09T17:14:03.451 INFO:teuthology.orchestra.run.smithi179.stdout:rbd-fuse-17.2.7-499.g89a83b98.el8.x86_64.rpm 414 kB/s | 90 kB 00:00 2024-02-09T17:14:03.452 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-02-09T17:14:03.452 INFO:teuthology.orchestra.run.smithi179.stdout:Total 412 kB/s | 90 kB 00:00 2024-02-09T17:14:03.452 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-02-09T17:14:03.462 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-02-09T17:14:03.462 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-02-09T17:14:03.508 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-02-09T17:14:03.511 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-02-09T17:14:03.604 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : rbd-fuse-2:17.2.7-499.g89a83b98.el8.x86_64 1/1 2024-02-09T17:14:03.610 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-02-09T17:14:03.737 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-fuse-2:17.2.7-499.g89a83b98.el8.x86_64 1/1 2024-02-09T17:14:03.737 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:14:03.737 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-02-09T17:14:03.737 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-fuse-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:14:03.737 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:14:03.737 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-09T17:14:03.834 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : rbd-fuse-2:17.2.7-499.g89a83b98.el8.x86_64 1/1 2024-02-09T17:14:03.886 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: rbd-fuse-2:17.2.7-499.g89a83b98.el8.x86_64 1/1 2024-02-09T17:14:03.986 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install rbd-mirror 2024-02-09T17:14:04.122 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: rbd-fuse-2:17.2.7-499.g89a83b98.el8.x86_64 1/1 2024-02-09T17:14:04.293 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : rbd-fuse-2:17.2.7-499.g89a83b98.el8.x86_64 1/1 2024-02-09T17:14:04.293 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:14:04.294 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-02-09T17:14:04.294 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-fuse-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:14:04.294 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:14:04.294 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-02-09T17:14:04.477 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : rbd-fuse-2:17.2.7-499.g89a83b98.el8.x86_64 1/1 2024-02-09T17:14:04.477 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:14:04.477 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-02-09T17:14:04.477 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-fuse-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:14:04.477 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:14:04.477 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-02-09T17:14:04.516 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install rbd-mirror 2024-02-09T17:14:04.687 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install rbd-mirror 2024-02-09T17:14:04.748 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:06 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:14:05.274 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:02:07 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:14:05.453 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:02:07 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:14:06.258 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-09T17:14:06.258 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-09T17:14:06.259 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-02-09T17:14:06.259 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-09T17:14:06.259 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-02-09T17:14:06.259 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-mirror x86_64 2:17.2.7-499.g89a83b98.el8 ceph 3.5 M 2024-02-09T17:14:06.259 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:14:06.259 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-02-09T17:14:06.259 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-09T17:14:06.259 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-02-09T17:14:06.259 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:14:06.260 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 3.5 M 2024-02-09T17:14:06.260 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 16 M 2024-02-09T17:14:06.260 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-02-09T17:14:06.844 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-02-09T17:14:06.845 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-02-09T17:14:06.845 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2024-02-09T17:14:06.845 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-02-09T17:14:06.845 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-02-09T17:14:06.845 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-mirror x86_64 2:17.2.7-499.g89a83b98.el8 ceph 3.5 M 2024-02-09T17:14:06.845 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:14:06.845 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-02-09T17:14:06.845 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-02-09T17:14:06.845 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2024-02-09T17:14:06.845 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:14:06.845 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 3.5 M 2024-02-09T17:14:06.845 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 16 M 2024-02-09T17:14:06.846 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-02-09T17:14:06.946 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-02-09T17:14:06.946 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-02-09T17:14:06.947 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repo Size 2024-02-09T17:14:06.947 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-02-09T17:14:06.947 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-02-09T17:14:06.947 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-mirror x86_64 2:17.2.7-499.g89a83b98.el8 ceph 3.5 M 2024-02-09T17:14:06.947 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:14:06.947 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-02-09T17:14:06.947 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-02-09T17:14:06.947 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-02-09T17:14:06.947 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:14:06.947 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 3.5 M 2024-02-09T17:14:06.948 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 16 M 2024-02-09T17:14:06.948 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-02-09T17:14:07.001 INFO:teuthology.orchestra.run.smithi162.stdout:rbd-mirror-17.2.7-499.g89a83b98.el8.x86_64.rpm 4.7 MB/s | 3.5 MB 00:00 2024-02-09T17:14:07.001 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-02-09T17:14:07.002 INFO:teuthology.orchestra.run.smithi162.stdout:Total 4.7 MB/s | 3.5 MB 00:00 2024-02-09T17:14:07.002 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-02-09T17:14:07.013 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-02-09T17:14:07.014 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-02-09T17:14:07.073 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-02-09T17:14:07.076 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-02-09T17:14:07.496 INFO:teuthology.orchestra.run.smithi099.stdout:rbd-mirror-17.2.7-499.g89a83b98.el8.x86_64.rpm 5.3 MB/s | 3.5 MB 00:00 2024-02-09T17:14:07.497 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-02-09T17:14:07.497 INFO:teuthology.orchestra.run.smithi099.stdout:Total 5.3 MB/s | 3.5 MB 00:00 2024-02-09T17:14:07.498 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-02-09T17:14:07.508 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-02-09T17:14:07.509 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-02-09T17:14:07.509 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-02-09T17:14:07.567 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-02-09T17:14:07.570 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-02-09T17:14:07.646 INFO:teuthology.orchestra.run.smithi179.stdout:rbd-mirror-17.2.7-499.g89a83b98.el8.x86_64.rpm 4.9 MB/s | 3.5 MB 00:00 2024-02-09T17:14:07.647 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-02-09T17:14:07.647 INFO:teuthology.orchestra.run.smithi179.stdout:Total 4.9 MB/s | 3.5 MB 00:00 2024-02-09T17:14:07.647 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-02-09T17:14:07.658 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-02-09T17:14:07.658 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-02-09T17:14:07.692 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-mirror-2:17.2.7-499.g89a83b98.el8.x86_64 1/1 2024-02-09T17:14:07.718 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-02-09T17:14:07.721 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-02-09T17:14:08.000 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-02-09T17:14:08.094 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: rbd-mirror-2:17.2.7-499.g89a83b98.el8.x86_64 1/1 2024-02-09T17:14:08.192 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-02-09T17:14:08.196 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : rbd-mirror-2:17.2.7-499.g89a83b98.el8.x86_64 1/1 2024-02-09T17:14:08.402 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : rbd-mirror-2:17.2.7-499.g89a83b98.el8.x86_64 1/1 2024-02-09T17:14:08.501 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-mirror-2:17.2.7-499.g89a83b98.el8.x86_64 1/1 2024-02-09T17:14:08.501 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:14:08.501 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-02-09T17:14:08.502 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-mirror-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:14:08.502 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:14:08.502 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-09T17:14:08.601 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: rbd-mirror-2:17.2.7-499.g89a83b98.el8.x86_64 1/1 2024-02-09T17:14:08.735 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install rbd-nbd 2024-02-09T17:14:08.823 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: rbd-mirror-2:17.2.7-499.g89a83b98.el8.x86_64 1/1 2024-02-09T17:14:09.006 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : rbd-mirror-2:17.2.7-499.g89a83b98.el8.x86_64 1/1 2024-02-09T17:14:09.006 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:14:09.007 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-02-09T17:14:09.007 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-mirror-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:14:09.007 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:14:09.007 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-02-09T17:14:09.193 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : rbd-mirror-2:17.2.7-499.g89a83b98.el8.x86_64 1/1 2024-02-09T17:14:09.193 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:14:09.193 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-02-09T17:14:09.193 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-mirror-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:14:09.193 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:14:09.194 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-02-09T17:14:09.217 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install rbd-nbd 2024-02-09T17:14:09.430 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install rbd-nbd 2024-02-09T17:14:09.504 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:11 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:14:09.983 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:02:11 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:14:10.202 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:02:12 ago on Fri 09 Feb 2024 05:11:58 PM UTC. 2024-02-09T17:14:11.007 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-02-09T17:14:11.008 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-09T17:14:11.008 INFO:teuthology.orchestra.run.smithi162.stdout: Package Architecture Version Repository Size 2024-02-09T17:14:11.008 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-09T17:14:11.008 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-02-09T17:14:11.008 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-nbd x86_64 2:17.2.7-499.g89a83b98.el8 ceph 175 k 2024-02-09T17:14:11.008 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:14:11.008 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-02-09T17:14:11.008 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-02-09T17:14:11.008 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-02-09T17:14:11.009 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:14:11.009 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 175 k 2024-02-09T17:14:11.009 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 556 k 2024-02-09T17:14:11.009 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-02-09T17:14:11.280 INFO:teuthology.orchestra.run.smithi162.stdout:rbd-nbd-17.2.7-499.g89a83b98.el8.x86_64.rpm 644 kB/s | 175 kB 00:00 2024-02-09T17:14:11.281 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-02-09T17:14:11.281 INFO:teuthology.orchestra.run.smithi162.stdout:Total 641 kB/s | 175 kB 00:00 2024-02-09T17:14:11.281 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-02-09T17:14:11.292 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-02-09T17:14:11.292 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-02-09T17:14:11.339 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-02-09T17:14:11.342 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-02-09T17:14:11.452 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-02-09T17:14:11.493 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-02-09T17:14:11.494 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-02-09T17:14:11.494 INFO:teuthology.orchestra.run.smithi099.stdout: Package Architecture Version Repository Size 2024-02-09T17:14:11.494 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-02-09T17:14:11.494 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-02-09T17:14:11.494 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-nbd x86_64 2:17.2.7-499.g89a83b98.el8 ceph 175 k 2024-02-09T17:14:11.494 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:14:11.494 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-02-09T17:14:11.494 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-02-09T17:14:11.495 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2024-02-09T17:14:11.495 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:14:11.495 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 175 k 2024-02-09T17:14:11.495 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 556 k 2024-02-09T17:14:11.495 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-02-09T17:14:11.631 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-nbd-2:17.2.7-499.g89a83b98.el8.x86_64 1/1 2024-02-09T17:14:11.699 INFO:teuthology.orchestra.run.smithi099.stdout:rbd-nbd-17.2.7-499.g89a83b98.el8.x86_64.rpm 860 kB/s | 175 kB 00:00 2024-02-09T17:14:11.699 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-02-09T17:14:11.699 INFO:teuthology.orchestra.run.smithi099.stdout:Total 854 kB/s | 175 kB 00:00 2024-02-09T17:14:11.700 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-02-09T17:14:11.710 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-02-09T17:14:11.710 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-02-09T17:14:11.718 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-02-09T17:14:11.719 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-02-09T17:14:11.719 INFO:teuthology.orchestra.run.smithi179.stdout: Package Architecture Version Repository Size 2024-02-09T17:14:11.719 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-02-09T17:14:11.719 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-02-09T17:14:11.719 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-nbd x86_64 2:17.2.7-499.g89a83b98.el8 ceph 175 k 2024-02-09T17:14:11.719 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:14:11.719 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-02-09T17:14:11.719 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-02-09T17:14:11.719 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-02-09T17:14:11.719 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:14:11.720 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 175 k 2024-02-09T17:14:11.720 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 556 k 2024-02-09T17:14:11.720 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-02-09T17:14:11.756 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-02-09T17:14:11.759 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-02-09T17:14:11.870 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-02-09T17:14:11.915 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: rbd-nbd-2:17.2.7-499.g89a83b98.el8.x86_64 1/1 2024-02-09T17:14:11.932 INFO:teuthology.orchestra.run.smithi179.stdout:rbd-nbd-17.2.7-499.g89a83b98.el8.x86_64.rpm 825 kB/s | 175 kB 00:00 2024-02-09T17:14:11.932 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-02-09T17:14:11.933 INFO:teuthology.orchestra.run.smithi179.stdout:Total 820 kB/s | 175 kB 00:00 2024-02-09T17:14:11.933 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-02-09T17:14:11.943 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-02-09T17:14:11.943 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-02-09T17:14:11.990 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-02-09T17:14:11.993 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-02-09T17:14:12.068 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : rbd-nbd-2:17.2.7-499.g89a83b98.el8.x86_64 1/1 2024-02-09T17:14:12.111 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-02-09T17:14:12.301 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-nbd-2:17.2.7-499.g89a83b98.el8.x86_64 1/1 2024-02-09T17:14:12.301 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:14:12.301 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-02-09T17:14:12.301 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-nbd-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:14:12.301 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-02-09T17:14:12.301 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-02-09T17:14:12.308 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : rbd-nbd-2:17.2.7-499.g89a83b98.el8.x86_64 1/1 2024-02-09T17:14:12.355 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: rbd-nbd-2:17.2.7-499.g89a83b98.el8.x86_64 1/1 2024-02-09T17:14:12.520 DEBUG:teuthology.parallel:result is None 2024-02-09T17:14:12.632 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: rbd-nbd-2:17.2.7-499.g89a83b98.el8.x86_64 1/1 2024-02-09T17:14:12.725 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : rbd-nbd-2:17.2.7-499.g89a83b98.el8.x86_64 1/1 2024-02-09T17:14:12.725 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:14:12.725 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-02-09T17:14:12.725 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-nbd-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:14:12.726 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:14:12.726 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-02-09T17:14:12.941 DEBUG:teuthology.parallel:result is None 2024-02-09T17:14:12.993 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : rbd-nbd-2:17.2.7-499.g89a83b98.el8.x86_64 1/1 2024-02-09T17:14:12.993 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:14:12.993 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-02-09T17:14:12.993 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-nbd-2:17.2.7-499.g89a83b98.el8.x86_64 2024-02-09T17:14:12.993 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:14:12.993 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-02-09T17:14:13.197 DEBUG:teuthology.parallel:result is None 2024-02-09T17:14:13.198 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-09T17:14:13.198 INFO:teuthology.packaging:ref: None 2024-02-09T17:14:13.198 INFO:teuthology.packaging:tag: None 2024-02-09T17:14:13.198 INFO:teuthology.packaging:branch: quincy 2024-02-09T17:14:13.198 INFO:teuthology.packaging:sha1: 720625d38f7f95e6fa31efce87133247d9d28517 2024-02-09T17:14:13.199 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=quincy 2024-02-09T17:14:13.446 DEBUG:teuthology.orchestra.run.smithi099:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-09T17:14:13.471 INFO:teuthology.orchestra.run.smithi099.stdout:17.2.7-499.g89a83b98.el8 2024-02-09T17:14:13.471 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-499.g89a83b98.el8 2024-02-09T17:14:13.471 INFO:teuthology.task.install:The correct ceph version 17.2.7-499.g89a83b98 is installed. 2024-02-09T17:14:13.473 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-09T17:14:13.473 INFO:teuthology.packaging:ref: None 2024-02-09T17:14:13.473 INFO:teuthology.packaging:tag: None 2024-02-09T17:14:13.473 INFO:teuthology.packaging:branch: quincy 2024-02-09T17:14:13.473 INFO:teuthology.packaging:sha1: 720625d38f7f95e6fa31efce87133247d9d28517 2024-02-09T17:14:13.473 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=quincy 2024-02-09T17:14:13.701 DEBUG:teuthology.orchestra.run.smithi162:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-09T17:14:13.726 INFO:teuthology.orchestra.run.smithi162.stdout:17.2.7-499.g89a83b98.el8 2024-02-09T17:14:13.726 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-499.g89a83b98.el8 2024-02-09T17:14:13.726 INFO:teuthology.task.install:The correct ceph version 17.2.7-499.g89a83b98 is installed. 2024-02-09T17:14:13.728 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-09T17:14:13.728 INFO:teuthology.packaging:ref: None 2024-02-09T17:14:13.728 INFO:teuthology.packaging:tag: None 2024-02-09T17:14:13.728 INFO:teuthology.packaging:branch: quincy 2024-02-09T17:14:13.728 INFO:teuthology.packaging:sha1: 720625d38f7f95e6fa31efce87133247d9d28517 2024-02-09T17:14:13.728 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=quincy 2024-02-09T17:14:13.955 DEBUG:teuthology.orchestra.run.smithi179:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-09T17:14:13.979 INFO:teuthology.orchestra.run.smithi179.stdout:17.2.7-499.g89a83b98.el8 2024-02-09T17:14:13.979 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-499.g89a83b98.el8 2024-02-09T17:14:13.980 INFO:teuthology.task.install:The correct ceph version 17.2.7-499.g89a83b98 is installed. 2024-02-09T17:14:13.981 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-09T17:14:13.982 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-02-09T17:14:13.982 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-09T17:14:14.015 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-02-09T17:14:14.015 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-09T17:14:14.049 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-02-09T17:14:14.049 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-09T17:14:14.078 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-09T17:14:14.078 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-02-09T17:14:14.078 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/usr/bin/daemon-helper 2024-02-09T17:14:14.108 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-09T17:14:14.177 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-02-09T17:14:14.177 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/daemon-helper 2024-02-09T17:14:14.210 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-09T17:14:14.282 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-02-09T17:14:14.282 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/usr/bin/daemon-helper 2024-02-09T17:14:14.310 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-09T17:14:14.378 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-09T17:14:14.378 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-02-09T17:14:14.378 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-09T17:14:14.409 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-09T17:14:14.479 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-02-09T17:14:14.479 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-09T17:14:14.511 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-09T17:14:14.582 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-02-09T17:14:14.582 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-09T17:14:14.609 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-09T17:14:14.676 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-09T17:14:14.677 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-02-09T17:14:14.677 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/usr/bin/stdin-killer 2024-02-09T17:14:14.707 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-09T17:14:14.776 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-02-09T17:14:14.776 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/stdin-killer 2024-02-09T17:14:14.808 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-09T17:14:14.878 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-02-09T17:14:14.878 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/usr/bin/stdin-killer 2024-02-09T17:14:14.905 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-09T17:14:14.972 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-09T17:14:15.066 INFO:tasks.cephadm:Config: {'conf': {'mon': {'auth allow insecure global id reclaim': True, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'client.2': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(MON_DOWN\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN'], 'sha1': '720625d38f7f95e6fa31efce87133247d9d28517'} 2024-02-09T17:14:15.066 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 2024-02-09T17:14:15.067 INFO:tasks.cephadm:Cluster fsid is a69c870a-c76e-11ee-95b7-87774f69a715 2024-02-09T17:14:15.067 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-09T17:14:15.067 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.99', 'mon.b': '172.21.15.162', 'mon.c': '172.21.15.179'} 2024-02-09T17:14:15.067 INFO:tasks.cephadm:First mon is mon.a on smithi099 2024-02-09T17:14:15.067 INFO:tasks.cephadm:First mgr is y 2024-02-09T17:14:15.067 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-09T17:14:15.067 DEBUG:teuthology.orchestra.run.smithi099:> sudo hostname $(hostname -s) 2024-02-09T17:14:15.100 DEBUG:teuthology.orchestra.run.smithi162:> sudo hostname $(hostname -s) 2024-02-09T17:14:15.131 DEBUG:teuthology.orchestra.run.smithi179:> sudo hostname $(hostname -s) 2024-02-09T17:14:15.158 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-02-09T17:14:15.158 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=720625d38f7f95e6fa31efce87133247d9d28517 2024-02-09T17:14:15.315 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '720625d38f7f95e6fa31efce87133247d9d28517', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/76568/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1271-g720625d3', 'node_name': '172.21.4.65+confusa03', 'job_name': 'ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1271.g720625d3'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-02-06-1154/720625d38f7f95e6fa31efce87133247d9d28517/centos/8/flavors/default/', 'distro_codename': None, 'modified': '2024-02-06 23:16:22.481793', 'distro_version': '8', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-yuri2-testing-2024-02-06-1154', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-yuri2-testing-2024-02-06-1154/720625d38f7f95e6fa31efce87133247d9d28517/centos/8/flavors/default/', 'archs': ['source', 'x86_64', 'arm64'], 'distro': 'centos'}] 2024-02-09T17:14:15.475 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-yuri2-testing-2024-02-06-1154, sha1 720625d38f7f95e6fa31efce87133247d9d28517 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F8%2Fx86_64&flavor=default&sha1=720625d38f7f95e6fa31efce87133247d9d28517 2024-02-09T17:14:15.477 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-yuri2-testing-2024-02-06-1154/720625d38f7f95e6fa31efce87133247d9d28517/centos/8/x86_64/flavors/default/cephadm 2024-02-09T17:14:15.477 DEBUG:teuthology.orchestra.run.smithi099:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-yuri2-testing-2024-02-06-1154/720625d38f7f95e6fa31efce87133247d9d28517/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-09T17:14:16.292 INFO:teuthology.orchestra.run.smithi099.stdout:-rw-rw-r--. 1 ubuntu ubuntu 2250325 Feb 9 17:14 /home/ubuntu/cephtest/cephadm 2024-02-09T17:14:16.293 DEBUG:teuthology.orchestra.run.smithi162:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-yuri2-testing-2024-02-06-1154/720625d38f7f95e6fa31efce87133247d9d28517/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-09T17:14:17.162 INFO:teuthology.orchestra.run.smithi162.stdout:-rw-rw-r--. 1 ubuntu ubuntu 2250325 Feb 9 17:14 /home/ubuntu/cephtest/cephadm 2024-02-09T17:14:17.162 DEBUG:teuthology.orchestra.run.smithi179:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-yuri2-testing-2024-02-06-1154/720625d38f7f95e6fa31efce87133247d9d28517/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-09T17:14:18.062 INFO:teuthology.orchestra.run.smithi179.stdout:-rw-rw-r--. 1 ubuntu ubuntu 2250325 Feb 9 17:14 /home/ubuntu/cephtest/cephadm 2024-02-09T17:14:18.062 DEBUG:teuthology.orchestra.run.smithi099:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-09T17:14:18.085 DEBUG:teuthology.orchestra.run.smithi162:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-09T17:14:18.108 DEBUG:teuthology.orchestra.run.smithi179:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-09T17:14:18.143 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 on all hosts... 2024-02-09T17:14:18.143 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 pull 2024-02-09T17:14:18.145 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 pull 2024-02-09T17:14:18.150 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 pull 2024-02-09T17:14:19.334 INFO:teuthology.orchestra.run.smithi162.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517... 2024-02-09T17:14:19.418 INFO:teuthology.orchestra.run.smithi099.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517... 2024-02-09T17:14:19.459 INFO:teuthology.orchestra.run.smithi179.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517... 2024-02-09T17:14:51.208 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:14:51.209 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_version": "ceph version 19.0.0-1271-g720625d3 (720625d38f7f95e6fa31efce87133247d9d28517) squid (dev)", 2024-02-09T17:14:51.209 INFO:teuthology.orchestra.run.smithi162.stdout: "image_id": "460e8c0526a4ac42c6f611aa7ad530641623fa5ddb07db758fca88a9b9da23e9", 2024-02-09T17:14:51.209 INFO:teuthology.orchestra.run.smithi162.stdout: "repo_digests": [ 2024-02-09T17:14:51.209 INFO:teuthology.orchestra.run.smithi162.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:96064cefa09b46b83404968abe29e7381bd2bc1fd07dadd4e4f9771696ab50aa" 2024-02-09T17:14:51.209 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:14:51.209 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:15:08.612 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:15:08.612 INFO:teuthology.orchestra.run.smithi099.stdout: "ceph_version": "ceph version 19.0.0-1271-g720625d3 (720625d38f7f95e6fa31efce87133247d9d28517) squid (dev)", 2024-02-09T17:15:08.612 INFO:teuthology.orchestra.run.smithi099.stdout: "image_id": "460e8c0526a4ac42c6f611aa7ad530641623fa5ddb07db758fca88a9b9da23e9", 2024-02-09T17:15:08.612 INFO:teuthology.orchestra.run.smithi099.stdout: "repo_digests": [ 2024-02-09T17:15:08.612 INFO:teuthology.orchestra.run.smithi099.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:96064cefa09b46b83404968abe29e7381bd2bc1fd07dadd4e4f9771696ab50aa" 2024-02-09T17:15:08.612 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:15:08.613 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:15:16.123 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:15:16.124 INFO:teuthology.orchestra.run.smithi179.stdout: "ceph_version": "ceph version 19.0.0-1271-g720625d3 (720625d38f7f95e6fa31efce87133247d9d28517) squid (dev)", 2024-02-09T17:15:16.124 INFO:teuthology.orchestra.run.smithi179.stdout: "image_id": "460e8c0526a4ac42c6f611aa7ad530641623fa5ddb07db758fca88a9b9da23e9", 2024-02-09T17:15:16.124 INFO:teuthology.orchestra.run.smithi179.stdout: "repo_digests": [ 2024-02-09T17:15:16.124 INFO:teuthology.orchestra.run.smithi179.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:96064cefa09b46b83404968abe29e7381bd2bc1fd07dadd4e4f9771696ab50aa" 2024-02-09T17:15:16.124 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:15:16.124 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:15:16.150 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /etc/ceph 2024-02-09T17:15:16.188 DEBUG:teuthology.orchestra.run.smithi162:> sudo mkdir -p /etc/ceph 2024-02-09T17:15:16.224 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /etc/ceph 2024-02-09T17:15:16.255 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 777 /etc/ceph 2024-02-09T17:15:16.286 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 777 /etc/ceph 2024-02-09T17:15:16.316 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod 777 /etc/ceph 2024-02-09T17:15:16.343 INFO:tasks.cephadm:Writing seed config... 2024-02-09T17:15:16.344 INFO:tasks.cephadm: override: [mon] auth allow insecure global id reclaim = True 2024-02-09T17:15:16.344 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-09T17:15:16.344 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-09T17:15:16.344 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-09T17:15:16.344 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-02-09T17:15:16.345 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-02-09T17:15:16.345 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-02-09T17:15:16.345 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-02-09T17:15:16.345 INFO:tasks.cephadm: override: [mon] mon scrub interval = 300 2024-02-09T17:15:16.345 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-02-09T17:15:16.345 INFO:tasks.cephadm: override: [client.2] debug ms = 1 2024-02-09T17:15:16.345 INFO:tasks.cephadm: override: [client.2] debug objecter = 20 2024-02-09T17:15:16.345 INFO:tasks.cephadm: override: [client.2] debug rados = 20 2024-02-09T17:15:16.345 INFO:tasks.cephadm: override: [global] mon client directed command retry = 5 2024-02-09T17:15:16.345 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2024-02-09T17:15:16.345 INFO:tasks.cephadm: override: [global] ms inject socket failures = 2500 2024-02-09T17:15:16.345 INFO:tasks.cephadm: override: [global] ms tcp read timeout = 5 2024-02-09T17:15:16.345 INFO:tasks.cephadm: override: [global] osd_pool_default_min_size = 2 2024-02-09T17:15:16.345 INFO:tasks.cephadm: override: [global] osd_pool_default_size = 3 2024-02-09T17:15:16.345 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-09T17:15:16.345 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-09T17:15:16.345 INFO:tasks.cephadm: override: [osd] bluestore zero block detection = True 2024-02-09T17:15:16.346 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-09T17:15:16.346 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-09T17:15:16.346 INFO:tasks.cephadm: override: [osd] osd blocked scrub grace period = 3600 2024-02-09T17:15:16.346 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-02-09T17:15:16.346 INFO:tasks.cephadm: override: [osd] osd debug verify cached snaps = True 2024-02-09T17:15:16.346 INFO:tasks.cephadm: override: [osd] osd debug verify missing on start = True 2024-02-09T17:15:16.346 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-02-09T17:15:16.346 INFO:tasks.cephadm: override: [osd] osd max markdown count = 1000 2024-02-09T17:15:16.346 INFO:tasks.cephadm: override: [osd] osd mclock override recovery settings = True 2024-02-09T17:15:16.346 INFO:tasks.cephadm: override: [osd] osd mclock profile = high_recovery_ops 2024-02-09T17:15:16.346 INFO:tasks.cephadm: override: [osd] osd op queue = debug_random 2024-02-09T17:15:16.346 INFO:tasks.cephadm: override: [osd] osd op queue cut off = debug_random 2024-02-09T17:15:16.346 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-02-09T17:15:16.346 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-02-09T17:15:16.346 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-02-09T17:15:16.346 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-02-09T17:15:16.347 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-02-09T17:15:16.347 DEBUG:teuthology.orchestra.run.smithi099:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-09T17:15:16.367 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = a69c870a-c76e-11ee-95b7-87774f69a715 mon client directed command retry = 5 mon election default strategy = 3 ms inject socket failures = 2500 ms tcp read timeout = 5 osd_pool_default_min_size = 2 osd_pool_default_size = 3 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = True osd debug verify missing on start = True osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore zero block detection = True debug ms = 1 debug osd = 20 osd blocked scrub grace period = 3600 osd debug reject backfill probability = 0.3 osd max backfills = 3 osd max markdown count = 1000 osd mclock override recovery settings = True osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false auth allow insecure global id reclaim = True debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 mon scrub interval = 300 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.2] debug ms = 1 debug objecter = 20 debug rados = 20 2024-02-09T17:15:16.367 DEBUG:teuthology.orchestra.run.smithi099:mon.a> sudo journalctl -f -n 0 -u ceph-a69c870a-c76e-11ee-95b7-87774f69a715@mon.a.service 2024-02-09T17:15:16.410 DEBUG:teuthology.orchestra.run.smithi099:mgr.y> sudo journalctl -f -n 0 -u ceph-a69c870a-c76e-11ee-95b7-87774f69a715@mgr.y.service 2024-02-09T17:15:16.439 INFO:journalctl@ceph.mon.a.smithi099.stdout:-- Logs begin at Fri 2024-02-09 17:04:25 UTC. -- 2024-02-09T17:15:16.440 INFO:tasks.cephadm:Bootstrapping... 2024-02-09T17:15:16.440 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 -v bootstrap --fsid a69c870a-c76e-11ee-95b7-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.99 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-09T17:15:16.468 INFO:journalctl@ceph.mgr.y.smithi099.stdout:-- Logs begin at Fri 2024-02-09 17:04:25 UTC. -- 2024-02-09T17:15:16.643 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-02-09T17:15:16.644 INFO:teuthology.orchestra.run.smithi099.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517', '-v', 'bootstrap', '--fsid', 'a69c870a-c76e-11ee-95b7-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.99', '--skip-admin-label'] 2024-02-09T17:15:16.678 INFO:teuthology.orchestra.run.smithi099.stdout:/bin/podman: stdout 4.8.3 2024-02-09T17:15:16.679 INFO:teuthology.orchestra.run.smithi099.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-02-09T17:15:16.679 INFO:teuthology.orchestra.run.smithi099.stdout:Verifying podman|docker is present... 2024-02-09T17:15:16.712 INFO:teuthology.orchestra.run.smithi099.stdout:/bin/podman: stdout 4.8.3 2024-02-09T17:15:16.713 INFO:teuthology.orchestra.run.smithi099.stdout:Verifying lvm2 is present... 2024-02-09T17:15:16.713 INFO:teuthology.orchestra.run.smithi099.stdout:Verifying time synchronization is in place... 2024-02-09T17:15:16.720 INFO:teuthology.orchestra.run.smithi099.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-02-09T17:15:16.720 INFO:teuthology.orchestra.run.smithi099.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-02-09T17:15:16.725 INFO:teuthology.orchestra.run.smithi099.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-02-09T17:15:16.725 INFO:teuthology.orchestra.run.smithi099.stdout:systemctl: stdout inactive 2024-02-09T17:15:16.731 INFO:teuthology.orchestra.run.smithi099.stdout:systemctl: stdout enabled 2024-02-09T17:15:16.737 INFO:teuthology.orchestra.run.smithi099.stdout:systemctl: stdout active 2024-02-09T17:15:16.737 INFO:teuthology.orchestra.run.smithi099.stdout:Unit chronyd.service is enabled and running 2024-02-09T17:15:16.737 INFO:teuthology.orchestra.run.smithi099.stdout:Repeating the final host check... 2024-02-09T17:15:16.768 INFO:teuthology.orchestra.run.smithi099.stdout:/bin/podman: stdout 4.8.3 2024-02-09T17:15:16.768 INFO:teuthology.orchestra.run.smithi099.stdout:podman (/bin/podman) version 4.8.3 is present 2024-02-09T17:15:16.768 INFO:teuthology.orchestra.run.smithi099.stdout:systemctl is present 2024-02-09T17:15:16.768 INFO:teuthology.orchestra.run.smithi099.stdout:lvcreate is present 2024-02-09T17:15:16.774 INFO:teuthology.orchestra.run.smithi099.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-02-09T17:15:16.775 INFO:teuthology.orchestra.run.smithi099.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-02-09T17:15:16.780 INFO:teuthology.orchestra.run.smithi099.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-02-09T17:15:16.780 INFO:teuthology.orchestra.run.smithi099.stdout:systemctl: stdout inactive 2024-02-09T17:15:16.786 INFO:teuthology.orchestra.run.smithi099.stdout:systemctl: stdout enabled 2024-02-09T17:15:16.792 INFO:teuthology.orchestra.run.smithi099.stdout:systemctl: stdout active 2024-02-09T17:15:16.792 INFO:teuthology.orchestra.run.smithi099.stdout:Unit chronyd.service is enabled and running 2024-02-09T17:15:16.792 INFO:teuthology.orchestra.run.smithi099.stdout:Host looks OK 2024-02-09T17:15:16.792 INFO:teuthology.orchestra.run.smithi099.stdout:Cluster fsid: a69c870a-c76e-11ee-95b7-87774f69a715 2024-02-09T17:15:16.792 INFO:teuthology.orchestra.run.smithi099.stdout:Acquiring lock 140344360194456 on /run/cephadm/a69c870a-c76e-11ee-95b7-87774f69a715.lock 2024-02-09T17:15:16.792 INFO:teuthology.orchestra.run.smithi099.stdout:Lock 140344360194456 acquired on /run/cephadm/a69c870a-c76e-11ee-95b7-87774f69a715.lock 2024-02-09T17:15:16.793 INFO:teuthology.orchestra.run.smithi099.stdout:Verifying IP 172.21.15.99 port 3300 ... 2024-02-09T17:15:16.793 INFO:teuthology.orchestra.run.smithi099.stdout:Verifying IP 172.21.15.99 port 6789 ... 2024-02-09T17:15:16.793 INFO:teuthology.orchestra.run.smithi099.stdout:Base mon IP(s) is [172.21.15.99:3300, 172.21.15.99:6789], mon addrv is [v2:172.21.15.99:3300,v1:172.21.15.99:6789] 2024-02-09T17:15:16.797 INFO:teuthology.orchestra.run.smithi099.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.99 metric 100 2024-02-09T17:15:16.797 INFO:teuthology.orchestra.run.smithi099.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.99 metric 100 2024-02-09T17:15:16.800 INFO:teuthology.orchestra.run.smithi099.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-02-09T17:15:16.800 INFO:teuthology.orchestra.run.smithi099.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-02-09T17:15:16.800 INFO:teuthology.orchestra.run.smithi099.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1598sec hoplimit 64 pref medium 2024-02-09T17:15:16.802 INFO:teuthology.orchestra.run.smithi099.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-09T17:15:16.802 INFO:teuthology.orchestra.run.smithi099.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-02-09T17:15:16.803 INFO:teuthology.orchestra.run.smithi099.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-02-09T17:15:16.803 INFO:teuthology.orchestra.run.smithi099.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-02-09T17:15:16.803 INFO:teuthology.orchestra.run.smithi099.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d803/64 scope link 2024-02-09T17:15:16.803 INFO:teuthology.orchestra.run.smithi099.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-02-09T17:15:16.803 INFO:teuthology.orchestra.run.smithi099.stdout:Mon IP `172.21.15.99` is in CIDR network `172.21.0.0/20` 2024-02-09T17:15:16.803 INFO:teuthology.orchestra.run.smithi099.stdout:Mon IP `172.21.15.99` is in CIDR network `172.21.0.0/20` 2024-02-09T17:15:16.803 INFO:teuthology.orchestra.run.smithi099.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-02-09T17:15:16.804 INFO:teuthology.orchestra.run.smithi099.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-09T17:15:16.805 INFO:teuthology.orchestra.run.smithi099.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517... 2024-02-09T17:15:17.391 INFO:teuthology.orchestra.run.smithi099.stdout:/bin/podman: stdout 460e8c0526a4ac42c6f611aa7ad530641623fa5ddb07db758fca88a9b9da23e9 2024-02-09T17:15:17.391 INFO:teuthology.orchestra.run.smithi099.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517... 2024-02-09T17:15:17.392 INFO:teuthology.orchestra.run.smithi099.stdout:/bin/podman: stderr Getting image source signatures 2024-02-09T17:15:17.392 INFO:teuthology.orchestra.run.smithi099.stdout:/bin/podman: stderr Copying blob sha256:69df73b173f30fb8bed22f8f830efa2b72d5c74df21a25271257860780efd9f6 2024-02-09T17:15:17.392 INFO:teuthology.orchestra.run.smithi099.stdout:/bin/podman: stderr Copying blob sha256:58df4e03ba066accf3a7e07426bab5f1c867e27a139801c642bdafb42b9a2d9a 2024-02-09T17:15:17.392 INFO:teuthology.orchestra.run.smithi099.stdout:/bin/podman: stderr Copying config sha256:460e8c0526a4ac42c6f611aa7ad530641623fa5ddb07db758fca88a9b9da23e9 2024-02-09T17:15:17.392 INFO:teuthology.orchestra.run.smithi099.stdout:/bin/podman: stderr Writing manifest to image destination 2024-02-09T17:15:18.518 INFO:teuthology.orchestra.run.smithi099.stdout:ceph: stdout ceph version 19.0.0-1271-g720625d3 (720625d38f7f95e6fa31efce87133247d9d28517) squid (dev) 2024-02-09T17:15:18.519 INFO:teuthology.orchestra.run.smithi099.stdout:Ceph version: ceph version 19.0.0-1271-g720625d3 (720625d38f7f95e6fa31efce87133247d9d28517) squid (dev) 2024-02-09T17:15:18.519 INFO:teuthology.orchestra.run.smithi099.stdout:Extracting ceph user uid/gid from container image... 2024-02-09T17:15:19.579 INFO:teuthology.orchestra.run.smithi099.stdout:stat: stdout 167 167 2024-02-09T17:15:19.579 INFO:teuthology.orchestra.run.smithi099.stdout:Creating initial keys... 2024-02-09T17:15:20.746 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph-authtool: stdout AQCoXcZluaWzARAAxK4RecBHKqFaJUDiJPKH6Q== 2024-02-09T17:15:21.784 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph-authtool: stdout AQCpXcZlj1PfBxAA7rwAb20inr3h7KXOACIe4Q== 2024-02-09T17:15:22.823 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph-authtool: stdout AQCqXcZlWrQgDRAAH7WYPvAe/NTehehaF7eWWA== 2024-02-09T17:15:22.823 INFO:teuthology.orchestra.run.smithi099.stdout:Creating initial monmap... 2024-02-09T17:15:23.854 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-09T17:15:23.854 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-02-09T17:15:23.854 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to a69c870a-c76e-11ee-95b7-87774f69a715 2024-02-09T17:15:23.854 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-09T17:15:23.854 INFO:teuthology.orchestra.run.smithi099.stdout:monmaptool for a [v2:172.21.15.99:3300,v1:172.21.15.99:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-09T17:15:23.854 INFO:teuthology.orchestra.run.smithi099.stdout:setting min_mon_release = quincy 2024-02-09T17:15:23.854 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/monmaptool: set fsid to a69c870a-c76e-11ee-95b7-87774f69a715 2024-02-09T17:15:23.854 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-09T17:15:23.854 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:15:23.854 INFO:teuthology.orchestra.run.smithi099.stdout:Creating mon... 2024-02-09T17:15:24.882 INFO:teuthology.orchestra.run.smithi099.stdout:create mon.a on 2024-02-09T17:15:25.011 INFO:teuthology.orchestra.run.smithi099.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-02-09T17:15:25.095 INFO:teuthology.orchestra.run.smithi099.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-09T17:15:25.211 INFO:teuthology.orchestra.run.smithi099.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-a69c870a-c76e-11ee-95b7-87774f69a715.target → /etc/systemd/system/ceph-a69c870a-c76e-11ee-95b7-87774f69a715.target. 2024-02-09T17:15:25.212 INFO:teuthology.orchestra.run.smithi099.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-a69c870a-c76e-11ee-95b7-87774f69a715.target → /etc/systemd/system/ceph-a69c870a-c76e-11ee-95b7-87774f69a715.target. 2024-02-09T17:15:25.401 INFO:teuthology.orchestra.run.smithi099.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-a69c870a-c76e-11ee-95b7-87774f69a715@mon.a 2024-02-09T17:15:25.401 INFO:teuthology.orchestra.run.smithi099.stdout:systemctl: stderr Failed to reset failed state of unit ceph-a69c870a-c76e-11ee-95b7-87774f69a715@mon.a.service: Unit ceph-a69c870a-c76e-11ee-95b7-87774f69a715@mon.a.service not loaded. 2024-02-09T17:15:25.492 INFO:teuthology.orchestra.run.smithi099.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-a69c870a-c76e-11ee-95b7-87774f69a715.target.wants/ceph-a69c870a-c76e-11ee-95b7-87774f69a715@mon.a.service → /etc/systemd/system/ceph-a69c870a-c76e-11ee-95b7-87774f69a715@.service. 2024-02-09T17:15:25.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:25 smithi099 systemd[1]: Starting Ceph mon.a for a69c870a-c76e-11ee-95b7-87774f69a715... 2024-02-09T17:15:26.177 INFO:teuthology.orchestra.run.smithi099.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-09T17:15:26.177 INFO:teuthology.orchestra.run.smithi099.stdout:systemctl: stdout disabled 2024-02-09T17:15:26.185 INFO:teuthology.orchestra.run.smithi099.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-09T17:15:26.185 INFO:teuthology.orchestra.run.smithi099.stdout:systemctl: stdout inactive 2024-02-09T17:15:26.185 INFO:teuthology.orchestra.run.smithi099.stdout:firewalld.service is not enabled 2024-02-09T17:15:26.185 INFO:teuthology.orchestra.run.smithi099.stdout:Not possible to enable service . firewalld.service is not available 2024-02-09T17:15:26.185 INFO:teuthology.orchestra.run.smithi099.stdout:Waiting for mon to start... 2024-02-09T17:15:26.185 INFO:teuthology.orchestra.run.smithi099.stdout:Waiting for mon... 2024-02-09T17:15:26.392 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:26 smithi099 ceph-mon[38935]: mkfs a69c870a-c76e-11ee-95b7-87774f69a715 2024-02-09T17:15:26.392 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:26 smithi099 ceph-mon[38935]: mon.a is new leader, mons a in quorum (ranks 0) 2024-02-09T17:15:26.392 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:26 smithi099 bash[38897]: 24f217954f67500b4660c6fbb3eebd26f34617b0ef736d59ce9c28040028262d 2024-02-09T17:15:26.392 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:26 smithi099 systemd[1]: Started Ceph mon.a for a69c870a-c76e-11ee-95b7-87774f69a715. 2024-02-09T17:15:27.290 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout cluster: 2024-02-09T17:15:27.290 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout id: a69c870a-c76e-11ee-95b7-87774f69a715 2024-02-09T17:15:27.290 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-02-09T17:15:27.291 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout 2024-02-09T17:15:27.291 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout services: 2024-02-09T17:15:27.291 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.651222s) 2024-02-09T17:15:27.291 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-02-09T17:15:27.291 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-02-09T17:15:27.291 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout 2024-02-09T17:15:27.291 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout data: 2024-02-09T17:15:27.291 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-02-09T17:15:27.291 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-02-09T17:15:27.291 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-02-09T17:15:27.291 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout pgs: 2024-02-09T17:15:27.291 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout 2024-02-09T17:15:27.291 INFO:teuthology.orchestra.run.smithi099.stdout:mon is available 2024-02-09T17:15:27.291 INFO:teuthology.orchestra.run.smithi099.stdout:Assimilating anything we can from ceph.conf... 2024-02-09T17:15:27.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:27 smithi099 ceph-mon[38935]: mon.a is new leader, mons a in quorum (ranks 0) 2024-02-09T17:15:27.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:27 smithi099 ceph-mon[38935]: monmap e1: 1 mons at {a=[v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-09T17:15:27.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:27 smithi099 ceph-mon[38935]: fsmap 2024-02-09T17:15:27.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:27 smithi099 ceph-mon[38935]: osdmap e1: 0 total, 0 up, 0 in 2024-02-09T17:15:27.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:27 smithi099 ceph-mon[38935]: mgrmap e1: no daemons active 2024-02-09T17:15:27.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:27 smithi099 ceph-mon[38935]: from='client.? 172.21.15.99:0/3054896253' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-09T17:15:28.217 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:28 smithi099 ceph-mon[38935]: from='client.? 172.21.15.99:0/1721071741' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-09T17:15:28.217 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:28 smithi099 ceph-mon[38935]: from='client.? 172.21.15.99:0/1721071741' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-09T17:15:28.486 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout 2024-02-09T17:15:28.488 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout [global] 2024-02-09T17:15:28.488 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout fsid = a69c870a-c76e-11ee-95b7-87774f69a715 2024-02-09T17:15:28.488 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.99:3300,v1:172.21.15.99:6789] 2024-02-09T17:15:28.488 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-02-09T17:15:28.488 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-02-09T17:15:28.488 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-02-09T17:15:28.488 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout ms_tcp_read_timeout = 5 2024-02-09T17:15:28.488 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-02-09T17:15:28.488 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout 2024-02-09T17:15:28.488 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout [mgr] 2024-02-09T17:15:28.489 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-02-09T17:15:28.489 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout 2024-02-09T17:15:28.489 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout [osd] 2024-02-09T17:15:28.489 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-02-09T17:15:28.489 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-02-09T17:15:28.489 INFO:teuthology.orchestra.run.smithi099.stdout:Generating new minimal ceph.conf... 2024-02-09T17:15:29.289 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:29 smithi099 ceph-mon[38935]: from='client.? 172.21.15.99:0/3920553640' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:15:29.500 INFO:teuthology.orchestra.run.smithi099.stdout:Restarting the monitor... 2024-02-09T17:15:29.733 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:29 smithi099 systemd[1]: Stopping Ceph mon.a for a69c870a-c76e-11ee-95b7-87774f69a715... 2024-02-09T17:15:29.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:29 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mon-a[38912]: 2024-02-09T17:15:29.729+0000 7f1c1395a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T17:15:29.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:29 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mon-a[38912]: 2024-02-09T17:15:29.729+0000 7f1c1395a700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-02-09T17:15:30.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 bash[39260]: ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mon-a 2024-02-09T17:15:30.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 systemd[1]: ceph-a69c870a-c76e-11ee-95b7-87774f69a715@mon.a.service: Succeeded. 2024-02-09T17:15:30.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 systemd[1]: Stopped Ceph mon.a for a69c870a-c76e-11ee-95b7-87774f69a715. 2024-02-09T17:15:30.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 systemd[1]: Starting Ceph mon.a for a69c870a-c76e-11ee-95b7-87774f69a715... 2024-02-09T17:15:30.975 INFO:teuthology.orchestra.run.smithi099.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-02-09T17:15:31.174 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: set uid:gid to 167:167 (ceph:ceph) 2024-02-09T17:15:31.174 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: ceph version 19.0.0-1271-g720625d3 (720625d38f7f95e6fa31efce87133247d9d28517) squid (dev), process ceph-mon, pid 7 2024-02-09T17:15:31.174 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: pidfile_write: ignore empty --pid-file 2024-02-09T17:15:31.174 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: load: jerasure load: lrc 2024-02-09T17:15:31.174 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: RocksDB version: 7.9.2 2024-02-09T17:15:31.174 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Git sha 0 2024-02-09T17:15:31.174 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Compile date 2024-02-06 20:16:13 2024-02-09T17:15:31.175 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: DB SUMMARY 2024-02-09T17:15:31.175 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: DB Session ID: USCXM8FHBGTIXSCKVN61 2024-02-09T17:15:31.175 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: CURRENT file: CURRENT 2024-02-09T17:15:31.175 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: IDENTITY file: IDENTITY 2024-02-09T17:15:31.175 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-02-09T17:15:31.175 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-02-09T17:15:31.175 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 95187 ; 2024-02-09T17:15:31.175 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.error_if_exists: 0 2024-02-09T17:15:31.175 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.create_if_missing: 0 2024-02-09T17:15:31.175 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.paranoid_checks: 1 2024-02-09T17:15:31.175 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-09T17:15:31.175 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-09T17:15:31.175 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-09T17:15:31.176 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.env: 0x5634e1668ae0 2024-02-09T17:15:31.176 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.fs: PosixFileSystem 2024-02-09T17:15:31.176 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.info_log: 0x5634e289b8c0 2024-02-09T17:15:31.176 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.max_file_opening_threads: 16 2024-02-09T17:15:31.176 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.statistics: (nil) 2024-02-09T17:15:31.176 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.use_fsync: 0 2024-02-09T17:15:31.176 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.max_log_file_size: 0 2024-02-09T17:15:31.176 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-09T17:15:31.176 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-09T17:15:31.176 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.keep_log_file_num: 1000 2024-02-09T17:15:31.176 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.recycle_log_file_num: 0 2024-02-09T17:15:31.177 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.allow_fallocate: 1 2024-02-09T17:15:31.177 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.allow_mmap_reads: 0 2024-02-09T17:15:31.177 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.allow_mmap_writes: 0 2024-02-09T17:15:31.177 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.use_direct_reads: 0 2024-02-09T17:15:31.177 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-09T17:15:31.177 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.create_missing_column_families: 0 2024-02-09T17:15:31.177 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.db_log_dir: 2024-02-09T17:15:31.177 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.wal_dir: 2024-02-09T17:15:31.177 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-09T17:15:31.177 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-09T17:15:31.177 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-09T17:15:31.177 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-09T17:15:31.177 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-09T17:15:31.177 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-09T17:15:31.178 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.advise_random_on_open: 1 2024-02-09T17:15:31.178 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.db_write_buffer_size: 0 2024-02-09T17:15:31.178 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.write_buffer_manager: 0x5634e286f400 2024-02-09T17:15:31.178 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-09T17:15:31.178 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-09T17:15:31.178 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-09T17:15:31.178 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.rate_limiter: (nil) 2024-02-09T17:15:31.178 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-09T17:15:31.178 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.wal_recovery_mode: 2 2024-02-09T17:15:31.178 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.enable_thread_tracking: 0 2024-02-09T17:15:31.178 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.enable_pipelined_write: 0 2024-02-09T17:15:31.178 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.unordered_write: 0 2024-02-09T17:15:31.178 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-09T17:15:31.179 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-09T17:15:31.179 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-09T17:15:31.179 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-09T17:15:31.179 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.row_cache: None 2024-02-09T17:15:31.179 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.wal_filter: None 2024-02-09T17:15:31.179 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-09T17:15:31.179 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.allow_ingest_behind: 0 2024-02-09T17:15:31.179 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.two_write_queues: 0 2024-02-09T17:15:31.179 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.manual_wal_flush: 0 2024-02-09T17:15:31.179 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.wal_compression: 0 2024-02-09T17:15:31.179 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.atomic_flush: 0 2024-02-09T17:15:31.179 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-09T17:15:31.179 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-09T17:15:31.179 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-09T17:15:31.179 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.log_readahead_size: 0 2024-02-09T17:15:31.179 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-09T17:15:31.180 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.best_efforts_recovery: 0 2024-02-09T17:15:31.180 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-09T17:15:31.180 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-09T17:15:31.180 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.allow_data_in_errors: 0 2024-02-09T17:15:31.180 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.db_host_id: __hostname__ 2024-02-09T17:15:31.180 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-09T17:15:31.181 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.max_background_jobs: 2 2024-02-09T17:15:31.181 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.max_background_compactions: -1 2024-02-09T17:15:31.181 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.max_subcompactions: 1 2024-02-09T17:15:31.181 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-09T17:15:31.181 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-09T17:15:31.181 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-09T17:15:31.181 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.max_total_wal_size: 0 2024-02-09T17:15:31.181 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-09T17:15:31.182 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-09T17:15:31.182 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-09T17:15:31.182 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-09T17:15:31.182 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.max_open_files: -1 2024-02-09T17:15:31.182 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.bytes_per_sync: 0 2024-02-09T17:15:31.182 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-09T17:15:31.182 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-09T17:15:31.182 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.compaction_readahead_size: 0 2024-02-09T17:15:31.182 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.max_background_flushes: -1 2024-02-09T17:15:31.182 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Compression algorithms supported: 2024-02-09T17:15:31.182 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-09T17:15:31.182 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: kZSTD supported: 0 2024-02-09T17:15:31.182 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: kXpressCompression supported: 0 2024-02-09T17:15:31.182 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: kLZ4HCCompression supported: 1 2024-02-09T17:15:31.182 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: kZlibCompression supported: 1 2024-02-09T17:15:31.183 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: kSnappyCompression supported: 1 2024-02-09T17:15:31.183 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: kLZ4Compression supported: 1 2024-02-09T17:15:31.183 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: kBZip2Compression supported: 0 2024-02-09T17:15:31.183 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-09T17:15:31.183 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-09T17:15:31.183 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-02-09T17:15:31.183 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-09T17:15:31.183 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-09T17:15:31.183 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.merge_operator: 2024-02-09T17:15:31.183 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.compaction_filter: None 2024-02-09T17:15:31.183 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.compaction_filter_factory: None 2024-02-09T17:15:31.183 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.sst_partitioner_factory: None 2024-02-09T17:15:31.183 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-09T17:15:31.183 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-09T17:15:31.183 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5634e289b9c0) 2024-02-09T17:15:31.183 INFO:journalctl@ceph.mon.a.smithi099.stdout: cache_index_and_filter_blocks: 1 2024-02-09T17:15:31.183 INFO:journalctl@ceph.mon.a.smithi099.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-09T17:15:31.184 INFO:journalctl@ceph.mon.a.smithi099.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-09T17:15:31.184 INFO:journalctl@ceph.mon.a.smithi099.stdout: pin_top_level_index_and_filter: 1 2024-02-09T17:15:31.184 INFO:journalctl@ceph.mon.a.smithi099.stdout: index_type: 0 2024-02-09T17:15:31.184 INFO:journalctl@ceph.mon.a.smithi099.stdout: data_block_index_type: 0 2024-02-09T17:15:31.184 INFO:journalctl@ceph.mon.a.smithi099.stdout: index_shortening: 1 2024-02-09T17:15:31.184 INFO:journalctl@ceph.mon.a.smithi099.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-09T17:15:31.184 INFO:journalctl@ceph.mon.a.smithi099.stdout: checksum: 4 2024-02-09T17:15:31.184 INFO:journalctl@ceph.mon.a.smithi099.stdout: no_block_cache: 0 2024-02-09T17:15:31.184 INFO:journalctl@ceph.mon.a.smithi099.stdout: block_cache: 0x5634e289f090 2024-02-09T17:15:31.184 INFO:journalctl@ceph.mon.a.smithi099.stdout: block_cache_name: BinnedLRUCache 2024-02-09T17:15:31.184 INFO:journalctl@ceph.mon.a.smithi099.stdout: block_cache_options: 2024-02-09T17:15:31.184 INFO:journalctl@ceph.mon.a.smithi099.stdout: capacity : 536870912 2024-02-09T17:15:31.184 INFO:journalctl@ceph.mon.a.smithi099.stdout: num_shard_bits : 4 2024-02-09T17:15:31.184 INFO:journalctl@ceph.mon.a.smithi099.stdout: strict_capacity_limit : 0 2024-02-09T17:15:31.184 INFO:journalctl@ceph.mon.a.smithi099.stdout: high_pri_pool_ratio: 0.000 2024-02-09T17:15:31.185 INFO:journalctl@ceph.mon.a.smithi099.stdout: block_cache_compressed: (nil) 2024-02-09T17:15:31.185 INFO:journalctl@ceph.mon.a.smithi099.stdout: persistent_cache: (nil) 2024-02-09T17:15:31.185 INFO:journalctl@ceph.mon.a.smithi099.stdout: block_size: 4096 2024-02-09T17:15:31.185 INFO:journalctl@ceph.mon.a.smithi099.stdout: block_size_deviation: 10 2024-02-09T17:15:31.185 INFO:journalctl@ceph.mon.a.smithi099.stdout: block_restart_interval: 16 2024-02-09T17:15:31.185 INFO:journalctl@ceph.mon.a.smithi099.stdout: index_block_restart_interval: 1 2024-02-09T17:15:31.185 INFO:journalctl@ceph.mon.a.smithi099.stdout: metadata_block_size: 4096 2024-02-09T17:15:31.185 INFO:journalctl@ceph.mon.a.smithi099.stdout: partition_filters: 0 2024-02-09T17:15:31.185 INFO:journalctl@ceph.mon.a.smithi099.stdout: use_delta_encoding: 1 2024-02-09T17:15:31.185 INFO:journalctl@ceph.mon.a.smithi099.stdout: filter_policy: bloomfilter 2024-02-09T17:15:31.185 INFO:journalctl@ceph.mon.a.smithi099.stdout: whole_key_filtering: 1 2024-02-09T17:15:31.186 INFO:journalctl@ceph.mon.a.smithi099.stdout: verify_compression: 0 2024-02-09T17:15:31.186 INFO:journalctl@ceph.mon.a.smithi099.stdout: read_amp_bytes_per_bit: 0 2024-02-09T17:15:31.186 INFO:journalctl@ceph.mon.a.smithi099.stdout: format_version: 5 2024-02-09T17:15:31.186 INFO:journalctl@ceph.mon.a.smithi099.stdout: enable_index_compression: 1 2024-02-09T17:15:31.186 INFO:journalctl@ceph.mon.a.smithi099.stdout: block_align: 0 2024-02-09T17:15:31.186 INFO:journalctl@ceph.mon.a.smithi099.stdout: max_auto_readahead_size: 262144 2024-02-09T17:15:31.186 INFO:journalctl@ceph.mon.a.smithi099.stdout: prepopulate_block_cache: 0 2024-02-09T17:15:31.187 INFO:journalctl@ceph.mon.a.smithi099.stdout: initial_auto_readahead_size: 8192 2024-02-09T17:15:31.187 INFO:journalctl@ceph.mon.a.smithi099.stdout: num_file_reads_for_auto_readahead: 2 2024-02-09T17:15:31.187 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.write_buffer_size: 33554432 2024-02-09T17:15:31.187 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.max_write_buffer_number: 2 2024-02-09T17:15:31.187 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.compression: NoCompression 2024-02-09T17:15:31.187 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.bottommost_compression: Disabled 2024-02-09T17:15:31.187 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.prefix_extractor: nullptr 2024-02-09T17:15:31.187 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-09T17:15:31.187 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.num_levels: 7 2024-02-09T17:15:31.187 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-09T17:15:31.187 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-09T17:15:31.187 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-09T17:15:31.187 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-09T17:15:31.187 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-09T17:15:31.188 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-09T17:15:31.188 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-09T17:15:31.188 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-09T17:15:31.188 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-09T17:15:31.188 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-09T17:15:31.188 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-09T17:15:31.188 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-09T17:15:31.188 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-09T17:15:31.189 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.compression_opts.level: 32767 2024-02-09T17:15:31.189 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.compression_opts.strategy: 0 2024-02-09T17:15:31.189 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-09T17:15:31.189 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-09T17:15:31.189 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-09T17:15:31.190 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-09T17:15:31.190 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.compression_opts.enabled: false 2024-02-09T17:15:31.190 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-09T17:15:31.190 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-09T17:15:31.190 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-09T17:15:31.190 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-09T17:15:31.191 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.target_file_size_base: 67108864 2024-02-09T17:15:31.191 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-09T17:15:31.191 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-09T17:15:31.191 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-09T17:15:31.191 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-09T17:15:31.191 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-09T17:15:31.191 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-09T17:15:31.192 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-09T17:15:31.192 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-09T17:15:31.192 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-09T17:15:31.192 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-09T17:15:31.192 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-09T17:15:31.192 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-09T17:15:31.192 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-09T17:15:31.192 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-09T17:15:31.192 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.arena_block_size: 1048576 2024-02-09T17:15:31.192 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-09T17:15:31.192 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-09T17:15:31.192 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.disable_auto_compactions: 0 2024-02-09T17:15:31.193 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-09T17:15:31.193 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-09T17:15:31.193 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-09T17:15:31.193 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-09T17:15:31.193 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-09T17:15:31.193 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-09T17:15:31.194 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-09T17:15:31.194 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-09T17:15:31.194 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-09T17:15:31.194 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-09T17:15:31.194 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.table_properties_collectors: 2024-02-09T17:15:31.194 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.inplace_update_support: 0 2024-02-09T17:15:31.194 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-09T17:15:31.194 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-09T17:15:31.194 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-09T17:15:31.194 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-09T17:15:31.194 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.bloom_locality: 0 2024-02-09T17:15:31.194 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.max_successive_merges: 0 2024-02-09T17:15:31.194 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-09T17:15:31.195 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.paranoid_file_checks: 0 2024-02-09T17:15:31.195 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.force_consistency_checks: 1 2024-02-09T17:15:31.195 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.report_bg_io_stats: 0 2024-02-09T17:15:31.195 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.ttl: 2592000 2024-02-09T17:15:31.195 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-09T17:15:31.195 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-09T17:15:31.195 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-09T17:15:31.195 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.enable_blob_files: false 2024-02-09T17:15:31.195 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.min_blob_size: 0 2024-02-09T17:15:31.195 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.blob_file_size: 268435456 2024-02-09T17:15:31.195 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-09T17:15:31.195 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-09T17:15:31.195 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-09T17:15:31.195 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-09T17:15:31.195 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-09T17:15:31.196 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.blob_file_starting_level: 0 2024-02-09T17:15:31.196 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-09T17:15:31.196 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-02-09T17:15:31.196 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-02-09T17:15:31.196 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: d77dbedc-86e0-455e-8907-463135f7384c 2024-02-09T17:15:31.196 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707498930889747, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-02-09T17:15:31.196 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-02-09T17:15:31.196 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707498930890700, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 90661, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 294, "table_properties": {"data_size": 88747, "index_size": 260, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 12730, "raw_average_key_size": 50, "raw_value_size": 81815, "raw_average_value_size": 323, "num_data_blocks": 11, "num_entries": 253, "num_filter_entries": 253, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1707498930, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d77dbedc-86e0-455e-8907-463135f7384c", "db_session_id": "USCXM8FHBGTIXSCKVN61", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-02-09T17:15:31.196 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707498930890756, "job": 1, "event": "recovery_finished"} 2024-02-09T17:15:31.196 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-02-09T17:15:31.196 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-09T17:15:31.196 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5634e2990000 2024-02-09T17:15:31.196 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: DB pointer 0x5634e297a000 2024-02-09T17:15:31.196 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-09T17:15:31.197 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-09T17:15:31.197 INFO:journalctl@ceph.mon.a.smithi099.stdout: ** DB Stats ** 2024-02-09T17:15:31.197 INFO:journalctl@ceph.mon.a.smithi099.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-09T17:15:31.197 INFO:journalctl@ceph.mon.a.smithi099.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-09T17:15:31.197 INFO:journalctl@ceph.mon.a.smithi099.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-09T17:15:31.197 INFO:journalctl@ceph.mon.a.smithi099.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-09T17:15:31.197 INFO:journalctl@ceph.mon.a.smithi099.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-09T17:15:31.197 INFO:journalctl@ceph.mon.a.smithi099.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-09T17:15:31.197 INFO:journalctl@ceph.mon.a.smithi099.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-09T17:15:31.197 INFO:journalctl@ceph.mon.a.smithi099.stdout: 2024-02-09T17:15:31.197 INFO:journalctl@ceph.mon.a.smithi099.stdout: ** Compaction Stats [default] ** 2024-02-09T17:15:31.197 INFO:journalctl@ceph.mon.a.smithi099.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-09T17:15:31.197 INFO:journalctl@ceph.mon.a.smithi099.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-09T17:15:31.197 INFO:journalctl@ceph.mon.a.smithi099.stdout: L0 2/0 90.37 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 137.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-09T17:15:31.197 INFO:journalctl@ceph.mon.a.smithi099.stdout: Sum 2/0 90.37 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 137.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-09T17:15:31.197 INFO:journalctl@ceph.mon.a.smithi099.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 137.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-09T17:15:31.198 INFO:journalctl@ceph.mon.a.smithi099.stdout: 2024-02-09T17:15:31.198 INFO:journalctl@ceph.mon.a.smithi099.stdout: ** Compaction Stats [default] ** 2024-02-09T17:15:31.198 INFO:journalctl@ceph.mon.a.smithi099.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-09T17:15:31.198 INFO:journalctl@ceph.mon.a.smithi099.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-09T17:15:31.198 INFO:journalctl@ceph.mon.a.smithi099.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 137.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-09T17:15:31.198 INFO:journalctl@ceph.mon.a.smithi099.stdout: 2024-02-09T17:15:31.198 INFO:journalctl@ceph.mon.a.smithi099.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-09T17:15:31.198 INFO:journalctl@ceph.mon.a.smithi099.stdout: 2024-02-09T17:15:31.198 INFO:journalctl@ceph.mon.a.smithi099.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-09T17:15:31.198 INFO:journalctl@ceph.mon.a.smithi099.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-09T17:15:31.198 INFO:journalctl@ceph.mon.a.smithi099.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-09T17:15:31.198 INFO:journalctl@ceph.mon.a.smithi099.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-09T17:15:31.198 INFO:journalctl@ceph.mon.a.smithi099.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-09T17:15:31.198 INFO:journalctl@ceph.mon.a.smithi099.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-09T17:15:31.198 INFO:journalctl@ceph.mon.a.smithi099.stdout: Cumulative compaction: 0.00 GB write, 29.81 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-09T17:15:31.198 INFO:journalctl@ceph.mon.a.smithi099.stdout: Interval compaction: 0.00 GB write, 29.81 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-09T17:15:31.199 INFO:journalctl@ceph.mon.a.smithi099.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-09T17:15:31.199 INFO:journalctl@ceph.mon.a.smithi099.stdout: Block cache BinnedLRUCache@0x5634e289f090#7 capacity: 512.00 MB usage: 1.30 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-02-09T17:15:31.199 INFO:journalctl@ceph.mon.a.smithi099.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-02-09T17:15:31.199 INFO:journalctl@ceph.mon.a.smithi099.stdout: 2024-02-09T17:15:31.199 INFO:journalctl@ceph.mon.a.smithi099.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-09T17:15:31.199 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: starting mon.a rank 0 at public addrs [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] at bind addrs [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid a69c870a-c76e-11ee-95b7-87774f69a715 2024-02-09T17:15:31.199 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: mon.a@-1(???) e1 preinit fsid a69c870a-c76e-11ee-95b7-87774f69a715 2024-02-09T17:15:31.199 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: mon.a@-1(???).mds e1 new map 2024-02-09T17:15:31.199 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: mon.a@-1(???).mds e1 print_map 2024-02-09T17:15:31.199 INFO:journalctl@ceph.mon.a.smithi099.stdout: e1 2024-02-09T17:15:31.199 INFO:journalctl@ceph.mon.a.smithi099.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-09T17:15:31.199 INFO:journalctl@ceph.mon.a.smithi099.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-02-09T17:15:31.199 INFO:journalctl@ceph.mon.a.smithi099.stdout: legacy client fscid: -1 2024-02-09T17:15:31.199 INFO:journalctl@ceph.mon.a.smithi099.stdout: 2024-02-09T17:15:31.199 INFO:journalctl@ceph.mon.a.smithi099.stdout: No filesystems configured 2024-02-09T17:15:31.199 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-09T17:15:31.200 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-09T17:15:31.200 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-09T17:15:31.200 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-09T17:15:31.200 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-09T17:15:31.200 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: mon.a is new leader, mons a in quorum (ranks 0) 2024-02-09T17:15:31.200 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: monmap e1: 1 mons at {a=[v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-09T17:15:31.200 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: fsmap 2024-02-09T17:15:31.200 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: osdmap e1: 0 total, 0 up, 0 in 2024-02-09T17:15:31.200 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 ceph-mon[39420]: mgrmap e1: no daemons active 2024-02-09T17:15:31.200 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 bash[39382]: a149c71620e59d630208b3c9bdf0b996b821f0870981be2589b58a5b451e7a5d 2024-02-09T17:15:31.200 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:30 smithi099 systemd[1]: Started Ceph mon.a for a69c870a-c76e-11ee-95b7-87774f69a715. 2024-02-09T17:15:32.038 INFO:teuthology.orchestra.run.smithi099.stdout:Wrote config to /etc/ceph/ceph.conf 2024-02-09T17:15:32.060 INFO:teuthology.orchestra.run.smithi099.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-09T17:15:32.060 INFO:teuthology.orchestra.run.smithi099.stdout:Creating mgr... 2024-02-09T17:15:32.063 INFO:teuthology.orchestra.run.smithi099.stdout:Verifying port 0.0.0.0:9283 ... 2024-02-09T17:15:32.064 INFO:teuthology.orchestra.run.smithi099.stdout:Verifying port 0.0.0.0:8765 ... 2024-02-09T17:15:32.260 INFO:teuthology.orchestra.run.smithi099.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-a69c870a-c76e-11ee-95b7-87774f69a715@mgr.y 2024-02-09T17:15:32.261 INFO:teuthology.orchestra.run.smithi099.stdout:systemctl: stderr Failed to reset failed state of unit ceph-a69c870a-c76e-11ee-95b7-87774f69a715@mgr.y.service: Unit ceph-a69c870a-c76e-11ee-95b7-87774f69a715@mgr.y.service not loaded. 2024-02-09T17:15:32.355 INFO:teuthology.orchestra.run.smithi099.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-a69c870a-c76e-11ee-95b7-87774f69a715.target.wants/ceph-a69c870a-c76e-11ee-95b7-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-a69c870a-c76e-11ee-95b7-87774f69a715@.service. 2024-02-09T17:15:32.495 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:32 smithi099 systemd[1]: Starting Ceph mgr.y for a69c870a-c76e-11ee-95b7-87774f69a715... 2024-02-09T17:15:32.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:32 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3629063180' entity='client.admin' 2024-02-09T17:15:35.246 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:34 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:34.858+0000 7ff729d11200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-09T17:15:35.246 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:35 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:35.022+0000 7ff729d11200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-09T17:15:35.661 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:35 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:35.351+0000 7ff729d11200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-09T17:15:35.676 INFO:teuthology.orchestra.run.smithi099.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-09T17:15:35.676 INFO:teuthology.orchestra.run.smithi099.stdout:systemctl: stdout disabled 2024-02-09T17:15:35.684 INFO:teuthology.orchestra.run.smithi099.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-09T17:15:35.684 INFO:teuthology.orchestra.run.smithi099.stdout:systemctl: stdout inactive 2024-02-09T17:15:35.684 INFO:teuthology.orchestra.run.smithi099.stdout:firewalld.service is not enabled 2024-02-09T17:15:35.684 INFO:teuthology.orchestra.run.smithi099.stdout:Not possible to enable service . firewalld.service is not available 2024-02-09T17:15:35.693 INFO:teuthology.orchestra.run.smithi099.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-09T17:15:35.693 INFO:teuthology.orchestra.run.smithi099.stdout:systemctl: stdout disabled 2024-02-09T17:15:35.702 INFO:teuthology.orchestra.run.smithi099.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-09T17:15:35.702 INFO:teuthology.orchestra.run.smithi099.stdout:systemctl: stdout inactive 2024-02-09T17:15:35.702 INFO:teuthology.orchestra.run.smithi099.stdout:firewalld.service is not enabled 2024-02-09T17:15:35.702 INFO:teuthology.orchestra.run.smithi099.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-02-09T17:15:35.702 INFO:teuthology.orchestra.run.smithi099.stdout:Waiting for mgr to start... 2024-02-09T17:15:35.702 INFO:teuthology.orchestra.run.smithi099.stdout:Waiting for mgr... 2024-02-09T17:15:35.995 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:35 smithi099 bash[39657]: 024c5520a33e1f2a0257e02eea3e21b757e0edc2b4cda69263f9c6b7e17bb73f 2024-02-09T17:15:35.995 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:35 smithi099 systemd[1]: Started Ceph mgr.y for a69c870a-c76e-11ee-95b7-87774f69a715. 2024-02-09T17:15:35.995 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:35 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:35.873+0000 7ff729d11200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-09T17:15:36.495 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:36 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:36.116+0000 7ff729d11200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-09T17:15:36.995 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:36 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:36.547+0000 7ff729d11200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-09T17:15:37.745 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:37 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:37.258+0000 7ff729d11200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-09T17:15:37.745 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:37 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:37.344+0000 7ff729d11200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-09T17:15:38.127 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:37 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:37.945+0000 7ff729d11200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-09T17:15:38.127 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:38 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:38.032+0000 7ff729d11200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-09T17:15:38.127 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:38 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:38.122+0000 7ff729d11200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-09T17:15:38.442 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:38 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:38.346+0000 7ff729d11200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-09T17:15:38.745 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:38 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:38.438+0000 7ff729d11200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-09T17:15:38.745 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:38 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:38.530+0000 7ff729d11200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-09T17:15:39.026 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:38 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:38.759+0000 7ff729d11200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-09T17:15:39.027 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:38 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:38.858+0000 7ff729d11200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-09T17:15:39.301 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:39 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1411536879' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-09T17:15:39.302 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:39 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:39.050+0000 7ff729d11200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-09T17:15:39.745 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:39 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:39.298+0000 7ff729d11200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-09T17:15:40.245 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:39 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:39.878+0000 7ff729d11200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-09T17:15:40.245 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:40 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:40.086+0000 7ff729d11200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-09T17:15:40.745 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:40 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:40.279+0000 7ff729d11200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-09T17:15:41.245 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:40 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:40.930+0000 7ff729d11200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-09T17:15:41.245 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:41 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:41.126+0000 7ff729d11200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-09T17:15:41.742 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout 2024-02-09T17:15:41.742 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout { 2024-02-09T17:15:41.742 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "fsid": "a69c870a-c76e-11ee-95b7-87774f69a715", 2024-02-09T17:15:41.742 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "health": { 2024-02-09T17:15:41.743 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-09T17:15:41.743 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-09T17:15:41.743 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-09T17:15:41.743 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout }, 2024-02-09T17:15:41.743 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-09T17:15:41.743 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-09T17:15:41.743 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout 0 2024-02-09T17:15:41.743 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout ], 2024-02-09T17:15:41.743 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-09T17:15:41.743 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "a" 2024-02-09T17:15:41.743 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout ], 2024-02-09T17:15:41.743 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-02-09T17:15:41.743 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-09T17:15:41.743 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-09T17:15:41.744 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-02-09T17:15:41.744 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-09T17:15:41.744 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout }, 2024-02-09T17:15:41.744 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-09T17:15:41.744 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-09T17:15:41.744 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-09T17:15:41.744 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-09T17:15:41.744 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-09T17:15:41.744 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-09T17:15:41.744 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-09T17:15:41.744 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-09T17:15:41.744 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout }, 2024-02-09T17:15:41.744 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-09T17:15:41.744 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-09T17:15:41.745 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-09T17:15:41.745 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-09T17:15:41.745 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-09T17:15:41.745 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-09T17:15:41.745 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-09T17:15:41.745 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-09T17:15:41.745 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-09T17:15:41.745 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout }, 2024-02-09T17:15:41.745 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-09T17:15:41.745 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-09T17:15:41.745 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-09T17:15:41.745 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-09T17:15:41.745 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout }, 2024-02-09T17:15:41.745 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-09T17:15:41.745 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-09T17:15:41.746 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-09T17:15:41.746 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-09T17:15:41.746 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-09T17:15:41.746 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-09T17:15:41.746 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "restful" 2024-02-09T17:15:41.746 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout ], 2024-02-09T17:15:41.746 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-09T17:15:41.746 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout }, 2024-02-09T17:15:41.746 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-09T17:15:41.746 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-09T17:15:41.746 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "modified": "2024-02-09T17:15:26.114056+0000", 2024-02-09T17:15:41.746 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-09T17:15:41.746 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout }, 2024-02-09T17:15:41.746 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-09T17:15:41.746 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout } 2024-02-09T17:15:41.746 INFO:teuthology.orchestra.run.smithi099.stdout:mgr not available, waiting (1/15)... 2024-02-09T17:15:41.747 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:41 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:41.308+0000 7ff729d11200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-09T17:15:41.747 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:41 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:41.424+0000 7ff729d11200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-09T17:15:41.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:41 smithi099 ceph-mon[39420]: Activating manager daemon y 2024-02-09T17:15:41.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:41 smithi099 ceph-mon[39420]: mgrmap e2: y(active, starting, since 0.0028651s) 2024-02-09T17:15:41.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:41 smithi099 ceph-mon[39420]: from='mgr.14100 172.21.15.99:0/3048245288' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-09T17:15:41.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:41 smithi099 ceph-mon[39420]: from='mgr.14100 172.21.15.99:0/3048245288' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-09T17:15:41.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:41 smithi099 ceph-mon[39420]: from='mgr.14100 172.21.15.99:0/3048245288' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-09T17:15:41.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:41 smithi099 ceph-mon[39420]: from='mgr.14100 172.21.15.99:0/3048245288' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:15:41.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:41 smithi099 ceph-mon[39420]: from='mgr.14100 172.21.15.99:0/3048245288' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-09T17:15:41.748 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:41 smithi099 ceph-mon[39420]: Manager daemon y is now available 2024-02-09T17:15:41.748 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:41 smithi099 ceph-mon[39420]: from='mgr.14100 172.21.15.99:0/3048245288' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-09T17:15:41.748 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:41 smithi099 ceph-mon[39420]: from='mgr.14100 172.21.15.99:0/3048245288' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-09T17:15:41.748 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:41 smithi099 ceph-mon[39420]: from='mgr.14100 172.21.15.99:0/3048245288' entity='mgr.y' 2024-02-09T17:15:41.748 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:41 smithi099 ceph-mon[39420]: from='mgr.14100 172.21.15.99:0/3048245288' entity='mgr.y' 2024-02-09T17:15:41.748 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:41 smithi099 ceph-mon[39420]: from='mgr.14100 172.21.15.99:0/3048245288' entity='mgr.y' 2024-02-09T17:15:43.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:43 smithi099 ceph-mon[39420]: mgrmap e3: y(active, since 1.00721s) 2024-02-09T17:15:45.132 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout 2024-02-09T17:15:45.133 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout { 2024-02-09T17:15:45.133 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "fsid": "a69c870a-c76e-11ee-95b7-87774f69a715", 2024-02-09T17:15:45.133 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "health": { 2024-02-09T17:15:45.133 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-09T17:15:45.133 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-09T17:15:45.133 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-09T17:15:45.133 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout }, 2024-02-09T17:15:45.133 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-09T17:15:45.133 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-09T17:15:45.133 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout 0 2024-02-09T17:15:45.133 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout ], 2024-02-09T17:15:45.133 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-09T17:15:45.134 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "a" 2024-02-09T17:15:45.134 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout ], 2024-02-09T17:15:45.134 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "quorum_age": 13, 2024-02-09T17:15:45.134 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-09T17:15:45.134 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-09T17:15:45.134 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-02-09T17:15:45.134 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-09T17:15:45.134 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout }, 2024-02-09T17:15:45.134 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-09T17:15:45.134 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-09T17:15:45.134 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-09T17:15:45.134 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-09T17:15:45.134 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-09T17:15:45.134 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-09T17:15:45.134 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-09T17:15:45.135 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-09T17:15:45.135 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout }, 2024-02-09T17:15:45.135 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-09T17:15:45.135 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-09T17:15:45.135 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-09T17:15:45.135 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-09T17:15:45.135 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-09T17:15:45.135 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-09T17:15:45.135 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-09T17:15:45.135 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-09T17:15:45.135 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-09T17:15:45.135 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout }, 2024-02-09T17:15:45.135 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-09T17:15:45.135 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-09T17:15:45.136 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-09T17:15:45.136 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-09T17:15:45.136 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout }, 2024-02-09T17:15:45.136 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-09T17:15:45.136 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-09T17:15:45.136 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-09T17:15:45.136 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-09T17:15:45.136 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-09T17:15:45.136 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-09T17:15:45.136 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "restful" 2024-02-09T17:15:45.136 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout ], 2024-02-09T17:15:45.136 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-09T17:15:45.136 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout }, 2024-02-09T17:15:45.136 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-09T17:15:45.136 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-09T17:15:45.137 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "modified": "2024-02-09T17:15:26.114056+0000", 2024-02-09T17:15:45.137 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-09T17:15:45.137 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout }, 2024-02-09T17:15:45.137 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-09T17:15:45.137 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout } 2024-02-09T17:15:45.137 INFO:teuthology.orchestra.run.smithi099.stdout:mgr is available 2024-02-09T17:15:45.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:45 smithi099 ceph-mon[39420]: mgrmap e4: y(active, since 3s) 2024-02-09T17:15:45.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:45 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3584638585' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-09T17:15:46.635 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout 2024-02-09T17:15:46.635 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout [global] 2024-02-09T17:15:46.635 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout fsid = a69c870a-c76e-11ee-95b7-87774f69a715 2024-02-09T17:15:46.635 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-02-09T17:15:46.635 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-02-09T17:15:46.635 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-02-09T17:15:46.635 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout ms_tcp_read_timeout = 5 2024-02-09T17:15:46.635 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-02-09T17:15:46.636 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout 2024-02-09T17:15:46.636 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout [mgr] 2024-02-09T17:15:46.636 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-02-09T17:15:46.636 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout 2024-02-09T17:15:46.636 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout [osd] 2024-02-09T17:15:46.636 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-02-09T17:15:46.636 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-02-09T17:15:46.636 INFO:teuthology.orchestra.run.smithi099.stdout:Enabling cephadm module... 2024-02-09T17:15:46.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:46 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2137107950' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-09T17:15:47.987 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:47 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3787930098' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-09T17:15:47.988 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:47 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: ignoring --setuser ceph since I am not root 2024-02-09T17:15:47.988 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:47 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: ignoring --setgroup ceph since I am not root 2024-02-09T17:15:47.988 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:47 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:47.831+0000 7f050053d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-09T17:15:48.242 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:47 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:47.994+0000 7f050053d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-09T17:15:48.495 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:48 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:48.329+0000 7f050053d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-09T17:15:48.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:48 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3787930098' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-09T17:15:48.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:48 smithi099 ceph-mon[39420]: mgrmap e5: y(active, since 6s) 2024-02-09T17:15:48.996 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:48 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:48.836+0000 7f050053d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-09T17:15:49.410 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:49 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:49.081+0000 7f050053d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-09T17:15:49.675 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:49 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:49.516+0000 7f050053d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-09T17:15:49.723 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout { 2024-02-09T17:15:49.723 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-02-09T17:15:49.723 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-09T17:15:49.724 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-02-09T17:15:49.724 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-02-09T17:15:49.724 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout } 2024-02-09T17:15:49.724 INFO:teuthology.orchestra.run.smithi099.stdout:Waiting for the mgr to restart... 2024-02-09T17:15:49.724 INFO:teuthology.orchestra.run.smithi099.stdout:Waiting for mgr epoch 5... 2024-02-09T17:15:49.926 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:49 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2787592108' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-09T17:15:50.745 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:50 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:50.258+0000 7f050053d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-09T17:15:50.745 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:50 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:50.345+0000 7f050053d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-09T17:15:51.245 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:50 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:50.931+0000 7f050053d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-09T17:15:51.245 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:51 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:51.015+0000 7f050053d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-09T17:15:51.245 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:51 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:51.102+0000 7f050053d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-09T17:15:51.745 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:51 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:51.334+0000 7f050053d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-09T17:15:51.745 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:51 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:51.425+0000 7f050053d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-09T17:15:51.745 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:51 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:51.518+0000 7f050053d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-09T17:15:52.029 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:51 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:51.745+0000 7f050053d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-09T17:15:52.029 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:51 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:51.842+0000 7f050053d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-09T17:15:52.495 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:52 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:52.025+0000 7f050053d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-09T17:15:52.495 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:52 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:52.272+0000 7f050053d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-09T17:15:53.245 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:52 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:52.847+0000 7f050053d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-09T17:15:53.245 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:53 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:53.054+0000 7f050053d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-09T17:15:53.745 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:53 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:53.246+0000 7f050053d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-09T17:15:54.245 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:53 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:53.895+0000 7f050053d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-09T17:15:54.245 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:54 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:54.091+0000 7f050053d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-09T17:15:54.745 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:54 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:54.271+0000 7f050053d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-09T17:15:54.746 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:54 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:15:54.385+0000 7f050053d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-09T17:15:54.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:54 smithi099 ceph-mon[39420]: Active manager daemon y restarted 2024-02-09T17:15:54.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:54 smithi099 ceph-mon[39420]: Activating manager daemon y 2024-02-09T17:15:54.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:54 smithi099 ceph-mon[39420]: osdmap e2: 0 total, 0 up, 0 in 2024-02-09T17:15:54.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:54 smithi099 ceph-mon[39420]: mgrmap e6: y(active, starting, since 0.00582922s) 2024-02-09T17:15:54.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:54 smithi099 ceph-mon[39420]: from='mgr.14116 172.21.15.99:0/2399482477' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:15:54.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:54 smithi099 ceph-mon[39420]: from='mgr.14116 172.21.15.99:0/2399482477' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-09T17:15:54.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:54 smithi099 ceph-mon[39420]: from='mgr.14116 172.21.15.99:0/2399482477' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-09T17:15:54.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:54 smithi099 ceph-mon[39420]: from='mgr.14116 172.21.15.99:0/2399482477' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-09T17:15:54.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:54 smithi099 ceph-mon[39420]: from='mgr.14116 172.21.15.99:0/2399482477' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-09T17:15:54.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:54 smithi099 ceph-mon[39420]: Manager daemon y is now available 2024-02-09T17:15:54.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:54 smithi099 ceph-mon[39420]: from='mgr.14116 172.21.15.99:0/2399482477' entity='mgr.y' 2024-02-09T17:15:54.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:54 smithi099 ceph-mon[39420]: from='mgr.14116 172.21.15.99:0/2399482477' entity='mgr.y' 2024-02-09T17:15:55.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:55 smithi099 ceph-mon[39420]: Found migration_current of "None". Setting to last migration. 2024-02-09T17:15:55.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:55 smithi099 ceph-mon[39420]: from='mgr.14116 172.21.15.99:0/2399482477' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:15:55.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:55 smithi099 ceph-mon[39420]: from='mgr.14116 172.21.15.99:0/2399482477' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:15:55.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:55 smithi099 ceph-mon[39420]: from='mgr.14116 172.21.15.99:0/2399482477' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:15:55.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:55 smithi099 ceph-mon[39420]: from='mgr.14116 172.21.15.99:0/2399482477' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-09T17:15:55.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:55 smithi099 ceph-mon[39420]: from='mgr.14116 172.21.15.99:0/2399482477' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-09T17:15:55.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:55 smithi099 ceph-mon[39420]: from='mgr.14116 172.21.15.99:0/2399482477' entity='mgr.y' 2024-02-09T17:15:55.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:55 smithi099 ceph-mon[39420]: from='mgr.14116 172.21.15.99:0/2399482477' entity='mgr.y' 2024-02-09T17:15:55.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:55 smithi099 ceph-mon[39420]: mgrmap e7: y(active, since 1.00564s) 2024-02-09T17:15:56.064 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout { 2024-02-09T17:15:56.064 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-02-09T17:15:56.064 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "initialized": true 2024-02-09T17:15:56.064 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout } 2024-02-09T17:15:56.064 INFO:teuthology.orchestra.run.smithi099.stdout:mgr epoch 5 is available 2024-02-09T17:15:56.064 INFO:teuthology.orchestra.run.smithi099.stdout:Setting orchestrator backend to cephadm... 2024-02-09T17:15:56.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:56 smithi099 ceph-mon[39420]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-09T17:15:56.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:56 smithi099 ceph-mon[39420]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-09T17:15:56.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:56 smithi099 ceph-mon[39420]: [09/Feb/2024:17:15:55] ENGINE Bus STARTING 2024-02-09T17:15:56.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:56 smithi099 ceph-mon[39420]: [09/Feb/2024:17:15:55] ENGINE Serving on http://172.21.15.99:8765 2024-02-09T17:15:56.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:56 smithi099 ceph-mon[39420]: [09/Feb/2024:17:15:55] ENGINE Serving on https://172.21.15.99:7150 2024-02-09T17:15:56.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:56 smithi099 ceph-mon[39420]: [09/Feb/2024:17:15:55] ENGINE Bus STARTED 2024-02-09T17:15:56.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:56 smithi099 ceph-mon[39420]: from='mgr.14116 172.21.15.99:0/2399482477' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:15:57.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:57 smithi099 ceph-mon[39420]: mgrmap e8: y(active, since 2s) 2024-02-09T17:15:57.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:57 smithi099 ceph-mon[39420]: from='mgr.14116 172.21.15.99:0/2399482477' entity='mgr.y' 2024-02-09T17:15:57.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:57 smithi099 ceph-mon[39420]: from='mgr.14116 172.21.15.99:0/2399482477' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:15:58.701 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout value unchanged 2024-02-09T17:15:58.701 INFO:teuthology.orchestra.run.smithi099.stdout:Generating ssh key... 2024-02-09T17:15:58.703 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:58 smithi099 ceph-mon[39420]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:15:59.733 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:15:59 smithi099 ceph-mon[39420]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:15:59.733 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:59 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: Generating public/private rsa key pair. 2024-02-09T17:15:59.734 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:59 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: Your identification has been saved in /tmp/tmp2jx4hn83/key. 2024-02-09T17:15:59.734 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:59 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: Your public key has been saved in /tmp/tmp2jx4hn83/key.pub. 2024-02-09T17:15:59.734 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:59 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: The key fingerprint is: 2024-02-09T17:15:59.734 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:59 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: SHA256:Qe2DGHnAl1laJrsuyTSaG764gv3Nm74hqGKtPCNguMI ceph-a69c870a-c76e-11ee-95b7-87774f69a715 2024-02-09T17:15:59.734 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:59 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: The key's randomart image is: 2024-02-09T17:15:59.734 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:59 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: +---[RSA 3072]----+ 2024-02-09T17:15:59.734 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:59 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: | ..oo=+ | 2024-02-09T17:15:59.734 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:59 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: | +.=*. | 2024-02-09T17:15:59.734 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:59 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: | =+o | 2024-02-09T17:15:59.734 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:59 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: | . .oo | 2024-02-09T17:15:59.734 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:59 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: |. o S . | 2024-02-09T17:15:59.734 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:59 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: |o. . = + | 2024-02-09T17:15:59.734 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:59 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: |=oo = = . | 2024-02-09T17:15:59.734 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:59 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: |OE.+ * + | 2024-02-09T17:15:59.735 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:59 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: |*+*o=oBo | 2024-02-09T17:15:59.735 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:15:59 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: +----[SHA256]-----+ 2024-02-09T17:16:00.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:00 smithi099 ceph-mon[39420]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:16:00.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:00 smithi099 ceph-mon[39420]: Generating ssh key... 2024-02-09T17:16:00.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:00 smithi099 ceph-mon[39420]: from='mgr.14116 172.21.15.99:0/2399482477' entity='mgr.y' 2024-02-09T17:16:00.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:00 smithi099 ceph-mon[39420]: from='mgr.14116 172.21.15.99:0/2399482477' entity='mgr.y' 2024-02-09T17:16:01.556 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSoO3S4tVb8t72oVO73Khx8s6h9Snxd09pp/GE/jxRJBhnG7z0LYypbaHuorVvLUoM5AiyPRF3qj8FiVst1jSkMUqcJ4iZO9pPxEe2FlC5aBVGesDe1jjLAP5JEtT3uuK3BUSj+t9JJtCUpzIEcFoBPMSzX4lFOjpfvdRlNO0ku25oZ6Ekr/kS+/SR0X/K5Zc6R1/4+B4l/YxvBpy0xtTgCcDwp1SOtPzZFJ8y/iUqBie28XPETCz7vcQOriFLsGyDh5vHIDE9Xi0Q+Bhdj1czfPBeBMh4/g4V2lv+hTwES+RNGyf07C3D97dkd55yByBjUu21VIAWxfjFzzTtWzgP+TmHkR4K5JtYQFMs+KGsQsOIeEhH/PcHs3KRoYFQZPXoLhzj6ptNBxrTMK0//ewZaxMGBcAaaLX+8M26jZu/pSg8JOZap6/wBxG/OebuaQOV47nkhH2QEYSkXpNJFZ73owkhNNOdaPij95Nv7CECB83Ad9n/Co4oEKnxSaRrndc= ceph-a69c870a-c76e-11ee-95b7-87774f69a715 2024-02-09T17:16:01.557 INFO:teuthology.orchestra.run.smithi099.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-09T17:16:01.557 INFO:teuthology.orchestra.run.smithi099.stdout:Adding key to root@localhost authorized_keys... 2024-02-09T17:16:01.557 INFO:teuthology.orchestra.run.smithi099.stdout:Adding host smithi099... 2024-02-09T17:16:02.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:01 smithi099 ceph-mon[39420]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:16:02.990 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:02 smithi099 ceph-mon[39420]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi099", "addr": "172.21.15.99", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:16:04.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:03 smithi099 ceph-mon[39420]: Deploying cephadm binary to smithi099 2024-02-09T17:16:06.665 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout Added host 'smithi099' with addr '172.21.15.99' 2024-02-09T17:16:06.665 INFO:teuthology.orchestra.run.smithi099.stdout:Deploying unmanaged mon service... 2024-02-09T17:16:07.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:07 smithi099 ceph-mon[39420]: from='mgr.14116 172.21.15.99:0/2399482477' entity='mgr.y' 2024-02-09T17:16:07.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:07 smithi099 ceph-mon[39420]: Added host smithi099 2024-02-09T17:16:07.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:07 smithi099 ceph-mon[39420]: from='mgr.14116 172.21.15.99:0/2399482477' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:16:09.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:08 smithi099 ceph-mon[39420]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:16:09.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:08 smithi099 ceph-mon[39420]: Saving service mon spec with placement count:5 2024-02-09T17:16:09.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:08 smithi099 ceph-mon[39420]: from='mgr.14116 172.21.15.99:0/2399482477' entity='mgr.y' 2024-02-09T17:16:10.700 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-02-09T17:16:10.700 INFO:teuthology.orchestra.run.smithi099.stdout:Deploying unmanaged mgr service... 2024-02-09T17:16:11.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:11 smithi099 ceph-mon[39420]: from='mgr.14116 172.21.15.99:0/2399482477' entity='mgr.y' 2024-02-09T17:16:12.023 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:11 smithi099 ceph-mon[39420]: from='mgr.14116 172.21.15.99:0/2399482477' entity='mgr.y' 2024-02-09T17:16:12.023 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:11 smithi099 ceph-mon[39420]: from='mgr.14116 172.21.15.99:0/2399482477' entity='mgr.y' 2024-02-09T17:16:12.417 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-02-09T17:16:12.671 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:12 smithi099 ceph-mon[39420]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:16:12.671 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:12 smithi099 ceph-mon[39420]: Saving service mgr spec with placement count:2 2024-02-09T17:16:14.280 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:14 smithi099 ceph-mon[39420]: from='mgr.14116 172.21.15.99:0/2399482477' entity='mgr.y' 2024-02-09T17:16:14.280 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:14 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1012436447' entity='client.admin' 2024-02-09T17:16:14.280 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:14 smithi099 ceph-mon[39420]: from='mgr.14116 172.21.15.99:0/2399482477' entity='mgr.y' 2024-02-09T17:16:15.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:15 smithi099 ceph-mon[39420]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:16:15.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:15 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3392812033' entity='client.admin' 2024-02-09T17:16:16.103 INFO:teuthology.orchestra.run.smithi099.stdout:Enabling the dashboard module... 2024-02-09T17:16:17.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:17 smithi099 ceph-mon[39420]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:16:17.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:17 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/327179164' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-09T17:16:17.746 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:16:17 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: ignoring --setuser ceph since I am not root 2024-02-09T17:16:17.746 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:16:17 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: ignoring --setgroup ceph since I am not root 2024-02-09T17:16:17.746 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:16:17 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:16:17.600+0000 7f51a74a2200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-09T17:16:18.073 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:16:17 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:16:17.766+0000 7f51a74a2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-09T17:16:18.331 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:16:18 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:16:18.098+0000 7f51a74a2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-09T17:16:18.745 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:16:18 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:16:18.608+0000 7f51a74a2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-09T17:16:18.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:18 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/327179164' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-09T17:16:18.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:18 smithi099 ceph-mon[39420]: mgrmap e9: y(active, since 23s) 2024-02-09T17:16:19.245 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:16:18 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:16:18.857+0000 7f51a74a2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-09T17:16:19.662 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout { 2024-02-09T17:16:19.662 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-02-09T17:16:19.662 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-09T17:16:19.662 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-02-09T17:16:19.662 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-02-09T17:16:19.662 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout } 2024-02-09T17:16:19.662 INFO:teuthology.orchestra.run.smithi099.stdout:Waiting for the mgr to restart... 2024-02-09T17:16:19.662 INFO:teuthology.orchestra.run.smithi099.stdout:Waiting for mgr epoch 9... 2024-02-09T17:16:19.664 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:19 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2897736840' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-09T17:16:19.664 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:16:19 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:16:19.308+0000 7f51a74a2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-09T17:16:20.495 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:16:20 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:16:20.045+0000 7f51a74a2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-09T17:16:20.495 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:16:20 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:16:20.133+0000 7f51a74a2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-09T17:16:20.995 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:16:20 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:16:20.734+0000 7f51a74a2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-09T17:16:20.995 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:16:20 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:16:20.819+0000 7f51a74a2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-09T17:16:20.995 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:16:20 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:16:20.907+0000 7f51a74a2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-09T17:16:21.495 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:16:21 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:16:21.132+0000 7f51a74a2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-09T17:16:21.495 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:16:21 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:16:21.224+0000 7f51a74a2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-09T17:16:21.495 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:16:21 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:16:21.316+0000 7f51a74a2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-09T17:16:21.833 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:16:21 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:16:21.546+0000 7f51a74a2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-09T17:16:21.833 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:16:21 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:16:21.643+0000 7f51a74a2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-09T17:16:22.245 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:16:21 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:16:21.829+0000 7f51a74a2200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-09T17:16:22.245 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:16:22 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:16:22.078+0000 7f51a74a2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-09T17:16:22.995 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:16:22 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:16:22.657+0000 7f51a74a2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-09T17:16:22.995 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:16:22 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:16:22.869+0000 7f51a74a2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-09T17:16:23.495 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:16:23 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:16:23.063+0000 7f51a74a2200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-09T17:16:23.995 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:16:23 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:16:23.719+0000 7f51a74a2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-09T17:16:23.995 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:16:23 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:16:23.918+0000 7f51a74a2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-09T17:16:24.495 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:16:24 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:16:24.099+0000 7f51a74a2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-09T17:16:24.495 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:16:24 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:16:24.214+0000 7f51a74a2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-09T17:16:24.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:24 smithi099 ceph-mon[39420]: Active manager daemon y restarted 2024-02-09T17:16:24.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:24 smithi099 ceph-mon[39420]: Activating manager daemon y 2024-02-09T17:16:24.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:24 smithi099 ceph-mon[39420]: osdmap e3: 0 total, 0 up, 0 in 2024-02-09T17:16:24.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:24 smithi099 ceph-mon[39420]: mgrmap e10: y(active, starting, since 0.0031939s) 2024-02-09T17:16:24.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:24 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:16:24.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:24 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-09T17:16:24.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:24 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-09T17:16:24.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:24 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-09T17:16:24.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:24 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-09T17:16:24.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:24 smithi099 ceph-mon[39420]: Manager daemon y is now available 2024-02-09T17:16:24.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:24 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:16:24.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:24 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:16:25.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:25 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-09T17:16:25.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:25 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-09T17:16:25.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:25 smithi099 ceph-mon[39420]: mgrmap e11: y(active, since 1.00545s) 2024-02-09T17:16:26.022 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout { 2024-02-09T17:16:26.022 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-02-09T17:16:26.022 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout "initialized": true 2024-02-09T17:16:26.022 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout } 2024-02-09T17:16:26.022 INFO:teuthology.orchestra.run.smithi099.stdout:mgr epoch 9 is available 2024-02-09T17:16:26.022 INFO:teuthology.orchestra.run.smithi099.stdout:Generating a dashboard self-signed certificate... 2024-02-09T17:16:26.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:26 smithi099 ceph-mon[39420]: [09/Feb/2024:17:16:24] ENGINE Bus STARTING 2024-02-09T17:16:26.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:26 smithi099 ceph-mon[39420]: [09/Feb/2024:17:16:25] ENGINE Serving on https://172.21.15.99:7150 2024-02-09T17:16:26.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:26 smithi099 ceph-mon[39420]: [09/Feb/2024:17:16:25] ENGINE Serving on http://172.21.15.99:8765 2024-02-09T17:16:26.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:26 smithi099 ceph-mon[39420]: [09/Feb/2024:17:16:25] ENGINE Bus STARTED 2024-02-09T17:16:26.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:26 smithi099 ceph-mon[39420]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-09T17:16:26.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:26 smithi099 ceph-mon[39420]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-09T17:16:26.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:26 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:16:27.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:27 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:16:27.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:27 smithi099 ceph-mon[39420]: mgrmap e12: y(active, since 2s) 2024-02-09T17:16:27.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:27 smithi099 ceph-mon[39420]: from='client.14160 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:16:27.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:27 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:16:27.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:27 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:16:28.476 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-02-09T17:16:28.476 INFO:teuthology.orchestra.run.smithi099.stdout:Creating initial admin user... 2024-02-09T17:16:30.100 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$HdioYsoVT/B2LSMWCqAh../Q2wcgl1BQbBk2.mqZNkkEy9jcezMAO", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1707498989, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-09T17:16:30.100 INFO:teuthology.orchestra.run.smithi099.stdout:Fetching dashboard port number... 2024-02-09T17:16:30.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:30 smithi099 ceph-mon[39420]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:16:30.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:30 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:16:31.391 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stdout 8443 2024-02-09T17:16:31.400 INFO:teuthology.orchestra.run.smithi099.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-09T17:16:31.400 INFO:teuthology.orchestra.run.smithi099.stdout:systemctl: stdout disabled 2024-02-09T17:16:31.408 INFO:teuthology.orchestra.run.smithi099.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-09T17:16:31.408 INFO:teuthology.orchestra.run.smithi099.stdout:systemctl: stdout inactive 2024-02-09T17:16:31.408 INFO:teuthology.orchestra.run.smithi099.stdout:firewalld.service is not enabled 2024-02-09T17:16:31.408 INFO:teuthology.orchestra.run.smithi099.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-09T17:16:31.409 INFO:teuthology.orchestra.run.smithi099.stdout:Ceph Dashboard is now available at: 2024-02-09T17:16:31.409 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:16:31.409 INFO:teuthology.orchestra.run.smithi099.stdout: URL: https://smithi099.front.sepia.ceph.com:8443/ 2024-02-09T17:16:31.410 INFO:teuthology.orchestra.run.smithi099.stdout: User: admin 2024-02-09T17:16:31.410 INFO:teuthology.orchestra.run.smithi099.stdout: Password: 1cvtts5gys 2024-02-09T17:16:31.410 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:16:31.410 INFO:teuthology.orchestra.run.smithi099.stdout:Saving cluster configuration to /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/config directory 2024-02-09T17:16:31.410 INFO:teuthology.orchestra.run.smithi099.stdout:Enabling autotune for osd_memory_target 2024-02-09T17:16:31.661 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:31 smithi099 ceph-mon[39420]: mgrmap e13: y(active, since 6s) 2024-02-09T17:16:31.661 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:31 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/636472409' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-09T17:16:34.186 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-02-09T17:16:34.187 INFO:teuthology.orchestra.run.smithi099.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-02-09T17:16:34.187 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:16:34.187 INFO:teuthology.orchestra.run.smithi099.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-09T17:16:34.187 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:16:34.187 INFO:teuthology.orchestra.run.smithi099.stdout:Or, if you are only running a single cluster on this host: 2024-02-09T17:16:34.187 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:16:34.187 INFO:teuthology.orchestra.run.smithi099.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-02-09T17:16:34.187 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:16:34.187 INFO:teuthology.orchestra.run.smithi099.stdout:Please consider enabling telemetry to help improve Ceph: 2024-02-09T17:16:34.187 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:16:34.187 INFO:teuthology.orchestra.run.smithi099.stdout: ceph telemetry on 2024-02-09T17:16:34.187 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:16:34.187 INFO:teuthology.orchestra.run.smithi099.stdout:For more information see: 2024-02-09T17:16:34.187 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:16:34.188 INFO:teuthology.orchestra.run.smithi099.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-02-09T17:16:34.188 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:16:34.188 INFO:teuthology.orchestra.run.smithi099.stdout:Bootstrap complete. 2024-02-09T17:16:34.232 INFO:tasks.cephadm:Fetching config... 2024-02-09T17:16:34.232 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-02-09T17:16:34.233 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-09T17:16:34.250 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-09T17:16:34.250 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-02-09T17:16:34.250 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-09T17:16:34.308 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-09T17:16:34.308 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-02-09T17:16:34.308 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/keyring of=/dev/stdout 2024-02-09T17:16:34.376 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-09T17:16:34.376 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-02-09T17:16:34.376 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-09T17:16:34.435 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-09T17:16:34.435 DEBUG:teuthology.orchestra.run.smithi099:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSoO3S4tVb8t72oVO73Khx8s6h9Snxd09pp/GE/jxRJBhnG7z0LYypbaHuorVvLUoM5AiyPRF3qj8FiVst1jSkMUqcJ4iZO9pPxEe2FlC5aBVGesDe1jjLAP5JEtT3uuK3BUSj+t9JJtCUpzIEcFoBPMSzX4lFOjpfvdRlNO0ku25oZ6Ekr/kS+/SR0X/K5Zc6R1/4+B4l/YxvBpy0xtTgCcDwp1SOtPzZFJ8y/iUqBie28XPETCz7vcQOriFLsGyDh5vHIDE9Xi0Q+Bhdj1czfPBeBMh4/g4V2lv+hTwES+RNGyf07C3D97dkd55yByBjUu21VIAWxfjFzzTtWzgP+TmHkR4K5JtYQFMs+KGsQsOIeEhH/PcHs3KRoYFQZPXoLhzj6ptNBxrTMK0//ewZaxMGBcAaaLX+8M26jZu/pSg8JOZap6/wBxG/OebuaQOV47nkhH2QEYSkXpNJFZ73owkhNNOdaPij95Nv7CECB83Ad9n/Co4oEKnxSaRrndc= ceph-a69c870a-c76e-11ee-95b7-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-09T17:16:34.511 INFO:teuthology.orchestra.run.smithi099.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSoO3S4tVb8t72oVO73Khx8s6h9Snxd09pp/GE/jxRJBhnG7z0LYypbaHuorVvLUoM5AiyPRF3qj8FiVst1jSkMUqcJ4iZO9pPxEe2FlC5aBVGesDe1jjLAP5JEtT3uuK3BUSj+t9JJtCUpzIEcFoBPMSzX4lFOjpfvdRlNO0ku25oZ6Ekr/kS+/SR0X/K5Zc6R1/4+B4l/YxvBpy0xtTgCcDwp1SOtPzZFJ8y/iUqBie28XPETCz7vcQOriFLsGyDh5vHIDE9Xi0Q+Bhdj1czfPBeBMh4/g4V2lv+hTwES+RNGyf07C3D97dkd55yByBjUu21VIAWxfjFzzTtWzgP+TmHkR4K5JtYQFMs+KGsQsOIeEhH/PcHs3KRoYFQZPXoLhzj6ptNBxrTMK0//ewZaxMGBcAaaLX+8M26jZu/pSg8JOZap6/wBxG/OebuaQOV47nkhH2QEYSkXpNJFZ73owkhNNOdaPij95Nv7CECB83Ad9n/Co4oEKnxSaRrndc= ceph-a69c870a-c76e-11ee-95b7-87774f69a715 2024-02-09T17:16:34.523 DEBUG:teuthology.orchestra.run.smithi162:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSoO3S4tVb8t72oVO73Khx8s6h9Snxd09pp/GE/jxRJBhnG7z0LYypbaHuorVvLUoM5AiyPRF3qj8FiVst1jSkMUqcJ4iZO9pPxEe2FlC5aBVGesDe1jjLAP5JEtT3uuK3BUSj+t9JJtCUpzIEcFoBPMSzX4lFOjpfvdRlNO0ku25oZ6Ekr/kS+/SR0X/K5Zc6R1/4+B4l/YxvBpy0xtTgCcDwp1SOtPzZFJ8y/iUqBie28XPETCz7vcQOriFLsGyDh5vHIDE9Xi0Q+Bhdj1czfPBeBMh4/g4V2lv+hTwES+RNGyf07C3D97dkd55yByBjUu21VIAWxfjFzzTtWzgP+TmHkR4K5JtYQFMs+KGsQsOIeEhH/PcHs3KRoYFQZPXoLhzj6ptNBxrTMK0//ewZaxMGBcAaaLX+8M26jZu/pSg8JOZap6/wBxG/OebuaQOV47nkhH2QEYSkXpNJFZ73owkhNNOdaPij95Nv7CECB83Ad9n/Co4oEKnxSaRrndc= ceph-a69c870a-c76e-11ee-95b7-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-09T17:16:34.567 INFO:teuthology.orchestra.run.smithi162.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSoO3S4tVb8t72oVO73Khx8s6h9Snxd09pp/GE/jxRJBhnG7z0LYypbaHuorVvLUoM5AiyPRF3qj8FiVst1jSkMUqcJ4iZO9pPxEe2FlC5aBVGesDe1jjLAP5JEtT3uuK3BUSj+t9JJtCUpzIEcFoBPMSzX4lFOjpfvdRlNO0ku25oZ6Ekr/kS+/SR0X/K5Zc6R1/4+B4l/YxvBpy0xtTgCcDwp1SOtPzZFJ8y/iUqBie28XPETCz7vcQOriFLsGyDh5vHIDE9Xi0Q+Bhdj1czfPBeBMh4/g4V2lv+hTwES+RNGyf07C3D97dkd55yByBjUu21VIAWxfjFzzTtWzgP+TmHkR4K5JtYQFMs+KGsQsOIeEhH/PcHs3KRoYFQZPXoLhzj6ptNBxrTMK0//ewZaxMGBcAaaLX+8M26jZu/pSg8JOZap6/wBxG/OebuaQOV47nkhH2QEYSkXpNJFZ73owkhNNOdaPij95Nv7CECB83Ad9n/Co4oEKnxSaRrndc= ceph-a69c870a-c76e-11ee-95b7-87774f69a715 2024-02-09T17:16:34.578 DEBUG:teuthology.orchestra.run.smithi179:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSoO3S4tVb8t72oVO73Khx8s6h9Snxd09pp/GE/jxRJBhnG7z0LYypbaHuorVvLUoM5AiyPRF3qj8FiVst1jSkMUqcJ4iZO9pPxEe2FlC5aBVGesDe1jjLAP5JEtT3uuK3BUSj+t9JJtCUpzIEcFoBPMSzX4lFOjpfvdRlNO0ku25oZ6Ekr/kS+/SR0X/K5Zc6R1/4+B4l/YxvBpy0xtTgCcDwp1SOtPzZFJ8y/iUqBie28XPETCz7vcQOriFLsGyDh5vHIDE9Xi0Q+Bhdj1czfPBeBMh4/g4V2lv+hTwES+RNGyf07C3D97dkd55yByBjUu21VIAWxfjFzzTtWzgP+TmHkR4K5JtYQFMs+KGsQsOIeEhH/PcHs3KRoYFQZPXoLhzj6ptNBxrTMK0//ewZaxMGBcAaaLX+8M26jZu/pSg8JOZap6/wBxG/OebuaQOV47nkhH2QEYSkXpNJFZ73owkhNNOdaPij95Nv7CECB83Ad9n/Co4oEKnxSaRrndc= ceph-a69c870a-c76e-11ee-95b7-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-09T17:16:34.623 INFO:teuthology.orchestra.run.smithi179.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSoO3S4tVb8t72oVO73Khx8s6h9Snxd09pp/GE/jxRJBhnG7z0LYypbaHuorVvLUoM5AiyPRF3qj8FiVst1jSkMUqcJ4iZO9pPxEe2FlC5aBVGesDe1jjLAP5JEtT3uuK3BUSj+t9JJtCUpzIEcFoBPMSzX4lFOjpfvdRlNO0ku25oZ6Ekr/kS+/SR0X/K5Zc6R1/4+B4l/YxvBpy0xtTgCcDwp1SOtPzZFJ8y/iUqBie28XPETCz7vcQOriFLsGyDh5vHIDE9Xi0Q+Bhdj1czfPBeBMh4/g4V2lv+hTwES+RNGyf07C3D97dkd55yByBjUu21VIAWxfjFzzTtWzgP+TmHkR4K5JtYQFMs+KGsQsOIeEhH/PcHs3KRoYFQZPXoLhzj6ptNBxrTMK0//ewZaxMGBcAaaLX+8M26jZu/pSg8JOZap6/wBxG/OebuaQOV47nkhH2QEYSkXpNJFZ73owkhNNOdaPij95Nv7CECB83Ad9n/Co4oEKnxSaRrndc= ceph-a69c870a-c76e-11ee-95b7-87774f69a715 2024-02-09T17:16:34.635 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-09T17:16:34.900 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:16:34.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:34 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2785236045' entity='client.admin' 2024-02-09T17:16:38.658 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-02-09T17:16:38.659 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-02-09T17:16:38.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:38 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/359245618' entity='client.admin' 2024-02-09T17:16:39.252 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:16:40.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:40 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:16:40.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:40 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:16:40.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:40 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi099", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:16:40.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:40 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:16:40.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:40 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:16:40.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:40 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:16:42.008 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi162 2024-02-09T17:16:42.008 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-02-09T17:16:42.008 DEBUG:teuthology.orchestra.run.smithi162:> dd of=/etc/ceph/ceph.conf 2024-02-09T17:16:42.030 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-02-09T17:16:42.030 DEBUG:teuthology.orchestra.run.smithi162:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-09T17:16:42.091 INFO:tasks.cephadm:Adding host smithi162 to orchestrator... 2024-02-09T17:16:42.091 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph orch host add smithi162 2024-02-09T17:16:42.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:42 smithi099 ceph-mon[39420]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:16:42.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:42 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:16:42.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:42 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:16:42.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:42 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:16:42.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:42 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:16:42.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:42 smithi099 ceph-mon[39420]: Updating smithi099:/etc/ceph/ceph.conf 2024-02-09T17:16:42.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:42 smithi099 ceph-mon[39420]: Updating smithi099:/var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/config/ceph.conf 2024-02-09T17:16:42.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:42 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:16:42.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:42 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:16:42.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:42 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:16:42.350 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:16:43.369 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:43 smithi099 ceph-mon[39420]: Updating smithi099:/etc/ceph/ceph.client.admin.keyring 2024-02-09T17:16:43.369 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:43 smithi099 ceph-mon[39420]: Updating smithi099:/var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/config/ceph.client.admin.keyring 2024-02-09T17:16:45.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:45 smithi099 ceph-mon[39420]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi162", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:16:45.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:45 smithi099 ceph-mon[39420]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:16:46.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:46 smithi099 ceph-mon[39420]: Deploying cephadm binary to smithi162 2024-02-09T17:16:47.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:47 smithi099 ceph-mon[39420]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:16:47.815 INFO:teuthology.orchestra.run.smithi099.stdout:Added host 'smithi162' with addr '172.21.15.162' 2024-02-09T17:16:48.469 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph orch host ls --format=json 2024-02-09T17:16:48.726 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:16:49.165 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:48 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:16:49.165 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:48 smithi099 ceph-mon[39420]: Added host smithi162 2024-02-09T17:16:49.165 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:48 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:16:49.165 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:48 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:16:49.165 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:48 smithi099 ceph-mon[39420]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:16:49.165 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:48 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:16:50.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:50 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:16:50.668 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:16:50.668 INFO:teuthology.orchestra.run.smithi099.stdout:[{"addr": "172.21.15.99", "hostname": "smithi099", "labels": [], "status": ""}, {"addr": "172.21.15.162", "hostname": "smithi162", "labels": [], "status": ""}] 2024-02-09T17:16:51.266 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi179 2024-02-09T17:16:51.267 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-02-09T17:16:51.267 DEBUG:teuthology.orchestra.run.smithi179:> dd of=/etc/ceph/ceph.conf 2024-02-09T17:16:51.288 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-02-09T17:16:51.288 DEBUG:teuthology.orchestra.run.smithi179:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-09T17:16:51.349 INFO:tasks.cephadm:Adding host smithi179 to orchestrator... 2024-02-09T17:16:51.349 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph orch host add smithi179 2024-02-09T17:16:51.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:51 smithi099 ceph-mon[39420]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:16:51.607 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:16:52.473 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:52 smithi099 ceph-mon[39420]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:16:53.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:53 smithi099 ceph-mon[39420]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:16:53.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:53 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:16:53.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:53 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:16:53.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:53 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:16:53.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:53 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:16:53.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:53 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi162", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:16:53.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:53 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:16:53.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:53 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:16:54.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:54 smithi099 ceph-mon[39420]: Updating smithi162:/etc/ceph/ceph.conf 2024-02-09T17:16:54.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:54 smithi099 ceph-mon[39420]: Updating smithi162:/var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/config/ceph.conf 2024-02-09T17:16:54.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:54 smithi099 ceph-mon[39420]: Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-02-09T17:16:54.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:54 smithi099 ceph-mon[39420]: Updating smithi162:/var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/config/ceph.client.admin.keyring 2024-02-09T17:16:54.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:54 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:16:54.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:54 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:16:54.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:54 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:16:55.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:55 smithi099 ceph-mon[39420]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi179", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:16:55.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:55 smithi099 ceph-mon[39420]: Deploying cephadm binary to smithi179 2024-02-09T17:16:55.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:55 smithi099 ceph-mon[39420]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:16:57.223 INFO:teuthology.orchestra.run.smithi099.stdout:Added host 'smithi179' with addr '172.21.15.179' 2024-02-09T17:16:57.598 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:57 smithi099 ceph-mon[39420]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:16:57.598 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:57 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:16:57.598 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:57 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:16:57.918 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph orch host ls --format=json 2024-02-09T17:16:58.173 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:16:58.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:58 smithi099 ceph-mon[39420]: Added host smithi179 2024-02-09T17:16:58.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:58 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:16:58.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:58 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:16:59.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:59 smithi099 ceph-mon[39420]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:16:59.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:16:59 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:16:59.990 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:16:59.990 INFO:teuthology.orchestra.run.smithi099.stdout:[{"addr": "172.21.15.99", "hostname": "smithi099", "labels": [], "status": ""}, {"addr": "172.21.15.162", "hostname": "smithi162", "labels": [], "status": ""}, {"addr": "172.21.15.179", "hostname": "smithi179", "labels": [], "status": ""}] 2024-02-09T17:17:00.679 INFO:tasks.cephadm:Setting crush tunables to default 2024-02-09T17:17:00.679 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd crush tunables default 2024-02-09T17:17:00.937 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:17:01.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:01 smithi099 ceph-mon[39420]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:17:01.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:01 smithi099 ceph-mon[39420]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:02.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:02 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:02.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:02 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:02.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:02 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:02.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:02 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:02.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:02 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi179", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:17:02.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:02 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:02.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:02 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:17:02.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:02 smithi099 ceph-mon[39420]: Updating smithi179:/etc/ceph/ceph.conf 2024-02-09T17:17:02.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:02 smithi099 ceph-mon[39420]: Updating smithi179:/var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/config/ceph.conf 2024-02-09T17:17:02.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:02 smithi099 ceph-mon[39420]: Updating smithi179:/etc/ceph/ceph.client.admin.keyring 2024-02-09T17:17:02.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:02 smithi099 ceph-mon[39420]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:02.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:02 smithi099 ceph-mon[39420]: Updating smithi179:/var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/config/ceph.client.admin.keyring 2024-02-09T17:17:02.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:02 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:02.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:02 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:02.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:02 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:03.791 INFO:teuthology.orchestra.run.smithi099.stderr:adjusted tunables profile to default 2024-02-09T17:17:04.185 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:03 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2579465243' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-09T17:17:04.455 INFO:tasks.cephadm:Adding mon.a on smithi099 2024-02-09T17:17:04.456 INFO:tasks.cephadm:Adding mon.b on smithi162 2024-02-09T17:17:04.456 INFO:tasks.cephadm:Adding mon.c on smithi179 2024-02-09T17:17:04.456 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph orch apply mon '3;smithi099:172.21.15.99=a;smithi162:172.21.15.162=b;smithi179:172.21.15.179=c' 2024-02-09T17:17:04.755 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/config/ceph.conf 2024-02-09T17:17:05.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:04 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2579465243' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-09T17:17:05.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:04 smithi099 ceph-mon[39420]: osdmap e4: 0 total, 0 up, 0 in 2024-02-09T17:17:05.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:04 smithi099 ceph-mon[39420]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:06.595 INFO:teuthology.orchestra.run.smithi179.stdout:Scheduled mon update... 2024-02-09T17:17:07.305 DEBUG:teuthology.orchestra.run.smithi162:mon.b> sudo journalctl -f -n 0 -u ceph-a69c870a-c76e-11ee-95b7-87774f69a715@mon.b.service 2024-02-09T17:17:07.308 DEBUG:teuthology.orchestra.run.smithi179:mon.c> sudo journalctl -f -n 0 -u ceph-a69c870a-c76e-11ee-95b7-87774f69a715@mon.c.service 2024-02-09T17:17:07.311 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-09T17:17:07.311 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph mon dump -f json 2024-02-09T17:17:07.340 INFO:journalctl@ceph.mon.c.smithi179.stdout:-- Logs begin at Fri 2024-02-09 17:04:31 UTC. -- 2024-02-09T17:17:07.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:-- Logs begin at Fri 2024-02-09 17:04:37 UTC. -- 2024-02-09T17:17:07.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:07 smithi099 ceph-mon[39420]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:07.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:07 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:07.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:07 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:17:07.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:07 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:07.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:07 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:17:07.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:07 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:07.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:07 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-09T17:17:07.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:07 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:08.045 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/config/ceph.conf 2024-02-09T17:17:08.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:08 smithi099 ceph-mon[39420]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi099:172.21.15.99=a;smithi162:172.21.15.162=b;smithi179:172.21.15.179=c", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:17:08.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:08 smithi099 ceph-mon[39420]: Saving service mon spec with placement smithi099:172.21.15.99=a;smithi162:172.21.15.162=b;smithi179:172.21.15.179=c;count:3 2024-02-09T17:17:08.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:08 smithi099 ceph-mon[39420]: Deploying daemon mon.c on smithi179 2024-02-09T17:17:09.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:09 smithi099 ceph-mon[39420]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:11.291 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:17:11.292 INFO:teuthology.orchestra.run.smithi179.stdout:{"epoch":1,"fsid":"a69c870a-c76e-11ee-95b7-87774f69a715","modified":"2024-02-09T17:15:23.317650Z","created":"2024-02-09T17:15:23.317650Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:3300","nonce":0},{"type":"v1","addr":"172.21.15.99:6789","nonce":0}]},"addr":"172.21.15.99:6789/0","public_addr":"172.21.15.99:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-09T17:17:11.293 INFO:teuthology.orchestra.run.smithi179.stderr:dumped monmap epoch 1 2024-02-09T17:17:11.605 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:11 smithi179 systemd[1]: Starting Ceph mon.c for a69c870a-c76e-11ee-95b7-87774f69a715... 2024-02-09T17:17:11.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:11 smithi099 ceph-mon[39420]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:11.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:11 smithi099 ceph-mon[39420]: from='client.? 172.21.15.179:0/2948853760' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-09T17:17:12.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: set uid:gid to 167:167 (ceph:ceph) 2024-02-09T17:17:12.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: ceph version 19.0.0-1271-g720625d3 (720625d38f7f95e6fa31efce87133247d9d28517) squid (dev), process ceph-mon, pid 7 2024-02-09T17:17:12.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: pidfile_write: ignore empty --pid-file 2024-02-09T17:17:12.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: load: jerasure load: lrc 2024-02-09T17:17:12.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: RocksDB version: 7.9.2 2024-02-09T17:17:12.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Git sha 0 2024-02-09T17:17:12.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Compile date 2024-02-06 20:16:13 2024-02-09T17:17:12.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: DB SUMMARY 2024-02-09T17:17:12.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: DB Session ID: DKV0Z0NMNWTSFS8NU2Y1 2024-02-09T17:17:12.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: CURRENT file: CURRENT 2024-02-09T17:17:12.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: IDENTITY file: IDENTITY 2024-02-09T17:17:12.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-02-09T17:17:12.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-02-09T17:17:12.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-02-09T17:17:12.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.error_if_exists: 0 2024-02-09T17:17:12.677 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.create_if_missing: 0 2024-02-09T17:17:12.677 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.paranoid_checks: 1 2024-02-09T17:17:12.677 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-09T17:17:12.677 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-09T17:17:12.677 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-09T17:17:12.677 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.env: 0x5614abbe2ae0 2024-02-09T17:17:12.677 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.fs: PosixFileSystem 2024-02-09T17:17:12.677 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.info_log: 0x5614acd838c0 2024-02-09T17:17:12.677 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.max_file_opening_threads: 16 2024-02-09T17:17:12.677 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.statistics: (nil) 2024-02-09T17:17:12.677 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.use_fsync: 0 2024-02-09T17:17:12.677 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.max_log_file_size: 0 2024-02-09T17:17:12.677 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-09T17:17:12.677 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-09T17:17:12.677 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.keep_log_file_num: 1000 2024-02-09T17:17:12.678 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.recycle_log_file_num: 0 2024-02-09T17:17:12.678 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.allow_fallocate: 1 2024-02-09T17:17:12.678 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.allow_mmap_reads: 0 2024-02-09T17:17:12.678 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.allow_mmap_writes: 0 2024-02-09T17:17:12.678 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.use_direct_reads: 0 2024-02-09T17:17:12.678 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-09T17:17:12.678 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.create_missing_column_families: 0 2024-02-09T17:17:12.678 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.db_log_dir: 2024-02-09T17:17:12.678 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.wal_dir: 2024-02-09T17:17:12.678 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-09T17:17:12.678 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-09T17:17:12.678 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-09T17:17:12.678 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-09T17:17:12.679 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-09T17:17:12.679 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-09T17:17:12.679 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.advise_random_on_open: 1 2024-02-09T17:17:12.679 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.db_write_buffer_size: 0 2024-02-09T17:17:12.679 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.write_buffer_manager: 0x5614acd57400 2024-02-09T17:17:12.679 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-09T17:17:12.679 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-09T17:17:12.679 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-09T17:17:12.679 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.rate_limiter: (nil) 2024-02-09T17:17:12.679 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-09T17:17:12.679 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.wal_recovery_mode: 2 2024-02-09T17:17:12.679 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.enable_thread_tracking: 0 2024-02-09T17:17:12.679 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.enable_pipelined_write: 0 2024-02-09T17:17:12.679 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.unordered_write: 0 2024-02-09T17:17:12.680 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-09T17:17:12.680 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-09T17:17:12.680 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-09T17:17:12.680 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-09T17:17:12.680 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.row_cache: None 2024-02-09T17:17:12.680 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.wal_filter: None 2024-02-09T17:17:12.680 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-09T17:17:12.680 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.allow_ingest_behind: 0 2024-02-09T17:17:12.680 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.two_write_queues: 0 2024-02-09T17:17:12.680 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.manual_wal_flush: 0 2024-02-09T17:17:12.680 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.wal_compression: 0 2024-02-09T17:17:12.680 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.atomic_flush: 0 2024-02-09T17:17:12.680 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-09T17:17:12.681 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-09T17:17:12.681 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-09T17:17:12.681 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.log_readahead_size: 0 2024-02-09T17:17:12.681 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-09T17:17:12.681 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.best_efforts_recovery: 0 2024-02-09T17:17:12.681 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-09T17:17:12.681 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-09T17:17:12.681 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.allow_data_in_errors: 0 2024-02-09T17:17:12.681 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.db_host_id: __hostname__ 2024-02-09T17:17:12.681 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-09T17:17:12.681 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.max_background_jobs: 2 2024-02-09T17:17:12.681 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.max_background_compactions: -1 2024-02-09T17:17:12.681 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.max_subcompactions: 1 2024-02-09T17:17:12.681 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-09T17:17:12.681 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-09T17:17:12.682 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-09T17:17:12.682 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.max_total_wal_size: 0 2024-02-09T17:17:12.682 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-09T17:17:12.682 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-09T17:17:12.682 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-09T17:17:12.682 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-09T17:17:12.682 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.max_open_files: -1 2024-02-09T17:17:12.682 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.bytes_per_sync: 0 2024-02-09T17:17:12.682 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-09T17:17:12.682 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-09T17:17:12.682 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.compaction_readahead_size: 0 2024-02-09T17:17:12.682 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.max_background_flushes: -1 2024-02-09T17:17:12.682 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Compression algorithms supported: 2024-02-09T17:17:12.683 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-09T17:17:12.683 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: kZSTD supported: 0 2024-02-09T17:17:12.683 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: kXpressCompression supported: 0 2024-02-09T17:17:12.683 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: kLZ4HCCompression supported: 1 2024-02-09T17:17:12.683 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: kZlibCompression supported: 1 2024-02-09T17:17:12.683 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: kSnappyCompression supported: 1 2024-02-09T17:17:12.683 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: kLZ4Compression supported: 1 2024-02-09T17:17:12.683 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: kBZip2Compression supported: 0 2024-02-09T17:17:12.683 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-09T17:17:12.683 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-09T17:17:12.683 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-02-09T17:17:12.683 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-09T17:17:12.683 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-09T17:17:12.683 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.merge_operator: 2024-02-09T17:17:12.684 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.compaction_filter: None 2024-02-09T17:17:12.684 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.compaction_filter_factory: None 2024-02-09T17:17:12.684 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.sst_partitioner_factory: None 2024-02-09T17:17:12.684 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-09T17:17:12.684 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-09T17:17:12.684 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5614acd839c0) 2024-02-09T17:17:12.684 INFO:journalctl@ceph.mon.c.smithi179.stdout: cache_index_and_filter_blocks: 1 2024-02-09T17:17:12.684 INFO:journalctl@ceph.mon.c.smithi179.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-09T17:17:12.684 INFO:journalctl@ceph.mon.c.smithi179.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-09T17:17:12.684 INFO:journalctl@ceph.mon.c.smithi179.stdout: pin_top_level_index_and_filter: 1 2024-02-09T17:17:12.684 INFO:journalctl@ceph.mon.c.smithi179.stdout: index_type: 0 2024-02-09T17:17:12.684 INFO:journalctl@ceph.mon.c.smithi179.stdout: data_block_index_type: 0 2024-02-09T17:17:12.684 INFO:journalctl@ceph.mon.c.smithi179.stdout: index_shortening: 1 2024-02-09T17:17:12.685 INFO:journalctl@ceph.mon.c.smithi179.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-09T17:17:12.685 INFO:journalctl@ceph.mon.c.smithi179.stdout: checksum: 4 2024-02-09T17:17:12.685 INFO:journalctl@ceph.mon.c.smithi179.stdout: no_block_cache: 0 2024-02-09T17:17:12.685 INFO:journalctl@ceph.mon.c.smithi179.stdout: block_cache: 0x5614acd87090 2024-02-09T17:17:12.685 INFO:journalctl@ceph.mon.c.smithi179.stdout: block_cache_name: BinnedLRUCache 2024-02-09T17:17:12.685 INFO:journalctl@ceph.mon.c.smithi179.stdout: block_cache_options: 2024-02-09T17:17:12.685 INFO:journalctl@ceph.mon.c.smithi179.stdout: capacity : 536870912 2024-02-09T17:17:12.685 INFO:journalctl@ceph.mon.c.smithi179.stdout: num_shard_bits : 4 2024-02-09T17:17:12.685 INFO:journalctl@ceph.mon.c.smithi179.stdout: strict_capacity_limit : 0 2024-02-09T17:17:12.685 INFO:journalctl@ceph.mon.c.smithi179.stdout: high_pri_pool_ratio: 0.000 2024-02-09T17:17:12.685 INFO:journalctl@ceph.mon.c.smithi179.stdout: block_cache_compressed: (nil) 2024-02-09T17:17:12.685 INFO:journalctl@ceph.mon.c.smithi179.stdout: persistent_cache: (nil) 2024-02-09T17:17:12.685 INFO:journalctl@ceph.mon.c.smithi179.stdout: block_size: 4096 2024-02-09T17:17:12.685 INFO:journalctl@ceph.mon.c.smithi179.stdout: block_size_deviation: 10 2024-02-09T17:17:12.685 INFO:journalctl@ceph.mon.c.smithi179.stdout: block_restart_interval: 16 2024-02-09T17:17:12.685 INFO:journalctl@ceph.mon.c.smithi179.stdout: index_block_restart_interval: 1 2024-02-09T17:17:12.686 INFO:journalctl@ceph.mon.c.smithi179.stdout: metadata_block_size: 4096 2024-02-09T17:17:12.686 INFO:journalctl@ceph.mon.c.smithi179.stdout: partition_filters: 0 2024-02-09T17:17:12.686 INFO:journalctl@ceph.mon.c.smithi179.stdout: use_delta_encoding: 1 2024-02-09T17:17:12.686 INFO:journalctl@ceph.mon.c.smithi179.stdout: filter_policy: bloomfilter 2024-02-09T17:17:12.686 INFO:journalctl@ceph.mon.c.smithi179.stdout: whole_key_filtering: 1 2024-02-09T17:17:12.686 INFO:journalctl@ceph.mon.c.smithi179.stdout: verify_compression: 0 2024-02-09T17:17:12.686 INFO:journalctl@ceph.mon.c.smithi179.stdout: read_amp_bytes_per_bit: 0 2024-02-09T17:17:12.686 INFO:journalctl@ceph.mon.c.smithi179.stdout: format_version: 5 2024-02-09T17:17:12.686 INFO:journalctl@ceph.mon.c.smithi179.stdout: enable_index_compression: 1 2024-02-09T17:17:12.686 INFO:journalctl@ceph.mon.c.smithi179.stdout: block_align: 0 2024-02-09T17:17:12.686 INFO:journalctl@ceph.mon.c.smithi179.stdout: max_auto_readahead_size: 262144 2024-02-09T17:17:12.686 INFO:journalctl@ceph.mon.c.smithi179.stdout: prepopulate_block_cache: 0 2024-02-09T17:17:12.686 INFO:journalctl@ceph.mon.c.smithi179.stdout: initial_auto_readahead_size: 8192 2024-02-09T17:17:12.686 INFO:journalctl@ceph.mon.c.smithi179.stdout: num_file_reads_for_auto_readahead: 2 2024-02-09T17:17:12.686 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.write_buffer_size: 33554432 2024-02-09T17:17:12.687 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.max_write_buffer_number: 2 2024-02-09T17:17:12.687 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.compression: NoCompression 2024-02-09T17:17:12.687 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.bottommost_compression: Disabled 2024-02-09T17:17:12.687 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.prefix_extractor: nullptr 2024-02-09T17:17:12.687 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-09T17:17:12.687 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.num_levels: 7 2024-02-09T17:17:12.687 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-09T17:17:12.687 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-09T17:17:12.687 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-09T17:17:12.687 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-09T17:17:12.687 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-09T17:17:12.687 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-09T17:17:12.687 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-09T17:17:12.687 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-09T17:17:12.687 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-09T17:17:12.687 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-09T17:17:12.688 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-09T17:17:12.688 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-09T17:17:12.688 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-09T17:17:12.688 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.compression_opts.level: 32767 2024-02-09T17:17:12.688 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.compression_opts.strategy: 0 2024-02-09T17:17:12.688 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-09T17:17:12.688 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-09T17:17:12.688 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-09T17:17:12.688 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-09T17:17:12.688 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.compression_opts.enabled: false 2024-02-09T17:17:12.688 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-09T17:17:12.688 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-09T17:17:12.688 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-09T17:17:12.688 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-09T17:17:12.688 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.target_file_size_base: 67108864 2024-02-09T17:17:12.689 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-09T17:17:12.689 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-09T17:17:12.689 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-09T17:17:12.689 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-09T17:17:12.689 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-09T17:17:12.689 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-09T17:17:12.689 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-09T17:17:12.689 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-09T17:17:12.689 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-09T17:17:12.689 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-09T17:17:12.689 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-09T17:17:12.689 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-09T17:17:12.689 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-09T17:17:12.689 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-09T17:17:12.690 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.arena_block_size: 1048576 2024-02-09T17:17:12.690 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-09T17:17:12.690 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-09T17:17:12.690 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.disable_auto_compactions: 0 2024-02-09T17:17:12.690 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-09T17:17:12.690 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-09T17:17:12.690 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-09T17:17:12.690 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-09T17:17:12.690 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-09T17:17:12.690 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-09T17:17:12.690 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-09T17:17:12.690 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-09T17:17:12.690 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-09T17:17:12.690 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-09T17:17:12.691 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.table_properties_collectors: 2024-02-09T17:17:12.691 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.inplace_update_support: 0 2024-02-09T17:17:12.691 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-09T17:17:12.691 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-09T17:17:12.691 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-09T17:17:12.691 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-09T17:17:12.691 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.bloom_locality: 0 2024-02-09T17:17:12.691 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.max_successive_merges: 0 2024-02-09T17:17:12.691 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-09T17:17:12.691 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.paranoid_file_checks: 0 2024-02-09T17:17:12.691 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.force_consistency_checks: 1 2024-02-09T17:17:12.691 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.report_bg_io_stats: 0 2024-02-09T17:17:12.691 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.ttl: 2592000 2024-02-09T17:17:12.691 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-09T17:17:12.691 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-09T17:17:12.692 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-09T17:17:12.692 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.enable_blob_files: false 2024-02-09T17:17:12.692 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.min_blob_size: 0 2024-02-09T17:17:12.692 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.blob_file_size: 268435456 2024-02-09T17:17:12.692 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-09T17:17:12.692 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-09T17:17:12.692 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-09T17:17:12.692 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-09T17:17:12.692 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-09T17:17:12.692 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.blob_file_starting_level: 0 2024-02-09T17:17:12.692 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-09T17:17:12.692 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-09T17:17:12.692 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-02-09T17:17:12.692 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 7657b6a9-c03e-467a-a192-5cec46930ae5 2024-02-09T17:17:12.693 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707499032403758, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-02-09T17:17:12.693 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-02-09T17:17:12.693 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707499032404322, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1707499032, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "7657b6a9-c03e-467a-a192-5cec46930ae5", "db_session_id": "DKV0Z0NMNWTSFS8NU2Y1", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-02-09T17:17:12.693 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707499032404425, "job": 1, "event": "recovery_finished"} 2024-02-09T17:17:12.693 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-02-09T17:17:12.693 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-09T17:17:12.693 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5614ace78000 2024-02-09T17:17:12.693 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: DB pointer 0x5614ace62000 2024-02-09T17:17:12.693 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-09T17:17:12.693 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-09T17:17:12.693 INFO:journalctl@ceph.mon.c.smithi179.stdout: ** DB Stats ** 2024-02-09T17:17:12.693 INFO:journalctl@ceph.mon.c.smithi179.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-09T17:17:12.693 INFO:journalctl@ceph.mon.c.smithi179.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-09T17:17:12.694 INFO:journalctl@ceph.mon.c.smithi179.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-09T17:17:12.694 INFO:journalctl@ceph.mon.c.smithi179.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-09T17:17:12.694 INFO:journalctl@ceph.mon.c.smithi179.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-09T17:17:12.694 INFO:journalctl@ceph.mon.c.smithi179.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-09T17:17:12.694 INFO:journalctl@ceph.mon.c.smithi179.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-09T17:17:12.694 INFO:journalctl@ceph.mon.c.smithi179.stdout: 2024-02-09T17:17:12.694 INFO:journalctl@ceph.mon.c.smithi179.stdout: ** Compaction Stats [default] ** 2024-02-09T17:17:12.694 INFO:journalctl@ceph.mon.c.smithi179.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-09T17:17:12.694 INFO:journalctl@ceph.mon.c.smithi179.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-09T17:17:12.694 INFO:journalctl@ceph.mon.c.smithi179.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-09T17:17:12.694 INFO:journalctl@ceph.mon.c.smithi179.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-09T17:17:12.694 INFO:journalctl@ceph.mon.c.smithi179.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-09T17:17:12.694 INFO:journalctl@ceph.mon.c.smithi179.stdout: 2024-02-09T17:17:12.694 INFO:journalctl@ceph.mon.c.smithi179.stdout: ** Compaction Stats [default] ** 2024-02-09T17:17:12.694 INFO:journalctl@ceph.mon.c.smithi179.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-09T17:17:12.695 INFO:journalctl@ceph.mon.c.smithi179.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-09T17:17:12.695 INFO:journalctl@ceph.mon.c.smithi179.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-09T17:17:12.695 INFO:journalctl@ceph.mon.c.smithi179.stdout: 2024-02-09T17:17:12.695 INFO:journalctl@ceph.mon.c.smithi179.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-09T17:17:12.695 INFO:journalctl@ceph.mon.c.smithi179.stdout: 2024-02-09T17:17:12.695 INFO:journalctl@ceph.mon.c.smithi179.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-09T17:17:12.695 INFO:journalctl@ceph.mon.c.smithi179.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-09T17:17:12.695 INFO:journalctl@ceph.mon.c.smithi179.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-09T17:17:12.695 INFO:journalctl@ceph.mon.c.smithi179.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-09T17:17:12.695 INFO:journalctl@ceph.mon.c.smithi179.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-09T17:17:12.695 INFO:journalctl@ceph.mon.c.smithi179.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-09T17:17:12.695 INFO:journalctl@ceph.mon.c.smithi179.stdout: Cumulative compaction: 0.00 GB write, 0.47 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-09T17:17:12.695 INFO:journalctl@ceph.mon.c.smithi179.stdout: Interval compaction: 0.00 GB write, 0.47 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-09T17:17:12.695 INFO:journalctl@ceph.mon.c.smithi179.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-09T17:17:12.696 INFO:journalctl@ceph.mon.c.smithi179.stdout: Block cache BinnedLRUCache@0x5614acd87090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-02-09T17:17:12.696 INFO:journalctl@ceph.mon.c.smithi179.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-02-09T17:17:12.696 INFO:journalctl@ceph.mon.c.smithi179.stdout: 2024-02-09T17:17:12.696 INFO:journalctl@ceph.mon.c.smithi179.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-09T17:17:12.696 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-02-09T17:17:12.696 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: using public_addr v2:172.21.15.179:0/0 -> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] 2024-02-09T17:17:12.696 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: starting mon.c rank -1 at public addrs [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] at bind addrs [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid a69c870a-c76e-11ee-95b7-87774f69a715 2024-02-09T17:17:12.696 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: mon.c@-1(???) e0 preinit fsid a69c870a-c76e-11ee-95b7-87774f69a715 2024-02-09T17:17:12.696 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: mon.c@-1(synchronizing).mds e1 new map 2024-02-09T17:17:12.696 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: mon.c@-1(synchronizing).mds e1 print_map 2024-02-09T17:17:12.696 INFO:journalctl@ceph.mon.c.smithi179.stdout: e1 2024-02-09T17:17:12.696 INFO:journalctl@ceph.mon.c.smithi179.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-09T17:17:12.696 INFO:journalctl@ceph.mon.c.smithi179.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-02-09T17:17:12.696 INFO:journalctl@ceph.mon.c.smithi179.stdout: legacy client fscid: -1 2024-02-09T17:17:12.697 INFO:journalctl@ceph.mon.c.smithi179.stdout: 2024-02-09T17:17:12.697 INFO:journalctl@ceph.mon.c.smithi179.stdout: No filesystems configured 2024-02-09T17:17:12.697 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: [db/db_impl/db_impl_write.cc:2098] [default] New memtable created with log file: #13. Immutable memtables: 0. 2024-02-09T17:17:12.697 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-02-09T17:17:12.697 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-02-09T17:17:12.697 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-02-09T17:17:12.697 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: (Original Log Time 2024/02/09-17:17:12.426416) [db/db_impl/db_impl_compaction_flush.cc:2832] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 2024-02-09T17:17:12.697 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: [db/flush_job.cc:856] [default] [JOB 3] Flushing memtable with next log file: 13 2024-02-09T17:17:12.697 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707499032426520, "job": 3, "event": "flush_started", "num_memtables": 1, "num_entries": 1095, "num_deletes": 24, "total_data_size": 2239432, "memory_usage": 2260368, "flush_reason": "Manual Compaction"} 2024-02-09T17:17:12.697 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-02-09T17:17:12.697 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: [db/flush_job.cc:885] [default] [JOB 3] Level-0 flush table #14: started 2024-02-09T17:17:12.697 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-02-09T17:17:12.697 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-02-09T17:17:12.697 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-09T17:17:12.697 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-09T17:17:12.698 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-09T17:17:12.698 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-09T17:17:12.698 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:17:12.698 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:12.698 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:12.698 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:12.698 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:12.698 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:12.698 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi162", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:17:12.698 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:12.698 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:17:12.698 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: Updating smithi162:/etc/ceph/ceph.conf 2024-02-09T17:17:12.698 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: Updating smithi162:/var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/config/ceph.conf 2024-02-09T17:17:12.698 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-02-09T17:17:12.698 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: Updating smithi162:/var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/config/ceph.client.admin.keyring 2024-02-09T17:17:12.699 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:12.699 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:12.699 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:12.699 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi179", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:17:12.699 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: Deploying cephadm binary to smithi179 2024-02-09T17:17:12.699 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:12.699 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:12.699 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:12.699 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:17:12.699 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: Added host smithi179 2024-02-09T17:17:12.699 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:12.699 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:12.699 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:12.699 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:12.700 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:17:12.700 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:12.700 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:12.700 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:12.700 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:12.700 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:12.700 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi179", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:17:12.700 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:12.700 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:17:12.700 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: Updating smithi179:/etc/ceph/ceph.conf 2024-02-09T17:17:12.700 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: Updating smithi179:/var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/config/ceph.conf 2024-02-09T17:17:12.700 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: Updating smithi179:/etc/ceph/ceph.client.admin.keyring 2024-02-09T17:17:12.700 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:12.700 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: Updating smithi179:/var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/config/ceph.client.admin.keyring 2024-02-09T17:17:12.700 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:12.701 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:12.701 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:12.701 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2579465243' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-09T17:17:12.701 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2579465243' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-09T17:17:12.701 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: osdmap e4: 0 total, 0 up, 0 in 2024-02-09T17:17:12.701 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:12.701 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:12.701 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:12.701 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:17:12.701 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:12.701 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:17:12.701 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:12.701 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-09T17:17:12.701 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:12.702 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi099:172.21.15.99=a;smithi162:172.21.15.162=b;smithi179:172.21.15.179=c", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:17:12.702 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: Saving service mon spec with placement smithi099:172.21.15.99=a;smithi162:172.21.15.162=b;smithi179:172.21.15.179=c;count:3 2024-02-09T17:17:12.702 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: Deploying daemon mon.c on smithi179 2024-02-09T17:17:12.702 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:12.702 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:12.702 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: from='client.? 172.21.15.179:0/2948853760' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-09T17:17:12.702 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-02-09T17:17:12.702 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707499032430322, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 14, "file_size": 1970614, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 6, "largest_seqno": 1100, "table_properties": {"data_size": 1965052, "index_size": 2472, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 2053, "raw_key_size": 25697, "raw_average_key_size": 31, "raw_value_size": 1947659, "raw_average_value_size": 2381, "num_data_blocks": 115, "num_entries": 818, "num_filter_entries": 818, "num_deletions": 24, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1707499032, "oldest_key_time": 1707499032, "file_creation_time": 1707499032, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "7657b6a9-c03e-467a-a192-5cec46930ae5", "db_session_id": "DKV0Z0NMNWTSFS8NU2Y1", "orig_file_number": 14, "seqno_to_time_mapping": "N/A"}} 2024-02-09T17:17:12.702 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: [db/flush_job.cc:1019] [default] [JOB 3] Flush lasted 3928 microseconds, and 2555 cpu microseconds. 2024-02-09T17:17:12.702 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: (Original Log Time 2024/02/09-17:17:12.430346) [db/flush_job.cc:967] [default] [JOB 3] Level-0 flush table #14: 1970614 bytes OK 2024-02-09T17:17:12.702 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: (Original Log Time 2024/02/09-17:17:12.430358) [db/memtable_list.cc:519] [default] Level-0 commit table #14 started 2024-02-09T17:17:12.702 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: (Original Log Time 2024/02/09-17:17:12.430463) [db/memtable_list.cc:722] [default] Level-0 commit table #14: memtable #1 done 2024-02-09T17:17:12.702 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: (Original Log Time 2024/02/09-17:17:12.430474) EVENT_LOG_v1 {"time_micros": 1707499032430471, "job": 3, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [2, 0, 0, 0, 0, 0, 0], "immutable_memtables": 0} 2024-02-09T17:17:12.702 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: (Original Log Time 2024/02/09-17:17:12.430486) [db/db_impl/db_impl_compaction_flush.cc:299] [default] Level summary: files[2 0 0 0 0 0 0] max score 0.50 2024-02-09T17:17:12.702 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: [db/db_impl/db_impl_files.cc:463] [JOB 3] Try to delete WAL files size 2231803, prev total WAL file size 2231944, number of live WAL files 2. 2024-02-09T17:17:12.703 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-09T17:17:12.703 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: (Original Log Time 2024/02/09-17:17:12.430812) [db/db_impl/db_impl_compaction_flush.cc:3165] [default] Manual compaction from level-0 to level-6 from '6D6772737461740030' seq:72057594037927935, type:22 .. '6D677273746174003132' seq:0, type:0; will stop at (end) 2024-02-09T17:17:12.703 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: [db/compaction/compaction_job.cc:1995] [default] [JOB 4] Compacting 2@0 files to L6, score -1.00 2024-02-09T17:17:12.703 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: [db/compaction/compaction_job.cc:2001] [default]: Compaction start summary: Base version 3 Base level 0, inputs: [14(1924KB) 8(1622B)] 2024-02-09T17:17:12.703 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 ceph-mon[43975]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707499032430858, "job": 4, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [14, 8], "score": -1, "input_data_size": 1972236, "oldest_snapshot_seqno": -1} 2024-02-09T17:17:12.703 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 bash[43937]: 2a6199e25cacfaacd4a683c668bf66ff8e35d65cbdfc07882d2a1eba49113706 2024-02-09T17:17:12.703 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:12 smithi179 systemd[1]: Started Ceph mon.c for a69c870a-c76e-11ee-95b7-87774f69a715. 2024-02-09T17:17:12.954 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-09T17:17:12.954 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph mon dump -f json 2024-02-09T17:17:13.210 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:17:17.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:17 smithi179 ceph-mon[43975]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:17.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:17 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:17:17.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:17 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:17:17.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:17 smithi179 ceph-mon[43975]: mon.a calling monitor election 2024-02-09T17:17:17.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:17 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:17:17.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:17 smithi179 ceph-mon[43975]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:17.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:17 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:17:17.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:17 smithi179 ceph-mon[43975]: mon.c calling monitor election 2024-02-09T17:17:17.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:17 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:17:17.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:17 smithi179 ceph-mon[43975]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:17.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:17 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:17:17.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:17 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:17:17.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:17 smithi179 ceph-mon[43975]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-09T17:17:17.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:17 smithi179 ceph-mon[43975]: monmap e2: 2 mons at {a=[v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0],c=[v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-09T17:17:17.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:17 smithi179 ceph-mon[43975]: fsmap 2024-02-09T17:17:17.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:17 smithi179 ceph-mon[43975]: osdmap e4: 0 total, 0 up, 0 in 2024-02-09T17:17:17.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:17 smithi179 ceph-mon[43975]: mgrmap e13: y(active, since 53s) 2024-02-09T17:17:17.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:17 smithi179 ceph-mon[43975]: overall HEALTH_OK 2024-02-09T17:17:17.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:17 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:17.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:17 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:17.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:17 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:17.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:17 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-09T17:17:17.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:17 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:17.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:17 smithi099 ceph-mon[39420]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:17.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:17 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:17:17.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:17 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:17:17.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:17 smithi099 ceph-mon[39420]: mon.a calling monitor election 2024-02-09T17:17:17.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:17 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:17:17.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:17 smithi099 ceph-mon[39420]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:17.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:17 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:17:17.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:17 smithi099 ceph-mon[39420]: mon.c calling monitor election 2024-02-09T17:17:17.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:17 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:17:17.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:17 smithi099 ceph-mon[39420]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:17.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:17 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:17:17.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:17 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:17:17.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:17 smithi099 ceph-mon[39420]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-09T17:17:17.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:17 smithi099 ceph-mon[39420]: monmap e2: 2 mons at {a=[v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0],c=[v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-09T17:17:17.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:17 smithi099 ceph-mon[39420]: fsmap 2024-02-09T17:17:17.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:17 smithi099 ceph-mon[39420]: osdmap e4: 0 total, 0 up, 0 in 2024-02-09T17:17:17.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:17 smithi099 ceph-mon[39420]: mgrmap e13: y(active, since 53s) 2024-02-09T17:17:17.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:17 smithi099 ceph-mon[39420]: overall HEALTH_OK 2024-02-09T17:17:17.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:17 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:17.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:17 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:17.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:17 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:17.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:17 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-09T17:17:17.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:17 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:18.157 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:17:18.157 INFO:teuthology.orchestra.run.smithi179.stdout:{"epoch":2,"fsid":"a69c870a-c76e-11ee-95b7-87774f69a715","modified":"2024-02-09T17:17:12.434235Z","created":"2024-02-09T17:15:23.317650Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:3300","nonce":0},{"type":"v1","addr":"172.21.15.99:6789","nonce":0}]},"addr":"172.21.15.99:6789/0","public_addr":"172.21.15.99:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:3300","nonce":0},{"type":"v1","addr":"172.21.15.179:6789","nonce":0}]},"addr":"172.21.15.179:6789/0","public_addr":"172.21.15.179:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-09T17:17:18.159 INFO:teuthology.orchestra.run.smithi179.stderr:dumped monmap epoch 2 2024-02-09T17:17:18.843 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:18 smithi179 ceph-mon[43975]: Deploying daemon mon.b on smithi162 2024-02-09T17:17:18.843 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:18 smithi179 ceph-mon[43975]: from='client.? 172.21.15.179:0/3248729134' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-09T17:17:18.843 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:18 smithi179 ceph-mon[43975]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:18.843 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:18 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:17:18.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:18 smithi099 ceph-mon[39420]: Deploying daemon mon.b on smithi162 2024-02-09T17:17:18.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:18 smithi099 ceph-mon[39420]: from='client.? 172.21.15.179:0/3248729134' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-09T17:17:18.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:18 smithi099 ceph-mon[39420]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:18.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:18 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:17:19.871 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-09T17:17:19.871 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph mon dump -f json 2024-02-09T17:17:20.142 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:17:20.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:20 smithi162 systemd[1]: Starting Ceph mon.b for a69c870a-c76e-11ee-95b7-87774f69a715... 2024-02-09T17:17:21.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: set uid:gid to 167:167 (ceph:ceph) 2024-02-09T17:17:21.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: ceph version 19.0.0-1271-g720625d3 (720625d38f7f95e6fa31efce87133247d9d28517) squid (dev), process ceph-mon, pid 7 2024-02-09T17:17:21.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: pidfile_write: ignore empty --pid-file 2024-02-09T17:17:21.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: load: jerasure load: lrc 2024-02-09T17:17:21.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: RocksDB version: 7.9.2 2024-02-09T17:17:21.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Git sha 0 2024-02-09T17:17:21.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Compile date 2024-02-06 20:16:13 2024-02-09T17:17:21.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: DB SUMMARY 2024-02-09T17:17:21.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: DB Session ID: LHTK4IEX8MPX61EJ2EXI 2024-02-09T17:17:21.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: CURRENT file: CURRENT 2024-02-09T17:17:21.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: IDENTITY file: IDENTITY 2024-02-09T17:17:21.431 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-02-09T17:17:21.431 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-02-09T17:17:21.431 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-02-09T17:17:21.431 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.error_if_exists: 0 2024-02-09T17:17:21.431 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.create_if_missing: 0 2024-02-09T17:17:21.431 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.paranoid_checks: 1 2024-02-09T17:17:21.431 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-09T17:17:21.431 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-09T17:17:21.431 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-09T17:17:21.431 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.env: 0x55b874295ae0 2024-02-09T17:17:21.431 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.fs: PosixFileSystem 2024-02-09T17:17:21.432 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.info_log: 0x55b87650f8c0 2024-02-09T17:17:21.432 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.max_file_opening_threads: 16 2024-02-09T17:17:21.432 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.statistics: (nil) 2024-02-09T17:17:21.432 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.use_fsync: 0 2024-02-09T17:17:21.432 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.max_log_file_size: 0 2024-02-09T17:17:21.432 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-09T17:17:21.432 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-09T17:17:21.432 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.keep_log_file_num: 1000 2024-02-09T17:17:21.432 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.recycle_log_file_num: 0 2024-02-09T17:17:21.432 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.allow_fallocate: 1 2024-02-09T17:17:21.432 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.allow_mmap_reads: 0 2024-02-09T17:17:21.433 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.allow_mmap_writes: 0 2024-02-09T17:17:21.433 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.use_direct_reads: 0 2024-02-09T17:17:21.433 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-09T17:17:21.433 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.create_missing_column_families: 0 2024-02-09T17:17:21.433 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.db_log_dir: 2024-02-09T17:17:21.433 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.wal_dir: 2024-02-09T17:17:21.433 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-09T17:17:21.433 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-09T17:17:21.433 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-09T17:17:21.433 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-09T17:17:21.434 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-09T17:17:21.434 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-09T17:17:21.434 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.advise_random_on_open: 1 2024-02-09T17:17:21.434 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.db_write_buffer_size: 0 2024-02-09T17:17:21.434 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.write_buffer_manager: 0x55b8764e3400 2024-02-09T17:17:21.434 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-09T17:17:21.434 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-09T17:17:21.434 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-09T17:17:21.434 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.rate_limiter: (nil) 2024-02-09T17:17:21.434 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-09T17:17:21.434 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.wal_recovery_mode: 2 2024-02-09T17:17:21.434 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.enable_thread_tracking: 0 2024-02-09T17:17:21.435 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.enable_pipelined_write: 0 2024-02-09T17:17:21.435 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.unordered_write: 0 2024-02-09T17:17:21.435 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-09T17:17:21.435 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-09T17:17:21.435 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-09T17:17:21.435 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-09T17:17:21.435 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.row_cache: None 2024-02-09T17:17:21.435 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.wal_filter: None 2024-02-09T17:17:21.435 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-09T17:17:21.435 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.allow_ingest_behind: 0 2024-02-09T17:17:21.435 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.two_write_queues: 0 2024-02-09T17:17:21.435 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.manual_wal_flush: 0 2024-02-09T17:17:21.435 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.wal_compression: 0 2024-02-09T17:17:21.436 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.atomic_flush: 0 2024-02-09T17:17:21.436 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-09T17:17:21.436 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-09T17:17:21.436 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-09T17:17:21.436 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.log_readahead_size: 0 2024-02-09T17:17:21.436 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-09T17:17:21.436 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.best_efforts_recovery: 0 2024-02-09T17:17:21.436 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-09T17:17:21.436 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-09T17:17:21.436 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.allow_data_in_errors: 0 2024-02-09T17:17:21.436 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.db_host_id: __hostname__ 2024-02-09T17:17:21.436 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-09T17:17:21.436 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.max_background_jobs: 2 2024-02-09T17:17:21.437 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.max_background_compactions: -1 2024-02-09T17:17:21.437 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.max_subcompactions: 1 2024-02-09T17:17:21.437 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-09T17:17:21.437 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-09T17:17:21.437 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-09T17:17:21.437 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.max_total_wal_size: 0 2024-02-09T17:17:21.437 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-09T17:17:21.437 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-09T17:17:21.437 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-09T17:17:21.437 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-09T17:17:21.437 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.max_open_files: -1 2024-02-09T17:17:21.438 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.bytes_per_sync: 0 2024-02-09T17:17:21.438 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-09T17:17:21.438 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-09T17:17:21.438 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.compaction_readahead_size: 0 2024-02-09T17:17:21.438 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.max_background_flushes: -1 2024-02-09T17:17:21.438 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Compression algorithms supported: 2024-02-09T17:17:21.438 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-09T17:17:21.438 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: kZSTD supported: 0 2024-02-09T17:17:21.438 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: kXpressCompression supported: 0 2024-02-09T17:17:21.438 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: kLZ4HCCompression supported: 1 2024-02-09T17:17:21.438 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: kZlibCompression supported: 1 2024-02-09T17:17:21.438 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: kSnappyCompression supported: 1 2024-02-09T17:17:21.439 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: kLZ4Compression supported: 1 2024-02-09T17:17:21.439 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: kBZip2Compression supported: 0 2024-02-09T17:17:21.439 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-09T17:17:21.439 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-09T17:17:21.439 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-02-09T17:17:21.439 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-09T17:17:21.439 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-09T17:17:21.439 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.merge_operator: 2024-02-09T17:17:21.439 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.compaction_filter: None 2024-02-09T17:17:21.439 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.compaction_filter_factory: None 2024-02-09T17:17:21.439 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.sst_partitioner_factory: None 2024-02-09T17:17:21.439 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-09T17:17:21.440 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-09T17:17:21.440 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55b87650f9c0) 2024-02-09T17:17:21.440 INFO:journalctl@ceph.mon.b.smithi162.stdout: cache_index_and_filter_blocks: 1 2024-02-09T17:17:21.440 INFO:journalctl@ceph.mon.b.smithi162.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-09T17:17:21.440 INFO:journalctl@ceph.mon.b.smithi162.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-09T17:17:21.440 INFO:journalctl@ceph.mon.b.smithi162.stdout: pin_top_level_index_and_filter: 1 2024-02-09T17:17:21.440 INFO:journalctl@ceph.mon.b.smithi162.stdout: index_type: 0 2024-02-09T17:17:21.440 INFO:journalctl@ceph.mon.b.smithi162.stdout: data_block_index_type: 0 2024-02-09T17:17:21.440 INFO:journalctl@ceph.mon.b.smithi162.stdout: index_shortening: 1 2024-02-09T17:17:21.440 INFO:journalctl@ceph.mon.b.smithi162.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-09T17:17:21.440 INFO:journalctl@ceph.mon.b.smithi162.stdout: checksum: 4 2024-02-09T17:17:21.440 INFO:journalctl@ceph.mon.b.smithi162.stdout: no_block_cache: 0 2024-02-09T17:17:21.440 INFO:journalctl@ceph.mon.b.smithi162.stdout: block_cache: 0x55b876513090 2024-02-09T17:17:21.441 INFO:journalctl@ceph.mon.b.smithi162.stdout: block_cache_name: BinnedLRUCache 2024-02-09T17:17:21.441 INFO:journalctl@ceph.mon.b.smithi162.stdout: block_cache_options: 2024-02-09T17:17:21.441 INFO:journalctl@ceph.mon.b.smithi162.stdout: capacity : 536870912 2024-02-09T17:17:21.441 INFO:journalctl@ceph.mon.b.smithi162.stdout: num_shard_bits : 4 2024-02-09T17:17:21.441 INFO:journalctl@ceph.mon.b.smithi162.stdout: strict_capacity_limit : 0 2024-02-09T17:17:21.441 INFO:journalctl@ceph.mon.b.smithi162.stdout: high_pri_pool_ratio: 0.000 2024-02-09T17:17:21.441 INFO:journalctl@ceph.mon.b.smithi162.stdout: block_cache_compressed: (nil) 2024-02-09T17:17:21.441 INFO:journalctl@ceph.mon.b.smithi162.stdout: persistent_cache: (nil) 2024-02-09T17:17:21.441 INFO:journalctl@ceph.mon.b.smithi162.stdout: block_size: 4096 2024-02-09T17:17:21.441 INFO:journalctl@ceph.mon.b.smithi162.stdout: block_size_deviation: 10 2024-02-09T17:17:21.441 INFO:journalctl@ceph.mon.b.smithi162.stdout: block_restart_interval: 16 2024-02-09T17:17:21.441 INFO:journalctl@ceph.mon.b.smithi162.stdout: index_block_restart_interval: 1 2024-02-09T17:17:21.441 INFO:journalctl@ceph.mon.b.smithi162.stdout: metadata_block_size: 4096 2024-02-09T17:17:21.442 INFO:journalctl@ceph.mon.b.smithi162.stdout: partition_filters: 0 2024-02-09T17:17:21.442 INFO:journalctl@ceph.mon.b.smithi162.stdout: use_delta_encoding: 1 2024-02-09T17:17:21.442 INFO:journalctl@ceph.mon.b.smithi162.stdout: filter_policy: bloomfilter 2024-02-09T17:17:21.442 INFO:journalctl@ceph.mon.b.smithi162.stdout: whole_key_filtering: 1 2024-02-09T17:17:21.442 INFO:journalctl@ceph.mon.b.smithi162.stdout: verify_compression: 0 2024-02-09T17:17:21.442 INFO:journalctl@ceph.mon.b.smithi162.stdout: read_amp_bytes_per_bit: 0 2024-02-09T17:17:21.442 INFO:journalctl@ceph.mon.b.smithi162.stdout: format_version: 5 2024-02-09T17:17:21.442 INFO:journalctl@ceph.mon.b.smithi162.stdout: enable_index_compression: 1 2024-02-09T17:17:21.442 INFO:journalctl@ceph.mon.b.smithi162.stdout: block_align: 0 2024-02-09T17:17:21.442 INFO:journalctl@ceph.mon.b.smithi162.stdout: max_auto_readahead_size: 262144 2024-02-09T17:17:21.442 INFO:journalctl@ceph.mon.b.smithi162.stdout: prepopulate_block_cache: 0 2024-02-09T17:17:21.442 INFO:journalctl@ceph.mon.b.smithi162.stdout: initial_auto_readahead_size: 8192 2024-02-09T17:17:21.442 INFO:journalctl@ceph.mon.b.smithi162.stdout: num_file_reads_for_auto_readahead: 2 2024-02-09T17:17:21.442 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.write_buffer_size: 33554432 2024-02-09T17:17:21.443 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.max_write_buffer_number: 2 2024-02-09T17:17:21.443 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.compression: NoCompression 2024-02-09T17:17:21.443 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.bottommost_compression: Disabled 2024-02-09T17:17:21.443 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.prefix_extractor: nullptr 2024-02-09T17:17:21.443 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-09T17:17:21.443 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.num_levels: 7 2024-02-09T17:17:21.443 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-09T17:17:21.443 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-09T17:17:21.443 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-09T17:17:21.443 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-09T17:17:21.443 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-09T17:17:21.443 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-09T17:17:21.443 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-09T17:17:21.444 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-09T17:17:21.444 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-09T17:17:21.444 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-09T17:17:21.444 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-09T17:17:21.444 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-09T17:17:21.444 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-09T17:17:21.444 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.compression_opts.level: 32767 2024-02-09T17:17:21.444 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.compression_opts.strategy: 0 2024-02-09T17:17:21.444 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-09T17:17:21.444 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-09T17:17:21.444 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-09T17:17:21.444 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-09T17:17:21.445 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.compression_opts.enabled: false 2024-02-09T17:17:21.445 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-09T17:17:21.445 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-09T17:17:21.445 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-09T17:17:21.445 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-09T17:17:21.445 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.target_file_size_base: 67108864 2024-02-09T17:17:21.445 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-09T17:17:21.445 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-09T17:17:21.445 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-09T17:17:21.445 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-09T17:17:21.445 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-09T17:17:21.445 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-09T17:17:21.445 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-09T17:17:21.445 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-09T17:17:21.446 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-09T17:17:21.446 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-09T17:17:21.446 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-09T17:17:21.446 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-09T17:17:21.446 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-09T17:17:21.446 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-09T17:17:21.446 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.arena_block_size: 1048576 2024-02-09T17:17:21.446 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-09T17:17:21.446 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-09T17:17:21.446 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.disable_auto_compactions: 0 2024-02-09T17:17:21.446 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-09T17:17:21.446 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-09T17:17:21.447 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-09T17:17:21.447 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-09T17:17:21.447 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-09T17:17:21.447 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-09T17:17:21.447 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-09T17:17:21.447 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-09T17:17:21.447 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-09T17:17:21.447 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-09T17:17:21.447 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.table_properties_collectors: 2024-02-09T17:17:21.447 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.inplace_update_support: 0 2024-02-09T17:17:21.447 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-09T17:17:21.447 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-09T17:17:21.448 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-09T17:17:21.448 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-09T17:17:21.448 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.bloom_locality: 0 2024-02-09T17:17:21.448 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.max_successive_merges: 0 2024-02-09T17:17:21.448 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-09T17:17:21.448 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.paranoid_file_checks: 0 2024-02-09T17:17:21.448 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.force_consistency_checks: 1 2024-02-09T17:17:21.448 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.report_bg_io_stats: 0 2024-02-09T17:17:21.448 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.ttl: 2592000 2024-02-09T17:17:21.448 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-09T17:17:21.448 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-09T17:17:21.448 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-09T17:17:21.448 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.enable_blob_files: false 2024-02-09T17:17:21.449 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.min_blob_size: 0 2024-02-09T17:17:21.449 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.blob_file_size: 268435456 2024-02-09T17:17:21.449 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-09T17:17:21.449 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-09T17:17:21.449 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-09T17:17:21.449 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-09T17:17:21.449 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-09T17:17:21.449 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.blob_file_starting_level: 0 2024-02-09T17:17:21.449 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-09T17:17:21.449 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-09T17:17:21.449 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-02-09T17:17:21.449 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 458ce824-ed8f-4cb5-b4ea-0caa9a6da4e5 2024-02-09T17:17:21.450 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707499041136360, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-02-09T17:17:21.450 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-02-09T17:17:21.450 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707499041136763, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1707499041, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "458ce824-ed8f-4cb5-b4ea-0caa9a6da4e5", "db_session_id": "LHTK4IEX8MPX61EJ2EXI", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-02-09T17:17:21.450 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707499041136842, "job": 1, "event": "recovery_finished"} 2024-02-09T17:17:21.450 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-02-09T17:17:21.450 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-09T17:17:21.450 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55b876604000 2024-02-09T17:17:21.450 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: DB pointer 0x55b8765ee000 2024-02-09T17:17:21.450 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-09T17:17:21.450 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-09T17:17:21.450 INFO:journalctl@ceph.mon.b.smithi162.stdout: ** DB Stats ** 2024-02-09T17:17:21.451 INFO:journalctl@ceph.mon.b.smithi162.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-09T17:17:21.451 INFO:journalctl@ceph.mon.b.smithi162.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-09T17:17:21.451 INFO:journalctl@ceph.mon.b.smithi162.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-09T17:17:21.451 INFO:journalctl@ceph.mon.b.smithi162.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-09T17:17:21.451 INFO:journalctl@ceph.mon.b.smithi162.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-09T17:17:21.451 INFO:journalctl@ceph.mon.b.smithi162.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-09T17:17:21.451 INFO:journalctl@ceph.mon.b.smithi162.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-09T17:17:21.451 INFO:journalctl@ceph.mon.b.smithi162.stdout: 2024-02-09T17:17:21.451 INFO:journalctl@ceph.mon.b.smithi162.stdout: ** Compaction Stats [default] ** 2024-02-09T17:17:21.451 INFO:journalctl@ceph.mon.b.smithi162.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-09T17:17:21.451 INFO:journalctl@ceph.mon.b.smithi162.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-09T17:17:21.452 INFO:journalctl@ceph.mon.b.smithi162.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-09T17:17:21.452 INFO:journalctl@ceph.mon.b.smithi162.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-09T17:17:21.452 INFO:journalctl@ceph.mon.b.smithi162.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-09T17:17:21.452 INFO:journalctl@ceph.mon.b.smithi162.stdout: 2024-02-09T17:17:21.452 INFO:journalctl@ceph.mon.b.smithi162.stdout: ** Compaction Stats [default] ** 2024-02-09T17:17:21.452 INFO:journalctl@ceph.mon.b.smithi162.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-09T17:17:21.452 INFO:journalctl@ceph.mon.b.smithi162.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-09T17:17:21.452 INFO:journalctl@ceph.mon.b.smithi162.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-09T17:17:21.452 INFO:journalctl@ceph.mon.b.smithi162.stdout: 2024-02-09T17:17:21.452 INFO:journalctl@ceph.mon.b.smithi162.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-09T17:17:21.453 INFO:journalctl@ceph.mon.b.smithi162.stdout: 2024-02-09T17:17:21.453 INFO:journalctl@ceph.mon.b.smithi162.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-09T17:17:21.453 INFO:journalctl@ceph.mon.b.smithi162.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-09T17:17:21.453 INFO:journalctl@ceph.mon.b.smithi162.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-09T17:17:21.453 INFO:journalctl@ceph.mon.b.smithi162.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-09T17:17:21.453 INFO:journalctl@ceph.mon.b.smithi162.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-09T17:17:21.453 INFO:journalctl@ceph.mon.b.smithi162.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-09T17:17:21.453 INFO:journalctl@ceph.mon.b.smithi162.stdout: Cumulative compaction: 0.00 GB write, 0.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-09T17:17:21.453 INFO:journalctl@ceph.mon.b.smithi162.stdout: Interval compaction: 0.00 GB write, 0.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-09T17:17:21.453 INFO:journalctl@ceph.mon.b.smithi162.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-09T17:17:21.453 INFO:journalctl@ceph.mon.b.smithi162.stdout: Block cache BinnedLRUCache@0x55b876513090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-02-09T17:17:21.453 INFO:journalctl@ceph.mon.b.smithi162.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-02-09T17:17:21.454 INFO:journalctl@ceph.mon.b.smithi162.stdout: 2024-02-09T17:17:21.454 INFO:journalctl@ceph.mon.b.smithi162.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-09T17:17:21.454 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-02-09T17:17:21.454 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: using public_addr v2:172.21.15.162:0/0 -> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-02-09T17:17:21.454 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: starting mon.b rank -1 at public addrs [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] at bind addrs [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid a69c870a-c76e-11ee-95b7-87774f69a715 2024-02-09T17:17:21.454 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: mon.b@-1(???) e0 preinit fsid a69c870a-c76e-11ee-95b7-87774f69a715 2024-02-09T17:17:21.454 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: [db/db_impl/db_impl_write.cc:2098] [default] New memtable created with log file: #13. Immutable memtables: 0. 2024-02-09T17:17:21.454 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: (Original Log Time 2024/02/09-17:17:21.153914) [db/db_impl/db_impl_compaction_flush.cc:2832] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 2024-02-09T17:17:21.454 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: [db/flush_job.cc:856] [default] [JOB 3] Flushing memtable with next log file: 13 2024-02-09T17:17:21.454 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707499041154026, "job": 3, "event": "flush_started", "num_memtables": 1, "num_entries": 1154, "num_deletes": 21, "total_data_size": 2288798, "memory_usage": 2310672, "flush_reason": "Manual Compaction"} 2024-02-09T17:17:21.454 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: rocksdb: [db/flush_job.cc:885] [default] [JOB 3] Level-0 flush table #14: started 2024-02-09T17:17:21.455 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: mon.b@-1(synchronizing).mds e1 new map 2024-02-09T17:17:21.455 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: mon.b@-1(synchronizing).mds e1 print_map 2024-02-09T17:17:21.455 INFO:journalctl@ceph.mon.b.smithi162.stdout: e1 2024-02-09T17:17:21.455 INFO:journalctl@ceph.mon.b.smithi162.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-09T17:17:21.455 INFO:journalctl@ceph.mon.b.smithi162.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-02-09T17:17:21.455 INFO:journalctl@ceph.mon.b.smithi162.stdout: legacy client fscid: -1 2024-02-09T17:17:21.455 INFO:journalctl@ceph.mon.b.smithi162.stdout: 2024-02-09T17:17:21.455 INFO:journalctl@ceph.mon.b.smithi162.stdout: No filesystems configured 2024-02-09T17:17:21.455 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-02-09T17:17:21.455 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-02-09T17:17:21.455 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-02-09T17:17:21.456 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-02-09T17:17:21.456 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-02-09T17:17:21.456 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-02-09T17:17:21.456 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-09T17:17:21.456 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-09T17:17:21.456 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-09T17:17:21.456 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-09T17:17:21.456 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:17:21.456 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:21.456 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:21.456 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:21.456 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:21.457 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:21.457 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi162", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:17:21.457 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:21.458 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:17:21.458 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: Updating smithi162:/etc/ceph/ceph.conf 2024-02-09T17:17:21.458 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: Updating smithi162:/var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/config/ceph.conf 2024-02-09T17:17:21.458 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-02-09T17:17:21.458 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: Updating smithi162:/var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/config/ceph.client.admin.keyring 2024-02-09T17:17:21.458 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:21.458 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:21.458 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:21.459 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi179", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:17:21.459 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: Deploying cephadm binary to smithi179 2024-02-09T17:17:21.459 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:21.459 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:21.459 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:21.459 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:17:21.459 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: Added host smithi179 2024-02-09T17:17:21.459 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:21.459 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:21.459 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:21.459 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:21.459 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:17:21.460 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:21.460 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:21.460 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:21.460 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:21.460 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:21.460 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi179", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:17:21.460 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:21.460 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:17:21.460 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: Updating smithi179:/etc/ceph/ceph.conf 2024-02-09T17:17:21.460 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: Updating smithi179:/var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/config/ceph.conf 2024-02-09T17:17:21.461 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: Updating smithi179:/etc/ceph/ceph.client.admin.keyring 2024-02-09T17:17:21.461 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:21.461 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: Updating smithi179:/var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/config/ceph.client.admin.keyring 2024-02-09T17:17:21.461 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:21.461 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:21.461 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:21.461 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2579465243' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-09T17:17:21.461 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2579465243' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-09T17:17:21.461 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: osdmap e4: 0 total, 0 up, 0 in 2024-02-09T17:17:21.461 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:21.462 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:21.462 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:21.462 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:17:21.462 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:21.462 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:17:21.462 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:21.462 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-09T17:17:21.462 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:21.462 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi099:172.21.15.99=a;smithi162:172.21.15.162=b;smithi179:172.21.15.179=c", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:17:21.462 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: Saving service mon spec with placement smithi099:172.21.15.99=a;smithi162:172.21.15.162=b;smithi179:172.21.15.179=c;count:3 2024-02-09T17:17:21.462 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: Deploying daemon mon.c on smithi179 2024-02-09T17:17:21.463 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:21.463 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:21.463 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='client.? 172.21.15.179:0/2948853760' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-09T17:17:21.463 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:21.463 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:17:21.463 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:17:21.463 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: mon.a calling monitor election 2024-02-09T17:17:21.463 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:17:21.463 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:21.463 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:17:21.463 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: mon.c calling monitor election 2024-02-09T17:17:21.464 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:17:21.464 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:21.464 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:17:21.464 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:17:21.464 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-09T17:17:21.464 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: monmap e2: 2 mons at {a=[v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0],c=[v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-09T17:17:21.464 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: fsmap 2024-02-09T17:17:21.464 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: osdmap e4: 0 total, 0 up, 0 in 2024-02-09T17:17:21.464 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: mgrmap e13: y(active, since 53s) 2024-02-09T17:17:21.464 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: overall HEALTH_OK 2024-02-09T17:17:21.464 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:21.465 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:21.465 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:21.465 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-09T17:17:21.465 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:21.465 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: Deploying daemon mon.b on smithi162 2024-02-09T17:17:21.465 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='client.? 172.21.15.179:0/3248729134' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-09T17:17:21.465 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:21.465 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:17:21.465 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 ceph-mon[44304]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-02-09T17:17:21.465 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 bash[44266]: c6daea2ab1df632ac630a3349a001e8e5566f69161f8f8aa64c0417d14b730a3 2024-02-09T17:17:21.465 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:21 smithi162 systemd[1]: Started Ceph mon.b for a69c870a-c76e-11ee-95b7-87774f69a715. 2024-02-09T17:17:26.535 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-02-09T17:17:26.535 INFO:teuthology.orchestra.run.smithi179.stdout:{"epoch":3,"fsid":"a69c870a-c76e-11ee-95b7-87774f69a715","modified":"2024-02-09T17:17:21.159235Z","created":"2024-02-09T17:15:23.317650Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:3300","nonce":0},{"type":"v1","addr":"172.21.15.99:6789","nonce":0}]},"addr":"172.21.15.99:6789/0","public_addr":"172.21.15.99:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:3300","nonce":0},{"type":"v1","addr":"172.21.15.179:6789","nonce":0}]},"addr":"172.21.15.179:6789/0","public_addr":"172.21.15.179:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:3300","nonce":0},{"type":"v1","addr":"172.21.15.162:6789","nonce":0}]},"addr":"172.21.15.162:6789/0","public_addr":"172.21.15.162:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-09T17:17:26.537 INFO:teuthology.orchestra.run.smithi179.stderr:dumped monmap epoch 3 2024-02-09T17:17:26.566 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:26 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:17:26.566 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:26 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:17:26.566 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:26 smithi179 ceph-mon[43975]: mon.a calling monitor election 2024-02-09T17:17:26.566 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:26 smithi179 ceph-mon[43975]: mon.c calling monitor election 2024-02-09T17:17:26.566 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:26 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:17:26.566 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:26 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:17:26.566 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:26 smithi179 ceph-mon[43975]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:26.566 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:26 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:17:26.566 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:26 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:17:26.566 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:26 smithi179 ceph-mon[43975]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:26.566 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:26 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:17:26.567 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:26 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:17:26.567 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:26 smithi179 ceph-mon[43975]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-09T17:17:26.567 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:26 smithi179 ceph-mon[43975]: monmap e3: 3 mons at {a=[v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0],b=[v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0],c=[v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-09T17:17:26.567 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:26 smithi179 ceph-mon[43975]: fsmap 2024-02-09T17:17:26.567 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:26 smithi179 ceph-mon[43975]: osdmap e4: 0 total, 0 up, 0 in 2024-02-09T17:17:26.567 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:26 smithi179 ceph-mon[43975]: mgrmap e13: y(active, since 61s) 2024-02-09T17:17:26.567 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:26 smithi179 ceph-mon[43975]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-02-09T17:17:26.567 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:26 smithi179 ceph-mon[43975]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-02-09T17:17:26.567 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:26 smithi179 ceph-mon[43975]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-02-09T17:17:26.567 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:26 smithi179 ceph-mon[43975]: mon.b (rank 2) addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] is down (out of quorum) 2024-02-09T17:17:26.567 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:26 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:26.567 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:26 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:26.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:26 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:17:26.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:26 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:17:26.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:26 smithi099 ceph-mon[39420]: mon.a calling monitor election 2024-02-09T17:17:26.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:26 smithi099 ceph-mon[39420]: mon.c calling monitor election 2024-02-09T17:17:26.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:26 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:17:26.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:26 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:17:26.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:26 smithi099 ceph-mon[39420]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:26.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:26 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:17:26.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:26 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:17:26.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:26 smithi099 ceph-mon[39420]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:26.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:26 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:17:26.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:26 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:17:26.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:26 smithi099 ceph-mon[39420]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-09T17:17:26.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:26 smithi099 ceph-mon[39420]: monmap e3: 3 mons at {a=[v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0],b=[v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0],c=[v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-09T17:17:26.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:26 smithi099 ceph-mon[39420]: fsmap 2024-02-09T17:17:26.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:26 smithi099 ceph-mon[39420]: osdmap e4: 0 total, 0 up, 0 in 2024-02-09T17:17:26.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:26 smithi099 ceph-mon[39420]: mgrmap e13: y(active, since 61s) 2024-02-09T17:17:26.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:26 smithi099 ceph-mon[39420]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-02-09T17:17:26.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:26 smithi099 ceph-mon[39420]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-02-09T17:17:26.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:26 smithi099 ceph-mon[39420]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-02-09T17:17:26.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:26 smithi099 ceph-mon[39420]: mon.b (rank 2) addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] is down (out of quorum) 2024-02-09T17:17:26.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:26 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:26.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:26 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:27.238 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-02-09T17:17:27.238 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph config generate-minimal-conf 2024-02-09T17:17:27.500 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:17:27.637 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:27 smithi179 ceph-mon[43975]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:27.637 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:27 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:27.637 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:27 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:27.637 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:27 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:17:27.637 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:27 smithi179 ceph-mon[43975]: from='client.? 172.21.15.179:0/3121204540' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-09T17:17:27.637 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:27 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:17:27.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:27 smithi099 ceph-mon[39420]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:27.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:27 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:27.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:27 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:27.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:27 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:17:27.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:27 smithi099 ceph-mon[39420]: from='client.? 172.21.15.179:0/3121204540' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-09T17:17:27.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:27 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:17:28.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:17:28.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:17:28.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: mon.a calling monitor election 2024-02-09T17:17:28.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: mon.c calling monitor election 2024-02-09T17:17:28.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:17:28.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:17:28.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:28.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:17:28.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:17:28.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:28.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:17:28.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:17:28.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-09T17:17:28.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: monmap e3: 3 mons at {a=[v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0],b=[v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0],c=[v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-09T17:17:28.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: fsmap 2024-02-09T17:17:28.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: osdmap e4: 0 total, 0 up, 0 in 2024-02-09T17:17:28.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: mgrmap e13: y(active, since 61s) 2024-02-09T17:17:28.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-02-09T17:17:28.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-02-09T17:17:28.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-02-09T17:17:28.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: mon.b (rank 2) addr [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] is down (out of quorum) 2024-02-09T17:17:28.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:28.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:28.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:28.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:28.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:28.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:17:28.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: from='client.? 172.21.15.179:0/3121204540' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-09T17:17:28.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:17:29.113 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:28 smithi099 ceph-mon[39420]: mon.b calling monitor election 2024-02-09T17:17:29.114 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:28 smithi099 ceph-mon[39420]: Updating smithi099:/etc/ceph/ceph.conf 2024-02-09T17:17:29.114 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:28 smithi099 ceph-mon[39420]: Updating smithi162:/etc/ceph/ceph.conf 2024-02-09T17:17:29.114 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:28 smithi099 ceph-mon[39420]: Updating smithi179:/etc/ceph/ceph.conf 2024-02-09T17:17:29.114 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:28 smithi099 ceph-mon[39420]: mon.b calling monitor election 2024-02-09T17:17:29.114 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:28 smithi099 ceph-mon[39420]: mon.c calling monitor election 2024-02-09T17:17:29.114 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:28 smithi099 ceph-mon[39420]: mon.a calling monitor election 2024-02-09T17:17:29.114 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:28 smithi099 ceph-mon[39420]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-09T17:17:29.114 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:28 smithi099 ceph-mon[39420]: monmap e3: 3 mons at {a=[v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0],b=[v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0],c=[v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-09T17:17:29.114 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:28 smithi099 ceph-mon[39420]: fsmap 2024-02-09T17:17:29.114 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:28 smithi099 ceph-mon[39420]: osdmap e4: 0 total, 0 up, 0 in 2024-02-09T17:17:29.114 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:28 smithi099 ceph-mon[39420]: mgrmap e13: y(active, since 64s) 2024-02-09T17:17:29.114 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:28 smithi099 ceph-mon[39420]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-02-09T17:17:29.114 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:28 smithi099 ceph-mon[39420]: Cluster is now healthy 2024-02-09T17:17:29.115 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:28 smithi099 ceph-mon[39420]: overall HEALTH_OK 2024-02-09T17:17:29.115 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:28 smithi099 ceph-mon[39420]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:29.115 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:28 smithi099 ceph-mon[39420]: Updating smithi099:/var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/config/ceph.conf 2024-02-09T17:17:29.115 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:28 smithi099 ceph-mon[39420]: Updating smithi179:/var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/config/ceph.conf 2024-02-09T17:17:29.115 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:28 smithi099 ceph-mon[39420]: Updating smithi162:/var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/config/ceph.conf 2024-02-09T17:17:29.115 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:28 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:29.115 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:28 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:29.115 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:28 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:29.115 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:28 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:29.115 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:28 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:29.115 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:28 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:29.115 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:28 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:29.115 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:28 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:29.115 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:28 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:29.115 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:28 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:29.115 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:28 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:29.115 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:28 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-09T17:17:29.116 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:28 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-09T17:17:29.116 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:28 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:29.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:28 smithi179 ceph-mon[43975]: mon.b calling monitor election 2024-02-09T17:17:29.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:28 smithi179 ceph-mon[43975]: Updating smithi099:/etc/ceph/ceph.conf 2024-02-09T17:17:29.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:28 smithi179 ceph-mon[43975]: Updating smithi162:/etc/ceph/ceph.conf 2024-02-09T17:17:29.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:28 smithi179 ceph-mon[43975]: Updating smithi179:/etc/ceph/ceph.conf 2024-02-09T17:17:29.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:28 smithi179 ceph-mon[43975]: mon.b calling monitor election 2024-02-09T17:17:29.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:28 smithi179 ceph-mon[43975]: mon.c calling monitor election 2024-02-09T17:17:29.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:28 smithi179 ceph-mon[43975]: mon.a calling monitor election 2024-02-09T17:17:29.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:28 smithi179 ceph-mon[43975]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-09T17:17:29.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:28 smithi179 ceph-mon[43975]: monmap e3: 3 mons at {a=[v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0],b=[v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0],c=[v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-09T17:17:29.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:28 smithi179 ceph-mon[43975]: fsmap 2024-02-09T17:17:29.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:28 smithi179 ceph-mon[43975]: osdmap e4: 0 total, 0 up, 0 in 2024-02-09T17:17:29.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:28 smithi179 ceph-mon[43975]: mgrmap e13: y(active, since 64s) 2024-02-09T17:17:29.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:28 smithi179 ceph-mon[43975]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-02-09T17:17:29.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:28 smithi179 ceph-mon[43975]: Cluster is now healthy 2024-02-09T17:17:29.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:28 smithi179 ceph-mon[43975]: overall HEALTH_OK 2024-02-09T17:17:29.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:28 smithi179 ceph-mon[43975]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:29.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:28 smithi179 ceph-mon[43975]: Updating smithi099:/var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/config/ceph.conf 2024-02-09T17:17:29.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:28 smithi179 ceph-mon[43975]: Updating smithi179:/var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/config/ceph.conf 2024-02-09T17:17:29.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:28 smithi179 ceph-mon[43975]: Updating smithi162:/var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/config/ceph.conf 2024-02-09T17:17:29.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:28 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:29.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:28 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:29.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:28 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:29.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:28 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:29.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:28 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:29.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:28 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:29.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:28 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:29.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:28 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:29.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:28 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:29.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:28 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:29.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:28 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:29.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:28 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-09T17:17:29.426 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:28 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-09T17:17:29.426 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:28 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:29.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: mon.b calling monitor election 2024-02-09T17:17:29.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: Updating smithi099:/etc/ceph/ceph.conf 2024-02-09T17:17:29.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: Updating smithi162:/etc/ceph/ceph.conf 2024-02-09T17:17:29.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: Updating smithi179:/etc/ceph/ceph.conf 2024-02-09T17:17:29.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: mon.b calling monitor election 2024-02-09T17:17:29.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: mon.c calling monitor election 2024-02-09T17:17:29.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: mon.a calling monitor election 2024-02-09T17:17:29.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-09T17:17:29.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: monmap e3: 3 mons at {a=[v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0],b=[v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0],c=[v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-09T17:17:29.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: fsmap 2024-02-09T17:17:29.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: osdmap e4: 0 total, 0 up, 0 in 2024-02-09T17:17:29.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: mgrmap e13: y(active, since 64s) 2024-02-09T17:17:29.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-02-09T17:17:29.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: Cluster is now healthy 2024-02-09T17:17:29.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: overall HEALTH_OK 2024-02-09T17:17:29.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:29.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: Updating smithi099:/var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/config/ceph.conf 2024-02-09T17:17:29.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: Updating smithi179:/var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/config/ceph.conf 2024-02-09T17:17:29.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: Updating smithi162:/var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/config/ceph.conf 2024-02-09T17:17:29.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:29.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:29.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:29.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:29.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:29.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:29.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:29.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:29.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:29.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:29.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:29.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-09T17:17:29.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-09T17:17:29.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:28 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:29.750 INFO:teuthology.orchestra.run.smithi099.stdout:# minimal ceph.conf for a69c870a-c76e-11ee-95b7-87774f69a715 2024-02-09T17:17:29.750 INFO:teuthology.orchestra.run.smithi099.stdout:[global] 2024-02-09T17:17:29.751 INFO:teuthology.orchestra.run.smithi099.stdout: fsid = a69c870a-c76e-11ee-95b7-87774f69a715 2024-02-09T17:17:29.751 INFO:teuthology.orchestra.run.smithi099.stdout: mon_host = [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] 2024-02-09T17:17:30.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:29 smithi099 ceph-mon[39420]: Reconfiguring mon.a (unknown last config time)... 2024-02-09T17:17:30.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:29 smithi099 ceph-mon[39420]: Reconfiguring daemon mon.a on smithi099 2024-02-09T17:17:30.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:29 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:17:30.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:29 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:30.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:29 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/35441120' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:30.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:29 smithi179 ceph-mon[43975]: Reconfiguring mon.a (unknown last config time)... 2024-02-09T17:17:30.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:29 smithi179 ceph-mon[43975]: Reconfiguring daemon mon.a on smithi099 2024-02-09T17:17:30.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:29 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:17:30.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:29 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:30.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:29 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/35441120' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:30.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:29 smithi162 ceph-mon[44304]: Reconfiguring mon.a (unknown last config time)... 2024-02-09T17:17:30.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:29 smithi162 ceph-mon[44304]: Reconfiguring daemon mon.a on smithi099 2024-02-09T17:17:30.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:29 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:17:30.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:29 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:30.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:29 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/35441120' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:30.946 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-02-09T17:17:30.946 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-02-09T17:17:30.946 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/ceph/ceph.conf 2024-02-09T17:17:30.976 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-02-09T17:17:30.976 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-09T17:17:31.043 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-02-09T17:17:31.043 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/ceph/ceph.conf 2024-02-09T17:17:31.074 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-02-09T17:17:31.074 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-09T17:17:31.144 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-02-09T17:17:31.144 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/ceph/ceph.conf 2024-02-09T17:17:31.180 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-02-09T17:17:31.180 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-09T17:17:31.252 INFO:tasks.cephadm:Adding mgr.y on smithi099 2024-02-09T17:17:31.252 INFO:tasks.cephadm:Adding mgr.x on smithi162 2024-02-09T17:17:31.252 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph orch apply mgr '2;smithi099=y;smithi162=x' 2024-02-09T17:17:31.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:31 smithi162 ceph-mon[44304]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:31.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:31 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:31.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:31 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:31.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:31 smithi162 ceph-mon[44304]: Reconfiguring mon.b (monmap changed)... 2024-02-09T17:17:31.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:31 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-09T17:17:31.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:31 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-09T17:17:31.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:31 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:31.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:31 smithi162 ceph-mon[44304]: Reconfiguring daemon mon.b on smithi162 2024-02-09T17:17:31.549 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:17:31.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:31 smithi179 ceph-mon[43975]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:31.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:31 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:31.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:31 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:31.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:31 smithi179 ceph-mon[43975]: Reconfiguring mon.b (monmap changed)... 2024-02-09T17:17:31.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:31 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-09T17:17:31.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:31 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-09T17:17:31.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:31 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:31.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:31 smithi179 ceph-mon[43975]: Reconfiguring daemon mon.b on smithi162 2024-02-09T17:17:31.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:31 smithi099 ceph-mon[39420]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:31.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:31 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:31.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:31 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:31.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:31 smithi099 ceph-mon[39420]: Reconfiguring mon.b (monmap changed)... 2024-02-09T17:17:31.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:31 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-09T17:17:31.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:31 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-09T17:17:31.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:31 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:31.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:31 smithi099 ceph-mon[39420]: Reconfiguring daemon mon.b on smithi162 2024-02-09T17:17:33.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:32 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:33.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:32 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:33.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:32 smithi179 ceph-mon[43975]: Reconfiguring mon.c (monmap changed)... 2024-02-09T17:17:33.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:32 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-09T17:17:33.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:32 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-09T17:17:33.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:32 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:33.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:32 smithi179 ceph-mon[43975]: Reconfiguring daemon mon.c on smithi179 2024-02-09T17:17:33.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:32 smithi179 ceph-mon[43975]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:33.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:32 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:33.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:32 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:33.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:32 smithi162 ceph-mon[44304]: Reconfiguring mon.c (monmap changed)... 2024-02-09T17:17:33.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:32 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-09T17:17:33.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:32 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-09T17:17:33.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:32 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:33.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:32 smithi162 ceph-mon[44304]: Reconfiguring daemon mon.c on smithi179 2024-02-09T17:17:33.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:32 smithi162 ceph-mon[44304]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:33.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:32 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:33.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:32 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:33.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:32 smithi099 ceph-mon[39420]: Reconfiguring mon.c (monmap changed)... 2024-02-09T17:17:33.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:32 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-09T17:17:33.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:32 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-09T17:17:33.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:32 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:33.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:32 smithi099 ceph-mon[39420]: Reconfiguring daemon mon.c on smithi179 2024-02-09T17:17:33.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:32 smithi099 ceph-mon[39420]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:33.747 INFO:teuthology.orchestra.run.smithi179.stdout:Scheduled mgr update... 2024-02-09T17:17:34.415 DEBUG:teuthology.orchestra.run.smithi162:mgr.x> sudo journalctl -f -n 0 -u ceph-a69c870a-c76e-11ee-95b7-87774f69a715@mgr.x.service 2024-02-09T17:17:34.417 INFO:tasks.cephadm:Deploying OSDs... 2024-02-09T17:17:34.417 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-02-09T17:17:34.417 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/scratch_devs of=/dev/stdout 2024-02-09T17:17:34.440 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-09T17:17:34.440 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_1 2024-02-09T17:17:34.450 INFO:journalctl@ceph.mgr.x.smithi162.stdout:-- Logs begin at Fri 2024-02-09 17:04:37 UTC. -- 2024-02-09T17:17:34.500 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-09T17:17:34.501 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-09T17:17:34.501 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 6h/6d Inode: 166470 Links: 1 2024-02-09T17:17:34.501 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-09T17:17:34.501 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2024-02-09T17:17:34.501 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2024-02-09 17:16:37.659598964 +0000 2024-02-09T17:17:34.501 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2024-02-09 17:10:51.280610549 +0000 2024-02-09T17:17:34.501 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2024-02-09 17:10:51.280610549 +0000 2024-02-09T17:17:34.501 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: - 2024-02-09T17:17:34.501 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-09T17:17:34.570 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2024-02-09T17:17:34.570 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2024-02-09T17:17:34.570 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000218852 s, 2.3 MB/s 2024-02-09T17:17:34.571 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-09T17:17:34.633 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_2 2024-02-09T17:17:34.692 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-09T17:17:34.692 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-09T17:17:34.692 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 6h/6d Inode: 164474 Links: 1 2024-02-09T17:17:34.692 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-09T17:17:34.692 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2024-02-09T17:17:34.693 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2024-02-09 17:16:37.715597666 +0000 2024-02-09T17:17:34.693 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2024-02-09 17:10:51.625602503 +0000 2024-02-09T17:17:34.693 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2024-02-09 17:10:51.625602503 +0000 2024-02-09T17:17:34.693 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: - 2024-02-09T17:17:34.693 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-09T17:17:34.759 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2024-02-09T17:17:34.759 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2024-02-09T17:17:34.759 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000245166 s, 2.1 MB/s 2024-02-09T17:17:34.760 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-09T17:17:34.820 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_3 2024-02-09T17:17:34.880 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-09T17:17:34.880 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-09T17:17:34.880 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 6h/6d Inode: 166603 Links: 1 2024-02-09T17:17:34.880 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-09T17:17:34.880 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2024-02-09T17:17:34.880 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2024-02-09 17:16:37.765596508 +0000 2024-02-09T17:17:34.880 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2024-02-09 17:10:51.983594155 +0000 2024-02-09T17:17:34.880 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2024-02-09 17:10:51.983594155 +0000 2024-02-09T17:17:34.880 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: - 2024-02-09T17:17:34.881 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-09T17:17:34.944 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:34 smithi099 ceph-mon[39420]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi099=y;smithi162=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:17:34.945 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:34 smithi099 ceph-mon[39420]: Saving service mgr spec with placement smithi099=y;smithi162=x;count:2 2024-02-09T17:17:34.945 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:34 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:34.945 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:34 smithi099 ceph-mon[39420]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:34.946 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2024-02-09T17:17:34.946 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2024-02-09T17:17:34.946 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000179137 s, 2.9 MB/s 2024-02-09T17:17:34.948 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-09T17:17:35.007 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_4 2024-02-09T17:17:35.065 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-09T17:17:35.065 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-09T17:17:35.065 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 6h/6d Inode: 166686 Links: 1 2024-02-09T17:17:35.065 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-09T17:17:35.065 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2024-02-09T17:17:35.065 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2024-02-09 17:16:37.814595373 +0000 2024-02-09T17:17:35.065 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2024-02-09 17:10:52.336585923 +0000 2024-02-09T17:17:35.065 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2024-02-09 17:10:52.336585923 +0000 2024-02-09T17:17:35.065 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: - 2024-02-09T17:17:35.065 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-09T17:17:35.130 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2024-02-09T17:17:35.130 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2024-02-09T17:17:35.131 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000226097 s, 2.3 MB/s 2024-02-09T17:17:35.131 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-09T17:17:35.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:34 smithi179 ceph-mon[43975]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi099=y;smithi162=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:17:35.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:34 smithi179 ceph-mon[43975]: Saving service mgr spec with placement smithi099=y;smithi162=x;count:2 2024-02-09T17:17:35.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:34 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:35.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:34 smithi179 ceph-mon[43975]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:35.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:34 smithi162 ceph-mon[44304]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi099=y;smithi162=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:17:35.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:34 smithi162 ceph-mon[44304]: Saving service mgr spec with placement smithi099=y;smithi162=x;count:2 2024-02-09T17:17:35.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:34 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:35.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:34 smithi162 ceph-mon[44304]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:35.191 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-02-09T17:17:35.191 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/scratch_devs of=/dev/stdout 2024-02-09T17:17:35.209 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-09T17:17:35.210 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_1 2024-02-09T17:17:35.268 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-09T17:17:35.268 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-09T17:17:35.268 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 6h/6d Inode: 169035 Links: 1 2024-02-09T17:17:35.268 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-09T17:17:35.269 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-02-09T17:17:35.269 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-02-09 17:17:20.512771477 +0000 2024-02-09T17:17:35.269 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-02-09 17:10:52.036512813 +0000 2024-02-09T17:17:35.269 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-02-09 17:10:52.036512813 +0000 2024-02-09T17:17:35.269 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: - 2024-02-09T17:17:35.269 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-09T17:17:35.335 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-02-09T17:17:35.335 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-02-09T17:17:35.335 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000202082 s, 2.5 MB/s 2024-02-09T17:17:35.336 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-09T17:17:35.395 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_2 2024-02-09T17:17:35.453 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-09T17:17:35.453 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-09T17:17:35.454 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 6h/6d Inode: 160707 Links: 1 2024-02-09T17:17:35.454 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-09T17:17:35.454 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-02-09T17:17:35.454 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-02-09 17:17:20.512771477 +0000 2024-02-09T17:17:35.454 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-02-09 17:10:52.364506328 +0000 2024-02-09T17:17:35.454 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-02-09 17:10:52.364506328 +0000 2024-02-09T17:17:35.454 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: - 2024-02-09T17:17:35.454 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-09T17:17:35.520 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-02-09T17:17:35.520 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-02-09T17:17:35.520 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000217394 s, 2.4 MB/s 2024-02-09T17:17:35.521 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-09T17:17:35.581 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_3 2024-02-09T17:17:35.640 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-09T17:17:35.640 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-09T17:17:35.640 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 6h/6d Inode: 169160 Links: 1 2024-02-09T17:17:35.640 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-09T17:17:35.640 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-02-09T17:17:35.640 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-02-09 17:17:20.513771457 +0000 2024-02-09T17:17:35.641 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-02-09 17:10:52.692499844 +0000 2024-02-09T17:17:35.641 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-02-09 17:10:52.692499844 +0000 2024-02-09T17:17:35.641 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: - 2024-02-09T17:17:35.641 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-09T17:17:35.705 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-02-09T17:17:35.706 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-02-09T17:17:35.706 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000190917 s, 2.7 MB/s 2024-02-09T17:17:35.707 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-09T17:17:35.765 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_4 2024-02-09T17:17:35.824 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-09T17:17:35.824 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-09T17:17:35.824 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 6h/6d Inode: 166673 Links: 1 2024-02-09T17:17:35.825 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-09T17:17:35.825 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-02-09T17:17:35.825 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-02-09 17:17:20.513771457 +0000 2024-02-09T17:17:35.825 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-02-09 17:10:53.033493103 +0000 2024-02-09T17:17:35.825 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-02-09 17:10:53.033493103 +0000 2024-02-09T17:17:35.825 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: - 2024-02-09T17:17:35.825 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-09T17:17:35.891 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-02-09T17:17:35.891 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-02-09T17:17:35.891 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000205056 s, 2.5 MB/s 2024-02-09T17:17:35.892 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-09T17:17:35.950 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-02-09T17:17:35.951 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/scratch_devs of=/dev/stdout 2024-02-09T17:17:35.973 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-09T17:17:35.973 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_1 2024-02-09T17:17:36.034 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-09T17:17:36.034 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-09T17:17:36.034 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 6h/6d Inode: 168128 Links: 1 2024-02-09T17:17:36.035 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-09T17:17:36.035 INFO:teuthology.orchestra.run.smithi179.stdout:Context: system_u:object_r:device_t:s0 2024-02-09T17:17:36.035 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2024-02-09 17:17:11.350955968 +0000 2024-02-09T17:17:36.035 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2024-02-09 17:10:52.607161303 +0000 2024-02-09T17:17:36.035 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2024-02-09 17:10:52.607161303 +0000 2024-02-09T17:17:36.035 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: - 2024-02-09T17:17:36.035 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-09T17:17:36.090 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:35 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:36.091 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:35 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:36.091 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:35 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:17:36.091 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:35 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:36.091 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:35 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:17:36.091 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:35 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:36.091 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:35 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-09T17:17:36.091 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:35 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-09T17:17:36.091 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:35 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-09T17:17:36.091 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:35 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:36.091 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:35 smithi162 ceph-mon[44304]: Deploying daemon mgr.x on smithi162 2024-02-09T17:17:36.105 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:35 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:36.105 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:35 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:36.105 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:35 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:17:36.105 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:35 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:36.105 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:35 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:17:36.105 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:35 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:36.105 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:35 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-09T17:17:36.105 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:35 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-09T17:17:36.105 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:35 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-09T17:17:36.105 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:35 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:36.105 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:35 smithi179 ceph-mon[43975]: Deploying daemon mgr.x on smithi162 2024-02-09T17:17:36.108 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2024-02-09T17:17:36.108 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2024-02-09T17:17:36.108 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.000217597 s, 2.4 MB/s 2024-02-09T17:17:36.109 DEBUG:teuthology.orchestra.run.smithi179:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-09T17:17:36.171 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_2 2024-02-09T17:17:36.231 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-09T17:17:36.231 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-09T17:17:36.231 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 6h/6d Inode: 167205 Links: 1 2024-02-09T17:17:36.231 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-09T17:17:36.231 INFO:teuthology.orchestra.run.smithi179.stdout:Context: system_u:object_r:device_t:s0 2024-02-09T17:17:36.231 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2024-02-09 17:17:11.350955968 +0000 2024-02-09T17:17:36.231 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2024-02-09 17:10:52.949153771 +0000 2024-02-09T17:17:36.232 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2024-02-09 17:10:52.949153771 +0000 2024-02-09T17:17:36.232 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: - 2024-02-09T17:17:36.232 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-09T17:17:36.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:35 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:36.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:35 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:36.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:35 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:17:36.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:35 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:36.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:35 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:17:36.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:35 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:36.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:35 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-09T17:17:36.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:35 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-09T17:17:36.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:35 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-09T17:17:36.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:35 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:36.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:35 smithi099 ceph-mon[39420]: Deploying daemon mgr.x on smithi162 2024-02-09T17:17:36.299 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2024-02-09T17:17:36.299 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2024-02-09T17:17:36.299 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.000218637 s, 2.3 MB/s 2024-02-09T17:17:36.300 DEBUG:teuthology.orchestra.run.smithi179:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-09T17:17:36.363 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_3 2024-02-09T17:17:36.422 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-09T17:17:36.422 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-09T17:17:36.422 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 6h/6d Inode: 168244 Links: 1 2024-02-09T17:17:36.422 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-09T17:17:36.422 INFO:teuthology.orchestra.run.smithi179.stdout:Context: system_u:object_r:device_t:s0 2024-02-09T17:17:36.422 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2024-02-09 17:17:11.350955968 +0000 2024-02-09T17:17:36.422 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2024-02-09 17:10:53.296146128 +0000 2024-02-09T17:17:36.423 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2024-02-09 17:10:53.296146128 +0000 2024-02-09T17:17:36.423 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: - 2024-02-09T17:17:36.423 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-09T17:17:36.489 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2024-02-09T17:17:36.489 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2024-02-09T17:17:36.489 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.000203755 s, 2.5 MB/s 2024-02-09T17:17:36.490 DEBUG:teuthology.orchestra.run.smithi179:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-09T17:17:36.550 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_4 2024-02-09T17:17:36.609 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-09T17:17:36.609 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-09T17:17:36.609 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 6h/6d Inode: 165504 Links: 1 2024-02-09T17:17:36.609 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-09T17:17:36.609 INFO:teuthology.orchestra.run.smithi179.stdout:Context: system_u:object_r:device_t:s0 2024-02-09T17:17:36.609 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2024-02-09 17:17:11.350955968 +0000 2024-02-09T17:17:36.609 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2024-02-09 17:10:53.646138421 +0000 2024-02-09T17:17:36.609 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2024-02-09 17:10:53.646138421 +0000 2024-02-09T17:17:36.609 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: - 2024-02-09T17:17:36.610 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-09T17:17:36.675 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2024-02-09T17:17:36.675 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2024-02-09T17:17:36.675 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.000231196 s, 2.2 MB/s 2024-02-09T17:17:36.676 DEBUG:teuthology.orchestra.run.smithi179:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-09T17:17:36.735 INFO:tasks.cephadm:Deploying osd.0 on smithi099 with /dev/vg_nvme/lv_4... 2024-02-09T17:17:36.736 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-09T17:17:36.922 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Feb 09 17:17:36 smithi162 systemd[1]: Starting Ceph mgr.x for a69c870a-c76e-11ee-95b7-87774f69a715... 2024-02-09T17:17:37.005 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:17:37.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:36 smithi179 ceph-mon[43975]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:37.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:36 smithi162 ceph-mon[44304]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:37.178 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Feb 09 17:17:37 smithi162 ceph-mgr[45954]: -- 172.21.15.162:0/3291481827 <== mon.2 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5637cc0381a0 con 0x5637cc00e800 2024-02-09T17:17:37.232 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:36 smithi099 ceph-mon[39420]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:37.678 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Feb 09 17:17:37 smithi162 bash[45916]: ed00418ed4598c0aa49956d69c7e384e4a4ea70156410aec599908cb26f889ab 2024-02-09T17:17:37.678 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Feb 09 17:17:37 smithi162 systemd[1]: Started Ceph mgr.x for a69c870a-c76e-11ee-95b7-87774f69a715. 2024-02-09T17:17:37.678 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Feb 09 17:17:37 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-x[45931]: 2024-02-09T17:17:37.272+0000 7f1e83054200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-09T17:17:37.678 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Feb 09 17:17:37 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-x[45931]: 2024-02-09T17:17:37.442+0000 7f1e83054200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-09T17:17:38.178 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Feb 09 17:17:37 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-x[45931]: 2024-02-09T17:17:37.796+0000 7f1e83054200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-09T17:17:38.610 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Feb 09 17:17:38 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-x[45931]: 2024-02-09T17:17:38.352+0000 7f1e83054200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-09T17:17:38.611 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:38 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:38.611 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:38 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:38.611 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:38 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:38.611 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:38 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:38.611 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:38 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:17:38.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:38 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:38.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:38 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:38.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:38 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:38.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:38 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:38.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:38 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:17:38.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:38 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:38.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:38 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:38.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:38 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:38.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:38 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:38.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:38 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:17:39.084 INFO:teuthology.orchestra.run.smithi099.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-09T17:17:39.084 INFO:teuthology.orchestra.run.smithi099.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-09T17:17:39.084 INFO:teuthology.orchestra.run.smithi099.stderr: stderr: 10+0 records in 2024-02-09T17:17:39.084 INFO:teuthology.orchestra.run.smithi099.stderr:10+0 records out 2024-02-09T17:17:39.084 INFO:teuthology.orchestra.run.smithi099.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0125655 s, 834 MB/s 2024-02-09T17:17:39.084 INFO:teuthology.orchestra.run.smithi099.stderr:--> Zapping successful for: 2024-02-09T17:17:39.368 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Feb 09 17:17:38 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-x[45931]: 2024-02-09T17:17:38.607+0000 7f1e83054200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-09T17:17:39.369 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Feb 09 17:17:39 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-x[45931]: 2024-02-09T17:17:39.043+0000 7f1e83054200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-09T17:17:39.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:39 smithi099 ceph-mon[39420]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:39.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:39 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:39.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:39 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:39.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:39 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:39.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:39 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:17:39.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:39 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:39.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:39 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-09T17:17:39.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:39 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-09T17:17:39.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:39 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:39.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:39 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:39.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:39 smithi179 ceph-mon[43975]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:39.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:39 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:39.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:39 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:39.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:39 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:39.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:39 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:17:39.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:39 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:39.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:39 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-09T17:17:39.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:39 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-09T17:17:39.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:39 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:39.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:39 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:39.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:39 smithi162 ceph-mon[44304]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:39.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:39 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:39.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:39 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:39.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:39 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:39.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:39 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:17:39.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:39 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:39.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:39 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-09T17:17:39.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:39 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-09T17:17:39.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:39 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:39.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:39 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:40.146 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph orch daemon add osd smithi099:vg_nvme/lv_4 2024-02-09T17:17:40.177 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Feb 09 17:17:39 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-x[45931]: 2024-02-09T17:17:39.750+0000 7f1e83054200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-09T17:17:40.177 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Feb 09 17:17:39 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-x[45931]: 2024-02-09T17:17:39.835+0000 7f1e83054200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-09T17:17:40.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:40 smithi179 ceph-mon[43975]: Reconfiguring mgr.y (unknown last config time)... 2024-02-09T17:17:40.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:40 smithi179 ceph-mon[43975]: Reconfiguring daemon mgr.y on smithi099 2024-02-09T17:17:40.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:40 smithi162 ceph-mon[44304]: Reconfiguring mgr.y (unknown last config time)... 2024-02-09T17:17:40.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:40 smithi162 ceph-mon[44304]: Reconfiguring daemon mgr.y on smithi099 2024-02-09T17:17:40.678 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Feb 09 17:17:40 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-x[45931]: 2024-02-09T17:17:40.429+0000 7f1e83054200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-09T17:17:40.678 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Feb 09 17:17:40 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-x[45931]: 2024-02-09T17:17:40.512+0000 7f1e83054200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-09T17:17:40.678 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Feb 09 17:17:40 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-x[45931]: 2024-02-09T17:17:40.599+0000 7f1e83054200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-09T17:17:40.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:40 smithi099 ceph-mon[39420]: Reconfiguring mgr.y (unknown last config time)... 2024-02-09T17:17:40.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:40 smithi099 ceph-mon[39420]: Reconfiguring daemon mgr.y on smithi099 2024-02-09T17:17:40.919 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:17:41.177 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Feb 09 17:17:40 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-x[45931]: 2024-02-09T17:17:40.825+0000 7f1e83054200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-09T17:17:41.178 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Feb 09 17:17:40 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-x[45931]: 2024-02-09T17:17:40.916+0000 7f1e83054200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-09T17:17:41.178 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Feb 09 17:17:41 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-x[45931]: 2024-02-09T17:17:41.011+0000 7f1e83054200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-09T17:17:41.406 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:41 smithi099 ceph-mon[39420]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:41.406 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:41 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:41.407 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:41 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:41.407 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:41 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:17:41.407 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:41 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:41.407 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:41 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:17:41.407 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:41 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:17:41.407 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:41 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:17:41.407 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:41 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:17:41.407 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:41 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:41.407 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:41 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:17:41.407 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:41 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:41.407 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:41 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:17:41.407 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:41 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:41.531 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Feb 09 17:17:41 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-x[45931]: 2024-02-09T17:17:41.244+0000 7f1e83054200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-09T17:17:41.531 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Feb 09 17:17:41 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-x[45931]: 2024-02-09T17:17:41.343+0000 7f1e83054200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-09T17:17:41.531 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:41 smithi162 ceph-mon[44304]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:41.531 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:41 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:41.531 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:41 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:41.531 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:41 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:17:41.531 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:41 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:41.532 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:41 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:17:41.532 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:41 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:17:41.532 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:41 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:17:41.532 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:41 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:17:41.532 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:41 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:41.532 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:41 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:17:41.532 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:41 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:41.532 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:41 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:17:41.532 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:41 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:41.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:41 smithi179 ceph-mon[43975]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:41.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:41 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:41.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:41 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:41.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:41 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:17:41.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:41 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:41.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:41 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:17:41.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:41 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:17:41.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:41 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:17:41.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:41 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:17:41.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:41 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:41.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:41 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:17:41.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:41 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:41.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:41 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:17:41.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:41 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:41.782 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Feb 09 17:17:41 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-x[45931]: 2024-02-09T17:17:41.527+0000 7f1e83054200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-09T17:17:42.177 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Feb 09 17:17:41 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-x[45931]: 2024-02-09T17:17:41.779+0000 7f1e83054200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-09T17:17:42.678 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Feb 09 17:17:42 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-x[45931]: 2024-02-09T17:17:42.353+0000 7f1e83054200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-09T17:17:42.678 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Feb 09 17:17:42 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-x[45931]: 2024-02-09T17:17:42.565+0000 7f1e83054200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-09T17:17:43.177 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Feb 09 17:17:42 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-x[45931]: 2024-02-09T17:17:42.759+0000 7f1e83054200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-09T17:17:43.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:43 smithi179 ceph-mon[43975]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:43.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:43 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T17:17:43.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:43 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T17:17:43.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:43 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:43.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:43 smithi162 ceph-mon[44304]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:43.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:43 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T17:17:43.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:43 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T17:17:43.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:43 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:43.678 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Feb 09 17:17:43 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-x[45931]: 2024-02-09T17:17:43.415+0000 7f1e83054200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-09T17:17:43.678 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Feb 09 17:17:43 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-x[45931]: 2024-02-09T17:17:43.611+0000 7f1e83054200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-09T17:17:43.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:43 smithi099 ceph-mon[39420]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:43.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:43 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T17:17:43.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:43 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T17:17:43.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:43 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:44.178 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Feb 09 17:17:43 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-x[45931]: 2024-02-09T17:17:43.791+0000 7f1e83054200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-09T17:17:44.178 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Feb 09 17:17:43 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-x[45931]: 2024-02-09T17:17:43.904+0000 7f1e83054200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-09T17:17:44.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:44 smithi179 ceph-mon[43975]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi099:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:17:44.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:44 smithi179 ceph-mon[43975]: Standby manager daemon x started 2024-02-09T17:17:44.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:44 smithi179 ceph-mon[43975]: from='mgr.? 172.21.15.162:0/691849981' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-09T17:17:44.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:44 smithi179 ceph-mon[43975]: from='mgr.? 172.21.15.162:0/691849981' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-09T17:17:44.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:44 smithi179 ceph-mon[43975]: from='mgr.? 172.21.15.162:0/691849981' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-09T17:17:44.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:44 smithi179 ceph-mon[43975]: from='mgr.? 172.21.15.162:0/691849981' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-09T17:17:44.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:44 smithi162 ceph-mon[44304]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi099:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:17:44.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:44 smithi162 ceph-mon[44304]: Standby manager daemon x started 2024-02-09T17:17:44.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:44 smithi162 ceph-mon[44304]: from='mgr.? 172.21.15.162:0/691849981' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-09T17:17:44.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:44 smithi162 ceph-mon[44304]: from='mgr.? 172.21.15.162:0/691849981' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-09T17:17:44.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:44 smithi162 ceph-mon[44304]: from='mgr.? 172.21.15.162:0/691849981' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-09T17:17:44.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:44 smithi162 ceph-mon[44304]: from='mgr.? 172.21.15.162:0/691849981' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-09T17:17:44.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:44 smithi099 ceph-mon[39420]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi099:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:17:44.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:44 smithi099 ceph-mon[39420]: Standby manager daemon x started 2024-02-09T17:17:44.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:44 smithi099 ceph-mon[39420]: from='mgr.? 172.21.15.162:0/691849981' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-09T17:17:44.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:44 smithi099 ceph-mon[39420]: from='mgr.? 172.21.15.162:0/691849981' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-09T17:17:44.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:44 smithi099 ceph-mon[39420]: from='mgr.? 172.21.15.162:0/691849981' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-09T17:17:44.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:44 smithi099 ceph-mon[39420]: from='mgr.? 172.21.15.162:0/691849981' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-09T17:17:45.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:45 smithi179 ceph-mon[43975]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:45.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:45 smithi179 ceph-mon[43975]: mgrmap e14: y(active, since 80s), standbys: x 2024-02-09T17:17:45.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:45 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-09T17:17:45.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:45 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/632535144' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "21739aa7-e278-45da-9a3f-867385f01baf"}]: dispatch 2024-02-09T17:17:45.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:45 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/632535144' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "21739aa7-e278-45da-9a3f-867385f01baf"}]': finished 2024-02-09T17:17:45.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:45 smithi179 ceph-mon[43975]: osdmap e5: 1 total, 0 up, 1 in 2024-02-09T17:17:45.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:45 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-09T17:17:45.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:45 smithi162 ceph-mon[44304]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:45.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:45 smithi162 ceph-mon[44304]: mgrmap e14: y(active, since 80s), standbys: x 2024-02-09T17:17:45.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:45 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-09T17:17:45.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:45 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/632535144' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "21739aa7-e278-45da-9a3f-867385f01baf"}]: dispatch 2024-02-09T17:17:45.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:45 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/632535144' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "21739aa7-e278-45da-9a3f-867385f01baf"}]': finished 2024-02-09T17:17:45.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:45 smithi162 ceph-mon[44304]: osdmap e5: 1 total, 0 up, 1 in 2024-02-09T17:17:45.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:45 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-09T17:17:45.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:45 smithi099 ceph-mon[39420]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:45.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:45 smithi099 ceph-mon[39420]: mgrmap e14: y(active, since 80s), standbys: x 2024-02-09T17:17:45.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:45 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-09T17:17:45.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:45 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/632535144' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "21739aa7-e278-45da-9a3f-867385f01baf"}]: dispatch 2024-02-09T17:17:45.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:45 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/632535144' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "21739aa7-e278-45da-9a3f-867385f01baf"}]': finished 2024-02-09T17:17:45.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:45 smithi099 ceph-mon[39420]: osdmap e5: 1 total, 0 up, 1 in 2024-02-09T17:17:45.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:45 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-09T17:17:46.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:46 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1006752643' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T17:17:46.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:46 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1006752643' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T17:17:46.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:46 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1006752643' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T17:17:47.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:47 smithi179 ceph-mon[43975]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:47.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:47 smithi162 ceph-mon[44304]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:47.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:47 smithi099 ceph-mon[39420]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:49.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:49 smithi179 ceph-mon[43975]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:49.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:49 smithi162 ceph-mon[44304]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:49.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:49 smithi099 ceph-mon[39420]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:51.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:51 smithi179 ceph-mon[43975]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:51.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:51 smithi162 ceph-mon[44304]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:51.680 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:51 smithi099 ceph-mon[39420]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:52.428 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:52 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-09T17:17:52.428 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:52 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:52.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:52 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-09T17:17:52.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:52 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:52.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:52 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-09T17:17:52.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:52 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:17:53.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:53 smithi179 ceph-mon[43975]: Deploying daemon osd.0 on smithi099 2024-02-09T17:17:53.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:53 smithi179 ceph-mon[43975]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:53.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:53 smithi162 ceph-mon[44304]: Deploying daemon osd.0 on smithi099 2024-02-09T17:17:53.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:53 smithi162 ceph-mon[44304]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:53.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:53 smithi099 ceph-mon[39420]: Deploying daemon osd.0 on smithi099 2024-02-09T17:17:53.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:53 smithi099 ceph-mon[39420]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:55.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:55 smithi099 ceph-mon[39420]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:55.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:55 smithi179 ceph-mon[43975]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:55.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:55 smithi162 ceph-mon[44304]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:57.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:56 smithi162 ceph-mon[44304]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:57.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:56 smithi099 ceph-mon[39420]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:57.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:56 smithi179 ceph-mon[43975]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:58.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:58 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:58.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:58 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:17:58.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:58 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:58.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:58 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:58.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:58 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:17:58.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:58 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:58.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:58 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:58.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:58 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:17:58.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:58 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:17:59.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:17:59 smithi179 ceph-mon[43975]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:59.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:17:59 smithi162 ceph-mon[44304]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:17:59.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:17:59 smithi099 ceph-mon[39420]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:18:00.433 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:00 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:00.433 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:00 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:00.433 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:00 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:18:00.434 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:00 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:18:00.434 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:00 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:00.434 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:00 smithi099 ceph-mon[39420]: from='osd.0 [v2:172.21.15.99:6802/3859868923,v1:172.21.15.99:6803/3859868923]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-09T17:18:00.499 INFO:teuthology.orchestra.run.smithi099.stdout:Created osd(s) 0 on host 'smithi099' 2024-02-09T17:18:00.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:00 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:00.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:00 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:00.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:00 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:18:00.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:00 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:18:00.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:00 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:00.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:00 smithi179 ceph-mon[43975]: from='osd.0 [v2:172.21.15.99:6802/3859868923,v1:172.21.15.99:6803/3859868923]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-09T17:18:00.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:00 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:00.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:00 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:00.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:00 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:18:00.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:00 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:18:00.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:00 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:00.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:00 smithi162 ceph-mon[44304]: from='osd.0 [v2:172.21.15.99:6802/3859868923,v1:172.21.15.99:6803/3859868923]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-09T17:18:01.171 DEBUG:teuthology.orchestra.run.smithi099:osd.0> sudo journalctl -f -n 0 -u ceph-a69c870a-c76e-11ee-95b7-87774f69a715@osd.0.service 2024-02-09T17:18:01.174 INFO:tasks.cephadm:Deploying osd.1 on smithi099 with /dev/vg_nvme/lv_3... 2024-02-09T17:18:01.174 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-09T17:18:01.203 INFO:journalctl@ceph.osd.0.smithi099.stdout:-- Logs begin at Fri 2024-02-09 17:04:25 UTC. -- 2024-02-09T17:18:01.427 INFO:journalctl@ceph.osd.0.smithi099.stdout:Feb 09 17:18:01 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-0[58275]: 2024-02-09T17:18:01.308+0000 7f0dce73c700 -1 osd.0 0 waiting for initial osdmap 2024-02-09T17:18:01.427 INFO:journalctl@ceph.osd.0.smithi099.stdout:Feb 09 17:18:01 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-0[58275]: 2024-02-09T17:18:01.311+0000 7f0dca536700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-09T17:18:01.428 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:01 smithi099 ceph-mon[39420]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:18:01.428 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:01 smithi099 ceph-mon[39420]: from='osd.0 [v2:172.21.15.99:6802/3859868923,v1:172.21.15.99:6803/3859868923]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-09T17:18:01.428 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:01 smithi099 ceph-mon[39420]: osdmap e6: 1 total, 0 up, 1 in 2024-02-09T17:18:01.428 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:01 smithi099 ceph-mon[39420]: from='osd.0 [v2:172.21.15.99:6802/3859868923,v1:172.21.15.99:6803/3859868923]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-02-09T17:18:01.428 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:01 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-09T17:18:01.428 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:01 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:01.428 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:01 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:18:01.428 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:01 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:01.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:01 smithi179 ceph-mon[43975]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:18:01.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:01 smithi179 ceph-mon[43975]: from='osd.0 [v2:172.21.15.99:6802/3859868923,v1:172.21.15.99:6803/3859868923]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-09T17:18:01.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:01 smithi179 ceph-mon[43975]: osdmap e6: 1 total, 0 up, 1 in 2024-02-09T17:18:01.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:01 smithi179 ceph-mon[43975]: from='osd.0 [v2:172.21.15.99:6802/3859868923,v1:172.21.15.99:6803/3859868923]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-02-09T17:18:01.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:01 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-09T17:18:01.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:01 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:01.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:01 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:18:01.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:01 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:01.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:01 smithi162 ceph-mon[44304]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:18:01.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:01 smithi162 ceph-mon[44304]: from='osd.0 [v2:172.21.15.99:6802/3859868923,v1:172.21.15.99:6803/3859868923]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-09T17:18:01.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:01 smithi162 ceph-mon[44304]: osdmap e6: 1 total, 0 up, 1 in 2024-02-09T17:18:01.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:01 smithi162 ceph-mon[44304]: from='osd.0 [v2:172.21.15.99:6802/3859868923,v1:172.21.15.99:6803/3859868923]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-02-09T17:18:01.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:01 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-09T17:18:01.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:01 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:01.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:01 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:18:01.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:01 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:02.260 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:18:02.568 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:02 smithi099 ceph-mon[39420]: from='osd.0 [v2:172.21.15.99:6802/3859868923,v1:172.21.15.99:6803/3859868923]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]': finished 2024-02-09T17:18:02.568 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:02 smithi099 ceph-mon[39420]: osdmap e7: 1 total, 0 up, 1 in 2024-02-09T17:18:02.568 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:02 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-09T17:18:02.568 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:02 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-09T17:18:02.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:02 smithi179 ceph-mon[43975]: from='osd.0 [v2:172.21.15.99:6802/3859868923,v1:172.21.15.99:6803/3859868923]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]': finished 2024-02-09T17:18:02.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:02 smithi179 ceph-mon[43975]: osdmap e7: 1 total, 0 up, 1 in 2024-02-09T17:18:02.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:02 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-09T17:18:02.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:02 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-09T17:18:02.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:02 smithi162 ceph-mon[44304]: from='osd.0 [v2:172.21.15.99:6802/3859868923,v1:172.21.15.99:6803/3859868923]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]': finished 2024-02-09T17:18:02.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:02 smithi162 ceph-mon[44304]: osdmap e7: 1 total, 0 up, 1 in 2024-02-09T17:18:02.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:02 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-09T17:18:02.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:02 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-09T17:18:03.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:03 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:18:03.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:03 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:18:03.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:03 smithi179 ceph-mon[43975]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:18:03.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:03 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-09T17:18:03.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:03 smithi179 ceph-mon[43975]: osd.0 [v2:172.21.15.99:6802/3859868923,v1:172.21.15.99:6803/3859868923] boot 2024-02-09T17:18:03.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:03 smithi179 ceph-mon[43975]: osdmap e8: 1 total, 1 up, 1 in 2024-02-09T17:18:03.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:03 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-09T17:18:03.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:03 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:18:03.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:03 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:18:03.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:03 smithi162 ceph-mon[44304]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:18:03.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:03 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-09T17:18:03.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:03 smithi162 ceph-mon[44304]: osd.0 [v2:172.21.15.99:6802/3859868923,v1:172.21.15.99:6803/3859868923] boot 2024-02-09T17:18:03.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:03 smithi162 ceph-mon[44304]: osdmap e8: 1 total, 1 up, 1 in 2024-02-09T17:18:03.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:03 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-09T17:18:03.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:03 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:18:03.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:03 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:18:03.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:03 smithi099 ceph-mon[39420]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T17:18:03.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:03 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-09T17:18:03.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:03 smithi099 ceph-mon[39420]: osd.0 [v2:172.21.15.99:6802/3859868923,v1:172.21.15.99:6803/3859868923] boot 2024-02-09T17:18:03.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:03 smithi099 ceph-mon[39420]: osdmap e8: 1 total, 1 up, 1 in 2024-02-09T17:18:03.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:03 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-09T17:18:04.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:04 smithi099 ceph-mon[39420]: osdmap e9: 1 total, 1 up, 1 in 2024-02-09T17:18:04.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:04 smithi179 ceph-mon[43975]: osdmap e9: 1 total, 1 up, 1 in 2024-02-09T17:18:04.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:04 smithi162 ceph-mon[44304]: osdmap e9: 1 total, 1 up, 1 in 2024-02-09T17:18:05.294 INFO:teuthology.orchestra.run.smithi099.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-09T17:18:05.294 INFO:teuthology.orchestra.run.smithi099.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-09T17:18:05.294 INFO:teuthology.orchestra.run.smithi099.stderr: stderr: 10+0 records in 2024-02-09T17:18:05.294 INFO:teuthology.orchestra.run.smithi099.stderr:10+0 records out 2024-02-09T17:18:05.294 INFO:teuthology.orchestra.run.smithi099.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112413 s, 933 MB/s 2024-02-09T17:18:05.295 INFO:teuthology.orchestra.run.smithi099.stderr:--> Zapping successful for: 2024-02-09T17:18:05.636 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:05 smithi099 ceph-mon[39420]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T17:18:05.636 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:05 smithi099 ceph-mon[39420]: Detected new or changed devices on smithi099 2024-02-09T17:18:05.637 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:05 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:05.637 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:05 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:05.637 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:05 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:18:05.637 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:05 smithi099 ceph-mon[39420]: Adjusting osd_memory_target on smithi099 to 17122M 2024-02-09T17:18:05.637 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:05 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:05.637 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:05 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:18:05.637 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:05 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:18:05.637 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:05 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:05.919 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph orch daemon add osd smithi099:vg_nvme/lv_3 2024-02-09T17:18:05.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:05 smithi179 ceph-mon[43975]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T17:18:05.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:05 smithi179 ceph-mon[43975]: Detected new or changed devices on smithi099 2024-02-09T17:18:05.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:05 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:05.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:05 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:05.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:05 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:18:05.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:05 smithi179 ceph-mon[43975]: Adjusting osd_memory_target on smithi099 to 17122M 2024-02-09T17:18:05.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:05 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:05.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:05 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:18:05.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:05 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:18:05.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:05 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:05.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:05 smithi162 ceph-mon[44304]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T17:18:05.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:05 smithi162 ceph-mon[44304]: Detected new or changed devices on smithi099 2024-02-09T17:18:05.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:05 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:05.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:05 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:05.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:05 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:18:05.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:05 smithi162 ceph-mon[44304]: Adjusting osd_memory_target on smithi099 to 17122M 2024-02-09T17:18:05.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:05 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:05.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:05 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:18:05.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:05 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:18:05.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:05 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:06.180 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:18:07.597 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:07 smithi099 ceph-mon[39420]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T17:18:07.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:07 smithi179 ceph-mon[43975]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T17:18:07.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:07 smithi162 ceph-mon[44304]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T17:18:08.770 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:08 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T17:18:08.770 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:08 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T17:18:08.771 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:08 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:18:08.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:08 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T17:18:08.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:08 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T17:18:08.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:08 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:18:08.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:08 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T17:18:08.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:08 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T17:18:08.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:08 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:18:09.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:09 smithi179 ceph-mon[43975]: from='client.14247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi099:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:18:09.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:09 smithi179 ceph-mon[43975]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T17:18:09.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:09 smithi162 ceph-mon[44304]: from='client.14247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi099:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:18:09.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:09 smithi162 ceph-mon[44304]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T17:18:09.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:09 smithi099 ceph-mon[39420]: from='client.14247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi099:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:18:09.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:09 smithi099 ceph-mon[39420]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T17:18:10.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:10 smithi179 ceph-mon[43975]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T17:18:10.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:10 smithi162 ceph-mon[44304]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T17:18:10.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:10 smithi099 ceph-mon[39420]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T17:18:11.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:11 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3270380518' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8eab347c-9065-4de1-93ae-e2eb7e11cab7"}]: dispatch 2024-02-09T17:18:11.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:11 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3270380518' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8eab347c-9065-4de1-93ae-e2eb7e11cab7"}]': finished 2024-02-09T17:18:11.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:11 smithi179 ceph-mon[43975]: osdmap e10: 2 total, 1 up, 2 in 2024-02-09T17:18:11.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:11 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-09T17:18:11.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:11 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1672156256' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T17:18:11.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:11 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3270380518' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8eab347c-9065-4de1-93ae-e2eb7e11cab7"}]: dispatch 2024-02-09T17:18:11.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:11 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3270380518' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8eab347c-9065-4de1-93ae-e2eb7e11cab7"}]': finished 2024-02-09T17:18:11.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:11 smithi162 ceph-mon[44304]: osdmap e10: 2 total, 1 up, 2 in 2024-02-09T17:18:11.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:11 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-09T17:18:11.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:11 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1672156256' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T17:18:11.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:11 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3270380518' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8eab347c-9065-4de1-93ae-e2eb7e11cab7"}]: dispatch 2024-02-09T17:18:11.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:11 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3270380518' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8eab347c-9065-4de1-93ae-e2eb7e11cab7"}]': finished 2024-02-09T17:18:11.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:11 smithi099 ceph-mon[39420]: osdmap e10: 2 total, 1 up, 2 in 2024-02-09T17:18:11.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:11 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-09T17:18:11.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:11 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1672156256' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T17:18:12.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:12 smithi179 ceph-mon[43975]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T17:18:12.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:12 smithi162 ceph-mon[44304]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T17:18:12.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:12 smithi099 ceph-mon[39420]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T17:18:15.452 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:15 smithi099 ceph-mon[39420]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T17:18:15.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:15 smithi179 ceph-mon[43975]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T17:18:15.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:15 smithi162 ceph-mon[44304]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T17:18:17.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:16 smithi099 ceph-mon[39420]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T17:18:17.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:16 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-09T17:18:17.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:16 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:18:17.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:16 smithi179 ceph-mon[43975]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T17:18:17.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:16 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-09T17:18:17.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:16 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:18:17.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:16 smithi162 ceph-mon[44304]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T17:18:17.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:16 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-09T17:18:17.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:16 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:18:18.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:17 smithi099 ceph-mon[39420]: Deploying daemon osd.1 on smithi099 2024-02-09T17:18:18.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:17 smithi179 ceph-mon[43975]: Deploying daemon osd.1 on smithi099 2024-02-09T17:18:18.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:17 smithi162 ceph-mon[44304]: Deploying daemon osd.1 on smithi099 2024-02-09T17:18:18.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:18 smithi099 ceph-mon[39420]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T17:18:19.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:18 smithi179 ceph-mon[43975]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T17:18:19.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:18 smithi162 ceph-mon[44304]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T17:18:21.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:21 smithi099 ceph-mon[39420]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T17:18:21.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:21 smithi179 ceph-mon[43975]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T17:18:21.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:21 smithi162 ceph-mon[44304]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T17:18:23.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:23 smithi099 ceph-mon[39420]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T17:18:23.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:23 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:23.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:23 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:23.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:23 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:18:23.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:23 smithi179 ceph-mon[43975]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T17:18:23.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:23 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:23.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:23 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:23.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:23 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:18:23.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:23 smithi162 ceph-mon[44304]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T17:18:23.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:23 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:23.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:23 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:23.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:23 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:18:25.722 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:25 smithi099 ceph-mon[39420]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T17:18:25.723 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:25 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:25.723 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:25 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:25.723 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:25 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:18:25.723 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:25 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:18:25.723 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:25 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:25.805 INFO:teuthology.orchestra.run.smithi099.stdout:Created osd(s) 1 on host 'smithi099' 2024-02-09T17:18:25.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:25 smithi179 ceph-mon[43975]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T17:18:25.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:25 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:25.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:25 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:25.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:25 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:18:25.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:25 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:18:25.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:25 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:25.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:25 smithi162 ceph-mon[44304]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T17:18:25.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:25 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:25.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:25 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:25.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:25 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:18:25.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:25 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:18:25.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:25 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:26.543 DEBUG:teuthology.orchestra.run.smithi099:osd.1> sudo journalctl -f -n 0 -u ceph-a69c870a-c76e-11ee-95b7-87774f69a715@osd.1.service 2024-02-09T17:18:26.545 INFO:tasks.cephadm:Deploying osd.2 on smithi099 with /dev/vg_nvme/lv_2... 2024-02-09T17:18:26.545 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-09T17:18:26.574 INFO:journalctl@ceph.osd.1.smithi099.stdout:-- Logs begin at Fri 2024-02-09 17:04:25 UTC. -- 2024-02-09T17:18:26.731 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:26 smithi099 ceph-mon[39420]: from='osd.1 [v2:172.21.15.99:6810/3868328452,v1:172.21.15.99:6811/3868328452]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-09T17:18:26.731 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:26 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:26.731 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:26 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:26.731 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:26 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:18:26.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:26 smithi179 ceph-mon[43975]: from='osd.1 [v2:172.21.15.99:6810/3868328452,v1:172.21.15.99:6811/3868328452]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-09T17:18:26.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:26 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:26.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:26 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:26.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:26 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:18:26.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:26 smithi162 ceph-mon[44304]: from='osd.1 [v2:172.21.15.99:6810/3868328452,v1:172.21.15.99:6811/3868328452]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-09T17:18:26.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:26 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:26.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:26 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:26.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:26 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:18:27.265 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:18:27.639 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:27 smithi099 ceph-mon[39420]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T17:18:27.639 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:27 smithi099 ceph-mon[39420]: from='osd.1 [v2:172.21.15.99:6810/3868328452,v1:172.21.15.99:6811/3868328452]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-09T17:18:27.639 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:27 smithi099 ceph-mon[39420]: osdmap e11: 2 total, 1 up, 2 in 2024-02-09T17:18:27.639 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:27 smithi099 ceph-mon[39420]: from='osd.1 [v2:172.21.15.99:6810/3868328452,v1:172.21.15.99:6811/3868328452]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-02-09T17:18:27.639 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:27 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-09T17:18:27.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:27 smithi179 ceph-mon[43975]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T17:18:27.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:27 smithi179 ceph-mon[43975]: from='osd.1 [v2:172.21.15.99:6810/3868328452,v1:172.21.15.99:6811/3868328452]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-09T17:18:27.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:27 smithi179 ceph-mon[43975]: osdmap e11: 2 total, 1 up, 2 in 2024-02-09T17:18:27.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:27 smithi179 ceph-mon[43975]: from='osd.1 [v2:172.21.15.99:6810/3868328452,v1:172.21.15.99:6811/3868328452]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-02-09T17:18:27.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:27 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-09T17:18:27.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:27 smithi162 ceph-mon[44304]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T17:18:27.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:27 smithi162 ceph-mon[44304]: from='osd.1 [v2:172.21.15.99:6810/3868328452,v1:172.21.15.99:6811/3868328452]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-09T17:18:27.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:27 smithi162 ceph-mon[44304]: osdmap e11: 2 total, 1 up, 2 in 2024-02-09T17:18:27.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:27 smithi162 ceph-mon[44304]: from='osd.1 [v2:172.21.15.99:6810/3868328452,v1:172.21.15.99:6811/3868328452]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-02-09T17:18:27.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:27 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-09T17:18:28.245 INFO:journalctl@ceph.osd.1.smithi099.stdout:Feb 09 17:18:27 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-1[65435]: 2024-02-09T17:18:27.928+0000 7fdf6410f700 -1 osd.1 0 waiting for initial osdmap 2024-02-09T17:18:28.245 INFO:journalctl@ceph.osd.1.smithi099.stdout:Feb 09 17:18:27 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-1[65435]: 2024-02-09T17:18:27.931+0000 7fdf5b700700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-09T17:18:28.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:28 smithi099 ceph-mon[39420]: from='osd.1 [v2:172.21.15.99:6810/3868328452,v1:172.21.15.99:6811/3868328452]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]': finished 2024-02-09T17:18:28.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:28 smithi099 ceph-mon[39420]: osdmap e12: 2 total, 1 up, 2 in 2024-02-09T17:18:28.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:28 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-09T17:18:28.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:28 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-09T17:18:28.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:28 smithi099 ceph-mon[39420]: from='osd.1 [v2:172.21.15.99:6810/3868328452,v1:172.21.15.99:6811/3868328452]' entity='osd.1' 2024-02-09T17:18:28.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:28 smithi099 ceph-mon[39420]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T17:18:28.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:28 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-09T17:18:28.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:28 smithi179 ceph-mon[43975]: from='osd.1 [v2:172.21.15.99:6810/3868328452,v1:172.21.15.99:6811/3868328452]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]': finished 2024-02-09T17:18:28.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:28 smithi179 ceph-mon[43975]: osdmap e12: 2 total, 1 up, 2 in 2024-02-09T17:18:28.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:28 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-09T17:18:28.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:28 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-09T17:18:28.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:28 smithi179 ceph-mon[43975]: from='osd.1 [v2:172.21.15.99:6810/3868328452,v1:172.21.15.99:6811/3868328452]' entity='osd.1' 2024-02-09T17:18:28.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:28 smithi179 ceph-mon[43975]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T17:18:28.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:28 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-09T17:18:28.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:28 smithi162 ceph-mon[44304]: from='osd.1 [v2:172.21.15.99:6810/3868328452,v1:172.21.15.99:6811/3868328452]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]': finished 2024-02-09T17:18:28.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:28 smithi162 ceph-mon[44304]: osdmap e12: 2 total, 1 up, 2 in 2024-02-09T17:18:28.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:28 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-09T17:18:28.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:28 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-09T17:18:28.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:28 smithi162 ceph-mon[44304]: from='osd.1 [v2:172.21.15.99:6810/3868328452,v1:172.21.15.99:6811/3868328452]' entity='osd.1' 2024-02-09T17:18:28.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:28 smithi162 ceph-mon[44304]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T17:18:28.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:28 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-09T17:18:29.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:29 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:18:29.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:29 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:18:29.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:29 smithi179 ceph-mon[43975]: osd.1 [v2:172.21.15.99:6810/3868328452,v1:172.21.15.99:6811/3868328452] boot 2024-02-09T17:18:29.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:29 smithi179 ceph-mon[43975]: osdmap e13: 2 total, 2 up, 2 in 2024-02-09T17:18:29.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:29 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-09T17:18:29.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:29 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:18:29.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:29 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:18:29.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:29 smithi162 ceph-mon[44304]: osd.1 [v2:172.21.15.99:6810/3868328452,v1:172.21.15.99:6811/3868328452] boot 2024-02-09T17:18:29.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:29 smithi162 ceph-mon[44304]: osdmap e13: 2 total, 2 up, 2 in 2024-02-09T17:18:29.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:29 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-09T17:18:29.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:29 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:18:29.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:29 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:18:29.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:29 smithi099 ceph-mon[39420]: osd.1 [v2:172.21.15.99:6810/3868328452,v1:172.21.15.99:6811/3868328452] boot 2024-02-09T17:18:29.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:29 smithi099 ceph-mon[39420]: osdmap e13: 2 total, 2 up, 2 in 2024-02-09T17:18:29.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:29 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-09T17:18:30.768 INFO:teuthology.orchestra.run.smithi099.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-09T17:18:30.768 INFO:teuthology.orchestra.run.smithi099.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-09T17:18:30.768 INFO:teuthology.orchestra.run.smithi099.stderr: stderr: 10+0 records in 2024-02-09T17:18:30.768 INFO:teuthology.orchestra.run.smithi099.stderr:10+0 records out 2024-02-09T17:18:30.768 INFO:teuthology.orchestra.run.smithi099.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0115764 s, 906 MB/s 2024-02-09T17:18:30.768 INFO:teuthology.orchestra.run.smithi099.stderr:--> Zapping successful for: 2024-02-09T17:18:30.807 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:30 smithi099 ceph-mon[39420]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T17:18:30.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:30 smithi179 ceph-mon[43975]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T17:18:30.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:30 smithi162 ceph-mon[44304]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T17:18:31.374 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph orch daemon add osd smithi099:vg_nvme/lv_2 2024-02-09T17:18:31.637 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:18:31.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:31 smithi099 ceph-mon[39420]: osdmap e14: 2 total, 2 up, 2 in 2024-02-09T17:18:31.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:31 smithi099 ceph-mon[39420]: Detected new or changed devices on smithi099 2024-02-09T17:18:31.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:31 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:31.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:31 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:31.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:31 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:18:31.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:31 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:18:31.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:31 smithi099 ceph-mon[39420]: Adjusting osd_memory_target on smithi099 to 8561M 2024-02-09T17:18:31.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:31 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:31.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:31 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:18:31.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:31 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:18:31.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:31 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:31.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:31 smithi179 ceph-mon[43975]: osdmap e14: 2 total, 2 up, 2 in 2024-02-09T17:18:31.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:31 smithi179 ceph-mon[43975]: Detected new or changed devices on smithi099 2024-02-09T17:18:31.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:31 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:31.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:31 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:31.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:31 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:18:31.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:31 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:18:31.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:31 smithi179 ceph-mon[43975]: Adjusting osd_memory_target on smithi099 to 8561M 2024-02-09T17:18:31.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:31 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:31.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:31 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:18:31.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:31 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:18:31.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:31 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:31.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:31 smithi162 ceph-mon[44304]: osdmap e14: 2 total, 2 up, 2 in 2024-02-09T17:18:31.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:31 smithi162 ceph-mon[44304]: Detected new or changed devices on smithi099 2024-02-09T17:18:31.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:31 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:31.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:31 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:31.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:31 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:18:31.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:31 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:18:31.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:31 smithi162 ceph-mon[44304]: Adjusting osd_memory_target on smithi099 to 8561M 2024-02-09T17:18:31.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:31 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:31.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:31 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:18:31.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:31 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:18:31.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:31 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:32.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:32 smithi099 ceph-mon[39420]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T17:18:32.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:32 smithi179 ceph-mon[43975]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T17:18:32.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:32 smithi162 ceph-mon[44304]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T17:18:33.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:33 smithi179 ceph-mon[43975]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi099:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:18:33.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:33 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T17:18:33.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:33 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T17:18:33.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:33 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:18:33.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:33 smithi162 ceph-mon[44304]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi099:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:18:33.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:33 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T17:18:33.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:33 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T17:18:33.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:33 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:18:33.929 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:33 smithi099 ceph-mon[39420]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi099:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:18:33.929 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:33 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T17:18:33.929 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:33 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T17:18:33.929 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:33 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:18:34.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:34 smithi179 ceph-mon[43975]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T17:18:34.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:34 smithi162 ceph-mon[44304]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T17:18:34.958 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:34 smithi099 ceph-mon[39420]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T17:18:37.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:36 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/223322596' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "575f4257-7e34-485e-963c-2607b69e1220"}]: dispatch 2024-02-09T17:18:37.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:36 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/223322596' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "575f4257-7e34-485e-963c-2607b69e1220"}]': finished 2024-02-09T17:18:37.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:36 smithi099 ceph-mon[39420]: osdmap e15: 3 total, 2 up, 3 in 2024-02-09T17:18:37.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:36 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-09T17:18:37.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:36 smithi099 ceph-mon[39420]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T17:18:37.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:36 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/45277088' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T17:18:37.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:36 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/223322596' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "575f4257-7e34-485e-963c-2607b69e1220"}]: dispatch 2024-02-09T17:18:37.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:36 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/223322596' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "575f4257-7e34-485e-963c-2607b69e1220"}]': finished 2024-02-09T17:18:37.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:36 smithi179 ceph-mon[43975]: osdmap e15: 3 total, 2 up, 3 in 2024-02-09T17:18:37.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:36 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-09T17:18:37.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:36 smithi179 ceph-mon[43975]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T17:18:37.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:36 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/45277088' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T17:18:37.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:36 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/223322596' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "575f4257-7e34-485e-963c-2607b69e1220"}]: dispatch 2024-02-09T17:18:37.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:36 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/223322596' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "575f4257-7e34-485e-963c-2607b69e1220"}]': finished 2024-02-09T17:18:37.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:36 smithi162 ceph-mon[44304]: osdmap e15: 3 total, 2 up, 3 in 2024-02-09T17:18:37.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:36 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-09T17:18:37.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:36 smithi162 ceph-mon[44304]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T17:18:37.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:36 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/45277088' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T17:18:39.593 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:39 smithi099 ceph-mon[39420]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T17:18:39.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:39 smithi179 ceph-mon[43975]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T17:18:39.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:39 smithi162 ceph-mon[44304]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T17:18:41.443 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:41 smithi099 ceph-mon[39420]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T17:18:41.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:41 smithi179 ceph-mon[43975]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T17:18:41.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:41 smithi162 ceph-mon[44304]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T17:18:43.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:43 smithi099 ceph-mon[39420]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T17:18:43.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:43 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-09T17:18:43.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:43 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:18:43.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:43 smithi179 ceph-mon[43975]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T17:18:43.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:43 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-09T17:18:43.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:43 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:18:43.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:43 smithi162 ceph-mon[44304]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T17:18:43.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:43 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-09T17:18:43.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:43 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:18:44.454 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:44 smithi099 ceph-mon[39420]: Deploying daemon osd.2 on smithi099 2024-02-09T17:18:44.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:44 smithi179 ceph-mon[43975]: Deploying daemon osd.2 on smithi099 2024-02-09T17:18:44.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:44 smithi162 ceph-mon[44304]: Deploying daemon osd.2 on smithi099 2024-02-09T17:18:45.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:45 smithi099 ceph-mon[39420]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T17:18:45.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:45 smithi179 ceph-mon[43975]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T17:18:45.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:45 smithi162 ceph-mon[44304]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T17:18:47.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:47 smithi099 ceph-mon[39420]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T17:18:47.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:47 smithi179 ceph-mon[43975]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T17:18:47.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:47 smithi162 ceph-mon[44304]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T17:18:49.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:49 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:49.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:49 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:49.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:49 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:18:49.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:49 smithi099 ceph-mon[39420]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T17:18:49.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:49 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:49.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:49 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:49.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:49 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:18:49.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:49 smithi179 ceph-mon[43975]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T17:18:49.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:49 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:49.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:49 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:49.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:49 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:18:49.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:49 smithi162 ceph-mon[44304]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T17:18:51.603 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:51 smithi099 ceph-mon[39420]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T17:18:51.603 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:51 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:51.603 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:51 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:51.603 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:51 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:18:51.603 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:51 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:18:51.603 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:51 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:51.603 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:51 smithi099 ceph-mon[39420]: from='osd.2 [v2:172.21.15.99:6818/537506237,v1:172.21.15.99:6819/537506237]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-09T17:18:51.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:51 smithi179 ceph-mon[43975]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T17:18:51.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:51 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:51.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:51 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:51.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:51 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:18:51.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:51 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:18:51.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:51 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:51.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:51 smithi179 ceph-mon[43975]: from='osd.2 [v2:172.21.15.99:6818/537506237,v1:172.21.15.99:6819/537506237]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-09T17:18:51.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:51 smithi162 ceph-mon[44304]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T17:18:51.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:51 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:51.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:51 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:51.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:51 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:18:51.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:51 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:18:51.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:51 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:51.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:51 smithi162 ceph-mon[44304]: from='osd.2 [v2:172.21.15.99:6818/537506237,v1:172.21.15.99:6819/537506237]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-09T17:18:51.687 INFO:teuthology.orchestra.run.smithi099.stdout:Created osd(s) 2 on host 'smithi099' 2024-02-09T17:18:52.332 DEBUG:teuthology.orchestra.run.smithi099:osd.2> sudo journalctl -f -n 0 -u ceph-a69c870a-c76e-11ee-95b7-87774f69a715@osd.2.service 2024-02-09T17:18:52.335 INFO:tasks.cephadm:Deploying osd.3 on smithi099 with /dev/vg_nvme/lv_1... 2024-02-09T17:18:52.335 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-09T17:18:52.368 INFO:journalctl@ceph.osd.2.smithi099.stdout:-- Logs begin at Fri 2024-02-09 17:04:25 UTC. -- 2024-02-09T17:18:52.837 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:52 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:52.837 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:52 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:52.837 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:52 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:18:52.837 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:52 smithi099 ceph-mon[39420]: from='osd.2 [v2:172.21.15.99:6818/537506237,v1:172.21.15.99:6819/537506237]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-09T17:18:52.837 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:52 smithi099 ceph-mon[39420]: osdmap e16: 3 total, 2 up, 3 in 2024-02-09T17:18:52.837 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:52 smithi099 ceph-mon[39420]: from='osd.2 [v2:172.21.15.99:6818/537506237,v1:172.21.15.99:6819/537506237]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-02-09T17:18:52.837 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:52 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-09T17:18:52.838 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:52 smithi099 ceph-mon[39420]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T17:18:52.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:52 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:52.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:52 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:52.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:52 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:18:52.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:52 smithi179 ceph-mon[43975]: from='osd.2 [v2:172.21.15.99:6818/537506237,v1:172.21.15.99:6819/537506237]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-09T17:18:52.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:52 smithi179 ceph-mon[43975]: osdmap e16: 3 total, 2 up, 3 in 2024-02-09T17:18:52.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:52 smithi179 ceph-mon[43975]: from='osd.2 [v2:172.21.15.99:6818/537506237,v1:172.21.15.99:6819/537506237]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-02-09T17:18:52.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:52 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-09T17:18:52.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:52 smithi179 ceph-mon[43975]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T17:18:52.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:52 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:52.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:52 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:52.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:52 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:18:52.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:52 smithi162 ceph-mon[44304]: from='osd.2 [v2:172.21.15.99:6818/537506237,v1:172.21.15.99:6819/537506237]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-09T17:18:52.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:52 smithi162 ceph-mon[44304]: osdmap e16: 3 total, 2 up, 3 in 2024-02-09T17:18:52.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:52 smithi162 ceph-mon[44304]: from='osd.2 [v2:172.21.15.99:6818/537506237,v1:172.21.15.99:6819/537506237]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-02-09T17:18:52.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:52 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-09T17:18:52.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:52 smithi162 ceph-mon[44304]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T17:18:53.494 INFO:journalctl@ceph.osd.2.smithi099.stdout:Feb 09 17:18:53 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-2[73177]: 2024-02-09T17:18:53.430+0000 7f382fa5c700 -1 osd.2 0 waiting for initial osdmap 2024-02-09T17:18:53.495 INFO:journalctl@ceph.osd.2.smithi099.stdout:Feb 09 17:18:53 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-2[73177]: 2024-02-09T17:18:53.434+0000 7f382784e700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-09T17:18:53.530 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:18:54.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:53 smithi099 ceph-mon[39420]: from='osd.2 [v2:172.21.15.99:6818/537506237,v1:172.21.15.99:6819/537506237]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]': finished 2024-02-09T17:18:54.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:53 smithi099 ceph-mon[39420]: osdmap e17: 3 total, 2 up, 3 in 2024-02-09T17:18:54.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:53 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-09T17:18:54.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:53 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-09T17:18:54.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:53 smithi099 ceph-mon[39420]: from='osd.2 [v2:172.21.15.99:6818/537506237,v1:172.21.15.99:6819/537506237]' entity='osd.2' 2024-02-09T17:18:54.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:53 smithi179 ceph-mon[43975]: from='osd.2 [v2:172.21.15.99:6818/537506237,v1:172.21.15.99:6819/537506237]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]': finished 2024-02-09T17:18:54.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:53 smithi179 ceph-mon[43975]: osdmap e17: 3 total, 2 up, 3 in 2024-02-09T17:18:54.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:53 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-09T17:18:54.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:53 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-09T17:18:54.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:53 smithi179 ceph-mon[43975]: from='osd.2 [v2:172.21.15.99:6818/537506237,v1:172.21.15.99:6819/537506237]' entity='osd.2' 2024-02-09T17:18:54.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:53 smithi162 ceph-mon[44304]: from='osd.2 [v2:172.21.15.99:6818/537506237,v1:172.21.15.99:6819/537506237]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]': finished 2024-02-09T17:18:54.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:53 smithi162 ceph-mon[44304]: osdmap e17: 3 total, 2 up, 3 in 2024-02-09T17:18:54.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:53 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-09T17:18:54.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:53 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-09T17:18:54.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:53 smithi162 ceph-mon[44304]: from='osd.2 [v2:172.21.15.99:6818/537506237,v1:172.21.15.99:6819/537506237]' entity='osd.2' 2024-02-09T17:18:55.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:54 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:18:55.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:54 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:18:55.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:54 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-09T17:18:55.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:54 smithi099 ceph-mon[39420]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T17:18:55.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:54 smithi099 ceph-mon[39420]: osd.2 [v2:172.21.15.99:6818/537506237,v1:172.21.15.99:6819/537506237] boot 2024-02-09T17:18:55.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:54 smithi099 ceph-mon[39420]: osdmap e18: 3 total, 3 up, 3 in 2024-02-09T17:18:55.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:54 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-09T17:18:55.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:54 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:18:55.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:54 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:18:55.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:54 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-09T17:18:55.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:54 smithi179 ceph-mon[43975]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T17:18:55.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:54 smithi179 ceph-mon[43975]: osd.2 [v2:172.21.15.99:6818/537506237,v1:172.21.15.99:6819/537506237] boot 2024-02-09T17:18:55.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:54 smithi179 ceph-mon[43975]: osdmap e18: 3 total, 3 up, 3 in 2024-02-09T17:18:55.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:54 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-09T17:18:55.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:54 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:18:55.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:54 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:18:55.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:54 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-09T17:18:55.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:54 smithi162 ceph-mon[44304]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T17:18:55.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:54 smithi162 ceph-mon[44304]: osd.2 [v2:172.21.15.99:6818/537506237,v1:172.21.15.99:6819/537506237] boot 2024-02-09T17:18:55.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:54 smithi162 ceph-mon[44304]: osdmap e18: 3 total, 3 up, 3 in 2024-02-09T17:18:55.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:54 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-09T17:18:56.170 INFO:teuthology.orchestra.run.smithi099.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-09T17:18:56.170 INFO:teuthology.orchestra.run.smithi099.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-09T17:18:56.170 INFO:teuthology.orchestra.run.smithi099.stderr: stderr: 10+0 records in 2024-02-09T17:18:56.170 INFO:teuthology.orchestra.run.smithi099.stderr:10+0 records out 2024-02-09T17:18:56.170 INFO:teuthology.orchestra.run.smithi099.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0144293 s, 727 MB/s 2024-02-09T17:18:56.170 INFO:teuthology.orchestra.run.smithi099.stderr:--> Zapping successful for: 2024-02-09T17:18:56.896 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph orch daemon add osd smithi099:vg_nvme/lv_1 2024-02-09T17:18:57.158 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:18:57.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:56 smithi099 ceph-mon[39420]: osdmap e19: 3 total, 3 up, 3 in 2024-02-09T17:18:57.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:56 smithi099 ceph-mon[39420]: pgmap v85: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-09T17:18:57.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:56 smithi099 ceph-mon[39420]: Detected new or changed devices on smithi099 2024-02-09T17:18:57.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:56 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:57.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:56 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:57.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:56 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:18:57.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:56 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:18:57.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:56 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:18:57.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:56 smithi099 ceph-mon[39420]: Adjusting osd_memory_target on smithi099 to 5707M 2024-02-09T17:18:57.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:56 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:57.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:56 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:18:57.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:56 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:18:57.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:56 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:57.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:56 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-02-09T17:18:57.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:56 smithi179 ceph-mon[43975]: osdmap e19: 3 total, 3 up, 3 in 2024-02-09T17:18:57.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:56 smithi179 ceph-mon[43975]: pgmap v85: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-09T17:18:57.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:56 smithi179 ceph-mon[43975]: Detected new or changed devices on smithi099 2024-02-09T17:18:57.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:56 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:57.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:56 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:57.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:56 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:18:57.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:56 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:18:57.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:56 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:18:57.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:56 smithi179 ceph-mon[43975]: Adjusting osd_memory_target on smithi099 to 5707M 2024-02-09T17:18:57.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:56 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:57.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:56 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:18:57.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:56 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:18:57.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:56 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:57.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:56 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-02-09T17:18:57.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:56 smithi162 ceph-mon[44304]: osdmap e19: 3 total, 3 up, 3 in 2024-02-09T17:18:57.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:56 smithi162 ceph-mon[44304]: pgmap v85: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-09T17:18:57.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:56 smithi162 ceph-mon[44304]: Detected new or changed devices on smithi099 2024-02-09T17:18:57.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:56 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:57.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:56 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:57.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:56 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:18:57.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:56 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:18:57.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:56 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:18:57.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:56 smithi162 ceph-mon[44304]: Adjusting osd_memory_target on smithi099 to 5707M 2024-02-09T17:18:57.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:56 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:57.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:56 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:18:57.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:56 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:18:57.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:56 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:18:57.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:56 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-02-09T17:18:58.609 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:58 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-02-09T17:18:58.610 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:58 smithi099 ceph-mon[39420]: osdmap e20: 3 total, 3 up, 3 in 2024-02-09T17:18:58.610 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:58 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-09T17:18:58.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:58 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-02-09T17:18:58.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:58 smithi179 ceph-mon[43975]: osdmap e20: 3 total, 3 up, 3 in 2024-02-09T17:18:58.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:58 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-09T17:18:58.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:58 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-02-09T17:18:58.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:58 smithi162 ceph-mon[44304]: osdmap e20: 3 total, 3 up, 3 in 2024-02-09T17:18:58.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:58 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-09T17:18:59.394 INFO:journalctl@ceph.osd.0.smithi099.stdout:Feb 09 17:18:59 smithi099 sudo[78306]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-09T17:18:59.394 INFO:journalctl@ceph.osd.0.smithi099.stdout:Feb 09 17:18:59 smithi099 sudo[78306]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-09T17:18:59.394 INFO:journalctl@ceph.osd.0.smithi099.stdout:Feb 09 17:18:59 smithi099 sudo[78306]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-09T17:18:59.394 INFO:journalctl@ceph.osd.0.smithi099.stdout:Feb 09 17:18:59 smithi099 sudo[78306]: pam_unix(sudo:session): session closed for user root 2024-02-09T17:18:59.395 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:59 smithi099 ceph-mon[39420]: pgmap v87: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-09T17:18:59.395 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:59 smithi099 ceph-mon[39420]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-09T17:18:59.395 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:59 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-09T17:18:59.395 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:59 smithi099 ceph-mon[39420]: osdmap e21: 3 total, 3 up, 3 in 2024-02-09T17:18:59.395 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:59 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T17:18:59.395 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:59 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T17:18:59.395 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:18:59 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:18:59.661 INFO:journalctl@ceph.osd.0.smithi099.stdout:Feb 09 17:18:59 smithi099 sudo[78391]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-09T17:18:59.661 INFO:journalctl@ceph.osd.0.smithi099.stdout:Feb 09 17:18:59 smithi099 sudo[78391]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-09T17:18:59.661 INFO:journalctl@ceph.osd.0.smithi099.stdout:Feb 09 17:18:59 smithi099 sudo[78391]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-09T17:18:59.661 INFO:journalctl@ceph.osd.0.smithi099.stdout:Feb 09 17:18:59 smithi099 sudo[78391]: pam_unix(sudo:session): session closed for user root 2024-02-09T17:18:59.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:59 smithi179 ceph-mon[43975]: pgmap v87: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-09T17:18:59.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:59 smithi179 ceph-mon[43975]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-09T17:18:59.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:59 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-09T17:18:59.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:59 smithi179 ceph-mon[43975]: osdmap e21: 3 total, 3 up, 3 in 2024-02-09T17:18:59.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:59 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T17:18:59.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:59 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T17:18:59.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:18:59 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:18:59.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:59 smithi162 ceph-mon[44304]: pgmap v87: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-09T17:18:59.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:59 smithi162 ceph-mon[44304]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-09T17:18:59.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:59 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-09T17:18:59.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:59 smithi162 ceph-mon[44304]: osdmap e21: 3 total, 3 up, 3 in 2024-02-09T17:18:59.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:59 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T17:18:59.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:59 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T17:18:59.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:18:59 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:18:59.919 INFO:journalctl@ceph.osd.1.smithi099.stdout:Feb 09 17:18:59 smithi099 sudo[78419]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-09T17:18:59.919 INFO:journalctl@ceph.osd.1.smithi099.stdout:Feb 09 17:18:59 smithi099 sudo[78419]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-09T17:18:59.919 INFO:journalctl@ceph.osd.1.smithi099.stdout:Feb 09 17:18:59 smithi099 sudo[78419]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-09T17:18:59.919 INFO:journalctl@ceph.osd.1.smithi099.stdout:Feb 09 17:18:59 smithi099 sudo[78419]: pam_unix(sudo:session): session closed for user root 2024-02-09T17:19:00.174 INFO:journalctl@ceph.osd.1.smithi099.stdout:Feb 09 17:18:59 smithi099 sudo[78441]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-09T17:19:00.174 INFO:journalctl@ceph.osd.1.smithi099.stdout:Feb 09 17:18:59 smithi099 sudo[78441]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-09T17:19:00.174 INFO:journalctl@ceph.osd.1.smithi099.stdout:Feb 09 17:18:59 smithi099 sudo[78441]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-09T17:19:00.174 INFO:journalctl@ceph.osd.1.smithi099.stdout:Feb 09 17:18:59 smithi099 sudo[78441]: pam_unix(sudo:session): session closed for user root 2024-02-09T17:19:00.174 INFO:journalctl@ceph.osd.2.smithi099.stdout:Feb 09 17:19:00 smithi099 sudo[78457]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-09T17:19:00.175 INFO:journalctl@ceph.osd.2.smithi099.stdout:Feb 09 17:19:00 smithi099 sudo[78457]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-09T17:19:00.175 INFO:journalctl@ceph.osd.2.smithi099.stdout:Feb 09 17:19:00 smithi099 sudo[78457]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-09T17:19:00.175 INFO:journalctl@ceph.osd.2.smithi099.stdout:Feb 09 17:19:00 smithi099 sudo[78457]: pam_unix(sudo:session): session closed for user root 2024-02-09T17:19:00.494 INFO:journalctl@ceph.osd.2.smithi099.stdout:Feb 09 17:19:00 smithi099 sudo[78466]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-09T17:19:00.495 INFO:journalctl@ceph.osd.2.smithi099.stdout:Feb 09 17:19:00 smithi099 sudo[78466]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-09T17:19:00.495 INFO:journalctl@ceph.osd.2.smithi099.stdout:Feb 09 17:19:00 smithi099 sudo[78466]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-09T17:19:00.495 INFO:journalctl@ceph.osd.2.smithi099.stdout:Feb 09 17:19:00 smithi099 sudo[78466]: pam_unix(sudo:session): session closed for user root 2024-02-09T17:19:00.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:00 smithi099 ceph-mon[39420]: from='client.24142 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi099:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:19:00.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:00 smithi099 ceph-mon[39420]: osdmap e22: 3 total, 3 up, 3 in 2024-02-09T17:19:00.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:00 smithi099 ceph-mon[39420]: mgrmap e15: y(active, since 2m), standbys: x 2024-02-09T17:19:00.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:00 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:00.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:00 smithi179 ceph-mon[43975]: from='client.24142 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi099:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:19:00.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:00 smithi179 ceph-mon[43975]: osdmap e22: 3 total, 3 up, 3 in 2024-02-09T17:19:00.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:00 smithi179 ceph-mon[43975]: mgrmap e15: y(active, since 2m), standbys: x 2024-02-09T17:19:00.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:00 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:00.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:00 smithi162 ceph-mon[44304]: from='client.24142 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi099:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:19:00.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:00 smithi162 ceph-mon[44304]: osdmap e22: 3 total, 3 up, 3 in 2024-02-09T17:19:00.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:00 smithi162 ceph-mon[44304]: mgrmap e15: y(active, since 2m), standbys: x 2024-02-09T17:19:00.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:00 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:00.907 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:00 smithi099 sudo[78469]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-09T17:19:00.907 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:00 smithi099 sudo[78469]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-09T17:19:00.907 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:00 smithi099 sudo[78469]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-09T17:19:00.907 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:00 smithi099 sudo[78469]: pam_unix(sudo:session): session closed for user root 2024-02-09T17:19:01.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:00 smithi099 sudo[78495]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-09T17:19:01.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:00 smithi099 sudo[78495]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-09T17:19:01.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:00 smithi099 sudo[78495]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-09T17:19:01.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:00 smithi099 sudo[78495]: pam_unix(sudo:session): session closed for user root 2024-02-09T17:19:01.398 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:01 smithi179 sudo[46409]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-09T17:19:01.398 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:01 smithi179 sudo[46409]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-09T17:19:01.398 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:01 smithi179 sudo[46409]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-09T17:19:01.398 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:01 smithi179 sudo[46409]: pam_unix(sudo:session): session closed for user root 2024-02-09T17:19:01.398 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:01 smithi179 sudo[46412]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-09T17:19:01.398 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:01 smithi179 sudo[46412]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-09T17:19:01.398 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:01 smithi179 sudo[46412]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-09T17:19:01.398 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:01 smithi179 sudo[46412]: pam_unix(sudo:session): session closed for user root 2024-02-09T17:19:01.650 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:01 smithi162 ceph-mon[44304]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-09T17:19:01.650 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:01 smithi162 ceph-mon[44304]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-09T17:19:01.650 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:01 smithi162 ceph-mon[44304]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-09T17:19:01.650 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:01 smithi162 ceph-mon[44304]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-09T17:19:01.650 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:01 smithi162 ceph-mon[44304]: Cluster is now healthy 2024-02-09T17:19:01.650 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:01 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:19:01.651 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:01 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:19:01.651 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:01 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:19:01.651 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:01 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:19:01.651 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:01 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:19:01.651 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:01 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:19:01.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:01 smithi179 ceph-mon[43975]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-09T17:19:01.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:01 smithi179 ceph-mon[43975]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-09T17:19:01.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:01 smithi179 ceph-mon[43975]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-09T17:19:01.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:01 smithi179 ceph-mon[43975]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-09T17:19:01.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:01 smithi179 ceph-mon[43975]: Cluster is now healthy 2024-02-09T17:19:01.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:01 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:19:01.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:01 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:19:01.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:01 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:19:01.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:01 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:19:01.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:01 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:19:01.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:01 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:19:01.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:01 smithi099 ceph-mon[39420]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-09T17:19:01.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:01 smithi099 ceph-mon[39420]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-09T17:19:01.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:01 smithi099 ceph-mon[39420]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-09T17:19:01.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:01 smithi099 ceph-mon[39420]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-09T17:19:01.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:01 smithi099 ceph-mon[39420]: Cluster is now healthy 2024-02-09T17:19:01.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:01 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:19:01.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:01 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:19:01.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:01 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:19:01.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:01 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:19:01.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:01 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:19:01.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:01 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:19:01.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:01 smithi162 sudo[46751]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-09T17:19:01.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:01 smithi162 sudo[46751]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-09T17:19:01.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:01 smithi162 sudo[46751]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-09T17:19:01.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:01 smithi162 sudo[46751]: pam_unix(sudo:session): session closed for user root 2024-02-09T17:19:01.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:01 smithi162 sudo[46754]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-09T17:19:01.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:01 smithi162 sudo[46754]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-09T17:19:01.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:01 smithi162 sudo[46754]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-09T17:19:01.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:01 smithi162 sudo[46754]: pam_unix(sudo:session): session closed for user root 2024-02-09T17:19:02.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:02 smithi179 ceph-mon[43975]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-09T17:19:02.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:02 smithi179 ceph-mon[43975]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-09T17:19:02.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:02 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:02.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:02 smithi179 ceph-mon[43975]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-09T17:19:02.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:02 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:19:02.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:02 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:19:02.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:02 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:19:02.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:02 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1629212102' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a520bcf-81bb-4f1d-80fb-11e98008e429"}]: dispatch 2024-02-09T17:19:02.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:02 smithi179 ceph-mon[43975]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-09T17:19:02.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:02 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1629212102' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1a520bcf-81bb-4f1d-80fb-11e98008e429"}]': finished 2024-02-09T17:19:02.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:02 smithi179 ceph-mon[43975]: osdmap e23: 4 total, 3 up, 4 in 2024-02-09T17:19:02.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:02 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T17:19:02.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:02 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:02.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:02 smithi162 ceph-mon[44304]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-09T17:19:02.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:02 smithi162 ceph-mon[44304]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-09T17:19:02.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:02 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:02.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:02 smithi162 ceph-mon[44304]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-09T17:19:02.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:02 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:19:02.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:02 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:19:02.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:02 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:19:02.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:02 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1629212102' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a520bcf-81bb-4f1d-80fb-11e98008e429"}]: dispatch 2024-02-09T17:19:02.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:02 smithi162 ceph-mon[44304]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-09T17:19:02.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:02 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1629212102' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1a520bcf-81bb-4f1d-80fb-11e98008e429"}]': finished 2024-02-09T17:19:02.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:02 smithi162 ceph-mon[44304]: osdmap e23: 4 total, 3 up, 4 in 2024-02-09T17:19:02.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:02 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T17:19:02.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:02 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:02.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:02 smithi099 ceph-mon[39420]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-09T17:19:02.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:02 smithi099 ceph-mon[39420]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-09T17:19:02.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:02 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:02.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:02 smithi099 ceph-mon[39420]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-09T17:19:02.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:02 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:19:02.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:02 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:19:02.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:02 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:19:02.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:02 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1629212102' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a520bcf-81bb-4f1d-80fb-11e98008e429"}]: dispatch 2024-02-09T17:19:02.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:02 smithi099 ceph-mon[39420]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-09T17:19:02.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:02 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1629212102' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1a520bcf-81bb-4f1d-80fb-11e98008e429"}]': finished 2024-02-09T17:19:02.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:02 smithi099 ceph-mon[39420]: osdmap e23: 4 total, 3 up, 4 in 2024-02-09T17:19:02.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:02 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T17:19:02.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:02 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:03.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:03 smithi179 ceph-mon[43975]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-09T17:19:03.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:03 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/687391795' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T17:19:03.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:03 smithi162 ceph-mon[44304]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-09T17:19:03.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:03 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/687391795' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T17:19:03.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:03 smithi099 ceph-mon[39420]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-09T17:19:03.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:03 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/687391795' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T17:19:05.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:05 smithi099 ceph-mon[39420]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-02-09T17:19:05.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:05 smithi179 ceph-mon[43975]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-02-09T17:19:05.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:05 smithi162 ceph-mon[44304]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-02-09T17:19:07.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:07 smithi179 ceph-mon[43975]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 218 KiB/s wr, 14 op/s 2024-02-09T17:19:07.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:07 smithi162 ceph-mon[44304]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 218 KiB/s wr, 14 op/s 2024-02-09T17:19:07.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:07 smithi099 ceph-mon[39420]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 218 KiB/s wr, 14 op/s 2024-02-09T17:19:09.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:09 smithi179 ceph-mon[43975]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 3.4 KiB/s rd, 163 KiB/s wr, 10 op/s 2024-02-09T17:19:09.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:09 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-09T17:19:09.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:09 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:19:09.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:09 smithi179 ceph-mon[43975]: Deploying daemon osd.3 on smithi099 2024-02-09T17:19:09.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:09 smithi162 ceph-mon[44304]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 3.4 KiB/s rd, 163 KiB/s wr, 10 op/s 2024-02-09T17:19:09.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:09 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-09T17:19:09.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:09 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:19:09.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:09 smithi162 ceph-mon[44304]: Deploying daemon osd.3 on smithi099 2024-02-09T17:19:09.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:09 smithi099 ceph-mon[39420]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 3.4 KiB/s rd, 163 KiB/s wr, 10 op/s 2024-02-09T17:19:09.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:09 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-09T17:19:09.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:09 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:19:09.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:09 smithi099 ceph-mon[39420]: Deploying daemon osd.3 on smithi099 2024-02-09T17:19:10.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:10 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:19:10.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:10 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:19:10.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:10 smithi179 ceph-mon[43975]: osdmap e24: 4 total, 3 up, 4 in 2024-02-09T17:19:10.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:10 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T17:19:10.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:10 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:19:10.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:10 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:19:10.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:10 smithi162 ceph-mon[44304]: osdmap e24: 4 total, 3 up, 4 in 2024-02-09T17:19:10.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:10 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T17:19:10.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:10 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:19:10.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:10 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:19:10.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:10 smithi099 ceph-mon[39420]: osdmap e24: 4 total, 3 up, 4 in 2024-02-09T17:19:10.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:10 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T17:19:11.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:11 smithi179 ceph-mon[43975]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 3.2 KiB/s rd, 156 KiB/s wr, 10 op/s 2024-02-09T17:19:11.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:11 smithi162 ceph-mon[44304]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 3.2 KiB/s rd, 156 KiB/s wr, 10 op/s 2024-02-09T17:19:11.690 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:11 smithi099 ceph-mon[39420]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 3.2 KiB/s rd, 156 KiB/s wr, 10 op/s 2024-02-09T17:19:13.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:13 smithi179 ceph-mon[43975]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-02-09T17:19:13.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:13 smithi162 ceph-mon[44304]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-02-09T17:19:13.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:13 smithi099 ceph-mon[39420]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-02-09T17:19:15.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:14 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:15.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:14 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:15.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:14 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:19:15.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:14 smithi179 ceph-mon[43975]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-02-09T17:19:15.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:14 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:15.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:14 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:15.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:14 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:19:15.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:14 smithi162 ceph-mon[44304]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-02-09T17:19:15.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:14 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:15.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:14 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:15.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:14 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:19:15.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:14 smithi099 ceph-mon[39420]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-02-09T17:19:17.176 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:16 smithi099 ceph-mon[39420]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-02-09T17:19:17.176 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:16 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:17.176 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:16 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:17.176 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:16 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:19:17.176 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:16 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:19:17.176 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:16 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:17.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:16 smithi179 ceph-mon[43975]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-02-09T17:19:17.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:16 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:17.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:16 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:17.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:16 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:19:17.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:16 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:19:17.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:16 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:17.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:16 smithi162 ceph-mon[44304]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-02-09T17:19:17.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:16 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:17.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:16 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:17.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:16 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:19:17.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:16 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:19:17.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:16 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:17.500 INFO:teuthology.orchestra.run.smithi099.stdout:Created osd(s) 3 on host 'smithi099' 2024-02-09T17:19:18.224 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:17 smithi099 ceph-mon[39420]: from='osd.3 [v2:172.21.15.99:6826/762784793,v1:172.21.15.99:6827/762784793]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-09T17:19:18.224 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:17 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:18.225 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:17 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:18.225 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:17 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:19:18.258 DEBUG:teuthology.orchestra.run.smithi099:osd.3> sudo journalctl -f -n 0 -u ceph-a69c870a-c76e-11ee-95b7-87774f69a715@osd.3.service 2024-02-09T17:19:18.260 INFO:tasks.cephadm:Deploying osd.4 on smithi162 with /dev/vg_nvme/lv_4... 2024-02-09T17:19:18.261 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-09T17:19:18.291 INFO:journalctl@ceph.osd.3.smithi099.stdout:-- Logs begin at Fri 2024-02-09 17:04:25 UTC. -- 2024-02-09T17:19:18.292 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:17 smithi162 ceph-mon[44304]: from='osd.3 [v2:172.21.15.99:6826/762784793,v1:172.21.15.99:6827/762784793]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-09T17:19:18.292 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:17 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:18.293 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:17 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:18.293 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:17 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:19:18.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:17 smithi179 ceph-mon[43975]: from='osd.3 [v2:172.21.15.99:6826/762784793,v1:172.21.15.99:6827/762784793]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-09T17:19:18.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:17 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:18.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:17 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:18.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:17 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:19:18.530 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:19:19.034 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:18 smithi162 ceph-mon[44304]: from='osd.3 [v2:172.21.15.99:6826/762784793,v1:172.21.15.99:6827/762784793]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-09T17:19:19.034 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:18 smithi162 ceph-mon[44304]: osdmap e25: 4 total, 3 up, 4 in 2024-02-09T17:19:19.034 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:18 smithi162 ceph-mon[44304]: from='osd.3 [v2:172.21.15.99:6826/762784793,v1:172.21.15.99:6827/762784793]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-02-09T17:19:19.034 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:18 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T17:19:19.035 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:18 smithi162 ceph-mon[44304]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-02-09T17:19:19.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:18 smithi099 ceph-mon[39420]: from='osd.3 [v2:172.21.15.99:6826/762784793,v1:172.21.15.99:6827/762784793]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-09T17:19:19.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:18 smithi099 ceph-mon[39420]: osdmap e25: 4 total, 3 up, 4 in 2024-02-09T17:19:19.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:18 smithi099 ceph-mon[39420]: from='osd.3 [v2:172.21.15.99:6826/762784793,v1:172.21.15.99:6827/762784793]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-02-09T17:19:19.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:18 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T17:19:19.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:18 smithi099 ceph-mon[39420]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-02-09T17:19:19.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:18 smithi179 ceph-mon[43975]: from='osd.3 [v2:172.21.15.99:6826/762784793,v1:172.21.15.99:6827/762784793]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-09T17:19:19.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:18 smithi179 ceph-mon[43975]: osdmap e25: 4 total, 3 up, 4 in 2024-02-09T17:19:19.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:18 smithi179 ceph-mon[43975]: from='osd.3 [v2:172.21.15.99:6826/762784793,v1:172.21.15.99:6827/762784793]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-02-09T17:19:19.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:18 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T17:19:19.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:18 smithi179 ceph-mon[43975]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-02-09T17:19:19.744 INFO:journalctl@ceph.osd.3.smithi099.stdout:Feb 09 17:19:19 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-3[80449]: 2024-02-09T17:19:19.388+0000 7fe43c3f6700 -1 osd.3 0 waiting for initial osdmap 2024-02-09T17:19:19.745 INFO:journalctl@ceph.osd.3.smithi099.stdout:Feb 09 17:19:19 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-3[80449]: 2024-02-09T17:19:19.392+0000 7fe4341e8700 -1 osd.3 26 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-09T17:19:20.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:19 smithi162 ceph-mon[44304]: from='osd.3 [v2:172.21.15.99:6826/762784793,v1:172.21.15.99:6827/762784793]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]': finished 2024-02-09T17:19:20.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:19 smithi162 ceph-mon[44304]: osdmap e26: 4 total, 3 up, 4 in 2024-02-09T17:19:20.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:19 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T17:19:20.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:19 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T17:19:20.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:19 smithi162 ceph-mon[44304]: from='osd.3 [v2:172.21.15.99:6826/762784793,v1:172.21.15.99:6827/762784793]' entity='osd.3' 2024-02-09T17:19:20.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:19 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T17:19:20.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:19 smithi099 ceph-mon[39420]: from='osd.3 [v2:172.21.15.99:6826/762784793,v1:172.21.15.99:6827/762784793]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]': finished 2024-02-09T17:19:20.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:19 smithi099 ceph-mon[39420]: osdmap e26: 4 total, 3 up, 4 in 2024-02-09T17:19:20.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:19 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T17:19:20.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:19 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T17:19:20.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:19 smithi099 ceph-mon[39420]: from='osd.3 [v2:172.21.15.99:6826/762784793,v1:172.21.15.99:6827/762784793]' entity='osd.3' 2024-02-09T17:19:20.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:19 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T17:19:20.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:19 smithi179 ceph-mon[43975]: from='osd.3 [v2:172.21.15.99:6826/762784793,v1:172.21.15.99:6827/762784793]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]': finished 2024-02-09T17:19:20.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:19 smithi179 ceph-mon[43975]: osdmap e26: 4 total, 3 up, 4 in 2024-02-09T17:19:20.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:19 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T17:19:20.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:19 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T17:19:20.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:19 smithi179 ceph-mon[43975]: from='osd.3 [v2:172.21.15.99:6826/762784793,v1:172.21.15.99:6827/762784793]' entity='osd.3' 2024-02-09T17:19:20.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:19 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T17:19:20.439 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-09T17:19:20.439 INFO:teuthology.orchestra.run.smithi162.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-09T17:19:20.439 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 10+0 records in 2024-02-09T17:19:20.439 INFO:teuthology.orchestra.run.smithi162.stderr:10+0 records out 2024-02-09T17:19:20.439 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0116411 s, 901 MB/s 2024-02-09T17:19:20.439 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping successful for: 2024-02-09T17:19:21.071 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:20 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:19:21.071 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:20 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:19:21.071 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:20 smithi162 ceph-mon[44304]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-02-09T17:19:21.071 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:20 smithi162 ceph-mon[44304]: osd.3 [v2:172.21.15.99:6826/762784793,v1:172.21.15.99:6827/762784793] boot 2024-02-09T17:19:21.071 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:20 smithi162 ceph-mon[44304]: osdmap e27: 4 total, 4 up, 4 in 2024-02-09T17:19:21.071 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:20 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T17:19:21.096 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph orch daemon add osd smithi162:vg_nvme/lv_4 2024-02-09T17:19:21.195 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:20 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:19:21.195 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:20 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:19:21.195 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:20 smithi099 ceph-mon[39420]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-02-09T17:19:21.195 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:20 smithi099 ceph-mon[39420]: osd.3 [v2:172.21.15.99:6826/762784793,v1:172.21.15.99:6827/762784793] boot 2024-02-09T17:19:21.195 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:20 smithi099 ceph-mon[39420]: osdmap e27: 4 total, 4 up, 4 in 2024-02-09T17:19:21.195 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:20 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T17:19:21.395 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:19:21.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:20 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:19:21.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:20 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:19:21.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:20 smithi179 ceph-mon[43975]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-02-09T17:19:21.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:20 smithi179 ceph-mon[43975]: osd.3 [v2:172.21.15.99:6826/762784793,v1:172.21.15.99:6827/762784793] boot 2024-02-09T17:19:21.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:20 smithi179 ceph-mon[43975]: osdmap e27: 4 total, 4 up, 4 in 2024-02-09T17:19:21.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:20 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T17:19:22.776 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:22 smithi162 ceph-mon[44304]: Detected new or changed devices on smithi099 2024-02-09T17:19:22.776 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:22 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:22.776 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:22 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:22.776 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:22 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:19:22.776 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:22 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:19:22.777 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:22 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:19:22.777 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:22 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:19:22.777 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:22 smithi162 ceph-mon[44304]: Adjusting osd_memory_target on smithi099 to 4280M 2024-02-09T17:19:22.777 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:22 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:22.777 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:22 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:19:22.777 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:22 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:19:22.777 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:22 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:22.777 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:22 smithi162 ceph-mon[44304]: osdmap e28: 4 total, 4 up, 4 in 2024-02-09T17:19:22.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:22 smithi179 ceph-mon[43975]: Detected new or changed devices on smithi099 2024-02-09T17:19:22.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:22 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:22.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:22 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:22.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:22 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:19:22.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:22 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:19:22.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:22 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:19:22.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:22 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:19:22.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:22 smithi179 ceph-mon[43975]: Adjusting osd_memory_target on smithi099 to 4280M 2024-02-09T17:19:22.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:22 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:22.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:22 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:19:22.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:22 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:19:22.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:22 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:22.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:22 smithi179 ceph-mon[43975]: osdmap e28: 4 total, 4 up, 4 in 2024-02-09T17:19:22.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:22 smithi099 ceph-mon[39420]: Detected new or changed devices on smithi099 2024-02-09T17:19:22.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:22 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:22.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:22 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:22.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:22 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:19:22.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:22 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:19:22.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:22 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:19:22.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:22 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:19:22.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:22 smithi099 ceph-mon[39420]: Adjusting osd_memory_target on smithi099 to 4280M 2024-02-09T17:19:22.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:22 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:22.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:22 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:19:22.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:22 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:19:22.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:22 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:22.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:22 smithi099 ceph-mon[39420]: osdmap e28: 4 total, 4 up, 4 in 2024-02-09T17:19:23.776 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:23 smithi162 ceph-mon[44304]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-09T17:19:23.776 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:23 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T17:19:23.776 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:23 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T17:19:23.776 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:23 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:19:23.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:23 smithi179 ceph-mon[43975]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-09T17:19:23.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:23 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T17:19:23.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:23 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T17:19:23.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:23 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:19:23.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:23 smithi099 ceph-mon[39420]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-09T17:19:23.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:23 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T17:19:23.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:23 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T17:19:23.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:23 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:19:24.662 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:24 smithi162 ceph-mon[44304]: from='client.24188 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:19:24.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:24 smithi179 ceph-mon[43975]: from='client.24188 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:19:24.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:24 smithi099 ceph-mon[39420]: from='client.24188 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:19:25.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:25 smithi179 ceph-mon[43975]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-09T17:19:25.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:25 smithi162 ceph-mon[44304]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-09T17:19:25.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:25 smithi099 ceph-mon[39420]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-09T17:19:26.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:26 smithi179 ceph-mon[43975]: from='client.? 172.21.15.162:0/1746730246' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4617c2fe-c9d8-40cd-887b-303e3215bdf8"}]: dispatch 2024-02-09T17:19:26.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:26 smithi179 ceph-mon[43975]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4617c2fe-c9d8-40cd-887b-303e3215bdf8"}]: dispatch 2024-02-09T17:19:26.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:26 smithi179 ceph-mon[43975]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4617c2fe-c9d8-40cd-887b-303e3215bdf8"}]': finished 2024-02-09T17:19:26.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:26 smithi179 ceph-mon[43975]: osdmap e29: 5 total, 4 up, 5 in 2024-02-09T17:19:26.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:26 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-09T17:19:26.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:26 smithi179 ceph-mon[43975]: from='client.? 172.21.15.162:0/680021469' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T17:19:26.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:26 smithi162 ceph-mon[44304]: from='client.? 172.21.15.162:0/1746730246' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4617c2fe-c9d8-40cd-887b-303e3215bdf8"}]: dispatch 2024-02-09T17:19:26.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:26 smithi162 ceph-mon[44304]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4617c2fe-c9d8-40cd-887b-303e3215bdf8"}]: dispatch 2024-02-09T17:19:26.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:26 smithi162 ceph-mon[44304]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4617c2fe-c9d8-40cd-887b-303e3215bdf8"}]': finished 2024-02-09T17:19:26.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:26 smithi162 ceph-mon[44304]: osdmap e29: 5 total, 4 up, 5 in 2024-02-09T17:19:26.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:26 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-09T17:19:26.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:26 smithi162 ceph-mon[44304]: from='client.? 172.21.15.162:0/680021469' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T17:19:26.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:26 smithi099 ceph-mon[39420]: from='client.? 172.21.15.162:0/1746730246' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4617c2fe-c9d8-40cd-887b-303e3215bdf8"}]: dispatch 2024-02-09T17:19:26.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:26 smithi099 ceph-mon[39420]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4617c2fe-c9d8-40cd-887b-303e3215bdf8"}]: dispatch 2024-02-09T17:19:26.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:26 smithi099 ceph-mon[39420]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4617c2fe-c9d8-40cd-887b-303e3215bdf8"}]': finished 2024-02-09T17:19:26.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:26 smithi099 ceph-mon[39420]: osdmap e29: 5 total, 4 up, 5 in 2024-02-09T17:19:26.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:26 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-09T17:19:26.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:26 smithi099 ceph-mon[39420]: from='client.? 172.21.15.162:0/680021469' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T17:19:27.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:27 smithi179 ceph-mon[43975]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-09T17:19:27.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:27 smithi162 ceph-mon[44304]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-09T17:19:27.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:27 smithi099 ceph-mon[39420]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-09T17:19:28.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:28 smithi179 ceph-mon[43975]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-09T17:19:28.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:28 smithi162 ceph-mon[44304]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-09T17:19:28.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:28 smithi099 ceph-mon[39420]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-09T17:19:31.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:31 smithi179 ceph-mon[43975]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-09T17:19:31.696 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:31 smithi162 ceph-mon[44304]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-09T17:19:31.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:31 smithi099 ceph-mon[39420]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-09T17:19:32.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:32 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-09T17:19:32.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:32 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:19:32.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:32 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-09T17:19:32.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:32 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:19:32.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:32 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-09T17:19:32.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:32 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:19:33.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:33 smithi162 ceph-mon[44304]: Deploying daemon osd.4 on smithi162 2024-02-09T17:19:33.558 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:33 smithi162 ceph-mon[44304]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-09T17:19:33.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:33 smithi179 ceph-mon[43975]: Deploying daemon osd.4 on smithi162 2024-02-09T17:19:33.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:33 smithi179 ceph-mon[43975]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-09T17:19:33.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:33 smithi099 ceph-mon[39420]: Deploying daemon osd.4 on smithi162 2024-02-09T17:19:33.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:33 smithi099 ceph-mon[39420]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-09T17:19:35.664 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:35 smithi162 ceph-mon[44304]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-09T17:19:35.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:35 smithi179 ceph-mon[43975]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-09T17:19:35.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:35 smithi099 ceph-mon[39420]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-09T17:19:37.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:36 smithi099 ceph-mon[39420]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-09T17:19:37.292 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:36 smithi162 ceph-mon[44304]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-09T17:19:37.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:36 smithi179 ceph-mon[43975]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-09T17:19:38.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:38 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:38.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:38 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:38.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:38 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:19:38.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:38 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:38.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:38 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:38.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:38 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:19:38.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:38 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:38.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:38 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:38.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:38 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:19:39.730 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:39 smithi162 ceph-mon[44304]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-09T17:19:39.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:39 smithi099 ceph-mon[39420]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-09T17:19:39.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:39 smithi179 ceph-mon[43975]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-09T17:19:40.993 INFO:teuthology.orchestra.run.smithi162.stdout:Created osd(s) 4 on host 'smithi162' 2024-02-09T17:19:41.020 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:40 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:41.020 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:40 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:41.020 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:40 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:19:41.020 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:40 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:19:41.021 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:40 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:41.021 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:40 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:19:41.021 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:40 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:19:41.021 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:40 smithi162 ceph-mon[44304]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-09T17:19:41.021 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:40 smithi162 ceph-mon[44304]: from='osd.4 [v2:172.21.15.162:6800/4178676504,v1:172.21.15.162:6801/4178676504]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-09T17:19:41.021 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:40 smithi162 ceph-mon[44304]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-09T17:19:41.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:40 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:41.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:40 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:41.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:40 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:19:41.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:40 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:19:41.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:40 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:41.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:40 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:19:41.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:40 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:19:41.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:40 smithi179 ceph-mon[43975]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-09T17:19:41.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:40 smithi179 ceph-mon[43975]: from='osd.4 [v2:172.21.15.162:6800/4178676504,v1:172.21.15.162:6801/4178676504]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-09T17:19:41.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:40 smithi179 ceph-mon[43975]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-09T17:19:41.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:40 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:41.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:40 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:41.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:40 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:19:41.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:40 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:19:41.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:40 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:41.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:40 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:19:41.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:40 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:19:41.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:40 smithi099 ceph-mon[39420]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-09T17:19:41.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:40 smithi099 ceph-mon[39420]: from='osd.4 [v2:172.21.15.162:6800/4178676504,v1:172.21.15.162:6801/4178676504]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-09T17:19:41.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:40 smithi099 ceph-mon[39420]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-09T17:19:41.691 DEBUG:teuthology.orchestra.run.smithi162:osd.4> sudo journalctl -f -n 0 -u ceph-a69c870a-c76e-11ee-95b7-87774f69a715@osd.4.service 2024-02-09T17:19:41.694 INFO:tasks.cephadm:Deploying osd.5 on smithi162 with /dev/vg_nvme/lv_3... 2024-02-09T17:19:41.694 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-09T17:19:41.726 INFO:journalctl@ceph.osd.4.smithi162.stdout:-- Logs begin at Fri 2024-02-09 17:04:37 UTC. -- 2024-02-09T17:19:41.927 INFO:journalctl@ceph.osd.4.smithi162.stdout:Feb 09 17:19:41 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-4[49654]: 2024-02-09T17:19:41.804+0000 7f1a22e08700 -1 osd.4 0 waiting for initial osdmap 2024-02-09T17:19:41.927 INFO:journalctl@ceph.osd.4.smithi162.stdout:Feb 09 17:19:41 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-4[49654]: 2024-02-09T17:19:41.811+0000 7f1a1ec02700 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-09T17:19:41.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:41 smithi162 ceph-mon[44304]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-09T17:19:41.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:41 smithi162 ceph-mon[44304]: from='osd.4 [v2:172.21.15.162:6800/4178676504,v1:172.21.15.162:6801/4178676504]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-02-09T17:19:41.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:41 smithi162 ceph-mon[44304]: osdmap e30: 5 total, 4 up, 5 in 2024-02-09T17:19:41.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:41 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-09T17:19:41.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:41 smithi162 ceph-mon[44304]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-02-09T17:19:41.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:41 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:41.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:41 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:41.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:41 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:19:42.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:41 smithi179 ceph-mon[43975]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-09T17:19:42.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:41 smithi179 ceph-mon[43975]: from='osd.4 [v2:172.21.15.162:6800/4178676504,v1:172.21.15.162:6801/4178676504]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-02-09T17:19:42.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:41 smithi179 ceph-mon[43975]: osdmap e30: 5 total, 4 up, 5 in 2024-02-09T17:19:42.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:41 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-09T17:19:42.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:41 smithi179 ceph-mon[43975]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-02-09T17:19:42.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:41 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:42.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:41 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:42.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:41 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:19:42.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:41 smithi099 ceph-mon[39420]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-09T17:19:42.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:41 smithi099 ceph-mon[39420]: from='osd.4 [v2:172.21.15.162:6800/4178676504,v1:172.21.15.162:6801/4178676504]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-02-09T17:19:42.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:41 smithi099 ceph-mon[39420]: osdmap e30: 5 total, 4 up, 5 in 2024-02-09T17:19:42.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:41 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-09T17:19:42.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:41 smithi099 ceph-mon[39420]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-02-09T17:19:42.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:41 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:42.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:41 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:42.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:41 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:19:42.532 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:19:43.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:42 smithi179 ceph-mon[43975]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-02-09T17:19:43.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:42 smithi179 ceph-mon[43975]: osdmap e31: 5 total, 4 up, 5 in 2024-02-09T17:19:43.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:42 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-09T17:19:43.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:42 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-09T17:19:43.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:42 smithi179 ceph-mon[43975]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-09T17:19:43.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:42 smithi162 ceph-mon[44304]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-02-09T17:19:43.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:42 smithi162 ceph-mon[44304]: osdmap e31: 5 total, 4 up, 5 in 2024-02-09T17:19:43.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:42 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-09T17:19:43.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:42 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-09T17:19:43.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:42 smithi162 ceph-mon[44304]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-09T17:19:43.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:42 smithi099 ceph-mon[39420]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-02-09T17:19:43.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:42 smithi099 ceph-mon[39420]: osdmap e31: 5 total, 4 up, 5 in 2024-02-09T17:19:43.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:42 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-09T17:19:43.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:42 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-09T17:19:43.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:42 smithi099 ceph-mon[39420]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-09T17:19:44.074 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:43 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:19:44.075 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:43 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:19:44.075 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:43 smithi162 ceph-mon[44304]: osd.4 [v2:172.21.15.162:6800/4178676504,v1:172.21.15.162:6801/4178676504] boot 2024-02-09T17:19:44.075 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:43 smithi162 ceph-mon[44304]: osdmap e32: 5 total, 5 up, 5 in 2024-02-09T17:19:44.075 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:43 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-09T17:19:44.075 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:43 smithi162 ceph-mon[44304]: osdmap e33: 5 total, 5 up, 5 in 2024-02-09T17:19:44.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:43 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:19:44.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:43 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:19:44.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:43 smithi179 ceph-mon[43975]: osd.4 [v2:172.21.15.162:6800/4178676504,v1:172.21.15.162:6801/4178676504] boot 2024-02-09T17:19:44.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:43 smithi179 ceph-mon[43975]: osdmap e32: 5 total, 5 up, 5 in 2024-02-09T17:19:44.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:43 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-09T17:19:44.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:43 smithi179 ceph-mon[43975]: osdmap e33: 5 total, 5 up, 5 in 2024-02-09T17:19:44.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:43 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:19:44.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:43 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:19:44.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:43 smithi099 ceph-mon[39420]: osd.4 [v2:172.21.15.162:6800/4178676504,v1:172.21.15.162:6801/4178676504] boot 2024-02-09T17:19:44.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:43 smithi099 ceph-mon[39420]: osdmap e32: 5 total, 5 up, 5 in 2024-02-09T17:19:44.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:43 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-09T17:19:44.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:43 smithi099 ceph-mon[39420]: osdmap e33: 5 total, 5 up, 5 in 2024-02-09T17:19:44.822 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:44 smithi162 ceph-mon[44304]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T17:19:44.822 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:44 smithi162 ceph-mon[44304]: osdmap e34: 5 total, 5 up, 5 in 2024-02-09T17:19:45.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:44 smithi179 ceph-mon[43975]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T17:19:45.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:44 smithi179 ceph-mon[43975]: osdmap e34: 5 total, 5 up, 5 in 2024-02-09T17:19:45.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:44 smithi099 ceph-mon[39420]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T17:19:45.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:44 smithi099 ceph-mon[39420]: osdmap e34: 5 total, 5 up, 5 in 2024-02-09T17:19:45.255 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-09T17:19:45.255 INFO:teuthology.orchestra.run.smithi162.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-09T17:19:45.255 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 10+0 records in 2024-02-09T17:19:45.255 INFO:teuthology.orchestra.run.smithi162.stderr:10+0 records out 2024-02-09T17:19:45.255 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0133953 s, 783 MB/s 2024-02-09T17:19:45.255 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping successful for: 2024-02-09T17:19:46.514 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph orch daemon add osd smithi162:vg_nvme/lv_3 2024-02-09T17:19:46.772 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:19:47.080 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:46 smithi162 ceph-mon[44304]: osdmap e35: 5 total, 5 up, 5 in 2024-02-09T17:19:47.080 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:46 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:47.080 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:46 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:19:47.080 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:46 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:19:47.080 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:46 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:19:47.080 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:46 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:47.080 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:46 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:19:47.080 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:46 smithi162 ceph-mon[44304]: Adjusting osd_memory_target on smithi162 to 17122M 2024-02-09T17:19:47.080 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:46 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:47.080 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:46 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:19:47.081 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:46 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:19:47.081 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:46 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:47.081 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:46 smithi162 ceph-mon[44304]: pgmap v126: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T17:19:47.081 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:46 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:19:47.081 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:46 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:19:47.081 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:46 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:19:47.081 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:46 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:47.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:46 smithi179 ceph-mon[43975]: osdmap e35: 5 total, 5 up, 5 in 2024-02-09T17:19:47.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:46 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:47.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:46 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:19:47.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:46 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:19:47.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:46 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:19:47.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:46 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:47.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:46 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:19:47.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:46 smithi179 ceph-mon[43975]: Adjusting osd_memory_target on smithi162 to 17122M 2024-02-09T17:19:47.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:46 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:47.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:46 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:19:47.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:46 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:19:47.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:46 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:47.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:46 smithi179 ceph-mon[43975]: pgmap v126: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T17:19:47.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:46 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:19:47.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:46 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:19:47.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:46 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:19:47.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:46 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:47.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:46 smithi099 ceph-mon[39420]: osdmap e35: 5 total, 5 up, 5 in 2024-02-09T17:19:47.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:46 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:47.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:46 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:19:47.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:46 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:19:47.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:46 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:19:47.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:46 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:47.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:46 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:19:47.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:46 smithi099 ceph-mon[39420]: Adjusting osd_memory_target on smithi162 to 17122M 2024-02-09T17:19:47.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:46 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:47.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:46 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:19:47.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:46 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:19:47.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:46 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:47.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:46 smithi099 ceph-mon[39420]: pgmap v126: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T17:19:47.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:46 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:19:47.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:46 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:19:47.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:46 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:19:47.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:46 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:47.920 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:47 smithi162 ceph-mon[44304]: osdmap e36: 5 total, 5 up, 5 in 2024-02-09T17:19:48.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:47 smithi179 ceph-mon[43975]: osdmap e36: 5 total, 5 up, 5 in 2024-02-09T17:19:48.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:47 smithi099 ceph-mon[39420]: osdmap e36: 5 total, 5 up, 5 in 2024-02-09T17:19:49.145 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:48 smithi162 ceph-mon[44304]: pgmap v128: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T17:19:49.145 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:48 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T17:19:49.145 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:48 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T17:19:49.145 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:48 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:19:49.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:48 smithi179 ceph-mon[43975]: pgmap v128: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T17:19:49.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:48 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T17:19:49.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:48 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T17:19:49.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:48 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:19:49.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:48 smithi099 ceph-mon[39420]: pgmap v128: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T17:19:49.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:48 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T17:19:49.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:48 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T17:19:49.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:48 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:19:50.030 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:49 smithi162 ceph-mon[44304]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:19:50.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:49 smithi179 ceph-mon[43975]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:19:50.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:49 smithi099 ceph-mon[39420]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:19:51.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:50 smithi179 ceph-mon[43975]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-02-09T17:19:51.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:50 smithi162 ceph-mon[44304]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-02-09T17:19:51.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:50 smithi099 ceph-mon[39420]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-02-09T17:19:52.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:51 smithi179 ceph-mon[43975]: from='client.? 172.21.15.162:0/84061505' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "da98e3f3-173e-46fe-be8b-1ccbe8b0dabc"}]: dispatch 2024-02-09T17:19:52.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:51 smithi179 ceph-mon[43975]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "da98e3f3-173e-46fe-be8b-1ccbe8b0dabc"}]: dispatch 2024-02-09T17:19:52.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:51 smithi179 ceph-mon[43975]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "da98e3f3-173e-46fe-be8b-1ccbe8b0dabc"}]': finished 2024-02-09T17:19:52.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:51 smithi179 ceph-mon[43975]: osdmap e37: 6 total, 5 up, 6 in 2024-02-09T17:19:52.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:51 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-09T17:19:52.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:51 smithi162 ceph-mon[44304]: from='client.? 172.21.15.162:0/84061505' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "da98e3f3-173e-46fe-be8b-1ccbe8b0dabc"}]: dispatch 2024-02-09T17:19:52.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:51 smithi162 ceph-mon[44304]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "da98e3f3-173e-46fe-be8b-1ccbe8b0dabc"}]: dispatch 2024-02-09T17:19:52.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:51 smithi162 ceph-mon[44304]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "da98e3f3-173e-46fe-be8b-1ccbe8b0dabc"}]': finished 2024-02-09T17:19:52.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:51 smithi162 ceph-mon[44304]: osdmap e37: 6 total, 5 up, 6 in 2024-02-09T17:19:52.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:51 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-09T17:19:52.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:51 smithi099 ceph-mon[39420]: from='client.? 172.21.15.162:0/84061505' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "da98e3f3-173e-46fe-be8b-1ccbe8b0dabc"}]: dispatch 2024-02-09T17:19:52.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:51 smithi099 ceph-mon[39420]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "da98e3f3-173e-46fe-be8b-1ccbe8b0dabc"}]: dispatch 2024-02-09T17:19:52.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:51 smithi099 ceph-mon[39420]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "da98e3f3-173e-46fe-be8b-1ccbe8b0dabc"}]': finished 2024-02-09T17:19:52.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:51 smithi099 ceph-mon[39420]: osdmap e37: 6 total, 5 up, 6 in 2024-02-09T17:19:52.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:51 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-09T17:19:53.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:52 smithi179 ceph-mon[43975]: from='client.? 172.21.15.162:0/3897055038' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T17:19:53.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:52 smithi179 ceph-mon[43975]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-02-09T17:19:53.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:52 smithi162 ceph-mon[44304]: from='client.? 172.21.15.162:0/3897055038' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T17:19:53.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:52 smithi162 ceph-mon[44304]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-02-09T17:19:53.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:52 smithi099 ceph-mon[39420]: from='client.? 172.21.15.162:0/3897055038' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T17:19:53.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:52 smithi099 ceph-mon[39420]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-02-09T17:19:55.615 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:55 smithi162 ceph-mon[44304]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-09T17:19:55.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:55 smithi179 ceph-mon[43975]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-09T17:19:55.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:55 smithi099 ceph-mon[39420]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-09T17:19:56.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:56 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:19:56.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:56 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:19:56.547 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:56 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:19:56.548 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:56 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:19:56.548 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:56 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:19:56.548 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:56 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:19:56.548 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:56 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:56.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:56 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:19:56.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:56 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:19:56.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:56 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:19:56.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:56 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:19:56.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:56 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:19:56.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:56 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:19:56.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:56 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:56.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:56 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:19:56.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:56 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:19:56.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:56 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:19:56.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:56 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:19:56.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:56 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:19:56.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:56 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:19:56.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:56 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:19:57.622 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:57 smithi162 ceph-mon[44304]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-02-09T17:19:57.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:57 smithi179 ceph-mon[43975]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-02-09T17:19:57.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:57 smithi099 ceph-mon[39420]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-02-09T17:19:58.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:58 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-09T17:19:58.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:58 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:19:58.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:58 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-09T17:19:58.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:58 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:19:58.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:58 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-09T17:19:58.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:58 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:19:59.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:59 smithi162 ceph-mon[44304]: Deploying daemon osd.5 on smithi162 2024-02-09T17:19:59.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:19:59 smithi162 ceph-mon[44304]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-09T17:19:59.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:59 smithi179 ceph-mon[43975]: Deploying daemon osd.5 on smithi162 2024-02-09T17:19:59.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:19:59 smithi179 ceph-mon[43975]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-09T17:19:59.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:59 smithi099 ceph-mon[39420]: Deploying daemon osd.5 on smithi162 2024-02-09T17:19:59.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:19:59 smithi099 ceph-mon[39420]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-09T17:20:00.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:00 smithi162 ceph-mon[44304]: 1.0 scrub starts 2024-02-09T17:20:00.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:00 smithi162 ceph-mon[44304]: 1.0 scrub ok 2024-02-09T17:20:00.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:00 smithi162 ceph-mon[44304]: overall HEALTH_OK 2024-02-09T17:20:00.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:00 smithi179 ceph-mon[43975]: 1.0 scrub starts 2024-02-09T17:20:00.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:00 smithi179 ceph-mon[43975]: 1.0 scrub ok 2024-02-09T17:20:00.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:00 smithi179 ceph-mon[43975]: overall HEALTH_OK 2024-02-09T17:20:00.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:00 smithi099 ceph-mon[39420]: 1.0 scrub starts 2024-02-09T17:20:00.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:00 smithi099 ceph-mon[39420]: 1.0 scrub ok 2024-02-09T17:20:00.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:00 smithi099 ceph-mon[39420]: overall HEALTH_OK 2024-02-09T17:20:01.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:01 smithi179 ceph-mon[43975]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T17:20:01.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:01 smithi162 ceph-mon[44304]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T17:20:01.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:01 smithi099 ceph-mon[39420]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T17:20:03.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:03 smithi162 ceph-mon[44304]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T17:20:03.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:03 smithi179 ceph-mon[43975]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T17:20:03.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:03 smithi099 ceph-mon[39420]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T17:20:04.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:04 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:04.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:04 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:04.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:04 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:20:04.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:04 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:04.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:04 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:04.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:04 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:20:04.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:04 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:04.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:04 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:04.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:04 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:20:05.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:05 smithi179 ceph-mon[43975]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T17:20:05.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:05 smithi162 ceph-mon[44304]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T17:20:05.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:05 smithi099 ceph-mon[39420]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T17:20:06.959 INFO:teuthology.orchestra.run.smithi162.stdout:Created osd(s) 5 on host 'smithi162' 2024-02-09T17:20:07.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:06 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:07.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:06 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:07.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:06 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:20:07.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:06 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:20:07.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:06 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:07.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:06 smithi099 ceph-mon[39420]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T17:20:07.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:06 smithi099 ceph-mon[39420]: from='osd.5 [v2:172.21.15.162:6808/1489371339,v1:172.21.15.162:6809/1489371339]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-09T17:20:07.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:06 smithi099 ceph-mon[39420]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-09T17:20:07.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:06 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:07.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:06 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:07.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:06 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:20:07.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:06 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:07.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:06 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:07.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:06 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:20:07.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:06 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:20:07.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:06 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:07.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:06 smithi162 ceph-mon[44304]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T17:20:07.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:06 smithi162 ceph-mon[44304]: from='osd.5 [v2:172.21.15.162:6808/1489371339,v1:172.21.15.162:6809/1489371339]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-09T17:20:07.306 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:06 smithi162 ceph-mon[44304]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-09T17:20:07.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:06 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:07.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:06 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:07.307 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:06 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:20:07.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:06 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:07.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:06 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:07.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:06 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:20:07.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:06 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:20:07.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:06 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:07.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:06 smithi179 ceph-mon[43975]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T17:20:07.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:06 smithi179 ceph-mon[43975]: from='osd.5 [v2:172.21.15.162:6808/1489371339,v1:172.21.15.162:6809/1489371339]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-09T17:20:07.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:06 smithi179 ceph-mon[43975]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-09T17:20:07.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:06 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:07.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:06 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:07.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:06 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:20:07.592 DEBUG:teuthology.orchestra.run.smithi162:osd.5> sudo journalctl -f -n 0 -u ceph-a69c870a-c76e-11ee-95b7-87774f69a715@osd.5.service 2024-02-09T17:20:07.595 INFO:tasks.cephadm:Deploying osd.6 on smithi162 with /dev/vg_nvme/lv_2... 2024-02-09T17:20:07.595 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-09T17:20:07.626 INFO:journalctl@ceph.osd.5.smithi162.stdout:-- Logs begin at Fri 2024-02-09 17:04:37 UTC. -- 2024-02-09T17:20:08.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:07 smithi162 ceph-mon[44304]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-09T17:20:08.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:07 smithi162 ceph-mon[44304]: from='osd.5 [v2:172.21.15.162:6808/1489371339,v1:172.21.15.162:6809/1489371339]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-02-09T17:20:08.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:07 smithi162 ceph-mon[44304]: osdmap e38: 6 total, 5 up, 6 in 2024-02-09T17:20:08.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:07 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-09T17:20:08.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:07 smithi162 ceph-mon[44304]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-02-09T17:20:08.178 INFO:journalctl@ceph.osd.5.smithi162.stdout:Feb 09 17:20:07 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-5[56859]: 2024-02-09T17:20:07.997+0000 7f8e23c2f700 -1 osd.5 0 waiting for initial osdmap 2024-02-09T17:20:08.178 INFO:journalctl@ceph.osd.5.smithi162.stdout:Feb 09 17:20:08 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-5[56859]: 2024-02-09T17:20:08.003+0000 7f8e1fa29700 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-09T17:20:08.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:07 smithi179 ceph-mon[43975]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-09T17:20:08.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:07 smithi179 ceph-mon[43975]: from='osd.5 [v2:172.21.15.162:6808/1489371339,v1:172.21.15.162:6809/1489371339]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-02-09T17:20:08.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:07 smithi179 ceph-mon[43975]: osdmap e38: 6 total, 5 up, 6 in 2024-02-09T17:20:08.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:07 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-09T17:20:08.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:07 smithi179 ceph-mon[43975]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-02-09T17:20:08.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:07 smithi099 ceph-mon[39420]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-09T17:20:08.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:07 smithi099 ceph-mon[39420]: from='osd.5 [v2:172.21.15.162:6808/1489371339,v1:172.21.15.162:6809/1489371339]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-02-09T17:20:08.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:07 smithi099 ceph-mon[39420]: osdmap e38: 6 total, 5 up, 6 in 2024-02-09T17:20:08.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:07 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-09T17:20:08.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:07 smithi099 ceph-mon[39420]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-02-09T17:20:08.802 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:20:09.082 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:08 smithi162 ceph-mon[44304]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-02-09T17:20:09.083 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:08 smithi162 ceph-mon[44304]: osdmap e39: 6 total, 5 up, 6 in 2024-02-09T17:20:09.083 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:09 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-09T17:20:09.083 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:09 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-09T17:20:09.083 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:09 smithi162 ceph-mon[44304]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T17:20:09.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:08 smithi179 ceph-mon[43975]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-02-09T17:20:09.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:08 smithi179 ceph-mon[43975]: osdmap e39: 6 total, 5 up, 6 in 2024-02-09T17:20:09.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:08 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-09T17:20:09.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:09 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-09T17:20:09.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:09 smithi179 ceph-mon[43975]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T17:20:09.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:08 smithi099 ceph-mon[39420]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-02-09T17:20:09.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:08 smithi099 ceph-mon[39420]: osdmap e39: 6 total, 5 up, 6 in 2024-02-09T17:20:09.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:08 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-09T17:20:09.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:08 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-09T17:20:09.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:08 smithi099 ceph-mon[39420]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T17:20:10.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:10 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:20:10.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:10 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:20:10.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:10 smithi179 ceph-mon[43975]: osd.5 [v2:172.21.15.162:6808/1489371339,v1:172.21.15.162:6809/1489371339] boot 2024-02-09T17:20:10.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:10 smithi179 ceph-mon[43975]: osdmap e40: 6 total, 6 up, 6 in 2024-02-09T17:20:10.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:10 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-09T17:20:10.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:10 smithi179 ceph-mon[43975]: osdmap e41: 6 total, 6 up, 6 in 2024-02-09T17:20:10.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:10 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:20:10.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:10 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:20:10.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:10 smithi162 ceph-mon[44304]: osd.5 [v2:172.21.15.162:6808/1489371339,v1:172.21.15.162:6809/1489371339] boot 2024-02-09T17:20:10.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:10 smithi162 ceph-mon[44304]: osdmap e40: 6 total, 6 up, 6 in 2024-02-09T17:20:10.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:10 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-09T17:20:10.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:10 smithi162 ceph-mon[44304]: osdmap e41: 6 total, 6 up, 6 in 2024-02-09T17:20:10.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:10 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:20:10.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:10 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:20:10.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:10 smithi099 ceph-mon[39420]: osd.5 [v2:172.21.15.162:6808/1489371339,v1:172.21.15.162:6809/1489371339] boot 2024-02-09T17:20:10.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:10 smithi099 ceph-mon[39420]: osdmap e40: 6 total, 6 up, 6 in 2024-02-09T17:20:10.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:10 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-09T17:20:10.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:10 smithi099 ceph-mon[39420]: osdmap e41: 6 total, 6 up, 6 in 2024-02-09T17:20:11.060 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-09T17:20:11.060 INFO:teuthology.orchestra.run.smithi162.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-09T17:20:11.060 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 10+0 records in 2024-02-09T17:20:11.061 INFO:teuthology.orchestra.run.smithi162.stderr:10+0 records out 2024-02-09T17:20:11.061 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0119157 s, 880 MB/s 2024-02-09T17:20:11.061 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping successful for: 2024-02-09T17:20:11.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:11 smithi179 ceph-mon[43975]: pgmap v144: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-02-09T17:20:11.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:11 smithi162 ceph-mon[44304]: pgmap v144: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-02-09T17:20:11.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:11 smithi099 ceph-mon[39420]: pgmap v144: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-02-09T17:20:12.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:12 smithi179 ceph-mon[43975]: osdmap e42: 6 total, 6 up, 6 in 2024-02-09T17:20:12.424 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:12 smithi162 ceph-mon[44304]: osdmap e42: 6 total, 6 up, 6 in 2024-02-09T17:20:12.451 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph orch daemon add osd smithi162:vg_nvme/lv_2 2024-02-09T17:20:12.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:12 smithi099 ceph-mon[39420]: osdmap e42: 6 total, 6 up, 6 in 2024-02-09T17:20:12.713 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:20:13.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:13 smithi179 ceph-mon[43975]: osdmap e43: 6 total, 6 up, 6 in 2024-02-09T17:20:13.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:13 smithi179 ceph-mon[43975]: Detected new or changed devices on smithi162 2024-02-09T17:20:13.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:13 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:13.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:13 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:13.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:13 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:20:13.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:13 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:20:13.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:13 smithi179 ceph-mon[43975]: Adjusting osd_memory_target on smithi162 to 8561M 2024-02-09T17:20:13.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:13 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:13.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:13 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:20:13.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:13 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:20:13.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:13 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:13.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:13 smithi179 ceph-mon[43975]: pgmap v147: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-02-09T17:20:13.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:13 smithi162 ceph-mon[44304]: osdmap e43: 6 total, 6 up, 6 in 2024-02-09T17:20:13.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:13 smithi162 ceph-mon[44304]: Detected new or changed devices on smithi162 2024-02-09T17:20:13.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:13 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:13.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:13 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:13.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:13 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:20:13.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:13 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:20:13.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:13 smithi162 ceph-mon[44304]: Adjusting osd_memory_target on smithi162 to 8561M 2024-02-09T17:20:13.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:13 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:13.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:13 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:20:13.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:13 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:20:13.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:13 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:13.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:13 smithi162 ceph-mon[44304]: pgmap v147: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-02-09T17:20:13.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:13 smithi099 ceph-mon[39420]: osdmap e43: 6 total, 6 up, 6 in 2024-02-09T17:20:13.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:13 smithi099 ceph-mon[39420]: Detected new or changed devices on smithi162 2024-02-09T17:20:13.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:13 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:13.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:13 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:13.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:13 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:20:13.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:13 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:20:13.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:13 smithi099 ceph-mon[39420]: Adjusting osd_memory_target on smithi162 to 8561M 2024-02-09T17:20:13.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:13 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:13.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:13 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:20:13.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:13 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:20:13.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:13 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:13.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:13 smithi099 ceph-mon[39420]: pgmap v147: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-02-09T17:20:14.120 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:14 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:20:14.120 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:14 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:20:14.120 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:14 smithi162 ceph-mon[44304]: osdmap e44: 6 total, 6 up, 6 in 2024-02-09T17:20:14.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:14 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:20:14.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:14 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:20:14.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:14 smithi179 ceph-mon[43975]: osdmap e44: 6 total, 6 up, 6 in 2024-02-09T17:20:14.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:14 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:20:14.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:14 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:20:14.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:14 smithi099 ceph-mon[39420]: osdmap e44: 6 total, 6 up, 6 in 2024-02-09T17:20:15.337 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:15 smithi162 ceph-mon[44304]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 121 KiB/s, 0 objects/s recovering 2024-02-09T17:20:15.337 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:15 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T17:20:15.337 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:15 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T17:20:15.337 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:15 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:20:15.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:15 smithi179 ceph-mon[43975]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 121 KiB/s, 0 objects/s recovering 2024-02-09T17:20:15.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:15 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T17:20:15.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:15 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T17:20:15.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:15 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:20:15.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:15 smithi099 ceph-mon[39420]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 121 KiB/s, 0 objects/s recovering 2024-02-09T17:20:15.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:15 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T17:20:15.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:15 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T17:20:15.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:15 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:20:16.295 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:16 smithi162 ceph-mon[44304]: from='client.24242 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:20:16.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:16 smithi179 ceph-mon[43975]: from='client.24242 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:20:16.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:16 smithi099 ceph-mon[39420]: from='client.24242 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:20:17.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:17 smithi179 ceph-mon[43975]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-02-09T17:20:17.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:17 smithi162 ceph-mon[44304]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-02-09T17:20:17.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:17 smithi099 ceph-mon[39420]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-02-09T17:20:18.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:18 smithi179 ceph-mon[43975]: from='client.? 172.21.15.162:0/4067514622' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff50382d-5ddf-4cda-bb0c-ff055feeaa40"}]: dispatch 2024-02-09T17:20:18.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:18 smithi179 ceph-mon[43975]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff50382d-5ddf-4cda-bb0c-ff055feeaa40"}]: dispatch 2024-02-09T17:20:18.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:18 smithi179 ceph-mon[43975]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ff50382d-5ddf-4cda-bb0c-ff055feeaa40"}]': finished 2024-02-09T17:20:18.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:18 smithi179 ceph-mon[43975]: osdmap e45: 7 total, 6 up, 7 in 2024-02-09T17:20:18.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:18 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-09T17:20:18.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:18 smithi179 ceph-mon[43975]: from='client.? 172.21.15.162:0/2475395820' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T17:20:18.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:18 smithi162 ceph-mon[44304]: from='client.? 172.21.15.162:0/4067514622' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff50382d-5ddf-4cda-bb0c-ff055feeaa40"}]: dispatch 2024-02-09T17:20:18.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:18 smithi162 ceph-mon[44304]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff50382d-5ddf-4cda-bb0c-ff055feeaa40"}]: dispatch 2024-02-09T17:20:18.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:18 smithi162 ceph-mon[44304]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ff50382d-5ddf-4cda-bb0c-ff055feeaa40"}]': finished 2024-02-09T17:20:18.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:18 smithi162 ceph-mon[44304]: osdmap e45: 7 total, 6 up, 7 in 2024-02-09T17:20:18.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:18 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-09T17:20:18.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:18 smithi162 ceph-mon[44304]: from='client.? 172.21.15.162:0/2475395820' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T17:20:18.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:18 smithi099 ceph-mon[39420]: from='client.? 172.21.15.162:0/4067514622' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff50382d-5ddf-4cda-bb0c-ff055feeaa40"}]: dispatch 2024-02-09T17:20:18.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:18 smithi099 ceph-mon[39420]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff50382d-5ddf-4cda-bb0c-ff055feeaa40"}]: dispatch 2024-02-09T17:20:18.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:18 smithi099 ceph-mon[39420]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ff50382d-5ddf-4cda-bb0c-ff055feeaa40"}]': finished 2024-02-09T17:20:18.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:18 smithi099 ceph-mon[39420]: osdmap e45: 7 total, 6 up, 7 in 2024-02-09T17:20:18.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:18 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-09T17:20:18.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:18 smithi099 ceph-mon[39420]: from='client.? 172.21.15.162:0/2475395820' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T17:20:19.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:19 smithi179 ceph-mon[43975]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-02-09T17:20:19.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:19 smithi162 ceph-mon[44304]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-02-09T17:20:19.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:19 smithi099 ceph-mon[39420]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-02-09T17:20:21.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:21 smithi179 ceph-mon[43975]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-09T17:20:21.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:21 smithi162 ceph-mon[44304]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-09T17:20:21.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:21 smithi099 ceph-mon[39420]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-09T17:20:23.529 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:23 smithi162 ceph-mon[44304]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-02-09T17:20:23.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:23 smithi179 ceph-mon[43975]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-02-09T17:20:23.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:23 smithi099 ceph-mon[39420]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-02-09T17:20:24.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:24 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-09T17:20:24.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:24 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:20:24.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:24 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-09T17:20:24.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:24 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:20:24.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:24 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-09T17:20:24.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:24 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:20:25.628 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:25 smithi162 ceph-mon[44304]: Deploying daemon osd.6 on smithi162 2024-02-09T17:20:25.628 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:25 smithi162 ceph-mon[44304]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-02-09T17:20:25.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:25 smithi179 ceph-mon[43975]: Deploying daemon osd.6 on smithi162 2024-02-09T17:20:25.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:25 smithi179 ceph-mon[43975]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-02-09T17:20:25.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:25 smithi099 ceph-mon[39420]: Deploying daemon osd.6 on smithi162 2024-02-09T17:20:25.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:25 smithi099 ceph-mon[39420]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-02-09T17:20:27.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:27 smithi179 ceph-mon[43975]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-02-09T17:20:27.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:27 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:20:27.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:27 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:20:27.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:27 smithi162 ceph-mon[44304]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-02-09T17:20:27.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:27 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:20:27.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:27 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:20:27.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:27 smithi099 ceph-mon[39420]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-02-09T17:20:27.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:27 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:20:27.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:27 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:20:28.615 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:28 smithi162 ceph-mon[44304]: osdmap e46: 7 total, 6 up, 7 in 2024-02-09T17:20:28.615 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:28 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-09T17:20:28.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:28 smithi179 ceph-mon[43975]: osdmap e46: 7 total, 6 up, 7 in 2024-02-09T17:20:28.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:28 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-09T17:20:28.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:28 smithi099 ceph-mon[39420]: osdmap e46: 7 total, 6 up, 7 in 2024-02-09T17:20:28.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:28 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-09T17:20:29.628 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:29 smithi162 ceph-mon[44304]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-02-09T17:20:29.628 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:29 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:29.628 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:29 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:29.628 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:29 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:20:29.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:29 smithi179 ceph-mon[43975]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-02-09T17:20:29.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:29 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:29.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:29 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:29.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:29 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:20:29.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:29 smithi099 ceph-mon[39420]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-02-09T17:20:29.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:29 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:29.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:29 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:29.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:29 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:20:31.574 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:31 smithi162 ceph-mon[44304]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-02-09T17:20:31.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:31 smithi179 ceph-mon[43975]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-02-09T17:20:31.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:31 smithi099 ceph-mon[39420]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-02-09T17:20:31.937 INFO:teuthology.orchestra.run.smithi162.stdout:Created osd(s) 6 on host 'smithi162' 2024-02-09T17:20:32.598 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:32 smithi162 ceph-mon[44304]: from='osd.6 [v2:172.21.15.162:6816/821169663,v1:172.21.15.162:6817/821169663]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-09T17:20:32.598 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:32 smithi162 ceph-mon[44304]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-09T17:20:32.598 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:32 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:32.598 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:32 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:32.598 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:32 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:32.598 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:32 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:32.624 DEBUG:teuthology.orchestra.run.smithi162:osd.6> sudo journalctl -f -n 0 -u ceph-a69c870a-c76e-11ee-95b7-87774f69a715@osd.6.service 2024-02-09T17:20:32.626 INFO:tasks.cephadm:Deploying osd.7 on smithi162 with /dev/vg_nvme/lv_1... 2024-02-09T17:20:32.627 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-09T17:20:32.655 INFO:journalctl@ceph.osd.6.smithi162.stdout:-- Logs begin at Fri 2024-02-09 17:04:37 UTC. -- 2024-02-09T17:20:32.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:32 smithi179 ceph-mon[43975]: from='osd.6 [v2:172.21.15.162:6816/821169663,v1:172.21.15.162:6817/821169663]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-09T17:20:32.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:32 smithi179 ceph-mon[43975]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-09T17:20:32.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:32 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:32.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:32 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:32.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:32 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:32.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:32 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:32.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:32 smithi099 ceph-mon[39420]: from='osd.6 [v2:172.21.15.162:6816/821169663,v1:172.21.15.162:6817/821169663]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-09T17:20:32.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:32 smithi099 ceph-mon[39420]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-09T17:20:32.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:32 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:32.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:32 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:32.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:32 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:32.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:32 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:32.921 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:20:33.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:33 smithi179 ceph-mon[43975]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-02-09T17:20:33.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:33 smithi179 ceph-mon[43975]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-09T17:20:33.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:33 smithi179 ceph-mon[43975]: osdmap e47: 7 total, 6 up, 7 in 2024-02-09T17:20:33.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:33 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-09T17:20:33.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:33 smithi179 ceph-mon[43975]: from='osd.6 [v2:172.21.15.162:6816/821169663,v1:172.21.15.162:6817/821169663]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-02-09T17:20:33.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:33 smithi179 ceph-mon[43975]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-02-09T17:20:33.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:33 smithi162 ceph-mon[44304]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-02-09T17:20:33.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:33 smithi162 ceph-mon[44304]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-09T17:20:33.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:33 smithi162 ceph-mon[44304]: osdmap e47: 7 total, 6 up, 7 in 2024-02-09T17:20:33.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:33 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-09T17:20:33.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:33 smithi162 ceph-mon[44304]: from='osd.6 [v2:172.21.15.162:6816/821169663,v1:172.21.15.162:6817/821169663]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-02-09T17:20:33.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:33 smithi162 ceph-mon[44304]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-02-09T17:20:33.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:33 smithi099 ceph-mon[39420]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-02-09T17:20:33.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:33 smithi099 ceph-mon[39420]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-09T17:20:33.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:33 smithi099 ceph-mon[39420]: osdmap e47: 7 total, 6 up, 7 in 2024-02-09T17:20:33.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:33 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-09T17:20:33.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:33 smithi099 ceph-mon[39420]: from='osd.6 [v2:172.21.15.162:6816/821169663,v1:172.21.15.162:6817/821169663]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-02-09T17:20:33.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:33 smithi099 ceph-mon[39420]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-02-09T17:20:34.161 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 17:20:33 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T17:20:33.815+0000 7f20302a1700 -1 osd.6 0 waiting for initial osdmap 2024-02-09T17:20:34.161 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 17:20:33 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T17:20:33.821+0000 7f2027892700 -1 osd.6 48 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-09T17:20:34.414 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:34 smithi162 ceph-mon[44304]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-02-09T17:20:34.414 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:34 smithi162 ceph-mon[44304]: osdmap e48: 7 total, 6 up, 7 in 2024-02-09T17:20:34.414 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:34 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-09T17:20:34.414 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:34 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-09T17:20:34.414 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:34 smithi162 ceph-mon[44304]: from='osd.6 ' entity='osd.6' 2024-02-09T17:20:34.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:34 smithi179 ceph-mon[43975]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-02-09T17:20:34.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:34 smithi179 ceph-mon[43975]: osdmap e48: 7 total, 6 up, 7 in 2024-02-09T17:20:34.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:34 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-09T17:20:34.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:34 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-09T17:20:34.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:34 smithi179 ceph-mon[43975]: from='osd.6 ' entity='osd.6' 2024-02-09T17:20:34.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:34 smithi099 ceph-mon[39420]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-02-09T17:20:34.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:34 smithi099 ceph-mon[39420]: osdmap e48: 7 total, 6 up, 7 in 2024-02-09T17:20:34.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:34 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-09T17:20:34.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:34 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-09T17:20:34.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:34 smithi099 ceph-mon[39420]: from='osd.6 ' entity='osd.6' 2024-02-09T17:20:35.617 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:35 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:20:35.617 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:35 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:20:35.617 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:35 smithi162 ceph-mon[44304]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-02-09T17:20:35.617 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:35 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-09T17:20:35.618 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:35 smithi162 ceph-mon[44304]: osd.6 [v2:172.21.15.162:6816/821169663,v1:172.21.15.162:6817/821169663] boot 2024-02-09T17:20:35.618 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:35 smithi162 ceph-mon[44304]: osdmap e49: 7 total, 7 up, 7 in 2024-02-09T17:20:35.618 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:35 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-09T17:20:35.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:35 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:20:35.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:35 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:20:35.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:35 smithi179 ceph-mon[43975]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-02-09T17:20:35.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:35 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-09T17:20:35.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:35 smithi179 ceph-mon[43975]: osd.6 [v2:172.21.15.162:6816/821169663,v1:172.21.15.162:6817/821169663] boot 2024-02-09T17:20:35.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:35 smithi179 ceph-mon[43975]: osdmap e49: 7 total, 7 up, 7 in 2024-02-09T17:20:35.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:35 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-09T17:20:35.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:35 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:20:35.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:35 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:20:35.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:35 smithi099 ceph-mon[39420]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-02-09T17:20:35.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:35 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-09T17:20:35.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:35 smithi099 ceph-mon[39420]: osd.6 [v2:172.21.15.162:6816/821169663,v1:172.21.15.162:6817/821169663] boot 2024-02-09T17:20:35.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:35 smithi099 ceph-mon[39420]: osdmap e49: 7 total, 7 up, 7 in 2024-02-09T17:20:35.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:35 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-09T17:20:36.226 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-09T17:20:36.226 INFO:teuthology.orchestra.run.smithi162.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-09T17:20:36.226 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 10+0 records in 2024-02-09T17:20:36.226 INFO:teuthology.orchestra.run.smithi162.stderr:10+0 records out 2024-02-09T17:20:36.226 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110644 s, 948 MB/s 2024-02-09T17:20:36.226 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping successful for: 2024-02-09T17:20:36.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:36 smithi179 ceph-mon[43975]: osdmap e50: 7 total, 7 up, 7 in 2024-02-09T17:20:36.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:36 smithi162 ceph-mon[44304]: osdmap e50: 7 total, 7 up, 7 in 2024-02-09T17:20:36.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:36 smithi099 ceph-mon[39420]: osdmap e50: 7 total, 7 up, 7 in 2024-02-09T17:20:37.088 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph orch daemon add osd smithi162:vg_nvme/lv_1 2024-02-09T17:20:37.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:37 smithi179 ceph-mon[43975]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-02-09T17:20:37.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:37 smithi179 ceph-mon[43975]: osdmap e51: 7 total, 7 up, 7 in 2024-02-09T17:20:37.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:37 smithi162 ceph-mon[44304]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-02-09T17:20:37.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:37 smithi162 ceph-mon[44304]: osdmap e51: 7 total, 7 up, 7 in 2024-02-09T17:20:37.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:37 smithi099 ceph-mon[39420]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-02-09T17:20:37.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:37 smithi099 ceph-mon[39420]: osdmap e51: 7 total, 7 up, 7 in 2024-02-09T17:20:37.851 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:20:38.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:38 smithi179 ceph-mon[43975]: osdmap e52: 7 total, 7 up, 7 in 2024-02-09T17:20:38.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:38 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:38.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:38 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:38.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:38 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:20:38.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:38 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:20:38.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:38 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:20:38.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:38 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:38.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:38 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:20:38.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:38 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:20:38.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:38 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:38.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:38 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:20:38.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:38 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:20:38.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:38 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:20:38.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:38 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:38.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:38 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:20:38.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:38 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:20:38.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:38 smithi162 ceph-mon[44304]: osdmap e52: 7 total, 7 up, 7 in 2024-02-09T17:20:38.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:38 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:38.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:38 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:38.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:38 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:20:38.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:38 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:20:38.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:38 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:20:38.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:38 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:38.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:38 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:20:38.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:38 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:20:38.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:38 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:38.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:38 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:20:38.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:38 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:20:38.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:38 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:20:38.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:38 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:38.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:38 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:20:38.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:38 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:20:38.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:38 smithi099 ceph-mon[39420]: osdmap e52: 7 total, 7 up, 7 in 2024-02-09T17:20:38.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:38 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:38.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:38 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:38.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:38 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:20:38.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:38 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:20:38.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:38 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:20:38.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:38 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:38.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:38 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:20:38.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:38 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:20:38.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:38 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:38.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:38 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:20:38.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:38 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:20:38.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:38 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:20:38.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:38 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:38.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:38 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:20:38.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:38 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:20:39.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:39 smithi179 ceph-mon[43975]: Detected new or changed devices on smithi162 2024-02-09T17:20:39.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:39 smithi179 ceph-mon[43975]: Adjusting osd_memory_target on smithi162 to 5707M 2024-02-09T17:20:39.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:39 smithi179 ceph-mon[43975]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-02-09T17:20:39.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:39 smithi179 ceph-mon[43975]: osdmap e53: 7 total, 7 up, 7 in 2024-02-09T17:20:39.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:39 smithi162 ceph-mon[44304]: Detected new or changed devices on smithi162 2024-02-09T17:20:39.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:39 smithi162 ceph-mon[44304]: Adjusting osd_memory_target on smithi162 to 5707M 2024-02-09T17:20:39.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:39 smithi162 ceph-mon[44304]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-02-09T17:20:39.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:39 smithi162 ceph-mon[44304]: osdmap e53: 7 total, 7 up, 7 in 2024-02-09T17:20:39.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:39 smithi099 ceph-mon[39420]: Detected new or changed devices on smithi162 2024-02-09T17:20:39.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:39 smithi099 ceph-mon[39420]: Adjusting osd_memory_target on smithi162 to 5707M 2024-02-09T17:20:39.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:39 smithi099 ceph-mon[39420]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-02-09T17:20:39.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:39 smithi099 ceph-mon[39420]: osdmap e53: 7 total, 7 up, 7 in 2024-02-09T17:20:40.617 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:40 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T17:20:40.617 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:40 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T17:20:40.617 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:40 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:20:40.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:40 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T17:20:40.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:40 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T17:20:40.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:40 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:20:40.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:40 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T17:20:40.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:40 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T17:20:40.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:40 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:20:41.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:41 smithi179 ceph-mon[43975]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:20:41.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:41 smithi179 ceph-mon[43975]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-02-09T17:20:41.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:41 smithi162 ceph-mon[44304]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:20:41.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:41 smithi162 ceph-mon[44304]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-02-09T17:20:41.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:41 smithi099 ceph-mon[39420]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:20:41.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:41 smithi099 ceph-mon[39420]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-02-09T17:20:42.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:42 smithi179 ceph-mon[43975]: from='client.? 172.21.15.162:0/4247703272' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a82984dc-5e22-424b-a446-ae2ee71eee19"}]: dispatch 2024-02-09T17:20:42.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:42 smithi179 ceph-mon[43975]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a82984dc-5e22-424b-a446-ae2ee71eee19"}]: dispatch 2024-02-09T17:20:42.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:42 smithi179 ceph-mon[43975]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a82984dc-5e22-424b-a446-ae2ee71eee19"}]': finished 2024-02-09T17:20:42.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:42 smithi179 ceph-mon[43975]: osdmap e54: 8 total, 7 up, 8 in 2024-02-09T17:20:42.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:42 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-09T17:20:42.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:42 smithi162 ceph-mon[44304]: from='client.? 172.21.15.162:0/4247703272' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a82984dc-5e22-424b-a446-ae2ee71eee19"}]: dispatch 2024-02-09T17:20:42.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:42 smithi162 ceph-mon[44304]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a82984dc-5e22-424b-a446-ae2ee71eee19"}]: dispatch 2024-02-09T17:20:42.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:42 smithi162 ceph-mon[44304]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a82984dc-5e22-424b-a446-ae2ee71eee19"}]': finished 2024-02-09T17:20:42.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:42 smithi162 ceph-mon[44304]: osdmap e54: 8 total, 7 up, 8 in 2024-02-09T17:20:42.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:42 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-09T17:20:42.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:42 smithi099 ceph-mon[39420]: from='client.? 172.21.15.162:0/4247703272' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a82984dc-5e22-424b-a446-ae2ee71eee19"}]: dispatch 2024-02-09T17:20:42.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:42 smithi099 ceph-mon[39420]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a82984dc-5e22-424b-a446-ae2ee71eee19"}]: dispatch 2024-02-09T17:20:42.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:42 smithi099 ceph-mon[39420]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a82984dc-5e22-424b-a446-ae2ee71eee19"}]': finished 2024-02-09T17:20:42.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:42 smithi099 ceph-mon[39420]: osdmap e54: 8 total, 7 up, 8 in 2024-02-09T17:20:42.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:42 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-09T17:20:43.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:43 smithi179 ceph-mon[43975]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-02-09T17:20:43.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:43 smithi179 ceph-mon[43975]: from='client.? 172.21.15.162:0/4024077959' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T17:20:43.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:43 smithi162 ceph-mon[44304]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-02-09T17:20:43.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:43 smithi162 ceph-mon[44304]: from='client.? 172.21.15.162:0/4024077959' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T17:20:43.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:43 smithi099 ceph-mon[39420]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-02-09T17:20:43.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:43 smithi099 ceph-mon[39420]: from='client.? 172.21.15.162:0/4024077959' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T17:20:45.624 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:45 smithi162 ceph-mon[44304]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-02-09T17:20:45.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:45 smithi179 ceph-mon[43975]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-02-09T17:20:45.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:45 smithi099 ceph-mon[39420]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-02-09T17:20:47.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:47 smithi179 ceph-mon[43975]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-09T17:20:47.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:47 smithi162 ceph-mon[44304]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-09T17:20:47.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:47 smithi099 ceph-mon[39420]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-09T17:20:49.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:49 smithi179 ceph-mon[43975]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-09T17:20:49.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:49 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-09T17:20:49.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:49 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:20:49.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:49 smithi162 ceph-mon[44304]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-09T17:20:49.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:49 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-09T17:20:49.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:49 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:20:49.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:49 smithi099 ceph-mon[39420]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-09T17:20:49.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:49 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-09T17:20:49.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:49 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:20:50.541 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:50 smithi162 ceph-mon[44304]: Deploying daemon osd.7 on smithi162 2024-02-09T17:20:50.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:50 smithi099 ceph-mon[39420]: Deploying daemon osd.7 on smithi162 2024-02-09T17:20:50.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:50 smithi179 ceph-mon[43975]: Deploying daemon osd.7 on smithi162 2024-02-09T17:20:51.627 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:51 smithi162 ceph-mon[44304]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-09T17:20:51.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:51 smithi099 ceph-mon[39420]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-09T17:20:51.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:51 smithi179 ceph-mon[43975]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-09T17:20:53.653 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:53 smithi162 ceph-mon[44304]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-02-09T17:20:53.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:53 smithi099 ceph-mon[39420]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-02-09T17:20:53.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:53 smithi179 ceph-mon[43975]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-02-09T17:20:55.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:55 smithi099 ceph-mon[39420]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-02-09T17:20:55.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:55 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:55.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:55 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:55.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:55 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:20:55.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:55 smithi179 ceph-mon[43975]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-02-09T17:20:55.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:55 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:55.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:55 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:55.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:55 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:20:55.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:55 smithi162 ceph-mon[44304]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-02-09T17:20:55.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:55 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:55.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:55 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:55.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:55 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:20:57.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:56 smithi099 ceph-mon[39420]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-02-09T17:20:57.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:56 smithi179 ceph-mon[43975]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-02-09T17:20:57.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:56 smithi162 ceph-mon[44304]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-02-09T17:20:58.128 INFO:teuthology.orchestra.run.smithi162.stdout:Created osd(s) 7 on host 'smithi162' 2024-02-09T17:20:58.380 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:58 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:58.380 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:58 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:58.380 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:58 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:20:58.380 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:58 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:20:58.380 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:58 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:58.380 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:58 smithi162 ceph-mon[44304]: from='osd.7 [v2:172.21.15.162:6824/965264369,v1:172.21.15.162:6825/965264369]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-09T17:20:58.380 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:58 smithi162 ceph-mon[44304]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-09T17:20:58.381 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:58 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:58.381 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:58 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:20:58.381 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:58 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:58.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:58 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:58.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:58 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:58.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:58 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:20:58.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:58 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:20:58.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:58 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:58.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:58 smithi179 ceph-mon[43975]: from='osd.7 [v2:172.21.15.162:6824/965264369,v1:172.21.15.162:6825/965264369]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-09T17:20:58.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:58 smithi179 ceph-mon[43975]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-09T17:20:58.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:58 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:58.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:58 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:20:58.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:58 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:58.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:58 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:58.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:58 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:58.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:58 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:20:58.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:58 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:20:58.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:58 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:58.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:58 smithi099 ceph-mon[39420]: from='osd.7 [v2:172.21.15.162:6824/965264369,v1:172.21.15.162:6825/965264369]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-09T17:20:58.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:58 smithi099 ceph-mon[39420]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-09T17:20:58.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:58 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:58.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:58 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:20:58.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:58 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:20:58.865 DEBUG:teuthology.orchestra.run.smithi162:osd.7> sudo journalctl -f -n 0 -u ceph-a69c870a-c76e-11ee-95b7-87774f69a715@osd.7.service 2024-02-09T17:20:58.867 INFO:tasks.cephadm:Deploying osd.8 on smithi179 with /dev/vg_nvme/lv_4... 2024-02-09T17:20:58.867 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-09T17:20:58.900 INFO:journalctl@ceph.osd.7.smithi162.stdout:-- Logs begin at Fri 2024-02-09 17:04:37 UTC. -- 2024-02-09T17:20:59.133 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:20:59.664 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 17:20:59 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T17:20:59.394+0000 7f570a6d1700 -1 osd.7 0 waiting for initial osdmap 2024-02-09T17:20:59.664 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 17:20:59 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T17:20:59.403+0000 7f57064cb700 -1 osd.7 56 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-09T17:20:59.665 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:59 smithi162 ceph-mon[44304]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-02-09T17:20:59.665 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:59 smithi162 ceph-mon[44304]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-09T17:20:59.665 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:59 smithi162 ceph-mon[44304]: osdmap e55: 8 total, 7 up, 8 in 2024-02-09T17:20:59.665 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:59 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-09T17:20:59.665 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:59 smithi162 ceph-mon[44304]: from='osd.7 [v2:172.21.15.162:6824/965264369,v1:172.21.15.162:6825/965264369]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-02-09T17:20:59.666 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:20:59 smithi162 ceph-mon[44304]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-02-09T17:20:59.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:59 smithi179 ceph-mon[43975]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-02-09T17:20:59.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:59 smithi179 ceph-mon[43975]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-09T17:20:59.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:59 smithi179 ceph-mon[43975]: osdmap e55: 8 total, 7 up, 8 in 2024-02-09T17:20:59.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:59 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-09T17:20:59.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:59 smithi179 ceph-mon[43975]: from='osd.7 [v2:172.21.15.162:6824/965264369,v1:172.21.15.162:6825/965264369]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-02-09T17:20:59.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:20:59 smithi179 ceph-mon[43975]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-02-09T17:20:59.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:59 smithi099 ceph-mon[39420]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-02-09T17:20:59.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:59 smithi099 ceph-mon[39420]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-09T17:20:59.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:59 smithi099 ceph-mon[39420]: osdmap e55: 8 total, 7 up, 8 in 2024-02-09T17:20:59.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:59 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-09T17:20:59.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:59 smithi099 ceph-mon[39420]: from='osd.7 [v2:172.21.15.162:6824/965264369,v1:172.21.15.162:6825/965264369]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-02-09T17:20:59.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:20:59 smithi099 ceph-mon[39420]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-02-09T17:21:00.619 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:00 smithi179 ceph-mon[43975]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-02-09T17:21:00.620 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:00 smithi179 ceph-mon[43975]: osdmap e56: 8 total, 7 up, 8 in 2024-02-09T17:21:00.620 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:00 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-09T17:21:00.620 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:00 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-09T17:21:00.620 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:00 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-09T17:21:00.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:00 smithi162 ceph-mon[44304]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-02-09T17:21:00.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:00 smithi162 ceph-mon[44304]: osdmap e56: 8 total, 7 up, 8 in 2024-02-09T17:21:00.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:00 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-09T17:21:00.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:00 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-09T17:21:00.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:00 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-09T17:21:00.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:00 smithi099 ceph-mon[39420]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-02-09T17:21:00.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:00 smithi099 ceph-mon[39420]: osdmap e56: 8 total, 7 up, 8 in 2024-02-09T17:21:00.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:00 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-09T17:21:00.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:00 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-09T17:21:00.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:00 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-09T17:21:01.216 INFO:teuthology.orchestra.run.smithi179.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-09T17:21:01.216 INFO:teuthology.orchestra.run.smithi179.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-09T17:21:01.216 INFO:teuthology.orchestra.run.smithi179.stderr: stderr: 10+0 records in 2024-02-09T17:21:01.216 INFO:teuthology.orchestra.run.smithi179.stderr:10+0 records out 2024-02-09T17:21:01.216 INFO:teuthology.orchestra.run.smithi179.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0111287 s, 942 MB/s 2024-02-09T17:21:01.217 INFO:teuthology.orchestra.run.smithi179.stderr:--> Zapping successful for: 2024-02-09T17:21:01.504 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:01 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:21:01.504 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:01 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:21:01.504 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:01 smithi162 ceph-mon[44304]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-02-09T17:21:01.504 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:01 smithi162 ceph-mon[44304]: osd.7 [v2:172.21.15.162:6824/965264369,v1:172.21.15.162:6825/965264369] boot 2024-02-09T17:21:01.504 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:01 smithi162 ceph-mon[44304]: osdmap e57: 8 total, 8 up, 8 in 2024-02-09T17:21:01.504 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:01 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-09T17:21:01.505 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:01 smithi162 ceph-mon[44304]: osdmap e58: 8 total, 8 up, 8 in 2024-02-09T17:21:01.537 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:01 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:21:01.537 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:01 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:21:01.537 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:01 smithi179 ceph-mon[43975]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-02-09T17:21:01.537 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:01 smithi179 ceph-mon[43975]: osd.7 [v2:172.21.15.162:6824/965264369,v1:172.21.15.162:6825/965264369] boot 2024-02-09T17:21:01.537 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:01 smithi179 ceph-mon[43975]: osdmap e57: 8 total, 8 up, 8 in 2024-02-09T17:21:01.537 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:01 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-09T17:21:01.537 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:01 smithi179 ceph-mon[43975]: osdmap e58: 8 total, 8 up, 8 in 2024-02-09T17:21:01.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:01 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:21:01.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:01 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:21:01.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:01 smithi099 ceph-mon[39420]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-02-09T17:21:01.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:01 smithi099 ceph-mon[39420]: osd.7 [v2:172.21.15.162:6824/965264369,v1:172.21.15.162:6825/965264369] boot 2024-02-09T17:21:01.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:01 smithi099 ceph-mon[39420]: osdmap e57: 8 total, 8 up, 8 in 2024-02-09T17:21:01.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:01 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-09T17:21:01.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:01 smithi099 ceph-mon[39420]: osdmap e58: 8 total, 8 up, 8 in 2024-02-09T17:21:01.818 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph orch daemon add osd smithi179:vg_nvme/lv_4 2024-02-09T17:21:02.080 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:21:03.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:02 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:21:03.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:02 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:21:03.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:02 smithi179 ceph-mon[43975]: Detected new or changed devices on smithi162 2024-02-09T17:21:03.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:02 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:03.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:02 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:03.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:02 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:21:03.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:02 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:21:03.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:02 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:21:03.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:02 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:21:03.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:02 smithi179 ceph-mon[43975]: Adjusting osd_memory_target on smithi162 to 4280M 2024-02-09T17:21:03.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:02 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:03.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:02 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:21:03.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:02 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:21:03.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:02 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:03.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:02 smithi179 ceph-mon[43975]: osdmap e59: 8 total, 8 up, 8 in 2024-02-09T17:21:03.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:02 smithi179 ceph-mon[43975]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-02-09T17:21:03.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:02 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:21:03.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:02 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:21:03.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:02 smithi162 ceph-mon[44304]: Detected new or changed devices on smithi162 2024-02-09T17:21:03.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:02 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:03.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:02 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:03.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:02 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:21:03.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:02 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:21:03.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:02 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:21:03.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:02 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:21:03.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:02 smithi162 ceph-mon[44304]: Adjusting osd_memory_target on smithi162 to 4280M 2024-02-09T17:21:03.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:02 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:03.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:02 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:21:03.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:02 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:21:03.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:02 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:03.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:02 smithi162 ceph-mon[44304]: osdmap e59: 8 total, 8 up, 8 in 2024-02-09T17:21:03.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:02 smithi162 ceph-mon[44304]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-02-09T17:21:03.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:02 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:21:03.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:02 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:21:03.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:02 smithi099 ceph-mon[39420]: Detected new or changed devices on smithi162 2024-02-09T17:21:03.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:02 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:03.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:02 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:03.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:02 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:21:03.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:02 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:21:03.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:02 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:21:03.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:02 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:21:03.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:02 smithi099 ceph-mon[39420]: Adjusting osd_memory_target on smithi162 to 4280M 2024-02-09T17:21:03.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:02 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:03.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:02 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:21:03.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:02 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:21:03.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:02 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:03.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:02 smithi099 ceph-mon[39420]: osdmap e59: 8 total, 8 up, 8 in 2024-02-09T17:21:03.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:02 smithi099 ceph-mon[39420]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-02-09T17:21:04.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:03 smithi099 ceph-mon[39420]: osdmap e60: 8 total, 8 up, 8 in 2024-02-09T17:21:04.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:03 smithi179 ceph-mon[43975]: osdmap e60: 8 total, 8 up, 8 in 2024-02-09T17:21:04.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:03 smithi162 ceph-mon[44304]: osdmap e60: 8 total, 8 up, 8 in 2024-02-09T17:21:05.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:04 smithi099 ceph-mon[39420]: osdmap e61: 8 total, 8 up, 8 in 2024-02-09T17:21:05.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:04 smithi099 ceph-mon[39420]: from='client.14418 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:21:05.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:04 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T17:21:05.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:04 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T17:21:05.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:04 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:21:05.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:04 smithi099 ceph-mon[39420]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-02-09T17:21:05.349 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:04 smithi179 ceph-mon[43975]: osdmap e61: 8 total, 8 up, 8 in 2024-02-09T17:21:05.349 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:04 smithi179 ceph-mon[43975]: from='client.14418 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:21:05.349 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:04 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T17:21:05.350 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:04 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T17:21:05.350 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:04 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:21:05.350 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:04 smithi179 ceph-mon[43975]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-02-09T17:21:05.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:04 smithi162 ceph-mon[44304]: osdmap e61: 8 total, 8 up, 8 in 2024-02-09T17:21:05.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:04 smithi162 ceph-mon[44304]: from='client.14418 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:21:05.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:04 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T17:21:05.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:04 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T17:21:05.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:04 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:21:05.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:04 smithi162 ceph-mon[44304]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-02-09T17:21:06.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:05 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:21:06.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:05 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:21:06.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:05 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:21:06.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:05 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:21:06.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:05 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:21:06.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:05 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:21:07.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:06 smithi179 ceph-mon[43975]: osdmap e62: 8 total, 8 up, 8 in 2024-02-09T17:21:07.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:06 smithi179 ceph-mon[43975]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-02-09T17:21:07.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:06 smithi179 ceph-mon[43975]: from='client.? 172.21.15.179:0/2004002839' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63f9d79f-0400-44dc-b388-df00fefb5820"}]: dispatch 2024-02-09T17:21:07.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:06 smithi179 ceph-mon[43975]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63f9d79f-0400-44dc-b388-df00fefb5820"}]: dispatch 2024-02-09T17:21:07.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:06 smithi179 ceph-mon[43975]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "63f9d79f-0400-44dc-b388-df00fefb5820"}]': finished 2024-02-09T17:21:07.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:06 smithi179 ceph-mon[43975]: osdmap e63: 9 total, 8 up, 9 in 2024-02-09T17:21:07.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:06 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-09T17:21:07.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:06 smithi162 ceph-mon[44304]: osdmap e62: 8 total, 8 up, 8 in 2024-02-09T17:21:07.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:06 smithi162 ceph-mon[44304]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-02-09T17:21:07.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:06 smithi162 ceph-mon[44304]: from='client.? 172.21.15.179:0/2004002839' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63f9d79f-0400-44dc-b388-df00fefb5820"}]: dispatch 2024-02-09T17:21:07.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:06 smithi162 ceph-mon[44304]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63f9d79f-0400-44dc-b388-df00fefb5820"}]: dispatch 2024-02-09T17:21:07.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:06 smithi162 ceph-mon[44304]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "63f9d79f-0400-44dc-b388-df00fefb5820"}]': finished 2024-02-09T17:21:07.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:06 smithi162 ceph-mon[44304]: osdmap e63: 9 total, 8 up, 9 in 2024-02-09T17:21:07.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:06 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-09T17:21:07.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:06 smithi099 ceph-mon[39420]: osdmap e62: 8 total, 8 up, 8 in 2024-02-09T17:21:07.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:06 smithi099 ceph-mon[39420]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-02-09T17:21:07.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:06 smithi099 ceph-mon[39420]: from='client.? 172.21.15.179:0/2004002839' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63f9d79f-0400-44dc-b388-df00fefb5820"}]: dispatch 2024-02-09T17:21:07.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:06 smithi099 ceph-mon[39420]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63f9d79f-0400-44dc-b388-df00fefb5820"}]: dispatch 2024-02-09T17:21:07.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:06 smithi099 ceph-mon[39420]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "63f9d79f-0400-44dc-b388-df00fefb5820"}]': finished 2024-02-09T17:21:07.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:06 smithi099 ceph-mon[39420]: osdmap e63: 9 total, 8 up, 9 in 2024-02-09T17:21:07.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:06 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-09T17:21:08.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:07 smithi179 ceph-mon[43975]: 1.0 deep-scrub starts 2024-02-09T17:21:08.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:07 smithi179 ceph-mon[43975]: 1.0 deep-scrub ok 2024-02-09T17:21:08.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:07 smithi179 ceph-mon[43975]: from='client.? 172.21.15.179:0/1651980341' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T17:21:08.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:07 smithi162 ceph-mon[44304]: 1.0 deep-scrub starts 2024-02-09T17:21:08.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:07 smithi162 ceph-mon[44304]: 1.0 deep-scrub ok 2024-02-09T17:21:08.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:07 smithi162 ceph-mon[44304]: from='client.? 172.21.15.179:0/1651980341' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T17:21:08.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:07 smithi099 ceph-mon[39420]: 1.0 deep-scrub starts 2024-02-09T17:21:08.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:07 smithi099 ceph-mon[39420]: 1.0 deep-scrub ok 2024-02-09T17:21:08.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:07 smithi099 ceph-mon[39420]: from='client.? 172.21.15.179:0/1651980341' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T17:21:09.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:09 smithi179 ceph-mon[43975]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-02-09T17:21:09.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:09 smithi162 ceph-mon[44304]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-02-09T17:21:09.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:09 smithi099 ceph-mon[39420]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-02-09T17:21:11.539 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:11 smithi179 ceph-mon[43975]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-02-09T17:21:11.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:11 smithi162 ceph-mon[44304]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-02-09T17:21:11.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:11 smithi099 ceph-mon[39420]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-02-09T17:21:13.647 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:13 smithi179 ceph-mon[43975]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-09T17:21:13.648 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:13 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-02-09T17:21:13.648 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:13 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:21:13.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:13 smithi162 ceph-mon[44304]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-09T17:21:13.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:13 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-02-09T17:21:13.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:13 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:21:13.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:13 smithi099 ceph-mon[39420]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-09T17:21:13.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:13 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-02-09T17:21:13.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:13 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:21:14.616 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:14 smithi179 ceph-mon[43975]: Deploying daemon osd.8 on smithi179 2024-02-09T17:21:14.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:14 smithi162 ceph-mon[44304]: Deploying daemon osd.8 on smithi179 2024-02-09T17:21:14.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:14 smithi099 ceph-mon[39420]: Deploying daemon osd.8 on smithi179 2024-02-09T17:21:15.609 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:15 smithi179 ceph-mon[43975]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-02-09T17:21:15.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:15 smithi162 ceph-mon[44304]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-02-09T17:21:15.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:15 smithi099 ceph-mon[39420]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-02-09T17:21:17.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:16 smithi099 ceph-mon[39420]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-09T17:21:17.302 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:16 smithi179 ceph-mon[43975]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-09T17:21:17.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:16 smithi162 ceph-mon[44304]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-09T17:21:19.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:19 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:19.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:19 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:21:19.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:19 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:19.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:19 smithi179 ceph-mon[43975]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-02-09T17:21:19.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:19 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:19.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:19 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:21:19.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:19 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:19.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:19 smithi162 ceph-mon[44304]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-02-09T17:21:19.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:19 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:19.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:19 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:21:19.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:19 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:19.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:19 smithi099 ceph-mon[39420]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-02-09T17:21:21.627 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:21 smithi179 ceph-mon[43975]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-02-09T17:21:21.627 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:21 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:21.627 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:21 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:21.627 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:21 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:21:21.627 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:21 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:21:21.627 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:21 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:21.627 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:21 smithi179 ceph-mon[43975]: from='osd.8 [v2:172.21.15.179:6800/4219839127,v1:172.21.15.179:6801/4219839127]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-02-09T17:21:21.627 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:21 smithi179 ceph-mon[43975]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-02-09T17:21:21.696 INFO:teuthology.orchestra.run.smithi179.stdout:Created osd(s) 8 on host 'smithi179' 2024-02-09T17:21:21.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:21 smithi099 ceph-mon[39420]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-02-09T17:21:21.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:21 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:21.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:21 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:21.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:21 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:21:21.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:21 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:21:21.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:21 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:21.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:21 smithi099 ceph-mon[39420]: from='osd.8 [v2:172.21.15.179:6800/4219839127,v1:172.21.15.179:6801/4219839127]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-02-09T17:21:21.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:21 smithi099 ceph-mon[39420]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-02-09T17:21:21.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:21 smithi162 ceph-mon[44304]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-02-09T17:21:21.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:21.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:21.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:21:21.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:21:21.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:21.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:21 smithi162 ceph-mon[44304]: from='osd.8 [v2:172.21.15.179:6800/4219839127,v1:172.21.15.179:6801/4219839127]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-02-09T17:21:21.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:21 smithi162 ceph-mon[44304]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-02-09T17:21:22.423 DEBUG:teuthology.orchestra.run.smithi179:osd.8> sudo journalctl -f -n 0 -u ceph-a69c870a-c76e-11ee-95b7-87774f69a715@osd.8.service 2024-02-09T17:21:22.425 INFO:tasks.cephadm:Deploying osd.9 on smithi179 with /dev/vg_nvme/lv_3... 2024-02-09T17:21:22.425 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-09T17:21:22.454 INFO:journalctl@ceph.osd.8.smithi179.stdout:-- Logs begin at Fri 2024-02-09 17:04:31 UTC. -- 2024-02-09T17:21:22.653 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:22 smithi179 ceph-mon[43975]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-02-09T17:21:22.654 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:22 smithi179 ceph-mon[43975]: osdmap e64: 9 total, 8 up, 9 in 2024-02-09T17:21:22.654 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:22 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-09T17:21:22.654 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:22 smithi179 ceph-mon[43975]: from='osd.8 [v2:172.21.15.179:6800/4219839127,v1:172.21.15.179:6801/4219839127]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-02-09T17:21:22.654 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:22 smithi179 ceph-mon[43975]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-02-09T17:21:22.654 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:22 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:22.654 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:22 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:21:22.654 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:22 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:22.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:22 smithi162 ceph-mon[44304]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-02-09T17:21:22.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:22 smithi162 ceph-mon[44304]: osdmap e64: 9 total, 8 up, 9 in 2024-02-09T17:21:22.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:22 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-09T17:21:22.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:22 smithi162 ceph-mon[44304]: from='osd.8 [v2:172.21.15.179:6800/4219839127,v1:172.21.15.179:6801/4219839127]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-02-09T17:21:22.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:22 smithi162 ceph-mon[44304]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-02-09T17:21:22.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:22 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:22.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:22 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:21:22.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:22 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:22.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:22 smithi099 ceph-mon[39420]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-02-09T17:21:22.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:22 smithi099 ceph-mon[39420]: osdmap e64: 9 total, 8 up, 9 in 2024-02-09T17:21:22.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:22 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-09T17:21:22.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:22 smithi099 ceph-mon[39420]: from='osd.8 [v2:172.21.15.179:6800/4219839127,v1:172.21.15.179:6801/4219839127]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-02-09T17:21:22.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:22 smithi099 ceph-mon[39420]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-02-09T17:21:22.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:22 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:22.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:22 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:21:22.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:22 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:23.223 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 17:21:22 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[49792]: 2024-02-09T17:21:22.943+0000 7f1023669700 -1 osd.8 0 waiting for initial osdmap 2024-02-09T17:21:23.223 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 17:21:22 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[49792]: 2024-02-09T17:21:22.944+0000 7f1016dfc700 -1 osd.8 0 failed to load OSD map for epoch 10, got 0 bytes 2024-02-09T17:21:23.224 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 17:21:22 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[49792]: 2024-02-09T17:21:22.952+0000 7f101b45b700 -1 osd.8 65 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-09T17:21:23.529 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:21:23.774 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:23 smithi179 ceph-mon[43975]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-02-09T17:21:23.774 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:23 smithi179 ceph-mon[43975]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-02-09T17:21:23.774 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:23 smithi179 ceph-mon[43975]: osdmap e65: 9 total, 8 up, 9 in 2024-02-09T17:21:23.774 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:23 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-09T17:21:23.774 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:23 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-09T17:21:23.774 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:23 smithi179 ceph-mon[43975]: from='osd.8 ' entity='osd.8' 2024-02-09T17:21:23.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:23 smithi162 ceph-mon[44304]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-02-09T17:21:23.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:23 smithi162 ceph-mon[44304]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-02-09T17:21:23.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:23 smithi162 ceph-mon[44304]: osdmap e65: 9 total, 8 up, 9 in 2024-02-09T17:21:23.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:23 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-09T17:21:23.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:23 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-09T17:21:23.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:23 smithi162 ceph-mon[44304]: from='osd.8 ' entity='osd.8' 2024-02-09T17:21:23.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:23 smithi099 ceph-mon[39420]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-02-09T17:21:23.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:23 smithi099 ceph-mon[39420]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-02-09T17:21:23.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:23 smithi099 ceph-mon[39420]: osdmap e65: 9 total, 8 up, 9 in 2024-02-09T17:21:23.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:23 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-09T17:21:23.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:23 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-09T17:21:23.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:23 smithi099 ceph-mon[39420]: from='osd.8 ' entity='osd.8' 2024-02-09T17:21:24.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:24 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:21:24.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:24 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:21:24.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:24 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-09T17:21:24.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:24 smithi179 ceph-mon[43975]: osd.8 [v2:172.21.15.179:6800/4219839127,v1:172.21.15.179:6801/4219839127] boot 2024-02-09T17:21:24.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:24 smithi179 ceph-mon[43975]: osdmap e66: 9 total, 9 up, 9 in 2024-02-09T17:21:24.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:24 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-09T17:21:24.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:24 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:21:24.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:24 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:21:24.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:24 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-09T17:21:24.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:24 smithi162 ceph-mon[44304]: osd.8 [v2:172.21.15.179:6800/4219839127,v1:172.21.15.179:6801/4219839127] boot 2024-02-09T17:21:24.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:24 smithi162 ceph-mon[44304]: osdmap e66: 9 total, 9 up, 9 in 2024-02-09T17:21:24.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:24 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-09T17:21:24.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:24 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:21:24.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:24 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:21:24.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:24 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-09T17:21:24.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:24 smithi099 ceph-mon[39420]: osd.8 [v2:172.21.15.179:6800/4219839127,v1:172.21.15.179:6801/4219839127] boot 2024-02-09T17:21:24.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:24 smithi099 ceph-mon[39420]: osdmap e66: 9 total, 9 up, 9 in 2024-02-09T17:21:24.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:24 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-09T17:21:25.669 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:25 smithi179 ceph-mon[43975]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-02-09T17:21:25.669 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:25 smithi179 ceph-mon[43975]: osdmap e67: 9 total, 9 up, 9 in 2024-02-09T17:21:25.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:25 smithi162 ceph-mon[44304]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-02-09T17:21:25.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:25 smithi162 ceph-mon[44304]: osdmap e67: 9 total, 9 up, 9 in 2024-02-09T17:21:25.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:25 smithi099 ceph-mon[39420]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-02-09T17:21:25.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:25 smithi099 ceph-mon[39420]: osdmap e67: 9 total, 9 up, 9 in 2024-02-09T17:21:26.680 INFO:teuthology.orchestra.run.smithi179.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-09T17:21:26.680 INFO:teuthology.orchestra.run.smithi179.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-09T17:21:26.680 INFO:teuthology.orchestra.run.smithi179.stderr: stderr: 10+0 records in 2024-02-09T17:21:26.680 INFO:teuthology.orchestra.run.smithi179.stderr:10+0 records out 2024-02-09T17:21:26.680 INFO:teuthology.orchestra.run.smithi179.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111294 s, 942 MB/s 2024-02-09T17:21:26.680 INFO:teuthology.orchestra.run.smithi179.stderr:--> Zapping successful for: 2024-02-09T17:21:26.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:26 smithi179 ceph-mon[43975]: osdmap e68: 9 total, 9 up, 9 in 2024-02-09T17:21:26.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:26 smithi179 ceph-mon[43975]: osdmap e69: 9 total, 9 up, 9 in 2024-02-09T17:21:26.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:26 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:26.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:26 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:26.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:26 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:21:26.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:26 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:26.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:26 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:21:26.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:26 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:21:26.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:26 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:26.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:26 smithi162 ceph-mon[44304]: osdmap e68: 9 total, 9 up, 9 in 2024-02-09T17:21:26.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:26 smithi162 ceph-mon[44304]: osdmap e69: 9 total, 9 up, 9 in 2024-02-09T17:21:26.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:26 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:26.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:26 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:26.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:26 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:21:26.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:26 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:26.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:26 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:21:26.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:26 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:21:26.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:26 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:26.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:26 smithi099 ceph-mon[39420]: osdmap e68: 9 total, 9 up, 9 in 2024-02-09T17:21:26.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:26 smithi099 ceph-mon[39420]: osdmap e69: 9 total, 9 up, 9 in 2024-02-09T17:21:26.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:26 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:26.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:26 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:26.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:26 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:21:26.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:26 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:26.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:26 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:21:26.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:26 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:21:26.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:26 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:27.420 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph orch daemon add osd smithi179:vg_nvme/lv_3 2024-02-09T17:21:27.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:27 smithi179 ceph-mon[43975]: Detected new or changed devices on smithi179 2024-02-09T17:21:27.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:27 smithi179 ceph-mon[43975]: Adjusting osd_memory_target on smithi179 to 21218M 2024-02-09T17:21:27.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:27 smithi179 ceph-mon[43975]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-02-09T17:21:27.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:27 smithi179 ceph-mon[43975]: osdmap e70: 9 total, 9 up, 9 in 2024-02-09T17:21:27.683 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:21:27.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:27 smithi162 ceph-mon[44304]: Detected new or changed devices on smithi179 2024-02-09T17:21:27.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:27 smithi162 ceph-mon[44304]: Adjusting osd_memory_target on smithi179 to 21218M 2024-02-09T17:21:27.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:27 smithi162 ceph-mon[44304]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-02-09T17:21:27.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:27 smithi162 ceph-mon[44304]: osdmap e70: 9 total, 9 up, 9 in 2024-02-09T17:21:27.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:27 smithi099 ceph-mon[39420]: Detected new or changed devices on smithi179 2024-02-09T17:21:27.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:27 smithi099 ceph-mon[39420]: Adjusting osd_memory_target on smithi179 to 21218M 2024-02-09T17:21:27.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:27 smithi099 ceph-mon[39420]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-02-09T17:21:27.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:27 smithi099 ceph-mon[39420]: osdmap e70: 9 total, 9 up, 9 in 2024-02-09T17:21:29.888 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:29 smithi179 ceph-mon[43975]: pgmap v212: 1 pgs: 1 peering; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-02-09T17:21:29.888 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:29 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T17:21:29.889 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:29 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T17:21:29.889 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:29 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:21:29.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:29 smithi162 ceph-mon[44304]: pgmap v212: 1 pgs: 1 peering; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-02-09T17:21:29.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:29 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T17:21:29.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:29 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T17:21:29.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:29 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:21:29.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:29 smithi099 ceph-mon[39420]: pgmap v212: 1 pgs: 1 peering; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-02-09T17:21:29.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:29 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T17:21:29.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:29 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T17:21:29.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:29 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:21:30.791 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:30 smithi179 ceph-mon[43975]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:21:30.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:30 smithi162 ceph-mon[44304]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:21:30.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:30 smithi099 ceph-mon[39420]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:21:31.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:31 smithi179 ceph-mon[43975]: pgmap v213: 1 pgs: 1 peering; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-02-09T17:21:31.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:31 smithi162 ceph-mon[44304]: pgmap v213: 1 pgs: 1 peering; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-02-09T17:21:31.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:31 smithi099 ceph-mon[39420]: pgmap v213: 1 pgs: 1 peering; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-02-09T17:21:32.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:32 smithi179 ceph-mon[43975]: from='client.? 172.21.15.179:0/2328504358' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d995854-7328-4dfc-b1a9-0edca8d74f46"}]: dispatch 2024-02-09T17:21:32.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:32 smithi179 ceph-mon[43975]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d995854-7328-4dfc-b1a9-0edca8d74f46"}]: dispatch 2024-02-09T17:21:32.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:32 smithi179 ceph-mon[43975]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d995854-7328-4dfc-b1a9-0edca8d74f46"}]': finished 2024-02-09T17:21:32.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:32 smithi179 ceph-mon[43975]: osdmap e71: 10 total, 9 up, 10 in 2024-02-09T17:21:32.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:32 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-09T17:21:32.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:32 smithi162 ceph-mon[44304]: from='client.? 172.21.15.179:0/2328504358' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d995854-7328-4dfc-b1a9-0edca8d74f46"}]: dispatch 2024-02-09T17:21:32.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:32 smithi162 ceph-mon[44304]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d995854-7328-4dfc-b1a9-0edca8d74f46"}]: dispatch 2024-02-09T17:21:32.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:32 smithi162 ceph-mon[44304]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d995854-7328-4dfc-b1a9-0edca8d74f46"}]': finished 2024-02-09T17:21:32.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:32 smithi162 ceph-mon[44304]: osdmap e71: 10 total, 9 up, 10 in 2024-02-09T17:21:32.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:32 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-09T17:21:32.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:32 smithi099 ceph-mon[39420]: from='client.? 172.21.15.179:0/2328504358' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d995854-7328-4dfc-b1a9-0edca8d74f46"}]: dispatch 2024-02-09T17:21:32.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:32 smithi099 ceph-mon[39420]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d995854-7328-4dfc-b1a9-0edca8d74f46"}]: dispatch 2024-02-09T17:21:32.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:32 smithi099 ceph-mon[39420]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d995854-7328-4dfc-b1a9-0edca8d74f46"}]': finished 2024-02-09T17:21:32.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:32 smithi099 ceph-mon[39420]: osdmap e71: 10 total, 9 up, 10 in 2024-02-09T17:21:32.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:32 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-09T17:21:33.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:33 smithi179 ceph-mon[43975]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-02-09T17:21:33.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:33 smithi179 ceph-mon[43975]: from='client.? 172.21.15.179:0/3669903854' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T17:21:33.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:33 smithi162 ceph-mon[44304]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-02-09T17:21:33.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:33 smithi162 ceph-mon[44304]: from='client.? 172.21.15.179:0/3669903854' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T17:21:33.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:33 smithi099 ceph-mon[39420]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-02-09T17:21:33.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:33 smithi099 ceph-mon[39420]: from='client.? 172.21.15.179:0/3669903854' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T17:21:34.848 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:34 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:21:34.848 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:34 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:21:34.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:34 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:21:34.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:34 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:21:34.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:34 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:21:34.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:34 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:21:35.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:35 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:21:35.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:35 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:21:35.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:35 smithi179 ceph-mon[43975]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-09T17:21:35.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:35 smithi179 ceph-mon[43975]: osdmap e72: 10 total, 9 up, 10 in 2024-02-09T17:21:35.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:35 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-09T17:21:35.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:35 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:21:35.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:35 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:21:35.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:35 smithi162 ceph-mon[44304]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-09T17:21:35.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:35 smithi162 ceph-mon[44304]: osdmap e72: 10 total, 9 up, 10 in 2024-02-09T17:21:35.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:35 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-09T17:21:35.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:35 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:21:35.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:35 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:21:35.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:35 smithi099 ceph-mon[39420]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-09T17:21:35.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:35 smithi099 ceph-mon[39420]: osdmap e72: 10 total, 9 up, 10 in 2024-02-09T17:21:35.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:35 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-09T17:21:37.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:36 smithi179 ceph-mon[43975]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-02-09T17:21:37.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:36 smithi099 ceph-mon[39420]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-02-09T17:21:37.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:36 smithi162 ceph-mon[44304]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-02-09T17:21:38.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:37 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:21:38.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:37 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:21:38.257 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:37 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:21:38.257 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:37 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:21:38.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:37 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:21:38.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:37 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:21:39.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:38 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:21:39.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:38 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:21:39.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:38 smithi099 ceph-mon[39420]: osdmap e73: 10 total, 9 up, 10 in 2024-02-09T17:21:39.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:38 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-09T17:21:39.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:38 smithi099 ceph-mon[39420]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-02-09T17:21:39.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:38 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-02-09T17:21:39.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:38 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:21:39.403 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:38 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:21:39.403 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:38 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:21:39.403 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:38 smithi179 ceph-mon[43975]: osdmap e73: 10 total, 9 up, 10 in 2024-02-09T17:21:39.403 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:38 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-09T17:21:39.403 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:38 smithi179 ceph-mon[43975]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-02-09T17:21:39.404 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:38 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-02-09T17:21:39.404 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:38 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:21:39.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:38 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:21:39.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:38 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:21:39.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:38 smithi162 ceph-mon[44304]: osdmap e73: 10 total, 9 up, 10 in 2024-02-09T17:21:39.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:38 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-09T17:21:39.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:38 smithi162 ceph-mon[44304]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-02-09T17:21:39.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:38 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-02-09T17:21:39.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:38 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:21:40.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:39 smithi179 ceph-mon[43975]: Deploying daemon osd.9 on smithi179 2024-02-09T17:21:40.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:39 smithi099 ceph-mon[39420]: Deploying daemon osd.9 on smithi179 2024-02-09T17:21:40.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:39 smithi162 ceph-mon[44304]: Deploying daemon osd.9 on smithi179 2024-02-09T17:21:41.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:40 smithi099 ceph-mon[39420]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-02-09T17:21:41.303 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:40 smithi179 ceph-mon[43975]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-02-09T17:21:41.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:40 smithi162 ceph-mon[44304]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-02-09T17:21:43.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:43 smithi179 ceph-mon[43975]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-02-09T17:21:43.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:43 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:21:43.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:43 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:21:43.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:43 smithi162 ceph-mon[44304]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-02-09T17:21:43.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:43 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:21:43.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:43 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:21:43.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:43 smithi099 ceph-mon[39420]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-02-09T17:21:43.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:43 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:21:43.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:43 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:21:44.649 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:44 smithi179 ceph-mon[43975]: osdmap e74: 10 total, 9 up, 10 in 2024-02-09T17:21:44.649 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:44 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-09T17:21:44.649 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:44 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:44.649 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:44 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:44.649 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:44 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:21:44.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:44 smithi162 ceph-mon[44304]: osdmap e74: 10 total, 9 up, 10 in 2024-02-09T17:21:44.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:44 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-09T17:21:44.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:44 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:44.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:44 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:44.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:44 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:21:44.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:44 smithi099 ceph-mon[39420]: osdmap e74: 10 total, 9 up, 10 in 2024-02-09T17:21:44.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:44 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-09T17:21:44.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:44 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:44.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:44 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:44.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:44 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:21:45.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:45 smithi179 ceph-mon[43975]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-02-09T17:21:45.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:45 smithi162 ceph-mon[44304]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-02-09T17:21:45.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:45 smithi099 ceph-mon[39420]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-02-09T17:21:47.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:46 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:47.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:46 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:47.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:46 smithi099 ceph-mon[39420]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-02-09T17:21:47.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:46 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:21:47.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:46 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:21:47.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:46 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:47.354 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:46 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:47.354 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:46 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:47.354 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:46 smithi179 ceph-mon[43975]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-02-09T17:21:47.354 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:46 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:21:47.354 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:46 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:21:47.354 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:46 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:47.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:46 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:47.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:46 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:47.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:46 smithi162 ceph-mon[44304]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-02-09T17:21:47.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:46 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:21:47.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:46 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:21:47.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:46 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:47.867 INFO:teuthology.orchestra.run.smithi179.stdout:Created osd(s) 9 on host 'smithi179' 2024-02-09T17:21:48.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:47 smithi179 ceph-mon[43975]: from='osd.9 [v2:172.21.15.179:6808/504427183,v1:172.21.15.179:6809/504427183]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-02-09T17:21:48.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:47 smithi179 ceph-mon[43975]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-02-09T17:21:48.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:47 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:48.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:47 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:48.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:47 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:21:48.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:47 smithi099 ceph-mon[39420]: from='osd.9 [v2:172.21.15.179:6808/504427183,v1:172.21.15.179:6809/504427183]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-02-09T17:21:48.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:47 smithi099 ceph-mon[39420]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-02-09T17:21:48.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:47 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:48.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:47 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:48.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:47 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:21:48.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:47 smithi162 ceph-mon[44304]: from='osd.9 [v2:172.21.15.179:6808/504427183,v1:172.21.15.179:6809/504427183]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-02-09T17:21:48.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:47 smithi162 ceph-mon[44304]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-02-09T17:21:48.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:47 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:48.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:47 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:48.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:47 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:21:48.516 DEBUG:teuthology.orchestra.run.smithi179:osd.9> sudo journalctl -f -n 0 -u ceph-a69c870a-c76e-11ee-95b7-87774f69a715@osd.9.service 2024-02-09T17:21:48.519 INFO:tasks.cephadm:Deploying osd.10 on smithi179 with /dev/vg_nvme/lv_2... 2024-02-09T17:21:48.519 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-09T17:21:48.551 INFO:journalctl@ceph.osd.9.smithi179.stdout:-- Logs begin at Fri 2024-02-09 17:04:31 UTC. -- 2024-02-09T17:21:49.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:49 smithi179 ceph-mon[43975]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-02-09T17:21:49.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:49 smithi179 ceph-mon[43975]: osdmap e75: 10 total, 9 up, 10 in 2024-02-09T17:21:49.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:49 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-09T17:21:49.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:49 smithi179 ceph-mon[43975]: from='osd.9 [v2:172.21.15.179:6808/504427183,v1:172.21.15.179:6809/504427183]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-02-09T17:21:49.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:49 smithi179 ceph-mon[43975]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-02-09T17:21:49.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:49 smithi179 ceph-mon[43975]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-02-09T17:21:49.333 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:21:49.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:49 smithi162 ceph-mon[44304]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-02-09T17:21:49.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:49 smithi162 ceph-mon[44304]: osdmap e75: 10 total, 9 up, 10 in 2024-02-09T17:21:49.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:49 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-09T17:21:49.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:49 smithi162 ceph-mon[44304]: from='osd.9 [v2:172.21.15.179:6808/504427183,v1:172.21.15.179:6809/504427183]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-02-09T17:21:49.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:49 smithi162 ceph-mon[44304]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-02-09T17:21:49.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:49 smithi162 ceph-mon[44304]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-02-09T17:21:49.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:49 smithi099 ceph-mon[39420]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-02-09T17:21:49.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:49 smithi099 ceph-mon[39420]: osdmap e75: 10 total, 9 up, 10 in 2024-02-09T17:21:49.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:49 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-09T17:21:49.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:49 smithi099 ceph-mon[39420]: from='osd.9 [v2:172.21.15.179:6808/504427183,v1:172.21.15.179:6809/504427183]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-02-09T17:21:49.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:49 smithi099 ceph-mon[39420]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-02-09T17:21:49.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:49 smithi099 ceph-mon[39420]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-02-09T17:21:49.826 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 17:21:49 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T17:21:49.440+0000 7f4fac332700 -1 osd.9 0 waiting for initial osdmap 2024-02-09T17:21:49.826 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 17:21:49 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T17:21:49.440+0000 7f4f9f2c4700 -1 osd.9 0 failed to load OSD map for epoch 21, got 0 bytes 2024-02-09T17:21:49.826 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 17:21:49 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T17:21:49.459+0000 7f4fa3923700 -1 osd.9 76 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-09T17:21:50.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:50 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:21:50.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:50 smithi179 ceph-mon[43975]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-02-09T17:21:50.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:50 smithi179 ceph-mon[43975]: osdmap e76: 10 total, 9 up, 10 in 2024-02-09T17:21:50.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:50 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-09T17:21:50.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:50 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-09T17:21:50.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:50 smithi179 ceph-mon[43975]: from='osd.9 ' entity='osd.9' 2024-02-09T17:21:50.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:50 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:21:50.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:50 smithi162 ceph-mon[44304]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-02-09T17:21:50.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:50 smithi162 ceph-mon[44304]: osdmap e76: 10 total, 9 up, 10 in 2024-02-09T17:21:50.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:50 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-09T17:21:50.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:50 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-09T17:21:50.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:50 smithi162 ceph-mon[44304]: from='osd.9 ' entity='osd.9' 2024-02-09T17:21:50.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:50 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:21:50.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:50 smithi099 ceph-mon[39420]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-02-09T17:21:50.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:50 smithi099 ceph-mon[39420]: osdmap e76: 10 total, 9 up, 10 in 2024-02-09T17:21:50.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:50 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-09T17:21:50.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:50 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-09T17:21:50.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:50 smithi099 ceph-mon[39420]: from='osd.9 ' entity='osd.9' 2024-02-09T17:21:51.151 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:51 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-09T17:21:51.151 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:51 smithi179 ceph-mon[43975]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-02-09T17:21:51.151 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:51 smithi179 ceph-mon[43975]: osd.9 [v2:172.21.15.179:6808/504427183,v1:172.21.15.179:6809/504427183] boot 2024-02-09T17:21:51.151 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:51 smithi179 ceph-mon[43975]: osdmap e77: 10 total, 10 up, 10 in 2024-02-09T17:21:51.151 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:51 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-09T17:21:51.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:51 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-09T17:21:51.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:51 smithi162 ceph-mon[44304]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-02-09T17:21:51.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:51 smithi162 ceph-mon[44304]: osd.9 [v2:172.21.15.179:6808/504427183,v1:172.21.15.179:6809/504427183] boot 2024-02-09T17:21:51.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:51 smithi162 ceph-mon[44304]: osdmap e77: 10 total, 10 up, 10 in 2024-02-09T17:21:51.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:51 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-09T17:21:51.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:51 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-09T17:21:51.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:51 smithi099 ceph-mon[39420]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-02-09T17:21:51.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:51 smithi099 ceph-mon[39420]: osd.9 [v2:172.21.15.179:6808/504427183,v1:172.21.15.179:6809/504427183] boot 2024-02-09T17:21:51.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:51 smithi099 ceph-mon[39420]: osdmap e77: 10 total, 10 up, 10 in 2024-02-09T17:21:51.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:51 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-09T17:21:51.793 INFO:teuthology.orchestra.run.smithi179.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-09T17:21:51.793 INFO:teuthology.orchestra.run.smithi179.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-09T17:21:51.793 INFO:teuthology.orchestra.run.smithi179.stderr: stderr: 10+0 records in 2024-02-09T17:21:51.793 INFO:teuthology.orchestra.run.smithi179.stderr:10+0 records out 2024-02-09T17:21:51.793 INFO:teuthology.orchestra.run.smithi179.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0111638 s, 939 MB/s 2024-02-09T17:21:51.793 INFO:teuthology.orchestra.run.smithi179.stderr:--> Zapping successful for: 2024-02-09T17:21:52.626 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph orch daemon add osd smithi179:vg_nvme/lv_2 2024-02-09T17:21:53.288 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:21:53.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:53 smithi179 ceph-mon[43975]: osdmap e78: 10 total, 10 up, 10 in 2024-02-09T17:21:53.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:53 smithi179 ceph-mon[43975]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-02-09T17:21:53.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:53 smithi162 ceph-mon[44304]: osdmap e78: 10 total, 10 up, 10 in 2024-02-09T17:21:53.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:53 smithi162 ceph-mon[44304]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-02-09T17:21:53.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:53 smithi099 ceph-mon[39420]: osdmap e78: 10 total, 10 up, 10 in 2024-02-09T17:21:53.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:53 smithi099 ceph-mon[39420]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-02-09T17:21:54.551 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:54 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:54.551 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:54 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:54.551 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:54 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:21:54.551 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:54 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:21:54.551 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:54 smithi179 ceph-mon[43975]: Adjusting osd_memory_target on smithi179 to 10609M 2024-02-09T17:21:54.551 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:54 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:54.551 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:54 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:21:54.551 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:54 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:21:54.551 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:54 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:21:54.552 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:54 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:21:54.552 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:54 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:21:54.552 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:54 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:54.552 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:54 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:21:54.552 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:54 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:21:54.552 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:54 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:21:54.552 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:54 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:54.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:54 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:54.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:54 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:54.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:54 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:21:54.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:54 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:21:54.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:54 smithi162 ceph-mon[44304]: Adjusting osd_memory_target on smithi179 to 10609M 2024-02-09T17:21:54.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:54 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:54.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:54 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:21:54.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:54 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:21:54.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:54 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:21:54.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:54 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:21:54.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:54 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:21:54.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:54 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:54.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:54 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:21:54.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:54 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:21:54.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:54 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:21:54.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:54 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:54.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:54 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:54.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:54 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:54.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:54 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:21:54.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:54 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:21:54.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:54 smithi099 ceph-mon[39420]: Adjusting osd_memory_target on smithi179 to 10609M 2024-02-09T17:21:54.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:54 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:54.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:54 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:21:54.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:54 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:21:54.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:54 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:21:54.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:54 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:21:54.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:54 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:21:54.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:54 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:54.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:54 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:21:54.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:54 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:21:54.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:54 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:21:54.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:54 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:21:55.801 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:55 smithi179 ceph-mon[43975]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-02-09T17:21:55.801 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:55 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T17:21:55.801 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:55 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T17:21:55.801 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:55 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:21:55.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:55 smithi162 ceph-mon[44304]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-02-09T17:21:55.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:55 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T17:21:55.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:55 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T17:21:55.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:55 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:21:55.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:55 smithi099 ceph-mon[39420]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-02-09T17:21:55.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:55 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T17:21:55.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:55 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T17:21:55.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:55 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:21:56.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:56 smithi179 ceph-mon[43975]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:21:56.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:56 smithi162 ceph-mon[44304]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:21:56.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:56 smithi099 ceph-mon[39420]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:21:57.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:57 smithi179 ceph-mon[43975]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-02-09T17:21:57.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:57 smithi162 ceph-mon[44304]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-02-09T17:21:57.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:57 smithi099 ceph-mon[39420]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-02-09T17:21:58.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:58 smithi179 ceph-mon[43975]: from='client.? 172.21.15.179:0/2514859907' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "151fbedd-08e6-46e6-a4e5-a14248399f61"}]: dispatch 2024-02-09T17:21:58.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:58 smithi179 ceph-mon[43975]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "151fbedd-08e6-46e6-a4e5-a14248399f61"}]: dispatch 2024-02-09T17:21:58.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:58 smithi179 ceph-mon[43975]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "151fbedd-08e6-46e6-a4e5-a14248399f61"}]': finished 2024-02-09T17:21:58.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:58 smithi179 ceph-mon[43975]: osdmap e79: 11 total, 10 up, 11 in 2024-02-09T17:21:58.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:58 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-09T17:21:58.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:58 smithi162 ceph-mon[44304]: from='client.? 172.21.15.179:0/2514859907' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "151fbedd-08e6-46e6-a4e5-a14248399f61"}]: dispatch 2024-02-09T17:21:58.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:58 smithi162 ceph-mon[44304]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "151fbedd-08e6-46e6-a4e5-a14248399f61"}]: dispatch 2024-02-09T17:21:58.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:58 smithi162 ceph-mon[44304]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "151fbedd-08e6-46e6-a4e5-a14248399f61"}]': finished 2024-02-09T17:21:58.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:58 smithi162 ceph-mon[44304]: osdmap e79: 11 total, 10 up, 11 in 2024-02-09T17:21:58.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:58 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-09T17:21:58.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:58 smithi099 ceph-mon[39420]: from='client.? 172.21.15.179:0/2514859907' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "151fbedd-08e6-46e6-a4e5-a14248399f61"}]: dispatch 2024-02-09T17:21:58.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:58 smithi099 ceph-mon[39420]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "151fbedd-08e6-46e6-a4e5-a14248399f61"}]: dispatch 2024-02-09T17:21:58.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:58 smithi099 ceph-mon[39420]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "151fbedd-08e6-46e6-a4e5-a14248399f61"}]': finished 2024-02-09T17:21:58.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:58 smithi099 ceph-mon[39420]: osdmap e79: 11 total, 10 up, 11 in 2024-02-09T17:21:58.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:58 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-09T17:21:59.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:59 smithi179 ceph-mon[43975]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-02-09T17:21:59.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:21:59 smithi179 ceph-mon[43975]: from='client.? 172.21.15.179:0/241939390' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T17:21:59.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:59 smithi162 ceph-mon[44304]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-02-09T17:21:59.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:21:59 smithi162 ceph-mon[44304]: from='client.? 172.21.15.179:0/241939390' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T17:21:59.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:59 smithi099 ceph-mon[39420]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-02-09T17:21:59.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:21:59 smithi099 ceph-mon[39420]: from='client.? 172.21.15.179:0/241939390' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T17:22:00.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:00 smithi179 ceph-mon[43975]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-02-09T17:22:00.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:00 smithi162 ceph-mon[44304]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-02-09T17:22:00.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:00 smithi099 ceph-mon[39420]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-02-09T17:22:03.571 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:03 smithi179 ceph-mon[43975]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-02-09T17:22:03.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:03 smithi162 ceph-mon[44304]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-02-09T17:22:03.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:03 smithi099 ceph-mon[39420]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-02-09T17:22:05.575 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:05 smithi179 ceph-mon[43975]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-02-09T17:22:05.576 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:05 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:22:05.576 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:05 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:22:05.576 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:05 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-02-09T17:22:05.576 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:05 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:22:05.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:05 smithi162 ceph-mon[44304]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-02-09T17:22:05.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:05 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:22:05.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:05 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:22:05.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:05 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-02-09T17:22:05.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:05 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:22:05.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:05 smithi099 ceph-mon[39420]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-02-09T17:22:05.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:05 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:22:05.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:05 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:22:05.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:05 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-02-09T17:22:05.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:05 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:22:06.579 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:06 smithi179 ceph-mon[43975]: Deploying daemon osd.10 on smithi179 2024-02-09T17:22:06.579 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:06 smithi179 ceph-mon[43975]: osdmap e80: 11 total, 10 up, 11 in 2024-02-09T17:22:06.579 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:06 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-09T17:22:06.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:06 smithi162 ceph-mon[44304]: Deploying daemon osd.10 on smithi179 2024-02-09T17:22:06.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:06 smithi162 ceph-mon[44304]: osdmap e80: 11 total, 10 up, 11 in 2024-02-09T17:22:06.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:06 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-09T17:22:06.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:06 smithi099 ceph-mon[39420]: Deploying daemon osd.10 on smithi179 2024-02-09T17:22:06.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:06 smithi099 ceph-mon[39420]: osdmap e80: 11 total, 10 up, 11 in 2024-02-09T17:22:06.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:06 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-09T17:22:07.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:07 smithi179 ceph-mon[43975]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-02-09T17:22:07.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:07 smithi162 ceph-mon[44304]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-02-09T17:22:07.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:07 smithi099 ceph-mon[39420]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-02-09T17:22:09.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:09 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:22:09.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:09 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:22:09.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:09 smithi179 ceph-mon[43975]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-02-09T17:22:09.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:09 smithi179 ceph-mon[43975]: osdmap e81: 11 total, 10 up, 11 in 2024-02-09T17:22:09.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:09 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-09T17:22:09.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:09 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:22:09.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:09 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:22:09.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:09 smithi162 ceph-mon[44304]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-02-09T17:22:09.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:09 smithi162 ceph-mon[44304]: osdmap e81: 11 total, 10 up, 11 in 2024-02-09T17:22:09.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:09 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-09T17:22:09.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:09 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:22:09.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:09 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:22:09.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:09 smithi099 ceph-mon[39420]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-02-09T17:22:09.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:09 smithi099 ceph-mon[39420]: osdmap e81: 11 total, 10 up, 11 in 2024-02-09T17:22:09.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:09 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-09T17:22:11.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:11 smithi179 ceph-mon[43975]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-02-09T17:22:11.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:11 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:11.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:11 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:11.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:11 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:22:11.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:11 smithi162 ceph-mon[44304]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-02-09T17:22:11.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:11 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:11.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:11 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:11.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:11 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:22:11.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:11 smithi099 ceph-mon[39420]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-02-09T17:22:11.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:11 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:11.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:11 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:11.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:11 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:22:12.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:12 smithi179 ceph-mon[43975]: 1.0 scrub starts 2024-02-09T17:22:12.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:12 smithi179 ceph-mon[43975]: 1.0 scrub ok 2024-02-09T17:22:12.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:12 smithi162 ceph-mon[44304]: 1.0 scrub starts 2024-02-09T17:22:12.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:12 smithi162 ceph-mon[44304]: 1.0 scrub ok 2024-02-09T17:22:12.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:12 smithi099 ceph-mon[39420]: 1.0 scrub starts 2024-02-09T17:22:12.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:12 smithi099 ceph-mon[39420]: 1.0 scrub ok 2024-02-09T17:22:13.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:13 smithi179 ceph-mon[43975]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-02-09T17:22:13.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:13 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:13.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:13 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:13.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:13 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:22:13.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:13 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:22:13.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:13 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:13.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:13 smithi162 ceph-mon[44304]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-02-09T17:22:13.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:13 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:13.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:13 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:13.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:13 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:22:13.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:13 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:22:13.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:13 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:13.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:13 smithi099 ceph-mon[39420]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-02-09T17:22:13.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:13 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:13.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:13 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:13.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:13 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:22:13.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:13 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:22:13.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:13 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:14.367 INFO:teuthology.orchestra.run.smithi179.stdout:Created osd(s) 10 on host 'smithi179' 2024-02-09T17:22:14.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:14 smithi179 ceph-mon[43975]: from='osd.10 [v2:172.21.15.179:6816/2403345164,v1:172.21.15.179:6817/2403345164]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-02-09T17:22:14.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:14 smithi179 ceph-mon[43975]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-02-09T17:22:14.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:14 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:14.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:14 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:14.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:14 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:22:14.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:14 smithi162 ceph-mon[44304]: from='osd.10 [v2:172.21.15.179:6816/2403345164,v1:172.21.15.179:6817/2403345164]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-02-09T17:22:14.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:14 smithi162 ceph-mon[44304]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-02-09T17:22:14.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:14 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:14.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:14 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:14.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:14 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:22:14.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:14 smithi099 ceph-mon[39420]: from='osd.10 [v2:172.21.15.179:6816/2403345164,v1:172.21.15.179:6817/2403345164]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-02-09T17:22:14.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:14 smithi099 ceph-mon[39420]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-02-09T17:22:14.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:14 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:14.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:14 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:14.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:14 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:22:15.173 DEBUG:teuthology.orchestra.run.smithi179:osd.10> sudo journalctl -f -n 0 -u ceph-a69c870a-c76e-11ee-95b7-87774f69a715@osd.10.service 2024-02-09T17:22:15.175 INFO:tasks.cephadm:Deploying osd.11 on smithi179 with /dev/vg_nvme/lv_1... 2024-02-09T17:22:15.175 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-09T17:22:15.204 INFO:journalctl@ceph.osd.10.smithi179.stdout:-- Logs begin at Fri 2024-02-09 17:04:31 UTC. -- 2024-02-09T17:22:15.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:15 smithi179 ceph-mon[43975]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-02-09T17:22:15.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:15 smithi179 ceph-mon[43975]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-02-09T17:22:15.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:15 smithi179 ceph-mon[43975]: from='osd.10 [v2:172.21.15.179:6816/2403345164,v1:172.21.15.179:6817/2403345164]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-02-09T17:22:15.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:15 smithi179 ceph-mon[43975]: osdmap e82: 11 total, 10 up, 11 in 2024-02-09T17:22:15.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:15 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-09T17:22:15.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:15 smithi179 ceph-mon[43975]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-02-09T17:22:15.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:15 smithi162 ceph-mon[44304]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-02-09T17:22:15.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:15 smithi162 ceph-mon[44304]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-02-09T17:22:15.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:15 smithi162 ceph-mon[44304]: from='osd.10 [v2:172.21.15.179:6816/2403345164,v1:172.21.15.179:6817/2403345164]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-02-09T17:22:15.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:15 smithi162 ceph-mon[44304]: osdmap e82: 11 total, 10 up, 11 in 2024-02-09T17:22:15.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:15 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-09T17:22:15.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:15 smithi162 ceph-mon[44304]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-02-09T17:22:15.930 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:22:15.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:15 smithi099 ceph-mon[39420]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-02-09T17:22:15.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:15 smithi099 ceph-mon[39420]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-02-09T17:22:15.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:15 smithi099 ceph-mon[39420]: from='osd.10 [v2:172.21.15.179:6816/2403345164,v1:172.21.15.179:6817/2403345164]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-02-09T17:22:15.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:15 smithi099 ceph-mon[39420]: osdmap e82: 11 total, 10 up, 11 in 2024-02-09T17:22:15.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:15 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-09T17:22:15.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:15 smithi099 ceph-mon[39420]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-02-09T17:22:16.292 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 17:22:15 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T17:22:15.993+0000 7f434f61f700 -1 osd.10 0 waiting for initial osdmap 2024-02-09T17:22:16.292 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 17:22:15 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T17:22:15.994+0000 7f4342db2700 -1 osd.10 0 failed to load OSD map for epoch 32, got 0 bytes 2024-02-09T17:22:16.293 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 17:22:16 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T17:22:16.003+0000 7f4347411700 -1 osd.10 83 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-09T17:22:16.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:16 smithi179 ceph-mon[43975]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-02-09T17:22:16.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:16 smithi179 ceph-mon[43975]: osdmap e83: 11 total, 10 up, 11 in 2024-02-09T17:22:16.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:16 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-09T17:22:16.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:16 smithi179 ceph-mon[43975]: from='osd.10 ' entity='osd.10' 2024-02-09T17:22:16.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:16 smithi179 ceph-mon[43975]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-02-09T17:22:16.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:16 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-09T17:22:16.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:16 smithi162 ceph-mon[44304]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-02-09T17:22:16.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:16 smithi162 ceph-mon[44304]: osdmap e83: 11 total, 10 up, 11 in 2024-02-09T17:22:16.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:16 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-09T17:22:16.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:16 smithi162 ceph-mon[44304]: from='osd.10 ' entity='osd.10' 2024-02-09T17:22:16.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:16 smithi162 ceph-mon[44304]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-02-09T17:22:16.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:16 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-09T17:22:16.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:16 smithi099 ceph-mon[39420]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-02-09T17:22:16.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:16 smithi099 ceph-mon[39420]: osdmap e83: 11 total, 10 up, 11 in 2024-02-09T17:22:16.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:16 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-09T17:22:16.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:16 smithi099 ceph-mon[39420]: from='osd.10 ' entity='osd.10' 2024-02-09T17:22:16.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:16 smithi099 ceph-mon[39420]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-02-09T17:22:16.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:16 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-09T17:22:17.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:17 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:22:17.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:17 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:22:17.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:17 smithi179 ceph-mon[43975]: osd.10 [v2:172.21.15.179:6816/2403345164,v1:172.21.15.179:6817/2403345164] boot 2024-02-09T17:22:17.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:17 smithi179 ceph-mon[43975]: osdmap e84: 11 total, 11 up, 11 in 2024-02-09T17:22:17.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:17 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-09T17:22:17.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:17 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:22:17.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:17 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:22:17.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:17 smithi162 ceph-mon[44304]: osd.10 [v2:172.21.15.179:6816/2403345164,v1:172.21.15.179:6817/2403345164] boot 2024-02-09T17:22:17.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:17 smithi162 ceph-mon[44304]: osdmap e84: 11 total, 11 up, 11 in 2024-02-09T17:22:17.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:17 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-09T17:22:17.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:17 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:22:17.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:17 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:22:17.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:17 smithi099 ceph-mon[39420]: osd.10 [v2:172.21.15.179:6816/2403345164,v1:172.21.15.179:6817/2403345164] boot 2024-02-09T17:22:17.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:17 smithi099 ceph-mon[39420]: osdmap e84: 11 total, 11 up, 11 in 2024-02-09T17:22:17.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:17 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-09T17:22:18.555 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:18 smithi179 ceph-mon[43975]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-02-09T17:22:18.889 INFO:teuthology.orchestra.run.smithi179.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-09T17:22:18.889 INFO:teuthology.orchestra.run.smithi179.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-09T17:22:18.889 INFO:teuthology.orchestra.run.smithi179.stderr: stderr: 10+0 records in 2024-02-09T17:22:18.889 INFO:teuthology.orchestra.run.smithi179.stderr:10+0 records out 2024-02-09T17:22:18.889 INFO:teuthology.orchestra.run.smithi179.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112602 s, 931 MB/s 2024-02-09T17:22:18.889 INFO:teuthology.orchestra.run.smithi179.stderr:--> Zapping successful for: 2024-02-09T17:22:18.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:18 smithi162 ceph-mon[44304]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-02-09T17:22:18.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:18 smithi099 ceph-mon[39420]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-02-09T17:22:19.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:19 smithi179 ceph-mon[43975]: osdmap e85: 11 total, 11 up, 11 in 2024-02-09T17:22:19.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:19 smithi179 ceph-mon[43975]: Detected new or changed devices on smithi179 2024-02-09T17:22:19.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:19 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:19.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:19 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:19.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:19 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:22:19.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:19 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:22:19.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:19 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:22:19.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:19 smithi179 ceph-mon[43975]: Adjusting osd_memory_target on smithi179 to 7072M 2024-02-09T17:22:19.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:19 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:19.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:19 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:22:19.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:19 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:22:19.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:19 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:19.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:19 smithi162 ceph-mon[44304]: osdmap e85: 11 total, 11 up, 11 in 2024-02-09T17:22:19.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:19 smithi162 ceph-mon[44304]: Detected new or changed devices on smithi179 2024-02-09T17:22:19.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:19 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:19.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:19 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:19.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:19 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:22:19.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:19 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:22:19.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:19 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:22:19.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:19 smithi162 ceph-mon[44304]: Adjusting osd_memory_target on smithi179 to 7072M 2024-02-09T17:22:19.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:19 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:19.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:19 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:22:19.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:19 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:22:19.932 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:19 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:19.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:19 smithi099 ceph-mon[39420]: osdmap e85: 11 total, 11 up, 11 in 2024-02-09T17:22:19.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:19 smithi099 ceph-mon[39420]: Detected new or changed devices on smithi179 2024-02-09T17:22:19.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:19 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:19.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:19 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:19.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:19 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:22:19.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:19 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:22:19.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:19 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:22:19.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:19 smithi099 ceph-mon[39420]: Adjusting osd_memory_target on smithi179 to 7072M 2024-02-09T17:22:19.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:19 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:19.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:19 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:22:19.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:19 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:22:19.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:19 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:20.134 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph orch daemon add osd smithi179:vg_nvme/lv_1 2024-02-09T17:22:20.397 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:22:20.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:20 smithi179 ceph-mon[43975]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-02-09T17:22:20.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:20 smithi162 ceph-mon[44304]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-02-09T17:22:20.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:20 smithi099 ceph-mon[39420]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-02-09T17:22:22.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:22 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:22:22.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:22 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:22:22.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:22 smithi179 ceph-mon[43975]: osdmap e86: 11 total, 11 up, 11 in 2024-02-09T17:22:22.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:22 smithi179 ceph-mon[43975]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-02-09T17:22:22.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:22 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:22:22.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:22 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:22:22.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:22 smithi162 ceph-mon[44304]: osdmap e86: 11 total, 11 up, 11 in 2024-02-09T17:22:22.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:22 smithi162 ceph-mon[44304]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-02-09T17:22:22.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:22 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:22:22.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:22 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:22:22.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:22 smithi099 ceph-mon[39420]: osdmap e86: 11 total, 11 up, 11 in 2024-02-09T17:22:22.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:22 smithi099 ceph-mon[39420]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-02-09T17:22:23.886 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:23 smithi179 ceph-mon[43975]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:22:23.886 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:23 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T17:22:23.886 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:23 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T17:22:23.886 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:23 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:22:23.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:23 smithi162 ceph-mon[44304]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:22:23.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:23 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T17:22:23.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:23 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T17:22:23.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:23 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:22:23.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:23 smithi099 ceph-mon[39420]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:22:23.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:23 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T17:22:23.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:23 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T17:22:23.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:23 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:22:24.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:24 smithi179 ceph-mon[43975]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-02-09T17:22:24.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:24 smithi162 ceph-mon[44304]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-02-09T17:22:24.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:24 smithi099 ceph-mon[39420]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-02-09T17:22:25.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:25 smithi179 ceph-mon[43975]: from='client.? 172.21.15.179:0/4067493486' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d3a3ad8-417a-483f-991c-52549dbe7d73"}]: dispatch 2024-02-09T17:22:25.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:25 smithi179 ceph-mon[43975]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d3a3ad8-417a-483f-991c-52549dbe7d73"}]: dispatch 2024-02-09T17:22:25.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:25 smithi179 ceph-mon[43975]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6d3a3ad8-417a-483f-991c-52549dbe7d73"}]': finished 2024-02-09T17:22:25.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:25 smithi179 ceph-mon[43975]: osdmap e87: 12 total, 11 up, 12 in 2024-02-09T17:22:25.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:25 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-09T17:22:25.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:25 smithi162 ceph-mon[44304]: from='client.? 172.21.15.179:0/4067493486' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d3a3ad8-417a-483f-991c-52549dbe7d73"}]: dispatch 2024-02-09T17:22:25.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:25 smithi162 ceph-mon[44304]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d3a3ad8-417a-483f-991c-52549dbe7d73"}]: dispatch 2024-02-09T17:22:25.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:25 smithi162 ceph-mon[44304]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6d3a3ad8-417a-483f-991c-52549dbe7d73"}]': finished 2024-02-09T17:22:25.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:25 smithi162 ceph-mon[44304]: osdmap e87: 12 total, 11 up, 12 in 2024-02-09T17:22:25.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:25 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-09T17:22:25.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:25 smithi099 ceph-mon[39420]: from='client.? 172.21.15.179:0/4067493486' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d3a3ad8-417a-483f-991c-52549dbe7d73"}]: dispatch 2024-02-09T17:22:25.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:25 smithi099 ceph-mon[39420]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d3a3ad8-417a-483f-991c-52549dbe7d73"}]: dispatch 2024-02-09T17:22:25.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:25 smithi099 ceph-mon[39420]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6d3a3ad8-417a-483f-991c-52549dbe7d73"}]': finished 2024-02-09T17:22:25.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:25 smithi099 ceph-mon[39420]: osdmap e87: 12 total, 11 up, 12 in 2024-02-09T17:22:25.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:25 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-09T17:22:26.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:26 smithi179 ceph-mon[43975]: from='client.? 172.21.15.179:0/2594668418' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T17:22:26.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:26 smithi179 ceph-mon[43975]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-02-09T17:22:26.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:26 smithi162 ceph-mon[44304]: from='client.? 172.21.15.179:0/2594668418' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T17:22:26.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:26 smithi162 ceph-mon[44304]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-02-09T17:22:26.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:26 smithi099 ceph-mon[39420]: from='client.? 172.21.15.179:0/2594668418' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T17:22:26.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:26 smithi099 ceph-mon[39420]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-02-09T17:22:28.663 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {auth=22,config=2,health=12,kv=58,logm=6} crc {auth=3891088204,config=752837470,health=3897446751,kv=1906335068,logm=4086943657}) 2024-02-09T17:22:28.663 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=437533630}) 2024-02-09T17:22:28.663 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1016060523}) 2024-02-09T17:22:28.663 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1398881885}) 2024-02-09T17:22:28.663 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1320221401}) 2024-02-09T17:22:28.664 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3563532348}) 2024-02-09T17:22:28.664 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1279845533}) 2024-02-09T17:22:28.664 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=709429396}) 2024-02-09T17:22:28.664 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3950524718}) 2024-02-09T17:22:28.664 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=115316212}) 2024-02-09T17:22:28.664 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2209752062}) 2024-02-09T17:22:28.664 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=223251183}) 2024-02-09T17:22:28.664 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1572498577}) 2024-02-09T17:22:28.664 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=667803934}) 2024-02-09T17:22:28.664 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2595337133}) 2024-02-09T17:22:28.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3854663127}) 2024-02-09T17:22:28.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=29,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=33} crc {logm=1359283780,mdsmap=3984425268,mgr=1708625883,mgr_command_descs=708342470,mgr_metadata=670763949,mgrstat=4150172400,mon_config_key=1935721327}) 2024-02-09T17:22:28.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1165259736}) 2024-02-09T17:22:28.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=808802728}) 2024-02-09T17:22:28.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=42,monmap=5,osd_metadata=11,osd_pg_creating=1,osdmap=41} crc {mon_config_key=300718405,monmap=423865542,osd_metadata=2714738301,osd_pg_creating=103948301,osdmap=2257923108}) 2024-02-09T17:22:28.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=72} crc {osdmap=1730578044}) 2024-02-09T17:22:28.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {auth=22,config=2,health=12,kv=58,logm=6} crc {auth=3891088204,config=752837470,health=3897446751,kv=1906335068,logm=4086943657}) 2024-02-09T17:22:28.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=437533630}) 2024-02-09T17:22:28.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1016060523}) 2024-02-09T17:22:28.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1398881885}) 2024-02-09T17:22:28.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1320221401}) 2024-02-09T17:22:28.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3563532348}) 2024-02-09T17:22:28.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1279845533}) 2024-02-09T17:22:28.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=709429396}) 2024-02-09T17:22:28.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3950524718}) 2024-02-09T17:22:28.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=115316212}) 2024-02-09T17:22:28.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2209752062}) 2024-02-09T17:22:28.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=223251183}) 2024-02-09T17:22:28.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1572498577}) 2024-02-09T17:22:28.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=667803934}) 2024-02-09T17:22:28.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2595337133}) 2024-02-09T17:22:28.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3854663127}) 2024-02-09T17:22:28.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=29,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=33} crc {logm=1359283780,mdsmap=3984425268,mgr=1708625883,mgr_command_descs=708342470,mgr_metadata=670763949,mgrstat=4150172400,mon_config_key=1935721327}) 2024-02-09T17:22:28.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1165259736}) 2024-02-09T17:22:28.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=808802728}) 2024-02-09T17:22:28.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=42,monmap=5,osd_metadata=11,osd_pg_creating=1,osdmap=41} crc {mon_config_key=300718405,monmap=423865542,osd_metadata=2714738301,osd_pg_creating=103948301,osdmap=2257923108}) 2024-02-09T17:22:28.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=72} crc {osdmap=1730578044}) 2024-02-09T17:22:28.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {auth=22,config=2,health=12,kv=58,logm=6} crc {auth=3891088204,config=752837470,health=3897446751,kv=1906335068,logm=4086943657}) 2024-02-09T17:22:28.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=437533630}) 2024-02-09T17:22:28.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1016060523}) 2024-02-09T17:22:28.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1398881885}) 2024-02-09T17:22:28.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1320221401}) 2024-02-09T17:22:28.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3563532348}) 2024-02-09T17:22:28.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1279845533}) 2024-02-09T17:22:28.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=709429396}) 2024-02-09T17:22:28.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3950524718}) 2024-02-09T17:22:28.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=115316212}) 2024-02-09T17:22:28.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2209752062}) 2024-02-09T17:22:28.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=223251183}) 2024-02-09T17:22:28.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1572498577}) 2024-02-09T17:22:28.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=667803934}) 2024-02-09T17:22:28.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2595337133}) 2024-02-09T17:22:28.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3854663127}) 2024-02-09T17:22:28.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=29,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=33} crc {logm=1359283780,mdsmap=3984425268,mgr=1708625883,mgr_command_descs=708342470,mgr_metadata=670763949,mgrstat=4150172400,mon_config_key=1935721327}) 2024-02-09T17:22:28.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1165259736}) 2024-02-09T17:22:28.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=808802728}) 2024-02-09T17:22:28.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=42,monmap=5,osd_metadata=11,osd_pg_creating=1,osdmap=41} crc {mon_config_key=300718405,monmap=423865542,osd_metadata=2714738301,osd_pg_creating=103948301,osdmap=2257923108}) 2024-02-09T17:22:28.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=72} crc {osdmap=1730578044}) 2024-02-09T17:22:29.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:29 smithi179 ceph-mon[43975]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-02-09T17:22:29.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:29 smithi162 ceph-mon[44304]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-02-09T17:22:29.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:29 smithi099 ceph-mon[39420]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-02-09T17:22:31.663 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:31 smithi179 ceph-mon[43975]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-02-09T17:22:31.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:31 smithi162 ceph-mon[44304]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-02-09T17:22:31.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:31 smithi099 ceph-mon[39420]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-02-09T17:22:32.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:32 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-02-09T17:22:32.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:32 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:22:32.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:32 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-02-09T17:22:32.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:32 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:22:32.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:32 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-02-09T17:22:32.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:32 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:22:33.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:33 smithi179 ceph-mon[43975]: Deploying daemon osd.11 on smithi179 2024-02-09T17:22:33.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:33 smithi179 ceph-mon[43975]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-02-09T17:22:33.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:33 smithi162 ceph-mon[44304]: Deploying daemon osd.11 on smithi179 2024-02-09T17:22:33.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:33 smithi162 ceph-mon[44304]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-02-09T17:22:33.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:33 smithi099 ceph-mon[39420]: Deploying daemon osd.11 on smithi179 2024-02-09T17:22:33.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:33 smithi099 ceph-mon[39420]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-02-09T17:22:35.667 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:35 smithi179 ceph-mon[43975]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-02-09T17:22:35.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:35 smithi162 ceph-mon[44304]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-02-09T17:22:35.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:35 smithi099 ceph-mon[39420]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-02-09T17:22:37.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:36 smithi099 ceph-mon[39420]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-02-09T17:22:37.335 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:36 smithi179 ceph-mon[43975]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-02-09T17:22:37.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:36 smithi162 ceph-mon[44304]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-02-09T17:22:38.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:38 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:38.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:38 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:38.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:38 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:22:38.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:38 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:38.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:38 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:38.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:38 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:22:38.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:38 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:38.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:38 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:38.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:38 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:22:39.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:39 smithi179 ceph-mon[43975]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-02-09T17:22:39.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:39 smithi162 ceph-mon[44304]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-02-09T17:22:39.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:39 smithi099 ceph-mon[39420]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-02-09T17:22:41.206 INFO:teuthology.orchestra.run.smithi179.stdout:Created osd(s) 11 on host 'smithi179' 2024-02-09T17:22:41.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:41 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:41.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:41 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:41.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:41 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:22:41.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:41 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:22:41.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:41 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:41.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:41 smithi179 ceph-mon[43975]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-02-09T17:22:41.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:41 smithi179 ceph-mon[43975]: from='osd.11 [v2:172.21.15.179:6824/2899402373,v1:172.21.15.179:6825/2899402373]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-02-09T17:22:41.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:41 smithi179 ceph-mon[43975]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-02-09T17:22:41.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:41 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:41.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:41 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:41.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:41 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:41.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:41 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:41.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:41 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:22:41.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:41 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:22:41.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:41 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:41.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:41 smithi099 ceph-mon[39420]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-02-09T17:22:41.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:41 smithi099 ceph-mon[39420]: from='osd.11 [v2:172.21.15.179:6824/2899402373,v1:172.21.15.179:6825/2899402373]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-02-09T17:22:41.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:41 smithi099 ceph-mon[39420]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-02-09T17:22:41.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:41 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:41.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:41 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:41.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:41 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:41.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:41 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:41.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:41 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:22:41.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:41 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:22:41.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:41 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:41.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:41 smithi162 ceph-mon[44304]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-02-09T17:22:41.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:41 smithi162 ceph-mon[44304]: from='osd.11 [v2:172.21.15.179:6824/2899402373,v1:172.21.15.179:6825/2899402373]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-02-09T17:22:41.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:41 smithi162 ceph-mon[44304]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-02-09T17:22:41.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:41 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:41.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:41 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:41.962 DEBUG:teuthology.orchestra.run.smithi179:osd.11> sudo journalctl -f -n 0 -u ceph-a69c870a-c76e-11ee-95b7-87774f69a715@osd.11.service 2024-02-09T17:22:41.964 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-02-09T17:22:41.965 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd stat -f json 2024-02-09T17:22:41.996 INFO:journalctl@ceph.osd.11.smithi179.stdout:-- Logs begin at Fri 2024-02-09 17:04:31 UTC. -- 2024-02-09T17:22:42.240 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:22:42.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:42 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:22:42.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:42 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:22:42.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:42 smithi099 ceph-mon[39420]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-02-09T17:22:42.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:42 smithi099 ceph-mon[39420]: osdmap e88: 12 total, 11 up, 12 in 2024-02-09T17:22:42.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:42 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-09T17:22:42.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:42 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:22:42.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:42 smithi099 ceph-mon[39420]: from='osd.11 [v2:172.21.15.179:6824/2899402373,v1:172.21.15.179:6825/2899402373]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-02-09T17:22:42.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:42 smithi099 ceph-mon[39420]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-02-09T17:22:42.672 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 17:22:42 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T17:22:42.238+0000 7f1099e0a700 -1 osd.11 0 waiting for initial osdmap 2024-02-09T17:22:42.672 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 17:22:42 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T17:22:42.238+0000 7f10915a5700 -1 osd.11 0 failed to load OSD map for epoch 32, got 0 bytes 2024-02-09T17:22:42.672 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 17:22:42 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T17:22:42.244+0000 7f1095c04700 -1 osd.11 89 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-09T17:22:42.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:42 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:22:42.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:42 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:22:42.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:42 smithi179 ceph-mon[43975]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-02-09T17:22:42.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:42 smithi179 ceph-mon[43975]: osdmap e88: 12 total, 11 up, 12 in 2024-02-09T17:22:42.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:42 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-09T17:22:42.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:42 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:22:42.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:42 smithi179 ceph-mon[43975]: from='osd.11 [v2:172.21.15.179:6824/2899402373,v1:172.21.15.179:6825/2899402373]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-02-09T17:22:42.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:42 smithi179 ceph-mon[43975]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-02-09T17:22:42.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:42 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:22:42.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:42 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:22:42.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:42 smithi162 ceph-mon[44304]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-02-09T17:22:42.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:42 smithi162 ceph-mon[44304]: osdmap e88: 12 total, 11 up, 12 in 2024-02-09T17:22:42.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:42 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-09T17:22:42.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:42 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:22:42.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:42 smithi162 ceph-mon[44304]: from='osd.11 [v2:172.21.15.179:6824/2899402373,v1:172.21.15.179:6825/2899402373]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-02-09T17:22:42.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:42 smithi162 ceph-mon[44304]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-02-09T17:22:43.469 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:43 smithi179 ceph-mon[43975]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-02-09T17:22:43.469 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:43 smithi179 ceph-mon[43975]: osdmap e89: 12 total, 11 up, 12 in 2024-02-09T17:22:43.469 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:43 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-09T17:22:43.469 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:43 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-09T17:22:43.469 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:43 smithi179 ceph-mon[43975]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-02-09T17:22:43.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:43 smithi099 ceph-mon[39420]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-02-09T17:22:43.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:43 smithi099 ceph-mon[39420]: osdmap e89: 12 total, 11 up, 12 in 2024-02-09T17:22:43.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:43 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-09T17:22:43.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:43 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-09T17:22:43.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:43 smithi099 ceph-mon[39420]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-02-09T17:22:43.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:43 smithi162 ceph-mon[44304]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-02-09T17:22:43.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:43 smithi162 ceph-mon[44304]: osdmap e89: 12 total, 11 up, 12 in 2024-02-09T17:22:43.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:43 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-09T17:22:43.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:43 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-09T17:22:43.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:43 smithi162 ceph-mon[44304]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-02-09T17:22:44.354 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:22:44.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:44 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:22:44.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:44 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:22:44.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:44 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-09T17:22:44.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:44 smithi099 ceph-mon[39420]: osd.11 [v2:172.21.15.179:6824/2899402373,v1:172.21.15.179:6825/2899402373] boot 2024-02-09T17:22:44.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:44 smithi099 ceph-mon[39420]: osdmap e90: 12 total, 12 up, 12 in 2024-02-09T17:22:44.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:44 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-09T17:22:44.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:44 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:22:44.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:44 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:22:44.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:44 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-09T17:22:44.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:44 smithi179 ceph-mon[43975]: osd.11 [v2:172.21.15.179:6824/2899402373,v1:172.21.15.179:6825/2899402373] boot 2024-02-09T17:22:44.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:44 smithi179 ceph-mon[43975]: osdmap e90: 12 total, 12 up, 12 in 2024-02-09T17:22:44.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:44 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-09T17:22:44.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:44 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:22:44.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:44 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:22:44.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:44 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-09T17:22:44.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:44 smithi162 ceph-mon[44304]: osd.11 [v2:172.21.15.179:6824/2899402373,v1:172.21.15.179:6825/2899402373] boot 2024-02-09T17:22:44.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:44 smithi162 ceph-mon[44304]: osdmap e90: 12 total, 12 up, 12 in 2024-02-09T17:22:44.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:44 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-09T17:22:44.963 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":91,"num_osds":12,"num_up_osds":12,"osd_up_since":1707499363,"num_in_osds":12,"osd_in_since":1707499345,"num_remapped_pgs":1} 2024-02-09T17:22:44.964 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd dump --format=json 2024-02-09T17:22:45.230 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:22:45.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:45 smithi179 ceph-mon[43975]: osdmap e91: 12 total, 12 up, 12 in 2024-02-09T17:22:45.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:45 smithi179 ceph-mon[43975]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:22:45.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:45 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/204796116' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-09T17:22:45.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:45 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:45.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:45 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:45.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:45 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:22:45.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:45 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:22:45.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:45 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:22:45.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:45 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:22:45.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:45 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:45.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:45 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:22:45.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:45 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:22:45.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:45 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:45.676 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:45 smithi099 ceph-mon[39420]: osdmap e91: 12 total, 12 up, 12 in 2024-02-09T17:22:45.677 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:45 smithi099 ceph-mon[39420]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:22:45.677 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:45 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/204796116' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-09T17:22:45.677 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:45 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:45.677 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:45 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:45.677 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:45 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:22:45.677 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:45 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:22:45.677 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:45 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:22:45.677 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:45 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:22:45.677 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:45 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:45.677 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:45 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:22:45.677 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:45 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:22:45.677 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:45 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:45.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:45 smithi162 ceph-mon[44304]: osdmap e91: 12 total, 12 up, 12 in 2024-02-09T17:22:45.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:45 smithi162 ceph-mon[44304]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:22:45.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:45 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/204796116' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-09T17:22:45.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:45 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:45.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:45 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:45.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:45 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:22:45.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:45 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:22:45.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:45 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:22:45.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:45 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-02-09T17:22:45.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:45 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:45.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:45 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:22:45.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:45 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:22:45.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:45 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:22:46.348 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:46 smithi099 ceph-mon[39420]: Detected new or changed devices on smithi179 2024-02-09T17:22:46.348 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:46 smithi099 ceph-mon[39420]: Adjusting osd_memory_target on smithi179 to 5304M 2024-02-09T17:22:46.348 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:46 smithi099 ceph-mon[39420]: osdmap e92: 12 total, 12 up, 12 in 2024-02-09T17:22:46.348 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:46 smithi099 ceph-mon[39420]: osdmap e93: 12 total, 12 up, 12 in 2024-02-09T17:22:46.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:46 smithi179 ceph-mon[43975]: Detected new or changed devices on smithi179 2024-02-09T17:22:46.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:46 smithi179 ceph-mon[43975]: Adjusting osd_memory_target on smithi179 to 5304M 2024-02-09T17:22:46.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:46 smithi179 ceph-mon[43975]: osdmap e92: 12 total, 12 up, 12 in 2024-02-09T17:22:46.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:46 smithi179 ceph-mon[43975]: osdmap e93: 12 total, 12 up, 12 in 2024-02-09T17:22:46.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:46 smithi162 ceph-mon[44304]: Detected new or changed devices on smithi179 2024-02-09T17:22:46.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:46 smithi162 ceph-mon[44304]: Adjusting osd_memory_target on smithi179 to 5304M 2024-02-09T17:22:46.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:46 smithi162 ceph-mon[44304]: osdmap e92: 12 total, 12 up, 12 in 2024-02-09T17:22:46.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:46 smithi162 ceph-mon[44304]: osdmap e93: 12 total, 12 up, 12 in 2024-02-09T17:22:47.114 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:22:47.114 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":94,"fsid":"a69c870a-c76e-11ee-95b7-87774f69a715","created":"2024-02-09T17:15:26.113606+0000","modified":"2024-02-09T17:22:46.976599+0000","last_up_change":"2024-02-09T17:22:43.235246+0000","last_in_change":"2024-02-09T17:22:25.344079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-09T17:18:56.552286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"21739aa7-e278-45da-9a3f-867385f01baf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":3859868923},{"type":"v1","addr":"172.21.15.99:6803","nonce":3859868923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":3859868923},{"type":"v1","addr":"172.21.15.99:6805","nonce":3859868923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":3859868923},{"type":"v1","addr":"172.21.15.99:6809","nonce":3859868923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":3859868923},{"type":"v1","addr":"172.21.15.99:6807","nonce":3859868923}]},"public_addr":"172.21.15.99:6803/3859868923","cluster_addr":"172.21.15.99:6805/3859868923","heartbeat_back_addr":"172.21.15.99:6809/3859868923","heartbeat_front_addr":"172.21.15.99:6807/3859868923","state":["exists","up"]},{"osd":1,"uuid":"8eab347c-9065-4de1-93ae-e2eb7e11cab7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":3868328452},{"type":"v1","addr":"172.21.15.99:6811","nonce":3868328452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":3868328452},{"type":"v1","addr":"172.21.15.99:6813","nonce":3868328452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":3868328452},{"type":"v1","addr":"172.21.15.99:6817","nonce":3868328452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":3868328452},{"type":"v1","addr":"172.21.15.99:6815","nonce":3868328452}]},"public_addr":"172.21.15.99:6811/3868328452","cluster_addr":"172.21.15.99:6813/3868328452","heartbeat_back_addr":"172.21.15.99:6817/3868328452","heartbeat_front_addr":"172.21.15.99:6815/3868328452","state":["exists","up"]},{"osd":2,"uuid":"575f4257-7e34-485e-963c-2607b69e1220","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":537506237},{"type":"v1","addr":"172.21.15.99:6819","nonce":537506237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":537506237},{"type":"v1","addr":"172.21.15.99:6821","nonce":537506237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":537506237},{"type":"v1","addr":"172.21.15.99:6825","nonce":537506237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":537506237},{"type":"v1","addr":"172.21.15.99:6823","nonce":537506237}]},"public_addr":"172.21.15.99:6819/537506237","cluster_addr":"172.21.15.99:6821/537506237","heartbeat_back_addr":"172.21.15.99:6825/537506237","heartbeat_front_addr":"172.21.15.99:6823/537506237","state":["exists","up"]},{"osd":3,"uuid":"1a520bcf-81bb-4f1d-80fb-11e98008e429","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":762784793},{"type":"v1","addr":"172.21.15.99:6827","nonce":762784793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":762784793},{"type":"v1","addr":"172.21.15.99:6829","nonce":762784793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":762784793},{"type":"v1","addr":"172.21.15.99:6833","nonce":762784793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":762784793},{"type":"v1","addr":"172.21.15.99:6831","nonce":762784793}]},"public_addr":"172.21.15.99:6827/762784793","cluster_addr":"172.21.15.99:6829/762784793","heartbeat_back_addr":"172.21.15.99:6833/762784793","heartbeat_front_addr":"172.21.15.99:6831/762784793","state":["exists","up"]},{"osd":4,"uuid":"4617c2fe-c9d8-40cd-887b-303e3215bdf8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":4178676504},{"type":"v1","addr":"172.21.15.162:6801","nonce":4178676504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":4178676504},{"type":"v1","addr":"172.21.15.162:6803","nonce":4178676504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":4178676504},{"type":"v1","addr":"172.21.15.162:6807","nonce":4178676504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":4178676504},{"type":"v1","addr":"172.21.15.162:6805","nonce":4178676504}]},"public_addr":"172.21.15.162:6801/4178676504","cluster_addr":"172.21.15.162:6803/4178676504","heartbeat_back_addr":"172.21.15.162:6807/4178676504","heartbeat_front_addr":"172.21.15.162:6805/4178676504","state":["exists","up"]},{"osd":5,"uuid":"da98e3f3-173e-46fe-be8b-1ccbe8b0dabc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1489371339},{"type":"v1","addr":"172.21.15.162:6809","nonce":1489371339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1489371339},{"type":"v1","addr":"172.21.15.162:6811","nonce":1489371339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1489371339},{"type":"v1","addr":"172.21.15.162:6815","nonce":1489371339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1489371339},{"type":"v1","addr":"172.21.15.162:6813","nonce":1489371339}]},"public_addr":"172.21.15.162:6809/1489371339","cluster_addr":"172.21.15.162:6811/1489371339","heartbeat_back_addr":"172.21.15.162:6815/1489371339","heartbeat_front_addr":"172.21.15.162:6813/1489371339","state":["exists","up"]},{"osd":6,"uuid":"ff50382d-5ddf-4cda-bb0c-ff055feeaa40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":821169663},{"type":"v1","addr":"172.21.15.162:6817","nonce":821169663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":821169663},{"type":"v1","addr":"172.21.15.162:6819","nonce":821169663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":821169663},{"type":"v1","addr":"172.21.15.162:6823","nonce":821169663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":821169663},{"type":"v1","addr":"172.21.15.162:6821","nonce":821169663}]},"public_addr":"172.21.15.162:6817/821169663","cluster_addr":"172.21.15.162:6819/821169663","heartbeat_back_addr":"172.21.15.162:6823/821169663","heartbeat_front_addr":"172.21.15.162:6821/821169663","state":["exists","up"]},{"osd":7,"uuid":"a82984dc-5e22-424b-a446-ae2ee71eee19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":965264369},{"type":"v1","addr":"172.21.15.162:6825","nonce":965264369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":965264369},{"type":"v1","addr":"172.21.15.162:6827","nonce":965264369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":965264369},{"type":"v1","addr":"172.21.15.162:6831","nonce":965264369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":965264369},{"type":"v1","addr":"172.21.15.162:6829","nonce":965264369}]},"public_addr":"172.21.15.162:6825/965264369","cluster_addr":"172.21.15.162:6827/965264369","heartbeat_back_addr":"172.21.15.162:6831/965264369","heartbeat_front_addr":"172.21.15.162:6829/965264369","state":["exists","up"]},{"osd":8,"uuid":"63f9d79f-0400-44dc-b388-df00fefb5820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":4219839127},{"type":"v1","addr":"172.21.15.179:6801","nonce":4219839127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":4219839127},{"type":"v1","addr":"172.21.15.179:6803","nonce":4219839127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":4219839127},{"type":"v1","addr":"172.21.15.179:6807","nonce":4219839127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":4219839127},{"type":"v1","addr":"172.21.15.179:6805","nonce":4219839127}]},"public_addr":"172.21.15.179:6801/4219839127","cluster_addr":"172.21.15.179:6803/4219839127","heartbeat_back_addr":"172.21.15.179:6807/4219839127","heartbeat_front_addr":"172.21.15.179:6805/4219839127","state":["exists","up"]},{"osd":9,"uuid":"7d995854-7328-4dfc-b1a9-0edca8d74f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":504427183},{"type":"v1","addr":"172.21.15.179:6809","nonce":504427183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":504427183},{"type":"v1","addr":"172.21.15.179:6811","nonce":504427183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":504427183},{"type":"v1","addr":"172.21.15.179:6815","nonce":504427183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":504427183},{"type":"v1","addr":"172.21.15.179:6813","nonce":504427183}]},"public_addr":"172.21.15.179:6809/504427183","cluster_addr":"172.21.15.179:6811/504427183","heartbeat_back_addr":"172.21.15.179:6815/504427183","heartbeat_front_addr":"172.21.15.179:6813/504427183","state":["exists","up"]},{"osd":10,"uuid":"151fbedd-08e6-46e6-a4e5-a14248399f61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":2403345164},{"type":"v1","addr":"172.21.15.179:6817","nonce":2403345164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":2403345164},{"type":"v1","addr":"172.21.15.179:6819","nonce":2403345164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":2403345164},{"type":"v1","addr":"172.21.15.179:6823","nonce":2403345164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":2403345164},{"type":"v1","addr":"172.21.15.179:6821","nonce":2403345164}]},"public_addr":"172.21.15.179:6817/2403345164","cluster_addr":"172.21.15.179:6819/2403345164","heartbeat_back_addr":"172.21.15.179:6823/2403345164","heartbeat_front_addr":"172.21.15.179:6821/2403345164","state":["exists","up"]},{"osd":11,"uuid":"6d3a3ad8-417a-483f-991c-52549dbe7d73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":90,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2899402373},{"type":"v1","addr":"172.21.15.179:6825","nonce":2899402373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2899402373},{"type":"v1","addr":"172.21.15.179:6827","nonce":2899402373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2899402373},{"type":"v1","addr":"172.21.15.179:6831","nonce":2899402373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2899402373},{"type":"v1","addr":"172.21.15.179:6829","nonce":2899402373}]},"public_addr":"172.21.15.179:6825/2899402373","cluster_addr":"172.21.15.179:6827/2899402373","heartbeat_back_addr":"172.21.15.179:6831/2899402373","heartbeat_front_addr":"172.21.15.179:6829/2899402373","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T17:21:43.002533+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T17:22:20.758805+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T17:21:37.704996+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T17:22:41.081741+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T17:22:07.675834+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T17:21:34.311619+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T17:21:37.035213+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T17:22:04.521484+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T17:21:22.258843+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T17:21:48.379858+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T17:22:14.679271+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T17:22:41.679203+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.99:0/3904172782":"2024-02-10T17:16:24.229365+0000","172.21.15.99:6801/1586034576":"2024-02-10T17:16:24.229365+0000","172.21.15.99:6800/746248609":"2024-02-10T17:15:54.399201+0000","172.21.15.99:0/4092784197":"2024-02-10T17:16:24.229365+0000","172.21.15.99:0/2844598101":"2024-02-10T17:15:54.399201+0000","172.21.15.99:6801/746248609":"2024-02-10T17:15:54.399201+0000","172.21.15.99:0/2528750050":"2024-02-10T17:15:54.399201+0000","172.21.15.99:6800/1586034576":"2024-02-10T17:16:24.229365+0000","172.21.15.99:0/2214140078":"2024-02-10T17:15:54.399201+0000","172.21.15.99:0/4217821965":"2024-02-10T17:16:24.229365+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-09T17:22:47.430 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:47 smithi099 ceph-mon[39420]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:22:47.430 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:47 smithi099 ceph-mon[39420]: osdmap e94: 12 total, 12 up, 12 in 2024-02-09T17:22:47.431 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:47 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/320878799' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T17:22:47.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:47 smithi179 ceph-mon[43975]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:22:47.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:47 smithi179 ceph-mon[43975]: osdmap e94: 12 total, 12 up, 12 in 2024-02-09T17:22:47.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:47 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/320878799' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T17:22:47.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:47 smithi162 ceph-mon[44304]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:22:47.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:47 smithi162 ceph-mon[44304]: osdmap e94: 12 total, 12 up, 12 in 2024-02-09T17:22:47.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:47 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/320878799' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T17:22:47.728 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-09T17:18:56.552286+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-09T17:22:47.729 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-02-09T17:22:47.991 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:22:49.441 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:49 smithi099 ceph-mon[39420]: pgmap v276: 1 pgs: 1 peering; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s, 0 objects/s recovering 2024-02-09T17:22:49.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:49 smithi179 ceph-mon[43975]: pgmap v276: 1 pgs: 1 peering; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s, 0 objects/s recovering 2024-02-09T17:22:49.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:49 smithi162 ceph-mon[44304]: pgmap v276: 1 pgs: 1 peering; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s, 0 objects/s recovering 2024-02-09T17:22:49.900 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 1 2024-02-09T17:22:50.661 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:50 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1856841854' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-09T17:22:50.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:50 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1856841854' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-09T17:22:50.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:50 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1856841854' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-09T17:22:50.685 INFO:tasks.cephadm:Setting up client nodes... 2024-02-09T17:22:50.686 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-09T17:22:50.946 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:22:51.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:51 smithi099 ceph-mon[39420]: pgmap v277: 1 pgs: 1 peering; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s, 0 objects/s recovering 2024-02-09T17:22:51.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:51 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:22:51.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:51 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:22:51.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:51 smithi179 ceph-mon[43975]: pgmap v277: 1 pgs: 1 peering; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s, 0 objects/s recovering 2024-02-09T17:22:51.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:51 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:22:51.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:51 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:22:51.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:51 smithi162 ceph-mon[44304]: pgmap v277: 1 pgs: 1 peering; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s, 0 objects/s recovering 2024-02-09T17:22:51.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:51 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:22:51.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:51 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:22:52.977 INFO:teuthology.orchestra.run.smithi099.stdout:[client.0] 2024-02-09T17:22:52.978 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQBsX8Zld7gbOhAA6IyTgSmbtz+TSWsEqxlApw== 2024-02-09T17:22:53.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:52 smithi099 ceph-mon[39420]: osdmap e95: 12 total, 12 up, 12 in 2024-02-09T17:22:53.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:52 smithi099 ceph-mon[39420]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s, 0 objects/s recovering 2024-02-09T17:22:53.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:52 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3958011648' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-09T17:22:53.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:52 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3958011648' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-09T17:22:53.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:52 smithi179 ceph-mon[43975]: osdmap e95: 12 total, 12 up, 12 in 2024-02-09T17:22:53.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:52 smithi179 ceph-mon[43975]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s, 0 objects/s recovering 2024-02-09T17:22:53.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:52 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3958011648' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-09T17:22:53.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:52 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3958011648' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-09T17:22:53.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:52 smithi162 ceph-mon[44304]: osdmap e95: 12 total, 12 up, 12 in 2024-02-09T17:22:53.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:52 smithi162 ceph-mon[44304]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s, 0 objects/s recovering 2024-02-09T17:22:53.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:52 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3958011648' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-09T17:22:53.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:52 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3958011648' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-09T17:22:53.683 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-02-09T17:22:53.683 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-02-09T17:22:53.683 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-02-09T17:22:53.722 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-09T17:22:53.997 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:22:55.480 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:55 smithi162 ceph-mon[44304]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-09T17:22:55.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:55 smithi179 ceph-mon[43975]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-09T17:22:55.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:55 smithi099 ceph-mon[39420]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-09T17:22:56.038 INFO:teuthology.orchestra.run.smithi162.stdout:[client.1] 2024-02-09T17:22:56.038 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQBwX8ZlQFvvARAAGpFthsydA/Qg+u4cXqxspA== 2024-02-09T17:22:56.664 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:56 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:22:56.664 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:56 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:22:56.664 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:56 smithi162 ceph-mon[44304]: osdmap e96: 12 total, 12 up, 12 in 2024-02-09T17:22:56.664 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:56 smithi162 ceph-mon[44304]: from='client.? 172.21.15.162:0/1333071783' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-09T17:22:56.664 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:56 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-09T17:22:56.665 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:56 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-09T17:22:56.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:56 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:22:56.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:56 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:22:56.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:56 smithi179 ceph-mon[43975]: osdmap e96: 12 total, 12 up, 12 in 2024-02-09T17:22:56.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:56 smithi179 ceph-mon[43975]: from='client.? 172.21.15.162:0/1333071783' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-09T17:22:56.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:56 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-09T17:22:56.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:56 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-09T17:22:56.691 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-02-09T17:22:56.691 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-02-09T17:22:56.691 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-02-09T17:22:56.731 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph auth get-or-create client.2 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-09T17:22:56.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:56 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:22:56.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:56 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:22:56.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:56 smithi099 ceph-mon[39420]: osdmap e96: 12 total, 12 up, 12 in 2024-02-09T17:22:56.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:56 smithi099 ceph-mon[39420]: from='client.? 172.21.15.162:0/1333071783' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-09T17:22:56.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:56 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-09T17:22:56.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:56 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-09T17:22:57.007 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:22:57.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:57 smithi179 ceph-mon[43975]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:22:57.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:57 smithi162 ceph-mon[44304]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:22:57.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:57 smithi099 ceph-mon[39420]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:22:59.333 INFO:teuthology.orchestra.run.smithi179.stdout:[client.2] 2024-02-09T17:22:59.333 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQBzX8ZlOph8ExAApXN+W9C1f/Q/GH+Ozony5A== 2024-02-09T17:22:59.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:59 smithi179 ceph-mon[43975]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:22:59.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:59 smithi179 ceph-mon[43975]: from='client.? 172.21.15.179:0/3600665208' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-09T17:22:59.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:59 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-09T17:22:59.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:22:59 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-09T17:22:59.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:59 smithi162 ceph-mon[44304]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:22:59.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:59 smithi162 ceph-mon[44304]: from='client.? 172.21.15.179:0/3600665208' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-09T17:22:59.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:59 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-09T17:22:59.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:22:59 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-09T17:22:59.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:59 smithi099 ceph-mon[39420]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:22:59.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:59 smithi099 ceph-mon[39420]: from='client.? 172.21.15.179:0/3600665208' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-09T17:22:59.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:59 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-09T17:22:59.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:22:59 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-09T17:23:00.039 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-02-09T17:23:00.039 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/ceph/ceph.client.2.keyring 2024-02-09T17:23:00.039 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-02-09T17:23:00.079 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-02-09T17:23:00.079 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-02-09T17:23:00.079 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph mgr dump --format=json 2024-02-09T17:23:00.352 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:23:00.651 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:00 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:23:00.651 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:00 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:23:00.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:00 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:23:00.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:00 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:23:00.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:00 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:23:00.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:00 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:23:01.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:01 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:23:01.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:01 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:23:01.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:01 smithi099 ceph-mon[39420]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:01.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:01 smithi099 ceph-mon[39420]: osdmap e97: 12 total, 12 up, 12 in 2024-02-09T17:23:01.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:01 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:23:01.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:01 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:23:01.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:01 smithi179 ceph-mon[43975]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:01.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:01 smithi179 ceph-mon[43975]: osdmap e97: 12 total, 12 up, 12 in 2024-02-09T17:23:01.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:01 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:23:01.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:01 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:23:01.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:01 smithi162 ceph-mon[44304]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:01.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:01 smithi162 ceph-mon[44304]: osdmap e97: 12 total, 12 up, 12 in 2024-02-09T17:23:02.287 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:23:02.666 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:02 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3060986629' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-09T17:23:02.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:02 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3060986629' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-09T17:23:02.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:02 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3060986629' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-09T17:23:02.992 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":15,"flags":0,"active_gid":14148,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":1682457041},{"type":"v1","addr":"172.21.15.99:6801","nonce":1682457041}]},"active_addr":"172.21.15.99:6801/1682457041","active_change":"2024-02-09T17:16:24.229481+0000","active_mgr_features":4540701272860131327,"available":true,"standbys":[{"gid":24107,"name":"x","mgr_features":4540701272860131327,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.99:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.99:0","nonce":2110133553}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.99:0","nonce":422221573}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.99:0","nonce":1633142596}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.99:0","nonce":3307096486}]}]} 2024-02-09T17:23:02.996 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-02-09T17:23:02.997 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-02-09T17:23:02.997 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd dump --format=json 2024-02-09T17:23:03.260 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:23:03.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:03 smithi179 ceph-mon[43975]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:03.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:03 smithi162 ceph-mon[44304]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:03.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:03 smithi099 ceph-mon[39420]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:04.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:04 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:23:04.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:04 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:23:04.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:04 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:23:04.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:04 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:23:04.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:04 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:23:04.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:04 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:23:05.307 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:23:05.307 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":98,"fsid":"a69c870a-c76e-11ee-95b7-87774f69a715","created":"2024-02-09T17:15:26.113606+0000","modified":"2024-02-09T17:23:04.380468+0000","last_up_change":"2024-02-09T17:22:43.235246+0000","last_in_change":"2024-02-09T17:22:25.344079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-09T17:18:56.552286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"21739aa7-e278-45da-9a3f-867385f01baf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":3859868923},{"type":"v1","addr":"172.21.15.99:6803","nonce":3859868923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":3859868923},{"type":"v1","addr":"172.21.15.99:6805","nonce":3859868923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":3859868923},{"type":"v1","addr":"172.21.15.99:6809","nonce":3859868923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":3859868923},{"type":"v1","addr":"172.21.15.99:6807","nonce":3859868923}]},"public_addr":"172.21.15.99:6803/3859868923","cluster_addr":"172.21.15.99:6805/3859868923","heartbeat_back_addr":"172.21.15.99:6809/3859868923","heartbeat_front_addr":"172.21.15.99:6807/3859868923","state":["exists","up"]},{"osd":1,"uuid":"8eab347c-9065-4de1-93ae-e2eb7e11cab7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":3868328452},{"type":"v1","addr":"172.21.15.99:6811","nonce":3868328452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":3868328452},{"type":"v1","addr":"172.21.15.99:6813","nonce":3868328452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":3868328452},{"type":"v1","addr":"172.21.15.99:6817","nonce":3868328452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":3868328452},{"type":"v1","addr":"172.21.15.99:6815","nonce":3868328452}]},"public_addr":"172.21.15.99:6811/3868328452","cluster_addr":"172.21.15.99:6813/3868328452","heartbeat_back_addr":"172.21.15.99:6817/3868328452","heartbeat_front_addr":"172.21.15.99:6815/3868328452","state":["exists","up"]},{"osd":2,"uuid":"575f4257-7e34-485e-963c-2607b69e1220","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":537506237},{"type":"v1","addr":"172.21.15.99:6819","nonce":537506237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":537506237},{"type":"v1","addr":"172.21.15.99:6821","nonce":537506237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":537506237},{"type":"v1","addr":"172.21.15.99:6825","nonce":537506237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":537506237},{"type":"v1","addr":"172.21.15.99:6823","nonce":537506237}]},"public_addr":"172.21.15.99:6819/537506237","cluster_addr":"172.21.15.99:6821/537506237","heartbeat_back_addr":"172.21.15.99:6825/537506237","heartbeat_front_addr":"172.21.15.99:6823/537506237","state":["exists","up"]},{"osd":3,"uuid":"1a520bcf-81bb-4f1d-80fb-11e98008e429","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":762784793},{"type":"v1","addr":"172.21.15.99:6827","nonce":762784793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":762784793},{"type":"v1","addr":"172.21.15.99:6829","nonce":762784793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":762784793},{"type":"v1","addr":"172.21.15.99:6833","nonce":762784793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":762784793},{"type":"v1","addr":"172.21.15.99:6831","nonce":762784793}]},"public_addr":"172.21.15.99:6827/762784793","cluster_addr":"172.21.15.99:6829/762784793","heartbeat_back_addr":"172.21.15.99:6833/762784793","heartbeat_front_addr":"172.21.15.99:6831/762784793","state":["exists","up"]},{"osd":4,"uuid":"4617c2fe-c9d8-40cd-887b-303e3215bdf8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":4178676504},{"type":"v1","addr":"172.21.15.162:6801","nonce":4178676504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":4178676504},{"type":"v1","addr":"172.21.15.162:6803","nonce":4178676504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":4178676504},{"type":"v1","addr":"172.21.15.162:6807","nonce":4178676504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":4178676504},{"type":"v1","addr":"172.21.15.162:6805","nonce":4178676504}]},"public_addr":"172.21.15.162:6801/4178676504","cluster_addr":"172.21.15.162:6803/4178676504","heartbeat_back_addr":"172.21.15.162:6807/4178676504","heartbeat_front_addr":"172.21.15.162:6805/4178676504","state":["exists","up"]},{"osd":5,"uuid":"da98e3f3-173e-46fe-be8b-1ccbe8b0dabc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1489371339},{"type":"v1","addr":"172.21.15.162:6809","nonce":1489371339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1489371339},{"type":"v1","addr":"172.21.15.162:6811","nonce":1489371339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1489371339},{"type":"v1","addr":"172.21.15.162:6815","nonce":1489371339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1489371339},{"type":"v1","addr":"172.21.15.162:6813","nonce":1489371339}]},"public_addr":"172.21.15.162:6809/1489371339","cluster_addr":"172.21.15.162:6811/1489371339","heartbeat_back_addr":"172.21.15.162:6815/1489371339","heartbeat_front_addr":"172.21.15.162:6813/1489371339","state":["exists","up"]},{"osd":6,"uuid":"ff50382d-5ddf-4cda-bb0c-ff055feeaa40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":821169663},{"type":"v1","addr":"172.21.15.162:6817","nonce":821169663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":821169663},{"type":"v1","addr":"172.21.15.162:6819","nonce":821169663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":821169663},{"type":"v1","addr":"172.21.15.162:6823","nonce":821169663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":821169663},{"type":"v1","addr":"172.21.15.162:6821","nonce":821169663}]},"public_addr":"172.21.15.162:6817/821169663","cluster_addr":"172.21.15.162:6819/821169663","heartbeat_back_addr":"172.21.15.162:6823/821169663","heartbeat_front_addr":"172.21.15.162:6821/821169663","state":["exists","up"]},{"osd":7,"uuid":"a82984dc-5e22-424b-a446-ae2ee71eee19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":965264369},{"type":"v1","addr":"172.21.15.162:6825","nonce":965264369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":965264369},{"type":"v1","addr":"172.21.15.162:6827","nonce":965264369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":965264369},{"type":"v1","addr":"172.21.15.162:6831","nonce":965264369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":965264369},{"type":"v1","addr":"172.21.15.162:6829","nonce":965264369}]},"public_addr":"172.21.15.162:6825/965264369","cluster_addr":"172.21.15.162:6827/965264369","heartbeat_back_addr":"172.21.15.162:6831/965264369","heartbeat_front_addr":"172.21.15.162:6829/965264369","state":["exists","up"]},{"osd":8,"uuid":"63f9d79f-0400-44dc-b388-df00fefb5820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":4219839127},{"type":"v1","addr":"172.21.15.179:6801","nonce":4219839127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":4219839127},{"type":"v1","addr":"172.21.15.179:6803","nonce":4219839127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":4219839127},{"type":"v1","addr":"172.21.15.179:6807","nonce":4219839127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":4219839127},{"type":"v1","addr":"172.21.15.179:6805","nonce":4219839127}]},"public_addr":"172.21.15.179:6801/4219839127","cluster_addr":"172.21.15.179:6803/4219839127","heartbeat_back_addr":"172.21.15.179:6807/4219839127","heartbeat_front_addr":"172.21.15.179:6805/4219839127","state":["exists","up"]},{"osd":9,"uuid":"7d995854-7328-4dfc-b1a9-0edca8d74f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":504427183},{"type":"v1","addr":"172.21.15.179:6809","nonce":504427183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":504427183},{"type":"v1","addr":"172.21.15.179:6811","nonce":504427183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":504427183},{"type":"v1","addr":"172.21.15.179:6815","nonce":504427183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":504427183},{"type":"v1","addr":"172.21.15.179:6813","nonce":504427183}]},"public_addr":"172.21.15.179:6809/504427183","cluster_addr":"172.21.15.179:6811/504427183","heartbeat_back_addr":"172.21.15.179:6815/504427183","heartbeat_front_addr":"172.21.15.179:6813/504427183","state":["exists","up"]},{"osd":10,"uuid":"151fbedd-08e6-46e6-a4e5-a14248399f61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":2403345164},{"type":"v1","addr":"172.21.15.179:6817","nonce":2403345164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":2403345164},{"type":"v1","addr":"172.21.15.179:6819","nonce":2403345164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":2403345164},{"type":"v1","addr":"172.21.15.179:6823","nonce":2403345164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":2403345164},{"type":"v1","addr":"172.21.15.179:6821","nonce":2403345164}]},"public_addr":"172.21.15.179:6817/2403345164","cluster_addr":"172.21.15.179:6819/2403345164","heartbeat_back_addr":"172.21.15.179:6823/2403345164","heartbeat_front_addr":"172.21.15.179:6821/2403345164","state":["exists","up"]},{"osd":11,"uuid":"6d3a3ad8-417a-483f-991c-52549dbe7d73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":90,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2899402373},{"type":"v1","addr":"172.21.15.179:6825","nonce":2899402373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2899402373},{"type":"v1","addr":"172.21.15.179:6827","nonce":2899402373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2899402373},{"type":"v1","addr":"172.21.15.179:6831","nonce":2899402373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2899402373},{"type":"v1","addr":"172.21.15.179:6829","nonce":2899402373}]},"public_addr":"172.21.15.179:6825/2899402373","cluster_addr":"172.21.15.179:6827/2899402373","heartbeat_back_addr":"172.21.15.179:6831/2899402373","heartbeat_front_addr":"172.21.15.179:6829/2899402373","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T17:23:00.068304+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T17:22:20.758805+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T17:23:03.775899+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T17:22:41.081741+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T17:22:07.675834+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T17:22:59.909988+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T17:22:54.927073+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T17:22:04.521484+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T17:22:51.006735+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T17:21:48.379858+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T17:22:14.679271+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T17:22:41.679203+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.99:0/3904172782":"2024-02-10T17:16:24.229365+0000","172.21.15.99:6801/1586034576":"2024-02-10T17:16:24.229365+0000","172.21.15.99:6800/746248609":"2024-02-10T17:15:54.399201+0000","172.21.15.99:0/4092784197":"2024-02-10T17:16:24.229365+0000","172.21.15.99:0/2844598101":"2024-02-10T17:15:54.399201+0000","172.21.15.99:6801/746248609":"2024-02-10T17:15:54.399201+0000","172.21.15.99:0/2528750050":"2024-02-10T17:15:54.399201+0000","172.21.15.99:6800/1586034576":"2024-02-10T17:16:24.229365+0000","172.21.15.99:0/2214140078":"2024-02-10T17:15:54.399201+0000","172.21.15.99:0/4217821965":"2024-02-10T17:16:24.229365+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-09T17:23:05.623 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:05 smithi099 ceph-mon[39420]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:05.623 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:05 smithi099 ceph-mon[39420]: osdmap e98: 12 total, 12 up, 12 in 2024-02-09T17:23:05.623 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:05 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4048020052' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T17:23:05.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:05 smithi179 ceph-mon[43975]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:05.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:05 smithi179 ceph-mon[43975]: osdmap e98: 12 total, 12 up, 12 in 2024-02-09T17:23:05.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:05 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4048020052' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T17:23:05.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:05 smithi162 ceph-mon[44304]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:05.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:05 smithi162 ceph-mon[44304]: osdmap e98: 12 total, 12 up, 12 in 2024-02-09T17:23:05.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:05 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4048020052' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T17:23:05.993 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-02-09T17:23:05.994 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd dump --format=json 2024-02-09T17:23:06.259 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:23:07.629 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:07 smithi099 ceph-mon[39420]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:07.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:07 smithi179 ceph-mon[43975]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:07.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:07 smithi162 ceph-mon[44304]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:08.360 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:23:08.361 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":98,"fsid":"a69c870a-c76e-11ee-95b7-87774f69a715","created":"2024-02-09T17:15:26.113606+0000","modified":"2024-02-09T17:23:04.380468+0000","last_up_change":"2024-02-09T17:22:43.235246+0000","last_in_change":"2024-02-09T17:22:25.344079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-09T17:18:56.552286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"21739aa7-e278-45da-9a3f-867385f01baf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":3859868923},{"type":"v1","addr":"172.21.15.99:6803","nonce":3859868923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":3859868923},{"type":"v1","addr":"172.21.15.99:6805","nonce":3859868923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":3859868923},{"type":"v1","addr":"172.21.15.99:6809","nonce":3859868923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":3859868923},{"type":"v1","addr":"172.21.15.99:6807","nonce":3859868923}]},"public_addr":"172.21.15.99:6803/3859868923","cluster_addr":"172.21.15.99:6805/3859868923","heartbeat_back_addr":"172.21.15.99:6809/3859868923","heartbeat_front_addr":"172.21.15.99:6807/3859868923","state":["exists","up"]},{"osd":1,"uuid":"8eab347c-9065-4de1-93ae-e2eb7e11cab7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":3868328452},{"type":"v1","addr":"172.21.15.99:6811","nonce":3868328452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":3868328452},{"type":"v1","addr":"172.21.15.99:6813","nonce":3868328452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":3868328452},{"type":"v1","addr":"172.21.15.99:6817","nonce":3868328452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":3868328452},{"type":"v1","addr":"172.21.15.99:6815","nonce":3868328452}]},"public_addr":"172.21.15.99:6811/3868328452","cluster_addr":"172.21.15.99:6813/3868328452","heartbeat_back_addr":"172.21.15.99:6817/3868328452","heartbeat_front_addr":"172.21.15.99:6815/3868328452","state":["exists","up"]},{"osd":2,"uuid":"575f4257-7e34-485e-963c-2607b69e1220","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":537506237},{"type":"v1","addr":"172.21.15.99:6819","nonce":537506237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":537506237},{"type":"v1","addr":"172.21.15.99:6821","nonce":537506237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":537506237},{"type":"v1","addr":"172.21.15.99:6825","nonce":537506237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":537506237},{"type":"v1","addr":"172.21.15.99:6823","nonce":537506237}]},"public_addr":"172.21.15.99:6819/537506237","cluster_addr":"172.21.15.99:6821/537506237","heartbeat_back_addr":"172.21.15.99:6825/537506237","heartbeat_front_addr":"172.21.15.99:6823/537506237","state":["exists","up"]},{"osd":3,"uuid":"1a520bcf-81bb-4f1d-80fb-11e98008e429","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":762784793},{"type":"v1","addr":"172.21.15.99:6827","nonce":762784793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":762784793},{"type":"v1","addr":"172.21.15.99:6829","nonce":762784793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":762784793},{"type":"v1","addr":"172.21.15.99:6833","nonce":762784793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":762784793},{"type":"v1","addr":"172.21.15.99:6831","nonce":762784793}]},"public_addr":"172.21.15.99:6827/762784793","cluster_addr":"172.21.15.99:6829/762784793","heartbeat_back_addr":"172.21.15.99:6833/762784793","heartbeat_front_addr":"172.21.15.99:6831/762784793","state":["exists","up"]},{"osd":4,"uuid":"4617c2fe-c9d8-40cd-887b-303e3215bdf8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":4178676504},{"type":"v1","addr":"172.21.15.162:6801","nonce":4178676504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":4178676504},{"type":"v1","addr":"172.21.15.162:6803","nonce":4178676504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":4178676504},{"type":"v1","addr":"172.21.15.162:6807","nonce":4178676504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":4178676504},{"type":"v1","addr":"172.21.15.162:6805","nonce":4178676504}]},"public_addr":"172.21.15.162:6801/4178676504","cluster_addr":"172.21.15.162:6803/4178676504","heartbeat_back_addr":"172.21.15.162:6807/4178676504","heartbeat_front_addr":"172.21.15.162:6805/4178676504","state":["exists","up"]},{"osd":5,"uuid":"da98e3f3-173e-46fe-be8b-1ccbe8b0dabc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1489371339},{"type":"v1","addr":"172.21.15.162:6809","nonce":1489371339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1489371339},{"type":"v1","addr":"172.21.15.162:6811","nonce":1489371339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1489371339},{"type":"v1","addr":"172.21.15.162:6815","nonce":1489371339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1489371339},{"type":"v1","addr":"172.21.15.162:6813","nonce":1489371339}]},"public_addr":"172.21.15.162:6809/1489371339","cluster_addr":"172.21.15.162:6811/1489371339","heartbeat_back_addr":"172.21.15.162:6815/1489371339","heartbeat_front_addr":"172.21.15.162:6813/1489371339","state":["exists","up"]},{"osd":6,"uuid":"ff50382d-5ddf-4cda-bb0c-ff055feeaa40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":821169663},{"type":"v1","addr":"172.21.15.162:6817","nonce":821169663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":821169663},{"type":"v1","addr":"172.21.15.162:6819","nonce":821169663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":821169663},{"type":"v1","addr":"172.21.15.162:6823","nonce":821169663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":821169663},{"type":"v1","addr":"172.21.15.162:6821","nonce":821169663}]},"public_addr":"172.21.15.162:6817/821169663","cluster_addr":"172.21.15.162:6819/821169663","heartbeat_back_addr":"172.21.15.162:6823/821169663","heartbeat_front_addr":"172.21.15.162:6821/821169663","state":["exists","up"]},{"osd":7,"uuid":"a82984dc-5e22-424b-a446-ae2ee71eee19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":965264369},{"type":"v1","addr":"172.21.15.162:6825","nonce":965264369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":965264369},{"type":"v1","addr":"172.21.15.162:6827","nonce":965264369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":965264369},{"type":"v1","addr":"172.21.15.162:6831","nonce":965264369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":965264369},{"type":"v1","addr":"172.21.15.162:6829","nonce":965264369}]},"public_addr":"172.21.15.162:6825/965264369","cluster_addr":"172.21.15.162:6827/965264369","heartbeat_back_addr":"172.21.15.162:6831/965264369","heartbeat_front_addr":"172.21.15.162:6829/965264369","state":["exists","up"]},{"osd":8,"uuid":"63f9d79f-0400-44dc-b388-df00fefb5820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":4219839127},{"type":"v1","addr":"172.21.15.179:6801","nonce":4219839127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":4219839127},{"type":"v1","addr":"172.21.15.179:6803","nonce":4219839127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":4219839127},{"type":"v1","addr":"172.21.15.179:6807","nonce":4219839127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":4219839127},{"type":"v1","addr":"172.21.15.179:6805","nonce":4219839127}]},"public_addr":"172.21.15.179:6801/4219839127","cluster_addr":"172.21.15.179:6803/4219839127","heartbeat_back_addr":"172.21.15.179:6807/4219839127","heartbeat_front_addr":"172.21.15.179:6805/4219839127","state":["exists","up"]},{"osd":9,"uuid":"7d995854-7328-4dfc-b1a9-0edca8d74f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":504427183},{"type":"v1","addr":"172.21.15.179:6809","nonce":504427183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":504427183},{"type":"v1","addr":"172.21.15.179:6811","nonce":504427183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":504427183},{"type":"v1","addr":"172.21.15.179:6815","nonce":504427183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":504427183},{"type":"v1","addr":"172.21.15.179:6813","nonce":504427183}]},"public_addr":"172.21.15.179:6809/504427183","cluster_addr":"172.21.15.179:6811/504427183","heartbeat_back_addr":"172.21.15.179:6815/504427183","heartbeat_front_addr":"172.21.15.179:6813/504427183","state":["exists","up"]},{"osd":10,"uuid":"151fbedd-08e6-46e6-a4e5-a14248399f61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":2403345164},{"type":"v1","addr":"172.21.15.179:6817","nonce":2403345164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":2403345164},{"type":"v1","addr":"172.21.15.179:6819","nonce":2403345164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":2403345164},{"type":"v1","addr":"172.21.15.179:6823","nonce":2403345164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":2403345164},{"type":"v1","addr":"172.21.15.179:6821","nonce":2403345164}]},"public_addr":"172.21.15.179:6817/2403345164","cluster_addr":"172.21.15.179:6819/2403345164","heartbeat_back_addr":"172.21.15.179:6823/2403345164","heartbeat_front_addr":"172.21.15.179:6821/2403345164","state":["exists","up"]},{"osd":11,"uuid":"6d3a3ad8-417a-483f-991c-52549dbe7d73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":90,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2899402373},{"type":"v1","addr":"172.21.15.179:6825","nonce":2899402373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2899402373},{"type":"v1","addr":"172.21.15.179:6827","nonce":2899402373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2899402373},{"type":"v1","addr":"172.21.15.179:6831","nonce":2899402373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2899402373},{"type":"v1","addr":"172.21.15.179:6829","nonce":2899402373}]},"public_addr":"172.21.15.179:6825/2899402373","cluster_addr":"172.21.15.179:6827/2899402373","heartbeat_back_addr":"172.21.15.179:6831/2899402373","heartbeat_front_addr":"172.21.15.179:6829/2899402373","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T17:23:00.068304+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T17:22:20.758805+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T17:23:03.775899+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T17:22:41.081741+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T17:22:07.675834+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T17:22:59.909988+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T17:22:54.927073+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T17:22:04.521484+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T17:22:51.006735+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T17:21:48.379858+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T17:22:14.679271+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T17:22:41.679203+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.99:0/3904172782":"2024-02-10T17:16:24.229365+0000","172.21.15.99:6801/1586034576":"2024-02-10T17:16:24.229365+0000","172.21.15.99:6800/746248609":"2024-02-10T17:15:54.399201+0000","172.21.15.99:0/4092784197":"2024-02-10T17:16:24.229365+0000","172.21.15.99:0/2844598101":"2024-02-10T17:15:54.399201+0000","172.21.15.99:6801/746248609":"2024-02-10T17:15:54.399201+0000","172.21.15.99:0/2528750050":"2024-02-10T17:15:54.399201+0000","172.21.15.99:6800/1586034576":"2024-02-10T17:16:24.229365+0000","172.21.15.99:0/2214140078":"2024-02-10T17:15:54.399201+0000","172.21.15.99:0/4217821965":"2024-02-10T17:16:24.229365+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-09T17:23:08.655 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:08 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2420201926' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T17:23:08.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:08 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2420201926' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T17:23:08.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:08 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2420201926' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T17:23:09.098 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-02-09T17:23:09.099 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-02-09T17:23:09.099 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-02-09T17:23:09.100 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-02-09T17:23:09.100 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-02-09T17:23:09.100 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-02-09T17:23:09.101 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-02-09T17:23:09.101 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-02-09T17:23:09.102 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-02-09T17:23:09.102 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-02-09T17:23:09.103 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-02-09T17:23:09.103 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-02-09T17:23:09.441 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:09 smithi099 ceph-mon[39420]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:09.800 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:23:09.840 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:23:09.848 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:23:09.878 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:23:09.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:09 smithi179 ceph-mon[43975]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:09.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:09 smithi162 ceph-mon[44304]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:10.304 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:23:10.305 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:23:10.305 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:23:10.305 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:23:10.311 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:23:10.311 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:23:10.315 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:23:11.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:11 smithi099 ceph-mon[39420]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:11.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:11 smithi179 ceph-mon[43975]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:11.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:11 smithi162 ceph-mon[44304]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:13.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:13 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:23:13.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:13 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:23:13.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:13 smithi099 ceph-mon[39420]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:13.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:13 smithi099 ceph-mon[39420]: osdmap e99: 12 total, 12 up, 12 in 2024-02-09T17:23:13.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:13 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:23:13.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:13 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:23:13.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:13 smithi179 ceph-mon[43975]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:13.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:13 smithi179 ceph-mon[43975]: osdmap e99: 12 total, 12 up, 12 in 2024-02-09T17:23:13.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:13 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:23:13.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:13 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:23:13.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:13 smithi162 ceph-mon[44304]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:13.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:13 smithi162 ceph-mon[44304]: osdmap e99: 12 total, 12 up, 12 in 2024-02-09T17:23:15.048 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:23:15.560 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:15 smithi099 ceph-mon[39420]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:15.560 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:15 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:23:15.561 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:15 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:23:15.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:15 smithi179 ceph-mon[43975]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:15.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:15 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:23:15.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:15 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:23:15.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:15 smithi162 ceph-mon[44304]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:15.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:15 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:23:15.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:15 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:23:16.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:16 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:23:16.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:16 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:23:16.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:16 smithi099 ceph-mon[39420]: osdmap e100: 12 total, 12 up, 12 in 2024-02-09T17:23:16.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:16 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:23:16.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:16 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:23:16.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:16 smithi179 ceph-mon[43975]: osdmap e100: 12 total, 12 up, 12 in 2024-02-09T17:23:16.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:16 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:23:16.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:16 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:23:16.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:16 smithi162 ceph-mon[44304]: osdmap e100: 12 total, 12 up, 12 in 2024-02-09T17:23:17.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:17 smithi099 ceph-mon[39420]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:17.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:17 smithi179 ceph-mon[43975]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:17.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:17 smithi162 ceph-mon[44304]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:19.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:19 smithi099 ceph-mon[39420]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:19.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:19 smithi179 ceph-mon[43975]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:19.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:19 smithi162 ceph-mon[44304]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:20.017 INFO:teuthology.orchestra.run.smithi099.stdout:34359738434 2024-02-09T17:23:20.018 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-02-09T17:23:21.699 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:21 smithi099 ceph-mon[39420]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:21.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:21 smithi179 ceph-mon[43975]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:21.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:21 smithi162 ceph-mon[44304]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:23.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:23 smithi099 ceph-mon[39420]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:23.773 INFO:teuthology.orchestra.run.smithi099.stdout:115964117044 2024-02-09T17:23:23.773 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-02-09T17:23:23.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:23 smithi179 ceph-mon[43975]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:23.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:23 smithi162 ceph-mon[44304]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:25.149 INFO:teuthology.orchestra.run.smithi099.stdout:386547056651 2024-02-09T17:23:25.150 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-02-09T17:23:25.708 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:25 smithi099 ceph-mon[39420]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:25.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:25 smithi179 ceph-mon[43975]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:25.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:25 smithi162 ceph-mon[44304]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:27.692 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:27 smithi099 ceph-mon[39420]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:27.692 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:27 smithi099 ceph-mon[39420]: 1.0 scrub starts 2024-02-09T17:23:27.692 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:27 smithi099 ceph-mon[39420]: 1.0 scrub ok 2024-02-09T17:23:27.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:27 smithi179 ceph-mon[43975]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:27.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:27 smithi179 ceph-mon[43975]: 1.0 scrub starts 2024-02-09T17:23:27.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:27 smithi179 ceph-mon[43975]: 1.0 scrub ok 2024-02-09T17:23:27.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:27 smithi162 ceph-mon[44304]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:27.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:27 smithi162 ceph-mon[44304]: 1.0 scrub starts 2024-02-09T17:23:27.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:27 smithi162 ceph-mon[44304]: 1.0 scrub ok 2024-02-09T17:23:28.497 INFO:teuthology.orchestra.run.smithi099.stdout:210453397542 2024-02-09T17:23:28.497 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-02-09T17:23:29.169 INFO:teuthology.orchestra.run.smithi099.stdout:55834574911 2024-02-09T17:23:29.169 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-02-09T17:23:29.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:29 smithi099 ceph-mon[39420]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:29.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:29 smithi179 ceph-mon[43975]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:29.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:29 smithi162 ceph-mon[44304]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:30.737 INFO:teuthology.orchestra.run.smithi099.stdout:360777252881 2024-02-09T17:23:30.738 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-02-09T17:23:31.390 INFO:teuthology.orchestra.run.smithi099.stdout:330712481814 2024-02-09T17:23:31.391 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-02-09T17:23:31.670 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:31 smithi099 ceph-mon[39420]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:31.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:31 smithi179 ceph-mon[43975]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:31.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:31 smithi162 ceph-mon[44304]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:32.816 INFO:teuthology.orchestra.run.smithi099.stdout:244813135904 2024-02-09T17:23:32.816 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-02-09T17:23:32.850 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:23:33.303 INFO:teuthology.orchestra.run.smithi099.stdout:77309411387 2024-02-09T17:23:33.303 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-02-09T17:23:33.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:33 smithi099 ceph-mon[39420]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:33.805 INFO:teuthology.orchestra.run.smithi099.stdout:137438953520 2024-02-09T17:23:33.806 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-02-09T17:23:33.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:33 smithi179 ceph-mon[43975]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:33.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:33 smithi162 ceph-mon[44304]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:34.059 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:23:34.059 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:23:34.062 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:23:34.066 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:23:35.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:35 smithi099 ceph-mon[39420]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:35.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:35 smithi179 ceph-mon[43975]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:35.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:35 smithi162 ceph-mon[44304]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:37.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:36 smithi099 ceph-mon[39420]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:37.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:36 smithi179 ceph-mon[43975]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:37.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:36 smithi162 ceph-mon[44304]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:37.513 INFO:teuthology.orchestra.run.smithi099.stdout:283467841565 2024-02-09T17:23:37.514 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-02-09T17:23:38.027 INFO:teuthology.orchestra.run.smithi099.stdout:171798691884 2024-02-09T17:23:38.027 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-02-09T17:23:39.078 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:23:39.080 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:23:39.417 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:38 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:23:39.417 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:38 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:23:39.418 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:38 smithi099 ceph-mon[39420]: osdmap e101: 12 total, 12 up, 12 in 2024-02-09T17:23:39.418 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:38 smithi099 ceph-mon[39420]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:39.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:38 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:23:39.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:38 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:23:39.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:38 smithi179 ceph-mon[43975]: osdmap e101: 12 total, 12 up, 12 in 2024-02-09T17:23:39.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:38 smithi179 ceph-mon[43975]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:39.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:38 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:23:39.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:38 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:23:39.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:38 smithi162 ceph-mon[44304]: osdmap e101: 12 total, 12 up, 12 in 2024-02-09T17:23:39.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:38 smithi162 ceph-mon[44304]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:41.350 INFO:teuthology.orchestra.run.smithi099.stdout:386547056655 2024-02-09T17:23:41.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:41 smithi099 ceph-mon[39420]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:41.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:41 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4033930831' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-02-09T17:23:41.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:41 smithi179 ceph-mon[43975]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:41.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:41 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4033930831' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-02-09T17:23:41.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:41 smithi162 ceph-mon[44304]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:41.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:41 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4033930831' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-02-09T17:23:41.830 INFO:teuthology.orchestra.run.smithi099.stdout:210453397544 2024-02-09T17:23:42.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:42 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3512137206' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-09T17:23:42.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:42 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3512137206' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-09T17:23:42.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:42 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3512137206' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-09T17:23:42.959 INFO:teuthology.orchestra.run.smithi099.stdout:55834574914 2024-02-09T17:23:43.243 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:23:43.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:43 smithi179 ceph-mon[43975]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:43.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:43 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/680040553' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-09T17:23:43.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:43 smithi162 ceph-mon[44304]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:43.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:43 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/680040553' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-09T17:23:43.681 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:43 smithi099 ceph-mon[39420]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:43.681 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:43 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/680040553' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-09T17:23:44.266 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574911 got 55834574914 for osd.1 2024-02-09T17:23:44.267 DEBUG:teuthology.parallel:result is None 2024-02-09T17:23:44.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:44 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/984760714' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-02-09T17:23:44.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:44 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/984760714' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-02-09T17:23:44.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:44 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/984760714' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-02-09T17:23:44.992 INFO:tasks.cephadm.ceph_manager.ceph:need seq 386547056651 got 386547056655 for osd.11 2024-02-09T17:23:44.992 DEBUG:teuthology.parallel:result is None 2024-02-09T17:23:45.328 INFO:tasks.cephadm.ceph_manager.ceph:need seq 210453397542 got 210453397544 for osd.6 2024-02-09T17:23:45.329 DEBUG:teuthology.parallel:result is None 2024-02-09T17:23:45.423 INFO:teuthology.orchestra.run.smithi099.stdout:360777252884 2024-02-09T17:23:45.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:45 smithi179 ceph-mon[43975]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:45.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:45 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:23:45.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:45 smithi162 ceph-mon[44304]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:45.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:45 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:23:45.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:45 smithi099 ceph-mon[39420]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:45.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:45 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:23:45.862 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:23:45.863 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:23:45.863 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:23:45.863 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:23:46.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:46 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:23:46.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:46 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:23:46.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:46 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:23:46.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:46 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:23:46.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:46 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:23:46.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:46 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:23:46.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:46 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:23:46.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:46 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:23:46.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:46 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:23:47.328 INFO:tasks.cephadm.ceph_manager.ceph:need seq 360777252881 got 360777252884 for osd.10 2024-02-09T17:23:47.328 DEBUG:teuthology.parallel:result is None 2024-02-09T17:23:47.558 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:47 smithi099 ceph-mon[39420]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:47.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:47 smithi179 ceph-mon[43975]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:47.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:47 smithi162 ceph-mon[44304]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:49.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:49 smithi179 ceph-mon[43975]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:49.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:49 smithi162 ceph-mon[44304]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:49.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:49 smithi099 ceph-mon[39420]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:50.361 INFO:teuthology.orchestra.run.smithi099.stdout:115964117049 2024-02-09T17:23:50.451 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:50 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1425780683' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-09T17:23:50.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:50 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1425780683' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-09T17:23:50.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:50 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1425780683' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-09T17:23:51.609 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:51 smithi099 ceph-mon[39420]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:51.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:51 smithi179 ceph-mon[43975]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:51.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:51 smithi162 ceph-mon[44304]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:52.611 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:52 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:23:52.611 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:52 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:23:52.611 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:52 smithi099 ceph-mon[39420]: osdmap e102: 12 total, 12 up, 12 in 2024-02-09T17:23:52.611 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:52 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:23:52.611 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:52 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:23:52.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:52 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:23:52.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:52 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:23:52.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:52 smithi179 ceph-mon[43975]: osdmap e102: 12 total, 12 up, 12 in 2024-02-09T17:23:52.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:52 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:23:52.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:52 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:23:52.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:52 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:23:52.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:52 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:23:52.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:52 smithi162 ceph-mon[44304]: osdmap e102: 12 total, 12 up, 12 in 2024-02-09T17:23:52.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:52 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:23:52.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:52 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:23:53.166 INFO:teuthology.orchestra.run.smithi099.stdout:330712481820 2024-02-09T17:23:53.665 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:53 smithi099 ceph-mon[39420]: pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:53.666 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:53 smithi099 ceph-mon[39420]: osdmap e103: 12 total, 12 up, 12 in 2024-02-09T17:23:53.666 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:53 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2622869405' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-02-09T17:23:53.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:53 smithi179 ceph-mon[43975]: pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:53.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:53 smithi179 ceph-mon[43975]: osdmap e103: 12 total, 12 up, 12 in 2024-02-09T17:23:53.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:53 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2622869405' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-02-09T17:23:53.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:53 smithi162 ceph-mon[44304]: pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:53.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:53 smithi162 ceph-mon[44304]: osdmap e103: 12 total, 12 up, 12 in 2024-02-09T17:23:53.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:53 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2622869405' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-02-09T17:23:54.549 INFO:teuthology.orchestra.run.smithi099.stdout:171798691888 2024-02-09T17:23:55.107 INFO:teuthology.orchestra.run.smithi099.stdout:77309411391 2024-02-09T17:23:55.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:55 smithi179 ceph-mon[43975]: pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:55.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:55 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/350723042' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-09T17:23:55.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:55 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1316391193' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-09T17:23:55.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:55 smithi162 ceph-mon[44304]: pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:55.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:55 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/350723042' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-09T17:23:55.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:55 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1316391193' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-09T17:23:55.733 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:55 smithi099 ceph-mon[39420]: pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:55.733 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:55 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/350723042' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-09T17:23:55.733 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:55 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1316391193' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-09T17:23:56.836 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411387 got 77309411391 for osd.2 2024-02-09T17:23:56.836 DEBUG:teuthology.parallel:result is None 2024-02-09T17:23:56.881 INFO:teuthology.orchestra.run.smithi099.stdout:34359738442 2024-02-09T17:23:57.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:57 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:23:57.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:57 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:23:57.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:57 smithi099 ceph-mon[39420]: pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:57.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:57 smithi099 ceph-mon[39420]: osdmap e104: 12 total, 12 up, 12 in 2024-02-09T17:23:57.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:57 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/951106287' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-09T17:23:57.575 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738434 got 34359738442 for osd.0 2024-02-09T17:23:57.575 DEBUG:teuthology.parallel:result is None 2024-02-09T17:23:57.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:57 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:23:57.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:57 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:23:57.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:57 smithi179 ceph-mon[43975]: pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:57.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:57 smithi179 ceph-mon[43975]: osdmap e104: 12 total, 12 up, 12 in 2024-02-09T17:23:57.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:57 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/951106287' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-09T17:23:57.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:57 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:23:57.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:57 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:23:57.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:57 smithi162 ceph-mon[44304]: pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:57.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:57 smithi162 ceph-mon[44304]: osdmap e104: 12 total, 12 up, 12 in 2024-02-09T17:23:57.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:57 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/951106287' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-09T17:23:57.919 INFO:teuthology.orchestra.run.smithi099.stdout:244813135910 2024-02-09T17:23:58.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:58 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/243575261' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-09T17:23:58.731 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:58 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/243575261' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-09T17:23:58.756 INFO:tasks.cephadm.ceph_manager.ceph:need seq 244813135904 got 244813135910 for osd.7 2024-02-09T17:23:58.757 DEBUG:teuthology.parallel:result is None 2024-02-09T17:23:58.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:58 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/243575261' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-09T17:23:59.695 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:23:59 smithi099 ceph-mon[39420]: pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:59.728 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691884 got 171798691888 for osd.5 2024-02-09T17:23:59.728 DEBUG:teuthology.parallel:result is None 2024-02-09T17:23:59.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:23:59 smithi179 ceph-mon[43975]: pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:23:59.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:23:59 smithi162 ceph-mon[44304]: pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:00.127 INFO:teuthology.orchestra.run.smithi099.stdout:283467841570 2024-02-09T17:24:00.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:00 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3204192728' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-02-09T17:24:00.869 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117044 got 115964117049 for osd.3 2024-02-09T17:24:00.869 DEBUG:teuthology.parallel:result is None 2024-02-09T17:24:00.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:00 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3204192728' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-02-09T17:24:00.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:00 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3204192728' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-02-09T17:24:01.660 INFO:tasks.cephadm.ceph_manager.ceph:need seq 283467841565 got 283467841570 for osd.8 2024-02-09T17:24:01.660 DEBUG:teuthology.parallel:result is None 2024-02-09T17:24:01.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:01 smithi099 ceph-mon[39420]: pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:01.762 INFO:teuthology.orchestra.run.smithi099.stdout:137438953527 2024-02-09T17:24:01.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:01 smithi179 ceph-mon[43975]: pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:01.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:01 smithi162 ceph-mon[44304]: pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:02.129 INFO:tasks.cephadm.ceph_manager.ceph:need seq 330712481814 got 330712481820 for osd.9 2024-02-09T17:24:02.130 DEBUG:teuthology.parallel:result is None 2024-02-09T17:24:02.567 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953520 got 137438953527 for osd.4 2024-02-09T17:24:02.567 DEBUG:teuthology.parallel:result is None 2024-02-09T17:24:02.567 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-02-09T17:24:02.568 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph pg dump --format=json 2024-02-09T17:24:02.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:02 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2955028164' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-09T17:24:02.833 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:24:02.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:02 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2955028164' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-09T17:24:02.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:02 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2955028164' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-09T17:24:03.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:03 smithi099 ceph-mon[39420]: pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:03.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:03 smithi179 ceph-mon[43975]: pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:03.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:03 smithi162 ceph-mon[44304]: pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:04.899 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:24:04.900 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-02-09T17:24:05.688 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:05 smithi099 ceph-mon[39420]: pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:05.716 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":324,"stamp":"2024-02-09T17:24:04.321688+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":343000,"kb_used_data":10440,"kb_used_omap":18,"kb_used_meta":332397,"kb_avail":1124598824,"statfs":{"total":1151940427776,"available":1151589195776,"internally_reserved":0,"allocated":10690560,"data_stored":6353472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":340374841},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001999"},"pg_stats":[{"pgid":"1.0","version":"63'139","reported_seq":377,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-09T17:23:56.472264+0000","last_change":"2024-02-09T17:23:26.651048+0000","last_active":"2024-02-09T17:23:56.472264+0000","last_peered":"2024-02-09T17:23:56.472264+0000","last_clean":"2024-02-09T17:23:56.472264+0000","last_became_active":"2024-02-09T17:22:46.986816+0000","last_became_peered":"2024-02-09T17:22:46.986816+0000","last_unstale":"2024-02-09T17:23:56.472264+0000","last_undegraded":"2024-02-09T17:23:56.472264+0000","last_fullsized":"2024-02-09T17:23:56.472264+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:23:26.650945+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:23:26.650945+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:24:46.463688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056660,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28280,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716872,"statfs":{"total":95995035648,"available":95966076928,"internally_reserved":0,"allocated":1290240,"data_stored":926137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.52600000000000002}]},{"osd":1,"last update":"Fri Feb 9 17:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.626},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.71699999999999997}]},{"osd":2,"last update":"Fri Feb 9 17:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.48799999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.60899999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.95999999999999996},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":0.81699999999999995}]},{"osd":5,"last update":"Fri Feb 9 17:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":1.0009999999999999},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.73499999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.98099999999999998},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.77100000000000002}]},{"osd":7,"last update":"Fri Feb 9 17:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.79300000000000004}]},{"osd":8,"last update":"Fri Feb 9 17:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.63700000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.78700000000000003}]}]},{"osd":10,"up_from":84,"seq":360777252889,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27760,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717392,"statfs":{"total":95995035648,"available":95966609408,"internally_reserved":0,"allocated":757760,"data_stored":397209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":1,"last update":"Fri Feb 9 17:23:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.85399999999999998}]},{"osd":2,"last update":"Fri Feb 9 17:23:17 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.74099999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:23:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.72099999999999997}]},{"osd":4,"last update":"Fri Feb 9 17:23:17 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1,"5min":1,"15min":1},"last":0.80400000000000005}]},{"osd":5,"last update":"Fri Feb 9 17:23:17 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.90500000000000003}]},{"osd":6,"last update":"Fri Feb 9 17:23:17 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.81699999999999995}]},{"osd":7,"last update":"Fri Feb 9 17:23:17 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.88800000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:23:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.61599999999999999}]},{"osd":9,"last update":"Fri Feb 9 17:23:17 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.78900000000000003}]},{"osd":11,"last update":"Fri Feb 9 17:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.877}]}]},{"osd":9,"up_from":77,"seq":330712481822,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27760,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717392,"statfs":{"total":95995035648,"available":95966609408,"internally_reserved":0,"allocated":757760,"data_stored":397209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.55200000000000005,"15min":0.55800000000000005},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.71799999999999997,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.55000000000000004,"15min":0.55200000000000005},"min":{"1min":0.315,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.75600000000000001,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.45700000000000002}]},{"osd":1,"last update":"Fri Feb 9 17:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.55600000000000005,"15min":0.56399999999999995},"min":{"1min":0.35999999999999999,"5min":0.318,"15min":0.318},"max":{"1min":0.78100000000000003,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.52700000000000002}]},{"osd":2,"last update":"Fri Feb 9 17:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56899999999999995,"15min":0.57099999999999995},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.70999999999999996,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.56499999999999995,"15min":0.56799999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.89600000000000002,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.61599999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.57299999999999995,"15min":0.57699999999999996},"min":{"1min":0.37,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.78800000000000003,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.56000000000000005,"15min":0.55200000000000005},"min":{"1min":0.45400000000000001,"5min":0.373,"15min":0.373},"max":{"1min":0.75600000000000001,"5min":0.88,"15min":0.88},"last":0.47899999999999998}]},{"osd":4,"last update":"Fri Feb 9 17:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.67200000000000004,"15min":0.66400000000000003},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.85199999999999998}]},{"osd":5,"last update":"Fri Feb 9 17:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.66000000000000003,"15min":0.64800000000000002},"min":{"1min":0.49399999999999999,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.66900000000000004,"15min":0.66000000000000003},"min":{"1min":0.49399999999999999,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.88400000000000001}]},{"osd":6,"last update":"Fri Feb 9 17:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.67900000000000005,"15min":0.66700000000000004},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.68300000000000005,"15min":0.67700000000000005},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.67400000000000004}]},{"osd":7,"last update":"Fri Feb 9 17:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.70599999999999996,"15min":0.69699999999999995},"min":{"1min":0.55400000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.91800000000000004},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.67700000000000005,"15min":0.66200000000000003},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.70399999999999996}]},{"osd":8,"last update":"Fri Feb 9 17:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.53900000000000003,"15min":0.51900000000000002},"min":{"1min":0.47899999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.56499999999999995,"15min":0.54400000000000004},"min":{"1min":0.498,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.85999999999999999}]},{"osd":10,"last update":"Fri Feb 9 17:23:20 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.79200000000000004}]}]},{"osd":8,"up_from":66,"seq":283467841572,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27760,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717392,"statfs":{"total":95995035648,"available":95966609408,"internally_reserved":0,"allocated":757760,"data_stored":397229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.58399999999999996,"15min":0.58199999999999996},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.56899999999999995,"15min":0.58099999999999996},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.751,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.83299999999999996}]},{"osd":1,"last update":"Fri Feb 9 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.56000000000000005,"15min":0.56299999999999994},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.57099999999999995,"15min":0.57899999999999996},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.84599999999999997}]},{"osd":2,"last update":"Fri Feb 9 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.56399999999999995,"15min":0.56899999999999995},"min":{"1min":0.38400000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.72599999999999998,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.85899999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.59399999999999997,"15min":0.60399999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.78400000000000003,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.59499999999999997,"15min":0.60499999999999998},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.67900000000000005,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.82199999999999995}]},{"osd":4,"last update":"Fri Feb 9 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.63,"15min":0.623},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.85599999999999998,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.67100000000000004,"15min":0.66300000000000003},"min":{"1min":0.55300000000000005,"5min":0.44,"15min":0.44},"max":{"1min":0.92000000000000004,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.80400000000000005}]},{"osd":5,"last update":"Fri Feb 9 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.65700000000000003,"15min":0.65100000000000002},"min":{"1min":0.504,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.84999999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.64600000000000002,"15min":0.63700000000000001},"min":{"1min":0.52100000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.77300000000000002}]},{"osd":6,"last update":"Fri Feb 9 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.65500000000000003,"15min":0.64700000000000002},"min":{"1min":0.51900000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.65600000000000003,"15min":0.64600000000000002},"min":{"1min":0.57999999999999996,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.745}]},{"osd":7,"last update":"Fri Feb 9 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.63400000000000001,"15min":0.61799999999999999},"min":{"1min":0.58399999999999996,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.67300000000000004,"15min":0.66800000000000004},"min":{"1min":0.47799999999999998,"5min":0.438,"15min":0.438},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.71699999999999997}]},{"osd":9,"last update":"Fri Feb 9 17:23:51 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.52900000000000003,"15min":0.51700000000000002},"min":{"1min":0.38800000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.76900000000000002,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.55400000000000005,"15min":0.53600000000000003},"min":{"1min":0.45600000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.76000000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.873},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.88900000000000001}]}]},{"osd":7,"up_from":57,"seq":244813135912,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28280,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716872,"statfs":{"total":95995035648,"available":95966076928,"internally_reserved":0,"allocated":1290240,"data_stored":926157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.622},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.67100000000000004}]},{"osd":1,"last update":"Fri Feb 9 17:23:03 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.61099999999999999,"15min":0.60599999999999998},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.626,"5min":0.63300000000000001,"15min":0.63400000000000001},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.501}]},{"osd":2,"last update":"Fri Feb 9 17:23:03 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.61199999999999999,"15min":0.61499999999999999},"min":{"1min":0.44700000000000001,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.66000000000000003,"15min":0.65200000000000002},"min":{"1min":0.52500000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.42799999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:23:03 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.48399999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.627,"5min":0.64800000000000002,"15min":0.65200000000000002},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.84899999999999998,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.48699999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:23:03 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.55700000000000005,"15min":0.55200000000000005},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.54600000000000004,"15min":0.54200000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.83799999999999997,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.55100000000000005}]},{"osd":5,"last update":"Fri Feb 9 17:23:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55300000000000005,"15min":0.55500000000000005},"min":{"1min":0.36299999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.75600000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63500000000000001,"15min":0.63600000000000001},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.88900000000000001,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.64200000000000002}]},{"osd":6,"last update":"Fri Feb 9 17:23:03 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60999999999999999,"15min":0.61199999999999999},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.79200000000000004,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.53200000000000003}]},{"osd":8,"last update":"Fri Feb 9 17:23:25 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.78000000000000003,"15min":0.77300000000000002},"min":{"1min":0.59599999999999997,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.77800000000000002,"15min":0.77600000000000002},"min":{"1min":0.56599999999999995,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.59299999999999997}]},{"osd":9,"last update":"Fri Feb 9 17:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.75800000000000001,"15min":0.753},"min":{"1min":0.65300000000000002,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.60299999999999998}]},{"osd":10,"last update":"Fri Feb 9 17:23:19 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.70299999999999996}]},{"osd":11,"last update":"Fri Feb 9 17:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.88,"15min":0.88},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.629}]}]},{"osd":6,"up_from":49,"seq":210453397549,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28280,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716872,"statfs":{"total":95995035648,"available":95966076928,"internally_reserved":0,"allocated":1290240,"data_stored":926157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:23:38 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":0.77000000000000002,"15min":0.66000000000000003},"min":{"1min":0.36799999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":24.544,"5min":24.544,"15min":24.544},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.624,"5min":0.64600000000000002,"15min":0.64500000000000002},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.80800000000000005,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.57499999999999996}]},{"osd":1,"last update":"Fri Feb 9 17:23:38 2024","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":0.74199999999999999,"15min":0.63100000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":23.954000000000001,"5min":23.954000000000001,"15min":23.954000000000001},"last":0.71799999999999997},{"interface":"front","average":{"1min":1.3560000000000001,"5min":0.75800000000000001,"15min":0.64900000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":23.984999999999999,"5min":23.984999999999999,"15min":23.984999999999999},"last":0.47799999999999998}]},{"osd":2,"last update":"Fri Feb 9 17:23:38 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":0.75900000000000001,"15min":0.66500000000000004},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":24.329999999999998,"5min":24.329999999999998,"15min":24.329999999999998},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.67300000000000004,"15min":0.61099999999999999},"min":{"1min":0.34699999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":12.906000000000001,"5min":12.906000000000001,"15min":12.906000000000001},"last":0.60199999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.59899999999999998,"15min":0.58499999999999996},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":2.609,"5min":2.609,"15min":2.609},"last":0.5},{"interface":"front","average":{"1min":1.3540000000000001,"5min":0.77100000000000002,"15min":0.66100000000000003},"min":{"1min":0.40899999999999997,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":24.219000000000001,"5min":24.219000000000001,"15min":24.219000000000001},"last":0.58799999999999997}]},{"osd":4,"last update":"Fri Feb 9 17:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.497,"15min":0.48099999999999998},"min":{"1min":0.26400000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.505,"15min":0.48399999999999999},"min":{"1min":0.33800000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.55200000000000005}]},{"osd":5,"last update":"Fri Feb 9 17:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.52400000000000002,"15min":0.50600000000000001},"min":{"1min":0.31900000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.52400000000000002,"15min":0.504},"min":{"1min":0.35599999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.81200000000000006,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.61799999999999999}]},{"osd":7,"last update":"Fri Feb 9 17:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.80900000000000005,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.69299999999999995}]},{"osd":8,"last update":"Fri Feb 9 17:23:27 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.76300000000000001,"15min":0.75900000000000001},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.73399999999999999,"15min":0.73199999999999998},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.0469999999999999,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.752}]},{"osd":9,"last update":"Fri Feb 9 17:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"min":{"1min":0.60599999999999998,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.81999999999999995}]},{"osd":10,"last update":"Fri Feb 9 17:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.98799999999999999},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.95399999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574918,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31920,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713232,"statfs":{"total":95995035648,"available":95962349568,"internally_reserved":0,"allocated":757760,"data_stored":397229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:23:47 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.376,"15min":0.33500000000000002},"min":{"1min":0.255,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.53000000000000003,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.441,"5min":0.442,"15min":0.35999999999999999},"min":{"1min":0.30499999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.64800000000000002,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.34699999999999998}]},{"osd":2,"last update":"Fri Feb 9 17:23:08 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.39400000000000002,"15min":0.38900000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.41199999999999998,"15min":0.40000000000000002},"min":{"1min":0.28699999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.56599999999999995,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.29699999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:23:34 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.46400000000000002,"15min":0.46200000000000002},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.58399999999999996,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.46600000000000003,"15min":0.46700000000000003},"min":{"1min":0.32200000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.56599999999999995,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.40000000000000002}]},{"osd":4,"last update":"Fri Feb 9 17:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.58599999999999997,"15min":0.56200000000000006},"min":{"1min":0.41399999999999998,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.78200000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.61599999999999999,"15min":0.57199999999999995},"min":{"1min":0.45400000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.77500000000000002,"5min":1.028,"15min":1.028},"last":0.68100000000000005}]},{"osd":5,"last update":"Fri Feb 9 17:23:16 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.624,"15min":0.60999999999999999},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.79600000000000004,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.64500000000000002,"15min":0.627},"min":{"1min":0.51900000000000002,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.67400000000000004}]},{"osd":6,"last update":"Fri Feb 9 17:23:47 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.66900000000000004,"15min":0.66300000000000003},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.83199999999999996,"5min":1.095,"15min":1.095},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.67000000000000004,"15min":0.68300000000000005},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.76800000000000002,"5min":1.044,"15min":1.044},"last":0.627}]},{"osd":7,"last update":"Fri Feb 9 17:23:08 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.72699999999999998,"15min":0.73199999999999998},"min":{"1min":0.54800000000000004,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.76900000000000002,"15min":0.77400000000000002},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.65600000000000003}]},{"osd":8,"last update":"Fri Feb 9 17:23:31 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.73299999999999998,"15min":0.73399999999999999},"min":{"1min":0.55800000000000005,"5min":0.501,"15min":0.501},"max":{"1min":0.875,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.044,"5min":1.115,"15min":1.115},"last":0.53500000000000003}]},{"osd":9,"last update":"Fri Feb 9 17:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.77600000000000002,"15min":0.78800000000000003},"min":{"1min":0.54800000000000004,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.89300000000000002,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.75700000000000001,"15min":0.76200000000000001},"min":{"1min":0.58099999999999996,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.95899999999999996,"5min":1.21,"15min":1.21},"last":0.56499999999999995}]},{"osd":10,"last update":"Fri Feb 9 17:23:17 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.1719999999999999,"5min":1.1719999999999999,"15min":1.1719999999999999},"last":0.63800000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738444,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27760,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717392,"statfs":{"total":95995035648,"available":95966609408,"internally_reserved":0,"allocated":757760,"data_stored":397229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 9 17:23:40 2024","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.36199999999999999,"15min":0.33500000000000002},"min":{"1min":0.253,"5min":0.223,"15min":0.223},"max":{"1min":0.39200000000000002,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.39700000000000002,"15min":0.36699999999999999},"min":{"1min":0.26000000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.48499999999999999,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.27500000000000002}]},{"osd":2,"last update":"Fri Feb 9 17:21:59 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.39000000000000001,"15min":0.376},"min":{"1min":0.28599999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.51600000000000001,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.40400000000000003,"15min":0.39900000000000002},"min":{"1min":0.28100000000000003,"5min":0.245,"15min":0.245},"max":{"1min":0.55700000000000005,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.29399999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:23:30 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.45200000000000001,"15min":0.45500000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.52400000000000002,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.314},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.437,"15min":0.42599999999999999},"min":{"1min":0.31,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.58799999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.29999999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:23:55 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.623,"15min":0.60399999999999998},"min":{"1min":0.47499999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.83599999999999997,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.62,"15min":0.60599999999999998},"min":{"1min":0.498,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.81399999999999995,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.72399999999999998}]},{"osd":5,"last update":"Fri Feb 9 17:23:15 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61299999999999999,"15min":0.60299999999999998},"min":{"1min":0.42799999999999999,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.89100000000000001,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.626,"15min":0.61799999999999999},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.82399999999999995,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.66300000000000003}]},{"osd":6,"last update":"Fri Feb 9 17:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.64500000000000002,"15min":0.65500000000000003},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78900000000000003,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.65400000000000003,"15min":0.66800000000000004},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.71499999999999997,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.63700000000000001}]},{"osd":7,"last update":"Fri Feb 9 17:23:04 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.68700000000000006,"15min":0.69699999999999995},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.745},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.69099999999999995,"15min":0.69599999999999995},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.58499999999999996}]},{"osd":8,"last update":"Fri Feb 9 17:23:30 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.73599999999999999,"15min":0.74199999999999999},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.86399999999999999,"5min":1.042,"15min":1.042},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.71999999999999997,"15min":0.72399999999999998},"min":{"1min":0.53800000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.88800000000000001,"5min":1.004,"15min":1.004},"last":0.85199999999999998}]},{"osd":9,"last update":"Fri Feb 9 17:23:55 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.74099999999999999,"15min":0.751},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.82899999999999996,"5min":1.056,"15min":1.056},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.755,"15min":0.76200000000000001},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.98799999999999999,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.79600000000000004}]},{"osd":10,"last update":"Fri Feb 9 17:23:17 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.77900000000000003}]},{"osd":11,"last update":"Fri Feb 9 17:23:48 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.75900000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411393,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31920,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713232,"statfs":{"total":95995035648,"available":95962349568,"internally_reserved":0,"allocated":757760,"data_stored":397229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:23:02 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.40500000000000003,"15min":0.38200000000000001},"min":{"1min":0.26400000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.443,"5min":0.41199999999999998,"15min":0.377},"min":{"1min":0.28999999999999998,"5min":0.246,"15min":0.246},"max":{"1min":0.73999999999999999,"5min":0.747,"15min":0.747},"last":0.443}]},{"osd":1,"last update":"Fri Feb 9 17:23:02 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.41499999999999998,"15min":0.38700000000000001},"min":{"1min":0.32400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.372},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.42699999999999999,"15min":0.39200000000000002},"min":{"1min":0.28000000000000003,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.42199999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.45800000000000002,"15min":0.44400000000000001},"min":{"1min":0.314,"5min":0.312,"15min":0.312},"max":{"1min":0.71799999999999997,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.44800000000000001,"15min":0.44400000000000001},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.67400000000000004,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.38500000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:23:47 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.60299999999999998,"15min":0.58499999999999996},"min":{"1min":0.44,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.86399999999999999,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.63600000000000001,"15min":0.61299999999999999},"min":{"1min":0.45500000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.84099999999999997,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.73899999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:23:16 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.622},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.96299999999999997,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.64000000000000001,"15min":0.621},"min":{"1min":0.504,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.57499999999999996}]},{"osd":6,"last update":"Fri Feb 9 17:23:37 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.67600000000000005,"15min":0.66800000000000004},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.748},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.69099999999999995,"15min":0.69199999999999995},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.66100000000000003}]},{"osd":7,"last update":"Fri Feb 9 17:23:02 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.68100000000000005,"15min":0.68300000000000005},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.88,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.77100000000000002}]},{"osd":8,"last update":"Fri Feb 9 17:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.76100000000000001,"15min":0.76600000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.79200000000000004,"15min":0.80000000000000004},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.65100000000000002}]},{"osd":9,"last update":"Fri Feb 9 17:23:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.74399999999999999,"15min":0.747},"min":{"1min":0.55200000000000005,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.0609999999999999,"5min":1.111,"15min":1.111},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.745,"5min":0.76500000000000001,"15min":0.76800000000000002},"min":{"1min":0.56000000000000005,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.61899999999999999}]},{"osd":10,"last update":"Fri Feb 9 17:23:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.68899999999999995}]}]},{"osd":3,"up_from":27,"seq":115964117052,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27760,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717392,"statfs":{"total":95995035648,"available":95966609408,"internally_reserved":0,"allocated":757760,"data_stored":397229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:22:21 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.39700000000000002,"15min":0.38800000000000001},"min":{"1min":0.33300000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.503,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.436,"15min":0.434},"min":{"1min":0.34899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.55100000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.46600000000000003}]},{"osd":1,"last update":"Fri Feb 9 17:23:26 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.434,"15min":0.441},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.57899999999999996,"5min":0.628,"15min":0.628},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.441,"15min":0.44600000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.59599999999999997,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.48199999999999998}]},{"osd":2,"last update":"Fri Feb 9 17:23:26 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.46300000000000002,"15min":0.46700000000000003},"min":{"1min":0.29899999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.69299999999999995,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.44,"5min":0.45000000000000001,"15min":0.442},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.503}]},{"osd":4,"last update":"Fri Feb 9 17:23:51 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.60599999999999998,"15min":0.58699999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.81999999999999995,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.59099999999999997,"15min":0.58499999999999996},"min":{"1min":0.36499999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.72799999999999998,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.55800000000000005}]},{"osd":5,"last update":"Fri Feb 9 17:23:13 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63800000000000001,"15min":0.63400000000000001},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.90300000000000002,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.63200000000000001,"15min":0.61599999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.66000000000000003}]},{"osd":6,"last update":"Fri Feb 9 17:23:39 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.65000000000000002,"15min":0.65800000000000003},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.79800000000000004,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.67200000000000004,"15min":0.68000000000000005},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.161,"5min":1.161,"15min":1.161},"last":0.70699999999999996}]},{"osd":7,"last update":"Fri Feb 9 17:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.69999999999999996,"15min":0.69899999999999995},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.876,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.69499999999999995,"15min":0.69899999999999995},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.71999999999999997}]},{"osd":8,"last update":"Fri Feb 9 17:23:27 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.53900000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.014,"5min":1.151,"15min":1.151},"last":0.98899999999999999},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.78000000000000003,"15min":0.78400000000000003},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.97699999999999998}]},{"osd":9,"last update":"Fri Feb 9 17:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.78800000000000003,"15min":0.78400000000000003},"min":{"1min":0.56999999999999995,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":1.018},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.78300000000000003,"15min":0.79000000000000004},"min":{"1min":0.57599999999999996,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.92900000000000005,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.92900000000000005}]},{"osd":10,"last update":"Fri Feb 9 17:23:20 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":1.008},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.94399999999999995}]},{"osd":11,"last update":"Fri Feb 9 17:23:47 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.96599999999999997},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.95799999999999996}]}]},{"osd":4,"up_from":32,"seq":137438953528,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27760,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717392,"statfs":{"total":95995035648,"available":95966609408,"internally_reserved":0,"allocated":757760,"data_stored":397229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:23:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.58499999999999996,"15min":0.57099999999999995},"min":{"1min":0.41899999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.73099999999999998,"5min":1.056,"15min":1.056},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.63,"5min":0.59899999999999998,"15min":0.58599999999999997},"min":{"1min":0.46200000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.90500000000000003,"5min":1.179,"15min":1.179},"last":0.56699999999999995}]},{"osd":1,"last update":"Fri Feb 9 17:23:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.58899999999999997,"15min":0.57799999999999996},"min":{"1min":0.439,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.73299999999999998,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.505},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.59599999999999997,"15min":0.58799999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.88700000000000001,"5min":1.113,"15min":1.113},"last":0.71099999999999997}]},{"osd":2,"last update":"Fri Feb 9 17:23:47 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.623,"15min":0.58899999999999997},"min":{"1min":0.441,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.83199999999999996,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.628},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60399999999999998,"15min":0.59999999999999998},"min":{"1min":0.45600000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.76300000000000001,"5min":1.196,"15min":1.196},"last":0.58899999999999997}]},{"osd":3,"last update":"Fri Feb 9 17:23:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.63600000000000001,"15min":0.61799999999999999},"min":{"1min":0.42199999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.76000000000000001,"5min":1.151,"15min":1.151},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61699999999999999,"15min":0.60099999999999998},"min":{"1min":0.48399999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.76900000000000002,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.60299999999999998}]},{"osd":5,"last update":"Fri Feb 9 17:23:13 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.46000000000000002,"15min":0.44700000000000001},"min":{"1min":0.35799999999999998,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.68600000000000005,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.52900000000000003,"15min":0.47699999999999998},"min":{"1min":0.36599999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.91600000000000004,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.80400000000000005}]},{"osd":6,"last update":"Fri Feb 9 17:23:42 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.51100000000000001,"15min":0.47599999999999998},"min":{"1min":0.33100000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.51600000000000001,"15min":0.48499999999999999},"min":{"1min":0.35299999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.73099999999999998,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.69799999999999995}]},{"osd":7,"last update":"Fri Feb 9 17:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.60599999999999998,"15min":0.59999999999999998},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.60799999999999998,"15min":0.60199999999999998},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.74099999999999999}]},{"osd":8,"last update":"Fri Feb 9 17:23:24 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.76300000000000001,"15min":0.75900000000000001},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.96799999999999997,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":1.194},{"interface":"front","average":{"1min":0.753,"5min":0.74099999999999999,"15min":0.73899999999999999},"min":{"1min":0.60499999999999998,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":0.97499999999999998,"5min":1.02,"15min":1.02},"last":1.1779999999999999}]},{"osd":9,"last update":"Fri Feb 9 17:23:54 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.81599999999999995,"15min":0.81299999999999994},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.139,"5min":1.139,"15min":1.139},"last":0.96799999999999997},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.97699999999999998,"5min":1.02,"15min":1.02},"last":1.0489999999999999}]},{"osd":10,"last update":"Fri Feb 9 17:23:19 2024","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":1.095},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.90600000000000003}]}]},{"osd":5,"up_from":40,"seq":171798691890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27760,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717392,"statfs":{"total":95995035648,"available":95966609408,"internally_reserved":0,"allocated":757760,"data_stored":397229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:23:13 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.57399999999999995,"15min":0.54300000000000004},"min":{"1min":0.42499999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.81699999999999995,"5min":0.877,"15min":0.877},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.54800000000000004,"15min":0.52300000000000002},"min":{"1min":0.46700000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.72499999999999998,"5min":0.753,"15min":0.753},"last":0.52100000000000002}]},{"osd":1,"last update":"Fri Feb 9 17:23:13 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.56499999999999995,"15min":0.52000000000000002},"min":{"1min":0.48999999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.89100000000000001,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.56000000000000005,"15min":0.54300000000000004},"min":{"1min":0.40400000000000003,"5min":0.379,"15min":0.379},"max":{"1min":0.78400000000000003,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.58599999999999997}]},{"osd":2,"last update":"Fri Feb 9 17:23:13 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.57499999999999996,"15min":0.55300000000000005},"min":{"1min":0.40799999999999997,"5min":0.373,"15min":0.373},"max":{"1min":0.78900000000000003,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.625},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.626,"15min":0.59399999999999997},"min":{"1min":0.51200000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.83499999999999996,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.73399999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:23:13 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.57699999999999996,"15min":0.55300000000000005},"min":{"1min":0.47899999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.83499999999999996,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.59999999999999998,"15min":0.56599999999999995},"min":{"1min":0.55000000000000004,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.92100000000000004,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.66300000000000003}]},{"osd":4,"last update":"Fri Feb 9 17:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.41399999999999998,"15min":0.377},"min":{"1min":0.254,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.71599999999999997,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.44500000000000001,"15min":0.40300000000000002},"min":{"1min":0.35099999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.76800000000000002,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.48399999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:23:40 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.50900000000000001,"15min":0.49299999999999999},"min":{"1min":0.309,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.80800000000000005,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.52800000000000002,"15min":0.48799999999999999},"min":{"1min":0.36599999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.83799999999999997,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.75600000000000001}]},{"osd":7,"last update":"Fri Feb 9 17:23:04 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.60999999999999999,"15min":0.60799999999999998},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.85599999999999998,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.59299999999999997,"15min":0.58699999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.87,"5min":0.872,"15min":0.872},"last":0.64000000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.72499999999999998,"15min":0.71799999999999997},"min":{"1min":0.58299999999999996,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":0.94699999999999995,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.78500000000000003,"15min":0.78300000000000003},"min":{"1min":0.66800000000000004,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":0.94499999999999995,"5min":1.097,"15min":1.097},"last":0.78400000000000003}]},{"osd":9,"last update":"Fri Feb 9 17:23:55 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":0.96199999999999997,"5min":1.171,"15min":1.171},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.83499999999999996,"15min":0.84099999999999997},"min":{"1min":0.63700000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":0.997,"5min":1.083,"15min":1.083},"last":0.875}]},{"osd":10,"last update":"Fri Feb 9 17:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.85399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-09T17:24:05.719 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph pg dump --format=json 2024-02-09T17:24:05.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:05 smithi179 ceph-mon[43975]: pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:05.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:05 smithi162 ceph-mon[44304]: pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:05.981 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:24:06.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:06 smithi099 ceph-mon[39420]: from='client.14691 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:24:06.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:06 smithi179 ceph-mon[43975]: from='client.14691 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:24:06.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:06 smithi162 ceph-mon[44304]: from='client.14691 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:24:07.490 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:07 smithi099 ceph-mon[39420]: pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:07.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:07 smithi179 ceph-mon[43975]: pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:07.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:07 smithi162 ceph-mon[44304]: pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:07.929 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:24:07.931 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-02-09T17:24:08.536 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2024-02-09T17:24:06.322142+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":343000,"kb_used_data":10440,"kb_used_omap":18,"kb_used_meta":332397,"kb_avail":1124598824,"statfs":{"total":1151940427776,"available":1151589195776,"internally_reserved":0,"allocated":10690560,"data_stored":6353472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":340374841},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002050"},"pg_stats":[{"pgid":"1.0","version":"63'139","reported_seq":377,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-09T17:23:56.472264+0000","last_change":"2024-02-09T17:23:26.651048+0000","last_active":"2024-02-09T17:23:56.472264+0000","last_peered":"2024-02-09T17:23:56.472264+0000","last_clean":"2024-02-09T17:23:56.472264+0000","last_became_active":"2024-02-09T17:22:46.986816+0000","last_became_peered":"2024-02-09T17:22:46.986816+0000","last_unstale":"2024-02-09T17:23:56.472264+0000","last_undegraded":"2024-02-09T17:23:56.472264+0000","last_fullsized":"2024-02-09T17:23:56.472264+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:23:26.650945+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:23:26.650945+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:24:46.463688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056660,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28280,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716872,"statfs":{"total":95995035648,"available":95966076928,"internally_reserved":0,"allocated":1290240,"data_stored":926137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.52600000000000002}]},{"osd":1,"last update":"Fri Feb 9 17:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.626},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.71699999999999997}]},{"osd":2,"last update":"Fri Feb 9 17:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.48799999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.60899999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.95999999999999996},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":0.81699999999999995}]},{"osd":5,"last update":"Fri Feb 9 17:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":1.0009999999999999},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.73499999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.98099999999999998},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.77100000000000002}]},{"osd":7,"last update":"Fri Feb 9 17:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.79300000000000004}]},{"osd":8,"last update":"Fri Feb 9 17:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.63700000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.78700000000000003}]}]},{"osd":10,"up_from":84,"seq":360777252889,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27760,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717392,"statfs":{"total":95995035648,"available":95966609408,"internally_reserved":0,"allocated":757760,"data_stored":397209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":1,"last update":"Fri Feb 9 17:23:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.85399999999999998}]},{"osd":2,"last update":"Fri Feb 9 17:23:17 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.74099999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:23:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.72099999999999997}]},{"osd":4,"last update":"Fri Feb 9 17:23:17 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1,"5min":1,"15min":1},"last":0.80400000000000005}]},{"osd":5,"last update":"Fri Feb 9 17:23:17 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.90500000000000003}]},{"osd":6,"last update":"Fri Feb 9 17:23:17 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.81699999999999995}]},{"osd":7,"last update":"Fri Feb 9 17:23:17 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.88800000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:23:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.61599999999999999}]},{"osd":9,"last update":"Fri Feb 9 17:23:17 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.78900000000000003}]},{"osd":11,"last update":"Fri Feb 9 17:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.877}]}]},{"osd":9,"up_from":77,"seq":330712481823,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27760,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717392,"statfs":{"total":95995035648,"available":95966609408,"internally_reserved":0,"allocated":757760,"data_stored":397209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.55200000000000005,"15min":0.55800000000000005},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.71799999999999997,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.501},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.55000000000000004,"15min":0.55200000000000005},"min":{"1min":0.315,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.75600000000000001,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.442}]},{"osd":1,"last update":"Fri Feb 9 17:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.55600000000000005,"15min":0.56399999999999995},"min":{"1min":0.35999999999999999,"5min":0.318,"15min":0.318},"max":{"1min":0.78100000000000003,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.41499999999999998}]},{"osd":2,"last update":"Fri Feb 9 17:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56899999999999995,"15min":0.57099999999999995},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.70999999999999996,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.56499999999999995,"15min":0.56799999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.89600000000000002,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.55600000000000005}]},{"osd":3,"last update":"Fri Feb 9 17:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.57299999999999995,"15min":0.57699999999999996},"min":{"1min":0.37,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.78800000000000003,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.56000000000000005,"15min":0.55200000000000005},"min":{"1min":0.45400000000000001,"5min":0.373,"15min":0.373},"max":{"1min":0.75600000000000001,"5min":0.88,"15min":0.88},"last":0.39400000000000002}]},{"osd":4,"last update":"Fri Feb 9 17:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.67200000000000004,"15min":0.66400000000000003},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.75}]},{"osd":5,"last update":"Fri Feb 9 17:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.66000000000000003,"15min":0.64800000000000002},"min":{"1min":0.49399999999999999,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.66900000000000004,"15min":0.66000000000000003},"min":{"1min":0.49399999999999999,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.68100000000000005}]},{"osd":6,"last update":"Fri Feb 9 17:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.67900000000000005,"15min":0.66700000000000004},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.628},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.68300000000000005,"15min":0.67700000000000005},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.65300000000000002}]},{"osd":7,"last update":"Fri Feb 9 17:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.70599999999999996,"15min":0.69699999999999995},"min":{"1min":0.55400000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.67700000000000005,"15min":0.66200000000000003},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.70099999999999996}]},{"osd":8,"last update":"Fri Feb 9 17:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.53900000000000003,"15min":0.51900000000000002},"min":{"1min":0.47899999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.56499999999999995,"15min":0.54400000000000004},"min":{"1min":0.498,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.63800000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:23:20 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.61399999999999999}]}]},{"osd":8,"up_from":66,"seq":283467841573,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27760,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717392,"statfs":{"total":95995035648,"available":95966609408,"internally_reserved":0,"allocated":757760,"data_stored":397229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.58399999999999996,"15min":0.58199999999999996},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.56899999999999995,"15min":0.58099999999999996},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.751,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.503}]},{"osd":1,"last update":"Fri Feb 9 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.56000000000000005,"15min":0.56299999999999994},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.57099999999999995,"15min":0.57899999999999996},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.377}]},{"osd":2,"last update":"Fri Feb 9 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.56399999999999995,"15min":0.56899999999999995},"min":{"1min":0.38400000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.72599999999999998,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.52700000000000002}]},{"osd":3,"last update":"Fri Feb 9 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.59399999999999997,"15min":0.60399999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.78400000000000003,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.59499999999999997,"15min":0.60499999999999998},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.67900000000000005,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.436}]},{"osd":4,"last update":"Fri Feb 9 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.63,"15min":0.623},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.85599999999999998,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.748},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.67100000000000004,"15min":0.66300000000000003},"min":{"1min":0.55300000000000005,"5min":0.44,"15min":0.44},"max":{"1min":0.92000000000000004,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.67400000000000004}]},{"osd":5,"last update":"Fri Feb 9 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.65700000000000003,"15min":0.65100000000000002},"min":{"1min":0.504,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.84999999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.63},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.64600000000000002,"15min":0.63700000000000001},"min":{"1min":0.52100000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.68999999999999995}]},{"osd":6,"last update":"Fri Feb 9 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.65500000000000003,"15min":0.64700000000000002},"min":{"1min":0.51900000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.65600000000000003,"15min":0.64600000000000002},"min":{"1min":0.57999999999999996,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.70899999999999996}]},{"osd":7,"last update":"Fri Feb 9 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.63400000000000001,"15min":0.61799999999999999},"min":{"1min":0.58399999999999996,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.67300000000000004,"15min":0.66800000000000004},"min":{"1min":0.47799999999999998,"5min":0.438,"15min":0.438},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.77100000000000002}]},{"osd":9,"last update":"Fri Feb 9 17:23:51 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.52900000000000003,"15min":0.51700000000000002},"min":{"1min":0.38800000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.76900000000000002,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.55400000000000005,"15min":0.53600000000000003},"min":{"1min":0.45600000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.59899999999999998}]},{"osd":10,"last update":"Fri Feb 9 17:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.58599999999999997}]}]},{"osd":7,"up_from":57,"seq":244813135912,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28280,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716872,"statfs":{"total":95995035648,"available":95966076928,"internally_reserved":0,"allocated":1290240,"data_stored":926157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.622},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.67100000000000004}]},{"osd":1,"last update":"Fri Feb 9 17:23:03 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.61099999999999999,"15min":0.60599999999999998},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.626,"5min":0.63300000000000001,"15min":0.63400000000000001},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.501}]},{"osd":2,"last update":"Fri Feb 9 17:23:03 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.61199999999999999,"15min":0.61499999999999999},"min":{"1min":0.44700000000000001,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.66000000000000003,"15min":0.65200000000000002},"min":{"1min":0.52500000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.42799999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:23:03 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.48399999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.627,"5min":0.64800000000000002,"15min":0.65200000000000002},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.84899999999999998,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.48699999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:23:03 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.55700000000000005,"15min":0.55200000000000005},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.54600000000000004,"15min":0.54200000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.83799999999999997,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.55100000000000005}]},{"osd":5,"last update":"Fri Feb 9 17:23:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55300000000000005,"15min":0.55500000000000005},"min":{"1min":0.36299999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.75600000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63500000000000001,"15min":0.63600000000000001},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.88900000000000001,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.64200000000000002}]},{"osd":6,"last update":"Fri Feb 9 17:23:03 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60999999999999999,"15min":0.61199999999999999},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.79200000000000004,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.53200000000000003}]},{"osd":8,"last update":"Fri Feb 9 17:23:25 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.78000000000000003,"15min":0.77300000000000002},"min":{"1min":0.59599999999999997,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.77800000000000002,"15min":0.77600000000000002},"min":{"1min":0.56599999999999995,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.59299999999999997}]},{"osd":9,"last update":"Fri Feb 9 17:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.75800000000000001,"15min":0.753},"min":{"1min":0.65300000000000002,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.60299999999999998}]},{"osd":10,"last update":"Fri Feb 9 17:23:19 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.70299999999999996}]},{"osd":11,"last update":"Fri Feb 9 17:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.88,"15min":0.88},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.629}]}]},{"osd":6,"up_from":49,"seq":210453397550,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28280,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716872,"statfs":{"total":95995035648,"available":95966076928,"internally_reserved":0,"allocated":1290240,"data_stored":926157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:23:38 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":0.77000000000000002,"15min":0.66000000000000003},"min":{"1min":0.36799999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":24.544,"5min":24.544,"15min":24.544},"last":0.94399999999999995},{"interface":"front","average":{"1min":0.624,"5min":0.64600000000000002,"15min":0.64500000000000002},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.80800000000000005,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.61499999999999999}]},{"osd":1,"last update":"Fri Feb 9 17:23:38 2024","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":0.74199999999999999,"15min":0.63100000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":23.954000000000001,"5min":23.954000000000001,"15min":23.954000000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":1.3560000000000001,"5min":0.75800000000000001,"15min":0.64900000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":23.984999999999999,"5min":23.984999999999999,"15min":23.984999999999999},"last":0.92100000000000004}]},{"osd":2,"last update":"Fri Feb 9 17:23:38 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":0.75900000000000001,"15min":0.66500000000000004},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":24.329999999999998,"5min":24.329999999999998,"15min":24.329999999999998},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.67300000000000004,"15min":0.61099999999999999},"min":{"1min":0.34699999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":12.906000000000001,"5min":12.906000000000001,"15min":12.906000000000001},"last":0.873}]},{"osd":3,"last update":"Fri Feb 9 17:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.59899999999999998,"15min":0.58499999999999996},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":2.609,"5min":2.609,"15min":2.609},"last":0.66600000000000004},{"interface":"front","average":{"1min":1.3540000000000001,"5min":0.77100000000000002,"15min":0.66100000000000003},"min":{"1min":0.40899999999999997,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":24.219000000000001,"5min":24.219000000000001,"15min":24.219000000000001},"last":0.64400000000000002}]},{"osd":4,"last update":"Fri Feb 9 17:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.497,"15min":0.48099999999999998},"min":{"1min":0.26400000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.998},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.505,"15min":0.48399999999999999},"min":{"1min":0.33800000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.72999999999999998}]},{"osd":5,"last update":"Fri Feb 9 17:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.52400000000000002,"15min":0.50600000000000001},"min":{"1min":0.31900000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.52400000000000002,"15min":0.504},"min":{"1min":0.35599999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.81200000000000006,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.89600000000000002}]},{"osd":7,"last update":"Fri Feb 9 17:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.80900000000000005,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":1.0740000000000001},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.97699999999999998}]},{"osd":8,"last update":"Fri Feb 9 17:23:27 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.76300000000000001,"15min":0.75900000000000001},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":1.0469999999999999},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.73399999999999999,"15min":0.73199999999999998},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.0469999999999999,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":1.113}]},{"osd":9,"last update":"Fri Feb 9 17:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"min":{"1min":0.60599999999999998,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":1.095},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":1.129}]},{"osd":10,"last update":"Fri Feb 9 17:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":1.0600000000000001},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":1.1579999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574919,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31920,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713232,"statfs":{"total":95995035648,"available":95962349568,"internally_reserved":0,"allocated":757760,"data_stored":397229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:23:47 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.376,"15min":0.33500000000000002},"min":{"1min":0.255,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.53000000000000003,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.37},{"interface":"front","average":{"1min":0.441,"5min":0.442,"15min":0.35999999999999999},"min":{"1min":0.30499999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.64800000000000002,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.33800000000000002}]},{"osd":2,"last update":"Fri Feb 9 17:23:08 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.39400000000000002,"15min":0.38900000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.312},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.41199999999999998,"15min":0.40000000000000002},"min":{"1min":0.28699999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.56599999999999995,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.40699999999999997}]},{"osd":3,"last update":"Fri Feb 9 17:23:34 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.46400000000000002,"15min":0.46200000000000002},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.58399999999999996,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.46600000000000003,"15min":0.46700000000000003},"min":{"1min":0.32200000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.56599999999999995,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.38400000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.58599999999999997,"15min":0.56200000000000006},"min":{"1min":0.41399999999999998,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.78200000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.61599999999999999,"15min":0.57199999999999995},"min":{"1min":0.45400000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.77500000000000002,"5min":1.028,"15min":1.028},"last":0.66500000000000004}]},{"osd":5,"last update":"Fri Feb 9 17:23:16 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.624,"15min":0.60999999999999999},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.79600000000000004,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.64500000000000002,"15min":0.627},"min":{"1min":0.51900000000000002,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.60999999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:23:47 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.66900000000000004,"15min":0.66300000000000003},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.83199999999999996,"5min":1.095,"15min":1.095},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.67000000000000004,"15min":0.68300000000000005},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.76800000000000002,"5min":1.044,"15min":1.044},"last":0.63100000000000001}]},{"osd":7,"last update":"Fri Feb 9 17:23:08 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.72699999999999998,"15min":0.73199999999999998},"min":{"1min":0.54800000000000004,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.76900000000000002,"15min":0.77400000000000002},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.64500000000000002}]},{"osd":8,"last update":"Fri Feb 9 17:23:31 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.73299999999999998,"15min":0.73399999999999999},"min":{"1min":0.55800000000000005,"5min":0.501,"15min":0.501},"max":{"1min":0.875,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.044,"5min":1.115,"15min":1.115},"last":0.42499999999999999}]},{"osd":9,"last update":"Fri Feb 9 17:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.77600000000000002,"15min":0.78800000000000003},"min":{"1min":0.54800000000000004,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.89300000000000002,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.75700000000000001,"15min":0.76200000000000001},"min":{"1min":0.58099999999999996,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.95899999999999996,"5min":1.21,"15min":1.21},"last":0.47999999999999998}]},{"osd":10,"last update":"Fri Feb 9 17:23:17 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.498},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.1719999999999999,"5min":1.1719999999999999,"15min":1.1719999999999999},"last":0.46400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738444,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27760,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717392,"statfs":{"total":95995035648,"available":95966609408,"internally_reserved":0,"allocated":757760,"data_stored":397229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 9 17:23:40 2024","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.36199999999999999,"15min":0.33500000000000002},"min":{"1min":0.253,"5min":0.223,"15min":0.223},"max":{"1min":0.39200000000000002,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.39700000000000002,"15min":0.36699999999999999},"min":{"1min":0.26000000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.48499999999999999,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.27500000000000002}]},{"osd":2,"last update":"Fri Feb 9 17:21:59 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.39000000000000001,"15min":0.376},"min":{"1min":0.28599999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.51600000000000001,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.40400000000000003,"15min":0.39900000000000002},"min":{"1min":0.28100000000000003,"5min":0.245,"15min":0.245},"max":{"1min":0.55700000000000005,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.29399999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:23:30 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.45200000000000001,"15min":0.45500000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.52400000000000002,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.314},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.437,"15min":0.42599999999999999},"min":{"1min":0.31,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.58799999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.29999999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:23:55 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.623,"15min":0.60399999999999998},"min":{"1min":0.47499999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.83599999999999997,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.62,"15min":0.60599999999999998},"min":{"1min":0.498,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.81399999999999995,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.72399999999999998}]},{"osd":5,"last update":"Fri Feb 9 17:23:15 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61299999999999999,"15min":0.60299999999999998},"min":{"1min":0.42799999999999999,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.89100000000000001,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.626,"15min":0.61799999999999999},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.82399999999999995,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.66300000000000003}]},{"osd":6,"last update":"Fri Feb 9 17:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.64500000000000002,"15min":0.65500000000000003},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78900000000000003,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.65400000000000003,"15min":0.66800000000000004},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.71499999999999997,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.63700000000000001}]},{"osd":7,"last update":"Fri Feb 9 17:23:04 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.68700000000000006,"15min":0.69699999999999995},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.745},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.69099999999999995,"15min":0.69599999999999995},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.58499999999999996}]},{"osd":8,"last update":"Fri Feb 9 17:23:30 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.73599999999999999,"15min":0.74199999999999999},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.86399999999999999,"5min":1.042,"15min":1.042},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.71999999999999997,"15min":0.72399999999999998},"min":{"1min":0.53800000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.88800000000000001,"5min":1.004,"15min":1.004},"last":0.85199999999999998}]},{"osd":9,"last update":"Fri Feb 9 17:23:55 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.74099999999999999,"15min":0.751},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.82899999999999996,"5min":1.056,"15min":1.056},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.755,"15min":0.76200000000000001},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.98799999999999999,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.79600000000000004}]},{"osd":10,"last update":"Fri Feb 9 17:23:17 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.77900000000000003}]},{"osd":11,"last update":"Fri Feb 9 17:23:48 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.75900000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411394,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31920,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713232,"statfs":{"total":95995035648,"available":95962349568,"internally_reserved":0,"allocated":757760,"data_stored":397229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:23:02 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.40500000000000003,"15min":0.38200000000000001},"min":{"1min":0.26400000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.443,"5min":0.41199999999999998,"15min":0.377},"min":{"1min":0.28999999999999998,"5min":0.246,"15min":0.246},"max":{"1min":0.73999999999999999,"5min":0.747,"15min":0.747},"last":0.39500000000000002}]},{"osd":1,"last update":"Fri Feb 9 17:23:02 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.41499999999999998,"15min":0.38700000000000001},"min":{"1min":0.32400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.42699999999999999,"15min":0.39200000000000002},"min":{"1min":0.28000000000000003,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.46000000000000002}]},{"osd":3,"last update":"Fri Feb 9 17:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.45800000000000002,"15min":0.44400000000000001},"min":{"1min":0.314,"5min":0.312,"15min":0.312},"max":{"1min":0.71799999999999997,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.44800000000000001,"15min":0.44400000000000001},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.67400000000000004,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.442}]},{"osd":4,"last update":"Fri Feb 9 17:23:47 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.60299999999999998,"15min":0.58499999999999996},"min":{"1min":0.44,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.86399999999999999,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.63600000000000001,"15min":0.61299999999999999},"min":{"1min":0.45500000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.84099999999999997,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.55900000000000005}]},{"osd":5,"last update":"Fri Feb 9 17:23:16 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.622},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.96299999999999997,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.64000000000000001,"15min":0.621},"min":{"1min":0.504,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.53300000000000003}]},{"osd":6,"last update":"Fri Feb 9 17:23:37 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.67600000000000005,"15min":0.66800000000000004},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.69099999999999995,"15min":0.69199999999999995},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.52300000000000002}]},{"osd":7,"last update":"Fri Feb 9 17:23:02 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.68100000000000005,"15min":0.68300000000000005},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.88,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.70399999999999996}]},{"osd":8,"last update":"Fri Feb 9 17:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.76100000000000001,"15min":0.76600000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.79200000000000004,"15min":0.80000000000000004},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.82199999999999995}]},{"osd":9,"last update":"Fri Feb 9 17:23:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.74399999999999999,"15min":0.747},"min":{"1min":0.55200000000000005,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.0609999999999999,"5min":1.111,"15min":1.111},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.745,"5min":0.76500000000000001,"15min":0.76800000000000002},"min":{"1min":0.56000000000000005,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.80100000000000005}]},{"osd":10,"last update":"Fri Feb 9 17:23:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.89100000000000001}]}]},{"osd":3,"up_from":27,"seq":115964117053,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27760,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717392,"statfs":{"total":95995035648,"available":95966609408,"internally_reserved":0,"allocated":757760,"data_stored":397229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:22:21 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.39700000000000002,"15min":0.38800000000000001},"min":{"1min":0.33300000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.503,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.436,"15min":0.434},"min":{"1min":0.34899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.55100000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.46600000000000003}]},{"osd":1,"last update":"Fri Feb 9 17:23:26 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.434,"15min":0.441},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.57899999999999996,"5min":0.628,"15min":0.628},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.441,"15min":0.44600000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.59599999999999997,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.249}]},{"osd":2,"last update":"Fri Feb 9 17:23:26 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.46300000000000002,"15min":0.46700000000000003},"min":{"1min":0.29899999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.69299999999999995,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.44,"5min":0.45000000000000001,"15min":0.442},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.33300000000000002}]},{"osd":4,"last update":"Fri Feb 9 17:23:51 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.60599999999999998,"15min":0.58699999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.81999999999999995,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.59099999999999997,"15min":0.58499999999999996},"min":{"1min":0.36499999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.72799999999999998,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.51900000000000002}]},{"osd":5,"last update":"Fri Feb 9 17:23:13 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63800000000000001,"15min":0.63400000000000001},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.90300000000000002,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.63200000000000001,"15min":0.61599999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.60799999999999998}]},{"osd":6,"last update":"Fri Feb 9 17:23:39 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.65000000000000002,"15min":0.65800000000000003},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.79800000000000004,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.67200000000000004,"15min":0.68000000000000005},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.161,"5min":1.161,"15min":1.161},"last":0.59699999999999998}]},{"osd":7,"last update":"Fri Feb 9 17:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.69999999999999996,"15min":0.69899999999999995},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.876,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.69499999999999995,"15min":0.69899999999999995},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.64700000000000002}]},{"osd":8,"last update":"Fri Feb 9 17:23:27 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.53900000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.014,"5min":1.151,"15min":1.151},"last":0.5},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.78000000000000003,"15min":0.78400000000000003},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.69099999999999995}]},{"osd":9,"last update":"Fri Feb 9 17:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.78800000000000003,"15min":0.78400000000000003},"min":{"1min":0.56999999999999995,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.78300000000000003,"15min":0.79000000000000004},"min":{"1min":0.57599999999999996,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.92900000000000005,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.67500000000000004}]},{"osd":10,"last update":"Fri Feb 9 17:23:20 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.54200000000000004}]},{"osd":11,"last update":"Fri Feb 9 17:23:47 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.63600000000000001}]}]},{"osd":4,"up_from":32,"seq":137438953528,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27760,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717392,"statfs":{"total":95995035648,"available":95966609408,"internally_reserved":0,"allocated":757760,"data_stored":397229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:23:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.58499999999999996,"15min":0.57099999999999995},"min":{"1min":0.41899999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.73099999999999998,"5min":1.056,"15min":1.056},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.63,"5min":0.59899999999999998,"15min":0.58599999999999997},"min":{"1min":0.46200000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.90500000000000003,"5min":1.179,"15min":1.179},"last":0.56699999999999995}]},{"osd":1,"last update":"Fri Feb 9 17:23:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.58899999999999997,"15min":0.57799999999999996},"min":{"1min":0.439,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.73299999999999998,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.505},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.59599999999999997,"15min":0.58799999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.88700000000000001,"5min":1.113,"15min":1.113},"last":0.71099999999999997}]},{"osd":2,"last update":"Fri Feb 9 17:23:47 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.623,"15min":0.58899999999999997},"min":{"1min":0.441,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.83199999999999996,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.628},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60399999999999998,"15min":0.59999999999999998},"min":{"1min":0.45600000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.76300000000000001,"5min":1.196,"15min":1.196},"last":0.58899999999999997}]},{"osd":3,"last update":"Fri Feb 9 17:23:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.63600000000000001,"15min":0.61799999999999999},"min":{"1min":0.42199999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.76000000000000001,"5min":1.151,"15min":1.151},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61699999999999999,"15min":0.60099999999999998},"min":{"1min":0.48399999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.76900000000000002,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.60299999999999998}]},{"osd":5,"last update":"Fri Feb 9 17:23:13 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.46000000000000002,"15min":0.44700000000000001},"min":{"1min":0.35799999999999998,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.68600000000000005,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.52900000000000003,"15min":0.47699999999999998},"min":{"1min":0.36599999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.91600000000000004,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.80400000000000005}]},{"osd":6,"last update":"Fri Feb 9 17:23:42 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.51100000000000001,"15min":0.47599999999999998},"min":{"1min":0.33100000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.51600000000000001,"15min":0.48499999999999999},"min":{"1min":0.35299999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.73099999999999998,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.69799999999999995}]},{"osd":7,"last update":"Fri Feb 9 17:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.60599999999999998,"15min":0.59999999999999998},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.60799999999999998,"15min":0.60199999999999998},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.74099999999999999}]},{"osd":8,"last update":"Fri Feb 9 17:23:24 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.76300000000000001,"15min":0.75900000000000001},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.96799999999999997,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":1.194},{"interface":"front","average":{"1min":0.753,"5min":0.74099999999999999,"15min":0.73899999999999999},"min":{"1min":0.60499999999999998,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":0.97499999999999998,"5min":1.02,"15min":1.02},"last":1.1779999999999999}]},{"osd":9,"last update":"Fri Feb 9 17:23:54 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.81599999999999995,"15min":0.81299999999999994},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.139,"5min":1.139,"15min":1.139},"last":0.96799999999999997},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.97699999999999998,"5min":1.02,"15min":1.02},"last":1.0489999999999999}]},{"osd":10,"last update":"Fri Feb 9 17:23:19 2024","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":1.095},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.90600000000000003}]}]},{"osd":5,"up_from":40,"seq":171798691890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27760,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717392,"statfs":{"total":95995035648,"available":95966609408,"internally_reserved":0,"allocated":757760,"data_stored":397229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:23:13 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.57399999999999995,"15min":0.54300000000000004},"min":{"1min":0.42499999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.81699999999999995,"5min":0.877,"15min":0.877},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.54800000000000004,"15min":0.52300000000000002},"min":{"1min":0.46700000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.72499999999999998,"5min":0.753,"15min":0.753},"last":0.52100000000000002}]},{"osd":1,"last update":"Fri Feb 9 17:23:13 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.56499999999999995,"15min":0.52000000000000002},"min":{"1min":0.48999999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.89100000000000001,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.56000000000000005,"15min":0.54300000000000004},"min":{"1min":0.40400000000000003,"5min":0.379,"15min":0.379},"max":{"1min":0.78400000000000003,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.58599999999999997}]},{"osd":2,"last update":"Fri Feb 9 17:23:13 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.57499999999999996,"15min":0.55300000000000005},"min":{"1min":0.40799999999999997,"5min":0.373,"15min":0.373},"max":{"1min":0.78900000000000003,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.625},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.626,"15min":0.59399999999999997},"min":{"1min":0.51200000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.83499999999999996,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.73399999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:23:13 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.57699999999999996,"15min":0.55300000000000005},"min":{"1min":0.47899999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.83499999999999996,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.59999999999999998,"15min":0.56599999999999995},"min":{"1min":0.55000000000000004,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.92100000000000004,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.66300000000000003}]},{"osd":4,"last update":"Fri Feb 9 17:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.41399999999999998,"15min":0.377},"min":{"1min":0.254,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.71599999999999997,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.44500000000000001,"15min":0.40300000000000002},"min":{"1min":0.35099999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.76800000000000002,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.48399999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:23:40 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.50900000000000001,"15min":0.49299999999999999},"min":{"1min":0.309,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.80800000000000005,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.52800000000000002,"15min":0.48799999999999999},"min":{"1min":0.36599999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.83799999999999997,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.75600000000000001}]},{"osd":7,"last update":"Fri Feb 9 17:23:04 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.60999999999999999,"15min":0.60799999999999998},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.85599999999999998,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.59299999999999997,"15min":0.58699999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.87,"5min":0.872,"15min":0.872},"last":0.64000000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.72499999999999998,"15min":0.71799999999999997},"min":{"1min":0.58299999999999996,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":0.94699999999999995,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.78500000000000003,"15min":0.78300000000000003},"min":{"1min":0.66800000000000004,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":0.94499999999999995,"5min":1.097,"15min":1.097},"last":0.78400000000000003}]},{"osd":9,"last update":"Fri Feb 9 17:23:55 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":0.96199999999999997,"5min":1.171,"15min":1.171},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.83499999999999996,"15min":0.84099999999999997},"min":{"1min":0.63700000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":0.997,"5min":1.083,"15min":1.083},"last":0.875}]},{"osd":10,"last update":"Fri Feb 9 17:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.85399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-09T17:24:08.539 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-02-09T17:24:08.539 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-02-09T17:24:08.539 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-02-09T17:24:08.539 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph health --format=json 2024-02-09T17:24:08.800 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:24:09.702 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:09 smithi099 ceph-mon[39420]: from='client.14697 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:24:09.702 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:09 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:24:09.703 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:09 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:24:09.703 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:09 smithi099 ceph-mon[39420]: pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:09.703 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:09 smithi099 ceph-mon[39420]: osdmap e105: 12 total, 12 up, 12 in 2024-02-09T17:24:09.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:09 smithi179 ceph-mon[43975]: from='client.14697 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:24:09.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:09 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:24:09.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:09 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:24:09.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:09 smithi179 ceph-mon[43975]: pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:09.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:09 smithi179 ceph-mon[43975]: osdmap e105: 12 total, 12 up, 12 in 2024-02-09T17:24:09.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:09 smithi162 ceph-mon[44304]: from='client.14697 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:24:09.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:09 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:24:09.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:09 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:24:09.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:09 smithi162 ceph-mon[44304]: pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:09.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:09 smithi162 ceph-mon[44304]: osdmap e105: 12 total, 12 up, 12 in 2024-02-09T17:24:10.928 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:24:10.928 INFO:teuthology.orchestra.run.smithi099.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-09T17:24:11.600 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:11 smithi099 ceph-mon[39420]: pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:11.600 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:11 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3792797712' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-09T17:24:11.631 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-02-09T17:24:11.631 INFO:tasks.cephadm:Setup complete, yielding 2024-02-09T17:24:11.632 INFO:teuthology.run_tasks:Running task thrashosds... 2024-02-09T17:24:11.641 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-02-09T17:24:11.642 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-yuri2-testing-2024-02-06-1154'}, 'ceph': {'conf': {'client.2': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(MON_DOWN\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN'], 'sha1': '720625d38f7f95e6fa31efce87133247d9d28517'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '720625d38f7f95e6fa31efce87133247d9d28517'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'wip-tracker-64343', 'sha1': '9dda454f26f61ca0da16ee8ddca7ba78eb9abae7'}} 2024-02-09T17:24:11.642 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'min_in': 4} 2024-02-09T17:24:11.642 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-02-09T17:24:11.907 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:24:11.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:11 smithi179 ceph-mon[43975]: pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:11.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:11 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3792797712' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-09T17:24:11.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:11 smithi162 ceph-mon[44304]: pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:11.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:11 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3792797712' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-09T17:24:13.730 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:13 smithi099 ceph-mon[39420]: pgmap v329: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:13.730 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:13 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:24:13.730 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:13 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:24:13.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:13 smithi179 ceph-mon[43975]: pgmap v329: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:13.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:13 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:24:13.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:13 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:24:13.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:13 smithi162 ceph-mon[44304]: pgmap v329: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:13.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:13 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:24:13.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:13 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:24:14.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:14 smithi099 ceph-mon[39420]: osdmap e106: 12 total, 12 up, 12 in 2024-02-09T17:24:14.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:14 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4205282853' entity='client.admin' 2024-02-09T17:24:14.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:14 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:24:14.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:14 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:24:14.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:14 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:24:14.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:14 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:24:14.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:14 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:24:14.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:14 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:24:14.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:14 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:24:14.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:14 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:24:14.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:14 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:24:14.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:14 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:24:14.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:14 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:24:14.920 INFO:tasks.thrashosds:Beginning thrashosds... 2024-02-09T17:24:14.920 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-02-09T17:24:14.921 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph pg dump --format=json 2024-02-09T17:24:14.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:14 smithi179 ceph-mon[43975]: osdmap e106: 12 total, 12 up, 12 in 2024-02-09T17:24:14.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:14 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4205282853' entity='client.admin' 2024-02-09T17:24:14.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:14 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:24:14.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:14 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:24:14.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:14 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:24:14.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:14 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:24:14.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:14 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:24:14.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:14 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:24:14.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:14 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:24:14.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:14 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:24:14.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:14 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:24:14.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:14 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:24:14.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:14 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:24:14.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:14 smithi162 ceph-mon[44304]: osdmap e106: 12 total, 12 up, 12 in 2024-02-09T17:24:14.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:14 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4205282853' entity='client.admin' 2024-02-09T17:24:14.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:14 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:24:14.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:14 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:24:14.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:14 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:24:14.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:14 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:24:14.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:14 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:24:14.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:14 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:24:14.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:14 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:24:14.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:14 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:24:14.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:14 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:24:14.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:14 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:24:14.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:14 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:24:15.183 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:24:15.720 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:15 smithi099 ceph-mon[39420]: pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:15.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:15 smithi179 ceph-mon[43975]: pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:15.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:15 smithi162 ceph-mon[44304]: pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:17.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:17 smithi099 ceph-mon[39420]: pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:17.282 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:24:17.284 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-02-09T17:24:17.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:17 smithi179 ceph-mon[43975]: pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:17.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:17 smithi162 ceph-mon[44304]: pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:18.098 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":332,"stamp":"2024-02-09T17:24:16.324200+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":342388,"kb_used_data":9228,"kb_used_omap":18,"kb_used_meta":332397,"kb_avail":1124599436,"statfs":{"total":1151940427776,"available":1151589822464,"internally_reserved":0,"allocated":9449472,"data_stored":5875071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":340374841},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001624"},"pg_stats":[{"pgid":"1.0","version":"63'139","reported_seq":379,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-02-09T17:24:09.075696+0000","last_change":"2024-02-09T17:23:26.651048+0000","last_active":"2024-02-09T17:24:09.075696+0000","last_peered":"2024-02-09T17:24:09.075696+0000","last_clean":"2024-02-09T17:24:09.075696+0000","last_became_active":"2024-02-09T17:22:46.986816+0000","last_became_peered":"2024-02-09T17:22:46.986816+0000","last_unstale":"2024-02-09T17:24:09.075696+0000","last_undegraded":"2024-02-09T17:24:09.075696+0000","last_fullsized":"2024-02-09T17:24:09.075696+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:23:26.650945+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:23:26.650945+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:24:46.463688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056662,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28292,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716860,"statfs":{"total":95995035648,"available":95966064640,"internally_reserved":0,"allocated":1179648,"data_stored":881967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.378},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.53300000000000003}]},{"osd":1,"last update":"Fri Feb 9 17:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.51400000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.47199999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.443}]},{"osd":4,"last update":"Fri Feb 9 17:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":0.60899999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.75800000000000001}]},{"osd":6,"last update":"Fri Feb 9 17:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.70999999999999996}]},{"osd":7,"last update":"Fri Feb 9 17:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.66100000000000003}]},{"osd":8,"last update":"Fri Feb 9 17:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.72799999999999998}]},{"osd":10,"last update":"Fri Feb 9 17:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.69399999999999995}]}]},{"osd":10,"up_from":84,"seq":360777252891,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27772,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717380,"statfs":{"total":95995035648,"available":95966597120,"internally_reserved":0,"allocated":647168,"data_stored":353039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":1,"last update":"Fri Feb 9 17:23:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.64000000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:23:17 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.55600000000000005}]},{"osd":3,"last update":"Fri Feb 9 17:23:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.58599999999999997}]},{"osd":4,"last update":"Fri Feb 9 17:23:17 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.87},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1,"5min":1,"15min":1},"last":0.63300000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:23:17 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.95199999999999996}]},{"osd":6,"last update":"Fri Feb 9 17:23:17 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":1.0389999999999999}]},{"osd":7,"last update":"Fri Feb 9 17:23:17 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":1.1040000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:23:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":1.069},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.745}]},{"osd":9,"last update":"Fri Feb 9 17:23:17 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":1.1659999999999999},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":1.208}]},{"osd":11,"last update":"Fri Feb 9 17:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":1.147},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":1.1279999999999999}]}]},{"osd":9,"up_from":77,"seq":330712481825,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27664,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717488,"statfs":{"total":95995035648,"available":95966707712,"internally_reserved":0,"allocated":659456,"data_stored":360416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.55200000000000005,"15min":0.55800000000000005},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.71799999999999997,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.55000000000000004,"15min":0.55200000000000005},"min":{"1min":0.315,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.75600000000000001,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.44900000000000001}]},{"osd":1,"last update":"Fri Feb 9 17:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.55600000000000005,"15min":0.56399999999999995},"min":{"1min":0.35999999999999999,"5min":0.318,"15min":0.318},"max":{"1min":0.78100000000000003,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.53800000000000003}]},{"osd":2,"last update":"Fri Feb 9 17:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56899999999999995,"15min":0.57099999999999995},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.70999999999999996,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.56499999999999995,"15min":0.56799999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.89600000000000002,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.61499999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.57299999999999995,"15min":0.57699999999999996},"min":{"1min":0.37,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.78800000000000003,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.56000000000000005,"15min":0.55200000000000005},"min":{"1min":0.45400000000000001,"5min":0.373,"15min":0.373},"max":{"1min":0.75600000000000001,"5min":0.88,"15min":0.88},"last":0.49199999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.67200000000000004,"15min":0.66400000000000003},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.81200000000000006}]},{"osd":5,"last update":"Fri Feb 9 17:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.66000000000000003,"15min":0.64800000000000002},"min":{"1min":0.49399999999999999,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.75},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.66900000000000004,"15min":0.66000000000000003},"min":{"1min":0.49399999999999999,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.86699999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.67900000000000005,"15min":0.66700000000000004},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.68300000000000005,"15min":0.67700000000000005},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.84199999999999997}]},{"osd":7,"last update":"Fri Feb 9 17:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.70599999999999996,"15min":0.69699999999999995},"min":{"1min":0.55400000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.67700000000000005,"15min":0.66200000000000003},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.82899999999999996}]},{"osd":8,"last update":"Fri Feb 9 17:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.53900000000000003,"15min":0.51900000000000002},"min":{"1min":0.47899999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.56499999999999995,"15min":0.54400000000000004},"min":{"1min":0.498,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.73099999999999998}]},{"osd":10,"last update":"Fri Feb 9 17:23:20 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.76300000000000001}]}]},{"osd":8,"up_from":66,"seq":283467841575,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27664,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717488,"statfs":{"total":95995035648,"available":95966707712,"internally_reserved":0,"allocated":659456,"data_stored":360436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.58399999999999996,"15min":0.58199999999999996},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.56899999999999995,"15min":0.58099999999999996},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.751,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.56299999999999994}]},{"osd":1,"last update":"Fri Feb 9 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.56000000000000005,"15min":0.56299999999999994},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.57099999999999995,"15min":0.57899999999999996},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.40600000000000003}]},{"osd":2,"last update":"Fri Feb 9 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.56399999999999995,"15min":0.56899999999999995},"min":{"1min":0.38400000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.72599999999999998,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.44900000000000001}]},{"osd":3,"last update":"Fri Feb 9 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.59399999999999997,"15min":0.60399999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.78400000000000003,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.59499999999999997,"15min":0.60499999999999998},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.67900000000000005,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.505}]},{"osd":4,"last update":"Fri Feb 9 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.63,"15min":0.623},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.85599999999999998,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.67100000000000004,"15min":0.66300000000000003},"min":{"1min":0.55300000000000005,"5min":0.44,"15min":0.44},"max":{"1min":0.92000000000000004,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.75}]},{"osd":5,"last update":"Fri Feb 9 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.65700000000000003,"15min":0.65100000000000002},"min":{"1min":0.504,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.84999999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.64600000000000002,"15min":0.63700000000000001},"min":{"1min":0.52100000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.69099999999999995}]},{"osd":6,"last update":"Fri Feb 9 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.65500000000000003,"15min":0.64700000000000002},"min":{"1min":0.51900000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.65600000000000003,"15min":0.64600000000000002},"min":{"1min":0.57999999999999996,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.78900000000000003}]},{"osd":7,"last update":"Fri Feb 9 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.63400000000000001,"15min":0.61799999999999999},"min":{"1min":0.58399999999999996,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.67300000000000004,"15min":0.66800000000000004},"min":{"1min":0.47799999999999998,"5min":0.438,"15min":0.438},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.80600000000000005}]},{"osd":9,"last update":"Fri Feb 9 17:23:51 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.52900000000000003,"15min":0.51700000000000002},"min":{"1min":0.38800000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.76900000000000002,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.55400000000000005,"15min":0.53600000000000003},"min":{"1min":0.45600000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.61299999999999999}]},{"osd":10,"last update":"Fri Feb 9 17:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.67600000000000005}]}]},{"osd":7,"up_from":57,"seq":244813135914,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28292,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716860,"statfs":{"total":95995035648,"available":95966064640,"internally_reserved":0,"allocated":1179648,"data_stored":881987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.622},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.67100000000000004}]},{"osd":1,"last update":"Fri Feb 9 17:24:03 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.60899999999999999,"15min":0.60499999999999998},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.871,"5min":1.113,"15min":1.113},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.624,"5min":0.63100000000000001,"15min":0.63300000000000001},"min":{"1min":0.438,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.82899999999999996,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.61599999999999999}]},{"osd":2,"last update":"Fri Feb 9 17:24:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.59499999999999997,"15min":0.60899999999999999},"min":{"1min":0.45500000000000002,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.65400000000000003,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.40300000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.76100000000000001,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.52100000000000002}]},{"osd":3,"last update":"Fri Feb 9 17:24:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.63500000000000001,"15min":0.63900000000000001},"min":{"1min":0.42999999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.78700000000000003,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.629,"15min":0.64500000000000002},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.72699999999999998,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.5}]},{"osd":4,"last update":"Fri Feb 9 17:24:03 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.76300000000000001,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.54300000000000004,"15min":0.54100000000000004},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.66300000000000003,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.55500000000000005}]},{"osd":5,"last update":"Fri Feb 9 17:24:03 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.54000000000000004,"15min":0.55100000000000005},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63500000000000001,"15min":0.63600000000000001},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.82799999999999996,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.41099999999999998}]},{"osd":6,"last update":"Fri Feb 9 17:24:03 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.60199999999999998,"15min":0.60999999999999999},"min":{"1min":0.42299999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.751,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.622,"15min":0.63},"min":{"1min":0.442,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.76900000000000002,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.57499999999999996}]},{"osd":8,"last update":"Fri Feb 9 17:23:25 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.78000000000000003,"15min":0.77300000000000002},"min":{"1min":0.59599999999999997,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.77800000000000002,"15min":0.77600000000000002},"min":{"1min":0.56599999999999995,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.64000000000000001}]},{"osd":9,"last update":"Fri Feb 9 17:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.75800000000000001,"15min":0.753},"min":{"1min":0.65300000000000002,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.70499999999999996}]},{"osd":10,"last update":"Fri Feb 9 17:23:19 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.73899999999999999}]},{"osd":11,"last update":"Fri Feb 9 17:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.88,"15min":0.88},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.66700000000000004}]}]},{"osd":6,"up_from":49,"seq":210453397552,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28184,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716968,"statfs":{"total":95995035648,"available":95966175232,"internally_reserved":0,"allocated":1191936,"data_stored":889364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:23:38 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":0.77000000000000002,"15min":0.66000000000000003},"min":{"1min":0.36799999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":24.544,"5min":24.544,"15min":24.544},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.624,"5min":0.64600000000000002,"15min":0.64500000000000002},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.80800000000000005,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.63100000000000001}]},{"osd":1,"last update":"Fri Feb 9 17:23:38 2024","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":0.74199999999999999,"15min":0.63100000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":23.954000000000001,"5min":23.954000000000001,"15min":23.954000000000001},"last":0.53300000000000003},{"interface":"front","average":{"1min":1.3560000000000001,"5min":0.75800000000000001,"15min":0.64900000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":23.984999999999999,"5min":23.984999999999999,"15min":23.984999999999999},"last":0.52200000000000002}]},{"osd":2,"last update":"Fri Feb 9 17:23:38 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":0.75900000000000001,"15min":0.66500000000000004},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":24.329999999999998,"5min":24.329999999999998,"15min":24.329999999999998},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.67300000000000004,"15min":0.61099999999999999},"min":{"1min":0.34699999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":12.906000000000001,"5min":12.906000000000001,"15min":12.906000000000001},"last":0.5}]},{"osd":3,"last update":"Fri Feb 9 17:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.59899999999999998,"15min":0.58499999999999996},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":2.609,"5min":2.609,"15min":2.609},"last":0.61299999999999999},{"interface":"front","average":{"1min":1.3540000000000001,"5min":0.77100000000000002,"15min":0.66100000000000003},"min":{"1min":0.40899999999999997,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":24.219000000000001,"5min":24.219000000000001,"15min":24.219000000000001},"last":0.46600000000000003}]},{"osd":4,"last update":"Fri Feb 9 17:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.497,"15min":0.48099999999999998},"min":{"1min":0.26400000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.505,"15min":0.48399999999999999},"min":{"1min":0.33800000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.74399999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.52400000000000002,"15min":0.50600000000000001},"min":{"1min":0.31900000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.52400000000000002,"15min":0.504},"min":{"1min":0.35599999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.81200000000000006,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.65300000000000002}]},{"osd":7,"last update":"Fri Feb 9 17:24:09 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.58999999999999997,"15min":0.59899999999999998},"min":{"1min":0.375,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.66900000000000004,"15min":0.67100000000000004},"min":{"1min":0.40300000000000002,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.97699999999999998,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.71599999999999997}]},{"osd":8,"last update":"Fri Feb 9 17:23:27 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.76300000000000001,"15min":0.75900000000000001},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.73399999999999999,"15min":0.73199999999999998},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.0469999999999999,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.68999999999999995}]},{"osd":9,"last update":"Fri Feb 9 17:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"min":{"1min":0.60599999999999998,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.79200000000000004}]},{"osd":10,"last update":"Fri Feb 9 17:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.73099999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574921,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31824,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713328,"statfs":{"total":95995035648,"available":95962447872,"internally_reserved":0,"allocated":659456,"data_stored":360436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:23:47 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.376,"15min":0.33500000000000002},"min":{"1min":0.255,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.53000000000000003,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.441,"5min":0.442,"15min":0.35999999999999999},"min":{"1min":0.30499999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.64800000000000002,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.371}]},{"osd":2,"last update":"Fri Feb 9 17:24:12 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.39300000000000002,"15min":0.38900000000000001},"min":{"1min":0.27700000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.308},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.40600000000000003,"15min":0.39800000000000002},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.54900000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.34799999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:23:34 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.46400000000000002,"15min":0.46200000000000002},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.58399999999999996,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.46600000000000003,"15min":0.46700000000000003},"min":{"1min":0.32200000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.56599999999999995,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.40699999999999997}]},{"osd":4,"last update":"Fri Feb 9 17:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.58599999999999997,"15min":0.56200000000000006},"min":{"1min":0.41399999999999998,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.78200000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.61599999999999999,"15min":0.57199999999999995},"min":{"1min":0.45400000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.77500000000000002,"5min":1.028,"15min":1.028},"last":0.44500000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:23:16 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.624,"15min":0.60999999999999999},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.79600000000000004,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.5},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.64500000000000002,"15min":0.627},"min":{"1min":0.51900000000000002,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.47999999999999998}]},{"osd":6,"last update":"Fri Feb 9 17:23:47 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.66900000000000004,"15min":0.66300000000000003},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.83199999999999996,"5min":1.095,"15min":1.095},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.67000000000000004,"15min":0.68300000000000005},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.76800000000000002,"5min":1.044,"15min":1.044},"last":0.52700000000000002}]},{"osd":7,"last update":"Fri Feb 9 17:23:08 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.72699999999999998,"15min":0.73199999999999998},"min":{"1min":0.54800000000000004,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.76900000000000002,"15min":0.77400000000000002},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.67600000000000005}]},{"osd":8,"last update":"Fri Feb 9 17:23:31 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.73299999999999998,"15min":0.73399999999999999},"min":{"1min":0.55800000000000005,"5min":0.501,"15min":0.501},"max":{"1min":0.875,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.044,"5min":1.115,"15min":1.115},"last":0.78100000000000003}]},{"osd":9,"last update":"Fri Feb 9 17:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.77600000000000002,"15min":0.78800000000000003},"min":{"1min":0.54800000000000004,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.89300000000000002,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.75700000000000001,"15min":0.76200000000000001},"min":{"1min":0.58099999999999996,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.95899999999999996,"5min":1.21,"15min":1.21},"last":0.82199999999999995}]},{"osd":10,"last update":"Fri Feb 9 17:23:17 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.1719999999999999,"5min":1.1719999999999999,"15min":1.1719999999999999},"last":0.84099999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738446,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27772,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717380,"statfs":{"total":95995035648,"available":95966597120,"internally_reserved":0,"allocated":647168,"data_stored":353059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 9 17:23:40 2024","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.36199999999999999,"15min":0.33500000000000002},"min":{"1min":0.253,"5min":0.223,"15min":0.223},"max":{"1min":0.39200000000000002,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.378},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.39700000000000002,"15min":0.36699999999999999},"min":{"1min":0.26000000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.48499999999999999,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.35299999999999998}]},{"osd":2,"last update":"Fri Feb 9 17:21:59 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.39000000000000001,"15min":0.376},"min":{"1min":0.28599999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.51600000000000001,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.40400000000000003,"15min":0.39900000000000002},"min":{"1min":0.28100000000000003,"5min":0.245,"15min":0.245},"max":{"1min":0.55700000000000005,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.29399999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:23:30 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.45200000000000001,"15min":0.45500000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.52400000000000002,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.437,"15min":0.42599999999999999},"min":{"1min":0.31,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.58799999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.32400000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:23:55 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.623,"15min":0.60399999999999998},"min":{"1min":0.47499999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.83599999999999997,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.62,"15min":0.60599999999999998},"min":{"1min":0.498,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.81399999999999995,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.65400000000000003}]},{"osd":5,"last update":"Fri Feb 9 17:23:15 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61299999999999999,"15min":0.60299999999999998},"min":{"1min":0.42799999999999999,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.89100000000000001,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.626,"15min":0.61799999999999999},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.82399999999999995,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.63700000000000001}]},{"osd":6,"last update":"Fri Feb 9 17:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.64500000000000002,"15min":0.65500000000000003},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78900000000000003,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.65400000000000003,"15min":0.66800000000000004},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.71499999999999997,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.64600000000000002}]},{"osd":7,"last update":"Fri Feb 9 17:24:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.66300000000000003,"15min":0.68899999999999995},"min":{"1min":0.42399999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.745,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.628},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.66700000000000004,"15min":0.68799999999999994},"min":{"1min":0.495,"5min":0.437,"15min":0.437},"max":{"1min":0.82599999999999996,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.53400000000000003}]},{"osd":8,"last update":"Fri Feb 9 17:23:30 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.73599999999999999,"15min":0.74199999999999999},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.86399999999999999,"5min":1.042,"15min":1.042},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.71999999999999997,"15min":0.72399999999999998},"min":{"1min":0.53800000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.88800000000000001,"5min":1.004,"15min":1.004},"last":0.47399999999999998}]},{"osd":9,"last update":"Fri Feb 9 17:23:55 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.74099999999999999,"15min":0.751},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.82899999999999996,"5min":1.056,"15min":1.056},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.755,"15min":0.76200000000000001},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.98799999999999999,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.42399999999999999}]},{"osd":10,"last update":"Fri Feb 9 17:23:17 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.436}]},{"osd":11,"last update":"Fri Feb 9 17:23:48 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.56699999999999995}]}]},{"osd":2,"up_from":18,"seq":77309411396,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31824,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713328,"statfs":{"total":95995035648,"available":95962447872,"internally_reserved":0,"allocated":659456,"data_stored":360436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:24:06 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.40300000000000002,"15min":0.38100000000000001},"min":{"1min":0.253,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.54400000000000004,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.378,"5min":0.41599999999999998,"15min":0.378},"min":{"1min":0.29799999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.47199999999999998,"5min":0.747,"15min":0.747},"last":0.313}]},{"osd":1,"last update":"Fri Feb 9 17:24:06 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.41199999999999998,"15min":0.38600000000000001},"min":{"1min":0.28100000000000003,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.42799999999999999,"15min":0.39200000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.63400000000000001,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.35199999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.45800000000000002,"15min":0.44400000000000001},"min":{"1min":0.314,"5min":0.312,"15min":0.312},"max":{"1min":0.71799999999999997,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.44800000000000001,"15min":0.44400000000000001},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.67400000000000004,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.28499999999999998}]},{"osd":4,"last update":"Fri Feb 9 17:23:47 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.60299999999999998,"15min":0.58499999999999996},"min":{"1min":0.44,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.86399999999999999,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.63600000000000001,"15min":0.61299999999999999},"min":{"1min":0.45500000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.84099999999999997,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.61299999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:23:16 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.622},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.96299999999999997,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.64000000000000001,"15min":0.621},"min":{"1min":0.504,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.48799999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:23:37 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.67600000000000005,"15min":0.66800000000000004},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.69099999999999995,"15min":0.69199999999999995},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.57999999999999996}]},{"osd":7,"last update":"Fri Feb 9 17:24:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.65800000000000003,"15min":0.67500000000000004},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.72799999999999998,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.69599999999999995,"15min":0.70399999999999996},"min":{"1min":0.499,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.79700000000000004,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.68100000000000005}]},{"osd":8,"last update":"Fri Feb 9 17:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.76100000000000001,"15min":0.76600000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.88300000000000001},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.79200000000000004,"15min":0.80000000000000004},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.56100000000000005}]},{"osd":9,"last update":"Fri Feb 9 17:23:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.74399999999999999,"15min":0.747},"min":{"1min":0.55200000000000005,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.0609999999999999,"5min":1.111,"15min":1.111},"last":0.94399999999999995},{"interface":"front","average":{"1min":0.745,"5min":0.76500000000000001,"15min":0.76800000000000002},"min":{"1min":0.56000000000000005,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.83699999999999997}]},{"osd":10,"last update":"Fri Feb 9 17:23:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.64900000000000002}]}]},{"osd":3,"up_from":27,"seq":115964117055,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27664,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717488,"statfs":{"total":95995035648,"available":95966707712,"internally_reserved":0,"allocated":659456,"data_stored":360436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:22:21 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.39700000000000002,"15min":0.38800000000000001},"min":{"1min":0.33300000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.503,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.436,"15min":0.434},"min":{"1min":0.34899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.55100000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.46600000000000003}]},{"osd":1,"last update":"Fri Feb 9 17:23:26 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.434,"15min":0.441},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.57899999999999996,"5min":0.628,"15min":0.628},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.441,"15min":0.44600000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.59599999999999997,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.46700000000000003}]},{"osd":2,"last update":"Fri Feb 9 17:23:26 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.46300000000000002,"15min":0.46700000000000003},"min":{"1min":0.29899999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.69299999999999995,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.44,"5min":0.45000000000000001,"15min":0.442},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.311}]},{"osd":4,"last update":"Fri Feb 9 17:23:51 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.60599999999999998,"15min":0.58699999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.81999999999999995,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.59099999999999997,"15min":0.58499999999999996},"min":{"1min":0.36499999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.72799999999999998,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.36599999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:24:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.627,"15min":0.63},"min":{"1min":0.45100000000000001,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.70099999999999996,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.504},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.623,"15min":0.61299999999999999},"min":{"1min":0.40300000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.66300000000000003,"5min":0.999,"15min":0.999},"last":0.435}]},{"osd":6,"last update":"Fri Feb 9 17:23:39 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.65000000000000002,"15min":0.65800000000000003},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.79800000000000004,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.67200000000000004,"15min":0.68000000000000005},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.161,"5min":1.161,"15min":1.161},"last":0.53000000000000003}]},{"osd":7,"last update":"Fri Feb 9 17:24:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.67800000000000005,"15min":0.69199999999999995},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.67900000000000005,"15min":0.69299999999999995},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.89000000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.45200000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:23:27 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.53900000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.014,"5min":1.151,"15min":1.151},"last":0.747},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.78000000000000003,"15min":0.78400000000000003},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.66900000000000004}]},{"osd":9,"last update":"Fri Feb 9 17:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.78800000000000003,"15min":0.78400000000000003},"min":{"1min":0.56999999999999995,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.78300000000000003,"15min":0.79000000000000004},"min":{"1min":0.57599999999999996,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.92900000000000005,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.629}]},{"osd":10,"last update":"Fri Feb 9 17:23:20 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.64600000000000002}]},{"osd":11,"last update":"Fri Feb 9 17:23:47 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.73299999999999998}]}]},{"osd":4,"up_from":32,"seq":137438953530,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27664,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717488,"statfs":{"total":95995035648,"available":95966707712,"internally_reserved":0,"allocated":659456,"data_stored":360436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:23:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.58499999999999996,"15min":0.57099999999999995},"min":{"1min":0.41899999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.73099999999999998,"5min":1.056,"15min":1.056},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.63,"5min":0.59899999999999998,"15min":0.58599999999999997},"min":{"1min":0.46200000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.90500000000000003,"5min":1.179,"15min":1.179},"last":0.57199999999999995}]},{"osd":1,"last update":"Fri Feb 9 17:23:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.58899999999999997,"15min":0.57799999999999996},"min":{"1min":0.439,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.73299999999999998,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.59599999999999997,"15min":0.58799999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.88700000000000001,"5min":1.113,"15min":1.113},"last":0.59399999999999997}]},{"osd":2,"last update":"Fri Feb 9 17:23:47 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.623,"15min":0.58899999999999997},"min":{"1min":0.441,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.83199999999999996,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60399999999999998,"15min":0.59999999999999998},"min":{"1min":0.45600000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.76300000000000001,"5min":1.196,"15min":1.196},"last":0.53300000000000003}]},{"osd":3,"last update":"Fri Feb 9 17:23:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.63600000000000001,"15min":0.61799999999999999},"min":{"1min":0.42199999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.76000000000000001,"5min":1.151,"15min":1.151},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61699999999999999,"15min":0.60099999999999998},"min":{"1min":0.48399999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.76900000000000002,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.5}]},{"osd":5,"last update":"Fri Feb 9 17:24:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.47999999999999998,"15min":0.45400000000000001},"min":{"1min":0.39900000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.66800000000000004,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.56499999999999995,"15min":0.48899999999999999},"min":{"1min":0.36099999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.80500000000000005,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.51000000000000001}]},{"osd":6,"last update":"Fri Feb 9 17:23:42 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.51100000000000001,"15min":0.47599999999999998},"min":{"1min":0.33100000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.51600000000000001,"15min":0.48499999999999999},"min":{"1min":0.35299999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.73099999999999998,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.48499999999999999}]},{"osd":7,"last update":"Fri Feb 9 17:24:09 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.61899999999999999,"15min":0.60399999999999998},"min":{"1min":0.44900000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.94499999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.61499999999999999,"15min":0.60499999999999998},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.77100000000000002,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.39900000000000002}]},{"osd":8,"last update":"Fri Feb 9 17:23:24 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.76300000000000001,"15min":0.75900000000000001},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.96799999999999997,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.753,"5min":0.74099999999999999,"15min":0.73899999999999999},"min":{"1min":0.60499999999999998,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":0.97499999999999998,"5min":1.02,"15min":1.02},"last":0.68100000000000005}]},{"osd":9,"last update":"Fri Feb 9 17:23:54 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.81599999999999995,"15min":0.81299999999999994},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.139,"5min":1.139,"15min":1.139},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.97699999999999998,"5min":1.02,"15min":1.02},"last":0.72799999999999998}]},{"osd":10,"last update":"Fri Feb 9 17:23:19 2024","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.751}]}]},{"osd":5,"up_from":40,"seq":171798691892,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27772,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717380,"statfs":{"total":95995035648,"available":95966597120,"internally_reserved":0,"allocated":647168,"data_stored":353059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:23:13 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.57399999999999995,"15min":0.54300000000000004},"min":{"1min":0.42499999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.81699999999999995,"5min":0.877,"15min":0.877},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.54800000000000004,"15min":0.52300000000000002},"min":{"1min":0.46700000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.72499999999999998,"5min":0.753,"15min":0.753},"last":0.52000000000000002}]},{"osd":1,"last update":"Fri Feb 9 17:23:13 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.56499999999999995,"15min":0.52000000000000002},"min":{"1min":0.48999999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.89100000000000001,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.56000000000000005,"15min":0.54300000000000004},"min":{"1min":0.40400000000000003,"5min":0.379,"15min":0.379},"max":{"1min":0.78400000000000003,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.42699999999999999}]},{"osd":2,"last update":"Fri Feb 9 17:23:13 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.57499999999999996,"15min":0.55300000000000005},"min":{"1min":0.40799999999999997,"5min":0.373,"15min":0.373},"max":{"1min":0.78900000000000003,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.626,"15min":0.59399999999999997},"min":{"1min":0.51200000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.83499999999999996,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.61599999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:23:13 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.57699999999999996,"15min":0.55300000000000005},"min":{"1min":0.47899999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.83499999999999996,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.59999999999999998,"15min":0.56599999999999995},"min":{"1min":0.55000000000000004,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.92100000000000004,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.56000000000000005}]},{"osd":4,"last update":"Fri Feb 9 17:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.41399999999999998,"15min":0.377},"min":{"1min":0.254,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.71599999999999997,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.44500000000000001,"15min":0.40300000000000002},"min":{"1min":0.35099999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.76800000000000002,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.375}]},{"osd":6,"last update":"Fri Feb 9 17:23:40 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.50900000000000001,"15min":0.49299999999999999},"min":{"1min":0.309,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.80800000000000005,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.52800000000000002,"15min":0.48799999999999999},"min":{"1min":0.36599999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.83799999999999997,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.495}]},{"osd":7,"last update":"Fri Feb 9 17:24:07 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.62,"15min":0.61199999999999999},"min":{"1min":0.33300000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.94799999999999995,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.60299999999999998,"15min":0.58999999999999997},"min":{"1min":0.39000000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.54800000000000004}]},{"osd":8,"last update":"Fri Feb 9 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.72499999999999998,"15min":0.71799999999999997},"min":{"1min":0.58299999999999996,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":0.94699999999999995,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.78500000000000003,"15min":0.78300000000000003},"min":{"1min":0.66800000000000004,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":0.94499999999999995,"5min":1.097,"15min":1.097},"last":0.69099999999999995}]},{"osd":9,"last update":"Fri Feb 9 17:23:55 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":0.96199999999999997,"5min":1.171,"15min":1.171},"last":0.628},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.83499999999999996,"15min":0.84099999999999997},"min":{"1min":0.63700000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":0.997,"5min":1.083,"15min":1.083},"last":0.71099999999999997}]},{"osd":10,"last update":"Fri Feb 9 17:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.60499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-09T17:24:18.100 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph pg dump --format=json 2024-02-09T17:24:18.364 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:24:18.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:18 smithi179 ceph-mon[43975]: from='client.14715 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:24:18.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:18 smithi162 ceph-mon[44304]: from='client.14715 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:24:18.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:18 smithi099 ceph-mon[39420]: from='client.14715 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:24:19.295 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:19 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:24:19.295 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:19 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:24:19.295 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:19 smithi099 ceph-mon[39420]: pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:19.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:19 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:24:19.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:19 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:24:19.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:19 smithi179 ceph-mon[43975]: pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:19.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:19 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:24:19.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:19 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:24:19.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:19 smithi162 ceph-mon[44304]: pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:20.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:20 smithi099 ceph-mon[39420]: osdmap e107: 12 total, 12 up, 12 in 2024-02-09T17:24:20.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:20 smithi179 ceph-mon[43975]: osdmap e107: 12 total, 12 up, 12 in 2024-02-09T17:24:20.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:20 smithi162 ceph-mon[44304]: osdmap e107: 12 total, 12 up, 12 in 2024-02-09T17:24:20.430 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:24:20.432 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-02-09T17:24:21.200 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:21 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:24:21.200 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:21 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:24:21.200 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:21 smithi099 ceph-mon[39420]: osdmap e108: 12 total, 12 up, 12 in 2024-02-09T17:24:21.200 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:21 smithi099 ceph-mon[39420]: pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:21.201 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:21 smithi099 ceph-mon[39420]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:24:21.227 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":336,"stamp":"2024-02-09T17:24:20.324880+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":341872,"kb_used_data":9312,"kb_used_omap":18,"kb_used_meta":332397,"kb_avail":1124599952,"statfs":{"total":1151940427776,"available":1151590350848,"internally_reserved":0,"allocated":9535488,"data_stored":5926710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":340374841},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.849625"},"pg_stats":[{"pgid":"1.0","version":"63'139","reported_seq":381,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-02-09T17:24:13.956529+0000","last_change":"2024-02-09T17:23:26.651048+0000","last_active":"2024-02-09T17:24:13.956529+0000","last_peered":"2024-02-09T17:24:13.956529+0000","last_clean":"2024-02-09T17:24:13.956529+0000","last_became_active":"2024-02-09T17:22:46.986816+0000","last_became_peered":"2024-02-09T17:22:46.986816+0000","last_unstale":"2024-02-09T17:24:13.956529+0000","last_undegraded":"2024-02-09T17:24:13.956529+0000","last_fullsized":"2024-02-09T17:24:13.956529+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:23:26.650945+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:23:26.650945+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:24:46.463688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056663,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28184,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716968,"statfs":{"total":95995035648,"available":95966175232,"internally_reserved":0,"allocated":1191936,"data_stored":889344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.71699999999999997}]},{"osd":1,"last update":"Fri Feb 9 17:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.627},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.56899999999999995}]},{"osd":2,"last update":"Fri Feb 9 17:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.54600000000000004}]},{"osd":3,"last update":"Fri Feb 9 17:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.65500000000000003}]},{"osd":4,"last update":"Fri Feb 9 17:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":0.85299999999999998}]},{"osd":5,"last update":"Fri Feb 9 17:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.753},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.92400000000000004}]},{"osd":6,"last update":"Fri Feb 9 17:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.89100000000000001}]},{"osd":7,"last update":"Fri Feb 9 17:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.871},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.80200000000000005}]},{"osd":8,"last update":"Fri Feb 9 17:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.81899999999999995}]},{"osd":10,"last update":"Fri Feb 9 17:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.58699999999999997}]}]},{"osd":10,"up_from":84,"seq":360777252892,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27664,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717488,"statfs":{"total":95995035648,"available":95966707712,"internally_reserved":0,"allocated":659456,"data_stored":360416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":1,"last update":"Fri Feb 9 17:23:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.64100000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:23:17 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.755},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.57199999999999995}]},{"osd":3,"last update":"Fri Feb 9 17:23:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.84499999999999997}]},{"osd":4,"last update":"Fri Feb 9 17:23:17 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.875},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1,"5min":1,"15min":1},"last":0.74099999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:23:17 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.92500000000000004}]},{"osd":6,"last update":"Fri Feb 9 17:23:17 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.97799999999999998}]},{"osd":7,"last update":"Fri Feb 9 17:23:17 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.93999999999999995}]},{"osd":8,"last update":"Fri Feb 9 17:23:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.67800000000000005}]},{"osd":9,"last update":"Fri Feb 9 17:23:17 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.98999999999999999}]},{"osd":11,"last update":"Fri Feb 9 17:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":1.004},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.95699999999999996}]}]},{"osd":9,"up_from":77,"seq":330712481825,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27664,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717488,"statfs":{"total":95995035648,"available":95966707712,"internally_reserved":0,"allocated":659456,"data_stored":360416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.55200000000000005,"15min":0.55800000000000005},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.71799999999999997,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.55000000000000004,"15min":0.55200000000000005},"min":{"1min":0.315,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.75600000000000001,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.44900000000000001}]},{"osd":1,"last update":"Fri Feb 9 17:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.55600000000000005,"15min":0.56399999999999995},"min":{"1min":0.35999999999999999,"5min":0.318,"15min":0.318},"max":{"1min":0.78100000000000003,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.53800000000000003}]},{"osd":2,"last update":"Fri Feb 9 17:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56899999999999995,"15min":0.57099999999999995},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.70999999999999996,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.56499999999999995,"15min":0.56799999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.89600000000000002,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.61499999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.57299999999999995,"15min":0.57699999999999996},"min":{"1min":0.37,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.78800000000000003,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.56000000000000005,"15min":0.55200000000000005},"min":{"1min":0.45400000000000001,"5min":0.373,"15min":0.373},"max":{"1min":0.75600000000000001,"5min":0.88,"15min":0.88},"last":0.49199999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.67200000000000004,"15min":0.66400000000000003},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.81200000000000006}]},{"osd":5,"last update":"Fri Feb 9 17:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.66000000000000003,"15min":0.64800000000000002},"min":{"1min":0.49399999999999999,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.75},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.66900000000000004,"15min":0.66000000000000003},"min":{"1min":0.49399999999999999,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.86699999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.67900000000000005,"15min":0.66700000000000004},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.68300000000000005,"15min":0.67700000000000005},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.84199999999999997}]},{"osd":7,"last update":"Fri Feb 9 17:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.70599999999999996,"15min":0.69699999999999995},"min":{"1min":0.55400000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.67700000000000005,"15min":0.66200000000000003},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.82899999999999996}]},{"osd":8,"last update":"Fri Feb 9 17:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.53900000000000003,"15min":0.51900000000000002},"min":{"1min":0.47899999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.56499999999999995,"15min":0.54400000000000004},"min":{"1min":0.498,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.73099999999999998}]},{"osd":10,"last update":"Fri Feb 9 17:23:20 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.76300000000000001}]}]},{"osd":8,"up_from":66,"seq":283467841576,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717476,"statfs":{"total":95995035648,"available":95966695424,"internally_reserved":0,"allocated":671744,"data_stored":367813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.58399999999999996,"15min":0.58199999999999996},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.56899999999999995,"15min":0.58099999999999996},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.751,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.56799999999999995}]},{"osd":1,"last update":"Fri Feb 9 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.56000000000000005,"15min":0.56299999999999994},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.57099999999999995,"15min":0.57899999999999996},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.61299999999999999}]},{"osd":2,"last update":"Fri Feb 9 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.56399999999999995,"15min":0.56899999999999995},"min":{"1min":0.38400000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.72599999999999998,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.59699999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.59399999999999997,"15min":0.60399999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.78400000000000003,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.59499999999999997,"15min":0.60499999999999998},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.67900000000000005,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.67100000000000004}]},{"osd":4,"last update":"Fri Feb 9 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.63,"15min":0.623},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.85599999999999998,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.67100000000000004,"15min":0.66300000000000003},"min":{"1min":0.55300000000000005,"5min":0.44,"15min":0.44},"max":{"1min":0.92000000000000004,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.73099999999999998}]},{"osd":5,"last update":"Fri Feb 9 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.65700000000000003,"15min":0.65100000000000002},"min":{"1min":0.504,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.84999999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.64600000000000002,"15min":0.63700000000000001},"min":{"1min":0.52100000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.69699999999999995}]},{"osd":6,"last update":"Fri Feb 9 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.65500000000000003,"15min":0.64700000000000002},"min":{"1min":0.51900000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.65600000000000003,"15min":0.64600000000000002},"min":{"1min":0.57999999999999996,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.70699999999999996}]},{"osd":7,"last update":"Fri Feb 9 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.63400000000000001,"15min":0.61799999999999999},"min":{"1min":0.58399999999999996,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.67300000000000004,"15min":0.66800000000000004},"min":{"1min":0.47799999999999998,"5min":0.438,"15min":0.438},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.745}]},{"osd":9,"last update":"Fri Feb 9 17:23:51 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.52900000000000003,"15min":0.51700000000000002},"min":{"1min":0.38800000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.76900000000000002,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.55400000000000005,"15min":0.53600000000000003},"min":{"1min":0.45600000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.71999999999999997}]},{"osd":10,"last update":"Fri Feb 9 17:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.77700000000000002}]}]},{"osd":7,"up_from":57,"seq":244813135915,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28184,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716968,"statfs":{"total":95995035648,"available":95966175232,"internally_reserved":0,"allocated":1191936,"data_stored":889364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.622},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.67100000000000004}]},{"osd":1,"last update":"Fri Feb 9 17:24:03 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.60899999999999999,"15min":0.60499999999999998},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.871,"5min":1.113,"15min":1.113},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.624,"5min":0.63100000000000001,"15min":0.63300000000000001},"min":{"1min":0.438,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.82899999999999996,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.55900000000000005}]},{"osd":2,"last update":"Fri Feb 9 17:24:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.59499999999999997,"15min":0.60899999999999999},"min":{"1min":0.45500000000000002,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.65400000000000003,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.40300000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.76100000000000001,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.53600000000000003}]},{"osd":3,"last update":"Fri Feb 9 17:24:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.63500000000000001,"15min":0.63900000000000001},"min":{"1min":0.42999999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.78700000000000003,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.629,"15min":0.64500000000000002},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.72699999999999998,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.40000000000000002}]},{"osd":4,"last update":"Fri Feb 9 17:24:03 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.76300000000000001,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.54300000000000004,"15min":0.54100000000000004},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.66300000000000003,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.61599999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:24:03 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.54000000000000004,"15min":0.55100000000000005},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63500000000000001,"15min":0.63600000000000001},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.82799999999999996,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.60099999999999998}]},{"osd":6,"last update":"Fri Feb 9 17:24:03 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.60199999999999998,"15min":0.60999999999999999},"min":{"1min":0.42299999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.751,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.496},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.622,"15min":0.63},"min":{"1min":0.442,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.76900000000000002,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.58099999999999996}]},{"osd":8,"last update":"Fri Feb 9 17:23:25 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.78000000000000003,"15min":0.77300000000000002},"min":{"1min":0.59599999999999997,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.77800000000000002,"15min":0.77600000000000002},"min":{"1min":0.56599999999999995,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.73199999999999998}]},{"osd":9,"last update":"Fri Feb 9 17:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.75800000000000001,"15min":0.753},"min":{"1min":0.65300000000000002,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.64200000000000002}]},{"osd":10,"last update":"Fri Feb 9 17:23:19 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.85699999999999998}]},{"osd":11,"last update":"Fri Feb 9 17:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.88,"15min":0.88},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.89900000000000002}]}]},{"osd":6,"up_from":49,"seq":210453397552,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28184,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716968,"statfs":{"total":95995035648,"available":95966175232,"internally_reserved":0,"allocated":1191936,"data_stored":889364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:23:38 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":0.77000000000000002,"15min":0.66000000000000003},"min":{"1min":0.36799999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":24.544,"5min":24.544,"15min":24.544},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.624,"5min":0.64600000000000002,"15min":0.64500000000000002},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.80800000000000005,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.63100000000000001}]},{"osd":1,"last update":"Fri Feb 9 17:23:38 2024","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":0.74199999999999999,"15min":0.63100000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":23.954000000000001,"5min":23.954000000000001,"15min":23.954000000000001},"last":0.53300000000000003},{"interface":"front","average":{"1min":1.3560000000000001,"5min":0.75800000000000001,"15min":0.64900000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":23.984999999999999,"5min":23.984999999999999,"15min":23.984999999999999},"last":0.52200000000000002}]},{"osd":2,"last update":"Fri Feb 9 17:23:38 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":0.75900000000000001,"15min":0.66500000000000004},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":24.329999999999998,"5min":24.329999999999998,"15min":24.329999999999998},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.67300000000000004,"15min":0.61099999999999999},"min":{"1min":0.34699999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":12.906000000000001,"5min":12.906000000000001,"15min":12.906000000000001},"last":0.5}]},{"osd":3,"last update":"Fri Feb 9 17:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.59899999999999998,"15min":0.58499999999999996},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":2.609,"5min":2.609,"15min":2.609},"last":0.61299999999999999},{"interface":"front","average":{"1min":1.3540000000000001,"5min":0.77100000000000002,"15min":0.66100000000000003},"min":{"1min":0.40899999999999997,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":24.219000000000001,"5min":24.219000000000001,"15min":24.219000000000001},"last":0.46600000000000003}]},{"osd":4,"last update":"Fri Feb 9 17:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.497,"15min":0.48099999999999998},"min":{"1min":0.26400000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.505,"15min":0.48399999999999999},"min":{"1min":0.33800000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.74399999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.52400000000000002,"15min":0.50600000000000001},"min":{"1min":0.31900000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.52400000000000002,"15min":0.504},"min":{"1min":0.35599999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.81200000000000006,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.65300000000000002}]},{"osd":7,"last update":"Fri Feb 9 17:24:09 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.58999999999999997,"15min":0.59899999999999998},"min":{"1min":0.375,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.66900000000000004,"15min":0.67100000000000004},"min":{"1min":0.40300000000000002,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.97699999999999998,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.71599999999999997}]},{"osd":8,"last update":"Fri Feb 9 17:23:27 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.76300000000000001,"15min":0.75900000000000001},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.73399999999999999,"15min":0.73199999999999998},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.0469999999999999,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.68999999999999995}]},{"osd":9,"last update":"Fri Feb 9 17:23:52 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"min":{"1min":0.60599999999999998,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.79200000000000004}]},{"osd":10,"last update":"Fri Feb 9 17:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.73099999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574922,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31824,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713328,"statfs":{"total":95995035648,"available":95962447872,"internally_reserved":0,"allocated":659456,"data_stored":360436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:23:47 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.376,"15min":0.33500000000000002},"min":{"1min":0.255,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.53000000000000003,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.441,"5min":0.442,"15min":0.35999999999999999},"min":{"1min":0.30499999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.64800000000000002,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.50800000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:24:12 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.39300000000000002,"15min":0.38900000000000001},"min":{"1min":0.27700000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.40600000000000003,"15min":0.39800000000000002},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.54900000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.47499999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:23:34 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.46400000000000002,"15min":0.46200000000000002},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.58399999999999996,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.46600000000000003,"15min":0.46700000000000003},"min":{"1min":0.32200000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.56599999999999995,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.53400000000000003}]},{"osd":4,"last update":"Fri Feb 9 17:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.58599999999999997,"15min":0.56200000000000006},"min":{"1min":0.41399999999999998,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.78200000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.61599999999999999,"15min":0.57199999999999995},"min":{"1min":0.45400000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.77500000000000002,"5min":1.028,"15min":1.028},"last":0.61099999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:23:16 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.624,"15min":0.60999999999999999},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.79600000000000004,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":1.0069999999999999},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.64500000000000002,"15min":0.627},"min":{"1min":0.51900000000000002,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":1.0189999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:23:47 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.66900000000000004,"15min":0.66300000000000003},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.83199999999999996,"5min":1.095,"15min":1.095},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.67000000000000004,"15min":0.68300000000000005},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.76800000000000002,"5min":1.044,"15min":1.044},"last":0.83899999999999997}]},{"osd":7,"last update":"Fri Feb 9 17:24:15 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.70899999999999996,"15min":0.72599999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.84699999999999998,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.753,"15min":0.76900000000000002},"min":{"1min":0.52200000000000002,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.99399999999999999,"5min":1.145,"15min":1.145},"last":0.99399999999999999}]},{"osd":8,"last update":"Fri Feb 9 17:23:31 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.73299999999999998,"15min":0.73399999999999999},"min":{"1min":0.55800000000000005,"5min":0.501,"15min":0.501},"max":{"1min":0.875,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.93700000000000006},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.044,"5min":1.115,"15min":1.115},"last":0.88300000000000001}]},{"osd":9,"last update":"Fri Feb 9 17:23:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.77600000000000002,"15min":0.78800000000000003},"min":{"1min":0.54800000000000004,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.89300000000000002,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.95799999999999996},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.75700000000000001,"15min":0.76200000000000001},"min":{"1min":0.58099999999999996,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.95899999999999996,"5min":1.21,"15min":1.21},"last":0.97199999999999998}]},{"osd":10,"last update":"Fri Feb 9 17:23:17 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.98499999999999999},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.1719999999999999,"5min":1.1719999999999999,"15min":1.1719999999999999},"last":1.0309999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738447,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27664,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717488,"statfs":{"total":95995035648,"available":95966707712,"internally_reserved":0,"allocated":659456,"data_stored":360436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 9 17:23:40 2024","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.36199999999999999,"15min":0.33500000000000002},"min":{"1min":0.253,"5min":0.223,"15min":0.223},"max":{"1min":0.39200000000000002,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.39700000000000002,"15min":0.36699999999999999},"min":{"1min":0.26000000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.48499999999999999,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.29899999999999999}]},{"osd":2,"last update":"Fri Feb 9 17:21:59 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.39000000000000001,"15min":0.376},"min":{"1min":0.28599999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.51600000000000001,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.40400000000000003,"15min":0.39900000000000002},"min":{"1min":0.28100000000000003,"5min":0.245,"15min":0.245},"max":{"1min":0.55700000000000005,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.29399999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:23:30 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.45200000000000001,"15min":0.45500000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.52400000000000002,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.437,"15min":0.42599999999999999},"min":{"1min":0.31,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.58799999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.34699999999999998}]},{"osd":4,"last update":"Fri Feb 9 17:23:55 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.623,"15min":0.60399999999999998},"min":{"1min":0.47499999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.83599999999999997,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.62,"15min":0.60599999999999998},"min":{"1min":0.498,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.81399999999999995,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.75}]},{"osd":5,"last update":"Fri Feb 9 17:23:15 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61299999999999999,"15min":0.60299999999999998},"min":{"1min":0.42799999999999999,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.89100000000000001,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.626,"15min":0.61799999999999999},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.82399999999999995,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.72199999999999998}]},{"osd":6,"last update":"Fri Feb 9 17:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.64500000000000002,"15min":0.65500000000000003},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78900000000000003,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.65400000000000003,"15min":0.66800000000000004},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.71499999999999997,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.63500000000000001}]},{"osd":7,"last update":"Fri Feb 9 17:24:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.66300000000000003,"15min":0.68899999999999995},"min":{"1min":0.42399999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.745,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.66700000000000004,"15min":0.68799999999999994},"min":{"1min":0.495,"5min":0.437,"15min":0.437},"max":{"1min":0.82599999999999996,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.60199999999999998}]},{"osd":8,"last update":"Fri Feb 9 17:23:30 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.73599999999999999,"15min":0.74199999999999999},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.86399999999999999,"5min":1.042,"15min":1.042},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.71999999999999997,"15min":0.72399999999999998},"min":{"1min":0.53800000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.88800000000000001,"5min":1.004,"15min":1.004},"last":0.52800000000000002}]},{"osd":9,"last update":"Fri Feb 9 17:23:55 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.74099999999999999,"15min":0.751},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.82899999999999996,"5min":1.056,"15min":1.056},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.755,"15min":0.76200000000000001},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.98799999999999999,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.44900000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:23:17 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.61499999999999999}]},{"osd":11,"last update":"Fri Feb 9 17:23:48 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.58699999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411397,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31836,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713316,"statfs":{"total":95995035648,"available":95962435584,"internally_reserved":0,"allocated":671744,"data_stored":367813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:24:06 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.40300000000000002,"15min":0.38100000000000001},"min":{"1min":0.253,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.54400000000000004,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.378,"5min":0.41599999999999998,"15min":0.378},"min":{"1min":0.29799999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.47199999999999998,"5min":0.747,"15min":0.747},"last":0.52400000000000002}]},{"osd":1,"last update":"Fri Feb 9 17:24:06 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.41199999999999998,"15min":0.38600000000000001},"min":{"1min":0.28100000000000003,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.502},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.42799999999999999,"15min":0.39200000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.63400000000000001,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.61599999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.45800000000000002,"15min":0.44400000000000001},"min":{"1min":0.314,"5min":0.312,"15min":0.312},"max":{"1min":0.71799999999999997,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.44800000000000001,"15min":0.44400000000000001},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.67400000000000004,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.60399999999999998}]},{"osd":4,"last update":"Fri Feb 9 17:23:47 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.60299999999999998,"15min":0.58499999999999996},"min":{"1min":0.44,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.86399999999999999,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.63600000000000001,"15min":0.61299999999999999},"min":{"1min":0.45500000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.84099999999999997,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.78300000000000003}]},{"osd":5,"last update":"Fri Feb 9 17:24:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.63200000000000001,"15min":0.61899999999999999},"min":{"1min":0.435,"5min":0.433,"15min":0.433},"max":{"1min":0.75,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.628,"15min":0.61699999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.70199999999999996,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.83299999999999996}]},{"osd":6,"last update":"Fri Feb 9 17:23:37 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.67600000000000005,"15min":0.66800000000000004},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.69099999999999995,"15min":0.69199999999999995},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.80000000000000004}]},{"osd":7,"last update":"Fri Feb 9 17:24:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.65800000000000003,"15min":0.67500000000000004},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.72799999999999998,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.69599999999999995,"15min":0.70399999999999996},"min":{"1min":0.499,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.79700000000000004,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.81399999999999995}]},{"osd":8,"last update":"Fri Feb 9 17:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.76100000000000001,"15min":0.76600000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.79200000000000004,"15min":0.80000000000000004},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.95699999999999996}]},{"osd":9,"last update":"Fri Feb 9 17:23:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.74399999999999999,"15min":0.747},"min":{"1min":0.55200000000000005,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.0609999999999999,"5min":1.111,"15min":1.111},"last":1.085},{"interface":"front","average":{"1min":0.745,"5min":0.76500000000000001,"15min":0.76800000000000002},"min":{"1min":0.56000000000000005,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.995}]},{"osd":10,"last update":"Fri Feb 9 17:23:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":1.0740000000000001},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.98599999999999999}]}]},{"osd":3,"up_from":27,"seq":115964117055,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27664,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717488,"statfs":{"total":95995035648,"available":95966707712,"internally_reserved":0,"allocated":659456,"data_stored":360436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:22:21 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.39700000000000002,"15min":0.38800000000000001},"min":{"1min":0.33300000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.503,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.436,"15min":0.434},"min":{"1min":0.34899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.55100000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.46600000000000003}]},{"osd":1,"last update":"Fri Feb 9 17:23:26 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.434,"15min":0.441},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.57899999999999996,"5min":0.628,"15min":0.628},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.441,"15min":0.44600000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.59599999999999997,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.46700000000000003}]},{"osd":2,"last update":"Fri Feb 9 17:23:26 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.46300000000000002,"15min":0.46700000000000003},"min":{"1min":0.29899999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.69299999999999995,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.44,"5min":0.45000000000000001,"15min":0.442},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.311}]},{"osd":4,"last update":"Fri Feb 9 17:23:51 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.60599999999999998,"15min":0.58699999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.81999999999999995,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.59099999999999997,"15min":0.58499999999999996},"min":{"1min":0.36499999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.72799999999999998,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.36599999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:24:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.627,"15min":0.63},"min":{"1min":0.45100000000000001,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.70099999999999996,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.504},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.623,"15min":0.61299999999999999},"min":{"1min":0.40300000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.66300000000000003,"5min":0.999,"15min":0.999},"last":0.435}]},{"osd":6,"last update":"Fri Feb 9 17:23:39 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.65000000000000002,"15min":0.65800000000000003},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.79800000000000004,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.67200000000000004,"15min":0.68000000000000005},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.161,"5min":1.161,"15min":1.161},"last":0.53000000000000003}]},{"osd":7,"last update":"Fri Feb 9 17:24:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.67800000000000005,"15min":0.69199999999999995},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.67900000000000005,"15min":0.69299999999999995},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.89000000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.45200000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:23:27 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.53900000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.014,"5min":1.151,"15min":1.151},"last":0.747},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.78000000000000003,"15min":0.78400000000000003},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.66900000000000004}]},{"osd":9,"last update":"Fri Feb 9 17:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.78800000000000003,"15min":0.78400000000000003},"min":{"1min":0.56999999999999995,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.78300000000000003,"15min":0.79000000000000004},"min":{"1min":0.57599999999999996,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.92900000000000005,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.629}]},{"osd":10,"last update":"Fri Feb 9 17:23:20 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.64600000000000002}]},{"osd":11,"last update":"Fri Feb 9 17:23:47 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.73299999999999998}]}]},{"osd":4,"up_from":32,"seq":137438953531,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27664,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717488,"statfs":{"total":95995035648,"available":95966707712,"internally_reserved":0,"allocated":659456,"data_stored":360436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:23:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.58499999999999996,"15min":0.57099999999999995},"min":{"1min":0.41899999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.73099999999999998,"5min":1.056,"15min":1.056},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.63,"5min":0.59899999999999998,"15min":0.58599999999999997},"min":{"1min":0.46200000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.90500000000000003,"5min":1.179,"15min":1.179},"last":0.57399999999999995}]},{"osd":1,"last update":"Fri Feb 9 17:23:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.58899999999999997,"15min":0.57799999999999996},"min":{"1min":0.439,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.73299999999999998,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.59599999999999997,"15min":0.58799999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.88700000000000001,"5min":1.113,"15min":1.113},"last":0.61899999999999999}]},{"osd":2,"last update":"Fri Feb 9 17:23:47 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.623,"15min":0.58899999999999997},"min":{"1min":0.441,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.83199999999999996,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60399999999999998,"15min":0.59999999999999998},"min":{"1min":0.45600000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.76300000000000001,"5min":1.196,"15min":1.196},"last":0.55600000000000005}]},{"osd":3,"last update":"Fri Feb 9 17:23:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.63600000000000001,"15min":0.61799999999999999},"min":{"1min":0.42199999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.76000000000000001,"5min":1.151,"15min":1.151},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61699999999999999,"15min":0.60099999999999998},"min":{"1min":0.48399999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.76900000000000002,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.46100000000000002}]},{"osd":5,"last update":"Fri Feb 9 17:24:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.47999999999999998,"15min":0.45400000000000001},"min":{"1min":0.39900000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.66800000000000004,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.56499999999999995,"15min":0.48899999999999999},"min":{"1min":0.36099999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.80500000000000005,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.54200000000000004}]},{"osd":6,"last update":"Fri Feb 9 17:23:42 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.51100000000000001,"15min":0.47599999999999998},"min":{"1min":0.33100000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.51600000000000001,"15min":0.48499999999999999},"min":{"1min":0.35299999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.73099999999999998,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.38100000000000001}]},{"osd":7,"last update":"Fri Feb 9 17:24:09 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.61899999999999999,"15min":0.60399999999999998},"min":{"1min":0.44900000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.94499999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.61499999999999999,"15min":0.60499999999999998},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.77100000000000002,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.437}]},{"osd":8,"last update":"Fri Feb 9 17:23:24 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.76300000000000001,"15min":0.75900000000000001},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.96799999999999997,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.753,"5min":0.74099999999999999,"15min":0.73899999999999999},"min":{"1min":0.60499999999999998,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":0.97499999999999998,"5min":1.02,"15min":1.02},"last":0.76800000000000002}]},{"osd":9,"last update":"Fri Feb 9 17:23:54 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.81599999999999995,"15min":0.81299999999999994},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.139,"5min":1.139,"15min":1.139},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.97699999999999998,"5min":1.02,"15min":1.02},"last":0.60699999999999998}]},{"osd":10,"last update":"Fri Feb 9 17:23:19 2024","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.78200000000000003}]}]},{"osd":5,"up_from":40,"seq":171798691893,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27664,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717488,"statfs":{"total":95995035648,"available":95966707712,"internally_reserved":0,"allocated":659456,"data_stored":360436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:24:15 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.58999999999999997,"15min":0.54800000000000004},"min":{"1min":0.36599999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.81499999999999995,"5min":0.877,"15min":0.877},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.56100000000000005,"15min":0.52700000000000002},"min":{"1min":0.443,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.70799999999999996,"5min":0.753,"15min":0.753},"last":0.70699999999999996}]},{"osd":1,"last update":"Fri Feb 9 17:24:15 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57999999999999996,"15min":0.52500000000000002},"min":{"1min":0.40300000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.77700000000000002,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.57599999999999996,"15min":0.54800000000000004},"min":{"1min":0.42699999999999999,"5min":0.379,"15min":0.379},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.53000000000000003}]},{"osd":2,"last update":"Fri Feb 9 17:24:15 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.61399999999999999,"15min":0.56599999999999995},"min":{"1min":0.47399999999999998,"5min":0.373,"15min":0.373},"max":{"1min":2.8410000000000002,"5min":2.8410000000000002,"15min":2.8410000000000002},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.66600000000000004,"15min":0.60699999999999998},"min":{"1min":0.47699999999999998,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":2.855,"5min":2.855,"15min":2.855},"last":0.73099999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:24:15 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.58799999999999997,"15min":0.55600000000000005},"min":{"1min":0.41199999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.73999999999999999,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.625,"15min":0.57499999999999996},"min":{"1min":0.505,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.88600000000000001,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.82599999999999996}]},{"osd":4,"last update":"Fri Feb 9 17:24:18 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.45300000000000001,"15min":0.39000000000000001},"min":{"1min":0.36399999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.79500000000000004,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.48999999999999999,"15min":0.41799999999999998},"min":{"1min":0.375,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.89100000000000001,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.752}]},{"osd":6,"last update":"Fri Feb 9 17:23:40 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.50900000000000001,"15min":0.49299999999999999},"min":{"1min":0.309,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.80800000000000005,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.52800000000000002,"15min":0.48799999999999999},"min":{"1min":0.36599999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.83799999999999997,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.85199999999999998}]},{"osd":7,"last update":"Fri Feb 9 17:24:07 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.62,"15min":0.61199999999999999},"min":{"1min":0.33300000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.94799999999999995,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.876},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.60299999999999998,"15min":0.58999999999999997},"min":{"1min":0.39000000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.77100000000000002}]},{"osd":8,"last update":"Fri Feb 9 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.72499999999999998,"15min":0.71799999999999997},"min":{"1min":0.58299999999999996,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":0.94699999999999995,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":1.0449999999999999},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.78500000000000003,"15min":0.78300000000000003},"min":{"1min":0.66800000000000004,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":0.94499999999999995,"5min":1.097,"15min":1.097},"last":1.0900000000000001}]},{"osd":9,"last update":"Fri Feb 9 17:23:55 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":0.96199999999999997,"5min":1.171,"15min":1.171},"last":1.0189999999999999},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.83499999999999996,"15min":0.84099999999999997},"min":{"1min":0.63700000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":0.997,"5min":1.083,"15min":1.083},"last":0.95699999999999996}]},{"osd":10,"last update":"Fri Feb 9 17:24:18 2024","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":0.82199999999999995,"15min":0.81299999999999994},"min":{"1min":0.66100000000000003,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":1.1180000000000001},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.79700000000000004,"15min":0.78900000000000003},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.141,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":1.103}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-09T17:24:21.230 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-02-09T17:24:21.230 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd dump 2024-02-09T17:24:21.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:21 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:24:21.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:21 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:24:21.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:21 smithi179 ceph-mon[43975]: osdmap e108: 12 total, 12 up, 12 in 2024-02-09T17:24:21.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:21 smithi179 ceph-mon[43975]: pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:21.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:21 smithi179 ceph-mon[43975]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:24:21.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:21 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:24:21.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:21 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:24:21.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:21 smithi162 ceph-mon[44304]: osdmap e108: 12 total, 12 up, 12 in 2024-02-09T17:24:21.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:21 smithi162 ceph-mon[44304]: pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:21.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:21 smithi162 ceph-mon[44304]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:24:21.493 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:24:23.529 INFO:teuthology.orchestra.run.smithi099.stdout:epoch 108 2024-02-09T17:24:23.530 INFO:teuthology.orchestra.run.smithi099.stdout:fsid a69c870a-c76e-11ee-95b7-87774f69a715 2024-02-09T17:24:23.530 INFO:teuthology.orchestra.run.smithi099.stdout:created 2024-02-09T17:15:26.113606+0000 2024-02-09T17:24:23.530 INFO:teuthology.orchestra.run.smithi099.stdout:modified 2024-02-09T17:24:20.009667+0000 2024-02-09T17:24:23.530 INFO:teuthology.orchestra.run.smithi099.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-02-09T17:24:23.530 INFO:teuthology.orchestra.run.smithi099.stdout:crush_version 26 2024-02-09T17:24:23.530 INFO:teuthology.orchestra.run.smithi099.stdout:full_ratio 0.95 2024-02-09T17:24:23.530 INFO:teuthology.orchestra.run.smithi099.stdout:backfillfull_ratio 0.9 2024-02-09T17:24:23.530 INFO:teuthology.orchestra.run.smithi099.stdout:nearfull_ratio 0.85 2024-02-09T17:24:23.530 INFO:teuthology.orchestra.run.smithi099.stdout:require_min_compat_client luminous 2024-02-09T17:24:23.530 INFO:teuthology.orchestra.run.smithi099.stdout:min_compat_client jewel 2024-02-09T17:24:23.530 INFO:teuthology.orchestra.run.smithi099.stdout:require_osd_release squid 2024-02-09T17:24:23.530 INFO:teuthology.orchestra.run.smithi099.stdout:stretch_mode_enabled false 2024-02-09T17:24:23.531 INFO:teuthology.orchestra.run.smithi099.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-02-09T17:24:23.531 INFO:teuthology.orchestra.run.smithi099.stdout:max_osd 12 2024-02-09T17:24:23.531 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0 up in weight 1 up_from 8 up_thru 59 down_at 0 last_clean_interval [0,0) [v2:172.21.15.99:6802/3859868923,v1:172.21.15.99:6803/3859868923] [v2:172.21.15.99:6804/3859868923,v1:172.21.15.99:6805/3859868923] exists,up 21739aa7-e278-45da-9a3f-867385f01baf 2024-02-09T17:24:23.531 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1 up in weight 1 up_from 13 up_thru 35 down_at 0 last_clean_interval [0,0) [v2:172.21.15.99:6810/3868328452,v1:172.21.15.99:6811/3868328452] [v2:172.21.15.99:6812/3868328452,v1:172.21.15.99:6813/3868328452] exists,up 8eab347c-9065-4de1-93ae-e2eb7e11cab7 2024-02-09T17:24:23.531 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.99:6818/537506237,v1:172.21.15.99:6819/537506237] [v2:172.21.15.99:6820/537506237,v1:172.21.15.99:6821/537506237] exists,up 575f4257-7e34-485e-963c-2607b69e1220 2024-02-09T17:24:23.531 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.99:6826/762784793,v1:172.21.15.99:6827/762784793] [v2:172.21.15.99:6828/762784793,v1:172.21.15.99:6829/762784793] exists,up 1a520bcf-81bb-4f1d-80fb-11e98008e429 2024-02-09T17:24:23.531 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4 up in weight 1 up_from 32 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.162:6800/4178676504,v1:172.21.15.162:6801/4178676504] [v2:172.21.15.162:6802/4178676504,v1:172.21.15.162:6803/4178676504] exists,up 4617c2fe-c9d8-40cd-887b-303e3215bdf8 2024-02-09T17:24:23.531 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5 up in weight 1 up_from 40 up_thru 43 down_at 0 last_clean_interval [0,0) [v2:172.21.15.162:6808/1489371339,v1:172.21.15.162:6809/1489371339] [v2:172.21.15.162:6810/1489371339,v1:172.21.15.162:6811/1489371339] exists,up da98e3f3-173e-46fe-be8b-1ccbe8b0dabc 2024-02-09T17:24:23.531 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6 up in weight 1 up_from 49 up_thru 52 down_at 0 last_clean_interval [0,0) [v2:172.21.15.162:6816/821169663,v1:172.21.15.162:6817/821169663] [v2:172.21.15.162:6818/821169663,v1:172.21.15.162:6819/821169663] exists,up ff50382d-5ddf-4cda-bb0c-ff055feeaa40 2024-02-09T17:24:23.531 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7 up in weight 1 up_from 57 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.162:6824/965264369,v1:172.21.15.162:6825/965264369] [v2:172.21.15.162:6826/965264369,v1:172.21.15.162:6827/965264369] exists,up a82984dc-5e22-424b-a446-ae2ee71eee19 2024-02-09T17:24:23.531 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8 up in weight 1 up_from 66 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.179:6800/4219839127,v1:172.21.15.179:6801/4219839127] [v2:172.21.15.179:6802/4219839127,v1:172.21.15.179:6803/4219839127] exists,up 63f9d79f-0400-44dc-b388-df00fefb5820 2024-02-09T17:24:23.531 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9 up in weight 1 up_from 77 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.179:6808/504427183,v1:172.21.15.179:6809/504427183] [v2:172.21.15.179:6810/504427183,v1:172.21.15.179:6811/504427183] exists,up 7d995854-7328-4dfc-b1a9-0edca8d74f46 2024-02-09T17:24:23.531 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10 up in weight 1 up_from 84 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.179:6816/2403345164,v1:172.21.15.179:6817/2403345164] [v2:172.21.15.179:6818/2403345164,v1:172.21.15.179:6819/2403345164] exists,up 151fbedd-08e6-46e6-a4e5-a14248399f61 2024-02-09T17:24:23.532 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11 up in weight 1 up_from 90 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.179:6824/2899402373,v1:172.21.15.179:6825/2899402373] [v2:172.21.15.179:6826/2899402373,v1:172.21.15.179:6827/2899402373] exists,up 6d3a3ad8-417a-483f-991c-52549dbe7d73 2024-02-09T17:24:23.532 INFO:teuthology.orchestra.run.smithi099.stdout:blocklist 172.21.15.99:0/3904172782 expires 2024-02-10T17:16:24.229365+0000 2024-02-09T17:24:23.532 INFO:teuthology.orchestra.run.smithi099.stdout:blocklist 172.21.15.99:6801/1586034576 expires 2024-02-10T17:16:24.229365+0000 2024-02-09T17:24:23.532 INFO:teuthology.orchestra.run.smithi099.stdout:blocklist 172.21.15.99:6800/746248609 expires 2024-02-10T17:15:54.399201+0000 2024-02-09T17:24:23.532 INFO:teuthology.orchestra.run.smithi099.stdout:blocklist 172.21.15.99:0/4092784197 expires 2024-02-10T17:16:24.229365+0000 2024-02-09T17:24:23.532 INFO:teuthology.orchestra.run.smithi099.stdout:blocklist 172.21.15.99:0/2844598101 expires 2024-02-10T17:15:54.399201+0000 2024-02-09T17:24:23.532 INFO:teuthology.orchestra.run.smithi099.stdout:blocklist 172.21.15.99:6801/746248609 expires 2024-02-10T17:15:54.399201+0000 2024-02-09T17:24:23.532 INFO:teuthology.orchestra.run.smithi099.stdout:blocklist 172.21.15.99:0/2528750050 expires 2024-02-10T17:15:54.399201+0000 2024-02-09T17:24:23.532 INFO:teuthology.orchestra.run.smithi099.stdout:blocklist 172.21.15.99:6800/1586034576 expires 2024-02-10T17:16:24.229365+0000 2024-02-09T17:24:23.532 INFO:teuthology.orchestra.run.smithi099.stdout:blocklist 172.21.15.99:0/2214140078 expires 2024-02-10T17:15:54.399201+0000 2024-02-09T17:24:23.532 INFO:teuthology.orchestra.run.smithi099.stdout:blocklist 172.21.15.99:0/4217821965 expires 2024-02-10T17:16:24.229365+0000 2024-02-09T17:24:23.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:23 smithi179 ceph-mon[43975]: pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:23.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:23 smithi162 ceph-mon[44304]: pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:23.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:23 smithi099 ceph-mon[39420]: pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:24.172 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 59 down_at 0 last_clean_interval [0,0) [v2:172.21.15.99:6802/3859868923,v1:172.21.15.99:6803/3859868923] [v2:172.21.15.99:6804/3859868923,v1:172.21.15.99:6805/3859868923] exists,up 21739aa7-e278-45da-9a3f-867385f01baf', 'osd.1 up in weight 1 up_from 13 up_thru 35 down_at 0 last_clean_interval [0,0) [v2:172.21.15.99:6810/3868328452,v1:172.21.15.99:6811/3868328452] [v2:172.21.15.99:6812/3868328452,v1:172.21.15.99:6813/3868328452] exists,up 8eab347c-9065-4de1-93ae-e2eb7e11cab7', 'osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.99:6818/537506237,v1:172.21.15.99:6819/537506237] [v2:172.21.15.99:6820/537506237,v1:172.21.15.99:6821/537506237] exists,up 575f4257-7e34-485e-963c-2607b69e1220', 'osd.3 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.99:6826/762784793,v1:172.21.15.99:6827/762784793] [v2:172.21.15.99:6828/762784793,v1:172.21.15.99:6829/762784793] exists,up 1a520bcf-81bb-4f1d-80fb-11e98008e429', 'osd.4 up in weight 1 up_from 32 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.162:6800/4178676504,v1:172.21.15.162:6801/4178676504] [v2:172.21.15.162:6802/4178676504,v1:172.21.15.162:6803/4178676504] exists,up 4617c2fe-c9d8-40cd-887b-303e3215bdf8', 'osd.5 up in weight 1 up_from 40 up_thru 43 down_at 0 last_clean_interval [0,0) [v2:172.21.15.162:6808/1489371339,v1:172.21.15.162:6809/1489371339] [v2:172.21.15.162:6810/1489371339,v1:172.21.15.162:6811/1489371339] exists,up da98e3f3-173e-46fe-be8b-1ccbe8b0dabc', 'osd.6 up in weight 1 up_from 49 up_thru 52 down_at 0 last_clean_interval [0,0) [v2:172.21.15.162:6816/821169663,v1:172.21.15.162:6817/821169663] [v2:172.21.15.162:6818/821169663,v1:172.21.15.162:6819/821169663] exists,up ff50382d-5ddf-4cda-bb0c-ff055feeaa40', 'osd.7 up in weight 1 up_from 57 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.162:6824/965264369,v1:172.21.15.162:6825/965264369] [v2:172.21.15.162:6826/965264369,v1:172.21.15.162:6827/965264369] exists,up a82984dc-5e22-424b-a446-ae2ee71eee19', 'osd.8 up in weight 1 up_from 66 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.179:6800/4219839127,v1:172.21.15.179:6801/4219839127] [v2:172.21.15.179:6802/4219839127,v1:172.21.15.179:6803/4219839127] exists,up 63f9d79f-0400-44dc-b388-df00fefb5820', 'osd.9 up in weight 1 up_from 77 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.179:6808/504427183,v1:172.21.15.179:6809/504427183] [v2:172.21.15.179:6810/504427183,v1:172.21.15.179:6811/504427183] exists,up 7d995854-7328-4dfc-b1a9-0edca8d74f46', 'osd.10 up in weight 1 up_from 84 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.179:6816/2403345164,v1:172.21.15.179:6817/2403345164] [v2:172.21.15.179:6818/2403345164,v1:172.21.15.179:6819/2403345164] exists,up 151fbedd-08e6-46e6-a4e5-a14248399f61', 'osd.11 up in weight 1 up_from 90 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.179:6824/2899402373,v1:172.21.15.179:6825/2899402373] [v2:172.21.15.179:6826/2899402373,v1:172.21.15.179:6827/2899402373] exists,up 6d3a3ad8-417a-483f-991c-52549dbe7d73'] 2024-02-09T17:24:24.172 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon mon.a config show 2024-02-09T17:24:24.436 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:24:24.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:24 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1918032976' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-09T17:24:24.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:24 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1918032976' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-09T17:24:24.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:24 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1918032976' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-09T17:24:25.670 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:25 smithi099 ceph-mon[39420]: pgmap v338: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:25.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:25 smithi179 ceph-mon[43975]: pgmap v338: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:25.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:25 smithi162 ceph-mon[44304]: pgmap v338: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:26.207 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:24:26.208 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "mon.a", 2024-02-09T17:24:26.208 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster": "ceph", 2024-02-09T17:24:26.208 INFO:teuthology.orchestra.run.smithi099.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-02-09T17:24:26.208 INFO:teuthology.orchestra.run.smithi099.stdout: "admin_socket_mode": "", 2024-02-09T17:24:26.208 INFO:teuthology.orchestra.run.smithi099.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-02-09T17:24:26.208 INFO:teuthology.orchestra.run.smithi099.stdout: "auth_client_required": "cephx, none", 2024-02-09T17:24:26.208 INFO:teuthology.orchestra.run.smithi099.stdout: "auth_cluster_required": "cephx", 2024-02-09T17:24:26.208 INFO:teuthology.orchestra.run.smithi099.stdout: "auth_debug": "true", 2024-02-09T17:24:26.208 INFO:teuthology.orchestra.run.smithi099.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-02-09T17:24:26.208 INFO:teuthology.orchestra.run.smithi099.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-02-09T17:24:26.208 INFO:teuthology.orchestra.run.smithi099.stdout: "auth_service_required": "cephx", 2024-02-09T17:24:26.209 INFO:teuthology.orchestra.run.smithi099.stdout: "auth_service_ticket_ttl": "240.000000", 2024-02-09T17:24:26.209 INFO:teuthology.orchestra.run.smithi099.stdout: "auth_supported": "", 2024-02-09T17:24:26.209 INFO:teuthology.orchestra.run.smithi099.stdout: "bdev_aio": "true", 2024-02-09T17:24:26.209 INFO:teuthology.orchestra.run.smithi099.stdout: "bdev_aio_max_queue_depth": "1024", 2024-02-09T17:24:26.209 INFO:teuthology.orchestra.run.smithi099.stdout: "bdev_aio_poll_ms": "250", 2024-02-09T17:24:26.209 INFO:teuthology.orchestra.run.smithi099.stdout: "bdev_aio_reap_max": "16", 2024-02-09T17:24:26.209 INFO:teuthology.orchestra.run.smithi099.stdout: "bdev_async_discard": "false", 2024-02-09T17:24:26.209 INFO:teuthology.orchestra.run.smithi099.stdout: "bdev_block_size": "4096", 2024-02-09T17:24:26.209 INFO:teuthology.orchestra.run.smithi099.stdout: "bdev_debug_aio": "false", 2024-02-09T17:24:26.209 INFO:teuthology.orchestra.run.smithi099.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-02-09T17:24:26.209 INFO:teuthology.orchestra.run.smithi099.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-02-09T17:24:26.210 INFO:teuthology.orchestra.run.smithi099.stdout: "bdev_debug_inflight_ios": "false", 2024-02-09T17:24:26.210 INFO:teuthology.orchestra.run.smithi099.stdout: "bdev_enable_discard": "false", 2024-02-09T17:24:26.210 INFO:teuthology.orchestra.run.smithi099.stdout: "bdev_flock_retry": "3", 2024-02-09T17:24:26.210 INFO:teuthology.orchestra.run.smithi099.stdout: "bdev_flock_retry_interval": "0.100000", 2024-02-09T17:24:26.210 INFO:teuthology.orchestra.run.smithi099.stdout: "bdev_inject_crash": "0", 2024-02-09T17:24:26.210 INFO:teuthology.orchestra.run.smithi099.stdout: "bdev_inject_crash_flush_delay": "2", 2024-02-09T17:24:26.210 INFO:teuthology.orchestra.run.smithi099.stdout: "bdev_ioring": "false", 2024-02-09T17:24:26.210 INFO:teuthology.orchestra.run.smithi099.stdout: "bdev_ioring_hipri": "false", 2024-02-09T17:24:26.210 INFO:teuthology.orchestra.run.smithi099.stdout: "bdev_ioring_sqthread_poll": "false", 2024-02-09T17:24:26.210 INFO:teuthology.orchestra.run.smithi099.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-02-09T17:24:26.210 INFO:teuthology.orchestra.run.smithi099.stdout: "bdev_read_buffer_alignment": "4096", 2024-02-09T17:24:26.210 INFO:teuthology.orchestra.run.smithi099.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-02-09T17:24:26.210 INFO:teuthology.orchestra.run.smithi099.stdout: "bdev_type": "", 2024-02-09T17:24:26.210 INFO:teuthology.orchestra.run.smithi099.stdout: "bluefs_alloc_size": "1048576", 2024-02-09T17:24:26.211 INFO:teuthology.orchestra.run.smithi099.stdout: "bluefs_allocator": "hybrid", 2024-02-09T17:24:26.211 INFO:teuthology.orchestra.run.smithi099.stdout: "bluefs_buffered_io": "true", 2024-02-09T17:24:26.211 INFO:teuthology.orchestra.run.smithi099.stdout: "bluefs_check_for_zeros": "false", 2024-02-09T17:24:26.211 INFO:teuthology.orchestra.run.smithi099.stdout: "bluefs_check_volume_selector_often": "false", 2024-02-09T17:24:26.211 INFO:teuthology.orchestra.run.smithi099.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-02-09T17:24:26.211 INFO:teuthology.orchestra.run.smithi099.stdout: "bluefs_compact_log_sync": "false", 2024-02-09T17:24:26.211 INFO:teuthology.orchestra.run.smithi099.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-02-09T17:24:26.211 INFO:teuthology.orchestra.run.smithi099.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-02-09T17:24:26.211 INFO:teuthology.orchestra.run.smithi099.stdout: "bluefs_log_compact_min_size": "16777216", 2024-02-09T17:24:26.211 INFO:teuthology.orchestra.run.smithi099.stdout: "bluefs_log_replay_check_allocations": "true", 2024-02-09T17:24:26.211 INFO:teuthology.orchestra.run.smithi099.stdout: "bluefs_max_log_runway": "4194304", 2024-02-09T17:24:26.211 INFO:teuthology.orchestra.run.smithi099.stdout: "bluefs_max_prefetch": "1048576", 2024-02-09T17:24:26.211 INFO:teuthology.orchestra.run.smithi099.stdout: "bluefs_min_flush_size": "524288", 2024-02-09T17:24:26.212 INFO:teuthology.orchestra.run.smithi099.stdout: "bluefs_min_log_runway": "1048576", 2024-02-09T17:24:26.212 INFO:teuthology.orchestra.run.smithi099.stdout: "bluefs_replay_recovery": "false", 2024-02-09T17:24:26.212 INFO:teuthology.orchestra.run.smithi099.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-02-09T17:24:26.212 INFO:teuthology.orchestra.run.smithi099.stdout: "bluefs_shared_alloc_size": "65536", 2024-02-09T17:24:26.212 INFO:teuthology.orchestra.run.smithi099.stdout: "bluefs_sync_write": "false", 2024-02-09T17:24:26.212 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-02-09T17:24:26.212 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-02-09T17:24:26.212 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-02-09T17:24:26.212 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_allocation_from_file": "true", 2024-02-09T17:24:26.212 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_allocator": "hybrid", 2024-02-09T17:24:26.212 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-02-09T17:24:26.212 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-02-09T17:24:26.213 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-02-09T17:24:26.213 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-02-09T17:24:26.213 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-02-09T17:24:26.213 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-02-09T17:24:26.213 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_blobid_prealloc": "10240", 2024-02-09T17:24:26.213 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_block_create": "true", 2024-02-09T17:24:26.213 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_block_db_create": "false", 2024-02-09T17:24:26.213 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_block_db_path": "", 2024-02-09T17:24:26.213 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_block_db_size": "0", 2024-02-09T17:24:26.213 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_block_path": "", 2024-02-09T17:24:26.213 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_block_preallocate_file": "false", 2024-02-09T17:24:26.213 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_block_size": "107374182400", 2024-02-09T17:24:26.214 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_block_wal_create": "false", 2024-02-09T17:24:26.214 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_block_wal_path": "", 2024-02-09T17:24:26.214 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_block_wal_size": "100663296", 2024-02-09T17:24:26.214 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_bluefs": "true", 2024-02-09T17:24:26.214 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-02-09T17:24:26.214 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_bluefs_env_mirror": "false", 2024-02-09T17:24:26.214 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-02-09T17:24:26.214 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-02-09T17:24:26.214 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-02-09T17:24:26.214 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-02-09T17:24:26.214 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-02-09T17:24:26.214 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-02-09T17:24:26.215 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_cache_autotune": "true", 2024-02-09T17:24:26.215 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-02-09T17:24:26.215 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-02-09T17:24:26.215 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-02-09T17:24:26.215 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-02-09T17:24:26.215 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_cache_size": "0", 2024-02-09T17:24:26.215 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-02-09T17:24:26.215 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-02-09T17:24:26.215 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-02-09T17:24:26.215 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-02-09T17:24:26.216 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_cache_type": "2q", 2024-02-09T17:24:26.216 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-02-09T17:24:26.216 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_clone_cow": "true", 2024-02-09T17:24:26.216 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_compression_algorithm": "snappy", 2024-02-09T17:24:26.216 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_compression_max_blob_size": "0", 2024-02-09T17:24:26.216 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-02-09T17:24:26.216 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-02-09T17:24:26.216 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_compression_min_blob_size": "0", 2024-02-09T17:24:26.216 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-02-09T17:24:26.216 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-02-09T17:24:26.216 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_compression_mode": "none", 2024-02-09T17:24:26.216 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-02-09T17:24:26.216 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_csum_type": "crc32c", 2024-02-09T17:24:26.217 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_debug_enforce_settings": "default", 2024-02-09T17:24:26.217 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_debug_freelist": "false", 2024-02-09T17:24:26.217 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_debug_fsck_abort": "false", 2024-02-09T17:24:26.217 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-02-09T17:24:26.217 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-02-09T17:24:26.218 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_debug_inject_read_err": "false", 2024-02-09T17:24:26.218 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_debug_legacy_omap": "false", 2024-02-09T17:24:26.218 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-02-09T17:24:26.218 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-02-09T17:24:26.218 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-02-09T17:24:26.218 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-02-09T17:24:26.218 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_debug_prefill": "0.000000", 2024-02-09T17:24:26.218 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-02-09T17:24:26.218 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-02-09T17:24:26.218 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-02-09T17:24:26.218 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_debug_small_allocations": "0", 2024-02-09T17:24:26.218 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-02-09T17:24:26.218 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_default_buffered_read": "true", 2024-02-09T17:24:26.218 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_default_buffered_write": "false", 2024-02-09T17:24:26.219 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_deferred_batch_ops": "0", 2024-02-09T17:24:26.219 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-02-09T17:24:26.219 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-02-09T17:24:26.219 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_elastic_shared_blobs": "true", 2024-02-09T17:24:26.219 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-02-09T17:24:26.219 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-02-09T17:24:26.219 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-02-09T17:24:26.219 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-02-09T17:24:26.219 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-02-09T17:24:26.219 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_fail_eio": "false", 2024-02-09T17:24:26.220 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-02-09T17:24:26.220 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-02-09T17:24:26.220 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-02-09T17:24:26.220 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-02-09T17:24:26.220 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_fsck_on_mkfs": "true", 2024-02-09T17:24:26.220 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-02-09T17:24:26.220 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_fsck_on_mount": "false", 2024-02-09T17:24:26.220 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-02-09T17:24:26.220 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_fsck_on_umount": "false", 2024-02-09T17:24:26.220 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-02-09T17:24:26.220 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-02-09T17:24:26.220 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-02-09T17:24:26.220 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-02-09T17:24:26.220 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-02-09T17:24:26.221 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-02-09T17:24:26.221 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-02-09T17:24:26.221 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-02-09T17:24:26.221 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_ignore_data_csum": "false", 2024-02-09T17:24:26.221 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-02-09T17:24:26.221 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_kvbackend": "rocksdb", 2024-02-09T17:24:26.221 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-02-09T17:24:26.221 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-02-09T17:24:26.221 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_log_op_age": "5.000000", 2024-02-09T17:24:26.221 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_max_alloc_size": "0", 2024-02-09T17:24:26.221 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_max_blob_size": "0", 2024-02-09T17:24:26.221 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-02-09T17:24:26.221 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-02-09T17:24:26.222 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_max_defer_interval": "3.000000", 2024-02-09T17:24:26.222 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_max_deferred_txc": "32", 2024-02-09T17:24:26.222 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_min_alloc_size": "0", 2024-02-09T17:24:26.222 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-02-09T17:24:26.222 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-02-09T17:24:26.222 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_nid_prealloc": "1024", 2024-02-09T17:24:26.222 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_prefer_deferred_size": "0", 2024-02-09T17:24:26.222 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-02-09T17:24:26.222 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-02-09T17:24:26.222 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_qfsck_on_mount": "true", 2024-02-09T17:24:26.222 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_retry_disk_reads": "3", 2024-02-09T17:24:26.222 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_rocksdb_cf": "true", 2024-02-09T17:24:26.223 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-02-09T17:24:26.223 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-02-09T17:24:26.223 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_rocksdb_options_annex": "", 2024-02-09T17:24:26.223 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_spdk_coremask": "0x1", 2024-02-09T17:24:26.223 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_spdk_io_sleep": "5", 2024-02-09T17:24:26.223 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_spdk_max_io_completion": "0", 2024-02-09T17:24:26.224 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_spdk_mem": "512", 2024-02-09T17:24:26.224 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_sync_submit_transaction": "false", 2024-02-09T17:24:26.224 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_throttle_bytes": "67108864", 2024-02-09T17:24:26.224 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_throttle_cost_per_io": "0", 2024-02-09T17:24:26.224 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-02-09T17:24:26.224 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-02-09T17:24:26.224 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-02-09T17:24:26.224 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-02-09T17:24:26.224 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_tracing": "false", 2024-02-09T17:24:26.224 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-02-09T17:24:26.224 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-02-09T17:24:26.224 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_volume_selection_reserved": "0", 2024-02-09T17:24:26.225 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-02-09T17:24:26.225 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-02-09T17:24:26.225 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-02-09T17:24:26.225 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-02-09T17:24:26.225 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-02-09T17:24:26.225 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-02-09T17:24:26.225 INFO:teuthology.orchestra.run.smithi099.stdout: "bluestore_zero_block_detection": "false", 2024-02-09T17:24:26.225 INFO:teuthology.orchestra.run.smithi099.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-02-09T17:24:26.225 INFO:teuthology.orchestra.run.smithi099.stdout: "cephfs_mirror_action_update_interval": "2", 2024-02-09T17:24:26.225 INFO:teuthology.orchestra.run.smithi099.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-02-09T17:24:26.225 INFO:teuthology.orchestra.run.smithi099.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-02-09T17:24:26.225 INFO:teuthology.orchestra.run.smithi099.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-02-09T17:24:26.225 INFO:teuthology.orchestra.run.smithi099.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-02-09T17:24:26.226 INFO:teuthology.orchestra.run.smithi099.stdout: "cephfs_mirror_mount_timeout": "10", 2024-02-09T17:24:26.226 INFO:teuthology.orchestra.run.smithi099.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-02-09T17:24:26.226 INFO:teuthology.orchestra.run.smithi099.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-02-09T17:24:26.226 INFO:teuthology.orchestra.run.smithi099.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-02-09T17:24:26.226 INFO:teuthology.orchestra.run.smithi099.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-02-09T17:24:26.226 INFO:teuthology.orchestra.run.smithi099.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-02-09T17:24:26.226 INFO:teuthology.orchestra.run.smithi099.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-02-09T17:24:26.226 INFO:teuthology.orchestra.run.smithi099.stdout: "cephx_cluster_require_signatures": "false", 2024-02-09T17:24:26.226 INFO:teuthology.orchestra.run.smithi099.stdout: "cephx_cluster_require_version": "2", 2024-02-09T17:24:26.226 INFO:teuthology.orchestra.run.smithi099.stdout: "cephx_require_signatures": "false", 2024-02-09T17:24:26.226 INFO:teuthology.orchestra.run.smithi099.stdout: "cephx_require_version": "2", 2024-02-09T17:24:26.226 INFO:teuthology.orchestra.run.smithi099.stdout: "cephx_service_require_signatures": "false", 2024-02-09T17:24:26.227 INFO:teuthology.orchestra.run.smithi099.stdout: "cephx_service_require_version": "2", 2024-02-09T17:24:26.227 INFO:teuthology.orchestra.run.smithi099.stdout: "cephx_sign_messages": "true", 2024-02-09T17:24:26.227 INFO:teuthology.orchestra.run.smithi099.stdout: "chdir": "", 2024-02-09T17:24:26.227 INFO:teuthology.orchestra.run.smithi099.stdout: "client_acl_type": "", 2024-02-09T17:24:26.227 INFO:teuthology.orchestra.run.smithi099.stdout: "client_asio_thread_count": "2", 2024-02-09T17:24:26.227 INFO:teuthology.orchestra.run.smithi099.stdout: "client_cache_mid": "0.750000", 2024-02-09T17:24:26.227 INFO:teuthology.orchestra.run.smithi099.stdout: "client_cache_size": "16384", 2024-02-09T17:24:26.227 INFO:teuthology.orchestra.run.smithi099.stdout: "client_caps_release_delay": "5", 2024-02-09T17:24:26.227 INFO:teuthology.orchestra.run.smithi099.stdout: "client_check_pool_perm": "true", 2024-02-09T17:24:26.227 INFO:teuthology.orchestra.run.smithi099.stdout: "client_collect_and_send_global_metrics": "false", 2024-02-09T17:24:26.227 INFO:teuthology.orchestra.run.smithi099.stdout: "client_debug_force_sync_read": "false", 2024-02-09T17:24:26.228 INFO:teuthology.orchestra.run.smithi099.stdout: "client_debug_getattr_caps": "false", 2024-02-09T17:24:26.228 INFO:teuthology.orchestra.run.smithi099.stdout: "client_debug_inject_tick_delay": "0", 2024-02-09T17:24:26.228 INFO:teuthology.orchestra.run.smithi099.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-02-09T17:24:26.228 INFO:teuthology.orchestra.run.smithi099.stdout: "client_die_on_failed_remount": "false", 2024-02-09T17:24:26.228 INFO:teuthology.orchestra.run.smithi099.stdout: "client_dirsize_rbytes": "true", 2024-02-09T17:24:26.228 INFO:teuthology.orchestra.run.smithi099.stdout: "client_force_lazyio": "false", 2024-02-09T17:24:26.228 INFO:teuthology.orchestra.run.smithi099.stdout: "client_fs": "", 2024-02-09T17:24:26.228 INFO:teuthology.orchestra.run.smithi099.stdout: "client_inject_fixed_oldest_tid": "false", 2024-02-09T17:24:26.228 INFO:teuthology.orchestra.run.smithi099.stdout: "client_inject_release_failure": "false", 2024-02-09T17:24:26.228 INFO:teuthology.orchestra.run.smithi099.stdout: "client_max_inline_size": "4096", 2024-02-09T17:24:26.231 INFO:teuthology.orchestra.run.smithi099.stdout: "client_max_retries_on_remount_failure": "5", 2024-02-09T17:24:26.231 INFO:teuthology.orchestra.run.smithi099.stdout: "client_mds_namespace": "", 2024-02-09T17:24:26.231 INFO:teuthology.orchestra.run.smithi099.stdout: "client_metadata": "", 2024-02-09T17:24:26.231 INFO:teuthology.orchestra.run.smithi099.stdout: "client_mount_gid": "-1", 2024-02-09T17:24:26.231 INFO:teuthology.orchestra.run.smithi099.stdout: "client_mount_timeout": "300", 2024-02-09T17:24:26.231 INFO:teuthology.orchestra.run.smithi099.stdout: "client_mount_uid": "-1", 2024-02-09T17:24:26.231 INFO:teuthology.orchestra.run.smithi099.stdout: "client_mountpoint": "/", 2024-02-09T17:24:26.231 INFO:teuthology.orchestra.run.smithi099.stdout: "client_notify_timeout": "10", 2024-02-09T17:24:26.231 INFO:teuthology.orchestra.run.smithi099.stdout: "client_oc": "true", 2024-02-09T17:24:26.232 INFO:teuthology.orchestra.run.smithi099.stdout: "client_oc_max_dirty": "104857600", 2024-02-09T17:24:26.232 INFO:teuthology.orchestra.run.smithi099.stdout: "client_oc_max_dirty_age": "5.000000", 2024-02-09T17:24:26.232 INFO:teuthology.orchestra.run.smithi099.stdout: "client_oc_max_objects": "1000", 2024-02-09T17:24:26.232 INFO:teuthology.orchestra.run.smithi099.stdout: "client_oc_size": "209715200", 2024-02-09T17:24:26.232 INFO:teuthology.orchestra.run.smithi099.stdout: "client_oc_target_dirty": "8388608", 2024-02-09T17:24:26.232 INFO:teuthology.orchestra.run.smithi099.stdout: "client_permissions": "true", 2024-02-09T17:24:26.232 INFO:teuthology.orchestra.run.smithi099.stdout: "client_quota": "true", 2024-02-09T17:24:26.232 INFO:teuthology.orchestra.run.smithi099.stdout: "client_quota_df": "true", 2024-02-09T17:24:26.232 INFO:teuthology.orchestra.run.smithi099.stdout: "client_readahead_max_bytes": "0", 2024-02-09T17:24:26.232 INFO:teuthology.orchestra.run.smithi099.stdout: "client_readahead_max_periods": "4", 2024-02-09T17:24:26.232 INFO:teuthology.orchestra.run.smithi099.stdout: "client_readahead_min": "131072", 2024-02-09T17:24:26.232 INFO:teuthology.orchestra.run.smithi099.stdout: "client_reconnect_stale": "false", 2024-02-09T17:24:26.232 INFO:teuthology.orchestra.run.smithi099.stdout: "client_shutdown_timeout": "30", 2024-02-09T17:24:26.232 INFO:teuthology.orchestra.run.smithi099.stdout: "client_snapdir": ".snap", 2024-02-09T17:24:26.233 INFO:teuthology.orchestra.run.smithi099.stdout: "client_tick_interval": "1", 2024-02-09T17:24:26.233 INFO:teuthology.orchestra.run.smithi099.stdout: "client_trace": "", 2024-02-09T17:24:26.233 INFO:teuthology.orchestra.run.smithi099.stdout: "client_try_dentry_invalidate": "false", 2024-02-09T17:24:26.233 INFO:teuthology.orchestra.run.smithi099.stdout: "client_use_faked_inos": "false", 2024-02-09T17:24:26.233 INFO:teuthology.orchestra.run.smithi099.stdout: "client_use_random_mds": "false", 2024-02-09T17:24:26.233 INFO:teuthology.orchestra.run.smithi099.stdout: "clog_to_graylog": "false", 2024-02-09T17:24:26.233 INFO:teuthology.orchestra.run.smithi099.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-02-09T17:24:26.233 INFO:teuthology.orchestra.run.smithi099.stdout: "clog_to_graylog_port": "12201", 2024-02-09T17:24:26.233 INFO:teuthology.orchestra.run.smithi099.stdout: "clog_to_monitors": "default=true", 2024-02-09T17:24:26.233 INFO:teuthology.orchestra.run.smithi099.stdout: "clog_to_syslog": "false", 2024-02-09T17:24:26.233 INFO:teuthology.orchestra.run.smithi099.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-02-09T17:24:26.233 INFO:teuthology.orchestra.run.smithi099.stdout: "clog_to_syslog_level": "info", 2024-02-09T17:24:26.233 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "-", 2024-02-09T17:24:26.233 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_network": "", 2024-02-09T17:24:26.233 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_network_interface": "", 2024-02-09T17:24:26.234 INFO:teuthology.orchestra.run.smithi099.stdout: "compressor_zlib_isal": "false", 2024-02-09T17:24:26.234 INFO:teuthology.orchestra.run.smithi099.stdout: "compressor_zlib_level": "5", 2024-02-09T17:24:26.234 INFO:teuthology.orchestra.run.smithi099.stdout: "compressor_zlib_winsize": "-15", 2024-02-09T17:24:26.234 INFO:teuthology.orchestra.run.smithi099.stdout: "compressor_zstd_level": "1", 2024-02-09T17:24:26.234 INFO:teuthology.orchestra.run.smithi099.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:96064cefa09b46b83404968abe29e7381bd2bc1fd07dadd4e4f9771696ab50aa", 2024-02-09T17:24:26.234 INFO:teuthology.orchestra.run.smithi099.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-02-09T17:24:26.234 INFO:teuthology.orchestra.run.smithi099.stdout: "crimson_alien_op_num_threads": "6", 2024-02-09T17:24:26.234 INFO:teuthology.orchestra.run.smithi099.stdout: "crimson_alien_thread_cpu_cores": "", 2024-02-09T17:24:26.234 INFO:teuthology.orchestra.run.smithi099.stdout: "crimson_osd_obc_lru_size": "512", 2024-02-09T17:24:26.234 INFO:teuthology.orchestra.run.smithi099.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-02-09T17:24:26.234 INFO:teuthology.orchestra.run.smithi099.stdout: "crimson_seastar_cpu_cores": "", 2024-02-09T17:24:26.234 INFO:teuthology.orchestra.run.smithi099.stdout: "crush_location": "", 2024-02-09T17:24:26.234 INFO:teuthology.orchestra.run.smithi099.stdout: "crush_location_hook": "", 2024-02-09T17:24:26.235 INFO:teuthology.orchestra.run.smithi099.stdout: "crush_location_hook_timeout": "10", 2024-02-09T17:24:26.235 INFO:teuthology.orchestra.run.smithi099.stdout: "daemonize": "false", 2024-02-09T17:24:26.235 INFO:teuthology.orchestra.run.smithi099.stdout: "daos_pool": "tank", 2024-02-09T17:24:26.235 INFO:teuthology.orchestra.run.smithi099.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-02-09T17:24:26.235 INFO:teuthology.orchestra.run.smithi099.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-02-09T17:24:26.235 INFO:teuthology.orchestra.run.smithi099.stdout: "dbstore_db_name_prefix": "dbstore", 2024-02-09T17:24:26.235 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_alienstore": "0/5", 2024-02-09T17:24:26.235 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_allow_any_pool_priority": "false", 2024-02-09T17:24:26.235 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_asok": "1/5", 2024-02-09T17:24:26.235 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_asok_assert_abort": "false", 2024-02-09T17:24:26.235 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_asserts_on_shutdown": "true", 2024-02-09T17:24:26.235 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_auth": "1/5", 2024-02-09T17:24:26.235 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_bdev": "1/3", 2024-02-09T17:24:26.235 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_bluefs": "1/5", 2024-02-09T17:24:26.235 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_bluestore": "1/5", 2024-02-09T17:24:26.236 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_buffer": "0/1", 2024-02-09T17:24:26.236 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_ceph_exporter": "1/5", 2024-02-09T17:24:26.236 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_cephfs_mirror": "0/5", 2024-02-09T17:24:26.236 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_cephsqlite": "0/5", 2024-02-09T17:24:26.236 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_client": "0/5", 2024-02-09T17:24:26.236 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_compressor": "1/5", 2024-02-09T17:24:26.236 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_context": "0/1", 2024-02-09T17:24:26.236 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_crimson_interrupt": "0/5", 2024-02-09T17:24:26.236 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_crush": "1/1", 2024-02-09T17:24:26.236 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_crypto": "1/5", 2024-02-09T17:24:26.236 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_cyanstore": "0/5", 2024-02-09T17:24:26.236 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_deliberately_leak_memory": "false", 2024-02-09T17:24:26.236 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_disable_randomized_ping": "false", 2024-02-09T17:24:26.236 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_dpdk": "1/5", 2024-02-09T17:24:26.237 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_eventtrace": "1/5", 2024-02-09T17:24:26.237 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_filer": "0/1", 2024-02-09T17:24:26.237 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_filestore": "1/3", 2024-02-09T17:24:26.237 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_finisher": "1/1", 2024-02-09T17:24:26.237 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_fuse": "1/5", 2024-02-09T17:24:26.237 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_heartbeat_testing_span": "0", 2024-02-09T17:24:26.237 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_heartbeatmap": "1/5", 2024-02-09T17:24:26.237 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_immutable_obj_cache": "0/5", 2024-02-09T17:24:26.237 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_javaclient": "1/5", 2024-02-09T17:24:26.237 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_journal": "1/3", 2024-02-09T17:24:26.237 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_journaler": "0/5", 2024-02-09T17:24:26.237 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_kstore": "1/5", 2024-02-09T17:24:26.237 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_lockdep": "0/1", 2024-02-09T17:24:26.237 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_mclock": "1/5", 2024-02-09T17:24:26.237 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_mds": "1/5", 2024-02-09T17:24:26.238 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_mds_balancer": "1/5", 2024-02-09T17:24:26.238 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_mds_locker": "1/5", 2024-02-09T17:24:26.238 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_mds_log": "1/5", 2024-02-09T17:24:26.238 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_mds_log_expire": "1/5", 2024-02-09T17:24:26.238 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_mds_migrator": "1/5", 2024-02-09T17:24:26.238 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_memstore": "1/5", 2024-02-09T17:24:26.238 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_mgr": "2/5", 2024-02-09T17:24:26.238 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_mgrc": "1/5", 2024-02-09T17:24:26.238 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_mon": "20/20", 2024-02-09T17:24:26.238 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_monc": "0/10", 2024-02-09T17:24:26.238 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_ms": "1/1", 2024-02-09T17:24:26.238 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_none": "0/5", 2024-02-09T17:24:26.238 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_objclass": "0/5", 2024-02-09T17:24:26.239 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_objectcacher": "0/5", 2024-02-09T17:24:26.239 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_objecter": "0/1", 2024-02-09T17:24:26.239 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_optracker": "0/5", 2024-02-09T17:24:26.239 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_osd": "1/5", 2024-02-09T17:24:26.239 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_paxos": "20/20", 2024-02-09T17:24:26.239 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_perfcounter": "1/5", 2024-02-09T17:24:26.239 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_prioritycache": "1/5", 2024-02-09T17:24:26.239 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_rados": "0/5", 2024-02-09T17:24:26.239 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_rbd": "0/5", 2024-02-09T17:24:26.239 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_rbd_mirror": "0/5", 2024-02-09T17:24:26.239 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_rbd_pwl": "0/5", 2024-02-09T17:24:26.239 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_rbd_replay": "0/5", 2024-02-09T17:24:26.239 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_refs": "0/0", 2024-02-09T17:24:26.239 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_reserver": "1/1", 2024-02-09T17:24:26.239 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_rgw": "1/5", 2024-02-09T17:24:26.239 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_rgw_access": "1/5", 2024-02-09T17:24:26.240 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_rgw_datacache": "1/5", 2024-02-09T17:24:26.240 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_rgw_dbstore": "1/5", 2024-02-09T17:24:26.240 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_rgw_flight": "1/5", 2024-02-09T17:24:26.240 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_rgw_lifecycle": "1/5", 2024-02-09T17:24:26.240 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_rgw_sync": "1/5", 2024-02-09T17:24:26.240 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_rocksdb": "4/5", 2024-02-09T17:24:26.240 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_seastore": "0/5", 2024-02-09T17:24:26.240 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_seastore_backref": "0/5", 2024-02-09T17:24:26.240 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_seastore_cache": "0/5", 2024-02-09T17:24:26.240 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_seastore_cleaner": "0/5", 2024-02-09T17:24:26.240 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_seastore_device": "0/5", 2024-02-09T17:24:26.240 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_seastore_epm": "0/5", 2024-02-09T17:24:26.240 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-02-09T17:24:26.241 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_seastore_journal": "0/5", 2024-02-09T17:24:26.241 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_seastore_lba": "0/5", 2024-02-09T17:24:26.241 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_seastore_odata": "0/5", 2024-02-09T17:24:26.241 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_seastore_omap": "0/5", 2024-02-09T17:24:26.241 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_seastore_onode": "0/5", 2024-02-09T17:24:26.241 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_seastore_t": "0/5", 2024-02-09T17:24:26.241 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_seastore_tm": "0/5", 2024-02-09T17:24:26.241 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_striper": "0/1", 2024-02-09T17:24:26.241 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_test": "0/5", 2024-02-09T17:24:26.241 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_throttle": "1/1", 2024-02-09T17:24:26.241 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_timer": "0/1", 2024-02-09T17:24:26.241 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_tp": "0/5", 2024-02-09T17:24:26.241 INFO:teuthology.orchestra.run.smithi099.stdout: "debug_trace": "1/5", 2024-02-09T17:24:26.242 INFO:teuthology.orchestra.run.smithi099.stdout: "defer_client_eviction_on_laggy_osds": "true", 2024-02-09T17:24:26.242 INFO:teuthology.orchestra.run.smithi099.stdout: "device_failure_prediction_mode": "none", 2024-02-09T17:24:26.242 INFO:teuthology.orchestra.run.smithi099.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-02-09T17:24:26.242 INFO:teuthology.orchestra.run.smithi099.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-02-09T17:24:26.242 INFO:teuthology.orchestra.run.smithi099.stdout: "err_to_graylog": "false", 2024-02-09T17:24:26.242 INFO:teuthology.orchestra.run.smithi099.stdout: "err_to_journald": "false", 2024-02-09T17:24:26.242 INFO:teuthology.orchestra.run.smithi099.stdout: "err_to_stderr": "true", 2024-02-09T17:24:26.242 INFO:teuthology.orchestra.run.smithi099.stdout: "err_to_syslog": "false", 2024-02-09T17:24:26.242 INFO:teuthology.orchestra.run.smithi099.stdout: "event_tracing": "false", 2024-02-09T17:24:26.242 INFO:teuthology.orchestra.run.smithi099.stdout: "exporter_addr": "0.0.0.0", 2024-02-09T17:24:26.242 INFO:teuthology.orchestra.run.smithi099.stdout: "exporter_http_port": "9926", 2024-02-09T17:24:26.242 INFO:teuthology.orchestra.run.smithi099.stdout: "exporter_prio_limit": "5", 2024-02-09T17:24:26.242 INFO:teuthology.orchestra.run.smithi099.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-02-09T17:24:26.243 INFO:teuthology.orchestra.run.smithi099.stdout: "exporter_sort_metrics": "true", 2024-02-09T17:24:26.243 INFO:teuthology.orchestra.run.smithi099.stdout: "exporter_stats_period": "5", 2024-02-09T17:24:26.243 INFO:teuthology.orchestra.run.smithi099.stdout: "fake_statfs_for_testing": "0", 2024-02-09T17:24:26.243 INFO:teuthology.orchestra.run.smithi099.stdout: "fatal_signal_handlers": "true", 2024-02-09T17:24:26.243 INFO:teuthology.orchestra.run.smithi099.stdout: "filer_max_purge_ops": "10", 2024-02-09T17:24:26.243 INFO:teuthology.orchestra.run.smithi099.stdout: "filer_max_truncate_ops": "128", 2024-02-09T17:24:26.243 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_apply_finisher_threads": "1", 2024-02-09T17:24:26.243 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_blackhole": "false", 2024-02-09T17:24:26.243 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_btrfs_clone_range": "true", 2024-02-09T17:24:26.243 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_btrfs_snap": "true", 2024-02-09T17:24:26.243 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_caller_concurrency": "10", 2024-02-09T17:24:26.243 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_collect_device_partition_information": "true", 2024-02-09T17:24:26.243 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_commit_timeout": "600.000000", 2024-02-09T17:24:26.243 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_debug_inject_read_err": "false", 2024-02-09T17:24:26.244 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_debug_omap_check": "false", 2024-02-09T17:24:26.244 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_debug_random_read_err": "0.000000", 2024-02-09T17:24:26.244 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_debug_verify_split": "false", 2024-02-09T17:24:26.244 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_dump_file": "", 2024-02-09T17:24:26.244 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-02-09T17:24:26.244 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-02-09T17:24:26.244 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_fadvise": "true", 2024-02-09T17:24:26.244 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_fail_eio": "true", 2024-02-09T17:24:26.244 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_fd_cache_shards": "16", 2024-02-09T17:24:26.244 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_fd_cache_size": "128", 2024-02-09T17:24:26.244 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_fiemap": "false", 2024-02-09T17:24:26.244 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_fiemap_threshold": "4096", 2024-02-09T17:24:26.244 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-02-09T17:24:26.245 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_index_retry_probability": "0.000000", 2024-02-09T17:24:26.245 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_inject_stall": "0", 2024-02-09T17:24:26.245 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_journal_parallel": "false", 2024-02-09T17:24:26.245 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_journal_trailing": "false", 2024-02-09T17:24:26.245 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_journal_writeahead": "false", 2024-02-09T17:24:26.245 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_kill_at": "0", 2024-02-09T17:24:26.245 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-02-09T17:24:26.245 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_max_inline_xattr_size": "0", 2024-02-09T17:24:26.245 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-02-09T17:24:26.245 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-02-09T17:24:26.245 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-02-09T17:24:26.245 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_max_inline_xattrs": "0", 2024-02-09T17:24:26.245 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-02-09T17:24:26.245 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_max_inline_xattrs_other": "2", 2024-02-09T17:24:26.246 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-02-09T17:24:26.246 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_max_sync_interval": "5.000000", 2024-02-09T17:24:26.246 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_max_xattr_value_size": "0", 2024-02-09T17:24:26.246 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-02-09T17:24:26.246 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-02-09T17:24:26.246 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-02-09T17:24:26.246 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_merge_threshold": "-10", 2024-02-09T17:24:26.246 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_min_sync_interval": "0.010000", 2024-02-09T17:24:26.246 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_odsync_write": "false", 2024-02-09T17:24:26.246 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_omap_backend": "rocksdb", 2024-02-09T17:24:26.246 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_omap_backend_path": "", 2024-02-09T17:24:26.246 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_omap_header_cache_size": "1024", 2024-02-09T17:24:26.246 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_ondisk_finisher_threads": "1", 2024-02-09T17:24:26.246 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-02-09T17:24:26.247 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_op_thread_timeout": "60", 2024-02-09T17:24:26.247 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_op_threads": "2", 2024-02-09T17:24:26.247 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_punch_hole": "false", 2024-02-09T17:24:26.247 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-02-09T17:24:26.247 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-02-09T17:24:26.247 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-02-09T17:24:26.247 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-02-09T17:24:26.247 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-02-09T17:24:26.247 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_queue_max_bytes": "104857600", 2024-02-09T17:24:26.247 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-02-09T17:24:26.247 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-02-09T17:24:26.247 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-02-09T17:24:26.247 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_queue_max_ops": "50", 2024-02-09T17:24:26.247 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-02-09T17:24:26.248 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_seek_data_hole": "false", 2024-02-09T17:24:26.248 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_sloppy_crc": "false", 2024-02-09T17:24:26.248 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-02-09T17:24:26.248 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_splice": "false", 2024-02-09T17:24:26.248 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_split_multiple": "2", 2024-02-09T17:24:26.248 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_split_rand_factor": "20", 2024-02-09T17:24:26.248 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_update_to": "1000", 2024-02-09T17:24:26.248 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-02-09T17:24:26.248 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-02-09T17:24:26.248 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-02-09T17:24:26.248 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-02-09T17:24:26.248 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-02-09T17:24:26.249 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-02-09T17:24:26.249 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_wbthrottle_enable": "true", 2024-02-09T17:24:26.249 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-02-09T17:24:26.249 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-02-09T17:24:26.249 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-02-09T17:24:26.249 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-02-09T17:24:26.249 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-02-09T17:24:26.249 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-02-09T17:24:26.249 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_xfs_extsize": "false", 2024-02-09T17:24:26.249 INFO:teuthology.orchestra.run.smithi099.stdout: "filestore_zfs_snap": "false", 2024-02-09T17:24:26.249 INFO:teuthology.orchestra.run.smithi099.stdout: "fio_dir": "/tmp/fio", 2024-02-09T17:24:26.249 INFO:teuthology.orchestra.run.smithi099.stdout: "fsid": "a69c870a-c76e-11ee-95b7-87774f69a715", 2024-02-09T17:24:26.249 INFO:teuthology.orchestra.run.smithi099.stdout: "fuse_allow_other": "true", 2024-02-09T17:24:26.249 INFO:teuthology.orchestra.run.smithi099.stdout: "fuse_atomic_o_trunc": "true", 2024-02-09T17:24:26.249 INFO:teuthology.orchestra.run.smithi099.stdout: "fuse_big_writes": "true", 2024-02-09T17:24:26.250 INFO:teuthology.orchestra.run.smithi099.stdout: "fuse_debug": "false", 2024-02-09T17:24:26.250 INFO:teuthology.orchestra.run.smithi099.stdout: "fuse_default_permissions": "false", 2024-02-09T17:24:26.250 INFO:teuthology.orchestra.run.smithi099.stdout: "fuse_disable_pagecache": "false", 2024-02-09T17:24:26.250 INFO:teuthology.orchestra.run.smithi099.stdout: "fuse_max_write": "0", 2024-02-09T17:24:26.250 INFO:teuthology.orchestra.run.smithi099.stdout: "fuse_multithreaded": "true", 2024-02-09T17:24:26.250 INFO:teuthology.orchestra.run.smithi099.stdout: "fuse_require_active_mds": "true", 2024-02-09T17:24:26.250 INFO:teuthology.orchestra.run.smithi099.stdout: "fuse_set_user_groups": "true", 2024-02-09T17:24:26.250 INFO:teuthology.orchestra.run.smithi099.stdout: "fuse_splice_move": "true", 2024-02-09T17:24:26.250 INFO:teuthology.orchestra.run.smithi099.stdout: "fuse_splice_read": "true", 2024-02-09T17:24:26.250 INFO:teuthology.orchestra.run.smithi099.stdout: "fuse_splice_write": "true", 2024-02-09T17:24:26.250 INFO:teuthology.orchestra.run.smithi099.stdout: "fuse_syncfs_on_mksnap": "true", 2024-02-09T17:24:26.250 INFO:teuthology.orchestra.run.smithi099.stdout: "fuse_use_invalidate_cb": "true", 2024-02-09T17:24:26.251 INFO:teuthology.orchestra.run.smithi099.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-02-09T17:24:26.251 INFO:teuthology.orchestra.run.smithi099.stdout: "gss_target_name": "ceph", 2024-02-09T17:24:26.251 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_file": "", 2024-02-09T17:24:26.251 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_inject_failure": "0", 2024-02-09T17:24:26.251 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_interval": "5", 2024-02-09T17:24:26.251 INFO:teuthology.orchestra.run.smithi099.stdout: "host": "smithi099", 2024-02-09T17:24:26.251 INFO:teuthology.orchestra.run.smithi099.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-02-09T17:24:26.251 INFO:teuthology.orchestra.run.smithi099.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-02-09T17:24:26.251 INFO:teuthology.orchestra.run.smithi099.stdout: "immutable_object_cache_max_size": "1073741824", 2024-02-09T17:24:26.251 INFO:teuthology.orchestra.run.smithi099.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-02-09T17:24:26.251 INFO:teuthology.orchestra.run.smithi099.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-02-09T17:24:26.251 INFO:teuthology.orchestra.run.smithi099.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-02-09T17:24:26.251 INFO:teuthology.orchestra.run.smithi099.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-02-09T17:24:26.252 INFO:teuthology.orchestra.run.smithi099.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-02-09T17:24:26.252 INFO:teuthology.orchestra.run.smithi099.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-02-09T17:24:26.252 INFO:teuthology.orchestra.run.smithi099.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-02-09T17:24:26.252 INFO:teuthology.orchestra.run.smithi099.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-02-09T17:24:26.252 INFO:teuthology.orchestra.run.smithi099.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-02-09T17:24:26.252 INFO:teuthology.orchestra.run.smithi099.stdout: "immutable_object_cache_watermark": "0.900000", 2024-02-09T17:24:26.252 INFO:teuthology.orchestra.run.smithi099.stdout: "inject_early_sigterm": "false", 2024-02-09T17:24:26.252 INFO:teuthology.orchestra.run.smithi099.stdout: "jaeger_agent_port": "6799", 2024-02-09T17:24:26.252 INFO:teuthology.orchestra.run.smithi099.stdout: "jaeger_tracing_enable": "false", 2024-02-09T17:24:26.252 INFO:teuthology.orchestra.run.smithi099.stdout: "journal_aio": "true", 2024-02-09T17:24:26.252 INFO:teuthology.orchestra.run.smithi099.stdout: "journal_align_min_size": "65536", 2024-02-09T17:24:26.252 INFO:teuthology.orchestra.run.smithi099.stdout: "journal_block_align": "true", 2024-02-09T17:24:26.252 INFO:teuthology.orchestra.run.smithi099.stdout: "journal_block_size": "4096", 2024-02-09T17:24:26.253 INFO:teuthology.orchestra.run.smithi099.stdout: "journal_dio": "true", 2024-02-09T17:24:26.253 INFO:teuthology.orchestra.run.smithi099.stdout: "journal_discard": "false", 2024-02-09T17:24:26.253 INFO:teuthology.orchestra.run.smithi099.stdout: "journal_force_aio": "false", 2024-02-09T17:24:26.253 INFO:teuthology.orchestra.run.smithi099.stdout: "journal_ignore_corruption": "false", 2024-02-09T17:24:26.253 INFO:teuthology.orchestra.run.smithi099.stdout: "journal_max_write_bytes": "10485760", 2024-02-09T17:24:26.253 INFO:teuthology.orchestra.run.smithi099.stdout: "journal_max_write_entries": "100", 2024-02-09T17:24:26.253 INFO:teuthology.orchestra.run.smithi099.stdout: "journal_replay_from": "0", 2024-02-09T17:24:26.253 INFO:teuthology.orchestra.run.smithi099.stdout: "journal_throttle_high_multiple": "0.000000", 2024-02-09T17:24:26.253 INFO:teuthology.orchestra.run.smithi099.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-02-09T17:24:26.253 INFO:teuthology.orchestra.run.smithi099.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-02-09T17:24:26.253 INFO:teuthology.orchestra.run.smithi099.stdout: "journal_throttle_max_multiple": "0.000000", 2024-02-09T17:24:26.253 INFO:teuthology.orchestra.run.smithi099.stdout: "journal_write_header_frequency": "0", 2024-02-09T17:24:26.253 INFO:teuthology.orchestra.run.smithi099.stdout: "journal_zero_on_create": "false", 2024-02-09T17:24:26.254 INFO:teuthology.orchestra.run.smithi099.stdout: "journaler_prefetch_periods": "10", 2024-02-09T17:24:26.254 INFO:teuthology.orchestra.run.smithi099.stdout: "journaler_prezero_periods": "5", 2024-02-09T17:24:26.254 INFO:teuthology.orchestra.run.smithi099.stdout: "journaler_write_head_interval": "15", 2024-02-09T17:24:26.254 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "", 2024-02-09T17:24:26.254 INFO:teuthology.orchestra.run.smithi099.stdout: "keyfile": "", 2024-02-09T17:24:26.254 INFO:teuthology.orchestra.run.smithi099.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-02-09T17:24:26.254 INFO:teuthology.orchestra.run.smithi099.stdout: "kstore_backend": "rocksdb", 2024-02-09T17:24:26.254 INFO:teuthology.orchestra.run.smithi099.stdout: "kstore_default_stripe_size": "65536", 2024-02-09T17:24:26.254 INFO:teuthology.orchestra.run.smithi099.stdout: "kstore_fsck_on_mount": "false", 2024-02-09T17:24:26.254 INFO:teuthology.orchestra.run.smithi099.stdout: "kstore_fsck_on_mount_deep": "true", 2024-02-09T17:24:26.254 INFO:teuthology.orchestra.run.smithi099.stdout: "kstore_max_bytes": "67108864", 2024-02-09T17:24:26.254 INFO:teuthology.orchestra.run.smithi099.stdout: "kstore_max_ops": "512", 2024-02-09T17:24:26.254 INFO:teuthology.orchestra.run.smithi099.stdout: "kstore_nid_prealloc": "1024", 2024-02-09T17:24:26.254 INFO:teuthology.orchestra.run.smithi099.stdout: "kstore_onode_map_size": "1024", 2024-02-09T17:24:26.254 INFO:teuthology.orchestra.run.smithi099.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-02-09T17:24:26.255 INFO:teuthology.orchestra.run.smithi099.stdout: "kstore_sync_submit_transaction": "false", 2024-02-09T17:24:26.255 INFO:teuthology.orchestra.run.smithi099.stdout: "kstore_sync_transaction": "false", 2024-02-09T17:24:26.255 INFO:teuthology.orchestra.run.smithi099.stdout: "librados_thread_count": "2", 2024-02-09T17:24:26.255 INFO:teuthology.orchestra.run.smithi099.stdout: "lockdep": "false", 2024-02-09T17:24:26.255 INFO:teuthology.orchestra.run.smithi099.stdout: "lockdep_force_backtrace": "false", 2024-02-09T17:24:26.255 INFO:teuthology.orchestra.run.smithi099.stdout: "log_coarse_timestamps": "true", 2024-02-09T17:24:26.255 INFO:teuthology.orchestra.run.smithi099.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-02-09T17:24:26.255 INFO:teuthology.orchestra.run.smithi099.stdout: "log_flush_on_exit": "false", 2024-02-09T17:24:26.255 INFO:teuthology.orchestra.run.smithi099.stdout: "log_graylog_host": "127.0.0.1", 2024-02-09T17:24:26.255 INFO:teuthology.orchestra.run.smithi099.stdout: "log_graylog_port": "12201", 2024-02-09T17:24:26.255 INFO:teuthology.orchestra.run.smithi099.stdout: "log_max_new": "1000", 2024-02-09T17:24:26.255 INFO:teuthology.orchestra.run.smithi099.stdout: "log_max_recent": "10000", 2024-02-09T17:24:26.255 INFO:teuthology.orchestra.run.smithi099.stdout: "log_stderr_prefix": "", 2024-02-09T17:24:26.255 INFO:teuthology.orchestra.run.smithi099.stdout: "log_stop_at_utilization": "0.970000", 2024-02-09T17:24:26.255 INFO:teuthology.orchestra.run.smithi099.stdout: "log_to_file": "true", 2024-02-09T17:24:26.256 INFO:teuthology.orchestra.run.smithi099.stdout: "log_to_graylog": "false", 2024-02-09T17:24:26.256 INFO:teuthology.orchestra.run.smithi099.stdout: "log_to_journald": "false", 2024-02-09T17:24:26.256 INFO:teuthology.orchestra.run.smithi099.stdout: "log_to_stderr": "false", 2024-02-09T17:24:26.256 INFO:teuthology.orchestra.run.smithi099.stdout: "log_to_syslog": "false", 2024-02-09T17:24:26.256 INFO:teuthology.orchestra.run.smithi099.stdout: "max_rotating_auth_attempts": "10", 2024-02-09T17:24:26.256 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-02-09T17:24:26.256 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_action_on_write_error": "1", 2024-02-09T17:24:26.256 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_alternate_name_max": "8192", 2024-02-09T17:24:26.256 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_asio_thread_count": "2", 2024-02-09T17:24:26.256 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_bal_export_pin": "true", 2024-02-09T17:24:26.256 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_bal_fragment_dirs": "true", 2024-02-09T17:24:26.256 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-02-09T17:24:26.256 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_bal_fragment_interval": "5", 2024-02-09T17:24:26.256 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_bal_fragment_size_max": "100000", 2024-02-09T17:24:26.257 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_bal_idle_threshold": "0.000000", 2024-02-09T17:24:26.257 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_bal_interval": "10", 2024-02-09T17:24:26.257 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_bal_max": "-1", 2024-02-09T17:24:26.257 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_bal_max_until": "-1", 2024-02-09T17:24:26.257 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_bal_merge_size": "50", 2024-02-09T17:24:26.257 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_bal_midchunk": "0.300000", 2024-02-09T17:24:26.257 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_bal_min_rebalance": "0.100000", 2024-02-09T17:24:26.257 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_bal_min_start": "0.200000", 2024-02-09T17:24:26.257 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_bal_minchunk": "0.001000", 2024-02-09T17:24:26.257 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_bal_mode": "0", 2024-02-09T17:24:26.257 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_bal_need_max": "1.200000", 2024-02-09T17:24:26.257 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_bal_need_min": "0.800000", 2024-02-09T17:24:26.257 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_bal_overload_epochs": "2", 2024-02-09T17:24:26.257 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-02-09T17:24:26.258 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_bal_sample_interval": "3.000000", 2024-02-09T17:24:26.258 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_bal_split_bits": "3", 2024-02-09T17:24:26.258 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_bal_split_rd": "25000.000000", 2024-02-09T17:24:26.258 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_bal_split_size": "10000", 2024-02-09T17:24:26.258 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_bal_split_wr": "10000.000000", 2024-02-09T17:24:26.258 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_bal_target_decay": "10.000000", 2024-02-09T17:24:26.258 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-02-09T17:24:26.258 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_beacon_grace": "15.000000", 2024-02-09T17:24:26.258 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_beacon_interval": "4.000000", 2024-02-09T17:24:26.258 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_beacon_mon_down_grace": "60", 2024-02-09T17:24:26.258 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_cache_memory_limit": "4294967296", 2024-02-09T17:24:26.258 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_cache_mid": "0.700000", 2024-02-09T17:24:26.258 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_cache_release_free_interval": "10", 2024-02-09T17:24:26.258 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_cache_reservation": "0.050000", 2024-02-09T17:24:26.259 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-02-09T17:24:26.259 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_cache_trim_interval": "1", 2024-02-09T17:24:26.259 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_cache_trim_threshold": "262144", 2024-02-09T17:24:26.259 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-02-09T17:24:26.259 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-02-09T17:24:26.259 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_client_delegate_inos_pct": "50", 2024-02-09T17:24:26.259 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_client_prealloc_inos": "1000", 2024-02-09T17:24:26.259 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-02-09T17:24:26.259 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_connect_bootstrapping": "false", 2024-02-09T17:24:26.259 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_damage_table_max_entries": "10000", 2024-02-09T17:24:26.259 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-02-09T17:24:26.259 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_debug_auth_pins": "false", 2024-02-09T17:24:26.259 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_debug_frag": "false", 2024-02-09T17:24:26.260 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_debug_scatterstat": "false", 2024-02-09T17:24:26.260 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_debug_subtrees": "false", 2024-02-09T17:24:26.260 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_decay_halflife": "5.000000", 2024-02-09T17:24:26.260 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_default_dir_hash": "2", 2024-02-09T17:24:26.260 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_defer_session_stale": "true", 2024-02-09T17:24:26.260 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_deny_all_reconnect": "false", 2024-02-09T17:24:26.260 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_dir_keys_per_op": "16384", 2024-02-09T17:24:26.260 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_dir_max_commit_size": "10", 2024-02-09T17:24:26.260 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_dir_max_entries": "0", 2024-02-09T17:24:26.260 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_dir_prefetch": "true", 2024-02-09T17:24:26.260 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_dirstat_min_interval": "1.000000", 2024-02-09T17:24:26.260 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_dump_cache_after_rejoin": "false", 2024-02-09T17:24:26.261 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_dump_cache_on_map": "false", 2024-02-09T17:24:26.261 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_dump_cache_threshold_file": "0", 2024-02-09T17:24:26.261 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-02-09T17:24:26.261 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_early_reply": "true", 2024-02-09T17:24:26.261 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_enable_op_tracker": "true", 2024-02-09T17:24:26.261 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_enforce_unique_name": "true", 2024-02-09T17:24:26.261 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_export_ephemeral_distributed": "true", 2024-02-09T17:24:26.261 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-02-09T17:24:26.261 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_export_ephemeral_random": "true", 2024-02-09T17:24:26.261 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-02-09T17:24:26.261 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-02-09T17:24:26.261 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_forward_all_requests_to_auth": "false", 2024-02-09T17:24:26.261 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-02-09T17:24:26.261 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-02-09T17:24:26.261 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-02-09T17:24:26.262 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-02-09T17:24:26.262 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_health_cache_threshold": "1.500000", 2024-02-09T17:24:26.262 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_health_summarize_threshold": "10", 2024-02-09T17:24:26.262 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_heartbeat_grace": "15.000000", 2024-02-09T17:24:26.262 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_heartbeat_reset_grace": "1000", 2024-02-09T17:24:26.262 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_inject_health_dummy": "false", 2024-02-09T17:24:26.262 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-02-09T17:24:26.262 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_inject_migrator_session_race": "false", 2024-02-09T17:24:26.262 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-02-09T17:24:26.262 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-02-09T17:24:26.262 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-02-09T17:24:26.262 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_join_fs": "", 2024-02-09T17:24:26.262 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_journal_format": "1", 2024-02-09T17:24:26.263 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_kill_create_at": "0", 2024-02-09T17:24:26.263 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_kill_export_at": "0", 2024-02-09T17:24:26.263 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_kill_import_at": "0", 2024-02-09T17:24:26.263 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_kill_journal_at": "0", 2024-02-09T17:24:26.263 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_kill_journal_expire_at": "0", 2024-02-09T17:24:26.263 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_kill_journal_replay_at": "0", 2024-02-09T17:24:26.263 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_kill_link_at": "0", 2024-02-09T17:24:26.263 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_kill_mdstable_at": "0", 2024-02-09T17:24:26.263 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_kill_openc_at": "0", 2024-02-09T17:24:26.263 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_kill_rename_at": "0", 2024-02-09T17:24:26.263 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_kill_shutdown_at": "0", 2024-02-09T17:24:26.263 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-02-09T17:24:26.263 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_log_event_large_threshold": "524288", 2024-02-09T17:24:26.264 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_log_events_per_segment": "1024", 2024-02-09T17:24:26.264 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_log_major_segment_event_ratio": "12", 2024-02-09T17:24:26.264 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_log_max_events": "-1", 2024-02-09T17:24:26.264 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_log_max_segments": "128", 2024-02-09T17:24:26.264 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_log_pause": "false", 2024-02-09T17:24:26.264 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_log_segment_size": "0", 2024-02-09T17:24:26.264 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_log_skip_corrupt_events": "false", 2024-02-09T17:24:26.264 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_log_skip_unbounded_events": "false", 2024-02-09T17:24:26.264 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-02-09T17:24:26.264 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_log_trim_threshold": "128", 2024-02-09T17:24:26.264 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-02-09T17:24:26.264 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_log_warn_factor": "2.000000", 2024-02-09T17:24:26.264 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_max_caps_per_client": "1048576", 2024-02-09T17:24:26.264 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_max_completed_flushes": "100000", 2024-02-09T17:24:26.265 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_max_completed_requests": "100000", 2024-02-09T17:24:26.265 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_max_export_size": "20971520", 2024-02-09T17:24:26.265 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_max_file_recover": "32", 2024-02-09T17:24:26.265 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_max_purge_files": "64", 2024-02-09T17:24:26.265 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_max_purge_ops": "8192", 2024-02-09T17:24:26.265 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-02-09T17:24:26.265 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-02-09T17:24:26.265 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_max_snaps_per_dir": "100", 2024-02-09T17:24:26.265 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_metrics_update_interval": "2", 2024-02-09T17:24:26.265 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_min_caps_per_client": "100", 2024-02-09T17:24:26.265 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_min_caps_working_set": "10000", 2024-02-09T17:24:26.265 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-02-09T17:24:26.265 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_numa_node": "-1", 2024-02-09T17:24:26.265 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-02-09T17:24:26.266 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_op_complaint_time": "30.000000", 2024-02-09T17:24:26.266 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_op_history_duration": "600", 2024-02-09T17:24:26.266 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_op_history_size": "20", 2024-02-09T17:24:26.266 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_op_history_slow_op_size": "20", 2024-02-09T17:24:26.266 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_op_history_slow_op_threshold": "10", 2024-02-09T17:24:26.266 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_op_log_threshold": "5", 2024-02-09T17:24:26.266 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_ping_grace": "15", 2024-02-09T17:24:26.266 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_ping_interval": "5", 2024-02-09T17:24:26.266 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-02-09T17:24:26.266 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-02-09T17:24:26.266 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_recall_max_caps": "30000", 2024-02-09T17:24:26.266 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-02-09T17:24:26.266 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_recall_max_decay_threshold": "131072", 2024-02-09T17:24:26.267 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-02-09T17:24:26.267 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_recall_warning_threshold": "262144", 2024-02-09T17:24:26.267 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_reconnect_timeout": "45.000000", 2024-02-09T17:24:26.267 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_replay_interval": "1.000000", 2024-02-09T17:24:26.267 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-02-09T17:24:26.267 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-02-09T17:24:26.267 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_root_ino_gid": "0", 2024-02-09T17:24:26.267 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_root_ino_uid": "0", 2024-02-09T17:24:26.267 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-02-09T17:24:26.267 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_session_blocklist_on_evict": "true", 2024-02-09T17:24:26.267 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_session_blocklist_on_timeout": "true", 2024-02-09T17:24:26.267 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-02-09T17:24:26.267 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-02-09T17:24:26.268 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-02-09T17:24:26.268 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-02-09T17:24:26.268 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-02-09T17:24:26.268 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_session_metadata_threshold": "16777216", 2024-02-09T17:24:26.268 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-02-09T17:24:26.268 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_shutdown_check": "0", 2024-02-09T17:24:26.268 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_skip_ino": "0", 2024-02-09T17:24:26.268 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_sleep_rank_change": "0.000000", 2024-02-09T17:24:26.268 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_snap_max_uid": "4294967294", 2024-02-09T17:24:26.268 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_snap_min_uid": "0", 2024-02-09T17:24:26.268 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_snap_rstat": "false", 2024-02-09T17:24:26.268 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_standby_replay_damaged": "false", 2024-02-09T17:24:26.268 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_symlink_recovery": "true", 2024-02-09T17:24:26.268 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_task_status_update_interval": "2.000000", 2024-02-09T17:24:26.269 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_thrash_exports": "0", 2024-02-09T17:24:26.269 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_thrash_fragments": "0", 2024-02-09T17:24:26.269 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_tick_interval": "5.000000", 2024-02-09T17:24:26.269 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_valgrind_exit": "false", 2024-02-09T17:24:26.270 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_verify_backtrace": "1", 2024-02-09T17:24:26.270 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_verify_scatter": "false", 2024-02-09T17:24:26.270 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_wipe_ino_prealloc": "false", 2024-02-09T17:24:26.270 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_wipe_sessions": "false", 2024-02-09T17:24:26.270 INFO:teuthology.orchestra.run.smithi099.stdout: "mempool_debug": "false", 2024-02-09T17:24:26.270 INFO:teuthology.orchestra.run.smithi099.stdout: "memstore_debug_omit_block_device_write": "false", 2024-02-09T17:24:26.270 INFO:teuthology.orchestra.run.smithi099.stdout: "memstore_device_bytes": "1073741824", 2024-02-09T17:24:26.270 INFO:teuthology.orchestra.run.smithi099.stdout: "memstore_page_set": "false", 2024-02-09T17:24:26.270 INFO:teuthology.orchestra.run.smithi099.stdout: "memstore_page_size": "65536", 2024-02-09T17:24:26.270 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr_client_bytes": "134217728", 2024-02-09T17:24:26.270 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr_client_messages": "512", 2024-02-09T17:24:26.270 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-02-09T17:24:26.270 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr_connect_retry_interval": "1.000000", 2024-02-09T17:24:26.270 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-02-09T17:24:26.270 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-02-09T17:24:26.271 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr_disabled_modules": "", 2024-02-09T17:24:26.271 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-02-09T17:24:26.271 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr_max_pg_num_change": "128", 2024-02-09T17:24:26.271 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr_mds_bytes": "134217728", 2024-02-09T17:24:26.271 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr_mds_messages": "128", 2024-02-09T17:24:26.271 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-02-09T17:24:26.271 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr_mon_bytes": "134217728", 2024-02-09T17:24:26.271 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr_mon_messages": "128", 2024-02-09T17:24:26.271 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr_osd_bytes": "536870912", 2024-02-09T17:24:26.271 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr_osd_messages": "8192", 2024-02-09T17:24:26.271 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr_pool": "true", 2024-02-09T17:24:26.271 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr_service_beacon_grace": "60.000000", 2024-02-09T17:24:26.271 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr_standby_modules": "true", 2024-02-09T17:24:26.271 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr_stats_period": "5", 2024-02-09T17:24:26.272 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr_stats_threshold": "5", 2024-02-09T17:24:26.272 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr_tick_period": "2", 2024-02-09T17:24:26.272 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-02-09T17:24:26.272 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_accept_timeout_factor": "2.000000", 2024-02-09T17:24:26.272 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_allow_pool_delete": "true", 2024-02-09T17:24:26.272 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_allow_pool_size_one": "false", 2024-02-09T17:24:26.272 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_auth_validate_all_caps": "true", 2024-02-09T17:24:26.272 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-02-09T17:24:26.272 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-02-09T17:24:26.272 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_client_bytes": "104857600", 2024-02-09T17:24:26.272 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_client_directed_command_retry": "5", 2024-02-09T17:24:26.272 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_client_hunt_interval": "3.000000", 2024-02-09T17:24:26.272 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-02-09T17:24:26.272 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-02-09T17:24:26.273 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-02-09T17:24:26.273 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_client_hunt_parallel": "3", 2024-02-09T17:24:26.273 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_client_log_interval": "1.000000", 2024-02-09T17:24:26.273 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-02-09T17:24:26.273 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_client_ping_interval": "10.000000", 2024-02-09T17:24:26.273 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_client_ping_timeout": "30.000000", 2024-02-09T17:24:26.273 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_client_target_rank": "-1", 2024-02-09T17:24:26.273 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_clock_drift_allowed": "1.000000", 2024-02-09T17:24:26.273 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-02-09T17:24:26.273 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-02-09T17:24:26.273 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_cluster_log_file_level": "debug", 2024-02-09T17:24:26.273 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_cluster_log_to_file": "true", 2024-02-09T17:24:26.273 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_cluster_log_to_graylog": "false", 2024-02-09T17:24:26.273 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-02-09T17:24:26.274 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-02-09T17:24:26.274 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_cluster_log_to_journald": "true", 2024-02-09T17:24:26.274 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_cluster_log_to_stderr": "false", 2024-02-09T17:24:26.274 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-02-09T17:24:26.274 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-02-09T17:24:26.274 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-02-09T17:24:26.274 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_compact_on_bootstrap": "false", 2024-02-09T17:24:26.274 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_compact_on_start": "false", 2024-02-09T17:24:26.274 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_compact_on_trim": "true", 2024-02-09T17:24:26.274 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_con_tracker_persist_interval": "10", 2024-02-09T17:24:26.274 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_con_tracker_score_halflife": "43200", 2024-02-09T17:24:26.274 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_config_key_max_entry_size": "65536", 2024-02-09T17:24:26.274 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_cpu_threads": "4", 2024-02-09T17:24:26.274 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_crush_min_required_version": "hammer", 2024-02-09T17:24:26.274 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_daemon_bytes": "419430400", 2024-02-09T17:24:26.275 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-02-09T17:24:26.275 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_data_avail_crit": "5", 2024-02-09T17:24:26.275 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_data_avail_warn": "5", 2024-02-09T17:24:26.275 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_data_size_warn": "16106127360", 2024-02-09T17:24:26.275 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_debug_block_osdmap_trim": "false", 2024-02-09T17:24:26.275 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-02-09T17:24:26.275 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_debug_dump_json": "false", 2024-02-09T17:24:26.275 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-02-09T17:24:26.275 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_debug_dump_transactions": "false", 2024-02-09T17:24:26.275 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_debug_extra_checks": "false", 2024-02-09T17:24:26.275 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-02-09T17:24:26.275 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-02-09T17:24:26.275 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_debug_no_require_reef": "false", 2024-02-09T17:24:26.275 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_debug_no_require_squid": "false", 2024-02-09T17:24:26.276 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-02-09T17:24:26.276 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_delta_reset_interval": "10.000000", 2024-02-09T17:24:26.276 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_dns_srv_name": "ceph-mon", 2024-02-09T17:24:26.276 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_down_mkfs_grace": "60", 2024-02-09T17:24:26.276 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_election_default_strategy": "3", 2024-02-09T17:24:26.276 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_election_timeout": "5.000000", 2024-02-09T17:24:26.276 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-02-09T17:24:26.276 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_elector_ping_divisor": "2", 2024-02-09T17:24:26.276 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_elector_ping_timeout": "2.000000", 2024-02-09T17:24:26.276 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_enable_op_tracker": "true", 2024-02-09T17:24:26.276 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_fake_pool_delete": "false", 2024-02-09T17:24:26.276 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_force_quorum_join": "false", 2024-02-09T17:24:26.276 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_globalid_prealloc": "10000", 2024-02-09T17:24:26.276 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_health_detail_to_clog": "true", 2024-02-09T17:24:26.277 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_health_log_update_period": "5", 2024-02-09T17:24:26.277 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_health_max_detail": "50", 2024-02-09T17:24:26.277 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_health_to_clog": "true", 2024-02-09T17:24:26.277 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_health_to_clog_interval": "600", 2024-02-09T17:24:26.277 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-02-09T17:24:26.277 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_host": "[v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0]", 2024-02-09T17:24:26.277 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_host_override": "", 2024-02-09T17:24:26.277 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_initial_members": "", 2024-02-09T17:24:26.277 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-02-09T17:24:26.277 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-02-09T17:24:26.277 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-02-09T17:24:26.277 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-02-09T17:24:26.277 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_keyvaluedb": "rocksdb", 2024-02-09T17:24:26.277 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_lease": "5.000000", 2024-02-09T17:24:26.277 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-02-09T17:24:26.278 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-02-09T17:24:26.278 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_log_full_interval": "50", 2024-02-09T17:24:26.278 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_log_max": "10000", 2024-02-09T17:24:26.278 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_log_max_summary": "50", 2024-02-09T17:24:26.278 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_max_log_entries_per_event": "4096", 2024-02-09T17:24:26.278 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_max_log_epochs": "500", 2024-02-09T17:24:26.278 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_max_mdsmap_epochs": "500", 2024-02-09T17:24:26.278 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_max_mgrmap_epochs": "500", 2024-02-09T17:24:26.278 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_max_osd": "10000", 2024-02-09T17:24:26.278 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_max_pg_per_osd": "10000", 2024-02-09T17:24:26.278 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_max_pool_pg_num": "65536", 2024-02-09T17:24:26.278 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-02-09T17:24:26.278 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-02-09T17:24:26.278 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_mds_force_trim_to": "0", 2024-02-09T17:24:26.278 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_mds_skip_sanity": "false", 2024-02-09T17:24:26.279 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_memory_autotune": "true", 2024-02-09T17:24:26.279 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_memory_target": "2147483648", 2024-02-09T17:24:26.279 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_mgr_beacon_grace": "30", 2024-02-09T17:24:26.279 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-02-09T17:24:26.279 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_mgr_digest_period": "5", 2024-02-09T17:24:26.279 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_mgr_inactive_grace": "60", 2024-02-09T17:24:26.279 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_mgr_mkfs_grace": "240", 2024-02-09T17:24:26.279 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-02-09T17:24:26.279 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_min_osdmap_epochs": "50", 2024-02-09T17:24:26.279 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_op_complaint_time": "30", 2024-02-09T17:24:26.279 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_op_history_duration": "600", 2024-02-09T17:24:26.279 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_op_history_size": "20", 2024-02-09T17:24:26.279 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_op_history_slow_op_size": "20", 2024-02-09T17:24:26.279 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_op_history_slow_op_threshold": "10", 2024-02-09T17:24:26.279 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_op_log_threshold": "5", 2024-02-09T17:24:26.279 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-02-09T17:24:26.280 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-02-09T17:24:26.280 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-02-09T17:24:26.280 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_osd_auto_mark_in": "false", 2024-02-09T17:24:26.280 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_osd_auto_mark_new_in": "true", 2024-02-09T17:24:26.280 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-02-09T17:24:26.280 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-02-09T17:24:26.280 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_osd_cache_size": "500", 2024-02-09T17:24:26.280 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_osd_cache_size_min": "134217728", 2024-02-09T17:24:26.280 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_osd_crush_smoke_test": "true", 2024-02-09T17:24:26.280 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_osd_destroyed_out_interval": "600", 2024-02-09T17:24:26.280 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_osd_down_out_interval": "600", 2024-02-09T17:24:26.281 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-02-09T17:24:26.281 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-02-09T17:24:26.281 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_osd_force_trim_to": "0", 2024-02-09T17:24:26.281 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_osd_full_ratio": "0.950000", 2024-02-09T17:24:26.281 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-02-09T17:24:26.281 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_osd_laggy_halflife": "3600", 2024-02-09T17:24:26.281 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_osd_laggy_max_interval": "300", 2024-02-09T17:24:26.281 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_osd_laggy_weight": "0.300000", 2024-02-09T17:24:26.281 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-02-09T17:24:26.281 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_osd_max_creating_pgs": "1024", 2024-02-09T17:24:26.281 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_osd_max_initial_pgs": "1024", 2024-02-09T17:24:26.281 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_osd_min_down_reporters": "2", 2024-02-09T17:24:26.281 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-02-09T17:24:26.281 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-02-09T17:24:26.281 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-02-09T17:24:26.282 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_osd_prime_pg_temp": "true", 2024-02-09T17:24:26.282 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-02-09T17:24:26.282 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-02-09T17:24:26.282 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_osd_report_timeout": "900", 2024-02-09T17:24:26.282 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-02-09T17:24:26.282 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-02-09T17:24:26.282 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_osd_warn_num_repaired": "10", 2024-02-09T17:24:26.282 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_osd_warn_op_age": "32.000000", 2024-02-09T17:24:26.282 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-02-09T17:24:26.282 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_osdmap_full_prune_interval": "2", 2024-02-09T17:24:26.282 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_osdmap_full_prune_min": "15", 2024-02-09T17:24:26.282 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-02-09T17:24:26.282 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-02-09T17:24:26.282 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_pg_stuck_threshold": "60", 2024-02-09T17:24:26.282 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-02-09T17:24:26.283 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_pg_warn_min_objects": "10000", 2024-02-09T17:24:26.283 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_pg_warn_min_per_osd": "0", 2024-02-09T17:24:26.283 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-02-09T17:24:26.283 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_pool_quota_crit_threshold": "0", 2024-02-09T17:24:26.283 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_pool_quota_warn_threshold": "0", 2024-02-09T17:24:26.283 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_probe_timeout": "2.000000", 2024-02-09T17:24:26.283 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_reweight_max_change": "0.050000", 2024-02-09T17:24:26.283 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_reweight_max_osds": "4", 2024-02-09T17:24:26.283 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-02-09T17:24:26.283 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-02-09T17:24:26.283 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-02-09T17:24:26.283 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-02-09T17:24:26.283 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-02-09T17:24:26.283 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_scrub_interval": "300", 2024-02-09T17:24:26.283 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_scrub_max_keys": "100", 2024-02-09T17:24:26.284 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_scrub_timeout": "300", 2024-02-09T17:24:26.284 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_session_timeout": "300", 2024-02-09T17:24:26.284 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_smart_report_timeout": "5", 2024-02-09T17:24:26.284 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_stat_smooth_intervals": "6", 2024-02-09T17:24:26.284 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-02-09T17:24:26.284 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_stretch_pool_min_size": "2", 2024-02-09T17:24:26.284 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_stretch_pool_size": "4", 2024-02-09T17:24:26.284 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-02-09T17:24:26.284 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_subscribe_interval": "86400.000000", 2024-02-09T17:24:26.284 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_sync_debug": "false", 2024-02-09T17:24:26.284 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_sync_max_payload_keys": "2000", 2024-02-09T17:24:26.284 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_sync_max_payload_size": "1048576", 2024-02-09T17:24:26.284 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_sync_provider_kill_at": "0", 2024-02-09T17:24:26.284 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_sync_requester_kill_at": "0", 2024-02-09T17:24:26.284 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_sync_timeout": "60.000000", 2024-02-09T17:24:26.285 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_target_pg_per_osd": "100", 2024-02-09T17:24:26.285 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_tick_interval": "5", 2024-02-09T17:24:26.285 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_timecheck_interval": "300.000000", 2024-02-09T17:24:26.285 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-02-09T17:24:26.285 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_use_min_delay_socket": "false", 2024-02-09T17:24:26.285 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_warn_older_version_delay": "604800", 2024-02-09T17:24:26.285 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-02-09T17:24:26.285 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-02-09T17:24:26.285 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-02-09T17:24:26.285 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_warn_on_filestore_osds": "true", 2024-02-09T17:24:26.285 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-02-09T17:24:26.285 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-02-09T17:24:26.285 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-02-09T17:24:26.285 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_warn_on_misplaced": "false", 2024-02-09T17:24:26.285 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-02-09T17:24:26.286 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_warn_on_older_version": "true", 2024-02-09T17:24:26.286 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-02-09T17:24:26.286 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_warn_on_pool_no_app": "true", 2024-02-09T17:24:26.286 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-02-09T17:24:26.286 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-02-09T17:24:26.286 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-02-09T17:24:26.286 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-02-09T17:24:26.286 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_warn_on_too_few_osds": "false", 2024-02-09T17:24:26.286 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-02-09T17:24:26.286 INFO:teuthology.orchestra.run.smithi099.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-02-09T17:24:26.286 INFO:teuthology.orchestra.run.smithi099.stdout: "monmap": "", 2024-02-09T17:24:26.286 INFO:teuthology.orchestra.run.smithi099.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-02-09T17:24:26.286 INFO:teuthology.orchestra.run.smithi099.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-02-09T17:24:26.286 INFO:teuthology.orchestra.run.smithi099.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-02-09T17:24:26.287 INFO:teuthology.orchestra.run.smithi099.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-02-09T17:24:26.287 INFO:teuthology.orchestra.run.smithi099.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-02-09T17:24:26.287 INFO:teuthology.orchestra.run.smithi099.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-02-09T17:24:26.287 INFO:teuthology.orchestra.run.smithi099.stdout: "motr_tracing_enabled": "false", 2024-02-09T17:24:26.287 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_async_op_threads": "3", 2024-02-09T17:24:26.287 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_async_rdma_buffer_size": "131072", 2024-02-09T17:24:26.287 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_async_rdma_cm": "false", 2024-02-09T17:24:26.287 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_async_rdma_device_name": "", 2024-02-09T17:24:26.287 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_async_rdma_dscp": "96", 2024-02-09T17:24:26.287 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-02-09T17:24:26.287 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_async_rdma_gid_idx": "0", 2024-02-09T17:24:26.287 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_async_rdma_local_gid": "", 2024-02-09T17:24:26.287 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_async_rdma_polling_us": "1000", 2024-02-09T17:24:26.287 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_async_rdma_port_num": "1", 2024-02-09T17:24:26.287 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-02-09T17:24:26.288 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-02-09T17:24:26.288 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_async_rdma_roce_ver": "1", 2024-02-09T17:24:26.288 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_async_rdma_send_buffers": "1024", 2024-02-09T17:24:26.288 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_async_rdma_sl": "3", 2024-02-09T17:24:26.288 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_async_rdma_support_srq": "true", 2024-02-09T17:24:26.288 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_async_rdma_type": "ib", 2024-02-09T17:24:26.288 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_async_reap_threshold": "5", 2024-02-09T17:24:26.288 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_bind_before_connect": "false", 2024-02-09T17:24:26.288 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_bind_ipv4": "true", 2024-02-09T17:24:26.288 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_bind_ipv6": "false", 2024-02-09T17:24:26.288 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_bind_msgr1": "true", 2024-02-09T17:24:26.288 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_bind_msgr2": "true", 2024-02-09T17:24:26.288 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_bind_port_max": "7568", 2024-02-09T17:24:26.288 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_bind_port_min": "6800", 2024-02-09T17:24:26.288 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_bind_prefer_ipv4": "false", 2024-02-09T17:24:26.289 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_bind_retry_count": "3", 2024-02-09T17:24:26.289 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_bind_retry_delay": "5", 2024-02-09T17:24:26.289 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_blackhole_client": "false", 2024-02-09T17:24:26.289 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_blackhole_mds": "false", 2024-02-09T17:24:26.289 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_blackhole_mgr": "false", 2024-02-09T17:24:26.289 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_blackhole_mon": "false", 2024-02-09T17:24:26.289 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_blackhole_osd": "false", 2024-02-09T17:24:26.289 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_client_mode": "crc secure", 2024-02-09T17:24:26.289 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_cluster_mode": "crc secure", 2024-02-09T17:24:26.289 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_cluster_type": "", 2024-02-09T17:24:26.289 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_compress_secure": "false", 2024-02-09T17:24:26.289 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_connection_idle_timeout": "900", 2024-02-09T17:24:26.289 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_connection_ready_timeout": "10", 2024-02-09T17:24:26.289 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_crc_data": "true", 2024-02-09T17:24:26.289 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_crc_header": "true", 2024-02-09T17:24:26.290 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_die_on_bad_msg": "false", 2024-02-09T17:24:26.290 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_die_on_bug": "true", 2024-02-09T17:24:26.290 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_die_on_old_message": "true", 2024-02-09T17:24:26.290 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_die_on_skipped_message": "false", 2024-02-09T17:24:26.290 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_die_on_unhandled_msg": "false", 2024-02-09T17:24:26.290 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-02-09T17:24:26.290 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_dpdk_coremask": "0xF", 2024-02-09T17:24:26.290 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-02-09T17:24:26.290 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_dpdk_devs_allowlist": "", 2024-02-09T17:24:26.290 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_dpdk_enable_tso": "true", 2024-02-09T17:24:26.290 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-02-09T17:24:26.290 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-02-09T17:24:26.290 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_dpdk_hugepages": "", 2024-02-09T17:24:26.290 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_dpdk_hw_flow_control": "true", 2024-02-09T17:24:26.290 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-02-09T17:24:26.291 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_dpdk_lro": "true", 2024-02-09T17:24:26.291 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_dpdk_memory_channel": "4", 2024-02-09T17:24:26.291 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-02-09T17:24:26.291 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_dpdk_pmd": "", 2024-02-09T17:24:26.291 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_dpdk_port_id": "0", 2024-02-09T17:24:26.291 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-02-09T17:24:26.291 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_dump_corrupt_message_level": "1", 2024-02-09T17:24:26.291 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_dump_on_send": "false", 2024-02-09T17:24:26.291 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_initial_backoff": "0.200000", 2024-02-09T17:24:26.291 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_inject_delay_max": "1.000000", 2024-02-09T17:24:26.291 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_inject_delay_probability": "0.000000", 2024-02-09T17:24:26.291 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_inject_delay_type": "", 2024-02-09T17:24:26.291 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_inject_internal_delays": "0.000000", 2024-02-09T17:24:26.291 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_inject_network_congestion": "0", 2024-02-09T17:24:26.292 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_inject_socket_failures": "2500", 2024-02-09T17:24:26.292 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_learn_addr_from_peer": "true", 2024-02-09T17:24:26.292 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_max_accept_failures": "4", 2024-02-09T17:24:26.292 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_max_backoff": "15.000000", 2024-02-09T17:24:26.292 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_mon_client_mode": "secure crc", 2024-02-09T17:24:26.292 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_mon_cluster_mode": "secure crc", 2024-02-09T17:24:26.292 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_mon_service_mode": "secure crc", 2024-02-09T17:24:26.292 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_osd_compress_min_size": "1024", 2024-02-09T17:24:26.292 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_osd_compress_mode": "none", 2024-02-09T17:24:26.292 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_osd_compression_algorithm": "snappy", 2024-02-09T17:24:26.292 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-02-09T17:24:26.292 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_pq_min_cost": "65536", 2024-02-09T17:24:26.292 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_public_type": "", 2024-02-09T17:24:26.292 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_service_mode": "crc secure", 2024-02-09T17:24:26.293 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_tcp_listen_backlog": "512", 2024-02-09T17:24:26.293 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_tcp_nodelay": "true", 2024-02-09T17:24:26.293 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-02-09T17:24:26.293 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_tcp_rcvbuf": "0", 2024-02-09T17:24:26.293 INFO:teuthology.orchestra.run.smithi099.stdout: "ms_type": "async+posix", 2024-02-09T17:24:26.293 INFO:teuthology.orchestra.run.smithi099.stdout: "no_config_file": "false", 2024-02-09T17:24:26.293 INFO:teuthology.orchestra.run.smithi099.stdout: "objecter_completion_locks_per_session": "32", 2024-02-09T17:24:26.293 INFO:teuthology.orchestra.run.smithi099.stdout: "objecter_debug_inject_relock_delay": "false", 2024-02-09T17:24:26.293 INFO:teuthology.orchestra.run.smithi099.stdout: "objecter_inflight_op_bytes": "104857600", 2024-02-09T17:24:26.293 INFO:teuthology.orchestra.run.smithi099.stdout: "objecter_inflight_ops": "1024", 2024-02-09T17:24:26.293 INFO:teuthology.orchestra.run.smithi099.stdout: "objecter_inject_no_watch_ping": "false", 2024-02-09T17:24:26.293 INFO:teuthology.orchestra.run.smithi099.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-02-09T17:24:26.293 INFO:teuthology.orchestra.run.smithi099.stdout: "objecter_tick_interval": "5.000000", 2024-02-09T17:24:26.294 INFO:teuthology.orchestra.run.smithi099.stdout: "objecter_timeout": "10.000000", 2024-02-09T17:24:26.294 INFO:teuthology.orchestra.run.smithi099.stdout: "objectstore_blackhole": "false", 2024-02-09T17:24:26.294 INFO:teuthology.orchestra.run.smithi099.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-02-09T17:24:26.294 INFO:teuthology.orchestra.run.smithi099.stdout: "openssl_engine_opts": "", 2024-02-09T17:24:26.294 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_agent_delay_time": "5.000000", 2024-02-09T17:24:26.294 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_agent_hist_halflife": "1000", 2024-02-09T17:24:26.294 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_agent_max_low_ops": "2", 2024-02-09T17:24:26.294 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_agent_max_ops": "4", 2024-02-09T17:24:26.294 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-02-09T17:24:26.294 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_agent_quantize_effort": "0.100000", 2024-02-09T17:24:26.294 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_agent_slop": "0.020000", 2024-02-09T17:24:26.294 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-02-09T17:24:26.294 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_allow_recovery_below_min_size": "true", 2024-02-09T17:24:26.294 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_asio_thread_count": "2", 2024-02-09T17:24:26.294 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_async_recovery_min_cost": "100", 2024-02-09T17:24:26.295 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_auto_mark_unfound_lost": "false", 2024-02-09T17:24:26.295 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_backfill_retry_interval": "30.000000", 2024-02-09T17:24:26.295 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_backfill_scan_max": "512", 2024-02-09T17:24:26.295 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_backfill_scan_min": "64", 2024-02-09T17:24:26.295 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_backoff_on_degraded": "false", 2024-02-09T17:24:26.295 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_backoff_on_peering": "false", 2024-02-09T17:24:26.295 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_backoff_on_unfound": "true", 2024-02-09T17:24:26.295 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_beacon_report_interval": "300", 2024-02-09T17:24:26.295 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_bench_duration": "30", 2024-02-09T17:24:26.295 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-02-09T17:24:26.295 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_bench_max_block_size": "67108864", 2024-02-09T17:24:26.295 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_bench_small_size_max_iops": "100", 2024-02-09T17:24:26.295 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_blkin_trace_all": "false", 2024-02-09T17:24:26.295 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_blocked_scrub_grace_period": "120", 2024-02-09T17:24:26.295 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-02-09T17:24:26.296 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-02-09T17:24:26.296 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-02-09T17:24:26.296 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_check_for_log_corruption": "false", 2024-02-09T17:24:26.296 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-02-09T17:24:26.296 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-02-09T17:24:26.296 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-02-09T17:24:26.296 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-02-09T17:24:26.296 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_class_update_on_start": "true", 2024-02-09T17:24:26.296 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_client_message_cap": "256", 2024-02-09T17:24:26.296 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_client_message_size_cap": "524288000", 2024-02-09T17:24:26.296 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_client_op_priority": "63", 2024-02-09T17:24:26.296 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_client_watch_timeout": "30", 2024-02-09T17:24:26.296 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_command_max_records": "256", 2024-02-09T17:24:26.296 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_compact_on_start": "false", 2024-02-09T17:24:26.297 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-02-09T17:24:26.297 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_crush_chooseleaf_type": "1", 2024-02-09T17:24:26.297 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_crush_initial_weight": "-1.000000", 2024-02-09T17:24:26.297 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_crush_update_on_start": "true", 2024-02-09T17:24:26.297 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_crush_update_weight_set": "true", 2024-02-09T17:24:26.297 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-02-09T17:24:26.297 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-02-09T17:24:26.297 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-02-09T17:24:26.297 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_debug_drop_ping_duration": "0", 2024-02-09T17:24:26.297 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-02-09T17:24:26.297 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_debug_feed_pullee": "-1", 2024-02-09T17:24:26.297 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-02-09T17:24:26.297 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-02-09T17:24:26.297 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-02-09T17:24:26.297 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_debug_misdirected_ops": "false", 2024-02-09T17:24:26.298 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_debug_no_acting_change": "false", 2024-02-09T17:24:26.298 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_debug_no_purge_strays": "false", 2024-02-09T17:24:26.298 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_debug_op_order": "false", 2024-02-09T17:24:26.298 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_debug_pg_log_writeout": "false", 2024-02-09T17:24:26.298 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_debug_pretend_recovery_active": "false", 2024-02-09T17:24:26.298 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-02-09T17:24:26.298 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-02-09T17:24:26.298 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_debug_shutdown": "false", 2024-02-09T17:24:26.298 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-02-09T17:24:26.298 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-02-09T17:24:26.298 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_debug_trim_objects": "false", 2024-02-09T17:24:26.298 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_debug_verify_cached_snaps": "false", 2024-02-09T17:24:26.298 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_debug_verify_missing_on_start": "false", 2024-02-09T17:24:26.298 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_debug_verify_snaps": "false", 2024-02-09T17:24:26.298 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-02-09T17:24:26.299 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-02-09T17:24:26.299 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_deep_scrub_keys": "1024", 2024-02-09T17:24:26.299 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-02-09T17:24:26.299 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-02-09T17:24:26.299 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-02-09T17:24:26.299 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_deep_scrub_stride": "524288", 2024-02-09T17:24:26.299 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-02-09T17:24:26.299 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_default_data_pool_replay_window": "45", 2024-02-09T17:24:26.299 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_default_notify_timeout": "30", 2024-02-09T17:24:26.299 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_delete_sleep": "0.000000", 2024-02-09T17:24:26.299 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-02-09T17:24:26.299 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-02-09T17:24:26.299 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-02-09T17:24:26.299 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_discard_disconnected_ops": "true", 2024-02-09T17:24:26.299 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_enable_op_tracker": "true", 2024-02-09T17:24:26.300 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-02-09T17:24:26.300 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_extblkdev_plugins": "vdo", 2024-02-09T17:24:26.300 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-02-09T17:24:26.300 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-02-09T17:24:26.300 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_fast_info": "true", 2024-02-09T17:24:26.300 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_fast_shutdown": "true", 2024-02-09T17:24:26.300 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-02-09T17:24:26.300 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_fast_shutdown_timeout": "15", 2024-02-09T17:24:26.300 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-02-09T17:24:26.300 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-02-09T17:24:26.300 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-02-09T17:24:26.300 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_function_tracing": "false", 2024-02-09T17:24:26.300 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_heartbeat_grace": "20", 2024-02-09T17:24:26.300 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_heartbeat_interval": "6", 2024-02-09T17:24:26.301 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-02-09T17:24:26.301 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_heartbeat_min_peers": "10", 2024-02-09T17:24:26.301 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_heartbeat_min_size": "2000", 2024-02-09T17:24:26.301 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_heartbeat_stale": "600", 2024-02-09T17:24:26.301 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-02-09T17:24:26.301 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_hit_set_max_size": "100000", 2024-02-09T17:24:26.301 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_hit_set_min_size": "1000", 2024-02-09T17:24:26.301 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-02-09T17:24:26.301 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-02-09T17:24:26.301 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-02-09T17:24:26.301 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-02-09T17:24:26.301 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-02-09T17:24:26.301 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_journal_flush_on_shutdown": "true", 2024-02-09T17:24:26.301 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_journal_size": "5120", 2024-02-09T17:24:26.301 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_kill_backfill_at": "0", 2024-02-09T17:24:26.302 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_loop_before_reset_tphandle": "64", 2024-02-09T17:24:26.302 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_map_cache_size": "50", 2024-02-09T17:24:26.302 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_map_dedup": "true", 2024-02-09T17:24:26.302 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_map_message_max": "40", 2024-02-09T17:24:26.302 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_map_message_max_bytes": "10485760", 2024-02-09T17:24:26.302 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_map_share_max_epochs": "40", 2024-02-09T17:24:26.302 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_max_attr_name_len": "100", 2024-02-09T17:24:26.302 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_max_attr_size": "0", 2024-02-09T17:24:26.302 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_max_backfills": "1", 2024-02-09T17:24:26.302 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_max_markdown_count": "5", 2024-02-09T17:24:26.302 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_max_markdown_period": "600", 2024-02-09T17:24:26.302 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_max_object_name_len": "2048", 2024-02-09T17:24:26.302 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_max_object_namespace_len": "256", 2024-02-09T17:24:26.302 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_max_object_size": "134217728", 2024-02-09T17:24:26.302 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-02-09T17:24:26.303 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_max_omap_entries_per_request": "1024", 2024-02-09T17:24:26.303 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_max_pg_blocked_by": "16", 2024-02-09T17:24:26.303 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_max_pg_log_entries": "10000", 2024-02-09T17:24:26.303 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-02-09T17:24:26.303 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_max_pgls": "1024", 2024-02-09T17:24:26.303 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_max_push_cost": "8388608", 2024-02-09T17:24:26.303 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_max_push_objects": "10", 2024-02-09T17:24:26.303 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_max_scrubs": "3", 2024-02-09T17:24:26.303 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-02-09T17:24:26.303 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_max_trimming_pgs": "2", 2024-02-09T17:24:26.303 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_max_write_op_reply_len": "64", 2024-02-09T17:24:26.303 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_max_write_size": "90", 2024-02-09T17:24:26.303 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-02-09T17:24:26.303 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-02-09T17:24:26.303 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-02-09T17:24:26.304 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-02-09T17:24:26.304 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-02-09T17:24:26.304 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-02-09T17:24:26.304 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-02-09T17:24:26.304 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_mclock_override_recovery_settings": "false", 2024-02-09T17:24:26.304 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_mclock_profile": "balanced", 2024-02-09T17:24:26.304 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-02-09T17:24:26.304 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-02-09T17:24:26.304 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-02-09T17:24:26.304 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-02-09T17:24:26.304 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-02-09T17:24:26.304 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-02-09T17:24:26.304 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-02-09T17:24:26.305 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-02-09T17:24:26.305 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-02-09T17:24:26.305 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-02-09T17:24:26.305 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_mclock_skip_benchmark": "false", 2024-02-09T17:24:26.305 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_memory_base": "805306368", 2024-02-09T17:24:26.305 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_memory_cache_min": "134217728", 2024-02-09T17:24:26.305 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-02-09T17:24:26.305 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-02-09T17:24:26.305 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_memory_target": "4294967296", 2024-02-09T17:24:26.305 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_memory_target_autotune": "false", 2024-02-09T17:24:26.305 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-02-09T17:24:26.305 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_min_pg_log_entries": "250", 2024-02-09T17:24:26.305 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_min_recovery_priority": "0", 2024-02-09T17:24:26.305 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_mon_heartbeat_interval": "30", 2024-02-09T17:24:26.305 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-02-09T17:24:26.306 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_mon_report_interval": "5", 2024-02-09T17:24:26.306 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_mon_report_max_in_flight": "2", 2024-02-09T17:24:26.306 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-02-09T17:24:26.306 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_num_cache_shards": "32", 2024-02-09T17:24:26.306 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_num_op_tracker_shard": "32", 2024-02-09T17:24:26.306 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_numa_auto_affinity": "true", 2024-02-09T17:24:26.306 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_numa_node": "-1", 2024-02-09T17:24:26.306 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_numa_prefer_iface": "true", 2024-02-09T17:24:26.306 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-02-09T17:24:26.306 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_objecter_finishers": "1", 2024-02-09T17:24:26.306 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_objectstore": "bluestore", 2024-02-09T17:24:26.306 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_objectstore_fuse": "false", 2024-02-09T17:24:26.306 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_objectstore_tracing": "false", 2024-02-09T17:24:26.306 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_op_complaint_time": "30.000000", 2024-02-09T17:24:26.306 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_op_history_duration": "600", 2024-02-09T17:24:26.306 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_op_history_size": "20", 2024-02-09T17:24:26.307 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_op_history_slow_op_size": "20", 2024-02-09T17:24:26.307 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-02-09T17:24:26.307 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_op_log_threshold": "5", 2024-02-09T17:24:26.307 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_op_num_shards": "0", 2024-02-09T17:24:26.307 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_op_num_shards_hdd": "5", 2024-02-09T17:24:26.307 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_op_num_shards_ssd": "8", 2024-02-09T17:24:26.307 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_op_num_threads_per_shard": "0", 2024-02-09T17:24:26.307 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-02-09T17:24:26.307 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-02-09T17:24:26.307 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-02-09T17:24:26.307 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_op_pq_min_cost": "65536", 2024-02-09T17:24:26.307 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_op_queue": "mclock_scheduler", 2024-02-09T17:24:26.307 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_op_queue_cut_off": "high", 2024-02-09T17:24:26.307 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_op_thread_suicide_timeout": "150", 2024-02-09T17:24:26.307 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_op_thread_timeout": "15", 2024-02-09T17:24:26.308 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_open_classes_on_start": "true", 2024-02-09T17:24:26.308 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_os_flags": "0", 2024-02-09T17:24:26.308 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_peering_op_priority": "255", 2024-02-09T17:24:26.308 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_pg_delete_cost": "1048576", 2024-02-09T17:24:26.308 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_pg_delete_priority": "5", 2024-02-09T17:24:26.308 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-02-09T17:24:26.308 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-02-09T17:24:26.308 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_pg_log_dups_tracked": "3000", 2024-02-09T17:24:26.308 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_pg_log_trim_max": "10000", 2024-02-09T17:24:26.308 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_pg_log_trim_min": "100", 2024-02-09T17:24:26.308 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-02-09T17:24:26.308 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_pg_object_context_cache_count": "64", 2024-02-09T17:24:26.308 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-02-09T17:24:26.309 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_pg_stat_report_interval_max_seconds": "500", 2024-02-09T17:24:26.309 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-02-09T17:24:26.309 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-02-09T17:24:26.309 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-02-09T17:24:26.309 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-02-09T17:24:26.309 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-02-09T17:24:26.309 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-02-09T17:24:26.309 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_pool_default_crimson": "false", 2024-02-09T17:24:26.309 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_pool_default_crush_rule": "-1", 2024-02-09T17:24:26.309 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_pool_default_ec_fast_read": "false", 2024-02-09T17:24:26.309 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-02-09T17:24:26.309 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_pool_default_flag_bulk": "false", 2024-02-09T17:24:26.309 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-02-09T17:24:26.309 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_pool_default_flag_nodelete": "false", 2024-02-09T17:24:26.309 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-02-09T17:24:26.310 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-02-09T17:24:26.310 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_pool_default_flags": "0", 2024-02-09T17:24:26.310 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-02-09T17:24:26.310 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_pool_default_min_size": "2", 2024-02-09T17:24:26.310 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-02-09T17:24:26.310 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_pool_default_pg_num": "32", 2024-02-09T17:24:26.310 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_pool_default_pgp_num": "0", 2024-02-09T17:24:26.310 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-02-09T17:24:26.310 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_pool_default_read_ratio": "70", 2024-02-09T17:24:26.310 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_pool_default_size": "3", 2024-02-09T17:24:26.310 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_pool_default_type": "replicated", 2024-02-09T17:24:26.310 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-02-09T17:24:26.310 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_pool_use_gmt_hitset": "true", 2024-02-09T17:24:26.310 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_push_per_object_cost": "1000", 2024-02-09T17:24:26.310 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_read_ec_check_for_errors": "false", 2024-02-09T17:24:26.311 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_recover_clone_overlap": "true", 2024-02-09T17:24:26.311 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_recover_clone_overlap_limit": "10", 2024-02-09T17:24:26.311 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_recovery_cost": "20971520", 2024-02-09T17:24:26.311 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_recovery_delay_start": "0.000000", 2024-02-09T17:24:26.311 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_recovery_max_active": "0", 2024-02-09T17:24:26.311 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_recovery_max_active_hdd": "3", 2024-02-09T17:24:26.311 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_recovery_max_active_ssd": "10", 2024-02-09T17:24:26.311 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_recovery_max_chunk": "8388608", 2024-02-09T17:24:26.311 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-02-09T17:24:26.311 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_recovery_max_single_start": "1", 2024-02-09T17:24:26.311 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_recovery_op_priority": "3", 2024-02-09T17:24:26.311 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_recovery_op_warn_multiple": "16", 2024-02-09T17:24:26.311 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_recovery_priority": "5", 2024-02-09T17:24:26.311 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_recovery_retry_interval": "30.000000", 2024-02-09T17:24:26.311 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_recovery_sleep": "0.000000", 2024-02-09T17:24:26.312 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-02-09T17:24:26.312 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-02-09T17:24:26.312 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-02-09T17:24:26.312 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_repair_during_recovery": "false", 2024-02-09T17:24:26.312 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_requested_scrub_priority": "120", 2024-02-09T17:24:26.312 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-02-09T17:24:26.312 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_rollback_to_cluster_snap": "", 2024-02-09T17:24:26.312 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_scrub_auto_repair": "false", 2024-02-09T17:24:26.312 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-02-09T17:24:26.312 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-02-09T17:24:26.312 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_scrub_begin_hour": "0", 2024-02-09T17:24:26.312 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_scrub_begin_week_day": "0", 2024-02-09T17:24:26.312 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_scrub_chunk_max": "25", 2024-02-09T17:24:26.312 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_scrub_chunk_min": "5", 2024-02-09T17:24:26.313 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_scrub_cost": "52428800", 2024-02-09T17:24:26.313 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-02-09T17:24:26.313 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_scrub_during_recovery": "false", 2024-02-09T17:24:26.313 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_scrub_end_hour": "0", 2024-02-09T17:24:26.313 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_scrub_end_week_day": "0", 2024-02-09T17:24:26.313 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_scrub_event_cost": "4096", 2024-02-09T17:24:26.313 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-02-09T17:24:26.313 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-02-09T17:24:26.313 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_scrub_invalid_stats": "true", 2024-02-09T17:24:26.313 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_scrub_load_threshold": "0.500000", 2024-02-09T17:24:26.313 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_scrub_max_interval": "604800.000000", 2024-02-09T17:24:26.313 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_scrub_max_preemptions": "5", 2024-02-09T17:24:26.313 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_scrub_min_interval": "86400.000000", 2024-02-09T17:24:26.313 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_scrub_priority": "5", 2024-02-09T17:24:26.314 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_scrub_reservation_timeout": "300000", 2024-02-09T17:24:26.314 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_scrub_sleep": "0.000000", 2024-02-09T17:24:26.314 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_scrub_slow_reservation_response": "30000", 2024-02-09T17:24:26.314 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-02-09T17:24:26.314 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-02-09T17:24:26.314 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_shutdown_pgref_assert": "false", 2024-02-09T17:24:26.314 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_skip_data_digest": "false", 2024-02-09T17:24:26.314 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_smart_report_timeout": "5", 2024-02-09T17:24:26.314 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_snap_trim_cost": "1048576", 2024-02-09T17:24:26.314 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_snap_trim_priority": "5", 2024-02-09T17:24:26.314 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_snap_trim_sleep": "0.000000", 2024-02-09T17:24:26.314 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-02-09T17:24:26.314 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-02-09T17:24:26.314 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-02-09T17:24:26.314 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-02-09T17:24:26.315 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_stats_update_period_scrubbing": "15", 2024-02-09T17:24:26.315 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-02-09T17:24:26.315 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_target_transaction_size": "30", 2024-02-09T17:24:26.315 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-02-09T17:24:26.315 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-02-09T17:24:26.315 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-02-09T17:24:26.315 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-02-09T17:24:26.315 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-02-09T17:24:26.315 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-02-09T17:24:26.315 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-02-09T17:24:26.315 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_tier_default_cache_mode": "writeback", 2024-02-09T17:24:26.315 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-02-09T17:24:26.315 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-02-09T17:24:26.315 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_tracing": "false", 2024-02-09T17:24:26.316 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_use_stale_snap": "false", 2024-02-09T17:24:26.316 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-02-09T17:24:26.316 INFO:teuthology.orchestra.run.smithi099.stdout: "osdc_blkin_trace_all": "false", 2024-02-09T17:24:26.316 INFO:teuthology.orchestra.run.smithi099.stdout: "paxos_kill_at": "0", 2024-02-09T17:24:26.316 INFO:teuthology.orchestra.run.smithi099.stdout: "paxos_max_join_drift": "10", 2024-02-09T17:24:26.316 INFO:teuthology.orchestra.run.smithi099.stdout: "paxos_min": "500", 2024-02-09T17:24:26.316 INFO:teuthology.orchestra.run.smithi099.stdout: "paxos_min_wait": "0.050000", 2024-02-09T17:24:26.316 INFO:teuthology.orchestra.run.smithi099.stdout: "paxos_propose_interval": "1.000000", 2024-02-09T17:24:26.316 INFO:teuthology.orchestra.run.smithi099.stdout: "paxos_service_trim_max": "500", 2024-02-09T17:24:26.316 INFO:teuthology.orchestra.run.smithi099.stdout: "paxos_service_trim_max_multiplier": "20", 2024-02-09T17:24:26.316 INFO:teuthology.orchestra.run.smithi099.stdout: "paxos_service_trim_min": "10", 2024-02-09T17:24:26.316 INFO:teuthology.orchestra.run.smithi099.stdout: "paxos_stash_full_interval": "25", 2024-02-09T17:24:26.316 INFO:teuthology.orchestra.run.smithi099.stdout: "paxos_trim_max": "500", 2024-02-09T17:24:26.317 INFO:teuthology.orchestra.run.smithi099.stdout: "paxos_trim_min": "250", 2024-02-09T17:24:26.317 INFO:teuthology.orchestra.run.smithi099.stdout: "perf": "true", 2024-02-09T17:24:26.317 INFO:teuthology.orchestra.run.smithi099.stdout: "pid_file": "", 2024-02-09T17:24:26.317 INFO:teuthology.orchestra.run.smithi099.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-02-09T17:24:26.317 INFO:teuthology.orchestra.run.smithi099.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-02-09T17:24:26.317 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "-", 2024-02-09T17:24:26.317 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrv": "", 2024-02-09T17:24:26.317 INFO:teuthology.orchestra.run.smithi099.stdout: "public_bind_addr": "-", 2024-02-09T17:24:26.317 INFO:teuthology.orchestra.run.smithi099.stdout: "public_network": "", 2024-02-09T17:24:26.317 INFO:teuthology.orchestra.run.smithi099.stdout: "public_network_interface": "", 2024-02-09T17:24:26.317 INFO:teuthology.orchestra.run.smithi099.stdout: "qat_compressor_enabled": "false", 2024-02-09T17:24:26.317 INFO:teuthology.orchestra.run.smithi099.stdout: "qat_compressor_session_max_number": "256", 2024-02-09T17:24:26.317 INFO:teuthology.orchestra.run.smithi099.stdout: "rados_mon_op_timeout": "0", 2024-02-09T17:24:26.317 INFO:teuthology.orchestra.run.smithi099.stdout: "rados_osd_op_timeout": "0", 2024-02-09T17:24:26.317 INFO:teuthology.orchestra.run.smithi099.stdout: "rados_tracing": "false", 2024-02-09T17:24:26.318 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_atime_update_interval": "60", 2024-02-09T17:24:26.318 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-02-09T17:24:26.318 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_balance_parent_reads": "false", 2024-02-09T17:24:26.318 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_balance_snap_reads": "false", 2024-02-09T17:24:26.318 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_blkin_trace_all": "false", 2024-02-09T17:24:26.318 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_blocklist_expire_seconds": "0", 2024-02-09T17:24:26.318 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_blocklist_on_break_lock": "true", 2024-02-09T17:24:26.318 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_cache": "true", 2024-02-09T17:24:26.318 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_cache_block_writes_upfront": "false", 2024-02-09T17:24:26.318 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_cache_max_dirty": "25165824", 2024-02-09T17:24:26.318 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-02-09T17:24:26.318 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_cache_max_dirty_object": "0", 2024-02-09T17:24:26.318 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_cache_policy": "writearound", 2024-02-09T17:24:26.318 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_cache_size": "33554432", 2024-02-09T17:24:26.318 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_cache_target_dirty": "16777216", 2024-02-09T17:24:26.319 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-02-09T17:24:26.319 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_clone_copy_on_read": "false", 2024-02-09T17:24:26.319 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_compression_hint": "none", 2024-02-09T17:24:26.319 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_concurrent_management_ops": "10", 2024-02-09T17:24:26.319 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-02-09T17:24:26.319 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_default_clone_format": "auto", 2024-02-09T17:24:26.319 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_default_data_pool": "", 2024-02-09T17:24:26.319 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_default_features": "61", 2024-02-09T17:24:26.319 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_default_format": "2", 2024-02-09T17:24:26.319 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_default_map_options": "", 2024-02-09T17:24:26.319 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_default_order": "22", 2024-02-09T17:24:26.319 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_default_pool": "rbd", 2024-02-09T17:24:26.319 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-02-09T17:24:26.319 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_default_stripe_count": "0", 2024-02-09T17:24:26.319 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_default_stripe_unit": "0", 2024-02-09T17:24:26.320 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_disable_zero_copy_writes": "true", 2024-02-09T17:24:26.320 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_discard_granularity_bytes": "65536", 2024-02-09T17:24:26.320 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-02-09T17:24:26.320 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_enable_alloc_hint": "true", 2024-02-09T17:24:26.320 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-02-09T17:24:26.320 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_io_scheduler": "simple", 2024-02-09T17:24:26.320 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-02-09T17:24:26.320 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_journal_commit_age": "5.000000", 2024-02-09T17:24:26.320 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-02-09T17:24:26.320 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-02-09T17:24:26.320 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-02-09T17:24:26.320 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-02-09T17:24:26.320 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_journal_object_flush_interval": "0", 2024-02-09T17:24:26.320 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-02-09T17:24:26.321 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-02-09T17:24:26.321 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_journal_order": "24", 2024-02-09T17:24:26.321 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_journal_pool": "", 2024-02-09T17:24:26.321 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_journal_splay_width": "4", 2024-02-09T17:24:26.321 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_localize_parent_reads": "false", 2024-02-09T17:24:26.321 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_localize_snap_reads": "false", 2024-02-09T17:24:26.321 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-02-09T17:24:26.321 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-02-09T17:24:26.321 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-02-09T17:24:26.321 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-02-09T17:24:26.321 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-02-09T17:24:26.321 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-02-09T17:24:26.321 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_mirror_image_policy_type": "simple", 2024-02-09T17:24:26.321 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-02-09T17:24:26.321 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-02-09T17:24:26.322 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-02-09T17:24:26.322 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-02-09T17:24:26.322 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-02-09T17:24:26.322 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-02-09T17:24:26.322 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-02-09T17:24:26.322 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_mirror_memory_autotune": "true", 2024-02-09T17:24:26.322 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_mirror_memory_base": "805306368", 2024-02-09T17:24:26.322 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-02-09T17:24:26.322 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-02-09T17:24:26.322 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-02-09T17:24:26.322 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-02-09T17:24:26.322 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_mirror_memory_target": "4294967296", 2024-02-09T17:24:26.322 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-02-09T17:24:26.322 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-02-09T17:24:26.322 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-02-09T17:24:26.322 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_mirroring_delete_delay": "0", 2024-02-09T17:24:26.323 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-02-09T17:24:26.323 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_mirroring_replay_delay": "0", 2024-02-09T17:24:26.323 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-02-09T17:24:26.323 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-02-09T17:24:26.323 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_move_to_trash_on_remove": "false", 2024-02-09T17:24:26.323 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-02-09T17:24:26.323 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_mtime_update_interval": "60", 2024-02-09T17:24:26.323 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_non_blocking_aio": "true", 2024-02-09T17:24:26.323 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_op_thread_timeout": "60", 2024-02-09T17:24:26.323 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_op_threads": "1", 2024-02-09T17:24:26.323 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_parent_cache_enabled": "false", 2024-02-09T17:24:26.323 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_persistent_cache_mode": "disabled", 2024-02-09T17:24:26.323 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_persistent_cache_path": "/tmp", 2024-02-09T17:24:26.323 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_persistent_cache_size": "1073741824", 2024-02-09T17:24:26.324 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_plugins": "", 2024-02-09T17:24:26.324 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_qos_bps_burst": "0", 2024-02-09T17:24:26.324 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-02-09T17:24:26.324 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_qos_bps_limit": "0", 2024-02-09T17:24:26.324 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_qos_exclude_ops": "0", 2024-02-09T17:24:26.324 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_qos_iops_burst": "0", 2024-02-09T17:24:26.324 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-02-09T17:24:26.324 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_qos_iops_limit": "0", 2024-02-09T17:24:26.324 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_qos_read_bps_burst": "0", 2024-02-09T17:24:26.324 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-02-09T17:24:26.324 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_qos_read_bps_limit": "0", 2024-02-09T17:24:26.324 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_qos_read_iops_burst": "0", 2024-02-09T17:24:26.324 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-02-09T17:24:26.324 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_qos_read_iops_limit": "0", 2024-02-09T17:24:26.325 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_qos_schedule_tick_min": "50", 2024-02-09T17:24:26.325 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_qos_write_bps_burst": "0", 2024-02-09T17:24:26.325 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-02-09T17:24:26.325 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_qos_write_bps_limit": "0", 2024-02-09T17:24:26.325 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_qos_write_iops_burst": "0", 2024-02-09T17:24:26.325 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-02-09T17:24:26.325 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_qos_write_iops_limit": "0", 2024-02-09T17:24:26.325 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_quiesce_notification_attempts": "10", 2024-02-09T17:24:26.325 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_read_from_replica_policy": "default", 2024-02-09T17:24:26.325 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-02-09T17:24:26.325 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_readahead_max_bytes": "524288", 2024-02-09T17:24:26.325 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_readahead_trigger_requests": "10", 2024-02-09T17:24:26.325 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_request_timed_out_seconds": "30", 2024-02-09T17:24:26.325 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_skip_partial_discard": "true", 2024-02-09T17:24:26.325 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-02-09T17:24:26.326 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_tracing": "false", 2024-02-09T17:24:26.326 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_validate_names": "true", 2024-02-09T17:24:26.326 INFO:teuthology.orchestra.run.smithi099.stdout: "rbd_validate_pool": "true", 2024-02-09T17:24:26.326 INFO:teuthology.orchestra.run.smithi099.stdout: "restapi_base_url": "", 2024-02-09T17:24:26.326 INFO:teuthology.orchestra.run.smithi099.stdout: "restapi_log_level": "", 2024-02-09T17:24:26.326 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_acl_grants_max_num": "100", 2024-02-09T17:24:26.326 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_admin_entry": "admin", 2024-02-09T17:24:26.326 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-02-09T17:24:26.326 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_backend_store": "rados", 2024-02-09T17:24:26.326 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_barbican_url": "", 2024-02-09T17:24:26.326 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_beast_enable_async": "true", 2024-02-09T17:24:26.326 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_bucket_counters_cache": "false", 2024-02-09T17:24:26.326 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-02-09T17:24:26.326 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-02-09T17:24:26.327 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-02-09T17:24:26.327 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_bucket_index_max_aio": "128", 2024-02-09T17:24:26.327 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-02-09T17:24:26.327 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-02-09T17:24:26.327 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_bucket_quota_ttl": "600", 2024-02-09T17:24:26.327 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-02-09T17:24:26.327 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_cache_enabled": "true", 2024-02-09T17:24:26.327 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_cache_expiry_interval": "900", 2024-02-09T17:24:26.327 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_cache_lru_size": "10000", 2024-02-09T17:24:26.327 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_config_store": "rados", 2024-02-09T17:24:26.327 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_content_length_compat": "false", 2024-02-09T17:24:26.327 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_copy_obj_progress": "true", 2024-02-09T17:24:26.327 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-02-09T17:24:26.327 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_cors_rules_max_num": "100", 2024-02-09T17:24:26.327 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_cross_domain_policy": "", 2024-02-09T17:24:26.328 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_crypt_default_encryption_key": "", 2024-02-09T17:24:26.328 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_crypt_kmip_addr": "", 2024-02-09T17:24:26.328 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_crypt_kmip_ca_path": "", 2024-02-09T17:24:26.328 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_crypt_kmip_client_cert": "", 2024-02-09T17:24:26.328 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_crypt_kmip_client_key": "", 2024-02-09T17:24:26.328 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-02-09T17:24:26.328 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_crypt_kmip_password": "", 2024-02-09T17:24:26.328 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-02-09T17:24:26.328 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_crypt_kmip_username": "", 2024-02-09T17:24:26.328 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_crypt_require_ssl": "true", 2024-02-09T17:24:26.328 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-02-09T17:24:26.328 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-02-09T17:24:26.328 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-02-09T17:24:26.329 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-02-09T17:24:26.329 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-02-09T17:24:26.329 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-02-09T17:24:26.329 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-02-09T17:24:26.329 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-02-09T17:24:26.329 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-02-09T17:24:26.329 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-02-09T17:24:26.329 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-02-09T17:24:26.329 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-02-09T17:24:26.329 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-02-09T17:24:26.329 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-02-09T17:24:26.329 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_crypt_suppress_logs": "true", 2024-02-09T17:24:26.329 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_crypt_vault_addr": "", 2024-02-09T17:24:26.329 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_crypt_vault_auth": "token", 2024-02-09T17:24:26.329 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_crypt_vault_namespace": "", 2024-02-09T17:24:26.330 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_crypt_vault_prefix": "", 2024-02-09T17:24:26.330 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-02-09T17:24:26.330 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-02-09T17:24:26.330 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-02-09T17:24:26.330 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-02-09T17:24:26.330 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_crypt_vault_token_file": "", 2024-02-09T17:24:26.330 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-02-09T17:24:26.330 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_curl_buffersize": "524288", 2024-02-09T17:24:26.330 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_curl_low_speed_limit": "1024", 2024-02-09T17:24:26.330 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_curl_low_speed_time": "300", 2024-02-09T17:24:26.330 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_curl_tcp_keepalive": "0", 2024-02-09T17:24:26.330 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-02-09T17:24:26.330 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-02-09T17:24:26.330 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-02-09T17:24:26.330 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-02-09T17:24:26.331 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-02-09T17:24:26.331 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_d3n_l1_fadvise": "4", 2024-02-09T17:24:26.331 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-02-09T17:24:26.331 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-02-09T17:24:26.331 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-02-09T17:24:26.331 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_d4n_host": "127.0.0.1", 2024-02-09T17:24:26.331 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_d4n_port": "6379", 2024-02-09T17:24:26.331 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-02-09T17:24:26.331 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_data_log_changes_size": "1000", 2024-02-09T17:24:26.331 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_data_log_num_shards": "128", 2024-02-09T17:24:26.331 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-02-09T17:24:26.331 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_data_log_window": "30", 2024-02-09T17:24:26.331 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_data_notify_interval_msec": "0", 2024-02-09T17:24:26.331 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_data_sync_poll_interval": "20", 2024-02-09T17:24:26.332 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_data_sync_spawn_window": "20", 2024-02-09T17:24:26.332 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-02-09T17:24:26.332 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-02-09T17:24:26.332 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-02-09T17:24:26.332 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_default_data_log_backing": "fifo", 2024-02-09T17:24:26.332 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-02-09T17:24:26.332 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_default_region_info_oid": "default.region", 2024-02-09T17:24:26.332 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-02-09T17:24:26.332 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-02-09T17:24:26.332 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_defer_to_bucket_acls": "", 2024-02-09T17:24:26.332 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-02-09T17:24:26.332 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-02-09T17:24:26.332 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-02-09T17:24:26.332 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-02-09T17:24:26.333 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-02-09T17:24:26.333 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-02-09T17:24:26.333 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-02-09T17:24:26.333 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-02-09T17:24:26.333 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_dmclock_data_res": "500.000000", 2024-02-09T17:24:26.333 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-02-09T17:24:26.333 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-02-09T17:24:26.333 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-02-09T17:24:26.333 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-02-09T17:24:26.333 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_dns_name": "", 2024-02-09T17:24:26.333 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_dns_s3website_name": "", 2024-02-09T17:24:26.333 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_dynamic_resharding": "true", 2024-02-09T17:24:26.333 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-02-09T17:24:26.333 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_enable_gc_threads": "true", 2024-02-09T17:24:26.333 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_enable_lc_threads": "true", 2024-02-09T17:24:26.334 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_enable_ops_log": "false", 2024-02-09T17:24:26.334 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_enable_quota_threads": "true", 2024-02-09T17:24:26.334 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_enable_static_website": "false", 2024-02-09T17:24:26.334 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_enable_usage_log": "false", 2024-02-09T17:24:26.334 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_enforce_swift_acls": "true", 2024-02-09T17:24:26.334 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_exit_timeout_secs": "120", 2024-02-09T17:24:26.334 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_expose_bucket": "false", 2024-02-09T17:24:26.334 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_extended_http_attrs": "", 2024-02-09T17:24:26.334 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_filter": "none", 2024-02-09T17:24:26.334 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-02-09T17:24:26.334 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_frontends": "beast port=7480", 2024-02-09T17:24:26.334 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_gc_max_concurrent_io": "10", 2024-02-09T17:24:26.334 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_gc_max_deferred": "50", 2024-02-09T17:24:26.334 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-02-09T17:24:26.334 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_gc_max_objs": "32", 2024-02-09T17:24:26.335 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_gc_max_queue_size": "134213632", 2024-02-09T17:24:26.335 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_gc_max_trim_chunk": "16", 2024-02-09T17:24:26.335 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_gc_obj_min_wait": "7200", 2024-02-09T17:24:26.335 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_gc_processor_max_time": "3600", 2024-02-09T17:24:26.335 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_gc_processor_period": "3600", 2024-02-09T17:24:26.335 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-02-09T17:24:26.335 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_get_obj_window_size": "16777216", 2024-02-09T17:24:26.335 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_healthcheck_disabling_path": "", 2024-02-09T17:24:26.335 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_ignore_get_invalid_range": "false", 2024-02-09T17:24:26.335 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_init_timeout": "300", 2024-02-09T17:24:26.335 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-02-09T17:24:26.335 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-02-09T17:24:26.335 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_kafka_connection_idle": "30", 2024-02-09T17:24:26.335 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_kafka_sleep_timeout": "10", 2024-02-09T17:24:26.335 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-02-09T17:24:26.336 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-02-09T17:24:26.336 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-02-09T17:24:26.336 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_keystone_admin_domain": "", 2024-02-09T17:24:26.336 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_keystone_admin_password": "", 2024-02-09T17:24:26.336 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_keystone_admin_password_path": "", 2024-02-09T17:24:26.336 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_keystone_admin_project": "", 2024-02-09T17:24:26.336 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_keystone_admin_tenant": "", 2024-02-09T17:24:26.336 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_keystone_admin_token": "", 2024-02-09T17:24:26.336 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_keystone_admin_token_path": "", 2024-02-09T17:24:26.336 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_keystone_admin_user": "", 2024-02-09T17:24:26.336 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_keystone_api_version": "2", 2024-02-09T17:24:26.336 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_keystone_barbican_domain": "", 2024-02-09T17:24:26.336 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_keystone_barbican_password": "", 2024-02-09T17:24:26.336 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_keystone_barbican_project": "", 2024-02-09T17:24:26.337 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_keystone_barbican_tenant": "", 2024-02-09T17:24:26.337 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_keystone_barbican_user": "", 2024-02-09T17:24:26.337 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-02-09T17:24:26.337 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_keystone_implicit_tenants": "false", 2024-02-09T17:24:26.337 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-02-09T17:24:26.337 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_keystone_service_token_enabled": "false", 2024-02-09T17:24:26.337 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_keystone_token_cache_size": "10000", 2024-02-09T17:24:26.337 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_keystone_url": "", 2024-02-09T17:24:26.337 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_keystone_verify_ssl": "true", 2024-02-09T17:24:26.337 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_lc_debug_interval": "-1", 2024-02-09T17:24:26.337 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_lc_lock_max_time": "90", 2024-02-09T17:24:26.337 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_lc_max_objs": "32", 2024-02-09T17:24:26.337 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_lc_max_rules": "1000", 2024-02-09T17:24:26.337 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_lc_max_worker": "3", 2024-02-09T17:24:26.337 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_lc_max_wp_worker": "3", 2024-02-09T17:24:26.338 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_lc_thread_delay": "0", 2024-02-09T17:24:26.338 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-02-09T17:24:26.338 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_ldap_dnattr": "uid", 2024-02-09T17:24:26.338 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-02-09T17:24:26.338 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_ldap_searchfilter": "", 2024-02-09T17:24:26.338 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-02-09T17:24:26.338 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_ldap_uri": "ldaps://", 2024-02-09T17:24:26.338 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-02-09T17:24:26.338 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-02-09T17:24:26.338 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-02-09T17:24:26.338 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_log_http_headers": "", 2024-02-09T17:24:26.338 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_log_nonexistent_bucket": "false", 2024-02-09T17:24:26.338 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-02-09T17:24:26.338 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_log_object_name_utc": "false", 2024-02-09T17:24:26.339 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-02-09T17:24:26.339 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-02-09T17:24:26.339 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_max_attr_name_len": "0", 2024-02-09T17:24:26.339 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_max_attr_size": "0", 2024-02-09T17:24:26.339 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_max_attrs_num_in_req": "0", 2024-02-09T17:24:26.339 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_max_chunk_size": "4194304", 2024-02-09T17:24:26.339 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_max_concurrent_requests": "1024", 2024-02-09T17:24:26.339 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-02-09T17:24:26.339 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_max_dynamic_shards": "1999", 2024-02-09T17:24:26.339 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_max_listing_results": "1000", 2024-02-09T17:24:26.339 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_max_notify_retries": "10", 2024-02-09T17:24:26.339 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_max_objs_per_shard": "100000", 2024-02-09T17:24:26.339 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_max_put_param_size": "1048576", 2024-02-09T17:24:26.339 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_max_put_size": "5368709120", 2024-02-09T17:24:26.339 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_max_slo_entries": "1000", 2024-02-09T17:24:26.340 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_md_log_max_shards": "64", 2024-02-09T17:24:26.340 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_md_notify_interval_msec": "200", 2024-02-09T17:24:26.340 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_meta_sync_poll_interval": "20", 2024-02-09T17:24:26.340 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_meta_sync_spawn_window": "20", 2024-02-09T17:24:26.340 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-02-09T17:24:26.340 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_mp_lock_max_time": "600", 2024-02-09T17:24:26.340 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-02-09T17:24:26.340 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_multipart_min_part_size": "5242880", 2024-02-09T17:24:26.340 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-02-09T17:24:26.340 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-02-09T17:24:26.340 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_nfs_fhcache_size": "2017", 2024-02-09T17:24:26.340 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-02-09T17:24:26.340 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-02-09T17:24:26.341 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_nfs_lru_lanes": "5", 2024-02-09T17:24:26.341 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_nfs_max_gc": "300", 2024-02-09T17:24:26.341 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-02-09T17:24:26.341 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_nfs_run_gc_threads": "false", 2024-02-09T17:24:26.341 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_nfs_run_lc_threads": "false", 2024-02-09T17:24:26.341 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_nfs_run_quota_threads": "false", 2024-02-09T17:24:26.341 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_nfs_run_sync_thread": "false", 2024-02-09T17:24:26.341 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-02-09T17:24:26.341 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-02-09T17:24:26.341 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_num_async_rados_threads": "32", 2024-02-09T17:24:26.341 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_num_control_oids": "8", 2024-02-09T17:24:26.341 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_numa_node": "-1", 2024-02-09T17:24:26.341 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_obj_stripe_size": "4194304", 2024-02-09T17:24:26.341 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-02-09T17:24:26.341 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_objexp_chunk_size": "100", 2024-02-09T17:24:26.342 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_objexp_gc_interval": "600", 2024-02-09T17:24:26.342 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_objexp_hints_num_shards": "127", 2024-02-09T17:24:26.342 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-02-09T17:24:26.342 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-02-09T17:24:26.342 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_op_thread_timeout": "600", 2024-02-09T17:24:26.342 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_op_tracing": "false", 2024-02-09T17:24:26.342 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_opa_token": "", 2024-02-09T17:24:26.342 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_opa_url": "", 2024-02-09T17:24:26.342 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_opa_verify_ssl": "true", 2024-02-09T17:24:26.342 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-02-09T17:24:26.342 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-02-09T17:24:26.342 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_ops_log_rados": "false", 2024-02-09T17:24:26.342 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_ops_log_socket_path": "", 2024-02-09T17:24:26.342 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-02-09T17:24:26.342 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-02-09T17:24:26.343 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-02-09T17:24:26.343 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_period_push_interval": "2.000000", 2024-02-09T17:24:26.343 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_period_push_interval_max": "30.000000", 2024-02-09T17:24:26.343 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_period_root_pool": ".rgw.root", 2024-02-09T17:24:26.343 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-02-09T17:24:26.343 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-02-09T17:24:26.343 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_posix_cache_lanes": "3", 2024-02-09T17:24:26.343 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-02-09T17:24:26.343 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_posix_cache_max_buckets": "100", 2024-02-09T17:24:26.343 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_posix_cache_partitions": "3", 2024-02-09T17:24:26.343 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-02-09T17:24:26.343 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_print_continue": "true", 2024-02-09T17:24:26.343 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_print_prohibited_content_length": "false", 2024-02-09T17:24:26.343 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-02-09T17:24:26.344 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-02-09T17:24:26.344 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-02-09T17:24:26.344 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-02-09T17:24:26.344 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_rados_tracing": "false", 2024-02-09T17:24:26.344 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_realm": "", 2024-02-09T17:24:26.344 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_realm_id": "", 2024-02-09T17:24:26.344 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-02-09T17:24:26.344 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_region": "", 2024-02-09T17:24:26.344 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_region_root_pool": ".rgw.root", 2024-02-09T17:24:26.344 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_relaxed_region_enforcement": "false", 2024-02-09T17:24:26.344 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-02-09T17:24:26.344 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-02-09T17:24:26.344 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_request_uri": "", 2024-02-09T17:24:26.344 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_reshard_batch_size": "64", 2024-02-09T17:24:26.345 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-02-09T17:24:26.345 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_reshard_max_aio": "128", 2024-02-09T17:24:26.345 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_reshard_num_logs": "16", 2024-02-09T17:24:26.345 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_reshard_thread_interval": "600", 2024-02-09T17:24:26.345 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_resolve_cname": "false", 2024-02-09T17:24:26.345 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_rest_getusage_op_compat": "false", 2024-02-09T17:24:26.345 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_run_sync_thread": "true", 2024-02-09T17:24:26.345 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-02-09T17:24:26.345 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_s3_auth_use_keystone": "false", 2024-02-09T17:24:26.345 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_s3_auth_use_ldap": "false", 2024-02-09T17:24:26.345 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_s3_auth_use_rados": "true", 2024-02-09T17:24:26.345 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_s3_auth_use_sts": "false", 2024-02-09T17:24:26.345 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-02-09T17:24:26.345 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_s3_success_create_obj_status": "0", 2024-02-09T17:24:26.345 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-02-09T17:24:26.346 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_scheduler_type": "throttler", 2024-02-09T17:24:26.346 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_script_uri": "", 2024-02-09T17:24:26.346 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_service_provider_name": "", 2024-02-09T17:24:26.346 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-02-09T17:24:26.346 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_sts_client_id": "", 2024-02-09T17:24:26.346 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_sts_client_secret": "", 2024-02-09T17:24:26.346 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_sts_entry": "sts", 2024-02-09T17:24:26.346 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_sts_key": "sts", 2024-02-09T17:24:26.346 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_sts_max_session_duration": "43200", 2024-02-09T17:24:26.346 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_sts_min_session_duration": "900", 2024-02-09T17:24:26.346 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_sts_token_introspection_url": "", 2024-02-09T17:24:26.346 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_swift_account_in_url": "false", 2024-02-09T17:24:26.346 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_swift_auth_entry": "auth", 2024-02-09T17:24:26.346 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_swift_auth_url": "", 2024-02-09T17:24:26.347 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_swift_custom_header": "", 2024-02-09T17:24:26.347 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_swift_enforce_content_length": "false", 2024-02-09T17:24:26.347 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_swift_need_stats": "true", 2024-02-09T17:24:26.347 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_swift_tenant_name": "", 2024-02-09T17:24:26.347 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_swift_token_expiration": "86400", 2024-02-09T17:24:26.347 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_swift_url": "", 2024-02-09T17:24:26.347 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_swift_url_prefix": "swift", 2024-02-09T17:24:26.347 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_swift_versioning_enabled": "false", 2024-02-09T17:24:26.347 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-02-09T17:24:26.347 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-02-09T17:24:26.347 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_sync_lease_period": "120", 2024-02-09T17:24:26.347 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-02-09T17:24:26.347 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_sync_log_trim_interval": "1200", 2024-02-09T17:24:26.347 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-02-09T17:24:26.348 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-02-09T17:24:26.348 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-02-09T17:24:26.348 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_sync_obj_etag_verify": "false", 2024-02-09T17:24:26.348 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_sync_trace_history_size": "4096", 2024-02-09T17:24:26.348 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-02-09T17:24:26.348 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-02-09T17:24:26.348 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_thread_pool_size": "512", 2024-02-09T17:24:26.348 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_topic_persistency_max_retries": "0", 2024-02-09T17:24:26.348 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-02-09T17:24:26.348 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-02-09T17:24:26.348 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_topic_require_publish_policy": "false", 2024-02-09T17:24:26.348 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_torrent_comment": "", 2024-02-09T17:24:26.348 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_torrent_createby": "", 2024-02-09T17:24:26.348 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_torrent_encoding": "", 2024-02-09T17:24:26.349 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_torrent_flag": "false", 2024-02-09T17:24:26.349 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_torrent_max_size": "5368709120", 2024-02-09T17:24:26.349 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_torrent_origin": "", 2024-02-09T17:24:26.349 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_torrent_sha_unit": "524288", 2024-02-09T17:24:26.349 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_torrent_tracker": "", 2024-02-09T17:24:26.349 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_trust_forwarded_https": "false", 2024-02-09T17:24:26.349 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-02-09T17:24:26.349 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_usage_log_tick_interval": "30", 2024-02-09T17:24:26.349 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_usage_max_shards": "32", 2024-02-09T17:24:26.349 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_usage_max_user_shards": "1", 2024-02-09T17:24:26.349 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_use_opa_authz": "false", 2024-02-09T17:24:26.349 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_user_counters_cache": "false", 2024-02-09T17:24:26.349 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_user_counters_cache_size": "10000", 2024-02-09T17:24:26.349 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-02-09T17:24:26.350 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_user_default_quota_max_size": "-1", 2024-02-09T17:24:26.350 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_user_max_buckets": "1000", 2024-02-09T17:24:26.350 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_user_policies_max_num": "100", 2024-02-09T17:24:26.350 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-02-09T17:24:26.350 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-02-09T17:24:26.350 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_user_quota_sync_interval": "86400", 2024-02-09T17:24:26.350 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-02-09T17:24:26.350 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_user_unique_email": "true", 2024-02-09T17:24:26.350 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_verify_ssl": "true", 2024-02-09T17:24:26.350 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_website_routing_rules_max_num": "50", 2024-02-09T17:24:26.350 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_zone": "", 2024-02-09T17:24:26.350 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_zone_id": "", 2024-02-09T17:24:26.350 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-02-09T17:24:26.350 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_zonegroup": "", 2024-02-09T17:24:26.350 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_zonegroup_id": "", 2024-02-09T17:24:26.351 INFO:teuthology.orchestra.run.smithi099.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-02-09T17:24:26.351 INFO:teuthology.orchestra.run.smithi099.stdout: "rgwlc_auto_session_clear": "true", 2024-02-09T17:24:26.351 INFO:teuthology.orchestra.run.smithi099.stdout: "rgwlc_skip_bucket_step": "false", 2024-02-09T17:24:26.351 INFO:teuthology.orchestra.run.smithi099.stdout: "rocksdb_block_size": "4096", 2024-02-09T17:24:26.351 INFO:teuthology.orchestra.run.smithi099.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-02-09T17:24:26.351 INFO:teuthology.orchestra.run.smithi099.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-02-09T17:24:26.351 INFO:teuthology.orchestra.run.smithi099.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-02-09T17:24:26.351 INFO:teuthology.orchestra.run.smithi099.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-02-09T17:24:26.351 INFO:teuthology.orchestra.run.smithi099.stdout: "rocksdb_cache_shard_bits": "4", 2024-02-09T17:24:26.351 INFO:teuthology.orchestra.run.smithi099.stdout: "rocksdb_cache_size": "536870912", 2024-02-09T17:24:26.351 INFO:teuthology.orchestra.run.smithi099.stdout: "rocksdb_cache_type": "binned_lru", 2024-02-09T17:24:26.351 INFO:teuthology.orchestra.run.smithi099.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-02-09T17:24:26.351 INFO:teuthology.orchestra.run.smithi099.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-02-09T17:24:26.351 INFO:teuthology.orchestra.run.smithi099.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-02-09T17:24:26.351 INFO:teuthology.orchestra.run.smithi099.stdout: "rocksdb_collect_compaction_stats": "false", 2024-02-09T17:24:26.352 INFO:teuthology.orchestra.run.smithi099.stdout: "rocksdb_collect_extended_stats": "false", 2024-02-09T17:24:26.352 INFO:teuthology.orchestra.run.smithi099.stdout: "rocksdb_collect_memory_stats": "false", 2024-02-09T17:24:26.352 INFO:teuthology.orchestra.run.smithi099.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-02-09T17:24:26.352 INFO:teuthology.orchestra.run.smithi099.stdout: "rocksdb_index_type": "binary_search", 2024-02-09T17:24:26.352 INFO:teuthology.orchestra.run.smithi099.stdout: "rocksdb_log_to_ceph_log": "true", 2024-02-09T17:24:26.352 INFO:teuthology.orchestra.run.smithi099.stdout: "rocksdb_metadata_block_size": "4096", 2024-02-09T17:24:26.352 INFO:teuthology.orchestra.run.smithi099.stdout: "rocksdb_partition_filters": "false", 2024-02-09T17:24:26.352 INFO:teuthology.orchestra.run.smithi099.stdout: "rocksdb_perf": "false", 2024-02-09T17:24:26.352 INFO:teuthology.orchestra.run.smithi099.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-02-09T17:24:26.352 INFO:teuthology.orchestra.run.smithi099.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-02-09T17:24:26.352 INFO:teuthology.orchestra.run.smithi099.stdout: "rotating_keys_renewal_timeout": "10", 2024-02-09T17:24:26.352 INFO:teuthology.orchestra.run.smithi099.stdout: "run_dir": "/var/run/ceph", 2024-02-09T17:24:26.352 INFO:teuthology.orchestra.run.smithi099.stdout: "seastore_block_create": "true", 2024-02-09T17:24:26.352 INFO:teuthology.orchestra.run.smithi099.stdout: "seastore_cache_lru_size": "67108864", 2024-02-09T17:24:26.353 INFO:teuthology.orchestra.run.smithi099.stdout: "seastore_cbjournal_size": "5368709120", 2024-02-09T17:24:26.353 INFO:teuthology.orchestra.run.smithi099.stdout: "seastore_data_delta_based_overwrite": "0", 2024-02-09T17:24:26.353 INFO:teuthology.orchestra.run.smithi099.stdout: "seastore_default_max_object_size": "16777216", 2024-02-09T17:24:26.353 INFO:teuthology.orchestra.run.smithi099.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-02-09T17:24:26.353 INFO:teuthology.orchestra.run.smithi099.stdout: "seastore_device_size": "53687091200", 2024-02-09T17:24:26.353 INFO:teuthology.orchestra.run.smithi099.stdout: "seastore_journal_batch_capacity": "16", 2024-02-09T17:24:26.353 INFO:teuthology.orchestra.run.smithi099.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-02-09T17:24:26.353 INFO:teuthology.orchestra.run.smithi099.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-02-09T17:24:26.353 INFO:teuthology.orchestra.run.smithi099.stdout: "seastore_journal_iodepth_limit": "5", 2024-02-09T17:24:26.353 INFO:teuthology.orchestra.run.smithi099.stdout: "seastore_main_device_type": "SSD", 2024-02-09T17:24:26.353 INFO:teuthology.orchestra.run.smithi099.stdout: "seastore_max_concurrent_transactions": "8", 2024-02-09T17:24:26.353 INFO:teuthology.orchestra.run.smithi099.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-02-09T17:24:26.353 INFO:teuthology.orchestra.run.smithi099.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-02-09T17:24:26.353 INFO:teuthology.orchestra.run.smithi099.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-02-09T17:24:26.353 INFO:teuthology.orchestra.run.smithi099.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-02-09T17:24:26.354 INFO:teuthology.orchestra.run.smithi099.stdout: "seastore_segment_size": "67108864", 2024-02-09T17:24:26.354 INFO:teuthology.orchestra.run.smithi099.stdout: "set_keepcaps": "false", 2024-02-09T17:24:26.354 INFO:teuthology.orchestra.run.smithi099.stdout: "setgroup": "ceph", 2024-02-09T17:24:26.354 INFO:teuthology.orchestra.run.smithi099.stdout: "setuser": "ceph", 2024-02-09T17:24:26.354 INFO:teuthology.orchestra.run.smithi099.stdout: "setuser_match_path": "", 2024-02-09T17:24:26.354 INFO:teuthology.orchestra.run.smithi099.stdout: "target_max_misplaced_ratio": "0.050000", 2024-02-09T17:24:26.354 INFO:teuthology.orchestra.run.smithi099.stdout: "thp": "false", 2024-02-09T17:24:26.354 INFO:teuthology.orchestra.run.smithi099.stdout: "threadpool_default_timeout": "60", 2024-02-09T17:24:26.354 INFO:teuthology.orchestra.run.smithi099.stdout: "threadpool_empty_queue_max_wait": "2", 2024-02-09T17:24:26.354 INFO:teuthology.orchestra.run.smithi099.stdout: "throttler_perf_counter": "true" 2024-02-09T17:24:26.354 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:24:26.913 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-02-09T17:24:27.176 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:24:27.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:27 smithi179 ceph-mon[43975]: pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:27.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:27 smithi162 ceph-mon[44304]: pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:27.698 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:27 smithi099 ceph-mon[39420]: pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:28.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:28 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:24:28.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:28 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:24:28.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:28 smithi099 ceph-mon[39420]: osdmap e109: 12 total, 12 up, 12 in 2024-02-09T17:24:28.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:28 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:24:28.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:28 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:24:28.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:28 smithi179 ceph-mon[43975]: osdmap e109: 12 total, 12 up, 12 in 2024-02-09T17:24:28.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:28 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:24:28.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:28 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:24:28.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:28 smithi162 ceph-mon[44304]: osdmap e109: 12 total, 12 up, 12 in 2024-02-09T17:24:29.047 INFO:teuthology.orchestra.run.smithi099.stdout:mon.a: {} 2024-02-09T17:24:29.058 INFO:teuthology.orchestra.run.smithi099.stdout:mon.c: {} 2024-02-09T17:24:29.069 INFO:teuthology.orchestra.run.smithi099.stdout:mon.b: {} 2024-02-09T17:24:29.070 INFO:teuthology.orchestra.run.smithi099.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-02-09T17:24:29.070 INFO:teuthology.orchestra.run.smithi099.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-02-09T17:24:29.070 INFO:teuthology.orchestra.run.smithi099.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-02-09T17:24:29.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:29 smithi179 ceph-mon[43975]: pgmap v341: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:29.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:29 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/708386013' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-09T17:24:29.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:29 smithi162 ceph-mon[44304]: pgmap v341: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:29.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:29 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/708386013' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-09T17:24:29.742 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:29 smithi099 ceph-mon[39420]: pgmap v341: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:29.742 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:29 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/708386013' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-09T17:24:29.770 INFO:teuthology.run_tasks:Running task full_sequential... 2024-02-09T17:24:29.809 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-02-09T17:24:29.811 INFO:tasks.radosbench:Beginning radosbench... 2024-02-09T17:24:29.811 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-02-09T17:24:29.811 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd pool create unique_pool_0 16 2024-02-09T17:24:29.812 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-02-09T17:24:29.812 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-02-09T17:24:29.812 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:24:29.812 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-02-09T17:24:29.812 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph tell osd.0 'trim stale osdmaps' 2024-02-09T17:24:29.813 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-02-09T17:24:29.813 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:24:29.813 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-02-09T17:24:29.813 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:24:29.814 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-02-09T17:24:29.814 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-09T17:24:29.814 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-02-09T17:24:29.814 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set noscrub 2024-02-09T17:24:29.850 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:29.871 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:24:29.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:29.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:24:29.976 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:24:30.022 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:30.045 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:24:30.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:30.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:24:30.148 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:24:30.149 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:24:30.156 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:24:30.191 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:24:30.192 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:30.454 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:24:30.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:30.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:24:30.558 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:24:30.586 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:31.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:31 smithi179 ceph-mon[43975]: pgmap v342: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:31.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:31 smithi162 ceph-mon[44304]: pgmap v342: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:31.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:31 smithi099 ceph-mon[39420]: pgmap v342: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:32.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:32 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:24:32.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:32 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:24:32.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:32 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:24:32.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:32 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:24:32.698 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:32 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:24:32.698 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:32 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:24:33.522 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:33 smithi099 ceph-mon[39420]: pgmap v343: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:33.522 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:33 smithi099 ceph-mon[39420]: osdmap e110: 12 total, 12 up, 12 in 2024-02-09T17:24:33.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:33 smithi179 ceph-mon[43975]: pgmap v343: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:33.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:33 smithi179 ceph-mon[43975]: osdmap e110: 12 total, 12 up, 12 in 2024-02-09T17:24:33.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:33 smithi162 ceph-mon[44304]: pgmap v343: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:33.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:33 smithi162 ceph-mon[44304]: osdmap e110: 12 total, 12 up, 12 in 2024-02-09T17:24:34.027 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:24:34.027 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [], 2024-02-09T17:24:34.027 INFO:teuthology.orchestra.run.smithi099.stdout: "num_ops": 0 2024-02-09T17:24:34.027 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:24:34.051 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:24:34.051 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:24:34.266 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:24:34.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:34.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:24:34.371 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:24:34.407 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:34.422 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'unique_pool_0' created 2024-02-09T17:24:34.431 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:24:34.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:34.434 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:24:34.535 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:24:34.564 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:34.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:34 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/524904886' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-02-09T17:24:34.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:34 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/524904886' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-02-09T17:24:34.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:34 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/524904886' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-02-09T17:24:35.284 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-09T17:24:35.313 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:24:35.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:35.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:24:35.417 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:24:35.449 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:35.469 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:24:35.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:35.472 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:24:35.510 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:35 smithi099 ceph-mon[39420]: pgmap v345: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:35.510 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:35 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/524904886' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-02-09T17:24:35.510 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:35 smithi099 ceph-mon[39420]: osdmap e111: 12 total, 12 up, 12 in 2024-02-09T17:24:35.573 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:24:35.600 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:35.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:35 smithi179 ceph-mon[43975]: pgmap v345: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:35.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:35 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/524904886' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-02-09T17:24:35.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:35 smithi179 ceph-mon[43975]: osdmap e111: 12 total, 12 up, 12 in 2024-02-09T17:24:35.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:35 smithi162 ceph-mon[44304]: pgmap v345: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:35.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:35 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/524904886' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-02-09T17:24:35.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:35 smithi162 ceph-mon[44304]: osdmap e111: 12 total, 12 up, 12 in 2024-02-09T17:24:36.730 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:36 smithi099 ceph-mon[39420]: osdmap e112: 12 total, 12 up, 12 in 2024-02-09T17:24:36.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:36 smithi179 ceph-mon[43975]: osdmap e112: 12 total, 12 up, 12 in 2024-02-09T17:24:36.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:36 smithi162 ceph-mon[44304]: osdmap e112: 12 total, 12 up, 12 in 2024-02-09T17:24:37.151 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:24:37.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:37.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:24:37.256 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:24:37.289 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:37.310 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:24:37.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:37.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:24:37.414 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:24:37.446 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:37.466 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:24:37.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:37.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:24:37.572 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:24:37.605 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:37.627 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:24:37.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:37.630 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:24:37.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:37 smithi162 ceph-mon[44304]: pgmap v348: 17 pgs: 9 creating+peering, 2 active+clean, 6 unknown; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:37.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:37 smithi162 ceph-mon[44304]: osdmap e113: 12 total, 12 up, 12 in 2024-02-09T17:24:37.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:37 smithi162 ceph-mon[44304]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-09T17:24:37.730 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:24:37.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:37 smithi099 ceph-mon[39420]: pgmap v348: 17 pgs: 9 creating+peering, 2 active+clean, 6 unknown; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:37.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:37 smithi099 ceph-mon[39420]: osdmap e113: 12 total, 12 up, 12 in 2024-02-09T17:24:37.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:37 smithi099 ceph-mon[39420]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-09T17:24:37.760 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:37.780 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:24:37.782 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:37.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:24:37.884 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:24:37.916 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:37.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:37 smithi179 ceph-mon[43975]: pgmap v348: 17 pgs: 9 creating+peering, 2 active+clean, 6 unknown; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:37.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:37 smithi179 ceph-mon[43975]: osdmap e113: 12 total, 12 up, 12 in 2024-02-09T17:24:37.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:37 smithi179 ceph-mon[43975]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-09T17:24:37.942 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:24:37.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:37.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:24:38.046 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:24:38.077 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:38.097 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:24:38.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:38.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:24:38.201 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:24:38.230 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:38.249 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:24:38.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:38.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:24:38.293 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:24:38.313 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:24:38.332 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:24:38.352 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:24:38.352 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:24:38.372 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:24:38.379 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:38.391 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:24:38.399 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:24:38.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:38.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:24:38.411 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:24:38.430 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:24:38.451 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:24:38.471 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:24:38.491 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:24:38.501 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:24:38.513 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:24:38.514 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:24:38.514 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:24:38.514 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:24:38.514 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:24:38.514 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:24:38.514 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:24:38.514 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:24:38.515 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:24:38.515 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:24:38.515 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:24:38.515 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:24:38.515 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:24:38.527 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:38.546 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:24:38.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:38.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:24:38.641 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-02-09T17:24:38.649 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:24:38.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:38 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:24:38.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:38 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:24:38.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:38 smithi162 ceph-mon[44304]: osdmap e114: 12 total, 12 up, 12 in 2024-02-09T17:24:38.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:38 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2799183754' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:24:38.681 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:38.702 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:24:38.703 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:38 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:24:38.703 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:38 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:24:38.703 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:38 smithi179 ceph-mon[43975]: osdmap e114: 12 total, 12 up, 12 in 2024-02-09T17:24:38.704 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:38 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2799183754' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:24:38.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:38.704 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:24:38.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:38 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:24:38.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:38 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:24:38.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:38 smithi099 ceph-mon[39420]: osdmap e114: 12 total, 12 up, 12 in 2024-02-09T17:24:38.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:38 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2799183754' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:24:38.805 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:24:38.835 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:38.855 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:24:38.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:38.858 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:24:38.959 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:24:38.988 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:39.008 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:24:39.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:39.011 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:24:39.111 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:24:39.138 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:39.157 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:24:39.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:39.160 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:24:39.262 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:24:39.292 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:39.314 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:24:39.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:39.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:24:39.323 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:24:39.421 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:24:39.448 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:39.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:39 smithi099 ceph-mon[39420]: pgmap v351: 17 pgs: 9 creating+peering, 4 active+clean, 4 unknown; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:39.796 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:24:39.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:39.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:24:39.802 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:24:39.903 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:24:39.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:39 smithi179 ceph-mon[43975]: pgmap v351: 17 pgs: 9 creating+peering, 4 active+clean, 4 unknown; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:39.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:39 smithi162 ceph-mon[44304]: pgmap v351: 17 pgs: 9 creating+peering, 4 active+clean, 4 unknown; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:39.930 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:40.388 INFO:teuthology.orchestra.run.smithi099.stderr:no valid command found; 10 closest matches: 2024-02-09T17:24:40.388 INFO:teuthology.orchestra.run.smithi099.stderr:0 2024-02-09T17:24:40.388 INFO:teuthology.orchestra.run.smithi099.stderr:1 2024-02-09T17:24:40.388 INFO:teuthology.orchestra.run.smithi099.stderr:2 2024-02-09T17:24:40.388 INFO:teuthology.orchestra.run.smithi099.stderr:abort 2024-02-09T17:24:40.388 INFO:teuthology.orchestra.run.smithi099.stderr:assert 2024-02-09T17:24:40.388 INFO:teuthology.orchestra.run.smithi099.stderr:bench [] [] [] [] 2024-02-09T17:24:40.388 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs debug_inject_read_zeros 2024-02-09T17:24:40.388 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs files list 2024-02-09T17:24:40.389 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs stats 2024-02-09T17:24:40.389 INFO:teuthology.orchestra.run.smithi099.stderr:bluestore allocator dump block 2024-02-09T17:24:40.389 INFO:teuthology.orchestra.run.smithi099.stderr:Error EINVAL: invalid command 2024-02-09T17:24:40.455 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is set 2024-02-09T17:24:40.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:40 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3028627316' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:24:40.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:40 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3028627316' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:24:40.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:40 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3028627316' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:24:41.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:41 smithi099 ceph-mon[39420]: pgmap v352: 17 pgs: 1 creating+peering, 16 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:41.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:41 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:24:41.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:41 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:24:41.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:41 smithi099 ceph-mon[39420]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:24:41.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:41 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3028627316' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:24:41.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:41 smithi099 ceph-mon[39420]: osdmap e115: 12 total, 12 up, 12 in 2024-02-09T17:24:41.833 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:24:41.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:41 smithi179 ceph-mon[43975]: pgmap v352: 17 pgs: 1 creating+peering, 16 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:41.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:41 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:24:41.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:41 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:24:41.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:41 smithi179 ceph-mon[43975]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:24:41.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:41 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3028627316' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:24:41.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:41 smithi179 ceph-mon[43975]: osdmap e115: 12 total, 12 up, 12 in 2024-02-09T17:24:41.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:41 smithi162 ceph-mon[44304]: pgmap v352: 17 pgs: 1 creating+peering, 16 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:41.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:41 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:24:41.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:41 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:24:41.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:41 smithi162 ceph-mon[44304]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:24:41.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:41 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3028627316' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:24:41.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:41 smithi162 ceph-mon[44304]: osdmap e115: 12 total, 12 up, 12 in 2024-02-09T17:24:42.953 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-09T17:24:42.954 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-02-09T17:24:42.954 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph tell osd.1 'trim stale osdmaps' 2024-02-09T17:24:43.181 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:24:43.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:43.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:24:43.286 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:24:43.318 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:43.339 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:24:43.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:43.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:24:43.443 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:24:43.475 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:43.497 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:24:43.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:43.500 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:24:43.602 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:24:43.632 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:43 smithi162 ceph-mon[44304]: pgmap v354: 17 pgs: 17 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:43.634 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:43.656 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:24:43.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:43.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:24:43.722 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:43 smithi099 ceph-mon[39420]: pgmap v354: 17 pgs: 17 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:43.760 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:24:43.790 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:43.810 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:24:43.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:43.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:24:43.915 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:24:43.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:43 smithi179 ceph-mon[43975]: pgmap v354: 17 pgs: 17 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:43.947 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:43.968 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:24:43.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:43.972 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:24:44.072 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:24:44.103 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:44.123 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:24:44.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:44.126 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:24:44.227 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:24:44.258 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:44.285 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:24:44.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:44.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:24:44.296 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:24:44.296 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:24:44.324 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:24:44.326 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:24:44.390 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:24:44.419 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:44.565 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:24:44.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:44.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:24:44.668 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:24:44.701 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:44.723 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:24:44.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:44.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:24:44.827 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:24:44.859 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:44.879 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:24:44.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:44.883 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:24:44.983 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:24:45.012 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:45.032 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:24:45.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:45.036 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:24:45.137 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:24:45.165 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:45.413 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:24:45.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:45.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:24:45.435 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-09T17:24:45.458 INFO:teuthology.orchestra.run.smithi099.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-02-09T17:24:45.519 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:24:45.551 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:45.572 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:24:45.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:45.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:24:45.676 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:24:45.705 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:45.724 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:24:45.726 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:45 smithi162 ceph-mon[44304]: pgmap v355: 17 pgs: 17 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:45.727 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:45 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/512058104' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-09T17:24:45.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:45.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:24:45.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:45 smithi099 ceph-mon[39420]: pgmap v355: 17 pgs: 17 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:45.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:45 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/512058104' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-09T17:24:45.829 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:24:45.858 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:45.876 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:24:45.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:45.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:24:45.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:45 smithi179 ceph-mon[43975]: pgmap v355: 17 pgs: 17 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:45.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:45 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/512058104' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-09T17:24:45.980 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:24:46.007 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:46.025 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:24:46.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:46.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:24:46.062 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-09T17:24:46.129 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:24:46.161 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:46.183 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:24:46.186 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:46.186 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:24:46.287 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:24:46.314 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:46.333 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:24:46.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:46.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:24:46.437 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:24:46.464 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:46.482 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:24:46.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:46.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:24:46.557 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:46 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/512058104' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-09T17:24:46.557 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:46 smithi099 ceph-mon[39420]: osdmap e116: 12 total, 12 up, 12 in 2024-02-09T17:24:46.586 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:24:46.610 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:46 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/512058104' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-09T17:24:46.610 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:46 smithi162 ceph-mon[44304]: osdmap e116: 12 total, 12 up, 12 in 2024-02-09T17:24:46.612 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:46.630 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:24:46.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:46.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:24:46.734 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:24:46.764 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:46 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/512058104' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-09T17:24:46.764 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:46 smithi179 ceph-mon[43975]: osdmap e116: 12 total, 12 up, 12 in 2024-02-09T17:24:46.766 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:46.787 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:24:46.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:46.790 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:24:46.891 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:24:46.918 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:46.937 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:24:46.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:46.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:24:47.041 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:24:47.067 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:47.085 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:24:47.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:47.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:24:47.189 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:24:47.215 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:47.233 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:24:47.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:47.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:24:47.337 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:24:47.369 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:47.391 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:24:47.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:47.394 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:24:47.496 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:24:47.524 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:47.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:47 smithi179 ceph-mon[43975]: pgmap v357: 17 pgs: 17 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:47.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:47 smithi179 ceph-mon[43975]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-09T17:24:47.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:47 smithi099 ceph-mon[39420]: pgmap v357: 17 pgs: 17 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:47.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:47 smithi099 ceph-mon[39420]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-09T17:24:47.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:47 smithi162 ceph-mon[44304]: pgmap v357: 17 pgs: 17 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:47.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:47 smithi162 ceph-mon[44304]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-09T17:24:48.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:48 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:24:48.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:48 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:24:48.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:48 smithi099 ceph-mon[39420]: osdmap e117: 12 total, 12 up, 12 in 2024-02-09T17:24:48.573 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:24:48.576 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:24:48.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:48.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:24:48.579 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:24:48.608 INFO:teuthology.orchestra.run.smithi099.stderr:no valid command found; 10 closest matches: 2024-02-09T17:24:48.608 INFO:teuthology.orchestra.run.smithi099.stderr:0 2024-02-09T17:24:48.608 INFO:teuthology.orchestra.run.smithi099.stderr:1 2024-02-09T17:24:48.608 INFO:teuthology.orchestra.run.smithi099.stderr:2 2024-02-09T17:24:48.609 INFO:teuthology.orchestra.run.smithi099.stderr:abort 2024-02-09T17:24:48.609 INFO:teuthology.orchestra.run.smithi099.stderr:assert 2024-02-09T17:24:48.609 INFO:teuthology.orchestra.run.smithi099.stderr:bench [] [] [] [] 2024-02-09T17:24:48.610 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs debug_inject_read_zeros 2024-02-09T17:24:48.610 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs files list 2024-02-09T17:24:48.610 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs stats 2024-02-09T17:24:48.610 INFO:teuthology.orchestra.run.smithi099.stderr:bluestore allocator dump block 2024-02-09T17:24:48.610 INFO:teuthology.orchestra.run.smithi099.stderr:Error EINVAL: invalid command 2024-02-09T17:24:48.679 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:24:48.711 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:48.732 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:24:48.734 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:48 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:24:48.734 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:48 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:24:48.734 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:48 smithi162 ceph-mon[44304]: osdmap e117: 12 total, 12 up, 12 in 2024-02-09T17:24:48.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:48.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:24:48.836 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:24:48.866 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:48.886 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:24:48.888 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:48.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:24:48.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:48 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:24:48.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:48 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:24:48.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:48 smithi179 ceph-mon[43975]: osdmap e117: 12 total, 12 up, 12 in 2024-02-09T17:24:48.989 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:24:49.021 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:49.042 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:24:49.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:49.045 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:24:49.146 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:24:49.179 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:49.278 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:24:49.278 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:24:49.278 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:24:49.278 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:24:49.287 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:24:49.307 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:24:49.327 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:24:49.347 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:24:49.368 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:24:49.375 DEBUG:teuthology.orchestra.run.smithi179:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-02-09T17:24:49.378 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-02-09T17:24:49.378 INFO:teuthology.orchestra.run:waiting for 3000 2024-02-09T17:24:49.390 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:24:49.410 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:24:49.440 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:hints = 1 2024-02-09T17:24:49.440 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-02-09T17:24:49.440 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Object prefix: benchmark_data_smithi179_77138 2024-02-09T17:24:49.440 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-09T17:24:49.441 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 0 0 0 0 0 0 - 0 2024-02-09T17:24:49.631 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:24:49.633 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:24:49.633 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:24:49.633 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:24:49.633 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:24:49.633 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:24:49.633 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:24:49.633 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:24:49.633 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:24:49.633 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:24:49.633 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:24:49.634 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:24:49.634 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:24:49.654 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:49 smithi099 ceph-mon[39420]: pgmap v359: 17 pgs: 17 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:49.655 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:49 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4085511085' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:24:49.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:49 smithi179 ceph-mon[43975]: pgmap v359: 17 pgs: 17 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:49.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:49 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4085511085' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:24:49.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:49 smithi162 ceph-mon[44304]: pgmap v359: 17 pgs: 17 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:24:49.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:49 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4085511085' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:24:50.440 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 1 16 2980 2964 185.24 185.25 0.00457296 0.00538191 2024-02-09T17:24:50.810 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:24:50.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:50.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:24:50.917 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:24:50.961 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:50.986 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:24:50.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:50.991 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:24:51.092 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:24:51.139 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:51.369 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:24:51.369 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 20, 2024-02-09T17:24:51.369 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 600, 2024-02-09T17:24:51.369 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [ 2024-02-09T17:24:51.369 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.369 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:435 2.5 e117/111 2:a3da9aec:::benchmark_data_smithi179_77138_object434:head v 117'21, mlcod=117'18)", 2024-02-09T17:24:51.369 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:24:49.899845+0000", 2024-02-09T17:24:51.369 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 1.453345157, 2024-02-09T17:24:51.369 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0082530030000000001, 2024-02-09T17:24:51.369 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:24:51.370 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:24:51.370 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:24:51.370 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.370 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:24:51.370 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:49.899845+0000", 2024-02-09T17:24:51.370 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:24:51.370 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.370 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.370 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:24:51.370 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:49.899845+0000", 2024-02-09T17:24:51.370 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:24:51.370 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.371 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.371 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:24:51.371 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:49.899846+0000", 2024-02-09T17:24:51.371 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.8599999999999998e-07 2024-02-09T17:24:51.371 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.371 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.371 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:24:51.371 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:49.899912+0000", 2024-02-09T17:24:51.371 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.5548000000000003e-05 2024-02-09T17:24:51.371 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.371 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.371 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:24:51.372 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:49.899922+0000", 2024-02-09T17:24:51.372 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0304e-05 2024-02-09T17:24:51.372 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.372 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.372 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:24:51.372 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:49.899935+0000", 2024-02-09T17:24:51.372 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2979999999999999e-05 2024-02-09T17:24:51.372 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.372 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.372 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:24:51.372 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:49.899990+0000", 2024-02-09T17:24:51.372 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.5438e-05 2024-02-09T17:24:51.372 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.373 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.373 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:24:51.373 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:49.900009+0000", 2024-02-09T17:24:51.373 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.9049999999999999e-05 2024-02-09T17:24:51.373 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.373 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.373 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:24:51.373 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:49.908053+0000", 2024-02-09T17:24:51.373 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0080438880000000008 2024-02-09T17:24:51.373 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.373 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.374 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:24:51.374 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:49.908098+0000", 2024-02-09T17:24:51.374 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.5209e-05 2024-02-09T17:24:51.374 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:24:51.374 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:24:51.374 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:24:51.374 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.374 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.374 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:442 2.5 e117/111 2:aabe466e:::benchmark_data_smithi179_77138_object441:head v 117'22, mlcod=117'19)", 2024-02-09T17:24:51.374 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:24:49.901045+0000", 2024-02-09T17:24:51.374 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 1.452145794, 2024-02-09T17:24:51.375 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0070839350000000004, 2024-02-09T17:24:51.375 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:24:51.375 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:24:51.375 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:24:51.375 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.375 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:24:51.375 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:49.901045+0000", 2024-02-09T17:24:51.375 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:24:51.375 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.375 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.375 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:24:51.375 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:49.901045+0000", 2024-02-09T17:24:51.375 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:24:51.376 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.376 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.376 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:24:51.376 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:49.901045+0000", 2024-02-09T17:24:51.376 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3900000000000005e-07 2024-02-09T17:24:51.376 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.376 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.376 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:24:51.376 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:49.901121+0000", 2024-02-09T17:24:51.376 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.5495999999999997e-05 2024-02-09T17:24:51.376 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.376 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.376 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:24:51.377 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:49.901130+0000", 2024-02-09T17:24:51.377 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.0350000000000007e-06 2024-02-09T17:24:51.377 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.377 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.377 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:24:51.377 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:49.901135+0000", 2024-02-09T17:24:51.377 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.9699999999999998e-06 2024-02-09T17:24:51.377 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.377 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.377 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:24:51.377 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:49.901171+0000", 2024-02-09T17:24:51.377 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.6569000000000002e-05 2024-02-09T17:24:51.378 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.378 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.378 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:24:51.378 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:49.901186+0000", 2024-02-09T17:24:51.378 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4171e-05 2024-02-09T17:24:51.378 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.378 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.378 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:24:51.378 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:49.908101+0000", 2024-02-09T17:24:51.378 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0069155889999999998 2024-02-09T17:24:51.378 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.378 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.379 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:24:51.379 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:49.908129+0000", 2024-02-09T17:24:51.379 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.7566e-05 2024-02-09T17:24:51.379 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:24:51.379 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:24:51.379 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:24:51.379 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.379 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.379 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:446 2.8 e117/111 2:1349afd0:::benchmark_data_smithi179_77138_object445:head v 117'29, mlcod=117'27)", 2024-02-09T17:24:51.379 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:24:49.901279+0000", 2024-02-09T17:24:51.379 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 1.45191179, 2024-02-09T17:24:51.379 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0066237839999999997, 2024-02-09T17:24:51.379 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:24:51.379 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:24:51.379 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:24:51.380 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.380 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:24:51.380 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:49.901279+0000", 2024-02-09T17:24:51.380 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:24:51.380 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.380 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.380 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:24:51.380 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:49.901279+0000", 2024-02-09T17:24:51.380 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:24:51.380 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.380 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.380 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:24:51.380 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:49.901279+0000", 2024-02-09T17:24:51.380 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.2699999999999999e-07 2024-02-09T17:24:51.380 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.381 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.381 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:24:51.381 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:49.901448+0000", 2024-02-09T17:24:51.381 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00016882 2024-02-09T17:24:51.381 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.381 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.381 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:24:51.381 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:49.901471+0000", 2024-02-09T17:24:51.381 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2402e-05 2024-02-09T17:24:51.381 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.383 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.383 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:24:51.383 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:49.901478+0000", 2024-02-09T17:24:51.383 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.8669999999999994e-06 2024-02-09T17:24:51.383 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.383 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.384 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:24:51.384 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:49.901526+0000", 2024-02-09T17:24:51.384 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.7886999999999997e-05 2024-02-09T17:24:51.384 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.384 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.384 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:24:51.384 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:49.901547+0000", 2024-02-09T17:24:51.384 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0630999999999999e-05 2024-02-09T17:24:51.384 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.384 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.384 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:24:51.384 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:49.907856+0000", 2024-02-09T17:24:51.384 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0063086089999999997 2024-02-09T17:24:51.384 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.385 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.385 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:24:51.385 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:49.907903+0000", 2024-02-09T17:24:51.385 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.6940999999999997e-05 2024-02-09T17:24:51.385 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:24:51.385 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:24:51.385 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:24:51.385 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.385 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.385 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:2576 2.c e117/111 2:3722d477:::benchmark_data_smithi179_77138_object2575:head v 117'175, mlcod=117'172)", 2024-02-09T17:24:51.385 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:24:50.347653+0000", 2024-02-09T17:24:51.385 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 1.0055372220000001, 2024-02-09T17:24:51.385 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0052031209999999998, 2024-02-09T17:24:51.385 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:24:51.386 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:24:51.386 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:24:51.386 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.386 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:24:51.386 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.347653+0000", 2024-02-09T17:24:51.386 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:24:51.386 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.386 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.386 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:24:51.386 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.347653+0000", 2024-02-09T17:24:51.386 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:24:51.386 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.387 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.387 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:24:51.387 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.347654+0000", 2024-02-09T17:24:51.387 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.5300000000000003e-07 2024-02-09T17:24:51.387 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.387 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.387 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:24:51.387 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.347683+0000", 2024-02-09T17:24:51.388 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.8619000000000002e-05 2024-02-09T17:24:51.388 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.388 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.388 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:24:51.388 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.347697+0000", 2024-02-09T17:24:51.388 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4449e-05 2024-02-09T17:24:51.388 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.388 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.389 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:24:51.389 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.347707+0000", 2024-02-09T17:24:51.389 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.2730000000000008e-06 2024-02-09T17:24:51.389 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.390 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.390 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:24:51.390 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.348149+0000", 2024-02-09T17:24:51.390 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00044242800000000003 2024-02-09T17:24:51.391 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.391 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.391 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:24:51.391 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.348168+0000", 2024-02-09T17:24:51.392 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.889e-05 2024-02-09T17:24:51.392 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.392 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.392 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:24:51.392 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.352831+0000", 2024-02-09T17:24:51.392 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0046631540000000001 2024-02-09T17:24:51.392 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.392 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.392 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:24:51.392 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.352856+0000", 2024-02-09T17:24:51.392 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.5454999999999999e-05 2024-02-09T17:24:51.393 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:24:51.393 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:24:51.393 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:24:51.393 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.393 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.393 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:2579 2.8 e117/111 2:1194e23a:::benchmark_data_smithi179_77138_object2578:head v 117'173, mlcod=117'171)", 2024-02-09T17:24:51.393 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:24:50.347691+0000", 2024-02-09T17:24:51.394 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 1.005499696, 2024-02-09T17:24:51.394 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0050394039999999999, 2024-02-09T17:24:51.394 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:24:51.394 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:24:51.394 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:24:51.394 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.394 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:24:51.394 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.347691+0000", 2024-02-09T17:24:51.394 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:24:51.395 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.395 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.395 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:24:51.395 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.347691+0000", 2024-02-09T17:24:51.395 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:24:51.396 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.397 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.397 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:24:51.397 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.347692+0000", 2024-02-09T17:24:51.397 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.2399999999999996e-07 2024-02-09T17:24:51.397 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.397 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.397 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:24:51.397 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.347730+0000", 2024-02-09T17:24:51.397 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.8389000000000001e-05 2024-02-09T17:24:51.397 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.397 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.398 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:24:51.398 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.347742+0000", 2024-02-09T17:24:51.398 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1911e-05 2024-02-09T17:24:51.398 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.398 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.398 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:24:51.398 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.347750+0000", 2024-02-09T17:24:51.398 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.5769999999999996e-06 2024-02-09T17:24:51.398 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.398 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.398 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:24:51.398 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.347824+0000", 2024-02-09T17:24:51.399 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.4395999999999997e-05 2024-02-09T17:24:51.399 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.399 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.399 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:24:51.399 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.347843+0000", 2024-02-09T17:24:51.399 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.8936999999999998e-05 2024-02-09T17:24:51.399 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.399 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.399 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:24:51.399 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.352699+0000", 2024-02-09T17:24:51.399 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0048560410000000002 2024-02-09T17:24:51.399 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.400 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.400 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:24:51.400 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.352730+0000", 2024-02-09T17:24:51.400 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.1229000000000002e-05 2024-02-09T17:24:51.400 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:24:51.400 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:24:51.400 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:24:51.400 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.400 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.400 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:2573 2.1 e117/111 2:8163679b:::benchmark_data_smithi179_77138_object2572:head v 117'180, mlcod=117'177)", 2024-02-09T17:24:51.400 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:24:50.347827+0000", 2024-02-09T17:24:51.400 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 1.0053634149999999, 2024-02-09T17:24:51.401 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0049349709999999998, 2024-02-09T17:24:51.401 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:24:51.401 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:24:51.401 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:24:51.401 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.401 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:24:51.401 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.347827+0000", 2024-02-09T17:24:51.401 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:24:51.401 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.401 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.401 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:24:51.401 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.347827+0000", 2024-02-09T17:24:51.401 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:24:51.402 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.402 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.402 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:24:51.402 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.347828+0000", 2024-02-09T17:24:51.402 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0160000000000001e-06 2024-02-09T17:24:51.402 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.402 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.402 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:24:51.402 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.347854+0000", 2024-02-09T17:24:51.402 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.5497e-05 2024-02-09T17:24:51.402 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.402 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.403 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:24:51.403 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.347864+0000", 2024-02-09T17:24:51.403 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0695e-05 2024-02-09T17:24:51.403 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.403 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.404 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:24:51.404 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.347872+0000", 2024-02-09T17:24:51.404 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.3209999999999999e-06 2024-02-09T17:24:51.404 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.404 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.404 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:24:51.404 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.347991+0000", 2024-02-09T17:24:51.404 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000119021 2024-02-09T17:24:51.404 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.405 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.405 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:24:51.405 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.348011+0000", 2024-02-09T17:24:51.405 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0139000000000001e-05 2024-02-09T17:24:51.405 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.405 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.405 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:24:51.405 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.352742+0000", 2024-02-09T17:24:51.405 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0047307759999999999 2024-02-09T17:24:51.405 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.405 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.405 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:24:51.406 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.352762+0000", 2024-02-09T17:24:51.406 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0506e-05 2024-02-09T17:24:51.406 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:24:51.406 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:24:51.406 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:24:51.406 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.406 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.406 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:2574 2.1 e117/111 2:8cc6901b:::benchmark_data_smithi179_77138_object2573:head v 117'181, mlcod=117'177)", 2024-02-09T17:24:51.406 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:24:50.347894+0000", 2024-02-09T17:24:51.406 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 1.005297042, 2024-02-09T17:24:51.406 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.004882763, 2024-02-09T17:24:51.406 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:24:51.407 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:24:51.407 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:24:51.407 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.407 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:24:51.407 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.347894+0000", 2024-02-09T17:24:51.407 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:24:51.407 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.407 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.407 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:24:51.408 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.347894+0000", 2024-02-09T17:24:51.408 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:24:51.408 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.408 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.408 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:24:51.408 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.347894+0000", 2024-02-09T17:24:51.409 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.4900000000000001e-07 2024-02-09T17:24:51.409 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.409 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.409 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:24:51.409 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.347915+0000", 2024-02-09T17:24:51.409 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.1565000000000001e-05 2024-02-09T17:24:51.409 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.409 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.409 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:24:51.409 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.347923+0000", 2024-02-09T17:24:51.409 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.5009999999999998e-06 2024-02-09T17:24:51.409 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.409 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.410 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:24:51.410 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.347928+0000", 2024-02-09T17:24:51.410 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.4349999999999996e-06 2024-02-09T17:24:51.410 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.410 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.410 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:24:51.410 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.348275+0000", 2024-02-09T17:24:51.410 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00034680799999999999 2024-02-09T17:24:51.410 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.410 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.410 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:24:51.410 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.348294+0000", 2024-02-09T17:24:51.410 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.8552000000000001e-05 2024-02-09T17:24:51.411 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.411 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.411 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:24:51.411 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.352764+0000", 2024-02-09T17:24:51.411 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0044699099999999997 2024-02-09T17:24:51.411 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.411 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.411 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:24:51.411 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.352776+0000", 2024-02-09T17:24:51.411 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2643000000000001e-05 2024-02-09T17:24:51.411 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:24:51.411 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:24:51.411 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:24:51.412 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.412 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.412 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:2931 2.c e117/111 2:3918683b:::benchmark_data_smithi179_77138_object2930:head v 117'204, mlcod=117'202)", 2024-02-09T17:24:51.412 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:24:50.424366+0000", 2024-02-09T17:24:51.412 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.92882497100000005, 2024-02-09T17:24:51.412 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0072091070000000002, 2024-02-09T17:24:51.412 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:24:51.412 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:24:51.412 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:24:51.412 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.412 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:24:51.412 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.424366+0000", 2024-02-09T17:24:51.412 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:24:51.413 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.413 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.413 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:24:51.413 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.424366+0000", 2024-02-09T17:24:51.413 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:24:51.413 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.413 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.413 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:24:51.413 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.424366+0000", 2024-02-09T17:24:51.413 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.13e-07 2024-02-09T17:24:51.413 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.413 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.413 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:24:51.413 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.424480+0000", 2024-02-09T17:24:51.414 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000114256 2024-02-09T17:24:51.414 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.414 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.414 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:24:51.414 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.424490+0000", 2024-02-09T17:24:51.414 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.9699999999999994e-06 2024-02-09T17:24:51.414 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.414 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.414 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:24:51.414 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.424496+0000", 2024-02-09T17:24:51.414 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1499999999999998e-06 2024-02-09T17:24:51.414 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.414 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.414 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:24:51.414 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.424549+0000", 2024-02-09T17:24:51.415 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2984e-05 2024-02-09T17:24:51.415 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.415 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.415 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:24:51.415 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.424569+0000", 2024-02-09T17:24:51.415 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0083999999999999e-05 2024-02-09T17:24:51.415 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.415 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.415 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:24:51.415 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.431527+0000", 2024-02-09T17:24:51.415 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0069585469999999998 2024-02-09T17:24:51.415 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.415 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.415 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:24:51.415 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.431575+0000", 2024-02-09T17:24:51.416 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.7503000000000002e-05 2024-02-09T17:24:51.416 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:24:51.416 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:24:51.416 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:24:51.416 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.416 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.416 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:2932 2.5 e117/111 2:a165a0ef:::benchmark_data_smithi179_77138_object2931:head v 117'167, mlcod=117'164)", 2024-02-09T17:24:51.416 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:24:50.424966+0000", 2024-02-09T17:24:51.416 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.92822417899999998, 2024-02-09T17:24:51.416 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0084189569999999995, 2024-02-09T17:24:51.416 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:24:51.416 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:24:51.416 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:24:51.416 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.417 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:24:51.417 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.424966+0000", 2024-02-09T17:24:51.417 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:24:51.417 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.417 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.417 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:24:51.417 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.424966+0000", 2024-02-09T17:24:51.417 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:24:51.417 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.417 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.417 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:24:51.417 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.424967+0000", 2024-02-09T17:24:51.417 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.7100000000000001e-07 2024-02-09T17:24:51.417 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.418 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.418 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:24:51.418 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.425102+0000", 2024-02-09T17:24:51.418 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000134571 2024-02-09T17:24:51.418 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.418 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.418 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:24:51.418 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.425114+0000", 2024-02-09T17:24:51.418 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1935e-05 2024-02-09T17:24:51.418 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.418 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.418 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:24:51.418 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.425121+0000", 2024-02-09T17:24:51.418 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.9419999999999998e-06 2024-02-09T17:24:51.419 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.419 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.419 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:24:51.419 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.425166+0000", 2024-02-09T17:24:51.419 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.5289e-05 2024-02-09T17:24:51.419 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.419 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.419 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:24:51.419 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.425182+0000", 2024-02-09T17:24:51.419 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.6572999999999999e-05 2024-02-09T17:24:51.419 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.419 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.419 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:24:51.419 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.433333+0000", 2024-02-09T17:24:51.419 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0081506870000000002 2024-02-09T17:24:51.419 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.420 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.420 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:24:51.420 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.433385+0000", 2024-02-09T17:24:51.420 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2289e-05 2024-02-09T17:24:51.420 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:24:51.420 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:24:51.420 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:24:51.420 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.420 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.420 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:2941 2.1 e117/111 2:8963f1a6:::benchmark_data_smithi179_77138_object2940:head v 117'210, mlcod=117'208)", 2024-02-09T17:24:51.420 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:24:50.426733+0000", 2024-02-09T17:24:51.420 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.92645766200000002, 2024-02-09T17:24:51.420 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0066502369999999998, 2024-02-09T17:24:51.421 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:24:51.421 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:24:51.421 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:24:51.421 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.421 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:24:51.421 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.426733+0000", 2024-02-09T17:24:51.421 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:24:51.421 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.421 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.421 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:24:51.421 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.426733+0000", 2024-02-09T17:24:51.421 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:24:51.421 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.421 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.421 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:24:51.422 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.426734+0000", 2024-02-09T17:24:51.422 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.8400000000000003e-07 2024-02-09T17:24:51.422 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.422 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.422 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:24:51.422 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.426752+0000", 2024-02-09T17:24:51.422 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.8014999999999999e-05 2024-02-09T17:24:51.422 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.422 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.422 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:24:51.422 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.426763+0000", 2024-02-09T17:24:51.422 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1191e-05 2024-02-09T17:24:51.422 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.422 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.423 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:24:51.423 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.426768+0000", 2024-02-09T17:24:51.423 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2830000000000001e-06 2024-02-09T17:24:51.423 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.423 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.423 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:24:51.423 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.426814+0000", 2024-02-09T17:24:51.423 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.5914000000000001e-05 2024-02-09T17:24:51.423 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.423 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.423 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:24:51.423 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.426834+0000", 2024-02-09T17:24:51.423 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.9899e-05 2024-02-09T17:24:51.423 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.424 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.424 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:24:51.424 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.433333+0000", 2024-02-09T17:24:51.424 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.006498467 2024-02-09T17:24:51.424 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.424 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.424 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:24:51.424 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.433383+0000", 2024-02-09T17:24:51.424 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.0584000000000002e-05 2024-02-09T17:24:51.424 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:24:51.424 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:24:51.424 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:24:51.424 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.424 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.425 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:3180 2.1 e117/111 2:894c70f6:::benchmark_data_smithi179_77138_object3179:head v 117'223, mlcod=117'220)", 2024-02-09T17:24:51.425 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:24:50.484059+0000", 2024-02-09T17:24:51.425 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.869131765, 2024-02-09T17:24:51.425 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0053958749999999996, 2024-02-09T17:24:51.425 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:24:51.426 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:24:51.426 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:24:51.426 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.426 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:24:51.426 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.484059+0000", 2024-02-09T17:24:51.426 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:24:51.426 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.426 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.426 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:24:51.427 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.484059+0000", 2024-02-09T17:24:51.427 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:24:51.427 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.427 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.427 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:24:51.427 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.484059+0000", 2024-02-09T17:24:51.427 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.9999999999999997e-07 2024-02-09T17:24:51.427 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.427 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.427 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:24:51.427 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.484086+0000", 2024-02-09T17:24:51.427 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.6659e-05 2024-02-09T17:24:51.427 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.427 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.428 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:24:51.428 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.484096+0000", 2024-02-09T17:24:51.428 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.7359999999999996e-06 2024-02-09T17:24:51.428 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.428 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.428 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:24:51.428 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.484103+0000", 2024-02-09T17:24:51.428 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.7619999999999998e-06 2024-02-09T17:24:51.428 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.428 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.428 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:24:51.429 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.485602+0000", 2024-02-09T17:24:51.429 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0014994979999999999 2024-02-09T17:24:51.429 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.429 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.429 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:24:51.429 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.485621+0000", 2024-02-09T17:24:51.429 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.9267999999999999e-05 2024-02-09T17:24:51.429 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.429 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.429 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:24:51.429 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.489408+0000", 2024-02-09T17:24:51.429 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0037864539999999999 2024-02-09T17:24:51.429 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.429 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.430 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:24:51.430 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.489455+0000", 2024-02-09T17:24:51.430 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.6798000000000002e-05 2024-02-09T17:24:51.430 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:24:51.430 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:24:51.430 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:24:51.430 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.430 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.430 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:3950 2.c e117/111 2:324d7025:::benchmark_data_smithi179_77138_object3949:head v 117'282, mlcod=117'280)", 2024-02-09T17:24:51.430 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:24:50.653286+0000", 2024-02-09T17:24:51.430 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.69990444699999999, 2024-02-09T17:24:51.430 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0048503640000000002, 2024-02-09T17:24:51.430 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:24:51.431 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:24:51.431 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:24:51.431 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.431 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:24:51.431 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.653286+0000", 2024-02-09T17:24:51.431 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:24:51.431 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.431 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.431 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:24:51.431 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.653286+0000", 2024-02-09T17:24:51.431 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:24:51.431 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.431 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.431 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:24:51.432 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.653287+0000", 2024-02-09T17:24:51.432 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.8599999999999997e-07 2024-02-09T17:24:51.432 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.432 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.432 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:24:51.432 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.653315+0000", 2024-02-09T17:24:51.432 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.8047000000000001e-05 2024-02-09T17:24:51.433 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.433 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.433 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:24:51.433 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.653325+0000", 2024-02-09T17:24:51.433 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.7960000000000007e-06 2024-02-09T17:24:51.433 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.433 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.433 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:24:51.433 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.653331+0000", 2024-02-09T17:24:51.433 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.3160000000000002e-06 2024-02-09T17:24:51.433 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.433 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.434 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:24:51.434 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.653398+0000", 2024-02-09T17:24:51.434 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.7016999999999999e-05 2024-02-09T17:24:51.434 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.434 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.434 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:24:51.434 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.653416+0000", 2024-02-09T17:24:51.434 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.7855000000000001e-05 2024-02-09T17:24:51.434 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.434 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.435 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:24:51.435 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.658098+0000", 2024-02-09T17:24:51.435 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.004682126 2024-02-09T17:24:51.435 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.435 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.435 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:24:51.435 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.658136+0000", 2024-02-09T17:24:51.435 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.8420999999999997e-05 2024-02-09T17:24:51.435 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:24:51.435 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:24:51.436 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:24:51.436 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.436 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.436 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:4761 2.5 e117/111 2:afa62986:::benchmark_data_smithi179_77138_object4760:head v 117'268, mlcod=117'266)", 2024-02-09T17:24:51.436 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:24:50.846160+0000", 2024-02-09T17:24:51.436 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.50703078499999998, 2024-02-09T17:24:51.436 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0056034850000000001, 2024-02-09T17:24:51.436 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:24:51.436 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:24:51.436 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:24:51.436 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.436 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:24:51.437 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.846160+0000", 2024-02-09T17:24:51.437 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:24:51.437 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.437 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.437 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:24:51.437 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.846160+0000", 2024-02-09T17:24:51.437 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:24:51.437 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.437 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.437 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:24:51.437 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.846160+0000", 2024-02-09T17:24:51.437 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.0699999999999996e-07 2024-02-09T17:24:51.437 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.438 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.438 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:24:51.438 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.846259+0000", 2024-02-09T17:24:51.438 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.8629999999999996e-05 2024-02-09T17:24:51.438 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.438 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.438 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:24:51.438 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.846268+0000", 2024-02-09T17:24:51.438 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.5399999999999996e-06 2024-02-09T17:24:51.439 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.439 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.439 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:24:51.439 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.846272+0000", 2024-02-09T17:24:51.439 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.702e-06 2024-02-09T17:24:51.439 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.439 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.439 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:24:51.439 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.846975+0000", 2024-02-09T17:24:51.439 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00070264399999999999 2024-02-09T17:24:51.439 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.440 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.440 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:24:51.440 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.846990+0000", 2024-02-09T17:24:51.440 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5265999999999999e-05 2024-02-09T17:24:51.440 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.440 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.440 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:24:51.440 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.851723+0000", 2024-02-09T17:24:51.440 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0047329980000000004 2024-02-09T17:24:51.440 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.440 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.440 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:24:51.440 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:50.851763+0000", 2024-02-09T17:24:51.441 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9997999999999999e-05 2024-02-09T17:24:51.441 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:24:51.441 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:24:51.441 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:24:51.441 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.441 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.441 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:6046 2.1 e117/111 2:8bf5d13f:::benchmark_data_smithi179_77138_object6045:head v 117'409, mlcod=117'406)", 2024-02-09T17:24:51.441 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:24:51.149442+0000", 2024-02-09T17:24:51.441 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.20374825799999999, 2024-02-09T17:24:51.441 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0052363510000000002, 2024-02-09T17:24:51.441 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:24:51.441 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:24:51.441 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:24:51.442 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.442 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:24:51.442 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.149442+0000", 2024-02-09T17:24:51.442 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:24:51.442 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.442 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.442 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:24:51.442 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.149442+0000", 2024-02-09T17:24:51.442 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:24:51.442 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.442 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.442 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:24:51.443 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.149443+0000", 2024-02-09T17:24:51.443 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.5099999999999999e-07 2024-02-09T17:24:51.443 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.443 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.443 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:24:51.443 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.149463+0000", 2024-02-09T17:24:51.443 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0197e-05 2024-02-09T17:24:51.443 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.443 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.443 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:24:51.443 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.149474+0000", 2024-02-09T17:24:51.443 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0509999999999999e-05 2024-02-09T17:24:51.443 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.443 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.444 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:24:51.444 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.149480+0000", 2024-02-09T17:24:51.444 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.7429999999999996e-06 2024-02-09T17:24:51.444 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.444 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.444 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:24:51.444 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.149599+0000", 2024-02-09T17:24:51.444 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000119094 2024-02-09T17:24:51.444 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.444 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.444 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:24:51.444 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.149637+0000", 2024-02-09T17:24:51.444 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.7039999999999998e-05 2024-02-09T17:24:51.444 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.445 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.445 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:24:51.445 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.154636+0000", 2024-02-09T17:24:51.445 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0049998960000000002 2024-02-09T17:24:51.445 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.445 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.445 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:24:51.445 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.154679+0000", 2024-02-09T17:24:51.445 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2219999999999999e-05 2024-02-09T17:24:51.445 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:24:51.445 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:24:51.445 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:24:51.446 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.446 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.446 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:6049 2.1 e117/111 2:883dc1c8:::benchmark_data_smithi179_77138_object6048:head v 117'410, mlcod=117'406)", 2024-02-09T17:24:51.446 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:24:51.149501+0000", 2024-02-09T17:24:51.446 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.20368935699999999, 2024-02-09T17:24:51.446 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0052058479999999999, 2024-02-09T17:24:51.446 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:24:51.446 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:24:51.446 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:24:51.446 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.446 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:24:51.446 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.149501+0000", 2024-02-09T17:24:51.446 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:24:51.446 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.447 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.447 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:24:51.447 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.149501+0000", 2024-02-09T17:24:51.447 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:24:51.447 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.447 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.447 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:24:51.447 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.149502+0000", 2024-02-09T17:24:51.447 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.1699999999999999e-07 2024-02-09T17:24:51.447 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.448 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.448 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:24:51.448 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.149523+0000", 2024-02-09T17:24:51.448 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.1202000000000001e-05 2024-02-09T17:24:51.448 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.448 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.448 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:24:51.448 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.149530+0000", 2024-02-09T17:24:51.448 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.8140000000000003e-06 2024-02-09T17:24:51.448 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.448 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.449 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:24:51.449 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.149534+0000", 2024-02-09T17:24:51.449 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.9529999999999998e-06 2024-02-09T17:24:51.449 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.449 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.449 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:24:51.449 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.150009+0000", 2024-02-09T17:24:51.449 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00047430200000000001 2024-02-09T17:24:51.449 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.449 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.449 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:24:51.449 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.150027+0000", 2024-02-09T17:24:51.449 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.8087000000000001e-05 2024-02-09T17:24:51.449 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.449 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.450 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:24:51.450 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.154681+0000", 2024-02-09T17:24:51.450 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0046544940000000003 2024-02-09T17:24:51.450 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.450 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.450 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:24:51.450 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.154707+0000", 2024-02-09T17:24:51.450 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.5678999999999999e-05 2024-02-09T17:24:51.450 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:24:51.450 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:24:51.450 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:24:51.450 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.450 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.450 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:6074 2.1 e117/111 2:863b1468:::benchmark_data_smithi179_77138_object6073:head v 117'413, mlcod=117'407)", 2024-02-09T17:24:51.451 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:24:51.157293+0000", 2024-02-09T17:24:51.451 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.19589752299999999, 2024-02-09T17:24:51.451 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.005517175, 2024-02-09T17:24:51.451 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:24:51.451 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:24:51.451 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:24:51.451 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.451 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:24:51.451 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.157293+0000", 2024-02-09T17:24:51.451 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:24:51.451 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.451 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.451 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:24:51.451 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.157293+0000", 2024-02-09T17:24:51.452 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:24:51.452 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.452 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.452 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:24:51.452 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.157294+0000", 2024-02-09T17:24:51.452 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.3399999999999998e-07 2024-02-09T17:24:51.452 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.452 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.452 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:24:51.452 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.157322+0000", 2024-02-09T17:24:51.452 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.7906999999999999e-05 2024-02-09T17:24:51.452 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.452 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.453 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:24:51.453 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.157341+0000", 2024-02-09T17:24:51.453 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.8787000000000002e-05 2024-02-09T17:24:51.453 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.453 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.453 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:24:51.453 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.157350+0000", 2024-02-09T17:24:51.453 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.2439999999999999e-06 2024-02-09T17:24:51.453 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.453 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.453 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:24:51.453 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.157442+0000", 2024-02-09T17:24:51.453 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.1694999999999995e-05 2024-02-09T17:24:51.453 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.454 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.454 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:24:51.454 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.157463+0000", 2024-02-09T17:24:51.454 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.1339999999999999e-05 2024-02-09T17:24:51.454 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.454 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.454 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:24:51.454 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.162766+0000", 2024-02-09T17:24:51.454 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0053036239999999998 2024-02-09T17:24:51.454 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.454 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.454 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:24:51.454 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.162810+0000", 2024-02-09T17:24:51.454 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.3744e-05 2024-02-09T17:24:51.455 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:24:51.455 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:24:51.455 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:24:51.455 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.455 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.455 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:6485 2.4 e117/111 2:276da765:::benchmark_data_smithi179_77138_object6484:head v 117'379, mlcod=117'377)", 2024-02-09T17:24:51.455 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:24:51.263573+0000", 2024-02-09T17:24:51.455 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.089617587999999998, 2024-02-09T17:24:51.455 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0050135529999999996, 2024-02-09T17:24:51.455 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:24:51.455 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:24:51.455 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:24:51.455 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.456 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:24:51.456 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.263573+0000", 2024-02-09T17:24:51.456 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:24:51.456 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.456 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.456 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:24:51.456 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.263573+0000", 2024-02-09T17:24:51.456 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:24:51.456 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.456 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.456 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:24:51.456 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.263573+0000", 2024-02-09T17:24:51.456 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2699999999999999e-07 2024-02-09T17:24:51.457 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.457 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.457 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:24:51.457 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.263595+0000", 2024-02-09T17:24:51.457 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.1016999999999999e-05 2024-02-09T17:24:51.457 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.457 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.457 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:24:51.457 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.263605+0000", 2024-02-09T17:24:51.457 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0030999999999999e-05 2024-02-09T17:24:51.457 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.457 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.457 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:24:51.458 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.263610+0000", 2024-02-09T17:24:51.458 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6890000000000001e-06 2024-02-09T17:24:51.458 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.458 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.458 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:24:51.458 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.263699+0000", 2024-02-09T17:24:51.458 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.8959e-05 2024-02-09T17:24:51.458 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.458 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.458 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:24:51.458 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.263729+0000", 2024-02-09T17:24:51.458 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.0176e-05 2024-02-09T17:24:51.458 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.458 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.458 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:24:51.459 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.268545+0000", 2024-02-09T17:24:51.459 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0048152280000000004 2024-02-09T17:24:51.459 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.459 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.459 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:24:51.459 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.268587+0000", 2024-02-09T17:24:51.459 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.1925999999999998e-05 2024-02-09T17:24:51.459 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:24:51.459 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:24:51.459 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:24:51.459 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.459 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.459 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:6680 2.5 e117/111 2:a0b71b83:::benchmark_data_smithi179_77138_object6679:head v 117'388, mlcod=117'385)", 2024-02-09T17:24:51.459 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:24:51.316489+0000", 2024-02-09T17:24:51.459 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.036701228000000002, 2024-02-09T17:24:51.460 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0069749579999999999, 2024-02-09T17:24:51.460 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:24:51.460 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:24:51.460 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:24:51.460 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.460 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:24:51.460 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.316489+0000", 2024-02-09T17:24:51.460 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:24:51.460 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.460 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.460 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:24:51.460 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.316489+0000", 2024-02-09T17:24:51.460 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:24:51.461 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.461 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.461 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:24:51.461 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.316490+0000", 2024-02-09T17:24:51.461 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.8100000000000003e-07 2024-02-09T17:24:51.461 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.461 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.461 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:24:51.461 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.316555+0000", 2024-02-09T17:24:51.461 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.4942000000000003e-05 2024-02-09T17:24:51.461 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.461 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.461 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:24:51.461 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.316565+0000", 2024-02-09T17:24:51.462 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0448999999999999e-05 2024-02-09T17:24:51.462 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.462 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.462 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:24:51.462 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.316573+0000", 2024-02-09T17:24:51.462 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.2439999999999996e-06 2024-02-09T17:24:51.462 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.462 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.462 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:24:51.462 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.316625+0000", 2024-02-09T17:24:51.462 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2583000000000001e-05 2024-02-09T17:24:51.462 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.462 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.462 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:24:51.463 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.316647+0000", 2024-02-09T17:24:51.463 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.141e-05 2024-02-09T17:24:51.463 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.463 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.463 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:24:51.463 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.323417+0000", 2024-02-09T17:24:51.463 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0067701790000000003 2024-02-09T17:24:51.463 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.463 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.463 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:24:51.463 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.323464+0000", 2024-02-09T17:24:51.463 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.757e-05 2024-02-09T17:24:51.463 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:24:51.463 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:24:51.464 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:24:51.464 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.464 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.464 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:6726 2.c e117/111 2:31c2354d:::benchmark_data_smithi179_77138_object6725:head v 117'428, mlcod=117'426)", 2024-02-09T17:24:51.464 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:24:51.329854+0000", 2024-02-09T17:24:51.464 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.023336593999999999, 2024-02-09T17:24:51.464 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0056745750000000003, 2024-02-09T17:24:51.464 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:24:51.464 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:24:51.464 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:24:51.464 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.464 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:24:51.464 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.329854+0000", 2024-02-09T17:24:51.464 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:24:51.465 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.465 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.465 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:24:51.465 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.329854+0000", 2024-02-09T17:24:51.465 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:24:51.465 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.465 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.465 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:24:51.465 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.329855+0000", 2024-02-09T17:24:51.465 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.7199999999999997e-07 2024-02-09T17:24:51.465 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.465 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.465 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:24:51.465 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.329885+0000", 2024-02-09T17:24:51.466 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.0645000000000002e-05 2024-02-09T17:24:51.466 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.466 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.466 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:24:51.466 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.329898+0000", 2024-02-09T17:24:51.466 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2836e-05 2024-02-09T17:24:51.466 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.466 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.466 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:24:51.466 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.329906+0000", 2024-02-09T17:24:51.466 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.1659999999999994e-06 2024-02-09T17:24:51.466 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.467 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.467 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:24:51.467 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.329984+0000", 2024-02-09T17:24:51.467 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.7768999999999994e-05 2024-02-09T17:24:51.467 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.467 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.467 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:24:51.467 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.330007+0000", 2024-02-09T17:24:51.467 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2297999999999999e-05 2024-02-09T17:24:51.467 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.467 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.467 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:24:51.467 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.335479+0000", 2024-02-09T17:24:51.468 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0054720819999999996 2024-02-09T17:24:51.468 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.468 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.468 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:24:51.468 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.335529+0000", 2024-02-09T17:24:51.468 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.9907000000000002e-05 2024-02-09T17:24:51.468 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:24:51.468 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:24:51.468 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:24:51.468 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.468 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.468 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:6729 2.c e117/111 2:353b38ad:::benchmark_data_smithi179_77138_object6728:head v 117'429, mlcod=117'426)", 2024-02-09T17:24:51.468 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:24:51.330465+0000", 2024-02-09T17:24:51.469 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.022725929999999998, 2024-02-09T17:24:51.469 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0055982979999999998, 2024-02-09T17:24:51.469 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:24:51.469 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:24:51.469 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:24:51.469 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.469 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:24:51.469 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.330465+0000", 2024-02-09T17:24:51.469 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:24:51.469 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.469 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.469 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:24:51.469 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.330465+0000", 2024-02-09T17:24:51.470 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:24:51.470 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.470 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.470 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:24:51.470 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.330465+0000", 2024-02-09T17:24:51.470 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.4499999999999997e-07 2024-02-09T17:24:51.470 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.470 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.470 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:24:51.470 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.330497+0000", 2024-02-09T17:24:51.470 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.1273999999999999e-05 2024-02-09T17:24:51.470 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.470 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.471 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:24:51.471 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.330506+0000", 2024-02-09T17:24:51.471 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.5310000000000007e-06 2024-02-09T17:24:51.471 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.471 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.471 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:24:51.471 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.330513+0000", 2024-02-09T17:24:51.471 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.6800000000000004e-06 2024-02-09T17:24:51.471 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.471 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.471 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:24:51.471 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.334691+0000", 2024-02-09T17:24:51.471 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0041779490000000002 2024-02-09T17:24:51.472 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.472 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.472 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:24:51.472 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.334710+0000", 2024-02-09T17:24:51.472 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.9652000000000001e-05 2024-02-09T17:24:51.472 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.472 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.472 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:24:51.472 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.336019+0000", 2024-02-09T17:24:51.472 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.001308965 2024-02-09T17:24:51.472 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:24:51.472 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:24:51.472 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:24:51.472 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:51.336063+0000", 2024-02-09T17:24:51.473 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.3602e-05 2024-02-09T17:24:51.473 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:24:51.473 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:24:51.473 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:24:51.473 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:24:51.473 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:24:51.473 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:24:51.474 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 2 16 7179 7163 223.786 262.438 0.00914854 0.00445776 2024-02-09T17:24:51.678 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:51 smithi099 ceph-mon[39420]: pgmap v360: 17 pgs: 17 active+clean; 11 MiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s wr, 20 op/s 2024-02-09T17:24:51.699 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:24:51.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:51.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:24:51.805 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:24:51.840 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:51 smithi162 ceph-mon[44304]: pgmap v360: 17 pgs: 17 active+clean; 11 MiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s wr, 20 op/s 2024-02-09T17:24:51.842 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:51.867 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:24:51.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:51.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:24:51.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:51 smithi179 ceph-mon[43975]: pgmap v360: 17 pgs: 17 active+clean; 11 MiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s wr, 20 op/s 2024-02-09T17:24:51.971 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:24:52.013 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:52.441 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 3 16 10743 10727 223.431 222.75 0.00126935 0.00425728 2024-02-09T17:24:52.661 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-09T17:24:52.662 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-02-09T17:24:52.662 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph tell osd.2 'trim stale osdmaps' 2024-02-09T17:24:53.441 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 4 16 13827 13811 215.748 192.75 0.00557986 0.00462987 2024-02-09T17:24:53.585 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-09T17:24:53.725 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:24:53.767 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:53 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:24:53.767 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:53 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:24:53.767 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:53 smithi099 ceph-mon[39420]: pgmap v361: 17 pgs: 17 active+clean; 98 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 195 op/s 2024-02-09T17:24:53.767 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:53 smithi099 ceph-mon[39420]: osdmap e118: 12 total, 12 up, 12 in 2024-02-09T17:24:53.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:53 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:24:53.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:53 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:24:53.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:53 smithi179 ceph-mon[43975]: pgmap v361: 17 pgs: 17 active+clean; 98 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 195 op/s 2024-02-09T17:24:53.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:53 smithi179 ceph-mon[43975]: osdmap e118: 12 total, 12 up, 12 in 2024-02-09T17:24:53.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:53 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:24:53.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:53 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:24:53.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:53 smithi162 ceph-mon[44304]: pgmap v361: 17 pgs: 17 active+clean; 98 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 195 op/s 2024-02-09T17:24:53.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:53 smithi162 ceph-mon[44304]: osdmap e118: 12 total, 12 up, 12 in 2024-02-09T17:24:54.066 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:24:54.073 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:24:54.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:54.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:24:54.122 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:24:54.179 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:24:54.226 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:54.255 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:24:54.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:54.260 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:24:54.361 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:24:54.408 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:54.432 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:24:54.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:54.436 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:24:54.441 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 5 16 16830 16814 210.133 187.688 0.00366306 0.00475355 2024-02-09T17:24:54.537 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:24:54.584 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:54.722 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:24:54.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:54.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:24:54.827 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:24:54.876 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:55.441 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 6 15 19882 19867 206.911 190.812 0.00608713 0.00482735 2024-02-09T17:24:55.539 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:24:55.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:55.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:24:55.563 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is set 2024-02-09T17:24:55.643 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:24:55.685 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:55.713 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:24:55.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:55.717 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:24:55.818 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:24:55.855 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:55 smithi179 ceph-mon[43975]: pgmap v363: 17 pgs: 17 active+clean; 348 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 43 MiB/s wr, 694 op/s 2024-02-09T17:24:55.855 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:55 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3468141100' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:24:55.856 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:55 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:24:55.857 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:55.884 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:24:55.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:55.886 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:24:55.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:55 smithi162 ceph-mon[44304]: pgmap v363: 17 pgs: 17 active+clean; 348 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 43 MiB/s wr, 694 op/s 2024-02-09T17:24:55.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:55 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3468141100' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:24:55.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:55 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:24:55.988 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:24:55.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:55 smithi099 ceph-mon[39420]: pgmap v363: 17 pgs: 17 active+clean; 348 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 43 MiB/s wr, 694 op/s 2024-02-09T17:24:55.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:55 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3468141100' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:24:55.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:55 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:24:56.014 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:56.033 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:24:56.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:56.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:24:56.136 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:24:56.185 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:56.252 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:24:56.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:56.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:24:56.357 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:24:56.399 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:56.441 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 7 15 22450 22435 200.274 160.5 0.00759571 0.00498822 2024-02-09T17:24:56.610 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:24:56.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:56.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:24:56.648 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:56 smithi099 ceph-mon[39420]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:24:56.648 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:56 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:24:56.648 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:56 smithi099 ceph-mon[39420]: osdmap e119: 12 total, 12 up, 12 in 2024-02-09T17:24:56.648 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:56 smithi099 ceph-mon[39420]: osdmap e120: 12 total, 12 up, 12 in 2024-02-09T17:24:56.715 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:24:56.750 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:56.776 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:24:56.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:56.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:24:56.781 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:56 smithi162 ceph-mon[44304]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:24:56.781 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:56 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:24:56.781 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:56 smithi162 ceph-mon[44304]: osdmap e119: 12 total, 12 up, 12 in 2024-02-09T17:24:56.781 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:56 smithi162 ceph-mon[44304]: osdmap e120: 12 total, 12 up, 12 in 2024-02-09T17:24:56.881 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:24:56.917 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:56.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:56 smithi179 ceph-mon[43975]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:24:56.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:56 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:24:56.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:56 smithi179 ceph-mon[43975]: osdmap e119: 12 total, 12 up, 12 in 2024-02-09T17:24:56.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:56 smithi179 ceph-mon[43975]: osdmap e120: 12 total, 12 up, 12 in 2024-02-09T17:24:56.943 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:24:56.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:56.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:24:57.047 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:24:57.085 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:57.114 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:24:57.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:57.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:24:57.217 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:24:57.276 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:57.338 INFO:teuthology.orchestra.run.smithi099.stderr:no valid command found; 10 closest matches: 2024-02-09T17:24:57.339 INFO:teuthology.orchestra.run.smithi099.stderr:0 2024-02-09T17:24:57.339 INFO:teuthology.orchestra.run.smithi099.stderr:1 2024-02-09T17:24:57.339 INFO:teuthology.orchestra.run.smithi099.stderr:2 2024-02-09T17:24:57.339 INFO:teuthology.orchestra.run.smithi099.stderr:abort 2024-02-09T17:24:57.339 INFO:teuthology.orchestra.run.smithi099.stderr:assert 2024-02-09T17:24:57.339 INFO:teuthology.orchestra.run.smithi099.stderr:bench [] [] [] [] 2024-02-09T17:24:57.339 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs debug_inject_read_zeros 2024-02-09T17:24:57.339 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs files list 2024-02-09T17:24:57.339 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs stats 2024-02-09T17:24:57.339 INFO:teuthology.orchestra.run.smithi099.stderr:bluestore allocator dump block 2024-02-09T17:24:57.339 INFO:teuthology.orchestra.run.smithi099.stderr:Error EINVAL: invalid command 2024-02-09T17:24:57.387 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:24:57.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:57.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:24:57.442 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 8 16 25368 25352 198.027 182.312 0.0067246 0.00504534 2024-02-09T17:24:57.490 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:24:57.538 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:57.570 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:24:57.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:57.575 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:24:57.676 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:24:57.704 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:57 smithi099 ceph-mon[39420]: pgmap v366: 17 pgs: 17 active+clean; 1013 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 167 MiB/s wr, 2.67k op/s 2024-02-09T17:24:57.706 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:57.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:57 smithi179 ceph-mon[43975]: pgmap v366: 17 pgs: 17 active+clean; 1013 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 167 MiB/s wr, 2.67k op/s 2024-02-09T17:24:57.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:57 smithi162 ceph-mon[44304]: pgmap v366: 17 pgs: 17 active+clean; 1013 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 167 MiB/s wr, 2.67k op/s 2024-02-09T17:24:57.935 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:24:57.954 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:24:57.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:57.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:24:58.062 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:24:58.111 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:58.442 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 9 15 27685 27670 192.12 144.875 0.0109328 0.00519992 2024-02-09T17:24:58.442 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-09T17:24:58.443 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-02-09T17:24:58.443 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph tell osd.3 'trim stale osdmaps' 2024-02-09T17:24:58.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:58 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:24:58.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:58 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:24:58.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:58 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:24:58.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:58 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:24:58.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:58 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:24:58.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:58 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:24:59.038 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset noscrub 2024-02-09T17:24:59.384 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:24:59.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:59.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:24:59.442 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 10 16 30152 30136 188.319 154.125 0.00232394 0.00526118 2024-02-09T17:24:59.487 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:24:59.514 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:59.618 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:24:59.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:59.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:24:59.639 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:59 smithi099 ceph-mon[39420]: pgmap v367: 17 pgs: 17 active+clean; 1.1 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 172 MiB/s wr, 2.75k op/s 2024-02-09T17:24:59.640 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:24:59 smithi099 ceph-mon[39420]: osdmap e121: 12 total, 12 up, 12 in 2024-02-09T17:24:59.723 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:24:59.762 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:24:59.796 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:24:59.798 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:59 smithi179 ceph-mon[43975]: pgmap v367: 17 pgs: 17 active+clean; 1.1 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 172 MiB/s wr, 2.75k op/s 2024-02-09T17:24:59.798 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:24:59 smithi179 ceph-mon[43975]: osdmap e121: 12 total, 12 up, 12 in 2024-02-09T17:24:59.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:24:59.799 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:24:59.900 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:24:59.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:59 smithi162 ceph-mon[44304]: pgmap v367: 17 pgs: 17 active+clean; 1.1 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 172 MiB/s wr, 2.75k op/s 2024-02-09T17:24:59.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:24:59 smithi162 ceph-mon[44304]: osdmap e121: 12 total, 12 up, 12 in 2024-02-09T17:24:59.948 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:00.060 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:25:00.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:00.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:25:00.167 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:25:00.216 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:00.246 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:25:00.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:00.249 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:25:00.351 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:25:00.400 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:00.432 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:25:00.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:00.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:25:00.442 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 11 16 32559 32543 184.875 150.438 0.00917432 0.00540381 2024-02-09T17:25:00.519 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:25:00.533 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:25:00.537 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:25:00.569 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:00.594 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:25:00.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:00.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:25:00.699 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:25:00.754 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:00.780 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:25:00.789 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:00.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:25:00.890 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:25:00.934 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:00.960 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:25:00.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:00.963 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:25:01.064 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:25:01.105 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:01.130 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:25:01.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:01.133 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:25:01.234 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:25:01.261 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:01.279 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:25:01.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:01.281 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:25:01.323 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:25:01.323 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [], 2024-02-09T17:25:01.323 INFO:teuthology.orchestra.run.smithi099.stdout: "num_ops": 0 2024-02-09T17:25:01.324 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:25:01.382 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:25:01.417 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:01.441 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:25:01.442 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 12 16 34591 34575 180.051 127 0.00238831 0.00554945 2024-02-09T17:25:01.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:01.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:25:01.546 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:25:01.581 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:01.605 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:25:01.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:01.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:25:01.624 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:01 smithi099 ceph-mon[39420]: pgmap v369: 17 pgs: 17 active+clean; 1.7 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 234 MiB/s wr, 3.74k op/s 2024-02-09T17:25:01.624 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:01 smithi099 ceph-mon[39420]: osdmap e122: 12 total, 12 up, 12 in 2024-02-09T17:25:01.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:01 smithi162 ceph-mon[44304]: pgmap v369: 17 pgs: 17 active+clean; 1.7 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 234 MiB/s wr, 3.74k op/s 2024-02-09T17:25:01.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:01 smithi162 ceph-mon[44304]: osdmap e122: 12 total, 12 up, 12 in 2024-02-09T17:25:01.711 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:25:01.753 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:01.781 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:25:01.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:01.787 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:25:01.888 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:25:01.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:01 smithi179 ceph-mon[43975]: pgmap v369: 17 pgs: 17 active+clean; 1.7 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 234 MiB/s wr, 3.74k op/s 2024-02-09T17:25:01.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:01 smithi179 ceph-mon[43975]: osdmap e122: 12 total, 12 up, 12 in 2024-02-09T17:25:01.939 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:01.970 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:25:01.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:01.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:25:02.074 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:25:02.114 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:02.144 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:25:02.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:02.147 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:25:02.248 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:25:02.284 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:02.310 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:25:02.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:02.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:25:02.415 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:25:02.442 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 13 16 37030 37014 177.926 152.438 0.0090878 0.00561559 2024-02-09T17:25:02.450 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:02.474 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:25:02.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:02.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:25:02.580 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:25:02.620 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:02.650 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:25:02.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:02.657 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:25:02.758 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:25:02.786 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-09T17:25:02.793 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:02.815 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:25:02.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:02.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:25:02.919 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:25:02.992 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:03.177 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:25:03.177 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:25:03.197 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:25:03.198 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:25:03.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:03.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:25:03.251 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:25:03.284 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:25:03.303 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:25:03.319 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:25:03.342 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:03.348 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:25:03.368 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:25:03.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:03.372 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:25:03.383 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:25:03.422 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:25:03.442 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 14 16 39094 39078 174.431 129 0.00421874 0.00572909 2024-02-09T17:25:03.457 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:25:03.473 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:25:03.492 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:25:03.529 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:03.541 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:25:03.561 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:25:03.561 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:25:03.561 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:25:03.561 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:25:03.561 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:25:03.561 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:25:03.561 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:25:03.561 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:25:03.561 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:25:03.561 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:25:03.562 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:25:03.562 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:25:03.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:03 smithi179 ceph-mon[43975]: pgmap v371: 17 pgs: 17 active+clean; 1.9 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 157 MiB/s wr, 2.52k op/s 2024-02-09T17:25:03.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:03 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/408500760' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:25:03.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:03 smithi099 ceph-mon[39420]: pgmap v371: 17 pgs: 17 active+clean; 1.9 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 157 MiB/s wr, 2.52k op/s 2024-02-09T17:25:03.748 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:03 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/408500760' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:25:03.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:03 smithi162 ceph-mon[44304]: pgmap v371: 17 pgs: 17 active+clean; 1.9 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 157 MiB/s wr, 2.52k op/s 2024-02-09T17:25:03.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:03 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/408500760' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:25:04.186 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:25:04.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:04.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:25:04.292 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:25:04.341 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:04.442 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 15 16 41452 41436 172.627 147.375 0.00595246 0.00578854 2024-02-09T17:25:04.902 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:25:04.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:04.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:25:05.007 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:25:05.053 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:05.374 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:25:05.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:05.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:25:05.442 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 16 16 43781 43765 170.935 145.562 0.00397691 0.00584588 2024-02-09T17:25:05.482 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:25:05.514 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:05.535 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:25:05.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:05.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:25:05.593 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:25:05.638 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:25:05.680 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:05.703 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:25:05.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:05.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:25:05.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:05 smithi099 ceph-mon[39420]: pgmap v372: 17 pgs: 17 active+clean; 2.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 138 MiB/s wr, 2.21k op/s 2024-02-09T17:25:05.808 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:25:05.840 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:05 smithi162 ceph-mon[44304]: pgmap v372: 17 pgs: 17 active+clean; 2.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 138 MiB/s wr, 2.21k op/s 2024-02-09T17:25:05.842 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:05.865 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:25:05.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:05.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:25:05.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:05 smithi179 ceph-mon[43975]: pgmap v372: 17 pgs: 17 active+clean; 2.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 138 MiB/s wr, 2.21k op/s 2024-02-09T17:25:05.969 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:25:06.004 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:06.029 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:25:06.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:06.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:25:06.135 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:25:06.184 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:06.442 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 17 16 45782 45766 168.235 125.062 0.00331118 0.00593942 2024-02-09T17:25:06.516 INFO:teuthology.orchestra.run.smithi099.stderr:no valid command found; 10 closest matches: 2024-02-09T17:25:06.517 INFO:teuthology.orchestra.run.smithi099.stderr:0 2024-02-09T17:25:06.517 INFO:teuthology.orchestra.run.smithi099.stderr:1 2024-02-09T17:25:06.517 INFO:teuthology.orchestra.run.smithi099.stderr:2 2024-02-09T17:25:06.517 INFO:teuthology.orchestra.run.smithi099.stderr:abort 2024-02-09T17:25:06.517 INFO:teuthology.orchestra.run.smithi099.stderr:assert 2024-02-09T17:25:06.517 INFO:teuthology.orchestra.run.smithi099.stderr:bench [] [] [] [] 2024-02-09T17:25:06.517 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs debug_inject_read_zeros 2024-02-09T17:25:06.517 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs files list 2024-02-09T17:25:06.517 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs stats 2024-02-09T17:25:06.517 INFO:teuthology.orchestra.run.smithi099.stderr:bluestore allocator dump block 2024-02-09T17:25:06.517 INFO:teuthology.orchestra.run.smithi099.stderr:Error EINVAL: invalid command 2024-02-09T17:25:06.537 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:25:06.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:06.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:25:06.645 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:25:06.685 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:06.714 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:25:06.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:06.717 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:25:06.818 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:25:06.857 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:06.883 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:25:06.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:06.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:25:06.988 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:25:07.015 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:07 smithi179 ceph-mon[43975]: osdmap e123: 12 total, 12 up, 12 in 2024-02-09T17:25:07.015 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:07 smithi179 ceph-mon[43975]: pgmap v374: 17 pgs: 17 active+clean; 2.5 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 184 MiB/s wr, 2.95k op/s 2024-02-09T17:25:07.015 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:07 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1870033578' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:25:07.015 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:07 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:25:07.042 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:07.054 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is unset 2024-02-09T17:25:07.167 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:25:07.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:07 smithi162 ceph-mon[44304]: osdmap e123: 12 total, 12 up, 12 in 2024-02-09T17:25:07.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:07 smithi162 ceph-mon[44304]: pgmap v374: 17 pgs: 17 active+clean; 2.5 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 184 MiB/s wr, 2.95k op/s 2024-02-09T17:25:07.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:07 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1870033578' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:25:07.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:07 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:25:07.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:07.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:25:07.285 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:25:07.300 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:07 smithi099 ceph-mon[39420]: osdmap e123: 12 total, 12 up, 12 in 2024-02-09T17:25:07.300 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:07 smithi099 ceph-mon[39420]: pgmap v374: 17 pgs: 17 active+clean; 2.5 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 184 MiB/s wr, 2.95k op/s 2024-02-09T17:25:07.300 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:07 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1870033578' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:25:07.300 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:07 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:25:07.332 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:07.361 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:25:07.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:07.364 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:25:07.408 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:25:07.442 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 18 16 47778 47762 165.819 124.75 0.0068946 0.00602594 2024-02-09T17:25:07.466 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:25:07.503 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:07.525 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:25:07.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:07.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:25:07.629 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:25:07.659 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:08.111 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-09T17:25:08.111 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-02-09T17:25:08.112 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph tell osd.4 'trim stale osdmaps' 2024-02-09T17:25:08.114 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:25:08.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:08.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:25:08.220 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:25:08.255 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:08.279 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:25:08.281 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:08 smithi162 ceph-mon[44304]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:25:08.281 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:08 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:25:08.281 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:08 smithi162 ceph-mon[44304]: osdmap e124: 12 total, 12 up, 12 in 2024-02-09T17:25:08.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:08.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:25:08.383 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:25:08.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:08 smithi179 ceph-mon[43975]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:25:08.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:08 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:25:08.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:08 smithi179 ceph-mon[43975]: osdmap e124: 12 total, 12 up, 12 in 2024-02-09T17:25:08.430 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:08 smithi099 ceph-mon[39420]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:25:08.430 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:08 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:25:08.430 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:08 smithi099 ceph-mon[39420]: osdmap e124: 12 total, 12 up, 12 in 2024-02-09T17:25:08.433 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:08.442 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 19 16 49613 49597 163.128 114.688 0.0110168 0.00612639 2024-02-09T17:25:08.786 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:25:08.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:08.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:25:08.891 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:25:08.927 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:09.201 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:09 smithi099 ceph-mon[39420]: pgmap v376: 17 pgs: 17 active+clean; 2.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 117 MiB/s wr, 1.87k op/s 2024-02-09T17:25:09.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:09 smithi179 ceph-mon[43975]: pgmap v376: 17 pgs: 17 active+clean; 2.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 117 MiB/s wr, 1.87k op/s 2024-02-09T17:25:09.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:09 smithi162 ceph-mon[44304]: pgmap v376: 17 pgs: 17 active+clean; 2.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 117 MiB/s wr, 1.87k op/s 2024-02-09T17:25:09.444 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:2024-02-09T17:25:09.443864+0000 min lat: 0.00106674 max lat: 0.214389 avg lat: 0.006229 2024-02-09T17:25:09.444 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-09T17:25:09.444 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 20 16 51365 51349 160.433 109.5 0.00544071 0.006229 2024-02-09T17:25:09.477 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:25:09.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:09.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:25:09.584 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:25:09.628 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:09.656 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:25:09.657 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:09.658 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:25:09.759 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:25:09.811 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:25:09.828 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:25:09.837 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:09.849 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:25:09.850 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:25:09.969 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:25:10.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:10.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:25:10.230 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:25:10.284 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:10.444 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 21 16 53141 53125 158.079 111 0.00265368 0.00632087 2024-02-09T17:25:11.247 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-09T17:25:11.444 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 22 16 54799 54783 155.604 103.625 0.0129393 0.00642114 2024-02-09T17:25:11.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:11 smithi179 ceph-mon[43975]: pgmap v377: 17 pgs: 17 active+clean; 3.0 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 147 MiB/s wr, 2.34k op/s 2024-02-09T17:25:11.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:11 smithi179 ceph-mon[43975]: osdmap e125: 12 total, 12 up, 12 in 2024-02-09T17:25:11.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:11 smithi162 ceph-mon[44304]: pgmap v377: 17 pgs: 17 active+clean; 3.0 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 147 MiB/s wr, 2.34k op/s 2024-02-09T17:25:11.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:11 smithi162 ceph-mon[44304]: osdmap e125: 12 total, 12 up, 12 in 2024-02-09T17:25:11.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:11 smithi099 ceph-mon[39420]: pgmap v377: 17 pgs: 17 active+clean; 3.0 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 147 MiB/s wr, 2.34k op/s 2024-02-09T17:25:11.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:11 smithi099 ceph-mon[39420]: osdmap e125: 12 total, 12 up, 12 in 2024-02-09T17:25:12.444 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 23 16 56514 56498 153.498 107.188 0.00540301 0.0065108 2024-02-09T17:25:12.666 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:12 smithi099 ceph-mon[39420]: 1.0 scrub starts 2024-02-09T17:25:12.667 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:12 smithi099 ceph-mon[39420]: 1.0 scrub ok 2024-02-09T17:25:12.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:12 smithi179 ceph-mon[43975]: 1.0 scrub starts 2024-02-09T17:25:12.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:12 smithi179 ceph-mon[43975]: 1.0 scrub ok 2024-02-09T17:25:12.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:12 smithi162 ceph-mon[44304]: 1.0 scrub starts 2024-02-09T17:25:12.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:12 smithi162 ceph-mon[44304]: 1.0 scrub ok 2024-02-09T17:25:13.183 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:25:13.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:13.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:25:13.294 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:25:13.327 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:13.352 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:25:13.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:13.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:25:13.444 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 24 16 58005 57989 150.985 93.1875 0.00235874 0.00660465 2024-02-09T17:25:13.456 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:25:13.485 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:13.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:13 smithi179 ceph-mon[43975]: pgmap v379: 17 pgs: 17 active+clean; 3.2 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 116 MiB/s wr, 1.86k op/s 2024-02-09T17:25:13.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:13 smithi162 ceph-mon[44304]: pgmap v379: 17 pgs: 17 active+clean; 3.2 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 116 MiB/s wr, 1.86k op/s 2024-02-09T17:25:13.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:13 smithi099 ceph-mon[39420]: pgmap v379: 17 pgs: 17 active+clean; 3.2 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 116 MiB/s wr, 1.86k op/s 2024-02-09T17:25:14.237 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:25:14.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:14.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:25:14.278 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-09T17:25:14.343 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:25:14.408 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:14.444 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 25 16 59375 59359 148.371 85.625 0.00304612 0.00673436 2024-02-09T17:25:15.103 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:25:15.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:15.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:25:15.211 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:25:15.246 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:15.268 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:25:15.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:15.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:25:15.376 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:25:15.421 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:15.444 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 26 16 60706 60690 145.864 83.1875 0.00436178 0.0068464 2024-02-09T17:25:15.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:15 smithi179 ceph-mon[43975]: pgmap v380: 17 pgs: 17 active+clean; 3.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 102 MiB/s wr, 1.63k op/s 2024-02-09T17:25:15.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:15 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:25:15.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:15 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:25:15.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:15 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:25:15.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:15 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:25:15.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:15 smithi162 ceph-mon[44304]: pgmap v380: 17 pgs: 17 active+clean; 3.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 102 MiB/s wr, 1.63k op/s 2024-02-09T17:25:15.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:15 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:25:15.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:15 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:25:15.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:15 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:25:15.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:15 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:25:15.730 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:15 smithi099 ceph-mon[39420]: pgmap v380: 17 pgs: 17 active+clean; 3.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 102 MiB/s wr, 1.63k op/s 2024-02-09T17:25:15.730 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:15 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:25:15.730 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:15 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:25:15.731 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:15 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:25:15.731 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:15 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:25:16.014 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:25:16.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:16.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:25:16.123 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:25:16.155 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:16.181 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:25:16.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:16.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:25:16.286 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:25:16.319 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:16.342 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:25:16.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:16.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:25:16.445 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 27 16 62107 62091 143.705 87.5625 0.00309979 0.00694005 2024-02-09T17:25:16.446 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:25:16.454 INFO:teuthology.orchestra.run.smithi099.stderr:no valid command found; 10 closest matches: 2024-02-09T17:25:16.454 INFO:teuthology.orchestra.run.smithi099.stderr:0 2024-02-09T17:25:16.454 INFO:teuthology.orchestra.run.smithi099.stderr:1 2024-02-09T17:25:16.454 INFO:teuthology.orchestra.run.smithi099.stderr:2 2024-02-09T17:25:16.455 INFO:teuthology.orchestra.run.smithi099.stderr:abort 2024-02-09T17:25:16.455 INFO:teuthology.orchestra.run.smithi099.stderr:assert 2024-02-09T17:25:16.455 INFO:teuthology.orchestra.run.smithi099.stderr:bench [] [] [] [] 2024-02-09T17:25:16.455 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs debug_inject_read_zeros 2024-02-09T17:25:16.455 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs files list 2024-02-09T17:25:16.455 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs stats 2024-02-09T17:25:16.455 INFO:teuthology.orchestra.run.smithi099.stderr:bluestore allocator dump block 2024-02-09T17:25:16.455 INFO:teuthology.orchestra.run.smithi099.stderr:Error EINVAL: invalid command 2024-02-09T17:25:16.462 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:25:16.462 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:25:16.472 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:16.490 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:25:16.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:16.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:25:16.593 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:25:16.633 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:16.663 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:25:16.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:16.667 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:25:16.768 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:25:16.826 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:16.900 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:25:16.900 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:25:16.900 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:25:16.900 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:25:16.900 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:25:16.901 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:25:16.901 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:25:16.901 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:25:16.901 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:25:16.901 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:25:16.901 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:25:16.903 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:25:16.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:16.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:25:17.009 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:25:17.058 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:17.141 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:25:17.150 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:25:17.151 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:25:17.151 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:25:17.151 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:25:17.151 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:25:17.151 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:25:17.151 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:25:17.151 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:25:17.151 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:25:17.151 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:25:17.151 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:25:17.151 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:25:17.176 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:17 smithi099 ceph-mon[39420]: pgmap v381: 17 pgs: 17 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 111 MiB/s wr, 1.77k op/s 2024-02-09T17:25:17.176 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:17 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2448761801' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:25:17.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:17 smithi179 ceph-mon[43975]: pgmap v381: 17 pgs: 17 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 111 MiB/s wr, 1.77k op/s 2024-02-09T17:25:17.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:17 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2448761801' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:25:17.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:17 smithi162 ceph-mon[44304]: pgmap v381: 17 pgs: 17 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 111 MiB/s wr, 1.77k op/s 2024-02-09T17:25:17.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:17 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2448761801' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:25:17.444 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 28 16 63586 63570 141.874 92.4375 0.00689921 0.00704345 2024-02-09T17:25:17.717 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:25:17.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:17.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:25:17.821 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:25:17.869 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:17.900 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:25:17.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:17.905 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:25:18.005 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:25:18.043 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:18.067 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:25:18.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:18.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:25:18.172 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:25:18.215 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:18.445 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 29 16 65381 65365 140.85 112.188 0.0112254 0.0070948 2024-02-09T17:25:18.586 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-09T17:25:18.586 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-02-09T17:25:18.587 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph tell osd.5 'trim stale osdmaps' 2024-02-09T17:25:18.673 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:25:18.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:18.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:25:18.778 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:25:18.812 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:18.837 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:25:18.839 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:18.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:25:18.940 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:25:18.977 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:19.004 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:25:19.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:19.009 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:25:19.110 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:25:19.174 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:19.400 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:19 smithi179 ceph-mon[43975]: pgmap v382: 17 pgs: 17 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 109 MiB/s wr, 1.74k op/s 2024-02-09T17:25:19.445 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 30 16 67051 67035 139.633 104.375 0.00347294 0.00715702 2024-02-09T17:25:19.454 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:19 smithi099 ceph-mon[39420]: pgmap v382: 17 pgs: 17 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 109 MiB/s wr, 1.74k op/s 2024-02-09T17:25:19.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:19 smithi162 ceph-mon[44304]: pgmap v382: 17 pgs: 17 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 109 MiB/s wr, 1.74k op/s 2024-02-09T17:25:19.936 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:25:19.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:19.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:25:20.042 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:25:20.080 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:20.116 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:25:20.119 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:20.119 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:25:20.220 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:25:20.257 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:20.281 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:25:20.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:20.285 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:25:20.386 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:25:20.424 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:20.445 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 31 16 68657 68641 138.366 100.375 0.00552327 0.0072224 2024-02-09T17:25:20.447 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:25:20.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:20.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:25:20.475 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:25:20.553 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:25:20.602 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:20.622 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:25:20.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:20.625 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:25:20.726 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:25:20.804 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:21.235 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:25:21.235 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 20, 2024-02-09T17:25:21.235 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 600, 2024-02-09T17:25:21.236 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [ 2024-02-09T17:25:21.236 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.236 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14154.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-02-09T17:25:21.236 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:18:58.291070+0000", 2024-02-09T17:25:21.236 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 382.85401434099998, 2024-02-09T17:25:21.237 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.52066127500000003, 2024-02-09T17:25:21.237 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:25:21.237 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:25:21.237 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:25:21.237 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14154", 2024-02-09T17:25:21.237 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.99:0/422221573", 2024-02-09T17:25:21.238 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 1 2024-02-09T17:25:21.238 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.238 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:25:21.238 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.238 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:25:21.238 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:18:58.291070+0000", 2024-02-09T17:25:21.238 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:25:21.238 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.238 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.238 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:25:21.239 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:18:58.291070+0000", 2024-02-09T17:25:21.239 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:25:21.239 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.239 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.239 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:25:21.239 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:18:58.291072+0000", 2024-02-09T17:25:21.240 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5659999999999999e-06 2024-02-09T17:25:21.240 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.240 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.240 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:25:21.240 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:18:58.291078+0000", 2024-02-09T17:25:21.240 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.1140000000000002e-06 2024-02-09T17:25:21.240 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.240 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.240 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:25:21.241 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:18:58.291087+0000", 2024-02-09T17:25:21.241 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.3270000000000007e-06 2024-02-09T17:25:21.241 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.241 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.241 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:25:21.241 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:18:58.291097+0000", 2024-02-09T17:25:21.241 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.7450000000000007e-06 2024-02-09T17:25:21.241 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.241 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.241 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:25:21.241 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:18:58.291156+0000", 2024-02-09T17:25:21.241 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.8801999999999997e-05 2024-02-09T17:25:21.241 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.241 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.242 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:25:21.242 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:18:58.291164+0000", 2024-02-09T17:25:21.242 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.5720000000000003e-06 2024-02-09T17:25:21.242 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.242 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.242 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:25:21.242 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:18:58.810527+0000", 2024-02-09T17:25:21.242 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.51936238300000004 2024-02-09T17:25:21.242 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.242 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.242 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:25:21.242 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:18:58.810689+0000", 2024-02-09T17:25:21.242 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00016201799999999999 2024-02-09T17:25:21.242 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.243 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.243 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,2", 2024-02-09T17:25:21.243 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:18:58.810888+0000", 2024-02-09T17:25:21.243 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000199605 2024-02-09T17:25:21.243 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.243 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.243 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:25:21.243 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:18:58.811578+0000", 2024-02-09T17:25:21.243 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00068957800000000004 2024-02-09T17:25:21.243 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.243 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.243 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:25:21.243 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:18:58.811611+0000", 2024-02-09T17:25:21.244 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.3541999999999998e-05 2024-02-09T17:25:21.244 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.244 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.244 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:25:21.244 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:18:58.811629+0000", 2024-02-09T17:25:21.244 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.7229999999999999e-05 2024-02-09T17:25:21.244 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.244 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.244 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:25:21.244 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:18:58.811691+0000", 2024-02-09T17:25:21.244 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.1864000000000003e-05 2024-02-09T17:25:21.244 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.244 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.245 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:25:21.245 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:18:58.811731+0000", 2024-02-09T17:25:21.245 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.0929000000000001e-05 2024-02-09T17:25:21.245 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:25:21.245 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:25:21.245 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:25:21.245 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.245 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.245 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:15741 2.9 2:950603f3:::benchmark_data_smithi179_77138_object15740:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-02-09T17:25:21.245 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:24:54.078345+0000", 2024-02-09T17:25:21.245 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 27.066739370000001, 2024-02-09T17:25:21.245 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.037667293999999997, 2024-02-09T17:25:21.245 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:25:21.245 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:25:21.246 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:25:21.246 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:25:21.246 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:25:21.246 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 15741 2024-02-09T17:25:21.246 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.246 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:25:21.246 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.246 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:25:21.246 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:54.078345+0000", 2024-02-09T17:25:21.246 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:25:21.246 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.246 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.246 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:25:21.247 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:54.078345+0000", 2024-02-09T17:25:21.247 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:25:21.247 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.247 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.247 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:25:21.247 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:54.078346+0000", 2024-02-09T17:25:21.247 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.6899999999999997e-07 2024-02-09T17:25:21.247 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.247 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.247 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:25:21.247 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:54.078449+0000", 2024-02-09T17:25:21.247 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000103056 2024-02-09T17:25:21.247 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.247 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.247 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:25:21.247 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:54.078461+0000", 2024-02-09T17:25:21.248 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.199e-05 2024-02-09T17:25:21.248 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.248 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.248 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:25:21.248 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:54.078469+0000", 2024-02-09T17:25:21.248 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.3920000000000003e-06 2024-02-09T17:25:21.248 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.248 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.248 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:25:21.248 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:54.082791+0000", 2024-02-09T17:25:21.248 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0043216110000000004 2024-02-09T17:25:21.248 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.248 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.249 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:25:21.249 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:54.082943+0000", 2024-02-09T17:25:21.249 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00015250100000000001 2024-02-09T17:25:21.249 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.249 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.249 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,7", 2024-02-09T17:25:21.249 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:54.083105+0000", 2024-02-09T17:25:21.249 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00016166800000000001 2024-02-09T17:25:21.249 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.249 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.249 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:25:21.249 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:54.084500+0000", 2024-02-09T17:25:21.249 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0013947320000000001 2024-02-09T17:25:21.249 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.249 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.250 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:25:21.250 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:54.084617+0000", 2024-02-09T17:25:21.250 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000117688 2024-02-09T17:25:21.250 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.250 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.250 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:25:21.250 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:54.115914+0000", 2024-02-09T17:25:21.250 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.031296168999999999 2024-02-09T17:25:21.250 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.250 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.250 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:25:21.250 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:54.115974+0000", 2024-02-09T17:25:21.250 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.05e-05 2024-02-09T17:25:21.250 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.250 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.251 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:25:21.251 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:24:54.116012+0000", 2024-02-09T17:25:21.251 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.8318000000000001e-05 2024-02-09T17:25:21.251 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:25:21.251 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:25:21.251 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:25:21.251 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.251 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.251 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:35127 2.9 2:9b938802:::benchmark_data_smithi179_77138_object35126:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-02-09T17:25:21.251 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:25:01.661497+0000", 2024-02-09T17:25:21.251 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 19.483587554, 2024-02-09T17:25:21.251 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.033805452, 2024-02-09T17:25:21.251 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:25:21.251 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:25:21.251 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:25:21.252 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:25:21.252 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:25:21.252 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 35127 2024-02-09T17:25:21.252 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.252 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:25:21.252 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.252 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:25:21.252 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.661497+0000", 2024-02-09T17:25:21.252 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:25:21.252 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.252 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.252 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:25:21.253 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.661497+0000", 2024-02-09T17:25:21.253 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:25:21.253 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.253 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.253 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:25:21.253 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.661497+0000", 2024-02-09T17:25:21.253 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.4900000000000001e-07 2024-02-09T17:25:21.253 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.253 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.253 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:25:21.253 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.661518+0000", 2024-02-09T17:25:21.253 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0361999999999999e-05 2024-02-09T17:25:21.253 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.253 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.254 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:25:21.254 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.661523+0000", 2024-02-09T17:25:21.254 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2689999999999999e-06 2024-02-09T17:25:21.254 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.254 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.254 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:25:21.254 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.661527+0000", 2024-02-09T17:25:21.254 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.5870000000000001e-06 2024-02-09T17:25:21.254 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.254 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.254 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:25:21.254 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.662687+0000", 2024-02-09T17:25:21.254 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0011608650000000001 2024-02-09T17:25:21.254 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.254 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.254 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:25:21.255 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.662766+0000", 2024-02-09T17:25:21.255 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.8250000000000005e-05 2024-02-09T17:25:21.255 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.255 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.255 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,7", 2024-02-09T17:25:21.255 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.662846+0000", 2024-02-09T17:25:21.255 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.0309000000000006e-05 2024-02-09T17:25:21.255 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.255 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.255 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:25:21.255 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.663857+0000", 2024-02-09T17:25:21.255 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0010111359999999999 2024-02-09T17:25:21.255 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.255 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.256 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:25:21.256 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.666594+0000", 2024-02-09T17:25:21.256 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.002737293 2024-02-09T17:25:21.256 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.256 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.256 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:25:21.259 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.695222+0000", 2024-02-09T17:25:21.259 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.028627792999999999 2024-02-09T17:25:21.259 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.259 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.259 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:25:21.259 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.695277+0000", 2024-02-09T17:25:21.260 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.5204999999999997e-05 2024-02-09T17:25:21.260 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.260 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.260 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:25:21.260 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.695302+0000", 2024-02-09T17:25:21.260 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.4933999999999999e-05 2024-02-09T17:25:21.260 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:25:21.260 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:25:21.260 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:25:21.260 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.260 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.261 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:62623 2.6 2:68a64834:::benchmark_data_smithi179_77138_object62622:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-02-09T17:25:21.261 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:25:16.831683+0000", 2024-02-09T17:25:21.261 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 4.3134017829999998, 2024-02-09T17:25:21.261 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.034620197999999998, 2024-02-09T17:25:21.261 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:25:21.261 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:25:21.261 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:25:21.261 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:25:21.261 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:25:21.261 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 62623 2024-02-09T17:25:21.261 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.261 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:25:21.261 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.261 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:25:21.262 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:16.831683+0000", 2024-02-09T17:25:21.262 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:25:21.262 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.262 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.262 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:25:21.262 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:16.831683+0000", 2024-02-09T17:25:21.262 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:25:21.262 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.262 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.262 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:25:21.262 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:16.831683+0000", 2024-02-09T17:25:21.262 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7000000000000005e-07 2024-02-09T17:25:21.262 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.263 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.263 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:25:21.263 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:16.831718+0000", 2024-02-09T17:25:21.263 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.4734999999999999e-05 2024-02-09T17:25:21.263 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.263 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.263 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:25:21.263 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:16.831725+0000", 2024-02-09T17:25:21.263 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.0389999999999999e-06 2024-02-09T17:25:21.263 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.263 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.263 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:25:21.263 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:16.831731+0000", 2024-02-09T17:25:21.263 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.5570000000000003e-06 2024-02-09T17:25:21.264 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.264 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.264 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:25:21.264 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:16.838420+0000", 2024-02-09T17:25:21.264 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0066896949999999998 2024-02-09T17:25:21.264 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.264 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.264 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:25:21.264 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:16.838567+0000", 2024-02-09T17:25:21.264 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000146203 2024-02-09T17:25:21.264 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.264 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.264 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,8", 2024-02-09T17:25:21.264 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:16.838726+0000", 2024-02-09T17:25:21.265 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000159631 2024-02-09T17:25:21.265 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.265 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.265 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:25:21.265 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:16.841752+0000", 2024-02-09T17:25:21.265 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0030253989999999998 2024-02-09T17:25:21.265 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.265 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.265 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:25:21.265 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:16.846001+0000", 2024-02-09T17:25:21.265 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0042492579999999997 2024-02-09T17:25:21.265 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.265 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.265 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:25:21.265 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:16.866214+0000", 2024-02-09T17:25:21.266 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.020212877000000001 2024-02-09T17:25:21.266 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.266 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.266 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:25:21.266 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:16.866266+0000", 2024-02-09T17:25:21.266 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1997999999999999e-05 2024-02-09T17:25:21.266 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.266 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.266 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:25:21.266 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:16.866303+0000", 2024-02-09T17:25:21.266 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.7236000000000001e-05 2024-02-09T17:25:21.266 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:25:21.266 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:25:21.266 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:25:21.266 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.267 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.267 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:65769 2.4 2:28adf554:::benchmark_data_smithi179_77138_object65768:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-02-09T17:25:21.267 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:25:18.666561+0000", 2024-02-09T17:25:21.267 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 2.4785234439999999, 2024-02-09T17:25:21.267 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.033618702, 2024-02-09T17:25:21.267 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:25:21.267 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:25:21.267 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:25:21.267 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:25:21.267 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:25:21.267 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 65769 2024-02-09T17:25:21.267 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.267 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:25:21.267 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.267 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:25:21.268 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:18.666561+0000", 2024-02-09T17:25:21.268 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:25:21.268 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.268 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.268 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:25:21.268 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:18.666561+0000", 2024-02-09T17:25:21.268 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:25:21.268 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.268 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.268 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:25:21.268 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:18.666562+0000", 2024-02-09T17:25:21.268 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.153e-06 2024-02-09T17:25:21.268 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.269 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.269 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:25:21.269 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:18.666617+0000", 2024-02-09T17:25:21.269 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.4784000000000003e-05 2024-02-09T17:25:21.269 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.269 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.269 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:25:21.269 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:18.666628+0000", 2024-02-09T17:25:21.269 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.116e-05 2024-02-09T17:25:21.269 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.269 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.269 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:25:21.269 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:18.666635+0000", 2024-02-09T17:25:21.269 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.2980000000000003e-06 2024-02-09T17:25:21.269 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.270 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.270 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:25:21.270 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:18.666705+0000", 2024-02-09T17:25:21.270 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.9165999999999995e-05 2024-02-09T17:25:21.270 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.270 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.270 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:25:21.270 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:18.666866+0000", 2024-02-09T17:25:21.270 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00016091200000000001 2024-02-09T17:25:21.270 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.270 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.270 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,10", 2024-02-09T17:25:21.270 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:18.667019+0000", 2024-02-09T17:25:21.270 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00015394200000000001 2024-02-09T17:25:21.270 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.271 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.271 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:25:21.271 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:18.677894+0000", 2024-02-09T17:25:21.271 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.010874959999999999 2024-02-09T17:25:21.271 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.271 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.271 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:25:21.271 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:18.681384+0000", 2024-02-09T17:25:21.271 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0034898490000000002 2024-02-09T17:25:21.271 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.271 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.271 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:25:21.271 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:18.700091+0000", 2024-02-09T17:25:21.271 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.018706699 2024-02-09T17:25:21.272 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.272 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.272 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:25:21.272 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:18.700143+0000", 2024-02-09T17:25:21.272 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2445999999999999e-05 2024-02-09T17:25:21.272 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.272 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.272 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:25:21.272 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:18.700180+0000", 2024-02-09T17:25:21.272 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.6332999999999999e-05 2024-02-09T17:25:21.272 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:25:21.272 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:25:21.272 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:25:21.273 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.273 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.273 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:67897 2.9 2:90d4b0b1:::benchmark_data_smithi179_77138_object67896:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-02-09T17:25:21.273 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:25:19.924303+0000", 2024-02-09T17:25:21.273 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 1.2207818880000001, 2024-02-09T17:25:21.273 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.033712029999999997, 2024-02-09T17:25:21.273 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:25:21.273 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:25:21.273 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:25:21.273 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:25:21.273 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:25:21.273 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 67897 2024-02-09T17:25:21.273 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.273 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:25:21.273 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.274 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:25:21.274 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:19.924303+0000", 2024-02-09T17:25:21.274 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:25:21.274 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.274 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.274 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:25:21.274 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:19.924303+0000", 2024-02-09T17:25:21.274 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:25:21.274 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.274 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.274 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:25:21.274 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:19.924304+0000", 2024-02-09T17:25:21.274 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.187e-06 2024-02-09T17:25:21.274 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.275 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.275 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:25:21.275 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:19.924332+0000", 2024-02-09T17:25:21.275 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.8442999999999999e-05 2024-02-09T17:25:21.275 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.275 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.275 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:25:21.275 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:19.924343+0000", 2024-02-09T17:25:21.275 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.119e-05 2024-02-09T17:25:21.275 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.275 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.275 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:25:21.275 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:19.924351+0000", 2024-02-09T17:25:21.276 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.2359999999999998e-06 2024-02-09T17:25:21.276 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.276 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.276 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:25:21.276 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:19.929492+0000", 2024-02-09T17:25:21.276 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0051412300000000001 2024-02-09T17:25:21.276 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.276 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.276 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:25:21.276 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:19.929623+0000", 2024-02-09T17:25:21.276 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013089500000000001 2024-02-09T17:25:21.276 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.276 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.276 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,7", 2024-02-09T17:25:21.277 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:19.929751+0000", 2024-02-09T17:25:21.277 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012777599999999999 2024-02-09T17:25:21.277 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.277 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.277 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:25:21.277 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:19.936151+0000", 2024-02-09T17:25:21.277 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0064002470000000004 2024-02-09T17:25:21.277 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.277 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.277 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:25:21.277 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:19.948017+0000", 2024-02-09T17:25:21.277 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.011866265000000001 2024-02-09T17:25:21.277 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.278 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.278 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:25:21.278 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:19.957933+0000", 2024-02-09T17:25:21.278 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0099157099999999995 2024-02-09T17:25:21.278 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.278 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.278 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:25:21.278 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:19.957981+0000", 2024-02-09T17:25:21.278 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.8059000000000002e-05 2024-02-09T17:25:21.278 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.278 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.278 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:25:21.278 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:19.958015+0000", 2024-02-09T17:25:21.278 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.3791999999999998e-05 2024-02-09T17:25:21.279 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:25:21.279 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:25:21.279 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:25:21.279 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.279 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.279 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:68200 2.4 2:2679e32a:::benchmark_data_smithi179_77138_object68199:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-02-09T17:25:21.279 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:25:20.117653+0000", 2024-02-09T17:25:21.279 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 1.027431014, 2024-02-09T17:25:21.279 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.21735884, 2024-02-09T17:25:21.279 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:25:21.279 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:25:21.279 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:25:21.279 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:25:21.279 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:25:21.280 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 68200 2024-02-09T17:25:21.280 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.280 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:25:21.280 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.280 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:25:21.280 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.117653+0000", 2024-02-09T17:25:21.280 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:25:21.280 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.280 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.280 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:25:21.280 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.117653+0000", 2024-02-09T17:25:21.280 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:25:21.280 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.280 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.281 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:25:21.281 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.117654+0000", 2024-02-09T17:25:21.281 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.2399999999999998e-07 2024-02-09T17:25:21.281 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.281 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.281 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:25:21.281 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.117749+0000", 2024-02-09T17:25:21.281 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.5290000000000004e-05 2024-02-09T17:25:21.281 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.281 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.281 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:25:21.281 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.117759+0000", 2024-02-09T17:25:21.281 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.6889999999999999e-06 2024-02-09T17:25:21.282 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.282 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.282 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:25:21.282 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.117766+0000", 2024-02-09T17:25:21.282 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.3780000000000003e-06 2024-02-09T17:25:21.282 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.282 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.282 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:25:21.282 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.118175+0000", 2024-02-09T17:25:21.282 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000408238 2024-02-09T17:25:21.282 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.282 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.282 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:25:21.282 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.118278+0000", 2024-02-09T17:25:21.283 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000102823 2024-02-09T17:25:21.283 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.283 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.283 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,10", 2024-02-09T17:25:21.283 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.118405+0000", 2024-02-09T17:25:21.283 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000127632 2024-02-09T17:25:21.283 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.283 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.283 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:25:21.283 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.123073+0000", 2024-02-09T17:25:21.283 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.004668312 2024-02-09T17:25:21.283 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.283 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.283 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:25:21.284 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.123123+0000", 2024-02-09T17:25:21.284 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.9954999999999999e-05 2024-02-09T17:25:21.284 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.284 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.284 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:25:21.284 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.334940+0000", 2024-02-09T17:25:21.284 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.211817015 2024-02-09T17:25:21.284 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.284 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.284 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:25:21.284 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.334976+0000", 2024-02-09T17:25:21.284 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.5244999999999999e-05 2024-02-09T17:25:21.284 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.284 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.285 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:25:21.285 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.335012+0000", 2024-02-09T17:25:21.285 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.6638999999999999e-05 2024-02-09T17:25:21.285 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:25:21.285 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:25:21.285 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:25:21.285 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.285 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.285 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:68207 2.4 2:2ed4664d:::benchmark_data_smithi179_77138_object68206:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-02-09T17:25:21.285 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:25:20.122888+0000", 2024-02-09T17:25:21.285 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 1.022196608, 2024-02-09T17:25:21.285 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.212263546, 2024-02-09T17:25:21.285 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:25:21.285 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:25:21.286 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:25:21.286 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:25:21.286 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:25:21.286 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 68207 2024-02-09T17:25:21.286 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.286 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:25:21.286 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.286 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:25:21.286 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.122888+0000", 2024-02-09T17:25:21.286 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:25:21.286 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.286 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.286 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:25:21.286 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.122888+0000", 2024-02-09T17:25:21.287 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:25:21.287 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.287 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.287 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:25:21.287 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.122889+0000", 2024-02-09T17:25:21.287 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4619999999999999e-06 2024-02-09T17:25:21.287 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.287 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.287 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:25:21.287 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.122973+0000", 2024-02-09T17:25:21.287 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.3955999999999994e-05 2024-02-09T17:25:21.287 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.287 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.287 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:25:21.288 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.122984+0000", 2024-02-09T17:25:21.288 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1109e-05 2024-02-09T17:25:21.288 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.288 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.288 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:25:21.288 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.122990+0000", 2024-02-09T17:25:21.288 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3140000000000003e-06 2024-02-09T17:25:21.288 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.288 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.288 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:25:21.288 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.123150+0000", 2024-02-09T17:25:21.288 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00016015499999999999 2024-02-09T17:25:21.289 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.289 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.289 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:25:21.289 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.123277+0000", 2024-02-09T17:25:21.289 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012703399999999999 2024-02-09T17:25:21.289 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.289 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.289 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,10", 2024-02-09T17:25:21.289 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.123415+0000", 2024-02-09T17:25:21.289 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013789999999999999 2024-02-09T17:25:21.289 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.289 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.289 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:25:21.289 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.125510+0000", 2024-02-09T17:25:21.290 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.002095501 2024-02-09T17:25:21.290 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.290 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.290 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:25:21.290 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.133075+0000", 2024-02-09T17:25:21.290 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0075643739999999996 2024-02-09T17:25:21.290 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.290 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.290 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:25:21.290 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.335060+0000", 2024-02-09T17:25:21.290 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.201985306 2024-02-09T17:25:21.290 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.290 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.290 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:25:21.291 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.335117+0000", 2024-02-09T17:25:21.291 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6991999999999999e-05 2024-02-09T17:25:21.291 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.291 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.291 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:25:21.291 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.335151+0000", 2024-02-09T17:25:21.291 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.4443000000000002e-05 2024-02-09T17:25:21.291 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:25:21.291 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:25:21.291 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:25:21.291 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.291 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.291 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:68223 2.4 2:2e2ca52e:::benchmark_data_smithi179_77138_object68222:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-02-09T17:25:21.291 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:25:20.133614+0000", 2024-02-09T17:25:21.292 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 1.011470662, 2024-02-09T17:25:21.292 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20952131800000001, 2024-02-09T17:25:21.292 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:25:21.292 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:25:21.292 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:25:21.292 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:25:21.292 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:25:21.292 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 68223 2024-02-09T17:25:21.292 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.292 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:25:21.292 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.292 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:25:21.292 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.133614+0000", 2024-02-09T17:25:21.292 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:25:21.292 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.293 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.293 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:25:21.293 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.133614+0000", 2024-02-09T17:25:21.293 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:25:21.293 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.293 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.293 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:25:21.293 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.133615+0000", 2024-02-09T17:25:21.293 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.457e-06 2024-02-09T17:25:21.293 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.293 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.293 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:25:21.293 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.133647+0000", 2024-02-09T17:25:21.293 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.1962000000000003e-05 2024-02-09T17:25:21.293 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.294 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.294 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:25:21.294 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.133657+0000", 2024-02-09T17:25:21.294 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.6770000000000007e-06 2024-02-09T17:25:21.294 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.294 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.294 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:25:21.294 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.133664+0000", 2024-02-09T17:25:21.294 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.7889999999999997e-06 2024-02-09T17:25:21.294 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.294 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.294 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:25:21.294 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.136487+0000", 2024-02-09T17:25:21.295 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.002823187 2024-02-09T17:25:21.295 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.295 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.295 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:25:21.295 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.136596+0000", 2024-02-09T17:25:21.295 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00010904999999999999 2024-02-09T17:25:21.295 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.295 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.295 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,10", 2024-02-09T17:25:21.295 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.136693+0000", 2024-02-09T17:25:21.295 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.6935000000000006e-05 2024-02-09T17:25:21.295 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.295 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.295 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:25:21.295 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.139831+0000", 2024-02-09T17:25:21.296 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0031380140000000002 2024-02-09T17:25:21.296 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.296 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.296 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:25:21.296 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.140618+0000", 2024-02-09T17:25:21.296 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00078751199999999998 2024-02-09T17:25:21.296 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.296 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.296 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:25:21.296 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.343024+0000", 2024-02-09T17:25:21.296 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20240518599999999 2024-02-09T17:25:21.296 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.296 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.297 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:25:21.297 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.343090+0000", 2024-02-09T17:25:21.297 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.6347999999999996e-05 2024-02-09T17:25:21.297 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.297 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.297 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:25:21.297 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.343135+0000", 2024-02-09T17:25:21.297 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.5201000000000003e-05 2024-02-09T17:25:21.297 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:25:21.297 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:25:21.297 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:25:21.297 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.297 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.297 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:68269 2.4 2:242c6c34:::benchmark_data_smithi179_77138_object68268:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-02-09T17:25:21.298 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:25:20.161075+0000", 2024-02-09T17:25:21.298 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.98400913499999998, 2024-02-09T17:25:21.298 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.18840509699999999, 2024-02-09T17:25:21.298 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:25:21.298 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:25:21.298 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:25:21.298 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:25:21.298 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:25:21.298 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 68269 2024-02-09T17:25:21.298 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.298 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:25:21.298 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.298 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:25:21.299 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.161075+0000", 2024-02-09T17:25:21.299 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:25:21.299 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.299 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.299 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:25:21.299 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.161075+0000", 2024-02-09T17:25:21.299 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:25:21.299 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.299 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.299 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:25:21.299 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.161076+0000", 2024-02-09T17:25:21.299 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.82e-07 2024-02-09T17:25:21.299 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.299 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.300 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:25:21.300 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.161104+0000", 2024-02-09T17:25:21.300 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.7895999999999999e-05 2024-02-09T17:25:21.300 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.300 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.300 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:25:21.300 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.161111+0000", 2024-02-09T17:25:21.300 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.9909999999999996e-06 2024-02-09T17:25:21.300 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.300 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.300 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:25:21.301 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.161117+0000", 2024-02-09T17:25:21.301 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7389999999999996e-06 2024-02-09T17:25:21.301 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.301 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.301 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:25:21.301 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.161166+0000", 2024-02-09T17:25:21.301 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.9446000000000001e-05 2024-02-09T17:25:21.301 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.301 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.301 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:25:21.301 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.161275+0000", 2024-02-09T17:25:21.301 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00010888200000000001 2024-02-09T17:25:21.301 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.301 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.302 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,10", 2024-02-09T17:25:21.302 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.163797+0000", 2024-02-09T17:25:21.302 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0025216909999999999 2024-02-09T17:25:21.302 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.302 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.302 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:25:21.302 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.167743+0000", 2024-02-09T17:25:21.302 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0039468070000000001 2024-02-09T17:25:21.302 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.302 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.302 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:25:21.302 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.167796+0000", 2024-02-09T17:25:21.302 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3047000000000002e-05 2024-02-09T17:25:21.303 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.303 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.303 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:25:21.303 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.349342+0000", 2024-02-09T17:25:21.303 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.18154573700000001 2024-02-09T17:25:21.303 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.303 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.303 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:25:21.303 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.349417+0000", 2024-02-09T17:25:21.303 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.4997e-05 2024-02-09T17:25:21.303 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.303 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.303 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:25:21.304 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.349480+0000", 2024-02-09T17:25:21.304 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.3281999999999995e-05 2024-02-09T17:25:21.304 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:25:21.304 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:25:21.304 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:25:21.304 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.304 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.304 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:68274 2.4 2:28e1b85d:::benchmark_data_smithi179_77138_object68273:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-02-09T17:25:21.304 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:25:20.162287+0000", 2024-02-09T17:25:21.304 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.98279747100000003, 2024-02-09T17:25:21.304 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.187309533, 2024-02-09T17:25:21.304 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:25:21.304 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:25:21.305 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:25:21.305 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:25:21.305 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:25:21.305 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 68274 2024-02-09T17:25:21.305 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.305 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:25:21.305 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.305 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:25:21.305 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.162287+0000", 2024-02-09T17:25:21.305 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:25:21.305 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.305 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.305 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:25:21.305 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.162287+0000", 2024-02-09T17:25:21.306 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:25:21.306 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.306 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.306 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:25:21.306 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.162288+0000", 2024-02-09T17:25:21.306 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.4799999999999997e-07 2024-02-09T17:25:21.306 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.306 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.306 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:25:21.306 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.162403+0000", 2024-02-09T17:25:21.306 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000115096 2024-02-09T17:25:21.306 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.306 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.306 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:25:21.306 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.162413+0000", 2024-02-09T17:25:21.307 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.5210000000000008e-06 2024-02-09T17:25:21.307 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.307 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.307 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:25:21.307 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.162418+0000", 2024-02-09T17:25:21.307 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6230000000000002e-06 2024-02-09T17:25:21.307 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.307 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.307 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:25:21.307 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.165950+0000", 2024-02-09T17:25:21.307 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0035316620000000001 2024-02-09T17:25:21.307 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.307 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.307 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:25:21.308 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.166048+0000", 2024-02-09T17:25:21.308 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.8302999999999997e-05 2024-02-09T17:25:21.308 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.308 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.308 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,10", 2024-02-09T17:25:21.308 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.166136+0000", 2024-02-09T17:25:21.308 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.7918999999999996e-05 2024-02-09T17:25:21.308 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.308 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.308 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:25:21.308 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.169470+0000", 2024-02-09T17:25:21.308 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0033337789999999998 2024-02-09T17:25:21.308 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.308 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.309 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:25:21.309 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.170179+0000", 2024-02-09T17:25:21.309 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00070950899999999999 2024-02-09T17:25:21.309 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.309 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.309 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:25:21.309 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.349526+0000", 2024-02-09T17:25:21.309 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.17934718099999999 2024-02-09T17:25:21.309 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.309 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.309 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:25:21.309 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.349577+0000", 2024-02-09T17:25:21.309 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.0039e-05 2024-02-09T17:25:21.309 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.310 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.310 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:25:21.310 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.349597+0000", 2024-02-09T17:25:21.310 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0052999999999999e-05 2024-02-09T17:25:21.310 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:25:21.310 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:25:21.310 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:25:21.310 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.310 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.310 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:68281 2.4 2:276f5379:::benchmark_data_smithi179_77138_object68280:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-02-09T17:25:21.310 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:25:20.166266+0000", 2024-02-09T17:25:21.310 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.97881827300000002, 2024-02-09T17:25:21.310 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.184377706, 2024-02-09T17:25:21.311 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:25:21.311 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:25:21.311 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:25:21.311 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:25:21.311 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:25:21.311 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 68281 2024-02-09T17:25:21.311 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.311 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:25:21.311 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.311 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:25:21.311 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.166266+0000", 2024-02-09T17:25:21.311 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:25:21.311 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.312 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.312 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:25:21.312 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.166266+0000", 2024-02-09T17:25:21.312 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:25:21.312 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.312 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.312 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:25:21.312 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.166267+0000", 2024-02-09T17:25:21.312 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1000000000000001e-06 2024-02-09T17:25:21.312 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.312 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.312 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:25:21.312 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.166369+0000", 2024-02-09T17:25:21.313 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00010189200000000001 2024-02-09T17:25:21.313 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.313 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.313 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:25:21.313 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.166378+0000", 2024-02-09T17:25:21.313 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.1309999999999995e-06 2024-02-09T17:25:21.313 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.313 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.313 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:25:21.313 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.166384+0000", 2024-02-09T17:25:21.313 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7189999999999998e-06 2024-02-09T17:25:21.313 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.313 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.314 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:25:21.314 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.167828+0000", 2024-02-09T17:25:21.314 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.001444243 2024-02-09T17:25:21.314 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.314 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.314 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:25:21.314 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.167938+0000", 2024-02-09T17:25:21.314 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000109234 2024-02-09T17:25:21.314 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.314 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.314 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,10", 2024-02-09T17:25:21.314 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.168026+0000", 2024-02-09T17:25:21.314 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.8317000000000002e-05 2024-02-09T17:25:21.314 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.315 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.315 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:25:21.315 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.169600+0000", 2024-02-09T17:25:21.315 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.001574102 2024-02-09T17:25:21.315 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.315 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.315 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:25:21.315 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.170413+0000", 2024-02-09T17:25:21.315 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00081325600000000005 2024-02-09T17:25:21.315 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.315 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.315 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:25:21.315 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.350557+0000", 2024-02-09T17:25:21.316 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.18014343499999999 2024-02-09T17:25:21.316 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.316 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.316 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:25:21.316 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.350610+0000", 2024-02-09T17:25:21.316 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3353000000000002e-05 2024-02-09T17:25:21.316 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.316 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.316 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:25:21.316 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.350644+0000", 2024-02-09T17:25:21.316 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.3924000000000002e-05 2024-02-09T17:25:21.316 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:25:21.316 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:25:21.316 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:25:21.317 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.317 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.317 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:68287 2.4 2:24b8d849:::benchmark_data_smithi179_77138_object68286:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-02-09T17:25:21.317 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:25:20.172962+0000", 2024-02-09T17:25:21.317 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.97212222199999998, 2024-02-09T17:25:21.317 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.177793757, 2024-02-09T17:25:21.317 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:25:21.317 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:25:21.317 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:25:21.317 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:25:21.317 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:25:21.317 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 68287 2024-02-09T17:25:21.317 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.318 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:25:21.318 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.318 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:25:21.318 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.172962+0000", 2024-02-09T17:25:21.318 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:25:21.318 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.318 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.318 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:25:21.318 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.172962+0000", 2024-02-09T17:25:21.318 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:25:21.318 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.318 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.318 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:25:21.318 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.172964+0000", 2024-02-09T17:25:21.318 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3400000000000001e-06 2024-02-09T17:25:21.319 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.319 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.319 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:25:21.319 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.173049+0000", 2024-02-09T17:25:21.319 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.5029000000000002e-05 2024-02-09T17:25:21.319 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.319 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.319 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:25:21.319 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.173072+0000", 2024-02-09T17:25:21.319 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.3683999999999999e-05 2024-02-09T17:25:21.319 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.319 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.319 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:25:21.320 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.173079+0000", 2024-02-09T17:25:21.320 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.7569999999999998e-06 2024-02-09T17:25:21.320 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.320 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.320 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:25:21.320 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.173133+0000", 2024-02-09T17:25:21.320 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3897999999999998e-05 2024-02-09T17:25:21.320 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.320 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.320 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:25:21.320 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.173277+0000", 2024-02-09T17:25:21.320 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000143969 2024-02-09T17:25:21.320 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.321 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.321 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,10", 2024-02-09T17:25:21.321 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.173451+0000", 2024-02-09T17:25:21.321 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00017415599999999999 2024-02-09T17:25:21.321 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.321 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.321 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:25:21.321 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.175861+0000", 2024-02-09T17:25:21.321 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0024101959999999999 2024-02-09T17:25:21.321 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.321 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.321 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:25:21.321 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.176234+0000", 2024-02-09T17:25:21.321 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00037265200000000001 2024-02-09T17:25:21.321 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.322 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.322 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:25:21.322 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.350681+0000", 2024-02-09T17:25:21.322 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.174447358 2024-02-09T17:25:21.322 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.322 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.322 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:25:21.322 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.350724+0000", 2024-02-09T17:25:21.322 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.3096000000000002e-05 2024-02-09T17:25:21.322 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.322 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.322 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:25:21.322 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.350756+0000", 2024-02-09T17:25:21.322 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.1622000000000003e-05 2024-02-09T17:25:21.323 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:25:21.323 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:25:21.323 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:25:21.323 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.323 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.323 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:68308 2.4 2:2d59d1f5:::benchmark_data_smithi179_77138_object68307:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-02-09T17:25:21.323 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:25:20.184820+0000", 2024-02-09T17:25:21.323 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.96026447100000001, 2024-02-09T17:25:21.323 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.16752827100000001, 2024-02-09T17:25:21.323 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:25:21.323 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:25:21.323 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:25:21.323 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:25:21.323 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:25:21.323 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 68308 2024-02-09T17:25:21.324 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.324 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:25:21.324 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.324 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:25:21.324 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.184820+0000", 2024-02-09T17:25:21.324 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:25:21.324 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.324 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.324 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:25:21.324 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.184820+0000", 2024-02-09T17:25:21.324 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:25:21.324 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.324 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.324 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:25:21.325 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.184821+0000", 2024-02-09T17:25:21.325 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.108e-06 2024-02-09T17:25:21.325 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.325 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.325 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:25:21.325 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.184916+0000", 2024-02-09T17:25:21.325 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.5186000000000006e-05 2024-02-09T17:25:21.325 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.325 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.325 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:25:21.325 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.184927+0000", 2024-02-09T17:25:21.325 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0355e-05 2024-02-09T17:25:21.325 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.325 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.326 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:25:21.326 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.184932+0000", 2024-02-09T17:25:21.326 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.5060000000000003e-06 2024-02-09T17:25:21.326 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.326 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.326 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:25:21.326 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.184983+0000", 2024-02-09T17:25:21.326 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.0590000000000002e-05 2024-02-09T17:25:21.326 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.326 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.326 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:25:21.326 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.185118+0000", 2024-02-09T17:25:21.326 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013488899999999999 2024-02-09T17:25:21.326 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.326 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.326 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,10", 2024-02-09T17:25:21.327 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.185269+0000", 2024-02-09T17:25:21.327 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00015179399999999999 2024-02-09T17:25:21.327 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.327 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.327 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:25:21.327 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.187313+0000", 2024-02-09T17:25:21.327 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0020433729999999998 2024-02-09T17:25:21.327 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.327 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.327 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:25:21.327 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.187640+0000", 2024-02-09T17:25:21.327 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00032729900000000003 2024-02-09T17:25:21.327 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.327 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.327 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:25:21.327 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.352256+0000", 2024-02-09T17:25:21.328 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.164616124 2024-02-09T17:25:21.328 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.328 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.328 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:25:21.328 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.352310+0000", 2024-02-09T17:25:21.328 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3897000000000003e-05 2024-02-09T17:25:21.328 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.328 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.328 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:25:21.328 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.352348+0000", 2024-02-09T17:25:21.328 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.8149999999999999e-05 2024-02-09T17:25:21.328 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:25:21.328 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:25:21.328 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:25:21.329 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.329 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.329 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:68349 2.4 2:2fb0a798:::benchmark_data_smithi179_77138_object68348:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-02-09T17:25:21.329 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:25:20.213725+0000", 2024-02-09T17:25:21.329 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.93135902299999995, 2024-02-09T17:25:21.329 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.14507325800000001, 2024-02-09T17:25:21.329 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:25:21.329 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:25:21.329 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:25:21.329 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:25:21.329 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:25:21.329 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 68349 2024-02-09T17:25:21.329 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.330 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:25:21.330 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.330 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:25:21.330 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.213725+0000", 2024-02-09T17:25:21.330 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:25:21.330 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.330 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.330 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:25:21.330 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.213725+0000", 2024-02-09T17:25:21.330 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:25:21.330 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.330 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.330 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:25:21.330 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.213727+0000", 2024-02-09T17:25:21.331 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.018e-06 2024-02-09T17:25:21.331 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.331 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.331 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:25:21.331 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.213804+0000", 2024-02-09T17:25:21.331 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.7585999999999999e-05 2024-02-09T17:25:21.331 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.331 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.331 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:25:21.331 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.213816+0000", 2024-02-09T17:25:21.331 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1894e-05 2024-02-09T17:25:21.331 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.331 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.331 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:25:21.331 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.213822+0000", 2024-02-09T17:25:21.332 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.0870000000000003e-06 2024-02-09T17:25:21.332 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.332 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.332 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:25:21.332 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.214075+0000", 2024-02-09T17:25:21.332 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00025306299999999999 2024-02-09T17:25:21.332 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.332 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.332 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:25:21.332 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.214192+0000", 2024-02-09T17:25:21.332 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000116883 2024-02-09T17:25:21.332 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.332 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.333 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,10", 2024-02-09T17:25:21.333 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.214351+0000", 2024-02-09T17:25:21.333 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000159202 2024-02-09T17:25:21.333 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.333 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.333 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:25:21.333 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.216486+0000", 2024-02-09T17:25:21.333 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.002134857 2024-02-09T17:25:21.333 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.333 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.333 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:25:21.333 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.217212+0000", 2024-02-09T17:25:21.333 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00072563800000000002 2024-02-09T17:25:21.333 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.334 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.334 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:25:21.334 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.358693+0000", 2024-02-09T17:25:21.334 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.14148118500000001 2024-02-09T17:25:21.334 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.334 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.334 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:25:21.334 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.358750+0000", 2024-02-09T17:25:21.334 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6966000000000003e-05 2024-02-09T17:25:21.334 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.334 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.334 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:25:21.334 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.358799+0000", 2024-02-09T17:25:21.334 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.8878999999999997e-05 2024-02-09T17:25:21.335 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:25:21.335 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:25:21.335 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:25:21.335 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.335 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.335 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:68384 2.4 2:2a5d00fc:::benchmark_data_smithi179_77138_object68383:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-02-09T17:25:21.335 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:25:20.239133+0000", 2024-02-09T17:25:21.335 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.90595197900000002, 2024-02-09T17:25:21.335 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.119817903, 2024-02-09T17:25:21.335 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:25:21.335 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:25:21.335 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:25:21.335 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:25:21.335 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:25:21.336 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 68384 2024-02-09T17:25:21.336 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.336 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:25:21.336 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.336 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:25:21.336 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.239133+0000", 2024-02-09T17:25:21.336 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:25:21.336 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.336 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.336 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:25:21.336 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.239133+0000", 2024-02-09T17:25:21.336 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:25:21.336 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.337 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.337 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:25:21.337 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.239134+0000", 2024-02-09T17:25:21.337 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4020000000000001e-06 2024-02-09T17:25:21.337 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.337 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.337 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:25:21.337 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.239182+0000", 2024-02-09T17:25:21.337 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.7698000000000003e-05 2024-02-09T17:25:21.337 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.337 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.337 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:25:21.337 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.239193+0000", 2024-02-09T17:25:21.337 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1657999999999999e-05 2024-02-09T17:25:21.337 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.338 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.338 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:25:21.338 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.239202+0000", 2024-02-09T17:25:21.338 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.4880000000000007e-06 2024-02-09T17:25:21.338 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.338 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.338 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:25:21.338 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.239260+0000", 2024-02-09T17:25:21.338 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.8254000000000002e-05 2024-02-09T17:25:21.338 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.338 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.338 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:25:21.338 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.239431+0000", 2024-02-09T17:25:21.338 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00017114499999999999 2024-02-09T17:25:21.338 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.339 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.339 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,10", 2024-02-09T17:25:21.339 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.239579+0000", 2024-02-09T17:25:21.339 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014734000000000001 2024-02-09T17:25:21.339 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.339 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.339 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:25:21.339 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.241451+0000", 2024-02-09T17:25:21.339 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.001872912 2024-02-09T17:25:21.339 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.339 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.339 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:25:21.339 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.242254+0000", 2024-02-09T17:25:21.339 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00080277600000000005 2024-02-09T17:25:21.340 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.340 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.340 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:25:21.340 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.358854+0000", 2024-02-09T17:25:21.340 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.116599999 2024-02-09T17:25:21.340 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.340 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.340 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:25:21.340 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.358914+0000", 2024-02-09T17:25:21.340 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.9392999999999999e-05 2024-02-09T17:25:21.340 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.340 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.341 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:25:21.341 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.358950+0000", 2024-02-09T17:25:21.341 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.6838000000000002e-05 2024-02-09T17:25:21.341 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:25:21.341 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:25:21.341 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:25:21.341 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.341 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.341 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:68406 2.4 2:297ec383:::benchmark_data_smithi179_77138_object68405:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-02-09T17:25:21.341 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:25:20.258648+0000", 2024-02-09T17:25:21.341 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.88643689199999998, 2024-02-09T17:25:21.341 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.10039636, 2024-02-09T17:25:21.341 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:25:21.341 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:25:21.342 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:25:21.342 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:25:21.342 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:25:21.342 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 68406 2024-02-09T17:25:21.342 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.342 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:25:21.342 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.342 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:25:21.342 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.258648+0000", 2024-02-09T17:25:21.342 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:25:21.342 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.342 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.342 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:25:21.342 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.258648+0000", 2024-02-09T17:25:21.343 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:25:21.343 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.343 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.343 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:25:21.343 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.258649+0000", 2024-02-09T17:25:21.343 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.333e-06 2024-02-09T17:25:21.343 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.343 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.343 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:25:21.343 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.258679+0000", 2024-02-09T17:25:21.343 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.0374000000000001e-05 2024-02-09T17:25:21.343 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.343 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.344 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:25:21.344 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.258694+0000", 2024-02-09T17:25:21.344 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4250000000000001e-05 2024-02-09T17:25:21.344 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.344 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.344 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:25:21.344 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.258702+0000", 2024-02-09T17:25:21.344 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.1519999999999992e-06 2024-02-09T17:25:21.344 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.344 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.344 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:25:21.344 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.258762+0000", 2024-02-09T17:25:21.345 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.0622000000000003e-05 2024-02-09T17:25:21.345 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.345 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.345 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:25:21.345 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.258883+0000", 2024-02-09T17:25:21.345 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012025500000000001 2024-02-09T17:25:21.345 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.345 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.345 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,10", 2024-02-09T17:25:21.345 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.258984+0000", 2024-02-09T17:25:21.345 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000101611 2024-02-09T17:25:21.345 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.345 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.345 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:25:21.346 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.260985+0000", 2024-02-09T17:25:21.346 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0020012749999999998 2024-02-09T17:25:21.346 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.346 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.346 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:25:21.346 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.261838+0000", 2024-02-09T17:25:21.346 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00085254600000000003 2024-02-09T17:25:21.346 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.346 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.346 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:25:21.346 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.358992+0000", 2024-02-09T17:25:21.346 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.097153844000000003 2024-02-09T17:25:21.346 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.347 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.347 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:25:21.347 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.359024+0000", 2024-02-09T17:25:21.347 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.1686999999999997e-05 2024-02-09T17:25:21.347 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.347 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.347 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:25:21.347 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.359044+0000", 2024-02-09T17:25:21.347 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0411000000000002e-05 2024-02-09T17:25:21.347 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:25:21.347 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:25:21.347 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:25:21.347 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.347 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.348 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:68418 2.4 2:2ddb1ba2:::benchmark_data_smithi179_77138_object68417:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-02-09T17:25:21.348 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:25:20.268002+0000", 2024-02-09T17:25:21.348 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.87708255999999996, 2024-02-09T17:25:21.348 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.091130806999999994, 2024-02-09T17:25:21.348 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:25:21.348 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:25:21.348 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:25:21.348 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:25:21.348 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:25:21.348 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 68418 2024-02-09T17:25:21.348 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.348 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:25:21.349 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.349 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:25:21.349 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.268002+0000", 2024-02-09T17:25:21.349 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:25:21.349 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.349 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.349 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:25:21.349 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.268002+0000", 2024-02-09T17:25:21.349 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:25:21.349 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.349 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.349 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:25:21.349 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.268003+0000", 2024-02-09T17:25:21.350 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3820000000000001e-06 2024-02-09T17:25:21.350 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.350 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.350 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:25:21.350 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.268104+0000", 2024-02-09T17:25:21.350 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00010044599999999999 2024-02-09T17:25:21.350 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.350 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.350 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:25:21.350 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.268115+0000", 2024-02-09T17:25:21.350 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0858e-05 2024-02-09T17:25:21.350 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.350 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.350 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:25:21.351 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.268120+0000", 2024-02-09T17:25:21.351 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7429999999999999e-06 2024-02-09T17:25:21.351 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.351 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.351 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:25:21.351 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.268170+0000", 2024-02-09T17:25:21.351 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.9719000000000003e-05 2024-02-09T17:25:21.351 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.351 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.351 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:25:21.351 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.268310+0000", 2024-02-09T17:25:21.351 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000139565 2024-02-09T17:25:21.351 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.351 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.352 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,10", 2024-02-09T17:25:21.352 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.268448+0000", 2024-02-09T17:25:21.352 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013839200000000001 2024-02-09T17:25:21.352 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.352 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.352 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:25:21.352 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.270498+0000", 2024-02-09T17:25:21.352 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0020500319999999998 2024-02-09T17:25:21.352 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.352 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.352 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:25:21.352 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.270857+0000", 2024-02-09T17:25:21.353 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00035938399999999997 2024-02-09T17:25:21.353 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.353 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.353 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:25:21.353 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.359083+0000", 2024-02-09T17:25:21.353 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.088225056999999996 2024-02-09T17:25:21.353 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.353 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.353 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:25:21.353 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.359114+0000", 2024-02-09T17:25:21.353 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.1319000000000003e-05 2024-02-09T17:25:21.353 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.353 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.353 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:25:21.354 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.359133+0000", 2024-02-09T17:25:21.354 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.891e-05 2024-02-09T17:25:21.354 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:25:21.354 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:25:21.354 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:25:21.354 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.354 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.354 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:68452 2.4 2:2657da92:::benchmark_data_smithi179_77138_object68451:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-02-09T17:25:21.354 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:25:20.303355+0000", 2024-02-09T17:25:21.354 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.84172957699999995, 2024-02-09T17:25:21.354 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.057089616000000003, 2024-02-09T17:25:21.354 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:25:21.354 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:25:21.354 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:25:21.354 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:25:21.355 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:25:21.355 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 68452 2024-02-09T17:25:21.355 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.355 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:25:21.355 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.355 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:25:21.355 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.303355+0000", 2024-02-09T17:25:21.355 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:25:21.355 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.355 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.355 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:25:21.355 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.303355+0000", 2024-02-09T17:25:21.355 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:25:21.355 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.356 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.356 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:25:21.356 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.303356+0000", 2024-02-09T17:25:21.356 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3820000000000001e-06 2024-02-09T17:25:21.356 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.356 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.356 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:25:21.356 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.303422+0000", 2024-02-09T17:25:21.356 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.5298999999999999e-05 2024-02-09T17:25:21.356 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.356 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.356 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:25:21.356 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.303435+0000", 2024-02-09T17:25:21.357 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2979999999999999e-05 2024-02-09T17:25:21.357 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.357 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.357 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:25:21.357 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.303442+0000", 2024-02-09T17:25:21.357 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.6939999999999995e-06 2024-02-09T17:25:21.357 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.357 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.357 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:25:21.357 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.303491+0000", 2024-02-09T17:25:21.357 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.8497e-05 2024-02-09T17:25:21.357 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.357 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.357 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:25:21.357 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.303593+0000", 2024-02-09T17:25:21.358 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000102379 2024-02-09T17:25:21.358 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.358 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.358 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,10", 2024-02-09T17:25:21.358 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.303689+0000", 2024-02-09T17:25:21.358 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.5575999999999994e-05 2024-02-09T17:25:21.358 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.358 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.358 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:25:21.358 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.305580+0000", 2024-02-09T17:25:21.358 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0018909840000000001 2024-02-09T17:25:21.358 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.358 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.358 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:25:21.359 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.305692+0000", 2024-02-09T17:25:21.359 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000112304 2024-02-09T17:25:21.359 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.359 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.359 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:25:21.359 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.360351+0000", 2024-02-09T17:25:21.359 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.054659174999999997 2024-02-09T17:25:21.359 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.359 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.359 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:25:21.359 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.360422+0000", 2024-02-09T17:25:21.359 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.1182999999999999e-05 2024-02-09T17:25:21.359 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.359 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.360 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:25:21.360 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.360445+0000", 2024-02-09T17:25:21.360 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2163000000000001e-05 2024-02-09T17:25:21.360 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:25:21.360 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:25:21.360 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:25:21.360 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.360 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.360 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:68468 2.4 2:28ce28d9:::benchmark_data_smithi179_77138_object68467:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-02-09T17:25:21.360 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:25:20.320467+0000", 2024-02-09T17:25:21.360 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.82461757400000002, 2024-02-09T17:25:21.360 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.040045005000000002, 2024-02-09T17:25:21.360 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:25:21.361 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:25:21.361 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:25:21.361 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:25:21.361 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:25:21.361 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 68468 2024-02-09T17:25:21.361 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.361 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:25:21.361 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.361 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:25:21.361 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.320467+0000", 2024-02-09T17:25:21.361 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:25:21.361 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.361 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.361 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:25:21.361 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.320467+0000", 2024-02-09T17:25:21.361 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:25:21.362 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.362 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.362 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:25:21.362 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.320468+0000", 2024-02-09T17:25:21.362 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.248e-06 2024-02-09T17:25:21.362 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.362 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.362 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:25:21.362 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.320569+0000", 2024-02-09T17:25:21.362 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00010098 2024-02-09T17:25:21.362 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.362 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.362 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:25:21.362 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.320580+0000", 2024-02-09T17:25:21.363 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0688e-05 2024-02-09T17:25:21.363 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.363 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.363 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:25:21.363 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.320586+0000", 2024-02-09T17:25:21.363 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.2040000000000002e-06 2024-02-09T17:25:21.363 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.363 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.363 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:25:21.363 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.320629+0000", 2024-02-09T17:25:21.363 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.3435e-05 2024-02-09T17:25:21.363 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.363 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.364 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:25:21.364 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.320721+0000", 2024-02-09T17:25:21.364 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.1262999999999997e-05 2024-02-09T17:25:21.364 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.364 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.364 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,10", 2024-02-09T17:25:21.364 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.320822+0000", 2024-02-09T17:25:21.364 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00010108299999999999 2024-02-09T17:25:21.364 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.364 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.364 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:25:21.364 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.322647+0000", 2024-02-09T17:25:21.364 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.001825112 2024-02-09T17:25:21.364 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.365 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.365 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:25:21.365 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.323304+0000", 2024-02-09T17:25:21.365 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00065691600000000001 2024-02-09T17:25:21.365 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.365 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.365 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:25:21.365 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.360469+0000", 2024-02-09T17:25:21.365 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.037164923000000002 2024-02-09T17:25:21.365 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.365 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.365 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:25:21.365 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.360495+0000", 2024-02-09T17:25:21.365 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.6367e-05 2024-02-09T17:25:21.365 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:21.365 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:21.366 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:25:21.366 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.360512+0000", 2024-02-09T17:25:21.366 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.6786000000000002e-05 2024-02-09T17:25:21.366 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:25:21.366 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:25:21.366 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:25:21.366 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:25:21.366 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:25:21.366 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:25:21.445 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 32 15 70151 70136 136.962 93.4375 0.00770434 0.00729751 2024-02-09T17:25:21.489 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:25:21.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:21 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:25:21.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:21 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:25:21.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:21 smithi179 ceph-mon[43975]: osdmap e126: 12 total, 12 up, 12 in 2024-02-09T17:25:21.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:21 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:25:21.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:21 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:25:21.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:21 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:25:21.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:21 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:25:21.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:21 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:25:21.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:21 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:25:21.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:21 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:25:21.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:21 smithi179 ceph-mon[43975]: osdmap e127: 12 total, 12 up, 12 in 2024-02-09T17:25:21.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:21 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:25:21.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:21 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:25:21.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:21 smithi162 ceph-mon[44304]: osdmap e126: 12 total, 12 up, 12 in 2024-02-09T17:25:21.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:25:21.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:25:21.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:25:21.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:25:21.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:25:21.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:25:21.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:25:21.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:21 smithi162 ceph-mon[44304]: osdmap e127: 12 total, 12 up, 12 in 2024-02-09T17:25:21.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:21 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:25:21.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:21 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:25:21.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:21 smithi099 ceph-mon[39420]: osdmap e126: 12 total, 12 up, 12 in 2024-02-09T17:25:21.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:21 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:25:21.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:21 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:25:21.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:21 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:25:21.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:21 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:25:21.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:21 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:25:21.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:21 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:25:21.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:21 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:25:21.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:21 smithi099 ceph-mon[39420]: osdmap e127: 12 total, 12 up, 12 in 2024-02-09T17:25:21.966 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-09T17:25:21.984 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:25:21.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:21.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:25:22.087 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:25:22.132 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:22.157 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:25:22.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:22.162 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:25:22.262 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:25:22.289 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:22.307 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:25:22.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:22.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:25:22.410 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:25:22.445 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 33 16 71175 71159 134.749 63.9375 0.00379704 0.00738511 2024-02-09T17:25:22.452 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:22.532 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:25:22.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:22.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:25:22.638 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:25:22.673 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:22.699 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:25:22.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:22.703 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:25:22.804 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:25:22.864 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:23.153 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:25:23.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:23.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:25:23.261 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:25:23.309 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:23.433 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is unset 2024-02-09T17:25:23.435 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:23 smithi099 ceph-mon[39420]: pgmap v386: 17 pgs: 17 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 106 MiB/s wr, 1.70k op/s 2024-02-09T17:25:23.435 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:23 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2173398918' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:25:23.435 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:23 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:25:23.445 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 34 16 72655 72639 133.507 92.5 0.00331371 0.0074854 2024-02-09T17:25:23.450 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:25:23.456 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:25:23.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:23.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:25:23.565 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:25:23.618 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:23.712 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:25:23.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:23.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:25:23.816 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:25:23.857 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:23 smithi179 ceph-mon[43975]: pgmap v386: 17 pgs: 17 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 106 MiB/s wr, 1.70k op/s 2024-02-09T17:25:23.858 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:23 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2173398918' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:25:23.858 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:23 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:25:23.860 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:23.884 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:25:23.888 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:23.888 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:25:23.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:23 smithi162 ceph-mon[44304]: pgmap v386: 17 pgs: 17 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 106 MiB/s wr, 1.70k op/s 2024-02-09T17:25:23.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:23 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2173398918' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:25:23.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:23 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:25:23.989 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:25:24.025 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:24.047 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:25:24.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:24.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:25:24.152 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:25:24.194 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:24.211 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:25:24.215 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:24.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:25:24.317 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:25:24.365 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:24.446 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 35 16 74141 74125 132.343 92.875 0.0097164 0.00755211 2024-02-09T17:25:24.625 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:24 smithi099 ceph-mon[39420]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:25:24.625 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:24 smithi099 ceph-mon[39420]: Cluster is now healthy 2024-02-09T17:25:24.625 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:24 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:25:24.625 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:24 smithi099 ceph-mon[39420]: osdmap e128: 12 total, 12 up, 12 in 2024-02-09T17:25:24.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:24 smithi179 ceph-mon[43975]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:25:24.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:24 smithi179 ceph-mon[43975]: Cluster is now healthy 2024-02-09T17:25:24.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:24 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:25:24.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:24 smithi179 ceph-mon[43975]: osdmap e128: 12 total, 12 up, 12 in 2024-02-09T17:25:24.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:24 smithi162 ceph-mon[44304]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:25:24.956 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:24 smithi162 ceph-mon[44304]: Cluster is now healthy 2024-02-09T17:25:24.956 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:24 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:25:24.956 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:24 smithi162 ceph-mon[44304]: osdmap e128: 12 total, 12 up, 12 in 2024-02-09T17:25:25.118 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:25:25.131 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:25:25.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:25.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:25:25.236 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:25:25.280 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:25.437 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:25:25.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:25.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:25:25.446 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 36 16 75705 75689 131.382 97.75 0.00524366 0.007607 2024-02-09T17:25:25.544 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:25:25.572 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:25.600 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:25:25.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:25.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:25:25.705 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:25:25.719 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:25 smithi099 ceph-mon[39420]: pgmap v388: 17 pgs: 17 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 100 MiB/s wr, 1.60k op/s 2024-02-09T17:25:25.739 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:25 smithi179 ceph-mon[43975]: pgmap v388: 17 pgs: 17 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 100 MiB/s wr, 1.60k op/s 2024-02-09T17:25:25.741 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:25.776 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:25:25.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:25.780 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:25:25.881 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:25:25.920 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:25.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:25 smithi162 ceph-mon[44304]: pgmap v388: 17 pgs: 17 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 100 MiB/s wr, 1.60k op/s 2024-02-09T17:25:25.945 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:25:25.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:25.951 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:25:26.053 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:25:26.103 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:26.181 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:25:26.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:26.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:25:26.294 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:25:26.326 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:26.346 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:25:26.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:26.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:25:26.446 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 37 16 77182 77166 130.326 92.3125 0.015171 0.00766848 2024-02-09T17:25:26.450 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:25:26.502 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:26.537 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:25:26.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:26.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:25:26.623 INFO:teuthology.orchestra.run.smithi099.stderr:no valid command found; 10 closest matches: 2024-02-09T17:25:26.623 INFO:teuthology.orchestra.run.smithi099.stderr:0 2024-02-09T17:25:26.623 INFO:teuthology.orchestra.run.smithi099.stderr:1 2024-02-09T17:25:26.623 INFO:teuthology.orchestra.run.smithi099.stderr:2 2024-02-09T17:25:26.623 INFO:teuthology.orchestra.run.smithi099.stderr:abort 2024-02-09T17:25:26.623 INFO:teuthology.orchestra.run.smithi099.stderr:assert 2024-02-09T17:25:26.623 INFO:teuthology.orchestra.run.smithi099.stderr:bench [] [] [] [] 2024-02-09T17:25:26.623 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs debug_inject_read_zeros 2024-02-09T17:25:26.624 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs files list 2024-02-09T17:25:26.624 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs stats 2024-02-09T17:25:26.624 INFO:teuthology.orchestra.run.smithi099.stderr:bluestore allocator dump block 2024-02-09T17:25:26.624 INFO:teuthology.orchestra.run.smithi099.stderr:Error EINVAL: invalid command 2024-02-09T17:25:26.649 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:25:26.675 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set noscrub 2024-02-09T17:25:26.681 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:26.709 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:25:26.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:26.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:25:26.813 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:25:26.850 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:26.878 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:25:26.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:26.881 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:25:26.982 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:25:27.021 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:27.056 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:25:27.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:27.062 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:25:27.163 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:25:27.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:27 smithi179 ceph-mon[43975]: osdmap e129: 12 total, 12 up, 12 in 2024-02-09T17:25:27.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:27 smithi179 ceph-mon[43975]: pgmap v390: 17 pgs: 17 active+clean; 4.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s 2024-02-09T17:25:27.217 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:27.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:27 smithi099 ceph-mon[39420]: osdmap e129: 12 total, 12 up, 12 in 2024-02-09T17:25:27.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:27 smithi099 ceph-mon[39420]: pgmap v390: 17 pgs: 17 active+clean; 4.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s 2024-02-09T17:25:27.325 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:25:27.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:27.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:25:27.362 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-09T17:25:27.363 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-02-09T17:25:27.363 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph tell osd.6 'trim stale osdmaps' 2024-02-09T17:25:27.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:27 smithi162 ceph-mon[44304]: osdmap e129: 12 total, 12 up, 12 in 2024-02-09T17:25:27.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:27 smithi162 ceph-mon[44304]: pgmap v390: 17 pgs: 17 active+clean; 4.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s 2024-02-09T17:25:27.435 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:25:27.446 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 38 16 78617 78601 129.257 89.6875 0.0115565 0.00773248 2024-02-09T17:25:27.475 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:27.500 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:25:27.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:27.503 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:25:27.604 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:25:27.632 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:27.655 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:25:27.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:27.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:25:27.762 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:25:27.794 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:27.819 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:25:27.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:27.824 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:25:27.925 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:25:27.983 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:28.149 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:25:28.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:28.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:25:28.251 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:25:28.251 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [ 2024-02-09T17:25:28.251 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:28.252 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:79648 2.e 2:7e9b5520:::benchmark_data_smithi179_77138_object79647:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-02-09T17:25:28.252 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:25:28.223190+0000", 2024-02-09T17:25:28.252 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.000313963, 2024-02-09T17:25:28.252 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00035281199999999998, 2024-02-09T17:25:28.252 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:25:28.252 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T17:25:28.252 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:25:28.252 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:25:28.252 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:25:28.252 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 79648 2024-02-09T17:25:28.252 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:28.252 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:25:28.252 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:28.253 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:25:28.253 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:28.223190+0000", 2024-02-09T17:25:28.253 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:25:28.253 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:28.253 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:28.253 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:25:28.253 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:28.223190+0000", 2024-02-09T17:25:28.253 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:25:28.253 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:28.253 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:28.253 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:25:28.253 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:28.223192+0000", 2024-02-09T17:25:28.253 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.8619999999999999e-06 2024-02-09T17:25:28.253 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:28.253 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:28.254 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:25:28.254 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:28.223231+0000", 2024-02-09T17:25:28.254 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.8940999999999999e-05 2024-02-09T17:25:28.254 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:28.254 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:28.254 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:25:28.254 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:28.223251+0000", 2024-02-09T17:25:28.254 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.9436000000000002e-05 2024-02-09T17:25:28.254 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:28.254 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:28.254 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:25:28.254 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:28.223261+0000", 2024-02-09T17:25:28.254 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.9119999999999993e-06 2024-02-09T17:25:28.254 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:28.255 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:28.255 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:25:28.255 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:28.223321+0000", 2024-02-09T17:25:28.255 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.9899999999999999e-05 2024-02-09T17:25:28.255 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:28.255 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:28.255 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:25:28.255 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:28.223442+0000", 2024-02-09T17:25:28.255 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000121834 2024-02-09T17:25:28.255 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:28.255 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:28.255 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,9", 2024-02-09T17:25:28.255 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:28.223557+0000", 2024-02-09T17:25:28.255 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00011496 2024-02-09T17:25:28.256 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:25:28.256 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:25:28.256 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:25:28.256 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:28.256 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:28.256 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:79649 2.7 e129/111 2:e679a550:::benchmark_data_smithi179_77138_object79648:head v 129'4872, mlcod=129'4870)", 2024-02-09T17:25:28.256 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:25:28.223297+0000", 2024-02-09T17:25:28.256 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.000207436, 2024-02-09T17:25:28.256 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000330589, 2024-02-09T17:25:28.256 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:25:28.256 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T17:25:28.256 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:25:28.256 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:28.257 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:25:28.257 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:28.223297+0000", 2024-02-09T17:25:28.257 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:25:28.257 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:28.257 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:28.257 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:25:28.257 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:28.223297+0000", 2024-02-09T17:25:28.257 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:25:28.257 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:28.257 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:28.257 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:25:28.257 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:28.223298+0000", 2024-02-09T17:25:28.257 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.6599999999999995e-07 2024-02-09T17:25:28.257 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:28.258 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:28.258 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:25:28.258 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:28.223336+0000", 2024-02-09T17:25:28.258 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.8275999999999997e-05 2024-02-09T17:25:28.258 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:28.258 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:28.258 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:25:28.258 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:28.223344+0000", 2024-02-09T17:25:28.258 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.6539999999999999e-06 2024-02-09T17:25:28.258 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:28.258 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:28.258 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:25:28.258 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:28.223349+0000", 2024-02-09T17:25:28.258 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1490000000000002e-06 2024-02-09T17:25:28.258 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:28.258 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:28.259 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:25:28.259 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:28.223411+0000", 2024-02-09T17:25:28.259 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.1712999999999997e-05 2024-02-09T17:25:28.259 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:28.259 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:28.259 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:25:28.259 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:28.223431+0000", 2024-02-09T17:25:28.259 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0188999999999999e-05 2024-02-09T17:25:28.259 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:25:28.259 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:25:28.259 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:25:28.259 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:28.259 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:28.259 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:79645 2.e 2:7015e498:::benchmark_data_smithi179_77138_object79644:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-02-09T17:25:28.259 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:25:28.213416+0000", 2024-02-09T17:25:28.260 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.010088666, 2024-02-09T17:25:28.260 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.010282416000000001, 2024-02-09T17:25:28.260 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:25:28.260 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "waiting for sub ops", 2024-02-09T17:25:28.260 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:25:28.260 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:25:28.260 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:25:28.260 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 79645 2024-02-09T17:25:28.260 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:28.260 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:25:28.260 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:28.260 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:25:28.260 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:28.213416+0000", 2024-02-09T17:25:28.260 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:25:28.261 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:28.261 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:28.261 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:25:28.261 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:28.213416+0000", 2024-02-09T17:25:28.261 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:25:28.261 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:28.261 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:28.261 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:25:28.261 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:28.213418+0000", 2024-02-09T17:25:28.261 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.8050000000000001e-06 2024-02-09T17:25:28.261 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:28.261 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:28.261 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:25:28.261 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:28.213459+0000", 2024-02-09T17:25:28.261 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.1564999999999999e-05 2024-02-09T17:25:28.262 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:28.262 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:28.262 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:25:28.262 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:28.213475+0000", 2024-02-09T17:25:28.262 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.6045e-05 2024-02-09T17:25:28.262 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:28.262 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:28.262 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:25:28.262 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:28.213484+0000", 2024-02-09T17:25:28.262 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.5350000000000004e-06 2024-02-09T17:25:28.262 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:28.262 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:28.262 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:25:28.262 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:28.213546+0000", 2024-02-09T17:25:28.262 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.2422000000000006e-05 2024-02-09T17:25:28.263 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:28.263 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:28.263 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:25:28.263 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:28.213661+0000", 2024-02-09T17:25:28.263 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000114666 2024-02-09T17:25:28.263 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:28.263 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:28.263 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,9", 2024-02-09T17:25:28.263 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:28.213797+0000", 2024-02-09T17:25:28.263 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013627499999999999 2024-02-09T17:25:28.263 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:28.263 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:28.263 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:25:28.263 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:28.216613+0000", 2024-02-09T17:25:28.263 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0028155770000000001 2024-02-09T17:25:28.263 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:25:28.264 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:25:28.264 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:25:28.264 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:28.217302+0000", 2024-02-09T17:25:28.264 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00068968299999999996 2024-02-09T17:25:28.264 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:25:28.264 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:25:28.264 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:25:28.264 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:25:28.264 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T17:25:28.264 INFO:teuthology.orchestra.run.smithi099.stdout: "num_ops": 3 2024-02-09T17:25:28.264 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:25:28.264 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:25:28.301 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:28.325 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:25:28.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:28.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:25:28.431 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:25:28.446 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 39 16 79963 79947 128.099 84.125 0.0123811 0.00780241 2024-02-09T17:25:28.466 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:28.495 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:25:28.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:28.501 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:25:28.602 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:25:28.642 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:28.931 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:25:29.446 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:2024-02-09T17:25:29.446306+0000 min lat: 0.00106674 max lat: 0.414288 avg lat: 0.00786354 2024-02-09T17:25:29.447 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-09T17:25:29.447 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 40 16 81374 81358 127.101 88.1875 0.0112774 0.00786354 2024-02-09T17:25:29.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:29 smithi099 ceph-mon[39420]: pgmap v391: 17 pgs: 17 active+clean; 4.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-02-09T17:25:29.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:29 smithi179 ceph-mon[43975]: pgmap v391: 17 pgs: 17 active+clean; 4.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-02-09T17:25:29.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:29 smithi162 ceph-mon[44304]: pgmap v391: 17 pgs: 17 active+clean; 4.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-02-09T17:25:29.727 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-09T17:25:29.728 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:25:29.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:29.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:25:29.833 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:25:29.916 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:30.231 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:25:30.231 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:25:30.231 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:25:30.231 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:25:30.235 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:25:30.283 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:25:30.286 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:25:30.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:30.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:25:30.326 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:25:30.354 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:25:30.391 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:25:30.394 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:25:30.440 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:25:30.446 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 41 16 82677 82661 125.987 81.4375 0.00302636 0.00793189 2024-02-09T17:25:30.449 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:30.487 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:25:30.518 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:25:30.538 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:25:30.538 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:25:30.538 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:25:30.538 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:25:30.538 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:25:30.538 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:25:30.538 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:25:30.538 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:25:30.538 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:25:30.538 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:25:30.538 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:25:30.538 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:25:30.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:30 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/505872952' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:25:30.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:30 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/505872952' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:25:30.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:30 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/505872952' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:25:31.447 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 42 16 84051 84035 125.032 85.875 0.0032613 0.00796871 2024-02-09T17:25:31.526 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:25:31.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:31.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:25:31.629 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:25:31.656 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:31.675 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:25:31.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:31 smithi162 ceph-mon[44304]: pgmap v392: 17 pgs: 17 active+clean; 4.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.56k op/s 2024-02-09T17:25:31.677 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:31 smithi179 ceph-mon[43975]: pgmap v392: 17 pgs: 17 active+clean; 4.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.56k op/s 2024-02-09T17:25:31.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:31.678 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:25:31.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:31 smithi099 ceph-mon[39420]: pgmap v392: 17 pgs: 17 active+clean; 4.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.56k op/s 2024-02-09T17:25:31.780 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:25:31.811 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:31.833 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:25:31.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:31.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:25:31.898 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:25:31.937 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:25:31.969 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:31.989 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:25:31.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:31.992 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:25:32.093 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:25:32.122 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:32.141 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:25:32.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:32.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:25:32.247 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:25:32.273 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:32.347 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:25:32.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:32.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:25:32.447 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 43 16 84051 84035 122.125 0 - 0.00796871 2024-02-09T17:25:32.450 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:25:32.478 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:32.974 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:25:33.227 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:25:33.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:33.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:25:33.331 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:25:33.364 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:33.384 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:25:33.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:33.388 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:25:33.433 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is set 2024-02-09T17:25:33.447 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 44 16 84051 84035 119.349 0 - 0.00796871 2024-02-09T17:25:33.468 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:25:33.489 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:25:33.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:33 smithi099 ceph-mon[39420]: pgmap v393: 17 pgs: 17 active+clean; 5.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 99 MiB/s wr, 1.59k op/s 2024-02-09T17:25:33.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:33 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4107054424' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:25:33.521 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:33.541 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:25:33.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:33.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:25:33.646 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:25:33.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:33 smithi179 ceph-mon[43975]: pgmap v393: 17 pgs: 17 active+clean; 5.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 99 MiB/s wr, 1.59k op/s 2024-02-09T17:25:33.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:33 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4107054424' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:25:33.675 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:33.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:33 smithi162 ceph-mon[44304]: pgmap v393: 17 pgs: 17 active+clean; 5.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 99 MiB/s wr, 1.59k op/s 2024-02-09T17:25:33.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:33 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4107054424' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:25:34.447 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 45 16 84051 84035 116.697 0 - 0.00796871 2024-02-09T17:25:34.456 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:25:34.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:34.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:25:34.563 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:25:34.595 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:34.615 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:25:34.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:34.618 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:25:34.719 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:25:34.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:34 smithi099 ceph-mon[39420]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:25:34.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:34 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4107054424' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:25:34.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:34 smithi099 ceph-mon[39420]: osdmap e130: 12 total, 12 up, 12 in 2024-02-09T17:25:34.747 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:34 smithi179 ceph-mon[43975]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:25:34.747 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:34 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4107054424' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:25:34.747 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:34 smithi179 ceph-mon[43975]: osdmap e130: 12 total, 12 up, 12 in 2024-02-09T17:25:34.749 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:34.769 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:25:34.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:34.772 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:25:34.873 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:25:34.917 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:34.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:34 smithi162 ceph-mon[44304]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:25:34.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:34 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4107054424' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:25:34.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:34 smithi162 ceph-mon[44304]: osdmap e130: 12 total, 12 up, 12 in 2024-02-09T17:25:35.447 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 46 15 84889 84874 115.3 13.1094 0.0144706 0.00866949 2024-02-09T17:25:35.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:35 smithi099 ceph-mon[39420]: pgmap v395: 17 pgs: 17 active+clean; 5.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.11k op/s 2024-02-09T17:25:35.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:35 smithi179 ceph-mon[43975]: pgmap v395: 17 pgs: 17 active+clean; 5.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.11k op/s 2024-02-09T17:25:35.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:35 smithi162 ceph-mon[44304]: pgmap v395: 17 pgs: 17 active+clean; 5.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.11k op/s 2024-02-09T17:25:36.447 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 47 16 86330 86314 114.762 90 0.00715544 0.00870986 2024-02-09T17:25:37.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:37 smithi179 ceph-mon[43975]: osdmap e131: 12 total, 12 up, 12 in 2024-02-09T17:25:37.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:37 smithi179 ceph-mon[43975]: pgmap v397: 17 pgs: 17 active+clean; 5.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-02-09T17:25:37.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:37 smithi162 ceph-mon[44304]: osdmap e131: 12 total, 12 up, 12 in 2024-02-09T17:25:37.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:37 smithi162 ceph-mon[44304]: pgmap v397: 17 pgs: 17 active+clean; 5.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-02-09T17:25:37.447 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 48 16 87769 87753 114.244 89.9375 0.00319653 0.0087479 2024-02-09T17:25:37.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:37 smithi099 ceph-mon[39420]: osdmap e131: 12 total, 12 up, 12 in 2024-02-09T17:25:37.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:37 smithi099 ceph-mon[39420]: pgmap v397: 17 pgs: 17 active+clean; 5.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-02-09T17:25:38.306 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:25:38.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:38.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:25:38.415 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:25:38.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:38 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:25:38.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:38 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:25:38.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:38 smithi179 ceph-mon[43975]: osdmap e132: 12 total, 12 up, 12 in 2024-02-09T17:25:38.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:38 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:25:38.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:38 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:25:38.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:38 smithi162 ceph-mon[44304]: osdmap e132: 12 total, 12 up, 12 in 2024-02-09T17:25:38.447 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 49 16 89144 89128 113.667 85.9375 0.00349711 0.00879201 2024-02-09T17:25:38.451 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:38 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:25:38.451 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:38 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:25:38.451 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:38 smithi099 ceph-mon[39420]: osdmap e132: 12 total, 12 up, 12 in 2024-02-09T17:25:38.454 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:39.348 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:39 smithi099 ceph-mon[39420]: pgmap v399: 17 pgs: 17 active+clean; 5.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s wr, 436 op/s 2024-02-09T17:25:39.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:39 smithi179 ceph-mon[43975]: pgmap v399: 17 pgs: 17 active+clean; 5.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s wr, 436 op/s 2024-02-09T17:25:39.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:39 smithi162 ceph-mon[44304]: pgmap v399: 17 pgs: 17 active+clean; 5.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s wr, 436 op/s 2024-02-09T17:25:39.447 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 50 16 90339 90323 112.887 74.6875 0.00354245 0.00885448 2024-02-09T17:25:40.448 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 51 16 91733 91717 112.382 87.125 0.013263 0.0088943 2024-02-09T17:25:41.448 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 52 16 92992 92976 111.734 78.6875 0.0126285 0.00894579 2024-02-09T17:25:41.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:41 smithi179 ceph-mon[43975]: pgmap v400: 17 pgs: 17 active+clean; 5.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s 2024-02-09T17:25:41.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:41 smithi179 ceph-mon[43975]: osdmap e133: 12 total, 12 up, 12 in 2024-02-09T17:25:41.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:41 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:25:41.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:41 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:25:41.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:41 smithi162 ceph-mon[44304]: pgmap v400: 17 pgs: 17 active+clean; 5.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s 2024-02-09T17:25:41.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:41 smithi162 ceph-mon[44304]: osdmap e133: 12 total, 12 up, 12 in 2024-02-09T17:25:41.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:41 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:25:41.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:41 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:25:41.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:41 smithi099 ceph-mon[39420]: pgmap v400: 17 pgs: 17 active+clean; 5.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s 2024-02-09T17:25:41.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:41 smithi099 ceph-mon[39420]: osdmap e133: 12 total, 12 up, 12 in 2024-02-09T17:25:41.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:41 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:25:41.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:41 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:25:42.448 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 53 16 94323 94307 111.195 83.1875 0.0135833 0.00898906 2024-02-09T17:25:43.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:43 smithi179 ceph-mon[43975]: osdmap e134: 12 total, 12 up, 12 in 2024-02-09T17:25:43.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:43 smithi179 ceph-mon[43975]: pgmap v403: 17 pgs: 17 active+clean; 5.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.11k op/s 2024-02-09T17:25:43.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:43 smithi162 ceph-mon[44304]: osdmap e134: 12 total, 12 up, 12 in 2024-02-09T17:25:43.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:43 smithi162 ceph-mon[44304]: pgmap v403: 17 pgs: 17 active+clean; 5.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.11k op/s 2024-02-09T17:25:43.448 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 54 16 95709 95693 110.74 86.625 0.00769689 0.0090264 2024-02-09T17:25:43.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:43 smithi099 ceph-mon[39420]: osdmap e134: 12 total, 12 up, 12 in 2024-02-09T17:25:43.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:43 smithi099 ceph-mon[39420]: pgmap v403: 17 pgs: 17 active+clean; 5.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.11k op/s 2024-02-09T17:25:44.254 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:25:44.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:44.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:25:44.294 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:44 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:25:44.295 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:44 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:25:44.295 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:44 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:25:44.295 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:44 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:25:44.295 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:44 smithi099 ceph-mon[39420]: osdmap e135: 12 total, 12 up, 12 in 2024-02-09T17:25:44.360 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:25:44.420 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:44.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:44 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:25:44.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:44 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:25:44.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:44 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:25:44.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:44 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:25:44.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:44 smithi179 ceph-mon[43975]: osdmap e135: 12 total, 12 up, 12 in 2024-02-09T17:25:44.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:44 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:25:44.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:44 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:25:44.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:44 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:25:44.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:44 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:25:44.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:44 smithi162 ceph-mon[44304]: osdmap e135: 12 total, 12 up, 12 in 2024-02-09T17:25:44.448 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 55 16 96878 96862 110.055 73.0625 0.0179277 0.00908236 2024-02-09T17:25:44.526 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:25:44.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:44.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:25:44.632 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:25:44.674 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:44.701 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:25:44.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:44.705 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:25:44.806 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:25:44.842 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:44.864 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:25:44.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:44.866 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:25:44.968 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:25:45.025 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:45.079 INFO:teuthology.orchestra.run.smithi099.stderr:no valid command found; 10 closest matches: 2024-02-09T17:25:45.079 INFO:teuthology.orchestra.run.smithi099.stderr:0 2024-02-09T17:25:45.079 INFO:teuthology.orchestra.run.smithi099.stderr:1 2024-02-09T17:25:45.079 INFO:teuthology.orchestra.run.smithi099.stderr:2 2024-02-09T17:25:45.079 INFO:teuthology.orchestra.run.smithi099.stderr:abort 2024-02-09T17:25:45.080 INFO:teuthology.orchestra.run.smithi099.stderr:assert 2024-02-09T17:25:45.080 INFO:teuthology.orchestra.run.smithi099.stderr:bench [] [] [] [] 2024-02-09T17:25:45.080 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs debug_inject_read_zeros 2024-02-09T17:25:45.080 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs files list 2024-02-09T17:25:45.080 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs stats 2024-02-09T17:25:45.080 INFO:teuthology.orchestra.run.smithi099.stderr:bluestore allocator dump block 2024-02-09T17:25:45.080 INFO:teuthology.orchestra.run.smithi099.stderr:Error EINVAL: invalid command 2024-02-09T17:25:45.295 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:45 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:25:45.295 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:45 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:25:45.295 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:45 smithi099 ceph-mon[39420]: pgmap v405: 17 pgs: 17 active+clean; 5.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.33k op/s 2024-02-09T17:25:45.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:45 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:25:45.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:45 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:25:45.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:45 smithi179 ceph-mon[43975]: pgmap v405: 17 pgs: 17 active+clean; 5.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.33k op/s 2024-02-09T17:25:45.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:45 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:25:45.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:45 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:25:45.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:45 smithi162 ceph-mon[44304]: pgmap v405: 17 pgs: 17 active+clean; 5.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.33k op/s 2024-02-09T17:25:45.448 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 56 16 98056 98040 109.404 73.625 0.00701601 0.00913664 2024-02-09T17:25:45.599 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:25:45.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:45.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:25:45.720 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:25:45.755 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:45.778 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:25:45.782 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:45.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:25:45.883 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:25:45.926 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:45.952 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:25:45.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:45.958 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:25:45.981 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-09T17:25:45.982 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-02-09T17:25:45.982 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph tell osd.7 'trim stale osdmaps' 2024-02-09T17:25:46.059 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:25:46.062 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:46 smithi162 ceph-mon[44304]: osdmap e136: 12 total, 12 up, 12 in 2024-02-09T17:25:46.062 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:46 smithi162 ceph-mon[44304]: osdmap e137: 12 total, 12 up, 12 in 2024-02-09T17:25:46.147 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:46.199 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:25:46.202 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:46.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:25:46.203 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:46 smithi099 ceph-mon[39420]: osdmap e136: 12 total, 12 up, 12 in 2024-02-09T17:25:46.204 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:46 smithi099 ceph-mon[39420]: osdmap e137: 12 total, 12 up, 12 in 2024-02-09T17:25:46.208 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:25:46.294 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-09T17:25:46.303 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:25:46.310 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:25:46.310 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:25:46.333 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:46.356 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:25:46.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:46.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:25:46.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:46 smithi179 ceph-mon[43975]: osdmap e136: 12 total, 12 up, 12 in 2024-02-09T17:25:46.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:46 smithi179 ceph-mon[43975]: osdmap e137: 12 total, 12 up, 12 in 2024-02-09T17:25:46.448 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 57 16 99256 99240 108.8 75 0.0259533 0.00918724 2024-02-09T17:25:46.460 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:25:46.589 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:47.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:47 smithi099 ceph-mon[39420]: pgmap v408: 17 pgs: 17 active+clean; 5.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.32k op/s 2024-02-09T17:25:47.367 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:25:47.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:47.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:25:47.405 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-09T17:25:47.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:47 smithi179 ceph-mon[43975]: pgmap v408: 17 pgs: 17 active+clean; 5.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.32k op/s 2024-02-09T17:25:47.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:47 smithi162 ceph-mon[44304]: pgmap v408: 17 pgs: 17 active+clean; 5.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.32k op/s 2024-02-09T17:25:47.448 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 58 15 100458 100443 108.221 75.1875 0.0155402 0.00923635 2024-02-09T17:25:47.475 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:25:47.551 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:47.670 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:25:47.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:47.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:25:47.780 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:25:47.813 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:47.846 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:25:47.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:47.852 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:25:47.953 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:25:47.987 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:48.011 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:25:48.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:48.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:25:48.117 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:25:48.167 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:48.213 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:25:48.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:48.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:25:48.261 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:25:48.284 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:25:48.292 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:25:48.329 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:25:48.370 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:48.398 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:25:48.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:48.404 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:25:48.448 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 59 16 101594 101578 107.589 70.9375 0.0101003 0.00929011 2024-02-09T17:25:48.505 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:25:48.550 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:48.578 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:25:48.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:48.580 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:25:48.682 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:25:48.746 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:49.448 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:2024-02-09T17:25:49.448124+0000 min lat: 0.00106674 max lat: 3.63891 avg lat: 0.00933597 2024-02-09T17:25:49.448 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-09T17:25:49.448 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 60 16 102809 102793 107.061 75.9375 0.00773157 0.00933597 2024-02-09T17:25:49.522 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:25:49.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:49.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:25:49.628 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:25:49.662 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:49.686 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:25:49.689 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:49 smithi162 ceph-mon[44304]: pgmap v409: 17 pgs: 17 active+clean; 5.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.08k op/s 2024-02-09T17:25:49.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:49.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:25:49.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:49 smithi099 ceph-mon[39420]: pgmap v409: 17 pgs: 17 active+clean; 5.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.08k op/s 2024-02-09T17:25:49.790 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:25:49.846 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:49.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:49 smithi179 ceph-mon[43975]: pgmap v409: 17 pgs: 17 active+clean; 5.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.08k op/s 2024-02-09T17:25:49.980 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:25:49.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:49.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:25:50.084 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:25:50.123 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:50.159 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:25:50.164 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:50.165 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:25:50.266 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:25:50.298 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:50.327 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:25:50.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:50.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:25:50.432 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:25:50.448 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 61 16 104066 104050 106.594 78.5625 0.0219772 0.00937675 2024-02-09T17:25:50.486 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:50.594 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:25:50.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:50.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:25:50.698 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:25:50.736 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:50.757 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:25:50.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:50.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:25:50.862 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:25:50.890 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:50.913 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:25:50.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:50.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:25:51.020 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:25:51.052 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:51.078 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:25:51.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:51.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:25:51.185 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:25:51.222 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:51.257 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:25:51.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:51.261 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:25:51.363 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:25:51.393 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:51.421 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:25:51.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:51.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:25:51.448 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 62 16 105147 105131 105.964 67.5625 0.00936878 0.009433 2024-02-09T17:25:51.526 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:25:51.584 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:51.613 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:25:51.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:51.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:25:51.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:51 smithi179 ceph-mon[43975]: pgmap v410: 17 pgs: 17 active+clean; 6.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 94 MiB/s wr, 1.50k op/s 2024-02-09T17:25:51.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:51 smithi162 ceph-mon[44304]: pgmap v410: 17 pgs: 17 active+clean; 6.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 94 MiB/s wr, 1.50k op/s 2024-02-09T17:25:51.718 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:25:51.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:51 smithi099 ceph-mon[39420]: pgmap v410: 17 pgs: 17 active+clean; 6.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 94 MiB/s wr, 1.50k op/s 2024-02-09T17:25:51.757 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:51.786 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:25:51.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:51.791 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:25:51.892 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:25:51.922 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:51.946 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:25:51.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:51.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:25:52.050 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:25:52.079 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:52.107 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:25:52.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:52.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:25:52.211 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:25:52.245 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:52.267 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:25:52.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:52.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:25:52.371 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:25:52.400 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:52.430 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:25:52.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:52.435 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:25:52.449 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 63 15 106285 106270 105.412 71.1875 0.0154096 0.00948213 2024-02-09T17:25:52.464 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:52 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/949569465' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:25:52.536 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:25:52.605 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:52.625 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:25:52.625 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:25:52.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:52 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/949569465' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:25:52.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:52 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/949569465' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:25:52.871 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:25:52.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:52.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:25:52.978 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:25:53.008 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:53.035 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:25:53.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:53.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:25:53.138 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:25:53.138 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:25:53.138 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:25:53.138 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:25:53.138 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:25:53.138 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:25:53.138 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:25:53.138 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:25:53.138 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:25:53.138 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:25:53.139 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:25:53.139 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:25:53.139 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:25:53.139 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:25:53.139 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:25:53.139 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:25:53.139 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:25:53.139 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:25:53.139 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:25:53.139 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:25:53.139 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:25:53.139 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:25:53.140 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:25:53.140 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:25:53.142 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:25:53.195 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:53.220 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:25:53.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:53.224 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:25:53.325 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:25:53.383 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:53.416 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:25:53.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:53.422 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:25:53.448 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 64 16 107475 107459 104.926 74.3125 0.0129739 0.00952618 2024-02-09T17:25:53.523 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:25:53.592 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:53.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:53 smithi099 ceph-mon[39420]: pgmap v411: 17 pgs: 17 active+clean; 6.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.35k op/s 2024-02-09T17:25:53.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:53 smithi162 ceph-mon[44304]: pgmap v411: 17 pgs: 17 active+clean; 6.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.35k op/s 2024-02-09T17:25:53.928 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:53 smithi179 ceph-mon[43975]: pgmap v411: 17 pgs: 17 active+clean; 6.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.35k op/s 2024-02-09T17:25:54.122 INFO:teuthology.orchestra.run.smithi099.stderr:no valid command found; 10 closest matches: 2024-02-09T17:25:54.122 INFO:teuthology.orchestra.run.smithi099.stderr:0 2024-02-09T17:25:54.122 INFO:teuthology.orchestra.run.smithi099.stderr:1 2024-02-09T17:25:54.122 INFO:teuthology.orchestra.run.smithi099.stderr:2 2024-02-09T17:25:54.122 INFO:teuthology.orchestra.run.smithi099.stderr:abort 2024-02-09T17:25:54.122 INFO:teuthology.orchestra.run.smithi099.stderr:assert 2024-02-09T17:25:54.122 INFO:teuthology.orchestra.run.smithi099.stderr:bench [] [] [] [] 2024-02-09T17:25:54.122 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs debug_inject_read_zeros 2024-02-09T17:25:54.122 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs files list 2024-02-09T17:25:54.123 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs stats 2024-02-09T17:25:54.123 INFO:teuthology.orchestra.run.smithi099.stderr:bluestore allocator dump block 2024-02-09T17:25:54.123 INFO:teuthology.orchestra.run.smithi099.stderr:Error EINVAL: invalid command 2024-02-09T17:25:54.449 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 65 16 108666 108650 104.457 74.4375 0.00669372 0.00956992 2024-02-09T17:25:54.785 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:25:54.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:54.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:25:54.819 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-09T17:25:54.820 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-02-09T17:25:54.820 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph tell osd.8 'trim stale osdmaps' 2024-02-09T17:25:54.893 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:25:54.936 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:54.965 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:25:54.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:54.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:25:55.070 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:25:55.106 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:55.132 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:25:55.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:55.135 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:25:55.236 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:25:55.246 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-09T17:25:55.291 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:55.449 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 66 16 109727 109711 103.879 66.3125 0.00491698 0.00962187 2024-02-09T17:25:55.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:55 smithi099 ceph-mon[39420]: pgmap v412: 17 pgs: 17 active+clean; 6.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 967 op/s 2024-02-09T17:25:55.906 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:25:55.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:55 smithi179 ceph-mon[43975]: pgmap v412: 17 pgs: 17 active+clean; 6.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 967 op/s 2024-02-09T17:25:55.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:55 smithi162 ceph-mon[44304]: pgmap v412: 17 pgs: 17 active+clean; 6.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 967 op/s 2024-02-09T17:25:55.967 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:25:55.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:55.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:25:56.074 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:25:56.113 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:56.142 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:25:56.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:56.147 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:25:56.248 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:25:56.294 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:56.325 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:25:56.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:56.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:25:56.431 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:25:56.449 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 67 16 110810 110794 103.339 67.6875 0.00593943 0.00966927 2024-02-09T17:25:56.463 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:56.486 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:25:56.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:56.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:25:56.494 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is set 2024-02-09T17:25:56.590 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:25:56.649 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:56.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:56 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2043577760' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:25:56.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:56 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2043577760' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:25:56.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:56 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2043577760' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:25:57.449 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 68 16 111916 111900 102.836 69.125 0.00753988 0.00972026 2024-02-09T17:25:57.513 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:25:57.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:57.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:25:57.622 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:25:57.674 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:57.749 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:57 smithi099 ceph-mon[39420]: pgmap v413: 17 pgs: 17 active+clean; 6.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.18k op/s 2024-02-09T17:25:57.749 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:57 smithi099 ceph-mon[39420]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:25:57.749 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:57 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2043577760' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:25:57.749 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:57 smithi099 ceph-mon[39420]: osdmap e138: 12 total, 12 up, 12 in 2024-02-09T17:25:57.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:57 smithi179 ceph-mon[43975]: pgmap v413: 17 pgs: 17 active+clean; 6.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.18k op/s 2024-02-09T17:25:57.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:57 smithi179 ceph-mon[43975]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:25:57.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:57 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2043577760' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:25:57.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:57 smithi179 ceph-mon[43975]: osdmap e138: 12 total, 12 up, 12 in 2024-02-09T17:25:57.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:57 smithi162 ceph-mon[44304]: pgmap v413: 17 pgs: 17 active+clean; 6.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.18k op/s 2024-02-09T17:25:57.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:57 smithi162 ceph-mon[44304]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:25:57.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:57 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2043577760' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:25:57.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:57 smithi162 ceph-mon[44304]: osdmap e138: 12 total, 12 up, 12 in 2024-02-09T17:25:57.940 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:25:58.050 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:25:58.054 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:58.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:25:58.077 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:25:58.155 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:25:58.214 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:58.379 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:25:58.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:58.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:25:58.449 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 69 16 113008 112992 102.334 68.25 0.0092285 0.00976783 2024-02-09T17:25:58.484 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:25:58.515 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:58.543 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:25:58.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:58.549 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:25:58.650 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:25:58.711 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:59.449 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 70 16 114094 114078 101.842 67.875 0.006658 0.00981503 2024-02-09T17:25:59.523 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset noscrub 2024-02-09T17:25:59.799 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:25:59.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:59.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:25:59.810 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:25:59 smithi099 ceph-mon[39420]: pgmap v415: 17 pgs: 17 active+clean; 6.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-02-09T17:25:59.904 INFO:teuthology.orchestra.run.smithi099.stderr:no valid command found; 10 closest matches: 2024-02-09T17:25:59.904 INFO:teuthology.orchestra.run.smithi099.stderr:0 2024-02-09T17:25:59.904 INFO:teuthology.orchestra.run.smithi099.stderr:1 2024-02-09T17:25:59.904 INFO:teuthology.orchestra.run.smithi099.stderr:2 2024-02-09T17:25:59.904 INFO:teuthology.orchestra.run.smithi099.stderr:abort 2024-02-09T17:25:59.905 INFO:teuthology.orchestra.run.smithi099.stderr:assert 2024-02-09T17:25:59.905 INFO:teuthology.orchestra.run.smithi099.stderr:bench [] [] [] [] 2024-02-09T17:25:59.905 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs debug_inject_read_zeros 2024-02-09T17:25:59.905 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs files list 2024-02-09T17:25:59.905 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs stats 2024-02-09T17:25:59.905 INFO:teuthology.orchestra.run.smithi099.stderr:bluestore allocator dump block 2024-02-09T17:25:59.905 INFO:teuthology.orchestra.run.smithi099.stderr:Error EINVAL: invalid command 2024-02-09T17:25:59.905 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:25:59.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:25:59 smithi179 ceph-mon[43975]: pgmap v415: 17 pgs: 17 active+clean; 6.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-02-09T17:25:59.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:25:59 smithi162 ceph-mon[44304]: pgmap v415: 17 pgs: 17 active+clean; 6.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-02-09T17:25:59.938 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:25:59.961 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:25:59.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:25:59.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:26:00.065 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:26:00.096 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:00.119 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:26:00.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:00.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:26:00.224 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:26:00.256 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:00.284 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:26:00.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:00.295 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:26:00.396 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:26:00.427 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:00.449 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 71 16 115219 115203 101.398 70.3125 0.0173775 0.00985819 2024-02-09T17:26:00.459 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:26:00.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:00.465 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:26:00.566 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:26:00.569 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-09T17:26:00.569 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-02-09T17:26:00.570 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph tell osd.9 'trim stale osdmaps' 2024-02-09T17:26:00.600 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:00.621 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:26:00.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:00.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:26:00.727 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:26:00.765 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:00.795 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:26:00.799 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:00 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:26:00.799 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:00 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:26:00.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:00.800 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:26:00.900 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:00 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:26:00.900 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:00 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:26:00.901 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:26:00.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:00 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:26:00.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:00 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:26:00.929 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:00.951 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:26:00.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:00.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:26:01.055 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:26:01.087 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:01.108 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:26:01.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:01.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:26:01.209 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:26:01.215 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:26:01.243 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:01.266 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:26:01.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:01.270 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:26:01.283 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:26:01.284 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [ 2024-02-09T17:26:01.284 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:01.284 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:116051 2.1 e139/111 2:8d11da54:::benchmark_data_smithi179_77138_object116050:head v 139'7126, mlcod=139'7124)", 2024-02-09T17:26:01.284 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:26:01.280415+0000", 2024-02-09T17:26:01.284 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.0017598710000000001, 2024-02-09T17:26:01.284 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.001793361, 2024-02-09T17:26:01.284 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:26:01.284 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T17:26:01.284 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:26:01.284 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:01.284 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:26:01.284 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.280415+0000", 2024-02-09T17:26:01.285 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:26:01.285 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:01.285 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:01.285 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:26:01.285 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.280415+0000", 2024-02-09T17:26:01.285 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:26:01.285 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:01.285 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:01.285 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:26:01.285 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.280416+0000", 2024-02-09T17:26:01.285 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.102e-06 2024-02-09T17:26:01.285 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:01.285 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:01.285 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:26:01.286 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.280437+0000", 2024-02-09T17:26:01.286 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0953000000000001e-05 2024-02-09T17:26:01.286 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:01.286 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:01.286 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:26:01.286 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.280447+0000", 2024-02-09T17:26:01.286 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0664e-05 2024-02-09T17:26:01.286 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:01.286 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:01.286 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:26:01.286 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.280454+0000", 2024-02-09T17:26:01.286 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.7819999999999996e-06 2024-02-09T17:26:01.286 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:01.286 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:01.286 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:26:01.287 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.280787+0000", 2024-02-09T17:26:01.287 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00033296999999999998 2024-02-09T17:26:01.287 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:01.287 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:01.287 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:26:01.287 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.280812+0000", 2024-02-09T17:26:01.287 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.5264000000000001e-05 2024-02-09T17:26:01.287 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:26:01.287 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:26:01.287 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:26:01.287 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:01.287 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:01.287 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:116055 2.a e139/111)", 2024-02-09T17:26:01.287 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:26:01.281221+0000", 2024-02-09T17:26:01.287 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.00095346799999999996, 2024-02-09T17:26:01.288 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0010485189999999999, 2024-02-09T17:26:01.288 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:26:01.288 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "queued for pg", 2024-02-09T17:26:01.288 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:26:01.288 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:01.288 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:26:01.288 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.281221+0000", 2024-02-09T17:26:01.288 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:26:01.288 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:01.288 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:01.288 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:26:01.288 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.281221+0000", 2024-02-09T17:26:01.288 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:26:01.288 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:01.289 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:01.289 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:26:01.289 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.281222+0000", 2024-02-09T17:26:01.289 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.1299999999999999e-07 2024-02-09T17:26:01.289 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:01.289 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:01.289 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:26:01.289 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.281341+0000", 2024-02-09T17:26:01.289 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00011934699999999999 2024-02-09T17:26:01.289 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:01.289 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:01.289 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:26:01.289 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.281352+0000", 2024-02-09T17:26:01.289 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0455000000000001e-05 2024-02-09T17:26:01.289 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:01.289 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:01.289 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:26:01.290 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.281358+0000", 2024-02-09T17:26:01.290 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.3550000000000001e-06 2024-02-09T17:26:01.290 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:26:01.290 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:26:01.290 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:26:01.290 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:01.290 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:01.290 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:116044 2.6 e139/111 2:6a019e36:::benchmark_data_smithi179_77138_object116043:head v 139'7290, mlcod=139'7288)", 2024-02-09T17:26:01.290 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:26:01.281457+0000", 2024-02-09T17:26:01.290 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.00071761299999999997, 2024-02-09T17:26:01.290 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00086089499999999998, 2024-02-09T17:26:01.290 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:26:01.290 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T17:26:01.290 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:26:01.290 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:01.290 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:26:01.291 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.281457+0000", 2024-02-09T17:26:01.291 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:26:01.291 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:01.291 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:01.291 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:26:01.291 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.281457+0000", 2024-02-09T17:26:01.291 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:26:01.291 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:01.291 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:01.291 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:26:01.291 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.281458+0000", 2024-02-09T17:26:01.291 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.4099999999999997e-07 2024-02-09T17:26:01.291 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:01.291 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:01.291 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:26:01.291 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.281480+0000", 2024-02-09T17:26:01.291 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2510999999999998e-05 2024-02-09T17:26:01.292 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:01.292 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:01.292 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:26:01.292 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.281491+0000", 2024-02-09T17:26:01.292 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0566e-05 2024-02-09T17:26:01.292 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:01.292 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:01.292 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:26:01.292 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.281497+0000", 2024-02-09T17:26:01.292 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.5649999999999998e-06 2024-02-09T17:26:01.292 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:01.292 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:01.292 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:26:01.292 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.281547+0000", 2024-02-09T17:26:01.292 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.9453000000000002e-05 2024-02-09T17:26:01.293 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:01.293 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:01.293 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:26:01.293 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.281565+0000", 2024-02-09T17:26:01.293 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.8400999999999999e-05 2024-02-09T17:26:01.293 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:26:01.293 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:26:01.293 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:26:01.293 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:26:01.293 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T17:26:01.293 INFO:teuthology.orchestra.run.smithi099.stdout: "num_ops": 3 2024-02-09T17:26:01.293 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:26:01.371 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:26:01.410 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:01.435 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:26:01.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:01.443 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:26:01.449 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 72 16 116240 116224 100.876 63.8125 0.00893455 0.00990846 2024-02-09T17:26:01.544 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:26:01.560 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:01 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:26:01.560 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:01 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:26:01.561 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:01 smithi099 ceph-mon[39420]: pgmap v416: 17 pgs: 17 active+clean; 6.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.13k op/s 2024-02-09T17:26:01.561 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:01 smithi099 ceph-mon[39420]: osdmap e139: 12 total, 12 up, 12 in 2024-02-09T17:26:01.612 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:01.830 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:26:01.839 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:01.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:26:01.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:01 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:26:01.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:01 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:26:01.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:01 smithi179 ceph-mon[43975]: pgmap v416: 17 pgs: 17 active+clean; 6.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.13k op/s 2024-02-09T17:26:01.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:01 smithi179 ceph-mon[43975]: osdmap e139: 12 total, 12 up, 12 in 2024-02-09T17:26:01.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:01 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:26:01.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:01 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:26:01.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:01 smithi162 ceph-mon[44304]: pgmap v416: 17 pgs: 17 active+clean; 6.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.13k op/s 2024-02-09T17:26:01.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:01 smithi162 ceph-mon[44304]: osdmap e139: 12 total, 12 up, 12 in 2024-02-09T17:26:01.940 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:26:01.975 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:01.996 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:26:01.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:01.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:26:02.100 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:26:02.167 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:02.296 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:26:02.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:02.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:26:02.409 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:26:02.449 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 73 16 117362 117346 100.455 70.125 0.00555239 0.00995086 2024-02-09T17:26:02.452 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:02.477 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:26:02.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:02.480 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:26:02.581 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:26:02.643 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:02.669 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:26:02.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:02.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:26:02.684 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-09T17:26:02.775 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:26:02.811 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:02.832 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:26:02.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:02.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:26:02.939 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:26:02.977 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:02.998 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:26:03.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:03.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:26:03.072 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:26:03.076 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:26:03.101 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:26:03.131 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:03.152 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:26:03.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:03.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:26:03.257 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:26:03.289 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:03.310 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:26:03.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:03.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:26:03.414 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:26:03.449 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 74 16 118407 118391 99.9796 65.3125 0.0126174 0.00999837 2024-02-09T17:26:03.468 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:03.698 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:03 smithi099 ceph-mon[39420]: pgmap v418: 17 pgs: 17 active+clean; 7.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-02-09T17:26:03.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:03 smithi179 ceph-mon[43975]: pgmap v418: 17 pgs: 17 active+clean; 7.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-02-09T17:26:03.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:03 smithi162 ceph-mon[44304]: pgmap v418: 17 pgs: 17 active+clean; 7.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-02-09T17:26:04.140 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:26:04.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:04.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:26:04.246 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:26:04.308 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:04.449 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 75 16 119489 119473 99.5482 67.625 0.00545082 0.0100415 2024-02-09T17:26:04.489 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:26:04.529 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:26:04.574 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:26:04.608 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:26:04.652 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:26:04.676 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:26:04.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:04.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:26:04.689 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:26:04.735 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:26:04.773 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:26:04.791 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:26:04.822 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:04 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/298609641' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:26:04.825 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:04.827 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:26:04.846 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:26:04.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:04.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:26:04.874 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:26:04.917 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:26:04.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:04 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/298609641' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:26:04.947 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:04 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/298609641' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:26:04.953 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:26:04.968 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:26:04.968 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:26:04.968 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:26:04.969 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:26:04.969 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:26:04.969 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:26:04.969 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:26:04.969 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:26:04.969 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:26:04.969 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:26:04.969 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:26:04.969 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:26:04.969 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:26:04.998 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:05.357 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:26:05.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:05.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:26:05.449 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 76 16 120395 120379 98.9834 56.625 0.0234027 0.0100982 2024-02-09T17:26:05.464 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:26:05.511 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:05.543 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:26:05.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:05.547 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:26:05.648 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:26:05.707 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:05.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:05 smithi179 ceph-mon[43975]: pgmap v419: 17 pgs: 17 active+clean; 7.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 946 op/s 2024-02-09T17:26:05.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:05 smithi162 ceph-mon[44304]: pgmap v419: 17 pgs: 17 active+clean; 7.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 946 op/s 2024-02-09T17:26:05.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:05 smithi099 ceph-mon[39420]: pgmap v419: 17 pgs: 17 active+clean; 7.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 946 op/s 2024-02-09T17:26:06.335 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:26:06.372 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:26:06.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:06.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:26:06.450 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 77 16 121491 121475 98.5874 68.5 0.0108011 0.0101386 2024-02-09T17:26:06.484 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:26:06.526 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:06.647 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:26:06.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:06.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:26:06.751 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:26:06.788 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:06.805 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:26:06.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:06.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:26:06.915 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:26:06.956 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:06.978 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:26:06.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:06.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:26:07.083 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:26:07.116 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:07.133 INFO:teuthology.orchestra.run.smithi099.stderr:no valid command found; 10 closest matches: 2024-02-09T17:26:07.133 INFO:teuthology.orchestra.run.smithi099.stderr:0 2024-02-09T17:26:07.133 INFO:teuthology.orchestra.run.smithi099.stderr:1 2024-02-09T17:26:07.133 INFO:teuthology.orchestra.run.smithi099.stderr:2 2024-02-09T17:26:07.133 INFO:teuthology.orchestra.run.smithi099.stderr:abort 2024-02-09T17:26:07.133 INFO:teuthology.orchestra.run.smithi099.stderr:assert 2024-02-09T17:26:07.133 INFO:teuthology.orchestra.run.smithi099.stderr:bench [] [] [] [] 2024-02-09T17:26:07.133 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs debug_inject_read_zeros 2024-02-09T17:26:07.133 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs files list 2024-02-09T17:26:07.133 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs stats 2024-02-09T17:26:07.133 INFO:teuthology.orchestra.run.smithi099.stderr:bluestore allocator dump block 2024-02-09T17:26:07.134 INFO:teuthology.orchestra.run.smithi099.stderr:Error EINVAL: invalid command 2024-02-09T17:26:07.142 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:26:07.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:07.146 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:26:07.246 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:26:07.274 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:07.296 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:26:07.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:07.301 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:26:07.402 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:26:07.435 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:07.450 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 78 16 122460 122444 98.0999 60.5625 0.0114492 0.0101896 2024-02-09T17:26:07.460 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:26:07.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:07.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:26:07.564 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:26:07.594 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:07.620 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:26:07.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:07.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:26:07.719 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:07 smithi099 ceph-mon[39420]: pgmap v420: 17 pgs: 17 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.03k op/s 2024-02-09T17:26:07.727 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:26:07.780 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:07.826 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:26:07.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:07.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:26:07.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:07 smithi179 ceph-mon[43975]: pgmap v420: 17 pgs: 17 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.03k op/s 2024-02-09T17:26:07.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:07 smithi162 ceph-mon[44304]: pgmap v420: 17 pgs: 17 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.03k op/s 2024-02-09T17:26:07.948 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:26:07.990 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:08.023 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:26:08.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:08.025 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:26:08.126 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:26:08.159 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:08.195 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:26:08.199 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:08.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:26:08.301 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:26:08.341 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:08.368 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:26:08.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:08.374 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:26:08.450 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 79 16 123539 123523 97.7117 67.4375 0.0106739 0.0102303 2024-02-09T17:26:08.474 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:26:08.508 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:08.539 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:26:08.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:08.545 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:26:08.646 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:26:08.675 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:08.700 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:26:08.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:08.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:26:08.762 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-09T17:26:08.763 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-02-09T17:26:08.763 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph tell osd.10 'trim stale osdmaps' 2024-02-09T17:26:08.803 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:26:08.836 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:08.862 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:26:08.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:08.868 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:26:08.969 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:26:09.002 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:09.027 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:26:09.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:09.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:26:09.132 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:26:09.185 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:09.332 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:26:09.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:09.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:26:09.446 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:26:09.456 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:2024-02-09T17:26:09.455739+0000 min lat: 0.00106674 max lat: 3.63891 avg lat: 0.010277 2024-02-09T17:26:09.456 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-09T17:26:09.456 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 80 16 124543 124527 97.2674 62.75 0.0184566 0.010277 2024-02-09T17:26:09.488 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:09.526 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:26:09.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:09.533 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:26:09.633 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:26:09.635 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is unset 2024-02-09T17:26:09.667 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:09.690 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:26:09.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:09.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:26:09.795 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:26:09.823 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:09.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:09 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:26:09.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:09 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:26:09.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:09 smithi179 ceph-mon[43975]: pgmap v421: 17 pgs: 17 active+clean; 7.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.08k op/s 2024-02-09T17:26:09.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:09 smithi179 ceph-mon[43975]: osdmap e140: 12 total, 12 up, 12 in 2024-02-09T17:26:09.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:09 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1344802786' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:26:09.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:09 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:26:09.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:09 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:26:09.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:09 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:26:09.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:09 smithi162 ceph-mon[44304]: pgmap v421: 17 pgs: 17 active+clean; 7.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.08k op/s 2024-02-09T17:26:09.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:09 smithi162 ceph-mon[44304]: osdmap e140: 12 total, 12 up, 12 in 2024-02-09T17:26:09.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:09 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1344802786' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:26:09.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:09 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:26:09.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:09 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:26:09.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:09 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:26:09.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:09 smithi099 ceph-mon[39420]: pgmap v421: 17 pgs: 17 active+clean; 7.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.08k op/s 2024-02-09T17:26:09.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:09 smithi099 ceph-mon[39420]: osdmap e140: 12 total, 12 up, 12 in 2024-02-09T17:26:09.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:09 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1344802786' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:26:09.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:09 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:26:10.298 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:26:10.456 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 81 16 125573 125557 96.8613 64.375 0.0126738 0.0103203 2024-02-09T17:26:10.612 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:10 smithi099 ceph-mon[39420]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:26:10.612 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:10 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:26:10.612 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:10 smithi099 ceph-mon[39420]: osdmap e141: 12 total, 12 up, 12 in 2024-02-09T17:26:10.612 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:10 smithi099 ceph-mon[39420]: 2.3 scrub starts 2024-02-09T17:26:10.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:10 smithi179 ceph-mon[43975]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:26:10.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:10 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:26:10.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:10 smithi179 ceph-mon[43975]: osdmap e141: 12 total, 12 up, 12 in 2024-02-09T17:26:10.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:10 smithi179 ceph-mon[43975]: 2.3 scrub starts 2024-02-09T17:26:10.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:10 smithi162 ceph-mon[44304]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:26:10.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:10 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:26:10.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:10 smithi162 ceph-mon[44304]: osdmap e141: 12 total, 12 up, 12 in 2024-02-09T17:26:10.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:10 smithi162 ceph-mon[44304]: 2.3 scrub starts 2024-02-09T17:26:11.456 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 82 16 126581 126565 96.4483 63 0.0162199 0.0103646 2024-02-09T17:26:11.697 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:26:11.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:11.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:26:11.803 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:26:11.871 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:11.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:11 smithi179 ceph-mon[43975]: pgmap v424: 17 pgs: 17 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-02-09T17:26:11.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:11 smithi162 ceph-mon[44304]: pgmap v424: 17 pgs: 17 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-02-09T17:26:11.933 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:26:11.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:11.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:26:11.938 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:11 smithi099 ceph-mon[39420]: pgmap v424: 17 pgs: 17 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-02-09T17:26:12.039 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:26:12.097 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:12.205 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:26:12.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:12.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:26:12.310 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:26:12.356 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:12.386 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:26:12.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:12.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:26:12.456 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 83 16 127357 127341 95.8706 48.5 0.0103669 0.0104257 2024-02-09T17:26:12.490 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:26:12.532 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:12.564 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:26:12.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:12.572 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:26:12.674 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:26:12.691 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:26:12.691 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:26:12.717 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:12.746 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:26:12.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:12.750 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:26:12.851 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:26:12.917 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:12 smithi099 ceph-mon[39420]: 2.7 scrub starts 2024-02-09T17:26:12.917 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:12 smithi099 ceph-mon[39420]: 2.1 scrub starts 2024-02-09T17:26:12.917 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:12 smithi099 ceph-mon[39420]: 2.b scrub starts 2024-02-09T17:26:12.917 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:12 smithi099 ceph-mon[39420]: 2.3 scrub ok 2024-02-09T17:26:12.917 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:12 smithi099 ceph-mon[39420]: 2.2 scrub starts 2024-02-09T17:26:12.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:12 smithi179 ceph-mon[43975]: 2.7 scrub starts 2024-02-09T17:26:12.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:12 smithi179 ceph-mon[43975]: 2.1 scrub starts 2024-02-09T17:26:12.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:12 smithi179 ceph-mon[43975]: 2.b scrub starts 2024-02-09T17:26:12.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:12 smithi179 ceph-mon[43975]: 2.3 scrub ok 2024-02-09T17:26:12.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:12 smithi179 ceph-mon[43975]: 2.2 scrub starts 2024-02-09T17:26:12.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:12 smithi162 ceph-mon[44304]: 2.7 scrub starts 2024-02-09T17:26:12.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:12 smithi162 ceph-mon[44304]: 2.1 scrub starts 2024-02-09T17:26:12.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:12 smithi162 ceph-mon[44304]: 2.b scrub starts 2024-02-09T17:26:12.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:12 smithi162 ceph-mon[44304]: 2.3 scrub ok 2024-02-09T17:26:12.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:12 smithi162 ceph-mon[44304]: 2.2 scrub starts 2024-02-09T17:26:12.932 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:13.401 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:26:13.401 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:26:13.410 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:13.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:26:13.445 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-09T17:26:13.456 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 84 16 128188 128172 95.3476 51.9375 0.00537329 0.0104834 2024-02-09T17:26:13.466 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:26:13.512 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:26:13.580 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:13.670 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:13 smithi099 ceph-mon[39420]: 2.6 scrub starts 2024-02-09T17:26:13.671 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:13 smithi099 ceph-mon[39420]: pgmap v425: 17 pgs: 17 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-02-09T17:26:13.671 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:13 smithi099 ceph-mon[39420]: 2.7 scrub ok 2024-02-09T17:26:13.671 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:13 smithi099 ceph-mon[39420]: 2.b scrub ok 2024-02-09T17:26:13.700 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:26:13.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:13.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:26:13.806 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:26:13.863 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:13.894 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:26:13.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:13.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:26:13.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:13 smithi179 ceph-mon[43975]: 2.6 scrub starts 2024-02-09T17:26:13.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:13 smithi179 ceph-mon[43975]: pgmap v425: 17 pgs: 17 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-02-09T17:26:13.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:13 smithi179 ceph-mon[43975]: 2.7 scrub ok 2024-02-09T17:26:13.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:13 smithi179 ceph-mon[43975]: 2.b scrub ok 2024-02-09T17:26:13.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:13 smithi162 ceph-mon[44304]: 2.6 scrub starts 2024-02-09T17:26:13.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:13 smithi162 ceph-mon[44304]: pgmap v425: 17 pgs: 17 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-02-09T17:26:13.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:13 smithi162 ceph-mon[44304]: 2.7 scrub ok 2024-02-09T17:26:13.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:13 smithi162 ceph-mon[44304]: 2.b scrub ok 2024-02-09T17:26:13.999 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:26:14.075 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:14.114 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:26:14.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:14.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:26:14.221 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:26:14.255 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:14.281 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:26:14.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:14.283 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:26:14.384 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:26:14.456 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 85 16 128987 128971 94.8134 49.9375 0.0113823 0.0105429 2024-02-09T17:26:14.480 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:14.571 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-09T17:26:14.611 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:26:14.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:14.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:26:14.718 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:26:14.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:14 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:26:14.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:14 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:26:14.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:14 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:26:14.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:14 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:26:14.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:14 smithi099 ceph-mon[39420]: 2.2 scrub ok 2024-02-09T17:26:14.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:14 smithi099 ceph-mon[39420]: osdmap e142: 12 total, 12 up, 12 in 2024-02-09T17:26:14.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:14 smithi099 ceph-mon[39420]: 2.1 scrub ok 2024-02-09T17:26:14.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:14 smithi099 ceph-mon[39420]: pgmap v427: 17 pgs: 17 active+clean; 7.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.03k op/s 2024-02-09T17:26:14.771 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:14.804 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:26:14.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:14.809 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:26:14.910 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:26:14.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:14 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:26:14.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:14 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:26:14.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:14 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:26:14.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:14 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:26:14.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:14 smithi162 ceph-mon[44304]: 2.2 scrub ok 2024-02-09T17:26:14.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:14 smithi162 ceph-mon[44304]: osdmap e142: 12 total, 12 up, 12 in 2024-02-09T17:26:14.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:14 smithi162 ceph-mon[44304]: 2.1 scrub ok 2024-02-09T17:26:14.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:14 smithi162 ceph-mon[44304]: pgmap v427: 17 pgs: 17 active+clean; 7.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.03k op/s 2024-02-09T17:26:14.997 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:15.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:14 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:26:15.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:14 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:26:15.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:14 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:26:15.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:14 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:26:15.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:14 smithi179 ceph-mon[43975]: 2.2 scrub ok 2024-02-09T17:26:15.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:14 smithi179 ceph-mon[43975]: osdmap e142: 12 total, 12 up, 12 in 2024-02-09T17:26:15.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:14 smithi179 ceph-mon[43975]: 2.1 scrub ok 2024-02-09T17:26:15.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:14 smithi179 ceph-mon[43975]: pgmap v427: 17 pgs: 17 active+clean; 7.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.03k op/s 2024-02-09T17:26:15.456 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 86 16 129829 129813 94.3229 52.625 0.0164777 0.0105967 2024-02-09T17:26:15.467 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:26:15.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:15.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:26:15.573 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:26:15.613 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:15.643 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:26:15.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:15.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:26:15.708 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:15 smithi099 ceph-mon[39420]: 2.e scrub starts 2024-02-09T17:26:15.709 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:15 smithi099 ceph-mon[39420]: 2.8 scrub starts 2024-02-09T17:26:15.709 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:15 smithi099 ceph-mon[39420]: 2.d scrub starts 2024-02-09T17:26:15.709 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:15 smithi099 ceph-mon[39420]: 2.6 scrub ok 2024-02-09T17:26:15.757 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:26:15.816 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:15.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:15 smithi162 ceph-mon[44304]: 2.e scrub starts 2024-02-09T17:26:15.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:15 smithi162 ceph-mon[44304]: 2.8 scrub starts 2024-02-09T17:26:15.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:15 smithi162 ceph-mon[44304]: 2.d scrub starts 2024-02-09T17:26:15.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:15 smithi162 ceph-mon[44304]: 2.6 scrub ok 2024-02-09T17:26:16.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:15 smithi179 ceph-mon[43975]: 2.e scrub starts 2024-02-09T17:26:16.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:15 smithi179 ceph-mon[43975]: 2.8 scrub starts 2024-02-09T17:26:16.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:15 smithi179 ceph-mon[43975]: 2.d scrub starts 2024-02-09T17:26:16.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:15 smithi179 ceph-mon[43975]: 2.6 scrub ok 2024-02-09T17:26:16.179 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:26:16.186 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:16.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:26:16.288 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:26:16.364 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:16.456 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 87 16 130685 130669 93.8536 53.5 0.0136469 0.010651 2024-02-09T17:26:16.459 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:26:16.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:16.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:26:16.564 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:26:16.616 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:16.656 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:26:16.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:16.661 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:26:16.762 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:26:16.799 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:16.832 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:26:16.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:16.837 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:26:16.841 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:26:16.903 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:26:16.938 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:26:16.947 INFO:teuthology.orchestra.run.smithi099.stderr:no valid command found; 10 closest matches: 2024-02-09T17:26:16.947 INFO:teuthology.orchestra.run.smithi099.stderr:0 2024-02-09T17:26:16.948 INFO:teuthology.orchestra.run.smithi099.stderr:1 2024-02-09T17:26:16.948 INFO:teuthology.orchestra.run.smithi099.stderr:2 2024-02-09T17:26:16.948 INFO:teuthology.orchestra.run.smithi099.stderr:abort 2024-02-09T17:26:16.948 INFO:teuthology.orchestra.run.smithi099.stderr:assert 2024-02-09T17:26:16.948 INFO:teuthology.orchestra.run.smithi099.stderr:bench [] [] [] [] 2024-02-09T17:26:16.948 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs debug_inject_read_zeros 2024-02-09T17:26:16.948 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs files list 2024-02-09T17:26:16.948 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs stats 2024-02-09T17:26:16.948 INFO:teuthology.orchestra.run.smithi099.stderr:bluestore allocator dump block 2024-02-09T17:26:16.948 INFO:teuthology.orchestra.run.smithi099.stderr:Error EINVAL: invalid command 2024-02-09T17:26:16.950 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:26:16.974 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:16.991 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:26:17.001 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:26:17.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:17.005 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:26:17.015 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:26:17.015 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:26:17.046 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:26:17.079 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:26:17.106 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:26:17.119 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:26:17.141 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:17.150 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:26:17.171 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:26:17.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:17.173 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:26:17.189 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:26:17.223 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:26:17.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:17 smithi099 ceph-mon[39420]: 2.8 scrub ok 2024-02-09T17:26:17.248 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:17 smithi099 ceph-mon[39420]: pgmap v428: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s 2024-02-09T17:26:17.248 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:17 smithi099 ceph-mon[39420]: 2.0 scrub starts 2024-02-09T17:26:17.248 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:17 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4079883828' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:26:17.274 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:26:17.287 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:26:17.296 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:26:17.303 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:26:17.303 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:26:17.303 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:26:17.303 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:26:17.303 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:26:17.303 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:26:17.303 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:26:17.303 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:26:17.303 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:26:17.303 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:26:17.303 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:26:17.303 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:26:17.314 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:17 smithi162 ceph-mon[44304]: 2.8 scrub ok 2024-02-09T17:26:17.314 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:17 smithi162 ceph-mon[44304]: pgmap v428: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s 2024-02-09T17:26:17.314 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:17 smithi162 ceph-mon[44304]: 2.0 scrub starts 2024-02-09T17:26:17.314 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:17 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4079883828' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:26:17.318 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:17.348 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:26:17.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:17.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:26:17.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:17 smithi179 ceph-mon[43975]: 2.8 scrub ok 2024-02-09T17:26:17.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:17 smithi179 ceph-mon[43975]: pgmap v428: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s 2024-02-09T17:26:17.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:17 smithi179 ceph-mon[43975]: 2.0 scrub starts 2024-02-09T17:26:17.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:17 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4079883828' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:26:17.453 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:26:17.461 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 88 16 131546 131530 93.3942 53.8125 0.0123936 0.0107032 2024-02-09T17:26:17.494 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:17.522 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:26:17.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:17.526 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:26:17.627 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:26:17.685 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:17.715 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:26:17.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:17.718 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:26:17.819 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:26:17.883 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:17.916 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:26:17.923 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:17.924 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:26:18.025 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:26:18.079 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:18.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:18 smithi179 ceph-mon[43975]: 2.d scrub ok 2024-02-09T17:26:18.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:18 smithi179 ceph-mon[43975]: 2.e scrub ok 2024-02-09T17:26:18.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:18 smithi162 ceph-mon[44304]: 2.d scrub ok 2024-02-09T17:26:18.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:18 smithi162 ceph-mon[44304]: 2.e scrub ok 2024-02-09T17:26:18.461 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 89 16 132539 132523 93.0422 62.0625 0.0419372 0.0107425 2024-02-09T17:26:18.502 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:18 smithi099 ceph-mon[39420]: 2.d scrub ok 2024-02-09T17:26:18.502 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:18 smithi099 ceph-mon[39420]: 2.e scrub ok 2024-02-09T17:26:19.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:19 smithi179 ceph-mon[43975]: 2.0 scrub ok 2024-02-09T17:26:19.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:19 smithi179 ceph-mon[43975]: 2.9 scrub starts 2024-02-09T17:26:19.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:19 smithi179 ceph-mon[43975]: pgmap v429: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 721 op/s 2024-02-09T17:26:19.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:19 smithi179 ceph-mon[43975]: 2.a scrub starts 2024-02-09T17:26:19.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:19 smithi162 ceph-mon[44304]: 2.0 scrub ok 2024-02-09T17:26:19.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:19 smithi162 ceph-mon[44304]: 2.9 scrub starts 2024-02-09T17:26:19.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:19 smithi162 ceph-mon[44304]: pgmap v429: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 721 op/s 2024-02-09T17:26:19.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:19 smithi162 ceph-mon[44304]: 2.a scrub starts 2024-02-09T17:26:19.461 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Total time run: 90.0083 2024-02-09T17:26:19.461 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Total writes made: 133405 2024-02-09T17:26:19.461 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Write size: 65536 2024-02-09T17:26:19.461 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Object size: 65536 2024-02-09T17:26:19.461 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Bandwidth (MB/sec): 92.6338 2024-02-09T17:26:19.461 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Stddev Bandwidth: 45.61 2024-02-09T17:26:19.462 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Max bandwidth (MB/sec): 262.438 2024-02-09T17:26:19.462 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Min bandwidth (MB/sec): 0 2024-02-09T17:26:19.462 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Average IOPS: 1482 2024-02-09T17:26:19.462 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Stddev IOPS: 729.775 2024-02-09T17:26:19.462 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Max IOPS: 4199 2024-02-09T17:26:19.462 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Min IOPS: 0 2024-02-09T17:26:19.462 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Average Latency(s): 0.010792 2024-02-09T17:26:19.462 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Stddev Latency(s): 0.040965 2024-02-09T17:26:19.462 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Max latency(s): 3.63891 2024-02-09T17:26:19.462 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Min latency(s): 0.00106674 2024-02-09T17:26:19.466 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Cleaning up (deleting benchmark objects) 2024-02-09T17:26:19.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:19 smithi099 ceph-mon[39420]: 2.0 scrub ok 2024-02-09T17:26:19.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:19 smithi099 ceph-mon[39420]: 2.9 scrub starts 2024-02-09T17:26:19.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:19 smithi099 ceph-mon[39420]: pgmap v429: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 721 op/s 2024-02-09T17:26:19.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:19 smithi099 ceph-mon[39420]: 2.a scrub starts 2024-02-09T17:26:20.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:20 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:26:20.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:20 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:26:20.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:20 smithi179 ceph-mon[43975]: osdmap e143: 12 total, 12 up, 12 in 2024-02-09T17:26:20.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:20 smithi179 ceph-mon[43975]: 2.c scrub starts 2024-02-09T17:26:20.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:20 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:26:20.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:20 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:26:20.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:20 smithi162 ceph-mon[44304]: osdmap e143: 12 total, 12 up, 12 in 2024-02-09T17:26:20.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:20 smithi162 ceph-mon[44304]: 2.c scrub starts 2024-02-09T17:26:20.502 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:20 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:26:20.502 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:20 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:26:20.502 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:20 smithi099 ceph-mon[39420]: osdmap e143: 12 total, 12 up, 12 in 2024-02-09T17:26:20.502 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:20 smithi099 ceph-mon[39420]: 2.c scrub starts 2024-02-09T17:26:21.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:21 smithi179 ceph-mon[43975]: 2.a scrub ok 2024-02-09T17:26:21.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:21 smithi179 ceph-mon[43975]: pgmap v431: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 960 op/s 2024-02-09T17:26:21.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:21 smithi179 ceph-mon[43975]: 2.9 scrub ok 2024-02-09T17:26:21.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:21 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:26:21.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:21 smithi162 ceph-mon[44304]: 2.a scrub ok 2024-02-09T17:26:21.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:21 smithi162 ceph-mon[44304]: pgmap v431: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 960 op/s 2024-02-09T17:26:21.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:21 smithi162 ceph-mon[44304]: 2.9 scrub ok 2024-02-09T17:26:21.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:26:21.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:21 smithi099 ceph-mon[39420]: 2.a scrub ok 2024-02-09T17:26:21.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:21 smithi099 ceph-mon[39420]: pgmap v431: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 960 op/s 2024-02-09T17:26:21.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:21 smithi099 ceph-mon[39420]: 2.9 scrub ok 2024-02-09T17:26:21.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:21 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:26:23.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:23 smithi179 ceph-mon[43975]: 2.c scrub ok 2024-02-09T17:26:23.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:23 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:26:23.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:23 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:26:23.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:23 smithi179 ceph-mon[43975]: 2.4 scrub starts 2024-02-09T17:26:23.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:23 smithi179 ceph-mon[43975]: pgmap v432: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 117 B/s rd, 60 MiB/s wr, 1.01k op/s 2024-02-09T17:26:23.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:23 smithi162 ceph-mon[44304]: 2.c scrub ok 2024-02-09T17:26:23.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:23 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:26:23.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:23 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:26:23.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:23 smithi162 ceph-mon[44304]: 2.4 scrub starts 2024-02-09T17:26:23.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:23 smithi162 ceph-mon[44304]: pgmap v432: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 117 B/s rd, 60 MiB/s wr, 1.01k op/s 2024-02-09T17:26:23.498 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:23 smithi099 ceph-mon[39420]: 2.c scrub ok 2024-02-09T17:26:23.498 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:23 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:26:23.499 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:23 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:26:23.499 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:23 smithi099 ceph-mon[39420]: 2.4 scrub starts 2024-02-09T17:26:23.499 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:23 smithi099 ceph-mon[39420]: pgmap v432: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 117 B/s rd, 60 MiB/s wr, 1.01k op/s 2024-02-09T17:26:23.547 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:26:23.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:23.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:26:23.655 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:26:23.687 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:23.708 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:26:23.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:23.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:26:23.811 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:26:23.894 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:24.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:24 smithi179 ceph-mon[43975]: 2.f scrub starts 2024-02-09T17:26:24.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:24 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:26:24.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:24 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:26:24.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:24 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:26:24.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:24 smithi162 ceph-mon[44304]: 2.f scrub starts 2024-02-09T17:26:24.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:24 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:26:24.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:24 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:26:24.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:24 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:26:24.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:24 smithi099 ceph-mon[39420]: 2.f scrub starts 2024-02-09T17:26:24.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:24 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:26:24.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:24 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:26:24.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:24 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:26:24.659 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:26:24.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:24.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:26:24.770 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:26:24.822 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:25.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:25 smithi099 ceph-mon[39420]: 2.5 scrub starts 2024-02-09T17:26:25.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:25 smithi099 ceph-mon[39420]: 2.f scrub ok 2024-02-09T17:26:25.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:25 smithi099 ceph-mon[39420]: pgmap v433: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 46 MiB/s wr, 876 op/s 2024-02-09T17:26:25.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:25 smithi179 ceph-mon[43975]: 2.5 scrub starts 2024-02-09T17:26:25.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:25 smithi179 ceph-mon[43975]: 2.f scrub ok 2024-02-09T17:26:25.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:25 smithi179 ceph-mon[43975]: pgmap v433: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 46 MiB/s wr, 876 op/s 2024-02-09T17:26:25.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:25 smithi162 ceph-mon[44304]: 2.5 scrub starts 2024-02-09T17:26:25.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:25 smithi162 ceph-mon[44304]: 2.f scrub ok 2024-02-09T17:26:25.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:25 smithi162 ceph-mon[44304]: pgmap v433: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 46 MiB/s wr, 876 op/s 2024-02-09T17:26:25.845 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-09T17:26:25.846 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-02-09T17:26:25.846 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph tell osd.11 'trim stale osdmaps' 2024-02-09T17:26:26.083 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:26:26.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:26.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:26:26.189 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:26:26.230 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:26.415 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:26 smithi099 ceph-mon[39420]: 2.4 scrub ok 2024-02-09T17:26:26.415 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:26 smithi099 ceph-mon[39420]: 2.5 scrub ok 2024-02-09T17:26:26.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:26 smithi179 ceph-mon[43975]: 2.4 scrub ok 2024-02-09T17:26:26.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:26 smithi179 ceph-mon[43975]: 2.5 scrub ok 2024-02-09T17:26:26.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:26 smithi162 ceph-mon[44304]: 2.4 scrub ok 2024-02-09T17:26:26.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:26 smithi162 ceph-mon[44304]: 2.5 scrub ok 2024-02-09T17:26:27.141 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:27 smithi099 ceph-mon[39420]: pgmap v434: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 28 MiB/s wr, 873 op/s 2024-02-09T17:26:27.217 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:26:27.218 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 20, 2024-02-09T17:26:27.218 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 600, 2024-02-09T17:26:27.218 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [ 2024-02-09T17:26:27.218 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.218 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:112746 2.9 e138/111 2:9f81da2f:::benchmark_data_smithi179_77138_object112745:head v 138'7056, mlcod=138'7051)", 2024-02-09T17:26:27.218 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:25:58.197218+0000", 2024-02-09T17:26:27.218 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 29.013630326000001, 2024-02-09T17:26:27.218 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.043284123000000001, 2024-02-09T17:26:27.218 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:26:27.218 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:26:27.218 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:26:27.219 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.219 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:26:27.219 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:58.197218+0000", 2024-02-09T17:26:27.219 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:26:27.219 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.219 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.219 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:26:27.219 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:58.197218+0000", 2024-02-09T17:26:27.219 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:26:27.219 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.219 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.219 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:26:27.220 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:58.197220+0000", 2024-02-09T17:26:27.220 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.373e-06 2024-02-09T17:26:27.220 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.220 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.220 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:26:27.220 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:58.197240+0000", 2024-02-09T17:26:27.220 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0367e-05 2024-02-09T17:26:27.220 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.220 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.220 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:26:27.220 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:58.197251+0000", 2024-02-09T17:26:27.220 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0933e-05 2024-02-09T17:26:27.221 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.221 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.221 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:26:27.221 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:58.197257+0000", 2024-02-09T17:26:27.221 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.1040000000000003e-06 2024-02-09T17:26:27.221 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.221 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.221 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:26:27.221 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:58.214905+0000", 2024-02-09T17:26:27.221 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.017647377999999998 2024-02-09T17:26:27.221 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.221 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.222 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:26:27.222 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:58.214924+0000", 2024-02-09T17:26:27.222 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.9222e-05 2024-02-09T17:26:27.222 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.222 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.222 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:26:27.222 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:58.240447+0000", 2024-02-09T17:26:27.222 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.025522960000000001 2024-02-09T17:26:27.222 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.222 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.222 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:26:27.222 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:58.240503+0000", 2024-02-09T17:26:27.222 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.5785999999999997e-05 2024-02-09T17:26:27.223 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:26:27.223 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:26:27.223 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:26:27.223 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.223 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.223 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:116008 2.a e139/111 2:59fd5f17:::benchmark_data_smithi179_77138_object116007:head v 139'7212, mlcod=139'7209)", 2024-02-09T17:26:27.223 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:26:01.215620+0000", 2024-02-09T17:26:27.223 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 25.995228710999999, 2024-02-09T17:26:27.223 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.040905585000000001, 2024-02-09T17:26:27.223 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:26:27.223 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:26:27.223 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:26:27.224 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.224 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:26:27.224 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.215620+0000", 2024-02-09T17:26:27.224 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:26:27.224 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.224 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.224 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:26:27.224 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.215620+0000", 2024-02-09T17:26:27.224 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:26:27.224 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.224 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.224 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:26:27.225 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.215621+0000", 2024-02-09T17:26:27.225 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1829999999999999e-06 2024-02-09T17:26:27.225 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.225 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.225 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:26:27.225 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.215681+0000", 2024-02-09T17:26:27.225 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.0171999999999999e-05 2024-02-09T17:26:27.225 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.225 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.225 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:26:27.225 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.215692+0000", 2024-02-09T17:26:27.225 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0804e-05 2024-02-09T17:26:27.226 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.226 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.226 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:26:27.226 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.215699+0000", 2024-02-09T17:26:27.226 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.0040000000000002e-06 2024-02-09T17:26:27.226 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.226 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.226 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:26:27.226 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.220222+0000", 2024-02-09T17:26:27.226 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0045225660000000004 2024-02-09T17:26:27.226 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.227 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.227 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:26:27.227 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.220244+0000", 2024-02-09T17:26:27.227 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2580000000000001e-05 2024-02-09T17:26:27.227 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.227 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.227 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:26:27.227 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.256477+0000", 2024-02-09T17:26:27.227 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.036232638999999997 2024-02-09T17:26:27.227 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.227 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.227 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:26:27.227 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.256526+0000", 2024-02-09T17:26:27.228 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.8637000000000002e-05 2024-02-09T17:26:27.228 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:26:27.228 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:26:27.228 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:26:27.228 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.228 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.228 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:116007 2.9 e139/111 2:93535257:::benchmark_data_smithi179_77138_object116006:head v 139'7272, mlcod=139'7270)", 2024-02-09T17:26:27.228 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:26:01.231148+0000", 2024-02-09T17:26:27.228 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 25.979700627, 2024-02-09T17:26:27.228 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.040146933000000003, 2024-02-09T17:26:27.228 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:26:27.228 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:26:27.228 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:26:27.228 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.229 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:26:27.229 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.231148+0000", 2024-02-09T17:26:27.229 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:26:27.229 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.229 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.229 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:26:27.229 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.231148+0000", 2024-02-09T17:26:27.229 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:26:27.229 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.229 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.229 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:26:27.229 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.231150+0000", 2024-02-09T17:26:27.229 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.7740000000000001e-06 2024-02-09T17:26:27.229 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.230 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.230 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:26:27.230 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.231178+0000", 2024-02-09T17:26:27.230 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.7866000000000001e-05 2024-02-09T17:26:27.230 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.230 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.230 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:26:27.230 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.231198+0000", 2024-02-09T17:26:27.230 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0063000000000001e-05 2024-02-09T17:26:27.230 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.230 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.230 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:26:27.230 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.231208+0000", 2024-02-09T17:26:27.230 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0045e-05 2024-02-09T17:26:27.231 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.231 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.231 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:26:27.231 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.240545+0000", 2024-02-09T17:26:27.231 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0093366260000000006 2024-02-09T17:26:27.231 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.231 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.231 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:26:27.231 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.240568+0000", 2024-02-09T17:26:27.231 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.3084000000000001e-05 2024-02-09T17:26:27.231 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.231 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.231 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:26:27.231 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.271241+0000", 2024-02-09T17:26:27.232 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.03067309 2024-02-09T17:26:27.232 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.232 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.232 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:26:27.232 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.271295+0000", 2024-02-09T17:26:27.232 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.4385000000000002e-05 2024-02-09T17:26:27.232 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:26:27.232 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:26:27.232 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:26:27.232 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.232 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.232 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:116014 2.1 e139/111 2:8a7f303b:::benchmark_data_smithi179_77138_object116013:head v 139'7124, mlcod=139'7122)", 2024-02-09T17:26:27.232 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:26:01.232480+0000", 2024-02-09T17:26:27.233 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 25.978368745000001, 2024-02-09T17:26:27.233 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.042577159000000003, 2024-02-09T17:26:27.233 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:26:27.233 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:26:27.233 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:26:27.233 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.233 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:26:27.233 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.232480+0000", 2024-02-09T17:26:27.233 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:26:27.233 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.233 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.233 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:26:27.233 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.232480+0000", 2024-02-09T17:26:27.234 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:26:27.234 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.234 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.234 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:26:27.234 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.232481+0000", 2024-02-09T17:26:27.234 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1069999999999999e-06 2024-02-09T17:26:27.234 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.234 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.234 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:26:27.234 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.232510+0000", 2024-02-09T17:26:27.234 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.9030000000000002e-05 2024-02-09T17:26:27.234 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.234 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.234 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:26:27.234 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.232528+0000", 2024-02-09T17:26:27.235 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.7689e-05 2024-02-09T17:26:27.235 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.235 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.235 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:26:27.235 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.232537+0000", 2024-02-09T17:26:27.235 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.3249999999999997e-06 2024-02-09T17:26:27.235 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.235 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.235 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:26:27.235 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.244180+0000", 2024-02-09T17:26:27.235 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.011642784 2024-02-09T17:26:27.235 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.235 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.236 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:26:27.236 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.244202+0000", 2024-02-09T17:26:27.236 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.1965999999999999e-05 2024-02-09T17:26:27.236 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.236 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.236 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:26:27.236 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.275017+0000", 2024-02-09T17:26:27.236 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.030815449000000002 2024-02-09T17:26:27.236 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.236 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.236 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:26:27.236 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.275057+0000", 2024-02-09T17:26:27.237 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9808999999999998e-05 2024-02-09T17:26:27.237 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:26:27.237 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:26:27.237 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:26:27.237 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.237 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.237 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:116012 2.9 e139/111 2:940737f4:::benchmark_data_smithi179_77138_object116011:head v 139'7273, mlcod=139'7270)", 2024-02-09T17:26:27.237 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:26:01.233493+0000", 2024-02-09T17:26:27.237 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 25.977355288999998, 2024-02-09T17:26:27.237 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.041600735, 2024-02-09T17:26:27.237 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:26:27.237 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:26:27.237 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:26:27.238 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.238 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:26:27.238 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.233493+0000", 2024-02-09T17:26:27.238 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:26:27.238 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.238 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.238 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:26:27.238 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.233493+0000", 2024-02-09T17:26:27.238 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:26:27.238 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.238 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.238 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:26:27.238 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.233494+0000", 2024-02-09T17:26:27.239 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.3200000000000004e-07 2024-02-09T17:26:27.239 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.239 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.239 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:26:27.239 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.233517+0000", 2024-02-09T17:26:27.239 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2656000000000001e-05 2024-02-09T17:26:27.239 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.239 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.239 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:26:27.239 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.233528+0000", 2024-02-09T17:26:27.239 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1233000000000001e-05 2024-02-09T17:26:27.239 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.240 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.240 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:26:27.240 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.233536+0000", 2024-02-09T17:26:27.240 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.4109999999999999e-06 2024-02-09T17:26:27.240 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.240 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.240 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:26:27.240 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.248078+0000", 2024-02-09T17:26:27.240 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.014542507 2024-02-09T17:26:27.240 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.240 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.240 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:26:27.241 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.248096+0000", 2024-02-09T17:26:27.241 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.8240000000000002e-05 2024-02-09T17:26:27.241 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.241 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.241 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:26:27.241 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.275062+0000", 2024-02-09T17:26:27.241 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.026966121999999999 2024-02-09T17:26:27.241 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.241 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.241 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:26:27.241 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.275094+0000", 2024-02-09T17:26:27.241 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.1733999999999998e-05 2024-02-09T17:26:27.241 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:26:27.241 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:26:27.242 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:26:27.242 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.242 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.242 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:116017 2.9 e139/111 2:98e98300:::benchmark_data_smithi179_77138_object116016:head v 139'7274, mlcod=139'7270)", 2024-02-09T17:26:27.242 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:26:01.233549+0000", 2024-02-09T17:26:27.242 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 25.977299872, 2024-02-09T17:26:27.242 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.041569862999999999, 2024-02-09T17:26:27.242 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:26:27.242 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:26:27.242 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:26:27.242 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.242 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:26:27.242 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.233549+0000", 2024-02-09T17:26:27.242 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:26:27.242 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.243 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.243 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:26:27.243 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.233549+0000", 2024-02-09T17:26:27.243 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:26:27.243 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.243 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.243 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:26:27.243 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.233549+0000", 2024-02-09T17:26:27.243 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.1e-07 2024-02-09T17:26:27.243 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.243 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.243 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:26:27.243 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.233568+0000", 2024-02-09T17:26:27.243 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.8626000000000001e-05 2024-02-09T17:26:27.244 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.244 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.244 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:26:27.244 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.233574+0000", 2024-02-09T17:26:27.244 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.2029999999999997e-06 2024-02-09T17:26:27.244 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.244 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.244 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:26:27.244 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.233578+0000", 2024-02-09T17:26:27.244 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.4020000000000004e-06 2024-02-09T17:26:27.244 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.244 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.244 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:26:27.245 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.251746+0000", 2024-02-09T17:26:27.245 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.018168006 2024-02-09T17:26:27.245 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.245 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.245 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:26:27.245 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.251767+0000", 2024-02-09T17:26:27.245 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0862000000000001e-05 2024-02-09T17:26:27.245 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.245 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.245 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:26:27.245 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.275096+0000", 2024-02-09T17:26:27.245 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.023328865000000001 2024-02-09T17:26:27.245 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.245 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.246 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:26:27.246 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.275119+0000", 2024-02-09T17:26:27.246 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2589e-05 2024-02-09T17:26:27.246 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:26:27.246 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:26:27.246 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:26:27.246 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.246 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.246 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:116018 2.9 e139/111 2:9eeef93a:::benchmark_data_smithi179_77138_object116017:head v 139'7275, mlcod=139'7270)", 2024-02-09T17:26:27.246 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:26:01.234854+0000", 2024-02-09T17:26:27.246 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 25.975995159, 2024-02-09T17:26:27.246 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.040287851, 2024-02-09T17:26:27.246 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:26:27.246 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:26:27.246 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:26:27.247 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.247 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:26:27.247 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.234854+0000", 2024-02-09T17:26:27.247 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:26:27.247 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.247 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.247 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:26:27.247 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.234854+0000", 2024-02-09T17:26:27.247 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:26:27.247 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.247 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.247 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:26:27.247 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.234855+0000", 2024-02-09T17:26:27.247 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.545e-06 2024-02-09T17:26:27.247 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.248 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.248 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:26:27.248 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.234880+0000", 2024-02-09T17:26:27.248 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.4729e-05 2024-02-09T17:26:27.248 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.248 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.248 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:26:27.248 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.234897+0000", 2024-02-09T17:26:27.248 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.6625000000000001e-05 2024-02-09T17:26:27.248 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.248 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.248 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:26:27.249 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.234906+0000", 2024-02-09T17:26:27.249 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.1009999999999998e-06 2024-02-09T17:26:27.249 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.249 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.249 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:26:27.249 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.255261+0000", 2024-02-09T17:26:27.249 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.020355874999999999 2024-02-09T17:26:27.249 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.249 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.249 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:26:27.249 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.255285+0000", 2024-02-09T17:26:27.249 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.3516e-05 2024-02-09T17:26:27.249 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.249 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.249 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:26:27.249 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.275121+0000", 2024-02-09T17:26:27.250 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.019835518 2024-02-09T17:26:27.250 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.250 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.250 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:26:27.250 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:01.275141+0000", 2024-02-09T17:26:27.250 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0942e-05 2024-02-09T17:26:27.250 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:26:27.250 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:26:27.250 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:26:27.250 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.251 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.251 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:117560 2.6 e139/111 2:6d771913:::benchmark_data_smithi179_77138_object117559:head v 139'7369, mlcod=139'7367)", 2024-02-09T17:26:27.251 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:26:02.620286+0000", 2024-02-09T17:26:27.251 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 24.590563275000001, 2024-02-09T17:26:27.251 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.052902339, 2024-02-09T17:26:27.251 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:26:27.251 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:26:27.251 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:26:27.251 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.251 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:26:27.251 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:02.620286+0000", 2024-02-09T17:26:27.251 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:26:27.251 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.251 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.251 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:26:27.252 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:02.620286+0000", 2024-02-09T17:26:27.252 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:26:27.252 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.252 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.252 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:26:27.252 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:02.620286+0000", 2024-02-09T17:26:27.252 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.1900000000000002e-07 2024-02-09T17:26:27.252 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.252 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.252 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:26:27.252 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:02.620306+0000", 2024-02-09T17:26:27.252 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0129999999999999e-05 2024-02-09T17:26:27.252 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.253 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.253 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:26:27.253 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:02.620324+0000", 2024-02-09T17:26:27.253 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.7394e-05 2024-02-09T17:26:27.253 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.253 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.253 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:26:27.253 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:02.620331+0000", 2024-02-09T17:26:27.253 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.8159999999999996e-06 2024-02-09T17:26:27.253 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.253 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.253 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:26:27.253 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:02.620416+0000", 2024-02-09T17:26:27.254 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.5410999999999999e-05 2024-02-09T17:26:27.254 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.254 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.254 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:26:27.254 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:02.620438+0000", 2024-02-09T17:26:27.254 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2177000000000001e-05 2024-02-09T17:26:27.254 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.254 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.254 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:26:27.254 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:02.673128+0000", 2024-02-09T17:26:27.254 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.052689690999999997 2024-02-09T17:26:27.254 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.254 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.255 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:26:27.255 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:02.673188+0000", 2024-02-09T17:26:27.255 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.0000999999999997e-05 2024-02-09T17:26:27.255 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:26:27.255 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:26:27.255 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:26:27.255 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.255 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.255 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:117564 2.e e139/111 2:7da21441:::benchmark_data_smithi179_77138_object117563:head v 139'7290, mlcod=139'7288)", 2024-02-09T17:26:27.255 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:26:02.625738+0000", 2024-02-09T17:26:27.255 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 24.585110646, 2024-02-09T17:26:27.255 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.048049758999999997, 2024-02-09T17:26:27.255 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:26:27.256 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:26:27.256 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:26:27.256 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.256 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:26:27.256 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:02.625738+0000", 2024-02-09T17:26:27.256 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:26:27.256 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.256 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.256 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:26:27.256 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:02.625738+0000", 2024-02-09T17:26:27.256 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:26:27.256 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.256 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.256 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:26:27.257 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:02.625739+0000", 2024-02-09T17:26:27.257 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.1200000000000001e-07 2024-02-09T17:26:27.257 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.257 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.257 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:26:27.257 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:02.625761+0000", 2024-02-09T17:26:27.257 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.1554999999999999e-05 2024-02-09T17:26:27.257 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.257 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.257 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:26:27.257 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:02.625773+0000", 2024-02-09T17:26:27.257 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2322e-05 2024-02-09T17:26:27.257 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.258 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.258 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:26:27.258 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:02.625779+0000", 2024-02-09T17:26:27.258 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.9939999999999997e-06 2024-02-09T17:26:27.258 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.258 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.258 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:26:27.258 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:02.647486+0000", 2024-02-09T17:26:27.258 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.021707400000000002 2024-02-09T17:26:27.258 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.258 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.258 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:26:27.258 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:02.647510+0000", 2024-02-09T17:26:27.258 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.4170000000000001e-05 2024-02-09T17:26:27.259 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.259 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.259 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:26:27.259 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:02.673769+0000", 2024-02-09T17:26:27.259 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.026258981000000001 2024-02-09T17:26:27.259 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.259 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.259 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:26:27.259 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:02.673788+0000", 2024-02-09T17:26:27.259 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.8425000000000001e-05 2024-02-09T17:26:27.259 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:26:27.259 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:26:27.259 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:26:27.259 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.259 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.260 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:133267 2.e e143/111 2:729e455d:::benchmark_data_smithi179_77138_object133266:head v 143'8317, mlcod=143'8315)", 2024-02-09T17:26:27.260 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:26:19.260944+0000", 2024-02-09T17:26:27.260 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 7.949905126, 2024-02-09T17:26:27.260 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.057743624, 2024-02-09T17:26:27.260 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:26:27.260 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:26:27.260 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:26:27.260 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.260 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:26:27.260 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.260944+0000", 2024-02-09T17:26:27.260 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:26:27.260 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.260 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.260 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:26:27.261 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.260944+0000", 2024-02-09T17:26:27.261 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:26:27.261 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.261 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.261 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:26:27.261 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.260944+0000", 2024-02-09T17:26:27.261 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.3900000000000004e-07 2024-02-09T17:26:27.261 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.261 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.261 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:26:27.261 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.260967+0000", 2024-02-09T17:26:27.261 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.3125999999999998e-05 2024-02-09T17:26:27.261 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.261 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.261 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:26:27.262 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.260978+0000", 2024-02-09T17:26:27.262 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0539e-05 2024-02-09T17:26:27.262 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.262 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.262 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:26:27.262 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.260985+0000", 2024-02-09T17:26:27.262 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.4950000000000002e-06 2024-02-09T17:26:27.262 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.262 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.262 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:26:27.262 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.261039+0000", 2024-02-09T17:26:27.262 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3066999999999999e-05 2024-02-09T17:26:27.262 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.262 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.262 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:26:27.263 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.261053+0000", 2024-02-09T17:26:27.263 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4418e-05 2024-02-09T17:26:27.263 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.263 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.263 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:26:27.263 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.318639+0000", 2024-02-09T17:26:27.263 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.057585721999999999 2024-02-09T17:26:27.263 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.263 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.263 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:26:27.263 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.318687+0000", 2024-02-09T17:26:27.263 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.8618000000000001e-05 2024-02-09T17:26:27.263 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:26:27.263 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:26:27.263 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:26:27.264 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.264 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.264 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:133268 2.e e143/111 2:75018e88:::benchmark_data_smithi179_77138_object133267:head v 143'8318, mlcod=143'8315)", 2024-02-09T17:26:27.264 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:26:19.262093+0000", 2024-02-09T17:26:27.264 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 7.9487554410000003, 2024-02-09T17:26:27.264 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.056620009999999998, 2024-02-09T17:26:27.264 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:26:27.264 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:26:27.264 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:26:27.264 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.264 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:26:27.264 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.262093+0000", 2024-02-09T17:26:27.264 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:26:27.264 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.265 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.265 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:26:27.265 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.262093+0000", 2024-02-09T17:26:27.265 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:26:27.265 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.265 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.265 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:26:27.265 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.262094+0000", 2024-02-09T17:26:27.265 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.2500000000000005e-07 2024-02-09T17:26:27.265 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.265 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.265 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:26:27.265 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.262117+0000", 2024-02-09T17:26:27.265 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.3170999999999999e-05 2024-02-09T17:26:27.267 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.267 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.267 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:26:27.267 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.262127+0000", 2024-02-09T17:26:27.267 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0271999999999999e-05 2024-02-09T17:26:27.268 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.268 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.268 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:26:27.268 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.262134+0000", 2024-02-09T17:26:27.268 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.7510000000000002e-06 2024-02-09T17:26:27.268 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.268 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.268 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:26:27.268 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.264860+0000", 2024-02-09T17:26:27.268 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.002726245 2024-02-09T17:26:27.268 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.269 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.269 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:26:27.269 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.264881+0000", 2024-02-09T17:26:27.269 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0119000000000002e-05 2024-02-09T17:26:27.269 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.269 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.269 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:26:27.269 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.318690+0000", 2024-02-09T17:26:27.269 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.053809546999999999 2024-02-09T17:26:27.269 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.269 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.269 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:26:27.269 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.318713+0000", 2024-02-09T17:26:27.270 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.3280000000000001e-05 2024-02-09T17:26:27.270 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:26:27.270 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:26:27.270 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:26:27.270 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.270 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.270 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:133273 2.e e143/111 2:7692b402:::benchmark_data_smithi179_77138_object133272:head v 143'8319, mlcod=143'8315)", 2024-02-09T17:26:27.270 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:26:19.266353+0000", 2024-02-09T17:26:27.270 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 7.9444953070000004, 2024-02-09T17:26:27.270 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.06080443, 2024-02-09T17:26:27.270 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:26:27.270 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:26:27.271 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:26:27.271 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.271 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:26:27.271 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.266353+0000", 2024-02-09T17:26:27.271 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:26:27.271 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.271 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.271 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:26:27.271 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.266353+0000", 2024-02-09T17:26:27.271 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:26:27.271 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.271 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.272 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:26:27.272 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.266354+0000", 2024-02-09T17:26:27.272 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.0399999999999995e-07 2024-02-09T17:26:27.272 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.272 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.272 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:26:27.272 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.266377+0000", 2024-02-09T17:26:27.272 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2983999999999999e-05 2024-02-09T17:26:27.272 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.272 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.273 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:26:27.273 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.266386+0000", 2024-02-09T17:26:27.273 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.9209999999999998e-06 2024-02-09T17:26:27.273 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.273 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.273 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:26:27.273 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.266398+0000", 2024-02-09T17:26:27.273 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1617000000000001e-05 2024-02-09T17:26:27.273 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.273 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.273 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:26:27.273 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.314510+0000", 2024-02-09T17:26:27.274 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.048111845 2024-02-09T17:26:27.274 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.274 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.274 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:26:27.274 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.314528+0000", 2024-02-09T17:26:27.274 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.8834e-05 2024-02-09T17:26:27.274 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.274 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.274 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:26:27.274 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.327112+0000", 2024-02-09T17:26:27.274 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.012583967 2024-02-09T17:26:27.274 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.275 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.275 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:26:27.275 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.327158+0000", 2024-02-09T17:26:27.275 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.5558e-05 2024-02-09T17:26:27.275 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:26:27.275 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:26:27.275 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:26:27.275 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.275 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.275 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:133278 2.e e143/111 2:7ef0de1c:::benchmark_data_smithi179_77138_object133277:head v 143'8320, mlcod=143'8315)", 2024-02-09T17:26:27.275 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:26:19.270915+0000", 2024-02-09T17:26:27.276 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 7.9399342439999998, 2024-02-09T17:26:27.276 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.056271237000000002, 2024-02-09T17:26:27.276 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:26:27.276 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:26:27.276 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:26:27.276 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.276 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:26:27.276 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.270915+0000", 2024-02-09T17:26:27.276 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:26:27.276 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.276 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.276 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:26:27.276 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.270915+0000", 2024-02-09T17:26:27.277 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:26:27.277 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.277 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.277 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:26:27.277 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.270915+0000", 2024-02-09T17:26:27.277 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.6400000000000001e-07 2024-02-09T17:26:27.277 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.277 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.277 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:26:27.277 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.270936+0000", 2024-02-09T17:26:27.277 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0526999999999998e-05 2024-02-09T17:26:27.277 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.277 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.277 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:26:27.278 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.314049+0000", 2024-02-09T17:26:27.278 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.043112959999999999 2024-02-09T17:26:27.278 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.278 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.278 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:26:27.278 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.314062+0000", 2024-02-09T17:26:27.278 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3088000000000001e-05 2024-02-09T17:26:27.278 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.278 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.278 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:26:27.278 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.318739+0000", 2024-02-09T17:26:27.278 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0046773860000000004 2024-02-09T17:26:27.278 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.279 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.279 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:26:27.279 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.318758+0000", 2024-02-09T17:26:27.279 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.9012e-05 2024-02-09T17:26:27.279 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.279 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.279 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:26:27.279 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.327163+0000", 2024-02-09T17:26:27.279 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00840488 2024-02-09T17:26:27.279 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.279 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.279 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:26:27.280 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.327186+0000", 2024-02-09T17:26:27.280 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.262e-05 2024-02-09T17:26:27.280 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:26:27.280 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:26:27.280 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:26:27.280 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.280 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.280 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:133740 2.d e143/111 2:bb74f4b7:::benchmark_data_smithi179_77138_object332:head v 143'8484, mlcod=143'8482)", 2024-02-09T17:26:27.280 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:26:19.838708+0000", 2024-02-09T17:26:27.280 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 7.3721403810000004, 2024-02-09T17:26:27.280 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.048770398999999999, 2024-02-09T17:26:27.280 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:26:27.280 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:26:27.281 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:26:27.281 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.281 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:26:27.281 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.838708+0000", 2024-02-09T17:26:27.281 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:26:27.281 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.281 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.281 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:26:27.281 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.838708+0000", 2024-02-09T17:26:27.281 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:26:27.281 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.281 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.281 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:26:27.282 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.838709+0000", 2024-02-09T17:26:27.282 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.6899999999999996e-07 2024-02-09T17:26:27.282 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.282 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.282 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:26:27.282 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.838715+0000", 2024-02-09T17:26:27.282 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.463e-06 2024-02-09T17:26:27.282 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.282 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.282 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:26:27.282 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.877218+0000", 2024-02-09T17:26:27.282 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.038503048999999998 2024-02-09T17:26:27.282 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.282 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.282 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:26:27.283 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.877231+0000", 2024-02-09T17:26:27.283 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2927e-05 2024-02-09T17:26:27.283 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.283 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.283 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:26:27.283 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.877298+0000", 2024-02-09T17:26:27.283 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.7623e-05 2024-02-09T17:26:27.283 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.283 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.283 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:26:27.283 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.877319+0000", 2024-02-09T17:26:27.283 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0295000000000001e-05 2024-02-09T17:26:27.283 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.283 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.284 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:26:27.284 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.887451+0000", 2024-02-09T17:26:27.284 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.010132304999999999 2024-02-09T17:26:27.284 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.284 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.284 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:26:27.284 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:19.887479+0000", 2024-02-09T17:26:27.284 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.7868000000000002e-05 2024-02-09T17:26:27.284 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:26:27.284 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:26:27.284 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:26:27.284 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.284 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.284 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:135926 2.1 e143/111 2:897c7fe5:::benchmark_data_smithi179_77138_object2518:head v 143'8391, mlcod=143'8388)", 2024-02-09T17:26:27.285 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:26:22.230426+0000", 2024-02-09T17:26:27.285 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 4.9804230460000003, 2024-02-09T17:26:27.285 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.03693424, 2024-02-09T17:26:27.285 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:26:27.285 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:26:27.285 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:26:27.285 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.285 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:26:27.285 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:22.230426+0000", 2024-02-09T17:26:27.285 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:26:27.285 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.285 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.285 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:26:27.285 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:22.230426+0000", 2024-02-09T17:26:27.286 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:26:27.286 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.286 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.286 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:26:27.286 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:22.230427+0000", 2024-02-09T17:26:27.286 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.097e-06 2024-02-09T17:26:27.286 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.286 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.286 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:26:27.286 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:22.230435+0000", 2024-02-09T17:26:27.286 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.9030000000000004e-06 2024-02-09T17:26:27.286 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.286 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.287 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:26:27.287 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:22.230449+0000", 2024-02-09T17:26:27.287 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4691999999999999e-05 2024-02-09T17:26:27.287 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.287 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.287 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:26:27.287 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:22.230458+0000", 2024-02-09T17:26:27.287 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.3839999999999997e-06 2024-02-09T17:26:27.287 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.287 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.287 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:26:27.287 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:22.234704+0000", 2024-02-09T17:26:27.287 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0042464779999999997 2024-02-09T17:26:27.288 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.288 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.288 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:26:27.288 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:22.234724+0000", 2024-02-09T17:26:27.288 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0112e-05 2024-02-09T17:26:27.288 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.288 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.288 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:26:27.288 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:22.267321+0000", 2024-02-09T17:26:27.288 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.032596848999999997 2024-02-09T17:26:27.288 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.288 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.288 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:26:27.289 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:22.267360+0000", 2024-02-09T17:26:27.289 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.8724999999999999e-05 2024-02-09T17:26:27.289 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:26:27.289 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:26:27.289 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:26:27.289 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.289 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.289 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:137277 2.a e143/111 2:59827782:::benchmark_data_smithi179_77138_object3869:head v 143'8508, mlcod=143'8506)", 2024-02-09T17:26:27.289 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:26:23.887507+0000", 2024-02-09T17:26:27.289 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 3.3233415530000001, 2024-02-09T17:26:27.289 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.039508548999999997, 2024-02-09T17:26:27.289 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:26:27.289 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:26:27.290 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:26:27.290 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.290 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:26:27.290 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:23.887507+0000", 2024-02-09T17:26:27.290 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:26:27.290 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.290 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.290 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:26:27.290 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:23.887507+0000", 2024-02-09T17:26:27.290 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:26:27.290 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.290 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.290 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:26:27.290 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:23.887509+0000", 2024-02-09T17:26:27.290 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5510000000000001e-06 2024-02-09T17:26:27.291 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.291 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.291 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:26:27.291 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:23.887518+0000", 2024-02-09T17:26:27.291 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.4269999999999997e-06 2024-02-09T17:26:27.291 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.291 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.291 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:26:27.291 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:23.887537+0000", 2024-02-09T17:26:27.291 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.8389e-05 2024-02-09T17:26:27.291 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.291 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.291 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:26:27.291 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:23.887546+0000", 2024-02-09T17:26:27.291 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.2310000000000002e-06 2024-02-09T17:26:27.291 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.292 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.292 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:26:27.292 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:23.889094+0000", 2024-02-09T17:26:27.292 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0015483129999999999 2024-02-09T17:26:27.292 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.292 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.292 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:26:27.292 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:23.889113+0000", 2024-02-09T17:26:27.292 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.9262999999999998e-05 2024-02-09T17:26:27.292 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.292 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.292 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:26:27.292 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:23.926970+0000", 2024-02-09T17:26:27.293 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.037856386999999998 2024-02-09T17:26:27.293 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.293 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.293 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:26:27.293 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:23.927016+0000", 2024-02-09T17:26:27.293 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.5988000000000001e-05 2024-02-09T17:26:27.293 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:26:27.293 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:26:27.293 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:26:27.293 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.293 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.293 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:137843 2.9 e143/111 2:982c91d8:::benchmark_data_smithi179_77138_object4435:head v 143'8629, mlcod=143'8624)", 2024-02-09T17:26:27.293 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:26:24.507594+0000", 2024-02-09T17:26:27.293 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 2.7032550450000001, 2024-02-09T17:26:27.294 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.038083032000000003, 2024-02-09T17:26:27.294 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:26:27.294 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:26:27.294 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:26:27.294 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.294 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:26:27.294 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:24.507594+0000", 2024-02-09T17:26:27.294 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:26:27.294 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.294 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.294 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:26:27.294 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:24.507594+0000", 2024-02-09T17:26:27.295 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:26:27.295 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.295 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.295 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:26:27.295 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:24.507594+0000", 2024-02-09T17:26:27.295 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.6e-07 2024-02-09T17:26:27.295 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.295 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.295 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:26:27.295 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:24.507599+0000", 2024-02-09T17:26:27.295 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.8849999999999998e-06 2024-02-09T17:26:27.295 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.295 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.296 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:26:27.296 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:24.507603+0000", 2024-02-09T17:26:27.296 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.7160000000000002e-06 2024-02-09T17:26:27.296 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.296 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.296 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:26:27.296 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:24.507607+0000", 2024-02-09T17:26:27.296 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.6210000000000001e-06 2024-02-09T17:26:27.296 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.296 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.296 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:26:27.297 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:24.530436+0000", 2024-02-09T17:26:27.297 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.022829180000000001 2024-02-09T17:26:27.297 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.297 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.297 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:26:27.297 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:24.530460+0000", 2024-02-09T17:26:27.297 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.3949000000000001e-05 2024-02-09T17:26:27.297 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.297 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.297 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:26:27.297 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:24.545632+0000", 2024-02-09T17:26:27.297 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.015172138 2024-02-09T17:26:27.298 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.298 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.298 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:26:27.298 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:24.545677+0000", 2024-02-09T17:26:27.298 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.4382999999999998e-05 2024-02-09T17:26:27.298 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:26:27.298 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:26:27.298 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:26:27.298 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.298 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.298 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:138136 2.9 e143/111 2:93b189f3:::benchmark_data_smithi179_77138_object4728:head v 143'8646, mlcod=143'8641)", 2024-02-09T17:26:27.298 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:26:24.828113+0000", 2024-02-09T17:26:27.298 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 2.3827355670000001, 2024-02-09T17:26:27.299 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.034463497000000003, 2024-02-09T17:26:27.299 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:26:27.299 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:26:27.299 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:26:27.299 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.299 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:26:27.299 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:24.828113+0000", 2024-02-09T17:26:27.299 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:26:27.299 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.299 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.299 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:26:27.299 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:24.828113+0000", 2024-02-09T17:26:27.299 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:26:27.299 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.300 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.300 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:26:27.300 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:24.828114+0000", 2024-02-09T17:26:27.300 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.1699999999999998e-07 2024-02-09T17:26:27.300 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.300 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.300 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:26:27.300 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:24.828120+0000", 2024-02-09T17:26:27.300 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7810000000000002e-06 2024-02-09T17:26:27.300 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.300 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.300 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:26:27.300 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:24.828129+0000", 2024-02-09T17:26:27.301 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.5300000000000002e-06 2024-02-09T17:26:27.301 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.301 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.301 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:26:27.301 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:24.828136+0000", 2024-02-09T17:26:27.301 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.0049999999999998e-06 2024-02-09T17:26:27.301 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.301 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.301 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:26:27.301 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:24.836968+0000", 2024-02-09T17:26:27.301 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0088321630000000005 2024-02-09T17:26:27.301 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.301 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.301 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:26:27.301 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:24.836985+0000", 2024-02-09T17:26:27.302 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.6555999999999999e-05 2024-02-09T17:26:27.302 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.302 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.302 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:26:27.302 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:24.862528+0000", 2024-02-09T17:26:27.302 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.025542777999999999 2024-02-09T17:26:27.302 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.302 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.302 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:26:27.302 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:24.862577+0000", 2024-02-09T17:26:27.302 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.9067000000000003e-05 2024-02-09T17:26:27.302 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:26:27.302 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:26:27.302 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:26:27.302 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.303 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.303 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:138556 2.1 e143/111 2:87ff7f55:::benchmark_data_smithi179_77138_object5148:head v 143'8568, mlcod=143'8565)", 2024-02-09T17:26:27.303 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:26:25.247586+0000", 2024-02-09T17:26:27.303 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 1.9632626119999999, 2024-02-09T17:26:27.303 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.034928946000000002, 2024-02-09T17:26:27.303 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:26:27.303 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:26:27.303 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:26:27.303 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.303 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:26:27.303 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:25.247586+0000", 2024-02-09T17:26:27.303 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:26:27.303 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.303 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.304 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:26:27.304 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:25.247586+0000", 2024-02-09T17:26:27.304 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:26:27.304 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.304 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.304 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:26:27.304 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:25.247586+0000", 2024-02-09T17:26:27.304 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5800000000000001e-07 2024-02-09T17:26:27.304 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.304 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.304 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:26:27.304 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:25.247592+0000", 2024-02-09T17:26:27.304 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.4310000000000002e-06 2024-02-09T17:26:27.304 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.305 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.305 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:26:27.305 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:25.247598+0000", 2024-02-09T17:26:27.305 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.0719999999999996e-06 2024-02-09T17:26:27.305 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.305 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.305 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:26:27.305 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:25.247602+0000", 2024-02-09T17:26:27.305 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.3989999999999997e-06 2024-02-09T17:26:27.305 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.305 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.305 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:26:27.305 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:25.259119+0000", 2024-02-09T17:26:27.305 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.011516788999999999 2024-02-09T17:26:27.306 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.306 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.306 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:26:27.306 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:25.259141+0000", 2024-02-09T17:26:27.306 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2339000000000001e-05 2024-02-09T17:26:27.306 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.306 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.306 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:26:27.306 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:25.282475+0000", 2024-02-09T17:26:27.306 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.023333501 2024-02-09T17:26:27.306 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.306 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.306 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:26:27.306 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:25.282515+0000", 2024-02-09T17:26:27.307 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.0256999999999998e-05 2024-02-09T17:26:27.307 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:26:27.307 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:26:27.307 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:26:27.307 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.307 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.307 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:139604 2.e e143/111 2:7f3a833e:::benchmark_data_smithi179_77138_object6196:head v 143'8721, mlcod=143'8718)", 2024-02-09T17:26:27.307 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:26:26.301881+0000", 2024-02-09T17:26:27.307 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.90896818599999996, 2024-02-09T17:26:27.307 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.038439864999999997, 2024-02-09T17:26:27.307 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:26:27.307 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:26:27.307 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:26:27.308 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.308 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:26:27.308 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:26.301881+0000", 2024-02-09T17:26:27.308 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:26:27.308 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.308 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.308 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:26:27.308 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:26.301881+0000", 2024-02-09T17:26:27.308 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:26:27.308 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.308 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.308 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:26:27.308 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:26.301881+0000", 2024-02-09T17:26:27.308 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.3399999999999998e-07 2024-02-09T17:26:27.309 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.309 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.309 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:26:27.309 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:26.301889+0000", 2024-02-09T17:26:27.309 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.0990000000000001e-06 2024-02-09T17:26:27.309 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.309 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.309 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:26:27.309 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:26.301903+0000", 2024-02-09T17:26:27.309 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4688999999999999e-05 2024-02-09T17:26:27.309 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.309 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.309 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:26:27.310 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:26.301910+0000", 2024-02-09T17:26:27.310 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.9700000000000002e-06 2024-02-09T17:26:27.310 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.310 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.310 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:26:27.310 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:26.308028+0000", 2024-02-09T17:26:27.310 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0061177469999999998 2024-02-09T17:26:27.310 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.310 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.310 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:26:27.310 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:26.308047+0000", 2024-02-09T17:26:27.310 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.8630999999999998e-05 2024-02-09T17:26:27.310 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.310 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.311 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:26:27.311 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:26.340010+0000", 2024-02-09T17:26:27.311 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.031963321000000003 2024-02-09T17:26:27.311 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:26:27.311 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:26:27.311 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:26:27.311 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:26.340320+0000", 2024-02-09T17:26:27.311 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00031057399999999998 2024-02-09T17:26:27.311 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:26:27.311 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:26:27.311 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:26:27.311 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:26:27.311 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:26:27.311 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:26:27.312 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:26:27.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:27.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:26:27.419 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:26:27.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:27 smithi179 ceph-mon[43975]: pgmap v434: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 28 MiB/s wr, 873 op/s 2024-02-09T17:26:27.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:27 smithi162 ceph-mon[44304]: pgmap v434: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 28 MiB/s wr, 873 op/s 2024-02-09T17:26:27.455 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:27.478 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:26:27.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:27.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:26:27.582 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:26:27.622 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:27.652 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:26:27.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:27.655 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:26:27.756 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:26:27.795 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:27.821 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:26:27.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:27.827 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:26:27.928 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:26:27.957 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:27.983 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:26:27.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:27.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:26:28.087 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:26:28.152 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:28.312 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:26:28.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:28.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:26:28.348 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-09T17:26:28.420 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:26:28.438 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:26:28.462 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:26:28.517 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:28.639 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:26:28.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:28.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:26:28.678 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:26:28.750 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:26:28.830 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:28.872 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:26:28.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:28.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:26:28.980 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:26:29.018 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:29.045 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:26:29.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:29.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:26:29.149 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:26:29.179 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:29.201 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:26:29.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:29.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:26:29.313 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:26:29.353 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:29.378 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:26:29.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:29.384 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:26:29.447 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:29 smithi099 ceph-mon[39420]: pgmap v435: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 25 MiB/s wr, 884 op/s 2024-02-09T17:26:29.447 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:29 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3303975452' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:26:29.479 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is unset 2024-02-09T17:26:29.485 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:26:29.549 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:29.599 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:26:29.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:29.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:26:29.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:29 smithi179 ceph-mon[43975]: pgmap v435: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 25 MiB/s wr, 884 op/s 2024-02-09T17:26:29.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:29 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3303975452' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:26:29.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:29 smithi162 ceph-mon[44304]: pgmap v435: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 25 MiB/s wr, 884 op/s 2024-02-09T17:26:29.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:29 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3303975452' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:26:29.719 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:26:29.765 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:29.811 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:26:29.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:29.816 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:26:29.917 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:26:29.963 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:29.998 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:26:30.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:30.006 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:26:30.107 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:26:30.142 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:30.172 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:26:30.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:30.180 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:26:30.197 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:26:30.281 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:26:30.313 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:30.452 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:30 smithi099 ceph-mon[39420]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:26:30.453 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:30 smithi099 ceph-mon[39420]: Cluster is now healthy 2024-02-09T17:26:30.453 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:30 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3303975452' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:26:30.453 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:30 smithi099 ceph-mon[39420]: osdmap e144: 12 total, 12 up, 12 in 2024-02-09T17:26:30.466 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:26:30.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:30.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:26:30.570 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:26:30.610 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:30.630 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:26:30.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:30.633 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:26:30.734 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:26:30.802 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:26:30.802 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-02-09T17:26:30.802 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:30.802 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.14790.0:143986 2.5 e144/111 2:ad5c14fb:::benchmark_data_smithi179_77138_object10578:head v 144'8922, mlcod=144'8920)", 2024-02-09T17:26:30.802 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:26:30.793279+0000", 2024-02-09T17:26:30.802 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0043965590000000004, 2024-02-09T17:26:30.802 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0044273419999999999, 2024-02-09T17:26:30.802 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:26:30.802 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-09T17:26:30.802 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:26:30.803 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:30.803 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:26:30.803 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:30.793279+0000", 2024-02-09T17:26:30.803 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:26:30.803 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:30.803 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:30.803 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:26:30.803 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:30.793279+0000", 2024-02-09T17:26:30.803 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:26:30.803 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:30.803 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:30.803 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:26:30.803 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:30.793280+0000", 2024-02-09T17:26:30.803 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1030000000000001e-06 2024-02-09T17:26:30.804 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:30.804 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:30.804 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:26:30.804 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:30.793288+0000", 2024-02-09T17:26:30.804 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8669999999999994e-06 2024-02-09T17:26:30.804 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:30.804 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:30.804 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:26:30.804 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:30.793302+0000", 2024-02-09T17:26:30.804 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4405e-05 2024-02-09T17:26:30.804 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:30.804 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:30.805 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:26:30.805 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:30.793311+0000", 2024-02-09T17:26:30.805 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.524e-06 2024-02-09T17:26:30.805 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:30.805 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:30.805 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:26:30.805 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:30.793355+0000", 2024-02-09T17:26:30.805 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4737999999999998e-05 2024-02-09T17:26:30.805 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:30.805 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:30.805 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:26:30.805 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:30.793370+0000", 2024-02-09T17:26:30.805 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4824000000000001e-05 2024-02-09T17:26:30.805 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:26:30.805 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:26:30.806 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:26:30.806 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:30.806 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:30.806 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.14790.0:143987 2.5 e144/111 2:a55479e6:::benchmark_data_smithi179_77138_object10579:head v 144'8923, mlcod=144'8920)", 2024-02-09T17:26:30.806 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:26:30.795568+0000", 2024-02-09T17:26:30.806 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0021069349999999999, 2024-02-09T17:26:30.806 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021806059999999999, 2024-02-09T17:26:30.806 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:26:30.806 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-09T17:26:30.806 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:26:30.806 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:30.806 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:26:30.806 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:30.795568+0000", 2024-02-09T17:26:30.806 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:26:30.806 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:30.807 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:30.807 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:26:30.807 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:30.795568+0000", 2024-02-09T17:26:30.807 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:26:30.807 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:30.807 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:30.807 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:26:30.807 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:30.795569+0000", 2024-02-09T17:26:30.807 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1900000000000001e-07 2024-02-09T17:26:30.807 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:30.807 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:30.807 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:26:30.807 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:30.795576+0000", 2024-02-09T17:26:30.807 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4119999999999998e-06 2024-02-09T17:26:30.807 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:30.808 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:30.808 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:26:30.808 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:30.795588+0000", 2024-02-09T17:26:30.808 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2605e-05 2024-02-09T17:26:30.808 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:30.808 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:30.808 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:26:30.808 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:30.795596+0000", 2024-02-09T17:26:30.808 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6559999999999992e-06 2024-02-09T17:26:30.808 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:30.808 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:30.808 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:26:30.808 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:30.796442+0000", 2024-02-09T17:26:30.808 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00084635699999999995 2024-02-09T17:26:30.809 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:30.809 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:30.809 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:26:30.809 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:30.796458+0000", 2024-02-09T17:26:30.809 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6181e-05 2024-02-09T17:26:30.809 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:26:30.809 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:26:30.809 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:26:30.809 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:26:30.809 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-09T17:26:30.809 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 2 2024-02-09T17:26:30.809 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:26:30.810 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:30.823 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:30 smithi162 ceph-mon[44304]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:26:30.823 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:30 smithi162 ceph-mon[44304]: Cluster is now healthy 2024-02-09T17:26:30.823 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:30 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3303975452' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:26:30.823 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:30 smithi162 ceph-mon[44304]: osdmap e144: 12 total, 12 up, 12 in 2024-02-09T17:26:30.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:30 smithi179 ceph-mon[43975]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:26:30.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:30 smithi179 ceph-mon[43975]: Cluster is now healthy 2024-02-09T17:26:30.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:30 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3303975452' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:26:30.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:30 smithi179 ceph-mon[43975]: osdmap e144: 12 total, 12 up, 12 in 2024-02-09T17:26:30.981 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:26:30.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:30.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:26:31.087 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:26:31.118 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:31.477 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-09T17:26:31.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:31 smithi099 ceph-mon[39420]: pgmap v437: 17 pgs: 17 active+clean; 7.6 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 5.2 MiB/s wr, 923 op/s 2024-02-09T17:26:31.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:31 smithi099 ceph-mon[39420]: 1.0 scrub starts 2024-02-09T17:26:31.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:31 smithi099 ceph-mon[39420]: 1.0 scrub ok 2024-02-09T17:26:31.761 INFO:teuthology.orchestra.run.smithi099.stderr:no valid command found; 10 closest matches: 2024-02-09T17:26:31.761 INFO:teuthology.orchestra.run.smithi099.stderr:0 2024-02-09T17:26:31.761 INFO:teuthology.orchestra.run.smithi099.stderr:1 2024-02-09T17:26:31.761 INFO:teuthology.orchestra.run.smithi099.stderr:2 2024-02-09T17:26:31.762 INFO:teuthology.orchestra.run.smithi099.stderr:abort 2024-02-09T17:26:31.762 INFO:teuthology.orchestra.run.smithi099.stderr:assert 2024-02-09T17:26:31.762 INFO:teuthology.orchestra.run.smithi099.stderr:bench [] [] [] [] 2024-02-09T17:26:31.762 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs debug_inject_read_zeros 2024-02-09T17:26:31.762 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs files list 2024-02-09T17:26:31.762 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs stats 2024-02-09T17:26:31.762 INFO:teuthology.orchestra.run.smithi099.stderr:bluestore allocator dump block 2024-02-09T17:26:31.762 INFO:teuthology.orchestra.run.smithi099.stderr:Error EINVAL: invalid command 2024-02-09T17:26:31.830 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:26:31.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:31 smithi179 ceph-mon[43975]: pgmap v437: 17 pgs: 17 active+clean; 7.6 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 5.2 MiB/s wr, 923 op/s 2024-02-09T17:26:31.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:31 smithi179 ceph-mon[43975]: 1.0 scrub starts 2024-02-09T17:26:31.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:31 smithi179 ceph-mon[43975]: 1.0 scrub ok 2024-02-09T17:26:31.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:31 smithi162 ceph-mon[44304]: pgmap v437: 17 pgs: 17 active+clean; 7.6 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 5.2 MiB/s wr, 923 op/s 2024-02-09T17:26:31.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:31 smithi162 ceph-mon[44304]: 1.0 scrub starts 2024-02-09T17:26:31.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:31 smithi162 ceph-mon[44304]: 1.0 scrub ok 2024-02-09T17:26:31.975 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:26:31.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:31.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:26:32.084 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:26:32.119 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:32.147 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:26:32.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:32.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:26:32.254 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:26:32.316 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:32.587 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:26:32.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:32.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:26:32.696 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:26:32.756 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:33.150 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-09T17:26:33.150 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-02-09T17:26:33.445 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:26:33.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:33.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:26:33.550 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:26:33.585 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:33.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:33 smithi162 ceph-mon[44304]: pgmap v438: 17 pgs: 17 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s wr, 930 op/s 2024-02-09T17:26:33.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:33 smithi099 ceph-mon[39420]: pgmap v438: 17 pgs: 17 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s wr, 930 op/s 2024-02-09T17:26:33.791 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:26:33.830 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:26:33.885 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:26:33.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:33 smithi179 ceph-mon[43975]: pgmap v438: 17 pgs: 17 active+clean; 7.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s wr, 930 op/s 2024-02-09T17:26:33.934 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:26:33.968 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:26:34.012 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:26:34.061 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:26:34.101 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:26:34.130 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:26:34.178 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:26:34.218 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:26:34.248 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set noscrub 2024-02-09T17:26:34.255 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:26:34.257 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:26:34.257 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:26:34.257 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:26:34.257 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:26:34.257 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:26:34.258 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:26:34.258 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:26:34.258 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:26:34.258 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:26:34.258 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:26:34.258 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:26:34.258 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:26:34.706 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:34 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/6112632' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:26:34.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:34 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/6112632' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:26:34.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:34 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/6112632' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:26:35.140 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:26:35.141 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:26:35.147 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:26:35.748 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:35 smithi099 ceph-mon[39420]: pgmap v439: 17 pgs: 17 active+clean; 7.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 939 op/s 2024-02-09T17:26:35.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:35 smithi179 ceph-mon[43975]: pgmap v439: 17 pgs: 17 active+clean; 7.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 939 op/s 2024-02-09T17:26:35.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:35 smithi162 ceph-mon[44304]: pgmap v439: 17 pgs: 17 active+clean; 7.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 939 op/s 2024-02-09T17:26:37.054 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:26:37.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:37 smithi179 ceph-mon[43975]: pgmap v440: 17 pgs: 17 active+clean; 7.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 965 op/s 2024-02-09T17:26:37.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:37 smithi162 ceph-mon[44304]: pgmap v440: 17 pgs: 17 active+clean; 7.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 965 op/s 2024-02-09T17:26:37.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:37 smithi099 ceph-mon[39420]: pgmap v440: 17 pgs: 17 active+clean; 7.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 965 op/s 2024-02-09T17:26:37.546 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:26:37.552 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:37.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:26:37.567 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:26:37.653 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:26:37.702 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:37.858 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:26:37.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:37.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:26:37.965 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:26:38.008 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:38.028 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:26:38.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:38.031 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:26:38.087 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is set 2024-02-09T17:26:38.132 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:26:38.151 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-02-09T17:26:38.152 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:26:38.152 INFO:tasks.thrashosds.thrasher:Killing osd 8, live_osds are [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-02-09T17:26:38.152 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-02-09T17:26:38.152 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-a69c870a-c76e-11ee-95b7-87774f69a715@osd.8 2024-02-09T17:26:38.178 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:38.210 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:26:38.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:38.218 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:26:38.319 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:26:38.371 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:38 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1604836634' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:26:38.371 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:38 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:26:38.374 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:38.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:38 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1604836634' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:26:38.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:38 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:26:38.422 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 17:26:38 smithi179 systemd[1]: Stopping Ceph osd.8 for a69c870a-c76e-11ee-95b7-87774f69a715... 2024-02-09T17:26:38.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:38 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1604836634' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:26:38.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:38 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:26:38.434 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:26:38.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:38.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:26:38.538 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:26:38.574 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:39.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:39 smithi099 ceph-mon[39420]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:26:39.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:39 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:26:39.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:39 smithi099 ceph-mon[39420]: osdmap e145: 12 total, 12 up, 12 in 2024-02-09T17:26:39.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:39 smithi099 ceph-mon[39420]: pgmap v442: 17 pgs: 17 active+clean; 7.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 944 op/s 2024-02-09T17:26:39.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:39 smithi179 ceph-mon[43975]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:26:39.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:39 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:26:39.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:39 smithi179 ceph-mon[43975]: osdmap e145: 12 total, 12 up, 12 in 2024-02-09T17:26:39.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:39 smithi179 ceph-mon[43975]: pgmap v442: 17 pgs: 17 active+clean; 7.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 944 op/s 2024-02-09T17:26:39.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:39 smithi162 ceph-mon[44304]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:26:39.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:39 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:26:39.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:39 smithi162 ceph-mon[44304]: osdmap e145: 12 total, 12 up, 12 in 2024-02-09T17:26:39.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:39 smithi162 ceph-mon[44304]: pgmap v442: 17 pgs: 17 active+clean; 7.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 944 op/s 2024-02-09T17:26:39.530 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-09T17:26:39.557 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:26:39.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:39.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:26:39.661 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:26:39.741 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:39.880 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:26:39.921 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 17:26:39 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[49792]: 2024-02-09T17:26:39.574+0000 7f102466b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T17:26:39.921 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 17:26:39 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[49792]: 2024-02-09T17:26:39.574+0000 7f102466b700 -1 osd.8 145 *** Got signal Terminated *** 2024-02-09T17:26:39.922 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 17:26:39 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[49792]: 2024-02-09T17:26:39.574+0000 7f102466b700 -1 osd.8 145 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-09T17:26:40.008 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:26:40.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:40.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:26:40.123 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:26:40.165 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:40.188 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:26:40.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:40.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:26:40.292 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:26:40.320 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:40.322 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:26:40.341 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:26:40.360 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:26:40.379 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:26:40.392 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:26:40.393 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:40 smithi162 ceph-mon[44304]: osd.8 marked itself down and dead 2024-02-09T17:26:40.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:40.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:26:40.399 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:26:40.418 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:26:40.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:40 smithi179 ceph-mon[43975]: osd.8 marked itself down and dead 2024-02-09T17:26:40.438 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:26:40.457 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:26:40.477 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:26:40.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:40 smithi099 ceph-mon[39420]: osd.8 marked itself down and dead 2024-02-09T17:26:40.496 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:26:40.498 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:26:40.518 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:26:40.518 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:26:40.518 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:26:40.518 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:26:40.518 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:26:40.518 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:26:40.518 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:26:40.518 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:26:40.519 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:26:40.519 INFO:teuthology.orchestra.run.smithi099.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-02-09T17:26:40.519 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:26:40.519 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:26:40.519 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:26:40.522 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:40.541 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:26:40.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:40.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:26:40.645 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:26:40.672 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:40.708 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:26:40.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:40.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:26:40.812 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:26:40.839 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:41.176 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:41 smithi099 ceph-mon[39420]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-09T17:26:41.176 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:41 smithi099 ceph-mon[39420]: osdmap e146: 12 total, 11 up, 12 in 2024-02-09T17:26:41.176 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:41 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2690465579' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:26:41.176 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:41 smithi099 ceph-mon[39420]: pgmap v444: 17 pgs: 3 stale+active+clean, 14 active+clean; 7.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s 2024-02-09T17:26:41.186 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:26:41.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:41.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:26:41.210 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-09T17:26:41.211 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-09T17:26:41.291 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:26:41.362 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:41.393 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:26:41.399 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:41.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:26:41.422 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 17:26:41 smithi179 bash[80413]: ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8 2024-02-09T17:26:41.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:41 smithi179 ceph-mon[43975]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-09T17:26:41.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:41 smithi179 ceph-mon[43975]: osdmap e146: 12 total, 11 up, 12 in 2024-02-09T17:26:41.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:41 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2690465579' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:26:41.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:41 smithi179 ceph-mon[43975]: pgmap v444: 17 pgs: 3 stale+active+clean, 14 active+clean; 7.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s 2024-02-09T17:26:41.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:41 smithi162 ceph-mon[44304]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-09T17:26:41.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:41 smithi162 ceph-mon[44304]: osdmap e146: 12 total, 11 up, 12 in 2024-02-09T17:26:41.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:41 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2690465579' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:26:41.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:41 smithi162 ceph-mon[44304]: pgmap v444: 17 pgs: 3 stale+active+clean, 14 active+clean; 7.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s 2024-02-09T17:26:41.500 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:26:41.568 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:41.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:41.607 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:26:41.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:26:41.707 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-09T17:26:41.708 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:26:41.744 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:41.766 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:26:41.766 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-02-09T17:26:41.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-02-09T17:26:41.766 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-02-09T17:26:41.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.766 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.14790.0:131405 2.5 e142/111 2:a7bdea8b:::benchmark_data_smithi179_77138_object131404:head v 142'8135, mlcod=142'8133)", 2024-02-09T17:26:41.767 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:26:17.332666+0000", 2024-02-09T17:26:41.767 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 24.429876142000001, 2024-02-09T17:26:41.767 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014522178, 2024-02-09T17:26:41.767 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:26:41.767 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:26:41.767 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:26:41.767 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.767 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:26:41.767 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:17.332666+0000", 2024-02-09T17:26:41.767 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:26:41.767 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.767 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.767 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:26:41.768 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:17.332666+0000", 2024-02-09T17:26:41.768 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:26:41.768 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.768 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.768 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:26:41.768 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:17.332667+0000", 2024-02-09T17:26:41.768 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.093e-06 2024-02-09T17:26:41.768 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.768 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.768 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:26:41.768 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:17.332741+0000", 2024-02-09T17:26:41.768 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4190999999999995e-05 2024-02-09T17:26:41.768 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.768 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.769 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:26:41.769 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:17.332755+0000", 2024-02-09T17:26:41.769 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3315999999999999e-05 2024-02-09T17:26:41.769 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.769 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.769 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:26:41.769 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:17.332763+0000", 2024-02-09T17:26:41.769 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0649999999999999e-06 2024-02-09T17:26:41.769 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.769 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.769 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:26:41.769 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:17.332809+0000", 2024-02-09T17:26:41.769 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6755000000000002e-05 2024-02-09T17:26:41.769 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:26:41.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:17.332826+0000", 2024-02-09T17:26:41.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6487999999999999e-05 2024-02-09T17:26:41.770 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:26:41.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:17.347141+0000", 2024-02-09T17:26:41.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014314742 2024-02-09T17:26:41.770 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:26:41.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:17.347188+0000", 2024-02-09T17:26:41.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7528000000000003e-05 2024-02-09T17:26:41.771 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:26:41.771 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:26:41.771 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:26:41.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.771 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.14790.0:135327 2.d e143/111 2:b3926817:::benchmark_data_smithi179_77138_object1919:head v 143'8597, mlcod=143'8595)", 2024-02-09T17:26:41.771 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:26:21.644299+0000", 2024-02-09T17:26:41.771 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 20.118242753000001, 2024-02-09T17:26:41.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014915266999999999, 2024-02-09T17:26:41.771 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:26:41.771 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:26:41.771 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:26:41.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.771 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:26:41.772 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:21.644299+0000", 2024-02-09T17:26:41.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:26:41.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.772 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.772 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:26:41.772 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:21.644299+0000", 2024-02-09T17:26:41.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:26:41.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.772 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.772 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:26:41.772 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:21.644300+0000", 2024-02-09T17:26:41.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9500000000000002e-07 2024-02-09T17:26:41.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.773 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:26:41.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:21.644306+0000", 2024-02-09T17:26:41.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6389999999999997e-06 2024-02-09T17:26:41.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.773 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:26:41.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:21.644312+0000", 2024-02-09T17:26:41.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.815e-06 2024-02-09T17:26:41.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.773 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:26:41.774 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:21.644317+0000", 2024-02-09T17:26:41.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4950000000000002e-06 2024-02-09T17:26:41.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.774 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.774 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:26:41.774 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:21.644362+0000", 2024-02-09T17:26:41.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4536999999999997e-05 2024-02-09T17:26:41.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.774 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.774 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:26:41.774 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:21.644381+0000", 2024-02-09T17:26:41.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9930000000000001e-05 2024-02-09T17:26:41.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.774 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.774 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:26:41.775 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:21.659168+0000", 2024-02-09T17:26:41.775 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014786875999999999 2024-02-09T17:26:41.775 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.775 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.775 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:26:41.775 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:21.659215+0000", 2024-02-09T17:26:41.775 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.638e-05 2024-02-09T17:26:41.775 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:26:41.775 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:26:41.775 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:26:41.775 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.775 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.775 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.14790.0:135335 2.d e143/111 2:b084792f:::benchmark_data_smithi179_77138_object1927:head v 143'8598, mlcod=143'8595)", 2024-02-09T17:26:41.775 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:26:21.644463+0000", 2024-02-09T17:26:41.776 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 20.118079181999999, 2024-02-09T17:26:41.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014790807, 2024-02-09T17:26:41.776 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:26:41.776 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:26:41.776 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:26:41.776 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.776 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:26:41.776 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:21.644463+0000", 2024-02-09T17:26:41.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:26:41.776 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.776 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.776 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:26:41.776 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:21.644463+0000", 2024-02-09T17:26:41.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:26:41.776 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:26:41.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:21.644464+0000", 2024-02-09T17:26:41.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3099999999999997e-07 2024-02-09T17:26:41.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:26:41.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:21.644469+0000", 2024-02-09T17:26:41.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5659999999999997e-06 2024-02-09T17:26:41.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:26:41.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:21.644480+0000", 2024-02-09T17:26:41.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0713999999999999e-05 2024-02-09T17:26:41.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:26:41.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:21.644486+0000", 2024-02-09T17:26:41.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2659999999999998e-06 2024-02-09T17:26:41.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:26:41.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:21.648200+0000", 2024-02-09T17:26:41.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0037136640000000002 2024-02-09T17:26:41.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:26:41.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:21.648222+0000", 2024-02-09T17:26:41.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2603e-05 2024-02-09T17:26:41.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:26:41.779 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:21.659218+0000", 2024-02-09T17:26:41.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010995477 2024-02-09T17:26:41.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.779 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:26:41.779 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:21.659254+0000", 2024-02-09T17:26:41.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5886000000000002e-05 2024-02-09T17:26:41.779 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:26:41.779 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:26:41.779 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:26:41.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.779 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.14790.0:137970 2.d e143/111 2:b3977e31:::benchmark_data_smithi179_77138_object4562:head v 143'8753, mlcod=143'8750)", 2024-02-09T17:26:41.779 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:26:24.648041+0000", 2024-02-09T17:26:41.779 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 17.114501293, 2024-02-09T17:26:41.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014274425, 2024-02-09T17:26:41.780 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:26:41.780 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:26:41.780 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:26:41.780 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.780 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:26:41.780 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:24.648041+0000", 2024-02-09T17:26:41.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:26:41.780 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.780 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.780 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:26:41.780 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:24.648041+0000", 2024-02-09T17:26:41.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:26:41.780 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.780 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.780 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:26:41.780 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:24.648042+0000", 2024-02-09T17:26:41.781 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0500000000000003e-07 2024-02-09T17:26:41.781 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.781 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:26:41.781 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:24.648046+0000", 2024-02-09T17:26:41.781 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.673e-06 2024-02-09T17:26:41.781 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.781 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:26:41.781 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:24.648055+0000", 2024-02-09T17:26:41.781 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0489999999999993e-06 2024-02-09T17:26:41.781 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.781 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:26:41.781 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:24.648063+0000", 2024-02-09T17:26:41.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3969999999999997e-06 2024-02-09T17:26:41.782 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.782 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.782 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:26:41.782 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:24.649395+0000", 2024-02-09T17:26:41.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001332216 2024-02-09T17:26:41.782 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.782 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.782 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:26:41.782 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:24.649412+0000", 2024-02-09T17:26:41.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7150999999999999e-05 2024-02-09T17:26:41.782 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.782 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.782 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:26:41.783 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:24.662275+0000", 2024-02-09T17:26:41.783 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012862946 2024-02-09T17:26:41.783 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.783 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:26:41.783 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:24.662315+0000", 2024-02-09T17:26:41.783 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0388e-05 2024-02-09T17:26:41.783 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:26:41.783 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:26:41.783 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:26:41.783 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.783 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.14790.0:141664 2.d e143/111 2:bfcdd0bf:::benchmark_data_smithi179_77138_object8256:head v 143'8989, mlcod=143'8987)", 2024-02-09T17:26:41.783 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:26:28.390241+0000", 2024-02-09T17:26:41.783 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 13.372300797999999, 2024-02-09T17:26:41.784 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.016689703, 2024-02-09T17:26:41.784 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:26:41.784 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:26:41.784 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:26:41.784 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.784 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:26:41.784 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:28.390241+0000", 2024-02-09T17:26:41.784 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:26:41.784 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.784 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.784 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:26:41.784 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:28.390241+0000", 2024-02-09T17:26:41.784 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:26:41.784 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.785 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.785 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:26:41.785 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:28.390243+0000", 2024-02-09T17:26:41.785 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.212e-06 2024-02-09T17:26:41.785 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.785 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.785 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:26:41.785 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:28.390251+0000", 2024-02-09T17:26:41.785 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1669999999999999e-06 2024-02-09T17:26:41.785 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.785 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.785 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:26:41.785 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:28.390264+0000", 2024-02-09T17:26:41.785 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2789000000000001e-05 2024-02-09T17:26:41.786 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.786 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.786 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:26:41.786 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:28.390272+0000", 2024-02-09T17:26:41.786 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9400000000000002e-06 2024-02-09T17:26:41.786 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.786 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.786 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:26:41.786 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:28.390338+0000", 2024-02-09T17:26:41.786 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6481000000000003e-05 2024-02-09T17:26:41.786 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.786 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.786 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:26:41.786 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:28.390358+0000", 2024-02-09T17:26:41.786 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9947999999999999e-05 2024-02-09T17:26:41.787 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.787 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:26:41.787 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:28.406888+0000", 2024-02-09T17:26:41.787 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.016529907999999999 2024-02-09T17:26:41.787 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.787 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:26:41.787 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:28.406931+0000", 2024-02-09T17:26:41.787 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3257999999999998e-05 2024-02-09T17:26:41.787 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:26:41.787 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:26:41.787 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:26:41.787 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.788 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.14790.0:142492 2.5 e143/111 2:af856e1f:::benchmark_data_smithi179_77138_object9084:head v 143'8823, mlcod=143'8821)", 2024-02-09T17:26:41.788 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:26:29.283472+0000", 2024-02-09T17:26:41.788 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 12.479069981, 2024-02-09T17:26:41.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015877091999999999, 2024-02-09T17:26:41.788 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:26:41.788 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:26:41.788 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:26:41.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.788 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:26:41.788 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.283472+0000", 2024-02-09T17:26:41.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:26:41.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.789 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:26:41.789 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.283472+0000", 2024-02-09T17:26:41.789 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:26:41.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.789 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:26:41.789 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.283474+0000", 2024-02-09T17:26:41.789 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5820000000000001e-06 2024-02-09T17:26:41.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.789 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:26:41.789 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.283487+0000", 2024-02-09T17:26:41.789 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3011e-05 2024-02-09T17:26:41.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.789 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:26:41.789 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.283501+0000", 2024-02-09T17:26:41.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4619e-05 2024-02-09T17:26:41.790 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.790 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.790 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:26:41.790 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.283509+0000", 2024-02-09T17:26:41.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9389999999999997e-06 2024-02-09T17:26:41.790 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.790 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.790 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:26:41.790 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.283571+0000", 2024-02-09T17:26:41.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2020999999999995e-05 2024-02-09T17:26:41.790 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.790 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.790 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:26:41.790 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.283594+0000", 2024-02-09T17:26:41.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2680999999999998e-05 2024-02-09T17:26:41.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:26:41.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.299294+0000", 2024-02-09T17:26:41.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015699787999999999 2024-02-09T17:26:41.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:26:41.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.299349+0000", 2024-02-09T17:26:41.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5451000000000002e-05 2024-02-09T17:26:41.791 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:26:41.791 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:26:41.791 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:26:41.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.792 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.14790.0:142984 2.d e144/111 2:bc962ec1:::benchmark_data_smithi179_77138_object9576:head v 144'9075, mlcod=144'9072)", 2024-02-09T17:26:41.792 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:26:29.764472+0000", 2024-02-09T17:26:41.792 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 11.998069888, 2024-02-09T17:26:41.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.022860015000000001, 2024-02-09T17:26:41.792 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:26:41.792 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:26:41.792 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:26:41.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.792 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:26:41.792 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.764472+0000", 2024-02-09T17:26:41.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:26:41.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.792 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:26:41.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.764472+0000", 2024-02-09T17:26:41.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:26:41.793 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.793 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.793 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:26:41.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.764473+0000", 2024-02-09T17:26:41.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1999999999999998e-07 2024-02-09T17:26:41.793 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.793 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.793 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:26:41.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.764480+0000", 2024-02-09T17:26:41.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7349999999999999e-06 2024-02-09T17:26:41.793 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.793 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.794 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:26:41.794 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.764492+0000", 2024-02-09T17:26:41.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1861999999999999e-05 2024-02-09T17:26:41.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.794 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:26:41.794 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.764500+0000", 2024-02-09T17:26:41.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6890000000000004e-06 2024-02-09T17:26:41.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.794 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:26:41.794 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.764546+0000", 2024-02-09T17:26:41.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6901999999999999e-05 2024-02-09T17:26:41.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.795 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.795 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:26:41.795 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.764566+0000", 2024-02-09T17:26:41.795 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.969e-05 2024-02-09T17:26:41.795 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.795 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.795 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:26:41.795 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.787304+0000", 2024-02-09T17:26:41.795 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.022737812 2024-02-09T17:26:41.795 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.795 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.795 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:26:41.795 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.787332+0000", 2024-02-09T17:26:41.795 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8405e-05 2024-02-09T17:26:41.795 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:26:41.796 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:26:41.796 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:26:41.796 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.796 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.796 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.14790.0:142974 2.5 e144/111 2:a36110d8:::benchmark_data_smithi179_77138_object9566:head v 144'8866, mlcod=144'8863)", 2024-02-09T17:26:41.796 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:26:29.767275+0000", 2024-02-09T17:26:41.796 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 11.995267025, 2024-02-09T17:26:41.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020025202999999998, 2024-02-09T17:26:41.796 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:26:41.796 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:26:41.796 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:26:41.796 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.796 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:26:41.796 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.767275+0000", 2024-02-09T17:26:41.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:26:41.797 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.797 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.797 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:26:41.797 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.767275+0000", 2024-02-09T17:26:41.797 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:26:41.797 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.797 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.797 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:26:41.797 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.767276+0000", 2024-02-09T17:26:41.797 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.08e-06 2024-02-09T17:26:41.797 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.797 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.797 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:26:41.797 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.767288+0000", 2024-02-09T17:26:41.797 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1504e-05 2024-02-09T17:26:41.797 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.798 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.798 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:26:41.798 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.767299+0000", 2024-02-09T17:26:41.798 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1226000000000001e-05 2024-02-09T17:26:41.798 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.798 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.798 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:26:41.798 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.767306+0000", 2024-02-09T17:26:41.798 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6189999999999996e-06 2024-02-09T17:26:41.798 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.798 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.798 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:26:41.798 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.767355+0000", 2024-02-09T17:26:41.798 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9001000000000001e-05 2024-02-09T17:26:41.798 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.799 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:26:41.799 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.767382+0000", 2024-02-09T17:26:41.799 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7690000000000001e-05 2024-02-09T17:26:41.799 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.799 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:26:41.799 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.787246+0000", 2024-02-09T17:26:41.799 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019863512999999999 2024-02-09T17:26:41.799 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.799 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:26:41.799 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.787300+0000", 2024-02-09T17:26:41.799 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4570000000000001e-05 2024-02-09T17:26:41.799 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:26:41.799 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:26:41.800 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:26:41.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.800 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.800 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.14790.0:142979 2.5 e144/111 2:a3730a5e:::benchmark_data_smithi179_77138_object9571:head v 144'8867, mlcod=144'8863)", 2024-02-09T17:26:41.800 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:26:29.767353+0000", 2024-02-09T17:26:41.800 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 11.995189043, 2024-02-09T17:26:41.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020002255, 2024-02-09T17:26:41.800 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:26:41.800 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:26:41.800 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:26:41.800 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.800 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:26:41.800 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.767353+0000", 2024-02-09T17:26:41.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:26:41.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.800 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.800 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:26:41.801 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.767353+0000", 2024-02-09T17:26:41.801 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:26:41.801 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.801 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.801 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:26:41.801 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.767353+0000", 2024-02-09T17:26:41.801 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9499999999999998e-07 2024-02-09T17:26:41.801 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.801 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.801 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:26:41.801 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.767357+0000", 2024-02-09T17:26:41.801 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.642e-06 2024-02-09T17:26:41.801 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.801 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.801 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:26:41.802 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.767369+0000", 2024-02-09T17:26:41.802 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2033e-05 2024-02-09T17:26:41.802 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.802 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.802 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:26:41.802 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.767374+0000", 2024-02-09T17:26:41.802 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1789999999999999e-06 2024-02-09T17:26:41.802 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.802 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.802 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:26:41.802 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.770756+0000", 2024-02-09T17:26:41.802 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0033820170000000002 2024-02-09T17:26:41.802 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.802 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.802 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:26:41.802 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.770776+0000", 2024-02-09T17:26:41.803 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9986000000000001e-05 2024-02-09T17:26:41.803 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.803 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.803 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:26:41.803 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.787334+0000", 2024-02-09T17:26:41.803 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.016557788 2024-02-09T17:26:41.803 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.803 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.803 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:26:41.803 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.787355+0000", 2024-02-09T17:26:41.803 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1314999999999998e-05 2024-02-09T17:26:41.803 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:26:41.803 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:26:41.803 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:26:41.803 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.803 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.804 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.14790.0:142988 2.5 e144/111 2:a5792247:::benchmark_data_smithi179_77138_object9580:head v 144'8868, mlcod=144'8864)", 2024-02-09T17:26:41.804 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:26:29.770487+0000", 2024-02-09T17:26:41.804 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 11.992054808000001, 2024-02-09T17:26:41.804 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.016892588, 2024-02-09T17:26:41.804 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:26:41.804 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:26:41.804 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:26:41.804 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.804 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:26:41.804 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.770487+0000", 2024-02-09T17:26:41.804 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:26:41.804 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.804 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.804 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:26:41.804 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.770487+0000", 2024-02-09T17:26:41.804 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:26:41.805 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.805 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.805 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:26:41.805 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.770488+0000", 2024-02-09T17:26:41.805 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1399999999999995e-07 2024-02-09T17:26:41.805 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.805 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.805 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:26:41.805 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.770496+0000", 2024-02-09T17:26:41.805 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3200000000000002e-06 2024-02-09T17:26:41.805 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.805 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.805 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:26:41.805 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.770510+0000", 2024-02-09T17:26:41.805 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4688999999999999e-05 2024-02-09T17:26:41.806 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.806 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.806 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:26:41.806 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.770518+0000", 2024-02-09T17:26:41.806 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1079999999999993e-06 2024-02-09T17:26:41.806 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.806 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.806 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:26:41.806 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.774701+0000", 2024-02-09T17:26:41.806 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041827100000000001 2024-02-09T17:26:41.806 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.806 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.806 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:26:41.806 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.774724+0000", 2024-02-09T17:26:41.806 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3144999999999999e-05 2024-02-09T17:26:41.806 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.807 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.807 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:26:41.807 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.787359+0000", 2024-02-09T17:26:41.807 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012634256 2024-02-09T17:26:41.807 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.807 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.807 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:26:41.807 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:29.787380+0000", 2024-02-09T17:26:41.807 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1446000000000001e-05 2024-02-09T17:26:41.807 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:26:41.807 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:26:41.807 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:26:41.807 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.807 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.807 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.14790.0:144452 2.5 e144/111 2:a3ac926d:::benchmark_data_smithi179_77138_object11044:head v 144'8949, mlcod=144'8945)", 2024-02-09T17:26:41.807 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:26:31.412478+0000", 2024-02-09T17:26:41.808 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 10.350063860000001, 2024-02-09T17:26:41.808 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015175085, 2024-02-09T17:26:41.808 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:26:41.808 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:26:41.808 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:26:41.808 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.808 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:26:41.808 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:31.412478+0000", 2024-02-09T17:26:41.808 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:26:41.808 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.808 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.808 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:26:41.808 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:31.412478+0000", 2024-02-09T17:26:41.808 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:26:41.809 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.809 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.809 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:26:41.809 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:31.412479+0000", 2024-02-09T17:26:41.809 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.156e-06 2024-02-09T17:26:41.809 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.809 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.809 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:26:41.809 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:31.412486+0000", 2024-02-09T17:26:41.809 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3790000000000003e-06 2024-02-09T17:26:41.809 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.809 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.809 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:26:41.809 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:31.412496+0000", 2024-02-09T17:26:41.810 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0427e-05 2024-02-09T17:26:41.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.810 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.810 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:26:41.810 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:31.412504+0000", 2024-02-09T17:26:41.810 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3320000000000003e-06 2024-02-09T17:26:41.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.810 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.810 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:26:41.810 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:31.415151+0000", 2024-02-09T17:26:41.810 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0026472039999999998 2024-02-09T17:26:41.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.810 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.810 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:26:41.811 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:31.415171+0000", 2024-02-09T17:26:41.811 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9760999999999999e-05 2024-02-09T17:26:41.811 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.811 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.811 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:26:41.811 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:31.427601+0000", 2024-02-09T17:26:41.811 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012429924 2024-02-09T17:26:41.811 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.811 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.811 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:26:41.811 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:31.427653+0000", 2024-02-09T17:26:41.811 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2902000000000003e-05 2024-02-09T17:26:41.811 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:26:41.811 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:26:41.811 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:26:41.812 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.812 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.812 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.14790.0:144456 2.5 e144/111 2:a1aba87b:::benchmark_data_smithi179_77138_object11048:head v 144'8950, mlcod=144'8946)", 2024-02-09T17:26:41.812 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:26:31.416186+0000", 2024-02-09T17:26:41.812 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 10.346356441999999, 2024-02-09T17:26:41.812 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015608059000000001, 2024-02-09T17:26:41.812 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:26:41.812 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:26:41.812 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:26:41.812 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.812 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:26:41.812 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:31.416186+0000", 2024-02-09T17:26:41.812 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:26:41.812 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.812 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.813 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:26:41.813 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:31.416186+0000", 2024-02-09T17:26:41.813 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:26:41.813 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.813 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.813 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:26:41.813 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:31.416186+0000", 2024-02-09T17:26:41.813 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9699999999999995e-07 2024-02-09T17:26:41.813 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.813 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.813 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:26:41.813 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:31.416193+0000", 2024-02-09T17:26:41.813 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.426e-06 2024-02-09T17:26:41.813 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.813 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.814 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:26:41.814 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:31.416204+0000", 2024-02-09T17:26:41.814 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0642000000000001e-05 2024-02-09T17:26:41.814 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.814 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.814 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:26:41.814 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:31.416210+0000", 2024-02-09T17:26:41.814 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1689999999999997e-06 2024-02-09T17:26:41.814 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.814 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.814 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:26:41.814 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:31.419113+0000", 2024-02-09T17:26:41.814 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0029035829999999999 2024-02-09T17:26:41.814 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.814 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.814 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:26:41.815 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:31.419127+0000", 2024-02-09T17:26:41.815 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3892e-05 2024-02-09T17:26:41.815 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.815 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.815 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:26:41.815 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:31.431747+0000", 2024-02-09T17:26:41.815 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012620189 2024-02-09T17:26:41.815 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.815 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.815 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:26:41.815 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:31.431794+0000", 2024-02-09T17:26:41.815 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6461000000000001e-05 2024-02-09T17:26:41.815 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:26:41.815 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:26:41.816 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:26:41.816 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.816 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.816 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.14790.0:145445 2.f e144/111 2:fddf6f62:::benchmark_data_smithi179_77138_object12037:head v 144'9163, mlcod=144'9161)", 2024-02-09T17:26:41.816 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:26:32.388911+0000", 2024-02-09T17:26:41.816 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 9.3736310970000005, 2024-02-09T17:26:41.816 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.01743043, 2024-02-09T17:26:41.816 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:26:41.816 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:26:41.816 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:26:41.816 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.816 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:26:41.816 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:32.388911+0000", 2024-02-09T17:26:41.816 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:26:41.817 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.817 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.817 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:26:41.817 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:32.388911+0000", 2024-02-09T17:26:41.817 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:26:41.817 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.817 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.817 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:26:41.817 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:32.388912+0000", 2024-02-09T17:26:41.817 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3170000000000001e-06 2024-02-09T17:26:41.817 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.817 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.817 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:26:41.817 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:32.388922+0000", 2024-02-09T17:26:41.817 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4439999999999997e-06 2024-02-09T17:26:41.818 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.818 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.818 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:26:41.818 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:32.388938+0000", 2024-02-09T17:26:41.818 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6507e-05 2024-02-09T17:26:41.818 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.818 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.818 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:26:41.818 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:32.388948+0000", 2024-02-09T17:26:41.818 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7089999999999997e-06 2024-02-09T17:26:41.818 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.818 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.818 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:26:41.818 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:32.389012+0000", 2024-02-09T17:26:41.818 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4191999999999998e-05 2024-02-09T17:26:41.819 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.819 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.819 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:26:41.819 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:32.389040+0000", 2024-02-09T17:26:41.819 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7351e-05 2024-02-09T17:26:41.819 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.819 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.819 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:26:41.819 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:32.406292+0000", 2024-02-09T17:26:41.819 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.017252640999999999 2024-02-09T17:26:41.819 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.819 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.819 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:26:41.819 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:32.406342+0000", 2024-02-09T17:26:41.820 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9268999999999999e-05 2024-02-09T17:26:41.820 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:26:41.820 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:26:41.820 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:26:41.820 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.820 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.820 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.14790.0:145448 2.f e144/111 2:f292e58f:::benchmark_data_smithi179_77138_object12040:head v 144'9164, mlcod=144'9161)", 2024-02-09T17:26:41.820 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:26:32.391295+0000", 2024-02-09T17:26:41.820 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 9.3712468149999992, 2024-02-09T17:26:41.820 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015097401999999999, 2024-02-09T17:26:41.820 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:26:41.820 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:26:41.820 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:26:41.820 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.820 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:26:41.820 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:32.391295+0000", 2024-02-09T17:26:41.821 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:26:41.821 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.821 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.821 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:26:41.821 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:32.391295+0000", 2024-02-09T17:26:41.821 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:26:41.821 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.821 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.821 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:26:41.821 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:32.391297+0000", 2024-02-09T17:26:41.821 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.176e-06 2024-02-09T17:26:41.821 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.821 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.821 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:26:41.821 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:32.391310+0000", 2024-02-09T17:26:41.822 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3023000000000001e-05 2024-02-09T17:26:41.822 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.822 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.822 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:26:41.822 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:32.391322+0000", 2024-02-09T17:26:41.822 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.289e-05 2024-02-09T17:26:41.822 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.822 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.822 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:26:41.822 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:32.391332+0000", 2024-02-09T17:26:41.822 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1900000000000001e-06 2024-02-09T17:26:41.822 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.822 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.823 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:26:41.823 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:32.393315+0000", 2024-02-09T17:26:41.823 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0019833820000000001 2024-02-09T17:26:41.823 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.823 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.823 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:26:41.823 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:32.393336+0000", 2024-02-09T17:26:41.823 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0916000000000001e-05 2024-02-09T17:26:41.823 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.823 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.823 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:26:41.823 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:32.406358+0000", 2024-02-09T17:26:41.823 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013022229999999999 2024-02-09T17:26:41.823 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.824 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.824 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:26:41.824 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:32.406393+0000", 2024-02-09T17:26:41.824 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4595000000000003e-05 2024-02-09T17:26:41.824 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:26:41.824 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:26:41.824 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:26:41.824 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.824 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.824 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.14790.0:146134 2.5 e144/111 2:aad1ce9e:::benchmark_data_smithi179_77138_object12726:head v 144'9046, mlcod=144'9044)", 2024-02-09T17:26:41.824 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:26:33.076071+0000", 2024-02-09T17:26:41.824 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 8.6864710810000005, 2024-02-09T17:26:41.824 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.016052522999999999, 2024-02-09T17:26:41.824 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:26:41.824 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:26:41.825 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:26:41.825 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.825 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:26:41.825 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.076071+0000", 2024-02-09T17:26:41.825 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:26:41.825 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.825 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.825 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:26:41.825 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.076071+0000", 2024-02-09T17:26:41.825 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:26:41.825 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.825 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.825 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:26:41.825 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.076072+0000", 2024-02-09T17:26:41.825 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4799999999999997e-07 2024-02-09T17:26:41.826 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.826 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.826 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:26:41.826 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.076079+0000", 2024-02-09T17:26:41.826 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7399999999999998e-06 2024-02-09T17:26:41.826 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.826 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.826 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:26:41.826 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.076089+0000", 2024-02-09T17:26:41.826 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0468e-05 2024-02-09T17:26:41.826 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.826 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.826 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:26:41.826 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.076096+0000", 2024-02-09T17:26:41.826 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.601e-06 2024-02-09T17:26:41.826 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.827 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.827 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:26:41.827 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.076139+0000", 2024-02-09T17:26:41.827 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2840999999999999e-05 2024-02-09T17:26:41.827 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.827 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.827 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:26:41.827 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.076155+0000", 2024-02-09T17:26:41.827 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6396999999999999e-05 2024-02-09T17:26:41.827 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.827 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.827 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:26:41.827 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.092075+0000", 2024-02-09T17:26:41.827 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015919698999999999 2024-02-09T17:26:41.827 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.827 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.828 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:26:41.828 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.092124+0000", 2024-02-09T17:26:41.828 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8829000000000003e-05 2024-02-09T17:26:41.828 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:26:41.828 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:26:41.828 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:26:41.828 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.828 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.828 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.14790.0:146132 2.d e144/111 2:bd0e6828:::benchmark_data_smithi179_77138_object12724:head v 144'9298, mlcod=144'9296)", 2024-02-09T17:26:41.828 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:26:33.076390+0000", 2024-02-09T17:26:41.828 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 8.6861525779999997, 2024-02-09T17:26:41.828 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015777005, 2024-02-09T17:26:41.828 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:26:41.828 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:26:41.828 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:26:41.828 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.828 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:26:41.829 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.076390+0000", 2024-02-09T17:26:41.829 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:26:41.829 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.829 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.829 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:26:41.829 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.076390+0000", 2024-02-09T17:26:41.829 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:26:41.829 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.829 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.829 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:26:41.829 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.076390+0000", 2024-02-09T17:26:41.829 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1899999999999998e-07 2024-02-09T17:26:41.829 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.829 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.829 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:26:41.829 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.076393+0000", 2024-02-09T17:26:41.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.219e-06 2024-02-09T17:26:41.830 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.830 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.830 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:26:41.830 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.076399+0000", 2024-02-09T17:26:41.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2790000000000004e-06 2024-02-09T17:26:41.830 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.830 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.830 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:26:41.830 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.076403+0000", 2024-02-09T17:26:41.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7340000000000002e-06 2024-02-09T17:26:41.830 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.830 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.830 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:26:41.830 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.076443+0000", 2024-02-09T17:26:41.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9894000000000001e-05 2024-02-09T17:26:41.830 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.831 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.831 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:26:41.831 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.076466+0000", 2024-02-09T17:26:41.831 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2680999999999998e-05 2024-02-09T17:26:41.831 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.831 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.831 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:26:41.831 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.092134+0000", 2024-02-09T17:26:41.831 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015668578999999998 2024-02-09T17:26:41.831 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.831 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.831 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:26:41.831 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.092167+0000", 2024-02-09T17:26:41.831 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2299999999999999e-05 2024-02-09T17:26:41.831 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:26:41.831 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:26:41.832 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:26:41.832 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.832 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.832 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.14790.0:146137 2.5 e144/111 2:af5bd374:::benchmark_data_smithi179_77138_object12729:head v 144'9047, mlcod=144'9044)", 2024-02-09T17:26:41.832 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:26:33.077416+0000", 2024-02-09T17:26:41.832 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 8.6851260180000001, 2024-02-09T17:26:41.832 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014779258999999999, 2024-02-09T17:26:41.832 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:26:41.832 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:26:41.832 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:26:41.832 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.832 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:26:41.832 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.077416+0000", 2024-02-09T17:26:41.833 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:26:41.833 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.833 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.833 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:26:41.833 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.077416+0000", 2024-02-09T17:26:41.833 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:26:41.833 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.833 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.833 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:26:41.833 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.077417+0000", 2024-02-09T17:26:41.833 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5799999999999999e-07 2024-02-09T17:26:41.833 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.833 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.833 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:26:41.834 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.077422+0000", 2024-02-09T17:26:41.834 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9779999999999996e-06 2024-02-09T17:26:41.834 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.834 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.834 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:26:41.834 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.077431+0000", 2024-02-09T17:26:41.834 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2420000000000006e-06 2024-02-09T17:26:41.834 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.834 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.834 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:26:41.834 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.077436+0000", 2024-02-09T17:26:41.834 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.254e-06 2024-02-09T17:26:41.834 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.834 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.835 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:26:41.835 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.079513+0000", 2024-02-09T17:26:41.835 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0020768359999999999 2024-02-09T17:26:41.835 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.835 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.835 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:26:41.835 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.079536+0000", 2024-02-09T17:26:41.835 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3062999999999999e-05 2024-02-09T17:26:41.835 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.835 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.835 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:26:41.835 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.092169+0000", 2024-02-09T17:26:41.835 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0126324 2024-02-09T17:26:41.835 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.835 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.835 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:26:41.836 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.092195+0000", 2024-02-09T17:26:41.836 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6828000000000001e-05 2024-02-09T17:26:41.836 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:26:41.836 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:26:41.836 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:26:41.836 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.836 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.836 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.14790.0:146145 2.5 e144/111 2:af741e91:::benchmark_data_smithi179_77138_object12737:head v 144'9048, mlcod=144'9044)", 2024-02-09T17:26:41.836 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:26:33.081389+0000", 2024-02-09T17:26:41.836 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 8.6811528750000004, 2024-02-09T17:26:41.836 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018904048, 2024-02-09T17:26:41.836 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:26:41.836 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:26:41.836 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:26:41.836 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.836 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:26:41.837 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.081389+0000", 2024-02-09T17:26:41.837 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:26:41.837 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.837 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.837 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:26:41.837 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.081389+0000", 2024-02-09T17:26:41.837 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:26:41.837 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.837 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.837 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:26:41.837 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.081390+0000", 2024-02-09T17:26:41.837 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3299999999999998e-07 2024-02-09T17:26:41.837 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.837 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.837 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:26:41.838 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.081394+0000", 2024-02-09T17:26:41.838 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4209999999999997e-06 2024-02-09T17:26:41.838 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.838 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.838 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:26:41.838 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.081401+0000", 2024-02-09T17:26:41.838 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5740000000000001e-06 2024-02-09T17:26:41.838 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.838 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.838 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:26:41.838 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.081406+0000", 2024-02-09T17:26:41.838 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1830000000000002e-06 2024-02-09T17:26:41.838 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.838 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.838 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:26:41.838 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.083541+0000", 2024-02-09T17:26:41.839 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021354350000000002 2024-02-09T17:26:41.839 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.839 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.839 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:26:41.839 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.083562+0000", 2024-02-09T17:26:41.839 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0846e-05 2024-02-09T17:26:41.839 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.839 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.839 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:26:41.839 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.100242+0000", 2024-02-09T17:26:41.839 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.016680118000000001 2024-02-09T17:26:41.839 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.839 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.839 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:26:41.839 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.100293+0000", 2024-02-09T17:26:41.840 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1137999999999997e-05 2024-02-09T17:26:41.840 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:26:41.840 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:26:41.840 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:26:41.840 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.840 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.840 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.14790.0:146452 2.5 e144/111 2:a10afa6b:::benchmark_data_smithi179_77138_object13044:head v 144'9078, mlcod=144'9074)", 2024-02-09T17:26:41.840 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:26:33.389180+0000", 2024-02-09T17:26:41.840 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 8.37336189, 2024-02-09T17:26:41.840 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.016823806, 2024-02-09T17:26:41.840 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:26:41.840 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:26:41.840 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:26:41.840 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.840 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:26:41.840 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.389180+0000", 2024-02-09T17:26:41.841 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:26:41.841 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.841 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.841 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:26:41.841 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.389180+0000", 2024-02-09T17:26:41.841 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:26:41.841 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.841 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.841 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:26:41.841 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.389181+0000", 2024-02-09T17:26:41.841 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2099999999999995e-07 2024-02-09T17:26:41.841 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.841 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.841 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:26:41.841 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.389187+0000", 2024-02-09T17:26:41.841 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3419999999999996e-06 2024-02-09T17:26:41.841 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.842 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.842 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:26:41.842 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.389201+0000", 2024-02-09T17:26:41.842 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3407999999999999e-05 2024-02-09T17:26:41.842 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.842 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.842 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:26:41.842 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.389208+0000", 2024-02-09T17:26:41.842 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2590000000000003e-06 2024-02-09T17:26:41.842 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.842 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.842 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:26:41.842 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.391128+0000", 2024-02-09T17:26:41.842 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001919373 2024-02-09T17:26:41.842 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.843 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.843 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:26:41.843 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.391151+0000", 2024-02-09T17:26:41.843 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3459e-05 2024-02-09T17:26:41.843 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.843 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.843 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:26:41.843 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.405968+0000", 2024-02-09T17:26:41.843 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014817359 2024-02-09T17:26:41.843 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.843 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.843 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:26:41.843 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.406004+0000", 2024-02-09T17:26:41.843 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5784999999999997e-05 2024-02-09T17:26:41.843 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:26:41.843 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:26:41.844 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:26:41.844 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.844 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.844 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.14790.0:146453 2.d e144/111 2:b2507c19:::benchmark_data_smithi179_77138_object13045:head v 144'9318, mlcod=144'9315)", 2024-02-09T17:26:41.844 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:26:33.391517+0000", 2024-02-09T17:26:41.844 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 8.3710252979999993, 2024-02-09T17:26:41.844 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014506744, 2024-02-09T17:26:41.844 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:26:41.844 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:26:41.844 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:26:41.844 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.844 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:26:41.844 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.391517+0000", 2024-02-09T17:26:41.844 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:26:41.844 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.845 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.845 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:26:41.845 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.391517+0000", 2024-02-09T17:26:41.845 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:26:41.845 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.845 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.845 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:26:41.845 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.391518+0000", 2024-02-09T17:26:41.845 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5099999999999998e-07 2024-02-09T17:26:41.845 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.845 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.845 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:26:41.845 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.391525+0000", 2024-02-09T17:26:41.845 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6340000000000001e-06 2024-02-09T17:26:41.845 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.845 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.846 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:26:41.846 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.391537+0000", 2024-02-09T17:26:41.846 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1389999999999999e-05 2024-02-09T17:26:41.846 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.846 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.846 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:26:41.846 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.391544+0000", 2024-02-09T17:26:41.846 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.52e-06 2024-02-09T17:26:41.846 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.846 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.846 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:26:41.846 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.392391+0000", 2024-02-09T17:26:41.846 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000847049 2024-02-09T17:26:41.846 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.846 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.846 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:26:41.847 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.392405+0000", 2024-02-09T17:26:41.847 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3339000000000001e-05 2024-02-09T17:26:41.847 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.847 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.847 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:26:41.847 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.406007+0000", 2024-02-09T17:26:41.847 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013601741000000001 2024-02-09T17:26:41.847 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:26:41.847 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:26:41.847 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:26:41.847 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.406024+0000", 2024-02-09T17:26:41.847 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7119999999999999e-05 2024-02-09T17:26:41.847 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:26:41.847 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:26:41.847 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:26:41.847 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:26:41.847 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:26:41.848 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:26:41.848 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:41.849 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:26:41.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:26:41.950 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:26:41.987 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:42.008 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:26:42.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:42.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:26:42.113 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:26:42.118 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:26:42.158 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:42.261 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:26:42.266 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:42.267 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:26:42.368 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:26:42.406 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:42 smithi099 ceph-mon[39420]: osdmap e147: 12 total, 11 up, 12 in 2024-02-09T17:26:42.421 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:42.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:42 smithi179 ceph-mon[43975]: osdmap e147: 12 total, 11 up, 12 in 2024-02-09T17:26:42.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:42 smithi162 ceph-mon[44304]: osdmap e147: 12 total, 11 up, 12 in 2024-02-09T17:26:42.445 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:26:42.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:42.449 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:26:42.533 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-09T17:26:42.550 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:26:42.609 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:42.642 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:26:42.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:42.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:26:42.753 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:26:42.828 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:42.877 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:26:42.994 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:26:42.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:42.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:26:43.100 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:26:43.132 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:43.169 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:26:43.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:43.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:26:43.212 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:26:43.273 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:26:43.283 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:43 smithi099 ceph-mon[39420]: pgmap v446: 17 pgs: 1 peering, 2 stale+active+clean, 14 active+clean; 7.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 854 op/s 2024-02-09T17:26:43.307 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:43 smithi179 ceph-mon[43975]: pgmap v446: 17 pgs: 1 peering, 2 stale+active+clean, 14 active+clean; 7.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 854 op/s 2024-02-09T17:26:43.310 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:43.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:43 smithi162 ceph-mon[44304]: pgmap v446: 17 pgs: 1 peering, 2 stale+active+clean, 14 active+clean; 7.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 854 op/s 2024-02-09T17:26:43.502 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a69c870a-c76e-11ee-95b7-87774f69a715@osd.8.service' 2024-02-09T17:26:43.508 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:26:43.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:43.513 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:26:43.575 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T17:26:43.575 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-02-09T17:26:43.575 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.8 2024-02-09T17:26:43.576 INFO:tasks.thrashosds.thrasher:remote for osd 8 is ubuntu@smithi179.front.sepia.ceph.com 2024-02-09T17:26:43.576 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell -n osd.8 --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-8 --journal-path /var/lib/ceph/osd/ceph-8/journal --op list-pgs 2024-02-09T17:26:43.614 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:26:43.687 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:43.824 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:26:43.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:43.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:26:43.874 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/osd.8/config 2024-02-09T17:26:43.939 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:26:44.012 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:44.567 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:44 smithi162 ceph-mon[44304]: Health check failed: Degraded data redundancy: 13884/339213 objects degraded (4.093%), 2 pgs degraded (PG_DEGRADED) 2024-02-09T17:26:44.748 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:44 smithi099 ceph-mon[39420]: Health check failed: Degraded data redundancy: 13884/339213 objects degraded (4.093%), 2 pgs degraded (PG_DEGRADED) 2024-02-09T17:26:44.869 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:26:44.869 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-09T17:26:44.869 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-02-09T17:26:44.869 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:26:44.892 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:44 smithi179 ceph-mon[43975]: Health check failed: Degraded data redundancy: 13884/339213 objects degraded (4.093%), 2 pgs degraded (PG_DEGRADED) 2024-02-09T17:26:45.364 INFO:teuthology.orchestra.run.smithi179.stderr:Creating an OSD daemon form without an OSD FSID value 2024-02-09T17:26:45.548 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:45 smithi162 ceph-mon[44304]: pgmap v447: 17 pgs: 2 active+undersized+degraded, 1 peering, 14 active+clean; 6.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 932 op/s; 13884/339213 objects degraded (4.093%) 2024-02-09T17:26:45.577 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-09T17:26:45.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:45 smithi179 ceph-mon[43975]: pgmap v447: 17 pgs: 2 active+undersized+degraded, 1 peering, 14 active+clean; 6.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 932 op/s; 13884/339213 objects degraded (4.093%) 2024-02-09T17:26:45.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:45 smithi099 ceph-mon[39420]: pgmap v447: 17 pgs: 2 active+undersized+degraded, 1 peering, 14 active+clean; 6.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 932 op/s; 13884/339213 objects degraded (4.093%) 2024-02-09T17:26:45.887 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:26:46.441 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:26:46.530 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is set 2024-02-09T17:26:46.704 INFO:teuthology.orchestra.run.smithi179.stdout:2.f 2024-02-09T17:26:46.704 INFO:teuthology.orchestra.run.smithi179.stdout:2.b 2024-02-09T17:26:46.704 INFO:teuthology.orchestra.run.smithi179.stdout:2.5 2024-02-09T17:26:46.704 INFO:teuthology.orchestra.run.smithi179.stdout:2.6 2024-02-09T17:26:46.790 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:46 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2268389599' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:26:46.790 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:46 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:26:46.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:46 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2268389599' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:26:46.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:46 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:26:46.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:46 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2268389599' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:26:46.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:46 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:26:47.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:47 smithi162 ceph-mon[44304]: pgmap v448: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 6.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 988 op/s; 27739/333141 objects degraded (8.327%); 0 B/s, 0 objects/s recovering 2024-02-09T17:26:47.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:47 smithi162 ceph-mon[44304]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:26:47.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:47 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:26:47.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:47 smithi162 ceph-mon[44304]: osdmap e148: 12 total, 11 up, 12 in 2024-02-09T17:26:47.911 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:47 smithi099 ceph-mon[39420]: pgmap v448: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 6.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 988 op/s; 27739/333141 objects degraded (8.327%); 0 B/s, 0 objects/s recovering 2024-02-09T17:26:47.911 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:47 smithi099 ceph-mon[39420]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:26:47.911 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:47 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:26:47.911 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:47 smithi099 ceph-mon[39420]: osdmap e148: 12 total, 11 up, 12 in 2024-02-09T17:26:47.921 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:26:47.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:47 smithi179 ceph-mon[43975]: pgmap v448: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 6.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 988 op/s; 27739/333141 objects degraded (8.327%); 0 B/s, 0 objects/s recovering 2024-02-09T17:26:47.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:47 smithi179 ceph-mon[43975]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:26:47.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:47 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:26:47.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:47 smithi179 ceph-mon[43975]: osdmap e148: 12 total, 11 up, 12 in 2024-02-09T17:26:47.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:47.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:26:47.997 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:26:47.998 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-09T17:26:47.998 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-02-09T17:26:47.998 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-02-09T17:26:47.998 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:26:48.026 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:26:48.067 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:48.086 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell -n osd.8 --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-8 --journal-path /var/lib/ceph/osd/ceph-8/journal --op export --pgid 2.f --file /var/log/ceph/exp.2.f.8 2024-02-09T17:26:48.099 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:26:48.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:48.102 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:26:48.203 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:26:48.238 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:48.279 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:26:48.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:48.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:26:48.355 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/osd.8/config 2024-02-09T17:26:48.383 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:26:48.437 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:48.721 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:26:48.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:48.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:26:48.802 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-09T17:26:48.827 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:26:48.864 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:48.907 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:26:48.916 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:48.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:26:49.017 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:26:49.060 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:49.109 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:26:49.135 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:26:49.143 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:26:49.187 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:26:49.212 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:26:49.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:49.216 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:26:49.243 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:26:49.283 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:26:49.317 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:26:49.317 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:26:49.351 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:26:49.390 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:26:49.409 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:49.425 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:26:49.468 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:26:49.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:49.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:26:49.479 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:26:49.529 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:26:49.530 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:26:49.530 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:26:49.530 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:26:49.530 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:26:49.530 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:26:49.530 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:26:49.530 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:26:49.530 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:26:49.530 INFO:teuthology.orchestra.run.smithi099.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-02-09T17:26:49.530 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:26:49.530 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:26:49.530 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:26:49.573 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:26:49.603 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:49.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:49 smithi162 ceph-mon[44304]: pgmap v450: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 6.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 612 op/s; 27739/331545 objects degraded (8.367%); 0 B/s, 0 objects/s recovering 2024-02-09T17:26:49.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:49 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3895719829' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:26:49.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:49 smithi099 ceph-mon[39420]: pgmap v450: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 6.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 612 op/s; 27739/331545 objects degraded (8.367%); 0 B/s, 0 objects/s recovering 2024-02-09T17:26:49.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:49 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3895719829' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:26:49.854 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:26:49.857 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:49 smithi179 ceph-mon[43975]: pgmap v450: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 6.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 612 op/s; 27739/331545 objects degraded (8.367%); 0 B/s, 0 objects/s recovering 2024-02-09T17:26:49.857 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:49 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3895719829' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:26:49.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:49.858 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:26:49.922 INFO:teuthology.orchestra.run.smithi179.stderr:Creating an OSD daemon form without an OSD FSID value 2024-02-09T17:26:49.959 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:26:50.002 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:50.275 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-09T17:26:50.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:50.276 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:26:50.277 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-09T17:26:50.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:26:50.379 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:26:50.419 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:50.445 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:26:50.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:50.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:26:50.510 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset noscrub 2024-02-09T17:26:50.552 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:26:50.587 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:50.710 INFO:teuthology.orchestra.run.smithi179.stderr:Exporting 2.f info 2.f( v 145'9589 (0'0,145'9589] local-lis/les=111/112 n=7225 ec=111/111 lis/c=111/111 les/c/f=112/112/0 sis=111) 2024-02-09T17:26:50.747 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:26:50.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:50.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:26:50.770 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0009b1d:::benchmark_data_smithi179_77138_object76587:head# 2024-02-09T17:26:50.770 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f00145b2:::benchmark_data_smithi179_77138_object92940:head# 2024-02-09T17:26:50.770 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0014e91:::benchmark_data_smithi179_77138_object25924:head# 2024-02-09T17:26:50.770 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f001b70e:::benchmark_data_smithi179_77138_object86702:head# 2024-02-09T17:26:50.770 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f00277de:::benchmark_data_smithi179_77138_object101862:head# 2024-02-09T17:26:50.770 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f002fc04:::benchmark_data_smithi179_77138_object103750:head# 2024-02-09T17:26:50.771 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0049663:::benchmark_data_smithi179_77138_object78361:head# 2024-02-09T17:26:50.771 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f005d6de:::benchmark_data_smithi179_77138_object83949:head# 2024-02-09T17:26:50.774 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f005ee7d:::benchmark_data_smithi179_77138_object93644:head# 2024-02-09T17:26:50.774 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f00665ea:::benchmark_data_smithi179_77138_object116732:head# 2024-02-09T17:26:50.775 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f007069d:::benchmark_data_smithi179_77138_object44669:head# 2024-02-09T17:26:50.775 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0071423:::benchmark_data_smithi179_77138_object78642:head# 2024-02-09T17:26:50.775 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f007ca94:::benchmark_data_smithi179_77138_object52821:head# 2024-02-09T17:26:50.776 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f007d73e:::benchmark_data_smithi179_77138_object39310:head# 2024-02-09T17:26:50.776 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0082e16:::benchmark_data_smithi179_77138_object62498:head# 2024-02-09T17:26:50.782 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0087052:::benchmark_data_smithi179_77138_object86817:head# 2024-02-09T17:26:50.782 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f009f166:::benchmark_data_smithi179_77138_object103815:head# 2024-02-09T17:26:50.783 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f00aa2a9:::benchmark_data_smithi179_77138_object121051:head# 2024-02-09T17:26:50.783 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f00cc996:::benchmark_data_smithi179_77138_object61202:head# 2024-02-09T17:26:50.784 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f00ccdc1:::benchmark_data_smithi179_77138_object94720:head# 2024-02-09T17:26:50.784 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f00d1199:::benchmark_data_smithi179_77138_object131602:head# 2024-02-09T17:26:50.785 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f00d3f76:::benchmark_data_smithi179_77138_object61291:head# 2024-02-09T17:26:50.789 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f00dcac1:::benchmark_data_smithi179_77138_object106431:head# 2024-02-09T17:26:50.789 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f00dd4cd:::benchmark_data_smithi179_77138_object78366:head# 2024-02-09T17:26:50.789 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f00e639e:::benchmark_data_smithi179_77138_object109593:head# 2024-02-09T17:26:50.789 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f00eeb75:::benchmark_data_smithi179_77138_object42343:head# 2024-02-09T17:26:50.790 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f00f5c07:::benchmark_data_smithi179_77138_object101059:head# 2024-02-09T17:26:50.790 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f00fdd95:::benchmark_data_smithi179_77138_object62588:head# 2024-02-09T17:26:50.790 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0100ff1:::benchmark_data_smithi179_77138_object83033:head# 2024-02-09T17:26:50.793 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0103c44:::benchmark_data_smithi179_77138_object62435:head# 2024-02-09T17:26:50.793 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f010525a:::benchmark_data_smithi179_77138_object50524:head# 2024-02-09T17:26:50.793 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f01071b2:::benchmark_data_smithi179_77138_object64831:head# 2024-02-09T17:26:50.794 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f011e672:::benchmark_data_smithi179_77138_object50037:head# 2024-02-09T17:26:50.794 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f012680f:::benchmark_data_smithi179_77138_object108555:head# 2024-02-09T17:26:50.794 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f01271e2:::benchmark_data_smithi179_77138_object74125:head# 2024-02-09T17:26:50.794 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f012dcab:::benchmark_data_smithi179_77138_object38915:head# 2024-02-09T17:26:50.794 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f014621f:::benchmark_data_smithi179_77138_object81355:head# 2024-02-09T17:26:50.794 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0154249:::benchmark_data_smithi179_77138_object19798:head# 2024-02-09T17:26:50.795 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f015cf31:::benchmark_data_smithi179_77138_object106013:head# 2024-02-09T17:26:50.795 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0164466:::benchmark_data_smithi179_77138_object122578:head# 2024-02-09T17:26:50.796 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0165fcd:::benchmark_data_smithi179_77138_object26528:head# 2024-02-09T17:26:50.796 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0170933:::benchmark_data_smithi179_77138_object63557:head# 2024-02-09T17:26:50.799 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f01786c7:::benchmark_data_smithi179_77138_object68930:head# 2024-02-09T17:26:50.800 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f019c5ec:::benchmark_data_smithi179_77138_object27826:head# 2024-02-09T17:26:50.800 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f019d14a:::benchmark_data_smithi179_77138_object71945:head# 2024-02-09T17:26:50.801 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f019ec6c:::benchmark_data_smithi179_77138_object113765:head# 2024-02-09T17:26:50.801 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f01aa986:::benchmark_data_smithi179_77138_object70740:head# 2024-02-09T17:26:50.802 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f01c8ca0:::benchmark_data_smithi179_77138_object63796:head# 2024-02-09T17:26:50.802 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f01ca4af:::benchmark_data_smithi179_77138_object80981:head# 2024-02-09T17:26:50.803 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f01d2e45:::benchmark_data_smithi179_77138_object105114:head# 2024-02-09T17:26:50.811 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f01d578f:::benchmark_data_smithi179_77138_object121030:head# 2024-02-09T17:26:50.811 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f01e59c9:::benchmark_data_smithi179_77138_object70880:head# 2024-02-09T17:26:50.811 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f01f6a98:::benchmark_data_smithi179_77138_object108599:head# 2024-02-09T17:26:50.811 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0200abe:::benchmark_data_smithi179_77138_object93795:head# 2024-02-09T17:26:50.811 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0209489:::benchmark_data_smithi179_77138_object71861:head# 2024-02-09T17:26:50.811 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f020b167:::benchmark_data_smithi179_77138_object70908:head# 2024-02-09T17:26:50.811 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f020bd30:::benchmark_data_smithi179_77138_object56493:head# 2024-02-09T17:26:50.811 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f021fb1a:::benchmark_data_smithi179_77138_object115266:head# 2024-02-09T17:26:50.811 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0234d87:::benchmark_data_smithi179_77138_object44098:head# 2024-02-09T17:26:50.812 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0239b60:::benchmark_data_smithi179_77138_object50518:head# 2024-02-09T17:26:50.812 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f023de06:::benchmark_data_smithi179_77138_object83562:head# 2024-02-09T17:26:50.812 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f02475ed:::benchmark_data_smithi179_77138_object19942:head# 2024-02-09T17:26:50.812 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f024ddbc:::benchmark_data_smithi179_77138_object86947:head# 2024-02-09T17:26:50.812 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f024fc6d:::benchmark_data_smithi179_77138_object63134:head# 2024-02-09T17:26:50.812 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f025b2e9:::benchmark_data_smithi179_77138_object91726:head# 2024-02-09T17:26:50.813 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f025f8c0:::benchmark_data_smithi179_77138_object128027:head# 2024-02-09T17:26:50.813 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f026b703:::benchmark_data_smithi179_77138_object59730:head# 2024-02-09T17:26:50.813 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f028626e:::benchmark_data_smithi179_77138_object107889:head# 2024-02-09T17:26:50.813 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f02a8e8d:::benchmark_data_smithi179_77138_object108825:head# 2024-02-09T17:26:50.813 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f02acf3b:::benchmark_data_smithi179_77138_object58076:head# 2024-02-09T17:26:50.814 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f02b330f:::benchmark_data_smithi179_77138_object128853:head# 2024-02-09T17:26:50.814 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f02cb235:::benchmark_data_smithi179_77138_object23970:head# 2024-02-09T17:26:50.814 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f02cdca4:::benchmark_data_smithi179_77138_object82479:head# 2024-02-09T17:26:50.815 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f02f0913:::benchmark_data_smithi179_77138_object88860:head# 2024-02-09T17:26:50.815 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f02f7251:::benchmark_data_smithi179_77138_object61612:head# 2024-02-09T17:26:50.815 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f02f91ec:::benchmark_data_smithi179_77138_object63392:head# 2024-02-09T17:26:50.816 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0300b1f:::benchmark_data_smithi179_77138_object26272:head# 2024-02-09T17:26:50.816 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0313443:::benchmark_data_smithi179_77138_object119842:head# 2024-02-09T17:26:50.817 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f03135ca:::benchmark_data_smithi179_77138_object102514:head# 2024-02-09T17:26:50.817 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f031b2e3:::benchmark_data_smithi179_77138_object46384:head# 2024-02-09T17:26:50.817 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0325e8c:::benchmark_data_smithi179_77138_object76009:head# 2024-02-09T17:26:50.818 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f032c8a8:::benchmark_data_smithi179_77138_object46933:head# 2024-02-09T17:26:50.819 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0334609:::benchmark_data_smithi179_77138_object67238:head# 2024-02-09T17:26:50.819 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f033b26a:::benchmark_data_smithi179_77138_object41806:head# 2024-02-09T17:26:50.819 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f033bcff:::benchmark_data_smithi179_77138_object81143:head# 2024-02-09T17:26:50.819 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f033e5dc:::benchmark_data_smithi179_77138_object36920:head# 2024-02-09T17:26:50.820 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f034a3ba:::benchmark_data_smithi179_77138_object57026:head# 2024-02-09T17:26:50.820 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f034c516:::benchmark_data_smithi179_77138_object113382:head# 2024-02-09T17:26:50.821 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f035dca4:::benchmark_data_smithi179_77138_object71087:head# 2024-02-09T17:26:50.821 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0362761:::benchmark_data_smithi179_77138_object32135:head# 2024-02-09T17:26:50.822 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0363375:::benchmark_data_smithi179_77138_object52864:head# 2024-02-09T17:26:50.822 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f036f0de:::benchmark_data_smithi179_77138_object43229:head# 2024-02-09T17:26:50.823 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0375fd4:::benchmark_data_smithi179_77138_object99643:head# 2024-02-09T17:26:50.823 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f037a121:::benchmark_data_smithi179_77138_object131840:head# 2024-02-09T17:26:50.824 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0384e9e:::benchmark_data_smithi179_77138_object83483:head# 2024-02-09T17:26:50.824 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f038e875:::benchmark_data_smithi179_77138_object97332:head# 2024-02-09T17:26:50.824 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f039ae41:::benchmark_data_smithi179_77138_object38478:head# 2024-02-09T17:26:50.825 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f03a559f:::benchmark_data_smithi179_77138_object131911:head# 2024-02-09T17:26:50.825 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f03b4be1:::benchmark_data_smithi179_77138_object84639:head# 2024-02-09T17:26:50.826 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f03c5c31:::benchmark_data_smithi179_77138_object123876:head# 2024-02-09T17:26:50.826 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f03d4c75:::benchmark_data_smithi179_77138_object121138:head# 2024-02-09T17:26:50.827 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f03d7c74:::benchmark_data_smithi179_77138_object61693:head# 2024-02-09T17:26:50.827 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f03da106:::benchmark_data_smithi179_77138_object110589:head# 2024-02-09T17:26:50.827 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f03e0107:::benchmark_data_smithi179_77138_object51014:head# 2024-02-09T17:26:50.828 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f03e4330:::benchmark_data_smithi179_77138_object38198:head# 2024-02-09T17:26:50.828 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f03ec1cb:::benchmark_data_smithi179_77138_object74956:head# 2024-02-09T17:26:50.829 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f03ee2e3:::benchmark_data_smithi179_77138_object95933:head# 2024-02-09T17:26:50.829 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f040592a:::benchmark_data_smithi179_77138_object50936:head# 2024-02-09T17:26:50.830 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0416e7a:::benchmark_data_smithi179_77138_object26840:head# 2024-02-09T17:26:50.830 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0418f63:::benchmark_data_smithi179_77138_object74194:head# 2024-02-09T17:26:50.831 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0428949:::benchmark_data_smithi179_77138_object44821:head# 2024-02-09T17:26:50.831 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0435b36:::benchmark_data_smithi179_77138_object107139:head# 2024-02-09T17:26:50.832 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0439ef1:::benchmark_data_smithi179_77138_object76874:head# 2024-02-09T17:26:50.832 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f043eed2:::benchmark_data_smithi179_77138_object81403:head# 2024-02-09T17:26:50.833 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f04473e5:::benchmark_data_smithi179_77138_object115473:head# 2024-02-09T17:26:50.833 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0465b7d:::benchmark_data_smithi179_77138_object118594:head# 2024-02-09T17:26:50.833 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0477f76:::benchmark_data_smithi179_77138_object123816:head# 2024-02-09T17:26:50.834 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f047cf49:::benchmark_data_smithi179_77138_object98654:head# 2024-02-09T17:26:50.834 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f04845e5:::benchmark_data_smithi179_77138_object30575:head# 2024-02-09T17:26:50.835 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f048b013:::benchmark_data_smithi179_77138_object29666:head# 2024-02-09T17:26:50.835 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f048fede:::benchmark_data_smithi179_77138_object20727:head# 2024-02-09T17:26:50.836 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f049bfc6:::benchmark_data_smithi179_77138_object120147:head# 2024-02-09T17:26:50.837 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f049cf2b:::benchmark_data_smithi179_77138_object80727:head# 2024-02-09T17:26:50.837 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f049ef92:::benchmark_data_smithi179_77138_object78866:head# 2024-02-09T17:26:50.838 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f04b07df:::benchmark_data_smithi179_77138_object20620:head# 2024-02-09T17:26:50.838 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f04bd63c:::benchmark_data_smithi179_77138_object32172:head# 2024-02-09T17:26:50.838 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f04d3f89:::benchmark_data_smithi179_77138_object26552:head# 2024-02-09T17:26:50.839 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f04de1f1:::benchmark_data_smithi179_77138_object131750:head# 2024-02-09T17:26:50.839 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f04df140:::benchmark_data_smithi179_77138_object67950:head# 2024-02-09T17:26:50.839 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f04ee360:::benchmark_data_smithi179_77138_object57352:head# 2024-02-09T17:26:50.840 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f04f467d:::benchmark_data_smithi179_77138_object26578:head# 2024-02-09T17:26:50.840 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0516145:::benchmark_data_smithi179_77138_object83753:head# 2024-02-09T17:26:50.840 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f052a853:::benchmark_data_smithi179_77138_object65318:head# 2024-02-09T17:26:50.841 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f052d2e3:::benchmark_data_smithi179_77138_object35833:head# 2024-02-09T17:26:50.841 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0551d31:::benchmark_data_smithi179_77138_object100194:head# 2024-02-09T17:26:50.842 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0563dec:::benchmark_data_smithi179_77138_object89077:head# 2024-02-09T17:26:50.842 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f056521f:::benchmark_data_smithi179_77138_object53693:head# 2024-02-09T17:26:50.843 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0567086:::benchmark_data_smithi179_77138_object116262:head# 2024-02-09T17:26:50.847 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0573f51:::benchmark_data_smithi179_77138_object27748:head# 2024-02-09T17:26:50.847 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f057920a:::benchmark_data_smithi179_77138_object30307:head# 2024-02-09T17:26:50.847 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f05810a4:::benchmark_data_smithi179_77138_object127587:head# 2024-02-09T17:26:50.848 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0585547:::benchmark_data_smithi179_77138_object35736:head# 2024-02-09T17:26:50.848 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f05910da:::benchmark_data_smithi179_77138_object64915:head# 2024-02-09T17:26:50.849 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0594072:::benchmark_data_smithi179_77138_object116293:head# 2024-02-09T17:26:50.849 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0598223:::benchmark_data_smithi179_77138_object101491:head# 2024-02-09T17:26:50.849 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f05a7377:::benchmark_data_smithi179_77138_object75626:head# 2024-02-09T17:26:50.850 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f05aacf9:::benchmark_data_smithi179_77138_object34727:head# 2024-02-09T17:26:50.850 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f05ad007:::benchmark_data_smithi179_77138_object60356:head# 2024-02-09T17:26:50.851 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f05b0af0:::benchmark_data_smithi179_77138_object101434:head# 2024-02-09T17:26:50.851 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f05b0db8:::benchmark_data_smithi179_77138_object130256:head# 2024-02-09T17:26:50.852 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f05b4d95:::benchmark_data_smithi179_77138_object89574:head# 2024-02-09T17:26:50.852 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f05c1cc5:::benchmark_data_smithi179_77138_object38207:head# 2024-02-09T17:26:50.853 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f05cc820:::benchmark_data_smithi179_77138_object103662:head# 2024-02-09T17:26:50.853 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:26:50.854 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f05cd621:::benchmark_data_smithi179_77138_object65959:head# 2024-02-09T17:26:50.854 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f05d3e2d:::benchmark_data_smithi179_77138_object34042:head# 2024-02-09T17:26:50.854 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f05e1f1e:::benchmark_data_smithi179_77138_object87156:head# 2024-02-09T17:26:50.855 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f060294c:::benchmark_data_smithi179_77138_object86460:head# 2024-02-09T17:26:50.855 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0604612:::benchmark_data_smithi179_77138_object76223:head# 2024-02-09T17:26:50.855 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f06057ad:::benchmark_data_smithi179_77138_object124688:head# 2024-02-09T17:26:50.857 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0615ebc:::benchmark_data_smithi179_77138_object108892:head# 2024-02-09T17:26:50.857 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f06360d5:::benchmark_data_smithi179_77138_object60783:head# 2024-02-09T17:26:50.858 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f06480cd:::benchmark_data_smithi179_77138_object35648:head# 2024-02-09T17:26:50.858 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f064c10e:::benchmark_data_smithi179_77138_object22594:head# 2024-02-09T17:26:50.858 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0650250:::benchmark_data_smithi179_77138_object120584:head# 2024-02-09T17:26:50.858 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f065c158:::benchmark_data_smithi179_77138_object90564:head# 2024-02-09T17:26:50.859 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f066152c:::benchmark_data_smithi179_77138_object89900:head# 2024-02-09T17:26:50.861 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0663842:::benchmark_data_smithi179_77138_object75827:head# 2024-02-09T17:26:50.862 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f066deef:::benchmark_data_smithi179_77138_object115985:head# 2024-02-09T17:26:50.862 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f06716f5:::benchmark_data_smithi179_77138_object66697:head# 2024-02-09T17:26:50.862 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0678444:::benchmark_data_smithi179_77138_object79175:head# 2024-02-09T17:26:50.862 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f067af82:::benchmark_data_smithi179_77138_object126514:head# 2024-02-09T17:26:50.862 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0680e99:::benchmark_data_smithi179_77138_object43363:head# 2024-02-09T17:26:50.862 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f06872cb:::benchmark_data_smithi179_77138_object61325:head# 2024-02-09T17:26:50.862 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f068c5d9:::benchmark_data_smithi179_77138_object40311:head# 2024-02-09T17:26:50.862 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f068cadd:::benchmark_data_smithi179_77138_object85122:head# 2024-02-09T17:26:50.863 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f06a416c:::benchmark_data_smithi179_77138_object106684:head# 2024-02-09T17:26:50.863 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f06bb2e9:::benchmark_data_smithi179_77138_object55991:head# 2024-02-09T17:26:50.863 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f06d4f4c:::benchmark_data_smithi179_77138_object47474:head# 2024-02-09T17:26:50.864 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f06dbe1b:::benchmark_data_smithi179_77138_object38518:head# 2024-02-09T17:26:50.864 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f06dc076:::benchmark_data_smithi179_77138_object40603:head# 2024-02-09T17:26:50.865 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f06e667b:::benchmark_data_smithi179_77138_object89306:head# 2024-02-09T17:26:50.866 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f06fa039:::benchmark_data_smithi179_77138_object75676:head# 2024-02-09T17:26:50.866 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f06fd209:::benchmark_data_smithi179_77138_object76437:head# 2024-02-09T17:26:50.867 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f06fdb68:::benchmark_data_smithi179_77138_object120202:head# 2024-02-09T17:26:50.867 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0704a7c:::benchmark_data_smithi179_77138_object40641:head# 2024-02-09T17:26:50.867 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f07062eb:::benchmark_data_smithi179_77138_object88630:head# 2024-02-09T17:26:50.868 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f070e062:::benchmark_data_smithi179_77138_object29065:head# 2024-02-09T17:26:50.868 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0720703:::benchmark_data_smithi179_77138_object130842:head# 2024-02-09T17:26:50.869 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0722ab7:::benchmark_data_smithi179_77138_object33812:head# 2024-02-09T17:26:50.869 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0723db8:::benchmark_data_smithi179_77138_object78601:head# 2024-02-09T17:26:50.869 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f073ee3a:::benchmark_data_smithi179_77138_object125923:head# 2024-02-09T17:26:50.870 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0748175:::benchmark_data_smithi179_77138_object118901:head# 2024-02-09T17:26:50.870 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f075857d:::benchmark_data_smithi179_77138_object126127:head# 2024-02-09T17:26:50.871 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f075a4f5:::benchmark_data_smithi179_77138_object41105:head# 2024-02-09T17:26:50.871 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0768437:::benchmark_data_smithi179_77138_object66572:head# 2024-02-09T17:26:50.871 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f076e6d2:::benchmark_data_smithi179_77138_object49506:head# 2024-02-09T17:26:50.872 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f077ad54:::benchmark_data_smithi179_77138_object116092:head# 2024-02-09T17:26:50.872 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0785faf:::benchmark_data_smithi179_77138_object117969:head# 2024-02-09T17:26:50.873 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0788f0b:::benchmark_data_smithi179_77138_object29606:head# 2024-02-09T17:26:50.873 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f078dfcc:::benchmark_data_smithi179_77138_object77617:head# 2024-02-09T17:26:50.874 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f078e741:::benchmark_data_smithi179_77138_object57882:head# 2024-02-09T17:26:50.874 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f07904ea:::benchmark_data_smithi179_77138_object123909:head# 2024-02-09T17:26:50.875 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0797559:::benchmark_data_smithi179_77138_object111449:head# 2024-02-09T17:26:50.875 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f07985b9:::benchmark_data_smithi179_77138_object42329:head# 2024-02-09T17:26:50.876 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f079ce89:::benchmark_data_smithi179_77138_object108428:head# 2024-02-09T17:26:50.876 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f07a30c3:::benchmark_data_smithi179_77138_object78046:head# 2024-02-09T17:26:50.877 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f07ac5f7:::benchmark_data_smithi179_77138_object27022:head# 2024-02-09T17:26:50.877 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f07aea53:::benchmark_data_smithi179_77138_object98097:head# 2024-02-09T17:26:50.878 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f07b088a:::benchmark_data_smithi179_77138_object131346:head# 2024-02-09T17:26:50.878 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f07b4dd5:::benchmark_data_smithi179_77138_object31636:head# 2024-02-09T17:26:50.879 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f07c5067:::benchmark_data_smithi179_77138_object90879:head# 2024-02-09T17:26:50.879 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f07cd205:::benchmark_data_smithi179_77138_object41256:head# 2024-02-09T17:26:50.880 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f07de7ac:::benchmark_data_smithi179_77138_object22481:head# 2024-02-09T17:26:50.881 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f07e7712:::benchmark_data_smithi179_77138_object98951:head# 2024-02-09T17:26:50.881 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f07efe63:::benchmark_data_smithi179_77138_object118720:head# 2024-02-09T17:26:50.881 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f07f6816:::benchmark_data_smithi179_77138_object111026:head# 2024-02-09T17:26:50.882 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f080de51:::benchmark_data_smithi179_77138_object60737:head# 2024-02-09T17:26:50.882 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0812ab9:::benchmark_data_smithi179_77138_object82572:head# 2024-02-09T17:26:50.883 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0824acd:::benchmark_data_smithi179_77138_object98533:head# 2024-02-09T17:26:50.883 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:50 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 26903/322257 objects degraded (8.348%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:26:50.883 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:50 smithi162 ceph-mon[44304]: osdmap e149: 12 total, 11 up, 12 in 2024-02-09T17:26:50.884 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f082ab2e:::benchmark_data_smithi179_77138_object20153:head# 2024-02-09T17:26:50.884 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0831511:::benchmark_data_smithi179_77138_object72833:head# 2024-02-09T17:26:50.884 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f083bb9f:::benchmark_data_smithi179_77138_object123483:head# 2024-02-09T17:26:50.884 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f083fcbf:::benchmark_data_smithi179_77138_object92193:head# 2024-02-09T17:26:50.885 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f08438c6:::benchmark_data_smithi179_77138_object116164:head# 2024-02-09T17:26:50.885 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0846fd3:::benchmark_data_smithi179_77138_object93218:head# 2024-02-09T17:26:50.886 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0847fef:::benchmark_data_smithi179_77138_object114962:head# 2024-02-09T17:26:50.886 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:50.886 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f08937a7:::benchmark_data_smithi179_77138_object86180:head# 2024-02-09T17:26:50.887 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f089f1c3:::benchmark_data_smithi179_77138_object103271:head# 2024-02-09T17:26:50.887 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f08a1f69:::benchmark_data_smithi179_77138_object127089:head# 2024-02-09T17:26:50.887 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f08a4b30:::benchmark_data_smithi179_77138_object113377:head# 2024-02-09T17:26:50.888 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f08aff12:::benchmark_data_smithi179_77138_object54940:head# 2024-02-09T17:26:50.888 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f08b6257:::benchmark_data_smithi179_77138_object90306:head# 2024-02-09T17:26:50.889 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f08bbef2:::benchmark_data_smithi179_77138_object104701:head# 2024-02-09T17:26:50.889 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f08c8715:::benchmark_data_smithi179_77138_object128388:head# 2024-02-09T17:26:50.890 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f08ced31:::benchmark_data_smithi179_77138_object85858:head# 2024-02-09T17:26:50.890 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f08d19df:::benchmark_data_smithi179_77138_object103987:head# 2024-02-09T17:26:50.891 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f08d5fd7:::benchmark_data_smithi179_77138_object41047:head# 2024-02-09T17:26:50.891 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f08e2674:::benchmark_data_smithi179_77138_object49455:head# 2024-02-09T17:26:50.892 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f08e8342:::benchmark_data_smithi179_77138_object133348:head# 2024-02-09T17:26:50.892 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f08e865f:::benchmark_data_smithi179_77138_object83189:head# 2024-02-09T17:26:50.893 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f08eed31:::benchmark_data_smithi179_77138_object38641:head# 2024-02-09T17:26:50.893 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f08f153d:::benchmark_data_smithi179_77138_object33995:head# 2024-02-09T17:26:50.894 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f08f7683:::benchmark_data_smithi179_77138_object64276:head# 2024-02-09T17:26:50.894 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f08fc08e:::benchmark_data_smithi179_77138_object117723:head# 2024-02-09T17:26:50.894 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0900b83:::benchmark_data_smithi179_77138_object105761:head# 2024-02-09T17:26:50.895 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0915201:::benchmark_data_smithi179_77138_object90425:head# 2024-02-09T17:26:50.895 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0924677:::benchmark_data_smithi179_77138_object85974:head# 2024-02-09T17:26:50.896 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f092c974:::benchmark_data_smithi179_77138_object44003:head# 2024-02-09T17:26:50.896 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f094e9ae:::benchmark_data_smithi179_77138_object122180:head# 2024-02-09T17:26:50.897 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0954375:::benchmark_data_smithi179_77138_object77134:head# 2024-02-09T17:26:50.897 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f09550e1:::benchmark_data_smithi179_77138_object58571:head# 2024-02-09T17:26:50.898 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f095cfb1:::benchmark_data_smithi179_77138_object76965:head# 2024-02-09T17:26:50.898 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f095ec9c:::benchmark_data_smithi179_77138_object107161:head# 2024-02-09T17:26:50.899 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0965a02:::benchmark_data_smithi179_77138_object57875:head# 2024-02-09T17:26:50.899 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f09681b3:::benchmark_data_smithi179_77138_object45461:head# 2024-02-09T17:26:50.900 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0974966:::benchmark_data_smithi179_77138_object108865:head# 2024-02-09T17:26:50.900 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f09753c7:::benchmark_data_smithi179_77138_object121991:head# 2024-02-09T17:26:50.900 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0985e37:::benchmark_data_smithi179_77138_object24771:head# 2024-02-09T17:26:50.901 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f099e566:::benchmark_data_smithi179_77138_object95579:head# 2024-02-09T17:26:50.902 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f09ab928:::benchmark_data_smithi179_77138_object31897:head# 2024-02-09T17:26:50.902 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f09af59d:::benchmark_data_smithi179_77138_object109134:head# 2024-02-09T17:26:50.903 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f09b25d4:::benchmark_data_smithi179_77138_object36086:head# 2024-02-09T17:26:50.903 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f09b7716:::benchmark_data_smithi179_77138_object108674:head# 2024-02-09T17:26:50.903 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f09bdebe:::benchmark_data_smithi179_77138_object40591:head# 2024-02-09T17:26:50.904 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f09be939:::benchmark_data_smithi179_77138_object78956:head# 2024-02-09T17:26:50.904 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f09cba00:::benchmark_data_smithi179_77138_object76345:head# 2024-02-09T17:26:50.905 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f09d9701:::benchmark_data_smithi179_77138_object128304:head# 2024-02-09T17:26:50.905 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f09dee97:::benchmark_data_smithi179_77138_object28282:head# 2024-02-09T17:26:50.906 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f09dff39:::benchmark_data_smithi179_77138_object40045:head# 2024-02-09T17:26:50.906 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f09ef315:::benchmark_data_smithi179_77138_object55997:head# 2024-02-09T17:26:50.907 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f09f1347:::benchmark_data_smithi179_77138_object21876:head# 2024-02-09T17:26:50.907 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f09f3ac0:::benchmark_data_smithi179_77138_object19753:head# 2024-02-09T17:26:50.909 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f09f9622:::benchmark_data_smithi179_77138_object63871:head# 2024-02-09T17:26:50.909 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0a0f333:::benchmark_data_smithi179_77138_object55989:head# 2024-02-09T17:26:50.909 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0a170d8:::benchmark_data_smithi179_77138_object117567:head# 2024-02-09T17:26:50.910 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0a19c02:::benchmark_data_smithi179_77138_object58900:head# 2024-02-09T17:26:50.910 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0a1e813:::benchmark_data_smithi179_77138_object124873:head# 2024-02-09T17:26:50.910 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0a282be:::benchmark_data_smithi179_77138_object21221:head# 2024-02-09T17:26:50.911 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0a2a253:::benchmark_data_smithi179_77138_object125327:head# 2024-02-09T17:26:50.911 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0a2c674:::benchmark_data_smithi179_77138_object20011:head# 2024-02-09T17:26:50.914 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0a350a1:::benchmark_data_smithi179_77138_object73561:head# 2024-02-09T17:26:50.918 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0a42ac2:::benchmark_data_smithi179_77138_object25933:head# 2024-02-09T17:26:50.918 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0a5334d:::benchmark_data_smithi179_77138_object117197:head# 2024-02-09T17:26:50.918 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0a53642:::benchmark_data_smithi179_77138_object48844:head# 2024-02-09T17:26:50.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:50 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 26903/322257 objects degraded (8.348%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:26:50.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:50 smithi179 ceph-mon[43975]: osdmap e149: 12 total, 11 up, 12 in 2024-02-09T17:26:50.922 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0a53820:::benchmark_data_smithi179_77138_object37766:head# 2024-02-09T17:26:50.922 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0a66a30:::benchmark_data_smithi179_77138_object123538:head# 2024-02-09T17:26:50.923 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0a6f2eb:::benchmark_data_smithi179_77138_object86013:head# 2024-02-09T17:26:50.923 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0a7b7bc:::benchmark_data_smithi179_77138_object59250:head# 2024-02-09T17:26:50.924 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0a7fbd2:::benchmark_data_smithi179_77138_object91437:head# 2024-02-09T17:26:50.924 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0a81180:::benchmark_data_smithi179_77138_object25840:head# 2024-02-09T17:26:50.925 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0a8e769:::benchmark_data_smithi179_77138_object85336:head# 2024-02-09T17:26:50.925 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0aa7f32:::benchmark_data_smithi179_77138_object83238:head# 2024-02-09T17:26:50.926 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0abca32:::benchmark_data_smithi179_77138_object133004:head# 2024-02-09T17:26:50.926 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0ac4c80:::benchmark_data_smithi179_77138_object106395:head# 2024-02-09T17:26:50.926 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0acfae8:::benchmark_data_smithi179_77138_object122920:head# 2024-02-09T17:26:50.927 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0aeebd0:::benchmark_data_smithi179_77138_object95488:head# 2024-02-09T17:26:50.927 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0afaa0a:::benchmark_data_smithi179_77138_object49362:head# 2024-02-09T17:26:50.928 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0b2bcf3:::benchmark_data_smithi179_77138_object87032:head# 2024-02-09T17:26:50.929 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0b3a54c:::benchmark_data_smithi179_77138_object112172:head# 2024-02-09T17:26:50.930 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0b469cf:::benchmark_data_smithi179_77138_object40887:head# 2024-02-09T17:26:50.930 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0b62663:::benchmark_data_smithi179_77138_object36310:head# 2024-02-09T17:26:50.931 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0b6a12e:::benchmark_data_smithi179_77138_object109553:head# 2024-02-09T17:26:50.931 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0b6e65b:::benchmark_data_smithi179_77138_object118690:head# 2024-02-09T17:26:50.931 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0b72b0a:::benchmark_data_smithi179_77138_object128292:head# 2024-02-09T17:26:50.932 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0b7a02c:::benchmark_data_smithi179_77138_object23194:head# 2024-02-09T17:26:50.932 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0b84303:::benchmark_data_smithi179_77138_object91717:head# 2024-02-09T17:26:50.933 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0b9613c:::benchmark_data_smithi179_77138_object53511:head# 2024-02-09T17:26:50.933 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0ba24d7:::benchmark_data_smithi179_77138_object91361:head# 2024-02-09T17:26:50.934 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0ba3ced:::benchmark_data_smithi179_77138_object121721:head# 2024-02-09T17:26:50.934 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0baeee5:::benchmark_data_smithi179_77138_object103065:head# 2024-02-09T17:26:50.935 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0bb3c1f:::benchmark_data_smithi179_77138_object40898:head# 2024-02-09T17:26:50.935 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0bbe780:::benchmark_data_smithi179_77138_object70296:head# 2024-02-09T17:26:50.936 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0bbf8bb:::benchmark_data_smithi179_77138_object95071:head# 2024-02-09T17:26:50.936 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0bcdb42:::benchmark_data_smithi179_77138_object65432:head# 2024-02-09T17:26:50.937 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0bff7ca:::benchmark_data_smithi179_77138_object121170:head# 2024-02-09T17:26:50.937 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0c03bb8:::benchmark_data_smithi179_77138_object38969:head# 2024-02-09T17:26:50.938 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0c066e1:::benchmark_data_smithi179_77138_object74621:head# 2024-02-09T17:26:50.938 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0c0c6b6:::benchmark_data_smithi179_77138_object88098:head# 2024-02-09T17:26:50.939 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0c0da51:::benchmark_data_smithi179_77138_object54324:head# 2024-02-09T17:26:50.939 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:26:50.939 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0c195cd:::benchmark_data_smithi179_77138_object91459:head# 2024-02-09T17:26:50.940 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0c1d8af:::benchmark_data_smithi179_77138_object47203:head# 2024-02-09T17:26:50.940 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0c1e0d3:::benchmark_data_smithi179_77138_object126401:head# 2024-02-09T17:26:50.940 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0c225b3:::benchmark_data_smithi179_77138_object103660:head# 2024-02-09T17:26:50.941 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0c2ab71:::benchmark_data_smithi179_77138_object132523:head# 2024-02-09T17:26:50.941 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0c2cfd0:::benchmark_data_smithi179_77138_object58534:head# 2024-02-09T17:26:50.942 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0c2d73d:::benchmark_data_smithi179_77138_object118053:head# 2024-02-09T17:26:50.942 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0c31faf:::benchmark_data_smithi179_77138_object83666:head# 2024-02-09T17:26:50.942 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0c4f4ce:::benchmark_data_smithi179_77138_object36701:head# 2024-02-09T17:26:50.943 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0c64677:::benchmark_data_smithi179_77138_object125580:head# 2024-02-09T17:26:50.943 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0c71fd8:::benchmark_data_smithi179_77138_object90141:head# 2024-02-09T17:26:50.944 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0c7af21:::benchmark_data_smithi179_77138_object121097:head# 2024-02-09T17:26:50.945 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0c8b7ae:::benchmark_data_smithi179_77138_object30404:head# 2024-02-09T17:26:50.945 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0c9b9ed:::benchmark_data_smithi179_77138_object117220:head# 2024-02-09T17:26:50.945 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0ca3608:::benchmark_data_smithi179_77138_object129985:head# 2024-02-09T17:26:50.946 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0caed23:::benchmark_data_smithi179_77138_object62386:head# 2024-02-09T17:26:50.946 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0cb5d09:::benchmark_data_smithi179_77138_object102666:head# 2024-02-09T17:26:50.947 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0cb64fb:::benchmark_data_smithi179_77138_object54030:head# 2024-02-09T17:26:50.949 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0cbebbc:::benchmark_data_smithi179_77138_object24069:head# 2024-02-09T17:26:50.950 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0cc6096:::benchmark_data_smithi179_77138_object119408:head# 2024-02-09T17:26:50.950 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0cc66d3:::benchmark_data_smithi179_77138_object129583:head# 2024-02-09T17:26:50.952 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0cccfca:::benchmark_data_smithi179_77138_object32202:head# 2024-02-09T17:26:50.952 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0cd023c:::benchmark_data_smithi179_77138_object56116:head# 2024-02-09T17:26:50.953 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0cdd007:::benchmark_data_smithi179_77138_object76034:head# 2024-02-09T17:26:50.954 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0ce233c:::benchmark_data_smithi179_77138_object76693:head# 2024-02-09T17:26:50.954 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0d00258:::benchmark_data_smithi179_77138_object56061:head# 2024-02-09T17:26:50.954 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0d019f7:::benchmark_data_smithi179_77138_object45109:head# 2024-02-09T17:26:50.955 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0d0297e:::benchmark_data_smithi179_77138_object51354:head# 2024-02-09T17:26:50.956 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0d050f6:::benchmark_data_smithi179_77138_object73086:head# 2024-02-09T17:26:50.956 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0d0a39d:::benchmark_data_smithi179_77138_object34467:head# 2024-02-09T17:26:50.956 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0d0fe08:::benchmark_data_smithi179_77138_object49754:head# 2024-02-09T17:26:50.958 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0d11d78:::benchmark_data_smithi179_77138_object122939:head# 2024-02-09T17:26:50.958 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0d1930b:::benchmark_data_smithi179_77138_object83180:head# 2024-02-09T17:26:50.958 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0d1cdf3:::benchmark_data_smithi179_77138_object29512:head# 2024-02-09T17:26:50.958 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0d215a0:::benchmark_data_smithi179_77138_object69166:head# 2024-02-09T17:26:50.959 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0d23254:::benchmark_data_smithi179_77138_object57981:head# 2024-02-09T17:26:50.960 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0d2a470:::benchmark_data_smithi179_77138_object110245:head# 2024-02-09T17:26:50.960 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0d3f7fe:::benchmark_data_smithi179_77138_object26566:head# 2024-02-09T17:26:50.960 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0d437e4:::benchmark_data_smithi179_77138_object119026:head# 2024-02-09T17:26:50.960 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0d47e11:::benchmark_data_smithi179_77138_object61638:head# 2024-02-09T17:26:50.961 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0d482cc:::benchmark_data_smithi179_77138_object50999:head# 2024-02-09T17:26:50.961 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0d49a46:::benchmark_data_smithi179_77138_object35420:head# 2024-02-09T17:26:50.962 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0d4e1d3:::benchmark_data_smithi179_77138_object87836:head# 2024-02-09T17:26:50.962 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0d571d7:::benchmark_data_smithi179_77138_object129640:head# 2024-02-09T17:26:50.965 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0d5c8c2:::benchmark_data_smithi179_77138_object108600:head# 2024-02-09T17:26:50.965 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0d609b3:::benchmark_data_smithi179_77138_object41430:head# 2024-02-09T17:26:50.966 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0d67e17:::benchmark_data_smithi179_77138_object37580:head# 2024-02-09T17:26:50.966 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0d772cc:::benchmark_data_smithi179_77138_object71677:head# 2024-02-09T17:26:50.967 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0d8d312:::benchmark_data_smithi179_77138_object118913:head# 2024-02-09T17:26:50.967 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0d94627:::benchmark_data_smithi179_77138_object122948:head# 2024-02-09T17:26:50.967 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0da0ade:::benchmark_data_smithi179_77138_object71205:head# 2024-02-09T17:26:50.968 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0daca70:::benchmark_data_smithi179_77138_object110900:head# 2024-02-09T17:26:50.968 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0db3cfd:::benchmark_data_smithi179_77138_object79862:head# 2024-02-09T17:26:50.969 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0dba4d1:::benchmark_data_smithi179_77138_object61085:head# 2024-02-09T17:26:50.969 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0dbd0bd:::benchmark_data_smithi179_77138_object29460:head# 2024-02-09T17:26:50.969 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0dde5cd:::benchmark_data_smithi179_77138_object92006:head# 2024-02-09T17:26:50.970 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0de168a:::benchmark_data_smithi179_77138_object35326:head# 2024-02-09T17:26:50.970 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0dea766:::benchmark_data_smithi179_77138_object41201:head# 2024-02-09T17:26:50.975 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0decd56:::benchmark_data_smithi179_77138_object61987:head# 2024-02-09T17:26:50.975 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0defd32:::benchmark_data_smithi179_77138_object60832:head# 2024-02-09T17:26:50.975 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0df9086:::benchmark_data_smithi179_77138_object105794:head# 2024-02-09T17:26:50.975 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0e02fb2:::benchmark_data_smithi179_77138_object84832:head# 2024-02-09T17:26:50.976 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0e061cb:::benchmark_data_smithi179_77138_object88303:head# 2024-02-09T17:26:50.976 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0e204a4:::benchmark_data_smithi179_77138_object95895:head# 2024-02-09T17:26:50.977 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0e396fa:::benchmark_data_smithi179_77138_object42666:head# 2024-02-09T17:26:50.977 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0e43158:::benchmark_data_smithi179_77138_object130369:head# 2024-02-09T17:26:50.978 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0e47f12:::benchmark_data_smithi179_77138_object22917:head# 2024-02-09T17:26:50.978 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0e4c999:::benchmark_data_smithi179_77138_object109952:head# 2024-02-09T17:26:50.979 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0e5141e:::benchmark_data_smithi179_77138_object118088:head# 2024-02-09T17:26:50.979 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0e56353:::benchmark_data_smithi179_77138_object85368:head# 2024-02-09T17:26:50.980 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0e571ec:::benchmark_data_smithi179_77138_object106606:head# 2024-02-09T17:26:50.980 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0e5fa51:::benchmark_data_smithi179_77138_object97579:head# 2024-02-09T17:26:50.981 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0e60799:::benchmark_data_smithi179_77138_object99769:head# 2024-02-09T17:26:50.981 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0e621d9:::benchmark_data_smithi179_77138_object97431:head# 2024-02-09T17:26:50.981 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0e75e48:::benchmark_data_smithi179_77138_object43875:head# 2024-02-09T17:26:50.982 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0e7d45a:::benchmark_data_smithi179_77138_object64906:head# 2024-02-09T17:26:50.982 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0e847f2:::benchmark_data_smithi179_77138_object87545:head# 2024-02-09T17:26:50.983 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0e898da:::benchmark_data_smithi179_77138_object75988:head# 2024-02-09T17:26:50.983 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0e9cde2:::benchmark_data_smithi179_77138_object88496:head# 2024-02-09T17:26:50.984 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0ea18e5:::benchmark_data_smithi179_77138_object31099:head# 2024-02-09T17:26:50.984 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0ea1928:::benchmark_data_smithi179_77138_object42437:head# 2024-02-09T17:26:50.985 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0eb0b1a:::benchmark_data_smithi179_77138_object33948:head# 2024-02-09T17:26:50.985 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0ec1665:::benchmark_data_smithi179_77138_object109665:head# 2024-02-09T17:26:50.985 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0ec43e6:::benchmark_data_smithi179_77138_object129102:head# 2024-02-09T17:26:50.986 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0ec6f75:::benchmark_data_smithi179_77138_object61576:head# 2024-02-09T17:26:50.986 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0ec7314:::benchmark_data_smithi179_77138_object92079:head# 2024-02-09T17:26:50.987 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0ec88a0:::benchmark_data_smithi179_77138_object96660:head# 2024-02-09T17:26:50.990 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0ecf80b:::benchmark_data_smithi179_77138_object61504:head# 2024-02-09T17:26:50.990 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0ee906e:::benchmark_data_smithi179_77138_object63865:head# 2024-02-09T17:26:50.990 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0ef260c:::benchmark_data_smithi179_77138_object119254:head# 2024-02-09T17:26:50.991 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0ef389f:::benchmark_data_smithi179_77138_object126055:head# 2024-02-09T17:26:50.992 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0ef7e6c:::benchmark_data_smithi179_77138_object67920:head# 2024-02-09T17:26:50.992 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0efa560:::benchmark_data_smithi179_77138_object75102:head# 2024-02-09T17:26:50.992 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0f04d98:::benchmark_data_smithi179_77138_object30491:head# 2024-02-09T17:26:50.993 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0f05f2a:::benchmark_data_smithi179_77138_object93978:head# 2024-02-09T17:26:50.993 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0f06a48:::benchmark_data_smithi179_77138_object49990:head# 2024-02-09T17:26:50.994 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0f070dc:::benchmark_data_smithi179_77138_object130287:head# 2024-02-09T17:26:50.994 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0f0c45f:::benchmark_data_smithi179_77138_object78712:head# 2024-02-09T17:26:50.994 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0f17db8:::benchmark_data_smithi179_77138_object49932:head# 2024-02-09T17:26:50.995 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0f26857:::benchmark_data_smithi179_77138_object86363:head# 2024-02-09T17:26:50.995 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0f2a21e:::benchmark_data_smithi179_77138_object64087:head# 2024-02-09T17:26:50.999 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0f2fb8b:::benchmark_data_smithi179_77138_object55429:head# 2024-02-09T17:26:50.999 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0f3b67d:::benchmark_data_smithi179_77138_object133201:head# 2024-02-09T17:26:50.999 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:50 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 26903/322257 objects degraded (8.348%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:26:51.000 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:50 smithi099 ceph-mon[39420]: osdmap e149: 12 total, 11 up, 12 in 2024-02-09T17:26:51.000 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0f3c99b:::benchmark_data_smithi179_77138_object125038:head# 2024-02-09T17:26:51.000 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0f3de5b:::benchmark_data_smithi179_77138_object66804:head# 2024-02-09T17:26:51.000 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0f445b9:::benchmark_data_smithi179_77138_object21410:head# 2024-02-09T17:26:51.000 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0f46ac8:::benchmark_data_smithi179_77138_object21844:head# 2024-02-09T17:26:51.001 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0f473bb:::benchmark_data_smithi179_77138_object62177:head# 2024-02-09T17:26:51.001 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0f61687:::benchmark_data_smithi179_77138_object104909:head# 2024-02-09T17:26:51.003 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0f6eb14:::benchmark_data_smithi179_77138_object80261:head# 2024-02-09T17:26:51.004 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0f74d46:::benchmark_data_smithi179_77138_object52086:head# 2024-02-09T17:26:51.006 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0f74ead:::benchmark_data_smithi179_77138_object106839:head# 2024-02-09T17:26:51.006 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0f9b048:::benchmark_data_smithi179_77138_object34885:head# 2024-02-09T17:26:51.007 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0fa1236:::benchmark_data_smithi179_77138_object23512:head# 2024-02-09T17:26:51.007 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0fa83e4:::benchmark_data_smithi179_77138_object43027:head# 2024-02-09T17:26:51.012 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0fb9330:::benchmark_data_smithi179_77138_object27182:head# 2024-02-09T17:26:51.012 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0fb971a:::benchmark_data_smithi179_77138_object35953:head# 2024-02-09T17:26:51.012 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0fc9be7:::benchmark_data_smithi179_77138_object65343:head# 2024-02-09T17:26:51.013 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0fcbc3d:::benchmark_data_smithi179_77138_object44963:head# 2024-02-09T17:26:51.014 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:26:51.014 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0fd0a0b:::benchmark_data_smithi179_77138_object57711:head# 2024-02-09T17:26:51.014 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0fd5c93:::benchmark_data_smithi179_77138_object47541:head# 2024-02-09T17:26:51.014 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0fd6ec6:::benchmark_data_smithi179_77138_object68438:head# 2024-02-09T17:26:51.015 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0fd7ba8:::benchmark_data_smithi179_77138_object95736:head# 2024-02-09T17:26:51.015 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f0ff6684:::benchmark_data_smithi179_77138_object47206:head# 2024-02-09T17:26:51.016 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1014a42:::benchmark_data_smithi179_77138_object104527:head# 2024-02-09T17:26:51.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:51.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:26:51.017 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1021e7a:::benchmark_data_smithi179_77138_object47224:head# 2024-02-09T17:26:51.018 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f102724c:::benchmark_data_smithi179_77138_object122775:head# 2024-02-09T17:26:51.018 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f103b077:::benchmark_data_smithi179_77138_object38481:head# 2024-02-09T17:26:51.018 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f103bdb3:::benchmark_data_smithi179_77138_object42345:head# 2024-02-09T17:26:51.018 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f104199d:::benchmark_data_smithi179_77138_object125138:head# 2024-02-09T17:26:51.019 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1046d2d:::benchmark_data_smithi179_77138_object133046:head# 2024-02-09T17:26:51.021 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f10681d5:::benchmark_data_smithi179_77138_object133352:head# 2024-02-09T17:26:51.021 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f106c32c:::benchmark_data_smithi179_77138_object117686:head# 2024-02-09T17:26:51.022 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1073111:::benchmark_data_smithi179_77138_object110646:head# 2024-02-09T17:26:51.022 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f107987b:::benchmark_data_smithi179_77138_object129226:head# 2024-02-09T17:26:51.023 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f107ef36:::benchmark_data_smithi179_77138_object112335:head# 2024-02-09T17:26:51.023 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1087b0c:::benchmark_data_smithi179_77138_object60300:head# 2024-02-09T17:26:51.024 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1091074:::benchmark_data_smithi179_77138_object102118:head# 2024-02-09T17:26:51.028 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f10968da:::benchmark_data_smithi179_77138_object125879:head# 2024-02-09T17:26:51.033 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f109a5ee:::benchmark_data_smithi179_77138_object120328:head# 2024-02-09T17:26:51.033 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f109d49b:::benchmark_data_smithi179_77138_object123337:head# 2024-02-09T17:26:51.035 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f109f007:::benchmark_data_smithi179_77138_object40374:head# 2024-02-09T17:26:51.035 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f10a9f10:::benchmark_data_smithi179_77138_object34395:head# 2024-02-09T17:26:51.035 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f10acd79:::benchmark_data_smithi179_77138_object82795:head# 2024-02-09T17:26:51.035 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f10b5a37:::benchmark_data_smithi179_77138_object55158:head# 2024-02-09T17:26:51.035 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f10c4a54:::benchmark_data_smithi179_77138_object49048:head# 2024-02-09T17:26:51.035 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f10c8a45:::benchmark_data_smithi179_77138_object73097:head# 2024-02-09T17:26:51.035 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f10d06cd:::benchmark_data_smithi179_77138_object109942:head# 2024-02-09T17:26:51.035 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f10db02b:::benchmark_data_smithi179_77138_object49751:head# 2024-02-09T17:26:51.035 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f10dee0b:::benchmark_data_smithi179_77138_object61082:head# 2024-02-09T17:26:51.035 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f10ee6a7:::benchmark_data_smithi179_77138_object82528:head# 2024-02-09T17:26:51.035 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f10ee6cc:::benchmark_data_smithi179_77138_object58766:head# 2024-02-09T17:26:51.036 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f10fbe0f:::benchmark_data_smithi179_77138_object131542:head# 2024-02-09T17:26:51.036 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1103341:::benchmark_data_smithi179_77138_object102116:head# 2024-02-09T17:26:51.036 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f110e4c9:::benchmark_data_smithi179_77138_object124833:head# 2024-02-09T17:26:51.037 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1134138:::benchmark_data_smithi179_77138_object122620:head# 2024-02-09T17:26:51.037 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f113e835:::benchmark_data_smithi179_77138_object59854:head# 2024-02-09T17:26:51.042 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1142227:::benchmark_data_smithi179_77138_object66882:head# 2024-02-09T17:26:51.042 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f114530f:::benchmark_data_smithi179_77138_object58311:head# 2024-02-09T17:26:51.042 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1145d5b:::benchmark_data_smithi179_77138_object102846:head# 2024-02-09T17:26:51.042 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f11549ae:::benchmark_data_smithi179_77138_object24680:head# 2024-02-09T17:26:51.042 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f115cb2a:::benchmark_data_smithi179_77138_object37770:head# 2024-02-09T17:26:51.042 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1178729:::benchmark_data_smithi179_77138_object43485:head# 2024-02-09T17:26:51.042 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1182dbb:::benchmark_data_smithi179_77138_object60679:head# 2024-02-09T17:26:51.043 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f11864f3:::benchmark_data_smithi179_77138_object81346:head# 2024-02-09T17:26:51.043 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1188c18:::benchmark_data_smithi179_77138_object88603:head# 2024-02-09T17:26:51.048 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f11912df:::benchmark_data_smithi179_77138_object121720:head# 2024-02-09T17:26:51.048 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f119ecf9:::benchmark_data_smithi179_77138_object91733:head# 2024-02-09T17:26:51.048 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f11a4434:::benchmark_data_smithi179_77138_object125551:head# 2024-02-09T17:26:51.048 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f11a9640:::benchmark_data_smithi179_77138_object53832:head# 2024-02-09T17:26:51.048 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f11b69a6:::benchmark_data_smithi179_77138_object131360:head# 2024-02-09T17:26:51.048 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f11b9f6c:::benchmark_data_smithi179_77138_object57381:head# 2024-02-09T17:26:51.048 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f11be021:::benchmark_data_smithi179_77138_object46817:head# 2024-02-09T17:26:51.048 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f11c67d4:::benchmark_data_smithi179_77138_object42816:head# 2024-02-09T17:26:51.048 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f11dc448:::benchmark_data_smithi179_77138_object113453:head# 2024-02-09T17:26:51.048 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f11e6851:::benchmark_data_smithi179_77138_object93799:head# 2024-02-09T17:26:51.049 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:26:51.049 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:26:51.049 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f11ed7c7:::benchmark_data_smithi179_77138_object28342:head# 2024-02-09T17:26:51.050 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1201adc:::benchmark_data_smithi179_77138_object41367:head# 2024-02-09T17:26:51.050 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f121e082:::benchmark_data_smithi179_77138_object124068:head# 2024-02-09T17:26:51.050 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1226caa:::benchmark_data_smithi179_77138_object19731:head# 2024-02-09T17:26:51.050 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f122ad75:::benchmark_data_smithi179_77138_object97085:head# 2024-02-09T17:26:51.050 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1233664:::benchmark_data_smithi179_77138_object20774:head# 2024-02-09T17:26:51.050 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1239d15:::benchmark_data_smithi179_77138_object129729:head# 2024-02-09T17:26:51.051 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1245a6d:::benchmark_data_smithi179_77138_object22042:head# 2024-02-09T17:26:51.051 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1251347:::benchmark_data_smithi179_77138_object130340:head# 2024-02-09T17:26:51.052 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1256e46:::benchmark_data_smithi179_77138_object33505:head# 2024-02-09T17:26:51.052 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f12572d5:::benchmark_data_smithi179_77138_object81076:head# 2024-02-09T17:26:51.053 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1257d36:::benchmark_data_smithi179_77138_object97164:head# 2024-02-09T17:26:51.053 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1258d44:::benchmark_data_smithi179_77138_object116606:head# 2024-02-09T17:26:51.054 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f12795f3:::benchmark_data_smithi179_77138_object56568:head# 2024-02-09T17:26:51.054 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f128d73c:::benchmark_data_smithi179_77138_object39584:head# 2024-02-09T17:26:51.055 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f12956d9:::benchmark_data_smithi179_77138_object128998:head# 2024-02-09T17:26:51.060 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1295ff4:::benchmark_data_smithi179_77138_object60877:head# 2024-02-09T17:26:51.061 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f129d4b3:::benchmark_data_smithi179_77138_object99319:head# 2024-02-09T17:26:51.062 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f12a8f8c:::benchmark_data_smithi179_77138_object116500:head# 2024-02-09T17:26:51.063 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f12aafba:::benchmark_data_smithi179_77138_object88446:head# 2024-02-09T17:26:51.063 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f12b22eb:::benchmark_data_smithi179_77138_object106566:head# 2024-02-09T17:26:51.063 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f12bec5a:::benchmark_data_smithi179_77138_object133249:head# 2024-02-09T17:26:51.063 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f12d708f:::benchmark_data_smithi179_77138_object58838:head# 2024-02-09T17:26:51.063 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f12e1f67:::benchmark_data_smithi179_77138_object33558:head# 2024-02-09T17:26:51.064 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f12e4520:::benchmark_data_smithi179_77138_object104890:head# 2024-02-09T17:26:51.064 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f12ea387:::benchmark_data_smithi179_77138_object97282:head# 2024-02-09T17:26:51.065 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f12f915f:::benchmark_data_smithi179_77138_object103722:head# 2024-02-09T17:26:51.065 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f12fd731:::benchmark_data_smithi179_77138_object46878:head# 2024-02-09T17:26:51.067 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f131689b:::benchmark_data_smithi179_77138_object130479:head# 2024-02-09T17:26:51.068 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1323487:::benchmark_data_smithi179_77138_object89374:head# 2024-02-09T17:26:51.068 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f13256e1:::benchmark_data_smithi179_77138_object92923:head# 2024-02-09T17:26:51.069 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f132bbdd:::benchmark_data_smithi179_77138_object101562:head# 2024-02-09T17:26:51.069 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f132cba2:::benchmark_data_smithi179_77138_object30810:head# 2024-02-09T17:26:51.070 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f132f755:::benchmark_data_smithi179_77138_object76793:head# 2024-02-09T17:26:51.070 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f133a3a7:::benchmark_data_smithi179_77138_object78566:head# 2024-02-09T17:26:51.071 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f133fa11:::benchmark_data_smithi179_77138_object113150:head# 2024-02-09T17:26:51.071 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1348276:::benchmark_data_smithi179_77138_object61025:head# 2024-02-09T17:26:51.076 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1353c6f:::benchmark_data_smithi179_77138_object52000:head# 2024-02-09T17:26:51.077 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1369688:::benchmark_data_smithi179_77138_object91779:head# 2024-02-09T17:26:51.077 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f136f1fb:::benchmark_data_smithi179_77138_object95260:head# 2024-02-09T17:26:51.077 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f13705b3:::benchmark_data_smithi179_77138_object30173:head# 2024-02-09T17:26:51.077 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f137ca79:::benchmark_data_smithi179_77138_object24619:head# 2024-02-09T17:26:51.078 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1383e33:::benchmark_data_smithi179_77138_object75609:head# 2024-02-09T17:26:51.080 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1388549:::benchmark_data_smithi179_77138_object47450:head# 2024-02-09T17:26:51.081 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f138ed9b:::benchmark_data_smithi179_77138_object76154:head# 2024-02-09T17:26:51.081 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f139587c:::benchmark_data_smithi179_77138_object83859:head# 2024-02-09T17:26:51.082 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1397218:::benchmark_data_smithi179_77138_object63068:head# 2024-02-09T17:26:51.085 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1397419:::benchmark_data_smithi179_77138_object63249:head# 2024-02-09T17:26:51.086 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f139c3e3:::benchmark_data_smithi179_77138_object111351:head# 2024-02-09T17:26:51.086 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f139c528:::benchmark_data_smithi179_77138_object36394:head# 2024-02-09T17:26:51.086 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f13a6b3f:::benchmark_data_smithi179_77138_object36399:head# 2024-02-09T17:26:51.087 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f13a7cf4:::benchmark_data_smithi179_77138_object40974:head# 2024-02-09T17:26:51.087 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f13b43b0:::benchmark_data_smithi179_77138_object53374:head# 2024-02-09T17:26:51.088 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f13b5508:::benchmark_data_smithi179_77138_object110946:head# 2024-02-09T17:26:51.089 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f13b951e:::benchmark_data_smithi179_77138_object45514:head# 2024-02-09T17:26:51.089 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f13c22c0:::benchmark_data_smithi179_77138_object83858:head# 2024-02-09T17:26:51.089 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f13dc1b6:::benchmark_data_smithi179_77138_object56547:head# 2024-02-09T17:26:51.090 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f13de626:::benchmark_data_smithi179_77138_object107036:head# 2024-02-09T17:26:51.090 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f13dea5b:::benchmark_data_smithi179_77138_object31244:head# 2024-02-09T17:26:51.091 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f13f6535:::benchmark_data_smithi179_77138_object109635:head# 2024-02-09T17:26:51.091 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1402a44:::benchmark_data_smithi179_77138_object45926:head# 2024-02-09T17:26:51.092 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f140a963:::benchmark_data_smithi179_77138_object38925:head# 2024-02-09T17:26:51.092 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f140ea14:::benchmark_data_smithi179_77138_object93626:head# 2024-02-09T17:26:51.093 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f140f710:::benchmark_data_smithi179_77138_object84752:head# 2024-02-09T17:26:51.094 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1410eab:::benchmark_data_smithi179_77138_object22015:head# 2024-02-09T17:26:51.095 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1428006:::benchmark_data_smithi179_77138_object41635:head# 2024-02-09T17:26:51.095 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f142d3a8:::benchmark_data_smithi179_77138_object124318:head# 2024-02-09T17:26:51.096 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1431c2d:::benchmark_data_smithi179_77138_object37832:head# 2024-02-09T17:26:51.096 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1431f61:::benchmark_data_smithi179_77138_object73135:head# 2024-02-09T17:26:51.096 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1443b37:::benchmark_data_smithi179_77138_object110806:head# 2024-02-09T17:26:51.097 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1444e30:::benchmark_data_smithi179_77138_object64715:head# 2024-02-09T17:26:51.097 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f145049a:::benchmark_data_smithi179_77138_object87201:head# 2024-02-09T17:26:51.098 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1451542:::benchmark_data_smithi179_77138_object43499:head# 2024-02-09T17:26:51.098 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f145412d:::benchmark_data_smithi179_77138_object124775:head# 2024-02-09T17:26:51.099 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1459465:::benchmark_data_smithi179_77138_object86178:head# 2024-02-09T17:26:51.099 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f14602e4:::benchmark_data_smithi179_77138_object130468:head# 2024-02-09T17:26:51.100 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1461d77:::benchmark_data_smithi179_77138_object127016:head# 2024-02-09T17:26:51.101 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f14756ed:::benchmark_data_smithi179_77138_object94951:head# 2024-02-09T17:26:51.102 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1479c3f:::benchmark_data_smithi179_77138_object130269:head# 2024-02-09T17:26:51.102 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f148293d:::benchmark_data_smithi179_77138_object76731:head# 2024-02-09T17:26:51.103 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1489e0a:::benchmark_data_smithi179_77138_object32604:head# 2024-02-09T17:26:51.103 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1489f83:::benchmark_data_smithi179_77138_object108341:head# 2024-02-09T17:26:51.104 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f149259e:::benchmark_data_smithi179_77138_object90792:head# 2024-02-09T17:26:51.104 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1493007:::benchmark_data_smithi179_77138_object116056:head# 2024-02-09T17:26:51.104 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f14a139a:::benchmark_data_smithi179_77138_object64004:head# 2024-02-09T17:26:51.105 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f14aba19:::benchmark_data_smithi179_77138_object45329:head# 2024-02-09T17:26:51.105 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f14b57ad:::benchmark_data_smithi179_77138_object76081:head# 2024-02-09T17:26:51.106 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f14ba16a:::benchmark_data_smithi179_77138_object49497:head# 2024-02-09T17:26:51.106 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f14bdf11:::benchmark_data_smithi179_77138_object58672:head# 2024-02-09T17:26:51.107 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f14cea78:::benchmark_data_smithi179_77138_object91240:head# 2024-02-09T17:26:51.107 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f14dcfd7:::benchmark_data_smithi179_77138_object83772:head# 2024-02-09T17:26:51.107 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f14e7d65:::benchmark_data_smithi179_77138_object35124:head# 2024-02-09T17:26:51.109 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f14ec763:::benchmark_data_smithi179_77138_object124572:head# 2024-02-09T17:26:51.109 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f14f1b82:::benchmark_data_smithi179_77138_object36943:head# 2024-02-09T17:26:51.109 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f151742a:::benchmark_data_smithi179_77138_object97596:head# 2024-02-09T17:26:51.109 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f152f3c6:::benchmark_data_smithi179_77138_object103846:head# 2024-02-09T17:26:51.110 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1533357:::benchmark_data_smithi179_77138_object133300:head# 2024-02-09T17:26:51.110 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f15339be:::benchmark_data_smithi179_77138_object37043:head# 2024-02-09T17:26:51.110 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f154d806:::benchmark_data_smithi179_77138_object83204:head# 2024-02-09T17:26:51.111 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f154e1bf:::benchmark_data_smithi179_77138_object67372:head# 2024-02-09T17:26:51.111 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1554da1:::benchmark_data_smithi179_77138_object113188:head# 2024-02-09T17:26:51.112 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f155f67e:::benchmark_data_smithi179_77138_object44815:head# 2024-02-09T17:26:51.112 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1563301:::benchmark_data_smithi179_77138_object46988:head# 2024-02-09T17:26:51.114 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f156b37a:::benchmark_data_smithi179_77138_object24386:head# 2024-02-09T17:26:51.114 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f15854e4:::benchmark_data_smithi179_77138_object60058:head# 2024-02-09T17:26:51.116 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1596ae8:::benchmark_data_smithi179_77138_object21512:head# 2024-02-09T17:26:51.116 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1597386:::benchmark_data_smithi179_77138_object100192:head# 2024-02-09T17:26:51.117 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:26:51.118 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f15a1432:::benchmark_data_smithi179_77138_object87954:head# 2024-02-09T17:26:51.118 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f15b68c3:::benchmark_data_smithi179_77138_object63013:head# 2024-02-09T17:26:51.118 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f15bcb59:::benchmark_data_smithi179_77138_object80418:head# 2024-02-09T17:26:51.118 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f15cbcf9:::benchmark_data_smithi179_77138_object63941:head# 2024-02-09T17:26:51.123 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f15d5d05:::benchmark_data_smithi179_77138_object70483:head# 2024-02-09T17:26:51.124 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f15e57e0:::benchmark_data_smithi179_77138_object77603:head# 2024-02-09T17:26:51.124 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f15e675b:::benchmark_data_smithi179_77138_object86767:head# 2024-02-09T17:26:51.124 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f15f3ca8:::benchmark_data_smithi179_77138_object25904:head# 2024-02-09T17:26:51.124 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f15f924b:::benchmark_data_smithi179_77138_object93134:head# 2024-02-09T17:26:51.124 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1604c12:::benchmark_data_smithi179_77138_object49209:head# 2024-02-09T17:26:51.124 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f160fc28:::benchmark_data_smithi179_77138_object88909:head# 2024-02-09T17:26:51.124 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f161026d:::benchmark_data_smithi179_77138_object131591:head# 2024-02-09T17:26:51.124 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f161462a:::benchmark_data_smithi179_77138_object29396:head# 2024-02-09T17:26:51.124 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f162f32e:::benchmark_data_smithi179_77138_object95422:head# 2024-02-09T17:26:51.124 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f164c3f2:::benchmark_data_smithi179_77138_object50717:head# 2024-02-09T17:26:51.124 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f165105f:::benchmark_data_smithi179_77138_object112139:head# 2024-02-09T17:26:51.125 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f165241c:::benchmark_data_smithi179_77138_object117971:head# 2024-02-09T17:26:51.125 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1653e54:::benchmark_data_smithi179_77138_object25320:head# 2024-02-09T17:26:51.125 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f16550f2:::benchmark_data_smithi179_77138_object29294:head# 2024-02-09T17:26:51.125 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f167386c:::benchmark_data_smithi179_77138_object82394:head# 2024-02-09T17:26:51.126 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f167b8bf:::benchmark_data_smithi179_77138_object123859:head# 2024-02-09T17:26:51.126 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f167d88e:::benchmark_data_smithi179_77138_object51383:head# 2024-02-09T17:26:51.127 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f16817a3:::benchmark_data_smithi179_77138_object75487:head# 2024-02-09T17:26:51.127 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f16898b2:::benchmark_data_smithi179_77138_object58432:head# 2024-02-09T17:26:51.127 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1693f58:::benchmark_data_smithi179_77138_object80457:head# 2024-02-09T17:26:51.128 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f169540d:::benchmark_data_smithi179_77138_object55995:head# 2024-02-09T17:26:51.128 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1697edc:::benchmark_data_smithi179_77138_object101308:head# 2024-02-09T17:26:51.128 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f169ae8b:::benchmark_data_smithi179_77138_object126163:head# 2024-02-09T17:26:51.134 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f16a2dd3:::benchmark_data_smithi179_77138_object30849:head# 2024-02-09T17:26:51.134 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f16a4cf1:::benchmark_data_smithi179_77138_object108863:head# 2024-02-09T17:26:51.134 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f16b2e69:::benchmark_data_smithi179_77138_object64391:head# 2024-02-09T17:26:51.134 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f16b9480:::benchmark_data_smithi179_77138_object116392:head# 2024-02-09T17:26:51.134 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f16c7a1c:::benchmark_data_smithi179_77138_object35278:head# 2024-02-09T17:26:51.134 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f16ce0af:::benchmark_data_smithi179_77138_object63170:head# 2024-02-09T17:26:51.134 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f16cfab3:::benchmark_data_smithi179_77138_object38841:head# 2024-02-09T17:26:51.134 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f16d82ff:::benchmark_data_smithi179_77138_object87075:head# 2024-02-09T17:26:51.134 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f16deeff:::benchmark_data_smithi179_77138_object53084:head# 2024-02-09T17:26:51.134 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f16e2ae6:::benchmark_data_smithi179_77138_object33847:head# 2024-02-09T17:26:51.134 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f16e6550:::benchmark_data_smithi179_77138_object119528:head# 2024-02-09T17:26:51.135 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f16ec19f:::benchmark_data_smithi179_77138_object44386:head# 2024-02-09T17:26:51.135 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f16f2324:::benchmark_data_smithi179_77138_object113741:head# 2024-02-09T17:26:51.135 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f16f388a:::benchmark_data_smithi179_77138_object23982:head# 2024-02-09T17:26:51.136 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f16feb53:::benchmark_data_smithi179_77138_object22389:head# 2024-02-09T17:26:51.136 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1703ee5:::benchmark_data_smithi179_77138_object52519:head# 2024-02-09T17:26:51.136 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f17100e9:::benchmark_data_smithi179_77138_object78834:head# 2024-02-09T17:26:51.137 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1717b7d:::benchmark_data_smithi179_77138_object66845:head# 2024-02-09T17:26:51.137 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f171dbd5:::benchmark_data_smithi179_77138_object125612:head# 2024-02-09T17:26:51.143 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f171e3db:::benchmark_data_smithi179_77138_object92843:head# 2024-02-09T17:26:51.143 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f17296ff:::benchmark_data_smithi179_77138_object121427:head# 2024-02-09T17:26:51.143 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f17474d9:::benchmark_data_smithi179_77138_object61372:head# 2024-02-09T17:26:51.144 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f174d063:::benchmark_data_smithi179_77138_object125095:head# 2024-02-09T17:26:51.144 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f175202d:::benchmark_data_smithi179_77138_object104721:head# 2024-02-09T17:26:51.144 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1752936:::benchmark_data_smithi179_77138_object48124:head# 2024-02-09T17:26:51.145 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1753ea8:::benchmark_data_smithi179_77138_object70789:head# 2024-02-09T17:26:51.145 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f177a410:::benchmark_data_smithi179_77138_object84151:head# 2024-02-09T17:26:51.145 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f177abf4:::benchmark_data_smithi179_77138_object75544:head# 2024-02-09T17:26:51.146 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f179407d:::benchmark_data_smithi179_77138_object20364:head# 2024-02-09T17:26:51.147 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f179dcfb:::benchmark_data_smithi179_77138_object126511:head# 2024-02-09T17:26:51.147 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f17abafb:::benchmark_data_smithi179_77138_object31512:head# 2024-02-09T17:26:51.148 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f17b2e4a:::benchmark_data_smithi179_77138_object130601:head# 2024-02-09T17:26:51.148 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f17dc295:::benchmark_data_smithi179_77138_object112961:head# 2024-02-09T17:26:51.148 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f17dec4f:::benchmark_data_smithi179_77138_object84490:head# 2024-02-09T17:26:51.149 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f17e0ff5:::benchmark_data_smithi179_77138_object121822:head# 2024-02-09T17:26:51.150 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f17e3356:::benchmark_data_smithi179_77138_object69048:head# 2024-02-09T17:26:51.150 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f17eedd6:::benchmark_data_smithi179_77138_object65423:head# 2024-02-09T17:26:51.150 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f17ef2d7:::benchmark_data_smithi179_77138_object124030:head# 2024-02-09T17:26:51.151 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f17f3967:::benchmark_data_smithi179_77138_object118645:head# 2024-02-09T17:26:51.152 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f17f69ea:::benchmark_data_smithi179_77138_object106975:head# 2024-02-09T17:26:51.156 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f17f6be4:::benchmark_data_smithi179_77138_object32278:head# 2024-02-09T17:26:51.156 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f17fab4b:::benchmark_data_smithi179_77138_object50227:head# 2024-02-09T17:26:51.157 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f18068c3:::benchmark_data_smithi179_77138_object31960:head# 2024-02-09T17:26:51.157 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f180d756:::benchmark_data_smithi179_77138_object105881:head# 2024-02-09T17:26:51.157 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f18155c7:::benchmark_data_smithi179_77138_object100530:head# 2024-02-09T17:26:51.157 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1815e06:::benchmark_data_smithi179_77138_object121794:head# 2024-02-09T17:26:51.157 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f181ff52:::benchmark_data_smithi179_77138_object77861:head# 2024-02-09T17:26:51.157 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1820a3e:::benchmark_data_smithi179_77138_object96435:head# 2024-02-09T17:26:51.157 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1827e80:::benchmark_data_smithi179_77138_object84799:head# 2024-02-09T17:26:51.158 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f18334b3:::benchmark_data_smithi179_77138_object41626:head# 2024-02-09T17:26:51.158 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f183d59c:::benchmark_data_smithi179_77138_object124852:head# 2024-02-09T17:26:51.158 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f18405ad:::benchmark_data_smithi179_77138_object109079:head# 2024-02-09T17:26:51.158 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1849c04:::benchmark_data_smithi179_77138_object90210:head# 2024-02-09T17:26:51.159 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f184a462:::benchmark_data_smithi179_77138_object26912:head# 2024-02-09T17:26:51.159 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f184dfa3:::benchmark_data_smithi179_77138_object46638:head# 2024-02-09T17:26:51.160 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:51.160 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f18581c0:::benchmark_data_smithi179_77138_object86275:head# 2024-02-09T17:26:51.160 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f185ac49:::benchmark_data_smithi179_77138_object112408:head# 2024-02-09T17:26:51.160 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f185e330:::benchmark_data_smithi179_77138_object60067:head# 2024-02-09T17:26:51.161 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f185ecd0:::benchmark_data_smithi179_77138_object37668:head# 2024-02-09T17:26:51.161 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1869380:::benchmark_data_smithi179_77138_object55943:head# 2024-02-09T17:26:51.162 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1872290:::benchmark_data_smithi179_77138_object93549:head# 2024-02-09T17:26:51.162 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f18771a5:::benchmark_data_smithi179_77138_object28321:head# 2024-02-09T17:26:51.162 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f187f7d9:::benchmark_data_smithi179_77138_object95493:head# 2024-02-09T17:26:51.163 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1881ce5:::benchmark_data_smithi179_77138_object110550:head# 2024-02-09T17:26:51.163 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f188eb0e:::benchmark_data_smithi179_77138_object47639:head# 2024-02-09T17:26:51.164 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f189c65d:::benchmark_data_smithi179_77138_object115005:head# 2024-02-09T17:26:51.164 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f189cabe:::benchmark_data_smithi179_77138_object39560:head# 2024-02-09T17:26:51.166 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f18a5231:::benchmark_data_smithi179_77138_object70414:head# 2024-02-09T17:26:51.166 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f18a73a6:::benchmark_data_smithi179_77138_object115789:head# 2024-02-09T17:26:51.166 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f18aec2b:::benchmark_data_smithi179_77138_object123786:head# 2024-02-09T17:26:51.167 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f18bb10e:::benchmark_data_smithi179_77138_object107039:head# 2024-02-09T17:26:51.167 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f18d0213:::benchmark_data_smithi179_77138_object127496:head# 2024-02-09T17:26:51.168 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f18d4bbb:::benchmark_data_smithi179_77138_object73577:head# 2024-02-09T17:26:51.168 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f18ec66f:::benchmark_data_smithi179_77138_object93107:head# 2024-02-09T17:26:51.168 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f18f07ec:::benchmark_data_smithi179_77138_object43162:head# 2024-02-09T17:26:51.168 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f18fb03f:::benchmark_data_smithi179_77138_object41909:head# 2024-02-09T17:26:51.169 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f190049b:::benchmark_data_smithi179_77138_object96712:head# 2024-02-09T17:26:51.169 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f19034bc:::benchmark_data_smithi179_77138_object54556:head# 2024-02-09T17:26:51.169 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1908c87:::benchmark_data_smithi179_77138_object106038:head# 2024-02-09T17:26:51.170 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1908cf0:::benchmark_data_smithi179_77138_object49617:head# 2024-02-09T17:26:51.171 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f190f1c4:::benchmark_data_smithi179_77138_object66116:head# 2024-02-09T17:26:51.171 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f19141df:::benchmark_data_smithi179_77138_object83656:head# 2024-02-09T17:26:51.171 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f191903b:::benchmark_data_smithi179_77138_object48166:head# 2024-02-09T17:26:51.172 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f193f379:::benchmark_data_smithi179_77138_object57612:head# 2024-02-09T17:26:51.172 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1947b91:::benchmark_data_smithi179_77138_object59107:head# 2024-02-09T17:26:51.173 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f194c98a:::benchmark_data_smithi179_77138_object40586:head# 2024-02-09T17:26:51.173 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1954709:::benchmark_data_smithi179_77138_object90914:head# 2024-02-09T17:26:51.174 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1962ce5:::benchmark_data_smithi179_77138_object59474:head# 2024-02-09T17:26:51.174 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f196b692:::benchmark_data_smithi179_77138_object115065:head# 2024-02-09T17:26:51.175 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f197b0c5:::benchmark_data_smithi179_77138_object55838:head# 2024-02-09T17:26:51.175 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f19866f1:::benchmark_data_smithi179_77138_object122887:head# 2024-02-09T17:26:51.176 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f19b6a21:::benchmark_data_smithi179_77138_object85488:head# 2024-02-09T17:26:51.176 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f19b6f8c:::benchmark_data_smithi179_77138_object95399:head# 2024-02-09T17:26:51.177 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f19bad25:::benchmark_data_smithi179_77138_object40830:head# 2024-02-09T17:26:51.177 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f19c0203:::benchmark_data_smithi179_77138_object48586:head# 2024-02-09T17:26:51.177 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f19c10b5:::benchmark_data_smithi179_77138_object84073:head# 2024-02-09T17:26:51.178 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f19c4565:::benchmark_data_smithi179_77138_object58281:head# 2024-02-09T17:26:51.178 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f19cdf58:::benchmark_data_smithi179_77138_object108858:head# 2024-02-09T17:26:51.179 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f19d5a2d:::benchmark_data_smithi179_77138_object109366:head# 2024-02-09T17:26:51.179 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f19da43c:::benchmark_data_smithi179_77138_object121147:head# 2024-02-09T17:26:51.179 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f19f2265:::benchmark_data_smithi179_77138_object67723:head# 2024-02-09T17:26:51.180 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1a054bc:::benchmark_data_smithi179_77138_object71170:head# 2024-02-09T17:26:51.180 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1a0ee6c:::benchmark_data_smithi179_77138_object84420:head# 2024-02-09T17:26:51.181 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1a12519:::benchmark_data_smithi179_77138_object53321:head# 2024-02-09T17:26:51.181 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1a12c87:::benchmark_data_smithi179_77138_object71877:head# 2024-02-09T17:26:51.182 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1a14cf6:::benchmark_data_smithi179_77138_object68979:head# 2024-02-09T17:26:51.182 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1a1f1bc:::benchmark_data_smithi179_77138_object133311:head# 2024-02-09T17:26:51.183 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1a2c762:::benchmark_data_smithi179_77138_object31866:head# 2024-02-09T17:26:51.183 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1a3c1fa:::benchmark_data_smithi179_77138_object93956:head# 2024-02-09T17:26:51.184 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1a49f32:::benchmark_data_smithi179_77138_object53462:head# 2024-02-09T17:26:51.184 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1a51cc0:::benchmark_data_smithi179_77138_object110254:head# 2024-02-09T17:26:51.185 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1a54677:::benchmark_data_smithi179_77138_object100445:head# 2024-02-09T17:26:51.185 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1a5a9c3:::benchmark_data_smithi179_77138_object39693:head# 2024-02-09T17:26:51.185 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1a606f6:::benchmark_data_smithi179_77138_object106501:head# 2024-02-09T17:26:51.186 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1a6c99f:::benchmark_data_smithi179_77138_object82996:head# 2024-02-09T17:26:51.186 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1a6d4de:::benchmark_data_smithi179_77138_object127894:head# 2024-02-09T17:26:51.187 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1a75f69:::benchmark_data_smithi179_77138_object127220:head# 2024-02-09T17:26:51.187 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1a76041:::benchmark_data_smithi179_77138_object104949:head# 2024-02-09T17:26:51.195 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1a7a28a:::benchmark_data_smithi179_77138_object71121:head# 2024-02-09T17:26:51.195 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1a7c53a:::benchmark_data_smithi179_77138_object48762:head# 2024-02-09T17:26:51.196 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1a7e2f4:::benchmark_data_smithi179_77138_object34241:head# 2024-02-09T17:26:51.200 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1a7eb3b:::benchmark_data_smithi179_77138_object25496:head# 2024-02-09T17:26:51.200 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1a80f2e:::benchmark_data_smithi179_77138_object62540:head# 2024-02-09T17:26:51.201 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1a8110f:::benchmark_data_smithi179_77138_object27909:head# 2024-02-09T17:26:51.208 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1a86b09:::benchmark_data_smithi179_77138_object96792:head# 2024-02-09T17:26:51.208 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1a8c30c:::benchmark_data_smithi179_77138_object118083:head# 2024-02-09T17:26:51.208 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1a984db:::benchmark_data_smithi179_77138_object47885:head# 2024-02-09T17:26:51.208 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1aae775:::benchmark_data_smithi179_77138_object124229:head# 2024-02-09T17:26:51.208 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1ab2103:::benchmark_data_smithi179_77138_object129483:head# 2024-02-09T17:26:51.209 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1ab4a37:::benchmark_data_smithi179_77138_object129037:head# 2024-02-09T17:26:51.209 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1ac2f25:::benchmark_data_smithi179_77138_object76501:head# 2024-02-09T17:26:51.209 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1ac4527:::benchmark_data_smithi179_77138_object113597:head# 2024-02-09T17:26:51.210 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1acf853:::benchmark_data_smithi179_77138_object58135:head# 2024-02-09T17:26:51.210 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1ad8865:::benchmark_data_smithi179_77138_object30487:head# 2024-02-09T17:26:51.210 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1adc04c:::benchmark_data_smithi179_77138_object28047:head# 2024-02-09T17:26:51.211 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1adfeb4:::benchmark_data_smithi179_77138_object111274:head# 2024-02-09T17:26:51.211 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1ae1c0e:::benchmark_data_smithi179_77138_object112564:head# 2024-02-09T17:26:51.212 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1ae44a7:::benchmark_data_smithi179_77138_object78363:head# 2024-02-09T17:26:51.214 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1ae8e7e:::benchmark_data_smithi179_77138_object113235:head# 2024-02-09T17:26:51.215 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1afb960:::benchmark_data_smithi179_77138_object77616:head# 2024-02-09T17:26:51.215 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1afc207:::benchmark_data_smithi179_77138_object126068:head# 2024-02-09T17:26:51.215 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1b12440:::benchmark_data_smithi179_77138_object65610:head# 2024-02-09T17:26:51.216 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1b146fc:::benchmark_data_smithi179_77138_object116517:head# 2024-02-09T17:26:51.217 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1b1d6e1:::benchmark_data_smithi179_77138_object83464:head# 2024-02-09T17:26:51.218 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1b2aa83:::benchmark_data_smithi179_77138_object133141:head# 2024-02-09T17:26:51.218 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1b46861:::benchmark_data_smithi179_77138_object57279:head# 2024-02-09T17:26:51.219 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1b57f2a:::benchmark_data_smithi179_77138_object54816:head# 2024-02-09T17:26:51.220 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1b5f078:::benchmark_data_smithi179_77138_object121123:head# 2024-02-09T17:26:51.220 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1b64cc8:::benchmark_data_smithi179_77138_object50729:head# 2024-02-09T17:26:51.221 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1b65ef3:::benchmark_data_smithi179_77138_object29758:head# 2024-02-09T17:26:51.221 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1b6bd7c:::benchmark_data_smithi179_77138_object118461:head# 2024-02-09T17:26:51.222 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1b6f5f8:::benchmark_data_smithi179_77138_object84394:head# 2024-02-09T17:26:51.222 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1b7ad61:::benchmark_data_smithi179_77138_object113050:head# 2024-02-09T17:26:51.225 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1b99ec7:::benchmark_data_smithi179_77138_object126672:head# 2024-02-09T17:26:51.225 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1ba0826:::benchmark_data_smithi179_77138_object58676:head# 2024-02-09T17:26:51.226 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1baaa23:::benchmark_data_smithi179_77138_object109227:head# 2024-02-09T17:26:51.226 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1badfcc:::benchmark_data_smithi179_77138_object44982:head# 2024-02-09T17:26:51.227 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1beb986:::benchmark_data_smithi179_77138_object32772:head# 2024-02-09T17:26:51.227 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1bef2a9:::benchmark_data_smithi179_77138_object51887:head# 2024-02-09T17:26:51.230 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1bfdc5e:::benchmark_data_smithi179_77138_object108952:head# 2024-02-09T17:26:51.231 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1bfebb5:::benchmark_data_smithi179_77138_object122926:head# 2024-02-09T17:26:51.231 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1c084a3:::benchmark_data_smithi179_77138_object23849:head# 2024-02-09T17:26:51.232 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1c11d57:::benchmark_data_smithi179_77138_object72927:head# 2024-02-09T17:26:51.232 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1c17389:::benchmark_data_smithi179_77138_object63153:head# 2024-02-09T17:26:51.232 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1c17865:::benchmark_data_smithi179_77138_object91398:head# 2024-02-09T17:26:51.233 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1c17eb1:::benchmark_data_smithi179_77138_object123815:head# 2024-02-09T17:26:51.233 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1c258a9:::benchmark_data_smithi179_77138_object104206:head# 2024-02-09T17:26:51.234 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1c30875:::benchmark_data_smithi179_77138_object89090:head# 2024-02-09T17:26:51.234 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1c34b1c:::benchmark_data_smithi179_77138_object76424:head# 2024-02-09T17:26:51.235 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1c4ac00:::benchmark_data_smithi179_77138_object91417:head# 2024-02-09T17:26:51.235 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1c57296:::benchmark_data_smithi179_77138_object112994:head# 2024-02-09T17:26:51.236 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1c5918a:::benchmark_data_smithi179_77138_object110395:head# 2024-02-09T17:26:51.236 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1c6ac88:::benchmark_data_smithi179_77138_object44361:head# 2024-02-09T17:26:51.237 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1c76d74:::benchmark_data_smithi179_77138_object53804:head# 2024-02-09T17:26:51.237 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1c8193b:::benchmark_data_smithi179_77138_object68616:head# 2024-02-09T17:26:51.240 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1c82e5e:::benchmark_data_smithi179_77138_object102239:head# 2024-02-09T17:26:51.240 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1c88b56:::benchmark_data_smithi179_77138_object97167:head# 2024-02-09T17:26:51.241 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1c8e04d:::benchmark_data_smithi179_77138_object121769:head# 2024-02-09T17:26:51.241 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1c8eab3:::benchmark_data_smithi179_77138_object40131:head# 2024-02-09T17:26:51.241 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1c9a670:::benchmark_data_smithi179_77138_object63620:head# 2024-02-09T17:26:51.242 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1c9dac5:::benchmark_data_smithi179_77138_object54155:head# 2024-02-09T17:26:51.244 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1c9e3bf:::benchmark_data_smithi179_77138_object107368:head# 2024-02-09T17:26:51.244 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1ca1e64:::benchmark_data_smithi179_77138_object42688:head# 2024-02-09T17:26:51.244 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1cb32ca:::benchmark_data_smithi179_77138_object128151:head# 2024-02-09T17:26:51.245 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1cb86eb:::benchmark_data_smithi179_77138_object34774:head# 2024-02-09T17:26:51.245 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1cbfbfc:::benchmark_data_smithi179_77138_object80149:head# 2024-02-09T17:26:51.245 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1cbff87:::benchmark_data_smithi179_77138_object81276:head# 2024-02-09T17:26:51.245 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1cc2c58:::benchmark_data_smithi179_77138_object55075:head# 2024-02-09T17:26:51.246 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1cc98f1:::benchmark_data_smithi179_77138_object101615:head# 2024-02-09T17:26:51.246 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1ccc5cd:::benchmark_data_smithi179_77138_object82044:head# 2024-02-09T17:26:51.247 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1cdd4cc:::benchmark_data_smithi179_77138_object117705:head# 2024-02-09T17:26:51.249 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1ce361b:::benchmark_data_smithi179_77138_object83938:head# 2024-02-09T17:26:51.249 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1ce4a0a:::benchmark_data_smithi179_77138_object103785:head# 2024-02-09T17:26:51.250 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1ce86fe:::benchmark_data_smithi179_77138_object26230:head# 2024-02-09T17:26:51.250 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1ce9210:::benchmark_data_smithi179_77138_object104382:head# 2024-02-09T17:26:51.251 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1cf6c71:::benchmark_data_smithi179_77138_object26384:head# 2024-02-09T17:26:51.251 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1d07393:::benchmark_data_smithi179_77138_object124912:head# 2024-02-09T17:26:51.252 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1d1ad82:::benchmark_data_smithi179_77138_object71711:head# 2024-02-09T17:26:51.252 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1d287dd:::benchmark_data_smithi179_77138_object104072:head# 2024-02-09T17:26:51.253 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1d2ceac:::benchmark_data_smithi179_77138_object53171:head# 2024-02-09T17:26:51.255 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1d419f6:::benchmark_data_smithi179_77138_object74101:head# 2024-02-09T17:26:51.255 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1d47f19:::benchmark_data_smithi179_77138_object45673:head# 2024-02-09T17:26:51.255 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1d5e941:::benchmark_data_smithi179_77138_object49175:head# 2024-02-09T17:26:51.255 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1d67ec1:::benchmark_data_smithi179_77138_object42812:head# 2024-02-09T17:26:51.255 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1d6f76b:::benchmark_data_smithi179_77138_object120376:head# 2024-02-09T17:26:51.256 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1d7247f:::benchmark_data_smithi179_77138_object77198:head# 2024-02-09T17:26:51.256 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1d7e584:::benchmark_data_smithi179_77138_object77177:head# 2024-02-09T17:26:51.261 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1d8ae54:::benchmark_data_smithi179_77138_object59837:head# 2024-02-09T17:26:51.261 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1d95e08:::benchmark_data_smithi179_77138_object113289:head# 2024-02-09T17:26:51.261 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1d98b5f:::benchmark_data_smithi179_77138_object107928:head# 2024-02-09T17:26:51.261 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1da2d08:::benchmark_data_smithi179_77138_object62475:head# 2024-02-09T17:26:51.261 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1dae9af:::benchmark_data_smithi179_77138_object80755:head# 2024-02-09T17:26:51.262 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1db7304:::benchmark_data_smithi179_77138_object51695:head# 2024-02-09T17:26:51.267 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1dde9a8:::benchmark_data_smithi179_77138_object77067:head# 2024-02-09T17:26:51.267 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1de0859:::benchmark_data_smithi179_77138_object38810:head# 2024-02-09T17:26:51.267 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1de722a:::benchmark_data_smithi179_77138_object38592:head# 2024-02-09T17:26:51.267 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1deb7b3:::benchmark_data_smithi179_77138_object100949:head# 2024-02-09T17:26:51.267 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1df23fd:::benchmark_data_smithi179_77138_object129153:head# 2024-02-09T17:26:51.267 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1df2667:::benchmark_data_smithi179_77138_object27872:head# 2024-02-09T17:26:51.267 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1dfa24b:::benchmark_data_smithi179_77138_object99401:head# 2024-02-09T17:26:51.267 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1dfa39f:::benchmark_data_smithi179_77138_object20583:head# 2024-02-09T17:26:51.270 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1dffe32:::benchmark_data_smithi179_77138_object69036:head# 2024-02-09T17:26:51.270 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1e04fb6:::benchmark_data_smithi179_77138_object65715:head# 2024-02-09T17:26:51.270 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1e087a1:::benchmark_data_smithi179_77138_object132635:head# 2024-02-09T17:26:51.271 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1e0dfb2:::benchmark_data_smithi179_77138_object74102:head# 2024-02-09T17:26:51.271 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1e15537:::benchmark_data_smithi179_77138_object75038:head# 2024-02-09T17:26:51.271 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1e1ec79:::benchmark_data_smithi179_77138_object92713:head# 2024-02-09T17:26:51.272 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1e3dc07:::benchmark_data_smithi179_77138_object82404:head# 2024-02-09T17:26:51.276 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1e4a30e:::benchmark_data_smithi179_77138_object27510:head# 2024-02-09T17:26:51.276 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1e526b1:::benchmark_data_smithi179_77138_object105185:head# 2024-02-09T17:26:51.279 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1e598db:::benchmark_data_smithi179_77138_object28187:head# 2024-02-09T17:26:51.282 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1e5e0a2:::benchmark_data_smithi179_77138_object112869:head# 2024-02-09T17:26:51.282 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1e640c0:::benchmark_data_smithi179_77138_object118691:head# 2024-02-09T17:26:51.283 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1e68731:::benchmark_data_smithi179_77138_object51819:head# 2024-02-09T17:26:51.285 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1e6c31d:::benchmark_data_smithi179_77138_object39520:head# 2024-02-09T17:26:51.285 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1e7a67a:::benchmark_data_smithi179_77138_object107415:head# 2024-02-09T17:26:51.287 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1e808a0:::benchmark_data_smithi179_77138_object115890:head# 2024-02-09T17:26:51.287 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1e8155d:::benchmark_data_smithi179_77138_object34773:head# 2024-02-09T17:26:51.288 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1e8e8cd:::benchmark_data_smithi179_77138_object127586:head# 2024-02-09T17:26:51.288 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1e9b09e:::benchmark_data_smithi179_77138_object36714:head# 2024-02-09T17:26:51.289 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1ea682b:::benchmark_data_smithi179_77138_object81154:head# 2024-02-09T17:26:51.289 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1eab609:::benchmark_data_smithi179_77138_object91017:head# 2024-02-09T17:26:51.289 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1eb4c61:::benchmark_data_smithi179_77138_object26077:head# 2024-02-09T17:26:51.290 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1ec8dcd:::benchmark_data_smithi179_77138_object82797:head# 2024-02-09T17:26:51.290 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1ed6181:::benchmark_data_smithi179_77138_object35533:head# 2024-02-09T17:26:51.291 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1eddeef:::benchmark_data_smithi179_77138_object80766:head# 2024-02-09T17:26:51.291 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1ef6a99:::benchmark_data_smithi179_77138_object71920:head# 2024-02-09T17:26:51.292 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1f064a8:::benchmark_data_smithi179_77138_object64410:head# 2024-02-09T17:26:51.293 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:26:51.295 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1f189f9:::benchmark_data_smithi179_77138_object38374:head# 2024-02-09T17:26:51.295 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1f23faa:::benchmark_data_smithi179_77138_object52801:head# 2024-02-09T17:26:51.295 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1f280ea:::benchmark_data_smithi179_77138_object124361:head# 2024-02-09T17:26:51.295 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1f33f1a:::benchmark_data_smithi179_77138_object37385:head# 2024-02-09T17:26:51.295 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1f3edec:::benchmark_data_smithi179_77138_object79015:head# 2024-02-09T17:26:51.296 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1f409de:::benchmark_data_smithi179_77138_object79504:head# 2024-02-09T17:26:51.296 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1f40a2e:::benchmark_data_smithi179_77138_object36491:head# 2024-02-09T17:26:51.296 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1f4209c:::benchmark_data_smithi179_77138_object83606:head# 2024-02-09T17:26:51.297 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1f44f19:::benchmark_data_smithi179_77138_object87738:head# 2024-02-09T17:26:51.298 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1f455a0:::benchmark_data_smithi179_77138_object93429:head# 2024-02-09T17:26:51.299 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1f640b6:::benchmark_data_smithi179_77138_object53337:head# 2024-02-09T17:26:51.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:51.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:26:51.301 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1f840d4:::benchmark_data_smithi179_77138_object71998:head# 2024-02-09T17:26:51.301 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1f90a84:::benchmark_data_smithi179_77138_object97008:head# 2024-02-09T17:26:51.301 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1f95189:::benchmark_data_smithi179_77138_object69334:head# 2024-02-09T17:26:51.301 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1fa9214:::benchmark_data_smithi179_77138_object74832:head# 2024-02-09T17:26:51.302 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1fabb80:::benchmark_data_smithi179_77138_object43909:head# 2024-02-09T17:26:51.302 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1fbdc27:::benchmark_data_smithi179_77138_object75550:head# 2024-02-09T17:26:51.302 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1fc7c5f:::benchmark_data_smithi179_77138_object78772:head# 2024-02-09T17:26:51.303 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1fcb422:::benchmark_data_smithi179_77138_object25888:head# 2024-02-09T17:26:51.303 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1fcbdf0:::benchmark_data_smithi179_77138_object67330:head# 2024-02-09T17:26:51.304 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1fcc769:::benchmark_data_smithi179_77138_object73961:head# 2024-02-09T17:26:51.305 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1fd14e9:::benchmark_data_smithi179_77138_object34702:head# 2024-02-09T17:26:51.305 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1fd26ff:::benchmark_data_smithi179_77138_object98363:head# 2024-02-09T17:26:51.306 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1fd6017:::benchmark_data_smithi179_77138_object54951:head# 2024-02-09T17:26:51.306 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1fed20e:::benchmark_data_smithi179_77138_object121522:head# 2024-02-09T17:26:51.307 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1fef110:::benchmark_data_smithi179_77138_object69350:head# 2024-02-09T17:26:51.307 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f1fefffd:::benchmark_data_smithi179_77138_object101412:head# 2024-02-09T17:26:51.311 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f200de02:::benchmark_data_smithi179_77138_object19325:head# 2024-02-09T17:26:51.311 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2035d4e:::benchmark_data_smithi179_77138_object43030:head# 2024-02-09T17:26:51.311 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f203d6f1:::benchmark_data_smithi179_77138_object34380:head# 2024-02-09T17:26:51.312 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f204180d:::benchmark_data_smithi179_77138_object100390:head# 2024-02-09T17:26:51.312 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2041c3b:::benchmark_data_smithi179_77138_object78621:head# 2024-02-09T17:26:51.312 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2043c25:::benchmark_data_smithi179_77138_object42833:head# 2024-02-09T17:26:51.312 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f20460a0:::benchmark_data_smithi179_77138_object126837:head# 2024-02-09T17:26:51.312 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2046f60:::benchmark_data_smithi179_77138_object91813:head# 2024-02-09T17:26:51.315 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2047b16:::benchmark_data_smithi179_77138_object81689:head# 2024-02-09T17:26:51.319 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f204fcd5:::benchmark_data_smithi179_77138_object63247:head# 2024-02-09T17:26:51.319 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2057d55:::benchmark_data_smithi179_77138_object28421:head# 2024-02-09T17:26:51.321 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f205c12b:::benchmark_data_smithi179_77138_object88003:head# 2024-02-09T17:26:51.322 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f20657e3:::benchmark_data_smithi179_77138_object28448:head# 2024-02-09T17:26:51.322 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f20684e4:::benchmark_data_smithi179_77138_object97950:head# 2024-02-09T17:26:51.325 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2079957:::benchmark_data_smithi179_77138_object73729:head# 2024-02-09T17:26:51.325 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2087989:::benchmark_data_smithi179_77138_object48834:head# 2024-02-09T17:26:51.326 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f208a601:::benchmark_data_smithi179_77138_object34838:head# 2024-02-09T17:26:51.326 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f208c432:::benchmark_data_smithi179_77138_object67872:head# 2024-02-09T17:26:51.327 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2091b6d:::benchmark_data_smithi179_77138_object78159:head# 2024-02-09T17:26:51.327 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2092363:::benchmark_data_smithi179_77138_object113426:head# 2024-02-09T17:26:51.328 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f20b5b29:::benchmark_data_smithi179_77138_object25584:head# 2024-02-09T17:26:51.328 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f20bcc1e:::benchmark_data_smithi179_77138_object20382:head# 2024-02-09T17:26:51.329 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f20c2393:::benchmark_data_smithi179_77138_object114421:head# 2024-02-09T17:26:51.329 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f20d91f4:::benchmark_data_smithi179_77138_object117025:head# 2024-02-09T17:26:51.330 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f20e7bad:::benchmark_data_smithi179_77138_object29016:head# 2024-02-09T17:26:51.330 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f20f2c64:::benchmark_data_smithi179_77138_object114600:head# 2024-02-09T17:26:51.330 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2101c30:::benchmark_data_smithi179_77138_object88459:head# 2024-02-09T17:26:51.331 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f21078d2:::benchmark_data_smithi179_77138_object122127:head# 2024-02-09T17:26:51.331 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f21209e8:::benchmark_data_smithi179_77138_object41123:head# 2024-02-09T17:26:51.332 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f212e17a:::benchmark_data_smithi179_77138_object132462:head# 2024-02-09T17:26:51.332 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f212fa22:::benchmark_data_smithi179_77138_object99451:head# 2024-02-09T17:26:51.333 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f21312ae:::benchmark_data_smithi179_77138_object67745:head# 2024-02-09T17:26:51.333 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f214745b:::benchmark_data_smithi179_77138_object45697:head# 2024-02-09T17:26:51.333 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2147cdf:::benchmark_data_smithi179_77138_object32750:head# 2024-02-09T17:26:51.334 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f214bed1:::benchmark_data_smithi179_77138_object113949:head# 2024-02-09T17:26:51.335 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2153f2a:::benchmark_data_smithi179_77138_object19856:head# 2024-02-09T17:26:51.335 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f21545a2:::benchmark_data_smithi179_77138_object25748:head# 2024-02-09T17:26:51.335 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f21742aa:::benchmark_data_smithi179_77138_object105458:head# 2024-02-09T17:26:51.336 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2174ad0:::benchmark_data_smithi179_77138_object128939:head# 2024-02-09T17:26:51.340 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f217fdcc:::benchmark_data_smithi179_77138_object129691:head# 2024-02-09T17:26:51.340 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f217ff45:::benchmark_data_smithi179_77138_object42405:head# 2024-02-09T17:26:51.340 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f218a8e0:::benchmark_data_smithi179_77138_object69034:head# 2024-02-09T17:26:51.341 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f218f974:::benchmark_data_smithi179_77138_object121368:head# 2024-02-09T17:26:51.341 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2190357:::benchmark_data_smithi179_77138_object43212:head# 2024-02-09T17:26:51.341 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f21914b9:::benchmark_data_smithi179_77138_object69834:head# 2024-02-09T17:26:51.341 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f219c72b:::benchmark_data_smithi179_77138_object95515:head# 2024-02-09T17:26:51.341 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f219cf4e:::benchmark_data_smithi179_77138_object38630:head# 2024-02-09T17:26:51.341 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f21b3831:::benchmark_data_smithi179_77138_object131038:head# 2024-02-09T17:26:51.341 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f21c7ce1:::benchmark_data_smithi179_77138_object111918:head# 2024-02-09T17:26:51.342 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f21cd53b:::benchmark_data_smithi179_77138_object131902:head# 2024-02-09T17:26:51.342 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f21cd57a:::benchmark_data_smithi179_77138_object83147:head# 2024-02-09T17:26:51.342 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f21d1efe:::benchmark_data_smithi179_77138_object39849:head# 2024-02-09T17:26:51.342 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f21f35c4:::benchmark_data_smithi179_77138_object61148:head# 2024-02-09T17:26:51.343 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f21fb0cf:::benchmark_data_smithi179_77138_object131362:head# 2024-02-09T17:26:51.343 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f21fd2a9:::benchmark_data_smithi179_77138_object39134:head# 2024-02-09T17:26:51.344 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f220932d:::benchmark_data_smithi179_77138_object48217:head# 2024-02-09T17:26:51.344 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2213042:::benchmark_data_smithi179_77138_object37337:head# 2024-02-09T17:26:51.345 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2214223:::benchmark_data_smithi179_77138_object102472:head# 2024-02-09T17:26:51.345 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2214789:::benchmark_data_smithi179_77138_object120080:head# 2024-02-09T17:26:51.345 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2214bea:::benchmark_data_smithi179_77138_object59008:head# 2024-02-09T17:26:51.346 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f222d717:::benchmark_data_smithi179_77138_object86314:head# 2024-02-09T17:26:51.346 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2230dd1:::benchmark_data_smithi179_77138_object104320:head# 2024-02-09T17:26:51.346 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2231bac:::benchmark_data_smithi179_77138_object29578:head# 2024-02-09T17:26:51.347 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2239f83:::benchmark_data_smithi179_77138_object89802:head# 2024-02-09T17:26:51.348 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f223a0d8:::benchmark_data_smithi179_77138_object64537:head# 2024-02-09T17:26:51.348 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f224f76c:::benchmark_data_smithi179_77138_object52878:head# 2024-02-09T17:26:51.349 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2252e63:::benchmark_data_smithi179_77138_object123457:head# 2024-02-09T17:26:51.349 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2259735:::benchmark_data_smithi179_77138_object122398:head# 2024-02-09T17:26:51.350 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f225aad3:::benchmark_data_smithi179_77138_object90261:head# 2024-02-09T17:26:51.350 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f225e3b7:::benchmark_data_smithi179_77138_object47347:head# 2024-02-09T17:26:51.350 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f226513f:::benchmark_data_smithi179_77138_object86588:head# 2024-02-09T17:26:51.351 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f22710f8:::benchmark_data_smithi179_77138_object56335:head# 2024-02-09T17:26:51.351 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2279e11:::benchmark_data_smithi179_77138_object88681:head# 2024-02-09T17:26:51.352 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f227a41f:::benchmark_data_smithi179_77138_object31543:head# 2024-02-09T17:26:51.352 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f227c7c2:::benchmark_data_smithi179_77138_object102051:head# 2024-02-09T17:26:51.353 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f22957f9:::benchmark_data_smithi179_77138_object51720:head# 2024-02-09T17:26:51.353 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f229600e:::benchmark_data_smithi179_77138_object59225:head# 2024-02-09T17:26:51.358 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f229a76d:::benchmark_data_smithi179_77138_object68297:head# 2024-02-09T17:26:51.358 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f229b792:::benchmark_data_smithi179_77138_object98655:head# 2024-02-09T17:26:51.358 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f22a0e4c:::benchmark_data_smithi179_77138_object102684:head# 2024-02-09T17:26:51.359 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f22abfa6:::benchmark_data_smithi179_77138_object20298:head# 2024-02-09T17:26:51.359 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f22acce2:::benchmark_data_smithi179_77138_object68828:head# 2024-02-09T17:26:51.359 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f22b15b4:::benchmark_data_smithi179_77138_object77922:head# 2024-02-09T17:26:51.359 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f22b3c81:::benchmark_data_smithi179_77138_object51431:head# 2024-02-09T17:26:51.359 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f22d5a14:::benchmark_data_smithi179_77138_object55973:head# 2024-02-09T17:26:51.359 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f22daa59:::benchmark_data_smithi179_77138_object128050:head# 2024-02-09T17:26:51.359 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f22db8ea:::benchmark_data_smithi179_77138_object55041:head# 2024-02-09T17:26:51.359 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f22dce60:::benchmark_data_smithi179_77138_object129571:head# 2024-02-09T17:26:51.359 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f22dd9fa:::benchmark_data_smithi179_77138_object57773:head# 2024-02-09T17:26:51.363 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f22e2da4:::benchmark_data_smithi179_77138_object100733:head# 2024-02-09T17:26:51.363 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f22eaad3:::benchmark_data_smithi179_77138_object77844:head# 2024-02-09T17:26:51.363 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2301c3f:::benchmark_data_smithi179_77138_object74134:head# 2024-02-09T17:26:51.364 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f230694a:::benchmark_data_smithi179_77138_object72800:head# 2024-02-09T17:26:51.364 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f231e252:::benchmark_data_smithi179_77138_object42258:head# 2024-02-09T17:26:51.365 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2325e1f:::benchmark_data_smithi179_77138_object97126:head# 2024-02-09T17:26:51.365 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f23280cf:::benchmark_data_smithi179_77138_object29655:head# 2024-02-09T17:26:51.366 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f23359dd:::benchmark_data_smithi179_77138_object42035:head# 2024-02-09T17:26:51.366 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f233dfec:::benchmark_data_smithi179_77138_object108912:head# 2024-02-09T17:26:51.367 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2342aea:::benchmark_data_smithi179_77138_object111088:head# 2024-02-09T17:26:51.371 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f23476df:::benchmark_data_smithi179_77138_object116305:head# 2024-02-09T17:26:51.371 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2348a3f:::benchmark_data_smithi179_77138_object36993:head# 2024-02-09T17:26:51.371 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f234a835:::benchmark_data_smithi179_77138_object124111:head# 2024-02-09T17:26:51.371 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2350450:::benchmark_data_smithi179_77138_object63242:head# 2024-02-09T17:26:51.371 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2351bf9:::benchmark_data_smithi179_77138_object29325:head# 2024-02-09T17:26:51.371 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2354e61:::benchmark_data_smithi179_77138_object125204:head# 2024-02-09T17:26:51.371 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2384a57:::benchmark_data_smithi179_77138_object62573:head# 2024-02-09T17:26:51.371 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f238d7a8:::benchmark_data_smithi179_77138_object100921:head# 2024-02-09T17:26:51.372 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2396057:::benchmark_data_smithi179_77138_object88144:head# 2024-02-09T17:26:51.372 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f239752c:::benchmark_data_smithi179_77138_object125682:head# 2024-02-09T17:26:51.372 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f23978c4:::benchmark_data_smithi179_77138_object97854:head# 2024-02-09T17:26:51.372 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f239a9a8:::benchmark_data_smithi179_77138_object84191:head# 2024-02-09T17:26:51.373 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f23a62c4:::benchmark_data_smithi179_77138_object28271:head# 2024-02-09T17:26:51.373 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f23ba2db:::benchmark_data_smithi179_77138_object129935:head# 2024-02-09T17:26:51.374 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f23c31f8:::benchmark_data_smithi179_77138_object112957:head# 2024-02-09T17:26:51.374 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f23c336d:::benchmark_data_smithi179_77138_object107329:head# 2024-02-09T17:26:51.378 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f23ce421:::benchmark_data_smithi179_77138_object50861:head# 2024-02-09T17:26:51.378 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f23d4e69:::benchmark_data_smithi179_77138_object67637:head# 2024-02-09T17:26:51.378 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f23d6793:::benchmark_data_smithi179_77138_object120561:head# 2024-02-09T17:26:51.378 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f23e940e:::benchmark_data_smithi179_77138_object26729:head# 2024-02-09T17:26:51.378 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f23f2689:::benchmark_data_smithi179_77138_object19683:head# 2024-02-09T17:26:51.378 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f23f36eb:::benchmark_data_smithi179_77138_object95122:head# 2024-02-09T17:26:51.381 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f23fcda7:::benchmark_data_smithi179_77138_object39617:head# 2024-02-09T17:26:51.382 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f24010eb:::benchmark_data_smithi179_77138_object71668:head# 2024-02-09T17:26:51.387 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2402cad:::benchmark_data_smithi179_77138_object108082:head# 2024-02-09T17:26:51.387 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2403890:::benchmark_data_smithi179_77138_object56028:head# 2024-02-09T17:26:51.387 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2403b1b:::benchmark_data_smithi179_77138_object126154:head# 2024-02-09T17:26:51.388 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2405aa4:::benchmark_data_smithi179_77138_object26188:head# 2024-02-09T17:26:51.388 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f241ba96:::benchmark_data_smithi179_77138_object109957:head# 2024-02-09T17:26:51.388 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f242cb95:::benchmark_data_smithi179_77138_object47836:head# 2024-02-09T17:26:51.388 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2438995:::benchmark_data_smithi179_77138_object106822:head# 2024-02-09T17:26:51.388 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f244476f:::benchmark_data_smithi179_77138_object122265:head# 2024-02-09T17:26:51.388 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f244d817:::benchmark_data_smithi179_77138_object43052:head# 2024-02-09T17:26:51.389 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f24597cc:::benchmark_data_smithi179_77138_object30393:head# 2024-02-09T17:26:51.389 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f245caba:::benchmark_data_smithi179_77138_object111420:head# 2024-02-09T17:26:51.391 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f246190c:::benchmark_data_smithi179_77138_object72195:head# 2024-02-09T17:26:51.391 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2465709:::benchmark_data_smithi179_77138_object32678:head# 2024-02-09T17:26:51.391 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f247b326:::benchmark_data_smithi179_77138_object110204:head# 2024-02-09T17:26:51.391 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f247bfb3:::benchmark_data_smithi179_77138_object88685:head# 2024-02-09T17:26:51.391 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f247fbec:::benchmark_data_smithi179_77138_object66357:head# 2024-02-09T17:26:51.391 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2486e33:::benchmark_data_smithi179_77138_object33875:head# 2024-02-09T17:26:51.391 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f248d43a:::benchmark_data_smithi179_77138_object24943:head# 2024-02-09T17:26:51.391 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f249169d:::benchmark_data_smithi179_77138_object57656:head# 2024-02-09T17:26:51.391 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f24922a8:::benchmark_data_smithi179_77138_object60402:head# 2024-02-09T17:26:51.392 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f249c499:::benchmark_data_smithi179_77138_object122532:head# 2024-02-09T17:26:51.394 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f24c80b0:::benchmark_data_smithi179_77138_object54848:head# 2024-02-09T17:26:51.394 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f24d752a:::benchmark_data_smithi179_77138_object48447:head# 2024-02-09T17:26:51.394 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f24df8ba:::benchmark_data_smithi179_77138_object26201:head# 2024-02-09T17:26:51.395 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f24e194f:::benchmark_data_smithi179_77138_object93784:head# 2024-02-09T17:26:51.395 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f24e2d31:::benchmark_data_smithi179_77138_object82105:head# 2024-02-09T17:26:51.395 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f24e490e:::benchmark_data_smithi179_77138_object120518:head# 2024-02-09T17:26:51.395 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f24f3e14:::benchmark_data_smithi179_77138_object76958:head# 2024-02-09T17:26:51.395 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f24fadcc:::benchmark_data_smithi179_77138_object73958:head# 2024-02-09T17:26:51.395 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f24fda9b:::benchmark_data_smithi179_77138_object37496:head# 2024-02-09T17:26:51.396 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f24fdc5c:::benchmark_data_smithi179_77138_object100426:head# 2024-02-09T17:26:51.396 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f25082e4:::benchmark_data_smithi179_77138_object129468:head# 2024-02-09T17:26:51.397 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f250a2e5:::benchmark_data_smithi179_77138_object103497:head# 2024-02-09T17:26:51.397 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f251df92:::benchmark_data_smithi179_77138_object68585:head# 2024-02-09T17:26:51.397 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2524024:::benchmark_data_smithi179_77138_object39707:head# 2024-02-09T17:26:51.398 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f252e716:::benchmark_data_smithi179_77138_object111917:head# 2024-02-09T17:26:51.398 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f25561cb:::benchmark_data_smithi179_77138_object65911:head# 2024-02-09T17:26:51.399 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2556c0d:::benchmark_data_smithi179_77138_object37122:head# 2024-02-09T17:26:51.399 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f25604fb:::benchmark_data_smithi179_77138_object104966:head# 2024-02-09T17:26:51.400 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2570139:::benchmark_data_smithi179_77138_object28547:head# 2024-02-09T17:26:51.400 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f257da7f:::benchmark_data_smithi179_77138_object39612:head# 2024-02-09T17:26:51.400 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:26:51.401 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f259b56f:::benchmark_data_smithi179_77138_object82676:head# 2024-02-09T17:26:51.401 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f25a39af:::benchmark_data_smithi179_77138_object112181:head# 2024-02-09T17:26:51.403 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f25ae92b:::benchmark_data_smithi179_77138_object56607:head# 2024-02-09T17:26:51.403 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f25b18f4:::benchmark_data_smithi179_77138_object77932:head# 2024-02-09T17:26:51.403 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f25e52ca:::benchmark_data_smithi179_77138_object118101:head# 2024-02-09T17:26:51.403 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f25e541f:::benchmark_data_smithi179_77138_object121480:head# 2024-02-09T17:26:51.405 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f25f859c:::benchmark_data_smithi179_77138_object24042:head# 2024-02-09T17:26:51.405 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f25fecff:::benchmark_data_smithi179_77138_object76977:head# 2024-02-09T17:26:51.405 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f260474b:::benchmark_data_smithi179_77138_object59438:head# 2024-02-09T17:26:51.405 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2605f31:::benchmark_data_smithi179_77138_object41957:head# 2024-02-09T17:26:51.405 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2625b29:::benchmark_data_smithi179_77138_object88075:head# 2024-02-09T17:26:51.406 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f262e4fd:::benchmark_data_smithi179_77138_object100970:head# 2024-02-09T17:26:51.406 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f262ed37:::benchmark_data_smithi179_77138_object127372:head# 2024-02-09T17:26:51.407 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2630be6:::benchmark_data_smithi179_77138_object87251:head# 2024-02-09T17:26:51.407 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2647b76:::benchmark_data_smithi179_77138_object36070:head# 2024-02-09T17:26:51.407 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f26482af:::benchmark_data_smithi179_77138_object101000:head# 2024-02-09T17:26:51.408 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f264c651:::benchmark_data_smithi179_77138_object103197:head# 2024-02-09T17:26:51.408 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f265426f:::benchmark_data_smithi179_77138_object55434:head# 2024-02-09T17:26:51.409 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2654e22:::benchmark_data_smithi179_77138_object62411:head# 2024-02-09T17:26:51.409 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2660c6a:::benchmark_data_smithi179_77138_object71785:head# 2024-02-09T17:26:51.410 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f266cc24:::benchmark_data_smithi179_77138_object63001:head# 2024-02-09T17:26:51.410 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2676100:::benchmark_data_smithi179_77138_object57842:head# 2024-02-09T17:26:51.411 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2682c52:::benchmark_data_smithi179_77138_object32973:head# 2024-02-09T17:26:51.411 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f268480c:::benchmark_data_smithi179_77138_object102768:head# 2024-02-09T17:26:51.412 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f268d75b:::benchmark_data_smithi179_77138_object129326:head# 2024-02-09T17:26:51.412 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f268fa00:::benchmark_data_smithi179_77138_object59552:head# 2024-02-09T17:26:51.413 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f269c1c3:::benchmark_data_smithi179_77138_object127244:head# 2024-02-09T17:26:51.413 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f26a6547:::benchmark_data_smithi179_77138_object84598:head# 2024-02-09T17:26:51.414 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f26a840e:::benchmark_data_smithi179_77138_object80404:head# 2024-02-09T17:26:51.414 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f26c7e88:::benchmark_data_smithi179_77138_object25741:head# 2024-02-09T17:26:51.414 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f26cd11b:::benchmark_data_smithi179_77138_object126053:head# 2024-02-09T17:26:51.415 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f26d4de9:::benchmark_data_smithi179_77138_object46244:head# 2024-02-09T17:26:51.416 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f26e5091:::benchmark_data_smithi179_77138_object30714:head# 2024-02-09T17:26:51.416 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f26feadb:::benchmark_data_smithi179_77138_object113584:head# 2024-02-09T17:26:51.418 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2700c05:::benchmark_data_smithi179_77138_object118775:head# 2024-02-09T17:26:51.418 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f270b8a7:::benchmark_data_smithi179_77138_object46151:head# 2024-02-09T17:26:51.418 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f271b930:::benchmark_data_smithi179_77138_object32193:head# 2024-02-09T17:26:51.418 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2729c02:::benchmark_data_smithi179_77138_object72867:head# 2024-02-09T17:26:51.419 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f272afba:::benchmark_data_smithi179_77138_object59205:head# 2024-02-09T17:26:51.419 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f27321c6:::benchmark_data_smithi179_77138_object98944:head# 2024-02-09T17:26:51.419 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f273822d:::benchmark_data_smithi179_77138_object77109:head# 2024-02-09T17:26:51.420 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f273cd93:::benchmark_data_smithi179_77138_object36723:head# 2024-02-09T17:26:51.420 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2745dfc:::benchmark_data_smithi179_77138_object130651:head# 2024-02-09T17:26:51.421 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f275dade:::benchmark_data_smithi179_77138_object69943:head# 2024-02-09T17:26:51.421 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2775bca:::benchmark_data_smithi179_77138_object121968:head# 2024-02-09T17:26:51.422 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2777b3b:::benchmark_data_smithi179_77138_object68808:head# 2024-02-09T17:26:51.422 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2779694:::benchmark_data_smithi179_77138_object83093:head# 2024-02-09T17:26:51.422 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f277fa07:::benchmark_data_smithi179_77138_object100992:head# 2024-02-09T17:26:51.423 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2788c86:::benchmark_data_smithi179_77138_object35413:head# 2024-02-09T17:26:51.424 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f278b1eb:::benchmark_data_smithi179_77138_object56461:head# 2024-02-09T17:26:51.424 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2796c55:::benchmark_data_smithi179_77138_object107702:head# 2024-02-09T17:26:51.425 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2797e9e:::benchmark_data_smithi179_77138_object86791:head# 2024-02-09T17:26:51.425 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f27991a4:::benchmark_data_smithi179_77138_object75757:head# 2024-02-09T17:26:51.425 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f279a2c9:::benchmark_data_smithi179_77138_object22551:head# 2024-02-09T17:26:51.426 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f27a1d9b:::benchmark_data_smithi179_77138_object54236:head# 2024-02-09T17:26:51.426 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f27ab2a6:::benchmark_data_smithi179_77138_object111176:head# 2024-02-09T17:26:51.427 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f27bce93:::benchmark_data_smithi179_77138_object97576:head# 2024-02-09T17:26:51.427 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f27c09e0:::benchmark_data_smithi179_77138_object109633:head# 2024-02-09T17:26:51.428 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f27c1efa:::benchmark_data_smithi179_77138_object118453:head# 2024-02-09T17:26:51.428 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f27dd971:::benchmark_data_smithi179_77138_object98105:head# 2024-02-09T17:26:51.429 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f27e4067:::benchmark_data_smithi179_77138_object22360:head# 2024-02-09T17:26:51.429 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f27f0043:::benchmark_data_smithi179_77138_object80769:head# 2024-02-09T17:26:51.430 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f27f2721:::benchmark_data_smithi179_77138_object63827:head# 2024-02-09T17:26:51.430 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f280d277:::benchmark_data_smithi179_77138_object88453:head# 2024-02-09T17:26:51.431 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2815231:::benchmark_data_smithi179_77138_object41854:head# 2024-02-09T17:26:51.431 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f281ce87:::benchmark_data_smithi179_77138_object48350:head# 2024-02-09T17:26:51.431 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2829254:::benchmark_data_smithi179_77138_object23923:head# 2024-02-09T17:26:51.433 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2835600:::benchmark_data_smithi179_77138_object56192:head# 2024-02-09T17:26:51.433 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f28367bd:::benchmark_data_smithi179_77138_object56580:head# 2024-02-09T17:26:51.433 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2868f0b:::benchmark_data_smithi179_77138_object85848:head# 2024-02-09T17:26:51.434 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f28817c9:::benchmark_data_smithi179_77138_object106780:head# 2024-02-09T17:26:51.435 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2886b84:::benchmark_data_smithi179_77138_object36873:head# 2024-02-09T17:26:51.435 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f288ab5e:::benchmark_data_smithi179_77138_object80282:head# 2024-02-09T17:26:51.436 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2894ab5:::benchmark_data_smithi179_77138_object19724:head# 2024-02-09T17:26:51.436 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f289d0c6:::benchmark_data_smithi179_77138_object115086:head# 2024-02-09T17:26:51.437 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f289e97e:::benchmark_data_smithi179_77138_object118595:head# 2024-02-09T17:26:51.437 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f28a6ba0:::benchmark_data_smithi179_77138_object67220:head# 2024-02-09T17:26:51.438 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f28b58ae:::benchmark_data_smithi179_77138_object105506:head# 2024-02-09T17:26:51.441 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f28d0522:::benchmark_data_smithi179_77138_object130189:head# 2024-02-09T17:26:51.441 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f28e206e:::benchmark_data_smithi179_77138_object108868:head# 2024-02-09T17:26:51.442 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f28e88f9:::benchmark_data_smithi179_77138_object127289:head# 2024-02-09T17:26:51.442 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f28ef563:::benchmark_data_smithi179_77138_object92389:head# 2024-02-09T17:26:51.442 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f28f6885:::benchmark_data_smithi179_77138_object88526:head# 2024-02-09T17:26:51.442 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f290f6a0:::benchmark_data_smithi179_77138_object21754:head# 2024-02-09T17:26:51.442 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f291800a:::benchmark_data_smithi179_77138_object88500:head# 2024-02-09T17:26:51.443 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:51.443 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2947e91:::benchmark_data_smithi179_77138_object108026:head# 2024-02-09T17:26:51.444 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2969f4f:::benchmark_data_smithi179_77138_object52298:head# 2024-02-09T17:26:51.445 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f29821bd:::benchmark_data_smithi179_77138_object92405:head# 2024-02-09T17:26:51.445 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f298b847:::benchmark_data_smithi179_77138_object59645:head# 2024-02-09T17:26:51.446 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f298e80f:::benchmark_data_smithi179_77138_object35917:head# 2024-02-09T17:26:51.446 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2991935:::benchmark_data_smithi179_77138_object105268:head# 2024-02-09T17:26:51.447 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f29961f7:::benchmark_data_smithi179_77138_object68281:head# 2024-02-09T17:26:51.447 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f299ad71:::benchmark_data_smithi179_77138_object125815:head# 2024-02-09T17:26:51.447 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f299d9ce:::benchmark_data_smithi179_77138_object86089:head# 2024-02-09T17:26:51.448 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f29a9d1a:::benchmark_data_smithi179_77138_object47028:head# 2024-02-09T17:26:51.448 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f29c693b:::benchmark_data_smithi179_77138_object68089:head# 2024-02-09T17:26:51.449 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f29ea7ca:::benchmark_data_smithi179_77138_object125800:head# 2024-02-09T17:26:51.449 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2a06128:::benchmark_data_smithi179_77138_object102009:head# 2024-02-09T17:26:51.449 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2a0bdd3:::benchmark_data_smithi179_77138_object46566:head# 2024-02-09T17:26:51.450 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2a160d2:::benchmark_data_smithi179_77138_object88612:head# 2024-02-09T17:26:51.450 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2a1c1fb:::benchmark_data_smithi179_77138_object21363:head# 2024-02-09T17:26:51.451 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2a2fb92:::benchmark_data_smithi179_77138_object25183:head# 2024-02-09T17:26:51.451 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2a38220:::benchmark_data_smithi179_77138_object95502:head# 2024-02-09T17:26:51.459 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2a40f3a:::benchmark_data_smithi179_77138_object34352:head# 2024-02-09T17:26:51.459 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2a465b9:::benchmark_data_smithi179_77138_object111934:head# 2024-02-09T17:26:51.459 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2a6eabe:::benchmark_data_smithi179_77138_object57382:head# 2024-02-09T17:26:51.460 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2a755ce:::benchmark_data_smithi179_77138_object29158:head# 2024-02-09T17:26:51.460 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2a76161:::benchmark_data_smithi179_77138_object133244:head# 2024-02-09T17:26:51.460 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2a762bd:::benchmark_data_smithi179_77138_object99163:head# 2024-02-09T17:26:51.460 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2a7817d:::benchmark_data_smithi179_77138_object84588:head# 2024-02-09T17:26:51.460 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2a8d068:::benchmark_data_smithi179_77138_object121526:head# 2024-02-09T17:26:51.460 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2aa2c9c:::benchmark_data_smithi179_77138_object129392:head# 2024-02-09T17:26:51.460 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2aa38e7:::benchmark_data_smithi179_77138_object77047:head# 2024-02-09T17:26:51.460 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2aa4b07:::benchmark_data_smithi179_77138_object26810:head# 2024-02-09T17:26:51.460 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2ab8375:::benchmark_data_smithi179_77138_object104074:head# 2024-02-09T17:26:51.460 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2abebae:::benchmark_data_smithi179_77138_object96372:head# 2024-02-09T17:26:51.460 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2aca0e7:::benchmark_data_smithi179_77138_object103679:head# 2024-02-09T17:26:51.460 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2acc2e2:::benchmark_data_smithi179_77138_object101786:head# 2024-02-09T17:26:51.460 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2aceb8a:::benchmark_data_smithi179_77138_object106816:head# 2024-02-09T17:26:51.461 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2ad646a:::benchmark_data_smithi179_77138_object60342:head# 2024-02-09T17:26:51.461 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2ae4151:::benchmark_data_smithi179_77138_object116537:head# 2024-02-09T17:26:51.461 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2aedb28:::benchmark_data_smithi179_77138_object57642:head# 2024-02-09T17:26:51.461 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2aede1a:::benchmark_data_smithi179_77138_object71482:head# 2024-02-09T17:26:51.462 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2af2ba2:::benchmark_data_smithi179_77138_object84036:head# 2024-02-09T17:26:51.462 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2b0c790:::benchmark_data_smithi179_77138_object121844:head# 2024-02-09T17:26:51.462 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2b17b7c:::benchmark_data_smithi179_77138_object79916:head# 2024-02-09T17:26:51.463 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2b18b10:::benchmark_data_smithi179_77138_object101829:head# 2024-02-09T17:26:51.464 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2b2a6b1:::benchmark_data_smithi179_77138_object39899:head# 2024-02-09T17:26:51.464 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2b2c192:::benchmark_data_smithi179_77138_object80358:head# 2024-02-09T17:26:51.465 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2b3a15b:::benchmark_data_smithi179_77138_object26413:head# 2024-02-09T17:26:51.466 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2b3b997:::benchmark_data_smithi179_77138_object119567:head# 2024-02-09T17:26:51.466 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2b411c5:::benchmark_data_smithi179_77138_object39906:head# 2024-02-09T17:26:51.466 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2b42853:::benchmark_data_smithi179_77138_object89119:head# 2024-02-09T17:26:51.467 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2b43063:::benchmark_data_smithi179_77138_object22690:head# 2024-02-09T17:26:51.467 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2b44947:::benchmark_data_smithi179_77138_object95882:head# 2024-02-09T17:26:51.468 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2b4e77c:::benchmark_data_smithi179_77138_object22659:head# 2024-02-09T17:26:51.469 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2b585f9:::benchmark_data_smithi179_77138_object19925:head# 2024-02-09T17:26:51.469 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2b6bf09:::benchmark_data_smithi179_77138_object95438:head# 2024-02-09T17:26:51.469 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2b7e6af:::benchmark_data_smithi179_77138_object104443:head# 2024-02-09T17:26:51.470 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2b82e82:::benchmark_data_smithi179_77138_object20242:head# 2024-02-09T17:26:51.470 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2b9338f:::benchmark_data_smithi179_77138_object70936:head# 2024-02-09T17:26:51.470 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2b950b4:::benchmark_data_smithi179_77138_object97767:head# 2024-02-09T17:26:51.471 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2b9efe9:::benchmark_data_smithi179_77138_object67476:head# 2024-02-09T17:26:51.471 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2ba3859:::benchmark_data_smithi179_77138_object99607:head# 2024-02-09T17:26:51.472 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2baf0b0:::benchmark_data_smithi179_77138_object44973:head# 2024-02-09T17:26:51.472 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2bc9dfd:::benchmark_data_smithi179_77138_object92700:head# 2024-02-09T17:26:51.473 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2bdce48:::benchmark_data_smithi179_77138_object35661:head# 2024-02-09T17:26:51.473 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2be8cd0:::benchmark_data_smithi179_77138_object65426:head# 2024-02-09T17:26:51.473 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2bedc60:::benchmark_data_smithi179_77138_object38061:head# 2024-02-09T17:26:51.474 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2bf0593:::benchmark_data_smithi179_77138_object101863:head# 2024-02-09T17:26:51.475 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2bf44ac:::benchmark_data_smithi179_77138_object87209:head# 2024-02-09T17:26:51.475 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2c0b648:::benchmark_data_smithi179_77138_object81064:head# 2024-02-09T17:26:51.475 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2c23513:::benchmark_data_smithi179_77138_object20683:head# 2024-02-09T17:26:51.476 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2c2db95:::benchmark_data_smithi179_77138_object33762:head# 2024-02-09T17:26:51.476 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2c3d331:::benchmark_data_smithi179_77138_object27802:head# 2024-02-09T17:26:51.477 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2c44916:::benchmark_data_smithi179_77138_object113815:head# 2024-02-09T17:26:51.477 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2c5cfbe:::benchmark_data_smithi179_77138_object103199:head# 2024-02-09T17:26:51.478 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2c6962a:::benchmark_data_smithi179_77138_object111983:head# 2024-02-09T17:26:51.478 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2c6e7be:::benchmark_data_smithi179_77138_object85502:head# 2024-02-09T17:26:51.478 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2c8eb8f:::benchmark_data_smithi179_77138_object103795:head# 2024-02-09T17:26:51.479 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2c920df:::benchmark_data_smithi179_77138_object88725:head# 2024-02-09T17:26:51.479 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2c95836:::benchmark_data_smithi179_77138_object51830:head# 2024-02-09T17:26:51.480 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2cabb6a:::benchmark_data_smithi179_77138_object34242:head# 2024-02-09T17:26:51.480 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2cd3ca1:::benchmark_data_smithi179_77138_object101208:head# 2024-02-09T17:26:51.481 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2cdb678:::benchmark_data_smithi179_77138_object57154:head# 2024-02-09T17:26:51.481 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2cdbe16:::benchmark_data_smithi179_77138_object28480:head# 2024-02-09T17:26:51.482 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2cedf6b:::benchmark_data_smithi179_77138_object46753:head# 2024-02-09T17:26:51.482 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2d1f50b:::benchmark_data_smithi179_77138_object110313:head# 2024-02-09T17:26:51.482 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2d1f718:::benchmark_data_smithi179_77138_object111939:head# 2024-02-09T17:26:51.483 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2d21596:::benchmark_data_smithi179_77138_object98828:head# 2024-02-09T17:26:51.486 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2d24221:::benchmark_data_smithi179_77138_object80893:head# 2024-02-09T17:26:51.486 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2d2fbd6:::benchmark_data_smithi179_77138_object28121:head# 2024-02-09T17:26:51.486 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2d38fde:::benchmark_data_smithi179_77138_object42739:head# 2024-02-09T17:26:51.487 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2d48927:::benchmark_data_smithi179_77138_object106308:head# 2024-02-09T17:26:51.487 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2d55a6a:::benchmark_data_smithi179_77138_object116783:head# 2024-02-09T17:26:51.487 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2d5880f:::benchmark_data_smithi179_77138_object45689:head# 2024-02-09T17:26:51.487 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2d640d9:::benchmark_data_smithi179_77138_object39034:head# 2024-02-09T17:26:51.490 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2d67bf0:::benchmark_data_smithi179_77138_object70077:head# 2024-02-09T17:26:51.490 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2d6deaf:::benchmark_data_smithi179_77138_object87321:head# 2024-02-09T17:26:51.491 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2d71468:::benchmark_data_smithi179_77138_object37739:head# 2024-02-09T17:26:51.491 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2d759bc:::benchmark_data_smithi179_77138_object47307:head# 2024-02-09T17:26:51.491 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2d75a47:::benchmark_data_smithi179_77138_object98978:head# 2024-02-09T17:26:51.491 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2d80c5c:::benchmark_data_smithi179_77138_object58351:head# 2024-02-09T17:26:51.491 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2d817ce:::benchmark_data_smithi179_77138_object25137:head# 2024-02-09T17:26:51.492 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2d82b33:::benchmark_data_smithi179_77138_object81670:head# 2024-02-09T17:26:51.492 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2d8398c:::benchmark_data_smithi179_77138_object22723:head# 2024-02-09T17:26:51.492 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2d847e8:::benchmark_data_smithi179_77138_object87218:head# 2024-02-09T17:26:51.492 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2d8ac07:::benchmark_data_smithi179_77138_object24702:head# 2024-02-09T17:26:51.493 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2da6b73:::benchmark_data_smithi179_77138_object110980:head# 2024-02-09T17:26:51.493 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2dabbc2:::benchmark_data_smithi179_77138_object46149:head# 2024-02-09T17:26:51.493 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2db8d51:::benchmark_data_smithi179_77138_object117541:head# 2024-02-09T17:26:51.494 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2dd533b:::benchmark_data_smithi179_77138_object75409:head# 2024-02-09T17:26:51.494 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2defb03:::benchmark_data_smithi179_77138_object112632:head# 2024-02-09T17:26:51.495 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2df1afc:::benchmark_data_smithi179_77138_object80651:head# 2024-02-09T17:26:51.495 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2dfa403:::benchmark_data_smithi179_77138_object104528:head# 2024-02-09T17:26:51.496 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2dfbe96:::benchmark_data_smithi179_77138_object129231:head# 2024-02-09T17:26:51.498 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2e00c75:::benchmark_data_smithi179_77138_object82574:head# 2024-02-09T17:26:51.498 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2e0575f:::benchmark_data_smithi179_77138_object29286:head# 2024-02-09T17:26:51.499 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2e23624:::benchmark_data_smithi179_77138_object76868:head# 2024-02-09T17:26:51.499 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2e3c45f:::benchmark_data_smithi179_77138_object99416:head# 2024-02-09T17:26:51.500 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2e47202:::benchmark_data_smithi179_77138_object86802:head# 2024-02-09T17:26:51.500 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2e56900:::benchmark_data_smithi179_77138_object81863:head# 2024-02-09T17:26:51.501 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2e56de8:::benchmark_data_smithi179_77138_object43740:head# 2024-02-09T17:26:51.501 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2e592c3:::benchmark_data_smithi179_77138_object120796:head# 2024-02-09T17:26:51.501 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2e5d27b:::benchmark_data_smithi179_77138_object27369:head# 2024-02-09T17:26:51.502 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2e69032:::benchmark_data_smithi179_77138_object68078:head# 2024-02-09T17:26:51.503 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2e7afc9:::benchmark_data_smithi179_77138_object29245:head# 2024-02-09T17:26:51.503 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2e7db89:::benchmark_data_smithi179_77138_object67591:head# 2024-02-09T17:26:51.503 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2e7df35:::benchmark_data_smithi179_77138_object58618:head# 2024-02-09T17:26:51.504 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2e7f4cd:::benchmark_data_smithi179_77138_object121846:head# 2024-02-09T17:26:51.504 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2e7f4e1:::benchmark_data_smithi179_77138_object74798:head# 2024-02-09T17:26:51.505 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2e82c20:::benchmark_data_smithi179_77138_object130537:head# 2024-02-09T17:26:51.505 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2ea36a6:::benchmark_data_smithi179_77138_object56044:head# 2024-02-09T17:26:51.506 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2ea6ca6:::benchmark_data_smithi179_77138_object36370:head# 2024-02-09T17:26:51.506 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2eb06d9:::benchmark_data_smithi179_77138_object128044:head# 2024-02-09T17:26:51.506 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2ec394f:::benchmark_data_smithi179_77138_object25907:head# 2024-02-09T17:26:51.506 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2ec57a6:::benchmark_data_smithi179_77138_object70115:head# 2024-02-09T17:26:51.507 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2ec62b5:::benchmark_data_smithi179_77138_object41461:head# 2024-02-09T17:26:51.507 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2ecb184:::benchmark_data_smithi179_77138_object48938:head# 2024-02-09T17:26:51.508 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2eebafd:::benchmark_data_smithi179_77138_object77631:head# 2024-02-09T17:26:51.508 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2ef02f6:::benchmark_data_smithi179_77138_object131787:head# 2024-02-09T17:26:51.509 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2ef071e:::benchmark_data_smithi179_77138_object61298:head# 2024-02-09T17:26:51.509 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2efa1b3:::benchmark_data_smithi179_77138_object39407:head# 2024-02-09T17:26:51.510 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2efbc27:::benchmark_data_smithi179_77138_object88930:head# 2024-02-09T17:26:51.510 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2f03f83:::benchmark_data_smithi179_77138_object124749:head# 2024-02-09T17:26:51.511 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2f0ab47:::benchmark_data_smithi179_77138_object111905:head# 2024-02-09T17:26:51.511 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2f12d1c:::benchmark_data_smithi179_77138_object94573:head# 2024-02-09T17:26:51.511 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2f15ea7:::benchmark_data_smithi179_77138_object98026:head# 2024-02-09T17:26:51.512 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2f18d8c:::benchmark_data_smithi179_77138_object78233:head# 2024-02-09T17:26:51.512 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2f1f104:::benchmark_data_smithi179_77138_object94752:head# 2024-02-09T17:26:51.513 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2f208d0:::benchmark_data_smithi179_77138_object92326:head# 2024-02-09T17:26:51.513 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2f2dfab:::benchmark_data_smithi179_77138_object132448:head# 2024-02-09T17:26:51.514 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2f31255:::benchmark_data_smithi179_77138_object40623:head# 2024-02-09T17:26:51.514 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2f356c6:::benchmark_data_smithi179_77138_object48445:head# 2024-02-09T17:26:51.514 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2f35e6c:::benchmark_data_smithi179_77138_object101482:head# 2024-02-09T17:26:51.515 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2f3b7fe:::benchmark_data_smithi179_77138_object39632:head# 2024-02-09T17:26:51.515 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2f3d8bf:::benchmark_data_smithi179_77138_object78739:head# 2024-02-09T17:26:51.516 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2f3dbe7:::benchmark_data_smithi179_77138_object79848:head# 2024-02-09T17:26:51.516 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2f49560:::benchmark_data_smithi179_77138_object34202:head# 2024-02-09T17:26:51.517 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2f4b7f5:::benchmark_data_smithi179_77138_object105064:head# 2024-02-09T17:26:51.517 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2f5691e:::benchmark_data_smithi179_77138_object62500:head# 2024-02-09T17:26:51.517 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2f5757b:::benchmark_data_smithi179_77138_object105742:head# 2024-02-09T17:26:51.518 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2f771af:::benchmark_data_smithi179_77138_object132420:head# 2024-02-09T17:26:51.518 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2f7f40f:::benchmark_data_smithi179_77138_object51216:head# 2024-02-09T17:26:51.519 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2f83531:::benchmark_data_smithi179_77138_object90138:head# 2024-02-09T17:26:51.519 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2f850a6:::benchmark_data_smithi179_77138_object25699:head# 2024-02-09T17:26:51.523 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2f85ec3:::benchmark_data_smithi179_77138_object33924:head# 2024-02-09T17:26:51.523 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2f8ce76:::benchmark_data_smithi179_77138_object65362:head# 2024-02-09T17:26:51.523 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2f8d7f5:::benchmark_data_smithi179_77138_object30289:head# 2024-02-09T17:26:51.523 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2f8df81:::benchmark_data_smithi179_77138_object70401:head# 2024-02-09T17:26:51.523 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2f8f2e3:::benchmark_data_smithi179_77138_object42659:head# 2024-02-09T17:26:51.523 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2f9407f:::benchmark_data_smithi179_77138_object76607:head# 2024-02-09T17:26:51.524 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2faebee:::benchmark_data_smithi179_77138_object123424:head# 2024-02-09T17:26:51.524 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2fbff8b:::benchmark_data_smithi179_77138_object32713:head# 2024-02-09T17:26:51.524 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2fc398a:::benchmark_data_smithi179_77138_object82945:head# 2024-02-09T17:26:51.524 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2fc7cab:::benchmark_data_smithi179_77138_object131445:head# 2024-02-09T17:26:51.527 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2fccee7:::benchmark_data_smithi179_77138_object65366:head# 2024-02-09T17:26:51.527 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2fd0244:::benchmark_data_smithi179_77138_object49942:head# 2024-02-09T17:26:51.527 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2fd29f6:::benchmark_data_smithi179_77138_object58011:head# 2024-02-09T17:26:51.528 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2fd3661:::benchmark_data_smithi179_77138_object104556:head# 2024-02-09T17:26:51.528 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2fd545f:::benchmark_data_smithi179_77138_object25498:head# 2024-02-09T17:26:51.529 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2fe61d4:::benchmark_data_smithi179_77138_object54989:head# 2024-02-09T17:26:51.529 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2fe6636:::benchmark_data_smithi179_77138_object54712:head# 2024-02-09T17:26:51.530 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2fe6cd3:::benchmark_data_smithi179_77138_object22820:head# 2024-02-09T17:26:51.530 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2fe710b:::benchmark_data_smithi179_77138_object121652:head# 2024-02-09T17:26:51.531 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2fea768:::benchmark_data_smithi179_77138_object28106:head# 2024-02-09T17:26:51.531 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f2ffd8b5:::benchmark_data_smithi179_77138_object54657:head# 2024-02-09T17:26:51.531 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f300f8df:::benchmark_data_smithi179_77138_object105903:head# 2024-02-09T17:26:51.532 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f301041b:::benchmark_data_smithi179_77138_object122629:head# 2024-02-09T17:26:51.532 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f30307c7:::benchmark_data_smithi179_77138_object54145:head# 2024-02-09T17:26:51.533 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f303fae7:::benchmark_data_smithi179_77138_object45016:head# 2024-02-09T17:26:51.533 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f303fb31:::benchmark_data_smithi179_77138_object109702:head# 2024-02-09T17:26:51.534 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f30420d9:::benchmark_data_smithi179_77138_object42042:head# 2024-02-09T17:26:51.534 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f30460dc:::benchmark_data_smithi179_77138_object58106:head# 2024-02-09T17:26:51.534 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f304f635:::benchmark_data_smithi179_77138_object36278:head# 2024-02-09T17:26:51.535 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f30547d1:::benchmark_data_smithi179_77138_object91645:head# 2024-02-09T17:26:51.535 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f305a03c:::benchmark_data_smithi179_77138_object42986:head# 2024-02-09T17:26:51.536 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f305a492:::benchmark_data_smithi179_77138_object105431:head# 2024-02-09T17:26:51.536 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3066850:::benchmark_data_smithi179_77138_object118470:head# 2024-02-09T17:26:51.537 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f306c5f1:::benchmark_data_smithi179_77138_object97999:head# 2024-02-09T17:26:51.537 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3082904:::benchmark_data_smithi179_77138_object71733:head# 2024-02-09T17:26:51.537 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f308975b:::benchmark_data_smithi179_77138_object125942:head# 2024-02-09T17:26:51.538 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3089780:::benchmark_data_smithi179_77138_object39189:head# 2024-02-09T17:26:51.538 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f308b7f2:::benchmark_data_smithi179_77138_object85643:head# 2024-02-09T17:26:51.539 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f308d8b1:::benchmark_data_smithi179_77138_object20363:head# 2024-02-09T17:26:51.539 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3092e1f:::benchmark_data_smithi179_77138_object44651:head# 2024-02-09T17:26:51.540 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f30afeab:::benchmark_data_smithi179_77138_object56802:head# 2024-02-09T17:26:51.540 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f30b0aed:::benchmark_data_smithi179_77138_object96163:head# 2024-02-09T17:26:51.541 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f30b6f49:::benchmark_data_smithi179_77138_object120120:head# 2024-02-09T17:26:51.541 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f30bd1be:::benchmark_data_smithi179_77138_object41680:head# 2024-02-09T17:26:51.542 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f30d830c:::benchmark_data_smithi179_77138_object74849:head# 2024-02-09T17:26:51.542 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f30d9425:::benchmark_data_smithi179_77138_object52992:head# 2024-02-09T17:26:51.542 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f30e5ad3:::benchmark_data_smithi179_77138_object107105:head# 2024-02-09T17:26:51.544 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f30ee2d9:::benchmark_data_smithi179_77138_object126596:head# 2024-02-09T17:26:51.547 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f31000e4:::benchmark_data_smithi179_77138_object39447:head# 2024-02-09T17:26:51.547 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3105235:::benchmark_data_smithi179_77138_object131068:head# 2024-02-09T17:26:51.547 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f310a8e1:::benchmark_data_smithi179_77138_object64244:head# 2024-02-09T17:26:51.548 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f310ac8c:::benchmark_data_smithi179_77138_object74862:head# 2024-02-09T17:26:51.548 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f310b47d:::benchmark_data_smithi179_77138_object116072:head# 2024-02-09T17:26:51.549 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f311e803:::benchmark_data_smithi179_77138_object101876:head# 2024-02-09T17:26:51.549 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f312bd71:::benchmark_data_smithi179_77138_object55916:head# 2024-02-09T17:26:51.550 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3130c8f:::benchmark_data_smithi179_77138_object57109:head# 2024-02-09T17:26:51.550 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3135e0d:::benchmark_data_smithi179_77138_object114672:head# 2024-02-09T17:26:51.550 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3136325:::benchmark_data_smithi179_77138_object50953:head# 2024-02-09T17:26:51.551 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f313eb27:::benchmark_data_smithi179_77138_object54512:head# 2024-02-09T17:26:51.552 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3146ee0:::benchmark_data_smithi179_77138_object89692:head# 2024-02-09T17:26:51.552 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3156ef7:::benchmark_data_smithi179_77138_object108184:head# 2024-02-09T17:26:51.553 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f315b07a:::benchmark_data_smithi179_77138_object86235:head# 2024-02-09T17:26:51.553 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f31632de:::benchmark_data_smithi179_77138_object42115:head# 2024-02-09T17:26:51.554 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3168476:::benchmark_data_smithi179_77138_object86375:head# 2024-02-09T17:26:51.554 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f316c331:::benchmark_data_smithi179_77138_object37700:head# 2024-02-09T17:26:51.557 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f31709f4:::benchmark_data_smithi179_77138_object30595:head# 2024-02-09T17:26:51.557 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3172b2a:::benchmark_data_smithi179_77138_object77524:head# 2024-02-09T17:26:51.558 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3178446:::benchmark_data_smithi179_77138_object114822:head# 2024-02-09T17:26:51.558 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3179bf4:::benchmark_data_smithi179_77138_object34310:head# 2024-02-09T17:26:51.559 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f318d48e:::benchmark_data_smithi179_77138_object87556:head# 2024-02-09T17:26:51.559 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f318e74a:::benchmark_data_smithi179_77138_object83841:head# 2024-02-09T17:26:51.560 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f318ec76:::benchmark_data_smithi179_77138_object32560:head# 2024-02-09T17:26:51.560 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3195ef3:::benchmark_data_smithi179_77138_object92905:head# 2024-02-09T17:26:51.562 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f31a0e37:::benchmark_data_smithi179_77138_object83618:head# 2024-02-09T17:26:51.562 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f31becc1:::benchmark_data_smithi179_77138_object132083:head# 2024-02-09T17:26:51.563 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f31c36b1:::benchmark_data_smithi179_77138_object41358:head# 2024-02-09T17:26:51.563 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f31c886b:::benchmark_data_smithi179_77138_object23221:head# 2024-02-09T17:26:51.563 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f31cd040:::benchmark_data_smithi179_77138_object114116:head# 2024-02-09T17:26:51.564 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f31cf826:::benchmark_data_smithi179_77138_object108819:head# 2024-02-09T17:26:51.564 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f31defcf:::benchmark_data_smithi179_77138_object49723:head# 2024-02-09T17:26:51.565 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f31df0ae:::benchmark_data_smithi179_77138_object41073:head# 2024-02-09T17:26:51.565 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f31e13f1:::benchmark_data_smithi179_77138_object111532:head# 2024-02-09T17:26:51.566 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f31e8d0c:::benchmark_data_smithi179_77138_object88773:head# 2024-02-09T17:26:51.566 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f31efac6:::benchmark_data_smithi179_77138_object126861:head# 2024-02-09T17:26:51.567 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f31fb995:::benchmark_data_smithi179_77138_object22746:head# 2024-02-09T17:26:51.568 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f32118ed:::benchmark_data_smithi179_77138_object73285:head# 2024-02-09T17:26:51.568 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3213e39:::benchmark_data_smithi179_77138_object64202:head# 2024-02-09T17:26:51.568 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3219cea:::benchmark_data_smithi179_77138_object105731:head# 2024-02-09T17:26:51.570 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3226e01:::benchmark_data_smithi179_77138_object117485:head# 2024-02-09T17:26:51.570 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3229f71:::benchmark_data_smithi179_77138_object100760:head# 2024-02-09T17:26:51.570 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f322c0be:::benchmark_data_smithi179_77138_object85790:head# 2024-02-09T17:26:51.570 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3232f21:::benchmark_data_smithi179_77138_object96322:head# 2024-02-09T17:26:51.570 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f323fa5d:::benchmark_data_smithi179_77138_object34567:head# 2024-02-09T17:26:51.571 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3246308:::benchmark_data_smithi179_77138_object102496:head# 2024-02-09T17:26:51.571 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3247fdd:::benchmark_data_smithi179_77138_object95560:head# 2024-02-09T17:26:51.572 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3264e67:::benchmark_data_smithi179_77138_object127732:head# 2024-02-09T17:26:51.572 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3268b5d:::benchmark_data_smithi179_77138_object21428:head# 2024-02-09T17:26:51.572 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f326bf2b:::benchmark_data_smithi179_77138_object58129:head# 2024-02-09T17:26:51.573 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f327354d:::benchmark_data_smithi179_77138_object108870:head# 2024-02-09T17:26:51.573 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f327f5a2:::benchmark_data_smithi179_77138_object115348:head# 2024-02-09T17:26:51.574 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f32874d1:::benchmark_data_smithi179_77138_object26842:head# 2024-02-09T17:26:51.574 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3298468:::benchmark_data_smithi179_77138_object43245:head# 2024-02-09T17:26:51.574 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f32b97fa:::benchmark_data_smithi179_77138_object29842:head# 2024-02-09T17:26:51.575 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f32c360b:::benchmark_data_smithi179_77138_object110928:head# 2024-02-09T17:26:51.575 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f32c6b6d:::benchmark_data_smithi179_77138_object73815:head# 2024-02-09T17:26:51.576 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f32cd48a:::benchmark_data_smithi179_77138_object37184:head# 2024-02-09T17:26:51.576 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f32dc7b6:::benchmark_data_smithi179_77138_object19461:head# 2024-02-09T17:26:51.576 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f32e3dfe:::benchmark_data_smithi179_77138_object123735:head# 2024-02-09T17:26:51.577 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f32ec74b:::benchmark_data_smithi179_77138_object47817:head# 2024-02-09T17:26:51.577 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f32fdb34:::benchmark_data_smithi179_77138_object27403:head# 2024-02-09T17:26:51.578 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f32ff56e:::benchmark_data_smithi179_77138_object127036:head# 2024-02-09T17:26:51.578 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3304b9d:::benchmark_data_smithi179_77138_object82537:head# 2024-02-09T17:26:51.579 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3306348:::benchmark_data_smithi179_77138_object59212:head# 2024-02-09T17:26:51.579 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f331baba:::benchmark_data_smithi179_77138_object116585:head# 2024-02-09T17:26:51.580 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f331bb53:::benchmark_data_smithi179_77138_object129602:head# 2024-02-09T17:26:51.580 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f332053e:::benchmark_data_smithi179_77138_object106726:head# 2024-02-09T17:26:51.581 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3321236:::benchmark_data_smithi179_77138_object80258:head# 2024-02-09T17:26:51.581 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f33212a6:::benchmark_data_smithi179_77138_object48475:head# 2024-02-09T17:26:51.582 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f33251a4:::benchmark_data_smithi179_77138_object58934:head# 2024-02-09T17:26:51.582 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3326774:::benchmark_data_smithi179_77138_object99400:head# 2024-02-09T17:26:51.582 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3326d2d:::benchmark_data_smithi179_77138_object108629:head# 2024-02-09T17:26:51.587 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f332b62c:::benchmark_data_smithi179_77138_object105771:head# 2024-02-09T17:26:51.587 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3331fd0:::benchmark_data_smithi179_77138_object72921:head# 2024-02-09T17:26:51.587 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f333810f:::benchmark_data_smithi179_77138_object49147:head# 2024-02-09T17:26:51.588 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f334155e:::benchmark_data_smithi179_77138_object88997:head# 2024-02-09T17:26:51.588 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f334156b:::benchmark_data_smithi179_77138_object58035:head# 2024-02-09T17:26:51.588 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f33658dd:::benchmark_data_smithi179_77138_object126746:head# 2024-02-09T17:26:51.588 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f336b4cc:::benchmark_data_smithi179_77138_object101246:head# 2024-02-09T17:26:51.588 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f336f4ce:::benchmark_data_smithi179_77138_object77173:head# 2024-02-09T17:26:51.588 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f337169f:::benchmark_data_smithi179_77138_object131010:head# 2024-02-09T17:26:51.588 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3377cc3:::benchmark_data_smithi179_77138_object39218:head# 2024-02-09T17:26:51.588 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3379c6f:::benchmark_data_smithi179_77138_object49835:head# 2024-02-09T17:26:51.589 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3385ae1:::benchmark_data_smithi179_77138_object102203:head# 2024-02-09T17:26:51.589 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f33863f7:::benchmark_data_smithi179_77138_object73072:head# 2024-02-09T17:26:51.589 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3386a55:::benchmark_data_smithi179_77138_object56705:head# 2024-02-09T17:26:51.594 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3388ba3:::benchmark_data_smithi179_77138_object95379:head# 2024-02-09T17:26:51.594 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3396654:::benchmark_data_smithi179_77138_object105498:head# 2024-02-09T17:26:51.594 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f339ba62:::benchmark_data_smithi179_77138_object66100:head# 2024-02-09T17:26:51.594 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f33a0bf0:::benchmark_data_smithi179_77138_object75349:head# 2024-02-09T17:26:51.594 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f33a8dfe:::benchmark_data_smithi179_77138_object41335:head# 2024-02-09T17:26:51.594 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f33bafb3:::benchmark_data_smithi179_77138_object43171:head# 2024-02-09T17:26:51.595 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f33c38c3:::benchmark_data_smithi179_77138_object46498:head# 2024-02-09T17:26:51.595 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f33c5f5e:::benchmark_data_smithi179_77138_object45535:head# 2024-02-09T17:26:51.595 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f33c62ed:::benchmark_data_smithi179_77138_object39822:head# 2024-02-09T17:26:51.595 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f33ce50e:::benchmark_data_smithi179_77138_object25039:head# 2024-02-09T17:26:51.596 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f33dcbb9:::benchmark_data_smithi179_77138_object26756:head# 2024-02-09T17:26:51.596 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f33f3c04:::benchmark_data_smithi179_77138_object70747:head# 2024-02-09T17:26:51.597 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f33f4e31:::benchmark_data_smithi179_77138_object57995:head# 2024-02-09T17:26:51.602 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3400cbf:::benchmark_data_smithi179_77138_object106495:head# 2024-02-09T17:26:51.602 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3414891:::benchmark_data_smithi179_77138_object116143:head# 2024-02-09T17:26:51.602 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3416a6d:::benchmark_data_smithi179_77138_object106400:head# 2024-02-09T17:26:51.603 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f341c7b2:::benchmark_data_smithi179_77138_object72145:head# 2024-02-09T17:26:51.603 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f34218c7:::benchmark_data_smithi179_77138_object93379:head# 2024-02-09T17:26:51.603 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3423439:::benchmark_data_smithi179_77138_object22129:head# 2024-02-09T17:26:51.604 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3424d6e:::benchmark_data_smithi179_77138_object72947:head# 2024-02-09T17:26:51.604 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f342c6a3:::benchmark_data_smithi179_77138_object101814:head# 2024-02-09T17:26:51.605 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f342c6cc:::benchmark_data_smithi179_77138_object73702:head# 2024-02-09T17:26:51.605 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f343090d:::benchmark_data_smithi179_77138_object107211:head# 2024-02-09T17:26:51.606 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3436dec:::benchmark_data_smithi179_77138_object131876:head# 2024-02-09T17:26:51.607 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f343b686:::benchmark_data_smithi179_77138_object109509:head# 2024-02-09T17:26:51.607 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f344100c:::benchmark_data_smithi179_77138_object91929:head# 2024-02-09T17:26:51.607 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f34488d9:::benchmark_data_smithi179_77138_object60057:head# 2024-02-09T17:26:51.608 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3454f27:::benchmark_data_smithi179_77138_object22022:head# 2024-02-09T17:26:51.608 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3466866:::benchmark_data_smithi179_77138_object43189:head# 2024-02-09T17:26:51.608 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3469493:::benchmark_data_smithi179_77138_object116845:head# 2024-02-09T17:26:51.609 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3469b0c:::benchmark_data_smithi179_77138_object86439:head# 2024-02-09T17:26:51.609 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f34711b9:::benchmark_data_smithi179_77138_object29476:head# 2024-02-09T17:26:51.610 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3474e83:::benchmark_data_smithi179_77138_object58460:head# 2024-02-09T17:26:51.610 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f348a1fc:::benchmark_data_smithi179_77138_object110635:head# 2024-02-09T17:26:51.610 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f348aec2:::benchmark_data_smithi179_77138_object102329:head# 2024-02-09T17:26:51.611 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f349171d:::benchmark_data_smithi179_77138_object127796:head# 2024-02-09T17:26:51.611 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f349639b:::benchmark_data_smithi179_77138_object82291:head# 2024-02-09T17:26:51.612 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f34986dd:::benchmark_data_smithi179_77138_object100635:head# 2024-02-09T17:26:51.612 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f34a734d:::benchmark_data_smithi179_77138_object59494:head# 2024-02-09T17:26:51.613 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f34a88ef:::benchmark_data_smithi179_77138_object119802:head# 2024-02-09T17:26:51.613 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f34a9462:::benchmark_data_smithi179_77138_object114111:head# 2024-02-09T17:26:51.613 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f34bab0d:::benchmark_data_smithi179_77138_object95880:head# 2024-02-09T17:26:51.614 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f34bd9d8:::benchmark_data_smithi179_77138_object109352:head# 2024-02-09T17:26:51.614 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f34c4357:::benchmark_data_smithi179_77138_object64414:head# 2024-02-09T17:26:51.615 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f34c9abf:::benchmark_data_smithi179_77138_object40643:head# 2024-02-09T17:26:51.615 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f34ca501:::benchmark_data_smithi179_77138_object50691:head# 2024-02-09T17:26:51.616 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f34cf3d4:::benchmark_data_smithi179_77138_object22616:head# 2024-02-09T17:26:51.616 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f34dd992:::benchmark_data_smithi179_77138_object92686:head# 2024-02-09T17:26:51.617 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f34e8484:::benchmark_data_smithi179_77138_object51573:head# 2024-02-09T17:26:51.618 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f34f4a57:::benchmark_data_smithi179_77138_object45179:head# 2024-02-09T17:26:51.618 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f34fd41c:::benchmark_data_smithi179_77138_object25489:head# 2024-02-09T17:26:51.619 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3505a03:::benchmark_data_smithi179_77138_object120360:head# 2024-02-09T17:26:51.619 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3513ce2:::benchmark_data_smithi179_77138_object121776:head# 2024-02-09T17:26:51.619 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3515002:::benchmark_data_smithi179_77138_object69461:head# 2024-02-09T17:26:51.620 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3517297:::benchmark_data_smithi179_77138_object126187:head# 2024-02-09T17:26:51.626 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f35192cf:::benchmark_data_smithi179_77138_object72793:head# 2024-02-09T17:26:51.626 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3521115:::benchmark_data_smithi179_77138_object85060:head# 2024-02-09T17:26:51.626 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3523d22:::benchmark_data_smithi179_77138_object44841:head# 2024-02-09T17:26:51.626 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3527c71:::benchmark_data_smithi179_77138_object59820:head# 2024-02-09T17:26:51.626 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f352bdd4:::benchmark_data_smithi179_77138_object31771:head# 2024-02-09T17:26:51.626 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f354622c:::benchmark_data_smithi179_77138_object127988:head# 2024-02-09T17:26:51.626 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f354956b:::benchmark_data_smithi179_77138_object22260:head# 2024-02-09T17:26:51.626 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3551b53:::benchmark_data_smithi179_77138_object120444:head# 2024-02-09T17:26:51.626 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f355a41c:::benchmark_data_smithi179_77138_object22613:head# 2024-02-09T17:26:51.626 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f355e0f0:::benchmark_data_smithi179_77138_object126618:head# 2024-02-09T17:26:51.626 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f355f439:::benchmark_data_smithi179_77138_object51668:head# 2024-02-09T17:26:51.626 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3562dc9:::benchmark_data_smithi179_77138_object42083:head# 2024-02-09T17:26:51.627 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3563209:::benchmark_data_smithi179_77138_object47510:head# 2024-02-09T17:26:51.627 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f356824a:::benchmark_data_smithi179_77138_object130263:head# 2024-02-09T17:26:51.627 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f35701cf:::benchmark_data_smithi179_77138_object104282:head# 2024-02-09T17:26:51.628 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3570db2:::benchmark_data_smithi179_77138_object83066:head# 2024-02-09T17:26:51.628 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f357858e:::benchmark_data_smithi179_77138_object63440:head# 2024-02-09T17:26:51.633 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3583b23:::benchmark_data_smithi179_77138_object53491:head# 2024-02-09T17:26:51.633 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3596074:::benchmark_data_smithi179_77138_object50805:head# 2024-02-09T17:26:51.633 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f359d7da:::benchmark_data_smithi179_77138_object105432:head# 2024-02-09T17:26:51.633 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f35b767a:::benchmark_data_smithi179_77138_object110820:head# 2024-02-09T17:26:51.633 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f35b7861:::benchmark_data_smithi179_77138_object126426:head# 2024-02-09T17:26:51.633 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f35d30b5:::benchmark_data_smithi179_77138_object113452:head# 2024-02-09T17:26:51.633 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f35d4311:::benchmark_data_smithi179_77138_object113153:head# 2024-02-09T17:26:51.633 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f35d4f3a:::benchmark_data_smithi179_77138_object31623:head# 2024-02-09T17:26:51.634 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f35e0e72:::benchmark_data_smithi179_77138_object128290:head# 2024-02-09T17:26:51.634 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f35ebc25:::benchmark_data_smithi179_77138_object60531:head# 2024-02-09T17:26:51.634 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f35f1526:::benchmark_data_smithi179_77138_object60215:head# 2024-02-09T17:26:51.634 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f35f69a7:::benchmark_data_smithi179_77138_object101290:head# 2024-02-09T17:26:51.635 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f35f7594:::benchmark_data_smithi179_77138_object59927:head# 2024-02-09T17:26:51.638 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3603e39:::benchmark_data_smithi179_77138_object82961:head# 2024-02-09T17:26:51.638 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3619847:::benchmark_data_smithi179_77138_object106935:head# 2024-02-09T17:26:51.638 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3625d53:::benchmark_data_smithi179_77138_object74210:head# 2024-02-09T17:26:51.639 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3628c17:::benchmark_data_smithi179_77138_object124041:head# 2024-02-09T17:26:51.639 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f363ee1a:::benchmark_data_smithi179_77138_object101752:head# 2024-02-09T17:26:51.639 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f365a335:::benchmark_data_smithi179_77138_object68365:head# 2024-02-09T17:26:51.639 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3661351:::benchmark_data_smithi179_77138_object115833:head# 2024-02-09T17:26:51.639 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3666e01:::benchmark_data_smithi179_77138_object83371:head# 2024-02-09T17:26:51.639 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f367122a:::benchmark_data_smithi179_77138_object118036:head# 2024-02-09T17:26:51.640 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f36769fd:::benchmark_data_smithi179_77138_object58519:head# 2024-02-09T17:26:51.640 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f367f6ca:::benchmark_data_smithi179_77138_object69375:head# 2024-02-09T17:26:51.641 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3685b11:::benchmark_data_smithi179_77138_object105665:head# 2024-02-09T17:26:51.641 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f368d943:::benchmark_data_smithi179_77138_object29698:head# 2024-02-09T17:26:51.642 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f368fdd7:::benchmark_data_smithi179_77138_object60804:head# 2024-02-09T17:26:51.642 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f369693a:::benchmark_data_smithi179_77138_object30948:head# 2024-02-09T17:26:51.642 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f36ad7a6:::benchmark_data_smithi179_77138_object25980:head# 2024-02-09T17:26:51.643 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f36b6ddf:::benchmark_data_smithi179_77138_object90411:head# 2024-02-09T17:26:51.643 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f36bf510:::benchmark_data_smithi179_77138_object98041:head# 2024-02-09T17:26:51.643 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f36c5d13:::benchmark_data_smithi179_77138_object112703:head# 2024-02-09T17:26:51.644 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f36dacfa:::benchmark_data_smithi179_77138_object62245:head# 2024-02-09T17:26:51.644 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f36de74b:::benchmark_data_smithi179_77138_object56010:head# 2024-02-09T17:26:51.646 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f36e032f:::benchmark_data_smithi179_77138_object114025:head# 2024-02-09T17:26:51.646 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f36eaf56:::benchmark_data_smithi179_77138_object105644:head# 2024-02-09T17:26:51.646 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f36ebec6:::benchmark_data_smithi179_77138_object40288:head# 2024-02-09T17:26:51.647 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f36ed44e:::benchmark_data_smithi179_77138_object110080:head# 2024-02-09T17:26:51.647 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f36f3f9c:::benchmark_data_smithi179_77138_object20388:head# 2024-02-09T17:26:51.647 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3710b9d:::benchmark_data_smithi179_77138_object78354:head# 2024-02-09T17:26:51.648 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3716141:::benchmark_data_smithi179_77138_object19499:head# 2024-02-09T17:26:51.648 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f371970d:::benchmark_data_smithi179_77138_object31382:head# 2024-02-09T17:26:51.648 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f371c748:::benchmark_data_smithi179_77138_object110485:head# 2024-02-09T17:26:51.649 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f371d621:::benchmark_data_smithi179_77138_object102910:head# 2024-02-09T17:26:51.649 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f371e563:::benchmark_data_smithi179_77138_object81458:head# 2024-02-09T17:26:51.650 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f372dc9d:::benchmark_data_smithi179_77138_object54481:head# 2024-02-09T17:26:51.650 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f37319de:::benchmark_data_smithi179_77138_object76313:head# 2024-02-09T17:26:51.651 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f373237e:::benchmark_data_smithi179_77138_object56550:head# 2024-02-09T17:26:51.651 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3732fba:::benchmark_data_smithi179_77138_object103708:head# 2024-02-09T17:26:51.651 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f373441a:::benchmark_data_smithi179_77138_object46635:head# 2024-02-09T17:26:51.652 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f374d7e3:::benchmark_data_smithi179_77138_object93019:head# 2024-02-09T17:26:51.652 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f375c169:::benchmark_data_smithi179_77138_object99523:head# 2024-02-09T17:26:51.653 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3763742:::benchmark_data_smithi179_77138_object25195:head# 2024-02-09T17:26:51.653 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f37668a9:::benchmark_data_smithi179_77138_object67133:head# 2024-02-09T17:26:51.654 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f377c78e:::benchmark_data_smithi179_77138_object121203:head# 2024-02-09T17:26:51.654 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3782076:::benchmark_data_smithi179_77138_object126982:head# 2024-02-09T17:26:51.655 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f378a7f8:::benchmark_data_smithi179_77138_object101175:head# 2024-02-09T17:26:51.655 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f378f48e:::benchmark_data_smithi179_77138_object23793:head# 2024-02-09T17:26:51.655 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f37a5eef:::benchmark_data_smithi179_77138_object35703:head# 2024-02-09T17:26:51.656 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f37a6280:::benchmark_data_smithi179_77138_object71730:head# 2024-02-09T17:26:51.656 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f37b91db:::benchmark_data_smithi179_77138_object112938:head# 2024-02-09T17:26:51.657 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f37f07f7:::benchmark_data_smithi179_77138_object110918:head# 2024-02-09T17:26:51.657 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f37fe900:::benchmark_data_smithi179_77138_object35046:head# 2024-02-09T17:26:51.658 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3802b4b:::benchmark_data_smithi179_77138_object116179:head# 2024-02-09T17:26:51.658 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f38072e5:::benchmark_data_smithi179_77138_object107544:head# 2024-02-09T17:26:51.659 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f380c735:::benchmark_data_smithi179_77138_object40124:head# 2024-02-09T17:26:51.659 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f382ce60:::benchmark_data_smithi179_77138_object79915:head# 2024-02-09T17:26:51.660 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f384600b:::benchmark_data_smithi179_77138_object55413:head# 2024-02-09T17:26:51.660 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f384bab3:::benchmark_data_smithi179_77138_object99924:head# 2024-02-09T17:26:51.661 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f384efbf:::benchmark_data_smithi179_77138_object79432:head# 2024-02-09T17:26:51.661 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3857419:::benchmark_data_smithi179_77138_object83723:head# 2024-02-09T17:26:51.661 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3857667:::benchmark_data_smithi179_77138_object109905:head# 2024-02-09T17:26:51.662 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f385d8fc:::benchmark_data_smithi179_77138_object118613:head# 2024-02-09T17:26:51.663 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f38631a4:::benchmark_data_smithi179_77138_object121377:head# 2024-02-09T17:26:51.663 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3866006:::benchmark_data_smithi179_77138_object29645:head# 2024-02-09T17:26:51.664 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f38812da:::benchmark_data_smithi179_77138_object94253:head# 2024-02-09T17:26:51.664 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3887c7b:::benchmark_data_smithi179_77138_object45426:head# 2024-02-09T17:26:51.664 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f388c1d2:::benchmark_data_smithi179_77138_object43091:head# 2024-02-09T17:26:51.669 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f388cbf3:::benchmark_data_smithi179_77138_object91051:head# 2024-02-09T17:26:51.669 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f388e824:::benchmark_data_smithi179_77138_object69739:head# 2024-02-09T17:26:51.669 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f389b6ab:::benchmark_data_smithi179_77138_object116980:head# 2024-02-09T17:26:51.669 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f38a3e6a:::benchmark_data_smithi179_77138_object24657:head# 2024-02-09T17:26:51.669 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f38a6bbc:::benchmark_data_smithi179_77138_object112460:head# 2024-02-09T17:26:51.669 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f38ab554:::benchmark_data_smithi179_77138_object43828:head# 2024-02-09T17:26:51.669 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f38b065a:::benchmark_data_smithi179_77138_object34880:head# 2024-02-09T17:26:51.669 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f38be686:::benchmark_data_smithi179_77138_object103634:head# 2024-02-09T17:26:51.670 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f38c3da4:::benchmark_data_smithi179_77138_object72506:head# 2024-02-09T17:26:51.670 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f38cf2ea:::benchmark_data_smithi179_77138_object94505:head# 2024-02-09T17:26:51.670 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f38d73c5:::benchmark_data_smithi179_77138_object55026:head# 2024-02-09T17:26:51.670 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f38f9211:::benchmark_data_smithi179_77138_object34938:head# 2024-02-09T17:26:51.670 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3901255:::benchmark_data_smithi179_77138_object87136:head# 2024-02-09T17:26:51.671 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f390630b:::benchmark_data_smithi179_77138_object61551:head# 2024-02-09T17:26:51.671 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3924d9b:::benchmark_data_smithi179_77138_object130407:head# 2024-02-09T17:26:51.672 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f39316b2:::benchmark_data_smithi179_77138_object84608:head# 2024-02-09T17:26:51.672 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3938dc7:::benchmark_data_smithi179_77138_object113948:head# 2024-02-09T17:26:51.673 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3945652:::benchmark_data_smithi179_77138_object21043:head# 2024-02-09T17:26:51.673 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f394cae8:::benchmark_data_smithi179_77138_object77319:head# 2024-02-09T17:26:51.674 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f394cc42:::benchmark_data_smithi179_77138_object36772:head# 2024-02-09T17:26:51.674 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3957591:::benchmark_data_smithi179_77138_object107670:head# 2024-02-09T17:26:51.674 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f395e5d4:::benchmark_data_smithi179_77138_object92779:head# 2024-02-09T17:26:51.675 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f39700ae:::benchmark_data_smithi179_77138_object99910:head# 2024-02-09T17:26:51.675 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3982074:::benchmark_data_smithi179_77138_object130278:head# 2024-02-09T17:26:51.676 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f398398e:::benchmark_data_smithi179_77138_object98517:head# 2024-02-09T17:26:51.676 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3993949:::benchmark_data_smithi179_77138_object94958:head# 2024-02-09T17:26:51.676 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f39a6155:::benchmark_data_smithi179_77138_object70022:head# 2024-02-09T17:26:51.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:51 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:26:51.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:51 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:26:51.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:51 smithi162 ceph-mon[44304]: pgmap v451: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 6.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 806 op/s; 26903/322257 objects degraded (8.348%); 0 B/s, 0 objects/s recovering 2024-02-09T17:26:51.677 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f39ac82f:::benchmark_data_smithi179_77138_object47532:head# 2024-02-09T17:26:51.678 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f39b14a9:::benchmark_data_smithi179_77138_object110518:head# 2024-02-09T17:26:51.678 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f39beecb:::benchmark_data_smithi179_77138_object118008:head# 2024-02-09T17:26:51.678 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f39c0e87:::benchmark_data_smithi179_77138_object34080:head# 2024-02-09T17:26:51.678 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f39c5c99:::benchmark_data_smithi179_77138_object70460:head# 2024-02-09T17:26:51.679 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f39ca776:::benchmark_data_smithi179_77138_object31702:head# 2024-02-09T17:26:51.679 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f39d2790:::benchmark_data_smithi179_77138_object123796:head# 2024-02-09T17:26:51.680 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f39df4b6:::benchmark_data_smithi179_77138_object125141:head# 2024-02-09T17:26:51.680 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f39f5afd:::benchmark_data_smithi179_77138_object48279:head# 2024-02-09T17:26:51.681 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f39fb346:::benchmark_data_smithi179_77138_object77032:head# 2024-02-09T17:26:51.684 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3a09a4c:::benchmark_data_smithi179_77138_object39278:head# 2024-02-09T17:26:51.684 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3a0e102:::benchmark_data_smithi179_77138_object78435:head# 2024-02-09T17:26:51.685 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3a3ef09:::benchmark_data_smithi179_77138_object109274:head# 2024-02-09T17:26:51.685 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3a3f583:::benchmark_data_smithi179_77138_object66154:head# 2024-02-09T17:26:51.687 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3a4887e:::benchmark_data_smithi179_77138_object111367:head# 2024-02-09T17:26:51.687 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3a6b87c:::benchmark_data_smithi179_77138_object35409:head# 2024-02-09T17:26:51.688 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3a7461b:::benchmark_data_smithi179_77138_object21237:head# 2024-02-09T17:26:51.688 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3a768c7:::benchmark_data_smithi179_77138_object27090:head# 2024-02-09T17:26:51.688 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3a7e7ee:::benchmark_data_smithi179_77138_object72990:head# 2024-02-09T17:26:51.689 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3a860d2:::benchmark_data_smithi179_77138_object88975:head# 2024-02-09T17:26:51.689 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3a92f43:::benchmark_data_smithi179_77138_object66554:head# 2024-02-09T17:26:51.690 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3a93fa8:::benchmark_data_smithi179_77138_object58649:head# 2024-02-09T17:26:51.692 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3a983b8:::benchmark_data_smithi179_77138_object41609:head# 2024-02-09T17:26:51.692 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3ab0cec:::benchmark_data_smithi179_77138_object113490:head# 2024-02-09T17:26:51.693 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3ab131d:::benchmark_data_smithi179_77138_object127321:head# 2024-02-09T17:26:51.693 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3ab27c4:::benchmark_data_smithi179_77138_object60853:head# 2024-02-09T17:26:51.694 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3ab74b0:::benchmark_data_smithi179_77138_object86309:head# 2024-02-09T17:26:51.694 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3ac2203:::benchmark_data_smithi179_77138_object39974:head# 2024-02-09T17:26:51.695 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3ad2cf4:::benchmark_data_smithi179_77138_object96596:head# 2024-02-09T17:26:51.695 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3adcf6b:::benchmark_data_smithi179_77138_object77945:head# 2024-02-09T17:26:51.695 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3ade0f0:::benchmark_data_smithi179_77138_object82091:head# 2024-02-09T17:26:51.696 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3ae0325:::benchmark_data_smithi179_77138_object79440:head# 2024-02-09T17:26:51.696 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3affab7:::benchmark_data_smithi179_77138_object43467:head# 2024-02-09T17:26:51.697 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3affb50:::benchmark_data_smithi179_77138_object39847:head# 2024-02-09T17:26:51.697 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3b0587d:::benchmark_data_smithi179_77138_object118275:head# 2024-02-09T17:26:51.697 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3b08962:::benchmark_data_smithi179_77138_object60133:head# 2024-02-09T17:26:51.698 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3b2390b:::benchmark_data_smithi179_77138_object85523:head# 2024-02-09T17:26:51.700 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3b24de8:::benchmark_data_smithi179_77138_object49523:head# 2024-02-09T17:26:51.701 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3b292b1:::benchmark_data_smithi179_77138_object96960:head# 2024-02-09T17:26:51.701 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3b31106:::benchmark_data_smithi179_77138_object58102:head# 2024-02-09T17:26:51.701 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3b37e0e:::benchmark_data_smithi179_77138_object132085:head# 2024-02-09T17:26:51.705 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3b3eeeb:::benchmark_data_smithi179_77138_object53066:head# 2024-02-09T17:26:51.707 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3b4783f:::benchmark_data_smithi179_77138_object61436:head# 2024-02-09T17:26:51.708 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3b4ed54:::benchmark_data_smithi179_77138_object119232:head# 2024-02-09T17:26:51.708 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3b521d1:::benchmark_data_smithi179_77138_object83160:head# 2024-02-09T17:26:51.709 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3b5bdac:::benchmark_data_smithi179_77138_object49263:head# 2024-02-09T17:26:51.709 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3b66979:::benchmark_data_smithi179_77138_object43729:head# 2024-02-09T17:26:51.710 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3b673e6:::benchmark_data_smithi179_77138_object81109:head# 2024-02-09T17:26:51.711 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3b6e0dc:::benchmark_data_smithi179_77138_object72759:head# 2024-02-09T17:26:51.711 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3b7accd:::benchmark_data_smithi179_77138_object86738:head# 2024-02-09T17:26:51.711 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3b7b554:::benchmark_data_smithi179_77138_object114994:head# 2024-02-09T17:26:51.714 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3b7d60f:::benchmark_data_smithi179_77138_object69940:head# 2024-02-09T17:26:51.714 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3b9d3dc:::benchmark_data_smithi179_77138_object77762:head# 2024-02-09T17:26:51.715 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3ba2bd8:::benchmark_data_smithi179_77138_object97168:head# 2024-02-09T17:26:51.715 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3ba2d5d:::benchmark_data_smithi179_77138_object84682:head# 2024-02-09T17:26:51.715 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3ba4ef6:::benchmark_data_smithi179_77138_object26116:head# 2024-02-09T17:26:51.716 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3bafce5:::benchmark_data_smithi179_77138_object34963:head# 2024-02-09T17:26:51.716 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3bb8a14:::benchmark_data_smithi179_77138_object102715:head# 2024-02-09T17:26:51.717 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3bc27b7:::benchmark_data_smithi179_77138_object32134:head# 2024-02-09T17:26:51.717 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3bcce2b:::benchmark_data_smithi179_77138_object21756:head# 2024-02-09T17:26:51.717 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3bcdba9:::benchmark_data_smithi179_77138_object122870:head# 2024-02-09T17:26:51.718 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3be66fe:::benchmark_data_smithi179_77138_object36596:head# 2024-02-09T17:26:51.721 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3bed4df:::benchmark_data_smithi179_77138_object29824:head# 2024-02-09T17:26:51.721 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3c0d6e3:::benchmark_data_smithi179_77138_object104524:head# 2024-02-09T17:26:51.721 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3c0df6d:::benchmark_data_smithi179_77138_object70346:head# 2024-02-09T17:26:51.721 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3c115c4:::benchmark_data_smithi179_77138_object93443:head# 2024-02-09T17:26:51.722 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3c5687c:::benchmark_data_smithi179_77138_object90001:head# 2024-02-09T17:26:51.722 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3c644ff:::benchmark_data_smithi179_77138_object36405:head# 2024-02-09T17:26:51.723 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3c673da:::benchmark_data_smithi179_77138_object101665:head# 2024-02-09T17:26:51.723 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3c681d2:::benchmark_data_smithi179_77138_object116687:head# 2024-02-09T17:26:51.723 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3c767ab:::benchmark_data_smithi179_77138_object104582:head# 2024-02-09T17:26:51.724 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3c87c10:::benchmark_data_smithi179_77138_object91508:head# 2024-02-09T17:26:51.724 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3c8995e:::benchmark_data_smithi179_77138_object122157:head# 2024-02-09T17:26:51.724 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3c917bb:::benchmark_data_smithi179_77138_object66582:head# 2024-02-09T17:26:51.725 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3c92743:::benchmark_data_smithi179_77138_object122736:head# 2024-02-09T17:26:51.725 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3c98906:::benchmark_data_smithi179_77138_object70005:head# 2024-02-09T17:26:51.726 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3cb4f55:::benchmark_data_smithi179_77138_object123253:head# 2024-02-09T17:26:51.726 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3cb559c:::benchmark_data_smithi179_77138_object123465:head# 2024-02-09T17:26:51.727 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3cc9180:::benchmark_data_smithi179_77138_object57098:head# 2024-02-09T17:26:51.727 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3cca11d:::benchmark_data_smithi179_77138_object119468:head# 2024-02-09T17:26:51.727 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3ce2e9b:::benchmark_data_smithi179_77138_object53468:head# 2024-02-09T17:26:51.728 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3cebee5:::benchmark_data_smithi179_77138_object81960:head# 2024-02-09T17:26:51.728 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3cf0e6e:::benchmark_data_smithi179_77138_object103201:head# 2024-02-09T17:26:51.729 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3cf91d4:::benchmark_data_smithi179_77138_object40451:head# 2024-02-09T17:26:51.729 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3cfab04:::benchmark_data_smithi179_77138_object46411:head# 2024-02-09T17:26:51.730 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3d0b9a1:::benchmark_data_smithi179_77138_object62353:head# 2024-02-09T17:26:51.730 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3d25fb1:::benchmark_data_smithi179_77138_object42240:head# 2024-02-09T17:26:51.730 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3d280f8:::benchmark_data_smithi179_77138_object48322:head# 2024-02-09T17:26:51.731 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3d29876:::benchmark_data_smithi179_77138_object102268:head# 2024-02-09T17:26:51.731 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3d2c9e5:::benchmark_data_smithi179_77138_object31257:head# 2024-02-09T17:26:51.732 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3d34b11:::benchmark_data_smithi179_77138_object119186:head# 2024-02-09T17:26:51.732 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3d3b032:::benchmark_data_smithi179_77138_object48056:head# 2024-02-09T17:26:51.733 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3d3cf83:::benchmark_data_smithi179_77138_object37576:head# 2024-02-09T17:26:51.733 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3d3f6d0:::benchmark_data_smithi179_77138_object27336:head# 2024-02-09T17:26:51.734 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3d3ff50:::benchmark_data_smithi179_77138_object28367:head# 2024-02-09T17:26:51.734 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3d47621:::benchmark_data_smithi179_77138_object39743:head# 2024-02-09T17:26:51.734 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3d4bae2:::benchmark_data_smithi179_77138_object79531:head# 2024-02-09T17:26:51.735 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3d5cd09:::benchmark_data_smithi179_77138_object51999:head# 2024-02-09T17:26:51.735 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3d6ae20:::benchmark_data_smithi179_77138_object107293:head# 2024-02-09T17:26:51.736 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3d7f8c2:::benchmark_data_smithi179_77138_object112452:head# 2024-02-09T17:26:51.736 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3d8f3a0:::benchmark_data_smithi179_77138_object35885:head# 2024-02-09T17:26:51.737 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3d90557:::benchmark_data_smithi179_77138_object38086:head# 2024-02-09T17:26:51.737 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3d9345c:::benchmark_data_smithi179_77138_object60432:head# 2024-02-09T17:26:51.742 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3d96f3c:::benchmark_data_smithi179_77138_object121620:head# 2024-02-09T17:26:51.742 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3d98682:::benchmark_data_smithi179_77138_object94572:head# 2024-02-09T17:26:51.742 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3da037f:::benchmark_data_smithi179_77138_object36247:head# 2024-02-09T17:26:51.743 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3da0944:::benchmark_data_smithi179_77138_object131269:head# 2024-02-09T17:26:51.743 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3db66a5:::benchmark_data_smithi179_77138_object72021:head# 2024-02-09T17:26:51.743 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3db755c:::benchmark_data_smithi179_77138_object51563:head# 2024-02-09T17:26:51.743 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3db8e3c:::benchmark_data_smithi179_77138_object133156:head# 2024-02-09T17:26:51.749 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3dbc159:::benchmark_data_smithi179_77138_object98406:head# 2024-02-09T17:26:51.750 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3dbf883:::benchmark_data_smithi179_77138_object52048:head# 2024-02-09T17:26:51.750 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3dc54df:::benchmark_data_smithi179_77138_object68997:head# 2024-02-09T17:26:51.751 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3dd0265:::benchmark_data_smithi179_77138_object20652:head# 2024-02-09T17:26:51.751 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3dd92e7:::benchmark_data_smithi179_77138_object126920:head# 2024-02-09T17:26:51.752 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3ddedbf:::benchmark_data_smithi179_77138_object30467:head# 2024-02-09T17:26:51.752 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3dfbf43:::benchmark_data_smithi179_77138_object88143:head# 2024-02-09T17:26:51.755 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3e00489:::benchmark_data_smithi179_77138_object39514:head# 2024-02-09T17:26:51.755 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3e113ff:::benchmark_data_smithi179_77138_object88223:head# 2024-02-09T17:26:51.755 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3e1cba1:::benchmark_data_smithi179_77138_object126626:head# 2024-02-09T17:26:51.756 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3e20475:::benchmark_data_smithi179_77138_object102884:head# 2024-02-09T17:26:51.756 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3e2c152:::benchmark_data_smithi179_77138_object30432:head# 2024-02-09T17:26:51.757 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3e3858e:::benchmark_data_smithi179_77138_object45322:head# 2024-02-09T17:26:51.757 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3e48dd8:::benchmark_data_smithi179_77138_object45459:head# 2024-02-09T17:26:51.758 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3e5bdb3:::benchmark_data_smithi179_77138_object119278:head# 2024-02-09T17:26:51.760 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3e657a3:::benchmark_data_smithi179_77138_object117900:head# 2024-02-09T17:26:51.760 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3e80ae8:::benchmark_data_smithi179_77138_object126760:head# 2024-02-09T17:26:51.760 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3e8922b:::benchmark_data_smithi179_77138_object42148:head# 2024-02-09T17:26:51.760 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3e93570:::benchmark_data_smithi179_77138_object24613:head# 2024-02-09T17:26:51.761 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-09T17:26:51.762 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3e953c2:::benchmark_data_smithi179_77138_object112088:head# 2024-02-09T17:26:51.762 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3eb4b85:::benchmark_data_smithi179_77138_object110710:head# 2024-02-09T17:26:51.762 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3eba75b:::benchmark_data_smithi179_77138_object23309:head# 2024-02-09T17:26:51.762 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3ebefd8:::benchmark_data_smithi179_77138_object131923:head# 2024-02-09T17:26:51.762 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3ec8533:::benchmark_data_smithi179_77138_object115444:head# 2024-02-09T17:26:51.763 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3ed1f80:::benchmark_data_smithi179_77138_object60973:head# 2024-02-09T17:26:51.763 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3ed248b:::benchmark_data_smithi179_77138_object42946:head# 2024-02-09T17:26:51.764 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3ed9344:::benchmark_data_smithi179_77138_object73832:head# 2024-02-09T17:26:51.764 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3edf694:::benchmark_data_smithi179_77138_object118512:head# 2024-02-09T17:26:51.764 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3ee611d:::benchmark_data_smithi179_77138_object63627:head# 2024-02-09T17:26:51.765 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3f0bbcc:::benchmark_data_smithi179_77138_object103824:head# 2024-02-09T17:26:51.765 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3f14dbc:::benchmark_data_smithi179_77138_object57776:head# 2024-02-09T17:26:51.766 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3f15f47:::benchmark_data_smithi179_77138_object33759:head# 2024-02-09T17:26:51.766 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3f1a69a:::benchmark_data_smithi179_77138_object58233:head# 2024-02-09T17:26:51.767 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3f1acea:::benchmark_data_smithi179_77138_object54177:head# 2024-02-09T17:26:51.767 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3f2162a:::benchmark_data_smithi179_77138_object73668:head# 2024-02-09T17:26:51.768 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3f23a48:::benchmark_data_smithi179_77138_object57450:head# 2024-02-09T17:26:51.768 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3f27d32:::benchmark_data_smithi179_77138_object112482:head# 2024-02-09T17:26:51.768 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3f28d11:::benchmark_data_smithi179_77138_object72989:head# 2024-02-09T17:26:51.769 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3f2b1a3:::benchmark_data_smithi179_77138_object49972:head# 2024-02-09T17:26:51.769 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3f30f44:::benchmark_data_smithi179_77138_object72838:head# 2024-02-09T17:26:51.770 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3f337a6:::benchmark_data_smithi179_77138_object41182:head# 2024-02-09T17:26:51.770 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3f3611a:::benchmark_data_smithi179_77138_object46261:head# 2024-02-09T17:26:51.771 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3f5419f:::benchmark_data_smithi179_77138_object115514:head# 2024-02-09T17:26:51.771 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3f5b250:::benchmark_data_smithi179_77138_object48918:head# 2024-02-09T17:26:51.771 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3f5ea27:::benchmark_data_smithi179_77138_object92145:head# 2024-02-09T17:26:51.772 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3f68665:::benchmark_data_smithi179_77138_object59549:head# 2024-02-09T17:26:51.772 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3f697ab:::benchmark_data_smithi179_77138_object107601:head# 2024-02-09T17:26:51.773 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3f6aea5:::benchmark_data_smithi179_77138_object109883:head# 2024-02-09T17:26:51.773 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3f6c9f3:::benchmark_data_smithi179_77138_object105463:head# 2024-02-09T17:26:51.774 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3f773cb:::benchmark_data_smithi179_77138_object94934:head# 2024-02-09T17:26:51.779 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3f7957c:::benchmark_data_smithi179_77138_object28436:head# 2024-02-09T17:26:51.779 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3f8797e:::benchmark_data_smithi179_77138_object50317:head# 2024-02-09T17:26:51.779 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3f8c82e:::benchmark_data_smithi179_77138_object121834:head# 2024-02-09T17:26:51.779 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3f8db21:::benchmark_data_smithi179_77138_object77849:head# 2024-02-09T17:26:51.779 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3f8f52a:::benchmark_data_smithi179_77138_object89557:head# 2024-02-09T17:26:51.779 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3f9b336:::benchmark_data_smithi179_77138_object99533:head# 2024-02-09T17:26:51.779 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3f9b876:::benchmark_data_smithi179_77138_object31724:head# 2024-02-09T17:26:51.779 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3f9e6a7:::benchmark_data_smithi179_77138_object122723:head# 2024-02-09T17:26:51.779 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3fb95ad:::benchmark_data_smithi179_77138_object127903:head# 2024-02-09T17:26:51.779 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3fdcfaf:::benchmark_data_smithi179_77138_object71429:head# 2024-02-09T17:26:51.780 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3fe8167:::benchmark_data_smithi179_77138_object65546:head# 2024-02-09T17:26:51.782 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3ff1182:::benchmark_data_smithi179_77138_object98802:head# 2024-02-09T17:26:51.782 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3ff1d0b:::benchmark_data_smithi179_77138_object19376:head# 2024-02-09T17:26:51.783 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3ff2b73:::benchmark_data_smithi179_77138_object60205:head# 2024-02-09T17:26:51.783 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f3ffb5e9:::benchmark_data_smithi179_77138_object127271:head# 2024-02-09T17:26:51.784 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4015af3:::benchmark_data_smithi179_77138_object128823:head# 2024-02-09T17:26:51.784 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f402abe0:::benchmark_data_smithi179_77138_object77073:head# 2024-02-09T17:26:51.785 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f402e873:::benchmark_data_smithi179_77138_object90607:head# 2024-02-09T17:26:51.785 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4031482:::benchmark_data_smithi179_77138_object40401:head# 2024-02-09T17:26:51.786 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4038bd2:::benchmark_data_smithi179_77138_object23638:head# 2024-02-09T17:26:51.786 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4041da8:::benchmark_data_smithi179_77138_object72111:head# 2024-02-09T17:26:51.786 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4043abd:::benchmark_data_smithi179_77138_object66429:head# 2024-02-09T17:26:51.787 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f404f260:::benchmark_data_smithi179_77138_object40779:head# 2024-02-09T17:26:51.787 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4064b9d:::benchmark_data_smithi179_77138_object78073:head# 2024-02-09T17:26:51.788 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4067565:::benchmark_data_smithi179_77138_object25000:head# 2024-02-09T17:26:51.788 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4067dcd:::benchmark_data_smithi179_77138_object116084:head# 2024-02-09T17:26:51.789 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f40733d6:::benchmark_data_smithi179_77138_object100756:head# 2024-02-09T17:26:51.789 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f408313d:::benchmark_data_smithi179_77138_object126115:head# 2024-02-09T17:26:51.790 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4083bfc:::benchmark_data_smithi179_77138_object110159:head# 2024-02-09T17:26:51.790 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f408a84b:::benchmark_data_smithi179_77138_object71951:head# 2024-02-09T17:26:51.791 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4095bea:::benchmark_data_smithi179_77138_object41538:head# 2024-02-09T17:26:51.791 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f409db89:::benchmark_data_smithi179_77138_object40498:head# 2024-02-09T17:26:51.792 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f40a44c0:::benchmark_data_smithi179_77138_object39552:head# 2024-02-09T17:26:51.796 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:26:51.796 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f40a6dab:::benchmark_data_smithi179_77138_object37741:head# 2024-02-09T17:26:51.796 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f40adc04:::benchmark_data_smithi179_77138_object132233:head# 2024-02-09T17:26:51.796 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f40c6bb1:::benchmark_data_smithi179_77138_object42917:head# 2024-02-09T17:26:51.796 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f40d0797:::benchmark_data_smithi179_77138_object84158:head# 2024-02-09T17:26:51.796 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f40e046d:::benchmark_data_smithi179_77138_object49308:head# 2024-02-09T17:26:51.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:51.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:26:51.799 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f40e88fc:::benchmark_data_smithi179_77138_object29517:head# 2024-02-09T17:26:51.799 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f40f4202:::benchmark_data_smithi179_77138_object25753:head# 2024-02-09T17:26:51.799 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f40fa10a:::benchmark_data_smithi179_77138_object64981:head# 2024-02-09T17:26:51.799 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4103277:::benchmark_data_smithi179_77138_object130947:head# 2024-02-09T17:26:51.799 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f41062fe:::benchmark_data_smithi179_77138_object121263:head# 2024-02-09T17:26:51.800 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f41188bb:::benchmark_data_smithi179_77138_object54861:head# 2024-02-09T17:26:51.800 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f413edad:::benchmark_data_smithi179_77138_object82940:head# 2024-02-09T17:26:51.800 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f414db3d:::benchmark_data_smithi179_77138_object48457:head# 2024-02-09T17:26:51.802 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f415020f:::benchmark_data_smithi179_77138_object115384:head# 2024-02-09T17:26:51.802 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f415eb24:::benchmark_data_smithi179_77138_object90618:head# 2024-02-09T17:26:51.803 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f416a826:::benchmark_data_smithi179_77138_object52728:head# 2024-02-09T17:26:51.803 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4186dee:::benchmark_data_smithi179_77138_object46124:head# 2024-02-09T17:26:51.803 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f41899fd:::benchmark_data_smithi179_77138_object123187:head# 2024-02-09T17:26:51.803 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f418a840:::benchmark_data_smithi179_77138_object112746:head# 2024-02-09T17:26:51.803 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f418b57d:::benchmark_data_smithi179_77138_object59834:head# 2024-02-09T17:26:51.803 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f418be91:::benchmark_data_smithi179_77138_object112220:head# 2024-02-09T17:26:51.803 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f41a24f7:::benchmark_data_smithi179_77138_object48427:head# 2024-02-09T17:26:51.803 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f41a2adb:::benchmark_data_smithi179_77138_object68093:head# 2024-02-09T17:26:51.804 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f41b54e8:::benchmark_data_smithi179_77138_object92590:head# 2024-02-09T17:26:51.804 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f41ca169:::benchmark_data_smithi179_77138_object91938:head# 2024-02-09T17:26:51.805 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f41d3cdd:::benchmark_data_smithi179_77138_object82509:head# 2024-02-09T17:26:51.805 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f41ebf07:::benchmark_data_smithi179_77138_object50384:head# 2024-02-09T17:26:51.806 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f41eefe6:::benchmark_data_smithi179_77138_object49125:head# 2024-02-09T17:26:51.806 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f41efa7c:::benchmark_data_smithi179_77138_object122445:head# 2024-02-09T17:26:51.806 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f41f3a74:::benchmark_data_smithi179_77138_object130152:head# 2024-02-09T17:26:51.807 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f41ff67a:::benchmark_data_smithi179_77138_object63768:head# 2024-02-09T17:26:51.807 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4203af2:::benchmark_data_smithi179_77138_object51253:head# 2024-02-09T17:26:51.808 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4213f91:::benchmark_data_smithi179_77138_object131471:head# 2024-02-09T17:26:51.809 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4218265:::benchmark_data_smithi179_77138_object109146:head# 2024-02-09T17:26:51.809 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4227873:::benchmark_data_smithi179_77138_object56143:head# 2024-02-09T17:26:51.809 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4229b5b:::benchmark_data_smithi179_77138_object84615:head# 2024-02-09T17:26:51.810 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f42324c6:::benchmark_data_smithi179_77138_object68673:head# 2024-02-09T17:26:51.811 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4235b76:::benchmark_data_smithi179_77138_object65748:head# 2024-02-09T17:26:51.811 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f423d949:::benchmark_data_smithi179_77138_object34199:head# 2024-02-09T17:26:51.811 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f424ac89:::benchmark_data_smithi179_77138_object111617:head# 2024-02-09T17:26:51.812 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4250ac9:::benchmark_data_smithi179_77138_object99691:head# 2024-02-09T17:26:51.813 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4260749:::benchmark_data_smithi179_77138_object103592:head# 2024-02-09T17:26:51.813 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4279081:::benchmark_data_smithi179_77138_object83550:head# 2024-02-09T17:26:51.813 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f427b8c7:::benchmark_data_smithi179_77138_object54880:head# 2024-02-09T17:26:51.813 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f427efa4:::benchmark_data_smithi179_77138_object57892:head# 2024-02-09T17:26:51.813 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4284c61:::benchmark_data_smithi179_77138_object128936:head# 2024-02-09T17:26:51.814 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f42937d5:::benchmark_data_smithi179_77138_object28898:head# 2024-02-09T17:26:51.814 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f42ae755:::benchmark_data_smithi179_77138_object71815:head# 2024-02-09T17:26:51.815 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f42bd8a6:::benchmark_data_smithi179_77138_object87359:head# 2024-02-09T17:26:51.815 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f42c8923:::benchmark_data_smithi179_77138_object24700:head# 2024-02-09T17:26:51.816 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f42d6830:::benchmark_data_smithi179_77138_object94109:head# 2024-02-09T17:26:51.819 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f42d7a42:::benchmark_data_smithi179_77138_object126839:head# 2024-02-09T17:26:51.819 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f42e8576:::benchmark_data_smithi179_77138_object21241:head# 2024-02-09T17:26:51.819 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f42ed6ae:::benchmark_data_smithi179_77138_object22963:head# 2024-02-09T17:26:51.819 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f42f7dfe:::benchmark_data_smithi179_77138_object94380:head# 2024-02-09T17:26:51.819 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4305e94:::benchmark_data_smithi179_77138_object30725:head# 2024-02-09T17:26:51.820 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f431103e:::benchmark_data_smithi179_77138_object109042:head# 2024-02-09T17:26:51.820 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4318322:::benchmark_data_smithi179_77138_object103054:head# 2024-02-09T17:26:51.820 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4318c61:::benchmark_data_smithi179_77138_object51036:head# 2024-02-09T17:26:51.821 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f43246d0:::benchmark_data_smithi179_77138_object82446:head# 2024-02-09T17:26:51.821 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f432e6d5:::benchmark_data_smithi179_77138_object54341:head# 2024-02-09T17:26:51.821 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f43370ff:::benchmark_data_smithi179_77138_object92067:head# 2024-02-09T17:26:51.822 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f43395ad:::benchmark_data_smithi179_77138_object107845:head# 2024-02-09T17:26:51.822 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f433c78e:::benchmark_data_smithi179_77138_object95067:head# 2024-02-09T17:26:51.822 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f433e2a2:::benchmark_data_smithi179_77138_object68553:head# 2024-02-09T17:26:51.823 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4347377:::benchmark_data_smithi179_77138_object68948:head# 2024-02-09T17:26:51.823 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4348ae9:::benchmark_data_smithi179_77138_object70103:head# 2024-02-09T17:26:51.824 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4354da3:::benchmark_data_smithi179_77138_object116080:head# 2024-02-09T17:26:51.825 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f435a1a0:::benchmark_data_smithi179_77138_object113466:head# 2024-02-09T17:26:51.825 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f435cbaf:::benchmark_data_smithi179_77138_object117557:head# 2024-02-09T17:26:51.826 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f436ac78:::benchmark_data_smithi179_77138_object115421:head# 2024-02-09T17:26:51.826 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f436f1f0:::benchmark_data_smithi179_77138_object122101:head# 2024-02-09T17:26:51.826 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f437a53d:::benchmark_data_smithi179_77138_object91485:head# 2024-02-09T17:26:51.826 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f43a049f:::benchmark_data_smithi179_77138_object128193:head# 2024-02-09T17:26:51.827 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f43a856a:::benchmark_data_smithi179_77138_object127414:head# 2024-02-09T17:26:51.827 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f43aea67:::benchmark_data_smithi179_77138_object106456:head# 2024-02-09T17:26:51.828 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f43b145c:::benchmark_data_smithi179_77138_object80599:head# 2024-02-09T17:26:51.828 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f43c609b:::benchmark_data_smithi179_77138_object23769:head# 2024-02-09T17:26:51.828 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f43cb164:::benchmark_data_smithi179_77138_object22046:head# 2024-02-09T17:26:51.829 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f43d32ea:::benchmark_data_smithi179_77138_object25011:head# 2024-02-09T17:26:51.829 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f43e1f34:::benchmark_data_smithi179_77138_object37004:head# 2024-02-09T17:26:51.829 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f43f3197:::benchmark_data_smithi179_77138_object131576:head# 2024-02-09T17:26:51.830 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f43f8238:::benchmark_data_smithi179_77138_object35239:head# 2024-02-09T17:26:51.830 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f43fa46c:::benchmark_data_smithi179_77138_object55948:head# 2024-02-09T17:26:51.831 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f43ff04c:::benchmark_data_smithi179_77138_object91264:head# 2024-02-09T17:26:51.831 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f44021d6:::benchmark_data_smithi179_77138_object64220:head# 2024-02-09T17:26:51.832 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4413123:::benchmark_data_smithi179_77138_object44977:head# 2024-02-09T17:26:51.832 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f441a267:::benchmark_data_smithi179_77138_object37759:head# 2024-02-09T17:26:51.833 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f443b70e:::benchmark_data_smithi179_77138_object114647:head# 2024-02-09T17:26:51.833 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f443c2d6:::benchmark_data_smithi179_77138_object81293:head# 2024-02-09T17:26:51.834 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f443ce2d:::benchmark_data_smithi179_77138_object29148:head# 2024-02-09T17:26:51.834 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4444c0f:::benchmark_data_smithi179_77138_object50112:head# 2024-02-09T17:26:51.835 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4464440:::benchmark_data_smithi179_77138_object114767:head# 2024-02-09T17:26:51.835 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4471508:::benchmark_data_smithi179_77138_object107703:head# 2024-02-09T17:26:51.836 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f447869a:::benchmark_data_smithi179_77138_object68776:head# 2024-02-09T17:26:51.836 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4478a8d:::benchmark_data_smithi179_77138_object52203:head# 2024-02-09T17:26:51.836 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4482c82:::benchmark_data_smithi179_77138_object98960:head# 2024-02-09T17:26:51.837 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f44874da:::benchmark_data_smithi179_77138_object101765:head# 2024-02-09T17:26:51.842 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f448c002:::benchmark_data_smithi179_77138_object33401:head# 2024-02-09T17:26:51.842 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f44959ef:::benchmark_data_smithi179_77138_object58607:head# 2024-02-09T17:26:51.842 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f44989f4:::benchmark_data_smithi179_77138_object30266:head# 2024-02-09T17:26:51.842 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f449bbb6:::benchmark_data_smithi179_77138_object117419:head# 2024-02-09T17:26:51.842 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f449c518:::benchmark_data_smithi179_77138_object69073:head# 2024-02-09T17:26:51.842 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f449f267:::benchmark_data_smithi179_77138_object109210:head# 2024-02-09T17:26:51.842 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f44a13c2:::benchmark_data_smithi179_77138_object37526:head# 2024-02-09T17:26:51.842 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f44a6cdb:::benchmark_data_smithi179_77138_object32057:head# 2024-02-09T17:26:51.842 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f44adb54:::benchmark_data_smithi179_77138_object76856:head# 2024-02-09T17:26:51.843 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f44b61cb:::benchmark_data_smithi179_77138_object95099:head# 2024-02-09T17:26:51.843 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f44bfdaa:::benchmark_data_smithi179_77138_object91377:head# 2024-02-09T17:26:51.843 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f44c513b:::benchmark_data_smithi179_77138_object26838:head# 2024-02-09T17:26:51.845 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f44c7fb5:::benchmark_data_smithi179_77138_object124918:head# 2024-02-09T17:26:51.845 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f44c86c2:::benchmark_data_smithi179_77138_object35180:head# 2024-02-09T17:26:51.845 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f44d0def:::benchmark_data_smithi179_77138_object32419:head# 2024-02-09T17:26:51.845 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f44f2261:::benchmark_data_smithi179_77138_object54694:head# 2024-02-09T17:26:51.846 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f45198b2:::benchmark_data_smithi179_77138_object41831:head# 2024-02-09T17:26:51.846 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f451ac4b:::benchmark_data_smithi179_77138_object104843:head# 2024-02-09T17:26:51.846 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4520760:::benchmark_data_smithi179_77138_object76141:head# 2024-02-09T17:26:51.847 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4529580:::benchmark_data_smithi179_77138_object112521:head# 2024-02-09T17:26:51.847 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f452a0b5:::benchmark_data_smithi179_77138_object107530:head# 2024-02-09T17:26:51.848 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f452de77:::benchmark_data_smithi179_77138_object33482:head# 2024-02-09T17:26:51.848 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f45300f4:::benchmark_data_smithi179_77138_object110624:head# 2024-02-09T17:26:51.853 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f45322be:::benchmark_data_smithi179_77138_object58828:head# 2024-02-09T17:26:51.854 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f453b72c:::benchmark_data_smithi179_77138_object75131:head# 2024-02-09T17:26:51.854 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4553b9a:::benchmark_data_smithi179_77138_object70847:head# 2024-02-09T17:26:51.854 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4561742:::benchmark_data_smithi179_77138_object93842:head# 2024-02-09T17:26:51.855 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f456407a:::benchmark_data_smithi179_77138_object32757:head# 2024-02-09T17:26:51.855 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f456fe87:::benchmark_data_smithi179_77138_object90856:head# 2024-02-09T17:26:51.855 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f45761a3:::benchmark_data_smithi179_77138_object120435:head# 2024-02-09T17:26:51.856 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4580cd4:::benchmark_data_smithi179_77138_object78286:head# 2024-02-09T17:26:51.856 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f459d584:::benchmark_data_smithi179_77138_object103167:head# 2024-02-09T17:26:51.857 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f45a5288:::benchmark_data_smithi179_77138_object63459:head# 2024-02-09T17:26:51.857 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f45b2279:::benchmark_data_smithi179_77138_object21474:head# 2024-02-09T17:26:51.857 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f45cabf8:::benchmark_data_smithi179_77138_object85238:head# 2024-02-09T17:26:51.858 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f45cffb8:::benchmark_data_smithi179_77138_object119199:head# 2024-02-09T17:26:51.859 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f45daf5c:::benchmark_data_smithi179_77138_object70068:head# 2024-02-09T17:26:51.859 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f45e3f20:::benchmark_data_smithi179_77138_object66184:head# 2024-02-09T17:26:51.859 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f45e53f2:::benchmark_data_smithi179_77138_object24697:head# 2024-02-09T17:26:51.860 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f45f07ef:::benchmark_data_smithi179_77138_object118026:head# 2024-02-09T17:26:51.860 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f45fec88:::benchmark_data_smithi179_77138_object128682:head# 2024-02-09T17:26:51.860 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f460bfd1:::benchmark_data_smithi179_77138_object61861:head# 2024-02-09T17:26:51.861 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f46144ea:::benchmark_data_smithi179_77138_object51492:head# 2024-02-09T17:26:51.861 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4614990:::benchmark_data_smithi179_77138_object37693:head# 2024-02-09T17:26:51.862 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4624a2c:::benchmark_data_smithi179_77138_object47477:head# 2024-02-09T17:26:51.862 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f46270da:::benchmark_data_smithi179_77138_object91199:head# 2024-02-09T17:26:51.863 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f46277ea:::benchmark_data_smithi179_77138_object40924:head# 2024-02-09T17:26:51.863 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f462806b:::benchmark_data_smithi179_77138_object48720:head# 2024-02-09T17:26:51.863 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4628cce:::benchmark_data_smithi179_77138_object59895:head# 2024-02-09T17:26:51.864 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f46347ca:::benchmark_data_smithi179_77138_object118606:head# 2024-02-09T17:26:51.864 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4635e73:::benchmark_data_smithi179_77138_object45463:head# 2024-02-09T17:26:51.865 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4636f97:::benchmark_data_smithi179_77138_object89960:head# 2024-02-09T17:26:51.865 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4647e71:::benchmark_data_smithi179_77138_object40668:head# 2024-02-09T17:26:51.866 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f464c126:::benchmark_data_smithi179_77138_object101600:head# 2024-02-09T17:26:51.866 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4653772:::benchmark_data_smithi179_77138_object109351:head# 2024-02-09T17:26:51.866 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f46552b6:::benchmark_data_smithi179_77138_object66342:head# 2024-02-09T17:26:51.867 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f466e771:::benchmark_data_smithi179_77138_object43835:head# 2024-02-09T17:26:51.867 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f467e036:::benchmark_data_smithi179_77138_object68338:head# 2024-02-09T17:26:51.868 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4689963:::benchmark_data_smithi179_77138_object128417:head# 2024-02-09T17:26:51.868 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f469729c:::benchmark_data_smithi179_77138_object53197:head# 2024-02-09T17:26:51.869 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f46978cc:::benchmark_data_smithi179_77138_object78299:head# 2024-02-09T17:26:51.869 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f46b7355:::benchmark_data_smithi179_77138_object92243:head# 2024-02-09T17:26:51.869 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f46c4d39:::benchmark_data_smithi179_77138_object63130:head# 2024-02-09T17:26:51.870 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f46d7389:::benchmark_data_smithi179_77138_object20339:head# 2024-02-09T17:26:51.870 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f46d9e3a:::benchmark_data_smithi179_77138_object65723:head# 2024-02-09T17:26:51.871 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f46e2fe8:::benchmark_data_smithi179_77138_object47504:head# 2024-02-09T17:26:51.871 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f46eaf21:::benchmark_data_smithi179_77138_object61086:head# 2024-02-09T17:26:51.872 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f46f0651:::benchmark_data_smithi179_77138_object122614:head# 2024-02-09T17:26:51.873 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f46f24e3:::benchmark_data_smithi179_77138_object81305:head# 2024-02-09T17:26:51.873 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f46fac4e:::benchmark_data_smithi179_77138_object108548:head# 2024-02-09T17:26:51.873 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f47099c6:::benchmark_data_smithi179_77138_object24399:head# 2024-02-09T17:26:51.874 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4712fd4:::benchmark_data_smithi179_77138_object26907:head# 2024-02-09T17:26:51.874 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f47190c7:::benchmark_data_smithi179_77138_object27993:head# 2024-02-09T17:26:51.875 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f471d841:::benchmark_data_smithi179_77138_object131623:head# 2024-02-09T17:26:51.875 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4721be0:::benchmark_data_smithi179_77138_object40168:head# 2024-02-09T17:26:51.875 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f47257b4:::benchmark_data_smithi179_77138_object78554:head# 2024-02-09T17:26:51.876 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4730a67:::benchmark_data_smithi179_77138_object131956:head# 2024-02-09T17:26:51.876 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f473af39:::benchmark_data_smithi179_77138_object35527:head# 2024-02-09T17:26:51.877 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f473bbe2:::benchmark_data_smithi179_77138_object68738:head# 2024-02-09T17:26:51.877 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f474d693:::benchmark_data_smithi179_77138_object25443:head# 2024-02-09T17:26:51.878 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4751726:::benchmark_data_smithi179_77138_object91292:head# 2024-02-09T17:26:51.878 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4757aa1:::benchmark_data_smithi179_77138_object111201:head# 2024-02-09T17:26:51.879 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f475adbc:::benchmark_data_smithi179_77138_object26991:head# 2024-02-09T17:26:51.879 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f475c8ad:::benchmark_data_smithi179_77138_object58428:head# 2024-02-09T17:26:51.881 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f476fd1d:::benchmark_data_smithi179_77138_object102384:head# 2024-02-09T17:26:51.882 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f477382b:::benchmark_data_smithi179_77138_object99975:head# 2024-02-09T17:26:51.882 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f477c9da:::benchmark_data_smithi179_77138_object40634:head# 2024-02-09T17:26:51.882 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f47815ed:::benchmark_data_smithi179_77138_object103109:head# 2024-02-09T17:26:51.883 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4789e16:::benchmark_data_smithi179_77138_object24120:head# 2024-02-09T17:26:51.883 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f479c40f:::benchmark_data_smithi179_77138_object94151:head# 2024-02-09T17:26:51.883 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f47a4680:::benchmark_data_smithi179_77138_object104161:head# 2024-02-09T17:26:51.884 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f47a6eb7:::benchmark_data_smithi179_77138_object82603:head# 2024-02-09T17:26:51.885 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f47b4041:::benchmark_data_smithi179_77138_object83103:head# 2024-02-09T17:26:51.885 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f47d7be3:::benchmark_data_smithi179_77138_object26364:head# 2024-02-09T17:26:51.886 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f47e5583:::benchmark_data_smithi179_77138_object64634:head# 2024-02-09T17:26:51.887 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f48001da:::benchmark_data_smithi179_77138_object127729:head# 2024-02-09T17:26:51.887 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f480282f:::benchmark_data_smithi179_77138_object68236:head# 2024-02-09T17:26:51.887 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4804ab8:::benchmark_data_smithi179_77138_object57065:head# 2024-02-09T17:26:51.888 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f481ffb7:::benchmark_data_smithi179_77138_object121231:head# 2024-02-09T17:26:51.888 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4825a7d:::benchmark_data_smithi179_77138_object48698:head# 2024-02-09T17:26:51.888 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f482c4d7:::benchmark_data_smithi179_77138_object40753:head# 2024-02-09T17:26:51.889 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4846157:::benchmark_data_smithi179_77138_object29887:head# 2024-02-09T17:26:51.890 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f484d366:::benchmark_data_smithi179_77138_object121449:head# 2024-02-09T17:26:51.890 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f485fac4:::benchmark_data_smithi179_77138_object64169:head# 2024-02-09T17:26:51.890 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4860992:::benchmark_data_smithi179_77138_object122023:head# 2024-02-09T17:26:51.891 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f48661a6:::benchmark_data_smithi179_77138_object68150:head# 2024-02-09T17:26:51.891 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f487120b:::benchmark_data_smithi179_77138_object72023:head# 2024-02-09T17:26:51.892 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f48826b0:::benchmark_data_smithi179_77138_object55810:head# 2024-02-09T17:26:51.893 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f48855b5:::benchmark_data_smithi179_77138_object57565:head# 2024-02-09T17:26:51.893 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f48859cf:::benchmark_data_smithi179_77138_object92783:head# 2024-02-09T17:26:51.894 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f48862dc:::benchmark_data_smithi179_77138_object37533:head# 2024-02-09T17:26:51.894 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f489811d:::benchmark_data_smithi179_77138_object51734:head# 2024-02-09T17:26:51.894 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f489994e:::benchmark_data_smithi179_77138_object40742:head# 2024-02-09T17:26:51.894 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f48a0fc2:::benchmark_data_smithi179_77138_object92256:head# 2024-02-09T17:26:51.896 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f48a10cd:::benchmark_data_smithi179_77138_object121862:head# 2024-02-09T17:26:51.896 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f48aac1d:::benchmark_data_smithi179_77138_object76956:head# 2024-02-09T17:26:51.896 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f48c7435:::benchmark_data_smithi179_77138_object80255:head# 2024-02-09T17:26:51.896 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f48ca8b0:::benchmark_data_smithi179_77138_object123932:head# 2024-02-09T17:26:51.899 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:26:51.902 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f48d55a9:::benchmark_data_smithi179_77138_object115332:head# 2024-02-09T17:26:51.902 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f48dad69:::benchmark_data_smithi179_77138_object115842:head# 2024-02-09T17:26:51.902 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f48dc358:::benchmark_data_smithi179_77138_object69671:head# 2024-02-09T17:26:51.903 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f48e6f45:::benchmark_data_smithi179_77138_object107094:head# 2024-02-09T17:26:51.903 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f48f8101:::benchmark_data_smithi179_77138_object85815:head# 2024-02-09T17:26:51.903 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f48fa9b4:::benchmark_data_smithi179_77138_object99191:head# 2024-02-09T17:26:51.904 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f490f5b6:::benchmark_data_smithi179_77138_object97183:head# 2024-02-09T17:26:51.905 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4912c50:::benchmark_data_smithi179_77138_object89065:head# 2024-02-09T17:26:51.906 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f492cd98:::benchmark_data_smithi179_77138_object89746:head# 2024-02-09T17:26:51.906 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4944bc7:::benchmark_data_smithi179_77138_object30918:head# 2024-02-09T17:26:51.907 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f494c94f:::benchmark_data_smithi179_77138_object63570:head# 2024-02-09T17:26:51.907 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f494f71a:::benchmark_data_smithi179_77138_object128145:head# 2024-02-09T17:26:51.907 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4955399:::benchmark_data_smithi179_77138_object78994:head# 2024-02-09T17:26:51.908 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f49570a6:::benchmark_data_smithi179_77138_object132389:head# 2024-02-09T17:26:51.908 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4960d9d:::benchmark_data_smithi179_77138_object84077:head# 2024-02-09T17:26:51.910 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f496cb08:::benchmark_data_smithi179_77138_object37459:head# 2024-02-09T17:26:51.910 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f496d002:::benchmark_data_smithi179_77138_object69957:head# 2024-02-09T17:26:51.911 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f497479d:::benchmark_data_smithi179_77138_object52061:head# 2024-02-09T17:26:51.911 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f497588b:::benchmark_data_smithi179_77138_object44571:head# 2024-02-09T17:26:51.912 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f497d8c9:::benchmark_data_smithi179_77138_object122741:head# 2024-02-09T17:26:51.912 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f49834f4:::benchmark_data_smithi179_77138_object88342:head# 2024-02-09T17:26:51.912 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4986b0d:::benchmark_data_smithi179_77138_object133347:head# 2024-02-09T17:26:51.912 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f499018f:::benchmark_data_smithi179_77138_object129186:head# 2024-02-09T17:26:51.913 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f49a8d6e:::benchmark_data_smithi179_77138_object131786:head# 2024-02-09T17:26:51.913 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f49b0120:::benchmark_data_smithi179_77138_object44858:head# 2024-02-09T17:26:51.914 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f49bebdd:::benchmark_data_smithi179_77138_object114438:head# 2024-02-09T17:26:51.914 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f49d0154:::benchmark_data_smithi179_77138_object79728:head# 2024-02-09T17:26:51.914 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f49d53de:::benchmark_data_smithi179_77138_object96856:head# 2024-02-09T17:26:51.915 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f49eaaf8:::benchmark_data_smithi179_77138_object61186:head# 2024-02-09T17:26:51.915 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f49f37fb:::benchmark_data_smithi179_77138_object117889:head# 2024-02-09T17:26:51.915 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f49f55bb:::benchmark_data_smithi179_77138_object35310:head# 2024-02-09T17:26:51.916 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f49f8b2f:::benchmark_data_smithi179_77138_object121680:head# 2024-02-09T17:26:51.917 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f49fe080:::benchmark_data_smithi179_77138_object103498:head# 2024-02-09T17:26:51.917 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4a05357:::benchmark_data_smithi179_77138_object89920:head# 2024-02-09T17:26:51.918 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4a0961b:::benchmark_data_smithi179_77138_object101887:head# 2024-02-09T17:26:51.919 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4a15a8e:::benchmark_data_smithi179_77138_object23060:head# 2024-02-09T17:26:51.920 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4a1fb7b:::benchmark_data_smithi179_77138_object82527:head# 2024-02-09T17:26:51.920 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4a27a26:::benchmark_data_smithi179_77138_object59962:head# 2024-02-09T17:26:51.921 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4a28b4b:::benchmark_data_smithi179_77138_object95675:head# 2024-02-09T17:26:51.921 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4a2acf8:::benchmark_data_smithi179_77138_object37937:head# 2024-02-09T17:26:51.922 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4a2ddf8:::benchmark_data_smithi179_77138_object31340:head# 2024-02-09T17:26:51.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:51 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:26:51.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:51 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:26:51.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:51 smithi179 ceph-mon[43975]: pgmap v451: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 6.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 806 op/s; 26903/322257 objects degraded (8.348%); 0 B/s, 0 objects/s recovering 2024-02-09T17:26:51.923 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4a34605:::benchmark_data_smithi179_77138_object88341:head# 2024-02-09T17:26:51.923 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4a3b526:::benchmark_data_smithi179_77138_object118213:head# 2024-02-09T17:26:51.923 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4a3bc94:::benchmark_data_smithi179_77138_object82137:head# 2024-02-09T17:26:51.924 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4a3e33e:::benchmark_data_smithi179_77138_object48966:head# 2024-02-09T17:26:51.924 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4a4928a:::benchmark_data_smithi179_77138_object80212:head# 2024-02-09T17:26:51.924 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4a4bc2b:::benchmark_data_smithi179_77138_object129500:head# 2024-02-09T17:26:51.925 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4a4f947:::benchmark_data_smithi179_77138_object54742:head# 2024-02-09T17:26:51.925 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4a5ef5d:::benchmark_data_smithi179_77138_object107326:head# 2024-02-09T17:26:51.925 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4a62b20:::benchmark_data_smithi179_77138_object111911:head# 2024-02-09T17:26:51.926 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4a830b4:::benchmark_data_smithi179_77138_object89378:head# 2024-02-09T17:26:51.928 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4a864b2:::benchmark_data_smithi179_77138_object62687:head# 2024-02-09T17:26:51.929 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4a8c1c6:::benchmark_data_smithi179_77138_object24832:head# 2024-02-09T17:26:51.929 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4a9d4ca:::benchmark_data_smithi179_77138_object42426:head# 2024-02-09T17:26:51.930 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4ab0d16:::benchmark_data_smithi179_77138_object105444:head# 2024-02-09T17:26:51.930 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4ab1a85:::benchmark_data_smithi179_77138_object125315:head# 2024-02-09T17:26:51.931 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4ab1c8b:::benchmark_data_smithi179_77138_object41588:head# 2024-02-09T17:26:51.931 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4ac8a26:::benchmark_data_smithi179_77138_object104569:head# 2024-02-09T17:26:51.931 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4acbf73:::benchmark_data_smithi179_77138_object35690:head# 2024-02-09T17:26:51.937 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4ad0f44:::benchmark_data_smithi179_77138_object41562:head# 2024-02-09T17:26:51.938 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4ad680d:::benchmark_data_smithi179_77138_object55398:head# 2024-02-09T17:26:51.938 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4adc594:::benchmark_data_smithi179_77138_object83206:head# 2024-02-09T17:26:51.938 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4adead1:::benchmark_data_smithi179_77138_object55892:head# 2024-02-09T17:26:51.938 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4ae8c87:::benchmark_data_smithi179_77138_object48955:head# 2024-02-09T17:26:51.938 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4af378e:::benchmark_data_smithi179_77138_object78734:head# 2024-02-09T17:26:51.938 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4af83f9:::benchmark_data_smithi179_77138_object128981:head# 2024-02-09T17:26:51.938 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4afe3c1:::benchmark_data_smithi179_77138_object58510:head# 2024-02-09T17:26:51.943 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4b00b79:::benchmark_data_smithi179_77138_object73496:head# 2024-02-09T17:26:51.943 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4b03f61:::benchmark_data_smithi179_77138_object99994:head# 2024-02-09T17:26:51.943 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4b04c39:::benchmark_data_smithi179_77138_object115139:head# 2024-02-09T17:26:51.944 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4b0b41a:::benchmark_data_smithi179_77138_object84589:head# 2024-02-09T17:26:51.944 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4b0fefa:::benchmark_data_smithi179_77138_object30610:head# 2024-02-09T17:26:51.944 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4b13aeb:::benchmark_data_smithi179_77138_object94317:head# 2024-02-09T17:26:51.947 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4b227a6:::benchmark_data_smithi179_77138_object25280:head# 2024-02-09T17:26:51.947 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4b26952:::benchmark_data_smithi179_77138_object26863:head# 2024-02-09T17:26:51.950 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4b285db:::benchmark_data_smithi179_77138_object97484:head# 2024-02-09T17:26:51.951 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:51.953 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4b3aa35:::benchmark_data_smithi179_77138_object91890:head# 2024-02-09T17:26:51.953 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4b3ea28:::benchmark_data_smithi179_77138_object126295:head# 2024-02-09T17:26:51.954 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4b3f127:::benchmark_data_smithi179_77138_object91309:head# 2024-02-09T17:26:51.954 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4b5bdaf:::benchmark_data_smithi179_77138_object35583:head# 2024-02-09T17:26:51.955 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4b5da44:::benchmark_data_smithi179_77138_object114687:head# 2024-02-09T17:26:51.956 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4b6e1e1:::benchmark_data_smithi179_77138_object97520:head# 2024-02-09T17:26:51.956 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4b6effa:::benchmark_data_smithi179_77138_object103659:head# 2024-02-09T17:26:51.956 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4b72309:::benchmark_data_smithi179_77138_object74373:head# 2024-02-09T17:26:51.957 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4b72a1b:::benchmark_data_smithi179_77138_object20611:head# 2024-02-09T17:26:51.958 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4b758a5:::benchmark_data_smithi179_77138_object129135:head# 2024-02-09T17:26:51.958 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4b75baf:::benchmark_data_smithi179_77138_object115719:head# 2024-02-09T17:26:51.959 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4b86361:::benchmark_data_smithi179_77138_object65111:head# 2024-02-09T17:26:51.959 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4b8c542:::benchmark_data_smithi179_77138_object89648:head# 2024-02-09T17:26:51.960 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4b8ded5:::benchmark_data_smithi179_77138_object67412:head# 2024-02-09T17:26:51.960 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4b95bb7:::benchmark_data_smithi179_77138_object91869:head# 2024-02-09T17:26:51.960 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4b9860d:::benchmark_data_smithi179_77138_object50730:head# 2024-02-09T17:26:51.961 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4ba6be3:::benchmark_data_smithi179_77138_object99740:head# 2024-02-09T17:26:51.961 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4ba7297:::benchmark_data_smithi179_77138_object33605:head# 2024-02-09T17:26:51.962 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4ba7dd3:::benchmark_data_smithi179_77138_object64383:head# 2024-02-09T17:26:51.962 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4baa07c:::benchmark_data_smithi179_77138_object91667:head# 2024-02-09T17:26:51.963 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4bb486c:::benchmark_data_smithi179_77138_object113565:head# 2024-02-09T17:26:51.963 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4bb98f7:::benchmark_data_smithi179_77138_object129605:head# 2024-02-09T17:26:51.964 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4bbf4ef:::benchmark_data_smithi179_77138_object130414:head# 2024-02-09T17:26:51.964 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4bcab05:::benchmark_data_smithi179_77138_object128138:head# 2024-02-09T17:26:51.965 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4bd47e3:::benchmark_data_smithi179_77138_object77828:head# 2024-02-09T17:26:51.965 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4bdb648:::benchmark_data_smithi179_77138_object33415:head# 2024-02-09T17:26:51.968 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4be5ae9:::benchmark_data_smithi179_77138_object107125:head# 2024-02-09T17:26:52.050 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:51 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:26:52.050 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:51 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:26:52.050 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:51 smithi099 ceph-mon[39420]: pgmap v451: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 6.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 806 op/s; 26903/322257 objects degraded (8.348%); 0 B/s, 0 objects/s recovering 2024-02-09T17:26:52.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:52.051 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:26:52.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:26:52.068 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4be949a:::benchmark_data_smithi179_77138_object101696:head# 2024-02-09T17:26:52.068 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4be9e45:::benchmark_data_smithi179_77138_object55910:head# 2024-02-09T17:26:52.068 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4bff853:::benchmark_data_smithi179_77138_object77627:head# 2024-02-09T17:26:52.068 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4c0058c:::benchmark_data_smithi179_77138_object98657:head# 2024-02-09T17:26:52.068 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4c01a8f:::benchmark_data_smithi179_77138_object20365:head# 2024-02-09T17:26:52.068 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4c0ff1b:::benchmark_data_smithi179_77138_object36932:head# 2024-02-09T17:26:52.068 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4c170b9:::benchmark_data_smithi179_77138_object54061:head# 2024-02-09T17:26:52.068 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4c19149:::benchmark_data_smithi179_77138_object96551:head# 2024-02-09T17:26:52.068 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4c3b222:::benchmark_data_smithi179_77138_object68063:head# 2024-02-09T17:26:52.068 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4c45563:::benchmark_data_smithi179_77138_object110391:head# 2024-02-09T17:26:52.068 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4c45a57:::benchmark_data_smithi179_77138_object104698:head# 2024-02-09T17:26:52.068 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4c46648:::benchmark_data_smithi179_77138_object62366:head# 2024-02-09T17:26:52.069 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4c6beb8:::benchmark_data_smithi179_77138_object89701:head# 2024-02-09T17:26:52.069 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4c82daa:::benchmark_data_smithi179_77138_object33214:head# 2024-02-09T17:26:52.069 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4c920a8:::benchmark_data_smithi179_77138_object90506:head# 2024-02-09T17:26:52.069 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4c96335:::benchmark_data_smithi179_77138_object129673:head# 2024-02-09T17:26:52.069 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4c9cf72:::benchmark_data_smithi179_77138_object22608:head# 2024-02-09T17:26:52.069 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4caf2d1:::benchmark_data_smithi179_77138_object82906:head# 2024-02-09T17:26:52.069 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4cb0049:::benchmark_data_smithi179_77138_object104126:head# 2024-02-09T17:26:52.069 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4cd18c7:::benchmark_data_smithi179_77138_object26714:head# 2024-02-09T17:26:52.069 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4ce07e7:::benchmark_data_smithi179_77138_object75433:head# 2024-02-09T17:26:52.069 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4cf376f:::benchmark_data_smithi179_77138_object98475:head# 2024-02-09T17:26:52.069 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4cf4210:::benchmark_data_smithi179_77138_object60261:head# 2024-02-09T17:26:52.069 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4cfa9d1:::benchmark_data_smithi179_77138_object46281:head# 2024-02-09T17:26:52.069 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4cfce15:::benchmark_data_smithi179_77138_object55630:head# 2024-02-09T17:26:52.069 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4d0831d:::benchmark_data_smithi179_77138_object31638:head# 2024-02-09T17:26:52.069 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4d0cb0f:::benchmark_data_smithi179_77138_object54699:head# 2024-02-09T17:26:52.069 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4d1925b:::benchmark_data_smithi179_77138_object29694:head# 2024-02-09T17:26:52.070 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4d26711:::benchmark_data_smithi179_77138_object52577:head# 2024-02-09T17:26:52.070 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4d2aed9:::benchmark_data_smithi179_77138_object38267:head# 2024-02-09T17:26:52.070 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4d2ea3c:::benchmark_data_smithi179_77138_object30471:head# 2024-02-09T17:26:52.070 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4d39310:::benchmark_data_smithi179_77138_object98822:head# 2024-02-09T17:26:52.070 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4d3a1a8:::benchmark_data_smithi179_77138_object76842:head# 2024-02-09T17:26:52.070 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4d3b510:::benchmark_data_smithi179_77138_object34132:head# 2024-02-09T17:26:52.070 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4d4d30d:::benchmark_data_smithi179_77138_object99012:head# 2024-02-09T17:26:52.070 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4d550be:::benchmark_data_smithi179_77138_object34836:head# 2024-02-09T17:26:52.070 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4d595f1:::benchmark_data_smithi179_77138_object62943:head# 2024-02-09T17:26:52.070 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4d5af9f:::benchmark_data_smithi179_77138_object63814:head# 2024-02-09T17:26:52.070 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4d5afcb:::benchmark_data_smithi179_77138_object34206:head# 2024-02-09T17:26:52.070 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4d5b5b2:::benchmark_data_smithi179_77138_object111748:head# 2024-02-09T17:26:52.070 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4d63853:::benchmark_data_smithi179_77138_object23126:head# 2024-02-09T17:26:52.070 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4d67eed:::benchmark_data_smithi179_77138_object106858:head# 2024-02-09T17:26:52.071 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4d700fe:::benchmark_data_smithi179_77138_object78228:head# 2024-02-09T17:26:52.071 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4d7648c:::benchmark_data_smithi179_77138_object130073:head# 2024-02-09T17:26:52.071 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4d783c7:::benchmark_data_smithi179_77138_object128720:head# 2024-02-09T17:26:52.071 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4d7e875:::benchmark_data_smithi179_77138_object88678:head# 2024-02-09T17:26:52.071 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4d8c1be:::benchmark_data_smithi179_77138_object89898:head# 2024-02-09T17:26:52.071 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4d9fc1e:::benchmark_data_smithi179_77138_object30013:head# 2024-02-09T17:26:52.071 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4da7f1e:::benchmark_data_smithi179_77138_object106777:head# 2024-02-09T17:26:52.071 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4dacc2a:::benchmark_data_smithi179_77138_object38930:head# 2024-02-09T17:26:52.071 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4db1d84:::benchmark_data_smithi179_77138_object130955:head# 2024-02-09T17:26:52.071 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4db1f25:::benchmark_data_smithi179_77138_object47756:head# 2024-02-09T17:26:52.071 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4db3c9c:::benchmark_data_smithi179_77138_object46601:head# 2024-02-09T17:26:52.071 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4db6e7a:::benchmark_data_smithi179_77138_object100521:head# 2024-02-09T17:26:52.071 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4dc1779:::benchmark_data_smithi179_77138_object83281:head# 2024-02-09T17:26:52.071 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4dca59e:::benchmark_data_smithi179_77138_object133096:head# 2024-02-09T17:26:52.072 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4dcc671:::benchmark_data_smithi179_77138_object69807:head# 2024-02-09T17:26:52.072 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4dd00aa:::benchmark_data_smithi179_77138_object57860:head# 2024-02-09T17:26:52.072 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4dd2b33:::benchmark_data_smithi179_77138_object34712:head# 2024-02-09T17:26:52.072 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4dda4ac:::benchmark_data_smithi179_77138_object41178:head# 2024-02-09T17:26:52.072 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4ddd6e4:::benchmark_data_smithi179_77138_object132761:head# 2024-02-09T17:26:52.072 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4de09f8:::benchmark_data_smithi179_77138_object78523:head# 2024-02-09T17:26:52.072 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4de7fee:::benchmark_data_smithi179_77138_object85028:head# 2024-02-09T17:26:52.072 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4df1216:::benchmark_data_smithi179_77138_object58898:head# 2024-02-09T17:26:52.072 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4dfbbdd:::benchmark_data_smithi179_77138_object48136:head# 2024-02-09T17:26:52.072 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4e03155:::benchmark_data_smithi179_77138_object76530:head# 2024-02-09T17:26:52.072 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4e0c4df:::benchmark_data_smithi179_77138_object49313:head# 2024-02-09T17:26:52.072 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4e0f7d6:::benchmark_data_smithi179_77138_object20119:head# 2024-02-09T17:26:52.072 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4e0f998:::benchmark_data_smithi179_77138_object96813:head# 2024-02-09T17:26:52.072 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4e229cd:::benchmark_data_smithi179_77138_object50664:head# 2024-02-09T17:26:52.073 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4e2737a:::benchmark_data_smithi179_77138_object41509:head# 2024-02-09T17:26:52.073 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4e2e1da:::benchmark_data_smithi179_77138_object70159:head# 2024-02-09T17:26:52.073 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4e30760:::benchmark_data_smithi179_77138_object72672:head# 2024-02-09T17:26:52.073 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4e4c42f:::benchmark_data_smithi179_77138_object65703:head# 2024-02-09T17:26:52.073 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4e4ee99:::benchmark_data_smithi179_77138_object90093:head# 2024-02-09T17:26:52.073 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4e50161:::benchmark_data_smithi179_77138_object70277:head# 2024-02-09T17:26:52.073 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4e5795e:::benchmark_data_smithi179_77138_object19404:head# 2024-02-09T17:26:52.073 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4e5dd15:::benchmark_data_smithi179_77138_object95326:head# 2024-02-09T17:26:52.073 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4e612ac:::benchmark_data_smithi179_77138_object25967:head# 2024-02-09T17:26:52.073 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4e62ec7:::benchmark_data_smithi179_77138_object110768:head# 2024-02-09T17:26:52.073 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4e64f66:::benchmark_data_smithi179_77138_object90853:head# 2024-02-09T17:26:52.073 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4e7129a:::benchmark_data_smithi179_77138_object22354:head# 2024-02-09T17:26:52.073 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4e99d01:::benchmark_data_smithi179_77138_object86769:head# 2024-02-09T17:26:52.073 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4ea8871:::benchmark_data_smithi179_77138_object110756:head# 2024-02-09T17:26:52.073 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4eb78cf:::benchmark_data_smithi179_77138_object89364:head# 2024-02-09T17:26:52.074 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4ebe356:::benchmark_data_smithi179_77138_object118536:head# 2024-02-09T17:26:52.074 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4ed0bed:::benchmark_data_smithi179_77138_object98604:head# 2024-02-09T17:26:52.074 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4ed1f6b:::benchmark_data_smithi179_77138_object86797:head# 2024-02-09T17:26:52.074 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4ed25e5:::benchmark_data_smithi179_77138_object56833:head# 2024-02-09T17:26:52.074 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4ed5715:::benchmark_data_smithi179_77138_object22080:head# 2024-02-09T17:26:52.074 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4ee4295:::benchmark_data_smithi179_77138_object67378:head# 2024-02-09T17:26:52.074 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4ee4395:::benchmark_data_smithi179_77138_object93858:head# 2024-02-09T17:26:52.074 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4eea2ba:::benchmark_data_smithi179_77138_object126296:head# 2024-02-09T17:26:52.074 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4eec3e4:::benchmark_data_smithi179_77138_object51613:head# 2024-02-09T17:26:52.074 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4ef2976:::benchmark_data_smithi179_77138_object24401:head# 2024-02-09T17:26:52.074 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4ef5f2b:::benchmark_data_smithi179_77138_object102367:head# 2024-02-09T17:26:52.074 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4ef9f91:::benchmark_data_smithi179_77138_object63736:head# 2024-02-09T17:26:52.074 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4efc8df:::benchmark_data_smithi179_77138_object46955:head# 2024-02-09T17:26:52.074 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4efcaac:::benchmark_data_smithi179_77138_object110765:head# 2024-02-09T17:26:52.074 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4efd10b:::benchmark_data_smithi179_77138_object53782:head# 2024-02-09T17:26:52.075 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4f0108e:::benchmark_data_smithi179_77138_object93384:head# 2024-02-09T17:26:52.075 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4f1d9d3:::benchmark_data_smithi179_77138_object92571:head# 2024-02-09T17:26:52.075 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4f3400b:::benchmark_data_smithi179_77138_object36550:head# 2024-02-09T17:26:52.075 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4f35caf:::benchmark_data_smithi179_77138_object54046:head# 2024-02-09T17:26:52.075 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4f3ae20:::benchmark_data_smithi179_77138_object67439:head# 2024-02-09T17:26:52.075 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4f3c99c:::benchmark_data_smithi179_77138_object44268:head# 2024-02-09T17:26:52.075 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4f40e5a:::benchmark_data_smithi179_77138_object61669:head# 2024-02-09T17:26:52.075 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4f45008:::benchmark_data_smithi179_77138_object44401:head# 2024-02-09T17:26:52.075 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4f492dc:::benchmark_data_smithi179_77138_object75914:head# 2024-02-09T17:26:52.075 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4f4e40a:::benchmark_data_smithi179_77138_object107667:head# 2024-02-09T17:26:52.075 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4f4ec8d:::benchmark_data_smithi179_77138_object131388:head# 2024-02-09T17:26:52.075 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4f6bf18:::benchmark_data_smithi179_77138_object63953:head# 2024-02-09T17:26:52.075 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4f6d719:::benchmark_data_smithi179_77138_object111058:head# 2024-02-09T17:26:52.075 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4f6e33c:::benchmark_data_smithi179_77138_object37221:head# 2024-02-09T17:26:52.075 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4f6f655:::benchmark_data_smithi179_77138_object110118:head# 2024-02-09T17:26:52.076 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4f7757e:::benchmark_data_smithi179_77138_object57717:head# 2024-02-09T17:26:52.076 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4f7fb6a:::benchmark_data_smithi179_77138_object88012:head# 2024-02-09T17:26:52.076 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4f84bd2:::benchmark_data_smithi179_77138_object22546:head# 2024-02-09T17:26:52.076 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4f8639a:::benchmark_data_smithi179_77138_object28373:head# 2024-02-09T17:26:52.076 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4f95acb:::benchmark_data_smithi179_77138_object91739:head# 2024-02-09T17:26:52.076 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4f99f06:::benchmark_data_smithi179_77138_object88030:head# 2024-02-09T17:26:52.076 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4fb09e1:::benchmark_data_smithi179_77138_object125756:head# 2024-02-09T17:26:52.076 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4fbb5a8:::benchmark_data_smithi179_77138_object109866:head# 2024-02-09T17:26:52.076 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4fcdcf1:::benchmark_data_smithi179_77138_object67898:head# 2024-02-09T17:26:52.076 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4fcfc3b:::benchmark_data_smithi179_77138_object132575:head# 2024-02-09T17:26:52.076 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4fd8c06:::benchmark_data_smithi179_77138_object24688:head# 2024-02-09T17:26:52.076 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4feba50:::benchmark_data_smithi179_77138_object81221:head# 2024-02-09T17:26:52.076 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4fef2ff:::benchmark_data_smithi179_77138_object22081:head# 2024-02-09T17:26:52.076 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4ff1cd5:::benchmark_data_smithi179_77138_object39710:head# 2024-02-09T17:26:52.077 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f4ff7cdc:::benchmark_data_smithi179_77138_object126868:head# 2024-02-09T17:26:52.077 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f500bc3a:::benchmark_data_smithi179_77138_object63905:head# 2024-02-09T17:26:52.077 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f500cda7:::benchmark_data_smithi179_77138_object60584:head# 2024-02-09T17:26:52.077 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f500f98f:::benchmark_data_smithi179_77138_object93789:head# 2024-02-09T17:26:52.077 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f501ec43:::benchmark_data_smithi179_77138_object84746:head# 2024-02-09T17:26:52.077 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5025aac:::benchmark_data_smithi179_77138_object116297:head# 2024-02-09T17:26:52.077 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5031953:::benchmark_data_smithi179_77138_object61446:head# 2024-02-09T17:26:52.077 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f504048f:::benchmark_data_smithi179_77138_object27876:head# 2024-02-09T17:26:52.077 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f504840b:::benchmark_data_smithi179_77138_object77061:head# 2024-02-09T17:26:52.077 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f505162d:::benchmark_data_smithi179_77138_object51739:head# 2024-02-09T17:26:52.077 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5055147:::benchmark_data_smithi179_77138_object84095:head# 2024-02-09T17:26:52.077 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f505ef7b:::benchmark_data_smithi179_77138_object90539:head# 2024-02-09T17:26:52.077 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5060311:::benchmark_data_smithi179_77138_object84719:head# 2024-02-09T17:26:52.077 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f506756a:::benchmark_data_smithi179_77138_object89846:head# 2024-02-09T17:26:52.077 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f50927f5:::benchmark_data_smithi179_77138_object85765:head# 2024-02-09T17:26:52.078 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5095f4c:::benchmark_data_smithi179_77138_object70998:head# 2024-02-09T17:26:52.078 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f509d8fb:::benchmark_data_smithi179_77138_object88846:head# 2024-02-09T17:26:52.078 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f50a5fcc:::benchmark_data_smithi179_77138_object116593:head# 2024-02-09T17:26:52.078 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f50a6a97:::benchmark_data_smithi179_77138_object20597:head# 2024-02-09T17:26:52.078 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f50a6d6d:::benchmark_data_smithi179_77138_object101883:head# 2024-02-09T17:26:52.078 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f50accd0:::benchmark_data_smithi179_77138_object35764:head# 2024-02-09T17:26:52.078 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f50b7cde:::benchmark_data_smithi179_77138_object114035:head# 2024-02-09T17:26:52.078 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f50bff95:::benchmark_data_smithi179_77138_object108459:head# 2024-02-09T17:26:52.078 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f50c3861:::benchmark_data_smithi179_77138_object65337:head# 2024-02-09T17:26:52.078 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f50cfaa9:::benchmark_data_smithi179_77138_object55798:head# 2024-02-09T17:26:52.078 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f50d4e18:::benchmark_data_smithi179_77138_object57527:head# 2024-02-09T17:26:52.078 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f50ea456:::benchmark_data_smithi179_77138_object41995:head# 2024-02-09T17:26:52.078 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f50ed249:::benchmark_data_smithi179_77138_object38569:head# 2024-02-09T17:26:52.078 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f50f7594:::benchmark_data_smithi179_77138_object33580:head# 2024-02-09T17:26:52.078 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f50fa4db:::benchmark_data_smithi179_77138_object38226:head# 2024-02-09T17:26:52.079 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f50ff66b:::benchmark_data_smithi179_77138_object94858:head# 2024-02-09T17:26:52.079 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f51062c8:::benchmark_data_smithi179_77138_object97591:head# 2024-02-09T17:26:52.079 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f512ded6:::benchmark_data_smithi179_77138_object28493:head# 2024-02-09T17:26:52.079 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f51359d1:::benchmark_data_smithi179_77138_object42687:head# 2024-02-09T17:26:52.079 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f51455d5:::benchmark_data_smithi179_77138_object102495:head# 2024-02-09T17:26:52.079 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f514ec7a:::benchmark_data_smithi179_77138_object48827:head# 2024-02-09T17:26:52.079 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5154e2b:::benchmark_data_smithi179_77138_object104971:head# 2024-02-09T17:26:52.079 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5163615:::benchmark_data_smithi179_77138_object69456:head# 2024-02-09T17:26:52.079 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f516a601:::benchmark_data_smithi179_77138_object121904:head# 2024-02-09T17:26:52.079 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5175a3c:::benchmark_data_smithi179_77138_object96946:head# 2024-02-09T17:26:52.079 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5184a0d:::benchmark_data_smithi179_77138_object95527:head# 2024-02-09T17:26:52.079 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f518f840:::benchmark_data_smithi179_77138_object48172:head# 2024-02-09T17:26:52.079 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5192089:::benchmark_data_smithi179_77138_object83621:head# 2024-02-09T17:26:52.079 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5195277:::benchmark_data_smithi179_77138_object103549:head# 2024-02-09T17:26:52.079 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f519af88:::benchmark_data_smithi179_77138_object51420:head# 2024-02-09T17:26:52.079 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f51bda34:::benchmark_data_smithi179_77138_object99413:head# 2024-02-09T17:26:52.080 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f51be0ff:::benchmark_data_smithi179_77138_object102288:head# 2024-02-09T17:26:52.080 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f51bfe6b:::benchmark_data_smithi179_77138_object125082:head# 2024-02-09T17:26:52.080 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f51c006e:::benchmark_data_smithi179_77138_object53574:head# 2024-02-09T17:26:52.080 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f51c37b5:::benchmark_data_smithi179_77138_object80425:head# 2024-02-09T17:26:52.080 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f51d9ebb:::benchmark_data_smithi179_77138_object106469:head# 2024-02-09T17:26:52.080 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f51e254f:::benchmark_data_smithi179_77138_object74069:head# 2024-02-09T17:26:52.080 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f51e2edb:::benchmark_data_smithi179_77138_object49661:head# 2024-02-09T17:26:52.080 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f51ed20e:::benchmark_data_smithi179_77138_object27610:head# 2024-02-09T17:26:52.080 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f51fcabf:::benchmark_data_smithi179_77138_object67471:head# 2024-02-09T17:26:52.080 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f52020bf:::benchmark_data_smithi179_77138_object123064:head# 2024-02-09T17:26:52.080 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f521399a:::benchmark_data_smithi179_77138_object103566:head# 2024-02-09T17:26:52.080 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5218213:::benchmark_data_smithi179_77138_object87260:head# 2024-02-09T17:26:52.080 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f521c1fd:::benchmark_data_smithi179_77138_object93079:head# 2024-02-09T17:26:52.080 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5223536:::benchmark_data_smithi179_77138_object104800:head# 2024-02-09T17:26:52.081 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f52291f4:::benchmark_data_smithi179_77138_object119332:head# 2024-02-09T17:26:52.081 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f522f2b3:::benchmark_data_smithi179_77138_object88128:head# 2024-02-09T17:26:52.081 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f523bab7:::benchmark_data_smithi179_77138_object83637:head# 2024-02-09T17:26:52.081 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f523f99a:::benchmark_data_smithi179_77138_object84794:head# 2024-02-09T17:26:52.081 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5241d78:::benchmark_data_smithi179_77138_object55595:head# 2024-02-09T17:26:52.081 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f524980f:::benchmark_data_smithi179_77138_object103784:head# 2024-02-09T17:26:52.081 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5270d56:::benchmark_data_smithi179_77138_object109821:head# 2024-02-09T17:26:52.081 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f527b7a1:::benchmark_data_smithi179_77138_object77876:head# 2024-02-09T17:26:52.082 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f52886ea:::benchmark_data_smithi179_77138_object93168:head# 2024-02-09T17:26:52.082 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f529a1e8:::benchmark_data_smithi179_77138_object26978:head# 2024-02-09T17:26:52.082 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f529c8fb:::benchmark_data_smithi179_77138_object47724:head# 2024-02-09T17:26:52.082 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f52abef2:::benchmark_data_smithi179_77138_object41035:head# 2024-02-09T17:26:52.082 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f52ad7fc:::benchmark_data_smithi179_77138_object78736:head# 2024-02-09T17:26:52.082 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f52adb8f:::benchmark_data_smithi179_77138_object26850:head# 2024-02-09T17:26:52.083 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f52b4639:::benchmark_data_smithi179_77138_object30739:head# 2024-02-09T17:26:52.083 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f52d0b22:::benchmark_data_smithi179_77138_object45339:head# 2024-02-09T17:26:52.083 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f52d5da9:::benchmark_data_smithi179_77138_object90180:head# 2024-02-09T17:26:52.083 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f52dcef1:::benchmark_data_smithi179_77138_object36922:head# 2024-02-09T17:26:52.083 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f52e229e:::benchmark_data_smithi179_77138_object127336:head# 2024-02-09T17:26:52.083 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f52e7cac:::benchmark_data_smithi179_77138_object42485:head# 2024-02-09T17:26:52.083 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f52ef8bb:::benchmark_data_smithi179_77138_object52072:head# 2024-02-09T17:26:52.084 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f52f96e7:::benchmark_data_smithi179_77138_object44200:head# 2024-02-09T17:26:52.084 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f530df98:::benchmark_data_smithi179_77138_object82220:head# 2024-02-09T17:26:52.084 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f53148b6:::benchmark_data_smithi179_77138_object64309:head# 2024-02-09T17:26:52.085 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5320ce4:::benchmark_data_smithi179_77138_object121177:head# 2024-02-09T17:26:52.085 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5356443:::benchmark_data_smithi179_77138_object37279:head# 2024-02-09T17:26:52.086 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5361457:::benchmark_data_smithi179_77138_object52499:head# 2024-02-09T17:26:52.086 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f536d9f9:::benchmark_data_smithi179_77138_object78176:head# 2024-02-09T17:26:52.087 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5375e2c:::benchmark_data_smithi179_77138_object77744:head# 2024-02-09T17:26:52.089 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5377a7b:::benchmark_data_smithi179_77138_object120201:head# 2024-02-09T17:26:52.089 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f537d2c9:::benchmark_data_smithi179_77138_object22341:head# 2024-02-09T17:26:52.090 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f538179c:::benchmark_data_smithi179_77138_object79633:head# 2024-02-09T17:26:52.091 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5383a1c:::benchmark_data_smithi179_77138_object82994:head# 2024-02-09T17:26:52.093 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f538b699:::benchmark_data_smithi179_77138_object51568:head# 2024-02-09T17:26:52.093 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f538b7ae:::benchmark_data_smithi179_77138_object88146:head# 2024-02-09T17:26:52.093 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f53a427d:::benchmark_data_smithi179_77138_object82011:head# 2024-02-09T17:26:52.094 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f53afcc9:::benchmark_data_smithi179_77138_object64811:head# 2024-02-09T17:26:52.094 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f53b3cba:::benchmark_data_smithi179_77138_object90378:head# 2024-02-09T17:26:52.095 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f53b861f:::benchmark_data_smithi179_77138_object120155:head# 2024-02-09T17:26:52.095 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f53bec54:::benchmark_data_smithi179_77138_object64909:head# 2024-02-09T17:26:52.096 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f53c11fe:::benchmark_data_smithi179_77138_object45316:head# 2024-02-09T17:26:52.096 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f53c3803:::benchmark_data_smithi179_77138_object62816:head# 2024-02-09T17:26:52.097 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f53c7e42:::benchmark_data_smithi179_77138_object28599:head# 2024-02-09T17:26:52.097 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f53cca90:::benchmark_data_smithi179_77138_object70334:head# 2024-02-09T17:26:52.098 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f53ceed1:::benchmark_data_smithi179_77138_object103527:head# 2024-02-09T17:26:52.098 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f53d3364:::benchmark_data_smithi179_77138_object80230:head# 2024-02-09T17:26:52.099 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f53d5701:::benchmark_data_smithi179_77138_object120704:head# 2024-02-09T17:26:52.099 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f53d6f38:::benchmark_data_smithi179_77138_object70207:head# 2024-02-09T17:26:52.100 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f53dcf4f:::benchmark_data_smithi179_77138_object68693:head# 2024-02-09T17:26:52.100 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f53eba8d:::benchmark_data_smithi179_77138_object129040:head# 2024-02-09T17:26:52.100 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f53ee90c:::benchmark_data_smithi179_77138_object89348:head# 2024-02-09T17:26:52.104 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f53f0e7e:::benchmark_data_smithi179_77138_object21478:head# 2024-02-09T17:26:52.104 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f53f10cf:::benchmark_data_smithi179_77138_object106004:head# 2024-02-09T17:26:52.104 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f53f5ca0:::benchmark_data_smithi179_77138_object67860:head# 2024-02-09T17:26:52.104 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f540efc5:::benchmark_data_smithi179_77138_object73314:head# 2024-02-09T17:26:52.104 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5426dc0:::benchmark_data_smithi179_77138_object50061:head# 2024-02-09T17:26:52.104 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5438d00:::benchmark_data_smithi179_77138_object30676:head# 2024-02-09T17:26:52.104 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f54493c8:::benchmark_data_smithi179_77138_object29861:head# 2024-02-09T17:26:52.105 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f54522ec:::benchmark_data_smithi179_77138_object58443:head# 2024-02-09T17:26:52.105 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f546c189:::benchmark_data_smithi179_77138_object66053:head# 2024-02-09T17:26:52.106 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5479124:::benchmark_data_smithi179_77138_object48131:head# 2024-02-09T17:26:52.107 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f547c134:::benchmark_data_smithi179_77138_object41849:head# 2024-02-09T17:26:52.107 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5481fb9:::benchmark_data_smithi179_77138_object69153:head# 2024-02-09T17:26:52.109 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f54851ba:::benchmark_data_smithi179_77138_object121141:head# 2024-02-09T17:26:52.109 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5486b31:::benchmark_data_smithi179_77138_object91313:head# 2024-02-09T17:26:52.109 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f548b120:::benchmark_data_smithi179_77138_object98616:head# 2024-02-09T17:26:52.109 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f548bee4:::benchmark_data_smithi179_77138_object30155:head# 2024-02-09T17:26:52.109 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f548e68e:::benchmark_data_smithi179_77138_object93370:head# 2024-02-09T17:26:52.110 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5491171:::benchmark_data_smithi179_77138_object41394:head# 2024-02-09T17:26:52.110 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f549476d:::benchmark_data_smithi179_77138_object119059:head# 2024-02-09T17:26:52.112 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f54a1185:::benchmark_data_smithi179_77138_object29647:head# 2024-02-09T17:26:52.113 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f54a3ecd:::benchmark_data_smithi179_77138_object33300:head# 2024-02-09T17:26:52.113 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f54b964c:::benchmark_data_smithi179_77138_object37670:head# 2024-02-09T17:26:52.114 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f54c9cdc:::benchmark_data_smithi179_77138_object103496:head# 2024-02-09T17:26:52.114 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f54cf8da:::benchmark_data_smithi179_77138_object26353:head# 2024-02-09T17:26:52.115 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f54ec100:::benchmark_data_smithi179_77138_object119140:head# 2024-02-09T17:26:52.115 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f54f0a6c:::benchmark_data_smithi179_77138_object33936:head# 2024-02-09T17:26:52.116 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f54f11c5:::benchmark_data_smithi179_77138_object87031:head# 2024-02-09T17:26:52.116 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f54f2712:::benchmark_data_smithi179_77138_object95391:head# 2024-02-09T17:26:52.117 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f54fc323:::benchmark_data_smithi179_77138_object27961:head# 2024-02-09T17:26:52.120 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f54ff8fb:::benchmark_data_smithi179_77138_object34073:head# 2024-02-09T17:26:52.120 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5512a46:::benchmark_data_smithi179_77138_object120550:head# 2024-02-09T17:26:52.120 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5528033:::benchmark_data_smithi179_77138_object124201:head# 2024-02-09T17:26:52.122 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f552ff9d:::benchmark_data_smithi179_77138_object30323:head# 2024-02-09T17:26:52.123 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f553e8aa:::benchmark_data_smithi179_77138_object121923:head# 2024-02-09T17:26:52.123 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f55434f7:::benchmark_data_smithi179_77138_object129405:head# 2024-02-09T17:26:52.124 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f554a031:::benchmark_data_smithi179_77138_object52553:head# 2024-02-09T17:26:52.124 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f554fb41:::benchmark_data_smithi179_77138_object75654:head# 2024-02-09T17:26:52.124 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5550031:::benchmark_data_smithi179_77138_object21330:head# 2024-02-09T17:26:52.125 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5557e0d:::benchmark_data_smithi179_77138_object100863:head# 2024-02-09T17:26:52.125 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f555accc:::benchmark_data_smithi179_77138_object28126:head# 2024-02-09T17:26:52.126 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f555dfca:::benchmark_data_smithi179_77138_object122936:head# 2024-02-09T17:26:52.126 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5561efe:::benchmark_data_smithi179_77138_object83447:head# 2024-02-09T17:26:52.127 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f556cb06:::benchmark_data_smithi179_77138_object81575:head# 2024-02-09T17:26:52.130 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f556ebda:::benchmark_data_smithi179_77138_object108641:head# 2024-02-09T17:26:52.130 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f55749bb:::benchmark_data_smithi179_77138_object22735:head# 2024-02-09T17:26:52.132 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5577afc:::benchmark_data_smithi179_77138_object36589:head# 2024-02-09T17:26:52.132 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5596381:::benchmark_data_smithi179_77138_object55064:head# 2024-02-09T17:26:52.133 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f559702d:::benchmark_data_smithi179_77138_object66709:head# 2024-02-09T17:26:52.136 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f55a0f24:::benchmark_data_smithi179_77138_object85573:head# 2024-02-09T17:26:52.136 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f55b21dd:::benchmark_data_smithi179_77138_object93184:head# 2024-02-09T17:26:52.136 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f55e9b89:::benchmark_data_smithi179_77138_object50897:head# 2024-02-09T17:26:52.136 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f55f2659:::benchmark_data_smithi179_77138_object129391:head# 2024-02-09T17:26:52.137 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f55f2d52:::benchmark_data_smithi179_77138_object76895:head# 2024-02-09T17:26:52.137 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f55f978e:::benchmark_data_smithi179_77138_object58712:head# 2024-02-09T17:26:52.137 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f55fad34:::benchmark_data_smithi179_77138_object74858:head# 2024-02-09T17:26:52.137 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5603f8c:::benchmark_data_smithi179_77138_object91383:head# 2024-02-09T17:26:52.138 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f56076f5:::benchmark_data_smithi179_77138_object68702:head# 2024-02-09T17:26:52.138 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f56135c1:::benchmark_data_smithi179_77138_object114704:head# 2024-02-09T17:26:52.139 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f56143d2:::benchmark_data_smithi179_77138_object124448:head# 2024-02-09T17:26:52.139 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f561e226:::benchmark_data_smithi179_77138_object125056:head# 2024-02-09T17:26:52.140 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f562fdc3:::benchmark_data_smithi179_77138_object71573:head# 2024-02-09T17:26:52.140 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f56361e7:::benchmark_data_smithi179_77138_object73835:head# 2024-02-09T17:26:52.143 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f563c18f:::benchmark_data_smithi179_77138_object124028:head# 2024-02-09T17:26:52.143 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5640584:::benchmark_data_smithi179_77138_object73309:head# 2024-02-09T17:26:52.143 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f56571da:::benchmark_data_smithi179_77138_object133155:head# 2024-02-09T17:26:52.144 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f566481e:::benchmark_data_smithi179_77138_object118376:head# 2024-02-09T17:26:52.144 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5672df9:::benchmark_data_smithi179_77138_object115948:head# 2024-02-09T17:26:52.144 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f567c860:::benchmark_data_smithi179_77138_object51595:head# 2024-02-09T17:26:52.145 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:26:52.146 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5682f9b:::benchmark_data_smithi179_77138_object75811:head# 2024-02-09T17:26:52.146 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5685e47:::benchmark_data_smithi179_77138_object102754:head# 2024-02-09T17:26:52.146 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f568669e:::benchmark_data_smithi179_77138_object47340:head# 2024-02-09T17:26:52.146 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5686f85:::benchmark_data_smithi179_77138_object105302:head# 2024-02-09T17:26:52.147 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5689adb:::benchmark_data_smithi179_77138_object59163:head# 2024-02-09T17:26:52.147 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f568c770:::benchmark_data_smithi179_77138_object68612:head# 2024-02-09T17:26:52.147 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f56a1357:::benchmark_data_smithi179_77138_object114601:head# 2024-02-09T17:26:52.148 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f56ad36d:::benchmark_data_smithi179_77138_object40009:head# 2024-02-09T17:26:52.148 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f56c0194:::benchmark_data_smithi179_77138_object67779:head# 2024-02-09T17:26:52.149 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f56c134e:::benchmark_data_smithi179_77138_object71151:head# 2024-02-09T17:26:52.149 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f56cba5e:::benchmark_data_smithi179_77138_object31080:head# 2024-02-09T17:26:52.151 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f56d2363:::benchmark_data_smithi179_77138_object33069:head# 2024-02-09T17:26:52.151 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f56f0f77:::benchmark_data_smithi179_77138_object69197:head# 2024-02-09T17:26:52.152 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f56f14aa:::benchmark_data_smithi179_77138_object59154:head# 2024-02-09T17:26:52.152 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:26:52.154 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f56f38ce:::benchmark_data_smithi179_77138_object75390:head# 2024-02-09T17:26:52.155 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f56fe48b:::benchmark_data_smithi179_77138_object61409:head# 2024-02-09T17:26:52.155 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f570a155:::benchmark_data_smithi179_77138_object53069:head# 2024-02-09T17:26:52.156 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f570e2fe:::benchmark_data_smithi179_77138_object31095:head# 2024-02-09T17:26:52.157 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f572d8af:::benchmark_data_smithi179_77138_object33778:head# 2024-02-09T17:26:52.157 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f572e8dc:::benchmark_data_smithi179_77138_object82409:head# 2024-02-09T17:26:52.158 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5730606:::benchmark_data_smithi179_77138_object69824:head# 2024-02-09T17:26:52.159 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5731e4a:::benchmark_data_smithi179_77138_object92582:head# 2024-02-09T17:26:52.159 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5737933:::benchmark_data_smithi179_77138_object28868:head# 2024-02-09T17:26:52.159 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5738df8:::benchmark_data_smithi179_77138_object35876:head# 2024-02-09T17:26:52.159 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f574141b:::benchmark_data_smithi179_77138_object88258:head# 2024-02-09T17:26:52.160 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5743306:::benchmark_data_smithi179_77138_object30831:head# 2024-02-09T17:26:52.160 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f575ba6c:::benchmark_data_smithi179_77138_object32384:head# 2024-02-09T17:26:52.160 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f57602d8:::benchmark_data_smithi179_77138_object75725:head# 2024-02-09T17:26:52.160 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5769952:::benchmark_data_smithi179_77138_object102862:head# 2024-02-09T17:26:52.161 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5770773:::benchmark_data_smithi179_77138_object86284:head# 2024-02-09T17:26:52.161 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5775884:::benchmark_data_smithi179_77138_object29751:head# 2024-02-09T17:26:52.162 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5776cae:::benchmark_data_smithi179_77138_object116343:head# 2024-02-09T17:26:52.162 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5790e56:::benchmark_data_smithi179_77138_object121207:head# 2024-02-09T17:26:52.163 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f57927b9:::benchmark_data_smithi179_77138_object131800:head# 2024-02-09T17:26:52.163 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f57a7165:::benchmark_data_smithi179_77138_object131422:head# 2024-02-09T17:26:52.164 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f57acbf4:::benchmark_data_smithi179_77138_object91582:head# 2024-02-09T17:26:52.164 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f57bfa3e:::benchmark_data_smithi179_77138_object54031:head# 2024-02-09T17:26:52.164 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f57c8a73:::benchmark_data_smithi179_77138_object116522:head# 2024-02-09T17:26:52.165 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f57ca848:::benchmark_data_smithi179_77138_object54756:head# 2024-02-09T17:26:52.166 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f57d938e:::benchmark_data_smithi179_77138_object21123:head# 2024-02-09T17:26:52.166 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f57d9a1c:::benchmark_data_smithi179_77138_object99795:head# 2024-02-09T17:26:52.166 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f57e53ed:::benchmark_data_smithi179_77138_object31612:head# 2024-02-09T17:26:52.167 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f57eb7da:::benchmark_data_smithi179_77138_object40167:head# 2024-02-09T17:26:52.170 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f57eed97:::benchmark_data_smithi179_77138_object118163:head# 2024-02-09T17:26:52.170 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f57f0f0b:::benchmark_data_smithi179_77138_object84952:head# 2024-02-09T17:26:52.170 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f57f5ed0:::benchmark_data_smithi179_77138_object49473:head# 2024-02-09T17:26:52.170 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5805af0:::benchmark_data_smithi179_77138_object42438:head# 2024-02-09T17:26:52.170 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5809137:::benchmark_data_smithi179_77138_object129955:head# 2024-02-09T17:26:52.170 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f580c6ac:::benchmark_data_smithi179_77138_object66723:head# 2024-02-09T17:26:52.171 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f580d474:::benchmark_data_smithi179_77138_object30457:head# 2024-02-09T17:26:52.171 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5819b99:::benchmark_data_smithi179_77138_object34784:head# 2024-02-09T17:26:52.172 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f581ad18:::benchmark_data_smithi179_77138_object79559:head# 2024-02-09T17:26:52.172 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f582fae5:::benchmark_data_smithi179_77138_object59133:head# 2024-02-09T17:26:52.173 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5833739:::benchmark_data_smithi179_77138_object44020:head# 2024-02-09T17:26:52.174 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f58340ab:::benchmark_data_smithi179_77138_object73968:head# 2024-02-09T17:26:52.174 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5835e6a:::benchmark_data_smithi179_77138_object114646:head# 2024-02-09T17:26:52.175 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f584ce3c:::benchmark_data_smithi179_77138_object78698:head# 2024-02-09T17:26:52.175 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f58521d0:::benchmark_data_smithi179_77138_object87589:head# 2024-02-09T17:26:52.175 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f585924c:::benchmark_data_smithi179_77138_object29572:head# 2024-02-09T17:26:52.179 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5863a75:::benchmark_data_smithi179_77138_object77741:head# 2024-02-09T17:26:52.179 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5868e05:::benchmark_data_smithi179_77138_object109363:head# 2024-02-09T17:26:52.179 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f586ec87:::benchmark_data_smithi179_77138_object116022:head# 2024-02-09T17:26:52.180 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f587fa27:::benchmark_data_smithi179_77138_object73422:head# 2024-02-09T17:26:52.180 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f58813c0:::benchmark_data_smithi179_77138_object130694:head# 2024-02-09T17:26:52.186 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:52.187 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f58a64bf:::benchmark_data_smithi179_77138_object110218:head# 2024-02-09T17:26:52.187 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f58a7e64:::benchmark_data_smithi179_77138_object35353:head# 2024-02-09T17:26:52.187 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f58be252:::benchmark_data_smithi179_77138_object85661:head# 2024-02-09T17:26:52.187 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f58c604c:::benchmark_data_smithi179_77138_object90045:head# 2024-02-09T17:26:52.187 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f58d41b1:::benchmark_data_smithi179_77138_object27494:head# 2024-02-09T17:26:52.187 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f58d5058:::benchmark_data_smithi179_77138_object83015:head# 2024-02-09T17:26:52.188 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f58dc534:::benchmark_data_smithi179_77138_object106750:head# 2024-02-09T17:26:52.188 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f58dc5b2:::benchmark_data_smithi179_77138_object51205:head# 2024-02-09T17:26:52.188 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f58e7985:::benchmark_data_smithi179_77138_object57638:head# 2024-02-09T17:26:52.188 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f58eb26b:::benchmark_data_smithi179_77138_object61140:head# 2024-02-09T17:26:52.188 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f58f9664:::benchmark_data_smithi179_77138_object69487:head# 2024-02-09T17:26:52.188 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f58fbe53:::benchmark_data_smithi179_77138_object130553:head# 2024-02-09T17:26:52.189 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5902226:::benchmark_data_smithi179_77138_object23257:head# 2024-02-09T17:26:52.189 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f59059fd:::benchmark_data_smithi179_77138_object37233:head# 2024-02-09T17:26:52.189 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f593171d:::benchmark_data_smithi179_77138_object104924:head# 2024-02-09T17:26:52.189 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f593c145:::benchmark_data_smithi179_77138_object35056:head# 2024-02-09T17:26:52.189 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f594c0d2:::benchmark_data_smithi179_77138_object93560:head# 2024-02-09T17:26:52.190 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f59715fa:::benchmark_data_smithi179_77138_object61628:head# 2024-02-09T17:26:52.190 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f59814ce:::benchmark_data_smithi179_77138_object94172:head# 2024-02-09T17:26:52.191 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f598f2eb:::benchmark_data_smithi179_77138_object103270:head# 2024-02-09T17:26:52.191 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5994f2c:::benchmark_data_smithi179_77138_object65602:head# 2024-02-09T17:26:52.192 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f59a5e89:::benchmark_data_smithi179_77138_object61090:head# 2024-02-09T17:26:52.192 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f59a6e4b:::benchmark_data_smithi179_77138_object81918:head# 2024-02-09T17:26:52.193 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f59ad674:::benchmark_data_smithi179_77138_object71618:head# 2024-02-09T17:26:52.193 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f59af731:::benchmark_data_smithi179_77138_object89521:head# 2024-02-09T17:26:52.194 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f59b69cb:::benchmark_data_smithi179_77138_object132991:head# 2024-02-09T17:26:52.194 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f59b7005:::benchmark_data_smithi179_77138_object43526:head# 2024-02-09T17:26:52.194 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f59bb47b:::benchmark_data_smithi179_77138_object50703:head# 2024-02-09T17:26:52.195 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f59c3776:::benchmark_data_smithi179_77138_object129599:head# 2024-02-09T17:26:52.195 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f59c599e:::benchmark_data_smithi179_77138_object127024:head# 2024-02-09T17:26:52.196 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f59cfddc:::benchmark_data_smithi179_77138_object42843:head# 2024-02-09T17:26:52.196 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f59d2652:::benchmark_data_smithi179_77138_object87607:head# 2024-02-09T17:26:52.197 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f59e49fa:::benchmark_data_smithi179_77138_object110679:head# 2024-02-09T17:26:52.197 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f59e6e98:::benchmark_data_smithi179_77138_object108784:head# 2024-02-09T17:26:52.200 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f59f1188:::benchmark_data_smithi179_77138_object60222:head# 2024-02-09T17:26:52.201 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f59f901f:::benchmark_data_smithi179_77138_object76420:head# 2024-02-09T17:26:52.201 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f59ffae1:::benchmark_data_smithi179_77138_object101152:head# 2024-02-09T17:26:52.201 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5a0c9fa:::benchmark_data_smithi179_77138_object27000:head# 2024-02-09T17:26:52.201 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5a0dab0:::benchmark_data_smithi179_77138_object32906:head# 2024-02-09T17:26:52.201 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5a14569:::benchmark_data_smithi179_77138_object98298:head# 2024-02-09T17:26:52.201 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5a1ba68:::benchmark_data_smithi179_77138_object95841:head# 2024-02-09T17:26:52.201 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5a2a6b4:::benchmark_data_smithi179_77138_object108886:head# 2024-02-09T17:26:52.202 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5a35a55:::benchmark_data_smithi179_77138_object85440:head# 2024-02-09T17:26:52.202 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5a3a0b0:::benchmark_data_smithi179_77138_object25349:head# 2024-02-09T17:26:52.202 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5a3e695:::benchmark_data_smithi179_77138_object123358:head# 2024-02-09T17:26:52.202 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5a3fb3f:::benchmark_data_smithi179_77138_object79493:head# 2024-02-09T17:26:52.203 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5a4afd0:::benchmark_data_smithi179_77138_object92615:head# 2024-02-09T17:26:52.204 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5a5eeaa:::benchmark_data_smithi179_77138_object122306:head# 2024-02-09T17:26:52.204 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5a6e4e5:::benchmark_data_smithi179_77138_object53436:head# 2024-02-09T17:26:52.204 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5a6ebb0:::benchmark_data_smithi179_77138_object119787:head# 2024-02-09T17:26:52.205 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5a88329:::benchmark_data_smithi179_77138_object96800:head# 2024-02-09T17:26:52.205 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5a8a99e:::benchmark_data_smithi179_77138_object90915:head# 2024-02-09T17:26:52.206 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5a9649d:::benchmark_data_smithi179_77138_object72516:head# 2024-02-09T17:26:52.206 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5aa5b54:::benchmark_data_smithi179_77138_object58806:head# 2024-02-09T17:26:52.207 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5aacc09:::benchmark_data_smithi179_77138_object106255:head# 2024-02-09T17:26:52.207 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5aaf385:::benchmark_data_smithi179_77138_object90812:head# 2024-02-09T17:26:52.208 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5abd24b:::benchmark_data_smithi179_77138_object55319:head# 2024-02-09T17:26:52.208 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5ac69a6:::benchmark_data_smithi179_77138_object47615:head# 2024-02-09T17:26:52.213 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5ad058d:::benchmark_data_smithi179_77138_object83019:head# 2024-02-09T17:26:52.213 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5ad2dfa:::benchmark_data_smithi179_77138_object87570:head# 2024-02-09T17:26:52.213 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5addec4:::benchmark_data_smithi179_77138_object80516:head# 2024-02-09T17:26:52.215 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5adfed6:::benchmark_data_smithi179_77138_object124680:head# 2024-02-09T17:26:52.215 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5aea2df:::benchmark_data_smithi179_77138_object103170:head# 2024-02-09T17:26:52.215 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5aeb4c0:::benchmark_data_smithi179_77138_object116857:head# 2024-02-09T17:26:52.216 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5aebea0:::benchmark_data_smithi179_77138_object111049:head# 2024-02-09T17:26:52.216 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5aedaf5:::benchmark_data_smithi179_77138_object44708:head# 2024-02-09T17:26:52.216 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5afd82f:::benchmark_data_smithi179_77138_object84956:head# 2024-02-09T17:26:52.217 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5b0f84e:::benchmark_data_smithi179_77138_object39930:head# 2024-02-09T17:26:52.217 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5b23ca3:::benchmark_data_smithi179_77138_object75235:head# 2024-02-09T17:26:52.217 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5b34e4b:::benchmark_data_smithi179_77138_object70944:head# 2024-02-09T17:26:52.218 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5b400e2:::benchmark_data_smithi179_77138_object76397:head# 2024-02-09T17:26:52.218 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5b44245:::benchmark_data_smithi179_77138_object54119:head# 2024-02-09T17:26:52.219 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5b44637:::benchmark_data_smithi179_77138_object121889:head# 2024-02-09T17:26:52.219 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5b570aa:::benchmark_data_smithi179_77138_object84922:head# 2024-02-09T17:26:52.219 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5b655f5:::benchmark_data_smithi179_77138_object74821:head# 2024-02-09T17:26:52.220 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5b74669:::benchmark_data_smithi179_77138_object112307:head# 2024-02-09T17:26:52.220 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5b77c29:::benchmark_data_smithi179_77138_object66684:head# 2024-02-09T17:26:52.221 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5b7d8bf:::benchmark_data_smithi179_77138_object62894:head# 2024-02-09T17:26:52.221 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5b87977:::benchmark_data_smithi179_77138_object101896:head# 2024-02-09T17:26:52.222 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5b8fa91:::benchmark_data_smithi179_77138_object93096:head# 2024-02-09T17:26:52.222 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5b9d047:::benchmark_data_smithi179_77138_object29479:head# 2024-02-09T17:26:52.223 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5ba2031:::benchmark_data_smithi179_77138_object106748:head# 2024-02-09T17:26:52.224 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5bb2549:::benchmark_data_smithi179_77138_object108044:head# 2024-02-09T17:26:52.224 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5bb2cf5:::benchmark_data_smithi179_77138_object42926:head# 2024-02-09T17:26:52.225 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5bb8f61:::benchmark_data_smithi179_77138_object56622:head# 2024-02-09T17:26:52.225 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5bca03f:::benchmark_data_smithi179_77138_object88074:head# 2024-02-09T17:26:52.226 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5be4800:::benchmark_data_smithi179_77138_object39163:head# 2024-02-09T17:26:52.226 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5be4fee:::benchmark_data_smithi179_77138_object111528:head# 2024-02-09T17:26:52.226 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5be503a:::benchmark_data_smithi179_77138_object45397:head# 2024-02-09T17:26:52.227 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5be5f85:::benchmark_data_smithi179_77138_object109185:head# 2024-02-09T17:26:52.228 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5be9bad:::benchmark_data_smithi179_77138_object99149:head# 2024-02-09T17:26:52.228 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5bea328:::benchmark_data_smithi179_77138_object67708:head# 2024-02-09T17:26:52.229 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5bf9fcc:::benchmark_data_smithi179_77138_object109163:head# 2024-02-09T17:26:52.229 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5c01697:::benchmark_data_smithi179_77138_object84766:head# 2024-02-09T17:26:52.230 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5c081cf:::benchmark_data_smithi179_77138_object23047:head# 2024-02-09T17:26:52.230 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5c16ef7:::benchmark_data_smithi179_77138_object49816:head# 2024-02-09T17:26:52.231 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5c18acd:::benchmark_data_smithi179_77138_object66989:head# 2024-02-09T17:26:52.231 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5c21859:::benchmark_data_smithi179_77138_object39249:head# 2024-02-09T17:26:52.231 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5c2ab08:::benchmark_data_smithi179_77138_object98387:head# 2024-02-09T17:26:52.232 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5c3049e:::benchmark_data_smithi179_77138_object24986:head# 2024-02-09T17:26:52.232 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5c4d9b4:::benchmark_data_smithi179_77138_object108011:head# 2024-02-09T17:26:52.233 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5c4e75d:::benchmark_data_smithi179_77138_object82726:head# 2024-02-09T17:26:52.233 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5c5091b:::benchmark_data_smithi179_77138_object94920:head# 2024-02-09T17:26:52.233 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5c5f421:::benchmark_data_smithi179_77138_object29854:head# 2024-02-09T17:26:52.234 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5c68041:::benchmark_data_smithi179_77138_object39934:head# 2024-02-09T17:26:52.234 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5c6d862:::benchmark_data_smithi179_77138_object103845:head# 2024-02-09T17:26:52.235 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5c6ef13:::benchmark_data_smithi179_77138_object110305:head# 2024-02-09T17:26:52.235 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5c73f91:::benchmark_data_smithi179_77138_object52746:head# 2024-02-09T17:26:52.236 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5c74c5d:::benchmark_data_smithi179_77138_object45729:head# 2024-02-09T17:26:52.236 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5c7d57e:::benchmark_data_smithi179_77138_object37735:head# 2024-02-09T17:26:52.236 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5c7e5e7:::benchmark_data_smithi179_77138_object60872:head# 2024-02-09T17:26:52.237 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5c81cf3:::benchmark_data_smithi179_77138_object34260:head# 2024-02-09T17:26:52.237 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5c866ba:::benchmark_data_smithi179_77138_object58540:head# 2024-02-09T17:26:52.238 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5c96404:::benchmark_data_smithi179_77138_object85423:head# 2024-02-09T17:26:52.238 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5c99988:::benchmark_data_smithi179_77138_object111935:head# 2024-02-09T17:26:52.239 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5ca34cd:::benchmark_data_smithi179_77138_object34721:head# 2024-02-09T17:26:52.239 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5caa720:::benchmark_data_smithi179_77138_object28742:head# 2024-02-09T17:26:52.239 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5cb4bd8:::benchmark_data_smithi179_77138_object83171:head# 2024-02-09T17:26:52.240 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5cb77ab:::benchmark_data_smithi179_77138_object78288:head# 2024-02-09T17:26:52.240 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5cba881:::benchmark_data_smithi179_77138_object48833:head# 2024-02-09T17:26:52.241 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5cbca69:::benchmark_data_smithi179_77138_object67529:head# 2024-02-09T17:26:52.241 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5cc09de:::benchmark_data_smithi179_77138_object70653:head# 2024-02-09T17:26:52.242 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5cda238:::benchmark_data_smithi179_77138_object76754:head# 2024-02-09T17:26:52.242 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5ce315b:::benchmark_data_smithi179_77138_object38747:head# 2024-02-09T17:26:52.243 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5ce4272:::benchmark_data_smithi179_77138_object45116:head# 2024-02-09T17:26:52.243 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5cee6e0:::benchmark_data_smithi179_77138_object131347:head# 2024-02-09T17:26:52.244 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5cfb9a8:::benchmark_data_smithi179_77138_object123838:head# 2024-02-09T17:26:52.244 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5cfe860:::benchmark_data_smithi179_77138_object116255:head# 2024-02-09T17:26:52.244 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5d06632:::benchmark_data_smithi179_77138_object115342:head# 2024-02-09T17:26:52.245 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5d10aa5:::benchmark_data_smithi179_77138_object56691:head# 2024-02-09T17:26:52.246 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5d15d25:::benchmark_data_smithi179_77138_object69447:head# 2024-02-09T17:26:52.250 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5d23eaf:::benchmark_data_smithi179_77138_object77518:head# 2024-02-09T17:26:52.250 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5d24659:::benchmark_data_smithi179_77138_object112479:head# 2024-02-09T17:26:52.250 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5d2c213:::benchmark_data_smithi179_77138_object30531:head# 2024-02-09T17:26:52.250 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5d66c6f:::benchmark_data_smithi179_77138_object98352:head# 2024-02-09T17:26:52.250 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5d6ab5b:::benchmark_data_smithi179_77138_object54542:head# 2024-02-09T17:26:52.250 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5d73e35:::benchmark_data_smithi179_77138_object86466:head# 2024-02-09T17:26:52.251 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5d7a649:::benchmark_data_smithi179_77138_object43577:head# 2024-02-09T17:26:52.251 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5d7cabe:::benchmark_data_smithi179_77138_object90140:head# 2024-02-09T17:26:52.251 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5d80e1d:::benchmark_data_smithi179_77138_object126842:head# 2024-02-09T17:26:52.251 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5d885cc:::benchmark_data_smithi179_77138_object53162:head# 2024-02-09T17:26:52.251 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5d8e347:::benchmark_data_smithi179_77138_object65785:head# 2024-02-09T17:26:52.252 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5d9305f:::benchmark_data_smithi179_77138_object114457:head# 2024-02-09T17:26:52.252 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5da9dbe:::benchmark_data_smithi179_77138_object67961:head# 2024-02-09T17:26:52.253 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5dab7bb:::benchmark_data_smithi179_77138_object43449:head# 2024-02-09T17:26:52.253 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5dbb8b2:::benchmark_data_smithi179_77138_object50856:head# 2024-02-09T17:26:52.254 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5dc0893:::benchmark_data_smithi179_77138_object107469:head# 2024-02-09T17:26:52.254 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5dc269e:::benchmark_data_smithi179_77138_object120210:head# 2024-02-09T17:26:52.255 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5dc3d77:::benchmark_data_smithi179_77138_object20526:head# 2024-02-09T17:26:52.255 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5dd8031:::benchmark_data_smithi179_77138_object76922:head# 2024-02-09T17:26:52.256 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5de7c8a:::benchmark_data_smithi179_77138_object116302:head# 2024-02-09T17:26:52.256 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5dfda5e:::benchmark_data_smithi179_77138_object110878:head# 2024-02-09T17:26:52.257 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5e0661e:::benchmark_data_smithi179_77138_object77704:head# 2024-02-09T17:26:52.257 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5e0a915:::benchmark_data_smithi179_77138_object97270:head# 2024-02-09T17:26:52.257 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5e1234c:::benchmark_data_smithi179_77138_object117345:head# 2024-02-09T17:26:52.258 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5e28d06:::benchmark_data_smithi179_77138_object109462:head# 2024-02-09T17:26:52.259 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5e34691:::benchmark_data_smithi179_77138_object67142:head# 2024-02-09T17:26:52.259 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5e3d1a7:::benchmark_data_smithi179_77138_object53967:head# 2024-02-09T17:26:52.261 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5e480d2:::benchmark_data_smithi179_77138_object114376:head# 2024-02-09T17:26:52.261 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5e50b41:::benchmark_data_smithi179_77138_object73648:head# 2024-02-09T17:26:52.261 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5e5eb9b:::benchmark_data_smithi179_77138_object59830:head# 2024-02-09T17:26:52.261 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5e62545:::benchmark_data_smithi179_77138_object55489:head# 2024-02-09T17:26:52.262 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5e660ca:::benchmark_data_smithi179_77138_object74291:head# 2024-02-09T17:26:52.262 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5e87f1f:::benchmark_data_smithi179_77138_object117005:head# 2024-02-09T17:26:52.262 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5e8fd8f:::benchmark_data_smithi179_77138_object108322:head# 2024-02-09T17:26:52.263 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5e99485:::benchmark_data_smithi179_77138_object44838:head# 2024-02-09T17:26:52.263 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5e9d943:::benchmark_data_smithi179_77138_object69501:head# 2024-02-09T17:26:52.264 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5e9d94d:::benchmark_data_smithi179_77138_object111010:head# 2024-02-09T17:26:52.264 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5eaa183:::benchmark_data_smithi179_77138_object51928:head# 2024-02-09T17:26:52.265 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5eac1ac:::benchmark_data_smithi179_77138_object100132:head# 2024-02-09T17:26:52.266 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5ead4a0:::benchmark_data_smithi179_77138_object33321:head# 2024-02-09T17:26:52.266 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5ead7ce:::benchmark_data_smithi179_77138_object125471:head# 2024-02-09T17:26:52.266 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5ead952:::benchmark_data_smithi179_77138_object24929:head# 2024-02-09T17:26:52.266 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5eb1848:::benchmark_data_smithi179_77138_object106438:head# 2024-02-09T17:26:52.267 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5ebcfa9:::benchmark_data_smithi179_77138_object47298:head# 2024-02-09T17:26:52.267 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5ec6ba0:::benchmark_data_smithi179_77138_object35093:head# 2024-02-09T17:26:52.267 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5eda4fa:::benchmark_data_smithi179_77138_object38535:head# 2024-02-09T17:26:52.268 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5edc73b:::benchmark_data_smithi179_77138_object96920:head# 2024-02-09T17:26:52.268 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5ee4c46:::benchmark_data_smithi179_77138_object62326:head# 2024-02-09T17:26:52.269 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5ee713c:::benchmark_data_smithi179_77138_object63611:head# 2024-02-09T17:26:52.269 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5ef7caf:::benchmark_data_smithi179_77138_object63578:head# 2024-02-09T17:26:52.273 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5f1138e:::benchmark_data_smithi179_77138_object55153:head# 2024-02-09T17:26:52.273 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5f143d5:::benchmark_data_smithi179_77138_object83592:head# 2024-02-09T17:26:52.273 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5f18696:::benchmark_data_smithi179_77138_object74327:head# 2024-02-09T17:26:52.273 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5f1c947:::benchmark_data_smithi179_77138_object35852:head# 2024-02-09T17:26:52.273 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5f2479b:::benchmark_data_smithi179_77138_object33024:head# 2024-02-09T17:26:52.273 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5f27669:::benchmark_data_smithi179_77138_object95490:head# 2024-02-09T17:26:52.274 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5f2769f:::benchmark_data_smithi179_77138_object81214:head# 2024-02-09T17:26:52.275 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5f28e16:::benchmark_data_smithi179_77138_object86382:head# 2024-02-09T17:26:52.275 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5f29b84:::benchmark_data_smithi179_77138_object67405:head# 2024-02-09T17:26:52.275 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5f2b70c:::benchmark_data_smithi179_77138_object52153:head# 2024-02-09T17:26:52.276 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5f33ea3:::benchmark_data_smithi179_77138_object52999:head# 2024-02-09T17:26:52.276 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5f37e29:::benchmark_data_smithi179_77138_object100885:head# 2024-02-09T17:26:52.277 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5f38d61:::benchmark_data_smithi179_77138_object85744:head# 2024-02-09T17:26:52.277 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:26:52.278 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5f41947:::benchmark_data_smithi179_77138_object61141:head# 2024-02-09T17:26:52.278 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5f54ab2:::benchmark_data_smithi179_77138_object72841:head# 2024-02-09T17:26:52.279 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5f56f95:::benchmark_data_smithi179_77138_object119789:head# 2024-02-09T17:26:52.283 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5f617ae:::benchmark_data_smithi179_77138_object67759:head# 2024-02-09T17:26:52.283 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5f68f4a:::benchmark_data_smithi179_77138_object110181:head# 2024-02-09T17:26:52.283 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5f6b809:::benchmark_data_smithi179_77138_object60787:head# 2024-02-09T17:26:52.283 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5f6ea3a:::benchmark_data_smithi179_77138_object35925:head# 2024-02-09T17:26:52.283 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5f7103f:::benchmark_data_smithi179_77138_object111739:head# 2024-02-09T17:26:52.283 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5f7bbac:::benchmark_data_smithi179_77138_object62053:head# 2024-02-09T17:26:52.283 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5f7ef84:::benchmark_data_smithi179_77138_object126047:head# 2024-02-09T17:26:52.284 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5f810ba:::benchmark_data_smithi179_77138_object98025:head# 2024-02-09T17:26:52.284 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5f825cd:::benchmark_data_smithi179_77138_object93072:head# 2024-02-09T17:26:52.284 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5f94dd5:::benchmark_data_smithi179_77138_object70076:head# 2024-02-09T17:26:52.284 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5f9e61d:::benchmark_data_smithi179_77138_object96344:head# 2024-02-09T17:26:52.284 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5fa4818:::benchmark_data_smithi179_77138_object34631:head# 2024-02-09T17:26:52.287 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5fac09d:::benchmark_data_smithi179_77138_object125655:head# 2024-02-09T17:26:52.287 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5face22:::benchmark_data_smithi179_77138_object50496:head# 2024-02-09T17:26:52.287 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5fd5807:::benchmark_data_smithi179_77138_object109589:head# 2024-02-09T17:26:52.288 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5fd6979:::benchmark_data_smithi179_77138_object59534:head# 2024-02-09T17:26:52.288 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5fda0cf:::benchmark_data_smithi179_77138_object69385:head# 2024-02-09T17:26:52.289 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5fdc248:::benchmark_data_smithi179_77138_object108247:head# 2024-02-09T17:26:52.289 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f5feb1de:::benchmark_data_smithi179_77138_object101662:head# 2024-02-09T17:26:52.290 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f600cf9b:::benchmark_data_smithi179_77138_object90890:head# 2024-02-09T17:26:52.290 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6010d00:::benchmark_data_smithi179_77138_object81901:head# 2024-02-09T17:26:52.291 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f601e225:::benchmark_data_smithi179_77138_object79575:head# 2024-02-09T17:26:52.292 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f602b943:::benchmark_data_smithi179_77138_object72787:head# 2024-02-09T17:26:52.292 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f604d5f0:::benchmark_data_smithi179_77138_object33112:head# 2024-02-09T17:26:52.292 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6064b01:::benchmark_data_smithi179_77138_object133036:head# 2024-02-09T17:26:52.292 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f60673d3:::benchmark_data_smithi179_77138_object79706:head# 2024-02-09T17:26:52.293 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f60694b9:::benchmark_data_smithi179_77138_object96363:head# 2024-02-09T17:26:52.293 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f606c8ae:::benchmark_data_smithi179_77138_object59508:head# 2024-02-09T17:26:52.294 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f606ced8:::benchmark_data_smithi179_77138_object131578:head# 2024-02-09T17:26:52.294 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6072d6b:::benchmark_data_smithi179_77138_object57162:head# 2024-02-09T17:26:52.294 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6083ef8:::benchmark_data_smithi179_77138_object111550:head# 2024-02-09T17:26:52.295 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6089a33:::benchmark_data_smithi179_77138_object81751:head# 2024-02-09T17:26:52.295 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f608becf:::benchmark_data_smithi179_77138_object37126:head# 2024-02-09T17:26:52.296 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6099185:::benchmark_data_smithi179_77138_object46054:head# 2024-02-09T17:26:52.296 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f609c370:::benchmark_data_smithi179_77138_object87047:head# 2024-02-09T17:26:52.297 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f60a1288:::benchmark_data_smithi179_77138_object82063:head# 2024-02-09T17:26:52.297 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f60a38a7:::benchmark_data_smithi179_77138_object91211:head# 2024-02-09T17:26:52.297 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f60a70ba:::benchmark_data_smithi179_77138_object43664:head# 2024-02-09T17:26:52.298 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f60b0b08:::benchmark_data_smithi179_77138_object29708:head# 2024-02-09T17:26:52.298 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f60b328a:::benchmark_data_smithi179_77138_object87907:head# 2024-02-09T17:26:52.299 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f60b8450:::benchmark_data_smithi179_77138_object114961:head# 2024-02-09T17:26:52.299 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f60bc1fb:::benchmark_data_smithi179_77138_object66250:head# 2024-02-09T17:26:52.300 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f60c1e0d:::benchmark_data_smithi179_77138_object109369:head# 2024-02-09T17:26:52.300 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f60c9e1c:::benchmark_data_smithi179_77138_object24988:head# 2024-02-09T17:26:52.301 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f60d9996:::benchmark_data_smithi179_77138_object79390:head# 2024-02-09T17:26:52.303 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f60d9af4:::benchmark_data_smithi179_77138_object99141:head# 2024-02-09T17:26:52.303 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f60df039:::benchmark_data_smithi179_77138_object86654:head# 2024-02-09T17:26:52.303 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f60e1c27:::benchmark_data_smithi179_77138_object85754:head# 2024-02-09T17:26:52.304 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f60e663b:::benchmark_data_smithi179_77138_object62579:head# 2024-02-09T17:26:52.304 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f60e7133:::benchmark_data_smithi179_77138_object120015:head# 2024-02-09T17:26:52.305 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f60ef9aa:::benchmark_data_smithi179_77138_object77238:head# 2024-02-09T17:26:52.305 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f610603f:::benchmark_data_smithi179_77138_object26219:head# 2024-02-09T17:26:52.305 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6107058:::benchmark_data_smithi179_77138_object28755:head# 2024-02-09T17:26:52.306 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f610c02c:::benchmark_data_smithi179_77138_object22358:head# 2024-02-09T17:26:52.306 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f610d0b1:::benchmark_data_smithi179_77138_object122298:head# 2024-02-09T17:26:52.307 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f61116eb:::benchmark_data_smithi179_77138_object129197:head# 2024-02-09T17:26:52.307 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6145b0f:::benchmark_data_smithi179_77138_object90444:head# 2024-02-09T17:26:52.307 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6153f55:::benchmark_data_smithi179_77138_object30618:head# 2024-02-09T17:26:52.308 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f615be04:::benchmark_data_smithi179_77138_object60288:head# 2024-02-09T17:26:52.308 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6169b4d:::benchmark_data_smithi179_77138_object53616:head# 2024-02-09T17:26:52.309 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f616be7c:::benchmark_data_smithi179_77138_object121949:head# 2024-02-09T17:26:52.309 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f616dc2c:::benchmark_data_smithi179_77138_object132853:head# 2024-02-09T17:26:52.310 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6184f7f:::benchmark_data_smithi179_77138_object62325:head# 2024-02-09T17:26:52.310 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f61acb79:::benchmark_data_smithi179_77138_object42726:head# 2024-02-09T17:26:52.314 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f61ae79f:::benchmark_data_smithi179_77138_object98794:head# 2024-02-09T17:26:52.314 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f61b6a7b:::benchmark_data_smithi179_77138_object38431:head# 2024-02-09T17:26:52.314 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f61baddc:::benchmark_data_smithi179_77138_object126281:head# 2024-02-09T17:26:52.314 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f61bbda3:::benchmark_data_smithi179_77138_object52276:head# 2024-02-09T17:26:52.315 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f61bc66d:::benchmark_data_smithi179_77138_object22399:head# 2024-02-09T17:26:52.315 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f61be9d9:::benchmark_data_smithi179_77138_object93702:head# 2024-02-09T17:26:52.315 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f61c5f79:::benchmark_data_smithi179_77138_object39655:head# 2024-02-09T17:26:52.315 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f61c9207:::benchmark_data_smithi179_77138_object114299:head# 2024-02-09T17:26:52.317 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f61d33e7:::benchmark_data_smithi179_77138_object69786:head# 2024-02-09T17:26:52.317 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f61f0a35:::benchmark_data_smithi179_77138_object42847:head# 2024-02-09T17:26:52.318 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f61f3352:::benchmark_data_smithi179_77138_object123229:head# 2024-02-09T17:26:52.318 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6217295:::benchmark_data_smithi179_77138_object84743:head# 2024-02-09T17:26:52.321 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6225638:::benchmark_data_smithi179_77138_object101621:head# 2024-02-09T17:26:52.321 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6226b2b:::benchmark_data_smithi179_77138_object58727:head# 2024-02-09T17:26:52.322 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6238141:::benchmark_data_smithi179_77138_object110276:head# 2024-02-09T17:26:52.323 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f623cc20:::benchmark_data_smithi179_77138_object51075:head# 2024-02-09T17:26:52.325 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f623e54a:::benchmark_data_smithi179_77138_object126955:head# 2024-02-09T17:26:52.325 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6252b2f:::benchmark_data_smithi179_77138_object121861:head# 2024-02-09T17:26:52.325 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f62544e0:::benchmark_data_smithi179_77138_object44891:head# 2024-02-09T17:26:52.325 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6254eb2:::benchmark_data_smithi179_77138_object129647:head# 2024-02-09T17:26:52.326 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6258c69:::benchmark_data_smithi179_77138_object75141:head# 2024-02-09T17:26:52.326 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f626d589:::benchmark_data_smithi179_77138_object34895:head# 2024-02-09T17:26:52.326 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f626ee27:::benchmark_data_smithi179_77138_object122395:head# 2024-02-09T17:26:52.327 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f62700fd:::benchmark_data_smithi179_77138_object74894:head# 2024-02-09T17:26:52.327 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6283193:::benchmark_data_smithi179_77138_object70797:head# 2024-02-09T17:26:52.327 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6283f2b:::benchmark_data_smithi179_77138_object25783:head# 2024-02-09T17:26:52.328 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f62893a6:::benchmark_data_smithi179_77138_object41890:head# 2024-02-09T17:26:52.328 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f628c22b:::benchmark_data_smithi179_77138_object87828:head# 2024-02-09T17:26:52.329 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f62a0ee5:::benchmark_data_smithi179_77138_object67964:head# 2024-02-09T17:26:52.329 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f62a2e2c:::benchmark_data_smithi179_77138_object48072:head# 2024-02-09T17:26:52.330 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f62aa987:::benchmark_data_smithi179_77138_object92151:head# 2024-02-09T17:26:52.330 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f62ae5de:::benchmark_data_smithi179_77138_object41251:head# 2024-02-09T17:26:52.331 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f62d1512:::benchmark_data_smithi179_77138_object74881:head# 2024-02-09T17:26:52.331 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f62d9f08:::benchmark_data_smithi179_77138_object75766:head# 2024-02-09T17:26:52.332 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f62e3480:::benchmark_data_smithi179_77138_object81673:head# 2024-02-09T17:26:52.332 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f62e5327:::benchmark_data_smithi179_77138_object71634:head# 2024-02-09T17:26:52.332 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f631221b:::benchmark_data_smithi179_77138_object67423:head# 2024-02-09T17:26:52.333 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f631d95a:::benchmark_data_smithi179_77138_object90008:head# 2024-02-09T17:26:52.333 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f632bdb1:::benchmark_data_smithi179_77138_object109832:head# 2024-02-09T17:26:52.334 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f632fdee:::benchmark_data_smithi179_77138_object103776:head# 2024-02-09T17:26:52.334 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6330e8e:::benchmark_data_smithi179_77138_object65376:head# 2024-02-09T17:26:52.335 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f633245a:::benchmark_data_smithi179_77138_object30032:head# 2024-02-09T17:26:52.335 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f63476e7:::benchmark_data_smithi179_77138_object26928:head# 2024-02-09T17:26:52.336 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f63504f6:::benchmark_data_smithi179_77138_object73826:head# 2024-02-09T17:26:52.336 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f635b995:::benchmark_data_smithi179_77138_object103685:head# 2024-02-09T17:26:52.336 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6363373:::benchmark_data_smithi179_77138_object79279:head# 2024-02-09T17:26:52.337 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6364915:::benchmark_data_smithi179_77138_object87646:head# 2024-02-09T17:26:52.337 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6385e2b:::benchmark_data_smithi179_77138_object32464:head# 2024-02-09T17:26:52.338 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f638a549:::benchmark_data_smithi179_77138_object129370:head# 2024-02-09T17:26:52.340 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f638c116:::benchmark_data_smithi179_77138_object74775:head# 2024-02-09T17:26:52.340 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f638f555:::benchmark_data_smithi179_77138_object47095:head# 2024-02-09T17:26:52.340 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6397788:::benchmark_data_smithi179_77138_object51634:head# 2024-02-09T17:26:52.341 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f63a3051:::benchmark_data_smithi179_77138_object74348:head# 2024-02-09T17:26:52.342 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f63ad25b:::benchmark_data_smithi179_77138_object89714:head# 2024-02-09T17:26:52.342 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f63b1de6:::benchmark_data_smithi179_77138_object27721:head# 2024-02-09T17:26:52.343 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f63b5204:::benchmark_data_smithi179_77138_object99090:head# 2024-02-09T17:26:52.348 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f63bc365:::benchmark_data_smithi179_77138_object51377:head# 2024-02-09T17:26:52.348 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f63d060e:::benchmark_data_smithi179_77138_object117779:head# 2024-02-09T17:26:52.349 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f63e38a2:::benchmark_data_smithi179_77138_object48418:head# 2024-02-09T17:26:52.349 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f63e477d:::benchmark_data_smithi179_77138_object127462:head# 2024-02-09T17:26:52.350 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f63e9de5:::benchmark_data_smithi179_77138_object35647:head# 2024-02-09T17:26:52.350 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f63ea381:::benchmark_data_smithi179_77138_object98658:head# 2024-02-09T17:26:52.351 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f63ec06b:::benchmark_data_smithi179_77138_object88824:head# 2024-02-09T17:26:52.351 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f63f11f5:::benchmark_data_smithi179_77138_object130224:head# 2024-02-09T17:26:52.352 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f63f15dc:::benchmark_data_smithi179_77138_object113368:head# 2024-02-09T17:26:52.352 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f63f5fbe:::benchmark_data_smithi179_77138_object83848:head# 2024-02-09T17:26:52.354 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f63f7620:::benchmark_data_smithi179_77138_object71419:head# 2024-02-09T17:26:52.355 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f63fcb7f:::benchmark_data_smithi179_77138_object66674:head# 2024-02-09T17:26:52.357 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f63fd7d5:::benchmark_data_smithi179_77138_object43963:head# 2024-02-09T17:26:52.357 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f640b480:::benchmark_data_smithi179_77138_object89308:head# 2024-02-09T17:26:52.357 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f641884d:::benchmark_data_smithi179_77138_object118356:head# 2024-02-09T17:26:52.357 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6422b8c:::benchmark_data_smithi179_77138_object74305:head# 2024-02-09T17:26:52.358 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f642de58:::benchmark_data_smithi179_77138_object44686:head# 2024-02-09T17:26:52.358 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6432492:::benchmark_data_smithi179_77138_object20109:head# 2024-02-09T17:26:52.358 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f64451e0:::benchmark_data_smithi179_77138_object82643:head# 2024-02-09T17:26:52.358 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6447f02:::benchmark_data_smithi179_77138_object43125:head# 2024-02-09T17:26:52.359 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f644b070:::benchmark_data_smithi179_77138_object75583:head# 2024-02-09T17:26:52.359 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6450cd0:::benchmark_data_smithi179_77138_object117119:head# 2024-02-09T17:26:52.360 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f645a6e3:::benchmark_data_smithi179_77138_object47624:head# 2024-02-09T17:26:52.360 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f645e1da:::benchmark_data_smithi179_77138_object58956:head# 2024-02-09T17:26:52.366 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6465cfd:::benchmark_data_smithi179_77138_object123571:head# 2024-02-09T17:26:52.366 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6470711:::benchmark_data_smithi179_77138_object45853:head# 2024-02-09T17:26:52.366 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f647d14c:::benchmark_data_smithi179_77138_object32385:head# 2024-02-09T17:26:52.366 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f647db37:::benchmark_data_smithi179_77138_object21413:head# 2024-02-09T17:26:52.366 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6480d4c:::benchmark_data_smithi179_77138_object124270:head# 2024-02-09T17:26:52.366 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6482d9b:::benchmark_data_smithi179_77138_object78098:head# 2024-02-09T17:26:52.366 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f648929e:::benchmark_data_smithi179_77138_object77156:head# 2024-02-09T17:26:52.366 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f648a3aa:::benchmark_data_smithi179_77138_object49230:head# 2024-02-09T17:26:52.366 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f649a1a9:::benchmark_data_smithi179_77138_object39703:head# 2024-02-09T17:26:52.367 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f649fed1:::benchmark_data_smithi179_77138_object96390:head# 2024-02-09T17:26:52.367 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f64a394a:::benchmark_data_smithi179_77138_object130953:head# 2024-02-09T17:26:52.367 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f64b9451:::benchmark_data_smithi179_77138_object112311:head# 2024-02-09T17:26:52.367 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f64bbf60:::benchmark_data_smithi179_77138_object56206:head# 2024-02-09T17:26:52.368 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f64bdcbe:::benchmark_data_smithi179_77138_object127017:head# 2024-02-09T17:26:52.368 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f64bf4fa:::benchmark_data_smithi179_77138_object96194:head# 2024-02-09T17:26:52.368 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f64c12ea:::benchmark_data_smithi179_77138_object94115:head# 2024-02-09T17:26:52.369 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f64d3294:::benchmark_data_smithi179_77138_object90412:head# 2024-02-09T17:26:52.369 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f64d5384:::benchmark_data_smithi179_77138_object33601:head# 2024-02-09T17:26:52.370 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f64e3ded:::benchmark_data_smithi179_77138_object37325:head# 2024-02-09T17:26:52.370 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f64e6d7d:::benchmark_data_smithi179_77138_object51880:head# 2024-02-09T17:26:52.371 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f64f6be0:::benchmark_data_smithi179_77138_object72866:head# 2024-02-09T17:26:52.371 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f64fe7a6:::benchmark_data_smithi179_77138_object73869:head# 2024-02-09T17:26:52.371 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6501da4:::benchmark_data_smithi179_77138_object63337:head# 2024-02-09T17:26:52.372 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6504afe:::benchmark_data_smithi179_77138_object91761:head# 2024-02-09T17:26:52.372 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6511e7f:::benchmark_data_smithi179_77138_object56041:head# 2024-02-09T17:26:52.373 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6529dba:::benchmark_data_smithi179_77138_object90243:head# 2024-02-09T17:26:52.373 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6535183:::benchmark_data_smithi179_77138_object84836:head# 2024-02-09T17:26:52.374 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f653ff6b:::benchmark_data_smithi179_77138_object113175:head# 2024-02-09T17:26:52.374 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6546e29:::benchmark_data_smithi179_77138_object64618:head# 2024-02-09T17:26:52.374 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f654c259:::benchmark_data_smithi179_77138_object106654:head# 2024-02-09T17:26:52.375 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f654ce77:::benchmark_data_smithi179_77138_object26570:head# 2024-02-09T17:26:52.375 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f65516b7:::benchmark_data_smithi179_77138_object38445:head# 2024-02-09T17:26:52.376 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f65624e1:::benchmark_data_smithi179_77138_object32067:head# 2024-02-09T17:26:52.376 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f65638ea:::benchmark_data_smithi179_77138_object109879:head# 2024-02-09T17:26:52.376 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f65742f7:::benchmark_data_smithi179_77138_object43617:head# 2024-02-09T17:26:52.377 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f657a00b:::benchmark_data_smithi179_77138_object65028:head# 2024-02-09T17:26:52.377 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6582124:::benchmark_data_smithi179_77138_object59026:head# 2024-02-09T17:26:52.378 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f658ceca:::benchmark_data_smithi179_77138_object89357:head# 2024-02-09T17:26:52.378 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f65b4f33:::benchmark_data_smithi179_77138_object74645:head# 2024-02-09T17:26:52.379 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f65c3a44:::benchmark_data_smithi179_77138_object93656:head# 2024-02-09T17:26:52.379 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f65c50a1:::benchmark_data_smithi179_77138_object20105:head# 2024-02-09T17:26:52.379 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f65ca0c7:::benchmark_data_smithi179_77138_object91096:head# 2024-02-09T17:26:52.383 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f65cfc47:::benchmark_data_smithi179_77138_object67231:head# 2024-02-09T17:26:52.383 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f65d2903:::benchmark_data_smithi179_77138_object33528:head# 2024-02-09T17:26:52.383 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f65de23d:::benchmark_data_smithi179_77138_object111305:head# 2024-02-09T17:26:52.383 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f65ef744:::benchmark_data_smithi179_77138_object109510:head# 2024-02-09T17:26:52.383 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f65f4906:::benchmark_data_smithi179_77138_object85137:head# 2024-02-09T17:26:52.384 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f65f7a11:::benchmark_data_smithi179_77138_object45861:head# 2024-02-09T17:26:52.384 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f65fbc08:::benchmark_data_smithi179_77138_object50907:head# 2024-02-09T17:26:52.384 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6615c4c:::benchmark_data_smithi179_77138_object72293:head# 2024-02-09T17:26:52.385 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6616d07:::benchmark_data_smithi179_77138_object31557:head# 2024-02-09T17:26:52.385 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f661e370:::benchmark_data_smithi179_77138_object106402:head# 2024-02-09T17:26:52.386 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f662523e:::benchmark_data_smithi179_77138_object122363:head# 2024-02-09T17:26:52.386 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f662cd68:::benchmark_data_smithi179_77138_object110546:head# 2024-02-09T17:26:52.388 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f66462b6:::benchmark_data_smithi179_77138_object24341:head# 2024-02-09T17:26:52.388 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f664c5ee:::benchmark_data_smithi179_77138_object107143:head# 2024-02-09T17:26:52.389 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f664d86d:::benchmark_data_smithi179_77138_object127055:head# 2024-02-09T17:26:52.389 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f664f513:::benchmark_data_smithi179_77138_object38048:head# 2024-02-09T17:26:52.391 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f66557b8:::benchmark_data_smithi179_77138_object133326:head# 2024-02-09T17:26:52.391 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f665650b:::benchmark_data_smithi179_77138_object110693:head# 2024-02-09T17:26:52.391 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f66565c2:::benchmark_data_smithi179_77138_object19419:head# 2024-02-09T17:26:52.391 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f665b823:::benchmark_data_smithi179_77138_object40752:head# 2024-02-09T17:26:52.392 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6661b79:::benchmark_data_smithi179_77138_object120551:head# 2024-02-09T17:26:52.392 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f66651ab:::benchmark_data_smithi179_77138_object39661:head# 2024-02-09T17:26:52.392 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6669920:::benchmark_data_smithi179_77138_object46625:head# 2024-02-09T17:26:52.393 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6678384:::benchmark_data_smithi179_77138_object125097:head# 2024-02-09T17:26:52.393 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6684309:::benchmark_data_smithi179_77138_object47982:head# 2024-02-09T17:26:52.394 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f668a05c:::benchmark_data_smithi179_77138_object72295:head# 2024-02-09T17:26:52.394 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f668b957:::benchmark_data_smithi179_77138_object46821:head# 2024-02-09T17:26:52.395 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f668d9e6:::benchmark_data_smithi179_77138_object41021:head# 2024-02-09T17:26:52.395 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6696cc8:::benchmark_data_smithi179_77138_object108377:head# 2024-02-09T17:26:52.395 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f669e3f1:::benchmark_data_smithi179_77138_object61211:head# 2024-02-09T17:26:52.396 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f66af516:::benchmark_data_smithi179_77138_object121974:head# 2024-02-09T17:26:52.396 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f66b1592:::benchmark_data_smithi179_77138_object51061:head# 2024-02-09T17:26:52.397 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f66b2d3c:::benchmark_data_smithi179_77138_object88184:head# 2024-02-09T17:26:52.397 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f66b82db:::benchmark_data_smithi179_77138_object29317:head# 2024-02-09T17:26:52.398 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f66b9edf:::benchmark_data_smithi179_77138_object92846:head# 2024-02-09T17:26:52.398 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f66c7fce:::benchmark_data_smithi179_77138_object54094:head# 2024-02-09T17:26:52.399 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f66c973d:::benchmark_data_smithi179_77138_object111213:head# 2024-02-09T17:26:52.402 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f66d23dd:::benchmark_data_smithi179_77138_object105575:head# 2024-02-09T17:26:52.402 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f66d9457:::benchmark_data_smithi179_77138_object39856:head# 2024-02-09T17:26:52.403 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f66da3c7:::benchmark_data_smithi179_77138_object94386:head# 2024-02-09T17:26:52.404 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f66dd637:::benchmark_data_smithi179_77138_object64098:head# 2024-02-09T17:26:52.404 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f66df584:::benchmark_data_smithi179_77138_object75702:head# 2024-02-09T17:26:52.404 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f66e10e9:::benchmark_data_smithi179_77138_object32987:head# 2024-02-09T17:26:52.405 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f66e21f0:::benchmark_data_smithi179_77138_object92798:head# 2024-02-09T17:26:52.410 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f66ec498:::benchmark_data_smithi179_77138_object115596:head# 2024-02-09T17:26:52.410 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6703a50:::benchmark_data_smithi179_77138_object84869:head# 2024-02-09T17:26:52.410 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6708c37:::benchmark_data_smithi179_77138_object132037:head# 2024-02-09T17:26:52.410 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f671345a:::benchmark_data_smithi179_77138_object81030:head# 2024-02-09T17:26:52.410 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f671c4b9:::benchmark_data_smithi179_77138_object93712:head# 2024-02-09T17:26:52.410 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6726315:::benchmark_data_smithi179_77138_object126964:head# 2024-02-09T17:26:52.410 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f672f70d:::benchmark_data_smithi179_77138_object104317:head# 2024-02-09T17:26:52.410 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f673194f:::benchmark_data_smithi179_77138_object104600:head# 2024-02-09T17:26:52.410 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6736df6:::benchmark_data_smithi179_77138_object74130:head# 2024-02-09T17:26:52.411 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f674ceaa:::benchmark_data_smithi179_77138_object29211:head# 2024-02-09T17:26:52.411 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6756ac2:::benchmark_data_smithi179_77138_object87795:head# 2024-02-09T17:26:52.411 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f675dab3:::benchmark_data_smithi179_77138_object26204:head# 2024-02-09T17:26:52.411 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f676fd3b:::benchmark_data_smithi179_77138_object67062:head# 2024-02-09T17:26:52.412 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f67a07f9:::benchmark_data_smithi179_77138_object37365:head# 2024-02-09T17:26:52.412 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f67a565d:::benchmark_data_smithi179_77138_object89695:head# 2024-02-09T17:26:52.413 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f67a885e:::benchmark_data_smithi179_77138_object26955:head# 2024-02-09T17:26:52.413 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f67b1279:::benchmark_data_smithi179_77138_object59858:head# 2024-02-09T17:26:52.413 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f67bb917:::benchmark_data_smithi179_77138_object86450:head# 2024-02-09T17:26:52.414 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f67bb919:::benchmark_data_smithi179_77138_object25339:head# 2024-02-09T17:26:52.414 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f67c5638:::benchmark_data_smithi179_77138_object52544:head# 2024-02-09T17:26:52.415 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f67cf93a:::benchmark_data_smithi179_77138_object74701:head# 2024-02-09T17:26:52.415 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f67d89db:::benchmark_data_smithi179_77138_object74244:head# 2024-02-09T17:26:52.415 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f67e1b5d:::benchmark_data_smithi179_77138_object72690:head# 2024-02-09T17:26:52.416 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f67f0f00:::benchmark_data_smithi179_77138_object117836:head# 2024-02-09T17:26:52.416 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f67fcb9a:::benchmark_data_smithi179_77138_object127975:head# 2024-02-09T17:26:52.417 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f67ff78f:::benchmark_data_smithi179_77138_object101892:head# 2024-02-09T17:26:52.417 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f680bc77:::benchmark_data_smithi179_77138_object21892:head# 2024-02-09T17:26:52.418 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6827ceb:::benchmark_data_smithi179_77138_object29802:head# 2024-02-09T17:26:52.418 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f682f6a4:::benchmark_data_smithi179_77138_object55363:head# 2024-02-09T17:26:52.418 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6840169:::benchmark_data_smithi179_77138_object79514:head# 2024-02-09T17:26:52.419 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6842688:::benchmark_data_smithi179_77138_object96981:head# 2024-02-09T17:26:52.419 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6849d1e:::benchmark_data_smithi179_77138_object58969:head# 2024-02-09T17:26:52.420 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6863911:::benchmark_data_smithi179_77138_object119556:head# 2024-02-09T17:26:52.420 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6881ebb:::benchmark_data_smithi179_77138_object97463:head# 2024-02-09T17:26:52.421 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f689ef6a:::benchmark_data_smithi179_77138_object59270:head# 2024-02-09T17:26:52.421 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f68a74b7:::benchmark_data_smithi179_77138_object123121:head# 2024-02-09T17:26:52.422 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f68b0584:::benchmark_data_smithi179_77138_object81345:head# 2024-02-09T17:26:52.422 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f68b0d58:::benchmark_data_smithi179_77138_object118054:head# 2024-02-09T17:26:52.423 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f68b2f10:::benchmark_data_smithi179_77138_object61868:head# 2024-02-09T17:26:52.423 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f68b3068:::benchmark_data_smithi179_77138_object32753:head# 2024-02-09T17:26:52.423 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f68c91ec:::benchmark_data_smithi179_77138_object39605:head# 2024-02-09T17:26:52.424 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f68cee8d:::benchmark_data_smithi179_77138_object40295:head# 2024-02-09T17:26:52.424 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f68e8b94:::benchmark_data_smithi179_77138_object66040:head# 2024-02-09T17:26:52.425 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f68f97db:::benchmark_data_smithi179_77138_object43478:head# 2024-02-09T17:26:52.425 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6901323:::benchmark_data_smithi179_77138_object131275:head# 2024-02-09T17:26:52.425 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6901d8f:::benchmark_data_smithi179_77138_object51440:head# 2024-02-09T17:26:52.426 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f69248e6:::benchmark_data_smithi179_77138_object37376:head# 2024-02-09T17:26:52.431 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f69249b9:::benchmark_data_smithi179_77138_object34646:head# 2024-02-09T17:26:52.431 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6926b4d:::benchmark_data_smithi179_77138_object92315:head# 2024-02-09T17:26:52.432 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6937f62:::benchmark_data_smithi179_77138_object60313:head# 2024-02-09T17:26:52.433 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6941086:::benchmark_data_smithi179_77138_object116925:head# 2024-02-09T17:26:52.433 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f69469cd:::benchmark_data_smithi179_77138_object53614:head# 2024-02-09T17:26:52.434 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6948118:::benchmark_data_smithi179_77138_object105686:head# 2024-02-09T17:26:52.434 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6958350:::benchmark_data_smithi179_77138_object99758:head# 2024-02-09T17:26:52.434 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f696e38c:::benchmark_data_smithi179_77138_object75549:head# 2024-02-09T17:26:52.435 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f697f0b2:::benchmark_data_smithi179_77138_object99625:head# 2024-02-09T17:26:52.435 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f698714a:::benchmark_data_smithi179_77138_object90843:head# 2024-02-09T17:26:52.436 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f698e15e:::benchmark_data_smithi179_77138_object118489:head# 2024-02-09T17:26:52.436 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f698f3ae:::benchmark_data_smithi179_77138_object122965:head# 2024-02-09T17:26:52.437 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f699762a:::benchmark_data_smithi179_77138_object123306:head# 2024-02-09T17:26:52.438 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f699a357:::benchmark_data_smithi179_77138_object116554:head# 2024-02-09T17:26:52.438 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f699d655:::benchmark_data_smithi179_77138_object53010:head# 2024-02-09T17:26:52.439 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f69cc344:::benchmark_data_smithi179_77138_object98243:head# 2024-02-09T17:26:52.439 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f69eb156:::benchmark_data_smithi179_77138_object96185:head# 2024-02-09T17:26:52.440 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f69f31d9:::benchmark_data_smithi179_77138_object124008:head# 2024-02-09T17:26:52.440 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6a094a3:::benchmark_data_smithi179_77138_object105002:head# 2024-02-09T17:26:52.440 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6a2520e:::benchmark_data_smithi179_77138_object96003:head# 2024-02-09T17:26:52.441 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6a25437:::benchmark_data_smithi179_77138_object92338:head# 2024-02-09T17:26:52.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:52.442 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6a28d99:::benchmark_data_smithi179_77138_object25507:head# 2024-02-09T17:26:52.442 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6a28f86:::benchmark_data_smithi179_77138_object96840:head# 2024-02-09T17:26:52.443 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:26:52.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:26:52.443 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6a32c77:::benchmark_data_smithi179_77138_object127678:head# 2024-02-09T17:26:52.444 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6a440e2:::benchmark_data_smithi179_77138_object129416:head# 2024-02-09T17:26:52.444 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6a71175:::benchmark_data_smithi179_77138_object32222:head# 2024-02-09T17:26:52.444 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6a72e02:::benchmark_data_smithi179_77138_object80049:head# 2024-02-09T17:26:52.444 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6a83edc:::benchmark_data_smithi179_77138_object20818:head# 2024-02-09T17:26:52.445 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6a8433f:::benchmark_data_smithi179_77138_object88470:head# 2024-02-09T17:26:52.445 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6a87446:::benchmark_data_smithi179_77138_object113836:head# 2024-02-09T17:26:52.446 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6a87f14:::benchmark_data_smithi179_77138_object115804:head# 2024-02-09T17:26:52.446 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6a8c93f:::benchmark_data_smithi179_77138_object115618:head# 2024-02-09T17:26:52.446 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6a90a1c:::benchmark_data_smithi179_77138_object24247:head# 2024-02-09T17:26:52.447 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6a93b53:::benchmark_data_smithi179_77138_object20642:head# 2024-02-09T17:26:52.447 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6a9fab1:::benchmark_data_smithi179_77138_object40863:head# 2024-02-09T17:26:52.448 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6aa6458:::benchmark_data_smithi179_77138_object47001:head# 2024-02-09T17:26:52.448 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6aa79cc:::benchmark_data_smithi179_77138_object70162:head# 2024-02-09T17:26:52.448 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6aa8263:::benchmark_data_smithi179_77138_object123232:head# 2024-02-09T17:26:52.449 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6aab50f:::benchmark_data_smithi179_77138_object117809:head# 2024-02-09T17:26:52.449 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6ab2b94:::benchmark_data_smithi179_77138_object31854:head# 2024-02-09T17:26:52.450 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6abc42d:::benchmark_data_smithi179_77138_object102640:head# 2024-02-09T17:26:52.450 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6ad4c5a:::benchmark_data_smithi179_77138_object124415:head# 2024-02-09T17:26:52.451 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6ad4e13:::benchmark_data_smithi179_77138_object54172:head# 2024-02-09T17:26:52.451 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6ada498:::benchmark_data_smithi179_77138_object63671:head# 2024-02-09T17:26:52.451 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6ae1763:::benchmark_data_smithi179_77138_object96278:head# 2024-02-09T17:26:52.452 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6ae7ba5:::benchmark_data_smithi179_77138_object99382:head# 2024-02-09T17:26:52.453 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6aec981:::benchmark_data_smithi179_77138_object31437:head# 2024-02-09T17:26:52.453 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6af77c5:::benchmark_data_smithi179_77138_object74157:head# 2024-02-09T17:26:52.454 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6af824d:::benchmark_data_smithi179_77138_object46610:head# 2024-02-09T17:26:52.454 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6affd8c:::benchmark_data_smithi179_77138_object79819:head# 2024-02-09T17:26:52.454 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6b16b1c:::benchmark_data_smithi179_77138_object74712:head# 2024-02-09T17:26:52.455 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6b181f5:::benchmark_data_smithi179_77138_object84781:head# 2024-02-09T17:26:52.455 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6b19c0d:::benchmark_data_smithi179_77138_object70670:head# 2024-02-09T17:26:52.456 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6b32dc8:::benchmark_data_smithi179_77138_object64009:head# 2024-02-09T17:26:52.456 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6b3573b:::benchmark_data_smithi179_77138_object52952:head# 2024-02-09T17:26:52.457 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6b3832f:::benchmark_data_smithi179_77138_object85674:head# 2024-02-09T17:26:52.457 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6b42501:::benchmark_data_smithi179_77138_object54306:head# 2024-02-09T17:26:52.458 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6b434d1:::benchmark_data_smithi179_77138_object30579:head# 2024-02-09T17:26:52.458 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6b45b5d:::benchmark_data_smithi179_77138_object113872:head# 2024-02-09T17:26:52.462 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6b5dd2a:::benchmark_data_smithi179_77138_object24457:head# 2024-02-09T17:26:52.462 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6b5f08b:::benchmark_data_smithi179_77138_object94103:head# 2024-02-09T17:26:52.462 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6b66f78:::benchmark_data_smithi179_77138_object24300:head# 2024-02-09T17:26:52.462 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6b77cbd:::benchmark_data_smithi179_77138_object124223:head# 2024-02-09T17:26:52.462 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6b8a71a:::benchmark_data_smithi179_77138_object35685:head# 2024-02-09T17:26:52.463 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6b94bf6:::benchmark_data_smithi179_77138_object20744:head# 2024-02-09T17:26:52.463 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6b9da7d:::benchmark_data_smithi179_77138_object110692:head# 2024-02-09T17:26:52.463 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6b9dbc0:::benchmark_data_smithi179_77138_object102697:head# 2024-02-09T17:26:52.463 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6b9f46d:::benchmark_data_smithi179_77138_object108595:head# 2024-02-09T17:26:52.463 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6baddc2:::benchmark_data_smithi179_77138_object56039:head# 2024-02-09T17:26:52.463 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6bae4bf:::benchmark_data_smithi179_77138_object48683:head# 2024-02-09T17:26:52.464 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6bcc438:::benchmark_data_smithi179_77138_object21325:head# 2024-02-09T17:26:52.464 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6bd2f06:::benchmark_data_smithi179_77138_object76017:head# 2024-02-09T17:26:52.465 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6bd5a57:::benchmark_data_smithi179_77138_object125527:head# 2024-02-09T17:26:52.465 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6bd7b30:::benchmark_data_smithi179_77138_object43237:head# 2024-02-09T17:26:52.465 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6be318f:::benchmark_data_smithi179_77138_object27167:head# 2024-02-09T17:26:52.466 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6be7840:::benchmark_data_smithi179_77138_object112041:head# 2024-02-09T17:26:52.466 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6be7cce:::benchmark_data_smithi179_77138_object55924:head# 2024-02-09T17:26:52.467 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6bef18a:::benchmark_data_smithi179_77138_object125450:head# 2024-02-09T17:26:52.467 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6bf8121:::benchmark_data_smithi179_77138_object32382:head# 2024-02-09T17:26:52.467 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6bfa139:::benchmark_data_smithi179_77138_object76824:head# 2024-02-09T17:26:52.468 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6c0a8d7:::benchmark_data_smithi179_77138_object86883:head# 2024-02-09T17:26:52.468 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6c10819:::benchmark_data_smithi179_77138_object119293:head# 2024-02-09T17:26:52.470 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6c10e44:::benchmark_data_smithi179_77138_object102882:head# 2024-02-09T17:26:52.470 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6c1e562:::benchmark_data_smithi179_77138_object120682:head# 2024-02-09T17:26:52.470 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6c26497:::benchmark_data_smithi179_77138_object26946:head# 2024-02-09T17:26:52.471 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6c2d45d:::benchmark_data_smithi179_77138_object90171:head# 2024-02-09T17:26:52.471 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6c39cab:::benchmark_data_smithi179_77138_object33840:head# 2024-02-09T17:26:52.471 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6c62206:::benchmark_data_smithi179_77138_object107000:head# 2024-02-09T17:26:52.472 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6c65576:::benchmark_data_smithi179_77138_object27709:head# 2024-02-09T17:26:52.473 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6c679fe:::benchmark_data_smithi179_77138_object46504:head# 2024-02-09T17:26:52.473 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6c6d243:::benchmark_data_smithi179_77138_object38273:head# 2024-02-09T17:26:52.473 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6c75942:::benchmark_data_smithi179_77138_object128372:head# 2024-02-09T17:26:52.474 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6c7878d:::benchmark_data_smithi179_77138_object97399:head# 2024-02-09T17:26:52.474 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6c7fbba:::benchmark_data_smithi179_77138_object36613:head# 2024-02-09T17:26:52.474 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6c83028:::benchmark_data_smithi179_77138_object25212:head# 2024-02-09T17:26:52.475 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6c84200:::benchmark_data_smithi179_77138_object52949:head# 2024-02-09T17:26:52.475 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6c859aa:::benchmark_data_smithi179_77138_object32565:head# 2024-02-09T17:26:52.476 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6c8735d:::benchmark_data_smithi179_77138_object71431:head# 2024-02-09T17:26:52.480 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6c9e8b9:::benchmark_data_smithi179_77138_object105011:head# 2024-02-09T17:26:52.480 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6ca352a:::benchmark_data_smithi179_77138_object69575:head# 2024-02-09T17:26:52.480 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6ca3d71:::benchmark_data_smithi179_77138_object95657:head# 2024-02-09T17:26:52.480 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6ca51db:::benchmark_data_smithi179_77138_object100293:head# 2024-02-09T17:26:52.480 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6cbd9f9:::benchmark_data_smithi179_77138_object63815:head# 2024-02-09T17:26:52.480 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6cc5084:::benchmark_data_smithi179_77138_object105124:head# 2024-02-09T17:26:52.480 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6cc7aaa:::benchmark_data_smithi179_77138_object108295:head# 2024-02-09T17:26:52.480 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6cce5e7:::benchmark_data_smithi179_77138_object44785:head# 2024-02-09T17:26:52.482 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6cd1657:::benchmark_data_smithi179_77138_object42575:head# 2024-02-09T17:26:52.482 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6cd50fd:::benchmark_data_smithi179_77138_object124651:head# 2024-02-09T17:26:52.483 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6cd7a5e:::benchmark_data_smithi179_77138_object129479:head# 2024-02-09T17:26:52.483 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6cda484:::benchmark_data_smithi179_77138_object90340:head# 2024-02-09T17:26:52.483 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6cdbf3e:::benchmark_data_smithi179_77138_object117502:head# 2024-02-09T17:26:52.483 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6ce1a35:::benchmark_data_smithi179_77138_object36253:head# 2024-02-09T17:26:52.483 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6cebf0f:::benchmark_data_smithi179_77138_object132337:head# 2024-02-09T17:26:52.484 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6cedbe8:::benchmark_data_smithi179_77138_object114411:head# 2024-02-09T17:26:52.484 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6cf99c6:::benchmark_data_smithi179_77138_object98555:head# 2024-02-09T17:26:52.485 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6d063f6:::benchmark_data_smithi179_77138_object21188:head# 2024-02-09T17:26:52.485 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6d27416:::benchmark_data_smithi179_77138_object52228:head# 2024-02-09T17:26:52.490 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6d367c7:::benchmark_data_smithi179_77138_object108359:head# 2024-02-09T17:26:52.490 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6d3b6bd:::benchmark_data_smithi179_77138_object35107:head# 2024-02-09T17:26:52.490 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6d41c51:::benchmark_data_smithi179_77138_object68564:head# 2024-02-09T17:26:52.490 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6d430d5:::benchmark_data_smithi179_77138_object102259:head# 2024-02-09T17:26:52.490 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6d460bc:::benchmark_data_smithi179_77138_object34167:head# 2024-02-09T17:26:52.490 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6d47964:::benchmark_data_smithi179_77138_object113852:head# 2024-02-09T17:26:52.490 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6d4c9db:::benchmark_data_smithi179_77138_object125123:head# 2024-02-09T17:26:52.491 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6d4f3b3:::benchmark_data_smithi179_77138_object104870:head# 2024-02-09T17:26:52.491 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6d5b694:::benchmark_data_smithi179_77138_object119037:head# 2024-02-09T17:26:52.494 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6d5cf0c:::benchmark_data_smithi179_77138_object98260:head# 2024-02-09T17:26:52.495 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6d6cd95:::benchmark_data_smithi179_77138_object94243:head# 2024-02-09T17:26:52.495 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6d8d9d1:::benchmark_data_smithi179_77138_object110066:head# 2024-02-09T17:26:52.495 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6d94fd1:::benchmark_data_smithi179_77138_object123292:head# 2024-02-09T17:26:52.495 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6da6525:::benchmark_data_smithi179_77138_object62193:head# 2024-02-09T17:26:52.495 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6db8717:::benchmark_data_smithi179_77138_object119742:head# 2024-02-09T17:26:52.495 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6db9b2d:::benchmark_data_smithi179_77138_object73270:head# 2024-02-09T17:26:52.495 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6dc1cd4:::benchmark_data_smithi179_77138_object111866:head# 2024-02-09T17:26:52.495 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6dc9c91:::benchmark_data_smithi179_77138_object131352:head# 2024-02-09T17:26:52.495 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6dd1979:::benchmark_data_smithi179_77138_object35162:head# 2024-02-09T17:26:52.496 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6dd1f0f:::benchmark_data_smithi179_77138_object56100:head# 2024-02-09T17:26:52.496 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6dd5e0e:::benchmark_data_smithi179_77138_object133391:head# 2024-02-09T17:26:52.496 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6de6e25:::benchmark_data_smithi179_77138_object101460:head# 2024-02-09T17:26:52.496 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6dfd19d:::benchmark_data_smithi179_77138_object88804:head# 2024-02-09T17:26:52.497 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6e01882:::benchmark_data_smithi179_77138_object43061:head# 2024-02-09T17:26:52.497 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6e0280d:::benchmark_data_smithi179_77138_object90956:head# 2024-02-09T17:26:52.498 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6e0880a:::benchmark_data_smithi179_77138_object53609:head# 2024-02-09T17:26:52.498 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6e0e2cf:::benchmark_data_smithi179_77138_object97447:head# 2024-02-09T17:26:52.498 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6e18152:::benchmark_data_smithi179_77138_object122220:head# 2024-02-09T17:26:52.499 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6e1a435:::benchmark_data_smithi179_77138_object43880:head# 2024-02-09T17:26:52.499 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6e22cc4:::benchmark_data_smithi179_77138_object107588:head# 2024-02-09T17:26:52.500 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6e291cf:::benchmark_data_smithi179_77138_object53169:head# 2024-02-09T17:26:52.500 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6e2a2da:::benchmark_data_smithi179_77138_object75273:head# 2024-02-09T17:26:52.501 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6e3382a:::benchmark_data_smithi179_77138_object43356:head# 2024-02-09T17:26:52.501 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6e5c4d1:::benchmark_data_smithi179_77138_object96887:head# 2024-02-09T17:26:52.502 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6e65a31:::benchmark_data_smithi179_77138_object98740:head# 2024-02-09T17:26:52.502 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6e68ee1:::benchmark_data_smithi179_77138_object114665:head# 2024-02-09T17:26:52.503 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6e74b91:::benchmark_data_smithi179_77138_object28097:head# 2024-02-09T17:26:52.503 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6e74e5c:::benchmark_data_smithi179_77138_object119204:head# 2024-02-09T17:26:52.504 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6e778a1:::benchmark_data_smithi179_77138_object66934:head# 2024-02-09T17:26:52.504 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6e81b8a:::benchmark_data_smithi179_77138_object77716:head# 2024-02-09T17:26:52.505 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6e86cb0:::benchmark_data_smithi179_77138_object60858:head# 2024-02-09T17:26:52.505 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6e8c93c:::benchmark_data_smithi179_77138_object103036:head# 2024-02-09T17:26:52.507 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6e9a103:::benchmark_data_smithi179_77138_object94079:head# 2024-02-09T17:26:52.508 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6e9e575:::benchmark_data_smithi179_77138_object36798:head# 2024-02-09T17:26:52.508 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6ea18ef:::benchmark_data_smithi179_77138_object74187:head# 2024-02-09T17:26:52.509 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6ea2797:::benchmark_data_smithi179_77138_object80878:head# 2024-02-09T17:26:52.509 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6ea31dd:::benchmark_data_smithi179_77138_object51712:head# 2024-02-09T17:26:52.510 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6ea39f6:::benchmark_data_smithi179_77138_object128363:head# 2024-02-09T17:26:52.510 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6eb7332:::benchmark_data_smithi179_77138_object52053:head# 2024-02-09T17:26:52.511 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6ec3020:::benchmark_data_smithi179_77138_object89419:head# 2024-02-09T17:26:52.511 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6ec9d68:::benchmark_data_smithi179_77138_object123775:head# 2024-02-09T17:26:52.512 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6edd549:::benchmark_data_smithi179_77138_object43977:head# 2024-02-09T17:26:52.512 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6edf458:::benchmark_data_smithi179_77138_object54913:head# 2024-02-09T17:26:52.513 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6ee6148:::benchmark_data_smithi179_77138_object97098:head# 2024-02-09T17:26:52.513 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6eef147:::benchmark_data_smithi179_77138_object43340:head# 2024-02-09T17:26:52.514 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6ef0c09:::benchmark_data_smithi179_77138_object33205:head# 2024-02-09T17:26:52.514 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6ef6f17:::benchmark_data_smithi179_77138_object65672:head# 2024-02-09T17:26:52.515 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6efddbe:::benchmark_data_smithi179_77138_object70447:head# 2024-02-09T17:26:52.515 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6eff0b6:::benchmark_data_smithi179_77138_object77334:head# 2024-02-09T17:26:52.516 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6f04055:::benchmark_data_smithi179_77138_object72647:head# 2024-02-09T17:26:52.516 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6f151d7:::benchmark_data_smithi179_77138_object50508:head# 2024-02-09T17:26:52.517 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6f17d5f:::benchmark_data_smithi179_77138_object74308:head# 2024-02-09T17:26:52.517 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6f269bf:::benchmark_data_smithi179_77138_object113918:head# 2024-02-09T17:26:52.518 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6f2c845:::benchmark_data_smithi179_77138_object95648:head# 2024-02-09T17:26:52.518 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6f37e32:::benchmark_data_smithi179_77138_object45795:head# 2024-02-09T17:26:52.519 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6f38e88:::benchmark_data_smithi179_77138_object109888:head# 2024-02-09T17:26:52.519 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6f3994d:::benchmark_data_smithi179_77138_object40963:head# 2024-02-09T17:26:52.520 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6f41e7b:::benchmark_data_smithi179_77138_object106976:head# 2024-02-09T17:26:52.520 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6f53f92:::benchmark_data_smithi179_77138_object120298:head# 2024-02-09T17:26:52.523 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6f54114:::benchmark_data_smithi179_77138_object66630:head# 2024-02-09T17:26:52.524 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6f58774:::benchmark_data_smithi179_77138_object83531:head# 2024-02-09T17:26:52.524 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6f5e451:::benchmark_data_smithi179_77138_object19459:head# 2024-02-09T17:26:52.524 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6f615c3:::benchmark_data_smithi179_77138_object52909:head# 2024-02-09T17:26:52.525 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6f8158c:::benchmark_data_smithi179_77138_object47860:head# 2024-02-09T17:26:52.525 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6f88548:::benchmark_data_smithi179_77138_object37338:head# 2024-02-09T17:26:52.526 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6f90148:::benchmark_data_smithi179_77138_object54999:head# 2024-02-09T17:26:52.526 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6f993c0:::benchmark_data_smithi179_77138_object129926:head# 2024-02-09T17:26:52.527 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6fb2b00:::benchmark_data_smithi179_77138_object108719:head# 2024-02-09T17:26:52.527 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6fb6c75:::benchmark_data_smithi179_77138_object104237:head# 2024-02-09T17:26:52.528 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6fbbb5b:::benchmark_data_smithi179_77138_object50828:head# 2024-02-09T17:26:52.528 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6fc845a:::benchmark_data_smithi179_77138_object93052:head# 2024-02-09T17:26:52.528 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6fcd028:::benchmark_data_smithi179_77138_object65675:head# 2024-02-09T17:26:52.529 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6fcdb96:::benchmark_data_smithi179_77138_object83490:head# 2024-02-09T17:26:52.529 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6fcfddb:::benchmark_data_smithi179_77138_object38906:head# 2024-02-09T17:26:52.530 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6ff7e89:::benchmark_data_smithi179_77138_object84585:head# 2024-02-09T17:26:52.530 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6ffd133:::benchmark_data_smithi179_77138_object124087:head# 2024-02-09T17:26:52.531 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f6ffe373:::benchmark_data_smithi179_77138_object94550:head# 2024-02-09T17:26:52.531 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7006677:::benchmark_data_smithi179_77138_object65671:head# 2024-02-09T17:26:52.531 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f70092df:::benchmark_data_smithi179_77138_object123287:head# 2024-02-09T17:26:52.532 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f700b84e:::benchmark_data_smithi179_77138_object29335:head# 2024-02-09T17:26:52.532 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7011d1d:::benchmark_data_smithi179_77138_object104135:head# 2024-02-09T17:26:52.533 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f70153e5:::benchmark_data_smithi179_77138_object105112:head# 2024-02-09T17:26:52.533 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f701842d:::benchmark_data_smithi179_77138_object99358:head# 2024-02-09T17:26:52.534 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7026d3d:::benchmark_data_smithi179_77138_object72441:head# 2024-02-09T17:26:52.534 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f703b64e:::benchmark_data_smithi179_77138_object76098:head# 2024-02-09T17:26:52.536 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f703e408:::benchmark_data_smithi179_77138_object78104:head# 2024-02-09T17:26:52.536 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f70423f3:::benchmark_data_smithi179_77138_object30480:head# 2024-02-09T17:26:52.538 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7042b9d:::benchmark_data_smithi179_77138_object116515:head# 2024-02-09T17:26:52.538 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f704509a:::benchmark_data_smithi179_77138_object87146:head# 2024-02-09T17:26:52.538 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7048e4d:::benchmark_data_smithi179_77138_object71539:head# 2024-02-09T17:26:52.539 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7051fda:::benchmark_data_smithi179_77138_object98527:head# 2024-02-09T17:26:52.543 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:26:52.544 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7052900:::benchmark_data_smithi179_77138_object65447:head# 2024-02-09T17:26:52.544 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f70589f9:::benchmark_data_smithi179_77138_object85957:head# 2024-02-09T17:26:52.544 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f705ba47:::benchmark_data_smithi179_77138_object95457:head# 2024-02-09T17:26:52.547 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f70685af:::benchmark_data_smithi179_77138_object79980:head# 2024-02-09T17:26:52.547 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7069aac:::benchmark_data_smithi179_77138_object49406:head# 2024-02-09T17:26:52.547 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f706ca68:::benchmark_data_smithi179_77138_object101293:head# 2024-02-09T17:26:52.552 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7071277:::benchmark_data_smithi179_77138_object97862:head# 2024-02-09T17:26:52.554 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f707879f:::benchmark_data_smithi179_77138_object79141:head# 2024-02-09T17:26:52.554 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f707ed75:::benchmark_data_smithi179_77138_object30444:head# 2024-02-09T17:26:52.554 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7081086:::benchmark_data_smithi179_77138_object47037:head# 2024-02-09T17:26:52.555 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7082436:::benchmark_data_smithi179_77138_object81091:head# 2024-02-09T17:26:52.556 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7085425:::benchmark_data_smithi179_77138_object108897:head# 2024-02-09T17:26:52.556 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7089deb:::benchmark_data_smithi179_77138_object21447:head# 2024-02-09T17:26:52.557 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7098ff9:::benchmark_data_smithi179_77138_object42817:head# 2024-02-09T17:26:52.557 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f709da36:::benchmark_data_smithi179_77138_object37052:head# 2024-02-09T17:26:52.557 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f70a560f:::benchmark_data_smithi179_77138_object89556:head# 2024-02-09T17:26:52.558 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f70aafed:::benchmark_data_smithi179_77138_object69751:head# 2024-02-09T17:26:52.559 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f70aec77:::benchmark_data_smithi179_77138_object78531:head# 2024-02-09T17:26:52.560 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f70af6d4:::benchmark_data_smithi179_77138_object116725:head# 2024-02-09T17:26:52.560 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f70bea4e:::benchmark_data_smithi179_77138_object52854:head# 2024-02-09T17:26:52.560 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f70c647b:::benchmark_data_smithi179_77138_object73725:head# 2024-02-09T17:26:52.560 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f70cd54a:::benchmark_data_smithi179_77138_object100585:head# 2024-02-09T17:26:52.562 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f70d264e:::benchmark_data_smithi179_77138_object109247:head# 2024-02-09T17:26:52.562 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f70daf5f:::benchmark_data_smithi179_77138_object101225:head# 2024-02-09T17:26:52.562 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f70df852:::benchmark_data_smithi179_77138_object82252:head# 2024-02-09T17:26:52.562 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f70e1914:::benchmark_data_smithi179_77138_object124206:head# 2024-02-09T17:26:52.563 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f70e2dc5:::benchmark_data_smithi179_77138_object30095:head# 2024-02-09T17:26:52.563 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f70e9329:::benchmark_data_smithi179_77138_object23169:head# 2024-02-09T17:26:52.563 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f70f1bbc:::benchmark_data_smithi179_77138_object125907:head# 2024-02-09T17:26:52.566 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f70f7bef:::benchmark_data_smithi179_77138_object44257:head# 2024-02-09T17:26:52.566 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f70f7df7:::benchmark_data_smithi179_77138_object89606:head# 2024-02-09T17:26:52.567 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7112cb9:::benchmark_data_smithi179_77138_object115283:head# 2024-02-09T17:26:52.567 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f71191e4:::benchmark_data_smithi179_77138_object56352:head# 2024-02-09T17:26:52.568 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7122279:::benchmark_data_smithi179_77138_object55459:head# 2024-02-09T17:26:52.568 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7129f79:::benchmark_data_smithi179_77138_object64330:head# 2024-02-09T17:26:52.569 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f713920b:::benchmark_data_smithi179_77138_object24636:head# 2024-02-09T17:26:52.569 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7142240:::benchmark_data_smithi179_77138_object112918:head# 2024-02-09T17:26:52.571 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f714db7f:::benchmark_data_smithi179_77138_object91819:head# 2024-02-09T17:26:52.571 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7152e85:::benchmark_data_smithi179_77138_object38358:head# 2024-02-09T17:26:52.571 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f715ee71:::benchmark_data_smithi179_77138_object88263:head# 2024-02-09T17:26:52.571 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f716c368:::benchmark_data_smithi179_77138_object46211:head# 2024-02-09T17:26:52.572 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f716f81c:::benchmark_data_smithi179_77138_object20253:head# 2024-02-09T17:26:52.572 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7170992:::benchmark_data_smithi179_77138_object96223:head# 2024-02-09T17:26:52.572 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7171a5f:::benchmark_data_smithi179_77138_object130201:head# 2024-02-09T17:26:52.573 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7187715:::benchmark_data_smithi179_77138_object33721:head# 2024-02-09T17:26:52.573 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f718cd2a:::benchmark_data_smithi179_77138_object101497:head# 2024-02-09T17:26:52.574 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f719c6fc:::benchmark_data_smithi179_77138_object121824:head# 2024-02-09T17:26:52.574 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f719c7d4:::benchmark_data_smithi179_77138_object108717:head# 2024-02-09T17:26:52.575 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f71a47f4:::benchmark_data_smithi179_77138_object92605:head# 2024-02-09T17:26:52.575 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f71a8c10:::benchmark_data_smithi179_77138_object131907:head# 2024-02-09T17:26:52.576 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f71aa14c:::benchmark_data_smithi179_77138_object46329:head# 2024-02-09T17:26:52.578 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f71ac567:::benchmark_data_smithi179_77138_object49119:head# 2024-02-09T17:26:52.579 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f71bd0ff:::benchmark_data_smithi179_77138_object124256:head# 2024-02-09T17:26:52.581 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:52.581 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f71bd48c:::benchmark_data_smithi179_77138_object99215:head# 2024-02-09T17:26:52.582 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f71d949e:::benchmark_data_smithi179_77138_object55035:head# 2024-02-09T17:26:52.582 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f71e0057:::benchmark_data_smithi179_77138_object124107:head# 2024-02-09T17:26:52.582 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f71e35b2:::benchmark_data_smithi179_77138_object85397:head# 2024-02-09T17:26:52.583 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f71e3d8e:::benchmark_data_smithi179_77138_object96355:head# 2024-02-09T17:26:52.583 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f71e898b:::benchmark_data_smithi179_77138_object47945:head# 2024-02-09T17:26:52.584 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f71fbd51:::benchmark_data_smithi179_77138_object41594:head# 2024-02-09T17:26:52.587 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f71fd3fe:::benchmark_data_smithi179_77138_object38941:head# 2024-02-09T17:26:52.587 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f720c7d8:::benchmark_data_smithi179_77138_object66183:head# 2024-02-09T17:26:52.587 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f72174b6:::benchmark_data_smithi179_77138_object33248:head# 2024-02-09T17:26:52.588 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f721904a:::benchmark_data_smithi179_77138_object82984:head# 2024-02-09T17:26:52.588 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f721aa31:::benchmark_data_smithi179_77138_object121296:head# 2024-02-09T17:26:52.589 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f722dd5b:::benchmark_data_smithi179_77138_object53254:head# 2024-02-09T17:26:52.589 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7235016:::benchmark_data_smithi179_77138_object112678:head# 2024-02-09T17:26:52.589 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f72389ab:::benchmark_data_smithi179_77138_object31273:head# 2024-02-09T17:26:52.590 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7241895:::benchmark_data_smithi179_77138_object62108:head# 2024-02-09T17:26:52.590 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7255478:::benchmark_data_smithi179_77138_object32147:head# 2024-02-09T17:26:52.592 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7256b03:::benchmark_data_smithi179_77138_object103163:head# 2024-02-09T17:26:52.592 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f725c601:::benchmark_data_smithi179_77138_object83900:head# 2024-02-09T17:26:52.593 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7261caf:::benchmark_data_smithi179_77138_object71441:head# 2024-02-09T17:26:52.596 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7265695:::benchmark_data_smithi179_77138_object107502:head# 2024-02-09T17:26:52.596 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7267933:::benchmark_data_smithi179_77138_object35394:head# 2024-02-09T17:26:52.596 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f72694fa:::benchmark_data_smithi179_77138_object30687:head# 2024-02-09T17:26:52.596 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f726e601:::benchmark_data_smithi179_77138_object37062:head# 2024-02-09T17:26:52.596 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7274e48:::benchmark_data_smithi179_77138_object30197:head# 2024-02-09T17:26:52.596 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f727aa7e:::benchmark_data_smithi179_77138_object132642:head# 2024-02-09T17:26:52.597 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f727b56c:::benchmark_data_smithi179_77138_object111833:head# 2024-02-09T17:26:52.597 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f727f573:::benchmark_data_smithi179_77138_object40179:head# 2024-02-09T17:26:52.597 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7285e1b:::benchmark_data_smithi179_77138_object56240:head# 2024-02-09T17:26:52.598 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f728659b:::benchmark_data_smithi179_77138_object98484:head# 2024-02-09T17:26:52.598 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f729fa34:::benchmark_data_smithi179_77138_object110082:head# 2024-02-09T17:26:52.599 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f72a1082:::benchmark_data_smithi179_77138_object123213:head# 2024-02-09T17:26:52.601 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f72a1a7e:::benchmark_data_smithi179_77138_object21486:head# 2024-02-09T17:26:52.601 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f72aab62:::benchmark_data_smithi179_77138_object120533:head# 2024-02-09T17:26:52.601 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f72c7b6c:::benchmark_data_smithi179_77138_object57500:head# 2024-02-09T17:26:52.602 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f72d1f3f:::benchmark_data_smithi179_77138_object92634:head# 2024-02-09T17:26:52.602 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f72d413c:::benchmark_data_smithi179_77138_object124443:head# 2024-02-09T17:26:52.604 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f72eb6c9:::benchmark_data_smithi179_77138_object126995:head# 2024-02-09T17:26:52.605 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f72ebf3b:::benchmark_data_smithi179_77138_object72368:head# 2024-02-09T17:26:52.605 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f72f2849:::benchmark_data_smithi179_77138_object114391:head# 2024-02-09T17:26:52.605 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f72f3a96:::benchmark_data_smithi179_77138_object28094:head# 2024-02-09T17:26:52.606 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:26:52.606 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f72fdb4f:::benchmark_data_smithi179_77138_object24253:head# 2024-02-09T17:26:52.607 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7302765:::benchmark_data_smithi179_77138_object121091:head# 2024-02-09T17:26:52.608 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f730d4af:::benchmark_data_smithi179_77138_object49728:head# 2024-02-09T17:26:52.608 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f731896f:::benchmark_data_smithi179_77138_object110840:head# 2024-02-09T17:26:52.609 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7319972:::benchmark_data_smithi179_77138_object82922:head# 2024-02-09T17:26:52.609 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7322620:::benchmark_data_smithi179_77138_object108236:head# 2024-02-09T17:26:52.610 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7323356:::benchmark_data_smithi179_77138_object100040:head# 2024-02-09T17:26:52.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:52.611 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:26:52.611 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f73242a3:::benchmark_data_smithi179_77138_object41404:head# 2024-02-09T17:26:52.611 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7338f29:::benchmark_data_smithi179_77138_object67542:head# 2024-02-09T17:26:52.612 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7345c13:::benchmark_data_smithi179_77138_object108675:head# 2024-02-09T17:26:52.612 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f734fe42:::benchmark_data_smithi179_77138_object54329:head# 2024-02-09T17:26:52.612 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f735562a:::benchmark_data_smithi179_77138_object34987:head# 2024-02-09T17:26:52.614 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7357261:::benchmark_data_smithi179_77138_object44747:head# 2024-02-09T17:26:52.614 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f735f034:::benchmark_data_smithi179_77138_object35777:head# 2024-02-09T17:26:52.615 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f73620a0:::benchmark_data_smithi179_77138_object33704:head# 2024-02-09T17:26:52.615 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7373ead:::benchmark_data_smithi179_77138_object103017:head# 2024-02-09T17:26:52.615 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7375f91:::benchmark_data_smithi179_77138_object54853:head# 2024-02-09T17:26:52.616 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7385284:::benchmark_data_smithi179_77138_object82503:head# 2024-02-09T17:26:52.616 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7386019:::benchmark_data_smithi179_77138_object90365:head# 2024-02-09T17:26:52.617 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f739b11b:::benchmark_data_smithi179_77138_object48055:head# 2024-02-09T17:26:52.617 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f73a43d1:::benchmark_data_smithi179_77138_object47274:head# 2024-02-09T17:26:52.618 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f73a5e4e:::benchmark_data_smithi179_77138_object38793:head# 2024-02-09T17:26:52.618 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f73b0b83:::benchmark_data_smithi179_77138_object33788:head# 2024-02-09T17:26:52.618 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f73b1941:::benchmark_data_smithi179_77138_object61062:head# 2024-02-09T17:26:52.619 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f73b2810:::benchmark_data_smithi179_77138_object43745:head# 2024-02-09T17:26:52.619 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f73b84e1:::benchmark_data_smithi179_77138_object92886:head# 2024-02-09T17:26:52.620 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f73b8d01:::benchmark_data_smithi179_77138_object99106:head# 2024-02-09T17:26:52.620 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f73c0521:::benchmark_data_smithi179_77138_object59841:head# 2024-02-09T17:26:52.621 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f73c1e25:::benchmark_data_smithi179_77138_object46401:head# 2024-02-09T17:26:52.621 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f73c55e8:::benchmark_data_smithi179_77138_object67789:head# 2024-02-09T17:26:52.622 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f73c5cbe:::benchmark_data_smithi179_77138_object52882:head# 2024-02-09T17:26:52.622 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f73c6d1b:::benchmark_data_smithi179_77138_object127715:head# 2024-02-09T17:26:52.624 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f73d00d4:::benchmark_data_smithi179_77138_object54219:head# 2024-02-09T17:26:52.624 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f73d2d2f:::benchmark_data_smithi179_77138_object37801:head# 2024-02-09T17:26:52.625 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f73dc7b7:::benchmark_data_smithi179_77138_object66359:head# 2024-02-09T17:26:52.625 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f73ea559:::benchmark_data_smithi179_77138_object85646:head# 2024-02-09T17:26:52.630 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f73eb815:::benchmark_data_smithi179_77138_object60031:head# 2024-02-09T17:26:52.631 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f73f2dae:::benchmark_data_smithi179_77138_object107713:head# 2024-02-09T17:26:52.631 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f73f4262:::benchmark_data_smithi179_77138_object74167:head# 2024-02-09T17:26:52.632 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f73fbf86:::benchmark_data_smithi179_77138_object22998:head# 2024-02-09T17:26:52.633 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f74068d9:::benchmark_data_smithi179_77138_object120934:head# 2024-02-09T17:26:52.633 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f740d527:::benchmark_data_smithi179_77138_object23346:head# 2024-02-09T17:26:52.634 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7416d7a:::benchmark_data_smithi179_77138_object61223:head# 2024-02-09T17:26:52.634 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f74305f7:::benchmark_data_smithi179_77138_object45059:head# 2024-02-09T17:26:52.634 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7453703:::benchmark_data_smithi179_77138_object60301:head# 2024-02-09T17:26:52.635 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7457eb3:::benchmark_data_smithi179_77138_object77577:head# 2024-02-09T17:26:52.635 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f745ff68:::benchmark_data_smithi179_77138_object96264:head# 2024-02-09T17:26:52.636 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7467cfa:::benchmark_data_smithi179_77138_object60459:head# 2024-02-09T17:26:52.636 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f746971b:::benchmark_data_smithi179_77138_object57349:head# 2024-02-09T17:26:52.640 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f746a075:::benchmark_data_smithi179_77138_object22069:head# 2024-02-09T17:26:52.640 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f746e382:::benchmark_data_smithi179_77138_object35851:head# 2024-02-09T17:26:52.640 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7476037:::benchmark_data_smithi179_77138_object45403:head# 2024-02-09T17:26:52.640 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f74784ce:::benchmark_data_smithi179_77138_object42852:head# 2024-02-09T17:26:52.640 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7486855:::benchmark_data_smithi179_77138_object29051:head# 2024-02-09T17:26:52.640 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7488c08:::benchmark_data_smithi179_77138_object116816:head# 2024-02-09T17:26:52.642 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f748c861:::benchmark_data_smithi179_77138_object120301:head# 2024-02-09T17:26:52.642 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f749b282:::benchmark_data_smithi179_77138_object25075:head# 2024-02-09T17:26:52.642 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f74ac6d7:::benchmark_data_smithi179_77138_object23493:head# 2024-02-09T17:26:52.642 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f74afddd:::benchmark_data_smithi179_77138_object32252:head# 2024-02-09T17:26:52.645 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f74b79a4:::benchmark_data_smithi179_77138_object20628:head# 2024-02-09T17:26:52.645 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f74b7d6a:::benchmark_data_smithi179_77138_object113396:head# 2024-02-09T17:26:52.645 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f74b86e1:::benchmark_data_smithi179_77138_object25961:head# 2024-02-09T17:26:52.645 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f74b965c:::benchmark_data_smithi179_77138_object63464:head# 2024-02-09T17:26:52.645 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f74c0af7:::benchmark_data_smithi179_77138_object108723:head# 2024-02-09T17:26:52.645 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f74d55f6:::benchmark_data_smithi179_77138_object55157:head# 2024-02-09T17:26:52.646 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f74dbdd1:::benchmark_data_smithi179_77138_object37065:head# 2024-02-09T17:26:52.646 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f74ec5da:::benchmark_data_smithi179_77138_object75133:head# 2024-02-09T17:26:52.646 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f74ee449:::benchmark_data_smithi179_77138_object35251:head# 2024-02-09T17:26:52.647 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f74ee770:::benchmark_data_smithi179_77138_object27509:head# 2024-02-09T17:26:52.647 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f74ef86a:::benchmark_data_smithi179_77138_object53115:head# 2024-02-09T17:26:52.648 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f74f3560:::benchmark_data_smithi179_77138_object29710:head# 2024-02-09T17:26:52.648 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f74f6a3e:::benchmark_data_smithi179_77138_object27211:head# 2024-02-09T17:26:52.648 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f74f783d:::benchmark_data_smithi179_77138_object23303:head# 2024-02-09T17:26:52.649 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f74fd95a:::benchmark_data_smithi179_77138_object132019:head# 2024-02-09T17:26:52.649 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f74ff70c:::benchmark_data_smithi179_77138_object46604:head# 2024-02-09T17:26:52.649 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7501d38:::benchmark_data_smithi179_77138_object101178:head# 2024-02-09T17:26:52.650 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f750c1b8:::benchmark_data_smithi179_77138_object31720:head# 2024-02-09T17:26:52.650 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f751fa30:::benchmark_data_smithi179_77138_object93041:head# 2024-02-09T17:26:52.651 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7526673:::benchmark_data_smithi179_77138_object77840:head# 2024-02-09T17:26:52.651 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7538bce:::benchmark_data_smithi179_77138_object119036:head# 2024-02-09T17:26:52.652 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7538dfd:::benchmark_data_smithi179_77138_object37626:head# 2024-02-09T17:26:52.652 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f753cbeb:::benchmark_data_smithi179_77138_object104332:head# 2024-02-09T17:26:52.652 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f75408f3:::benchmark_data_smithi179_77138_object109881:head# 2024-02-09T17:26:52.653 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f754c16f:::benchmark_data_smithi179_77138_object69755:head# 2024-02-09T17:26:52.653 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f75563b9:::benchmark_data_smithi179_77138_object119231:head# 2024-02-09T17:26:52.653 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f75589fc:::benchmark_data_smithi179_77138_object30169:head# 2024-02-09T17:26:52.654 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f756e6e0:::benchmark_data_smithi179_77138_object27501:head# 2024-02-09T17:26:52.655 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f75765e5:::benchmark_data_smithi179_77138_object25672:head# 2024-02-09T17:26:52.655 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7579200:::benchmark_data_smithi179_77138_object52675:head# 2024-02-09T17:26:52.655 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f757f864:::benchmark_data_smithi179_77138_object59493:head# 2024-02-09T17:26:52.656 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f75913f7:::benchmark_data_smithi179_77138_object131004:head# 2024-02-09T17:26:52.656 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7596b39:::benchmark_data_smithi179_77138_object79608:head# 2024-02-09T17:26:52.657 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f759bd19:::benchmark_data_smithi179_77138_object76349:head# 2024-02-09T17:26:52.657 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f75a3a48:::benchmark_data_smithi179_77138_object63222:head# 2024-02-09T17:26:52.657 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f75b62d2:::benchmark_data_smithi179_77138_object54098:head# 2024-02-09T17:26:52.661 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f75bcf5f:::benchmark_data_smithi179_77138_object32359:head# 2024-02-09T17:26:52.661 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f75c9efe:::benchmark_data_smithi179_77138_object36593:head# 2024-02-09T17:26:52.661 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f75df61f:::benchmark_data_smithi179_77138_object102184:head# 2024-02-09T17:26:52.661 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f75f4ae2:::benchmark_data_smithi179_77138_object60286:head# 2024-02-09T17:26:52.661 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f760287b:::benchmark_data_smithi179_77138_object129003:head# 2024-02-09T17:26:52.661 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f760d677:::benchmark_data_smithi179_77138_object72848:head# 2024-02-09T17:26:52.661 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7610cb3:::benchmark_data_smithi179_77138_object21335:head# 2024-02-09T17:26:52.661 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7611a7e:::benchmark_data_smithi179_77138_object48002:head# 2024-02-09T17:26:52.666 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7618531:::benchmark_data_smithi179_77138_object87481:head# 2024-02-09T17:26:52.666 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f761e5ac:::benchmark_data_smithi179_77138_object27322:head# 2024-02-09T17:26:52.666 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7631e47:::benchmark_data_smithi179_77138_object68943:head# 2024-02-09T17:26:52.666 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f763795e:::benchmark_data_smithi179_77138_object34829:head# 2024-02-09T17:26:52.666 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f763a676:::benchmark_data_smithi179_77138_object47628:head# 2024-02-09T17:26:52.667 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f764318d:::benchmark_data_smithi179_77138_object70781:head# 2024-02-09T17:26:52.667 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f764464c:::benchmark_data_smithi179_77138_object77890:head# 2024-02-09T17:26:52.668 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7645b2a:::benchmark_data_smithi179_77138_object54433:head# 2024-02-09T17:26:52.668 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f765124b:::benchmark_data_smithi179_77138_object66327:head# 2024-02-09T17:26:52.668 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7666c78:::benchmark_data_smithi179_77138_object126829:head# 2024-02-09T17:26:52.669 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f766ee88:::benchmark_data_smithi179_77138_object51919:head# 2024-02-09T17:26:52.674 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f767190d:::benchmark_data_smithi179_77138_object87302:head# 2024-02-09T17:26:52.674 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7676b37:::benchmark_data_smithi179_77138_object94238:head# 2024-02-09T17:26:52.674 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f767e556:::benchmark_data_smithi179_77138_object59363:head# 2024-02-09T17:26:52.674 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7686c88:::benchmark_data_smithi179_77138_object87997:head# 2024-02-09T17:26:52.674 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f768f09a:::benchmark_data_smithi179_77138_object127542:head# 2024-02-09T17:26:52.674 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7691c32:::benchmark_data_smithi179_77138_object90188:head# 2024-02-09T17:26:52.674 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f76aa06c:::benchmark_data_smithi179_77138_object70837:head# 2024-02-09T17:26:52.675 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f76ac8dc:::benchmark_data_smithi179_77138_object117469:head# 2024-02-09T17:26:52.675 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f76bc0cb:::benchmark_data_smithi179_77138_object129700:head# 2024-02-09T17:26:52.676 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f76c697c:::benchmark_data_smithi179_77138_object97974:head# 2024-02-09T17:26:52.681 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f76c942e:::benchmark_data_smithi179_77138_object28165:head# 2024-02-09T17:26:52.682 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f76cb6c9:::benchmark_data_smithi179_77138_object52426:head# 2024-02-09T17:26:52.683 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f76cd64f:::benchmark_data_smithi179_77138_object131641:head# 2024-02-09T17:26:52.683 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f76ce8d9:::benchmark_data_smithi179_77138_object38794:head# 2024-02-09T17:26:52.686 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f76e0d58:::benchmark_data_smithi179_77138_object130773:head# 2024-02-09T17:26:52.686 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7705859:::benchmark_data_smithi179_77138_object30374:head# 2024-02-09T17:26:52.687 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f770d442:::benchmark_data_smithi179_77138_object20130:head# 2024-02-09T17:26:52.687 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f771c2cc:::benchmark_data_smithi179_77138_object76612:head# 2024-02-09T17:26:52.688 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f771e920:::benchmark_data_smithi179_77138_object77202:head# 2024-02-09T17:26:52.688 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f772204d:::benchmark_data_smithi179_77138_object54101:head# 2024-02-09T17:26:52.688 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7729790:::benchmark_data_smithi179_77138_object113309:head# 2024-02-09T17:26:52.689 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7730af5:::benchmark_data_smithi179_77138_object71665:head# 2024-02-09T17:26:52.689 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f77343bc:::benchmark_data_smithi179_77138_object25111:head# 2024-02-09T17:26:52.690 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f773a523:::benchmark_data_smithi179_77138_object116040:head# 2024-02-09T17:26:52.690 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f77424f4:::benchmark_data_smithi179_77138_object110469:head# 2024-02-09T17:26:52.690 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f774d50b:::benchmark_data_smithi179_77138_object55822:head# 2024-02-09T17:26:52.691 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f775050f:::benchmark_data_smithi179_77138_object104525:head# 2024-02-09T17:26:52.691 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f77578d3:::benchmark_data_smithi179_77138_object114719:head# 2024-02-09T17:26:52.692 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7757fa8:::benchmark_data_smithi179_77138_object26901:head# 2024-02-09T17:26:52.692 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f77581f4:::benchmark_data_smithi179_77138_object123781:head# 2024-02-09T17:26:52.692 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7778c89:::benchmark_data_smithi179_77138_object20142:head# 2024-02-09T17:26:52.693 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7780aef:::benchmark_data_smithi179_77138_object46824:head# 2024-02-09T17:26:52.693 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f77826ec:::benchmark_data_smithi179_77138_object28887:head# 2024-02-09T17:26:52.694 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7786418:::benchmark_data_smithi179_77138_object63660:head# 2024-02-09T17:26:52.694 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f779cd19:::benchmark_data_smithi179_77138_object124836:head# 2024-02-09T17:26:52.694 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f77acbb3:::benchmark_data_smithi179_77138_object100026:head# 2024-02-09T17:26:52.697 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f77bc716:::benchmark_data_smithi179_77138_object35263:head# 2024-02-09T17:26:52.697 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f77c48ee:::benchmark_data_smithi179_77138_object73223:head# 2024-02-09T17:26:52.698 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f77c574f:::benchmark_data_smithi179_77138_object58925:head# 2024-02-09T17:26:52.698 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f77d5c43:::benchmark_data_smithi179_77138_object113689:head# 2024-02-09T17:26:52.699 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f77d7b0e:::benchmark_data_smithi179_77138_object82784:head# 2024-02-09T17:26:52.699 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f77de48e:::benchmark_data_smithi179_77138_object25926:head# 2024-02-09T17:26:52.700 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f77e4e12:::benchmark_data_smithi179_77138_object127818:head# 2024-02-09T17:26:52.700 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f77e5bcc:::benchmark_data_smithi179_77138_object33667:head# 2024-02-09T17:26:52.701 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f77f013f:::benchmark_data_smithi179_77138_object41788:head# 2024-02-09T17:26:52.701 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f77f5188:::benchmark_data_smithi179_77138_object130313:head# 2024-02-09T17:26:52.701 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f77fb7bf:::benchmark_data_smithi179_77138_object31885:head# 2024-02-09T17:26:52.701 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7804850:::benchmark_data_smithi179_77138_object66571:head# 2024-02-09T17:26:52.702 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f780aa42:::benchmark_data_smithi179_77138_object23729:head# 2024-02-09T17:26:52.702 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7814243:::benchmark_data_smithi179_77138_object104796:head# 2024-02-09T17:26:52.703 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7829c0b:::benchmark_data_smithi179_77138_object82036:head# 2024-02-09T17:26:52.703 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7832bab:::benchmark_data_smithi179_77138_object77757:head# 2024-02-09T17:26:52.704 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f78331a7:::benchmark_data_smithi179_77138_object95196:head# 2024-02-09T17:26:52.704 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f783638f:::benchmark_data_smithi179_77138_object95786:head# 2024-02-09T17:26:52.704 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f784e830:::benchmark_data_smithi179_77138_object31748:head# 2024-02-09T17:26:52.705 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f786336e:::benchmark_data_smithi179_77138_object27031:head# 2024-02-09T17:26:52.705 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f78669ce:::benchmark_data_smithi179_77138_object98835:head# 2024-02-09T17:26:52.706 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7867440:::benchmark_data_smithi179_77138_object64773:head# 2024-02-09T17:26:52.706 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f786c978:::benchmark_data_smithi179_77138_object27303:head# 2024-02-09T17:26:52.706 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7873f6a:::benchmark_data_smithi179_77138_object53999:head# 2024-02-09T17:26:52.707 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f787b819:::benchmark_data_smithi179_77138_object132355:head# 2024-02-09T17:26:52.707 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7882fdd:::benchmark_data_smithi179_77138_object127473:head# 2024-02-09T17:26:52.707 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7893ac7:::benchmark_data_smithi179_77138_object130813:head# 2024-02-09T17:26:52.708 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f789e076:::benchmark_data_smithi179_77138_object95532:head# 2024-02-09T17:26:52.708 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f78a0461:::benchmark_data_smithi179_77138_object46776:head# 2024-02-09T17:26:52.709 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f78a6c79:::benchmark_data_smithi179_77138_object100964:head# 2024-02-09T17:26:52.709 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f78a7069:::benchmark_data_smithi179_77138_object119924:head# 2024-02-09T17:26:52.709 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f78b2389:::benchmark_data_smithi179_77138_object70801:head# 2024-02-09T17:26:52.712 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:26:52.717 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f78c9690:::benchmark_data_smithi179_77138_object85034:head# 2024-02-09T17:26:52.717 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f78d2eda:::benchmark_data_smithi179_77138_object30654:head# 2024-02-09T17:26:52.717 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f78da392:::benchmark_data_smithi179_77138_object25132:head# 2024-02-09T17:26:52.717 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f78db402:::benchmark_data_smithi179_77138_object95109:head# 2024-02-09T17:26:52.717 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f78e34b2:::benchmark_data_smithi179_77138_object60544:head# 2024-02-09T17:26:52.717 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f78e96ab:::benchmark_data_smithi179_77138_object19956:head# 2024-02-09T17:26:52.717 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f78eb87e:::benchmark_data_smithi179_77138_object81911:head# 2024-02-09T17:26:52.717 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f78f03f9:::benchmark_data_smithi179_77138_object29391:head# 2024-02-09T17:26:52.717 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f78f571c:::benchmark_data_smithi179_77138_object66982:head# 2024-02-09T17:26:52.717 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f78f583e:::benchmark_data_smithi179_77138_object57547:head# 2024-02-09T17:26:52.717 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f78fb029:::benchmark_data_smithi179_77138_object65047:head# 2024-02-09T17:26:52.717 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f790ba4a:::benchmark_data_smithi179_77138_object94276:head# 2024-02-09T17:26:52.718 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7911654:::benchmark_data_smithi179_77138_object43668:head# 2024-02-09T17:26:52.718 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f792e5ac:::benchmark_data_smithi179_77138_object103238:head# 2024-02-09T17:26:52.718 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f792e9ea:::benchmark_data_smithi179_77138_object96823:head# 2024-02-09T17:26:52.719 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7942114:::benchmark_data_smithi179_77138_object117055:head# 2024-02-09T17:26:52.720 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f794b5e9:::benchmark_data_smithi179_77138_object89862:head# 2024-02-09T17:26:52.720 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7958d92:::benchmark_data_smithi179_77138_object114130:head# 2024-02-09T17:26:52.722 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f798f2b0:::benchmark_data_smithi179_77138_object45500:head# 2024-02-09T17:26:52.722 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7999fce:::benchmark_data_smithi179_77138_object132762:head# 2024-02-09T17:26:52.722 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f799cb2d:::benchmark_data_smithi179_77138_object41067:head# 2024-02-09T17:26:52.723 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f79af97e:::benchmark_data_smithi179_77138_object98184:head# 2024-02-09T17:26:52.723 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f79ba04c:::benchmark_data_smithi179_77138_object21881:head# 2024-02-09T17:26:52.723 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f79bf031:::benchmark_data_smithi179_77138_object102641:head# 2024-02-09T17:26:52.723 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f79cb99d:::benchmark_data_smithi179_77138_object35457:head# 2024-02-09T17:26:52.724 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f79cc300:::benchmark_data_smithi179_77138_object122811:head# 2024-02-09T17:26:52.724 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f79d3c97:::benchmark_data_smithi179_77138_object54423:head# 2024-02-09T17:26:52.725 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f79d586b:::benchmark_data_smithi179_77138_object100472:head# 2024-02-09T17:26:52.725 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f79db0fd:::benchmark_data_smithi179_77138_object36430:head# 2024-02-09T17:26:52.726 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f79dec25:::benchmark_data_smithi179_77138_object89332:head# 2024-02-09T17:26:52.726 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f79dff49:::benchmark_data_smithi179_77138_object103902:head# 2024-02-09T17:26:52.727 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f79e22c6:::benchmark_data_smithi179_77138_object65107:head# 2024-02-09T17:26:52.727 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f79e3e64:::benchmark_data_smithi179_77138_object112314:head# 2024-02-09T17:26:52.727 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f79ea06a:::benchmark_data_smithi179_77138_object126509:head# 2024-02-09T17:26:52.728 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f79f20f8:::benchmark_data_smithi179_77138_object63437:head# 2024-02-09T17:26:52.728 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f79f89a5:::benchmark_data_smithi179_77138_object47706:head# 2024-02-09T17:26:52.729 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f79f94e9:::benchmark_data_smithi179_77138_object49654:head# 2024-02-09T17:26:52.729 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7a0cbaa:::benchmark_data_smithi179_77138_object88762:head# 2024-02-09T17:26:52.730 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7a0e9df:::benchmark_data_smithi179_77138_object60117:head# 2024-02-09T17:26:52.731 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7a2b12b:::benchmark_data_smithi179_77138_object83925:head# 2024-02-09T17:26:52.731 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7a2c09a:::benchmark_data_smithi179_77138_object74260:head# 2024-02-09T17:26:52.731 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7a361d8:::benchmark_data_smithi179_77138_object26294:head# 2024-02-09T17:26:52.732 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7a3a430:::benchmark_data_smithi179_77138_object24612:head# 2024-02-09T17:26:52.737 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7a466ab:::benchmark_data_smithi179_77138_object61602:head# 2024-02-09T17:26:52.737 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7a4a0b1:::benchmark_data_smithi179_77138_object117146:head# 2024-02-09T17:26:52.737 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7a54c94:::benchmark_data_smithi179_77138_object120790:head# 2024-02-09T17:26:52.737 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7a5f623:::benchmark_data_smithi179_77138_object127744:head# 2024-02-09T17:26:52.737 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7a66ba5:::benchmark_data_smithi179_77138_object89025:head# 2024-02-09T17:26:52.737 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7a67fd8:::benchmark_data_smithi179_77138_object111337:head# 2024-02-09T17:26:52.737 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7a7149b:::benchmark_data_smithi179_77138_object41916:head# 2024-02-09T17:26:52.737 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7a736cd:::benchmark_data_smithi179_77138_object112783:head# 2024-02-09T17:26:52.738 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7a957dc:::benchmark_data_smithi179_77138_object43157:head# 2024-02-09T17:26:52.738 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7ab0dc9:::benchmark_data_smithi179_77138_object64656:head# 2024-02-09T17:26:52.739 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7ab6e9c:::benchmark_data_smithi179_77138_object36363:head# 2024-02-09T17:26:52.740 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7ab8af5:::benchmark_data_smithi179_77138_object76305:head# 2024-02-09T17:26:52.740 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7abca98:::benchmark_data_smithi179_77138_object29689:head# 2024-02-09T17:26:52.741 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7acb71f:::benchmark_data_smithi179_77138_object78367:head# 2024-02-09T17:26:52.741 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7ace351:::benchmark_data_smithi179_77138_object76387:head# 2024-02-09T17:26:52.741 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7ade12d:::benchmark_data_smithi179_77138_object114493:head# 2024-02-09T17:26:52.742 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7ae4225:::benchmark_data_smithi179_77138_object53284:head# 2024-02-09T17:26:52.742 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7ae9469:::benchmark_data_smithi179_77138_object51661:head# 2024-02-09T17:26:52.742 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7af0ea8:::benchmark_data_smithi179_77138_object76012:head# 2024-02-09T17:26:52.743 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7af12ca:::benchmark_data_smithi179_77138_object104138:head# 2024-02-09T17:26:52.743 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7af48c0:::benchmark_data_smithi179_77138_object57996:head# 2024-02-09T17:26:52.744 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7af8d6d:::benchmark_data_smithi179_77138_object38664:head# 2024-02-09T17:26:52.744 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7af91df:::benchmark_data_smithi179_77138_object50562:head# 2024-02-09T17:26:52.745 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7b085e0:::benchmark_data_smithi179_77138_object132438:head# 2024-02-09T17:26:52.745 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7b201da:::benchmark_data_smithi179_77138_object132691:head# 2024-02-09T17:26:52.746 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7b26d82:::benchmark_data_smithi179_77138_object38292:head# 2024-02-09T17:26:52.746 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7b2bd9d:::benchmark_data_smithi179_77138_object119491:head# 2024-02-09T17:26:52.746 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7b3fbad:::benchmark_data_smithi179_77138_object116899:head# 2024-02-09T17:26:52.747 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7b4c097:::benchmark_data_smithi179_77138_object48665:head# 2024-02-09T17:26:52.749 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7b579b8:::benchmark_data_smithi179_77138_object43986:head# 2024-02-09T17:26:52.749 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7b5855d:::benchmark_data_smithi179_77138_object31342:head# 2024-02-09T17:26:52.750 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7b6e94c:::benchmark_data_smithi179_77138_object110144:head# 2024-02-09T17:26:52.750 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7b7cdbb:::benchmark_data_smithi179_77138_object84567:head# 2024-02-09T17:26:52.751 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:52.751 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7b84d06:::benchmark_data_smithi179_77138_object92260:head# 2024-02-09T17:26:52.751 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7b85177:::benchmark_data_smithi179_77138_object122685:head# 2024-02-09T17:26:52.751 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7b8b0cd:::benchmark_data_smithi179_77138_object77540:head# 2024-02-09T17:26:52.752 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7b90064:::benchmark_data_smithi179_77138_object62468:head# 2024-02-09T17:26:52.752 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7b92e3b:::benchmark_data_smithi179_77138_object32667:head# 2024-02-09T17:26:52.753 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7b99b98:::benchmark_data_smithi179_77138_object23189:head# 2024-02-09T17:26:52.755 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7b9acda:::benchmark_data_smithi179_77138_object27470:head# 2024-02-09T17:26:52.755 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7baac7d:::benchmark_data_smithi179_77138_object74299:head# 2024-02-09T17:26:52.759 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7bac92a:::benchmark_data_smithi179_77138_object116465:head# 2024-02-09T17:26:52.759 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7bb53bd:::benchmark_data_smithi179_77138_object35824:head# 2024-02-09T17:26:52.759 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7bc0cc8:::benchmark_data_smithi179_77138_object49078:head# 2024-02-09T17:26:52.759 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7bc6979:::benchmark_data_smithi179_77138_object23747:head# 2024-02-09T17:26:52.759 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7bdb403:::benchmark_data_smithi179_77138_object120174:head# 2024-02-09T17:26:52.759 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7bdead9:::benchmark_data_smithi179_77138_object56912:head# 2024-02-09T17:26:52.760 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7bdec3b:::benchmark_data_smithi179_77138_object54587:head# 2024-02-09T17:26:52.760 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7beaf65:::benchmark_data_smithi179_77138_object58875:head# 2024-02-09T17:26:52.760 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7c02f05:::benchmark_data_smithi179_77138_object35826:head# 2024-02-09T17:26:52.760 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7c0ff0c:::benchmark_data_smithi179_77138_object105083:head# 2024-02-09T17:26:52.761 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7c15aec:::benchmark_data_smithi179_77138_object71636:head# 2024-02-09T17:26:52.761 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7c1cfe7:::benchmark_data_smithi179_77138_object75923:head# 2024-02-09T17:26:52.762 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7c1ebe9:::benchmark_data_smithi179_77138_object115027:head# 2024-02-09T17:26:52.762 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7c470a4:::benchmark_data_smithi179_77138_object80253:head# 2024-02-09T17:26:52.763 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7c4e0c5:::benchmark_data_smithi179_77138_object50898:head# 2024-02-09T17:26:52.763 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7c4e264:::benchmark_data_smithi179_77138_object23995:head# 2024-02-09T17:26:52.764 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7c5de9e:::benchmark_data_smithi179_77138_object104011:head# 2024-02-09T17:26:52.765 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7c6108a:::benchmark_data_smithi179_77138_object105565:head# 2024-02-09T17:26:52.766 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7c61967:::benchmark_data_smithi179_77138_object110701:head# 2024-02-09T17:26:52.766 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7c65158:::benchmark_data_smithi179_77138_object20212:head# 2024-02-09T17:26:52.767 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7c7bb81:::benchmark_data_smithi179_77138_object130239:head# 2024-02-09T17:26:52.767 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7c81bf8:::benchmark_data_smithi179_77138_object109513:head# 2024-02-09T17:26:52.767 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7c8dffd:::benchmark_data_smithi179_77138_object101920:head# 2024-02-09T17:26:52.768 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7c96a2a:::benchmark_data_smithi179_77138_object67106:head# 2024-02-09T17:26:52.768 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7c99280:::benchmark_data_smithi179_77138_object40991:head# 2024-02-09T17:26:52.769 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7ca33ab:::benchmark_data_smithi179_77138_object129915:head# 2024-02-09T17:26:52.769 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7cac746:::benchmark_data_smithi179_77138_object41533:head# 2024-02-09T17:26:52.772 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7cb8b5b:::benchmark_data_smithi179_77138_object100929:head# 2024-02-09T17:26:52.772 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7ccedde:::benchmark_data_smithi179_77138_object103122:head# 2024-02-09T17:26:52.772 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7ccee8c:::benchmark_data_smithi179_77138_object92918:head# 2024-02-09T17:26:52.772 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7cd279a:::benchmark_data_smithi179_77138_object37245:head# 2024-02-09T17:26:52.772 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7cd9645:::benchmark_data_smithi179_77138_object25824:head# 2024-02-09T17:26:52.773 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7cde14f:::benchmark_data_smithi179_77138_object35038:head# 2024-02-09T17:26:52.773 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7ce7983:::benchmark_data_smithi179_77138_object24976:head# 2024-02-09T17:26:52.773 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7d1f8b9:::benchmark_data_smithi179_77138_object52592:head# 2024-02-09T17:26:52.774 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7d3289b:::benchmark_data_smithi179_77138_object94166:head# 2024-02-09T17:26:52.774 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7d482a7:::benchmark_data_smithi179_77138_object22807:head# 2024-02-09T17:26:52.775 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7d4dec6:::benchmark_data_smithi179_77138_object78895:head# 2024-02-09T17:26:52.779 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:26:52.786 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7d541f3:::benchmark_data_smithi179_77138_object129735:head# 2024-02-09T17:26:52.786 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7d5988a:::benchmark_data_smithi179_77138_object72051:head# 2024-02-09T17:26:52.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:52.788 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:26:52.788 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7d5af5e:::benchmark_data_smithi179_77138_object79412:head# 2024-02-09T17:26:52.788 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7d5b998:::benchmark_data_smithi179_77138_object108699:head# 2024-02-09T17:26:52.789 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7d6a525:::benchmark_data_smithi179_77138_object112717:head# 2024-02-09T17:26:52.789 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7d6e6e6:::benchmark_data_smithi179_77138_object107249:head# 2024-02-09T17:26:52.789 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7d80cc5:::benchmark_data_smithi179_77138_object110916:head# 2024-02-09T17:26:52.789 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7d9492f:::benchmark_data_smithi179_77138_object85889:head# 2024-02-09T17:26:52.789 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7da22d2:::benchmark_data_smithi179_77138_object119606:head# 2024-02-09T17:26:52.789 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7da29eb:::benchmark_data_smithi179_77138_object47039:head# 2024-02-09T17:26:52.794 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7da4744:::benchmark_data_smithi179_77138_object39549:head# 2024-02-09T17:26:52.794 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7dad0cc:::benchmark_data_smithi179_77138_object67313:head# 2024-02-09T17:26:52.794 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7db3a7a:::benchmark_data_smithi179_77138_object46774:head# 2024-02-09T17:26:52.794 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7db4504:::benchmark_data_smithi179_77138_object61934:head# 2024-02-09T17:26:52.794 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7db933c:::benchmark_data_smithi179_77138_object109949:head# 2024-02-09T17:26:52.798 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7db9d4f:::benchmark_data_smithi179_77138_object125761:head# 2024-02-09T17:26:52.798 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7dba184:::benchmark_data_smithi179_77138_object73095:head# 2024-02-09T17:26:52.799 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7dba34e:::benchmark_data_smithi179_77138_object117551:head# 2024-02-09T17:26:52.799 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7dcc669:::benchmark_data_smithi179_77138_object78202:head# 2024-02-09T17:26:52.799 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7dd8a72:::benchmark_data_smithi179_77138_object65574:head# 2024-02-09T17:26:52.802 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7deb09c:::benchmark_data_smithi179_77138_object110988:head# 2024-02-09T17:26:52.802 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7decc30:::benchmark_data_smithi179_77138_object64464:head# 2024-02-09T17:26:52.802 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7df6263:::benchmark_data_smithi179_77138_object46363:head# 2024-02-09T17:26:52.802 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7df7a49:::benchmark_data_smithi179_77138_object111180:head# 2024-02-09T17:26:52.803 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7e162d8:::benchmark_data_smithi179_77138_object106957:head# 2024-02-09T17:26:52.803 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7e1870e:::benchmark_data_smithi179_77138_object24673:head# 2024-02-09T17:26:52.803 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7e26037:::benchmark_data_smithi179_77138_object106388:head# 2024-02-09T17:26:52.803 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7e33452:::benchmark_data_smithi179_77138_object101118:head# 2024-02-09T17:26:52.804 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7e35cbe:::benchmark_data_smithi179_77138_object96013:head# 2024-02-09T17:26:52.804 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7e87529:::benchmark_data_smithi179_77138_object48232:head# 2024-02-09T17:26:52.804 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7e9b763:::benchmark_data_smithi179_77138_object43102:head# 2024-02-09T17:26:52.804 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7e9c745:::benchmark_data_smithi179_77138_object24557:head# 2024-02-09T17:26:52.804 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7e9ec8a:::benchmark_data_smithi179_77138_object128448:head# 2024-02-09T17:26:52.804 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7e9fa56:::benchmark_data_smithi179_77138_object96083:head# 2024-02-09T17:26:52.805 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7ea24ce:::benchmark_data_smithi179_77138_object93102:head# 2024-02-09T17:26:52.805 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7eb99f9:::benchmark_data_smithi179_77138_object32941:head# 2024-02-09T17:26:52.805 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7ed9f08:::benchmark_data_smithi179_77138_object38148:head# 2024-02-09T17:26:52.806 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7ee27cd:::benchmark_data_smithi179_77138_object77951:head# 2024-02-09T17:26:52.806 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7ee7400:::benchmark_data_smithi179_77138_object98516:head# 2024-02-09T17:26:52.807 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7f03fae:::benchmark_data_smithi179_77138_object120050:head# 2024-02-09T17:26:52.807 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7f09fd5:::benchmark_data_smithi179_77138_object132700:head# 2024-02-09T17:26:52.808 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7f1272d:::benchmark_data_smithi179_77138_object29954:head# 2024-02-09T17:26:52.808 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7f12e3b:::benchmark_data_smithi179_77138_object44012:head# 2024-02-09T17:26:52.808 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7f15d62:::benchmark_data_smithi179_77138_object83029:head# 2024-02-09T17:26:52.809 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7f19ddc:::benchmark_data_smithi179_77138_object37581:head# 2024-02-09T17:26:52.809 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7f26f1e:::benchmark_data_smithi179_77138_object43566:head# 2024-02-09T17:26:52.810 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7f289b1:::benchmark_data_smithi179_77138_object62859:head# 2024-02-09T17:26:52.810 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7f2fb93:::benchmark_data_smithi179_77138_object94745:head# 2024-02-09T17:26:52.810 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7f3b709:::benchmark_data_smithi179_77138_object42722:head# 2024-02-09T17:26:52.811 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7f3fedd:::benchmark_data_smithi179_77138_object121933:head# 2024-02-09T17:26:52.811 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7f529d8:::benchmark_data_smithi179_77138_object84338:head# 2024-02-09T17:26:52.812 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7f53255:::benchmark_data_smithi179_77138_object29617:head# 2024-02-09T17:26:52.812 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7f5dac2:::benchmark_data_smithi179_77138_object84106:head# 2024-02-09T17:26:52.813 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7f76feb:::benchmark_data_smithi179_77138_object72983:head# 2024-02-09T17:26:52.813 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7f7aafe:::benchmark_data_smithi179_77138_object115463:head# 2024-02-09T17:26:52.813 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7f7d9fa:::benchmark_data_smithi179_77138_object66765:head# 2024-02-09T17:26:52.814 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7f7fb5b:::benchmark_data_smithi179_77138_object37912:head# 2024-02-09T17:26:52.814 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7f91965:::benchmark_data_smithi179_77138_object84130:head# 2024-02-09T17:26:52.815 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7faa3a1:::benchmark_data_smithi179_77138_object117663:head# 2024-02-09T17:26:52.815 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7faeb9b:::benchmark_data_smithi179_77138_object104880:head# 2024-02-09T17:26:52.815 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7fb16cb:::benchmark_data_smithi179_77138_object121228:head# 2024-02-09T17:26:52.816 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7fb5a8b:::benchmark_data_smithi179_77138_object53196:head# 2024-02-09T17:26:52.816 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7fb6ab7:::benchmark_data_smithi179_77138_object101641:head# 2024-02-09T17:26:52.817 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7fd5cca:::benchmark_data_smithi179_77138_object77894:head# 2024-02-09T17:26:52.817 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7fdbbd9:::benchmark_data_smithi179_77138_object114099:head# 2024-02-09T17:26:52.817 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7fe9bbd:::benchmark_data_smithi179_77138_object109215:head# 2024-02-09T17:26:52.818 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7fed295:::benchmark_data_smithi179_77138_object85035:head# 2024-02-09T17:26:52.818 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7fed35f:::benchmark_data_smithi179_77138_object21248:head# 2024-02-09T17:26:52.819 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f7ff1511:::benchmark_data_smithi179_77138_object31895:head# 2024-02-09T17:26:52.819 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8018948:::benchmark_data_smithi179_77138_object83910:head# 2024-02-09T17:26:52.820 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f801aa65:::benchmark_data_smithi179_77138_object82190:head# 2024-02-09T17:26:52.820 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f801d357:::benchmark_data_smithi179_77138_object48083:head# 2024-02-09T17:26:52.820 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f801fb74:::benchmark_data_smithi179_77138_object106792:head# 2024-02-09T17:26:52.821 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8051aff:::benchmark_data_smithi179_77138_object31829:head# 2024-02-09T17:26:52.821 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f80556b1:::benchmark_data_smithi179_77138_object43136:head# 2024-02-09T17:26:52.822 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f805d6d6:::benchmark_data_smithi179_77138_object21579:head# 2024-02-09T17:26:52.822 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8061503:::benchmark_data_smithi179_77138_object103283:head# 2024-02-09T17:26:52.822 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8066fcf:::benchmark_data_smithi179_77138_object122178:head# 2024-02-09T17:26:52.823 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f806a61d:::benchmark_data_smithi179_77138_object81536:head# 2024-02-09T17:26:52.823 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f806d8c9:::benchmark_data_smithi179_77138_object80034:head# 2024-02-09T17:26:52.824 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f809d402:::benchmark_data_smithi179_77138_object98567:head# 2024-02-09T17:26:52.824 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f80b1f6f:::benchmark_data_smithi179_77138_object118664:head# 2024-02-09T17:26:52.825 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f80b7205:::benchmark_data_smithi179_77138_object19375:head# 2024-02-09T17:26:52.825 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f80b9a4d:::benchmark_data_smithi179_77138_object42420:head# 2024-02-09T17:26:52.825 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f80be2b2:::benchmark_data_smithi179_77138_object59095:head# 2024-02-09T17:26:52.826 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f80d3bad:::benchmark_data_smithi179_77138_object77259:head# 2024-02-09T17:26:52.826 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f80e8979:::benchmark_data_smithi179_77138_object92437:head# 2024-02-09T17:26:52.827 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f80ed599:::benchmark_data_smithi179_77138_object38296:head# 2024-02-09T17:26:52.827 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f80f002c:::benchmark_data_smithi179_77138_object76722:head# 2024-02-09T17:26:52.827 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f80f43ed:::benchmark_data_smithi179_77138_object96084:head# 2024-02-09T17:26:52.828 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f810617b:::benchmark_data_smithi179_77138_object20618:head# 2024-02-09T17:26:52.828 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f813459f:::benchmark_data_smithi179_77138_object55052:head# 2024-02-09T17:26:52.829 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f81499af:::benchmark_data_smithi179_77138_object36422:head# 2024-02-09T17:26:52.829 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f814ffab:::benchmark_data_smithi179_77138_object51757:head# 2024-02-09T17:26:52.829 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8157963:::benchmark_data_smithi179_77138_object71150:head# 2024-02-09T17:26:52.830 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8159f7b:::benchmark_data_smithi179_77138_object46157:head# 2024-02-09T17:26:52.830 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f815efa7:::benchmark_data_smithi179_77138_object41206:head# 2024-02-09T17:26:52.831 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8160bc4:::benchmark_data_smithi179_77138_object39813:head# 2024-02-09T17:26:52.831 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8162db2:::benchmark_data_smithi179_77138_object114348:head# 2024-02-09T17:26:52.831 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8163623:::benchmark_data_smithi179_77138_object55332:head# 2024-02-09T17:26:52.832 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f816c893:::benchmark_data_smithi179_77138_object88560:head# 2024-02-09T17:26:52.832 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8174329:::benchmark_data_smithi179_77138_object120505:head# 2024-02-09T17:26:52.833 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8178970:::benchmark_data_smithi179_77138_object57603:head# 2024-02-09T17:26:52.833 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f818bad6:::benchmark_data_smithi179_77138_object28462:head# 2024-02-09T17:26:52.833 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f819b138:::benchmark_data_smithi179_77138_object94576:head# 2024-02-09T17:26:52.834 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f819f30f:::benchmark_data_smithi179_77138_object97121:head# 2024-02-09T17:26:52.834 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f81a0b49:::benchmark_data_smithi179_77138_object101666:head# 2024-02-09T17:26:52.837 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f81a6761:::benchmark_data_smithi179_77138_object45285:head# 2024-02-09T17:26:52.837 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f81a888e:::benchmark_data_smithi179_77138_object35589:head# 2024-02-09T17:26:52.837 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f81a893c:::benchmark_data_smithi179_77138_object20507:head# 2024-02-09T17:26:52.837 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f81b2ef0:::benchmark_data_smithi179_77138_object95261:head# 2024-02-09T17:26:52.838 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f81c0c29:::benchmark_data_smithi179_77138_object95429:head# 2024-02-09T17:26:52.838 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f81c106d:::benchmark_data_smithi179_77138_object44541:head# 2024-02-09T17:26:52.839 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f81cb7df:::benchmark_data_smithi179_77138_object42881:head# 2024-02-09T17:26:52.839 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f81d0c5f:::benchmark_data_smithi179_77138_object75649:head# 2024-02-09T17:26:52.839 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f81de696:::benchmark_data_smithi179_77138_object82737:head# 2024-02-09T17:26:52.839 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f81deab3:::benchmark_data_smithi179_77138_object87145:head# 2024-02-09T17:26:52.840 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f81e59bc:::benchmark_data_smithi179_77138_object46733:head# 2024-02-09T17:26:52.840 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f81e74bd:::benchmark_data_smithi179_77138_object127976:head# 2024-02-09T17:26:52.840 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f81ff3c2:::benchmark_data_smithi179_77138_object67584:head# 2024-02-09T17:26:52.841 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8201302:::benchmark_data_smithi179_77138_object28164:head# 2024-02-09T17:26:52.841 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f82197c5:::benchmark_data_smithi179_77138_object123048:head# 2024-02-09T17:26:52.841 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f821e891:::benchmark_data_smithi179_77138_object117463:head# 2024-02-09T17:26:52.842 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f821eab2:::benchmark_data_smithi179_77138_object39761:head# 2024-02-09T17:26:52.847 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f822a915:::benchmark_data_smithi179_77138_object54416:head# 2024-02-09T17:26:52.847 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8238098:::benchmark_data_smithi179_77138_object34350:head# 2024-02-09T17:26:52.847 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f823bdaa:::benchmark_data_smithi179_77138_object90201:head# 2024-02-09T17:26:52.847 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f825198b:::benchmark_data_smithi179_77138_object40878:head# 2024-02-09T17:26:52.847 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f826194c:::benchmark_data_smithi179_77138_object108792:head# 2024-02-09T17:26:52.847 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8261b34:::benchmark_data_smithi179_77138_object37826:head# 2024-02-09T17:26:52.849 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f826903c:::benchmark_data_smithi179_77138_object39316:head# 2024-02-09T17:26:52.849 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8289f91:::benchmark_data_smithi179_77138_object80291:head# 2024-02-09T17:26:52.850 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f828d227:::benchmark_data_smithi179_77138_object50964:head# 2024-02-09T17:26:52.850 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f82957f4:::benchmark_data_smithi179_77138_object99070:head# 2024-02-09T17:26:52.850 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f829f1c8:::benchmark_data_smithi179_77138_object98102:head# 2024-02-09T17:26:52.850 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f82a5366:::benchmark_data_smithi179_77138_object58896:head# 2024-02-09T17:26:52.850 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f82c3b9c:::benchmark_data_smithi179_77138_object24915:head# 2024-02-09T17:26:52.850 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f82cfe20:::benchmark_data_smithi179_77138_object24227:head# 2024-02-09T17:26:52.850 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f82d5f91:::benchmark_data_smithi179_77138_object31121:head# 2024-02-09T17:26:52.850 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f82dd04b:::benchmark_data_smithi179_77138_object79821:head# 2024-02-09T17:26:52.850 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f82eba07:::benchmark_data_smithi179_77138_object130329:head# 2024-02-09T17:26:52.851 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8301ca6:::benchmark_data_smithi179_77138_object26291:head# 2024-02-09T17:26:52.851 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f830ab58:::benchmark_data_smithi179_77138_object106985:head# 2024-02-09T17:26:52.851 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f830ca2b:::benchmark_data_smithi179_77138_object32509:head# 2024-02-09T17:26:52.852 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f830e8eb:::benchmark_data_smithi179_77138_object88292:head# 2024-02-09T17:26:52.852 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8318f41:::benchmark_data_smithi179_77138_object124596:head# 2024-02-09T17:26:52.853 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f831c25b:::benchmark_data_smithi179_77138_object101952:head# 2024-02-09T17:26:52.853 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f832b458:::benchmark_data_smithi179_77138_object85507:head# 2024-02-09T17:26:52.853 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f832bb17:::benchmark_data_smithi179_77138_object82607:head# 2024-02-09T17:26:52.854 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f83344ba:::benchmark_data_smithi179_77138_object19598:head# 2024-02-09T17:26:52.854 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f83439cf:::benchmark_data_smithi179_77138_object60009:head# 2024-02-09T17:26:52.855 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8347c0c:::benchmark_data_smithi179_77138_object91641:head# 2024-02-09T17:26:52.855 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8355966:::benchmark_data_smithi179_77138_object29623:head# 2024-02-09T17:26:52.856 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8358ff1:::benchmark_data_smithi179_77138_object120827:head# 2024-02-09T17:26:52.856 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f83598ae:::benchmark_data_smithi179_77138_object53395:head# 2024-02-09T17:26:52.857 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f835b7d9:::benchmark_data_smithi179_77138_object78746:head# 2024-02-09T17:26:52.857 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f83634ef:::benchmark_data_smithi179_77138_object96425:head# 2024-02-09T17:26:52.857 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8373f0a:::benchmark_data_smithi179_77138_object49053:head# 2024-02-09T17:26:52.859 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f83755c4:::benchmark_data_smithi179_77138_object116869:head# 2024-02-09T17:26:52.859 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f837ba09:::benchmark_data_smithi179_77138_object68356:head# 2024-02-09T17:26:52.860 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f837e48d:::benchmark_data_smithi179_77138_object78105:head# 2024-02-09T17:26:52.860 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f838055d:::benchmark_data_smithi179_77138_object110587:head# 2024-02-09T17:26:52.861 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f83878d3:::benchmark_data_smithi179_77138_object22475:head# 2024-02-09T17:26:52.861 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f838d71a:::benchmark_data_smithi179_77138_object53559:head# 2024-02-09T17:26:52.862 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f838ebfe:::benchmark_data_smithi179_77138_object86612:head# 2024-02-09T17:26:52.862 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f839c5b5:::benchmark_data_smithi179_77138_object67571:head# 2024-02-09T17:26:52.862 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f839d430:::benchmark_data_smithi179_77138_object56492:head# 2024-02-09T17:26:52.863 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f83b79d4:::benchmark_data_smithi179_77138_object125103:head# 2024-02-09T17:26:52.863 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f83c4bbb:::benchmark_data_smithi179_77138_object112141:head# 2024-02-09T17:26:52.864 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f83c4cc8:::benchmark_data_smithi179_77138_object80947:head# 2024-02-09T17:26:52.864 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f83c930b:::benchmark_data_smithi179_77138_object53135:head# 2024-02-09T17:26:52.865 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f83df600:::benchmark_data_smithi179_77138_object77158:head# 2024-02-09T17:26:52.865 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f83e0ba5:::benchmark_data_smithi179_77138_object104348:head# 2024-02-09T17:26:52.865 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f83f06de:::benchmark_data_smithi179_77138_object85138:head# 2024-02-09T17:26:52.866 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f83f5983:::benchmark_data_smithi179_77138_object113789:head# 2024-02-09T17:26:52.866 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f83f64b1:::benchmark_data_smithi179_77138_object130207:head# 2024-02-09T17:26:52.866 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f83ffb59:::benchmark_data_smithi179_77138_object82920:head# 2024-02-09T17:26:52.867 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f841358b:::benchmark_data_smithi179_77138_object85202:head# 2024-02-09T17:26:52.867 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f84179ab:::benchmark_data_smithi179_77138_object81997:head# 2024-02-09T17:26:52.868 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8427b9c:::benchmark_data_smithi179_77138_object62770:head# 2024-02-09T17:26:52.868 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8428c22:::benchmark_data_smithi179_77138_object57473:head# 2024-02-09T17:26:52.869 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f842ac57:::benchmark_data_smithi179_77138_object61776:head# 2024-02-09T17:26:52.869 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f84305b7:::benchmark_data_smithi179_77138_object61964:head# 2024-02-09T17:26:52.870 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f843b837:::benchmark_data_smithi179_77138_object67436:head# 2024-02-09T17:26:52.870 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f843c326:::benchmark_data_smithi179_77138_object74510:head# 2024-02-09T17:26:52.870 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8441794:::benchmark_data_smithi179_77138_object119910:head# 2024-02-09T17:26:52.871 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f844a5e6:::benchmark_data_smithi179_77138_object99761:head# 2024-02-09T17:26:52.871 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f844ac3d:::benchmark_data_smithi179_77138_object77201:head# 2024-02-09T17:26:52.871 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f845bf17:::benchmark_data_smithi179_77138_object88566:head# 2024-02-09T17:26:52.872 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f845ed62:::benchmark_data_smithi179_77138_object130902:head# 2024-02-09T17:26:52.872 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8469890:::benchmark_data_smithi179_77138_object98078:head# 2024-02-09T17:26:52.873 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8470f94:::benchmark_data_smithi179_77138_object29480:head# 2024-02-09T17:26:52.873 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f847371e:::benchmark_data_smithi179_77138_object69983:head# 2024-02-09T17:26:52.874 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f84799cc:::benchmark_data_smithi179_77138_object34898:head# 2024-02-09T17:26:52.874 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8482a0c:::benchmark_data_smithi179_77138_object82720:head# 2024-02-09T17:26:52.876 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f849c669:::benchmark_data_smithi179_77138_object59613:head# 2024-02-09T17:26:52.876 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f849e807:::benchmark_data_smithi179_77138_object81922:head# 2024-02-09T17:26:52.876 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f84a64c5:::benchmark_data_smithi179_77138_object25681:head# 2024-02-09T17:26:52.878 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f84a7a79:::benchmark_data_smithi179_77138_object94624:head# 2024-02-09T17:26:52.878 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f84b3139:::benchmark_data_smithi179_77138_object75082:head# 2024-02-09T17:26:52.878 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f84bb30d:::benchmark_data_smithi179_77138_object100409:head# 2024-02-09T17:26:52.879 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f84cc341:::benchmark_data_smithi179_77138_object22960:head# 2024-02-09T17:26:52.879 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f84d7050:::benchmark_data_smithi179_77138_object53636:head# 2024-02-09T17:26:52.880 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f84ea705:::benchmark_data_smithi179_77138_object121856:head# 2024-02-09T17:26:52.880 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f84f0193:::benchmark_data_smithi179_77138_object24681:head# 2024-02-09T17:26:52.880 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f84f52b1:::benchmark_data_smithi179_77138_object28343:head# 2024-02-09T17:26:52.883 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f84f8a6d:::benchmark_data_smithi179_77138_object104512:head# 2024-02-09T17:26:52.885 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f84f926e:::benchmark_data_smithi179_77138_object65308:head# 2024-02-09T17:26:52.885 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f84fe504:::benchmark_data_smithi179_77138_object102308:head# 2024-02-09T17:26:52.885 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8504e99:::benchmark_data_smithi179_77138_object37694:head# 2024-02-09T17:26:52.885 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f850b071:::benchmark_data_smithi179_77138_object124551:head# 2024-02-09T17:26:52.886 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f851cb64:::benchmark_data_smithi179_77138_object23137:head# 2024-02-09T17:26:52.888 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:26:52.889 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f851fb37:::benchmark_data_smithi179_77138_object25964:head# 2024-02-09T17:26:52.889 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f85203a3:::benchmark_data_smithi179_77138_object57219:head# 2024-02-09T17:26:52.889 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f852b656:::benchmark_data_smithi179_77138_object41273:head# 2024-02-09T17:26:52.890 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f852ff9e:::benchmark_data_smithi179_77138_object73183:head# 2024-02-09T17:26:52.890 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f853a8ad:::benchmark_data_smithi179_77138_object54599:head# 2024-02-09T17:26:52.890 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f853e3d1:::benchmark_data_smithi179_77138_object55211:head# 2024-02-09T17:26:52.891 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8541611:::benchmark_data_smithi179_77138_object67741:head# 2024-02-09T17:26:52.892 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f85485b4:::benchmark_data_smithi179_77138_object34701:head# 2024-02-09T17:26:52.892 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8549a38:::benchmark_data_smithi179_77138_object108614:head# 2024-02-09T17:26:52.892 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f855dbbe:::benchmark_data_smithi179_77138_object96154:head# 2024-02-09T17:26:52.893 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f856b42b:::benchmark_data_smithi179_77138_object106721:head# 2024-02-09T17:26:52.893 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f85780ac:::benchmark_data_smithi179_77138_object131461:head# 2024-02-09T17:26:52.893 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f858af99:::benchmark_data_smithi179_77138_object61880:head# 2024-02-09T17:26:52.894 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8599ed7:::benchmark_data_smithi179_77138_object81897:head# 2024-02-09T17:26:52.895 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f85a0660:::benchmark_data_smithi179_77138_object29024:head# 2024-02-09T17:26:52.895 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f85a0887:::benchmark_data_smithi179_77138_object70677:head# 2024-02-09T17:26:52.895 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f85a4c93:::benchmark_data_smithi179_77138_object129717:head# 2024-02-09T17:26:52.895 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f85a797f:::benchmark_data_smithi179_77138_object56410:head# 2024-02-09T17:26:52.896 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f85c217f:::benchmark_data_smithi179_77138_object75446:head# 2024-02-09T17:26:52.896 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f85cfb17:::benchmark_data_smithi179_77138_object43897:head# 2024-02-09T17:26:52.903 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f85d8494:::benchmark_data_smithi179_77138_object77136:head# 2024-02-09T17:26:52.903 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f85da420:::benchmark_data_smithi179_77138_object42658:head# 2024-02-09T17:26:52.903 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f85db365:::benchmark_data_smithi179_77138_object73976:head# 2024-02-09T17:26:52.903 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f85e15cd:::benchmark_data_smithi179_77138_object72981:head# 2024-02-09T17:26:52.903 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f85e19ac:::benchmark_data_smithi179_77138_object74439:head# 2024-02-09T17:26:52.904 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f85e7f48:::benchmark_data_smithi179_77138_object77170:head# 2024-02-09T17:26:52.904 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f85eecf3:::benchmark_data_smithi179_77138_object118497:head# 2024-02-09T17:26:52.904 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f85f2a97:::benchmark_data_smithi179_77138_object100996:head# 2024-02-09T17:26:52.905 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f85f900a:::benchmark_data_smithi179_77138_object50349:head# 2024-02-09T17:26:52.905 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f861004c:::benchmark_data_smithi179_77138_object105010:head# 2024-02-09T17:26:52.906 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8617942:::benchmark_data_smithi179_77138_object21148:head# 2024-02-09T17:26:52.906 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f861c3c3:::benchmark_data_smithi179_77138_object100587:head# 2024-02-09T17:26:52.907 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f86256f7:::benchmark_data_smithi179_77138_object130887:head# 2024-02-09T17:26:52.907 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8626382:::benchmark_data_smithi179_77138_object75895:head# 2024-02-09T17:26:52.908 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8628f50:::benchmark_data_smithi179_77138_object125951:head# 2024-02-09T17:26:52.908 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f863df36:::benchmark_data_smithi179_77138_object31637:head# 2024-02-09T17:26:52.908 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f863f53f:::benchmark_data_smithi179_77138_object34196:head# 2024-02-09T17:26:52.909 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f864ec36:::benchmark_data_smithi179_77138_object68506:head# 2024-02-09T17:26:52.909 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f865bf8f:::benchmark_data_smithi179_77138_object83149:head# 2024-02-09T17:26:52.910 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f865d1ee:::benchmark_data_smithi179_77138_object51109:head# 2024-02-09T17:26:52.912 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f866034f:::benchmark_data_smithi179_77138_object56850:head# 2024-02-09T17:26:52.912 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8663173:::benchmark_data_smithi179_77138_object59940:head# 2024-02-09T17:26:52.916 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f866392f:::benchmark_data_smithi179_77138_object71107:head# 2024-02-09T17:26:52.916 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8674f48:::benchmark_data_smithi179_77138_object121894:head# 2024-02-09T17:26:52.916 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8679a54:::benchmark_data_smithi179_77138_object50482:head# 2024-02-09T17:26:52.916 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f86867a7:::benchmark_data_smithi179_77138_object77286:head# 2024-02-09T17:26:52.916 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f868d28d:::benchmark_data_smithi179_77138_object46083:head# 2024-02-09T17:26:52.916 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f868dddc:::benchmark_data_smithi179_77138_object103539:head# 2024-02-09T17:26:52.916 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f869129f:::benchmark_data_smithi179_77138_object20592:head# 2024-02-09T17:26:52.916 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8694796:::benchmark_data_smithi179_77138_object119351:head# 2024-02-09T17:26:52.917 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f86976e5:::benchmark_data_smithi179_77138_object85629:head# 2024-02-09T17:26:52.917 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f86b378c:::benchmark_data_smithi179_77138_object30280:head# 2024-02-09T17:26:52.917 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f86cd5e5:::benchmark_data_smithi179_77138_object113081:head# 2024-02-09T17:26:52.917 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f86d64cd:::benchmark_data_smithi179_77138_object58414:head# 2024-02-09T17:26:52.919 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f86d667a:::benchmark_data_smithi179_77138_object125380:head# 2024-02-09T17:26:52.920 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f86e2502:::benchmark_data_smithi179_77138_object130502:head# 2024-02-09T17:26:52.920 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f86f64fa:::benchmark_data_smithi179_77138_object98177:head# 2024-02-09T17:26:52.920 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f86ff873:::benchmark_data_smithi179_77138_object108646:head# 2024-02-09T17:26:52.920 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f870d237:::benchmark_data_smithi179_77138_object64724:head# 2024-02-09T17:26:52.922 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f87326b4:::benchmark_data_smithi179_77138_object23310:head# 2024-02-09T17:26:52.922 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f87327d2:::benchmark_data_smithi179_77138_object28413:head# 2024-02-09T17:26:52.925 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f873bb87:::benchmark_data_smithi179_77138_object96404:head# 2024-02-09T17:26:52.925 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8740a85:::benchmark_data_smithi179_77138_object95151:head# 2024-02-09T17:26:52.926 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8748bf7:::benchmark_data_smithi179_77138_object66034:head# 2024-02-09T17:26:52.926 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f87499b5:::benchmark_data_smithi179_77138_object20682:head# 2024-02-09T17:26:52.926 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8753050:::benchmark_data_smithi179_77138_object80856:head# 2024-02-09T17:26:52.927 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8757209:::benchmark_data_smithi179_77138_object118537:head# 2024-02-09T17:26:52.927 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f875b106:::benchmark_data_smithi179_77138_object20759:head# 2024-02-09T17:26:52.928 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8762f84:::benchmark_data_smithi179_77138_object91441:head# 2024-02-09T17:26:52.928 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f87669cf:::benchmark_data_smithi179_77138_object27928:head# 2024-02-09T17:26:52.928 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f87786a5:::benchmark_data_smithi179_77138_object51675:head# 2024-02-09T17:26:52.929 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f877db1f:::benchmark_data_smithi179_77138_object71843:head# 2024-02-09T17:26:52.929 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:52.929 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f87803ac:::benchmark_data_smithi179_77138_object115329:head# 2024-02-09T17:26:52.930 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f878471d:::benchmark_data_smithi179_77138_object44223:head# 2024-02-09T17:26:52.930 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f879322d:::benchmark_data_smithi179_77138_object120629:head# 2024-02-09T17:26:52.930 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f879efac:::benchmark_data_smithi179_77138_object65135:head# 2024-02-09T17:26:52.931 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f87a01e7:::benchmark_data_smithi179_77138_object51964:head# 2024-02-09T17:26:52.931 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f87a3956:::benchmark_data_smithi179_77138_object77405:head# 2024-02-09T17:26:52.932 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f87c1092:::benchmark_data_smithi179_77138_object70083:head# 2024-02-09T17:26:52.932 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f87c7783:::benchmark_data_smithi179_77138_object105457:head# 2024-02-09T17:26:52.938 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f87d5141:::benchmark_data_smithi179_77138_object74920:head# 2024-02-09T17:26:52.938 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f87e397d:::benchmark_data_smithi179_77138_object39504:head# 2024-02-09T17:26:52.938 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f87f149c:::benchmark_data_smithi179_77138_object80313:head# 2024-02-09T17:26:52.938 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f87f3ef7:::benchmark_data_smithi179_77138_object26953:head# 2024-02-09T17:26:52.938 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f87f5923:::benchmark_data_smithi179_77138_object77832:head# 2024-02-09T17:26:52.938 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8806087:::benchmark_data_smithi179_77138_object115423:head# 2024-02-09T17:26:52.938 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f881ba07:::benchmark_data_smithi179_77138_object85054:head# 2024-02-09T17:26:52.940 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8824175:::benchmark_data_smithi179_77138_object34769:head# 2024-02-09T17:26:52.941 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f883e133:::benchmark_data_smithi179_77138_object56777:head# 2024-02-09T17:26:52.941 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8844370:::benchmark_data_smithi179_77138_object119587:head# 2024-02-09T17:26:52.941 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8848ef0:::benchmark_data_smithi179_77138_object69191:head# 2024-02-09T17:26:52.941 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f885ad50:::benchmark_data_smithi179_77138_object83010:head# 2024-02-09T17:26:52.941 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8868172:::benchmark_data_smithi179_77138_object22088:head# 2024-02-09T17:26:52.941 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8870220:::benchmark_data_smithi179_77138_object112445:head# 2024-02-09T17:26:52.941 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f88773ce:::benchmark_data_smithi179_77138_object130915:head# 2024-02-09T17:26:52.941 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8877c68:::benchmark_data_smithi179_77138_object132165:head# 2024-02-09T17:26:52.942 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8889dc0:::benchmark_data_smithi179_77138_object110076:head# 2024-02-09T17:26:52.942 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f888b63c:::benchmark_data_smithi179_77138_object114541:head# 2024-02-09T17:26:52.943 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f888e64c:::benchmark_data_smithi179_77138_object97564:head# 2024-02-09T17:26:52.945 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8899eb1:::benchmark_data_smithi179_77138_object59108:head# 2024-02-09T17:26:52.945 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f88a0550:::benchmark_data_smithi179_77138_object80986:head# 2024-02-09T17:26:52.945 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f88bc8d2:::benchmark_data_smithi179_77138_object81510:head# 2024-02-09T17:26:52.946 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f88c6a8c:::benchmark_data_smithi179_77138_object70652:head# 2024-02-09T17:26:52.946 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f88c8210:::benchmark_data_smithi179_77138_object112613:head# 2024-02-09T17:26:52.946 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f88c962e:::benchmark_data_smithi179_77138_object132908:head# 2024-02-09T17:26:52.947 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f88cf6f9:::benchmark_data_smithi179_77138_object40862:head# 2024-02-09T17:26:52.947 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f88f42a6:::benchmark_data_smithi179_77138_object64486:head# 2024-02-09T17:26:52.948 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f890db58:::benchmark_data_smithi179_77138_object68774:head# 2024-02-09T17:26:52.948 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f891386b:::benchmark_data_smithi179_77138_object97337:head# 2024-02-09T17:26:52.948 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8915b2f:::benchmark_data_smithi179_77138_object90295:head# 2024-02-09T17:26:52.949 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f891aa09:::benchmark_data_smithi179_77138_object71860:head# 2024-02-09T17:26:52.949 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8942f3e:::benchmark_data_smithi179_77138_object89407:head# 2024-02-09T17:26:52.950 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f894bdc1:::benchmark_data_smithi179_77138_object115297:head# 2024-02-09T17:26:52.950 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8956e34:::benchmark_data_smithi179_77138_object107134:head# 2024-02-09T17:26:52.951 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f896026b:::benchmark_data_smithi179_77138_object74741:head# 2024-02-09T17:26:52.951 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f89681be:::benchmark_data_smithi179_77138_object100423:head# 2024-02-09T17:26:52.951 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f896d193:::benchmark_data_smithi179_77138_object58703:head# 2024-02-09T17:26:52.952 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f896ef5e:::benchmark_data_smithi179_77138_object132818:head# 2024-02-09T17:26:52.952 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f89709e9:::benchmark_data_smithi179_77138_object19908:head# 2024-02-09T17:26:52.952 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8971a22:::benchmark_data_smithi179_77138_object91337:head# 2024-02-09T17:26:52.953 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8975ea9:::benchmark_data_smithi179_77138_object92677:head# 2024-02-09T17:26:52.953 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f898eae4:::benchmark_data_smithi179_77138_object112144:head# 2024-02-09T17:26:52.954 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8992c90:::benchmark_data_smithi179_77138_object68112:head# 2024-02-09T17:26:52.954 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8996516:::benchmark_data_smithi179_77138_object86197:head# 2024-02-09T17:26:52.954 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f899824a:::benchmark_data_smithi179_77138_object71303:head# 2024-02-09T17:26:52.955 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f899a3db:::benchmark_data_smithi179_77138_object21880:head# 2024-02-09T17:26:52.955 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f89a5bae:::benchmark_data_smithi179_77138_object48308:head# 2024-02-09T17:26:52.956 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f89b8cab:::benchmark_data_smithi179_77138_object66167:head# 2024-02-09T17:26:52.956 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f89c3d7d:::benchmark_data_smithi179_77138_object77637:head# 2024-02-09T17:26:52.956 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f89da4be:::benchmark_data_smithi179_77138_object74414:head# 2024-02-09T17:26:52.957 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f89dac33:::benchmark_data_smithi179_77138_object94051:head# 2024-02-09T17:26:52.957 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f89ddbb8:::benchmark_data_smithi179_77138_object66138:head# 2024-02-09T17:26:52.958 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f89de14b:::benchmark_data_smithi179_77138_object47204:head# 2024-02-09T17:26:52.958 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8a25396:::benchmark_data_smithi179_77138_object45027:head# 2024-02-09T17:26:52.958 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8a26347:::benchmark_data_smithi179_77138_object42701:head# 2024-02-09T17:26:52.959 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8a2779e:::benchmark_data_smithi179_77138_object120917:head# 2024-02-09T17:26:52.959 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8a29604:::benchmark_data_smithi179_77138_object57067:head# 2024-02-09T17:26:52.960 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8a3a984:::benchmark_data_smithi179_77138_object50958:head# 2024-02-09T17:26:52.960 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8a3dc7a:::benchmark_data_smithi179_77138_object81135:head# 2024-02-09T17:26:52.961 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8a3f71c:::benchmark_data_smithi179_77138_object86080:head# 2024-02-09T17:26:52.964 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8a458dd:::benchmark_data_smithi179_77138_object116882:head# 2024-02-09T17:26:52.964 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8a474a3:::benchmark_data_smithi179_77138_object64796:head# 2024-02-09T17:26:52.964 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8a4cdec:::benchmark_data_smithi179_77138_object42196:head# 2024-02-09T17:26:52.965 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8a62ebe:::benchmark_data_smithi179_77138_object25389:head# 2024-02-09T17:26:52.965 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8a670a2:::benchmark_data_smithi179_77138_object58746:head# 2024-02-09T17:26:52.966 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8a67530:::benchmark_data_smithi179_77138_object60330:head# 2024-02-09T17:26:52.966 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8a676af:::benchmark_data_smithi179_77138_object71158:head# 2024-02-09T17:26:52.967 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8a6b808:::benchmark_data_smithi179_77138_object125142:head# 2024-02-09T17:26:52.967 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8a6cac2:::benchmark_data_smithi179_77138_object42994:head# 2024-02-09T17:26:52.967 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8a72f64:::benchmark_data_smithi179_77138_object103852:head# 2024-02-09T17:26:52.968 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8a7a951:::benchmark_data_smithi179_77138_object102810:head# 2024-02-09T17:26:52.969 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8a7c1de:::benchmark_data_smithi179_77138_object111067:head# 2024-02-09T17:26:52.969 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8a80a83:::benchmark_data_smithi179_77138_object119383:head# 2024-02-09T17:26:52.969 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8a87ac6:::benchmark_data_smithi179_77138_object68796:head# 2024-02-09T17:26:52.970 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8a910d4:::benchmark_data_smithi179_77138_object52114:head# 2024-02-09T17:26:52.970 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8a94742:::benchmark_data_smithi179_77138_object56231:head# 2024-02-09T17:26:52.971 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8ab2b98:::benchmark_data_smithi179_77138_object117824:head# 2024-02-09T17:26:52.971 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8aba317:::benchmark_data_smithi179_77138_object40700:head# 2024-02-09T17:26:52.971 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8ababb6:::benchmark_data_smithi179_77138_object130992:head# 2024-02-09T17:26:52.972 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8abe5f9:::benchmark_data_smithi179_77138_object101231:head# 2024-02-09T17:26:52.972 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8ac1fd5:::benchmark_data_smithi179_77138_object122609:head# 2024-02-09T17:26:52.972 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8ac6ba0:::benchmark_data_smithi179_77138_object99453:head# 2024-02-09T17:26:52.973 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8ac798f:::benchmark_data_smithi179_77138_object54335:head# 2024-02-09T17:26:52.973 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8acdb4d:::benchmark_data_smithi179_77138_object96415:head# 2024-02-09T17:26:52.978 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8adb279:::benchmark_data_smithi179_77138_object53470:head# 2024-02-09T17:26:52.979 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8ae2155:::benchmark_data_smithi179_77138_object91448:head# 2024-02-09T17:26:52.979 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8aee991:::benchmark_data_smithi179_77138_object90338:head# 2024-02-09T17:26:52.979 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8afad0d:::benchmark_data_smithi179_77138_object90669:head# 2024-02-09T17:26:52.979 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8b04b54:::benchmark_data_smithi179_77138_object52987:head# 2024-02-09T17:26:52.979 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8b05a8c:::benchmark_data_smithi179_77138_object47697:head# 2024-02-09T17:26:52.979 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8b0d3b8:::benchmark_data_smithi179_77138_object41734:head# 2024-02-09T17:26:52.979 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8b11cf0:::benchmark_data_smithi179_77138_object19870:head# 2024-02-09T17:26:52.979 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8b21892:::benchmark_data_smithi179_77138_object111883:head# 2024-02-09T17:26:52.979 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8b28b99:::benchmark_data_smithi179_77138_object47735:head# 2024-02-09T17:26:52.979 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8b29f36:::benchmark_data_smithi179_77138_object41715:head# 2024-02-09T17:26:52.979 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8b2d1a6:::benchmark_data_smithi179_77138_object21243:head# 2024-02-09T17:26:52.980 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8b326c9:::benchmark_data_smithi179_77138_object35412:head# 2024-02-09T17:26:52.982 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8b32dfe:::benchmark_data_smithi179_77138_object25338:head# 2024-02-09T17:26:52.982 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8b503b0:::benchmark_data_smithi179_77138_object129903:head# 2024-02-09T17:26:52.982 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8b8cf1f:::benchmark_data_smithi179_77138_object108472:head# 2024-02-09T17:26:52.983 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8b908c0:::benchmark_data_smithi179_77138_object67611:head# 2024-02-09T17:26:52.983 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8b962cd:::benchmark_data_smithi179_77138_object25277:head# 2024-02-09T17:26:52.984 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8b9cca1:::benchmark_data_smithi179_77138_object78917:head# 2024-02-09T17:26:52.984 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8ba0417:::benchmark_data_smithi179_77138_object110960:head# 2024-02-09T17:26:52.986 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8ba5458:::benchmark_data_smithi179_77138_object116744:head# 2024-02-09T17:26:52.986 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8baf976:::benchmark_data_smithi179_77138_object62535:head# 2024-02-09T17:26:52.986 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8bbae34:::benchmark_data_smithi179_77138_object20969:head# 2024-02-09T17:26:52.987 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8bd9d2c:::benchmark_data_smithi179_77138_object51851:head# 2024-02-09T17:26:52.987 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8bed305:::benchmark_data_smithi179_77138_object101399:head# 2024-02-09T17:26:52.988 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8bef0d6:::benchmark_data_smithi179_77138_object52588:head# 2024-02-09T17:26:52.988 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8bf01fc:::benchmark_data_smithi179_77138_object45075:head# 2024-02-09T17:26:52.989 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8bf59fd:::benchmark_data_smithi179_77138_object46730:head# 2024-02-09T17:26:52.990 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8c0a97d:::benchmark_data_smithi179_77138_object78083:head# 2024-02-09T17:26:52.991 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8c10321:::benchmark_data_smithi179_77138_object105500:head# 2024-02-09T17:26:52.991 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8c12847:::benchmark_data_smithi179_77138_object29382:head# 2024-02-09T17:26:52.991 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8c195ae:::benchmark_data_smithi179_77138_object61586:head# 2024-02-09T17:26:52.992 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8c1d29c:::benchmark_data_smithi179_77138_object21568:head# 2024-02-09T17:26:52.992 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8c2401d:::benchmark_data_smithi179_77138_object89773:head# 2024-02-09T17:26:52.993 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8c25e59:::benchmark_data_smithi179_77138_object57959:head# 2024-02-09T17:26:52.993 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8c2c231:::benchmark_data_smithi179_77138_object105198:head# 2024-02-09T17:26:52.993 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8c2ee2f:::benchmark_data_smithi179_77138_object31833:head# 2024-02-09T17:26:52.994 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8c39a6f:::benchmark_data_smithi179_77138_object132770:head# 2024-02-09T17:26:52.994 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8c4063e:::benchmark_data_smithi179_77138_object112062:head# 2024-02-09T17:26:52.995 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8c494b6:::benchmark_data_smithi179_77138_object42993:head# 2024-02-09T17:26:52.995 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8c580f6:::benchmark_data_smithi179_77138_object52289:head# 2024-02-09T17:26:52.996 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8c5d062:::benchmark_data_smithi179_77138_object73890:head# 2024-02-09T17:26:52.996 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8c7340d:::benchmark_data_smithi179_77138_object97890:head# 2024-02-09T17:26:52.997 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8c85fc0:::benchmark_data_smithi179_77138_object95498:head# 2024-02-09T17:26:52.997 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8c97928:::benchmark_data_smithi179_77138_object132941:head# 2024-02-09T17:26:52.998 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8cb1db5:::benchmark_data_smithi179_77138_object82711:head# 2024-02-09T17:26:52.998 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8ce612c:::benchmark_data_smithi179_77138_object132323:head# 2024-02-09T17:26:52.998 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8cf1876:::benchmark_data_smithi179_77138_object83731:head# 2024-02-09T17:26:52.999 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8cf513d:::benchmark_data_smithi179_77138_object96707:head# 2024-02-09T17:26:52.999 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8d122a6:::benchmark_data_smithi179_77138_object63831:head# 2024-02-09T17:26:53.000 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8d14896:::benchmark_data_smithi179_77138_object93735:head# 2024-02-09T17:26:53.000 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8d1c79f:::benchmark_data_smithi179_77138_object132667:head# 2024-02-09T17:26:53.000 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8d2385e:::benchmark_data_smithi179_77138_object115913:head# 2024-02-09T17:26:53.001 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8d3fbb2:::benchmark_data_smithi179_77138_object121960:head# 2024-02-09T17:26:53.001 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8d45d20:::benchmark_data_smithi179_77138_object76483:head# 2024-02-09T17:26:53.002 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8d48c6e:::benchmark_data_smithi179_77138_object48733:head# 2024-02-09T17:26:53.002 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8d61b4c:::benchmark_data_smithi179_77138_object42856:head# 2024-02-09T17:26:53.002 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8d6620c:::benchmark_data_smithi179_77138_object129698:head# 2024-02-09T17:26:53.003 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8d6a977:::benchmark_data_smithi179_77138_object104915:head# 2024-02-09T17:26:53.004 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8d77174:::benchmark_data_smithi179_77138_object43395:head# 2024-02-09T17:26:53.004 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8d799c7:::benchmark_data_smithi179_77138_object63356:head# 2024-02-09T17:26:53.005 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8d8c5dc:::benchmark_data_smithi179_77138_object123395:head# 2024-02-09T17:26:53.005 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8d8cdec:::benchmark_data_smithi179_77138_object99037:head# 2024-02-09T17:26:53.006 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8d93b1d:::benchmark_data_smithi179_77138_object91475:head# 2024-02-09T17:26:53.006 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8d98043:::benchmark_data_smithi179_77138_object76108:head# 2024-02-09T17:26:53.007 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8d98b47:::benchmark_data_smithi179_77138_object129991:head# 2024-02-09T17:26:53.007 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8daff14:::benchmark_data_smithi179_77138_object126161:head# 2024-02-09T17:26:53.008 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8db2c3a:::benchmark_data_smithi179_77138_object62998:head# 2024-02-09T17:26:53.011 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8db4d05:::benchmark_data_smithi179_77138_object25005:head# 2024-02-09T17:26:53.011 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8dc98dc:::benchmark_data_smithi179_77138_object103603:head# 2024-02-09T17:26:53.011 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8deec19:::benchmark_data_smithi179_77138_object95423:head# 2024-02-09T17:26:53.012 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8dfa2b9:::benchmark_data_smithi179_77138_object79896:head# 2024-02-09T17:26:53.012 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8e04606:::benchmark_data_smithi179_77138_object50445:head# 2024-02-09T17:26:53.012 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8e0a630:::benchmark_data_smithi179_77138_object111170:head# 2024-02-09T17:26:53.012 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8e0d3c5:::benchmark_data_smithi179_77138_object73136:head# 2024-02-09T17:26:53.012 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8e0d44a:::benchmark_data_smithi179_77138_object43271:head# 2024-02-09T17:26:53.012 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8e194fb:::benchmark_data_smithi179_77138_object95872:head# 2024-02-09T17:26:53.012 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8e1c4e8:::benchmark_data_smithi179_77138_object82083:head# 2024-02-09T17:26:53.012 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8e2c650:::benchmark_data_smithi179_77138_object76414:head# 2024-02-09T17:26:53.013 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8e2e7ab:::benchmark_data_smithi179_77138_object73171:head# 2024-02-09T17:26:53.013 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8e33670:::benchmark_data_smithi179_77138_object103404:head# 2024-02-09T17:26:53.013 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8e43b57:::benchmark_data_smithi179_77138_object120079:head# 2024-02-09T17:26:53.014 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8e44df1:::benchmark_data_smithi179_77138_object41617:head# 2024-02-09T17:26:53.014 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8e4b06d:::benchmark_data_smithi179_77138_object63554:head# 2024-02-09T17:26:53.014 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8e55114:::benchmark_data_smithi179_77138_object35276:head# 2024-02-09T17:26:53.015 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8e660cf:::benchmark_data_smithi179_77138_object85077:head# 2024-02-09T17:26:53.015 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8e6b710:::benchmark_data_smithi179_77138_object51285:head# 2024-02-09T17:26:53.016 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8e6c945:::benchmark_data_smithi179_77138_object118572:head# 2024-02-09T17:26:53.016 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8e74d0e:::benchmark_data_smithi179_77138_object35031:head# 2024-02-09T17:26:53.017 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8e7f38d:::benchmark_data_smithi179_77138_object72916:head# 2024-02-09T17:26:53.017 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8e99709:::benchmark_data_smithi179_77138_object63677:head# 2024-02-09T17:26:53.017 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8eb87f7:::benchmark_data_smithi179_77138_object97774:head# 2024-02-09T17:26:53.018 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8ec6595:::benchmark_data_smithi179_77138_object89096:head# 2024-02-09T17:26:53.018 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8ec77fc:::benchmark_data_smithi179_77138_object24653:head# 2024-02-09T17:26:53.019 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8ecbd95:::benchmark_data_smithi179_77138_object128295:head# 2024-02-09T17:26:53.019 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8ecc86e:::benchmark_data_smithi179_77138_object123936:head# 2024-02-09T17:26:53.019 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8eda3ba:::benchmark_data_smithi179_77138_object54214:head# 2024-02-09T17:26:53.020 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8ee7ff4:::benchmark_data_smithi179_77138_object44923:head# 2024-02-09T17:26:53.020 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8eea117:::benchmark_data_smithi179_77138_object102350:head# 2024-02-09T17:26:53.021 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8ef36e6:::benchmark_data_smithi179_77138_object85191:head# 2024-02-09T17:26:53.021 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8efb731:::benchmark_data_smithi179_77138_object76492:head# 2024-02-09T17:26:53.022 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8eff535:::benchmark_data_smithi179_77138_object37254:head# 2024-02-09T17:26:53.022 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8f07c91:::benchmark_data_smithi179_77138_object77062:head# 2024-02-09T17:26:53.023 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8f0f916:::benchmark_data_smithi179_77138_object39643:head# 2024-02-09T17:26:53.023 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8f16e0d:::benchmark_data_smithi179_77138_object76564:head# 2024-02-09T17:26:53.023 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8f1ac3d:::benchmark_data_smithi179_77138_object131764:head# 2024-02-09T17:26:53.024 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8f285c6:::benchmark_data_smithi179_77138_object131108:head# 2024-02-09T17:26:53.024 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8f2d710:::benchmark_data_smithi179_77138_object46877:head# 2024-02-09T17:26:53.024 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8f3350c:::benchmark_data_smithi179_77138_object129730:head# 2024-02-09T17:26:53.025 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8f38564:::benchmark_data_smithi179_77138_object21212:head# 2024-02-09T17:26:53.025 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8f3e88a:::benchmark_data_smithi179_77138_object66734:head# 2024-02-09T17:26:53.026 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8f4877c:::benchmark_data_smithi179_77138_object35571:head# 2024-02-09T17:26:53.026 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8f490a7:::benchmark_data_smithi179_77138_object87540:head# 2024-02-09T17:26:53.027 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8f4f366:::benchmark_data_smithi179_77138_object77458:head# 2024-02-09T17:26:53.027 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8f51b26:::benchmark_data_smithi179_77138_object72707:head# 2024-02-09T17:26:53.027 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8f58aa9:::benchmark_data_smithi179_77138_object60890:head# 2024-02-09T17:26:53.028 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8f5b3aa:::benchmark_data_smithi179_77138_object92722:head# 2024-02-09T17:26:53.028 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8f61e4e:::benchmark_data_smithi179_77138_object67574:head# 2024-02-09T17:26:53.029 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8f67665:::benchmark_data_smithi179_77138_object51827:head# 2024-02-09T17:26:53.029 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8f6fb34:::benchmark_data_smithi179_77138_object58323:head# 2024-02-09T17:26:53.031 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8f7375e:::benchmark_data_smithi179_77138_object56546:head# 2024-02-09T17:26:53.031 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8f74aa1:::benchmark_data_smithi179_77138_object36988:head# 2024-02-09T17:26:53.031 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8f793d9:::benchmark_data_smithi179_77138_object64439:head# 2024-02-09T17:26:53.032 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8f95ddc:::benchmark_data_smithi179_77138_object82495:head# 2024-02-09T17:26:53.032 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8f9884f:::benchmark_data_smithi179_77138_object115523:head# 2024-02-09T17:26:53.032 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:26:53.033 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8fa0a8f:::benchmark_data_smithi179_77138_object70457:head# 2024-02-09T17:26:53.033 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8fa6669:::benchmark_data_smithi179_77138_object102108:head# 2024-02-09T17:26:53.033 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8fb12e3:::benchmark_data_smithi179_77138_object74388:head# 2024-02-09T17:26:53.034 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8fb596d:::benchmark_data_smithi179_77138_object55806:head# 2024-02-09T17:26:53.034 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8fc00af:::benchmark_data_smithi179_77138_object107388:head# 2024-02-09T17:26:53.034 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8fc188b:::benchmark_data_smithi179_77138_object99521:head# 2024-02-09T17:26:53.035 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8fc8923:::benchmark_data_smithi179_77138_object113804:head# 2024-02-09T17:26:53.035 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8fcbcf9:::benchmark_data_smithi179_77138_object96619:head# 2024-02-09T17:26:53.035 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8fce684:::benchmark_data_smithi179_77138_object80405:head# 2024-02-09T17:26:53.036 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8fdd646:::benchmark_data_smithi179_77138_object105838:head# 2024-02-09T17:26:53.036 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8fed861:::benchmark_data_smithi179_77138_object78903:head# 2024-02-09T17:26:53.037 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f8ff40a2:::benchmark_data_smithi179_77138_object35669:head# 2024-02-09T17:26:53.038 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f90047f2:::benchmark_data_smithi179_77138_object82208:head# 2024-02-09T17:26:53.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:53.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:26:53.042 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f900c266:::benchmark_data_smithi179_77138_object77189:head# 2024-02-09T17:26:53.042 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f90105ca:::benchmark_data_smithi179_77138_object126236:head# 2024-02-09T17:26:53.042 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9010c53:::benchmark_data_smithi179_77138_object77205:head# 2024-02-09T17:26:53.042 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9014699:::benchmark_data_smithi179_77138_object28517:head# 2024-02-09T17:26:53.042 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f901f921:::benchmark_data_smithi179_77138_object34185:head# 2024-02-09T17:26:53.042 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9028634:::benchmark_data_smithi179_77138_object84976:head# 2024-02-09T17:26:53.042 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f902d575:::benchmark_data_smithi179_77138_object97358:head# 2024-02-09T17:26:53.042 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f903077f:::benchmark_data_smithi179_77138_object97306:head# 2024-02-09T17:26:53.043 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f903196c:::benchmark_data_smithi179_77138_object27551:head# 2024-02-09T17:26:53.043 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f904fac8:::benchmark_data_smithi179_77138_object29052:head# 2024-02-09T17:26:53.047 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f905551d:::benchmark_data_smithi179_77138_object59052:head# 2024-02-09T17:26:53.047 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f905891f:::benchmark_data_smithi179_77138_object25068:head# 2024-02-09T17:26:53.047 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f907a985:::benchmark_data_smithi179_77138_object111496:head# 2024-02-09T17:26:53.051 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f90852a7:::benchmark_data_smithi179_77138_object49776:head# 2024-02-09T17:26:53.052 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9089deb:::benchmark_data_smithi179_77138_object75968:head# 2024-02-09T17:26:53.052 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f909352a:::benchmark_data_smithi179_77138_object95826:head# 2024-02-09T17:26:53.052 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f90a56bd:::benchmark_data_smithi179_77138_object68768:head# 2024-02-09T17:26:53.053 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f90aa573:::benchmark_data_smithi179_77138_object66917:head# 2024-02-09T17:26:53.053 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f90c68e4:::benchmark_data_smithi179_77138_object101500:head# 2024-02-09T17:26:53.053 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f90e08fb:::benchmark_data_smithi179_77138_object95923:head# 2024-02-09T17:26:53.054 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f90e9e99:::benchmark_data_smithi179_77138_object63785:head# 2024-02-09T17:26:53.054 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f90eb53c:::benchmark_data_smithi179_77138_object68718:head# 2024-02-09T17:26:53.055 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f90f4733:::benchmark_data_smithi179_77138_object93715:head# 2024-02-09T17:26:53.056 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f90f7a36:::benchmark_data_smithi179_77138_object21443:head# 2024-02-09T17:26:53.056 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f910d620:::benchmark_data_smithi179_77138_object65336:head# 2024-02-09T17:26:53.057 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f911777c:::benchmark_data_smithi179_77138_object68411:head# 2024-02-09T17:26:53.057 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f911f9d7:::benchmark_data_smithi179_77138_object49733:head# 2024-02-09T17:26:53.057 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9125705:::benchmark_data_smithi179_77138_object123371:head# 2024-02-09T17:26:53.058 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9125b3f:::benchmark_data_smithi179_77138_object114349:head# 2024-02-09T17:26:53.058 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9128627:::benchmark_data_smithi179_77138_object27230:head# 2024-02-09T17:26:53.058 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9129237:::benchmark_data_smithi179_77138_object52258:head# 2024-02-09T17:26:53.059 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f912ab3f:::benchmark_data_smithi179_77138_object43731:head# 2024-02-09T17:26:53.059 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f912ad7a:::benchmark_data_smithi179_77138_object20416:head# 2024-02-09T17:26:53.060 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9151b42:::benchmark_data_smithi179_77138_object95445:head# 2024-02-09T17:26:53.061 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9167748:::benchmark_data_smithi179_77138_object104371:head# 2024-02-09T17:26:53.061 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f916818d:::benchmark_data_smithi179_77138_object115542:head# 2024-02-09T17:26:53.061 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f916e3c9:::benchmark_data_smithi179_77138_object41186:head# 2024-02-09T17:26:53.062 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f917659f:::benchmark_data_smithi179_77138_object107652:head# 2024-02-09T17:26:53.062 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9177e79:::benchmark_data_smithi179_77138_object85361:head# 2024-02-09T17:26:53.063 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f91787d8:::benchmark_data_smithi179_77138_object79388:head# 2024-02-09T17:26:53.063 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f917bb47:::benchmark_data_smithi179_77138_object39179:head# 2024-02-09T17:26:53.064 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f917fb67:::benchmark_data_smithi179_77138_object28056:head# 2024-02-09T17:26:53.064 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f918eb1a:::benchmark_data_smithi179_77138_object84945:head# 2024-02-09T17:26:53.064 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9190291:::benchmark_data_smithi179_77138_object116437:head# 2024-02-09T17:26:53.065 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f919dc06:::benchmark_data_smithi179_77138_object52752:head# 2024-02-09T17:26:53.065 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f91a0e03:::benchmark_data_smithi179_77138_object107529:head# 2024-02-09T17:26:53.066 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f91a4ac0:::benchmark_data_smithi179_77138_object122419:head# 2024-02-09T17:26:53.066 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f91ab4bc:::benchmark_data_smithi179_77138_object89026:head# 2024-02-09T17:26:53.069 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f91bec39:::benchmark_data_smithi179_77138_object51307:head# 2024-02-09T17:26:53.072 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f91c6ef2:::benchmark_data_smithi179_77138_object102321:head# 2024-02-09T17:26:53.072 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f91c86f5:::benchmark_data_smithi179_77138_object45014:head# 2024-02-09T17:26:53.072 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f91cd247:::benchmark_data_smithi179_77138_object72968:head# 2024-02-09T17:26:53.073 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f91e8ca8:::benchmark_data_smithi179_77138_object64823:head# 2024-02-09T17:26:53.073 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f91ed8a0:::benchmark_data_smithi179_77138_object73732:head# 2024-02-09T17:26:53.074 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f91f5a4c:::benchmark_data_smithi179_77138_object122575:head# 2024-02-09T17:26:53.074 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9209748:::benchmark_data_smithi179_77138_object132999:head# 2024-02-09T17:26:53.075 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9209d51:::benchmark_data_smithi179_77138_object46982:head# 2024-02-09T17:26:53.075 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9213292:::benchmark_data_smithi179_77138_object20829:head# 2024-02-09T17:26:53.075 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f922bcb6:::benchmark_data_smithi179_77138_object51300:head# 2024-02-09T17:26:53.077 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f92322e2:::benchmark_data_smithi179_77138_object117819:head# 2024-02-09T17:26:53.077 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f92360a0:::benchmark_data_smithi179_77138_object76813:head# 2024-02-09T17:26:53.077 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f923d2af:::benchmark_data_smithi179_77138_object115480:head# 2024-02-09T17:26:53.078 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f923ff7e:::benchmark_data_smithi179_77138_object129842:head# 2024-02-09T17:26:53.078 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9246749:::benchmark_data_smithi179_77138_object39795:head# 2024-02-09T17:26:53.078 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9250375:::benchmark_data_smithi179_77138_object81994:head# 2024-02-09T17:26:53.079 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f925714d:::benchmark_data_smithi179_77138_object30886:head# 2024-02-09T17:26:53.079 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f92618d9:::benchmark_data_smithi179_77138_object59855:head# 2024-02-09T17:26:53.080 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f92664fb:::benchmark_data_smithi179_77138_object44214:head# 2024-02-09T17:26:53.080 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f926a6e2:::benchmark_data_smithi179_77138_object52139:head# 2024-02-09T17:26:53.080 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f926b20b:::benchmark_data_smithi179_77138_object79455:head# 2024-02-09T17:26:53.083 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9279eee:::benchmark_data_smithi179_77138_object118074:head# 2024-02-09T17:26:53.083 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f927b604:::benchmark_data_smithi179_77138_object54761:head# 2024-02-09T17:26:53.083 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f928b0fa:::benchmark_data_smithi179_77138_object44354:head# 2024-02-09T17:26:53.083 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f928b901:::benchmark_data_smithi179_77138_object103862:head# 2024-02-09T17:26:53.084 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f928c7e0:::benchmark_data_smithi179_77138_object131224:head# 2024-02-09T17:26:53.084 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f928f541:::benchmark_data_smithi179_77138_object89586:head# 2024-02-09T17:26:53.084 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f92939c8:::benchmark_data_smithi179_77138_object81343:head# 2024-02-09T17:26:53.084 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9299595:::benchmark_data_smithi179_77138_object130678:head# 2024-02-09T17:26:53.085 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f929a6f0:::benchmark_data_smithi179_77138_object121059:head# 2024-02-09T17:26:53.087 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f92a828c:::benchmark_data_smithi179_77138_object74967:head# 2024-02-09T17:26:53.088 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f92d1f10:::benchmark_data_smithi179_77138_object43116:head# 2024-02-09T17:26:53.088 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f92d74b3:::benchmark_data_smithi179_77138_object108062:head# 2024-02-09T17:26:53.089 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f92dffee:::benchmark_data_smithi179_77138_object71971:head# 2024-02-09T17:26:53.090 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f92ec29b:::benchmark_data_smithi179_77138_object122174:head# 2024-02-09T17:26:53.092 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f92f219e:::benchmark_data_smithi179_77138_object55348:head# 2024-02-09T17:26:53.094 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f92fc0c4:::benchmark_data_smithi179_77138_object129754:head# 2024-02-09T17:26:53.095 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f92ff216:::benchmark_data_smithi179_77138_object107133:head# 2024-02-09T17:26:53.098 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f93001cd:::benchmark_data_smithi179_77138_object87455:head# 2024-02-09T17:26:53.098 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9306927:::benchmark_data_smithi179_77138_object106324:head# 2024-02-09T17:26:53.098 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f930e453:::benchmark_data_smithi179_77138_object78867:head# 2024-02-09T17:26:53.099 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9319d83:::benchmark_data_smithi179_77138_object51697:head# 2024-02-09T17:26:53.100 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f931b8b7:::benchmark_data_smithi179_77138_object62126:head# 2024-02-09T17:26:53.100 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9321c95:::benchmark_data_smithi179_77138_object115045:head# 2024-02-09T17:26:53.101 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f93345ae:::benchmark_data_smithi179_77138_object40938:head# 2024-02-09T17:26:53.101 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9337c39:::benchmark_data_smithi179_77138_object19741:head# 2024-02-09T17:26:53.102 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f933bd49:::benchmark_data_smithi179_77138_object61399:head# 2024-02-09T17:26:53.102 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f933e864:::benchmark_data_smithi179_77138_object99385:head# 2024-02-09T17:26:53.102 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9344651:::benchmark_data_smithi179_77138_object26099:head# 2024-02-09T17:26:53.103 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f934862e:::benchmark_data_smithi179_77138_object40432:head# 2024-02-09T17:26:53.103 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f934a30f:::benchmark_data_smithi179_77138_object60314:head# 2024-02-09T17:26:53.104 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9352dfb:::benchmark_data_smithi179_77138_object115934:head# 2024-02-09T17:26:53.104 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9357e14:::benchmark_data_smithi179_77138_object63880:head# 2024-02-09T17:26:53.105 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f935cf27:::benchmark_data_smithi179_77138_object98230:head# 2024-02-09T17:26:53.105 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f936a9b3:::benchmark_data_smithi179_77138_object127141:head# 2024-02-09T17:26:53.106 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f937107a:::benchmark_data_smithi179_77138_object124674:head# 2024-02-09T17:26:53.106 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f93722f8:::benchmark_data_smithi179_77138_object126033:head# 2024-02-09T17:26:53.107 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f937a960:::benchmark_data_smithi179_77138_object47603:head# 2024-02-09T17:26:53.107 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9387124:::benchmark_data_smithi179_77138_object52749:head# 2024-02-09T17:26:53.108 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9392fe5:::benchmark_data_smithi179_77138_object31873:head# 2024-02-09T17:26:53.108 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f939b058:::benchmark_data_smithi179_77138_object41320:head# 2024-02-09T17:26:53.109 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f93acd30:::benchmark_data_smithi179_77138_object96776:head# 2024-02-09T17:26:53.109 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f93b41a0:::benchmark_data_smithi179_77138_object81991:head# 2024-02-09T17:26:53.109 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f93c309a:::benchmark_data_smithi179_77138_object122950:head# 2024-02-09T17:26:53.112 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f93e51aa:::benchmark_data_smithi179_77138_object80016:head# 2024-02-09T17:26:53.112 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f93ecb98:::benchmark_data_smithi179_77138_object61566:head# 2024-02-09T17:26:53.113 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f93f97f7:::benchmark_data_smithi179_77138_object25826:head# 2024-02-09T17:26:53.113 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f93ff44d:::benchmark_data_smithi179_77138_object69156:head# 2024-02-09T17:26:53.113 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9403987:::benchmark_data_smithi179_77138_object22639:head# 2024-02-09T17:26:53.114 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9408dea:::benchmark_data_smithi179_77138_object109466:head# 2024-02-09T17:26:53.114 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f940b360:::benchmark_data_smithi179_77138_object51951:head# 2024-02-09T17:26:53.115 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f941b9b3:::benchmark_data_smithi179_77138_object22282:head# 2024-02-09T17:26:53.115 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9423971:::benchmark_data_smithi179_77138_object104389:head# 2024-02-09T17:26:53.115 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9429048:::benchmark_data_smithi179_77138_object27181:head# 2024-02-09T17:26:53.118 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9453b01:::benchmark_data_smithi179_77138_object128169:head# 2024-02-09T17:26:53.119 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9456cfa:::benchmark_data_smithi179_77138_object34351:head# 2024-02-09T17:26:53.119 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f945ddf2:::benchmark_data_smithi179_77138_object87044:head# 2024-02-09T17:26:53.119 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f945e23a:::benchmark_data_smithi179_77138_object39914:head# 2024-02-09T17:26:53.120 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f945e796:::benchmark_data_smithi179_77138_object87307:head# 2024-02-09T17:26:53.121 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9467d19:::benchmark_data_smithi179_77138_object48920:head# 2024-02-09T17:26:53.121 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f946b697:::benchmark_data_smithi179_77138_object52871:head# 2024-02-09T17:26:53.122 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f947e7f4:::benchmark_data_smithi179_77138_object60080:head# 2024-02-09T17:26:53.122 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f947f437:::benchmark_data_smithi179_77138_object79527:head# 2024-02-09T17:26:53.123 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f947fd46:::benchmark_data_smithi179_77138_object100493:head# 2024-02-09T17:26:53.123 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f94863e0:::benchmark_data_smithi179_77138_object43423:head# 2024-02-09T17:26:53.123 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f948943f:::benchmark_data_smithi179_77138_object55070:head# 2024-02-09T17:26:53.124 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f94927df:::benchmark_data_smithi179_77138_object25802:head# 2024-02-09T17:26:53.124 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f949995d:::benchmark_data_smithi179_77138_object80905:head# 2024-02-09T17:26:53.124 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f949faa7:::benchmark_data_smithi179_77138_object48758:head# 2024-02-09T17:26:53.125 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f949fce1:::benchmark_data_smithi179_77138_object121473:head# 2024-02-09T17:26:53.125 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f94a6e15:::benchmark_data_smithi179_77138_object75810:head# 2024-02-09T17:26:53.126 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f94b27b0:::benchmark_data_smithi179_77138_object101698:head# 2024-02-09T17:26:53.126 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f94bca77:::benchmark_data_smithi179_77138_object63529:head# 2024-02-09T17:26:53.127 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f94c1155:::benchmark_data_smithi179_77138_object90328:head# 2024-02-09T17:26:53.127 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f94cbf94:::benchmark_data_smithi179_77138_object105038:head# 2024-02-09T17:26:53.128 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f94cdabe:::benchmark_data_smithi179_77138_object39544:head# 2024-02-09T17:26:53.128 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f94df194:::benchmark_data_smithi179_77138_object24950:head# 2024-02-09T17:26:53.128 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f94ef686:::benchmark_data_smithi179_77138_object124261:head# 2024-02-09T17:26:53.129 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f94f5a6b:::benchmark_data_smithi179_77138_object112819:head# 2024-02-09T17:26:53.129 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9508196:::benchmark_data_smithi179_77138_object31052:head# 2024-02-09T17:26:53.130 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9512109:::benchmark_data_smithi179_77138_object44454:head# 2024-02-09T17:26:53.130 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9519305:::benchmark_data_smithi179_77138_object32309:head# 2024-02-09T17:26:53.130 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f951af9a:::benchmark_data_smithi179_77138_object64337:head# 2024-02-09T17:26:53.131 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f951dc7e:::benchmark_data_smithi179_77138_object109367:head# 2024-02-09T17:26:53.132 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9523cf5:::benchmark_data_smithi179_77138_object86687:head# 2024-02-09T17:26:53.132 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9524296:::benchmark_data_smithi179_77138_object44227:head# 2024-02-09T17:26:53.133 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9524840:::benchmark_data_smithi179_77138_object118468:head# 2024-02-09T17:26:53.133 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9524e82:::benchmark_data_smithi179_77138_object101825:head# 2024-02-09T17:26:53.133 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f952adcd:::benchmark_data_smithi179_77138_object43088:head# 2024-02-09T17:26:53.134 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f953c361:::benchmark_data_smithi179_77138_object75206:head# 2024-02-09T17:26:53.134 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f953e3d8:::benchmark_data_smithi179_77138_object25644:head# 2024-02-09T17:26:53.135 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f954b228:::benchmark_data_smithi179_77138_object132430:head# 2024-02-09T17:26:53.135 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f954df45:::benchmark_data_smithi179_77138_object114393:head# 2024-02-09T17:26:53.135 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f95640e3:::benchmark_data_smithi179_77138_object34818:head# 2024-02-09T17:26:53.136 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9567079:::benchmark_data_smithi179_77138_object71805:head# 2024-02-09T17:26:53.136 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f956d9b3:::benchmark_data_smithi179_77138_object125170:head# 2024-02-09T17:26:53.137 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9570c6c:::benchmark_data_smithi179_77138_object84698:head# 2024-02-09T17:26:53.137 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f95732fa:::benchmark_data_smithi179_77138_object108755:head# 2024-02-09T17:26:53.137 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f95a141d:::benchmark_data_smithi179_77138_object94438:head# 2024-02-09T17:26:53.138 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f95a7793:::benchmark_data_smithi179_77138_object81609:head# 2024-02-09T17:26:53.138 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f95a8c7c:::benchmark_data_smithi179_77138_object48411:head# 2024-02-09T17:26:53.139 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f95ad770:::benchmark_data_smithi179_77138_object101712:head# 2024-02-09T17:26:53.139 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:26:53.140 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f95afbaa:::benchmark_data_smithi179_77138_object109179:head# 2024-02-09T17:26:53.140 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f95b73a9:::benchmark_data_smithi179_77138_object70590:head# 2024-02-09T17:26:53.141 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f95d1db8:::benchmark_data_smithi179_77138_object109647:head# 2024-02-09T17:26:53.141 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f95d1fca:::benchmark_data_smithi179_77138_object57715:head# 2024-02-09T17:26:53.141 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f95deadf:::benchmark_data_smithi179_77138_object53922:head# 2024-02-09T17:26:53.141 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f95defee:::benchmark_data_smithi179_77138_object70968:head# 2024-02-09T17:26:53.142 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9602ba8:::benchmark_data_smithi179_77138_object74208:head# 2024-02-09T17:26:53.142 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9624537:::benchmark_data_smithi179_77138_object54700:head# 2024-02-09T17:26:53.142 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9629d71:::benchmark_data_smithi179_77138_object22885:head# 2024-02-09T17:26:53.143 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9632ad6:::benchmark_data_smithi179_77138_object124584:head# 2024-02-09T17:26:53.144 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f963f19e:::benchmark_data_smithi179_77138_object103808:head# 2024-02-09T17:26:53.144 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f964b8b5:::benchmark_data_smithi179_77138_object100668:head# 2024-02-09T17:26:53.145 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9662fa2:::benchmark_data_smithi179_77138_object120522:head# 2024-02-09T17:26:53.145 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9667dc7:::benchmark_data_smithi179_77138_object72179:head# 2024-02-09T17:26:53.145 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f966b7d2:::benchmark_data_smithi179_77138_object63114:head# 2024-02-09T17:26:53.146 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f967d0c4:::benchmark_data_smithi179_77138_object93165:head# 2024-02-09T17:26:53.146 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f96815fc:::benchmark_data_smithi179_77138_object101372:head# 2024-02-09T17:26:53.146 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9687fcf:::benchmark_data_smithi179_77138_object90270:head# 2024-02-09T17:26:53.147 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f96898fd:::benchmark_data_smithi179_77138_object33774:head# 2024-02-09T17:26:53.147 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9690fe3:::benchmark_data_smithi179_77138_object70505:head# 2024-02-09T17:26:53.148 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f96acf8e:::benchmark_data_smithi179_77138_object59696:head# 2024-02-09T17:26:53.148 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f96d1975:::benchmark_data_smithi179_77138_object61653:head# 2024-02-09T17:26:53.148 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f96e750f:::benchmark_data_smithi179_77138_object24259:head# 2024-02-09T17:26:53.149 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f96f3605:::benchmark_data_smithi179_77138_object107089:head# 2024-02-09T17:26:53.149 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f96f62de:::benchmark_data_smithi179_77138_object54990:head# 2024-02-09T17:26:53.154 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9711561:::benchmark_data_smithi179_77138_object76849:head# 2024-02-09T17:26:53.154 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f971693e:::benchmark_data_smithi179_77138_object67156:head# 2024-02-09T17:26:53.154 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9719ec6:::benchmark_data_smithi179_77138_object47172:head# 2024-02-09T17:26:53.154 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f971eafc:::benchmark_data_smithi179_77138_object60102:head# 2024-02-09T17:26:53.156 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f972dce9:::benchmark_data_smithi179_77138_object113853:head# 2024-02-09T17:26:53.156 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f973dbde:::benchmark_data_smithi179_77138_object97473:head# 2024-02-09T17:26:53.156 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f974a405:::benchmark_data_smithi179_77138_object106486:head# 2024-02-09T17:26:53.157 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f974b9fb:::benchmark_data_smithi179_77138_object63810:head# 2024-02-09T17:26:53.157 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f975adf3:::benchmark_data_smithi179_77138_object106499:head# 2024-02-09T17:26:53.157 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f977ab59:::benchmark_data_smithi179_77138_object103548:head# 2024-02-09T17:26:53.157 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f978c2aa:::benchmark_data_smithi179_77138_object26405:head# 2024-02-09T17:26:53.157 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f979bb8b:::benchmark_data_smithi179_77138_object80788:head# 2024-02-09T17:26:53.157 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f979f7f7:::benchmark_data_smithi179_77138_object130876:head# 2024-02-09T17:26:53.157 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f97a24ea:::benchmark_data_smithi179_77138_object57130:head# 2024-02-09T17:26:53.159 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f97a5713:::benchmark_data_smithi179_77138_object44976:head# 2024-02-09T17:26:53.160 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f97a885e:::benchmark_data_smithi179_77138_object21703:head# 2024-02-09T17:26:53.160 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f97a9678:::benchmark_data_smithi179_77138_object54652:head# 2024-02-09T17:26:53.161 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f97c2994:::benchmark_data_smithi179_77138_object28252:head# 2024-02-09T17:26:53.161 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f97ca267:::benchmark_data_smithi179_77138_object120511:head# 2024-02-09T17:26:53.161 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f97cf8f4:::benchmark_data_smithi179_77138_object84651:head# 2024-02-09T17:26:53.162 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f97df1da:::benchmark_data_smithi179_77138_object97297:head# 2024-02-09T17:26:53.162 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f97e7f54:::benchmark_data_smithi179_77138_object52107:head# 2024-02-09T17:26:53.163 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f97ecc74:::benchmark_data_smithi179_77138_object78971:head# 2024-02-09T17:26:53.163 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f97ef575:::benchmark_data_smithi179_77138_object75130:head# 2024-02-09T17:26:53.163 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f97f5488:::benchmark_data_smithi179_77138_object89477:head# 2024-02-09T17:26:53.164 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9804dac:::benchmark_data_smithi179_77138_object23805:head# 2024-02-09T17:26:53.164 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9807d39:::benchmark_data_smithi179_77138_object37852:head# 2024-02-09T17:26:53.165 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9811c1e:::benchmark_data_smithi179_77138_object87642:head# 2024-02-09T17:26:53.165 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9813e9e:::benchmark_data_smithi179_77138_object79096:head# 2024-02-09T17:26:53.166 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f981717a:::benchmark_data_smithi179_77138_object58167:head# 2024-02-09T17:26:53.166 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f98201cd:::benchmark_data_smithi179_77138_object56771:head# 2024-02-09T17:26:53.166 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9823157:::benchmark_data_smithi179_77138_object112540:head# 2024-02-09T17:26:53.167 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f982fef3:::benchmark_data_smithi179_77138_object109542:head# 2024-02-09T17:26:53.167 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9833277:::benchmark_data_smithi179_77138_object131465:head# 2024-02-09T17:26:53.168 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9837c1e:::benchmark_data_smithi179_77138_object97234:head# 2024-02-09T17:26:53.168 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9844681:::benchmark_data_smithi179_77138_object26114:head# 2024-02-09T17:26:53.168 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9846dcc:::benchmark_data_smithi179_77138_object125673:head# 2024-02-09T17:26:53.169 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9847205:::benchmark_data_smithi179_77138_object92885:head# 2024-02-09T17:26:53.170 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9847585:::benchmark_data_smithi179_77138_object41013:head# 2024-02-09T17:26:53.170 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f984ab30:::benchmark_data_smithi179_77138_object26766:head# 2024-02-09T17:26:53.170 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f98521d6:::benchmark_data_smithi179_77138_object86623:head# 2024-02-09T17:26:53.172 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9852ad6:::benchmark_data_smithi179_77138_object91886:head# 2024-02-09T17:26:53.172 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9859d57:::benchmark_data_smithi179_77138_object113001:head# 2024-02-09T17:26:53.175 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f985df66:::benchmark_data_smithi179_77138_object87542:head# 2024-02-09T17:26:53.175 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f986d30d:::benchmark_data_smithi179_77138_object87305:head# 2024-02-09T17:26:53.175 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f987bd44:::benchmark_data_smithi179_77138_object116062:head# 2024-02-09T17:26:53.175 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f987c1b9:::benchmark_data_smithi179_77138_object121099:head# 2024-02-09T17:26:53.176 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f987cd8e:::benchmark_data_smithi179_77138_object102352:head# 2024-02-09T17:26:53.176 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f98a709b:::benchmark_data_smithi179_77138_object64155:head# 2024-02-09T17:26:53.176 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f98a8e74:::benchmark_data_smithi179_77138_object52284:head# 2024-02-09T17:26:53.176 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f98b8a10:::benchmark_data_smithi179_77138_object61883:head# 2024-02-09T17:26:53.176 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f98b99a9:::benchmark_data_smithi179_77138_object22795:head# 2024-02-09T17:26:53.177 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f98c1d8c:::benchmark_data_smithi179_77138_object61220:head# 2024-02-09T17:26:53.178 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f98e5166:::benchmark_data_smithi179_77138_object31087:head# 2024-02-09T17:26:53.178 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f98e6f9d:::benchmark_data_smithi179_77138_object123419:head# 2024-02-09T17:26:53.178 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f98ed341:::benchmark_data_smithi179_77138_object126613:head# 2024-02-09T17:26:53.179 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f98f6a20:::benchmark_data_smithi179_77138_object88902:head# 2024-02-09T17:26:53.181 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f990ef9e:::benchmark_data_smithi179_77138_object126181:head# 2024-02-09T17:26:53.182 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f990fa0d:::benchmark_data_smithi179_77138_object119385:head# 2024-02-09T17:26:53.182 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9918f64:::benchmark_data_smithi179_77138_object84111:head# 2024-02-09T17:26:53.182 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9919ab0:::benchmark_data_smithi179_77138_object37435:head# 2024-02-09T17:26:53.183 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f992ce16:::benchmark_data_smithi179_77138_object91078:head# 2024-02-09T17:26:53.183 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f993c4bf:::benchmark_data_smithi179_77138_object32478:head# 2024-02-09T17:26:53.183 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f993fe81:::benchmark_data_smithi179_77138_object24604:head# 2024-02-09T17:26:53.184 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f994095b:::benchmark_data_smithi179_77138_object39123:head# 2024-02-09T17:26:53.184 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f994225f:::benchmark_data_smithi179_77138_object80241:head# 2024-02-09T17:26:53.185 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f994f95f:::benchmark_data_smithi179_77138_object28208:head# 2024-02-09T17:26:53.185 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f995ce1d:::benchmark_data_smithi179_77138_object118394:head# 2024-02-09T17:26:53.185 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f997d102:::benchmark_data_smithi179_77138_object36138:head# 2024-02-09T17:26:53.186 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f997e05c:::benchmark_data_smithi179_77138_object131009:head# 2024-02-09T17:26:53.186 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f997eb92:::benchmark_data_smithi179_77138_object66951:head# 2024-02-09T17:26:53.187 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9986771:::benchmark_data_smithi179_77138_object120923:head# 2024-02-09T17:26:53.187 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f998830a:::benchmark_data_smithi179_77138_object23355:head# 2024-02-09T17:26:53.188 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f99945d7:::benchmark_data_smithi179_77138_object35062:head# 2024-02-09T17:26:53.188 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9998a46:::benchmark_data_smithi179_77138_object40842:head# 2024-02-09T17:26:53.189 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9998b57:::benchmark_data_smithi179_77138_object128345:head# 2024-02-09T17:26:53.190 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f99b7bb7:::benchmark_data_smithi179_77138_object56359:head# 2024-02-09T17:26:53.190 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f99bf315:::benchmark_data_smithi179_77138_object25356:head# 2024-02-09T17:26:53.191 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f99d1cb7:::benchmark_data_smithi179_77138_object106828:head# 2024-02-09T17:26:53.191 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f99d6d83:::benchmark_data_smithi179_77138_object80249:head# 2024-02-09T17:26:53.192 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f99d8d70:::benchmark_data_smithi179_77138_object71645:head# 2024-02-09T17:26:53.192 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f99dc7e6:::benchmark_data_smithi179_77138_object119192:head# 2024-02-09T17:26:53.192 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f99f3cc2:::benchmark_data_smithi179_77138_object78262:head# 2024-02-09T17:26:53.193 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f99fde23:::benchmark_data_smithi179_77138_object87010:head# 2024-02-09T17:26:53.193 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9a0cb42:::benchmark_data_smithi179_77138_object104173:head# 2024-02-09T17:26:53.194 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9a0de41:::benchmark_data_smithi179_77138_object46343:head# 2024-02-09T17:26:53.194 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9a112e6:::benchmark_data_smithi179_77138_object73672:head# 2024-02-09T17:26:53.195 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9a12a4b:::benchmark_data_smithi179_77138_object75056:head# 2024-02-09T17:26:53.195 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9a13651:::benchmark_data_smithi179_77138_object39370:head# 2024-02-09T17:26:53.196 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9a230b5:::benchmark_data_smithi179_77138_object34221:head# 2024-02-09T17:26:53.196 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9a27071:::benchmark_data_smithi179_77138_object39850:head# 2024-02-09T17:26:53.196 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9a2ca20:::benchmark_data_smithi179_77138_object19930:head# 2024-02-09T17:26:53.197 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9a3efa9:::benchmark_data_smithi179_77138_object114511:head# 2024-02-09T17:26:53.197 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9a44950:::benchmark_data_smithi179_77138_object83558:head# 2024-02-09T17:26:53.198 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9a67022:::benchmark_data_smithi179_77138_object67352:head# 2024-02-09T17:26:53.198 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9a6ec30:::benchmark_data_smithi179_77138_object130075:head# 2024-02-09T17:26:53.198 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9a7cd76:::benchmark_data_smithi179_77138_object22023:head# 2024-02-09T17:26:53.199 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9a7fc63:::benchmark_data_smithi179_77138_object132711:head# 2024-02-09T17:26:53.199 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9a8206e:::benchmark_data_smithi179_77138_object67771:head# 2024-02-09T17:26:53.200 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9a82cdf:::benchmark_data_smithi179_77138_object27378:head# 2024-02-09T17:26:53.200 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9a9292b:::benchmark_data_smithi179_77138_object70382:head# 2024-02-09T17:26:53.200 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9a95a20:::benchmark_data_smithi179_77138_object119411:head# 2024-02-09T17:26:53.201 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9a96a92:::benchmark_data_smithi179_77138_object79566:head# 2024-02-09T17:26:53.201 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9a97a65:::benchmark_data_smithi179_77138_object23564:head# 2024-02-09T17:26:53.204 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9a9c7a0:::benchmark_data_smithi179_77138_object93718:head# 2024-02-09T17:26:53.204 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9a9fa28:::benchmark_data_smithi179_77138_object70332:head# 2024-02-09T17:26:53.204 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9aa8a40:::benchmark_data_smithi179_77138_object26906:head# 2024-02-09T17:26:53.204 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9ac1097:::benchmark_data_smithi179_77138_object98699:head# 2024-02-09T17:26:53.204 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9ac7c25:::benchmark_data_smithi179_77138_object97367:head# 2024-02-09T17:26:53.205 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9ad06bf:::benchmark_data_smithi179_77138_object44403:head# 2024-02-09T17:26:53.205 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9aec103:::benchmark_data_smithi179_77138_object24815:head# 2024-02-09T17:26:53.205 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9af0339:::benchmark_data_smithi179_77138_object25092:head# 2024-02-09T17:26:53.206 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9b075f6:::benchmark_data_smithi179_77138_object92055:head# 2024-02-09T17:26:53.206 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9b0996b:::benchmark_data_smithi179_77138_object104939:head# 2024-02-09T17:26:53.206 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9b11fd1:::benchmark_data_smithi179_77138_object22999:head# 2024-02-09T17:26:53.206 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9b1ead7:::benchmark_data_smithi179_77138_object132693:head# 2024-02-09T17:26:53.207 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9b23e04:::benchmark_data_smithi179_77138_object73197:head# 2024-02-09T17:26:53.207 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9b2ff81:::benchmark_data_smithi179_77138_object122810:head# 2024-02-09T17:26:53.208 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9b37c47:::benchmark_data_smithi179_77138_object72019:head# 2024-02-09T17:26:53.208 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9b3b121:::benchmark_data_smithi179_77138_object47864:head# 2024-02-09T17:26:53.210 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9b481ef:::benchmark_data_smithi179_77138_object38916:head# 2024-02-09T17:26:53.210 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9b4f45d:::benchmark_data_smithi179_77138_object55385:head# 2024-02-09T17:26:53.210 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9b570e9:::benchmark_data_smithi179_77138_object42668:head# 2024-02-09T17:26:53.210 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9b65185:::benchmark_data_smithi179_77138_object55539:head# 2024-02-09T17:26:53.210 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9b6e62a:::benchmark_data_smithi179_77138_object89595:head# 2024-02-09T17:26:53.211 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9b70991:::benchmark_data_smithi179_77138_object127306:head# 2024-02-09T17:26:53.211 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9b755a1:::benchmark_data_smithi179_77138_object45909:head# 2024-02-09T17:26:53.211 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9b78dd3:::benchmark_data_smithi179_77138_object106387:head# 2024-02-09T17:26:53.212 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9b79c6e:::benchmark_data_smithi179_77138_object52827:head# 2024-02-09T17:26:53.212 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9b9df1e:::benchmark_data_smithi179_77138_object66676:head# 2024-02-09T17:26:53.213 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9ba6f45:::benchmark_data_smithi179_77138_object109515:head# 2024-02-09T17:26:53.213 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:53.214 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9bab8d4:::benchmark_data_smithi179_77138_object41975:head# 2024-02-09T17:26:53.215 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9bb0708:::benchmark_data_smithi179_77138_object104069:head# 2024-02-09T17:26:53.215 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9bb9e46:::benchmark_data_smithi179_77138_object125420:head# 2024-02-09T17:26:53.215 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9bde3a2:::benchmark_data_smithi179_77138_object125872:head# 2024-02-09T17:26:53.216 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9bee9a1:::benchmark_data_smithi179_77138_object30985:head# 2024-02-09T17:26:53.216 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9bef44f:::benchmark_data_smithi179_77138_object30721:head# 2024-02-09T17:26:53.216 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9bf1314:::benchmark_data_smithi179_77138_object117988:head# 2024-02-09T17:26:53.216 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9bf526d:::benchmark_data_smithi179_77138_object73181:head# 2024-02-09T17:26:53.216 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9bf5b1a:::benchmark_data_smithi179_77138_object46757:head# 2024-02-09T17:26:53.217 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9c00f94:::benchmark_data_smithi179_77138_object75081:head# 2024-02-09T17:26:53.217 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9c18c16:::benchmark_data_smithi179_77138_object96027:head# 2024-02-09T17:26:53.218 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9c20635:::benchmark_data_smithi179_77138_object70530:head# 2024-02-09T17:26:53.218 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9c2573d:::benchmark_data_smithi179_77138_object28479:head# 2024-02-09T17:26:53.220 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9c2d336:::benchmark_data_smithi179_77138_object20112:head# 2024-02-09T17:26:53.220 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9c2fdee:::benchmark_data_smithi179_77138_object53901:head# 2024-02-09T17:26:53.221 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9c40cc1:::benchmark_data_smithi179_77138_object27676:head# 2024-02-09T17:26:53.221 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9c431d4:::benchmark_data_smithi179_77138_object46471:head# 2024-02-09T17:26:53.222 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9c661b3:::benchmark_data_smithi179_77138_object23286:head# 2024-02-09T17:26:53.222 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9c69ce0:::benchmark_data_smithi179_77138_object115966:head# 2024-02-09T17:26:53.226 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9c6ade1:::benchmark_data_smithi179_77138_object54406:head# 2024-02-09T17:26:53.227 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9c92119:::benchmark_data_smithi179_77138_object23023:head# 2024-02-09T17:26:53.227 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9c9276a:::benchmark_data_smithi179_77138_object82020:head# 2024-02-09T17:26:53.227 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9c99c04:::benchmark_data_smithi179_77138_object58603:head# 2024-02-09T17:26:53.227 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9ca6c83:::benchmark_data_smithi179_77138_object102128:head# 2024-02-09T17:26:53.227 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9cc873c:::benchmark_data_smithi179_77138_object106141:head# 2024-02-09T17:26:53.227 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9ccaa02:::benchmark_data_smithi179_77138_object48495:head# 2024-02-09T17:26:53.227 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9ccf928:::benchmark_data_smithi179_77138_object32027:head# 2024-02-09T17:26:53.227 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9cd0094:::benchmark_data_smithi179_77138_object132798:head# 2024-02-09T17:26:53.227 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9cd2d56:::benchmark_data_smithi179_77138_object67488:head# 2024-02-09T17:26:53.232 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9cda960:::benchmark_data_smithi179_77138_object104794:head# 2024-02-09T17:26:53.232 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9ce9cd7:::benchmark_data_smithi179_77138_object88668:head# 2024-02-09T17:26:53.233 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9cfb703:::benchmark_data_smithi179_77138_object85831:head# 2024-02-09T17:26:53.233 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9d089d2:::benchmark_data_smithi179_77138_object24486:head# 2024-02-09T17:26:53.233 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9d08ee4:::benchmark_data_smithi179_77138_object34821:head# 2024-02-09T17:26:53.234 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9d0b37c:::benchmark_data_smithi179_77138_object73522:head# 2024-02-09T17:26:53.234 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9d0c40f:::benchmark_data_smithi179_77138_object83230:head# 2024-02-09T17:26:53.235 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9d0ca36:::benchmark_data_smithi179_77138_object94020:head# 2024-02-09T17:26:53.235 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9d0f221:::benchmark_data_smithi179_77138_object101912:head# 2024-02-09T17:26:53.235 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9d29db0:::benchmark_data_smithi179_77138_object79621:head# 2024-02-09T17:26:53.236 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9d3e3ce:::benchmark_data_smithi179_77138_object97254:head# 2024-02-09T17:26:53.237 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9d5f9d6:::benchmark_data_smithi179_77138_object47921:head# 2024-02-09T17:26:53.237 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9d604c9:::benchmark_data_smithi179_77138_object88847:head# 2024-02-09T17:26:53.241 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:26:53.242 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9d69479:::benchmark_data_smithi179_77138_object100640:head# 2024-02-09T17:26:53.242 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9d71dd3:::benchmark_data_smithi179_77138_object47978:head# 2024-02-09T17:26:53.243 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9d72d77:::benchmark_data_smithi179_77138_object29983:head# 2024-02-09T17:26:53.243 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9d7dd4c:::benchmark_data_smithi179_77138_object81257:head# 2024-02-09T17:26:53.243 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9d7e8de:::benchmark_data_smithi179_77138_object49771:head# 2024-02-09T17:26:53.243 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9d8611a:::benchmark_data_smithi179_77138_object35544:head# 2024-02-09T17:26:53.243 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9d95d52:::benchmark_data_smithi179_77138_object84763:head# 2024-02-09T17:26:53.243 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9d9678f:::benchmark_data_smithi179_77138_object127046:head# 2024-02-09T17:26:53.243 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9d9b20e:::benchmark_data_smithi179_77138_object31134:head# 2024-02-09T17:26:53.244 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9da0646:::benchmark_data_smithi179_77138_object22823:head# 2024-02-09T17:26:53.244 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9db1638:::benchmark_data_smithi179_77138_object39086:head# 2024-02-09T17:26:53.244 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9dc04e6:::benchmark_data_smithi179_77138_object109880:head# 2024-02-09T17:26:53.245 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9dc0fa8:::benchmark_data_smithi179_77138_object81010:head# 2024-02-09T17:26:53.245 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9dc47c9:::benchmark_data_smithi179_77138_object95378:head# 2024-02-09T17:26:53.246 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9dc97d3:::benchmark_data_smithi179_77138_object72661:head# 2024-02-09T17:26:53.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:53.248 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9de8a14:::benchmark_data_smithi179_77138_object119226:head# 2024-02-09T17:26:53.248 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9df170b:::benchmark_data_smithi179_77138_object97541:head# 2024-02-09T17:26:53.248 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9e076a0:::benchmark_data_smithi179_77138_object121045:head# 2024-02-09T17:26:53.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:26:53.249 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9e0a14c:::benchmark_data_smithi179_77138_object32673:head# 2024-02-09T17:26:53.250 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9e1349a:::benchmark_data_smithi179_77138_object79634:head# 2024-02-09T17:26:53.250 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9e15195:::benchmark_data_smithi179_77138_object94346:head# 2024-02-09T17:26:53.251 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9e17826:::benchmark_data_smithi179_77138_object108774:head# 2024-02-09T17:26:53.251 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9e1a94c:::benchmark_data_smithi179_77138_object112751:head# 2024-02-09T17:26:53.251 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9e1cd6b:::benchmark_data_smithi179_77138_object48882:head# 2024-02-09T17:26:53.252 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9e2194b:::benchmark_data_smithi179_77138_object53689:head# 2024-02-09T17:26:53.252 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9e23819:::benchmark_data_smithi179_77138_object59383:head# 2024-02-09T17:26:53.252 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9e2aac2:::benchmark_data_smithi179_77138_object23935:head# 2024-02-09T17:26:53.253 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9e2fdc1:::benchmark_data_smithi179_77138_object20811:head# 2024-02-09T17:26:53.253 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9e3eea9:::benchmark_data_smithi179_77138_object94742:head# 2024-02-09T17:26:53.254 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9e4ec0a:::benchmark_data_smithi179_77138_object87107:head# 2024-02-09T17:26:53.254 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9e5899d:::benchmark_data_smithi179_77138_object87955:head# 2024-02-09T17:26:53.255 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9e61bb1:::benchmark_data_smithi179_77138_object104529:head# 2024-02-09T17:26:53.259 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9e63859:::benchmark_data_smithi179_77138_object79702:head# 2024-02-09T17:26:53.259 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9e6b8cc:::benchmark_data_smithi179_77138_object84691:head# 2024-02-09T17:26:53.259 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9e7931d:::benchmark_data_smithi179_77138_object103758:head# 2024-02-09T17:26:53.260 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9e88be4:::benchmark_data_smithi179_77138_object64604:head# 2024-02-09T17:26:53.260 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9e97cac:::benchmark_data_smithi179_77138_object40444:head# 2024-02-09T17:26:53.260 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9eaea53:::benchmark_data_smithi179_77138_object129203:head# 2024-02-09T17:26:53.260 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9eb0804:::benchmark_data_smithi179_77138_object96613:head# 2024-02-09T17:26:53.260 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9eb6556:::benchmark_data_smithi179_77138_object88853:head# 2024-02-09T17:26:53.261 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9eccdc5:::benchmark_data_smithi179_77138_object85079:head# 2024-02-09T17:26:53.261 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9ed10ca:::benchmark_data_smithi179_77138_object117433:head# 2024-02-09T17:26:53.261 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9edb574:::benchmark_data_smithi179_77138_object71162:head# 2024-02-09T17:26:53.262 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9edffdf:::benchmark_data_smithi179_77138_object126448:head# 2024-02-09T17:26:53.263 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:26:53.263 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9ee3d68:::benchmark_data_smithi179_77138_object60929:head# 2024-02-09T17:26:53.263 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9ee98ef:::benchmark_data_smithi179_77138_object129113:head# 2024-02-09T17:26:53.263 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9ef8b5f:::benchmark_data_smithi179_77138_object129260:head# 2024-02-09T17:26:53.264 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9f07bb6:::benchmark_data_smithi179_77138_object29448:head# 2024-02-09T17:26:53.267 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9f25b60:::benchmark_data_smithi179_77138_object76974:head# 2024-02-09T17:26:53.268 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9f3aab4:::benchmark_data_smithi179_77138_object110892:head# 2024-02-09T17:26:53.268 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9f429e3:::benchmark_data_smithi179_77138_object19641:head# 2024-02-09T17:26:53.271 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9f5a4f1:::benchmark_data_smithi179_77138_object107666:head# 2024-02-09T17:26:53.271 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9f604f6:::benchmark_data_smithi179_77138_object130084:head# 2024-02-09T17:26:53.272 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9f6ec9a:::benchmark_data_smithi179_77138_object65992:head# 2024-02-09T17:26:53.272 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9f890bf:::benchmark_data_smithi179_77138_object47006:head# 2024-02-09T17:26:53.273 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9f8a002:::benchmark_data_smithi179_77138_object115320:head# 2024-02-09T17:26:53.273 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9f8acce:::benchmark_data_smithi179_77138_object52760:head# 2024-02-09T17:26:53.273 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9fa2884:::benchmark_data_smithi179_77138_object96935:head# 2024-02-09T17:26:53.274 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9fa3d85:::benchmark_data_smithi179_77138_object54310:head# 2024-02-09T17:26:53.274 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9fc2afb:::benchmark_data_smithi179_77138_object70226:head# 2024-02-09T17:26:53.276 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9fc3792:::benchmark_data_smithi179_77138_object25107:head# 2024-02-09T17:26:53.276 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9fccbff:::benchmark_data_smithi179_77138_object73199:head# 2024-02-09T17:26:53.277 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9fcd7da:::benchmark_data_smithi179_77138_object21480:head# 2024-02-09T17:26:53.277 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9fd9688:::benchmark_data_smithi179_77138_object65859:head# 2024-02-09T17:26:53.277 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9fdc379:::benchmark_data_smithi179_77138_object36711:head# 2024-02-09T17:26:53.278 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:f9fe8457:::benchmark_data_smithi179_77138_object114356:head# 2024-02-09T17:26:53.278 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa005e1b:::benchmark_data_smithi179_77138_object43305:head# 2024-02-09T17:26:53.278 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa00aada:::benchmark_data_smithi179_77138_object78301:head# 2024-02-09T17:26:53.279 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa00bab8:::benchmark_data_smithi179_77138_object33022:head# 2024-02-09T17:26:53.279 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa00c937:::benchmark_data_smithi179_77138_object126073:head# 2024-02-09T17:26:53.280 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa029c0f:::benchmark_data_smithi179_77138_object107355:head# 2024-02-09T17:26:53.280 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa03cfd3:::benchmark_data_smithi179_77138_object21833:head# 2024-02-09T17:26:53.280 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa03dfcc:::benchmark_data_smithi179_77138_object45236:head# 2024-02-09T17:26:53.281 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa0459ff:::benchmark_data_smithi179_77138_object24954:head# 2024-02-09T17:26:53.281 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa046286:::benchmark_data_smithi179_77138_object28244:head# 2024-02-09T17:26:53.282 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa04a38d:::benchmark_data_smithi179_77138_object68874:head# 2024-02-09T17:26:53.282 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa04f7d7:::benchmark_data_smithi179_77138_object78336:head# 2024-02-09T17:26:53.283 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa06f4ab:::benchmark_data_smithi179_77138_object91728:head# 2024-02-09T17:26:53.283 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa08047c:::benchmark_data_smithi179_77138_object30029:head# 2024-02-09T17:26:53.283 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa081541:::benchmark_data_smithi179_77138_object76175:head# 2024-02-09T17:26:53.284 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa08e3d2:::benchmark_data_smithi179_77138_object94221:head# 2024-02-09T17:26:53.284 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa08eb40:::benchmark_data_smithi179_77138_object49399:head# 2024-02-09T17:26:53.284 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa090534:::benchmark_data_smithi179_77138_object128974:head# 2024-02-09T17:26:53.285 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa09ca01:::benchmark_data_smithi179_77138_object45385:head# 2024-02-09T17:26:53.285 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa0a1365:::benchmark_data_smithi179_77138_object98174:head# 2024-02-09T17:26:53.286 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa0a17b6:::benchmark_data_smithi179_77138_object56960:head# 2024-02-09T17:26:53.288 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa0a9058:::benchmark_data_smithi179_77138_object46177:head# 2024-02-09T17:26:53.288 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa0a9e3f:::benchmark_data_smithi179_77138_object41148:head# 2024-02-09T17:26:53.288 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa0bb5bb:::benchmark_data_smithi179_77138_object100176:head# 2024-02-09T17:26:53.289 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa0bd095:::benchmark_data_smithi179_77138_object66484:head# 2024-02-09T17:26:53.289 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa0c86c5:::benchmark_data_smithi179_77138_object39357:head# 2024-02-09T17:26:53.289 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa10a25c:::benchmark_data_smithi179_77138_object39964:head# 2024-02-09T17:26:53.290 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa10e0da:::benchmark_data_smithi179_77138_object59893:head# 2024-02-09T17:26:53.290 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa119309:::benchmark_data_smithi179_77138_object88008:head# 2024-02-09T17:26:53.291 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa125ed4:::benchmark_data_smithi179_77138_object51226:head# 2024-02-09T17:26:53.291 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa131ed9:::benchmark_data_smithi179_77138_object30545:head# 2024-02-09T17:26:53.291 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa144d3a:::benchmark_data_smithi179_77138_object72037:head# 2024-02-09T17:26:53.292 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa147cb8:::benchmark_data_smithi179_77138_object63667:head# 2024-02-09T17:26:53.292 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa154d63:::benchmark_data_smithi179_77138_object72471:head# 2024-02-09T17:26:53.293 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa16705b:::benchmark_data_smithi179_77138_object25261:head# 2024-02-09T17:26:53.293 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa16bd33:::benchmark_data_smithi179_77138_object57794:head# 2024-02-09T17:26:53.293 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa18a509:::benchmark_data_smithi179_77138_object98574:head# 2024-02-09T17:26:53.294 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa18e3a4:::benchmark_data_smithi179_77138_object119836:head# 2024-02-09T17:26:53.294 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa196519:::benchmark_data_smithi179_77138_object64261:head# 2024-02-09T17:26:53.295 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa198247:::benchmark_data_smithi179_77138_object102146:head# 2024-02-09T17:26:53.295 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa1aa882:::benchmark_data_smithi179_77138_object51721:head# 2024-02-09T17:26:53.296 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa1b7d5f:::benchmark_data_smithi179_77138_object131598:head# 2024-02-09T17:26:53.297 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa1c0853:::benchmark_data_smithi179_77138_object122325:head# 2024-02-09T17:26:53.298 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa1cf469:::benchmark_data_smithi179_77138_object77651:head# 2024-02-09T17:26:53.298 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa1d061a:::benchmark_data_smithi179_77138_object73557:head# 2024-02-09T17:26:53.299 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa1e7965:::benchmark_data_smithi179_77138_object96627:head# 2024-02-09T17:26:53.299 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa1ef08b:::benchmark_data_smithi179_77138_object114844:head# 2024-02-09T17:26:53.299 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa1f30c3:::benchmark_data_smithi179_77138_object27652:head# 2024-02-09T17:26:53.300 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa1f71c7:::benchmark_data_smithi179_77138_object123429:head# 2024-02-09T17:26:53.300 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa1f81af:::benchmark_data_smithi179_77138_object41756:head# 2024-02-09T17:26:53.301 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa1fb9f9:::benchmark_data_smithi179_77138_object47319:head# 2024-02-09T17:26:53.301 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa206fc9:::benchmark_data_smithi179_77138_object79883:head# 2024-02-09T17:26:53.301 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa20849d:::benchmark_data_smithi179_77138_object131919:head# 2024-02-09T17:26:53.303 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa20ce12:::benchmark_data_smithi179_77138_object75907:head# 2024-02-09T17:26:53.304 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa213ee9:::benchmark_data_smithi179_77138_object27088:head# 2024-02-09T17:26:53.304 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa222314:::benchmark_data_smithi179_77138_object108546:head# 2024-02-09T17:26:53.305 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa22557f:::benchmark_data_smithi179_77138_object129900:head# 2024-02-09T17:26:53.305 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa240ef6:::benchmark_data_smithi179_77138_object121319:head# 2024-02-09T17:26:53.306 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa24ce24:::benchmark_data_smithi179_77138_object94071:head# 2024-02-09T17:26:53.306 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa25e606:::benchmark_data_smithi179_77138_object55267:head# 2024-02-09T17:26:53.306 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa25f2ee:::benchmark_data_smithi179_77138_object23806:head# 2024-02-09T17:26:53.307 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa25fa9f:::benchmark_data_smithi179_77138_object32924:head# 2024-02-09T17:26:53.307 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa26402c:::benchmark_data_smithi179_77138_object104602:head# 2024-02-09T17:26:53.308 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa26d6c0:::benchmark_data_smithi179_77138_object44498:head# 2024-02-09T17:26:53.311 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa279be5:::benchmark_data_smithi179_77138_object109132:head# 2024-02-09T17:26:53.311 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa284324:::benchmark_data_smithi179_77138_object120546:head# 2024-02-09T17:26:53.312 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa28eb31:::benchmark_data_smithi179_77138_object79909:head# 2024-02-09T17:26:53.312 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa294d54:::benchmark_data_smithi179_77138_object44787:head# 2024-02-09T17:26:53.313 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa296952:::benchmark_data_smithi179_77138_object77804:head# 2024-02-09T17:26:53.315 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa29a5d9:::benchmark_data_smithi179_77138_object29508:head# 2024-02-09T17:26:53.315 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa29bafb:::benchmark_data_smithi179_77138_object101354:head# 2024-02-09T17:26:53.316 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa29ce67:::benchmark_data_smithi179_77138_object37507:head# 2024-02-09T17:26:53.316 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa29fc74:::benchmark_data_smithi179_77138_object62306:head# 2024-02-09T17:26:53.316 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa2a13b9:::benchmark_data_smithi179_77138_object68840:head# 2024-02-09T17:26:53.318 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa2a1e25:::benchmark_data_smithi179_77138_object132003:head# 2024-02-09T17:26:53.318 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa2a6765:::benchmark_data_smithi179_77138_object92295:head# 2024-02-09T17:26:53.318 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa2aadf7:::benchmark_data_smithi179_77138_object77307:head# 2024-02-09T17:26:53.318 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa2acdcd:::benchmark_data_smithi179_77138_object79158:head# 2024-02-09T17:26:53.319 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa2b487d:::benchmark_data_smithi179_77138_object47022:head# 2024-02-09T17:26:53.319 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa2bb237:::benchmark_data_smithi179_77138_object75826:head# 2024-02-09T17:26:53.319 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa2bb427:::benchmark_data_smithi179_77138_object121058:head# 2024-02-09T17:26:53.320 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa2bc8e7:::benchmark_data_smithi179_77138_object64088:head# 2024-02-09T17:26:53.324 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa2d4520:::benchmark_data_smithi179_77138_object80367:head# 2024-02-09T17:26:53.324 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa2e6b93:::benchmark_data_smithi179_77138_object67129:head# 2024-02-09T17:26:53.324 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa2ec11d:::benchmark_data_smithi179_77138_object83642:head# 2024-02-09T17:26:53.324 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa306835:::benchmark_data_smithi179_77138_object24959:head# 2024-02-09T17:26:53.324 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa30796a:::benchmark_data_smithi179_77138_object76210:head# 2024-02-09T17:26:53.325 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa30c1d8:::benchmark_data_smithi179_77138_object27559:head# 2024-02-09T17:26:53.325 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa316762:::benchmark_data_smithi179_77138_object54814:head# 2024-02-09T17:26:53.325 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa31715c:::benchmark_data_smithi179_77138_object20100:head# 2024-02-09T17:26:53.325 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa31829f:::benchmark_data_smithi179_77138_object47851:head# 2024-02-09T17:26:53.328 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa31bb60:::benchmark_data_smithi179_77138_object123218:head# 2024-02-09T17:26:53.328 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa3284a8:::benchmark_data_smithi179_77138_object109852:head# 2024-02-09T17:26:53.328 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa32a132:::benchmark_data_smithi179_77138_object127674:head# 2024-02-09T17:26:53.328 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa33136c:::benchmark_data_smithi179_77138_object108589:head# 2024-02-09T17:26:53.328 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa333adb:::benchmark_data_smithi179_77138_object69025:head# 2024-02-09T17:26:53.331 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa336640:::benchmark_data_smithi179_77138_object117810:head# 2024-02-09T17:26:53.331 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa33e10b:::benchmark_data_smithi179_77138_object96779:head# 2024-02-09T17:26:53.331 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa348679:::benchmark_data_smithi179_77138_object58967:head# 2024-02-09T17:26:53.336 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa353e55:::benchmark_data_smithi179_77138_object27674:head# 2024-02-09T17:26:53.336 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa35669e:::benchmark_data_smithi179_77138_object19413:head# 2024-02-09T17:26:53.336 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa364520:::benchmark_data_smithi179_77138_object87483:head# 2024-02-09T17:26:53.336 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa36f6a1:::benchmark_data_smithi179_77138_object127598:head# 2024-02-09T17:26:53.336 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa3766ae:::benchmark_data_smithi179_77138_object64685:head# 2024-02-09T17:26:53.337 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa380d42:::benchmark_data_smithi179_77138_object72709:head# 2024-02-09T17:26:53.337 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa3821c7:::benchmark_data_smithi179_77138_object123629:head# 2024-02-09T17:26:53.337 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa383c9e:::benchmark_data_smithi179_77138_object124185:head# 2024-02-09T17:26:53.338 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa38ce06:::benchmark_data_smithi179_77138_object70751:head# 2024-02-09T17:26:53.342 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa3afd99:::benchmark_data_smithi179_77138_object100801:head# 2024-02-09T17:26:53.342 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa3b1649:::benchmark_data_smithi179_77138_object104705:head# 2024-02-09T17:26:53.343 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa3b8b83:::benchmark_data_smithi179_77138_object92917:head# 2024-02-09T17:26:53.343 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa3ca332:::benchmark_data_smithi179_77138_object23810:head# 2024-02-09T17:26:53.348 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa3ce975:::benchmark_data_smithi179_77138_object35843:head# 2024-02-09T17:26:53.348 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa3d14ab:::benchmark_data_smithi179_77138_object47684:head# 2024-02-09T17:26:53.348 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa3d16e7:::benchmark_data_smithi179_77138_object66240:head# 2024-02-09T17:26:53.348 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa3d5bb4:::benchmark_data_smithi179_77138_object57495:head# 2024-02-09T17:26:53.348 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa3e2dfe:::benchmark_data_smithi179_77138_object19809:head# 2024-02-09T17:26:53.348 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa3e8845:::benchmark_data_smithi179_77138_object72076:head# 2024-02-09T17:26:53.348 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa3f33be:::benchmark_data_smithi179_77138_object34186:head# 2024-02-09T17:26:53.348 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa403f5d:::benchmark_data_smithi179_77138_object96817:head# 2024-02-09T17:26:53.348 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa4301c4:::benchmark_data_smithi179_77138_object66704:head# 2024-02-09T17:26:53.349 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa4320fc:::benchmark_data_smithi179_77138_object108769:head# 2024-02-09T17:26:53.349 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa43286d:::benchmark_data_smithi179_77138_object108989:head# 2024-02-09T17:26:53.349 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa436b3b:::benchmark_data_smithi179_77138_object75119:head# 2024-02-09T17:26:53.349 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa43b034:::benchmark_data_smithi179_77138_object129636:head# 2024-02-09T17:26:53.349 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa449279:::benchmark_data_smithi179_77138_object31979:head# 2024-02-09T17:26:53.349 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa46a279:::benchmark_data_smithi179_77138_object117101:head# 2024-02-09T17:26:53.349 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:26:53.350 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa46e330:::benchmark_data_smithi179_77138_object21309:head# 2024-02-09T17:26:53.350 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa498d13:::benchmark_data_smithi179_77138_object57502:head# 2024-02-09T17:26:53.350 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa49bb1d:::benchmark_data_smithi179_77138_object106934:head# 2024-02-09T17:26:53.353 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa49e55d:::benchmark_data_smithi179_77138_object81518:head# 2024-02-09T17:26:53.353 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa4a3afb:::benchmark_data_smithi179_77138_object128163:head# 2024-02-09T17:26:53.353 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa4a7895:::benchmark_data_smithi179_77138_object99155:head# 2024-02-09T17:26:53.353 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa4a7ab6:::benchmark_data_smithi179_77138_object97108:head# 2024-02-09T17:26:53.353 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa4ab1ab:::benchmark_data_smithi179_77138_object87588:head# 2024-02-09T17:26:53.353 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa4b1331:::benchmark_data_smithi179_77138_object92496:head# 2024-02-09T17:26:53.354 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa4bbe54:::benchmark_data_smithi179_77138_object131249:head# 2024-02-09T17:26:53.354 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa4bbf35:::benchmark_data_smithi179_77138_object57797:head# 2024-02-09T17:26:53.354 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa4dbf49:::benchmark_data_smithi179_77138_object64481:head# 2024-02-09T17:26:53.355 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa4e2618:::benchmark_data_smithi179_77138_object38798:head# 2024-02-09T17:26:53.355 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa4e9d52:::benchmark_data_smithi179_77138_object85246:head# 2024-02-09T17:26:53.355 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa4fce31:::benchmark_data_smithi179_77138_object98032:head# 2024-02-09T17:26:53.355 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa5075a5:::benchmark_data_smithi179_77138_object129779:head# 2024-02-09T17:26:53.355 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa52bbee:::benchmark_data_smithi179_77138_object29367:head# 2024-02-09T17:26:53.356 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa53532f:::benchmark_data_smithi179_77138_object118753:head# 2024-02-09T17:26:53.356 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa538eab:::benchmark_data_smithi179_77138_object107379:head# 2024-02-09T17:26:53.356 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa5589ba:::benchmark_data_smithi179_77138_object88700:head# 2024-02-09T17:26:53.357 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa56265b:::benchmark_data_smithi179_77138_object95210:head# 2024-02-09T17:26:53.357 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa565d20:::benchmark_data_smithi179_77138_object131091:head# 2024-02-09T17:26:53.357 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa571ffd:::benchmark_data_smithi179_77138_object21748:head# 2024-02-09T17:26:53.359 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa573ba4:::benchmark_data_smithi179_77138_object69972:head# 2024-02-09T17:26:53.359 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa57af92:::benchmark_data_smithi179_77138_object110180:head# 2024-02-09T17:26:53.359 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa57f412:::benchmark_data_smithi179_77138_object43554:head# 2024-02-09T17:26:53.359 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa58200e:::benchmark_data_smithi179_77138_object52959:head# 2024-02-09T17:26:53.359 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa5866df:::benchmark_data_smithi179_77138_object81200:head# 2024-02-09T17:26:53.360 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa59e4e4:::benchmark_data_smithi179_77138_object68447:head# 2024-02-09T17:26:53.360 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa5a0fb6:::benchmark_data_smithi179_77138_object61170:head# 2024-02-09T17:26:53.361 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa5a42e5:::benchmark_data_smithi179_77138_object69389:head# 2024-02-09T17:26:53.362 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa5ada02:::benchmark_data_smithi179_77138_object114751:head# 2024-02-09T17:26:53.362 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa5b4172:::benchmark_data_smithi179_77138_object49277:head# 2024-02-09T17:26:53.362 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa5b5336:::benchmark_data_smithi179_77138_object102010:head# 2024-02-09T17:26:53.362 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa5bbcd3:::benchmark_data_smithi179_77138_object59838:head# 2024-02-09T17:26:53.363 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa5be452:::benchmark_data_smithi179_77138_object46957:head# 2024-02-09T17:26:53.363 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa5c18be:::benchmark_data_smithi179_77138_object48128:head# 2024-02-09T17:26:53.363 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa5c5084:::benchmark_data_smithi179_77138_object95458:head# 2024-02-09T17:26:53.364 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa5cd479:::benchmark_data_smithi179_77138_object54246:head# 2024-02-09T17:26:53.364 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa5d360f:::benchmark_data_smithi179_77138_object86691:head# 2024-02-09T17:26:53.365 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa5e4094:::benchmark_data_smithi179_77138_object38479:head# 2024-02-09T17:26:53.365 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa5ed617:::benchmark_data_smithi179_77138_object31924:head# 2024-02-09T17:26:53.367 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa5ee11c:::benchmark_data_smithi179_77138_object107847:head# 2024-02-09T17:26:53.367 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa5f7fbc:::benchmark_data_smithi179_77138_object67665:head# 2024-02-09T17:26:53.367 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa60155c:::benchmark_data_smithi179_77138_object42572:head# 2024-02-09T17:26:53.367 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa6034a0:::benchmark_data_smithi179_77138_object97526:head# 2024-02-09T17:26:53.368 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa619409:::benchmark_data_smithi179_77138_object50243:head# 2024-02-09T17:26:53.368 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa6257d2:::benchmark_data_smithi179_77138_object113848:head# 2024-02-09T17:26:53.369 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa628086:::benchmark_data_smithi179_77138_object122013:head# 2024-02-09T17:26:53.370 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa62b194:::benchmark_data_smithi179_77138_object56119:head# 2024-02-09T17:26:53.370 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa6377c2:::benchmark_data_smithi179_77138_object115034:head# 2024-02-09T17:26:53.370 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa63925f:::benchmark_data_smithi179_77138_object56177:head# 2024-02-09T17:26:53.371 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa647c0e:::benchmark_data_smithi179_77138_object44473:head# 2024-02-09T17:26:53.371 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa659d6d:::benchmark_data_smithi179_77138_object96082:head# 2024-02-09T17:26:53.371 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa65e530:::benchmark_data_smithi179_77138_object110328:head# 2024-02-09T17:26:53.372 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa661861:::benchmark_data_smithi179_77138_object131536:head# 2024-02-09T17:26:53.373 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa6722b9:::benchmark_data_smithi179_77138_object56350:head# 2024-02-09T17:26:53.373 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa6756fb:::benchmark_data_smithi179_77138_object77618:head# 2024-02-09T17:26:53.373 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa677175:::benchmark_data_smithi179_77138_object62880:head# 2024-02-09T17:26:53.374 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa678a08:::benchmark_data_smithi179_77138_object33171:head# 2024-02-09T17:26:53.374 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa682507:::benchmark_data_smithi179_77138_object40284:head# 2024-02-09T17:26:53.374 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa68344c:::benchmark_data_smithi179_77138_object96577:head# 2024-02-09T17:26:53.375 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa686108:::benchmark_data_smithi179_77138_object40958:head# 2024-02-09T17:26:53.375 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa68a213:::benchmark_data_smithi179_77138_object54753:head# 2024-02-09T17:26:53.375 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa69515e:::benchmark_data_smithi179_77138_object70476:head# 2024-02-09T17:26:53.376 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa6a6fb6:::benchmark_data_smithi179_77138_object80392:head# 2024-02-09T17:26:53.376 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa6a7b32:::benchmark_data_smithi179_77138_object105276:head# 2024-02-09T17:26:53.378 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa6b0858:::benchmark_data_smithi179_77138_object23653:head# 2024-02-09T17:26:53.378 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa6b66aa:::benchmark_data_smithi179_77138_object28161:head# 2024-02-09T17:26:53.378 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa6b81df:::benchmark_data_smithi179_77138_object43692:head# 2024-02-09T17:26:53.378 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa6b8bb6:::benchmark_data_smithi179_77138_object71800:head# 2024-02-09T17:26:53.379 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa6d0920:::benchmark_data_smithi179_77138_object76442:head# 2024-02-09T17:26:53.379 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa6d681d:::benchmark_data_smithi179_77138_object122073:head# 2024-02-09T17:26:53.380 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa6e7cb9:::benchmark_data_smithi179_77138_object38293:head# 2024-02-09T17:26:53.381 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa6ece2d:::benchmark_data_smithi179_77138_object54745:head# 2024-02-09T17:26:53.381 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa6f0fb3:::benchmark_data_smithi179_77138_object95999:head# 2024-02-09T17:26:53.381 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa6fa810:::benchmark_data_smithi179_77138_object72732:head# 2024-02-09T17:26:53.382 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa70808d:::benchmark_data_smithi179_77138_object55700:head# 2024-02-09T17:26:53.383 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa721473:::benchmark_data_smithi179_77138_object66521:head# 2024-02-09T17:26:53.383 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa73dfcf:::benchmark_data_smithi179_77138_object30664:head# 2024-02-09T17:26:53.383 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa7470b6:::benchmark_data_smithi179_77138_object105769:head# 2024-02-09T17:26:53.383 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa74ffb5:::benchmark_data_smithi179_77138_object113301:head# 2024-02-09T17:26:53.384 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa75fa1e:::benchmark_data_smithi179_77138_object91550:head# 2024-02-09T17:26:53.384 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa7651d0:::benchmark_data_smithi179_77138_object76329:head# 2024-02-09T17:26:53.385 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa779c8f:::benchmark_data_smithi179_77138_object66478:head# 2024-02-09T17:26:53.386 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa7837e6:::benchmark_data_smithi179_77138_object129586:head# 2024-02-09T17:26:53.386 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa7885fb:::benchmark_data_smithi179_77138_object82525:head# 2024-02-09T17:26:53.387 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa7a2afe:::benchmark_data_smithi179_77138_object126752:head# 2024-02-09T17:26:53.389 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa7a674c:::benchmark_data_smithi179_77138_object40371:head# 2024-02-09T17:26:53.389 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa7ab99a:::benchmark_data_smithi179_77138_object80936:head# 2024-02-09T17:26:53.389 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa7ad164:::benchmark_data_smithi179_77138_object55137:head# 2024-02-09T17:26:53.389 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa7bc028:::benchmark_data_smithi179_77138_object28831:head# 2024-02-09T17:26:53.389 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa7caa06:::benchmark_data_smithi179_77138_object125217:head# 2024-02-09T17:26:53.390 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa7debc1:::benchmark_data_smithi179_77138_object81505:head# 2024-02-09T17:26:53.390 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa7e119f:::benchmark_data_smithi179_77138_object121001:head# 2024-02-09T17:26:53.391 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa7e62c3:::benchmark_data_smithi179_77138_object78496:head# 2024-02-09T17:26:53.392 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa7f64c7:::benchmark_data_smithi179_77138_object66566:head# 2024-02-09T17:26:53.392 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa7fc7ec:::benchmark_data_smithi179_77138_object47660:head# 2024-02-09T17:26:53.392 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa7fd4ee:::benchmark_data_smithi179_77138_object35192:head# 2024-02-09T17:26:53.393 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa8053b4:::benchmark_data_smithi179_77138_object72606:head# 2024-02-09T17:26:53.397 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa80c9b7:::benchmark_data_smithi179_77138_object115429:head# 2024-02-09T17:26:53.397 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa80e663:::benchmark_data_smithi179_77138_object76258:head# 2024-02-09T17:26:53.397 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa826287:::benchmark_data_smithi179_77138_object59189:head# 2024-02-09T17:26:53.397 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa82c62c:::benchmark_data_smithi179_77138_object106810:head# 2024-02-09T17:26:53.397 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa82f955:::benchmark_data_smithi179_77138_object117418:head# 2024-02-09T17:26:53.397 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa84cc2c:::benchmark_data_smithi179_77138_object93175:head# 2024-02-09T17:26:53.397 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa85f3c1:::benchmark_data_smithi179_77138_object110531:head# 2024-02-09T17:26:53.398 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa88252f:::benchmark_data_smithi179_77138_object119458:head# 2024-02-09T17:26:53.398 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa8847ec:::benchmark_data_smithi179_77138_object32986:head# 2024-02-09T17:26:53.398 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa886130:::benchmark_data_smithi179_77138_object63048:head# 2024-02-09T17:26:53.398 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa896cc3:::benchmark_data_smithi179_77138_object62220:head# 2024-02-09T17:26:53.398 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa897da3:::benchmark_data_smithi179_77138_object74974:head# 2024-02-09T17:26:53.399 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa89a76e:::benchmark_data_smithi179_77138_object67531:head# 2024-02-09T17:26:53.399 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa8a4406:::benchmark_data_smithi179_77138_object128880:head# 2024-02-09T17:26:53.399 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa8ac88d:::benchmark_data_smithi179_77138_object43002:head# 2024-02-09T17:26:53.399 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa8b34b6:::benchmark_data_smithi179_77138_object26616:head# 2024-02-09T17:26:53.399 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa8b3944:::benchmark_data_smithi179_77138_object100407:head# 2024-02-09T17:26:53.400 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa8b3cc8:::benchmark_data_smithi179_77138_object37083:head# 2024-02-09T17:26:53.400 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa8d3c4a:::benchmark_data_smithi179_77138_object93289:head# 2024-02-09T17:26:53.400 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa8e151e:::benchmark_data_smithi179_77138_object58173:head# 2024-02-09T17:26:53.400 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa8f001e:::benchmark_data_smithi179_77138_object50374:head# 2024-02-09T17:26:53.401 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa8f2f09:::benchmark_data_smithi179_77138_object66858:head# 2024-02-09T17:26:53.402 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa8f2fa1:::benchmark_data_smithi179_77138_object108349:head# 2024-02-09T17:26:53.403 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa8fb812:::benchmark_data_smithi179_77138_object73802:head# 2024-02-09T17:26:53.403 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa9036dc:::benchmark_data_smithi179_77138_object80454:head# 2024-02-09T17:26:53.403 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa90ee46:::benchmark_data_smithi179_77138_object87473:head# 2024-02-09T17:26:53.404 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa90f914:::benchmark_data_smithi179_77138_object55777:head# 2024-02-09T17:26:53.404 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa9107c4:::benchmark_data_smithi179_77138_object36254:head# 2024-02-09T17:26:53.405 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa913187:::benchmark_data_smithi179_77138_object42341:head# 2024-02-09T17:26:53.405 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa917326:::benchmark_data_smithi179_77138_object106189:head# 2024-02-09T17:26:53.405 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa920039:::benchmark_data_smithi179_77138_object75212:head# 2024-02-09T17:26:53.406 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa922694:::benchmark_data_smithi179_77138_object52306:head# 2024-02-09T17:26:53.406 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa927d54:::benchmark_data_smithi179_77138_object46294:head# 2024-02-09T17:26:53.407 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa92a1ed:::benchmark_data_smithi179_77138_object104751:head# 2024-02-09T17:26:53.408 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa930330:::benchmark_data_smithi179_77138_object72213:head# 2024-02-09T17:26:53.408 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa93c3b7:::benchmark_data_smithi179_77138_object71779:head# 2024-02-09T17:26:53.409 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa93ece1:::benchmark_data_smithi179_77138_object66394:head# 2024-02-09T17:26:53.409 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa93eef9:::benchmark_data_smithi179_77138_object38752:head# 2024-02-09T17:26:53.409 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa940bfd:::benchmark_data_smithi179_77138_object85143:head# 2024-02-09T17:26:53.409 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa9429d5:::benchmark_data_smithi179_77138_object47099:head# 2024-02-09T17:26:53.409 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa94a9d2:::benchmark_data_smithi179_77138_object32201:head# 2024-02-09T17:26:53.410 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa952cd2:::benchmark_data_smithi179_77138_object41056:head# 2024-02-09T17:26:53.410 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa954315:::benchmark_data_smithi179_77138_object126499:head# 2024-02-09T17:26:53.410 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa959c72:::benchmark_data_smithi179_77138_object31639:head# 2024-02-09T17:26:53.411 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa95d6c6:::benchmark_data_smithi179_77138_object122492:head# 2024-02-09T17:26:53.411 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa9601d4:::benchmark_data_smithi179_77138_object129778:head# 2024-02-09T17:26:53.412 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa970253:::benchmark_data_smithi179_77138_object64513:head# 2024-02-09T17:26:53.412 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa97344a:::benchmark_data_smithi179_77138_object119584:head# 2024-02-09T17:26:53.413 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa976fe2:::benchmark_data_smithi179_77138_object21673:head# 2024-02-09T17:26:53.413 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa983f00:::benchmark_data_smithi179_77138_object25076:head# 2024-02-09T17:26:53.413 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:53.414 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa985f8a:::benchmark_data_smithi179_77138_object55952:head# 2024-02-09T17:26:53.414 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa988414:::benchmark_data_smithi179_77138_object28088:head# 2024-02-09T17:26:53.414 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa992651:::benchmark_data_smithi179_77138_object129106:head# 2024-02-09T17:26:53.415 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa9948b4:::benchmark_data_smithi179_77138_object113085:head# 2024-02-09T17:26:53.415 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa99961f:::benchmark_data_smithi179_77138_object105999:head# 2024-02-09T17:26:53.416 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa99f3fb:::benchmark_data_smithi179_77138_object98324:head# 2024-02-09T17:26:53.416 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa9a5461:::benchmark_data_smithi179_77138_object45247:head# 2024-02-09T17:26:53.421 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa9a5819:::benchmark_data_smithi179_77138_object63156:head# 2024-02-09T17:26:53.421 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa9a7a7d:::benchmark_data_smithi179_77138_object59936:head# 2024-02-09T17:26:53.421 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa9a7c51:::benchmark_data_smithi179_77138_object101303:head# 2024-02-09T17:26:53.421 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa9a9c61:::benchmark_data_smithi179_77138_object37732:head# 2024-02-09T17:26:53.421 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa9b5e21:::benchmark_data_smithi179_77138_object75458:head# 2024-02-09T17:26:53.421 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa9b9bd8:::benchmark_data_smithi179_77138_object53962:head# 2024-02-09T17:26:53.422 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa9cc8c3:::benchmark_data_smithi179_77138_object47744:head# 2024-02-09T17:26:53.422 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa9d2a0c:::benchmark_data_smithi179_77138_object34291:head# 2024-02-09T17:26:53.422 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa9d4721:::benchmark_data_smithi179_77138_object49423:head# 2024-02-09T17:26:53.422 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa9da503:::benchmark_data_smithi179_77138_object90574:head# 2024-02-09T17:26:53.422 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa9dd32e:::benchmark_data_smithi179_77138_object99111:head# 2024-02-09T17:26:53.422 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa9dfdd4:::benchmark_data_smithi179_77138_object33097:head# 2024-02-09T17:26:53.423 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa9f8ea3:::benchmark_data_smithi179_77138_object74625:head# 2024-02-09T17:26:53.423 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fa9ffd05:::benchmark_data_smithi179_77138_object51174:head# 2024-02-09T17:26:53.423 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faa03729:::benchmark_data_smithi179_77138_object55536:head# 2024-02-09T17:26:53.423 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faa04765:::benchmark_data_smithi179_77138_object63701:head# 2024-02-09T17:26:53.423 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faa06c2a:::benchmark_data_smithi179_77138_object114479:head# 2024-02-09T17:26:53.424 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faa08076:::benchmark_data_smithi179_77138_object72996:head# 2024-02-09T17:26:53.424 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faa0dee9:::benchmark_data_smithi179_77138_object47773:head# 2024-02-09T17:26:53.425 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faa11512:::benchmark_data_smithi179_77138_object130868:head# 2024-02-09T17:26:53.425 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faa219a7:::benchmark_data_smithi179_77138_object72092:head# 2024-02-09T17:26:53.425 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faa35d0b:::benchmark_data_smithi179_77138_object80181:head# 2024-02-09T17:26:53.426 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faa3bd0c:::benchmark_data_smithi179_77138_object86377:head# 2024-02-09T17:26:53.426 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faa4c27d:::benchmark_data_smithi179_77138_object82600:head# 2024-02-09T17:26:53.427 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faa6f0fa:::benchmark_data_smithi179_77138_object67732:head# 2024-02-09T17:26:53.427 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faa70448:::benchmark_data_smithi179_77138_object35903:head# 2024-02-09T17:26:53.427 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faa7d027:::benchmark_data_smithi179_77138_object70520:head# 2024-02-09T17:26:53.428 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faa87f1e:::benchmark_data_smithi179_77138_object91390:head# 2024-02-09T17:26:53.429 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faa8daab:::benchmark_data_smithi179_77138_object126436:head# 2024-02-09T17:26:53.429 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faa908ed:::benchmark_data_smithi179_77138_object129413:head# 2024-02-09T17:26:53.429 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faa96007:::benchmark_data_smithi179_77138_object96046:head# 2024-02-09T17:26:53.429 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faa9768d:::benchmark_data_smithi179_77138_object56846:head# 2024-02-09T17:26:53.430 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faa97ea9:::benchmark_data_smithi179_77138_object62875:head# 2024-02-09T17:26:53.430 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faaa3f08:::benchmark_data_smithi179_77138_object26794:head# 2024-02-09T17:26:53.431 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faaa8e04:::benchmark_data_smithi179_77138_object127404:head# 2024-02-09T17:26:53.431 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faaaf884:::benchmark_data_smithi179_77138_object51439:head# 2024-02-09T17:26:53.432 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faab3bcf:::benchmark_data_smithi179_77138_object62190:head# 2024-02-09T17:26:53.432 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faab6141:::benchmark_data_smithi179_77138_object90570:head# 2024-02-09T17:26:53.432 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faadc507:::benchmark_data_smithi179_77138_object99291:head# 2024-02-09T17:26:53.433 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faae0adc:::benchmark_data_smithi179_77138_object66077:head# 2024-02-09T17:26:53.433 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faaee3d2:::benchmark_data_smithi179_77138_object127423:head# 2024-02-09T17:26:53.434 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faaf12d7:::benchmark_data_smithi179_77138_object77390:head# 2024-02-09T17:26:53.434 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faaf311a:::benchmark_data_smithi179_77138_object44671:head# 2024-02-09T17:26:53.435 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fab17ad2:::benchmark_data_smithi179_77138_object129341:head# 2024-02-09T17:26:53.435 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fab1d16c:::benchmark_data_smithi179_77138_object105433:head# 2024-02-09T17:26:53.436 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fab233eb:::benchmark_data_smithi179_77138_object45343:head# 2024-02-09T17:26:53.439 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fab273e7:::benchmark_data_smithi179_77138_object99184:head# 2024-02-09T17:26:53.440 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fab2c969:::benchmark_data_smithi179_77138_object20686:head# 2024-02-09T17:26:53.440 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fab35723:::benchmark_data_smithi179_77138_object26679:head# 2024-02-09T17:26:53.441 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fab40c6e:::benchmark_data_smithi179_77138_object38571:head# 2024-02-09T17:26:53.442 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fab43c51:::benchmark_data_smithi179_77138_object55461:head# 2024-02-09T17:26:53.442 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fab5233f:::benchmark_data_smithi179_77138_object48805:head# 2024-02-09T17:26:53.442 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fab60703:::benchmark_data_smithi179_77138_object111162:head# 2024-02-09T17:26:53.442 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fab6fda4:::benchmark_data_smithi179_77138_object43682:head# 2024-02-09T17:26:53.444 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fab6ff21:::benchmark_data_smithi179_77138_object41581:head# 2024-02-09T17:26:53.444 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fab7b819:::benchmark_data_smithi179_77138_object128476:head# 2024-02-09T17:26:53.444 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fab7c6d3:::benchmark_data_smithi179_77138_object19632:head# 2024-02-09T17:26:53.445 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fab80947:::benchmark_data_smithi179_77138_object45776:head# 2024-02-09T17:26:53.445 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fab8dad7:::benchmark_data_smithi179_77138_object97299:head# 2024-02-09T17:26:53.445 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fab90ec2:::benchmark_data_smithi179_77138_object26893:head# 2024-02-09T17:26:53.445 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fab97ecb:::benchmark_data_smithi179_77138_object42068:head# 2024-02-09T17:26:53.446 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faba0a1b:::benchmark_data_smithi179_77138_object54834:head# 2024-02-09T17:26:53.446 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fabac3ec:::benchmark_data_smithi179_77138_object89875:head# 2024-02-09T17:26:53.446 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fabae10b:::benchmark_data_smithi179_77138_object20260:head# 2024-02-09T17:26:53.447 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fabb0be5:::benchmark_data_smithi179_77138_object131032:head# 2024-02-09T17:26:53.447 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fabb21ab:::benchmark_data_smithi179_77138_object20859:head# 2024-02-09T17:26:53.448 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fabb76e6:::benchmark_data_smithi179_77138_object26582:head# 2024-02-09T17:26:53.448 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fabb87db:::benchmark_data_smithi179_77138_object72042:head# 2024-02-09T17:26:53.449 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fabbc1d3:::benchmark_data_smithi179_77138_object73589:head# 2024-02-09T17:26:53.449 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fabc1f5c:::benchmark_data_smithi179_77138_object131774:head# 2024-02-09T17:26:53.449 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fabc71f3:::benchmark_data_smithi179_77138_object62852:head# 2024-02-09T17:26:53.450 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fabd7b3b:::benchmark_data_smithi179_77138_object130230:head# 2024-02-09T17:26:53.450 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fabd8fdf:::benchmark_data_smithi179_77138_object48769:head# 2024-02-09T17:26:53.451 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fabdd7ad:::benchmark_data_smithi179_77138_object127931:head# 2024-02-09T17:26:53.451 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fabde182:::benchmark_data_smithi179_77138_object64243:head# 2024-02-09T17:26:53.456 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fabe0e60:::benchmark_data_smithi179_77138_object99937:head# 2024-02-09T17:26:53.456 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fabe152e:::benchmark_data_smithi179_77138_object43902:head# 2024-02-09T17:26:53.457 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fabe254c:::benchmark_data_smithi179_77138_object43704:head# 2024-02-09T17:26:53.457 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fabfa896:::benchmark_data_smithi179_77138_object20036:head# 2024-02-09T17:26:53.457 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fac038a5:::benchmark_data_smithi179_77138_object110379:head# 2024-02-09T17:26:53.457 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fac060ae:::benchmark_data_smithi179_77138_object56215:head# 2024-02-09T17:26:53.458 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fac0e831:::benchmark_data_smithi179_77138_object28551:head# 2024-02-09T17:26:53.458 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fac132f2:::benchmark_data_smithi179_77138_object125491:head# 2024-02-09T17:26:53.458 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fac1f901:::benchmark_data_smithi179_77138_object127911:head# 2024-02-09T17:26:53.458 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fac1ff9e:::benchmark_data_smithi179_77138_object99467:head# 2024-02-09T17:26:53.458 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fac2211c:::benchmark_data_smithi179_77138_object108633:head# 2024-02-09T17:26:53.458 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fac231a5:::benchmark_data_smithi179_77138_object105115:head# 2024-02-09T17:26:53.459 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fac26fee:::benchmark_data_smithi179_77138_object120211:head# 2024-02-09T17:26:53.459 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fac3b2ae:::benchmark_data_smithi179_77138_object56094:head# 2024-02-09T17:26:53.460 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fac513ba:::benchmark_data_smithi179_77138_object24173:head# 2024-02-09T17:26:53.460 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fac54cd1:::benchmark_data_smithi179_77138_object78818:head# 2024-02-09T17:26:53.461 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fac5c591:::benchmark_data_smithi179_77138_object62321:head# 2024-02-09T17:26:53.461 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fac5d2e1:::benchmark_data_smithi179_77138_object73838:head# 2024-02-09T17:26:53.461 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fac5ed23:::benchmark_data_smithi179_77138_object77772:head# 2024-02-09T17:26:53.462 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fac5f33a:::benchmark_data_smithi179_77138_object24775:head# 2024-02-09T17:26:53.462 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fac5f5a3:::benchmark_data_smithi179_77138_object125366:head# 2024-02-09T17:26:53.462 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fac6116c:::benchmark_data_smithi179_77138_object63235:head# 2024-02-09T17:26:53.463 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fac65d29:::benchmark_data_smithi179_77138_object19656:head# 2024-02-09T17:26:53.463 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fac70f79:::benchmark_data_smithi179_77138_object29868:head# 2024-02-09T17:26:53.464 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fac74b7d:::benchmark_data_smithi179_77138_object22996:head# 2024-02-09T17:26:53.464 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fac94431:::benchmark_data_smithi179_77138_object89028:head# 2024-02-09T17:26:53.465 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:facabd5f:::benchmark_data_smithi179_77138_object47644:head# 2024-02-09T17:26:53.465 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:facbbd03:::benchmark_data_smithi179_77138_object128150:head# 2024-02-09T17:26:53.465 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:face187c:::benchmark_data_smithi179_77138_object76125:head# 2024-02-09T17:26:53.466 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:face447a:::benchmark_data_smithi179_77138_object36372:head# 2024-02-09T17:26:53.466 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:facfaa65:::benchmark_data_smithi179_77138_object60446:head# 2024-02-09T17:26:53.466 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fad07c9b:::benchmark_data_smithi179_77138_object66861:head# 2024-02-09T17:26:53.467 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fad090fb:::benchmark_data_smithi179_77138_object95609:head# 2024-02-09T17:26:53.468 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fad0a551:::benchmark_data_smithi179_77138_object24366:head# 2024-02-09T17:26:53.468 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fad135f7:::benchmark_data_smithi179_77138_object32293:head# 2024-02-09T17:26:53.474 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fad2d302:::benchmark_data_smithi179_77138_object21759:head# 2024-02-09T17:26:53.474 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fad2d54b:::benchmark_data_smithi179_77138_object36250:head# 2024-02-09T17:26:53.474 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fad42f49:::benchmark_data_smithi179_77138_object123118:head# 2024-02-09T17:26:53.474 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fad45db3:::benchmark_data_smithi179_77138_object73243:head# 2024-02-09T17:26:53.474 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fad5c5e2:::benchmark_data_smithi179_77138_object34434:head# 2024-02-09T17:26:53.474 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fad6c3fb:::benchmark_data_smithi179_77138_object81048:head# 2024-02-09T17:26:53.474 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fad705d6:::benchmark_data_smithi179_77138_object103995:head# 2024-02-09T17:26:53.474 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fad71bbd:::benchmark_data_smithi179_77138_object39484:head# 2024-02-09T17:26:53.474 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fad8c76b:::benchmark_data_smithi179_77138_object107103:head# 2024-02-09T17:26:53.474 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fad8d6f7:::benchmark_data_smithi179_77138_object80251:head# 2024-02-09T17:26:53.474 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fad91fa5:::benchmark_data_smithi179_77138_object56297:head# 2024-02-09T17:26:53.474 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fad9a46c:::benchmark_data_smithi179_77138_object117524:head# 2024-02-09T17:26:53.474 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fada3b58:::benchmark_data_smithi179_77138_object94663:head# 2024-02-09T17:26:53.476 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fadb0ca0:::benchmark_data_smithi179_77138_object23752:head# 2024-02-09T17:26:53.476 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fadb1018:::benchmark_data_smithi179_77138_object52191:head# 2024-02-09T17:26:53.476 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fadb4187:::benchmark_data_smithi179_77138_object118835:head# 2024-02-09T17:26:53.476 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fadbe255:::benchmark_data_smithi179_77138_object39239:head# 2024-02-09T17:26:53.476 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fadc7f67:::benchmark_data_smithi179_77138_object121426:head# 2024-02-09T17:26:53.480 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fadd43ee:::benchmark_data_smithi179_77138_object33522:head# 2024-02-09T17:26:53.480 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fadd713d:::benchmark_data_smithi179_77138_object113537:head# 2024-02-09T17:26:53.481 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faddfb20:::benchmark_data_smithi179_77138_object22161:head# 2024-02-09T17:26:53.482 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fade1224:::benchmark_data_smithi179_77138_object20823:head# 2024-02-09T17:26:53.483 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fade77f0:::benchmark_data_smithi179_77138_object107313:head# 2024-02-09T17:26:53.484 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fadfdcb1:::benchmark_data_smithi179_77138_object120939:head# 2024-02-09T17:26:53.484 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fae07bf7:::benchmark_data_smithi179_77138_object87701:head# 2024-02-09T17:26:53.485 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fae0fac3:::benchmark_data_smithi179_77138_object29677:head# 2024-02-09T17:26:53.485 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fae15342:::benchmark_data_smithi179_77138_object19362:head# 2024-02-09T17:26:53.486 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fae1ee5f:::benchmark_data_smithi179_77138_object39430:head# 2024-02-09T17:26:53.490 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fae23053:::benchmark_data_smithi179_77138_object22799:head# 2024-02-09T17:26:53.490 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fae27c94:::benchmark_data_smithi179_77138_object56610:head# 2024-02-09T17:26:53.490 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fae34b2f:::benchmark_data_smithi179_77138_object128272:head# 2024-02-09T17:26:53.490 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fae3773f:::benchmark_data_smithi179_77138_object43628:head# 2024-02-09T17:26:53.490 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fae4460f:::benchmark_data_smithi179_77138_object89546:head# 2024-02-09T17:26:53.491 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fae4f29b:::benchmark_data_smithi179_77138_object42694:head# 2024-02-09T17:26:53.491 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fae62470:::benchmark_data_smithi179_77138_object79565:head# 2024-02-09T17:26:53.491 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fae6b75e:::benchmark_data_smithi179_77138_object112061:head# 2024-02-09T17:26:53.491 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fae783d5:::benchmark_data_smithi179_77138_object42869:head# 2024-02-09T17:26:53.491 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fae7ca7b:::benchmark_data_smithi179_77138_object101755:head# 2024-02-09T17:26:53.491 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fae81bab:::benchmark_data_smithi179_77138_object103237:head# 2024-02-09T17:26:53.491 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fae9490d:::benchmark_data_smithi179_77138_object93530:head# 2024-02-09T17:26:53.492 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fae96f83:::benchmark_data_smithi179_77138_object113503:head# 2024-02-09T17:26:53.492 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faeab799:::benchmark_data_smithi179_77138_object74479:head# 2024-02-09T17:26:53.492 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faeac6d3:::benchmark_data_smithi179_77138_object127315:head# 2024-02-09T17:26:53.492 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faed4933:::benchmark_data_smithi179_77138_object131000:head# 2024-02-09T17:26:53.493 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faed941c:::benchmark_data_smithi179_77138_object40587:head# 2024-02-09T17:26:53.493 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faedcfb8:::benchmark_data_smithi179_77138_object27933:head# 2024-02-09T17:26:53.493 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faee2f8f:::benchmark_data_smithi179_77138_object28209:head# 2024-02-09T17:26:53.494 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faee5c39:::benchmark_data_smithi179_77138_object85377:head# 2024-02-09T17:26:53.494 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faee670b:::benchmark_data_smithi179_77138_object90447:head# 2024-02-09T17:26:53.495 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faef3e21:::benchmark_data_smithi179_77138_object107743:head# 2024-02-09T17:26:53.495 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faefa220:::benchmark_data_smithi179_77138_object32424:head# 2024-02-09T17:26:53.495 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faefdb35:::benchmark_data_smithi179_77138_object70086:head# 2024-02-09T17:26:53.496 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faf142a5:::benchmark_data_smithi179_77138_object20227:head# 2024-02-09T17:26:53.496 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faf35ed7:::benchmark_data_smithi179_77138_object49028:head# 2024-02-09T17:26:53.498 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faf38098:::benchmark_data_smithi179_77138_object85084:head# 2024-02-09T17:26:53.499 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faf41df5:::benchmark_data_smithi179_77138_object65459:head# 2024-02-09T17:26:53.499 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faf5aa3f:::benchmark_data_smithi179_77138_object94973:head# 2024-02-09T17:26:53.500 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faf6c8a6:::benchmark_data_smithi179_77138_object51358:head# 2024-02-09T17:26:53.500 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faf7d635:::benchmark_data_smithi179_77138_object49011:head# 2024-02-09T17:26:53.501 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faf951fa:::benchmark_data_smithi179_77138_object103353:head# 2024-02-09T17:26:53.501 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faf96fd9:::benchmark_data_smithi179_77138_object89162:head# 2024-02-09T17:26:53.501 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fafa30c5:::benchmark_data_smithi179_77138_object90742:head# 2024-02-09T17:26:53.502 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fafacfd7:::benchmark_data_smithi179_77138_object126859:head# 2024-02-09T17:26:53.503 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fafbedf2:::benchmark_data_smithi179_77138_object102557:head# 2024-02-09T17:26:53.503 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fafbf844:::benchmark_data_smithi179_77138_object31628:head# 2024-02-09T17:26:53.503 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fafc246e:::benchmark_data_smithi179_77138_object44947:head# 2024-02-09T17:26:53.504 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fafe9261:::benchmark_data_smithi179_77138_object116148:head# 2024-02-09T17:26:53.504 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fafea24d:::benchmark_data_smithi179_77138_object21308:head# 2024-02-09T17:26:53.510 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:faff23d7:::benchmark_data_smithi179_77138_object28660:head# 2024-02-09T17:26:53.510 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb0053fb:::benchmark_data_smithi179_77138_object73607:head# 2024-02-09T17:26:53.510 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb00b094:::benchmark_data_smithi179_77138_object95961:head# 2024-02-09T17:26:53.510 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb00d07f:::benchmark_data_smithi179_77138_object125461:head# 2024-02-09T17:26:53.510 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb00e55f:::benchmark_data_smithi179_77138_object61761:head# 2024-02-09T17:26:53.511 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb00e72d:::benchmark_data_smithi179_77138_object25030:head# 2024-02-09T17:26:53.511 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb013bb2:::benchmark_data_smithi179_77138_object65571:head# 2024-02-09T17:26:53.511 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb017c31:::benchmark_data_smithi179_77138_object21787:head# 2024-02-09T17:26:53.511 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb01ed45:::benchmark_data_smithi179_77138_object73319:head# 2024-02-09T17:26:53.513 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb0214cc:::benchmark_data_smithi179_77138_object129788:head# 2024-02-09T17:26:53.514 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb02291e:::benchmark_data_smithi179_77138_object87447:head# 2024-02-09T17:26:53.514 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb022ddd:::benchmark_data_smithi179_77138_object27774:head# 2024-02-09T17:26:53.515 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb02ae7a:::benchmark_data_smithi179_77138_object101724:head# 2024-02-09T17:26:53.515 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb0337b4:::benchmark_data_smithi179_77138_object29799:head# 2024-02-09T17:26:53.515 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb033c97:::benchmark_data_smithi179_77138_object119283:head# 2024-02-09T17:26:53.516 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb058ec7:::benchmark_data_smithi179_77138_object111444:head# 2024-02-09T17:26:53.516 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb05d5f7:::benchmark_data_smithi179_77138_object59633:head# 2024-02-09T17:26:53.518 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb0766da:::benchmark_data_smithi179_77138_object120248:head# 2024-02-09T17:26:53.518 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb07970d:::benchmark_data_smithi179_77138_object44384:head# 2024-02-09T17:26:53.518 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb07d9e1:::benchmark_data_smithi179_77138_object31094:head# 2024-02-09T17:26:53.519 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb08aab0:::benchmark_data_smithi179_77138_object71390:head# 2024-02-09T17:26:53.520 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb08ff2f:::benchmark_data_smithi179_77138_object56891:head# 2024-02-09T17:26:53.522 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:26:53.526 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb09d67f:::benchmark_data_smithi179_77138_object42352:head# 2024-02-09T17:26:53.526 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb0a26e8:::benchmark_data_smithi179_77138_object117387:head# 2024-02-09T17:26:53.526 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb0b2263:::benchmark_data_smithi179_77138_object104604:head# 2024-02-09T17:26:53.527 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb0c8954:::benchmark_data_smithi179_77138_object91318:head# 2024-02-09T17:26:53.527 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb0d7e8f:::benchmark_data_smithi179_77138_object124554:head# 2024-02-09T17:26:53.528 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb0e0fb6:::benchmark_data_smithi179_77138_object21869:head# 2024-02-09T17:26:53.528 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb0e9564:::benchmark_data_smithi179_77138_object25767:head# 2024-02-09T17:26:53.529 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb0ed28f:::benchmark_data_smithi179_77138_object68399:head# 2024-02-09T17:26:53.530 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb10f4a3:::benchmark_data_smithi179_77138_object82640:head# 2024-02-09T17:26:53.530 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb1325a5:::benchmark_data_smithi179_77138_object86976:head# 2024-02-09T17:26:53.531 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb13777c:::benchmark_data_smithi179_77138_object127513:head# 2024-02-09T17:26:53.531 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb13c7c3:::benchmark_data_smithi179_77138_object49323:head# 2024-02-09T17:26:53.531 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb145ef3:::benchmark_data_smithi179_77138_object64895:head# 2024-02-09T17:26:53.532 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb1487ee:::benchmark_data_smithi179_77138_object46688:head# 2024-02-09T17:26:53.532 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb14eec0:::benchmark_data_smithi179_77138_object90385:head# 2024-02-09T17:26:53.533 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb14f56f:::benchmark_data_smithi179_77138_object34436:head# 2024-02-09T17:26:53.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:53.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:26:53.534 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb15835a:::benchmark_data_smithi179_77138_object102566:head# 2024-02-09T17:26:53.535 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb17091c:::benchmark_data_smithi179_77138_object112034:head# 2024-02-09T17:26:53.535 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb1716c1:::benchmark_data_smithi179_77138_object48500:head# 2024-02-09T17:26:53.535 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb18f869:::benchmark_data_smithi179_77138_object40988:head# 2024-02-09T17:26:53.535 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb19d400:::benchmark_data_smithi179_77138_object108057:head# 2024-02-09T17:26:53.535 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb19ec89:::benchmark_data_smithi179_77138_object105803:head# 2024-02-09T17:26:53.536 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb1aebd1:::benchmark_data_smithi179_77138_object77145:head# 2024-02-09T17:26:53.536 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb1af90b:::benchmark_data_smithi179_77138_object48567:head# 2024-02-09T17:26:53.537 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb1b8b84:::benchmark_data_smithi179_77138_object109906:head# 2024-02-09T17:26:53.537 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb1bb00e:::benchmark_data_smithi179_77138_object65329:head# 2024-02-09T17:26:53.539 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb1bb139:::benchmark_data_smithi179_77138_object56427:head# 2024-02-09T17:26:53.539 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb1c90e7:::benchmark_data_smithi179_77138_object89258:head# 2024-02-09T17:26:53.540 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb1ceeea:::benchmark_data_smithi179_77138_object73305:head# 2024-02-09T17:26:53.540 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb2009eb:::benchmark_data_smithi179_77138_object119745:head# 2024-02-09T17:26:53.541 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb20ded1:::benchmark_data_smithi179_77138_object73137:head# 2024-02-09T17:26:53.541 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb210d52:::benchmark_data_smithi179_77138_object125137:head# 2024-02-09T17:26:53.542 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb25109a:::benchmark_data_smithi179_77138_object116742:head# 2024-02-09T17:26:53.542 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb25cb09:::benchmark_data_smithi179_77138_object121911:head# 2024-02-09T17:26:53.542 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb260ddc:::benchmark_data_smithi179_77138_object49582:head# 2024-02-09T17:26:53.543 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb2679ba:::benchmark_data_smithi179_77138_object74508:head# 2024-02-09T17:26:53.543 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb26c75a:::benchmark_data_smithi179_77138_object107107:head# 2024-02-09T17:26:53.544 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb26d886:::benchmark_data_smithi179_77138_object40379:head# 2024-02-09T17:26:53.544 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb27278f:::benchmark_data_smithi179_77138_object48196:head# 2024-02-09T17:26:53.544 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb27e1ed:::benchmark_data_smithi179_77138_object113834:head# 2024-02-09T17:26:53.545 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb293e01:::benchmark_data_smithi179_77138_object23148:head# 2024-02-09T17:26:53.545 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb295e29:::benchmark_data_smithi179_77138_object114096:head# 2024-02-09T17:26:53.546 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb2a33e0:::benchmark_data_smithi179_77138_object34184:head# 2024-02-09T17:26:53.546 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb2a680b:::benchmark_data_smithi179_77138_object81413:head# 2024-02-09T17:26:53.547 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb2c9093:::benchmark_data_smithi179_77138_object75312:head# 2024-02-09T17:26:53.547 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb2cd384:::benchmark_data_smithi179_77138_object43224:head# 2024-02-09T17:26:53.547 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb2d0299:::benchmark_data_smithi179_77138_object85313:head# 2024-02-09T17:26:53.548 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb2d6bcd:::benchmark_data_smithi179_77138_object54765:head# 2024-02-09T17:26:53.548 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb2f1a08:::benchmark_data_smithi179_77138_object85955:head# 2024-02-09T17:26:53.549 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb2f2dd0:::benchmark_data_smithi179_77138_object111801:head# 2024-02-09T17:26:53.549 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb2f4285:::benchmark_data_smithi179_77138_object65017:head# 2024-02-09T17:26:53.550 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb2f4a8b:::benchmark_data_smithi179_77138_object77677:head# 2024-02-09T17:26:53.550 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb2fc7b9:::benchmark_data_smithi179_77138_object34781:head# 2024-02-09T17:26:53.551 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb303462:::benchmark_data_smithi179_77138_object78268:head# 2024-02-09T17:26:53.551 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb312c64:::benchmark_data_smithi179_77138_object83679:head# 2024-02-09T17:26:53.552 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb3134a4:::benchmark_data_smithi179_77138_object115968:head# 2024-02-09T17:26:53.552 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb3149ed:::benchmark_data_smithi179_77138_object26489:head# 2024-02-09T17:26:53.553 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb32369a:::benchmark_data_smithi179_77138_object92764:head# 2024-02-09T17:26:53.553 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb32544f:::benchmark_data_smithi179_77138_object116667:head# 2024-02-09T17:26:53.553 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb3310b4:::benchmark_data_smithi179_77138_object64267:head# 2024-02-09T17:26:53.554 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb33d3f1:::benchmark_data_smithi179_77138_object78350:head# 2024-02-09T17:26:53.554 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb3433b5:::benchmark_data_smithi179_77138_object76291:head# 2024-02-09T17:26:53.554 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb3473ac:::benchmark_data_smithi179_77138_object114544:head# 2024-02-09T17:26:53.555 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb355c56:::benchmark_data_smithi179_77138_object88068:head# 2024-02-09T17:26:53.555 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb35d8a5:::benchmark_data_smithi179_77138_object47064:head# 2024-02-09T17:26:53.557 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb369401:::benchmark_data_smithi179_77138_object33382:head# 2024-02-09T17:26:53.557 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb37df57:::benchmark_data_smithi179_77138_object128185:head# 2024-02-09T17:26:53.557 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb37e4ca:::benchmark_data_smithi179_77138_object52302:head# 2024-02-09T17:26:53.558 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb38486d:::benchmark_data_smithi179_77138_object38088:head# 2024-02-09T17:26:53.558 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb38d18a:::benchmark_data_smithi179_77138_object60670:head# 2024-02-09T17:26:53.561 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb393305:::benchmark_data_smithi179_77138_object98147:head# 2024-02-09T17:26:53.561 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb39f4fb:::benchmark_data_smithi179_77138_object78742:head# 2024-02-09T17:26:53.561 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb3a75e5:::benchmark_data_smithi179_77138_object23942:head# 2024-02-09T17:26:53.562 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb3ae83c:::benchmark_data_smithi179_77138_object34797:head# 2024-02-09T17:26:53.562 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb3af25f:::benchmark_data_smithi179_77138_object28624:head# 2024-02-09T17:26:53.563 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb3b9b7c:::benchmark_data_smithi179_77138_object99099:head# 2024-02-09T17:26:53.564 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb3bdc74:::benchmark_data_smithi179_77138_object114916:head# 2024-02-09T17:26:53.564 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb3c3887:::benchmark_data_smithi179_77138_object80862:head# 2024-02-09T17:26:53.565 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb3c54c1:::benchmark_data_smithi179_77138_object55755:head# 2024-02-09T17:26:53.565 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb3c5bd2:::benchmark_data_smithi179_77138_object117781:head# 2024-02-09T17:26:53.565 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb3c9a69:::benchmark_data_smithi179_77138_object131238:head# 2024-02-09T17:26:53.565 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb3ce721:::benchmark_data_smithi179_77138_object56071:head# 2024-02-09T17:26:53.568 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb3d7436:::benchmark_data_smithi179_77138_object114378:head# 2024-02-09T17:26:53.568 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb3eee84:::benchmark_data_smithi179_77138_object126866:head# 2024-02-09T17:26:53.568 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb3fc24c:::benchmark_data_smithi179_77138_object99201:head# 2024-02-09T17:26:53.568 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb40fa2e:::benchmark_data_smithi179_77138_object79682:head# 2024-02-09T17:26:53.568 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb411a9c:::benchmark_data_smithi179_77138_object63596:head# 2024-02-09T17:26:53.570 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb4149e0:::benchmark_data_smithi179_77138_object69081:head# 2024-02-09T17:26:53.570 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb416bfc:::benchmark_data_smithi179_77138_object114456:head# 2024-02-09T17:26:53.570 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb4239a7:::benchmark_data_smithi179_77138_object70028:head# 2024-02-09T17:26:53.570 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb427d1e:::benchmark_data_smithi179_77138_object127014:head# 2024-02-09T17:26:53.570 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb429c64:::benchmark_data_smithi179_77138_object120413:head# 2024-02-09T17:26:53.570 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb435781:::benchmark_data_smithi179_77138_object67559:head# 2024-02-09T17:26:53.575 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb44fdb0:::benchmark_data_smithi179_77138_object90376:head# 2024-02-09T17:26:53.576 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb452ebd:::benchmark_data_smithi179_77138_object35410:head# 2024-02-09T17:26:53.576 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb452f90:::benchmark_data_smithi179_77138_object100886:head# 2024-02-09T17:26:53.576 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb45f21c:::benchmark_data_smithi179_77138_object19921:head# 2024-02-09T17:26:53.576 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb4645a6:::benchmark_data_smithi179_77138_object96469:head# 2024-02-09T17:26:53.576 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb46b338:::benchmark_data_smithi179_77138_object21927:head# 2024-02-09T17:26:53.576 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb46bfd0:::benchmark_data_smithi179_77138_object31006:head# 2024-02-09T17:26:53.576 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb46cd85:::benchmark_data_smithi179_77138_object110497:head# 2024-02-09T17:26:53.578 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb476082:::benchmark_data_smithi179_77138_object116702:head# 2024-02-09T17:26:53.579 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb482fcf:::benchmark_data_smithi179_77138_object83492:head# 2024-02-09T17:26:53.579 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb485d95:::benchmark_data_smithi179_77138_object23565:head# 2024-02-09T17:26:53.579 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb4884da:::benchmark_data_smithi179_77138_object68422:head# 2024-02-09T17:26:53.580 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb4902f7:::benchmark_data_smithi179_77138_object120170:head# 2024-02-09T17:26:53.580 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb499c14:::benchmark_data_smithi179_77138_object63542:head# 2024-02-09T17:26:53.581 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb49abdf:::benchmark_data_smithi179_77138_object40268:head# 2024-02-09T17:26:53.582 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb4ae8a9:::benchmark_data_smithi179_77138_object27936:head# 2024-02-09T17:26:53.582 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb4af525:::benchmark_data_smithi179_77138_object41454:head# 2024-02-09T17:26:53.582 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb4b7bbb:::benchmark_data_smithi179_77138_object115271:head# 2024-02-09T17:26:53.583 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb4b9b73:::benchmark_data_smithi179_77138_object108967:head# 2024-02-09T17:26:53.583 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb4ba1e1:::benchmark_data_smithi179_77138_object54826:head# 2024-02-09T17:26:53.584 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb4c5797:::benchmark_data_smithi179_77138_object116434:head# 2024-02-09T17:26:53.584 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb4de436:::benchmark_data_smithi179_77138_object32303:head# 2024-02-09T17:26:53.584 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb4e6a1c:::benchmark_data_smithi179_77138_object47093:head# 2024-02-09T17:26:53.585 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb4f4869:::benchmark_data_smithi179_77138_object72524:head# 2024-02-09T17:26:53.585 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb4f8e89:::benchmark_data_smithi179_77138_object43111:head# 2024-02-09T17:26:53.585 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb4fec8b:::benchmark_data_smithi179_77138_object64628:head# 2024-02-09T17:26:53.586 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb505988:::benchmark_data_smithi179_77138_object99418:head# 2024-02-09T17:26:53.586 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb50b9f5:::benchmark_data_smithi179_77138_object27290:head# 2024-02-09T17:26:53.592 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb50e399:::benchmark_data_smithi179_77138_object89422:head# 2024-02-09T17:26:53.592 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb512338:::benchmark_data_smithi179_77138_object71909:head# 2024-02-09T17:26:53.592 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb512d1c:::benchmark_data_smithi179_77138_object95785:head# 2024-02-09T17:26:53.595 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb522de0:::benchmark_data_smithi179_77138_object25292:head# 2024-02-09T17:26:53.597 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb525ece:::benchmark_data_smithi179_77138_object56174:head# 2024-02-09T17:26:53.600 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb53a77d:::benchmark_data_smithi179_77138_object52439:head# 2024-02-09T17:26:53.600 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb545065:::benchmark_data_smithi179_77138_object116365:head# 2024-02-09T17:26:53.600 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb54a278:::benchmark_data_smithi179_77138_object124606:head# 2024-02-09T17:26:53.600 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb553d0f:::benchmark_data_smithi179_77138_object27027:head# 2024-02-09T17:26:53.601 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb565407:::benchmark_data_smithi179_77138_object55274:head# 2024-02-09T17:26:53.601 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb56ba35:::benchmark_data_smithi179_77138_object49586:head# 2024-02-09T17:26:53.601 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb57299a:::benchmark_data_smithi179_77138_object57588:head# 2024-02-09T17:26:53.602 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb57b773:::benchmark_data_smithi179_77138_object51722:head# 2024-02-09T17:26:53.602 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb58310d:::benchmark_data_smithi179_77138_object117835:head# 2024-02-09T17:26:53.603 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb588796:::benchmark_data_smithi179_77138_object46438:head# 2024-02-09T17:26:53.603 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb59a976:::benchmark_data_smithi179_77138_object53883:head# 2024-02-09T17:26:53.604 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb5adf59:::benchmark_data_smithi179_77138_object113243:head# 2024-02-09T17:26:53.604 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb5b2330:::benchmark_data_smithi179_77138_object71423:head# 2024-02-09T17:26:53.604 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb5b9842:::benchmark_data_smithi179_77138_object30483:head# 2024-02-09T17:26:53.605 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb5bbcc6:::benchmark_data_smithi179_77138_object118951:head# 2024-02-09T17:26:53.605 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb5bce70:::benchmark_data_smithi179_77138_object85114:head# 2024-02-09T17:26:53.606 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb5dd372:::benchmark_data_smithi179_77138_object48569:head# 2024-02-09T17:26:53.606 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb5df347:::benchmark_data_smithi179_77138_object70166:head# 2024-02-09T17:26:53.607 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb5f3dd2:::benchmark_data_smithi179_77138_object43612:head# 2024-02-09T17:26:53.607 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb5fdbf9:::benchmark_data_smithi179_77138_object59079:head# 2024-02-09T17:26:53.607 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb604757:::benchmark_data_smithi179_77138_object96977:head# 2024-02-09T17:26:53.608 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb606596:::benchmark_data_smithi179_77138_object91747:head# 2024-02-09T17:26:53.608 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb609354:::benchmark_data_smithi179_77138_object42167:head# 2024-02-09T17:26:53.609 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb610df7:::benchmark_data_smithi179_77138_object43479:head# 2024-02-09T17:26:53.609 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb6154d3:::benchmark_data_smithi179_77138_object19363:head# 2024-02-09T17:26:53.609 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb6180ec:::benchmark_data_smithi179_77138_object29075:head# 2024-02-09T17:26:53.610 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb61f614:::benchmark_data_smithi179_77138_object46996:head# 2024-02-09T17:26:53.610 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb62cc87:::benchmark_data_smithi179_77138_object114373:head# 2024-02-09T17:26:53.611 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb631675:::benchmark_data_smithi179_77138_object45601:head# 2024-02-09T17:26:53.611 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb642659:::benchmark_data_smithi179_77138_object33085:head# 2024-02-09T17:26:53.611 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb6428b7:::benchmark_data_smithi179_77138_object125468:head# 2024-02-09T17:26:53.612 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb645247:::benchmark_data_smithi179_77138_object57117:head# 2024-02-09T17:26:53.612 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb6718f6:::benchmark_data_smithi179_77138_object56277:head# 2024-02-09T17:26:53.613 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb67355d:::benchmark_data_smithi179_77138_object133163:head# 2024-02-09T17:26:53.613 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb6877ac:::benchmark_data_smithi179_77138_object47574:head# 2024-02-09T17:26:53.614 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb68c9de:::benchmark_data_smithi179_77138_object61166:head# 2024-02-09T17:26:53.614 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb68f48a:::benchmark_data_smithi179_77138_object95610:head# 2024-02-09T17:26:53.614 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb69586f:::benchmark_data_smithi179_77138_object67040:head# 2024-02-09T17:26:53.615 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb69d53d:::benchmark_data_smithi179_77138_object41859:head# 2024-02-09T17:26:53.615 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb6a28d8:::benchmark_data_smithi179_77138_object48886:head# 2024-02-09T17:26:53.616 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb6b9add:::benchmark_data_smithi179_77138_object113155:head# 2024-02-09T17:26:53.616 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb6be971:::benchmark_data_smithi179_77138_object120808:head# 2024-02-09T17:26:53.617 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb6c308f:::benchmark_data_smithi179_77138_object19510:head# 2024-02-09T17:26:53.617 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb6dba4f:::benchmark_data_smithi179_77138_object31470:head# 2024-02-09T17:26:53.618 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb6e3ef2:::benchmark_data_smithi179_77138_object34267:head# 2024-02-09T17:26:53.618 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb6ff23f:::benchmark_data_smithi179_77138_object125156:head# 2024-02-09T17:26:53.618 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb706804:::benchmark_data_smithi179_77138_object130574:head# 2024-02-09T17:26:53.619 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb722e66:::benchmark_data_smithi179_77138_object102965:head# 2024-02-09T17:26:53.619 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb7249de:::benchmark_data_smithi179_77138_object38650:head# 2024-02-09T17:26:53.620 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb73097a:::benchmark_data_smithi179_77138_object57365:head# 2024-02-09T17:26:53.620 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb73b913:::benchmark_data_smithi179_77138_object72656:head# 2024-02-09T17:26:53.620 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb7437a6:::benchmark_data_smithi179_77138_object22595:head# 2024-02-09T17:26:53.621 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb745030:::benchmark_data_smithi179_77138_object97477:head# 2024-02-09T17:26:53.621 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb74f8c8:::benchmark_data_smithi179_77138_object20181:head# 2024-02-09T17:26:53.622 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb757a78:::benchmark_data_smithi179_77138_object64607:head# 2024-02-09T17:26:53.622 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb7651d5:::benchmark_data_smithi179_77138_object133103:head# 2024-02-09T17:26:53.622 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb775c2d:::benchmark_data_smithi179_77138_object57063:head# 2024-02-09T17:26:53.623 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb7865ad:::benchmark_data_smithi179_77138_object77781:head# 2024-02-09T17:26:53.623 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb788a42:::benchmark_data_smithi179_77138_object70244:head# 2024-02-09T17:26:53.624 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb78c111:::benchmark_data_smithi179_77138_object116947:head# 2024-02-09T17:26:53.624 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb78ce1f:::benchmark_data_smithi179_77138_object95397:head# 2024-02-09T17:26:53.625 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb7a7c54:::benchmark_data_smithi179_77138_object85685:head# 2024-02-09T17:26:53.625 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb7b322a:::benchmark_data_smithi179_77138_object53216:head# 2024-02-09T17:26:53.625 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb7bbda1:::benchmark_data_smithi179_77138_object111963:head# 2024-02-09T17:26:53.626 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb7c2715:::benchmark_data_smithi179_77138_object81889:head# 2024-02-09T17:26:53.626 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb7d1107:::benchmark_data_smithi179_77138_object62664:head# 2024-02-09T17:26:53.626 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb7da9ba:::benchmark_data_smithi179_77138_object124835:head# 2024-02-09T17:26:53.627 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb7dd779:::benchmark_data_smithi179_77138_object120387:head# 2024-02-09T17:26:53.627 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb7e561c:::benchmark_data_smithi179_77138_object105688:head# 2024-02-09T17:26:53.627 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb7e6273:::benchmark_data_smithi179_77138_object24304:head# 2024-02-09T17:26:53.628 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb7fd374:::benchmark_data_smithi179_77138_object50888:head# 2024-02-09T17:26:53.628 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb80795d:::benchmark_data_smithi179_77138_object122668:head# 2024-02-09T17:26:53.629 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb82e77b:::benchmark_data_smithi179_77138_object21903:head# 2024-02-09T17:26:53.629 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb83c5f1:::benchmark_data_smithi179_77138_object85556:head# 2024-02-09T17:26:53.630 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb83c731:::benchmark_data_smithi179_77138_object66383:head# 2024-02-09T17:26:53.630 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb854f44:::benchmark_data_smithi179_77138_object77608:head# 2024-02-09T17:26:53.630 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb8585bb:::benchmark_data_smithi179_77138_object34625:head# 2024-02-09T17:26:53.631 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb859984:::benchmark_data_smithi179_77138_object86364:head# 2024-02-09T17:26:53.631 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb85aaf7:::benchmark_data_smithi179_77138_object32011:head# 2024-02-09T17:26:53.635 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:26:53.636 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb85b60c:::benchmark_data_smithi179_77138_object20541:head# 2024-02-09T17:26:53.636 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb85fa1e:::benchmark_data_smithi179_77138_object117358:head# 2024-02-09T17:26:53.636 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb8606c6:::benchmark_data_smithi179_77138_object49269:head# 2024-02-09T17:26:53.636 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb86e3c7:::benchmark_data_smithi179_77138_object20486:head# 2024-02-09T17:26:53.636 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb88a19d:::benchmark_data_smithi179_77138_object23508:head# 2024-02-09T17:26:53.636 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb8a6e18:::benchmark_data_smithi179_77138_object62769:head# 2024-02-09T17:26:53.637 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb8ad7f5:::benchmark_data_smithi179_77138_object69608:head# 2024-02-09T17:26:53.637 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb8af2aa:::benchmark_data_smithi179_77138_object57659:head# 2024-02-09T17:26:53.637 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb8b8133:::benchmark_data_smithi179_77138_object92302:head# 2024-02-09T17:26:53.637 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb8ba91b:::benchmark_data_smithi179_77138_object127101:head# 2024-02-09T17:26:53.637 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb8c0afb:::benchmark_data_smithi179_77138_object90225:head# 2024-02-09T17:26:53.637 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb8c12ac:::benchmark_data_smithi179_77138_object41843:head# 2024-02-09T17:26:53.637 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb8c5975:::benchmark_data_smithi179_77138_object55825:head# 2024-02-09T17:26:53.637 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb8cca33:::benchmark_data_smithi179_77138_object63244:head# 2024-02-09T17:26:53.638 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb8ce860:::benchmark_data_smithi179_77138_object32771:head# 2024-02-09T17:26:53.638 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb8ed748:::benchmark_data_smithi179_77138_object85201:head# 2024-02-09T17:26:53.639 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb8f127a:::benchmark_data_smithi179_77138_object122860:head# 2024-02-09T17:26:53.640 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb8f57a6:::benchmark_data_smithi179_77138_object122476:head# 2024-02-09T17:26:53.640 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb9426f8:::benchmark_data_smithi179_77138_object65942:head# 2024-02-09T17:26:53.640 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb947ec2:::benchmark_data_smithi179_77138_object96591:head# 2024-02-09T17:26:53.640 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb95397f:::benchmark_data_smithi179_77138_object131376:head# 2024-02-09T17:26:53.641 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb95e62b:::benchmark_data_smithi179_77138_object83805:head# 2024-02-09T17:26:53.641 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb98c3e7:::benchmark_data_smithi179_77138_object109860:head# 2024-02-09T17:26:53.641 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb99318c:::benchmark_data_smithi179_77138_object74796:head# 2024-02-09T17:26:53.642 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb9ae248:::benchmark_data_smithi179_77138_object62441:head# 2024-02-09T17:26:53.643 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb9b67ec:::benchmark_data_smithi179_77138_object40829:head# 2024-02-09T17:26:53.643 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb9cb30b:::benchmark_data_smithi179_77138_object123081:head# 2024-02-09T17:26:53.643 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb9cf7f4:::benchmark_data_smithi179_77138_object69201:head# 2024-02-09T17:26:53.644 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb9d8de0:::benchmark_data_smithi179_77138_object116582:head# 2024-02-09T17:26:53.644 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb9e9d0c:::benchmark_data_smithi179_77138_object119743:head# 2024-02-09T17:26:53.645 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fb9fe858:::benchmark_data_smithi179_77138_object89933:head# 2024-02-09T17:26:53.645 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fba086b2:::benchmark_data_smithi179_77138_object82622:head# 2024-02-09T17:26:53.645 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fba1689b:::benchmark_data_smithi179_77138_object41860:head# 2024-02-09T17:26:53.647 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fba1752c:::benchmark_data_smithi179_77138_object120481:head# 2024-02-09T17:26:53.647 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fba1896c:::benchmark_data_smithi179_77138_object26031:head# 2024-02-09T17:26:53.647 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fba1a354:::benchmark_data_smithi179_77138_object130562:head# 2024-02-09T17:26:53.647 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fba3fbac:::benchmark_data_smithi179_77138_object82575:head# 2024-02-09T17:26:53.648 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fba44bf4:::benchmark_data_smithi179_77138_object124070:head# 2024-02-09T17:26:53.648 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fba44c6a:::benchmark_data_smithi179_77138_object115325:head# 2024-02-09T17:26:53.649 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fba506f0:::benchmark_data_smithi179_77138_object88867:head# 2024-02-09T17:26:53.649 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fba525d8:::benchmark_data_smithi179_77138_object92663:head# 2024-02-09T17:26:53.649 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fba71e44:::benchmark_data_smithi179_77138_object44991:head# 2024-02-09T17:26:53.650 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fba78331:::benchmark_data_smithi179_77138_object91651:head# 2024-02-09T17:26:53.650 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fba85c88:::benchmark_data_smithi179_77138_object54837:head# 2024-02-09T17:26:53.651 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fba94e70:::benchmark_data_smithi179_77138_object112712:head# 2024-02-09T17:26:53.651 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fba9e29e:::benchmark_data_smithi179_77138_object30496:head# 2024-02-09T17:26:53.651 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbaaa0bb:::benchmark_data_smithi179_77138_object52668:head# 2024-02-09T17:26:53.651 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbab791b:::benchmark_data_smithi179_77138_object60200:head# 2024-02-09T17:26:53.652 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbab7aab:::benchmark_data_smithi179_77138_object93885:head# 2024-02-09T17:26:53.652 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbaba2b3:::benchmark_data_smithi179_77138_object76507:head# 2024-02-09T17:26:53.653 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbac0c19:::benchmark_data_smithi179_77138_object130067:head# 2024-02-09T17:26:53.653 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbacadd6:::benchmark_data_smithi179_77138_object55662:head# 2024-02-09T17:26:53.654 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbad02fb:::benchmark_data_smithi179_77138_object61288:head# 2024-02-09T17:26:53.654 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbad89b8:::benchmark_data_smithi179_77138_object80554:head# 2024-02-09T17:26:53.654 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbadb7b7:::benchmark_data_smithi179_77138_object43839:head# 2024-02-09T17:26:53.655 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbadbf2e:::benchmark_data_smithi179_77138_object21820:head# 2024-02-09T17:26:53.655 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbadec57:::benchmark_data_smithi179_77138_object117698:head# 2024-02-09T17:26:53.656 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbadfae9:::benchmark_data_smithi179_77138_object43318:head# 2024-02-09T17:26:53.656 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbaf3436:::benchmark_data_smithi179_77138_object121547:head# 2024-02-09T17:26:53.656 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbaf88e8:::benchmark_data_smithi179_77138_object122801:head# 2024-02-09T17:26:53.657 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbafc781:::benchmark_data_smithi179_77138_object84475:head# 2024-02-09T17:26:53.657 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbafd9f7:::benchmark_data_smithi179_77138_object51417:head# 2024-02-09T17:26:53.658 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbb1d4cd:::benchmark_data_smithi179_77138_object26303:head# 2024-02-09T17:26:53.658 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbb2f57e:::benchmark_data_smithi179_77138_object102651:head# 2024-02-09T17:26:53.658 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbb34aae:::benchmark_data_smithi179_77138_object90457:head# 2024-02-09T17:26:53.659 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbb38e1b:::benchmark_data_smithi179_77138_object106011:head# 2024-02-09T17:26:53.659 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbb39805:::benchmark_data_smithi179_77138_object65615:head# 2024-02-09T17:26:53.660 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbb3da92:::benchmark_data_smithi179_77138_object70567:head# 2024-02-09T17:26:53.660 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbb42dba:::benchmark_data_smithi179_77138_object132662:head# 2024-02-09T17:26:53.660 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbb42e4e:::benchmark_data_smithi179_77138_object30828:head# 2024-02-09T17:26:53.661 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbb4d9ca:::benchmark_data_smithi179_77138_object65735:head# 2024-02-09T17:26:53.661 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbb512e5:::benchmark_data_smithi179_77138_object91214:head# 2024-02-09T17:26:53.661 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbb56137:::benchmark_data_smithi179_77138_object33111:head# 2024-02-09T17:26:53.662 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbb5e85d:::benchmark_data_smithi179_77138_object95528:head# 2024-02-09T17:26:53.662 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbb62899:::benchmark_data_smithi179_77138_object22400:head# 2024-02-09T17:26:53.663 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbb6a732:::benchmark_data_smithi179_77138_object30204:head# 2024-02-09T17:26:53.663 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbb6adec:::benchmark_data_smithi179_77138_object68579:head# 2024-02-09T17:26:53.664 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbb8a3b1:::benchmark_data_smithi179_77138_object120976:head# 2024-02-09T17:26:53.664 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbb941ba:::benchmark_data_smithi179_77138_object100541:head# 2024-02-09T17:26:53.664 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbb964e7:::benchmark_data_smithi179_77138_object56026:head# 2024-02-09T17:26:53.665 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbb9d781:::benchmark_data_smithi179_77138_object99498:head# 2024-02-09T17:26:53.665 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbb9e130:::benchmark_data_smithi179_77138_object55283:head# 2024-02-09T17:26:53.666 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbba79d7:::benchmark_data_smithi179_77138_object120723:head# 2024-02-09T17:26:53.666 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbbabb56:::benchmark_data_smithi179_77138_object88588:head# 2024-02-09T17:26:53.666 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbbae57c:::benchmark_data_smithi179_77138_object112930:head# 2024-02-09T17:26:53.667 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbbaf7f0:::benchmark_data_smithi179_77138_object106482:head# 2024-02-09T17:26:53.667 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbbb38a2:::benchmark_data_smithi179_77138_object93743:head# 2024-02-09T17:26:53.668 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbbb5a95:::benchmark_data_smithi179_77138_object25250:head# 2024-02-09T17:26:53.668 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbbb7101:::benchmark_data_smithi179_77138_object86105:head# 2024-02-09T17:26:53.669 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbbbcf8a:::benchmark_data_smithi179_77138_object96377:head# 2024-02-09T17:26:53.669 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbbbe0d2:::benchmark_data_smithi179_77138_object30216:head# 2024-02-09T17:26:53.669 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbbc27a7:::benchmark_data_smithi179_77138_object124971:head# 2024-02-09T17:26:53.670 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbbc99b1:::benchmark_data_smithi179_77138_object103526:head# 2024-02-09T17:26:53.671 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbbd08cf:::benchmark_data_smithi179_77138_object46691:head# 2024-02-09T17:26:53.671 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbbdf0ea:::benchmark_data_smithi179_77138_object51288:head# 2024-02-09T17:26:53.671 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbbeddd6:::benchmark_data_smithi179_77138_object37094:head# 2024-02-09T17:26:53.672 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbbef2ed:::benchmark_data_smithi179_77138_object39363:head# 2024-02-09T17:26:53.672 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbbff750:::benchmark_data_smithi179_77138_object45656:head# 2024-02-09T17:26:53.673 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbc03e8f:::benchmark_data_smithi179_77138_object71628:head# 2024-02-09T17:26:53.673 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbc08a91:::benchmark_data_smithi179_77138_object37688:head# 2024-02-09T17:26:53.674 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbc0bc14:::benchmark_data_smithi179_77138_object35342:head# 2024-02-09T17:26:53.674 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbc0c334:::benchmark_data_smithi179_77138_object70027:head# 2024-02-09T17:26:53.675 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:53.676 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbc0e1d6:::benchmark_data_smithi179_77138_object74195:head# 2024-02-09T17:26:53.676 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbc0e974:::benchmark_data_smithi179_77138_object79555:head# 2024-02-09T17:26:53.676 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbc117c4:::benchmark_data_smithi179_77138_object113114:head# 2024-02-09T17:26:53.676 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbc183aa:::benchmark_data_smithi179_77138_object68389:head# 2024-02-09T17:26:53.677 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbc22e1b:::benchmark_data_smithi179_77138_object129145:head# 2024-02-09T17:26:53.677 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbc2a609:::benchmark_data_smithi179_77138_object108545:head# 2024-02-09T17:26:53.677 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbc2e23b:::benchmark_data_smithi179_77138_object40428:head# 2024-02-09T17:26:53.679 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbc2ffa8:::benchmark_data_smithi179_77138_object21605:head# 2024-02-09T17:26:53.679 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbc33474:::benchmark_data_smithi179_77138_object105409:head# 2024-02-09T17:26:53.679 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbc3b9d3:::benchmark_data_smithi179_77138_object90316:head# 2024-02-09T17:26:53.679 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbc484c2:::benchmark_data_smithi179_77138_object19502:head# 2024-02-09T17:26:53.680 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbc58300:::benchmark_data_smithi179_77138_object130950:head# 2024-02-09T17:26:53.680 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbc590d5:::benchmark_data_smithi179_77138_object22093:head# 2024-02-09T17:26:53.680 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbc61d67:::benchmark_data_smithi179_77138_object69905:head# 2024-02-09T17:26:53.682 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbc6c9f7:::benchmark_data_smithi179_77138_object113231:head# 2024-02-09T17:26:53.682 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbc6ce51:::benchmark_data_smithi179_77138_object22830:head# 2024-02-09T17:26:53.682 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbc6e439:::benchmark_data_smithi179_77138_object98553:head# 2024-02-09T17:26:53.682 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbc6efa4:::benchmark_data_smithi179_77138_object129190:head# 2024-02-09T17:26:53.682 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbc74183:::benchmark_data_smithi179_77138_object128866:head# 2024-02-09T17:26:53.682 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbc79054:::benchmark_data_smithi179_77138_object91662:head# 2024-02-09T17:26:53.683 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbc7dcd4:::benchmark_data_smithi179_77138_object93188:head# 2024-02-09T17:26:53.683 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbc8b8d9:::benchmark_data_smithi179_77138_object80063:head# 2024-02-09T17:26:53.684 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbca9605:::benchmark_data_smithi179_77138_object97236:head# 2024-02-09T17:26:53.684 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbcb3720:::benchmark_data_smithi179_77138_object127249:head# 2024-02-09T17:26:53.685 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbcb4512:::benchmark_data_smithi179_77138_object107299:head# 2024-02-09T17:26:53.685 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbcb6f84:::benchmark_data_smithi179_77138_object73501:head# 2024-02-09T17:26:53.686 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbcd0729:::benchmark_data_smithi179_77138_object105873:head# 2024-02-09T17:26:53.686 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbcd3bb7:::benchmark_data_smithi179_77138_object86897:head# 2024-02-09T17:26:53.686 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbcf3688:::benchmark_data_smithi179_77138_object119056:head# 2024-02-09T17:26:53.687 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbcf4d9a:::benchmark_data_smithi179_77138_object93090:head# 2024-02-09T17:26:53.687 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbcfe0af:::benchmark_data_smithi179_77138_object81295:head# 2024-02-09T17:26:53.688 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbd11c41:::benchmark_data_smithi179_77138_object61131:head# 2024-02-09T17:26:53.688 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbd14c90:::benchmark_data_smithi179_77138_object73625:head# 2024-02-09T17:26:53.689 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbd1a9c9:::benchmark_data_smithi179_77138_object99302:head# 2024-02-09T17:26:53.689 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbd21e56:::benchmark_data_smithi179_77138_object66099:head# 2024-02-09T17:26:53.690 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbd258e3:::benchmark_data_smithi179_77138_object50829:head# 2024-02-09T17:26:53.691 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbd2bfda:::benchmark_data_smithi179_77138_object88318:head# 2024-02-09T17:26:53.691 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbd3040b:::benchmark_data_smithi179_77138_object60790:head# 2024-02-09T17:26:53.692 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbd33c48:::benchmark_data_smithi179_77138_object109382:head# 2024-02-09T17:26:53.692 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbd358ec:::benchmark_data_smithi179_77138_object80584:head# 2024-02-09T17:26:53.693 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbd366e6:::benchmark_data_smithi179_77138_object65194:head# 2024-02-09T17:26:53.693 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbd3be16:::benchmark_data_smithi179_77138_object127922:head# 2024-02-09T17:26:53.693 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbd3c926:::benchmark_data_smithi179_77138_object108749:head# 2024-02-09T17:26:53.694 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbd46c34:::benchmark_data_smithi179_77138_object131886:head# 2024-02-09T17:26:53.694 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbd65968:::benchmark_data_smithi179_77138_object131183:head# 2024-02-09T17:26:53.695 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbd66455:::benchmark_data_smithi179_77138_object55172:head# 2024-02-09T17:26:53.695 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbd6e916:::benchmark_data_smithi179_77138_object71095:head# 2024-02-09T17:26:53.695 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbd74f5c:::benchmark_data_smithi179_77138_object28670:head# 2024-02-09T17:26:53.696 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbd7cf42:::benchmark_data_smithi179_77138_object55185:head# 2024-02-09T17:26:53.696 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbd809be:::benchmark_data_smithi179_77138_object52466:head# 2024-02-09T17:26:53.697 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbd872e7:::benchmark_data_smithi179_77138_object86408:head# 2024-02-09T17:26:53.699 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbd8b02b:::benchmark_data_smithi179_77138_object25555:head# 2024-02-09T17:26:53.699 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbda442d:::benchmark_data_smithi179_77138_object107644:head# 2024-02-09T17:26:53.699 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbdbc51e:::benchmark_data_smithi179_77138_object124025:head# 2024-02-09T17:26:53.699 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbdbe435:::benchmark_data_smithi179_77138_object83288:head# 2024-02-09T17:26:53.700 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbdc6c24:::benchmark_data_smithi179_77138_object23775:head# 2024-02-09T17:26:53.700 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbdca4fa:::benchmark_data_smithi179_77138_object100350:head# 2024-02-09T17:26:53.701 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbde9a47:::benchmark_data_smithi179_77138_object75204:head# 2024-02-09T17:26:53.702 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbdf0a37:::benchmark_data_smithi179_77138_object129674:head# 2024-02-09T17:26:53.702 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbe050e0:::benchmark_data_smithi179_77138_object90550:head# 2024-02-09T17:26:53.703 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbe16d4a:::benchmark_data_smithi179_77138_object35277:head# 2024-02-09T17:26:53.703 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbe16f57:::benchmark_data_smithi179_77138_object105969:head# 2024-02-09T17:26:53.703 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbe1ac0d:::benchmark_data_smithi179_77138_object28141:head# 2024-02-09T17:26:53.704 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbe1b02c:::benchmark_data_smithi179_77138_object124940:head# 2024-02-09T17:26:53.704 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbe1cd4c:::benchmark_data_smithi179_77138_object38397:head# 2024-02-09T17:26:53.705 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbe1e129:::benchmark_data_smithi179_77138_object122078:head# 2024-02-09T17:26:53.705 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbe24eb4:::benchmark_data_smithi179_77138_object67790:head# 2024-02-09T17:26:53.706 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbe2a89a:::benchmark_data_smithi179_77138_object112981:head# 2024-02-09T17:26:53.706 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbe338b9:::benchmark_data_smithi179_77138_object20849:head# 2024-02-09T17:26:53.706 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbe34408:::benchmark_data_smithi179_77138_object45666:head# 2024-02-09T17:26:53.707 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbe37af8:::benchmark_data_smithi179_77138_object107877:head# 2024-02-09T17:26:53.708 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbe37df9:::benchmark_data_smithi179_77138_object103030:head# 2024-02-09T17:26:53.708 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbe57402:::benchmark_data_smithi179_77138_object75792:head# 2024-02-09T17:26:53.709 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbe69cf8:::benchmark_data_smithi179_77138_object39334:head# 2024-02-09T17:26:53.709 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbe75653:::benchmark_data_smithi179_77138_object24741:head# 2024-02-09T17:26:53.709 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbe786dc:::benchmark_data_smithi179_77138_object85326:head# 2024-02-09T17:26:53.710 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbe7cce3:::benchmark_data_smithi179_77138_object69779:head# 2024-02-09T17:26:53.710 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbe8818a:::benchmark_data_smithi179_77138_object81308:head# 2024-02-09T17:26:53.710 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbe95200:::benchmark_data_smithi179_77138_object36337:head# 2024-02-09T17:26:53.711 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbe9a00e:::benchmark_data_smithi179_77138_object39243:head# 2024-02-09T17:26:53.711 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbeba4ef:::benchmark_data_smithi179_77138_object30001:head# 2024-02-09T17:26:53.712 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbebf16f:::benchmark_data_smithi179_77138_object34071:head# 2024-02-09T17:26:53.712 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbec945e:::benchmark_data_smithi179_77138_object126790:head# 2024-02-09T17:26:53.713 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbecc1b2:::benchmark_data_smithi179_77138_object98682:head# 2024-02-09T17:26:53.713 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbed2c13:::benchmark_data_smithi179_77138_object130869:head# 2024-02-09T17:26:53.713 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbed3170:::benchmark_data_smithi179_77138_object112559:head# 2024-02-09T17:26:53.714 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbed4b28:::benchmark_data_smithi179_77138_object75315:head# 2024-02-09T17:26:53.714 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbed77a2:::benchmark_data_smithi179_77138_object53275:head# 2024-02-09T17:26:53.714 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbf00f31:::benchmark_data_smithi179_77138_object76815:head# 2024-02-09T17:26:53.715 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbf05889:::benchmark_data_smithi179_77138_object76893:head# 2024-02-09T17:26:53.721 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbf07a8a:::benchmark_data_smithi179_77138_object83246:head# 2024-02-09T17:26:53.721 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbf09ce8:::benchmark_data_smithi179_77138_object54100:head# 2024-02-09T17:26:53.721 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbf0dec3:::benchmark_data_smithi179_77138_object76191:head# 2024-02-09T17:26:53.721 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbf0e2da:::benchmark_data_smithi179_77138_object121110:head# 2024-02-09T17:26:53.721 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbf2c578:::benchmark_data_smithi179_77138_object35979:head# 2024-02-09T17:26:53.721 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbf364a2:::benchmark_data_smithi179_77138_object60033:head# 2024-02-09T17:26:53.721 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbf586e8:::benchmark_data_smithi179_77138_object97723:head# 2024-02-09T17:26:53.721 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbf6746a:::benchmark_data_smithi179_77138_object38598:head# 2024-02-09T17:26:53.721 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbf780bb:::benchmark_data_smithi179_77138_object75113:head# 2024-02-09T17:26:53.721 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbf7ad09:::benchmark_data_smithi179_77138_object69076:head# 2024-02-09T17:26:53.722 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbf98b77:::benchmark_data_smithi179_77138_object100422:head# 2024-02-09T17:26:53.722 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbf9ed6e:::benchmark_data_smithi179_77138_object95893:head# 2024-02-09T17:26:53.722 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbfb42a4:::benchmark_data_smithi179_77138_object50229:head# 2024-02-09T17:26:53.722 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbfbf89a:::benchmark_data_smithi179_77138_object116933:head# 2024-02-09T17:26:53.722 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbfc4e35:::benchmark_data_smithi179_77138_object30081:head# 2024-02-09T17:26:53.723 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbfcee22:::benchmark_data_smithi179_77138_object122824:head# 2024-02-09T17:26:53.723 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbfd0a50:::benchmark_data_smithi179_77138_object89754:head# 2024-02-09T17:26:53.723 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbfef18d:::benchmark_data_smithi179_77138_object102992:head# 2024-02-09T17:26:53.724 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fbff48c8:::benchmark_data_smithi179_77138_object44342:head# 2024-02-09T17:26:53.724 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc0027fe:::benchmark_data_smithi179_77138_object25198:head# 2024-02-09T17:26:53.725 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc00f687:::benchmark_data_smithi179_77138_object132649:head# 2024-02-09T17:26:53.725 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc013d6f:::benchmark_data_smithi179_77138_object76744:head# 2024-02-09T17:26:53.725 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc014913:::benchmark_data_smithi179_77138_object61593:head# 2024-02-09T17:26:53.726 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc014d32:::benchmark_data_smithi179_77138_object31368:head# 2024-02-09T17:26:53.726 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc0180d2:::benchmark_data_smithi179_77138_object121396:head# 2024-02-09T17:26:53.727 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc020487:::benchmark_data_smithi179_77138_object76547:head# 2024-02-09T17:26:53.727 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc03e97b:::benchmark_data_smithi179_77138_object125031:head# 2024-02-09T17:26:53.727 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc03f2fa:::benchmark_data_smithi179_77138_object75241:head# 2024-02-09T17:26:53.728 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc05a2cb:::benchmark_data_smithi179_77138_object110184:head# 2024-02-09T17:26:53.728 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc05abdb:::benchmark_data_smithi179_77138_object97878:head# 2024-02-09T17:26:53.729 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc07b443:::benchmark_data_smithi179_77138_object119501:head# 2024-02-09T17:26:53.729 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc087a0d:::benchmark_data_smithi179_77138_object71629:head# 2024-02-09T17:26:53.730 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc088462:::benchmark_data_smithi179_77138_object129079:head# 2024-02-09T17:26:53.730 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc08d8d4:::benchmark_data_smithi179_77138_object99242:head# 2024-02-09T17:26:53.736 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc08de54:::benchmark_data_smithi179_77138_object115965:head# 2024-02-09T17:26:53.736 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc09b3f4:::benchmark_data_smithi179_77138_object122503:head# 2024-02-09T17:26:53.736 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc0a731e:::benchmark_data_smithi179_77138_object38180:head# 2024-02-09T17:26:53.736 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc0b3e58:::benchmark_data_smithi179_77138_object51422:head# 2024-02-09T17:26:53.736 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc0b9db3:::benchmark_data_smithi179_77138_object56939:head# 2024-02-09T17:26:53.736 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc0ba302:::benchmark_data_smithi179_77138_object91737:head# 2024-02-09T17:26:53.736 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc0c52ce:::benchmark_data_smithi179_77138_object41006:head# 2024-02-09T17:26:53.736 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc0d03db:::benchmark_data_smithi179_77138_object28644:head# 2024-02-09T17:26:53.737 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc0d3ba3:::benchmark_data_smithi179_77138_object39688:head# 2024-02-09T17:26:53.737 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc0d64a3:::benchmark_data_smithi179_77138_object99675:head# 2024-02-09T17:26:53.738 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc0dd8f0:::benchmark_data_smithi179_77138_object118251:head# 2024-02-09T17:26:53.738 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc0ea31d:::benchmark_data_smithi179_77138_object74109:head# 2024-02-09T17:26:53.738 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc104995:::benchmark_data_smithi179_77138_object43932:head# 2024-02-09T17:26:53.739 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc108d9a:::benchmark_data_smithi179_77138_object110123:head# 2024-02-09T17:26:53.739 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc1151c0:::benchmark_data_smithi179_77138_object80853:head# 2024-02-09T17:26:53.740 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc119d00:::benchmark_data_smithi179_77138_object81933:head# 2024-02-09T17:26:53.740 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc11f7df:::benchmark_data_smithi179_77138_object80409:head# 2024-02-09T17:26:53.741 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc126558:::benchmark_data_smithi179_77138_object125662:head# 2024-02-09T17:26:53.741 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc126a18:::benchmark_data_smithi179_77138_object53023:head# 2024-02-09T17:26:53.741 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc12717e:::benchmark_data_smithi179_77138_object94367:head# 2024-02-09T17:26:53.742 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc12862b:::benchmark_data_smithi179_77138_object30259:head# 2024-02-09T17:26:53.742 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc12ab02:::benchmark_data_smithi179_77138_object73786:head# 2024-02-09T17:26:53.743 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc12b8af:::benchmark_data_smithi179_77138_object71890:head# 2024-02-09T17:26:53.743 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc1320d5:::benchmark_data_smithi179_77138_object54228:head# 2024-02-09T17:26:53.744 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc14278b:::benchmark_data_smithi179_77138_object23916:head# 2024-02-09T17:26:53.744 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc14dc7f:::benchmark_data_smithi179_77138_object73972:head# 2024-02-09T17:26:53.745 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc14e70b:::benchmark_data_smithi179_77138_object85656:head# 2024-02-09T17:26:53.747 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc15ab01:::benchmark_data_smithi179_77138_object124102:head# 2024-02-09T17:26:53.748 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc1612e0:::benchmark_data_smithi179_77138_object24597:head# 2024-02-09T17:26:53.748 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc17dabb:::benchmark_data_smithi179_77138_object69010:head# 2024-02-09T17:26:53.748 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc180cff:::benchmark_data_smithi179_77138_object123844:head# 2024-02-09T17:26:53.749 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc18287a:::benchmark_data_smithi179_77138_object20393:head# 2024-02-09T17:26:53.749 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc182b09:::benchmark_data_smithi179_77138_object110481:head# 2024-02-09T17:26:53.750 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc18b15b:::benchmark_data_smithi179_77138_object91976:head# 2024-02-09T17:26:53.750 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc191cfe:::benchmark_data_smithi179_77138_object91385:head# 2024-02-09T17:26:53.751 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc193a30:::benchmark_data_smithi179_77138_object23200:head# 2024-02-09T17:26:53.751 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc199006:::benchmark_data_smithi179_77138_object84601:head# 2024-02-09T17:26:53.751 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc1a1601:::benchmark_data_smithi179_77138_object47155:head# 2024-02-09T17:26:53.754 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc1a8b93:::benchmark_data_smithi179_77138_object108384:head# 2024-02-09T17:26:53.755 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc1bad91:::benchmark_data_smithi179_77138_object63109:head# 2024-02-09T17:26:53.755 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc1dcb11:::benchmark_data_smithi179_77138_object22699:head# 2024-02-09T17:26:53.756 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc1e31dc:::benchmark_data_smithi179_77138_object123501:head# 2024-02-09T17:26:53.756 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc1e6e88:::benchmark_data_smithi179_77138_object118003:head# 2024-02-09T17:26:53.757 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc1f9014:::benchmark_data_smithi179_77138_object71334:head# 2024-02-09T17:26:53.757 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc1f9051:::benchmark_data_smithi179_77138_object119480:head# 2024-02-09T17:26:53.757 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc1fd953:::benchmark_data_smithi179_77138_object48672:head# 2024-02-09T17:26:53.758 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc21ee78:::benchmark_data_smithi179_77138_object120841:head# 2024-02-09T17:26:53.758 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc21f99b:::benchmark_data_smithi179_77138_object92913:head# 2024-02-09T17:26:53.759 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc2262ba:::benchmark_data_smithi179_77138_object96057:head# 2024-02-09T17:26:53.759 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc22ee65:::benchmark_data_smithi179_77138_object44347:head# 2024-02-09T17:26:53.759 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc2305f7:::benchmark_data_smithi179_77138_object119568:head# 2024-02-09T17:26:53.760 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc241e61:::benchmark_data_smithi179_77138_object33697:head# 2024-02-09T17:26:53.760 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc2498e5:::benchmark_data_smithi179_77138_object90622:head# 2024-02-09T17:26:53.760 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc24cbca:::benchmark_data_smithi179_77138_object69149:head# 2024-02-09T17:26:53.761 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc2558f5:::benchmark_data_smithi179_77138_object61138:head# 2024-02-09T17:26:53.761 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc258371:::benchmark_data_smithi179_77138_object79428:head# 2024-02-09T17:26:53.762 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc25937a:::benchmark_data_smithi179_77138_object26185:head# 2024-02-09T17:26:53.762 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc25f899:::benchmark_data_smithi179_77138_object110295:head# 2024-02-09T17:26:53.763 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc2607e3:::benchmark_data_smithi179_77138_object59673:head# 2024-02-09T17:26:53.763 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc263708:::benchmark_data_smithi179_77138_object105370:head# 2024-02-09T17:26:53.763 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc2784d9:::benchmark_data_smithi179_77138_object74328:head# 2024-02-09T17:26:53.764 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc278c64:::benchmark_data_smithi179_77138_object59785:head# 2024-02-09T17:26:53.764 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc288e92:::benchmark_data_smithi179_77138_object121799:head# 2024-02-09T17:26:53.765 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc291f37:::benchmark_data_smithi179_77138_object128797:head# 2024-02-09T17:26:53.765 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc2936c4:::benchmark_data_smithi179_77138_object108848:head# 2024-02-09T17:26:53.765 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc29a191:::benchmark_data_smithi179_77138_object114760:head# 2024-02-09T17:26:53.766 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc29cf61:::benchmark_data_smithi179_77138_object69757:head# 2024-02-09T17:26:53.767 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc2bfff6:::benchmark_data_smithi179_77138_object63661:head# 2024-02-09T17:26:53.767 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc2c07c3:::benchmark_data_smithi179_77138_object78775:head# 2024-02-09T17:26:53.768 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc2c2b01:::benchmark_data_smithi179_77138_object84978:head# 2024-02-09T17:26:53.768 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc2c8305:::benchmark_data_smithi179_77138_object94626:head# 2024-02-09T17:26:53.769 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc2c983f:::benchmark_data_smithi179_77138_object31352:head# 2024-02-09T17:26:53.770 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc2d5d1a:::benchmark_data_smithi179_77138_object131850:head# 2024-02-09T17:26:53.770 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc2ec06e:::benchmark_data_smithi179_77138_object128359:head# 2024-02-09T17:26:53.770 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc30457b:::benchmark_data_smithi179_77138_object126480:head# 2024-02-09T17:26:53.771 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc306e1a:::benchmark_data_smithi179_77138_object37383:head# 2024-02-09T17:26:53.772 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc307788:::benchmark_data_smithi179_77138_object116736:head# 2024-02-09T17:26:53.772 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc3088a0:::benchmark_data_smithi179_77138_object75463:head# 2024-02-09T17:26:53.773 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc3095ae:::benchmark_data_smithi179_77138_object65345:head# 2024-02-09T17:26:53.773 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc309c5f:::benchmark_data_smithi179_77138_object60149:head# 2024-02-09T17:26:53.773 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc30d726:::benchmark_data_smithi179_77138_object61277:head# 2024-02-09T17:26:53.773 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc31797e:::benchmark_data_smithi179_77138_object51850:head# 2024-02-09T17:26:53.774 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc327703:::benchmark_data_smithi179_77138_object53715:head# 2024-02-09T17:26:53.774 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc335498:::benchmark_data_smithi179_77138_object61083:head# 2024-02-09T17:26:53.775 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc344859:::benchmark_data_smithi179_77138_object97227:head# 2024-02-09T17:26:53.782 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc35a011:::benchmark_data_smithi179_77138_object65697:head# 2024-02-09T17:26:53.782 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc35d463:::benchmark_data_smithi179_77138_object61810:head# 2024-02-09T17:26:53.782 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc377ffe:::benchmark_data_smithi179_77138_object87110:head# 2024-02-09T17:26:53.782 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc37eaa6:::benchmark_data_smithi179_77138_object67493:head# 2024-02-09T17:26:53.783 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc381151:::benchmark_data_smithi179_77138_object116233:head# 2024-02-09T17:26:53.783 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc38e4e5:::benchmark_data_smithi179_77138_object42567:head# 2024-02-09T17:26:53.783 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc391198:::benchmark_data_smithi179_77138_object55706:head# 2024-02-09T17:26:53.783 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc3975fc:::benchmark_data_smithi179_77138_object36668:head# 2024-02-09T17:26:53.783 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc39e4fc:::benchmark_data_smithi179_77138_object111284:head# 2024-02-09T17:26:53.783 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc39efe5:::benchmark_data_smithi179_77138_object96207:head# 2024-02-09T17:26:53.783 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc3a5b7c:::benchmark_data_smithi179_77138_object104658:head# 2024-02-09T17:26:53.785 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc3a8f8e:::benchmark_data_smithi179_77138_object44772:head# 2024-02-09T17:26:53.785 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc3b13e8:::benchmark_data_smithi179_77138_object50411:head# 2024-02-09T17:26:53.785 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc3b1eb8:::benchmark_data_smithi179_77138_object133254:head# 2024-02-09T17:26:53.786 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc3b8096:::benchmark_data_smithi179_77138_object107985:head# 2024-02-09T17:26:53.789 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc3bdb4a:::benchmark_data_smithi179_77138_object126382:head# 2024-02-09T17:26:53.789 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc3ce435:::benchmark_data_smithi179_77138_object112146:head# 2024-02-09T17:26:53.789 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc3ddabd:::benchmark_data_smithi179_77138_object83493:head# 2024-02-09T17:26:53.790 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc3dfdc4:::benchmark_data_smithi179_77138_object50787:head# 2024-02-09T17:26:53.790 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc3f2367:::benchmark_data_smithi179_77138_object103825:head# 2024-02-09T17:26:53.790 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc3f57f0:::benchmark_data_smithi179_77138_object46680:head# 2024-02-09T17:26:53.791 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc40804b:::benchmark_data_smithi179_77138_object42568:head# 2024-02-09T17:26:53.791 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc4141b5:::benchmark_data_smithi179_77138_object37347:head# 2024-02-09T17:26:53.792 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc419dc4:::benchmark_data_smithi179_77138_object98061:head# 2024-02-09T17:26:53.792 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc43723d:::benchmark_data_smithi179_77138_object117811:head# 2024-02-09T17:26:53.792 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc43a3c9:::benchmark_data_smithi179_77138_object132346:head# 2024-02-09T17:26:53.793 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc451c12:::benchmark_data_smithi179_77138_object82318:head# 2024-02-09T17:26:53.795 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc45d103:::benchmark_data_smithi179_77138_object51235:head# 2024-02-09T17:26:53.795 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc45d78e:::benchmark_data_smithi179_77138_object21608:head# 2024-02-09T17:26:53.795 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc472f5b:::benchmark_data_smithi179_77138_object30848:head# 2024-02-09T17:26:53.795 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc47fcc5:::benchmark_data_smithi179_77138_object28824:head# 2024-02-09T17:26:53.796 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc4c2a6f:::benchmark_data_smithi179_77138_object54706:head# 2024-02-09T17:26:53.796 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc4cb367:::benchmark_data_smithi179_77138_object111325:head# 2024-02-09T17:26:53.796 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc4d68bc:::benchmark_data_smithi179_77138_object98808:head# 2024-02-09T17:26:53.797 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc4dc426:::benchmark_data_smithi179_77138_object121983:head# 2024-02-09T17:26:53.797 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc4eac35:::benchmark_data_smithi179_77138_object73834:head# 2024-02-09T17:26:53.798 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc4f1dca:::benchmark_data_smithi179_77138_object108943:head# 2024-02-09T17:26:53.798 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc4f28a8:::benchmark_data_smithi179_77138_object66947:head# 2024-02-09T17:26:53.798 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc4f4774:::benchmark_data_smithi179_77138_object114293:head# 2024-02-09T17:26:53.799 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc4f7367:::benchmark_data_smithi179_77138_object45803:head# 2024-02-09T17:26:53.799 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc4f7c9d:::benchmark_data_smithi179_77138_object120887:head# 2024-02-09T17:26:53.800 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc4fca18:::benchmark_data_smithi179_77138_object111372:head# 2024-02-09T17:26:53.801 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc51106b:::benchmark_data_smithi179_77138_object55683:head# 2024-02-09T17:26:53.801 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc516376:::benchmark_data_smithi179_77138_object67548:head# 2024-02-09T17:26:53.802 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc520557:::benchmark_data_smithi179_77138_object107719:head# 2024-02-09T17:26:53.803 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc523f51:::benchmark_data_smithi179_77138_object119734:head# 2024-02-09T17:26:53.803 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc524769:::benchmark_data_smithi179_77138_object91110:head# 2024-02-09T17:26:53.805 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc52503a:::benchmark_data_smithi179_77138_object22476:head# 2024-02-09T17:26:53.805 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc52e4ff:::benchmark_data_smithi179_77138_object106545:head# 2024-02-09T17:26:53.806 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc5302ee:::benchmark_data_smithi179_77138_object62889:head# 2024-02-09T17:26:53.806 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc54532e:::benchmark_data_smithi179_77138_object72809:head# 2024-02-09T17:26:53.806 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc54b9c3:::benchmark_data_smithi179_77138_object118390:head# 2024-02-09T17:26:53.809 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc555720:::benchmark_data_smithi179_77138_object108218:head# 2024-02-09T17:26:53.809 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc560e93:::benchmark_data_smithi179_77138_object30820:head# 2024-02-09T17:26:53.809 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc5659d7:::benchmark_data_smithi179_77138_object34509:head# 2024-02-09T17:26:53.810 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc576470:::benchmark_data_smithi179_77138_object65048:head# 2024-02-09T17:26:53.810 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc589759:::benchmark_data_smithi179_77138_object111484:head# 2024-02-09T17:26:53.811 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc5a565d:::benchmark_data_smithi179_77138_object58277:head# 2024-02-09T17:26:53.811 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc5aa845:::benchmark_data_smithi179_77138_object83818:head# 2024-02-09T17:26:53.812 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc5c65fd:::benchmark_data_smithi179_77138_object42263:head# 2024-02-09T17:26:53.812 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc5c775d:::benchmark_data_smithi179_77138_object92178:head# 2024-02-09T17:26:53.812 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc5c8350:::benchmark_data_smithi179_77138_object72594:head# 2024-02-09T17:26:53.813 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc5caebb:::benchmark_data_smithi179_77138_object107608:head# 2024-02-09T17:26:53.813 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc5d313f:::benchmark_data_smithi179_77138_object43793:head# 2024-02-09T17:26:53.816 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc5e8991:::benchmark_data_smithi179_77138_object45856:head# 2024-02-09T17:26:53.817 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc609835:::benchmark_data_smithi179_77138_object123512:head# 2024-02-09T17:26:53.817 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc60df9f:::benchmark_data_smithi179_77138_object62958:head# 2024-02-09T17:26:53.817 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc611a35:::benchmark_data_smithi179_77138_object89637:head# 2024-02-09T17:26:53.818 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc6323fa:::benchmark_data_smithi179_77138_object113547:head# 2024-02-09T17:26:53.818 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc646018:::benchmark_data_smithi179_77138_object85255:head# 2024-02-09T17:26:53.820 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc64c569:::benchmark_data_smithi179_77138_object55323:head# 2024-02-09T17:26:53.820 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc64f89a:::benchmark_data_smithi179_77138_object20536:head# 2024-02-09T17:26:53.820 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc653c42:::benchmark_data_smithi179_77138_object132875:head# 2024-02-09T17:26:53.821 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc65c335:::benchmark_data_smithi179_77138_object77171:head# 2024-02-09T17:26:53.821 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc65e320:::benchmark_data_smithi179_77138_object94715:head# 2024-02-09T17:26:53.822 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc665cd6:::benchmark_data_smithi179_77138_object38483:head# 2024-02-09T17:26:53.822 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc66968e:::benchmark_data_smithi179_77138_object29596:head# 2024-02-09T17:26:53.823 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc6772d6:::benchmark_data_smithi179_77138_object24373:head# 2024-02-09T17:26:53.824 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc68874f:::benchmark_data_smithi179_77138_object98440:head# 2024-02-09T17:26:53.824 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc696c94:::benchmark_data_smithi179_77138_object112731:head# 2024-02-09T17:26:53.824 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc6ab1e2:::benchmark_data_smithi179_77138_object102816:head# 2024-02-09T17:26:53.824 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc6b7e7e:::benchmark_data_smithi179_77138_object68856:head# 2024-02-09T17:26:53.825 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc6bc308:::benchmark_data_smithi179_77138_object83393:head# 2024-02-09T17:26:53.825 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc6c1ff6:::benchmark_data_smithi179_77138_object47820:head# 2024-02-09T17:26:53.826 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc6cdbe6:::benchmark_data_smithi179_77138_object56663:head# 2024-02-09T17:26:53.826 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc6d0219:::benchmark_data_smithi179_77138_object64902:head# 2024-02-09T17:26:53.828 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc6d170d:::benchmark_data_smithi179_77138_object93886:head# 2024-02-09T17:26:53.828 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc6e17fe:::benchmark_data_smithi179_77138_object71542:head# 2024-02-09T17:26:53.829 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:26:53.830 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:53 smithi162 ceph-mon[44304]: pgmap v453: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 6.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 881 op/s; 26608/318045 objects degraded (8.366%); 0 B/s, 0 objects/s recovering 2024-02-09T17:26:53.830 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc6fd654:::benchmark_data_smithi179_77138_object49099:head# 2024-02-09T17:26:53.833 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc70de7a:::benchmark_data_smithi179_77138_object69712:head# 2024-02-09T17:26:53.834 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc7178e3:::benchmark_data_smithi179_77138_object127247:head# 2024-02-09T17:26:53.834 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc71a946:::benchmark_data_smithi179_77138_object44920:head# 2024-02-09T17:26:53.835 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc725f99:::benchmark_data_smithi179_77138_object44567:head# 2024-02-09T17:26:53.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:53.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:26:53.840 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc738ad5:::benchmark_data_smithi179_77138_object108076:head# 2024-02-09T17:26:53.840 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc74fdb8:::benchmark_data_smithi179_77138_object95194:head# 2024-02-09T17:26:53.841 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc751135:::benchmark_data_smithi179_77138_object48108:head# 2024-02-09T17:26:53.841 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc75a813:::benchmark_data_smithi179_77138_object33050:head# 2024-02-09T17:26:53.841 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc7954cd:::benchmark_data_smithi179_77138_object25384:head# 2024-02-09T17:26:53.841 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc7a4899:::benchmark_data_smithi179_77138_object63439:head# 2024-02-09T17:26:53.841 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc7b15e7:::benchmark_data_smithi179_77138_object40850:head# 2024-02-09T17:26:53.841 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc7b92ff:::benchmark_data_smithi179_77138_object27482:head# 2024-02-09T17:26:53.841 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc7bb0d3:::benchmark_data_smithi179_77138_object25795:head# 2024-02-09T17:26:53.843 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc7c5437:::benchmark_data_smithi179_77138_object69296:head# 2024-02-09T17:26:53.843 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc7e20da:::benchmark_data_smithi179_77138_object68101:head# 2024-02-09T17:26:53.844 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc7ee92e:::benchmark_data_smithi179_77138_object85979:head# 2024-02-09T17:26:53.844 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc7ee958:::benchmark_data_smithi179_77138_object54469:head# 2024-02-09T17:26:53.845 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc8040f4:::benchmark_data_smithi179_77138_object29767:head# 2024-02-09T17:26:53.845 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc804375:::benchmark_data_smithi179_77138_object57112:head# 2024-02-09T17:26:53.846 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc80bd55:::benchmark_data_smithi179_77138_object99553:head# 2024-02-09T17:26:53.846 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc80c2c9:::benchmark_data_smithi179_77138_object25412:head# 2024-02-09T17:26:53.847 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc80c8b5:::benchmark_data_smithi179_77138_object117542:head# 2024-02-09T17:26:53.849 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc833a08:::benchmark_data_smithi179_77138_object131309:head# 2024-02-09T17:26:53.849 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc83ba1a:::benchmark_data_smithi179_77138_object119661:head# 2024-02-09T17:26:53.851 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc83c8ce:::benchmark_data_smithi179_77138_object96430:head# 2024-02-09T17:26:53.851 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc83ebe7:::benchmark_data_smithi179_77138_object34614:head# 2024-02-09T17:26:53.852 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc84d1d0:::benchmark_data_smithi179_77138_object103690:head# 2024-02-09T17:26:53.854 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc858f60:::benchmark_data_smithi179_77138_object111922:head# 2024-02-09T17:26:53.855 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc85fa07:::benchmark_data_smithi179_77138_object97497:head# 2024-02-09T17:26:53.855 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc864847:::benchmark_data_smithi179_77138_object27585:head# 2024-02-09T17:26:53.855 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc86aa58:::benchmark_data_smithi179_77138_object124075:head# 2024-02-09T17:26:53.855 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc87188e:::benchmark_data_smithi179_77138_object57682:head# 2024-02-09T17:26:53.855 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc887d83:::benchmark_data_smithi179_77138_object118910:head# 2024-02-09T17:26:53.855 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc889f1d:::benchmark_data_smithi179_77138_object74117:head# 2024-02-09T17:26:53.855 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc893259:::benchmark_data_smithi179_77138_object96152:head# 2024-02-09T17:26:53.856 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc897529:::benchmark_data_smithi179_77138_object112266:head# 2024-02-09T17:26:53.857 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc8aa698:::benchmark_data_smithi179_77138_object91917:head# 2024-02-09T17:26:53.857 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc8cae61:::benchmark_data_smithi179_77138_object35327:head# 2024-02-09T17:26:53.857 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc8d4df6:::benchmark_data_smithi179_77138_object20057:head# 2024-02-09T17:26:53.858 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc8e250b:::benchmark_data_smithi179_77138_object88777:head# 2024-02-09T17:26:53.858 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc8f577d:::benchmark_data_smithi179_77138_object104452:head# 2024-02-09T17:26:53.859 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc8f7558:::benchmark_data_smithi179_77138_object61950:head# 2024-02-09T17:26:53.859 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc8fbbf0:::benchmark_data_smithi179_77138_object93035:head# 2024-02-09T17:26:53.859 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc908cfa:::benchmark_data_smithi179_77138_object72365:head# 2024-02-09T17:26:53.860 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc910f87:::benchmark_data_smithi179_77138_object68798:head# 2024-02-09T17:26:53.860 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc916e7a:::benchmark_data_smithi179_77138_object47856:head# 2024-02-09T17:26:53.861 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc91cff6:::benchmark_data_smithi179_77138_object68550:head# 2024-02-09T17:26:53.861 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc92755d:::benchmark_data_smithi179_77138_object115593:head# 2024-02-09T17:26:53.861 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc933cc5:::benchmark_data_smithi179_77138_object27535:head# 2024-02-09T17:26:53.862 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc9347d9:::benchmark_data_smithi179_77138_object127469:head# 2024-02-09T17:26:53.863 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc9421a8:::benchmark_data_smithi179_77138_object98688:head# 2024-02-09T17:26:53.863 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc94620c:::benchmark_data_smithi179_77138_object67165:head# 2024-02-09T17:26:53.863 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc94bca2:::benchmark_data_smithi179_77138_object45493:head# 2024-02-09T17:26:53.867 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc950d68:::benchmark_data_smithi179_77138_object82130:head# 2024-02-09T17:26:53.868 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc9613fe:::benchmark_data_smithi179_77138_object89068:head# 2024-02-09T17:26:53.868 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc967447:::benchmark_data_smithi179_77138_object57912:head# 2024-02-09T17:26:53.868 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc96d4c1:::benchmark_data_smithi179_77138_object117370:head# 2024-02-09T17:26:53.868 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc96e1c6:::benchmark_data_smithi179_77138_object29686:head# 2024-02-09T17:26:53.871 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:53 smithi099 ceph-mon[39420]: pgmap v453: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 6.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 881 op/s; 26608/318045 objects degraded (8.366%); 0 B/s, 0 objects/s recovering 2024-02-09T17:26:53.871 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc96ec62:::benchmark_data_smithi179_77138_object33420:head# 2024-02-09T17:26:53.871 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc970819:::benchmark_data_smithi179_77138_object69815:head# 2024-02-09T17:26:53.873 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc981046:::benchmark_data_smithi179_77138_object96747:head# 2024-02-09T17:26:53.874 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc983410:::benchmark_data_smithi179_77138_object124056:head# 2024-02-09T17:26:53.875 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc98373c:::benchmark_data_smithi179_77138_object21800:head# 2024-02-09T17:26:53.876 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc984fca:::benchmark_data_smithi179_77138_object52326:head# 2024-02-09T17:26:53.876 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc9943a9:::benchmark_data_smithi179_77138_object117269:head# 2024-02-09T17:26:53.877 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc9b5910:::benchmark_data_smithi179_77138_object124754:head# 2024-02-09T17:26:53.877 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc9b9b2c:::benchmark_data_smithi179_77138_object65984:head# 2024-02-09T17:26:53.877 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc9bd8a2:::benchmark_data_smithi179_77138_object55519:head# 2024-02-09T17:26:53.878 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc9ce3af:::benchmark_data_smithi179_77138_object41233:head# 2024-02-09T17:26:53.878 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc9d6c77:::benchmark_data_smithi179_77138_object130911:head# 2024-02-09T17:26:53.880 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc9d80e8:::benchmark_data_smithi179_77138_object86226:head# 2024-02-09T17:26:53.880 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc9da2c1:::benchmark_data_smithi179_77138_object93368:head# 2024-02-09T17:26:53.880 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc9e1f63:::benchmark_data_smithi179_77138_object123990:head# 2024-02-09T17:26:53.881 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fc9e96d3:::benchmark_data_smithi179_77138_object55358:head# 2024-02-09T17:26:53.881 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fca0fa7f:::benchmark_data_smithi179_77138_object126258:head# 2024-02-09T17:26:53.882 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fca12f7c:::benchmark_data_smithi179_77138_object69222:head# 2024-02-09T17:26:53.882 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fca1e516:::benchmark_data_smithi179_77138_object26171:head# 2024-02-09T17:26:53.883 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fca1ee81:::benchmark_data_smithi179_77138_object128090:head# 2024-02-09T17:26:53.883 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fca20e11:::benchmark_data_smithi179_77138_object78108:head# 2024-02-09T17:26:53.883 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fca21401:::benchmark_data_smithi179_77138_object112662:head# 2024-02-09T17:26:53.884 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fca27242:::benchmark_data_smithi179_77138_object40349:head# 2024-02-09T17:26:53.884 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fca2bf4f:::benchmark_data_smithi179_77138_object25538:head# 2024-02-09T17:26:53.885 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fca2f295:::benchmark_data_smithi179_77138_object20852:head# 2024-02-09T17:26:53.885 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fca32f8f:::benchmark_data_smithi179_77138_object104867:head# 2024-02-09T17:26:53.886 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fca333ee:::benchmark_data_smithi179_77138_object22168:head# 2024-02-09T17:26:53.886 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fca3abb4:::benchmark_data_smithi179_77138_object128876:head# 2024-02-09T17:26:53.887 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fca4cf04:::benchmark_data_smithi179_77138_object123134:head# 2024-02-09T17:26:53.887 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fca5a325:::benchmark_data_smithi179_77138_object27624:head# 2024-02-09T17:26:53.888 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fca5ed29:::benchmark_data_smithi179_77138_object22420:head# 2024-02-09T17:26:53.888 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fca5f9a1:::benchmark_data_smithi179_77138_object125424:head# 2024-02-09T17:26:53.888 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fca613f1:::benchmark_data_smithi179_77138_object106618:head# 2024-02-09T17:26:53.889 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fca65cde:::benchmark_data_smithi179_77138_object108105:head# 2024-02-09T17:26:53.889 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fca6b4d5:::benchmark_data_smithi179_77138_object87756:head# 2024-02-09T17:26:53.892 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fca6db61:::benchmark_data_smithi179_77138_object43246:head# 2024-02-09T17:26:53.893 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fca6e4d0:::benchmark_data_smithi179_77138_object45175:head# 2024-02-09T17:26:53.893 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fca6f5d3:::benchmark_data_smithi179_77138_object75677:head# 2024-02-09T17:26:53.893 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fca72f12:::benchmark_data_smithi179_77138_object27052:head# 2024-02-09T17:26:53.894 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fca7f635:::benchmark_data_smithi179_77138_object102829:head# 2024-02-09T17:26:53.894 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fca8b82d:::benchmark_data_smithi179_77138_object126218:head# 2024-02-09T17:26:53.895 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fca928f3:::benchmark_data_smithi179_77138_object117337:head# 2024-02-09T17:26:53.896 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fca95771:::benchmark_data_smithi179_77138_object125581:head# 2024-02-09T17:26:53.896 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fca9b0b0:::benchmark_data_smithi179_77138_object49211:head# 2024-02-09T17:26:53.897 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fca9e957:::benchmark_data_smithi179_77138_object103364:head# 2024-02-09T17:26:53.897 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcaa3b42:::benchmark_data_smithi179_77138_object39969:head# 2024-02-09T17:26:53.897 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcaa8fce:::benchmark_data_smithi179_77138_object29971:head# 2024-02-09T17:26:53.898 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcab355d:::benchmark_data_smithi179_77138_object120309:head# 2024-02-09T17:26:53.901 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcab854e:::benchmark_data_smithi179_77138_object108951:head# 2024-02-09T17:26:53.901 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcad3c13:::benchmark_data_smithi179_77138_object120084:head# 2024-02-09T17:26:53.901 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcad698c:::benchmark_data_smithi179_77138_object62705:head# 2024-02-09T17:26:53.904 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcad6fd0:::benchmark_data_smithi179_77138_object19941:head# 2024-02-09T17:26:53.905 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcad883e:::benchmark_data_smithi179_77138_object127751:head# 2024-02-09T17:26:53.905 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcae11ab:::benchmark_data_smithi179_77138_object110256:head# 2024-02-09T17:26:53.906 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcafffc4:::benchmark_data_smithi179_77138_object109390:head# 2024-02-09T17:26:53.906 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcb0bea1:::benchmark_data_smithi179_77138_object115510:head# 2024-02-09T17:26:53.907 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcb11918:::benchmark_data_smithi179_77138_object39067:head# 2024-02-09T17:26:53.907 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcb154e1:::benchmark_data_smithi179_77138_object105787:head# 2024-02-09T17:26:53.908 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcb1568a:::benchmark_data_smithi179_77138_object112711:head# 2024-02-09T17:26:53.912 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcb17ef0:::benchmark_data_smithi179_77138_object48389:head# 2024-02-09T17:26:53.912 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcb19f33:::benchmark_data_smithi179_77138_object103875:head# 2024-02-09T17:26:53.912 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcb1b49a:::benchmark_data_smithi179_77138_object45781:head# 2024-02-09T17:26:53.912 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcb486ab:::benchmark_data_smithi179_77138_object32606:head# 2024-02-09T17:26:53.912 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcb4ad62:::benchmark_data_smithi179_77138_object102789:head# 2024-02-09T17:26:53.912 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcb5dd03:::benchmark_data_smithi179_77138_object79126:head# 2024-02-09T17:26:53.913 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcb6549c:::benchmark_data_smithi179_77138_object92995:head# 2024-02-09T17:26:53.913 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcb65811:::benchmark_data_smithi179_77138_object49084:head# 2024-02-09T17:26:53.913 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcb76d7a:::benchmark_data_smithi179_77138_object113786:head# 2024-02-09T17:26:53.913 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcb7d130:::benchmark_data_smithi179_77138_object118523:head# 2024-02-09T17:26:53.913 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcb81a32:::benchmark_data_smithi179_77138_object114856:head# 2024-02-09T17:26:53.913 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcb826ed:::benchmark_data_smithi179_77138_object109230:head# 2024-02-09T17:26:53.913 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcb8ed7e:::benchmark_data_smithi179_77138_object68862:head# 2024-02-09T17:26:53.914 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcb9cfd0:::benchmark_data_smithi179_77138_object116482:head# 2024-02-09T17:26:53.914 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcba5b9b:::benchmark_data_smithi179_77138_object46780:head# 2024-02-09T17:26:53.915 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcba8f4f:::benchmark_data_smithi179_77138_object88780:head# 2024-02-09T17:26:53.915 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcbad4a7:::benchmark_data_smithi179_77138_object107344:head# 2024-02-09T17:26:53.916 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcbaf166:::benchmark_data_smithi179_77138_object61251:head# 2024-02-09T17:26:53.916 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcbb2b43:::benchmark_data_smithi179_77138_object92216:head# 2024-02-09T17:26:53.917 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcbb3a1d:::benchmark_data_smithi179_77138_object42550:head# 2024-02-09T17:26:53.917 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcbb98ef:::benchmark_data_smithi179_77138_object66746:head# 2024-02-09T17:26:53.917 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcbc5687:::benchmark_data_smithi179_77138_object78184:head# 2024-02-09T17:26:53.918 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcbcdff5:::benchmark_data_smithi179_77138_object101628:head# 2024-02-09T17:26:53.918 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcbd2e42:::benchmark_data_smithi179_77138_object63120:head# 2024-02-09T17:26:53.919 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcbd4385:::benchmark_data_smithi179_77138_object72105:head# 2024-02-09T17:26:53.919 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcbd7000:::benchmark_data_smithi179_77138_object74776:head# 2024-02-09T17:26:53.920 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcbddddd:::benchmark_data_smithi179_77138_object67522:head# 2024-02-09T17:26:53.920 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcbe06e5:::benchmark_data_smithi179_77138_object39111:head# 2024-02-09T17:26:53.921 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcbff5db:::benchmark_data_smithi179_77138_object102872:head# 2024-02-09T17:26:53.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:53 smithi179 ceph-mon[43975]: pgmap v453: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 6.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 881 op/s; 26608/318045 objects degraded (8.366%); 0 B/s, 0 objects/s recovering 2024-02-09T17:26:53.921 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcc02088:::benchmark_data_smithi179_77138_object23217:head# 2024-02-09T17:26:53.922 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcc04f9a:::benchmark_data_smithi179_77138_object104887:head# 2024-02-09T17:26:53.922 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcc058f0:::benchmark_data_smithi179_77138_object112209:head# 2024-02-09T17:26:53.923 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcc0d346:::benchmark_data_smithi179_77138_object72442:head# 2024-02-09T17:26:53.923 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcc0dd08:::benchmark_data_smithi179_77138_object31160:head# 2024-02-09T17:26:53.924 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcc113c1:::benchmark_data_smithi179_77138_object50685:head# 2024-02-09T17:26:53.924 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcc11e44:::benchmark_data_smithi179_77138_object29900:head# 2024-02-09T17:26:53.924 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcc19077:::benchmark_data_smithi179_77138_object123311:head# 2024-02-09T17:26:53.925 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcc2e243:::benchmark_data_smithi179_77138_object63750:head# 2024-02-09T17:26:53.925 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcc385ac:::benchmark_data_smithi179_77138_object64231:head# 2024-02-09T17:26:53.926 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcc41078:::benchmark_data_smithi179_77138_object71614:head# 2024-02-09T17:26:53.926 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcc71ebc:::benchmark_data_smithi179_77138_object44874:head# 2024-02-09T17:26:53.926 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcc745c9:::benchmark_data_smithi179_77138_object124747:head# 2024-02-09T17:26:53.927 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcc7660b:::benchmark_data_smithi179_77138_object123203:head# 2024-02-09T17:26:53.927 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcc7d16b:::benchmark_data_smithi179_77138_object102101:head# 2024-02-09T17:26:53.928 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcc90d3c:::benchmark_data_smithi179_77138_object87242:head# 2024-02-09T17:26:53.928 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcca38cf:::benchmark_data_smithi179_77138_object107189:head# 2024-02-09T17:26:53.929 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fccaac70:::benchmark_data_smithi179_77138_object101512:head# 2024-02-09T17:26:53.929 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fccaf54b:::benchmark_data_smithi179_77138_object77276:head# 2024-02-09T17:26:53.930 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fccc0ef0:::benchmark_data_smithi179_77138_object89351:head# 2024-02-09T17:26:53.930 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fccc2635:::benchmark_data_smithi179_77138_object82098:head# 2024-02-09T17:26:53.931 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fccc4ee4:::benchmark_data_smithi179_77138_object37930:head# 2024-02-09T17:26:53.931 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fccd5e5a:::benchmark_data_smithi179_77138_object105174:head# 2024-02-09T17:26:53.931 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fccdc4cb:::benchmark_data_smithi179_77138_object22290:head# 2024-02-09T17:26:53.932 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcce1960:::benchmark_data_smithi179_77138_object40397:head# 2024-02-09T17:26:53.932 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fccead8b:::benchmark_data_smithi179_77138_object86799:head# 2024-02-09T17:26:53.933 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fccf76dd:::benchmark_data_smithi179_77138_object55068:head# 2024-02-09T17:26:53.933 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcd00dda:::benchmark_data_smithi179_77138_object21760:head# 2024-02-09T17:26:53.933 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcd07f1a:::benchmark_data_smithi179_77138_object51669:head# 2024-02-09T17:26:53.934 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcd0d964:::benchmark_data_smithi179_77138_object112469:head# 2024-02-09T17:26:53.934 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcd0ed02:::benchmark_data_smithi179_77138_object53705:head# 2024-02-09T17:26:53.934 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcd140d4:::benchmark_data_smithi179_77138_object107410:head# 2024-02-09T17:26:53.935 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcd142dd:::benchmark_data_smithi179_77138_object126628:head# 2024-02-09T17:26:53.935 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcd146bd:::benchmark_data_smithi179_77138_object50236:head# 2024-02-09T17:26:53.936 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcd19de3:::benchmark_data_smithi179_77138_object66051:head# 2024-02-09T17:26:53.936 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:26:53.937 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcd354c7:::benchmark_data_smithi179_77138_object66667:head# 2024-02-09T17:26:53.937 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcd3fe39:::benchmark_data_smithi179_77138_object80895:head# 2024-02-09T17:26:53.937 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcd5600c:::benchmark_data_smithi179_77138_object64256:head# 2024-02-09T17:26:53.938 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcd6787e:::benchmark_data_smithi179_77138_object63028:head# 2024-02-09T17:26:53.938 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcd72275:::benchmark_data_smithi179_77138_object20048:head# 2024-02-09T17:26:53.938 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcd8e6cc:::benchmark_data_smithi179_77138_object63355:head# 2024-02-09T17:26:53.939 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcd90cf2:::benchmark_data_smithi179_77138_object95982:head# 2024-02-09T17:26:53.939 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcda1ca8:::benchmark_data_smithi179_77138_object25730:head# 2024-02-09T17:26:53.939 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcdb5aaf:::benchmark_data_smithi179_77138_object131103:head# 2024-02-09T17:26:53.940 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcdb93ab:::benchmark_data_smithi179_77138_object94517:head# 2024-02-09T17:26:53.941 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcdbd126:::benchmark_data_smithi179_77138_object101074:head# 2024-02-09T17:26:53.941 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcdbf6fd:::benchmark_data_smithi179_77138_object64307:head# 2024-02-09T17:26:53.941 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcdc4dcc:::benchmark_data_smithi179_77138_object126901:head# 2024-02-09T17:26:53.941 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcde87da:::benchmark_data_smithi179_77138_object114637:head# 2024-02-09T17:26:53.942 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcdeeced:::benchmark_data_smithi179_77138_object102091:head# 2024-02-09T17:26:53.942 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcdf4b7f:::benchmark_data_smithi179_77138_object52868:head# 2024-02-09T17:26:53.943 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fce01541:::benchmark_data_smithi179_77138_object50228:head# 2024-02-09T17:26:53.943 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fce06030:::benchmark_data_smithi179_77138_object117591:head# 2024-02-09T17:26:53.943 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fce06548:::benchmark_data_smithi179_77138_object84633:head# 2024-02-09T17:26:53.944 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fce0c9a1:::benchmark_data_smithi179_77138_object25999:head# 2024-02-09T17:26:53.946 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fce204e4:::benchmark_data_smithi179_77138_object19890:head# 2024-02-09T17:26:53.946 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fce25882:::benchmark_data_smithi179_77138_object131332:head# 2024-02-09T17:26:53.946 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fce2cd60:::benchmark_data_smithi179_77138_object49981:head# 2024-02-09T17:26:53.946 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fce346f3:::benchmark_data_smithi179_77138_object60964:head# 2024-02-09T17:26:53.946 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fce35cdc:::benchmark_data_smithi179_77138_object75805:head# 2024-02-09T17:26:53.947 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fce3c79d:::benchmark_data_smithi179_77138_object59932:head# 2024-02-09T17:26:53.947 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fce3e961:::benchmark_data_smithi179_77138_object108208:head# 2024-02-09T17:26:53.947 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fce3f02b:::benchmark_data_smithi179_77138_object21661:head# 2024-02-09T17:26:53.948 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fce43084:::benchmark_data_smithi179_77138_object94062:head# 2024-02-09T17:26:53.948 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fce45ac7:::benchmark_data_smithi179_77138_object103540:head# 2024-02-09T17:26:53.949 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fce4c963:::benchmark_data_smithi179_77138_object45347:head# 2024-02-09T17:26:53.949 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fce4f632:::benchmark_data_smithi179_77138_object40823:head# 2024-02-09T17:26:53.949 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fce60c33:::benchmark_data_smithi179_77138_object98750:head# 2024-02-09T17:26:53.950 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fce6225f:::benchmark_data_smithi179_77138_object114476:head# 2024-02-09T17:26:53.950 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fce646c3:::benchmark_data_smithi179_77138_object35308:head# 2024-02-09T17:26:53.950 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fce6ba16:::benchmark_data_smithi179_77138_object42466:head# 2024-02-09T17:26:53.951 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fce79031:::benchmark_data_smithi179_77138_object88267:head# 2024-02-09T17:26:53.951 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fce7b3b8:::benchmark_data_smithi179_77138_object93563:head# 2024-02-09T17:26:53.952 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fce7d400:::benchmark_data_smithi179_77138_object82348:head# 2024-02-09T17:26:53.952 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fce7e01d:::benchmark_data_smithi179_77138_object32873:head# 2024-02-09T17:26:53.953 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fce7e341:::benchmark_data_smithi179_77138_object62673:head# 2024-02-09T17:26:53.954 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fce814f1:::benchmark_data_smithi179_77138_object132041:head# 2024-02-09T17:26:53.954 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fce817c3:::benchmark_data_smithi179_77138_object68682:head# 2024-02-09T17:26:53.954 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fce84dc6:::benchmark_data_smithi179_77138_object91182:head# 2024-02-09T17:26:53.954 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fce86bea:::benchmark_data_smithi179_77138_object57937:head# 2024-02-09T17:26:53.955 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fce90555:::benchmark_data_smithi179_77138_object103096:head# 2024-02-09T17:26:53.955 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fce9344a:::benchmark_data_smithi179_77138_object119084:head# 2024-02-09T17:26:53.956 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fce94144:::benchmark_data_smithi179_77138_object34594:head# 2024-02-09T17:26:53.956 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcea2767:::benchmark_data_smithi179_77138_object65769:head# 2024-02-09T17:26:53.956 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcea372f:::benchmark_data_smithi179_77138_object43531:head# 2024-02-09T17:26:53.957 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcea44a5:::benchmark_data_smithi179_77138_object81095:head# 2024-02-09T17:26:53.957 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcea5f4d:::benchmark_data_smithi179_77138_object74462:head# 2024-02-09T17:26:53.958 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcea7fc6:::benchmark_data_smithi179_77138_object94028:head# 2024-02-09T17:26:53.958 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fceab00b:::benchmark_data_smithi179_77138_object25745:head# 2024-02-09T17:26:53.959 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcead19d:::benchmark_data_smithi179_77138_object64255:head# 2024-02-09T17:26:53.959 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fceaf817:::benchmark_data_smithi179_77138_object22436:head# 2024-02-09T17:26:53.960 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fceb40fc:::benchmark_data_smithi179_77138_object41865:head# 2024-02-09T17:26:53.960 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcebc719:::benchmark_data_smithi179_77138_object53833:head# 2024-02-09T17:26:53.960 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcec12e7:::benchmark_data_smithi179_77138_object123671:head# 2024-02-09T17:26:53.961 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fced2d2e:::benchmark_data_smithi179_77138_object64086:head# 2024-02-09T17:26:53.962 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcee9a97:::benchmark_data_smithi179_77138_object70681:head# 2024-02-09T17:26:53.962 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcefce09:::benchmark_data_smithi179_77138_object52589:head# 2024-02-09T17:26:53.962 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fceff29b:::benchmark_data_smithi179_77138_object108754:head# 2024-02-09T17:26:53.963 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcf03946:::benchmark_data_smithi179_77138_object81817:head# 2024-02-09T17:26:53.963 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcf03b61:::benchmark_data_smithi179_77138_object119603:head# 2024-02-09T17:26:53.963 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcf367ab:::benchmark_data_smithi179_77138_object73131:head# 2024-02-09T17:26:53.964 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcf3b211:::benchmark_data_smithi179_77138_object42146:head# 2024-02-09T17:26:53.964 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcf3eece:::benchmark_data_smithi179_77138_object112665:head# 2024-02-09T17:26:53.965 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcf4ee2e:::benchmark_data_smithi179_77138_object85828:head# 2024-02-09T17:26:53.965 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcf5a44a:::benchmark_data_smithi179_77138_object125720:head# 2024-02-09T17:26:53.966 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcf6184b:::benchmark_data_smithi179_77138_object125938:head# 2024-02-09T17:26:53.966 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcf627c9:::benchmark_data_smithi179_77138_object60911:head# 2024-02-09T17:26:53.966 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcf65643:::benchmark_data_smithi179_77138_object24576:head# 2024-02-09T17:26:53.968 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcf70721:::benchmark_data_smithi179_77138_object30740:head# 2024-02-09T17:26:53.968 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcf78d06:::benchmark_data_smithi179_77138_object61458:head# 2024-02-09T17:26:53.968 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcf80861:::benchmark_data_smithi179_77138_object124447:head# 2024-02-09T17:26:53.968 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcf8194d:::benchmark_data_smithi179_77138_object106739:head# 2024-02-09T17:26:53.969 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcfb0f7f:::benchmark_data_smithi179_77138_object29230:head# 2024-02-09T17:26:53.969 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcfb2710:::benchmark_data_smithi179_77138_object111435:head# 2024-02-09T17:26:53.969 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcfb4680:::benchmark_data_smithi179_77138_object102949:head# 2024-02-09T17:26:53.970 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcfbd511:::benchmark_data_smithi179_77138_object127929:head# 2024-02-09T17:26:53.970 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcfc4401:::benchmark_data_smithi179_77138_object81282:head# 2024-02-09T17:26:53.971 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcfccf88:::benchmark_data_smithi179_77138_object122907:head# 2024-02-09T17:26:53.971 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcfd9f92:::benchmark_data_smithi179_77138_object59224:head# 2024-02-09T17:26:53.972 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcfda0c1:::benchmark_data_smithi179_77138_object52968:head# 2024-02-09T17:26:53.974 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fcff3905:::benchmark_data_smithi179_77138_object116336:head# 2024-02-09T17:26:53.974 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd00593a:::benchmark_data_smithi179_77138_object24542:head# 2024-02-09T17:26:53.974 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd006b9a:::benchmark_data_smithi179_77138_object68225:head# 2024-02-09T17:26:53.974 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd017bae:::benchmark_data_smithi179_77138_object88591:head# 2024-02-09T17:26:53.974 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd0244e3:::benchmark_data_smithi179_77138_object79911:head# 2024-02-09T17:26:53.975 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd02e621:::benchmark_data_smithi179_77138_object82837:head# 2024-02-09T17:26:53.975 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd037a65:::benchmark_data_smithi179_77138_object50878:head# 2024-02-09T17:26:53.975 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd03f5b4:::benchmark_data_smithi179_77138_object33429:head# 2024-02-09T17:26:53.978 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd045931:::benchmark_data_smithi179_77138_object117172:head# 2024-02-09T17:26:53.978 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd051f69:::benchmark_data_smithi179_77138_object67116:head# 2024-02-09T17:26:53.978 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd05eeaa:::benchmark_data_smithi179_77138_object83053:head# 2024-02-09T17:26:53.978 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd0642f8:::benchmark_data_smithi179_77138_object88851:head# 2024-02-09T17:26:53.978 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd07aa62:::benchmark_data_smithi179_77138_object116943:head# 2024-02-09T17:26:53.981 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd09b0e3:::benchmark_data_smithi179_77138_object106950:head# 2024-02-09T17:26:53.981 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd09dd58:::benchmark_data_smithi179_77138_object105235:head# 2024-02-09T17:26:53.981 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd0adf5e:::benchmark_data_smithi179_77138_object76886:head# 2024-02-09T17:26:53.981 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd0ae0cd:::benchmark_data_smithi179_77138_object105984:head# 2024-02-09T17:26:53.981 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd0b448f:::benchmark_data_smithi179_77138_object78476:head# 2024-02-09T17:26:53.981 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd0d9e53:::benchmark_data_smithi179_77138_object99859:head# 2024-02-09T17:26:53.981 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd0de44d:::benchmark_data_smithi179_77138_object38265:head# 2024-02-09T17:26:53.981 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd0e39b9:::benchmark_data_smithi179_77138_object62065:head# 2024-02-09T17:26:53.982 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd0f51af:::benchmark_data_smithi179_77138_object20650:head# 2024-02-09T17:26:53.982 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd0f7526:::benchmark_data_smithi179_77138_object82915:head# 2024-02-09T17:26:53.982 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd0fd7c1:::benchmark_data_smithi179_77138_object110573:head# 2024-02-09T17:26:53.983 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd102e4f:::benchmark_data_smithi179_77138_object83187:head# 2024-02-09T17:26:53.983 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd11060f:::benchmark_data_smithi179_77138_object58739:head# 2024-02-09T17:26:53.983 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd11721f:::benchmark_data_smithi179_77138_object68202:head# 2024-02-09T17:26:53.984 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd118823:::benchmark_data_smithi179_77138_object95574:head# 2024-02-09T17:26:53.984 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd11c7f4:::benchmark_data_smithi179_77138_object63639:head# 2024-02-09T17:26:53.985 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd12575e:::benchmark_data_smithi179_77138_object41069:head# 2024-02-09T17:26:53.985 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd13c9c6:::benchmark_data_smithi179_77138_object100705:head# 2024-02-09T17:26:53.986 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd13d8bf:::benchmark_data_smithi179_77138_object40844:head# 2024-02-09T17:26:53.986 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd1457d1:::benchmark_data_smithi179_77138_object58641:head# 2024-02-09T17:26:53.987 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd1474fc:::benchmark_data_smithi179_77138_object54800:head# 2024-02-09T17:26:53.987 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd14c95c:::benchmark_data_smithi179_77138_object107407:head# 2024-02-09T17:26:53.988 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd1561b6:::benchmark_data_smithi179_77138_object21424:head# 2024-02-09T17:26:53.988 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd163b13:::benchmark_data_smithi179_77138_object43316:head# 2024-02-09T17:26:53.991 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd167ff5:::benchmark_data_smithi179_77138_object104434:head# 2024-02-09T17:26:53.991 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd16ff2e:::benchmark_data_smithi179_77138_object51173:head# 2024-02-09T17:26:53.991 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd17336a:::benchmark_data_smithi179_77138_object72696:head# 2024-02-09T17:26:53.993 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd180a59:::benchmark_data_smithi179_77138_object103265:head# 2024-02-09T17:26:53.993 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd181092:::benchmark_data_smithi179_77138_object81577:head# 2024-02-09T17:26:53.993 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd18f40d:::benchmark_data_smithi179_77138_object66888:head# 2024-02-09T17:26:53.993 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd198575:::benchmark_data_smithi179_77138_object119221:head# 2024-02-09T17:26:53.993 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd19900f:::benchmark_data_smithi179_77138_object96414:head# 2024-02-09T17:26:53.995 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd1a0411:::benchmark_data_smithi179_77138_object23440:head# 2024-02-09T17:26:53.995 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd1bf345:::benchmark_data_smithi179_77138_object59979:head# 2024-02-09T17:26:53.995 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd1c115d:::benchmark_data_smithi179_77138_object63168:head# 2024-02-09T17:26:53.995 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd1e0a7d:::benchmark_data_smithi179_77138_object83706:head# 2024-02-09T17:26:53.996 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd1e629c:::benchmark_data_smithi179_77138_object101209:head# 2024-02-09T17:26:53.996 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd1e7492:::benchmark_data_smithi179_77138_object91767:head# 2024-02-09T17:26:53.996 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd1fd1fa:::benchmark_data_smithi179_77138_object95615:head# 2024-02-09T17:26:53.996 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd218245:::benchmark_data_smithi179_77138_object68608:head# 2024-02-09T17:26:53.998 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd2241b8:::benchmark_data_smithi179_77138_object87993:head# 2024-02-09T17:26:53.998 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd23beaf:::benchmark_data_smithi179_77138_object40828:head# 2024-02-09T17:26:53.998 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd23cc36:::benchmark_data_smithi179_77138_object127545:head# 2024-02-09T17:26:53.998 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd250f35:::benchmark_data_smithi179_77138_object101085:head# 2024-02-09T17:26:53.999 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:54.005 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd26416e:::benchmark_data_smithi179_77138_object114830:head# 2024-02-09T17:26:54.005 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd2652ea:::benchmark_data_smithi179_77138_object33084:head# 2024-02-09T17:26:54.005 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd267c17:::benchmark_data_smithi179_77138_object34638:head# 2024-02-09T17:26:54.005 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd26b35e:::benchmark_data_smithi179_77138_object90286:head# 2024-02-09T17:26:54.005 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd26c355:::benchmark_data_smithi179_77138_object48443:head# 2024-02-09T17:26:54.006 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd282fa3:::benchmark_data_smithi179_77138_object97875:head# 2024-02-09T17:26:54.006 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd286524:::benchmark_data_smithi179_77138_object37345:head# 2024-02-09T17:26:54.006 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd28dd31:::benchmark_data_smithi179_77138_object68944:head# 2024-02-09T17:26:54.007 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd298d04:::benchmark_data_smithi179_77138_object20530:head# 2024-02-09T17:26:54.007 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd29b3af:::benchmark_data_smithi179_77138_object116877:head# 2024-02-09T17:26:54.007 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd2a3756:::benchmark_data_smithi179_77138_object58101:head# 2024-02-09T17:26:54.009 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd2afa3a:::benchmark_data_smithi179_77138_object88020:head# 2024-02-09T17:26:54.009 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd2b206e:::benchmark_data_smithi179_77138_object103455:head# 2024-02-09T17:26:54.009 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd2b29a1:::benchmark_data_smithi179_77138_object29730:head# 2024-02-09T17:26:54.009 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd2bb687:::benchmark_data_smithi179_77138_object44862:head# 2024-02-09T17:26:54.009 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd2c73a2:::benchmark_data_smithi179_77138_object102903:head# 2024-02-09T17:26:54.010 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd2d7f6b:::benchmark_data_smithi179_77138_object67572:head# 2024-02-09T17:26:54.010 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd2e4416:::benchmark_data_smithi179_77138_object106815:head# 2024-02-09T17:26:54.011 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd2e939d:::benchmark_data_smithi179_77138_object101366:head# 2024-02-09T17:26:54.011 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd2e956a:::benchmark_data_smithi179_77138_object65104:head# 2024-02-09T17:26:54.011 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd2ec134:::benchmark_data_smithi179_77138_object101865:head# 2024-02-09T17:26:54.012 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd308d29:::benchmark_data_smithi179_77138_object122727:head# 2024-02-09T17:26:54.012 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd30b894:::benchmark_data_smithi179_77138_object25291:head# 2024-02-09T17:26:54.013 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd3123b4:::benchmark_data_smithi179_77138_object87323:head# 2024-02-09T17:26:54.013 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd31328d:::benchmark_data_smithi179_77138_object76966:head# 2024-02-09T17:26:54.016 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd3166ee:::benchmark_data_smithi179_77138_object33738:head# 2024-02-09T17:26:54.024 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd33e5a2:::benchmark_data_smithi179_77138_object112560:head# 2024-02-09T17:26:54.024 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd356353:::benchmark_data_smithi179_77138_object26368:head# 2024-02-09T17:26:54.025 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd359499:::benchmark_data_smithi179_77138_object122085:head# 2024-02-09T17:26:54.025 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd370770:::benchmark_data_smithi179_77138_object64371:head# 2024-02-09T17:26:54.027 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd37afc1:::benchmark_data_smithi179_77138_object26623:head# 2024-02-09T17:26:54.027 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd3839e1:::benchmark_data_smithi179_77138_object47809:head# 2024-02-09T17:26:54.027 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd384496:::benchmark_data_smithi179_77138_object76536:head# 2024-02-09T17:26:54.028 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd394c9d:::benchmark_data_smithi179_77138_object45021:head# 2024-02-09T17:26:54.028 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd3a57a2:::benchmark_data_smithi179_77138_object101172:head# 2024-02-09T17:26:54.029 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd3a8eba:::benchmark_data_smithi179_77138_object22310:head# 2024-02-09T17:26:54.029 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd3c75cf:::benchmark_data_smithi179_77138_object40756:head# 2024-02-09T17:26:54.031 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd3cdf78:::benchmark_data_smithi179_77138_object39206:head# 2024-02-09T17:26:54.035 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd3d043b:::benchmark_data_smithi179_77138_object52567:head# 2024-02-09T17:26:54.035 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd3d712e:::benchmark_data_smithi179_77138_object77877:head# 2024-02-09T17:26:54.035 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd3dcd6c:::benchmark_data_smithi179_77138_object61460:head# 2024-02-09T17:26:54.035 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd3e263f:::benchmark_data_smithi179_77138_object62218:head# 2024-02-09T17:26:54.035 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd3eafb8:::benchmark_data_smithi179_77138_object87492:head# 2024-02-09T17:26:54.036 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd3f0d10:::benchmark_data_smithi179_77138_object42489:head# 2024-02-09T17:26:54.036 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd3f5724:::benchmark_data_smithi179_77138_object97010:head# 2024-02-09T17:26:54.036 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd3fcf96:::benchmark_data_smithi179_77138_object92919:head# 2024-02-09T17:26:54.037 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd402262:::benchmark_data_smithi179_77138_object60602:head# 2024-02-09T17:26:54.037 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd40583c:::benchmark_data_smithi179_77138_object94057:head# 2024-02-09T17:26:54.038 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd405ee5:::benchmark_data_smithi179_77138_object27362:head# 2024-02-09T17:26:54.038 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd42287a:::benchmark_data_smithi179_77138_object22494:head# 2024-02-09T17:26:54.039 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd42f86e:::benchmark_data_smithi179_77138_object79716:head# 2024-02-09T17:26:54.039 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd432a0b:::benchmark_data_smithi179_77138_object130801:head# 2024-02-09T17:26:54.040 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd436964:::benchmark_data_smithi179_77138_object90454:head# 2024-02-09T17:26:54.040 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd438514:::benchmark_data_smithi179_77138_object38920:head# 2024-02-09T17:26:54.040 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd439323:::benchmark_data_smithi179_77138_object86880:head# 2024-02-09T17:26:54.042 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd450b13:::benchmark_data_smithi179_77138_object72321:head# 2024-02-09T17:26:54.042 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd46b954:::benchmark_data_smithi179_77138_object96384:head# 2024-02-09T17:26:54.042 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd479234:::benchmark_data_smithi179_77138_object130488:head# 2024-02-09T17:26:54.042 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd47b2d7:::benchmark_data_smithi179_77138_object129246:head# 2024-02-09T17:26:54.043 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd47f8d8:::benchmark_data_smithi179_77138_object21220:head# 2024-02-09T17:26:54.043 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd483aed:::benchmark_data_smithi179_77138_object52711:head# 2024-02-09T17:26:54.044 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd4960bb:::benchmark_data_smithi179_77138_object118336:head# 2024-02-09T17:26:54.044 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd4971ce:::benchmark_data_smithi179_77138_object118852:head# 2024-02-09T17:26:54.045 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd49910f:::benchmark_data_smithi179_77138_object72934:head# 2024-02-09T17:26:54.045 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd4999cf:::benchmark_data_smithi179_77138_object112621:head# 2024-02-09T17:26:54.046 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd4a9fa5:::benchmark_data_smithi179_77138_object36862:head# 2024-02-09T17:26:54.046 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd4b794b:::benchmark_data_smithi179_77138_object73020:head# 2024-02-09T17:26:54.047 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd4b941b:::benchmark_data_smithi179_77138_object87088:head# 2024-02-09T17:26:54.047 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd4ba38a:::benchmark_data_smithi179_77138_object124449:head# 2024-02-09T17:26:54.048 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd4badda:::benchmark_data_smithi179_77138_object92855:head# 2024-02-09T17:26:54.048 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd4bed0c:::benchmark_data_smithi179_77138_object105937:head# 2024-02-09T17:26:54.048 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd4c46fc:::benchmark_data_smithi179_77138_object28565:head# 2024-02-09T17:26:54.048 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd4c5333:::benchmark_data_smithi179_77138_object107349:head# 2024-02-09T17:26:54.049 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd4c7e50:::benchmark_data_smithi179_77138_object20104:head# 2024-02-09T17:26:54.049 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd4d8e84:::benchmark_data_smithi179_77138_object90256:head# 2024-02-09T17:26:54.050 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd4da2e5:::benchmark_data_smithi179_77138_object107975:head# 2024-02-09T17:26:54.050 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd4dd76e:::benchmark_data_smithi179_77138_object38588:head# 2024-02-09T17:26:54.050 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd4dfe7e:::benchmark_data_smithi179_77138_object58215:head# 2024-02-09T17:26:54.051 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd4e734c:::benchmark_data_smithi179_77138_object43307:head# 2024-02-09T17:26:54.051 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd4f117f:::benchmark_data_smithi179_77138_object124841:head# 2024-02-09T17:26:54.052 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd4f5403:::benchmark_data_smithi179_77138_object99722:head# 2024-02-09T17:26:54.052 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd50c40e:::benchmark_data_smithi179_77138_object40499:head# 2024-02-09T17:26:54.052 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd50c681:::benchmark_data_smithi179_77138_object60136:head# 2024-02-09T17:26:54.053 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd50c781:::benchmark_data_smithi179_77138_object25423:head# 2024-02-09T17:26:54.053 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd5153ee:::benchmark_data_smithi179_77138_object106526:head# 2024-02-09T17:26:54.054 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd524f50:::benchmark_data_smithi179_77138_object76946:head# 2024-02-09T17:26:54.054 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd5262ea:::benchmark_data_smithi179_77138_object79796:head# 2024-02-09T17:26:54.055 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd532d14:::benchmark_data_smithi179_77138_object82326:head# 2024-02-09T17:26:54.055 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd53f460:::benchmark_data_smithi179_77138_object129684:head# 2024-02-09T17:26:54.056 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd5687c7:::benchmark_data_smithi179_77138_object53882:head# 2024-02-09T17:26:54.056 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd569e17:::benchmark_data_smithi179_77138_object112143:head# 2024-02-09T17:26:54.056 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd58d23b:::benchmark_data_smithi179_77138_object99927:head# 2024-02-09T17:26:54.057 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd592b10:::benchmark_data_smithi179_77138_object31688:head# 2024-02-09T17:26:54.057 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd596b2f:::benchmark_data_smithi179_77138_object109801:head# 2024-02-09T17:26:54.058 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd59c346:::benchmark_data_smithi179_77138_object96323:head# 2024-02-09T17:26:54.058 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd59d4cc:::benchmark_data_smithi179_77138_object107978:head# 2024-02-09T17:26:54.058 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd5ac776:::benchmark_data_smithi179_77138_object96653:head# 2024-02-09T17:26:54.059 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd5b4520:::benchmark_data_smithi179_77138_object25283:head# 2024-02-09T17:26:54.063 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd5b98c5:::benchmark_data_smithi179_77138_object41638:head# 2024-02-09T17:26:54.063 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd5c723a:::benchmark_data_smithi179_77138_object58230:head# 2024-02-09T17:26:54.063 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd5c7622:::benchmark_data_smithi179_77138_object78049:head# 2024-02-09T17:26:54.063 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd5ca3f8:::benchmark_data_smithi179_77138_object68056:head# 2024-02-09T17:26:54.064 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd5d1bb2:::benchmark_data_smithi179_77138_object85024:head# 2024-02-09T17:26:54.064 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd5d535a:::benchmark_data_smithi179_77138_object112051:head# 2024-02-09T17:26:54.066 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd5efa4e:::benchmark_data_smithi179_77138_object61455:head# 2024-02-09T17:26:54.066 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd6058b7:::benchmark_data_smithi179_77138_object34061:head# 2024-02-09T17:26:54.066 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd60f59c:::benchmark_data_smithi179_77138_object132997:head# 2024-02-09T17:26:54.067 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd63ea85:::benchmark_data_smithi179_77138_object127997:head# 2024-02-09T17:26:54.067 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd6512ce:::benchmark_data_smithi179_77138_object25522:head# 2024-02-09T17:26:54.068 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd659421:::benchmark_data_smithi179_77138_object121729:head# 2024-02-09T17:26:54.068 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd675d4f:::benchmark_data_smithi179_77138_object103595:head# 2024-02-09T17:26:54.069 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd67de47:::benchmark_data_smithi179_77138_object112597:head# 2024-02-09T17:26:54.069 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd680c42:::benchmark_data_smithi179_77138_object69017:head# 2024-02-09T17:26:54.070 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd6843f2:::benchmark_data_smithi179_77138_object102000:head# 2024-02-09T17:26:54.073 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd684ee5:::benchmark_data_smithi179_77138_object52536:head# 2024-02-09T17:26:54.073 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd68900a:::benchmark_data_smithi179_77138_object123221:head# 2024-02-09T17:26:54.073 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd691f49:::benchmark_data_smithi179_77138_object73778:head# 2024-02-09T17:26:54.073 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd6924f5:::benchmark_data_smithi179_77138_object28998:head# 2024-02-09T17:26:54.073 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd693150:::benchmark_data_smithi179_77138_object131012:head# 2024-02-09T17:26:54.073 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd697fe4:::benchmark_data_smithi179_77138_object39714:head# 2024-02-09T17:26:54.074 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd69a05e:::benchmark_data_smithi179_77138_object91586:head# 2024-02-09T17:26:54.074 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd69adbc:::benchmark_data_smithi179_77138_object68292:head# 2024-02-09T17:26:54.074 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd6a2c6e:::benchmark_data_smithi179_77138_object57137:head# 2024-02-09T17:26:54.075 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd6aaa6d:::benchmark_data_smithi179_77138_object50782:head# 2024-02-09T17:26:54.076 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd6b4992:::benchmark_data_smithi179_77138_object59825:head# 2024-02-09T17:26:54.076 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd6b4c1b:::benchmark_data_smithi179_77138_object62713:head# 2024-02-09T17:26:54.076 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd6c37ba:::benchmark_data_smithi179_77138_object19318:head# 2024-02-09T17:26:54.076 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd6c7759:::benchmark_data_smithi179_77138_object130834:head# 2024-02-09T17:26:54.076 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd6dc69b:::benchmark_data_smithi179_77138_object118502:head# 2024-02-09T17:26:54.079 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd6dde16:::benchmark_data_smithi179_77138_object92027:head# 2024-02-09T17:26:54.079 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd6e465a:::benchmark_data_smithi179_77138_object97176:head# 2024-02-09T17:26:54.080 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd6e58af:::benchmark_data_smithi179_77138_object100989:head# 2024-02-09T17:26:54.082 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd6e7fe7:::benchmark_data_smithi179_77138_object131725:head# 2024-02-09T17:26:54.082 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd6f0e21:::benchmark_data_smithi179_77138_object112674:head# 2024-02-09T17:26:54.083 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd700a0f:::benchmark_data_smithi179_77138_object93889:head# 2024-02-09T17:26:54.083 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd702b9b:::benchmark_data_smithi179_77138_object102376:head# 2024-02-09T17:26:54.084 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd7056f9:::benchmark_data_smithi179_77138_object32120:head# 2024-02-09T17:26:54.084 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd70baa1:::benchmark_data_smithi179_77138_object21588:head# 2024-02-09T17:26:54.084 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd70ef97:::benchmark_data_smithi179_77138_object66736:head# 2024-02-09T17:26:54.085 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd716fa2:::benchmark_data_smithi179_77138_object120183:head# 2024-02-09T17:26:54.085 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd71b0c0:::benchmark_data_smithi179_77138_object82399:head# 2024-02-09T17:26:54.086 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd72817a:::benchmark_data_smithi179_77138_object66701:head# 2024-02-09T17:26:54.086 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd72bdec:::benchmark_data_smithi179_77138_object20174:head# 2024-02-09T17:26:54.086 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd72e794:::benchmark_data_smithi179_77138_object112669:head# 2024-02-09T17:26:54.087 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd732b0b:::benchmark_data_smithi179_77138_object126345:head# 2024-02-09T17:26:54.087 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd74127b:::benchmark_data_smithi179_77138_object116371:head# 2024-02-09T17:26:54.088 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd7549c5:::benchmark_data_smithi179_77138_object79982:head# 2024-02-09T17:26:54.088 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd760b9a:::benchmark_data_smithi179_77138_object30566:head# 2024-02-09T17:26:54.089 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd761a83:::benchmark_data_smithi179_77138_object126848:head# 2024-02-09T17:26:54.089 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd761c7a:::benchmark_data_smithi179_77138_object49708:head# 2024-02-09T17:26:54.090 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd77a162:::benchmark_data_smithi179_77138_object66624:head# 2024-02-09T17:26:54.090 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd77f369:::benchmark_data_smithi179_77138_object85180:head# 2024-02-09T17:26:54.091 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd789d9e:::benchmark_data_smithi179_77138_object48546:head# 2024-02-09T17:26:54.091 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd7a5bc0:::benchmark_data_smithi179_77138_object132350:head# 2024-02-09T17:26:54.092 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd7ab41b:::benchmark_data_smithi179_77138_object89944:head# 2024-02-09T17:26:54.092 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd7b5558:::benchmark_data_smithi179_77138_object52185:head# 2024-02-09T17:26:54.092 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd7de3db:::benchmark_data_smithi179_77138_object109874:head# 2024-02-09T17:26:54.096 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd7ef964:::benchmark_data_smithi179_77138_object127411:head# 2024-02-09T17:26:54.096 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd7f08da:::benchmark_data_smithi179_77138_object85072:head# 2024-02-09T17:26:54.096 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd7f552a:::benchmark_data_smithi179_77138_object49471:head# 2024-02-09T17:26:54.096 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd8005d9:::benchmark_data_smithi179_77138_object45450:head# 2024-02-09T17:26:54.097 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd80abfa:::benchmark_data_smithi179_77138_object127032:head# 2024-02-09T17:26:54.098 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd80bfb7:::benchmark_data_smithi179_77138_object26704:head# 2024-02-09T17:26:54.098 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd810c67:::benchmark_data_smithi179_77138_object95863:head# 2024-02-09T17:26:54.098 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd818d57:::benchmark_data_smithi179_77138_object92671:head# 2024-02-09T17:26:54.099 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd81c6ef:::benchmark_data_smithi179_77138_object28944:head# 2024-02-09T17:26:54.099 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd826950:::benchmark_data_smithi179_77138_object66395:head# 2024-02-09T17:26:54.100 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd82711a:::benchmark_data_smithi179_77138_object91473:head# 2024-02-09T17:26:54.100 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd837022:::benchmark_data_smithi179_77138_object100031:head# 2024-02-09T17:26:54.100 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd8373f0:::benchmark_data_smithi179_77138_object37223:head# 2024-02-09T17:26:54.101 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd83f035:::benchmark_data_smithi179_77138_object91936:head# 2024-02-09T17:26:54.102 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd8414ef:::benchmark_data_smithi179_77138_object62197:head# 2024-02-09T17:26:54.103 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd84700a:::benchmark_data_smithi179_77138_object20479:head# 2024-02-09T17:26:54.103 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd8494e9:::benchmark_data_smithi179_77138_object21151:head# 2024-02-09T17:26:54.103 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd866d9f:::benchmark_data_smithi179_77138_object22305:head# 2024-02-09T17:26:54.103 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd8856cc:::benchmark_data_smithi179_77138_object85396:head# 2024-02-09T17:26:54.104 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd88a82a:::benchmark_data_smithi179_77138_object116752:head# 2024-02-09T17:26:54.106 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd88e60d:::benchmark_data_smithi179_77138_object65617:head# 2024-02-09T17:26:54.107 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd8957ff:::benchmark_data_smithi179_77138_object92992:head# 2024-02-09T17:26:54.107 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd896603:::benchmark_data_smithi179_77138_object65496:head# 2024-02-09T17:26:54.108 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd897602:::benchmark_data_smithi179_77138_object91757:head# 2024-02-09T17:26:54.108 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd8a573a:::benchmark_data_smithi179_77138_object107067:head# 2024-02-09T17:26:54.108 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd8ac443:::benchmark_data_smithi179_77138_object82282:head# 2024-02-09T17:26:54.109 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd8bc801:::benchmark_data_smithi179_77138_object120726:head# 2024-02-09T17:26:54.109 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd8c52c6:::benchmark_data_smithi179_77138_object43938:head# 2024-02-09T17:26:54.110 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd8c7fa4:::benchmark_data_smithi179_77138_object97907:head# 2024-02-09T17:26:54.110 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd8d08a2:::benchmark_data_smithi179_77138_object124671:head# 2024-02-09T17:26:54.113 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd8d1acd:::benchmark_data_smithi179_77138_object68025:head# 2024-02-09T17:26:54.113 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd8d666e:::benchmark_data_smithi179_77138_object75638:head# 2024-02-09T17:26:54.113 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd8dccf6:::benchmark_data_smithi179_77138_object19538:head# 2024-02-09T17:26:54.113 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd8e9c12:::benchmark_data_smithi179_77138_object98737:head# 2024-02-09T17:26:54.114 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd8f0f12:::benchmark_data_smithi179_77138_object72329:head# 2024-02-09T17:26:54.115 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd8f23a9:::benchmark_data_smithi179_77138_object26324:head# 2024-02-09T17:26:54.115 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd9136b8:::benchmark_data_smithi179_77138_object111977:head# 2024-02-09T17:26:54.116 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd915b7c:::benchmark_data_smithi179_77138_object63958:head# 2024-02-09T17:26:54.116 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd9161de:::benchmark_data_smithi179_77138_object65530:head# 2024-02-09T17:26:54.117 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd917c86:::benchmark_data_smithi179_77138_object51958:head# 2024-02-09T17:26:54.117 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd929981:::benchmark_data_smithi179_77138_object49165:head# 2024-02-09T17:26:54.118 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd935071:::benchmark_data_smithi179_77138_object57413:head# 2024-02-09T17:26:54.118 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd938743:::benchmark_data_smithi179_77138_object114198:head# 2024-02-09T17:26:54.119 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd93a1b6:::benchmark_data_smithi179_77138_object79775:head# 2024-02-09T17:26:54.119 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd952721:::benchmark_data_smithi179_77138_object95097:head# 2024-02-09T17:26:54.120 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd95583e:::benchmark_data_smithi179_77138_object58905:head# 2024-02-09T17:26:54.120 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd95e867:::benchmark_data_smithi179_77138_object123251:head# 2024-02-09T17:26:54.120 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd981701:::benchmark_data_smithi179_77138_object59721:head# 2024-02-09T17:26:54.121 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd988a42:::benchmark_data_smithi179_77138_object20327:head# 2024-02-09T17:26:54.121 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd98e70e:::benchmark_data_smithi179_77138_object95456:head# 2024-02-09T17:26:54.122 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd99814c:::benchmark_data_smithi179_77138_object45782:head# 2024-02-09T17:26:54.122 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd9a3694:::benchmark_data_smithi179_77138_object113805:head# 2024-02-09T17:26:54.123 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd9a9525:::benchmark_data_smithi179_77138_object102086:head# 2024-02-09T17:26:54.124 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd9af947:::benchmark_data_smithi179_77138_object105799:head# 2024-02-09T17:26:54.124 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd9bc94d:::benchmark_data_smithi179_77138_object76964:head# 2024-02-09T17:26:54.124 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd9e904c:::benchmark_data_smithi179_77138_object53051:head# 2024-02-09T17:26:54.125 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd9ec30e:::benchmark_data_smithi179_77138_object32212:head# 2024-02-09T17:26:54.125 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fd9f00c3:::benchmark_data_smithi179_77138_object39853:head# 2024-02-09T17:26:54.126 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fda26eb0:::benchmark_data_smithi179_77138_object97077:head# 2024-02-09T17:26:54.126 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fda291cd:::benchmark_data_smithi179_77138_object28006:head# 2024-02-09T17:26:54.126 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fda2b67c:::benchmark_data_smithi179_77138_object133048:head# 2024-02-09T17:26:54.127 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fda3725b:::benchmark_data_smithi179_77138_object24805:head# 2024-02-09T17:26:54.128 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fda394f2:::benchmark_data_smithi179_77138_object129419:head# 2024-02-09T17:26:54.129 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fda47e47:::benchmark_data_smithi179_77138_object101537:head# 2024-02-09T17:26:54.129 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fda4c309:::benchmark_data_smithi179_77138_object132834:head# 2024-02-09T17:26:54.130 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fda5390c:::benchmark_data_smithi179_77138_object63254:head# 2024-02-09T17:26:54.130 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fda705e6:::benchmark_data_smithi179_77138_object132187:head# 2024-02-09T17:26:54.130 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fda85216:::benchmark_data_smithi179_77138_object132877:head# 2024-02-09T17:26:54.131 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdaa667e:::benchmark_data_smithi179_77138_object98014:head# 2024-02-09T17:26:54.131 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdabd36b:::benchmark_data_smithi179_77138_object73867:head# 2024-02-09T17:26:54.133 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdac232c:::benchmark_data_smithi179_77138_object52568:head# 2024-02-09T17:26:54.134 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdacf794:::benchmark_data_smithi179_77138_object109507:head# 2024-02-09T17:26:54.134 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdad055d:::benchmark_data_smithi179_77138_object30692:head# 2024-02-09T17:26:54.135 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdad832f:::benchmark_data_smithi179_77138_object37527:head# 2024-02-09T17:26:54.135 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdad98ed:::benchmark_data_smithi179_77138_object131246:head# 2024-02-09T17:26:54.136 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdaf3431:::benchmark_data_smithi179_77138_object45714:head# 2024-02-09T17:26:54.136 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdaf78b8:::benchmark_data_smithi179_77138_object47659:head# 2024-02-09T17:26:54.137 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdafea3c:::benchmark_data_smithi179_77138_object83274:head# 2024-02-09T17:26:54.137 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdb00d97:::benchmark_data_smithi179_77138_object37106:head# 2024-02-09T17:26:54.137 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdb0c5b0:::benchmark_data_smithi179_77138_object131098:head# 2024-02-09T17:26:54.138 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdb38cb4:::benchmark_data_smithi179_77138_object124026:head# 2024-02-09T17:26:54.138 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdb3ae65:::benchmark_data_smithi179_77138_object80717:head# 2024-02-09T17:26:54.139 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdb3d79f:::benchmark_data_smithi179_77138_object70704:head# 2024-02-09T17:26:54.139 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdb50b6c:::benchmark_data_smithi179_77138_object103772:head# 2024-02-09T17:26:54.140 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdb51550:::benchmark_data_smithi179_77138_object46047:head# 2024-02-09T17:26:54.140 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdb64af2:::benchmark_data_smithi179_77138_object41144:head# 2024-02-09T17:26:54.140 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdb6a1b2:::benchmark_data_smithi179_77138_object92848:head# 2024-02-09T17:26:54.141 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdb74f9e:::benchmark_data_smithi179_77138_object79060:head# 2024-02-09T17:26:54.144 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdb775f0:::benchmark_data_smithi179_77138_object37536:head# 2024-02-09T17:26:54.144 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdb7a619:::benchmark_data_smithi179_77138_object100195:head# 2024-02-09T17:26:54.144 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdb7aaad:::benchmark_data_smithi179_77138_object47409:head# 2024-02-09T17:26:54.144 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdb803f7:::benchmark_data_smithi179_77138_object96075:head# 2024-02-09T17:26:54.144 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdb8d6ff:::benchmark_data_smithi179_77138_object48502:head# 2024-02-09T17:26:54.144 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdb91d00:::benchmark_data_smithi179_77138_object72885:head# 2024-02-09T17:26:54.144 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdb96c4f:::benchmark_data_smithi179_77138_object109033:head# 2024-02-09T17:26:54.145 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdb98319:::benchmark_data_smithi179_77138_object126010:head# 2024-02-09T17:26:54.146 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdb9f3d3:::benchmark_data_smithi179_77138_object73195:head# 2024-02-09T17:26:54.146 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdba39f5:::benchmark_data_smithi179_77138_object44581:head# 2024-02-09T17:26:54.146 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdba46d1:::benchmark_data_smithi179_77138_object88416:head# 2024-02-09T17:26:54.146 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdba5b42:::benchmark_data_smithi179_77138_object65196:head# 2024-02-09T17:26:54.146 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdba7b4e:::benchmark_data_smithi179_77138_object90723:head# 2024-02-09T17:26:54.147 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdba82aa:::benchmark_data_smithi179_77138_object120363:head# 2024-02-09T17:26:54.150 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdbb2171:::benchmark_data_smithi179_77138_object121016:head# 2024-02-09T17:26:54.150 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdbb6606:::benchmark_data_smithi179_77138_object93233:head# 2024-02-09T17:26:54.150 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdbc4156:::benchmark_data_smithi179_77138_object81587:head# 2024-02-09T17:26:54.151 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdbc8035:::benchmark_data_smithi179_77138_object33190:head# 2024-02-09T17:26:54.151 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdbd158a:::benchmark_data_smithi179_77138_object83527:head# 2024-02-09T17:26:54.152 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdbd6a45:::benchmark_data_smithi179_77138_object35738:head# 2024-02-09T17:26:54.152 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdbd8598:::benchmark_data_smithi179_77138_object60405:head# 2024-02-09T17:26:54.152 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdbe3646:::benchmark_data_smithi179_77138_object42845:head# 2024-02-09T17:26:54.153 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdbf0469:::benchmark_data_smithi179_77138_object115394:head# 2024-02-09T17:26:54.153 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdbf1b9a:::benchmark_data_smithi179_77138_object35972:head# 2024-02-09T17:26:54.154 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdbf4010:::benchmark_data_smithi179_77138_object118379:head# 2024-02-09T17:26:54.155 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdc028cd:::benchmark_data_smithi179_77138_object58045:head# 2024-02-09T17:26:54.155 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdc05619:::benchmark_data_smithi179_77138_object102654:head# 2024-02-09T17:26:54.155 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdc08429:::benchmark_data_smithi179_77138_object88023:head# 2024-02-09T17:26:54.155 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdc0bba7:::benchmark_data_smithi179_77138_object131695:head# 2024-02-09T17:26:54.156 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdc17e38:::benchmark_data_smithi179_77138_object73059:head# 2024-02-09T17:26:54.156 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:26:54.156 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:26:54.157 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdc38eee:::benchmark_data_smithi179_77138_object68723:head# 2024-02-09T17:26:54.157 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdc45c4b:::benchmark_data_smithi179_77138_object121371:head# 2024-02-09T17:26:54.158 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdc5d563:::benchmark_data_smithi179_77138_object80829:head# 2024-02-09T17:26:54.158 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdc666b3:::benchmark_data_smithi179_77138_object46189:head# 2024-02-09T17:26:54.158 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdc6bfe7:::benchmark_data_smithi179_77138_object124204:head# 2024-02-09T17:26:54.159 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdc7595a:::benchmark_data_smithi179_77138_object26990:head# 2024-02-09T17:26:54.159 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdc80a1c:::benchmark_data_smithi179_77138_object29622:head# 2024-02-09T17:26:54.160 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdc83b7d:::benchmark_data_smithi179_77138_object91422:head# 2024-02-09T17:26:54.160 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdc8bbe2:::benchmark_data_smithi179_77138_object38546:head# 2024-02-09T17:26:54.161 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdca0f51:::benchmark_data_smithi179_77138_object76463:head# 2024-02-09T17:26:54.161 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdcbe79d:::benchmark_data_smithi179_77138_object129649:head# 2024-02-09T17:26:54.162 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdcc08b3:::benchmark_data_smithi179_77138_object72917:head# 2024-02-09T17:26:54.162 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdcc0c0e:::benchmark_data_smithi179_77138_object95190:head# 2024-02-09T17:26:54.163 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdcc0c97:::benchmark_data_smithi179_77138_object90818:head# 2024-02-09T17:26:54.164 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdcc42d2:::benchmark_data_smithi179_77138_object36407:head# 2024-02-09T17:26:54.164 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdcc7a31:::benchmark_data_smithi179_77138_object57401:head# 2024-02-09T17:26:54.166 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdccc8c8:::benchmark_data_smithi179_77138_object84324:head# 2024-02-09T17:26:54.166 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdcd0b2c:::benchmark_data_smithi179_77138_object81338:head# 2024-02-09T17:26:54.166 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdcd4e24:::benchmark_data_smithi179_77138_object23711:head# 2024-02-09T17:26:54.166 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdcd8077:::benchmark_data_smithi179_77138_object130625:head# 2024-02-09T17:26:54.167 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdcdd988:::benchmark_data_smithi179_77138_object68315:head# 2024-02-09T17:26:54.167 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdce0d30:::benchmark_data_smithi179_77138_object101903:head# 2024-02-09T17:26:54.167 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdcefa3d:::benchmark_data_smithi179_77138_object27415:head# 2024-02-09T17:26:54.168 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdd03297:::benchmark_data_smithi179_77138_object26553:head# 2024-02-09T17:26:54.168 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdd0e777:::benchmark_data_smithi179_77138_object43856:head# 2024-02-09T17:26:54.168 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdd186dd:::benchmark_data_smithi179_77138_object130940:head# 2024-02-09T17:26:54.169 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdd2c92b:::benchmark_data_smithi179_77138_object70571:head# 2024-02-09T17:26:54.169 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdd31ad5:::benchmark_data_smithi179_77138_object108563:head# 2024-02-09T17:26:54.170 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdd3290c:::benchmark_data_smithi179_77138_object77670:head# 2024-02-09T17:26:54.170 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdd3bd18:::benchmark_data_smithi179_77138_object42095:head# 2024-02-09T17:26:54.170 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdd587af:::benchmark_data_smithi179_77138_object112833:head# 2024-02-09T17:26:54.171 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdd5d41f:::benchmark_data_smithi179_77138_object32659:head# 2024-02-09T17:26:54.171 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdd5d6a7:::benchmark_data_smithi179_77138_object106885:head# 2024-02-09T17:26:54.172 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdd5ecb9:::benchmark_data_smithi179_77138_object67800:head# 2024-02-09T17:26:54.172 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdd64628:::benchmark_data_smithi179_77138_object119110:head# 2024-02-09T17:26:54.173 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdd66d42:::benchmark_data_smithi179_77138_object126415:head# 2024-02-09T17:26:54.173 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdd6802e:::benchmark_data_smithi179_77138_object61644:head# 2024-02-09T17:26:54.174 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdd6a9ec:::benchmark_data_smithi179_77138_object27146:head# 2024-02-09T17:26:54.174 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdd6d444:::benchmark_data_smithi179_77138_object34907:head# 2024-02-09T17:26:54.174 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdd70507:::benchmark_data_smithi179_77138_object52698:head# 2024-02-09T17:26:54.175 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdd71c2c:::benchmark_data_smithi179_77138_object98754:head# 2024-02-09T17:26:54.179 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdd723a8:::benchmark_data_smithi179_77138_object22429:head# 2024-02-09T17:26:54.179 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdd7256b:::benchmark_data_smithi179_77138_object89043:head# 2024-02-09T17:26:54.180 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdd72a91:::benchmark_data_smithi179_77138_object30064:head# 2024-02-09T17:26:54.180 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdd7b260:::benchmark_data_smithi179_77138_object85173:head# 2024-02-09T17:26:54.180 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdd82689:::benchmark_data_smithi179_77138_object111657:head# 2024-02-09T17:26:54.180 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdd854bd:::benchmark_data_smithi179_77138_object22101:head# 2024-02-09T17:26:54.180 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdd89bf9:::benchmark_data_smithi179_77138_object104756:head# 2024-02-09T17:26:54.180 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdd8e514:::benchmark_data_smithi179_77138_object128042:head# 2024-02-09T17:26:54.180 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdd9c071:::benchmark_data_smithi179_77138_object75085:head# 2024-02-09T17:26:54.180 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdda5bc4:::benchmark_data_smithi179_77138_object74847:head# 2024-02-09T17:26:54.181 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fddbfb3c:::benchmark_data_smithi179_77138_object119388:head# 2024-02-09T17:26:54.181 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fddce621:::benchmark_data_smithi179_77138_object75617:head# 2024-02-09T17:26:54.181 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fddd71a7:::benchmark_data_smithi179_77138_object108695:head# 2024-02-09T17:26:54.181 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdddc6e8:::benchmark_data_smithi179_77138_object29324:head# 2024-02-09T17:26:54.181 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fde0c99e:::benchmark_data_smithi179_77138_object56875:head# 2024-02-09T17:26:54.182 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fde1d57a:::benchmark_data_smithi179_77138_object67388:head# 2024-02-09T17:26:54.182 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fde25425:::benchmark_data_smithi179_77138_object58914:head# 2024-02-09T17:26:54.183 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fde25ae9:::benchmark_data_smithi179_77138_object28394:head# 2024-02-09T17:26:54.183 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fde3612f:::benchmark_data_smithi179_77138_object102018:head# 2024-02-09T17:26:54.184 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fde4aabc:::benchmark_data_smithi179_77138_object99652:head# 2024-02-09T17:26:54.184 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fde4f1ce:::benchmark_data_smithi179_77138_object24337:head# 2024-02-09T17:26:54.185 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fde50f2c:::benchmark_data_smithi179_77138_object53754:head# 2024-02-09T17:26:54.186 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fde52d10:::benchmark_data_smithi179_77138_object38845:head# 2024-02-09T17:26:54.186 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fde5ad92:::benchmark_data_smithi179_77138_object105737:head# 2024-02-09T17:26:54.186 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fde6e18a:::benchmark_data_smithi179_77138_object81798:head# 2024-02-09T17:26:54.187 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fde83230:::benchmark_data_smithi179_77138_object103964:head# 2024-02-09T17:26:54.187 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fde8bbb7:::benchmark_data_smithi179_77138_object26871:head# 2024-02-09T17:26:54.187 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fde9442e:::benchmark_data_smithi179_77138_object29891:head# 2024-02-09T17:26:54.188 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fde98b2a:::benchmark_data_smithi179_77138_object111132:head# 2024-02-09T17:26:54.188 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fde9c1b6:::benchmark_data_smithi179_77138_object81001:head# 2024-02-09T17:26:54.188 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdea11e0:::benchmark_data_smithi179_77138_object84090:head# 2024-02-09T17:26:54.189 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdeae8e4:::benchmark_data_smithi179_77138_object87720:head# 2024-02-09T17:26:54.190 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdeaff39:::benchmark_data_smithi179_77138_object43064:head# 2024-02-09T17:26:54.190 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdeb4be9:::benchmark_data_smithi179_77138_object102751:head# 2024-02-09T17:26:54.190 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdec14f3:::benchmark_data_smithi179_77138_object30807:head# 2024-02-09T17:26:54.191 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdec57ec:::benchmark_data_smithi179_77138_object111136:head# 2024-02-09T17:26:54.191 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdec8e8c:::benchmark_data_smithi179_77138_object81237:head# 2024-02-09T17:26:54.192 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdee3b27:::benchmark_data_smithi179_77138_object19506:head# 2024-02-09T17:26:54.192 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdee6eb5:::benchmark_data_smithi179_77138_object65410:head# 2024-02-09T17:26:54.193 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdeedfac:::benchmark_data_smithi179_77138_object89691:head# 2024-02-09T17:26:54.196 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdef388f:::benchmark_data_smithi179_77138_object76649:head# 2024-02-09T17:26:54.196 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdf00c57:::benchmark_data_smithi179_77138_object59616:head# 2024-02-09T17:26:54.197 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdf0b5c3:::benchmark_data_smithi179_77138_object58830:head# 2024-02-09T17:26:54.198 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdf1fb62:::benchmark_data_smithi179_77138_object47857:head# 2024-02-09T17:26:54.198 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdf2c449:::benchmark_data_smithi179_77138_object38800:head# 2024-02-09T17:26:54.198 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdf2e808:::benchmark_data_smithi179_77138_object54439:head# 2024-02-09T17:26:54.199 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdf2f311:::benchmark_data_smithi179_77138_object113271:head# 2024-02-09T17:26:54.199 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdf33281:::benchmark_data_smithi179_77138_object33148:head# 2024-02-09T17:26:54.205 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdf390a2:::benchmark_data_smithi179_77138_object94450:head# 2024-02-09T17:26:54.205 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdf57b9e:::benchmark_data_smithi179_77138_object55229:head# 2024-02-09T17:26:54.205 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdf5b00a:::benchmark_data_smithi179_77138_object61592:head# 2024-02-09T17:26:54.205 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdf686fa:::benchmark_data_smithi179_77138_object30040:head# 2024-02-09T17:26:54.205 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdf69d0e:::benchmark_data_smithi179_77138_object82544:head# 2024-02-09T17:26:54.205 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdf7deec:::benchmark_data_smithi179_77138_object100714:head# 2024-02-09T17:26:54.211 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdf903eb:::benchmark_data_smithi179_77138_object129715:head# 2024-02-09T17:26:54.212 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdf9ec6d:::benchmark_data_smithi179_77138_object122700:head# 2024-02-09T17:26:54.212 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdfa3b32:::benchmark_data_smithi179_77138_object33963:head# 2024-02-09T17:26:54.212 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdfaa187:::benchmark_data_smithi179_77138_object58210:head# 2024-02-09T17:26:54.212 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdfb581f:::benchmark_data_smithi179_77138_object107471:head# 2024-02-09T17:26:54.212 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdfbd182:::benchmark_data_smithi179_77138_object96714:head# 2024-02-09T17:26:54.212 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdfcd9b0:::benchmark_data_smithi179_77138_object81044:head# 2024-02-09T17:26:54.212 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdfd57e7:::benchmark_data_smithi179_77138_object37510:head# 2024-02-09T17:26:54.212 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fdff05a5:::benchmark_data_smithi179_77138_object102951:head# 2024-02-09T17:26:54.214 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe003356:::benchmark_data_smithi179_77138_object86451:head# 2024-02-09T17:26:54.214 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe005eb7:::benchmark_data_smithi179_77138_object59813:head# 2024-02-09T17:26:54.214 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe009c42:::benchmark_data_smithi179_77138_object57283:head# 2024-02-09T17:26:54.214 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe00bd09:::benchmark_data_smithi179_77138_object61261:head# 2024-02-09T17:26:54.214 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe0117e5:::benchmark_data_smithi179_77138_object129124:head# 2024-02-09T17:26:54.214 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe0221ae:::benchmark_data_smithi179_77138_object23766:head# 2024-02-09T17:26:54.215 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe028c3d:::benchmark_data_smithi179_77138_object54672:head# 2024-02-09T17:26:54.215 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe03c7cb:::benchmark_data_smithi179_77138_object74818:head# 2024-02-09T17:26:54.215 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe0505c1:::benchmark_data_smithi179_77138_object100998:head# 2024-02-09T17:26:54.216 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe05dd12:::benchmark_data_smithi179_77138_object94164:head# 2024-02-09T17:26:54.216 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe05e441:::benchmark_data_smithi179_77138_object122456:head# 2024-02-09T17:26:54.216 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe0617b8:::benchmark_data_smithi179_77138_object95804:head# 2024-02-09T17:26:54.217 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe064121:::benchmark_data_smithi179_77138_object66865:head# 2024-02-09T17:26:54.217 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe06a739:::benchmark_data_smithi179_77138_object37454:head# 2024-02-09T17:26:54.218 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe0744da:::benchmark_data_smithi179_77138_object130048:head# 2024-02-09T17:26:54.218 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:26:54.218 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe083ecb:::benchmark_data_smithi179_77138_object43528:head# 2024-02-09T17:26:54.219 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe087d06:::benchmark_data_smithi179_77138_object122339:head# 2024-02-09T17:26:54.219 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe08f400:::benchmark_data_smithi179_77138_object101240:head# 2024-02-09T17:26:54.220 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe095a23:::benchmark_data_smithi179_77138_object44300:head# 2024-02-09T17:26:54.220 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe0a6157:::benchmark_data_smithi179_77138_object66897:head# 2024-02-09T17:26:54.220 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe0bb3fd:::benchmark_data_smithi179_77138_object91952:head# 2024-02-09T17:26:54.221 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe0c25ca:::benchmark_data_smithi179_77138_object103386:head# 2024-02-09T17:26:54.221 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe0c7575:::benchmark_data_smithi179_77138_object119607:head# 2024-02-09T17:26:54.222 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe0cbe0d:::benchmark_data_smithi179_77138_object83465:head# 2024-02-09T17:26:54.222 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe0cbe81:::benchmark_data_smithi179_77138_object84474:head# 2024-02-09T17:26:54.223 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe0cea57:::benchmark_data_smithi179_77138_object125699:head# 2024-02-09T17:26:54.223 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe0d152b:::benchmark_data_smithi179_77138_object50136:head# 2024-02-09T17:26:54.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:54.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:26:54.225 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe0db5e6:::benchmark_data_smithi179_77138_object36279:head# 2024-02-09T17:26:54.225 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe0e056b:::benchmark_data_smithi179_77138_object32812:head# 2024-02-09T17:26:54.225 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe0e256d:::benchmark_data_smithi179_77138_object66766:head# 2024-02-09T17:26:54.225 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe0e7ab0:::benchmark_data_smithi179_77138_object66262:head# 2024-02-09T17:26:54.226 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe0f276c:::benchmark_data_smithi179_77138_object123917:head# 2024-02-09T17:26:54.226 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe0fc3f4:::benchmark_data_smithi179_77138_object119721:head# 2024-02-09T17:26:54.226 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe10085d:::benchmark_data_smithi179_77138_object113641:head# 2024-02-09T17:26:54.231 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe103e6a:::benchmark_data_smithi179_77138_object94083:head# 2024-02-09T17:26:54.231 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe1073b9:::benchmark_data_smithi179_77138_object106447:head# 2024-02-09T17:26:54.231 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe1144d1:::benchmark_data_smithi179_77138_object26564:head# 2024-02-09T17:26:54.231 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe114883:::benchmark_data_smithi179_77138_object23650:head# 2024-02-09T17:26:54.231 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe1150b8:::benchmark_data_smithi179_77138_object63975:head# 2024-02-09T17:26:54.231 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe116032:::benchmark_data_smithi179_77138_object89744:head# 2024-02-09T17:26:54.231 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe11b4f2:::benchmark_data_smithi179_77138_object43914:head# 2024-02-09T17:26:54.232 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe11e7ab:::benchmark_data_smithi179_77138_object93751:head# 2024-02-09T17:26:54.232 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe125460:::benchmark_data_smithi179_77138_object86592:head# 2024-02-09T17:26:54.232 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe1298e6:::benchmark_data_smithi179_77138_object96880:head# 2024-02-09T17:26:54.232 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe131714:::benchmark_data_smithi179_77138_object125112:head# 2024-02-09T17:26:54.234 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe161f90:::benchmark_data_smithi179_77138_object65743:head# 2024-02-09T17:26:54.237 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe16862e:::benchmark_data_smithi179_77138_object28853:head# 2024-02-09T17:26:54.237 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe16c366:::benchmark_data_smithi179_77138_object88399:head# 2024-02-09T17:26:54.237 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe16e409:::benchmark_data_smithi179_77138_object55207:head# 2024-02-09T17:26:54.237 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe170d5d:::benchmark_data_smithi179_77138_object126528:head# 2024-02-09T17:26:54.238 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe1759dd:::benchmark_data_smithi179_77138_object49710:head# 2024-02-09T17:26:54.238 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe17c667:::benchmark_data_smithi179_77138_object81516:head# 2024-02-09T17:26:54.238 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe1864f8:::benchmark_data_smithi179_77138_object76475:head# 2024-02-09T17:26:54.238 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe187427:::benchmark_data_smithi179_77138_object50565:head# 2024-02-09T17:26:54.240 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe18c913:::benchmark_data_smithi179_77138_object34554:head# 2024-02-09T17:26:54.240 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe193d27:::benchmark_data_smithi179_77138_object125855:head# 2024-02-09T17:26:54.240 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe19afb2:::benchmark_data_smithi179_77138_object114909:head# 2024-02-09T17:26:54.241 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe19ee97:::benchmark_data_smithi179_77138_object78610:head# 2024-02-09T17:26:54.241 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe1a18f1:::benchmark_data_smithi179_77138_object122522:head# 2024-02-09T17:26:54.241 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe1a472c:::benchmark_data_smithi179_77138_object22972:head# 2024-02-09T17:26:54.241 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe1ab04a:::benchmark_data_smithi179_77138_object112180:head# 2024-02-09T17:26:54.241 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe1adc69:::benchmark_data_smithi179_77138_object55705:head# 2024-02-09T17:26:54.242 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe1b3d89:::benchmark_data_smithi179_77138_object108518:head# 2024-02-09T17:26:54.242 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe1bd69b:::benchmark_data_smithi179_77138_object51185:head# 2024-02-09T17:26:54.242 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe1c7d9b:::benchmark_data_smithi179_77138_object35567:head# 2024-02-09T17:26:54.243 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe1c9e9e:::benchmark_data_smithi179_77138_object93398:head# 2024-02-09T17:26:54.243 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe1cc9e7:::benchmark_data_smithi179_77138_object92085:head# 2024-02-09T17:26:54.245 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe1ce5a2:::benchmark_data_smithi179_77138_object49332:head# 2024-02-09T17:26:54.245 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe1e497e:::benchmark_data_smithi179_77138_object38696:head# 2024-02-09T17:26:54.246 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe1e514b:::benchmark_data_smithi179_77138_object41166:head# 2024-02-09T17:26:54.246 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe1e8e4d:::benchmark_data_smithi179_77138_object101787:head# 2024-02-09T17:26:54.247 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe1ec75e:::benchmark_data_smithi179_77138_object100680:head# 2024-02-09T17:26:54.247 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe1f282e:::benchmark_data_smithi179_77138_object79519:head# 2024-02-09T17:26:54.248 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe1f55a0:::benchmark_data_smithi179_77138_object124416:head# 2024-02-09T17:26:54.248 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe1f9647:::benchmark_data_smithi179_77138_object106035:head# 2024-02-09T17:26:54.249 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe1fe320:::benchmark_data_smithi179_77138_object56637:head# 2024-02-09T17:26:54.249 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe2005b6:::benchmark_data_smithi179_77138_object82331:head# 2024-02-09T17:26:54.250 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe20d049:::benchmark_data_smithi179_77138_object107184:head# 2024-02-09T17:26:54.250 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe20d94f:::benchmark_data_smithi179_77138_object83892:head# 2024-02-09T17:26:54.250 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe21a930:::benchmark_data_smithi179_77138_object69014:head# 2024-02-09T17:26:54.251 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe223ec0:::benchmark_data_smithi179_77138_object36120:head# 2024-02-09T17:26:54.251 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe2282e8:::benchmark_data_smithi179_77138_object113130:head# 2024-02-09T17:26:54.252 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe2372e6:::benchmark_data_smithi179_77138_object74284:head# 2024-02-09T17:26:54.252 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe24925d:::benchmark_data_smithi179_77138_object119053:head# 2024-02-09T17:26:54.253 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe24d7a6:::benchmark_data_smithi179_77138_object100865:head# 2024-02-09T17:26:54.254 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe250a6d:::benchmark_data_smithi179_77138_object111086:head# 2024-02-09T17:26:54.254 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe261942:::benchmark_data_smithi179_77138_object80242:head# 2024-02-09T17:26:54.254 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe266885:::benchmark_data_smithi179_77138_object106040:head# 2024-02-09T17:26:54.255 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe26e1f7:::benchmark_data_smithi179_77138_object36767:head# 2024-02-09T17:26:54.255 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe274166:::benchmark_data_smithi179_77138_object51507:head# 2024-02-09T17:26:54.256 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe27984e:::benchmark_data_smithi179_77138_object94250:head# 2024-02-09T17:26:54.256 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe28441c:::benchmark_data_smithi179_77138_object36961:head# 2024-02-09T17:26:54.257 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe2a0146:::benchmark_data_smithi179_77138_object31231:head# 2024-02-09T17:26:54.257 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe2a7781:::benchmark_data_smithi179_77138_object56402:head# 2024-02-09T17:26:54.257 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe2ae493:::benchmark_data_smithi179_77138_object62447:head# 2024-02-09T17:26:54.258 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe2aff8c:::benchmark_data_smithi179_77138_object53383:head# 2024-02-09T17:26:54.262 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe2b03be:::benchmark_data_smithi179_77138_object114956:head# 2024-02-09T17:26:54.264 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe2ba690:::benchmark_data_smithi179_77138_object87479:head# 2024-02-09T17:26:54.265 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe2c549e:::benchmark_data_smithi179_77138_object57950:head# 2024-02-09T17:26:54.265 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe2c8f47:::benchmark_data_smithi179_77138_object73929:head# 2024-02-09T17:26:54.267 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe2cce6f:::benchmark_data_smithi179_77138_object78421:head# 2024-02-09T17:26:54.267 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe2cd08a:::benchmark_data_smithi179_77138_object89443:head# 2024-02-09T17:26:54.268 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe2d268c:::benchmark_data_smithi179_77138_object132961:head# 2024-02-09T17:26:54.270 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe2d6612:::benchmark_data_smithi179_77138_object42326:head# 2024-02-09T17:26:54.270 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe2d6d09:::benchmark_data_smithi179_77138_object45313:head# 2024-02-09T17:26:54.271 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe2ddb33:::benchmark_data_smithi179_77138_object62318:head# 2024-02-09T17:26:54.271 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe2e700d:::benchmark_data_smithi179_77138_object39264:head# 2024-02-09T17:26:54.272 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe2ee6ba:::benchmark_data_smithi179_77138_object74319:head# 2024-02-09T17:26:54.272 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe2f3087:::benchmark_data_smithi179_77138_object45615:head# 2024-02-09T17:26:54.272 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe2f3860:::benchmark_data_smithi179_77138_object97709:head# 2024-02-09T17:26:54.273 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe2f713c:::benchmark_data_smithi179_77138_object31484:head# 2024-02-09T17:26:54.273 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe2f75e4:::benchmark_data_smithi179_77138_object28468:head# 2024-02-09T17:26:54.276 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe2f94dd:::benchmark_data_smithi179_77138_object130457:head# 2024-02-09T17:26:54.277 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe2fcc21:::benchmark_data_smithi179_77138_object72740:head# 2024-02-09T17:26:54.277 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe305f87:::benchmark_data_smithi179_77138_object83852:head# 2024-02-09T17:26:54.277 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe32694e:::benchmark_data_smithi179_77138_object103515:head# 2024-02-09T17:26:54.278 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe32ba1b:::benchmark_data_smithi179_77138_object23033:head# 2024-02-09T17:26:54.280 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe32cc75:::benchmark_data_smithi179_77138_object90535:head# 2024-02-09T17:26:54.280 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe333e3e:::benchmark_data_smithi179_77138_object67913:head# 2024-02-09T17:26:54.280 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe34446b:::benchmark_data_smithi179_77138_object26073:head# 2024-02-09T17:26:54.280 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe347b00:::benchmark_data_smithi179_77138_object56014:head# 2024-02-09T17:26:54.280 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe34eb74:::benchmark_data_smithi179_77138_object82853:head# 2024-02-09T17:26:54.281 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe350122:::benchmark_data_smithi179_77138_object80761:head# 2024-02-09T17:26:54.281 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe3521dc:::benchmark_data_smithi179_77138_object48353:head# 2024-02-09T17:26:54.281 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe356be0:::benchmark_data_smithi179_77138_object101043:head# 2024-02-09T17:26:54.281 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe35922d:::benchmark_data_smithi179_77138_object48577:head# 2024-02-09T17:26:54.282 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe374bc9:::benchmark_data_smithi179_77138_object26409:head# 2024-02-09T17:26:54.282 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe397452:::benchmark_data_smithi179_77138_object98775:head# 2024-02-09T17:26:54.283 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe3b668f:::benchmark_data_smithi179_77138_object70672:head# 2024-02-09T17:26:54.283 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe3c9c44:::benchmark_data_smithi179_77138_object65865:head# 2024-02-09T17:26:54.284 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe3dd6e0:::benchmark_data_smithi179_77138_object32770:head# 2024-02-09T17:26:54.284 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe3e3784:::benchmark_data_smithi179_77138_object41181:head# 2024-02-09T17:26:54.286 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe3f60d5:::benchmark_data_smithi179_77138_object22054:head# 2024-02-09T17:26:54.287 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe4005c3:::benchmark_data_smithi179_77138_object34634:head# 2024-02-09T17:26:54.287 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe401643:::benchmark_data_smithi179_77138_object39474:head# 2024-02-09T17:26:54.287 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe418218:::benchmark_data_smithi179_77138_object67152:head# 2024-02-09T17:26:54.288 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe41bfb6:::benchmark_data_smithi179_77138_object69450:head# 2024-02-09T17:26:54.289 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe424df1:::benchmark_data_smithi179_77138_object118699:head# 2024-02-09T17:26:54.289 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe437b89:::benchmark_data_smithi179_77138_object35389:head# 2024-02-09T17:26:54.289 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe442ac2:::benchmark_data_smithi179_77138_object121954:head# 2024-02-09T17:26:54.290 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe44f084:::benchmark_data_smithi179_77138_object104938:head# 2024-02-09T17:26:54.290 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe451f3d:::benchmark_data_smithi179_77138_object32668:head# 2024-02-09T17:26:54.291 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe452a5f:::benchmark_data_smithi179_77138_object40051:head# 2024-02-09T17:26:54.292 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe4543f3:::benchmark_data_smithi179_77138_object125092:head# 2024-02-09T17:26:54.292 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe455491:::benchmark_data_smithi179_77138_object25247:head# 2024-02-09T17:26:54.292 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe45be35:::benchmark_data_smithi179_77138_object57023:head# 2024-02-09T17:26:54.293 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe462046:::benchmark_data_smithi179_77138_object92449:head# 2024-02-09T17:26:54.294 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe463a33:::benchmark_data_smithi179_77138_object80032:head# 2024-02-09T17:26:54.294 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe47a18b:::benchmark_data_smithi179_77138_object79359:head# 2024-02-09T17:26:54.297 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe480195:::benchmark_data_smithi179_77138_object126681:head# 2024-02-09T17:26:54.297 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe4815ca:::benchmark_data_smithi179_77138_object47124:head# 2024-02-09T17:26:54.298 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe4973f5:::benchmark_data_smithi179_77138_object57324:head# 2024-02-09T17:26:54.299 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe4a229d:::benchmark_data_smithi179_77138_object70612:head# 2024-02-09T17:26:54.299 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe4af0a4:::benchmark_data_smithi179_77138_object98734:head# 2024-02-09T17:26:54.300 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe4bdf94:::benchmark_data_smithi179_77138_object70328:head# 2024-02-09T17:26:54.300 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe4c15b8:::benchmark_data_smithi179_77138_object19557:head# 2024-02-09T17:26:54.301 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe4c3d81:::benchmark_data_smithi179_77138_object33432:head# 2024-02-09T17:26:54.301 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe4cad75:::benchmark_data_smithi179_77138_object85069:head# 2024-02-09T17:26:54.303 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe4d1c2c:::benchmark_data_smithi179_77138_object72385:head# 2024-02-09T17:26:54.303 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe4deef1:::benchmark_data_smithi179_77138_object51908:head# 2024-02-09T17:26:54.303 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe5051d3:::benchmark_data_smithi179_77138_object112229:head# 2024-02-09T17:26:54.304 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe50bf0b:::benchmark_data_smithi179_77138_object117455:head# 2024-02-09T17:26:54.304 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe51478a:::benchmark_data_smithi179_77138_object46953:head# 2024-02-09T17:26:54.304 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe531cb7:::benchmark_data_smithi179_77138_object94574:head# 2024-02-09T17:26:54.305 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe53c679:::benchmark_data_smithi179_77138_object81038:head# 2024-02-09T17:26:54.305 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe5405d9:::benchmark_data_smithi179_77138_object102466:head# 2024-02-09T17:26:54.306 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe54e0f7:::benchmark_data_smithi179_77138_object47974:head# 2024-02-09T17:26:54.306 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe54e51f:::benchmark_data_smithi179_77138_object109870:head# 2024-02-09T17:26:54.306 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe54ea12:::benchmark_data_smithi179_77138_object103293:head# 2024-02-09T17:26:54.307 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe55132d:::benchmark_data_smithi179_77138_object26181:head# 2024-02-09T17:26:54.307 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe5592df:::benchmark_data_smithi179_77138_object65413:head# 2024-02-09T17:26:54.308 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe55a451:::benchmark_data_smithi179_77138_object79125:head# 2024-02-09T17:26:54.308 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe55b19d:::benchmark_data_smithi179_77138_object52888:head# 2024-02-09T17:26:54.309 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe55b89d:::benchmark_data_smithi179_77138_object83647:head# 2024-02-09T17:26:54.309 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe56cad2:::benchmark_data_smithi179_77138_object55548:head# 2024-02-09T17:26:54.310 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe56f9c0:::benchmark_data_smithi179_77138_object77131:head# 2024-02-09T17:26:54.310 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe5798a2:::benchmark_data_smithi179_77138_object77793:head# 2024-02-09T17:26:54.313 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe5826d2:::benchmark_data_smithi179_77138_object44568:head# 2024-02-09T17:26:54.313 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe584336:::benchmark_data_smithi179_77138_object36256:head# 2024-02-09T17:26:54.314 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe585931:::benchmark_data_smithi179_77138_object68250:head# 2024-02-09T17:26:54.314 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe58699a:::benchmark_data_smithi179_77138_object23477:head# 2024-02-09T17:26:54.314 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe590f34:::benchmark_data_smithi179_77138_object79551:head# 2024-02-09T17:26:54.315 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe599545:::benchmark_data_smithi179_77138_object67420:head# 2024-02-09T17:26:54.315 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe5abd77:::benchmark_data_smithi179_77138_object25252:head# 2024-02-09T17:26:54.316 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe5b8186:::benchmark_data_smithi179_77138_object48037:head# 2024-02-09T17:26:54.316 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe5c0081:::benchmark_data_smithi179_77138_object94069:head# 2024-02-09T17:26:54.317 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe5c7d61:::benchmark_data_smithi179_77138_object40426:head# 2024-02-09T17:26:54.317 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe5c99ed:::benchmark_data_smithi179_77138_object114224:head# 2024-02-09T17:26:54.318 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe5c9ab7:::benchmark_data_smithi179_77138_object63358:head# 2024-02-09T17:26:54.318 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe5d8d01:::benchmark_data_smithi179_77138_object71278:head# 2024-02-09T17:26:54.318 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe5dd2cd:::benchmark_data_smithi179_77138_object105875:head# 2024-02-09T17:26:54.319 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe5e86cf:::benchmark_data_smithi179_77138_object93535:head# 2024-02-09T17:26:54.319 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe5ee57f:::benchmark_data_smithi179_77138_object114458:head# 2024-02-09T17:26:54.320 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe5f28c9:::benchmark_data_smithi179_77138_object80699:head# 2024-02-09T17:26:54.320 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe5f652a:::benchmark_data_smithi179_77138_object100080:head# 2024-02-09T17:26:54.321 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe5f9beb:::benchmark_data_smithi179_77138_object80000:head# 2024-02-09T17:26:54.321 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe60b3b7:::benchmark_data_smithi179_77138_object93773:head# 2024-02-09T17:26:54.321 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe60eb4d:::benchmark_data_smithi179_77138_object54060:head# 2024-02-09T17:26:54.322 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe6161fa:::benchmark_data_smithi179_77138_object52371:head# 2024-02-09T17:26:54.322 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe6213c8:::benchmark_data_smithi179_77138_object101650:head# 2024-02-09T17:26:54.323 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe6267a9:::benchmark_data_smithi179_77138_object81252:head# 2024-02-09T17:26:54.323 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe629041:::benchmark_data_smithi179_77138_object51404:head# 2024-02-09T17:26:54.324 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe62ce2d:::benchmark_data_smithi179_77138_object32277:head# 2024-02-09T17:26:54.324 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe62f3d7:::benchmark_data_smithi179_77138_object20176:head# 2024-02-09T17:26:54.325 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:26:54.326 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe63042d:::benchmark_data_smithi179_77138_object105510:head# 2024-02-09T17:26:54.326 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe63bd37:::benchmark_data_smithi179_77138_object28426:head# 2024-02-09T17:26:54.326 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe6465a2:::benchmark_data_smithi179_77138_object106071:head# 2024-02-09T17:26:54.327 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe64890c:::benchmark_data_smithi179_77138_object104313:head# 2024-02-09T17:26:54.331 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe64a290:::benchmark_data_smithi179_77138_object74054:head# 2024-02-09T17:26:54.331 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe652c4e:::benchmark_data_smithi179_77138_object72859:head# 2024-02-09T17:26:54.331 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe654cd0:::benchmark_data_smithi179_77138_object31319:head# 2024-02-09T17:26:54.331 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe65def2:::benchmark_data_smithi179_77138_object102524:head# 2024-02-09T17:26:54.331 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe65ee7d:::benchmark_data_smithi179_77138_object64159:head# 2024-02-09T17:26:54.332 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe66d7bb:::benchmark_data_smithi179_77138_object29790:head# 2024-02-09T17:26:54.332 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe66dbfa:::benchmark_data_smithi179_77138_object93268:head# 2024-02-09T17:26:54.333 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe674524:::benchmark_data_smithi179_77138_object111704:head# 2024-02-09T17:26:54.333 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe6827b6:::benchmark_data_smithi179_77138_object42311:head# 2024-02-09T17:26:54.333 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe68dda3:::benchmark_data_smithi179_77138_object86985:head# 2024-02-09T17:26:54.334 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe696942:::benchmark_data_smithi179_77138_object106904:head# 2024-02-09T17:26:54.334 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe6a273d:::benchmark_data_smithi179_77138_object69804:head# 2024-02-09T17:26:54.334 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe6aa985:::benchmark_data_smithi179_77138_object74480:head# 2024-02-09T17:26:54.335 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe6c1cd7:::benchmark_data_smithi179_77138_object92336:head# 2024-02-09T17:26:54.335 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe6c8d50:::benchmark_data_smithi179_77138_object33845:head# 2024-02-09T17:26:54.336 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe6cade6:::benchmark_data_smithi179_77138_object131986:head# 2024-02-09T17:26:54.337 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe6d2746:::benchmark_data_smithi179_77138_object106571:head# 2024-02-09T17:26:54.337 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe6de3c4:::benchmark_data_smithi179_77138_object32343:head# 2024-02-09T17:26:54.337 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe6f4817:::benchmark_data_smithi179_77138_object59786:head# 2024-02-09T17:26:54.342 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe6fa88b:::benchmark_data_smithi179_77138_object84917:head# 2024-02-09T17:26:54.342 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe708e59:::benchmark_data_smithi179_77138_object92104:head# 2024-02-09T17:26:54.342 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe70b068:::benchmark_data_smithi179_77138_object96440:head# 2024-02-09T17:26:54.342 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe715e73:::benchmark_data_smithi179_77138_object95111:head# 2024-02-09T17:26:54.342 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe7175b3:::benchmark_data_smithi179_77138_object95865:head# 2024-02-09T17:26:54.342 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe717dcd:::benchmark_data_smithi179_77138_object79550:head# 2024-02-09T17:26:54.343 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe718b77:::benchmark_data_smithi179_77138_object74053:head# 2024-02-09T17:26:54.343 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe71fdb4:::benchmark_data_smithi179_77138_object102231:head# 2024-02-09T17:26:54.343 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe728340:::benchmark_data_smithi179_77138_object64107:head# 2024-02-09T17:26:54.344 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe72e504:::benchmark_data_smithi179_77138_object100518:head# 2024-02-09T17:26:54.344 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe73885a:::benchmark_data_smithi179_77138_object127671:head# 2024-02-09T17:26:54.346 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe747a18:::benchmark_data_smithi179_77138_object83087:head# 2024-02-09T17:26:54.346 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe75b2aa:::benchmark_data_smithi179_77138_object38433:head# 2024-02-09T17:26:54.346 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe75b5fc:::benchmark_data_smithi179_77138_object92431:head# 2024-02-09T17:26:54.346 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe75cb86:::benchmark_data_smithi179_77138_object50756:head# 2024-02-09T17:26:54.347 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe76933a:::benchmark_data_smithi179_77138_object48715:head# 2024-02-09T17:26:54.348 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe7730db:::benchmark_data_smithi179_77138_object30789:head# 2024-02-09T17:26:54.348 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe77b562:::benchmark_data_smithi179_77138_object102841:head# 2024-02-09T17:26:54.348 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe781f39:::benchmark_data_smithi179_77138_object60065:head# 2024-02-09T17:26:54.350 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe79f0ac:::benchmark_data_smithi179_77138_object88293:head# 2024-02-09T17:26:54.351 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe7b148b:::benchmark_data_smithi179_77138_object87054:head# 2024-02-09T17:26:54.351 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe7b7357:::benchmark_data_smithi179_77138_object49814:head# 2024-02-09T17:26:54.352 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe7ba7fe:::benchmark_data_smithi179_77138_object114404:head# 2024-02-09T17:26:54.357 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe7bb882:::benchmark_data_smithi179_77138_object44238:head# 2024-02-09T17:26:54.357 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe7c17cc:::benchmark_data_smithi179_77138_object81480:head# 2024-02-09T17:26:54.357 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe7db89d:::benchmark_data_smithi179_77138_object63625:head# 2024-02-09T17:26:54.357 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe7e025b:::benchmark_data_smithi179_77138_object70234:head# 2024-02-09T17:26:54.357 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe7ed3a9:::benchmark_data_smithi179_77138_object127780:head# 2024-02-09T17:26:54.357 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe7f1aed:::benchmark_data_smithi179_77138_object24569:head# 2024-02-09T17:26:54.364 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:54.368 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe7f3b36:::benchmark_data_smithi179_77138_object31835:head# 2024-02-09T17:26:54.368 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe8014c6:::benchmark_data_smithi179_77138_object35873:head# 2024-02-09T17:26:54.368 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe80c2a8:::benchmark_data_smithi179_77138_object49492:head# 2024-02-09T17:26:54.371 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe80e009:::benchmark_data_smithi179_77138_object33352:head# 2024-02-09T17:26:54.373 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe8104ca:::benchmark_data_smithi179_77138_object33483:head# 2024-02-09T17:26:54.373 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe813fa1:::benchmark_data_smithi179_77138_object125464:head# 2024-02-09T17:26:54.374 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe81654e:::benchmark_data_smithi179_77138_object66148:head# 2024-02-09T17:26:54.378 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe81f730:::benchmark_data_smithi179_77138_object125667:head# 2024-02-09T17:26:54.378 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe823a9b:::benchmark_data_smithi179_77138_object27250:head# 2024-02-09T17:26:54.378 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe82c113:::benchmark_data_smithi179_77138_object29043:head# 2024-02-09T17:26:54.378 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe82e56e:::benchmark_data_smithi179_77138_object94403:head# 2024-02-09T17:26:54.378 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe832443:::benchmark_data_smithi179_77138_object121369:head# 2024-02-09T17:26:54.378 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe83c9fe:::benchmark_data_smithi179_77138_object42183:head# 2024-02-09T17:26:54.378 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe84ef79:::benchmark_data_smithi179_77138_object126725:head# 2024-02-09T17:26:54.378 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe856def:::benchmark_data_smithi179_77138_object111249:head# 2024-02-09T17:26:54.378 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe87cff8:::benchmark_data_smithi179_77138_object86726:head# 2024-02-09T17:26:54.379 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe893035:::benchmark_data_smithi179_77138_object93767:head# 2024-02-09T17:26:54.379 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe89d3d2:::benchmark_data_smithi179_77138_object58150:head# 2024-02-09T17:26:54.379 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe8a275f:::benchmark_data_smithi179_77138_object21430:head# 2024-02-09T17:26:54.380 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe8a4eb4:::benchmark_data_smithi179_77138_object62733:head# 2024-02-09T17:26:54.381 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe8a70c7:::benchmark_data_smithi179_77138_object43319:head# 2024-02-09T17:26:54.381 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe8b49ee:::benchmark_data_smithi179_77138_object133125:head# 2024-02-09T17:26:54.382 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe8c22eb:::benchmark_data_smithi179_77138_object108566:head# 2024-02-09T17:26:54.382 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe8cb5db:::benchmark_data_smithi179_77138_object105217:head# 2024-02-09T17:26:54.383 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe8d1d7e:::benchmark_data_smithi179_77138_object36026:head# 2024-02-09T17:26:54.383 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe8d4b59:::benchmark_data_smithi179_77138_object27730:head# 2024-02-09T17:26:54.383 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe8d4d50:::benchmark_data_smithi179_77138_object55501:head# 2024-02-09T17:26:54.384 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe8d84c6:::benchmark_data_smithi179_77138_object49724:head# 2024-02-09T17:26:54.384 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe8dc682:::benchmark_data_smithi179_77138_object49019:head# 2024-02-09T17:26:54.385 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe8dc80d:::benchmark_data_smithi179_77138_object112167:head# 2024-02-09T17:26:54.385 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe8e61b2:::benchmark_data_smithi179_77138_object93785:head# 2024-02-09T17:26:54.386 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe8e7ff2:::benchmark_data_smithi179_77138_object116822:head# 2024-02-09T17:26:54.386 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe8ed9b9:::benchmark_data_smithi179_77138_object79500:head# 2024-02-09T17:26:54.386 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe8f19e1:::benchmark_data_smithi179_77138_object75733:head# 2024-02-09T17:26:54.387 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe8f35da:::benchmark_data_smithi179_77138_object118267:head# 2024-02-09T17:26:54.388 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe9073c1:::benchmark_data_smithi179_77138_object27363:head# 2024-02-09T17:26:54.388 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe92544f:::benchmark_data_smithi179_77138_object74432:head# 2024-02-09T17:26:54.388 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe935218:::benchmark_data_smithi179_77138_object99203:head# 2024-02-09T17:26:54.389 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe9362e2:::benchmark_data_smithi179_77138_object100056:head# 2024-02-09T17:26:54.391 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:26:54.392 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe93e103:::benchmark_data_smithi179_77138_object73326:head# 2024-02-09T17:26:54.392 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe96351d:::benchmark_data_smithi179_77138_object127596:head# 2024-02-09T17:26:54.392 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe968b5c:::benchmark_data_smithi179_77138_object21528:head# 2024-02-09T17:26:54.392 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe96f073:::benchmark_data_smithi179_77138_object39424:head# 2024-02-09T17:26:54.392 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe9739cc:::benchmark_data_smithi179_77138_object48557:head# 2024-02-09T17:26:54.392 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe97b83c:::benchmark_data_smithi179_77138_object55999:head# 2024-02-09T17:26:54.393 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe983f00:::benchmark_data_smithi179_77138_object81256:head# 2024-02-09T17:26:54.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:54.394 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:26:54.394 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe99a31c:::benchmark_data_smithi179_77138_object84855:head# 2024-02-09T17:26:54.394 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe9ae494:::benchmark_data_smithi179_77138_object127749:head# 2024-02-09T17:26:54.396 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe9c1656:::benchmark_data_smithi179_77138_object123454:head# 2024-02-09T17:26:54.396 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe9c5809:::benchmark_data_smithi179_77138_object56058:head# 2024-02-09T17:26:54.396 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe9c584e:::benchmark_data_smithi179_77138_object110156:head# 2024-02-09T17:26:54.396 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe9d5417:::benchmark_data_smithi179_77138_object128610:head# 2024-02-09T17:26:54.396 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe9d766e:::benchmark_data_smithi179_77138_object130871:head# 2024-02-09T17:26:54.396 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe9d893f:::benchmark_data_smithi179_77138_object108550:head# 2024-02-09T17:26:54.397 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe9dcfe1:::benchmark_data_smithi179_77138_object107328:head# 2024-02-09T17:26:54.397 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe9deec7:::benchmark_data_smithi179_77138_object109205:head# 2024-02-09T17:26:54.397 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe9f15c8:::benchmark_data_smithi179_77138_object62606:head# 2024-02-09T17:26:54.397 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fe9f3cdd:::benchmark_data_smithi179_77138_object115693:head# 2024-02-09T17:26:54.399 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fea02305:::benchmark_data_smithi179_77138_object95105:head# 2024-02-09T17:26:54.399 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fea04289:::benchmark_data_smithi179_77138_object119462:head# 2024-02-09T17:26:54.399 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fea0bfe5:::benchmark_data_smithi179_77138_object54372:head# 2024-02-09T17:26:54.400 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fea0de54:::benchmark_data_smithi179_77138_object128099:head# 2024-02-09T17:26:54.400 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fea13ea6:::benchmark_data_smithi179_77138_object115962:head# 2024-02-09T17:26:54.401 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fea177de:::benchmark_data_smithi179_77138_object75100:head# 2024-02-09T17:26:54.401 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fea2308b:::benchmark_data_smithi179_77138_object79327:head# 2024-02-09T17:26:54.401 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fea2988d:::benchmark_data_smithi179_77138_object109708:head# 2024-02-09T17:26:54.402 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fea2b3d9:::benchmark_data_smithi179_77138_object25001:head# 2024-02-09T17:26:54.402 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fea32310:::benchmark_data_smithi179_77138_object66379:head# 2024-02-09T17:26:54.403 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fea37291:::benchmark_data_smithi179_77138_object39314:head# 2024-02-09T17:26:54.403 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fea42de1:::benchmark_data_smithi179_77138_object33472:head# 2024-02-09T17:26:54.403 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fea6e74a:::benchmark_data_smithi179_77138_object21796:head# 2024-02-09T17:26:54.404 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fea73194:::benchmark_data_smithi179_77138_object85379:head# 2024-02-09T17:26:54.404 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fea78049:::benchmark_data_smithi179_77138_object126582:head# 2024-02-09T17:26:54.405 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fea8941b:::benchmark_data_smithi179_77138_object61144:head# 2024-02-09T17:26:54.405 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fea89821:::benchmark_data_smithi179_77138_object50824:head# 2024-02-09T17:26:54.405 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fea8c630:::benchmark_data_smithi179_77138_object75011:head# 2024-02-09T17:26:54.406 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:feab7dc6:::benchmark_data_smithi179_77138_object122903:head# 2024-02-09T17:26:54.407 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:feaba071:::benchmark_data_smithi179_77138_object86357:head# 2024-02-09T17:26:54.407 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fead165a:::benchmark_data_smithi179_77138_object23426:head# 2024-02-09T17:26:54.408 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:feae217a:::benchmark_data_smithi179_77138_object129685:head# 2024-02-09T17:26:54.408 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:feae47dd:::benchmark_data_smithi179_77138_object123772:head# 2024-02-09T17:26:54.408 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:feaeac9e:::benchmark_data_smithi179_77138_object132054:head# 2024-02-09T17:26:54.408 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:feb00f0f:::benchmark_data_smithi179_77138_object36676:head# 2024-02-09T17:26:54.409 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:feb09f42:::benchmark_data_smithi179_77138_object46181:head# 2024-02-09T17:26:54.409 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:feb0cc0b:::benchmark_data_smithi179_77138_object106256:head# 2024-02-09T17:26:54.410 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:feb1a582:::benchmark_data_smithi179_77138_object77729:head# 2024-02-09T17:26:54.410 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:feb1e840:::benchmark_data_smithi179_77138_object121985:head# 2024-02-09T17:26:54.410 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:feb2a074:::benchmark_data_smithi179_77138_object76724:head# 2024-02-09T17:26:54.411 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:feb2a476:::benchmark_data_smithi179_77138_object87378:head# 2024-02-09T17:26:54.411 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:feb372ae:::benchmark_data_smithi179_77138_object36604:head# 2024-02-09T17:26:54.411 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:feb49fa3:::benchmark_data_smithi179_77138_object80825:head# 2024-02-09T17:26:54.412 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:feb4e82b:::benchmark_data_smithi179_77138_object86923:head# 2024-02-09T17:26:54.412 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:feb50d65:::benchmark_data_smithi179_77138_object36187:head# 2024-02-09T17:26:54.413 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:feb722ca:::benchmark_data_smithi179_77138_object63699:head# 2024-02-09T17:26:54.413 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:feb75b0f:::benchmark_data_smithi179_77138_object43277:head# 2024-02-09T17:26:54.414 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:feb77841:::benchmark_data_smithi179_77138_object101008:head# 2024-02-09T17:26:54.414 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:feb86775:::benchmark_data_smithi179_77138_object112761:head# 2024-02-09T17:26:54.414 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:feb935f9:::benchmark_data_smithi179_77138_object32337:head# 2024-02-09T17:26:54.415 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:feb99e0c:::benchmark_data_smithi179_77138_object83276:head# 2024-02-09T17:26:54.415 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:feb9d885:::benchmark_data_smithi179_77138_object133098:head# 2024-02-09T17:26:54.416 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:feba3369:::benchmark_data_smithi179_77138_object128233:head# 2024-02-09T17:26:54.416 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:febb7ddc:::benchmark_data_smithi179_77138_object40747:head# 2024-02-09T17:26:54.416 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:febbb080:::benchmark_data_smithi179_77138_object49863:head# 2024-02-09T17:26:54.417 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:febc4be0:::benchmark_data_smithi179_77138_object33309:head# 2024-02-09T17:26:54.417 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:febca6b0:::benchmark_data_smithi179_77138_object107509:head# 2024-02-09T17:26:54.418 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:febcc805:::benchmark_data_smithi179_77138_object34112:head# 2024-02-09T17:26:54.418 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:febd1946:::benchmark_data_smithi179_77138_object33915:head# 2024-02-09T17:26:54.419 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:febde508:::benchmark_data_smithi179_77138_object30562:head# 2024-02-09T17:26:54.419 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:febe1525:::benchmark_data_smithi179_77138_object32476:head# 2024-02-09T17:26:54.420 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:febe2e10:::benchmark_data_smithi179_77138_object31341:head# 2024-02-09T17:26:54.420 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:febf310f:::benchmark_data_smithi179_77138_object48624:head# 2024-02-09T17:26:54.420 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fec01f10:::benchmark_data_smithi179_77138_object132128:head# 2024-02-09T17:26:54.421 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fec04992:::benchmark_data_smithi179_77138_object81227:head# 2024-02-09T17:26:54.421 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fec1ffb2:::benchmark_data_smithi179_77138_object77936:head# 2024-02-09T17:26:54.422 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fec348de:::benchmark_data_smithi179_77138_object87888:head# 2024-02-09T17:26:54.423 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fec3912f:::benchmark_data_smithi179_77138_object23736:head# 2024-02-09T17:26:54.423 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fec3c84f:::benchmark_data_smithi179_77138_object59906:head# 2024-02-09T17:26:54.424 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fec4342e:::benchmark_data_smithi179_77138_object93793:head# 2024-02-09T17:26:54.424 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fec54acd:::benchmark_data_smithi179_77138_object34102:head# 2024-02-09T17:26:54.426 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fec6a634:::benchmark_data_smithi179_77138_object87413:head# 2024-02-09T17:26:54.426 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fec73195:::benchmark_data_smithi179_77138_object111742:head# 2024-02-09T17:26:54.427 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fec745ee:::benchmark_data_smithi179_77138_object71042:head# 2024-02-09T17:26:54.428 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fec8c8f7:::benchmark_data_smithi179_77138_object54274:head# 2024-02-09T17:26:54.428 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fec8cdfe:::benchmark_data_smithi179_77138_object109835:head# 2024-02-09T17:26:54.428 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fec8dfdc:::benchmark_data_smithi179_77138_object36090:head# 2024-02-09T17:26:54.429 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fec8f4a9:::benchmark_data_smithi179_77138_object109857:head# 2024-02-09T17:26:54.429 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fec93e34:::benchmark_data_smithi179_77138_object76873:head# 2024-02-09T17:26:54.430 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fecaa83b:::benchmark_data_smithi179_77138_object35484:head# 2024-02-09T17:26:54.430 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fecac443:::benchmark_data_smithi179_77138_object64571:head# 2024-02-09T17:26:54.432 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fecb181d:::benchmark_data_smithi179_77138_object55876:head# 2024-02-09T17:26:54.433 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fecd0c7e:::benchmark_data_smithi179_77138_object83426:head# 2024-02-09T17:26:54.433 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:feced9c4:::benchmark_data_smithi179_77138_object96867:head# 2024-02-09T17:26:54.435 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fed06e34:::benchmark_data_smithi179_77138_object95290:head# 2024-02-09T17:26:54.436 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fed1f3e3:::benchmark_data_smithi179_77138_object60524:head# 2024-02-09T17:26:54.436 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fed23f3c:::benchmark_data_smithi179_77138_object131126:head# 2024-02-09T17:26:54.437 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fed2bd31:::benchmark_data_smithi179_77138_object56819:head# 2024-02-09T17:26:54.437 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fed2c612:::benchmark_data_smithi179_77138_object20221:head# 2024-02-09T17:26:54.437 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fed30865:::benchmark_data_smithi179_77138_object130758:head# 2024-02-09T17:26:54.438 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fed33147:::benchmark_data_smithi179_77138_object129545:head# 2024-02-09T17:26:54.438 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fed33716:::benchmark_data_smithi179_77138_object35848:head# 2024-02-09T17:26:54.439 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fed430f3:::benchmark_data_smithi179_77138_object50017:head# 2024-02-09T17:26:54.439 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fed47c9f:::benchmark_data_smithi179_77138_object36402:head# 2024-02-09T17:26:54.440 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fed4e2a3:::benchmark_data_smithi179_77138_object117652:head# 2024-02-09T17:26:54.440 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fed78ec5:::benchmark_data_smithi179_77138_object89509:head# 2024-02-09T17:26:54.443 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fed7ab2f:::benchmark_data_smithi179_77138_object41889:head# 2024-02-09T17:26:54.444 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fed7c1a6:::benchmark_data_smithi179_77138_object124409:head# 2024-02-09T17:26:54.444 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fed84db5:::benchmark_data_smithi179_77138_object90750:head# 2024-02-09T17:26:54.445 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fed9022e:::benchmark_data_smithi179_77138_object44789:head# 2024-02-09T17:26:54.445 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fedc7cba:::benchmark_data_smithi179_77138_object90560:head# 2024-02-09T17:26:54.445 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fedca3b2:::benchmark_data_smithi179_77138_object63040:head# 2024-02-09T17:26:54.446 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fedcddcd:::benchmark_data_smithi179_77138_object126856:head# 2024-02-09T17:26:54.446 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:feddf5d3:::benchmark_data_smithi179_77138_object91135:head# 2024-02-09T17:26:54.447 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fedecce5:::benchmark_data_smithi179_77138_object98167:head# 2024-02-09T17:26:54.448 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fedf20dd:::benchmark_data_smithi179_77138_object77237:head# 2024-02-09T17:26:54.448 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fedf4703:::benchmark_data_smithi179_77138_object37465:head# 2024-02-09T17:26:54.448 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fedff09f:::benchmark_data_smithi179_77138_object105807:head# 2024-02-09T17:26:54.449 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fee0f421:::benchmark_data_smithi179_77138_object83089:head# 2024-02-09T17:26:54.449 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fee1091e:::benchmark_data_smithi179_77138_object131969:head# 2024-02-09T17:26:54.450 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fee14306:::benchmark_data_smithi179_77138_object110789:head# 2024-02-09T17:26:54.451 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fee20a05:::benchmark_data_smithi179_77138_object53956:head# 2024-02-09T17:26:54.451 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fee23bb1:::benchmark_data_smithi179_77138_object96411:head# 2024-02-09T17:26:54.451 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fee27673:::benchmark_data_smithi179_77138_object21054:head# 2024-02-09T17:26:54.452 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fee3665d:::benchmark_data_smithi179_77138_object61978:head# 2024-02-09T17:26:54.452 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fee3f07c:::benchmark_data_smithi179_77138_object30326:head# 2024-02-09T17:26:54.452 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fee44171:::benchmark_data_smithi179_77138_object122367:head# 2024-02-09T17:26:54.453 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fee447bf:::benchmark_data_smithi179_77138_object85525:head# 2024-02-09T17:26:54.453 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fee4d610:::benchmark_data_smithi179_77138_object75435:head# 2024-02-09T17:26:54.454 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fee4f0e0:::benchmark_data_smithi179_77138_object61331:head# 2024-02-09T17:26:54.454 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fee4f5e7:::benchmark_data_smithi179_77138_object32200:head# 2024-02-09T17:26:54.455 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fee6df5f:::benchmark_data_smithi179_77138_object104936:head# 2024-02-09T17:26:54.455 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fee7368d:::benchmark_data_smithi179_77138_object78662:head# 2024-02-09T17:26:54.455 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fee74718:::benchmark_data_smithi179_77138_object33677:head# 2024-02-09T17:26:54.456 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fee76145:::benchmark_data_smithi179_77138_object26316:head# 2024-02-09T17:26:54.456 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fee7dd04:::benchmark_data_smithi179_77138_object31432:head# 2024-02-09T17:26:54.457 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fee92a2f:::benchmark_data_smithi179_77138_object34430:head# 2024-02-09T17:26:54.457 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fee98717:::benchmark_data_smithi179_77138_object58817:head# 2024-02-09T17:26:54.458 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fee9efa5:::benchmark_data_smithi179_77138_object104789:head# 2024-02-09T17:26:54.458 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:feea4029:::benchmark_data_smithi179_77138_object94851:head# 2024-02-09T17:26:54.458 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:feea819e:::benchmark_data_smithi179_77138_object82153:head# 2024-02-09T17:26:54.459 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:feeaca1b:::benchmark_data_smithi179_77138_object118352:head# 2024-02-09T17:26:54.459 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:feeaf6df:::benchmark_data_smithi179_77138_object52906:head# 2024-02-09T17:26:54.460 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:feebe508:::benchmark_data_smithi179_77138_object55521:head# 2024-02-09T17:26:54.460 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:feecb98d:::benchmark_data_smithi179_77138_object53020:head# 2024-02-09T17:26:54.460 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:feed1157:::benchmark_data_smithi179_77138_object115748:head# 2024-02-09T17:26:54.464 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:feed4cc6:::benchmark_data_smithi179_77138_object42143:head# 2024-02-09T17:26:54.464 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:feee028c:::benchmark_data_smithi179_77138_object26235:head# 2024-02-09T17:26:54.464 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:feee12c7:::benchmark_data_smithi179_77138_object65950:head# 2024-02-09T17:26:54.464 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:feee961c:::benchmark_data_smithi179_77138_object83002:head# 2024-02-09T17:26:54.464 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:feeeb6f9:::benchmark_data_smithi179_77138_object129374:head# 2024-02-09T17:26:54.465 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fef0555b:::benchmark_data_smithi179_77138_object89229:head# 2024-02-09T17:26:54.465 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fef0c83d:::benchmark_data_smithi179_77138_object117369:head# 2024-02-09T17:26:54.465 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fef1096a:::benchmark_data_smithi179_77138_object39436:head# 2024-02-09T17:26:54.465 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fef2cc1c:::benchmark_data_smithi179_77138_object63092:head# 2024-02-09T17:26:54.465 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fef3e0c8:::benchmark_data_smithi179_77138_object117211:head# 2024-02-09T17:26:54.466 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fef3f956:::benchmark_data_smithi179_77138_object22990:head# 2024-02-09T17:26:54.466 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fef446bb:::benchmark_data_smithi179_77138_object53726:head# 2024-02-09T17:26:54.467 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fef4619a:::benchmark_data_smithi179_77138_object27128:head# 2024-02-09T17:26:54.467 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fef49a6a:::benchmark_data_smithi179_77138_object93258:head# 2024-02-09T17:26:54.467 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fef4db6e:::benchmark_data_smithi179_77138_object83442:head# 2024-02-09T17:26:54.468 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fef6e960:::benchmark_data_smithi179_77138_object60379:head# 2024-02-09T17:26:54.468 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fef836df:::benchmark_data_smithi179_77138_object114069:head# 2024-02-09T17:26:54.469 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fef8e771:::benchmark_data_smithi179_77138_object89139:head# 2024-02-09T17:26:54.469 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fef91944:::benchmark_data_smithi179_77138_object120000:head# 2024-02-09T17:26:54.470 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fefba326:::benchmark_data_smithi179_77138_object86737:head# 2024-02-09T17:26:54.470 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fefbf5c3:::benchmark_data_smithi179_77138_object69418:head# 2024-02-09T17:26:54.470 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fefc6598:::benchmark_data_smithi179_77138_object110208:head# 2024-02-09T17:26:54.471 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fefceefb:::benchmark_data_smithi179_77138_object97764:head# 2024-02-09T17:26:54.471 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fefd7921:::benchmark_data_smithi179_77138_object63340:head# 2024-02-09T17:26:54.472 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fefe7479:::benchmark_data_smithi179_77138_object118802:head# 2024-02-09T17:26:54.475 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:feff5f57:::benchmark_data_smithi179_77138_object20707:head# 2024-02-09T17:26:54.475 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff006362:::benchmark_data_smithi179_77138_object39079:head# 2024-02-09T17:26:54.475 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff00c5f9:::benchmark_data_smithi179_77138_object129696:head# 2024-02-09T17:26:54.476 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff019799:::benchmark_data_smithi179_77138_object73547:head# 2024-02-09T17:26:54.477 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff051e6e:::benchmark_data_smithi179_77138_object113861:head# 2024-02-09T17:26:54.478 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff05e5bc:::benchmark_data_smithi179_77138_object96099:head# 2024-02-09T17:26:54.478 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff0681f0:::benchmark_data_smithi179_77138_object122863:head# 2024-02-09T17:26:54.478 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff0883a9:::benchmark_data_smithi179_77138_object128615:head# 2024-02-09T17:26:54.478 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff08e3f4:::benchmark_data_smithi179_77138_object64302:head# 2024-02-09T17:26:54.479 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff09ce39:::benchmark_data_smithi179_77138_object48556:head# 2024-02-09T17:26:54.480 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff0a272a:::benchmark_data_smithi179_77138_object50327:head# 2024-02-09T17:26:54.480 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff0a6cfa:::benchmark_data_smithi179_77138_object118629:head# 2024-02-09T17:26:54.480 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff0a9e9d:::benchmark_data_smithi179_77138_object125745:head# 2024-02-09T17:26:54.480 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff0adde0:::benchmark_data_smithi179_77138_object101148:head# 2024-02-09T17:26:54.481 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff0b659b:::benchmark_data_smithi179_77138_object124456:head# 2024-02-09T17:26:54.481 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff0c6502:::benchmark_data_smithi179_77138_object118585:head# 2024-02-09T17:26:54.482 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff0cb190:::benchmark_data_smithi179_77138_object77831:head# 2024-02-09T17:26:54.482 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff0d0d54:::benchmark_data_smithi179_77138_object68536:head# 2024-02-09T17:26:54.483 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff0d4b9f:::benchmark_data_smithi179_77138_object30642:head# 2024-02-09T17:26:54.483 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff0de4d9:::benchmark_data_smithi179_77138_object129047:head# 2024-02-09T17:26:54.485 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff0e482b:::benchmark_data_smithi179_77138_object30683:head# 2024-02-09T17:26:54.485 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff0e7d2b:::benchmark_data_smithi179_77138_object99025:head# 2024-02-09T17:26:54.486 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff0f8798:::benchmark_data_smithi179_77138_object54641:head# 2024-02-09T17:26:54.486 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff0fd3b5:::benchmark_data_smithi179_77138_object97285:head# 2024-02-09T17:26:54.487 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff10188e:::benchmark_data_smithi179_77138_object61790:head# 2024-02-09T17:26:54.487 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff120a58:::benchmark_data_smithi179_77138_object39782:head# 2024-02-09T17:26:54.488 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff122fd0:::benchmark_data_smithi179_77138_object97994:head# 2024-02-09T17:26:54.488 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff131123:::benchmark_data_smithi179_77138_object33555:head# 2024-02-09T17:26:54.489 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff138902:::benchmark_data_smithi179_77138_object62451:head# 2024-02-09T17:26:54.491 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff141195:::benchmark_data_smithi179_77138_object43326:head# 2024-02-09T17:26:54.491 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff154065:::benchmark_data_smithi179_77138_object124634:head# 2024-02-09T17:26:54.491 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff15971e:::benchmark_data_smithi179_77138_object38439:head# 2024-02-09T17:26:54.492 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff164a51:::benchmark_data_smithi179_77138_object114483:head# 2024-02-09T17:26:54.492 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff172ae1:::benchmark_data_smithi179_77138_object88551:head# 2024-02-09T17:26:54.493 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff17f209:::benchmark_data_smithi179_77138_object118274:head# 2024-02-09T17:26:54.493 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff180f36:::benchmark_data_smithi179_77138_object126617:head# 2024-02-09T17:26:54.494 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff1877da:::benchmark_data_smithi179_77138_object105544:head# 2024-02-09T17:26:54.494 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff18a859:::benchmark_data_smithi179_77138_object116077:head# 2024-02-09T17:26:54.494 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff192c70:::benchmark_data_smithi179_77138_object130753:head# 2024-02-09T17:26:54.495 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff1a24a7:::benchmark_data_smithi179_77138_object67897:head# 2024-02-09T17:26:54.495 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:26:54.496 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff1a3835:::benchmark_data_smithi179_77138_object28871:head# 2024-02-09T17:26:54.496 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff1a4845:::benchmark_data_smithi179_77138_object42092:head# 2024-02-09T17:26:54.496 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff1d6dd3:::benchmark_data_smithi179_77138_object107917:head# 2024-02-09T17:26:54.496 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff1ea826:::benchmark_data_smithi179_77138_object132951:head# 2024-02-09T17:26:54.497 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff1f6804:::benchmark_data_smithi179_77138_object130691:head# 2024-02-09T17:26:54.498 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff21a2e7:::benchmark_data_smithi179_77138_object22721:head# 2024-02-09T17:26:54.498 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff222800:::benchmark_data_smithi179_77138_object109339:head# 2024-02-09T17:26:54.498 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff22d98b:::benchmark_data_smithi179_77138_object96135:head# 2024-02-09T17:26:54.499 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff22deb2:::benchmark_data_smithi179_77138_object103957:head# 2024-02-09T17:26:54.500 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff239ee5:::benchmark_data_smithi179_77138_object93828:head# 2024-02-09T17:26:54.500 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff245685:::benchmark_data_smithi179_77138_object114673:head# 2024-02-09T17:26:54.500 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff24af6c:::benchmark_data_smithi179_77138_object102459:head# 2024-02-09T17:26:54.501 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff25cf1b:::benchmark_data_smithi179_77138_object67078:head# 2024-02-09T17:26:54.501 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff262869:::benchmark_data_smithi179_77138_object89647:head# 2024-02-09T17:26:54.501 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff2645e6:::benchmark_data_smithi179_77138_object84468:head# 2024-02-09T17:26:54.502 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff26548e:::benchmark_data_smithi179_77138_object47592:head# 2024-02-09T17:26:54.505 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff270c9e:::benchmark_data_smithi179_77138_object23749:head# 2024-02-09T17:26:54.506 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff27261d:::benchmark_data_smithi179_77138_object105374:head# 2024-02-09T17:26:54.506 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff27f6c2:::benchmark_data_smithi179_77138_object68783:head# 2024-02-09T17:26:54.506 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff292b22:::benchmark_data_smithi179_77138_object91604:head# 2024-02-09T17:26:54.507 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff2940f9:::benchmark_data_smithi179_77138_object70907:head# 2024-02-09T17:26:54.507 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff298e90:::benchmark_data_smithi179_77138_object49838:head# 2024-02-09T17:26:54.508 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff29fec1:::benchmark_data_smithi179_77138_object68805:head# 2024-02-09T17:26:54.508 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff2a0a35:::benchmark_data_smithi179_77138_object107637:head# 2024-02-09T17:26:54.509 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff2a1414:::benchmark_data_smithi179_77138_object33739:head# 2024-02-09T17:26:54.509 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff2b1b8b:::benchmark_data_smithi179_77138_object44770:head# 2024-02-09T17:26:54.510 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff2bc313:::benchmark_data_smithi179_77138_object65917:head# 2024-02-09T17:26:54.510 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff2cb328:::benchmark_data_smithi179_77138_object30873:head# 2024-02-09T17:26:54.510 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff2cd011:::benchmark_data_smithi179_77138_object48366:head# 2024-02-09T17:26:54.511 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff2cd2e6:::benchmark_data_smithi179_77138_object22666:head# 2024-02-09T17:26:54.512 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff2dd79a:::benchmark_data_smithi179_77138_object108562:head# 2024-02-09T17:26:54.512 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff2ea0f8:::benchmark_data_smithi179_77138_object65775:head# 2024-02-09T17:26:54.513 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff2ec20e:::benchmark_data_smithi179_77138_object66293:head# 2024-02-09T17:26:54.513 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff2edffb:::benchmark_data_smithi179_77138_object69884:head# 2024-02-09T17:26:54.513 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff2ef597:::benchmark_data_smithi179_77138_object63826:head# 2024-02-09T17:26:54.514 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff2f8db3:::benchmark_data_smithi179_77138_object82826:head# 2024-02-09T17:26:54.514 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff308bf6:::benchmark_data_smithi179_77138_object61822:head# 2024-02-09T17:26:54.517 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff318800:::benchmark_data_smithi179_77138_object78515:head# 2024-02-09T17:26:54.518 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff3205a4:::benchmark_data_smithi179_77138_object74381:head# 2024-02-09T17:26:54.518 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff32566d:::benchmark_data_smithi179_77138_object114399:head# 2024-02-09T17:26:54.518 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff327c19:::benchmark_data_smithi179_77138_object116796:head# 2024-02-09T17:26:54.519 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff32a665:::benchmark_data_smithi179_77138_object129551:head# 2024-02-09T17:26:54.519 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff330b42:::benchmark_data_smithi179_77138_object25910:head# 2024-02-09T17:26:54.520 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff331af7:::benchmark_data_smithi179_77138_object117648:head# 2024-02-09T17:26:54.520 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff33ce0d:::benchmark_data_smithi179_77138_object77673:head# 2024-02-09T17:26:54.521 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff340dd3:::benchmark_data_smithi179_77138_object25896:head# 2024-02-09T17:26:54.521 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff34addd:::benchmark_data_smithi179_77138_object62209:head# 2024-02-09T17:26:54.522 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff34b2b3:::benchmark_data_smithi179_77138_object129343:head# 2024-02-09T17:26:54.522 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff34e55f:::benchmark_data_smithi179_77138_object73451:head# 2024-02-09T17:26:54.522 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff34f880:::benchmark_data_smithi179_77138_object78454:head# 2024-02-09T17:26:54.523 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff35542b:::benchmark_data_smithi179_77138_object71142:head# 2024-02-09T17:26:54.523 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff36974d:::benchmark_data_smithi179_77138_object40654:head# 2024-02-09T17:26:54.526 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff36f3f4:::benchmark_data_smithi179_77138_object44434:head# 2024-02-09T17:26:54.526 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff373d35:::benchmark_data_smithi179_77138_object116085:head# 2024-02-09T17:26:54.526 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff3789c2:::benchmark_data_smithi179_77138_object23573:head# 2024-02-09T17:26:54.526 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff393588:::benchmark_data_smithi179_77138_object60422:head# 2024-02-09T17:26:54.526 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff395d61:::benchmark_data_smithi179_77138_object117329:head# 2024-02-09T17:26:54.527 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff39b6a3:::benchmark_data_smithi179_77138_object27343:head# 2024-02-09T17:26:54.527 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff3b5967:::benchmark_data_smithi179_77138_object96393:head# 2024-02-09T17:26:54.527 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff3c7baa:::benchmark_data_smithi179_77138_object119938:head# 2024-02-09T17:26:54.533 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff3ce26e:::benchmark_data_smithi179_77138_object25963:head# 2024-02-09T17:26:54.533 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff3d9a3f:::benchmark_data_smithi179_77138_object104468:head# 2024-02-09T17:26:54.533 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff3daf0d:::benchmark_data_smithi179_77138_object69772:head# 2024-02-09T17:26:54.533 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff3dc199:::benchmark_data_smithi179_77138_object89382:head# 2024-02-09T17:26:54.533 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff3ddc34:::benchmark_data_smithi179_77138_object125534:head# 2024-02-09T17:26:54.533 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff3e5333:::benchmark_data_smithi179_77138_object111005:head# 2024-02-09T17:26:54.533 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff3f3fad:::benchmark_data_smithi179_77138_object33327:head# 2024-02-09T17:26:54.533 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff40b6e4:::benchmark_data_smithi179_77138_object31971:head# 2024-02-09T17:26:54.533 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff4286e6:::benchmark_data_smithi179_77138_object71842:head# 2024-02-09T17:26:54.534 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff43015d:::benchmark_data_smithi179_77138_object39428:head# 2024-02-09T17:26:54.534 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff4302e0:::benchmark_data_smithi179_77138_object90955:head# 2024-02-09T17:26:54.534 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff435473:::benchmark_data_smithi179_77138_object125537:head# 2024-02-09T17:26:54.535 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff43ce97:::benchmark_data_smithi179_77138_object76717:head# 2024-02-09T17:26:54.535 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff442e6a:::benchmark_data_smithi179_77138_object65027:head# 2024-02-09T17:26:54.535 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff443d4f:::benchmark_data_smithi179_77138_object91204:head# 2024-02-09T17:26:54.535 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff448f0b:::benchmark_data_smithi179_77138_object131429:head# 2024-02-09T17:26:54.535 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff449afe:::benchmark_data_smithi179_77138_object54520:head# 2024-02-09T17:26:54.536 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff4592ae:::benchmark_data_smithi179_77138_object44704:head# 2024-02-09T17:26:54.536 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff45b7e2:::benchmark_data_smithi179_77138_object85064:head# 2024-02-09T17:26:54.537 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff45c582:::benchmark_data_smithi179_77138_object36032:head# 2024-02-09T17:26:54.537 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff464569:::benchmark_data_smithi179_77138_object126064:head# 2024-02-09T17:26:54.537 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff47776e:::benchmark_data_smithi179_77138_object78801:head# 2024-02-09T17:26:54.538 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff478ebf:::benchmark_data_smithi179_77138_object58976:head# 2024-02-09T17:26:54.538 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff47a164:::benchmark_data_smithi179_77138_object121102:head# 2024-02-09T17:26:54.539 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff480c21:::benchmark_data_smithi179_77138_object106434:head# 2024-02-09T17:26:54.539 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff491951:::benchmark_data_smithi179_77138_object102293:head# 2024-02-09T17:26:54.539 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff4a9b67:::benchmark_data_smithi179_77138_object46746:head# 2024-02-09T17:26:54.541 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff4a9f0e:::benchmark_data_smithi179_77138_object96523:head# 2024-02-09T17:26:54.542 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff4b7a32:::benchmark_data_smithi179_77138_object95280:head# 2024-02-09T17:26:54.542 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff4bc702:::benchmark_data_smithi179_77138_object122927:head# 2024-02-09T17:26:54.545 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff4be6ef:::benchmark_data_smithi179_77138_object123264:head# 2024-02-09T17:26:54.545 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff4c15c4:::benchmark_data_smithi179_77138_object63972:head# 2024-02-09T17:26:54.546 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff4c7da9:::benchmark_data_smithi179_77138_object66396:head# 2024-02-09T17:26:54.546 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff4d18af:::benchmark_data_smithi179_77138_object54065:head# 2024-02-09T17:26:54.546 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff4d8690:::benchmark_data_smithi179_77138_object64934:head# 2024-02-09T17:26:54.547 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff4e7fd3:::benchmark_data_smithi179_77138_object38982:head# 2024-02-09T17:26:54.547 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff4ef156:::benchmark_data_smithi179_77138_object51337:head# 2024-02-09T17:26:54.548 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff4fca5e:::benchmark_data_smithi179_77138_object122678:head# 2024-02-09T17:26:54.550 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff500c7a:::benchmark_data_smithi179_77138_object37309:head# 2024-02-09T17:26:54.550 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff506d09:::benchmark_data_smithi179_77138_object106762:head# 2024-02-09T17:26:54.550 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff50a38e:::benchmark_data_smithi179_77138_object131099:head# 2024-02-09T17:26:54.551 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff535ed3:::benchmark_data_smithi179_77138_object98955:head# 2024-02-09T17:26:54.551 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff53c36f:::benchmark_data_smithi179_77138_object61607:head# 2024-02-09T17:26:54.552 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:54.552 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff543b7b:::benchmark_data_smithi179_77138_object75422:head# 2024-02-09T17:26:54.553 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff546105:::benchmark_data_smithi179_77138_object35718:head# 2024-02-09T17:26:54.553 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff548f95:::benchmark_data_smithi179_77138_object94275:head# 2024-02-09T17:26:54.553 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff558076:::benchmark_data_smithi179_77138_object92704:head# 2024-02-09T17:26:54.554 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff55c924:::benchmark_data_smithi179_77138_object87566:head# 2024-02-09T17:26:54.554 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff5668fc:::benchmark_data_smithi179_77138_object99819:head# 2024-02-09T17:26:54.555 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff56b058:::benchmark_data_smithi179_77138_object55188:head# 2024-02-09T17:26:54.555 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff56b792:::benchmark_data_smithi179_77138_object110272:head# 2024-02-09T17:26:54.555 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff570521:::benchmark_data_smithi179_77138_object77889:head# 2024-02-09T17:26:54.556 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff570c53:::benchmark_data_smithi179_77138_object108035:head# 2024-02-09T17:26:54.558 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff576459:::benchmark_data_smithi179_77138_object79090:head# 2024-02-09T17:26:54.562 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff576b6b:::benchmark_data_smithi179_77138_object21334:head# 2024-02-09T17:26:54.562 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff57f039:::benchmark_data_smithi179_77138_object92606:head# 2024-02-09T17:26:54.562 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff582066:::benchmark_data_smithi179_77138_object79652:head# 2024-02-09T17:26:54.563 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff5846c1:::benchmark_data_smithi179_77138_object25173:head# 2024-02-09T17:26:54.563 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff586fba:::benchmark_data_smithi179_77138_object78054:head# 2024-02-09T17:26:54.564 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff592b9c:::benchmark_data_smithi179_77138_object104629:head# 2024-02-09T17:26:54.565 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff5a7285:::benchmark_data_smithi179_77138_object41191:head# 2024-02-09T17:26:54.565 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff5c587f:::benchmark_data_smithi179_77138_object129693:head# 2024-02-09T17:26:54.566 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff5e08b8:::benchmark_data_smithi179_77138_object76458:head# 2024-02-09T17:26:54.566 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff5e7f21:::benchmark_data_smithi179_77138_object84594:head# 2024-02-09T17:26:54.567 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff5eecbd:::benchmark_data_smithi179_77138_object22887:head# 2024-02-09T17:26:54.567 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff604da7:::benchmark_data_smithi179_77138_object132769:head# 2024-02-09T17:26:54.568 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff60819b:::benchmark_data_smithi179_77138_object130191:head# 2024-02-09T17:26:54.568 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff609ac6:::benchmark_data_smithi179_77138_object123096:head# 2024-02-09T17:26:54.569 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff60b234:::benchmark_data_smithi179_77138_object98136:head# 2024-02-09T17:26:54.569 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff619778:::benchmark_data_smithi179_77138_object36142:head# 2024-02-09T17:26:54.569 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff627541:::benchmark_data_smithi179_77138_object42807:head# 2024-02-09T17:26:54.570 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff628895:::benchmark_data_smithi179_77138_object21922:head# 2024-02-09T17:26:54.570 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff62b882:::benchmark_data_smithi179_77138_object94144:head# 2024-02-09T17:26:54.571 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff63000b:::benchmark_data_smithi179_77138_object59343:head# 2024-02-09T17:26:54.571 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff638a2d:::benchmark_data_smithi179_77138_object113455:head# 2024-02-09T17:26:54.571 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff6512f7:::benchmark_data_smithi179_77138_object33575:head# 2024-02-09T17:26:54.572 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff6568a9:::benchmark_data_smithi179_77138_object87115:head# 2024-02-09T17:26:54.572 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff65a5d7:::benchmark_data_smithi179_77138_object118518:head# 2024-02-09T17:26:54.574 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff661100:::benchmark_data_smithi179_77138_object89917:head# 2024-02-09T17:26:54.575 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff66ab41:::benchmark_data_smithi179_77138_object49055:head# 2024-02-09T17:26:54.575 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff67ef64:::benchmark_data_smithi179_77138_object114400:head# 2024-02-09T17:26:54.575 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff68520d:::benchmark_data_smithi179_77138_object117195:head# 2024-02-09T17:26:54.576 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff68911f:::benchmark_data_smithi179_77138_object21593:head# 2024-02-09T17:26:54.576 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff68d074:::benchmark_data_smithi179_77138_object99173:head# 2024-02-09T17:26:54.577 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff6917d0:::benchmark_data_smithi179_77138_object92833:head# 2024-02-09T17:26:54.577 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff694997:::benchmark_data_smithi179_77138_object82350:head# 2024-02-09T17:26:54.577 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff6a169b:::benchmark_data_smithi179_77138_object45482:head# 2024-02-09T17:26:54.578 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff6aa81f:::benchmark_data_smithi179_77138_object94604:head# 2024-02-09T17:26:54.578 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff6af65f:::benchmark_data_smithi179_77138_object22308:head# 2024-02-09T17:26:54.579 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff6b3de3:::benchmark_data_smithi179_77138_object107343:head# 2024-02-09T17:26:54.579 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff6bcf12:::benchmark_data_smithi179_77138_object103407:head# 2024-02-09T17:26:54.580 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff6c8781:::benchmark_data_smithi179_77138_object61673:head# 2024-02-09T17:26:54.580 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff6ca901:::benchmark_data_smithi179_77138_object34133:head# 2024-02-09T17:26:54.580 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff6cf4e8:::benchmark_data_smithi179_77138_object69861:head# 2024-02-09T17:26:54.581 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff6d3d78:::benchmark_data_smithi179_77138_object70972:head# 2024-02-09T17:26:54.581 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff6d3ee1:::benchmark_data_smithi179_77138_object36602:head# 2024-02-09T17:26:54.582 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff6d406a:::benchmark_data_smithi179_77138_object69835:head# 2024-02-09T17:26:54.584 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff6e40e1:::benchmark_data_smithi179_77138_object89485:head# 2024-02-09T17:26:54.585 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff6ee750:::benchmark_data_smithi179_77138_object72946:head# 2024-02-09T17:26:54.585 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff6f1ebd:::benchmark_data_smithi179_77138_object124810:head# 2024-02-09T17:26:54.585 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff6f7edd:::benchmark_data_smithi179_77138_object90606:head# 2024-02-09T17:26:54.585 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff6fc846:::benchmark_data_smithi179_77138_object125979:head# 2024-02-09T17:26:54.585 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff711edb:::benchmark_data_smithi179_77138_object24663:head# 2024-02-09T17:26:54.585 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff721d0f:::benchmark_data_smithi179_77138_object122370:head# 2024-02-09T17:26:54.585 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff7237e4:::benchmark_data_smithi179_77138_object66742:head# 2024-02-09T17:26:54.586 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff724872:::benchmark_data_smithi179_77138_object44144:head# 2024-02-09T17:26:54.586 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff72a4ef:::benchmark_data_smithi179_77138_object53301:head# 2024-02-09T17:26:54.587 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff75b000:::benchmark_data_smithi179_77138_object42094:head# 2024-02-09T17:26:54.587 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff767580:::benchmark_data_smithi179_77138_object53148:head# 2024-02-09T17:26:54.588 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff76be6d:::benchmark_data_smithi179_77138_object64960:head# 2024-02-09T17:26:54.588 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff76d0ad:::benchmark_data_smithi179_77138_object115247:head# 2024-02-09T17:26:54.588 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff7716b4:::benchmark_data_smithi179_77138_object102331:head# 2024-02-09T17:26:54.589 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff771f5e:::benchmark_data_smithi179_77138_object97626:head# 2024-02-09T17:26:54.589 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff788bdb:::benchmark_data_smithi179_77138_object46442:head# 2024-02-09T17:26:54.590 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff78cb9b:::benchmark_data_smithi179_77138_object97004:head# 2024-02-09T17:26:54.590 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff78e9a5:::benchmark_data_smithi179_77138_object125789:head# 2024-02-09T17:26:54.591 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff79b1eb:::benchmark_data_smithi179_77138_object85124:head# 2024-02-09T17:26:54.591 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff7d1840:::benchmark_data_smithi179_77138_object34057:head# 2024-02-09T17:26:54.591 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff7ddaee:::benchmark_data_smithi179_77138_object66800:head# 2024-02-09T17:26:54.592 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff7e5225:::benchmark_data_smithi179_77138_object62691:head# 2024-02-09T17:26:54.592 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff7e9997:::benchmark_data_smithi179_77138_object42369:head# 2024-02-09T17:26:54.593 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff7ecf36:::benchmark_data_smithi179_77138_object35101:head# 2024-02-09T17:26:54.593 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff7ef6ca:::benchmark_data_smithi179_77138_object132322:head# 2024-02-09T17:26:54.593 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff7efb96:::benchmark_data_smithi179_77138_object79824:head# 2024-02-09T17:26:54.594 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff7f8fb3:::benchmark_data_smithi179_77138_object54591:head# 2024-02-09T17:26:54.594 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff7f95e0:::benchmark_data_smithi179_77138_object31297:head# 2024-02-09T17:26:54.595 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff80272e:::benchmark_data_smithi179_77138_object54607:head# 2024-02-09T17:26:54.595 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff8054da:::benchmark_data_smithi179_77138_object95547:head# 2024-02-09T17:26:54.596 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff80963d:::benchmark_data_smithi179_77138_object126052:head# 2024-02-09T17:26:54.596 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff8139f5:::benchmark_data_smithi179_77138_object115672:head# 2024-02-09T17:26:54.596 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff81de59:::benchmark_data_smithi179_77138_object47153:head# 2024-02-09T17:26:54.597 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff82456d:::benchmark_data_smithi179_77138_object44797:head# 2024-02-09T17:26:54.597 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff835dc7:::benchmark_data_smithi179_77138_object72548:head# 2024-02-09T17:26:54.597 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff83b768:::benchmark_data_smithi179_77138_object111946:head# 2024-02-09T17:26:54.598 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff83c09e:::benchmark_data_smithi179_77138_object109815:head# 2024-02-09T17:26:54.598 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff8409ae:::benchmark_data_smithi179_77138_object121344:head# 2024-02-09T17:26:54.599 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff861113:::benchmark_data_smithi179_77138_object114291:head# 2024-02-09T17:26:54.599 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff87f3d6:::benchmark_data_smithi179_77138_object133179:head# 2024-02-09T17:26:54.599 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff880cb5:::benchmark_data_smithi179_77138_object107959:head# 2024-02-09T17:26:54.600 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff8a2b0b:::benchmark_data_smithi179_77138_object62689:head# 2024-02-09T17:26:54.600 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff8a4768:::benchmark_data_smithi179_77138_object29470:head# 2024-02-09T17:26:54.600 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff8c3557:::benchmark_data_smithi179_77138_object38964:head# 2024-02-09T17:26:54.601 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff8c9b47:::benchmark_data_smithi179_77138_object19600:head# 2024-02-09T17:26:54.601 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff8ccb7e:::benchmark_data_smithi179_77138_object96732:head# 2024-02-09T17:26:54.602 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff8dbb7c:::benchmark_data_smithi179_77138_object125120:head# 2024-02-09T17:26:54.602 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff8e058c:::benchmark_data_smithi179_77138_object88271:head# 2024-02-09T17:26:54.603 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff8ed459:::benchmark_data_smithi179_77138_object33490:head# 2024-02-09T17:26:54.603 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff8f2403:::benchmark_data_smithi179_77138_object64569:head# 2024-02-09T17:26:54.603 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff9035ea:::benchmark_data_smithi179_77138_object45901:head# 2024-02-09T17:26:54.604 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff905f93:::benchmark_data_smithi179_77138_object33062:head# 2024-02-09T17:26:54.604 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff908961:::benchmark_data_smithi179_77138_object58075:head# 2024-02-09T17:26:54.605 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff919e53:::benchmark_data_smithi179_77138_object90591:head# 2024-02-09T17:26:54.605 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff91bb3a:::benchmark_data_smithi179_77138_object126477:head# 2024-02-09T17:26:54.606 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff91eee5:::benchmark_data_smithi179_77138_object46749:head# 2024-02-09T17:26:54.606 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff92decd:::benchmark_data_smithi179_77138_object54238:head# 2024-02-09T17:26:54.607 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff92fa77:::benchmark_data_smithi179_77138_object56593:head# 2024-02-09T17:26:54.607 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff933dc4:::benchmark_data_smithi179_77138_object70468:head# 2024-02-09T17:26:54.607 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff93e5c1:::benchmark_data_smithi179_77138_object77103:head# 2024-02-09T17:26:54.608 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff9451f1:::benchmark_data_smithi179_77138_object92910:head# 2024-02-09T17:26:54.608 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff94f8ce:::benchmark_data_smithi179_77138_object46679:head# 2024-02-09T17:26:54.609 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff9518f8:::benchmark_data_smithi179_77138_object82246:head# 2024-02-09T17:26:54.609 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff9900eb:::benchmark_data_smithi179_77138_object57030:head# 2024-02-09T17:26:54.610 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff992af2:::benchmark_data_smithi179_77138_object65314:head# 2024-02-09T17:26:54.610 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff9993f4:::benchmark_data_smithi179_77138_object112875:head# 2024-02-09T17:26:54.610 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff9997f6:::benchmark_data_smithi179_77138_object71098:head# 2024-02-09T17:26:54.611 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff9a1ff5:::benchmark_data_smithi179_77138_object119297:head# 2024-02-09T17:26:54.611 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff9bcc03:::benchmark_data_smithi179_77138_object126845:head# 2024-02-09T17:26:54.612 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff9c47cd:::benchmark_data_smithi179_77138_object111336:head# 2024-02-09T17:26:54.612 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff9c645f:::benchmark_data_smithi179_77138_object49794:head# 2024-02-09T17:26:54.613 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff9d6201:::benchmark_data_smithi179_77138_object117218:head# 2024-02-09T17:26:54.613 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff9d94b0:::benchmark_data_smithi179_77138_object128019:head# 2024-02-09T17:26:54.613 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff9e644e:::benchmark_data_smithi179_77138_object133338:head# 2024-02-09T17:26:54.614 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff9ecb13:::benchmark_data_smithi179_77138_object49197:head# 2024-02-09T17:26:54.614 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff9f4fb7:::benchmark_data_smithi179_77138_object114285:head# 2024-02-09T17:26:54.615 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ff9f96b3:::benchmark_data_smithi179_77138_object56408:head# 2024-02-09T17:26:54.615 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffa14601:::benchmark_data_smithi179_77138_object33140:head# 2024-02-09T17:26:54.616 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffa1627e:::benchmark_data_smithi179_77138_object37492:head# 2024-02-09T17:26:54.616 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffa1bef5:::benchmark_data_smithi179_77138_object77721:head# 2024-02-09T17:26:54.617 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffa37e40:::benchmark_data_smithi179_77138_object68606:head# 2024-02-09T17:26:54.617 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffa52d61:::benchmark_data_smithi179_77138_object108738:head# 2024-02-09T17:26:54.620 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffa5484a:::benchmark_data_smithi179_77138_object109081:head# 2024-02-09T17:26:54.620 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffa5512c:::benchmark_data_smithi179_77138_object131475:head# 2024-02-09T17:26:54.620 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffa596cb:::benchmark_data_smithi179_77138_object129761:head# 2024-02-09T17:26:54.621 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffa5eae4:::benchmark_data_smithi179_77138_object97207:head# 2024-02-09T17:26:54.621 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffa66e4b:::benchmark_data_smithi179_77138_object24629:head# 2024-02-09T17:26:54.622 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffa731b0:::benchmark_data_smithi179_77138_object59314:head# 2024-02-09T17:26:54.623 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffa77c9b:::benchmark_data_smithi179_77138_object112498:head# 2024-02-09T17:26:54.623 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffa791dc:::benchmark_data_smithi179_77138_object106790:head# 2024-02-09T17:26:54.623 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffa7f15f:::benchmark_data_smithi179_77138_object61745:head# 2024-02-09T17:26:54.624 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffa8979a:::benchmark_data_smithi179_77138_object26323:head# 2024-02-09T17:26:54.624 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffa9e8c0:::benchmark_data_smithi179_77138_object83195:head# 2024-02-09T17:26:54.625 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffaa7898:::benchmark_data_smithi179_77138_object92049:head# 2024-02-09T17:26:54.625 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffab8b26:::benchmark_data_smithi179_77138_object79217:head# 2024-02-09T17:26:54.625 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffab8c99:::benchmark_data_smithi179_77138_object81406:head# 2024-02-09T17:26:54.625 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffad6bfd:::benchmark_data_smithi179_77138_object42620:head# 2024-02-09T17:26:54.626 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffad8a09:::benchmark_data_smithi179_77138_object67029:head# 2024-02-09T17:26:54.627 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffadcc70:::benchmark_data_smithi179_77138_object71239:head# 2024-02-09T17:26:54.629 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffae1c61:::benchmark_data_smithi179_77138_object69168:head# 2024-02-09T17:26:54.629 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffae75e7:::benchmark_data_smithi179_77138_object44115:head# 2024-02-09T17:26:54.630 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffaf9eb7:::benchmark_data_smithi179_77138_object86750:head# 2024-02-09T17:26:54.630 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffb07889:::benchmark_data_smithi179_77138_object98295:head# 2024-02-09T17:26:54.631 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffb11d6c:::benchmark_data_smithi179_77138_object106666:head# 2024-02-09T17:26:54.631 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffb1286a:::benchmark_data_smithi179_77138_object49306:head# 2024-02-09T17:26:54.631 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffb18e12:::benchmark_data_smithi179_77138_object51041:head# 2024-02-09T17:26:54.632 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffb29ce9:::benchmark_data_smithi179_77138_object85101:head# 2024-02-09T17:26:54.632 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffb5ef7d:::benchmark_data_smithi179_77138_object60856:head# 2024-02-09T17:26:54.632 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffb6572c:::benchmark_data_smithi179_77138_object71617:head# 2024-02-09T17:26:54.633 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffb68df8:::benchmark_data_smithi179_77138_object55849:head# 2024-02-09T17:26:54.633 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffb7a75e:::benchmark_data_smithi179_77138_object42090:head# 2024-02-09T17:26:54.634 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffb7f4b1:::benchmark_data_smithi179_77138_object125998:head# 2024-02-09T17:26:54.634 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffb85a56:::benchmark_data_smithi179_77138_object123252:head# 2024-02-09T17:26:54.636 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffb9b27d:::benchmark_data_smithi179_77138_object88561:head# 2024-02-09T17:26:54.636 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffba4dca:::benchmark_data_smithi179_77138_object90960:head# 2024-02-09T17:26:54.636 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffbaf2d0:::benchmark_data_smithi179_77138_object125306:head# 2024-02-09T17:26:54.637 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffbb32c4:::benchmark_data_smithi179_77138_object36082:head# 2024-02-09T17:26:54.638 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffbb4754:::benchmark_data_smithi179_77138_object25729:head# 2024-02-09T17:26:54.638 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffbd34f5:::benchmark_data_smithi179_77138_object29766:head# 2024-02-09T17:26:54.638 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffbd482a:::benchmark_data_smithi179_77138_object116110:head# 2024-02-09T17:26:54.639 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffbd6c47:::benchmark_data_smithi179_77138_object112797:head# 2024-02-09T17:26:54.639 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffbe5230:::benchmark_data_smithi179_77138_object52279:head# 2024-02-09T17:26:54.640 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffbfb322:::benchmark_data_smithi179_77138_object47669:head# 2024-02-09T17:26:54.640 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffc05327:::benchmark_data_smithi179_77138_object132611:head# 2024-02-09T17:26:54.641 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffc08b7a:::benchmark_data_smithi179_77138_object112406:head# 2024-02-09T17:26:54.641 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffc2a232:::benchmark_data_smithi179_77138_object96191:head# 2024-02-09T17:26:54.641 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffc2baca:::benchmark_data_smithi179_77138_object88758:head# 2024-02-09T17:26:54.642 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffc2c381:::benchmark_data_smithi179_77138_object25097:head# 2024-02-09T17:26:54.645 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffc35124:::benchmark_data_smithi179_77138_object88099:head# 2024-02-09T17:26:54.645 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffc4bc7b:::benchmark_data_smithi179_77138_object87571:head# 2024-02-09T17:26:54.646 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffc507a7:::benchmark_data_smithi179_77138_object57060:head# 2024-02-09T17:26:54.646 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffc66fc3:::benchmark_data_smithi179_77138_object126858:head# 2024-02-09T17:26:54.646 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffc6bddf:::benchmark_data_smithi179_77138_object63017:head# 2024-02-09T17:26:54.647 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffc6f2df:::benchmark_data_smithi179_77138_object123867:head# 2024-02-09T17:26:54.647 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffc7ad8f:::benchmark_data_smithi179_77138_object104383:head# 2024-02-09T17:26:54.648 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffc7de18:::benchmark_data_smithi179_77138_object26022:head# 2024-02-09T17:26:54.648 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffcaba87:::benchmark_data_smithi179_77138_object71147:head# 2024-02-09T17:26:54.650 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffcabe00:::benchmark_data_smithi179_77138_object52458:head# 2024-02-09T17:26:54.650 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffcaf3c2:::benchmark_data_smithi179_77138_object103433:head# 2024-02-09T17:26:54.650 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffcb19dc:::benchmark_data_smithi179_77138_object99811:head# 2024-02-09T17:26:54.650 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffce53a8:::benchmark_data_smithi179_77138_object110415:head# 2024-02-09T17:26:54.650 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffce8f1b:::benchmark_data_smithi179_77138_object66231:head# 2024-02-09T17:26:54.651 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffcffaa6:::benchmark_data_smithi179_77138_object27915:head# 2024-02-09T17:26:54.651 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffd13d41:::benchmark_data_smithi179_77138_object81578:head# 2024-02-09T17:26:54.652 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffd1ca07:::benchmark_data_smithi179_77138_object45912:head# 2024-02-09T17:26:54.652 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffd2ac1a:::benchmark_data_smithi179_77138_object117640:head# 2024-02-09T17:26:54.653 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffd3563a:::benchmark_data_smithi179_77138_object113040:head# 2024-02-09T17:26:54.653 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffd376da:::benchmark_data_smithi179_77138_object52018:head# 2024-02-09T17:26:54.653 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffd396ab:::benchmark_data_smithi179_77138_object120601:head# 2024-02-09T17:26:54.654 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffd3a552:::benchmark_data_smithi179_77138_object101414:head# 2024-02-09T17:26:54.654 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffd3ef88:::benchmark_data_smithi179_77138_object106462:head# 2024-02-09T17:26:54.655 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffd40528:::benchmark_data_smithi179_77138_object129841:head# 2024-02-09T17:26:54.658 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffd64556:::benchmark_data_smithi179_77138_object26795:head# 2024-02-09T17:26:54.658 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffd778a7:::benchmark_data_smithi179_77138_object94540:head# 2024-02-09T17:26:54.658 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffd85df5:::benchmark_data_smithi179_77138_object133106:head# 2024-02-09T17:26:54.658 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffd90254:::benchmark_data_smithi179_77138_object62203:head# 2024-02-09T17:26:54.658 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffd956c5:::benchmark_data_smithi179_77138_object121528:head# 2024-02-09T17:26:54.659 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffda407f:::benchmark_data_smithi179_77138_object81454:head# 2024-02-09T17:26:54.659 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffdb475d:::benchmark_data_smithi179_77138_object122731:head# 2024-02-09T17:26:54.659 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffdbcc47:::benchmark_data_smithi179_77138_object35944:head# 2024-02-09T17:26:54.659 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffdc1169:::benchmark_data_smithi179_77138_object78135:head# 2024-02-09T17:26:54.660 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffdc3d7c:::benchmark_data_smithi179_77138_object48157:head# 2024-02-09T17:26:54.660 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffdca279:::benchmark_data_smithi179_77138_object105417:head# 2024-02-09T17:26:54.662 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffdcb1ad:::benchmark_data_smithi179_77138_object23087:head# 2024-02-09T17:26:54.662 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffddd849:::benchmark_data_smithi179_77138_object68349:head# 2024-02-09T17:26:54.662 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:54 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3660921371' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:26:54.662 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:54 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:26:54.663 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffde3d8d:::benchmark_data_smithi179_77138_object40393:head# 2024-02-09T17:26:54.663 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffde689f:::benchmark_data_smithi179_77138_object128700:head# 2024-02-09T17:26:54.663 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffdf1fba:::benchmark_data_smithi179_77138_object28313:head# 2024-02-09T17:26:54.664 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffdfc7fb:::benchmark_data_smithi179_77138_object76771:head# 2024-02-09T17:26:54.664 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffe0908c:::benchmark_data_smithi179_77138_object125648:head# 2024-02-09T17:26:54.665 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffe33fb1:::benchmark_data_smithi179_77138_object96769:head# 2024-02-09T17:26:54.665 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffe3e4bb:::benchmark_data_smithi179_77138_object66091:head# 2024-02-09T17:26:54.666 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffe40994:::benchmark_data_smithi179_77138_object42928:head# 2024-02-09T17:26:54.666 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffe43aae:::benchmark_data_smithi179_77138_object22619:head# 2024-02-09T17:26:54.667 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffe44e65:::benchmark_data_smithi179_77138_object27612:head# 2024-02-09T17:26:54.667 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffe50551:::benchmark_data_smithi179_77138_object70133:head# 2024-02-09T17:26:54.667 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffe76b59:::benchmark_data_smithi179_77138_object107634:head# 2024-02-09T17:26:54.668 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffe7f740:::benchmark_data_smithi179_77138_object41711:head# 2024-02-09T17:26:54.669 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffe8a376:::benchmark_data_smithi179_77138_object27913:head# 2024-02-09T17:26:54.670 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffe8b48b:::benchmark_data_smithi179_77138_object41376:head# 2024-02-09T17:26:54.670 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffe945a7:::benchmark_data_smithi179_77138_object129261:head# 2024-02-09T17:26:54.671 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffea7e28:::benchmark_data_smithi179_77138_object45447:head# 2024-02-09T17:26:54.671 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffeb4f01:::benchmark_data_smithi179_77138_object67500:head# 2024-02-09T17:26:54.674 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffeb8ca4:::benchmark_data_smithi179_77138_object100023:head# 2024-02-09T17:26:54.675 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffed7260:::benchmark_data_smithi179_77138_object68433:head# 2024-02-09T17:26:54.675 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffed96ff:::benchmark_data_smithi179_77138_object85392:head# 2024-02-09T17:26:54.676 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffed9764:::benchmark_data_smithi179_77138_object56590:head# 2024-02-09T17:26:54.676 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:26:54.676 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffee906c:::benchmark_data_smithi179_77138_object45574:head# 2024-02-09T17:26:54.677 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffeed7ba:::benchmark_data_smithi179_77138_object20294:head# 2024-02-09T17:26:54.677 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffefb832:::benchmark_data_smithi179_77138_object84643:head# 2024-02-09T17:26:54.677 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffefb8eb:::benchmark_data_smithi179_77138_object29347:head# 2024-02-09T17:26:54.678 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffefeb9a:::benchmark_data_smithi179_77138_object62514:head# 2024-02-09T17:26:54.678 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fff05de5:::benchmark_data_smithi179_77138_object102592:head# 2024-02-09T17:26:54.679 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fff07db2:::benchmark_data_smithi179_77138_object63769:head# 2024-02-09T17:26:54.679 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fff0e7df:::benchmark_data_smithi179_77138_object57890:head# 2024-02-09T17:26:54.681 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fff27efd:::benchmark_data_smithi179_77138_object95249:head# 2024-02-09T17:26:54.682 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fff2a1a5:::benchmark_data_smithi179_77138_object96780:head# 2024-02-09T17:26:54.682 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fff2fc92:::benchmark_data_smithi179_77138_object64190:head# 2024-02-09T17:26:54.683 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is unset 2024-02-09T17:26:54.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:54.684 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fff358fb:::benchmark_data_smithi179_77138_object104546:head# 2024-02-09T17:26:54.684 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fff3d3ea:::benchmark_data_smithi179_77138_object130682:head# 2024-02-09T17:26:54.684 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fff3d706:::benchmark_data_smithi179_77138_object45467:head# 2024-02-09T17:26:54.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:26:54.685 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fff5e815:::benchmark_data_smithi179_77138_object47811:head# 2024-02-09T17:26:54.685 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fff626b4:::benchmark_data_smithi179_77138_object122720:head# 2024-02-09T17:26:54.685 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fff6d277:::benchmark_data_smithi179_77138_object109155:head# 2024-02-09T17:26:54.685 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fff74b41:::benchmark_data_smithi179_77138_object118660:head# 2024-02-09T17:26:54.686 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fff93583:::benchmark_data_smithi179_77138_object23131:head# 2024-02-09T17:26:54.686 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fff9f60c:::benchmark_data_smithi179_77138_object62999:head# 2024-02-09T17:26:54.687 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fffa1ff9:::benchmark_data_smithi179_77138_object26494:head# 2024-02-09T17:26:54.687 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fffa61f6:::benchmark_data_smithi179_77138_object61184:head# 2024-02-09T17:26:54.687 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fffaf8af:::benchmark_data_smithi179_77138_object105171:head# 2024-02-09T17:26:54.687 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fffbae02:::benchmark_data_smithi179_77138_object48692:head# 2024-02-09T17:26:54.688 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fffc8031:::benchmark_data_smithi179_77138_object118162:head# 2024-02-09T17:26:54.688 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fffcd3d5:::benchmark_data_smithi179_77138_object83845:head# 2024-02-09T17:26:54.689 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fffd1eb2:::benchmark_data_smithi179_77138_object68998:head# 2024-02-09T17:26:54.689 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fffd2fa3:::benchmark_data_smithi179_77138_object64490:head# 2024-02-09T17:26:54.689 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fffd36f6:::benchmark_data_smithi179_77138_object81192:head# 2024-02-09T17:26:54.690 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fffd5f7c:::benchmark_data_smithi179_77138_object119124:head# 2024-02-09T17:26:54.690 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fffd6e0d:::benchmark_data_smithi179_77138_object127538:head# 2024-02-09T17:26:54.691 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fffdff23:::benchmark_data_smithi179_77138_object132726:head# 2024-02-09T17:26:54.691 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fffe844c:::benchmark_data_smithi179_77138_object111341:head# 2024-02-09T17:26:54.691 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:fffebbea:::benchmark_data_smithi179_77138_object127823:head# 2024-02-09T17:26:54.692 INFO:teuthology.orchestra.run.smithi179.stderr:Read #2:ffff8a59:::benchmark_data_smithi179_77138_object128340:head# 2024-02-09T17:26:54.696 INFO:teuthology.orchestra.run.smithi179.stderr:Export successful 2024-02-09T17:26:54.785 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:26:54.814 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:54 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3660921371' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:26:54.814 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:54 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:26:54.818 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:54.864 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:26:54.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:54.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:26:54.886 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-09T17:26:54.970 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:26:54.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:54 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3660921371' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:26:54.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:54 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:26:55.020 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:55.106 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:26:55.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:55.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:26:55.220 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:26:55.258 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:26:55.261 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:55.293 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:26:55.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:55.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:26:55.399 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:26:55.471 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:55.524 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:26:55.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:55.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:26:55.632 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:26:55.663 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:55.746 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:26:55.800 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:26:55.815 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:55 smithi162 ceph-mon[44304]: pgmap v454: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 6.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 842 op/s; 26071/312909 objects degraded (8.332%) 2024-02-09T17:26:55.815 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:55 smithi162 ceph-mon[44304]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:26:55.815 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:55 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:26:55.815 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:55 smithi162 ceph-mon[44304]: osdmap e150: 12 total, 11 up, 12 in 2024-02-09T17:26:55.848 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:26:55.898 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:26:55.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:55 smithi179 ceph-mon[43975]: pgmap v454: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 6.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 842 op/s; 26071/312909 objects degraded (8.332%) 2024-02-09T17:26:55.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:55 smithi179 ceph-mon[43975]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:26:55.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:55 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:26:55.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:55 smithi179 ceph-mon[43975]: osdmap e150: 12 total, 11 up, 12 in 2024-02-09T17:26:55.938 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:26:55.975 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:26:55.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:55 smithi099 ceph-mon[39420]: pgmap v454: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 6.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 842 op/s; 26071/312909 objects degraded (8.332%) 2024-02-09T17:26:55.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:55 smithi099 ceph-mon[39420]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:26:55.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:55 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:26:55.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:55 smithi099 ceph-mon[39420]: osdmap e150: 12 total, 11 up, 12 in 2024-02-09T17:26:56.028 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:26:56.086 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:26:56.093 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell -n osd.8 --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-8 --journal-path /var/lib/ceph/osd/ceph-8/journal --force --op remove --pgid 2.f 2024-02-09T17:26:56.121 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:26:56.124 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:26:56.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:56.128 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:26:56.158 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:26:56.203 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:26:56.203 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:26:56.203 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:26:56.203 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:26:56.203 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:26:56.203 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:26:56.203 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:26:56.203 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:26:56.204 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:26:56.204 INFO:teuthology.orchestra.run.smithi099.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-02-09T17:26:56.204 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:26:56.204 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:26:56.204 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:26:56.228 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:26:56.281 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:56.306 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:26:56.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:56.313 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:26:56.402 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/osd.8/config 2024-02-09T17:26:56.414 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:26:56.445 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:56.696 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:26:56.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:56.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:26:56.800 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:26:56.848 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:56.950 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:26:56.955 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:56 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2340215070' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:26:56.955 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:56 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 26071/312909 objects degraded (8.332%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:26:56.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:56.956 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:26:56.998 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:56 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2340215070' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:26:56.998 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:56 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 26071/312909 objects degraded (8.332%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:26:57.012 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:56 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2340215070' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:26:57.012 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:56 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 26071/312909 objects degraded (8.332%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:26:57.057 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:26:57.097 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:57.175 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:26:57.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:57.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:26:57.281 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:26:57.317 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:57.326 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:26:57.326 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-09T17:26:57.326 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-02-09T17:26:57.326 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-02-09T17:26:57.326 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:26:57.337 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:26:57.339 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:57.340 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:26:57.441 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:26:57.478 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:57.970 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:57 smithi179 ceph-mon[43975]: pgmap v456: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 6.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 25512/306201 objects degraded (8.332%) 2024-02-09T17:26:57.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:57 smithi099 ceph-mon[39420]: pgmap v456: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 6.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 25512/306201 objects degraded (8.332%) 2024-02-09T17:26:58.041 INFO:teuthology.orchestra.run.smithi179.stderr:Creating an OSD daemon form without an OSD FSID value 2024-02-09T17:26:58.042 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:26:58.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:58.046 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:26:58.084 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:57 smithi162 ceph-mon[44304]: pgmap v456: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 6.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 25512/306201 objects degraded (8.332%) 2024-02-09T17:26:58.085 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-09T17:26:58.124 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-09T17:26:58.146 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:26:58.208 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:58.250 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:26:58.253 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:58.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:26:58.296 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-09T17:26:58.297 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-09T17:26:58.354 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:26:58.392 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:58.431 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:26:58.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:58.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:26:58.457 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:26:58.535 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:26:58.538 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:26:58.594 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:58.683 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:26:58.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:58.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:26:58.793 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:26:58.795 INFO:teuthology.orchestra.run.smithi179.stdout: marking collection for removal 2024-02-09T17:26:58.795 INFO:teuthology.orchestra.run.smithi179.stdout:setting '_remove' omap key 2024-02-09T17:26:58.795 INFO:teuthology.orchestra.run.smithi179.stdout:finish_remove_pgs 2.f_head removing 2.f 2024-02-09T17:26:58.847 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:58.875 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:26:58.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:26:58.880 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:26:58.967 INFO:teuthology.orchestra.run.smithi179.stdout:Remove successful 2024-02-09T17:26:58.981 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:26:59.025 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:26:59.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:26:59 smithi099 ceph-mon[39420]: pgmap v457: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 6.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 734 op/s; 25512/304596 objects degraded (8.376%) 2024-02-09T17:27:00.079 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:26:59 smithi179 ceph-mon[43975]: pgmap v457: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 6.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 734 op/s; 25512/304596 objects degraded (8.376%) 2024-02-09T17:27:00.113 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell -n osd.8 --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-8 --journal-path /var/lib/ceph/osd/ceph-8/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.2.f.8 2024-02-09T17:27:00.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:26:59 smithi162 ceph-mon[44304]: pgmap v457: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 6.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 734 op/s; 25512/304596 objects degraded (8.376%) 2024-02-09T17:27:00.298 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:27:00.387 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/osd.8/config 2024-02-09T17:27:00.487 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:27:00.487 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-02-09T17:27:00.487 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-02-09T17:27:00.487 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-02-09T17:27:00.487 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.487 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14790.0:82043 2.7 2:e5a2f4ba:::benchmark_data_smithi179_77138_object82042:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-02-09T17:27:00.487 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:25:29.912435+0000", 2024-02-09T17:27:00.487 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 90.570883261999995, 2024-02-09T17:27:00.487 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.037257132999999998, 2024-02-09T17:27:00.488 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:27:00.488 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:00.488 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:27:00.488 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14790", 2024-02-09T17:27:00.488 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:27:00.488 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 82043 2024-02-09T17:27:00.488 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.488 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:27:00.488 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.488 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:27:00.488 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:25:29.912435+0000", 2024-02-09T17:27:00.488 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:27:00.488 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.489 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:27:00.489 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:25:29.912435+0000", 2024-02-09T17:27:00.489 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:27:00.489 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.489 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:27:00.489 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:25:29.912436+0000", 2024-02-09T17:27:00.489 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7000000000000004e-07 2024-02-09T17:27:00.489 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.489 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:27:00.489 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:25:29.912510+0000", 2024-02-09T17:27:00.489 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3627999999999994e-05 2024-02-09T17:27:00.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.490 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:27:00.490 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:25:29.912525+0000", 2024-02-09T17:27:00.490 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5218000000000001e-05 2024-02-09T17:27:00.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.490 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:27:00.490 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:25:29.912530+0000", 2024-02-09T17:27:00.490 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2129999999999999e-06 2024-02-09T17:27:00.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.490 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:27:00.490 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:25:29.913086+0000", 2024-02-09T17:27:00.490 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00055603300000000005 2024-02-09T17:27:00.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.491 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.491 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:27:00.491 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:25:29.913163+0000", 2024-02-09T17:27:00.491 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7194999999999995e-05 2024-02-09T17:27:00.491 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.491 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.491 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,10", 2024-02-09T17:27:00.491 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:25:29.913274+0000", 2024-02-09T17:27:00.491 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011118800000000001 2024-02-09T17:27:00.491 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.491 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.491 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:27:00.491 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:25:29.918798+0000", 2024-02-09T17:27:00.491 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0055232780000000004 2024-02-09T17:27:00.492 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.492 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:00.492 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:25:29.918847+0000", 2024-02-09T17:27:00.492 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8869000000000003e-05 2024-02-09T17:27:00.492 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.492 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:00.492 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:25:29.949624+0000", 2024-02-09T17:27:00.492 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.030777828 2024-02-09T17:27:00.492 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.492 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:27:00.493 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:25:29.949666+0000", 2024-02-09T17:27:00.493 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1489999999999997e-05 2024-02-09T17:27:00.493 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.493 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.493 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:27:00.493 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:25:29.949692+0000", 2024-02-09T17:27:00.493 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6523e-05 2024-02-09T17:27:00.493 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:27:00.493 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:27:00.493 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:27:00.493 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.493 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.493 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14790.0:82044 2.7 2:e68f35ad:::benchmark_data_smithi179_77138_object82043:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-02-09T17:27:00.493 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:25:29.912552+0000", 2024-02-09T17:27:00.494 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 90.570766946000006, 2024-02-09T17:27:00.494 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.037246754, 2024-02-09T17:27:00.494 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:27:00.494 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:00.494 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:27:00.494 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14790", 2024-02-09T17:27:00.494 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:27:00.494 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 82044 2024-02-09T17:27:00.494 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.494 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:27:00.494 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.494 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:27:00.494 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:25:29.912552+0000", 2024-02-09T17:27:00.494 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:27:00.494 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.495 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.495 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:27:00.495 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:25:29.912552+0000", 2024-02-09T17:27:00.495 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:27:00.495 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.495 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.495 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:27:00.495 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:25:29.912552+0000", 2024-02-09T17:27:00.495 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2200000000000004e-07 2024-02-09T17:27:00.495 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.495 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.495 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:27:00.495 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:25:29.912610+0000", 2024-02-09T17:27:00.495 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7457999999999998e-05 2024-02-09T17:27:00.496 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.496 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.496 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:27:00.496 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:25:29.912616+0000", 2024-02-09T17:27:00.496 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5350000000000001e-06 2024-02-09T17:27:00.496 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.496 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.496 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:27:00.496 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:25:29.912620+0000", 2024-02-09T17:27:00.496 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8449999999999996e-06 2024-02-09T17:27:00.496 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.496 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.496 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:27:00.496 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:25:29.914757+0000", 2024-02-09T17:27:00.497 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021373009999999999 2024-02-09T17:27:00.497 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.497 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.497 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:27:00.497 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:25:29.914871+0000", 2024-02-09T17:27:00.497 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011342500000000001 2024-02-09T17:27:00.497 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.497 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.497 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,10", 2024-02-09T17:27:00.497 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:25:29.915011+0000", 2024-02-09T17:27:00.497 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014076299999999999 2024-02-09T17:27:00.497 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.497 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.497 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:27:00.497 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:25:29.918808+0000", 2024-02-09T17:27:00.498 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0037965270000000001 2024-02-09T17:27:00.498 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.498 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.498 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:00.498 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:25:29.919699+0000", 2024-02-09T17:27:00.498 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00089077899999999996 2024-02-09T17:27:00.498 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.498 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.498 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:00.498 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:25:29.949735+0000", 2024-02-09T17:27:00.498 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.030035765999999998 2024-02-09T17:27:00.498 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.498 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.498 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:27:00.498 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:25:29.949774+0000", 2024-02-09T17:27:00.498 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9650000000000002e-05 2024-02-09T17:27:00.499 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.499 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.499 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:27:00.499 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:25:29.949798+0000", 2024-02-09T17:27:00.499 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4182999999999999e-05 2024-02-09T17:27:00.499 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:27:00.499 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:27:00.499 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:27:00.499 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.499 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.499 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14790.0:116411 2.7 2:e574b900:::benchmark_data_smithi179_77138_object116410:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-02-09T17:27:00.499 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:26:01.597610+0000", 2024-02-09T17:27:00.499 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 58.885708575999999, 2024-02-09T17:27:00.499 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.036036574000000002, 2024-02-09T17:27:00.500 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:27:00.500 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:00.500 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:27:00.500 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14790", 2024-02-09T17:27:00.500 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:27:00.500 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 116411 2024-02-09T17:27:00.500 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.500 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:27:00.500 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.500 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:27:00.500 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:01.597610+0000", 2024-02-09T17:27:00.500 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:27:00.500 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.500 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.501 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:27:00.501 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:01.597610+0000", 2024-02-09T17:27:00.501 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:27:00.501 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.501 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:27:00.501 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:01.597611+0000", 2024-02-09T17:27:00.501 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3200000000000003e-07 2024-02-09T17:27:00.501 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.501 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:27:00.501 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:01.597670+0000", 2024-02-09T17:27:00.501 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8807000000000001e-05 2024-02-09T17:27:00.501 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.502 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:27:00.502 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:01.597680+0000", 2024-02-09T17:27:00.502 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0604e-05 2024-02-09T17:27:00.502 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.502 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.502 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:27:00.502 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:01.597686+0000", 2024-02-09T17:27:00.502 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9240000000000004e-06 2024-02-09T17:27:00.502 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.502 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.502 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:27:00.502 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:01.597732+0000", 2024-02-09T17:27:00.502 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5445999999999998e-05 2024-02-09T17:27:00.502 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.502 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.502 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:27:00.503 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:01.597827+0000", 2024-02-09T17:27:00.503 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5617000000000003e-05 2024-02-09T17:27:00.503 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.503 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.503 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,10", 2024-02-09T17:27:00.503 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:01.597923+0000", 2024-02-09T17:27:00.503 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5855999999999998e-05 2024-02-09T17:27:00.503 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.503 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.503 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:27:00.503 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:01.600417+0000", 2024-02-09T17:27:00.503 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0024943180000000001 2024-02-09T17:27:00.503 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.503 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.503 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:00.503 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:01.604402+0000", 2024-02-09T17:27:00.504 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0039849200000000003 2024-02-09T17:27:00.504 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.504 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.504 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:00.504 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:01.633545+0000", 2024-02-09T17:27:00.504 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.029142143999999998 2024-02-09T17:27:00.504 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.504 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.504 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:27:00.504 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:01.633608+0000", 2024-02-09T17:27:00.504 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3437999999999998e-05 2024-02-09T17:27:00.504 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.504 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.505 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:27:00.505 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:01.633647+0000", 2024-02-09T17:27:00.505 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8568000000000001e-05 2024-02-09T17:27:00.505 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:27:00.505 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:27:00.505 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:27:00.505 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.505 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.505 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14790.0:133285 2.7 2:e64a52d2:::benchmark_data_smithi179_77138_object133284:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-02-09T17:27:00.505 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:26:19.274741+0000", 2024-02-09T17:27:00.505 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 41.208577560000002, 2024-02-09T17:27:00.505 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.049628528999999998, 2024-02-09T17:27:00.505 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:27:00.505 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:00.506 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:27:00.506 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14790", 2024-02-09T17:27:00.506 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:27:00.506 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 133285 2024-02-09T17:27:00.506 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.506 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:27:00.506 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.506 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:27:00.506 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:19.274741+0000", 2024-02-09T17:27:00.506 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:27:00.506 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.506 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.506 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:27:00.506 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:19.274741+0000", 2024-02-09T17:27:00.507 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:27:00.507 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.507 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.507 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:27:00.507 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:19.274742+0000", 2024-02-09T17:27:00.507 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1599999999999999e-06 2024-02-09T17:27:00.507 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.507 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.507 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:27:00.507 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:19.274786+0000", 2024-02-09T17:27:00.507 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4319e-05 2024-02-09T17:27:00.507 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.507 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.507 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:27:00.507 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:19.274797+0000", 2024-02-09T17:27:00.508 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0917999999999999e-05 2024-02-09T17:27:00.508 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.508 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.508 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:27:00.508 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:19.274803+0000", 2024-02-09T17:27:00.508 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0730000000000001e-06 2024-02-09T17:27:00.508 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.508 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.508 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:27:00.508 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:19.274853+0000", 2024-02-09T17:27:00.508 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9138000000000003e-05 2024-02-09T17:27:00.508 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.508 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.508 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:27:00.508 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:19.274942+0000", 2024-02-09T17:27:00.509 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9778999999999995e-05 2024-02-09T17:27:00.509 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.509 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.509 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,10", 2024-02-09T17:27:00.509 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:19.275035+0000", 2024-02-09T17:27:00.509 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3014000000000001e-05 2024-02-09T17:27:00.509 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.509 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.509 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:00.509 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:19.278943+0000", 2024-02-09T17:27:00.509 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0039072079999999997 2024-02-09T17:27:00.509 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.509 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.509 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:27:00.509 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:19.279049+0000", 2024-02-09T17:27:00.510 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000106499 2024-02-09T17:27:00.510 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.510 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.510 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:00.510 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:19.324260+0000", 2024-02-09T17:27:00.510 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.045210881000000001 2024-02-09T17:27:00.510 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.510 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.510 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:27:00.510 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:19.324321+0000", 2024-02-09T17:27:00.510 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0708999999999998e-05 2024-02-09T17:27:00.510 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.510 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.510 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:27:00.510 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:19.324370+0000", 2024-02-09T17:27:00.510 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8831000000000001e-05 2024-02-09T17:27:00.510 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:27:00.511 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:27:00.511 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:27:00.511 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.511 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.511 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14790.0:143448 2.7 2:e72e1557:::benchmark_data_smithi179_77138_object10040:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-02-09T17:27:00.511 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:26:30.208117+0000", 2024-02-09T17:27:00.511 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 30.275201529, 2024-02-09T17:27:00.511 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.036720377999999998, 2024-02-09T17:27:00.511 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:27:00.511 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:00.511 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:27:00.511 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14790", 2024-02-09T17:27:00.511 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:27:00.511 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 143448 2024-02-09T17:27:00.511 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.511 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:27:00.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.512 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:27:00.512 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:30.208117+0000", 2024-02-09T17:27:00.512 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:27:00.512 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.512 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:27:00.512 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:30.208117+0000", 2024-02-09T17:27:00.512 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:27:00.512 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.512 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:27:00.512 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:30.208119+0000", 2024-02-09T17:27:00.512 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1739999999999999e-06 2024-02-09T17:27:00.512 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.513 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.513 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:27:00.513 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:30.208127+0000", 2024-02-09T17:27:00.513 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7209999999999994e-06 2024-02-09T17:27:00.513 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.513 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.513 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:27:00.513 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:30.208144+0000", 2024-02-09T17:27:00.513 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7249999999999999e-05 2024-02-09T17:27:00.513 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.513 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.513 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:27:00.513 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:30.208153+0000", 2024-02-09T17:27:00.513 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5830000000000007e-06 2024-02-09T17:27:00.513 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.513 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.514 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:27:00.514 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:30.208223+0000", 2024-02-09T17:27:00.514 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0140000000000003e-05 2024-02-09T17:27:00.514 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.514 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.514 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:27:00.514 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:30.208385+0000", 2024-02-09T17:27:00.514 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000162041 2024-02-09T17:27:00.514 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.514 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.514 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,10", 2024-02-09T17:27:00.514 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:30.208507+0000", 2024-02-09T17:27:00.514 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000122438 2024-02-09T17:27:00.514 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.514 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.515 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:00.515 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:30.212959+0000", 2024-02-09T17:27:00.515 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.004452035 2024-02-09T17:27:00.515 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.515 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.515 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:27:00.515 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:30.213061+0000", 2024-02-09T17:27:00.515 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000101271 2024-02-09T17:27:00.515 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.515 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.515 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:00.515 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:30.244774+0000", 2024-02-09T17:27:00.515 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.031713558000000003 2024-02-09T17:27:00.515 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.515 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.515 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:27:00.516 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:30.244812+0000", 2024-02-09T17:27:00.516 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7397000000000001e-05 2024-02-09T17:27:00.516 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.516 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.516 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:27:00.516 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:30.244837+0000", 2024-02-09T17:27:00.516 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5769999999999999e-05 2024-02-09T17:27:00.516 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:27:00.516 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:27:00.516 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:27:00.516 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.516 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.516 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14790.0:146946 2.7 2:e490c70e:::benchmark_data_smithi179_77138_object13538:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-02-09T17:27:00.516 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:26:33.894273+0000", 2024-02-09T17:27:00.517 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 26.589045522999999, 2024-02-09T17:27:00.517 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.053209737, 2024-02-09T17:27:00.517 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:27:00.517 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:00.517 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:27:00.517 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14790", 2024-02-09T17:27:00.517 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:27:00.517 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 146946 2024-02-09T17:27:00.517 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.517 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:27:00.517 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.517 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:27:00.517 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.894273+0000", 2024-02-09T17:27:00.517 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:27:00.517 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.518 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.518 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:27:00.518 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.894273+0000", 2024-02-09T17:27:00.518 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:27:00.518 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.518 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.518 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:27:00.518 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.894273+0000", 2024-02-09T17:27:00.518 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9999999999999998e-07 2024-02-09T17:27:00.518 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.518 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.518 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:27:00.518 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.894276+0000", 2024-02-09T17:27:00.519 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5809999999999999e-06 2024-02-09T17:27:00.519 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.519 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.519 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:27:00.519 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.894283+0000", 2024-02-09T17:27:00.519 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6279999999999999e-06 2024-02-09T17:27:00.519 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.519 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.519 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:27:00.519 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.894288+0000", 2024-02-09T17:27:00.519 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4530000000000001e-06 2024-02-09T17:27:00.519 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.519 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.519 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:27:00.519 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.898941+0000", 2024-02-09T17:27:00.519 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0046531790000000003 2024-02-09T17:27:00.519 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.520 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.520 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:27:00.520 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.899054+0000", 2024-02-09T17:27:00.520 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000112633 2024-02-09T17:27:00.520 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.520 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.520 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,10", 2024-02-09T17:27:00.520 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.899154+0000", 2024-02-09T17:27:00.520 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9679999999999994e-05 2024-02-09T17:27:00.520 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.520 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.520 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:27:00.520 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.903397+0000", 2024-02-09T17:27:00.520 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042431109999999999 2024-02-09T17:27:00.520 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.521 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.521 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:00.521 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.904895+0000", 2024-02-09T17:27:00.521 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014981040000000001 2024-02-09T17:27:00.521 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.521 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.521 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:00.521 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.925296+0000", 2024-02-09T17:27:00.521 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020401633999999998 2024-02-09T17:27:00.521 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.521 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.521 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:27:00.521 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.947434+0000", 2024-02-09T17:27:00.521 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.022137093 2024-02-09T17:27:00.521 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.521 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.522 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:27:00.522 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.947483+0000", 2024-02-09T17:27:00.522 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9240999999999998e-05 2024-02-09T17:27:00.522 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:27:00.522 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:27:00.522 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:27:00.522 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.522 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.522 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14790.0:154958 2.7 2:e48b6079:::benchmark_data_smithi179_77138_object21550:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-02-09T17:27:00.522 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:26:43.466612+0000", 2024-02-09T17:27:00.522 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 17.016706904999999, 2024-02-09T17:27:00.522 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.039145480000000003, 2024-02-09T17:27:00.522 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:27:00.522 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:00.522 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:27:00.522 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14790", 2024-02-09T17:27:00.523 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:27:00.523 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 154958 2024-02-09T17:27:00.523 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.523 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:27:00.523 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.523 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:27:00.523 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:43.466612+0000", 2024-02-09T17:27:00.523 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:27:00.523 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.523 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.523 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:27:00.523 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:43.466612+0000", 2024-02-09T17:27:00.523 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:27:00.523 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.523 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.524 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:27:00.524 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:43.466613+0000", 2024-02-09T17:27:00.524 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.159e-06 2024-02-09T17:27:00.524 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.524 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.524 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:27:00.524 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:43.466617+0000", 2024-02-09T17:27:00.524 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3749999999999996e-06 2024-02-09T17:27:00.524 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.524 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.524 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:27:00.524 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:43.466629+0000", 2024-02-09T17:27:00.524 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1747e-05 2024-02-09T17:27:00.524 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.524 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.525 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:27:00.525 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:43.466635+0000", 2024-02-09T17:27:00.525 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5370000000000003e-06 2024-02-09T17:27:00.525 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.525 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.525 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:27:00.525 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:43.466689+0000", 2024-02-09T17:27:00.525 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3361e-05 2024-02-09T17:27:00.525 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.525 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.525 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:27:00.525 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:43.466824+0000", 2024-02-09T17:27:00.525 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013542599999999999 2024-02-09T17:27:00.525 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.525 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.525 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,10", 2024-02-09T17:27:00.525 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:43.466947+0000", 2024-02-09T17:27:00.526 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012323 2024-02-09T17:27:00.526 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.526 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.526 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:00.526 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:43.470340+0000", 2024-02-09T17:27:00.526 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0033930119999999999 2024-02-09T17:27:00.526 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.526 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.526 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:27:00.526 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:43.470409+0000", 2024-02-09T17:27:00.526 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8678999999999998e-05 2024-02-09T17:27:00.526 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.526 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.526 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:00.526 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:43.505685+0000", 2024-02-09T17:27:00.526 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.035276069 2024-02-09T17:27:00.527 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.527 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.527 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:27:00.527 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:43.505726+0000", 2024-02-09T17:27:00.527 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1155999999999997e-05 2024-02-09T17:27:00.527 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.527 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.527 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:27:00.527 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:43.505757+0000", 2024-02-09T17:27:00.527 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0728999999999997e-05 2024-02-09T17:27:00.527 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:27:00.527 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:27:00.527 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:27:00.527 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.527 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.528 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14790.0:159508 2.7 2:e418ab49:::benchmark_data_smithi179_77138_object26100:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-02-09T17:27:00.528 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:26:48.463993+0000", 2024-02-09T17:27:00.528 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 12.019325233, 2024-02-09T17:27:00.528 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.036910168, 2024-02-09T17:27:00.528 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:27:00.528 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:00.528 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:27:00.528 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14790", 2024-02-09T17:27:00.528 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:27:00.528 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 159508 2024-02-09T17:27:00.528 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.528 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:27:00.528 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.528 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:27:00.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:48.463993+0000", 2024-02-09T17:27:00.529 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:27:00.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:27:00.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:48.463993+0000", 2024-02-09T17:27:00.529 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:27:00.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:27:00.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:48.463995+0000", 2024-02-09T17:27:00.529 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.221e-06 2024-02-09T17:27:00.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:27:00.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:48.464001+0000", 2024-02-09T17:27:00.530 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1410000000000002e-06 2024-02-09T17:27:00.530 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.530 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.530 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:27:00.530 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:48.464015+0000", 2024-02-09T17:27:00.530 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3903000000000001e-05 2024-02-09T17:27:00.530 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.530 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.530 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:27:00.530 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:48.464022+0000", 2024-02-09T17:27:00.530 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6979999999999998e-06 2024-02-09T17:27:00.530 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.530 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.530 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:27:00.531 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:48.464080+0000", 2024-02-09T17:27:00.531 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7416000000000001e-05 2024-02-09T17:27:00.531 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.531 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.531 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:27:00.531 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:48.464216+0000", 2024-02-09T17:27:00.531 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000136191 2024-02-09T17:27:00.531 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.531 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.531 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,10", 2024-02-09T17:27:00.531 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:48.464360+0000", 2024-02-09T17:27:00.531 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014412499999999999 2024-02-09T17:27:00.531 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.531 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.532 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:00.532 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:48.473201+0000", 2024-02-09T17:27:00.532 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0088412899999999999 2024-02-09T17:27:00.532 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.532 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.532 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:27:00.532 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:48.473263+0000", 2024-02-09T17:27:00.532 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1556000000000005e-05 2024-02-09T17:27:00.532 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.532 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.532 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:00.532 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:48.500808+0000", 2024-02-09T17:27:00.532 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.027545508999999999 2024-02-09T17:27:00.533 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.533 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.533 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:27:00.533 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:48.500864+0000", 2024-02-09T17:27:00.533 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6028e-05 2024-02-09T17:27:00.533 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.533 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.533 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:27:00.533 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:48.500903+0000", 2024-02-09T17:27:00.533 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.909e-05 2024-02-09T17:27:00.533 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:27:00.533 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:27:00.533 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:27:00.533 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.533 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.534 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14790.0:159509 2.7 2:e8edef05:::benchmark_data_smithi179_77138_object26101:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-02-09T17:27:00.534 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:26:48.464392+0000", 2024-02-09T17:27:00.534 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 12.018927028, 2024-02-09T17:27:00.534 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.051121611999999997, 2024-02-09T17:27:00.534 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:27:00.534 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:00.534 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:27:00.534 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14790", 2024-02-09T17:27:00.534 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:27:00.534 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 159509 2024-02-09T17:27:00.534 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.534 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:27:00.534 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.534 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:27:00.535 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:48.464392+0000", 2024-02-09T17:27:00.535 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:27:00.535 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.535 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.535 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:27:00.535 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:48.464392+0000", 2024-02-09T17:27:00.535 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:27:00.535 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.535 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.535 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:27:00.535 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:48.464393+0000", 2024-02-09T17:27:00.535 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0389999999999999e-06 2024-02-09T17:27:00.535 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.535 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.536 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:27:00.536 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:48.464398+0000", 2024-02-09T17:27:00.536 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6799999999999998e-06 2024-02-09T17:27:00.536 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.536 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.536 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:27:00.536 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:48.464410+0000", 2024-02-09T17:27:00.536 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1419e-05 2024-02-09T17:27:00.536 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.536 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.536 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:27:00.536 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:48.464418+0000", 2024-02-09T17:27:00.536 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4619999999999996e-06 2024-02-09T17:27:00.536 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.537 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.537 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:27:00.537 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:48.469054+0000", 2024-02-09T17:27:00.537 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0046354430000000004 2024-02-09T17:27:00.537 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.537 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.537 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:27:00.537 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:48.469163+0000", 2024-02-09T17:27:00.537 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000109683 2024-02-09T17:27:00.537 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.537 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.537 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,10", 2024-02-09T17:27:00.537 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:48.469253+0000", 2024-02-09T17:27:00.537 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0153000000000002e-05 2024-02-09T17:27:00.537 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.538 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.538 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:00.538 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:48.473252+0000", 2024-02-09T17:27:00.538 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0039990240000000003 2024-02-09T17:27:00.538 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.538 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.538 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:27:00.538 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:48.473315+0000", 2024-02-09T17:27:00.538 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2143000000000005e-05 2024-02-09T17:27:00.538 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.538 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.538 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:00.538 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:48.515402+0000", 2024-02-09T17:27:00.538 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.042087362000000003 2024-02-09T17:27:00.538 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.539 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.539 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:27:00.539 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:48.515469+0000", 2024-02-09T17:27:00.539 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7534999999999996e-05 2024-02-09T17:27:00.539 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.539 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.539 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:27:00.539 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:48.515513+0000", 2024-02-09T17:27:00.539 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3668999999999998e-05 2024-02-09T17:27:00.539 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:27:00.539 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:27:00.539 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:27:00.539 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.539 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.539 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14790.0:159517 2.7 2:e61ee0b0:::benchmark_data_smithi179_77138_object26109:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-02-09T17:27:00.539 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:26:48.477050+0000", 2024-02-09T17:27:00.540 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 12.006268213, 2024-02-09T17:27:00.540 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.038622998999999998, 2024-02-09T17:27:00.540 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:27:00.540 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:00.540 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:27:00.540 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14790", 2024-02-09T17:27:00.540 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:27:00.540 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 159517 2024-02-09T17:27:00.540 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.540 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:27:00.540 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.540 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:27:00.540 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:48.477050+0000", 2024-02-09T17:27:00.540 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:27:00.541 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.541 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.541 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:27:00.541 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:48.477050+0000", 2024-02-09T17:27:00.541 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:27:00.541 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.541 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.541 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:27:00.541 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:48.477052+0000", 2024-02-09T17:27:00.541 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3200000000000001e-06 2024-02-09T17:27:00.541 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.541 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.541 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:27:00.541 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:48.477057+0000", 2024-02-09T17:27:00.541 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.755e-06 2024-02-09T17:27:00.541 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.542 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.542 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:27:00.542 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:48.477070+0000", 2024-02-09T17:27:00.542 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2539e-05 2024-02-09T17:27:00.542 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.542 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.542 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:27:00.542 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:48.477079+0000", 2024-02-09T17:27:00.542 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5070000000000006e-06 2024-02-09T17:27:00.542 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.542 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.542 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:27:00.542 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:48.477139+0000", 2024-02-09T17:27:00.542 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9061000000000003e-05 2024-02-09T17:27:00.542 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.542 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.543 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:27:00.543 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:48.477268+0000", 2024-02-09T17:27:00.543 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000129789 2024-02-09T17:27:00.543 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.543 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.543 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,10", 2024-02-09T17:27:00.543 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:48.477386+0000", 2024-02-09T17:27:00.543 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000117238 2024-02-09T17:27:00.543 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.543 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.543 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:27:00.543 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:48.480873+0000", 2024-02-09T17:27:00.543 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003487221 2024-02-09T17:27:00.543 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.543 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.543 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:00.543 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:48.482564+0000", 2024-02-09T17:27:00.544 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016914709999999999 2024-02-09T17:27:00.544 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.544 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.544 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:00.544 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:48.515573+0000", 2024-02-09T17:27:00.544 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.033008509999999998 2024-02-09T17:27:00.544 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.544 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.544 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:27:00.544 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:48.515628+0000", 2024-02-09T17:27:00.544 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5584999999999997e-05 2024-02-09T17:27:00.544 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.544 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.544 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:27:00.544 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:48.515673+0000", 2024-02-09T17:27:00.545 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5003000000000003e-05 2024-02-09T17:27:00.545 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:27:00.545 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:27:00.545 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:27:00.545 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.545 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.545 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14790.0:160100 2.7 2:e4d1b2c0:::benchmark_data_smithi179_77138_object26692:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-02-09T17:27:00.545 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:26:49.183826+0000", 2024-02-09T17:27:00.545 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 11.299492951, 2024-02-09T17:27:00.545 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.041175206999999998, 2024-02-09T17:27:00.545 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:27:00.545 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:00.545 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:27:00.545 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14790", 2024-02-09T17:27:00.545 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:27:00.545 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 160100 2024-02-09T17:27:00.545 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.546 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:27:00.546 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.546 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:27:00.546 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:49.183826+0000", 2024-02-09T17:27:00.546 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:27:00.546 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.546 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.546 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:27:00.546 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:49.183826+0000", 2024-02-09T17:27:00.546 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:27:00.546 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.546 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.546 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:27:00.546 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:49.183827+0000", 2024-02-09T17:27:00.546 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.372e-06 2024-02-09T17:27:00.546 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.546 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.547 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:27:00.547 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:49.183832+0000", 2024-02-09T17:27:00.547 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.95e-06 2024-02-09T17:27:00.547 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.547 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.547 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:27:00.547 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:49.183846+0000", 2024-02-09T17:27:00.547 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4219999999999999e-05 2024-02-09T17:27:00.547 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.547 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.547 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:27:00.547 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:49.183854+0000", 2024-02-09T17:27:00.547 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.396e-06 2024-02-09T17:27:00.547 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.547 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.547 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:27:00.547 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:49.183908+0000", 2024-02-09T17:27:00.548 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4360999999999997e-05 2024-02-09T17:27:00.548 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.548 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.548 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:27:00.548 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:49.184015+0000", 2024-02-09T17:27:00.548 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000107524 2024-02-09T17:27:00.548 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.548 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.548 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,10", 2024-02-09T17:27:00.548 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:49.184142+0000", 2024-02-09T17:27:00.548 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012661199999999999 2024-02-09T17:27:00.548 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.548 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.548 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:27:00.548 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:49.187719+0000", 2024-02-09T17:27:00.548 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0035766019999999999 2024-02-09T17:27:00.548 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.549 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.549 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:00.549 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:49.190460+0000", 2024-02-09T17:27:00.549 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0027410149999999999 2024-02-09T17:27:00.549 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.549 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.549 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:00.549 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:49.224914+0000", 2024-02-09T17:27:00.549 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.034454295000000003 2024-02-09T17:27:00.549 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.549 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.549 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:27:00.549 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:49.224960+0000", 2024-02-09T17:27:00.549 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5815000000000001e-05 2024-02-09T17:27:00.549 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.549 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.549 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:27:00.550 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:49.225001+0000", 2024-02-09T17:27:00.550 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1044999999999998e-05 2024-02-09T17:27:00.550 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:27:00.550 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:27:00.550 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:27:00.550 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.550 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.550 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14790.0:168363 2.7 2:e413ab3f:::benchmark_data_smithi179_77138_object34955:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-02-09T17:27:00.550 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:26:58.293252+0000", 2024-02-09T17:27:00.550 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 2.1900665579999998, 2024-02-09T17:27:00.550 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.044117027000000003, 2024-02-09T17:27:00.550 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:27:00.550 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:00.550 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:27:00.550 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14790", 2024-02-09T17:27:00.550 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:27:00.551 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 168363 2024-02-09T17:27:00.551 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.551 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:27:00.551 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.551 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:27:00.551 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:58.293252+0000", 2024-02-09T17:27:00.551 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:27:00.551 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.551 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.551 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:27:00.551 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:58.293252+0000", 2024-02-09T17:27:00.551 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:27:00.551 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.551 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.551 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:27:00.552 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:58.293254+0000", 2024-02-09T17:27:00.552 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.756e-06 2024-02-09T17:27:00.552 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.552 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.552 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:27:00.552 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:58.293262+0000", 2024-02-09T17:27:00.552 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0409999999999998e-06 2024-02-09T17:27:00.552 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.552 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.552 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:27:00.552 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:58.293281+0000", 2024-02-09T17:27:00.552 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8726e-05 2024-02-09T17:27:00.552 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.552 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.553 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:27:00.553 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:58.293291+0000", 2024-02-09T17:27:00.553 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0352e-05 2024-02-09T17:27:00.553 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.553 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.553 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:27:00.553 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:58.293345+0000", 2024-02-09T17:27:00.553 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4353e-05 2024-02-09T17:27:00.553 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.553 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.553 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:27:00.553 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:58.293441+0000", 2024-02-09T17:27:00.553 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6181000000000006e-05 2024-02-09T17:27:00.553 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.553 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.553 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,10", 2024-02-09T17:27:00.554 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:58.293522+0000", 2024-02-09T17:27:00.554 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0263e-05 2024-02-09T17:27:00.554 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.554 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.554 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:27:00.554 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:58.298141+0000", 2024-02-09T17:27:00.554 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0046196900000000001 2024-02-09T17:27:00.554 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.554 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.554 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:00.554 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:58.298954+0000", 2024-02-09T17:27:00.554 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00081270500000000005 2024-02-09T17:27:00.554 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.554 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.554 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:00.554 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:58.337296+0000", 2024-02-09T17:27:00.555 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.038341548000000003 2024-02-09T17:27:00.555 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.555 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.555 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:27:00.555 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:58.337341+0000", 2024-02-09T17:27:00.555 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4972000000000002e-05 2024-02-09T17:27:00.555 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.555 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.555 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:27:00.555 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:58.337369+0000", 2024-02-09T17:27:00.555 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8439999999999999e-05 2024-02-09T17:27:00.555 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:27:00.555 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:27:00.555 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:27:00.555 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.555 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.555 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14790.0:168646 2.7 2:ef9f7691:::benchmark_data_smithi179_77138_object35238:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-02-09T17:27:00.556 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:26:58.636899+0000", 2024-02-09T17:27:00.556 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.8464196130000001, 2024-02-09T17:27:00.556 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.042392633999999998, 2024-02-09T17:27:00.556 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:27:00.556 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:00.556 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:27:00.556 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14790", 2024-02-09T17:27:00.556 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:27:00.556 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 168646 2024-02-09T17:27:00.556 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.556 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:27:00.556 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.556 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:27:00.556 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:58.636899+0000", 2024-02-09T17:27:00.556 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:27:00.557 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.557 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.557 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:27:00.557 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:58.636899+0000", 2024-02-09T17:27:00.557 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:27:00.557 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.557 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.557 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:27:00.557 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:58.636900+0000", 2024-02-09T17:27:00.557 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5489999999999999e-06 2024-02-09T17:27:00.557 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.557 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.557 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:27:00.557 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:58.636907+0000", 2024-02-09T17:27:00.558 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3579999999999999e-06 2024-02-09T17:27:00.558 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.558 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.558 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:27:00.558 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:58.636921+0000", 2024-02-09T17:27:00.558 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4436999999999999e-05 2024-02-09T17:27:00.558 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.558 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.558 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:27:00.558 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:58.636930+0000", 2024-02-09T17:27:00.558 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9919999999999996e-06 2024-02-09T17:27:00.558 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.558 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.558 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:27:00.558 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:58.636991+0000", 2024-02-09T17:27:00.559 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0380999999999997e-05 2024-02-09T17:27:00.559 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.559 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.559 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:27:00.559 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:58.637076+0000", 2024-02-09T17:27:00.559 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5228999999999993e-05 2024-02-09T17:27:00.559 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.559 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.559 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,10", 2024-02-09T17:27:00.559 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:58.637145+0000", 2024-02-09T17:27:00.559 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9355000000000003e-05 2024-02-09T17:27:00.559 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.559 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.559 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:27:00.559 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:58.641075+0000", 2024-02-09T17:27:00.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0039293160000000004 2024-02-09T17:27:00.560 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.560 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.560 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:00.560 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:58.641307+0000", 2024-02-09T17:27:00.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023238299999999999 2024-02-09T17:27:00.560 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.560 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.560 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:00.560 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:58.679214+0000", 2024-02-09T17:27:00.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.037906673000000002 2024-02-09T17:27:00.560 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.560 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.560 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:27:00.560 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:58.679263+0000", 2024-02-09T17:27:00.561 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8955000000000001e-05 2024-02-09T17:27:00.561 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.561 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.561 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:27:00.561 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:58.679292+0000", 2024-02-09T17:27:00.561 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9006e-05 2024-02-09T17:27:00.561 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:27:00.561 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:27:00.561 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:27:00.561 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.561 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.561 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14790.0:168669 2.7 2:e47f9e59:::benchmark_data_smithi179_77138_object35261:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-02-09T17:27:00.561 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:26:58.658325+0000", 2024-02-09T17:27:00.561 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 1.824993576, 2024-02-09T17:27:00.561 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.040527324000000003, 2024-02-09T17:27:00.561 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:27:00.561 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:00.562 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:27:00.562 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14790", 2024-02-09T17:27:00.562 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:27:00.562 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 168669 2024-02-09T17:27:00.562 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.562 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:27:00.562 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.562 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:27:00.562 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:58.658325+0000", 2024-02-09T17:27:00.562 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:27:00.562 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.562 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.562 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:27:00.562 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:58.658325+0000", 2024-02-09T17:27:00.562 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:27:00.562 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.562 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.563 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:27:00.563 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:58.658326+0000", 2024-02-09T17:27:00.563 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7199999999999997e-07 2024-02-09T17:27:00.563 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.563 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.563 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:27:00.563 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:58.658330+0000", 2024-02-09T17:27:00.563 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4950000000000002e-06 2024-02-09T17:27:00.563 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.563 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.563 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:27:00.563 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:58.658341+0000", 2024-02-09T17:27:00.563 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0515e-05 2024-02-09T17:27:00.563 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.563 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.563 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:27:00.564 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:58.658348+0000", 2024-02-09T17:27:00.564 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1369999999999996e-06 2024-02-09T17:27:00.564 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.564 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.564 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:27:00.564 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:58.660928+0000", 2024-02-09T17:27:00.564 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0025801420000000001 2024-02-09T17:27:00.564 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.564 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.564 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:27:00.564 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:58.661044+0000", 2024-02-09T17:27:00.564 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000115469 2024-02-09T17:27:00.564 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.564 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.564 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,10", 2024-02-09T17:27:00.565 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:58.661115+0000", 2024-02-09T17:27:00.565 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1227000000000001e-05 2024-02-09T17:27:00.565 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.565 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.565 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:27:00.565 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:58.663899+0000", 2024-02-09T17:27:00.565 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0027842349999999999 2024-02-09T17:27:00.565 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.565 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.565 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:00.565 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:58.665388+0000", 2024-02-09T17:27:00.565 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014892309999999999 2024-02-09T17:27:00.565 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.565 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.565 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:00.565 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:58.698754+0000", 2024-02-09T17:27:00.566 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.033366016999999998 2024-02-09T17:27:00.566 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.566 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.566 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:27:00.566 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:58.698814+0000", 2024-02-09T17:27:00.566 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9929000000000002e-05 2024-02-09T17:27:00.566 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.566 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.566 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:27:00.566 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:58.698852+0000", 2024-02-09T17:27:00.566 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8055000000000001e-05 2024-02-09T17:27:00.566 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:27:00.566 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:27:00.566 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:27:00.566 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.566 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.567 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14790.0:169919 2.7 2:e68977f8:::benchmark_data_smithi179_77138_object36511:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-02-09T17:27:00.567 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:27:00.008799+0000", 2024-02-09T17:27:00.567 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.47451933600000001, 2024-02-09T17:27:00.567 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.036992023999999998, 2024-02-09T17:27:00.567 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:27:00.567 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:00.567 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:27:00.567 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14790", 2024-02-09T17:27:00.567 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:27:00.567 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 169919 2024-02-09T17:27:00.567 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.567 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:27:00.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.567 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:27:00.567 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.008799+0000", 2024-02-09T17:27:00.568 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:27:00.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.568 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.568 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:27:00.568 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.008799+0000", 2024-02-09T17:27:00.568 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:27:00.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.568 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.568 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:27:00.568 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.008800+0000", 2024-02-09T17:27:00.568 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7999999999999996e-07 2024-02-09T17:27:00.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.568 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.569 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:27:00.569 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.008803+0000", 2024-02-09T17:27:00.569 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1870000000000001e-06 2024-02-09T17:27:00.569 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.569 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.569 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:27:00.569 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.008809+0000", 2024-02-09T17:27:00.569 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5549999999999999e-06 2024-02-09T17:27:00.569 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.569 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.569 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:27:00.569 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.008815+0000", 2024-02-09T17:27:00.569 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6749999999999999e-06 2024-02-09T17:27:00.569 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.569 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.570 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:27:00.570 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.021569+0000", 2024-02-09T17:27:00.570 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012753621999999999 2024-02-09T17:27:00.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.570 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.570 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:27:00.570 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.021676+0000", 2024-02-09T17:27:00.570 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010721399999999999 2024-02-09T17:27:00.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.570 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.570 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,10", 2024-02-09T17:27:00.570 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.021763+0000", 2024-02-09T17:27:00.570 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7130000000000001e-05 2024-02-09T17:27:00.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.570 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.571 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:27:00.571 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.029047+0000", 2024-02-09T17:27:00.571 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0072840209999999999 2024-02-09T17:27:00.571 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.571 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.571 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:00.571 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.032781+0000", 2024-02-09T17:27:00.571 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0037337289999999999 2024-02-09T17:27:00.571 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.571 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.571 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:00.571 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.045735+0000", 2024-02-09T17:27:00.571 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012953800999999999 2024-02-09T17:27:00.571 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.571 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.572 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:27:00.572 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.045768+0000", 2024-02-09T17:27:00.572 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3130999999999998e-05 2024-02-09T17:27:00.572 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.572 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.572 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:27:00.572 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.045791+0000", 2024-02-09T17:27:00.572 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3479e-05 2024-02-09T17:27:00.572 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:27:00.572 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:27:00.572 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:27:00.572 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.572 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.572 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14790.0:169927 2.7 2:e1a48d61:::benchmark_data_smithi179_77138_object36519:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-02-09T17:27:00.572 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:27:00.014319+0000", 2024-02-09T17:27:00.572 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.46899928800000001, 2024-02-09T17:27:00.573 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.047063630000000002, 2024-02-09T17:27:00.573 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:27:00.573 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:00.573 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:27:00.573 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14790", 2024-02-09T17:27:00.573 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:27:00.573 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 169927 2024-02-09T17:27:00.573 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.573 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:27:00.573 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.573 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:27:00.573 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.014319+0000", 2024-02-09T17:27:00.573 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:27:00.573 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.573 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.573 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:27:00.573 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.014319+0000", 2024-02-09T17:27:00.573 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:27:00.574 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.574 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.574 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:27:00.574 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.014321+0000", 2024-02-09T17:27:00.574 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3239999999999999e-06 2024-02-09T17:27:00.574 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.574 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.574 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:27:00.574 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.014324+0000", 2024-02-09T17:27:00.574 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7699999999999999e-06 2024-02-09T17:27:00.574 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.574 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.574 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:27:00.574 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.014335+0000", 2024-02-09T17:27:00.574 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0933e-05 2024-02-09T17:27:00.574 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.574 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.575 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:27:00.575 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.014341+0000", 2024-02-09T17:27:00.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.823e-06 2024-02-09T17:27:00.575 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.575 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.575 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:27:00.575 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.025375+0000", 2024-02-09T17:27:00.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011034297 2024-02-09T17:27:00.575 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.575 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.575 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:27:00.575 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.025485+0000", 2024-02-09T17:27:00.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000109268 2024-02-09T17:27:00.575 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.575 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.575 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,10", 2024-02-09T17:27:00.576 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.025553+0000", 2024-02-09T17:27:00.576 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8764999999999996e-05 2024-02-09T17:27:00.576 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.576 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.576 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:27:00.576 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.032899+0000", 2024-02-09T17:27:00.576 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.007345575 2024-02-09T17:27:00.576 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.576 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.576 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:00.576 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.045630+0000", 2024-02-09T17:27:00.576 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012731436 2024-02-09T17:27:00.576 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.576 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.577 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:00.577 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.061280+0000", 2024-02-09T17:27:00.577 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015649111 2024-02-09T17:27:00.577 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.577 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.577 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:27:00.577 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.061320+0000", 2024-02-09T17:27:00.577 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0871000000000002e-05 2024-02-09T17:27:00.577 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.577 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.577 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:27:00.577 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.061383+0000", 2024-02-09T17:27:00.577 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2457000000000002e-05 2024-02-09T17:27:00.577 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:27:00.577 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:27:00.577 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:27:00.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.578 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.578 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14790.0:169928 2.7 2:e31d55e3:::benchmark_data_smithi179_77138_object36520:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-02-09T17:27:00.578 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:27:00.014352+0000", 2024-02-09T17:27:00.578 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.46896673100000003, 2024-02-09T17:27:00.578 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.047188510000000003, 2024-02-09T17:27:00.578 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:27:00.578 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:00.578 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:27:00.578 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14790", 2024-02-09T17:27:00.578 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:27:00.578 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 169928 2024-02-09T17:27:00.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.578 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:27:00.578 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.579 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:27:00.579 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.014352+0000", 2024-02-09T17:27:00.579 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:27:00.579 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.579 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.579 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:27:00.579 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.014352+0000", 2024-02-09T17:27:00.579 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:27:00.579 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.579 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.579 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:27:00.579 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.014352+0000", 2024-02-09T17:27:00.579 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9299999999999999e-07 2024-02-09T17:27:00.579 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.579 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.580 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:27:00.580 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.014354+0000", 2024-02-09T17:27:00.580 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5549999999999999e-06 2024-02-09T17:27:00.580 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.580 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.580 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:27:00.580 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.014359+0000", 2024-02-09T17:27:00.580 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8890000000000001e-06 2024-02-09T17:27:00.580 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.580 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.580 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:27:00.580 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.014362+0000", 2024-02-09T17:27:00.580 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3840000000000001e-06 2024-02-09T17:27:00.580 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.580 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.581 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:27:00.581 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.029091+0000", 2024-02-09T17:27:00.581 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014729106 2024-02-09T17:27:00.581 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.581 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.581 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:27:00.581 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.029209+0000", 2024-02-09T17:27:00.581 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000118176 2024-02-09T17:27:00.581 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.581 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.581 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,10", 2024-02-09T17:27:00.581 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.029340+0000", 2024-02-09T17:27:00.581 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013079300000000001 2024-02-09T17:27:00.581 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.581 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.581 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:27:00.582 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.041255+0000", 2024-02-09T17:27:00.582 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.01191478 2024-02-09T17:27:00.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.582 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.582 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:00.582 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.045668+0000", 2024-02-09T17:27:00.582 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0044134170000000002 2024-02-09T17:27:00.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.582 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.582 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:00.582 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.061440+0000", 2024-02-09T17:27:00.582 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.01577228 2024-02-09T17:27:00.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.582 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.582 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:27:00.582 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.061505+0000", 2024-02-09T17:27:00.583 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4030999999999997e-05 2024-02-09T17:27:00.583 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.583 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.583 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:27:00.583 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.061540+0000", 2024-02-09T17:27:00.583 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5806000000000002e-05 2024-02-09T17:27:00.583 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:27:00.583 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:27:00.583 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:27:00.583 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.583 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.583 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14790.0:169929 2.7 2:ea7ea63f:::benchmark_data_smithi179_77138_object36521:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-02-09T17:27:00.583 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:27:00.017431+0000", 2024-02-09T17:27:00.583 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.46588747400000002, 2024-02-09T17:27:00.583 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.044249262999999997, 2024-02-09T17:27:00.583 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:27:00.583 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:00.584 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:27:00.584 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14790", 2024-02-09T17:27:00.584 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:27:00.584 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 169929 2024-02-09T17:27:00.584 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.584 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:27:00.584 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.584 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:27:00.584 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.017431+0000", 2024-02-09T17:27:00.584 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:27:00.584 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.584 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.584 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:27:00.584 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.017431+0000", 2024-02-09T17:27:00.584 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:27:00.585 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.585 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.585 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:27:00.585 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.017432+0000", 2024-02-09T17:27:00.585 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.051e-06 2024-02-09T17:27:00.585 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.585 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.585 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:27:00.585 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.017436+0000", 2024-02-09T17:27:00.585 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2339999999999996e-06 2024-02-09T17:27:00.585 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.585 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.585 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:27:00.585 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.017445+0000", 2024-02-09T17:27:00.585 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9390000000000003e-06 2024-02-09T17:27:00.585 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.586 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.586 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:27:00.586 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.017451+0000", 2024-02-09T17:27:00.586 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1959999999999999e-06 2024-02-09T17:27:00.586 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.586 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.586 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:27:00.586 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.032939+0000", 2024-02-09T17:27:00.586 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015488475 2024-02-09T17:27:00.586 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.586 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.586 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:27:00.586 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.033045+0000", 2024-02-09T17:27:00.586 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010597199999999999 2024-02-09T17:27:00.586 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.587 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.587 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,10", 2024-02-09T17:27:00.587 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.033161+0000", 2024-02-09T17:27:00.587 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000116298 2024-02-09T17:27:00.587 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.587 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.587 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:27:00.587 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.045642+0000", 2024-02-09T17:27:00.587 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012480618000000001 2024-02-09T17:27:00.587 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.587 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.587 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:00.587 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.045698+0000", 2024-02-09T17:27:00.587 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5708999999999998e-05 2024-02-09T17:27:00.587 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.588 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.588 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:00.588 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.061596+0000", 2024-02-09T17:27:00.588 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015898511000000001 2024-02-09T17:27:00.588 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.588 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.588 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:27:00.588 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.061650+0000", 2024-02-09T17:27:00.588 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3848999999999999e-05 2024-02-09T17:27:00.588 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.588 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.588 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:27:00.588 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.061680+0000", 2024-02-09T17:27:00.588 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0411000000000001e-05 2024-02-09T17:27:00.589 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:27:00.589 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:27:00.589 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:27:00.589 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.589 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.589 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14790.0:169937 2.7 2:e268d2c8:::benchmark_data_smithi179_77138_object36529:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-02-09T17:27:00.589 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:27:00.025045+0000", 2024-02-09T17:27:00.589 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.45827324800000002, 2024-02-09T17:27:00.589 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.036769069000000001, 2024-02-09T17:27:00.589 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:27:00.589 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:00.589 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:27:00.589 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14790", 2024-02-09T17:27:00.589 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:27:00.589 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 169937 2024-02-09T17:27:00.589 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.589 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:27:00.590 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.590 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:27:00.590 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.025045+0000", 2024-02-09T17:27:00.590 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:27:00.590 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.590 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.590 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:27:00.590 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.025045+0000", 2024-02-09T17:27:00.590 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:27:00.590 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.590 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.590 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:27:00.590 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.025046+0000", 2024-02-09T17:27:00.590 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.139e-06 2024-02-09T17:27:00.590 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.591 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.591 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:27:00.591 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.025051+0000", 2024-02-09T17:27:00.591 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3019999999999996e-06 2024-02-09T17:27:00.591 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.591 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.591 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:27:00.591 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.025060+0000", 2024-02-09T17:27:00.591 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.935e-06 2024-02-09T17:27:00.591 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.591 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.591 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:27:00.591 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.025069+0000", 2024-02-09T17:27:00.591 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8249999999999994e-06 2024-02-09T17:27:00.591 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.592 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.592 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:27:00.592 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.037067+0000", 2024-02-09T17:27:00.592 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011998086 2024-02-09T17:27:00.592 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.592 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.592 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:27:00.592 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.037189+0000", 2024-02-09T17:27:00.592 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000121951 2024-02-09T17:27:00.592 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.592 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.592 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,10", 2024-02-09T17:27:00.592 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.037317+0000", 2024-02-09T17:27:00.592 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012841 2024-02-09T17:27:00.592 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.592 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.593 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:27:00.593 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.045707+0000", 2024-02-09T17:27:00.593 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0083900720000000002 2024-02-09T17:27:00.593 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.593 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.593 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:00.593 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.045840+0000", 2024-02-09T17:27:00.593 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013333400000000001 2024-02-09T17:27:00.593 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.593 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.593 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:00.593 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.061734+0000", 2024-02-09T17:27:00.593 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015894005999999999 2024-02-09T17:27:00.593 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.593 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.594 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:27:00.594 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.061783+0000", 2024-02-09T17:27:00.594 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8201999999999997e-05 2024-02-09T17:27:00.594 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.594 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.594 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:27:00.594 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.061814+0000", 2024-02-09T17:27:00.594 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1807000000000002e-05 2024-02-09T17:27:00.594 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:27:00.594 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:27:00.594 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:27:00.594 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.594 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.594 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14790.0:169945 2.7 2:e0943b70:::benchmark_data_smithi179_77138_object36537:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-02-09T17:27:00.594 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:27:00.034769+0000", 2024-02-09T17:27:00.594 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.44854963199999998, 2024-02-09T17:27:00.594 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.040366899999999997, 2024-02-09T17:27:00.595 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:27:00.595 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:00.595 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:27:00.595 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14790", 2024-02-09T17:27:00.595 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:27:00.595 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 169945 2024-02-09T17:27:00.595 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.595 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:27:00.595 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.595 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:27:00.595 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.034769+0000", 2024-02-09T17:27:00.595 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:27:00.595 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.595 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.595 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:27:00.595 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.034769+0000", 2024-02-09T17:27:00.596 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:27:00.596 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.596 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.596 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:27:00.596 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.034770+0000", 2024-02-09T17:27:00.596 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3599999999999999e-06 2024-02-09T17:27:00.596 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.596 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.596 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:27:00.596 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.034776+0000", 2024-02-09T17:27:00.596 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5280000000000003e-06 2024-02-09T17:27:00.596 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.596 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.596 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:27:00.596 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.034791+0000", 2024-02-09T17:27:00.597 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5270000000000001e-05 2024-02-09T17:27:00.597 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.597 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.597 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:27:00.597 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.034799+0000", 2024-02-09T17:27:00.597 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9440000000000005e-06 2024-02-09T17:27:00.597 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.597 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.597 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:27:00.597 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.045868+0000", 2024-02-09T17:27:00.597 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011068568000000001 2024-02-09T17:27:00.597 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.597 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.597 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:27:00.597 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.045944+0000", 2024-02-09T17:27:00.597 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6108999999999999e-05 2024-02-09T17:27:00.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.598 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.598 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,10", 2024-02-09T17:27:00.598 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.046019+0000", 2024-02-09T17:27:00.598 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5486000000000002e-05 2024-02-09T17:27:00.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.598 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.598 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:27:00.598 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.049520+0000", 2024-02-09T17:27:00.598 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0035008079999999998 2024-02-09T17:27:00.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.598 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.598 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:00.598 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.051333+0000", 2024-02-09T17:27:00.598 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018128739999999999 2024-02-09T17:27:00.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.599 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.599 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:00.599 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.075060+0000", 2024-02-09T17:27:00.599 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.023727549000000001 2024-02-09T17:27:00.599 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.599 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.599 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:27:00.599 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.075096+0000", 2024-02-09T17:27:00.599 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5815999999999997e-05 2024-02-09T17:27:00.599 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:27:00.599 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:27:00.599 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:27:00.599 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:27:00.075136+0000", 2024-02-09T17:27:00.599 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9588000000000001e-05 2024-02-09T17:27:00.599 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:27:00.599 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:27:00.600 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:27:00.600 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:27:00.600 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:27:00.600 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:27:00.609 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:27:00.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:00.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:27:00.713 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:27:00.763 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:00.790 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:27:00.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:00.793 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:27:00.894 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:27:00.926 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:01.195 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-09T17:27:01.279 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:27:01.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:01.282 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:27:01.383 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:27:01.416 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:01.566 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:27:01.701 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:01 smithi099 ceph-mon[39420]: pgmap v458: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 6.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 890 op/s; 24679/295248 objects degraded (8.359%) 2024-02-09T17:27:01.702 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:01 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 24679/295248 objects degraded (8.359%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:27:01.749 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:27:01.971 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:27:01.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:01.978 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:27:01.978 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:01 smithi179 ceph-mon[43975]: pgmap v458: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 6.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 890 op/s; 24679/295248 objects degraded (8.359%) 2024-02-09T17:27:01.978 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:01 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 24679/295248 objects degraded (8.359%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:27:01.979 INFO:teuthology.orchestra.run.smithi179.stderr:Creating an OSD daemon form without an OSD FSID value 2024-02-09T17:27:02.079 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:27:02.105 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:01 smithi162 ceph-mon[44304]: pgmap v458: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 6.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 890 op/s; 24679/295248 objects degraded (8.359%) 2024-02-09T17:27:02.105 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:01 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 24679/295248 objects degraded (8.359%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:27:02.147 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:02.204 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:27:02.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:02.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:27:02.311 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:27:02.387 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:02.832 INFO:teuthology.orchestra.run.smithi179.stdout:get_pg_num_history pg_num_history pg_num_history(e145 pg_nums {1={20=1},2={111=16}} deleted_pools ) 2024-02-09T17:27:02.832 INFO:teuthology.orchestra.run.smithi179.stdout:Importing pgid 2.f 2024-02-09T17:27:02.841 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0009b1d:::benchmark_data_smithi179_77138_object76587:head# 2024-02-09T17:27:02.841 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.843 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f00145b2:::benchmark_data_smithi179_77138_object92940:head# 2024-02-09T17:27:02.843 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.844 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0014e91:::benchmark_data_smithi179_77138_object25924:head# 2024-02-09T17:27:02.844 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.845 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f001b70e:::benchmark_data_smithi179_77138_object86702:head# 2024-02-09T17:27:02.845 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.845 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f00277de:::benchmark_data_smithi179_77138_object101862:head# 2024-02-09T17:27:02.845 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.845 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f002fc04:::benchmark_data_smithi179_77138_object103750:head# 2024-02-09T17:27:02.845 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.845 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0049663:::benchmark_data_smithi179_77138_object78361:head# 2024-02-09T17:27:02.845 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.846 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f005d6de:::benchmark_data_smithi179_77138_object83949:head# 2024-02-09T17:27:02.846 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.846 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f005ee7d:::benchmark_data_smithi179_77138_object93644:head# 2024-02-09T17:27:02.846 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.846 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f00665ea:::benchmark_data_smithi179_77138_object116732:head# 2024-02-09T17:27:02.846 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.847 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f007069d:::benchmark_data_smithi179_77138_object44669:head# 2024-02-09T17:27:02.847 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.847 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0071423:::benchmark_data_smithi179_77138_object78642:head# 2024-02-09T17:27:02.847 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.847 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f007ca94:::benchmark_data_smithi179_77138_object52821:head# 2024-02-09T17:27:02.847 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.847 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f007d73e:::benchmark_data_smithi179_77138_object39310:head# 2024-02-09T17:27:02.847 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.849 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0082e16:::benchmark_data_smithi179_77138_object62498:head# 2024-02-09T17:27:02.849 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.849 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0087052:::benchmark_data_smithi179_77138_object86817:head# 2024-02-09T17:27:02.849 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.849 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f009f166:::benchmark_data_smithi179_77138_object103815:head# 2024-02-09T17:27:02.849 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.849 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f00aa2a9:::benchmark_data_smithi179_77138_object121051:head# 2024-02-09T17:27:02.849 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.850 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f00cc996:::benchmark_data_smithi179_77138_object61202:head# 2024-02-09T17:27:02.850 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.850 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f00ccdc1:::benchmark_data_smithi179_77138_object94720:head# 2024-02-09T17:27:02.850 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.850 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f00d1199:::benchmark_data_smithi179_77138_object131602:head# 2024-02-09T17:27:02.850 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.852 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f00d3f76:::benchmark_data_smithi179_77138_object61291:head# 2024-02-09T17:27:02.852 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.852 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f00dcac1:::benchmark_data_smithi179_77138_object106431:head# 2024-02-09T17:27:02.852 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.852 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f00dd4cd:::benchmark_data_smithi179_77138_object78366:head# 2024-02-09T17:27:02.852 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.852 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f00e639e:::benchmark_data_smithi179_77138_object109593:head# 2024-02-09T17:27:02.852 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.852 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f00eeb75:::benchmark_data_smithi179_77138_object42343:head# 2024-02-09T17:27:02.852 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.852 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f00f5c07:::benchmark_data_smithi179_77138_object101059:head# 2024-02-09T17:27:02.852 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.852 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f00fdd95:::benchmark_data_smithi179_77138_object62588:head# 2024-02-09T17:27:02.852 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.853 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0100ff1:::benchmark_data_smithi179_77138_object83033:head# 2024-02-09T17:27:02.853 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.853 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0103c44:::benchmark_data_smithi179_77138_object62435:head# 2024-02-09T17:27:02.853 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.855 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f010525a:::benchmark_data_smithi179_77138_object50524:head# 2024-02-09T17:27:02.855 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.855 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f01071b2:::benchmark_data_smithi179_77138_object64831:head# 2024-02-09T17:27:02.855 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.855 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f011e672:::benchmark_data_smithi179_77138_object50037:head# 2024-02-09T17:27:02.855 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.855 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f012680f:::benchmark_data_smithi179_77138_object108555:head# 2024-02-09T17:27:02.855 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.855 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f01271e2:::benchmark_data_smithi179_77138_object74125:head# 2024-02-09T17:27:02.856 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.856 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f012dcab:::benchmark_data_smithi179_77138_object38915:head# 2024-02-09T17:27:02.856 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.856 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f014621f:::benchmark_data_smithi179_77138_object81355:head# 2024-02-09T17:27:02.856 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.856 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0154249:::benchmark_data_smithi179_77138_object19798:head# 2024-02-09T17:27:02.856 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.856 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f015cf31:::benchmark_data_smithi179_77138_object106013:head# 2024-02-09T17:27:02.856 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.856 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0164466:::benchmark_data_smithi179_77138_object122578:head# 2024-02-09T17:27:02.856 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.856 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0165fcd:::benchmark_data_smithi179_77138_object26528:head# 2024-02-09T17:27:02.856 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.856 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0170933:::benchmark_data_smithi179_77138_object63557:head# 2024-02-09T17:27:02.857 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.857 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f01786c7:::benchmark_data_smithi179_77138_object68930:head# 2024-02-09T17:27:02.857 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.857 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f019c5ec:::benchmark_data_smithi179_77138_object27826:head# 2024-02-09T17:27:02.857 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.860 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f019d14a:::benchmark_data_smithi179_77138_object71945:head# 2024-02-09T17:27:02.860 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.860 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f019ec6c:::benchmark_data_smithi179_77138_object113765:head# 2024-02-09T17:27:02.861 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.861 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f01aa986:::benchmark_data_smithi179_77138_object70740:head# 2024-02-09T17:27:02.861 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.861 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f01c8ca0:::benchmark_data_smithi179_77138_object63796:head# 2024-02-09T17:27:02.861 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.861 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f01ca4af:::benchmark_data_smithi179_77138_object80981:head# 2024-02-09T17:27:02.861 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.861 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f01d2e45:::benchmark_data_smithi179_77138_object105114:head# 2024-02-09T17:27:02.861 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.861 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f01d578f:::benchmark_data_smithi179_77138_object121030:head# 2024-02-09T17:27:02.861 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.861 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f01e59c9:::benchmark_data_smithi179_77138_object70880:head# 2024-02-09T17:27:02.861 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.861 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f01f6a98:::benchmark_data_smithi179_77138_object108599:head# 2024-02-09T17:27:02.862 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.862 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0200abe:::benchmark_data_smithi179_77138_object93795:head# 2024-02-09T17:27:02.862 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.862 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0209489:::benchmark_data_smithi179_77138_object71861:head# 2024-02-09T17:27:02.862 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.862 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f020b167:::benchmark_data_smithi179_77138_object70908:head# 2024-02-09T17:27:02.862 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.862 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f020bd30:::benchmark_data_smithi179_77138_object56493:head# 2024-02-09T17:27:02.862 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.862 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f021fb1a:::benchmark_data_smithi179_77138_object115266:head# 2024-02-09T17:27:02.862 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.862 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0234d87:::benchmark_data_smithi179_77138_object44098:head# 2024-02-09T17:27:02.862 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.862 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0239b60:::benchmark_data_smithi179_77138_object50518:head# 2024-02-09T17:27:02.863 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.863 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f023de06:::benchmark_data_smithi179_77138_object83562:head# 2024-02-09T17:27:02.863 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.863 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f02475ed:::benchmark_data_smithi179_77138_object19942:head# 2024-02-09T17:27:02.863 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.863 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f024ddbc:::benchmark_data_smithi179_77138_object86947:head# 2024-02-09T17:27:02.863 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.863 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f024fc6d:::benchmark_data_smithi179_77138_object63134:head# 2024-02-09T17:27:02.863 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.863 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f025b2e9:::benchmark_data_smithi179_77138_object91726:head# 2024-02-09T17:27:02.863 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.871 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f025f8c0:::benchmark_data_smithi179_77138_object128027:head# 2024-02-09T17:27:02.871 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.871 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f026b703:::benchmark_data_smithi179_77138_object59730:head# 2024-02-09T17:27:02.871 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.871 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f028626e:::benchmark_data_smithi179_77138_object107889:head# 2024-02-09T17:27:02.871 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.871 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f02a8e8d:::benchmark_data_smithi179_77138_object108825:head# 2024-02-09T17:27:02.871 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.872 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f02acf3b:::benchmark_data_smithi179_77138_object58076:head# 2024-02-09T17:27:02.872 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.872 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f02b330f:::benchmark_data_smithi179_77138_object128853:head# 2024-02-09T17:27:02.872 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.872 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f02cb235:::benchmark_data_smithi179_77138_object23970:head# 2024-02-09T17:27:02.872 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.872 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f02cdca4:::benchmark_data_smithi179_77138_object82479:head# 2024-02-09T17:27:02.872 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.872 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f02f0913:::benchmark_data_smithi179_77138_object88860:head# 2024-02-09T17:27:02.872 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.872 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f02f7251:::benchmark_data_smithi179_77138_object61612:head# 2024-02-09T17:27:02.872 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.873 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f02f91ec:::benchmark_data_smithi179_77138_object63392:head# 2024-02-09T17:27:02.873 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.873 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0300b1f:::benchmark_data_smithi179_77138_object26272:head# 2024-02-09T17:27:02.873 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.873 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0313443:::benchmark_data_smithi179_77138_object119842:head# 2024-02-09T17:27:02.873 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.873 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f03135ca:::benchmark_data_smithi179_77138_object102514:head# 2024-02-09T17:27:02.873 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.873 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f031b2e3:::benchmark_data_smithi179_77138_object46384:head# 2024-02-09T17:27:02.873 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.873 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0325e8c:::benchmark_data_smithi179_77138_object76009:head# 2024-02-09T17:27:02.873 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.873 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f032c8a8:::benchmark_data_smithi179_77138_object46933:head# 2024-02-09T17:27:02.873 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.874 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0334609:::benchmark_data_smithi179_77138_object67238:head# 2024-02-09T17:27:02.874 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.874 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f033b26a:::benchmark_data_smithi179_77138_object41806:head# 2024-02-09T17:27:02.874 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.874 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f033bcff:::benchmark_data_smithi179_77138_object81143:head# 2024-02-09T17:27:02.874 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.874 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f033e5dc:::benchmark_data_smithi179_77138_object36920:head# 2024-02-09T17:27:02.874 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.874 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f034a3ba:::benchmark_data_smithi179_77138_object57026:head# 2024-02-09T17:27:02.874 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.874 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f034c516:::benchmark_data_smithi179_77138_object113382:head# 2024-02-09T17:27:02.874 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.874 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f035dca4:::benchmark_data_smithi179_77138_object71087:head# 2024-02-09T17:27:02.874 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.874 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0362761:::benchmark_data_smithi179_77138_object32135:head# 2024-02-09T17:27:02.875 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.875 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0363375:::benchmark_data_smithi179_77138_object52864:head# 2024-02-09T17:27:02.875 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.875 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f036f0de:::benchmark_data_smithi179_77138_object43229:head# 2024-02-09T17:27:02.875 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.875 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0375fd4:::benchmark_data_smithi179_77138_object99643:head# 2024-02-09T17:27:02.875 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.875 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f037a121:::benchmark_data_smithi179_77138_object131840:head# 2024-02-09T17:27:02.875 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.875 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0384e9e:::benchmark_data_smithi179_77138_object83483:head# 2024-02-09T17:27:02.875 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.875 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f038e875:::benchmark_data_smithi179_77138_object97332:head# 2024-02-09T17:27:02.875 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.875 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f039ae41:::benchmark_data_smithi179_77138_object38478:head# 2024-02-09T17:27:02.875 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.876 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f03a559f:::benchmark_data_smithi179_77138_object131911:head# 2024-02-09T17:27:02.876 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.876 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f03b4be1:::benchmark_data_smithi179_77138_object84639:head# 2024-02-09T17:27:02.876 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.876 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f03c5c31:::benchmark_data_smithi179_77138_object123876:head# 2024-02-09T17:27:02.876 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.876 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f03d4c75:::benchmark_data_smithi179_77138_object121138:head# 2024-02-09T17:27:02.876 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.876 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f03d7c74:::benchmark_data_smithi179_77138_object61693:head# 2024-02-09T17:27:02.876 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.876 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f03da106:::benchmark_data_smithi179_77138_object110589:head# 2024-02-09T17:27:02.876 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.876 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f03e0107:::benchmark_data_smithi179_77138_object51014:head# 2024-02-09T17:27:02.876 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.877 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f03e4330:::benchmark_data_smithi179_77138_object38198:head# 2024-02-09T17:27:02.877 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.877 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f03ec1cb:::benchmark_data_smithi179_77138_object74956:head# 2024-02-09T17:27:02.877 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.877 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f03ee2e3:::benchmark_data_smithi179_77138_object95933:head# 2024-02-09T17:27:02.877 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.877 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f040592a:::benchmark_data_smithi179_77138_object50936:head# 2024-02-09T17:27:02.877 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.877 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0416e7a:::benchmark_data_smithi179_77138_object26840:head# 2024-02-09T17:27:02.877 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.877 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0418f63:::benchmark_data_smithi179_77138_object74194:head# 2024-02-09T17:27:02.877 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.877 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0428949:::benchmark_data_smithi179_77138_object44821:head# 2024-02-09T17:27:02.877 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.877 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0435b36:::benchmark_data_smithi179_77138_object107139:head# 2024-02-09T17:27:02.878 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.884 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0439ef1:::benchmark_data_smithi179_77138_object76874:head# 2024-02-09T17:27:02.884 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.884 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f043eed2:::benchmark_data_smithi179_77138_object81403:head# 2024-02-09T17:27:02.884 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.884 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f04473e5:::benchmark_data_smithi179_77138_object115473:head# 2024-02-09T17:27:02.884 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.884 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0465b7d:::benchmark_data_smithi179_77138_object118594:head# 2024-02-09T17:27:02.884 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.884 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0477f76:::benchmark_data_smithi179_77138_object123816:head# 2024-02-09T17:27:02.884 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.884 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f047cf49:::benchmark_data_smithi179_77138_object98654:head# 2024-02-09T17:27:02.884 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.884 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f04845e5:::benchmark_data_smithi179_77138_object30575:head# 2024-02-09T17:27:02.885 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.885 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f048b013:::benchmark_data_smithi179_77138_object29666:head# 2024-02-09T17:27:02.885 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.885 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f048fede:::benchmark_data_smithi179_77138_object20727:head# 2024-02-09T17:27:02.885 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.885 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f049bfc6:::benchmark_data_smithi179_77138_object120147:head# 2024-02-09T17:27:02.885 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.885 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f049cf2b:::benchmark_data_smithi179_77138_object80727:head# 2024-02-09T17:27:02.885 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.885 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f049ef92:::benchmark_data_smithi179_77138_object78866:head# 2024-02-09T17:27:02.885 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.885 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f04b07df:::benchmark_data_smithi179_77138_object20620:head# 2024-02-09T17:27:02.885 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.885 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f04bd63c:::benchmark_data_smithi179_77138_object32172:head# 2024-02-09T17:27:02.885 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.886 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f04d3f89:::benchmark_data_smithi179_77138_object26552:head# 2024-02-09T17:27:02.886 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.886 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f04de1f1:::benchmark_data_smithi179_77138_object131750:head# 2024-02-09T17:27:02.886 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.886 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f04df140:::benchmark_data_smithi179_77138_object67950:head# 2024-02-09T17:27:02.886 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.886 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f04ee360:::benchmark_data_smithi179_77138_object57352:head# 2024-02-09T17:27:02.886 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.886 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f04f467d:::benchmark_data_smithi179_77138_object26578:head# 2024-02-09T17:27:02.886 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.886 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0516145:::benchmark_data_smithi179_77138_object83753:head# 2024-02-09T17:27:02.886 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.886 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f052a853:::benchmark_data_smithi179_77138_object65318:head# 2024-02-09T17:27:02.886 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.887 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f052d2e3:::benchmark_data_smithi179_77138_object35833:head# 2024-02-09T17:27:02.887 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.887 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0551d31:::benchmark_data_smithi179_77138_object100194:head# 2024-02-09T17:27:02.887 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.887 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0563dec:::benchmark_data_smithi179_77138_object89077:head# 2024-02-09T17:27:02.887 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.887 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f056521f:::benchmark_data_smithi179_77138_object53693:head# 2024-02-09T17:27:02.887 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.887 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0567086:::benchmark_data_smithi179_77138_object116262:head# 2024-02-09T17:27:02.887 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.887 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0573f51:::benchmark_data_smithi179_77138_object27748:head# 2024-02-09T17:27:02.887 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.887 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f057920a:::benchmark_data_smithi179_77138_object30307:head# 2024-02-09T17:27:02.887 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.887 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f05810a4:::benchmark_data_smithi179_77138_object127587:head# 2024-02-09T17:27:02.888 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.888 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0585547:::benchmark_data_smithi179_77138_object35736:head# 2024-02-09T17:27:02.888 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.888 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f05910da:::benchmark_data_smithi179_77138_object64915:head# 2024-02-09T17:27:02.888 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.888 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0594072:::benchmark_data_smithi179_77138_object116293:head# 2024-02-09T17:27:02.888 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.888 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0598223:::benchmark_data_smithi179_77138_object101491:head# 2024-02-09T17:27:02.888 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.888 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f05a7377:::benchmark_data_smithi179_77138_object75626:head# 2024-02-09T17:27:02.888 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.888 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f05aacf9:::benchmark_data_smithi179_77138_object34727:head# 2024-02-09T17:27:02.888 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.888 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f05ad007:::benchmark_data_smithi179_77138_object60356:head# 2024-02-09T17:27:02.889 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.889 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f05b0af0:::benchmark_data_smithi179_77138_object101434:head# 2024-02-09T17:27:02.889 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.889 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f05b0db8:::benchmark_data_smithi179_77138_object130256:head# 2024-02-09T17:27:02.889 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.889 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f05b4d95:::benchmark_data_smithi179_77138_object89574:head# 2024-02-09T17:27:02.889 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.889 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f05c1cc5:::benchmark_data_smithi179_77138_object38207:head# 2024-02-09T17:27:02.889 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.889 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f05cc820:::benchmark_data_smithi179_77138_object103662:head# 2024-02-09T17:27:02.889 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.889 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f05cd621:::benchmark_data_smithi179_77138_object65959:head# 2024-02-09T17:27:02.889 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.889 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f05d3e2d:::benchmark_data_smithi179_77138_object34042:head# 2024-02-09T17:27:02.889 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.890 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f05e1f1e:::benchmark_data_smithi179_77138_object87156:head# 2024-02-09T17:27:02.890 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.890 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f060294c:::benchmark_data_smithi179_77138_object86460:head# 2024-02-09T17:27:02.890 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.890 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0604612:::benchmark_data_smithi179_77138_object76223:head# 2024-02-09T17:27:02.890 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.890 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f06057ad:::benchmark_data_smithi179_77138_object124688:head# 2024-02-09T17:27:02.890 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.896 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0615ebc:::benchmark_data_smithi179_77138_object108892:head# 2024-02-09T17:27:02.896 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.896 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f06360d5:::benchmark_data_smithi179_77138_object60783:head# 2024-02-09T17:27:02.896 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.896 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f06480cd:::benchmark_data_smithi179_77138_object35648:head# 2024-02-09T17:27:02.896 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.896 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f064c10e:::benchmark_data_smithi179_77138_object22594:head# 2024-02-09T17:27:02.896 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.896 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0650250:::benchmark_data_smithi179_77138_object120584:head# 2024-02-09T17:27:02.896 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.897 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f065c158:::benchmark_data_smithi179_77138_object90564:head# 2024-02-09T17:27:02.897 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.897 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f066152c:::benchmark_data_smithi179_77138_object89900:head# 2024-02-09T17:27:02.897 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.897 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0663842:::benchmark_data_smithi179_77138_object75827:head# 2024-02-09T17:27:02.897 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.897 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f066deef:::benchmark_data_smithi179_77138_object115985:head# 2024-02-09T17:27:02.897 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.897 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f06716f5:::benchmark_data_smithi179_77138_object66697:head# 2024-02-09T17:27:02.897 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.897 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0678444:::benchmark_data_smithi179_77138_object79175:head# 2024-02-09T17:27:02.897 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.897 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f067af82:::benchmark_data_smithi179_77138_object126514:head# 2024-02-09T17:27:02.898 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.898 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0680e99:::benchmark_data_smithi179_77138_object43363:head# 2024-02-09T17:27:02.898 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.898 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f06872cb:::benchmark_data_smithi179_77138_object61325:head# 2024-02-09T17:27:02.898 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.898 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f068c5d9:::benchmark_data_smithi179_77138_object40311:head# 2024-02-09T17:27:02.898 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.898 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f068cadd:::benchmark_data_smithi179_77138_object85122:head# 2024-02-09T17:27:02.898 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.898 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f06a416c:::benchmark_data_smithi179_77138_object106684:head# 2024-02-09T17:27:02.898 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.898 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f06bb2e9:::benchmark_data_smithi179_77138_object55991:head# 2024-02-09T17:27:02.898 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.898 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f06d4f4c:::benchmark_data_smithi179_77138_object47474:head# 2024-02-09T17:27:02.898 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.899 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f06dbe1b:::benchmark_data_smithi179_77138_object38518:head# 2024-02-09T17:27:02.899 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.899 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f06dc076:::benchmark_data_smithi179_77138_object40603:head# 2024-02-09T17:27:02.899 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.899 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f06e667b:::benchmark_data_smithi179_77138_object89306:head# 2024-02-09T17:27:02.899 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.899 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f06fa039:::benchmark_data_smithi179_77138_object75676:head# 2024-02-09T17:27:02.899 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.899 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f06fd209:::benchmark_data_smithi179_77138_object76437:head# 2024-02-09T17:27:02.899 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.899 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f06fdb68:::benchmark_data_smithi179_77138_object120202:head# 2024-02-09T17:27:02.899 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.899 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0704a7c:::benchmark_data_smithi179_77138_object40641:head# 2024-02-09T17:27:02.900 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.900 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f07062eb:::benchmark_data_smithi179_77138_object88630:head# 2024-02-09T17:27:02.900 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.900 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f070e062:::benchmark_data_smithi179_77138_object29065:head# 2024-02-09T17:27:02.900 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.900 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0720703:::benchmark_data_smithi179_77138_object130842:head# 2024-02-09T17:27:02.900 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.900 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0722ab7:::benchmark_data_smithi179_77138_object33812:head# 2024-02-09T17:27:02.900 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.900 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0723db8:::benchmark_data_smithi179_77138_object78601:head# 2024-02-09T17:27:02.900 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.900 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f073ee3a:::benchmark_data_smithi179_77138_object125923:head# 2024-02-09T17:27:02.900 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.900 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0748175:::benchmark_data_smithi179_77138_object118901:head# 2024-02-09T17:27:02.901 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.901 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f075857d:::benchmark_data_smithi179_77138_object126127:head# 2024-02-09T17:27:02.901 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.901 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f075a4f5:::benchmark_data_smithi179_77138_object41105:head# 2024-02-09T17:27:02.901 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.901 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0768437:::benchmark_data_smithi179_77138_object66572:head# 2024-02-09T17:27:02.901 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.901 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f076e6d2:::benchmark_data_smithi179_77138_object49506:head# 2024-02-09T17:27:02.901 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.901 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f077ad54:::benchmark_data_smithi179_77138_object116092:head# 2024-02-09T17:27:02.901 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.901 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0785faf:::benchmark_data_smithi179_77138_object117969:head# 2024-02-09T17:27:02.901 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.901 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0788f0b:::benchmark_data_smithi179_77138_object29606:head# 2024-02-09T17:27:02.901 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.902 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f078dfcc:::benchmark_data_smithi179_77138_object77617:head# 2024-02-09T17:27:02.902 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.902 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f078e741:::benchmark_data_smithi179_77138_object57882:head# 2024-02-09T17:27:02.902 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.902 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f07904ea:::benchmark_data_smithi179_77138_object123909:head# 2024-02-09T17:27:02.902 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.902 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0797559:::benchmark_data_smithi179_77138_object111449:head# 2024-02-09T17:27:02.902 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.902 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f07985b9:::benchmark_data_smithi179_77138_object42329:head# 2024-02-09T17:27:02.902 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.902 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f079ce89:::benchmark_data_smithi179_77138_object108428:head# 2024-02-09T17:27:02.902 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.902 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f07a30c3:::benchmark_data_smithi179_77138_object78046:head# 2024-02-09T17:27:02.902 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.912 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f07ac5f7:::benchmark_data_smithi179_77138_object27022:head# 2024-02-09T17:27:02.912 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.912 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f07aea53:::benchmark_data_smithi179_77138_object98097:head# 2024-02-09T17:27:02.912 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.912 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f07b088a:::benchmark_data_smithi179_77138_object131346:head# 2024-02-09T17:27:02.912 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.912 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f07b4dd5:::benchmark_data_smithi179_77138_object31636:head# 2024-02-09T17:27:02.912 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.912 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f07c5067:::benchmark_data_smithi179_77138_object90879:head# 2024-02-09T17:27:02.912 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.912 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f07cd205:::benchmark_data_smithi179_77138_object41256:head# 2024-02-09T17:27:02.912 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.912 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f07de7ac:::benchmark_data_smithi179_77138_object22481:head# 2024-02-09T17:27:02.913 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.913 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f07e7712:::benchmark_data_smithi179_77138_object98951:head# 2024-02-09T17:27:02.913 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.913 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f07efe63:::benchmark_data_smithi179_77138_object118720:head# 2024-02-09T17:27:02.913 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.913 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f07f6816:::benchmark_data_smithi179_77138_object111026:head# 2024-02-09T17:27:02.913 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.913 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f080de51:::benchmark_data_smithi179_77138_object60737:head# 2024-02-09T17:27:02.913 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.913 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0812ab9:::benchmark_data_smithi179_77138_object82572:head# 2024-02-09T17:27:02.913 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.913 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0824acd:::benchmark_data_smithi179_77138_object98533:head# 2024-02-09T17:27:02.913 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.913 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f082ab2e:::benchmark_data_smithi179_77138_object20153:head# 2024-02-09T17:27:02.914 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.914 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0831511:::benchmark_data_smithi179_77138_object72833:head# 2024-02-09T17:27:02.914 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.914 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f083bb9f:::benchmark_data_smithi179_77138_object123483:head# 2024-02-09T17:27:02.914 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.914 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f083fcbf:::benchmark_data_smithi179_77138_object92193:head# 2024-02-09T17:27:02.914 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.914 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f08438c6:::benchmark_data_smithi179_77138_object116164:head# 2024-02-09T17:27:02.914 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.914 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0846fd3:::benchmark_data_smithi179_77138_object93218:head# 2024-02-09T17:27:02.914 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.914 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0847fef:::benchmark_data_smithi179_77138_object114962:head# 2024-02-09T17:27:02.914 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.914 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f08937a7:::benchmark_data_smithi179_77138_object86180:head# 2024-02-09T17:27:02.915 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.915 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f089f1c3:::benchmark_data_smithi179_77138_object103271:head# 2024-02-09T17:27:02.915 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.915 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f08a1f69:::benchmark_data_smithi179_77138_object127089:head# 2024-02-09T17:27:02.915 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.915 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f08a4b30:::benchmark_data_smithi179_77138_object113377:head# 2024-02-09T17:27:02.915 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.915 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f08aff12:::benchmark_data_smithi179_77138_object54940:head# 2024-02-09T17:27:02.915 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.915 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f08b6257:::benchmark_data_smithi179_77138_object90306:head# 2024-02-09T17:27:02.915 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.915 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f08bbef2:::benchmark_data_smithi179_77138_object104701:head# 2024-02-09T17:27:02.915 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.916 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f08c8715:::benchmark_data_smithi179_77138_object128388:head# 2024-02-09T17:27:02.916 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.916 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f08ced31:::benchmark_data_smithi179_77138_object85858:head# 2024-02-09T17:27:02.916 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.916 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f08d19df:::benchmark_data_smithi179_77138_object103987:head# 2024-02-09T17:27:02.916 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.916 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f08d5fd7:::benchmark_data_smithi179_77138_object41047:head# 2024-02-09T17:27:02.916 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.916 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f08e2674:::benchmark_data_smithi179_77138_object49455:head# 2024-02-09T17:27:02.916 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.916 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f08e8342:::benchmark_data_smithi179_77138_object133348:head# 2024-02-09T17:27:02.916 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.916 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f08e865f:::benchmark_data_smithi179_77138_object83189:head# 2024-02-09T17:27:02.917 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.917 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f08eed31:::benchmark_data_smithi179_77138_object38641:head# 2024-02-09T17:27:02.917 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.917 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f08f153d:::benchmark_data_smithi179_77138_object33995:head# 2024-02-09T17:27:02.917 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.917 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f08f7683:::benchmark_data_smithi179_77138_object64276:head# 2024-02-09T17:27:02.917 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.917 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f08fc08e:::benchmark_data_smithi179_77138_object117723:head# 2024-02-09T17:27:02.917 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.917 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0900b83:::benchmark_data_smithi179_77138_object105761:head# 2024-02-09T17:27:02.917 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.917 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0915201:::benchmark_data_smithi179_77138_object90425:head# 2024-02-09T17:27:02.917 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.917 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0924677:::benchmark_data_smithi179_77138_object85974:head# 2024-02-09T17:27:02.918 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.918 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f092c974:::benchmark_data_smithi179_77138_object44003:head# 2024-02-09T17:27:02.918 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.918 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f094e9ae:::benchmark_data_smithi179_77138_object122180:head# 2024-02-09T17:27:02.918 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.918 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0954375:::benchmark_data_smithi179_77138_object77134:head# 2024-02-09T17:27:02.918 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.918 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f09550e1:::benchmark_data_smithi179_77138_object58571:head# 2024-02-09T17:27:02.918 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.918 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f095cfb1:::benchmark_data_smithi179_77138_object76965:head# 2024-02-09T17:27:02.918 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.918 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f095ec9c:::benchmark_data_smithi179_77138_object107161:head# 2024-02-09T17:27:02.918 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.918 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0965a02:::benchmark_data_smithi179_77138_object57875:head# 2024-02-09T17:27:02.919 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.919 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f09681b3:::benchmark_data_smithi179_77138_object45461:head# 2024-02-09T17:27:02.919 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.919 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0974966:::benchmark_data_smithi179_77138_object108865:head# 2024-02-09T17:27:02.919 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.919 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f09753c7:::benchmark_data_smithi179_77138_object121991:head# 2024-02-09T17:27:02.919 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.919 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0985e37:::benchmark_data_smithi179_77138_object24771:head# 2024-02-09T17:27:02.919 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.919 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f099e566:::benchmark_data_smithi179_77138_object95579:head# 2024-02-09T17:27:02.919 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.919 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f09ab928:::benchmark_data_smithi179_77138_object31897:head# 2024-02-09T17:27:02.919 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.919 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f09af59d:::benchmark_data_smithi179_77138_object109134:head# 2024-02-09T17:27:02.920 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.920 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f09b25d4:::benchmark_data_smithi179_77138_object36086:head# 2024-02-09T17:27:02.920 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.921 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f09b7716:::benchmark_data_smithi179_77138_object108674:head# 2024-02-09T17:27:02.922 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.922 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f09bdebe:::benchmark_data_smithi179_77138_object40591:head# 2024-02-09T17:27:02.922 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.922 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f09be939:::benchmark_data_smithi179_77138_object78956:head# 2024-02-09T17:27:02.922 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.922 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f09cba00:::benchmark_data_smithi179_77138_object76345:head# 2024-02-09T17:27:02.922 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.922 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f09d9701:::benchmark_data_smithi179_77138_object128304:head# 2024-02-09T17:27:02.922 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.922 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f09dee97:::benchmark_data_smithi179_77138_object28282:head# 2024-02-09T17:27:02.922 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.922 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f09dff39:::benchmark_data_smithi179_77138_object40045:head# 2024-02-09T17:27:02.922 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.923 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f09ef315:::benchmark_data_smithi179_77138_object55997:head# 2024-02-09T17:27:02.923 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.923 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f09f1347:::benchmark_data_smithi179_77138_object21876:head# 2024-02-09T17:27:02.923 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.923 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f09f3ac0:::benchmark_data_smithi179_77138_object19753:head# 2024-02-09T17:27:02.923 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.923 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f09f9622:::benchmark_data_smithi179_77138_object63871:head# 2024-02-09T17:27:02.923 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.923 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0a0f333:::benchmark_data_smithi179_77138_object55989:head# 2024-02-09T17:27:02.923 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.923 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0a170d8:::benchmark_data_smithi179_77138_object117567:head# 2024-02-09T17:27:02.923 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.923 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0a19c02:::benchmark_data_smithi179_77138_object58900:head# 2024-02-09T17:27:02.923 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.923 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0a1e813:::benchmark_data_smithi179_77138_object124873:head# 2024-02-09T17:27:02.924 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.927 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0a282be:::benchmark_data_smithi179_77138_object21221:head# 2024-02-09T17:27:02.927 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.928 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0a2a253:::benchmark_data_smithi179_77138_object125327:head# 2024-02-09T17:27:02.928 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.928 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0a2c674:::benchmark_data_smithi179_77138_object20011:head# 2024-02-09T17:27:02.929 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.929 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0a350a1:::benchmark_data_smithi179_77138_object73561:head# 2024-02-09T17:27:02.929 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.929 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0a42ac2:::benchmark_data_smithi179_77138_object25933:head# 2024-02-09T17:27:02.929 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.931 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0a5334d:::benchmark_data_smithi179_77138_object117197:head# 2024-02-09T17:27:02.931 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.931 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0a53642:::benchmark_data_smithi179_77138_object48844:head# 2024-02-09T17:27:02.931 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.932 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0a53820:::benchmark_data_smithi179_77138_object37766:head# 2024-02-09T17:27:02.932 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.932 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0a66a30:::benchmark_data_smithi179_77138_object123538:head# 2024-02-09T17:27:02.932 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.932 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0a6f2eb:::benchmark_data_smithi179_77138_object86013:head# 2024-02-09T17:27:02.932 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.932 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0a7b7bc:::benchmark_data_smithi179_77138_object59250:head# 2024-02-09T17:27:02.932 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.932 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0a7fbd2:::benchmark_data_smithi179_77138_object91437:head# 2024-02-09T17:27:02.932 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.932 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0a81180:::benchmark_data_smithi179_77138_object25840:head# 2024-02-09T17:27:02.932 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.932 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0a8e769:::benchmark_data_smithi179_77138_object85336:head# 2024-02-09T17:27:02.933 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.934 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0aa7f32:::benchmark_data_smithi179_77138_object83238:head# 2024-02-09T17:27:02.934 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.934 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0abca32:::benchmark_data_smithi179_77138_object133004:head# 2024-02-09T17:27:02.934 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.935 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0ac4c80:::benchmark_data_smithi179_77138_object106395:head# 2024-02-09T17:27:02.935 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.935 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0acfae8:::benchmark_data_smithi179_77138_object122920:head# 2024-02-09T17:27:02.935 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.935 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0aeebd0:::benchmark_data_smithi179_77138_object95488:head# 2024-02-09T17:27:02.935 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.935 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0afaa0a:::benchmark_data_smithi179_77138_object49362:head# 2024-02-09T17:27:02.935 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.935 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0b2bcf3:::benchmark_data_smithi179_77138_object87032:head# 2024-02-09T17:27:02.935 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.935 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0b3a54c:::benchmark_data_smithi179_77138_object112172:head# 2024-02-09T17:27:02.935 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.935 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0b469cf:::benchmark_data_smithi179_77138_object40887:head# 2024-02-09T17:27:02.935 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.935 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0b62663:::benchmark_data_smithi179_77138_object36310:head# 2024-02-09T17:27:02.936 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.936 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0b6a12e:::benchmark_data_smithi179_77138_object109553:head# 2024-02-09T17:27:02.936 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.938 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0b6e65b:::benchmark_data_smithi179_77138_object118690:head# 2024-02-09T17:27:02.938 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.938 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0b72b0a:::benchmark_data_smithi179_77138_object128292:head# 2024-02-09T17:27:02.939 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.939 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0b7a02c:::benchmark_data_smithi179_77138_object23194:head# 2024-02-09T17:27:02.939 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.939 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0b84303:::benchmark_data_smithi179_77138_object91717:head# 2024-02-09T17:27:02.939 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.939 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0b9613c:::benchmark_data_smithi179_77138_object53511:head# 2024-02-09T17:27:02.939 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.939 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0ba24d7:::benchmark_data_smithi179_77138_object91361:head# 2024-02-09T17:27:02.939 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.939 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0ba3ced:::benchmark_data_smithi179_77138_object121721:head# 2024-02-09T17:27:02.939 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.939 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0baeee5:::benchmark_data_smithi179_77138_object103065:head# 2024-02-09T17:27:02.939 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.940 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0bb3c1f:::benchmark_data_smithi179_77138_object40898:head# 2024-02-09T17:27:02.940 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.940 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0bbe780:::benchmark_data_smithi179_77138_object70296:head# 2024-02-09T17:27:02.940 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.941 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0bbf8bb:::benchmark_data_smithi179_77138_object95071:head# 2024-02-09T17:27:02.941 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.941 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0bcdb42:::benchmark_data_smithi179_77138_object65432:head# 2024-02-09T17:27:02.941 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.941 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0bff7ca:::benchmark_data_smithi179_77138_object121170:head# 2024-02-09T17:27:02.941 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.942 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0c03bb8:::benchmark_data_smithi179_77138_object38969:head# 2024-02-09T17:27:02.942 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.943 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0c066e1:::benchmark_data_smithi179_77138_object74621:head# 2024-02-09T17:27:02.943 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.943 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0c0c6b6:::benchmark_data_smithi179_77138_object88098:head# 2024-02-09T17:27:02.943 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.943 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0c0da51:::benchmark_data_smithi179_77138_object54324:head# 2024-02-09T17:27:02.943 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.944 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0c195cd:::benchmark_data_smithi179_77138_object91459:head# 2024-02-09T17:27:02.944 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.944 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0c1d8af:::benchmark_data_smithi179_77138_object47203:head# 2024-02-09T17:27:02.944 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.944 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0c1e0d3:::benchmark_data_smithi179_77138_object126401:head# 2024-02-09T17:27:02.944 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.944 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0c225b3:::benchmark_data_smithi179_77138_object103660:head# 2024-02-09T17:27:02.944 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.944 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0c2ab71:::benchmark_data_smithi179_77138_object132523:head# 2024-02-09T17:27:02.944 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.946 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0c2cfd0:::benchmark_data_smithi179_77138_object58534:head# 2024-02-09T17:27:02.946 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.946 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0c2d73d:::benchmark_data_smithi179_77138_object118053:head# 2024-02-09T17:27:02.946 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.946 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0c31faf:::benchmark_data_smithi179_77138_object83666:head# 2024-02-09T17:27:02.946 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.947 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0c4f4ce:::benchmark_data_smithi179_77138_object36701:head# 2024-02-09T17:27:02.947 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.947 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0c64677:::benchmark_data_smithi179_77138_object125580:head# 2024-02-09T17:27:02.947 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.947 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0c71fd8:::benchmark_data_smithi179_77138_object90141:head# 2024-02-09T17:27:02.947 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.947 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0c7af21:::benchmark_data_smithi179_77138_object121097:head# 2024-02-09T17:27:02.947 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.947 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0c8b7ae:::benchmark_data_smithi179_77138_object30404:head# 2024-02-09T17:27:02.947 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.947 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0c9b9ed:::benchmark_data_smithi179_77138_object117220:head# 2024-02-09T17:27:02.947 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.947 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0ca3608:::benchmark_data_smithi179_77138_object129985:head# 2024-02-09T17:27:02.947 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.948 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0caed23:::benchmark_data_smithi179_77138_object62386:head# 2024-02-09T17:27:02.948 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.948 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0cb5d09:::benchmark_data_smithi179_77138_object102666:head# 2024-02-09T17:27:02.948 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.951 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0cb64fb:::benchmark_data_smithi179_77138_object54030:head# 2024-02-09T17:27:02.951 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.951 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0cbebbc:::benchmark_data_smithi179_77138_object24069:head# 2024-02-09T17:27:02.951 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.952 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0cc6096:::benchmark_data_smithi179_77138_object119408:head# 2024-02-09T17:27:02.952 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.952 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0cc66d3:::benchmark_data_smithi179_77138_object129583:head# 2024-02-09T17:27:02.952 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.952 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0cccfca:::benchmark_data_smithi179_77138_object32202:head# 2024-02-09T17:27:02.952 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.952 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0cd023c:::benchmark_data_smithi179_77138_object56116:head# 2024-02-09T17:27:02.952 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.952 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0cdd007:::benchmark_data_smithi179_77138_object76034:head# 2024-02-09T17:27:02.952 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.952 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0ce233c:::benchmark_data_smithi179_77138_object76693:head# 2024-02-09T17:27:02.952 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.952 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0d00258:::benchmark_data_smithi179_77138_object56061:head# 2024-02-09T17:27:02.952 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.952 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0d019f7:::benchmark_data_smithi179_77138_object45109:head# 2024-02-09T17:27:02.952 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.953 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0d0297e:::benchmark_data_smithi179_77138_object51354:head# 2024-02-09T17:27:02.953 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.953 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0d050f6:::benchmark_data_smithi179_77138_object73086:head# 2024-02-09T17:27:02.953 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.953 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0d0a39d:::benchmark_data_smithi179_77138_object34467:head# 2024-02-09T17:27:02.953 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.953 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0d0fe08:::benchmark_data_smithi179_77138_object49754:head# 2024-02-09T17:27:02.953 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.953 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0d11d78:::benchmark_data_smithi179_77138_object122939:head# 2024-02-09T17:27:02.953 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.953 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0d1930b:::benchmark_data_smithi179_77138_object83180:head# 2024-02-09T17:27:02.953 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.953 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0d1cdf3:::benchmark_data_smithi179_77138_object29512:head# 2024-02-09T17:27:02.953 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.954 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0d215a0:::benchmark_data_smithi179_77138_object69166:head# 2024-02-09T17:27:02.954 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.954 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0d23254:::benchmark_data_smithi179_77138_object57981:head# 2024-02-09T17:27:02.954 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.954 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0d2a470:::benchmark_data_smithi179_77138_object110245:head# 2024-02-09T17:27:02.954 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.957 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0d3f7fe:::benchmark_data_smithi179_77138_object26566:head# 2024-02-09T17:27:02.957 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.957 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0d437e4:::benchmark_data_smithi179_77138_object119026:head# 2024-02-09T17:27:02.957 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.957 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0d47e11:::benchmark_data_smithi179_77138_object61638:head# 2024-02-09T17:27:02.958 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.958 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0d482cc:::benchmark_data_smithi179_77138_object50999:head# 2024-02-09T17:27:02.958 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.958 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0d49a46:::benchmark_data_smithi179_77138_object35420:head# 2024-02-09T17:27:02.958 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.958 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0d4e1d3:::benchmark_data_smithi179_77138_object87836:head# 2024-02-09T17:27:02.958 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.958 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0d571d7:::benchmark_data_smithi179_77138_object129640:head# 2024-02-09T17:27:02.958 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.958 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0d5c8c2:::benchmark_data_smithi179_77138_object108600:head# 2024-02-09T17:27:02.958 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.958 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0d609b3:::benchmark_data_smithi179_77138_object41430:head# 2024-02-09T17:27:02.958 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.958 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0d67e17:::benchmark_data_smithi179_77138_object37580:head# 2024-02-09T17:27:02.958 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.959 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0d772cc:::benchmark_data_smithi179_77138_object71677:head# 2024-02-09T17:27:02.959 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.959 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0d8d312:::benchmark_data_smithi179_77138_object118913:head# 2024-02-09T17:27:02.959 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.959 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0d94627:::benchmark_data_smithi179_77138_object122948:head# 2024-02-09T17:27:02.959 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.959 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0da0ade:::benchmark_data_smithi179_77138_object71205:head# 2024-02-09T17:27:02.959 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.959 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0daca70:::benchmark_data_smithi179_77138_object110900:head# 2024-02-09T17:27:02.959 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.959 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0db3cfd:::benchmark_data_smithi179_77138_object79862:head# 2024-02-09T17:27:02.959 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.959 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0dba4d1:::benchmark_data_smithi179_77138_object61085:head# 2024-02-09T17:27:02.959 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.960 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0dbd0bd:::benchmark_data_smithi179_77138_object29460:head# 2024-02-09T17:27:02.960 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.960 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0dde5cd:::benchmark_data_smithi179_77138_object92006:head# 2024-02-09T17:27:02.960 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.960 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0de168a:::benchmark_data_smithi179_77138_object35326:head# 2024-02-09T17:27:02.960 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.960 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0dea766:::benchmark_data_smithi179_77138_object41201:head# 2024-02-09T17:27:02.960 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.960 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0decd56:::benchmark_data_smithi179_77138_object61987:head# 2024-02-09T17:27:02.960 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.967 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0defd32:::benchmark_data_smithi179_77138_object60832:head# 2024-02-09T17:27:02.967 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.967 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0df9086:::benchmark_data_smithi179_77138_object105794:head# 2024-02-09T17:27:02.967 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.967 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0e02fb2:::benchmark_data_smithi179_77138_object84832:head# 2024-02-09T17:27:02.967 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.967 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0e061cb:::benchmark_data_smithi179_77138_object88303:head# 2024-02-09T17:27:02.968 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.968 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0e204a4:::benchmark_data_smithi179_77138_object95895:head# 2024-02-09T17:27:02.968 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.968 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0e396fa:::benchmark_data_smithi179_77138_object42666:head# 2024-02-09T17:27:02.968 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.968 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0e43158:::benchmark_data_smithi179_77138_object130369:head# 2024-02-09T17:27:02.968 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.968 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0e47f12:::benchmark_data_smithi179_77138_object22917:head# 2024-02-09T17:27:02.968 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.968 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0e4c999:::benchmark_data_smithi179_77138_object109952:head# 2024-02-09T17:27:02.968 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.968 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0e5141e:::benchmark_data_smithi179_77138_object118088:head# 2024-02-09T17:27:02.968 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.969 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0e56353:::benchmark_data_smithi179_77138_object85368:head# 2024-02-09T17:27:02.969 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.969 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0e571ec:::benchmark_data_smithi179_77138_object106606:head# 2024-02-09T17:27:02.969 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.969 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0e5fa51:::benchmark_data_smithi179_77138_object97579:head# 2024-02-09T17:27:02.969 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.969 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0e60799:::benchmark_data_smithi179_77138_object99769:head# 2024-02-09T17:27:02.969 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.969 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0e621d9:::benchmark_data_smithi179_77138_object97431:head# 2024-02-09T17:27:02.969 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.969 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0e75e48:::benchmark_data_smithi179_77138_object43875:head# 2024-02-09T17:27:02.969 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.969 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0e7d45a:::benchmark_data_smithi179_77138_object64906:head# 2024-02-09T17:27:02.970 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.970 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0e847f2:::benchmark_data_smithi179_77138_object87545:head# 2024-02-09T17:27:02.970 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.970 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0e898da:::benchmark_data_smithi179_77138_object75988:head# 2024-02-09T17:27:02.970 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.970 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0e9cde2:::benchmark_data_smithi179_77138_object88496:head# 2024-02-09T17:27:02.970 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.970 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0ea18e5:::benchmark_data_smithi179_77138_object31099:head# 2024-02-09T17:27:02.970 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.970 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0ea1928:::benchmark_data_smithi179_77138_object42437:head# 2024-02-09T17:27:02.970 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.970 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0eb0b1a:::benchmark_data_smithi179_77138_object33948:head# 2024-02-09T17:27:02.970 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.970 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0ec1665:::benchmark_data_smithi179_77138_object109665:head# 2024-02-09T17:27:02.971 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.971 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0ec43e6:::benchmark_data_smithi179_77138_object129102:head# 2024-02-09T17:27:02.971 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.971 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0ec6f75:::benchmark_data_smithi179_77138_object61576:head# 2024-02-09T17:27:02.971 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.971 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0ec7314:::benchmark_data_smithi179_77138_object92079:head# 2024-02-09T17:27:02.971 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.971 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0ec88a0:::benchmark_data_smithi179_77138_object96660:head# 2024-02-09T17:27:02.971 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.971 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0ecf80b:::benchmark_data_smithi179_77138_object61504:head# 2024-02-09T17:27:02.971 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.971 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0ee906e:::benchmark_data_smithi179_77138_object63865:head# 2024-02-09T17:27:02.971 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.972 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0ef260c:::benchmark_data_smithi179_77138_object119254:head# 2024-02-09T17:27:02.972 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.972 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0ef389f:::benchmark_data_smithi179_77138_object126055:head# 2024-02-09T17:27:02.972 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.972 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0ef7e6c:::benchmark_data_smithi179_77138_object67920:head# 2024-02-09T17:27:02.972 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.972 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0efa560:::benchmark_data_smithi179_77138_object75102:head# 2024-02-09T17:27:02.972 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.972 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0f04d98:::benchmark_data_smithi179_77138_object30491:head# 2024-02-09T17:27:02.972 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.972 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0f05f2a:::benchmark_data_smithi179_77138_object93978:head# 2024-02-09T17:27:02.972 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.973 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0f06a48:::benchmark_data_smithi179_77138_object49990:head# 2024-02-09T17:27:02.973 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.973 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0f070dc:::benchmark_data_smithi179_77138_object130287:head# 2024-02-09T17:27:02.973 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.973 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0f0c45f:::benchmark_data_smithi179_77138_object78712:head# 2024-02-09T17:27:02.973 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.973 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0f17db8:::benchmark_data_smithi179_77138_object49932:head# 2024-02-09T17:27:02.973 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.973 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0f26857:::benchmark_data_smithi179_77138_object86363:head# 2024-02-09T17:27:02.973 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.973 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0f2a21e:::benchmark_data_smithi179_77138_object64087:head# 2024-02-09T17:27:02.973 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.973 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0f2fb8b:::benchmark_data_smithi179_77138_object55429:head# 2024-02-09T17:27:02.973 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.973 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0f3b67d:::benchmark_data_smithi179_77138_object133201:head# 2024-02-09T17:27:02.974 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.974 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0f3c99b:::benchmark_data_smithi179_77138_object125038:head# 2024-02-09T17:27:02.974 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.981 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0f3de5b:::benchmark_data_smithi179_77138_object66804:head# 2024-02-09T17:27:02.981 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.981 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0f445b9:::benchmark_data_smithi179_77138_object21410:head# 2024-02-09T17:27:02.981 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.981 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0f46ac8:::benchmark_data_smithi179_77138_object21844:head# 2024-02-09T17:27:02.981 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.981 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0f473bb:::benchmark_data_smithi179_77138_object62177:head# 2024-02-09T17:27:02.981 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.981 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0f61687:::benchmark_data_smithi179_77138_object104909:head# 2024-02-09T17:27:02.981 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.981 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0f6eb14:::benchmark_data_smithi179_77138_object80261:head# 2024-02-09T17:27:02.982 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.982 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0f74d46:::benchmark_data_smithi179_77138_object52086:head# 2024-02-09T17:27:02.982 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.982 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0f74ead:::benchmark_data_smithi179_77138_object106839:head# 2024-02-09T17:27:02.982 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.982 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0f9b048:::benchmark_data_smithi179_77138_object34885:head# 2024-02-09T17:27:02.982 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.982 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0fa1236:::benchmark_data_smithi179_77138_object23512:head# 2024-02-09T17:27:02.982 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.982 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0fa83e4:::benchmark_data_smithi179_77138_object43027:head# 2024-02-09T17:27:02.982 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.982 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0fb9330:::benchmark_data_smithi179_77138_object27182:head# 2024-02-09T17:27:02.982 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.982 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0fb971a:::benchmark_data_smithi179_77138_object35953:head# 2024-02-09T17:27:02.983 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.983 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0fc9be7:::benchmark_data_smithi179_77138_object65343:head# 2024-02-09T17:27:02.983 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.983 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0fcbc3d:::benchmark_data_smithi179_77138_object44963:head# 2024-02-09T17:27:02.983 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.983 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0fd0a0b:::benchmark_data_smithi179_77138_object57711:head# 2024-02-09T17:27:02.983 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.983 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0fd5c93:::benchmark_data_smithi179_77138_object47541:head# 2024-02-09T17:27:02.983 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.983 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0fd6ec6:::benchmark_data_smithi179_77138_object68438:head# 2024-02-09T17:27:02.983 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.983 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0fd7ba8:::benchmark_data_smithi179_77138_object95736:head# 2024-02-09T17:27:02.983 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.984 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f0ff6684:::benchmark_data_smithi179_77138_object47206:head# 2024-02-09T17:27:02.984 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.984 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1014a42:::benchmark_data_smithi179_77138_object104527:head# 2024-02-09T17:27:02.984 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.984 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1021e7a:::benchmark_data_smithi179_77138_object47224:head# 2024-02-09T17:27:02.984 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.984 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f102724c:::benchmark_data_smithi179_77138_object122775:head# 2024-02-09T17:27:02.984 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.984 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f103b077:::benchmark_data_smithi179_77138_object38481:head# 2024-02-09T17:27:02.984 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.984 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f103bdb3:::benchmark_data_smithi179_77138_object42345:head# 2024-02-09T17:27:02.984 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.984 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f104199d:::benchmark_data_smithi179_77138_object125138:head# 2024-02-09T17:27:02.984 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.985 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1046d2d:::benchmark_data_smithi179_77138_object133046:head# 2024-02-09T17:27:02.985 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.985 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f10681d5:::benchmark_data_smithi179_77138_object133352:head# 2024-02-09T17:27:02.985 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.985 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f106c32c:::benchmark_data_smithi179_77138_object117686:head# 2024-02-09T17:27:02.985 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.985 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1073111:::benchmark_data_smithi179_77138_object110646:head# 2024-02-09T17:27:02.985 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.985 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f107987b:::benchmark_data_smithi179_77138_object129226:head# 2024-02-09T17:27:02.985 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.985 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f107ef36:::benchmark_data_smithi179_77138_object112335:head# 2024-02-09T17:27:02.985 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.985 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1087b0c:::benchmark_data_smithi179_77138_object60300:head# 2024-02-09T17:27:02.985 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.986 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1091074:::benchmark_data_smithi179_77138_object102118:head# 2024-02-09T17:27:02.986 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.986 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f10968da:::benchmark_data_smithi179_77138_object125879:head# 2024-02-09T17:27:02.986 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.986 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f109a5ee:::benchmark_data_smithi179_77138_object120328:head# 2024-02-09T17:27:02.986 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.986 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f109d49b:::benchmark_data_smithi179_77138_object123337:head# 2024-02-09T17:27:02.986 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.986 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f109f007:::benchmark_data_smithi179_77138_object40374:head# 2024-02-09T17:27:02.986 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.986 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f10a9f10:::benchmark_data_smithi179_77138_object34395:head# 2024-02-09T17:27:02.986 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.986 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f10acd79:::benchmark_data_smithi179_77138_object82795:head# 2024-02-09T17:27:02.987 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.987 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f10b5a37:::benchmark_data_smithi179_77138_object55158:head# 2024-02-09T17:27:02.987 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.987 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f10c4a54:::benchmark_data_smithi179_77138_object49048:head# 2024-02-09T17:27:02.987 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.987 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f10c8a45:::benchmark_data_smithi179_77138_object73097:head# 2024-02-09T17:27:02.987 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.987 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f10d06cd:::benchmark_data_smithi179_77138_object109942:head# 2024-02-09T17:27:02.987 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.987 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f10db02b:::benchmark_data_smithi179_77138_object49751:head# 2024-02-09T17:27:02.987 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.987 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f10dee0b:::benchmark_data_smithi179_77138_object61082:head# 2024-02-09T17:27:02.987 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.987 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f10ee6a7:::benchmark_data_smithi179_77138_object82528:head# 2024-02-09T17:27:02.987 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.996 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f10ee6cc:::benchmark_data_smithi179_77138_object58766:head# 2024-02-09T17:27:02.997 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.997 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f10fbe0f:::benchmark_data_smithi179_77138_object131542:head# 2024-02-09T17:27:02.997 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.997 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1103341:::benchmark_data_smithi179_77138_object102116:head# 2024-02-09T17:27:02.997 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.997 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f110e4c9:::benchmark_data_smithi179_77138_object124833:head# 2024-02-09T17:27:02.997 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.997 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1134138:::benchmark_data_smithi179_77138_object122620:head# 2024-02-09T17:27:02.998 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.998 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f113e835:::benchmark_data_smithi179_77138_object59854:head# 2024-02-09T17:27:02.998 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.998 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1142227:::benchmark_data_smithi179_77138_object66882:head# 2024-02-09T17:27:02.998 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.998 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f114530f:::benchmark_data_smithi179_77138_object58311:head# 2024-02-09T17:27:02.998 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.998 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1145d5b:::benchmark_data_smithi179_77138_object102846:head# 2024-02-09T17:27:02.998 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.998 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f11549ae:::benchmark_data_smithi179_77138_object24680:head# 2024-02-09T17:27:02.999 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.999 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f115cb2a:::benchmark_data_smithi179_77138_object37770:head# 2024-02-09T17:27:02.999 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.999 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1178729:::benchmark_data_smithi179_77138_object43485:head# 2024-02-09T17:27:02.999 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.999 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1182dbb:::benchmark_data_smithi179_77138_object60679:head# 2024-02-09T17:27:02.999 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.999 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f11864f3:::benchmark_data_smithi179_77138_object81346:head# 2024-02-09T17:27:02.999 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:02.999 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1188c18:::benchmark_data_smithi179_77138_object88603:head# 2024-02-09T17:27:03.000 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.000 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f11912df:::benchmark_data_smithi179_77138_object121720:head# 2024-02-09T17:27:03.000 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.000 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f119ecf9:::benchmark_data_smithi179_77138_object91733:head# 2024-02-09T17:27:03.000 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.000 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f11a4434:::benchmark_data_smithi179_77138_object125551:head# 2024-02-09T17:27:03.000 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.000 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f11a9640:::benchmark_data_smithi179_77138_object53832:head# 2024-02-09T17:27:03.000 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.000 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f11b69a6:::benchmark_data_smithi179_77138_object131360:head# 2024-02-09T17:27:03.001 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.001 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f11b9f6c:::benchmark_data_smithi179_77138_object57381:head# 2024-02-09T17:27:03.001 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.001 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f11be021:::benchmark_data_smithi179_77138_object46817:head# 2024-02-09T17:27:03.001 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.001 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f11c67d4:::benchmark_data_smithi179_77138_object42816:head# 2024-02-09T17:27:03.001 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.001 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f11dc448:::benchmark_data_smithi179_77138_object113453:head# 2024-02-09T17:27:03.001 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.001 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f11e6851:::benchmark_data_smithi179_77138_object93799:head# 2024-02-09T17:27:03.001 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.001 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f11ed7c7:::benchmark_data_smithi179_77138_object28342:head# 2024-02-09T17:27:03.002 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.002 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1201adc:::benchmark_data_smithi179_77138_object41367:head# 2024-02-09T17:27:03.002 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.002 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f121e082:::benchmark_data_smithi179_77138_object124068:head# 2024-02-09T17:27:03.002 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.002 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1226caa:::benchmark_data_smithi179_77138_object19731:head# 2024-02-09T17:27:03.002 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.002 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f122ad75:::benchmark_data_smithi179_77138_object97085:head# 2024-02-09T17:27:03.002 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.002 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1233664:::benchmark_data_smithi179_77138_object20774:head# 2024-02-09T17:27:03.002 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.002 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1239d15:::benchmark_data_smithi179_77138_object129729:head# 2024-02-09T17:27:03.002 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.003 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1245a6d:::benchmark_data_smithi179_77138_object22042:head# 2024-02-09T17:27:03.003 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.003 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1251347:::benchmark_data_smithi179_77138_object130340:head# 2024-02-09T17:27:03.003 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.003 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1256e46:::benchmark_data_smithi179_77138_object33505:head# 2024-02-09T17:27:03.003 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.003 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f12572d5:::benchmark_data_smithi179_77138_object81076:head# 2024-02-09T17:27:03.003 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.003 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1257d36:::benchmark_data_smithi179_77138_object97164:head# 2024-02-09T17:27:03.003 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.003 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1258d44:::benchmark_data_smithi179_77138_object116606:head# 2024-02-09T17:27:03.003 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.003 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f12795f3:::benchmark_data_smithi179_77138_object56568:head# 2024-02-09T17:27:03.004 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.004 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f128d73c:::benchmark_data_smithi179_77138_object39584:head# 2024-02-09T17:27:03.004 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.004 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f12956d9:::benchmark_data_smithi179_77138_object128998:head# 2024-02-09T17:27:03.004 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.004 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1295ff4:::benchmark_data_smithi179_77138_object60877:head# 2024-02-09T17:27:03.004 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.004 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f129d4b3:::benchmark_data_smithi179_77138_object99319:head# 2024-02-09T17:27:03.004 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.004 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f12a8f8c:::benchmark_data_smithi179_77138_object116500:head# 2024-02-09T17:27:03.004 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.004 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f12aafba:::benchmark_data_smithi179_77138_object88446:head# 2024-02-09T17:27:03.004 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.004 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f12b22eb:::benchmark_data_smithi179_77138_object106566:head# 2024-02-09T17:27:03.005 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.005 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f12bec5a:::benchmark_data_smithi179_77138_object133249:head# 2024-02-09T17:27:03.005 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.005 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f12d708f:::benchmark_data_smithi179_77138_object58838:head# 2024-02-09T17:27:03.005 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.005 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f12e1f67:::benchmark_data_smithi179_77138_object33558:head# 2024-02-09T17:27:03.005 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.005 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f12e4520:::benchmark_data_smithi179_77138_object104890:head# 2024-02-09T17:27:03.005 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.005 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f12ea387:::benchmark_data_smithi179_77138_object97282:head# 2024-02-09T17:27:03.005 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.005 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f12f915f:::benchmark_data_smithi179_77138_object103722:head# 2024-02-09T17:27:03.005 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.005 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f12fd731:::benchmark_data_smithi179_77138_object46878:head# 2024-02-09T17:27:03.005 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.005 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f131689b:::benchmark_data_smithi179_77138_object130479:head# 2024-02-09T17:27:03.005 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.006 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1323487:::benchmark_data_smithi179_77138_object89374:head# 2024-02-09T17:27:03.006 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.006 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f13256e1:::benchmark_data_smithi179_77138_object92923:head# 2024-02-09T17:27:03.006 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.006 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f132bbdd:::benchmark_data_smithi179_77138_object101562:head# 2024-02-09T17:27:03.006 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.006 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f132cba2:::benchmark_data_smithi179_77138_object30810:head# 2024-02-09T17:27:03.006 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.006 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f132f755:::benchmark_data_smithi179_77138_object76793:head# 2024-02-09T17:27:03.006 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.006 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f133a3a7:::benchmark_data_smithi179_77138_object78566:head# 2024-02-09T17:27:03.006 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.006 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f133fa11:::benchmark_data_smithi179_77138_object113150:head# 2024-02-09T17:27:03.006 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.007 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1348276:::benchmark_data_smithi179_77138_object61025:head# 2024-02-09T17:27:03.007 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.007 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1353c6f:::benchmark_data_smithi179_77138_object52000:head# 2024-02-09T17:27:03.007 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.007 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1369688:::benchmark_data_smithi179_77138_object91779:head# 2024-02-09T17:27:03.007 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.007 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f136f1fb:::benchmark_data_smithi179_77138_object95260:head# 2024-02-09T17:27:03.007 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.007 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f13705b3:::benchmark_data_smithi179_77138_object30173:head# 2024-02-09T17:27:03.007 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.007 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f137ca79:::benchmark_data_smithi179_77138_object24619:head# 2024-02-09T17:27:03.007 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.013 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1383e33:::benchmark_data_smithi179_77138_object75609:head# 2024-02-09T17:27:03.013 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.013 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1388549:::benchmark_data_smithi179_77138_object47450:head# 2024-02-09T17:27:03.013 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.013 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f138ed9b:::benchmark_data_smithi179_77138_object76154:head# 2024-02-09T17:27:03.013 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.013 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f139587c:::benchmark_data_smithi179_77138_object83859:head# 2024-02-09T17:27:03.013 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.013 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1397218:::benchmark_data_smithi179_77138_object63068:head# 2024-02-09T17:27:03.013 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.013 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1397419:::benchmark_data_smithi179_77138_object63249:head# 2024-02-09T17:27:03.014 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.014 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f139c3e3:::benchmark_data_smithi179_77138_object111351:head# 2024-02-09T17:27:03.014 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.014 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f139c528:::benchmark_data_smithi179_77138_object36394:head# 2024-02-09T17:27:03.014 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.014 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f13a6b3f:::benchmark_data_smithi179_77138_object36399:head# 2024-02-09T17:27:03.014 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.014 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f13a7cf4:::benchmark_data_smithi179_77138_object40974:head# 2024-02-09T17:27:03.014 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.014 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f13b43b0:::benchmark_data_smithi179_77138_object53374:head# 2024-02-09T17:27:03.014 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.014 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f13b5508:::benchmark_data_smithi179_77138_object110946:head# 2024-02-09T17:27:03.014 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.015 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f13b951e:::benchmark_data_smithi179_77138_object45514:head# 2024-02-09T17:27:03.015 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.015 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f13c22c0:::benchmark_data_smithi179_77138_object83858:head# 2024-02-09T17:27:03.015 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.015 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f13dc1b6:::benchmark_data_smithi179_77138_object56547:head# 2024-02-09T17:27:03.015 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.015 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f13de626:::benchmark_data_smithi179_77138_object107036:head# 2024-02-09T17:27:03.015 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.015 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f13dea5b:::benchmark_data_smithi179_77138_object31244:head# 2024-02-09T17:27:03.015 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.015 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f13f6535:::benchmark_data_smithi179_77138_object109635:head# 2024-02-09T17:27:03.015 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.015 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1402a44:::benchmark_data_smithi179_77138_object45926:head# 2024-02-09T17:27:03.015 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.016 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f140a963:::benchmark_data_smithi179_77138_object38925:head# 2024-02-09T17:27:03.016 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.016 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f140ea14:::benchmark_data_smithi179_77138_object93626:head# 2024-02-09T17:27:03.016 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.016 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f140f710:::benchmark_data_smithi179_77138_object84752:head# 2024-02-09T17:27:03.016 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.016 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1410eab:::benchmark_data_smithi179_77138_object22015:head# 2024-02-09T17:27:03.016 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.016 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1428006:::benchmark_data_smithi179_77138_object41635:head# 2024-02-09T17:27:03.016 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.016 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f142d3a8:::benchmark_data_smithi179_77138_object124318:head# 2024-02-09T17:27:03.016 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.016 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1431c2d:::benchmark_data_smithi179_77138_object37832:head# 2024-02-09T17:27:03.017 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.017 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1431f61:::benchmark_data_smithi179_77138_object73135:head# 2024-02-09T17:27:03.017 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.017 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1443b37:::benchmark_data_smithi179_77138_object110806:head# 2024-02-09T17:27:03.017 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.017 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1444e30:::benchmark_data_smithi179_77138_object64715:head# 2024-02-09T17:27:03.017 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.017 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f145049a:::benchmark_data_smithi179_77138_object87201:head# 2024-02-09T17:27:03.017 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.017 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1451542:::benchmark_data_smithi179_77138_object43499:head# 2024-02-09T17:27:03.017 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.017 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f145412d:::benchmark_data_smithi179_77138_object124775:head# 2024-02-09T17:27:03.017 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.018 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1459465:::benchmark_data_smithi179_77138_object86178:head# 2024-02-09T17:27:03.018 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.018 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f14602e4:::benchmark_data_smithi179_77138_object130468:head# 2024-02-09T17:27:03.018 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.018 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1461d77:::benchmark_data_smithi179_77138_object127016:head# 2024-02-09T17:27:03.018 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.018 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f14756ed:::benchmark_data_smithi179_77138_object94951:head# 2024-02-09T17:27:03.018 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.018 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1479c3f:::benchmark_data_smithi179_77138_object130269:head# 2024-02-09T17:27:03.018 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.018 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f148293d:::benchmark_data_smithi179_77138_object76731:head# 2024-02-09T17:27:03.018 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.018 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1489e0a:::benchmark_data_smithi179_77138_object32604:head# 2024-02-09T17:27:03.019 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.019 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1489f83:::benchmark_data_smithi179_77138_object108341:head# 2024-02-09T17:27:03.019 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.019 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f149259e:::benchmark_data_smithi179_77138_object90792:head# 2024-02-09T17:27:03.019 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.019 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1493007:::benchmark_data_smithi179_77138_object116056:head# 2024-02-09T17:27:03.019 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.019 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f14a139a:::benchmark_data_smithi179_77138_object64004:head# 2024-02-09T17:27:03.019 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.019 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f14aba19:::benchmark_data_smithi179_77138_object45329:head# 2024-02-09T17:27:03.019 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.019 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f14b57ad:::benchmark_data_smithi179_77138_object76081:head# 2024-02-09T17:27:03.019 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.020 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f14ba16a:::benchmark_data_smithi179_77138_object49497:head# 2024-02-09T17:27:03.020 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.020 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f14bdf11:::benchmark_data_smithi179_77138_object58672:head# 2024-02-09T17:27:03.020 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.020 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f14cea78:::benchmark_data_smithi179_77138_object91240:head# 2024-02-09T17:27:03.020 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.029 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f14dcfd7:::benchmark_data_smithi179_77138_object83772:head# 2024-02-09T17:27:03.029 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.029 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f14e7d65:::benchmark_data_smithi179_77138_object35124:head# 2024-02-09T17:27:03.029 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.029 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f14ec763:::benchmark_data_smithi179_77138_object124572:head# 2024-02-09T17:27:03.029 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.030 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f14f1b82:::benchmark_data_smithi179_77138_object36943:head# 2024-02-09T17:27:03.030 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.030 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f151742a:::benchmark_data_smithi179_77138_object97596:head# 2024-02-09T17:27:03.030 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.030 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f152f3c6:::benchmark_data_smithi179_77138_object103846:head# 2024-02-09T17:27:03.030 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.030 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1533357:::benchmark_data_smithi179_77138_object133300:head# 2024-02-09T17:27:03.030 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.030 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f15339be:::benchmark_data_smithi179_77138_object37043:head# 2024-02-09T17:27:03.030 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.030 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f154d806:::benchmark_data_smithi179_77138_object83204:head# 2024-02-09T17:27:03.030 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.031 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f154e1bf:::benchmark_data_smithi179_77138_object67372:head# 2024-02-09T17:27:03.031 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.031 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1554da1:::benchmark_data_smithi179_77138_object113188:head# 2024-02-09T17:27:03.031 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.031 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f155f67e:::benchmark_data_smithi179_77138_object44815:head# 2024-02-09T17:27:03.031 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.031 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1563301:::benchmark_data_smithi179_77138_object46988:head# 2024-02-09T17:27:03.031 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.031 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f156b37a:::benchmark_data_smithi179_77138_object24386:head# 2024-02-09T17:27:03.031 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.031 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f15854e4:::benchmark_data_smithi179_77138_object60058:head# 2024-02-09T17:27:03.031 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.032 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1596ae8:::benchmark_data_smithi179_77138_object21512:head# 2024-02-09T17:27:03.032 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.032 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1597386:::benchmark_data_smithi179_77138_object100192:head# 2024-02-09T17:27:03.032 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.032 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f15a1432:::benchmark_data_smithi179_77138_object87954:head# 2024-02-09T17:27:03.032 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.032 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f15b68c3:::benchmark_data_smithi179_77138_object63013:head# 2024-02-09T17:27:03.032 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.032 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f15bcb59:::benchmark_data_smithi179_77138_object80418:head# 2024-02-09T17:27:03.032 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.032 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f15cbcf9:::benchmark_data_smithi179_77138_object63941:head# 2024-02-09T17:27:03.032 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.033 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f15d5d05:::benchmark_data_smithi179_77138_object70483:head# 2024-02-09T17:27:03.033 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.033 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f15e57e0:::benchmark_data_smithi179_77138_object77603:head# 2024-02-09T17:27:03.033 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.033 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f15e675b:::benchmark_data_smithi179_77138_object86767:head# 2024-02-09T17:27:03.033 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.033 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f15f3ca8:::benchmark_data_smithi179_77138_object25904:head# 2024-02-09T17:27:03.033 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.033 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f15f924b:::benchmark_data_smithi179_77138_object93134:head# 2024-02-09T17:27:03.033 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.033 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1604c12:::benchmark_data_smithi179_77138_object49209:head# 2024-02-09T17:27:03.033 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.034 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f160fc28:::benchmark_data_smithi179_77138_object88909:head# 2024-02-09T17:27:03.034 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.034 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f161026d:::benchmark_data_smithi179_77138_object131591:head# 2024-02-09T17:27:03.034 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.034 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f161462a:::benchmark_data_smithi179_77138_object29396:head# 2024-02-09T17:27:03.034 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.034 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f162f32e:::benchmark_data_smithi179_77138_object95422:head# 2024-02-09T17:27:03.034 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.034 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f164c3f2:::benchmark_data_smithi179_77138_object50717:head# 2024-02-09T17:27:03.034 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.034 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f165105f:::benchmark_data_smithi179_77138_object112139:head# 2024-02-09T17:27:03.034 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.035 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f165241c:::benchmark_data_smithi179_77138_object117971:head# 2024-02-09T17:27:03.035 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.035 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1653e54:::benchmark_data_smithi179_77138_object25320:head# 2024-02-09T17:27:03.035 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.035 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f16550f2:::benchmark_data_smithi179_77138_object29294:head# 2024-02-09T17:27:03.035 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.035 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f167386c:::benchmark_data_smithi179_77138_object82394:head# 2024-02-09T17:27:03.035 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.035 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f167b8bf:::benchmark_data_smithi179_77138_object123859:head# 2024-02-09T17:27:03.035 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.035 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f167d88e:::benchmark_data_smithi179_77138_object51383:head# 2024-02-09T17:27:03.036 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.036 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f16817a3:::benchmark_data_smithi179_77138_object75487:head# 2024-02-09T17:27:03.036 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.036 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f16898b2:::benchmark_data_smithi179_77138_object58432:head# 2024-02-09T17:27:03.036 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.036 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1693f58:::benchmark_data_smithi179_77138_object80457:head# 2024-02-09T17:27:03.036 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.036 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f169540d:::benchmark_data_smithi179_77138_object55995:head# 2024-02-09T17:27:03.036 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.036 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1697edc:::benchmark_data_smithi179_77138_object101308:head# 2024-02-09T17:27:03.036 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.037 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f169ae8b:::benchmark_data_smithi179_77138_object126163:head# 2024-02-09T17:27:03.037 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.037 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f16a2dd3:::benchmark_data_smithi179_77138_object30849:head# 2024-02-09T17:27:03.037 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.037 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f16a4cf1:::benchmark_data_smithi179_77138_object108863:head# 2024-02-09T17:27:03.037 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.037 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f16b2e69:::benchmark_data_smithi179_77138_object64391:head# 2024-02-09T17:27:03.037 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.037 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f16b9480:::benchmark_data_smithi179_77138_object116392:head# 2024-02-09T17:27:03.037 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.037 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f16c7a1c:::benchmark_data_smithi179_77138_object35278:head# 2024-02-09T17:27:03.037 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.037 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f16ce0af:::benchmark_data_smithi179_77138_object63170:head# 2024-02-09T17:27:03.037 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.037 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f16cfab3:::benchmark_data_smithi179_77138_object38841:head# 2024-02-09T17:27:03.038 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.038 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f16d82ff:::benchmark_data_smithi179_77138_object87075:head# 2024-02-09T17:27:03.038 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.038 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f16deeff:::benchmark_data_smithi179_77138_object53084:head# 2024-02-09T17:27:03.038 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.038 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f16e2ae6:::benchmark_data_smithi179_77138_object33847:head# 2024-02-09T17:27:03.038 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.038 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f16e6550:::benchmark_data_smithi179_77138_object119528:head# 2024-02-09T17:27:03.038 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.038 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f16ec19f:::benchmark_data_smithi179_77138_object44386:head# 2024-02-09T17:27:03.038 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.038 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f16f2324:::benchmark_data_smithi179_77138_object113741:head# 2024-02-09T17:27:03.038 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.038 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f16f388a:::benchmark_data_smithi179_77138_object23982:head# 2024-02-09T17:27:03.038 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.039 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f16feb53:::benchmark_data_smithi179_77138_object22389:head# 2024-02-09T17:27:03.039 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.039 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1703ee5:::benchmark_data_smithi179_77138_object52519:head# 2024-02-09T17:27:03.039 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.039 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f17100e9:::benchmark_data_smithi179_77138_object78834:head# 2024-02-09T17:27:03.039 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.039 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1717b7d:::benchmark_data_smithi179_77138_object66845:head# 2024-02-09T17:27:03.039 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.039 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f171dbd5:::benchmark_data_smithi179_77138_object125612:head# 2024-02-09T17:27:03.039 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.039 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f171e3db:::benchmark_data_smithi179_77138_object92843:head# 2024-02-09T17:27:03.039 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.039 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f17296ff:::benchmark_data_smithi179_77138_object121427:head# 2024-02-09T17:27:03.039 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.039 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f17474d9:::benchmark_data_smithi179_77138_object61372:head# 2024-02-09T17:27:03.039 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.039 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f174d063:::benchmark_data_smithi179_77138_object125095:head# 2024-02-09T17:27:03.040 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.040 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f175202d:::benchmark_data_smithi179_77138_object104721:head# 2024-02-09T17:27:03.040 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.050 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1752936:::benchmark_data_smithi179_77138_object48124:head# 2024-02-09T17:27:03.050 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.050 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1753ea8:::benchmark_data_smithi179_77138_object70789:head# 2024-02-09T17:27:03.050 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.051 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f177a410:::benchmark_data_smithi179_77138_object84151:head# 2024-02-09T17:27:03.051 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.051 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f177abf4:::benchmark_data_smithi179_77138_object75544:head# 2024-02-09T17:27:03.051 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.051 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f179407d:::benchmark_data_smithi179_77138_object20364:head# 2024-02-09T17:27:03.051 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.051 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f179dcfb:::benchmark_data_smithi179_77138_object126511:head# 2024-02-09T17:27:03.051 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.051 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f17abafb:::benchmark_data_smithi179_77138_object31512:head# 2024-02-09T17:27:03.051 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.051 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f17b2e4a:::benchmark_data_smithi179_77138_object130601:head# 2024-02-09T17:27:03.051 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.051 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f17dc295:::benchmark_data_smithi179_77138_object112961:head# 2024-02-09T17:27:03.051 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.052 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f17dec4f:::benchmark_data_smithi179_77138_object84490:head# 2024-02-09T17:27:03.052 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.052 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f17e0ff5:::benchmark_data_smithi179_77138_object121822:head# 2024-02-09T17:27:03.052 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.052 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f17e3356:::benchmark_data_smithi179_77138_object69048:head# 2024-02-09T17:27:03.052 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.052 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f17eedd6:::benchmark_data_smithi179_77138_object65423:head# 2024-02-09T17:27:03.052 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.052 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f17ef2d7:::benchmark_data_smithi179_77138_object124030:head# 2024-02-09T17:27:03.052 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.052 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f17f3967:::benchmark_data_smithi179_77138_object118645:head# 2024-02-09T17:27:03.052 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.052 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f17f69ea:::benchmark_data_smithi179_77138_object106975:head# 2024-02-09T17:27:03.053 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.053 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f17f6be4:::benchmark_data_smithi179_77138_object32278:head# 2024-02-09T17:27:03.053 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.053 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f17fab4b:::benchmark_data_smithi179_77138_object50227:head# 2024-02-09T17:27:03.053 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.053 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f18068c3:::benchmark_data_smithi179_77138_object31960:head# 2024-02-09T17:27:03.053 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.053 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f180d756:::benchmark_data_smithi179_77138_object105881:head# 2024-02-09T17:27:03.053 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.053 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f18155c7:::benchmark_data_smithi179_77138_object100530:head# 2024-02-09T17:27:03.053 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.053 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1815e06:::benchmark_data_smithi179_77138_object121794:head# 2024-02-09T17:27:03.053 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.054 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f181ff52:::benchmark_data_smithi179_77138_object77861:head# 2024-02-09T17:27:03.054 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.054 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1820a3e:::benchmark_data_smithi179_77138_object96435:head# 2024-02-09T17:27:03.054 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.054 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1827e80:::benchmark_data_smithi179_77138_object84799:head# 2024-02-09T17:27:03.054 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.054 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f18334b3:::benchmark_data_smithi179_77138_object41626:head# 2024-02-09T17:27:03.054 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.054 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f183d59c:::benchmark_data_smithi179_77138_object124852:head# 2024-02-09T17:27:03.054 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.054 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f18405ad:::benchmark_data_smithi179_77138_object109079:head# 2024-02-09T17:27:03.054 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.054 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1849c04:::benchmark_data_smithi179_77138_object90210:head# 2024-02-09T17:27:03.054 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.055 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f184a462:::benchmark_data_smithi179_77138_object26912:head# 2024-02-09T17:27:03.055 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.055 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f184dfa3:::benchmark_data_smithi179_77138_object46638:head# 2024-02-09T17:27:03.055 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.055 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f18581c0:::benchmark_data_smithi179_77138_object86275:head# 2024-02-09T17:27:03.055 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.055 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f185ac49:::benchmark_data_smithi179_77138_object112408:head# 2024-02-09T17:27:03.055 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.055 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f185e330:::benchmark_data_smithi179_77138_object60067:head# 2024-02-09T17:27:03.055 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.055 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f185ecd0:::benchmark_data_smithi179_77138_object37668:head# 2024-02-09T17:27:03.055 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.055 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1869380:::benchmark_data_smithi179_77138_object55943:head# 2024-02-09T17:27:03.056 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.056 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1872290:::benchmark_data_smithi179_77138_object93549:head# 2024-02-09T17:27:03.056 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.056 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f18771a5:::benchmark_data_smithi179_77138_object28321:head# 2024-02-09T17:27:03.056 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.056 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f187f7d9:::benchmark_data_smithi179_77138_object95493:head# 2024-02-09T17:27:03.056 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.056 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1881ce5:::benchmark_data_smithi179_77138_object110550:head# 2024-02-09T17:27:03.056 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.056 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f188eb0e:::benchmark_data_smithi179_77138_object47639:head# 2024-02-09T17:27:03.056 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.056 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f189c65d:::benchmark_data_smithi179_77138_object115005:head# 2024-02-09T17:27:03.057 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.057 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f189cabe:::benchmark_data_smithi179_77138_object39560:head# 2024-02-09T17:27:03.057 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.057 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f18a5231:::benchmark_data_smithi179_77138_object70414:head# 2024-02-09T17:27:03.057 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.057 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f18a73a6:::benchmark_data_smithi179_77138_object115789:head# 2024-02-09T17:27:03.057 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.057 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f18aec2b:::benchmark_data_smithi179_77138_object123786:head# 2024-02-09T17:27:03.057 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.057 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f18bb10e:::benchmark_data_smithi179_77138_object107039:head# 2024-02-09T17:27:03.057 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.057 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f18d0213:::benchmark_data_smithi179_77138_object127496:head# 2024-02-09T17:27:03.057 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.058 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f18d4bbb:::benchmark_data_smithi179_77138_object73577:head# 2024-02-09T17:27:03.058 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.058 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f18ec66f:::benchmark_data_smithi179_77138_object93107:head# 2024-02-09T17:27:03.058 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.058 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f18f07ec:::benchmark_data_smithi179_77138_object43162:head# 2024-02-09T17:27:03.058 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.058 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f18fb03f:::benchmark_data_smithi179_77138_object41909:head# 2024-02-09T17:27:03.058 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.058 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f190049b:::benchmark_data_smithi179_77138_object96712:head# 2024-02-09T17:27:03.058 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.058 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f19034bc:::benchmark_data_smithi179_77138_object54556:head# 2024-02-09T17:27:03.058 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.059 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1908c87:::benchmark_data_smithi179_77138_object106038:head# 2024-02-09T17:27:03.059 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.059 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1908cf0:::benchmark_data_smithi179_77138_object49617:head# 2024-02-09T17:27:03.059 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.059 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f190f1c4:::benchmark_data_smithi179_77138_object66116:head# 2024-02-09T17:27:03.059 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.059 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f19141df:::benchmark_data_smithi179_77138_object83656:head# 2024-02-09T17:27:03.059 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.059 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f191903b:::benchmark_data_smithi179_77138_object48166:head# 2024-02-09T17:27:03.059 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.059 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f193f379:::benchmark_data_smithi179_77138_object57612:head# 2024-02-09T17:27:03.059 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.059 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1947b91:::benchmark_data_smithi179_77138_object59107:head# 2024-02-09T17:27:03.060 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.060 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f194c98a:::benchmark_data_smithi179_77138_object40586:head# 2024-02-09T17:27:03.060 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.060 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1954709:::benchmark_data_smithi179_77138_object90914:head# 2024-02-09T17:27:03.060 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.060 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1962ce5:::benchmark_data_smithi179_77138_object59474:head# 2024-02-09T17:27:03.060 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.060 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f196b692:::benchmark_data_smithi179_77138_object115065:head# 2024-02-09T17:27:03.060 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.060 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f197b0c5:::benchmark_data_smithi179_77138_object55838:head# 2024-02-09T17:27:03.060 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.067 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f19866f1:::benchmark_data_smithi179_77138_object122887:head# 2024-02-09T17:27:03.068 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.068 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f19b6a21:::benchmark_data_smithi179_77138_object85488:head# 2024-02-09T17:27:03.068 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.068 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f19b6f8c:::benchmark_data_smithi179_77138_object95399:head# 2024-02-09T17:27:03.068 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.068 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f19bad25:::benchmark_data_smithi179_77138_object40830:head# 2024-02-09T17:27:03.068 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.068 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f19c0203:::benchmark_data_smithi179_77138_object48586:head# 2024-02-09T17:27:03.068 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.068 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f19c10b5:::benchmark_data_smithi179_77138_object84073:head# 2024-02-09T17:27:03.068 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.069 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f19c4565:::benchmark_data_smithi179_77138_object58281:head# 2024-02-09T17:27:03.069 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.069 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f19cdf58:::benchmark_data_smithi179_77138_object108858:head# 2024-02-09T17:27:03.069 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.069 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f19d5a2d:::benchmark_data_smithi179_77138_object109366:head# 2024-02-09T17:27:03.069 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.069 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f19da43c:::benchmark_data_smithi179_77138_object121147:head# 2024-02-09T17:27:03.069 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.069 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f19f2265:::benchmark_data_smithi179_77138_object67723:head# 2024-02-09T17:27:03.069 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.069 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1a054bc:::benchmark_data_smithi179_77138_object71170:head# 2024-02-09T17:27:03.069 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.069 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1a0ee6c:::benchmark_data_smithi179_77138_object84420:head# 2024-02-09T17:27:03.070 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.070 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1a12519:::benchmark_data_smithi179_77138_object53321:head# 2024-02-09T17:27:03.070 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.070 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1a12c87:::benchmark_data_smithi179_77138_object71877:head# 2024-02-09T17:27:03.070 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.070 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1a14cf6:::benchmark_data_smithi179_77138_object68979:head# 2024-02-09T17:27:03.070 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.070 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1a1f1bc:::benchmark_data_smithi179_77138_object133311:head# 2024-02-09T17:27:03.070 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.070 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1a2c762:::benchmark_data_smithi179_77138_object31866:head# 2024-02-09T17:27:03.070 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.070 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1a3c1fa:::benchmark_data_smithi179_77138_object93956:head# 2024-02-09T17:27:03.071 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.071 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1a49f32:::benchmark_data_smithi179_77138_object53462:head# 2024-02-09T17:27:03.071 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.071 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1a51cc0:::benchmark_data_smithi179_77138_object110254:head# 2024-02-09T17:27:03.071 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.071 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1a54677:::benchmark_data_smithi179_77138_object100445:head# 2024-02-09T17:27:03.071 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.071 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1a5a9c3:::benchmark_data_smithi179_77138_object39693:head# 2024-02-09T17:27:03.071 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.071 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1a606f6:::benchmark_data_smithi179_77138_object106501:head# 2024-02-09T17:27:03.071 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.071 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1a6c99f:::benchmark_data_smithi179_77138_object82996:head# 2024-02-09T17:27:03.071 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.071 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1a6d4de:::benchmark_data_smithi179_77138_object127894:head# 2024-02-09T17:27:03.071 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.072 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1a75f69:::benchmark_data_smithi179_77138_object127220:head# 2024-02-09T17:27:03.072 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.072 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1a76041:::benchmark_data_smithi179_77138_object104949:head# 2024-02-09T17:27:03.072 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.072 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1a7a28a:::benchmark_data_smithi179_77138_object71121:head# 2024-02-09T17:27:03.072 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.072 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1a7c53a:::benchmark_data_smithi179_77138_object48762:head# 2024-02-09T17:27:03.072 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.072 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1a7e2f4:::benchmark_data_smithi179_77138_object34241:head# 2024-02-09T17:27:03.072 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.072 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1a7eb3b:::benchmark_data_smithi179_77138_object25496:head# 2024-02-09T17:27:03.072 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.072 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1a80f2e:::benchmark_data_smithi179_77138_object62540:head# 2024-02-09T17:27:03.072 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.072 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1a8110f:::benchmark_data_smithi179_77138_object27909:head# 2024-02-09T17:27:03.073 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.073 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1a86b09:::benchmark_data_smithi179_77138_object96792:head# 2024-02-09T17:27:03.073 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.073 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1a8c30c:::benchmark_data_smithi179_77138_object118083:head# 2024-02-09T17:27:03.073 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.073 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1a984db:::benchmark_data_smithi179_77138_object47885:head# 2024-02-09T17:27:03.073 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.073 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1aae775:::benchmark_data_smithi179_77138_object124229:head# 2024-02-09T17:27:03.073 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.073 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1ab2103:::benchmark_data_smithi179_77138_object129483:head# 2024-02-09T17:27:03.073 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.073 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1ab4a37:::benchmark_data_smithi179_77138_object129037:head# 2024-02-09T17:27:03.073 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.074 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1ac2f25:::benchmark_data_smithi179_77138_object76501:head# 2024-02-09T17:27:03.074 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.074 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1ac4527:::benchmark_data_smithi179_77138_object113597:head# 2024-02-09T17:27:03.074 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.074 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1acf853:::benchmark_data_smithi179_77138_object58135:head# 2024-02-09T17:27:03.074 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.074 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1ad8865:::benchmark_data_smithi179_77138_object30487:head# 2024-02-09T17:27:03.074 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.074 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1adc04c:::benchmark_data_smithi179_77138_object28047:head# 2024-02-09T17:27:03.074 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.074 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1adfeb4:::benchmark_data_smithi179_77138_object111274:head# 2024-02-09T17:27:03.074 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.074 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1ae1c0e:::benchmark_data_smithi179_77138_object112564:head# 2024-02-09T17:27:03.074 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.074 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1ae44a7:::benchmark_data_smithi179_77138_object78363:head# 2024-02-09T17:27:03.074 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.075 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1ae8e7e:::benchmark_data_smithi179_77138_object113235:head# 2024-02-09T17:27:03.075 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.075 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1afb960:::benchmark_data_smithi179_77138_object77616:head# 2024-02-09T17:27:03.075 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.078 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1afc207:::benchmark_data_smithi179_77138_object126068:head# 2024-02-09T17:27:03.078 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.078 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1b12440:::benchmark_data_smithi179_77138_object65610:head# 2024-02-09T17:27:03.078 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.078 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1b146fc:::benchmark_data_smithi179_77138_object116517:head# 2024-02-09T17:27:03.078 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.078 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1b1d6e1:::benchmark_data_smithi179_77138_object83464:head# 2024-02-09T17:27:03.079 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.079 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1b2aa83:::benchmark_data_smithi179_77138_object133141:head# 2024-02-09T17:27:03.079 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.079 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1b46861:::benchmark_data_smithi179_77138_object57279:head# 2024-02-09T17:27:03.079 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.079 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1b57f2a:::benchmark_data_smithi179_77138_object54816:head# 2024-02-09T17:27:03.079 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.079 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1b5f078:::benchmark_data_smithi179_77138_object121123:head# 2024-02-09T17:27:03.079 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.079 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1b64cc8:::benchmark_data_smithi179_77138_object50729:head# 2024-02-09T17:27:03.079 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.079 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1b65ef3:::benchmark_data_smithi179_77138_object29758:head# 2024-02-09T17:27:03.079 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.079 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1b6bd7c:::benchmark_data_smithi179_77138_object118461:head# 2024-02-09T17:27:03.080 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.080 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1b6f5f8:::benchmark_data_smithi179_77138_object84394:head# 2024-02-09T17:27:03.080 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.080 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1b7ad61:::benchmark_data_smithi179_77138_object113050:head# 2024-02-09T17:27:03.080 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.080 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1b99ec7:::benchmark_data_smithi179_77138_object126672:head# 2024-02-09T17:27:03.080 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.080 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1ba0826:::benchmark_data_smithi179_77138_object58676:head# 2024-02-09T17:27:03.080 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.080 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1baaa23:::benchmark_data_smithi179_77138_object109227:head# 2024-02-09T17:27:03.080 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.080 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1badfcc:::benchmark_data_smithi179_77138_object44982:head# 2024-02-09T17:27:03.080 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.081 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1beb986:::benchmark_data_smithi179_77138_object32772:head# 2024-02-09T17:27:03.081 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.081 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1bef2a9:::benchmark_data_smithi179_77138_object51887:head# 2024-02-09T17:27:03.081 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.081 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1bfdc5e:::benchmark_data_smithi179_77138_object108952:head# 2024-02-09T17:27:03.081 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.081 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1bfebb5:::benchmark_data_smithi179_77138_object122926:head# 2024-02-09T17:27:03.081 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.081 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1c084a3:::benchmark_data_smithi179_77138_object23849:head# 2024-02-09T17:27:03.081 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.081 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1c11d57:::benchmark_data_smithi179_77138_object72927:head# 2024-02-09T17:27:03.081 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.081 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1c17389:::benchmark_data_smithi179_77138_object63153:head# 2024-02-09T17:27:03.081 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.082 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1c17865:::benchmark_data_smithi179_77138_object91398:head# 2024-02-09T17:27:03.082 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.082 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1c17eb1:::benchmark_data_smithi179_77138_object123815:head# 2024-02-09T17:27:03.082 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.082 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1c258a9:::benchmark_data_smithi179_77138_object104206:head# 2024-02-09T17:27:03.082 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.082 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1c30875:::benchmark_data_smithi179_77138_object89090:head# 2024-02-09T17:27:03.082 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.082 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1c34b1c:::benchmark_data_smithi179_77138_object76424:head# 2024-02-09T17:27:03.082 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.082 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1c4ac00:::benchmark_data_smithi179_77138_object91417:head# 2024-02-09T17:27:03.082 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.082 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1c57296:::benchmark_data_smithi179_77138_object112994:head# 2024-02-09T17:27:03.082 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.082 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1c5918a:::benchmark_data_smithi179_77138_object110395:head# 2024-02-09T17:27:03.083 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.083 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1c6ac88:::benchmark_data_smithi179_77138_object44361:head# 2024-02-09T17:27:03.083 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.083 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1c76d74:::benchmark_data_smithi179_77138_object53804:head# 2024-02-09T17:27:03.083 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.083 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1c8193b:::benchmark_data_smithi179_77138_object68616:head# 2024-02-09T17:27:03.083 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.084 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1c82e5e:::benchmark_data_smithi179_77138_object102239:head# 2024-02-09T17:27:03.085 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.085 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1c88b56:::benchmark_data_smithi179_77138_object97167:head# 2024-02-09T17:27:03.085 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.085 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1c8e04d:::benchmark_data_smithi179_77138_object121769:head# 2024-02-09T17:27:03.085 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.085 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1c8eab3:::benchmark_data_smithi179_77138_object40131:head# 2024-02-09T17:27:03.085 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.085 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1c9a670:::benchmark_data_smithi179_77138_object63620:head# 2024-02-09T17:27:03.085 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.085 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1c9dac5:::benchmark_data_smithi179_77138_object54155:head# 2024-02-09T17:27:03.085 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.086 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1c9e3bf:::benchmark_data_smithi179_77138_object107368:head# 2024-02-09T17:27:03.086 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.086 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1ca1e64:::benchmark_data_smithi179_77138_object42688:head# 2024-02-09T17:27:03.086 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.086 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1cb32ca:::benchmark_data_smithi179_77138_object128151:head# 2024-02-09T17:27:03.086 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.086 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1cb86eb:::benchmark_data_smithi179_77138_object34774:head# 2024-02-09T17:27:03.086 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.086 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1cbfbfc:::benchmark_data_smithi179_77138_object80149:head# 2024-02-09T17:27:03.086 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.086 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1cbff87:::benchmark_data_smithi179_77138_object81276:head# 2024-02-09T17:27:03.086 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.087 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1cc2c58:::benchmark_data_smithi179_77138_object55075:head# 2024-02-09T17:27:03.087 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.087 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1cc98f1:::benchmark_data_smithi179_77138_object101615:head# 2024-02-09T17:27:03.087 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.087 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1ccc5cd:::benchmark_data_smithi179_77138_object82044:head# 2024-02-09T17:27:03.087 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.092 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1cdd4cc:::benchmark_data_smithi179_77138_object117705:head# 2024-02-09T17:27:03.093 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.093 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1ce361b:::benchmark_data_smithi179_77138_object83938:head# 2024-02-09T17:27:03.093 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.093 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1ce4a0a:::benchmark_data_smithi179_77138_object103785:head# 2024-02-09T17:27:03.093 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.093 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1ce86fe:::benchmark_data_smithi179_77138_object26230:head# 2024-02-09T17:27:03.093 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.093 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1ce9210:::benchmark_data_smithi179_77138_object104382:head# 2024-02-09T17:27:03.093 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.093 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1cf6c71:::benchmark_data_smithi179_77138_object26384:head# 2024-02-09T17:27:03.094 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.094 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1d07393:::benchmark_data_smithi179_77138_object124912:head# 2024-02-09T17:27:03.094 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.094 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1d1ad82:::benchmark_data_smithi179_77138_object71711:head# 2024-02-09T17:27:03.094 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.094 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1d287dd:::benchmark_data_smithi179_77138_object104072:head# 2024-02-09T17:27:03.094 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.094 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1d2ceac:::benchmark_data_smithi179_77138_object53171:head# 2024-02-09T17:27:03.094 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.094 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1d419f6:::benchmark_data_smithi179_77138_object74101:head# 2024-02-09T17:27:03.094 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.094 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1d47f19:::benchmark_data_smithi179_77138_object45673:head# 2024-02-09T17:27:03.095 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.095 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1d5e941:::benchmark_data_smithi179_77138_object49175:head# 2024-02-09T17:27:03.095 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.095 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1d67ec1:::benchmark_data_smithi179_77138_object42812:head# 2024-02-09T17:27:03.095 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.095 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1d6f76b:::benchmark_data_smithi179_77138_object120376:head# 2024-02-09T17:27:03.095 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.095 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1d7247f:::benchmark_data_smithi179_77138_object77198:head# 2024-02-09T17:27:03.095 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.095 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1d7e584:::benchmark_data_smithi179_77138_object77177:head# 2024-02-09T17:27:03.095 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.095 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1d8ae54:::benchmark_data_smithi179_77138_object59837:head# 2024-02-09T17:27:03.095 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.096 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1d95e08:::benchmark_data_smithi179_77138_object113289:head# 2024-02-09T17:27:03.096 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.096 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1d98b5f:::benchmark_data_smithi179_77138_object107928:head# 2024-02-09T17:27:03.096 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.096 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1da2d08:::benchmark_data_smithi179_77138_object62475:head# 2024-02-09T17:27:03.096 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.096 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1dae9af:::benchmark_data_smithi179_77138_object80755:head# 2024-02-09T17:27:03.096 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.096 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1db7304:::benchmark_data_smithi179_77138_object51695:head# 2024-02-09T17:27:03.096 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.096 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1dde9a8:::benchmark_data_smithi179_77138_object77067:head# 2024-02-09T17:27:03.096 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.096 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1de0859:::benchmark_data_smithi179_77138_object38810:head# 2024-02-09T17:27:03.097 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.097 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1de722a:::benchmark_data_smithi179_77138_object38592:head# 2024-02-09T17:27:03.097 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.097 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1deb7b3:::benchmark_data_smithi179_77138_object100949:head# 2024-02-09T17:27:03.097 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.097 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1df23fd:::benchmark_data_smithi179_77138_object129153:head# 2024-02-09T17:27:03.097 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.097 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1df2667:::benchmark_data_smithi179_77138_object27872:head# 2024-02-09T17:27:03.097 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.097 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1dfa24b:::benchmark_data_smithi179_77138_object99401:head# 2024-02-09T17:27:03.097 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.107 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1dfa39f:::benchmark_data_smithi179_77138_object20583:head# 2024-02-09T17:27:03.107 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.107 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1dffe32:::benchmark_data_smithi179_77138_object69036:head# 2024-02-09T17:27:03.107 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.107 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1e04fb6:::benchmark_data_smithi179_77138_object65715:head# 2024-02-09T17:27:03.107 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.107 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1e087a1:::benchmark_data_smithi179_77138_object132635:head# 2024-02-09T17:27:03.107 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.107 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1e0dfb2:::benchmark_data_smithi179_77138_object74102:head# 2024-02-09T17:27:03.107 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.107 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1e15537:::benchmark_data_smithi179_77138_object75038:head# 2024-02-09T17:27:03.108 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.108 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1e1ec79:::benchmark_data_smithi179_77138_object92713:head# 2024-02-09T17:27:03.108 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.108 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1e3dc07:::benchmark_data_smithi179_77138_object82404:head# 2024-02-09T17:27:03.108 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.108 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1e4a30e:::benchmark_data_smithi179_77138_object27510:head# 2024-02-09T17:27:03.108 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.108 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1e526b1:::benchmark_data_smithi179_77138_object105185:head# 2024-02-09T17:27:03.108 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.108 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1e598db:::benchmark_data_smithi179_77138_object28187:head# 2024-02-09T17:27:03.108 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.108 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1e5e0a2:::benchmark_data_smithi179_77138_object112869:head# 2024-02-09T17:27:03.108 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.109 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1e640c0:::benchmark_data_smithi179_77138_object118691:head# 2024-02-09T17:27:03.109 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.109 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1e68731:::benchmark_data_smithi179_77138_object51819:head# 2024-02-09T17:27:03.109 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.109 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1e6c31d:::benchmark_data_smithi179_77138_object39520:head# 2024-02-09T17:27:03.109 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.109 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1e7a67a:::benchmark_data_smithi179_77138_object107415:head# 2024-02-09T17:27:03.109 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.109 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1e808a0:::benchmark_data_smithi179_77138_object115890:head# 2024-02-09T17:27:03.109 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.109 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1e8155d:::benchmark_data_smithi179_77138_object34773:head# 2024-02-09T17:27:03.109 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.109 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1e8e8cd:::benchmark_data_smithi179_77138_object127586:head# 2024-02-09T17:27:03.109 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.110 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1e9b09e:::benchmark_data_smithi179_77138_object36714:head# 2024-02-09T17:27:03.110 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.110 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1ea682b:::benchmark_data_smithi179_77138_object81154:head# 2024-02-09T17:27:03.110 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.110 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1eab609:::benchmark_data_smithi179_77138_object91017:head# 2024-02-09T17:27:03.110 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.110 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1eb4c61:::benchmark_data_smithi179_77138_object26077:head# 2024-02-09T17:27:03.110 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.110 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1ec8dcd:::benchmark_data_smithi179_77138_object82797:head# 2024-02-09T17:27:03.110 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.110 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1ed6181:::benchmark_data_smithi179_77138_object35533:head# 2024-02-09T17:27:03.110 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.111 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1eddeef:::benchmark_data_smithi179_77138_object80766:head# 2024-02-09T17:27:03.111 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.111 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1ef6a99:::benchmark_data_smithi179_77138_object71920:head# 2024-02-09T17:27:03.111 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.111 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1f064a8:::benchmark_data_smithi179_77138_object64410:head# 2024-02-09T17:27:03.111 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.111 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1f189f9:::benchmark_data_smithi179_77138_object38374:head# 2024-02-09T17:27:03.111 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.111 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1f23faa:::benchmark_data_smithi179_77138_object52801:head# 2024-02-09T17:27:03.111 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.111 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1f280ea:::benchmark_data_smithi179_77138_object124361:head# 2024-02-09T17:27:03.111 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.111 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1f33f1a:::benchmark_data_smithi179_77138_object37385:head# 2024-02-09T17:27:03.111 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.112 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1f3edec:::benchmark_data_smithi179_77138_object79015:head# 2024-02-09T17:27:03.112 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.112 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1f409de:::benchmark_data_smithi179_77138_object79504:head# 2024-02-09T17:27:03.112 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.112 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1f40a2e:::benchmark_data_smithi179_77138_object36491:head# 2024-02-09T17:27:03.112 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.112 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1f4209c:::benchmark_data_smithi179_77138_object83606:head# 2024-02-09T17:27:03.112 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.112 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1f44f19:::benchmark_data_smithi179_77138_object87738:head# 2024-02-09T17:27:03.112 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.112 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1f455a0:::benchmark_data_smithi179_77138_object93429:head# 2024-02-09T17:27:03.112 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.112 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1f640b6:::benchmark_data_smithi179_77138_object53337:head# 2024-02-09T17:27:03.113 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.113 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1f840d4:::benchmark_data_smithi179_77138_object71998:head# 2024-02-09T17:27:03.113 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.113 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1f90a84:::benchmark_data_smithi179_77138_object97008:head# 2024-02-09T17:27:03.113 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.113 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1f95189:::benchmark_data_smithi179_77138_object69334:head# 2024-02-09T17:27:03.113 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.113 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1fa9214:::benchmark_data_smithi179_77138_object74832:head# 2024-02-09T17:27:03.113 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.113 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1fabb80:::benchmark_data_smithi179_77138_object43909:head# 2024-02-09T17:27:03.113 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.113 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1fbdc27:::benchmark_data_smithi179_77138_object75550:head# 2024-02-09T17:27:03.113 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.113 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1fc7c5f:::benchmark_data_smithi179_77138_object78772:head# 2024-02-09T17:27:03.113 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.113 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1fcb422:::benchmark_data_smithi179_77138_object25888:head# 2024-02-09T17:27:03.114 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.114 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1fcbdf0:::benchmark_data_smithi179_77138_object67330:head# 2024-02-09T17:27:03.114 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.114 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1fcc769:::benchmark_data_smithi179_77138_object73961:head# 2024-02-09T17:27:03.114 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.114 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1fd14e9:::benchmark_data_smithi179_77138_object34702:head# 2024-02-09T17:27:03.114 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.114 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1fd26ff:::benchmark_data_smithi179_77138_object98363:head# 2024-02-09T17:27:03.114 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.114 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1fd6017:::benchmark_data_smithi179_77138_object54951:head# 2024-02-09T17:27:03.114 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.116 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1fed20e:::benchmark_data_smithi179_77138_object121522:head# 2024-02-09T17:27:03.116 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.116 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1fef110:::benchmark_data_smithi179_77138_object69350:head# 2024-02-09T17:27:03.116 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.116 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f1fefffd:::benchmark_data_smithi179_77138_object101412:head# 2024-02-09T17:27:03.116 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.116 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f200de02:::benchmark_data_smithi179_77138_object19325:head# 2024-02-09T17:27:03.116 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.116 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2035d4e:::benchmark_data_smithi179_77138_object43030:head# 2024-02-09T17:27:03.116 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.116 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f203d6f1:::benchmark_data_smithi179_77138_object34380:head# 2024-02-09T17:27:03.117 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.117 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f204180d:::benchmark_data_smithi179_77138_object100390:head# 2024-02-09T17:27:03.117 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.117 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2041c3b:::benchmark_data_smithi179_77138_object78621:head# 2024-02-09T17:27:03.117 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.117 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2043c25:::benchmark_data_smithi179_77138_object42833:head# 2024-02-09T17:27:03.117 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.117 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f20460a0:::benchmark_data_smithi179_77138_object126837:head# 2024-02-09T17:27:03.117 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.117 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2046f60:::benchmark_data_smithi179_77138_object91813:head# 2024-02-09T17:27:03.117 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.117 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2047b16:::benchmark_data_smithi179_77138_object81689:head# 2024-02-09T17:27:03.117 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.117 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f204fcd5:::benchmark_data_smithi179_77138_object63247:head# 2024-02-09T17:27:03.118 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.118 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2057d55:::benchmark_data_smithi179_77138_object28421:head# 2024-02-09T17:27:03.118 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.118 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f205c12b:::benchmark_data_smithi179_77138_object88003:head# 2024-02-09T17:27:03.118 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.118 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f20657e3:::benchmark_data_smithi179_77138_object28448:head# 2024-02-09T17:27:03.118 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.118 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f20684e4:::benchmark_data_smithi179_77138_object97950:head# 2024-02-09T17:27:03.118 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.118 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2079957:::benchmark_data_smithi179_77138_object73729:head# 2024-02-09T17:27:03.118 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.118 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2087989:::benchmark_data_smithi179_77138_object48834:head# 2024-02-09T17:27:03.118 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.118 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f208a601:::benchmark_data_smithi179_77138_object34838:head# 2024-02-09T17:27:03.119 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.119 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f208c432:::benchmark_data_smithi179_77138_object67872:head# 2024-02-09T17:27:03.119 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.119 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2091b6d:::benchmark_data_smithi179_77138_object78159:head# 2024-02-09T17:27:03.119 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.119 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2092363:::benchmark_data_smithi179_77138_object113426:head# 2024-02-09T17:27:03.119 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.119 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f20b5b29:::benchmark_data_smithi179_77138_object25584:head# 2024-02-09T17:27:03.119 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.119 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f20bcc1e:::benchmark_data_smithi179_77138_object20382:head# 2024-02-09T17:27:03.119 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.119 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f20c2393:::benchmark_data_smithi179_77138_object114421:head# 2024-02-09T17:27:03.119 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.120 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f20d91f4:::benchmark_data_smithi179_77138_object117025:head# 2024-02-09T17:27:03.120 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.120 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f20e7bad:::benchmark_data_smithi179_77138_object29016:head# 2024-02-09T17:27:03.120 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.120 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f20f2c64:::benchmark_data_smithi179_77138_object114600:head# 2024-02-09T17:27:03.120 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.120 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2101c30:::benchmark_data_smithi179_77138_object88459:head# 2024-02-09T17:27:03.120 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.124 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f21078d2:::benchmark_data_smithi179_77138_object122127:head# 2024-02-09T17:27:03.124 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.125 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f21209e8:::benchmark_data_smithi179_77138_object41123:head# 2024-02-09T17:27:03.125 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.125 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f212e17a:::benchmark_data_smithi179_77138_object132462:head# 2024-02-09T17:27:03.125 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.125 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f212fa22:::benchmark_data_smithi179_77138_object99451:head# 2024-02-09T17:27:03.125 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.128 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f21312ae:::benchmark_data_smithi179_77138_object67745:head# 2024-02-09T17:27:03.128 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.128 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f214745b:::benchmark_data_smithi179_77138_object45697:head# 2024-02-09T17:27:03.128 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.128 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2147cdf:::benchmark_data_smithi179_77138_object32750:head# 2024-02-09T17:27:03.129 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.129 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f214bed1:::benchmark_data_smithi179_77138_object113949:head# 2024-02-09T17:27:03.129 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.129 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2153f2a:::benchmark_data_smithi179_77138_object19856:head# 2024-02-09T17:27:03.129 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.129 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f21545a2:::benchmark_data_smithi179_77138_object25748:head# 2024-02-09T17:27:03.129 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.129 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f21742aa:::benchmark_data_smithi179_77138_object105458:head# 2024-02-09T17:27:03.129 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.129 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2174ad0:::benchmark_data_smithi179_77138_object128939:head# 2024-02-09T17:27:03.129 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.129 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f217fdcc:::benchmark_data_smithi179_77138_object129691:head# 2024-02-09T17:27:03.129 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.130 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f217ff45:::benchmark_data_smithi179_77138_object42405:head# 2024-02-09T17:27:03.130 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.130 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f218a8e0:::benchmark_data_smithi179_77138_object69034:head# 2024-02-09T17:27:03.130 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.130 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f218f974:::benchmark_data_smithi179_77138_object121368:head# 2024-02-09T17:27:03.130 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.132 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2190357:::benchmark_data_smithi179_77138_object43212:head# 2024-02-09T17:27:03.132 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.132 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f21914b9:::benchmark_data_smithi179_77138_object69834:head# 2024-02-09T17:27:03.132 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.132 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f219c72b:::benchmark_data_smithi179_77138_object95515:head# 2024-02-09T17:27:03.132 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.132 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f219cf4e:::benchmark_data_smithi179_77138_object38630:head# 2024-02-09T17:27:03.132 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.132 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f21b3831:::benchmark_data_smithi179_77138_object131038:head# 2024-02-09T17:27:03.133 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.133 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f21c7ce1:::benchmark_data_smithi179_77138_object111918:head# 2024-02-09T17:27:03.133 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.133 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f21cd53b:::benchmark_data_smithi179_77138_object131902:head# 2024-02-09T17:27:03.133 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.133 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f21cd57a:::benchmark_data_smithi179_77138_object83147:head# 2024-02-09T17:27:03.133 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.133 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f21d1efe:::benchmark_data_smithi179_77138_object39849:head# 2024-02-09T17:27:03.133 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.133 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f21f35c4:::benchmark_data_smithi179_77138_object61148:head# 2024-02-09T17:27:03.133 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.133 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f21fb0cf:::benchmark_data_smithi179_77138_object131362:head# 2024-02-09T17:27:03.133 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.133 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f21fd2a9:::benchmark_data_smithi179_77138_object39134:head# 2024-02-09T17:27:03.134 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.134 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f220932d:::benchmark_data_smithi179_77138_object48217:head# 2024-02-09T17:27:03.134 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.134 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2213042:::benchmark_data_smithi179_77138_object37337:head# 2024-02-09T17:27:03.134 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.134 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2214223:::benchmark_data_smithi179_77138_object102472:head# 2024-02-09T17:27:03.134 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.134 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2214789:::benchmark_data_smithi179_77138_object120080:head# 2024-02-09T17:27:03.135 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.135 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2214bea:::benchmark_data_smithi179_77138_object59008:head# 2024-02-09T17:27:03.135 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.135 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f222d717:::benchmark_data_smithi179_77138_object86314:head# 2024-02-09T17:27:03.135 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.135 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2230dd1:::benchmark_data_smithi179_77138_object104320:head# 2024-02-09T17:27:03.135 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.136 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2231bac:::benchmark_data_smithi179_77138_object29578:head# 2024-02-09T17:27:03.136 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.136 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2239f83:::benchmark_data_smithi179_77138_object89802:head# 2024-02-09T17:27:03.136 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.136 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f223a0d8:::benchmark_data_smithi179_77138_object64537:head# 2024-02-09T17:27:03.136 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.136 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f224f76c:::benchmark_data_smithi179_77138_object52878:head# 2024-02-09T17:27:03.136 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.136 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2252e63:::benchmark_data_smithi179_77138_object123457:head# 2024-02-09T17:27:03.136 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.137 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2259735:::benchmark_data_smithi179_77138_object122398:head# 2024-02-09T17:27:03.137 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.137 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f225aad3:::benchmark_data_smithi179_77138_object90261:head# 2024-02-09T17:27:03.137 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.137 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f225e3b7:::benchmark_data_smithi179_77138_object47347:head# 2024-02-09T17:27:03.137 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.137 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f226513f:::benchmark_data_smithi179_77138_object86588:head# 2024-02-09T17:27:03.137 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.139 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f22710f8:::benchmark_data_smithi179_77138_object56335:head# 2024-02-09T17:27:03.139 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.139 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2279e11:::benchmark_data_smithi179_77138_object88681:head# 2024-02-09T17:27:03.139 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.139 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f227a41f:::benchmark_data_smithi179_77138_object31543:head# 2024-02-09T17:27:03.139 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.139 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f227c7c2:::benchmark_data_smithi179_77138_object102051:head# 2024-02-09T17:27:03.139 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.140 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f22957f9:::benchmark_data_smithi179_77138_object51720:head# 2024-02-09T17:27:03.140 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.140 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f229600e:::benchmark_data_smithi179_77138_object59225:head# 2024-02-09T17:27:03.140 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.140 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f229a76d:::benchmark_data_smithi179_77138_object68297:head# 2024-02-09T17:27:03.140 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.140 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f229b792:::benchmark_data_smithi179_77138_object98655:head# 2024-02-09T17:27:03.140 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.140 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f22a0e4c:::benchmark_data_smithi179_77138_object102684:head# 2024-02-09T17:27:03.140 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.140 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f22abfa6:::benchmark_data_smithi179_77138_object20298:head# 2024-02-09T17:27:03.140 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.140 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f22acce2:::benchmark_data_smithi179_77138_object68828:head# 2024-02-09T17:27:03.140 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.143 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f22b15b4:::benchmark_data_smithi179_77138_object77922:head# 2024-02-09T17:27:03.144 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.144 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f22b3c81:::benchmark_data_smithi179_77138_object51431:head# 2024-02-09T17:27:03.144 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.144 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f22d5a14:::benchmark_data_smithi179_77138_object55973:head# 2024-02-09T17:27:03.144 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.144 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f22daa59:::benchmark_data_smithi179_77138_object128050:head# 2024-02-09T17:27:03.144 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.144 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f22db8ea:::benchmark_data_smithi179_77138_object55041:head# 2024-02-09T17:27:03.144 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.145 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f22dce60:::benchmark_data_smithi179_77138_object129571:head# 2024-02-09T17:27:03.145 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.145 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f22dd9fa:::benchmark_data_smithi179_77138_object57773:head# 2024-02-09T17:27:03.145 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.146 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f22e2da4:::benchmark_data_smithi179_77138_object100733:head# 2024-02-09T17:27:03.146 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.146 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f22eaad3:::benchmark_data_smithi179_77138_object77844:head# 2024-02-09T17:27:03.146 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.146 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2301c3f:::benchmark_data_smithi179_77138_object74134:head# 2024-02-09T17:27:03.146 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.146 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f230694a:::benchmark_data_smithi179_77138_object72800:head# 2024-02-09T17:27:03.146 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.147 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f231e252:::benchmark_data_smithi179_77138_object42258:head# 2024-02-09T17:27:03.148 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.148 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2325e1f:::benchmark_data_smithi179_77138_object97126:head# 2024-02-09T17:27:03.148 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.148 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f23280cf:::benchmark_data_smithi179_77138_object29655:head# 2024-02-09T17:27:03.148 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.148 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f23359dd:::benchmark_data_smithi179_77138_object42035:head# 2024-02-09T17:27:03.148 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.148 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f233dfec:::benchmark_data_smithi179_77138_object108912:head# 2024-02-09T17:27:03.148 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.148 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2342aea:::benchmark_data_smithi179_77138_object111088:head# 2024-02-09T17:27:03.148 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.148 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f23476df:::benchmark_data_smithi179_77138_object116305:head# 2024-02-09T17:27:03.148 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.148 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2348a3f:::benchmark_data_smithi179_77138_object36993:head# 2024-02-09T17:27:03.149 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.149 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f234a835:::benchmark_data_smithi179_77138_object124111:head# 2024-02-09T17:27:03.149 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.151 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2350450:::benchmark_data_smithi179_77138_object63242:head# 2024-02-09T17:27:03.151 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.151 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2351bf9:::benchmark_data_smithi179_77138_object29325:head# 2024-02-09T17:27:03.151 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.151 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2354e61:::benchmark_data_smithi179_77138_object125204:head# 2024-02-09T17:27:03.151 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.151 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2384a57:::benchmark_data_smithi179_77138_object62573:head# 2024-02-09T17:27:03.151 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.151 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f238d7a8:::benchmark_data_smithi179_77138_object100921:head# 2024-02-09T17:27:03.152 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.152 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2396057:::benchmark_data_smithi179_77138_object88144:head# 2024-02-09T17:27:03.152 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.152 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f239752c:::benchmark_data_smithi179_77138_object125682:head# 2024-02-09T17:27:03.152 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.152 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f23978c4:::benchmark_data_smithi179_77138_object97854:head# 2024-02-09T17:27:03.152 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.152 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f239a9a8:::benchmark_data_smithi179_77138_object84191:head# 2024-02-09T17:27:03.152 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.152 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f23a62c4:::benchmark_data_smithi179_77138_object28271:head# 2024-02-09T17:27:03.152 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.152 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f23ba2db:::benchmark_data_smithi179_77138_object129935:head# 2024-02-09T17:27:03.152 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.152 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f23c31f8:::benchmark_data_smithi179_77138_object112957:head# 2024-02-09T17:27:03.153 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.153 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f23c336d:::benchmark_data_smithi179_77138_object107329:head# 2024-02-09T17:27:03.153 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.155 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f23ce421:::benchmark_data_smithi179_77138_object50861:head# 2024-02-09T17:27:03.155 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.155 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f23d4e69:::benchmark_data_smithi179_77138_object67637:head# 2024-02-09T17:27:03.155 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.155 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f23d6793:::benchmark_data_smithi179_77138_object120561:head# 2024-02-09T17:27:03.155 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.155 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f23e940e:::benchmark_data_smithi179_77138_object26729:head# 2024-02-09T17:27:03.155 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.155 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f23f2689:::benchmark_data_smithi179_77138_object19683:head# 2024-02-09T17:27:03.155 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.155 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f23f36eb:::benchmark_data_smithi179_77138_object95122:head# 2024-02-09T17:27:03.155 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.155 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f23fcda7:::benchmark_data_smithi179_77138_object39617:head# 2024-02-09T17:27:03.156 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.156 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f24010eb:::benchmark_data_smithi179_77138_object71668:head# 2024-02-09T17:27:03.156 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.156 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2402cad:::benchmark_data_smithi179_77138_object108082:head# 2024-02-09T17:27:03.156 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.156 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2403890:::benchmark_data_smithi179_77138_object56028:head# 2024-02-09T17:27:03.156 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.156 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2403b1b:::benchmark_data_smithi179_77138_object126154:head# 2024-02-09T17:27:03.156 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.156 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2405aa4:::benchmark_data_smithi179_77138_object26188:head# 2024-02-09T17:27:03.156 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.156 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f241ba96:::benchmark_data_smithi179_77138_object109957:head# 2024-02-09T17:27:03.156 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.156 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f242cb95:::benchmark_data_smithi179_77138_object47836:head# 2024-02-09T17:27:03.156 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.161 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2438995:::benchmark_data_smithi179_77138_object106822:head# 2024-02-09T17:27:03.161 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.161 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f244476f:::benchmark_data_smithi179_77138_object122265:head# 2024-02-09T17:27:03.161 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.161 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f244d817:::benchmark_data_smithi179_77138_object43052:head# 2024-02-09T17:27:03.161 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.161 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f24597cc:::benchmark_data_smithi179_77138_object30393:head# 2024-02-09T17:27:03.161 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.162 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f245caba:::benchmark_data_smithi179_77138_object111420:head# 2024-02-09T17:27:03.162 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.162 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f246190c:::benchmark_data_smithi179_77138_object72195:head# 2024-02-09T17:27:03.162 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.162 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2465709:::benchmark_data_smithi179_77138_object32678:head# 2024-02-09T17:27:03.162 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.162 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f247b326:::benchmark_data_smithi179_77138_object110204:head# 2024-02-09T17:27:03.162 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.162 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f247bfb3:::benchmark_data_smithi179_77138_object88685:head# 2024-02-09T17:27:03.162 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.162 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f247fbec:::benchmark_data_smithi179_77138_object66357:head# 2024-02-09T17:27:03.162 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.162 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2486e33:::benchmark_data_smithi179_77138_object33875:head# 2024-02-09T17:27:03.162 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.163 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f248d43a:::benchmark_data_smithi179_77138_object24943:head# 2024-02-09T17:27:03.163 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.163 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f249169d:::benchmark_data_smithi179_77138_object57656:head# 2024-02-09T17:27:03.163 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.163 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f24922a8:::benchmark_data_smithi179_77138_object60402:head# 2024-02-09T17:27:03.163 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.163 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f249c499:::benchmark_data_smithi179_77138_object122532:head# 2024-02-09T17:27:03.163 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.163 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f24c80b0:::benchmark_data_smithi179_77138_object54848:head# 2024-02-09T17:27:03.163 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.163 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f24d752a:::benchmark_data_smithi179_77138_object48447:head# 2024-02-09T17:27:03.163 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.163 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f24df8ba:::benchmark_data_smithi179_77138_object26201:head# 2024-02-09T17:27:03.163 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.164 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f24e194f:::benchmark_data_smithi179_77138_object93784:head# 2024-02-09T17:27:03.164 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.164 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f24e2d31:::benchmark_data_smithi179_77138_object82105:head# 2024-02-09T17:27:03.164 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.164 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f24e490e:::benchmark_data_smithi179_77138_object120518:head# 2024-02-09T17:27:03.164 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.164 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f24f3e14:::benchmark_data_smithi179_77138_object76958:head# 2024-02-09T17:27:03.164 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.164 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f24fadcc:::benchmark_data_smithi179_77138_object73958:head# 2024-02-09T17:27:03.164 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.164 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f24fda9b:::benchmark_data_smithi179_77138_object37496:head# 2024-02-09T17:27:03.164 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.164 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f24fdc5c:::benchmark_data_smithi179_77138_object100426:head# 2024-02-09T17:27:03.164 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.164 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f25082e4:::benchmark_data_smithi179_77138_object129468:head# 2024-02-09T17:27:03.165 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.165 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f250a2e5:::benchmark_data_smithi179_77138_object103497:head# 2024-02-09T17:27:03.165 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.168 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f251df92:::benchmark_data_smithi179_77138_object68585:head# 2024-02-09T17:27:03.168 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.168 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2524024:::benchmark_data_smithi179_77138_object39707:head# 2024-02-09T17:27:03.168 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.168 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f252e716:::benchmark_data_smithi179_77138_object111917:head# 2024-02-09T17:27:03.168 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.168 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f25561cb:::benchmark_data_smithi179_77138_object65911:head# 2024-02-09T17:27:03.168 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.168 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2556c0d:::benchmark_data_smithi179_77138_object37122:head# 2024-02-09T17:27:03.168 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.168 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f25604fb:::benchmark_data_smithi179_77138_object104966:head# 2024-02-09T17:27:03.168 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.168 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2570139:::benchmark_data_smithi179_77138_object28547:head# 2024-02-09T17:27:03.169 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.169 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f257da7f:::benchmark_data_smithi179_77138_object39612:head# 2024-02-09T17:27:03.169 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.169 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f259b56f:::benchmark_data_smithi179_77138_object82676:head# 2024-02-09T17:27:03.169 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.169 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f25a39af:::benchmark_data_smithi179_77138_object112181:head# 2024-02-09T17:27:03.169 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.169 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f25ae92b:::benchmark_data_smithi179_77138_object56607:head# 2024-02-09T17:27:03.169 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.169 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f25b18f4:::benchmark_data_smithi179_77138_object77932:head# 2024-02-09T17:27:03.169 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.169 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f25e52ca:::benchmark_data_smithi179_77138_object118101:head# 2024-02-09T17:27:03.169 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.169 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f25e541f:::benchmark_data_smithi179_77138_object121480:head# 2024-02-09T17:27:03.169 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.170 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f25f859c:::benchmark_data_smithi179_77138_object24042:head# 2024-02-09T17:27:03.170 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.170 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f25fecff:::benchmark_data_smithi179_77138_object76977:head# 2024-02-09T17:27:03.170 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.170 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f260474b:::benchmark_data_smithi179_77138_object59438:head# 2024-02-09T17:27:03.170 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.170 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2605f31:::benchmark_data_smithi179_77138_object41957:head# 2024-02-09T17:27:03.170 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.170 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2625b29:::benchmark_data_smithi179_77138_object88075:head# 2024-02-09T17:27:03.170 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.170 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f262e4fd:::benchmark_data_smithi179_77138_object100970:head# 2024-02-09T17:27:03.170 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.170 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f262ed37:::benchmark_data_smithi179_77138_object127372:head# 2024-02-09T17:27:03.170 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.171 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2630be6:::benchmark_data_smithi179_77138_object87251:head# 2024-02-09T17:27:03.171 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.171 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2647b76:::benchmark_data_smithi179_77138_object36070:head# 2024-02-09T17:27:03.171 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.171 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f26482af:::benchmark_data_smithi179_77138_object101000:head# 2024-02-09T17:27:03.171 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.172 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f264c651:::benchmark_data_smithi179_77138_object103197:head# 2024-02-09T17:27:03.173 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.173 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f265426f:::benchmark_data_smithi179_77138_object55434:head# 2024-02-09T17:27:03.173 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.173 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2654e22:::benchmark_data_smithi179_77138_object62411:head# 2024-02-09T17:27:03.173 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.173 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2660c6a:::benchmark_data_smithi179_77138_object71785:head# 2024-02-09T17:27:03.173 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.173 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f266cc24:::benchmark_data_smithi179_77138_object63001:head# 2024-02-09T17:27:03.173 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.173 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2676100:::benchmark_data_smithi179_77138_object57842:head# 2024-02-09T17:27:03.173 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.173 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2682c52:::benchmark_data_smithi179_77138_object32973:head# 2024-02-09T17:27:03.173 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.173 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f268480c:::benchmark_data_smithi179_77138_object102768:head# 2024-02-09T17:27:03.173 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.174 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f268d75b:::benchmark_data_smithi179_77138_object129326:head# 2024-02-09T17:27:03.174 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.179 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f268fa00:::benchmark_data_smithi179_77138_object59552:head# 2024-02-09T17:27:03.180 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.180 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f269c1c3:::benchmark_data_smithi179_77138_object127244:head# 2024-02-09T17:27:03.180 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.180 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f26a6547:::benchmark_data_smithi179_77138_object84598:head# 2024-02-09T17:27:03.180 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.180 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f26a840e:::benchmark_data_smithi179_77138_object80404:head# 2024-02-09T17:27:03.180 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.180 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f26c7e88:::benchmark_data_smithi179_77138_object25741:head# 2024-02-09T17:27:03.180 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.180 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f26cd11b:::benchmark_data_smithi179_77138_object126053:head# 2024-02-09T17:27:03.180 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.180 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f26d4de9:::benchmark_data_smithi179_77138_object46244:head# 2024-02-09T17:27:03.180 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.181 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f26e5091:::benchmark_data_smithi179_77138_object30714:head# 2024-02-09T17:27:03.181 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.181 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f26feadb:::benchmark_data_smithi179_77138_object113584:head# 2024-02-09T17:27:03.181 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.184 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2700c05:::benchmark_data_smithi179_77138_object118775:head# 2024-02-09T17:27:03.184 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.184 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f270b8a7:::benchmark_data_smithi179_77138_object46151:head# 2024-02-09T17:27:03.185 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.185 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f271b930:::benchmark_data_smithi179_77138_object32193:head# 2024-02-09T17:27:03.185 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.185 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2729c02:::benchmark_data_smithi179_77138_object72867:head# 2024-02-09T17:27:03.185 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.185 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f272afba:::benchmark_data_smithi179_77138_object59205:head# 2024-02-09T17:27:03.185 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.185 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f27321c6:::benchmark_data_smithi179_77138_object98944:head# 2024-02-09T17:27:03.185 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.185 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f273822d:::benchmark_data_smithi179_77138_object77109:head# 2024-02-09T17:27:03.185 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.185 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f273cd93:::benchmark_data_smithi179_77138_object36723:head# 2024-02-09T17:27:03.185 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.187 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2745dfc:::benchmark_data_smithi179_77138_object130651:head# 2024-02-09T17:27:03.187 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.187 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f275dade:::benchmark_data_smithi179_77138_object69943:head# 2024-02-09T17:27:03.187 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.187 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2775bca:::benchmark_data_smithi179_77138_object121968:head# 2024-02-09T17:27:03.188 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.188 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2777b3b:::benchmark_data_smithi179_77138_object68808:head# 2024-02-09T17:27:03.188 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.188 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2779694:::benchmark_data_smithi179_77138_object83093:head# 2024-02-09T17:27:03.188 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.188 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f277fa07:::benchmark_data_smithi179_77138_object100992:head# 2024-02-09T17:27:03.188 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.188 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2788c86:::benchmark_data_smithi179_77138_object35413:head# 2024-02-09T17:27:03.188 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.188 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f278b1eb:::benchmark_data_smithi179_77138_object56461:head# 2024-02-09T17:27:03.188 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.189 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2796c55:::benchmark_data_smithi179_77138_object107702:head# 2024-02-09T17:27:03.189 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.189 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2797e9e:::benchmark_data_smithi179_77138_object86791:head# 2024-02-09T17:27:03.189 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.189 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f27991a4:::benchmark_data_smithi179_77138_object75757:head# 2024-02-09T17:27:03.189 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.190 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f279a2c9:::benchmark_data_smithi179_77138_object22551:head# 2024-02-09T17:27:03.190 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.191 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f27a1d9b:::benchmark_data_smithi179_77138_object54236:head# 2024-02-09T17:27:03.191 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.191 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f27ab2a6:::benchmark_data_smithi179_77138_object111176:head# 2024-02-09T17:27:03.191 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.191 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f27bce93:::benchmark_data_smithi179_77138_object97576:head# 2024-02-09T17:27:03.191 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.191 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f27c09e0:::benchmark_data_smithi179_77138_object109633:head# 2024-02-09T17:27:03.191 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.191 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f27c1efa:::benchmark_data_smithi179_77138_object118453:head# 2024-02-09T17:27:03.191 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.191 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f27dd971:::benchmark_data_smithi179_77138_object98105:head# 2024-02-09T17:27:03.191 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.192 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f27e4067:::benchmark_data_smithi179_77138_object22360:head# 2024-02-09T17:27:03.192 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.192 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f27f0043:::benchmark_data_smithi179_77138_object80769:head# 2024-02-09T17:27:03.192 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.192 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f27f2721:::benchmark_data_smithi179_77138_object63827:head# 2024-02-09T17:27:03.192 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.192 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f280d277:::benchmark_data_smithi179_77138_object88453:head# 2024-02-09T17:27:03.192 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.192 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2815231:::benchmark_data_smithi179_77138_object41854:head# 2024-02-09T17:27:03.192 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.193 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f281ce87:::benchmark_data_smithi179_77138_object48350:head# 2024-02-09T17:27:03.193 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.193 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2829254:::benchmark_data_smithi179_77138_object23923:head# 2024-02-09T17:27:03.194 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.194 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2835600:::benchmark_data_smithi179_77138_object56192:head# 2024-02-09T17:27:03.194 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.194 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f28367bd:::benchmark_data_smithi179_77138_object56580:head# 2024-02-09T17:27:03.194 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.194 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2868f0b:::benchmark_data_smithi179_77138_object85848:head# 2024-02-09T17:27:03.194 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.194 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f28817c9:::benchmark_data_smithi179_77138_object106780:head# 2024-02-09T17:27:03.194 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.197 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2886b84:::benchmark_data_smithi179_77138_object36873:head# 2024-02-09T17:27:03.197 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.199 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f288ab5e:::benchmark_data_smithi179_77138_object80282:head# 2024-02-09T17:27:03.199 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.203 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2894ab5:::benchmark_data_smithi179_77138_object19724:head# 2024-02-09T17:27:03.203 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.204 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f289d0c6:::benchmark_data_smithi179_77138_object115086:head# 2024-02-09T17:27:03.204 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.205 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f289e97e:::benchmark_data_smithi179_77138_object118595:head# 2024-02-09T17:27:03.205 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.205 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f28a6ba0:::benchmark_data_smithi179_77138_object67220:head# 2024-02-09T17:27:03.205 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.205 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f28b58ae:::benchmark_data_smithi179_77138_object105506:head# 2024-02-09T17:27:03.205 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.206 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f28d0522:::benchmark_data_smithi179_77138_object130189:head# 2024-02-09T17:27:03.207 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.207 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f28e206e:::benchmark_data_smithi179_77138_object108868:head# 2024-02-09T17:27:03.207 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.207 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f28e88f9:::benchmark_data_smithi179_77138_object127289:head# 2024-02-09T17:27:03.207 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.207 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f28ef563:::benchmark_data_smithi179_77138_object92389:head# 2024-02-09T17:27:03.207 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.207 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f28f6885:::benchmark_data_smithi179_77138_object88526:head# 2024-02-09T17:27:03.207 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.207 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f290f6a0:::benchmark_data_smithi179_77138_object21754:head# 2024-02-09T17:27:03.207 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.208 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f291800a:::benchmark_data_smithi179_77138_object88500:head# 2024-02-09T17:27:03.208 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.209 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2947e91:::benchmark_data_smithi179_77138_object108026:head# 2024-02-09T17:27:03.209 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.210 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2969f4f:::benchmark_data_smithi179_77138_object52298:head# 2024-02-09T17:27:03.210 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.210 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f29821bd:::benchmark_data_smithi179_77138_object92405:head# 2024-02-09T17:27:03.210 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.210 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f298b847:::benchmark_data_smithi179_77138_object59645:head# 2024-02-09T17:27:03.210 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.210 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f298e80f:::benchmark_data_smithi179_77138_object35917:head# 2024-02-09T17:27:03.210 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.210 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2991935:::benchmark_data_smithi179_77138_object105268:head# 2024-02-09T17:27:03.210 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.210 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f29961f7:::benchmark_data_smithi179_77138_object68281:head# 2024-02-09T17:27:03.210 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.211 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f299ad71:::benchmark_data_smithi179_77138_object125815:head# 2024-02-09T17:27:03.211 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.211 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f299d9ce:::benchmark_data_smithi179_77138_object86089:head# 2024-02-09T17:27:03.212 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.212 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f29a9d1a:::benchmark_data_smithi179_77138_object47028:head# 2024-02-09T17:27:03.212 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.212 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f29c693b:::benchmark_data_smithi179_77138_object68089:head# 2024-02-09T17:27:03.212 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.215 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f29ea7ca:::benchmark_data_smithi179_77138_object125800:head# 2024-02-09T17:27:03.215 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.216 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2a06128:::benchmark_data_smithi179_77138_object102009:head# 2024-02-09T17:27:03.216 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.216 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2a0bdd3:::benchmark_data_smithi179_77138_object46566:head# 2024-02-09T17:27:03.216 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.217 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2a160d2:::benchmark_data_smithi179_77138_object88612:head# 2024-02-09T17:27:03.217 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.217 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2a1c1fb:::benchmark_data_smithi179_77138_object21363:head# 2024-02-09T17:27:03.217 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.217 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2a2fb92:::benchmark_data_smithi179_77138_object25183:head# 2024-02-09T17:27:03.217 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.217 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2a38220:::benchmark_data_smithi179_77138_object95502:head# 2024-02-09T17:27:03.217 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.221 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2a40f3a:::benchmark_data_smithi179_77138_object34352:head# 2024-02-09T17:27:03.221 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.222 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2a465b9:::benchmark_data_smithi179_77138_object111934:head# 2024-02-09T17:27:03.222 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.222 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2a6eabe:::benchmark_data_smithi179_77138_object57382:head# 2024-02-09T17:27:03.222 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.222 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2a755ce:::benchmark_data_smithi179_77138_object29158:head# 2024-02-09T17:27:03.222 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.222 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2a76161:::benchmark_data_smithi179_77138_object133244:head# 2024-02-09T17:27:03.222 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.222 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2a762bd:::benchmark_data_smithi179_77138_object99163:head# 2024-02-09T17:27:03.222 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.222 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2a7817d:::benchmark_data_smithi179_77138_object84588:head# 2024-02-09T17:27:03.222 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.222 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2a8d068:::benchmark_data_smithi179_77138_object121526:head# 2024-02-09T17:27:03.222 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.222 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2aa2c9c:::benchmark_data_smithi179_77138_object129392:head# 2024-02-09T17:27:03.223 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.223 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2aa38e7:::benchmark_data_smithi179_77138_object77047:head# 2024-02-09T17:27:03.223 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.223 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2aa4b07:::benchmark_data_smithi179_77138_object26810:head# 2024-02-09T17:27:03.223 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.223 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2ab8375:::benchmark_data_smithi179_77138_object104074:head# 2024-02-09T17:27:03.223 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.223 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2abebae:::benchmark_data_smithi179_77138_object96372:head# 2024-02-09T17:27:03.223 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.223 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2aca0e7:::benchmark_data_smithi179_77138_object103679:head# 2024-02-09T17:27:03.223 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.224 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2acc2e2:::benchmark_data_smithi179_77138_object101786:head# 2024-02-09T17:27:03.224 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.225 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2aceb8a:::benchmark_data_smithi179_77138_object106816:head# 2024-02-09T17:27:03.225 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.225 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2ad646a:::benchmark_data_smithi179_77138_object60342:head# 2024-02-09T17:27:03.225 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.225 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2ae4151:::benchmark_data_smithi179_77138_object116537:head# 2024-02-09T17:27:03.225 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.225 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2aedb28:::benchmark_data_smithi179_77138_object57642:head# 2024-02-09T17:27:03.225 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.225 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2aede1a:::benchmark_data_smithi179_77138_object71482:head# 2024-02-09T17:27:03.225 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.225 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2af2ba2:::benchmark_data_smithi179_77138_object84036:head# 2024-02-09T17:27:03.225 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.226 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2b0c790:::benchmark_data_smithi179_77138_object121844:head# 2024-02-09T17:27:03.226 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.226 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2b17b7c:::benchmark_data_smithi179_77138_object79916:head# 2024-02-09T17:27:03.226 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.227 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2b18b10:::benchmark_data_smithi179_77138_object101829:head# 2024-02-09T17:27:03.227 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.227 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2b2a6b1:::benchmark_data_smithi179_77138_object39899:head# 2024-02-09T17:27:03.227 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.227 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2b2c192:::benchmark_data_smithi179_77138_object80358:head# 2024-02-09T17:27:03.227 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.228 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2b3a15b:::benchmark_data_smithi179_77138_object26413:head# 2024-02-09T17:27:03.228 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.228 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2b3b997:::benchmark_data_smithi179_77138_object119567:head# 2024-02-09T17:27:03.228 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.228 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2b411c5:::benchmark_data_smithi179_77138_object39906:head# 2024-02-09T17:27:03.228 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.228 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2b42853:::benchmark_data_smithi179_77138_object89119:head# 2024-02-09T17:27:03.228 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.231 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2b43063:::benchmark_data_smithi179_77138_object22690:head# 2024-02-09T17:27:03.231 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.231 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2b44947:::benchmark_data_smithi179_77138_object95882:head# 2024-02-09T17:27:03.231 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.231 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2b4e77c:::benchmark_data_smithi179_77138_object22659:head# 2024-02-09T17:27:03.231 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.231 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2b585f9:::benchmark_data_smithi179_77138_object19925:head# 2024-02-09T17:27:03.231 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.231 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2b6bf09:::benchmark_data_smithi179_77138_object95438:head# 2024-02-09T17:27:03.231 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.232 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2b7e6af:::benchmark_data_smithi179_77138_object104443:head# 2024-02-09T17:27:03.232 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.232 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2b82e82:::benchmark_data_smithi179_77138_object20242:head# 2024-02-09T17:27:03.232 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.232 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2b9338f:::benchmark_data_smithi179_77138_object70936:head# 2024-02-09T17:27:03.232 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.232 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2b950b4:::benchmark_data_smithi179_77138_object97767:head# 2024-02-09T17:27:03.232 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.232 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2b9efe9:::benchmark_data_smithi179_77138_object67476:head# 2024-02-09T17:27:03.232 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.232 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2ba3859:::benchmark_data_smithi179_77138_object99607:head# 2024-02-09T17:27:03.232 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.234 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2baf0b0:::benchmark_data_smithi179_77138_object44973:head# 2024-02-09T17:27:03.234 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.234 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2bc9dfd:::benchmark_data_smithi179_77138_object92700:head# 2024-02-09T17:27:03.234 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.234 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2bdce48:::benchmark_data_smithi179_77138_object35661:head# 2024-02-09T17:27:03.234 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.235 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2be8cd0:::benchmark_data_smithi179_77138_object65426:head# 2024-02-09T17:27:03.235 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.235 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2bedc60:::benchmark_data_smithi179_77138_object38061:head# 2024-02-09T17:27:03.235 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.235 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2bf0593:::benchmark_data_smithi179_77138_object101863:head# 2024-02-09T17:27:03.235 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.235 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2bf44ac:::benchmark_data_smithi179_77138_object87209:head# 2024-02-09T17:27:03.235 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.235 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2c0b648:::benchmark_data_smithi179_77138_object81064:head# 2024-02-09T17:27:03.235 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.235 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2c23513:::benchmark_data_smithi179_77138_object20683:head# 2024-02-09T17:27:03.235 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.235 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2c2db95:::benchmark_data_smithi179_77138_object33762:head# 2024-02-09T17:27:03.235 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.235 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2c3d331:::benchmark_data_smithi179_77138_object27802:head# 2024-02-09T17:27:03.235 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.239 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2c44916:::benchmark_data_smithi179_77138_object113815:head# 2024-02-09T17:27:03.239 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.239 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2c5cfbe:::benchmark_data_smithi179_77138_object103199:head# 2024-02-09T17:27:03.239 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.239 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2c6962a:::benchmark_data_smithi179_77138_object111983:head# 2024-02-09T17:27:03.239 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.242 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2c6e7be:::benchmark_data_smithi179_77138_object85502:head# 2024-02-09T17:27:03.242 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.242 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2c8eb8f:::benchmark_data_smithi179_77138_object103795:head# 2024-02-09T17:27:03.242 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.243 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2c920df:::benchmark_data_smithi179_77138_object88725:head# 2024-02-09T17:27:03.243 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.251 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2c95836:::benchmark_data_smithi179_77138_object51830:head# 2024-02-09T17:27:03.251 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.251 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2cabb6a:::benchmark_data_smithi179_77138_object34242:head# 2024-02-09T17:27:03.251 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.251 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2cd3ca1:::benchmark_data_smithi179_77138_object101208:head# 2024-02-09T17:27:03.251 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.251 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2cdb678:::benchmark_data_smithi179_77138_object57154:head# 2024-02-09T17:27:03.251 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.251 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2cdbe16:::benchmark_data_smithi179_77138_object28480:head# 2024-02-09T17:27:03.251 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.251 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2cedf6b:::benchmark_data_smithi179_77138_object46753:head# 2024-02-09T17:27:03.251 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.251 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2d1f50b:::benchmark_data_smithi179_77138_object110313:head# 2024-02-09T17:27:03.252 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.252 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2d1f718:::benchmark_data_smithi179_77138_object111939:head# 2024-02-09T17:27:03.252 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.252 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2d21596:::benchmark_data_smithi179_77138_object98828:head# 2024-02-09T17:27:03.252 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.252 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2d24221:::benchmark_data_smithi179_77138_object80893:head# 2024-02-09T17:27:03.252 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.252 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2d2fbd6:::benchmark_data_smithi179_77138_object28121:head# 2024-02-09T17:27:03.252 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.252 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2d38fde:::benchmark_data_smithi179_77138_object42739:head# 2024-02-09T17:27:03.252 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.252 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2d48927:::benchmark_data_smithi179_77138_object106308:head# 2024-02-09T17:27:03.252 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.254 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2d55a6a:::benchmark_data_smithi179_77138_object116783:head# 2024-02-09T17:27:03.254 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.255 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2d5880f:::benchmark_data_smithi179_77138_object45689:head# 2024-02-09T17:27:03.255 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.255 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2d640d9:::benchmark_data_smithi179_77138_object39034:head# 2024-02-09T17:27:03.255 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.255 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2d67bf0:::benchmark_data_smithi179_77138_object70077:head# 2024-02-09T17:27:03.255 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.255 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2d6deaf:::benchmark_data_smithi179_77138_object87321:head# 2024-02-09T17:27:03.255 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.255 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2d71468:::benchmark_data_smithi179_77138_object37739:head# 2024-02-09T17:27:03.255 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.255 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2d759bc:::benchmark_data_smithi179_77138_object47307:head# 2024-02-09T17:27:03.255 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.255 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2d75a47:::benchmark_data_smithi179_77138_object98978:head# 2024-02-09T17:27:03.256 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.256 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2d80c5c:::benchmark_data_smithi179_77138_object58351:head# 2024-02-09T17:27:03.256 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.256 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2d817ce:::benchmark_data_smithi179_77138_object25137:head# 2024-02-09T17:27:03.256 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.256 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2d82b33:::benchmark_data_smithi179_77138_object81670:head# 2024-02-09T17:27:03.256 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.256 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2d8398c:::benchmark_data_smithi179_77138_object22723:head# 2024-02-09T17:27:03.256 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.256 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2d847e8:::benchmark_data_smithi179_77138_object87218:head# 2024-02-09T17:27:03.256 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.257 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2d8ac07:::benchmark_data_smithi179_77138_object24702:head# 2024-02-09T17:27:03.257 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.257 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2da6b73:::benchmark_data_smithi179_77138_object110980:head# 2024-02-09T17:27:03.257 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.258 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2dabbc2:::benchmark_data_smithi179_77138_object46149:head# 2024-02-09T17:27:03.258 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.258 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2db8d51:::benchmark_data_smithi179_77138_object117541:head# 2024-02-09T17:27:03.258 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.258 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2dd533b:::benchmark_data_smithi179_77138_object75409:head# 2024-02-09T17:27:03.258 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.258 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2defb03:::benchmark_data_smithi179_77138_object112632:head# 2024-02-09T17:27:03.258 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.258 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2df1afc:::benchmark_data_smithi179_77138_object80651:head# 2024-02-09T17:27:03.258 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.258 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2dfa403:::benchmark_data_smithi179_77138_object104528:head# 2024-02-09T17:27:03.258 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.259 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2dfbe96:::benchmark_data_smithi179_77138_object129231:head# 2024-02-09T17:27:03.259 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.259 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2e00c75:::benchmark_data_smithi179_77138_object82574:head# 2024-02-09T17:27:03.259 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.261 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2e0575f:::benchmark_data_smithi179_77138_object29286:head# 2024-02-09T17:27:03.261 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.261 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2e23624:::benchmark_data_smithi179_77138_object76868:head# 2024-02-09T17:27:03.261 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.261 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2e3c45f:::benchmark_data_smithi179_77138_object99416:head# 2024-02-09T17:27:03.261 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.261 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2e47202:::benchmark_data_smithi179_77138_object86802:head# 2024-02-09T17:27:03.262 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.262 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2e56900:::benchmark_data_smithi179_77138_object81863:head# 2024-02-09T17:27:03.262 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.262 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2e56de8:::benchmark_data_smithi179_77138_object43740:head# 2024-02-09T17:27:03.262 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.262 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2e592c3:::benchmark_data_smithi179_77138_object120796:head# 2024-02-09T17:27:03.262 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.264 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2e5d27b:::benchmark_data_smithi179_77138_object27369:head# 2024-02-09T17:27:03.264 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.264 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2e69032:::benchmark_data_smithi179_77138_object68078:head# 2024-02-09T17:27:03.264 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.264 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2e7afc9:::benchmark_data_smithi179_77138_object29245:head# 2024-02-09T17:27:03.264 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.264 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2e7db89:::benchmark_data_smithi179_77138_object67591:head# 2024-02-09T17:27:03.264 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.264 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2e7df35:::benchmark_data_smithi179_77138_object58618:head# 2024-02-09T17:27:03.264 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.264 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2e7f4cd:::benchmark_data_smithi179_77138_object121846:head# 2024-02-09T17:27:03.264 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.264 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2e7f4e1:::benchmark_data_smithi179_77138_object74798:head# 2024-02-09T17:27:03.264 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.264 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2e82c20:::benchmark_data_smithi179_77138_object130537:head# 2024-02-09T17:27:03.264 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.265 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2ea36a6:::benchmark_data_smithi179_77138_object56044:head# 2024-02-09T17:27:03.265 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.265 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2ea6ca6:::benchmark_data_smithi179_77138_object36370:head# 2024-02-09T17:27:03.265 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.265 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2eb06d9:::benchmark_data_smithi179_77138_object128044:head# 2024-02-09T17:27:03.265 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.266 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2ec394f:::benchmark_data_smithi179_77138_object25907:head# 2024-02-09T17:27:03.266 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.266 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2ec57a6:::benchmark_data_smithi179_77138_object70115:head# 2024-02-09T17:27:03.266 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.266 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2ec62b5:::benchmark_data_smithi179_77138_object41461:head# 2024-02-09T17:27:03.266 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.266 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2ecb184:::benchmark_data_smithi179_77138_object48938:head# 2024-02-09T17:27:03.266 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.267 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2eebafd:::benchmark_data_smithi179_77138_object77631:head# 2024-02-09T17:27:03.267 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.267 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2ef02f6:::benchmark_data_smithi179_77138_object131787:head# 2024-02-09T17:27:03.267 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.267 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2ef071e:::benchmark_data_smithi179_77138_object61298:head# 2024-02-09T17:27:03.267 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.267 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2efa1b3:::benchmark_data_smithi179_77138_object39407:head# 2024-02-09T17:27:03.267 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.269 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2efbc27:::benchmark_data_smithi179_77138_object88930:head# 2024-02-09T17:27:03.269 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.269 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2f03f83:::benchmark_data_smithi179_77138_object124749:head# 2024-02-09T17:27:03.269 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.269 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2f0ab47:::benchmark_data_smithi179_77138_object111905:head# 2024-02-09T17:27:03.269 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.269 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2f12d1c:::benchmark_data_smithi179_77138_object94573:head# 2024-02-09T17:27:03.269 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.269 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2f15ea7:::benchmark_data_smithi179_77138_object98026:head# 2024-02-09T17:27:03.269 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.270 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2f18d8c:::benchmark_data_smithi179_77138_object78233:head# 2024-02-09T17:27:03.270 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.270 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2f1f104:::benchmark_data_smithi179_77138_object94752:head# 2024-02-09T17:27:03.270 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.270 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2f208d0:::benchmark_data_smithi179_77138_object92326:head# 2024-02-09T17:27:03.270 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.270 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2f2dfab:::benchmark_data_smithi179_77138_object132448:head# 2024-02-09T17:27:03.270 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.270 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2f31255:::benchmark_data_smithi179_77138_object40623:head# 2024-02-09T17:27:03.270 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.270 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2f356c6:::benchmark_data_smithi179_77138_object48445:head# 2024-02-09T17:27:03.270 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.272 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2f35e6c:::benchmark_data_smithi179_77138_object101482:head# 2024-02-09T17:27:03.272 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.272 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2f3b7fe:::benchmark_data_smithi179_77138_object39632:head# 2024-02-09T17:27:03.272 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.272 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2f3d8bf:::benchmark_data_smithi179_77138_object78739:head# 2024-02-09T17:27:03.272 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.272 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2f3dbe7:::benchmark_data_smithi179_77138_object79848:head# 2024-02-09T17:27:03.272 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.272 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2f49560:::benchmark_data_smithi179_77138_object34202:head# 2024-02-09T17:27:03.272 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.272 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2f4b7f5:::benchmark_data_smithi179_77138_object105064:head# 2024-02-09T17:27:03.272 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.272 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2f5691e:::benchmark_data_smithi179_77138_object62500:head# 2024-02-09T17:27:03.273 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.273 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2f5757b:::benchmark_data_smithi179_77138_object105742:head# 2024-02-09T17:27:03.273 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.275 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2f771af:::benchmark_data_smithi179_77138_object132420:head# 2024-02-09T17:27:03.275 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.276 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2f7f40f:::benchmark_data_smithi179_77138_object51216:head# 2024-02-09T17:27:03.276 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.285 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2f83531:::benchmark_data_smithi179_77138_object90138:head# 2024-02-09T17:27:03.285 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.285 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2f850a6:::benchmark_data_smithi179_77138_object25699:head# 2024-02-09T17:27:03.286 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.286 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2f85ec3:::benchmark_data_smithi179_77138_object33924:head# 2024-02-09T17:27:03.286 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.286 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2f8ce76:::benchmark_data_smithi179_77138_object65362:head# 2024-02-09T17:27:03.286 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.286 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2f8d7f5:::benchmark_data_smithi179_77138_object30289:head# 2024-02-09T17:27:03.286 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.286 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2f8df81:::benchmark_data_smithi179_77138_object70401:head# 2024-02-09T17:27:03.286 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.286 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2f8f2e3:::benchmark_data_smithi179_77138_object42659:head# 2024-02-09T17:27:03.286 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.286 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2f9407f:::benchmark_data_smithi179_77138_object76607:head# 2024-02-09T17:27:03.286 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.286 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2faebee:::benchmark_data_smithi179_77138_object123424:head# 2024-02-09T17:27:03.287 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.287 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2fbff8b:::benchmark_data_smithi179_77138_object32713:head# 2024-02-09T17:27:03.287 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.287 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2fc398a:::benchmark_data_smithi179_77138_object82945:head# 2024-02-09T17:27:03.287 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.287 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2fc7cab:::benchmark_data_smithi179_77138_object131445:head# 2024-02-09T17:27:03.287 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.287 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2fccee7:::benchmark_data_smithi179_77138_object65366:head# 2024-02-09T17:27:03.287 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.289 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2fd0244:::benchmark_data_smithi179_77138_object49942:head# 2024-02-09T17:27:03.289 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.289 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2fd29f6:::benchmark_data_smithi179_77138_object58011:head# 2024-02-09T17:27:03.290 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.290 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2fd3661:::benchmark_data_smithi179_77138_object104556:head# 2024-02-09T17:27:03.290 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.290 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2fd545f:::benchmark_data_smithi179_77138_object25498:head# 2024-02-09T17:27:03.290 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.290 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2fe61d4:::benchmark_data_smithi179_77138_object54989:head# 2024-02-09T17:27:03.290 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.290 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2fe6636:::benchmark_data_smithi179_77138_object54712:head# 2024-02-09T17:27:03.290 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.290 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2fe6cd3:::benchmark_data_smithi179_77138_object22820:head# 2024-02-09T17:27:03.290 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.290 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2fe710b:::benchmark_data_smithi179_77138_object121652:head# 2024-02-09T17:27:03.290 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.290 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2fea768:::benchmark_data_smithi179_77138_object28106:head# 2024-02-09T17:27:03.290 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.291 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f2ffd8b5:::benchmark_data_smithi179_77138_object54657:head# 2024-02-09T17:27:03.291 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.291 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f300f8df:::benchmark_data_smithi179_77138_object105903:head# 2024-02-09T17:27:03.291 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.291 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f301041b:::benchmark_data_smithi179_77138_object122629:head# 2024-02-09T17:27:03.291 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.291 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f30307c7:::benchmark_data_smithi179_77138_object54145:head# 2024-02-09T17:27:03.291 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.291 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f303fae7:::benchmark_data_smithi179_77138_object45016:head# 2024-02-09T17:27:03.291 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.295 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f303fb31:::benchmark_data_smithi179_77138_object109702:head# 2024-02-09T17:27:03.295 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.295 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f30420d9:::benchmark_data_smithi179_77138_object42042:head# 2024-02-09T17:27:03.295 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.295 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f30460dc:::benchmark_data_smithi179_77138_object58106:head# 2024-02-09T17:27:03.295 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.295 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f304f635:::benchmark_data_smithi179_77138_object36278:head# 2024-02-09T17:27:03.295 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.295 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f30547d1:::benchmark_data_smithi179_77138_object91645:head# 2024-02-09T17:27:03.295 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.296 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f305a03c:::benchmark_data_smithi179_77138_object42986:head# 2024-02-09T17:27:03.296 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.296 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f305a492:::benchmark_data_smithi179_77138_object105431:head# 2024-02-09T17:27:03.296 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.296 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3066850:::benchmark_data_smithi179_77138_object118470:head# 2024-02-09T17:27:03.296 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.296 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f306c5f1:::benchmark_data_smithi179_77138_object97999:head# 2024-02-09T17:27:03.296 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.296 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3082904:::benchmark_data_smithi179_77138_object71733:head# 2024-02-09T17:27:03.296 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.296 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f308975b:::benchmark_data_smithi179_77138_object125942:head# 2024-02-09T17:27:03.296 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.296 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3089780:::benchmark_data_smithi179_77138_object39189:head# 2024-02-09T17:27:03.296 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.297 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f308b7f2:::benchmark_data_smithi179_77138_object85643:head# 2024-02-09T17:27:03.297 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.297 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f308d8b1:::benchmark_data_smithi179_77138_object20363:head# 2024-02-09T17:27:03.297 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.297 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3092e1f:::benchmark_data_smithi179_77138_object44651:head# 2024-02-09T17:27:03.297 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.297 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f30afeab:::benchmark_data_smithi179_77138_object56802:head# 2024-02-09T17:27:03.297 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.297 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f30b0aed:::benchmark_data_smithi179_77138_object96163:head# 2024-02-09T17:27:03.297 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.297 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f30b6f49:::benchmark_data_smithi179_77138_object120120:head# 2024-02-09T17:27:03.297 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.297 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f30bd1be:::benchmark_data_smithi179_77138_object41680:head# 2024-02-09T17:27:03.297 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.300 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f30d830c:::benchmark_data_smithi179_77138_object74849:head# 2024-02-09T17:27:03.301 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.301 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f30d9425:::benchmark_data_smithi179_77138_object52992:head# 2024-02-09T17:27:03.301 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.301 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f30e5ad3:::benchmark_data_smithi179_77138_object107105:head# 2024-02-09T17:27:03.301 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.301 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f30ee2d9:::benchmark_data_smithi179_77138_object126596:head# 2024-02-09T17:27:03.301 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.301 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f31000e4:::benchmark_data_smithi179_77138_object39447:head# 2024-02-09T17:27:03.301 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.301 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3105235:::benchmark_data_smithi179_77138_object131068:head# 2024-02-09T17:27:03.301 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.301 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f310a8e1:::benchmark_data_smithi179_77138_object64244:head# 2024-02-09T17:27:03.301 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.301 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f310ac8c:::benchmark_data_smithi179_77138_object74862:head# 2024-02-09T17:27:03.302 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.302 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f310b47d:::benchmark_data_smithi179_77138_object116072:head# 2024-02-09T17:27:03.302 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.302 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f311e803:::benchmark_data_smithi179_77138_object101876:head# 2024-02-09T17:27:03.302 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.302 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f312bd71:::benchmark_data_smithi179_77138_object55916:head# 2024-02-09T17:27:03.302 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.302 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3130c8f:::benchmark_data_smithi179_77138_object57109:head# 2024-02-09T17:27:03.302 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.302 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3135e0d:::benchmark_data_smithi179_77138_object114672:head# 2024-02-09T17:27:03.302 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.302 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3136325:::benchmark_data_smithi179_77138_object50953:head# 2024-02-09T17:27:03.302 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.303 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f313eb27:::benchmark_data_smithi179_77138_object54512:head# 2024-02-09T17:27:03.303 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.303 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3146ee0:::benchmark_data_smithi179_77138_object89692:head# 2024-02-09T17:27:03.303 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.303 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3156ef7:::benchmark_data_smithi179_77138_object108184:head# 2024-02-09T17:27:03.303 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.303 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f315b07a:::benchmark_data_smithi179_77138_object86235:head# 2024-02-09T17:27:03.303 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.303 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f31632de:::benchmark_data_smithi179_77138_object42115:head# 2024-02-09T17:27:03.303 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.303 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3168476:::benchmark_data_smithi179_77138_object86375:head# 2024-02-09T17:27:03.303 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.303 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f316c331:::benchmark_data_smithi179_77138_object37700:head# 2024-02-09T17:27:03.303 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.304 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f31709f4:::benchmark_data_smithi179_77138_object30595:head# 2024-02-09T17:27:03.304 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.305 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3172b2a:::benchmark_data_smithi179_77138_object77524:head# 2024-02-09T17:27:03.306 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.306 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3178446:::benchmark_data_smithi179_77138_object114822:head# 2024-02-09T17:27:03.306 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.306 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3179bf4:::benchmark_data_smithi179_77138_object34310:head# 2024-02-09T17:27:03.306 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.306 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f318d48e:::benchmark_data_smithi179_77138_object87556:head# 2024-02-09T17:27:03.306 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.306 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f318e74a:::benchmark_data_smithi179_77138_object83841:head# 2024-02-09T17:27:03.306 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.306 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f318ec76:::benchmark_data_smithi179_77138_object32560:head# 2024-02-09T17:27:03.306 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.306 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3195ef3:::benchmark_data_smithi179_77138_object92905:head# 2024-02-09T17:27:03.306 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.307 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f31a0e37:::benchmark_data_smithi179_77138_object83618:head# 2024-02-09T17:27:03.307 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.307 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f31becc1:::benchmark_data_smithi179_77138_object132083:head# 2024-02-09T17:27:03.307 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.307 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f31c36b1:::benchmark_data_smithi179_77138_object41358:head# 2024-02-09T17:27:03.307 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.307 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f31c886b:::benchmark_data_smithi179_77138_object23221:head# 2024-02-09T17:27:03.307 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.307 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f31cd040:::benchmark_data_smithi179_77138_object114116:head# 2024-02-09T17:27:03.307 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.307 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f31cf826:::benchmark_data_smithi179_77138_object108819:head# 2024-02-09T17:27:03.307 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.307 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f31defcf:::benchmark_data_smithi179_77138_object49723:head# 2024-02-09T17:27:03.307 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.307 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f31df0ae:::benchmark_data_smithi179_77138_object41073:head# 2024-02-09T17:27:03.308 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.308 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f31e13f1:::benchmark_data_smithi179_77138_object111532:head# 2024-02-09T17:27:03.308 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.308 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f31e8d0c:::benchmark_data_smithi179_77138_object88773:head# 2024-02-09T17:27:03.308 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.308 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f31efac6:::benchmark_data_smithi179_77138_object126861:head# 2024-02-09T17:27:03.308 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.308 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f31fb995:::benchmark_data_smithi179_77138_object22746:head# 2024-02-09T17:27:03.308 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.308 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f32118ed:::benchmark_data_smithi179_77138_object73285:head# 2024-02-09T17:27:03.309 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.309 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3213e39:::benchmark_data_smithi179_77138_object64202:head# 2024-02-09T17:27:03.309 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.309 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3219cea:::benchmark_data_smithi179_77138_object105731:head# 2024-02-09T17:27:03.309 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.309 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3226e01:::benchmark_data_smithi179_77138_object117485:head# 2024-02-09T17:27:03.309 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.309 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3229f71:::benchmark_data_smithi179_77138_object100760:head# 2024-02-09T17:27:03.309 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.310 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f322c0be:::benchmark_data_smithi179_77138_object85790:head# 2024-02-09T17:27:03.310 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.310 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3232f21:::benchmark_data_smithi179_77138_object96322:head# 2024-02-09T17:27:03.310 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.310 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f323fa5d:::benchmark_data_smithi179_77138_object34567:head# 2024-02-09T17:27:03.310 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.311 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3246308:::benchmark_data_smithi179_77138_object102496:head# 2024-02-09T17:27:03.311 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.311 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3247fdd:::benchmark_data_smithi179_77138_object95560:head# 2024-02-09T17:27:03.311 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.311 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3264e67:::benchmark_data_smithi179_77138_object127732:head# 2024-02-09T17:27:03.311 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.311 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3268b5d:::benchmark_data_smithi179_77138_object21428:head# 2024-02-09T17:27:03.311 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.311 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f326bf2b:::benchmark_data_smithi179_77138_object58129:head# 2024-02-09T17:27:03.311 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.311 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f327354d:::benchmark_data_smithi179_77138_object108870:head# 2024-02-09T17:27:03.311 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.311 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f327f5a2:::benchmark_data_smithi179_77138_object115348:head# 2024-02-09T17:27:03.311 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.312 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f32874d1:::benchmark_data_smithi179_77138_object26842:head# 2024-02-09T17:27:03.312 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.312 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3298468:::benchmark_data_smithi179_77138_object43245:head# 2024-02-09T17:27:03.312 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.317 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f32b97fa:::benchmark_data_smithi179_77138_object29842:head# 2024-02-09T17:27:03.317 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.317 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f32c360b:::benchmark_data_smithi179_77138_object110928:head# 2024-02-09T17:27:03.317 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.317 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f32c6b6d:::benchmark_data_smithi179_77138_object73815:head# 2024-02-09T17:27:03.317 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.317 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f32cd48a:::benchmark_data_smithi179_77138_object37184:head# 2024-02-09T17:27:03.317 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.318 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f32dc7b6:::benchmark_data_smithi179_77138_object19461:head# 2024-02-09T17:27:03.318 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.318 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f32e3dfe:::benchmark_data_smithi179_77138_object123735:head# 2024-02-09T17:27:03.318 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.318 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f32ec74b:::benchmark_data_smithi179_77138_object47817:head# 2024-02-09T17:27:03.318 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.318 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f32fdb34:::benchmark_data_smithi179_77138_object27403:head# 2024-02-09T17:27:03.318 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.318 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f32ff56e:::benchmark_data_smithi179_77138_object127036:head# 2024-02-09T17:27:03.318 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.318 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3304b9d:::benchmark_data_smithi179_77138_object82537:head# 2024-02-09T17:27:03.318 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.319 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3306348:::benchmark_data_smithi179_77138_object59212:head# 2024-02-09T17:27:03.319 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.319 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f331baba:::benchmark_data_smithi179_77138_object116585:head# 2024-02-09T17:27:03.319 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.319 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f331bb53:::benchmark_data_smithi179_77138_object129602:head# 2024-02-09T17:27:03.319 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.319 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f332053e:::benchmark_data_smithi179_77138_object106726:head# 2024-02-09T17:27:03.319 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.319 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3321236:::benchmark_data_smithi179_77138_object80258:head# 2024-02-09T17:27:03.319 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.319 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f33212a6:::benchmark_data_smithi179_77138_object48475:head# 2024-02-09T17:27:03.319 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.319 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f33251a4:::benchmark_data_smithi179_77138_object58934:head# 2024-02-09T17:27:03.320 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.320 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3326774:::benchmark_data_smithi179_77138_object99400:head# 2024-02-09T17:27:03.320 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.320 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3326d2d:::benchmark_data_smithi179_77138_object108629:head# 2024-02-09T17:27:03.320 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.322 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f332b62c:::benchmark_data_smithi179_77138_object105771:head# 2024-02-09T17:27:03.322 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.322 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3331fd0:::benchmark_data_smithi179_77138_object72921:head# 2024-02-09T17:27:03.322 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.322 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f333810f:::benchmark_data_smithi179_77138_object49147:head# 2024-02-09T17:27:03.322 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.322 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f334155e:::benchmark_data_smithi179_77138_object88997:head# 2024-02-09T17:27:03.322 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.322 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f334156b:::benchmark_data_smithi179_77138_object58035:head# 2024-02-09T17:27:03.322 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.323 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f33658dd:::benchmark_data_smithi179_77138_object126746:head# 2024-02-09T17:27:03.323 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.323 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f336b4cc:::benchmark_data_smithi179_77138_object101246:head# 2024-02-09T17:27:03.323 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.323 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f336f4ce:::benchmark_data_smithi179_77138_object77173:head# 2024-02-09T17:27:03.323 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.323 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f337169f:::benchmark_data_smithi179_77138_object131010:head# 2024-02-09T17:27:03.323 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.324 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3377cc3:::benchmark_data_smithi179_77138_object39218:head# 2024-02-09T17:27:03.325 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.325 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3379c6f:::benchmark_data_smithi179_77138_object49835:head# 2024-02-09T17:27:03.325 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.326 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3385ae1:::benchmark_data_smithi179_77138_object102203:head# 2024-02-09T17:27:03.326 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.326 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f33863f7:::benchmark_data_smithi179_77138_object73072:head# 2024-02-09T17:27:03.326 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.326 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3386a55:::benchmark_data_smithi179_77138_object56705:head# 2024-02-09T17:27:03.326 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.326 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3388ba3:::benchmark_data_smithi179_77138_object95379:head# 2024-02-09T17:27:03.327 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.327 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3396654:::benchmark_data_smithi179_77138_object105498:head# 2024-02-09T17:27:03.327 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.327 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f339ba62:::benchmark_data_smithi179_77138_object66100:head# 2024-02-09T17:27:03.327 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.329 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f33a0bf0:::benchmark_data_smithi179_77138_object75349:head# 2024-02-09T17:27:03.329 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.329 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f33a8dfe:::benchmark_data_smithi179_77138_object41335:head# 2024-02-09T17:27:03.329 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.329 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f33bafb3:::benchmark_data_smithi179_77138_object43171:head# 2024-02-09T17:27:03.329 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.329 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f33c38c3:::benchmark_data_smithi179_77138_object46498:head# 2024-02-09T17:27:03.329 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.329 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f33c5f5e:::benchmark_data_smithi179_77138_object45535:head# 2024-02-09T17:27:03.330 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.330 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f33c62ed:::benchmark_data_smithi179_77138_object39822:head# 2024-02-09T17:27:03.330 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.330 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f33ce50e:::benchmark_data_smithi179_77138_object25039:head# 2024-02-09T17:27:03.330 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.330 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f33dcbb9:::benchmark_data_smithi179_77138_object26756:head# 2024-02-09T17:27:03.330 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.330 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f33f3c04:::benchmark_data_smithi179_77138_object70747:head# 2024-02-09T17:27:03.330 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.330 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f33f4e31:::benchmark_data_smithi179_77138_object57995:head# 2024-02-09T17:27:03.330 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.330 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3400cbf:::benchmark_data_smithi179_77138_object106495:head# 2024-02-09T17:27:03.330 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.331 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3414891:::benchmark_data_smithi179_77138_object116143:head# 2024-02-09T17:27:03.331 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.334 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3416a6d:::benchmark_data_smithi179_77138_object106400:head# 2024-02-09T17:27:03.334 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.335 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f341c7b2:::benchmark_data_smithi179_77138_object72145:head# 2024-02-09T17:27:03.335 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.335 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f34218c7:::benchmark_data_smithi179_77138_object93379:head# 2024-02-09T17:27:03.335 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.335 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3423439:::benchmark_data_smithi179_77138_object22129:head# 2024-02-09T17:27:03.335 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.336 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3424d6e:::benchmark_data_smithi179_77138_object72947:head# 2024-02-09T17:27:03.336 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.337 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f342c6a3:::benchmark_data_smithi179_77138_object101814:head# 2024-02-09T17:27:03.337 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.337 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f342c6cc:::benchmark_data_smithi179_77138_object73702:head# 2024-02-09T17:27:03.337 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.337 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f343090d:::benchmark_data_smithi179_77138_object107211:head# 2024-02-09T17:27:03.337 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.337 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3436dec:::benchmark_data_smithi179_77138_object131876:head# 2024-02-09T17:27:03.337 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.338 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f343b686:::benchmark_data_smithi179_77138_object109509:head# 2024-02-09T17:27:03.338 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.339 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f344100c:::benchmark_data_smithi179_77138_object91929:head# 2024-02-09T17:27:03.339 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.339 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f34488d9:::benchmark_data_smithi179_77138_object60057:head# 2024-02-09T17:27:03.339 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.339 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3454f27:::benchmark_data_smithi179_77138_object22022:head# 2024-02-09T17:27:03.339 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.339 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3466866:::benchmark_data_smithi179_77138_object43189:head# 2024-02-09T17:27:03.339 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.339 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3469493:::benchmark_data_smithi179_77138_object116845:head# 2024-02-09T17:27:03.339 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.340 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3469b0c:::benchmark_data_smithi179_77138_object86439:head# 2024-02-09T17:27:03.341 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.341 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f34711b9:::benchmark_data_smithi179_77138_object29476:head# 2024-02-09T17:27:03.341 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.341 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3474e83:::benchmark_data_smithi179_77138_object58460:head# 2024-02-09T17:27:03.341 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.341 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f348a1fc:::benchmark_data_smithi179_77138_object110635:head# 2024-02-09T17:27:03.341 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.341 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f348aec2:::benchmark_data_smithi179_77138_object102329:head# 2024-02-09T17:27:03.341 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.341 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f349171d:::benchmark_data_smithi179_77138_object127796:head# 2024-02-09T17:27:03.341 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.341 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f349639b:::benchmark_data_smithi179_77138_object82291:head# 2024-02-09T17:27:03.341 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.342 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f34986dd:::benchmark_data_smithi179_77138_object100635:head# 2024-02-09T17:27:03.342 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.342 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f34a734d:::benchmark_data_smithi179_77138_object59494:head# 2024-02-09T17:27:03.342 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.342 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f34a88ef:::benchmark_data_smithi179_77138_object119802:head# 2024-02-09T17:27:03.342 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.343 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f34a9462:::benchmark_data_smithi179_77138_object114111:head# 2024-02-09T17:27:03.343 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.343 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f34bab0d:::benchmark_data_smithi179_77138_object95880:head# 2024-02-09T17:27:03.343 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.343 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f34bd9d8:::benchmark_data_smithi179_77138_object109352:head# 2024-02-09T17:27:03.343 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.344 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f34c4357:::benchmark_data_smithi179_77138_object64414:head# 2024-02-09T17:27:03.344 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.344 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f34c9abf:::benchmark_data_smithi179_77138_object40643:head# 2024-02-09T17:27:03.344 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.344 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f34ca501:::benchmark_data_smithi179_77138_object50691:head# 2024-02-09T17:27:03.344 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.344 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f34cf3d4:::benchmark_data_smithi179_77138_object22616:head# 2024-02-09T17:27:03.344 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.345 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f34dd992:::benchmark_data_smithi179_77138_object92686:head# 2024-02-09T17:27:03.346 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.346 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f34e8484:::benchmark_data_smithi179_77138_object51573:head# 2024-02-09T17:27:03.346 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.346 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f34f4a57:::benchmark_data_smithi179_77138_object45179:head# 2024-02-09T17:27:03.346 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.346 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f34fd41c:::benchmark_data_smithi179_77138_object25489:head# 2024-02-09T17:27:03.346 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.346 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3505a03:::benchmark_data_smithi179_77138_object120360:head# 2024-02-09T17:27:03.346 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.346 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3513ce2:::benchmark_data_smithi179_77138_object121776:head# 2024-02-09T17:27:03.346 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.346 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3515002:::benchmark_data_smithi179_77138_object69461:head# 2024-02-09T17:27:03.346 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.346 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3517297:::benchmark_data_smithi179_77138_object126187:head# 2024-02-09T17:27:03.347 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.347 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f35192cf:::benchmark_data_smithi179_77138_object72793:head# 2024-02-09T17:27:03.347 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.347 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3521115:::benchmark_data_smithi179_77138_object85060:head# 2024-02-09T17:27:03.347 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.347 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3523d22:::benchmark_data_smithi179_77138_object44841:head# 2024-02-09T17:27:03.348 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.351 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3527c71:::benchmark_data_smithi179_77138_object59820:head# 2024-02-09T17:27:03.351 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.352 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f352bdd4:::benchmark_data_smithi179_77138_object31771:head# 2024-02-09T17:27:03.353 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.353 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f354622c:::benchmark_data_smithi179_77138_object127988:head# 2024-02-09T17:27:03.353 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.353 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f354956b:::benchmark_data_smithi179_77138_object22260:head# 2024-02-09T17:27:03.353 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.353 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3551b53:::benchmark_data_smithi179_77138_object120444:head# 2024-02-09T17:27:03.353 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.354 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f355a41c:::benchmark_data_smithi179_77138_object22613:head# 2024-02-09T17:27:03.354 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.354 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f355e0f0:::benchmark_data_smithi179_77138_object126618:head# 2024-02-09T17:27:03.354 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.354 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f355f439:::benchmark_data_smithi179_77138_object51668:head# 2024-02-09T17:27:03.354 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.354 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3562dc9:::benchmark_data_smithi179_77138_object42083:head# 2024-02-09T17:27:03.354 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.355 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3563209:::benchmark_data_smithi179_77138_object47510:head# 2024-02-09T17:27:03.355 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.355 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f356824a:::benchmark_data_smithi179_77138_object130263:head# 2024-02-09T17:27:03.356 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.356 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f35701cf:::benchmark_data_smithi179_77138_object104282:head# 2024-02-09T17:27:03.356 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.356 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3570db2:::benchmark_data_smithi179_77138_object83066:head# 2024-02-09T17:27:03.356 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.356 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f357858e:::benchmark_data_smithi179_77138_object63440:head# 2024-02-09T17:27:03.356 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.358 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3583b23:::benchmark_data_smithi179_77138_object53491:head# 2024-02-09T17:27:03.358 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.359 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3596074:::benchmark_data_smithi179_77138_object50805:head# 2024-02-09T17:27:03.359 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.359 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f359d7da:::benchmark_data_smithi179_77138_object105432:head# 2024-02-09T17:27:03.359 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.359 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f35b767a:::benchmark_data_smithi179_77138_object110820:head# 2024-02-09T17:27:03.359 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.365 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f35b7861:::benchmark_data_smithi179_77138_object126426:head# 2024-02-09T17:27:03.365 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.365 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f35d30b5:::benchmark_data_smithi179_77138_object113452:head# 2024-02-09T17:27:03.365 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.365 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f35d4311:::benchmark_data_smithi179_77138_object113153:head# 2024-02-09T17:27:03.365 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.365 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f35d4f3a:::benchmark_data_smithi179_77138_object31623:head# 2024-02-09T17:27:03.365 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.366 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f35e0e72:::benchmark_data_smithi179_77138_object128290:head# 2024-02-09T17:27:03.366 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.366 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f35ebc25:::benchmark_data_smithi179_77138_object60531:head# 2024-02-09T17:27:03.366 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.366 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f35f1526:::benchmark_data_smithi179_77138_object60215:head# 2024-02-09T17:27:03.367 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.367 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f35f69a7:::benchmark_data_smithi179_77138_object101290:head# 2024-02-09T17:27:03.367 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.367 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f35f7594:::benchmark_data_smithi179_77138_object59927:head# 2024-02-09T17:27:03.367 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.369 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3603e39:::benchmark_data_smithi179_77138_object82961:head# 2024-02-09T17:27:03.369 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.369 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3619847:::benchmark_data_smithi179_77138_object106935:head# 2024-02-09T17:27:03.369 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.369 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3625d53:::benchmark_data_smithi179_77138_object74210:head# 2024-02-09T17:27:03.369 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.369 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3628c17:::benchmark_data_smithi179_77138_object124041:head# 2024-02-09T17:27:03.369 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.369 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f363ee1a:::benchmark_data_smithi179_77138_object101752:head# 2024-02-09T17:27:03.369 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.369 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f365a335:::benchmark_data_smithi179_77138_object68365:head# 2024-02-09T17:27:03.369 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.370 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3661351:::benchmark_data_smithi179_77138_object115833:head# 2024-02-09T17:27:03.370 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.370 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3666e01:::benchmark_data_smithi179_77138_object83371:head# 2024-02-09T17:27:03.370 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.370 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f367122a:::benchmark_data_smithi179_77138_object118036:head# 2024-02-09T17:27:03.370 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.370 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f36769fd:::benchmark_data_smithi179_77138_object58519:head# 2024-02-09T17:27:03.370 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.373 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f367f6ca:::benchmark_data_smithi179_77138_object69375:head# 2024-02-09T17:27:03.373 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.373 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3685b11:::benchmark_data_smithi179_77138_object105665:head# 2024-02-09T17:27:03.373 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.373 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f368d943:::benchmark_data_smithi179_77138_object29698:head# 2024-02-09T17:27:03.373 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.373 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f368fdd7:::benchmark_data_smithi179_77138_object60804:head# 2024-02-09T17:27:03.374 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.374 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f369693a:::benchmark_data_smithi179_77138_object30948:head# 2024-02-09T17:27:03.374 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.374 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f36ad7a6:::benchmark_data_smithi179_77138_object25980:head# 2024-02-09T17:27:03.374 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.374 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f36b6ddf:::benchmark_data_smithi179_77138_object90411:head# 2024-02-09T17:27:03.374 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.374 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f36bf510:::benchmark_data_smithi179_77138_object98041:head# 2024-02-09T17:27:03.374 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.374 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f36c5d13:::benchmark_data_smithi179_77138_object112703:head# 2024-02-09T17:27:03.374 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.374 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f36dacfa:::benchmark_data_smithi179_77138_object62245:head# 2024-02-09T17:27:03.374 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.374 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f36de74b:::benchmark_data_smithi179_77138_object56010:head# 2024-02-09T17:27:03.374 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.375 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f36e032f:::benchmark_data_smithi179_77138_object114025:head# 2024-02-09T17:27:03.375 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.375 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f36eaf56:::benchmark_data_smithi179_77138_object105644:head# 2024-02-09T17:27:03.375 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.375 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f36ebec6:::benchmark_data_smithi179_77138_object40288:head# 2024-02-09T17:27:03.375 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.375 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f36ed44e:::benchmark_data_smithi179_77138_object110080:head# 2024-02-09T17:27:03.375 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.375 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f36f3f9c:::benchmark_data_smithi179_77138_object20388:head# 2024-02-09T17:27:03.375 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.375 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3710b9d:::benchmark_data_smithi179_77138_object78354:head# 2024-02-09T17:27:03.375 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.375 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3716141:::benchmark_data_smithi179_77138_object19499:head# 2024-02-09T17:27:03.375 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.376 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f371970d:::benchmark_data_smithi179_77138_object31382:head# 2024-02-09T17:27:03.376 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.380 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f371c748:::benchmark_data_smithi179_77138_object110485:head# 2024-02-09T17:27:03.381 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.381 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f371d621:::benchmark_data_smithi179_77138_object102910:head# 2024-02-09T17:27:03.381 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.381 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f371e563:::benchmark_data_smithi179_77138_object81458:head# 2024-02-09T17:27:03.381 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.381 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f372dc9d:::benchmark_data_smithi179_77138_object54481:head# 2024-02-09T17:27:03.381 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.381 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f37319de:::benchmark_data_smithi179_77138_object76313:head# 2024-02-09T17:27:03.381 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.381 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f373237e:::benchmark_data_smithi179_77138_object56550:head# 2024-02-09T17:27:03.381 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.381 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3732fba:::benchmark_data_smithi179_77138_object103708:head# 2024-02-09T17:27:03.381 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.381 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f373441a:::benchmark_data_smithi179_77138_object46635:head# 2024-02-09T17:27:03.382 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.382 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f374d7e3:::benchmark_data_smithi179_77138_object93019:head# 2024-02-09T17:27:03.382 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.382 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f375c169:::benchmark_data_smithi179_77138_object99523:head# 2024-02-09T17:27:03.382 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.382 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3763742:::benchmark_data_smithi179_77138_object25195:head# 2024-02-09T17:27:03.382 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.382 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f37668a9:::benchmark_data_smithi179_77138_object67133:head# 2024-02-09T17:27:03.382 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.382 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f377c78e:::benchmark_data_smithi179_77138_object121203:head# 2024-02-09T17:27:03.382 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.382 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3782076:::benchmark_data_smithi179_77138_object126982:head# 2024-02-09T17:27:03.382 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.383 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f378a7f8:::benchmark_data_smithi179_77138_object101175:head# 2024-02-09T17:27:03.383 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.383 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f378f48e:::benchmark_data_smithi179_77138_object23793:head# 2024-02-09T17:27:03.383 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.383 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f37a5eef:::benchmark_data_smithi179_77138_object35703:head# 2024-02-09T17:27:03.383 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.383 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f37a6280:::benchmark_data_smithi179_77138_object71730:head# 2024-02-09T17:27:03.383 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.383 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f37b91db:::benchmark_data_smithi179_77138_object112938:head# 2024-02-09T17:27:03.383 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.383 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f37f07f7:::benchmark_data_smithi179_77138_object110918:head# 2024-02-09T17:27:03.383 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.383 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f37fe900:::benchmark_data_smithi179_77138_object35046:head# 2024-02-09T17:27:03.383 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.384 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3802b4b:::benchmark_data_smithi179_77138_object116179:head# 2024-02-09T17:27:03.384 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.384 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f38072e5:::benchmark_data_smithi179_77138_object107544:head# 2024-02-09T17:27:03.384 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.384 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f380c735:::benchmark_data_smithi179_77138_object40124:head# 2024-02-09T17:27:03.384 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.384 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f382ce60:::benchmark_data_smithi179_77138_object79915:head# 2024-02-09T17:27:03.384 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.384 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f384600b:::benchmark_data_smithi179_77138_object55413:head# 2024-02-09T17:27:03.384 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.384 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f384bab3:::benchmark_data_smithi179_77138_object99924:head# 2024-02-09T17:27:03.384 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.384 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f384efbf:::benchmark_data_smithi179_77138_object79432:head# 2024-02-09T17:27:03.384 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.385 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3857419:::benchmark_data_smithi179_77138_object83723:head# 2024-02-09T17:27:03.385 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.385 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3857667:::benchmark_data_smithi179_77138_object109905:head# 2024-02-09T17:27:03.385 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.385 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f385d8fc:::benchmark_data_smithi179_77138_object118613:head# 2024-02-09T17:27:03.385 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.385 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f38631a4:::benchmark_data_smithi179_77138_object121377:head# 2024-02-09T17:27:03.385 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.385 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3866006:::benchmark_data_smithi179_77138_object29645:head# 2024-02-09T17:27:03.385 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.385 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f38812da:::benchmark_data_smithi179_77138_object94253:head# 2024-02-09T17:27:03.385 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.392 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3887c7b:::benchmark_data_smithi179_77138_object45426:head# 2024-02-09T17:27:03.392 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.392 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f388c1d2:::benchmark_data_smithi179_77138_object43091:head# 2024-02-09T17:27:03.392 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.392 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f388cbf3:::benchmark_data_smithi179_77138_object91051:head# 2024-02-09T17:27:03.392 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.392 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f388e824:::benchmark_data_smithi179_77138_object69739:head# 2024-02-09T17:27:03.392 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.392 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f389b6ab:::benchmark_data_smithi179_77138_object116980:head# 2024-02-09T17:27:03.392 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.392 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f38a3e6a:::benchmark_data_smithi179_77138_object24657:head# 2024-02-09T17:27:03.393 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.393 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f38a6bbc:::benchmark_data_smithi179_77138_object112460:head# 2024-02-09T17:27:03.393 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.393 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f38ab554:::benchmark_data_smithi179_77138_object43828:head# 2024-02-09T17:27:03.393 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.393 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f38b065a:::benchmark_data_smithi179_77138_object34880:head# 2024-02-09T17:27:03.393 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.393 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f38be686:::benchmark_data_smithi179_77138_object103634:head# 2024-02-09T17:27:03.393 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.393 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f38c3da4:::benchmark_data_smithi179_77138_object72506:head# 2024-02-09T17:27:03.393 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.393 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f38cf2ea:::benchmark_data_smithi179_77138_object94505:head# 2024-02-09T17:27:03.393 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.393 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f38d73c5:::benchmark_data_smithi179_77138_object55026:head# 2024-02-09T17:27:03.394 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.394 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f38f9211:::benchmark_data_smithi179_77138_object34938:head# 2024-02-09T17:27:03.394 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.394 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3901255:::benchmark_data_smithi179_77138_object87136:head# 2024-02-09T17:27:03.394 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.394 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f390630b:::benchmark_data_smithi179_77138_object61551:head# 2024-02-09T17:27:03.394 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.394 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3924d9b:::benchmark_data_smithi179_77138_object130407:head# 2024-02-09T17:27:03.394 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.394 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f39316b2:::benchmark_data_smithi179_77138_object84608:head# 2024-02-09T17:27:03.394 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.394 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3938dc7:::benchmark_data_smithi179_77138_object113948:head# 2024-02-09T17:27:03.394 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.394 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3945652:::benchmark_data_smithi179_77138_object21043:head# 2024-02-09T17:27:03.395 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.395 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f394cae8:::benchmark_data_smithi179_77138_object77319:head# 2024-02-09T17:27:03.395 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.395 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f394cc42:::benchmark_data_smithi179_77138_object36772:head# 2024-02-09T17:27:03.395 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.395 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3957591:::benchmark_data_smithi179_77138_object107670:head# 2024-02-09T17:27:03.395 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.395 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f395e5d4:::benchmark_data_smithi179_77138_object92779:head# 2024-02-09T17:27:03.395 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.395 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f39700ae:::benchmark_data_smithi179_77138_object99910:head# 2024-02-09T17:27:03.395 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.395 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3982074:::benchmark_data_smithi179_77138_object130278:head# 2024-02-09T17:27:03.395 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.395 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f398398e:::benchmark_data_smithi179_77138_object98517:head# 2024-02-09T17:27:03.395 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.396 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3993949:::benchmark_data_smithi179_77138_object94958:head# 2024-02-09T17:27:03.396 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.396 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f39a6155:::benchmark_data_smithi179_77138_object70022:head# 2024-02-09T17:27:03.396 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.396 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f39ac82f:::benchmark_data_smithi179_77138_object47532:head# 2024-02-09T17:27:03.396 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.396 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f39b14a9:::benchmark_data_smithi179_77138_object110518:head# 2024-02-09T17:27:03.396 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.396 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f39beecb:::benchmark_data_smithi179_77138_object118008:head# 2024-02-09T17:27:03.396 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.396 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f39c0e87:::benchmark_data_smithi179_77138_object34080:head# 2024-02-09T17:27:03.396 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.396 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f39c5c99:::benchmark_data_smithi179_77138_object70460:head# 2024-02-09T17:27:03.397 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.397 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f39ca776:::benchmark_data_smithi179_77138_object31702:head# 2024-02-09T17:27:03.397 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.397 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f39d2790:::benchmark_data_smithi179_77138_object123796:head# 2024-02-09T17:27:03.397 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.397 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f39df4b6:::benchmark_data_smithi179_77138_object125141:head# 2024-02-09T17:27:03.397 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.397 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f39f5afd:::benchmark_data_smithi179_77138_object48279:head# 2024-02-09T17:27:03.397 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.397 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f39fb346:::benchmark_data_smithi179_77138_object77032:head# 2024-02-09T17:27:03.397 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.397 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3a09a4c:::benchmark_data_smithi179_77138_object39278:head# 2024-02-09T17:27:03.397 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.397 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3a0e102:::benchmark_data_smithi179_77138_object78435:head# 2024-02-09T17:27:03.397 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.398 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3a3ef09:::benchmark_data_smithi179_77138_object109274:head# 2024-02-09T17:27:03.398 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.398 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3a3f583:::benchmark_data_smithi179_77138_object66154:head# 2024-02-09T17:27:03.398 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.398 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3a4887e:::benchmark_data_smithi179_77138_object111367:head# 2024-02-09T17:27:03.398 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.398 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3a6b87c:::benchmark_data_smithi179_77138_object35409:head# 2024-02-09T17:27:03.398 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.398 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3a7461b:::benchmark_data_smithi179_77138_object21237:head# 2024-02-09T17:27:03.398 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.400 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3a768c7:::benchmark_data_smithi179_77138_object27090:head# 2024-02-09T17:27:03.400 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.400 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3a7e7ee:::benchmark_data_smithi179_77138_object72990:head# 2024-02-09T17:27:03.400 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.400 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3a860d2:::benchmark_data_smithi179_77138_object88975:head# 2024-02-09T17:27:03.400 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.401 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3a92f43:::benchmark_data_smithi179_77138_object66554:head# 2024-02-09T17:27:03.401 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.401 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3a93fa8:::benchmark_data_smithi179_77138_object58649:head# 2024-02-09T17:27:03.401 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.401 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3a983b8:::benchmark_data_smithi179_77138_object41609:head# 2024-02-09T17:27:03.401 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.401 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3ab0cec:::benchmark_data_smithi179_77138_object113490:head# 2024-02-09T17:27:03.401 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.401 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3ab131d:::benchmark_data_smithi179_77138_object127321:head# 2024-02-09T17:27:03.401 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.401 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3ab27c4:::benchmark_data_smithi179_77138_object60853:head# 2024-02-09T17:27:03.401 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.401 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3ab74b0:::benchmark_data_smithi179_77138_object86309:head# 2024-02-09T17:27:03.401 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.402 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3ac2203:::benchmark_data_smithi179_77138_object39974:head# 2024-02-09T17:27:03.402 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.402 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3ad2cf4:::benchmark_data_smithi179_77138_object96596:head# 2024-02-09T17:27:03.402 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.402 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3adcf6b:::benchmark_data_smithi179_77138_object77945:head# 2024-02-09T17:27:03.402 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.402 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3ade0f0:::benchmark_data_smithi179_77138_object82091:head# 2024-02-09T17:27:03.402 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.402 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3ae0325:::benchmark_data_smithi179_77138_object79440:head# 2024-02-09T17:27:03.402 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.402 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3affab7:::benchmark_data_smithi179_77138_object43467:head# 2024-02-09T17:27:03.402 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.402 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3affb50:::benchmark_data_smithi179_77138_object39847:head# 2024-02-09T17:27:03.402 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.402 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3b0587d:::benchmark_data_smithi179_77138_object118275:head# 2024-02-09T17:27:03.403 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.403 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3b08962:::benchmark_data_smithi179_77138_object60133:head# 2024-02-09T17:27:03.403 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.403 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3b2390b:::benchmark_data_smithi179_77138_object85523:head# 2024-02-09T17:27:03.403 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.403 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3b24de8:::benchmark_data_smithi179_77138_object49523:head# 2024-02-09T17:27:03.403 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.403 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3b292b1:::benchmark_data_smithi179_77138_object96960:head# 2024-02-09T17:27:03.403 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.403 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3b31106:::benchmark_data_smithi179_77138_object58102:head# 2024-02-09T17:27:03.403 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.403 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3b37e0e:::benchmark_data_smithi179_77138_object132085:head# 2024-02-09T17:27:03.403 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.403 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3b3eeeb:::benchmark_data_smithi179_77138_object53066:head# 2024-02-09T17:27:03.404 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.404 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3b4783f:::benchmark_data_smithi179_77138_object61436:head# 2024-02-09T17:27:03.404 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.404 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3b4ed54:::benchmark_data_smithi179_77138_object119232:head# 2024-02-09T17:27:03.404 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.404 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3b521d1:::benchmark_data_smithi179_77138_object83160:head# 2024-02-09T17:27:03.404 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.405 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3b5bdac:::benchmark_data_smithi179_77138_object49263:head# 2024-02-09T17:27:03.405 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.405 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3b66979:::benchmark_data_smithi179_77138_object43729:head# 2024-02-09T17:27:03.405 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.406 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3b673e6:::benchmark_data_smithi179_77138_object81109:head# 2024-02-09T17:27:03.406 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.406 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3b6e0dc:::benchmark_data_smithi179_77138_object72759:head# 2024-02-09T17:27:03.406 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.406 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3b7accd:::benchmark_data_smithi179_77138_object86738:head# 2024-02-09T17:27:03.406 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.406 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3b7b554:::benchmark_data_smithi179_77138_object114994:head# 2024-02-09T17:27:03.406 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.406 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3b7d60f:::benchmark_data_smithi179_77138_object69940:head# 2024-02-09T17:27:03.406 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.406 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3b9d3dc:::benchmark_data_smithi179_77138_object77762:head# 2024-02-09T17:27:03.406 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.406 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3ba2bd8:::benchmark_data_smithi179_77138_object97168:head# 2024-02-09T17:27:03.406 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.406 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3ba2d5d:::benchmark_data_smithi179_77138_object84682:head# 2024-02-09T17:27:03.407 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.407 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3ba4ef6:::benchmark_data_smithi179_77138_object26116:head# 2024-02-09T17:27:03.407 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.407 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3bafce5:::benchmark_data_smithi179_77138_object34963:head# 2024-02-09T17:27:03.407 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.407 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3bb8a14:::benchmark_data_smithi179_77138_object102715:head# 2024-02-09T17:27:03.407 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.407 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3bc27b7:::benchmark_data_smithi179_77138_object32134:head# 2024-02-09T17:27:03.407 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.407 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3bcce2b:::benchmark_data_smithi179_77138_object21756:head# 2024-02-09T17:27:03.407 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.407 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3bcdba9:::benchmark_data_smithi179_77138_object122870:head# 2024-02-09T17:27:03.407 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.407 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3be66fe:::benchmark_data_smithi179_77138_object36596:head# 2024-02-09T17:27:03.407 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.411 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3bed4df:::benchmark_data_smithi179_77138_object29824:head# 2024-02-09T17:27:03.411 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.411 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3c0d6e3:::benchmark_data_smithi179_77138_object104524:head# 2024-02-09T17:27:03.411 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.412 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3c0df6d:::benchmark_data_smithi179_77138_object70346:head# 2024-02-09T17:27:03.412 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.412 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3c115c4:::benchmark_data_smithi179_77138_object93443:head# 2024-02-09T17:27:03.412 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.412 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3c5687c:::benchmark_data_smithi179_77138_object90001:head# 2024-02-09T17:27:03.412 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.412 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3c644ff:::benchmark_data_smithi179_77138_object36405:head# 2024-02-09T17:27:03.412 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.412 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3c673da:::benchmark_data_smithi179_77138_object101665:head# 2024-02-09T17:27:03.412 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.412 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3c681d2:::benchmark_data_smithi179_77138_object116687:head# 2024-02-09T17:27:03.412 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.412 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3c767ab:::benchmark_data_smithi179_77138_object104582:head# 2024-02-09T17:27:03.412 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.413 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3c87c10:::benchmark_data_smithi179_77138_object91508:head# 2024-02-09T17:27:03.413 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.413 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3c8995e:::benchmark_data_smithi179_77138_object122157:head# 2024-02-09T17:27:03.413 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.413 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3c917bb:::benchmark_data_smithi179_77138_object66582:head# 2024-02-09T17:27:03.413 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.413 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3c92743:::benchmark_data_smithi179_77138_object122736:head# 2024-02-09T17:27:03.413 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.413 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3c98906:::benchmark_data_smithi179_77138_object70005:head# 2024-02-09T17:27:03.413 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.413 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3cb4f55:::benchmark_data_smithi179_77138_object123253:head# 2024-02-09T17:27:03.413 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.413 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3cb559c:::benchmark_data_smithi179_77138_object123465:head# 2024-02-09T17:27:03.413 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.414 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3cc9180:::benchmark_data_smithi179_77138_object57098:head# 2024-02-09T17:27:03.414 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.414 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3cca11d:::benchmark_data_smithi179_77138_object119468:head# 2024-02-09T17:27:03.414 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.414 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3ce2e9b:::benchmark_data_smithi179_77138_object53468:head# 2024-02-09T17:27:03.414 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.414 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3cebee5:::benchmark_data_smithi179_77138_object81960:head# 2024-02-09T17:27:03.414 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.414 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3cf0e6e:::benchmark_data_smithi179_77138_object103201:head# 2024-02-09T17:27:03.414 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.414 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3cf91d4:::benchmark_data_smithi179_77138_object40451:head# 2024-02-09T17:27:03.414 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.414 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3cfab04:::benchmark_data_smithi179_77138_object46411:head# 2024-02-09T17:27:03.414 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.415 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3d0b9a1:::benchmark_data_smithi179_77138_object62353:head# 2024-02-09T17:27:03.415 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.415 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3d25fb1:::benchmark_data_smithi179_77138_object42240:head# 2024-02-09T17:27:03.415 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.415 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3d280f8:::benchmark_data_smithi179_77138_object48322:head# 2024-02-09T17:27:03.415 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.415 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3d29876:::benchmark_data_smithi179_77138_object102268:head# 2024-02-09T17:27:03.415 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.415 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3d2c9e5:::benchmark_data_smithi179_77138_object31257:head# 2024-02-09T17:27:03.415 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.417 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3d34b11:::benchmark_data_smithi179_77138_object119186:head# 2024-02-09T17:27:03.417 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.417 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3d3b032:::benchmark_data_smithi179_77138_object48056:head# 2024-02-09T17:27:03.417 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.417 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3d3cf83:::benchmark_data_smithi179_77138_object37576:head# 2024-02-09T17:27:03.418 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.418 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3d3f6d0:::benchmark_data_smithi179_77138_object27336:head# 2024-02-09T17:27:03.418 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.418 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3d3ff50:::benchmark_data_smithi179_77138_object28367:head# 2024-02-09T17:27:03.418 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.418 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3d47621:::benchmark_data_smithi179_77138_object39743:head# 2024-02-09T17:27:03.418 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.418 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3d4bae2:::benchmark_data_smithi179_77138_object79531:head# 2024-02-09T17:27:03.418 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.418 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3d5cd09:::benchmark_data_smithi179_77138_object51999:head# 2024-02-09T17:27:03.418 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.418 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3d6ae20:::benchmark_data_smithi179_77138_object107293:head# 2024-02-09T17:27:03.419 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.419 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3d7f8c2:::benchmark_data_smithi179_77138_object112452:head# 2024-02-09T17:27:03.419 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.419 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3d8f3a0:::benchmark_data_smithi179_77138_object35885:head# 2024-02-09T17:27:03.419 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.419 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3d90557:::benchmark_data_smithi179_77138_object38086:head# 2024-02-09T17:27:03.419 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.420 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3d9345c:::benchmark_data_smithi179_77138_object60432:head# 2024-02-09T17:27:03.420 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.421 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3d96f3c:::benchmark_data_smithi179_77138_object121620:head# 2024-02-09T17:27:03.421 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.421 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3d98682:::benchmark_data_smithi179_77138_object94572:head# 2024-02-09T17:27:03.421 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.421 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3da037f:::benchmark_data_smithi179_77138_object36247:head# 2024-02-09T17:27:03.421 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.421 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3da0944:::benchmark_data_smithi179_77138_object131269:head# 2024-02-09T17:27:03.421 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.421 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3db66a5:::benchmark_data_smithi179_77138_object72021:head# 2024-02-09T17:27:03.421 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.421 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3db755c:::benchmark_data_smithi179_77138_object51563:head# 2024-02-09T17:27:03.421 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.422 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3db8e3c:::benchmark_data_smithi179_77138_object133156:head# 2024-02-09T17:27:03.422 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.422 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3dbc159:::benchmark_data_smithi179_77138_object98406:head# 2024-02-09T17:27:03.422 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.422 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3dbf883:::benchmark_data_smithi179_77138_object52048:head# 2024-02-09T17:27:03.422 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.422 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3dc54df:::benchmark_data_smithi179_77138_object68997:head# 2024-02-09T17:27:03.422 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.422 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3dd0265:::benchmark_data_smithi179_77138_object20652:head# 2024-02-09T17:27:03.422 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.422 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3dd92e7:::benchmark_data_smithi179_77138_object126920:head# 2024-02-09T17:27:03.422 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.422 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3ddedbf:::benchmark_data_smithi179_77138_object30467:head# 2024-02-09T17:27:03.423 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.423 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3dfbf43:::benchmark_data_smithi179_77138_object88143:head# 2024-02-09T17:27:03.423 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.423 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3e00489:::benchmark_data_smithi179_77138_object39514:head# 2024-02-09T17:27:03.423 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.423 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3e113ff:::benchmark_data_smithi179_77138_object88223:head# 2024-02-09T17:27:03.423 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.423 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3e1cba1:::benchmark_data_smithi179_77138_object126626:head# 2024-02-09T17:27:03.423 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.423 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3e20475:::benchmark_data_smithi179_77138_object102884:head# 2024-02-09T17:27:03.423 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.423 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3e2c152:::benchmark_data_smithi179_77138_object30432:head# 2024-02-09T17:27:03.423 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.424 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3e3858e:::benchmark_data_smithi179_77138_object45322:head# 2024-02-09T17:27:03.424 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.424 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3e48dd8:::benchmark_data_smithi179_77138_object45459:head# 2024-02-09T17:27:03.424 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.424 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3e5bdb3:::benchmark_data_smithi179_77138_object119278:head# 2024-02-09T17:27:03.424 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.428 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3e657a3:::benchmark_data_smithi179_77138_object117900:head# 2024-02-09T17:27:03.428 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.428 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3e80ae8:::benchmark_data_smithi179_77138_object126760:head# 2024-02-09T17:27:03.428 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.428 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3e8922b:::benchmark_data_smithi179_77138_object42148:head# 2024-02-09T17:27:03.428 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.428 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3e93570:::benchmark_data_smithi179_77138_object24613:head# 2024-02-09T17:27:03.428 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.429 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3e953c2:::benchmark_data_smithi179_77138_object112088:head# 2024-02-09T17:27:03.429 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.429 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3eb4b85:::benchmark_data_smithi179_77138_object110710:head# 2024-02-09T17:27:03.429 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.429 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3eba75b:::benchmark_data_smithi179_77138_object23309:head# 2024-02-09T17:27:03.429 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.429 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3ebefd8:::benchmark_data_smithi179_77138_object131923:head# 2024-02-09T17:27:03.429 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.429 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3ec8533:::benchmark_data_smithi179_77138_object115444:head# 2024-02-09T17:27:03.429 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.429 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3ed1f80:::benchmark_data_smithi179_77138_object60973:head# 2024-02-09T17:27:03.429 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.429 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3ed248b:::benchmark_data_smithi179_77138_object42946:head# 2024-02-09T17:27:03.430 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.430 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3ed9344:::benchmark_data_smithi179_77138_object73832:head# 2024-02-09T17:27:03.430 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.430 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3edf694:::benchmark_data_smithi179_77138_object118512:head# 2024-02-09T17:27:03.430 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.430 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3ee611d:::benchmark_data_smithi179_77138_object63627:head# 2024-02-09T17:27:03.430 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.430 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3f0bbcc:::benchmark_data_smithi179_77138_object103824:head# 2024-02-09T17:27:03.430 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.430 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3f14dbc:::benchmark_data_smithi179_77138_object57776:head# 2024-02-09T17:27:03.430 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.430 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3f15f47:::benchmark_data_smithi179_77138_object33759:head# 2024-02-09T17:27:03.430 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.430 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3f1a69a:::benchmark_data_smithi179_77138_object58233:head# 2024-02-09T17:27:03.431 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.431 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3f1acea:::benchmark_data_smithi179_77138_object54177:head# 2024-02-09T17:27:03.431 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.431 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3f2162a:::benchmark_data_smithi179_77138_object73668:head# 2024-02-09T17:27:03.431 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.431 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3f23a48:::benchmark_data_smithi179_77138_object57450:head# 2024-02-09T17:27:03.431 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.431 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3f27d32:::benchmark_data_smithi179_77138_object112482:head# 2024-02-09T17:27:03.431 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.431 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3f28d11:::benchmark_data_smithi179_77138_object72989:head# 2024-02-09T17:27:03.431 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.431 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3f2b1a3:::benchmark_data_smithi179_77138_object49972:head# 2024-02-09T17:27:03.431 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.431 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3f30f44:::benchmark_data_smithi179_77138_object72838:head# 2024-02-09T17:27:03.431 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.432 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3f337a6:::benchmark_data_smithi179_77138_object41182:head# 2024-02-09T17:27:03.432 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.432 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3f3611a:::benchmark_data_smithi179_77138_object46261:head# 2024-02-09T17:27:03.432 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.432 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3f5419f:::benchmark_data_smithi179_77138_object115514:head# 2024-02-09T17:27:03.432 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.433 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3f5b250:::benchmark_data_smithi179_77138_object48918:head# 2024-02-09T17:27:03.433 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.433 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3f5ea27:::benchmark_data_smithi179_77138_object92145:head# 2024-02-09T17:27:03.433 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.433 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3f68665:::benchmark_data_smithi179_77138_object59549:head# 2024-02-09T17:27:03.433 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.433 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3f697ab:::benchmark_data_smithi179_77138_object107601:head# 2024-02-09T17:27:03.433 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.434 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3f6aea5:::benchmark_data_smithi179_77138_object109883:head# 2024-02-09T17:27:03.434 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.434 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3f6c9f3:::benchmark_data_smithi179_77138_object105463:head# 2024-02-09T17:27:03.435 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.435 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3f773cb:::benchmark_data_smithi179_77138_object94934:head# 2024-02-09T17:27:03.435 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.435 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3f7957c:::benchmark_data_smithi179_77138_object28436:head# 2024-02-09T17:27:03.435 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.437 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3f8797e:::benchmark_data_smithi179_77138_object50317:head# 2024-02-09T17:27:03.437 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.437 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3f8c82e:::benchmark_data_smithi179_77138_object121834:head# 2024-02-09T17:27:03.437 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.437 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3f8db21:::benchmark_data_smithi179_77138_object77849:head# 2024-02-09T17:27:03.437 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.437 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3f8f52a:::benchmark_data_smithi179_77138_object89557:head# 2024-02-09T17:27:03.437 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.437 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3f9b336:::benchmark_data_smithi179_77138_object99533:head# 2024-02-09T17:27:03.437 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.437 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3f9b876:::benchmark_data_smithi179_77138_object31724:head# 2024-02-09T17:27:03.438 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.438 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3f9e6a7:::benchmark_data_smithi179_77138_object122723:head# 2024-02-09T17:27:03.438 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.438 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3fb95ad:::benchmark_data_smithi179_77138_object127903:head# 2024-02-09T17:27:03.438 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.438 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3fdcfaf:::benchmark_data_smithi179_77138_object71429:head# 2024-02-09T17:27:03.438 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.438 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3fe8167:::benchmark_data_smithi179_77138_object65546:head# 2024-02-09T17:27:03.438 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.440 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3ff1182:::benchmark_data_smithi179_77138_object98802:head# 2024-02-09T17:27:03.440 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.440 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3ff1d0b:::benchmark_data_smithi179_77138_object19376:head# 2024-02-09T17:27:03.440 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.440 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3ff2b73:::benchmark_data_smithi179_77138_object60205:head# 2024-02-09T17:27:03.440 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.440 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f3ffb5e9:::benchmark_data_smithi179_77138_object127271:head# 2024-02-09T17:27:03.441 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.441 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4015af3:::benchmark_data_smithi179_77138_object128823:head# 2024-02-09T17:27:03.441 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.441 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f402abe0:::benchmark_data_smithi179_77138_object77073:head# 2024-02-09T17:27:03.441 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.441 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f402e873:::benchmark_data_smithi179_77138_object90607:head# 2024-02-09T17:27:03.441 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.441 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4031482:::benchmark_data_smithi179_77138_object40401:head# 2024-02-09T17:27:03.441 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.441 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4038bd2:::benchmark_data_smithi179_77138_object23638:head# 2024-02-09T17:27:03.441 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.441 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4041da8:::benchmark_data_smithi179_77138_object72111:head# 2024-02-09T17:27:03.441 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.441 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4043abd:::benchmark_data_smithi179_77138_object66429:head# 2024-02-09T17:27:03.442 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.442 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f404f260:::benchmark_data_smithi179_77138_object40779:head# 2024-02-09T17:27:03.442 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.445 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4064b9d:::benchmark_data_smithi179_77138_object78073:head# 2024-02-09T17:27:03.445 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.445 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4067565:::benchmark_data_smithi179_77138_object25000:head# 2024-02-09T17:27:03.445 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.445 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4067dcd:::benchmark_data_smithi179_77138_object116084:head# 2024-02-09T17:27:03.445 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.445 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f40733d6:::benchmark_data_smithi179_77138_object100756:head# 2024-02-09T17:27:03.445 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.446 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f408313d:::benchmark_data_smithi179_77138_object126115:head# 2024-02-09T17:27:03.446 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.446 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4083bfc:::benchmark_data_smithi179_77138_object110159:head# 2024-02-09T17:27:03.446 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.446 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f408a84b:::benchmark_data_smithi179_77138_object71951:head# 2024-02-09T17:27:03.446 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.446 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4095bea:::benchmark_data_smithi179_77138_object41538:head# 2024-02-09T17:27:03.446 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.446 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f409db89:::benchmark_data_smithi179_77138_object40498:head# 2024-02-09T17:27:03.446 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.446 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f40a44c0:::benchmark_data_smithi179_77138_object39552:head# 2024-02-09T17:27:03.446 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.447 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f40a6dab:::benchmark_data_smithi179_77138_object37741:head# 2024-02-09T17:27:03.447 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.447 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f40adc04:::benchmark_data_smithi179_77138_object132233:head# 2024-02-09T17:27:03.447 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.447 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f40c6bb1:::benchmark_data_smithi179_77138_object42917:head# 2024-02-09T17:27:03.447 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.447 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f40d0797:::benchmark_data_smithi179_77138_object84158:head# 2024-02-09T17:27:03.447 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.447 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f40e046d:::benchmark_data_smithi179_77138_object49308:head# 2024-02-09T17:27:03.447 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.447 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f40e88fc:::benchmark_data_smithi179_77138_object29517:head# 2024-02-09T17:27:03.447 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.447 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f40f4202:::benchmark_data_smithi179_77138_object25753:head# 2024-02-09T17:27:03.447 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.447 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f40fa10a:::benchmark_data_smithi179_77138_object64981:head# 2024-02-09T17:27:03.448 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.448 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4103277:::benchmark_data_smithi179_77138_object130947:head# 2024-02-09T17:27:03.448 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.449 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f41062fe:::benchmark_data_smithi179_77138_object121263:head# 2024-02-09T17:27:03.449 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.450 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f41188bb:::benchmark_data_smithi179_77138_object54861:head# 2024-02-09T17:27:03.450 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.450 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f413edad:::benchmark_data_smithi179_77138_object82940:head# 2024-02-09T17:27:03.450 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.450 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f414db3d:::benchmark_data_smithi179_77138_object48457:head# 2024-02-09T17:27:03.450 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.450 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f415020f:::benchmark_data_smithi179_77138_object115384:head# 2024-02-09T17:27:03.450 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.450 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f415eb24:::benchmark_data_smithi179_77138_object90618:head# 2024-02-09T17:27:03.450 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.450 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f416a826:::benchmark_data_smithi179_77138_object52728:head# 2024-02-09T17:27:03.450 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.450 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4186dee:::benchmark_data_smithi179_77138_object46124:head# 2024-02-09T17:27:03.450 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.450 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f41899fd:::benchmark_data_smithi179_77138_object123187:head# 2024-02-09T17:27:03.451 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.451 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f418a840:::benchmark_data_smithi179_77138_object112746:head# 2024-02-09T17:27:03.451 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.451 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f418b57d:::benchmark_data_smithi179_77138_object59834:head# 2024-02-09T17:27:03.451 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.452 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f418be91:::benchmark_data_smithi179_77138_object112220:head# 2024-02-09T17:27:03.452 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.452 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f41a24f7:::benchmark_data_smithi179_77138_object48427:head# 2024-02-09T17:27:03.452 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.453 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f41a2adb:::benchmark_data_smithi179_77138_object68093:head# 2024-02-09T17:27:03.453 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.453 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f41b54e8:::benchmark_data_smithi179_77138_object92590:head# 2024-02-09T17:27:03.453 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.458 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f41ca169:::benchmark_data_smithi179_77138_object91938:head# 2024-02-09T17:27:03.458 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.458 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f41d3cdd:::benchmark_data_smithi179_77138_object82509:head# 2024-02-09T17:27:03.458 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.458 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f41ebf07:::benchmark_data_smithi179_77138_object50384:head# 2024-02-09T17:27:03.458 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.458 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f41eefe6:::benchmark_data_smithi179_77138_object49125:head# 2024-02-09T17:27:03.458 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.458 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f41efa7c:::benchmark_data_smithi179_77138_object122445:head# 2024-02-09T17:27:03.459 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.459 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f41f3a74:::benchmark_data_smithi179_77138_object130152:head# 2024-02-09T17:27:03.459 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.459 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f41ff67a:::benchmark_data_smithi179_77138_object63768:head# 2024-02-09T17:27:03.459 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.459 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4203af2:::benchmark_data_smithi179_77138_object51253:head# 2024-02-09T17:27:03.459 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.459 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4213f91:::benchmark_data_smithi179_77138_object131471:head# 2024-02-09T17:27:03.459 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.459 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4218265:::benchmark_data_smithi179_77138_object109146:head# 2024-02-09T17:27:03.459 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.459 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4227873:::benchmark_data_smithi179_77138_object56143:head# 2024-02-09T17:27:03.459 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.459 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4229b5b:::benchmark_data_smithi179_77138_object84615:head# 2024-02-09T17:27:03.460 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.460 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f42324c6:::benchmark_data_smithi179_77138_object68673:head# 2024-02-09T17:27:03.460 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.460 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4235b76:::benchmark_data_smithi179_77138_object65748:head# 2024-02-09T17:27:03.460 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.460 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f423d949:::benchmark_data_smithi179_77138_object34199:head# 2024-02-09T17:27:03.460 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.460 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f424ac89:::benchmark_data_smithi179_77138_object111617:head# 2024-02-09T17:27:03.460 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.460 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4250ac9:::benchmark_data_smithi179_77138_object99691:head# 2024-02-09T17:27:03.460 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.460 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4260749:::benchmark_data_smithi179_77138_object103592:head# 2024-02-09T17:27:03.460 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.460 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4279081:::benchmark_data_smithi179_77138_object83550:head# 2024-02-09T17:27:03.461 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.461 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f427b8c7:::benchmark_data_smithi179_77138_object54880:head# 2024-02-09T17:27:03.461 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.461 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f427efa4:::benchmark_data_smithi179_77138_object57892:head# 2024-02-09T17:27:03.461 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.461 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4284c61:::benchmark_data_smithi179_77138_object128936:head# 2024-02-09T17:27:03.461 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.461 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f42937d5:::benchmark_data_smithi179_77138_object28898:head# 2024-02-09T17:27:03.461 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.461 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f42ae755:::benchmark_data_smithi179_77138_object71815:head# 2024-02-09T17:27:03.461 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.461 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f42bd8a6:::benchmark_data_smithi179_77138_object87359:head# 2024-02-09T17:27:03.461 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.466 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f42c8923:::benchmark_data_smithi179_77138_object24700:head# 2024-02-09T17:27:03.466 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.466 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f42d6830:::benchmark_data_smithi179_77138_object94109:head# 2024-02-09T17:27:03.466 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.466 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f42d7a42:::benchmark_data_smithi179_77138_object126839:head# 2024-02-09T17:27:03.466 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.466 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f42e8576:::benchmark_data_smithi179_77138_object21241:head# 2024-02-09T17:27:03.466 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.466 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f42ed6ae:::benchmark_data_smithi179_77138_object22963:head# 2024-02-09T17:27:03.467 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.467 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f42f7dfe:::benchmark_data_smithi179_77138_object94380:head# 2024-02-09T17:27:03.467 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.467 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4305e94:::benchmark_data_smithi179_77138_object30725:head# 2024-02-09T17:27:03.467 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.467 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f431103e:::benchmark_data_smithi179_77138_object109042:head# 2024-02-09T17:27:03.467 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.467 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4318322:::benchmark_data_smithi179_77138_object103054:head# 2024-02-09T17:27:03.467 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.467 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4318c61:::benchmark_data_smithi179_77138_object51036:head# 2024-02-09T17:27:03.467 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.467 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f43246d0:::benchmark_data_smithi179_77138_object82446:head# 2024-02-09T17:27:03.467 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.467 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f432e6d5:::benchmark_data_smithi179_77138_object54341:head# 2024-02-09T17:27:03.468 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.468 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f43370ff:::benchmark_data_smithi179_77138_object92067:head# 2024-02-09T17:27:03.468 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.468 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f43395ad:::benchmark_data_smithi179_77138_object107845:head# 2024-02-09T17:27:03.468 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.468 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f433c78e:::benchmark_data_smithi179_77138_object95067:head# 2024-02-09T17:27:03.468 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.468 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f433e2a2:::benchmark_data_smithi179_77138_object68553:head# 2024-02-09T17:27:03.468 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.468 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4347377:::benchmark_data_smithi179_77138_object68948:head# 2024-02-09T17:27:03.468 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.468 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4348ae9:::benchmark_data_smithi179_77138_object70103:head# 2024-02-09T17:27:03.469 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.469 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4354da3:::benchmark_data_smithi179_77138_object116080:head# 2024-02-09T17:27:03.469 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.469 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f435a1a0:::benchmark_data_smithi179_77138_object113466:head# 2024-02-09T17:27:03.469 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.469 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f435cbaf:::benchmark_data_smithi179_77138_object117557:head# 2024-02-09T17:27:03.469 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.469 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f436ac78:::benchmark_data_smithi179_77138_object115421:head# 2024-02-09T17:27:03.469 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.469 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f436f1f0:::benchmark_data_smithi179_77138_object122101:head# 2024-02-09T17:27:03.469 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.469 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f437a53d:::benchmark_data_smithi179_77138_object91485:head# 2024-02-09T17:27:03.469 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.470 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f43a049f:::benchmark_data_smithi179_77138_object128193:head# 2024-02-09T17:27:03.470 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.472 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f43a856a:::benchmark_data_smithi179_77138_object127414:head# 2024-02-09T17:27:03.472 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.472 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f43aea67:::benchmark_data_smithi179_77138_object106456:head# 2024-02-09T17:27:03.472 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.472 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f43b145c:::benchmark_data_smithi179_77138_object80599:head# 2024-02-09T17:27:03.472 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.472 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f43c609b:::benchmark_data_smithi179_77138_object23769:head# 2024-02-09T17:27:03.472 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.472 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f43cb164:::benchmark_data_smithi179_77138_object22046:head# 2024-02-09T17:27:03.473 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.473 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f43d32ea:::benchmark_data_smithi179_77138_object25011:head# 2024-02-09T17:27:03.473 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.473 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f43e1f34:::benchmark_data_smithi179_77138_object37004:head# 2024-02-09T17:27:03.473 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.473 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f43f3197:::benchmark_data_smithi179_77138_object131576:head# 2024-02-09T17:27:03.473 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.473 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f43f8238:::benchmark_data_smithi179_77138_object35239:head# 2024-02-09T17:27:03.473 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.473 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f43fa46c:::benchmark_data_smithi179_77138_object55948:head# 2024-02-09T17:27:03.473 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.473 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f43ff04c:::benchmark_data_smithi179_77138_object91264:head# 2024-02-09T17:27:03.474 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.474 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f44021d6:::benchmark_data_smithi179_77138_object64220:head# 2024-02-09T17:27:03.474 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.474 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4413123:::benchmark_data_smithi179_77138_object44977:head# 2024-02-09T17:27:03.474 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.474 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f441a267:::benchmark_data_smithi179_77138_object37759:head# 2024-02-09T17:27:03.474 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.474 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f443b70e:::benchmark_data_smithi179_77138_object114647:head# 2024-02-09T17:27:03.474 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.474 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f443c2d6:::benchmark_data_smithi179_77138_object81293:head# 2024-02-09T17:27:03.474 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.474 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f443ce2d:::benchmark_data_smithi179_77138_object29148:head# 2024-02-09T17:27:03.475 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.475 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4444c0f:::benchmark_data_smithi179_77138_object50112:head# 2024-02-09T17:27:03.475 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.475 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4464440:::benchmark_data_smithi179_77138_object114767:head# 2024-02-09T17:27:03.475 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.475 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4471508:::benchmark_data_smithi179_77138_object107703:head# 2024-02-09T17:27:03.475 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.475 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f447869a:::benchmark_data_smithi179_77138_object68776:head# 2024-02-09T17:27:03.475 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.475 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4478a8d:::benchmark_data_smithi179_77138_object52203:head# 2024-02-09T17:27:03.475 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.475 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4482c82:::benchmark_data_smithi179_77138_object98960:head# 2024-02-09T17:27:03.475 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.475 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f44874da:::benchmark_data_smithi179_77138_object101765:head# 2024-02-09T17:27:03.475 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.476 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f448c002:::benchmark_data_smithi179_77138_object33401:head# 2024-02-09T17:27:03.476 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.479 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f44959ef:::benchmark_data_smithi179_77138_object58607:head# 2024-02-09T17:27:03.479 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.479 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f44989f4:::benchmark_data_smithi179_77138_object30266:head# 2024-02-09T17:27:03.479 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.479 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f449bbb6:::benchmark_data_smithi179_77138_object117419:head# 2024-02-09T17:27:03.479 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.480 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f449c518:::benchmark_data_smithi179_77138_object69073:head# 2024-02-09T17:27:03.480 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.480 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f449f267:::benchmark_data_smithi179_77138_object109210:head# 2024-02-09T17:27:03.480 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.480 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f44a13c2:::benchmark_data_smithi179_77138_object37526:head# 2024-02-09T17:27:03.480 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.480 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f44a6cdb:::benchmark_data_smithi179_77138_object32057:head# 2024-02-09T17:27:03.480 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.480 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f44adb54:::benchmark_data_smithi179_77138_object76856:head# 2024-02-09T17:27:03.480 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.480 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f44b61cb:::benchmark_data_smithi179_77138_object95099:head# 2024-02-09T17:27:03.480 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.480 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f44bfdaa:::benchmark_data_smithi179_77138_object91377:head# 2024-02-09T17:27:03.481 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.481 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f44c513b:::benchmark_data_smithi179_77138_object26838:head# 2024-02-09T17:27:03.481 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.481 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f44c7fb5:::benchmark_data_smithi179_77138_object124918:head# 2024-02-09T17:27:03.481 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.481 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f44c86c2:::benchmark_data_smithi179_77138_object35180:head# 2024-02-09T17:27:03.481 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.481 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f44d0def:::benchmark_data_smithi179_77138_object32419:head# 2024-02-09T17:27:03.481 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.481 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f44f2261:::benchmark_data_smithi179_77138_object54694:head# 2024-02-09T17:27:03.481 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.481 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f45198b2:::benchmark_data_smithi179_77138_object41831:head# 2024-02-09T17:27:03.482 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.482 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f451ac4b:::benchmark_data_smithi179_77138_object104843:head# 2024-02-09T17:27:03.482 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.482 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4520760:::benchmark_data_smithi179_77138_object76141:head# 2024-02-09T17:27:03.482 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.482 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4529580:::benchmark_data_smithi179_77138_object112521:head# 2024-02-09T17:27:03.482 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.482 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f452a0b5:::benchmark_data_smithi179_77138_object107530:head# 2024-02-09T17:27:03.482 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.482 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f452de77:::benchmark_data_smithi179_77138_object33482:head# 2024-02-09T17:27:03.482 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.482 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f45300f4:::benchmark_data_smithi179_77138_object110624:head# 2024-02-09T17:27:03.482 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.482 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f45322be:::benchmark_data_smithi179_77138_object58828:head# 2024-02-09T17:27:03.483 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.483 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f453b72c:::benchmark_data_smithi179_77138_object75131:head# 2024-02-09T17:27:03.484 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.484 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4553b9a:::benchmark_data_smithi179_77138_object70847:head# 2024-02-09T17:27:03.484 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.484 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4561742:::benchmark_data_smithi179_77138_object93842:head# 2024-02-09T17:27:03.484 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.484 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f456407a:::benchmark_data_smithi179_77138_object32757:head# 2024-02-09T17:27:03.484 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.484 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f456fe87:::benchmark_data_smithi179_77138_object90856:head# 2024-02-09T17:27:03.484 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.484 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f45761a3:::benchmark_data_smithi179_77138_object120435:head# 2024-02-09T17:27:03.484 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.484 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4580cd4:::benchmark_data_smithi179_77138_object78286:head# 2024-02-09T17:27:03.484 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.485 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f459d584:::benchmark_data_smithi179_77138_object103167:head# 2024-02-09T17:27:03.485 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.485 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f45a5288:::benchmark_data_smithi179_77138_object63459:head# 2024-02-09T17:27:03.485 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.485 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f45b2279:::benchmark_data_smithi179_77138_object21474:head# 2024-02-09T17:27:03.485 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.485 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f45cabf8:::benchmark_data_smithi179_77138_object85238:head# 2024-02-09T17:27:03.485 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.485 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f45cffb8:::benchmark_data_smithi179_77138_object119199:head# 2024-02-09T17:27:03.485 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.485 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f45daf5c:::benchmark_data_smithi179_77138_object70068:head# 2024-02-09T17:27:03.485 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.485 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f45e3f20:::benchmark_data_smithi179_77138_object66184:head# 2024-02-09T17:27:03.485 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.485 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f45e53f2:::benchmark_data_smithi179_77138_object24697:head# 2024-02-09T17:27:03.486 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.486 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f45f07ef:::benchmark_data_smithi179_77138_object118026:head# 2024-02-09T17:27:03.486 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.486 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f45fec88:::benchmark_data_smithi179_77138_object128682:head# 2024-02-09T17:27:03.486 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.486 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f460bfd1:::benchmark_data_smithi179_77138_object61861:head# 2024-02-09T17:27:03.486 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.486 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f46144ea:::benchmark_data_smithi179_77138_object51492:head# 2024-02-09T17:27:03.486 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.494 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4614990:::benchmark_data_smithi179_77138_object37693:head# 2024-02-09T17:27:03.494 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.496 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4624a2c:::benchmark_data_smithi179_77138_object47477:head# 2024-02-09T17:27:03.496 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.496 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f46270da:::benchmark_data_smithi179_77138_object91199:head# 2024-02-09T17:27:03.496 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.496 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f46277ea:::benchmark_data_smithi179_77138_object40924:head# 2024-02-09T17:27:03.496 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.496 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f462806b:::benchmark_data_smithi179_77138_object48720:head# 2024-02-09T17:27:03.496 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.496 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4628cce:::benchmark_data_smithi179_77138_object59895:head# 2024-02-09T17:27:03.496 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.496 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f46347ca:::benchmark_data_smithi179_77138_object118606:head# 2024-02-09T17:27:03.497 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.498 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4635e73:::benchmark_data_smithi179_77138_object45463:head# 2024-02-09T17:27:03.498 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.498 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4636f97:::benchmark_data_smithi179_77138_object89960:head# 2024-02-09T17:27:03.499 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.499 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4647e71:::benchmark_data_smithi179_77138_object40668:head# 2024-02-09T17:27:03.499 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.499 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f464c126:::benchmark_data_smithi179_77138_object101600:head# 2024-02-09T17:27:03.499 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.499 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4653772:::benchmark_data_smithi179_77138_object109351:head# 2024-02-09T17:27:03.499 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.499 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f46552b6:::benchmark_data_smithi179_77138_object66342:head# 2024-02-09T17:27:03.499 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.499 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f466e771:::benchmark_data_smithi179_77138_object43835:head# 2024-02-09T17:27:03.499 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.499 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f467e036:::benchmark_data_smithi179_77138_object68338:head# 2024-02-09T17:27:03.499 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.499 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4689963:::benchmark_data_smithi179_77138_object128417:head# 2024-02-09T17:27:03.499 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.500 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f469729c:::benchmark_data_smithi179_77138_object53197:head# 2024-02-09T17:27:03.501 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.501 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f46978cc:::benchmark_data_smithi179_77138_object78299:head# 2024-02-09T17:27:03.501 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.501 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f46b7355:::benchmark_data_smithi179_77138_object92243:head# 2024-02-09T17:27:03.501 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.501 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f46c4d39:::benchmark_data_smithi179_77138_object63130:head# 2024-02-09T17:27:03.501 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.501 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f46d7389:::benchmark_data_smithi179_77138_object20339:head# 2024-02-09T17:27:03.501 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.501 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f46d9e3a:::benchmark_data_smithi179_77138_object65723:head# 2024-02-09T17:27:03.501 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.501 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f46e2fe8:::benchmark_data_smithi179_77138_object47504:head# 2024-02-09T17:27:03.501 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.502 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f46eaf21:::benchmark_data_smithi179_77138_object61086:head# 2024-02-09T17:27:03.502 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.502 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f46f0651:::benchmark_data_smithi179_77138_object122614:head# 2024-02-09T17:27:03.502 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.502 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f46f24e3:::benchmark_data_smithi179_77138_object81305:head# 2024-02-09T17:27:03.502 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.503 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f46fac4e:::benchmark_data_smithi179_77138_object108548:head# 2024-02-09T17:27:03.503 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.503 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f47099c6:::benchmark_data_smithi179_77138_object24399:head# 2024-02-09T17:27:03.503 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.503 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4712fd4:::benchmark_data_smithi179_77138_object26907:head# 2024-02-09T17:27:03.504 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.504 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f47190c7:::benchmark_data_smithi179_77138_object27993:head# 2024-02-09T17:27:03.504 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.504 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f471d841:::benchmark_data_smithi179_77138_object131623:head# 2024-02-09T17:27:03.504 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.504 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4721be0:::benchmark_data_smithi179_77138_object40168:head# 2024-02-09T17:27:03.504 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.504 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f47257b4:::benchmark_data_smithi179_77138_object78554:head# 2024-02-09T17:27:03.504 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.504 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4730a67:::benchmark_data_smithi179_77138_object131956:head# 2024-02-09T17:27:03.504 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.504 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f473af39:::benchmark_data_smithi179_77138_object35527:head# 2024-02-09T17:27:03.504 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.504 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f473bbe2:::benchmark_data_smithi179_77138_object68738:head# 2024-02-09T17:27:03.504 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.505 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f474d693:::benchmark_data_smithi179_77138_object25443:head# 2024-02-09T17:27:03.505 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.506 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4751726:::benchmark_data_smithi179_77138_object91292:head# 2024-02-09T17:27:03.506 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.507 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4757aa1:::benchmark_data_smithi179_77138_object111201:head# 2024-02-09T17:27:03.507 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.507 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f475adbc:::benchmark_data_smithi179_77138_object26991:head# 2024-02-09T17:27:03.507 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.507 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f475c8ad:::benchmark_data_smithi179_77138_object58428:head# 2024-02-09T17:27:03.507 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.507 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f476fd1d:::benchmark_data_smithi179_77138_object102384:head# 2024-02-09T17:27:03.507 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.507 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f477382b:::benchmark_data_smithi179_77138_object99975:head# 2024-02-09T17:27:03.507 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.507 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f477c9da:::benchmark_data_smithi179_77138_object40634:head# 2024-02-09T17:27:03.507 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.507 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f47815ed:::benchmark_data_smithi179_77138_object103109:head# 2024-02-09T17:27:03.508 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.508 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4789e16:::benchmark_data_smithi179_77138_object24120:head# 2024-02-09T17:27:03.508 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.508 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f479c40f:::benchmark_data_smithi179_77138_object94151:head# 2024-02-09T17:27:03.508 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.508 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f47a4680:::benchmark_data_smithi179_77138_object104161:head# 2024-02-09T17:27:03.508 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.508 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f47a6eb7:::benchmark_data_smithi179_77138_object82603:head# 2024-02-09T17:27:03.508 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.508 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f47b4041:::benchmark_data_smithi179_77138_object83103:head# 2024-02-09T17:27:03.508 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.508 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f47d7be3:::benchmark_data_smithi179_77138_object26364:head# 2024-02-09T17:27:03.508 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.511 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f47e5583:::benchmark_data_smithi179_77138_object64634:head# 2024-02-09T17:27:03.511 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.511 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f48001da:::benchmark_data_smithi179_77138_object127729:head# 2024-02-09T17:27:03.511 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.511 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f480282f:::benchmark_data_smithi179_77138_object68236:head# 2024-02-09T17:27:03.511 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.511 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4804ab8:::benchmark_data_smithi179_77138_object57065:head# 2024-02-09T17:27:03.511 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.511 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f481ffb7:::benchmark_data_smithi179_77138_object121231:head# 2024-02-09T17:27:03.511 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.512 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4825a7d:::benchmark_data_smithi179_77138_object48698:head# 2024-02-09T17:27:03.512 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.512 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f482c4d7:::benchmark_data_smithi179_77138_object40753:head# 2024-02-09T17:27:03.512 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.512 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4846157:::benchmark_data_smithi179_77138_object29887:head# 2024-02-09T17:27:03.512 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.512 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f484d366:::benchmark_data_smithi179_77138_object121449:head# 2024-02-09T17:27:03.512 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.512 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f485fac4:::benchmark_data_smithi179_77138_object64169:head# 2024-02-09T17:27:03.512 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.512 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4860992:::benchmark_data_smithi179_77138_object122023:head# 2024-02-09T17:27:03.512 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.512 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f48661a6:::benchmark_data_smithi179_77138_object68150:head# 2024-02-09T17:27:03.512 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.513 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f487120b:::benchmark_data_smithi179_77138_object72023:head# 2024-02-09T17:27:03.513 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.513 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f48826b0:::benchmark_data_smithi179_77138_object55810:head# 2024-02-09T17:27:03.513 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.513 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f48855b5:::benchmark_data_smithi179_77138_object57565:head# 2024-02-09T17:27:03.513 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.513 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f48859cf:::benchmark_data_smithi179_77138_object92783:head# 2024-02-09T17:27:03.513 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.513 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f48862dc:::benchmark_data_smithi179_77138_object37533:head# 2024-02-09T17:27:03.513 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.516 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f489811d:::benchmark_data_smithi179_77138_object51734:head# 2024-02-09T17:27:03.516 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.516 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f489994e:::benchmark_data_smithi179_77138_object40742:head# 2024-02-09T17:27:03.516 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.516 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f48a0fc2:::benchmark_data_smithi179_77138_object92256:head# 2024-02-09T17:27:03.516 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.516 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f48a10cd:::benchmark_data_smithi179_77138_object121862:head# 2024-02-09T17:27:03.516 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.516 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f48aac1d:::benchmark_data_smithi179_77138_object76956:head# 2024-02-09T17:27:03.516 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.517 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f48c7435:::benchmark_data_smithi179_77138_object80255:head# 2024-02-09T17:27:03.517 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.517 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f48ca8b0:::benchmark_data_smithi179_77138_object123932:head# 2024-02-09T17:27:03.517 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.517 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f48d55a9:::benchmark_data_smithi179_77138_object115332:head# 2024-02-09T17:27:03.517 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.517 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f48dad69:::benchmark_data_smithi179_77138_object115842:head# 2024-02-09T17:27:03.517 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.517 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f48dc358:::benchmark_data_smithi179_77138_object69671:head# 2024-02-09T17:27:03.517 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.517 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f48e6f45:::benchmark_data_smithi179_77138_object107094:head# 2024-02-09T17:27:03.517 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.517 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f48f8101:::benchmark_data_smithi179_77138_object85815:head# 2024-02-09T17:27:03.517 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.518 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f48fa9b4:::benchmark_data_smithi179_77138_object99191:head# 2024-02-09T17:27:03.518 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.518 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f490f5b6:::benchmark_data_smithi179_77138_object97183:head# 2024-02-09T17:27:03.518 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.518 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4912c50:::benchmark_data_smithi179_77138_object89065:head# 2024-02-09T17:27:03.518 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.518 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f492cd98:::benchmark_data_smithi179_77138_object89746:head# 2024-02-09T17:27:03.518 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.518 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4944bc7:::benchmark_data_smithi179_77138_object30918:head# 2024-02-09T17:27:03.518 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.518 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f494c94f:::benchmark_data_smithi179_77138_object63570:head# 2024-02-09T17:27:03.518 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.522 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f494f71a:::benchmark_data_smithi179_77138_object128145:head# 2024-02-09T17:27:03.523 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.523 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4955399:::benchmark_data_smithi179_77138_object78994:head# 2024-02-09T17:27:03.523 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.523 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f49570a6:::benchmark_data_smithi179_77138_object132389:head# 2024-02-09T17:27:03.523 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.523 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4960d9d:::benchmark_data_smithi179_77138_object84077:head# 2024-02-09T17:27:03.523 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.523 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f496cb08:::benchmark_data_smithi179_77138_object37459:head# 2024-02-09T17:27:03.523 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.523 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f496d002:::benchmark_data_smithi179_77138_object69957:head# 2024-02-09T17:27:03.523 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.523 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f497479d:::benchmark_data_smithi179_77138_object52061:head# 2024-02-09T17:27:03.523 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.523 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f497588b:::benchmark_data_smithi179_77138_object44571:head# 2024-02-09T17:27:03.524 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.524 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f497d8c9:::benchmark_data_smithi179_77138_object122741:head# 2024-02-09T17:27:03.524 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.524 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f49834f4:::benchmark_data_smithi179_77138_object88342:head# 2024-02-09T17:27:03.524 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.524 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4986b0d:::benchmark_data_smithi179_77138_object133347:head# 2024-02-09T17:27:03.524 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.524 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f499018f:::benchmark_data_smithi179_77138_object129186:head# 2024-02-09T17:27:03.524 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.524 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f49a8d6e:::benchmark_data_smithi179_77138_object131786:head# 2024-02-09T17:27:03.524 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.524 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f49b0120:::benchmark_data_smithi179_77138_object44858:head# 2024-02-09T17:27:03.525 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.525 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f49bebdd:::benchmark_data_smithi179_77138_object114438:head# 2024-02-09T17:27:03.525 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.525 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f49d0154:::benchmark_data_smithi179_77138_object79728:head# 2024-02-09T17:27:03.525 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.525 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f49d53de:::benchmark_data_smithi179_77138_object96856:head# 2024-02-09T17:27:03.525 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.525 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f49eaaf8:::benchmark_data_smithi179_77138_object61186:head# 2024-02-09T17:27:03.525 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.525 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f49f37fb:::benchmark_data_smithi179_77138_object117889:head# 2024-02-09T17:27:03.525 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.525 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f49f55bb:::benchmark_data_smithi179_77138_object35310:head# 2024-02-09T17:27:03.526 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.526 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f49f8b2f:::benchmark_data_smithi179_77138_object121680:head# 2024-02-09T17:27:03.526 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.526 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f49fe080:::benchmark_data_smithi179_77138_object103498:head# 2024-02-09T17:27:03.526 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.526 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4a05357:::benchmark_data_smithi179_77138_object89920:head# 2024-02-09T17:27:03.526 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.526 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4a0961b:::benchmark_data_smithi179_77138_object101887:head# 2024-02-09T17:27:03.526 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.526 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4a15a8e:::benchmark_data_smithi179_77138_object23060:head# 2024-02-09T17:27:03.526 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.526 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4a1fb7b:::benchmark_data_smithi179_77138_object82527:head# 2024-02-09T17:27:03.526 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.526 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4a27a26:::benchmark_data_smithi179_77138_object59962:head# 2024-02-09T17:27:03.527 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.542 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4a28b4b:::benchmark_data_smithi179_77138_object95675:head# 2024-02-09T17:27:03.542 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.542 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4a2acf8:::benchmark_data_smithi179_77138_object37937:head# 2024-02-09T17:27:03.542 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.542 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4a2ddf8:::benchmark_data_smithi179_77138_object31340:head# 2024-02-09T17:27:03.542 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.542 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4a34605:::benchmark_data_smithi179_77138_object88341:head# 2024-02-09T17:27:03.542 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.542 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4a3b526:::benchmark_data_smithi179_77138_object118213:head# 2024-02-09T17:27:03.543 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.543 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4a3bc94:::benchmark_data_smithi179_77138_object82137:head# 2024-02-09T17:27:03.543 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.543 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4a3e33e:::benchmark_data_smithi179_77138_object48966:head# 2024-02-09T17:27:03.543 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.543 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4a4928a:::benchmark_data_smithi179_77138_object80212:head# 2024-02-09T17:27:03.543 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.543 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4a4bc2b:::benchmark_data_smithi179_77138_object129500:head# 2024-02-09T17:27:03.543 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.543 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4a4f947:::benchmark_data_smithi179_77138_object54742:head# 2024-02-09T17:27:03.543 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.543 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4a5ef5d:::benchmark_data_smithi179_77138_object107326:head# 2024-02-09T17:27:03.544 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.544 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4a62b20:::benchmark_data_smithi179_77138_object111911:head# 2024-02-09T17:27:03.544 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.544 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4a830b4:::benchmark_data_smithi179_77138_object89378:head# 2024-02-09T17:27:03.544 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.544 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4a864b2:::benchmark_data_smithi179_77138_object62687:head# 2024-02-09T17:27:03.544 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.544 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4a8c1c6:::benchmark_data_smithi179_77138_object24832:head# 2024-02-09T17:27:03.544 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.544 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4a9d4ca:::benchmark_data_smithi179_77138_object42426:head# 2024-02-09T17:27:03.544 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.545 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4ab0d16:::benchmark_data_smithi179_77138_object105444:head# 2024-02-09T17:27:03.545 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.545 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4ab1a85:::benchmark_data_smithi179_77138_object125315:head# 2024-02-09T17:27:03.545 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.545 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4ab1c8b:::benchmark_data_smithi179_77138_object41588:head# 2024-02-09T17:27:03.545 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.545 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4ac8a26:::benchmark_data_smithi179_77138_object104569:head# 2024-02-09T17:27:03.545 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.545 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4acbf73:::benchmark_data_smithi179_77138_object35690:head# 2024-02-09T17:27:03.545 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.545 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4ad0f44:::benchmark_data_smithi179_77138_object41562:head# 2024-02-09T17:27:03.545 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.546 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4ad680d:::benchmark_data_smithi179_77138_object55398:head# 2024-02-09T17:27:03.546 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.546 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4adc594:::benchmark_data_smithi179_77138_object83206:head# 2024-02-09T17:27:03.546 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.546 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4adead1:::benchmark_data_smithi179_77138_object55892:head# 2024-02-09T17:27:03.546 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.546 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4ae8c87:::benchmark_data_smithi179_77138_object48955:head# 2024-02-09T17:27:03.546 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.546 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4af378e:::benchmark_data_smithi179_77138_object78734:head# 2024-02-09T17:27:03.546 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.546 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4af83f9:::benchmark_data_smithi179_77138_object128981:head# 2024-02-09T17:27:03.546 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.547 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4afe3c1:::benchmark_data_smithi179_77138_object58510:head# 2024-02-09T17:27:03.547 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.547 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4b00b79:::benchmark_data_smithi179_77138_object73496:head# 2024-02-09T17:27:03.547 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.547 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4b03f61:::benchmark_data_smithi179_77138_object99994:head# 2024-02-09T17:27:03.547 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.547 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4b04c39:::benchmark_data_smithi179_77138_object115139:head# 2024-02-09T17:27:03.547 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.547 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4b0b41a:::benchmark_data_smithi179_77138_object84589:head# 2024-02-09T17:27:03.547 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.547 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4b0fefa:::benchmark_data_smithi179_77138_object30610:head# 2024-02-09T17:27:03.547 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.547 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4b13aeb:::benchmark_data_smithi179_77138_object94317:head# 2024-02-09T17:27:03.548 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.548 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4b227a6:::benchmark_data_smithi179_77138_object25280:head# 2024-02-09T17:27:03.548 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.548 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4b26952:::benchmark_data_smithi179_77138_object26863:head# 2024-02-09T17:27:03.548 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.548 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4b285db:::benchmark_data_smithi179_77138_object97484:head# 2024-02-09T17:27:03.548 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.548 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4b3aa35:::benchmark_data_smithi179_77138_object91890:head# 2024-02-09T17:27:03.548 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.548 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4b3ea28:::benchmark_data_smithi179_77138_object126295:head# 2024-02-09T17:27:03.548 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.548 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4b3f127:::benchmark_data_smithi179_77138_object91309:head# 2024-02-09T17:27:03.549 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.549 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4b5bdaf:::benchmark_data_smithi179_77138_object35583:head# 2024-02-09T17:27:03.551 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.551 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4b5da44:::benchmark_data_smithi179_77138_object114687:head# 2024-02-09T17:27:03.551 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.551 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4b6e1e1:::benchmark_data_smithi179_77138_object97520:head# 2024-02-09T17:27:03.551 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.551 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4b6effa:::benchmark_data_smithi179_77138_object103659:head# 2024-02-09T17:27:03.551 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.551 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4b72309:::benchmark_data_smithi179_77138_object74373:head# 2024-02-09T17:27:03.551 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.551 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4b72a1b:::benchmark_data_smithi179_77138_object20611:head# 2024-02-09T17:27:03.552 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.552 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4b758a5:::benchmark_data_smithi179_77138_object129135:head# 2024-02-09T17:27:03.552 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.552 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4b75baf:::benchmark_data_smithi179_77138_object115719:head# 2024-02-09T17:27:03.552 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.552 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4b86361:::benchmark_data_smithi179_77138_object65111:head# 2024-02-09T17:27:03.552 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.552 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4b8c542:::benchmark_data_smithi179_77138_object89648:head# 2024-02-09T17:27:03.552 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.552 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4b8ded5:::benchmark_data_smithi179_77138_object67412:head# 2024-02-09T17:27:03.552 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.552 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4b95bb7:::benchmark_data_smithi179_77138_object91869:head# 2024-02-09T17:27:03.552 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.552 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4b9860d:::benchmark_data_smithi179_77138_object50730:head# 2024-02-09T17:27:03.553 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.553 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4ba6be3:::benchmark_data_smithi179_77138_object99740:head# 2024-02-09T17:27:03.553 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.553 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4ba7297:::benchmark_data_smithi179_77138_object33605:head# 2024-02-09T17:27:03.553 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.553 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4ba7dd3:::benchmark_data_smithi179_77138_object64383:head# 2024-02-09T17:27:03.553 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.553 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4baa07c:::benchmark_data_smithi179_77138_object91667:head# 2024-02-09T17:27:03.553 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.553 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4bb486c:::benchmark_data_smithi179_77138_object113565:head# 2024-02-09T17:27:03.553 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.553 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4bb98f7:::benchmark_data_smithi179_77138_object129605:head# 2024-02-09T17:27:03.553 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.553 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4bbf4ef:::benchmark_data_smithi179_77138_object130414:head# 2024-02-09T17:27:03.554 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.554 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4bcab05:::benchmark_data_smithi179_77138_object128138:head# 2024-02-09T17:27:03.554 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.554 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4bd47e3:::benchmark_data_smithi179_77138_object77828:head# 2024-02-09T17:27:03.554 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.554 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4bdb648:::benchmark_data_smithi179_77138_object33415:head# 2024-02-09T17:27:03.554 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.554 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4be5ae9:::benchmark_data_smithi179_77138_object107125:head# 2024-02-09T17:27:03.554 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.554 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4be949a:::benchmark_data_smithi179_77138_object101696:head# 2024-02-09T17:27:03.554 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.554 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4be9e45:::benchmark_data_smithi179_77138_object55910:head# 2024-02-09T17:27:03.554 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.554 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4bff853:::benchmark_data_smithi179_77138_object77627:head# 2024-02-09T17:27:03.554 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.554 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4c0058c:::benchmark_data_smithi179_77138_object98657:head# 2024-02-09T17:27:03.555 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.555 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4c01a8f:::benchmark_data_smithi179_77138_object20365:head# 2024-02-09T17:27:03.555 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.555 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4c0ff1b:::benchmark_data_smithi179_77138_object36932:head# 2024-02-09T17:27:03.555 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.555 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4c170b9:::benchmark_data_smithi179_77138_object54061:head# 2024-02-09T17:27:03.555 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.555 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4c19149:::benchmark_data_smithi179_77138_object96551:head# 2024-02-09T17:27:03.555 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.555 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4c3b222:::benchmark_data_smithi179_77138_object68063:head# 2024-02-09T17:27:03.555 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.555 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4c45563:::benchmark_data_smithi179_77138_object110391:head# 2024-02-09T17:27:03.555 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.555 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4c45a57:::benchmark_data_smithi179_77138_object104698:head# 2024-02-09T17:27:03.555 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.555 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4c46648:::benchmark_data_smithi179_77138_object62366:head# 2024-02-09T17:27:03.556 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.556 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4c6beb8:::benchmark_data_smithi179_77138_object89701:head# 2024-02-09T17:27:03.556 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.556 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4c82daa:::benchmark_data_smithi179_77138_object33214:head# 2024-02-09T17:27:03.556 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.566 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4c920a8:::benchmark_data_smithi179_77138_object90506:head# 2024-02-09T17:27:03.566 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.566 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4c96335:::benchmark_data_smithi179_77138_object129673:head# 2024-02-09T17:27:03.566 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.566 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4c9cf72:::benchmark_data_smithi179_77138_object22608:head# 2024-02-09T17:27:03.566 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.566 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4caf2d1:::benchmark_data_smithi179_77138_object82906:head# 2024-02-09T17:27:03.566 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.567 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4cb0049:::benchmark_data_smithi179_77138_object104126:head# 2024-02-09T17:27:03.567 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.567 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4cd18c7:::benchmark_data_smithi179_77138_object26714:head# 2024-02-09T17:27:03.567 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.567 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4ce07e7:::benchmark_data_smithi179_77138_object75433:head# 2024-02-09T17:27:03.567 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.567 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4cf376f:::benchmark_data_smithi179_77138_object98475:head# 2024-02-09T17:27:03.567 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.567 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4cf4210:::benchmark_data_smithi179_77138_object60261:head# 2024-02-09T17:27:03.567 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.567 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4cfa9d1:::benchmark_data_smithi179_77138_object46281:head# 2024-02-09T17:27:03.567 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.567 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4cfce15:::benchmark_data_smithi179_77138_object55630:head# 2024-02-09T17:27:03.568 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.568 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4d0831d:::benchmark_data_smithi179_77138_object31638:head# 2024-02-09T17:27:03.568 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.568 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4d0cb0f:::benchmark_data_smithi179_77138_object54699:head# 2024-02-09T17:27:03.568 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.568 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4d1925b:::benchmark_data_smithi179_77138_object29694:head# 2024-02-09T17:27:03.568 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.568 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4d26711:::benchmark_data_smithi179_77138_object52577:head# 2024-02-09T17:27:03.568 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.568 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4d2aed9:::benchmark_data_smithi179_77138_object38267:head# 2024-02-09T17:27:03.568 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.568 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4d2ea3c:::benchmark_data_smithi179_77138_object30471:head# 2024-02-09T17:27:03.568 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.569 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4d39310:::benchmark_data_smithi179_77138_object98822:head# 2024-02-09T17:27:03.569 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.569 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4d3a1a8:::benchmark_data_smithi179_77138_object76842:head# 2024-02-09T17:27:03.569 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.569 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4d3b510:::benchmark_data_smithi179_77138_object34132:head# 2024-02-09T17:27:03.569 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.569 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4d4d30d:::benchmark_data_smithi179_77138_object99012:head# 2024-02-09T17:27:03.569 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.569 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4d550be:::benchmark_data_smithi179_77138_object34836:head# 2024-02-09T17:27:03.569 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.569 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4d595f1:::benchmark_data_smithi179_77138_object62943:head# 2024-02-09T17:27:03.569 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.570 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4d5af9f:::benchmark_data_smithi179_77138_object63814:head# 2024-02-09T17:27:03.570 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.570 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4d5afcb:::benchmark_data_smithi179_77138_object34206:head# 2024-02-09T17:27:03.570 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.570 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4d5b5b2:::benchmark_data_smithi179_77138_object111748:head# 2024-02-09T17:27:03.570 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.570 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4d63853:::benchmark_data_smithi179_77138_object23126:head# 2024-02-09T17:27:03.570 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.570 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4d67eed:::benchmark_data_smithi179_77138_object106858:head# 2024-02-09T17:27:03.570 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.570 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4d700fe:::benchmark_data_smithi179_77138_object78228:head# 2024-02-09T17:27:03.570 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.570 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4d7648c:::benchmark_data_smithi179_77138_object130073:head# 2024-02-09T17:27:03.570 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.571 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4d783c7:::benchmark_data_smithi179_77138_object128720:head# 2024-02-09T17:27:03.571 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.571 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4d7e875:::benchmark_data_smithi179_77138_object88678:head# 2024-02-09T17:27:03.571 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.571 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4d8c1be:::benchmark_data_smithi179_77138_object89898:head# 2024-02-09T17:27:03.571 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.571 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4d9fc1e:::benchmark_data_smithi179_77138_object30013:head# 2024-02-09T17:27:03.571 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.571 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4da7f1e:::benchmark_data_smithi179_77138_object106777:head# 2024-02-09T17:27:03.571 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.571 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4dacc2a:::benchmark_data_smithi179_77138_object38930:head# 2024-02-09T17:27:03.571 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.572 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4db1d84:::benchmark_data_smithi179_77138_object130955:head# 2024-02-09T17:27:03.572 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.572 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4db1f25:::benchmark_data_smithi179_77138_object47756:head# 2024-02-09T17:27:03.572 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.572 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4db3c9c:::benchmark_data_smithi179_77138_object46601:head# 2024-02-09T17:27:03.572 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.572 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4db6e7a:::benchmark_data_smithi179_77138_object100521:head# 2024-02-09T17:27:03.572 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.572 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4dc1779:::benchmark_data_smithi179_77138_object83281:head# 2024-02-09T17:27:03.572 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.572 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4dca59e:::benchmark_data_smithi179_77138_object133096:head# 2024-02-09T17:27:03.572 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.572 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4dcc671:::benchmark_data_smithi179_77138_object69807:head# 2024-02-09T17:27:03.572 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.573 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4dd00aa:::benchmark_data_smithi179_77138_object57860:head# 2024-02-09T17:27:03.573 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.573 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4dd2b33:::benchmark_data_smithi179_77138_object34712:head# 2024-02-09T17:27:03.573 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.573 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4dda4ac:::benchmark_data_smithi179_77138_object41178:head# 2024-02-09T17:27:03.573 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.573 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4ddd6e4:::benchmark_data_smithi179_77138_object132761:head# 2024-02-09T17:27:03.573 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.573 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4de09f8:::benchmark_data_smithi179_77138_object78523:head# 2024-02-09T17:27:03.573 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.573 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4de7fee:::benchmark_data_smithi179_77138_object85028:head# 2024-02-09T17:27:03.573 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.573 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4df1216:::benchmark_data_smithi179_77138_object58898:head# 2024-02-09T17:27:03.573 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.574 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4dfbbdd:::benchmark_data_smithi179_77138_object48136:head# 2024-02-09T17:27:03.574 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.574 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4e03155:::benchmark_data_smithi179_77138_object76530:head# 2024-02-09T17:27:03.574 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.574 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4e0c4df:::benchmark_data_smithi179_77138_object49313:head# 2024-02-09T17:27:03.574 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.574 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4e0f7d6:::benchmark_data_smithi179_77138_object20119:head# 2024-02-09T17:27:03.574 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.574 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4e0f998:::benchmark_data_smithi179_77138_object96813:head# 2024-02-09T17:27:03.574 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.574 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4e229cd:::benchmark_data_smithi179_77138_object50664:head# 2024-02-09T17:27:03.574 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.574 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4e2737a:::benchmark_data_smithi179_77138_object41509:head# 2024-02-09T17:27:03.575 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.575 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4e2e1da:::benchmark_data_smithi179_77138_object70159:head# 2024-02-09T17:27:03.575 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.575 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4e30760:::benchmark_data_smithi179_77138_object72672:head# 2024-02-09T17:27:03.575 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.575 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4e4c42f:::benchmark_data_smithi179_77138_object65703:head# 2024-02-09T17:27:03.575 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.575 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4e4ee99:::benchmark_data_smithi179_77138_object90093:head# 2024-02-09T17:27:03.575 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.575 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4e50161:::benchmark_data_smithi179_77138_object70277:head# 2024-02-09T17:27:03.575 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.575 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4e5795e:::benchmark_data_smithi179_77138_object19404:head# 2024-02-09T17:27:03.575 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.575 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4e5dd15:::benchmark_data_smithi179_77138_object95326:head# 2024-02-09T17:27:03.575 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.576 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4e612ac:::benchmark_data_smithi179_77138_object25967:head# 2024-02-09T17:27:03.576 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.576 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4e62ec7:::benchmark_data_smithi179_77138_object110768:head# 2024-02-09T17:27:03.576 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.576 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4e64f66:::benchmark_data_smithi179_77138_object90853:head# 2024-02-09T17:27:03.576 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.576 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4e7129a:::benchmark_data_smithi179_77138_object22354:head# 2024-02-09T17:27:03.576 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.576 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4e99d01:::benchmark_data_smithi179_77138_object86769:head# 2024-02-09T17:27:03.576 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.576 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4ea8871:::benchmark_data_smithi179_77138_object110756:head# 2024-02-09T17:27:03.576 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.576 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4eb78cf:::benchmark_data_smithi179_77138_object89364:head# 2024-02-09T17:27:03.577 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.577 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4ebe356:::benchmark_data_smithi179_77138_object118536:head# 2024-02-09T17:27:03.577 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.577 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4ed0bed:::benchmark_data_smithi179_77138_object98604:head# 2024-02-09T17:27:03.577 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.577 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4ed1f6b:::benchmark_data_smithi179_77138_object86797:head# 2024-02-09T17:27:03.577 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.577 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4ed25e5:::benchmark_data_smithi179_77138_object56833:head# 2024-02-09T17:27:03.577 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.577 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4ed5715:::benchmark_data_smithi179_77138_object22080:head# 2024-02-09T17:27:03.577 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.577 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4ee4295:::benchmark_data_smithi179_77138_object67378:head# 2024-02-09T17:27:03.577 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.577 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4ee4395:::benchmark_data_smithi179_77138_object93858:head# 2024-02-09T17:27:03.577 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.578 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4eea2ba:::benchmark_data_smithi179_77138_object126296:head# 2024-02-09T17:27:03.578 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.578 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4eec3e4:::benchmark_data_smithi179_77138_object51613:head# 2024-02-09T17:27:03.578 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.578 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4ef2976:::benchmark_data_smithi179_77138_object24401:head# 2024-02-09T17:27:03.578 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.578 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4ef5f2b:::benchmark_data_smithi179_77138_object102367:head# 2024-02-09T17:27:03.578 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.578 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4ef9f91:::benchmark_data_smithi179_77138_object63736:head# 2024-02-09T17:27:03.578 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.578 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4efc8df:::benchmark_data_smithi179_77138_object46955:head# 2024-02-09T17:27:03.578 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.578 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4efcaac:::benchmark_data_smithi179_77138_object110765:head# 2024-02-09T17:27:03.578 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.578 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4efd10b:::benchmark_data_smithi179_77138_object53782:head# 2024-02-09T17:27:03.579 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.579 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4f0108e:::benchmark_data_smithi179_77138_object93384:head# 2024-02-09T17:27:03.579 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.579 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4f1d9d3:::benchmark_data_smithi179_77138_object92571:head# 2024-02-09T17:27:03.579 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.579 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4f3400b:::benchmark_data_smithi179_77138_object36550:head# 2024-02-09T17:27:03.579 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.579 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4f35caf:::benchmark_data_smithi179_77138_object54046:head# 2024-02-09T17:27:03.579 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.580 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:27:03.580 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-09T17:27:03.580 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-02-09T17:27:03.580 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:27:03.580 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:27:03.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:03.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:27:03.590 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4f3ae20:::benchmark_data_smithi179_77138_object67439:head# 2024-02-09T17:27:03.590 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.590 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4f3c99c:::benchmark_data_smithi179_77138_object44268:head# 2024-02-09T17:27:03.590 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.590 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4f40e5a:::benchmark_data_smithi179_77138_object61669:head# 2024-02-09T17:27:03.591 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.591 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4f45008:::benchmark_data_smithi179_77138_object44401:head# 2024-02-09T17:27:03.591 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.591 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4f492dc:::benchmark_data_smithi179_77138_object75914:head# 2024-02-09T17:27:03.591 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.591 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4f4e40a:::benchmark_data_smithi179_77138_object107667:head# 2024-02-09T17:27:03.591 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.591 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4f4ec8d:::benchmark_data_smithi179_77138_object131388:head# 2024-02-09T17:27:03.591 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.591 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4f6bf18:::benchmark_data_smithi179_77138_object63953:head# 2024-02-09T17:27:03.591 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.591 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4f6d719:::benchmark_data_smithi179_77138_object111058:head# 2024-02-09T17:27:03.591 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.591 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4f6e33c:::benchmark_data_smithi179_77138_object37221:head# 2024-02-09T17:27:03.592 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.592 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4f6f655:::benchmark_data_smithi179_77138_object110118:head# 2024-02-09T17:27:03.592 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.592 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4f7757e:::benchmark_data_smithi179_77138_object57717:head# 2024-02-09T17:27:03.592 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.592 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4f7fb6a:::benchmark_data_smithi179_77138_object88012:head# 2024-02-09T17:27:03.592 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.592 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4f84bd2:::benchmark_data_smithi179_77138_object22546:head# 2024-02-09T17:27:03.592 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.592 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4f8639a:::benchmark_data_smithi179_77138_object28373:head# 2024-02-09T17:27:03.592 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.592 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4f95acb:::benchmark_data_smithi179_77138_object91739:head# 2024-02-09T17:27:03.592 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.592 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4f99f06:::benchmark_data_smithi179_77138_object88030:head# 2024-02-09T17:27:03.593 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.593 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4fb09e1:::benchmark_data_smithi179_77138_object125756:head# 2024-02-09T17:27:03.593 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.593 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4fbb5a8:::benchmark_data_smithi179_77138_object109866:head# 2024-02-09T17:27:03.593 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.593 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4fcdcf1:::benchmark_data_smithi179_77138_object67898:head# 2024-02-09T17:27:03.593 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.593 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4fcfc3b:::benchmark_data_smithi179_77138_object132575:head# 2024-02-09T17:27:03.593 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.593 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4fd8c06:::benchmark_data_smithi179_77138_object24688:head# 2024-02-09T17:27:03.593 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.593 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4feba50:::benchmark_data_smithi179_77138_object81221:head# 2024-02-09T17:27:03.593 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.594 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4fef2ff:::benchmark_data_smithi179_77138_object22081:head# 2024-02-09T17:27:03.594 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.594 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4ff1cd5:::benchmark_data_smithi179_77138_object39710:head# 2024-02-09T17:27:03.594 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.594 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f4ff7cdc:::benchmark_data_smithi179_77138_object126868:head# 2024-02-09T17:27:03.594 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.594 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f500bc3a:::benchmark_data_smithi179_77138_object63905:head# 2024-02-09T17:27:03.594 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.594 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f500cda7:::benchmark_data_smithi179_77138_object60584:head# 2024-02-09T17:27:03.594 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.594 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f500f98f:::benchmark_data_smithi179_77138_object93789:head# 2024-02-09T17:27:03.594 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.594 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f501ec43:::benchmark_data_smithi179_77138_object84746:head# 2024-02-09T17:27:03.594 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.595 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5025aac:::benchmark_data_smithi179_77138_object116297:head# 2024-02-09T17:27:03.595 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.595 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5031953:::benchmark_data_smithi179_77138_object61446:head# 2024-02-09T17:27:03.595 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.595 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f504048f:::benchmark_data_smithi179_77138_object27876:head# 2024-02-09T17:27:03.595 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.595 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f504840b:::benchmark_data_smithi179_77138_object77061:head# 2024-02-09T17:27:03.595 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.595 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f505162d:::benchmark_data_smithi179_77138_object51739:head# 2024-02-09T17:27:03.595 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.595 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5055147:::benchmark_data_smithi179_77138_object84095:head# 2024-02-09T17:27:03.595 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.595 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f505ef7b:::benchmark_data_smithi179_77138_object90539:head# 2024-02-09T17:27:03.595 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.596 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5060311:::benchmark_data_smithi179_77138_object84719:head# 2024-02-09T17:27:03.596 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.596 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f506756a:::benchmark_data_smithi179_77138_object89846:head# 2024-02-09T17:27:03.596 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.596 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f50927f5:::benchmark_data_smithi179_77138_object85765:head# 2024-02-09T17:27:03.596 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.596 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5095f4c:::benchmark_data_smithi179_77138_object70998:head# 2024-02-09T17:27:03.596 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.596 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f509d8fb:::benchmark_data_smithi179_77138_object88846:head# 2024-02-09T17:27:03.596 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.596 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f50a5fcc:::benchmark_data_smithi179_77138_object116593:head# 2024-02-09T17:27:03.596 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.596 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f50a6a97:::benchmark_data_smithi179_77138_object20597:head# 2024-02-09T17:27:03.596 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.596 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f50a6d6d:::benchmark_data_smithi179_77138_object101883:head# 2024-02-09T17:27:03.597 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.597 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f50accd0:::benchmark_data_smithi179_77138_object35764:head# 2024-02-09T17:27:03.597 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.597 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f50b7cde:::benchmark_data_smithi179_77138_object114035:head# 2024-02-09T17:27:03.597 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.597 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f50bff95:::benchmark_data_smithi179_77138_object108459:head# 2024-02-09T17:27:03.597 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.597 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f50c3861:::benchmark_data_smithi179_77138_object65337:head# 2024-02-09T17:27:03.597 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.597 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f50cfaa9:::benchmark_data_smithi179_77138_object55798:head# 2024-02-09T17:27:03.597 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.597 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f50d4e18:::benchmark_data_smithi179_77138_object57527:head# 2024-02-09T17:27:03.597 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.597 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f50ea456:::benchmark_data_smithi179_77138_object41995:head# 2024-02-09T17:27:03.598 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.598 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f50ed249:::benchmark_data_smithi179_77138_object38569:head# 2024-02-09T17:27:03.598 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.598 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f50f7594:::benchmark_data_smithi179_77138_object33580:head# 2024-02-09T17:27:03.598 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.598 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f50fa4db:::benchmark_data_smithi179_77138_object38226:head# 2024-02-09T17:27:03.598 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.598 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f50ff66b:::benchmark_data_smithi179_77138_object94858:head# 2024-02-09T17:27:03.598 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.598 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f51062c8:::benchmark_data_smithi179_77138_object97591:head# 2024-02-09T17:27:03.598 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.598 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f512ded6:::benchmark_data_smithi179_77138_object28493:head# 2024-02-09T17:27:03.598 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.598 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f51359d1:::benchmark_data_smithi179_77138_object42687:head# 2024-02-09T17:27:03.598 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.598 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f51455d5:::benchmark_data_smithi179_77138_object102495:head# 2024-02-09T17:27:03.598 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.599 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f514ec7a:::benchmark_data_smithi179_77138_object48827:head# 2024-02-09T17:27:03.599 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.599 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5154e2b:::benchmark_data_smithi179_77138_object104971:head# 2024-02-09T17:27:03.599 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.599 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5163615:::benchmark_data_smithi179_77138_object69456:head# 2024-02-09T17:27:03.599 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.599 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f516a601:::benchmark_data_smithi179_77138_object121904:head# 2024-02-09T17:27:03.599 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.599 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5175a3c:::benchmark_data_smithi179_77138_object96946:head# 2024-02-09T17:27:03.599 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.599 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5184a0d:::benchmark_data_smithi179_77138_object95527:head# 2024-02-09T17:27:03.599 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.599 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f518f840:::benchmark_data_smithi179_77138_object48172:head# 2024-02-09T17:27:03.599 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.599 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5192089:::benchmark_data_smithi179_77138_object83621:head# 2024-02-09T17:27:03.599 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.600 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5195277:::benchmark_data_smithi179_77138_object103549:head# 2024-02-09T17:27:03.600 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.600 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f519af88:::benchmark_data_smithi179_77138_object51420:head# 2024-02-09T17:27:03.600 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.600 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f51bda34:::benchmark_data_smithi179_77138_object99413:head# 2024-02-09T17:27:03.600 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.600 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f51be0ff:::benchmark_data_smithi179_77138_object102288:head# 2024-02-09T17:27:03.600 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.600 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f51bfe6b:::benchmark_data_smithi179_77138_object125082:head# 2024-02-09T17:27:03.600 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.610 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f51c006e:::benchmark_data_smithi179_77138_object53574:head# 2024-02-09T17:27:03.610 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.610 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f51c37b5:::benchmark_data_smithi179_77138_object80425:head# 2024-02-09T17:27:03.610 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.610 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f51d9ebb:::benchmark_data_smithi179_77138_object106469:head# 2024-02-09T17:27:03.611 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.611 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f51e254f:::benchmark_data_smithi179_77138_object74069:head# 2024-02-09T17:27:03.611 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.611 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f51e2edb:::benchmark_data_smithi179_77138_object49661:head# 2024-02-09T17:27:03.611 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.611 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f51ed20e:::benchmark_data_smithi179_77138_object27610:head# 2024-02-09T17:27:03.611 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.611 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f51fcabf:::benchmark_data_smithi179_77138_object67471:head# 2024-02-09T17:27:03.611 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.611 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f52020bf:::benchmark_data_smithi179_77138_object123064:head# 2024-02-09T17:27:03.611 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.611 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f521399a:::benchmark_data_smithi179_77138_object103566:head# 2024-02-09T17:27:03.611 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.611 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5218213:::benchmark_data_smithi179_77138_object87260:head# 2024-02-09T17:27:03.612 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.612 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f521c1fd:::benchmark_data_smithi179_77138_object93079:head# 2024-02-09T17:27:03.612 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.612 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5223536:::benchmark_data_smithi179_77138_object104800:head# 2024-02-09T17:27:03.612 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.612 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f52291f4:::benchmark_data_smithi179_77138_object119332:head# 2024-02-09T17:27:03.612 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.612 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f522f2b3:::benchmark_data_smithi179_77138_object88128:head# 2024-02-09T17:27:03.612 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.612 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f523bab7:::benchmark_data_smithi179_77138_object83637:head# 2024-02-09T17:27:03.612 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.612 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f523f99a:::benchmark_data_smithi179_77138_object84794:head# 2024-02-09T17:27:03.612 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.613 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5241d78:::benchmark_data_smithi179_77138_object55595:head# 2024-02-09T17:27:03.613 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.613 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f524980f:::benchmark_data_smithi179_77138_object103784:head# 2024-02-09T17:27:03.613 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.613 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5270d56:::benchmark_data_smithi179_77138_object109821:head# 2024-02-09T17:27:03.613 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.613 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f527b7a1:::benchmark_data_smithi179_77138_object77876:head# 2024-02-09T17:27:03.613 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.613 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f52886ea:::benchmark_data_smithi179_77138_object93168:head# 2024-02-09T17:27:03.613 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.613 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f529a1e8:::benchmark_data_smithi179_77138_object26978:head# 2024-02-09T17:27:03.613 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.613 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f529c8fb:::benchmark_data_smithi179_77138_object47724:head# 2024-02-09T17:27:03.614 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.614 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f52abef2:::benchmark_data_smithi179_77138_object41035:head# 2024-02-09T17:27:03.614 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.614 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f52ad7fc:::benchmark_data_smithi179_77138_object78736:head# 2024-02-09T17:27:03.614 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.614 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f52adb8f:::benchmark_data_smithi179_77138_object26850:head# 2024-02-09T17:27:03.614 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.614 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f52b4639:::benchmark_data_smithi179_77138_object30739:head# 2024-02-09T17:27:03.614 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.614 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f52d0b22:::benchmark_data_smithi179_77138_object45339:head# 2024-02-09T17:27:03.614 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.614 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f52d5da9:::benchmark_data_smithi179_77138_object90180:head# 2024-02-09T17:27:03.614 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.615 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f52dcef1:::benchmark_data_smithi179_77138_object36922:head# 2024-02-09T17:27:03.615 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.615 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f52e229e:::benchmark_data_smithi179_77138_object127336:head# 2024-02-09T17:27:03.615 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.615 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f52e7cac:::benchmark_data_smithi179_77138_object42485:head# 2024-02-09T17:27:03.615 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.615 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f52ef8bb:::benchmark_data_smithi179_77138_object52072:head# 2024-02-09T17:27:03.615 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.615 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f52f96e7:::benchmark_data_smithi179_77138_object44200:head# 2024-02-09T17:27:03.615 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.615 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f530df98:::benchmark_data_smithi179_77138_object82220:head# 2024-02-09T17:27:03.615 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.615 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f53148b6:::benchmark_data_smithi179_77138_object64309:head# 2024-02-09T17:27:03.615 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.616 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5320ce4:::benchmark_data_smithi179_77138_object121177:head# 2024-02-09T17:27:03.616 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.616 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5356443:::benchmark_data_smithi179_77138_object37279:head# 2024-02-09T17:27:03.616 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.616 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5361457:::benchmark_data_smithi179_77138_object52499:head# 2024-02-09T17:27:03.616 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.616 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f536d9f9:::benchmark_data_smithi179_77138_object78176:head# 2024-02-09T17:27:03.616 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.616 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5375e2c:::benchmark_data_smithi179_77138_object77744:head# 2024-02-09T17:27:03.616 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.616 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5377a7b:::benchmark_data_smithi179_77138_object120201:head# 2024-02-09T17:27:03.616 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.616 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f537d2c9:::benchmark_data_smithi179_77138_object22341:head# 2024-02-09T17:27:03.617 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.617 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f538179c:::benchmark_data_smithi179_77138_object79633:head# 2024-02-09T17:27:03.617 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.617 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5383a1c:::benchmark_data_smithi179_77138_object82994:head# 2024-02-09T17:27:03.617 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.617 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f538b699:::benchmark_data_smithi179_77138_object51568:head# 2024-02-09T17:27:03.617 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.617 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f538b7ae:::benchmark_data_smithi179_77138_object88146:head# 2024-02-09T17:27:03.617 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.617 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f53a427d:::benchmark_data_smithi179_77138_object82011:head# 2024-02-09T17:27:03.617 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.617 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f53afcc9:::benchmark_data_smithi179_77138_object64811:head# 2024-02-09T17:27:03.617 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.617 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f53b3cba:::benchmark_data_smithi179_77138_object90378:head# 2024-02-09T17:27:03.618 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.618 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f53b861f:::benchmark_data_smithi179_77138_object120155:head# 2024-02-09T17:27:03.618 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.618 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f53bec54:::benchmark_data_smithi179_77138_object64909:head# 2024-02-09T17:27:03.618 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.618 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f53c11fe:::benchmark_data_smithi179_77138_object45316:head# 2024-02-09T17:27:03.618 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.618 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f53c3803:::benchmark_data_smithi179_77138_object62816:head# 2024-02-09T17:27:03.618 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.618 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f53c7e42:::benchmark_data_smithi179_77138_object28599:head# 2024-02-09T17:27:03.618 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.618 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f53cca90:::benchmark_data_smithi179_77138_object70334:head# 2024-02-09T17:27:03.618 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.618 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f53ceed1:::benchmark_data_smithi179_77138_object103527:head# 2024-02-09T17:27:03.619 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.619 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f53d3364:::benchmark_data_smithi179_77138_object80230:head# 2024-02-09T17:27:03.619 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.619 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f53d5701:::benchmark_data_smithi179_77138_object120704:head# 2024-02-09T17:27:03.619 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.619 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f53d6f38:::benchmark_data_smithi179_77138_object70207:head# 2024-02-09T17:27:03.619 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.619 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f53dcf4f:::benchmark_data_smithi179_77138_object68693:head# 2024-02-09T17:27:03.619 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.619 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f53eba8d:::benchmark_data_smithi179_77138_object129040:head# 2024-02-09T17:27:03.619 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.619 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f53ee90c:::benchmark_data_smithi179_77138_object89348:head# 2024-02-09T17:27:03.619 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.619 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f53f0e7e:::benchmark_data_smithi179_77138_object21478:head# 2024-02-09T17:27:03.620 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.620 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f53f10cf:::benchmark_data_smithi179_77138_object106004:head# 2024-02-09T17:27:03.620 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.620 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f53f5ca0:::benchmark_data_smithi179_77138_object67860:head# 2024-02-09T17:27:03.620 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.620 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f540efc5:::benchmark_data_smithi179_77138_object73314:head# 2024-02-09T17:27:03.620 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.623 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5426dc0:::benchmark_data_smithi179_77138_object50061:head# 2024-02-09T17:27:03.623 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.623 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5438d00:::benchmark_data_smithi179_77138_object30676:head# 2024-02-09T17:27:03.623 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.623 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f54493c8:::benchmark_data_smithi179_77138_object29861:head# 2024-02-09T17:27:03.623 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.623 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f54522ec:::benchmark_data_smithi179_77138_object58443:head# 2024-02-09T17:27:03.624 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.624 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f546c189:::benchmark_data_smithi179_77138_object66053:head# 2024-02-09T17:27:03.624 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.624 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5479124:::benchmark_data_smithi179_77138_object48131:head# 2024-02-09T17:27:03.624 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.624 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f547c134:::benchmark_data_smithi179_77138_object41849:head# 2024-02-09T17:27:03.624 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.624 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5481fb9:::benchmark_data_smithi179_77138_object69153:head# 2024-02-09T17:27:03.624 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.624 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f54851ba:::benchmark_data_smithi179_77138_object121141:head# 2024-02-09T17:27:03.624 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.624 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5486b31:::benchmark_data_smithi179_77138_object91313:head# 2024-02-09T17:27:03.624 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.624 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f548b120:::benchmark_data_smithi179_77138_object98616:head# 2024-02-09T17:27:03.625 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.625 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f548bee4:::benchmark_data_smithi179_77138_object30155:head# 2024-02-09T17:27:03.625 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.625 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f548e68e:::benchmark_data_smithi179_77138_object93370:head# 2024-02-09T17:27:03.625 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.625 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5491171:::benchmark_data_smithi179_77138_object41394:head# 2024-02-09T17:27:03.625 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.625 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f549476d:::benchmark_data_smithi179_77138_object119059:head# 2024-02-09T17:27:03.625 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.625 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f54a1185:::benchmark_data_smithi179_77138_object29647:head# 2024-02-09T17:27:03.625 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.625 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f54a3ecd:::benchmark_data_smithi179_77138_object33300:head# 2024-02-09T17:27:03.625 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.625 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f54b964c:::benchmark_data_smithi179_77138_object37670:head# 2024-02-09T17:27:03.625 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.626 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f54c9cdc:::benchmark_data_smithi179_77138_object103496:head# 2024-02-09T17:27:03.626 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.626 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f54cf8da:::benchmark_data_smithi179_77138_object26353:head# 2024-02-09T17:27:03.626 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.626 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f54ec100:::benchmark_data_smithi179_77138_object119140:head# 2024-02-09T17:27:03.626 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.626 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f54f0a6c:::benchmark_data_smithi179_77138_object33936:head# 2024-02-09T17:27:03.626 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.626 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f54f11c5:::benchmark_data_smithi179_77138_object87031:head# 2024-02-09T17:27:03.626 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.626 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f54f2712:::benchmark_data_smithi179_77138_object95391:head# 2024-02-09T17:27:03.626 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.626 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f54fc323:::benchmark_data_smithi179_77138_object27961:head# 2024-02-09T17:27:03.626 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.626 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f54ff8fb:::benchmark_data_smithi179_77138_object34073:head# 2024-02-09T17:27:03.627 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.627 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5512a46:::benchmark_data_smithi179_77138_object120550:head# 2024-02-09T17:27:03.627 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.627 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5528033:::benchmark_data_smithi179_77138_object124201:head# 2024-02-09T17:27:03.627 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.627 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f552ff9d:::benchmark_data_smithi179_77138_object30323:head# 2024-02-09T17:27:03.627 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.629 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f553e8aa:::benchmark_data_smithi179_77138_object121923:head# 2024-02-09T17:27:03.629 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.629 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f55434f7:::benchmark_data_smithi179_77138_object129405:head# 2024-02-09T17:27:03.629 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.629 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f554a031:::benchmark_data_smithi179_77138_object52553:head# 2024-02-09T17:27:03.629 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.629 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f554fb41:::benchmark_data_smithi179_77138_object75654:head# 2024-02-09T17:27:03.629 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.629 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5550031:::benchmark_data_smithi179_77138_object21330:head# 2024-02-09T17:27:03.629 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.629 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5557e0d:::benchmark_data_smithi179_77138_object100863:head# 2024-02-09T17:27:03.630 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.630 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f555accc:::benchmark_data_smithi179_77138_object28126:head# 2024-02-09T17:27:03.630 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.630 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f555dfca:::benchmark_data_smithi179_77138_object122936:head# 2024-02-09T17:27:03.630 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.630 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5561efe:::benchmark_data_smithi179_77138_object83447:head# 2024-02-09T17:27:03.630 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.630 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f556cb06:::benchmark_data_smithi179_77138_object81575:head# 2024-02-09T17:27:03.630 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.630 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f556ebda:::benchmark_data_smithi179_77138_object108641:head# 2024-02-09T17:27:03.630 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.630 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f55749bb:::benchmark_data_smithi179_77138_object22735:head# 2024-02-09T17:27:03.630 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.630 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5577afc:::benchmark_data_smithi179_77138_object36589:head# 2024-02-09T17:27:03.631 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.631 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5596381:::benchmark_data_smithi179_77138_object55064:head# 2024-02-09T17:27:03.631 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.631 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f559702d:::benchmark_data_smithi179_77138_object66709:head# 2024-02-09T17:27:03.631 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.631 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f55a0f24:::benchmark_data_smithi179_77138_object85573:head# 2024-02-09T17:27:03.631 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.631 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f55b21dd:::benchmark_data_smithi179_77138_object93184:head# 2024-02-09T17:27:03.631 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.631 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f55e9b89:::benchmark_data_smithi179_77138_object50897:head# 2024-02-09T17:27:03.631 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.631 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f55f2659:::benchmark_data_smithi179_77138_object129391:head# 2024-02-09T17:27:03.631 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.631 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f55f2d52:::benchmark_data_smithi179_77138_object76895:head# 2024-02-09T17:27:03.631 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.632 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f55f978e:::benchmark_data_smithi179_77138_object58712:head# 2024-02-09T17:27:03.632 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.632 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f55fad34:::benchmark_data_smithi179_77138_object74858:head# 2024-02-09T17:27:03.632 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.632 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5603f8c:::benchmark_data_smithi179_77138_object91383:head# 2024-02-09T17:27:03.632 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.632 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f56076f5:::benchmark_data_smithi179_77138_object68702:head# 2024-02-09T17:27:03.632 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.634 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f56135c1:::benchmark_data_smithi179_77138_object114704:head# 2024-02-09T17:27:03.634 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.634 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f56143d2:::benchmark_data_smithi179_77138_object124448:head# 2024-02-09T17:27:03.634 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.634 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f561e226:::benchmark_data_smithi179_77138_object125056:head# 2024-02-09T17:27:03.634 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.634 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f562fdc3:::benchmark_data_smithi179_77138_object71573:head# 2024-02-09T17:27:03.634 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.634 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f56361e7:::benchmark_data_smithi179_77138_object73835:head# 2024-02-09T17:27:03.634 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.634 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f563c18f:::benchmark_data_smithi179_77138_object124028:head# 2024-02-09T17:27:03.634 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.634 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5640584:::benchmark_data_smithi179_77138_object73309:head# 2024-02-09T17:27:03.635 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.635 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f56571da:::benchmark_data_smithi179_77138_object133155:head# 2024-02-09T17:27:03.635 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.635 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f566481e:::benchmark_data_smithi179_77138_object118376:head# 2024-02-09T17:27:03.635 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.635 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5672df9:::benchmark_data_smithi179_77138_object115948:head# 2024-02-09T17:27:03.635 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.635 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f567c860:::benchmark_data_smithi179_77138_object51595:head# 2024-02-09T17:27:03.635 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.635 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5682f9b:::benchmark_data_smithi179_77138_object75811:head# 2024-02-09T17:27:03.635 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.635 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5685e47:::benchmark_data_smithi179_77138_object102754:head# 2024-02-09T17:27:03.635 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.635 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f568669e:::benchmark_data_smithi179_77138_object47340:head# 2024-02-09T17:27:03.636 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.636 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5686f85:::benchmark_data_smithi179_77138_object105302:head# 2024-02-09T17:27:03.636 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.636 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5689adb:::benchmark_data_smithi179_77138_object59163:head# 2024-02-09T17:27:03.636 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.636 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f568c770:::benchmark_data_smithi179_77138_object68612:head# 2024-02-09T17:27:03.636 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.636 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f56a1357:::benchmark_data_smithi179_77138_object114601:head# 2024-02-09T17:27:03.636 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.636 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f56ad36d:::benchmark_data_smithi179_77138_object40009:head# 2024-02-09T17:27:03.636 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.636 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f56c0194:::benchmark_data_smithi179_77138_object67779:head# 2024-02-09T17:27:03.636 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.639 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f56c134e:::benchmark_data_smithi179_77138_object71151:head# 2024-02-09T17:27:03.639 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.639 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f56cba5e:::benchmark_data_smithi179_77138_object31080:head# 2024-02-09T17:27:03.639 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.639 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f56d2363:::benchmark_data_smithi179_77138_object33069:head# 2024-02-09T17:27:03.639 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.639 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f56f0f77:::benchmark_data_smithi179_77138_object69197:head# 2024-02-09T17:27:03.639 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.639 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f56f14aa:::benchmark_data_smithi179_77138_object59154:head# 2024-02-09T17:27:03.640 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.640 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f56f38ce:::benchmark_data_smithi179_77138_object75390:head# 2024-02-09T17:27:03.640 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.640 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f56fe48b:::benchmark_data_smithi179_77138_object61409:head# 2024-02-09T17:27:03.640 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.640 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f570a155:::benchmark_data_smithi179_77138_object53069:head# 2024-02-09T17:27:03.640 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.640 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f570e2fe:::benchmark_data_smithi179_77138_object31095:head# 2024-02-09T17:27:03.640 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.640 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f572d8af:::benchmark_data_smithi179_77138_object33778:head# 2024-02-09T17:27:03.640 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.640 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f572e8dc:::benchmark_data_smithi179_77138_object82409:head# 2024-02-09T17:27:03.640 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.640 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5730606:::benchmark_data_smithi179_77138_object69824:head# 2024-02-09T17:27:03.641 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.641 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5731e4a:::benchmark_data_smithi179_77138_object92582:head# 2024-02-09T17:27:03.641 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.641 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5737933:::benchmark_data_smithi179_77138_object28868:head# 2024-02-09T17:27:03.641 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.641 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5738df8:::benchmark_data_smithi179_77138_object35876:head# 2024-02-09T17:27:03.641 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.641 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f574141b:::benchmark_data_smithi179_77138_object88258:head# 2024-02-09T17:27:03.641 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.641 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5743306:::benchmark_data_smithi179_77138_object30831:head# 2024-02-09T17:27:03.641 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.641 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f575ba6c:::benchmark_data_smithi179_77138_object32384:head# 2024-02-09T17:27:03.641 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.643 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f57602d8:::benchmark_data_smithi179_77138_object75725:head# 2024-02-09T17:27:03.643 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.643 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5769952:::benchmark_data_smithi179_77138_object102862:head# 2024-02-09T17:27:03.643 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.643 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5770773:::benchmark_data_smithi179_77138_object86284:head# 2024-02-09T17:27:03.643 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.643 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5775884:::benchmark_data_smithi179_77138_object29751:head# 2024-02-09T17:27:03.643 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.643 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5776cae:::benchmark_data_smithi179_77138_object116343:head# 2024-02-09T17:27:03.643 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.643 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5790e56:::benchmark_data_smithi179_77138_object121207:head# 2024-02-09T17:27:03.643 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.643 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f57927b9:::benchmark_data_smithi179_77138_object131800:head# 2024-02-09T17:27:03.644 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.645 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f57a7165:::benchmark_data_smithi179_77138_object131422:head# 2024-02-09T17:27:03.645 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.645 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f57acbf4:::benchmark_data_smithi179_77138_object91582:head# 2024-02-09T17:27:03.645 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.645 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f57bfa3e:::benchmark_data_smithi179_77138_object54031:head# 2024-02-09T17:27:03.645 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.645 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f57c8a73:::benchmark_data_smithi179_77138_object116522:head# 2024-02-09T17:27:03.645 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.645 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f57ca848:::benchmark_data_smithi179_77138_object54756:head# 2024-02-09T17:27:03.645 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.645 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f57d938e:::benchmark_data_smithi179_77138_object21123:head# 2024-02-09T17:27:03.645 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.645 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f57d9a1c:::benchmark_data_smithi179_77138_object99795:head# 2024-02-09T17:27:03.646 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.646 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f57e53ed:::benchmark_data_smithi179_77138_object31612:head# 2024-02-09T17:27:03.646 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.649 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f57eb7da:::benchmark_data_smithi179_77138_object40167:head# 2024-02-09T17:27:03.649 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.651 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f57eed97:::benchmark_data_smithi179_77138_object118163:head# 2024-02-09T17:27:03.651 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.651 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f57f0f0b:::benchmark_data_smithi179_77138_object84952:head# 2024-02-09T17:27:03.651 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.651 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f57f5ed0:::benchmark_data_smithi179_77138_object49473:head# 2024-02-09T17:27:03.651 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.652 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5805af0:::benchmark_data_smithi179_77138_object42438:head# 2024-02-09T17:27:03.652 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.652 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5809137:::benchmark_data_smithi179_77138_object129955:head# 2024-02-09T17:27:03.652 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.655 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f580c6ac:::benchmark_data_smithi179_77138_object66723:head# 2024-02-09T17:27:03.655 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.655 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f580d474:::benchmark_data_smithi179_77138_object30457:head# 2024-02-09T17:27:03.655 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.655 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5819b99:::benchmark_data_smithi179_77138_object34784:head# 2024-02-09T17:27:03.655 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.656 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f581ad18:::benchmark_data_smithi179_77138_object79559:head# 2024-02-09T17:27:03.656 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.656 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f582fae5:::benchmark_data_smithi179_77138_object59133:head# 2024-02-09T17:27:03.656 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.656 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5833739:::benchmark_data_smithi179_77138_object44020:head# 2024-02-09T17:27:03.656 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.656 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f58340ab:::benchmark_data_smithi179_77138_object73968:head# 2024-02-09T17:27:03.656 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.656 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5835e6a:::benchmark_data_smithi179_77138_object114646:head# 2024-02-09T17:27:03.656 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.656 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f584ce3c:::benchmark_data_smithi179_77138_object78698:head# 2024-02-09T17:27:03.656 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.656 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f58521d0:::benchmark_data_smithi179_77138_object87589:head# 2024-02-09T17:27:03.656 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.657 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f585924c:::benchmark_data_smithi179_77138_object29572:head# 2024-02-09T17:27:03.657 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.657 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5863a75:::benchmark_data_smithi179_77138_object77741:head# 2024-02-09T17:27:03.657 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.657 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5868e05:::benchmark_data_smithi179_77138_object109363:head# 2024-02-09T17:27:03.657 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.659 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f586ec87:::benchmark_data_smithi179_77138_object116022:head# 2024-02-09T17:27:03.660 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.660 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f587fa27:::benchmark_data_smithi179_77138_object73422:head# 2024-02-09T17:27:03.660 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.660 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f58813c0:::benchmark_data_smithi179_77138_object130694:head# 2024-02-09T17:27:03.660 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.660 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f58a64bf:::benchmark_data_smithi179_77138_object110218:head# 2024-02-09T17:27:03.660 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.660 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f58a7e64:::benchmark_data_smithi179_77138_object35353:head# 2024-02-09T17:27:03.660 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.660 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f58be252:::benchmark_data_smithi179_77138_object85661:head# 2024-02-09T17:27:03.660 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.660 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f58c604c:::benchmark_data_smithi179_77138_object90045:head# 2024-02-09T17:27:03.660 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.660 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f58d41b1:::benchmark_data_smithi179_77138_object27494:head# 2024-02-09T17:27:03.661 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.661 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f58d5058:::benchmark_data_smithi179_77138_object83015:head# 2024-02-09T17:27:03.661 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.661 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f58dc534:::benchmark_data_smithi179_77138_object106750:head# 2024-02-09T17:27:03.661 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.661 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f58dc5b2:::benchmark_data_smithi179_77138_object51205:head# 2024-02-09T17:27:03.661 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.661 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f58e7985:::benchmark_data_smithi179_77138_object57638:head# 2024-02-09T17:27:03.661 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.661 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f58eb26b:::benchmark_data_smithi179_77138_object61140:head# 2024-02-09T17:27:03.661 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.662 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f58f9664:::benchmark_data_smithi179_77138_object69487:head# 2024-02-09T17:27:03.662 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.662 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f58fbe53:::benchmark_data_smithi179_77138_object130553:head# 2024-02-09T17:27:03.662 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.662 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5902226:::benchmark_data_smithi179_77138_object23257:head# 2024-02-09T17:27:03.662 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.662 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f59059fd:::benchmark_data_smithi179_77138_object37233:head# 2024-02-09T17:27:03.662 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.662 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f593171d:::benchmark_data_smithi179_77138_object104924:head# 2024-02-09T17:27:03.662 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.665 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f593c145:::benchmark_data_smithi179_77138_object35056:head# 2024-02-09T17:27:03.665 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.665 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f594c0d2:::benchmark_data_smithi179_77138_object93560:head# 2024-02-09T17:27:03.665 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.666 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f59715fa:::benchmark_data_smithi179_77138_object61628:head# 2024-02-09T17:27:03.666 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.666 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f59814ce:::benchmark_data_smithi179_77138_object94172:head# 2024-02-09T17:27:03.666 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.666 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f598f2eb:::benchmark_data_smithi179_77138_object103270:head# 2024-02-09T17:27:03.666 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.668 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5994f2c:::benchmark_data_smithi179_77138_object65602:head# 2024-02-09T17:27:03.668 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.668 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f59a5e89:::benchmark_data_smithi179_77138_object61090:head# 2024-02-09T17:27:03.668 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.668 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f59a6e4b:::benchmark_data_smithi179_77138_object81918:head# 2024-02-09T17:27:03.669 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.669 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f59ad674:::benchmark_data_smithi179_77138_object71618:head# 2024-02-09T17:27:03.669 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.669 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f59af731:::benchmark_data_smithi179_77138_object89521:head# 2024-02-09T17:27:03.669 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.669 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f59b69cb:::benchmark_data_smithi179_77138_object132991:head# 2024-02-09T17:27:03.669 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.669 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f59b7005:::benchmark_data_smithi179_77138_object43526:head# 2024-02-09T17:27:03.669 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.669 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f59bb47b:::benchmark_data_smithi179_77138_object50703:head# 2024-02-09T17:27:03.669 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.669 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f59c3776:::benchmark_data_smithi179_77138_object129599:head# 2024-02-09T17:27:03.669 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.669 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f59c599e:::benchmark_data_smithi179_77138_object127024:head# 2024-02-09T17:27:03.670 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.670 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f59cfddc:::benchmark_data_smithi179_77138_object42843:head# 2024-02-09T17:27:03.670 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.674 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f59d2652:::benchmark_data_smithi179_77138_object87607:head# 2024-02-09T17:27:03.674 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.674 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f59e49fa:::benchmark_data_smithi179_77138_object110679:head# 2024-02-09T17:27:03.674 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.674 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f59e6e98:::benchmark_data_smithi179_77138_object108784:head# 2024-02-09T17:27:03.674 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.674 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f59f1188:::benchmark_data_smithi179_77138_object60222:head# 2024-02-09T17:27:03.674 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.674 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f59f901f:::benchmark_data_smithi179_77138_object76420:head# 2024-02-09T17:27:03.674 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.674 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f59ffae1:::benchmark_data_smithi179_77138_object101152:head# 2024-02-09T17:27:03.674 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.675 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5a0c9fa:::benchmark_data_smithi179_77138_object27000:head# 2024-02-09T17:27:03.675 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.675 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5a0dab0:::benchmark_data_smithi179_77138_object32906:head# 2024-02-09T17:27:03.675 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.675 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5a14569:::benchmark_data_smithi179_77138_object98298:head# 2024-02-09T17:27:03.675 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.675 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5a1ba68:::benchmark_data_smithi179_77138_object95841:head# 2024-02-09T17:27:03.675 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.675 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5a2a6b4:::benchmark_data_smithi179_77138_object108886:head# 2024-02-09T17:27:03.675 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.675 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5a35a55:::benchmark_data_smithi179_77138_object85440:head# 2024-02-09T17:27:03.675 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.675 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5a3a0b0:::benchmark_data_smithi179_77138_object25349:head# 2024-02-09T17:27:03.675 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.676 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5a3e695:::benchmark_data_smithi179_77138_object123358:head# 2024-02-09T17:27:03.676 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.676 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5a3fb3f:::benchmark_data_smithi179_77138_object79493:head# 2024-02-09T17:27:03.676 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.676 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5a4afd0:::benchmark_data_smithi179_77138_object92615:head# 2024-02-09T17:27:03.676 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.676 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5a5eeaa:::benchmark_data_smithi179_77138_object122306:head# 2024-02-09T17:27:03.676 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.676 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5a6e4e5:::benchmark_data_smithi179_77138_object53436:head# 2024-02-09T17:27:03.676 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.676 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5a6ebb0:::benchmark_data_smithi179_77138_object119787:head# 2024-02-09T17:27:03.676 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.676 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5a88329:::benchmark_data_smithi179_77138_object96800:head# 2024-02-09T17:27:03.676 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.676 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5a8a99e:::benchmark_data_smithi179_77138_object90915:head# 2024-02-09T17:27:03.677 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.679 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5a9649d:::benchmark_data_smithi179_77138_object72516:head# 2024-02-09T17:27:03.679 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.679 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5aa5b54:::benchmark_data_smithi179_77138_object58806:head# 2024-02-09T17:27:03.679 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.679 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5aacc09:::benchmark_data_smithi179_77138_object106255:head# 2024-02-09T17:27:03.679 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.679 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5aaf385:::benchmark_data_smithi179_77138_object90812:head# 2024-02-09T17:27:03.679 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.679 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5abd24b:::benchmark_data_smithi179_77138_object55319:head# 2024-02-09T17:27:03.679 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.679 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5ac69a6:::benchmark_data_smithi179_77138_object47615:head# 2024-02-09T17:27:03.679 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.679 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5ad058d:::benchmark_data_smithi179_77138_object83019:head# 2024-02-09T17:27:03.679 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.680 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5ad2dfa:::benchmark_data_smithi179_77138_object87570:head# 2024-02-09T17:27:03.680 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.680 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5addec4:::benchmark_data_smithi179_77138_object80516:head# 2024-02-09T17:27:03.680 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.680 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5adfed6:::benchmark_data_smithi179_77138_object124680:head# 2024-02-09T17:27:03.680 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.680 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5aea2df:::benchmark_data_smithi179_77138_object103170:head# 2024-02-09T17:27:03.680 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.680 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5aeb4c0:::benchmark_data_smithi179_77138_object116857:head# 2024-02-09T17:27:03.680 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.680 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5aebea0:::benchmark_data_smithi179_77138_object111049:head# 2024-02-09T17:27:03.680 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.680 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5aedaf5:::benchmark_data_smithi179_77138_object44708:head# 2024-02-09T17:27:03.680 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.681 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5afd82f:::benchmark_data_smithi179_77138_object84956:head# 2024-02-09T17:27:03.681 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.681 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5b0f84e:::benchmark_data_smithi179_77138_object39930:head# 2024-02-09T17:27:03.681 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.681 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5b23ca3:::benchmark_data_smithi179_77138_object75235:head# 2024-02-09T17:27:03.681 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.683 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5b34e4b:::benchmark_data_smithi179_77138_object70944:head# 2024-02-09T17:27:03.684 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.684 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5b400e2:::benchmark_data_smithi179_77138_object76397:head# 2024-02-09T17:27:03.684 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.684 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5b44245:::benchmark_data_smithi179_77138_object54119:head# 2024-02-09T17:27:03.684 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.684 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5b44637:::benchmark_data_smithi179_77138_object121889:head# 2024-02-09T17:27:03.684 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.684 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5b570aa:::benchmark_data_smithi179_77138_object84922:head# 2024-02-09T17:27:03.684 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.684 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5b655f5:::benchmark_data_smithi179_77138_object74821:head# 2024-02-09T17:27:03.684 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.684 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5b74669:::benchmark_data_smithi179_77138_object112307:head# 2024-02-09T17:27:03.685 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.685 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5b77c29:::benchmark_data_smithi179_77138_object66684:head# 2024-02-09T17:27:03.685 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.685 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5b7d8bf:::benchmark_data_smithi179_77138_object62894:head# 2024-02-09T17:27:03.685 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.685 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5b87977:::benchmark_data_smithi179_77138_object101896:head# 2024-02-09T17:27:03.685 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.685 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5b8fa91:::benchmark_data_smithi179_77138_object93096:head# 2024-02-09T17:27:03.685 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.685 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5b9d047:::benchmark_data_smithi179_77138_object29479:head# 2024-02-09T17:27:03.685 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.685 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5ba2031:::benchmark_data_smithi179_77138_object106748:head# 2024-02-09T17:27:03.685 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.685 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5bb2549:::benchmark_data_smithi179_77138_object108044:head# 2024-02-09T17:27:03.685 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.686 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5bb2cf5:::benchmark_data_smithi179_77138_object42926:head# 2024-02-09T17:27:03.686 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.687 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5bb8f61:::benchmark_data_smithi179_77138_object56622:head# 2024-02-09T17:27:03.687 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.687 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5bca03f:::benchmark_data_smithi179_77138_object88074:head# 2024-02-09T17:27:03.687 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.687 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5be4800:::benchmark_data_smithi179_77138_object39163:head# 2024-02-09T17:27:03.687 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.687 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5be4fee:::benchmark_data_smithi179_77138_object111528:head# 2024-02-09T17:27:03.687 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.687 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5be503a:::benchmark_data_smithi179_77138_object45397:head# 2024-02-09T17:27:03.688 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.688 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5be5f85:::benchmark_data_smithi179_77138_object109185:head# 2024-02-09T17:27:03.688 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.688 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5be9bad:::benchmark_data_smithi179_77138_object99149:head# 2024-02-09T17:27:03.688 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.688 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5bea328:::benchmark_data_smithi179_77138_object67708:head# 2024-02-09T17:27:03.688 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.688 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5bf9fcc:::benchmark_data_smithi179_77138_object109163:head# 2024-02-09T17:27:03.688 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.688 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5c01697:::benchmark_data_smithi179_77138_object84766:head# 2024-02-09T17:27:03.688 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.688 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:27:03.690 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5c081cf:::benchmark_data_smithi179_77138_object23047:head# 2024-02-09T17:27:03.690 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.690 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5c16ef7:::benchmark_data_smithi179_77138_object49816:head# 2024-02-09T17:27:03.690 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.690 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5c18acd:::benchmark_data_smithi179_77138_object66989:head# 2024-02-09T17:27:03.690 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.693 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5c21859:::benchmark_data_smithi179_77138_object39249:head# 2024-02-09T17:27:03.693 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.693 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5c2ab08:::benchmark_data_smithi179_77138_object98387:head# 2024-02-09T17:27:03.694 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.694 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5c3049e:::benchmark_data_smithi179_77138_object24986:head# 2024-02-09T17:27:03.694 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.694 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5c4d9b4:::benchmark_data_smithi179_77138_object108011:head# 2024-02-09T17:27:03.694 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.697 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5c4e75d:::benchmark_data_smithi179_77138_object82726:head# 2024-02-09T17:27:03.697 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.702 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5c5091b:::benchmark_data_smithi179_77138_object94920:head# 2024-02-09T17:27:03.702 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.707 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5c5f421:::benchmark_data_smithi179_77138_object29854:head# 2024-02-09T17:27:03.707 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.707 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5c68041:::benchmark_data_smithi179_77138_object39934:head# 2024-02-09T17:27:03.707 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.707 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5c6d862:::benchmark_data_smithi179_77138_object103845:head# 2024-02-09T17:27:03.707 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.707 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5c6ef13:::benchmark_data_smithi179_77138_object110305:head# 2024-02-09T17:27:03.707 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.707 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5c73f91:::benchmark_data_smithi179_77138_object52746:head# 2024-02-09T17:27:03.707 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.707 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5c74c5d:::benchmark_data_smithi179_77138_object45729:head# 2024-02-09T17:27:03.707 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.707 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5c7d57e:::benchmark_data_smithi179_77138_object37735:head# 2024-02-09T17:27:03.708 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.708 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5c7e5e7:::benchmark_data_smithi179_77138_object60872:head# 2024-02-09T17:27:03.708 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.708 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5c81cf3:::benchmark_data_smithi179_77138_object34260:head# 2024-02-09T17:27:03.708 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.708 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5c866ba:::benchmark_data_smithi179_77138_object58540:head# 2024-02-09T17:27:03.708 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.708 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5c96404:::benchmark_data_smithi179_77138_object85423:head# 2024-02-09T17:27:03.708 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.708 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5c99988:::benchmark_data_smithi179_77138_object111935:head# 2024-02-09T17:27:03.708 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.712 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5ca34cd:::benchmark_data_smithi179_77138_object34721:head# 2024-02-09T17:27:03.712 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.713 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5caa720:::benchmark_data_smithi179_77138_object28742:head# 2024-02-09T17:27:03.713 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.713 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5cb4bd8:::benchmark_data_smithi179_77138_object83171:head# 2024-02-09T17:27:03.713 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.713 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5cb77ab:::benchmark_data_smithi179_77138_object78288:head# 2024-02-09T17:27:03.713 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.713 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5cba881:::benchmark_data_smithi179_77138_object48833:head# 2024-02-09T17:27:03.713 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.713 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5cbca69:::benchmark_data_smithi179_77138_object67529:head# 2024-02-09T17:27:03.713 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.713 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5cc09de:::benchmark_data_smithi179_77138_object70653:head# 2024-02-09T17:27:03.713 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.713 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5cda238:::benchmark_data_smithi179_77138_object76754:head# 2024-02-09T17:27:03.713 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.714 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5ce315b:::benchmark_data_smithi179_77138_object38747:head# 2024-02-09T17:27:03.714 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.714 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5ce4272:::benchmark_data_smithi179_77138_object45116:head# 2024-02-09T17:27:03.714 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.714 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5cee6e0:::benchmark_data_smithi179_77138_object131347:head# 2024-02-09T17:27:03.714 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.714 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5cfb9a8:::benchmark_data_smithi179_77138_object123838:head# 2024-02-09T17:27:03.714 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.714 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5cfe860:::benchmark_data_smithi179_77138_object116255:head# 2024-02-09T17:27:03.714 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.714 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5d06632:::benchmark_data_smithi179_77138_object115342:head# 2024-02-09T17:27:03.714 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.714 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5d10aa5:::benchmark_data_smithi179_77138_object56691:head# 2024-02-09T17:27:03.714 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.714 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5d15d25:::benchmark_data_smithi179_77138_object69447:head# 2024-02-09T17:27:03.715 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.715 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5d23eaf:::benchmark_data_smithi179_77138_object77518:head# 2024-02-09T17:27:03.715 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.715 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5d24659:::benchmark_data_smithi179_77138_object112479:head# 2024-02-09T17:27:03.715 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.715 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5d2c213:::benchmark_data_smithi179_77138_object30531:head# 2024-02-09T17:27:03.715 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.715 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5d66c6f:::benchmark_data_smithi179_77138_object98352:head# 2024-02-09T17:27:03.715 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.715 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5d6ab5b:::benchmark_data_smithi179_77138_object54542:head# 2024-02-09T17:27:03.715 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.715 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5d73e35:::benchmark_data_smithi179_77138_object86466:head# 2024-02-09T17:27:03.715 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.721 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5d7a649:::benchmark_data_smithi179_77138_object43577:head# 2024-02-09T17:27:03.721 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.721 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5d7cabe:::benchmark_data_smithi179_77138_object90140:head# 2024-02-09T17:27:03.722 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.722 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5d80e1d:::benchmark_data_smithi179_77138_object126842:head# 2024-02-09T17:27:03.722 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.722 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5d885cc:::benchmark_data_smithi179_77138_object53162:head# 2024-02-09T17:27:03.722 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.722 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5d8e347:::benchmark_data_smithi179_77138_object65785:head# 2024-02-09T17:27:03.722 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.722 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5d9305f:::benchmark_data_smithi179_77138_object114457:head# 2024-02-09T17:27:03.722 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.722 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5da9dbe:::benchmark_data_smithi179_77138_object67961:head# 2024-02-09T17:27:03.722 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.722 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5dab7bb:::benchmark_data_smithi179_77138_object43449:head# 2024-02-09T17:27:03.722 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.722 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5dbb8b2:::benchmark_data_smithi179_77138_object50856:head# 2024-02-09T17:27:03.722 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.723 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5dc0893:::benchmark_data_smithi179_77138_object107469:head# 2024-02-09T17:27:03.723 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.723 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5dc269e:::benchmark_data_smithi179_77138_object120210:head# 2024-02-09T17:27:03.723 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.723 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5dc3d77:::benchmark_data_smithi179_77138_object20526:head# 2024-02-09T17:27:03.723 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.723 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5dd8031:::benchmark_data_smithi179_77138_object76922:head# 2024-02-09T17:27:03.723 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.723 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5de7c8a:::benchmark_data_smithi179_77138_object116302:head# 2024-02-09T17:27:03.723 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.723 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5dfda5e:::benchmark_data_smithi179_77138_object110878:head# 2024-02-09T17:27:03.723 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.723 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5e0661e:::benchmark_data_smithi179_77138_object77704:head# 2024-02-09T17:27:03.723 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.723 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5e0a915:::benchmark_data_smithi179_77138_object97270:head# 2024-02-09T17:27:03.723 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.724 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5e1234c:::benchmark_data_smithi179_77138_object117345:head# 2024-02-09T17:27:03.724 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.724 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5e28d06:::benchmark_data_smithi179_77138_object109462:head# 2024-02-09T17:27:03.724 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.724 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5e34691:::benchmark_data_smithi179_77138_object67142:head# 2024-02-09T17:27:03.724 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.724 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5e3d1a7:::benchmark_data_smithi179_77138_object53967:head# 2024-02-09T17:27:03.724 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.724 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5e480d2:::benchmark_data_smithi179_77138_object114376:head# 2024-02-09T17:27:03.724 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.724 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5e50b41:::benchmark_data_smithi179_77138_object73648:head# 2024-02-09T17:27:03.724 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.724 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5e5eb9b:::benchmark_data_smithi179_77138_object59830:head# 2024-02-09T17:27:03.724 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.725 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5e62545:::benchmark_data_smithi179_77138_object55489:head# 2024-02-09T17:27:03.725 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.725 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5e660ca:::benchmark_data_smithi179_77138_object74291:head# 2024-02-09T17:27:03.725 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.725 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5e87f1f:::benchmark_data_smithi179_77138_object117005:head# 2024-02-09T17:27:03.725 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.725 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5e8fd8f:::benchmark_data_smithi179_77138_object108322:head# 2024-02-09T17:27:03.725 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.725 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5e99485:::benchmark_data_smithi179_77138_object44838:head# 2024-02-09T17:27:03.725 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.725 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5e9d943:::benchmark_data_smithi179_77138_object69501:head# 2024-02-09T17:27:03.725 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.725 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5e9d94d:::benchmark_data_smithi179_77138_object111010:head# 2024-02-09T17:27:03.725 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.726 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5eaa183:::benchmark_data_smithi179_77138_object51928:head# 2024-02-09T17:27:03.726 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.726 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5eac1ac:::benchmark_data_smithi179_77138_object100132:head# 2024-02-09T17:27:03.726 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.726 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5ead4a0:::benchmark_data_smithi179_77138_object33321:head# 2024-02-09T17:27:03.726 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.726 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5ead7ce:::benchmark_data_smithi179_77138_object125471:head# 2024-02-09T17:27:03.726 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.726 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5ead952:::benchmark_data_smithi179_77138_object24929:head# 2024-02-09T17:27:03.726 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.727 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:03.727 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5eb1848:::benchmark_data_smithi179_77138_object106438:head# 2024-02-09T17:27:03.727 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.727 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5ebcfa9:::benchmark_data_smithi179_77138_object47298:head# 2024-02-09T17:27:03.728 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.728 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5ec6ba0:::benchmark_data_smithi179_77138_object35093:head# 2024-02-09T17:27:03.728 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.728 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5eda4fa:::benchmark_data_smithi179_77138_object38535:head# 2024-02-09T17:27:03.728 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.728 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5edc73b:::benchmark_data_smithi179_77138_object96920:head# 2024-02-09T17:27:03.728 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.735 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5ee4c46:::benchmark_data_smithi179_77138_object62326:head# 2024-02-09T17:27:03.735 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.736 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5ee713c:::benchmark_data_smithi179_77138_object63611:head# 2024-02-09T17:27:03.736 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.736 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5ef7caf:::benchmark_data_smithi179_77138_object63578:head# 2024-02-09T17:27:03.736 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.736 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5f1138e:::benchmark_data_smithi179_77138_object55153:head# 2024-02-09T17:27:03.736 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.740 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5f143d5:::benchmark_data_smithi179_77138_object83592:head# 2024-02-09T17:27:03.741 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.741 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5f18696:::benchmark_data_smithi179_77138_object74327:head# 2024-02-09T17:27:03.741 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.741 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5f1c947:::benchmark_data_smithi179_77138_object35852:head# 2024-02-09T17:27:03.741 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.741 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5f2479b:::benchmark_data_smithi179_77138_object33024:head# 2024-02-09T17:27:03.741 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.741 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5f27669:::benchmark_data_smithi179_77138_object95490:head# 2024-02-09T17:27:03.741 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.741 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5f2769f:::benchmark_data_smithi179_77138_object81214:head# 2024-02-09T17:27:03.741 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.741 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5f28e16:::benchmark_data_smithi179_77138_object86382:head# 2024-02-09T17:27:03.741 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.741 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5f29b84:::benchmark_data_smithi179_77138_object67405:head# 2024-02-09T17:27:03.742 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.742 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5f2b70c:::benchmark_data_smithi179_77138_object52153:head# 2024-02-09T17:27:03.742 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.742 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5f33ea3:::benchmark_data_smithi179_77138_object52999:head# 2024-02-09T17:27:03.742 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.742 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5f37e29:::benchmark_data_smithi179_77138_object100885:head# 2024-02-09T17:27:03.742 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.742 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5f38d61:::benchmark_data_smithi179_77138_object85744:head# 2024-02-09T17:27:03.743 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.745 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5f41947:::benchmark_data_smithi179_77138_object61141:head# 2024-02-09T17:27:03.745 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.745 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5f54ab2:::benchmark_data_smithi179_77138_object72841:head# 2024-02-09T17:27:03.745 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.745 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5f56f95:::benchmark_data_smithi179_77138_object119789:head# 2024-02-09T17:27:03.745 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.746 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5f617ae:::benchmark_data_smithi179_77138_object67759:head# 2024-02-09T17:27:03.746 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.746 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5f68f4a:::benchmark_data_smithi179_77138_object110181:head# 2024-02-09T17:27:03.746 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.746 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5f6b809:::benchmark_data_smithi179_77138_object60787:head# 2024-02-09T17:27:03.746 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.746 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5f6ea3a:::benchmark_data_smithi179_77138_object35925:head# 2024-02-09T17:27:03.746 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.746 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5f7103f:::benchmark_data_smithi179_77138_object111739:head# 2024-02-09T17:27:03.746 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.746 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5f7bbac:::benchmark_data_smithi179_77138_object62053:head# 2024-02-09T17:27:03.746 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.746 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5f7ef84:::benchmark_data_smithi179_77138_object126047:head# 2024-02-09T17:27:03.746 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.747 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5f810ba:::benchmark_data_smithi179_77138_object98025:head# 2024-02-09T17:27:03.747 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.747 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5f825cd:::benchmark_data_smithi179_77138_object93072:head# 2024-02-09T17:27:03.747 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.747 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5f94dd5:::benchmark_data_smithi179_77138_object70076:head# 2024-02-09T17:27:03.747 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.747 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5f9e61d:::benchmark_data_smithi179_77138_object96344:head# 2024-02-09T17:27:03.747 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.747 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5fa4818:::benchmark_data_smithi179_77138_object34631:head# 2024-02-09T17:27:03.747 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.747 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5fac09d:::benchmark_data_smithi179_77138_object125655:head# 2024-02-09T17:27:03.747 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.747 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5face22:::benchmark_data_smithi179_77138_object50496:head# 2024-02-09T17:27:03.747 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.752 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5fd5807:::benchmark_data_smithi179_77138_object109589:head# 2024-02-09T17:27:03.752 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.752 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5fd6979:::benchmark_data_smithi179_77138_object59534:head# 2024-02-09T17:27:03.752 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.752 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5fda0cf:::benchmark_data_smithi179_77138_object69385:head# 2024-02-09T17:27:03.752 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.752 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5fdc248:::benchmark_data_smithi179_77138_object108247:head# 2024-02-09T17:27:03.752 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.752 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f5feb1de:::benchmark_data_smithi179_77138_object101662:head# 2024-02-09T17:27:03.753 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.753 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f600cf9b:::benchmark_data_smithi179_77138_object90890:head# 2024-02-09T17:27:03.753 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.753 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6010d00:::benchmark_data_smithi179_77138_object81901:head# 2024-02-09T17:27:03.753 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.753 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f601e225:::benchmark_data_smithi179_77138_object79575:head# 2024-02-09T17:27:03.753 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.753 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f602b943:::benchmark_data_smithi179_77138_object72787:head# 2024-02-09T17:27:03.753 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.753 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f604d5f0:::benchmark_data_smithi179_77138_object33112:head# 2024-02-09T17:27:03.753 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.753 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6064b01:::benchmark_data_smithi179_77138_object133036:head# 2024-02-09T17:27:03.753 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.753 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f60673d3:::benchmark_data_smithi179_77138_object79706:head# 2024-02-09T17:27:03.754 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.754 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f60694b9:::benchmark_data_smithi179_77138_object96363:head# 2024-02-09T17:27:03.754 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.754 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f606c8ae:::benchmark_data_smithi179_77138_object59508:head# 2024-02-09T17:27:03.754 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.754 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f606ced8:::benchmark_data_smithi179_77138_object131578:head# 2024-02-09T17:27:03.754 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.754 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6072d6b:::benchmark_data_smithi179_77138_object57162:head# 2024-02-09T17:27:03.754 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.754 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6083ef8:::benchmark_data_smithi179_77138_object111550:head# 2024-02-09T17:27:03.754 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.754 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6089a33:::benchmark_data_smithi179_77138_object81751:head# 2024-02-09T17:27:03.754 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.754 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f608becf:::benchmark_data_smithi179_77138_object37126:head# 2024-02-09T17:27:03.755 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.755 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6099185:::benchmark_data_smithi179_77138_object46054:head# 2024-02-09T17:27:03.755 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.755 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f609c370:::benchmark_data_smithi179_77138_object87047:head# 2024-02-09T17:27:03.755 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.755 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f60a1288:::benchmark_data_smithi179_77138_object82063:head# 2024-02-09T17:27:03.755 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.755 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f60a38a7:::benchmark_data_smithi179_77138_object91211:head# 2024-02-09T17:27:03.755 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.755 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f60a70ba:::benchmark_data_smithi179_77138_object43664:head# 2024-02-09T17:27:03.755 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.755 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f60b0b08:::benchmark_data_smithi179_77138_object29708:head# 2024-02-09T17:27:03.755 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.755 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f60b328a:::benchmark_data_smithi179_77138_object87907:head# 2024-02-09T17:27:03.756 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.756 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f60b8450:::benchmark_data_smithi179_77138_object114961:head# 2024-02-09T17:27:03.756 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.756 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f60bc1fb:::benchmark_data_smithi179_77138_object66250:head# 2024-02-09T17:27:03.756 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.757 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f60c1e0d:::benchmark_data_smithi179_77138_object109369:head# 2024-02-09T17:27:03.757 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.757 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f60c9e1c:::benchmark_data_smithi179_77138_object24988:head# 2024-02-09T17:27:03.757 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.758 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f60d9996:::benchmark_data_smithi179_77138_object79390:head# 2024-02-09T17:27:03.758 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.758 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f60d9af4:::benchmark_data_smithi179_77138_object99141:head# 2024-02-09T17:27:03.758 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.758 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f60df039:::benchmark_data_smithi179_77138_object86654:head# 2024-02-09T17:27:03.758 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.758 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f60e1c27:::benchmark_data_smithi179_77138_object85754:head# 2024-02-09T17:27:03.759 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.759 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f60e663b:::benchmark_data_smithi179_77138_object62579:head# 2024-02-09T17:27:03.759 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.759 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f60e7133:::benchmark_data_smithi179_77138_object120015:head# 2024-02-09T17:27:03.759 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.759 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:27:03.762 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f60ef9aa:::benchmark_data_smithi179_77138_object77238:head# 2024-02-09T17:27:03.762 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.762 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f610603f:::benchmark_data_smithi179_77138_object26219:head# 2024-02-09T17:27:03.762 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.763 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6107058:::benchmark_data_smithi179_77138_object28755:head# 2024-02-09T17:27:03.763 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.763 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f610c02c:::benchmark_data_smithi179_77138_object22358:head# 2024-02-09T17:27:03.763 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.763 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f610d0b1:::benchmark_data_smithi179_77138_object122298:head# 2024-02-09T17:27:03.763 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.763 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f61116eb:::benchmark_data_smithi179_77138_object129197:head# 2024-02-09T17:27:03.763 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.763 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6145b0f:::benchmark_data_smithi179_77138_object90444:head# 2024-02-09T17:27:03.763 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.763 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6153f55:::benchmark_data_smithi179_77138_object30618:head# 2024-02-09T17:27:03.763 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.763 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f615be04:::benchmark_data_smithi179_77138_object60288:head# 2024-02-09T17:27:03.763 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.764 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6169b4d:::benchmark_data_smithi179_77138_object53616:head# 2024-02-09T17:27:03.764 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.764 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f616be7c:::benchmark_data_smithi179_77138_object121949:head# 2024-02-09T17:27:03.764 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.764 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f616dc2c:::benchmark_data_smithi179_77138_object132853:head# 2024-02-09T17:27:03.764 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.764 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6184f7f:::benchmark_data_smithi179_77138_object62325:head# 2024-02-09T17:27:03.764 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.764 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f61acb79:::benchmark_data_smithi179_77138_object42726:head# 2024-02-09T17:27:03.764 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.764 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f61ae79f:::benchmark_data_smithi179_77138_object98794:head# 2024-02-09T17:27:03.764 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.764 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f61b6a7b:::benchmark_data_smithi179_77138_object38431:head# 2024-02-09T17:27:03.764 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.765 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f61baddc:::benchmark_data_smithi179_77138_object126281:head# 2024-02-09T17:27:03.765 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.765 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f61bbda3:::benchmark_data_smithi179_77138_object52276:head# 2024-02-09T17:27:03.765 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.765 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:03 smithi099 ceph-mon[39420]: pgmap v459: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 5.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 900 op/s; 24390/290988 objects degraded (8.382%) 2024-02-09T17:27:03.766 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:03 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2213779330' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:27:03.766 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:03 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:27:03.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:03.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:27:03.771 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is unset 2024-02-09T17:27:03.772 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f61bc66d:::benchmark_data_smithi179_77138_object22399:head# 2024-02-09T17:27:03.772 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.772 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f61be9d9:::benchmark_data_smithi179_77138_object93702:head# 2024-02-09T17:27:03.772 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.772 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f61c5f79:::benchmark_data_smithi179_77138_object39655:head# 2024-02-09T17:27:03.772 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.772 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f61c9207:::benchmark_data_smithi179_77138_object114299:head# 2024-02-09T17:27:03.772 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.773 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f61d33e7:::benchmark_data_smithi179_77138_object69786:head# 2024-02-09T17:27:03.773 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.773 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f61f0a35:::benchmark_data_smithi179_77138_object42847:head# 2024-02-09T17:27:03.773 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.773 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f61f3352:::benchmark_data_smithi179_77138_object123229:head# 2024-02-09T17:27:03.773 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.773 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6217295:::benchmark_data_smithi179_77138_object84743:head# 2024-02-09T17:27:03.773 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.773 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6225638:::benchmark_data_smithi179_77138_object101621:head# 2024-02-09T17:27:03.773 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.773 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6226b2b:::benchmark_data_smithi179_77138_object58727:head# 2024-02-09T17:27:03.773 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.773 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6238141:::benchmark_data_smithi179_77138_object110276:head# 2024-02-09T17:27:03.773 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.773 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f623cc20:::benchmark_data_smithi179_77138_object51075:head# 2024-02-09T17:27:03.774 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.774 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f623e54a:::benchmark_data_smithi179_77138_object126955:head# 2024-02-09T17:27:03.774 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.774 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6252b2f:::benchmark_data_smithi179_77138_object121861:head# 2024-02-09T17:27:03.774 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.774 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f62544e0:::benchmark_data_smithi179_77138_object44891:head# 2024-02-09T17:27:03.774 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.774 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6254eb2:::benchmark_data_smithi179_77138_object129647:head# 2024-02-09T17:27:03.774 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.774 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6258c69:::benchmark_data_smithi179_77138_object75141:head# 2024-02-09T17:27:03.774 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.774 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f626d589:::benchmark_data_smithi179_77138_object34895:head# 2024-02-09T17:27:03.774 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.774 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f626ee27:::benchmark_data_smithi179_77138_object122395:head# 2024-02-09T17:27:03.775 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.775 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f62700fd:::benchmark_data_smithi179_77138_object74894:head# 2024-02-09T17:27:03.775 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.775 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6283193:::benchmark_data_smithi179_77138_object70797:head# 2024-02-09T17:27:03.775 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.775 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6283f2b:::benchmark_data_smithi179_77138_object25783:head# 2024-02-09T17:27:03.775 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.775 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f62893a6:::benchmark_data_smithi179_77138_object41890:head# 2024-02-09T17:27:03.775 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.775 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f628c22b:::benchmark_data_smithi179_77138_object87828:head# 2024-02-09T17:27:03.775 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.775 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f62a0ee5:::benchmark_data_smithi179_77138_object67964:head# 2024-02-09T17:27:03.775 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.775 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f62a2e2c:::benchmark_data_smithi179_77138_object48072:head# 2024-02-09T17:27:03.775 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.776 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f62aa987:::benchmark_data_smithi179_77138_object92151:head# 2024-02-09T17:27:03.776 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.776 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f62ae5de:::benchmark_data_smithi179_77138_object41251:head# 2024-02-09T17:27:03.776 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.777 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f62d1512:::benchmark_data_smithi179_77138_object74881:head# 2024-02-09T17:27:03.777 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.777 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f62d9f08:::benchmark_data_smithi179_77138_object75766:head# 2024-02-09T17:27:03.778 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.778 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f62e3480:::benchmark_data_smithi179_77138_object81673:head# 2024-02-09T17:27:03.778 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.778 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f62e5327:::benchmark_data_smithi179_77138_object71634:head# 2024-02-09T17:27:03.778 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.778 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f631221b:::benchmark_data_smithi179_77138_object67423:head# 2024-02-09T17:27:03.778 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.778 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f631d95a:::benchmark_data_smithi179_77138_object90008:head# 2024-02-09T17:27:03.778 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.779 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f632bdb1:::benchmark_data_smithi179_77138_object109832:head# 2024-02-09T17:27:03.779 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.779 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f632fdee:::benchmark_data_smithi179_77138_object103776:head# 2024-02-09T17:27:03.779 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.779 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6330e8e:::benchmark_data_smithi179_77138_object65376:head# 2024-02-09T17:27:03.780 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.780 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f633245a:::benchmark_data_smithi179_77138_object30032:head# 2024-02-09T17:27:03.780 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.782 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f63476e7:::benchmark_data_smithi179_77138_object26928:head# 2024-02-09T17:27:03.782 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.782 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f63504f6:::benchmark_data_smithi179_77138_object73826:head# 2024-02-09T17:27:03.782 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.782 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f635b995:::benchmark_data_smithi179_77138_object103685:head# 2024-02-09T17:27:03.782 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.782 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6363373:::benchmark_data_smithi179_77138_object79279:head# 2024-02-09T17:27:03.782 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.782 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6364915:::benchmark_data_smithi179_77138_object87646:head# 2024-02-09T17:27:03.782 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.783 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6385e2b:::benchmark_data_smithi179_77138_object32464:head# 2024-02-09T17:27:03.783 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.783 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f638a549:::benchmark_data_smithi179_77138_object129370:head# 2024-02-09T17:27:03.783 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.783 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f638c116:::benchmark_data_smithi179_77138_object74775:head# 2024-02-09T17:27:03.783 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.783 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f638f555:::benchmark_data_smithi179_77138_object47095:head# 2024-02-09T17:27:03.783 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.783 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6397788:::benchmark_data_smithi179_77138_object51634:head# 2024-02-09T17:27:03.783 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.783 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f63a3051:::benchmark_data_smithi179_77138_object74348:head# 2024-02-09T17:27:03.783 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.784 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f63ad25b:::benchmark_data_smithi179_77138_object89714:head# 2024-02-09T17:27:03.784 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.784 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f63b1de6:::benchmark_data_smithi179_77138_object27721:head# 2024-02-09T17:27:03.784 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.784 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f63b5204:::benchmark_data_smithi179_77138_object99090:head# 2024-02-09T17:27:03.785 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.785 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f63bc365:::benchmark_data_smithi179_77138_object51377:head# 2024-02-09T17:27:03.785 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.787 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f63d060e:::benchmark_data_smithi179_77138_object117779:head# 2024-02-09T17:27:03.787 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.787 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f63e38a2:::benchmark_data_smithi179_77138_object48418:head# 2024-02-09T17:27:03.788 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.788 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f63e477d:::benchmark_data_smithi179_77138_object127462:head# 2024-02-09T17:27:03.788 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.788 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f63e9de5:::benchmark_data_smithi179_77138_object35647:head# 2024-02-09T17:27:03.788 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.788 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f63ea381:::benchmark_data_smithi179_77138_object98658:head# 2024-02-09T17:27:03.788 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.788 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f63ec06b:::benchmark_data_smithi179_77138_object88824:head# 2024-02-09T17:27:03.788 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.790 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f63f11f5:::benchmark_data_smithi179_77138_object130224:head# 2024-02-09T17:27:03.791 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.791 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f63f15dc:::benchmark_data_smithi179_77138_object113368:head# 2024-02-09T17:27:03.791 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.791 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f63f5fbe:::benchmark_data_smithi179_77138_object83848:head# 2024-02-09T17:27:03.791 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.791 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f63f7620:::benchmark_data_smithi179_77138_object71419:head# 2024-02-09T17:27:03.791 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.791 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f63fcb7f:::benchmark_data_smithi179_77138_object66674:head# 2024-02-09T17:27:03.791 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.791 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f63fd7d5:::benchmark_data_smithi179_77138_object43963:head# 2024-02-09T17:27:03.791 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.791 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f640b480:::benchmark_data_smithi179_77138_object89308:head# 2024-02-09T17:27:03.791 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.791 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f641884d:::benchmark_data_smithi179_77138_object118356:head# 2024-02-09T17:27:03.792 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.792 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6422b8c:::benchmark_data_smithi179_77138_object74305:head# 2024-02-09T17:27:03.792 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.793 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f642de58:::benchmark_data_smithi179_77138_object44686:head# 2024-02-09T17:27:03.793 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.793 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6432492:::benchmark_data_smithi179_77138_object20109:head# 2024-02-09T17:27:03.793 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.793 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f64451e0:::benchmark_data_smithi179_77138_object82643:head# 2024-02-09T17:27:03.794 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.794 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6447f02:::benchmark_data_smithi179_77138_object43125:head# 2024-02-09T17:27:03.794 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.794 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f644b070:::benchmark_data_smithi179_77138_object75583:head# 2024-02-09T17:27:03.794 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.794 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6450cd0:::benchmark_data_smithi179_77138_object117119:head# 2024-02-09T17:27:03.794 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.794 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f645a6e3:::benchmark_data_smithi179_77138_object47624:head# 2024-02-09T17:27:03.794 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.794 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f645e1da:::benchmark_data_smithi179_77138_object58956:head# 2024-02-09T17:27:03.794 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.794 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6465cfd:::benchmark_data_smithi179_77138_object123571:head# 2024-02-09T17:27:03.794 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.794 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6470711:::benchmark_data_smithi179_77138_object45853:head# 2024-02-09T17:27:03.794 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.796 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f647d14c:::benchmark_data_smithi179_77138_object32385:head# 2024-02-09T17:27:03.796 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.796 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f647db37:::benchmark_data_smithi179_77138_object21413:head# 2024-02-09T17:27:03.796 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.796 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6480d4c:::benchmark_data_smithi179_77138_object124270:head# 2024-02-09T17:27:03.796 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.796 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6482d9b:::benchmark_data_smithi179_77138_object78098:head# 2024-02-09T17:27:03.796 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.796 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f648929e:::benchmark_data_smithi179_77138_object77156:head# 2024-02-09T17:27:03.797 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.797 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f648a3aa:::benchmark_data_smithi179_77138_object49230:head# 2024-02-09T17:27:03.797 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.797 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f649a1a9:::benchmark_data_smithi179_77138_object39703:head# 2024-02-09T17:27:03.797 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.797 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f649fed1:::benchmark_data_smithi179_77138_object96390:head# 2024-02-09T17:27:03.797 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.797 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f64a394a:::benchmark_data_smithi179_77138_object130953:head# 2024-02-09T17:27:03.797 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.799 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f64b9451:::benchmark_data_smithi179_77138_object112311:head# 2024-02-09T17:27:03.799 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.800 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f64bbf60:::benchmark_data_smithi179_77138_object56206:head# 2024-02-09T17:27:03.800 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.800 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f64bdcbe:::benchmark_data_smithi179_77138_object127017:head# 2024-02-09T17:27:03.800 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.800 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f64bf4fa:::benchmark_data_smithi179_77138_object96194:head# 2024-02-09T17:27:03.800 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.800 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f64c12ea:::benchmark_data_smithi179_77138_object94115:head# 2024-02-09T17:27:03.800 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.800 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f64d3294:::benchmark_data_smithi179_77138_object90412:head# 2024-02-09T17:27:03.800 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.800 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f64d5384:::benchmark_data_smithi179_77138_object33601:head# 2024-02-09T17:27:03.800 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.800 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f64e3ded:::benchmark_data_smithi179_77138_object37325:head# 2024-02-09T17:27:03.801 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.801 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f64e6d7d:::benchmark_data_smithi179_77138_object51880:head# 2024-02-09T17:27:03.801 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.801 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f64f6be0:::benchmark_data_smithi179_77138_object72866:head# 2024-02-09T17:27:03.801 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.801 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f64fe7a6:::benchmark_data_smithi179_77138_object73869:head# 2024-02-09T17:27:03.801 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.801 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6501da4:::benchmark_data_smithi179_77138_object63337:head# 2024-02-09T17:27:03.801 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.801 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6504afe:::benchmark_data_smithi179_77138_object91761:head# 2024-02-09T17:27:03.801 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.802 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6511e7f:::benchmark_data_smithi179_77138_object56041:head# 2024-02-09T17:27:03.802 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.802 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6529dba:::benchmark_data_smithi179_77138_object90243:head# 2024-02-09T17:27:03.802 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.807 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6535183:::benchmark_data_smithi179_77138_object84836:head# 2024-02-09T17:27:03.807 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.807 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f653ff6b:::benchmark_data_smithi179_77138_object113175:head# 2024-02-09T17:27:03.807 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.807 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6546e29:::benchmark_data_smithi179_77138_object64618:head# 2024-02-09T17:27:03.807 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.807 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f654c259:::benchmark_data_smithi179_77138_object106654:head# 2024-02-09T17:27:03.808 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.808 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f654ce77:::benchmark_data_smithi179_77138_object26570:head# 2024-02-09T17:27:03.808 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.808 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f65516b7:::benchmark_data_smithi179_77138_object38445:head# 2024-02-09T17:27:03.808 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.808 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f65624e1:::benchmark_data_smithi179_77138_object32067:head# 2024-02-09T17:27:03.808 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.808 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f65638ea:::benchmark_data_smithi179_77138_object109879:head# 2024-02-09T17:27:03.808 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.808 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f65742f7:::benchmark_data_smithi179_77138_object43617:head# 2024-02-09T17:27:03.808 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.808 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f657a00b:::benchmark_data_smithi179_77138_object65028:head# 2024-02-09T17:27:03.808 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.809 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6582124:::benchmark_data_smithi179_77138_object59026:head# 2024-02-09T17:27:03.809 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.809 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f658ceca:::benchmark_data_smithi179_77138_object89357:head# 2024-02-09T17:27:03.809 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.809 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f65b4f33:::benchmark_data_smithi179_77138_object74645:head# 2024-02-09T17:27:03.809 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.809 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f65c3a44:::benchmark_data_smithi179_77138_object93656:head# 2024-02-09T17:27:03.809 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.809 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f65c50a1:::benchmark_data_smithi179_77138_object20105:head# 2024-02-09T17:27:03.809 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.809 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f65ca0c7:::benchmark_data_smithi179_77138_object91096:head# 2024-02-09T17:27:03.809 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.809 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f65cfc47:::benchmark_data_smithi179_77138_object67231:head# 2024-02-09T17:27:03.809 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.809 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f65d2903:::benchmark_data_smithi179_77138_object33528:head# 2024-02-09T17:27:03.810 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.810 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f65de23d:::benchmark_data_smithi179_77138_object111305:head# 2024-02-09T17:27:03.810 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.810 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f65ef744:::benchmark_data_smithi179_77138_object109510:head# 2024-02-09T17:27:03.810 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.810 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f65f4906:::benchmark_data_smithi179_77138_object85137:head# 2024-02-09T17:27:03.810 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.810 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f65f7a11:::benchmark_data_smithi179_77138_object45861:head# 2024-02-09T17:27:03.810 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.810 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f65fbc08:::benchmark_data_smithi179_77138_object50907:head# 2024-02-09T17:27:03.810 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.810 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6615c4c:::benchmark_data_smithi179_77138_object72293:head# 2024-02-09T17:27:03.810 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.811 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6616d07:::benchmark_data_smithi179_77138_object31557:head# 2024-02-09T17:27:03.811 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.811 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f661e370:::benchmark_data_smithi179_77138_object106402:head# 2024-02-09T17:27:03.811 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.811 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f662523e:::benchmark_data_smithi179_77138_object122363:head# 2024-02-09T17:27:03.811 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.811 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f662cd68:::benchmark_data_smithi179_77138_object110546:head# 2024-02-09T17:27:03.811 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.812 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f66462b6:::benchmark_data_smithi179_77138_object24341:head# 2024-02-09T17:27:03.812 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.812 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f664c5ee:::benchmark_data_smithi179_77138_object107143:head# 2024-02-09T17:27:03.812 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.812 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f664d86d:::benchmark_data_smithi179_77138_object127055:head# 2024-02-09T17:27:03.812 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.812 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f664f513:::benchmark_data_smithi179_77138_object38048:head# 2024-02-09T17:27:03.812 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.820 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f66557b8:::benchmark_data_smithi179_77138_object133326:head# 2024-02-09T17:27:03.820 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.821 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f665650b:::benchmark_data_smithi179_77138_object110693:head# 2024-02-09T17:27:03.821 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.821 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f66565c2:::benchmark_data_smithi179_77138_object19419:head# 2024-02-09T17:27:03.821 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.821 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f665b823:::benchmark_data_smithi179_77138_object40752:head# 2024-02-09T17:27:03.821 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.822 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6661b79:::benchmark_data_smithi179_77138_object120551:head# 2024-02-09T17:27:03.822 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.822 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f66651ab:::benchmark_data_smithi179_77138_object39661:head# 2024-02-09T17:27:03.822 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.822 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6669920:::benchmark_data_smithi179_77138_object46625:head# 2024-02-09T17:27:03.822 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.822 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6678384:::benchmark_data_smithi179_77138_object125097:head# 2024-02-09T17:27:03.822 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.822 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6684309:::benchmark_data_smithi179_77138_object47982:head# 2024-02-09T17:27:03.822 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.823 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f668a05c:::benchmark_data_smithi179_77138_object72295:head# 2024-02-09T17:27:03.823 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.823 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f668b957:::benchmark_data_smithi179_77138_object46821:head# 2024-02-09T17:27:03.823 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.823 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f668d9e6:::benchmark_data_smithi179_77138_object41021:head# 2024-02-09T17:27:03.823 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.823 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6696cc8:::benchmark_data_smithi179_77138_object108377:head# 2024-02-09T17:27:03.823 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.823 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f669e3f1:::benchmark_data_smithi179_77138_object61211:head# 2024-02-09T17:27:03.823 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.823 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f66af516:::benchmark_data_smithi179_77138_object121974:head# 2024-02-09T17:27:03.823 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.823 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f66b1592:::benchmark_data_smithi179_77138_object51061:head# 2024-02-09T17:27:03.824 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.824 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f66b2d3c:::benchmark_data_smithi179_77138_object88184:head# 2024-02-09T17:27:03.824 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.824 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f66b82db:::benchmark_data_smithi179_77138_object29317:head# 2024-02-09T17:27:03.824 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.824 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f66b9edf:::benchmark_data_smithi179_77138_object92846:head# 2024-02-09T17:27:03.824 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.824 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f66c7fce:::benchmark_data_smithi179_77138_object54094:head# 2024-02-09T17:27:03.824 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.824 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f66c973d:::benchmark_data_smithi179_77138_object111213:head# 2024-02-09T17:27:03.824 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.824 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f66d23dd:::benchmark_data_smithi179_77138_object105575:head# 2024-02-09T17:27:03.824 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.824 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f66d9457:::benchmark_data_smithi179_77138_object39856:head# 2024-02-09T17:27:03.825 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.825 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f66da3c7:::benchmark_data_smithi179_77138_object94386:head# 2024-02-09T17:27:03.825 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.825 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f66dd637:::benchmark_data_smithi179_77138_object64098:head# 2024-02-09T17:27:03.825 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.825 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f66df584:::benchmark_data_smithi179_77138_object75702:head# 2024-02-09T17:27:03.825 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.825 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f66e10e9:::benchmark_data_smithi179_77138_object32987:head# 2024-02-09T17:27:03.825 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.825 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f66e21f0:::benchmark_data_smithi179_77138_object92798:head# 2024-02-09T17:27:03.825 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.825 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f66ec498:::benchmark_data_smithi179_77138_object115596:head# 2024-02-09T17:27:03.825 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.825 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6703a50:::benchmark_data_smithi179_77138_object84869:head# 2024-02-09T17:27:03.826 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.826 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6708c37:::benchmark_data_smithi179_77138_object132037:head# 2024-02-09T17:27:03.826 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.826 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f671345a:::benchmark_data_smithi179_77138_object81030:head# 2024-02-09T17:27:03.826 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.826 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f671c4b9:::benchmark_data_smithi179_77138_object93712:head# 2024-02-09T17:27:03.826 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.826 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6726315:::benchmark_data_smithi179_77138_object126964:head# 2024-02-09T17:27:03.826 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.826 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f672f70d:::benchmark_data_smithi179_77138_object104317:head# 2024-02-09T17:27:03.826 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.826 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f673194f:::benchmark_data_smithi179_77138_object104600:head# 2024-02-09T17:27:03.826 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.826 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6736df6:::benchmark_data_smithi179_77138_object74130:head# 2024-02-09T17:27:03.827 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.827 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f674ceaa:::benchmark_data_smithi179_77138_object29211:head# 2024-02-09T17:27:03.827 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.827 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6756ac2:::benchmark_data_smithi179_77138_object87795:head# 2024-02-09T17:27:03.827 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.827 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f675dab3:::benchmark_data_smithi179_77138_object26204:head# 2024-02-09T17:27:03.827 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.827 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f676fd3b:::benchmark_data_smithi179_77138_object67062:head# 2024-02-09T17:27:03.827 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.827 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f67a07f9:::benchmark_data_smithi179_77138_object37365:head# 2024-02-09T17:27:03.827 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.827 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f67a565d:::benchmark_data_smithi179_77138_object89695:head# 2024-02-09T17:27:03.827 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.827 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f67a885e:::benchmark_data_smithi179_77138_object26955:head# 2024-02-09T17:27:03.827 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.828 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f67b1279:::benchmark_data_smithi179_77138_object59858:head# 2024-02-09T17:27:03.828 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.828 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f67bb917:::benchmark_data_smithi179_77138_object86450:head# 2024-02-09T17:27:03.828 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.828 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f67bb919:::benchmark_data_smithi179_77138_object25339:head# 2024-02-09T17:27:03.828 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.828 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f67c5638:::benchmark_data_smithi179_77138_object52544:head# 2024-02-09T17:27:03.828 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.828 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f67cf93a:::benchmark_data_smithi179_77138_object74701:head# 2024-02-09T17:27:03.828 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.828 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f67d89db:::benchmark_data_smithi179_77138_object74244:head# 2024-02-09T17:27:03.828 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.828 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f67e1b5d:::benchmark_data_smithi179_77138_object72690:head# 2024-02-09T17:27:03.828 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.828 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f67f0f00:::benchmark_data_smithi179_77138_object117836:head# 2024-02-09T17:27:03.828 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.829 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f67fcb9a:::benchmark_data_smithi179_77138_object127975:head# 2024-02-09T17:27:03.829 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.829 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f67ff78f:::benchmark_data_smithi179_77138_object101892:head# 2024-02-09T17:27:03.829 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.836 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f680bc77:::benchmark_data_smithi179_77138_object21892:head# 2024-02-09T17:27:03.836 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.836 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6827ceb:::benchmark_data_smithi179_77138_object29802:head# 2024-02-09T17:27:03.836 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.836 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f682f6a4:::benchmark_data_smithi179_77138_object55363:head# 2024-02-09T17:27:03.836 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.836 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6840169:::benchmark_data_smithi179_77138_object79514:head# 2024-02-09T17:27:03.836 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.837 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6842688:::benchmark_data_smithi179_77138_object96981:head# 2024-02-09T17:27:03.837 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.837 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6849d1e:::benchmark_data_smithi179_77138_object58969:head# 2024-02-09T17:27:03.837 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.837 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6863911:::benchmark_data_smithi179_77138_object119556:head# 2024-02-09T17:27:03.837 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.837 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6881ebb:::benchmark_data_smithi179_77138_object97463:head# 2024-02-09T17:27:03.837 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.837 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f689ef6a:::benchmark_data_smithi179_77138_object59270:head# 2024-02-09T17:27:03.837 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.837 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f68a74b7:::benchmark_data_smithi179_77138_object123121:head# 2024-02-09T17:27:03.837 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.837 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f68b0584:::benchmark_data_smithi179_77138_object81345:head# 2024-02-09T17:27:03.837 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.838 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f68b0d58:::benchmark_data_smithi179_77138_object118054:head# 2024-02-09T17:27:03.838 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.838 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f68b2f10:::benchmark_data_smithi179_77138_object61868:head# 2024-02-09T17:27:03.838 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.838 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f68b3068:::benchmark_data_smithi179_77138_object32753:head# 2024-02-09T17:27:03.838 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.838 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f68c91ec:::benchmark_data_smithi179_77138_object39605:head# 2024-02-09T17:27:03.838 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.838 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f68cee8d:::benchmark_data_smithi179_77138_object40295:head# 2024-02-09T17:27:03.838 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.838 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f68e8b94:::benchmark_data_smithi179_77138_object66040:head# 2024-02-09T17:27:03.838 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.838 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f68f97db:::benchmark_data_smithi179_77138_object43478:head# 2024-02-09T17:27:03.838 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.839 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6901323:::benchmark_data_smithi179_77138_object131275:head# 2024-02-09T17:27:03.839 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.839 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6901d8f:::benchmark_data_smithi179_77138_object51440:head# 2024-02-09T17:27:03.839 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.839 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f69248e6:::benchmark_data_smithi179_77138_object37376:head# 2024-02-09T17:27:03.839 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.839 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f69249b9:::benchmark_data_smithi179_77138_object34646:head# 2024-02-09T17:27:03.839 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.839 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6926b4d:::benchmark_data_smithi179_77138_object92315:head# 2024-02-09T17:27:03.839 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.839 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6937f62:::benchmark_data_smithi179_77138_object60313:head# 2024-02-09T17:27:03.839 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.839 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6941086:::benchmark_data_smithi179_77138_object116925:head# 2024-02-09T17:27:03.839 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.840 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f69469cd:::benchmark_data_smithi179_77138_object53614:head# 2024-02-09T17:27:03.840 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.840 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6948118:::benchmark_data_smithi179_77138_object105686:head# 2024-02-09T17:27:03.840 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.840 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6958350:::benchmark_data_smithi179_77138_object99758:head# 2024-02-09T17:27:03.840 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.840 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f696e38c:::benchmark_data_smithi179_77138_object75549:head# 2024-02-09T17:27:03.840 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.840 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f697f0b2:::benchmark_data_smithi179_77138_object99625:head# 2024-02-09T17:27:03.840 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.840 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f698714a:::benchmark_data_smithi179_77138_object90843:head# 2024-02-09T17:27:03.840 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.840 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f698e15e:::benchmark_data_smithi179_77138_object118489:head# 2024-02-09T17:27:03.841 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.841 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f698f3ae:::benchmark_data_smithi179_77138_object122965:head# 2024-02-09T17:27:03.841 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.841 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f699762a:::benchmark_data_smithi179_77138_object123306:head# 2024-02-09T17:27:03.841 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.841 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f699a357:::benchmark_data_smithi179_77138_object116554:head# 2024-02-09T17:27:03.841 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.841 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f699d655:::benchmark_data_smithi179_77138_object53010:head# 2024-02-09T17:27:03.841 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.841 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f69cc344:::benchmark_data_smithi179_77138_object98243:head# 2024-02-09T17:27:03.841 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.841 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f69eb156:::benchmark_data_smithi179_77138_object96185:head# 2024-02-09T17:27:03.841 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.841 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f69f31d9:::benchmark_data_smithi179_77138_object124008:head# 2024-02-09T17:27:03.841 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.842 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6a094a3:::benchmark_data_smithi179_77138_object105002:head# 2024-02-09T17:27:03.842 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.842 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6a2520e:::benchmark_data_smithi179_77138_object96003:head# 2024-02-09T17:27:03.842 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.842 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6a25437:::benchmark_data_smithi179_77138_object92338:head# 2024-02-09T17:27:03.842 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.842 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6a28d99:::benchmark_data_smithi179_77138_object25507:head# 2024-02-09T17:27:03.842 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.842 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6a28f86:::benchmark_data_smithi179_77138_object96840:head# 2024-02-09T17:27:03.842 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.842 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6a32c77:::benchmark_data_smithi179_77138_object127678:head# 2024-02-09T17:27:03.842 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.842 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6a440e2:::benchmark_data_smithi179_77138_object129416:head# 2024-02-09T17:27:03.842 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.847 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6a71175:::benchmark_data_smithi179_77138_object32222:head# 2024-02-09T17:27:03.847 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.847 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6a72e02:::benchmark_data_smithi179_77138_object80049:head# 2024-02-09T17:27:03.847 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.848 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6a83edc:::benchmark_data_smithi179_77138_object20818:head# 2024-02-09T17:27:03.848 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.848 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6a8433f:::benchmark_data_smithi179_77138_object88470:head# 2024-02-09T17:27:03.848 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.848 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6a87446:::benchmark_data_smithi179_77138_object113836:head# 2024-02-09T17:27:03.848 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.848 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6a87f14:::benchmark_data_smithi179_77138_object115804:head# 2024-02-09T17:27:03.848 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.848 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6a8c93f:::benchmark_data_smithi179_77138_object115618:head# 2024-02-09T17:27:03.848 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.848 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6a90a1c:::benchmark_data_smithi179_77138_object24247:head# 2024-02-09T17:27:03.848 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.848 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6a93b53:::benchmark_data_smithi179_77138_object20642:head# 2024-02-09T17:27:03.848 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.849 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6a9fab1:::benchmark_data_smithi179_77138_object40863:head# 2024-02-09T17:27:03.849 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.849 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6aa6458:::benchmark_data_smithi179_77138_object47001:head# 2024-02-09T17:27:03.849 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.849 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6aa79cc:::benchmark_data_smithi179_77138_object70162:head# 2024-02-09T17:27:03.849 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.849 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6aa8263:::benchmark_data_smithi179_77138_object123232:head# 2024-02-09T17:27:03.849 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.849 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6aab50f:::benchmark_data_smithi179_77138_object117809:head# 2024-02-09T17:27:03.849 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.849 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6ab2b94:::benchmark_data_smithi179_77138_object31854:head# 2024-02-09T17:27:03.849 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.849 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6abc42d:::benchmark_data_smithi179_77138_object102640:head# 2024-02-09T17:27:03.849 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.850 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6ad4c5a:::benchmark_data_smithi179_77138_object124415:head# 2024-02-09T17:27:03.850 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.850 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6ad4e13:::benchmark_data_smithi179_77138_object54172:head# 2024-02-09T17:27:03.850 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.850 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6ada498:::benchmark_data_smithi179_77138_object63671:head# 2024-02-09T17:27:03.850 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.850 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6ae1763:::benchmark_data_smithi179_77138_object96278:head# 2024-02-09T17:27:03.850 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.850 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6ae7ba5:::benchmark_data_smithi179_77138_object99382:head# 2024-02-09T17:27:03.850 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.850 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6aec981:::benchmark_data_smithi179_77138_object31437:head# 2024-02-09T17:27:03.850 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.850 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6af77c5:::benchmark_data_smithi179_77138_object74157:head# 2024-02-09T17:27:03.850 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.850 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6af824d:::benchmark_data_smithi179_77138_object46610:head# 2024-02-09T17:27:03.851 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.851 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6affd8c:::benchmark_data_smithi179_77138_object79819:head# 2024-02-09T17:27:03.851 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.851 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6b16b1c:::benchmark_data_smithi179_77138_object74712:head# 2024-02-09T17:27:03.851 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.851 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6b181f5:::benchmark_data_smithi179_77138_object84781:head# 2024-02-09T17:27:03.851 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.851 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6b19c0d:::benchmark_data_smithi179_77138_object70670:head# 2024-02-09T17:27:03.851 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.851 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6b32dc8:::benchmark_data_smithi179_77138_object64009:head# 2024-02-09T17:27:03.851 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.851 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6b3573b:::benchmark_data_smithi179_77138_object52952:head# 2024-02-09T17:27:03.851 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.851 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6b3832f:::benchmark_data_smithi179_77138_object85674:head# 2024-02-09T17:27:03.852 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.852 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6b42501:::benchmark_data_smithi179_77138_object54306:head# 2024-02-09T17:27:03.852 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.852 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6b434d1:::benchmark_data_smithi179_77138_object30579:head# 2024-02-09T17:27:03.852 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.852 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6b45b5d:::benchmark_data_smithi179_77138_object113872:head# 2024-02-09T17:27:03.852 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.852 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6b5dd2a:::benchmark_data_smithi179_77138_object24457:head# 2024-02-09T17:27:03.852 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.852 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6b5f08b:::benchmark_data_smithi179_77138_object94103:head# 2024-02-09T17:27:03.852 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.852 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6b66f78:::benchmark_data_smithi179_77138_object24300:head# 2024-02-09T17:27:03.852 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.852 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6b77cbd:::benchmark_data_smithi179_77138_object124223:head# 2024-02-09T17:27:03.852 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.853 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6b8a71a:::benchmark_data_smithi179_77138_object35685:head# 2024-02-09T17:27:03.853 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.853 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6b94bf6:::benchmark_data_smithi179_77138_object20744:head# 2024-02-09T17:27:03.853 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.853 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6b9da7d:::benchmark_data_smithi179_77138_object110692:head# 2024-02-09T17:27:03.853 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.853 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6b9dbc0:::benchmark_data_smithi179_77138_object102697:head# 2024-02-09T17:27:03.853 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.853 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6b9f46d:::benchmark_data_smithi179_77138_object108595:head# 2024-02-09T17:27:03.853 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.854 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6baddc2:::benchmark_data_smithi179_77138_object56039:head# 2024-02-09T17:27:03.855 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.855 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6bae4bf:::benchmark_data_smithi179_77138_object48683:head# 2024-02-09T17:27:03.855 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.855 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6bcc438:::benchmark_data_smithi179_77138_object21325:head# 2024-02-09T17:27:03.855 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.855 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6bd2f06:::benchmark_data_smithi179_77138_object76017:head# 2024-02-09T17:27:03.855 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.855 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6bd5a57:::benchmark_data_smithi179_77138_object125527:head# 2024-02-09T17:27:03.855 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.855 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6bd7b30:::benchmark_data_smithi179_77138_object43237:head# 2024-02-09T17:27:03.855 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.855 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6be318f:::benchmark_data_smithi179_77138_object27167:head# 2024-02-09T17:27:03.855 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.856 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6be7840:::benchmark_data_smithi179_77138_object112041:head# 2024-02-09T17:27:03.856 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.856 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6be7cce:::benchmark_data_smithi179_77138_object55924:head# 2024-02-09T17:27:03.856 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.856 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6bef18a:::benchmark_data_smithi179_77138_object125450:head# 2024-02-09T17:27:03.856 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.856 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6bf8121:::benchmark_data_smithi179_77138_object32382:head# 2024-02-09T17:27:03.856 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.856 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6bfa139:::benchmark_data_smithi179_77138_object76824:head# 2024-02-09T17:27:03.856 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.856 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6c0a8d7:::benchmark_data_smithi179_77138_object86883:head# 2024-02-09T17:27:03.856 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.856 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6c10819:::benchmark_data_smithi179_77138_object119293:head# 2024-02-09T17:27:03.857 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.857 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6c10e44:::benchmark_data_smithi179_77138_object102882:head# 2024-02-09T17:27:03.857 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.857 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6c1e562:::benchmark_data_smithi179_77138_object120682:head# 2024-02-09T17:27:03.857 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.857 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6c26497:::benchmark_data_smithi179_77138_object26946:head# 2024-02-09T17:27:03.857 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.857 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6c2d45d:::benchmark_data_smithi179_77138_object90171:head# 2024-02-09T17:27:03.857 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.857 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6c39cab:::benchmark_data_smithi179_77138_object33840:head# 2024-02-09T17:27:03.857 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.861 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6c62206:::benchmark_data_smithi179_77138_object107000:head# 2024-02-09T17:27:03.862 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.862 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6c65576:::benchmark_data_smithi179_77138_object27709:head# 2024-02-09T17:27:03.862 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.862 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6c679fe:::benchmark_data_smithi179_77138_object46504:head# 2024-02-09T17:27:03.862 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.862 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6c6d243:::benchmark_data_smithi179_77138_object38273:head# 2024-02-09T17:27:03.862 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.862 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6c75942:::benchmark_data_smithi179_77138_object128372:head# 2024-02-09T17:27:03.862 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.862 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6c7878d:::benchmark_data_smithi179_77138_object97399:head# 2024-02-09T17:27:03.862 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.862 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6c7fbba:::benchmark_data_smithi179_77138_object36613:head# 2024-02-09T17:27:03.862 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.863 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6c83028:::benchmark_data_smithi179_77138_object25212:head# 2024-02-09T17:27:03.863 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.863 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6c84200:::benchmark_data_smithi179_77138_object52949:head# 2024-02-09T17:27:03.863 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.863 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6c859aa:::benchmark_data_smithi179_77138_object32565:head# 2024-02-09T17:27:03.863 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.863 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6c8735d:::benchmark_data_smithi179_77138_object71431:head# 2024-02-09T17:27:03.863 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.863 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6c9e8b9:::benchmark_data_smithi179_77138_object105011:head# 2024-02-09T17:27:03.863 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.863 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6ca352a:::benchmark_data_smithi179_77138_object69575:head# 2024-02-09T17:27:03.863 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.863 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6ca3d71:::benchmark_data_smithi179_77138_object95657:head# 2024-02-09T17:27:03.863 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.864 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6ca51db:::benchmark_data_smithi179_77138_object100293:head# 2024-02-09T17:27:03.864 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.864 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6cbd9f9:::benchmark_data_smithi179_77138_object63815:head# 2024-02-09T17:27:03.864 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.865 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6cc5084:::benchmark_data_smithi179_77138_object105124:head# 2024-02-09T17:27:03.865 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.865 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6cc7aaa:::benchmark_data_smithi179_77138_object108295:head# 2024-02-09T17:27:03.865 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.865 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6cce5e7:::benchmark_data_smithi179_77138_object44785:head# 2024-02-09T17:27:03.866 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.866 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6cd1657:::benchmark_data_smithi179_77138_object42575:head# 2024-02-09T17:27:03.866 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.866 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6cd50fd:::benchmark_data_smithi179_77138_object124651:head# 2024-02-09T17:27:03.866 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.866 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6cd7a5e:::benchmark_data_smithi179_77138_object129479:head# 2024-02-09T17:27:03.866 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.866 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6cda484:::benchmark_data_smithi179_77138_object90340:head# 2024-02-09T17:27:03.866 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.866 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6cdbf3e:::benchmark_data_smithi179_77138_object117502:head# 2024-02-09T17:27:03.866 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.866 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6ce1a35:::benchmark_data_smithi179_77138_object36253:head# 2024-02-09T17:27:03.866 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.867 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6cebf0f:::benchmark_data_smithi179_77138_object132337:head# 2024-02-09T17:27:03.868 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.868 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6cedbe8:::benchmark_data_smithi179_77138_object114411:head# 2024-02-09T17:27:03.868 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.868 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6cf99c6:::benchmark_data_smithi179_77138_object98555:head# 2024-02-09T17:27:03.868 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.868 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6d063f6:::benchmark_data_smithi179_77138_object21188:head# 2024-02-09T17:27:03.868 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.868 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6d27416:::benchmark_data_smithi179_77138_object52228:head# 2024-02-09T17:27:03.868 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.868 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6d367c7:::benchmark_data_smithi179_77138_object108359:head# 2024-02-09T17:27:03.868 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.868 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6d3b6bd:::benchmark_data_smithi179_77138_object35107:head# 2024-02-09T17:27:03.868 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.869 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6d41c51:::benchmark_data_smithi179_77138_object68564:head# 2024-02-09T17:27:03.869 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.869 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6d430d5:::benchmark_data_smithi179_77138_object102259:head# 2024-02-09T17:27:03.869 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.869 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6d460bc:::benchmark_data_smithi179_77138_object34167:head# 2024-02-09T17:27:03.869 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.869 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6d47964:::benchmark_data_smithi179_77138_object113852:head# 2024-02-09T17:27:03.869 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.869 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:27:03.873 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6d4c9db:::benchmark_data_smithi179_77138_object125123:head# 2024-02-09T17:27:03.873 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.873 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6d4f3b3:::benchmark_data_smithi179_77138_object104870:head# 2024-02-09T17:27:03.874 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.874 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6d5b694:::benchmark_data_smithi179_77138_object119037:head# 2024-02-09T17:27:03.874 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.874 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6d5cf0c:::benchmark_data_smithi179_77138_object98260:head# 2024-02-09T17:27:03.874 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.874 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6d6cd95:::benchmark_data_smithi179_77138_object94243:head# 2024-02-09T17:27:03.874 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.874 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6d8d9d1:::benchmark_data_smithi179_77138_object110066:head# 2024-02-09T17:27:03.874 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.874 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6d94fd1:::benchmark_data_smithi179_77138_object123292:head# 2024-02-09T17:27:03.874 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.874 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6da6525:::benchmark_data_smithi179_77138_object62193:head# 2024-02-09T17:27:03.874 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.874 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6db8717:::benchmark_data_smithi179_77138_object119742:head# 2024-02-09T17:27:03.874 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.875 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6db9b2d:::benchmark_data_smithi179_77138_object73270:head# 2024-02-09T17:27:03.875 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.875 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6dc1cd4:::benchmark_data_smithi179_77138_object111866:head# 2024-02-09T17:27:03.875 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.875 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6dc9c91:::benchmark_data_smithi179_77138_object131352:head# 2024-02-09T17:27:03.875 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.875 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6dd1979:::benchmark_data_smithi179_77138_object35162:head# 2024-02-09T17:27:03.875 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.875 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6dd1f0f:::benchmark_data_smithi179_77138_object56100:head# 2024-02-09T17:27:03.875 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.875 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6dd5e0e:::benchmark_data_smithi179_77138_object133391:head# 2024-02-09T17:27:03.875 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.875 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6de6e25:::benchmark_data_smithi179_77138_object101460:head# 2024-02-09T17:27:03.875 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.876 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6dfd19d:::benchmark_data_smithi179_77138_object88804:head# 2024-02-09T17:27:03.876 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.876 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6e01882:::benchmark_data_smithi179_77138_object43061:head# 2024-02-09T17:27:03.876 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.876 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6e0280d:::benchmark_data_smithi179_77138_object90956:head# 2024-02-09T17:27:03.876 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.876 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6e0880a:::benchmark_data_smithi179_77138_object53609:head# 2024-02-09T17:27:03.876 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.876 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6e0e2cf:::benchmark_data_smithi179_77138_object97447:head# 2024-02-09T17:27:03.876 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.876 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6e18152:::benchmark_data_smithi179_77138_object122220:head# 2024-02-09T17:27:03.876 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.876 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6e1a435:::benchmark_data_smithi179_77138_object43880:head# 2024-02-09T17:27:03.876 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.878 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6e22cc4:::benchmark_data_smithi179_77138_object107588:head# 2024-02-09T17:27:03.878 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.879 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6e291cf:::benchmark_data_smithi179_77138_object53169:head# 2024-02-09T17:27:03.879 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.879 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6e2a2da:::benchmark_data_smithi179_77138_object75273:head# 2024-02-09T17:27:03.879 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.879 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6e3382a:::benchmark_data_smithi179_77138_object43356:head# 2024-02-09T17:27:03.879 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.879 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6e5c4d1:::benchmark_data_smithi179_77138_object96887:head# 2024-02-09T17:27:03.879 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.879 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6e65a31:::benchmark_data_smithi179_77138_object98740:head# 2024-02-09T17:27:03.879 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.879 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6e68ee1:::benchmark_data_smithi179_77138_object114665:head# 2024-02-09T17:27:03.879 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.879 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6e74b91:::benchmark_data_smithi179_77138_object28097:head# 2024-02-09T17:27:03.879 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.879 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6e74e5c:::benchmark_data_smithi179_77138_object119204:head# 2024-02-09T17:27:03.880 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.880 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6e778a1:::benchmark_data_smithi179_77138_object66934:head# 2024-02-09T17:27:03.880 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.880 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6e81b8a:::benchmark_data_smithi179_77138_object77716:head# 2024-02-09T17:27:03.880 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.880 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6e86cb0:::benchmark_data_smithi179_77138_object60858:head# 2024-02-09T17:27:03.880 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.880 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6e8c93c:::benchmark_data_smithi179_77138_object103036:head# 2024-02-09T17:27:03.880 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.880 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6e9a103:::benchmark_data_smithi179_77138_object94079:head# 2024-02-09T17:27:03.880 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.881 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6e9e575:::benchmark_data_smithi179_77138_object36798:head# 2024-02-09T17:27:03.881 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.881 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6ea18ef:::benchmark_data_smithi179_77138_object74187:head# 2024-02-09T17:27:03.881 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.881 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6ea2797:::benchmark_data_smithi179_77138_object80878:head# 2024-02-09T17:27:03.881 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.884 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6ea31dd:::benchmark_data_smithi179_77138_object51712:head# 2024-02-09T17:27:03.884 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.884 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6ea39f6:::benchmark_data_smithi179_77138_object128363:head# 2024-02-09T17:27:03.884 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.885 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6eb7332:::benchmark_data_smithi179_77138_object52053:head# 2024-02-09T17:27:03.885 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.885 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6ec3020:::benchmark_data_smithi179_77138_object89419:head# 2024-02-09T17:27:03.885 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.885 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6ec9d68:::benchmark_data_smithi179_77138_object123775:head# 2024-02-09T17:27:03.885 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.885 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6edd549:::benchmark_data_smithi179_77138_object43977:head# 2024-02-09T17:27:03.885 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.885 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6edf458:::benchmark_data_smithi179_77138_object54913:head# 2024-02-09T17:27:03.885 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.885 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6ee6148:::benchmark_data_smithi179_77138_object97098:head# 2024-02-09T17:27:03.885 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.885 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6eef147:::benchmark_data_smithi179_77138_object43340:head# 2024-02-09T17:27:03.885 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.886 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6ef0c09:::benchmark_data_smithi179_77138_object33205:head# 2024-02-09T17:27:03.886 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.886 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6ef6f17:::benchmark_data_smithi179_77138_object65672:head# 2024-02-09T17:27:03.886 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.886 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6efddbe:::benchmark_data_smithi179_77138_object70447:head# 2024-02-09T17:27:03.887 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.887 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6eff0b6:::benchmark_data_smithi179_77138_object77334:head# 2024-02-09T17:27:03.887 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.887 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6f04055:::benchmark_data_smithi179_77138_object72647:head# 2024-02-09T17:27:03.887 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.887 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6f151d7:::benchmark_data_smithi179_77138_object50508:head# 2024-02-09T17:27:03.887 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.887 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6f17d5f:::benchmark_data_smithi179_77138_object74308:head# 2024-02-09T17:27:03.887 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.888 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6f269bf:::benchmark_data_smithi179_77138_object113918:head# 2024-02-09T17:27:03.888 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.888 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6f2c845:::benchmark_data_smithi179_77138_object95648:head# 2024-02-09T17:27:03.888 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.888 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6f37e32:::benchmark_data_smithi179_77138_object45795:head# 2024-02-09T17:27:03.888 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.888 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6f38e88:::benchmark_data_smithi179_77138_object109888:head# 2024-02-09T17:27:03.889 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.889 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6f3994d:::benchmark_data_smithi179_77138_object40963:head# 2024-02-09T17:27:03.889 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.889 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6f41e7b:::benchmark_data_smithi179_77138_object106976:head# 2024-02-09T17:27:03.889 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.889 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6f53f92:::benchmark_data_smithi179_77138_object120298:head# 2024-02-09T17:27:03.889 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.889 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6f54114:::benchmark_data_smithi179_77138_object66630:head# 2024-02-09T17:27:03.890 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.893 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6f58774:::benchmark_data_smithi179_77138_object83531:head# 2024-02-09T17:27:03.893 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.893 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6f5e451:::benchmark_data_smithi179_77138_object19459:head# 2024-02-09T17:27:03.894 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.894 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6f615c3:::benchmark_data_smithi179_77138_object52909:head# 2024-02-09T17:27:03.894 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.894 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6f8158c:::benchmark_data_smithi179_77138_object47860:head# 2024-02-09T17:27:03.894 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.894 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6f88548:::benchmark_data_smithi179_77138_object37338:head# 2024-02-09T17:27:03.894 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.894 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6f90148:::benchmark_data_smithi179_77138_object54999:head# 2024-02-09T17:27:03.894 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.894 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6f993c0:::benchmark_data_smithi179_77138_object129926:head# 2024-02-09T17:27:03.894 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.894 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6fb2b00:::benchmark_data_smithi179_77138_object108719:head# 2024-02-09T17:27:03.894 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.894 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6fb6c75:::benchmark_data_smithi179_77138_object104237:head# 2024-02-09T17:27:03.895 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.895 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6fbbb5b:::benchmark_data_smithi179_77138_object50828:head# 2024-02-09T17:27:03.895 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.895 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6fc845a:::benchmark_data_smithi179_77138_object93052:head# 2024-02-09T17:27:03.895 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.895 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6fcd028:::benchmark_data_smithi179_77138_object65675:head# 2024-02-09T17:27:03.895 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.895 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6fcdb96:::benchmark_data_smithi179_77138_object83490:head# 2024-02-09T17:27:03.895 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.896 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6fcfddb:::benchmark_data_smithi179_77138_object38906:head# 2024-02-09T17:27:03.896 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.896 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6ff7e89:::benchmark_data_smithi179_77138_object84585:head# 2024-02-09T17:27:03.896 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.900 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6ffd133:::benchmark_data_smithi179_77138_object124087:head# 2024-02-09T17:27:03.900 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.900 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f6ffe373:::benchmark_data_smithi179_77138_object94550:head# 2024-02-09T17:27:03.900 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.900 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7006677:::benchmark_data_smithi179_77138_object65671:head# 2024-02-09T17:27:03.900 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.900 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f70092df:::benchmark_data_smithi179_77138_object123287:head# 2024-02-09T17:27:03.900 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.900 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f700b84e:::benchmark_data_smithi179_77138_object29335:head# 2024-02-09T17:27:03.900 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.901 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7011d1d:::benchmark_data_smithi179_77138_object104135:head# 2024-02-09T17:27:03.901 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.901 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f70153e5:::benchmark_data_smithi179_77138_object105112:head# 2024-02-09T17:27:03.901 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.901 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f701842d:::benchmark_data_smithi179_77138_object99358:head# 2024-02-09T17:27:03.901 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.901 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7026d3d:::benchmark_data_smithi179_77138_object72441:head# 2024-02-09T17:27:03.901 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.901 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f703b64e:::benchmark_data_smithi179_77138_object76098:head# 2024-02-09T17:27:03.901 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.901 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f703e408:::benchmark_data_smithi179_77138_object78104:head# 2024-02-09T17:27:03.901 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.901 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f70423f3:::benchmark_data_smithi179_77138_object30480:head# 2024-02-09T17:27:03.901 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.901 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7042b9d:::benchmark_data_smithi179_77138_object116515:head# 2024-02-09T17:27:03.902 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.902 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f704509a:::benchmark_data_smithi179_77138_object87146:head# 2024-02-09T17:27:03.902 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.903 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7048e4d:::benchmark_data_smithi179_77138_object71539:head# 2024-02-09T17:27:03.903 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.904 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7051fda:::benchmark_data_smithi179_77138_object98527:head# 2024-02-09T17:27:03.904 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.904 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7052900:::benchmark_data_smithi179_77138_object65447:head# 2024-02-09T17:27:03.904 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.905 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f70589f9:::benchmark_data_smithi179_77138_object85957:head# 2024-02-09T17:27:03.905 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.905 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f705ba47:::benchmark_data_smithi179_77138_object95457:head# 2024-02-09T17:27:03.905 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.905 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f70685af:::benchmark_data_smithi179_77138_object79980:head# 2024-02-09T17:27:03.905 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.905 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7069aac:::benchmark_data_smithi179_77138_object49406:head# 2024-02-09T17:27:03.906 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.906 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f706ca68:::benchmark_data_smithi179_77138_object101293:head# 2024-02-09T17:27:03.906 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.907 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7071277:::benchmark_data_smithi179_77138_object97862:head# 2024-02-09T17:27:03.907 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.908 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f707879f:::benchmark_data_smithi179_77138_object79141:head# 2024-02-09T17:27:03.908 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.908 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f707ed75:::benchmark_data_smithi179_77138_object30444:head# 2024-02-09T17:27:03.908 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.908 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7081086:::benchmark_data_smithi179_77138_object47037:head# 2024-02-09T17:27:03.908 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.908 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7082436:::benchmark_data_smithi179_77138_object81091:head# 2024-02-09T17:27:03.908 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.908 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7085425:::benchmark_data_smithi179_77138_object108897:head# 2024-02-09T17:27:03.908 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.908 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7089deb:::benchmark_data_smithi179_77138_object21447:head# 2024-02-09T17:27:03.908 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.912 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7098ff9:::benchmark_data_smithi179_77138_object42817:head# 2024-02-09T17:27:03.913 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.913 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f709da36:::benchmark_data_smithi179_77138_object37052:head# 2024-02-09T17:27:03.913 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.913 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f70a560f:::benchmark_data_smithi179_77138_object89556:head# 2024-02-09T17:27:03.913 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.913 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f70aafed:::benchmark_data_smithi179_77138_object69751:head# 2024-02-09T17:27:03.913 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.913 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f70aec77:::benchmark_data_smithi179_77138_object78531:head# 2024-02-09T17:27:03.913 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.913 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f70af6d4:::benchmark_data_smithi179_77138_object116725:head# 2024-02-09T17:27:03.913 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.913 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f70bea4e:::benchmark_data_smithi179_77138_object52854:head# 2024-02-09T17:27:03.913 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.914 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f70c647b:::benchmark_data_smithi179_77138_object73725:head# 2024-02-09T17:27:03.914 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.914 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f70cd54a:::benchmark_data_smithi179_77138_object100585:head# 2024-02-09T17:27:03.914 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.914 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f70d264e:::benchmark_data_smithi179_77138_object109247:head# 2024-02-09T17:27:03.914 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.914 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f70daf5f:::benchmark_data_smithi179_77138_object101225:head# 2024-02-09T17:27:03.914 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.914 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f70df852:::benchmark_data_smithi179_77138_object82252:head# 2024-02-09T17:27:03.914 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.914 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f70e1914:::benchmark_data_smithi179_77138_object124206:head# 2024-02-09T17:27:03.914 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.914 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f70e2dc5:::benchmark_data_smithi179_77138_object30095:head# 2024-02-09T17:27:03.914 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.914 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f70e9329:::benchmark_data_smithi179_77138_object23169:head# 2024-02-09T17:27:03.915 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.915 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f70f1bbc:::benchmark_data_smithi179_77138_object125907:head# 2024-02-09T17:27:03.915 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.915 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f70f7bef:::benchmark_data_smithi179_77138_object44257:head# 2024-02-09T17:27:03.915 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.915 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f70f7df7:::benchmark_data_smithi179_77138_object89606:head# 2024-02-09T17:27:03.916 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.916 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7112cb9:::benchmark_data_smithi179_77138_object115283:head# 2024-02-09T17:27:03.916 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.916 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f71191e4:::benchmark_data_smithi179_77138_object56352:head# 2024-02-09T17:27:03.916 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.917 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7122279:::benchmark_data_smithi179_77138_object55459:head# 2024-02-09T17:27:03.917 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.917 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7129f79:::benchmark_data_smithi179_77138_object64330:head# 2024-02-09T17:27:03.917 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.917 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f713920b:::benchmark_data_smithi179_77138_object24636:head# 2024-02-09T17:27:03.917 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.917 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7142240:::benchmark_data_smithi179_77138_object112918:head# 2024-02-09T17:27:03.918 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.918 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f714db7f:::benchmark_data_smithi179_77138_object91819:head# 2024-02-09T17:27:03.918 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.918 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7152e85:::benchmark_data_smithi179_77138_object38358:head# 2024-02-09T17:27:03.918 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.918 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f715ee71:::benchmark_data_smithi179_77138_object88263:head# 2024-02-09T17:27:03.918 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.918 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f716c368:::benchmark_data_smithi179_77138_object46211:head# 2024-02-09T17:27:03.918 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.918 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f716f81c:::benchmark_data_smithi179_77138_object20253:head# 2024-02-09T17:27:03.918 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.918 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7170992:::benchmark_data_smithi179_77138_object96223:head# 2024-02-09T17:27:03.918 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.918 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7171a5f:::benchmark_data_smithi179_77138_object130201:head# 2024-02-09T17:27:03.918 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.919 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7187715:::benchmark_data_smithi179_77138_object33721:head# 2024-02-09T17:27:03.919 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.919 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f718cd2a:::benchmark_data_smithi179_77138_object101497:head# 2024-02-09T17:27:03.919 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.919 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f719c6fc:::benchmark_data_smithi179_77138_object121824:head# 2024-02-09T17:27:03.920 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.920 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f719c7d4:::benchmark_data_smithi179_77138_object108717:head# 2024-02-09T17:27:03.920 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.920 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f71a47f4:::benchmark_data_smithi179_77138_object92605:head# 2024-02-09T17:27:03.920 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.920 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:03.924 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f71a8c10:::benchmark_data_smithi179_77138_object131907:head# 2024-02-09T17:27:03.924 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.924 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f71aa14c:::benchmark_data_smithi179_77138_object46329:head# 2024-02-09T17:27:03.925 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.925 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f71ac567:::benchmark_data_smithi179_77138_object49119:head# 2024-02-09T17:27:03.925 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.925 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f71bd0ff:::benchmark_data_smithi179_77138_object124256:head# 2024-02-09T17:27:03.925 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.925 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f71bd48c:::benchmark_data_smithi179_77138_object99215:head# 2024-02-09T17:27:03.925 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.925 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f71d949e:::benchmark_data_smithi179_77138_object55035:head# 2024-02-09T17:27:03.925 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.925 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f71e0057:::benchmark_data_smithi179_77138_object124107:head# 2024-02-09T17:27:03.925 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.925 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f71e35b2:::benchmark_data_smithi179_77138_object85397:head# 2024-02-09T17:27:03.925 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.925 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f71e3d8e:::benchmark_data_smithi179_77138_object96355:head# 2024-02-09T17:27:03.925 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.925 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f71e898b:::benchmark_data_smithi179_77138_object47945:head# 2024-02-09T17:27:03.926 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.926 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f71fbd51:::benchmark_data_smithi179_77138_object41594:head# 2024-02-09T17:27:03.926 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.926 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f71fd3fe:::benchmark_data_smithi179_77138_object38941:head# 2024-02-09T17:27:03.926 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.926 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f720c7d8:::benchmark_data_smithi179_77138_object66183:head# 2024-02-09T17:27:03.926 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.926 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f72174b6:::benchmark_data_smithi179_77138_object33248:head# 2024-02-09T17:27:03.926 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.926 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f721904a:::benchmark_data_smithi179_77138_object82984:head# 2024-02-09T17:27:03.926 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.926 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f721aa31:::benchmark_data_smithi179_77138_object121296:head# 2024-02-09T17:27:03.926 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.926 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f722dd5b:::benchmark_data_smithi179_77138_object53254:head# 2024-02-09T17:27:03.926 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.926 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7235016:::benchmark_data_smithi179_77138_object112678:head# 2024-02-09T17:27:03.926 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.927 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f72389ab:::benchmark_data_smithi179_77138_object31273:head# 2024-02-09T17:27:03.927 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.927 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7241895:::benchmark_data_smithi179_77138_object62108:head# 2024-02-09T17:27:03.927 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.927 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7255478:::benchmark_data_smithi179_77138_object32147:head# 2024-02-09T17:27:03.927 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.927 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7256b03:::benchmark_data_smithi179_77138_object103163:head# 2024-02-09T17:27:03.927 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.927 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f725c601:::benchmark_data_smithi179_77138_object83900:head# 2024-02-09T17:27:03.927 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.927 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7261caf:::benchmark_data_smithi179_77138_object71441:head# 2024-02-09T17:27:03.927 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.927 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7265695:::benchmark_data_smithi179_77138_object107502:head# 2024-02-09T17:27:03.927 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:03 smithi162 ceph-mon[44304]: pgmap v459: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 5.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 900 op/s; 24390/290988 objects degraded (8.382%) 2024-02-09T17:27:03.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:03 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2213779330' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:27:03.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:03 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:27:03.930 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7267933:::benchmark_data_smithi179_77138_object35394:head# 2024-02-09T17:27:03.930 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.930 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f72694fa:::benchmark_data_smithi179_77138_object30687:head# 2024-02-09T17:27:03.930 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.931 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f726e601:::benchmark_data_smithi179_77138_object37062:head# 2024-02-09T17:27:03.931 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.932 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7274e48:::benchmark_data_smithi179_77138_object30197:head# 2024-02-09T17:27:03.932 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.936 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f727aa7e:::benchmark_data_smithi179_77138_object132642:head# 2024-02-09T17:27:03.936 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.937 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f727b56c:::benchmark_data_smithi179_77138_object111833:head# 2024-02-09T17:27:03.937 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.937 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f727f573:::benchmark_data_smithi179_77138_object40179:head# 2024-02-09T17:27:03.937 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.938 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7285e1b:::benchmark_data_smithi179_77138_object56240:head# 2024-02-09T17:27:03.938 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.939 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f728659b:::benchmark_data_smithi179_77138_object98484:head# 2024-02-09T17:27:03.939 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.939 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f729fa34:::benchmark_data_smithi179_77138_object110082:head# 2024-02-09T17:27:03.939 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.939 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f72a1082:::benchmark_data_smithi179_77138_object123213:head# 2024-02-09T17:27:03.940 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.940 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f72a1a7e:::benchmark_data_smithi179_77138_object21486:head# 2024-02-09T17:27:03.940 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.940 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f72aab62:::benchmark_data_smithi179_77138_object120533:head# 2024-02-09T17:27:03.940 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.940 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f72c7b6c:::benchmark_data_smithi179_77138_object57500:head# 2024-02-09T17:27:03.940 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.942 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f72d1f3f:::benchmark_data_smithi179_77138_object92634:head# 2024-02-09T17:27:03.942 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.942 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f72d413c:::benchmark_data_smithi179_77138_object124443:head# 2024-02-09T17:27:03.942 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.942 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f72eb6c9:::benchmark_data_smithi179_77138_object126995:head# 2024-02-09T17:27:03.942 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.942 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f72ebf3b:::benchmark_data_smithi179_77138_object72368:head# 2024-02-09T17:27:03.942 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.942 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f72f2849:::benchmark_data_smithi179_77138_object114391:head# 2024-02-09T17:27:03.942 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.942 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f72f3a96:::benchmark_data_smithi179_77138_object28094:head# 2024-02-09T17:27:03.942 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.943 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f72fdb4f:::benchmark_data_smithi179_77138_object24253:head# 2024-02-09T17:27:03.943 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.943 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7302765:::benchmark_data_smithi179_77138_object121091:head# 2024-02-09T17:27:03.943 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.943 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f730d4af:::benchmark_data_smithi179_77138_object49728:head# 2024-02-09T17:27:03.943 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.943 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f731896f:::benchmark_data_smithi179_77138_object110840:head# 2024-02-09T17:27:03.943 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.944 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7319972:::benchmark_data_smithi179_77138_object82922:head# 2024-02-09T17:27:03.944 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.944 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7322620:::benchmark_data_smithi179_77138_object108236:head# 2024-02-09T17:27:03.944 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.944 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7323356:::benchmark_data_smithi179_77138_object100040:head# 2024-02-09T17:27:03.945 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.945 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f73242a3:::benchmark_data_smithi179_77138_object41404:head# 2024-02-09T17:27:03.945 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.945 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7338f29:::benchmark_data_smithi179_77138_object67542:head# 2024-02-09T17:27:03.945 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.945 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7345c13:::benchmark_data_smithi179_77138_object108675:head# 2024-02-09T17:27:03.945 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.946 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f734fe42:::benchmark_data_smithi179_77138_object54329:head# 2024-02-09T17:27:03.946 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.946 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f735562a:::benchmark_data_smithi179_77138_object34987:head# 2024-02-09T17:27:03.946 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.946 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7357261:::benchmark_data_smithi179_77138_object44747:head# 2024-02-09T17:27:03.946 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.946 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f735f034:::benchmark_data_smithi179_77138_object35777:head# 2024-02-09T17:27:03.946 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.946 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f73620a0:::benchmark_data_smithi179_77138_object33704:head# 2024-02-09T17:27:03.946 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.947 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7373ead:::benchmark_data_smithi179_77138_object103017:head# 2024-02-09T17:27:03.947 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.947 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7375f91:::benchmark_data_smithi179_77138_object54853:head# 2024-02-09T17:27:03.947 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.947 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7385284:::benchmark_data_smithi179_77138_object82503:head# 2024-02-09T17:27:03.947 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.947 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7386019:::benchmark_data_smithi179_77138_object90365:head# 2024-02-09T17:27:03.947 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.947 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:27:03.948 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f739b11b:::benchmark_data_smithi179_77138_object48055:head# 2024-02-09T17:27:03.949 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.949 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f73a43d1:::benchmark_data_smithi179_77138_object47274:head# 2024-02-09T17:27:03.949 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.949 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f73a5e4e:::benchmark_data_smithi179_77138_object38793:head# 2024-02-09T17:27:03.949 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.949 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f73b0b83:::benchmark_data_smithi179_77138_object33788:head# 2024-02-09T17:27:03.949 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.949 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f73b1941:::benchmark_data_smithi179_77138_object61062:head# 2024-02-09T17:27:03.949 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.949 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f73b2810:::benchmark_data_smithi179_77138_object43745:head# 2024-02-09T17:27:03.949 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.949 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f73b84e1:::benchmark_data_smithi179_77138_object92886:head# 2024-02-09T17:27:03.949 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:03.950 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:27:03.958 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f73b8d01:::benchmark_data_smithi179_77138_object99106:head# 2024-02-09T17:27:03.958 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.958 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f73c0521:::benchmark_data_smithi179_77138_object59841:head# 2024-02-09T17:27:03.958 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.958 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f73c1e25:::benchmark_data_smithi179_77138_object46401:head# 2024-02-09T17:27:03.958 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.958 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f73c55e8:::benchmark_data_smithi179_77138_object67789:head# 2024-02-09T17:27:03.958 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.958 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f73c5cbe:::benchmark_data_smithi179_77138_object52882:head# 2024-02-09T17:27:03.958 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.959 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f73c6d1b:::benchmark_data_smithi179_77138_object127715:head# 2024-02-09T17:27:03.959 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.959 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f73d00d4:::benchmark_data_smithi179_77138_object54219:head# 2024-02-09T17:27:03.959 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.959 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f73d2d2f:::benchmark_data_smithi179_77138_object37801:head# 2024-02-09T17:27:03.959 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.959 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f73dc7b7:::benchmark_data_smithi179_77138_object66359:head# 2024-02-09T17:27:03.959 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.959 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f73ea559:::benchmark_data_smithi179_77138_object85646:head# 2024-02-09T17:27:03.959 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.959 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f73eb815:::benchmark_data_smithi179_77138_object60031:head# 2024-02-09T17:27:03.959 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.959 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f73f2dae:::benchmark_data_smithi179_77138_object107713:head# 2024-02-09T17:27:03.959 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.959 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f73f4262:::benchmark_data_smithi179_77138_object74167:head# 2024-02-09T17:27:03.960 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.960 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f73fbf86:::benchmark_data_smithi179_77138_object22998:head# 2024-02-09T17:27:03.960 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.960 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f74068d9:::benchmark_data_smithi179_77138_object120934:head# 2024-02-09T17:27:03.960 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.960 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f740d527:::benchmark_data_smithi179_77138_object23346:head# 2024-02-09T17:27:03.960 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.960 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7416d7a:::benchmark_data_smithi179_77138_object61223:head# 2024-02-09T17:27:03.960 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.960 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f74305f7:::benchmark_data_smithi179_77138_object45059:head# 2024-02-09T17:27:03.960 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.960 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7453703:::benchmark_data_smithi179_77138_object60301:head# 2024-02-09T17:27:03.960 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.960 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7457eb3:::benchmark_data_smithi179_77138_object77577:head# 2024-02-09T17:27:03.961 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.961 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f745ff68:::benchmark_data_smithi179_77138_object96264:head# 2024-02-09T17:27:03.961 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.961 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7467cfa:::benchmark_data_smithi179_77138_object60459:head# 2024-02-09T17:27:03.961 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.961 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f746971b:::benchmark_data_smithi179_77138_object57349:head# 2024-02-09T17:27:03.961 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.961 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f746a075:::benchmark_data_smithi179_77138_object22069:head# 2024-02-09T17:27:03.961 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.961 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f746e382:::benchmark_data_smithi179_77138_object35851:head# 2024-02-09T17:27:03.961 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.963 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7476037:::benchmark_data_smithi179_77138_object45403:head# 2024-02-09T17:27:03.963 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.963 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f74784ce:::benchmark_data_smithi179_77138_object42852:head# 2024-02-09T17:27:03.963 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.963 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7486855:::benchmark_data_smithi179_77138_object29051:head# 2024-02-09T17:27:03.963 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.963 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7488c08:::benchmark_data_smithi179_77138_object116816:head# 2024-02-09T17:27:03.963 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.963 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f748c861:::benchmark_data_smithi179_77138_object120301:head# 2024-02-09T17:27:03.963 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.963 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f749b282:::benchmark_data_smithi179_77138_object25075:head# 2024-02-09T17:27:03.964 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.964 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f74ac6d7:::benchmark_data_smithi179_77138_object23493:head# 2024-02-09T17:27:03.964 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.964 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f74afddd:::benchmark_data_smithi179_77138_object32252:head# 2024-02-09T17:27:03.964 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.964 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f74b79a4:::benchmark_data_smithi179_77138_object20628:head# 2024-02-09T17:27:03.964 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.964 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f74b7d6a:::benchmark_data_smithi179_77138_object113396:head# 2024-02-09T17:27:03.964 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.964 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f74b86e1:::benchmark_data_smithi179_77138_object25961:head# 2024-02-09T17:27:03.964 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.964 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f74b965c:::benchmark_data_smithi179_77138_object63464:head# 2024-02-09T17:27:03.964 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.964 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f74c0af7:::benchmark_data_smithi179_77138_object108723:head# 2024-02-09T17:27:03.964 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.965 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f74d55f6:::benchmark_data_smithi179_77138_object55157:head# 2024-02-09T17:27:03.965 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.965 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f74dbdd1:::benchmark_data_smithi179_77138_object37065:head# 2024-02-09T17:27:03.965 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.965 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f74ec5da:::benchmark_data_smithi179_77138_object75133:head# 2024-02-09T17:27:03.965 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.965 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f74ee449:::benchmark_data_smithi179_77138_object35251:head# 2024-02-09T17:27:03.965 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.965 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f74ee770:::benchmark_data_smithi179_77138_object27509:head# 2024-02-09T17:27:03.965 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.965 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f74ef86a:::benchmark_data_smithi179_77138_object53115:head# 2024-02-09T17:27:03.965 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.965 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f74f3560:::benchmark_data_smithi179_77138_object29710:head# 2024-02-09T17:27:03.965 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.965 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f74f6a3e:::benchmark_data_smithi179_77138_object27211:head# 2024-02-09T17:27:03.966 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.966 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f74f783d:::benchmark_data_smithi179_77138_object23303:head# 2024-02-09T17:27:03.966 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.966 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f74fd95a:::benchmark_data_smithi179_77138_object132019:head# 2024-02-09T17:27:03.966 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.966 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f74ff70c:::benchmark_data_smithi179_77138_object46604:head# 2024-02-09T17:27:03.966 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.970 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7501d38:::benchmark_data_smithi179_77138_object101178:head# 2024-02-09T17:27:03.970 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.970 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f750c1b8:::benchmark_data_smithi179_77138_object31720:head# 2024-02-09T17:27:03.970 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.971 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f751fa30:::benchmark_data_smithi179_77138_object93041:head# 2024-02-09T17:27:03.971 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.971 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7526673:::benchmark_data_smithi179_77138_object77840:head# 2024-02-09T17:27:03.971 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.971 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7538bce:::benchmark_data_smithi179_77138_object119036:head# 2024-02-09T17:27:03.971 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.971 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7538dfd:::benchmark_data_smithi179_77138_object37626:head# 2024-02-09T17:27:03.971 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.971 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f753cbeb:::benchmark_data_smithi179_77138_object104332:head# 2024-02-09T17:27:03.971 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.971 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f75408f3:::benchmark_data_smithi179_77138_object109881:head# 2024-02-09T17:27:03.971 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.971 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f754c16f:::benchmark_data_smithi179_77138_object69755:head# 2024-02-09T17:27:03.971 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.972 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f75563b9:::benchmark_data_smithi179_77138_object119231:head# 2024-02-09T17:27:03.972 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.972 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f75589fc:::benchmark_data_smithi179_77138_object30169:head# 2024-02-09T17:27:03.972 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.972 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f756e6e0:::benchmark_data_smithi179_77138_object27501:head# 2024-02-09T17:27:03.972 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.972 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f75765e5:::benchmark_data_smithi179_77138_object25672:head# 2024-02-09T17:27:03.972 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.972 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7579200:::benchmark_data_smithi179_77138_object52675:head# 2024-02-09T17:27:03.972 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.972 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f757f864:::benchmark_data_smithi179_77138_object59493:head# 2024-02-09T17:27:03.972 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.972 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f75913f7:::benchmark_data_smithi179_77138_object131004:head# 2024-02-09T17:27:03.972 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.972 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7596b39:::benchmark_data_smithi179_77138_object79608:head# 2024-02-09T17:27:03.973 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.973 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f759bd19:::benchmark_data_smithi179_77138_object76349:head# 2024-02-09T17:27:03.973 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.973 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f75a3a48:::benchmark_data_smithi179_77138_object63222:head# 2024-02-09T17:27:03.973 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.973 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f75b62d2:::benchmark_data_smithi179_77138_object54098:head# 2024-02-09T17:27:03.973 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.973 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f75bcf5f:::benchmark_data_smithi179_77138_object32359:head# 2024-02-09T17:27:03.973 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.973 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f75c9efe:::benchmark_data_smithi179_77138_object36593:head# 2024-02-09T17:27:03.973 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.973 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f75df61f:::benchmark_data_smithi179_77138_object102184:head# 2024-02-09T17:27:03.973 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.973 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f75f4ae2:::benchmark_data_smithi179_77138_object60286:head# 2024-02-09T17:27:03.973 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.973 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f760287b:::benchmark_data_smithi179_77138_object129003:head# 2024-02-09T17:27:03.974 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.974 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f760d677:::benchmark_data_smithi179_77138_object72848:head# 2024-02-09T17:27:03.974 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.974 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7610cb3:::benchmark_data_smithi179_77138_object21335:head# 2024-02-09T17:27:03.974 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.974 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7611a7e:::benchmark_data_smithi179_77138_object48002:head# 2024-02-09T17:27:03.974 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.974 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7618531:::benchmark_data_smithi179_77138_object87481:head# 2024-02-09T17:27:03.974 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.974 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f761e5ac:::benchmark_data_smithi179_77138_object27322:head# 2024-02-09T17:27:03.974 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.976 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7631e47:::benchmark_data_smithi179_77138_object68943:head# 2024-02-09T17:27:03.976 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.976 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f763795e:::benchmark_data_smithi179_77138_object34829:head# 2024-02-09T17:27:03.976 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.976 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f763a676:::benchmark_data_smithi179_77138_object47628:head# 2024-02-09T17:27:03.976 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.976 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f764318d:::benchmark_data_smithi179_77138_object70781:head# 2024-02-09T17:27:03.976 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.976 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f764464c:::benchmark_data_smithi179_77138_object77890:head# 2024-02-09T17:27:03.977 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.977 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7645b2a:::benchmark_data_smithi179_77138_object54433:head# 2024-02-09T17:27:03.977 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.977 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f765124b:::benchmark_data_smithi179_77138_object66327:head# 2024-02-09T17:27:03.977 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.977 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7666c78:::benchmark_data_smithi179_77138_object126829:head# 2024-02-09T17:27:03.977 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.977 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f766ee88:::benchmark_data_smithi179_77138_object51919:head# 2024-02-09T17:27:03.977 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.977 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f767190d:::benchmark_data_smithi179_77138_object87302:head# 2024-02-09T17:27:03.977 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.977 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7676b37:::benchmark_data_smithi179_77138_object94238:head# 2024-02-09T17:27:03.977 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.977 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f767e556:::benchmark_data_smithi179_77138_object59363:head# 2024-02-09T17:27:03.978 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.978 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7686c88:::benchmark_data_smithi179_77138_object87997:head# 2024-02-09T17:27:03.978 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.978 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f768f09a:::benchmark_data_smithi179_77138_object127542:head# 2024-02-09T17:27:03.978 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.978 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7691c32:::benchmark_data_smithi179_77138_object90188:head# 2024-02-09T17:27:03.978 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.978 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f76aa06c:::benchmark_data_smithi179_77138_object70837:head# 2024-02-09T17:27:03.978 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.978 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f76ac8dc:::benchmark_data_smithi179_77138_object117469:head# 2024-02-09T17:27:03.978 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.978 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f76bc0cb:::benchmark_data_smithi179_77138_object129700:head# 2024-02-09T17:27:03.978 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.978 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f76c697c:::benchmark_data_smithi179_77138_object97974:head# 2024-02-09T17:27:03.979 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.979 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f76c942e:::benchmark_data_smithi179_77138_object28165:head# 2024-02-09T17:27:03.979 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.979 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f76cb6c9:::benchmark_data_smithi179_77138_object52426:head# 2024-02-09T17:27:03.979 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.979 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f76cd64f:::benchmark_data_smithi179_77138_object131641:head# 2024-02-09T17:27:03.979 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.979 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f76ce8d9:::benchmark_data_smithi179_77138_object38794:head# 2024-02-09T17:27:03.979 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.979 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f76e0d58:::benchmark_data_smithi179_77138_object130773:head# 2024-02-09T17:27:03.979 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.985 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7705859:::benchmark_data_smithi179_77138_object30374:head# 2024-02-09T17:27:03.985 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.985 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f770d442:::benchmark_data_smithi179_77138_object20130:head# 2024-02-09T17:27:03.985 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.985 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f771c2cc:::benchmark_data_smithi179_77138_object76612:head# 2024-02-09T17:27:03.985 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.985 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f771e920:::benchmark_data_smithi179_77138_object77202:head# 2024-02-09T17:27:03.985 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.986 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f772204d:::benchmark_data_smithi179_77138_object54101:head# 2024-02-09T17:27:03.986 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.986 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7729790:::benchmark_data_smithi179_77138_object113309:head# 2024-02-09T17:27:03.986 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.986 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7730af5:::benchmark_data_smithi179_77138_object71665:head# 2024-02-09T17:27:03.986 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.986 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f77343bc:::benchmark_data_smithi179_77138_object25111:head# 2024-02-09T17:27:03.986 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.986 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f773a523:::benchmark_data_smithi179_77138_object116040:head# 2024-02-09T17:27:03.986 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.986 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f77424f4:::benchmark_data_smithi179_77138_object110469:head# 2024-02-09T17:27:03.986 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.986 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f774d50b:::benchmark_data_smithi179_77138_object55822:head# 2024-02-09T17:27:03.986 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.987 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f775050f:::benchmark_data_smithi179_77138_object104525:head# 2024-02-09T17:27:03.987 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.987 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f77578d3:::benchmark_data_smithi179_77138_object114719:head# 2024-02-09T17:27:03.987 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.987 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7757fa8:::benchmark_data_smithi179_77138_object26901:head# 2024-02-09T17:27:03.987 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.987 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f77581f4:::benchmark_data_smithi179_77138_object123781:head# 2024-02-09T17:27:03.987 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.987 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7778c89:::benchmark_data_smithi179_77138_object20142:head# 2024-02-09T17:27:03.987 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.987 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7780aef:::benchmark_data_smithi179_77138_object46824:head# 2024-02-09T17:27:03.987 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.987 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f77826ec:::benchmark_data_smithi179_77138_object28887:head# 2024-02-09T17:27:03.987 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.987 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7786418:::benchmark_data_smithi179_77138_object63660:head# 2024-02-09T17:27:03.988 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.988 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f779cd19:::benchmark_data_smithi179_77138_object124836:head# 2024-02-09T17:27:03.988 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.988 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f77acbb3:::benchmark_data_smithi179_77138_object100026:head# 2024-02-09T17:27:03.988 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.988 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f77bc716:::benchmark_data_smithi179_77138_object35263:head# 2024-02-09T17:27:03.988 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.988 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f77c48ee:::benchmark_data_smithi179_77138_object73223:head# 2024-02-09T17:27:03.988 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.988 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f77c574f:::benchmark_data_smithi179_77138_object58925:head# 2024-02-09T17:27:03.988 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.988 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f77d5c43:::benchmark_data_smithi179_77138_object113689:head# 2024-02-09T17:27:03.988 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.988 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f77d7b0e:::benchmark_data_smithi179_77138_object82784:head# 2024-02-09T17:27:03.989 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.989 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f77de48e:::benchmark_data_smithi179_77138_object25926:head# 2024-02-09T17:27:03.989 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.989 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f77e4e12:::benchmark_data_smithi179_77138_object127818:head# 2024-02-09T17:27:03.989 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.989 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f77e5bcc:::benchmark_data_smithi179_77138_object33667:head# 2024-02-09T17:27:03.989 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.989 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f77f013f:::benchmark_data_smithi179_77138_object41788:head# 2024-02-09T17:27:03.989 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.989 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f77f5188:::benchmark_data_smithi179_77138_object130313:head# 2024-02-09T17:27:03.989 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.989 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f77fb7bf:::benchmark_data_smithi179_77138_object31885:head# 2024-02-09T17:27:03.989 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.989 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7804850:::benchmark_data_smithi179_77138_object66571:head# 2024-02-09T17:27:03.989 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.990 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f780aa42:::benchmark_data_smithi179_77138_object23729:head# 2024-02-09T17:27:03.990 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.990 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7814243:::benchmark_data_smithi179_77138_object104796:head# 2024-02-09T17:27:03.990 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.990 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7829c0b:::benchmark_data_smithi179_77138_object82036:head# 2024-02-09T17:27:03.990 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.990 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7832bab:::benchmark_data_smithi179_77138_object77757:head# 2024-02-09T17:27:03.990 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.995 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f78331a7:::benchmark_data_smithi179_77138_object95196:head# 2024-02-09T17:27:03.995 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.995 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f783638f:::benchmark_data_smithi179_77138_object95786:head# 2024-02-09T17:27:03.995 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.995 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f784e830:::benchmark_data_smithi179_77138_object31748:head# 2024-02-09T17:27:03.995 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.995 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f786336e:::benchmark_data_smithi179_77138_object27031:head# 2024-02-09T17:27:03.996 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.996 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f78669ce:::benchmark_data_smithi179_77138_object98835:head# 2024-02-09T17:27:03.996 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.996 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7867440:::benchmark_data_smithi179_77138_object64773:head# 2024-02-09T17:27:03.996 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.996 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f786c978:::benchmark_data_smithi179_77138_object27303:head# 2024-02-09T17:27:03.996 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.996 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7873f6a:::benchmark_data_smithi179_77138_object53999:head# 2024-02-09T17:27:03.996 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.996 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f787b819:::benchmark_data_smithi179_77138_object132355:head# 2024-02-09T17:27:03.996 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.996 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7882fdd:::benchmark_data_smithi179_77138_object127473:head# 2024-02-09T17:27:03.996 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.996 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7893ac7:::benchmark_data_smithi179_77138_object130813:head# 2024-02-09T17:27:03.997 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.997 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f789e076:::benchmark_data_smithi179_77138_object95532:head# 2024-02-09T17:27:03.997 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.997 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f78a0461:::benchmark_data_smithi179_77138_object46776:head# 2024-02-09T17:27:03.997 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.997 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f78a6c79:::benchmark_data_smithi179_77138_object100964:head# 2024-02-09T17:27:03.997 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.997 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f78a7069:::benchmark_data_smithi179_77138_object119924:head# 2024-02-09T17:27:03.997 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.997 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f78b2389:::benchmark_data_smithi179_77138_object70801:head# 2024-02-09T17:27:03.997 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.997 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f78c9690:::benchmark_data_smithi179_77138_object85034:head# 2024-02-09T17:27:03.997 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.997 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f78d2eda:::benchmark_data_smithi179_77138_object30654:head# 2024-02-09T17:27:03.997 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.998 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f78da392:::benchmark_data_smithi179_77138_object25132:head# 2024-02-09T17:27:03.998 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.998 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f78db402:::benchmark_data_smithi179_77138_object95109:head# 2024-02-09T17:27:03.998 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.998 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f78e34b2:::benchmark_data_smithi179_77138_object60544:head# 2024-02-09T17:27:03.998 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.998 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f78e96ab:::benchmark_data_smithi179_77138_object19956:head# 2024-02-09T17:27:03.998 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.998 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f78eb87e:::benchmark_data_smithi179_77138_object81911:head# 2024-02-09T17:27:03.998 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.998 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f78f03f9:::benchmark_data_smithi179_77138_object29391:head# 2024-02-09T17:27:03.998 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.998 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f78f571c:::benchmark_data_smithi179_77138_object66982:head# 2024-02-09T17:27:03.998 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.999 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f78f583e:::benchmark_data_smithi179_77138_object57547:head# 2024-02-09T17:27:03.999 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.999 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f78fb029:::benchmark_data_smithi179_77138_object65047:head# 2024-02-09T17:27:03.999 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.999 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f790ba4a:::benchmark_data_smithi179_77138_object94276:head# 2024-02-09T17:27:03.999 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.999 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7911654:::benchmark_data_smithi179_77138_object43668:head# 2024-02-09T17:27:03.999 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.999 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f792e5ac:::benchmark_data_smithi179_77138_object103238:head# 2024-02-09T17:27:03.999 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.999 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f792e9ea:::benchmark_data_smithi179_77138_object96823:head# 2024-02-09T17:27:03.999 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:03.999 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7942114:::benchmark_data_smithi179_77138_object117055:head# 2024-02-09T17:27:03.999 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.006 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f794b5e9:::benchmark_data_smithi179_77138_object89862:head# 2024-02-09T17:27:04.006 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.007 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7958d92:::benchmark_data_smithi179_77138_object114130:head# 2024-02-09T17:27:04.007 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.007 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f798f2b0:::benchmark_data_smithi179_77138_object45500:head# 2024-02-09T17:27:04.007 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.007 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7999fce:::benchmark_data_smithi179_77138_object132762:head# 2024-02-09T17:27:04.007 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.007 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f799cb2d:::benchmark_data_smithi179_77138_object41067:head# 2024-02-09T17:27:04.007 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.007 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f79af97e:::benchmark_data_smithi179_77138_object98184:head# 2024-02-09T17:27:04.007 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.007 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f79ba04c:::benchmark_data_smithi179_77138_object21881:head# 2024-02-09T17:27:04.007 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.007 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f79bf031:::benchmark_data_smithi179_77138_object102641:head# 2024-02-09T17:27:04.007 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.008 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f79cb99d:::benchmark_data_smithi179_77138_object35457:head# 2024-02-09T17:27:04.008 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.008 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f79cc300:::benchmark_data_smithi179_77138_object122811:head# 2024-02-09T17:27:04.008 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.008 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f79d3c97:::benchmark_data_smithi179_77138_object54423:head# 2024-02-09T17:27:04.008 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.008 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f79d586b:::benchmark_data_smithi179_77138_object100472:head# 2024-02-09T17:27:04.008 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.008 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f79db0fd:::benchmark_data_smithi179_77138_object36430:head# 2024-02-09T17:27:04.008 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.008 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f79dec25:::benchmark_data_smithi179_77138_object89332:head# 2024-02-09T17:27:04.008 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.008 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f79dff49:::benchmark_data_smithi179_77138_object103902:head# 2024-02-09T17:27:04.008 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.009 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f79e22c6:::benchmark_data_smithi179_77138_object65107:head# 2024-02-09T17:27:04.009 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.009 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f79e3e64:::benchmark_data_smithi179_77138_object112314:head# 2024-02-09T17:27:04.009 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.009 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f79ea06a:::benchmark_data_smithi179_77138_object126509:head# 2024-02-09T17:27:04.009 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.009 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f79f20f8:::benchmark_data_smithi179_77138_object63437:head# 2024-02-09T17:27:04.009 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.009 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f79f89a5:::benchmark_data_smithi179_77138_object47706:head# 2024-02-09T17:27:04.009 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.009 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f79f94e9:::benchmark_data_smithi179_77138_object49654:head# 2024-02-09T17:27:04.009 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.009 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7a0cbaa:::benchmark_data_smithi179_77138_object88762:head# 2024-02-09T17:27:04.009 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.009 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7a0e9df:::benchmark_data_smithi179_77138_object60117:head# 2024-02-09T17:27:04.009 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.010 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7a2b12b:::benchmark_data_smithi179_77138_object83925:head# 2024-02-09T17:27:04.010 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.010 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7a2c09a:::benchmark_data_smithi179_77138_object74260:head# 2024-02-09T17:27:04.010 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.010 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7a361d8:::benchmark_data_smithi179_77138_object26294:head# 2024-02-09T17:27:04.010 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.010 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7a3a430:::benchmark_data_smithi179_77138_object24612:head# 2024-02-09T17:27:04.010 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.010 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7a466ab:::benchmark_data_smithi179_77138_object61602:head# 2024-02-09T17:27:04.010 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.010 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7a4a0b1:::benchmark_data_smithi179_77138_object117146:head# 2024-02-09T17:27:04.010 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.010 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7a54c94:::benchmark_data_smithi179_77138_object120790:head# 2024-02-09T17:27:04.010 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.011 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7a5f623:::benchmark_data_smithi179_77138_object127744:head# 2024-02-09T17:27:04.011 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.011 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7a66ba5:::benchmark_data_smithi179_77138_object89025:head# 2024-02-09T17:27:04.011 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.011 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7a67fd8:::benchmark_data_smithi179_77138_object111337:head# 2024-02-09T17:27:04.011 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.011 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7a7149b:::benchmark_data_smithi179_77138_object41916:head# 2024-02-09T17:27:04.011 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.011 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7a736cd:::benchmark_data_smithi179_77138_object112783:head# 2024-02-09T17:27:04.011 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.011 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7a957dc:::benchmark_data_smithi179_77138_object43157:head# 2024-02-09T17:27:04.011 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.011 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7ab0dc9:::benchmark_data_smithi179_77138_object64656:head# 2024-02-09T17:27:04.011 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.011 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7ab6e9c:::benchmark_data_smithi179_77138_object36363:head# 2024-02-09T17:27:04.012 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.012 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7ab8af5:::benchmark_data_smithi179_77138_object76305:head# 2024-02-09T17:27:04.012 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.012 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7abca98:::benchmark_data_smithi179_77138_object29689:head# 2024-02-09T17:27:04.012 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.012 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7acb71f:::benchmark_data_smithi179_77138_object78367:head# 2024-02-09T17:27:04.012 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.012 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7ace351:::benchmark_data_smithi179_77138_object76387:head# 2024-02-09T17:27:04.012 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.012 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7ade12d:::benchmark_data_smithi179_77138_object114493:head# 2024-02-09T17:27:04.012 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.012 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7ae4225:::benchmark_data_smithi179_77138_object53284:head# 2024-02-09T17:27:04.012 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.013 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7ae9469:::benchmark_data_smithi179_77138_object51661:head# 2024-02-09T17:27:04.013 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.020 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7af0ea8:::benchmark_data_smithi179_77138_object76012:head# 2024-02-09T17:27:04.020 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.020 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7af12ca:::benchmark_data_smithi179_77138_object104138:head# 2024-02-09T17:27:04.020 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.020 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7af48c0:::benchmark_data_smithi179_77138_object57996:head# 2024-02-09T17:27:04.020 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.020 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7af8d6d:::benchmark_data_smithi179_77138_object38664:head# 2024-02-09T17:27:04.021 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.021 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7af91df:::benchmark_data_smithi179_77138_object50562:head# 2024-02-09T17:27:04.021 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.021 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7b085e0:::benchmark_data_smithi179_77138_object132438:head# 2024-02-09T17:27:04.021 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.021 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7b201da:::benchmark_data_smithi179_77138_object132691:head# 2024-02-09T17:27:04.021 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.021 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7b26d82:::benchmark_data_smithi179_77138_object38292:head# 2024-02-09T17:27:04.021 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.021 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7b2bd9d:::benchmark_data_smithi179_77138_object119491:head# 2024-02-09T17:27:04.021 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.021 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7b3fbad:::benchmark_data_smithi179_77138_object116899:head# 2024-02-09T17:27:04.021 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.021 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7b4c097:::benchmark_data_smithi179_77138_object48665:head# 2024-02-09T17:27:04.022 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.022 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7b579b8:::benchmark_data_smithi179_77138_object43986:head# 2024-02-09T17:27:04.022 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.022 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7b5855d:::benchmark_data_smithi179_77138_object31342:head# 2024-02-09T17:27:04.022 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.022 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7b6e94c:::benchmark_data_smithi179_77138_object110144:head# 2024-02-09T17:27:04.022 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.022 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7b7cdbb:::benchmark_data_smithi179_77138_object84567:head# 2024-02-09T17:27:04.022 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.022 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7b84d06:::benchmark_data_smithi179_77138_object92260:head# 2024-02-09T17:27:04.022 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.022 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7b85177:::benchmark_data_smithi179_77138_object122685:head# 2024-02-09T17:27:04.022 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.023 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7b8b0cd:::benchmark_data_smithi179_77138_object77540:head# 2024-02-09T17:27:04.023 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.023 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7b90064:::benchmark_data_smithi179_77138_object62468:head# 2024-02-09T17:27:04.023 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.023 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7b92e3b:::benchmark_data_smithi179_77138_object32667:head# 2024-02-09T17:27:04.023 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.023 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7b99b98:::benchmark_data_smithi179_77138_object23189:head# 2024-02-09T17:27:04.023 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.023 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7b9acda:::benchmark_data_smithi179_77138_object27470:head# 2024-02-09T17:27:04.023 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.023 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7baac7d:::benchmark_data_smithi179_77138_object74299:head# 2024-02-09T17:27:04.023 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.023 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7bac92a:::benchmark_data_smithi179_77138_object116465:head# 2024-02-09T17:27:04.024 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.024 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7bb53bd:::benchmark_data_smithi179_77138_object35824:head# 2024-02-09T17:27:04.024 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.024 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7bc0cc8:::benchmark_data_smithi179_77138_object49078:head# 2024-02-09T17:27:04.024 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.024 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7bc6979:::benchmark_data_smithi179_77138_object23747:head# 2024-02-09T17:27:04.024 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.024 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7bdb403:::benchmark_data_smithi179_77138_object120174:head# 2024-02-09T17:27:04.024 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.024 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7bdead9:::benchmark_data_smithi179_77138_object56912:head# 2024-02-09T17:27:04.024 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.024 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7bdec3b:::benchmark_data_smithi179_77138_object54587:head# 2024-02-09T17:27:04.024 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.025 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7beaf65:::benchmark_data_smithi179_77138_object58875:head# 2024-02-09T17:27:04.025 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.025 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7c02f05:::benchmark_data_smithi179_77138_object35826:head# 2024-02-09T17:27:04.025 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.025 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7c0ff0c:::benchmark_data_smithi179_77138_object105083:head# 2024-02-09T17:27:04.025 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.025 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7c15aec:::benchmark_data_smithi179_77138_object71636:head# 2024-02-09T17:27:04.025 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.025 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7c1cfe7:::benchmark_data_smithi179_77138_object75923:head# 2024-02-09T17:27:04.025 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.025 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7c1ebe9:::benchmark_data_smithi179_77138_object115027:head# 2024-02-09T17:27:04.025 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.025 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7c470a4:::benchmark_data_smithi179_77138_object80253:head# 2024-02-09T17:27:04.025 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.025 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7c4e0c5:::benchmark_data_smithi179_77138_object50898:head# 2024-02-09T17:27:04.026 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.026 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7c4e264:::benchmark_data_smithi179_77138_object23995:head# 2024-02-09T17:27:04.026 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.026 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7c5de9e:::benchmark_data_smithi179_77138_object104011:head# 2024-02-09T17:27:04.026 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.026 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7c6108a:::benchmark_data_smithi179_77138_object105565:head# 2024-02-09T17:27:04.026 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.026 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7c61967:::benchmark_data_smithi179_77138_object110701:head# 2024-02-09T17:27:04.026 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.026 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7c65158:::benchmark_data_smithi179_77138_object20212:head# 2024-02-09T17:27:04.026 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.026 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7c7bb81:::benchmark_data_smithi179_77138_object130239:head# 2024-02-09T17:27:04.026 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.026 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7c81bf8:::benchmark_data_smithi179_77138_object109513:head# 2024-02-09T17:27:04.026 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.027 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7c8dffd:::benchmark_data_smithi179_77138_object101920:head# 2024-02-09T17:27:04.027 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.027 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7c96a2a:::benchmark_data_smithi179_77138_object67106:head# 2024-02-09T17:27:04.027 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.027 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7c99280:::benchmark_data_smithi179_77138_object40991:head# 2024-02-09T17:27:04.027 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.027 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7ca33ab:::benchmark_data_smithi179_77138_object129915:head# 2024-02-09T17:27:04.027 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.027 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7cac746:::benchmark_data_smithi179_77138_object41533:head# 2024-02-09T17:27:04.027 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.027 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7cb8b5b:::benchmark_data_smithi179_77138_object100929:head# 2024-02-09T17:27:04.027 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.027 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7ccedde:::benchmark_data_smithi179_77138_object103122:head# 2024-02-09T17:27:04.027 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.027 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7ccee8c:::benchmark_data_smithi179_77138_object92918:head# 2024-02-09T17:27:04.028 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.028 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7cd279a:::benchmark_data_smithi179_77138_object37245:head# 2024-02-09T17:27:04.028 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.028 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7cd9645:::benchmark_data_smithi179_77138_object25824:head# 2024-02-09T17:27:04.028 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.039 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7cde14f:::benchmark_data_smithi179_77138_object35038:head# 2024-02-09T17:27:04.039 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.039 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7ce7983:::benchmark_data_smithi179_77138_object24976:head# 2024-02-09T17:27:04.039 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.039 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7d1f8b9:::benchmark_data_smithi179_77138_object52592:head# 2024-02-09T17:27:04.039 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.040 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7d3289b:::benchmark_data_smithi179_77138_object94166:head# 2024-02-09T17:27:04.040 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.040 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7d482a7:::benchmark_data_smithi179_77138_object22807:head# 2024-02-09T17:27:04.040 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.040 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7d4dec6:::benchmark_data_smithi179_77138_object78895:head# 2024-02-09T17:27:04.040 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.040 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7d541f3:::benchmark_data_smithi179_77138_object129735:head# 2024-02-09T17:27:04.040 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.040 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7d5988a:::benchmark_data_smithi179_77138_object72051:head# 2024-02-09T17:27:04.040 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.040 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7d5af5e:::benchmark_data_smithi179_77138_object79412:head# 2024-02-09T17:27:04.040 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.040 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7d5b998:::benchmark_data_smithi179_77138_object108699:head# 2024-02-09T17:27:04.040 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.041 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7d6a525:::benchmark_data_smithi179_77138_object112717:head# 2024-02-09T17:27:04.041 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.041 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7d6e6e6:::benchmark_data_smithi179_77138_object107249:head# 2024-02-09T17:27:04.041 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.041 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7d80cc5:::benchmark_data_smithi179_77138_object110916:head# 2024-02-09T17:27:04.041 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.041 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7d9492f:::benchmark_data_smithi179_77138_object85889:head# 2024-02-09T17:27:04.041 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.041 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7da22d2:::benchmark_data_smithi179_77138_object119606:head# 2024-02-09T17:27:04.041 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.041 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7da29eb:::benchmark_data_smithi179_77138_object47039:head# 2024-02-09T17:27:04.041 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.041 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7da4744:::benchmark_data_smithi179_77138_object39549:head# 2024-02-09T17:27:04.041 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.042 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7dad0cc:::benchmark_data_smithi179_77138_object67313:head# 2024-02-09T17:27:04.042 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.042 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7db3a7a:::benchmark_data_smithi179_77138_object46774:head# 2024-02-09T17:27:04.042 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.042 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7db4504:::benchmark_data_smithi179_77138_object61934:head# 2024-02-09T17:27:04.042 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.042 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7db933c:::benchmark_data_smithi179_77138_object109949:head# 2024-02-09T17:27:04.042 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.042 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7db9d4f:::benchmark_data_smithi179_77138_object125761:head# 2024-02-09T17:27:04.042 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.042 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7dba184:::benchmark_data_smithi179_77138_object73095:head# 2024-02-09T17:27:04.042 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.042 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7dba34e:::benchmark_data_smithi179_77138_object117551:head# 2024-02-09T17:27:04.043 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.043 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7dcc669:::benchmark_data_smithi179_77138_object78202:head# 2024-02-09T17:27:04.043 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.043 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7dd8a72:::benchmark_data_smithi179_77138_object65574:head# 2024-02-09T17:27:04.043 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.043 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7deb09c:::benchmark_data_smithi179_77138_object110988:head# 2024-02-09T17:27:04.043 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.043 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7decc30:::benchmark_data_smithi179_77138_object64464:head# 2024-02-09T17:27:04.043 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.043 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7df6263:::benchmark_data_smithi179_77138_object46363:head# 2024-02-09T17:27:04.043 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.043 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7df7a49:::benchmark_data_smithi179_77138_object111180:head# 2024-02-09T17:27:04.043 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.043 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7e162d8:::benchmark_data_smithi179_77138_object106957:head# 2024-02-09T17:27:04.044 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.044 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7e1870e:::benchmark_data_smithi179_77138_object24673:head# 2024-02-09T17:27:04.044 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.044 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7e26037:::benchmark_data_smithi179_77138_object106388:head# 2024-02-09T17:27:04.044 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.044 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7e33452:::benchmark_data_smithi179_77138_object101118:head# 2024-02-09T17:27:04.044 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.044 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7e35cbe:::benchmark_data_smithi179_77138_object96013:head# 2024-02-09T17:27:04.044 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.044 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7e87529:::benchmark_data_smithi179_77138_object48232:head# 2024-02-09T17:27:04.044 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.044 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7e9b763:::benchmark_data_smithi179_77138_object43102:head# 2024-02-09T17:27:04.044 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.044 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7e9c745:::benchmark_data_smithi179_77138_object24557:head# 2024-02-09T17:27:04.045 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.045 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7e9ec8a:::benchmark_data_smithi179_77138_object128448:head# 2024-02-09T17:27:04.045 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.045 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7e9fa56:::benchmark_data_smithi179_77138_object96083:head# 2024-02-09T17:27:04.045 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.045 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7ea24ce:::benchmark_data_smithi179_77138_object93102:head# 2024-02-09T17:27:04.045 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.045 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7eb99f9:::benchmark_data_smithi179_77138_object32941:head# 2024-02-09T17:27:04.045 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.045 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7ed9f08:::benchmark_data_smithi179_77138_object38148:head# 2024-02-09T17:27:04.045 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.045 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7ee27cd:::benchmark_data_smithi179_77138_object77951:head# 2024-02-09T17:27:04.045 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.045 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7ee7400:::benchmark_data_smithi179_77138_object98516:head# 2024-02-09T17:27:04.046 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.046 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7f03fae:::benchmark_data_smithi179_77138_object120050:head# 2024-02-09T17:27:04.046 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.046 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7f09fd5:::benchmark_data_smithi179_77138_object132700:head# 2024-02-09T17:27:04.046 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.046 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7f1272d:::benchmark_data_smithi179_77138_object29954:head# 2024-02-09T17:27:04.046 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.046 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7f12e3b:::benchmark_data_smithi179_77138_object44012:head# 2024-02-09T17:27:04.046 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.046 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7f15d62:::benchmark_data_smithi179_77138_object83029:head# 2024-02-09T17:27:04.046 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.046 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7f19ddc:::benchmark_data_smithi179_77138_object37581:head# 2024-02-09T17:27:04.046 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.046 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7f26f1e:::benchmark_data_smithi179_77138_object43566:head# 2024-02-09T17:27:04.047 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.047 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7f289b1:::benchmark_data_smithi179_77138_object62859:head# 2024-02-09T17:27:04.047 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.047 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7f2fb93:::benchmark_data_smithi179_77138_object94745:head# 2024-02-09T17:27:04.047 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.047 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7f3b709:::benchmark_data_smithi179_77138_object42722:head# 2024-02-09T17:27:04.047 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.047 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7f3fedd:::benchmark_data_smithi179_77138_object121933:head# 2024-02-09T17:27:04.047 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.047 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7f529d8:::benchmark_data_smithi179_77138_object84338:head# 2024-02-09T17:27:04.047 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.047 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7f53255:::benchmark_data_smithi179_77138_object29617:head# 2024-02-09T17:27:04.047 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.047 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7f5dac2:::benchmark_data_smithi179_77138_object84106:head# 2024-02-09T17:27:04.047 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.048 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7f76feb:::benchmark_data_smithi179_77138_object72983:head# 2024-02-09T17:27:04.048 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.048 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7f7aafe:::benchmark_data_smithi179_77138_object115463:head# 2024-02-09T17:27:04.048 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.048 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7f7d9fa:::benchmark_data_smithi179_77138_object66765:head# 2024-02-09T17:27:04.048 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.048 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7f7fb5b:::benchmark_data_smithi179_77138_object37912:head# 2024-02-09T17:27:04.048 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.048 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7f91965:::benchmark_data_smithi179_77138_object84130:head# 2024-02-09T17:27:04.048 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.048 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7faa3a1:::benchmark_data_smithi179_77138_object117663:head# 2024-02-09T17:27:04.048 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.048 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7faeb9b:::benchmark_data_smithi179_77138_object104880:head# 2024-02-09T17:27:04.048 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.049 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7fb16cb:::benchmark_data_smithi179_77138_object121228:head# 2024-02-09T17:27:04.049 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.049 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7fb5a8b:::benchmark_data_smithi179_77138_object53196:head# 2024-02-09T17:27:04.049 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.049 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7fb6ab7:::benchmark_data_smithi179_77138_object101641:head# 2024-02-09T17:27:04.049 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.049 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7fd5cca:::benchmark_data_smithi179_77138_object77894:head# 2024-02-09T17:27:04.049 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.049 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7fdbbd9:::benchmark_data_smithi179_77138_object114099:head# 2024-02-09T17:27:04.049 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.049 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7fe9bbd:::benchmark_data_smithi179_77138_object109215:head# 2024-02-09T17:27:04.049 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.049 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7fed295:::benchmark_data_smithi179_77138_object85035:head# 2024-02-09T17:27:04.049 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.049 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7fed35f:::benchmark_data_smithi179_77138_object21248:head# 2024-02-09T17:27:04.050 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.050 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f7ff1511:::benchmark_data_smithi179_77138_object31895:head# 2024-02-09T17:27:04.050 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.050 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8018948:::benchmark_data_smithi179_77138_object83910:head# 2024-02-09T17:27:04.050 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.050 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f801aa65:::benchmark_data_smithi179_77138_object82190:head# 2024-02-09T17:27:04.050 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.050 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f801d357:::benchmark_data_smithi179_77138_object48083:head# 2024-02-09T17:27:04.050 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.050 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f801fb74:::benchmark_data_smithi179_77138_object106792:head# 2024-02-09T17:27:04.050 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.057 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:27:04.060 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8051aff:::benchmark_data_smithi179_77138_object31829:head# 2024-02-09T17:27:04.060 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.060 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f80556b1:::benchmark_data_smithi179_77138_object43136:head# 2024-02-09T17:27:04.061 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.061 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f805d6d6:::benchmark_data_smithi179_77138_object21579:head# 2024-02-09T17:27:04.061 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.061 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8061503:::benchmark_data_smithi179_77138_object103283:head# 2024-02-09T17:27:04.061 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.061 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8066fcf:::benchmark_data_smithi179_77138_object122178:head# 2024-02-09T17:27:04.061 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.061 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f806a61d:::benchmark_data_smithi179_77138_object81536:head# 2024-02-09T17:27:04.061 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.061 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f806d8c9:::benchmark_data_smithi179_77138_object80034:head# 2024-02-09T17:27:04.061 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.061 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f809d402:::benchmark_data_smithi179_77138_object98567:head# 2024-02-09T17:27:04.061 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.061 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f80b1f6f:::benchmark_data_smithi179_77138_object118664:head# 2024-02-09T17:27:04.062 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.062 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f80b7205:::benchmark_data_smithi179_77138_object19375:head# 2024-02-09T17:27:04.062 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.062 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f80b9a4d:::benchmark_data_smithi179_77138_object42420:head# 2024-02-09T17:27:04.062 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.062 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f80be2b2:::benchmark_data_smithi179_77138_object59095:head# 2024-02-09T17:27:04.062 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.062 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f80d3bad:::benchmark_data_smithi179_77138_object77259:head# 2024-02-09T17:27:04.062 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.062 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f80e8979:::benchmark_data_smithi179_77138_object92437:head# 2024-02-09T17:27:04.062 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.062 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f80ed599:::benchmark_data_smithi179_77138_object38296:head# 2024-02-09T17:27:04.062 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.063 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f80f002c:::benchmark_data_smithi179_77138_object76722:head# 2024-02-09T17:27:04.063 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.063 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f80f43ed:::benchmark_data_smithi179_77138_object96084:head# 2024-02-09T17:27:04.063 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.063 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f810617b:::benchmark_data_smithi179_77138_object20618:head# 2024-02-09T17:27:04.063 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.063 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f813459f:::benchmark_data_smithi179_77138_object55052:head# 2024-02-09T17:27:04.063 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.063 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f81499af:::benchmark_data_smithi179_77138_object36422:head# 2024-02-09T17:27:04.063 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.063 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f814ffab:::benchmark_data_smithi179_77138_object51757:head# 2024-02-09T17:27:04.063 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.063 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8157963:::benchmark_data_smithi179_77138_object71150:head# 2024-02-09T17:27:04.063 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.063 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8159f7b:::benchmark_data_smithi179_77138_object46157:head# 2024-02-09T17:27:04.064 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.064 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f815efa7:::benchmark_data_smithi179_77138_object41206:head# 2024-02-09T17:27:04.064 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.064 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8160bc4:::benchmark_data_smithi179_77138_object39813:head# 2024-02-09T17:27:04.064 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.064 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8162db2:::benchmark_data_smithi179_77138_object114348:head# 2024-02-09T17:27:04.064 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.064 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8163623:::benchmark_data_smithi179_77138_object55332:head# 2024-02-09T17:27:04.064 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.064 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f816c893:::benchmark_data_smithi179_77138_object88560:head# 2024-02-09T17:27:04.064 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.064 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8174329:::benchmark_data_smithi179_77138_object120505:head# 2024-02-09T17:27:04.064 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.064 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8178970:::benchmark_data_smithi179_77138_object57603:head# 2024-02-09T17:27:04.065 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.065 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f818bad6:::benchmark_data_smithi179_77138_object28462:head# 2024-02-09T17:27:04.065 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.065 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f819b138:::benchmark_data_smithi179_77138_object94576:head# 2024-02-09T17:27:04.065 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.065 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f819f30f:::benchmark_data_smithi179_77138_object97121:head# 2024-02-09T17:27:04.065 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.065 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f81a0b49:::benchmark_data_smithi179_77138_object101666:head# 2024-02-09T17:27:04.065 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.065 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f81a6761:::benchmark_data_smithi179_77138_object45285:head# 2024-02-09T17:27:04.065 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.065 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f81a888e:::benchmark_data_smithi179_77138_object35589:head# 2024-02-09T17:27:04.065 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.065 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f81a893c:::benchmark_data_smithi179_77138_object20507:head# 2024-02-09T17:27:04.065 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.066 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f81b2ef0:::benchmark_data_smithi179_77138_object95261:head# 2024-02-09T17:27:04.066 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.066 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f81c0c29:::benchmark_data_smithi179_77138_object95429:head# 2024-02-09T17:27:04.066 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.066 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f81c106d:::benchmark_data_smithi179_77138_object44541:head# 2024-02-09T17:27:04.066 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.066 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f81cb7df:::benchmark_data_smithi179_77138_object42881:head# 2024-02-09T17:27:04.066 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.066 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f81d0c5f:::benchmark_data_smithi179_77138_object75649:head# 2024-02-09T17:27:04.066 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.066 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f81de696:::benchmark_data_smithi179_77138_object82737:head# 2024-02-09T17:27:04.066 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.066 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f81deab3:::benchmark_data_smithi179_77138_object87145:head# 2024-02-09T17:27:04.066 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.066 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f81e59bc:::benchmark_data_smithi179_77138_object46733:head# 2024-02-09T17:27:04.067 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.067 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f81e74bd:::benchmark_data_smithi179_77138_object127976:head# 2024-02-09T17:27:04.067 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.067 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f81ff3c2:::benchmark_data_smithi179_77138_object67584:head# 2024-02-09T17:27:04.067 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.067 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8201302:::benchmark_data_smithi179_77138_object28164:head# 2024-02-09T17:27:04.067 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.067 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f82197c5:::benchmark_data_smithi179_77138_object123048:head# 2024-02-09T17:27:04.067 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.067 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f821e891:::benchmark_data_smithi179_77138_object117463:head# 2024-02-09T17:27:04.067 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.067 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f821eab2:::benchmark_data_smithi179_77138_object39761:head# 2024-02-09T17:27:04.067 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.068 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f822a915:::benchmark_data_smithi179_77138_object54416:head# 2024-02-09T17:27:04.068 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.068 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8238098:::benchmark_data_smithi179_77138_object34350:head# 2024-02-09T17:27:04.068 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.068 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f823bdaa:::benchmark_data_smithi179_77138_object90201:head# 2024-02-09T17:27:04.068 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.068 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f825198b:::benchmark_data_smithi179_77138_object40878:head# 2024-02-09T17:27:04.068 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.068 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f826194c:::benchmark_data_smithi179_77138_object108792:head# 2024-02-09T17:27:04.068 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.068 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8261b34:::benchmark_data_smithi179_77138_object37826:head# 2024-02-09T17:27:04.068 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.068 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f826903c:::benchmark_data_smithi179_77138_object39316:head# 2024-02-09T17:27:04.068 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.069 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8289f91:::benchmark_data_smithi179_77138_object80291:head# 2024-02-09T17:27:04.069 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.069 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f828d227:::benchmark_data_smithi179_77138_object50964:head# 2024-02-09T17:27:04.069 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.069 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f82957f4:::benchmark_data_smithi179_77138_object99070:head# 2024-02-09T17:27:04.069 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.069 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f829f1c8:::benchmark_data_smithi179_77138_object98102:head# 2024-02-09T17:27:04.069 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.069 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f82a5366:::benchmark_data_smithi179_77138_object58896:head# 2024-02-09T17:27:04.069 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.069 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f82c3b9c:::benchmark_data_smithi179_77138_object24915:head# 2024-02-09T17:27:04.069 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.069 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f82cfe20:::benchmark_data_smithi179_77138_object24227:head# 2024-02-09T17:27:04.069 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.069 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f82d5f91:::benchmark_data_smithi179_77138_object31121:head# 2024-02-09T17:27:04.070 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.070 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f82dd04b:::benchmark_data_smithi179_77138_object79821:head# 2024-02-09T17:27:04.070 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.070 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f82eba07:::benchmark_data_smithi179_77138_object130329:head# 2024-02-09T17:27:04.070 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.070 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8301ca6:::benchmark_data_smithi179_77138_object26291:head# 2024-02-09T17:27:04.070 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.070 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f830ab58:::benchmark_data_smithi179_77138_object106985:head# 2024-02-09T17:27:04.070 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.070 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f830ca2b:::benchmark_data_smithi179_77138_object32509:head# 2024-02-09T17:27:04.070 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.070 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f830e8eb:::benchmark_data_smithi179_77138_object88292:head# 2024-02-09T17:27:04.070 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.070 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8318f41:::benchmark_data_smithi179_77138_object124596:head# 2024-02-09T17:27:04.070 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.071 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f831c25b:::benchmark_data_smithi179_77138_object101952:head# 2024-02-09T17:27:04.071 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.071 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f832b458:::benchmark_data_smithi179_77138_object85507:head# 2024-02-09T17:27:04.071 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.071 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f832bb17:::benchmark_data_smithi179_77138_object82607:head# 2024-02-09T17:27:04.071 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.071 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f83344ba:::benchmark_data_smithi179_77138_object19598:head# 2024-02-09T17:27:04.071 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.071 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f83439cf:::benchmark_data_smithi179_77138_object60009:head# 2024-02-09T17:27:04.071 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.071 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8347c0c:::benchmark_data_smithi179_77138_object91641:head# 2024-02-09T17:27:04.071 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.071 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8355966:::benchmark_data_smithi179_77138_object29623:head# 2024-02-09T17:27:04.071 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.071 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8358ff1:::benchmark_data_smithi179_77138_object120827:head# 2024-02-09T17:27:04.072 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.072 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f83598ae:::benchmark_data_smithi179_77138_object53395:head# 2024-02-09T17:27:04.072 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.072 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f835b7d9:::benchmark_data_smithi179_77138_object78746:head# 2024-02-09T17:27:04.072 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.072 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f83634ef:::benchmark_data_smithi179_77138_object96425:head# 2024-02-09T17:27:04.072 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.072 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8373f0a:::benchmark_data_smithi179_77138_object49053:head# 2024-02-09T17:27:04.072 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.092 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f83755c4:::benchmark_data_smithi179_77138_object116869:head# 2024-02-09T17:27:04.092 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.092 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f837ba09:::benchmark_data_smithi179_77138_object68356:head# 2024-02-09T17:27:04.092 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.092 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f837e48d:::benchmark_data_smithi179_77138_object78105:head# 2024-02-09T17:27:04.092 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.092 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f838055d:::benchmark_data_smithi179_77138_object110587:head# 2024-02-09T17:27:04.092 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.092 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f83878d3:::benchmark_data_smithi179_77138_object22475:head# 2024-02-09T17:27:04.092 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.092 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f838d71a:::benchmark_data_smithi179_77138_object53559:head# 2024-02-09T17:27:04.093 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.093 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f838ebfe:::benchmark_data_smithi179_77138_object86612:head# 2024-02-09T17:27:04.093 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.093 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f839c5b5:::benchmark_data_smithi179_77138_object67571:head# 2024-02-09T17:27:04.093 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.093 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f839d430:::benchmark_data_smithi179_77138_object56492:head# 2024-02-09T17:27:04.093 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.093 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f83b79d4:::benchmark_data_smithi179_77138_object125103:head# 2024-02-09T17:27:04.093 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.093 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f83c4bbb:::benchmark_data_smithi179_77138_object112141:head# 2024-02-09T17:27:04.093 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.093 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f83c4cc8:::benchmark_data_smithi179_77138_object80947:head# 2024-02-09T17:27:04.093 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.093 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f83c930b:::benchmark_data_smithi179_77138_object53135:head# 2024-02-09T17:27:04.094 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.094 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f83df600:::benchmark_data_smithi179_77138_object77158:head# 2024-02-09T17:27:04.094 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.094 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f83e0ba5:::benchmark_data_smithi179_77138_object104348:head# 2024-02-09T17:27:04.094 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.094 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f83f06de:::benchmark_data_smithi179_77138_object85138:head# 2024-02-09T17:27:04.094 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.094 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f83f5983:::benchmark_data_smithi179_77138_object113789:head# 2024-02-09T17:27:04.094 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.094 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f83f64b1:::benchmark_data_smithi179_77138_object130207:head# 2024-02-09T17:27:04.094 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.094 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f83ffb59:::benchmark_data_smithi179_77138_object82920:head# 2024-02-09T17:27:04.094 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.095 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f841358b:::benchmark_data_smithi179_77138_object85202:head# 2024-02-09T17:27:04.095 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.095 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f84179ab:::benchmark_data_smithi179_77138_object81997:head# 2024-02-09T17:27:04.095 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.095 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8427b9c:::benchmark_data_smithi179_77138_object62770:head# 2024-02-09T17:27:04.095 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.095 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8428c22:::benchmark_data_smithi179_77138_object57473:head# 2024-02-09T17:27:04.095 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.095 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f842ac57:::benchmark_data_smithi179_77138_object61776:head# 2024-02-09T17:27:04.095 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.095 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f84305b7:::benchmark_data_smithi179_77138_object61964:head# 2024-02-09T17:27:04.095 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.095 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f843b837:::benchmark_data_smithi179_77138_object67436:head# 2024-02-09T17:27:04.096 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.096 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f843c326:::benchmark_data_smithi179_77138_object74510:head# 2024-02-09T17:27:04.096 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.096 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8441794:::benchmark_data_smithi179_77138_object119910:head# 2024-02-09T17:27:04.096 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.096 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f844a5e6:::benchmark_data_smithi179_77138_object99761:head# 2024-02-09T17:27:04.096 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.096 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f844ac3d:::benchmark_data_smithi179_77138_object77201:head# 2024-02-09T17:27:04.096 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.096 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f845bf17:::benchmark_data_smithi179_77138_object88566:head# 2024-02-09T17:27:04.096 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.096 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f845ed62:::benchmark_data_smithi179_77138_object130902:head# 2024-02-09T17:27:04.096 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.097 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8469890:::benchmark_data_smithi179_77138_object98078:head# 2024-02-09T17:27:04.097 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.097 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8470f94:::benchmark_data_smithi179_77138_object29480:head# 2024-02-09T17:27:04.097 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.097 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f847371e:::benchmark_data_smithi179_77138_object69983:head# 2024-02-09T17:27:04.097 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.097 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f84799cc:::benchmark_data_smithi179_77138_object34898:head# 2024-02-09T17:27:04.097 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.097 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8482a0c:::benchmark_data_smithi179_77138_object82720:head# 2024-02-09T17:27:04.097 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.097 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f849c669:::benchmark_data_smithi179_77138_object59613:head# 2024-02-09T17:27:04.097 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.097 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f849e807:::benchmark_data_smithi179_77138_object81922:head# 2024-02-09T17:27:04.097 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.097 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f84a64c5:::benchmark_data_smithi179_77138_object25681:head# 2024-02-09T17:27:04.098 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.098 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f84a7a79:::benchmark_data_smithi179_77138_object94624:head# 2024-02-09T17:27:04.098 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.098 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f84b3139:::benchmark_data_smithi179_77138_object75082:head# 2024-02-09T17:27:04.098 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.098 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f84bb30d:::benchmark_data_smithi179_77138_object100409:head# 2024-02-09T17:27:04.098 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.098 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f84cc341:::benchmark_data_smithi179_77138_object22960:head# 2024-02-09T17:27:04.098 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.098 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f84d7050:::benchmark_data_smithi179_77138_object53636:head# 2024-02-09T17:27:04.098 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.098 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f84ea705:::benchmark_data_smithi179_77138_object121856:head# 2024-02-09T17:27:04.098 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.098 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f84f0193:::benchmark_data_smithi179_77138_object24681:head# 2024-02-09T17:27:04.099 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.099 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f84f52b1:::benchmark_data_smithi179_77138_object28343:head# 2024-02-09T17:27:04.099 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.099 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f84f8a6d:::benchmark_data_smithi179_77138_object104512:head# 2024-02-09T17:27:04.099 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.099 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f84f926e:::benchmark_data_smithi179_77138_object65308:head# 2024-02-09T17:27:04.099 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.099 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f84fe504:::benchmark_data_smithi179_77138_object102308:head# 2024-02-09T17:27:04.099 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.099 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8504e99:::benchmark_data_smithi179_77138_object37694:head# 2024-02-09T17:27:04.099 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.099 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f850b071:::benchmark_data_smithi179_77138_object124551:head# 2024-02-09T17:27:04.099 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.100 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f851cb64:::benchmark_data_smithi179_77138_object23137:head# 2024-02-09T17:27:04.100 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.100 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f851fb37:::benchmark_data_smithi179_77138_object25964:head# 2024-02-09T17:27:04.100 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.100 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f85203a3:::benchmark_data_smithi179_77138_object57219:head# 2024-02-09T17:27:04.100 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.100 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f852b656:::benchmark_data_smithi179_77138_object41273:head# 2024-02-09T17:27:04.100 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.100 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f852ff9e:::benchmark_data_smithi179_77138_object73183:head# 2024-02-09T17:27:04.100 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.100 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f853a8ad:::benchmark_data_smithi179_77138_object54599:head# 2024-02-09T17:27:04.100 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.100 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f853e3d1:::benchmark_data_smithi179_77138_object55211:head# 2024-02-09T17:27:04.100 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.101 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8541611:::benchmark_data_smithi179_77138_object67741:head# 2024-02-09T17:27:04.101 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.101 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f85485b4:::benchmark_data_smithi179_77138_object34701:head# 2024-02-09T17:27:04.101 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.101 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8549a38:::benchmark_data_smithi179_77138_object108614:head# 2024-02-09T17:27:04.101 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.101 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f855dbbe:::benchmark_data_smithi179_77138_object96154:head# 2024-02-09T17:27:04.101 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.101 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f856b42b:::benchmark_data_smithi179_77138_object106721:head# 2024-02-09T17:27:04.101 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.101 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f85780ac:::benchmark_data_smithi179_77138_object131461:head# 2024-02-09T17:27:04.101 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.101 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f858af99:::benchmark_data_smithi179_77138_object61880:head# 2024-02-09T17:27:04.101 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.102 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8599ed7:::benchmark_data_smithi179_77138_object81897:head# 2024-02-09T17:27:04.102 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.102 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f85a0660:::benchmark_data_smithi179_77138_object29024:head# 2024-02-09T17:27:04.102 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.102 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f85a0887:::benchmark_data_smithi179_77138_object70677:head# 2024-02-09T17:27:04.102 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.102 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f85a4c93:::benchmark_data_smithi179_77138_object129717:head# 2024-02-09T17:27:04.102 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.102 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f85a797f:::benchmark_data_smithi179_77138_object56410:head# 2024-02-09T17:27:04.102 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.102 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f85c217f:::benchmark_data_smithi179_77138_object75446:head# 2024-02-09T17:27:04.102 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.102 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f85cfb17:::benchmark_data_smithi179_77138_object43897:head# 2024-02-09T17:27:04.102 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.103 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f85d8494:::benchmark_data_smithi179_77138_object77136:head# 2024-02-09T17:27:04.103 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.103 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f85da420:::benchmark_data_smithi179_77138_object42658:head# 2024-02-09T17:27:04.103 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.103 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f85db365:::benchmark_data_smithi179_77138_object73976:head# 2024-02-09T17:27:04.103 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.103 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f85e15cd:::benchmark_data_smithi179_77138_object72981:head# 2024-02-09T17:27:04.103 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.103 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f85e19ac:::benchmark_data_smithi179_77138_object74439:head# 2024-02-09T17:27:04.103 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.103 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f85e7f48:::benchmark_data_smithi179_77138_object77170:head# 2024-02-09T17:27:04.103 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.103 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f85eecf3:::benchmark_data_smithi179_77138_object118497:head# 2024-02-09T17:27:04.103 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.103 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f85f2a97:::benchmark_data_smithi179_77138_object100996:head# 2024-02-09T17:27:04.104 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.104 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f85f900a:::benchmark_data_smithi179_77138_object50349:head# 2024-02-09T17:27:04.104 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.104 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f861004c:::benchmark_data_smithi179_77138_object105010:head# 2024-02-09T17:27:04.104 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.104 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8617942:::benchmark_data_smithi179_77138_object21148:head# 2024-02-09T17:27:04.104 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.104 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f861c3c3:::benchmark_data_smithi179_77138_object100587:head# 2024-02-09T17:27:04.104 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.104 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f86256f7:::benchmark_data_smithi179_77138_object130887:head# 2024-02-09T17:27:04.104 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.104 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8626382:::benchmark_data_smithi179_77138_object75895:head# 2024-02-09T17:27:04.104 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.105 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8628f50:::benchmark_data_smithi179_77138_object125951:head# 2024-02-09T17:27:04.105 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.105 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f863df36:::benchmark_data_smithi179_77138_object31637:head# 2024-02-09T17:27:04.105 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.105 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f863f53f:::benchmark_data_smithi179_77138_object34196:head# 2024-02-09T17:27:04.105 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.105 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f864ec36:::benchmark_data_smithi179_77138_object68506:head# 2024-02-09T17:27:04.105 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.105 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f865bf8f:::benchmark_data_smithi179_77138_object83149:head# 2024-02-09T17:27:04.105 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.105 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f865d1ee:::benchmark_data_smithi179_77138_object51109:head# 2024-02-09T17:27:04.105 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.105 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f866034f:::benchmark_data_smithi179_77138_object56850:head# 2024-02-09T17:27:04.105 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.105 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8663173:::benchmark_data_smithi179_77138_object59940:head# 2024-02-09T17:27:04.105 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.106 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f866392f:::benchmark_data_smithi179_77138_object71107:head# 2024-02-09T17:27:04.106 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.106 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8674f48:::benchmark_data_smithi179_77138_object121894:head# 2024-02-09T17:27:04.106 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.106 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8679a54:::benchmark_data_smithi179_77138_object50482:head# 2024-02-09T17:27:04.106 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.106 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f86867a7:::benchmark_data_smithi179_77138_object77286:head# 2024-02-09T17:27:04.106 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.106 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f868d28d:::benchmark_data_smithi179_77138_object46083:head# 2024-02-09T17:27:04.106 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.106 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f868dddc:::benchmark_data_smithi179_77138_object103539:head# 2024-02-09T17:27:04.106 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.106 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f869129f:::benchmark_data_smithi179_77138_object20592:head# 2024-02-09T17:27:04.106 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.107 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8694796:::benchmark_data_smithi179_77138_object119351:head# 2024-02-09T17:27:04.107 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.107 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f86976e5:::benchmark_data_smithi179_77138_object85629:head# 2024-02-09T17:27:04.107 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.107 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f86b378c:::benchmark_data_smithi179_77138_object30280:head# 2024-02-09T17:27:04.107 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.107 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f86cd5e5:::benchmark_data_smithi179_77138_object113081:head# 2024-02-09T17:27:04.107 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.107 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f86d64cd:::benchmark_data_smithi179_77138_object58414:head# 2024-02-09T17:27:04.107 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.107 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f86d667a:::benchmark_data_smithi179_77138_object125380:head# 2024-02-09T17:27:04.107 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.107 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f86e2502:::benchmark_data_smithi179_77138_object130502:head# 2024-02-09T17:27:04.108 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.108 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f86f64fa:::benchmark_data_smithi179_77138_object98177:head# 2024-02-09T17:27:04.108 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.108 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f86ff873:::benchmark_data_smithi179_77138_object108646:head# 2024-02-09T17:27:04.108 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.108 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f870d237:::benchmark_data_smithi179_77138_object64724:head# 2024-02-09T17:27:04.108 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.108 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f87326b4:::benchmark_data_smithi179_77138_object23310:head# 2024-02-09T17:27:04.108 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.108 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f87327d2:::benchmark_data_smithi179_77138_object28413:head# 2024-02-09T17:27:04.108 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.108 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f873bb87:::benchmark_data_smithi179_77138_object96404:head# 2024-02-09T17:27:04.108 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.108 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8740a85:::benchmark_data_smithi179_77138_object95151:head# 2024-02-09T17:27:04.109 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.109 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8748bf7:::benchmark_data_smithi179_77138_object66034:head# 2024-02-09T17:27:04.109 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.109 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f87499b5:::benchmark_data_smithi179_77138_object20682:head# 2024-02-09T17:27:04.109 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.109 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8753050:::benchmark_data_smithi179_77138_object80856:head# 2024-02-09T17:27:04.109 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.109 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8757209:::benchmark_data_smithi179_77138_object118537:head# 2024-02-09T17:27:04.109 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.109 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f875b106:::benchmark_data_smithi179_77138_object20759:head# 2024-02-09T17:27:04.109 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.109 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8762f84:::benchmark_data_smithi179_77138_object91441:head# 2024-02-09T17:27:04.109 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.109 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f87669cf:::benchmark_data_smithi179_77138_object27928:head# 2024-02-09T17:27:04.109 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.110 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f87786a5:::benchmark_data_smithi179_77138_object51675:head# 2024-02-09T17:27:04.110 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.110 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f877db1f:::benchmark_data_smithi179_77138_object71843:head# 2024-02-09T17:27:04.110 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.110 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f87803ac:::benchmark_data_smithi179_77138_object115329:head# 2024-02-09T17:27:04.110 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.110 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f878471d:::benchmark_data_smithi179_77138_object44223:head# 2024-02-09T17:27:04.110 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.111 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:04.122 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f879322d:::benchmark_data_smithi179_77138_object120629:head# 2024-02-09T17:27:04.122 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.122 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f879efac:::benchmark_data_smithi179_77138_object65135:head# 2024-02-09T17:27:04.122 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.122 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f87a01e7:::benchmark_data_smithi179_77138_object51964:head# 2024-02-09T17:27:04.122 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.122 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f87a3956:::benchmark_data_smithi179_77138_object77405:head# 2024-02-09T17:27:04.122 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.122 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f87c1092:::benchmark_data_smithi179_77138_object70083:head# 2024-02-09T17:27:04.123 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.123 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f87c7783:::benchmark_data_smithi179_77138_object105457:head# 2024-02-09T17:27:04.123 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.123 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f87d5141:::benchmark_data_smithi179_77138_object74920:head# 2024-02-09T17:27:04.123 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.123 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f87e397d:::benchmark_data_smithi179_77138_object39504:head# 2024-02-09T17:27:04.123 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.123 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f87f149c:::benchmark_data_smithi179_77138_object80313:head# 2024-02-09T17:27:04.123 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.123 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f87f3ef7:::benchmark_data_smithi179_77138_object26953:head# 2024-02-09T17:27:04.123 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.123 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f87f5923:::benchmark_data_smithi179_77138_object77832:head# 2024-02-09T17:27:04.123 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.123 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8806087:::benchmark_data_smithi179_77138_object115423:head# 2024-02-09T17:27:04.123 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.124 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f881ba07:::benchmark_data_smithi179_77138_object85054:head# 2024-02-09T17:27:04.124 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.124 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8824175:::benchmark_data_smithi179_77138_object34769:head# 2024-02-09T17:27:04.124 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.124 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f883e133:::benchmark_data_smithi179_77138_object56777:head# 2024-02-09T17:27:04.124 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.124 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8844370:::benchmark_data_smithi179_77138_object119587:head# 2024-02-09T17:27:04.124 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.124 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8848ef0:::benchmark_data_smithi179_77138_object69191:head# 2024-02-09T17:27:04.124 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.124 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f885ad50:::benchmark_data_smithi179_77138_object83010:head# 2024-02-09T17:27:04.124 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.124 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8868172:::benchmark_data_smithi179_77138_object22088:head# 2024-02-09T17:27:04.124 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.125 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8870220:::benchmark_data_smithi179_77138_object112445:head# 2024-02-09T17:27:04.125 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.125 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f88773ce:::benchmark_data_smithi179_77138_object130915:head# 2024-02-09T17:27:04.125 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.125 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8877c68:::benchmark_data_smithi179_77138_object132165:head# 2024-02-09T17:27:04.125 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.125 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8889dc0:::benchmark_data_smithi179_77138_object110076:head# 2024-02-09T17:27:04.125 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.125 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f888b63c:::benchmark_data_smithi179_77138_object114541:head# 2024-02-09T17:27:04.125 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.125 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f888e64c:::benchmark_data_smithi179_77138_object97564:head# 2024-02-09T17:27:04.125 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.125 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8899eb1:::benchmark_data_smithi179_77138_object59108:head# 2024-02-09T17:27:04.125 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.126 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f88a0550:::benchmark_data_smithi179_77138_object80986:head# 2024-02-09T17:27:04.126 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.126 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f88bc8d2:::benchmark_data_smithi179_77138_object81510:head# 2024-02-09T17:27:04.126 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.126 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f88c6a8c:::benchmark_data_smithi179_77138_object70652:head# 2024-02-09T17:27:04.126 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.126 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f88c8210:::benchmark_data_smithi179_77138_object112613:head# 2024-02-09T17:27:04.126 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.126 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f88c962e:::benchmark_data_smithi179_77138_object132908:head# 2024-02-09T17:27:04.126 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.126 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f88cf6f9:::benchmark_data_smithi179_77138_object40862:head# 2024-02-09T17:27:04.126 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.126 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f88f42a6:::benchmark_data_smithi179_77138_object64486:head# 2024-02-09T17:27:04.126 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.126 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f890db58:::benchmark_data_smithi179_77138_object68774:head# 2024-02-09T17:27:04.126 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.127 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f891386b:::benchmark_data_smithi179_77138_object97337:head# 2024-02-09T17:27:04.127 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.127 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8915b2f:::benchmark_data_smithi179_77138_object90295:head# 2024-02-09T17:27:04.127 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.127 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f891aa09:::benchmark_data_smithi179_77138_object71860:head# 2024-02-09T17:27:04.127 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.127 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8942f3e:::benchmark_data_smithi179_77138_object89407:head# 2024-02-09T17:27:04.127 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.127 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f894bdc1:::benchmark_data_smithi179_77138_object115297:head# 2024-02-09T17:27:04.127 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.127 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8956e34:::benchmark_data_smithi179_77138_object107134:head# 2024-02-09T17:27:04.127 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.127 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f896026b:::benchmark_data_smithi179_77138_object74741:head# 2024-02-09T17:27:04.128 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.128 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f89681be:::benchmark_data_smithi179_77138_object100423:head# 2024-02-09T17:27:04.128 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.128 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f896d193:::benchmark_data_smithi179_77138_object58703:head# 2024-02-09T17:27:04.128 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.128 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f896ef5e:::benchmark_data_smithi179_77138_object132818:head# 2024-02-09T17:27:04.128 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.128 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f89709e9:::benchmark_data_smithi179_77138_object19908:head# 2024-02-09T17:27:04.128 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.128 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8971a22:::benchmark_data_smithi179_77138_object91337:head# 2024-02-09T17:27:04.128 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.128 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8975ea9:::benchmark_data_smithi179_77138_object92677:head# 2024-02-09T17:27:04.128 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.128 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f898eae4:::benchmark_data_smithi179_77138_object112144:head# 2024-02-09T17:27:04.129 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.129 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8992c90:::benchmark_data_smithi179_77138_object68112:head# 2024-02-09T17:27:04.129 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.129 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8996516:::benchmark_data_smithi179_77138_object86197:head# 2024-02-09T17:27:04.129 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.129 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f899824a:::benchmark_data_smithi179_77138_object71303:head# 2024-02-09T17:27:04.129 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.129 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f899a3db:::benchmark_data_smithi179_77138_object21880:head# 2024-02-09T17:27:04.129 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.129 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f89a5bae:::benchmark_data_smithi179_77138_object48308:head# 2024-02-09T17:27:04.129 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.129 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f89b8cab:::benchmark_data_smithi179_77138_object66167:head# 2024-02-09T17:27:04.129 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.129 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f89c3d7d:::benchmark_data_smithi179_77138_object77637:head# 2024-02-09T17:27:04.129 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.129 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f89da4be:::benchmark_data_smithi179_77138_object74414:head# 2024-02-09T17:27:04.130 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.130 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f89dac33:::benchmark_data_smithi179_77138_object94051:head# 2024-02-09T17:27:04.130 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.130 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f89ddbb8:::benchmark_data_smithi179_77138_object66138:head# 2024-02-09T17:27:04.130 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.130 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f89de14b:::benchmark_data_smithi179_77138_object47204:head# 2024-02-09T17:27:04.130 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.130 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8a25396:::benchmark_data_smithi179_77138_object45027:head# 2024-02-09T17:27:04.130 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.130 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8a26347:::benchmark_data_smithi179_77138_object42701:head# 2024-02-09T17:27:04.130 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.130 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8a2779e:::benchmark_data_smithi179_77138_object120917:head# 2024-02-09T17:27:04.130 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.130 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8a29604:::benchmark_data_smithi179_77138_object57067:head# 2024-02-09T17:27:04.130 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.131 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8a3a984:::benchmark_data_smithi179_77138_object50958:head# 2024-02-09T17:27:04.131 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.131 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8a3dc7a:::benchmark_data_smithi179_77138_object81135:head# 2024-02-09T17:27:04.131 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.131 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8a3f71c:::benchmark_data_smithi179_77138_object86080:head# 2024-02-09T17:27:04.131 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.131 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8a458dd:::benchmark_data_smithi179_77138_object116882:head# 2024-02-09T17:27:04.131 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.131 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8a474a3:::benchmark_data_smithi179_77138_object64796:head# 2024-02-09T17:27:04.131 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.131 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8a4cdec:::benchmark_data_smithi179_77138_object42196:head# 2024-02-09T17:27:04.131 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.131 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8a62ebe:::benchmark_data_smithi179_77138_object25389:head# 2024-02-09T17:27:04.131 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.132 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8a670a2:::benchmark_data_smithi179_77138_object58746:head# 2024-02-09T17:27:04.132 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.132 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8a67530:::benchmark_data_smithi179_77138_object60330:head# 2024-02-09T17:27:04.132 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.132 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8a676af:::benchmark_data_smithi179_77138_object71158:head# 2024-02-09T17:27:04.132 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.132 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8a6b808:::benchmark_data_smithi179_77138_object125142:head# 2024-02-09T17:27:04.132 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.132 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8a6cac2:::benchmark_data_smithi179_77138_object42994:head# 2024-02-09T17:27:04.132 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.132 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8a72f64:::benchmark_data_smithi179_77138_object103852:head# 2024-02-09T17:27:04.132 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.132 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8a7a951:::benchmark_data_smithi179_77138_object102810:head# 2024-02-09T17:27:04.132 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.133 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8a7c1de:::benchmark_data_smithi179_77138_object111067:head# 2024-02-09T17:27:04.133 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.133 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8a80a83:::benchmark_data_smithi179_77138_object119383:head# 2024-02-09T17:27:04.133 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.133 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8a87ac6:::benchmark_data_smithi179_77138_object68796:head# 2024-02-09T17:27:04.133 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.133 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8a910d4:::benchmark_data_smithi179_77138_object52114:head# 2024-02-09T17:27:04.133 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.133 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8a94742:::benchmark_data_smithi179_77138_object56231:head# 2024-02-09T17:27:04.133 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.133 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8ab2b98:::benchmark_data_smithi179_77138_object117824:head# 2024-02-09T17:27:04.133 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.133 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8aba317:::benchmark_data_smithi179_77138_object40700:head# 2024-02-09T17:27:04.133 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.134 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8ababb6:::benchmark_data_smithi179_77138_object130992:head# 2024-02-09T17:27:04.134 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.134 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8abe5f9:::benchmark_data_smithi179_77138_object101231:head# 2024-02-09T17:27:04.134 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.134 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8ac1fd5:::benchmark_data_smithi179_77138_object122609:head# 2024-02-09T17:27:04.134 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.134 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8ac6ba0:::benchmark_data_smithi179_77138_object99453:head# 2024-02-09T17:27:04.134 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.134 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8ac798f:::benchmark_data_smithi179_77138_object54335:head# 2024-02-09T17:27:04.134 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.134 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8acdb4d:::benchmark_data_smithi179_77138_object96415:head# 2024-02-09T17:27:04.134 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.134 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8adb279:::benchmark_data_smithi179_77138_object53470:head# 2024-02-09T17:27:04.134 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.134 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8ae2155:::benchmark_data_smithi179_77138_object91448:head# 2024-02-09T17:27:04.135 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.135 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8aee991:::benchmark_data_smithi179_77138_object90338:head# 2024-02-09T17:27:04.135 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.135 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8afad0d:::benchmark_data_smithi179_77138_object90669:head# 2024-02-09T17:27:04.135 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.135 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8b04b54:::benchmark_data_smithi179_77138_object52987:head# 2024-02-09T17:27:04.135 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.135 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8b05a8c:::benchmark_data_smithi179_77138_object47697:head# 2024-02-09T17:27:04.135 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.142 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8b0d3b8:::benchmark_data_smithi179_77138_object41734:head# 2024-02-09T17:27:04.142 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.142 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8b11cf0:::benchmark_data_smithi179_77138_object19870:head# 2024-02-09T17:27:04.142 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.142 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8b21892:::benchmark_data_smithi179_77138_object111883:head# 2024-02-09T17:27:04.142 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.142 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8b28b99:::benchmark_data_smithi179_77138_object47735:head# 2024-02-09T17:27:04.143 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.143 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8b29f36:::benchmark_data_smithi179_77138_object41715:head# 2024-02-09T17:27:04.143 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.143 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8b2d1a6:::benchmark_data_smithi179_77138_object21243:head# 2024-02-09T17:27:04.143 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.143 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8b326c9:::benchmark_data_smithi179_77138_object35412:head# 2024-02-09T17:27:04.143 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.143 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8b32dfe:::benchmark_data_smithi179_77138_object25338:head# 2024-02-09T17:27:04.143 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.143 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8b503b0:::benchmark_data_smithi179_77138_object129903:head# 2024-02-09T17:27:04.143 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.143 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8b8cf1f:::benchmark_data_smithi179_77138_object108472:head# 2024-02-09T17:27:04.143 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.143 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8b908c0:::benchmark_data_smithi179_77138_object67611:head# 2024-02-09T17:27:04.144 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.144 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8b962cd:::benchmark_data_smithi179_77138_object25277:head# 2024-02-09T17:27:04.144 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.144 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8b9cca1:::benchmark_data_smithi179_77138_object78917:head# 2024-02-09T17:27:04.144 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.144 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8ba0417:::benchmark_data_smithi179_77138_object110960:head# 2024-02-09T17:27:04.144 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.144 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8ba5458:::benchmark_data_smithi179_77138_object116744:head# 2024-02-09T17:27:04.144 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.144 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8baf976:::benchmark_data_smithi179_77138_object62535:head# 2024-02-09T17:27:04.144 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.144 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8bbae34:::benchmark_data_smithi179_77138_object20969:head# 2024-02-09T17:27:04.144 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.144 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8bd9d2c:::benchmark_data_smithi179_77138_object51851:head# 2024-02-09T17:27:04.145 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.145 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8bed305:::benchmark_data_smithi179_77138_object101399:head# 2024-02-09T17:27:04.145 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.145 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8bef0d6:::benchmark_data_smithi179_77138_object52588:head# 2024-02-09T17:27:04.145 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.145 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8bf01fc:::benchmark_data_smithi179_77138_object45075:head# 2024-02-09T17:27:04.145 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.145 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8bf59fd:::benchmark_data_smithi179_77138_object46730:head# 2024-02-09T17:27:04.145 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.145 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8c0a97d:::benchmark_data_smithi179_77138_object78083:head# 2024-02-09T17:27:04.145 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.145 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8c10321:::benchmark_data_smithi179_77138_object105500:head# 2024-02-09T17:27:04.145 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.145 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8c12847:::benchmark_data_smithi179_77138_object29382:head# 2024-02-09T17:27:04.146 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.146 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8c195ae:::benchmark_data_smithi179_77138_object61586:head# 2024-02-09T17:27:04.146 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.146 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8c1d29c:::benchmark_data_smithi179_77138_object21568:head# 2024-02-09T17:27:04.146 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.146 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8c2401d:::benchmark_data_smithi179_77138_object89773:head# 2024-02-09T17:27:04.146 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.146 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8c25e59:::benchmark_data_smithi179_77138_object57959:head# 2024-02-09T17:27:04.146 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.146 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8c2c231:::benchmark_data_smithi179_77138_object105198:head# 2024-02-09T17:27:04.146 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.146 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8c2ee2f:::benchmark_data_smithi179_77138_object31833:head# 2024-02-09T17:27:04.146 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.146 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8c39a6f:::benchmark_data_smithi179_77138_object132770:head# 2024-02-09T17:27:04.147 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.147 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8c4063e:::benchmark_data_smithi179_77138_object112062:head# 2024-02-09T17:27:04.147 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.147 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8c494b6:::benchmark_data_smithi179_77138_object42993:head# 2024-02-09T17:27:04.147 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.147 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8c580f6:::benchmark_data_smithi179_77138_object52289:head# 2024-02-09T17:27:04.147 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.147 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8c5d062:::benchmark_data_smithi179_77138_object73890:head# 2024-02-09T17:27:04.147 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.147 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8c7340d:::benchmark_data_smithi179_77138_object97890:head# 2024-02-09T17:27:04.147 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.147 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8c85fc0:::benchmark_data_smithi179_77138_object95498:head# 2024-02-09T17:27:04.147 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.148 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8c97928:::benchmark_data_smithi179_77138_object132941:head# 2024-02-09T17:27:04.148 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.148 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8cb1db5:::benchmark_data_smithi179_77138_object82711:head# 2024-02-09T17:27:04.148 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.148 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8ce612c:::benchmark_data_smithi179_77138_object132323:head# 2024-02-09T17:27:04.148 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.148 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8cf1876:::benchmark_data_smithi179_77138_object83731:head# 2024-02-09T17:27:04.148 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.148 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8cf513d:::benchmark_data_smithi179_77138_object96707:head# 2024-02-09T17:27:04.148 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.148 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8d122a6:::benchmark_data_smithi179_77138_object63831:head# 2024-02-09T17:27:04.148 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.148 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8d14896:::benchmark_data_smithi179_77138_object93735:head# 2024-02-09T17:27:04.149 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.149 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8d1c79f:::benchmark_data_smithi179_77138_object132667:head# 2024-02-09T17:27:04.149 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.149 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8d2385e:::benchmark_data_smithi179_77138_object115913:head# 2024-02-09T17:27:04.149 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.149 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8d3fbb2:::benchmark_data_smithi179_77138_object121960:head# 2024-02-09T17:27:04.149 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.149 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8d45d20:::benchmark_data_smithi179_77138_object76483:head# 2024-02-09T17:27:04.149 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.149 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8d48c6e:::benchmark_data_smithi179_77138_object48733:head# 2024-02-09T17:27:04.149 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.149 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8d61b4c:::benchmark_data_smithi179_77138_object42856:head# 2024-02-09T17:27:04.149 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.149 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8d6620c:::benchmark_data_smithi179_77138_object129698:head# 2024-02-09T17:27:04.149 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.150 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8d6a977:::benchmark_data_smithi179_77138_object104915:head# 2024-02-09T17:27:04.150 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.150 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8d77174:::benchmark_data_smithi179_77138_object43395:head# 2024-02-09T17:27:04.150 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.150 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8d799c7:::benchmark_data_smithi179_77138_object63356:head# 2024-02-09T17:27:04.150 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.150 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8d8c5dc:::benchmark_data_smithi179_77138_object123395:head# 2024-02-09T17:27:04.150 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.150 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8d8cdec:::benchmark_data_smithi179_77138_object99037:head# 2024-02-09T17:27:04.150 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.150 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8d93b1d:::benchmark_data_smithi179_77138_object91475:head# 2024-02-09T17:27:04.150 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.150 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8d98043:::benchmark_data_smithi179_77138_object76108:head# 2024-02-09T17:27:04.150 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.150 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8d98b47:::benchmark_data_smithi179_77138_object129991:head# 2024-02-09T17:27:04.151 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.151 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8daff14:::benchmark_data_smithi179_77138_object126161:head# 2024-02-09T17:27:04.151 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.151 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8db2c3a:::benchmark_data_smithi179_77138_object62998:head# 2024-02-09T17:27:04.151 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.151 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8db4d05:::benchmark_data_smithi179_77138_object25005:head# 2024-02-09T17:27:04.151 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.151 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8dc98dc:::benchmark_data_smithi179_77138_object103603:head# 2024-02-09T17:27:04.151 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.151 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8deec19:::benchmark_data_smithi179_77138_object95423:head# 2024-02-09T17:27:04.151 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.151 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8dfa2b9:::benchmark_data_smithi179_77138_object79896:head# 2024-02-09T17:27:04.151 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.151 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8e04606:::benchmark_data_smithi179_77138_object50445:head# 2024-02-09T17:27:04.152 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.152 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8e0a630:::benchmark_data_smithi179_77138_object111170:head# 2024-02-09T17:27:04.152 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.152 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8e0d3c5:::benchmark_data_smithi179_77138_object73136:head# 2024-02-09T17:27:04.152 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.152 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8e0d44a:::benchmark_data_smithi179_77138_object43271:head# 2024-02-09T17:27:04.152 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.152 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8e194fb:::benchmark_data_smithi179_77138_object95872:head# 2024-02-09T17:27:04.152 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.152 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8e1c4e8:::benchmark_data_smithi179_77138_object82083:head# 2024-02-09T17:27:04.152 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.152 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8e2c650:::benchmark_data_smithi179_77138_object76414:head# 2024-02-09T17:27:04.152 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.152 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8e2e7ab:::benchmark_data_smithi179_77138_object73171:head# 2024-02-09T17:27:04.153 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.157 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8e33670:::benchmark_data_smithi179_77138_object103404:head# 2024-02-09T17:27:04.157 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.158 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8e43b57:::benchmark_data_smithi179_77138_object120079:head# 2024-02-09T17:27:04.158 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.158 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8e44df1:::benchmark_data_smithi179_77138_object41617:head# 2024-02-09T17:27:04.158 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.158 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8e4b06d:::benchmark_data_smithi179_77138_object63554:head# 2024-02-09T17:27:04.158 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.158 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8e55114:::benchmark_data_smithi179_77138_object35276:head# 2024-02-09T17:27:04.158 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.158 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8e660cf:::benchmark_data_smithi179_77138_object85077:head# 2024-02-09T17:27:04.158 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.158 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8e6b710:::benchmark_data_smithi179_77138_object51285:head# 2024-02-09T17:27:04.158 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.158 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8e6c945:::benchmark_data_smithi179_77138_object118572:head# 2024-02-09T17:27:04.158 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.158 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8e74d0e:::benchmark_data_smithi179_77138_object35031:head# 2024-02-09T17:27:04.159 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.159 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8e7f38d:::benchmark_data_smithi179_77138_object72916:head# 2024-02-09T17:27:04.159 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.159 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8e99709:::benchmark_data_smithi179_77138_object63677:head# 2024-02-09T17:27:04.159 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.159 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8eb87f7:::benchmark_data_smithi179_77138_object97774:head# 2024-02-09T17:27:04.159 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.159 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8ec6595:::benchmark_data_smithi179_77138_object89096:head# 2024-02-09T17:27:04.159 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.159 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8ec77fc:::benchmark_data_smithi179_77138_object24653:head# 2024-02-09T17:27:04.159 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.159 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8ecbd95:::benchmark_data_smithi179_77138_object128295:head# 2024-02-09T17:27:04.159 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.159 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8ecc86e:::benchmark_data_smithi179_77138_object123936:head# 2024-02-09T17:27:04.160 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.160 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8eda3ba:::benchmark_data_smithi179_77138_object54214:head# 2024-02-09T17:27:04.160 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.160 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8ee7ff4:::benchmark_data_smithi179_77138_object44923:head# 2024-02-09T17:27:04.160 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.160 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8eea117:::benchmark_data_smithi179_77138_object102350:head# 2024-02-09T17:27:04.160 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.160 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8ef36e6:::benchmark_data_smithi179_77138_object85191:head# 2024-02-09T17:27:04.160 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.160 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8efb731:::benchmark_data_smithi179_77138_object76492:head# 2024-02-09T17:27:04.160 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.160 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8eff535:::benchmark_data_smithi179_77138_object37254:head# 2024-02-09T17:27:04.160 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.160 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8f07c91:::benchmark_data_smithi179_77138_object77062:head# 2024-02-09T17:27:04.161 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.161 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8f0f916:::benchmark_data_smithi179_77138_object39643:head# 2024-02-09T17:27:04.161 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.161 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8f16e0d:::benchmark_data_smithi179_77138_object76564:head# 2024-02-09T17:27:04.161 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.161 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8f1ac3d:::benchmark_data_smithi179_77138_object131764:head# 2024-02-09T17:27:04.161 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.161 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8f285c6:::benchmark_data_smithi179_77138_object131108:head# 2024-02-09T17:27:04.161 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.161 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8f2d710:::benchmark_data_smithi179_77138_object46877:head# 2024-02-09T17:27:04.161 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.161 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8f3350c:::benchmark_data_smithi179_77138_object129730:head# 2024-02-09T17:27:04.161 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.161 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8f38564:::benchmark_data_smithi179_77138_object21212:head# 2024-02-09T17:27:04.162 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.162 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8f3e88a:::benchmark_data_smithi179_77138_object66734:head# 2024-02-09T17:27:04.162 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.162 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8f4877c:::benchmark_data_smithi179_77138_object35571:head# 2024-02-09T17:27:04.162 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.162 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8f490a7:::benchmark_data_smithi179_77138_object87540:head# 2024-02-09T17:27:04.162 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.162 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8f4f366:::benchmark_data_smithi179_77138_object77458:head# 2024-02-09T17:27:04.162 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.162 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8f51b26:::benchmark_data_smithi179_77138_object72707:head# 2024-02-09T17:27:04.162 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.162 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8f58aa9:::benchmark_data_smithi179_77138_object60890:head# 2024-02-09T17:27:04.162 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.162 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8f5b3aa:::benchmark_data_smithi179_77138_object92722:head# 2024-02-09T17:27:04.162 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.163 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8f61e4e:::benchmark_data_smithi179_77138_object67574:head# 2024-02-09T17:27:04.163 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.163 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8f67665:::benchmark_data_smithi179_77138_object51827:head# 2024-02-09T17:27:04.163 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.163 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8f6fb34:::benchmark_data_smithi179_77138_object58323:head# 2024-02-09T17:27:04.163 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.163 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8f7375e:::benchmark_data_smithi179_77138_object56546:head# 2024-02-09T17:27:04.163 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.163 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8f74aa1:::benchmark_data_smithi179_77138_object36988:head# 2024-02-09T17:27:04.163 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.163 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8f793d9:::benchmark_data_smithi179_77138_object64439:head# 2024-02-09T17:27:04.163 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.163 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8f95ddc:::benchmark_data_smithi179_77138_object82495:head# 2024-02-09T17:27:04.163 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.163 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8f9884f:::benchmark_data_smithi179_77138_object115523:head# 2024-02-09T17:27:04.164 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.164 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8fa0a8f:::benchmark_data_smithi179_77138_object70457:head# 2024-02-09T17:27:04.164 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.164 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8fa6669:::benchmark_data_smithi179_77138_object102108:head# 2024-02-09T17:27:04.164 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.172 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8fb12e3:::benchmark_data_smithi179_77138_object74388:head# 2024-02-09T17:27:04.172 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.172 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8fb596d:::benchmark_data_smithi179_77138_object55806:head# 2024-02-09T17:27:04.172 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.172 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8fc00af:::benchmark_data_smithi179_77138_object107388:head# 2024-02-09T17:27:04.172 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.172 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8fc188b:::benchmark_data_smithi179_77138_object99521:head# 2024-02-09T17:27:04.172 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.172 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8fc8923:::benchmark_data_smithi179_77138_object113804:head# 2024-02-09T17:27:04.172 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.173 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8fcbcf9:::benchmark_data_smithi179_77138_object96619:head# 2024-02-09T17:27:04.173 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.173 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8fce684:::benchmark_data_smithi179_77138_object80405:head# 2024-02-09T17:27:04.173 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.173 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8fdd646:::benchmark_data_smithi179_77138_object105838:head# 2024-02-09T17:27:04.173 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.173 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8fed861:::benchmark_data_smithi179_77138_object78903:head# 2024-02-09T17:27:04.173 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.173 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f8ff40a2:::benchmark_data_smithi179_77138_object35669:head# 2024-02-09T17:27:04.173 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.174 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f90047f2:::benchmark_data_smithi179_77138_object82208:head# 2024-02-09T17:27:04.174 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.174 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f900c266:::benchmark_data_smithi179_77138_object77189:head# 2024-02-09T17:27:04.174 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.174 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f90105ca:::benchmark_data_smithi179_77138_object126236:head# 2024-02-09T17:27:04.174 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.174 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9010c53:::benchmark_data_smithi179_77138_object77205:head# 2024-02-09T17:27:04.175 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.175 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9014699:::benchmark_data_smithi179_77138_object28517:head# 2024-02-09T17:27:04.175 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.175 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f901f921:::benchmark_data_smithi179_77138_object34185:head# 2024-02-09T17:27:04.175 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.175 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9028634:::benchmark_data_smithi179_77138_object84976:head# 2024-02-09T17:27:04.175 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.175 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f902d575:::benchmark_data_smithi179_77138_object97358:head# 2024-02-09T17:27:04.176 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.176 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f903077f:::benchmark_data_smithi179_77138_object97306:head# 2024-02-09T17:27:04.176 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.176 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f903196c:::benchmark_data_smithi179_77138_object27551:head# 2024-02-09T17:27:04.176 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.176 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f904fac8:::benchmark_data_smithi179_77138_object29052:head# 2024-02-09T17:27:04.176 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.177 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f905551d:::benchmark_data_smithi179_77138_object59052:head# 2024-02-09T17:27:04.177 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.177 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f905891f:::benchmark_data_smithi179_77138_object25068:head# 2024-02-09T17:27:04.177 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.177 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f907a985:::benchmark_data_smithi179_77138_object111496:head# 2024-02-09T17:27:04.177 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.177 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f90852a7:::benchmark_data_smithi179_77138_object49776:head# 2024-02-09T17:27:04.177 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.178 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9089deb:::benchmark_data_smithi179_77138_object75968:head# 2024-02-09T17:27:04.178 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.178 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f909352a:::benchmark_data_smithi179_77138_object95826:head# 2024-02-09T17:27:04.178 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.178 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f90a56bd:::benchmark_data_smithi179_77138_object68768:head# 2024-02-09T17:27:04.178 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.178 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f90aa573:::benchmark_data_smithi179_77138_object66917:head# 2024-02-09T17:27:04.179 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.179 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f90c68e4:::benchmark_data_smithi179_77138_object101500:head# 2024-02-09T17:27:04.179 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.179 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f90e08fb:::benchmark_data_smithi179_77138_object95923:head# 2024-02-09T17:27:04.179 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.179 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f90e9e99:::benchmark_data_smithi179_77138_object63785:head# 2024-02-09T17:27:04.179 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.179 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f90eb53c:::benchmark_data_smithi179_77138_object68718:head# 2024-02-09T17:27:04.180 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.180 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f90f4733:::benchmark_data_smithi179_77138_object93715:head# 2024-02-09T17:27:04.180 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.180 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f90f7a36:::benchmark_data_smithi179_77138_object21443:head# 2024-02-09T17:27:04.180 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.180 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f910d620:::benchmark_data_smithi179_77138_object65336:head# 2024-02-09T17:27:04.180 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.181 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f911777c:::benchmark_data_smithi179_77138_object68411:head# 2024-02-09T17:27:04.181 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.181 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f911f9d7:::benchmark_data_smithi179_77138_object49733:head# 2024-02-09T17:27:04.181 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.181 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9125705:::benchmark_data_smithi179_77138_object123371:head# 2024-02-09T17:27:04.181 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.182 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9125b3f:::benchmark_data_smithi179_77138_object114349:head# 2024-02-09T17:27:04.182 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.182 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9128627:::benchmark_data_smithi179_77138_object27230:head# 2024-02-09T17:27:04.182 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.182 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9129237:::benchmark_data_smithi179_77138_object52258:head# 2024-02-09T17:27:04.182 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.182 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f912ab3f:::benchmark_data_smithi179_77138_object43731:head# 2024-02-09T17:27:04.182 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.182 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f912ad7a:::benchmark_data_smithi179_77138_object20416:head# 2024-02-09T17:27:04.182 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.182 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9151b42:::benchmark_data_smithi179_77138_object95445:head# 2024-02-09T17:27:04.183 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.183 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9167748:::benchmark_data_smithi179_77138_object104371:head# 2024-02-09T17:27:04.183 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.183 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f916818d:::benchmark_data_smithi179_77138_object115542:head# 2024-02-09T17:27:04.183 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.183 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f916e3c9:::benchmark_data_smithi179_77138_object41186:head# 2024-02-09T17:27:04.183 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.183 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f917659f:::benchmark_data_smithi179_77138_object107652:head# 2024-02-09T17:27:04.183 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.183 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9177e79:::benchmark_data_smithi179_77138_object85361:head# 2024-02-09T17:27:04.183 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.183 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f91787d8:::benchmark_data_smithi179_77138_object79388:head# 2024-02-09T17:27:04.183 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.183 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f917bb47:::benchmark_data_smithi179_77138_object39179:head# 2024-02-09T17:27:04.184 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.184 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f917fb67:::benchmark_data_smithi179_77138_object28056:head# 2024-02-09T17:27:04.184 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.184 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f918eb1a:::benchmark_data_smithi179_77138_object84945:head# 2024-02-09T17:27:04.184 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.184 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9190291:::benchmark_data_smithi179_77138_object116437:head# 2024-02-09T17:27:04.184 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.184 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f919dc06:::benchmark_data_smithi179_77138_object52752:head# 2024-02-09T17:27:04.184 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.184 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f91a0e03:::benchmark_data_smithi179_77138_object107529:head# 2024-02-09T17:27:04.184 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.184 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f91a4ac0:::benchmark_data_smithi179_77138_object122419:head# 2024-02-09T17:27:04.184 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.184 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f91ab4bc:::benchmark_data_smithi179_77138_object89026:head# 2024-02-09T17:27:04.185 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.185 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f91bec39:::benchmark_data_smithi179_77138_object51307:head# 2024-02-09T17:27:04.185 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.185 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f91c6ef2:::benchmark_data_smithi179_77138_object102321:head# 2024-02-09T17:27:04.185 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.185 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f91c86f5:::benchmark_data_smithi179_77138_object45014:head# 2024-02-09T17:27:04.185 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.185 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:03 smithi179 ceph-mon[43975]: pgmap v459: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 5.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 900 op/s; 24390/290988 objects degraded (8.382%) 2024-02-09T17:27:04.185 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:03 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2213779330' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:27:04.185 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:03 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:27:04.198 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f91cd247:::benchmark_data_smithi179_77138_object72968:head# 2024-02-09T17:27:04.198 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.198 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f91e8ca8:::benchmark_data_smithi179_77138_object64823:head# 2024-02-09T17:27:04.198 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.198 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f91ed8a0:::benchmark_data_smithi179_77138_object73732:head# 2024-02-09T17:27:04.198 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.198 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f91f5a4c:::benchmark_data_smithi179_77138_object122575:head# 2024-02-09T17:27:04.198 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.198 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9209748:::benchmark_data_smithi179_77138_object132999:head# 2024-02-09T17:27:04.198 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.199 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9209d51:::benchmark_data_smithi179_77138_object46982:head# 2024-02-09T17:27:04.199 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.199 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9213292:::benchmark_data_smithi179_77138_object20829:head# 2024-02-09T17:27:04.199 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.199 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f922bcb6:::benchmark_data_smithi179_77138_object51300:head# 2024-02-09T17:27:04.199 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.199 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f92322e2:::benchmark_data_smithi179_77138_object117819:head# 2024-02-09T17:27:04.199 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.199 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f92360a0:::benchmark_data_smithi179_77138_object76813:head# 2024-02-09T17:27:04.199 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.199 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f923d2af:::benchmark_data_smithi179_77138_object115480:head# 2024-02-09T17:27:04.199 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.199 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f923ff7e:::benchmark_data_smithi179_77138_object129842:head# 2024-02-09T17:27:04.199 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.199 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9246749:::benchmark_data_smithi179_77138_object39795:head# 2024-02-09T17:27:04.200 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.200 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9250375:::benchmark_data_smithi179_77138_object81994:head# 2024-02-09T17:27:04.200 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.200 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f925714d:::benchmark_data_smithi179_77138_object30886:head# 2024-02-09T17:27:04.200 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.200 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f92618d9:::benchmark_data_smithi179_77138_object59855:head# 2024-02-09T17:27:04.200 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.200 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f92664fb:::benchmark_data_smithi179_77138_object44214:head# 2024-02-09T17:27:04.200 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.200 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f926a6e2:::benchmark_data_smithi179_77138_object52139:head# 2024-02-09T17:27:04.200 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.200 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f926b20b:::benchmark_data_smithi179_77138_object79455:head# 2024-02-09T17:27:04.200 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.201 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9279eee:::benchmark_data_smithi179_77138_object118074:head# 2024-02-09T17:27:04.201 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.201 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f927b604:::benchmark_data_smithi179_77138_object54761:head# 2024-02-09T17:27:04.201 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.201 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f928b0fa:::benchmark_data_smithi179_77138_object44354:head# 2024-02-09T17:27:04.201 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.201 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f928b901:::benchmark_data_smithi179_77138_object103862:head# 2024-02-09T17:27:04.201 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.201 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f928c7e0:::benchmark_data_smithi179_77138_object131224:head# 2024-02-09T17:27:04.201 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.201 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f928f541:::benchmark_data_smithi179_77138_object89586:head# 2024-02-09T17:27:04.201 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.201 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f92939c8:::benchmark_data_smithi179_77138_object81343:head# 2024-02-09T17:27:04.201 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.202 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9299595:::benchmark_data_smithi179_77138_object130678:head# 2024-02-09T17:27:04.202 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.202 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f929a6f0:::benchmark_data_smithi179_77138_object121059:head# 2024-02-09T17:27:04.202 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.202 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f92a828c:::benchmark_data_smithi179_77138_object74967:head# 2024-02-09T17:27:04.202 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.202 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f92d1f10:::benchmark_data_smithi179_77138_object43116:head# 2024-02-09T17:27:04.202 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.202 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f92d74b3:::benchmark_data_smithi179_77138_object108062:head# 2024-02-09T17:27:04.202 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.202 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f92dffee:::benchmark_data_smithi179_77138_object71971:head# 2024-02-09T17:27:04.202 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.202 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f92ec29b:::benchmark_data_smithi179_77138_object122174:head# 2024-02-09T17:27:04.202 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.203 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f92f219e:::benchmark_data_smithi179_77138_object55348:head# 2024-02-09T17:27:04.203 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.203 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f92fc0c4:::benchmark_data_smithi179_77138_object129754:head# 2024-02-09T17:27:04.203 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.203 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f92ff216:::benchmark_data_smithi179_77138_object107133:head# 2024-02-09T17:27:04.203 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.203 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f93001cd:::benchmark_data_smithi179_77138_object87455:head# 2024-02-09T17:27:04.203 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.203 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9306927:::benchmark_data_smithi179_77138_object106324:head# 2024-02-09T17:27:04.203 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.203 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f930e453:::benchmark_data_smithi179_77138_object78867:head# 2024-02-09T17:27:04.203 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.203 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9319d83:::benchmark_data_smithi179_77138_object51697:head# 2024-02-09T17:27:04.203 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.204 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f931b8b7:::benchmark_data_smithi179_77138_object62126:head# 2024-02-09T17:27:04.204 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.204 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9321c95:::benchmark_data_smithi179_77138_object115045:head# 2024-02-09T17:27:04.204 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.204 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f93345ae:::benchmark_data_smithi179_77138_object40938:head# 2024-02-09T17:27:04.204 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.204 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9337c39:::benchmark_data_smithi179_77138_object19741:head# 2024-02-09T17:27:04.204 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.204 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f933bd49:::benchmark_data_smithi179_77138_object61399:head# 2024-02-09T17:27:04.204 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.204 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f933e864:::benchmark_data_smithi179_77138_object99385:head# 2024-02-09T17:27:04.204 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.204 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9344651:::benchmark_data_smithi179_77138_object26099:head# 2024-02-09T17:27:04.204 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.205 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f934862e:::benchmark_data_smithi179_77138_object40432:head# 2024-02-09T17:27:04.205 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.205 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f934a30f:::benchmark_data_smithi179_77138_object60314:head# 2024-02-09T17:27:04.205 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.205 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9352dfb:::benchmark_data_smithi179_77138_object115934:head# 2024-02-09T17:27:04.205 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.205 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9357e14:::benchmark_data_smithi179_77138_object63880:head# 2024-02-09T17:27:04.205 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.205 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f935cf27:::benchmark_data_smithi179_77138_object98230:head# 2024-02-09T17:27:04.205 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.205 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f936a9b3:::benchmark_data_smithi179_77138_object127141:head# 2024-02-09T17:27:04.205 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.205 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f937107a:::benchmark_data_smithi179_77138_object124674:head# 2024-02-09T17:27:04.206 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.206 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f93722f8:::benchmark_data_smithi179_77138_object126033:head# 2024-02-09T17:27:04.206 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.206 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f937a960:::benchmark_data_smithi179_77138_object47603:head# 2024-02-09T17:27:04.206 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.206 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9387124:::benchmark_data_smithi179_77138_object52749:head# 2024-02-09T17:27:04.206 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.206 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9392fe5:::benchmark_data_smithi179_77138_object31873:head# 2024-02-09T17:27:04.206 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.206 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f939b058:::benchmark_data_smithi179_77138_object41320:head# 2024-02-09T17:27:04.206 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.206 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f93acd30:::benchmark_data_smithi179_77138_object96776:head# 2024-02-09T17:27:04.206 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.206 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f93b41a0:::benchmark_data_smithi179_77138_object81991:head# 2024-02-09T17:27:04.207 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.207 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f93c309a:::benchmark_data_smithi179_77138_object122950:head# 2024-02-09T17:27:04.207 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.207 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f93e51aa:::benchmark_data_smithi179_77138_object80016:head# 2024-02-09T17:27:04.207 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.207 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f93ecb98:::benchmark_data_smithi179_77138_object61566:head# 2024-02-09T17:27:04.207 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.207 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f93f97f7:::benchmark_data_smithi179_77138_object25826:head# 2024-02-09T17:27:04.207 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.207 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f93ff44d:::benchmark_data_smithi179_77138_object69156:head# 2024-02-09T17:27:04.207 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.207 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9403987:::benchmark_data_smithi179_77138_object22639:head# 2024-02-09T17:27:04.207 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.207 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9408dea:::benchmark_data_smithi179_77138_object109466:head# 2024-02-09T17:27:04.208 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.208 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f940b360:::benchmark_data_smithi179_77138_object51951:head# 2024-02-09T17:27:04.208 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.208 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f941b9b3:::benchmark_data_smithi179_77138_object22282:head# 2024-02-09T17:27:04.208 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.208 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9423971:::benchmark_data_smithi179_77138_object104389:head# 2024-02-09T17:27:04.208 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.208 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9429048:::benchmark_data_smithi179_77138_object27181:head# 2024-02-09T17:27:04.208 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.208 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9453b01:::benchmark_data_smithi179_77138_object128169:head# 2024-02-09T17:27:04.208 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.208 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9456cfa:::benchmark_data_smithi179_77138_object34351:head# 2024-02-09T17:27:04.208 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.209 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f945ddf2:::benchmark_data_smithi179_77138_object87044:head# 2024-02-09T17:27:04.209 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.209 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f945e23a:::benchmark_data_smithi179_77138_object39914:head# 2024-02-09T17:27:04.209 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.209 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f945e796:::benchmark_data_smithi179_77138_object87307:head# 2024-02-09T17:27:04.209 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.209 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9467d19:::benchmark_data_smithi179_77138_object48920:head# 2024-02-09T17:27:04.209 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.209 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f946b697:::benchmark_data_smithi179_77138_object52871:head# 2024-02-09T17:27:04.209 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.209 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f947e7f4:::benchmark_data_smithi179_77138_object60080:head# 2024-02-09T17:27:04.209 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.209 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f947f437:::benchmark_data_smithi179_77138_object79527:head# 2024-02-09T17:27:04.209 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.210 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f947fd46:::benchmark_data_smithi179_77138_object100493:head# 2024-02-09T17:27:04.210 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.210 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f94863e0:::benchmark_data_smithi179_77138_object43423:head# 2024-02-09T17:27:04.210 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.210 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f948943f:::benchmark_data_smithi179_77138_object55070:head# 2024-02-09T17:27:04.210 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.210 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f94927df:::benchmark_data_smithi179_77138_object25802:head# 2024-02-09T17:27:04.210 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.210 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f949995d:::benchmark_data_smithi179_77138_object80905:head# 2024-02-09T17:27:04.210 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.210 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f949faa7:::benchmark_data_smithi179_77138_object48758:head# 2024-02-09T17:27:04.210 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.210 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f949fce1:::benchmark_data_smithi179_77138_object121473:head# 2024-02-09T17:27:04.210 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.210 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f94a6e15:::benchmark_data_smithi179_77138_object75810:head# 2024-02-09T17:27:04.211 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.211 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f94b27b0:::benchmark_data_smithi179_77138_object101698:head# 2024-02-09T17:27:04.211 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.211 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f94bca77:::benchmark_data_smithi179_77138_object63529:head# 2024-02-09T17:27:04.211 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.211 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f94c1155:::benchmark_data_smithi179_77138_object90328:head# 2024-02-09T17:27:04.211 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.211 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f94cbf94:::benchmark_data_smithi179_77138_object105038:head# 2024-02-09T17:27:04.211 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.211 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f94cdabe:::benchmark_data_smithi179_77138_object39544:head# 2024-02-09T17:27:04.211 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.211 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f94df194:::benchmark_data_smithi179_77138_object24950:head# 2024-02-09T17:27:04.211 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.211 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f94ef686:::benchmark_data_smithi179_77138_object124261:head# 2024-02-09T17:27:04.212 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.224 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f94f5a6b:::benchmark_data_smithi179_77138_object112819:head# 2024-02-09T17:27:04.224 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.224 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9508196:::benchmark_data_smithi179_77138_object31052:head# 2024-02-09T17:27:04.224 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.224 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9512109:::benchmark_data_smithi179_77138_object44454:head# 2024-02-09T17:27:04.225 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.225 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9519305:::benchmark_data_smithi179_77138_object32309:head# 2024-02-09T17:27:04.225 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.225 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f951af9a:::benchmark_data_smithi179_77138_object64337:head# 2024-02-09T17:27:04.225 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.225 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f951dc7e:::benchmark_data_smithi179_77138_object109367:head# 2024-02-09T17:27:04.225 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.225 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9523cf5:::benchmark_data_smithi179_77138_object86687:head# 2024-02-09T17:27:04.225 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.225 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9524296:::benchmark_data_smithi179_77138_object44227:head# 2024-02-09T17:27:04.225 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.225 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9524840:::benchmark_data_smithi179_77138_object118468:head# 2024-02-09T17:27:04.225 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.225 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9524e82:::benchmark_data_smithi179_77138_object101825:head# 2024-02-09T17:27:04.226 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.226 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f952adcd:::benchmark_data_smithi179_77138_object43088:head# 2024-02-09T17:27:04.226 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.226 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f953c361:::benchmark_data_smithi179_77138_object75206:head# 2024-02-09T17:27:04.226 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.226 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f953e3d8:::benchmark_data_smithi179_77138_object25644:head# 2024-02-09T17:27:04.226 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.226 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f954b228:::benchmark_data_smithi179_77138_object132430:head# 2024-02-09T17:27:04.226 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.226 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f954df45:::benchmark_data_smithi179_77138_object114393:head# 2024-02-09T17:27:04.226 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.226 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f95640e3:::benchmark_data_smithi179_77138_object34818:head# 2024-02-09T17:27:04.226 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.226 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9567079:::benchmark_data_smithi179_77138_object71805:head# 2024-02-09T17:27:04.226 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.227 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f956d9b3:::benchmark_data_smithi179_77138_object125170:head# 2024-02-09T17:27:04.227 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.227 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9570c6c:::benchmark_data_smithi179_77138_object84698:head# 2024-02-09T17:27:04.227 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.227 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f95732fa:::benchmark_data_smithi179_77138_object108755:head# 2024-02-09T17:27:04.227 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.227 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f95a141d:::benchmark_data_smithi179_77138_object94438:head# 2024-02-09T17:27:04.227 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.227 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f95a7793:::benchmark_data_smithi179_77138_object81609:head# 2024-02-09T17:27:04.227 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.227 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f95a8c7c:::benchmark_data_smithi179_77138_object48411:head# 2024-02-09T17:27:04.227 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.227 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f95ad770:::benchmark_data_smithi179_77138_object101712:head# 2024-02-09T17:27:04.227 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.228 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f95afbaa:::benchmark_data_smithi179_77138_object109179:head# 2024-02-09T17:27:04.228 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.228 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f95b73a9:::benchmark_data_smithi179_77138_object70590:head# 2024-02-09T17:27:04.228 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.228 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f95d1db8:::benchmark_data_smithi179_77138_object109647:head# 2024-02-09T17:27:04.228 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.228 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f95d1fca:::benchmark_data_smithi179_77138_object57715:head# 2024-02-09T17:27:04.228 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.228 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f95deadf:::benchmark_data_smithi179_77138_object53922:head# 2024-02-09T17:27:04.228 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.228 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f95defee:::benchmark_data_smithi179_77138_object70968:head# 2024-02-09T17:27:04.228 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.229 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9602ba8:::benchmark_data_smithi179_77138_object74208:head# 2024-02-09T17:27:04.229 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.229 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9624537:::benchmark_data_smithi179_77138_object54700:head# 2024-02-09T17:27:04.229 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.229 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9629d71:::benchmark_data_smithi179_77138_object22885:head# 2024-02-09T17:27:04.229 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.229 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9632ad6:::benchmark_data_smithi179_77138_object124584:head# 2024-02-09T17:27:04.229 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.229 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f963f19e:::benchmark_data_smithi179_77138_object103808:head# 2024-02-09T17:27:04.229 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.229 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f964b8b5:::benchmark_data_smithi179_77138_object100668:head# 2024-02-09T17:27:04.229 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.229 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9662fa2:::benchmark_data_smithi179_77138_object120522:head# 2024-02-09T17:27:04.229 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.230 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9667dc7:::benchmark_data_smithi179_77138_object72179:head# 2024-02-09T17:27:04.230 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.230 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f966b7d2:::benchmark_data_smithi179_77138_object63114:head# 2024-02-09T17:27:04.230 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.230 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f967d0c4:::benchmark_data_smithi179_77138_object93165:head# 2024-02-09T17:27:04.230 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.230 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f96815fc:::benchmark_data_smithi179_77138_object101372:head# 2024-02-09T17:27:04.230 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.230 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9687fcf:::benchmark_data_smithi179_77138_object90270:head# 2024-02-09T17:27:04.230 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.230 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f96898fd:::benchmark_data_smithi179_77138_object33774:head# 2024-02-09T17:27:04.230 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.230 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9690fe3:::benchmark_data_smithi179_77138_object70505:head# 2024-02-09T17:27:04.231 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.231 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f96acf8e:::benchmark_data_smithi179_77138_object59696:head# 2024-02-09T17:27:04.231 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.231 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f96d1975:::benchmark_data_smithi179_77138_object61653:head# 2024-02-09T17:27:04.231 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.231 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f96e750f:::benchmark_data_smithi179_77138_object24259:head# 2024-02-09T17:27:04.231 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.231 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f96f3605:::benchmark_data_smithi179_77138_object107089:head# 2024-02-09T17:27:04.231 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.231 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f96f62de:::benchmark_data_smithi179_77138_object54990:head# 2024-02-09T17:27:04.231 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.231 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9711561:::benchmark_data_smithi179_77138_object76849:head# 2024-02-09T17:27:04.231 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.231 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f971693e:::benchmark_data_smithi179_77138_object67156:head# 2024-02-09T17:27:04.231 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.232 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9719ec6:::benchmark_data_smithi179_77138_object47172:head# 2024-02-09T17:27:04.232 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.232 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f971eafc:::benchmark_data_smithi179_77138_object60102:head# 2024-02-09T17:27:04.232 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.232 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f972dce9:::benchmark_data_smithi179_77138_object113853:head# 2024-02-09T17:27:04.232 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.232 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f973dbde:::benchmark_data_smithi179_77138_object97473:head# 2024-02-09T17:27:04.232 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.232 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f974a405:::benchmark_data_smithi179_77138_object106486:head# 2024-02-09T17:27:04.232 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.232 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f974b9fb:::benchmark_data_smithi179_77138_object63810:head# 2024-02-09T17:27:04.232 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.232 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f975adf3:::benchmark_data_smithi179_77138_object106499:head# 2024-02-09T17:27:04.232 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.233 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f977ab59:::benchmark_data_smithi179_77138_object103548:head# 2024-02-09T17:27:04.233 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.233 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f978c2aa:::benchmark_data_smithi179_77138_object26405:head# 2024-02-09T17:27:04.233 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.233 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f979bb8b:::benchmark_data_smithi179_77138_object80788:head# 2024-02-09T17:27:04.233 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.233 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f979f7f7:::benchmark_data_smithi179_77138_object130876:head# 2024-02-09T17:27:04.233 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.233 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f97a24ea:::benchmark_data_smithi179_77138_object57130:head# 2024-02-09T17:27:04.233 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.233 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f97a5713:::benchmark_data_smithi179_77138_object44976:head# 2024-02-09T17:27:04.233 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.233 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f97a885e:::benchmark_data_smithi179_77138_object21703:head# 2024-02-09T17:27:04.233 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.233 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f97a9678:::benchmark_data_smithi179_77138_object54652:head# 2024-02-09T17:27:04.234 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.234 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f97c2994:::benchmark_data_smithi179_77138_object28252:head# 2024-02-09T17:27:04.234 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.234 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f97ca267:::benchmark_data_smithi179_77138_object120511:head# 2024-02-09T17:27:04.234 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.234 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f97cf8f4:::benchmark_data_smithi179_77138_object84651:head# 2024-02-09T17:27:04.234 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.234 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f97df1da:::benchmark_data_smithi179_77138_object97297:head# 2024-02-09T17:27:04.234 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.234 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f97e7f54:::benchmark_data_smithi179_77138_object52107:head# 2024-02-09T17:27:04.234 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.234 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f97ecc74:::benchmark_data_smithi179_77138_object78971:head# 2024-02-09T17:27:04.234 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.234 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f97ef575:::benchmark_data_smithi179_77138_object75130:head# 2024-02-09T17:27:04.235 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.235 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f97f5488:::benchmark_data_smithi179_77138_object89477:head# 2024-02-09T17:27:04.235 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.235 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9804dac:::benchmark_data_smithi179_77138_object23805:head# 2024-02-09T17:27:04.235 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.235 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9807d39:::benchmark_data_smithi179_77138_object37852:head# 2024-02-09T17:27:04.235 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.235 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9811c1e:::benchmark_data_smithi179_77138_object87642:head# 2024-02-09T17:27:04.235 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.235 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9813e9e:::benchmark_data_smithi179_77138_object79096:head# 2024-02-09T17:27:04.235 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.235 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f981717a:::benchmark_data_smithi179_77138_object58167:head# 2024-02-09T17:27:04.235 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.235 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f98201cd:::benchmark_data_smithi179_77138_object56771:head# 2024-02-09T17:27:04.236 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.236 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9823157:::benchmark_data_smithi179_77138_object112540:head# 2024-02-09T17:27:04.236 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.236 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f982fef3:::benchmark_data_smithi179_77138_object109542:head# 2024-02-09T17:27:04.236 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.236 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9833277:::benchmark_data_smithi179_77138_object131465:head# 2024-02-09T17:27:04.236 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.240 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9837c1e:::benchmark_data_smithi179_77138_object97234:head# 2024-02-09T17:27:04.240 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.240 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9844681:::benchmark_data_smithi179_77138_object26114:head# 2024-02-09T17:27:04.240 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.240 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9846dcc:::benchmark_data_smithi179_77138_object125673:head# 2024-02-09T17:27:04.241 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.241 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9847205:::benchmark_data_smithi179_77138_object92885:head# 2024-02-09T17:27:04.241 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.241 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9847585:::benchmark_data_smithi179_77138_object41013:head# 2024-02-09T17:27:04.241 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.241 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f984ab30:::benchmark_data_smithi179_77138_object26766:head# 2024-02-09T17:27:04.241 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.241 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f98521d6:::benchmark_data_smithi179_77138_object86623:head# 2024-02-09T17:27:04.241 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.241 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9852ad6:::benchmark_data_smithi179_77138_object91886:head# 2024-02-09T17:27:04.241 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.241 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9859d57:::benchmark_data_smithi179_77138_object113001:head# 2024-02-09T17:27:04.241 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.242 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f985df66:::benchmark_data_smithi179_77138_object87542:head# 2024-02-09T17:27:04.242 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.242 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f986d30d:::benchmark_data_smithi179_77138_object87305:head# 2024-02-09T17:27:04.242 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.242 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f987bd44:::benchmark_data_smithi179_77138_object116062:head# 2024-02-09T17:27:04.242 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.242 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f987c1b9:::benchmark_data_smithi179_77138_object121099:head# 2024-02-09T17:27:04.242 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.242 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f987cd8e:::benchmark_data_smithi179_77138_object102352:head# 2024-02-09T17:27:04.242 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.242 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f98a709b:::benchmark_data_smithi179_77138_object64155:head# 2024-02-09T17:27:04.242 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.242 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f98a8e74:::benchmark_data_smithi179_77138_object52284:head# 2024-02-09T17:27:04.243 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.243 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f98b8a10:::benchmark_data_smithi179_77138_object61883:head# 2024-02-09T17:27:04.243 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.243 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f98b99a9:::benchmark_data_smithi179_77138_object22795:head# 2024-02-09T17:27:04.243 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.243 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f98c1d8c:::benchmark_data_smithi179_77138_object61220:head# 2024-02-09T17:27:04.243 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.243 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f98e5166:::benchmark_data_smithi179_77138_object31087:head# 2024-02-09T17:27:04.243 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.243 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f98e6f9d:::benchmark_data_smithi179_77138_object123419:head# 2024-02-09T17:27:04.243 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.243 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f98ed341:::benchmark_data_smithi179_77138_object126613:head# 2024-02-09T17:27:04.243 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.243 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f98f6a20:::benchmark_data_smithi179_77138_object88902:head# 2024-02-09T17:27:04.244 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.244 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f990ef9e:::benchmark_data_smithi179_77138_object126181:head# 2024-02-09T17:27:04.244 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.244 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f990fa0d:::benchmark_data_smithi179_77138_object119385:head# 2024-02-09T17:27:04.244 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.244 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9918f64:::benchmark_data_smithi179_77138_object84111:head# 2024-02-09T17:27:04.244 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.244 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9919ab0:::benchmark_data_smithi179_77138_object37435:head# 2024-02-09T17:27:04.244 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.244 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f992ce16:::benchmark_data_smithi179_77138_object91078:head# 2024-02-09T17:27:04.244 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.244 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f993c4bf:::benchmark_data_smithi179_77138_object32478:head# 2024-02-09T17:27:04.244 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.244 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f993fe81:::benchmark_data_smithi179_77138_object24604:head# 2024-02-09T17:27:04.245 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.245 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f994095b:::benchmark_data_smithi179_77138_object39123:head# 2024-02-09T17:27:04.245 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.245 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f994225f:::benchmark_data_smithi179_77138_object80241:head# 2024-02-09T17:27:04.245 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.245 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f994f95f:::benchmark_data_smithi179_77138_object28208:head# 2024-02-09T17:27:04.245 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.245 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f995ce1d:::benchmark_data_smithi179_77138_object118394:head# 2024-02-09T17:27:04.245 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.245 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f997d102:::benchmark_data_smithi179_77138_object36138:head# 2024-02-09T17:27:04.245 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.245 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f997e05c:::benchmark_data_smithi179_77138_object131009:head# 2024-02-09T17:27:04.245 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.245 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f997eb92:::benchmark_data_smithi179_77138_object66951:head# 2024-02-09T17:27:04.245 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.246 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9986771:::benchmark_data_smithi179_77138_object120923:head# 2024-02-09T17:27:04.246 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.246 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f998830a:::benchmark_data_smithi179_77138_object23355:head# 2024-02-09T17:27:04.246 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.246 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f99945d7:::benchmark_data_smithi179_77138_object35062:head# 2024-02-09T17:27:04.246 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.246 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9998a46:::benchmark_data_smithi179_77138_object40842:head# 2024-02-09T17:27:04.246 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.246 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9998b57:::benchmark_data_smithi179_77138_object128345:head# 2024-02-09T17:27:04.246 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.246 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f99b7bb7:::benchmark_data_smithi179_77138_object56359:head# 2024-02-09T17:27:04.246 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.246 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f99bf315:::benchmark_data_smithi179_77138_object25356:head# 2024-02-09T17:27:04.246 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.247 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f99d1cb7:::benchmark_data_smithi179_77138_object106828:head# 2024-02-09T17:27:04.247 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.247 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f99d6d83:::benchmark_data_smithi179_77138_object80249:head# 2024-02-09T17:27:04.247 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.247 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f99d8d70:::benchmark_data_smithi179_77138_object71645:head# 2024-02-09T17:27:04.247 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.254 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f99dc7e6:::benchmark_data_smithi179_77138_object119192:head# 2024-02-09T17:27:04.254 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.254 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f99f3cc2:::benchmark_data_smithi179_77138_object78262:head# 2024-02-09T17:27:04.254 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.254 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f99fde23:::benchmark_data_smithi179_77138_object87010:head# 2024-02-09T17:27:04.254 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.254 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9a0cb42:::benchmark_data_smithi179_77138_object104173:head# 2024-02-09T17:27:04.254 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.254 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9a0de41:::benchmark_data_smithi179_77138_object46343:head# 2024-02-09T17:27:04.254 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.254 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9a112e6:::benchmark_data_smithi179_77138_object73672:head# 2024-02-09T17:27:04.255 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.255 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9a12a4b:::benchmark_data_smithi179_77138_object75056:head# 2024-02-09T17:27:04.255 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.255 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9a13651:::benchmark_data_smithi179_77138_object39370:head# 2024-02-09T17:27:04.255 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.255 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9a230b5:::benchmark_data_smithi179_77138_object34221:head# 2024-02-09T17:27:04.255 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.255 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9a27071:::benchmark_data_smithi179_77138_object39850:head# 2024-02-09T17:27:04.255 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.255 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9a2ca20:::benchmark_data_smithi179_77138_object19930:head# 2024-02-09T17:27:04.255 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.255 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9a3efa9:::benchmark_data_smithi179_77138_object114511:head# 2024-02-09T17:27:04.255 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.255 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9a44950:::benchmark_data_smithi179_77138_object83558:head# 2024-02-09T17:27:04.256 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.256 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9a67022:::benchmark_data_smithi179_77138_object67352:head# 2024-02-09T17:27:04.256 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.256 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9a6ec30:::benchmark_data_smithi179_77138_object130075:head# 2024-02-09T17:27:04.256 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.256 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9a7cd76:::benchmark_data_smithi179_77138_object22023:head# 2024-02-09T17:27:04.256 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.256 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9a7fc63:::benchmark_data_smithi179_77138_object132711:head# 2024-02-09T17:27:04.256 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.256 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9a8206e:::benchmark_data_smithi179_77138_object67771:head# 2024-02-09T17:27:04.256 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.256 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9a82cdf:::benchmark_data_smithi179_77138_object27378:head# 2024-02-09T17:27:04.256 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.256 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9a9292b:::benchmark_data_smithi179_77138_object70382:head# 2024-02-09T17:27:04.257 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.257 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9a95a20:::benchmark_data_smithi179_77138_object119411:head# 2024-02-09T17:27:04.257 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.257 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9a96a92:::benchmark_data_smithi179_77138_object79566:head# 2024-02-09T17:27:04.257 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.257 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9a97a65:::benchmark_data_smithi179_77138_object23564:head# 2024-02-09T17:27:04.257 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.257 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9a9c7a0:::benchmark_data_smithi179_77138_object93718:head# 2024-02-09T17:27:04.257 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.257 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9a9fa28:::benchmark_data_smithi179_77138_object70332:head# 2024-02-09T17:27:04.257 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.257 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9aa8a40:::benchmark_data_smithi179_77138_object26906:head# 2024-02-09T17:27:04.257 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.257 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9ac1097:::benchmark_data_smithi179_77138_object98699:head# 2024-02-09T17:27:04.257 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.258 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9ac7c25:::benchmark_data_smithi179_77138_object97367:head# 2024-02-09T17:27:04.258 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.258 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9ad06bf:::benchmark_data_smithi179_77138_object44403:head# 2024-02-09T17:27:04.258 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.258 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9aec103:::benchmark_data_smithi179_77138_object24815:head# 2024-02-09T17:27:04.258 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.258 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9af0339:::benchmark_data_smithi179_77138_object25092:head# 2024-02-09T17:27:04.258 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.258 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9b075f6:::benchmark_data_smithi179_77138_object92055:head# 2024-02-09T17:27:04.258 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.258 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9b0996b:::benchmark_data_smithi179_77138_object104939:head# 2024-02-09T17:27:04.258 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.258 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9b11fd1:::benchmark_data_smithi179_77138_object22999:head# 2024-02-09T17:27:04.258 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.258 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9b1ead7:::benchmark_data_smithi179_77138_object132693:head# 2024-02-09T17:27:04.258 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.259 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9b23e04:::benchmark_data_smithi179_77138_object73197:head# 2024-02-09T17:27:04.259 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.259 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9b2ff81:::benchmark_data_smithi179_77138_object122810:head# 2024-02-09T17:27:04.259 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.259 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9b37c47:::benchmark_data_smithi179_77138_object72019:head# 2024-02-09T17:27:04.259 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.259 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9b3b121:::benchmark_data_smithi179_77138_object47864:head# 2024-02-09T17:27:04.259 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.259 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9b481ef:::benchmark_data_smithi179_77138_object38916:head# 2024-02-09T17:27:04.259 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.259 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9b4f45d:::benchmark_data_smithi179_77138_object55385:head# 2024-02-09T17:27:04.259 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.259 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9b570e9:::benchmark_data_smithi179_77138_object42668:head# 2024-02-09T17:27:04.259 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.259 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9b65185:::benchmark_data_smithi179_77138_object55539:head# 2024-02-09T17:27:04.260 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.260 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9b6e62a:::benchmark_data_smithi179_77138_object89595:head# 2024-02-09T17:27:04.260 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.260 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9b70991:::benchmark_data_smithi179_77138_object127306:head# 2024-02-09T17:27:04.260 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.260 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9b755a1:::benchmark_data_smithi179_77138_object45909:head# 2024-02-09T17:27:04.260 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.260 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9b78dd3:::benchmark_data_smithi179_77138_object106387:head# 2024-02-09T17:27:04.260 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.268 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9b79c6e:::benchmark_data_smithi179_77138_object52827:head# 2024-02-09T17:27:04.268 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.268 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9b9df1e:::benchmark_data_smithi179_77138_object66676:head# 2024-02-09T17:27:04.268 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.268 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9ba6f45:::benchmark_data_smithi179_77138_object109515:head# 2024-02-09T17:27:04.268 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.269 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9bab8d4:::benchmark_data_smithi179_77138_object41975:head# 2024-02-09T17:27:04.269 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.269 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9bb0708:::benchmark_data_smithi179_77138_object104069:head# 2024-02-09T17:27:04.269 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.269 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9bb9e46:::benchmark_data_smithi179_77138_object125420:head# 2024-02-09T17:27:04.269 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.269 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9bde3a2:::benchmark_data_smithi179_77138_object125872:head# 2024-02-09T17:27:04.269 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.269 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9bee9a1:::benchmark_data_smithi179_77138_object30985:head# 2024-02-09T17:27:04.269 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.269 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9bef44f:::benchmark_data_smithi179_77138_object30721:head# 2024-02-09T17:27:04.269 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.269 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9bf1314:::benchmark_data_smithi179_77138_object117988:head# 2024-02-09T17:27:04.269 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.270 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9bf526d:::benchmark_data_smithi179_77138_object73181:head# 2024-02-09T17:27:04.270 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.270 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9bf5b1a:::benchmark_data_smithi179_77138_object46757:head# 2024-02-09T17:27:04.270 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.270 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9c00f94:::benchmark_data_smithi179_77138_object75081:head# 2024-02-09T17:27:04.270 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.270 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9c18c16:::benchmark_data_smithi179_77138_object96027:head# 2024-02-09T17:27:04.270 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.270 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9c20635:::benchmark_data_smithi179_77138_object70530:head# 2024-02-09T17:27:04.270 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.270 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9c2573d:::benchmark_data_smithi179_77138_object28479:head# 2024-02-09T17:27:04.270 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.270 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9c2d336:::benchmark_data_smithi179_77138_object20112:head# 2024-02-09T17:27:04.270 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.271 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9c2fdee:::benchmark_data_smithi179_77138_object53901:head# 2024-02-09T17:27:04.271 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.271 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9c40cc1:::benchmark_data_smithi179_77138_object27676:head# 2024-02-09T17:27:04.271 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.271 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9c431d4:::benchmark_data_smithi179_77138_object46471:head# 2024-02-09T17:27:04.271 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.271 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9c661b3:::benchmark_data_smithi179_77138_object23286:head# 2024-02-09T17:27:04.271 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.271 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9c69ce0:::benchmark_data_smithi179_77138_object115966:head# 2024-02-09T17:27:04.271 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.271 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9c6ade1:::benchmark_data_smithi179_77138_object54406:head# 2024-02-09T17:27:04.271 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.271 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9c92119:::benchmark_data_smithi179_77138_object23023:head# 2024-02-09T17:27:04.272 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.272 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9c9276a:::benchmark_data_smithi179_77138_object82020:head# 2024-02-09T17:27:04.272 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.272 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9c99c04:::benchmark_data_smithi179_77138_object58603:head# 2024-02-09T17:27:04.272 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.272 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9ca6c83:::benchmark_data_smithi179_77138_object102128:head# 2024-02-09T17:27:04.272 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.272 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9cc873c:::benchmark_data_smithi179_77138_object106141:head# 2024-02-09T17:27:04.272 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.272 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9ccaa02:::benchmark_data_smithi179_77138_object48495:head# 2024-02-09T17:27:04.272 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.272 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9ccf928:::benchmark_data_smithi179_77138_object32027:head# 2024-02-09T17:27:04.272 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.272 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9cd0094:::benchmark_data_smithi179_77138_object132798:head# 2024-02-09T17:27:04.273 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.273 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9cd2d56:::benchmark_data_smithi179_77138_object67488:head# 2024-02-09T17:27:04.273 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.273 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9cda960:::benchmark_data_smithi179_77138_object104794:head# 2024-02-09T17:27:04.273 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.273 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9ce9cd7:::benchmark_data_smithi179_77138_object88668:head# 2024-02-09T17:27:04.273 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.273 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9cfb703:::benchmark_data_smithi179_77138_object85831:head# 2024-02-09T17:27:04.273 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.273 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9d089d2:::benchmark_data_smithi179_77138_object24486:head# 2024-02-09T17:27:04.273 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.273 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9d08ee4:::benchmark_data_smithi179_77138_object34821:head# 2024-02-09T17:27:04.273 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.273 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9d0b37c:::benchmark_data_smithi179_77138_object73522:head# 2024-02-09T17:27:04.273 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.274 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9d0c40f:::benchmark_data_smithi179_77138_object83230:head# 2024-02-09T17:27:04.274 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.274 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9d0ca36:::benchmark_data_smithi179_77138_object94020:head# 2024-02-09T17:27:04.274 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.274 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9d0f221:::benchmark_data_smithi179_77138_object101912:head# 2024-02-09T17:27:04.274 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.274 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9d29db0:::benchmark_data_smithi179_77138_object79621:head# 2024-02-09T17:27:04.274 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.274 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9d3e3ce:::benchmark_data_smithi179_77138_object97254:head# 2024-02-09T17:27:04.274 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.274 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9d5f9d6:::benchmark_data_smithi179_77138_object47921:head# 2024-02-09T17:27:04.274 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.274 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9d604c9:::benchmark_data_smithi179_77138_object88847:head# 2024-02-09T17:27:04.274 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.275 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9d69479:::benchmark_data_smithi179_77138_object100640:head# 2024-02-09T17:27:04.275 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.275 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9d71dd3:::benchmark_data_smithi179_77138_object47978:head# 2024-02-09T17:27:04.275 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.275 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9d72d77:::benchmark_data_smithi179_77138_object29983:head# 2024-02-09T17:27:04.275 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.275 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9d7dd4c:::benchmark_data_smithi179_77138_object81257:head# 2024-02-09T17:27:04.275 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.275 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9d7e8de:::benchmark_data_smithi179_77138_object49771:head# 2024-02-09T17:27:04.275 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.275 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9d8611a:::benchmark_data_smithi179_77138_object35544:head# 2024-02-09T17:27:04.275 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.275 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9d95d52:::benchmark_data_smithi179_77138_object84763:head# 2024-02-09T17:27:04.275 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.276 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9d9678f:::benchmark_data_smithi179_77138_object127046:head# 2024-02-09T17:27:04.276 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.276 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9d9b20e:::benchmark_data_smithi179_77138_object31134:head# 2024-02-09T17:27:04.276 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.276 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9da0646:::benchmark_data_smithi179_77138_object22823:head# 2024-02-09T17:27:04.276 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.276 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9db1638:::benchmark_data_smithi179_77138_object39086:head# 2024-02-09T17:27:04.276 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.276 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9dc04e6:::benchmark_data_smithi179_77138_object109880:head# 2024-02-09T17:27:04.276 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.276 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9dc0fa8:::benchmark_data_smithi179_77138_object81010:head# 2024-02-09T17:27:04.276 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.276 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9dc47c9:::benchmark_data_smithi179_77138_object95378:head# 2024-02-09T17:27:04.276 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.277 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9dc97d3:::benchmark_data_smithi179_77138_object72661:head# 2024-02-09T17:27:04.277 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.277 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9de8a14:::benchmark_data_smithi179_77138_object119226:head# 2024-02-09T17:27:04.277 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.277 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9df170b:::benchmark_data_smithi179_77138_object97541:head# 2024-02-09T17:27:04.277 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.278 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9e076a0:::benchmark_data_smithi179_77138_object121045:head# 2024-02-09T17:27:04.278 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.278 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9e0a14c:::benchmark_data_smithi179_77138_object32673:head# 2024-02-09T17:27:04.278 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.278 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9e1349a:::benchmark_data_smithi179_77138_object79634:head# 2024-02-09T17:27:04.278 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.278 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9e15195:::benchmark_data_smithi179_77138_object94346:head# 2024-02-09T17:27:04.278 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.278 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9e17826:::benchmark_data_smithi179_77138_object108774:head# 2024-02-09T17:27:04.278 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.278 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9e1a94c:::benchmark_data_smithi179_77138_object112751:head# 2024-02-09T17:27:04.278 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.278 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9e1cd6b:::benchmark_data_smithi179_77138_object48882:head# 2024-02-09T17:27:04.278 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.279 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9e2194b:::benchmark_data_smithi179_77138_object53689:head# 2024-02-09T17:27:04.279 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.279 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9e23819:::benchmark_data_smithi179_77138_object59383:head# 2024-02-09T17:27:04.279 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.279 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9e2aac2:::benchmark_data_smithi179_77138_object23935:head# 2024-02-09T17:27:04.279 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.280 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9e2fdc1:::benchmark_data_smithi179_77138_object20811:head# 2024-02-09T17:27:04.280 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.280 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9e3eea9:::benchmark_data_smithi179_77138_object94742:head# 2024-02-09T17:27:04.280 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.280 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9e4ec0a:::benchmark_data_smithi179_77138_object87107:head# 2024-02-09T17:27:04.280 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.280 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9e5899d:::benchmark_data_smithi179_77138_object87955:head# 2024-02-09T17:27:04.280 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.280 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9e61bb1:::benchmark_data_smithi179_77138_object104529:head# 2024-02-09T17:27:04.280 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.280 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9e63859:::benchmark_data_smithi179_77138_object79702:head# 2024-02-09T17:27:04.280 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.280 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9e6b8cc:::benchmark_data_smithi179_77138_object84691:head# 2024-02-09T17:27:04.280 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.280 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9e7931d:::benchmark_data_smithi179_77138_object103758:head# 2024-02-09T17:27:04.281 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.281 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9e88be4:::benchmark_data_smithi179_77138_object64604:head# 2024-02-09T17:27:04.281 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.281 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9e97cac:::benchmark_data_smithi179_77138_object40444:head# 2024-02-09T17:27:04.281 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.281 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9eaea53:::benchmark_data_smithi179_77138_object129203:head# 2024-02-09T17:27:04.281 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.281 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9eb0804:::benchmark_data_smithi179_77138_object96613:head# 2024-02-09T17:27:04.281 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.283 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9eb6556:::benchmark_data_smithi179_77138_object88853:head# 2024-02-09T17:27:04.283 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.283 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9eccdc5:::benchmark_data_smithi179_77138_object85079:head# 2024-02-09T17:27:04.284 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.284 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9ed10ca:::benchmark_data_smithi179_77138_object117433:head# 2024-02-09T17:27:04.284 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.284 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9edb574:::benchmark_data_smithi179_77138_object71162:head# 2024-02-09T17:27:04.284 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.284 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9edffdf:::benchmark_data_smithi179_77138_object126448:head# 2024-02-09T17:27:04.284 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.284 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9ee3d68:::benchmark_data_smithi179_77138_object60929:head# 2024-02-09T17:27:04.284 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.284 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9ee98ef:::benchmark_data_smithi179_77138_object129113:head# 2024-02-09T17:27:04.284 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.284 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9ef8b5f:::benchmark_data_smithi179_77138_object129260:head# 2024-02-09T17:27:04.284 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.284 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9f07bb6:::benchmark_data_smithi179_77138_object29448:head# 2024-02-09T17:27:04.285 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.285 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9f25b60:::benchmark_data_smithi179_77138_object76974:head# 2024-02-09T17:27:04.285 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.285 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9f3aab4:::benchmark_data_smithi179_77138_object110892:head# 2024-02-09T17:27:04.285 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.285 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9f429e3:::benchmark_data_smithi179_77138_object19641:head# 2024-02-09T17:27:04.285 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.285 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9f5a4f1:::benchmark_data_smithi179_77138_object107666:head# 2024-02-09T17:27:04.285 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.286 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9f604f6:::benchmark_data_smithi179_77138_object130084:head# 2024-02-09T17:27:04.286 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.287 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9f6ec9a:::benchmark_data_smithi179_77138_object65992:head# 2024-02-09T17:27:04.287 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.287 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9f890bf:::benchmark_data_smithi179_77138_object47006:head# 2024-02-09T17:27:04.287 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.287 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9f8a002:::benchmark_data_smithi179_77138_object115320:head# 2024-02-09T17:27:04.287 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.287 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9f8acce:::benchmark_data_smithi179_77138_object52760:head# 2024-02-09T17:27:04.287 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.288 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9fa2884:::benchmark_data_smithi179_77138_object96935:head# 2024-02-09T17:27:04.289 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.289 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9fa3d85:::benchmark_data_smithi179_77138_object54310:head# 2024-02-09T17:27:04.289 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.289 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9fc2afb:::benchmark_data_smithi179_77138_object70226:head# 2024-02-09T17:27:04.289 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.289 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9fc3792:::benchmark_data_smithi179_77138_object25107:head# 2024-02-09T17:27:04.289 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.289 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9fccbff:::benchmark_data_smithi179_77138_object73199:head# 2024-02-09T17:27:04.289 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.289 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9fcd7da:::benchmark_data_smithi179_77138_object21480:head# 2024-02-09T17:27:04.289 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.290 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9fd9688:::benchmark_data_smithi179_77138_object65859:head# 2024-02-09T17:27:04.290 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.290 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9fdc379:::benchmark_data_smithi179_77138_object36711:head# 2024-02-09T17:27:04.290 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.290 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:f9fe8457:::benchmark_data_smithi179_77138_object114356:head# 2024-02-09T17:27:04.290 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.291 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa005e1b:::benchmark_data_smithi179_77138_object43305:head# 2024-02-09T17:27:04.291 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.291 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa00aada:::benchmark_data_smithi179_77138_object78301:head# 2024-02-09T17:27:04.291 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.293 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa00bab8:::benchmark_data_smithi179_77138_object33022:head# 2024-02-09T17:27:04.293 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.293 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa00c937:::benchmark_data_smithi179_77138_object126073:head# 2024-02-09T17:27:04.293 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.293 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa029c0f:::benchmark_data_smithi179_77138_object107355:head# 2024-02-09T17:27:04.293 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.293 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa03cfd3:::benchmark_data_smithi179_77138_object21833:head# 2024-02-09T17:27:04.293 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.293 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa03dfcc:::benchmark_data_smithi179_77138_object45236:head# 2024-02-09T17:27:04.293 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.293 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa0459ff:::benchmark_data_smithi179_77138_object24954:head# 2024-02-09T17:27:04.294 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.294 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa046286:::benchmark_data_smithi179_77138_object28244:head# 2024-02-09T17:27:04.294 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.294 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa04a38d:::benchmark_data_smithi179_77138_object68874:head# 2024-02-09T17:27:04.294 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.294 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa04f7d7:::benchmark_data_smithi179_77138_object78336:head# 2024-02-09T17:27:04.294 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.294 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa06f4ab:::benchmark_data_smithi179_77138_object91728:head# 2024-02-09T17:27:04.294 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.294 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa08047c:::benchmark_data_smithi179_77138_object30029:head# 2024-02-09T17:27:04.294 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.295 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa081541:::benchmark_data_smithi179_77138_object76175:head# 2024-02-09T17:27:04.295 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.295 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa08e3d2:::benchmark_data_smithi179_77138_object94221:head# 2024-02-09T17:27:04.295 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.295 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa08eb40:::benchmark_data_smithi179_77138_object49399:head# 2024-02-09T17:27:04.295 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.295 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa090534:::benchmark_data_smithi179_77138_object128974:head# 2024-02-09T17:27:04.296 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.298 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa09ca01:::benchmark_data_smithi179_77138_object45385:head# 2024-02-09T17:27:04.298 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.298 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa0a1365:::benchmark_data_smithi179_77138_object98174:head# 2024-02-09T17:27:04.299 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.299 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa0a17b6:::benchmark_data_smithi179_77138_object56960:head# 2024-02-09T17:27:04.299 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.299 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa0a9058:::benchmark_data_smithi179_77138_object46177:head# 2024-02-09T17:27:04.299 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.301 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa0a9e3f:::benchmark_data_smithi179_77138_object41148:head# 2024-02-09T17:27:04.301 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.301 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa0bb5bb:::benchmark_data_smithi179_77138_object100176:head# 2024-02-09T17:27:04.301 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.302 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa0bd095:::benchmark_data_smithi179_77138_object66484:head# 2024-02-09T17:27:04.302 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.302 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa0c86c5:::benchmark_data_smithi179_77138_object39357:head# 2024-02-09T17:27:04.302 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.302 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa10a25c:::benchmark_data_smithi179_77138_object39964:head# 2024-02-09T17:27:04.302 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.302 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa10e0da:::benchmark_data_smithi179_77138_object59893:head# 2024-02-09T17:27:04.302 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.302 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa119309:::benchmark_data_smithi179_77138_object88008:head# 2024-02-09T17:27:04.302 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.302 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa125ed4:::benchmark_data_smithi179_77138_object51226:head# 2024-02-09T17:27:04.302 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.302 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa131ed9:::benchmark_data_smithi179_77138_object30545:head# 2024-02-09T17:27:04.302 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.303 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa144d3a:::benchmark_data_smithi179_77138_object72037:head# 2024-02-09T17:27:04.303 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.303 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa147cb8:::benchmark_data_smithi179_77138_object63667:head# 2024-02-09T17:27:04.303 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.303 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa154d63:::benchmark_data_smithi179_77138_object72471:head# 2024-02-09T17:27:04.303 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.306 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa16705b:::benchmark_data_smithi179_77138_object25261:head# 2024-02-09T17:27:04.306 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.307 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa16bd33:::benchmark_data_smithi179_77138_object57794:head# 2024-02-09T17:27:04.307 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.307 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa18a509:::benchmark_data_smithi179_77138_object98574:head# 2024-02-09T17:27:04.307 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.307 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa18e3a4:::benchmark_data_smithi179_77138_object119836:head# 2024-02-09T17:27:04.307 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.307 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa196519:::benchmark_data_smithi179_77138_object64261:head# 2024-02-09T17:27:04.307 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.307 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa198247:::benchmark_data_smithi179_77138_object102146:head# 2024-02-09T17:27:04.307 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.307 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa1aa882:::benchmark_data_smithi179_77138_object51721:head# 2024-02-09T17:27:04.307 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.307 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa1b7d5f:::benchmark_data_smithi179_77138_object131598:head# 2024-02-09T17:27:04.307 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.308 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa1c0853:::benchmark_data_smithi179_77138_object122325:head# 2024-02-09T17:27:04.308 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.308 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa1cf469:::benchmark_data_smithi179_77138_object77651:head# 2024-02-09T17:27:04.308 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.308 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa1d061a:::benchmark_data_smithi179_77138_object73557:head# 2024-02-09T17:27:04.308 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.308 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa1e7965:::benchmark_data_smithi179_77138_object96627:head# 2024-02-09T17:27:04.308 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.308 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa1ef08b:::benchmark_data_smithi179_77138_object114844:head# 2024-02-09T17:27:04.308 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.308 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa1f30c3:::benchmark_data_smithi179_77138_object27652:head# 2024-02-09T17:27:04.308 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.308 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa1f71c7:::benchmark_data_smithi179_77138_object123429:head# 2024-02-09T17:27:04.308 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.309 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa1f81af:::benchmark_data_smithi179_77138_object41756:head# 2024-02-09T17:27:04.309 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.309 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa1fb9f9:::benchmark_data_smithi179_77138_object47319:head# 2024-02-09T17:27:04.309 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.309 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa206fc9:::benchmark_data_smithi179_77138_object79883:head# 2024-02-09T17:27:04.309 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.309 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa20849d:::benchmark_data_smithi179_77138_object131919:head# 2024-02-09T17:27:04.309 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.309 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa20ce12:::benchmark_data_smithi179_77138_object75907:head# 2024-02-09T17:27:04.309 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.309 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa213ee9:::benchmark_data_smithi179_77138_object27088:head# 2024-02-09T17:27:04.309 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.309 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa222314:::benchmark_data_smithi179_77138_object108546:head# 2024-02-09T17:27:04.309 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.312 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa22557f:::benchmark_data_smithi179_77138_object129900:head# 2024-02-09T17:27:04.312 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.312 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa240ef6:::benchmark_data_smithi179_77138_object121319:head# 2024-02-09T17:27:04.312 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.313 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa24ce24:::benchmark_data_smithi179_77138_object94071:head# 2024-02-09T17:27:04.313 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.313 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa25e606:::benchmark_data_smithi179_77138_object55267:head# 2024-02-09T17:27:04.313 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.313 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa25f2ee:::benchmark_data_smithi179_77138_object23806:head# 2024-02-09T17:27:04.313 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.313 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa25fa9f:::benchmark_data_smithi179_77138_object32924:head# 2024-02-09T17:27:04.313 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.313 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa26402c:::benchmark_data_smithi179_77138_object104602:head# 2024-02-09T17:27:04.313 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.313 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa26d6c0:::benchmark_data_smithi179_77138_object44498:head# 2024-02-09T17:27:04.313 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.313 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa279be5:::benchmark_data_smithi179_77138_object109132:head# 2024-02-09T17:27:04.313 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.314 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa284324:::benchmark_data_smithi179_77138_object120546:head# 2024-02-09T17:27:04.314 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.314 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa28eb31:::benchmark_data_smithi179_77138_object79909:head# 2024-02-09T17:27:04.314 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.314 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa294d54:::benchmark_data_smithi179_77138_object44787:head# 2024-02-09T17:27:04.314 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.314 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa296952:::benchmark_data_smithi179_77138_object77804:head# 2024-02-09T17:27:04.314 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.314 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa29a5d9:::benchmark_data_smithi179_77138_object29508:head# 2024-02-09T17:27:04.314 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.314 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa29bafb:::benchmark_data_smithi179_77138_object101354:head# 2024-02-09T17:27:04.314 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.314 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa29ce67:::benchmark_data_smithi179_77138_object37507:head# 2024-02-09T17:27:04.314 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.314 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa29fc74:::benchmark_data_smithi179_77138_object62306:head# 2024-02-09T17:27:04.315 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.315 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa2a13b9:::benchmark_data_smithi179_77138_object68840:head# 2024-02-09T17:27:04.315 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.315 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa2a1e25:::benchmark_data_smithi179_77138_object132003:head# 2024-02-09T17:27:04.315 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.317 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa2a6765:::benchmark_data_smithi179_77138_object92295:head# 2024-02-09T17:27:04.317 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.317 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa2aadf7:::benchmark_data_smithi179_77138_object77307:head# 2024-02-09T17:27:04.317 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.317 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa2acdcd:::benchmark_data_smithi179_77138_object79158:head# 2024-02-09T17:27:04.317 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.317 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa2b487d:::benchmark_data_smithi179_77138_object47022:head# 2024-02-09T17:27:04.317 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.317 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa2bb237:::benchmark_data_smithi179_77138_object75826:head# 2024-02-09T17:27:04.318 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.318 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa2bb427:::benchmark_data_smithi179_77138_object121058:head# 2024-02-09T17:27:04.318 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.318 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa2bc8e7:::benchmark_data_smithi179_77138_object64088:head# 2024-02-09T17:27:04.318 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.318 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa2d4520:::benchmark_data_smithi179_77138_object80367:head# 2024-02-09T17:27:04.318 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.318 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa2e6b93:::benchmark_data_smithi179_77138_object67129:head# 2024-02-09T17:27:04.318 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.318 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa2ec11d:::benchmark_data_smithi179_77138_object83642:head# 2024-02-09T17:27:04.318 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.318 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa306835:::benchmark_data_smithi179_77138_object24959:head# 2024-02-09T17:27:04.318 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.319 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa30796a:::benchmark_data_smithi179_77138_object76210:head# 2024-02-09T17:27:04.319 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.319 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa30c1d8:::benchmark_data_smithi179_77138_object27559:head# 2024-02-09T17:27:04.319 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.319 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa316762:::benchmark_data_smithi179_77138_object54814:head# 2024-02-09T17:27:04.319 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.319 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa31715c:::benchmark_data_smithi179_77138_object20100:head# 2024-02-09T17:27:04.319 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.319 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa31829f:::benchmark_data_smithi179_77138_object47851:head# 2024-02-09T17:27:04.319 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.319 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa31bb60:::benchmark_data_smithi179_77138_object123218:head# 2024-02-09T17:27:04.319 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.319 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa3284a8:::benchmark_data_smithi179_77138_object109852:head# 2024-02-09T17:27:04.319 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.320 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa32a132:::benchmark_data_smithi179_77138_object127674:head# 2024-02-09T17:27:04.320 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.320 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa33136c:::benchmark_data_smithi179_77138_object108589:head# 2024-02-09T17:27:04.320 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.320 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa333adb:::benchmark_data_smithi179_77138_object69025:head# 2024-02-09T17:27:04.320 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.320 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa336640:::benchmark_data_smithi179_77138_object117810:head# 2024-02-09T17:27:04.320 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.326 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa33e10b:::benchmark_data_smithi179_77138_object96779:head# 2024-02-09T17:27:04.326 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.326 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa348679:::benchmark_data_smithi179_77138_object58967:head# 2024-02-09T17:27:04.326 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.326 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa353e55:::benchmark_data_smithi179_77138_object27674:head# 2024-02-09T17:27:04.326 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.327 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa35669e:::benchmark_data_smithi179_77138_object19413:head# 2024-02-09T17:27:04.327 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.327 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa364520:::benchmark_data_smithi179_77138_object87483:head# 2024-02-09T17:27:04.327 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.327 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa36f6a1:::benchmark_data_smithi179_77138_object127598:head# 2024-02-09T17:27:04.327 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.327 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa3766ae:::benchmark_data_smithi179_77138_object64685:head# 2024-02-09T17:27:04.327 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.327 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa380d42:::benchmark_data_smithi179_77138_object72709:head# 2024-02-09T17:27:04.327 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.327 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa3821c7:::benchmark_data_smithi179_77138_object123629:head# 2024-02-09T17:27:04.327 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.327 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa383c9e:::benchmark_data_smithi179_77138_object124185:head# 2024-02-09T17:27:04.327 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.328 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa38ce06:::benchmark_data_smithi179_77138_object70751:head# 2024-02-09T17:27:04.328 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.328 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa3afd99:::benchmark_data_smithi179_77138_object100801:head# 2024-02-09T17:27:04.328 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.328 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa3b1649:::benchmark_data_smithi179_77138_object104705:head# 2024-02-09T17:27:04.328 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.328 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa3b8b83:::benchmark_data_smithi179_77138_object92917:head# 2024-02-09T17:27:04.328 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.328 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa3ca332:::benchmark_data_smithi179_77138_object23810:head# 2024-02-09T17:27:04.328 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.328 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa3ce975:::benchmark_data_smithi179_77138_object35843:head# 2024-02-09T17:27:04.328 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.328 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa3d14ab:::benchmark_data_smithi179_77138_object47684:head# 2024-02-09T17:27:04.328 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.329 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa3d16e7:::benchmark_data_smithi179_77138_object66240:head# 2024-02-09T17:27:04.329 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.329 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa3d5bb4:::benchmark_data_smithi179_77138_object57495:head# 2024-02-09T17:27:04.329 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.329 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa3e2dfe:::benchmark_data_smithi179_77138_object19809:head# 2024-02-09T17:27:04.329 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.329 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa3e8845:::benchmark_data_smithi179_77138_object72076:head# 2024-02-09T17:27:04.329 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.329 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa3f33be:::benchmark_data_smithi179_77138_object34186:head# 2024-02-09T17:27:04.329 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.329 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa403f5d:::benchmark_data_smithi179_77138_object96817:head# 2024-02-09T17:27:04.329 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.329 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa4301c4:::benchmark_data_smithi179_77138_object66704:head# 2024-02-09T17:27:04.329 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.329 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa4320fc:::benchmark_data_smithi179_77138_object108769:head# 2024-02-09T17:27:04.330 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.330 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa43286d:::benchmark_data_smithi179_77138_object108989:head# 2024-02-09T17:27:04.330 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.330 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa436b3b:::benchmark_data_smithi179_77138_object75119:head# 2024-02-09T17:27:04.330 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.330 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa43b034:::benchmark_data_smithi179_77138_object129636:head# 2024-02-09T17:27:04.330 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.330 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa449279:::benchmark_data_smithi179_77138_object31979:head# 2024-02-09T17:27:04.330 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.330 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa46a279:::benchmark_data_smithi179_77138_object117101:head# 2024-02-09T17:27:04.330 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.330 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa46e330:::benchmark_data_smithi179_77138_object21309:head# 2024-02-09T17:27:04.330 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.330 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa498d13:::benchmark_data_smithi179_77138_object57502:head# 2024-02-09T17:27:04.330 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.330 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa49bb1d:::benchmark_data_smithi179_77138_object106934:head# 2024-02-09T17:27:04.330 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.331 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa49e55d:::benchmark_data_smithi179_77138_object81518:head# 2024-02-09T17:27:04.331 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.331 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa4a3afb:::benchmark_data_smithi179_77138_object128163:head# 2024-02-09T17:27:04.331 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.331 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa4a7895:::benchmark_data_smithi179_77138_object99155:head# 2024-02-09T17:27:04.331 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.331 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa4a7ab6:::benchmark_data_smithi179_77138_object97108:head# 2024-02-09T17:27:04.331 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.336 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa4ab1ab:::benchmark_data_smithi179_77138_object87588:head# 2024-02-09T17:27:04.336 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.336 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa4b1331:::benchmark_data_smithi179_77138_object92496:head# 2024-02-09T17:27:04.337 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.337 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa4bbe54:::benchmark_data_smithi179_77138_object131249:head# 2024-02-09T17:27:04.337 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.337 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa4bbf35:::benchmark_data_smithi179_77138_object57797:head# 2024-02-09T17:27:04.337 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.337 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa4dbf49:::benchmark_data_smithi179_77138_object64481:head# 2024-02-09T17:27:04.337 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.337 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa4e2618:::benchmark_data_smithi179_77138_object38798:head# 2024-02-09T17:27:04.337 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.337 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa4e9d52:::benchmark_data_smithi179_77138_object85246:head# 2024-02-09T17:27:04.337 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.337 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa4fce31:::benchmark_data_smithi179_77138_object98032:head# 2024-02-09T17:27:04.337 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.337 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa5075a5:::benchmark_data_smithi179_77138_object129779:head# 2024-02-09T17:27:04.338 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.338 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa52bbee:::benchmark_data_smithi179_77138_object29367:head# 2024-02-09T17:27:04.338 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.338 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa53532f:::benchmark_data_smithi179_77138_object118753:head# 2024-02-09T17:27:04.338 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.338 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa538eab:::benchmark_data_smithi179_77138_object107379:head# 2024-02-09T17:27:04.338 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.338 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa5589ba:::benchmark_data_smithi179_77138_object88700:head# 2024-02-09T17:27:04.338 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.338 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa56265b:::benchmark_data_smithi179_77138_object95210:head# 2024-02-09T17:27:04.338 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.338 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa565d20:::benchmark_data_smithi179_77138_object131091:head# 2024-02-09T17:27:04.338 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.338 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa571ffd:::benchmark_data_smithi179_77138_object21748:head# 2024-02-09T17:27:04.338 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.339 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa573ba4:::benchmark_data_smithi179_77138_object69972:head# 2024-02-09T17:27:04.339 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.339 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa57af92:::benchmark_data_smithi179_77138_object110180:head# 2024-02-09T17:27:04.339 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.339 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa57f412:::benchmark_data_smithi179_77138_object43554:head# 2024-02-09T17:27:04.339 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.339 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa58200e:::benchmark_data_smithi179_77138_object52959:head# 2024-02-09T17:27:04.339 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.339 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa5866df:::benchmark_data_smithi179_77138_object81200:head# 2024-02-09T17:27:04.339 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.339 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa59e4e4:::benchmark_data_smithi179_77138_object68447:head# 2024-02-09T17:27:04.339 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.339 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa5a0fb6:::benchmark_data_smithi179_77138_object61170:head# 2024-02-09T17:27:04.339 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.340 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa5a42e5:::benchmark_data_smithi179_77138_object69389:head# 2024-02-09T17:27:04.340 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.340 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa5ada02:::benchmark_data_smithi179_77138_object114751:head# 2024-02-09T17:27:04.340 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.340 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa5b4172:::benchmark_data_smithi179_77138_object49277:head# 2024-02-09T17:27:04.340 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.340 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa5b5336:::benchmark_data_smithi179_77138_object102010:head# 2024-02-09T17:27:04.340 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.340 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa5bbcd3:::benchmark_data_smithi179_77138_object59838:head# 2024-02-09T17:27:04.340 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.340 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa5be452:::benchmark_data_smithi179_77138_object46957:head# 2024-02-09T17:27:04.340 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.340 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa5c18be:::benchmark_data_smithi179_77138_object48128:head# 2024-02-09T17:27:04.341 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.341 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa5c5084:::benchmark_data_smithi179_77138_object95458:head# 2024-02-09T17:27:04.341 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.341 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa5cd479:::benchmark_data_smithi179_77138_object54246:head# 2024-02-09T17:27:04.341 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.341 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa5d360f:::benchmark_data_smithi179_77138_object86691:head# 2024-02-09T17:27:04.341 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.341 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa5e4094:::benchmark_data_smithi179_77138_object38479:head# 2024-02-09T17:27:04.341 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.341 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa5ed617:::benchmark_data_smithi179_77138_object31924:head# 2024-02-09T17:27:04.341 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.341 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa5ee11c:::benchmark_data_smithi179_77138_object107847:head# 2024-02-09T17:27:04.341 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.341 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa5f7fbc:::benchmark_data_smithi179_77138_object67665:head# 2024-02-09T17:27:04.342 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.342 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa60155c:::benchmark_data_smithi179_77138_object42572:head# 2024-02-09T17:27:04.342 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.342 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa6034a0:::benchmark_data_smithi179_77138_object97526:head# 2024-02-09T17:27:04.342 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.342 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa619409:::benchmark_data_smithi179_77138_object50243:head# 2024-02-09T17:27:04.342 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.342 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa6257d2:::benchmark_data_smithi179_77138_object113848:head# 2024-02-09T17:27:04.342 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.342 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa628086:::benchmark_data_smithi179_77138_object122013:head# 2024-02-09T17:27:04.342 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.342 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa62b194:::benchmark_data_smithi179_77138_object56119:head# 2024-02-09T17:27:04.342 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.351 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-09T17:27:04.353 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa6377c2:::benchmark_data_smithi179_77138_object115034:head# 2024-02-09T17:27:04.353 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.353 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa63925f:::benchmark_data_smithi179_77138_object56177:head# 2024-02-09T17:27:04.353 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.353 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa647c0e:::benchmark_data_smithi179_77138_object44473:head# 2024-02-09T17:27:04.353 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.353 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa659d6d:::benchmark_data_smithi179_77138_object96082:head# 2024-02-09T17:27:04.353 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.354 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa65e530:::benchmark_data_smithi179_77138_object110328:head# 2024-02-09T17:27:04.354 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.354 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa661861:::benchmark_data_smithi179_77138_object131536:head# 2024-02-09T17:27:04.354 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.354 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa6722b9:::benchmark_data_smithi179_77138_object56350:head# 2024-02-09T17:27:04.354 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.354 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa6756fb:::benchmark_data_smithi179_77138_object77618:head# 2024-02-09T17:27:04.354 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.354 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa677175:::benchmark_data_smithi179_77138_object62880:head# 2024-02-09T17:27:04.354 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.354 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa678a08:::benchmark_data_smithi179_77138_object33171:head# 2024-02-09T17:27:04.354 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.354 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa682507:::benchmark_data_smithi179_77138_object40284:head# 2024-02-09T17:27:04.354 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.355 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa68344c:::benchmark_data_smithi179_77138_object96577:head# 2024-02-09T17:27:04.355 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.355 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa686108:::benchmark_data_smithi179_77138_object40958:head# 2024-02-09T17:27:04.355 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.355 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa68a213:::benchmark_data_smithi179_77138_object54753:head# 2024-02-09T17:27:04.355 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.355 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa69515e:::benchmark_data_smithi179_77138_object70476:head# 2024-02-09T17:27:04.355 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.355 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa6a6fb6:::benchmark_data_smithi179_77138_object80392:head# 2024-02-09T17:27:04.355 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.355 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa6a7b32:::benchmark_data_smithi179_77138_object105276:head# 2024-02-09T17:27:04.356 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.356 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa6b0858:::benchmark_data_smithi179_77138_object23653:head# 2024-02-09T17:27:04.356 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.356 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa6b66aa:::benchmark_data_smithi179_77138_object28161:head# 2024-02-09T17:27:04.356 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.356 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa6b81df:::benchmark_data_smithi179_77138_object43692:head# 2024-02-09T17:27:04.356 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.356 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa6b8bb6:::benchmark_data_smithi179_77138_object71800:head# 2024-02-09T17:27:04.356 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.356 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa6d0920:::benchmark_data_smithi179_77138_object76442:head# 2024-02-09T17:27:04.356 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.356 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa6d681d:::benchmark_data_smithi179_77138_object122073:head# 2024-02-09T17:27:04.357 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.357 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa6e7cb9:::benchmark_data_smithi179_77138_object38293:head# 2024-02-09T17:27:04.357 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.357 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa6ece2d:::benchmark_data_smithi179_77138_object54745:head# 2024-02-09T17:27:04.357 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.357 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa6f0fb3:::benchmark_data_smithi179_77138_object95999:head# 2024-02-09T17:27:04.357 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.357 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa6fa810:::benchmark_data_smithi179_77138_object72732:head# 2024-02-09T17:27:04.357 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.357 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa70808d:::benchmark_data_smithi179_77138_object55700:head# 2024-02-09T17:27:04.357 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.357 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa721473:::benchmark_data_smithi179_77138_object66521:head# 2024-02-09T17:27:04.357 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.357 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa73dfcf:::benchmark_data_smithi179_77138_object30664:head# 2024-02-09T17:27:04.358 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.358 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa7470b6:::benchmark_data_smithi179_77138_object105769:head# 2024-02-09T17:27:04.358 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.358 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa74ffb5:::benchmark_data_smithi179_77138_object113301:head# 2024-02-09T17:27:04.358 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.358 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa75fa1e:::benchmark_data_smithi179_77138_object91550:head# 2024-02-09T17:27:04.358 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.358 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa7651d0:::benchmark_data_smithi179_77138_object76329:head# 2024-02-09T17:27:04.358 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.358 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa779c8f:::benchmark_data_smithi179_77138_object66478:head# 2024-02-09T17:27:04.358 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.358 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa7837e6:::benchmark_data_smithi179_77138_object129586:head# 2024-02-09T17:27:04.358 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.359 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa7885fb:::benchmark_data_smithi179_77138_object82525:head# 2024-02-09T17:27:04.359 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.359 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa7a2afe:::benchmark_data_smithi179_77138_object126752:head# 2024-02-09T17:27:04.359 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.359 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa7a674c:::benchmark_data_smithi179_77138_object40371:head# 2024-02-09T17:27:04.359 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.359 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa7ab99a:::benchmark_data_smithi179_77138_object80936:head# 2024-02-09T17:27:04.359 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.359 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa7ad164:::benchmark_data_smithi179_77138_object55137:head# 2024-02-09T17:27:04.359 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.359 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa7bc028:::benchmark_data_smithi179_77138_object28831:head# 2024-02-09T17:27:04.359 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.359 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa7caa06:::benchmark_data_smithi179_77138_object125217:head# 2024-02-09T17:27:04.359 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.359 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa7debc1:::benchmark_data_smithi179_77138_object81505:head# 2024-02-09T17:27:04.360 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.360 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa7e119f:::benchmark_data_smithi179_77138_object121001:head# 2024-02-09T17:27:04.360 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.360 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa7e62c3:::benchmark_data_smithi179_77138_object78496:head# 2024-02-09T17:27:04.360 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.360 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa7f64c7:::benchmark_data_smithi179_77138_object66566:head# 2024-02-09T17:27:04.360 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.360 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa7fc7ec:::benchmark_data_smithi179_77138_object47660:head# 2024-02-09T17:27:04.360 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.360 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa7fd4ee:::benchmark_data_smithi179_77138_object35192:head# 2024-02-09T17:27:04.360 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.360 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa8053b4:::benchmark_data_smithi179_77138_object72606:head# 2024-02-09T17:27:04.360 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.361 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa80c9b7:::benchmark_data_smithi179_77138_object115429:head# 2024-02-09T17:27:04.361 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.361 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa80e663:::benchmark_data_smithi179_77138_object76258:head# 2024-02-09T17:27:04.361 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.361 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa826287:::benchmark_data_smithi179_77138_object59189:head# 2024-02-09T17:27:04.361 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.361 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa82c62c:::benchmark_data_smithi179_77138_object106810:head# 2024-02-09T17:27:04.361 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.361 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa82f955:::benchmark_data_smithi179_77138_object117418:head# 2024-02-09T17:27:04.361 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.361 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa84cc2c:::benchmark_data_smithi179_77138_object93175:head# 2024-02-09T17:27:04.361 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.361 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa85f3c1:::benchmark_data_smithi179_77138_object110531:head# 2024-02-09T17:27:04.362 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.362 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa88252f:::benchmark_data_smithi179_77138_object119458:head# 2024-02-09T17:27:04.362 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.362 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa8847ec:::benchmark_data_smithi179_77138_object32986:head# 2024-02-09T17:27:04.362 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.362 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa886130:::benchmark_data_smithi179_77138_object63048:head# 2024-02-09T17:27:04.362 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.362 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa896cc3:::benchmark_data_smithi179_77138_object62220:head# 2024-02-09T17:27:04.362 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.362 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa897da3:::benchmark_data_smithi179_77138_object74974:head# 2024-02-09T17:27:04.362 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.362 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa89a76e:::benchmark_data_smithi179_77138_object67531:head# 2024-02-09T17:27:04.362 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.363 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa8a4406:::benchmark_data_smithi179_77138_object128880:head# 2024-02-09T17:27:04.363 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.363 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa8ac88d:::benchmark_data_smithi179_77138_object43002:head# 2024-02-09T17:27:04.363 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.364 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:27:04.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:04.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:27:04.379 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa8b34b6:::benchmark_data_smithi179_77138_object26616:head# 2024-02-09T17:27:04.380 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.380 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa8b3944:::benchmark_data_smithi179_77138_object100407:head# 2024-02-09T17:27:04.380 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.380 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa8b3cc8:::benchmark_data_smithi179_77138_object37083:head# 2024-02-09T17:27:04.380 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.380 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa8d3c4a:::benchmark_data_smithi179_77138_object93289:head# 2024-02-09T17:27:04.380 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.380 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa8e151e:::benchmark_data_smithi179_77138_object58173:head# 2024-02-09T17:27:04.380 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.380 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa8f001e:::benchmark_data_smithi179_77138_object50374:head# 2024-02-09T17:27:04.380 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.380 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa8f2f09:::benchmark_data_smithi179_77138_object66858:head# 2024-02-09T17:27:04.381 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.381 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa8f2fa1:::benchmark_data_smithi179_77138_object108349:head# 2024-02-09T17:27:04.381 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.381 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa8fb812:::benchmark_data_smithi179_77138_object73802:head# 2024-02-09T17:27:04.381 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.381 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa9036dc:::benchmark_data_smithi179_77138_object80454:head# 2024-02-09T17:27:04.381 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.381 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa90ee46:::benchmark_data_smithi179_77138_object87473:head# 2024-02-09T17:27:04.381 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.381 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa90f914:::benchmark_data_smithi179_77138_object55777:head# 2024-02-09T17:27:04.381 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.381 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa9107c4:::benchmark_data_smithi179_77138_object36254:head# 2024-02-09T17:27:04.381 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.381 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa913187:::benchmark_data_smithi179_77138_object42341:head# 2024-02-09T17:27:04.382 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.382 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa917326:::benchmark_data_smithi179_77138_object106189:head# 2024-02-09T17:27:04.382 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.382 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa920039:::benchmark_data_smithi179_77138_object75212:head# 2024-02-09T17:27:04.382 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.382 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa922694:::benchmark_data_smithi179_77138_object52306:head# 2024-02-09T17:27:04.382 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.382 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa927d54:::benchmark_data_smithi179_77138_object46294:head# 2024-02-09T17:27:04.382 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.382 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa92a1ed:::benchmark_data_smithi179_77138_object104751:head# 2024-02-09T17:27:04.382 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.382 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa930330:::benchmark_data_smithi179_77138_object72213:head# 2024-02-09T17:27:04.382 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.383 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa93c3b7:::benchmark_data_smithi179_77138_object71779:head# 2024-02-09T17:27:04.383 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.383 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa93ece1:::benchmark_data_smithi179_77138_object66394:head# 2024-02-09T17:27:04.383 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.383 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa93eef9:::benchmark_data_smithi179_77138_object38752:head# 2024-02-09T17:27:04.383 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.383 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa940bfd:::benchmark_data_smithi179_77138_object85143:head# 2024-02-09T17:27:04.383 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.383 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa9429d5:::benchmark_data_smithi179_77138_object47099:head# 2024-02-09T17:27:04.383 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.383 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa94a9d2:::benchmark_data_smithi179_77138_object32201:head# 2024-02-09T17:27:04.383 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.383 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa952cd2:::benchmark_data_smithi179_77138_object41056:head# 2024-02-09T17:27:04.384 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.384 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa954315:::benchmark_data_smithi179_77138_object126499:head# 2024-02-09T17:27:04.384 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.384 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa959c72:::benchmark_data_smithi179_77138_object31639:head# 2024-02-09T17:27:04.384 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.384 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa95d6c6:::benchmark_data_smithi179_77138_object122492:head# 2024-02-09T17:27:04.384 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.384 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa9601d4:::benchmark_data_smithi179_77138_object129778:head# 2024-02-09T17:27:04.384 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.384 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa970253:::benchmark_data_smithi179_77138_object64513:head# 2024-02-09T17:27:04.384 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.384 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa97344a:::benchmark_data_smithi179_77138_object119584:head# 2024-02-09T17:27:04.385 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.385 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa976fe2:::benchmark_data_smithi179_77138_object21673:head# 2024-02-09T17:27:04.385 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.385 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa983f00:::benchmark_data_smithi179_77138_object25076:head# 2024-02-09T17:27:04.385 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.385 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa985f8a:::benchmark_data_smithi179_77138_object55952:head# 2024-02-09T17:27:04.385 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.385 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa988414:::benchmark_data_smithi179_77138_object28088:head# 2024-02-09T17:27:04.385 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.385 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa992651:::benchmark_data_smithi179_77138_object129106:head# 2024-02-09T17:27:04.385 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.385 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa9948b4:::benchmark_data_smithi179_77138_object113085:head# 2024-02-09T17:27:04.385 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.385 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa99961f:::benchmark_data_smithi179_77138_object105999:head# 2024-02-09T17:27:04.386 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.386 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa99f3fb:::benchmark_data_smithi179_77138_object98324:head# 2024-02-09T17:27:04.386 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.386 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa9a5461:::benchmark_data_smithi179_77138_object45247:head# 2024-02-09T17:27:04.386 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.386 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa9a5819:::benchmark_data_smithi179_77138_object63156:head# 2024-02-09T17:27:04.386 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.386 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa9a7a7d:::benchmark_data_smithi179_77138_object59936:head# 2024-02-09T17:27:04.386 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.386 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa9a7c51:::benchmark_data_smithi179_77138_object101303:head# 2024-02-09T17:27:04.386 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.386 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa9a9c61:::benchmark_data_smithi179_77138_object37732:head# 2024-02-09T17:27:04.386 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.387 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa9b5e21:::benchmark_data_smithi179_77138_object75458:head# 2024-02-09T17:27:04.387 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.387 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa9b9bd8:::benchmark_data_smithi179_77138_object53962:head# 2024-02-09T17:27:04.387 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.387 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa9cc8c3:::benchmark_data_smithi179_77138_object47744:head# 2024-02-09T17:27:04.387 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.387 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa9d2a0c:::benchmark_data_smithi179_77138_object34291:head# 2024-02-09T17:27:04.387 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.387 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa9d4721:::benchmark_data_smithi179_77138_object49423:head# 2024-02-09T17:27:04.387 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.387 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa9da503:::benchmark_data_smithi179_77138_object90574:head# 2024-02-09T17:27:04.387 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.387 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa9dd32e:::benchmark_data_smithi179_77138_object99111:head# 2024-02-09T17:27:04.387 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.388 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa9dfdd4:::benchmark_data_smithi179_77138_object33097:head# 2024-02-09T17:27:04.388 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.388 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa9f8ea3:::benchmark_data_smithi179_77138_object74625:head# 2024-02-09T17:27:04.388 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.388 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fa9ffd05:::benchmark_data_smithi179_77138_object51174:head# 2024-02-09T17:27:04.388 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.388 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faa03729:::benchmark_data_smithi179_77138_object55536:head# 2024-02-09T17:27:04.388 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.388 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faa04765:::benchmark_data_smithi179_77138_object63701:head# 2024-02-09T17:27:04.388 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.388 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faa06c2a:::benchmark_data_smithi179_77138_object114479:head# 2024-02-09T17:27:04.388 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.388 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faa08076:::benchmark_data_smithi179_77138_object72996:head# 2024-02-09T17:27:04.388 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.389 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faa0dee9:::benchmark_data_smithi179_77138_object47773:head# 2024-02-09T17:27:04.389 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.389 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faa11512:::benchmark_data_smithi179_77138_object130868:head# 2024-02-09T17:27:04.389 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.389 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faa219a7:::benchmark_data_smithi179_77138_object72092:head# 2024-02-09T17:27:04.389 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.389 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faa35d0b:::benchmark_data_smithi179_77138_object80181:head# 2024-02-09T17:27:04.389 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.389 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faa3bd0c:::benchmark_data_smithi179_77138_object86377:head# 2024-02-09T17:27:04.389 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.389 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faa4c27d:::benchmark_data_smithi179_77138_object82600:head# 2024-02-09T17:27:04.389 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.389 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faa6f0fa:::benchmark_data_smithi179_77138_object67732:head# 2024-02-09T17:27:04.389 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.390 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faa70448:::benchmark_data_smithi179_77138_object35903:head# 2024-02-09T17:27:04.390 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.390 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faa7d027:::benchmark_data_smithi179_77138_object70520:head# 2024-02-09T17:27:04.390 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.390 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faa87f1e:::benchmark_data_smithi179_77138_object91390:head# 2024-02-09T17:27:04.390 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.390 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faa8daab:::benchmark_data_smithi179_77138_object126436:head# 2024-02-09T17:27:04.390 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.390 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faa908ed:::benchmark_data_smithi179_77138_object129413:head# 2024-02-09T17:27:04.390 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.390 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faa96007:::benchmark_data_smithi179_77138_object96046:head# 2024-02-09T17:27:04.390 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.390 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faa9768d:::benchmark_data_smithi179_77138_object56846:head# 2024-02-09T17:27:04.390 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.391 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faa97ea9:::benchmark_data_smithi179_77138_object62875:head# 2024-02-09T17:27:04.391 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.391 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faaa3f08:::benchmark_data_smithi179_77138_object26794:head# 2024-02-09T17:27:04.391 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.391 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faaa8e04:::benchmark_data_smithi179_77138_object127404:head# 2024-02-09T17:27:04.391 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.391 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faaaf884:::benchmark_data_smithi179_77138_object51439:head# 2024-02-09T17:27:04.391 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.391 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faab3bcf:::benchmark_data_smithi179_77138_object62190:head# 2024-02-09T17:27:04.391 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.391 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faab6141:::benchmark_data_smithi179_77138_object90570:head# 2024-02-09T17:27:04.391 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.391 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faadc507:::benchmark_data_smithi179_77138_object99291:head# 2024-02-09T17:27:04.391 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.392 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faae0adc:::benchmark_data_smithi179_77138_object66077:head# 2024-02-09T17:27:04.392 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.392 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faaee3d2:::benchmark_data_smithi179_77138_object127423:head# 2024-02-09T17:27:04.392 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.392 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faaf12d7:::benchmark_data_smithi179_77138_object77390:head# 2024-02-09T17:27:04.392 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.392 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faaf311a:::benchmark_data_smithi179_77138_object44671:head# 2024-02-09T17:27:04.392 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.392 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fab17ad2:::benchmark_data_smithi179_77138_object129341:head# 2024-02-09T17:27:04.392 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.392 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fab1d16c:::benchmark_data_smithi179_77138_object105433:head# 2024-02-09T17:27:04.392 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.392 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fab233eb:::benchmark_data_smithi179_77138_object45343:head# 2024-02-09T17:27:04.392 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.392 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fab273e7:::benchmark_data_smithi179_77138_object99184:head# 2024-02-09T17:27:04.393 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.393 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fab2c969:::benchmark_data_smithi179_77138_object20686:head# 2024-02-09T17:27:04.393 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.393 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fab35723:::benchmark_data_smithi179_77138_object26679:head# 2024-02-09T17:27:04.393 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.393 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fab40c6e:::benchmark_data_smithi179_77138_object38571:head# 2024-02-09T17:27:04.393 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.393 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fab43c51:::benchmark_data_smithi179_77138_object55461:head# 2024-02-09T17:27:04.393 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.408 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fab5233f:::benchmark_data_smithi179_77138_object48805:head# 2024-02-09T17:27:04.408 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.408 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fab60703:::benchmark_data_smithi179_77138_object111162:head# 2024-02-09T17:27:04.409 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.409 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fab6fda4:::benchmark_data_smithi179_77138_object43682:head# 2024-02-09T17:27:04.409 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.409 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fab6ff21:::benchmark_data_smithi179_77138_object41581:head# 2024-02-09T17:27:04.409 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.409 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fab7b819:::benchmark_data_smithi179_77138_object128476:head# 2024-02-09T17:27:04.409 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.409 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fab7c6d3:::benchmark_data_smithi179_77138_object19632:head# 2024-02-09T17:27:04.409 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.409 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fab80947:::benchmark_data_smithi179_77138_object45776:head# 2024-02-09T17:27:04.409 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.409 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fab8dad7:::benchmark_data_smithi179_77138_object97299:head# 2024-02-09T17:27:04.409 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.410 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fab90ec2:::benchmark_data_smithi179_77138_object26893:head# 2024-02-09T17:27:04.410 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.410 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fab97ecb:::benchmark_data_smithi179_77138_object42068:head# 2024-02-09T17:27:04.410 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.410 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faba0a1b:::benchmark_data_smithi179_77138_object54834:head# 2024-02-09T17:27:04.410 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.410 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fabac3ec:::benchmark_data_smithi179_77138_object89875:head# 2024-02-09T17:27:04.410 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.410 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fabae10b:::benchmark_data_smithi179_77138_object20260:head# 2024-02-09T17:27:04.410 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.410 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fabb0be5:::benchmark_data_smithi179_77138_object131032:head# 2024-02-09T17:27:04.410 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.410 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fabb21ab:::benchmark_data_smithi179_77138_object20859:head# 2024-02-09T17:27:04.410 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.411 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fabb76e6:::benchmark_data_smithi179_77138_object26582:head# 2024-02-09T17:27:04.411 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.411 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fabb87db:::benchmark_data_smithi179_77138_object72042:head# 2024-02-09T17:27:04.411 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.411 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fabbc1d3:::benchmark_data_smithi179_77138_object73589:head# 2024-02-09T17:27:04.411 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.411 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fabc1f5c:::benchmark_data_smithi179_77138_object131774:head# 2024-02-09T17:27:04.411 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.411 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fabc71f3:::benchmark_data_smithi179_77138_object62852:head# 2024-02-09T17:27:04.411 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.411 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fabd7b3b:::benchmark_data_smithi179_77138_object130230:head# 2024-02-09T17:27:04.411 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.411 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fabd8fdf:::benchmark_data_smithi179_77138_object48769:head# 2024-02-09T17:27:04.411 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.411 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fabdd7ad:::benchmark_data_smithi179_77138_object127931:head# 2024-02-09T17:27:04.412 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.412 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fabde182:::benchmark_data_smithi179_77138_object64243:head# 2024-02-09T17:27:04.412 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.412 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fabe0e60:::benchmark_data_smithi179_77138_object99937:head# 2024-02-09T17:27:04.412 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.412 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fabe152e:::benchmark_data_smithi179_77138_object43902:head# 2024-02-09T17:27:04.412 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.412 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fabe254c:::benchmark_data_smithi179_77138_object43704:head# 2024-02-09T17:27:04.412 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.412 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fabfa896:::benchmark_data_smithi179_77138_object20036:head# 2024-02-09T17:27:04.412 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.412 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fac038a5:::benchmark_data_smithi179_77138_object110379:head# 2024-02-09T17:27:04.412 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.412 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fac060ae:::benchmark_data_smithi179_77138_object56215:head# 2024-02-09T17:27:04.413 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.413 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fac0e831:::benchmark_data_smithi179_77138_object28551:head# 2024-02-09T17:27:04.413 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.413 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fac132f2:::benchmark_data_smithi179_77138_object125491:head# 2024-02-09T17:27:04.413 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.413 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fac1f901:::benchmark_data_smithi179_77138_object127911:head# 2024-02-09T17:27:04.413 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.413 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fac1ff9e:::benchmark_data_smithi179_77138_object99467:head# 2024-02-09T17:27:04.413 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.413 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fac2211c:::benchmark_data_smithi179_77138_object108633:head# 2024-02-09T17:27:04.413 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.413 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fac231a5:::benchmark_data_smithi179_77138_object105115:head# 2024-02-09T17:27:04.413 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.413 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fac26fee:::benchmark_data_smithi179_77138_object120211:head# 2024-02-09T17:27:04.414 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.414 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fac3b2ae:::benchmark_data_smithi179_77138_object56094:head# 2024-02-09T17:27:04.414 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.414 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fac513ba:::benchmark_data_smithi179_77138_object24173:head# 2024-02-09T17:27:04.414 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.414 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fac54cd1:::benchmark_data_smithi179_77138_object78818:head# 2024-02-09T17:27:04.414 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.414 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fac5c591:::benchmark_data_smithi179_77138_object62321:head# 2024-02-09T17:27:04.414 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.414 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fac5d2e1:::benchmark_data_smithi179_77138_object73838:head# 2024-02-09T17:27:04.414 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.414 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fac5ed23:::benchmark_data_smithi179_77138_object77772:head# 2024-02-09T17:27:04.414 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.415 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fac5f33a:::benchmark_data_smithi179_77138_object24775:head# 2024-02-09T17:27:04.415 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.415 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fac5f5a3:::benchmark_data_smithi179_77138_object125366:head# 2024-02-09T17:27:04.415 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.415 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fac6116c:::benchmark_data_smithi179_77138_object63235:head# 2024-02-09T17:27:04.415 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.415 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fac65d29:::benchmark_data_smithi179_77138_object19656:head# 2024-02-09T17:27:04.415 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.415 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fac70f79:::benchmark_data_smithi179_77138_object29868:head# 2024-02-09T17:27:04.415 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.415 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fac74b7d:::benchmark_data_smithi179_77138_object22996:head# 2024-02-09T17:27:04.415 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.415 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fac94431:::benchmark_data_smithi179_77138_object89028:head# 2024-02-09T17:27:04.415 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.416 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:facabd5f:::benchmark_data_smithi179_77138_object47644:head# 2024-02-09T17:27:04.416 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.416 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:facbbd03:::benchmark_data_smithi179_77138_object128150:head# 2024-02-09T17:27:04.416 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.416 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:face187c:::benchmark_data_smithi179_77138_object76125:head# 2024-02-09T17:27:04.416 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.416 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:face447a:::benchmark_data_smithi179_77138_object36372:head# 2024-02-09T17:27:04.416 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.416 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:facfaa65:::benchmark_data_smithi179_77138_object60446:head# 2024-02-09T17:27:04.416 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.416 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fad07c9b:::benchmark_data_smithi179_77138_object66861:head# 2024-02-09T17:27:04.416 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.416 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fad090fb:::benchmark_data_smithi179_77138_object95609:head# 2024-02-09T17:27:04.417 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.417 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fad0a551:::benchmark_data_smithi179_77138_object24366:head# 2024-02-09T17:27:04.417 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.417 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fad135f7:::benchmark_data_smithi179_77138_object32293:head# 2024-02-09T17:27:04.417 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.417 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fad2d302:::benchmark_data_smithi179_77138_object21759:head# 2024-02-09T17:27:04.417 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.417 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fad2d54b:::benchmark_data_smithi179_77138_object36250:head# 2024-02-09T17:27:04.417 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.417 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fad42f49:::benchmark_data_smithi179_77138_object123118:head# 2024-02-09T17:27:04.417 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.417 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fad45db3:::benchmark_data_smithi179_77138_object73243:head# 2024-02-09T17:27:04.417 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.417 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fad5c5e2:::benchmark_data_smithi179_77138_object34434:head# 2024-02-09T17:27:04.418 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.418 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fad6c3fb:::benchmark_data_smithi179_77138_object81048:head# 2024-02-09T17:27:04.418 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.418 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fad705d6:::benchmark_data_smithi179_77138_object103995:head# 2024-02-09T17:27:04.418 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.418 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fad71bbd:::benchmark_data_smithi179_77138_object39484:head# 2024-02-09T17:27:04.418 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.418 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fad8c76b:::benchmark_data_smithi179_77138_object107103:head# 2024-02-09T17:27:04.418 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.418 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fad8d6f7:::benchmark_data_smithi179_77138_object80251:head# 2024-02-09T17:27:04.418 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.418 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fad91fa5:::benchmark_data_smithi179_77138_object56297:head# 2024-02-09T17:27:04.418 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.419 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fad9a46c:::benchmark_data_smithi179_77138_object117524:head# 2024-02-09T17:27:04.419 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.419 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fada3b58:::benchmark_data_smithi179_77138_object94663:head# 2024-02-09T17:27:04.419 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.419 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fadb0ca0:::benchmark_data_smithi179_77138_object23752:head# 2024-02-09T17:27:04.419 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.419 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fadb1018:::benchmark_data_smithi179_77138_object52191:head# 2024-02-09T17:27:04.419 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.419 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fadb4187:::benchmark_data_smithi179_77138_object118835:head# 2024-02-09T17:27:04.419 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.419 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fadbe255:::benchmark_data_smithi179_77138_object39239:head# 2024-02-09T17:27:04.419 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.419 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fadc7f67:::benchmark_data_smithi179_77138_object121426:head# 2024-02-09T17:27:04.419 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.420 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fadd43ee:::benchmark_data_smithi179_77138_object33522:head# 2024-02-09T17:27:04.420 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.420 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fadd713d:::benchmark_data_smithi179_77138_object113537:head# 2024-02-09T17:27:04.420 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.420 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faddfb20:::benchmark_data_smithi179_77138_object22161:head# 2024-02-09T17:27:04.420 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.420 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fade1224:::benchmark_data_smithi179_77138_object20823:head# 2024-02-09T17:27:04.420 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.420 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fade77f0:::benchmark_data_smithi179_77138_object107313:head# 2024-02-09T17:27:04.420 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.420 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fadfdcb1:::benchmark_data_smithi179_77138_object120939:head# 2024-02-09T17:27:04.420 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.420 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fae07bf7:::benchmark_data_smithi179_77138_object87701:head# 2024-02-09T17:27:04.420 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.421 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fae0fac3:::benchmark_data_smithi179_77138_object29677:head# 2024-02-09T17:27:04.421 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.421 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fae15342:::benchmark_data_smithi179_77138_object19362:head# 2024-02-09T17:27:04.421 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.421 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fae1ee5f:::benchmark_data_smithi179_77138_object39430:head# 2024-02-09T17:27:04.421 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.421 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fae23053:::benchmark_data_smithi179_77138_object22799:head# 2024-02-09T17:27:04.421 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.421 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fae27c94:::benchmark_data_smithi179_77138_object56610:head# 2024-02-09T17:27:04.421 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.421 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fae34b2f:::benchmark_data_smithi179_77138_object128272:head# 2024-02-09T17:27:04.421 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.421 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fae3773f:::benchmark_data_smithi179_77138_object43628:head# 2024-02-09T17:27:04.422 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.422 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fae4460f:::benchmark_data_smithi179_77138_object89546:head# 2024-02-09T17:27:04.422 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.422 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fae4f29b:::benchmark_data_smithi179_77138_object42694:head# 2024-02-09T17:27:04.422 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.422 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fae62470:::benchmark_data_smithi179_77138_object79565:head# 2024-02-09T17:27:04.422 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.422 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fae6b75e:::benchmark_data_smithi179_77138_object112061:head# 2024-02-09T17:27:04.422 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.422 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fae783d5:::benchmark_data_smithi179_77138_object42869:head# 2024-02-09T17:27:04.422 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.422 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fae7ca7b:::benchmark_data_smithi179_77138_object101755:head# 2024-02-09T17:27:04.422 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.423 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fae81bab:::benchmark_data_smithi179_77138_object103237:head# 2024-02-09T17:27:04.423 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.423 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fae9490d:::benchmark_data_smithi179_77138_object93530:head# 2024-02-09T17:27:04.423 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.423 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fae96f83:::benchmark_data_smithi179_77138_object113503:head# 2024-02-09T17:27:04.423 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.423 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faeab799:::benchmark_data_smithi179_77138_object74479:head# 2024-02-09T17:27:04.423 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.423 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faeac6d3:::benchmark_data_smithi179_77138_object127315:head# 2024-02-09T17:27:04.423 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.423 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faed4933:::benchmark_data_smithi179_77138_object131000:head# 2024-02-09T17:27:04.423 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.423 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faed941c:::benchmark_data_smithi179_77138_object40587:head# 2024-02-09T17:27:04.423 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.424 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faedcfb8:::benchmark_data_smithi179_77138_object27933:head# 2024-02-09T17:27:04.424 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.424 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faee2f8f:::benchmark_data_smithi179_77138_object28209:head# 2024-02-09T17:27:04.424 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.424 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faee5c39:::benchmark_data_smithi179_77138_object85377:head# 2024-02-09T17:27:04.424 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.424 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faee670b:::benchmark_data_smithi179_77138_object90447:head# 2024-02-09T17:27:04.424 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.424 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faef3e21:::benchmark_data_smithi179_77138_object107743:head# 2024-02-09T17:27:04.424 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.424 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faefa220:::benchmark_data_smithi179_77138_object32424:head# 2024-02-09T17:27:04.424 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.424 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faefdb35:::benchmark_data_smithi179_77138_object70086:head# 2024-02-09T17:27:04.424 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.425 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faf142a5:::benchmark_data_smithi179_77138_object20227:head# 2024-02-09T17:27:04.425 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.425 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faf35ed7:::benchmark_data_smithi179_77138_object49028:head# 2024-02-09T17:27:04.425 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.425 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faf38098:::benchmark_data_smithi179_77138_object85084:head# 2024-02-09T17:27:04.425 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.425 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faf41df5:::benchmark_data_smithi179_77138_object65459:head# 2024-02-09T17:27:04.425 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.461 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:27:04.461 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:27:04.461 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:27:04.467 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faf5aa3f:::benchmark_data_smithi179_77138_object94973:head# 2024-02-09T17:27:04.467 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.467 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faf6c8a6:::benchmark_data_smithi179_77138_object51358:head# 2024-02-09T17:27:04.467 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.467 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faf7d635:::benchmark_data_smithi179_77138_object49011:head# 2024-02-09T17:27:04.467 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.468 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faf951fa:::benchmark_data_smithi179_77138_object103353:head# 2024-02-09T17:27:04.468 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.468 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faf96fd9:::benchmark_data_smithi179_77138_object89162:head# 2024-02-09T17:27:04.468 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.468 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fafa30c5:::benchmark_data_smithi179_77138_object90742:head# 2024-02-09T17:27:04.468 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.468 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fafacfd7:::benchmark_data_smithi179_77138_object126859:head# 2024-02-09T17:27:04.468 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.468 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fafbedf2:::benchmark_data_smithi179_77138_object102557:head# 2024-02-09T17:27:04.468 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.468 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fafbf844:::benchmark_data_smithi179_77138_object31628:head# 2024-02-09T17:27:04.468 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.468 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fafc246e:::benchmark_data_smithi179_77138_object44947:head# 2024-02-09T17:27:04.469 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.469 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fafe9261:::benchmark_data_smithi179_77138_object116148:head# 2024-02-09T17:27:04.469 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.469 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fafea24d:::benchmark_data_smithi179_77138_object21308:head# 2024-02-09T17:27:04.469 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.469 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:faff23d7:::benchmark_data_smithi179_77138_object28660:head# 2024-02-09T17:27:04.469 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.469 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb0053fb:::benchmark_data_smithi179_77138_object73607:head# 2024-02-09T17:27:04.469 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.469 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb00b094:::benchmark_data_smithi179_77138_object95961:head# 2024-02-09T17:27:04.469 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.469 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb00d07f:::benchmark_data_smithi179_77138_object125461:head# 2024-02-09T17:27:04.469 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.470 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb00e55f:::benchmark_data_smithi179_77138_object61761:head# 2024-02-09T17:27:04.470 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.470 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb00e72d:::benchmark_data_smithi179_77138_object25030:head# 2024-02-09T17:27:04.470 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.470 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb013bb2:::benchmark_data_smithi179_77138_object65571:head# 2024-02-09T17:27:04.470 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.470 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb017c31:::benchmark_data_smithi179_77138_object21787:head# 2024-02-09T17:27:04.470 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.470 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb01ed45:::benchmark_data_smithi179_77138_object73319:head# 2024-02-09T17:27:04.470 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.470 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb0214cc:::benchmark_data_smithi179_77138_object129788:head# 2024-02-09T17:27:04.470 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.470 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb02291e:::benchmark_data_smithi179_77138_object87447:head# 2024-02-09T17:27:04.470 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.470 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb022ddd:::benchmark_data_smithi179_77138_object27774:head# 2024-02-09T17:27:04.471 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.471 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb02ae7a:::benchmark_data_smithi179_77138_object101724:head# 2024-02-09T17:27:04.471 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.471 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb0337b4:::benchmark_data_smithi179_77138_object29799:head# 2024-02-09T17:27:04.471 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.471 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb033c97:::benchmark_data_smithi179_77138_object119283:head# 2024-02-09T17:27:04.471 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.471 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb058ec7:::benchmark_data_smithi179_77138_object111444:head# 2024-02-09T17:27:04.471 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.471 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb05d5f7:::benchmark_data_smithi179_77138_object59633:head# 2024-02-09T17:27:04.471 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.471 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb0766da:::benchmark_data_smithi179_77138_object120248:head# 2024-02-09T17:27:04.471 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.471 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb07970d:::benchmark_data_smithi179_77138_object44384:head# 2024-02-09T17:27:04.472 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.472 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb07d9e1:::benchmark_data_smithi179_77138_object31094:head# 2024-02-09T17:27:04.472 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.472 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb08aab0:::benchmark_data_smithi179_77138_object71390:head# 2024-02-09T17:27:04.472 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.472 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb08ff2f:::benchmark_data_smithi179_77138_object56891:head# 2024-02-09T17:27:04.472 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.472 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb09d67f:::benchmark_data_smithi179_77138_object42352:head# 2024-02-09T17:27:04.472 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.472 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb0a26e8:::benchmark_data_smithi179_77138_object117387:head# 2024-02-09T17:27:04.472 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.472 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb0b2263:::benchmark_data_smithi179_77138_object104604:head# 2024-02-09T17:27:04.472 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.472 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb0c8954:::benchmark_data_smithi179_77138_object91318:head# 2024-02-09T17:27:04.473 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.473 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb0d7e8f:::benchmark_data_smithi179_77138_object124554:head# 2024-02-09T17:27:04.473 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.473 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb0e0fb6:::benchmark_data_smithi179_77138_object21869:head# 2024-02-09T17:27:04.473 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.473 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb0e9564:::benchmark_data_smithi179_77138_object25767:head# 2024-02-09T17:27:04.473 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.473 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb0ed28f:::benchmark_data_smithi179_77138_object68399:head# 2024-02-09T17:27:04.473 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.473 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb10f4a3:::benchmark_data_smithi179_77138_object82640:head# 2024-02-09T17:27:04.473 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.473 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb1325a5:::benchmark_data_smithi179_77138_object86976:head# 2024-02-09T17:27:04.473 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.474 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb13777c:::benchmark_data_smithi179_77138_object127513:head# 2024-02-09T17:27:04.474 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.474 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb13c7c3:::benchmark_data_smithi179_77138_object49323:head# 2024-02-09T17:27:04.474 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.474 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb145ef3:::benchmark_data_smithi179_77138_object64895:head# 2024-02-09T17:27:04.474 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.474 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb1487ee:::benchmark_data_smithi179_77138_object46688:head# 2024-02-09T17:27:04.474 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.474 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb14eec0:::benchmark_data_smithi179_77138_object90385:head# 2024-02-09T17:27:04.474 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.474 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb14f56f:::benchmark_data_smithi179_77138_object34436:head# 2024-02-09T17:27:04.474 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.474 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb15835a:::benchmark_data_smithi179_77138_object102566:head# 2024-02-09T17:27:04.475 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.475 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb17091c:::benchmark_data_smithi179_77138_object112034:head# 2024-02-09T17:27:04.475 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.475 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb1716c1:::benchmark_data_smithi179_77138_object48500:head# 2024-02-09T17:27:04.475 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.475 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb18f869:::benchmark_data_smithi179_77138_object40988:head# 2024-02-09T17:27:04.475 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.475 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb19d400:::benchmark_data_smithi179_77138_object108057:head# 2024-02-09T17:27:04.475 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.475 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb19ec89:::benchmark_data_smithi179_77138_object105803:head# 2024-02-09T17:27:04.475 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.475 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb1aebd1:::benchmark_data_smithi179_77138_object77145:head# 2024-02-09T17:27:04.475 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.475 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb1af90b:::benchmark_data_smithi179_77138_object48567:head# 2024-02-09T17:27:04.476 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.476 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb1b8b84:::benchmark_data_smithi179_77138_object109906:head# 2024-02-09T17:27:04.476 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.476 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb1bb00e:::benchmark_data_smithi179_77138_object65329:head# 2024-02-09T17:27:04.476 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.476 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb1bb139:::benchmark_data_smithi179_77138_object56427:head# 2024-02-09T17:27:04.476 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.476 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb1c90e7:::benchmark_data_smithi179_77138_object89258:head# 2024-02-09T17:27:04.476 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.476 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb1ceeea:::benchmark_data_smithi179_77138_object73305:head# 2024-02-09T17:27:04.476 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.476 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb2009eb:::benchmark_data_smithi179_77138_object119745:head# 2024-02-09T17:27:04.477 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.477 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb20ded1:::benchmark_data_smithi179_77138_object73137:head# 2024-02-09T17:27:04.477 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.477 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb210d52:::benchmark_data_smithi179_77138_object125137:head# 2024-02-09T17:27:04.477 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.477 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb25109a:::benchmark_data_smithi179_77138_object116742:head# 2024-02-09T17:27:04.477 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.477 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb25cb09:::benchmark_data_smithi179_77138_object121911:head# 2024-02-09T17:27:04.477 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.477 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb260ddc:::benchmark_data_smithi179_77138_object49582:head# 2024-02-09T17:27:04.477 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.477 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb2679ba:::benchmark_data_smithi179_77138_object74508:head# 2024-02-09T17:27:04.477 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.477 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb26c75a:::benchmark_data_smithi179_77138_object107107:head# 2024-02-09T17:27:04.478 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.478 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb26d886:::benchmark_data_smithi179_77138_object40379:head# 2024-02-09T17:27:04.478 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.478 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb27278f:::benchmark_data_smithi179_77138_object48196:head# 2024-02-09T17:27:04.478 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.478 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb27e1ed:::benchmark_data_smithi179_77138_object113834:head# 2024-02-09T17:27:04.478 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.478 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb293e01:::benchmark_data_smithi179_77138_object23148:head# 2024-02-09T17:27:04.478 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.478 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb295e29:::benchmark_data_smithi179_77138_object114096:head# 2024-02-09T17:27:04.478 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.478 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb2a33e0:::benchmark_data_smithi179_77138_object34184:head# 2024-02-09T17:27:04.478 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.478 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb2a680b:::benchmark_data_smithi179_77138_object81413:head# 2024-02-09T17:27:04.479 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.479 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb2c9093:::benchmark_data_smithi179_77138_object75312:head# 2024-02-09T17:27:04.479 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.479 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb2cd384:::benchmark_data_smithi179_77138_object43224:head# 2024-02-09T17:27:04.479 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.479 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb2d0299:::benchmark_data_smithi179_77138_object85313:head# 2024-02-09T17:27:04.479 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.479 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb2d6bcd:::benchmark_data_smithi179_77138_object54765:head# 2024-02-09T17:27:04.479 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.479 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb2f1a08:::benchmark_data_smithi179_77138_object85955:head# 2024-02-09T17:27:04.479 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.479 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb2f2dd0:::benchmark_data_smithi179_77138_object111801:head# 2024-02-09T17:27:04.479 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.480 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb2f4285:::benchmark_data_smithi179_77138_object65017:head# 2024-02-09T17:27:04.480 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.480 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb2f4a8b:::benchmark_data_smithi179_77138_object77677:head# 2024-02-09T17:27:04.480 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.480 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb2fc7b9:::benchmark_data_smithi179_77138_object34781:head# 2024-02-09T17:27:04.480 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.480 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb303462:::benchmark_data_smithi179_77138_object78268:head# 2024-02-09T17:27:04.480 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.480 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb312c64:::benchmark_data_smithi179_77138_object83679:head# 2024-02-09T17:27:04.480 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.480 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb3134a4:::benchmark_data_smithi179_77138_object115968:head# 2024-02-09T17:27:04.480 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.480 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb3149ed:::benchmark_data_smithi179_77138_object26489:head# 2024-02-09T17:27:04.480 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.481 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb32369a:::benchmark_data_smithi179_77138_object92764:head# 2024-02-09T17:27:04.481 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.481 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb32544f:::benchmark_data_smithi179_77138_object116667:head# 2024-02-09T17:27:04.481 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.481 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb3310b4:::benchmark_data_smithi179_77138_object64267:head# 2024-02-09T17:27:04.481 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.481 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb33d3f1:::benchmark_data_smithi179_77138_object78350:head# 2024-02-09T17:27:04.481 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.481 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb3433b5:::benchmark_data_smithi179_77138_object76291:head# 2024-02-09T17:27:04.481 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.481 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb3473ac:::benchmark_data_smithi179_77138_object114544:head# 2024-02-09T17:27:04.481 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.481 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb355c56:::benchmark_data_smithi179_77138_object88068:head# 2024-02-09T17:27:04.482 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.482 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb35d8a5:::benchmark_data_smithi179_77138_object47064:head# 2024-02-09T17:27:04.482 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.482 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb369401:::benchmark_data_smithi179_77138_object33382:head# 2024-02-09T17:27:04.482 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.482 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb37df57:::benchmark_data_smithi179_77138_object128185:head# 2024-02-09T17:27:04.482 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.482 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb37e4ca:::benchmark_data_smithi179_77138_object52302:head# 2024-02-09T17:27:04.482 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.482 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb38486d:::benchmark_data_smithi179_77138_object38088:head# 2024-02-09T17:27:04.482 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.482 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb38d18a:::benchmark_data_smithi179_77138_object60670:head# 2024-02-09T17:27:04.482 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.483 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb393305:::benchmark_data_smithi179_77138_object98147:head# 2024-02-09T17:27:04.483 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.483 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb39f4fb:::benchmark_data_smithi179_77138_object78742:head# 2024-02-09T17:27:04.483 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.483 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb3a75e5:::benchmark_data_smithi179_77138_object23942:head# 2024-02-09T17:27:04.483 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.483 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb3ae83c:::benchmark_data_smithi179_77138_object34797:head# 2024-02-09T17:27:04.483 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.483 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb3af25f:::benchmark_data_smithi179_77138_object28624:head# 2024-02-09T17:27:04.483 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.483 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb3b9b7c:::benchmark_data_smithi179_77138_object99099:head# 2024-02-09T17:27:04.483 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.483 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb3bdc74:::benchmark_data_smithi179_77138_object114916:head# 2024-02-09T17:27:04.483 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.484 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb3c3887:::benchmark_data_smithi179_77138_object80862:head# 2024-02-09T17:27:04.484 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.484 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb3c54c1:::benchmark_data_smithi179_77138_object55755:head# 2024-02-09T17:27:04.484 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.484 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb3c5bd2:::benchmark_data_smithi179_77138_object117781:head# 2024-02-09T17:27:04.484 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.484 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb3c9a69:::benchmark_data_smithi179_77138_object131238:head# 2024-02-09T17:27:04.484 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.484 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb3ce721:::benchmark_data_smithi179_77138_object56071:head# 2024-02-09T17:27:04.484 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.484 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb3d7436:::benchmark_data_smithi179_77138_object114378:head# 2024-02-09T17:27:04.484 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.484 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb3eee84:::benchmark_data_smithi179_77138_object126866:head# 2024-02-09T17:27:04.485 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.485 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb3fc24c:::benchmark_data_smithi179_77138_object99201:head# 2024-02-09T17:27:04.485 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.485 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb40fa2e:::benchmark_data_smithi179_77138_object79682:head# 2024-02-09T17:27:04.485 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.485 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb411a9c:::benchmark_data_smithi179_77138_object63596:head# 2024-02-09T17:27:04.485 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.485 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb4149e0:::benchmark_data_smithi179_77138_object69081:head# 2024-02-09T17:27:04.485 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.485 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb416bfc:::benchmark_data_smithi179_77138_object114456:head# 2024-02-09T17:27:04.485 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.485 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb4239a7:::benchmark_data_smithi179_77138_object70028:head# 2024-02-09T17:27:04.485 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.485 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb427d1e:::benchmark_data_smithi179_77138_object127014:head# 2024-02-09T17:27:04.485 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.486 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb429c64:::benchmark_data_smithi179_77138_object120413:head# 2024-02-09T17:27:04.486 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.486 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb435781:::benchmark_data_smithi179_77138_object67559:head# 2024-02-09T17:27:04.486 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.486 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb44fdb0:::benchmark_data_smithi179_77138_object90376:head# 2024-02-09T17:27:04.486 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.486 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb452ebd:::benchmark_data_smithi179_77138_object35410:head# 2024-02-09T17:27:04.486 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.486 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb452f90:::benchmark_data_smithi179_77138_object100886:head# 2024-02-09T17:27:04.486 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.486 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb45f21c:::benchmark_data_smithi179_77138_object19921:head# 2024-02-09T17:27:04.486 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.486 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb4645a6:::benchmark_data_smithi179_77138_object96469:head# 2024-02-09T17:27:04.486 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.487 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb46b338:::benchmark_data_smithi179_77138_object21927:head# 2024-02-09T17:27:04.487 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.487 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb46bfd0:::benchmark_data_smithi179_77138_object31006:head# 2024-02-09T17:27:04.487 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.487 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb46cd85:::benchmark_data_smithi179_77138_object110497:head# 2024-02-09T17:27:04.487 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.487 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb476082:::benchmark_data_smithi179_77138_object116702:head# 2024-02-09T17:27:04.487 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.487 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb482fcf:::benchmark_data_smithi179_77138_object83492:head# 2024-02-09T17:27:04.487 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.487 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb485d95:::benchmark_data_smithi179_77138_object23565:head# 2024-02-09T17:27:04.487 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.487 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb4884da:::benchmark_data_smithi179_77138_object68422:head# 2024-02-09T17:27:04.487 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.488 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb4902f7:::benchmark_data_smithi179_77138_object120170:head# 2024-02-09T17:27:04.488 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.488 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb499c14:::benchmark_data_smithi179_77138_object63542:head# 2024-02-09T17:27:04.488 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.488 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb49abdf:::benchmark_data_smithi179_77138_object40268:head# 2024-02-09T17:27:04.488 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.488 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb4ae8a9:::benchmark_data_smithi179_77138_object27936:head# 2024-02-09T17:27:04.488 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.488 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb4af525:::benchmark_data_smithi179_77138_object41454:head# 2024-02-09T17:27:04.488 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.488 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb4b7bbb:::benchmark_data_smithi179_77138_object115271:head# 2024-02-09T17:27:04.488 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.488 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb4b9b73:::benchmark_data_smithi179_77138_object108967:head# 2024-02-09T17:27:04.488 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.489 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb4ba1e1:::benchmark_data_smithi179_77138_object54826:head# 2024-02-09T17:27:04.489 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.489 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb4c5797:::benchmark_data_smithi179_77138_object116434:head# 2024-02-09T17:27:04.489 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.489 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb4de436:::benchmark_data_smithi179_77138_object32303:head# 2024-02-09T17:27:04.489 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.489 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb4e6a1c:::benchmark_data_smithi179_77138_object47093:head# 2024-02-09T17:27:04.489 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.489 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb4f4869:::benchmark_data_smithi179_77138_object72524:head# 2024-02-09T17:27:04.489 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.489 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb4f8e89:::benchmark_data_smithi179_77138_object43111:head# 2024-02-09T17:27:04.489 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.489 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb4fec8b:::benchmark_data_smithi179_77138_object64628:head# 2024-02-09T17:27:04.489 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.490 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb505988:::benchmark_data_smithi179_77138_object99418:head# 2024-02-09T17:27:04.490 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.490 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb50b9f5:::benchmark_data_smithi179_77138_object27290:head# 2024-02-09T17:27:04.490 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.490 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb50e399:::benchmark_data_smithi179_77138_object89422:head# 2024-02-09T17:27:04.490 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.490 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb512338:::benchmark_data_smithi179_77138_object71909:head# 2024-02-09T17:27:04.490 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.490 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb512d1c:::benchmark_data_smithi179_77138_object95785:head# 2024-02-09T17:27:04.490 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.490 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb522de0:::benchmark_data_smithi179_77138_object25292:head# 2024-02-09T17:27:04.490 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.490 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb525ece:::benchmark_data_smithi179_77138_object56174:head# 2024-02-09T17:27:04.490 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.491 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb53a77d:::benchmark_data_smithi179_77138_object52439:head# 2024-02-09T17:27:04.491 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.491 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb545065:::benchmark_data_smithi179_77138_object116365:head# 2024-02-09T17:27:04.491 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.491 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb54a278:::benchmark_data_smithi179_77138_object124606:head# 2024-02-09T17:27:04.491 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.491 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb553d0f:::benchmark_data_smithi179_77138_object27027:head# 2024-02-09T17:27:04.491 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.491 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb565407:::benchmark_data_smithi179_77138_object55274:head# 2024-02-09T17:27:04.491 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.491 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb56ba35:::benchmark_data_smithi179_77138_object49586:head# 2024-02-09T17:27:04.491 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.491 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb57299a:::benchmark_data_smithi179_77138_object57588:head# 2024-02-09T17:27:04.491 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.492 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb57b773:::benchmark_data_smithi179_77138_object51722:head# 2024-02-09T17:27:04.492 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.492 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb58310d:::benchmark_data_smithi179_77138_object117835:head# 2024-02-09T17:27:04.492 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.492 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb588796:::benchmark_data_smithi179_77138_object46438:head# 2024-02-09T17:27:04.492 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.492 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb59a976:::benchmark_data_smithi179_77138_object53883:head# 2024-02-09T17:27:04.492 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.492 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb5adf59:::benchmark_data_smithi179_77138_object113243:head# 2024-02-09T17:27:04.492 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.492 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb5b2330:::benchmark_data_smithi179_77138_object71423:head# 2024-02-09T17:27:04.492 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.493 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb5b9842:::benchmark_data_smithi179_77138_object30483:head# 2024-02-09T17:27:04.493 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.493 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb5bbcc6:::benchmark_data_smithi179_77138_object118951:head# 2024-02-09T17:27:04.493 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.493 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb5bce70:::benchmark_data_smithi179_77138_object85114:head# 2024-02-09T17:27:04.493 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.493 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb5dd372:::benchmark_data_smithi179_77138_object48569:head# 2024-02-09T17:27:04.493 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.493 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb5df347:::benchmark_data_smithi179_77138_object70166:head# 2024-02-09T17:27:04.493 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.493 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb5f3dd2:::benchmark_data_smithi179_77138_object43612:head# 2024-02-09T17:27:04.493 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.493 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb5fdbf9:::benchmark_data_smithi179_77138_object59079:head# 2024-02-09T17:27:04.493 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.494 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb604757:::benchmark_data_smithi179_77138_object96977:head# 2024-02-09T17:27:04.494 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.494 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb606596:::benchmark_data_smithi179_77138_object91747:head# 2024-02-09T17:27:04.494 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.494 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb609354:::benchmark_data_smithi179_77138_object42167:head# 2024-02-09T17:27:04.494 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.494 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb610df7:::benchmark_data_smithi179_77138_object43479:head# 2024-02-09T17:27:04.494 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.494 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb6154d3:::benchmark_data_smithi179_77138_object19363:head# 2024-02-09T17:27:04.494 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.494 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb6180ec:::benchmark_data_smithi179_77138_object29075:head# 2024-02-09T17:27:04.494 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.494 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb61f614:::benchmark_data_smithi179_77138_object46996:head# 2024-02-09T17:27:04.495 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.495 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb62cc87:::benchmark_data_smithi179_77138_object114373:head# 2024-02-09T17:27:04.495 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.495 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb631675:::benchmark_data_smithi179_77138_object45601:head# 2024-02-09T17:27:04.495 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.495 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb642659:::benchmark_data_smithi179_77138_object33085:head# 2024-02-09T17:27:04.495 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.495 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb6428b7:::benchmark_data_smithi179_77138_object125468:head# 2024-02-09T17:27:04.495 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.495 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb645247:::benchmark_data_smithi179_77138_object57117:head# 2024-02-09T17:27:04.495 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.495 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb6718f6:::benchmark_data_smithi179_77138_object56277:head# 2024-02-09T17:27:04.495 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.496 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb67355d:::benchmark_data_smithi179_77138_object133163:head# 2024-02-09T17:27:04.496 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.496 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb6877ac:::benchmark_data_smithi179_77138_object47574:head# 2024-02-09T17:27:04.496 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.496 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb68c9de:::benchmark_data_smithi179_77138_object61166:head# 2024-02-09T17:27:04.496 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.496 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb68f48a:::benchmark_data_smithi179_77138_object95610:head# 2024-02-09T17:27:04.496 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.496 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb69586f:::benchmark_data_smithi179_77138_object67040:head# 2024-02-09T17:27:04.496 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.496 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb69d53d:::benchmark_data_smithi179_77138_object41859:head# 2024-02-09T17:27:04.496 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.496 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb6a28d8:::benchmark_data_smithi179_77138_object48886:head# 2024-02-09T17:27:04.496 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.497 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb6b9add:::benchmark_data_smithi179_77138_object113155:head# 2024-02-09T17:27:04.497 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.497 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb6be971:::benchmark_data_smithi179_77138_object120808:head# 2024-02-09T17:27:04.497 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.497 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb6c308f:::benchmark_data_smithi179_77138_object19510:head# 2024-02-09T17:27:04.497 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.497 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb6dba4f:::benchmark_data_smithi179_77138_object31470:head# 2024-02-09T17:27:04.497 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.497 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb6e3ef2:::benchmark_data_smithi179_77138_object34267:head# 2024-02-09T17:27:04.497 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.497 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb6ff23f:::benchmark_data_smithi179_77138_object125156:head# 2024-02-09T17:27:04.497 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.497 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb706804:::benchmark_data_smithi179_77138_object130574:head# 2024-02-09T17:27:04.497 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.497 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb722e66:::benchmark_data_smithi179_77138_object102965:head# 2024-02-09T17:27:04.498 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.498 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb7249de:::benchmark_data_smithi179_77138_object38650:head# 2024-02-09T17:27:04.498 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.498 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb73097a:::benchmark_data_smithi179_77138_object57365:head# 2024-02-09T17:27:04.498 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.498 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb73b913:::benchmark_data_smithi179_77138_object72656:head# 2024-02-09T17:27:04.498 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.498 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb7437a6:::benchmark_data_smithi179_77138_object22595:head# 2024-02-09T17:27:04.498 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.498 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb745030:::benchmark_data_smithi179_77138_object97477:head# 2024-02-09T17:27:04.498 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.498 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb74f8c8:::benchmark_data_smithi179_77138_object20181:head# 2024-02-09T17:27:04.498 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.498 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb757a78:::benchmark_data_smithi179_77138_object64607:head# 2024-02-09T17:27:04.499 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.499 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb7651d5:::benchmark_data_smithi179_77138_object133103:head# 2024-02-09T17:27:04.499 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.499 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb775c2d:::benchmark_data_smithi179_77138_object57063:head# 2024-02-09T17:27:04.499 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.499 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb7865ad:::benchmark_data_smithi179_77138_object77781:head# 2024-02-09T17:27:04.499 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.499 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb788a42:::benchmark_data_smithi179_77138_object70244:head# 2024-02-09T17:27:04.499 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.499 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb78c111:::benchmark_data_smithi179_77138_object116947:head# 2024-02-09T17:27:04.499 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.499 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb78ce1f:::benchmark_data_smithi179_77138_object95397:head# 2024-02-09T17:27:04.499 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.499 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb7a7c54:::benchmark_data_smithi179_77138_object85685:head# 2024-02-09T17:27:04.499 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.500 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb7b322a:::benchmark_data_smithi179_77138_object53216:head# 2024-02-09T17:27:04.500 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.500 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb7bbda1:::benchmark_data_smithi179_77138_object111963:head# 2024-02-09T17:27:04.500 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.500 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb7c2715:::benchmark_data_smithi179_77138_object81889:head# 2024-02-09T17:27:04.500 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.500 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb7d1107:::benchmark_data_smithi179_77138_object62664:head# 2024-02-09T17:27:04.500 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.500 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb7da9ba:::benchmark_data_smithi179_77138_object124835:head# 2024-02-09T17:27:04.500 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.500 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb7dd779:::benchmark_data_smithi179_77138_object120387:head# 2024-02-09T17:27:04.500 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.500 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb7e561c:::benchmark_data_smithi179_77138_object105688:head# 2024-02-09T17:27:04.500 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.501 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb7e6273:::benchmark_data_smithi179_77138_object24304:head# 2024-02-09T17:27:04.501 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.501 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb7fd374:::benchmark_data_smithi179_77138_object50888:head# 2024-02-09T17:27:04.501 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.501 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb80795d:::benchmark_data_smithi179_77138_object122668:head# 2024-02-09T17:27:04.501 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.501 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb82e77b:::benchmark_data_smithi179_77138_object21903:head# 2024-02-09T17:27:04.501 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.501 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb83c5f1:::benchmark_data_smithi179_77138_object85556:head# 2024-02-09T17:27:04.501 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.501 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb83c731:::benchmark_data_smithi179_77138_object66383:head# 2024-02-09T17:27:04.501 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.501 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb854f44:::benchmark_data_smithi179_77138_object77608:head# 2024-02-09T17:27:04.501 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.502 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb8585bb:::benchmark_data_smithi179_77138_object34625:head# 2024-02-09T17:27:04.502 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.502 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb859984:::benchmark_data_smithi179_77138_object86364:head# 2024-02-09T17:27:04.502 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.502 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb85aaf7:::benchmark_data_smithi179_77138_object32011:head# 2024-02-09T17:27:04.502 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.502 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb85b60c:::benchmark_data_smithi179_77138_object20541:head# 2024-02-09T17:27:04.502 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.502 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb85fa1e:::benchmark_data_smithi179_77138_object117358:head# 2024-02-09T17:27:04.502 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.502 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb8606c6:::benchmark_data_smithi179_77138_object49269:head# 2024-02-09T17:27:04.502 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.502 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb86e3c7:::benchmark_data_smithi179_77138_object20486:head# 2024-02-09T17:27:04.502 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.502 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb88a19d:::benchmark_data_smithi179_77138_object23508:head# 2024-02-09T17:27:04.503 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.503 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb8a6e18:::benchmark_data_smithi179_77138_object62769:head# 2024-02-09T17:27:04.503 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.503 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb8ad7f5:::benchmark_data_smithi179_77138_object69608:head# 2024-02-09T17:27:04.503 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.503 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb8af2aa:::benchmark_data_smithi179_77138_object57659:head# 2024-02-09T17:27:04.503 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.503 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb8b8133:::benchmark_data_smithi179_77138_object92302:head# 2024-02-09T17:27:04.503 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.503 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb8ba91b:::benchmark_data_smithi179_77138_object127101:head# 2024-02-09T17:27:04.503 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.503 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb8c0afb:::benchmark_data_smithi179_77138_object90225:head# 2024-02-09T17:27:04.503 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.503 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb8c12ac:::benchmark_data_smithi179_77138_object41843:head# 2024-02-09T17:27:04.503 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.504 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb8c5975:::benchmark_data_smithi179_77138_object55825:head# 2024-02-09T17:27:04.504 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.504 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb8cca33:::benchmark_data_smithi179_77138_object63244:head# 2024-02-09T17:27:04.504 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.504 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:27:04.506 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:27:04.506 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:27:04.539 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb8ce860:::benchmark_data_smithi179_77138_object32771:head# 2024-02-09T17:27:04.539 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.539 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb8ed748:::benchmark_data_smithi179_77138_object85201:head# 2024-02-09T17:27:04.539 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.539 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb8f127a:::benchmark_data_smithi179_77138_object122860:head# 2024-02-09T17:27:04.539 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.539 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb8f57a6:::benchmark_data_smithi179_77138_object122476:head# 2024-02-09T17:27:04.540 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.540 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb9426f8:::benchmark_data_smithi179_77138_object65942:head# 2024-02-09T17:27:04.540 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.540 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb947ec2:::benchmark_data_smithi179_77138_object96591:head# 2024-02-09T17:27:04.540 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.540 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb95397f:::benchmark_data_smithi179_77138_object131376:head# 2024-02-09T17:27:04.540 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.540 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb95e62b:::benchmark_data_smithi179_77138_object83805:head# 2024-02-09T17:27:04.540 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.540 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb98c3e7:::benchmark_data_smithi179_77138_object109860:head# 2024-02-09T17:27:04.540 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.540 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb99318c:::benchmark_data_smithi179_77138_object74796:head# 2024-02-09T17:27:04.541 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.541 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb9ae248:::benchmark_data_smithi179_77138_object62441:head# 2024-02-09T17:27:04.541 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.541 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb9b67ec:::benchmark_data_smithi179_77138_object40829:head# 2024-02-09T17:27:04.541 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.541 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb9cb30b:::benchmark_data_smithi179_77138_object123081:head# 2024-02-09T17:27:04.541 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.541 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb9cf7f4:::benchmark_data_smithi179_77138_object69201:head# 2024-02-09T17:27:04.541 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.541 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb9d8de0:::benchmark_data_smithi179_77138_object116582:head# 2024-02-09T17:27:04.541 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.541 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb9e9d0c:::benchmark_data_smithi179_77138_object119743:head# 2024-02-09T17:27:04.541 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.541 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fb9fe858:::benchmark_data_smithi179_77138_object89933:head# 2024-02-09T17:27:04.542 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.542 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fba086b2:::benchmark_data_smithi179_77138_object82622:head# 2024-02-09T17:27:04.542 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.542 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fba1689b:::benchmark_data_smithi179_77138_object41860:head# 2024-02-09T17:27:04.542 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.542 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fba1752c:::benchmark_data_smithi179_77138_object120481:head# 2024-02-09T17:27:04.542 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.542 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fba1896c:::benchmark_data_smithi179_77138_object26031:head# 2024-02-09T17:27:04.542 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.542 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fba1a354:::benchmark_data_smithi179_77138_object130562:head# 2024-02-09T17:27:04.542 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.542 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fba3fbac:::benchmark_data_smithi179_77138_object82575:head# 2024-02-09T17:27:04.542 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.543 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fba44bf4:::benchmark_data_smithi179_77138_object124070:head# 2024-02-09T17:27:04.543 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.543 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fba44c6a:::benchmark_data_smithi179_77138_object115325:head# 2024-02-09T17:27:04.543 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.543 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fba506f0:::benchmark_data_smithi179_77138_object88867:head# 2024-02-09T17:27:04.543 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.543 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fba525d8:::benchmark_data_smithi179_77138_object92663:head# 2024-02-09T17:27:04.543 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.543 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fba71e44:::benchmark_data_smithi179_77138_object44991:head# 2024-02-09T17:27:04.543 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.543 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fba78331:::benchmark_data_smithi179_77138_object91651:head# 2024-02-09T17:27:04.543 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.543 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fba85c88:::benchmark_data_smithi179_77138_object54837:head# 2024-02-09T17:27:04.543 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.543 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fba94e70:::benchmark_data_smithi179_77138_object112712:head# 2024-02-09T17:27:04.544 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.544 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fba9e29e:::benchmark_data_smithi179_77138_object30496:head# 2024-02-09T17:27:04.544 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.544 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbaaa0bb:::benchmark_data_smithi179_77138_object52668:head# 2024-02-09T17:27:04.544 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.544 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbab791b:::benchmark_data_smithi179_77138_object60200:head# 2024-02-09T17:27:04.544 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.544 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbab7aab:::benchmark_data_smithi179_77138_object93885:head# 2024-02-09T17:27:04.544 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.544 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbaba2b3:::benchmark_data_smithi179_77138_object76507:head# 2024-02-09T17:27:04.544 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.544 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbac0c19:::benchmark_data_smithi179_77138_object130067:head# 2024-02-09T17:27:04.545 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.545 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbacadd6:::benchmark_data_smithi179_77138_object55662:head# 2024-02-09T17:27:04.545 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.545 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbad02fb:::benchmark_data_smithi179_77138_object61288:head# 2024-02-09T17:27:04.545 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.545 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbad89b8:::benchmark_data_smithi179_77138_object80554:head# 2024-02-09T17:27:04.545 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.545 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbadb7b7:::benchmark_data_smithi179_77138_object43839:head# 2024-02-09T17:27:04.545 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.545 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbadbf2e:::benchmark_data_smithi179_77138_object21820:head# 2024-02-09T17:27:04.545 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.545 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbadec57:::benchmark_data_smithi179_77138_object117698:head# 2024-02-09T17:27:04.545 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.545 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbadfae9:::benchmark_data_smithi179_77138_object43318:head# 2024-02-09T17:27:04.546 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.546 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbaf3436:::benchmark_data_smithi179_77138_object121547:head# 2024-02-09T17:27:04.546 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.546 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbaf88e8:::benchmark_data_smithi179_77138_object122801:head# 2024-02-09T17:27:04.546 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.546 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbafc781:::benchmark_data_smithi179_77138_object84475:head# 2024-02-09T17:27:04.546 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.546 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbafd9f7:::benchmark_data_smithi179_77138_object51417:head# 2024-02-09T17:27:04.546 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.546 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbb1d4cd:::benchmark_data_smithi179_77138_object26303:head# 2024-02-09T17:27:04.546 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.546 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbb2f57e:::benchmark_data_smithi179_77138_object102651:head# 2024-02-09T17:27:04.546 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.546 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbb34aae:::benchmark_data_smithi179_77138_object90457:head# 2024-02-09T17:27:04.547 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.547 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbb38e1b:::benchmark_data_smithi179_77138_object106011:head# 2024-02-09T17:27:04.547 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.547 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbb39805:::benchmark_data_smithi179_77138_object65615:head# 2024-02-09T17:27:04.547 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.547 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbb3da92:::benchmark_data_smithi179_77138_object70567:head# 2024-02-09T17:27:04.547 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.547 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbb42dba:::benchmark_data_smithi179_77138_object132662:head# 2024-02-09T17:27:04.547 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.547 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbb42e4e:::benchmark_data_smithi179_77138_object30828:head# 2024-02-09T17:27:04.547 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.547 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbb4d9ca:::benchmark_data_smithi179_77138_object65735:head# 2024-02-09T17:27:04.547 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.548 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbb512e5:::benchmark_data_smithi179_77138_object91214:head# 2024-02-09T17:27:04.548 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.548 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbb56137:::benchmark_data_smithi179_77138_object33111:head# 2024-02-09T17:27:04.548 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.548 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbb5e85d:::benchmark_data_smithi179_77138_object95528:head# 2024-02-09T17:27:04.548 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.548 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbb62899:::benchmark_data_smithi179_77138_object22400:head# 2024-02-09T17:27:04.548 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.548 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbb6a732:::benchmark_data_smithi179_77138_object30204:head# 2024-02-09T17:27:04.548 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.548 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbb6adec:::benchmark_data_smithi179_77138_object68579:head# 2024-02-09T17:27:04.548 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.548 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbb8a3b1:::benchmark_data_smithi179_77138_object120976:head# 2024-02-09T17:27:04.549 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.549 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbb941ba:::benchmark_data_smithi179_77138_object100541:head# 2024-02-09T17:27:04.549 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.549 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbb964e7:::benchmark_data_smithi179_77138_object56026:head# 2024-02-09T17:27:04.549 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.549 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbb9d781:::benchmark_data_smithi179_77138_object99498:head# 2024-02-09T17:27:04.549 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.549 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbb9e130:::benchmark_data_smithi179_77138_object55283:head# 2024-02-09T17:27:04.549 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.549 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbba79d7:::benchmark_data_smithi179_77138_object120723:head# 2024-02-09T17:27:04.549 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.549 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbbabb56:::benchmark_data_smithi179_77138_object88588:head# 2024-02-09T17:27:04.549 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.550 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbbae57c:::benchmark_data_smithi179_77138_object112930:head# 2024-02-09T17:27:04.550 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.550 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbbaf7f0:::benchmark_data_smithi179_77138_object106482:head# 2024-02-09T17:27:04.550 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.550 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbbb38a2:::benchmark_data_smithi179_77138_object93743:head# 2024-02-09T17:27:04.550 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.550 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbbb5a95:::benchmark_data_smithi179_77138_object25250:head# 2024-02-09T17:27:04.550 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.550 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbbb7101:::benchmark_data_smithi179_77138_object86105:head# 2024-02-09T17:27:04.550 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.550 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbbbcf8a:::benchmark_data_smithi179_77138_object96377:head# 2024-02-09T17:27:04.550 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.550 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbbbe0d2:::benchmark_data_smithi179_77138_object30216:head# 2024-02-09T17:27:04.551 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.551 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbbc27a7:::benchmark_data_smithi179_77138_object124971:head# 2024-02-09T17:27:04.551 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.551 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbbc99b1:::benchmark_data_smithi179_77138_object103526:head# 2024-02-09T17:27:04.551 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.551 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbbd08cf:::benchmark_data_smithi179_77138_object46691:head# 2024-02-09T17:27:04.551 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.551 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbbdf0ea:::benchmark_data_smithi179_77138_object51288:head# 2024-02-09T17:27:04.551 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.551 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbbeddd6:::benchmark_data_smithi179_77138_object37094:head# 2024-02-09T17:27:04.551 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.551 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbbef2ed:::benchmark_data_smithi179_77138_object39363:head# 2024-02-09T17:27:04.551 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.552 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbbff750:::benchmark_data_smithi179_77138_object45656:head# 2024-02-09T17:27:04.552 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.552 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbc03e8f:::benchmark_data_smithi179_77138_object71628:head# 2024-02-09T17:27:04.552 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.552 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbc08a91:::benchmark_data_smithi179_77138_object37688:head# 2024-02-09T17:27:04.552 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.552 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbc0bc14:::benchmark_data_smithi179_77138_object35342:head# 2024-02-09T17:27:04.552 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.552 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbc0c334:::benchmark_data_smithi179_77138_object70027:head# 2024-02-09T17:27:04.552 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.552 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbc0e1d6:::benchmark_data_smithi179_77138_object74195:head# 2024-02-09T17:27:04.552 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.552 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbc0e974:::benchmark_data_smithi179_77138_object79555:head# 2024-02-09T17:27:04.552 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.552 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbc117c4:::benchmark_data_smithi179_77138_object113114:head# 2024-02-09T17:27:04.553 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.553 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbc183aa:::benchmark_data_smithi179_77138_object68389:head# 2024-02-09T17:27:04.553 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.553 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbc22e1b:::benchmark_data_smithi179_77138_object129145:head# 2024-02-09T17:27:04.553 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.553 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbc2a609:::benchmark_data_smithi179_77138_object108545:head# 2024-02-09T17:27:04.553 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.553 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbc2e23b:::benchmark_data_smithi179_77138_object40428:head# 2024-02-09T17:27:04.553 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.553 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbc2ffa8:::benchmark_data_smithi179_77138_object21605:head# 2024-02-09T17:27:04.553 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.554 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbc33474:::benchmark_data_smithi179_77138_object105409:head# 2024-02-09T17:27:04.554 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.554 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbc3b9d3:::benchmark_data_smithi179_77138_object90316:head# 2024-02-09T17:27:04.554 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.554 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbc484c2:::benchmark_data_smithi179_77138_object19502:head# 2024-02-09T17:27:04.554 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.554 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbc58300:::benchmark_data_smithi179_77138_object130950:head# 2024-02-09T17:27:04.554 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.554 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbc590d5:::benchmark_data_smithi179_77138_object22093:head# 2024-02-09T17:27:04.554 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.554 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbc61d67:::benchmark_data_smithi179_77138_object69905:head# 2024-02-09T17:27:04.555 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.555 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbc6c9f7:::benchmark_data_smithi179_77138_object113231:head# 2024-02-09T17:27:04.555 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.555 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbc6ce51:::benchmark_data_smithi179_77138_object22830:head# 2024-02-09T17:27:04.555 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.555 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbc6e439:::benchmark_data_smithi179_77138_object98553:head# 2024-02-09T17:27:04.556 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.556 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbc6efa4:::benchmark_data_smithi179_77138_object129190:head# 2024-02-09T17:27:04.556 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.556 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbc74183:::benchmark_data_smithi179_77138_object128866:head# 2024-02-09T17:27:04.556 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.556 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbc79054:::benchmark_data_smithi179_77138_object91662:head# 2024-02-09T17:27:04.556 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.557 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbc7dcd4:::benchmark_data_smithi179_77138_object93188:head# 2024-02-09T17:27:04.557 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.557 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbc8b8d9:::benchmark_data_smithi179_77138_object80063:head# 2024-02-09T17:27:04.557 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.557 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbca9605:::benchmark_data_smithi179_77138_object97236:head# 2024-02-09T17:27:04.557 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.557 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbcb3720:::benchmark_data_smithi179_77138_object127249:head# 2024-02-09T17:27:04.557 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.558 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbcb4512:::benchmark_data_smithi179_77138_object107299:head# 2024-02-09T17:27:04.558 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.558 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbcb6f84:::benchmark_data_smithi179_77138_object73501:head# 2024-02-09T17:27:04.558 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.558 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbcd0729:::benchmark_data_smithi179_77138_object105873:head# 2024-02-09T17:27:04.558 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.558 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbcd3bb7:::benchmark_data_smithi179_77138_object86897:head# 2024-02-09T17:27:04.558 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.558 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbcf3688:::benchmark_data_smithi179_77138_object119056:head# 2024-02-09T17:27:04.558 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.558 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbcf4d9a:::benchmark_data_smithi179_77138_object93090:head# 2024-02-09T17:27:04.558 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.558 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbcfe0af:::benchmark_data_smithi179_77138_object81295:head# 2024-02-09T17:27:04.559 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.559 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbd11c41:::benchmark_data_smithi179_77138_object61131:head# 2024-02-09T17:27:04.559 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.559 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbd14c90:::benchmark_data_smithi179_77138_object73625:head# 2024-02-09T17:27:04.559 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.559 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbd1a9c9:::benchmark_data_smithi179_77138_object99302:head# 2024-02-09T17:27:04.559 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.559 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbd21e56:::benchmark_data_smithi179_77138_object66099:head# 2024-02-09T17:27:04.559 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.559 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbd258e3:::benchmark_data_smithi179_77138_object50829:head# 2024-02-09T17:27:04.559 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.559 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbd2bfda:::benchmark_data_smithi179_77138_object88318:head# 2024-02-09T17:27:04.559 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.559 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbd3040b:::benchmark_data_smithi179_77138_object60790:head# 2024-02-09T17:27:04.560 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.560 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbd33c48:::benchmark_data_smithi179_77138_object109382:head# 2024-02-09T17:27:04.560 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.560 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbd358ec:::benchmark_data_smithi179_77138_object80584:head# 2024-02-09T17:27:04.560 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.560 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbd366e6:::benchmark_data_smithi179_77138_object65194:head# 2024-02-09T17:27:04.560 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.560 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbd3be16:::benchmark_data_smithi179_77138_object127922:head# 2024-02-09T17:27:04.560 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.560 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbd3c926:::benchmark_data_smithi179_77138_object108749:head# 2024-02-09T17:27:04.560 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.560 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbd46c34:::benchmark_data_smithi179_77138_object131886:head# 2024-02-09T17:27:04.560 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.561 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbd65968:::benchmark_data_smithi179_77138_object131183:head# 2024-02-09T17:27:04.561 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.561 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbd66455:::benchmark_data_smithi179_77138_object55172:head# 2024-02-09T17:27:04.561 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.561 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbd6e916:::benchmark_data_smithi179_77138_object71095:head# 2024-02-09T17:27:04.561 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.561 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbd74f5c:::benchmark_data_smithi179_77138_object28670:head# 2024-02-09T17:27:04.561 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.561 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbd7cf42:::benchmark_data_smithi179_77138_object55185:head# 2024-02-09T17:27:04.561 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.561 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbd809be:::benchmark_data_smithi179_77138_object52466:head# 2024-02-09T17:27:04.561 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.561 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbd872e7:::benchmark_data_smithi179_77138_object86408:head# 2024-02-09T17:27:04.561 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.562 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbd8b02b:::benchmark_data_smithi179_77138_object25555:head# 2024-02-09T17:27:04.562 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.562 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbda442d:::benchmark_data_smithi179_77138_object107644:head# 2024-02-09T17:27:04.562 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.562 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbdbc51e:::benchmark_data_smithi179_77138_object124025:head# 2024-02-09T17:27:04.562 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.562 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbdbe435:::benchmark_data_smithi179_77138_object83288:head# 2024-02-09T17:27:04.562 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.562 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbdc6c24:::benchmark_data_smithi179_77138_object23775:head# 2024-02-09T17:27:04.562 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.562 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbdca4fa:::benchmark_data_smithi179_77138_object100350:head# 2024-02-09T17:27:04.562 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.562 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbde9a47:::benchmark_data_smithi179_77138_object75204:head# 2024-02-09T17:27:04.562 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.563 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbdf0a37:::benchmark_data_smithi179_77138_object129674:head# 2024-02-09T17:27:04.563 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.563 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbe050e0:::benchmark_data_smithi179_77138_object90550:head# 2024-02-09T17:27:04.563 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.563 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbe16d4a:::benchmark_data_smithi179_77138_object35277:head# 2024-02-09T17:27:04.563 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.563 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbe16f57:::benchmark_data_smithi179_77138_object105969:head# 2024-02-09T17:27:04.563 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.563 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbe1ac0d:::benchmark_data_smithi179_77138_object28141:head# 2024-02-09T17:27:04.563 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.563 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbe1b02c:::benchmark_data_smithi179_77138_object124940:head# 2024-02-09T17:27:04.563 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.563 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbe1cd4c:::benchmark_data_smithi179_77138_object38397:head# 2024-02-09T17:27:04.563 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.564 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbe1e129:::benchmark_data_smithi179_77138_object122078:head# 2024-02-09T17:27:04.564 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.564 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbe24eb4:::benchmark_data_smithi179_77138_object67790:head# 2024-02-09T17:27:04.564 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.564 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbe2a89a:::benchmark_data_smithi179_77138_object112981:head# 2024-02-09T17:27:04.564 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.564 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbe338b9:::benchmark_data_smithi179_77138_object20849:head# 2024-02-09T17:27:04.564 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.564 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbe34408:::benchmark_data_smithi179_77138_object45666:head# 2024-02-09T17:27:04.564 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.564 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbe37af8:::benchmark_data_smithi179_77138_object107877:head# 2024-02-09T17:27:04.564 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.564 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbe37df9:::benchmark_data_smithi179_77138_object103030:head# 2024-02-09T17:27:04.565 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.565 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbe57402:::benchmark_data_smithi179_77138_object75792:head# 2024-02-09T17:27:04.565 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.565 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbe69cf8:::benchmark_data_smithi179_77138_object39334:head# 2024-02-09T17:27:04.565 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.565 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbe75653:::benchmark_data_smithi179_77138_object24741:head# 2024-02-09T17:27:04.565 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.565 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbe786dc:::benchmark_data_smithi179_77138_object85326:head# 2024-02-09T17:27:04.565 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.565 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbe7cce3:::benchmark_data_smithi179_77138_object69779:head# 2024-02-09T17:27:04.565 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.565 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbe8818a:::benchmark_data_smithi179_77138_object81308:head# 2024-02-09T17:27:04.565 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.565 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbe95200:::benchmark_data_smithi179_77138_object36337:head# 2024-02-09T17:27:04.566 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.566 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbe9a00e:::benchmark_data_smithi179_77138_object39243:head# 2024-02-09T17:27:04.566 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.566 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbeba4ef:::benchmark_data_smithi179_77138_object30001:head# 2024-02-09T17:27:04.566 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.566 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbebf16f:::benchmark_data_smithi179_77138_object34071:head# 2024-02-09T17:27:04.566 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.566 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbec945e:::benchmark_data_smithi179_77138_object126790:head# 2024-02-09T17:27:04.566 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.566 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbecc1b2:::benchmark_data_smithi179_77138_object98682:head# 2024-02-09T17:27:04.566 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.566 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbed2c13:::benchmark_data_smithi179_77138_object130869:head# 2024-02-09T17:27:04.566 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.567 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbed3170:::benchmark_data_smithi179_77138_object112559:head# 2024-02-09T17:27:04.567 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.567 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbed4b28:::benchmark_data_smithi179_77138_object75315:head# 2024-02-09T17:27:04.567 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.567 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbed77a2:::benchmark_data_smithi179_77138_object53275:head# 2024-02-09T17:27:04.567 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.567 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbf00f31:::benchmark_data_smithi179_77138_object76815:head# 2024-02-09T17:27:04.567 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.567 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbf05889:::benchmark_data_smithi179_77138_object76893:head# 2024-02-09T17:27:04.567 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.567 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbf07a8a:::benchmark_data_smithi179_77138_object83246:head# 2024-02-09T17:27:04.567 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.567 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbf09ce8:::benchmark_data_smithi179_77138_object54100:head# 2024-02-09T17:27:04.567 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.568 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbf0dec3:::benchmark_data_smithi179_77138_object76191:head# 2024-02-09T17:27:04.568 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.568 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbf0e2da:::benchmark_data_smithi179_77138_object121110:head# 2024-02-09T17:27:04.568 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.568 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbf2c578:::benchmark_data_smithi179_77138_object35979:head# 2024-02-09T17:27:04.568 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.568 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbf364a2:::benchmark_data_smithi179_77138_object60033:head# 2024-02-09T17:27:04.568 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.568 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbf586e8:::benchmark_data_smithi179_77138_object97723:head# 2024-02-09T17:27:04.568 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.568 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbf6746a:::benchmark_data_smithi179_77138_object38598:head# 2024-02-09T17:27:04.568 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.568 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbf780bb:::benchmark_data_smithi179_77138_object75113:head# 2024-02-09T17:27:04.569 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.569 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbf7ad09:::benchmark_data_smithi179_77138_object69076:head# 2024-02-09T17:27:04.569 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.569 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbf98b77:::benchmark_data_smithi179_77138_object100422:head# 2024-02-09T17:27:04.569 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.569 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbf9ed6e:::benchmark_data_smithi179_77138_object95893:head# 2024-02-09T17:27:04.569 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.569 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbfb42a4:::benchmark_data_smithi179_77138_object50229:head# 2024-02-09T17:27:04.569 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.569 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbfbf89a:::benchmark_data_smithi179_77138_object116933:head# 2024-02-09T17:27:04.569 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.569 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbfc4e35:::benchmark_data_smithi179_77138_object30081:head# 2024-02-09T17:27:04.569 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.569 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbfcee22:::benchmark_data_smithi179_77138_object122824:head# 2024-02-09T17:27:04.570 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.570 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbfd0a50:::benchmark_data_smithi179_77138_object89754:head# 2024-02-09T17:27:04.570 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.570 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbfef18d:::benchmark_data_smithi179_77138_object102992:head# 2024-02-09T17:27:04.570 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.570 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fbff48c8:::benchmark_data_smithi179_77138_object44342:head# 2024-02-09T17:27:04.570 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.570 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc0027fe:::benchmark_data_smithi179_77138_object25198:head# 2024-02-09T17:27:04.570 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.570 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc00f687:::benchmark_data_smithi179_77138_object132649:head# 2024-02-09T17:27:04.570 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.570 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc013d6f:::benchmark_data_smithi179_77138_object76744:head# 2024-02-09T17:27:04.570 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.571 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc014913:::benchmark_data_smithi179_77138_object61593:head# 2024-02-09T17:27:04.571 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.571 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc014d32:::benchmark_data_smithi179_77138_object31368:head# 2024-02-09T17:27:04.571 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.571 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc0180d2:::benchmark_data_smithi179_77138_object121396:head# 2024-02-09T17:27:04.571 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.571 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc020487:::benchmark_data_smithi179_77138_object76547:head# 2024-02-09T17:27:04.571 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.571 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc03e97b:::benchmark_data_smithi179_77138_object125031:head# 2024-02-09T17:27:04.571 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.571 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc03f2fa:::benchmark_data_smithi179_77138_object75241:head# 2024-02-09T17:27:04.571 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.571 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc05a2cb:::benchmark_data_smithi179_77138_object110184:head# 2024-02-09T17:27:04.571 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.572 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc05abdb:::benchmark_data_smithi179_77138_object97878:head# 2024-02-09T17:27:04.572 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.572 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc07b443:::benchmark_data_smithi179_77138_object119501:head# 2024-02-09T17:27:04.572 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.572 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc087a0d:::benchmark_data_smithi179_77138_object71629:head# 2024-02-09T17:27:04.572 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.572 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc088462:::benchmark_data_smithi179_77138_object129079:head# 2024-02-09T17:27:04.572 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.572 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc08d8d4:::benchmark_data_smithi179_77138_object99242:head# 2024-02-09T17:27:04.572 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.572 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc08de54:::benchmark_data_smithi179_77138_object115965:head# 2024-02-09T17:27:04.572 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.573 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc09b3f4:::benchmark_data_smithi179_77138_object122503:head# 2024-02-09T17:27:04.573 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.573 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc0a731e:::benchmark_data_smithi179_77138_object38180:head# 2024-02-09T17:27:04.573 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.573 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc0b3e58:::benchmark_data_smithi179_77138_object51422:head# 2024-02-09T17:27:04.573 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.573 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc0b9db3:::benchmark_data_smithi179_77138_object56939:head# 2024-02-09T17:27:04.573 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.573 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc0ba302:::benchmark_data_smithi179_77138_object91737:head# 2024-02-09T17:27:04.573 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.573 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc0c52ce:::benchmark_data_smithi179_77138_object41006:head# 2024-02-09T17:27:04.573 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.573 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc0d03db:::benchmark_data_smithi179_77138_object28644:head# 2024-02-09T17:27:04.574 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.574 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc0d3ba3:::benchmark_data_smithi179_77138_object39688:head# 2024-02-09T17:27:04.574 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.574 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc0d64a3:::benchmark_data_smithi179_77138_object99675:head# 2024-02-09T17:27:04.574 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.574 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc0dd8f0:::benchmark_data_smithi179_77138_object118251:head# 2024-02-09T17:27:04.574 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.574 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc0ea31d:::benchmark_data_smithi179_77138_object74109:head# 2024-02-09T17:27:04.574 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.574 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc104995:::benchmark_data_smithi179_77138_object43932:head# 2024-02-09T17:27:04.574 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.574 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc108d9a:::benchmark_data_smithi179_77138_object110123:head# 2024-02-09T17:27:04.575 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.575 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc1151c0:::benchmark_data_smithi179_77138_object80853:head# 2024-02-09T17:27:04.575 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.575 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc119d00:::benchmark_data_smithi179_77138_object81933:head# 2024-02-09T17:27:04.575 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.575 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc11f7df:::benchmark_data_smithi179_77138_object80409:head# 2024-02-09T17:27:04.575 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.575 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc126558:::benchmark_data_smithi179_77138_object125662:head# 2024-02-09T17:27:04.575 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.575 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc126a18:::benchmark_data_smithi179_77138_object53023:head# 2024-02-09T17:27:04.575 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.575 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc12717e:::benchmark_data_smithi179_77138_object94367:head# 2024-02-09T17:27:04.575 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.575 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc12862b:::benchmark_data_smithi179_77138_object30259:head# 2024-02-09T17:27:04.576 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.576 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc12ab02:::benchmark_data_smithi179_77138_object73786:head# 2024-02-09T17:27:04.576 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.576 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc12b8af:::benchmark_data_smithi179_77138_object71890:head# 2024-02-09T17:27:04.576 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.576 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc1320d5:::benchmark_data_smithi179_77138_object54228:head# 2024-02-09T17:27:04.576 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.576 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc14278b:::benchmark_data_smithi179_77138_object23916:head# 2024-02-09T17:27:04.576 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.576 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc14dc7f:::benchmark_data_smithi179_77138_object73972:head# 2024-02-09T17:27:04.576 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.576 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc14e70b:::benchmark_data_smithi179_77138_object85656:head# 2024-02-09T17:27:04.576 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.577 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc15ab01:::benchmark_data_smithi179_77138_object124102:head# 2024-02-09T17:27:04.577 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.577 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc1612e0:::benchmark_data_smithi179_77138_object24597:head# 2024-02-09T17:27:04.577 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.577 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc17dabb:::benchmark_data_smithi179_77138_object69010:head# 2024-02-09T17:27:04.577 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.577 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc180cff:::benchmark_data_smithi179_77138_object123844:head# 2024-02-09T17:27:04.577 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.577 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc18287a:::benchmark_data_smithi179_77138_object20393:head# 2024-02-09T17:27:04.577 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.577 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc182b09:::benchmark_data_smithi179_77138_object110481:head# 2024-02-09T17:27:04.577 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.577 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc18b15b:::benchmark_data_smithi179_77138_object91976:head# 2024-02-09T17:27:04.578 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.578 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc191cfe:::benchmark_data_smithi179_77138_object91385:head# 2024-02-09T17:27:04.578 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.578 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc193a30:::benchmark_data_smithi179_77138_object23200:head# 2024-02-09T17:27:04.578 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.578 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc199006:::benchmark_data_smithi179_77138_object84601:head# 2024-02-09T17:27:04.578 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.578 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc1a1601:::benchmark_data_smithi179_77138_object47155:head# 2024-02-09T17:27:04.578 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.578 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc1a8b93:::benchmark_data_smithi179_77138_object108384:head# 2024-02-09T17:27:04.578 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.578 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc1bad91:::benchmark_data_smithi179_77138_object63109:head# 2024-02-09T17:27:04.578 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.579 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc1dcb11:::benchmark_data_smithi179_77138_object22699:head# 2024-02-09T17:27:04.579 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.579 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc1e31dc:::benchmark_data_smithi179_77138_object123501:head# 2024-02-09T17:27:04.579 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.579 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc1e6e88:::benchmark_data_smithi179_77138_object118003:head# 2024-02-09T17:27:04.579 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.579 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc1f9014:::benchmark_data_smithi179_77138_object71334:head# 2024-02-09T17:27:04.579 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.579 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc1f9051:::benchmark_data_smithi179_77138_object119480:head# 2024-02-09T17:27:04.579 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.579 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc1fd953:::benchmark_data_smithi179_77138_object48672:head# 2024-02-09T17:27:04.579 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.579 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc21ee78:::benchmark_data_smithi179_77138_object120841:head# 2024-02-09T17:27:04.579 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.580 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc21f99b:::benchmark_data_smithi179_77138_object92913:head# 2024-02-09T17:27:04.580 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.580 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc2262ba:::benchmark_data_smithi179_77138_object96057:head# 2024-02-09T17:27:04.580 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.580 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc22ee65:::benchmark_data_smithi179_77138_object44347:head# 2024-02-09T17:27:04.580 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.580 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc2305f7:::benchmark_data_smithi179_77138_object119568:head# 2024-02-09T17:27:04.580 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.580 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc241e61:::benchmark_data_smithi179_77138_object33697:head# 2024-02-09T17:27:04.580 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.580 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc2498e5:::benchmark_data_smithi179_77138_object90622:head# 2024-02-09T17:27:04.580 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.580 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc24cbca:::benchmark_data_smithi179_77138_object69149:head# 2024-02-09T17:27:04.581 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.581 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc2558f5:::benchmark_data_smithi179_77138_object61138:head# 2024-02-09T17:27:04.581 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.581 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc258371:::benchmark_data_smithi179_77138_object79428:head# 2024-02-09T17:27:04.581 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.581 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc25937a:::benchmark_data_smithi179_77138_object26185:head# 2024-02-09T17:27:04.581 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.581 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc25f899:::benchmark_data_smithi179_77138_object110295:head# 2024-02-09T17:27:04.581 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.582 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:04.582 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:27:04.583 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:27:04.583 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:27:04.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:04.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:27:04.609 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc2607e3:::benchmark_data_smithi179_77138_object59673:head# 2024-02-09T17:27:04.609 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.609 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc263708:::benchmark_data_smithi179_77138_object105370:head# 2024-02-09T17:27:04.610 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.610 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc2784d9:::benchmark_data_smithi179_77138_object74328:head# 2024-02-09T17:27:04.610 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.610 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc278c64:::benchmark_data_smithi179_77138_object59785:head# 2024-02-09T17:27:04.610 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.610 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc288e92:::benchmark_data_smithi179_77138_object121799:head# 2024-02-09T17:27:04.610 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.610 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc291f37:::benchmark_data_smithi179_77138_object128797:head# 2024-02-09T17:27:04.610 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.610 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc2936c4:::benchmark_data_smithi179_77138_object108848:head# 2024-02-09T17:27:04.610 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.610 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc29a191:::benchmark_data_smithi179_77138_object114760:head# 2024-02-09T17:27:04.610 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.611 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc29cf61:::benchmark_data_smithi179_77138_object69757:head# 2024-02-09T17:27:04.611 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.611 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc2bfff6:::benchmark_data_smithi179_77138_object63661:head# 2024-02-09T17:27:04.611 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.611 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc2c07c3:::benchmark_data_smithi179_77138_object78775:head# 2024-02-09T17:27:04.611 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.611 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc2c2b01:::benchmark_data_smithi179_77138_object84978:head# 2024-02-09T17:27:04.611 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.611 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc2c8305:::benchmark_data_smithi179_77138_object94626:head# 2024-02-09T17:27:04.611 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.611 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc2c983f:::benchmark_data_smithi179_77138_object31352:head# 2024-02-09T17:27:04.611 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.611 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc2d5d1a:::benchmark_data_smithi179_77138_object131850:head# 2024-02-09T17:27:04.611 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.612 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc2ec06e:::benchmark_data_smithi179_77138_object128359:head# 2024-02-09T17:27:04.612 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.612 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc30457b:::benchmark_data_smithi179_77138_object126480:head# 2024-02-09T17:27:04.612 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.612 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc306e1a:::benchmark_data_smithi179_77138_object37383:head# 2024-02-09T17:27:04.612 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.612 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc307788:::benchmark_data_smithi179_77138_object116736:head# 2024-02-09T17:27:04.612 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.612 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc3088a0:::benchmark_data_smithi179_77138_object75463:head# 2024-02-09T17:27:04.612 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.612 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc3095ae:::benchmark_data_smithi179_77138_object65345:head# 2024-02-09T17:27:04.612 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.612 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc309c5f:::benchmark_data_smithi179_77138_object60149:head# 2024-02-09T17:27:04.613 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.613 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc30d726:::benchmark_data_smithi179_77138_object61277:head# 2024-02-09T17:27:04.613 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.613 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc31797e:::benchmark_data_smithi179_77138_object51850:head# 2024-02-09T17:27:04.613 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.613 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc327703:::benchmark_data_smithi179_77138_object53715:head# 2024-02-09T17:27:04.613 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.613 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc335498:::benchmark_data_smithi179_77138_object61083:head# 2024-02-09T17:27:04.613 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.613 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc344859:::benchmark_data_smithi179_77138_object97227:head# 2024-02-09T17:27:04.613 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.613 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc35a011:::benchmark_data_smithi179_77138_object65697:head# 2024-02-09T17:27:04.613 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.613 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc35d463:::benchmark_data_smithi179_77138_object61810:head# 2024-02-09T17:27:04.614 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.614 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc377ffe:::benchmark_data_smithi179_77138_object87110:head# 2024-02-09T17:27:04.614 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.614 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc37eaa6:::benchmark_data_smithi179_77138_object67493:head# 2024-02-09T17:27:04.614 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.614 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc381151:::benchmark_data_smithi179_77138_object116233:head# 2024-02-09T17:27:04.614 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.614 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc38e4e5:::benchmark_data_smithi179_77138_object42567:head# 2024-02-09T17:27:04.614 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.614 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc391198:::benchmark_data_smithi179_77138_object55706:head# 2024-02-09T17:27:04.614 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.614 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc3975fc:::benchmark_data_smithi179_77138_object36668:head# 2024-02-09T17:27:04.614 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.615 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc39e4fc:::benchmark_data_smithi179_77138_object111284:head# 2024-02-09T17:27:04.615 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.615 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc39efe5:::benchmark_data_smithi179_77138_object96207:head# 2024-02-09T17:27:04.615 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.615 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc3a5b7c:::benchmark_data_smithi179_77138_object104658:head# 2024-02-09T17:27:04.615 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.615 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc3a8f8e:::benchmark_data_smithi179_77138_object44772:head# 2024-02-09T17:27:04.615 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.615 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc3b13e8:::benchmark_data_smithi179_77138_object50411:head# 2024-02-09T17:27:04.615 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.615 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc3b1eb8:::benchmark_data_smithi179_77138_object133254:head# 2024-02-09T17:27:04.615 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.615 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc3b8096:::benchmark_data_smithi179_77138_object107985:head# 2024-02-09T17:27:04.615 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.615 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc3bdb4a:::benchmark_data_smithi179_77138_object126382:head# 2024-02-09T17:27:04.616 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.616 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc3ce435:::benchmark_data_smithi179_77138_object112146:head# 2024-02-09T17:27:04.616 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.616 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc3ddabd:::benchmark_data_smithi179_77138_object83493:head# 2024-02-09T17:27:04.616 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.616 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc3dfdc4:::benchmark_data_smithi179_77138_object50787:head# 2024-02-09T17:27:04.616 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.616 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc3f2367:::benchmark_data_smithi179_77138_object103825:head# 2024-02-09T17:27:04.616 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.616 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc3f57f0:::benchmark_data_smithi179_77138_object46680:head# 2024-02-09T17:27:04.616 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.616 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc40804b:::benchmark_data_smithi179_77138_object42568:head# 2024-02-09T17:27:04.616 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.617 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc4141b5:::benchmark_data_smithi179_77138_object37347:head# 2024-02-09T17:27:04.617 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.617 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc419dc4:::benchmark_data_smithi179_77138_object98061:head# 2024-02-09T17:27:04.617 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.617 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc43723d:::benchmark_data_smithi179_77138_object117811:head# 2024-02-09T17:27:04.617 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.617 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc43a3c9:::benchmark_data_smithi179_77138_object132346:head# 2024-02-09T17:27:04.617 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.617 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc451c12:::benchmark_data_smithi179_77138_object82318:head# 2024-02-09T17:27:04.617 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.617 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc45d103:::benchmark_data_smithi179_77138_object51235:head# 2024-02-09T17:27:04.617 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.617 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc45d78e:::benchmark_data_smithi179_77138_object21608:head# 2024-02-09T17:27:04.617 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.618 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc472f5b:::benchmark_data_smithi179_77138_object30848:head# 2024-02-09T17:27:04.618 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.618 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc47fcc5:::benchmark_data_smithi179_77138_object28824:head# 2024-02-09T17:27:04.618 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.618 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc4c2a6f:::benchmark_data_smithi179_77138_object54706:head# 2024-02-09T17:27:04.618 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.618 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc4cb367:::benchmark_data_smithi179_77138_object111325:head# 2024-02-09T17:27:04.618 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.618 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc4d68bc:::benchmark_data_smithi179_77138_object98808:head# 2024-02-09T17:27:04.618 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.618 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc4dc426:::benchmark_data_smithi179_77138_object121983:head# 2024-02-09T17:27:04.618 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.618 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc4eac35:::benchmark_data_smithi179_77138_object73834:head# 2024-02-09T17:27:04.618 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.619 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc4f1dca:::benchmark_data_smithi179_77138_object108943:head# 2024-02-09T17:27:04.619 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.619 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc4f28a8:::benchmark_data_smithi179_77138_object66947:head# 2024-02-09T17:27:04.619 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.619 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc4f4774:::benchmark_data_smithi179_77138_object114293:head# 2024-02-09T17:27:04.619 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.619 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc4f7367:::benchmark_data_smithi179_77138_object45803:head# 2024-02-09T17:27:04.619 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.619 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc4f7c9d:::benchmark_data_smithi179_77138_object120887:head# 2024-02-09T17:27:04.619 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.619 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc4fca18:::benchmark_data_smithi179_77138_object111372:head# 2024-02-09T17:27:04.619 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.619 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc51106b:::benchmark_data_smithi179_77138_object55683:head# 2024-02-09T17:27:04.619 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.620 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc516376:::benchmark_data_smithi179_77138_object67548:head# 2024-02-09T17:27:04.620 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.620 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc520557:::benchmark_data_smithi179_77138_object107719:head# 2024-02-09T17:27:04.620 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.620 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc523f51:::benchmark_data_smithi179_77138_object119734:head# 2024-02-09T17:27:04.620 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.620 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc524769:::benchmark_data_smithi179_77138_object91110:head# 2024-02-09T17:27:04.620 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.620 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc52503a:::benchmark_data_smithi179_77138_object22476:head# 2024-02-09T17:27:04.620 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.620 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc52e4ff:::benchmark_data_smithi179_77138_object106545:head# 2024-02-09T17:27:04.620 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.620 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc5302ee:::benchmark_data_smithi179_77138_object62889:head# 2024-02-09T17:27:04.621 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.621 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc54532e:::benchmark_data_smithi179_77138_object72809:head# 2024-02-09T17:27:04.621 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.621 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc54b9c3:::benchmark_data_smithi179_77138_object118390:head# 2024-02-09T17:27:04.621 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.621 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc555720:::benchmark_data_smithi179_77138_object108218:head# 2024-02-09T17:27:04.621 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.621 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc560e93:::benchmark_data_smithi179_77138_object30820:head# 2024-02-09T17:27:04.621 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.621 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc5659d7:::benchmark_data_smithi179_77138_object34509:head# 2024-02-09T17:27:04.621 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.621 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc576470:::benchmark_data_smithi179_77138_object65048:head# 2024-02-09T17:27:04.621 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.621 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc589759:::benchmark_data_smithi179_77138_object111484:head# 2024-02-09T17:27:04.622 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.622 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc5a565d:::benchmark_data_smithi179_77138_object58277:head# 2024-02-09T17:27:04.622 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.622 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc5aa845:::benchmark_data_smithi179_77138_object83818:head# 2024-02-09T17:27:04.622 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.622 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc5c65fd:::benchmark_data_smithi179_77138_object42263:head# 2024-02-09T17:27:04.622 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.622 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc5c775d:::benchmark_data_smithi179_77138_object92178:head# 2024-02-09T17:27:04.622 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.622 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc5c8350:::benchmark_data_smithi179_77138_object72594:head# 2024-02-09T17:27:04.622 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.622 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc5caebb:::benchmark_data_smithi179_77138_object107608:head# 2024-02-09T17:27:04.622 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.623 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc5d313f:::benchmark_data_smithi179_77138_object43793:head# 2024-02-09T17:27:04.623 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.623 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc5e8991:::benchmark_data_smithi179_77138_object45856:head# 2024-02-09T17:27:04.623 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.623 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc609835:::benchmark_data_smithi179_77138_object123512:head# 2024-02-09T17:27:04.623 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.623 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc60df9f:::benchmark_data_smithi179_77138_object62958:head# 2024-02-09T17:27:04.623 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.623 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc611a35:::benchmark_data_smithi179_77138_object89637:head# 2024-02-09T17:27:04.623 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.623 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc6323fa:::benchmark_data_smithi179_77138_object113547:head# 2024-02-09T17:27:04.623 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.623 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc646018:::benchmark_data_smithi179_77138_object85255:head# 2024-02-09T17:27:04.624 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.624 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc64c569:::benchmark_data_smithi179_77138_object55323:head# 2024-02-09T17:27:04.624 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.624 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc64f89a:::benchmark_data_smithi179_77138_object20536:head# 2024-02-09T17:27:04.624 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.624 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc653c42:::benchmark_data_smithi179_77138_object132875:head# 2024-02-09T17:27:04.624 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.624 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc65c335:::benchmark_data_smithi179_77138_object77171:head# 2024-02-09T17:27:04.624 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.624 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc65e320:::benchmark_data_smithi179_77138_object94715:head# 2024-02-09T17:27:04.624 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.624 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc665cd6:::benchmark_data_smithi179_77138_object38483:head# 2024-02-09T17:27:04.624 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.625 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc66968e:::benchmark_data_smithi179_77138_object29596:head# 2024-02-09T17:27:04.625 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.625 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc6772d6:::benchmark_data_smithi179_77138_object24373:head# 2024-02-09T17:27:04.625 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.625 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc68874f:::benchmark_data_smithi179_77138_object98440:head# 2024-02-09T17:27:04.625 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.625 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc696c94:::benchmark_data_smithi179_77138_object112731:head# 2024-02-09T17:27:04.625 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.625 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc6ab1e2:::benchmark_data_smithi179_77138_object102816:head# 2024-02-09T17:27:04.625 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.625 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc6b7e7e:::benchmark_data_smithi179_77138_object68856:head# 2024-02-09T17:27:04.625 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.625 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc6bc308:::benchmark_data_smithi179_77138_object83393:head# 2024-02-09T17:27:04.625 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.626 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc6c1ff6:::benchmark_data_smithi179_77138_object47820:head# 2024-02-09T17:27:04.626 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.626 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc6cdbe6:::benchmark_data_smithi179_77138_object56663:head# 2024-02-09T17:27:04.626 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.626 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc6d0219:::benchmark_data_smithi179_77138_object64902:head# 2024-02-09T17:27:04.626 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.626 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc6d170d:::benchmark_data_smithi179_77138_object93886:head# 2024-02-09T17:27:04.626 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.626 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc6e17fe:::benchmark_data_smithi179_77138_object71542:head# 2024-02-09T17:27:04.626 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.626 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc6fd654:::benchmark_data_smithi179_77138_object49099:head# 2024-02-09T17:27:04.626 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.626 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc70de7a:::benchmark_data_smithi179_77138_object69712:head# 2024-02-09T17:27:04.626 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.627 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc7178e3:::benchmark_data_smithi179_77138_object127247:head# 2024-02-09T17:27:04.627 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.627 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc71a946:::benchmark_data_smithi179_77138_object44920:head# 2024-02-09T17:27:04.627 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.627 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc725f99:::benchmark_data_smithi179_77138_object44567:head# 2024-02-09T17:27:04.627 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.627 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc738ad5:::benchmark_data_smithi179_77138_object108076:head# 2024-02-09T17:27:04.627 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.627 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc74fdb8:::benchmark_data_smithi179_77138_object95194:head# 2024-02-09T17:27:04.627 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.627 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc751135:::benchmark_data_smithi179_77138_object48108:head# 2024-02-09T17:27:04.627 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.627 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc75a813:::benchmark_data_smithi179_77138_object33050:head# 2024-02-09T17:27:04.627 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.627 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc7954cd:::benchmark_data_smithi179_77138_object25384:head# 2024-02-09T17:27:04.628 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.628 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc7a4899:::benchmark_data_smithi179_77138_object63439:head# 2024-02-09T17:27:04.628 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.628 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc7b15e7:::benchmark_data_smithi179_77138_object40850:head# 2024-02-09T17:27:04.628 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.628 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc7b92ff:::benchmark_data_smithi179_77138_object27482:head# 2024-02-09T17:27:04.628 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.628 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc7bb0d3:::benchmark_data_smithi179_77138_object25795:head# 2024-02-09T17:27:04.628 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.628 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc7c5437:::benchmark_data_smithi179_77138_object69296:head# 2024-02-09T17:27:04.628 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.628 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc7e20da:::benchmark_data_smithi179_77138_object68101:head# 2024-02-09T17:27:04.628 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.629 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc7ee92e:::benchmark_data_smithi179_77138_object85979:head# 2024-02-09T17:27:04.629 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.629 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc7ee958:::benchmark_data_smithi179_77138_object54469:head# 2024-02-09T17:27:04.629 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.629 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc8040f4:::benchmark_data_smithi179_77138_object29767:head# 2024-02-09T17:27:04.629 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.629 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc804375:::benchmark_data_smithi179_77138_object57112:head# 2024-02-09T17:27:04.629 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.629 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc80bd55:::benchmark_data_smithi179_77138_object99553:head# 2024-02-09T17:27:04.629 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.629 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc80c2c9:::benchmark_data_smithi179_77138_object25412:head# 2024-02-09T17:27:04.629 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.629 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc80c8b5:::benchmark_data_smithi179_77138_object117542:head# 2024-02-09T17:27:04.630 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.630 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc833a08:::benchmark_data_smithi179_77138_object131309:head# 2024-02-09T17:27:04.630 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.630 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc83ba1a:::benchmark_data_smithi179_77138_object119661:head# 2024-02-09T17:27:04.630 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.630 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc83c8ce:::benchmark_data_smithi179_77138_object96430:head# 2024-02-09T17:27:04.630 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.630 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc83ebe7:::benchmark_data_smithi179_77138_object34614:head# 2024-02-09T17:27:04.630 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.630 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc84d1d0:::benchmark_data_smithi179_77138_object103690:head# 2024-02-09T17:27:04.630 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.630 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc858f60:::benchmark_data_smithi179_77138_object111922:head# 2024-02-09T17:27:04.630 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.630 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc85fa07:::benchmark_data_smithi179_77138_object97497:head# 2024-02-09T17:27:04.631 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.631 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc864847:::benchmark_data_smithi179_77138_object27585:head# 2024-02-09T17:27:04.631 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.631 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc86aa58:::benchmark_data_smithi179_77138_object124075:head# 2024-02-09T17:27:04.631 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.631 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc87188e:::benchmark_data_smithi179_77138_object57682:head# 2024-02-09T17:27:04.631 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.631 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc887d83:::benchmark_data_smithi179_77138_object118910:head# 2024-02-09T17:27:04.631 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.631 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc889f1d:::benchmark_data_smithi179_77138_object74117:head# 2024-02-09T17:27:04.631 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.631 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc893259:::benchmark_data_smithi179_77138_object96152:head# 2024-02-09T17:27:04.631 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.632 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc897529:::benchmark_data_smithi179_77138_object112266:head# 2024-02-09T17:27:04.632 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.632 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc8aa698:::benchmark_data_smithi179_77138_object91917:head# 2024-02-09T17:27:04.632 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.632 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc8cae61:::benchmark_data_smithi179_77138_object35327:head# 2024-02-09T17:27:04.632 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.632 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc8d4df6:::benchmark_data_smithi179_77138_object20057:head# 2024-02-09T17:27:04.632 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.632 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc8e250b:::benchmark_data_smithi179_77138_object88777:head# 2024-02-09T17:27:04.632 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.632 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc8f577d:::benchmark_data_smithi179_77138_object104452:head# 2024-02-09T17:27:04.632 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.632 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc8f7558:::benchmark_data_smithi179_77138_object61950:head# 2024-02-09T17:27:04.632 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.633 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc8fbbf0:::benchmark_data_smithi179_77138_object93035:head# 2024-02-09T17:27:04.633 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.633 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc908cfa:::benchmark_data_smithi179_77138_object72365:head# 2024-02-09T17:27:04.633 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.633 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc910f87:::benchmark_data_smithi179_77138_object68798:head# 2024-02-09T17:27:04.633 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.633 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc916e7a:::benchmark_data_smithi179_77138_object47856:head# 2024-02-09T17:27:04.633 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.633 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc91cff6:::benchmark_data_smithi179_77138_object68550:head# 2024-02-09T17:27:04.633 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.633 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc92755d:::benchmark_data_smithi179_77138_object115593:head# 2024-02-09T17:27:04.633 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.633 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc933cc5:::benchmark_data_smithi179_77138_object27535:head# 2024-02-09T17:27:04.633 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.634 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc9347d9:::benchmark_data_smithi179_77138_object127469:head# 2024-02-09T17:27:04.634 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.634 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc9421a8:::benchmark_data_smithi179_77138_object98688:head# 2024-02-09T17:27:04.634 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.634 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc94620c:::benchmark_data_smithi179_77138_object67165:head# 2024-02-09T17:27:04.634 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.634 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc94bca2:::benchmark_data_smithi179_77138_object45493:head# 2024-02-09T17:27:04.634 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.634 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc950d68:::benchmark_data_smithi179_77138_object82130:head# 2024-02-09T17:27:04.634 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.634 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc9613fe:::benchmark_data_smithi179_77138_object89068:head# 2024-02-09T17:27:04.634 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.634 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc967447:::benchmark_data_smithi179_77138_object57912:head# 2024-02-09T17:27:04.634 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.634 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc96d4c1:::benchmark_data_smithi179_77138_object117370:head# 2024-02-09T17:27:04.635 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.635 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc96e1c6:::benchmark_data_smithi179_77138_object29686:head# 2024-02-09T17:27:04.635 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.635 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc96ec62:::benchmark_data_smithi179_77138_object33420:head# 2024-02-09T17:27:04.635 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.635 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc970819:::benchmark_data_smithi179_77138_object69815:head# 2024-02-09T17:27:04.635 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.635 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc981046:::benchmark_data_smithi179_77138_object96747:head# 2024-02-09T17:27:04.635 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.635 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc983410:::benchmark_data_smithi179_77138_object124056:head# 2024-02-09T17:27:04.635 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.635 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc98373c:::benchmark_data_smithi179_77138_object21800:head# 2024-02-09T17:27:04.635 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.635 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc984fca:::benchmark_data_smithi179_77138_object52326:head# 2024-02-09T17:27:04.636 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.636 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc9943a9:::benchmark_data_smithi179_77138_object117269:head# 2024-02-09T17:27:04.636 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.636 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc9b5910:::benchmark_data_smithi179_77138_object124754:head# 2024-02-09T17:27:04.636 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.636 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc9b9b2c:::benchmark_data_smithi179_77138_object65984:head# 2024-02-09T17:27:04.636 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.636 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc9bd8a2:::benchmark_data_smithi179_77138_object55519:head# 2024-02-09T17:27:04.636 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.636 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc9ce3af:::benchmark_data_smithi179_77138_object41233:head# 2024-02-09T17:27:04.636 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.636 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc9d6c77:::benchmark_data_smithi179_77138_object130911:head# 2024-02-09T17:27:04.636 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.636 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc9d80e8:::benchmark_data_smithi179_77138_object86226:head# 2024-02-09T17:27:04.636 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.637 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc9da2c1:::benchmark_data_smithi179_77138_object93368:head# 2024-02-09T17:27:04.637 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.637 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc9e1f63:::benchmark_data_smithi179_77138_object123990:head# 2024-02-09T17:27:04.637 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.637 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fc9e96d3:::benchmark_data_smithi179_77138_object55358:head# 2024-02-09T17:27:04.637 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.637 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fca0fa7f:::benchmark_data_smithi179_77138_object126258:head# 2024-02-09T17:27:04.637 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.637 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fca12f7c:::benchmark_data_smithi179_77138_object69222:head# 2024-02-09T17:27:04.637 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.637 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fca1e516:::benchmark_data_smithi179_77138_object26171:head# 2024-02-09T17:27:04.637 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.637 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fca1ee81:::benchmark_data_smithi179_77138_object128090:head# 2024-02-09T17:27:04.637 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.638 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fca20e11:::benchmark_data_smithi179_77138_object78108:head# 2024-02-09T17:27:04.638 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.638 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fca21401:::benchmark_data_smithi179_77138_object112662:head# 2024-02-09T17:27:04.638 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.638 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fca27242:::benchmark_data_smithi179_77138_object40349:head# 2024-02-09T17:27:04.638 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.638 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fca2bf4f:::benchmark_data_smithi179_77138_object25538:head# 2024-02-09T17:27:04.638 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.638 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fca2f295:::benchmark_data_smithi179_77138_object20852:head# 2024-02-09T17:27:04.638 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.638 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fca32f8f:::benchmark_data_smithi179_77138_object104867:head# 2024-02-09T17:27:04.638 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.638 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fca333ee:::benchmark_data_smithi179_77138_object22168:head# 2024-02-09T17:27:04.638 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.639 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fca3abb4:::benchmark_data_smithi179_77138_object128876:head# 2024-02-09T17:27:04.639 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.639 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fca4cf04:::benchmark_data_smithi179_77138_object123134:head# 2024-02-09T17:27:04.639 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.639 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fca5a325:::benchmark_data_smithi179_77138_object27624:head# 2024-02-09T17:27:04.639 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.639 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fca5ed29:::benchmark_data_smithi179_77138_object22420:head# 2024-02-09T17:27:04.639 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.639 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fca5f9a1:::benchmark_data_smithi179_77138_object125424:head# 2024-02-09T17:27:04.639 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.639 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fca613f1:::benchmark_data_smithi179_77138_object106618:head# 2024-02-09T17:27:04.639 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.639 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fca65cde:::benchmark_data_smithi179_77138_object108105:head# 2024-02-09T17:27:04.639 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.640 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fca6b4d5:::benchmark_data_smithi179_77138_object87756:head# 2024-02-09T17:27:04.640 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.640 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fca6db61:::benchmark_data_smithi179_77138_object43246:head# 2024-02-09T17:27:04.640 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.640 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fca6e4d0:::benchmark_data_smithi179_77138_object45175:head# 2024-02-09T17:27:04.640 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.640 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fca6f5d3:::benchmark_data_smithi179_77138_object75677:head# 2024-02-09T17:27:04.640 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.640 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fca72f12:::benchmark_data_smithi179_77138_object27052:head# 2024-02-09T17:27:04.640 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.640 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fca7f635:::benchmark_data_smithi179_77138_object102829:head# 2024-02-09T17:27:04.640 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.640 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fca8b82d:::benchmark_data_smithi179_77138_object126218:head# 2024-02-09T17:27:04.640 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.640 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fca928f3:::benchmark_data_smithi179_77138_object117337:head# 2024-02-09T17:27:04.641 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.641 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fca95771:::benchmark_data_smithi179_77138_object125581:head# 2024-02-09T17:27:04.641 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.641 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fca9b0b0:::benchmark_data_smithi179_77138_object49211:head# 2024-02-09T17:27:04.641 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.641 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fca9e957:::benchmark_data_smithi179_77138_object103364:head# 2024-02-09T17:27:04.641 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.641 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcaa3b42:::benchmark_data_smithi179_77138_object39969:head# 2024-02-09T17:27:04.641 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.641 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcaa8fce:::benchmark_data_smithi179_77138_object29971:head# 2024-02-09T17:27:04.641 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.641 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcab355d:::benchmark_data_smithi179_77138_object120309:head# 2024-02-09T17:27:04.641 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.642 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcab854e:::benchmark_data_smithi179_77138_object108951:head# 2024-02-09T17:27:04.642 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.642 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcad3c13:::benchmark_data_smithi179_77138_object120084:head# 2024-02-09T17:27:04.642 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.642 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcad698c:::benchmark_data_smithi179_77138_object62705:head# 2024-02-09T17:27:04.642 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.642 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcad6fd0:::benchmark_data_smithi179_77138_object19941:head# 2024-02-09T17:27:04.642 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.642 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcad883e:::benchmark_data_smithi179_77138_object127751:head# 2024-02-09T17:27:04.642 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.642 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcae11ab:::benchmark_data_smithi179_77138_object110256:head# 2024-02-09T17:27:04.642 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.642 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcafffc4:::benchmark_data_smithi179_77138_object109390:head# 2024-02-09T17:27:04.642 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.642 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcb0bea1:::benchmark_data_smithi179_77138_object115510:head# 2024-02-09T17:27:04.643 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.643 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcb11918:::benchmark_data_smithi179_77138_object39067:head# 2024-02-09T17:27:04.643 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.643 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcb154e1:::benchmark_data_smithi179_77138_object105787:head# 2024-02-09T17:27:04.643 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.643 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcb1568a:::benchmark_data_smithi179_77138_object112711:head# 2024-02-09T17:27:04.643 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.643 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcb17ef0:::benchmark_data_smithi179_77138_object48389:head# 2024-02-09T17:27:04.643 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.643 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcb19f33:::benchmark_data_smithi179_77138_object103875:head# 2024-02-09T17:27:04.643 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.643 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcb1b49a:::benchmark_data_smithi179_77138_object45781:head# 2024-02-09T17:27:04.643 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.643 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcb486ab:::benchmark_data_smithi179_77138_object32606:head# 2024-02-09T17:27:04.644 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.644 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcb4ad62:::benchmark_data_smithi179_77138_object102789:head# 2024-02-09T17:27:04.644 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.644 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcb5dd03:::benchmark_data_smithi179_77138_object79126:head# 2024-02-09T17:27:04.644 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.644 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcb6549c:::benchmark_data_smithi179_77138_object92995:head# 2024-02-09T17:27:04.644 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.644 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcb65811:::benchmark_data_smithi179_77138_object49084:head# 2024-02-09T17:27:04.644 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.645 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:27:04.657 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:27:04.665 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcb76d7a:::benchmark_data_smithi179_77138_object113786:head# 2024-02-09T17:27:04.665 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.665 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcb7d130:::benchmark_data_smithi179_77138_object118523:head# 2024-02-09T17:27:04.665 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.665 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcb81a32:::benchmark_data_smithi179_77138_object114856:head# 2024-02-09T17:27:04.665 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.665 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcb826ed:::benchmark_data_smithi179_77138_object109230:head# 2024-02-09T17:27:04.665 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.666 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcb8ed7e:::benchmark_data_smithi179_77138_object68862:head# 2024-02-09T17:27:04.666 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.666 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcb9cfd0:::benchmark_data_smithi179_77138_object116482:head# 2024-02-09T17:27:04.666 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.666 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcba5b9b:::benchmark_data_smithi179_77138_object46780:head# 2024-02-09T17:27:04.666 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.666 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcba8f4f:::benchmark_data_smithi179_77138_object88780:head# 2024-02-09T17:27:04.666 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.666 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcbad4a7:::benchmark_data_smithi179_77138_object107344:head# 2024-02-09T17:27:04.666 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.666 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcbaf166:::benchmark_data_smithi179_77138_object61251:head# 2024-02-09T17:27:04.666 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.666 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcbb2b43:::benchmark_data_smithi179_77138_object92216:head# 2024-02-09T17:27:04.666 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.667 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcbb3a1d:::benchmark_data_smithi179_77138_object42550:head# 2024-02-09T17:27:04.667 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.667 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcbb98ef:::benchmark_data_smithi179_77138_object66746:head# 2024-02-09T17:27:04.667 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.667 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcbc5687:::benchmark_data_smithi179_77138_object78184:head# 2024-02-09T17:27:04.667 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.667 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcbcdff5:::benchmark_data_smithi179_77138_object101628:head# 2024-02-09T17:27:04.667 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.667 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcbd2e42:::benchmark_data_smithi179_77138_object63120:head# 2024-02-09T17:27:04.667 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.667 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcbd4385:::benchmark_data_smithi179_77138_object72105:head# 2024-02-09T17:27:04.667 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.667 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcbd7000:::benchmark_data_smithi179_77138_object74776:head# 2024-02-09T17:27:04.667 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.667 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcbddddd:::benchmark_data_smithi179_77138_object67522:head# 2024-02-09T17:27:04.668 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.668 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcbe06e5:::benchmark_data_smithi179_77138_object39111:head# 2024-02-09T17:27:04.668 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.668 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcbff5db:::benchmark_data_smithi179_77138_object102872:head# 2024-02-09T17:27:04.668 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.668 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcc02088:::benchmark_data_smithi179_77138_object23217:head# 2024-02-09T17:27:04.668 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.668 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcc04f9a:::benchmark_data_smithi179_77138_object104887:head# 2024-02-09T17:27:04.668 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.668 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcc058f0:::benchmark_data_smithi179_77138_object112209:head# 2024-02-09T17:27:04.668 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.668 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcc0d346:::benchmark_data_smithi179_77138_object72442:head# 2024-02-09T17:27:04.669 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.669 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcc0dd08:::benchmark_data_smithi179_77138_object31160:head# 2024-02-09T17:27:04.669 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.669 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcc113c1:::benchmark_data_smithi179_77138_object50685:head# 2024-02-09T17:27:04.669 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.669 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcc11e44:::benchmark_data_smithi179_77138_object29900:head# 2024-02-09T17:27:04.669 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.669 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcc19077:::benchmark_data_smithi179_77138_object123311:head# 2024-02-09T17:27:04.669 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.669 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcc2e243:::benchmark_data_smithi179_77138_object63750:head# 2024-02-09T17:27:04.669 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.669 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcc385ac:::benchmark_data_smithi179_77138_object64231:head# 2024-02-09T17:27:04.669 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.669 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcc41078:::benchmark_data_smithi179_77138_object71614:head# 2024-02-09T17:27:04.670 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.670 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcc71ebc:::benchmark_data_smithi179_77138_object44874:head# 2024-02-09T17:27:04.670 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.670 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcc745c9:::benchmark_data_smithi179_77138_object124747:head# 2024-02-09T17:27:04.670 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.670 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcc7660b:::benchmark_data_smithi179_77138_object123203:head# 2024-02-09T17:27:04.670 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.670 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcc7d16b:::benchmark_data_smithi179_77138_object102101:head# 2024-02-09T17:27:04.670 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.670 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcc90d3c:::benchmark_data_smithi179_77138_object87242:head# 2024-02-09T17:27:04.670 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.670 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcca38cf:::benchmark_data_smithi179_77138_object107189:head# 2024-02-09T17:27:04.671 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.671 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fccaac70:::benchmark_data_smithi179_77138_object101512:head# 2024-02-09T17:27:04.671 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.671 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fccaf54b:::benchmark_data_smithi179_77138_object77276:head# 2024-02-09T17:27:04.671 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.671 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fccc0ef0:::benchmark_data_smithi179_77138_object89351:head# 2024-02-09T17:27:04.671 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.671 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fccc2635:::benchmark_data_smithi179_77138_object82098:head# 2024-02-09T17:27:04.671 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.671 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fccc4ee4:::benchmark_data_smithi179_77138_object37930:head# 2024-02-09T17:27:04.671 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.671 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fccd5e5a:::benchmark_data_smithi179_77138_object105174:head# 2024-02-09T17:27:04.671 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.672 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fccdc4cb:::benchmark_data_smithi179_77138_object22290:head# 2024-02-09T17:27:04.672 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.672 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcce1960:::benchmark_data_smithi179_77138_object40397:head# 2024-02-09T17:27:04.672 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.672 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fccead8b:::benchmark_data_smithi179_77138_object86799:head# 2024-02-09T17:27:04.672 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.672 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fccf76dd:::benchmark_data_smithi179_77138_object55068:head# 2024-02-09T17:27:04.672 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.672 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcd00dda:::benchmark_data_smithi179_77138_object21760:head# 2024-02-09T17:27:04.672 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.672 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcd07f1a:::benchmark_data_smithi179_77138_object51669:head# 2024-02-09T17:27:04.672 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.672 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcd0d964:::benchmark_data_smithi179_77138_object112469:head# 2024-02-09T17:27:04.673 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.673 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcd0ed02:::benchmark_data_smithi179_77138_object53705:head# 2024-02-09T17:27:04.673 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.673 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcd140d4:::benchmark_data_smithi179_77138_object107410:head# 2024-02-09T17:27:04.673 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.673 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcd142dd:::benchmark_data_smithi179_77138_object126628:head# 2024-02-09T17:27:04.673 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.673 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcd146bd:::benchmark_data_smithi179_77138_object50236:head# 2024-02-09T17:27:04.673 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.673 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcd19de3:::benchmark_data_smithi179_77138_object66051:head# 2024-02-09T17:27:04.673 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.673 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcd354c7:::benchmark_data_smithi179_77138_object66667:head# 2024-02-09T17:27:04.673 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.674 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcd3fe39:::benchmark_data_smithi179_77138_object80895:head# 2024-02-09T17:27:04.674 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.674 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcd5600c:::benchmark_data_smithi179_77138_object64256:head# 2024-02-09T17:27:04.674 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.674 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcd6787e:::benchmark_data_smithi179_77138_object63028:head# 2024-02-09T17:27:04.674 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.674 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcd72275:::benchmark_data_smithi179_77138_object20048:head# 2024-02-09T17:27:04.674 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.674 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcd8e6cc:::benchmark_data_smithi179_77138_object63355:head# 2024-02-09T17:27:04.674 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.674 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcd90cf2:::benchmark_data_smithi179_77138_object95982:head# 2024-02-09T17:27:04.674 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.674 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcda1ca8:::benchmark_data_smithi179_77138_object25730:head# 2024-02-09T17:27:04.674 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.675 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcdb5aaf:::benchmark_data_smithi179_77138_object131103:head# 2024-02-09T17:27:04.675 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.675 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcdb93ab:::benchmark_data_smithi179_77138_object94517:head# 2024-02-09T17:27:04.675 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.675 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcdbd126:::benchmark_data_smithi179_77138_object101074:head# 2024-02-09T17:27:04.675 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.675 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcdbf6fd:::benchmark_data_smithi179_77138_object64307:head# 2024-02-09T17:27:04.675 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.675 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcdc4dcc:::benchmark_data_smithi179_77138_object126901:head# 2024-02-09T17:27:04.675 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.675 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcde87da:::benchmark_data_smithi179_77138_object114637:head# 2024-02-09T17:27:04.675 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.675 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcdeeced:::benchmark_data_smithi179_77138_object102091:head# 2024-02-09T17:27:04.675 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.676 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcdf4b7f:::benchmark_data_smithi179_77138_object52868:head# 2024-02-09T17:27:04.676 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.676 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fce01541:::benchmark_data_smithi179_77138_object50228:head# 2024-02-09T17:27:04.676 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.676 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fce06030:::benchmark_data_smithi179_77138_object117591:head# 2024-02-09T17:27:04.676 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.676 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fce06548:::benchmark_data_smithi179_77138_object84633:head# 2024-02-09T17:27:04.676 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.676 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fce0c9a1:::benchmark_data_smithi179_77138_object25999:head# 2024-02-09T17:27:04.676 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.676 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fce204e4:::benchmark_data_smithi179_77138_object19890:head# 2024-02-09T17:27:04.676 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.676 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fce25882:::benchmark_data_smithi179_77138_object131332:head# 2024-02-09T17:27:04.677 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.677 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fce2cd60:::benchmark_data_smithi179_77138_object49981:head# 2024-02-09T17:27:04.677 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.677 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fce346f3:::benchmark_data_smithi179_77138_object60964:head# 2024-02-09T17:27:04.677 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.677 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fce35cdc:::benchmark_data_smithi179_77138_object75805:head# 2024-02-09T17:27:04.677 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.677 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fce3c79d:::benchmark_data_smithi179_77138_object59932:head# 2024-02-09T17:27:04.677 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.677 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fce3e961:::benchmark_data_smithi179_77138_object108208:head# 2024-02-09T17:27:04.677 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.677 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fce3f02b:::benchmark_data_smithi179_77138_object21661:head# 2024-02-09T17:27:04.677 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.678 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fce43084:::benchmark_data_smithi179_77138_object94062:head# 2024-02-09T17:27:04.678 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.678 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fce45ac7:::benchmark_data_smithi179_77138_object103540:head# 2024-02-09T17:27:04.678 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.678 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fce4c963:::benchmark_data_smithi179_77138_object45347:head# 2024-02-09T17:27:04.678 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.678 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fce4f632:::benchmark_data_smithi179_77138_object40823:head# 2024-02-09T17:27:04.678 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.678 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fce60c33:::benchmark_data_smithi179_77138_object98750:head# 2024-02-09T17:27:04.678 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.678 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fce6225f:::benchmark_data_smithi179_77138_object114476:head# 2024-02-09T17:27:04.678 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.678 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fce646c3:::benchmark_data_smithi179_77138_object35308:head# 2024-02-09T17:27:04.678 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.678 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fce6ba16:::benchmark_data_smithi179_77138_object42466:head# 2024-02-09T17:27:04.679 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.679 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fce79031:::benchmark_data_smithi179_77138_object88267:head# 2024-02-09T17:27:04.679 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.679 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fce7b3b8:::benchmark_data_smithi179_77138_object93563:head# 2024-02-09T17:27:04.679 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.679 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fce7d400:::benchmark_data_smithi179_77138_object82348:head# 2024-02-09T17:27:04.679 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.679 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fce7e01d:::benchmark_data_smithi179_77138_object32873:head# 2024-02-09T17:27:04.679 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.679 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fce7e341:::benchmark_data_smithi179_77138_object62673:head# 2024-02-09T17:27:04.679 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.679 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fce814f1:::benchmark_data_smithi179_77138_object132041:head# 2024-02-09T17:27:04.679 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.679 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fce817c3:::benchmark_data_smithi179_77138_object68682:head# 2024-02-09T17:27:04.680 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.680 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fce84dc6:::benchmark_data_smithi179_77138_object91182:head# 2024-02-09T17:27:04.680 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.680 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fce86bea:::benchmark_data_smithi179_77138_object57937:head# 2024-02-09T17:27:04.680 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.680 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fce90555:::benchmark_data_smithi179_77138_object103096:head# 2024-02-09T17:27:04.680 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.680 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fce9344a:::benchmark_data_smithi179_77138_object119084:head# 2024-02-09T17:27:04.680 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.680 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fce94144:::benchmark_data_smithi179_77138_object34594:head# 2024-02-09T17:27:04.680 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.680 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcea2767:::benchmark_data_smithi179_77138_object65769:head# 2024-02-09T17:27:04.680 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.680 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcea372f:::benchmark_data_smithi179_77138_object43531:head# 2024-02-09T17:27:04.681 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.681 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcea44a5:::benchmark_data_smithi179_77138_object81095:head# 2024-02-09T17:27:04.681 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.681 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcea5f4d:::benchmark_data_smithi179_77138_object74462:head# 2024-02-09T17:27:04.681 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.681 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcea7fc6:::benchmark_data_smithi179_77138_object94028:head# 2024-02-09T17:27:04.681 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.681 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fceab00b:::benchmark_data_smithi179_77138_object25745:head# 2024-02-09T17:27:04.681 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.681 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcead19d:::benchmark_data_smithi179_77138_object64255:head# 2024-02-09T17:27:04.681 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.681 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fceaf817:::benchmark_data_smithi179_77138_object22436:head# 2024-02-09T17:27:04.681 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.681 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fceb40fc:::benchmark_data_smithi179_77138_object41865:head# 2024-02-09T17:27:04.682 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.682 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcebc719:::benchmark_data_smithi179_77138_object53833:head# 2024-02-09T17:27:04.682 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.682 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcec12e7:::benchmark_data_smithi179_77138_object123671:head# 2024-02-09T17:27:04.682 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.682 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fced2d2e:::benchmark_data_smithi179_77138_object64086:head# 2024-02-09T17:27:04.682 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.682 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcee9a97:::benchmark_data_smithi179_77138_object70681:head# 2024-02-09T17:27:04.682 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.682 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcefce09:::benchmark_data_smithi179_77138_object52589:head# 2024-02-09T17:27:04.682 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.682 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fceff29b:::benchmark_data_smithi179_77138_object108754:head# 2024-02-09T17:27:04.682 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.682 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcf03946:::benchmark_data_smithi179_77138_object81817:head# 2024-02-09T17:27:04.683 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.683 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcf03b61:::benchmark_data_smithi179_77138_object119603:head# 2024-02-09T17:27:04.683 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.683 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcf367ab:::benchmark_data_smithi179_77138_object73131:head# 2024-02-09T17:27:04.683 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.683 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcf3b211:::benchmark_data_smithi179_77138_object42146:head# 2024-02-09T17:27:04.683 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.683 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcf3eece:::benchmark_data_smithi179_77138_object112665:head# 2024-02-09T17:27:04.683 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.683 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcf4ee2e:::benchmark_data_smithi179_77138_object85828:head# 2024-02-09T17:27:04.683 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.683 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcf5a44a:::benchmark_data_smithi179_77138_object125720:head# 2024-02-09T17:27:04.683 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.683 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcf6184b:::benchmark_data_smithi179_77138_object125938:head# 2024-02-09T17:27:04.684 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.684 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcf627c9:::benchmark_data_smithi179_77138_object60911:head# 2024-02-09T17:27:04.684 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.684 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcf65643:::benchmark_data_smithi179_77138_object24576:head# 2024-02-09T17:27:04.684 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.684 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcf70721:::benchmark_data_smithi179_77138_object30740:head# 2024-02-09T17:27:04.684 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.684 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcf78d06:::benchmark_data_smithi179_77138_object61458:head# 2024-02-09T17:27:04.684 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.684 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcf80861:::benchmark_data_smithi179_77138_object124447:head# 2024-02-09T17:27:04.684 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.684 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcf8194d:::benchmark_data_smithi179_77138_object106739:head# 2024-02-09T17:27:04.684 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.684 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcfb0f7f:::benchmark_data_smithi179_77138_object29230:head# 2024-02-09T17:27:04.685 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.685 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcfb2710:::benchmark_data_smithi179_77138_object111435:head# 2024-02-09T17:27:04.685 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.685 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcfb4680:::benchmark_data_smithi179_77138_object102949:head# 2024-02-09T17:27:04.685 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.685 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcfbd511:::benchmark_data_smithi179_77138_object127929:head# 2024-02-09T17:27:04.685 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.685 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcfc4401:::benchmark_data_smithi179_77138_object81282:head# 2024-02-09T17:27:04.685 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.685 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcfccf88:::benchmark_data_smithi179_77138_object122907:head# 2024-02-09T17:27:04.685 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.685 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcfd9f92:::benchmark_data_smithi179_77138_object59224:head# 2024-02-09T17:27:04.685 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.685 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcfda0c1:::benchmark_data_smithi179_77138_object52968:head# 2024-02-09T17:27:04.686 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.686 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fcff3905:::benchmark_data_smithi179_77138_object116336:head# 2024-02-09T17:27:04.686 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.686 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd00593a:::benchmark_data_smithi179_77138_object24542:head# 2024-02-09T17:27:04.686 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.686 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd006b9a:::benchmark_data_smithi179_77138_object68225:head# 2024-02-09T17:27:04.686 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.686 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd017bae:::benchmark_data_smithi179_77138_object88591:head# 2024-02-09T17:27:04.686 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.686 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd0244e3:::benchmark_data_smithi179_77138_object79911:head# 2024-02-09T17:27:04.686 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.686 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd02e621:::benchmark_data_smithi179_77138_object82837:head# 2024-02-09T17:27:04.686 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.686 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd037a65:::benchmark_data_smithi179_77138_object50878:head# 2024-02-09T17:27:04.687 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.687 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd03f5b4:::benchmark_data_smithi179_77138_object33429:head# 2024-02-09T17:27:04.687 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.687 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd045931:::benchmark_data_smithi179_77138_object117172:head# 2024-02-09T17:27:04.687 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.687 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd051f69:::benchmark_data_smithi179_77138_object67116:head# 2024-02-09T17:27:04.687 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.687 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd05eeaa:::benchmark_data_smithi179_77138_object83053:head# 2024-02-09T17:27:04.687 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.687 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd0642f8:::benchmark_data_smithi179_77138_object88851:head# 2024-02-09T17:27:04.687 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.687 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd07aa62:::benchmark_data_smithi179_77138_object116943:head# 2024-02-09T17:27:04.687 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.687 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd09b0e3:::benchmark_data_smithi179_77138_object106950:head# 2024-02-09T17:27:04.687 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.688 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd09dd58:::benchmark_data_smithi179_77138_object105235:head# 2024-02-09T17:27:04.688 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.688 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd0adf5e:::benchmark_data_smithi179_77138_object76886:head# 2024-02-09T17:27:04.688 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.688 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd0ae0cd:::benchmark_data_smithi179_77138_object105984:head# 2024-02-09T17:27:04.688 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.688 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd0b448f:::benchmark_data_smithi179_77138_object78476:head# 2024-02-09T17:27:04.688 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.688 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd0d9e53:::benchmark_data_smithi179_77138_object99859:head# 2024-02-09T17:27:04.688 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.688 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd0de44d:::benchmark_data_smithi179_77138_object38265:head# 2024-02-09T17:27:04.688 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.688 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd0e39b9:::benchmark_data_smithi179_77138_object62065:head# 2024-02-09T17:27:04.689 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.689 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd0f51af:::benchmark_data_smithi179_77138_object20650:head# 2024-02-09T17:27:04.689 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.689 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd0f7526:::benchmark_data_smithi179_77138_object82915:head# 2024-02-09T17:27:04.689 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.689 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd0fd7c1:::benchmark_data_smithi179_77138_object110573:head# 2024-02-09T17:27:04.689 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.689 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd102e4f:::benchmark_data_smithi179_77138_object83187:head# 2024-02-09T17:27:04.689 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.689 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd11060f:::benchmark_data_smithi179_77138_object58739:head# 2024-02-09T17:27:04.689 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.689 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd11721f:::benchmark_data_smithi179_77138_object68202:head# 2024-02-09T17:27:04.689 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.689 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd118823:::benchmark_data_smithi179_77138_object95574:head# 2024-02-09T17:27:04.689 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.690 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd11c7f4:::benchmark_data_smithi179_77138_object63639:head# 2024-02-09T17:27:04.690 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.690 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd12575e:::benchmark_data_smithi179_77138_object41069:head# 2024-02-09T17:27:04.690 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.690 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd13c9c6:::benchmark_data_smithi179_77138_object100705:head# 2024-02-09T17:27:04.690 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.690 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd13d8bf:::benchmark_data_smithi179_77138_object40844:head# 2024-02-09T17:27:04.690 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.690 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd1457d1:::benchmark_data_smithi179_77138_object58641:head# 2024-02-09T17:27:04.690 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.690 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd1474fc:::benchmark_data_smithi179_77138_object54800:head# 2024-02-09T17:27:04.690 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.690 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd14c95c:::benchmark_data_smithi179_77138_object107407:head# 2024-02-09T17:27:04.690 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.690 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd1561b6:::benchmark_data_smithi179_77138_object21424:head# 2024-02-09T17:27:04.691 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.691 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd163b13:::benchmark_data_smithi179_77138_object43316:head# 2024-02-09T17:27:04.691 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.691 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd167ff5:::benchmark_data_smithi179_77138_object104434:head# 2024-02-09T17:27:04.691 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.691 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd16ff2e:::benchmark_data_smithi179_77138_object51173:head# 2024-02-09T17:27:04.691 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.691 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd17336a:::benchmark_data_smithi179_77138_object72696:head# 2024-02-09T17:27:04.691 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.691 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd180a59:::benchmark_data_smithi179_77138_object103265:head# 2024-02-09T17:27:04.691 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.691 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd181092:::benchmark_data_smithi179_77138_object81577:head# 2024-02-09T17:27:04.691 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.691 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd18f40d:::benchmark_data_smithi179_77138_object66888:head# 2024-02-09T17:27:04.692 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.692 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd198575:::benchmark_data_smithi179_77138_object119221:head# 2024-02-09T17:27:04.692 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.692 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd19900f:::benchmark_data_smithi179_77138_object96414:head# 2024-02-09T17:27:04.692 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.692 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd1a0411:::benchmark_data_smithi179_77138_object23440:head# 2024-02-09T17:27:04.692 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.692 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd1bf345:::benchmark_data_smithi179_77138_object59979:head# 2024-02-09T17:27:04.692 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.692 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd1c115d:::benchmark_data_smithi179_77138_object63168:head# 2024-02-09T17:27:04.692 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.692 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd1e0a7d:::benchmark_data_smithi179_77138_object83706:head# 2024-02-09T17:27:04.692 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.692 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd1e629c:::benchmark_data_smithi179_77138_object101209:head# 2024-02-09T17:27:04.693 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.693 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd1e7492:::benchmark_data_smithi179_77138_object91767:head# 2024-02-09T17:27:04.693 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.693 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:27:04.701 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:27:04.703 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd1fd1fa:::benchmark_data_smithi179_77138_object95615:head# 2024-02-09T17:27:04.703 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.703 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd218245:::benchmark_data_smithi179_77138_object68608:head# 2024-02-09T17:27:04.703 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.703 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd2241b8:::benchmark_data_smithi179_77138_object87993:head# 2024-02-09T17:27:04.703 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.703 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd23beaf:::benchmark_data_smithi179_77138_object40828:head# 2024-02-09T17:27:04.703 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.703 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd23cc36:::benchmark_data_smithi179_77138_object127545:head# 2024-02-09T17:27:04.703 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.703 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd250f35:::benchmark_data_smithi179_77138_object101085:head# 2024-02-09T17:27:04.703 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.703 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd26416e:::benchmark_data_smithi179_77138_object114830:head# 2024-02-09T17:27:04.704 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.704 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd2652ea:::benchmark_data_smithi179_77138_object33084:head# 2024-02-09T17:27:04.704 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.704 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd267c17:::benchmark_data_smithi179_77138_object34638:head# 2024-02-09T17:27:04.704 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.704 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd26b35e:::benchmark_data_smithi179_77138_object90286:head# 2024-02-09T17:27:04.704 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.704 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd26c355:::benchmark_data_smithi179_77138_object48443:head# 2024-02-09T17:27:04.704 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.704 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd282fa3:::benchmark_data_smithi179_77138_object97875:head# 2024-02-09T17:27:04.704 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.704 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd286524:::benchmark_data_smithi179_77138_object37345:head# 2024-02-09T17:27:04.704 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.704 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd28dd31:::benchmark_data_smithi179_77138_object68944:head# 2024-02-09T17:27:04.705 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.705 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd298d04:::benchmark_data_smithi179_77138_object20530:head# 2024-02-09T17:27:04.705 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.705 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd29b3af:::benchmark_data_smithi179_77138_object116877:head# 2024-02-09T17:27:04.705 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.705 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd2a3756:::benchmark_data_smithi179_77138_object58101:head# 2024-02-09T17:27:04.705 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.705 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd2afa3a:::benchmark_data_smithi179_77138_object88020:head# 2024-02-09T17:27:04.705 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.705 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd2b206e:::benchmark_data_smithi179_77138_object103455:head# 2024-02-09T17:27:04.705 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.705 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd2b29a1:::benchmark_data_smithi179_77138_object29730:head# 2024-02-09T17:27:04.705 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.706 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd2bb687:::benchmark_data_smithi179_77138_object44862:head# 2024-02-09T17:27:04.706 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.706 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd2c73a2:::benchmark_data_smithi179_77138_object102903:head# 2024-02-09T17:27:04.706 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.706 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd2d7f6b:::benchmark_data_smithi179_77138_object67572:head# 2024-02-09T17:27:04.706 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.706 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd2e4416:::benchmark_data_smithi179_77138_object106815:head# 2024-02-09T17:27:04.706 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.706 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd2e939d:::benchmark_data_smithi179_77138_object101366:head# 2024-02-09T17:27:04.706 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.706 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd2e956a:::benchmark_data_smithi179_77138_object65104:head# 2024-02-09T17:27:04.706 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.706 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd2ec134:::benchmark_data_smithi179_77138_object101865:head# 2024-02-09T17:27:04.706 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.707 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd308d29:::benchmark_data_smithi179_77138_object122727:head# 2024-02-09T17:27:04.707 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.707 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd30b894:::benchmark_data_smithi179_77138_object25291:head# 2024-02-09T17:27:04.707 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.707 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd3123b4:::benchmark_data_smithi179_77138_object87323:head# 2024-02-09T17:27:04.707 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.707 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd31328d:::benchmark_data_smithi179_77138_object76966:head# 2024-02-09T17:27:04.707 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.707 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd3166ee:::benchmark_data_smithi179_77138_object33738:head# 2024-02-09T17:27:04.707 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.707 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd33e5a2:::benchmark_data_smithi179_77138_object112560:head# 2024-02-09T17:27:04.707 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.707 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd356353:::benchmark_data_smithi179_77138_object26368:head# 2024-02-09T17:27:04.707 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.708 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd359499:::benchmark_data_smithi179_77138_object122085:head# 2024-02-09T17:27:04.708 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.708 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd370770:::benchmark_data_smithi179_77138_object64371:head# 2024-02-09T17:27:04.708 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.708 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd37afc1:::benchmark_data_smithi179_77138_object26623:head# 2024-02-09T17:27:04.708 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.708 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd3839e1:::benchmark_data_smithi179_77138_object47809:head# 2024-02-09T17:27:04.708 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.708 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd384496:::benchmark_data_smithi179_77138_object76536:head# 2024-02-09T17:27:04.708 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.708 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd394c9d:::benchmark_data_smithi179_77138_object45021:head# 2024-02-09T17:27:04.708 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.708 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd3a57a2:::benchmark_data_smithi179_77138_object101172:head# 2024-02-09T17:27:04.709 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.709 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd3a8eba:::benchmark_data_smithi179_77138_object22310:head# 2024-02-09T17:27:04.709 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.709 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd3c75cf:::benchmark_data_smithi179_77138_object40756:head# 2024-02-09T17:27:04.709 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.709 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd3cdf78:::benchmark_data_smithi179_77138_object39206:head# 2024-02-09T17:27:04.709 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.709 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd3d043b:::benchmark_data_smithi179_77138_object52567:head# 2024-02-09T17:27:04.709 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.709 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd3d712e:::benchmark_data_smithi179_77138_object77877:head# 2024-02-09T17:27:04.709 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.709 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd3dcd6c:::benchmark_data_smithi179_77138_object61460:head# 2024-02-09T17:27:04.709 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.709 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd3e263f:::benchmark_data_smithi179_77138_object62218:head# 2024-02-09T17:27:04.709 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.710 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd3eafb8:::benchmark_data_smithi179_77138_object87492:head# 2024-02-09T17:27:04.710 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.710 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd3f0d10:::benchmark_data_smithi179_77138_object42489:head# 2024-02-09T17:27:04.710 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.710 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd3f5724:::benchmark_data_smithi179_77138_object97010:head# 2024-02-09T17:27:04.710 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.710 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd3fcf96:::benchmark_data_smithi179_77138_object92919:head# 2024-02-09T17:27:04.710 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.710 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd402262:::benchmark_data_smithi179_77138_object60602:head# 2024-02-09T17:27:04.710 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.710 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd40583c:::benchmark_data_smithi179_77138_object94057:head# 2024-02-09T17:27:04.710 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.710 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd405ee5:::benchmark_data_smithi179_77138_object27362:head# 2024-02-09T17:27:04.710 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.711 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd42287a:::benchmark_data_smithi179_77138_object22494:head# 2024-02-09T17:27:04.711 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.711 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd42f86e:::benchmark_data_smithi179_77138_object79716:head# 2024-02-09T17:27:04.711 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.711 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd432a0b:::benchmark_data_smithi179_77138_object130801:head# 2024-02-09T17:27:04.711 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.711 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd436964:::benchmark_data_smithi179_77138_object90454:head# 2024-02-09T17:27:04.711 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.711 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd438514:::benchmark_data_smithi179_77138_object38920:head# 2024-02-09T17:27:04.711 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.711 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd439323:::benchmark_data_smithi179_77138_object86880:head# 2024-02-09T17:27:04.711 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.711 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd450b13:::benchmark_data_smithi179_77138_object72321:head# 2024-02-09T17:27:04.711 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.711 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd46b954:::benchmark_data_smithi179_77138_object96384:head# 2024-02-09T17:27:04.712 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.712 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd479234:::benchmark_data_smithi179_77138_object130488:head# 2024-02-09T17:27:04.712 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.712 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd47b2d7:::benchmark_data_smithi179_77138_object129246:head# 2024-02-09T17:27:04.712 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.712 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd47f8d8:::benchmark_data_smithi179_77138_object21220:head# 2024-02-09T17:27:04.712 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.712 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd483aed:::benchmark_data_smithi179_77138_object52711:head# 2024-02-09T17:27:04.712 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.712 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd4960bb:::benchmark_data_smithi179_77138_object118336:head# 2024-02-09T17:27:04.712 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.712 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd4971ce:::benchmark_data_smithi179_77138_object118852:head# 2024-02-09T17:27:04.712 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.712 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd49910f:::benchmark_data_smithi179_77138_object72934:head# 2024-02-09T17:27:04.712 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.713 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd4999cf:::benchmark_data_smithi179_77138_object112621:head# 2024-02-09T17:27:04.713 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.713 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd4a9fa5:::benchmark_data_smithi179_77138_object36862:head# 2024-02-09T17:27:04.713 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.713 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd4b794b:::benchmark_data_smithi179_77138_object73020:head# 2024-02-09T17:27:04.713 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.713 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd4b941b:::benchmark_data_smithi179_77138_object87088:head# 2024-02-09T17:27:04.713 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.713 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd4ba38a:::benchmark_data_smithi179_77138_object124449:head# 2024-02-09T17:27:04.713 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.713 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd4badda:::benchmark_data_smithi179_77138_object92855:head# 2024-02-09T17:27:04.713 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.713 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd4bed0c:::benchmark_data_smithi179_77138_object105937:head# 2024-02-09T17:27:04.713 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.714 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd4c46fc:::benchmark_data_smithi179_77138_object28565:head# 2024-02-09T17:27:04.714 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.714 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd4c5333:::benchmark_data_smithi179_77138_object107349:head# 2024-02-09T17:27:04.714 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.714 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd4c7e50:::benchmark_data_smithi179_77138_object20104:head# 2024-02-09T17:27:04.714 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.714 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd4d8e84:::benchmark_data_smithi179_77138_object90256:head# 2024-02-09T17:27:04.714 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.714 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd4da2e5:::benchmark_data_smithi179_77138_object107975:head# 2024-02-09T17:27:04.714 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.714 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd4dd76e:::benchmark_data_smithi179_77138_object38588:head# 2024-02-09T17:27:04.714 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.714 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd4dfe7e:::benchmark_data_smithi179_77138_object58215:head# 2024-02-09T17:27:04.714 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.714 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd4e734c:::benchmark_data_smithi179_77138_object43307:head# 2024-02-09T17:27:04.715 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.715 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd4f117f:::benchmark_data_smithi179_77138_object124841:head# 2024-02-09T17:27:04.715 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.715 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd4f5403:::benchmark_data_smithi179_77138_object99722:head# 2024-02-09T17:27:04.715 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.715 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd50c40e:::benchmark_data_smithi179_77138_object40499:head# 2024-02-09T17:27:04.715 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.715 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd50c681:::benchmark_data_smithi179_77138_object60136:head# 2024-02-09T17:27:04.715 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.715 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd50c781:::benchmark_data_smithi179_77138_object25423:head# 2024-02-09T17:27:04.715 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.715 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd5153ee:::benchmark_data_smithi179_77138_object106526:head# 2024-02-09T17:27:04.715 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.715 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd524f50:::benchmark_data_smithi179_77138_object76946:head# 2024-02-09T17:27:04.715 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.716 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd5262ea:::benchmark_data_smithi179_77138_object79796:head# 2024-02-09T17:27:04.716 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.716 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd532d14:::benchmark_data_smithi179_77138_object82326:head# 2024-02-09T17:27:04.716 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.716 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd53f460:::benchmark_data_smithi179_77138_object129684:head# 2024-02-09T17:27:04.716 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.716 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd5687c7:::benchmark_data_smithi179_77138_object53882:head# 2024-02-09T17:27:04.716 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.716 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd569e17:::benchmark_data_smithi179_77138_object112143:head# 2024-02-09T17:27:04.716 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.716 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd58d23b:::benchmark_data_smithi179_77138_object99927:head# 2024-02-09T17:27:04.716 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.716 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd592b10:::benchmark_data_smithi179_77138_object31688:head# 2024-02-09T17:27:04.716 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.717 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd596b2f:::benchmark_data_smithi179_77138_object109801:head# 2024-02-09T17:27:04.717 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.717 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd59c346:::benchmark_data_smithi179_77138_object96323:head# 2024-02-09T17:27:04.717 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.717 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd59d4cc:::benchmark_data_smithi179_77138_object107978:head# 2024-02-09T17:27:04.717 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.719 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:27:04.719 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd5ac776:::benchmark_data_smithi179_77138_object96653:head# 2024-02-09T17:27:04.719 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.720 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd5b4520:::benchmark_data_smithi179_77138_object25283:head# 2024-02-09T17:27:04.720 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.720 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd5b98c5:::benchmark_data_smithi179_77138_object41638:head# 2024-02-09T17:27:04.720 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.720 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd5c723a:::benchmark_data_smithi179_77138_object58230:head# 2024-02-09T17:27:04.720 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.720 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd5c7622:::benchmark_data_smithi179_77138_object78049:head# 2024-02-09T17:27:04.720 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.720 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd5ca3f8:::benchmark_data_smithi179_77138_object68056:head# 2024-02-09T17:27:04.720 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.720 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd5d1bb2:::benchmark_data_smithi179_77138_object85024:head# 2024-02-09T17:27:04.720 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.720 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd5d535a:::benchmark_data_smithi179_77138_object112051:head# 2024-02-09T17:27:04.721 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.721 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd5efa4e:::benchmark_data_smithi179_77138_object61455:head# 2024-02-09T17:27:04.721 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.721 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd6058b7:::benchmark_data_smithi179_77138_object34061:head# 2024-02-09T17:27:04.721 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.721 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd60f59c:::benchmark_data_smithi179_77138_object132997:head# 2024-02-09T17:27:04.721 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.721 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd63ea85:::benchmark_data_smithi179_77138_object127997:head# 2024-02-09T17:27:04.721 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.721 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd6512ce:::benchmark_data_smithi179_77138_object25522:head# 2024-02-09T17:27:04.721 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.721 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd659421:::benchmark_data_smithi179_77138_object121729:head# 2024-02-09T17:27:04.721 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.721 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd675d4f:::benchmark_data_smithi179_77138_object103595:head# 2024-02-09T17:27:04.722 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.722 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd67de47:::benchmark_data_smithi179_77138_object112597:head# 2024-02-09T17:27:04.722 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.722 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd680c42:::benchmark_data_smithi179_77138_object69017:head# 2024-02-09T17:27:04.722 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.722 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd6843f2:::benchmark_data_smithi179_77138_object102000:head# 2024-02-09T17:27:04.722 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.722 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd684ee5:::benchmark_data_smithi179_77138_object52536:head# 2024-02-09T17:27:04.722 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.722 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd68900a:::benchmark_data_smithi179_77138_object123221:head# 2024-02-09T17:27:04.722 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.722 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd691f49:::benchmark_data_smithi179_77138_object73778:head# 2024-02-09T17:27:04.722 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.722 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd6924f5:::benchmark_data_smithi179_77138_object28998:head# 2024-02-09T17:27:04.723 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.723 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd693150:::benchmark_data_smithi179_77138_object131012:head# 2024-02-09T17:27:04.723 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.723 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd697fe4:::benchmark_data_smithi179_77138_object39714:head# 2024-02-09T17:27:04.723 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.723 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd69a05e:::benchmark_data_smithi179_77138_object91586:head# 2024-02-09T17:27:04.723 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.723 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd69adbc:::benchmark_data_smithi179_77138_object68292:head# 2024-02-09T17:27:04.723 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.723 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd6a2c6e:::benchmark_data_smithi179_77138_object57137:head# 2024-02-09T17:27:04.723 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.723 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd6aaa6d:::benchmark_data_smithi179_77138_object50782:head# 2024-02-09T17:27:04.723 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.723 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd6b4992:::benchmark_data_smithi179_77138_object59825:head# 2024-02-09T17:27:04.723 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.724 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd6b4c1b:::benchmark_data_smithi179_77138_object62713:head# 2024-02-09T17:27:04.724 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.724 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd6c37ba:::benchmark_data_smithi179_77138_object19318:head# 2024-02-09T17:27:04.724 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.724 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd6c7759:::benchmark_data_smithi179_77138_object130834:head# 2024-02-09T17:27:04.724 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.724 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd6dc69b:::benchmark_data_smithi179_77138_object118502:head# 2024-02-09T17:27:04.724 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.724 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd6dde16:::benchmark_data_smithi179_77138_object92027:head# 2024-02-09T17:27:04.724 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.724 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd6e465a:::benchmark_data_smithi179_77138_object97176:head# 2024-02-09T17:27:04.724 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.724 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd6e58af:::benchmark_data_smithi179_77138_object100989:head# 2024-02-09T17:27:04.724 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.725 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd6e7fe7:::benchmark_data_smithi179_77138_object131725:head# 2024-02-09T17:27:04.725 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.725 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd6f0e21:::benchmark_data_smithi179_77138_object112674:head# 2024-02-09T17:27:04.725 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.725 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd700a0f:::benchmark_data_smithi179_77138_object93889:head# 2024-02-09T17:27:04.725 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.725 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd702b9b:::benchmark_data_smithi179_77138_object102376:head# 2024-02-09T17:27:04.725 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.725 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd7056f9:::benchmark_data_smithi179_77138_object32120:head# 2024-02-09T17:27:04.725 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.725 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd70baa1:::benchmark_data_smithi179_77138_object21588:head# 2024-02-09T17:27:04.725 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.725 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd70ef97:::benchmark_data_smithi179_77138_object66736:head# 2024-02-09T17:27:04.725 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.725 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd716fa2:::benchmark_data_smithi179_77138_object120183:head# 2024-02-09T17:27:04.726 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.726 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd71b0c0:::benchmark_data_smithi179_77138_object82399:head# 2024-02-09T17:27:04.726 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.726 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd72817a:::benchmark_data_smithi179_77138_object66701:head# 2024-02-09T17:27:04.726 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.726 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd72bdec:::benchmark_data_smithi179_77138_object20174:head# 2024-02-09T17:27:04.726 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.727 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd72e794:::benchmark_data_smithi179_77138_object112669:head# 2024-02-09T17:27:04.727 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.728 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd732b0b:::benchmark_data_smithi179_77138_object126345:head# 2024-02-09T17:27:04.728 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.728 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd74127b:::benchmark_data_smithi179_77138_object116371:head# 2024-02-09T17:27:04.728 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.729 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd7549c5:::benchmark_data_smithi179_77138_object79982:head# 2024-02-09T17:27:04.729 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.729 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd760b9a:::benchmark_data_smithi179_77138_object30566:head# 2024-02-09T17:27:04.729 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.729 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd761a83:::benchmark_data_smithi179_77138_object126848:head# 2024-02-09T17:27:04.729 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.729 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd761c7a:::benchmark_data_smithi179_77138_object49708:head# 2024-02-09T17:27:04.729 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.730 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd77a162:::benchmark_data_smithi179_77138_object66624:head# 2024-02-09T17:27:04.730 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.731 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd77f369:::benchmark_data_smithi179_77138_object85180:head# 2024-02-09T17:27:04.731 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.731 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd789d9e:::benchmark_data_smithi179_77138_object48546:head# 2024-02-09T17:27:04.731 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.731 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd7a5bc0:::benchmark_data_smithi179_77138_object132350:head# 2024-02-09T17:27:04.731 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.731 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd7ab41b:::benchmark_data_smithi179_77138_object89944:head# 2024-02-09T17:27:04.731 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.732 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd7b5558:::benchmark_data_smithi179_77138_object52185:head# 2024-02-09T17:27:04.732 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.733 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd7de3db:::benchmark_data_smithi179_77138_object109874:head# 2024-02-09T17:27:04.733 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.733 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd7ef964:::benchmark_data_smithi179_77138_object127411:head# 2024-02-09T17:27:04.733 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.733 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd7f08da:::benchmark_data_smithi179_77138_object85072:head# 2024-02-09T17:27:04.733 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.733 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd7f552a:::benchmark_data_smithi179_77138_object49471:head# 2024-02-09T17:27:04.733 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.733 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd8005d9:::benchmark_data_smithi179_77138_object45450:head# 2024-02-09T17:27:04.733 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.733 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd80abfa:::benchmark_data_smithi179_77138_object127032:head# 2024-02-09T17:27:04.733 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.737 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd80bfb7:::benchmark_data_smithi179_77138_object26704:head# 2024-02-09T17:27:04.737 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.737 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd810c67:::benchmark_data_smithi179_77138_object95863:head# 2024-02-09T17:27:04.737 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.741 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:04 smithi099 ceph-mon[39420]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:27:04.741 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:04 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:27:04.741 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:04 smithi099 ceph-mon[39420]: osdmap e151: 12 total, 11 up, 12 in 2024-02-09T17:27:04.741 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:04 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2831009301' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:27:04.742 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd818d57:::benchmark_data_smithi179_77138_object92671:head# 2024-02-09T17:27:04.742 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.744 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd81c6ef:::benchmark_data_smithi179_77138_object28944:head# 2024-02-09T17:27:04.744 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.748 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd826950:::benchmark_data_smithi179_77138_object66395:head# 2024-02-09T17:27:04.748 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.748 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd82711a:::benchmark_data_smithi179_77138_object91473:head# 2024-02-09T17:27:04.748 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.748 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd837022:::benchmark_data_smithi179_77138_object100031:head# 2024-02-09T17:27:04.748 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.748 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd8373f0:::benchmark_data_smithi179_77138_object37223:head# 2024-02-09T17:27:04.748 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.748 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd83f035:::benchmark_data_smithi179_77138_object91936:head# 2024-02-09T17:27:04.748 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.748 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd8414ef:::benchmark_data_smithi179_77138_object62197:head# 2024-02-09T17:27:04.749 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.750 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd84700a:::benchmark_data_smithi179_77138_object20479:head# 2024-02-09T17:27:04.750 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.751 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd8494e9:::benchmark_data_smithi179_77138_object21151:head# 2024-02-09T17:27:04.751 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.751 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd866d9f:::benchmark_data_smithi179_77138_object22305:head# 2024-02-09T17:27:04.751 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.751 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd8856cc:::benchmark_data_smithi179_77138_object85396:head# 2024-02-09T17:27:04.751 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.751 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd88a82a:::benchmark_data_smithi179_77138_object116752:head# 2024-02-09T17:27:04.751 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.751 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd88e60d:::benchmark_data_smithi179_77138_object65617:head# 2024-02-09T17:27:04.751 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.751 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd8957ff:::benchmark_data_smithi179_77138_object92992:head# 2024-02-09T17:27:04.751 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.751 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd896603:::benchmark_data_smithi179_77138_object65496:head# 2024-02-09T17:27:04.751 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.751 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd897602:::benchmark_data_smithi179_77138_object91757:head# 2024-02-09T17:27:04.752 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.752 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd8a573a:::benchmark_data_smithi179_77138_object107067:head# 2024-02-09T17:27:04.752 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.752 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:27:04.753 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:27:04.753 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:27:04.753 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:27:04.753 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:27:04.753 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:27:04.753 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:27:04.753 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:27:04.753 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:27:04.753 INFO:teuthology.orchestra.run.smithi099.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-02-09T17:27:04.753 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:27:04.753 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:27:04.754 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:27:04.754 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd8ac443:::benchmark_data_smithi179_77138_object82282:head# 2024-02-09T17:27:04.754 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.754 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd8bc801:::benchmark_data_smithi179_77138_object120726:head# 2024-02-09T17:27:04.754 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.754 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd8c52c6:::benchmark_data_smithi179_77138_object43938:head# 2024-02-09T17:27:04.754 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.755 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:04.755 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd8c7fa4:::benchmark_data_smithi179_77138_object97907:head# 2024-02-09T17:27:04.755 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.757 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd8d08a2:::benchmark_data_smithi179_77138_object124671:head# 2024-02-09T17:27:04.757 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.764 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd8d1acd:::benchmark_data_smithi179_77138_object68025:head# 2024-02-09T17:27:04.764 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.771 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd8d666e:::benchmark_data_smithi179_77138_object75638:head# 2024-02-09T17:27:04.771 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.774 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd8dccf6:::benchmark_data_smithi179_77138_object19538:head# 2024-02-09T17:27:04.774 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.774 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd8e9c12:::benchmark_data_smithi179_77138_object98737:head# 2024-02-09T17:27:04.774 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.776 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:27:04.779 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd8f0f12:::benchmark_data_smithi179_77138_object72329:head# 2024-02-09T17:27:04.779 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.779 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd8f23a9:::benchmark_data_smithi179_77138_object26324:head# 2024-02-09T17:27:04.779 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.780 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd9136b8:::benchmark_data_smithi179_77138_object111977:head# 2024-02-09T17:27:04.781 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.781 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd915b7c:::benchmark_data_smithi179_77138_object63958:head# 2024-02-09T17:27:04.781 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.781 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd9161de:::benchmark_data_smithi179_77138_object65530:head# 2024-02-09T17:27:04.781 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.781 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd917c86:::benchmark_data_smithi179_77138_object51958:head# 2024-02-09T17:27:04.781 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.781 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd929981:::benchmark_data_smithi179_77138_object49165:head# 2024-02-09T17:27:04.781 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.782 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd935071:::benchmark_data_smithi179_77138_object57413:head# 2024-02-09T17:27:04.782 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.782 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd938743:::benchmark_data_smithi179_77138_object114198:head# 2024-02-09T17:27:04.782 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.782 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd93a1b6:::benchmark_data_smithi179_77138_object79775:head# 2024-02-09T17:27:04.782 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:04.783 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:27:04.785 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd952721:::benchmark_data_smithi179_77138_object95097:head# 2024-02-09T17:27:04.785 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.785 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd95583e:::benchmark_data_smithi179_77138_object58905:head# 2024-02-09T17:27:04.785 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.785 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd95e867:::benchmark_data_smithi179_77138_object123251:head# 2024-02-09T17:27:04.785 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.785 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd981701:::benchmark_data_smithi179_77138_object59721:head# 2024-02-09T17:27:04.785 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.786 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd988a42:::benchmark_data_smithi179_77138_object20327:head# 2024-02-09T17:27:04.786 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.786 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd98e70e:::benchmark_data_smithi179_77138_object95456:head# 2024-02-09T17:27:04.786 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.786 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd99814c:::benchmark_data_smithi179_77138_object45782:head# 2024-02-09T17:27:04.786 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.786 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd9a3694:::benchmark_data_smithi179_77138_object113805:head# 2024-02-09T17:27:04.786 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.786 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd9a9525:::benchmark_data_smithi179_77138_object102086:head# 2024-02-09T17:27:04.786 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.786 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd9af947:::benchmark_data_smithi179_77138_object105799:head# 2024-02-09T17:27:04.786 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.786 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd9bc94d:::benchmark_data_smithi179_77138_object76964:head# 2024-02-09T17:27:04.786 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.787 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd9e904c:::benchmark_data_smithi179_77138_object53051:head# 2024-02-09T17:27:04.787 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.793 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd9ec30e:::benchmark_data_smithi179_77138_object32212:head# 2024-02-09T17:27:04.793 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.793 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fd9f00c3:::benchmark_data_smithi179_77138_object39853:head# 2024-02-09T17:27:04.793 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.793 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fda26eb0:::benchmark_data_smithi179_77138_object97077:head# 2024-02-09T17:27:04.793 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.793 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fda291cd:::benchmark_data_smithi179_77138_object28006:head# 2024-02-09T17:27:04.793 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.794 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fda2b67c:::benchmark_data_smithi179_77138_object133048:head# 2024-02-09T17:27:04.794 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.794 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fda3725b:::benchmark_data_smithi179_77138_object24805:head# 2024-02-09T17:27:04.794 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.794 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fda394f2:::benchmark_data_smithi179_77138_object129419:head# 2024-02-09T17:27:04.794 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.794 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fda47e47:::benchmark_data_smithi179_77138_object101537:head# 2024-02-09T17:27:04.794 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.794 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fda4c309:::benchmark_data_smithi179_77138_object132834:head# 2024-02-09T17:27:04.794 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.794 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fda5390c:::benchmark_data_smithi179_77138_object63254:head# 2024-02-09T17:27:04.794 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.794 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fda705e6:::benchmark_data_smithi179_77138_object132187:head# 2024-02-09T17:27:04.795 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.795 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fda85216:::benchmark_data_smithi179_77138_object132877:head# 2024-02-09T17:27:04.795 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.795 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdaa667e:::benchmark_data_smithi179_77138_object98014:head# 2024-02-09T17:27:04.795 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.795 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdabd36b:::benchmark_data_smithi179_77138_object73867:head# 2024-02-09T17:27:04.795 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.795 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdac232c:::benchmark_data_smithi179_77138_object52568:head# 2024-02-09T17:27:04.795 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.795 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdacf794:::benchmark_data_smithi179_77138_object109507:head# 2024-02-09T17:27:04.795 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.795 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdad055d:::benchmark_data_smithi179_77138_object30692:head# 2024-02-09T17:27:04.795 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.796 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdad832f:::benchmark_data_smithi179_77138_object37527:head# 2024-02-09T17:27:04.796 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.796 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdad98ed:::benchmark_data_smithi179_77138_object131246:head# 2024-02-09T17:27:04.796 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.796 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdaf3431:::benchmark_data_smithi179_77138_object45714:head# 2024-02-09T17:27:04.796 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.796 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdaf78b8:::benchmark_data_smithi179_77138_object47659:head# 2024-02-09T17:27:04.796 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.796 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdafea3c:::benchmark_data_smithi179_77138_object83274:head# 2024-02-09T17:27:04.796 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.796 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdb00d97:::benchmark_data_smithi179_77138_object37106:head# 2024-02-09T17:27:04.796 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.796 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdb0c5b0:::benchmark_data_smithi179_77138_object131098:head# 2024-02-09T17:27:04.796 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.796 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdb38cb4:::benchmark_data_smithi179_77138_object124026:head# 2024-02-09T17:27:04.797 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.797 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdb3ae65:::benchmark_data_smithi179_77138_object80717:head# 2024-02-09T17:27:04.797 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.797 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdb3d79f:::benchmark_data_smithi179_77138_object70704:head# 2024-02-09T17:27:04.797 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.797 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdb50b6c:::benchmark_data_smithi179_77138_object103772:head# 2024-02-09T17:27:04.797 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.797 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdb51550:::benchmark_data_smithi179_77138_object46047:head# 2024-02-09T17:27:04.797 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.797 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdb64af2:::benchmark_data_smithi179_77138_object41144:head# 2024-02-09T17:27:04.797 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.797 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdb6a1b2:::benchmark_data_smithi179_77138_object92848:head# 2024-02-09T17:27:04.797 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.798 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdb74f9e:::benchmark_data_smithi179_77138_object79060:head# 2024-02-09T17:27:04.798 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.798 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdb775f0:::benchmark_data_smithi179_77138_object37536:head# 2024-02-09T17:27:04.798 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.798 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdb7a619:::benchmark_data_smithi179_77138_object100195:head# 2024-02-09T17:27:04.798 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.804 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdb7aaad:::benchmark_data_smithi179_77138_object47409:head# 2024-02-09T17:27:04.804 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.805 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdb803f7:::benchmark_data_smithi179_77138_object96075:head# 2024-02-09T17:27:04.805 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.805 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdb8d6ff:::benchmark_data_smithi179_77138_object48502:head# 2024-02-09T17:27:04.805 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.805 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdb91d00:::benchmark_data_smithi179_77138_object72885:head# 2024-02-09T17:27:04.805 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.805 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdb96c4f:::benchmark_data_smithi179_77138_object109033:head# 2024-02-09T17:27:04.805 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.805 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdb98319:::benchmark_data_smithi179_77138_object126010:head# 2024-02-09T17:27:04.805 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.805 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdb9f3d3:::benchmark_data_smithi179_77138_object73195:head# 2024-02-09T17:27:04.805 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.805 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdba39f5:::benchmark_data_smithi179_77138_object44581:head# 2024-02-09T17:27:04.806 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.806 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdba46d1:::benchmark_data_smithi179_77138_object88416:head# 2024-02-09T17:27:04.806 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.806 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdba5b42:::benchmark_data_smithi179_77138_object65196:head# 2024-02-09T17:27:04.806 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.806 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdba7b4e:::benchmark_data_smithi179_77138_object90723:head# 2024-02-09T17:27:04.806 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.806 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdba82aa:::benchmark_data_smithi179_77138_object120363:head# 2024-02-09T17:27:04.806 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.806 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdbb2171:::benchmark_data_smithi179_77138_object121016:head# 2024-02-09T17:27:04.806 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.806 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdbb6606:::benchmark_data_smithi179_77138_object93233:head# 2024-02-09T17:27:04.806 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.807 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdbc4156:::benchmark_data_smithi179_77138_object81587:head# 2024-02-09T17:27:04.807 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.807 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdbc8035:::benchmark_data_smithi179_77138_object33190:head# 2024-02-09T17:27:04.807 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.807 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdbd158a:::benchmark_data_smithi179_77138_object83527:head# 2024-02-09T17:27:04.807 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.807 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdbd6a45:::benchmark_data_smithi179_77138_object35738:head# 2024-02-09T17:27:04.807 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.807 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdbd8598:::benchmark_data_smithi179_77138_object60405:head# 2024-02-09T17:27:04.807 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.807 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdbe3646:::benchmark_data_smithi179_77138_object42845:head# 2024-02-09T17:27:04.807 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.807 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdbf0469:::benchmark_data_smithi179_77138_object115394:head# 2024-02-09T17:27:04.807 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.807 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdbf1b9a:::benchmark_data_smithi179_77138_object35972:head# 2024-02-09T17:27:04.808 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.808 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdbf4010:::benchmark_data_smithi179_77138_object118379:head# 2024-02-09T17:27:04.808 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.808 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdc028cd:::benchmark_data_smithi179_77138_object58045:head# 2024-02-09T17:27:04.808 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.808 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdc05619:::benchmark_data_smithi179_77138_object102654:head# 2024-02-09T17:27:04.808 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.808 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdc08429:::benchmark_data_smithi179_77138_object88023:head# 2024-02-09T17:27:04.808 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.808 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdc0bba7:::benchmark_data_smithi179_77138_object131695:head# 2024-02-09T17:27:04.808 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.808 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdc17e38:::benchmark_data_smithi179_77138_object73059:head# 2024-02-09T17:27:04.808 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.808 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdc38eee:::benchmark_data_smithi179_77138_object68723:head# 2024-02-09T17:27:04.809 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.809 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdc45c4b:::benchmark_data_smithi179_77138_object121371:head# 2024-02-09T17:27:04.809 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.809 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdc5d563:::benchmark_data_smithi179_77138_object80829:head# 2024-02-09T17:27:04.809 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.809 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdc666b3:::benchmark_data_smithi179_77138_object46189:head# 2024-02-09T17:27:04.809 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.809 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdc6bfe7:::benchmark_data_smithi179_77138_object124204:head# 2024-02-09T17:27:04.809 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.809 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdc7595a:::benchmark_data_smithi179_77138_object26990:head# 2024-02-09T17:27:04.809 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.809 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdc80a1c:::benchmark_data_smithi179_77138_object29622:head# 2024-02-09T17:27:04.809 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.809 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdc83b7d:::benchmark_data_smithi179_77138_object91422:head# 2024-02-09T17:27:04.810 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.810 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdc8bbe2:::benchmark_data_smithi179_77138_object38546:head# 2024-02-09T17:27:04.810 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.810 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdca0f51:::benchmark_data_smithi179_77138_object76463:head# 2024-02-09T17:27:04.810 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.810 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdcbe79d:::benchmark_data_smithi179_77138_object129649:head# 2024-02-09T17:27:04.810 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.810 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdcc08b3:::benchmark_data_smithi179_77138_object72917:head# 2024-02-09T17:27:04.810 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.810 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdcc0c0e:::benchmark_data_smithi179_77138_object95190:head# 2024-02-09T17:27:04.810 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.810 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdcc0c97:::benchmark_data_smithi179_77138_object90818:head# 2024-02-09T17:27:04.810 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.810 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdcc42d2:::benchmark_data_smithi179_77138_object36407:head# 2024-02-09T17:27:04.811 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.811 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdcc7a31:::benchmark_data_smithi179_77138_object57401:head# 2024-02-09T17:27:04.811 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.811 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdccc8c8:::benchmark_data_smithi179_77138_object84324:head# 2024-02-09T17:27:04.811 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.811 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdcd0b2c:::benchmark_data_smithi179_77138_object81338:head# 2024-02-09T17:27:04.811 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.811 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdcd4e24:::benchmark_data_smithi179_77138_object23711:head# 2024-02-09T17:27:04.811 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.811 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdcd8077:::benchmark_data_smithi179_77138_object130625:head# 2024-02-09T17:27:04.811 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.823 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdcdd988:::benchmark_data_smithi179_77138_object68315:head# 2024-02-09T17:27:04.823 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.824 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdce0d30:::benchmark_data_smithi179_77138_object101903:head# 2024-02-09T17:27:04.824 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.824 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdcefa3d:::benchmark_data_smithi179_77138_object27415:head# 2024-02-09T17:27:04.824 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.824 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdd03297:::benchmark_data_smithi179_77138_object26553:head# 2024-02-09T17:27:04.824 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.824 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdd0e777:::benchmark_data_smithi179_77138_object43856:head# 2024-02-09T17:27:04.824 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.824 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdd186dd:::benchmark_data_smithi179_77138_object130940:head# 2024-02-09T17:27:04.824 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.824 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdd2c92b:::benchmark_data_smithi179_77138_object70571:head# 2024-02-09T17:27:04.824 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.825 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdd31ad5:::benchmark_data_smithi179_77138_object108563:head# 2024-02-09T17:27:04.825 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.825 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdd3290c:::benchmark_data_smithi179_77138_object77670:head# 2024-02-09T17:27:04.825 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.825 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdd3bd18:::benchmark_data_smithi179_77138_object42095:head# 2024-02-09T17:27:04.825 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.825 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdd587af:::benchmark_data_smithi179_77138_object112833:head# 2024-02-09T17:27:04.825 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.825 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdd5d41f:::benchmark_data_smithi179_77138_object32659:head# 2024-02-09T17:27:04.825 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.825 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdd5d6a7:::benchmark_data_smithi179_77138_object106885:head# 2024-02-09T17:27:04.825 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.825 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdd5ecb9:::benchmark_data_smithi179_77138_object67800:head# 2024-02-09T17:27:04.825 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.826 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdd64628:::benchmark_data_smithi179_77138_object119110:head# 2024-02-09T17:27:04.826 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.826 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdd66d42:::benchmark_data_smithi179_77138_object126415:head# 2024-02-09T17:27:04.826 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.826 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdd6802e:::benchmark_data_smithi179_77138_object61644:head# 2024-02-09T17:27:04.826 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.826 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdd6a9ec:::benchmark_data_smithi179_77138_object27146:head# 2024-02-09T17:27:04.826 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.826 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdd6d444:::benchmark_data_smithi179_77138_object34907:head# 2024-02-09T17:27:04.826 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.826 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdd70507:::benchmark_data_smithi179_77138_object52698:head# 2024-02-09T17:27:04.826 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.826 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdd71c2c:::benchmark_data_smithi179_77138_object98754:head# 2024-02-09T17:27:04.827 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.827 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdd723a8:::benchmark_data_smithi179_77138_object22429:head# 2024-02-09T17:27:04.827 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.827 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdd7256b:::benchmark_data_smithi179_77138_object89043:head# 2024-02-09T17:27:04.827 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.827 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdd72a91:::benchmark_data_smithi179_77138_object30064:head# 2024-02-09T17:27:04.827 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.827 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdd7b260:::benchmark_data_smithi179_77138_object85173:head# 2024-02-09T17:27:04.827 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.827 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdd82689:::benchmark_data_smithi179_77138_object111657:head# 2024-02-09T17:27:04.827 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.827 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdd854bd:::benchmark_data_smithi179_77138_object22101:head# 2024-02-09T17:27:04.827 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.827 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdd89bf9:::benchmark_data_smithi179_77138_object104756:head# 2024-02-09T17:27:04.827 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.828 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdd8e514:::benchmark_data_smithi179_77138_object128042:head# 2024-02-09T17:27:04.828 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.828 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdd9c071:::benchmark_data_smithi179_77138_object75085:head# 2024-02-09T17:27:04.828 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.828 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdda5bc4:::benchmark_data_smithi179_77138_object74847:head# 2024-02-09T17:27:04.828 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.828 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fddbfb3c:::benchmark_data_smithi179_77138_object119388:head# 2024-02-09T17:27:04.828 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.828 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fddce621:::benchmark_data_smithi179_77138_object75617:head# 2024-02-09T17:27:04.828 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.828 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fddd71a7:::benchmark_data_smithi179_77138_object108695:head# 2024-02-09T17:27:04.828 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.828 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdddc6e8:::benchmark_data_smithi179_77138_object29324:head# 2024-02-09T17:27:04.828 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.829 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fde0c99e:::benchmark_data_smithi179_77138_object56875:head# 2024-02-09T17:27:04.829 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.829 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fde1d57a:::benchmark_data_smithi179_77138_object67388:head# 2024-02-09T17:27:04.829 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.829 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fde25425:::benchmark_data_smithi179_77138_object58914:head# 2024-02-09T17:27:04.829 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.829 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fde25ae9:::benchmark_data_smithi179_77138_object28394:head# 2024-02-09T17:27:04.829 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.829 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fde3612f:::benchmark_data_smithi179_77138_object102018:head# 2024-02-09T17:27:04.829 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.829 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fde4aabc:::benchmark_data_smithi179_77138_object99652:head# 2024-02-09T17:27:04.829 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.829 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fde4f1ce:::benchmark_data_smithi179_77138_object24337:head# 2024-02-09T17:27:04.829 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.829 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fde50f2c:::benchmark_data_smithi179_77138_object53754:head# 2024-02-09T17:27:04.830 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.830 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fde52d10:::benchmark_data_smithi179_77138_object38845:head# 2024-02-09T17:27:04.830 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.830 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fde5ad92:::benchmark_data_smithi179_77138_object105737:head# 2024-02-09T17:27:04.830 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.830 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fde6e18a:::benchmark_data_smithi179_77138_object81798:head# 2024-02-09T17:27:04.830 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.830 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fde83230:::benchmark_data_smithi179_77138_object103964:head# 2024-02-09T17:27:04.830 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.830 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fde8bbb7:::benchmark_data_smithi179_77138_object26871:head# 2024-02-09T17:27:04.830 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.830 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fde9442e:::benchmark_data_smithi179_77138_object29891:head# 2024-02-09T17:27:04.830 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.830 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fde98b2a:::benchmark_data_smithi179_77138_object111132:head# 2024-02-09T17:27:04.831 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.831 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fde9c1b6:::benchmark_data_smithi179_77138_object81001:head# 2024-02-09T17:27:04.831 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.831 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdea11e0:::benchmark_data_smithi179_77138_object84090:head# 2024-02-09T17:27:04.831 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.831 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdeae8e4:::benchmark_data_smithi179_77138_object87720:head# 2024-02-09T17:27:04.831 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.831 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdeaff39:::benchmark_data_smithi179_77138_object43064:head# 2024-02-09T17:27:04.831 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.831 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdeb4be9:::benchmark_data_smithi179_77138_object102751:head# 2024-02-09T17:27:04.831 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.831 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdec14f3:::benchmark_data_smithi179_77138_object30807:head# 2024-02-09T17:27:04.831 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.831 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdec57ec:::benchmark_data_smithi179_77138_object111136:head# 2024-02-09T17:27:04.832 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.832 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdec8e8c:::benchmark_data_smithi179_77138_object81237:head# 2024-02-09T17:27:04.832 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.832 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdee3b27:::benchmark_data_smithi179_77138_object19506:head# 2024-02-09T17:27:04.832 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.832 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdee6eb5:::benchmark_data_smithi179_77138_object65410:head# 2024-02-09T17:27:04.832 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.832 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdeedfac:::benchmark_data_smithi179_77138_object89691:head# 2024-02-09T17:27:04.832 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.832 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdef388f:::benchmark_data_smithi179_77138_object76649:head# 2024-02-09T17:27:04.832 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.832 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdf00c57:::benchmark_data_smithi179_77138_object59616:head# 2024-02-09T17:27:04.832 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.832 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdf0b5c3:::benchmark_data_smithi179_77138_object58830:head# 2024-02-09T17:27:04.833 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.833 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdf1fb62:::benchmark_data_smithi179_77138_object47857:head# 2024-02-09T17:27:04.833 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.833 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdf2c449:::benchmark_data_smithi179_77138_object38800:head# 2024-02-09T17:27:04.833 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.833 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdf2e808:::benchmark_data_smithi179_77138_object54439:head# 2024-02-09T17:27:04.833 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.833 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdf2f311:::benchmark_data_smithi179_77138_object113271:head# 2024-02-09T17:27:04.833 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.833 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdf33281:::benchmark_data_smithi179_77138_object33148:head# 2024-02-09T17:27:04.833 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.833 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdf390a2:::benchmark_data_smithi179_77138_object94450:head# 2024-02-09T17:27:04.833 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.833 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdf57b9e:::benchmark_data_smithi179_77138_object55229:head# 2024-02-09T17:27:04.833 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.834 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdf5b00a:::benchmark_data_smithi179_77138_object61592:head# 2024-02-09T17:27:04.834 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.834 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdf686fa:::benchmark_data_smithi179_77138_object30040:head# 2024-02-09T17:27:04.834 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.834 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdf69d0e:::benchmark_data_smithi179_77138_object82544:head# 2024-02-09T17:27:04.834 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.842 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdf7deec:::benchmark_data_smithi179_77138_object100714:head# 2024-02-09T17:27:04.842 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.842 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdf903eb:::benchmark_data_smithi179_77138_object129715:head# 2024-02-09T17:27:04.842 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.842 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdf9ec6d:::benchmark_data_smithi179_77138_object122700:head# 2024-02-09T17:27:04.842 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.842 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdfa3b32:::benchmark_data_smithi179_77138_object33963:head# 2024-02-09T17:27:04.842 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.843 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdfaa187:::benchmark_data_smithi179_77138_object58210:head# 2024-02-09T17:27:04.843 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.843 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdfb581f:::benchmark_data_smithi179_77138_object107471:head# 2024-02-09T17:27:04.843 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.843 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdfbd182:::benchmark_data_smithi179_77138_object96714:head# 2024-02-09T17:27:04.843 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.843 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdfcd9b0:::benchmark_data_smithi179_77138_object81044:head# 2024-02-09T17:27:04.843 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.843 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdfd57e7:::benchmark_data_smithi179_77138_object37510:head# 2024-02-09T17:27:04.843 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.843 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fdff05a5:::benchmark_data_smithi179_77138_object102951:head# 2024-02-09T17:27:04.843 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.843 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe003356:::benchmark_data_smithi179_77138_object86451:head# 2024-02-09T17:27:04.844 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.844 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe005eb7:::benchmark_data_smithi179_77138_object59813:head# 2024-02-09T17:27:04.844 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.844 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe009c42:::benchmark_data_smithi179_77138_object57283:head# 2024-02-09T17:27:04.844 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.844 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe00bd09:::benchmark_data_smithi179_77138_object61261:head# 2024-02-09T17:27:04.844 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.844 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe0117e5:::benchmark_data_smithi179_77138_object129124:head# 2024-02-09T17:27:04.844 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.844 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe0221ae:::benchmark_data_smithi179_77138_object23766:head# 2024-02-09T17:27:04.844 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.844 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe028c3d:::benchmark_data_smithi179_77138_object54672:head# 2024-02-09T17:27:04.844 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.845 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe03c7cb:::benchmark_data_smithi179_77138_object74818:head# 2024-02-09T17:27:04.845 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.845 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe0505c1:::benchmark_data_smithi179_77138_object100998:head# 2024-02-09T17:27:04.845 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.845 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe05dd12:::benchmark_data_smithi179_77138_object94164:head# 2024-02-09T17:27:04.845 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.845 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe05e441:::benchmark_data_smithi179_77138_object122456:head# 2024-02-09T17:27:04.845 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.845 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe0617b8:::benchmark_data_smithi179_77138_object95804:head# 2024-02-09T17:27:04.845 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.845 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe064121:::benchmark_data_smithi179_77138_object66865:head# 2024-02-09T17:27:04.845 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.845 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe06a739:::benchmark_data_smithi179_77138_object37454:head# 2024-02-09T17:27:04.846 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.846 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe0744da:::benchmark_data_smithi179_77138_object130048:head# 2024-02-09T17:27:04.846 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.846 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe083ecb:::benchmark_data_smithi179_77138_object43528:head# 2024-02-09T17:27:04.846 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.846 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe087d06:::benchmark_data_smithi179_77138_object122339:head# 2024-02-09T17:27:04.846 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.846 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe08f400:::benchmark_data_smithi179_77138_object101240:head# 2024-02-09T17:27:04.846 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.846 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe095a23:::benchmark_data_smithi179_77138_object44300:head# 2024-02-09T17:27:04.846 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.846 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe0a6157:::benchmark_data_smithi179_77138_object66897:head# 2024-02-09T17:27:04.846 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.846 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe0bb3fd:::benchmark_data_smithi179_77138_object91952:head# 2024-02-09T17:27:04.847 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.847 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe0c25ca:::benchmark_data_smithi179_77138_object103386:head# 2024-02-09T17:27:04.847 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.847 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe0c7575:::benchmark_data_smithi179_77138_object119607:head# 2024-02-09T17:27:04.847 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.847 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe0cbe0d:::benchmark_data_smithi179_77138_object83465:head# 2024-02-09T17:27:04.847 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.847 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe0cbe81:::benchmark_data_smithi179_77138_object84474:head# 2024-02-09T17:27:04.847 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.847 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe0cea57:::benchmark_data_smithi179_77138_object125699:head# 2024-02-09T17:27:04.847 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.847 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe0d152b:::benchmark_data_smithi179_77138_object50136:head# 2024-02-09T17:27:04.847 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.847 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe0db5e6:::benchmark_data_smithi179_77138_object36279:head# 2024-02-09T17:27:04.848 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.848 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe0e056b:::benchmark_data_smithi179_77138_object32812:head# 2024-02-09T17:27:04.848 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.848 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe0e256d:::benchmark_data_smithi179_77138_object66766:head# 2024-02-09T17:27:04.848 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.848 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe0e7ab0:::benchmark_data_smithi179_77138_object66262:head# 2024-02-09T17:27:04.848 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.848 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe0f276c:::benchmark_data_smithi179_77138_object123917:head# 2024-02-09T17:27:04.848 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.848 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe0fc3f4:::benchmark_data_smithi179_77138_object119721:head# 2024-02-09T17:27:04.848 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.848 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe10085d:::benchmark_data_smithi179_77138_object113641:head# 2024-02-09T17:27:04.848 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.849 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe103e6a:::benchmark_data_smithi179_77138_object94083:head# 2024-02-09T17:27:04.849 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.849 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe1073b9:::benchmark_data_smithi179_77138_object106447:head# 2024-02-09T17:27:04.849 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.849 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe1144d1:::benchmark_data_smithi179_77138_object26564:head# 2024-02-09T17:27:04.849 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.849 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe114883:::benchmark_data_smithi179_77138_object23650:head# 2024-02-09T17:27:04.849 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.849 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe1150b8:::benchmark_data_smithi179_77138_object63975:head# 2024-02-09T17:27:04.849 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.849 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe116032:::benchmark_data_smithi179_77138_object89744:head# 2024-02-09T17:27:04.849 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.849 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe11b4f2:::benchmark_data_smithi179_77138_object43914:head# 2024-02-09T17:27:04.849 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.850 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe11e7ab:::benchmark_data_smithi179_77138_object93751:head# 2024-02-09T17:27:04.850 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.850 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe125460:::benchmark_data_smithi179_77138_object86592:head# 2024-02-09T17:27:04.850 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.850 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe1298e6:::benchmark_data_smithi179_77138_object96880:head# 2024-02-09T17:27:04.850 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.850 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe131714:::benchmark_data_smithi179_77138_object125112:head# 2024-02-09T17:27:04.850 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.850 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe161f90:::benchmark_data_smithi179_77138_object65743:head# 2024-02-09T17:27:04.850 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.850 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe16862e:::benchmark_data_smithi179_77138_object28853:head# 2024-02-09T17:27:04.850 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.850 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe16c366:::benchmark_data_smithi179_77138_object88399:head# 2024-02-09T17:27:04.850 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.851 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe16e409:::benchmark_data_smithi179_77138_object55207:head# 2024-02-09T17:27:04.851 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.851 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe170d5d:::benchmark_data_smithi179_77138_object126528:head# 2024-02-09T17:27:04.851 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.851 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe1759dd:::benchmark_data_smithi179_77138_object49710:head# 2024-02-09T17:27:04.851 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.851 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe17c667:::benchmark_data_smithi179_77138_object81516:head# 2024-02-09T17:27:04.851 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.851 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe1864f8:::benchmark_data_smithi179_77138_object76475:head# 2024-02-09T17:27:04.851 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.851 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe187427:::benchmark_data_smithi179_77138_object50565:head# 2024-02-09T17:27:04.851 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.851 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe18c913:::benchmark_data_smithi179_77138_object34554:head# 2024-02-09T17:27:04.851 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.852 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe193d27:::benchmark_data_smithi179_77138_object125855:head# 2024-02-09T17:27:04.852 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.852 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe19afb2:::benchmark_data_smithi179_77138_object114909:head# 2024-02-09T17:27:04.852 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.852 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe19ee97:::benchmark_data_smithi179_77138_object78610:head# 2024-02-09T17:27:04.852 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.852 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe1a18f1:::benchmark_data_smithi179_77138_object122522:head# 2024-02-09T17:27:04.852 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.852 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe1a472c:::benchmark_data_smithi179_77138_object22972:head# 2024-02-09T17:27:04.852 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.852 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe1ab04a:::benchmark_data_smithi179_77138_object112180:head# 2024-02-09T17:27:04.852 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.852 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe1adc69:::benchmark_data_smithi179_77138_object55705:head# 2024-02-09T17:27:04.852 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.853 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe1b3d89:::benchmark_data_smithi179_77138_object108518:head# 2024-02-09T17:27:04.853 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.853 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe1bd69b:::benchmark_data_smithi179_77138_object51185:head# 2024-02-09T17:27:04.853 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.853 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe1c7d9b:::benchmark_data_smithi179_77138_object35567:head# 2024-02-09T17:27:04.853 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.853 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe1c9e9e:::benchmark_data_smithi179_77138_object93398:head# 2024-02-09T17:27:04.853 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.853 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe1cc9e7:::benchmark_data_smithi179_77138_object92085:head# 2024-02-09T17:27:04.853 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.853 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe1ce5a2:::benchmark_data_smithi179_77138_object49332:head# 2024-02-09T17:27:04.853 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.853 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe1e497e:::benchmark_data_smithi179_77138_object38696:head# 2024-02-09T17:27:04.853 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.854 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe1e514b:::benchmark_data_smithi179_77138_object41166:head# 2024-02-09T17:27:04.854 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.858 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe1e8e4d:::benchmark_data_smithi179_77138_object101787:head# 2024-02-09T17:27:04.858 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.858 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe1ec75e:::benchmark_data_smithi179_77138_object100680:head# 2024-02-09T17:27:04.859 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.859 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe1f282e:::benchmark_data_smithi179_77138_object79519:head# 2024-02-09T17:27:04.859 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.859 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe1f55a0:::benchmark_data_smithi179_77138_object124416:head# 2024-02-09T17:27:04.859 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.859 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe1f9647:::benchmark_data_smithi179_77138_object106035:head# 2024-02-09T17:27:04.859 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.859 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe1fe320:::benchmark_data_smithi179_77138_object56637:head# 2024-02-09T17:27:04.859 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.859 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe2005b6:::benchmark_data_smithi179_77138_object82331:head# 2024-02-09T17:27:04.859 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.859 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe20d049:::benchmark_data_smithi179_77138_object107184:head# 2024-02-09T17:27:04.860 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.860 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe20d94f:::benchmark_data_smithi179_77138_object83892:head# 2024-02-09T17:27:04.860 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.860 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe21a930:::benchmark_data_smithi179_77138_object69014:head# 2024-02-09T17:27:04.860 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.860 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe223ec0:::benchmark_data_smithi179_77138_object36120:head# 2024-02-09T17:27:04.860 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.860 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe2282e8:::benchmark_data_smithi179_77138_object113130:head# 2024-02-09T17:27:04.860 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.860 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe2372e6:::benchmark_data_smithi179_77138_object74284:head# 2024-02-09T17:27:04.860 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.860 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe24925d:::benchmark_data_smithi179_77138_object119053:head# 2024-02-09T17:27:04.860 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.860 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe24d7a6:::benchmark_data_smithi179_77138_object100865:head# 2024-02-09T17:27:04.860 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.861 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe250a6d:::benchmark_data_smithi179_77138_object111086:head# 2024-02-09T17:27:04.861 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.861 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe261942:::benchmark_data_smithi179_77138_object80242:head# 2024-02-09T17:27:04.861 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.861 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe266885:::benchmark_data_smithi179_77138_object106040:head# 2024-02-09T17:27:04.861 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.861 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe26e1f7:::benchmark_data_smithi179_77138_object36767:head# 2024-02-09T17:27:04.861 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.861 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe274166:::benchmark_data_smithi179_77138_object51507:head# 2024-02-09T17:27:04.861 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.861 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe27984e:::benchmark_data_smithi179_77138_object94250:head# 2024-02-09T17:27:04.861 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.862 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe28441c:::benchmark_data_smithi179_77138_object36961:head# 2024-02-09T17:27:04.862 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.862 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe2a0146:::benchmark_data_smithi179_77138_object31231:head# 2024-02-09T17:27:04.862 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.862 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe2a7781:::benchmark_data_smithi179_77138_object56402:head# 2024-02-09T17:27:04.862 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.862 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe2ae493:::benchmark_data_smithi179_77138_object62447:head# 2024-02-09T17:27:04.862 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.862 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe2aff8c:::benchmark_data_smithi179_77138_object53383:head# 2024-02-09T17:27:04.862 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.862 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe2b03be:::benchmark_data_smithi179_77138_object114956:head# 2024-02-09T17:27:04.862 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.862 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe2ba690:::benchmark_data_smithi179_77138_object87479:head# 2024-02-09T17:27:04.862 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.863 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe2c549e:::benchmark_data_smithi179_77138_object57950:head# 2024-02-09T17:27:04.863 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.863 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe2c8f47:::benchmark_data_smithi179_77138_object73929:head# 2024-02-09T17:27:04.863 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.863 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe2cce6f:::benchmark_data_smithi179_77138_object78421:head# 2024-02-09T17:27:04.863 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.863 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe2cd08a:::benchmark_data_smithi179_77138_object89443:head# 2024-02-09T17:27:04.863 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.863 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe2d268c:::benchmark_data_smithi179_77138_object132961:head# 2024-02-09T17:27:04.863 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.863 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe2d6612:::benchmark_data_smithi179_77138_object42326:head# 2024-02-09T17:27:04.863 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.863 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe2d6d09:::benchmark_data_smithi179_77138_object45313:head# 2024-02-09T17:27:04.863 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.864 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe2ddb33:::benchmark_data_smithi179_77138_object62318:head# 2024-02-09T17:27:04.864 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.867 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe2e700d:::benchmark_data_smithi179_77138_object39264:head# 2024-02-09T17:27:04.867 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.867 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe2ee6ba:::benchmark_data_smithi179_77138_object74319:head# 2024-02-09T17:27:04.867 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.867 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe2f3087:::benchmark_data_smithi179_77138_object45615:head# 2024-02-09T17:27:04.867 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.867 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe2f3860:::benchmark_data_smithi179_77138_object97709:head# 2024-02-09T17:27:04.867 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.868 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe2f713c:::benchmark_data_smithi179_77138_object31484:head# 2024-02-09T17:27:04.868 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.868 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe2f75e4:::benchmark_data_smithi179_77138_object28468:head# 2024-02-09T17:27:04.868 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.868 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe2f94dd:::benchmark_data_smithi179_77138_object130457:head# 2024-02-09T17:27:04.868 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.868 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe2fcc21:::benchmark_data_smithi179_77138_object72740:head# 2024-02-09T17:27:04.868 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.868 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe305f87:::benchmark_data_smithi179_77138_object83852:head# 2024-02-09T17:27:04.868 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.868 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe32694e:::benchmark_data_smithi179_77138_object103515:head# 2024-02-09T17:27:04.868 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.868 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe32ba1b:::benchmark_data_smithi179_77138_object23033:head# 2024-02-09T17:27:04.868 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.869 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe32cc75:::benchmark_data_smithi179_77138_object90535:head# 2024-02-09T17:27:04.869 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.869 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe333e3e:::benchmark_data_smithi179_77138_object67913:head# 2024-02-09T17:27:04.869 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.869 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe34446b:::benchmark_data_smithi179_77138_object26073:head# 2024-02-09T17:27:04.869 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.869 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe347b00:::benchmark_data_smithi179_77138_object56014:head# 2024-02-09T17:27:04.869 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.869 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe34eb74:::benchmark_data_smithi179_77138_object82853:head# 2024-02-09T17:27:04.869 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.869 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe350122:::benchmark_data_smithi179_77138_object80761:head# 2024-02-09T17:27:04.869 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.869 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe3521dc:::benchmark_data_smithi179_77138_object48353:head# 2024-02-09T17:27:04.869 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.869 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe356be0:::benchmark_data_smithi179_77138_object101043:head# 2024-02-09T17:27:04.870 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.870 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe35922d:::benchmark_data_smithi179_77138_object48577:head# 2024-02-09T17:27:04.870 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.870 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe374bc9:::benchmark_data_smithi179_77138_object26409:head# 2024-02-09T17:27:04.870 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.870 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe397452:::benchmark_data_smithi179_77138_object98775:head# 2024-02-09T17:27:04.870 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.870 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe3b668f:::benchmark_data_smithi179_77138_object70672:head# 2024-02-09T17:27:04.870 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.870 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe3c9c44:::benchmark_data_smithi179_77138_object65865:head# 2024-02-09T17:27:04.870 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.870 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe3dd6e0:::benchmark_data_smithi179_77138_object32770:head# 2024-02-09T17:27:04.870 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.870 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe3e3784:::benchmark_data_smithi179_77138_object41181:head# 2024-02-09T17:27:04.871 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.871 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe3f60d5:::benchmark_data_smithi179_77138_object22054:head# 2024-02-09T17:27:04.871 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.871 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe4005c3:::benchmark_data_smithi179_77138_object34634:head# 2024-02-09T17:27:04.871 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.871 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe401643:::benchmark_data_smithi179_77138_object39474:head# 2024-02-09T17:27:04.871 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.871 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe418218:::benchmark_data_smithi179_77138_object67152:head# 2024-02-09T17:27:04.871 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.871 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe41bfb6:::benchmark_data_smithi179_77138_object69450:head# 2024-02-09T17:27:04.871 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.871 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe424df1:::benchmark_data_smithi179_77138_object118699:head# 2024-02-09T17:27:04.871 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.871 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe437b89:::benchmark_data_smithi179_77138_object35389:head# 2024-02-09T17:27:04.871 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.872 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe442ac2:::benchmark_data_smithi179_77138_object121954:head# 2024-02-09T17:27:04.872 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.872 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe44f084:::benchmark_data_smithi179_77138_object104938:head# 2024-02-09T17:27:04.872 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.872 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe451f3d:::benchmark_data_smithi179_77138_object32668:head# 2024-02-09T17:27:04.872 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.872 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe452a5f:::benchmark_data_smithi179_77138_object40051:head# 2024-02-09T17:27:04.872 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.875 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe4543f3:::benchmark_data_smithi179_77138_object125092:head# 2024-02-09T17:27:04.875 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.875 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe455491:::benchmark_data_smithi179_77138_object25247:head# 2024-02-09T17:27:04.875 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.875 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe45be35:::benchmark_data_smithi179_77138_object57023:head# 2024-02-09T17:27:04.875 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.875 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe462046:::benchmark_data_smithi179_77138_object92449:head# 2024-02-09T17:27:04.875 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.875 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe463a33:::benchmark_data_smithi179_77138_object80032:head# 2024-02-09T17:27:04.876 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.876 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe47a18b:::benchmark_data_smithi179_77138_object79359:head# 2024-02-09T17:27:04.876 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.876 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe480195:::benchmark_data_smithi179_77138_object126681:head# 2024-02-09T17:27:04.876 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.876 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe4815ca:::benchmark_data_smithi179_77138_object47124:head# 2024-02-09T17:27:04.876 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.876 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe4973f5:::benchmark_data_smithi179_77138_object57324:head# 2024-02-09T17:27:04.876 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.876 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe4a229d:::benchmark_data_smithi179_77138_object70612:head# 2024-02-09T17:27:04.876 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.876 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe4af0a4:::benchmark_data_smithi179_77138_object98734:head# 2024-02-09T17:27:04.876 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.877 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe4bdf94:::benchmark_data_smithi179_77138_object70328:head# 2024-02-09T17:27:04.877 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.877 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe4c15b8:::benchmark_data_smithi179_77138_object19557:head# 2024-02-09T17:27:04.877 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.877 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe4c3d81:::benchmark_data_smithi179_77138_object33432:head# 2024-02-09T17:27:04.877 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.877 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe4cad75:::benchmark_data_smithi179_77138_object85069:head# 2024-02-09T17:27:04.877 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.877 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe4d1c2c:::benchmark_data_smithi179_77138_object72385:head# 2024-02-09T17:27:04.877 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.877 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe4deef1:::benchmark_data_smithi179_77138_object51908:head# 2024-02-09T17:27:04.877 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.877 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe5051d3:::benchmark_data_smithi179_77138_object112229:head# 2024-02-09T17:27:04.877 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.877 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe50bf0b:::benchmark_data_smithi179_77138_object117455:head# 2024-02-09T17:27:04.878 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.878 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe51478a:::benchmark_data_smithi179_77138_object46953:head# 2024-02-09T17:27:04.878 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.882 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe531cb7:::benchmark_data_smithi179_77138_object94574:head# 2024-02-09T17:27:04.882 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.882 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe53c679:::benchmark_data_smithi179_77138_object81038:head# 2024-02-09T17:27:04.882 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.882 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe5405d9:::benchmark_data_smithi179_77138_object102466:head# 2024-02-09T17:27:04.882 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.882 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe54e0f7:::benchmark_data_smithi179_77138_object47974:head# 2024-02-09T17:27:04.882 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.882 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe54e51f:::benchmark_data_smithi179_77138_object109870:head# 2024-02-09T17:27:04.883 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.883 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe54ea12:::benchmark_data_smithi179_77138_object103293:head# 2024-02-09T17:27:04.883 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.883 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe55132d:::benchmark_data_smithi179_77138_object26181:head# 2024-02-09T17:27:04.883 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.883 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe5592df:::benchmark_data_smithi179_77138_object65413:head# 2024-02-09T17:27:04.883 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.883 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe55a451:::benchmark_data_smithi179_77138_object79125:head# 2024-02-09T17:27:04.883 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.883 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe55b19d:::benchmark_data_smithi179_77138_object52888:head# 2024-02-09T17:27:04.883 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.883 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe55b89d:::benchmark_data_smithi179_77138_object83647:head# 2024-02-09T17:27:04.884 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.884 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe56cad2:::benchmark_data_smithi179_77138_object55548:head# 2024-02-09T17:27:04.884 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.884 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe56f9c0:::benchmark_data_smithi179_77138_object77131:head# 2024-02-09T17:27:04.884 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.884 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe5798a2:::benchmark_data_smithi179_77138_object77793:head# 2024-02-09T17:27:04.884 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.884 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe5826d2:::benchmark_data_smithi179_77138_object44568:head# 2024-02-09T17:27:04.884 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.884 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe584336:::benchmark_data_smithi179_77138_object36256:head# 2024-02-09T17:27:04.884 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.884 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe585931:::benchmark_data_smithi179_77138_object68250:head# 2024-02-09T17:27:04.884 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.885 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe58699a:::benchmark_data_smithi179_77138_object23477:head# 2024-02-09T17:27:04.885 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.885 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe590f34:::benchmark_data_smithi179_77138_object79551:head# 2024-02-09T17:27:04.885 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.885 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe599545:::benchmark_data_smithi179_77138_object67420:head# 2024-02-09T17:27:04.885 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.885 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe5abd77:::benchmark_data_smithi179_77138_object25252:head# 2024-02-09T17:27:04.885 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.885 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe5b8186:::benchmark_data_smithi179_77138_object48037:head# 2024-02-09T17:27:04.885 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.885 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe5c0081:::benchmark_data_smithi179_77138_object94069:head# 2024-02-09T17:27:04.885 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.885 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:27:04.890 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe5c7d61:::benchmark_data_smithi179_77138_object40426:head# 2024-02-09T17:27:04.890 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.890 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe5c99ed:::benchmark_data_smithi179_77138_object114224:head# 2024-02-09T17:27:04.890 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.890 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe5c9ab7:::benchmark_data_smithi179_77138_object63358:head# 2024-02-09T17:27:04.890 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.890 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe5d8d01:::benchmark_data_smithi179_77138_object71278:head# 2024-02-09T17:27:04.890 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.890 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe5dd2cd:::benchmark_data_smithi179_77138_object105875:head# 2024-02-09T17:27:04.890 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.890 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe5e86cf:::benchmark_data_smithi179_77138_object93535:head# 2024-02-09T17:27:04.890 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.891 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe5ee57f:::benchmark_data_smithi179_77138_object114458:head# 2024-02-09T17:27:04.891 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.891 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe5f28c9:::benchmark_data_smithi179_77138_object80699:head# 2024-02-09T17:27:04.891 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.891 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe5f652a:::benchmark_data_smithi179_77138_object100080:head# 2024-02-09T17:27:04.891 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.891 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe5f9beb:::benchmark_data_smithi179_77138_object80000:head# 2024-02-09T17:27:04.891 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.891 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe60b3b7:::benchmark_data_smithi179_77138_object93773:head# 2024-02-09T17:27:04.891 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.891 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe60eb4d:::benchmark_data_smithi179_77138_object54060:head# 2024-02-09T17:27:04.891 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.891 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe6161fa:::benchmark_data_smithi179_77138_object52371:head# 2024-02-09T17:27:04.891 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.892 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe6213c8:::benchmark_data_smithi179_77138_object101650:head# 2024-02-09T17:27:04.892 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.892 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe6267a9:::benchmark_data_smithi179_77138_object81252:head# 2024-02-09T17:27:04.892 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.892 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe629041:::benchmark_data_smithi179_77138_object51404:head# 2024-02-09T17:27:04.892 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.892 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe62ce2d:::benchmark_data_smithi179_77138_object32277:head# 2024-02-09T17:27:04.892 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.892 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe62f3d7:::benchmark_data_smithi179_77138_object20176:head# 2024-02-09T17:27:04.892 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.892 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe63042d:::benchmark_data_smithi179_77138_object105510:head# 2024-02-09T17:27:04.892 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.892 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe63bd37:::benchmark_data_smithi179_77138_object28426:head# 2024-02-09T17:27:04.892 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.893 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe6465a2:::benchmark_data_smithi179_77138_object106071:head# 2024-02-09T17:27:04.893 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.893 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe64890c:::benchmark_data_smithi179_77138_object104313:head# 2024-02-09T17:27:04.893 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.893 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe64a290:::benchmark_data_smithi179_77138_object74054:head# 2024-02-09T17:27:04.893 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.893 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe652c4e:::benchmark_data_smithi179_77138_object72859:head# 2024-02-09T17:27:04.893 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.893 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe654cd0:::benchmark_data_smithi179_77138_object31319:head# 2024-02-09T17:27:04.893 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.893 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe65def2:::benchmark_data_smithi179_77138_object102524:head# 2024-02-09T17:27:04.893 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.893 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe65ee7d:::benchmark_data_smithi179_77138_object64159:head# 2024-02-09T17:27:04.893 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.893 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe66d7bb:::benchmark_data_smithi179_77138_object29790:head# 2024-02-09T17:27:04.894 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.894 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe66dbfa:::benchmark_data_smithi179_77138_object93268:head# 2024-02-09T17:27:04.894 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.894 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe674524:::benchmark_data_smithi179_77138_object111704:head# 2024-02-09T17:27:04.894 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.894 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe6827b6:::benchmark_data_smithi179_77138_object42311:head# 2024-02-09T17:27:04.894 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.896 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe68dda3:::benchmark_data_smithi179_77138_object86985:head# 2024-02-09T17:27:04.896 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.896 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe696942:::benchmark_data_smithi179_77138_object106904:head# 2024-02-09T17:27:04.896 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.896 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe6a273d:::benchmark_data_smithi179_77138_object69804:head# 2024-02-09T17:27:04.896 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.897 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe6aa985:::benchmark_data_smithi179_77138_object74480:head# 2024-02-09T17:27:04.897 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.897 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe6c1cd7:::benchmark_data_smithi179_77138_object92336:head# 2024-02-09T17:27:04.897 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.897 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe6c8d50:::benchmark_data_smithi179_77138_object33845:head# 2024-02-09T17:27:04.897 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.897 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe6cade6:::benchmark_data_smithi179_77138_object131986:head# 2024-02-09T17:27:04.897 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.899 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe6d2746:::benchmark_data_smithi179_77138_object106571:head# 2024-02-09T17:27:04.899 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.899 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe6de3c4:::benchmark_data_smithi179_77138_object32343:head# 2024-02-09T17:27:04.900 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.900 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe6f4817:::benchmark_data_smithi179_77138_object59786:head# 2024-02-09T17:27:04.900 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.900 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe6fa88b:::benchmark_data_smithi179_77138_object84917:head# 2024-02-09T17:27:04.900 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.900 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe708e59:::benchmark_data_smithi179_77138_object92104:head# 2024-02-09T17:27:04.900 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.900 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe70b068:::benchmark_data_smithi179_77138_object96440:head# 2024-02-09T17:27:04.900 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.900 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe715e73:::benchmark_data_smithi179_77138_object95111:head# 2024-02-09T17:27:04.900 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.900 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe7175b3:::benchmark_data_smithi179_77138_object95865:head# 2024-02-09T17:27:04.901 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.901 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe717dcd:::benchmark_data_smithi179_77138_object79550:head# 2024-02-09T17:27:04.901 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.901 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe718b77:::benchmark_data_smithi179_77138_object74053:head# 2024-02-09T17:27:04.901 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.901 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe71fdb4:::benchmark_data_smithi179_77138_object102231:head# 2024-02-09T17:27:04.901 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.901 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe728340:::benchmark_data_smithi179_77138_object64107:head# 2024-02-09T17:27:04.901 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.903 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe72e504:::benchmark_data_smithi179_77138_object100518:head# 2024-02-09T17:27:04.903 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.903 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe73885a:::benchmark_data_smithi179_77138_object127671:head# 2024-02-09T17:27:04.903 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.903 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe747a18:::benchmark_data_smithi179_77138_object83087:head# 2024-02-09T17:27:04.903 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.903 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe75b2aa:::benchmark_data_smithi179_77138_object38433:head# 2024-02-09T17:27:04.903 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.904 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe75b5fc:::benchmark_data_smithi179_77138_object92431:head# 2024-02-09T17:27:04.904 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.904 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe75cb86:::benchmark_data_smithi179_77138_object50756:head# 2024-02-09T17:27:04.904 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.904 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe76933a:::benchmark_data_smithi179_77138_object48715:head# 2024-02-09T17:27:04.904 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.904 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe7730db:::benchmark_data_smithi179_77138_object30789:head# 2024-02-09T17:27:04.904 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.906 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe77b562:::benchmark_data_smithi179_77138_object102841:head# 2024-02-09T17:27:04.906 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.906 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe781f39:::benchmark_data_smithi179_77138_object60065:head# 2024-02-09T17:27:04.906 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.906 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe79f0ac:::benchmark_data_smithi179_77138_object88293:head# 2024-02-09T17:27:04.906 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.906 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe7b148b:::benchmark_data_smithi179_77138_object87054:head# 2024-02-09T17:27:04.906 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.907 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe7b7357:::benchmark_data_smithi179_77138_object49814:head# 2024-02-09T17:27:04.907 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.907 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe7ba7fe:::benchmark_data_smithi179_77138_object114404:head# 2024-02-09T17:27:04.907 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.907 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe7bb882:::benchmark_data_smithi179_77138_object44238:head# 2024-02-09T17:27:04.907 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.907 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe7c17cc:::benchmark_data_smithi179_77138_object81480:head# 2024-02-09T17:27:04.907 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.907 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe7db89d:::benchmark_data_smithi179_77138_object63625:head# 2024-02-09T17:27:04.907 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.907 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe7e025b:::benchmark_data_smithi179_77138_object70234:head# 2024-02-09T17:27:04.907 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.908 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe7ed3a9:::benchmark_data_smithi179_77138_object127780:head# 2024-02-09T17:27:04.908 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.908 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe7f1aed:::benchmark_data_smithi179_77138_object24569:head# 2024-02-09T17:27:04.908 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.908 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe7f3b36:::benchmark_data_smithi179_77138_object31835:head# 2024-02-09T17:27:04.908 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.908 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe8014c6:::benchmark_data_smithi179_77138_object35873:head# 2024-02-09T17:27:04.908 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.908 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe80c2a8:::benchmark_data_smithi179_77138_object49492:head# 2024-02-09T17:27:04.908 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.908 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe80e009:::benchmark_data_smithi179_77138_object33352:head# 2024-02-09T17:27:04.908 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.908 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe8104ca:::benchmark_data_smithi179_77138_object33483:head# 2024-02-09T17:27:04.909 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.913 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe813fa1:::benchmark_data_smithi179_77138_object125464:head# 2024-02-09T17:27:04.913 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.913 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe81654e:::benchmark_data_smithi179_77138_object66148:head# 2024-02-09T17:27:04.913 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.913 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe81f730:::benchmark_data_smithi179_77138_object125667:head# 2024-02-09T17:27:04.913 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.913 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe823a9b:::benchmark_data_smithi179_77138_object27250:head# 2024-02-09T17:27:04.914 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.914 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe82c113:::benchmark_data_smithi179_77138_object29043:head# 2024-02-09T17:27:04.914 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.914 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe82e56e:::benchmark_data_smithi179_77138_object94403:head# 2024-02-09T17:27:04.914 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.914 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe832443:::benchmark_data_smithi179_77138_object121369:head# 2024-02-09T17:27:04.914 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.914 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe83c9fe:::benchmark_data_smithi179_77138_object42183:head# 2024-02-09T17:27:04.914 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.914 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe84ef79:::benchmark_data_smithi179_77138_object126725:head# 2024-02-09T17:27:04.914 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.914 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe856def:::benchmark_data_smithi179_77138_object111249:head# 2024-02-09T17:27:04.914 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.914 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe87cff8:::benchmark_data_smithi179_77138_object86726:head# 2024-02-09T17:27:04.915 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.915 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe893035:::benchmark_data_smithi179_77138_object93767:head# 2024-02-09T17:27:04.915 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.915 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe89d3d2:::benchmark_data_smithi179_77138_object58150:head# 2024-02-09T17:27:04.915 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.915 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe8a275f:::benchmark_data_smithi179_77138_object21430:head# 2024-02-09T17:27:04.915 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.915 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe8a4eb4:::benchmark_data_smithi179_77138_object62733:head# 2024-02-09T17:27:04.915 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.915 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe8a70c7:::benchmark_data_smithi179_77138_object43319:head# 2024-02-09T17:27:04.915 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.915 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe8b49ee:::benchmark_data_smithi179_77138_object133125:head# 2024-02-09T17:27:04.915 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.915 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe8c22eb:::benchmark_data_smithi179_77138_object108566:head# 2024-02-09T17:27:04.915 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.916 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe8cb5db:::benchmark_data_smithi179_77138_object105217:head# 2024-02-09T17:27:04.916 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.916 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe8d1d7e:::benchmark_data_smithi179_77138_object36026:head# 2024-02-09T17:27:04.916 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.916 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe8d4b59:::benchmark_data_smithi179_77138_object27730:head# 2024-02-09T17:27:04.916 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.916 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe8d4d50:::benchmark_data_smithi179_77138_object55501:head# 2024-02-09T17:27:04.916 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.916 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe8d84c6:::benchmark_data_smithi179_77138_object49724:head# 2024-02-09T17:27:04.916 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.916 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe8dc682:::benchmark_data_smithi179_77138_object49019:head# 2024-02-09T17:27:04.916 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.916 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe8dc80d:::benchmark_data_smithi179_77138_object112167:head# 2024-02-09T17:27:04.916 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.917 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe8e61b2:::benchmark_data_smithi179_77138_object93785:head# 2024-02-09T17:27:04.917 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.917 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe8e7ff2:::benchmark_data_smithi179_77138_object116822:head# 2024-02-09T17:27:04.917 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.917 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe8ed9b9:::benchmark_data_smithi179_77138_object79500:head# 2024-02-09T17:27:04.917 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.919 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe8f19e1:::benchmark_data_smithi179_77138_object75733:head# 2024-02-09T17:27:04.919 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.919 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe8f35da:::benchmark_data_smithi179_77138_object118267:head# 2024-02-09T17:27:04.919 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.919 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe9073c1:::benchmark_data_smithi179_77138_object27363:head# 2024-02-09T17:27:04.919 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.919 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe92544f:::benchmark_data_smithi179_77138_object74432:head# 2024-02-09T17:27:04.919 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.919 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe935218:::benchmark_data_smithi179_77138_object99203:head# 2024-02-09T17:27:04.919 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.919 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe9362e2:::benchmark_data_smithi179_77138_object100056:head# 2024-02-09T17:27:04.919 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.919 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe93e103:::benchmark_data_smithi179_77138_object73326:head# 2024-02-09T17:27:04.920 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.920 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe96351d:::benchmark_data_smithi179_77138_object127596:head# 2024-02-09T17:27:04.920 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.920 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe968b5c:::benchmark_data_smithi179_77138_object21528:head# 2024-02-09T17:27:04.920 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.920 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe96f073:::benchmark_data_smithi179_77138_object39424:head# 2024-02-09T17:27:04.920 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.920 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe9739cc:::benchmark_data_smithi179_77138_object48557:head# 2024-02-09T17:27:04.920 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.921 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe97b83c:::benchmark_data_smithi179_77138_object55999:head# 2024-02-09T17:27:04.921 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.921 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe983f00:::benchmark_data_smithi179_77138_object81256:head# 2024-02-09T17:27:04.921 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.921 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe99a31c:::benchmark_data_smithi179_77138_object84855:head# 2024-02-09T17:27:04.921 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.921 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe9ae494:::benchmark_data_smithi179_77138_object127749:head# 2024-02-09T17:27:04.921 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.921 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe9c1656:::benchmark_data_smithi179_77138_object123454:head# 2024-02-09T17:27:04.922 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.922 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe9c5809:::benchmark_data_smithi179_77138_object56058:head# 2024-02-09T17:27:04.922 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.922 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe9c584e:::benchmark_data_smithi179_77138_object110156:head# 2024-02-09T17:27:04.922 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.922 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe9d5417:::benchmark_data_smithi179_77138_object128610:head# 2024-02-09T17:27:04.922 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.922 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe9d766e:::benchmark_data_smithi179_77138_object130871:head# 2024-02-09T17:27:04.922 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.922 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe9d893f:::benchmark_data_smithi179_77138_object108550:head# 2024-02-09T17:27:04.922 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.922 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe9dcfe1:::benchmark_data_smithi179_77138_object107328:head# 2024-02-09T17:27:04.922 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.922 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe9deec7:::benchmark_data_smithi179_77138_object109205:head# 2024-02-09T17:27:04.922 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.923 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe9f15c8:::benchmark_data_smithi179_77138_object62606:head# 2024-02-09T17:27:04.923 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.923 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fe9f3cdd:::benchmark_data_smithi179_77138_object115693:head# 2024-02-09T17:27:04.923 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.924 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fea02305:::benchmark_data_smithi179_77138_object95105:head# 2024-02-09T17:27:04.924 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.924 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fea04289:::benchmark_data_smithi179_77138_object119462:head# 2024-02-09T17:27:04.924 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.924 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fea0bfe5:::benchmark_data_smithi179_77138_object54372:head# 2024-02-09T17:27:04.924 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.924 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fea0de54:::benchmark_data_smithi179_77138_object128099:head# 2024-02-09T17:27:04.925 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.925 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fea13ea6:::benchmark_data_smithi179_77138_object115962:head# 2024-02-09T17:27:04.925 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.925 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fea177de:::benchmark_data_smithi179_77138_object75100:head# 2024-02-09T17:27:04.925 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.925 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fea2308b:::benchmark_data_smithi179_77138_object79327:head# 2024-02-09T17:27:04.925 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.925 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fea2988d:::benchmark_data_smithi179_77138_object109708:head# 2024-02-09T17:27:04.925 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.925 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fea2b3d9:::benchmark_data_smithi179_77138_object25001:head# 2024-02-09T17:27:04.925 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.925 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fea32310:::benchmark_data_smithi179_77138_object66379:head# 2024-02-09T17:27:04.925 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.925 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fea37291:::benchmark_data_smithi179_77138_object39314:head# 2024-02-09T17:27:04.925 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.925 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fea42de1:::benchmark_data_smithi179_77138_object33472:head# 2024-02-09T17:27:04.926 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.926 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fea6e74a:::benchmark_data_smithi179_77138_object21796:head# 2024-02-09T17:27:04.926 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.926 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fea73194:::benchmark_data_smithi179_77138_object85379:head# 2024-02-09T17:27:04.926 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.926 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fea78049:::benchmark_data_smithi179_77138_object126582:head# 2024-02-09T17:27:04.926 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.930 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fea8941b:::benchmark_data_smithi179_77138_object61144:head# 2024-02-09T17:27:04.930 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.930 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fea89821:::benchmark_data_smithi179_77138_object50824:head# 2024-02-09T17:27:04.930 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.930 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fea8c630:::benchmark_data_smithi179_77138_object75011:head# 2024-02-09T17:27:04.930 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.930 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:feab7dc6:::benchmark_data_smithi179_77138_object122903:head# 2024-02-09T17:27:04.931 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.931 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:feaba071:::benchmark_data_smithi179_77138_object86357:head# 2024-02-09T17:27:04.931 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.931 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fead165a:::benchmark_data_smithi179_77138_object23426:head# 2024-02-09T17:27:04.931 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.931 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:feae217a:::benchmark_data_smithi179_77138_object129685:head# 2024-02-09T17:27:04.931 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.931 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:feae47dd:::benchmark_data_smithi179_77138_object123772:head# 2024-02-09T17:27:04.931 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.931 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:feaeac9e:::benchmark_data_smithi179_77138_object132054:head# 2024-02-09T17:27:04.931 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.931 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:feb00f0f:::benchmark_data_smithi179_77138_object36676:head# 2024-02-09T17:27:04.931 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.931 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:feb09f42:::benchmark_data_smithi179_77138_object46181:head# 2024-02-09T17:27:04.932 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.932 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:feb0cc0b:::benchmark_data_smithi179_77138_object106256:head# 2024-02-09T17:27:04.932 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.932 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:feb1a582:::benchmark_data_smithi179_77138_object77729:head# 2024-02-09T17:27:04.932 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.932 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:feb1e840:::benchmark_data_smithi179_77138_object121985:head# 2024-02-09T17:27:04.932 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.932 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:feb2a074:::benchmark_data_smithi179_77138_object76724:head# 2024-02-09T17:27:04.932 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.932 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:feb2a476:::benchmark_data_smithi179_77138_object87378:head# 2024-02-09T17:27:04.932 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.932 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:feb372ae:::benchmark_data_smithi179_77138_object36604:head# 2024-02-09T17:27:04.932 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.932 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:feb49fa3:::benchmark_data_smithi179_77138_object80825:head# 2024-02-09T17:27:04.933 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.933 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:feb4e82b:::benchmark_data_smithi179_77138_object86923:head# 2024-02-09T17:27:04.933 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.933 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:feb50d65:::benchmark_data_smithi179_77138_object36187:head# 2024-02-09T17:27:04.933 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.933 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:feb722ca:::benchmark_data_smithi179_77138_object63699:head# 2024-02-09T17:27:04.933 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.933 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:feb75b0f:::benchmark_data_smithi179_77138_object43277:head# 2024-02-09T17:27:04.933 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.933 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:feb77841:::benchmark_data_smithi179_77138_object101008:head# 2024-02-09T17:27:04.933 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.933 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:feb86775:::benchmark_data_smithi179_77138_object112761:head# 2024-02-09T17:27:04.933 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.934 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:04 smithi162 ceph-mon[44304]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:27:04.934 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:04 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:27:04.934 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:04 smithi162 ceph-mon[44304]: osdmap e151: 12 total, 11 up, 12 in 2024-02-09T17:27:04.934 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:04 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2831009301' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:27:04.935 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:04.937 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:feb935f9:::benchmark_data_smithi179_77138_object32337:head# 2024-02-09T17:27:04.937 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.937 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:feb99e0c:::benchmark_data_smithi179_77138_object83276:head# 2024-02-09T17:27:04.937 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.937 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:feb9d885:::benchmark_data_smithi179_77138_object133098:head# 2024-02-09T17:27:04.937 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.937 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:feba3369:::benchmark_data_smithi179_77138_object128233:head# 2024-02-09T17:27:04.937 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.938 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:febb7ddc:::benchmark_data_smithi179_77138_object40747:head# 2024-02-09T17:27:04.938 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.938 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:febbb080:::benchmark_data_smithi179_77138_object49863:head# 2024-02-09T17:27:04.938 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.938 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:febc4be0:::benchmark_data_smithi179_77138_object33309:head# 2024-02-09T17:27:04.938 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.938 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:febca6b0:::benchmark_data_smithi179_77138_object107509:head# 2024-02-09T17:27:04.938 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.938 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:febcc805:::benchmark_data_smithi179_77138_object34112:head# 2024-02-09T17:27:04.938 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.938 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:febd1946:::benchmark_data_smithi179_77138_object33915:head# 2024-02-09T17:27:04.938 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.938 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:febde508:::benchmark_data_smithi179_77138_object30562:head# 2024-02-09T17:27:04.938 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.939 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:febe1525:::benchmark_data_smithi179_77138_object32476:head# 2024-02-09T17:27:04.939 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.939 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:febe2e10:::benchmark_data_smithi179_77138_object31341:head# 2024-02-09T17:27:04.939 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.939 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:febf310f:::benchmark_data_smithi179_77138_object48624:head# 2024-02-09T17:27:04.939 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.939 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fec01f10:::benchmark_data_smithi179_77138_object132128:head# 2024-02-09T17:27:04.939 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.939 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fec04992:::benchmark_data_smithi179_77138_object81227:head# 2024-02-09T17:27:04.939 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.939 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fec1ffb2:::benchmark_data_smithi179_77138_object77936:head# 2024-02-09T17:27:04.939 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.939 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fec348de:::benchmark_data_smithi179_77138_object87888:head# 2024-02-09T17:27:04.939 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.939 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fec3912f:::benchmark_data_smithi179_77138_object23736:head# 2024-02-09T17:27:04.940 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.940 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fec3c84f:::benchmark_data_smithi179_77138_object59906:head# 2024-02-09T17:27:04.940 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.940 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fec4342e:::benchmark_data_smithi179_77138_object93793:head# 2024-02-09T17:27:04.940 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.940 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fec54acd:::benchmark_data_smithi179_77138_object34102:head# 2024-02-09T17:27:04.940 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.941 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fec6a634:::benchmark_data_smithi179_77138_object87413:head# 2024-02-09T17:27:04.941 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.941 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fec73195:::benchmark_data_smithi179_77138_object111742:head# 2024-02-09T17:27:04.941 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.941 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fec745ee:::benchmark_data_smithi179_77138_object71042:head# 2024-02-09T17:27:04.941 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.941 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fec8c8f7:::benchmark_data_smithi179_77138_object54274:head# 2024-02-09T17:27:04.941 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.941 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fec8cdfe:::benchmark_data_smithi179_77138_object109835:head# 2024-02-09T17:27:04.941 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.946 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fec8dfdc:::benchmark_data_smithi179_77138_object36090:head# 2024-02-09T17:27:04.946 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.950 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fec8f4a9:::benchmark_data_smithi179_77138_object109857:head# 2024-02-09T17:27:04.950 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.951 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fec93e34:::benchmark_data_smithi179_77138_object76873:head# 2024-02-09T17:27:04.951 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.951 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fecaa83b:::benchmark_data_smithi179_77138_object35484:head# 2024-02-09T17:27:04.951 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.951 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fecac443:::benchmark_data_smithi179_77138_object64571:head# 2024-02-09T17:27:04.951 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.952 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fecb181d:::benchmark_data_smithi179_77138_object55876:head# 2024-02-09T17:27:04.952 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.952 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fecd0c7e:::benchmark_data_smithi179_77138_object83426:head# 2024-02-09T17:27:04.952 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.952 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:feced9c4:::benchmark_data_smithi179_77138_object96867:head# 2024-02-09T17:27:04.953 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.953 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fed06e34:::benchmark_data_smithi179_77138_object95290:head# 2024-02-09T17:27:04.953 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.955 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fed1f3e3:::benchmark_data_smithi179_77138_object60524:head# 2024-02-09T17:27:04.955 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.955 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fed23f3c:::benchmark_data_smithi179_77138_object131126:head# 2024-02-09T17:27:04.955 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.955 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fed2bd31:::benchmark_data_smithi179_77138_object56819:head# 2024-02-09T17:27:04.955 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.955 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fed2c612:::benchmark_data_smithi179_77138_object20221:head# 2024-02-09T17:27:04.955 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.955 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fed30865:::benchmark_data_smithi179_77138_object130758:head# 2024-02-09T17:27:04.955 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.955 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fed33147:::benchmark_data_smithi179_77138_object129545:head# 2024-02-09T17:27:04.955 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.956 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fed33716:::benchmark_data_smithi179_77138_object35848:head# 2024-02-09T17:27:04.956 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.956 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fed430f3:::benchmark_data_smithi179_77138_object50017:head# 2024-02-09T17:27:04.956 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.956 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fed47c9f:::benchmark_data_smithi179_77138_object36402:head# 2024-02-09T17:27:04.956 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.956 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fed4e2a3:::benchmark_data_smithi179_77138_object117652:head# 2024-02-09T17:27:04.956 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.956 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fed78ec5:::benchmark_data_smithi179_77138_object89509:head# 2024-02-09T17:27:04.956 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.958 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fed7ab2f:::benchmark_data_smithi179_77138_object41889:head# 2024-02-09T17:27:04.958 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.958 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fed7c1a6:::benchmark_data_smithi179_77138_object124409:head# 2024-02-09T17:27:04.958 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.958 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fed84db5:::benchmark_data_smithi179_77138_object90750:head# 2024-02-09T17:27:04.958 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.958 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fed9022e:::benchmark_data_smithi179_77138_object44789:head# 2024-02-09T17:27:04.959 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.959 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fedc7cba:::benchmark_data_smithi179_77138_object90560:head# 2024-02-09T17:27:04.959 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.959 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fedca3b2:::benchmark_data_smithi179_77138_object63040:head# 2024-02-09T17:27:04.959 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.959 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fedcddcd:::benchmark_data_smithi179_77138_object126856:head# 2024-02-09T17:27:04.959 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.959 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:feddf5d3:::benchmark_data_smithi179_77138_object91135:head# 2024-02-09T17:27:04.959 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.959 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fedecce5:::benchmark_data_smithi179_77138_object98167:head# 2024-02-09T17:27:04.959 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.959 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fedf20dd:::benchmark_data_smithi179_77138_object77237:head# 2024-02-09T17:27:04.959 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.959 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fedf4703:::benchmark_data_smithi179_77138_object37465:head# 2024-02-09T17:27:04.960 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.960 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fedff09f:::benchmark_data_smithi179_77138_object105807:head# 2024-02-09T17:27:04.960 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.960 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fee0f421:::benchmark_data_smithi179_77138_object83089:head# 2024-02-09T17:27:04.960 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.960 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fee1091e:::benchmark_data_smithi179_77138_object131969:head# 2024-02-09T17:27:04.961 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.961 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fee14306:::benchmark_data_smithi179_77138_object110789:head# 2024-02-09T17:27:04.961 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.961 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fee20a05:::benchmark_data_smithi179_77138_object53956:head# 2024-02-09T17:27:04.961 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.962 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fee23bb1:::benchmark_data_smithi179_77138_object96411:head# 2024-02-09T17:27:04.962 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.962 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fee27673:::benchmark_data_smithi179_77138_object21054:head# 2024-02-09T17:27:04.962 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.962 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fee3665d:::benchmark_data_smithi179_77138_object61978:head# 2024-02-09T17:27:04.963 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.963 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fee3f07c:::benchmark_data_smithi179_77138_object30326:head# 2024-02-09T17:27:04.963 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.963 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fee44171:::benchmark_data_smithi179_77138_object122367:head# 2024-02-09T17:27:04.963 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.963 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fee447bf:::benchmark_data_smithi179_77138_object85525:head# 2024-02-09T17:27:04.963 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.963 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fee4d610:::benchmark_data_smithi179_77138_object75435:head# 2024-02-09T17:27:04.963 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.963 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fee4f0e0:::benchmark_data_smithi179_77138_object61331:head# 2024-02-09T17:27:04.963 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.963 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fee4f5e7:::benchmark_data_smithi179_77138_object32200:head# 2024-02-09T17:27:04.963 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.963 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fee6df5f:::benchmark_data_smithi179_77138_object104936:head# 2024-02-09T17:27:04.963 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.965 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fee7368d:::benchmark_data_smithi179_77138_object78662:head# 2024-02-09T17:27:04.965 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.965 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fee74718:::benchmark_data_smithi179_77138_object33677:head# 2024-02-09T17:27:04.965 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.965 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fee76145:::benchmark_data_smithi179_77138_object26316:head# 2024-02-09T17:27:04.965 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.965 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fee7dd04:::benchmark_data_smithi179_77138_object31432:head# 2024-02-09T17:27:04.965 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.965 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fee92a2f:::benchmark_data_smithi179_77138_object34430:head# 2024-02-09T17:27:04.965 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.966 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fee98717:::benchmark_data_smithi179_77138_object58817:head# 2024-02-09T17:27:04.966 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.966 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fee9efa5:::benchmark_data_smithi179_77138_object104789:head# 2024-02-09T17:27:04.966 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.966 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:feea4029:::benchmark_data_smithi179_77138_object94851:head# 2024-02-09T17:27:04.966 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.966 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:feea819e:::benchmark_data_smithi179_77138_object82153:head# 2024-02-09T17:27:04.966 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.967 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:feeaca1b:::benchmark_data_smithi179_77138_object118352:head# 2024-02-09T17:27:04.967 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.967 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:feeaf6df:::benchmark_data_smithi179_77138_object52906:head# 2024-02-09T17:27:04.967 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.967 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:feebe508:::benchmark_data_smithi179_77138_object55521:head# 2024-02-09T17:27:04.967 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.967 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:feecb98d:::benchmark_data_smithi179_77138_object53020:head# 2024-02-09T17:27:04.967 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.967 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:feed1157:::benchmark_data_smithi179_77138_object115748:head# 2024-02-09T17:27:04.968 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.974 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:feed4cc6:::benchmark_data_smithi179_77138_object42143:head# 2024-02-09T17:27:04.974 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.975 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:feee028c:::benchmark_data_smithi179_77138_object26235:head# 2024-02-09T17:27:04.975 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.975 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:feee12c7:::benchmark_data_smithi179_77138_object65950:head# 2024-02-09T17:27:04.975 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.976 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:feee961c:::benchmark_data_smithi179_77138_object83002:head# 2024-02-09T17:27:04.976 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.976 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:feeeb6f9:::benchmark_data_smithi179_77138_object129374:head# 2024-02-09T17:27:04.976 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.976 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fef0555b:::benchmark_data_smithi179_77138_object89229:head# 2024-02-09T17:27:04.976 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.976 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fef0c83d:::benchmark_data_smithi179_77138_object117369:head# 2024-02-09T17:27:04.976 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.977 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:27:04.979 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fef1096a:::benchmark_data_smithi179_77138_object39436:head# 2024-02-09T17:27:04.979 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.979 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fef2cc1c:::benchmark_data_smithi179_77138_object63092:head# 2024-02-09T17:27:04.979 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.979 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fef3e0c8:::benchmark_data_smithi179_77138_object117211:head# 2024-02-09T17:27:04.979 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.979 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fef3f956:::benchmark_data_smithi179_77138_object22990:head# 2024-02-09T17:27:04.980 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.980 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fef446bb:::benchmark_data_smithi179_77138_object53726:head# 2024-02-09T17:27:04.980 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.980 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fef4619a:::benchmark_data_smithi179_77138_object27128:head# 2024-02-09T17:27:04.980 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.980 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fef49a6a:::benchmark_data_smithi179_77138_object93258:head# 2024-02-09T17:27:04.980 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.980 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fef4db6e:::benchmark_data_smithi179_77138_object83442:head# 2024-02-09T17:27:04.980 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.980 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fef6e960:::benchmark_data_smithi179_77138_object60379:head# 2024-02-09T17:27:04.980 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.980 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fef836df:::benchmark_data_smithi179_77138_object114069:head# 2024-02-09T17:27:04.980 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.980 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fef8e771:::benchmark_data_smithi179_77138_object89139:head# 2024-02-09T17:27:04.980 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.981 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fef91944:::benchmark_data_smithi179_77138_object120000:head# 2024-02-09T17:27:04.981 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.981 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fefba326:::benchmark_data_smithi179_77138_object86737:head# 2024-02-09T17:27:04.981 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.981 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fefbf5c3:::benchmark_data_smithi179_77138_object69418:head# 2024-02-09T17:27:04.981 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:04.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:27:04.986 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fefc6598:::benchmark_data_smithi179_77138_object110208:head# 2024-02-09T17:27:04.986 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.986 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fefceefb:::benchmark_data_smithi179_77138_object97764:head# 2024-02-09T17:27:04.986 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.986 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fefd7921:::benchmark_data_smithi179_77138_object63340:head# 2024-02-09T17:27:04.986 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.986 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fefe7479:::benchmark_data_smithi179_77138_object118802:head# 2024-02-09T17:27:04.987 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.987 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:feff5f57:::benchmark_data_smithi179_77138_object20707:head# 2024-02-09T17:27:04.987 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.987 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff006362:::benchmark_data_smithi179_77138_object39079:head# 2024-02-09T17:27:04.987 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.987 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff00c5f9:::benchmark_data_smithi179_77138_object129696:head# 2024-02-09T17:27:04.987 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.987 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff019799:::benchmark_data_smithi179_77138_object73547:head# 2024-02-09T17:27:04.987 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.987 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff051e6e:::benchmark_data_smithi179_77138_object113861:head# 2024-02-09T17:27:04.987 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.987 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff05e5bc:::benchmark_data_smithi179_77138_object96099:head# 2024-02-09T17:27:04.987 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.987 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff0681f0:::benchmark_data_smithi179_77138_object122863:head# 2024-02-09T17:27:04.987 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.987 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff0883a9:::benchmark_data_smithi179_77138_object128615:head# 2024-02-09T17:27:04.988 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.988 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff08e3f4:::benchmark_data_smithi179_77138_object64302:head# 2024-02-09T17:27:04.988 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.988 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff09ce39:::benchmark_data_smithi179_77138_object48556:head# 2024-02-09T17:27:04.988 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.988 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff0a272a:::benchmark_data_smithi179_77138_object50327:head# 2024-02-09T17:27:04.988 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.988 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff0a6cfa:::benchmark_data_smithi179_77138_object118629:head# 2024-02-09T17:27:04.988 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.988 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff0a9e9d:::benchmark_data_smithi179_77138_object125745:head# 2024-02-09T17:27:04.988 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.988 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff0adde0:::benchmark_data_smithi179_77138_object101148:head# 2024-02-09T17:27:04.988 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.988 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff0b659b:::benchmark_data_smithi179_77138_object124456:head# 2024-02-09T17:27:04.988 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.989 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff0c6502:::benchmark_data_smithi179_77138_object118585:head# 2024-02-09T17:27:04.989 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.989 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff0cb190:::benchmark_data_smithi179_77138_object77831:head# 2024-02-09T17:27:04.989 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.989 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff0d0d54:::benchmark_data_smithi179_77138_object68536:head# 2024-02-09T17:27:04.989 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.989 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff0d4b9f:::benchmark_data_smithi179_77138_object30642:head# 2024-02-09T17:27:04.989 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.989 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff0de4d9:::benchmark_data_smithi179_77138_object129047:head# 2024-02-09T17:27:04.989 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.989 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff0e482b:::benchmark_data_smithi179_77138_object30683:head# 2024-02-09T17:27:04.989 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.990 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff0e7d2b:::benchmark_data_smithi179_77138_object99025:head# 2024-02-09T17:27:04.990 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.990 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff0f8798:::benchmark_data_smithi179_77138_object54641:head# 2024-02-09T17:27:04.990 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.991 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff0fd3b5:::benchmark_data_smithi179_77138_object97285:head# 2024-02-09T17:27:04.991 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.991 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff10188e:::benchmark_data_smithi179_77138_object61790:head# 2024-02-09T17:27:04.991 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.992 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff120a58:::benchmark_data_smithi179_77138_object39782:head# 2024-02-09T17:27:04.992 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.992 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff122fd0:::benchmark_data_smithi179_77138_object97994:head# 2024-02-09T17:27:04.992 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.992 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff131123:::benchmark_data_smithi179_77138_object33555:head# 2024-02-09T17:27:04.992 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.992 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff138902:::benchmark_data_smithi179_77138_object62451:head# 2024-02-09T17:27:04.992 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.993 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff141195:::benchmark_data_smithi179_77138_object43326:head# 2024-02-09T17:27:04.993 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.998 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff154065:::benchmark_data_smithi179_77138_object124634:head# 2024-02-09T17:27:04.998 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.999 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff15971e:::benchmark_data_smithi179_77138_object38439:head# 2024-02-09T17:27:04.999 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.999 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff164a51:::benchmark_data_smithi179_77138_object114483:head# 2024-02-09T17:27:04.999 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:04.999 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff172ae1:::benchmark_data_smithi179_77138_object88551:head# 2024-02-09T17:27:04.999 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.000 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff17f209:::benchmark_data_smithi179_77138_object118274:head# 2024-02-09T17:27:05.000 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.000 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff180f36:::benchmark_data_smithi179_77138_object126617:head# 2024-02-09T17:27:05.000 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.000 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff1877da:::benchmark_data_smithi179_77138_object105544:head# 2024-02-09T17:27:05.000 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.001 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff18a859:::benchmark_data_smithi179_77138_object116077:head# 2024-02-09T17:27:05.001 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.002 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff192c70:::benchmark_data_smithi179_77138_object130753:head# 2024-02-09T17:27:05.002 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.002 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff1a24a7:::benchmark_data_smithi179_77138_object67897:head# 2024-02-09T17:27:05.002 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.002 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff1a3835:::benchmark_data_smithi179_77138_object28871:head# 2024-02-09T17:27:05.002 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.003 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff1a4845:::benchmark_data_smithi179_77138_object42092:head# 2024-02-09T17:27:05.003 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.003 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff1d6dd3:::benchmark_data_smithi179_77138_object107917:head# 2024-02-09T17:27:05.003 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.003 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff1ea826:::benchmark_data_smithi179_77138_object132951:head# 2024-02-09T17:27:05.003 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.003 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff1f6804:::benchmark_data_smithi179_77138_object130691:head# 2024-02-09T17:27:05.004 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.010 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff21a2e7:::benchmark_data_smithi179_77138_object22721:head# 2024-02-09T17:27:05.010 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.010 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff222800:::benchmark_data_smithi179_77138_object109339:head# 2024-02-09T17:27:05.010 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.015 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff22d98b:::benchmark_data_smithi179_77138_object96135:head# 2024-02-09T17:27:05.015 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.015 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff22deb2:::benchmark_data_smithi179_77138_object103957:head# 2024-02-09T17:27:05.015 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.015 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff239ee5:::benchmark_data_smithi179_77138_object93828:head# 2024-02-09T17:27:05.015 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.015 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff245685:::benchmark_data_smithi179_77138_object114673:head# 2024-02-09T17:27:05.015 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.015 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff24af6c:::benchmark_data_smithi179_77138_object102459:head# 2024-02-09T17:27:05.015 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.015 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff25cf1b:::benchmark_data_smithi179_77138_object67078:head# 2024-02-09T17:27:05.016 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.016 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff262869:::benchmark_data_smithi179_77138_object89647:head# 2024-02-09T17:27:05.016 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.016 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff2645e6:::benchmark_data_smithi179_77138_object84468:head# 2024-02-09T17:27:05.016 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.016 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff26548e:::benchmark_data_smithi179_77138_object47592:head# 2024-02-09T17:27:05.016 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.016 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff270c9e:::benchmark_data_smithi179_77138_object23749:head# 2024-02-09T17:27:05.016 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.016 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff27261d:::benchmark_data_smithi179_77138_object105374:head# 2024-02-09T17:27:05.016 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.016 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff27f6c2:::benchmark_data_smithi179_77138_object68783:head# 2024-02-09T17:27:05.016 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.016 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff292b22:::benchmark_data_smithi179_77138_object91604:head# 2024-02-09T17:27:05.017 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.017 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff2940f9:::benchmark_data_smithi179_77138_object70907:head# 2024-02-09T17:27:05.017 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.019 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff298e90:::benchmark_data_smithi179_77138_object49838:head# 2024-02-09T17:27:05.019 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.019 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff29fec1:::benchmark_data_smithi179_77138_object68805:head# 2024-02-09T17:27:05.019 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.019 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff2a0a35:::benchmark_data_smithi179_77138_object107637:head# 2024-02-09T17:27:05.019 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.019 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff2a1414:::benchmark_data_smithi179_77138_object33739:head# 2024-02-09T17:27:05.019 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.019 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff2b1b8b:::benchmark_data_smithi179_77138_object44770:head# 2024-02-09T17:27:05.020 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.020 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff2bc313:::benchmark_data_smithi179_77138_object65917:head# 2024-02-09T17:27:05.020 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.020 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff2cb328:::benchmark_data_smithi179_77138_object30873:head# 2024-02-09T17:27:05.020 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.020 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff2cd011:::benchmark_data_smithi179_77138_object48366:head# 2024-02-09T17:27:05.020 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.020 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff2cd2e6:::benchmark_data_smithi179_77138_object22666:head# 2024-02-09T17:27:05.020 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.020 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff2dd79a:::benchmark_data_smithi179_77138_object108562:head# 2024-02-09T17:27:05.020 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.020 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff2ea0f8:::benchmark_data_smithi179_77138_object65775:head# 2024-02-09T17:27:05.020 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.020 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff2ec20e:::benchmark_data_smithi179_77138_object66293:head# 2024-02-09T17:27:05.021 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.021 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff2edffb:::benchmark_data_smithi179_77138_object69884:head# 2024-02-09T17:27:05.021 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.023 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff2ef597:::benchmark_data_smithi179_77138_object63826:head# 2024-02-09T17:27:05.023 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.024 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff2f8db3:::benchmark_data_smithi179_77138_object82826:head# 2024-02-09T17:27:05.024 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.024 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff308bf6:::benchmark_data_smithi179_77138_object61822:head# 2024-02-09T17:27:05.024 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.024 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff318800:::benchmark_data_smithi179_77138_object78515:head# 2024-02-09T17:27:05.024 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.024 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff3205a4:::benchmark_data_smithi179_77138_object74381:head# 2024-02-09T17:27:05.024 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.024 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff32566d:::benchmark_data_smithi179_77138_object114399:head# 2024-02-09T17:27:05.024 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.024 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff327c19:::benchmark_data_smithi179_77138_object116796:head# 2024-02-09T17:27:05.024 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.025 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff32a665:::benchmark_data_smithi179_77138_object129551:head# 2024-02-09T17:27:05.025 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.025 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff330b42:::benchmark_data_smithi179_77138_object25910:head# 2024-02-09T17:27:05.025 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.025 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff331af7:::benchmark_data_smithi179_77138_object117648:head# 2024-02-09T17:27:05.025 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.025 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff33ce0d:::benchmark_data_smithi179_77138_object77673:head# 2024-02-09T17:27:05.025 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.025 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff340dd3:::benchmark_data_smithi179_77138_object25896:head# 2024-02-09T17:27:05.025 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.025 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff34addd:::benchmark_data_smithi179_77138_object62209:head# 2024-02-09T17:27:05.025 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.025 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff34b2b3:::benchmark_data_smithi179_77138_object129343:head# 2024-02-09T17:27:05.025 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.026 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff34e55f:::benchmark_data_smithi179_77138_object73451:head# 2024-02-09T17:27:05.026 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.026 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff34f880:::benchmark_data_smithi179_77138_object78454:head# 2024-02-09T17:27:05.026 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.026 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff35542b:::benchmark_data_smithi179_77138_object71142:head# 2024-02-09T17:27:05.026 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.027 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff36974d:::benchmark_data_smithi179_77138_object40654:head# 2024-02-09T17:27:05.027 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.027 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff36f3f4:::benchmark_data_smithi179_77138_object44434:head# 2024-02-09T17:27:05.027 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.027 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff373d35:::benchmark_data_smithi179_77138_object116085:head# 2024-02-09T17:27:05.027 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.027 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff3789c2:::benchmark_data_smithi179_77138_object23573:head# 2024-02-09T17:27:05.027 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.027 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff393588:::benchmark_data_smithi179_77138_object60422:head# 2024-02-09T17:27:05.027 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.027 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff395d61:::benchmark_data_smithi179_77138_object117329:head# 2024-02-09T17:27:05.027 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.027 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff39b6a3:::benchmark_data_smithi179_77138_object27343:head# 2024-02-09T17:27:05.027 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.028 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff3b5967:::benchmark_data_smithi179_77138_object96393:head# 2024-02-09T17:27:05.028 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.028 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff3c7baa:::benchmark_data_smithi179_77138_object119938:head# 2024-02-09T17:27:05.028 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.029 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff3ce26e:::benchmark_data_smithi179_77138_object25963:head# 2024-02-09T17:27:05.029 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.029 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff3d9a3f:::benchmark_data_smithi179_77138_object104468:head# 2024-02-09T17:27:05.029 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.029 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff3daf0d:::benchmark_data_smithi179_77138_object69772:head# 2024-02-09T17:27:05.029 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.029 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff3dc199:::benchmark_data_smithi179_77138_object89382:head# 2024-02-09T17:27:05.030 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.030 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff3ddc34:::benchmark_data_smithi179_77138_object125534:head# 2024-02-09T17:27:05.030 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.030 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff3e5333:::benchmark_data_smithi179_77138_object111005:head# 2024-02-09T17:27:05.030 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.030 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff3f3fad:::benchmark_data_smithi179_77138_object33327:head# 2024-02-09T17:27:05.030 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.030 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff40b6e4:::benchmark_data_smithi179_77138_object31971:head# 2024-02-09T17:27:05.030 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.030 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff4286e6:::benchmark_data_smithi179_77138_object71842:head# 2024-02-09T17:27:05.030 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.030 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff43015d:::benchmark_data_smithi179_77138_object39428:head# 2024-02-09T17:27:05.030 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.032 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff4302e0:::benchmark_data_smithi179_77138_object90955:head# 2024-02-09T17:27:05.033 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.033 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff435473:::benchmark_data_smithi179_77138_object125537:head# 2024-02-09T17:27:05.033 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.033 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff43ce97:::benchmark_data_smithi179_77138_object76717:head# 2024-02-09T17:27:05.033 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.033 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff442e6a:::benchmark_data_smithi179_77138_object65027:head# 2024-02-09T17:27:05.033 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.033 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff443d4f:::benchmark_data_smithi179_77138_object91204:head# 2024-02-09T17:27:05.033 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.033 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff448f0b:::benchmark_data_smithi179_77138_object131429:head# 2024-02-09T17:27:05.033 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.033 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff449afe:::benchmark_data_smithi179_77138_object54520:head# 2024-02-09T17:27:05.033 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.033 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff4592ae:::benchmark_data_smithi179_77138_object44704:head# 2024-02-09T17:27:05.034 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.034 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff45b7e2:::benchmark_data_smithi179_77138_object85064:head# 2024-02-09T17:27:05.034 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.034 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff45c582:::benchmark_data_smithi179_77138_object36032:head# 2024-02-09T17:27:05.034 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.034 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff464569:::benchmark_data_smithi179_77138_object126064:head# 2024-02-09T17:27:05.034 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.034 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff47776e:::benchmark_data_smithi179_77138_object78801:head# 2024-02-09T17:27:05.034 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.034 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff478ebf:::benchmark_data_smithi179_77138_object58976:head# 2024-02-09T17:27:05.034 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.035 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff47a164:::benchmark_data_smithi179_77138_object121102:head# 2024-02-09T17:27:05.035 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.036 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff480c21:::benchmark_data_smithi179_77138_object106434:head# 2024-02-09T17:27:05.036 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.036 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff491951:::benchmark_data_smithi179_77138_object102293:head# 2024-02-09T17:27:05.036 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.036 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff4a9b67:::benchmark_data_smithi179_77138_object46746:head# 2024-02-09T17:27:05.036 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.036 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff4a9f0e:::benchmark_data_smithi179_77138_object96523:head# 2024-02-09T17:27:05.036 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.036 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff4b7a32:::benchmark_data_smithi179_77138_object95280:head# 2024-02-09T17:27:05.037 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.037 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff4bc702:::benchmark_data_smithi179_77138_object122927:head# 2024-02-09T17:27:05.037 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.037 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff4be6ef:::benchmark_data_smithi179_77138_object123264:head# 2024-02-09T17:27:05.037 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.038 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff4c15c4:::benchmark_data_smithi179_77138_object63972:head# 2024-02-09T17:27:05.038 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.039 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff4c7da9:::benchmark_data_smithi179_77138_object66396:head# 2024-02-09T17:27:05.039 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.039 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff4d18af:::benchmark_data_smithi179_77138_object54065:head# 2024-02-09T17:27:05.039 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.039 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff4d8690:::benchmark_data_smithi179_77138_object64934:head# 2024-02-09T17:27:05.039 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.039 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff4e7fd3:::benchmark_data_smithi179_77138_object38982:head# 2024-02-09T17:27:05.039 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.039 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff4ef156:::benchmark_data_smithi179_77138_object51337:head# 2024-02-09T17:27:05.039 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.039 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff4fca5e:::benchmark_data_smithi179_77138_object122678:head# 2024-02-09T17:27:05.039 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.043 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff500c7a:::benchmark_data_smithi179_77138_object37309:head# 2024-02-09T17:27:05.043 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.043 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff506d09:::benchmark_data_smithi179_77138_object106762:head# 2024-02-09T17:27:05.043 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.043 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff50a38e:::benchmark_data_smithi179_77138_object131099:head# 2024-02-09T17:27:05.043 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.043 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff535ed3:::benchmark_data_smithi179_77138_object98955:head# 2024-02-09T17:27:05.043 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.043 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff53c36f:::benchmark_data_smithi179_77138_object61607:head# 2024-02-09T17:27:05.043 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.043 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff543b7b:::benchmark_data_smithi179_77138_object75422:head# 2024-02-09T17:27:05.044 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.044 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff546105:::benchmark_data_smithi179_77138_object35718:head# 2024-02-09T17:27:05.044 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.044 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff548f95:::benchmark_data_smithi179_77138_object94275:head# 2024-02-09T17:27:05.044 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.044 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff558076:::benchmark_data_smithi179_77138_object92704:head# 2024-02-09T17:27:05.044 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.044 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff55c924:::benchmark_data_smithi179_77138_object87566:head# 2024-02-09T17:27:05.044 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.044 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff5668fc:::benchmark_data_smithi179_77138_object99819:head# 2024-02-09T17:27:05.044 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.044 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff56b058:::benchmark_data_smithi179_77138_object55188:head# 2024-02-09T17:27:05.044 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.044 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff56b792:::benchmark_data_smithi179_77138_object110272:head# 2024-02-09T17:27:05.045 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.045 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff570521:::benchmark_data_smithi179_77138_object77889:head# 2024-02-09T17:27:05.045 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.045 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff570c53:::benchmark_data_smithi179_77138_object108035:head# 2024-02-09T17:27:05.045 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.045 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff576459:::benchmark_data_smithi179_77138_object79090:head# 2024-02-09T17:27:05.045 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.045 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff576b6b:::benchmark_data_smithi179_77138_object21334:head# 2024-02-09T17:27:05.045 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.047 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff57f039:::benchmark_data_smithi179_77138_object92606:head# 2024-02-09T17:27:05.047 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.047 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff582066:::benchmark_data_smithi179_77138_object79652:head# 2024-02-09T17:27:05.047 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.047 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff5846c1:::benchmark_data_smithi179_77138_object25173:head# 2024-02-09T17:27:05.047 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.047 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff586fba:::benchmark_data_smithi179_77138_object78054:head# 2024-02-09T17:27:05.047 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.048 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff592b9c:::benchmark_data_smithi179_77138_object104629:head# 2024-02-09T17:27:05.048 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.048 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff5a7285:::benchmark_data_smithi179_77138_object41191:head# 2024-02-09T17:27:05.048 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.048 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff5c587f:::benchmark_data_smithi179_77138_object129693:head# 2024-02-09T17:27:05.048 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.048 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff5e08b8:::benchmark_data_smithi179_77138_object76458:head# 2024-02-09T17:27:05.048 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.048 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff5e7f21:::benchmark_data_smithi179_77138_object84594:head# 2024-02-09T17:27:05.048 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.048 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff5eecbd:::benchmark_data_smithi179_77138_object22887:head# 2024-02-09T17:27:05.048 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.048 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff604da7:::benchmark_data_smithi179_77138_object132769:head# 2024-02-09T17:27:05.049 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.049 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff60819b:::benchmark_data_smithi179_77138_object130191:head# 2024-02-09T17:27:05.049 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.051 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff609ac6:::benchmark_data_smithi179_77138_object123096:head# 2024-02-09T17:27:05.051 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.051 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff60b234:::benchmark_data_smithi179_77138_object98136:head# 2024-02-09T17:27:05.051 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.052 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff619778:::benchmark_data_smithi179_77138_object36142:head# 2024-02-09T17:27:05.052 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.052 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff627541:::benchmark_data_smithi179_77138_object42807:head# 2024-02-09T17:27:05.052 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.052 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff628895:::benchmark_data_smithi179_77138_object21922:head# 2024-02-09T17:27:05.052 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.058 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff62b882:::benchmark_data_smithi179_77138_object94144:head# 2024-02-09T17:27:05.058 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.058 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff63000b:::benchmark_data_smithi179_77138_object59343:head# 2024-02-09T17:27:05.058 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.058 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff638a2d:::benchmark_data_smithi179_77138_object113455:head# 2024-02-09T17:27:05.058 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.058 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff6512f7:::benchmark_data_smithi179_77138_object33575:head# 2024-02-09T17:27:05.058 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.058 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff6568a9:::benchmark_data_smithi179_77138_object87115:head# 2024-02-09T17:27:05.058 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.058 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff65a5d7:::benchmark_data_smithi179_77138_object118518:head# 2024-02-09T17:27:05.059 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.059 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff661100:::benchmark_data_smithi179_77138_object89917:head# 2024-02-09T17:27:05.059 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.060 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff66ab41:::benchmark_data_smithi179_77138_object49055:head# 2024-02-09T17:27:05.060 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.060 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff67ef64:::benchmark_data_smithi179_77138_object114400:head# 2024-02-09T17:27:05.061 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.061 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff68520d:::benchmark_data_smithi179_77138_object117195:head# 2024-02-09T17:27:05.061 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.061 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff68911f:::benchmark_data_smithi179_77138_object21593:head# 2024-02-09T17:27:05.061 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.061 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff68d074:::benchmark_data_smithi179_77138_object99173:head# 2024-02-09T17:27:05.061 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.061 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff6917d0:::benchmark_data_smithi179_77138_object92833:head# 2024-02-09T17:27:05.061 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.061 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff694997:::benchmark_data_smithi179_77138_object82350:head# 2024-02-09T17:27:05.061 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.061 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff6a169b:::benchmark_data_smithi179_77138_object45482:head# 2024-02-09T17:27:05.061 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.062 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff6aa81f:::benchmark_data_smithi179_77138_object94604:head# 2024-02-09T17:27:05.062 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.065 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff6af65f:::benchmark_data_smithi179_77138_object22308:head# 2024-02-09T17:27:05.065 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.066 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff6b3de3:::benchmark_data_smithi179_77138_object107343:head# 2024-02-09T17:27:05.066 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.066 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff6bcf12:::benchmark_data_smithi179_77138_object103407:head# 2024-02-09T17:27:05.066 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.066 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff6c8781:::benchmark_data_smithi179_77138_object61673:head# 2024-02-09T17:27:05.066 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.068 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff6ca901:::benchmark_data_smithi179_77138_object34133:head# 2024-02-09T17:27:05.068 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.068 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff6cf4e8:::benchmark_data_smithi179_77138_object69861:head# 2024-02-09T17:27:05.068 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.068 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff6d3d78:::benchmark_data_smithi179_77138_object70972:head# 2024-02-09T17:27:05.068 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.068 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff6d3ee1:::benchmark_data_smithi179_77138_object36602:head# 2024-02-09T17:27:05.068 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.068 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff6d406a:::benchmark_data_smithi179_77138_object69835:head# 2024-02-09T17:27:05.069 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.069 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff6e40e1:::benchmark_data_smithi179_77138_object89485:head# 2024-02-09T17:27:05.069 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.069 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff6ee750:::benchmark_data_smithi179_77138_object72946:head# 2024-02-09T17:27:05.069 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.069 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff6f1ebd:::benchmark_data_smithi179_77138_object124810:head# 2024-02-09T17:27:05.069 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.072 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff6f7edd:::benchmark_data_smithi179_77138_object90606:head# 2024-02-09T17:27:05.072 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.073 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff6fc846:::benchmark_data_smithi179_77138_object125979:head# 2024-02-09T17:27:05.073 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.073 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff711edb:::benchmark_data_smithi179_77138_object24663:head# 2024-02-09T17:27:05.073 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.073 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff721d0f:::benchmark_data_smithi179_77138_object122370:head# 2024-02-09T17:27:05.073 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.073 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff7237e4:::benchmark_data_smithi179_77138_object66742:head# 2024-02-09T17:27:05.073 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.073 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff724872:::benchmark_data_smithi179_77138_object44144:head# 2024-02-09T17:27:05.073 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.073 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff72a4ef:::benchmark_data_smithi179_77138_object53301:head# 2024-02-09T17:27:05.073 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.073 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff75b000:::benchmark_data_smithi179_77138_object42094:head# 2024-02-09T17:27:05.074 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.074 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff767580:::benchmark_data_smithi179_77138_object53148:head# 2024-02-09T17:27:05.074 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.074 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff76be6d:::benchmark_data_smithi179_77138_object64960:head# 2024-02-09T17:27:05.074 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.074 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff76d0ad:::benchmark_data_smithi179_77138_object115247:head# 2024-02-09T17:27:05.074 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.074 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff7716b4:::benchmark_data_smithi179_77138_object102331:head# 2024-02-09T17:27:05.074 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.074 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff771f5e:::benchmark_data_smithi179_77138_object97626:head# 2024-02-09T17:27:05.074 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.074 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff788bdb:::benchmark_data_smithi179_77138_object46442:head# 2024-02-09T17:27:05.075 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.075 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff78cb9b:::benchmark_data_smithi179_77138_object97004:head# 2024-02-09T17:27:05.075 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.075 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff78e9a5:::benchmark_data_smithi179_77138_object125789:head# 2024-02-09T17:27:05.075 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.075 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff79b1eb:::benchmark_data_smithi179_77138_object85124:head# 2024-02-09T17:27:05.075 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.076 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff7d1840:::benchmark_data_smithi179_77138_object34057:head# 2024-02-09T17:27:05.076 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.076 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff7ddaee:::benchmark_data_smithi179_77138_object66800:head# 2024-02-09T17:27:05.076 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.076 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff7e5225:::benchmark_data_smithi179_77138_object62691:head# 2024-02-09T17:27:05.077 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.077 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff7e9997:::benchmark_data_smithi179_77138_object42369:head# 2024-02-09T17:27:05.077 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.077 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff7ecf36:::benchmark_data_smithi179_77138_object35101:head# 2024-02-09T17:27:05.077 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.077 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff7ef6ca:::benchmark_data_smithi179_77138_object132322:head# 2024-02-09T17:27:05.077 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.079 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff7efb96:::benchmark_data_smithi179_77138_object79824:head# 2024-02-09T17:27:05.079 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.079 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff7f8fb3:::benchmark_data_smithi179_77138_object54591:head# 2024-02-09T17:27:05.079 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.079 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff7f95e0:::benchmark_data_smithi179_77138_object31297:head# 2024-02-09T17:27:05.079 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.079 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff80272e:::benchmark_data_smithi179_77138_object54607:head# 2024-02-09T17:27:05.079 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.079 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff8054da:::benchmark_data_smithi179_77138_object95547:head# 2024-02-09T17:27:05.079 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.079 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff80963d:::benchmark_data_smithi179_77138_object126052:head# 2024-02-09T17:27:05.079 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.080 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff8139f5:::benchmark_data_smithi179_77138_object115672:head# 2024-02-09T17:27:05.080 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.080 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff81de59:::benchmark_data_smithi179_77138_object47153:head# 2024-02-09T17:27:05.080 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.080 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff82456d:::benchmark_data_smithi179_77138_object44797:head# 2024-02-09T17:27:05.080 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.080 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff835dc7:::benchmark_data_smithi179_77138_object72548:head# 2024-02-09T17:27:05.081 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.081 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff83b768:::benchmark_data_smithi179_77138_object111946:head# 2024-02-09T17:27:05.081 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.082 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff83c09e:::benchmark_data_smithi179_77138_object109815:head# 2024-02-09T17:27:05.082 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.082 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff8409ae:::benchmark_data_smithi179_77138_object121344:head# 2024-02-09T17:27:05.082 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.082 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff861113:::benchmark_data_smithi179_77138_object114291:head# 2024-02-09T17:27:05.082 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.083 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff87f3d6:::benchmark_data_smithi179_77138_object133179:head# 2024-02-09T17:27:05.083 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.083 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff880cb5:::benchmark_data_smithi179_77138_object107959:head# 2024-02-09T17:27:05.083 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.084 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:27:05.088 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff8a2b0b:::benchmark_data_smithi179_77138_object62689:head# 2024-02-09T17:27:05.088 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.088 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff8a4768:::benchmark_data_smithi179_77138_object29470:head# 2024-02-09T17:27:05.088 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.088 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff8c3557:::benchmark_data_smithi179_77138_object38964:head# 2024-02-09T17:27:05.088 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.088 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff8c9b47:::benchmark_data_smithi179_77138_object19600:head# 2024-02-09T17:27:05.088 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.088 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff8ccb7e:::benchmark_data_smithi179_77138_object96732:head# 2024-02-09T17:27:05.089 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.089 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff8dbb7c:::benchmark_data_smithi179_77138_object125120:head# 2024-02-09T17:27:05.089 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.089 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff8e058c:::benchmark_data_smithi179_77138_object88271:head# 2024-02-09T17:27:05.089 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.089 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff8ed459:::benchmark_data_smithi179_77138_object33490:head# 2024-02-09T17:27:05.089 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.089 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff8f2403:::benchmark_data_smithi179_77138_object64569:head# 2024-02-09T17:27:05.089 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.089 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff9035ea:::benchmark_data_smithi179_77138_object45901:head# 2024-02-09T17:27:05.089 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.090 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff905f93:::benchmark_data_smithi179_77138_object33062:head# 2024-02-09T17:27:05.090 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.090 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff908961:::benchmark_data_smithi179_77138_object58075:head# 2024-02-09T17:27:05.090 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.090 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff919e53:::benchmark_data_smithi179_77138_object90591:head# 2024-02-09T17:27:05.090 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.090 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff91bb3a:::benchmark_data_smithi179_77138_object126477:head# 2024-02-09T17:27:05.090 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.093 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff91eee5:::benchmark_data_smithi179_77138_object46749:head# 2024-02-09T17:27:05.094 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.094 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff92decd:::benchmark_data_smithi179_77138_object54238:head# 2024-02-09T17:27:05.094 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.094 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff92fa77:::benchmark_data_smithi179_77138_object56593:head# 2024-02-09T17:27:05.094 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.094 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff933dc4:::benchmark_data_smithi179_77138_object70468:head# 2024-02-09T17:27:05.094 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.094 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff93e5c1:::benchmark_data_smithi179_77138_object77103:head# 2024-02-09T17:27:05.094 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.094 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff9451f1:::benchmark_data_smithi179_77138_object92910:head# 2024-02-09T17:27:05.095 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.095 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff94f8ce:::benchmark_data_smithi179_77138_object46679:head# 2024-02-09T17:27:05.095 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.095 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff9518f8:::benchmark_data_smithi179_77138_object82246:head# 2024-02-09T17:27:05.095 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.095 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff9900eb:::benchmark_data_smithi179_77138_object57030:head# 2024-02-09T17:27:05.095 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.095 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff992af2:::benchmark_data_smithi179_77138_object65314:head# 2024-02-09T17:27:05.095 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.095 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff9993f4:::benchmark_data_smithi179_77138_object112875:head# 2024-02-09T17:27:05.095 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.095 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff9997f6:::benchmark_data_smithi179_77138_object71098:head# 2024-02-09T17:27:05.095 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.096 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff9a1ff5:::benchmark_data_smithi179_77138_object119297:head# 2024-02-09T17:27:05.096 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.096 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff9bcc03:::benchmark_data_smithi179_77138_object126845:head# 2024-02-09T17:27:05.096 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.096 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff9c47cd:::benchmark_data_smithi179_77138_object111336:head# 2024-02-09T17:27:05.096 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.096 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff9c645f:::benchmark_data_smithi179_77138_object49794:head# 2024-02-09T17:27:05.096 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.099 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff9d6201:::benchmark_data_smithi179_77138_object117218:head# 2024-02-09T17:27:05.099 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.099 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff9d94b0:::benchmark_data_smithi179_77138_object128019:head# 2024-02-09T17:27:05.099 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.099 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff9e644e:::benchmark_data_smithi179_77138_object133338:head# 2024-02-09T17:27:05.099 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.099 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff9ecb13:::benchmark_data_smithi179_77138_object49197:head# 2024-02-09T17:27:05.099 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.100 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff9f4fb7:::benchmark_data_smithi179_77138_object114285:head# 2024-02-09T17:27:05.100 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.100 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ff9f96b3:::benchmark_data_smithi179_77138_object56408:head# 2024-02-09T17:27:05.100 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.100 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffa14601:::benchmark_data_smithi179_77138_object33140:head# 2024-02-09T17:27:05.100 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.100 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffa1627e:::benchmark_data_smithi179_77138_object37492:head# 2024-02-09T17:27:05.100 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.100 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffa1bef5:::benchmark_data_smithi179_77138_object77721:head# 2024-02-09T17:27:05.100 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.100 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffa37e40:::benchmark_data_smithi179_77138_object68606:head# 2024-02-09T17:27:05.100 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.101 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffa52d61:::benchmark_data_smithi179_77138_object108738:head# 2024-02-09T17:27:05.101 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.101 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffa5484a:::benchmark_data_smithi179_77138_object109081:head# 2024-02-09T17:27:05.101 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.101 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffa5512c:::benchmark_data_smithi179_77138_object131475:head# 2024-02-09T17:27:05.101 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.101 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffa596cb:::benchmark_data_smithi179_77138_object129761:head# 2024-02-09T17:27:05.101 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.101 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffa5eae4:::benchmark_data_smithi179_77138_object97207:head# 2024-02-09T17:27:05.101 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.102 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffa66e4b:::benchmark_data_smithi179_77138_object24629:head# 2024-02-09T17:27:05.102 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.102 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffa731b0:::benchmark_data_smithi179_77138_object59314:head# 2024-02-09T17:27:05.103 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.103 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffa77c9b:::benchmark_data_smithi179_77138_object112498:head# 2024-02-09T17:27:05.103 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.103 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffa791dc:::benchmark_data_smithi179_77138_object106790:head# 2024-02-09T17:27:05.103 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.103 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffa7f15f:::benchmark_data_smithi179_77138_object61745:head# 2024-02-09T17:27:05.103 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.103 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffa8979a:::benchmark_data_smithi179_77138_object26323:head# 2024-02-09T17:27:05.103 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.103 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffa9e8c0:::benchmark_data_smithi179_77138_object83195:head# 2024-02-09T17:27:05.103 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.104 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffaa7898:::benchmark_data_smithi179_77138_object92049:head# 2024-02-09T17:27:05.104 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.104 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffab8b26:::benchmark_data_smithi179_77138_object79217:head# 2024-02-09T17:27:05.104 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.105 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffab8c99:::benchmark_data_smithi179_77138_object81406:head# 2024-02-09T17:27:05.105 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.106 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffad6bfd:::benchmark_data_smithi179_77138_object42620:head# 2024-02-09T17:27:05.106 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.106 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffad8a09:::benchmark_data_smithi179_77138_object67029:head# 2024-02-09T17:27:05.106 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.106 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffadcc70:::benchmark_data_smithi179_77138_object71239:head# 2024-02-09T17:27:05.106 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.107 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffae1c61:::benchmark_data_smithi179_77138_object69168:head# 2024-02-09T17:27:05.107 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.108 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffae75e7:::benchmark_data_smithi179_77138_object44115:head# 2024-02-09T17:27:05.108 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.108 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffaf9eb7:::benchmark_data_smithi179_77138_object86750:head# 2024-02-09T17:27:05.108 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.108 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffb07889:::benchmark_data_smithi179_77138_object98295:head# 2024-02-09T17:27:05.108 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.108 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffb11d6c:::benchmark_data_smithi179_77138_object106666:head# 2024-02-09T17:27:05.108 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.109 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffb1286a:::benchmark_data_smithi179_77138_object49306:head# 2024-02-09T17:27:05.109 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.109 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffb18e12:::benchmark_data_smithi179_77138_object51041:head# 2024-02-09T17:27:05.109 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.109 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffb29ce9:::benchmark_data_smithi179_77138_object85101:head# 2024-02-09T17:27:05.110 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.110 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffb5ef7d:::benchmark_data_smithi179_77138_object60856:head# 2024-02-09T17:27:05.110 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.113 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffb6572c:::benchmark_data_smithi179_77138_object71617:head# 2024-02-09T17:27:05.113 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.115 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffb68df8:::benchmark_data_smithi179_77138_object55849:head# 2024-02-09T17:27:05.115 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.115 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffb7a75e:::benchmark_data_smithi179_77138_object42090:head# 2024-02-09T17:27:05.116 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.116 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffb7f4b1:::benchmark_data_smithi179_77138_object125998:head# 2024-02-09T17:27:05.116 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.116 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffb85a56:::benchmark_data_smithi179_77138_object123252:head# 2024-02-09T17:27:05.116 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.116 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffb9b27d:::benchmark_data_smithi179_77138_object88561:head# 2024-02-09T17:27:05.116 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.116 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffba4dca:::benchmark_data_smithi179_77138_object90960:head# 2024-02-09T17:27:05.116 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.116 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffbaf2d0:::benchmark_data_smithi179_77138_object125306:head# 2024-02-09T17:27:05.116 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.116 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffbb32c4:::benchmark_data_smithi179_77138_object36082:head# 2024-02-09T17:27:05.117 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.117 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffbb4754:::benchmark_data_smithi179_77138_object25729:head# 2024-02-09T17:27:05.117 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.119 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffbd34f5:::benchmark_data_smithi179_77138_object29766:head# 2024-02-09T17:27:05.119 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.119 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffbd482a:::benchmark_data_smithi179_77138_object116110:head# 2024-02-09T17:27:05.119 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.119 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffbd6c47:::benchmark_data_smithi179_77138_object112797:head# 2024-02-09T17:27:05.119 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.119 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffbe5230:::benchmark_data_smithi179_77138_object52279:head# 2024-02-09T17:27:05.119 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.119 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffbfb322:::benchmark_data_smithi179_77138_object47669:head# 2024-02-09T17:27:05.120 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.120 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffc05327:::benchmark_data_smithi179_77138_object132611:head# 2024-02-09T17:27:05.120 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.120 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffc08b7a:::benchmark_data_smithi179_77138_object112406:head# 2024-02-09T17:27:05.120 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.120 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffc2a232:::benchmark_data_smithi179_77138_object96191:head# 2024-02-09T17:27:05.120 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.120 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffc2baca:::benchmark_data_smithi179_77138_object88758:head# 2024-02-09T17:27:05.120 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.120 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffc2c381:::benchmark_data_smithi179_77138_object25097:head# 2024-02-09T17:27:05.120 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.120 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffc35124:::benchmark_data_smithi179_77138_object88099:head# 2024-02-09T17:27:05.120 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.121 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffc4bc7b:::benchmark_data_smithi179_77138_object87571:head# 2024-02-09T17:27:05.121 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.121 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffc507a7:::benchmark_data_smithi179_77138_object57060:head# 2024-02-09T17:27:05.121 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.123 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffc66fc3:::benchmark_data_smithi179_77138_object126858:head# 2024-02-09T17:27:05.123 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.123 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffc6bddf:::benchmark_data_smithi179_77138_object63017:head# 2024-02-09T17:27:05.123 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.124 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffc6f2df:::benchmark_data_smithi179_77138_object123867:head# 2024-02-09T17:27:05.124 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.124 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffc7ad8f:::benchmark_data_smithi179_77138_object104383:head# 2024-02-09T17:27:05.124 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.124 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffc7de18:::benchmark_data_smithi179_77138_object26022:head# 2024-02-09T17:27:05.124 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.124 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffcaba87:::benchmark_data_smithi179_77138_object71147:head# 2024-02-09T17:27:05.124 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.124 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffcabe00:::benchmark_data_smithi179_77138_object52458:head# 2024-02-09T17:27:05.124 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.124 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffcaf3c2:::benchmark_data_smithi179_77138_object103433:head# 2024-02-09T17:27:05.124 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.125 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffcb19dc:::benchmark_data_smithi179_77138_object99811:head# 2024-02-09T17:27:05.125 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.125 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffce53a8:::benchmark_data_smithi179_77138_object110415:head# 2024-02-09T17:27:05.125 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.125 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffce8f1b:::benchmark_data_smithi179_77138_object66231:head# 2024-02-09T17:27:05.125 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.125 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffcffaa6:::benchmark_data_smithi179_77138_object27915:head# 2024-02-09T17:27:05.125 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.125 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffd13d41:::benchmark_data_smithi179_77138_object81578:head# 2024-02-09T17:27:05.125 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.125 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffd1ca07:::benchmark_data_smithi179_77138_object45912:head# 2024-02-09T17:27:05.125 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.126 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffd2ac1a:::benchmark_data_smithi179_77138_object117640:head# 2024-02-09T17:27:05.126 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.126 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffd3563a:::benchmark_data_smithi179_77138_object113040:head# 2024-02-09T17:27:05.126 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.127 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:05.128 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffd376da:::benchmark_data_smithi179_77138_object52018:head# 2024-02-09T17:27:05.129 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.129 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffd396ab:::benchmark_data_smithi179_77138_object120601:head# 2024-02-09T17:27:05.129 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.129 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffd3a552:::benchmark_data_smithi179_77138_object101414:head# 2024-02-09T17:27:05.129 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.129 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffd3ef88:::benchmark_data_smithi179_77138_object106462:head# 2024-02-09T17:27:05.129 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.129 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffd40528:::benchmark_data_smithi179_77138_object129841:head# 2024-02-09T17:27:05.129 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.129 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffd64556:::benchmark_data_smithi179_77138_object26795:head# 2024-02-09T17:27:05.129 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.129 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffd778a7:::benchmark_data_smithi179_77138_object94540:head# 2024-02-09T17:27:05.130 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.130 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffd85df5:::benchmark_data_smithi179_77138_object133106:head# 2024-02-09T17:27:05.130 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.130 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffd90254:::benchmark_data_smithi179_77138_object62203:head# 2024-02-09T17:27:05.130 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.130 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffd956c5:::benchmark_data_smithi179_77138_object121528:head# 2024-02-09T17:27:05.130 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.130 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffda407f:::benchmark_data_smithi179_77138_object81454:head# 2024-02-09T17:27:05.130 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.131 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffdb475d:::benchmark_data_smithi179_77138_object122731:head# 2024-02-09T17:27:05.131 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.132 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffdbcc47:::benchmark_data_smithi179_77138_object35944:head# 2024-02-09T17:27:05.132 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.134 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffdc1169:::benchmark_data_smithi179_77138_object78135:head# 2024-02-09T17:27:05.134 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.135 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffdc3d7c:::benchmark_data_smithi179_77138_object48157:head# 2024-02-09T17:27:05.135 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.135 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffdca279:::benchmark_data_smithi179_77138_object105417:head# 2024-02-09T17:27:05.135 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.135 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffdcb1ad:::benchmark_data_smithi179_77138_object23087:head# 2024-02-09T17:27:05.135 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.135 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffddd849:::benchmark_data_smithi179_77138_object68349:head# 2024-02-09T17:27:05.135 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.135 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffde3d8d:::benchmark_data_smithi179_77138_object40393:head# 2024-02-09T17:27:05.135 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.135 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffde689f:::benchmark_data_smithi179_77138_object128700:head# 2024-02-09T17:27:05.135 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.135 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffdf1fba:::benchmark_data_smithi179_77138_object28313:head# 2024-02-09T17:27:05.136 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.136 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffdfc7fb:::benchmark_data_smithi179_77138_object76771:head# 2024-02-09T17:27:05.136 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.136 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffe0908c:::benchmark_data_smithi179_77138_object125648:head# 2024-02-09T17:27:05.136 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.136 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffe33fb1:::benchmark_data_smithi179_77138_object96769:head# 2024-02-09T17:27:05.137 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.137 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffe3e4bb:::benchmark_data_smithi179_77138_object66091:head# 2024-02-09T17:27:05.137 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.139 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffe40994:::benchmark_data_smithi179_77138_object42928:head# 2024-02-09T17:27:05.139 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.142 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffe43aae:::benchmark_data_smithi179_77138_object22619:head# 2024-02-09T17:27:05.142 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.142 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffe44e65:::benchmark_data_smithi179_77138_object27612:head# 2024-02-09T17:27:05.143 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.143 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffe50551:::benchmark_data_smithi179_77138_object70133:head# 2024-02-09T17:27:05.143 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.143 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffe76b59:::benchmark_data_smithi179_77138_object107634:head# 2024-02-09T17:27:05.143 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.143 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffe7f740:::benchmark_data_smithi179_77138_object41711:head# 2024-02-09T17:27:05.143 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.143 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffe8a376:::benchmark_data_smithi179_77138_object27913:head# 2024-02-09T17:27:05.143 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.143 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffe8b48b:::benchmark_data_smithi179_77138_object41376:head# 2024-02-09T17:27:05.143 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.143 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffe945a7:::benchmark_data_smithi179_77138_object129261:head# 2024-02-09T17:27:05.144 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.144 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffea7e28:::benchmark_data_smithi179_77138_object45447:head# 2024-02-09T17:27:05.144 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.144 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffeb4f01:::benchmark_data_smithi179_77138_object67500:head# 2024-02-09T17:27:05.144 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.144 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffeb8ca4:::benchmark_data_smithi179_77138_object100023:head# 2024-02-09T17:27:05.144 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.144 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffed7260:::benchmark_data_smithi179_77138_object68433:head# 2024-02-09T17:27:05.144 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.144 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffed96ff:::benchmark_data_smithi179_77138_object85392:head# 2024-02-09T17:27:05.144 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.145 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffed9764:::benchmark_data_smithi179_77138_object56590:head# 2024-02-09T17:27:05.145 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.145 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffee906c:::benchmark_data_smithi179_77138_object45574:head# 2024-02-09T17:27:05.145 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.145 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffeed7ba:::benchmark_data_smithi179_77138_object20294:head# 2024-02-09T17:27:05.145 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.150 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffefb832:::benchmark_data_smithi179_77138_object84643:head# 2024-02-09T17:27:05.150 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.150 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffefb8eb:::benchmark_data_smithi179_77138_object29347:head# 2024-02-09T17:27:05.150 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.150 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffefeb9a:::benchmark_data_smithi179_77138_object62514:head# 2024-02-09T17:27:05.151 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.151 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fff05de5:::benchmark_data_smithi179_77138_object102592:head# 2024-02-09T17:27:05.151 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.151 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fff07db2:::benchmark_data_smithi179_77138_object63769:head# 2024-02-09T17:27:05.151 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.151 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fff0e7df:::benchmark_data_smithi179_77138_object57890:head# 2024-02-09T17:27:05.151 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.151 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fff27efd:::benchmark_data_smithi179_77138_object95249:head# 2024-02-09T17:27:05.151 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.151 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fff2a1a5:::benchmark_data_smithi179_77138_object96780:head# 2024-02-09T17:27:05.151 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.151 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fff2fc92:::benchmark_data_smithi179_77138_object64190:head# 2024-02-09T17:27:05.152 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.152 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fff358fb:::benchmark_data_smithi179_77138_object104546:head# 2024-02-09T17:27:05.152 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.152 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fff3d3ea:::benchmark_data_smithi179_77138_object130682:head# 2024-02-09T17:27:05.152 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.152 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fff3d706:::benchmark_data_smithi179_77138_object45467:head# 2024-02-09T17:27:05.152 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.152 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fff5e815:::benchmark_data_smithi179_77138_object47811:head# 2024-02-09T17:27:05.152 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.152 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fff626b4:::benchmark_data_smithi179_77138_object122720:head# 2024-02-09T17:27:05.152 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.152 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fff6d277:::benchmark_data_smithi179_77138_object109155:head# 2024-02-09T17:27:05.153 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.153 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fff74b41:::benchmark_data_smithi179_77138_object118660:head# 2024-02-09T17:27:05.153 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.153 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fff93583:::benchmark_data_smithi179_77138_object23131:head# 2024-02-09T17:27:05.153 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.153 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fff9f60c:::benchmark_data_smithi179_77138_object62999:head# 2024-02-09T17:27:05.153 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.153 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fffa1ff9:::benchmark_data_smithi179_77138_object26494:head# 2024-02-09T17:27:05.153 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.153 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fffa61f6:::benchmark_data_smithi179_77138_object61184:head# 2024-02-09T17:27:05.153 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.154 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fffaf8af:::benchmark_data_smithi179_77138_object105171:head# 2024-02-09T17:27:05.154 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.154 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fffbae02:::benchmark_data_smithi179_77138_object48692:head# 2024-02-09T17:27:05.154 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.154 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fffc8031:::benchmark_data_smithi179_77138_object118162:head# 2024-02-09T17:27:05.154 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.154 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fffcd3d5:::benchmark_data_smithi179_77138_object83845:head# 2024-02-09T17:27:05.154 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.154 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fffd1eb2:::benchmark_data_smithi179_77138_object68998:head# 2024-02-09T17:27:05.154 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.154 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fffd2fa3:::benchmark_data_smithi179_77138_object64490:head# 2024-02-09T17:27:05.154 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.155 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fffd36f6:::benchmark_data_smithi179_77138_object81192:head# 2024-02-09T17:27:05.155 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.155 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fffd5f7c:::benchmark_data_smithi179_77138_object119124:head# 2024-02-09T17:27:05.155 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.155 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fffd6e0d:::benchmark_data_smithi179_77138_object127538:head# 2024-02-09T17:27:05.155 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.155 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fffdff23:::benchmark_data_smithi179_77138_object132726:head# 2024-02-09T17:27:05.155 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.155 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fffe844c:::benchmark_data_smithi179_77138_object111341:head# 2024-02-09T17:27:05.155 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.155 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:fffebbea:::benchmark_data_smithi179_77138_object127823:head# 2024-02-09T17:27:05.155 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.156 INFO:teuthology.orchestra.run.smithi179.stdout:Write #2:ffff8a59:::benchmark_data_smithi179_77138_object128340:head# 2024-02-09T17:27:05.156 INFO:teuthology.orchestra.run.smithi179.stdout:snapset 0=[]:{} 2024-02-09T17:27:05.160 INFO:teuthology.orchestra.run.smithi179.stdout:write_pg epoch 145 info 2.f( v 145'9589 (0'0,145'9589] local-lis/les=111/112 n=7225 ec=111/111 lis/c=111/111 les/c/f=112/112/0 sis=111) 2024-02-09T17:27:05.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:04 smithi179 ceph-mon[43975]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:27:05.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:04 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:27:05.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:04 smithi179 ceph-mon[43975]: osdmap e151: 12 total, 11 up, 12 in 2024-02-09T17:27:05.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:04 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2831009301' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:27:05.223 INFO:teuthology.orchestra.run.smithi179.stdout:Import successful 2024-02-09T17:27:05.257 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:27:05.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:05.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:27:05.366 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:27:05.405 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:05.557 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:27:05.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:05.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:27:05.608 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-09T17:27:05.609 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-09T17:27:05.664 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:27:05.694 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:05.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:05 smithi162 ceph-mon[44304]: pgmap v461: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 5.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 836 op/s; 23812/285939 objects degraded (8.328%) 2024-02-09T17:27:05.998 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:05 smithi099 ceph-mon[39420]: pgmap v461: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 5.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 836 op/s; 23812/285939 objects degraded (8.328%) 2024-02-09T17:27:06.053 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:27:06.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:06.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:27:06.081 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:05 smithi179 ceph-mon[43975]: pgmap v461: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 5.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 836 op/s; 23812/285939 objects degraded (8.328%) 2024-02-09T17:27:06.160 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:27:06.195 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:06.628 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:27:06.628 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:27:06.820 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set noscrub 2024-02-09T17:27:06.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:06 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:27:06.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:06 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:27:06.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:06 smithi162 ceph-mon[44304]: osdmap e152: 12 total, 11 up, 12 in 2024-02-09T17:27:06.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:06 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 23812/285939 objects degraded (8.328%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:27:06.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:06 smithi162 ceph-mon[44304]: pgmap v463: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 5.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 23216/279330 objects degraded (8.311%) 2024-02-09T17:27:07.029 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:06 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:27:07.029 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:06 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:27:07.029 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:06 smithi179 ceph-mon[43975]: osdmap e152: 12 total, 11 up, 12 in 2024-02-09T17:27:07.029 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:06 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 23812/285939 objects degraded (8.328%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:27:07.029 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:06 smithi179 ceph-mon[43975]: pgmap v463: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 5.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 23216/279330 objects degraded (8.311%) 2024-02-09T17:27:07.046 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:27:07.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:07.057 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:27:07.065 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -f /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/exp.2.f.8 2024-02-09T17:27:07.159 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:27:07.193 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:07.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:06 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:27:07.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:06 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:27:07.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:06 smithi099 ceph-mon[39420]: osdmap e152: 12 total, 11 up, 12 in 2024-02-09T17:27:07.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:06 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 23812/285939 objects degraded (8.328%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:27:07.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:06 smithi099 ceph-mon[39420]: pgmap v463: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 5.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 23216/279330 objects degraded (8.311%) 2024-02-09T17:27:07.256 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:27:07.349 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:27:07.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:07.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:27:07.377 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-09T17:27:07.455 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:27:07.529 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:07.584 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:27:07.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:07.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:27:07.610 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:27:07.688 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:27:07.760 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:27:07.762 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:08.169 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:27:08.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:08.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:27:08.281 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:27:08.352 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:08.636 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:27:08.649 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:27:08.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:08.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:27:08.754 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:27:08.815 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:08.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:08 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:27:08.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:08 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:27:08.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:08 smithi099 ceph-mon[39420]: osdmap e153: 12 total, 11 up, 12 in 2024-02-09T17:27:08.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:08 smithi099 ceph-mon[39420]: pgmap v465: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 5.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 746 op/s; 23216/277545 objects degraded (8.365%) 2024-02-09T17:27:09.016 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:27:09.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:09.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:27:09.128 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:27:09.159 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:08 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:27:09.159 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:08 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:27:09.159 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:08 smithi162 ceph-mon[44304]: osdmap e153: 12 total, 11 up, 12 in 2024-02-09T17:27:09.159 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:08 smithi162 ceph-mon[44304]: pgmap v465: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 5.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 746 op/s; 23216/277545 objects degraded (8.365%) 2024-02-09T17:27:09.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:08 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:27:09.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:08 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:27:09.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:08 smithi179 ceph-mon[43975]: osdmap e153: 12 total, 11 up, 12 in 2024-02-09T17:27:09.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:08 smithi179 ceph-mon[43975]: pgmap v465: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 5.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 746 op/s; 23216/277545 objects degraded (8.365%) 2024-02-09T17:27:09.174 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:09.537 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:27:09.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:09.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:27:09.650 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:27:09.679 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:09.704 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:27:09.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:09.710 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:27:09.811 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:27:09.814 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:27:09.815 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:27:09.840 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:09.993 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:27:09.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:09.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:27:10.103 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:27:10.141 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:10.200 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is set 2024-02-09T17:27:10.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:10 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/539015874' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:27:10.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:10 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/539015874' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:27:10.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:10 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/539015874' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:27:11.194 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:11 smithi099 ceph-mon[39420]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:27:11.194 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:11 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/539015874' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:27:11.194 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:11 smithi099 ceph-mon[39420]: osdmap e154: 12 total, 11 up, 12 in 2024-02-09T17:27:11.194 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:11 smithi099 ceph-mon[39420]: pgmap v467: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 5.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 978 op/s; 22385/268326 objects degraded (8.342%) 2024-02-09T17:27:11.194 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:11 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 23216/277545 objects degraded (8.365%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:27:11.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:11 smithi179 ceph-mon[43975]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:27:11.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:11 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/539015874' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:27:11.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:11 smithi179 ceph-mon[43975]: osdmap e154: 12 total, 11 up, 12 in 2024-02-09T17:27:11.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:11 smithi179 ceph-mon[43975]: pgmap v467: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 5.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 978 op/s; 22385/268326 objects degraded (8.342%) 2024-02-09T17:27:11.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:11 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 23216/277545 objects degraded (8.365%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:27:11.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:11 smithi162 ceph-mon[44304]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:27:11.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:11 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/539015874' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:27:11.681 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:11 smithi162 ceph-mon[44304]: osdmap e154: 12 total, 11 up, 12 in 2024-02-09T17:27:11.681 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:11 smithi162 ceph-mon[44304]: pgmap v467: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 5.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 978 op/s; 22385/268326 objects degraded (8.342%) 2024-02-09T17:27:11.681 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:11 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 23216/277545 objects degraded (8.365%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:27:12.213 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [8] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11] 2024-02-09T17:27:12.213 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:27:12.213 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-02-09T17:27:12.213 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 0.360185 2024-02-09T17:27:12.213 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd primary-affinity 2 0.3601854260758467 2024-02-09T17:27:12.253 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:27:12.302 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:27:12.366 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:27:12.423 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:27:12.454 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:27:12.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:12 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:27:12.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:12 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:27:12.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:12 smithi099 ceph-mon[39420]: osdmap e155: 12 total, 11 up, 12 in 2024-02-09T17:27:12.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:12 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3972741463' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:27:12.520 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:27:12.560 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:27:12.593 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:27:12.642 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:27:12.668 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:27:12.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:12 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:27:12.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:12 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:27:12.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:12 smithi179 ceph-mon[43975]: osdmap e155: 12 total, 11 up, 12 in 2024-02-09T17:27:12.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:12 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3972741463' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:27:12.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:12 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:27:12.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:12 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:27:12.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:12 smithi162 ceph-mon[44304]: osdmap e155: 12 total, 11 up, 12 in 2024-02-09T17:27:12.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:12 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3972741463' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:27:12.696 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:27:12.733 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:27:12.734 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:27:12.734 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:27:12.734 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:27:12.734 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:27:12.734 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:27:12.734 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:27:12.734 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:27:12.734 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:27:12.734 INFO:teuthology.orchestra.run.smithi099.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-02-09T17:27:12.734 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:27:12.734 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:27:12.734 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:27:13.332 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:13 smithi099 ceph-mon[39420]: pgmap v469: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 5.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 850 op/s; 22100/264024 objects degraded (8.370%) 2024-02-09T17:27:13.369 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-09T17:27:13.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:13 smithi162 ceph-mon[44304]: pgmap v469: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 5.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 850 op/s; 22100/264024 objects degraded (8.370%) 2024-02-09T17:27:13.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:13 smithi179 ceph-mon[43975]: pgmap v469: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 5.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 850 op/s; 22100/264024 objects degraded (8.370%) 2024-02-09T17:27:13.776 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.9 dump_ops_in_flight 2024-02-09T17:27:13.799 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:27:13.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:13.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:27:13.902 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:27:13.950 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:13.986 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:27:13.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:13.989 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:27:14.090 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:27:14.121 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:14.125 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:27:14.143 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:27:14.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:14.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:27:14.152 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-09T17:27:14.153 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-09T17:27:14.247 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:27:14.311 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:14.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:14 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:27:14.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:14 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:27:14.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:14 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:27:14.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:14 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:27:14.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:14 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:27:14.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:14 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:27:14.701 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:27:14.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:14.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:27:14.721 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:27:14.823 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:27:14.853 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:14.876 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:27:14.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:14.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:27:14.981 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:27:15.011 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:15.035 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:27:15.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:15.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:27:15.141 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:27:15.176 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:15.276 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:27:15.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:15.278 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:27:15.290 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:15 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:27:15.291 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:15 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:27:15.291 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:15 smithi099 ceph-mon[39420]: osdmap e156: 12 total, 11 up, 12 in 2024-02-09T17:27:15.291 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:15 smithi099 ceph-mon[39420]: pgmap v471: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 5.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 21511/259080 objects degraded (8.303%) 2024-02-09T17:27:15.379 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:27:15.409 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:15.500 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:27:15.502 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:15 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:27:15.502 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:15 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:27:15.502 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:15 smithi179 ceph-mon[43975]: osdmap e156: 12 total, 11 up, 12 in 2024-02-09T17:27:15.503 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:15 smithi179 ceph-mon[43975]: pgmap v471: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 5.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 21511/259080 objects degraded (8.303%) 2024-02-09T17:27:15.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:15.504 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:27:15.605 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:27:15.638 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:15 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:27:15.638 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:15 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:27:15.638 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:15 smithi162 ceph-mon[44304]: osdmap e156: 12 total, 11 up, 12 in 2024-02-09T17:27:15.638 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:15 smithi162 ceph-mon[44304]: pgmap v471: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 5.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 21511/259080 objects degraded (8.303%) 2024-02-09T17:27:15.640 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:15.661 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:27:15.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:15.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:27:15.768 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:27:15.798 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:15.822 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:27:15.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:15.826 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:27:15.927 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:27:15.958 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:16.153 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:27:16.415 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:27:16.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:16.420 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:27:16.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:16 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 21511/259080 objects degraded (8.303%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:27:16.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:16 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3312021473' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.3601854260758467}]: dispatch 2024-02-09T17:27:16.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:16 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.3601854260758467}]: dispatch 2024-02-09T17:27:16.521 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:27:16.547 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:16 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 21511/259080 objects degraded (8.303%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:27:16.547 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:16 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3312021473' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.3601854260758467}]: dispatch 2024-02-09T17:27:16.547 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:16 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.3601854260758467}]: dispatch 2024-02-09T17:27:16.550 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:16.570 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:27:16.576 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:16.576 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:27:16.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:16 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 21511/259080 objects degraded (8.303%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:27:16.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:16 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3312021473' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.3601854260758467}]: dispatch 2024-02-09T17:27:16.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:16 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.3601854260758467}]: dispatch 2024-02-09T17:27:16.677 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:27:16.743 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:16.781 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:27:16.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:16.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:27:16.889 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:27:16.927 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:16.958 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:27:16.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:16.963 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:27:16.991 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:27:16.991 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-02-09T17:27:16.991 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:16.991 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:184867 2.d 2:b3082818:::benchmark_data_smithi179_77138_object51459:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-02-09T17:27:16.991 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:16.986931+0000", 2024-02-09T17:27:16.992 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.0033669490000000002, 2024-02-09T17:27:16.992 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.003397995, 2024-02-09T17:27:16.992 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:27:16.992 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "waiting for sub ops", 2024-02-09T17:27:16.992 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:27:16.992 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:27:16.992 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:27:16.992 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 184867 2024-02-09T17:27:16.992 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:16.992 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:27:16.992 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:16.992 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:27:16.992 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:16.986931+0000", 2024-02-09T17:27:16.993 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:16.993 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:16.993 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:16.993 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:27:16.993 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:16.986931+0000", 2024-02-09T17:27:16.993 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:16.993 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:16.993 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:16.993 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:27:16.993 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:16.986932+0000", 2024-02-09T17:27:16.993 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.47e-07 2024-02-09T17:27:16.993 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:16.993 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:16.994 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:27:16.994 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:16.986936+0000", 2024-02-09T17:27:16.994 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.985e-06 2024-02-09T17:27:16.994 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:16.994 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:16.994 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:27:16.994 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:16.986946+0000", 2024-02-09T17:27:16.994 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5470000000000002e-06 2024-02-09T17:27:16.994 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:16.994 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:16.994 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:27:16.994 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:16.986951+0000", 2024-02-09T17:27:16.994 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6080000000000003e-06 2024-02-09T17:27:16.994 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:16.995 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:16.995 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:27:16.995 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:16.986993+0000", 2024-02-09T17:27:16.995 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1718999999999998e-05 2024-02-09T17:27:16.995 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:16.995 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:16.995 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:27:16.995 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:16.987085+0000", 2024-02-09T17:27:16.995 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.1513000000000003e-05 2024-02-09T17:27:16.995 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:16.995 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:16.995 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,4", 2024-02-09T17:27:16.995 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:16.987177+0000", 2024-02-09T17:27:16.995 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2825000000000007e-05 2024-02-09T17:27:16.996 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:16.996 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:27:16.996 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:16.996 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:16.996 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:16.996 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.14790.0:184861 2.e e156/111 2:73e5eefa:::benchmark_data_smithi179_77138_object51453:head v 156'11584, mlcod=156'11582)", 2024-02-09T17:27:16.996 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:16.989807+0000", 2024-02-09T17:27:16.996 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.00049092299999999999, 2024-02-09T17:27:16.996 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00060417800000000003, 2024-02-09T17:27:16.996 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:27:16.996 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-02-09T17:27:16.996 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:27:16.996 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:16.996 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:27:16.997 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:16.989807+0000", 2024-02-09T17:27:16.997 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:16.997 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:16.997 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:16.997 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:27:16.997 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:16.989807+0000", 2024-02-09T17:27:16.997 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:16.997 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:16.997 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:16.997 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:27:16.997 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:16.989808+0000", 2024-02-09T17:27:16.997 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6400000000000002e-07 2024-02-09T17:27:16.997 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:16.997 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:16.997 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:27:16.998 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:16.989831+0000", 2024-02-09T17:27:16.998 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3476e-05 2024-02-09T17:27:16.998 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:16.998 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:16.998 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:27:16.998 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:16.989843+0000", 2024-02-09T17:27:16.998 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1211e-05 2024-02-09T17:27:16.998 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:16.998 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:16.998 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:27:16.998 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:16.989849+0000", 2024-02-09T17:27:16.998 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8279999999999997e-06 2024-02-09T17:27:16.998 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:16.999 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:16.999 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:27:16.999 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:16.989908+0000", 2024-02-09T17:27:16.999 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8479e-05 2024-02-09T17:27:16.999 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:16.999 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:16.999 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:27:16.999 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:16.989932+0000", 2024-02-09T17:27:16.999 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4221999999999999e-05 2024-02-09T17:27:16.999 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:16.999 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:27:16.999 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:16.999 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:16.999 INFO:teuthology.orchestra.run.smithi179.stdout: ], 2024-02-09T17:27:17.000 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 2 2024-02-09T17:27:17.000 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:27:17.064 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:27:17.115 INFO:teuthology.orchestra.run.smithi099.stderr:set osd.2 primary-affinity to 0.360185 (8236052) 2024-02-09T17:27:17.124 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:17.483 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:27:17.484 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:27:17.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:17.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:27:17.587 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:27:17.613 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:17 smithi179 ceph-mon[43975]: pgmap v472: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 5.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 842 op/s; 20993/252546 objects degraded (8.313%) 2024-02-09T17:27:17.613 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:17 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.3601854260758467}]': finished 2024-02-09T17:27:17.613 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:17 smithi179 ceph-mon[43975]: osdmap e157: 12 total, 11 up, 12 in 2024-02-09T17:27:17.615 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:17.668 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:27:17.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:17.672 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:27:17.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:17 smithi162 ceph-mon[44304]: pgmap v472: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 5.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 842 op/s; 20993/252546 objects degraded (8.313%) 2024-02-09T17:27:17.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:17 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.3601854260758467}]': finished 2024-02-09T17:27:17.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:17 smithi162 ceph-mon[44304]: osdmap e157: 12 total, 11 up, 12 in 2024-02-09T17:27:17.686 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.9 dump_blocked_ops 2024-02-09T17:27:17.740 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:17 smithi099 ceph-mon[39420]: pgmap v472: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 5.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 842 op/s; 20993/252546 objects degraded (8.313%) 2024-02-09T17:27:17.740 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:17 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.3601854260758467}]': finished 2024-02-09T17:27:17.741 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:17 smithi099 ceph-mon[39420]: osdmap e157: 12 total, 11 up, 12 in 2024-02-09T17:27:17.773 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:27:17.800 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:17.820 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:27:17.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:17.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:27:17.924 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:27:17.951 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:17.971 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:27:17.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:17.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:27:17.981 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:27:18.075 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:27:18.093 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is set 2024-02-09T17:27:18.103 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:18.125 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:27:18.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:18.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:27:18.229 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:27:18.265 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:18.291 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:27:18.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:18.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:27:18.398 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:27:18.434 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:18.474 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:27:18.480 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:18 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4030693867' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:27:18.480 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:18 smithi179 ceph-mon[43975]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:27:18.481 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:18 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4030693867' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:27:18.481 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:18 smithi179 ceph-mon[43975]: osdmap e158: 12 total, 11 up, 12 in 2024-02-09T17:27:18.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:18.482 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:27:18.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:18 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4030693867' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:27:18.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:18 smithi099 ceph-mon[39420]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:27:18.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:18 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4030693867' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:27:18.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:18 smithi099 ceph-mon[39420]: osdmap e158: 12 total, 11 up, 12 in 2024-02-09T17:27:18.582 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:27:18.615 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:18 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4030693867' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:27:18.615 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:18 smithi162 ceph-mon[44304]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:27:18.615 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:18 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4030693867' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:27:18.615 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:18 smithi162 ceph-mon[44304]: osdmap e158: 12 total, 11 up, 12 in 2024-02-09T17:27:18.615 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:18.641 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:27:18.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:18.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:27:18.745 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:27:18.812 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:19.029 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:27:19.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:19.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:27:19.143 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:27:19.224 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:19.648 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:19 smithi099 ceph-mon[39420]: pgmap v475: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 5.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 726 op/s; 20993/250947 objects degraded (8.366%) 2024-02-09T17:27:19.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:19 smithi179 ceph-mon[43975]: pgmap v475: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 5.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 726 op/s; 20993/250947 objects degraded (8.366%) 2024-02-09T17:27:19.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:19 smithi162 ceph-mon[44304]: pgmap v475: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 5.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 726 op/s; 20993/250947 objects degraded (8.366%) 2024-02-09T17:27:20.012 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:27:20.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:20.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:27:20.071 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:27:20.071 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-02-09T17:27:20.071 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-02-09T17:27:20.071 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-02-09T17:27:20.071 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:27:20.116 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:27:20.169 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:20.229 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:27:20.234 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:20.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:27:20.335 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:27:20.375 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:20.401 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:27:20.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:20.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:27:20.505 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:27:20.539 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:20.563 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:27:20.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:20.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:27:20.671 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:27:20.733 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:20.777 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:27:20.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:20.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:27:20.797 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:27:20.825 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:27:20.847 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:27:20.869 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:27:20.875 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.9 dump_historic_ops 2024-02-09T17:27:20.884 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:27:20.889 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:27:20.913 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:20.929 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:27:20.949 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:27:20.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:20.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:27:20.993 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:27:21.027 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:27:21.055 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:27:21.072 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:27:21.117 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:27:21.122 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:21.166 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:27:21.167 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:27:21.167 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:27:21.167 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:27:21.167 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:27:21.167 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:27:21.167 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:27:21.167 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:27:21.167 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:27:21.167 INFO:teuthology.orchestra.run.smithi099.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-02-09T17:27:21.167 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:27:21.167 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:27:21.167 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:27:21.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:21.175 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:27:21.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:27:21.229 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:27:21.276 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:27:21.313 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:21.353 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:27:21.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:21.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:27:21.461 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:27:21.497 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:21.519 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:27:21.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:21.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:27:21.623 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:27:21.652 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:21.681 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset noscrub 2024-02-09T17:27:21.687 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:27:21.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:21.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:27:21.798 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:27:21.802 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:21 smithi179 ceph-mon[43975]: pgmap v476: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 891 op/s; 20338/245130 objects degraded (8.297%) 2024-02-09T17:27:21.802 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:21 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4119971232' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:27:21.802 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:21 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 20338/245130 objects degraded (8.297%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:27:21.825 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:21 smithi162 ceph-mon[44304]: pgmap v476: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 891 op/s; 20338/245130 objects degraded (8.297%) 2024-02-09T17:27:21.825 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:21 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4119971232' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:27:21.825 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:21 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 20338/245130 objects degraded (8.297%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:27:21.826 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:21.849 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:27:21.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:21.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:27:21.954 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:27:21.985 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:22.000 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:21 smithi099 ceph-mon[39420]: pgmap v476: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 891 op/s; 20338/245130 objects degraded (8.297%) 2024-02-09T17:27:22.000 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:21 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4119971232' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:27:22.001 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:21 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 20338/245130 objects degraded (8.297%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:27:22.139 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:27:22.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:22.144 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:27:22.245 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:27:22.278 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:22.322 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:27:22.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:22.331 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:27:22.432 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:27:22.463 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:22.785 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:22 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:27:22.868 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:22 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:27:22.869 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:22 smithi179 ceph-mon[43975]: osdmap e159: 12 total, 11 up, 12 in 2024-02-09T17:27:22.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:22.870 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:27:22.870 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:27:22.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:22 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:27:22.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:22 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:27:22.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:22 smithi162 ceph-mon[44304]: osdmap e159: 12 total, 11 up, 12 in 2024-02-09T17:27:22.971 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:27:22.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:22 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:27:22.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:22 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:27:22.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:22 smithi099 ceph-mon[39420]: osdmap e159: 12 total, 11 up, 12 in 2024-02-09T17:27:23.034 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:23.321 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:27:23.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:23.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:27:23.429 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:27:23.466 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:23.488 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:27:23.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:23.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:27:23.510 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T17:27:23.510 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:27:23.592 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:27:23.648 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:23 smithi099 ceph-mon[39420]: pgmap v478: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 4.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 688 op/s; 20108/240159 objects degraded (8.373%); 0 B/s, 0 objects/s recovering 2024-02-09T17:27:23.648 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:23 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:27:23.648 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:23 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:27:23.654 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:23.807 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [8] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11] 2024-02-09T17:27:23.807 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:27:23.808 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-02-09T17:27:23.808 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight-by-utilization 1000 0.05 2024-02-09T17:27:23.843 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:27:23.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:23.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:27:23.868 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-09T17:27:23.868 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-09T17:27:23.912 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:23 smithi179 ceph-mon[43975]: pgmap v478: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 4.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 688 op/s; 20108/240159 objects degraded (8.373%); 0 B/s, 0 objects/s recovering 2024-02-09T17:27:23.912 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:23 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:27:23.913 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:23 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:27:23.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:23 smithi162 ceph-mon[44304]: pgmap v478: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 4.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 688 op/s; 20108/240159 objects degraded (8.373%); 0 B/s, 0 objects/s recovering 2024-02-09T17:27:23.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:23 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:27:23.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:23 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:27:23.942 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:27:23.949 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:27:23.980 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:24.000 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:27:24.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:24.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:27:24.104 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:27:24.186 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:24.235 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:27:24.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:24.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:27:24.344 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:27:24.363 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.10 dump_ops_in_flight 2024-02-09T17:27:24.395 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:24.419 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:27:24.421 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:24.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:27:24.523 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:27:24.555 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:24.583 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:27:24.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:24.588 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:27:24.588 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:24 smithi179 ceph-mon[43975]: osdmap e160: 12 total, 11 up, 12 in 2024-02-09T17:27:24.588 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:24 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:27:24.588 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:24 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:27:24.588 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:24 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:27:24.589 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:24 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:27:24.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:24 smithi162 ceph-mon[44304]: osdmap e160: 12 total, 11 up, 12 in 2024-02-09T17:27:24.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:24 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:27:24.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:24 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:27:24.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:24 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:27:24.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:24 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:27:24.689 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:27:24.723 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:24.735 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:27:24.745 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:27:24.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:24.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:27:24.852 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:27:24.909 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:24 smithi099 ceph-mon[39420]: osdmap e160: 12 total, 11 up, 12 in 2024-02-09T17:27:24.909 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:24 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:27:24.909 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:24 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:27:24.909 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:24 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:27:24.909 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:24 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:27:24.911 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:24.955 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:27:24.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:24.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:27:25.063 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:27:25.105 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:25.129 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:27:25.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:25.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:27:25.233 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:27:25.261 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:25.283 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:27:25.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:25.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:27:25.390 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:27:25.418 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:25.442 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:27:25.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:25.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:27:25.451 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:27:25.546 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:27:25.583 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:25.654 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:27:25.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:25.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:27:25.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:25 smithi162 ceph-mon[44304]: pgmap v480: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 4.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 829 op/s; 19566/235197 objects degraded (8.319%); 0 B/s, 0 objects/s recovering 2024-02-09T17:27:25.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:25 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:27:25.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:25 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:27:25.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:25 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:27:25.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:25 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:27:25.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:25 smithi099 ceph-mon[39420]: pgmap v480: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 4.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 829 op/s; 19566/235197 objects degraded (8.319%); 0 B/s, 0 objects/s recovering 2024-02-09T17:27:25.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:25 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:27:25.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:25 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:27:25.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:25 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:27:25.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:25 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:27:25.767 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:27:25.794 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:25.815 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:27:25.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:25.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:27:25.869 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:27:25.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:25 smithi179 ceph-mon[43975]: pgmap v480: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 4.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 829 op/s; 19566/235197 objects degraded (8.319%); 0 B/s, 0 objects/s recovering 2024-02-09T17:27:25.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:25 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:27:25.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:25 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:27:25.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:25 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:27:25.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:25 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:27:25.922 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:27:25.954 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:25.980 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:27:25.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:25.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:27:26.085 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:27:26.113 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:26.133 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:27:26.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:26.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:27:26.239 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:27:26.292 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:26.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:26 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:27:26.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:26 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:27:26.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:26 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 19566/235197 objects degraded (8.319%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:27:26.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:26 smithi179 ceph-mon[43975]: osdmap e161: 12 total, 11 up, 12 in 2024-02-09T17:27:26.762 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:27:26.763 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T17:27:26.763 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:27:26.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:26.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:27:26.867 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:26 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:27:26.867 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:26 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:27:26.867 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:26 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 19566/235197 objects degraded (8.319%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:27:26.867 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:26 smithi099 ceph-mon[39420]: osdmap e161: 12 total, 11 up, 12 in 2024-02-09T17:27:26.869 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:27:26.898 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:26 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:27:26.899 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:26 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:27:26.899 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:26 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 19566/235197 objects degraded (8.319%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:27:26.899 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:26 smithi162 ceph-mon[44304]: osdmap e161: 12 total, 11 up, 12 in 2024-02-09T17:27:26.901 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:26.924 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:27:26.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:26.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:27:27.028 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:27:27.075 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:27.102 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:27:27.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:27.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:27:27.206 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:27:27.243 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:27.307 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:27:27.495 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:27:27.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:27.499 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:27:27.530 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.10 dump_blocked_ops 2024-02-09T17:27:27.594 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is unset 2024-02-09T17:27:27.600 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:27:27.630 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:27.654 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:27:27.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:27.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:27:27.758 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:27:27.791 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:27.818 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:27:27.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:27.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:27:27.867 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:27 smithi099 ceph-mon[39420]: pgmap v482: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 4.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 911 op/s; 19024/228711 objects degraded (8.318%); 0 B/s, 0 objects/s recovering 2024-02-09T17:27:27.867 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:27 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2606153801' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:27:27.867 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:27 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:27:27.919 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:27:27.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:27 smithi179 ceph-mon[43975]: pgmap v482: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 4.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 911 op/s; 19024/228711 objects degraded (8.318%); 0 B/s, 0 objects/s recovering 2024-02-09T17:27:27.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:27 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2606153801' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:27:27.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:27 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:27:27.926 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:27:27.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:27 smithi162 ceph-mon[44304]: pgmap v482: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 4.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 911 op/s; 19024/228711 objects degraded (8.318%); 0 B/s, 0 objects/s recovering 2024-02-09T17:27:27.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:27 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2606153801' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:27:27.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:27 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:27:27.975 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:28.076 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:27:28.079 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:28.079 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:27:28.180 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:27:28.212 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:28.236 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:27:28.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:28.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:27:28.340 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:27:28.398 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:28.463 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:27:28.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:28.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:27:28.502 INFO:teuthology.orchestra.run.smithi099.stdout:moved 0 / 51 (0%) 2024-02-09T17:27:28.502 INFO:teuthology.orchestra.run.smithi099.stdout:avg 4.63636 2024-02-09T17:27:28.502 INFO:teuthology.orchestra.run.smithi099.stdout:stddev 1.40248 -> 1.40248 (expected baseline 2.05302) 2024-02-09T17:27:28.502 INFO:teuthology.orchestra.run.smithi099.stdout:min osd.5 with 2 -> 2 pgs (0.431373 -> 0.431373 * mean) 2024-02-09T17:27:28.502 INFO:teuthology.orchestra.run.smithi099.stdout:max osd.3 with 6 -> 6 pgs (1.29412 -> 1.29412 * mean) 2024-02-09T17:27:28.502 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:27:28.502 INFO:teuthology.orchestra.run.smithi099.stdout:oload 1000 2024-02-09T17:27:28.502 INFO:teuthology.orchestra.run.smithi099.stdout:max_change 0.05 2024-02-09T17:27:28.502 INFO:teuthology.orchestra.run.smithi099.stdout:max_change_osds 4 2024-02-09T17:27:28.502 INFO:teuthology.orchestra.run.smithi099.stdout:average_utilization 0.0144 2024-02-09T17:27:28.503 INFO:teuthology.orchestra.run.smithi099.stdout:overload_utilization 0.1442 2024-02-09T17:27:28.532 INFO:teuthology.orchestra.run.smithi099.stderr:no change 2024-02-09T17:27:28.574 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:27:28.615 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:28 smithi162 ceph-mon[44304]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:27:28.615 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:28 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:27:28.615 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:28 smithi162 ceph-mon[44304]: osdmap e162: 12 total, 11 up, 12 in 2024-02-09T17:27:28.615 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=53,logm=7} crc {auth=3329010490,config=2679774077,health=3729121333,kv=3382112463,logm=745300946}) 2024-02-09T17:27:28.615 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2816134768}) 2024-02-09T17:27:28.615 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1300300781}) 2024-02-09T17:27:28.615 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3978572232}) 2024-02-09T17:27:28.615 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=357694623}) 2024-02-09T17:27:28.615 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2058891222}) 2024-02-09T17:27:28.615 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3871411908}) 2024-02-09T17:27:28.616 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2488200294}) 2024-02-09T17:27:28.616 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1791122756}) 2024-02-09T17:27:28.616 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2037867039}) 2024-02-09T17:27:28.616 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1720194272}) 2024-02-09T17:27:28.616 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1249130144}) 2024-02-09T17:27:28.616 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2795776881}) 2024-02-09T17:27:28.616 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1998478726}) 2024-02-09T17:27:28.616 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2589008878}) 2024-02-09T17:27:28.616 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2089805986}) 2024-02-09T17:27:28.616 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2525942273}) 2024-02-09T17:27:28.616 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619739444}) 2024-02-09T17:27:28.616 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2950400276}) 2024-02-09T17:27:28.616 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2155431284}) 2024-02-09T17:27:28.617 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1354958885}) 2024-02-09T17:27:28.617 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3691148013}) 2024-02-09T17:27:28.617 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=830577004}) 2024-02-09T17:27:28.617 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=591307711}) 2024-02-09T17:27:28.617 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=18,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=44} crc {logm=1434244818,mdsmap=3984425268,mgr=1708625883,mgr_command_descs=708342470,mgr_metadata=670763949,mgrstat=568593314,mon_config_key=4183486463}) 2024-02-09T17:27:28.617 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1317726899}) 2024-02-09T17:27:28.617 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1367840681}) 2024-02-09T17:27:28.617 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=37,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=45} crc {mon_config_key=2165145333,monmap=423865542,osd_metadata=2787106199,osd_pg_creating=1417969174,osdmap=596661807}) 2024-02-09T17:27:28.617 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=64} crc {osdmap=3608665156}) 2024-02-09T17:27:28.646 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:28 smithi099 ceph-mon[39420]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:27:28.646 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:28 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:27:28.646 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:28 smithi099 ceph-mon[39420]: osdmap e162: 12 total, 11 up, 12 in 2024-02-09T17:27:28.646 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=53,logm=7} crc {auth=3329010490,config=2679774077,health=3729121333,kv=3382112463,logm=745300946}) 2024-02-09T17:27:28.646 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2816134768}) 2024-02-09T17:27:28.646 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1300300781}) 2024-02-09T17:27:28.646 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3978572232}) 2024-02-09T17:27:28.646 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=357694623}) 2024-02-09T17:27:28.646 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2058891222}) 2024-02-09T17:27:28.647 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3871411908}) 2024-02-09T17:27:28.647 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2488200294}) 2024-02-09T17:27:28.647 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1791122756}) 2024-02-09T17:27:28.647 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2037867039}) 2024-02-09T17:27:28.647 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1720194272}) 2024-02-09T17:27:28.647 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1249130144}) 2024-02-09T17:27:28.647 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2795776881}) 2024-02-09T17:27:28.647 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1998478726}) 2024-02-09T17:27:28.647 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2589008878}) 2024-02-09T17:27:28.647 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2089805986}) 2024-02-09T17:27:28.647 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2525942273}) 2024-02-09T17:27:28.647 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619739444}) 2024-02-09T17:27:28.647 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2950400276}) 2024-02-09T17:27:28.647 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2155431284}) 2024-02-09T17:27:28.648 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1354958885}) 2024-02-09T17:27:28.648 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3691148013}) 2024-02-09T17:27:28.648 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=830577004}) 2024-02-09T17:27:28.648 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=591307711}) 2024-02-09T17:27:28.648 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=18,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=44} crc {logm=1434244818,mdsmap=3984425268,mgr=1708625883,mgr_command_descs=708342470,mgr_metadata=670763949,mgrstat=568593314,mon_config_key=4183486463}) 2024-02-09T17:27:28.648 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1317726899}) 2024-02-09T17:27:28.648 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1367840681}) 2024-02-09T17:27:28.648 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=37,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=45} crc {mon_config_key=2165145333,monmap=423865542,osd_metadata=2787106199,osd_pg_creating=1417969174,osdmap=596661807}) 2024-02-09T17:27:28.648 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=64} crc {osdmap=3608665156}) 2024-02-09T17:27:28.648 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:28.839 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:28 smithi179 ceph-mon[43975]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:27:28.839 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:28 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:27:28.839 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:28 smithi179 ceph-mon[43975]: osdmap e162: 12 total, 11 up, 12 in 2024-02-09T17:27:28.839 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=53,logm=7} crc {auth=3329010490,config=2679774077,health=3729121333,kv=3382112463,logm=745300946}) 2024-02-09T17:27:28.840 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2816134768}) 2024-02-09T17:27:28.840 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1300300781}) 2024-02-09T17:27:28.840 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3978572232}) 2024-02-09T17:27:28.840 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=357694623}) 2024-02-09T17:27:28.840 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2058891222}) 2024-02-09T17:27:28.840 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3871411908}) 2024-02-09T17:27:28.840 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2488200294}) 2024-02-09T17:27:28.840 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1791122756}) 2024-02-09T17:27:28.840 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2037867039}) 2024-02-09T17:27:28.840 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1720194272}) 2024-02-09T17:27:28.840 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1249130144}) 2024-02-09T17:27:28.840 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2795776881}) 2024-02-09T17:27:28.840 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1998478726}) 2024-02-09T17:27:28.840 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2589008878}) 2024-02-09T17:27:28.840 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2089805986}) 2024-02-09T17:27:28.841 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2525942273}) 2024-02-09T17:27:28.841 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619739444}) 2024-02-09T17:27:28.841 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2950400276}) 2024-02-09T17:27:28.841 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2155431284}) 2024-02-09T17:27:28.841 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1354958885}) 2024-02-09T17:27:28.841 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3691148013}) 2024-02-09T17:27:28.841 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=830577004}) 2024-02-09T17:27:28.841 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=591307711}) 2024-02-09T17:27:28.841 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=18,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=44} crc {logm=1434244818,mdsmap=3984425268,mgr=1708625883,mgr_command_descs=708342470,mgr_metadata=670763949,mgrstat=568593314,mon_config_key=4183486463}) 2024-02-09T17:27:28.841 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1317726899}) 2024-02-09T17:27:28.841 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1367840681}) 2024-02-09T17:27:28.841 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=37,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=45} crc {mon_config_key=2165145333,monmap=423865542,osd_metadata=2787106199,osd_pg_creating=1417969174,osdmap=596661807}) 2024-02-09T17:27:28.841 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=64} crc {osdmap=3608665156}) 2024-02-09T17:27:28.884 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:27:28.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:28.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:27:28.992 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:27:29.022 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:29.238 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:27:29.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:29.241 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:27:29.341 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:27:29.375 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:29.614 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-02-09T17:27:29.614 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd test-reweight-by-utilization 1000 3.0 2024-02-09T17:27:29.629 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:27:29.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:29.634 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:27:29.734 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:27:29.766 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:29.892 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:27:29.896 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:29 smithi179 ceph-mon[43975]: pgmap v484: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 4.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 729 op/s; 19024/227025 objects degraded (8.380%) 2024-02-09T17:27:29.896 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:29 smithi179 ceph-mon[43975]: from='client.24664 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:27:29.896 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:29 smithi179 ceph-mon[43975]: 2.8 scrub starts 2024-02-09T17:27:29.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:29.897 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:27:29.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:29 smithi162 ceph-mon[44304]: pgmap v484: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 4.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 729 op/s; 19024/227025 objects degraded (8.380%) 2024-02-09T17:27:29.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:29 smithi162 ceph-mon[44304]: from='client.24664 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:27:29.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:29 smithi162 ceph-mon[44304]: 2.8 scrub starts 2024-02-09T17:27:29.928 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T17:27:29.928 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:27:29.999 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:27:30.000 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:29 smithi099 ceph-mon[39420]: pgmap v484: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 4.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 729 op/s; 19024/227025 objects degraded (8.380%) 2024-02-09T17:27:30.000 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:29 smithi099 ceph-mon[39420]: from='client.24664 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:27:30.000 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:29 smithi099 ceph-mon[39420]: 2.8 scrub starts 2024-02-09T17:27:30.032 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:30.161 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:27:30.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:30.164 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:27:30.265 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:27:30.320 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:30.400 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:27:30.402 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:30.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:27:30.504 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:27:30.533 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:30.621 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:30 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:27:30.621 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:30 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:27:30.621 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:30 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:27:30.621 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:30 smithi099 ceph-mon[39420]: 2.8 scrub ok 2024-02-09T17:27:30.621 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:30 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:27:30.621 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:30 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:27:30.621 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:30 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:27:30.621 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:30 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:27:30.621 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:30 smithi099 ceph-mon[39420]: 2.1 scrub starts 2024-02-09T17:27:30.638 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:27:30.640 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:27:30.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:30.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:27:30.750 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:27:30.808 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:30.831 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.10 dump_historic_ops 2024-02-09T17:27:30.920 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:27:30.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:30 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:27:30.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:30 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:27:30.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:30 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:27:30.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:30 smithi179 ceph-mon[43975]: 2.8 scrub ok 2024-02-09T17:27:30.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:30 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:27:30.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:30 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:27:30.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:30 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:27:30.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:30 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:27:30.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:30 smithi179 ceph-mon[43975]: 2.1 scrub starts 2024-02-09T17:27:30.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:30 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:27:30.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:30 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:27:30.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:30 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:27:30.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:30 smithi162 ceph-mon[44304]: 2.8 scrub ok 2024-02-09T17:27:30.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:30 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:27:30.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:30 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:27:30.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:30 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:27:30.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:30 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:27:30.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:30 smithi162 ceph-mon[44304]: 2.1 scrub starts 2024-02-09T17:27:30.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:30.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:27:31.030 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:27:31.121 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:31.165 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:27:31.230 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:27:31.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:31.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:27:31.337 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:27:31.408 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:31.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:31 smithi099 ceph-mon[39420]: pgmap v485: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 4.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 951 op/s; 18204/219714 objects degraded (8.285%) 2024-02-09T17:27:31.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:31 smithi099 ceph-mon[39420]: 2.7 scrub starts 2024-02-09T17:27:31.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:31 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 18204/219714 objects degraded (8.285%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:27:31.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:31 smithi099 ceph-mon[39420]: 2.3 scrub starts 2024-02-09T17:27:31.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:31 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1396057576' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:27:32.099 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-09T17:27:32.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:31 smithi179 ceph-mon[43975]: pgmap v485: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 4.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 951 op/s; 18204/219714 objects degraded (8.285%) 2024-02-09T17:27:32.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:31 smithi179 ceph-mon[43975]: 2.7 scrub starts 2024-02-09T17:27:32.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:31 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 18204/219714 objects degraded (8.285%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:27:32.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:31 smithi179 ceph-mon[43975]: 2.3 scrub starts 2024-02-09T17:27:32.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:31 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1396057576' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:27:32.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:31 smithi162 ceph-mon[44304]: pgmap v485: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 4.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 951 op/s; 18204/219714 objects degraded (8.285%) 2024-02-09T17:27:32.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:31 smithi162 ceph-mon[44304]: 2.7 scrub starts 2024-02-09T17:27:32.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:31 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 18204/219714 objects degraded (8.285%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:27:32.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:31 smithi162 ceph-mon[44304]: 2.3 scrub starts 2024-02-09T17:27:32.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:31 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1396057576' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:27:32.413 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:27:32.413 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:27:32.413 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:27:32.413 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:27:32.413 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:27:32.413 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:27:32.414 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:27:32.414 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:27:32.414 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:27:32.414 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:27:32.414 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:27:32.414 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:27:32.414 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:27:32.414 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:27:32.414 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:27:32.414 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:27:32.414 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:27:32.415 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:27:32.415 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:27:32.415 INFO:teuthology.orchestra.run.smithi099.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-02-09T17:27:32.415 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:27:32.415 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:27:32.415 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:27:32.696 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:27:32.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:32.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:27:32.802 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:27:32.837 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:33.000 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:32 smithi099 ceph-mon[39420]: 2.1 scrub ok 2024-02-09T17:27:33.000 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:32 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:27:33.000 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:32 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:27:33.000 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:32 smithi099 ceph-mon[39420]: 2.7 scrub ok 2024-02-09T17:27:33.000 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:32 smithi099 ceph-mon[39420]: osdmap e163: 12 total, 11 up, 12 in 2024-02-09T17:27:33.000 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:32 smithi099 ceph-mon[39420]: 2.3 scrub ok 2024-02-09T17:27:33.023 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:32 smithi179 ceph-mon[43975]: 2.1 scrub ok 2024-02-09T17:27:33.023 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:32 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:27:33.023 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:32 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:27:33.023 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:32 smithi179 ceph-mon[43975]: 2.7 scrub ok 2024-02-09T17:27:33.023 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:32 smithi179 ceph-mon[43975]: osdmap e163: 12 total, 11 up, 12 in 2024-02-09T17:27:33.023 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:32 smithi179 ceph-mon[43975]: 2.3 scrub ok 2024-02-09T17:27:33.049 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:27:33.056 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:33.057 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:27:33.158 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:27:33.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:32 smithi162 ceph-mon[44304]: 2.1 scrub ok 2024-02-09T17:27:33.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:32 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:27:33.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:32 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:27:33.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:32 smithi162 ceph-mon[44304]: 2.7 scrub ok 2024-02-09T17:27:33.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:32 smithi162 ceph-mon[44304]: osdmap e163: 12 total, 11 up, 12 in 2024-02-09T17:27:33.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:32 smithi162 ceph-mon[44304]: 2.3 scrub ok 2024-02-09T17:27:33.191 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:33.313 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:27:33.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:33.316 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:27:33.373 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:27:33.374 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-02-09T17:27:33.374 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-02-09T17:27:33.374 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-02-09T17:27:33.374 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.374 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.14790.0:116262 2.0 e139/111 2:0c0e0535:::benchmark_data_smithi179_77138_object116261:head v 139'7249, mlcod=139'7246)", 2024-02-09T17:27:33.374 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:26:01.482325+0000", 2024-02-09T17:27:33.374 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 91.881943651, 2024-02-09T17:27:33.374 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017600088999999999, 2024-02-09T17:27:33.374 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:27:33.374 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:33.374 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:27:33.374 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.374 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:27:33.375 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:01.482325+0000", 2024-02-09T17:27:33.375 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:33.375 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.375 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.375 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:27:33.375 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:01.482325+0000", 2024-02-09T17:27:33.375 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:33.375 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.375 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.375 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:27:33.375 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:01.482326+0000", 2024-02-09T17:27:33.375 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.158e-06 2024-02-09T17:27:33.375 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.375 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.376 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:27:33.376 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:01.482403+0000", 2024-02-09T17:27:33.376 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6921000000000004e-05 2024-02-09T17:27:33.376 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.376 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.376 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:27:33.376 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:01.482418+0000", 2024-02-09T17:27:33.376 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5415000000000001e-05 2024-02-09T17:27:33.376 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.376 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.376 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:27:33.376 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:01.482427+0000", 2024-02-09T17:27:33.376 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4610000000000008e-06 2024-02-09T17:27:33.376 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.377 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.377 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:27:33.377 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:01.482501+0000", 2024-02-09T17:27:33.377 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4231000000000001e-05 2024-02-09T17:27:33.377 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.377 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.377 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:27:33.377 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:01.482516+0000", 2024-02-09T17:27:33.377 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4825e-05 2024-02-09T17:27:33.377 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.377 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.377 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:27:33.377 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:01.499882+0000", 2024-02-09T17:27:33.377 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017366719999999999 2024-02-09T17:27:33.377 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.378 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.378 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:27:33.378 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:01.499925+0000", 2024-02-09T17:27:33.378 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2357999999999997e-05 2024-02-09T17:27:33.378 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:33.378 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:27:33.378 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:33.378 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.378 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.378 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.14790.0:134913 2.7 e143/111 2:eeada740:::benchmark_data_smithi179_77138_object1505:head v 143'8336, mlcod=143'8332)", 2024-02-09T17:27:33.378 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:26:21.195015+0000", 2024-02-09T17:27:33.378 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 72.169253416999993, 2024-02-09T17:27:33.378 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.021894823000000001, 2024-02-09T17:27:33.378 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:27:33.379 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:33.379 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:27:33.379 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.379 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:27:33.379 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:21.195015+0000", 2024-02-09T17:27:33.379 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:33.379 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.379 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.379 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:27:33.379 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:21.195015+0000", 2024-02-09T17:27:33.379 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:33.379 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.379 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.379 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:27:33.380 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:21.195016+0000", 2024-02-09T17:27:33.380 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7700000000000003e-07 2024-02-09T17:27:33.380 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.380 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.380 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:27:33.380 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:21.195023+0000", 2024-02-09T17:27:33.380 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7740000000000004e-06 2024-02-09T17:27:33.380 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.380 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.380 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:27:33.380 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:21.195038+0000", 2024-02-09T17:27:33.380 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4842e-05 2024-02-09T17:27:33.380 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.380 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.381 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:27:33.381 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:21.195052+0000", 2024-02-09T17:27:33.381 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3603e-05 2024-02-09T17:27:33.381 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.381 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.381 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:27:33.381 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:21.195204+0000", 2024-02-09T17:27:33.381 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015165099999999999 2024-02-09T17:27:33.381 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.381 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.381 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:27:33.381 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:21.195226+0000", 2024-02-09T17:27:33.381 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2869000000000001e-05 2024-02-09T17:27:33.381 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.382 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.382 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:27:33.382 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:21.216874+0000", 2024-02-09T17:27:33.382 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.021647996999999999 2024-02-09T17:27:33.382 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.382 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.382 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:27:33.382 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:21.216910+0000", 2024-02-09T17:27:33.382 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.5209999999999997e-05 2024-02-09T17:27:33.382 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:33.382 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:27:33.382 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:33.382 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.382 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.383 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.14790.0:134920 2.7 e143/111 2:e2881413:::benchmark_data_smithi179_77138_object1512:head v 143'8337, mlcod=143'8333)", 2024-02-09T17:27:33.383 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:26:21.197967+0000", 2024-02-09T17:27:33.383 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 72.166301068999999, 2024-02-09T17:27:33.383 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.018972359000000001, 2024-02-09T17:27:33.383 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:27:33.383 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:33.383 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:27:33.383 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.383 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:27:33.383 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:21.197967+0000", 2024-02-09T17:27:33.383 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:33.383 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.383 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.383 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:27:33.384 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:21.197967+0000", 2024-02-09T17:27:33.384 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:33.384 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.384 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.384 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:27:33.384 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:21.197968+0000", 2024-02-09T17:27:33.384 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1399999999999996e-07 2024-02-09T17:27:33.384 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.384 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.384 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:27:33.384 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:21.197975+0000", 2024-02-09T17:27:33.384 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4139999999999997e-06 2024-02-09T17:27:33.384 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.384 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.385 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:27:33.385 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:21.197989+0000", 2024-02-09T17:27:33.385 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3464e-05 2024-02-09T17:27:33.385 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.385 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.385 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:27:33.385 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:21.197996+0000", 2024-02-09T17:27:33.385 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4889999999999998e-06 2024-02-09T17:27:33.385 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.385 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.385 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:27:33.385 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:21.199565+0000", 2024-02-09T17:27:33.385 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0015682280000000001 2024-02-09T17:27:33.385 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.385 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.385 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:27:33.386 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:21.199589+0000", 2024-02-09T17:27:33.386 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4479999999999999e-05 2024-02-09T17:27:33.386 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.386 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.386 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:27:33.386 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:21.216914+0000", 2024-02-09T17:27:33.386 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017324705999999999 2024-02-09T17:27:33.386 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.386 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.386 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:27:33.386 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:21.216940+0000", 2024-02-09T17:27:33.386 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5763999999999999e-05 2024-02-09T17:27:33.387 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:33.387 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:27:33.387 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:33.387 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.387 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.387 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.14790.0:135345 2.7 e143/111 2:e6a5e41a:::benchmark_data_smithi179_77138_object1937:head v 143'8360, mlcod=143'8358)", 2024-02-09T17:27:33.387 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:26:21.657804+0000", 2024-02-09T17:27:33.387 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 71.706464018999995, 2024-02-09T17:27:33.387 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.020278379999999999, 2024-02-09T17:27:33.387 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:27:33.387 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:33.388 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:27:33.388 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.388 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:27:33.388 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:21.657804+0000", 2024-02-09T17:27:33.388 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:33.388 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.388 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.388 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:27:33.388 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:21.657804+0000", 2024-02-09T17:27:33.388 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:33.388 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.388 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.388 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:27:33.388 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:21.657805+0000", 2024-02-09T17:27:33.389 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.133e-06 2024-02-09T17:27:33.389 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.389 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.389 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:27:33.389 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:21.657815+0000", 2024-02-09T17:27:33.389 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.4259999999999992e-06 2024-02-09T17:27:33.389 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.389 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.389 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:27:33.389 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:21.657833+0000", 2024-02-09T17:27:33.389 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8590999999999999e-05 2024-02-09T17:27:33.389 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.389 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.389 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:27:33.389 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:21.657843+0000", 2024-02-09T17:27:33.390 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7049999999999994e-06 2024-02-09T17:27:33.390 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.390 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.390 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:27:33.390 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:21.657907+0000", 2024-02-09T17:27:33.390 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3848000000000003e-05 2024-02-09T17:27:33.390 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.390 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.390 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:27:33.390 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:21.657932+0000", 2024-02-09T17:27:33.390 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5066e-05 2024-02-09T17:27:33.390 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.390 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.390 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:27:33.391 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:21.678041+0000", 2024-02-09T17:27:33.391 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.020108529 2024-02-09T17:27:33.391 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.391 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.391 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:27:33.391 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:21.678083+0000", 2024-02-09T17:27:33.391 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2082000000000001e-05 2024-02-09T17:27:33.391 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:33.391 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:27:33.391 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:33.391 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.391 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.391 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.14790.0:137447 2.7 e143/111 2:e0c67f23:::benchmark_data_smithi179_77138_object4039:head v 143'8476, mlcod=143'8473)", 2024-02-09T17:27:33.391 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:26:24.086222+0000", 2024-02-09T17:27:33.391 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 69.278046748999998, 2024-02-09T17:27:33.391 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017732062999999999, 2024-02-09T17:27:33.392 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:27:33.392 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:33.392 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:27:33.392 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.392 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:27:33.392 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:24.086222+0000", 2024-02-09T17:27:33.392 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:33.392 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.392 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.392 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:27:33.392 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:24.086222+0000", 2024-02-09T17:27:33.392 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:33.392 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.393 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.393 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:27:33.393 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:24.086222+0000", 2024-02-09T17:27:33.393 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.9999999999999999e-07 2024-02-09T17:27:33.393 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.393 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.393 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:27:33.393 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:24.086227+0000", 2024-02-09T17:27:33.393 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7099999999999998e-06 2024-02-09T17:27:33.393 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.393 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.393 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:27:33.393 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:24.086232+0000", 2024-02-09T17:27:33.393 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.818e-06 2024-02-09T17:27:33.393 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.394 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.394 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:27:33.394 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:24.086237+0000", 2024-02-09T17:27:33.394 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.42e-06 2024-02-09T17:27:33.394 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.394 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.394 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:27:33.394 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:24.090005+0000", 2024-02-09T17:27:33.394 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00376787 2024-02-09T17:27:33.394 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.394 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.394 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:27:33.394 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:24.090024+0000", 2024-02-09T17:27:33.394 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9188999999999999e-05 2024-02-09T17:27:33.394 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.394 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.395 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:27:33.395 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:24.103763+0000", 2024-02-09T17:27:33.395 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013739240999999999 2024-02-09T17:27:33.395 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.395 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.395 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:27:33.395 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:24.103954+0000", 2024-02-09T17:27:33.395 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00019051499999999999 2024-02-09T17:27:33.395 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:33.395 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:27:33.395 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:33.396 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.396 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.396 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.14790.0:139967 2.0 e143/111 2:05d5ff1a:::benchmark_data_smithi179_77138_object6559:head v 143'8712, mlcod=143'8710)", 2024-02-09T17:27:33.396 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:26:26.654621+0000", 2024-02-09T17:27:33.396 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 66.709647343, 2024-02-09T17:27:33.396 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017790988000000001, 2024-02-09T17:27:33.396 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:27:33.396 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:33.396 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:27:33.396 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.396 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:27:33.396 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:26.654621+0000", 2024-02-09T17:27:33.396 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:33.397 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.397 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.397 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:27:33.397 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:26.654621+0000", 2024-02-09T17:27:33.397 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:33.397 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.397 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.397 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:27:33.397 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:26.654622+0000", 2024-02-09T17:27:33.397 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0699999999999999e-06 2024-02-09T17:27:33.397 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.397 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.397 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:27:33.397 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:26.654630+0000", 2024-02-09T17:27:33.397 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.622e-06 2024-02-09T17:27:33.398 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.398 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.398 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:27:33.398 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:26.654643+0000", 2024-02-09T17:27:33.398 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3152999999999999e-05 2024-02-09T17:27:33.398 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.398 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.398 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:27:33.398 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:26.654650+0000", 2024-02-09T17:27:33.398 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6159999999999996e-06 2024-02-09T17:27:33.398 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.398 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.398 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:27:33.398 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:26.654710+0000", 2024-02-09T17:27:33.399 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9948000000000003e-05 2024-02-09T17:27:33.399 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.399 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.399 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:27:33.399 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:26.654730+0000", 2024-02-09T17:27:33.399 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9409999999999999e-05 2024-02-09T17:27:33.399 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.399 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.399 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:27:33.399 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:26.672370+0000", 2024-02-09T17:27:33.399 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017639841999999999 2024-02-09T17:27:33.399 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.399 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.399 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:27:33.399 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:26.672412+0000", 2024-02-09T17:27:33.399 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2327000000000003e-05 2024-02-09T17:27:33.400 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:33.400 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:27:33.400 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:33.400 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.400 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.400 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.14790.0:142845 2.7 e144/111 2:e7ffca81:::benchmark_data_smithi179_77138_object9437:head v 144'8810, mlcod=144'8807)", 2024-02-09T17:27:33.400 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:26:29.641507+0000", 2024-02-09T17:27:33.400 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 63.722761052000003, 2024-02-09T17:27:33.400 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.022092434000000001, 2024-02-09T17:27:33.400 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:27:33.400 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:33.400 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:27:33.400 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.400 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:27:33.400 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:29.641507+0000", 2024-02-09T17:27:33.400 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:33.401 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.401 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.401 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:27:33.401 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:29.641507+0000", 2024-02-09T17:27:33.401 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:33.401 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.401 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.401 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:27:33.401 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:29.641508+0000", 2024-02-09T17:27:33.401 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.3399999999999997e-07 2024-02-09T17:27:33.401 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.401 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.401 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:27:33.401 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:29.641515+0000", 2024-02-09T17:27:33.401 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.669e-06 2024-02-09T17:27:33.402 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.402 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.402 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:27:33.402 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:29.641528+0000", 2024-02-09T17:27:33.402 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3623e-05 2024-02-09T17:27:33.402 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.402 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.402 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:27:33.402 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:29.641536+0000", 2024-02-09T17:27:33.402 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4390000000000003e-06 2024-02-09T17:27:33.402 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.402 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.403 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:27:33.403 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:29.641601+0000", 2024-02-09T17:27:33.403 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5272999999999997e-05 2024-02-09T17:27:33.403 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.403 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.403 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:27:33.403 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:29.641621+0000", 2024-02-09T17:27:33.403 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9698999999999999e-05 2024-02-09T17:27:33.403 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.403 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.403 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:27:33.404 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:29.663555+0000", 2024-02-09T17:27:33.404 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.021933736999999998 2024-02-09T17:27:33.404 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.404 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.404 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:27:33.404 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:29.663600+0000", 2024-02-09T17:27:33.404 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5059999999999999e-05 2024-02-09T17:27:33.404 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:33.404 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:27:33.404 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:33.404 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.404 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.405 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.14790.0:144396 2.7 e144/111 2:e8a2cdad:::benchmark_data_smithi179_77138_object10988:head v 144'8915, mlcod=144'8913)", 2024-02-09T17:27:33.405 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:26:31.351554+0000", 2024-02-09T17:27:33.405 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 62.012714297999999, 2024-02-09T17:27:33.405 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.018848812, 2024-02-09T17:27:33.405 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:27:33.406 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:33.406 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:27:33.406 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.406 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:27:33.406 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:31.351554+0000", 2024-02-09T17:27:33.406 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:33.407 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.407 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.407 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:27:33.407 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:31.351554+0000", 2024-02-09T17:27:33.407 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:33.407 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.407 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.408 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:27:33.408 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:31.351555+0000", 2024-02-09T17:27:33.408 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.057e-06 2024-02-09T17:27:33.408 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.408 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.408 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:27:33.408 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:31.351562+0000", 2024-02-09T17:27:33.408 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2370000000000003e-06 2024-02-09T17:27:33.408 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.408 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.408 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:27:33.408 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:31.351578+0000", 2024-02-09T17:27:33.409 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5345999999999998e-05 2024-02-09T17:27:33.409 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.409 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.409 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:27:33.409 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:31.351586+0000", 2024-02-09T17:27:33.409 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.354e-06 2024-02-09T17:27:33.410 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.410 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.410 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:27:33.410 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:31.355549+0000", 2024-02-09T17:27:33.410 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.003962859 2024-02-09T17:27:33.410 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.410 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.410 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:27:33.410 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:31.355568+0000", 2024-02-09T17:27:33.410 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9612000000000001e-05 2024-02-09T17:27:33.410 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.410 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.410 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:27:33.410 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:31.370360+0000", 2024-02-09T17:27:33.411 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014791484000000001 2024-02-09T17:27:33.411 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.411 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.411 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:27:33.411 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:31.370403+0000", 2024-02-09T17:27:33.411 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2862999999999999e-05 2024-02-09T17:27:33.411 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:33.411 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:27:33.411 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:33.411 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.411 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.411 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.14790.0:144394 2.0 e144/111 2:03be6a8c:::benchmark_data_smithi179_77138_object10986:head v 144'8968, mlcod=144'8965)", 2024-02-09T17:27:33.411 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:26:31.354991+0000", 2024-02-09T17:27:33.411 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 62.009276823, 2024-02-09T17:27:33.411 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017418326000000001, 2024-02-09T17:27:33.412 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:27:33.412 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:33.412 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:27:33.412 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.412 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:27:33.412 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:31.354991+0000", 2024-02-09T17:27:33.412 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:33.412 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.412 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.412 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:27:33.412 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:31.354991+0000", 2024-02-09T17:27:33.412 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:33.412 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.412 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.412 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:27:33.413 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:31.354992+0000", 2024-02-09T17:27:33.413 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2699999999999999e-07 2024-02-09T17:27:33.413 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.413 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.413 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:27:33.413 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:31.354998+0000", 2024-02-09T17:27:33.413 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5790000000000003e-06 2024-02-09T17:27:33.413 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.413 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.413 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:27:33.413 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:31.355009+0000", 2024-02-09T17:27:33.413 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1819e-05 2024-02-09T17:27:33.413 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.413 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.413 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:27:33.414 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:31.355016+0000", 2024-02-09T17:27:33.414 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2149999999999998e-06 2024-02-09T17:27:33.414 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.414 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.414 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:27:33.414 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:31.358012+0000", 2024-02-09T17:27:33.414 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002996425 2024-02-09T17:27:33.414 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.414 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.414 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:27:33.414 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:31.358026+0000", 2024-02-09T17:27:33.414 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4279e-05 2024-02-09T17:27:33.414 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.414 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.414 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:27:33.415 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:31.372371+0000", 2024-02-09T17:27:33.415 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014344806999999999 2024-02-09T17:27:33.415 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.415 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.415 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:27:33.415 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:31.372410+0000", 2024-02-09T17:27:33.415 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8575000000000003e-05 2024-02-09T17:27:33.415 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:33.415 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:27:33.415 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:33.415 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.415 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.415 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.14790.0:145492 2.0 e144/111 2:03a13ba0:::benchmark_data_smithi179_77138_object12084:head v 144'9036, mlcod=144'9033)", 2024-02-09T17:27:33.415 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:26:32.428613+0000", 2024-02-09T17:27:33.415 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 60.935655107000002, 2024-02-09T17:27:33.415 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.019197809999999999, 2024-02-09T17:27:33.416 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:27:33.416 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:33.416 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:27:33.416 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.416 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:27:33.416 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:32.428613+0000", 2024-02-09T17:27:33.416 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:33.416 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.416 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.416 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:27:33.416 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:32.428613+0000", 2024-02-09T17:27:33.416 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:33.416 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.416 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.416 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:27:33.417 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:32.428614+0000", 2024-02-09T17:27:33.417 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.161e-06 2024-02-09T17:27:33.417 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.417 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.417 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:27:33.417 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:32.428623+0000", 2024-02-09T17:27:33.417 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.0089999999999996e-06 2024-02-09T17:27:33.417 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.417 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.417 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:27:33.417 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:32.428641+0000", 2024-02-09T17:27:33.417 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7218999999999999e-05 2024-02-09T17:27:33.417 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.418 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.418 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:27:33.418 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:32.428650+0000", 2024-02-09T17:27:33.418 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6909999999999992e-06 2024-02-09T17:27:33.418 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.418 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.418 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:27:33.418 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:32.432415+0000", 2024-02-09T17:27:33.418 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0037649129999999999 2024-02-09T17:27:33.418 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.418 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.418 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:27:33.418 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:32.432443+0000", 2024-02-09T17:27:33.418 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.8153999999999998e-05 2024-02-09T17:27:33.419 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.419 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.419 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:27:33.419 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:32.447764+0000", 2024-02-09T17:27:33.419 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.015321034000000001 2024-02-09T17:27:33.419 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.419 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.419 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:27:33.419 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:32.447811+0000", 2024-02-09T17:27:33.419 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6628999999999997e-05 2024-02-09T17:27:33.419 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:33.419 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:27:33.419 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:33.420 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.420 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.420 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.14790.0:146360 2.4 e144/111 2:2634f499:::benchmark_data_smithi179_77138_object12952:head v 144'9097, mlcod=144'9094)", 2024-02-09T17:27:33.420 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:26:33.303523+0000", 2024-02-09T17:27:33.420 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 60.060744986000003, 2024-02-09T17:27:33.420 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.018543517999999998, 2024-02-09T17:27:33.420 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:27:33.420 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:33.421 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:27:33.421 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.421 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:27:33.421 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:33.303523+0000", 2024-02-09T17:27:33.421 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:33.421 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.421 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.421 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:27:33.421 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:33.303523+0000", 2024-02-09T17:27:33.421 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:33.421 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.421 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.421 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:27:33.422 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:33.303524+0000", 2024-02-09T17:27:33.422 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9400000000000001e-07 2024-02-09T17:27:33.422 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.422 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.422 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:27:33.422 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:33.303528+0000", 2024-02-09T17:27:33.422 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.605e-06 2024-02-09T17:27:33.422 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.422 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.422 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:27:33.422 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:33.303536+0000", 2024-02-09T17:27:33.422 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2590000000000003e-06 2024-02-09T17:27:33.422 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.422 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.422 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:27:33.423 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:33.303541+0000", 2024-02-09T17:27:33.423 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4729999999999999e-06 2024-02-09T17:27:33.423 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.423 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.423 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:27:33.423 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:33.306941+0000", 2024-02-09T17:27:33.423 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0033998639999999998 2024-02-09T17:27:33.423 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.423 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.423 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:27:33.423 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:33.306963+0000", 2024-02-09T17:27:33.423 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2589999999999999e-05 2024-02-09T17:27:33.423 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.423 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.423 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:27:33.423 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:33.322025+0000", 2024-02-09T17:27:33.423 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.015061774999999999 2024-02-09T17:27:33.424 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.424 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.424 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:27:33.424 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:33.322067+0000", 2024-02-09T17:27:33.424 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1557999999999998e-05 2024-02-09T17:27:33.424 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:33.424 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:27:33.424 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:33.424 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.424 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.424 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.14790.0:153827 2.4 e147/111 2:25270f2d:::benchmark_data_smithi179_77138_object20419:head v 147'9580, mlcod=147'9575)", 2024-02-09T17:27:33.424 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:26:42.264249+0000", 2024-02-09T17:27:33.424 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 51.100019551000003, 2024-02-09T17:27:33.425 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.023264401000000001, 2024-02-09T17:27:33.425 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:27:33.425 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:33.425 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:27:33.425 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.425 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:27:33.425 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:42.264249+0000", 2024-02-09T17:27:33.425 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:33.425 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.425 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.425 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:27:33.425 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:42.264249+0000", 2024-02-09T17:27:33.425 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:33.425 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.426 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.426 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:27:33.426 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:42.264249+0000", 2024-02-09T17:27:33.426 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4000000000000002e-07 2024-02-09T17:27:33.426 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.426 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.426 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:27:33.426 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:42.264256+0000", 2024-02-09T17:27:33.426 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2700000000000001e-06 2024-02-09T17:27:33.426 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.426 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.426 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:27:33.426 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:42.264267+0000", 2024-02-09T17:27:33.427 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1779e-05 2024-02-09T17:27:33.427 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.427 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.427 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:27:33.427 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:42.264275+0000", 2024-02-09T17:27:33.427 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4340000000000003e-06 2024-02-09T17:27:33.427 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.427 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.427 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:27:33.427 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:42.268734+0000", 2024-02-09T17:27:33.427 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0044592099999999999 2024-02-09T17:27:33.427 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.427 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.427 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:27:33.427 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:42.268756+0000", 2024-02-09T17:27:33.428 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2101000000000001e-05 2024-02-09T17:27:33.428 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.428 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.428 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:27:33.428 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:42.287464+0000", 2024-02-09T17:27:33.428 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.018708358000000001 2024-02-09T17:27:33.428 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.428 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.428 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:27:33.428 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:42.287513+0000", 2024-02-09T17:27:33.428 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8708999999999997e-05 2024-02-09T17:27:33.428 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:33.429 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:27:33.429 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:33.429 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.429 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.429 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.14790.0:153828 2.4 e147/111 2:2d175386:::benchmark_data_smithi179_77138_object20420:head v 147'9581, mlcod=147'9575)", 2024-02-09T17:27:33.429 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:26:42.267119+0000", 2024-02-09T17:27:33.429 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 51.097148926000003, 2024-02-09T17:27:33.429 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.020428993999999999, 2024-02-09T17:27:33.429 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:27:33.429 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:33.429 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:27:33.429 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.429 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:27:33.430 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:42.267119+0000", 2024-02-09T17:27:33.430 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:33.430 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.430 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.430 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:27:33.430 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:42.267119+0000", 2024-02-09T17:27:33.430 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:33.430 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.430 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.430 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:27:33.430 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:42.267121+0000", 2024-02-09T17:27:33.430 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.663e-06 2024-02-09T17:27:33.430 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.431 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.431 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:27:33.431 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:42.267130+0000", 2024-02-09T17:27:33.431 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7770000000000008e-06 2024-02-09T17:27:33.431 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.431 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.431 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:27:33.431 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:42.267153+0000", 2024-02-09T17:27:33.431 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3020999999999999e-05 2024-02-09T17:27:33.431 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.431 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.431 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:27:33.431 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:42.267163+0000", 2024-02-09T17:27:33.431 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0022e-05 2024-02-09T17:27:33.431 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.431 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.432 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:27:33.432 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:42.272003+0000", 2024-02-09T17:27:33.432 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0048404650000000004 2024-02-09T17:27:33.432 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.432 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.432 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:27:33.432 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:42.272023+0000", 2024-02-09T17:27:33.432 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9304e-05 2024-02-09T17:27:33.432 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.432 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.432 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:27:33.432 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:42.287518+0000", 2024-02-09T17:27:33.432 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.015495282000000001 2024-02-09T17:27:33.432 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.433 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.433 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:27:33.433 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:42.287548+0000", 2024-02-09T17:27:33.433 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.046e-05 2024-02-09T17:27:33.433 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:33.433 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:27:33.433 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:33.433 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.433 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.433 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.14790.0:166445 2.0 e150/111 2:0018a928:::benchmark_data_smithi179_77138_object33037:head v 150'10367, mlcod=150'10364)", 2024-02-09T17:27:33.433 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:26:56.238313+0000", 2024-02-09T17:27:33.433 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 37.125955331999997, 2024-02-09T17:27:33.433 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.019898306000000001, 2024-02-09T17:27:33.433 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:27:33.433 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:33.433 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:27:33.433 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.434 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:27:33.434 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:56.238313+0000", 2024-02-09T17:27:33.434 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:33.434 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.434 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.434 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:27:33.434 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:56.238313+0000", 2024-02-09T17:27:33.434 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:33.434 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.434 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.434 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:27:33.434 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:56.238313+0000", 2024-02-09T17:27:33.434 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5400000000000002e-07 2024-02-09T17:27:33.434 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.434 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.434 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:27:33.434 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:56.238317+0000", 2024-02-09T17:27:33.435 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4020000000000002e-06 2024-02-09T17:27:33.435 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.435 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.435 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:27:33.435 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:56.239626+0000", 2024-02-09T17:27:33.435 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0013093779999999999 2024-02-09T17:27:33.435 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.435 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.435 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:27:33.435 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:56.239637+0000", 2024-02-09T17:27:33.435 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0873000000000001e-05 2024-02-09T17:27:33.435 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.435 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.435 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:27:33.436 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:56.243043+0000", 2024-02-09T17:27:33.436 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0034057570000000001 2024-02-09T17:27:33.436 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.436 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.436 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:27:33.436 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:56.243067+0000", 2024-02-09T17:27:33.436 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.48e-05 2024-02-09T17:27:33.436 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.436 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.436 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:27:33.436 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:56.258149+0000", 2024-02-09T17:27:33.436 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.01508163 2024-02-09T17:27:33.436 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.437 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.437 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:27:33.437 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:56.258211+0000", 2024-02-09T17:27:33.437 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2212000000000007e-05 2024-02-09T17:27:33.437 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:33.437 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:27:33.437 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:33.437 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.437 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.437 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.14790.0:172395 2.0 e150/111 2:0680ad1e:::benchmark_data_smithi179_77138_object38987:head v 150'10732, mlcod=150'10730)", 2024-02-09T17:27:33.437 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:02.816376+0000", 2024-02-09T17:27:33.437 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 30.547892397999998, 2024-02-09T17:27:33.437 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.018592431999999999, 2024-02-09T17:27:33.437 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:27:33.437 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:33.437 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:27:33.437 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.438 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:27:33.438 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:02.816376+0000", 2024-02-09T17:27:33.438 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:33.438 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.438 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.438 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:27:33.438 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:02.816376+0000", 2024-02-09T17:27:33.438 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:33.438 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.438 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.438 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:27:33.438 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:02.816377+0000", 2024-02-09T17:27:33.438 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2500000000000005e-07 2024-02-09T17:27:33.438 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.439 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.439 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:27:33.439 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:02.816383+0000", 2024-02-09T17:27:33.439 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.409e-06 2024-02-09T17:27:33.439 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.439 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.439 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:27:33.439 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:02.816397+0000", 2024-02-09T17:27:33.439 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4103e-05 2024-02-09T17:27:33.439 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.439 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.439 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:27:33.439 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:02.816404+0000", 2024-02-09T17:27:33.439 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8499999999999996e-06 2024-02-09T17:27:33.440 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.440 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.440 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:27:33.440 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:02.816455+0000", 2024-02-09T17:27:33.440 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1345999999999999e-05 2024-02-09T17:27:33.440 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.440 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.440 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:27:33.440 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:02.816477+0000", 2024-02-09T17:27:33.440 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2138e-05 2024-02-09T17:27:33.440 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.440 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.440 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:27:33.441 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:02.834923+0000", 2024-02-09T17:27:33.441 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.018446002 2024-02-09T17:27:33.441 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.441 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.441 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:27:33.441 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:02.834968+0000", 2024-02-09T17:27:33.441 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4858999999999999e-05 2024-02-09T17:27:33.441 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:33.441 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:27:33.441 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:33.441 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.441 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.441 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.14790.0:173017 2.0 e150/111 2:060cb559:::benchmark_data_smithi179_77138_object39609:head v 150'10783, mlcod=150'10780)", 2024-02-09T17:27:33.442 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:03.479974+0000", 2024-02-09T17:27:33.442 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 29.884293964000001, 2024-02-09T17:27:33.442 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017568853999999998, 2024-02-09T17:27:33.442 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:27:33.442 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:33.442 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:27:33.443 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.443 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:27:33.443 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:03.479974+0000", 2024-02-09T17:27:33.443 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:33.443 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.443 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.443 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:27:33.443 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:03.479974+0000", 2024-02-09T17:27:33.443 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:33.443 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.443 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.443 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:27:33.443 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:03.479975+0000", 2024-02-09T17:27:33.443 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3499999999999996e-07 2024-02-09T17:27:33.443 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.443 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.444 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:27:33.444 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:03.479981+0000", 2024-02-09T17:27:33.444 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9680000000000003e-06 2024-02-09T17:27:33.444 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.444 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.444 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:27:33.445 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:03.479991+0000", 2024-02-09T17:27:33.445 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7620000000000007e-06 2024-02-09T17:27:33.445 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.445 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.445 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:27:33.445 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:03.479997+0000", 2024-02-09T17:27:33.446 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1480000000000003e-06 2024-02-09T17:27:33.446 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.446 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.446 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:27:33.446 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:03.483926+0000", 2024-02-09T17:27:33.446 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0039295470000000002 2024-02-09T17:27:33.446 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.447 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.447 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:27:33.447 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:03.483953+0000", 2024-02-09T17:27:33.447 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6588999999999999e-05 2024-02-09T17:27:33.447 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.447 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.447 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:27:33.447 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:03.497494+0000", 2024-02-09T17:27:33.447 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013540916 2024-02-09T17:27:33.447 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.448 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.448 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:27:33.448 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:03.497543+0000", 2024-02-09T17:27:33.448 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9289000000000002e-05 2024-02-09T17:27:33.448 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:33.448 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:27:33.449 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:33.449 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.449 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.449 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.14790.0:182210 2.0 e155/111 2:016ed69b:::benchmark_data_smithi179_77138_object48802:head v 155'11399, mlcod=155'11397)", 2024-02-09T17:27:33.449 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:13.822419+0000", 2024-02-09T17:27:33.449 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 19.541848847000001, 2024-02-09T17:27:33.449 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.018413109, 2024-02-09T17:27:33.449 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:27:33.449 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:33.449 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:27:33.450 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.450 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:27:33.450 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.822419+0000", 2024-02-09T17:27:33.450 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:33.450 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.450 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.450 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:27:33.450 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.822419+0000", 2024-02-09T17:27:33.450 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:33.450 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.450 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.450 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:27:33.450 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.822420+0000", 2024-02-09T17:27:33.450 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.8899999999999998e-07 2024-02-09T17:27:33.450 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.450 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.451 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:27:33.451 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.822427+0000", 2024-02-09T17:27:33.451 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.0190000000000001e-06 2024-02-09T17:27:33.451 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.451 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.451 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:27:33.451 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.822446+0000", 2024-02-09T17:27:33.451 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8189000000000002e-05 2024-02-09T17:27:33.451 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.451 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.451 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:27:33.451 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.822455+0000", 2024-02-09T17:27:33.451 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2220000000000008e-06 2024-02-09T17:27:33.451 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.451 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.452 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:27:33.452 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.822515+0000", 2024-02-09T17:27:33.452 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0659999999999999e-05 2024-02-09T17:27:33.452 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.452 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.452 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:27:33.452 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.822539+0000", 2024-02-09T17:27:33.452 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3578e-05 2024-02-09T17:27:33.452 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.452 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.452 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:27:33.452 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.840798+0000", 2024-02-09T17:27:33.452 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.018258921000000001 2024-02-09T17:27:33.453 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.453 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.453 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:27:33.453 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.840833+0000", 2024-02-09T17:27:33.453 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4530999999999999e-05 2024-02-09T17:27:33.453 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:33.453 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:27:33.453 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:33.453 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.453 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.453 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.14790.0:183581 2.4 e156/111 2:20de99f4:::benchmark_data_smithi179_77138_object50173:head v 156'11413, mlcod=156'11411)", 2024-02-09T17:27:33.453 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:15.508792+0000", 2024-02-09T17:27:33.453 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 17.855475932000001, 2024-02-09T17:27:33.453 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.018715331000000002, 2024-02-09T17:27:33.453 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:27:33.453 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:33.453 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:27:33.454 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.454 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:27:33.454 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:15.508792+0000", 2024-02-09T17:27:33.454 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:33.454 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.454 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.454 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:27:33.454 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:15.508792+0000", 2024-02-09T17:27:33.454 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:33.454 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.454 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.454 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:27:33.454 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:15.508793+0000", 2024-02-09T17:27:33.454 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0449999999999999e-06 2024-02-09T17:27:33.454 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.454 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.455 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:27:33.455 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:15.508803+0000", 2024-02-09T17:27:33.455 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5610000000000004e-06 2024-02-09T17:27:33.455 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.455 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.455 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:27:33.455 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:15.508816+0000", 2024-02-09T17:27:33.455 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3407999999999999e-05 2024-02-09T17:27:33.455 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.455 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.455 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:27:33.455 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:15.508824+0000", 2024-02-09T17:27:33.455 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.0199999999999994e-06 2024-02-09T17:27:33.455 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.455 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.456 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:27:33.456 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:15.508885+0000", 2024-02-09T17:27:33.456 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0402000000000002e-05 2024-02-09T17:27:33.456 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.456 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.456 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:27:33.456 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:15.508909+0000", 2024-02-09T17:27:33.456 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4059000000000001e-05 2024-02-09T17:27:33.456 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.456 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.456 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:27:33.456 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:15.527475+0000", 2024-02-09T17:27:33.457 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.018566205999999998 2024-02-09T17:27:33.457 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.457 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.457 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:27:33.457 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:15.527508+0000", 2024-02-09T17:27:33.457 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2629999999999998e-05 2024-02-09T17:27:33.457 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:33.457 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:27:33.457 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:33.457 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.457 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.457 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.14790.0:183583 2.4 e156/111 2:22fbd71c:::benchmark_data_smithi179_77138_object50175:head v 156'11414, mlcod=156'11411)", 2024-02-09T17:27:33.457 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:15.508848+0000", 2024-02-09T17:27:33.457 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 17.855420008999999, 2024-02-09T17:27:33.457 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.018696829000000002, 2024-02-09T17:27:33.457 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:27:33.457 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:33.458 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:27:33.458 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.458 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:27:33.458 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:15.508848+0000", 2024-02-09T17:27:33.458 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:33.458 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.458 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.458 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:27:33.458 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:15.508848+0000", 2024-02-09T17:27:33.458 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:33.458 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.458 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.458 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:27:33.458 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:15.508849+0000", 2024-02-09T17:27:33.458 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7599999999999998e-07 2024-02-09T17:27:33.458 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.459 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.459 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:27:33.459 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:15.508853+0000", 2024-02-09T17:27:33.459 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6210000000000003e-06 2024-02-09T17:27:33.459 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.459 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.459 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:27:33.459 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:15.508860+0000", 2024-02-09T17:27:33.459 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1049999999999997e-06 2024-02-09T17:27:33.459 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.459 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.459 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:27:33.459 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:15.508866+0000", 2024-02-09T17:27:33.459 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.733e-06 2024-02-09T17:27:33.459 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.459 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.460 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:27:33.460 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:15.513451+0000", 2024-02-09T17:27:33.460 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0045848199999999999 2024-02-09T17:27:33.460 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.460 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.460 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:27:33.460 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:15.513474+0000", 2024-02-09T17:27:33.460 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3450000000000001e-05 2024-02-09T17:27:33.460 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.460 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.460 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:27:33.460 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:15.527513+0000", 2024-02-09T17:27:33.460 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014038893 2024-02-09T17:27:33.460 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.461 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.461 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:27:33.461 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:15.527545+0000", 2024-02-09T17:27:33.461 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1831000000000001e-05 2024-02-09T17:27:33.461 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:33.461 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:27:33.461 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:33.461 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.461 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.461 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.14790.0:183589 2.4 e156/111 2:2119cdac:::benchmark_data_smithi179_77138_object50181:head v 156'11415, mlcod=156'11411)", 2024-02-09T17:27:33.461 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:15.508875+0000", 2024-02-09T17:27:33.461 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 17.855393568, 2024-02-09T17:27:33.461 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.018695845999999999, 2024-02-09T17:27:33.462 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:27:33.462 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:33.462 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:27:33.462 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.462 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:27:33.462 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:15.508875+0000", 2024-02-09T17:27:33.462 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:33.462 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.462 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.462 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:27:33.462 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:15.508875+0000", 2024-02-09T17:27:33.462 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:33.462 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.462 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.463 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:27:33.463 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:15.508875+0000", 2024-02-09T17:27:33.463 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.9200000000000002e-07 2024-02-09T17:27:33.463 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.463 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.463 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:27:33.463 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:15.508879+0000", 2024-02-09T17:27:33.463 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0509999999999998e-06 2024-02-09T17:27:33.463 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.463 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.463 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:27:33.463 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:15.508885+0000", 2024-02-09T17:27:33.463 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8810000000000001e-06 2024-02-09T17:27:33.463 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.463 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.463 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:27:33.463 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:15.508890+0000", 2024-02-09T17:27:33.464 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6090000000000003e-06 2024-02-09T17:27:33.464 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.464 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.464 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:27:33.464 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:15.517508+0000", 2024-02-09T17:27:33.464 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0086182989999999994 2024-02-09T17:27:33.464 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.464 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.464 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:27:33.464 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:15.517531+0000", 2024-02-09T17:27:33.464 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3552000000000001e-05 2024-02-09T17:27:33.464 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.464 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.464 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:27:33.464 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:15.527547+0000", 2024-02-09T17:27:33.464 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010015639999999999 2024-02-09T17:27:33.465 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:33.465 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:33.465 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:27:33.465 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:15.527571+0000", 2024-02-09T17:27:33.465 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3521999999999999e-05 2024-02-09T17:27:33.465 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:33.465 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:27:33.465 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:33.465 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:33.465 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:27:33.465 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:27:33.465 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:27:33.496 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:33.519 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:27:33.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:33.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:27:33.583 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-09T17:27:33.583 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-09T17:27:33.625 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:27:33.658 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:33.684 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:27:33.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:33.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:27:33.726 INFO:teuthology.orchestra.run.smithi099.stdout:moved 0 / 51 (0%) 2024-02-09T17:27:33.727 INFO:teuthology.orchestra.run.smithi099.stdout:avg 4.63636 2024-02-09T17:27:33.727 INFO:teuthology.orchestra.run.smithi099.stdout:stddev 1.40248 -> 1.40248 (expected baseline 2.05302) 2024-02-09T17:27:33.727 INFO:teuthology.orchestra.run.smithi099.stdout:min osd.5 with 2 -> 2 pgs (0.431373 -> 0.431373 * mean) 2024-02-09T17:27:33.727 INFO:teuthology.orchestra.run.smithi099.stdout:max osd.3 with 6 -> 6 pgs (1.29412 -> 1.29412 * mean) 2024-02-09T17:27:33.727 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:27:33.727 INFO:teuthology.orchestra.run.smithi099.stdout:oload 1000 2024-02-09T17:27:33.727 INFO:teuthology.orchestra.run.smithi099.stdout:max_change 3 2024-02-09T17:27:33.727 INFO:teuthology.orchestra.run.smithi099.stdout:max_change_osds 4 2024-02-09T17:27:33.727 INFO:teuthology.orchestra.run.smithi099.stdout:average_utilization 0.0138 2024-02-09T17:27:33.727 INFO:teuthology.orchestra.run.smithi099.stdout:overload_utilization 0.1379 2024-02-09T17:27:33.728 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:27:33.742 INFO:teuthology.orchestra.run.smithi099.stderr:no change 2024-02-09T17:27:33.788 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:27:33.828 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:33 smithi162 ceph-mon[44304]: pgmap v487: 17 pgs: 1 active+clean+scrubbing, 4 active+undersized+degraded, 12 active+clean; 4.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 776 op/s; 17942/214257 objects degraded (8.374%) 2024-02-09T17:27:33.828 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:33 smithi162 ceph-mon[44304]: 2.2 scrub starts 2024-02-09T17:27:33.836 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:33.856 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:27:33.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:33.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:27:33.963 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:27:33.993 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:33.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:33 smithi099 ceph-mon[39420]: pgmap v487: 17 pgs: 1 active+clean+scrubbing, 4 active+undersized+degraded, 12 active+clean; 4.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 776 op/s; 17942/214257 objects degraded (8.374%) 2024-02-09T17:27:33.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:33 smithi099 ceph-mon[39420]: 2.2 scrub starts 2024-02-09T17:27:34.019 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:27:34.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:34.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:27:34.074 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:33 smithi179 ceph-mon[43975]: pgmap v487: 17 pgs: 1 active+clean+scrubbing, 4 active+undersized+degraded, 12 active+clean; 4.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 776 op/s; 17942/214257 objects degraded (8.374%) 2024-02-09T17:27:34.074 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:33 smithi179 ceph-mon[43975]: 2.2 scrub starts 2024-02-09T17:27:34.099 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.11 dump_ops_in_flight 2024-02-09T17:27:34.124 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:27:34.153 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:34.419 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:27:34.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:34 smithi099 ceph-mon[39420]: from='client.24676 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:27:34.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:34 smithi099 ceph-mon[39420]: 2.2 scrub ok 2024-02-09T17:27:34.998 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:34 smithi179 ceph-mon[43975]: from='client.24676 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:27:34.999 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:34 smithi179 ceph-mon[43975]: 2.2 scrub ok 2024-02-09T17:27:35.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:34 smithi162 ceph-mon[44304]: from='client.24676 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:27:35.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:34 smithi162 ceph-mon[44304]: 2.2 scrub ok 2024-02-09T17:27:35.584 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:27:35.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:35 smithi179 ceph-mon[43975]: pgmap v488: 17 pgs: 2 active+clean+scrubbing, 4 active+undersized+degraded, 11 active+clean; 4.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 793 op/s; 17422/209655 objects degraded (8.310%) 2024-02-09T17:27:35.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:35 smithi099 ceph-mon[39420]: pgmap v488: 17 pgs: 2 active+clean+scrubbing, 4 active+undersized+degraded, 11 active+clean; 4.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 793 op/s; 17422/209655 objects degraded (8.310%) 2024-02-09T17:27:36.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:35 smithi162 ceph-mon[44304]: pgmap v488: 17 pgs: 2 active+clean+scrubbing, 4 active+undersized+degraded, 11 active+clean; 4.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 793 op/s; 17422/209655 objects degraded (8.310%) 2024-02-09T17:27:36.628 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:27:36.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:36.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:27:36.663 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:27:36.663 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-02-09T17:27:36.663 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.663 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:200941 2.c 2:37f8fcde:::benchmark_data_smithi179_77138_object67533:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-02-09T17:27:36.663 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:36.636770+0000", 2024-02-09T17:27:36.663 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.025012801000000001, 2024-02-09T17:27:36.664 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.025051324999999999, 2024-02-09T17:27:36.664 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:27:36.664 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "waiting for sub ops", 2024-02-09T17:27:36.664 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:27:36.664 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:27:36.664 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:27:36.664 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 200941 2024-02-09T17:27:36.664 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.664 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:27:36.664 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.664 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:27:36.664 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.636770+0000", 2024-02-09T17:27:36.664 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:36.665 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.665 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.665 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:27:36.665 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.636770+0000", 2024-02-09T17:27:36.665 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:36.665 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.665 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.665 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:27:36.665 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.636770+0000", 2024-02-09T17:27:36.665 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9400000000000005e-07 2024-02-09T17:27:36.665 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.665 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.665 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:27:36.666 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.636774+0000", 2024-02-09T17:27:36.666 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2849999999999999e-06 2024-02-09T17:27:36.666 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.666 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.666 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:27:36.666 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.636783+0000", 2024-02-09T17:27:36.666 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8650000000000007e-06 2024-02-09T17:27:36.666 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.666 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.666 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:27:36.666 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.636789+0000", 2024-02-09T17:27:36.666 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4409999999999998e-06 2024-02-09T17:27:36.667 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.667 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.667 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:27:36.667 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.636839+0000", 2024-02-09T17:27:36.667 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0047999999999999e-05 2024-02-09T17:27:36.667 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.667 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.667 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:27:36.667 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.636966+0000", 2024-02-09T17:27:36.667 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000126988 2024-02-09T17:27:36.667 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.667 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.667 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,1", 2024-02-09T17:27:36.667 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.637091+0000", 2024-02-09T17:27:36.668 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012506799999999999 2024-02-09T17:27:36.668 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.668 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.668 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:36.668 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.647231+0000", 2024-02-09T17:27:36.668 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010139758 2024-02-09T17:27:36.668 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.668 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.668 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:27:36.668 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.647391+0000", 2024-02-09T17:27:36.668 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000159667 2024-02-09T17:27:36.668 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:36.668 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:27:36.669 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:36.669 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.669 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.669 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:200943 2.a 2:54c3b5c9:::benchmark_data_smithi179_77138_object67535:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-02-09T17:27:36.669 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:36.642214+0000", 2024-02-09T17:27:36.669 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.019568807000000001, 2024-02-09T17:27:36.669 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.019711124, 2024-02-09T17:27:36.669 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:27:36.669 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "waiting for sub ops", 2024-02-09T17:27:36.669 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:27:36.669 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:27:36.669 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:27:36.669 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 200943 2024-02-09T17:27:36.669 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.670 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:27:36.670 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.670 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:27:36.670 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.642214+0000", 2024-02-09T17:27:36.670 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:36.670 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.670 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.670 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:27:36.670 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.642214+0000", 2024-02-09T17:27:36.670 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:36.670 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.670 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.670 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:27:36.670 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.642216+0000", 2024-02-09T17:27:36.670 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4849999999999999e-06 2024-02-09T17:27:36.670 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.671 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.671 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:27:36.671 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.642223+0000", 2024-02-09T17:27:36.671 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6089999999999997e-06 2024-02-09T17:27:36.671 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.671 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.671 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:27:36.671 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.642237+0000", 2024-02-09T17:27:36.671 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3973e-05 2024-02-09T17:27:36.671 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.671 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.671 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:27:36.671 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.642248+0000", 2024-02-09T17:27:36.671 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0966e-05 2024-02-09T17:27:36.672 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.672 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.672 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:27:36.672 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.642306+0000", 2024-02-09T17:27:36.672 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7874000000000002e-05 2024-02-09T17:27:36.672 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.672 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.672 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:27:36.672 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.642423+0000", 2024-02-09T17:27:36.672 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000117343 2024-02-09T17:27:36.672 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.672 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.672 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,9", 2024-02-09T17:27:36.673 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.642542+0000", 2024-02-09T17:27:36.673 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000119092 2024-02-09T17:27:36.673 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.673 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.673 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:27:36.673 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.647671+0000", 2024-02-09T17:27:36.673 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0051285769999999996 2024-02-09T17:27:36.673 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.673 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.673 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:36.673 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.647745+0000", 2024-02-09T17:27:36.673 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4809999999999997e-05 2024-02-09T17:27:36.673 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:36.674 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:27:36.674 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:36.674 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.674 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.674 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:200953 2.c 2:31d8e808:::benchmark_data_smithi179_77138_object67545:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-02-09T17:27:36.674 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:36.647733+0000", 2024-02-09T17:27:36.674 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.01405, 2024-02-09T17:27:36.674 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014277508, 2024-02-09T17:27:36.674 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:27:36.674 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "waiting for sub ops", 2024-02-09T17:27:36.674 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:27:36.674 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:27:36.674 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:27:36.674 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 200953 2024-02-09T17:27:36.675 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.675 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:27:36.675 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.675 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:27:36.675 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.647733+0000", 2024-02-09T17:27:36.675 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:36.675 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.675 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.675 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:27:36.675 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.647733+0000", 2024-02-09T17:27:36.675 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:36.675 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.675 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.675 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:27:36.675 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.647734+0000", 2024-02-09T17:27:36.675 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.189e-06 2024-02-09T17:27:36.675 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.676 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.676 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:27:36.676 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.647738+0000", 2024-02-09T17:27:36.676 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9319999999999998e-06 2024-02-09T17:27:36.676 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.676 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.676 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:27:36.676 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.647747+0000", 2024-02-09T17:27:36.676 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.3079999999999997e-06 2024-02-09T17:27:36.676 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.676 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.676 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:27:36.676 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.647754+0000", 2024-02-09T17:27:36.677 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.0360000000000001e-06 2024-02-09T17:27:36.677 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.677 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.677 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:27:36.677 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.647800+0000", 2024-02-09T17:27:36.677 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5729000000000002e-05 2024-02-09T17:27:36.677 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.677 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.677 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:27:36.677 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.647928+0000", 2024-02-09T17:27:36.677 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000128706 2024-02-09T17:27:36.677 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.677 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.677 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,1", 2024-02-09T17:27:36.678 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.648059+0000", 2024-02-09T17:27:36.678 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000130576 2024-02-09T17:27:36.678 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.678 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.678 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:27:36.678 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.652951+0000", 2024-02-09T17:27:36.678 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0048922260000000004 2024-02-09T17:27:36.678 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:36.678 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:27:36.678 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:36.678 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.678 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.678 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:200955 2.a 2:5ca38f2b:::benchmark_data_smithi179_77138_object67547:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-02-09T17:27:36.679 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:36.647769+0000", 2024-02-09T17:27:36.679 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.014013126000000001, 2024-02-09T17:27:36.679 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014316626000000001, 2024-02-09T17:27:36.679 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:27:36.679 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "waiting for sub ops", 2024-02-09T17:27:36.679 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:27:36.679 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:27:36.679 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:27:36.679 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 200955 2024-02-09T17:27:36.679 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.679 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:27:36.679 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.679 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:27:36.679 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.647769+0000", 2024-02-09T17:27:36.679 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:36.680 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.680 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.680 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:27:36.680 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.647769+0000", 2024-02-09T17:27:36.680 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:36.680 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.680 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.680 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:27:36.680 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.647770+0000", 2024-02-09T17:27:36.680 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3000000000000001e-07 2024-02-09T17:27:36.680 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.680 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.680 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:27:36.681 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.647773+0000", 2024-02-09T17:27:36.681 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0869999999999998e-06 2024-02-09T17:27:36.681 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.681 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.681 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:27:36.681 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.647781+0000", 2024-02-09T17:27:36.681 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7829999999999999e-06 2024-02-09T17:27:36.681 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.681 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.681 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:27:36.681 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.647787+0000", 2024-02-09T17:27:36.681 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7790000000000001e-06 2024-02-09T17:27:36.681 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.682 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.682 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:27:36.682 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.647857+0000", 2024-02-09T17:27:36.682 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.0353999999999998e-05 2024-02-09T17:27:36.682 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.682 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.682 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:27:36.682 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.647999+0000", 2024-02-09T17:27:36.682 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000141967 2024-02-09T17:27:36.682 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.682 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.682 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,9", 2024-02-09T17:27:36.682 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.648122+0000", 2024-02-09T17:27:36.682 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000123199 2024-02-09T17:27:36.683 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.683 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.683 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:27:36.683 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.652159+0000", 2024-02-09T17:27:36.683 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0040367570000000002 2024-02-09T17:27:36.683 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.683 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.683 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:36.683 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.656891+0000", 2024-02-09T17:27:36.683 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0047325680000000004 2024-02-09T17:27:36.683 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:36.683 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:27:36.683 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:36.684 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.684 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.684 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:200958 2.c 2:33255050:::benchmark_data_smithi179_77138_object67550:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-02-09T17:27:36.684 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:36.653621+0000", 2024-02-09T17:27:36.684 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.0081613750000000002, 2024-02-09T17:27:36.684 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0085518590000000002, 2024-02-09T17:27:36.684 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:27:36.684 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "waiting for sub ops", 2024-02-09T17:27:36.684 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:27:36.684 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:27:36.684 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:27:36.684 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 200958 2024-02-09T17:27:36.684 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.685 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:27:36.685 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.685 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:27:36.685 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.653621+0000", 2024-02-09T17:27:36.685 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:36.685 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.685 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.685 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:27:36.685 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.653621+0000", 2024-02-09T17:27:36.685 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:36.685 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.685 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.685 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:27:36.685 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.653623+0000", 2024-02-09T17:27:36.686 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5230000000000001e-06 2024-02-09T17:27:36.686 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.686 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.686 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:27:36.686 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.653628+0000", 2024-02-09T17:27:36.686 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7819999999999999e-06 2024-02-09T17:27:36.686 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.686 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.686 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:27:36.686 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.653642+0000", 2024-02-09T17:27:36.686 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3797e-05 2024-02-09T17:27:36.686 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.686 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.686 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:27:36.687 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.653651+0000", 2024-02-09T17:27:36.687 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3620000000000006e-06 2024-02-09T17:27:36.687 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.687 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.687 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:27:36.687 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.653704+0000", 2024-02-09T17:27:36.687 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2973999999999998e-05 2024-02-09T17:27:36.687 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.687 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.687 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:27:36.687 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.653780+0000", 2024-02-09T17:27:36.687 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6129000000000002e-05 2024-02-09T17:27:36.687 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.687 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.688 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,1", 2024-02-09T17:27:36.688 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.653848+0000", 2024-02-09T17:27:36.688 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8118000000000001e-05 2024-02-09T17:27:36.688 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:36.688 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:36.688 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:27:36.688 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:36.657677+0000", 2024-02-09T17:27:36.688 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0038288089999999999 2024-02-09T17:27:36.688 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:36.688 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:27:36.688 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:36.688 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:36.688 INFO:teuthology.orchestra.run.smithi179.stdout: ], 2024-02-09T17:27:36.689 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 5 2024-02-09T17:27:36.689 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:27:36.739 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:27:36.779 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:36.808 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:27:36.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:36.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:27:36.915 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:27:36.951 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:36.972 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:27:36.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:36.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:27:36.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:36 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 17422/209655 objects degraded (8.310%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:27:37.080 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:27:37.103 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:36 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 17422/209655 objects degraded (8.310%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:27:37.145 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:37.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:36 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 17422/209655 objects degraded (8.310%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:27:37.426 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:27:37.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:37.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:27:37.528 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:27:37.564 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.11 dump_blocked_ops 2024-02-09T17:27:37.570 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:37.594 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:27:37.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:37.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:27:37.698 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:27:37.734 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:37.768 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:27:37.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:37.774 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:27:37.829 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-02-09T17:27:37.829 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd test-reweight-by-utilization 110 0.05 2024-02-09T17:27:37.875 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:27:37.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:37 smithi179 ceph-mon[43975]: pgmap v489: 17 pgs: 1 active+clean+scrubbing, 4 active+undersized+degraded, 12 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 885 op/s; 16899/203538 objects degraded (8.303%) 2024-02-09T17:27:37.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:37 smithi179 ceph-mon[43975]: 2.d scrub starts 2024-02-09T17:27:37.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:37 smithi162 ceph-mon[44304]: pgmap v489: 17 pgs: 1 active+clean+scrubbing, 4 active+undersized+degraded, 12 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 885 op/s; 16899/203538 objects degraded (8.303%) 2024-02-09T17:27:37.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:37 smithi162 ceph-mon[44304]: 2.d scrub starts 2024-02-09T17:27:37.941 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:27:37.985 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:38.204 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:27:38.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:38.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:27:38.210 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:37 smithi099 ceph-mon[39420]: pgmap v489: 17 pgs: 1 active+clean+scrubbing, 4 active+undersized+degraded, 12 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 885 op/s; 16899/203538 objects degraded (8.303%) 2024-02-09T17:27:38.210 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:37 smithi099 ceph-mon[39420]: 2.d scrub starts 2024-02-09T17:27:38.298 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:27:38.310 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:27:38.343 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:38.346 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:27:38.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:38 smithi099 ceph-mon[39420]: 2.d scrub ok 2024-02-09T17:27:38.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:38 smithi099 ceph-mon[39420]: 2.e scrub starts 2024-02-09T17:27:38.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:38 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:27:38.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:38 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:27:39.002 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:27:39.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:39.012 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:27:39.113 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:27:39.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:38 smithi179 ceph-mon[43975]: 2.d scrub ok 2024-02-09T17:27:39.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:38 smithi179 ceph-mon[43975]: 2.e scrub starts 2024-02-09T17:27:39.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:38 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:27:39.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:38 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:27:39.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:38 smithi162 ceph-mon[44304]: 2.d scrub ok 2024-02-09T17:27:39.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:38 smithi162 ceph-mon[44304]: 2.e scrub starts 2024-02-09T17:27:39.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:38 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:27:39.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:38 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:27:39.238 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:39.821 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is unset 2024-02-09T17:27:40.155 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:27:40.155 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-02-09T17:27:40.155 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-02-09T17:27:40.155 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-02-09T17:27:40.156 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:27:40.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:39 smithi179 ceph-mon[43975]: 2.4 scrub starts 2024-02-09T17:27:40.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:39 smithi179 ceph-mon[43975]: pgmap v490: 17 pgs: 1 active+clean+scrubbing, 4 active+undersized+degraded, 12 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 836 op/s; 16899/201924 objects degraded (8.369%) 2024-02-09T17:27:40.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:39 smithi179 ceph-mon[43975]: osdmap e164: 12 total, 11 up, 12 in 2024-02-09T17:27:40.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:39 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1703311021' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:27:40.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:39 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:27:40.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:39 smithi162 ceph-mon[44304]: 2.4 scrub starts 2024-02-09T17:27:40.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:39 smithi162 ceph-mon[44304]: pgmap v490: 17 pgs: 1 active+clean+scrubbing, 4 active+undersized+degraded, 12 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 836 op/s; 16899/201924 objects degraded (8.369%) 2024-02-09T17:27:40.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:39 smithi162 ceph-mon[44304]: osdmap e164: 12 total, 11 up, 12 in 2024-02-09T17:27:40.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:39 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1703311021' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:27:40.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:39 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:27:40.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:39 smithi099 ceph-mon[39420]: 2.4 scrub starts 2024-02-09T17:27:40.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:39 smithi099 ceph-mon[39420]: pgmap v490: 17 pgs: 1 active+clean+scrubbing, 4 active+undersized+degraded, 12 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 836 op/s; 16899/201924 objects degraded (8.369%) 2024-02-09T17:27:40.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:39 smithi099 ceph-mon[39420]: osdmap e164: 12 total, 11 up, 12 in 2024-02-09T17:27:40.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:39 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1703311021' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:27:40.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:39 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:27:40.320 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:27:40.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:40.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:27:40.462 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:27:40.492 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:40.519 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:27:40.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:40.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:27:40.623 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:27:40.657 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:40.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:40 smithi179 ceph-mon[43975]: 2.4 scrub ok 2024-02-09T17:27:40.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:40 smithi179 ceph-mon[43975]: 2.e scrub ok 2024-02-09T17:27:40.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:40 smithi179 ceph-mon[43975]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:27:40.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:40 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:27:40.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:40 smithi179 ceph-mon[43975]: osdmap e165: 12 total, 11 up, 12 in 2024-02-09T17:27:40.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:40 smithi179 ceph-mon[43975]: pgmap v493: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 817 op/s; 16081/194613 objects degraded (8.263%) 2024-02-09T17:27:40.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:40 smithi179 ceph-mon[43975]: 1.0 scrub starts 2024-02-09T17:27:40.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:40 smithi179 ceph-mon[43975]: 1.0 scrub ok 2024-02-09T17:27:41.147 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:40 smithi099 ceph-mon[39420]: 2.4 scrub ok 2024-02-09T17:27:41.147 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:40 smithi099 ceph-mon[39420]: 2.e scrub ok 2024-02-09T17:27:41.147 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:40 smithi099 ceph-mon[39420]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:27:41.147 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:40 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:27:41.147 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:40 smithi099 ceph-mon[39420]: osdmap e165: 12 total, 11 up, 12 in 2024-02-09T17:27:41.147 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:40 smithi099 ceph-mon[39420]: pgmap v493: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 817 op/s; 16081/194613 objects degraded (8.263%) 2024-02-09T17:27:41.147 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:40 smithi099 ceph-mon[39420]: 1.0 scrub starts 2024-02-09T17:27:41.147 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:40 smithi099 ceph-mon[39420]: 1.0 scrub ok 2024-02-09T17:27:41.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:40 smithi162 ceph-mon[44304]: 2.4 scrub ok 2024-02-09T17:27:41.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:40 smithi162 ceph-mon[44304]: 2.e scrub ok 2024-02-09T17:27:41.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:40 smithi162 ceph-mon[44304]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:27:41.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:40 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:27:41.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:40 smithi162 ceph-mon[44304]: osdmap e165: 12 total, 11 up, 12 in 2024-02-09T17:27:41.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:40 smithi162 ceph-mon[44304]: pgmap v493: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 817 op/s; 16081/194613 objects degraded (8.263%) 2024-02-09T17:27:41.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:40 smithi162 ceph-mon[44304]: 1.0 scrub starts 2024-02-09T17:27:41.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:40 smithi162 ceph-mon[44304]: 1.0 scrub ok 2024-02-09T17:27:41.709 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:27:41.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:41.713 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:27:41.731 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.11 dump_historic_ops 2024-02-09T17:27:41.813 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:27:41.886 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:41.915 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:27:41.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:41.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:27:42.019 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:27:42.044 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:42.059 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:27:42.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:41 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 16081/194613 objects degraded (8.263%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:27:42.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:41 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 16081/194613 objects degraded (8.263%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:27:42.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:41 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 16081/194613 objects degraded (8.263%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:27:42.520 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:27:42.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:42.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:27:42.633 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:27:42.666 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:42.692 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:27:42.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:42.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:27:42.797 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:27:42.834 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:42.839 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:42 smithi099 ceph-mon[39420]: pgmap v494: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 3.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 856 op/s; 15814/189075 objects degraded (8.364%) 2024-02-09T17:27:42.865 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:27:42.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:42.870 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:27:42.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:42 smithi162 ceph-mon[44304]: pgmap v494: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 3.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 856 op/s; 15814/189075 objects degraded (8.364%) 2024-02-09T17:27:42.936 INFO:teuthology.orchestra.run.smithi099.stdout:moved 1 / 51 (1.96078%) 2024-02-09T17:27:42.936 INFO:teuthology.orchestra.run.smithi099.stdout:avg 4.63636 2024-02-09T17:27:42.936 INFO:teuthology.orchestra.run.smithi099.stdout:stddev 1.40248 -> 1.46587 (expected baseline 2.05302) 2024-02-09T17:27:42.936 INFO:teuthology.orchestra.run.smithi099.stdout:min osd.5 with 2 -> 2 pgs (0.431373 -> 0.431373 * mean) 2024-02-09T17:27:42.936 INFO:teuthology.orchestra.run.smithi099.stdout:max osd.3 with 6 -> 6 pgs (1.29412 -> 1.29412 * mean) 2024-02-09T17:27:42.936 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:27:42.936 INFO:teuthology.orchestra.run.smithi099.stdout:oload 110 2024-02-09T17:27:42.937 INFO:teuthology.orchestra.run.smithi099.stdout:max_change 0.05 2024-02-09T17:27:42.937 INFO:teuthology.orchestra.run.smithi099.stdout:max_change_osds 4 2024-02-09T17:27:42.937 INFO:teuthology.orchestra.run.smithi099.stdout:average_utilization 0.0124 2024-02-09T17:27:42.937 INFO:teuthology.orchestra.run.smithi099.stdout:overload_utilization 0.0137 2024-02-09T17:27:42.937 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8 weight 1.0000 -> 0.9500 2024-02-09T17:27:42.937 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3 weight 1.0000 -> 0.9500 2024-02-09T17:27:42.937 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9 weight 1.0000 -> 0.9500 2024-02-09T17:27:42.937 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0 weight 1.0000 -> 0.9500 2024-02-09T17:27:42.951 INFO:teuthology.orchestra.run.smithi099.stderr:no change 2024-02-09T17:27:42.971 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:27:43.052 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:43.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:42 smithi179 ceph-mon[43975]: pgmap v494: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 3.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 856 op/s; 15814/189075 objects degraded (8.364%) 2024-02-09T17:27:43.180 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set noscrub 2024-02-09T17:27:43.227 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:27:42 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:27:42.928+0000 7f5169e95700 -1 mgr.server reply reply (4) Interrupted system call no change 2024-02-09T17:27:43.228 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:27:43.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:43.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:27:43.342 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:27:43.370 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:43.395 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:27:43.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:43.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:27:43.499 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:27:43.575 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:43.841 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:43 smithi179 ceph-mon[43975]: from='client.24685 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:27:43.844 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:43 smithi099 ceph-mon[39420]: from='client.24685 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:27:43.954 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-02-09T17:27:43.954 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd test-reweight-by-utilization 110 0.05 2024-02-09T17:27:43.976 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:27:43.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:43.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:27:44.081 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:27:44.114 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:43 smithi162 ceph-mon[44304]: from='client.24685 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:27:44.116 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:44.138 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:27:44.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:44.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:27:44.226 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:27:44.227 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-02-09T17:27:44.227 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-02-09T17:27:44.227 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-02-09T17:27:44.227 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.227 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:133709 2.c 2:36d46332:::benchmark_data_smithi179_77138_object301:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-02-09T17:27:44.227 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:26:19.805806+0000", 2024-02-09T17:27:44.227 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 84.415821084000001, 2024-02-09T17:27:44.227 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.109725028, 2024-02-09T17:27:44.227 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:27:44.227 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:44.227 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:27:44.227 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:27:44.227 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:27:44.228 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 133709 2024-02-09T17:27:44.228 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.228 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:27:44.228 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.228 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:27:44.228 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:19.805806+0000", 2024-02-09T17:27:44.228 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:44.228 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.228 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.228 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:27:44.228 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:19.805806+0000", 2024-02-09T17:27:44.228 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:44.228 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.229 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.229 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:27:44.229 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:19.805807+0000", 2024-02-09T17:27:44.229 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7570000000000001e-06 2024-02-09T17:27:44.229 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.229 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.229 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:27:44.229 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:19.805814+0000", 2024-02-09T17:27:44.229 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5740000000000001e-06 2024-02-09T17:27:44.229 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.229 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.229 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:27:44.229 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:19.805827+0000", 2024-02-09T17:27:44.229 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2795e-05 2024-02-09T17:27:44.229 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.230 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.230 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:27:44.230 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:19.805835+0000", 2024-02-09T17:27:44.230 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.0940000000000008e-06 2024-02-09T17:27:44.230 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.230 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.230 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:27:44.230 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:19.812536+0000", 2024-02-09T17:27:44.230 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0067011859999999996 2024-02-09T17:27:44.230 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.230 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.230 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:27:44.231 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:19.812647+0000", 2024-02-09T17:27:44.231 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000110392 2024-02-09T17:27:44.231 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.231 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.231 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,1", 2024-02-09T17:27:44.231 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:19.812749+0000", 2024-02-09T17:27:44.231 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000102409 2024-02-09T17:27:44.231 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.231 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.231 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:27:44.231 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:19.824100+0000", 2024-02-09T17:27:44.231 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011351228999999999 2024-02-09T17:27:44.231 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.231 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.232 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:44.232 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:19.824141+0000", 2024-02-09T17:27:44.232 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0970999999999998e-05 2024-02-09T17:27:44.232 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.232 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.232 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:44.232 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:19.915446+0000", 2024-02-09T17:27:44.232 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.091304838999999999 2024-02-09T17:27:44.232 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.232 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.232 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:27:44.232 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:19.915483+0000", 2024-02-09T17:27:44.232 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6977999999999997e-05 2024-02-09T17:27:44.233 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.233 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.233 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:27:44.233 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:19.915531+0000", 2024-02-09T17:27:44.233 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7803999999999998e-05 2024-02-09T17:27:44.233 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:44.233 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:27:44.233 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:44.233 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.233 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.233 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:134313 2.c 2:300b6bf1:::benchmark_data_smithi179_77138_object905:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-02-09T17:27:44.233 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:26:20.499896+0000", 2024-02-09T17:27:44.233 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 83.721730531000006, 2024-02-09T17:27:44.233 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.102593965, 2024-02-09T17:27:44.234 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:27:44.234 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:44.234 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:27:44.234 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:27:44.234 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:27:44.234 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 134313 2024-02-09T17:27:44.234 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.234 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:27:44.234 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.234 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:27:44.234 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:20.499896+0000", 2024-02-09T17:27:44.234 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:44.234 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.234 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.235 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:27:44.235 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:20.499896+0000", 2024-02-09T17:27:44.235 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:44.235 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.235 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.235 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:27:44.235 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:20.499897+0000", 2024-02-09T17:27:44.235 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1179999999999999e-06 2024-02-09T17:27:44.235 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.235 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.235 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:27:44.235 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:20.499901+0000", 2024-02-09T17:27:44.235 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.092e-06 2024-02-09T17:27:44.235 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.236 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.236 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:27:44.236 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:20.499912+0000", 2024-02-09T17:27:44.236 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.082e-05 2024-02-09T17:27:44.236 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.236 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.236 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:27:44.236 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:20.499918+0000", 2024-02-09T17:27:44.236 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8490000000000002e-06 2024-02-09T17:27:44.236 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.236 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.236 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:27:44.237 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:20.499976+0000", 2024-02-09T17:27:44.237 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7577000000000001e-05 2024-02-09T17:27:44.237 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.237 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.237 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:27:44.237 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:20.500200+0000", 2024-02-09T17:27:44.237 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00022447499999999999 2024-02-09T17:27:44.237 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.237 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.237 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,1", 2024-02-09T17:27:44.237 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:20.500281+0000", 2024-02-09T17:27:44.237 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1003999999999999e-05 2024-02-09T17:27:44.237 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.238 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.238 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:27:44.238 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:20.503923+0000", 2024-02-09T17:27:44.238 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0036414419999999999 2024-02-09T17:27:44.238 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.238 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.238 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:44.238 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:20.504118+0000", 2024-02-09T17:27:44.238 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000194969 2024-02-09T17:27:44.238 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.238 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.238 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:44.238 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:20.602370+0000", 2024-02-09T17:27:44.239 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.098252330999999998 2024-02-09T17:27:44.239 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.239 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.239 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:27:44.239 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:20.602432+0000", 2024-02-09T17:27:44.239 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1749999999999997e-05 2024-02-09T17:27:44.239 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.239 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.239 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:27:44.239 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:20.602490+0000", 2024-02-09T17:27:44.239 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8538000000000001e-05 2024-02-09T17:27:44.239 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:44.239 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:27:44.239 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:44.240 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.240 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.240 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:136232 2.c 2:3097e830:::benchmark_data_smithi179_77138_object2824:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-02-09T17:27:44.240 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:26:22.609931+0000", 2024-02-09T17:27:44.240 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 81.611695564000001, 2024-02-09T17:27:44.240 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.10327204299999999, 2024-02-09T17:27:44.240 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:27:44.240 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:44.240 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:27:44.240 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:27:44.240 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:27:44.240 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 136232 2024-02-09T17:27:44.240 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.240 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:27:44.241 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.241 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:27:44.241 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:22.609931+0000", 2024-02-09T17:27:44.241 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:44.241 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.241 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.241 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:27:44.241 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:22.609931+0000", 2024-02-09T17:27:44.241 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:44.241 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.241 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.241 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:27:44.241 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:22.609933+0000", 2024-02-09T17:27:44.241 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4839999999999999e-06 2024-02-09T17:27:44.242 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.242 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.242 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:27:44.242 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:22.609939+0000", 2024-02-09T17:27:44.242 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9889999999999997e-06 2024-02-09T17:27:44.242 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.242 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.242 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:27:44.242 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:22.609953+0000", 2024-02-09T17:27:44.242 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4357e-05 2024-02-09T17:27:44.242 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.242 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.242 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:27:44.242 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:22.609961+0000", 2024-02-09T17:27:44.243 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1319999999999994e-06 2024-02-09T17:27:44.243 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.243 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.243 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:27:44.243 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:22.610023+0000", 2024-02-09T17:27:44.243 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1338000000000001e-05 2024-02-09T17:27:44.243 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.243 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.243 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:27:44.243 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:22.610170+0000", 2024-02-09T17:27:44.243 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014755700000000001 2024-02-09T17:27:44.243 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.243 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.243 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,1", 2024-02-09T17:27:44.243 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:22.610283+0000", 2024-02-09T17:27:44.243 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000112878 2024-02-09T17:27:44.244 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.244 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.244 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:27:44.244 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:22.614621+0000", 2024-02-09T17:27:44.244 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0043382769999999998 2024-02-09T17:27:44.244 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.244 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.244 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:44.244 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:22.616637+0000", 2024-02-09T17:27:44.244 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0020161810000000001 2024-02-09T17:27:44.244 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.244 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.244 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:44.245 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:22.713103+0000", 2024-02-09T17:27:44.245 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.096465974999999995 2024-02-09T17:27:44.245 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.245 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.245 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:27:44.245 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:22.713159+0000", 2024-02-09T17:27:44.245 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5387999999999999e-05 2024-02-09T17:27:44.245 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.245 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.245 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:27:44.245 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:22.713203+0000", 2024-02-09T17:27:44.245 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4487000000000003e-05 2024-02-09T17:27:44.245 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:44.245 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:27:44.246 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:44.246 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.246 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.246 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:136243 2.c 2:3be08b8b:::benchmark_data_smithi179_77138_object2835:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-02-09T17:27:44.246 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:26:22.618859+0000", 2024-02-09T17:27:44.246 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 81.60276795, 2024-02-09T17:27:44.246 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.108967218, 2024-02-09T17:27:44.246 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:27:44.246 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:44.246 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:27:44.246 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:27:44.246 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:27:44.246 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 136243 2024-02-09T17:27:44.247 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.247 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:27:44.247 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.247 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:27:44.247 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:22.618859+0000", 2024-02-09T17:27:44.247 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:44.247 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.247 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.247 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:27:44.247 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:22.618859+0000", 2024-02-09T17:27:44.247 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:44.247 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.248 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.248 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:27:44.248 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:22.618860+0000", 2024-02-09T17:27:44.248 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.012e-06 2024-02-09T17:27:44.248 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.248 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.248 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:27:44.248 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:22.618864+0000", 2024-02-09T17:27:44.248 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1189999999999999e-06 2024-02-09T17:27:44.248 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.248 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.248 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:27:44.248 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:22.618874+0000", 2024-02-09T17:27:44.249 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0071000000000001e-05 2024-02-09T17:27:44.249 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.249 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.249 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:27:44.249 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:22.618880+0000", 2024-02-09T17:27:44.249 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5690000000000004e-06 2024-02-09T17:27:44.249 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.249 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.249 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:27:44.249 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:22.619336+0000", 2024-02-09T17:27:44.249 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00045622700000000002 2024-02-09T17:27:44.249 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.249 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.250 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:27:44.250 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:22.619422+0000", 2024-02-09T17:27:44.250 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5951000000000004e-05 2024-02-09T17:27:44.250 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.250 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.250 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,1", 2024-02-09T17:27:44.250 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:22.619494+0000", 2024-02-09T17:27:44.250 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1946000000000005e-05 2024-02-09T17:27:44.250 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.250 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.250 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:27:44.250 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:22.623895+0000", 2024-02-09T17:27:44.250 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0044010389999999998 2024-02-09T17:27:44.250 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.250 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.251 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:44.251 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:22.628717+0000", 2024-02-09T17:27:44.251 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0048221119999999999 2024-02-09T17:27:44.251 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.251 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.251 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:44.251 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:22.727714+0000", 2024-02-09T17:27:44.251 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.098997598000000006 2024-02-09T17:27:44.251 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.251 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.251 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:27:44.251 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:22.727781+0000", 2024-02-09T17:27:44.251 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6029000000000001e-05 2024-02-09T17:27:44.251 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.251 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.251 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:27:44.252 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:22.727826+0000", 2024-02-09T17:27:44.252 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5544999999999998e-05 2024-02-09T17:27:44.252 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:44.252 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:27:44.252 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:44.252 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.252 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.252 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:170253 2.c 2:33b79755:::benchmark_data_smithi179_77138_object36845:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-02-09T17:27:44.252 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:00.372563+0000", 2024-02-09T17:27:44.252 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 43.849063575999999, 2024-02-09T17:27:44.252 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.11809602900000001, 2024-02-09T17:27:44.252 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:27:44.252 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:44.253 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:27:44.253 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:27:44.253 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:27:44.253 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 170253 2024-02-09T17:27:44.253 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.253 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:27:44.253 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.253 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:27:44.253 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:00.372563+0000", 2024-02-09T17:27:44.253 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:44.253 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.253 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.253 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:27:44.253 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:00.372563+0000", 2024-02-09T17:27:44.253 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:44.253 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.254 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.254 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:27:44.254 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:00.372564+0000", 2024-02-09T17:27:44.254 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0720000000000001e-06 2024-02-09T17:27:44.254 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.254 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.254 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:27:44.254 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:00.372569+0000", 2024-02-09T17:27:44.254 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2889999999999999e-06 2024-02-09T17:27:44.254 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.254 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.254 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:27:44.254 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:00.372580+0000", 2024-02-09T17:27:44.254 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1225e-05 2024-02-09T17:27:44.254 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.255 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.255 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:27:44.255 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:00.372588+0000", 2024-02-09T17:27:44.255 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7100000000000007e-06 2024-02-09T17:27:44.255 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.255 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.255 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:27:44.255 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:00.372926+0000", 2024-02-09T17:27:44.255 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00033813799999999998 2024-02-09T17:27:44.255 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.255 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.255 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:27:44.255 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:00.373013+0000", 2024-02-09T17:27:44.255 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7814999999999999e-05 2024-02-09T17:27:44.256 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.256 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.256 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,1", 2024-02-09T17:27:44.256 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:00.373100+0000", 2024-02-09T17:27:44.256 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.6187999999999995e-05 2024-02-09T17:27:44.256 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.256 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.256 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:44.256 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:00.376947+0000", 2024-02-09T17:27:44.256 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.003847329 2024-02-09T17:27:44.256 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.256 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.256 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:27:44.257 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:00.377062+0000", 2024-02-09T17:27:44.257 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011547799999999999 2024-02-09T17:27:44.257 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.257 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.257 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:44.257 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:00.490604+0000", 2024-02-09T17:27:44.257 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.11354122899999999 2024-02-09T17:27:44.257 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.257 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.257 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:27:44.257 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:00.490635+0000", 2024-02-09T17:27:44.257 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1216000000000001e-05 2024-02-09T17:27:44.257 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.257 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.258 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:27:44.258 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:00.490659+0000", 2024-02-09T17:27:44.258 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4340000000000001e-05 2024-02-09T17:27:44.258 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:44.258 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:27:44.258 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:44.258 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.258 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.258 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:170271 2.c 2:3bbc20c7:::benchmark_data_smithi179_77138_object36863:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-02-09T17:27:44.258 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:00.406535+0000", 2024-02-09T17:27:44.258 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 43.815091324000001, 2024-02-09T17:27:44.258 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.107413996, 2024-02-09T17:27:44.258 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:27:44.258 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:44.258 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:27:44.259 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:27:44.259 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:27:44.259 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 170271 2024-02-09T17:27:44.259 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.259 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:27:44.259 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.259 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:27:44.259 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:00.406535+0000", 2024-02-09T17:27:44.259 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:44.259 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.259 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.259 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:27:44.259 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:00.406535+0000", 2024-02-09T17:27:44.259 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:44.260 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.260 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.260 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:27:44.260 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:00.406536+0000", 2024-02-09T17:27:44.260 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3200000000000004e-07 2024-02-09T17:27:44.260 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.260 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.260 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:27:44.260 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:00.406541+0000", 2024-02-09T17:27:44.260 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6840000000000004e-06 2024-02-09T17:27:44.260 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.260 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.260 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:27:44.261 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:00.406568+0000", 2024-02-09T17:27:44.261 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7373999999999999e-05 2024-02-09T17:27:44.261 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.261 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.261 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:27:44.261 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:00.406579+0000", 2024-02-09T17:27:44.261 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0152e-05 2024-02-09T17:27:44.261 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.261 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.261 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:27:44.261 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:00.412206+0000", 2024-02-09T17:27:44.261 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0056277230000000003 2024-02-09T17:27:44.261 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.261 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.262 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:27:44.262 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:00.412352+0000", 2024-02-09T17:27:44.262 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014582199999999999 2024-02-09T17:27:44.262 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.262 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.262 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,1", 2024-02-09T17:27:44.262 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:00.412463+0000", 2024-02-09T17:27:44.262 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000111203 2024-02-09T17:27:44.262 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.262 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.262 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:27:44.262 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:00.420584+0000", 2024-02-09T17:27:44.262 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0081204720000000001 2024-02-09T17:27:44.262 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.263 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.263 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:44.263 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:00.431453+0000", 2024-02-09T17:27:44.263 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010868948999999999 2024-02-09T17:27:44.263 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.263 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.263 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:44.263 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:00.513875+0000", 2024-02-09T17:27:44.263 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.082422244000000006 2024-02-09T17:27:44.263 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.263 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.263 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:27:44.263 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:00.513919+0000", 2024-02-09T17:27:44.263 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3591999999999999e-05 2024-02-09T17:27:44.263 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.264 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.264 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:27:44.264 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:00.513949+0000", 2024-02-09T17:27:44.264 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0948999999999998e-05 2024-02-09T17:27:44.264 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:44.264 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:27:44.264 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:44.264 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.264 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.264 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:173669 2.a 2:5049bf51:::benchmark_data_smithi179_77138_object40261:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-02-09T17:27:44.264 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:04.251927+0000", 2024-02-09T17:27:44.264 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 39.969699902999999, 2024-02-09T17:27:44.264 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.21543948900000001, 2024-02-09T17:27:44.264 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:27:44.265 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:44.265 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:27:44.265 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:27:44.265 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:27:44.265 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 173669 2024-02-09T17:27:44.265 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.265 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:27:44.265 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.265 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:27:44.265 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.251927+0000", 2024-02-09T17:27:44.265 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:44.265 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.265 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.265 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:27:44.266 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.251927+0000", 2024-02-09T17:27:44.266 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:44.266 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.266 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.266 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:27:44.266 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.251929+0000", 2024-02-09T17:27:44.266 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7579999999999999e-06 2024-02-09T17:27:44.266 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.266 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.266 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:27:44.266 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.251935+0000", 2024-02-09T17:27:44.266 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9329999999999998e-06 2024-02-09T17:27:44.266 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.266 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.267 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:27:44.267 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.251949+0000", 2024-02-09T17:27:44.267 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4268e-05 2024-02-09T17:27:44.267 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.267 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.267 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:27:44.267 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.251957+0000", 2024-02-09T17:27:44.267 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5290000000000008e-06 2024-02-09T17:27:44.267 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.267 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.267 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:27:44.267 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.252013+0000", 2024-02-09T17:27:44.267 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5619999999999999e-05 2024-02-09T17:27:44.267 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.267 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.268 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:27:44.268 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.252115+0000", 2024-02-09T17:27:44.268 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000102315 2024-02-09T17:27:44.268 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.268 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.268 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,9", 2024-02-09T17:27:44.268 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.252237+0000", 2024-02-09T17:27:44.268 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000121951 2024-02-09T17:27:44.268 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.268 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.268 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:27:44.268 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.256532+0000", 2024-02-09T17:27:44.268 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0042949540000000001 2024-02-09T17:27:44.268 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.269 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.269 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:44.269 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.259361+0000", 2024-02-09T17:27:44.269 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002829163 2024-02-09T17:27:44.269 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.269 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.269 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:44.269 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.467273+0000", 2024-02-09T17:27:44.269 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.207911866 2024-02-09T17:27:44.269 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.269 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.269 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:27:44.269 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.467324+0000", 2024-02-09T17:27:44.269 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0763000000000002e-05 2024-02-09T17:27:44.269 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.270 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.270 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:27:44.270 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.467366+0000", 2024-02-09T17:27:44.270 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2369e-05 2024-02-09T17:27:44.270 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:44.270 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:27:44.270 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:44.270 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.270 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.270 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:173689 2.a 2:55e666c2:::benchmark_data_smithi179_77138_object40281:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-02-09T17:27:44.270 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:04.273390+0000", 2024-02-09T17:27:44.270 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 39.948236592000001, 2024-02-09T17:27:44.270 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.201714904, 2024-02-09T17:27:44.270 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:27:44.271 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:44.271 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:27:44.271 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:27:44.271 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:27:44.271 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 173689 2024-02-09T17:27:44.271 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.271 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:27:44.271 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.271 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:27:44.271 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.273390+0000", 2024-02-09T17:27:44.271 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:44.271 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.271 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.271 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:27:44.272 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.273390+0000", 2024-02-09T17:27:44.272 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:44.272 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.272 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.272 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:27:44.272 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.273392+0000", 2024-02-09T17:27:44.272 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3340000000000001e-06 2024-02-09T17:27:44.272 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.272 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.272 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:27:44.272 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.273396+0000", 2024-02-09T17:27:44.272 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2640000000000002e-06 2024-02-09T17:27:44.272 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.273 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.273 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:27:44.273 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.273406+0000", 2024-02-09T17:27:44.273 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0135999999999999e-05 2024-02-09T17:27:44.273 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.273 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.273 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:27:44.273 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.273412+0000", 2024-02-09T17:27:44.273 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8289999999999996e-06 2024-02-09T17:27:44.273 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.273 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.273 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:27:44.273 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.273457+0000", 2024-02-09T17:27:44.273 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5225000000000002e-05 2024-02-09T17:27:44.273 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.273 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.274 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:27:44.274 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.273543+0000", 2024-02-09T17:27:44.274 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5606999999999995e-05 2024-02-09T17:27:44.274 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.274 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.274 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,9", 2024-02-09T17:27:44.274 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.273618+0000", 2024-02-09T17:27:44.274 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5615e-05 2024-02-09T17:27:44.274 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.274 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.274 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:27:44.274 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.277650+0000", 2024-02-09T17:27:44.274 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0040314699999999997 2024-02-09T17:27:44.274 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.274 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.275 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:44.275 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.288200+0000", 2024-02-09T17:27:44.275 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010550744000000001 2024-02-09T17:27:44.275 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.275 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.275 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:44.275 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.475013+0000", 2024-02-09T17:27:44.275 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.186812582 2024-02-09T17:27:44.275 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.275 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.275 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:27:44.275 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.475063+0000", 2024-02-09T17:27:44.275 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0197000000000001e-05 2024-02-09T17:27:44.275 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.275 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.276 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:27:44.276 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.475105+0000", 2024-02-09T17:27:44.276 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1900999999999997e-05 2024-02-09T17:27:44.276 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:44.276 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:27:44.276 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:44.276 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.276 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.276 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:173697 2.a 2:55ecfd26:::benchmark_data_smithi179_77138_object40289:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-02-09T17:27:44.276 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:04.282923+0000", 2024-02-09T17:27:44.276 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 39.938703381000003, 2024-02-09T17:27:44.276 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.19232663799999999, 2024-02-09T17:27:44.276 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:27:44.276 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:44.277 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:27:44.277 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:27:44.277 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:27:44.277 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 173697 2024-02-09T17:27:44.277 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.277 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:27:44.277 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.277 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:27:44.277 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.282923+0000", 2024-02-09T17:27:44.277 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:44.277 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.277 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.277 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:27:44.277 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.282923+0000", 2024-02-09T17:27:44.278 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:44.278 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.278 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.278 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:27:44.278 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.282925+0000", 2024-02-09T17:27:44.278 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4190000000000001e-06 2024-02-09T17:27:44.278 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.278 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.278 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:27:44.278 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.282930+0000", 2024-02-09T17:27:44.278 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9880000000000004e-06 2024-02-09T17:27:44.278 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.278 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.278 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:27:44.278 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.282952+0000", 2024-02-09T17:27:44.279 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1787999999999999e-05 2024-02-09T17:27:44.279 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.279 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.279 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:27:44.279 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.282960+0000", 2024-02-09T17:27:44.279 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7280000000000001e-06 2024-02-09T17:27:44.279 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.279 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.279 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:27:44.279 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.283139+0000", 2024-02-09T17:27:44.279 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00017882299999999999 2024-02-09T17:27:44.279 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.279 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.279 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:27:44.280 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.283270+0000", 2024-02-09T17:27:44.280 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013041300000000001 2024-02-09T17:27:44.280 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.280 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.280 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,9", 2024-02-09T17:27:44.280 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.283394+0000", 2024-02-09T17:27:44.280 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012484099999999999 2024-02-09T17:27:44.280 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.280 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.280 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:44.280 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.288244+0000", 2024-02-09T17:27:44.280 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0048494940000000002 2024-02-09T17:27:44.280 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.280 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.281 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:27:44.281 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.288278+0000", 2024-02-09T17:27:44.281 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.3902000000000002e-05 2024-02-09T17:27:44.281 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.281 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.281 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:44.281 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.475158+0000", 2024-02-09T17:27:44.281 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.18688034100000001 2024-02-09T17:27:44.281 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.281 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.281 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:27:44.281 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.475212+0000", 2024-02-09T17:27:44.281 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3863000000000003e-05 2024-02-09T17:27:44.282 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.282 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.282 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:27:44.282 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.475250+0000", 2024-02-09T17:27:44.282 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8037999999999997e-05 2024-02-09T17:27:44.282 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:44.282 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:27:44.282 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:44.282 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.282 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.282 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:173701 2.a 2:54bdf07e:::benchmark_data_smithi179_77138_object40293:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-02-09T17:27:44.282 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:04.288307+0000", 2024-02-09T17:27:44.282 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 39.933320168000002, 2024-02-09T17:27:44.283 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.19479152299999999, 2024-02-09T17:27:44.283 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:27:44.283 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:44.283 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:27:44.283 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:27:44.283 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:27:44.283 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 173701 2024-02-09T17:27:44.283 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.283 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:27:44.283 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.283 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:27:44.283 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.288307+0000", 2024-02-09T17:27:44.283 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:44.283 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.284 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.284 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:27:44.284 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.288307+0000", 2024-02-09T17:27:44.284 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:44.284 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.284 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.284 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:27:44.284 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.288308+0000", 2024-02-09T17:27:44.284 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.376e-06 2024-02-09T17:27:44.284 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.284 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.284 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:27:44.284 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.288314+0000", 2024-02-09T17:27:44.284 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8309999999999997e-06 2024-02-09T17:27:44.285 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.285 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.285 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:27:44.285 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.288330+0000", 2024-02-09T17:27:44.285 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5869999999999999e-05 2024-02-09T17:27:44.285 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.285 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.285 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:27:44.285 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.288338+0000", 2024-02-09T17:27:44.285 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7390000000000005e-06 2024-02-09T17:27:44.285 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.285 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.285 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:27:44.286 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.288385+0000", 2024-02-09T17:27:44.286 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6099e-05 2024-02-09T17:27:44.286 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.286 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.286 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:27:44.286 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.288505+0000", 2024-02-09T17:27:44.286 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000120623 2024-02-09T17:27:44.286 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.286 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.286 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,9", 2024-02-09T17:27:44.286 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.288636+0000", 2024-02-09T17:27:44.286 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000131013 2024-02-09T17:27:44.286 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.286 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.287 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:44.287 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.293939+0000", 2024-02-09T17:27:44.287 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0053033109999999998 2024-02-09T17:27:44.287 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.287 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.287 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:27:44.287 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.297980+0000", 2024-02-09T17:27:44.287 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00404004 2024-02-09T17:27:44.287 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.287 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.287 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:44.288 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.482999+0000", 2024-02-09T17:27:44.288 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.18501925899999999 2024-02-09T17:27:44.288 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.288 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.288 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:27:44.288 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.483057+0000", 2024-02-09T17:27:44.288 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8619000000000002e-05 2024-02-09T17:27:44.288 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.288 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.288 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:27:44.288 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.483098+0000", 2024-02-09T17:27:44.288 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0742999999999999e-05 2024-02-09T17:27:44.289 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:44.289 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:27:44.289 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:44.289 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.289 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.289 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:173707 2.a 2:5f5eaec0:::benchmark_data_smithi179_77138_object40299:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-02-09T17:27:44.289 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:04.293626+0000", 2024-02-09T17:27:44.289 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 39.928000580000003, 2024-02-09T17:27:44.289 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.189622768, 2024-02-09T17:27:44.289 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:27:44.289 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:44.289 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:27:44.290 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:27:44.290 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:27:44.290 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 173707 2024-02-09T17:27:44.290 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.290 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:27:44.290 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.290 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:27:44.290 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.293626+0000", 2024-02-09T17:27:44.290 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:44.290 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.290 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.290 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:27:44.290 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.293626+0000", 2024-02-09T17:27:44.290 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:44.291 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.291 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.291 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:27:44.291 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.293627+0000", 2024-02-09T17:27:44.291 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.19e-06 2024-02-09T17:27:44.291 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.291 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.291 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:27:44.291 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.293633+0000", 2024-02-09T17:27:44.291 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1549999999999998e-06 2024-02-09T17:27:44.291 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.291 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.291 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:27:44.291 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.293643+0000", 2024-02-09T17:27:44.292 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0394000000000001e-05 2024-02-09T17:27:44.292 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.292 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.292 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:27:44.292 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.293649+0000", 2024-02-09T17:27:44.292 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4119999999999998e-06 2024-02-09T17:27:44.292 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.292 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.292 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:27:44.292 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.293974+0000", 2024-02-09T17:27:44.292 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00032459299999999997 2024-02-09T17:27:44.292 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.292 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.293 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:27:44.293 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.294093+0000", 2024-02-09T17:27:44.293 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000119394 2024-02-09T17:27:44.293 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.293 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.293 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,9", 2024-02-09T17:27:44.293 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.294245+0000", 2024-02-09T17:27:44.293 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000151333 2024-02-09T17:27:44.293 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.293 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.293 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:27:44.293 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.305358+0000", 2024-02-09T17:27:44.293 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011113125999999999 2024-02-09T17:27:44.293 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.293 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.294 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:44.294 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.307269+0000", 2024-02-09T17:27:44.294 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0019107779999999999 2024-02-09T17:27:44.294 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.294 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.294 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:44.294 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.483159+0000", 2024-02-09T17:27:44.294 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.17589016900000001 2024-02-09T17:27:44.294 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.294 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.294 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:27:44.294 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.483213+0000", 2024-02-09T17:27:44.294 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4048999999999997e-05 2024-02-09T17:27:44.294 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.294 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.295 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:27:44.295 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.483249+0000", 2024-02-09T17:27:44.295 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6174999999999998e-05 2024-02-09T17:27:44.295 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:44.295 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:27:44.295 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:44.295 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.295 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.295 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:173712 2.a 2:5a5d0456:::benchmark_data_smithi179_77138_object40304:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-02-09T17:27:44.295 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:04.297430+0000", 2024-02-09T17:27:44.295 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 39.924196852000001, 2024-02-09T17:27:44.295 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.19325016, 2024-02-09T17:27:44.295 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:27:44.295 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:44.296 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:27:44.296 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:27:44.296 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:27:44.296 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 173712 2024-02-09T17:27:44.296 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.296 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:27:44.296 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.296 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:27:44.296 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.297430+0000", 2024-02-09T17:27:44.296 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:44.296 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.296 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.297 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:27:44.297 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.297430+0000", 2024-02-09T17:27:44.297 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:44.297 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.297 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.297 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:27:44.297 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.297432+0000", 2024-02-09T17:27:44.297 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8419999999999999e-06 2024-02-09T17:27:44.297 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.297 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.297 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:27:44.297 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.297439+0000", 2024-02-09T17:27:44.297 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7959999999999998e-06 2024-02-09T17:27:44.297 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.298 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.298 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:27:44.298 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.297470+0000", 2024-02-09T17:27:44.298 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1498000000000002e-05 2024-02-09T17:27:44.298 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.298 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.298 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:27:44.298 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.297480+0000", 2024-02-09T17:27:44.298 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.8740000000000007e-06 2024-02-09T17:27:44.298 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.298 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.298 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:27:44.298 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.298020+0000", 2024-02-09T17:27:44.298 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00054042499999999998 2024-02-09T17:27:44.298 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.299 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.299 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:27:44.299 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.298155+0000", 2024-02-09T17:27:44.299 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000134881 2024-02-09T17:27:44.299 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.299 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.299 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,9", 2024-02-09T17:27:44.299 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.298250+0000", 2024-02-09T17:27:44.299 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5222000000000004e-05 2024-02-09T17:27:44.299 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.299 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.299 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:27:44.299 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.305364+0000", 2024-02-09T17:27:44.299 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0071130430000000003 2024-02-09T17:27:44.300 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.300 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.300 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:44.300 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.308096+0000", 2024-02-09T17:27:44.300 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002732557 2024-02-09T17:27:44.300 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.300 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.300 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:44.300 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.490616+0000", 2024-02-09T17:27:44.300 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.18251959600000001 2024-02-09T17:27:44.300 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.300 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.300 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:27:44.301 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.490651+0000", 2024-02-09T17:27:44.301 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.5692999999999999e-05 2024-02-09T17:27:44.301 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.301 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.301 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:27:44.301 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.490680+0000", 2024-02-09T17:27:44.301 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.8733000000000001e-05 2024-02-09T17:27:44.301 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:44.301 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:27:44.301 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:44.301 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.301 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.301 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:173713 2.a 2:5c91d2d5:::benchmark_data_smithi179_77138_object40305:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-02-09T17:27:44.302 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:04.297886+0000", 2024-02-09T17:27:44.302 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 39.923740459000001, 2024-02-09T17:27:44.302 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.19289321800000001, 2024-02-09T17:27:44.302 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:27:44.302 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:44.302 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:27:44.302 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:27:44.302 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:27:44.302 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 173713 2024-02-09T17:27:44.302 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.302 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:27:44.302 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.302 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:27:44.302 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.297886+0000", 2024-02-09T17:27:44.302 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:44.303 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.303 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.303 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:27:44.303 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.297886+0000", 2024-02-09T17:27:44.303 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:44.303 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.303 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.303 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:27:44.303 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.297887+0000", 2024-02-09T17:27:44.303 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1340000000000001e-06 2024-02-09T17:27:44.303 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.303 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.303 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:27:44.303 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.297893+0000", 2024-02-09T17:27:44.303 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3700000000000003e-06 2024-02-09T17:27:44.303 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.304 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.304 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:27:44.304 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.297908+0000", 2024-02-09T17:27:44.304 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.504e-05 2024-02-09T17:27:44.304 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.304 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.304 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:27:44.304 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.297915+0000", 2024-02-09T17:27:44.304 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6020000000000002e-06 2024-02-09T17:27:44.304 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.304 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.304 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:27:44.304 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.301953+0000", 2024-02-09T17:27:44.304 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0040373529999999996 2024-02-09T17:27:44.304 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.305 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.305 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:27:44.305 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.302039+0000", 2024-02-09T17:27:44.305 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.6116e-05 2024-02-09T17:27:44.305 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.305 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.305 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,9", 2024-02-09T17:27:44.305 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.302130+0000", 2024-02-09T17:27:44.305 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.0859000000000005e-05 2024-02-09T17:27:44.305 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.305 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.305 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:27:44.305 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.305481+0000", 2024-02-09T17:27:44.306 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0033510129999999999 2024-02-09T17:27:44.306 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.306 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.306 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:44.306 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.320854+0000", 2024-02-09T17:27:44.306 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.015373658 2024-02-09T17:27:44.306 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.306 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.306 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:44.306 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.490718+0000", 2024-02-09T17:27:44.306 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.16986306000000001 2024-02-09T17:27:44.306 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.306 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.306 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:27:44.307 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.490760+0000", 2024-02-09T17:27:44.307 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2525999999999999e-05 2024-02-09T17:27:44.307 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.307 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.307 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:27:44.307 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.490780+0000", 2024-02-09T17:27:44.307 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9487000000000002e-05 2024-02-09T17:27:44.307 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:44.307 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:27:44.307 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:44.307 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.307 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.307 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:173721 2.a 2:526418d3:::benchmark_data_smithi179_77138_object40313:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-02-09T17:27:44.307 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:04.311408+0000", 2024-02-09T17:27:44.308 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 39.910219019000003, 2024-02-09T17:27:44.308 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.183574246, 2024-02-09T17:27:44.308 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:27:44.308 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:44.308 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:27:44.308 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:27:44.308 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:27:44.308 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 173721 2024-02-09T17:27:44.308 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.308 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:27:44.308 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.308 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:27:44.308 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.311408+0000", 2024-02-09T17:27:44.309 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:44.309 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.309 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.309 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:27:44.309 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.311408+0000", 2024-02-09T17:27:44.309 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:44.309 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.309 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.309 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:27:44.309 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.311409+0000", 2024-02-09T17:27:44.309 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0049999999999999e-06 2024-02-09T17:27:44.309 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.309 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.309 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:27:44.309 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.311413+0000", 2024-02-09T17:27:44.310 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4859999999999999e-06 2024-02-09T17:27:44.310 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.310 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.310 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:27:44.310 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.311424+0000", 2024-02-09T17:27:44.310 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0403e-05 2024-02-09T17:27:44.310 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.310 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.310 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:27:44.310 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.311430+0000", 2024-02-09T17:27:44.310 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1330000000000004e-06 2024-02-09T17:27:44.310 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.310 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.311 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:27:44.311 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.311479+0000", 2024-02-09T17:27:44.311 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8979e-05 2024-02-09T17:27:44.311 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.311 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.311 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:27:44.311 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.311557+0000", 2024-02-09T17:27:44.311 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7886000000000006e-05 2024-02-09T17:27:44.311 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.311 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.311 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,9", 2024-02-09T17:27:44.311 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.311638+0000", 2024-02-09T17:27:44.311 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1574000000000002e-05 2024-02-09T17:27:44.311 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.312 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.312 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:27:44.312 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.315588+0000", 2024-02-09T17:27:44.312 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0039498340000000002 2024-02-09T17:27:44.312 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.312 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.312 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:44.312 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.320939+0000", 2024-02-09T17:27:44.312 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0053514269999999997 2024-02-09T17:27:44.312 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.312 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.312 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:44.312 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.494887+0000", 2024-02-09T17:27:44.312 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.17394747799999999 2024-02-09T17:27:44.313 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.313 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.313 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:27:44.313 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.494942+0000", 2024-02-09T17:27:44.313 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5154000000000001e-05 2024-02-09T17:27:44.313 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.313 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.313 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:27:44.313 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.494982+0000", 2024-02-09T17:27:44.313 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9886999999999999e-05 2024-02-09T17:27:44.313 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:44.313 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:27:44.313 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:44.313 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.314 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.314 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:173735 2.a 2:56d1d430:::benchmark_data_smithi179_77138_object40327:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-02-09T17:27:44.314 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:04.331216+0000", 2024-02-09T17:27:44.314 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 39.890410752000001, 2024-02-09T17:27:44.314 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.16389076299999999, 2024-02-09T17:27:44.314 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:27:44.314 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:44.314 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:27:44.314 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:27:44.314 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:27:44.314 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 173735 2024-02-09T17:27:44.314 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.314 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:27:44.314 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.314 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:27:44.315 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.331216+0000", 2024-02-09T17:27:44.315 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:44.315 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.315 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.315 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:27:44.315 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.331216+0000", 2024-02-09T17:27:44.315 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:44.315 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.315 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.315 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:27:44.315 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.331218+0000", 2024-02-09T17:27:44.315 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9520000000000001e-06 2024-02-09T17:27:44.315 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.316 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.316 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:27:44.316 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.331230+0000", 2024-02-09T17:27:44.316 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2106000000000001e-05 2024-02-09T17:27:44.316 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.316 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.316 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:27:44.316 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.331245+0000", 2024-02-09T17:27:44.316 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5373e-05 2024-02-09T17:27:44.316 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.316 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.316 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:27:44.316 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.331255+0000", 2024-02-09T17:27:44.316 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.8269999999999993e-06 2024-02-09T17:27:44.316 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.317 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.317 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:27:44.317 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.331302+0000", 2024-02-09T17:27:44.317 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6808999999999999e-05 2024-02-09T17:27:44.317 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.317 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.317 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:27:44.317 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.331430+0000", 2024-02-09T17:27:44.317 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012804400000000001 2024-02-09T17:27:44.317 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.317 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.317 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,9", 2024-02-09T17:27:44.317 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.331521+0000", 2024-02-09T17:27:44.317 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.0412000000000001e-05 2024-02-09T17:27:44.317 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.317 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.318 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:27:44.318 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.335423+0000", 2024-02-09T17:27:44.318 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0039019829999999999 2024-02-09T17:27:44.318 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.318 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.318 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:44.318 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.335458+0000", 2024-02-09T17:27:44.318 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.5023e-05 2024-02-09T17:27:44.318 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.318 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.318 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:44.318 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.495032+0000", 2024-02-09T17:27:44.318 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15957394699999999 2024-02-09T17:27:44.318 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.318 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.318 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:27:44.319 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.495074+0000", 2024-02-09T17:27:44.319 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2463000000000003e-05 2024-02-09T17:27:44.319 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.319 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.319 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:27:44.319 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.495107+0000", 2024-02-09T17:27:44.319 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2824000000000003e-05 2024-02-09T17:27:44.319 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:44.319 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:27:44.319 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:44.319 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.319 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.319 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:173761 2.a 2:5adb3f55:::benchmark_data_smithi179_77138_object40353:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-02-09T17:27:44.320 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:04.370802+0000", 2024-02-09T17:27:44.320 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 39.850824385999999, 2024-02-09T17:27:44.320 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.128114384, 2024-02-09T17:27:44.320 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:27:44.320 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:44.320 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:27:44.320 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:27:44.320 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:27:44.320 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 173761 2024-02-09T17:27:44.320 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.320 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:27:44.320 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.320 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:27:44.320 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.370802+0000", 2024-02-09T17:27:44.321 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:44.321 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.321 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.321 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:27:44.321 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.370802+0000", 2024-02-09T17:27:44.321 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:44.321 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.321 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.321 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:27:44.321 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.370803+0000", 2024-02-09T17:27:44.321 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4600000000000004e-07 2024-02-09T17:27:44.321 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.321 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.322 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:27:44.322 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.370807+0000", 2024-02-09T17:27:44.322 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0300000000000004e-06 2024-02-09T17:27:44.322 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.322 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.322 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:27:44.322 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.370815+0000", 2024-02-09T17:27:44.322 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1149999999999994e-06 2024-02-09T17:27:44.322 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.322 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.322 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:27:44.322 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.370822+0000", 2024-02-09T17:27:44.322 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9280000000000004e-06 2024-02-09T17:27:44.322 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.323 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.323 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:27:44.323 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.370875+0000", 2024-02-09T17:27:44.323 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2380000000000003e-05 2024-02-09T17:27:44.323 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.323 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.323 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:27:44.323 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.370985+0000", 2024-02-09T17:27:44.323 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000109993 2024-02-09T17:27:44.323 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.323 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.323 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,9", 2024-02-09T17:27:44.323 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.371108+0000", 2024-02-09T17:27:44.323 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000123091 2024-02-09T17:27:44.324 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.324 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.324 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:27:44.324 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.375928+0000", 2024-02-09T17:27:44.324 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0048205840000000002 2024-02-09T17:27:44.324 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.324 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.324 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:44.324 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.376700+0000", 2024-02-09T17:27:44.324 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00077203700000000005 2024-02-09T17:27:44.324 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.324 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.324 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:44.324 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.498837+0000", 2024-02-09T17:27:44.325 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.122136288 2024-02-09T17:27:44.325 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.325 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.325 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:27:44.325 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.498889+0000", 2024-02-09T17:27:44.325 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1965999999999997e-05 2024-02-09T17:27:44.325 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.325 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.325 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:27:44.325 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.498917+0000", 2024-02-09T17:27:44.325 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.8226e-05 2024-02-09T17:27:44.325 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:44.325 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:27:44.325 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:44.326 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.326 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.326 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:173778 2.a 2:5f827783:::benchmark_data_smithi179_77138_object40370:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-02-09T17:27:44.326 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:04.398174+0000", 2024-02-09T17:27:44.326 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 39.823452543999998, 2024-02-09T17:27:44.326 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.10768032600000001, 2024-02-09T17:27:44.326 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:27:44.326 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:44.326 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:27:44.326 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:27:44.326 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:27:44.326 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 173778 2024-02-09T17:27:44.326 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.327 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:27:44.327 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.327 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:27:44.327 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.398174+0000", 2024-02-09T17:27:44.327 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:44.327 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.327 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.327 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:27:44.327 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.398174+0000", 2024-02-09T17:27:44.327 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:44.327 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.327 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.327 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:27:44.327 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.398176+0000", 2024-02-09T17:27:44.328 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.381e-06 2024-02-09T17:27:44.328 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.328 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.328 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:27:44.328 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.398180+0000", 2024-02-09T17:27:44.328 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2440000000000004e-06 2024-02-09T17:27:44.328 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.328 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.328 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:27:44.328 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.398191+0000", 2024-02-09T17:27:44.328 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0701e-05 2024-02-09T17:27:44.328 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.328 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.329 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:27:44.329 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.398198+0000", 2024-02-09T17:27:44.329 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5329999999999997e-06 2024-02-09T17:27:44.329 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.329 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.329 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:27:44.329 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.398239+0000", 2024-02-09T17:27:44.329 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1236000000000003e-05 2024-02-09T17:27:44.329 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.329 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.329 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:27:44.329 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.398371+0000", 2024-02-09T17:27:44.329 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013168699999999999 2024-02-09T17:27:44.330 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.330 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.330 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,9", 2024-02-09T17:27:44.330 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.398482+0000", 2024-02-09T17:27:44.330 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000111254 2024-02-09T17:27:44.330 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.330 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.330 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:27:44.330 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.403127+0000", 2024-02-09T17:27:44.330 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0046451579999999999 2024-02-09T17:27:44.330 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.330 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.330 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:44.330 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.403533+0000", 2024-02-09T17:27:44.331 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00040593600000000002 2024-02-09T17:27:44.331 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.331 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.331 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:44.331 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.505754+0000", 2024-02-09T17:27:44.331 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.10222039099999999 2024-02-09T17:27:44.331 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.331 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.331 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:27:44.331 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.505815+0000", 2024-02-09T17:27:44.331 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1079000000000003e-05 2024-02-09T17:27:44.331 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.331 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.332 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:27:44.332 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.505855+0000", 2024-02-09T17:27:44.332 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9725999999999999e-05 2024-02-09T17:27:44.332 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:44.332 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:27:44.332 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:44.332 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.332 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.332 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:173780 2.a 2:5f6bf9e0:::benchmark_data_smithi179_77138_object40372:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-02-09T17:27:44.332 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:04.403512+0000", 2024-02-09T17:27:44.332 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 39.818114364000003, 2024-02-09T17:27:44.332 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.109091655, 2024-02-09T17:27:44.332 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:27:44.333 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:44.333 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:27:44.333 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:27:44.333 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:27:44.333 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 173780 2024-02-09T17:27:44.333 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.333 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:27:44.333 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.333 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:27:44.333 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.403512+0000", 2024-02-09T17:27:44.333 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:44.333 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.333 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.334 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:27:44.334 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.403512+0000", 2024-02-09T17:27:44.334 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:44.334 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.334 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.334 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:27:44.334 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.403513+0000", 2024-02-09T17:27:44.334 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.068e-06 2024-02-09T17:27:44.334 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.334 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.334 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:27:44.334 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.403518+0000", 2024-02-09T17:27:44.334 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4909999999999999e-06 2024-02-09T17:27:44.335 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.335 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.335 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:27:44.335 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.403528+0000", 2024-02-09T17:27:44.335 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5210000000000008e-06 2024-02-09T17:27:44.335 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.335 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.335 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:27:44.335 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.403535+0000", 2024-02-09T17:27:44.335 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1400000000000002e-06 2024-02-09T17:27:44.335 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.335 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.335 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:27:44.335 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.403591+0000", 2024-02-09T17:27:44.335 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5965999999999999e-05 2024-02-09T17:27:44.335 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.336 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.336 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:27:44.336 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.403706+0000", 2024-02-09T17:27:44.336 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000115307 2024-02-09T17:27:44.336 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.336 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.336 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,9", 2024-02-09T17:27:44.336 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.403811+0000", 2024-02-09T17:27:44.336 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000105507 2024-02-09T17:27:44.336 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.336 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.336 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:44.336 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.412034+0000", 2024-02-09T17:27:44.336 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0082223490000000003 2024-02-09T17:27:44.337 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.337 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.337 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:27:44.337 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.415359+0000", 2024-02-09T17:27:44.337 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0033255400000000001 2024-02-09T17:27:44.337 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.337 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.337 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:44.337 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.512501+0000", 2024-02-09T17:27:44.337 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.097141874000000003 2024-02-09T17:27:44.337 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.337 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.337 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:27:44.337 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.512561+0000", 2024-02-09T17:27:44.338 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9851e-05 2024-02-09T17:27:44.338 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.338 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.338 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:27:44.338 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.512604+0000", 2024-02-09T17:27:44.338 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3040999999999997e-05 2024-02-09T17:27:44.338 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:44.338 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:27:44.338 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:44.338 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.338 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.338 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:173783 2.a 2:5bec6a3b:::benchmark_data_smithi179_77138_object40375:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-02-09T17:27:44.338 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:04.403928+0000", 2024-02-09T17:27:44.338 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 39.817698634000003, 2024-02-09T17:27:44.338 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.108807521, 2024-02-09T17:27:44.338 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:27:44.339 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:44.339 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:27:44.339 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:27:44.339 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:27:44.339 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 173783 2024-02-09T17:27:44.339 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.339 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:27:44.339 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.339 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:27:44.339 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.403928+0000", 2024-02-09T17:27:44.339 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:44.339 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.339 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.340 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:27:44.340 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.403928+0000", 2024-02-09T17:27:44.340 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:44.340 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.340 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.340 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:27:44.340 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.403929+0000", 2024-02-09T17:27:44.340 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7399999999999991e-07 2024-02-09T17:27:44.340 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.340 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.340 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:27:44.340 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.403934+0000", 2024-02-09T17:27:44.340 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9350000000000002e-06 2024-02-09T17:27:44.341 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.341 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.341 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:27:44.341 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.403947+0000", 2024-02-09T17:27:44.341 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2864000000000001e-05 2024-02-09T17:27:44.341 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.341 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.341 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:27:44.341 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.403955+0000", 2024-02-09T17:27:44.341 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7540000000000006e-06 2024-02-09T17:27:44.341 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.341 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.341 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:27:44.341 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.408183+0000", 2024-02-09T17:27:44.342 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.004228206 2024-02-09T17:27:44.342 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.342 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.342 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:27:44.342 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.408283+0000", 2024-02-09T17:27:44.342 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000100256 2024-02-09T17:27:44.342 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.342 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.342 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,9", 2024-02-09T17:27:44.342 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.408365+0000", 2024-02-09T17:27:44.342 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1427000000000005e-05 2024-02-09T17:27:44.342 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.342 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.342 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:44.342 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.415396+0000", 2024-02-09T17:27:44.343 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0070318530000000002 2024-02-09T17:27:44.343 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.343 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.343 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:27:44.343 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.415418+0000", 2024-02-09T17:27:44.343 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1699999999999999e-05 2024-02-09T17:27:44.343 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.343 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.343 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:44.343 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.512645+0000", 2024-02-09T17:27:44.343 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.097226983000000003 2024-02-09T17:27:44.343 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.343 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.343 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:27:44.344 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.512701+0000", 2024-02-09T17:27:44.344 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6017000000000003e-05 2024-02-09T17:27:44.344 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.344 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.344 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:27:44.344 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.512736+0000", 2024-02-09T17:27:44.344 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4551999999999997e-05 2024-02-09T17:27:44.344 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:44.344 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:27:44.344 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:44.344 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.344 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.344 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:173785 2.a 2:50bf97e6:::benchmark_data_smithi179_77138_object40377:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-02-09T17:27:44.345 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:04.410070+0000", 2024-02-09T17:27:44.345 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 39.811557278000002, 2024-02-09T17:27:44.345 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.10294313099999999, 2024-02-09T17:27:44.345 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:27:44.345 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:27:44.345 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:27:44.345 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:27:44.345 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:27:44.345 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 173785 2024-02-09T17:27:44.345 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.345 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:27:44.345 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.345 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:27:44.346 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.410070+0000", 2024-02-09T17:27:44.346 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:44.346 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.346 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.346 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:27:44.346 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.410070+0000", 2024-02-09T17:27:44.346 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:27:44.346 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.346 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.346 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:27:44.346 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.410071+0000", 2024-02-09T17:27:44.346 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3319999999999999e-06 2024-02-09T17:27:44.346 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.346 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.346 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:27:44.347 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.410076+0000", 2024-02-09T17:27:44.347 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.631e-06 2024-02-09T17:27:44.347 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.347 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.347 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:27:44.347 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.410091+0000", 2024-02-09T17:27:44.347 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4793999999999999e-05 2024-02-09T17:27:44.347 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.347 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.347 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:27:44.347 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.410100+0000", 2024-02-09T17:27:44.347 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.6689999999999995e-06 2024-02-09T17:27:44.347 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.347 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.347 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:27:44.348 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.412069+0000", 2024-02-09T17:27:44.348 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0019694320000000001 2024-02-09T17:27:44.348 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.348 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.348 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:27:44.348 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.412205+0000", 2024-02-09T17:27:44.348 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000135847 2024-02-09T17:27:44.348 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.348 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.348 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,9", 2024-02-09T17:27:44.348 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.412301+0000", 2024-02-09T17:27:44.348 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6249000000000006e-05 2024-02-09T17:27:44.348 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.348 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.349 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:27:44.349 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.415476+0000", 2024-02-09T17:27:44.349 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0031745520000000002 2024-02-09T17:27:44.349 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.349 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.349 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:44.349 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.418017+0000", 2024-02-09T17:27:44.349 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0025409059999999999 2024-02-09T17:27:44.349 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.349 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.349 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:27:44.349 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.512931+0000", 2024-02-09T17:27:44.349 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.094913708999999999 2024-02-09T17:27:44.349 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.349 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.350 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:27:44.350 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.512976+0000", 2024-02-09T17:27:44.350 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5286e-05 2024-02-09T17:27:44.350 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:27:44.350 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:27:44.350 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:27:44.350 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.513013+0000", 2024-02-09T17:27:44.350 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6724000000000003e-05 2024-02-09T17:27:44.350 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:44.350 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:27:44.350 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:44.350 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:27:44.350 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:27:44.350 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:27:44.351 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:27:44.352 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:27:44.380 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:44.403 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:27:44.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:44.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:27:44.410 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:27:44.459 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:27:44.509 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:27:44.518 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:27:44.572 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:27:44.587 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:44.612 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:27:44.612 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:27:44.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:44.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:27:44.649 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:27:44.702 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:27:44.716 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:27:44.739 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:27:44.750 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:44.795 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:27:44.837 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:27:44.858 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:44 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1897444481' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:27:44.858 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:44 smithi099 ceph-mon[39420]: pgmap v495: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 3.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 796 op/s; 15326/184422 objects degraded (8.310%) 2024-02-09T17:27:44.879 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:27:44.879 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:27:44.879 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:27:44.879 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:27:44.879 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:27:44.879 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:27:44.879 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:27:44.879 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:27:44.879 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:27:44.879 INFO:teuthology.orchestra.run.smithi099.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-02-09T17:27:44.880 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:27:44.880 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:27:44.880 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:27:44.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:44 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1897444481' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:27:44.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:44 smithi179 ceph-mon[43975]: pgmap v495: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 3.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 796 op/s; 15326/184422 objects degraded (8.310%) 2024-02-09T17:27:45.104 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:27:45.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:45.110 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:27:45.124 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-09T17:27:45.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:44 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1897444481' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:27:45.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:44 smithi162 ceph-mon[44304]: pgmap v495: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 3.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 796 op/s; 15326/184422 objects degraded (8.310%) 2024-02-09T17:27:45.211 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:27:45.239 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:45.261 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:27:45.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:45.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:27:45.366 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:27:45.399 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:45.424 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:27:45.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:45.428 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:27:45.529 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:27:45.545 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:27:45.595 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:45.754 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:27:45.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:45.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:27:45.865 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:45 smithi099 ceph-mon[39420]: 2.0 scrub starts 2024-02-09T17:27:45.872 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:27:45.902 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:45.930 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:27:45.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:45.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:27:46.037 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:27:46.073 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:46.097 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:27:46.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:46.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:27:46.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:45 smithi179 ceph-mon[43975]: 2.0 scrub starts 2024-02-09T17:27:46.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:45 smithi162 ceph-mon[44304]: 2.0 scrub starts 2024-02-09T17:27:46.202 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:27:46.239 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:46.264 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:27:46.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:46.272 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:27:46.373 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:27:46.432 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:46.603 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:27:46.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:46.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:27:46.708 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:27:46.763 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:46.792 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-09T17:27:46.793 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-09T17:27:46.795 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:27:46.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:46.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:27:46.900 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:27:46.940 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:46.964 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:27:46.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:46.967 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:27:47.068 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:27:47.096 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:47.119 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:27:47.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:47.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:27:47.168 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:46 smithi099 ceph-mon[39420]: 2.a scrub starts 2024-02-09T17:27:47.168 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:46 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 15326/184422 objects degraded (8.310%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:27:47.168 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:46 smithi099 ceph-mon[39420]: 2.0 scrub ok 2024-02-09T17:27:47.168 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:46 smithi099 ceph-mon[39420]: pgmap v496: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 3.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 993 op/s; 14818/178056 objects degraded (8.322%) 2024-02-09T17:27:47.168 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:46 smithi099 ceph-mon[39420]: 2.a scrub ok 2024-02-09T17:27:47.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:46 smithi179 ceph-mon[43975]: 2.a scrub starts 2024-02-09T17:27:47.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:46 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 15326/184422 objects degraded (8.310%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:27:47.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:46 smithi179 ceph-mon[43975]: 2.0 scrub ok 2024-02-09T17:27:47.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:46 smithi179 ceph-mon[43975]: pgmap v496: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 3.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 993 op/s; 14818/178056 objects degraded (8.322%) 2024-02-09T17:27:47.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:46 smithi179 ceph-mon[43975]: 2.a scrub ok 2024-02-09T17:27:47.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:46 smithi162 ceph-mon[44304]: 2.a scrub starts 2024-02-09T17:27:47.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:46 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 15326/184422 objects degraded (8.310%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:27:47.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:46 smithi162 ceph-mon[44304]: 2.0 scrub ok 2024-02-09T17:27:47.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:46 smithi162 ceph-mon[44304]: pgmap v496: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 3.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 993 op/s; 14818/178056 objects degraded (8.322%) 2024-02-09T17:27:47.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:46 smithi162 ceph-mon[44304]: 2.a scrub ok 2024-02-09T17:27:47.224 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:27:47.257 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:47.290 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:27:47.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:47.294 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:27:47.395 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:27:47.422 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:47.442 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:27:47.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:47.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:27:47.547 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:27:47.584 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:47.618 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:27:47.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:47.622 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:27:47.723 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:27:47.755 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:47.782 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:27:47.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:47.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:27:47.888 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:27:47.920 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:47.949 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:27:47.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:47.954 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:27:48.055 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:27:48.112 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:48.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:47 smithi179 ceph-mon[43975]: 2.c scrub starts 2024-02-09T17:27:48.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:47 smithi162 ceph-mon[44304]: 2.c scrub starts 2024-02-09T17:27:48.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:47 smithi099 ceph-mon[39420]: 2.c scrub starts 2024-02-09T17:27:48.267 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:27:48.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:48.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:27:48.385 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:27:48.418 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:48.440 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:27:48.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:48.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:27:48.548 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:27:48.595 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:48.620 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:27:48.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:48.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:27:48.672 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:27:48.724 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:27:48.765 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:48.793 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:27:48.851 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:27:48.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:48.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:27:48.929 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is set 2024-02-09T17:27:48.950 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:48 smithi099 ceph-mon[39420]: 2.c scrub ok 2024-02-09T17:27:48.950 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:48 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3193505005' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:27:48.951 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:48 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:27:48.951 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:48 smithi099 ceph-mon[39420]: pgmap v497: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 3.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 855 op/s; 14818/176421 objects degraded (8.399%) 2024-02-09T17:27:48.958 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:27:48.986 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:49.009 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:27:49.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:49.013 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:27:49.114 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:27:49.144 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:49.167 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:27:49.170 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:48 smithi162 ceph-mon[44304]: 2.c scrub ok 2024-02-09T17:27:49.170 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:48 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3193505005' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:27:49.170 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:48 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:27:49.170 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:48 smithi162 ceph-mon[44304]: pgmap v497: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 3.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 855 op/s; 14818/176421 objects degraded (8.399%) 2024-02-09T17:27:49.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:49.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:27:49.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:48 smithi179 ceph-mon[43975]: 2.c scrub ok 2024-02-09T17:27:49.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:48 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3193505005' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:27:49.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:48 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:27:49.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:48 smithi179 ceph-mon[43975]: pgmap v497: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 3.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 855 op/s; 14818/176421 objects degraded (8.399%) 2024-02-09T17:27:49.272 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:27:49.338 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:49.912 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:27:49.923 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:49.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:27:50.025 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:27:50.056 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:50.086 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:27:50.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:50.090 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:27:50.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:49 smithi179 ceph-mon[43975]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:27:50.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:49 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:27:50.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:49 smithi179 ceph-mon[43975]: osdmap e166: 12 total, 11 up, 12 in 2024-02-09T17:27:50.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:49 smithi162 ceph-mon[44304]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:27:50.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:49 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:27:50.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:49 smithi162 ceph-mon[44304]: osdmap e166: 12 total, 11 up, 12 in 2024-02-09T17:27:50.191 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:27:50.220 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:50.246 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:27:50.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:49 smithi099 ceph-mon[39420]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:27:50.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:49 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:27:50.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:49 smithi099 ceph-mon[39420]: osdmap e166: 12 total, 11 up, 12 in 2024-02-09T17:27:50.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:50.250 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:27:50.308 INFO:teuthology.orchestra.run.smithi099.stdout:moved 1 / 51 (1.96078%) 2024-02-09T17:27:50.309 INFO:teuthology.orchestra.run.smithi099.stdout:avg 4.63636 2024-02-09T17:27:50.309 INFO:teuthology.orchestra.run.smithi099.stdout:stddev 1.40248 -> 1.46587 (expected baseline 2.05302) 2024-02-09T17:27:50.309 INFO:teuthology.orchestra.run.smithi099.stdout:min osd.5 with 2 -> 2 pgs (0.431373 -> 0.431373 * mean) 2024-02-09T17:27:50.309 INFO:teuthology.orchestra.run.smithi099.stdout:max osd.3 with 6 -> 6 pgs (1.29412 -> 1.29412 * mean) 2024-02-09T17:27:50.309 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:27:50.309 INFO:teuthology.orchestra.run.smithi099.stdout:oload 110 2024-02-09T17:27:50.309 INFO:teuthology.orchestra.run.smithi099.stdout:max_change 0.05 2024-02-09T17:27:50.309 INFO:teuthology.orchestra.run.smithi099.stdout:max_change_osds 4 2024-02-09T17:27:50.309 INFO:teuthology.orchestra.run.smithi099.stdout:average_utilization 0.0117 2024-02-09T17:27:50.309 INFO:teuthology.orchestra.run.smithi099.stdout:overload_utilization 0.0128 2024-02-09T17:27:50.309 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8 weight 1.0000 -> 0.9500 2024-02-09T17:27:50.309 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3 weight 1.0000 -> 0.9500 2024-02-09T17:27:50.309 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9 weight 1.0000 -> 0.9500 2024-02-09T17:27:50.309 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1 weight 1.0000 -> 0.9500 2024-02-09T17:27:50.337 INFO:teuthology.orchestra.run.smithi099.stderr:no change 2024-02-09T17:27:50.351 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:27:50.385 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:50.409 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:27:50.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:50.412 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:27:50.512 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:27:50.541 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:50.567 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:27:50.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:50.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:27:50.676 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:27:50.721 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:27:50 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:27:50.298+0000 7f5169e95700 -1 mgr.server reply reply (4) Interrupted system call no change 2024-02-09T17:27:50.740 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:50.936 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:27:50.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:50.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:27:50.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:50 smithi099 ceph-mon[39420]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:27:50.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:50 smithi099 ceph-mon[39420]: pgmap v499: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 3.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 843 op/s; 14069/169296 objects degraded (8.310%) 2024-02-09T17:27:51.043 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:27:51.070 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:51.092 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:27:51.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:51.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:27:51.200 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:27:51.259 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:51.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:50 smithi179 ceph-mon[43975]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:27:51.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:50 smithi179 ceph-mon[43975]: pgmap v499: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 3.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 843 op/s; 14069/169296 objects degraded (8.310%) 2024-02-09T17:27:51.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:50 smithi162 ceph-mon[44304]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:27:51.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:50 smithi162 ceph-mon[44304]: pgmap v499: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 3.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 843 op/s; 14069/169296 objects degraded (8.310%) 2024-02-09T17:27:51.538 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:27:51.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:51.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:27:51.591 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:27:51.643 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:27:51.672 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:51.696 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:27:51.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:51.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:27:51.803 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:27:51.832 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:51.861 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:27:51.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:51.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:27:51.966 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:27:52.011 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:52.145 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-02-09T17:27:52.145 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd test-reweight-by-utilization 110 0.05 2024-02-09T17:27:52.225 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:51 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 14069/169296 objects degraded (8.310%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:27:52.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:51 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 14069/169296 objects degraded (8.310%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:27:52.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:51 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 14069/169296 objects degraded (8.310%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:27:52.782 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-09T17:27:52.960 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:52 smithi099 ceph-mon[39420]: pgmap v500: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 847 op/s; 13775/163641 objects degraded (8.418%) 2024-02-09T17:27:53.018 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:27:53.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:53.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:27:53.130 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:27:53.131 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:27:53.132 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:27:53.215 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:53.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:52 smithi179 ceph-mon[43975]: pgmap v500: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 847 op/s; 13775/163641 objects degraded (8.418%) 2024-02-09T17:27:53.477 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:52 smithi162 ceph-mon[44304]: pgmap v500: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 847 op/s; 13775/163641 objects degraded (8.418%) 2024-02-09T17:27:54.186 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:27:54.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:54.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:27:54.293 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:27:54.325 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:54.352 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:27:54.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:54.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:27:54.456 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:27:54.486 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:54.507 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:27:54.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:54.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:27:54.613 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:27:54.688 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:54.776 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:27:54.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:54.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:27:54.881 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:27:54.907 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:54.910 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:27:54.932 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:27:54.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:54.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:27:54.999 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:27:55.036 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:27:55.050 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:27:55.077 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:55.111 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:27:55.157 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:27:55.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:55 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:27:55.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:55 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:27:55.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:55 smithi162 ceph-mon[44304]: osdmap e167: 12 total, 11 up, 12 in 2024-02-09T17:27:55.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:55 smithi162 ceph-mon[44304]: pgmap v502: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 803 op/s; 13266/158757 objects degraded (8.356%) 2024-02-09T17:27:55.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:55 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1238548055' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:27:55.195 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:27:55.229 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:27:55.257 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:27:55.321 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:27:55.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:55.328 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:55 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:27:55.328 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:55 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:27:55.328 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:55 smithi099 ceph-mon[39420]: osdmap e167: 12 total, 11 up, 12 in 2024-02-09T17:27:55.328 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:55 smithi099 ceph-mon[39420]: pgmap v502: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 803 op/s; 13266/158757 objects degraded (8.356%) 2024-02-09T17:27:55.328 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:55 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1238548055' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:27:55.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:27:55.329 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:27:55.348 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-09T17:27:55.382 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:27:55.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:55 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:27:55.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:55 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:27:55.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:55 smithi179 ceph-mon[43975]: osdmap e167: 12 total, 11 up, 12 in 2024-02-09T17:27:55.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:55 smithi179 ceph-mon[43975]: pgmap v502: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 803 op/s; 13266/158757 objects degraded (8.356%) 2024-02-09T17:27:55.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:55 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1238548055' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:27:55.427 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:27:55.430 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:27:55.480 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:27:55.480 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:27:55.480 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:27:55.480 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:27:55.480 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:27:55.480 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:27:55.480 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:27:55.480 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:27:55.480 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:27:55.480 INFO:teuthology.orchestra.run.smithi099.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-02-09T17:27:55.481 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:27:55.481 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:27:55.481 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:27:55.507 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:27:56.386 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:56 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 13266/158757 objects degraded (8.356%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:27:56.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:56 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 13266/158757 objects degraded (8.356%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:27:56.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:56 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 13266/158757 objects degraded (8.356%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:27:56.428 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-09T17:27:56.429 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-09T17:27:56.784 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:27:56.841 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:27:57.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:57 smithi179 ceph-mon[43975]: pgmap v503: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 3.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s; 12718/152205 objects degraded (8.356%) 2024-02-09T17:27:57.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:57 smithi162 ceph-mon[44304]: pgmap v503: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 3.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s; 12718/152205 objects degraded (8.356%) 2024-02-09T17:27:57.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:57 smithi099 ceph-mon[39420]: pgmap v503: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 3.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s; 12718/152205 objects degraded (8.356%) 2024-02-09T17:27:58.430 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:27:59.853 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:27:59 smithi099 ceph-mon[39420]: pgmap v504: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 3.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 876 op/s; 12718/150600 objects degraded (8.445%) 2024-02-09T17:27:59.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:27:59 smithi179 ceph-mon[43975]: pgmap v504: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 3.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 876 op/s; 12718/150600 objects degraded (8.445%) 2024-02-09T17:27:59.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:27:59 smithi162 ceph-mon[44304]: pgmap v504: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 3.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 876 op/s; 12718/150600 objects degraded (8.445%) 2024-02-09T17:27:59.983 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:27:59.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:27:59.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:28:00.093 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:28:00.121 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:00.144 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:28:00.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:00.149 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:28:00.250 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:28:00.278 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:00.301 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:28:00.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:00.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:28:00.406 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:28:00.483 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:01.578 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is set 2024-02-09T17:28:01.698 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:28:01.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:01.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:28:01.717 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:01 smithi099 ceph-mon[39420]: pgmap v505: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 2.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 907 op/s; 11849/142041 objects degraded (8.342%) 2024-02-09T17:28:01.717 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:01 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 11849/142041 objects degraded (8.342%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:28:01.717 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:01 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2926822519' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:28:01.717 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:01 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:28:01.813 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:28:01.872 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:01.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:01 smithi179 ceph-mon[43975]: pgmap v505: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 2.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 907 op/s; 11849/142041 objects degraded (8.342%) 2024-02-09T17:28:01.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:01 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 11849/142041 objects degraded (8.342%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:28:01.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:01 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2926822519' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:28:01.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:01 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:28:01.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:01 smithi162 ceph-mon[44304]: pgmap v505: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 2.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 907 op/s; 11849/142041 objects degraded (8.342%) 2024-02-09T17:28:01.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:01 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 11849/142041 objects degraded (8.342%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:28:01.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:01 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2926822519' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:28:01.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:01 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:28:02.029 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:28:02.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:02.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:28:02.134 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:28:02.166 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:02.195 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:28:02.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:02.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:28:02.304 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:28:02.341 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:02.363 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:28:02.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:02.368 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:28:02.469 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:28:02.496 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:02.517 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:28:02.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:02.522 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:28:02.623 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:28:02.667 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:02.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:02 smithi179 ceph-mon[43975]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:28:02.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:02 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:28:02.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:02 smithi179 ceph-mon[43975]: osdmap e168: 12 total, 11 up, 12 in 2024-02-09T17:28:02.699 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:28:02.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:02.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:28:02.735 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:02 smithi099 ceph-mon[39420]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:28:02.735 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:02 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:28:02.736 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:02 smithi099 ceph-mon[39420]: osdmap e168: 12 total, 11 up, 12 in 2024-02-09T17:28:02.805 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:28:02.876 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:02.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:02 smithi162 ceph-mon[44304]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:28:02.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:02 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:28:02.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:02 smithi162 ceph-mon[44304]: osdmap e168: 12 total, 11 up, 12 in 2024-02-09T17:28:03.370 INFO:teuthology.orchestra.run.smithi099.stdout:moved 1 / 51 (1.96078%) 2024-02-09T17:28:03.370 INFO:teuthology.orchestra.run.smithi099.stdout:avg 4.63636 2024-02-09T17:28:03.370 INFO:teuthology.orchestra.run.smithi099.stdout:stddev 1.40248 -> 1.46587 (expected baseline 2.05302) 2024-02-09T17:28:03.370 INFO:teuthology.orchestra.run.smithi099.stdout:min osd.5 with 2 -> 2 pgs (0.431373 -> 0.431373 * mean) 2024-02-09T17:28:03.371 INFO:teuthology.orchestra.run.smithi099.stdout:max osd.3 with 6 -> 6 pgs (1.29412 -> 1.29412 * mean) 2024-02-09T17:28:03.371 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:28:03.371 INFO:teuthology.orchestra.run.smithi099.stdout:oload 110 2024-02-09T17:28:03.371 INFO:teuthology.orchestra.run.smithi099.stdout:max_change 0.05 2024-02-09T17:28:03.371 INFO:teuthology.orchestra.run.smithi099.stdout:max_change_osds 4 2024-02-09T17:28:03.372 INFO:teuthology.orchestra.run.smithi099.stdout:average_utilization 0.0097 2024-02-09T17:28:03.372 INFO:teuthology.orchestra.run.smithi099.stdout:overload_utilization 0.0107 2024-02-09T17:28:03.372 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8 weight 1.0000 -> 0.9500 2024-02-09T17:28:03.372 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3 weight 1.0000 -> 0.9500 2024-02-09T17:28:03.372 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9 weight 1.0000 -> 0.9500 2024-02-09T17:28:03.372 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0 weight 1.0000 -> 0.9500 2024-02-09T17:28:03.386 INFO:teuthology.orchestra.run.smithi099.stderr:no change 2024-02-09T17:28:03.744 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:28:03 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:28:03.367+0000 7f5169e95700 -1 mgr.server reply reply (4) Interrupted system call no change 2024-02-09T17:28:03.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:03 smithi099 ceph-mon[39420]: pgmap v507: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 2.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s; 11571/136875 objects degraded (8.454%) 2024-02-09T17:28:03.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:03 smithi179 ceph-mon[43975]: pgmap v507: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 2.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s; 11571/136875 objects degraded (8.454%) 2024-02-09T17:28:03.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:03 smithi162 ceph-mon[44304]: pgmap v507: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 2.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s; 11571/136875 objects degraded (8.454%) 2024-02-09T17:28:04.564 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:28:04.587 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:28:04.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:04.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:28:04.692 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:28:04.725 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:04.751 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:28:04.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:04.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:28:04.855 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:28:04.860 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:04 smithi099 ceph-mon[39420]: from='client.14961 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:28:04.889 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:04 smithi179 ceph-mon[43975]: from='client.14961 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:28:04.891 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:04.915 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:28:04.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:04.918 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:28:04.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:04 smithi162 ceph-mon[44304]: from='client.14961 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:28:05.019 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:28:05.053 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:05.078 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:28:05.081 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:05.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:28:05.183 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:28:05.211 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:05.237 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:28:05.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:05.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:28:05.345 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:28:05.374 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:05.394 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:28:05.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:05.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:28:05.442 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:28:05.443 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [], 2024-02-09T17:28:05.443 INFO:teuthology.orchestra.run.smithi099.stdout: "complaint_time": 30, 2024-02-09T17:28:05.443 INFO:teuthology.orchestra.run.smithi099.stdout: "num_blocked_ops": 0 2024-02-09T17:28:05.443 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:28:05.497 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:28:05.580 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:05.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:05 smithi162 ceph-mon[44304]: pgmap v508: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 891 op/s; 11017/131985 objects degraded (8.347%) 2024-02-09T17:28:05.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:05 smithi099 ceph-mon[39420]: pgmap v508: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 891 op/s; 11017/131985 objects degraded (8.347%) 2024-02-09T17:28:05.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:05 smithi179 ceph-mon[43975]: pgmap v508: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 891 op/s; 11017/131985 objects degraded (8.347%) 2024-02-09T17:28:06.073 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:28:06.081 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:06.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:28:06.182 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:28:06.214 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:06.242 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:28:06.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:06.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:28:06.346 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:28:06.378 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:06.402 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:28:06.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:06.405 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:28:06.506 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:28:06.534 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:06.557 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:28:06.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:06.560 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:28:06.599 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset noscrub 2024-02-09T17:28:06.606 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:06 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 11017/131985 objects degraded (8.347%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:28:06.660 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:28:06.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:06 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 11017/131985 objects degraded (8.347%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:28:06.690 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:06.714 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:28:06.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:06.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:28:06.722 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-09T17:28:06.817 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:28:06.868 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:06 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 11017/131985 objects degraded (8.347%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:28:06.903 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:07.292 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:28:07.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:07.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:28:07.359 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:28:07.395 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:28:07.402 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:28:07.454 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:07.674 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:28:07.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:07.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:28:07.785 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:28:07.818 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:07.840 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:28:07.845 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:07.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:28:07.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:07 smithi179 ceph-mon[43975]: pgmap v509: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 2.6 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 892 op/s; 10444/125388 objects degraded (8.329%) 2024-02-09T17:28:07.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:07 smithi162 ceph-mon[44304]: pgmap v509: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 2.6 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 892 op/s; 10444/125388 objects degraded (8.329%) 2024-02-09T17:28:07.946 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:28:07.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:07 smithi099 ceph-mon[39420]: pgmap v509: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 2.6 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 892 op/s; 10444/125388 objects degraded (8.329%) 2024-02-09T17:28:08.027 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:08.119 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:28:08.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:08.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:28:08.136 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:28:08.173 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:28:08.209 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:28:08.232 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:28:08.236 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:28:08.260 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:08.276 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:28:08.291 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:28:08.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:08.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:28:08.312 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:28:08.344 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:28:08.373 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:28:08.394 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:28:08.397 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:28:08.423 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:08.441 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:28:08.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:08.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:28:08.545 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:28:08.577 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:08.596 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:28:08.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:08.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:28:08.661 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:28:08.700 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:28:08.702 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:28:08.702 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:28:08.702 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:28:08.702 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:28:08.702 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:28:08.702 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:28:08.702 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:28:08.702 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:28:08.702 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:28:08.703 INFO:teuthology.orchestra.run.smithi099.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-02-09T17:28:08.703 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:28:08.703 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:28:08.703 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:28:08.703 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:08 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1863527382' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:28:08.755 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:08.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:08 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1863527382' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:28:08.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:08 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1863527382' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:28:09.259 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [8] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11] 2024-02-09T17:28:09.259 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:28:09.259 INFO:tasks.thrashosds.thrasher:Reviving osd 8 2024-02-09T17:28:09.259 INFO:tasks.cephadm.osd.8:Restarting osd.8 (starting--it wasn't running)... 2024-02-09T17:28:09.259 DEBUG:teuthology.orchestra.run.smithi179:osd.8> sudo journalctl -f -n 0 -u ceph-a69c870a-c76e-11ee-95b7-87774f69a715@osd.8.service 2024-02-09T17:28:09.262 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl start ceph-a69c870a-c76e-11ee-95b7-87774f69a715@osd.8 2024-02-09T17:28:09.294 INFO:journalctl@ceph.osd.8.smithi179.stdout:-- Logs begin at Fri 2024-02-09 17:04:31 UTC. -- 2024-02-09T17:28:09.640 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 17:28:09 smithi179 systemd[1]: Starting Ceph osd.8 for a69c870a-c76e-11ee-95b7-87774f69a715... 2024-02-09T17:28:09.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:09 smithi179 ceph-mon[43975]: pgmap v510: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 2.5 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 892 op/s; 10444/123771 objects degraded (8.438%) 2024-02-09T17:28:09.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:09 smithi162 ceph-mon[44304]: pgmap v510: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 2.5 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 892 op/s; 10444/123771 objects degraded (8.438%) 2024-02-09T17:28:09.998 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:09 smithi099 ceph-mon[39420]: pgmap v510: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 2.5 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 892 op/s; 10444/123771 objects degraded (8.438%) 2024-02-09T17:28:10.060 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:28:10.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:10.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:28:10.169 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:28:10.205 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:10.226 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:28:10.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:10.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:28:10.330 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:28:10.379 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:10.917 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 17:28:10 smithi179 bash[87008]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 2024-02-09T17:28:10.917 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 17:28:10 smithi179 bash[87008]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-8 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-02-09T17:28:10.917 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 17:28:10 smithi179 bash[87008]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-02-09T17:28:10.917 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 17:28:10 smithi179 bash[87008]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-09T17:28:10.917 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 17:28:10 smithi179 bash[87008]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-8/block 2024-02-09T17:28:10.917 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 17:28:10 smithi179 bash[87008]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 2024-02-09T17:28:10.917 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 17:28:10 smithi179 bash[87008]: --> ceph-volume raw activate successful for osd ID: 8 2024-02-09T17:28:11.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:11 smithi179 ceph-mon[43975]: pgmap v511: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 2.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 882 op/s; 9596/115560 objects degraded (8.304%) 2024-02-09T17:28:11.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:11 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 9596/115560 objects degraded (8.304%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:28:11.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:11 smithi162 ceph-mon[44304]: pgmap v511: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 2.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 882 op/s; 9596/115560 objects degraded (8.304%) 2024-02-09T17:28:11.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:11 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 9596/115560 objects degraded (8.304%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:28:11.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:11 smithi099 ceph-mon[39420]: pgmap v511: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 2.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 882 op/s; 9596/115560 objects degraded (8.304%) 2024-02-09T17:28:11.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:11 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 9596/115560 objects degraded (8.304%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:28:12.169 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.8 dump_ops_in_flight 2024-02-09T17:28:12.230 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-09T17:28:12.231 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-09T17:28:12.267 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:28:12.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:12.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:28:12.377 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:28:12.410 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 17:28:12 smithi179 ceph-osd[87279]: -- 172.21.15.179:0/3465922783 <== mon.1 v2:172.21.15.179:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55fdf716f1e0 con 0x55fdf7140c00 2024-02-09T17:28:12.410 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 17:28:12 smithi179 bash[87243]: bf1f3830c374d01610b5dc3b737addf5ee6b46c1c81339998cd92f121e767f12 2024-02-09T17:28:12.410 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 17:28:12 smithi179 systemd[1]: Started Ceph osd.8 for a69c870a-c76e-11ee-95b7-87774f69a715. 2024-02-09T17:28:12.414 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:12.448 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:28:12.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:12.459 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:28:12.543 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:28:12.560 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:28:12.611 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:12.667 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 17:28:12 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T17:28:12.587+0000 7f2f1e7807c0 -1 Falling back to public interface 2024-02-09T17:28:12.680 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is unset 2024-02-09T17:28:12.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:12 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4012679081' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:28:12.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:12 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4012679081' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:28:12.984 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:28:12.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:12.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:28:12.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:12 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4012679081' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:28:13.094 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:28:13.122 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:13.146 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:28:13.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:13.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:28:13.249 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:28:13.281 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:13.304 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:28:13.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:13.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:28:13.411 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:28:13.458 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:13.489 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:28:13.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:13.495 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:28:13.596 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:28:13.627 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:13.650 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:28:13.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:13.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:28:13.753 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:28:13.807 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:13.848 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:28:13.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:13.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:28:13.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:13 smithi179 ceph-mon[43975]: pgmap v512: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 2.3 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 847 op/s; 9320/110850 objects degraded (8.408%) 2024-02-09T17:28:13.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:13 smithi179 ceph-mon[43975]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:28:13.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:13 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4012679081' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:28:13.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:13 smithi179 ceph-mon[43975]: osdmap e169: 12 total, 11 up, 12 in 2024-02-09T17:28:13.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:13 smithi162 ceph-mon[44304]: pgmap v512: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 2.3 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 847 op/s; 9320/110850 objects degraded (8.408%) 2024-02-09T17:28:13.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:13 smithi162 ceph-mon[44304]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:28:13.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:13 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4012679081' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:28:13.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:13 smithi162 ceph-mon[44304]: osdmap e169: 12 total, 11 up, 12 in 2024-02-09T17:28:13.956 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:28:13.989 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:13.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:13 smithi099 ceph-mon[39420]: pgmap v512: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 2.3 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 847 op/s; 9320/110850 objects degraded (8.408%) 2024-02-09T17:28:13.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:13 smithi099 ceph-mon[39420]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:28:13.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:13 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4012679081' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:28:13.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:13 smithi099 ceph-mon[39420]: osdmap e169: 12 total, 11 up, 12 in 2024-02-09T17:28:14.019 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:28:14.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:14.025 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:28:14.126 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:28:14.162 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:14.232 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:28:14.366 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:28:14.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:14.370 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:28:14.413 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:28:14.413 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:28:14.471 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:28:14.498 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:14.517 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:28:14.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:14.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:28:14.621 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:28:14.676 INFO:teuthology.orchestra.run.smithi179.stderr:no valid command found; 10 closest matches: 2024-02-09T17:28:14.676 INFO:teuthology.orchestra.run.smithi179.stderr:0 2024-02-09T17:28:14.676 INFO:teuthology.orchestra.run.smithi179.stderr:1 2024-02-09T17:28:14.676 INFO:teuthology.orchestra.run.smithi179.stderr:2 2024-02-09T17:28:14.677 INFO:teuthology.orchestra.run.smithi179.stderr:abort 2024-02-09T17:28:14.677 INFO:teuthology.orchestra.run.smithi179.stderr:assert 2024-02-09T17:28:14.677 INFO:teuthology.orchestra.run.smithi179.stderr:bluestore allocator dump block 2024-02-09T17:28:14.677 INFO:teuthology.orchestra.run.smithi179.stderr:bluestore allocator fragmentation block 2024-02-09T17:28:14.677 INFO:teuthology.orchestra.run.smithi179.stderr:bluestore allocator fragmentation histogram block [] [] 2024-02-09T17:28:14.677 INFO:teuthology.orchestra.run.smithi179.stderr:bluestore allocator score block 2024-02-09T17:28:14.677 INFO:teuthology.orchestra.run.smithi179.stderr:config diff 2024-02-09T17:28:14.677 INFO:teuthology.orchestra.run.smithi179.stderr:admin_socket: invalid command 2024-02-09T17:28:14.678 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:14.719 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:28:14.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:14.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:28:14.825 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:28:14.900 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:15.384 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:28:15.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:15.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:28:15.410 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-09T17:28:15.420 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:28:15.439 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-09T17:28:15.439 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-8, ['dump_ops_in_flight'] 2024-02-09T17:28:15.495 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:28:15.559 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:15.619 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:28:15.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:15.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:28:15.723 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:28:15.748 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-09T17:28:15.750 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:15.772 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:28:15.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:15.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:28:15.876 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:28:15.929 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:15 smithi099 ceph-mon[39420]: 2.9 scrub starts 2024-02-09T17:28:15.929 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:15 smithi099 ceph-mon[39420]: pgmap v514: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 2.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 870 op/s; 8764/105831 objects degraded (8.281%) 2024-02-09T17:28:15.929 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:15 smithi099 ceph-mon[39420]: 2.9 scrub ok 2024-02-09T17:28:15.935 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:16.081 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:28:16.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:16.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:28:16.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:15 smithi179 ceph-mon[43975]: 2.9 scrub starts 2024-02-09T17:28:16.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:15 smithi179 ceph-mon[43975]: pgmap v514: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 2.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 870 op/s; 8764/105831 objects degraded (8.281%) 2024-02-09T17:28:16.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:15 smithi179 ceph-mon[43975]: 2.9 scrub ok 2024-02-09T17:28:16.173 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 17:28:15 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T17:28:15.777+0000 7f2f1e7807c0 -1 osd.8 145 log_to_monitors true 2024-02-09T17:28:16.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:15 smithi162 ceph-mon[44304]: 2.9 scrub starts 2024-02-09T17:28:16.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:15 smithi162 ceph-mon[44304]: pgmap v514: 17 pgs: 4 active+undersized+degraded, 13 active+clean; 2.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 870 op/s; 8764/105831 objects degraded (8.281%) 2024-02-09T17:28:16.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:15 smithi162 ceph-mon[44304]: 2.9 scrub ok 2024-02-09T17:28:16.187 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:28:16.232 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:16.998 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:16 smithi099 ceph-mon[39420]: from='osd.8 [v2:172.21.15.179:6800/2558294495,v1:172.21.15.179:6801/2558294495]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-02-09T17:28:16.998 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:16 smithi099 ceph-mon[39420]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-02-09T17:28:16.998 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:16 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 8764/105831 objects degraded (8.281%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:28:17.093 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:28:17.152 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:28:17.171 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 17:28:16 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T17:28:16.713+0000 7f2f0bbd7700 -1 osd.8 145 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-09T17:28:17.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:16 smithi179 ceph-mon[43975]: from='osd.8 [v2:172.21.15.179:6800/2558294495,v1:172.21.15.179:6801/2558294495]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-02-09T17:28:17.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:16 smithi179 ceph-mon[43975]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-02-09T17:28:17.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:16 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 8764/105831 objects degraded (8.281%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:28:17.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:16 smithi162 ceph-mon[44304]: from='osd.8 [v2:172.21.15.179:6800/2558294495,v1:172.21.15.179:6801/2558294495]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-02-09T17:28:17.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:16 smithi162 ceph-mon[44304]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-02-09T17:28:17.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:16 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 8764/105831 objects degraded (8.281%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:28:17.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:17 smithi099 ceph-mon[39420]: pgmap v515: 17 pgs: 1 active+clean+scrubbing, 4 active+undersized+degraded, 12 active+clean; 2.0 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 869 op/s; 8181/99276 objects degraded (8.241%) 2024-02-09T17:28:17.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:17 smithi099 ceph-mon[39420]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-02-09T17:28:17.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:17 smithi099 ceph-mon[39420]: from='osd.8 [v2:172.21.15.179:6800/2558294495,v1:172.21.15.179:6801/2558294495]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-02-09T17:28:17.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:17 smithi099 ceph-mon[39420]: osdmap e170: 12 total, 11 up, 12 in 2024-02-09T17:28:17.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:17 smithi099 ceph-mon[39420]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-02-09T17:28:18.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:17 smithi179 ceph-mon[43975]: pgmap v515: 17 pgs: 1 active+clean+scrubbing, 4 active+undersized+degraded, 12 active+clean; 2.0 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 869 op/s; 8181/99276 objects degraded (8.241%) 2024-02-09T17:28:18.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:17 smithi179 ceph-mon[43975]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-02-09T17:28:18.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:17 smithi179 ceph-mon[43975]: from='osd.8 [v2:172.21.15.179:6800/2558294495,v1:172.21.15.179:6801/2558294495]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-02-09T17:28:18.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:17 smithi179 ceph-mon[43975]: osdmap e170: 12 total, 11 up, 12 in 2024-02-09T17:28:18.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:17 smithi179 ceph-mon[43975]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-02-09T17:28:18.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:17 smithi162 ceph-mon[44304]: pgmap v515: 17 pgs: 1 active+clean+scrubbing, 4 active+undersized+degraded, 12 active+clean; 2.0 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 869 op/s; 8181/99276 objects degraded (8.241%) 2024-02-09T17:28:18.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:17 smithi162 ceph-mon[44304]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-02-09T17:28:18.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:17 smithi162 ceph-mon[44304]: from='osd.8 [v2:172.21.15.179:6800/2558294495,v1:172.21.15.179:6801/2558294495]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-02-09T17:28:18.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:17 smithi162 ceph-mon[44304]: osdmap e170: 12 total, 11 up, 12 in 2024-02-09T17:28:18.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:17 smithi162 ceph-mon[44304]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-02-09T17:28:18.430 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:28:18.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:18.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:28:18.534 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:28:18.571 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:18.594 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:28:18.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:18.597 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:28:18.698 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:28:18.730 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:18.752 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:28:18.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:18.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:28:18.856 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:28:18.886 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:18.907 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:28:18.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:18.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:28:18.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:18 smithi179 ceph-mon[43975]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-09T17:28:18.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:18 smithi179 ceph-mon[43975]: osd.8 [v2:172.21.15.179:6800/2558294495,v1:172.21.15.179:6801/2558294495] boot 2024-02-09T17:28:18.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:18 smithi179 ceph-mon[43975]: osdmap e171: 12 total, 12 up, 12 in 2024-02-09T17:28:18.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:18 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-09T17:28:18.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:18 smithi099 ceph-mon[39420]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-09T17:28:18.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:18 smithi099 ceph-mon[39420]: osd.8 [v2:172.21.15.179:6800/2558294495,v1:172.21.15.179:6801/2558294495] boot 2024-02-09T17:28:18.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:18 smithi099 ceph-mon[39420]: osdmap e171: 12 total, 12 up, 12 in 2024-02-09T17:28:18.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:18 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-09T17:28:19.011 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:28:19.051 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:28:19.051 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:28:19.068 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:28:19.083 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:19.088 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:28:19.104 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:28:19.106 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:19.107 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:28:19.108 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:28:19.128 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:28:19.148 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:28:19.167 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:28:19.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:18 smithi162 ceph-mon[44304]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-09T17:28:19.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:18 smithi162 ceph-mon[44304]: osd.8 [v2:172.21.15.179:6800/2558294495,v1:172.21.15.179:6801/2558294495] boot 2024-02-09T17:28:19.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:18 smithi162 ceph-mon[44304]: osdmap e171: 12 total, 12 up, 12 in 2024-02-09T17:28:19.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:18 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-09T17:28:19.186 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:28:19.206 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:28:19.208 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:28:19.229 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:28:19.237 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:19.250 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:28:19.252 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:28:19.252 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:28:19.252 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:28:19.252 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:28:19.252 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:28:19.252 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:28:19.252 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:28:19.252 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:28:19.252 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-02-09T17:28:19.253 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:28:19.253 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:28:19.253 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:28:19.834 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:19 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:28:19.835 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:19 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:28:19.835 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:19 smithi099 ceph-mon[39420]: pgmap v518: 17 pgs: 1 active+clean+scrubbing, 4 active+undersized+degraded, 12 active+clean; 2.0 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 732 op/s; 8181/97635 objects degraded (8.379%) 2024-02-09T17:28:19.835 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:19 smithi099 ceph-mon[39420]: osdmap e172: 12 total, 12 up, 12 in 2024-02-09T17:28:19.835 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:19 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2512327242' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:28:20.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:19 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:28:20.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:19 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:28:20.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:19 smithi179 ceph-mon[43975]: pgmap v518: 17 pgs: 1 active+clean+scrubbing, 4 active+undersized+degraded, 12 active+clean; 2.0 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 732 op/s; 8181/97635 objects degraded (8.379%) 2024-02-09T17:28:20.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:19 smithi179 ceph-mon[43975]: osdmap e172: 12 total, 12 up, 12 in 2024-02-09T17:28:20.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:19 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2512327242' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:28:20.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:19 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:28:20.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:19 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:28:20.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:19 smithi162 ceph-mon[44304]: pgmap v518: 17 pgs: 1 active+clean+scrubbing, 4 active+undersized+degraded, 12 active+clean; 2.0 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 732 op/s; 8181/97635 objects degraded (8.379%) 2024-02-09T17:28:20.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:19 smithi162 ceph-mon[44304]: osdmap e172: 12 total, 12 up, 12 in 2024-02-09T17:28:20.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:19 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2512327242' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:28:20.190 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:28:20.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:20.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:28:20.298 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:28:20.337 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:20.361 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:28:20.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:20.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:28:20.440 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.8 dump_ops_in_flight 2024-02-09T17:28:20.465 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:28:20.494 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:20.520 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:28:20.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:20.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:28:20.625 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:28:20.654 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:20.678 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:28:20.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:20.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:28:20.780 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:20 smithi179 ceph-mon[43975]: osdmap e173: 12 total, 12 up, 12 in 2024-02-09T17:28:20.782 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:28:20.811 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:20.835 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:28:20.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:20.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:28:20.844 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:28:20.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:20 smithi162 ceph-mon[44304]: osdmap e173: 12 total, 12 up, 12 in 2024-02-09T17:28:20.939 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:28:20.973 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:20.997 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:28:20.998 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:20 smithi099 ceph-mon[39420]: osdmap e173: 12 total, 12 up, 12 in 2024-02-09T17:28:21.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:21.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:28:21.102 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:28:21.175 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:21.228 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:28:21.228 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [ 2024-02-09T17:28:21.228 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.228 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:236983 2.6 2:6404cc5c:::benchmark_data_smithi179_77138_object103575:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-02-09T17:28:21.228 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:28:20.951273+0000", 2024-02-09T17:28:21.228 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.025998897999999999, 2024-02-09T17:28:21.228 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.026031985000000001, 2024-02-09T17:28:21.228 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:28:21.228 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "waiting for sub ops", 2024-02-09T17:28:21.228 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:28:21.229 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:28:21.229 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:28:21.229 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 236983 2024-02-09T17:28:21.229 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.229 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:28:21.229 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.229 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:28:21.229 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.951273+0000", 2024-02-09T17:28:21.229 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:28:21.229 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.229 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.229 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:28:21.229 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.951273+0000", 2024-02-09T17:28:21.230 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:28:21.230 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.230 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.230 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:28:21.230 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.951274+0000", 2024-02-09T17:28:21.230 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4529999999999999e-06 2024-02-09T17:28:21.230 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.230 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.230 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:28:21.230 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.951280+0000", 2024-02-09T17:28:21.230 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.066e-06 2024-02-09T17:28:21.230 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.231 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.231 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:28:21.231 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.951293+0000", 2024-02-09T17:28:21.231 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2561999999999999e-05 2024-02-09T17:28:21.231 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.231 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.231 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:28:21.231 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.951302+0000", 2024-02-09T17:28:21.231 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.0180000000000007e-06 2024-02-09T17:28:21.231 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.231 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.231 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:28:21.232 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.952984+0000", 2024-02-09T17:28:21.232 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.001681971 2024-02-09T17:28:21.232 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.232 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.232 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:28:21.232 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.953116+0000", 2024-02-09T17:28:21.232 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013187600000000001 2024-02-09T17:28:21.232 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.232 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.232 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,8", 2024-02-09T17:28:21.232 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.953252+0000", 2024-02-09T17:28:21.232 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013674 2024-02-09T17:28:21.232 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.233 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.233 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:28:21.233 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.958172+0000", 2024-02-09T17:28:21.233 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0049191699999999996 2024-02-09T17:28:21.233 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.233 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.233 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:28:21.233 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.963671+0000", 2024-02-09T17:28:21.233 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0054989160000000004 2024-02-09T17:28:21.233 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:28:21.233 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:28:21.233 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:28:21.233 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.234 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.234 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:236992 2.4 2:27fe8677:::benchmark_data_smithi179_77138_object103584:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-02-09T17:28:21.234 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:28:20.963622+0000", 2024-02-09T17:28:21.234 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.013649942, 2024-02-09T17:28:21.234 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.013777879999999999, 2024-02-09T17:28:21.234 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:28:21.234 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "waiting for sub ops", 2024-02-09T17:28:21.234 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:28:21.234 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:28:21.234 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:28:21.234 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 236992 2024-02-09T17:28:21.234 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.234 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:28:21.235 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.235 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:28:21.235 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.963622+0000", 2024-02-09T17:28:21.235 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:28:21.235 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.235 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.235 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:28:21.235 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.963622+0000", 2024-02-09T17:28:21.235 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:28:21.235 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.235 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.235 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:28:21.235 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.963623+0000", 2024-02-09T17:28:21.236 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.736e-06 2024-02-09T17:28:21.236 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.236 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.236 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:28:21.236 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.963631+0000", 2024-02-09T17:28:21.236 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.0320000000000003e-06 2024-02-09T17:28:21.236 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.236 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.236 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:28:21.236 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.963651+0000", 2024-02-09T17:28:21.236 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.9896000000000001e-05 2024-02-09T17:28:21.236 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.236 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.236 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:28:21.237 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.963659+0000", 2024-02-09T17:28:21.237 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.9079999999999995e-06 2024-02-09T17:28:21.237 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.237 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.237 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:28:21.237 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.963734+0000", 2024-02-09T17:28:21.237 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.4579000000000005e-05 2024-02-09T17:28:21.237 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.237 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.237 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:28:21.237 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.963814+0000", 2024-02-09T17:28:21.237 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.9883999999999996e-05 2024-02-09T17:28:21.237 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.237 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.238 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,10", 2024-02-09T17:28:21.238 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.963883+0000", 2024-02-09T17:28:21.238 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.9207000000000003e-05 2024-02-09T17:28:21.238 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.238 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.238 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:28:21.238 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.970335+0000", 2024-02-09T17:28:21.238 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0064524539999999998 2024-02-09T17:28:21.238 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.238 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.238 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:28:21.238 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.971427+0000", 2024-02-09T17:28:21.238 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.001091978 2024-02-09T17:28:21.238 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:28:21.239 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:28:21.239 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:28:21.239 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.239 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.239 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:236995 2.6 2:6d5785a6:::benchmark_data_smithi179_77138_object103587:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-02-09T17:28:21.239 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:28:20.965263+0000", 2024-02-09T17:28:21.239 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.012008591000000001, 2024-02-09T17:28:21.239 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.012226468000000001, 2024-02-09T17:28:21.239 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:28:21.239 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "waiting for sub ops", 2024-02-09T17:28:21.239 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:28:21.239 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:28:21.240 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:28:21.240 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 236995 2024-02-09T17:28:21.240 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.240 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:28:21.240 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.240 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:28:21.240 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.965263+0000", 2024-02-09T17:28:21.241 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:28:21.241 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.241 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.241 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:28:21.241 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.965263+0000", 2024-02-09T17:28:21.241 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:28:21.241 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.241 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.241 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:28:21.241 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.965264+0000", 2024-02-09T17:28:21.241 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.7999999999999993e-07 2024-02-09T17:28:21.242 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.242 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.242 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:28:21.242 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.965269+0000", 2024-02-09T17:28:21.242 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.6280000000000004e-06 2024-02-09T17:28:21.242 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.242 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.242 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:28:21.242 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.965280+0000", 2024-02-09T17:28:21.243 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1313e-05 2024-02-09T17:28:21.243 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.243 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.243 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:28:21.243 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.965287+0000", 2024-02-09T17:28:21.243 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.3270000000000003e-06 2024-02-09T17:28:21.244 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.244 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.244 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:28:21.244 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.968041+0000", 2024-02-09T17:28:21.244 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0027536489999999999 2024-02-09T17:28:21.245 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.245 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.245 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:28:21.245 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.968181+0000", 2024-02-09T17:28:21.245 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000139734 2024-02-09T17:28:21.245 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.245 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.245 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,8", 2024-02-09T17:28:21.245 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.968334+0000", 2024-02-09T17:28:21.246 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00015309500000000001 2024-02-09T17:28:21.246 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:28:21.246 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:28:21.246 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:28:21.246 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.246 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.246 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:236996 2.9 2:92d1274c:::benchmark_data_smithi179_77138_object103588:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-02-09T17:28:21.246 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:28:20.971449+0000", 2024-02-09T17:28:21.246 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.005822514, 2024-02-09T17:28:21.246 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0061105869999999998, 2024-02-09T17:28:21.247 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:28:21.247 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "waiting for sub ops", 2024-02-09T17:28:21.247 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:28:21.247 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:28:21.247 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:28:21.247 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 236996 2024-02-09T17:28:21.248 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.248 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:28:21.248 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.248 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:28:21.248 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.971449+0000", 2024-02-09T17:28:21.248 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:28:21.248 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.248 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.248 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:28:21.248 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.971449+0000", 2024-02-09T17:28:21.248 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:28:21.248 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.248 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.249 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:28:21.249 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.971451+0000", 2024-02-09T17:28:21.249 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4720000000000001e-06 2024-02-09T17:28:21.249 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.249 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.249 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:28:21.249 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.971457+0000", 2024-02-09T17:28:21.249 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.0419999999999999e-06 2024-02-09T17:28:21.249 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.249 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.249 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:28:21.249 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.971470+0000", 2024-02-09T17:28:21.249 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3661e-05 2024-02-09T17:28:21.249 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.249 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.249 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:28:21.250 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.971479+0000", 2024-02-09T17:28:21.250 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.9600000000000006e-06 2024-02-09T17:28:21.250 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.250 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.250 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:28:21.250 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.971608+0000", 2024-02-09T17:28:21.250 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012917399999999999 2024-02-09T17:28:21.250 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.250 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.250 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:28:21.250 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.971739+0000", 2024-02-09T17:28:21.250 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013078699999999999 2024-02-09T17:28:21.250 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.250 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.251 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,7", 2024-02-09T17:28:21.251 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.971864+0000", 2024-02-09T17:28:21.251 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000124898 2024-02-09T17:28:21.251 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.251 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.251 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:28:21.251 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.977332+0000", 2024-02-09T17:28:21.251 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0054680429999999997 2024-02-09T17:28:21.251 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:28:21.251 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:28:21.251 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:28:21.251 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.251 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.251 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:236999 2.6 2:6db32926:::benchmark_data_smithi179_77138_object103591:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-02-09T17:28:21.251 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:28:20.971503+0000", 2024-02-09T17:28:21.252 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.0057687729999999996, 2024-02-09T17:28:21.252 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0061423950000000001, 2024-02-09T17:28:21.252 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:28:21.252 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "waiting for sub ops", 2024-02-09T17:28:21.252 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:28:21.252 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:28:21.252 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:28:21.252 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 236999 2024-02-09T17:28:21.252 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.252 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:28:21.252 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.252 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:28:21.252 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.971503+0000", 2024-02-09T17:28:21.253 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:28:21.253 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.253 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.253 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:28:21.253 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.971503+0000", 2024-02-09T17:28:21.253 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:28:21.253 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.253 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.253 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:28:21.253 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.971503+0000", 2024-02-09T17:28:21.253 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.44e-07 2024-02-09T17:28:21.253 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.253 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.253 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:28:21.254 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.971506+0000", 2024-02-09T17:28:21.254 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.897e-06 2024-02-09T17:28:21.254 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.254 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.254 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:28:21.254 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.971514+0000", 2024-02-09T17:28:21.254 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.6699999999999994e-06 2024-02-09T17:28:21.254 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.254 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.254 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:28:21.254 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.971520+0000", 2024-02-09T17:28:21.254 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.1059999999999996e-06 2024-02-09T17:28:21.254 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.255 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.255 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:28:21.255 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.972230+0000", 2024-02-09T17:28:21.255 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00070941099999999996 2024-02-09T17:28:21.255 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.255 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.255 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:28:21.255 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.972349+0000", 2024-02-09T17:28:21.255 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000119856 2024-02-09T17:28:21.255 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.255 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.255 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,8", 2024-02-09T17:28:21.255 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.972498+0000", 2024-02-09T17:28:21.255 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014875700000000001 2024-02-09T17:28:21.255 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:21.256 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:21.256 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:28:21.256 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:20.977557+0000", 2024-02-09T17:28:21.256 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.005059142 2024-02-09T17:28:21.256 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:28:21.256 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:28:21.256 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:28:21.256 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:28:21.256 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T17:28:21.256 INFO:teuthology.orchestra.run.smithi099.stdout: "num_ops": 5 2024-02-09T17:28:21.256 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:28:21.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:21 smithi099 ceph-mon[39420]: pgmap v521: 17 pgs: 1 remapped+peering, 1 activating+undersized+degraded+remapped, 2 active+undersized+degraded, 13 active+clean; 1.9 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 492 op/s; 9468/93363 objects degraded (10.141%) 2024-02-09T17:28:21.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:21 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 9468/93363 objects degraded (10.141%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-09T17:28:22.126 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-09T17:28:22.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:21 smithi179 ceph-mon[43975]: pgmap v521: 17 pgs: 1 remapped+peering, 1 activating+undersized+degraded+remapped, 2 active+undersized+degraded, 13 active+clean; 1.9 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 492 op/s; 9468/93363 objects degraded (10.141%) 2024-02-09T17:28:22.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:21 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 9468/93363 objects degraded (10.141%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-09T17:28:22.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:21 smithi162 ceph-mon[44304]: pgmap v521: 17 pgs: 1 remapped+peering, 1 activating+undersized+degraded+remapped, 2 active+undersized+degraded, 13 active+clean; 1.9 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 492 op/s; 9468/93363 objects degraded (10.141%) 2024-02-09T17:28:22.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:21 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 9468/93363 objects degraded (10.141%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-09T17:28:22.219 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:28:22.781 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:28:22.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:22.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:28:22.874 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:28:22.887 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:28:22.892 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:28:22.994 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:23.155 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:28:23.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:23.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:28:23.264 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:28:23.294 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:23.315 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:28:23.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:23.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:28:23.419 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:28:23.450 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:23.473 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:28:23.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:23.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:28:23.578 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:28:23.606 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:23.628 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:28:23.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:23.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:28:23.736 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:28:23.795 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:23.817 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is unset 2024-02-09T17:28:23.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:23 smithi162 ceph-mon[44304]: pgmap v522: 17 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 1 activating+undersized+degraded+remapped, 1 active+undersized+degraded, 13 active+clean; 1.9 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 387 op/s; 7392/90930 objects degraded (8.129%); 0 B/s, 8 objects/s recovering 2024-02-09T17:28:23.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:23 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:28:23.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:23 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:28:23.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:23 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4130593076' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:28:23.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:23 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:28:24.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:23 smithi179 ceph-mon[43975]: pgmap v522: 17 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 1 activating+undersized+degraded+remapped, 1 active+undersized+degraded, 13 active+clean; 1.9 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 387 op/s; 7392/90930 objects degraded (8.129%); 0 B/s, 8 objects/s recovering 2024-02-09T17:28:24.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:23 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:28:24.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:23 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:28:24.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:23 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4130593076' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:28:24.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:23 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:28:24.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:23 smithi099 ceph-mon[39420]: pgmap v522: 17 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 1 activating+undersized+degraded+remapped, 1 active+undersized+degraded, 13 active+clean; 1.9 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 387 op/s; 7392/90930 objects degraded (8.129%); 0 B/s, 8 objects/s recovering 2024-02-09T17:28:24.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:23 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:28:24.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:23 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:28:24.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:23 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4130593076' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:28:24.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:23 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:28:24.383 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:28:24.399 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:24.400 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:28:24.501 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:28:24.535 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:24.558 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:28:24.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:24.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:28:24.663 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:28:24.710 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:24.845 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:28:24.848 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:24.848 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:28:24.949 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:28:24.978 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:25.002 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:28:25.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:25.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:28:25.111 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:28:25.146 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:25.146 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:24 smithi162 ceph-mon[44304]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:28:25.146 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:24 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:28:25.146 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:24 smithi162 ceph-mon[44304]: osdmap e174: 12 total, 12 up, 12 in 2024-02-09T17:28:25.169 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:28:25.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:24 smithi179 ceph-mon[43975]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:28:25.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:24 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:28:25.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:24 smithi179 ceph-mon[43975]: osdmap e174: 12 total, 12 up, 12 in 2024-02-09T17:28:25.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:25.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:28:25.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:24 smithi099 ceph-mon[39420]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:28:25.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:24 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:28:25.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:24 smithi099 ceph-mon[39420]: osdmap e174: 12 total, 12 up, 12 in 2024-02-09T17:28:25.274 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:28:25.323 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:25.575 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:28:25.575 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-02-09T17:28:25.575 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 0 2024-02-09T17:28:25.575 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:28:25.720 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:28:25.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:25.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:28:25.829 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:28:25.849 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:25 smithi179 ceph-mon[43975]: pgmap v524: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 13 active+clean; 1.8 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 577 op/s; 10594/87240 objects degraded (12.144%); 0 B/s, 27 objects/s recovering 2024-02-09T17:28:25.849 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:25 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:28:25.849 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:25 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:28:25.883 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:25.885 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:25 smithi099 ceph-mon[39420]: pgmap v524: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 13 active+clean; 1.8 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 577 op/s; 10594/87240 objects degraded (12.144%); 0 B/s, 27 objects/s recovering 2024-02-09T17:28:25.885 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:25 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:28:25.886 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:25 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:28:26.166 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:28:26.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:26.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:28:26.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:25 smithi162 ceph-mon[44304]: pgmap v524: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 13 active+clean; 1.8 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 577 op/s; 10594/87240 objects degraded (12.144%); 0 B/s, 27 objects/s recovering 2024-02-09T17:28:26.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:25 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:28:26.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:25 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:28:26.275 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:28:26.324 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:26.437 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:28:26.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:26.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:28:26.551 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:28:26.608 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:26.639 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:28:26.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:26.649 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:28:26.750 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:28:26.822 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:26.878 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set noscrub 2024-02-09T17:28:26.911 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:28:26.918 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:26.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:28:26.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:26 smithi179 ceph-mon[43975]: osdmap e175: 12 total, 12 up, 12 in 2024-02-09T17:28:26.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:26 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 10594/87240 objects degraded (12.144%), 2 pgs degraded (PG_DEGRADED) 2024-02-09T17:28:27.020 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:28:27.156 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:26 smithi099 ceph-mon[39420]: osdmap e175: 12 total, 12 up, 12 in 2024-02-09T17:28:27.156 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:26 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 10594/87240 objects degraded (12.144%), 2 pgs degraded (PG_DEGRADED) 2024-02-09T17:28:27.167 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:27.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:26 smithi162 ceph-mon[44304]: osdmap e175: 12 total, 12 up, 12 in 2024-02-09T17:28:27.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:26 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 10594/87240 objects degraded (12.144%), 2 pgs degraded (PG_DEGRADED) 2024-02-09T17:28:27.247 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:28:27.247 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [], 2024-02-09T17:28:27.247 INFO:teuthology.orchestra.run.smithi099.stdout: "complaint_time": 30, 2024-02-09T17:28:27.247 INFO:teuthology.orchestra.run.smithi099.stdout: "num_blocked_ops": 0 2024-02-09T17:28:27.248 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:28:27.387 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:28:27.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:27.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:28:27.407 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:28:27.467 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:28:27.501 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:28:27.502 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:28:27.564 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:28:27.592 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:27.598 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:28:27.632 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:28:27.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:27.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:28:27.647 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:28:27.681 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:28:27.737 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:28:27.742 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:28:27.771 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:27.794 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:28:27.804 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:28:27.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:27.807 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:28:27.822 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:28:27.862 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:27 smithi099 ceph-mon[39420]: pgmap v526: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 1.7 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 608 op/s; 5327/82827 objects degraded (6.431%); 0 B/s, 50 objects/s recovering 2024-02-09T17:28:27.862 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:27 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1069960167' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:28:27.876 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:28:27.908 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:28:27.921 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:28:27.936 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:28:27.936 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:28:27.937 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:28:27.937 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:28:27.937 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:28:27.937 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:28:27.937 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:28:27.937 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:28:27.937 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:28:27.937 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:28:27.937 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:28:27.937 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:28:27.944 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:27.973 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:28:27.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:27.981 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:28:28.082 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:28:28.124 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:27 smithi179 ceph-mon[43975]: pgmap v526: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 1.7 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 608 op/s; 5327/82827 objects degraded (6.431%); 0 B/s, 50 objects/s recovering 2024-02-09T17:28:28.124 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:27 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1069960167' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:28:28.126 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:28.165 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:28:28.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:28.172 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:28:28.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:27 smithi162 ceph-mon[44304]: pgmap v526: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 1.7 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 608 op/s; 5327/82827 objects degraded (6.431%); 0 B/s, 50 objects/s recovering 2024-02-09T17:28:28.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:27 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1069960167' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:28:28.272 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:28:28.319 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-09T17:28:28.334 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:28.464 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:28:28.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:28.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:28:28.576 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:28:28.619 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:28.655 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:28:28.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:28.662 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:28:28.763 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:28:28.807 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:28.879 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:28 smithi179 ceph-mon[43975]: pgmap v527: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 1.6 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 518 op/s; 5327/80916 objects degraded (6.583%); 0 B/s, 73 objects/s recovering 2024-02-09T17:28:29.080 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:28:29.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:29.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:28:29.143 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:28:29.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:28 smithi162 ceph-mon[44304]: pgmap v527: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 1.6 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 518 op/s; 5327/80916 objects degraded (6.583%); 0 B/s, 73 objects/s recovering 2024-02-09T17:28:29.190 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:28:29.232 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:29.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:28 smithi099 ceph-mon[39420]: pgmap v527: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 1.6 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 518 op/s; 5327/80916 objects degraded (6.583%); 0 B/s, 73 objects/s recovering 2024-02-09T17:28:29.286 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:28:29.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:29.298 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:28:29.399 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:28:29.444 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:29.507 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:28:29.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:29.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:28:29.615 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:28:29.655 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:29.704 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:28:29.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:29.710 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:28:29.811 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:28:29.847 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:29.870 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:28:29.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:29.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:28:29.975 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:28:30.056 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:30.087 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:28:30.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:30 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:28:30.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:30 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:28:30.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:30 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:28:30.648 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:28:30.657 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:30.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:28:30.758 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:28:30.801 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:30.834 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:28:30.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:30.842 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:28:30.943 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:28:30.984 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:31.011 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:28:31.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:31.013 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:28:31.110 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:28:31.114 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:28:31.132 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:31 smithi179 ceph-mon[43975]: pgmap v528: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 1.5 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 683 op/s; 5327/74523 objects degraded (7.148%); 0 B/s, 118 objects/s recovering 2024-02-09T17:28:31.132 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:31 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:28:31.132 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:31 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:28:31.133 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:31 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:28:31.147 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:31.175 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:28:31.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:31.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:28:31.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:31 smithi099 ceph-mon[39420]: pgmap v528: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 1.5 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 683 op/s; 5327/74523 objects degraded (7.148%); 0 B/s, 118 objects/s recovering 2024-02-09T17:28:31.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:31 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:28:31.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:31 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:28:31.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:31 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:28:31.282 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:28:31.362 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:31.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:31 smithi162 ceph-mon[44304]: pgmap v528: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 1.5 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 683 op/s; 5327/74523 objects degraded (7.148%); 0 B/s, 118 objects/s recovering 2024-02-09T17:28:31.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:31 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:28:31.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:31 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:28:31.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:31 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:28:31.431 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] 2024-02-09T17:28:31.431 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:28:31.431 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-02-09T17:28:31.431 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph tell osd.0 'trim stale osdmaps' 2024-02-09T17:28:32.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:32 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 5327/74523 objects degraded (7.148%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:28:32.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:32 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 5327/74523 objects degraded (7.148%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:28:32.502 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:32 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 5327/74523 objects degraded (7.148%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:28:33.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:33 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:28:33.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:33 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:28:33.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:33 smithi162 ceph-mon[44304]: osdmap e176: 12 total, 12 up, 12 in 2024-02-09T17:28:33.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:33 smithi162 ceph-mon[44304]: pgmap v530: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 1.4 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 687 op/s; 5327/70725 objects degraded (7.532%); 0 B/s, 136 objects/s recovering 2024-02-09T17:28:33.502 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:33 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:28:33.502 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:33 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:28:33.502 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:33 smithi099 ceph-mon[39420]: osdmap e176: 12 total, 12 up, 12 in 2024-02-09T17:28:33.502 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:33 smithi099 ceph-mon[39420]: pgmap v530: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 1.4 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 687 op/s; 5327/70725 objects degraded (7.532%); 0 B/s, 136 objects/s recovering 2024-02-09T17:28:33.562 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:28:33.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:33.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:28:33.669 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:28:33.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:33 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:28:33.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:33 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:28:33.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:33 smithi179 ceph-mon[43975]: osdmap e176: 12 total, 12 up, 12 in 2024-02-09T17:28:33.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:33 smithi179 ceph-mon[43975]: pgmap v530: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 1.4 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 687 op/s; 5327/70725 objects degraded (7.532%); 0 B/s, 136 objects/s recovering 2024-02-09T17:28:33.708 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:34.708 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:28:34.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:34.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:28:34.817 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:28:34.852 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:34.874 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:28:34.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:34.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:28:34.979 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:28:35.020 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:35.058 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:28:35.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:35.067 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:28:35.168 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:28:35.211 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:35.256 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:28:35.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:35.262 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:28:35.363 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:28:35.411 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:35.449 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:28:35.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:35.455 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:28:35.556 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:28:35.605 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:35.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:35 smithi179 ceph-mon[43975]: pgmap v531: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 1.4 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 698 op/s; 5327/66690 objects degraded (7.988%); 0 B/s, 130 objects/s recovering 2024-02-09T17:28:35.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:35 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2497258784' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:28:35.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:35 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:28:35.896 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is set 2024-02-09T17:28:35.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:35 smithi162 ceph-mon[44304]: pgmap v531: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 1.4 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 698 op/s; 5327/66690 objects degraded (7.988%); 0 B/s, 130 objects/s recovering 2024-02-09T17:28:35.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:35 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2497258784' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:28:35.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:35 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:28:35.999 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:35 smithi099 ceph-mon[39420]: pgmap v531: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 1.4 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 698 op/s; 5327/66690 objects degraded (7.988%); 0 B/s, 130 objects/s recovering 2024-02-09T17:28:35.999 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:35 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2497258784' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:28:35.999 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:35 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:28:36.574 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:36 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:28:36.574 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:36 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:28:36.574 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:36 smithi099 ceph-mon[39420]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:28:36.574 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:36 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:28:36.575 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:36 smithi099 ceph-mon[39420]: osdmap e177: 12 total, 12 up, 12 in 2024-02-09T17:28:36.575 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:36 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 5327/66690 objects degraded (7.988%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:28:36.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:36 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:28:36.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:36 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:28:36.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:36 smithi179 ceph-mon[43975]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:28:36.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:36 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:28:36.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:36 smithi179 ceph-mon[43975]: osdmap e177: 12 total, 12 up, 12 in 2024-02-09T17:28:36.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:36 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 5327/66690 objects degraded (7.988%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:28:36.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:36 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:28:36.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:36 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:28:36.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:36 smithi162 ceph-mon[44304]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:28:36.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:36 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:28:36.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:36 smithi162 ceph-mon[44304]: osdmap e177: 12 total, 12 up, 12 in 2024-02-09T17:28:36.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:36 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 5327/66690 objects degraded (7.988%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:28:37.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:37 smithi179 ceph-mon[43975]: pgmap v533: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 1.3 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 791 op/s; 5327/61911 objects degraded (8.604%); 0 B/s, 130 objects/s recovering 2024-02-09T17:28:37.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:37 smithi162 ceph-mon[44304]: pgmap v533: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 1.3 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 791 op/s; 5327/61911 objects degraded (8.604%); 0 B/s, 130 objects/s recovering 2024-02-09T17:28:37.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:37 smithi099 ceph-mon[39420]: pgmap v533: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 1.3 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 791 op/s; 5327/61911 objects degraded (8.604%); 0 B/s, 130 objects/s recovering 2024-02-09T17:28:39.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:39 smithi179 ceph-mon[43975]: pgmap v534: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 1.2 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 611 op/s; 5327/59841 objects degraded (8.902%); 0 B/s, 114 objects/s recovering 2024-02-09T17:28:39.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:39 smithi162 ceph-mon[44304]: pgmap v534: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 1.2 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 611 op/s; 5327/59841 objects degraded (8.902%); 0 B/s, 114 objects/s recovering 2024-02-09T17:28:39.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:39 smithi099 ceph-mon[39420]: pgmap v534: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 1.2 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 611 op/s; 5327/59841 objects degraded (8.902%); 0 B/s, 114 objects/s recovering 2024-02-09T17:28:40.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:40 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:28:40.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:40 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:28:40.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:40 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:28:40.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:40 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:28:40.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:40 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:28:40.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:40 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:28:41.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:41 smithi179 ceph-mon[43975]: pgmap v535: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 1.1 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 716 op/s; 5327/52836 objects degraded (10.082%); 0 B/s, 126 objects/s recovering 2024-02-09T17:28:41.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:41 smithi179 ceph-mon[43975]: osdmap e178: 12 total, 12 up, 12 in 2024-02-09T17:28:41.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:41 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 5327/52836 objects degraded (10.082%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:28:41.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:41 smithi162 ceph-mon[44304]: pgmap v535: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 1.1 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 716 op/s; 5327/52836 objects degraded (10.082%); 0 B/s, 126 objects/s recovering 2024-02-09T17:28:41.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:41 smithi162 ceph-mon[44304]: osdmap e178: 12 total, 12 up, 12 in 2024-02-09T17:28:41.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:41 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 5327/52836 objects degraded (10.082%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:28:41.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:41 smithi099 ceph-mon[39420]: pgmap v535: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 1.1 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 716 op/s; 5327/52836 objects degraded (10.082%); 0 B/s, 126 objects/s recovering 2024-02-09T17:28:41.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:41 smithi099 ceph-mon[39420]: osdmap e178: 12 total, 12 up, 12 in 2024-02-09T17:28:41.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:41 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 5327/52836 objects degraded (10.082%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:28:43.330 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:28:43.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:43.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:28:43.439 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:28:43.472 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:43.501 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:28:43.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:43.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:28:43.606 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:28:43.672 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:43.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:43 smithi179 ceph-mon[43975]: pgmap v537: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 1013 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 753 op/s; 5327/48606 objects degraded (10.960%); 0 B/s, 141 objects/s recovering 2024-02-09T17:28:43.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:43 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:28:43.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:43 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:28:43.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:43 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:28:43.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:43 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:28:43.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:43 smithi162 ceph-mon[44304]: pgmap v537: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 1013 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 753 op/s; 5327/48606 objects degraded (10.960%); 0 B/s, 141 objects/s recovering 2024-02-09T17:28:43.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:43 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:28:43.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:43 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:28:43.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:43 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:28:43.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:43 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:28:43.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:43 smithi099 ceph-mon[39420]: pgmap v537: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 1013 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 753 op/s; 5327/48606 objects degraded (10.960%); 0 B/s, 141 objects/s recovering 2024-02-09T17:28:43.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:43 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:28:43.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:43 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:28:43.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:43 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:28:43.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:43 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:28:44.631 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:44 smithi179 ceph-mon[43975]: osdmap e179: 12 total, 12 up, 12 in 2024-02-09T17:28:44.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:44 smithi099 ceph-mon[39420]: osdmap e179: 12 total, 12 up, 12 in 2024-02-09T17:28:44.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:44 smithi162 ceph-mon[44304]: osdmap e179: 12 total, 12 up, 12 in 2024-02-09T17:28:45.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:45 smithi179 ceph-mon[43975]: pgmap v539: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 926 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 727 op/s; 5327/44433 objects degraded (11.989%); 0 B/s, 146 objects/s recovering 2024-02-09T17:28:45.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:45 smithi162 ceph-mon[44304]: pgmap v539: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 926 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 727 op/s; 5327/44433 objects degraded (11.989%); 0 B/s, 146 objects/s recovering 2024-02-09T17:28:45.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:45 smithi099 ceph-mon[39420]: pgmap v539: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 926 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 727 op/s; 5327/44433 objects degraded (11.989%); 0 B/s, 146 objects/s recovering 2024-02-09T17:28:46.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:46 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 5327/44433 objects degraded (11.989%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:28:46.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:46 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 5327/44433 objects degraded (11.989%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:28:46.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:46 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 5327/44433 objects degraded (11.989%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:28:47.537 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:28:47.846 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:47 smithi099 ceph-mon[39420]: pgmap v540: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 821 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 851 op/s; 5327/39381 objects degraded (13.527%); 0 B/s, 135 objects/s recovering 2024-02-09T17:28:47.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:47 smithi179 ceph-mon[43975]: pgmap v540: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 821 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 851 op/s; 5327/39381 objects degraded (13.527%); 0 B/s, 135 objects/s recovering 2024-02-09T17:28:47.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:47 smithi162 ceph-mon[44304]: pgmap v540: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 821 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 851 op/s; 5327/39381 objects degraded (13.527%); 0 B/s, 135 objects/s recovering 2024-02-09T17:28:48.291 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:28:48.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:48.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:28:48.360 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:28:48.360 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:28:48.414 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:28:48.447 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:48.470 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:28:48.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:48.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:28:48.574 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:28:48.611 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:48.646 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:28:48.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:48.652 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:28:48.753 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:28:48.828 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:49.176 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:28:49.545 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-09T17:28:49.832 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-09T17:28:49.930 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:49 smithi099 ceph-mon[39420]: pgmap v541: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 773 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 655 op/s; 5327/37086 objects degraded (14.364%); 0 B/s, 119 objects/s recovering 2024-02-09T17:28:50.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:49 smithi162 ceph-mon[44304]: pgmap v541: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 773 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 655 op/s; 5327/37086 objects degraded (14.364%); 0 B/s, 119 objects/s recovering 2024-02-09T17:28:50.180 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:49 smithi179 ceph-mon[43975]: pgmap v541: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 773 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 655 op/s; 5327/37086 objects degraded (14.364%); 0 B/s, 119 objects/s recovering 2024-02-09T17:28:50.739 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:28:50.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:50.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:28:50.851 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:28:50.901 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:50 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:28:50.902 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:50 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:28:50.902 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:50 smithi099 ceph-mon[39420]: osdmap e180: 12 total, 12 up, 12 in 2024-02-09T17:28:50.904 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:51.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:50 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:28:51.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:50 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:28:51.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:50 smithi179 ceph-mon[43975]: osdmap e180: 12 total, 12 up, 12 in 2024-02-09T17:28:51.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:50 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:28:51.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:50 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:28:51.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:50 smithi162 ceph-mon[44304]: osdmap e180: 12 total, 12 up, 12 in 2024-02-09T17:28:51.716 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:51 smithi099 ceph-mon[39420]: pgmap v543: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 630 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 764 op/s; 5327/30228 objects degraded (17.623%); 0 B/s, 138 objects/s recovering 2024-02-09T17:28:51.717 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:51 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 5327/30228 objects degraded (17.623%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:28:51.717 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:51 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3881917295' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:28:52.091 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:28:52.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:52.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:28:52.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:51 smithi179 ceph-mon[43975]: pgmap v543: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 630 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 764 op/s; 5327/30228 objects degraded (17.623%); 0 B/s, 138 objects/s recovering 2024-02-09T17:28:52.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:51 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 5327/30228 objects degraded (17.623%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:28:52.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:51 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3881917295' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:28:52.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:51 smithi162 ceph-mon[44304]: pgmap v543: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 630 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 764 op/s; 5327/30228 objects degraded (17.623%); 0 B/s, 138 objects/s recovering 2024-02-09T17:28:52.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:51 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 5327/30228 objects degraded (17.623%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:28:52.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:51 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3881917295' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:28:52.187 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:28:52.188 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:28:52.188 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:28:52.188 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:28:52.188 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:28:52.188 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:28:52.188 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:28:52.188 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:28:52.188 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:28:52.188 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:28:52.188 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:28:52.195 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:28:52.209 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:28:52.210 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:28:52.210 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:28:52.243 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:28:52.243 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:28:52.244 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:28:52.244 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:28:52.244 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:28:52.244 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:28:52.244 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:28:52.244 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:28:52.244 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:28:52.244 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:28:52.244 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:28:52.244 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:28:52.292 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:52.678 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:28:52.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:52.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:28:52.782 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:28:52.814 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:52.837 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:28:52.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:52.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:28:52.944 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:28:52.973 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:52.996 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:28:53.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:53.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:28:53.101 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:28:53.132 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:53.155 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:28:53.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:53.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:28:53.260 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:28:53.344 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:53.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:53 smithi099 ceph-mon[39420]: pgmap v544: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 541 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 824 op/s; 5327/25968 objects degraded (20.514%); 0 B/s, 137 objects/s recovering 2024-02-09T17:28:53.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:53 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:28:53.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:53 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:28:54.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:53 smithi179 ceph-mon[43975]: pgmap v544: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 541 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 824 op/s; 5327/25968 objects degraded (20.514%); 0 B/s, 137 objects/s recovering 2024-02-09T17:28:54.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:53 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:28:54.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:53 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:28:54.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:53 smithi162 ceph-mon[44304]: pgmap v544: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 541 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 824 op/s; 5327/25968 objects degraded (20.514%); 0 B/s, 137 objects/s recovering 2024-02-09T17:28:54.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:53 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:28:54.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:53 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:28:54.931 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:54 smithi099 ceph-mon[39420]: osdmap e181: 12 total, 12 up, 12 in 2024-02-09T17:28:55.175 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:54 smithi179 ceph-mon[43975]: osdmap e181: 12 total, 12 up, 12 in 2024-02-09T17:28:55.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:54 smithi162 ceph-mon[44304]: osdmap e181: 12 total, 12 up, 12 in 2024-02-09T17:28:55.960 INFO:teuthology.orchestra.run.smithi099.stderr:no valid command found; 10 closest matches: 2024-02-09T17:28:55.960 INFO:teuthology.orchestra.run.smithi099.stderr:0 2024-02-09T17:28:55.960 INFO:teuthology.orchestra.run.smithi099.stderr:1 2024-02-09T17:28:55.960 INFO:teuthology.orchestra.run.smithi099.stderr:2 2024-02-09T17:28:55.960 INFO:teuthology.orchestra.run.smithi099.stderr:abort 2024-02-09T17:28:55.960 INFO:teuthology.orchestra.run.smithi099.stderr:assert 2024-02-09T17:28:55.961 INFO:teuthology.orchestra.run.smithi099.stderr:bench [] [] [] [] 2024-02-09T17:28:55.961 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs debug_inject_read_zeros 2024-02-09T17:28:55.961 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs files list 2024-02-09T17:28:55.961 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs stats 2024-02-09T17:28:55.961 INFO:teuthology.orchestra.run.smithi099.stderr:bluestore allocator dump block 2024-02-09T17:28:55.961 INFO:teuthology.orchestra.run.smithi099.stderr:Error EINVAL: invalid command 2024-02-09T17:28:55.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:55 smithi099 ceph-mon[39420]: pgmap v546: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 457 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 725 op/s; 5327/21927 objects degraded (24.294%); 0 B/s, 155 objects/s recovering 2024-02-09T17:28:56.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:55 smithi179 ceph-mon[43975]: pgmap v546: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 457 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 725 op/s; 5327/21927 objects degraded (24.294%); 0 B/s, 155 objects/s recovering 2024-02-09T17:28:56.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:55 smithi162 ceph-mon[44304]: pgmap v546: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 457 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 725 op/s; 5327/21927 objects degraded (24.294%); 0 B/s, 155 objects/s recovering 2024-02-09T17:28:56.755 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:28:56.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:56.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:28:56.863 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:28:56.901 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:56.949 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:28:56.957 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:56.958 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:28:56.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:56 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 5327/21927 objects degraded (24.294%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:28:56.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:56 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:28:56.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:56 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:28:57.059 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:28:57.100 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:56 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 5327/21927 objects degraded (24.294%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:28:57.100 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:56 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:28:57.100 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:56 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:28:57.104 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:57.108 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-09T17:28:57.109 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-02-09T17:28:57.109 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph tell osd.1 'trim stale osdmaps' 2024-02-09T17:28:57.141 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:28:57.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:57.146 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:28:57.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:56 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 5327/21927 objects degraded (24.294%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:28:57.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:56 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:28:57.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:56 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:28:57.247 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:28:57.298 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:57.324 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:28:57.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:57.330 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:28:57.431 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:28:57.472 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:57.505 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:28:57.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:57.509 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:28:57.610 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:28:57.638 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:57.662 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:28:57.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:57.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:28:57.764 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:57 smithi179 ceph-mon[43975]: pgmap v547: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 356 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 831 op/s; 5327/17091 objects degraded (31.168%); 0 B/s, 148 objects/s recovering 2024-02-09T17:28:57.764 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:57 smithi179 ceph-mon[43975]: osdmap e182: 12 total, 12 up, 12 in 2024-02-09T17:28:57.766 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:28:57.813 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:57.852 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:28:57.860 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:57.861 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:28:57.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:57 smithi162 ceph-mon[44304]: pgmap v547: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 356 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 831 op/s; 5327/17091 objects degraded (31.168%); 0 B/s, 148 objects/s recovering 2024-02-09T17:28:57.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:57 smithi162 ceph-mon[44304]: osdmap e182: 12 total, 12 up, 12 in 2024-02-09T17:28:57.961 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:28:57.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:57 smithi099 ceph-mon[39420]: pgmap v547: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 356 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 831 op/s; 5327/17091 objects degraded (31.168%); 0 B/s, 148 objects/s recovering 2024-02-09T17:28:57.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:57 smithi099 ceph-mon[39420]: osdmap e182: 12 total, 12 up, 12 in 2024-02-09T17:28:58.016 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:58.141 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:28:58.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:58.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:28:58.247 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:28:58.276 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:58.297 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:28:58.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:58.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:28:58.405 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:28:58.434 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:58.458 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:28:58.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:58.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:28:58.504 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:28:58.504 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [ 2024-02-09T17:28:58.504 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.505 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:264119 2.1 e182/157 2:85cc70b6:::benchmark_data_smithi179_77138_object130711:head v 182'16262, mlcod=182'16259)", 2024-02-09T17:28:58.505 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:28:58.470592+0000", 2024-02-09T17:28:58.505 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.028937575, 2024-02-09T17:28:58.505 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.028969567000000002, 2024-02-09T17:28:58.505 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:28:58.505 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T17:28:58.505 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:28:58.505 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.505 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:28:58.505 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:58.470592+0000", 2024-02-09T17:28:58.505 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:28:58.505 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:58.505 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.505 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:28:58.506 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:58.470592+0000", 2024-02-09T17:28:58.506 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:28:58.506 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:58.506 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.506 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:28:58.506 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:58.470593+0000", 2024-02-09T17:28:58.506 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4440000000000001e-06 2024-02-09T17:28:58.506 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:58.506 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.506 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:28:58.506 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:58.470601+0000", 2024-02-09T17:28:58.506 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.9179999999999994e-06 2024-02-09T17:28:58.506 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:58.507 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.507 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:28:58.507 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:58.470614+0000", 2024-02-09T17:28:58.507 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.254e-05 2024-02-09T17:28:58.507 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:58.507 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.507 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:28:58.507 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:58.470622+0000", 2024-02-09T17:28:58.507 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.6500000000000002e-06 2024-02-09T17:28:58.507 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:58.507 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.507 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:28:58.507 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:58.475732+0000", 2024-02-09T17:28:58.507 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0051102350000000003 2024-02-09T17:28:58.507 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:58.507 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.508 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:28:58.508 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:58.475752+0000", 2024-02-09T17:28:58.508 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.9692999999999999e-05 2024-02-09T17:28:58.508 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:28:58.508 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:28:58.508 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:28:58.508 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:58.508 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.508 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:264126 2.1 e182/157 2:8d06d98d:::benchmark_data_smithi179_77138_object130718:head v 182'16263, mlcod=182'16260)", 2024-02-09T17:28:58.508 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:28:58.470675+0000", 2024-02-09T17:28:58.508 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.028854555, 2024-02-09T17:28:58.508 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.028957469, 2024-02-09T17:28:58.508 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:28:58.509 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T17:28:58.509 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:28:58.509 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.509 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:28:58.509 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:58.470675+0000", 2024-02-09T17:28:58.509 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:28:58.509 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:58.509 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.509 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:28:58.509 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:58.470675+0000", 2024-02-09T17:28:58.509 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:28:58.509 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:58.509 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.510 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:28:58.510 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:58.470675+0000", 2024-02-09T17:28:58.510 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.92e-07 2024-02-09T17:28:58.510 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:58.510 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.510 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:28:58.510 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:58.470682+0000", 2024-02-09T17:28:58.510 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.6780000000000003e-06 2024-02-09T17:28:58.510 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:58.510 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.510 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:28:58.510 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:58.470687+0000", 2024-02-09T17:28:58.510 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3029999999999999e-06 2024-02-09T17:28:58.510 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:58.511 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.511 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:28:58.511 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:58.470691+0000", 2024-02-09T17:28:58.511 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.8959999999999996e-06 2024-02-09T17:28:58.511 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:58.511 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.511 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:28:58.511 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:58.480604+0000", 2024-02-09T17:28:58.511 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0099137019999999999 2024-02-09T17:28:58.511 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:58.511 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.511 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:28:58.511 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:58.480628+0000", 2024-02-09T17:28:58.511 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.338e-05 2024-02-09T17:28:58.512 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:28:58.512 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:28:58.512 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:28:58.512 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:58.512 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.512 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:264127 2.e e182/157 2:766e5fdf:::benchmark_data_smithi179_77138_object130719:head v 182'16487, mlcod=182'16484)", 2024-02-09T17:28:58.512 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:28:58.491406+0000", 2024-02-09T17:28:58.512 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.0081228340000000007, 2024-02-09T17:28:58.512 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0082916269999999993, 2024-02-09T17:28:58.512 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:28:58.512 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T17:28:58.512 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:28:58.512 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.513 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:28:58.513 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:58.491406+0000", 2024-02-09T17:28:58.513 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:28:58.513 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:58.513 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.513 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:28:58.513 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:58.491406+0000", 2024-02-09T17:28:58.513 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:28:58.513 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:58.513 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.513 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:28:58.513 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:58.491407+0000", 2024-02-09T17:28:58.513 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.4099999999999997e-07 2024-02-09T17:28:58.513 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:58.513 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.514 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:28:58.514 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:58.491415+0000", 2024-02-09T17:28:58.514 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.267e-06 2024-02-09T17:28:58.514 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:58.514 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.514 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:28:58.514 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:58.491430+0000", 2024-02-09T17:28:58.514 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5588000000000001e-05 2024-02-09T17:28:58.514 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:58.514 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.514 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:28:58.514 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:58.491438+0000", 2024-02-09T17:28:58.514 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.8189999999999992e-06 2024-02-09T17:28:58.514 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:58.514 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.515 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:28:58.515 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:58.493516+0000", 2024-02-09T17:28:58.515 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0020781990000000002 2024-02-09T17:28:58.515 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:58.515 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.515 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:28:58.515 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:58.493535+0000", 2024-02-09T17:28:58.515 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.8736000000000002e-05 2024-02-09T17:28:58.515 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:28:58.515 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:28:58.515 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:28:58.515 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:58.515 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.515 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:264129 2.1 e182/157 2:8245e204:::benchmark_data_smithi179_77138_object130721:head v 182'16264, mlcod=182'16260)", 2024-02-09T17:28:58.515 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:28:58.491459+0000", 2024-02-09T17:28:58.515 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.0080702299999999994, 2024-02-09T17:28:58.515 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0083025900000000003, 2024-02-09T17:28:58.516 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:28:58.516 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T17:28:58.516 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:28:58.516 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.516 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:28:58.516 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:58.491459+0000", 2024-02-09T17:28:58.516 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:28:58.516 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:58.516 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.516 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:28:58.516 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:58.491459+0000", 2024-02-09T17:28:58.516 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:28:58.516 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:58.516 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.517 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:28:58.517 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:58.491459+0000", 2024-02-09T17:28:58.517 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.5400000000000002e-07 2024-02-09T17:28:58.517 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:58.517 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.517 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:28:58.517 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:58.491463+0000", 2024-02-09T17:28:58.517 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.8970000000000001e-06 2024-02-09T17:28:58.517 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:58.517 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.517 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:28:58.517 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:58.491469+0000", 2024-02-09T17:28:58.517 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6149999999999996e-06 2024-02-09T17:28:58.517 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:58.518 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.518 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:28:58.518 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:58.491473+0000", 2024-02-09T17:28:58.518 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.3390000000000003e-06 2024-02-09T17:28:58.518 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:58.518 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.518 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:28:58.518 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:58.491605+0000", 2024-02-09T17:28:58.518 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013188100000000001 2024-02-09T17:28:58.518 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:58.518 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.518 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:28:58.518 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:58.491624+0000", 2024-02-09T17:28:58.518 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.9490000000000001e-05 2024-02-09T17:28:58.518 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:28:58.519 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:28:58.519 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:28:58.519 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:58.519 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.519 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:264132 2.1 e182/157 2:8ebd7d97:::benchmark_data_smithi179_77138_object130724:head v 182'16265, mlcod=182'16260)", 2024-02-09T17:28:58.519 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:28:58.491484+0000", 2024-02-09T17:28:58.519 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.0080455600000000002, 2024-02-09T17:28:58.519 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0083422189999999997, 2024-02-09T17:28:58.519 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:28:58.519 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T17:28:58.519 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:28:58.519 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.519 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:28:58.519 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:58.491484+0000", 2024-02-09T17:28:58.519 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:28:58.520 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:58.520 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.520 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:28:58.520 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:58.491484+0000", 2024-02-09T17:28:58.520 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:28:58.520 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:58.520 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.520 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:28:58.520 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:58.491484+0000", 2024-02-09T17:28:58.520 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.6400000000000001e-07 2024-02-09T17:28:58.520 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:58.520 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.520 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:28:58.520 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:58.491491+0000", 2024-02-09T17:28:58.520 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.9700000000000002e-06 2024-02-09T17:28:58.521 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:58.521 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.521 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:28:58.521 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:58.491496+0000", 2024-02-09T17:28:58.521 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.0509999999999996e-06 2024-02-09T17:28:58.521 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:58.521 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.521 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:28:58.521 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:58.491500+0000", 2024-02-09T17:28:58.521 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.0779999999999997e-06 2024-02-09T17:28:58.521 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:58.521 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.521 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:28:58.521 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:58.497668+0000", 2024-02-09T17:28:58.522 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0061684879999999997 2024-02-09T17:28:58.522 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:58.522 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.522 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:28:58.522 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:58.497690+0000", 2024-02-09T17:28:58.522 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.139e-05 2024-02-09T17:28:58.522 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:28:58.522 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:28:58.522 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:28:58.522 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:58.522 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.522 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:264140 2.7 e182/111)", 2024-02-09T17:28:58.522 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:28:58.499500+0000", 2024-02-09T17:28:58.522 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 2.8912999999999999e-05, 2024-02-09T17:28:58.522 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00038446200000000002, 2024-02-09T17:28:58.523 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:28:58.523 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "queued for pg", 2024-02-09T17:28:58.523 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:28:58.523 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.523 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:28:58.523 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:58.499500+0000", 2024-02-09T17:28:58.523 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:28:58.523 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:58.523 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.523 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:28:58.523 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:58.499500+0000", 2024-02-09T17:28:58.523 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:28:58.523 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:58.524 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.524 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:28:58.524 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:58.499501+0000", 2024-02-09T17:28:58.524 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.02e-06 2024-02-09T17:28:58.524 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:58.524 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.524 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:28:58.524 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:58.499511+0000", 2024-02-09T17:28:58.524 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.3640000000000005e-06 2024-02-09T17:28:58.524 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:58.524 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.524 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:28:58.524 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:58.499525+0000", 2024-02-09T17:28:58.524 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4419e-05 2024-02-09T17:28:58.524 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:28:58.525 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:28:58.525 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:28:58.525 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:58.499533+0000", 2024-02-09T17:28:58.525 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.9359999999999999e-06 2024-02-09T17:28:58.525 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:28:58.525 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:28:58.525 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:28:58.525 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:28:58.525 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T17:28:58.525 INFO:teuthology.orchestra.run.smithi099.stdout: "num_ops": 6 2024-02-09T17:28:58.525 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:28:58.562 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:28:58.590 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:58.611 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:28:58.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:58.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:28:58.686 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:28:58.717 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:28:58.770 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:58.802 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:28:58.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:58.808 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:28:58.909 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:28:58.940 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:58.961 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:28:58.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:58.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:28:59.068 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:28:59.119 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:59.146 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:28:59.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:59.158 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:28:59.259 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:28:59.291 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:59.315 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:28:59.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:59.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:28:59.421 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:28:59.447 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:59.468 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:28:59.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:59.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:28:59.571 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:28:59.600 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:59.634 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:28:59.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:59.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:28:59.737 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:28:59.773 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:59.800 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:28:59.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:59.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:28:59.910 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-09T17:28:59.913 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:28:59.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:59 smithi162 ceph-mon[44304]: pgmap v549: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 315 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 628 op/s; 5327/15117 objects degraded (35.238%); 0 B/s, 139 objects/s recovering 2024-02-09T17:28:59.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:59 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4269535497' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:28:59.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:28:59 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:28:59.946 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:28:59.971 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is set 2024-02-09T17:28:59.973 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:28:59.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:28:59.977 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:29:00.078 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:29:00.112 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:00.137 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:29:00.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:00.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:29:00.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:59 smithi179 ceph-mon[43975]: pgmap v549: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 315 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 628 op/s; 5327/15117 objects degraded (35.238%); 0 B/s, 139 objects/s recovering 2024-02-09T17:29:00.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:59 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4269535497' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:29:00.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:28:59 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:29:00.193 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:29:00.244 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:29:00.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:59 smithi099 ceph-mon[39420]: pgmap v549: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 315 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 628 op/s; 5327/15117 objects degraded (35.238%); 0 B/s, 139 objects/s recovering 2024-02-09T17:29:00.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:59 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4269535497' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:29:00.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:28:59 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:29:00.285 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:00.318 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:29:00.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:00.321 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:29:00.422 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:29:00.481 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:00.747 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:29:00.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:00.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:29:00.820 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:29:00.861 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:29:00.900 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:00.925 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:29:00.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:00.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:29:01.031 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:29:01.097 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:00 smithi099 ceph-mon[39420]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:29:01.097 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:00 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:29:01.098 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:00 smithi099 ceph-mon[39420]: osdmap e183: 12 total, 12 up, 12 in 2024-02-09T17:29:01.117 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:01.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:00 smithi179 ceph-mon[43975]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:29:01.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:00 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:29:01.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:00 smithi179 ceph-mon[43975]: osdmap e183: 12 total, 12 up, 12 in 2024-02-09T17:29:01.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:00 smithi162 ceph-mon[44304]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:29:01.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:00 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:29:01.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:00 smithi162 ceph-mon[44304]: osdmap e183: 12 total, 12 up, 12 in 2024-02-09T17:29:01.226 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:29:01.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:01.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:29:01.337 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:29:01.368 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:01.398 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:29:01.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:01.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:29:01.507 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:29:01.539 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:01.587 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:29:01.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:01.593 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:29:01.694 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:29:01.723 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:01.750 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:29:01.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:01.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:29:01.855 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:29:01.919 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:01.944 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:29:01.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:01.949 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:29:02.050 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:29:02.086 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:01 smithi162 ceph-mon[44304]: pgmap v551: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 174 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 837 op/s; 5327/8328 objects degraded (63.965%); 0 B/s, 155 objects/s recovering 2024-02-09T17:29:02.087 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:01 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 5327/8328 objects degraded (63.965%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:29:02.087 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:02.107 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:29:02.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:02.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:29:02.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:01 smithi179 ceph-mon[43975]: pgmap v551: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 174 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 837 op/s; 5327/8328 objects degraded (63.965%); 0 B/s, 155 objects/s recovering 2024-02-09T17:29:02.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:01 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 5327/8328 objects degraded (63.965%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:29:02.211 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:29:02.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:01 smithi099 ceph-mon[39420]: pgmap v551: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 174 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 837 op/s; 5327/8328 objects degraded (63.965%); 0 B/s, 155 objects/s recovering 2024-02-09T17:29:02.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:01 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 5327/8328 objects degraded (63.965%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:29:02.274 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:02.289 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Removed 133405 objects 2024-02-09T17:29:02.289 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Clean up completed and total clean up time :162.824 2024-02-09T17:29:02.783 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset noscrub 2024-02-09T17:29:02.827 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:29:02.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:02.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:29:02.932 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:29:02.959 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:02.977 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:29:02.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:02.981 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:29:03.082 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:29:03.110 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:03.362 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:29:03.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:03.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:29:03.396 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:29:03.471 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:29:03.498 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:03.517 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:29:03.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:03.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:29:03.621 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:29:03.647 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:03.666 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:29:03.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:03.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:29:03.770 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:29:03.801 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:03.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:03 smithi099 ceph-mon[39420]: pgmap v552: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 93 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 729 op/s; 5327/4422 objects degraded (120.466%); 0 B/s, 174 objects/s recovering 2024-02-09T17:29:04.032 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:29:04.052 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:29:04.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:04.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:29:04.077 INFO:teuthology.orchestra.run.smithi099.stderr:no valid command found; 10 closest matches: 2024-02-09T17:29:04.077 INFO:teuthology.orchestra.run.smithi099.stderr:0 2024-02-09T17:29:04.078 INFO:teuthology.orchestra.run.smithi099.stderr:1 2024-02-09T17:29:04.078 INFO:teuthology.orchestra.run.smithi099.stderr:2 2024-02-09T17:29:04.078 INFO:teuthology.orchestra.run.smithi099.stderr:abort 2024-02-09T17:29:04.078 INFO:teuthology.orchestra.run.smithi099.stderr:assert 2024-02-09T17:29:04.078 INFO:teuthology.orchestra.run.smithi099.stderr:bench [] [] [] [] 2024-02-09T17:29:04.078 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs debug_inject_read_zeros 2024-02-09T17:29:04.078 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs files list 2024-02-09T17:29:04.078 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs stats 2024-02-09T17:29:04.078 INFO:teuthology.orchestra.run.smithi099.stderr:bluestore allocator dump block 2024-02-09T17:29:04.078 INFO:teuthology.orchestra.run.smithi099.stderr:Error EINVAL: invalid command 2024-02-09T17:29:04.163 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:29:04.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:03 smithi179 ceph-mon[43975]: pgmap v552: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 93 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 729 op/s; 5327/4422 objects degraded (120.466%); 0 B/s, 174 objects/s recovering 2024-02-09T17:29:04.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:03 smithi162 ceph-mon[44304]: pgmap v552: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 93 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 729 op/s; 5327/4422 objects degraded (120.466%); 0 B/s, 174 objects/s recovering 2024-02-09T17:29:04.190 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:04.208 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:29:04.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:04.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:29:04.312 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:29:04.345 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:05.396 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:29:05.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:05.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:29:05.500 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:29:05.528 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:05.634 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-02-09T17:29:05.635 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-02-09T17:29:05.933 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:05 smithi099 ceph-mon[39420]: pgmap v553: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 41 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 631 op/s; 5327/1941 objects degraded (274.446%); 0 B/s, 187 objects/s recovering 2024-02-09T17:29:06.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:05 smithi179 ceph-mon[43975]: pgmap v553: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 41 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 631 op/s; 5327/1941 objects degraded (274.446%); 0 B/s, 187 objects/s recovering 2024-02-09T17:29:06.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:05 smithi162 ceph-mon[44304]: pgmap v553: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 41 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 631 op/s; 5327/1941 objects degraded (274.446%); 0 B/s, 187 objects/s recovering 2024-02-09T17:29:06.180 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:29:06.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:06.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:29:06.286 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:29:06.312 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:06.332 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:29:06.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:06.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:29:06.436 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:29:06.464 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:06.483 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:29:06.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:06.486 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:29:06.587 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:29:06.614 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:06.634 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:29:06.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:06.637 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:29:06.738 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:29:06.771 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:06.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:06 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 5327/1941 objects degraded (274.446%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:29:06.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:06 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 5327/1941 objects degraded (274.446%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:29:07.011 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:29:07.012 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:29:07.012 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:29:07.012 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:29:07.017 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:29:07.025 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:29:07.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:07.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:29:07.046 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:29:07.068 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:29:07.089 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:29:07.118 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:29:07.129 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:29:07.137 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:29:07.138 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:29:07.141 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:29:07.156 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:07.165 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:29:07.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:06 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 5327/1941 objects degraded (274.446%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:29:07.177 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:29:07.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:07.181 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:29:07.282 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:29:07.313 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:07.391 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:29:07.393 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:29:07.393 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:29:07.393 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:29:07.393 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:29:07.393 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:29:07.393 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:29:07.393 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:29:07.393 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:29:07.393 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:29:07.394 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:29:07.394 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:29:07.394 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:29:08.159 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:07 smithi099 ceph-mon[39420]: pgmap v554: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 6.8 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 531 op/s; 5327/306 objects degraded (1740.850%); 0 B/s, 168 objects/s recovering 2024-02-09T17:29:08.159 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:07 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3480437974' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:29:08.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:07 smithi179 ceph-mon[43975]: pgmap v554: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 6.8 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 531 op/s; 5327/306 objects degraded (1740.850%); 0 B/s, 168 objects/s recovering 2024-02-09T17:29:08.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:07 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3480437974' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:29:08.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:07 smithi162 ceph-mon[44304]: pgmap v554: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 6.8 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 531 op/s; 5327/306 objects degraded (1740.850%); 0 B/s, 168 objects/s recovering 2024-02-09T17:29:08.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:07 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3480437974' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:29:08.191 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-09T17:29:08.195 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:29:08.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:08.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:29:08.300 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:29:08.326 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:08.357 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:29:08.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:08.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:29:08.474 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:29:08.502 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:10.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:09 smithi179 ceph-mon[43975]: pgmap v555: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 577 KiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 503 op/s; 5327/6 objects degraded (88783.333%); 0 B/s, 239 objects/s recovering 2024-02-09T17:29:10.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:09 smithi162 ceph-mon[44304]: pgmap v555: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 577 KiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 503 op/s; 5327/6 objects degraded (88783.333%); 0 B/s, 239 objects/s recovering 2024-02-09T17:29:10.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:09 smithi099 ceph-mon[39420]: pgmap v555: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 577 KiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 503 op/s; 5327/6 objects degraded (88783.333%); 0 B/s, 239 objects/s recovering 2024-02-09T17:29:12.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:11 smithi179 ceph-mon[43975]: pgmap v556: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 577 KiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 342 op/s; 5327/6 objects degraded (88783.333%); 0 B/s, 320 objects/s recovering 2024-02-09T17:29:12.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:11 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 5327/6 objects degraded (88783.333%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:29:12.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:11 smithi162 ceph-mon[44304]: pgmap v556: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 577 KiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 342 op/s; 5327/6 objects degraded (88783.333%); 0 B/s, 320 objects/s recovering 2024-02-09T17:29:12.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:11 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 5327/6 objects degraded (88783.333%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:29:12.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:11 smithi099 ceph-mon[39420]: pgmap v556: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 577 KiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 342 op/s; 5327/6 objects degraded (88783.333%); 0 B/s, 320 objects/s recovering 2024-02-09T17:29:12.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:11 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 5327/6 objects degraded (88783.333%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:29:14.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:13 smithi179 ceph-mon[43975]: pgmap v557: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 577 KiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 231 op/s; 5327/6 objects degraded (88783.333%); 0 B/s, 359 objects/s recovering 2024-02-09T17:29:14.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:13 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:29:14.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:13 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:29:14.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:13 smithi162 ceph-mon[44304]: pgmap v557: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 577 KiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 231 op/s; 5327/6 objects degraded (88783.333%); 0 B/s, 359 objects/s recovering 2024-02-09T17:29:14.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:13 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:29:14.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:13 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:29:14.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:13 smithi099 ceph-mon[39420]: pgmap v557: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 577 KiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 231 op/s; 5327/6 objects degraded (88783.333%); 0 B/s, 359 objects/s recovering 2024-02-09T17:29:14.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:13 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:29:14.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:13 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:29:15.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:14 smithi179 ceph-mon[43975]: osdmap e184: 12 total, 12 up, 12 in 2024-02-09T17:29:15.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:14 smithi162 ceph-mon[44304]: osdmap e184: 12 total, 12 up, 12 in 2024-02-09T17:29:15.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:14 smithi099 ceph-mon[39420]: osdmap e184: 12 total, 12 up, 12 in 2024-02-09T17:29:16.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:15 smithi179 ceph-mon[43975]: pgmap v559: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 577 KiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 64 op/s; 5327/6 objects degraded (88783.333%); 0 B/s, 442 objects/s recovering 2024-02-09T17:29:16.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:15 smithi179 ceph-mon[43975]: osdmap e185: 12 total, 12 up, 12 in 2024-02-09T17:29:16.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:15 smithi162 ceph-mon[44304]: pgmap v559: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 577 KiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 64 op/s; 5327/6 objects degraded (88783.333%); 0 B/s, 442 objects/s recovering 2024-02-09T17:29:16.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:15 smithi162 ceph-mon[44304]: osdmap e185: 12 total, 12 up, 12 in 2024-02-09T17:29:16.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:15 smithi099 ceph-mon[39420]: pgmap v559: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 577 KiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 64 op/s; 5327/6 objects degraded (88783.333%); 0 B/s, 442 objects/s recovering 2024-02-09T17:29:16.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:15 smithi099 ceph-mon[39420]: osdmap e185: 12 total, 12 up, 12 in 2024-02-09T17:29:17.139 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:16 smithi099 ceph-mon[39420]: osdmap e186: 12 total, 12 up, 12 in 2024-02-09T17:29:17.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:16 smithi179 ceph-mon[43975]: osdmap e186: 12 total, 12 up, 12 in 2024-02-09T17:29:17.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:16 smithi162 ceph-mon[44304]: osdmap e186: 12 total, 12 up, 12 in 2024-02-09T17:29:18.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:17 smithi179 ceph-mon[43975]: pgmap v562: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 577 KiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5327/6 objects degraded (88783.333%); 0 B/s, 467 objects/s recovering 2024-02-09T17:29:18.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:17 smithi162 ceph-mon[44304]: pgmap v562: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 577 KiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5327/6 objects degraded (88783.333%); 0 B/s, 467 objects/s recovering 2024-02-09T17:29:18.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:17 smithi099 ceph-mon[39420]: pgmap v562: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 577 KiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5327/6 objects degraded (88783.333%); 0 B/s, 467 objects/s recovering 2024-02-09T17:29:20.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:19 smithi179 ceph-mon[43975]: pgmap v563: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 14 active+clean; 577 KiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5327/6 objects degraded (88783.333%); 0 B/s, 425 objects/s recovering 2024-02-09T17:29:20.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:19 smithi162 ceph-mon[44304]: pgmap v563: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 14 active+clean; 577 KiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5327/6 objects degraded (88783.333%); 0 B/s, 425 objects/s recovering 2024-02-09T17:29:20.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:19 smithi099 ceph-mon[39420]: pgmap v563: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 14 active+clean; 577 KiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5327/6 objects degraded (88783.333%); 0 B/s, 425 objects/s recovering 2024-02-09T17:29:20.317 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:29:21.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:20 smithi179 ceph-mon[43975]: pgmap v564: 17 pgs: 3 active+recovering+undersized+remapped, 14 active+clean; 577 KiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 777 objects/s recovering 2024-02-09T17:29:21.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:20 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 3 pgs undersized (PG_DEGRADED) 2024-02-09T17:29:21.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:20 smithi179 ceph-mon[43975]: osdmap e187: 12 total, 12 up, 12 in 2024-02-09T17:29:21.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:20 smithi162 ceph-mon[44304]: pgmap v564: 17 pgs: 3 active+recovering+undersized+remapped, 14 active+clean; 577 KiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 777 objects/s recovering 2024-02-09T17:29:21.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:20 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 3 pgs undersized (PG_DEGRADED) 2024-02-09T17:29:21.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:20 smithi162 ceph-mon[44304]: osdmap e187: 12 total, 12 up, 12 in 2024-02-09T17:29:21.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:20 smithi099 ceph-mon[39420]: pgmap v564: 17 pgs: 3 active+recovering+undersized+remapped, 14 active+clean; 577 KiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 777 objects/s recovering 2024-02-09T17:29:21.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:20 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 3 pgs undersized (PG_DEGRADED) 2024-02-09T17:29:21.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:20 smithi099 ceph-mon[39420]: osdmap e187: 12 total, 12 up, 12 in 2024-02-09T17:29:21.665 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:29:21.703 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-09T17:29:21.703 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-02-09T17:29:21.704 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph tell osd.2 'trim stale osdmaps' 2024-02-09T17:29:22.207 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:22 smithi099 ceph-mon[39420]: osdmap e188: 12 total, 12 up, 12 in 2024-02-09T17:29:22.584 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:29:22.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:22 smithi179 ceph-mon[43975]: osdmap e188: 12 total, 12 up, 12 in 2024-02-09T17:29:22.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:22 smithi162 ceph-mon[44304]: osdmap e188: 12 total, 12 up, 12 in 2024-02-09T17:29:23.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:23 smithi099 ceph-mon[39420]: osdmap e189: 12 total, 12 up, 12 in 2024-02-09T17:29:23.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:23 smithi099 ceph-mon[39420]: pgmap v568: 17 pgs: 1 peering, 2 active+recovering+undersized+remapped, 14 active+clean; 577 KiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 246 objects/s recovering 2024-02-09T17:29:23.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:23 smithi179 ceph-mon[43975]: osdmap e189: 12 total, 12 up, 12 in 2024-02-09T17:29:23.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:23 smithi179 ceph-mon[43975]: pgmap v568: 17 pgs: 1 peering, 2 active+recovering+undersized+remapped, 14 active+clean; 577 KiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 246 objects/s recovering 2024-02-09T17:29:23.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:23 smithi162 ceph-mon[44304]: osdmap e189: 12 total, 12 up, 12 in 2024-02-09T17:29:23.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:23 smithi162 ceph-mon[44304]: pgmap v568: 17 pgs: 1 peering, 2 active+recovering+undersized+remapped, 14 active+clean; 577 KiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 246 objects/s recovering 2024-02-09T17:29:23.921 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:29:23.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:23.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:29:23.927 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:29:23.927 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:29:24.026 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:29:24.054 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:24.073 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:29:24.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:24.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:29:24.177 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:29:24.194 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is unset 2024-02-09T17:29:24.206 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:24.225 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:29:24.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:24.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:29:24.328 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:29:24.353 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:24 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3717498605' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:29:24.355 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:24.375 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:29:24.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:24.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:29:24.479 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:29:24.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:24 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3717498605' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:29:24.504 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:24 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3717498605' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:29:24.506 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:24.527 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:29:24.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:24.531 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:29:24.632 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:29:24.664 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:24.736 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:29:24.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:24.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:29:24.789 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:29:24.809 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:29:24.828 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:29:24.840 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:29:24.848 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:29:24.866 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:24.868 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:29:24.890 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:29:24.909 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:29:24.931 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:29:24.950 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:29:24.970 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:29:24.989 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:29:25.008 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:29:25.010 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:29:25.010 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:29:25.010 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:29:25.010 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:29:25.010 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:29:25.010 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:29:25.010 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:29:25.010 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:29:25.010 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:29:25.010 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:29:25.010 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:29:25.010 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:29:25.294 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:25 smithi099 ceph-mon[39420]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:29:25.294 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:25 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3717498605' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:29:25.294 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:25 smithi099 ceph-mon[39420]: osdmap e190: 12 total, 12 up, 12 in 2024-02-09T17:29:25.294 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:25 smithi099 ceph-mon[39420]: pgmap v570: 17 pgs: 1 peering, 2 active+recovering+undersized+remapped, 14 active+clean; 577 KiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:25.294 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:25 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:29:25.295 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:25 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:29:25.295 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:25 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3901275451' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:29:25.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:25 smithi179 ceph-mon[43975]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:29:25.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:25 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3717498605' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:29:25.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:25 smithi179 ceph-mon[43975]: osdmap e190: 12 total, 12 up, 12 in 2024-02-09T17:29:25.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:25 smithi179 ceph-mon[43975]: pgmap v570: 17 pgs: 1 peering, 2 active+recovering+undersized+remapped, 14 active+clean; 577 KiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:25.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:25 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:29:25.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:25 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:29:25.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:25 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3901275451' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:29:25.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:25 smithi162 ceph-mon[44304]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:29:25.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:25 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3717498605' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:29:25.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:25 smithi162 ceph-mon[44304]: osdmap e190: 12 total, 12 up, 12 in 2024-02-09T17:29:25.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:25 smithi162 ceph-mon[44304]: pgmap v570: 17 pgs: 1 peering, 2 active+recovering+undersized+remapped, 14 active+clean; 577 KiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:25.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:25 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:29:25.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:25 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:29:25.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:25 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3901275451' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:29:26.117 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:29:26.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:26.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:29:26.221 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:29:26.251 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:26.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:26 smithi099 ceph-mon[39420]: osdmap e191: 12 total, 12 up, 12 in 2024-02-09T17:29:26.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:26 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-02-09T17:29:26.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:26 smithi099 ceph-mon[39420]: osdmap e192: 12 total, 12 up, 12 in 2024-02-09T17:29:26.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:26 smithi179 ceph-mon[43975]: osdmap e191: 12 total, 12 up, 12 in 2024-02-09T17:29:26.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:26 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-02-09T17:29:26.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:26 smithi179 ceph-mon[43975]: osdmap e192: 12 total, 12 up, 12 in 2024-02-09T17:29:26.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:26 smithi162 ceph-mon[44304]: osdmap e191: 12 total, 12 up, 12 in 2024-02-09T17:29:26.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:26 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-02-09T17:29:26.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:26 smithi162 ceph-mon[44304]: osdmap e192: 12 total, 12 up, 12 in 2024-02-09T17:29:27.266 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'unique_pool_0' removed 2024-02-09T17:29:27.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:27 smithi099 ceph-mon[39420]: pgmap v573: 17 pgs: 1 peering, 1 active+recovering+undersized+remapped, 15 active+clean; 577 KiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 276 objects/s recovering 2024-02-09T17:29:27.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:27 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3545837098' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-09T17:29:27.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:27 smithi179 ceph-mon[43975]: pgmap v573: 17 pgs: 1 peering, 1 active+recovering+undersized+remapped, 15 active+clean; 577 KiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 276 objects/s recovering 2024-02-09T17:29:27.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:27 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3545837098' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-09T17:29:27.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:27 smithi162 ceph-mon[44304]: pgmap v573: 17 pgs: 1 peering, 1 active+recovering+undersized+remapped, 15 active+clean; 577 KiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 276 objects/s recovering 2024-02-09T17:29:27.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:27 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3545837098' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-09T17:29:28.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:28 smithi179 ceph-mon[43975]: 2.e scrub starts 2024-02-09T17:29:28.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:28 smithi179 ceph-mon[43975]: 2.e scrub ok 2024-02-09T17:29:28.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:28 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3545837098' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-02-09T17:29:28.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:28 smithi179 ceph-mon[43975]: osdmap e193: 12 total, 12 up, 12 in 2024-02-09T17:29:28.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:28 smithi162 ceph-mon[44304]: 2.e scrub starts 2024-02-09T17:29:28.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:28 smithi162 ceph-mon[44304]: 2.e scrub ok 2024-02-09T17:29:28.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:28 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3545837098' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-02-09T17:29:28.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:28 smithi162 ceph-mon[44304]: osdmap e193: 12 total, 12 up, 12 in 2024-02-09T17:29:28.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:28 smithi099 ceph-mon[39420]: 2.e scrub starts 2024-02-09T17:29:28.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:28 smithi099 ceph-mon[39420]: 2.e scrub ok 2024-02-09T17:29:28.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:28 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3545837098' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-02-09T17:29:28.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:28 smithi099 ceph-mon[39420]: osdmap e193: 12 total, 12 up, 12 in 2024-02-09T17:29:28.822 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:29:29.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:29 smithi179 ceph-mon[43975]: pgmap v575: 1 pgs: 1 active+clean; 577 KiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:29.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:29 smithi162 ceph-mon[44304]: pgmap v575: 1 pgs: 1 active+clean; 577 KiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:29.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:29 smithi099 ceph-mon[39420]: pgmap v575: 1 pgs: 1 active+clean; 577 KiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:30.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:30 smithi179 ceph-mon[43975]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-02-09T17:29:30.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:30 smithi162 ceph-mon[44304]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-02-09T17:29:30.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:30 smithi099 ceph-mon[39420]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-02-09T17:29:31.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:31 smithi179 ceph-mon[43975]: pgmap v576: 1 pgs: 1 active+clean; 577 KiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:31.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:31 smithi179 ceph-mon[43975]: osdmap e194: 12 total, 12 up, 12 in 2024-02-09T17:29:31.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:31 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:29:31.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:31 smithi162 ceph-mon[44304]: pgmap v576: 1 pgs: 1 active+clean; 577 KiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:31.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:31 smithi162 ceph-mon[44304]: osdmap e194: 12 total, 12 up, 12 in 2024-02-09T17:29:31.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:31 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:29:31.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:31 smithi099 ceph-mon[39420]: pgmap v576: 1 pgs: 1 active+clean; 577 KiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:31.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:31 smithi099 ceph-mon[39420]: osdmap e194: 12 total, 12 up, 12 in 2024-02-09T17:29:31.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:31 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:29:32.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:32 smithi179 ceph-mon[43975]: 1.0 scrub starts 2024-02-09T17:29:32.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:32 smithi179 ceph-mon[43975]: 1.0 scrub ok 2024-02-09T17:29:32.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:32 smithi162 ceph-mon[44304]: 1.0 scrub starts 2024-02-09T17:29:32.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:32 smithi162 ceph-mon[44304]: 1.0 scrub ok 2024-02-09T17:29:32.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:32 smithi099 ceph-mon[39420]: 1.0 scrub starts 2024-02-09T17:29:32.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:32 smithi099 ceph-mon[39420]: 1.0 scrub ok 2024-02-09T17:29:33.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:33 smithi179 ceph-mon[43975]: pgmap v578: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:33.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:33 smithi162 ceph-mon[44304]: pgmap v578: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:33.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:33 smithi099 ceph-mon[39420]: pgmap v578: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:35.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:35 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:29:35.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:35 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:29:35.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:35 smithi179 ceph-mon[43975]: osdmap e195: 12 total, 12 up, 12 in 2024-02-09T17:29:35.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:35 smithi179 ceph-mon[43975]: pgmap v580: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:35.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:35 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:29:35.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:35 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:29:35.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:35 smithi162 ceph-mon[44304]: osdmap e195: 12 total, 12 up, 12 in 2024-02-09T17:29:35.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:35 smithi162 ceph-mon[44304]: pgmap v580: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:35.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:35 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:29:35.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:35 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:29:35.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:35 smithi099 ceph-mon[39420]: osdmap e195: 12 total, 12 up, 12 in 2024-02-09T17:29:35.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:35 smithi099 ceph-mon[39420]: pgmap v580: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:37.444 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:29:37.446 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:37 smithi099 ceph-mon[39420]: osdmap e196: 12 total, 12 up, 12 in 2024-02-09T17:29:37.446 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:37 smithi099 ceph-mon[39420]: pgmap v582: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:37.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:37.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:29:37.549 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:29:37.578 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:37 smithi179 ceph-mon[43975]: osdmap e196: 12 total, 12 up, 12 in 2024-02-09T17:29:37.578 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:37 smithi179 ceph-mon[43975]: pgmap v582: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:37.581 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:37.602 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:29:37.605 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:37.606 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:29:37.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:37 smithi162 ceph-mon[44304]: osdmap e196: 12 total, 12 up, 12 in 2024-02-09T17:29:37.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:37 smithi162 ceph-mon[44304]: pgmap v582: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:37.707 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:29:37.739 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:37.760 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:29:37.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:37.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:29:37.864 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:29:37.896 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:37.917 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:29:37.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:37.920 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:29:38.021 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:29:38.051 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:38.071 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:29:38.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:38.074 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:29:38.175 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:29:38.207 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:38.228 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:29:38.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:38.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:29:38.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:38 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:29:38.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:38 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:29:38.332 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:29:38.339 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-02-09T17:29:38.340 INFO:tasks.radosbench:Beginning radosbench... 2024-02-09T17:29:38.341 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-02-09T17:29:38.341 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd pool create unique_pool_1 16 2024-02-09T17:29:38.362 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:38.383 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:29:38.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:38.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:29:38.487 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:29:38.513 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:38.594 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:29:38.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:38.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:29:38.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:38 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:29:38.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:38 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:29:38.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:38 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:29:38.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:38 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:29:38.699 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:29:38.704 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-09T17:29:38.727 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:38.781 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:29:38.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:38.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:29:38.886 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:29:38.918 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:38.936 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:29:38.937 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 20, 2024-02-09T17:29:38.937 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 600, 2024-02-09T17:29:38.937 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [ 2024-02-09T17:29:38.937 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.937 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:34129 2.e 2:74e6a5ce:::benchmark_data_smithi179_77138_object34128:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-02-09T17:29:38.937 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:25:01.115737+0000", 2024-02-09T17:29:38.937 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 277.81701346699998, 2024-02-09T17:29:38.937 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.206674047, 2024-02-09T17:29:38.937 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:29:38.937 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:29:38.938 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:29:38.938 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:29:38.938 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:29:38.938 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 34129 2024-02-09T17:29:38.938 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.938 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:29:38.939 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.939 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:29:38.939 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.115737+0000", 2024-02-09T17:29:38.939 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:29:38.939 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.939 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.939 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:29:38.939 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.115737+0000", 2024-02-09T17:29:38.939 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:29:38.939 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.939 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.939 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:29:38.939 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.115739+0000", 2024-02-09T17:29:38.940 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.7e-06 2024-02-09T17:29:38.940 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.940 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.940 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:29:38.940 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.115799+0000", 2024-02-09T17:29:38.941 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.9962e-05 2024-02-09T17:29:38.941 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.941 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.941 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:29:38.941 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.115812+0000", 2024-02-09T17:29:38.941 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3736e-05 2024-02-09T17:29:38.942 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.942 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.942 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:29:38.942 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.115821+0000", 2024-02-09T17:29:38.942 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.0510000000000002e-06 2024-02-09T17:29:38.942 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.943 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.943 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:29:38.943 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.115876+0000", 2024-02-09T17:29:38.943 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.4496000000000001e-05 2024-02-09T17:29:38.943 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.943 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.943 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:29:38.943 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.115970+0000", 2024-02-09T17:29:38.944 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.4052e-05 2024-02-09T17:29:38.944 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.944 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.944 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,9", 2024-02-09T17:29:38.944 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.116102+0000", 2024-02-09T17:29:38.944 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000132247 2024-02-09T17:29:38.944 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.945 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.945 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:29:38.945 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.119108+0000", 2024-02-09T17:29:38.945 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0030056929999999998 2024-02-09T17:29:38.946 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.946 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.946 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:29:38.946 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.121849+0000", 2024-02-09T17:29:38.946 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0027413200000000002 2024-02-09T17:29:38.946 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.946 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.946 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:29:38.946 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.322292+0000", 2024-02-09T17:29:38.946 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.200443226 2024-02-09T17:29:38.946 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.946 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.946 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:29:38.947 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.322358+0000", 2024-02-09T17:29:38.947 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.5629999999999993e-05 2024-02-09T17:29:38.947 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.947 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.947 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:29:38.947 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.322411+0000", 2024-02-09T17:29:38.947 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2933999999999998e-05 2024-02-09T17:29:38.947 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:29:38.947 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:29:38.947 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:29:38.947 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.947 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.947 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:34131 2.1 2:8e8b8e0c:::benchmark_data_smithi179_77138_object34130:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-02-09T17:29:38.948 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:25:01.115889+0000", 2024-02-09T17:29:38.948 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 277.816861733, 2024-02-09T17:29:38.948 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20654767199999999, 2024-02-09T17:29:38.948 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:29:38.948 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:29:38.948 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:29:38.948 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:29:38.948 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:29:38.948 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 34131 2024-02-09T17:29:38.948 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.949 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:29:38.949 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.949 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:29:38.949 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.115889+0000", 2024-02-09T17:29:38.949 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:29:38.949 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.949 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.949 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:29:38.949 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.115889+0000", 2024-02-09T17:29:38.949 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:29:38.949 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.949 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.949 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:29:38.949 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.115890+0000", 2024-02-09T17:29:38.949 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.6300000000000004e-07 2024-02-09T17:29:38.950 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.950 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.950 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:29:38.950 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.115973+0000", 2024-02-09T17:29:38.950 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.3898000000000002e-05 2024-02-09T17:29:38.950 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.950 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.950 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:29:38.950 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.115982+0000", 2024-02-09T17:29:38.950 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.4999999999999999e-06 2024-02-09T17:29:38.950 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.950 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.950 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:29:38.950 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.115991+0000", 2024-02-09T17:29:38.951 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.3440000000000007e-06 2024-02-09T17:29:38.951 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.951 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.951 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:29:38.951 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.116043+0000", 2024-02-09T17:29:38.951 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1861999999999999e-05 2024-02-09T17:29:38.951 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.951 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.951 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:29:38.951 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.116173+0000", 2024-02-09T17:29:38.951 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012961099999999999 2024-02-09T17:29:38.951 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.951 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.951 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,3", 2024-02-09T17:29:38.951 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.116302+0000", 2024-02-09T17:29:38.951 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012955 2024-02-09T17:29:38.951 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.952 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.952 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:29:38.952 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.118828+0000", 2024-02-09T17:29:38.952 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0025252870000000002 2024-02-09T17:29:38.952 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.952 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.952 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:29:38.952 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.119242+0000", 2024-02-09T17:29:38.952 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00041434499999999999 2024-02-09T17:29:38.952 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.952 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.952 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:29:38.952 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.322323+0000", 2024-02-09T17:29:38.953 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.203080811 2024-02-09T17:29:38.953 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.953 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.953 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:29:38.953 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.322381+0000", 2024-02-09T17:29:38.953 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.8187999999999999e-05 2024-02-09T17:29:38.953 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.953 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.953 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:29:38.953 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.322436+0000", 2024-02-09T17:29:38.953 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.5413e-05 2024-02-09T17:29:38.953 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:29:38.953 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:29:38.953 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:29:38.953 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.953 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.954 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:34132 2.1 2:89f3148b:::benchmark_data_smithi179_77138_object34131:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-02-09T17:29:38.954 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:25:01.116006+0000", 2024-02-09T17:29:38.954 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 277.816744363, 2024-02-09T17:29:38.954 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20658180400000001, 2024-02-09T17:29:38.954 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:29:38.954 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:29:38.954 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:29:38.954 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:29:38.954 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:29:38.954 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 34132 2024-02-09T17:29:38.954 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.954 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:29:38.954 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.954 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:29:38.954 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.116006+0000", 2024-02-09T17:29:38.955 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:29:38.955 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.955 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.955 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:29:38.955 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.116006+0000", 2024-02-09T17:29:38.955 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:29:38.955 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.955 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.955 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:29:38.955 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.116007+0000", 2024-02-09T17:29:38.955 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.8999999999999997e-07 2024-02-09T17:29:38.955 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.955 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.955 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:29:38.956 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.116089+0000", 2024-02-09T17:29:38.956 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.2526000000000002e-05 2024-02-09T17:29:38.956 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.956 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.956 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:29:38.956 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.116096+0000", 2024-02-09T17:29:38.956 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.2819999999999999e-06 2024-02-09T17:29:38.956 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.956 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.956 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:29:38.956 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.116103+0000", 2024-02-09T17:29:38.956 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.2949999999999999e-06 2024-02-09T17:29:38.956 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.956 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.957 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:29:38.957 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.117470+0000", 2024-02-09T17:29:38.957 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0013677380000000001 2024-02-09T17:29:38.957 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.957 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.957 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:29:38.957 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.117580+0000", 2024-02-09T17:29:38.957 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000109372 2024-02-09T17:29:38.957 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.957 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.957 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,3", 2024-02-09T17:29:38.957 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.117715+0000", 2024-02-09T17:29:38.957 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000135079 2024-02-09T17:29:38.957 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.958 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.958 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:29:38.958 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.119190+0000", 2024-02-09T17:29:38.958 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.001475557 2024-02-09T17:29:38.958 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.958 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.958 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:29:38.958 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.119248+0000", 2024-02-09T17:29:38.958 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7858000000000001e-05 2024-02-09T17:29:38.959 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.959 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.959 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:29:38.959 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.322496+0000", 2024-02-09T17:29:38.959 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.203247389 2024-02-09T17:29:38.959 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.959 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.959 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:29:38.959 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.322552+0000", 2024-02-09T17:29:38.959 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6094000000000002e-05 2024-02-09T17:29:38.959 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.959 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.959 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:29:38.959 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.322588+0000", 2024-02-09T17:29:38.959 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.6124000000000002e-05 2024-02-09T17:29:38.959 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:29:38.960 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:29:38.960 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:29:38.960 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.960 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.960 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:34139 2.1 2:8872856e:::benchmark_data_smithi179_77138_object34138:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-02-09T17:29:38.960 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:25:01.119878+0000", 2024-02-09T17:29:38.960 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 277.81287242100001, 2024-02-09T17:29:38.960 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.206187176, 2024-02-09T17:29:38.960 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:29:38.960 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:29:38.960 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:29:38.960 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:29:38.960 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:29:38.961 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 34139 2024-02-09T17:29:38.961 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.961 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:29:38.961 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.961 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:29:38.961 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.119878+0000", 2024-02-09T17:29:38.961 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:29:38.961 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.961 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.961 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:29:38.961 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.119878+0000", 2024-02-09T17:29:38.961 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:29:38.961 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.961 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.961 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:29:38.962 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.119879+0000", 2024-02-09T17:29:38.962 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3430000000000001e-06 2024-02-09T17:29:38.962 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.962 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.962 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:29:38.962 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.120180+0000", 2024-02-09T17:29:38.962 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00030018999999999998 2024-02-09T17:29:38.962 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.962 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.962 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:29:38.962 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.120190+0000", 2024-02-09T17:29:38.962 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.023e-05 2024-02-09T17:29:38.962 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.962 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.962 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:29:38.963 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.120196+0000", 2024-02-09T17:29:38.963 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7749999999999998e-06 2024-02-09T17:29:38.963 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.963 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.963 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:29:38.963 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.120248+0000", 2024-02-09T17:29:38.963 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2336000000000002e-05 2024-02-09T17:29:38.963 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.963 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.963 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:29:38.963 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.120369+0000", 2024-02-09T17:29:38.963 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000121557 2024-02-09T17:29:38.963 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.963 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.963 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,3", 2024-02-09T17:29:38.963 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.120517+0000", 2024-02-09T17:29:38.964 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014778600000000001 2024-02-09T17:29:38.964 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.964 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.964 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:29:38.964 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.121946+0000", 2024-02-09T17:29:38.964 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00142847 2024-02-09T17:29:38.964 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.964 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.964 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:29:38.964 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.122597+0000", 2024-02-09T17:29:38.964 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00065173300000000004 2024-02-09T17:29:38.964 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.964 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.965 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:29:38.965 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.325959+0000", 2024-02-09T17:29:38.965 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.203361975 2024-02-09T17:29:38.965 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.965 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.965 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:29:38.965 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.326028+0000", 2024-02-09T17:29:38.965 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.8308999999999999e-05 2024-02-09T17:29:38.965 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.965 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.966 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:29:38.966 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.326065+0000", 2024-02-09T17:29:38.966 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.7472000000000003e-05 2024-02-09T17:29:38.966 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:29:38.966 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:29:38.966 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:29:38.966 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.966 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.966 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:34146 2.e 2:7b1b2f4a:::benchmark_data_smithi179_77138_object34145:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-02-09T17:29:38.966 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:25:01.123983+0000", 2024-02-09T17:29:38.966 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 277.80876790600001, 2024-02-09T17:29:38.966 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20148813600000001, 2024-02-09T17:29:38.966 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:29:38.966 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:29:38.966 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:29:38.967 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:29:38.967 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:29:38.967 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 34146 2024-02-09T17:29:38.967 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.967 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:29:38.967 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.967 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:29:38.967 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.123983+0000", 2024-02-09T17:29:38.967 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:29:38.967 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.967 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.967 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:29:38.967 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.123983+0000", 2024-02-09T17:29:38.967 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:29:38.967 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.968 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.968 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:29:38.968 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.123984+0000", 2024-02-09T17:29:38.968 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5489999999999999e-06 2024-02-09T17:29:38.968 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.968 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.968 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:29:38.968 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.124126+0000", 2024-02-09T17:29:38.968 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000142075 2024-02-09T17:29:38.968 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.968 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.968 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:29:38.968 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.124142+0000", 2024-02-09T17:29:38.969 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.6082e-05 2024-02-09T17:29:38.969 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.969 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.969 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:29:38.969 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.124151+0000", 2024-02-09T17:29:38.969 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.4460000000000001e-06 2024-02-09T17:29:38.969 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.969 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.969 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:29:38.969 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.124199+0000", 2024-02-09T17:29:38.969 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.7917000000000002e-05 2024-02-09T17:29:38.969 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.969 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.969 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:29:38.969 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.124301+0000", 2024-02-09T17:29:38.970 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000102429 2024-02-09T17:29:38.970 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.970 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.970 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,9", 2024-02-09T17:29:38.970 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.124441+0000", 2024-02-09T17:29:38.970 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000139639 2024-02-09T17:29:38.970 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.970 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.970 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:29:38.970 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.125658+0000", 2024-02-09T17:29:38.970 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0012172319999999999 2024-02-09T17:29:38.970 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.970 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.971 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:29:38.971 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.129228+0000", 2024-02-09T17:29:38.971 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0035698040000000002 2024-02-09T17:29:38.971 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.971 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.971 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:29:38.971 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.325379+0000", 2024-02-09T17:29:38.971 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.19615143299999999 2024-02-09T17:29:38.971 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.971 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.971 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:29:38.971 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.325432+0000", 2024-02-09T17:29:38.971 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2777e-05 2024-02-09T17:29:38.971 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.971 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.972 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:29:38.972 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.325471+0000", 2024-02-09T17:29:38.972 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.8753e-05 2024-02-09T17:29:38.972 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:29:38.972 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:29:38.972 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:29:38.972 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.972 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.972 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:34153 2.1 2:8ce2f7a5:::benchmark_data_smithi179_77138_object34152:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-02-09T17:29:38.972 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:25:01.128178+0000", 2024-02-09T17:29:38.972 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 277.804572044, 2024-02-09T17:29:38.972 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.19945995799999999, 2024-02-09T17:29:38.973 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:29:38.973 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:29:38.973 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:29:38.973 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:29:38.973 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:29:38.973 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 34153 2024-02-09T17:29:38.973 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.973 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:29:38.973 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.973 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:29:38.973 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.128178+0000", 2024-02-09T17:29:38.973 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:29:38.973 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.973 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.974 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:29:38.974 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.128178+0000", 2024-02-09T17:29:38.974 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:29:38.974 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.974 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.974 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:29:38.974 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.128180+0000", 2024-02-09T17:29:38.974 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1489999999999999e-06 2024-02-09T17:29:38.974 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.974 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.974 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:29:38.974 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.128295+0000", 2024-02-09T17:29:38.974 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000115677 2024-02-09T17:29:38.974 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.974 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.975 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:29:38.975 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.128308+0000", 2024-02-09T17:29:38.975 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2961e-05 2024-02-09T17:29:38.975 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.975 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.975 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:29:38.975 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.128316+0000", 2024-02-09T17:29:38.975 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.2109999999999998e-06 2024-02-09T17:29:38.975 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.975 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.975 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:29:38.975 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.128359+0000", 2024-02-09T17:29:38.975 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2911999999999998e-05 2024-02-09T17:29:38.975 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.976 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.976 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:29:38.976 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.128449+0000", 2024-02-09T17:29:38.976 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.9265000000000007e-05 2024-02-09T17:29:38.976 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.976 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.976 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,3", 2024-02-09T17:29:38.977 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.128533+0000", 2024-02-09T17:29:38.977 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.4869000000000003e-05 2024-02-09T17:29:38.977 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.977 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.977 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:29:38.977 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.129584+0000", 2024-02-09T17:29:38.977 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0010507369999999999 2024-02-09T17:29:38.977 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.977 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.977 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:29:38.977 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.130833+0000", 2024-02-09T17:29:38.977 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.001248386 2024-02-09T17:29:38.977 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.977 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.978 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:29:38.978 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.327537+0000", 2024-02-09T17:29:38.978 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.196704882 2024-02-09T17:29:38.978 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.978 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.978 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:29:38.978 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.327597+0000", 2024-02-09T17:29:38.978 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.9546000000000002e-05 2024-02-09T17:29:38.978 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.978 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.978 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:29:38.978 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.327638+0000", 2024-02-09T17:29:38.978 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.1362999999999997e-05 2024-02-09T17:29:38.979 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:29:38.979 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:29:38.979 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:29:38.979 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.979 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.979 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:34172 2.e 2:755c0a2b:::benchmark_data_smithi179_77138_object34171:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-02-09T17:29:38.979 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:25:01.137230+0000", 2024-02-09T17:29:38.979 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 277.795520485, 2024-02-09T17:29:38.979 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.18963250400000001, 2024-02-09T17:29:38.979 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:29:38.979 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:29:38.979 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:29:38.979 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:29:38.979 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:29:38.980 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 34172 2024-02-09T17:29:38.980 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.980 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:29:38.980 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.980 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:29:38.980 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.137230+0000", 2024-02-09T17:29:38.980 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:29:38.980 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.980 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.980 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:29:38.980 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.137230+0000", 2024-02-09T17:29:38.980 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:29:38.980 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.981 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.981 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:29:38.981 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.137231+0000", 2024-02-09T17:29:38.981 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2729999999999999e-06 2024-02-09T17:29:38.981 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.981 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.981 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:29:38.981 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.137385+0000", 2024-02-09T17:29:38.981 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000153318 2024-02-09T17:29:38.981 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.981 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.981 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:29:38.981 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.137402+0000", 2024-02-09T17:29:38.981 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.7674000000000001e-05 2024-02-09T17:29:38.982 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.982 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.982 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:29:38.982 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.137410+0000", 2024-02-09T17:29:38.982 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.9570000000000002e-06 2024-02-09T17:29:38.982 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.982 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.982 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:29:38.982 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.137470+0000", 2024-02-09T17:29:38.982 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.0183999999999998e-05 2024-02-09T17:29:38.982 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.982 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.982 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:29:38.982 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.137646+0000", 2024-02-09T17:29:38.983 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00017514399999999999 2024-02-09T17:29:38.983 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.983 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.983 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,9", 2024-02-09T17:29:38.983 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.137796+0000", 2024-02-09T17:29:38.983 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00015012000000000001 2024-02-09T17:29:38.984 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.984 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.984 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:29:38.984 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.139302+0000", 2024-02-09T17:29:38.984 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0015068289999999999 2024-02-09T17:29:38.984 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.985 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.985 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:29:38.985 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.139596+0000", 2024-02-09T17:29:38.985 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00029373099999999998 2024-02-09T17:29:38.985 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.985 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.985 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:29:38.985 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.326763+0000", 2024-02-09T17:29:38.985 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.18716690499999999 2024-02-09T17:29:38.985 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.985 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.985 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:29:38.985 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.326824+0000", 2024-02-09T17:29:38.986 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.1100999999999997e-05 2024-02-09T17:29:38.986 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.986 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.986 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:29:38.986 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.326862+0000", 2024-02-09T17:29:38.986 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.8268e-05 2024-02-09T17:29:38.987 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:29:38.987 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:29:38.987 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:29:38.987 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.987 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.987 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:34195 2.1 2:8834143b:::benchmark_data_smithi179_77138_object34194:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-02-09T17:29:38.987 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:25:01.147437+0000", 2024-02-09T17:29:38.987 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 277.78531341600001, 2024-02-09T17:29:38.987 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.18031862100000001, 2024-02-09T17:29:38.988 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:29:38.988 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:29:38.988 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:29:38.988 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:29:38.988 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:29:38.988 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 34195 2024-02-09T17:29:38.988 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.988 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:29:38.989 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.989 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:29:38.989 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.147437+0000", 2024-02-09T17:29:38.989 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:29:38.989 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.989 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.989 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:29:38.989 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.147437+0000", 2024-02-09T17:29:38.989 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:29:38.989 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.989 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.989 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:29:38.989 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.147438+0000", 2024-02-09T17:29:38.989 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.305e-06 2024-02-09T17:29:38.989 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.990 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.990 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:29:38.990 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.147503+0000", 2024-02-09T17:29:38.990 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.4379000000000001e-05 2024-02-09T17:29:38.990 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.990 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.990 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:29:38.991 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.147515+0000", 2024-02-09T17:29:38.991 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1917e-05 2024-02-09T17:29:38.991 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.991 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.991 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:29:38.991 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.147521+0000", 2024-02-09T17:29:38.992 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.5139999999999998e-06 2024-02-09T17:29:38.992 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.992 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.992 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:29:38.992 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.147580+0000", 2024-02-09T17:29:38.992 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.8928999999999998e-05 2024-02-09T17:29:38.993 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.993 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.993 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:29:38.993 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.147708+0000", 2024-02-09T17:29:38.993 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012768400000000001 2024-02-09T17:29:38.993 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.993 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.993 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,3", 2024-02-09T17:29:38.993 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.147962+0000", 2024-02-09T17:29:38.993 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00025464599999999997 2024-02-09T17:29:38.994 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.994 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.994 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:29:38.994 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.150003+0000", 2024-02-09T17:29:38.994 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0020407580000000002 2024-02-09T17:29:38.994 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.994 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.994 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:29:38.995 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.150209+0000", 2024-02-09T17:29:38.995 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00020579600000000001 2024-02-09T17:29:38.995 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.995 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.995 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:29:38.995 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.327676+0000", 2024-02-09T17:29:38.995 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.17746682699999999 2024-02-09T17:29:38.995 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.995 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.995 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:29:38.996 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.327725+0000", 2024-02-09T17:29:38.996 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.9310000000000001e-05 2024-02-09T17:29:38.996 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.996 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.996 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:29:38.996 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.327756+0000", 2024-02-09T17:29:38.996 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.0555999999999997e-05 2024-02-09T17:29:38.996 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:29:38.996 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:29:38.996 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:29:38.996 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.996 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.996 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:34225 2.1 2:8ca4ebbe:::benchmark_data_smithi179_77138_object34224:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-02-09T17:29:38.997 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:25:01.160563+0000", 2024-02-09T17:29:38.997 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 277.77218747000001, 2024-02-09T17:29:38.997 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.16730151400000001, 2024-02-09T17:29:38.997 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:29:38.997 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:29:38.997 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:29:38.997 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:29:38.997 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:29:38.997 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 34225 2024-02-09T17:29:38.997 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.997 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:29:38.998 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.998 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:29:38.998 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.160563+0000", 2024-02-09T17:29:38.998 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:29:38.998 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.998 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.998 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:29:38.998 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.160563+0000", 2024-02-09T17:29:38.998 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:29:38.998 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.998 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.998 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:29:38.998 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.160564+0000", 2024-02-09T17:29:38.998 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.372e-06 2024-02-09T17:29:38.999 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.999 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.999 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:29:38.999 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.160705+0000", 2024-02-09T17:29:38.999 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014037500000000001 2024-02-09T17:29:38.999 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:38.999 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:38.999 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:29:38.999 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.160722+0000", 2024-02-09T17:29:38.999 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.753e-05 2024-02-09T17:29:39.000 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.000 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.000 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:29:39.000 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.160731+0000", 2024-02-09T17:29:39.000 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.8799999999999997e-06 2024-02-09T17:29:39.000 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.001 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.001 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:29:39.001 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.160794+0000", 2024-02-09T17:29:39.001 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.2426000000000002e-05 2024-02-09T17:29:39.001 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.001 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.001 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:29:39.002 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.160890+0000", 2024-02-09T17:29:39.002 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.6496000000000006e-05 2024-02-09T17:29:39.002 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.002 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.002 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,3", 2024-02-09T17:29:39.002 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.160991+0000", 2024-02-09T17:29:39.002 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00010101 2024-02-09T17:29:39.002 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.002 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.002 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:29:39.002 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.162206+0000", 2024-02-09T17:29:39.003 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.001214964 2024-02-09T17:29:39.003 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.003 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.003 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:29:39.003 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.163243+0000", 2024-02-09T17:29:39.003 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.001036946 2024-02-09T17:29:39.003 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.003 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.003 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:29:39.003 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.327791+0000", 2024-02-09T17:29:39.003 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.16454820000000001 2024-02-09T17:29:39.003 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.003 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.003 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:29:39.004 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.327834+0000", 2024-02-09T17:29:39.004 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.3198999999999998e-05 2024-02-09T17:29:39.004 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.004 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.004 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:29:39.004 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.327865+0000", 2024-02-09T17:29:39.004 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.0116000000000001e-05 2024-02-09T17:29:39.004 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:29:39.004 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:29:39.004 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:29:39.004 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.004 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.004 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:34231 2.e 2:7edadf75:::benchmark_data_smithi179_77138_object34230:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-02-09T17:29:39.004 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:25:01.163465+0000", 2024-02-09T17:29:39.005 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 277.76928527799998, 2024-02-09T17:29:39.005 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.16354817199999999, 2024-02-09T17:29:39.005 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:29:39.005 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:29:39.005 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:29:39.005 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:29:39.005 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:29:39.005 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 34231 2024-02-09T17:29:39.005 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.005 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:29:39.005 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.005 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:29:39.005 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.163465+0000", 2024-02-09T17:29:39.005 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:29:39.005 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.005 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.006 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:29:39.006 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.163465+0000", 2024-02-09T17:29:39.006 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:29:39.006 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.006 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.006 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:29:39.006 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.163466+0000", 2024-02-09T17:29:39.006 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0899999999999999e-06 2024-02-09T17:29:39.006 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.006 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.006 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:29:39.006 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.163556+0000", 2024-02-09T17:29:39.006 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.9988999999999995e-05 2024-02-09T17:29:39.006 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.006 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.007 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:29:39.007 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.163566+0000", 2024-02-09T17:29:39.007 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.8220000000000002e-06 2024-02-09T17:29:39.007 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.007 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.007 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:29:39.007 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.163572+0000", 2024-02-09T17:29:39.007 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.4879999999999998e-06 2024-02-09T17:29:39.007 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.007 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.007 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:29:39.007 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.163616+0000", 2024-02-09T17:29:39.007 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.3948e-05 2024-02-09T17:29:39.007 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.007 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.008 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:29:39.008 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.163737+0000", 2024-02-09T17:29:39.008 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012142300000000001 2024-02-09T17:29:39.008 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.008 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.008 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,9", 2024-02-09T17:29:39.008 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.163848+0000", 2024-02-09T17:29:39.008 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000111451 2024-02-09T17:29:39.008 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.008 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.008 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:29:39.008 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.165131+0000", 2024-02-09T17:29:39.008 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.001282954 2024-02-09T17:29:39.008 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.009 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.009 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:29:39.009 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.166132+0000", 2024-02-09T17:29:39.009 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0010011149999999999 2024-02-09T17:29:39.009 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.009 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.009 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:29:39.009 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.326915+0000", 2024-02-09T17:29:39.009 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.160782593 2024-02-09T17:29:39.009 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.009 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.009 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:29:39.009 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.326976+0000", 2024-02-09T17:29:39.009 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.0897999999999999e-05 2024-02-09T17:29:39.009 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.010 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.010 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:29:39.010 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.327013+0000", 2024-02-09T17:29:39.010 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.7401000000000003e-05 2024-02-09T17:29:39.010 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:29:39.010 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:29:39.010 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:29:39.010 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.010 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.010 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:34238 2.1 2:84d4c0d4:::benchmark_data_smithi179_77138_object34237:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-02-09T17:29:39.010 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:25:01.166477+0000", 2024-02-09T17:29:39.010 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 277.76627365399997, 2024-02-09T17:29:39.010 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.16232606699999999, 2024-02-09T17:29:39.010 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:29:39.011 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:29:39.011 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:29:39.011 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:29:39.011 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:29:39.011 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 34238 2024-02-09T17:29:39.011 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.011 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:29:39.011 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.011 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:29:39.011 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.166477+0000", 2024-02-09T17:29:39.011 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:29:39.011 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.011 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.012 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:29:39.012 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.166477+0000", 2024-02-09T17:29:39.012 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:29:39.012 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.012 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.012 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:29:39.012 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.166478+0000", 2024-02-09T17:29:39.012 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0640000000000001e-06 2024-02-09T17:29:39.013 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.013 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.013 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:29:39.013 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.166623+0000", 2024-02-09T17:29:39.013 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014474600000000001 2024-02-09T17:29:39.013 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.013 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.013 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:29:39.013 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.166635+0000", 2024-02-09T17:29:39.013 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2762000000000001e-05 2024-02-09T17:29:39.013 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.014 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.014 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:29:39.014 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.166642+0000", 2024-02-09T17:29:39.014 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.0169999999999999e-06 2024-02-09T17:29:39.014 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.014 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.014 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:29:39.014 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.166698+0000", 2024-02-09T17:29:39.014 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.5825999999999997e-05 2024-02-09T17:29:39.014 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.014 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.015 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:29:39.015 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.166799+0000", 2024-02-09T17:29:39.015 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00010118 2024-02-09T17:29:39.015 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.015 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.015 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,3", 2024-02-09T17:29:39.015 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.166913+0000", 2024-02-09T17:29:39.015 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000113754 2024-02-09T17:29:39.015 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.015 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.015 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:29:39.015 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.168168+0000", 2024-02-09T17:29:39.015 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0012544430000000001 2024-02-09T17:29:39.015 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.015 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.016 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:29:39.016 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.168525+0000", 2024-02-09T17:29:39.016 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00035716300000000002 2024-02-09T17:29:39.016 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.016 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.016 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:29:39.016 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.328713+0000", 2024-02-09T17:29:39.016 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.160188206 2024-02-09T17:29:39.016 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.016 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.016 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:29:39.016 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.328766+0000", 2024-02-09T17:29:39.016 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3288000000000002e-05 2024-02-09T17:29:39.016 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.017 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.017 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:29:39.017 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.328803+0000", 2024-02-09T17:29:39.017 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.6618000000000001e-05 2024-02-09T17:29:39.017 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:29:39.017 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:29:39.017 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:29:39.017 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.017 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.017 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:34246 2.1 2:8e2746d4:::benchmark_data_smithi179_77138_object34245:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-02-09T17:29:39.017 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:25:01.169484+0000", 2024-02-09T17:29:39.017 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 277.76326676600002, 2024-02-09T17:29:39.017 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.15959520499999999, 2024-02-09T17:29:39.018 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:29:39.018 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:29:39.018 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:29:39.018 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:29:39.018 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:29:39.018 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 34246 2024-02-09T17:29:39.018 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.018 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:29:39.019 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.019 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:29:39.019 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.169484+0000", 2024-02-09T17:29:39.019 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:29:39.019 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.019 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.019 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:29:39.019 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.169484+0000", 2024-02-09T17:29:39.019 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:29:39.019 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.019 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.019 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:29:39.019 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.169485+0000", 2024-02-09T17:29:39.019 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.268e-06 2024-02-09T17:29:39.019 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.020 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.020 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:29:39.020 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.169611+0000", 2024-02-09T17:29:39.020 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012564400000000001 2024-02-09T17:29:39.020 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.020 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.020 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:29:39.020 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.169623+0000", 2024-02-09T17:29:39.020 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2234e-05 2024-02-09T17:29:39.020 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.020 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.021 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:29:39.021 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.169631+0000", 2024-02-09T17:29:39.021 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.1450000000000008e-06 2024-02-09T17:29:39.021 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.021 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.021 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:29:39.021 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.169673+0000", 2024-02-09T17:29:39.021 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2113999999999997e-05 2024-02-09T17:29:39.021 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.021 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.021 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:29:39.021 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.169755+0000", 2024-02-09T17:29:39.022 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.2223000000000002e-05 2024-02-09T17:29:39.022 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.022 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.022 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,3", 2024-02-09T17:29:39.022 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.169926+0000", 2024-02-09T17:29:39.022 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00017060299999999999 2024-02-09T17:29:39.022 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.022 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.022 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:29:39.022 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.170975+0000", 2024-02-09T17:29:39.022 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.001049063 2024-02-09T17:29:39.022 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.022 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.022 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:29:39.022 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.171300+0000", 2024-02-09T17:29:39.023 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00032474199999999999 2024-02-09T17:29:39.023 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.023 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.023 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:29:39.023 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.328996+0000", 2024-02-09T17:29:39.023 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.15769657300000001 2024-02-09T17:29:39.023 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.023 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.023 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:29:39.023 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.329045+0000", 2024-02-09T17:29:39.023 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.8544000000000001e-05 2024-02-09T17:29:39.023 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.024 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.024 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:29:39.024 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.329079+0000", 2024-02-09T17:29:39.024 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.4051999999999999e-05 2024-02-09T17:29:39.024 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:29:39.024 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:29:39.025 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:29:39.025 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.025 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.025 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:34247 2.e 2:757500ad:::benchmark_data_smithi179_77138_object34246:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-02-09T17:29:39.025 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:25:01.171083+0000", 2024-02-09T17:29:39.026 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 277.76166792599997, 2024-02-09T17:29:39.026 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.156083956, 2024-02-09T17:29:39.026 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:29:39.026 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:29:39.026 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:29:39.026 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:29:39.027 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:29:39.027 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 34247 2024-02-09T17:29:39.027 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.027 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:29:39.027 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.028 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:29:39.028 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.171083+0000", 2024-02-09T17:29:39.028 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:29:39.028 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.028 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.028 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:29:39.028 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.171083+0000", 2024-02-09T17:29:39.028 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:29:39.028 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.029 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.029 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:29:39.029 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.171083+0000", 2024-02-09T17:29:39.029 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.8899999999999998e-07 2024-02-09T17:29:39.029 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.029 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.030 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:29:39.030 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.171178+0000", 2024-02-09T17:29:39.030 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.4858000000000006e-05 2024-02-09T17:29:39.030 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.030 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.030 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:29:39.030 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.171189+0000", 2024-02-09T17:29:39.030 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0461e-05 2024-02-09T17:29:39.031 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.031 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.031 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:29:39.031 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.171194+0000", 2024-02-09T17:29:39.031 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6729999999999997e-06 2024-02-09T17:29:39.031 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.031 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.032 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:29:39.032 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.171235+0000", 2024-02-09T17:29:39.032 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.0166000000000001e-05 2024-02-09T17:29:39.032 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.032 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.032 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:29:39.032 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.171322+0000", 2024-02-09T17:29:39.032 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.7569000000000001e-05 2024-02-09T17:29:39.032 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.032 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.032 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,9", 2024-02-09T17:29:39.032 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.171418+0000", 2024-02-09T17:29:39.033 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.6111000000000002e-05 2024-02-09T17:29:39.033 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.033 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.033 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:29:39.033 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.172409+0000", 2024-02-09T17:29:39.033 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00099119799999999999 2024-02-09T17:29:39.033 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.033 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.033 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:29:39.033 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.172840+0000", 2024-02-09T17:29:39.033 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00043027899999999999 2024-02-09T17:29:39.033 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.034 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.034 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:29:39.034 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.327055+0000", 2024-02-09T17:29:39.034 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.154215353 2024-02-09T17:29:39.034 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.034 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.035 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:29:39.035 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.327125+0000", 2024-02-09T17:29:39.035 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.9469999999999997e-05 2024-02-09T17:29:39.035 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.035 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.035 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:29:39.035 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.327166+0000", 2024-02-09T17:29:39.035 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.1928999999999998e-05 2024-02-09T17:29:39.035 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:29:39.035 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:29:39.035 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:29:39.035 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.036 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.036 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:34266 2.e 2:7944c0da:::benchmark_data_smithi179_77138_object34265:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-02-09T17:29:39.036 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:25:01.184096+0000", 2024-02-09T17:29:39.036 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 277.74865452099999, 2024-02-09T17:29:39.036 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.14485842900000001, 2024-02-09T17:29:39.036 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:29:39.036 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:29:39.037 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:29:39.037 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:29:39.037 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:29:39.037 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 34266 2024-02-09T17:29:39.037 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.038 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:29:39.038 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.038 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:29:39.038 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.184096+0000", 2024-02-09T17:29:39.038 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:29:39.038 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.039 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.039 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:29:39.039 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.184096+0000", 2024-02-09T17:29:39.039 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:29:39.039 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.039 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.039 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:29:39.039 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.184098+0000", 2024-02-09T17:29:39.039 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5939999999999999e-06 2024-02-09T17:29:39.039 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.039 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.040 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:29:39.040 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.184153+0000", 2024-02-09T17:29:39.040 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.5625999999999999e-05 2024-02-09T17:29:39.040 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.040 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.040 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:29:39.040 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.184167+0000", 2024-02-09T17:29:39.041 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4087000000000001e-05 2024-02-09T17:29:39.041 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.041 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.041 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:29:39.041 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.184175+0000", 2024-02-09T17:29:39.041 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.4989999999999997e-06 2024-02-09T17:29:39.041 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.041 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.041 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:29:39.041 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.184224+0000", 2024-02-09T17:29:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.8996999999999999e-05 2024-02-09T17:29:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:29:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.184319+0000", 2024-02-09T17:29:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.5542000000000001e-05 2024-02-09T17:29:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,9", 2024-02-09T17:29:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.184419+0000", 2024-02-09T17:29:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00010006299999999999 2024-02-09T17:29:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:29:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.185556+0000", 2024-02-09T17:29:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0011365360000000001 2024-02-09T17:29:39.043 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.043 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.043 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:29:39.043 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.185784+0000", 2024-02-09T17:29:39.043 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00022769899999999999 2024-02-09T17:29:39.043 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.043 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.043 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:29:39.043 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.328863+0000", 2024-02-09T17:29:39.043 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.14307929899999999 2024-02-09T17:29:39.043 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.043 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.044 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:29:39.044 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.328915+0000", 2024-02-09T17:29:39.044 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2448999999999999e-05 2024-02-09T17:29:39.044 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.044 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.044 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:29:39.044 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.328954+0000", 2024-02-09T17:29:39.044 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9038000000000001e-05 2024-02-09T17:29:39.044 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:29:39.044 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:29:39.044 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:29:39.044 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.044 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.045 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:34274 2.e 2:7faa851f:::benchmark_data_smithi179_77138_object34273:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-02-09T17:29:39.045 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:25:01.190577+0000", 2024-02-09T17:29:39.045 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 277.74217353, 2024-02-09T17:29:39.045 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.138482838, 2024-02-09T17:29:39.045 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:29:39.045 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:29:39.045 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:29:39.045 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:29:39.045 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:29:39.045 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 34274 2024-02-09T17:29:39.045 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.045 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:29:39.045 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.046 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:29:39.046 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.190577+0000", 2024-02-09T17:29:39.046 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:29:39.046 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.046 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.046 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:29:39.046 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.190577+0000", 2024-02-09T17:29:39.046 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:29:39.046 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.046 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.047 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:29:39.047 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.190578+0000", 2024-02-09T17:29:39.047 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.4099999999999997e-07 2024-02-09T17:29:39.047 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.047 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.047 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:29:39.047 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.190654+0000", 2024-02-09T17:29:39.047 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.6178000000000001e-05 2024-02-09T17:29:39.047 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.047 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.047 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:29:39.047 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.190664+0000", 2024-02-09T17:29:39.047 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0261000000000001e-05 2024-02-09T17:29:39.048 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.048 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.048 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:29:39.048 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.190670+0000", 2024-02-09T17:29:39.048 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.5829999999999997e-06 2024-02-09T17:29:39.048 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.048 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.048 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:29:39.048 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.190711+0000", 2024-02-09T17:29:39.048 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.1298000000000003e-05 2024-02-09T17:29:39.048 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.049 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.049 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:29:39.049 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.190806+0000", 2024-02-09T17:29:39.049 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.5067000000000003e-05 2024-02-09T17:29:39.049 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.049 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.049 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,9", 2024-02-09T17:29:39.049 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.190897+0000", 2024-02-09T17:29:39.049 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.1230000000000006e-05 2024-02-09T17:29:39.049 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.049 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.049 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:29:39.049 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.191906+0000", 2024-02-09T17:29:39.049 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0010085949999999999 2024-02-09T17:29:39.050 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.050 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.050 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:29:39.050 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.192164+0000", 2024-02-09T17:29:39.050 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00025828299999999999 2024-02-09T17:29:39.050 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.050 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.050 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:29:39.050 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.328988+0000", 2024-02-09T17:29:39.050 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.13682372400000001 2024-02-09T17:29:39.050 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.050 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.050 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:29:39.050 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.329031+0000", 2024-02-09T17:29:39.050 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2899999999999999e-05 2024-02-09T17:29:39.051 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.051 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.051 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:29:39.051 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.329060+0000", 2024-02-09T17:29:39.051 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.8778000000000001e-05 2024-02-09T17:29:39.051 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:29:39.051 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:29:39.051 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:29:39.051 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.051 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.051 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:34282 2.1 2:8dfa6836:::benchmark_data_smithi179_77138_object34281:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-02-09T17:29:39.051 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:25:01.206524+0000", 2024-02-09T17:29:39.051 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 277.726226053, 2024-02-09T17:29:39.051 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.122693835, 2024-02-09T17:29:39.051 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:29:39.052 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:29:39.052 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:29:39.052 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:29:39.052 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:29:39.052 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 34282 2024-02-09T17:29:39.052 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.052 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:29:39.052 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.052 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:29:39.052 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.206524+0000", 2024-02-09T17:29:39.052 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:29:39.052 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.052 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.053 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:29:39.053 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.206524+0000", 2024-02-09T17:29:39.053 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:29:39.053 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.053 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.053 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:29:39.053 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.206526+0000", 2024-02-09T17:29:39.053 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.75e-06 2024-02-09T17:29:39.053 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.053 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.053 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:29:39.053 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.206584+0000", 2024-02-09T17:29:39.053 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.8059000000000001e-05 2024-02-09T17:29:39.054 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.054 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.054 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:29:39.054 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.206600+0000", 2024-02-09T17:29:39.054 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5804e-05 2024-02-09T17:29:39.054 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.054 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.054 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:29:39.054 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.206609+0000", 2024-02-09T17:29:39.054 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.8470000000000002e-06 2024-02-09T17:29:39.054 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.054 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.054 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:29:39.054 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.206659+0000", 2024-02-09T17:29:39.054 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.0506000000000001e-05 2024-02-09T17:29:39.055 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.055 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.055 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:29:39.055 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.206769+0000", 2024-02-09T17:29:39.055 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000109714 2024-02-09T17:29:39.055 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.055 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.055 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,3", 2024-02-09T17:29:39.055 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.206867+0000", 2024-02-09T17:29:39.055 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.7664999999999994e-05 2024-02-09T17:29:39.055 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.055 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.055 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:29:39.055 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.207935+0000", 2024-02-09T17:29:39.055 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0010682420000000001 2024-02-09T17:29:39.056 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.056 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.056 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:29:39.056 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.208115+0000", 2024-02-09T17:29:39.056 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000180087 2024-02-09T17:29:39.056 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.056 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.056 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:29:39.056 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.329141+0000", 2024-02-09T17:29:39.056 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.12102542500000001 2024-02-09T17:29:39.056 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.056 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.056 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:29:39.057 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.329193+0000", 2024-02-09T17:29:39.057 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2769000000000002e-05 2024-02-09T17:29:39.057 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.057 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.057 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:29:39.057 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:01.329218+0000", 2024-02-09T17:29:39.057 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.4967e-05 2024-02-09T17:29:39.057 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:29:39.057 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:29:39.057 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:29:39.057 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.057 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.057 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:155026 2.1 2:85ba9a6e:::benchmark_data_smithi179_77138_object21618:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-02-09T17:29:39.057 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:26:43.558578+0000", 2024-02-09T17:29:39.057 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 175.37417222799999, 2024-02-09T17:29:39.057 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.084151366000000005, 2024-02-09T17:29:39.058 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:29:39.058 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:29:39.058 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:29:39.058 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:29:39.058 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:29:39.058 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 155026 2024-02-09T17:29:39.058 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.058 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:29:39.058 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.058 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:29:39.058 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:43.558578+0000", 2024-02-09T17:29:39.059 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:29:39.059 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.059 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.059 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:29:39.059 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:43.558578+0000", 2024-02-09T17:29:39.059 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:29:39.059 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.059 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.059 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:29:39.059 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:43.558580+0000", 2024-02-09T17:29:39.059 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.28e-06 2024-02-09T17:29:39.059 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.059 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.059 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:29:39.060 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:43.558582+0000", 2024-02-09T17:29:39.060 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.9079999999999999e-06 2024-02-09T17:29:39.060 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.060 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.060 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:29:39.060 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:43.558589+0000", 2024-02-09T17:29:39.060 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.7429999999999996e-06 2024-02-09T17:29:39.060 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.060 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.060 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:29:39.060 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:43.558595+0000", 2024-02-09T17:29:39.060 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7080000000000002e-06 2024-02-09T17:29:39.060 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.061 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.061 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:29:39.061 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:43.566658+0000", 2024-02-09T17:29:39.061 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.008062975 2024-02-09T17:29:39.061 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.061 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.061 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:29:39.061 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:43.566762+0000", 2024-02-09T17:29:39.061 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00010413599999999999 2024-02-09T17:29:39.061 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.061 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.061 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,3", 2024-02-09T17:29:39.061 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:43.566860+0000", 2024-02-09T17:29:39.061 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.7631000000000001e-05 2024-02-09T17:29:39.062 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.062 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.062 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:29:39.062 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:43.577402+0000", 2024-02-09T17:29:39.062 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.010542780999999999 2024-02-09T17:29:39.062 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.062 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.062 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:29:39.062 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:43.597573+0000", 2024-02-09T17:29:39.062 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.02017041 2024-02-09T17:29:39.062 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.062 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.062 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:29:39.062 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:43.642640+0000", 2024-02-09T17:29:39.062 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.045067100999999998 2024-02-09T17:29:39.063 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.063 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.063 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:29:39.063 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:43.642691+0000", 2024-02-09T17:29:39.063 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.0966e-05 2024-02-09T17:29:39.063 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.063 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.063 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:29:39.063 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:43.642730+0000", 2024-02-09T17:29:39.063 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.8726999999999997e-05 2024-02-09T17:29:39.063 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:29:39.063 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:29:39.063 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:29:39.063 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.063 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.064 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:155029 2.1 2:8801e464:::benchmark_data_smithi179_77138_object21621:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-02-09T17:29:39.064 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:26:43.563436+0000", 2024-02-09T17:29:39.064 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 175.36931473300001, 2024-02-09T17:29:39.064 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.079427361000000002, 2024-02-09T17:29:39.064 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:29:39.064 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:29:39.064 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:29:39.064 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:29:39.064 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:29:39.064 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 155029 2024-02-09T17:29:39.064 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.064 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:29:39.064 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.064 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:29:39.065 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:43.563436+0000", 2024-02-09T17:29:39.065 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:29:39.065 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.065 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.065 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:29:39.065 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:43.563436+0000", 2024-02-09T17:29:39.065 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:29:39.065 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.065 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.065 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:29:39.065 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:43.563437+0000", 2024-02-09T17:29:39.065 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4330000000000001e-06 2024-02-09T17:29:39.065 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.065 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.065 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:29:39.065 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:43.563442+0000", 2024-02-09T17:29:39.066 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.3529999999999997e-06 2024-02-09T17:29:39.066 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.066 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.066 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:29:39.066 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:43.563454+0000", 2024-02-09T17:29:39.066 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.234e-05 2024-02-09T17:29:39.066 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.066 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.066 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:29:39.066 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:43.563464+0000", 2024-02-09T17:29:39.066 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.9569999999999997e-06 2024-02-09T17:29:39.066 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.066 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.066 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:29:39.066 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:43.571756+0000", 2024-02-09T17:29:39.067 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0082918539999999995 2024-02-09T17:29:39.067 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.067 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.067 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:29:39.067 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:43.571884+0000", 2024-02-09T17:29:39.067 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012832699999999999 2024-02-09T17:29:39.067 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.067 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.067 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,3", 2024-02-09T17:29:39.067 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:43.571999+0000", 2024-02-09T17:29:39.067 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00011459 2024-02-09T17:29:39.067 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.067 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.067 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:29:39.068 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:43.586193+0000", 2024-02-09T17:29:39.068 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.014194506000000001 2024-02-09T17:29:39.068 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.068 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.068 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:29:39.068 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:43.597610+0000", 2024-02-09T17:29:39.068 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.011416433 2024-02-09T17:29:39.068 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.068 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.068 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:29:39.068 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:43.642775+0000", 2024-02-09T17:29:39.068 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.045165354999999997 2024-02-09T17:29:39.068 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.069 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.069 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:29:39.069 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:43.642827+0000", 2024-02-09T17:29:39.069 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2215000000000001e-05 2024-02-09T17:29:39.069 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.069 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.069 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:29:39.069 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:43.642863+0000", 2024-02-09T17:29:39.069 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.5998000000000003e-05 2024-02-09T17:29:39.069 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:29:39.069 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:29:39.069 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:29:39.069 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.069 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.070 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:238250 2.7 e173/111 2:e6fac0c4:::benchmark_data_smithi179_77138_object104842:head v 173'14701, mlcod=173'14698)", 2024-02-09T17:29:39.070 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:28:22.835303+0000", 2024-02-09T17:29:39.070 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 76.097447473000003, 2024-02-09T17:29:39.070 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.086551764000000003, 2024-02-09T17:29:39.070 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:29:39.070 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:29:39.070 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:29:39.070 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.070 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:29:39.070 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:22.835303+0000", 2024-02-09T17:29:39.070 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:29:39.070 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.070 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.071 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:29:39.071 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:22.835303+0000", 2024-02-09T17:29:39.071 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:29:39.071 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.071 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.071 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:29:39.071 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:22.835304+0000", 2024-02-09T17:29:39.072 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.8700000000000005e-07 2024-02-09T17:29:39.072 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.072 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.072 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:29:39.072 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:22.835312+0000", 2024-02-09T17:29:39.072 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.8220000000000007e-06 2024-02-09T17:29:39.072 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.072 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.072 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:29:39.072 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:22.835322+0000", 2024-02-09T17:29:39.072 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0668e-05 2024-02-09T17:29:39.072 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.072 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.073 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:29:39.073 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:22.835331+0000", 2024-02-09T17:29:39.073 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.9230000000000008e-06 2024-02-09T17:29:39.073 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.073 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.073 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:29:39.074 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:22.840982+0000", 2024-02-09T17:29:39.074 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0056506639999999997 2024-02-09T17:29:39.074 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.074 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.074 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:29:39.075 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:22.841035+0000", 2024-02-09T17:29:39.075 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2834999999999998e-05 2024-02-09T17:29:39.075 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.075 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.075 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:29:39.075 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:22.921798+0000", 2024-02-09T17:29:39.076 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.080763540999999994 2024-02-09T17:29:39.076 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.076 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.076 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:29:39.076 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:22.921855+0000", 2024-02-09T17:29:39.076 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6524000000000003e-05 2024-02-09T17:29:39.076 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:29:39.076 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:29:39.076 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:29:39.076 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.077 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.077 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.14790.0:238256 2.7 e173/111 2:e73e60d4:::benchmark_data_smithi179_77138_object104848:head v 173'14702, mlcod=173'14698)", 2024-02-09T17:29:39.077 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:28:22.839363+0000", 2024-02-09T17:29:39.077 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 76.093387105000005, 2024-02-09T17:29:39.077 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.082522656, 2024-02-09T17:29:39.077 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:29:39.077 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:29:39.078 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:29:39.078 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.078 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:29:39.078 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:22.839363+0000", 2024-02-09T17:29:39.078 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:29:39.078 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.078 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.078 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:29:39.078 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:22.839363+0000", 2024-02-09T17:29:39.078 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:29:39.078 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.079 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.079 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:29:39.079 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:22.839365+0000", 2024-02-09T17:29:39.079 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.469e-06 2024-02-09T17:29:39.079 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.079 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.079 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:29:39.079 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:22.839374+0000", 2024-02-09T17:29:39.079 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.6139999999999998e-06 2024-02-09T17:29:39.079 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.079 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.080 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:29:39.080 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:22.839393+0000", 2024-02-09T17:29:39.080 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.9048000000000001e-05 2024-02-09T17:29:39.080 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.080 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.080 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:29:39.080 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:22.839404+0000", 2024-02-09T17:29:39.080 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0263e-05 2024-02-09T17:29:39.080 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.080 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.080 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:29:39.080 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:22.889829+0000", 2024-02-09T17:29:39.080 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.050425155999999999 2024-02-09T17:29:39.081 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.081 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.081 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:29:39.081 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:22.889858+0000", 2024-02-09T17:29:39.081 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.8867e-05 2024-02-09T17:29:39.081 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.081 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.081 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:29:39.081 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:22.921860+0000", 2024-02-09T17:29:39.081 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.032002190999999999 2024-02-09T17:29:39.081 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:29:39.081 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:29:39.081 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:29:39.081 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:28:22.921886+0000", 2024-02-09T17:29:39.081 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.6047999999999999e-05 2024-02-09T17:29:39.082 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:29:39.082 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:29:39.082 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:29:39.082 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:29:39.082 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:29:39.082 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:29:39.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:39.083 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:29:39.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:29:39.184 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:29:39.216 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:39.237 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:29:39.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:39.241 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:29:39.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:39 smithi099 ceph-mon[39420]: osdmap e197: 12 total, 12 up, 12 in 2024-02-09T17:29:39.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:39 smithi099 ceph-mon[39420]: pgmap v584: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:39.341 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:29:39.370 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:39.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:39 smithi179 ceph-mon[43975]: osdmap e197: 12 total, 12 up, 12 in 2024-02-09T17:29:39.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:39 smithi179 ceph-mon[43975]: pgmap v584: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:39.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:39 smithi162 ceph-mon[44304]: osdmap e197: 12 total, 12 up, 12 in 2024-02-09T17:29:39.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:39 smithi162 ceph-mon[44304]: pgmap v584: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:39.922 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:29:39.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:39.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:29:40.026 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:29:40.059 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:40.080 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:29:40.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:40.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:29:40.183 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:29:40.213 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:40.233 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:29:40.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:40.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:29:40.337 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:29:40.369 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:40.377 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:29:40.389 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:29:40.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:40.393 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:29:40.415 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-09T17:29:40.493 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:29:40.522 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:40.541 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:29:40.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:40.544 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:29:40.645 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:29:40.673 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:40.692 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:29:40.694 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:40.695 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:29:40.796 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:29:40.828 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:40.849 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:29:40.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:40.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:29:40.954 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:29:40.982 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:41.000 INFO:teuthology.orchestra.run.smithi099.stderr:no valid command found; 10 closest matches: 2024-02-09T17:29:41.000 INFO:teuthology.orchestra.run.smithi099.stderr:0 2024-02-09T17:29:41.000 INFO:teuthology.orchestra.run.smithi099.stderr:1 2024-02-09T17:29:41.000 INFO:teuthology.orchestra.run.smithi099.stderr:2 2024-02-09T17:29:41.000 INFO:teuthology.orchestra.run.smithi099.stderr:abort 2024-02-09T17:29:41.000 INFO:teuthology.orchestra.run.smithi099.stderr:assert 2024-02-09T17:29:41.000 INFO:teuthology.orchestra.run.smithi099.stderr:bench [] [] [] [] 2024-02-09T17:29:41.000 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs debug_inject_read_zeros 2024-02-09T17:29:41.001 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs files list 2024-02-09T17:29:41.001 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs stats 2024-02-09T17:29:41.001 INFO:teuthology.orchestra.run.smithi099.stderr:bluestore allocator dump block 2024-02-09T17:29:41.001 INFO:teuthology.orchestra.run.smithi099.stderr:Error EINVAL: invalid command 2024-02-09T17:29:41.002 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:29:41.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:41.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:29:41.006 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:29:41.107 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:29:41.133 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:41.547 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:29:41.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:41 smithi179 ceph-mon[43975]: pgmap v585: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:41.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:41 smithi179 ceph-mon[43975]: osdmap e198: 12 total, 12 up, 12 in 2024-02-09T17:29:41.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:41 smithi162 ceph-mon[44304]: pgmap v585: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:41.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:41 smithi162 ceph-mon[44304]: osdmap e198: 12 total, 12 up, 12 in 2024-02-09T17:29:41.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:41 smithi099 ceph-mon[39420]: pgmap v585: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:41.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:41 smithi099 ceph-mon[39420]: osdmap e198: 12 total, 12 up, 12 in 2024-02-09T17:29:42.182 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:29:42.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:42.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:29:42.289 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:29:42.315 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:42.925 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-09T17:29:42.925 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-02-09T17:29:42.925 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph tell osd.3 'trim stale osdmaps' 2024-02-09T17:29:43.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:43 smithi179 ceph-mon[43975]: pgmap v587: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:43.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:43 smithi162 ceph-mon[44304]: pgmap v587: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:43.961 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:43 smithi099 ceph-mon[39420]: pgmap v587: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:43.963 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:29:43.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:43.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:29:44.078 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:29:44.110 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:44.131 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:29:44.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:44.134 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:29:44.234 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:29:44.262 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:44.361 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:29:44.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:44.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:29:44.464 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:29:44.491 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:44.824 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:29:45.919 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:45 smithi099 ceph-mon[39420]: pgmap v588: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:45.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:45 smithi179 ceph-mon[43975]: pgmap v588: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:45.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:45 smithi162 ceph-mon[44304]: pgmap v588: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:45.940 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:29:45.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:45.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:29:46.045 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:29:46.075 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:46.134 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:29:46.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:46.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:29:46.190 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:29:46.209 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:29:46.238 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:29:46.264 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:46.337 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:29:46.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:46.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:29:46.428 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:29:46.441 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:29:46.447 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:29:46.468 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:29:46.476 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:46.490 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:29:46.495 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:29:46.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:46.498 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:29:46.511 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:29:46.531 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:29:46.557 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:29:46.575 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is unset 2024-02-09T17:29:46.576 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'unique_pool_1' created 2024-02-09T17:29:46.579 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:29:46.599 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:29:46.602 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:29:46.621 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:29:46.623 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:29:46.623 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:29:46.623 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:29:46.623 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:29:46.623 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:29:46.623 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:29:46.623 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:29:46.623 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:29:46.623 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:29:46.623 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:29:46.623 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:29:46.623 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:29:46.627 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:46.646 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:29:46.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:46.649 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:29:46.750 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:29:46.777 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:46.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:46 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1033416791' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-02-09T17:29:46.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:46 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/165340913' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:29:46.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:46 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3455289561' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:29:46.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:46 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1033416791' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-02-09T17:29:46.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:46 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/165340913' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:29:46.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:46 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3455289561' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:29:46.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:46 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1033416791' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-02-09T17:29:46.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:46 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/165340913' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:29:46.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:46 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3455289561' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:29:47.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:47 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:29:47.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:47 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:29:47.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:47 smithi099 ceph-mon[39420]: pgmap v589: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:47.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:47 smithi099 ceph-mon[39420]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:29:47.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:47 smithi099 ceph-mon[39420]: Cluster is now healthy 2024-02-09T17:29:47.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:47 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1033416791' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-02-09T17:29:47.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:47 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3455289561' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:29:47.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:47 smithi099 ceph-mon[39420]: osdmap e199: 12 total, 12 up, 12 in 2024-02-09T17:29:47.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:47 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:29:47.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:47 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:29:47.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:47 smithi179 ceph-mon[43975]: pgmap v589: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:47.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:47 smithi179 ceph-mon[43975]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:29:47.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:47 smithi179 ceph-mon[43975]: Cluster is now healthy 2024-02-09T17:29:47.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:47 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1033416791' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-02-09T17:29:47.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:47 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3455289561' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:29:47.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:47 smithi179 ceph-mon[43975]: osdmap e199: 12 total, 12 up, 12 in 2024-02-09T17:29:47.926 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:29:47.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:47 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:29:47.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:47 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:29:47.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:47 smithi162 ceph-mon[44304]: pgmap v589: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:47.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:47 smithi162 ceph-mon[44304]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:29:47.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:47 smithi162 ceph-mon[44304]: Cluster is now healthy 2024-02-09T17:29:47.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:47 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1033416791' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-02-09T17:29:47.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:47 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3455289561' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:29:47.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:47 smithi162 ceph-mon[44304]: osdmap e199: 12 total, 12 up, 12 in 2024-02-09T17:29:47.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:47.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:29:47.935 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:29:48.030 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:29:48.058 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:48.399 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-02-09T17:29:48.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:48 smithi099 ceph-mon[39420]: osdmap e200: 12 total, 12 up, 12 in 2024-02-09T17:29:48.907 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:29:48.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:48.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:29:48.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:48 smithi179 ceph-mon[43975]: osdmap e200: 12 total, 12 up, 12 in 2024-02-09T17:29:48.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:48 smithi162 ceph-mon[44304]: osdmap e200: 12 total, 12 up, 12 in 2024-02-09T17:29:49.013 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:29:49.045 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:49.066 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:29:49.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:49.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:29:49.170 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:29:49.199 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:49.218 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:29:49.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:49.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:29:49.322 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:29:49.351 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:49.370 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:29:49.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:49.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:29:49.475 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:29:49.507 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:49.527 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:29:49.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:49.530 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:29:49.631 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:29:49.660 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:49 smithi162 ceph-mon[44304]: pgmap v592: 17 pgs: 3 creating+peering, 1 active+clean, 13 unknown; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:49.660 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:49 smithi162 ceph-mon[44304]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-09T17:29:49.660 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:49 smithi162 ceph-mon[44304]: osdmap e201: 12 total, 12 up, 12 in 2024-02-09T17:29:49.662 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:49.682 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:29:49.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:49.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:29:49.786 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:29:49.817 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:49.837 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:29:49.840 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:49 smithi099 ceph-mon[39420]: pgmap v592: 17 pgs: 3 creating+peering, 1 active+clean, 13 unknown; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:49.840 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:49 smithi099 ceph-mon[39420]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-09T17:29:49.840 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:49 smithi099 ceph-mon[39420]: osdmap e201: 12 total, 12 up, 12 in 2024-02-09T17:29:49.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:49.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:29:49.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:49 smithi179 ceph-mon[43975]: pgmap v592: 17 pgs: 3 creating+peering, 1 active+clean, 13 unknown; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:49.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:49 smithi179 ceph-mon[43975]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-09T17:29:49.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:49 smithi179 ceph-mon[43975]: osdmap e201: 12 total, 12 up, 12 in 2024-02-09T17:29:49.941 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set noscrub 2024-02-09T17:29:49.941 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:29:49.970 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:50.069 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:29:50.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:50.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:29:50.174 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:29:50.207 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:50.228 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:29:50.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:50.231 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:29:50.332 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:29:50.362 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:50.382 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:29:50.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:50.385 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:29:50.487 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:29:50.516 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:50.587 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:29:50.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:50.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:29:50.694 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:29:50.721 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:50.919 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:29:50.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:50.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:29:51.023 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:29:51.055 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:51.077 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:29:51.079 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:51.080 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:29:51.181 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:29:51.209 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:51.261 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:29:51.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:51.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:29:51.365 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:29:51.391 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:51.539 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:29:51.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:51.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:29:51.644 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:29:51.652 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:29:51.676 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:51.697 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:29:51.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:51.701 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:29:51.802 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:29:51.827 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:51 smithi099 ceph-mon[39420]: pgmap v594: 17 pgs: 3 creating+peering, 11 active+clean, 3 unknown; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:51.828 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:51.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:51 smithi179 ceph-mon[43975]: pgmap v594: 17 pgs: 3 creating+peering, 11 active+clean, 3 unknown; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:51.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:51 smithi162 ceph-mon[44304]: pgmap v594: 17 pgs: 3 creating+peering, 11 active+clean, 3 unknown; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:51.935 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:29:51.935 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:29:52.111 INFO:teuthology.orchestra.run.smithi099.stderr:no valid command found; 10 closest matches: 2024-02-09T17:29:52.112 INFO:teuthology.orchestra.run.smithi099.stderr:0 2024-02-09T17:29:52.112 INFO:teuthology.orchestra.run.smithi099.stderr:1 2024-02-09T17:29:52.112 INFO:teuthology.orchestra.run.smithi099.stderr:2 2024-02-09T17:29:52.112 INFO:teuthology.orchestra.run.smithi099.stderr:abort 2024-02-09T17:29:52.112 INFO:teuthology.orchestra.run.smithi099.stderr:assert 2024-02-09T17:29:52.112 INFO:teuthology.orchestra.run.smithi099.stderr:bench [] [] [] [] 2024-02-09T17:29:52.112 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs debug_inject_read_zeros 2024-02-09T17:29:52.112 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs files list 2024-02-09T17:29:52.112 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs stats 2024-02-09T17:29:52.112 INFO:teuthology.orchestra.run.smithi099.stderr:bluestore allocator dump block 2024-02-09T17:29:52.113 INFO:teuthology.orchestra.run.smithi099.stderr:Error EINVAL: invalid command 2024-02-09T17:29:52.380 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:29:52.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:52.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:29:52.484 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:29:52.516 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:52.537 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:29:52.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:52.540 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:29:52.641 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:29:52.673 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:52.694 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:29:52.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:52.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:29:52.798 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:29:52.830 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:52.851 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:29:52.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:52.855 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:29:52.921 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-09T17:29:52.921 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-02-09T17:29:52.922 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph tell osd.4 'trim stale osdmaps' 2024-02-09T17:29:52.955 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:29:52.992 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:53.164 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:29:53.584 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:29:53.618 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:29:53.625 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-09T17:29:53.635 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:29:53.866 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:53 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:29:53.867 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:53 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:29:53.867 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:53 smithi099 ceph-mon[39420]: pgmap v595: 17 pgs: 17 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:53.867 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:53 smithi099 ceph-mon[39420]: osdmap e202: 12 total, 12 up, 12 in 2024-02-09T17:29:53.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:53 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:29:53.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:53 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:29:53.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:53 smithi179 ceph-mon[43975]: pgmap v595: 17 pgs: 17 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:53.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:53 smithi179 ceph-mon[43975]: osdmap e202: 12 total, 12 up, 12 in 2024-02-09T17:29:53.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:53 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:29:53.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:53 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:29:53.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:53 smithi162 ceph-mon[44304]: pgmap v595: 17 pgs: 17 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:53.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:53 smithi162 ceph-mon[44304]: osdmap e202: 12 total, 12 up, 12 in 2024-02-09T17:29:54.040 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:29:54.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:54.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:29:54.145 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:29:54.173 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:55.022 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:29:55.025 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:55.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:29:55.126 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:29:55.158 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:55.179 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:29:55.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:55.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:29:55.283 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:29:55.314 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:55.334 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:29:55.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:55.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:29:55.439 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:29:55.469 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:55.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:55 smithi099 ceph-mon[39420]: pgmap v597: 17 pgs: 17 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:55.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:55 smithi179 ceph-mon[43975]: pgmap v597: 17 pgs: 17 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:55.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:55 smithi162 ceph-mon[44304]: pgmap v597: 17 pgs: 17 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:56.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:56 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:29:56.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:56 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:29:56.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:56 smithi179 ceph-mon[43975]: osdmap e203: 12 total, 12 up, 12 in 2024-02-09T17:29:56.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:56 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:29:56.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:56 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:29:56.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:56 smithi162 ceph-mon[44304]: osdmap e203: 12 total, 12 up, 12 in 2024-02-09T17:29:56.970 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:56 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:29:56.971 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:56 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:29:56.971 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:56 smithi099 ceph-mon[39420]: osdmap e203: 12 total, 12 up, 12 in 2024-02-09T17:29:57.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:57 smithi099 ceph-mon[39420]: pgmap v599: 17 pgs: 17 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:57.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:57 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:29:57.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:57 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:29:57.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:57 smithi179 ceph-mon[43975]: pgmap v599: 17 pgs: 17 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:57.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:57 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:29:57.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:57 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:29:57.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:57 smithi162 ceph-mon[44304]: pgmap v599: 17 pgs: 17 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:57.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:57 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:29:57.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:57 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:29:58.025 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:29:58.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:58.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:29:58.131 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:29:58.163 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:58.184 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:29:58.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:58.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:29:58.288 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:29:58.320 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:58.341 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:29:58.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:58.344 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:29:58.445 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:29:58.474 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:58.548 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:29:58.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:58.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:29:58.651 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:29:58.679 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:58.701 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:29:58.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:58.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:29:58.808 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:29:58.840 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:58.861 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:29:58.863 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:58 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:29:58.863 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:58 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:29:58.863 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:58 smithi162 ceph-mon[44304]: osdmap e204: 12 total, 12 up, 12 in 2024-02-09T17:29:58.863 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:58 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:29:58.863 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:58 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:29:58.863 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:58 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:29:58.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:58.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:29:58.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:58 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:29:58.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:58 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:29:58.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:58 smithi179 ceph-mon[43975]: osdmap e204: 12 total, 12 up, 12 in 2024-02-09T17:29:58.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:58 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:29:58.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:58 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:29:58.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:58 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:29:58.965 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:29:58.967 INFO:teuthology.orchestra.run.smithi099.stderr:no valid command found; 10 closest matches: 2024-02-09T17:29:58.967 INFO:teuthology.orchestra.run.smithi099.stderr:0 2024-02-09T17:29:58.967 INFO:teuthology.orchestra.run.smithi099.stderr:1 2024-02-09T17:29:58.967 INFO:teuthology.orchestra.run.smithi099.stderr:2 2024-02-09T17:29:58.967 INFO:teuthology.orchestra.run.smithi099.stderr:abort 2024-02-09T17:29:58.967 INFO:teuthology.orchestra.run.smithi099.stderr:assert 2024-02-09T17:29:58.968 INFO:teuthology.orchestra.run.smithi099.stderr:bench [] [] [] [] 2024-02-09T17:29:58.968 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs debug_inject_read_zeros 2024-02-09T17:29:58.968 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs files list 2024-02-09T17:29:58.968 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs stats 2024-02-09T17:29:58.968 INFO:teuthology.orchestra.run.smithi099.stderr:bluestore allocator dump block 2024-02-09T17:29:58.968 INFO:teuthology.orchestra.run.smithi099.stderr:Error EINVAL: invalid command 2024-02-09T17:29:58.984 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:58 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:29:58.984 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:58 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:29:58.984 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:58 smithi099 ceph-mon[39420]: osdmap e204: 12 total, 12 up, 12 in 2024-02-09T17:29:58.984 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:58 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:29:58.984 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:58 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:29:58.985 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:58 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:29:58.995 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:59.014 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:29:59.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:59.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:29:59.118 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:29:59.150 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:59.172 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:29:59.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:59.175 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:29:59.276 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:29:59.306 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:59.326 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:29:59.328 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:29:59.329 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:29:59.360 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:29:59.361 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:29:59.361 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:29:59.361 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:29:59.361 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:29:59.361 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:29:59.361 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:29:59.361 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:29:59.361 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:29:59.364 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:29:59.384 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:29:59.404 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:29:59.406 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:29:59.406 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:29:59.406 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:29:59.406 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:29:59.406 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:29:59.406 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:29:59.406 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:29:59.406 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:29:59.406 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:29:59.406 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:29:59.406 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:29:59.407 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:29:59.429 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:29:59.458 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:29:59.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:59 smithi099 ceph-mon[39420]: pgmap v601: 17 pgs: 17 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:59.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:59 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:29:59.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:59 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:29:59.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:29:59 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2665872078' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:29:59.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:59 smithi179 ceph-mon[43975]: pgmap v601: 17 pgs: 17 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:59.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:59 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:29:59.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:59 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:29:59.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:29:59 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2665872078' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:29:59.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:59 smithi162 ceph-mon[44304]: pgmap v601: 17 pgs: 17 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:29:59.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:59 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:29:59.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:59 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:29:59.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:29:59 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2665872078' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:30:00.053 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:30:00.096 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-09T17:30:00.097 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-02-09T17:30:00.097 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph tell osd.5 'trim stale osdmaps' 2024-02-09T17:30:00.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:00 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:30:00.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:00 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:30:00.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:00 smithi179 ceph-mon[43975]: osdmap e205: 12 total, 12 up, 12 in 2024-02-09T17:30:00.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:00 smithi179 ceph-mon[43975]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-02-09T17:30:00.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:00 smithi179 ceph-mon[43975]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-02-09T17:30:00.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:00 smithi179 ceph-mon[43975]: application not enabled on pool 'unique_pool_1' 2024-02-09T17:30:00.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:00 smithi179 ceph-mon[43975]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-02-09T17:30:00.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:00 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:30:00.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:00 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:30:00.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:00 smithi162 ceph-mon[44304]: osdmap e205: 12 total, 12 up, 12 in 2024-02-09T17:30:00.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:00 smithi162 ceph-mon[44304]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-02-09T17:30:00.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:00 smithi162 ceph-mon[44304]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-02-09T17:30:00.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:00 smithi162 ceph-mon[44304]: application not enabled on pool 'unique_pool_1' 2024-02-09T17:30:00.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:00 smithi162 ceph-mon[44304]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-02-09T17:30:00.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:00 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:30:00.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:00 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:30:00.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:00 smithi099 ceph-mon[39420]: osdmap e205: 12 total, 12 up, 12 in 2024-02-09T17:30:00.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:00 smithi099 ceph-mon[39420]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-02-09T17:30:00.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:00 smithi099 ceph-mon[39420]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-02-09T17:30:00.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:00 smithi099 ceph-mon[39420]: application not enabled on pool 'unique_pool_1' 2024-02-09T17:30:00.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:00 smithi099 ceph-mon[39420]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-02-09T17:30:01.237 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:30:01.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:01.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:30:01.343 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:30:01.372 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:01.470 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:30:01.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:01.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:30:01.574 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:30:01.608 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:01.629 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:30:01.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:01.632 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:30:01.634 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is set 2024-02-09T17:30:01.733 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:30:01.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:01 smithi099 ceph-mon[39420]: pgmap v603: 17 pgs: 17 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:30:01.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:01 smithi099 ceph-mon[39420]: osdmap e206: 12 total, 12 up, 12 in 2024-02-09T17:30:01.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:01 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3467109668' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:30:01.761 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:01.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:01 smithi179 ceph-mon[43975]: pgmap v603: 17 pgs: 17 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:30:01.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:01 smithi179 ceph-mon[43975]: osdmap e206: 12 total, 12 up, 12 in 2024-02-09T17:30:01.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:01 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3467109668' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:30:01.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:01 smithi162 ceph-mon[44304]: pgmap v603: 17 pgs: 17 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:30:01.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:01 smithi162 ceph-mon[44304]: osdmap e206: 12 total, 12 up, 12 in 2024-02-09T17:30:01.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:01 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3467109668' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:30:02.610 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:30:02.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:02.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:30:02.644 INFO:teuthology.orchestra.run.smithi099.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-02-09T17:30:02.743 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:30:02.777 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:02.798 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:30:02.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:02.801 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:30:02.902 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:30:02.908 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:02 smithi099 ceph-mon[39420]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:30:02.908 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:02 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3467109668' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:30:02.908 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:02 smithi099 ceph-mon[39420]: osdmap e207: 12 total, 12 up, 12 in 2024-02-09T17:30:02.908 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:02 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:30:02.908 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:02 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:30:02.908 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:02 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2720154792' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-09T17:30:02.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:02 smithi179 ceph-mon[43975]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:30:02.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:02 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3467109668' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:30:02.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:02 smithi179 ceph-mon[43975]: osdmap e207: 12 total, 12 up, 12 in 2024-02-09T17:30:02.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:02 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:30:02.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:02 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:30:02.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:02 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2720154792' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-09T17:30:02.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:02 smithi162 ceph-mon[44304]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:30:02.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:02 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3467109668' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:30:02.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:02 smithi162 ceph-mon[44304]: osdmap e207: 12 total, 12 up, 12 in 2024-02-09T17:30:02.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:02 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:30:02.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:02 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:30:02.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:02 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2720154792' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-09T17:30:02.931 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:03.680 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:30:03.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:03.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:30:03.785 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:30:03.819 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:03.844 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:30:03.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:03.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:30:03.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:03 smithi179 ceph-mon[43975]: pgmap v606: 17 pgs: 17 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:30:03.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:03 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2720154792' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-09T17:30:03.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:03 smithi179 ceph-mon[43975]: osdmap e208: 12 total, 12 up, 12 in 2024-02-09T17:30:03.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:03 smithi162 ceph-mon[44304]: pgmap v606: 17 pgs: 17 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:30:03.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:03 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2720154792' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-09T17:30:03.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:03 smithi162 ceph-mon[44304]: osdmap e208: 12 total, 12 up, 12 in 2024-02-09T17:30:03.930 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:03 smithi099 ceph-mon[39420]: pgmap v606: 17 pgs: 17 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:30:03.930 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:03 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2720154792' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-09T17:30:03.930 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:03 smithi099 ceph-mon[39420]: osdmap e208: 12 total, 12 up, 12 in 2024-02-09T17:30:03.948 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:30:03.979 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:03.998 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:30:04.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:04.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:30:04.102 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:30:04.131 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:04.151 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:30:04.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:04.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:30:04.213 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:30:04.213 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [], 2024-02-09T17:30:04.213 INFO:teuthology.orchestra.run.smithi099.stdout: "complaint_time": 30, 2024-02-09T17:30:04.213 INFO:teuthology.orchestra.run.smithi099.stdout: "num_blocked_ops": 0 2024-02-09T17:30:04.213 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:30:04.255 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:30:04.284 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:04.733 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:30:04.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:04.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:30:04.837 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:30:04.869 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:04.890 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:30:04.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:04.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:30:04.938 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-09T17:30:04.965 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:04 smithi099 ceph-mon[39420]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-09T17:30:04.993 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:30:05.023 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:04 smithi179 ceph-mon[43975]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-09T17:30:05.025 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:05.047 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:30:05.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:05.050 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:30:05.151 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:30:05.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:04 smithi162 ceph-mon[44304]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-09T17:30:05.183 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:05.204 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:30:05.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:05.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:30:05.309 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:30:05.337 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:05.344 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:30:05.357 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:30:05.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:05.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:30:05.376 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-09T17:30:05.462 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:30:05.495 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:05.515 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:30:05.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:05.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:30:05.619 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:30:05.649 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:05.651 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:30:05.670 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:30:05.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:05.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:30:05.774 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:30:05.800 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:05.881 DEBUG:teuthology.orchestra.run.smithi179:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-02-09T17:30:05.883 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-02-09T17:30:05.884 INFO:teuthology.orchestra.run:waiting for 3000 2024-02-09T17:30:05.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:05 smithi162 ceph-mon[44304]: pgmap v608: 17 pgs: 17 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:30:05.937 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:hints = 1 2024-02-09T17:30:05.938 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-02-09T17:30:05.938 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Object prefix: benchmark_data_smithi179_90440 2024-02-09T17:30:05.938 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-09T17:30:05.938 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 0 0 0 0 0 0 - 0 2024-02-09T17:30:05.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:05 smithi099 ceph-mon[39420]: pgmap v608: 17 pgs: 17 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:30:06.151 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:30:06.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:06.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:30:06.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:05 smithi179 ceph-mon[43975]: pgmap v608: 17 pgs: 17 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:30:06.200 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:30:06.206 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:30:06.262 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:30:06.304 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:06.333 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:30:06.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:06.338 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:30:06.439 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:30:06.481 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:06.796 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:30:06.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:06.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:30:06.847 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:30:06.901 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:30:06.938 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 1 16 4541 4525 282.797 282.812 0.00330275 0.00352741 2024-02-09T17:30:06.949 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:06.969 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:30:06.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:06.973 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:30:07.074 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:30:07.127 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:07.356 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:07 smithi099 ceph-mon[39420]: pgmap v609: 17 pgs: 17 active+clean; 24 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s wr, 61 op/s 2024-02-09T17:30:07.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:07 smithi179 ceph-mon[43975]: pgmap v609: 17 pgs: 17 active+clean; 24 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s wr, 61 op/s 2024-02-09T17:30:07.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:07 smithi162 ceph-mon[44304]: pgmap v609: 17 pgs: 17 active+clean; 24 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s wr, 61 op/s 2024-02-09T17:30:07.938 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 2 16 8769 8753 273.508 264.25 0.0022398 0.00364956 2024-02-09T17:30:08.938 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 3 16 11868 11852 246.895 193.688 0.00361319 0.0040401 2024-02-09T17:30:09.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:09 smithi179 ceph-mon[43975]: pgmap v610: 17 pgs: 17 active+clean; 62 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s wr, 124 op/s 2024-02-09T17:30:09.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:09 smithi162 ceph-mon[44304]: pgmap v610: 17 pgs: 17 active+clean; 62 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s wr, 124 op/s 2024-02-09T17:30:09.938 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 4 16 14949 14933 233.308 192.562 0.00201659 0.00427648 2024-02-09T17:30:09.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:09 smithi099 ceph-mon[39420]: pgmap v610: 17 pgs: 17 active+clean; 62 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s wr, 124 op/s 2024-02-09T17:30:10.938 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 5 15 17873 17858 223.206 182.812 0.00565833 0.00447494 2024-02-09T17:30:11.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:11 smithi179 ceph-mon[43975]: pgmap v611: 17 pgs: 17 active+clean; 561 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s 2024-02-09T17:30:11.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:11 smithi162 ceph-mon[44304]: pgmap v611: 17 pgs: 17 active+clean; 561 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s 2024-02-09T17:30:11.938 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 6 16 20176 20160 209.982 143.875 0.00133342 0.00473991 2024-02-09T17:30:11.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:11 smithi099 ceph-mon[39420]: pgmap v611: 17 pgs: 17 active+clean; 561 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s 2024-02-09T17:30:12.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:12 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:30:12.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:12 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:30:12.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:12 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:30:12.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:12 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:30:12.938 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 7 16 21973 21957 196.028 112.312 0.00382978 0.00509514 2024-02-09T17:30:12.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:12 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:30:12.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:12 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:30:13.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:13 smithi179 ceph-mon[43975]: pgmap v612: 17 pgs: 17 active+clean; 948 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.52k op/s 2024-02-09T17:30:13.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:13 smithi179 ceph-mon[43975]: osdmap e209: 12 total, 12 up, 12 in 2024-02-09T17:30:13.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:13 smithi162 ceph-mon[44304]: pgmap v612: 17 pgs: 17 active+clean; 948 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.52k op/s 2024-02-09T17:30:13.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:13 smithi162 ceph-mon[44304]: osdmap e209: 12 total, 12 up, 12 in 2024-02-09T17:30:13.939 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 8 16 24858 24842 194.062 180.312 0.00522643 0.0051485 2024-02-09T17:30:13.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:13 smithi099 ceph-mon[39420]: pgmap v612: 17 pgs: 17 active+clean; 948 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.52k op/s 2024-02-09T17:30:13.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:13 smithi099 ceph-mon[39420]: osdmap e209: 12 total, 12 up, 12 in 2024-02-09T17:30:13.994 INFO:teuthology.orchestra.run.smithi099.stderr:no valid command found; 10 closest matches: 2024-02-09T17:30:13.994 INFO:teuthology.orchestra.run.smithi099.stderr:0 2024-02-09T17:30:13.994 INFO:teuthology.orchestra.run.smithi099.stderr:1 2024-02-09T17:30:13.995 INFO:teuthology.orchestra.run.smithi099.stderr:2 2024-02-09T17:30:13.995 INFO:teuthology.orchestra.run.smithi099.stderr:abort 2024-02-09T17:30:13.995 INFO:teuthology.orchestra.run.smithi099.stderr:assert 2024-02-09T17:30:13.995 INFO:teuthology.orchestra.run.smithi099.stderr:bench [] [] [] [] 2024-02-09T17:30:13.995 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs debug_inject_read_zeros 2024-02-09T17:30:13.995 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs files list 2024-02-09T17:30:13.995 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs stats 2024-02-09T17:30:13.995 INFO:teuthology.orchestra.run.smithi099.stderr:bluestore allocator dump block 2024-02-09T17:30:13.995 INFO:teuthology.orchestra.run.smithi099.stderr:Error EINVAL: invalid command 2024-02-09T17:30:14.939 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 9 16 27499 27483 190.838 165.062 0.00602947 0.005235 2024-02-09T17:30:15.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:15 smithi179 ceph-mon[43975]: pgmap v614: 17 pgs: 17 active+clean; 1.3 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 136 MiB/s wr, 2.18k op/s 2024-02-09T17:30:15.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:15 smithi162 ceph-mon[44304]: pgmap v614: 17 pgs: 17 active+clean; 1.3 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 136 MiB/s wr, 2.18k op/s 2024-02-09T17:30:15.939 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 10 16 30052 30036 187.71 159.562 0.0039506 0.00532316 2024-02-09T17:30:15.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:15 smithi099 ceph-mon[39420]: pgmap v614: 17 pgs: 17 active+clean; 1.3 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 136 MiB/s wr, 2.18k op/s 2024-02-09T17:30:16.939 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 11 16 32524 32508 184.689 154.5 0.00221504 0.00537102 2024-02-09T17:30:17.353 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-09T17:30:17.354 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-02-09T17:30:17.354 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph tell osd.6 'trim stale osdmaps' 2024-02-09T17:30:17.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:17 smithi099 ceph-mon[39420]: pgmap v615: 17 pgs: 17 active+clean; 1.6 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 158 MiB/s wr, 2.53k op/s 2024-02-09T17:30:17.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:17 smithi179 ceph-mon[43975]: pgmap v615: 17 pgs: 17 active+clean; 1.6 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 158 MiB/s wr, 2.53k op/s 2024-02-09T17:30:17.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:17 smithi162 ceph-mon[44304]: pgmap v615: 17 pgs: 17 active+clean; 1.6 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 158 MiB/s wr, 2.53k op/s 2024-02-09T17:30:17.939 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 12 16 34876 34860 181.548 147 0.00608283 0.00550375 2024-02-09T17:30:18.149 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:30:18.149 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 20, 2024-02-09T17:30:18.149 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 600, 2024-02-09T17:30:18.149 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [ 2024-02-09T17:30:18.149 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.150 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:184931 2.e 2:79df8df1:::benchmark_data_smithi179_77138_object51523:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e157)", 2024-02-09T17:30:18.150 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:27:17.092846+0000", 2024-02-09T17:30:18.150 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 181.05221978, 2024-02-09T17:30:18.150 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0303135729999999, 2024-02-09T17:30:18.150 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:30:18.150 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "reached pg", 2024-02-09T17:30:18.150 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:30:18.150 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:30:18.150 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:30:18.150 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 184931 2024-02-09T17:30:18.150 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.150 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:30:18.150 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.151 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:30:18.151 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.092846+0000", 2024-02-09T17:30:18.151 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:30:18.151 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.151 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.151 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:30:18.151 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.092846+0000", 2024-02-09T17:30:18.151 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:30:18.151 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.151 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.151 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:30:18.151 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.092847+0000", 2024-02-09T17:30:18.151 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.339e-06 2024-02-09T17:30:18.151 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.152 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.152 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:30:18.152 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.092852+0000", 2024-02-09T17:30:18.152 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3519999999999997e-06 2024-02-09T17:30:18.152 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.152 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.152 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:30:18.152 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.092861+0000", 2024-02-09T17:30:18.152 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.9789999999999999e-06 2024-02-09T17:30:18.152 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.152 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.152 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:30:18.152 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.092869+0000", 2024-02-09T17:30:18.153 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.6869999999999994e-06 2024-02-09T17:30:18.153 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.153 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.153 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:30:18.153 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.092925+0000", 2024-02-09T17:30:18.153 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6218000000000003e-05 2024-02-09T17:30:18.153 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.153 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.153 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:30:18.153 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.092936+0000", 2024-02-09T17:30:18.153 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1080000000000001e-05 2024-02-09T17:30:18.153 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.153 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.154 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:30:18.154 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.120181+0000", 2024-02-09T17:30:18.154 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0272447149999999 2024-02-09T17:30:18.154 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.154 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.154 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:30:18.154 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.123159+0000", 2024-02-09T17:30:18.154 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.002978203 2024-02-09T17:30:18.154 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:30:18.154 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:30:18.154 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:30:18.154 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.154 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.154 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:184932 2.1 2:8ba2ef2a:::benchmark_data_smithi179_77138_object51524:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e157)", 2024-02-09T17:30:18.155 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:27:17.092888+0000", 2024-02-09T17:30:18.155 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 181.052177871, 2024-02-09T17:30:18.155 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0186303029999999, 2024-02-09T17:30:18.155 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:30:18.155 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "reached pg", 2024-02-09T17:30:18.155 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:30:18.155 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:30:18.155 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:30:18.155 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 184932 2024-02-09T17:30:18.155 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.155 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:30:18.155 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.156 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:30:18.156 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.092888+0000", 2024-02-09T17:30:18.156 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:30:18.156 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.156 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.156 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:30:18.156 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.092888+0000", 2024-02-09T17:30:18.156 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:30:18.156 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.156 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.156 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:30:18.157 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.092888+0000", 2024-02-09T17:30:18.157 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2899999999999999e-07 2024-02-09T17:30:18.157 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.157 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.157 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:30:18.157 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.092890+0000", 2024-02-09T17:30:18.157 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.3020000000000002e-06 2024-02-09T17:30:18.157 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.157 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.157 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:30:18.157 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.092897+0000", 2024-02-09T17:30:18.157 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.4999999999999996e-06 2024-02-09T17:30:18.158 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.158 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.158 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:30:18.158 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.092903+0000", 2024-02-09T17:30:18.158 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.1179999999999997e-06 2024-02-09T17:30:18.158 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.158 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.158 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:30:18.158 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.095987+0000", 2024-02-09T17:30:18.158 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0030837009999999999 2024-02-09T17:30:18.158 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.158 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.158 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:30:18.159 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.095995+0000", 2024-02-09T17:30:18.159 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.1370000000000002e-06 2024-02-09T17:30:18.159 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.159 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.159 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:30:18.159 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.111427+0000", 2024-02-09T17:30:18.159 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0154317429999999 2024-02-09T17:30:18.159 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.159 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.159 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:30:18.159 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.111518+0000", 2024-02-09T17:30:18.159 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.1372999999999994e-05 2024-02-09T17:30:18.159 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:30:18.160 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:30:18.160 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:30:18.160 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.160 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.160 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:184936 2.1 2:8468682c:::benchmark_data_smithi179_77138_object51528:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e157)", 2024-02-09T17:30:18.160 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:27:17.092912+0000", 2024-02-09T17:30:18.160 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 181.05215338100001, 2024-02-09T17:30:18.160 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.018846699, 2024-02-09T17:30:18.160 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:30:18.160 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "reached pg", 2024-02-09T17:30:18.160 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:30:18.160 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:30:18.160 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:30:18.160 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 184936 2024-02-09T17:30:18.161 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.161 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:30:18.161 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.161 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:30:18.161 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.092912+0000", 2024-02-09T17:30:18.161 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:30:18.161 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.161 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.161 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:30:18.161 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.092912+0000", 2024-02-09T17:30:18.161 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:30:18.161 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.161 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.161 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:30:18.162 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.092912+0000", 2024-02-09T17:30:18.162 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.1199999999999998e-07 2024-02-09T17:30:18.162 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.162 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.162 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:30:18.162 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.092914+0000", 2024-02-09T17:30:18.162 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.995e-06 2024-02-09T17:30:18.162 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.162 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.162 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:30:18.162 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.092920+0000", 2024-02-09T17:30:18.162 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7830000000000004e-06 2024-02-09T17:30:18.162 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.162 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.163 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:30:18.163 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.092926+0000", 2024-02-09T17:30:18.163 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2560000000000002e-06 2024-02-09T17:30:18.163 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.163 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.163 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:30:18.163 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.096017+0000", 2024-02-09T17:30:18.163 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0030913889999999999 2024-02-09T17:30:18.163 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.163 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.163 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:30:18.163 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.096024+0000", 2024-02-09T17:30:18.163 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.703e-06 2024-02-09T17:30:18.163 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.164 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.164 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:30:18.164 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.111712+0000", 2024-02-09T17:30:18.164 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.015688776 2024-02-09T17:30:18.164 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.164 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.164 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:30:18.164 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.111759+0000", 2024-02-09T17:30:18.164 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.6384999999999997e-05 2024-02-09T17:30:18.164 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:30:18.164 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:30:18.165 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:30:18.165 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.165 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.165 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:184940 2.e 2:7f15bfe2:::benchmark_data_smithi179_77138_object51532:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e157)", 2024-02-09T17:30:18.165 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:27:17.092931+0000", 2024-02-09T17:30:18.165 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 181.05213398999999, 2024-02-09T17:30:18.165 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0307621140000001, 2024-02-09T17:30:18.165 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:30:18.165 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "reached pg", 2024-02-09T17:30:18.165 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:30:18.165 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:30:18.165 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:30:18.165 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 184940 2024-02-09T17:30:18.166 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.166 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:30:18.166 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.166 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:30:18.166 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.092931+0000", 2024-02-09T17:30:18.166 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:30:18.166 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.166 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.166 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:30:18.166 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.092931+0000", 2024-02-09T17:30:18.166 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:30:18.166 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.166 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.166 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:30:18.167 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.092932+0000", 2024-02-09T17:30:18.167 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.89e-07 2024-02-09T17:30:18.167 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.167 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.167 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:30:18.167 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.092934+0000", 2024-02-09T17:30:18.167 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.1789999999999998e-06 2024-02-09T17:30:18.167 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.167 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.167 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:30:18.167 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.092940+0000", 2024-02-09T17:30:18.167 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.5779999999999998e-06 2024-02-09T17:30:18.167 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.167 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.168 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:30:18.168 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.092946+0000", 2024-02-09T17:30:18.168 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.0889999999999996e-06 2024-02-09T17:30:18.168 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.168 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.168 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:30:18.168 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.092985+0000", 2024-02-09T17:30:18.168 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9660999999999999e-05 2024-02-09T17:30:18.168 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.168 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.168 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:30:18.168 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.092995+0000", 2024-02-09T17:30:18.168 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.3829999999999998e-06 2024-02-09T17:30:18.168 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.169 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.169 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:30:18.169 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.123627+0000", 2024-02-09T17:30:18.169 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.030631861 2024-02-09T17:30:18.169 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.169 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.169 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:30:18.169 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.123694+0000", 2024-02-09T17:30:18.169 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.6973999999999999e-05 2024-02-09T17:30:18.169 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:30:18.169 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:30:18.169 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:30:18.169 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.169 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.169 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:184944 2.1 2:8b653cfb:::benchmark_data_smithi179_77138_object51536:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e157)", 2024-02-09T17:30:18.170 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:27:17.092956+0000", 2024-02-09T17:30:18.170 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 181.052109685, 2024-02-09T17:30:18.170 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.018872239, 2024-02-09T17:30:18.170 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:30:18.170 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "reached pg", 2024-02-09T17:30:18.170 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:30:18.170 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:30:18.170 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:30:18.170 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 184944 2024-02-09T17:30:18.170 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.170 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:30:18.170 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.170 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:30:18.170 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.092956+0000", 2024-02-09T17:30:18.170 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:30:18.171 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.171 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.171 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:30:18.171 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.092956+0000", 2024-02-09T17:30:18.171 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:30:18.171 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.171 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.171 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:30:18.171 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.092956+0000", 2024-02-09T17:30:18.171 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2e-07 2024-02-09T17:30:18.171 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.171 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.171 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:30:18.171 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.092958+0000", 2024-02-09T17:30:18.171 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.08e-06 2024-02-09T17:30:18.172 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.172 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.172 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:30:18.172 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.092964+0000", 2024-02-09T17:30:18.172 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.139e-06 2024-02-09T17:30:18.172 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.172 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.172 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:30:18.172 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.092970+0000", 2024-02-09T17:30:18.172 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3720000000000004e-06 2024-02-09T17:30:18.172 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.172 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.172 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:30:18.172 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.096045+0000", 2024-02-09T17:30:18.172 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0030755880000000002 2024-02-09T17:30:18.173 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.173 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.173 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:30:18.173 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.096052+0000", 2024-02-09T17:30:18.173 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.4189999999999999e-06 2024-02-09T17:30:18.173 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.173 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.173 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:30:18.173 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.111791+0000", 2024-02-09T17:30:18.173 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0157392620000001 2024-02-09T17:30:18.173 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.173 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.173 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:30:18.173 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.111828+0000", 2024-02-09T17:30:18.174 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.6959000000000003e-05 2024-02-09T17:30:18.174 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:30:18.174 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:30:18.174 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:30:18.174 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.174 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.174 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:184945 2.1 2:87bdbebb:::benchmark_data_smithi179_77138_object51537:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e157)", 2024-02-09T17:30:18.174 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:27:17.092976+0000", 2024-02-09T17:30:18.174 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 181.052089752, 2024-02-09T17:30:18.174 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0189067279999999, 2024-02-09T17:30:18.174 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:30:18.174 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "reached pg", 2024-02-09T17:30:18.174 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:30:18.174 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:30:18.174 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:30:18.175 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 184945 2024-02-09T17:30:18.175 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.175 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:30:18.175 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.175 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:30:18.175 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.092976+0000", 2024-02-09T17:30:18.175 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:30:18.175 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.175 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.175 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:30:18.175 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.092976+0000", 2024-02-09T17:30:18.175 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:30:18.175 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.175 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.176 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:30:18.176 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.092976+0000", 2024-02-09T17:30:18.176 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.6399999999999998e-07 2024-02-09T17:30:18.176 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.176 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.176 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:30:18.176 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.092979+0000", 2024-02-09T17:30:18.176 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.4820000000000001e-06 2024-02-09T17:30:18.176 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.176 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.176 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:30:18.176 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.092985+0000", 2024-02-09T17:30:18.176 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.1330000000000004e-06 2024-02-09T17:30:18.176 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.177 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.177 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:30:18.177 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.092990+0000", 2024-02-09T17:30:18.177 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2349999999999999e-06 2024-02-09T17:30:18.177 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.177 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.177 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:30:18.177 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.096074+0000", 2024-02-09T17:30:18.177 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.003083658 2024-02-09T17:30:18.177 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.177 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.177 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:30:18.177 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.096080+0000", 2024-02-09T17:30:18.177 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.6930000000000001e-06 2024-02-09T17:30:18.178 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.178 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.178 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:30:18.178 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.111849+0000", 2024-02-09T17:30:18.178 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.015768615 2024-02-09T17:30:18.178 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.178 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.178 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:30:18.178 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.111882+0000", 2024-02-09T17:30:18.178 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.3547999999999998e-05 2024-02-09T17:30:18.178 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:30:18.178 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:30:18.178 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:30:18.178 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.179 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.179 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:184947 2.1 2:8a065b8a:::benchmark_data_smithi179_77138_object51539:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e157)", 2024-02-09T17:30:18.179 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:27:17.092997+0000", 2024-02-09T17:30:18.179 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 181.052068827, 2024-02-09T17:30:18.179 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0190275390000001, 2024-02-09T17:30:18.179 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:30:18.179 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "reached pg", 2024-02-09T17:30:18.179 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:30:18.179 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:30:18.179 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:30:18.179 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 184947 2024-02-09T17:30:18.179 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.179 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:30:18.179 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.180 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:30:18.180 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.092997+0000", 2024-02-09T17:30:18.180 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:30:18.180 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.180 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.180 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:30:18.180 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.092997+0000", 2024-02-09T17:30:18.180 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:30:18.180 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.180 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.180 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:30:18.180 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.092997+0000", 2024-02-09T17:30:18.180 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.6199999999999999e-07 2024-02-09T17:30:18.181 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.181 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.181 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:30:18.181 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.092999+0000", 2024-02-09T17:30:18.181 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.9290000000000001e-06 2024-02-09T17:30:18.181 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.181 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.181 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:30:18.181 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.093005+0000", 2024-02-09T17:30:18.181 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.8189999999999997e-06 2024-02-09T17:30:18.181 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.181 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.181 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:30:18.181 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.093010+0000", 2024-02-09T17:30:18.182 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2059999999999998e-06 2024-02-09T17:30:18.182 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.182 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.182 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:30:18.182 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.096101+0000", 2024-02-09T17:30:18.182 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0030907030000000002 2024-02-09T17:30:18.182 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.182 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.182 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:30:18.182 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.096107+0000", 2024-02-09T17:30:18.182 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.584e-06 2024-02-09T17:30:18.182 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.182 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.183 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:30:18.183 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.111965+0000", 2024-02-09T17:30:18.183 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0158575459999999 2024-02-09T17:30:18.183 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.183 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.183 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:30:18.183 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.112024+0000", 2024-02-09T17:30:18.183 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.9389999999999999e-05 2024-02-09T17:30:18.183 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:30:18.183 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:30:18.183 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:30:18.183 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.183 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.183 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:184954 2.1 2:8cd4e379:::benchmark_data_smithi179_77138_object51546:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e157)", 2024-02-09T17:30:18.184 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:27:17.093016+0000", 2024-02-09T17:30:18.184 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 181.052049455, 2024-02-09T17:30:18.184 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.04845681, 2024-02-09T17:30:18.184 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:30:18.184 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "reached pg", 2024-02-09T17:30:18.184 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:30:18.184 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:30:18.184 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:30:18.184 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 184954 2024-02-09T17:30:18.184 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.184 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:30:18.184 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.184 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:30:18.184 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.093016+0000", 2024-02-09T17:30:18.185 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:30:18.185 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.185 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.185 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:30:18.185 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.093016+0000", 2024-02-09T17:30:18.185 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:30:18.185 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.185 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.185 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:30:18.185 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.093016+0000", 2024-02-09T17:30:18.185 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.3700000000000001e-07 2024-02-09T17:30:18.185 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.185 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.185 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:30:18.186 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.093019+0000", 2024-02-09T17:30:18.186 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.1619999999999998e-06 2024-02-09T17:30:18.186 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.186 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.186 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:30:18.186 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.093024+0000", 2024-02-09T17:30:18.186 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.9170000000000003e-06 2024-02-09T17:30:18.186 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.186 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.186 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:30:18.186 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.093029+0000", 2024-02-09T17:30:18.186 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.9570000000000001e-06 2024-02-09T17:30:18.186 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.186 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.186 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:30:18.187 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.096128+0000", 2024-02-09T17:30:18.187 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0030987290000000002 2024-02-09T17:30:18.187 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.187 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.187 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:30:18.187 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.096135+0000", 2024-02-09T17:30:18.187 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.6390000000000003e-06 2024-02-09T17:30:18.187 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.187 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.187 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:30:18.187 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.112051+0000", 2024-02-09T17:30:18.187 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.01591578 2024-02-09T17:30:18.187 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.187 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.187 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for missing object", 2024-02-09T17:30:18.188 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.112160+0000", 2024-02-09T17:30:18.188 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000108997 2024-02-09T17:30:18.188 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.188 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.188 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:30:18.188 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.141373+0000", 2024-02-09T17:30:18.188 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.029213948999999999 2024-02-09T17:30:18.188 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.188 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.188 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:30:18.188 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.141473+0000", 2024-02-09T17:30:18.188 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.9343000000000001e-05 2024-02-09T17:30:18.188 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:30:18.188 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:30:18.189 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:30:18.189 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.189 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.189 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:184960 2.e 2:7ca10664:::benchmark_data_smithi179_77138_object51552:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-02-09T17:30:18.189 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:27:17.094412+0000", 2024-02-09T17:30:18.189 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 181.05065379999999, 2024-02-09T17:30:18.189 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0601470550000001, 2024-02-09T17:30:18.189 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:30:18.189 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:30:18.189 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:30:18.189 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:30:18.189 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:30:18.189 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 184960 2024-02-09T17:30:18.189 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.189 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:30:18.190 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.190 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:30:18.190 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.094412+0000", 2024-02-09T17:30:18.190 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:30:18.190 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.190 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.190 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:30:18.190 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.094412+0000", 2024-02-09T17:30:18.190 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:30:18.190 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.190 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.190 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:30:18.190 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.094413+0000", 2024-02-09T17:30:18.190 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.094e-06 2024-02-09T17:30:18.191 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.191 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.191 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:30:18.191 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.094417+0000", 2024-02-09T17:30:18.191 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9090000000000002e-06 2024-02-09T17:30:18.191 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.191 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.191 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:30:18.191 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.094442+0000", 2024-02-09T17:30:18.191 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.5318000000000001e-05 2024-02-09T17:30:18.191 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.191 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.191 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:30:18.191 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.094451+0000", 2024-02-09T17:30:18.192 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.9029999999999993e-06 2024-02-09T17:30:18.192 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.192 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.192 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:30:18.192 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.094495+0000", 2024-02-09T17:30:18.192 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.4149e-05 2024-02-09T17:30:18.192 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.192 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.192 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:30:18.192 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.094501+0000", 2024-02-09T17:30:18.192 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.1580000000000002e-06 2024-02-09T17:30:18.192 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.192 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.192 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:30:18.193 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.123719+0000", 2024-02-09T17:30:18.193 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.029218159 2024-02-09T17:30:18.193 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.193 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.193 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:30:18.193 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.124720+0000", 2024-02-09T17:30:18.193 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.001000686 2024-02-09T17:30:18.193 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.193 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.193 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 2,9", 2024-02-09T17:30:18.193 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.124823+0000", 2024-02-09T17:30:18.193 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000103393 2024-02-09T17:30:18.194 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.194 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.194 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:30:18.194 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.140086+0000", 2024-02-09T17:30:18.194 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.015262354000000001 2024-02-09T17:30:18.194 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.194 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.195 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:30:18.195 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.152069+0000", 2024-02-09T17:30:18.195 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.011983428000000001 2024-02-09T17:30:18.195 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.195 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.195 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:30:18.195 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.154456+0000", 2024-02-09T17:30:18.195 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0023868320000000002 2024-02-09T17:30:18.195 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.195 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.195 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:30:18.195 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.154516+0000", 2024-02-09T17:30:18.196 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.0018e-05 2024-02-09T17:30:18.196 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.196 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.196 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:30:18.196 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.154559+0000", 2024-02-09T17:30:18.196 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2654000000000002e-05 2024-02-09T17:30:18.196 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:30:18.196 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:30:18.196 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:30:18.196 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.196 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.197 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:184962 2.1 2:852fa7f1:::benchmark_data_smithi179_77138_object51554:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-02-09T17:30:18.197 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:27:17.100142+0000", 2024-02-09T17:30:18.197 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 181.044922985, 2024-02-09T17:30:18.197 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.040636034, 2024-02-09T17:30:18.197 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:30:18.197 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:30:18.197 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:30:18.197 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:30:18.197 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:30:18.197 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 184962 2024-02-09T17:30:18.197 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.197 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:30:18.198 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.198 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:30:18.198 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.100142+0000", 2024-02-09T17:30:18.198 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:30:18.198 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.198 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.198 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:30:18.199 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.100142+0000", 2024-02-09T17:30:18.199 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:30:18.199 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.199 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.199 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:30:18.199 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.100145+0000", 2024-02-09T17:30:18.199 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.1469999999999999e-06 2024-02-09T17:30:18.199 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.199 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.200 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:30:18.200 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.100150+0000", 2024-02-09T17:30:18.200 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7620000000000001e-06 2024-02-09T17:30:18.200 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.200 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.200 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:30:18.200 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.100168+0000", 2024-02-09T17:30:18.200 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.7669e-05 2024-02-09T17:30:18.200 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.200 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.201 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:30:18.201 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.100177+0000", 2024-02-09T17:30:18.201 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.3589999999999997e-06 2024-02-09T17:30:18.201 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.201 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.201 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:30:18.201 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.100234+0000", 2024-02-09T17:30:18.201 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6372999999999997e-05 2024-02-09T17:30:18.201 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.201 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.201 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:30:18.201 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.100242+0000", 2024-02-09T17:30:18.201 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.1089999999999998e-06 2024-02-09T17:30:18.202 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.202 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.202 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:30:18.202 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.112193+0000", 2024-02-09T17:30:18.202 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.011950908 2024-02-09T17:30:18.202 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.202 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.202 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:30:18.202 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.112296+0000", 2024-02-09T17:30:18.202 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000103564 2024-02-09T17:30:18.202 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.203 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.203 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,2", 2024-02-09T17:30:18.203 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.112444+0000", 2024-02-09T17:30:18.203 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014731499999999999 2024-02-09T17:30:18.203 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.203 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.203 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:30:18.203 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.131924+0000", 2024-02-09T17:30:18.203 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.019480417999999999 2024-02-09T17:30:18.203 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.203 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.203 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:30:18.204 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.136490+0000", 2024-02-09T17:30:18.204 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0045657029999999999 2024-02-09T17:30:18.204 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.204 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.204 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:30:18.204 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.140631+0000", 2024-02-09T17:30:18.204 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00414081 2024-02-09T17:30:18.204 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.204 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.204 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:30:18.204 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.140685+0000", 2024-02-09T17:30:18.205 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.4129000000000003e-05 2024-02-09T17:30:18.205 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.205 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.205 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:30:18.205 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.140779+0000", 2024-02-09T17:30:18.205 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.3768000000000001e-05 2024-02-09T17:30:18.205 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:30:18.205 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:30:18.205 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:30:18.205 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.205 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.206 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:184970 2.e 2:7014206b:::benchmark_data_smithi179_77138_object51562:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-02-09T17:30:18.206 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:27:17.116640+0000", 2024-02-09T17:30:18.206 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 181.02842583399999, 2024-02-09T17:30:18.206 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.038024061, 2024-02-09T17:30:18.206 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:30:18.206 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:30:18.206 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:30:18.206 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:30:18.206 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:30:18.206 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 184970 2024-02-09T17:30:18.206 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.206 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:30:18.207 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.207 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:30:18.207 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.116640+0000", 2024-02-09T17:30:18.207 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:30:18.207 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.207 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.207 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:30:18.207 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.116640+0000", 2024-02-09T17:30:18.207 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:30:18.207 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.207 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.207 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:30:18.208 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.116642+0000", 2024-02-09T17:30:18.208 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.075e-06 2024-02-09T17:30:18.208 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.208 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.208 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:30:18.208 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.116648+0000", 2024-02-09T17:30:18.208 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.3740000000000003e-06 2024-02-09T17:30:18.208 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.208 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.208 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:30:18.208 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.116665+0000", 2024-02-09T17:30:18.208 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.7189000000000001e-05 2024-02-09T17:30:18.209 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.209 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.209 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:30:18.209 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.116676+0000", 2024-02-09T17:30:18.209 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0382e-05 2024-02-09T17:30:18.209 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.209 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.209 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:30:18.209 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.116749+0000", 2024-02-09T17:30:18.209 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.3664000000000005e-05 2024-02-09T17:30:18.209 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.209 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.209 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:30:18.209 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.116761+0000", 2024-02-09T17:30:18.210 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2179e-05 2024-02-09T17:30:18.210 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.210 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.210 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:30:18.210 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.129777+0000", 2024-02-09T17:30:18.210 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.013015362 2024-02-09T17:30:18.210 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.210 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.210 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:30:18.210 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.129887+0000", 2024-02-09T17:30:18.210 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000110572 2024-02-09T17:30:18.210 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.210 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.210 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 2,9", 2024-02-09T17:30:18.211 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.129999+0000", 2024-02-09T17:30:18.211 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00011152799999999999 2024-02-09T17:30:18.211 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.211 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.211 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:30:18.211 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.140094+0000", 2024-02-09T17:30:18.211 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.010094629000000001 2024-02-09T17:30:18.211 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.211 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.211 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:30:18.211 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.152148+0000", 2024-02-09T17:30:18.211 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.012054252 2024-02-09T17:30:18.212 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.212 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.212 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:30:18.212 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.154593+0000", 2024-02-09T17:30:18.212 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0024455470000000002 2024-02-09T17:30:18.212 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.212 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.212 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:30:18.212 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.154634+0000", 2024-02-09T17:30:18.212 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.0747999999999997e-05 2024-02-09T17:30:18.212 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.212 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.213 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:30:18.213 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.154664+0000", 2024-02-09T17:30:18.213 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.9560000000000002e-05 2024-02-09T17:30:18.213 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:30:18.213 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:30:18.213 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:30:18.213 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.213 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.213 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:184974 2.e 2:76fcde3c:::benchmark_data_smithi179_77138_object51566:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-02-09T17:30:18.213 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:27:17.120619+0000", 2024-02-09T17:30:18.213 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 181.02444661499999, 2024-02-09T17:30:18.213 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.034143357, 2024-02-09T17:30:18.214 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:30:18.214 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:30:18.214 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:30:18.214 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:30:18.214 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:30:18.214 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 184974 2024-02-09T17:30:18.214 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.214 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:30:18.214 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.214 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:30:18.214 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.120619+0000", 2024-02-09T17:30:18.214 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:30:18.214 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.215 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.215 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:30:18.215 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.120619+0000", 2024-02-09T17:30:18.215 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:30:18.215 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.215 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.215 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:30:18.215 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.120621+0000", 2024-02-09T17:30:18.215 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0219999999999999e-06 2024-02-09T17:30:18.215 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.215 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.215 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:30:18.216 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.120642+0000", 2024-02-09T17:30:18.216 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.1217e-05 2024-02-09T17:30:18.216 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.216 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.216 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:30:18.216 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.120658+0000", 2024-02-09T17:30:18.216 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5982999999999999e-05 2024-02-09T17:30:18.216 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.216 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.216 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:30:18.216 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.120670+0000", 2024-02-09T17:30:18.217 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2319e-05 2024-02-09T17:30:18.217 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.217 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.217 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:30:18.217 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.120742+0000", 2024-02-09T17:30:18.217 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.1298999999999996e-05 2024-02-09T17:30:18.217 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.217 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.217 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:30:18.217 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.120751+0000", 2024-02-09T17:30:18.217 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.4550000000000001e-06 2024-02-09T17:30:18.217 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.217 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.218 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:30:18.218 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.134861+0000", 2024-02-09T17:30:18.218 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.01410986 2024-02-09T17:30:18.218 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.218 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.218 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:30:18.218 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.134980+0000", 2024-02-09T17:30:18.218 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000119104 2024-02-09T17:30:18.218 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.218 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.218 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 2,9", 2024-02-09T17:30:18.218 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.135091+0000", 2024-02-09T17:30:18.218 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000111371 2024-02-09T17:30:18.219 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.219 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.219 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:30:18.219 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.140654+0000", 2024-02-09T17:30:18.219 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0055621079999999996 2024-02-09T17:30:18.219 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.219 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.219 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:30:18.219 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.152182+0000", 2024-02-09T17:30:18.220 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.011528007999999999 2024-02-09T17:30:18.220 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.220 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.220 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:30:18.220 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.154693+0000", 2024-02-09T17:30:18.220 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.002511882 2024-02-09T17:30:18.220 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.220 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.220 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:30:18.220 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.154733+0000", 2024-02-09T17:30:18.220 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9700999999999998e-05 2024-02-09T17:30:18.220 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.221 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.221 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:30:18.221 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.154762+0000", 2024-02-09T17:30:18.221 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.9028000000000001e-05 2024-02-09T17:30:18.221 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:30:18.221 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:30:18.221 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:30:18.221 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.221 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.221 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:184988 2.1 2:874576dd:::benchmark_data_smithi179_77138_object51580:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-02-09T17:30:18.221 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:27:17.146034+0000", 2024-02-09T17:30:18.221 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 180.99903168399999, 2024-02-09T17:30:18.221 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.99503818200000005, 2024-02-09T17:30:18.222 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:30:18.222 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:30:18.222 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:30:18.222 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:30:18.222 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:30:18.222 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 184988 2024-02-09T17:30:18.222 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.222 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:30:18.222 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.222 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:30:18.222 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.146034+0000", 2024-02-09T17:30:18.222 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:30:18.223 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.223 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.223 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:30:18.223 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.146034+0000", 2024-02-09T17:30:18.223 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:30:18.223 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.223 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.223 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:30:18.223 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.146036+0000", 2024-02-09T17:30:18.223 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0209999999999998e-06 2024-02-09T17:30:18.223 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.223 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.223 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:30:18.224 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.146046+0000", 2024-02-09T17:30:18.224 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0037999999999999e-05 2024-02-09T17:30:18.224 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.224 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.224 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:30:18.224 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.146065+0000", 2024-02-09T17:30:18.224 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.9548e-05 2024-02-09T17:30:18.224 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.224 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.224 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:30:18.224 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.146076+0000", 2024-02-09T17:30:18.224 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0662000000000001e-05 2024-02-09T17:30:18.224 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.225 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.225 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:30:18.225 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.146154+0000", 2024-02-09T17:30:18.225 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.8104999999999998e-05 2024-02-09T17:30:18.225 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.225 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.225 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:30:18.225 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.146166+0000", 2024-02-09T17:30:18.225 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1999e-05 2024-02-09T17:30:18.225 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.225 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.225 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:30:18.226 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.117484+0000", 2024-02-09T17:30:18.226 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.97131762300000002 2024-02-09T17:30:18.226 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.226 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.226 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:30:18.226 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.117590+0000", 2024-02-09T17:30:18.226 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000106617 2024-02-09T17:30:18.226 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.226 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.226 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,2", 2024-02-09T17:30:18.226 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.117702+0000", 2024-02-09T17:30:18.226 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000111999 2024-02-09T17:30:18.227 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.227 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.227 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:30:18.227 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.136530+0000", 2024-02-09T17:30:18.227 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.018827831 2024-02-09T17:30:18.227 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.227 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.227 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:30:18.228 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.140585+0000", 2024-02-09T17:30:18.228 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0040543089999999999 2024-02-09T17:30:18.228 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.228 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.228 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:30:18.228 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.140954+0000", 2024-02-09T17:30:18.228 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00036901700000000002 2024-02-09T17:30:18.228 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.228 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.228 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:30:18.228 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.140993+0000", 2024-02-09T17:30:18.228 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9190000000000003e-05 2024-02-09T17:30:18.229 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.229 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.229 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:30:18.229 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.141072+0000", 2024-02-09T17:30:18.229 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.9222999999999997e-05 2024-02-09T17:30:18.229 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:30:18.229 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:30:18.229 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:30:18.229 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.229 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.229 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:185008 2.1 2:82559698:::benchmark_data_smithi179_77138_object51600:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-02-09T17:30:18.229 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:27:17.187102+0000", 2024-02-09T17:30:18.229 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 180.95796385099999, 2024-02-09T17:30:18.230 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.95420838900000005, 2024-02-09T17:30:18.230 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:30:18.230 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:30:18.230 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:30:18.230 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:30:18.230 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:30:18.230 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 185008 2024-02-09T17:30:18.230 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.230 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:30:18.230 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.230 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:30:18.230 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.187102+0000", 2024-02-09T17:30:18.230 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:30:18.230 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.230 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.231 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:30:18.231 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.187102+0000", 2024-02-09T17:30:18.231 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:30:18.231 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.231 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.231 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:30:18.231 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.187103+0000", 2024-02-09T17:30:18.231 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3030000000000001e-06 2024-02-09T17:30:18.231 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.231 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.231 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:30:18.231 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.187108+0000", 2024-02-09T17:30:18.231 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.7879999999999997e-06 2024-02-09T17:30:18.231 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.231 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.232 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:30:18.232 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.187117+0000", 2024-02-09T17:30:18.232 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.5610000000000004e-06 2024-02-09T17:30:18.232 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.232 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.232 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:30:18.232 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.187125+0000", 2024-02-09T17:30:18.232 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.9670000000000001e-06 2024-02-09T17:30:18.232 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.232 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.232 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:30:18.232 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.187194+0000", 2024-02-09T17:30:18.232 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.9233000000000006e-05 2024-02-09T17:30:18.232 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.233 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.233 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:30:18.233 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.187207+0000", 2024-02-09T17:30:18.233 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.242e-05 2024-02-09T17:30:18.233 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.233 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.233 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:30:18.233 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.122722+0000", 2024-02-09T17:30:18.233 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.93551549300000003 2024-02-09T17:30:18.233 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.233 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.233 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:30:18.233 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.122870+0000", 2024-02-09T17:30:18.233 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014733300000000001 2024-02-09T17:30:18.233 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.234 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.234 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,2", 2024-02-09T17:30:18.234 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.122968+0000", 2024-02-09T17:30:18.234 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.7922000000000002e-05 2024-02-09T17:30:18.234 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.234 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.234 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:30:18.234 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.140563+0000", 2024-02-09T17:30:18.234 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.017595823999999999 2024-02-09T17:30:18.234 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.234 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.234 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:30:18.234 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.140590+0000", 2024-02-09T17:30:18.234 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.6601000000000002e-05 2024-02-09T17:30:18.234 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.235 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.235 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:30:18.235 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.141197+0000", 2024-02-09T17:30:18.235 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000606497 2024-02-09T17:30:18.235 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.235 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.235 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:30:18.235 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.141237+0000", 2024-02-09T17:30:18.235 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.0674999999999999e-05 2024-02-09T17:30:18.235 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.235 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.235 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:30:18.235 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.141310+0000", 2024-02-09T17:30:18.235 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.2772000000000001e-05 2024-02-09T17:30:18.235 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:30:18.236 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:30:18.236 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:30:18.236 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.236 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.236 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:185010 2.1 2:8bb6ab63:::benchmark_data_smithi179_77138_object51602:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-02-09T17:30:18.236 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:27:17.188080+0000", 2024-02-09T17:30:18.236 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 180.95698503400001, 2024-02-09T17:30:18.236 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.98684645299999996, 2024-02-09T17:30:18.236 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:30:18.236 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:30:18.236 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:30:18.236 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:30:18.236 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:30:18.236 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 185010 2024-02-09T17:30:18.236 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.237 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:30:18.237 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.237 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:30:18.237 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.188080+0000", 2024-02-09T17:30:18.237 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:30:18.237 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.237 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.237 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:30:18.237 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.188080+0000", 2024-02-09T17:30:18.237 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:30:18.237 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.237 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.237 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:30:18.237 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.188081+0000", 2024-02-09T17:30:18.237 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.051e-06 2024-02-09T17:30:18.238 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.238 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.238 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:30:18.238 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.188086+0000", 2024-02-09T17:30:18.238 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.3109999999999999e-06 2024-02-09T17:30:18.238 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.238 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.238 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:30:18.238 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.188095+0000", 2024-02-09T17:30:18.238 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.5589999999999994e-06 2024-02-09T17:30:18.238 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.238 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.238 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:30:18.238 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.188102+0000", 2024-02-09T17:30:18.238 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.793e-06 2024-02-09T17:30:18.239 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.239 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.239 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:30:18.239 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.188148+0000", 2024-02-09T17:30:18.239 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.5485000000000002e-05 2024-02-09T17:30:18.239 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.239 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.239 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:30:18.239 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.188154+0000", 2024-02-09T17:30:18.239 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.7320000000000001e-06 2024-02-09T17:30:18.239 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.239 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.239 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:30:18.239 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.131962+0000", 2024-02-09T17:30:18.240 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.94380807 2024-02-09T17:30:18.240 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.240 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.240 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:30:18.240 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.132042+0000", 2024-02-09T17:30:18.240 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.9592999999999995e-05 2024-02-09T17:30:18.240 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.240 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.240 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,2", 2024-02-09T17:30:18.240 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.132130+0000", 2024-02-09T17:30:18.240 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.7619000000000003e-05 2024-02-09T17:30:18.240 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.240 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.241 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:30:18.241 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.140920+0000", 2024-02-09T17:30:18.241 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0087908540000000007 2024-02-09T17:30:18.241 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.241 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.241 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:30:18.241 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.158320+0000", 2024-02-09T17:30:18.241 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.017399936000000001 2024-02-09T17:30:18.241 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.241 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.241 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:30:18.241 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.174829+0000", 2024-02-09T17:30:18.241 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.01650861 2024-02-09T17:30:18.242 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.242 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.242 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:30:18.242 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.174886+0000", 2024-02-09T17:30:18.242 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6555000000000003e-05 2024-02-09T17:30:18.242 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.242 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.242 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:30:18.242 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.174927+0000", 2024-02-09T17:30:18.242 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.1285000000000002e-05 2024-02-09T17:30:18.242 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:30:18.242 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:30:18.242 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:30:18.242 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.243 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.243 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:185011 2.1 2:8f900dc9:::benchmark_data_smithi179_77138_object51603:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-02-09T17:30:18.243 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:27:17.192112+0000", 2024-02-09T17:30:18.243 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 180.952953038, 2024-02-09T17:30:18.243 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.98298666000000001, 2024-02-09T17:30:18.243 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:30:18.243 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:30:18.243 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:30:18.243 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:30:18.243 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:30:18.243 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 185011 2024-02-09T17:30:18.243 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.243 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:30:18.243 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.244 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:30:18.244 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.192112+0000", 2024-02-09T17:30:18.244 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:30:18.244 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.244 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.244 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:30:18.244 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.192112+0000", 2024-02-09T17:30:18.244 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:30:18.244 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.244 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.244 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:30:18.244 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.192113+0000", 2024-02-09T17:30:18.244 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.850000000000001e-07 2024-02-09T17:30:18.244 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.244 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.245 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:30:18.245 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.192118+0000", 2024-02-09T17:30:18.245 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.0420000000000002e-06 2024-02-09T17:30:18.245 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.245 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.245 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:30:18.245 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.192128+0000", 2024-02-09T17:30:18.245 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0029e-05 2024-02-09T17:30:18.245 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.245 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.245 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:30:18.245 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.192135+0000", 2024-02-09T17:30:18.245 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.4370000000000004e-06 2024-02-09T17:30:18.245 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.246 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.246 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:30:18.246 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.192207+0000", 2024-02-09T17:30:18.246 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.1892999999999998e-05 2024-02-09T17:30:18.246 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.246 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.246 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:30:18.246 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:17.192220+0000", 2024-02-09T17:30:18.246 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3016999999999999e-05 2024-02-09T17:30:18.246 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.246 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.246 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:30:18.246 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.141494+0000", 2024-02-09T17:30:18.246 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.94927378699999998 2024-02-09T17:30:18.246 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.247 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.247 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:30:18.247 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.141609+0000", 2024-02-09T17:30:18.247 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00011494599999999999 2024-02-09T17:30:18.247 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.247 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.247 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,2", 2024-02-09T17:30:18.247 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.141732+0000", 2024-02-09T17:30:18.247 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012374400000000001 2024-02-09T17:30:18.247 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.247 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.247 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:30:18.247 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.153713+0000", 2024-02-09T17:30:18.247 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.011980263999999999 2024-02-09T17:30:18.247 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.248 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.248 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:30:18.248 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.174963+0000", 2024-02-09T17:30:18.248 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.021250905 2024-02-09T17:30:18.248 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.248 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.248 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:30:18.248 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.175031+0000", 2024-02-09T17:30:18.248 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.8028000000000007e-05 2024-02-09T17:30:18.248 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.248 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.248 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:30:18.248 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.175072+0000", 2024-02-09T17:30:18.248 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.0287000000000002e-05 2024-02-09T17:30:18.248 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.249 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.249 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:30:18.249 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:27:18.175099+0000", 2024-02-09T17:30:18.249 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.7296000000000001e-05 2024-02-09T17:30:18.249 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:30:18.249 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:30:18.249 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:30:18.249 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.249 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.249 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.24817.0:11279 3.7 3:e75b4de5:::benchmark_data_smithi179_90440_object11278:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e208)", 2024-02-09T17:30:18.249 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:30:08.639528+0000", 2024-02-09T17:30:18.249 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 9.5055379450000004, 2024-02-09T17:30:18.249 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20480663800000001, 2024-02-09T17:30:18.249 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:30:18.249 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:30:18.250 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:30:18.250 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.24817", 2024-02-09T17:30:18.250 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:30:18.250 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 11279 2024-02-09T17:30:18.250 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.250 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:30:18.250 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.250 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:30:18.250 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.639528+0000", 2024-02-09T17:30:18.250 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:30:18.250 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.250 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.250 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:30:18.250 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.639528+0000", 2024-02-09T17:30:18.250 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:30:18.251 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.251 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.251 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:30:18.251 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.639528+0000", 2024-02-09T17:30:18.251 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.3399999999999997e-07 2024-02-09T17:30:18.251 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.251 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.251 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:30:18.251 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.639557+0000", 2024-02-09T17:30:18.251 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.8126000000000001e-05 2024-02-09T17:30:18.251 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.251 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.251 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:30:18.251 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.639567+0000", 2024-02-09T17:30:18.252 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0445000000000001e-05 2024-02-09T17:30:18.252 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.252 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.252 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:30:18.252 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.639575+0000", 2024-02-09T17:30:18.252 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.537e-06 2024-02-09T17:30:18.252 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.252 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.252 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:30:18.252 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.639642+0000", 2024-02-09T17:30:18.252 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.7873999999999994e-05 2024-02-09T17:30:18.252 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.252 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.253 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:30:18.253 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.639763+0000", 2024-02-09T17:30:18.253 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000120549 2024-02-09T17:30:18.253 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.253 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.253 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 7,11", 2024-02-09T17:30:18.253 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.639902+0000", 2024-02-09T17:30:18.253 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000138897 2024-02-09T17:30:18.253 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.253 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.253 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:30:18.253 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.641224+0000", 2024-02-09T17:30:18.253 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0013224129999999999 2024-02-09T17:30:18.253 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.253 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.253 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:30:18.254 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.642077+0000", 2024-02-09T17:30:18.254 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000853019 2024-02-09T17:30:18.254 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.254 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.254 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:30:18.254 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.844250+0000", 2024-02-09T17:30:18.254 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20217227300000001 2024-02-09T17:30:18.254 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.254 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.254 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:30:18.254 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.844295+0000", 2024-02-09T17:30:18.254 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.5271000000000001e-05 2024-02-09T17:30:18.254 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.254 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.255 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:30:18.255 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.844334+0000", 2024-02-09T17:30:18.255 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.93e-05 2024-02-09T17:30:18.255 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:30:18.255 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:30:18.255 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:30:18.255 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.255 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.255 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.24817.0:11292 3.7 3:e99cce9b:::benchmark_data_smithi179_90440_object11291:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e208)", 2024-02-09T17:30:18.255 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:30:08.642034+0000", 2024-02-09T17:30:18.255 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 9.5030317750000002, 2024-02-09T17:30:18.255 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20453410599999999, 2024-02-09T17:30:18.255 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:30:18.255 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:30:18.255 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:30:18.256 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.24817", 2024-02-09T17:30:18.256 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:30:18.256 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 11292 2024-02-09T17:30:18.256 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.256 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:30:18.256 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.256 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:30:18.256 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.642034+0000", 2024-02-09T17:30:18.256 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:30:18.256 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.256 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.256 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:30:18.256 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.642034+0000", 2024-02-09T17:30:18.256 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:30:18.256 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.257 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.257 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:30:18.257 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.642035+0000", 2024-02-09T17:30:18.257 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.4799999999999997e-07 2024-02-09T17:30:18.257 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.257 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.257 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:30:18.257 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.642065+0000", 2024-02-09T17:30:18.257 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.0340000000000001e-05 2024-02-09T17:30:18.257 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.257 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.257 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:30:18.257 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.642076+0000", 2024-02-09T17:30:18.257 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0869999999999999e-05 2024-02-09T17:30:18.258 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.258 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.258 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:30:18.258 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.642084+0000", 2024-02-09T17:30:18.258 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.7859999999999997e-06 2024-02-09T17:30:18.258 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.258 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.258 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:30:18.258 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.642147+0000", 2024-02-09T17:30:18.258 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.2891000000000005e-05 2024-02-09T17:30:18.258 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.258 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.258 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:30:18.258 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.642376+0000", 2024-02-09T17:30:18.259 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00022913600000000001 2024-02-09T17:30:18.259 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.259 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.259 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 7,11", 2024-02-09T17:30:18.259 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.642479+0000", 2024-02-09T17:30:18.259 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000103137 2024-02-09T17:30:18.259 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.259 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.259 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:30:18.259 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.643039+0000", 2024-02-09T17:30:18.259 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00056032800000000002 2024-02-09T17:30:18.259 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.259 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.259 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:30:18.259 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.645347+0000", 2024-02-09T17:30:18.260 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0023074620000000001 2024-02-09T17:30:18.260 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.260 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.260 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:30:18.260 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.846464+0000", 2024-02-09T17:30:18.260 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.201117196 2024-02-09T17:30:18.260 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.260 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.260 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:30:18.260 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.846519+0000", 2024-02-09T17:30:18.260 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.5399000000000003e-05 2024-02-09T17:30:18.260 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.260 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.260 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:30:18.260 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.846568+0000", 2024-02-09T17:30:18.261 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.8612999999999997e-05 2024-02-09T17:30:18.261 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:30:18.261 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:30:18.261 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:30:18.261 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.261 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.261 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.24817.0:11796 3.b 3:de431111:::benchmark_data_smithi179_90440_object11795:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e208)", 2024-02-09T17:30:18.261 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:30:08.920262+0000", 2024-02-09T17:30:18.261 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 9.2248033409999994, 2024-02-09T17:30:18.261 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.204893666, 2024-02-09T17:30:18.261 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:30:18.261 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:30:18.261 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:30:18.261 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.24817", 2024-02-09T17:30:18.261 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:30:18.262 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 11796 2024-02-09T17:30:18.262 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.262 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:30:18.262 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.262 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:30:18.262 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.920262+0000", 2024-02-09T17:30:18.262 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:30:18.262 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.262 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.262 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:30:18.262 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.920262+0000", 2024-02-09T17:30:18.262 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:30:18.262 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.262 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.262 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:30:18.263 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.920263+0000", 2024-02-09T17:30:18.263 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.540000000000001e-07 2024-02-09T17:30:18.263 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.263 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.263 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:30:18.263 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.920689+0000", 2024-02-09T17:30:18.263 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00042572999999999998 2024-02-09T17:30:18.263 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.263 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.263 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:30:18.263 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.920701+0000", 2024-02-09T17:30:18.263 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1926999999999999e-05 2024-02-09T17:30:18.263 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.263 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.263 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:30:18.264 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.920709+0000", 2024-02-09T17:30:18.264 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.3210000000000005e-06 2024-02-09T17:30:18.264 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.264 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.264 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:30:18.264 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.920777+0000", 2024-02-09T17:30:18.264 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.7860000000000004e-05 2024-02-09T17:30:18.264 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.264 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.264 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:30:18.264 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.920899+0000", 2024-02-09T17:30:18.264 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012191700000000001 2024-02-09T17:30:18.264 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.265 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.265 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,4", 2024-02-09T17:30:18.265 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.921039+0000", 2024-02-09T17:30:18.265 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014013900000000001 2024-02-09T17:30:18.265 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.265 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.265 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:30:18.265 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.923413+0000", 2024-02-09T17:30:18.265 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0023736959999999998 2024-02-09T17:30:18.265 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.265 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.265 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:30:18.265 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.923950+0000", 2024-02-09T17:30:18.265 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00053755899999999995 2024-02-09T17:30:18.266 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.266 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.266 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:30:18.266 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:09.125066+0000", 2024-02-09T17:30:18.266 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.201115567 2024-02-09T17:30:18.266 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.266 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.266 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:30:18.266 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:09.125113+0000", 2024-02-09T17:30:18.266 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.7540000000000002e-05 2024-02-09T17:30:18.266 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.266 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.266 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:30:18.266 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:09.125156+0000", 2024-02-09T17:30:18.266 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2456000000000001e-05 2024-02-09T17:30:18.267 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:30:18.267 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:30:18.267 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:30:18.267 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.267 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.267 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.24817.0:11797 3.b 3:d443f766:::benchmark_data_smithi179_90440_object11796:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e208)", 2024-02-09T17:30:18.267 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:30:08.920740+0000", 2024-02-09T17:30:18.267 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 9.2243254510000003, 2024-02-09T17:30:18.267 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20456191600000001, 2024-02-09T17:30:18.267 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:30:18.267 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:30:18.267 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:30:18.267 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.24817", 2024-02-09T17:30:18.267 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:30:18.268 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 11797 2024-02-09T17:30:18.268 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.268 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:30:18.268 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.268 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:30:18.268 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.920740+0000", 2024-02-09T17:30:18.268 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:30:18.268 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.268 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.268 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:30:18.268 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.920740+0000", 2024-02-09T17:30:18.268 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:30:18.268 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.268 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.269 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:30:18.269 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.920741+0000", 2024-02-09T17:30:18.269 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.6000000000000003e-07 2024-02-09T17:30:18.269 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.269 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.269 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:30:18.269 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.920787+0000", 2024-02-09T17:30:18.269 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.6125000000000003e-05 2024-02-09T17:30:18.269 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.269 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.269 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:30:18.269 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.920795+0000", 2024-02-09T17:30:18.269 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.3399999999999998e-06 2024-02-09T17:30:18.269 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.270 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.270 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:30:18.270 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.920802+0000", 2024-02-09T17:30:18.270 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.9630000000000001e-06 2024-02-09T17:30:18.270 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.270 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.270 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:30:18.270 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.921783+0000", 2024-02-09T17:30:18.270 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00098128299999999989 2024-02-09T17:30:18.270 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.270 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.270 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:30:18.270 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.921915+0000", 2024-02-09T17:30:18.271 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013138000000000001 2024-02-09T17:30:18.271 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.271 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.271 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,4", 2024-02-09T17:30:18.271 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.922085+0000", 2024-02-09T17:30:18.271 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000170302 2024-02-09T17:30:18.271 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.271 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.271 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:30:18.271 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.923417+0000", 2024-02-09T17:30:18.271 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0013324109999999999 2024-02-09T17:30:18.271 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.271 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.272 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:30:18.272 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:08.923973+0000", 2024-02-09T17:30:18.272 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00055583599999999998 2024-02-09T17:30:18.272 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.272 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.272 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:30:18.272 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:09.125203+0000", 2024-02-09T17:30:18.272 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20122973899999999 2024-02-09T17:30:18.272 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.272 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.272 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:30:18.272 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:09.125262+0000", 2024-02-09T17:30:18.272 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.9295000000000001e-05 2024-02-09T17:30:18.272 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:30:18.273 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:30:18.273 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:30:18.273 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:09.125302+0000", 2024-02-09T17:30:18.273 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9582000000000002e-05 2024-02-09T17:30:18.273 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:30:18.273 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:30:18.273 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:30:18.273 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:30:18.273 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:30:18.273 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:30:18.532 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:30:18.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:18.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:30:18.638 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:30:18.685 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:18.718 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:30:18.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:18.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:30:18.824 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:30:18.867 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:18.939 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 13 16 37380 37364 179.62 156.5 0.00486531 0.0055624 2024-02-09T17:30:18.951 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:30:18.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:18.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:30:19.057 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:30:19.097 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:19.312 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:30:19.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:19.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:30:19.416 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:30:19.456 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:19.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:19 smithi099 ceph-mon[39420]: pgmap v616: 17 pgs: 17 active+clean; 1.7 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 168 MiB/s wr, 2.69k op/s 2024-02-09T17:30:19.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:19 smithi179 ceph-mon[43975]: pgmap v616: 17 pgs: 17 active+clean; 1.7 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 168 MiB/s wr, 2.69k op/s 2024-02-09T17:30:19.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:19 smithi162 ceph-mon[44304]: pgmap v616: 17 pgs: 17 active+clean; 1.7 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 168 MiB/s wr, 2.69k op/s 2024-02-09T17:30:19.939 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 14 16 39439 39423 175.982 128.688 0.0040795 0.00567779 2024-02-09T17:30:20.008 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:30:20.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:20.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:30:20.117 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:30:20.171 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:20.207 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:30:20.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:20.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:30:20.312 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:30:20.353 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:20.381 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:30:20.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:20.385 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:30:20.486 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:30:20.541 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:20.939 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 15 16 41736 41720 173.82 143.562 0.00571316 0.00574867 2024-02-09T17:30:21.197 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:30:21.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:21.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:30:21.245 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-09T17:30:21.308 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:30:21.363 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:21.387 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:30:21.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:21.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:30:21.493 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:30:21.547 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:21.579 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:30:21.579 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is set 2024-02-09T17:30:21.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:21.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:30:21.678 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:30:21.685 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:30:21.727 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:21.754 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:30:21.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:21.759 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:30:21.860 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:30:21.908 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:21 smithi099 ceph-mon[39420]: pgmap v617: 17 pgs: 17 active+clean; 2.2 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 167 MiB/s wr, 2.67k op/s 2024-02-09T17:30:21.908 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:21 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/401935775' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:30:21.910 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:21.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:21 smithi179 ceph-mon[43975]: pgmap v617: 17 pgs: 17 active+clean; 2.2 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 167 MiB/s wr, 2.67k op/s 2024-02-09T17:30:21.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:21 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/401935775' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:30:21.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:21 smithi162 ceph-mon[44304]: pgmap v617: 17 pgs: 17 active+clean; 2.2 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 167 MiB/s wr, 2.67k op/s 2024-02-09T17:30:21.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:21 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/401935775' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:30:21.939 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 16 16 43777 43761 170.928 127.562 0.00465385 0.00584622 2024-02-09T17:30:22.151 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:30:22.171 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:30:22.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:22.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:30:22.211 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:30:22.239 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:30:22.269 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:30:22.277 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:30:22.300 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:30:22.318 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:22.336 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:30:22.381 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:30:22.414 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:30:22.453 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:30:22.487 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:30:22.518 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:30:22.546 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:30:22.578 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:30:22.580 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:30:22.580 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:30:22.580 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:30:22.580 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:30:22.580 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:30:22.581 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:30:22.581 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:30:22.581 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:30:22.581 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:30:22.581 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:30:22.581 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:30:22.581 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:30:22.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:22 smithi179 ceph-mon[43975]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:30:22.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:22 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/401935775' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:30:22.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:22 smithi179 ceph-mon[43975]: osdmap e210: 12 total, 12 up, 12 in 2024-02-09T17:30:22.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:22 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/836850009' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:30:22.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:22 smithi162 ceph-mon[44304]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:30:22.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:22 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/401935775' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:30:22.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:22 smithi162 ceph-mon[44304]: osdmap e210: 12 total, 12 up, 12 in 2024-02-09T17:30:22.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:22 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/836850009' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:30:22.939 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 17 16 45888 45872 168.634 131.938 0.0153259 0.00592539 2024-02-09T17:30:22.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:22 smithi099 ceph-mon[39420]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:30:22.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:22 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/401935775' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:30:22.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:22 smithi099 ceph-mon[39420]: osdmap e210: 12 total, 12 up, 12 in 2024-02-09T17:30:22.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:22 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/836850009' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:30:23.822 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:23 smithi099 ceph-mon[39420]: pgmap v619: 17 pgs: 17 active+clean; 2.4 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 156 MiB/s wr, 2.50k op/s 2024-02-09T17:30:23.861 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:30:23.861 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:30:23.877 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:30:23.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:23.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:30:23.884 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:23 smithi162 ceph-mon[44304]: pgmap v619: 17 pgs: 17 active+clean; 2.4 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 156 MiB/s wr, 2.50k op/s 2024-02-09T17:30:23.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:23 smithi179 ceph-mon[43975]: pgmap v619: 17 pgs: 17 active+clean; 2.4 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 156 MiB/s wr, 2.50k op/s 2024-02-09T17:30:23.939 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 18 16 47899 47883 166.247 125.688 0.00794197 0.0060101 2024-02-09T17:30:23.982 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:30:24.035 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:24.096 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:30:24.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:24.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:30:24.201 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:30:24.251 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:24.601 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-09T17:30:24.611 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:30:24.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:24.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:30:24.715 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:30:24.757 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:24.769 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset noscrub 2024-02-09T17:30:24.792 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:30:24.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:24.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:30:24.898 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:30:24.930 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:24.937 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:30:24.939 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 19 16 49167 49151 161.668 79.25 0.00899026 0.00618125 2024-02-09T17:30:24.957 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:30:24.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:24.961 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:30:25.062 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:30:25.103 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:25.186 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:30:25.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:25.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:30:25.294 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:30:25.329 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:25.354 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:30:25.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:25.359 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:30:25.460 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:30:25.506 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:25.538 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:30:25.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:25.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:30:25.592 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:30:25.609 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:25 smithi179 ceph-mon[43975]: pgmap v620: 17 pgs: 17 active+clean; 2.8 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 146 MiB/s wr, 2.34k op/s 2024-02-09T17:30:25.641 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:30:25.647 INFO:teuthology.orchestra.run.smithi099.stderr:no valid command found; 10 closest matches: 2024-02-09T17:30:25.647 INFO:teuthology.orchestra.run.smithi099.stderr:0 2024-02-09T17:30:25.647 INFO:teuthology.orchestra.run.smithi099.stderr:1 2024-02-09T17:30:25.647 INFO:teuthology.orchestra.run.smithi099.stderr:2 2024-02-09T17:30:25.648 INFO:teuthology.orchestra.run.smithi099.stderr:abort 2024-02-09T17:30:25.648 INFO:teuthology.orchestra.run.smithi099.stderr:assert 2024-02-09T17:30:25.648 INFO:teuthology.orchestra.run.smithi099.stderr:bench [] [] [] [] 2024-02-09T17:30:25.648 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs debug_inject_read_zeros 2024-02-09T17:30:25.648 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs files list 2024-02-09T17:30:25.648 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs stats 2024-02-09T17:30:25.648 INFO:teuthology.orchestra.run.smithi099.stderr:bluestore allocator dump block 2024-02-09T17:30:25.648 INFO:teuthology.orchestra.run.smithi099.stderr:Error EINVAL: invalid command 2024-02-09T17:30:25.679 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:30:25.705 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:25.828 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:25 smithi162 ceph-mon[44304]: pgmap v620: 17 pgs: 17 active+clean; 2.8 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 146 MiB/s wr, 2.34k op/s 2024-02-09T17:30:25.922 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:30:25.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:25.928 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:25 smithi099 ceph-mon[39420]: pgmap v620: 17 pgs: 17 active+clean; 2.8 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 146 MiB/s wr, 2.34k op/s 2024-02-09T17:30:25.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:30:25.939 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:2024-02-09T17:30:25.939619+0000 min lat: 0.00110138 max lat: 0.211514 avg lat: 0.00627069 2024-02-09T17:30:25.940 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-09T17:30:25.940 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 20 16 51018 51002 159.369 115.688 0.0116457 0.00627069 2024-02-09T17:30:26.029 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:30:26.063 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:26.092 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:30:26.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:26.097 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:30:26.198 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:30:26.237 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:26.493 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:30:26.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:26.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:30:26.537 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-09T17:30:26.538 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-02-09T17:30:26.538 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph tell osd.7 'trim stale osdmaps' 2024-02-09T17:30:26.601 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:30:26.650 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:26.789 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:30:26.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:26.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:30:26.893 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:30:26.940 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 21 16 52896 52880 157.368 117.375 0.00607578 0.00635011 2024-02-09T17:30:26.954 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:26.984 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:30:26.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:26.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:30:27.088 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:30:27.129 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:27.156 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:30:27.156 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:30:27.192 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:30:27.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:27.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:30:27.297 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:30:27.335 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:27.361 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:30:27.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:27.366 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:30:27.466 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:30:27.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:27 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:30:27.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:27 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:30:27.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:27 smithi099 ceph-mon[39420]: pgmap v621: 17 pgs: 17 active+clean; 2.9 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 140 MiB/s wr, 2.24k op/s 2024-02-09T17:30:27.507 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:27.535 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:27 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:30:27.535 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:27 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:30:27.535 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:27 smithi162 ceph-mon[44304]: pgmap v621: 17 pgs: 17 active+clean; 2.9 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 140 MiB/s wr, 2.24k op/s 2024-02-09T17:30:27.589 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:30:27.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:27.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:30:27.637 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:30:27.646 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:30:27.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:27 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:30:27.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:27 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:30:27.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:27 smithi179 ceph-mon[43975]: pgmap v621: 17 pgs: 17 active+clean; 2.9 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 140 MiB/s wr, 2.24k op/s 2024-02-09T17:30:27.699 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:30:27.742 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:27.791 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:30:27.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:27.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:30:27.897 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:30:27.899 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-09T17:30:27.934 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:27.940 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 22 16 54738 54722 155.448 115.125 0.0112556 0.00642813 2024-02-09T17:30:27.959 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:30:27.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:27.963 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:30:28.064 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:30:28.101 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:28.130 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:30:28.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:28.133 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:30:28.234 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:30:28.299 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:28.319 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:30:28.335 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:30:28.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:28.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:30:28.440 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:30:28.474 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:28 smithi099 ceph-mon[39420]: osdmap e211: 12 total, 12 up, 12 in 2024-02-09T17:30:28.494 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:28.597 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:28 smithi162 ceph-mon[44304]: osdmap e211: 12 total, 12 up, 12 in 2024-02-09T17:30:28.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:28 smithi179 ceph-mon[43975]: osdmap e211: 12 total, 12 up, 12 in 2024-02-09T17:30:28.849 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:30:28.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:28.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:30:28.940 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 23 16 56516 56500 153.52 111.125 0.00380453 0.00651048 2024-02-09T17:30:28.957 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:30:28.989 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:29.017 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:30:29.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:29.020 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:30:29.121 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:30:29.156 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:29.197 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:30:29.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:29.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:30:29.307 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:30:29.336 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:29.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:29 smithi179 ceph-mon[43975]: pgmap v623: 17 pgs: 17 active+clean; 3.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 111 MiB/s wr, 1.78k op/s 2024-02-09T17:30:29.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:29 smithi099 ceph-mon[39420]: pgmap v623: 17 pgs: 17 active+clean; 3.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 111 MiB/s wr, 1.78k op/s 2024-02-09T17:30:29.538 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:29 smithi162 ceph-mon[44304]: pgmap v623: 17 pgs: 17 active+clean; 3.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 111 MiB/s wr, 1.78k op/s 2024-02-09T17:30:29.539 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:30:29.542 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:29.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:30:29.643 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:30:29.671 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:29.770 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:30:29.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:29.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:30:29.875 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:30:29.921 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:29.940 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 24 16 57278 57262 149.108 47.625 0.00534889 0.00669822 2024-02-09T17:30:30.132 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:30:30.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:30.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:30:30.155 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:30:30.155 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:30:30.237 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:30:30.267 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:30.351 INFO:teuthology.orchestra.run.smithi099.stderr:2024-02-09T17:30:30.350+0000 7ffb2a7fc700 0 --2- 172.21.15.99:0/2394989195 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb2c100f90 0x7ffb2c19aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-02-09T17:30:30.467 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:30:30.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:30.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:30:30.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-02-09T17:30:30.551+0000 7ffb2b7fe700 0 --2- 172.21.15.99:0/2394989195 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb14008b90 0x7ffb1400b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-02-09T17:30:30.573 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:30:30.624 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:30.847 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:30:30.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:30.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:30:30.940 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 25 16 58581 58565 146.401 81.4375 0.010326 0.00682516 2024-02-09T17:30:30.955 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:30:30.961 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-09T17:30:30.994 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:31.219 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:31 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:30:31.219 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:31 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:30:31.219 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:31 smithi099 ceph-mon[39420]: osdmap e212: 12 total, 12 up, 12 in 2024-02-09T17:30:31.220 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:31 smithi099 ceph-mon[39420]: pgmap v625: 17 pgs: 17 active+clean; 3.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 118 MiB/s wr, 1.88k op/s 2024-02-09T17:30:31.220 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:31 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2394989195' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:30:31.223 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is unset 2024-02-09T17:30:31.323 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:30:31.616 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:31 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:30:31.616 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:31 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:30:31.616 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:31 smithi162 ceph-mon[44304]: osdmap e212: 12 total, 12 up, 12 in 2024-02-09T17:30:31.616 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:31 smithi162 ceph-mon[44304]: pgmap v625: 17 pgs: 17 active+clean; 3.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 118 MiB/s wr, 1.88k op/s 2024-02-09T17:30:31.617 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:31 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2394989195' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:30:31.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:31 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:30:31.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:31 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:30:31.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:31 smithi179 ceph-mon[43975]: osdmap e212: 12 total, 12 up, 12 in 2024-02-09T17:30:31.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:31 smithi179 ceph-mon[43975]: pgmap v625: 17 pgs: 17 active+clean; 3.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 118 MiB/s wr, 1.88k op/s 2024-02-09T17:30:31.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:31 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2394989195' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:30:31.940 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 26 16 60130 60114 144.493 96.8125 0.0118754 0.00691615 2024-02-09T17:30:32.179 INFO:teuthology.orchestra.run.smithi099.stderr:no valid command found; 10 closest matches: 2024-02-09T17:30:32.179 INFO:teuthology.orchestra.run.smithi099.stderr:0 2024-02-09T17:30:32.179 INFO:teuthology.orchestra.run.smithi099.stderr:1 2024-02-09T17:30:32.179 INFO:teuthology.orchestra.run.smithi099.stderr:2 2024-02-09T17:30:32.179 INFO:teuthology.orchestra.run.smithi099.stderr:abort 2024-02-09T17:30:32.179 INFO:teuthology.orchestra.run.smithi099.stderr:assert 2024-02-09T17:30:32.179 INFO:teuthology.orchestra.run.smithi099.stderr:bench [] [] [] [] 2024-02-09T17:30:32.179 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs debug_inject_read_zeros 2024-02-09T17:30:32.179 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs files list 2024-02-09T17:30:32.180 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs stats 2024-02-09T17:30:32.180 INFO:teuthology.orchestra.run.smithi099.stderr:bluestore allocator dump block 2024-02-09T17:30:32.180 INFO:teuthology.orchestra.run.smithi099.stderr:Error EINVAL: invalid command 2024-02-09T17:30:32.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:32 smithi099 ceph-mon[39420]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:30:32.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:32 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2394989195' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:30:32.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:32 smithi099 ceph-mon[39420]: osdmap e213: 12 total, 12 up, 12 in 2024-02-09T17:30:32.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:32 smithi179 ceph-mon[43975]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:30:32.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:32 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2394989195' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:30:32.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:32 smithi179 ceph-mon[43975]: osdmap e213: 12 total, 12 up, 12 in 2024-02-09T17:30:32.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:32 smithi162 ceph-mon[44304]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:30:32.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:32 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2394989195' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:30:32.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:32 smithi162 ceph-mon[44304]: osdmap e213: 12 total, 12 up, 12 in 2024-02-09T17:30:32.940 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 27 16 61879 61863 143.19 109.312 0.00767567 0.00698021 2024-02-09T17:30:33.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:33 smithi099 ceph-mon[39420]: pgmap v627: 17 pgs: 17 active+clean; 3.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 101 MiB/s wr, 1.61k op/s 2024-02-09T17:30:33.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:33 smithi179 ceph-mon[43975]: pgmap v627: 17 pgs: 17 active+clean; 3.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 101 MiB/s wr, 1.61k op/s 2024-02-09T17:30:33.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:33 smithi162 ceph-mon[44304]: pgmap v627: 17 pgs: 17 active+clean; 3.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 101 MiB/s wr, 1.61k op/s 2024-02-09T17:30:33.941 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 28 16 63439 63423 141.558 97.5 0.00460567 0.00705989 2024-02-09T17:30:34.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:34 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4176111340' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:30:34.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:34 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4176111340' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:30:34.940 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 29 16 65024 65008 140.092 99.0625 0.00272198 0.00712923 2024-02-09T17:30:34.942 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:34 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4176111340' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:30:35.425 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:30:35.425 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:30:35.425 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:30:35.425 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:30:35.425 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:30:35.425 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:30:35.425 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:30:35.425 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:30:35.425 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:30:35.425 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:30:35.425 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:30:35.425 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:30:35.426 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:30:35.426 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:30:35.426 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:30:35.426 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:30:35.426 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:30:35.426 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:30:35.426 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:30:35.426 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:30:35.426 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:30:35.426 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:30:35.426 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:30:35.426 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:30:35.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:35 smithi179 ceph-mon[43975]: 1.0 scrub starts 2024-02-09T17:30:35.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:35 smithi179 ceph-mon[43975]: 1.0 scrub ok 2024-02-09T17:30:35.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:35 smithi179 ceph-mon[43975]: pgmap v628: 17 pgs: 17 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 98 MiB/s wr, 1.56k op/s 2024-02-09T17:30:35.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:35 smithi162 ceph-mon[44304]: 1.0 scrub starts 2024-02-09T17:30:35.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:35 smithi162 ceph-mon[44304]: 1.0 scrub ok 2024-02-09T17:30:35.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:35 smithi162 ceph-mon[44304]: pgmap v628: 17 pgs: 17 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 98 MiB/s wr, 1.56k op/s 2024-02-09T17:30:35.940 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 30 16 66324 66308 138.13 81.25 0.00892486 0.00723575 2024-02-09T17:30:35.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:35 smithi099 ceph-mon[39420]: 1.0 scrub starts 2024-02-09T17:30:35.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:35 smithi099 ceph-mon[39420]: 1.0 scrub ok 2024-02-09T17:30:35.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:35 smithi099 ceph-mon[39420]: pgmap v628: 17 pgs: 17 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 98 MiB/s wr, 1.56k op/s 2024-02-09T17:30:36.462 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:30:36.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:36.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:30:36.570 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:30:36.617 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:36.940 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 31 16 67943 67927 136.938 101.188 0.00637091 0.00729866 2024-02-09T17:30:37.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:37 smithi099 ceph-mon[39420]: osdmap e214: 12 total, 12 up, 12 in 2024-02-09T17:30:37.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:37 smithi099 ceph-mon[39420]: pgmap v630: 17 pgs: 17 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.46k op/s 2024-02-09T17:30:37.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:37 smithi179 ceph-mon[43975]: osdmap e214: 12 total, 12 up, 12 in 2024-02-09T17:30:37.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:37 smithi179 ceph-mon[43975]: pgmap v630: 17 pgs: 17 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.46k op/s 2024-02-09T17:30:37.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:37 smithi162 ceph-mon[44304]: osdmap e214: 12 total, 12 up, 12 in 2024-02-09T17:30:37.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:37 smithi162 ceph-mon[44304]: pgmap v630: 17 pgs: 17 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.46k op/s 2024-02-09T17:30:37.784 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-09T17:30:37.787 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-02-09T17:30:37.788 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph tell osd.8 'trim stale osdmaps' 2024-02-09T17:30:37.941 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 32 16 69472 69456 135.645 95.5625 0.00473941 0.00736847 2024-02-09T17:30:38.090 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:30:38.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:38.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:30:38.128 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:30:38.195 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:30:38.224 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:38.421 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:30:38.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:38.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:30:38.525 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:30:38.557 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:38.941 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 33 16 69970 69954 132.478 31.125 0.0097226 0.00754446 2024-02-09T17:30:39.018 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-09T17:30:39.476 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:30:39.649 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:30:39.868 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:39 smithi099 ceph-mon[39420]: pgmap v631: 17 pgs: 17 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-02-09T17:30:39.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:39 smithi179 ceph-mon[43975]: pgmap v631: 17 pgs: 17 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-02-09T17:30:39.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:39 smithi162 ceph-mon[44304]: pgmap v631: 17 pgs: 17 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-02-09T17:30:39.941 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 34 16 71153 71137 130.756 73.9375 0.00993098 0.00764255 2024-02-09T17:30:40.424 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:30:40.429 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:40.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:30:40.531 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:30:40.588 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:40.803 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:30:40.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:40.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:30:40.874 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:30:40.874 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-02-09T17:30:40.874 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.874 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:72520 3.c 3:364bf2ce:::benchmark_data_smithi179_90440_object72519:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-02-09T17:30:40.874 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:30:40.857852+0000", 2024-02-09T17:30:40.874 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.012527753000000001, 2024-02-09T17:30:40.874 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012564163, 2024-02-09T17:30:40.874 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:30:40.874 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-02-09T17:30:40.874 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:30:40.875 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:30:40.875 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:30:40.875 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 72520 2024-02-09T17:30:40.875 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.875 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:30:40.875 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.875 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:30:40.875 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.857852+0000", 2024-02-09T17:30:40.875 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:30:40.875 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.875 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.875 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:30:40.875 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.857852+0000", 2024-02-09T17:30:40.876 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:30:40.876 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.876 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.876 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:30:40.876 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.857853+0000", 2024-02-09T17:30:40.876 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0389999999999999e-06 2024-02-09T17:30:40.876 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.876 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.876 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:30:40.876 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.857934+0000", 2024-02-09T17:30:40.876 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1464000000000005e-05 2024-02-09T17:30:40.876 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.876 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.876 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:30:40.877 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.857948+0000", 2024-02-09T17:30:40.877 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3791000000000001e-05 2024-02-09T17:30:40.877 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.877 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.877 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:30:40.877 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.857955+0000", 2024-02-09T17:30:40.877 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4939999999999997e-06 2024-02-09T17:30:40.877 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.877 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.877 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:30:40.877 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.861203+0000", 2024-02-09T17:30:40.877 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0032475009999999999 2024-02-09T17:30:40.877 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.877 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.877 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:30:40.878 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.861320+0000", 2024-02-09T17:30:40.878 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000117361 2024-02-09T17:30:40.878 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.878 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.878 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,6", 2024-02-09T17:30:40.878 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.861443+0000", 2024-02-09T17:30:40.878 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012318600000000001 2024-02-09T17:30:40.878 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.878 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.878 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:30:40.878 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.870000+0000", 2024-02-09T17:30:40.878 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0085570669999999998 2024-02-09T17:30:40.878 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:30:40.878 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:30:40.879 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:30:40.879 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.879 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.879 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:72524 3.c 3:3180c95a:::benchmark_data_smithi179_90440_object72523:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-02-09T17:30:40.879 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:30:40.864681+0000", 2024-02-09T17:30:40.879 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0056987019999999999, 2024-02-09T17:30:40.879 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0058381020000000004, 2024-02-09T17:30:40.879 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:30:40.879 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-02-09T17:30:40.879 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:30:40.879 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:30:40.879 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:30:40.879 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 72524 2024-02-09T17:30:40.879 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.879 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:30:40.880 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.880 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:30:40.880 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.864681+0000", 2024-02-09T17:30:40.880 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:30:40.880 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.880 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.880 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:30:40.880 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.864681+0000", 2024-02-09T17:30:40.880 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:30:40.880 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.880 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.880 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:30:40.880 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.864683+0000", 2024-02-09T17:30:40.881 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9850000000000001e-06 2024-02-09T17:30:40.881 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.881 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.881 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:30:40.881 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.864721+0000", 2024-02-09T17:30:40.881 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8655000000000002e-05 2024-02-09T17:30:40.881 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.881 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.881 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:30:40.881 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.864734+0000", 2024-02-09T17:30:40.881 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2724e-05 2024-02-09T17:30:40.881 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.881 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.881 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:30:40.881 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.864743+0000", 2024-02-09T17:30:40.882 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6640000000000004e-06 2024-02-09T17:30:40.882 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.882 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.882 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:30:40.882 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.865038+0000", 2024-02-09T17:30:40.882 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00029583799999999998 2024-02-09T17:30:40.882 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.882 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.882 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:30:40.882 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.865150+0000", 2024-02-09T17:30:40.882 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000111327 2024-02-09T17:30:40.882 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.882 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.882 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,6", 2024-02-09T17:30:40.882 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.865286+0000", 2024-02-09T17:30:40.882 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000135809 2024-02-09T17:30:40.883 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.883 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.883 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:30:40.883 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.870009+0000", 2024-02-09T17:30:40.883 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.004723372 2024-02-09T17:30:40.883 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:30:40.883 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:30:40.883 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:30:40.883 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.883 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.883 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:72528 3.c 3:30ed780a:::benchmark_data_smithi179_90440_object72527:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-02-09T17:30:40.883 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:30:40.866710+0000", 2024-02-09T17:30:40.883 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0036693260000000001, 2024-02-09T17:30:40.883 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0039001029999999998, 2024-02-09T17:30:40.883 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:30:40.884 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-02-09T17:30:40.884 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:30:40.884 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:30:40.884 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:30:40.884 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 72528 2024-02-09T17:30:40.884 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.884 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:30:40.884 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.884 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:30:40.884 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.866710+0000", 2024-02-09T17:30:40.884 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:30:40.884 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.884 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.885 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:30:40.885 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.866710+0000", 2024-02-09T17:30:40.885 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:30:40.885 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.885 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.885 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:30:40.885 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.866711+0000", 2024-02-09T17:30:40.885 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3939999999999999e-06 2024-02-09T17:30:40.885 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.885 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.885 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:30:40.885 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.866770+0000", 2024-02-09T17:30:40.885 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8164000000000001e-05 2024-02-09T17:30:40.885 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.885 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.886 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:30:40.886 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.866782+0000", 2024-02-09T17:30:40.886 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2449999999999999e-05 2024-02-09T17:30:40.886 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.886 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.886 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:30:40.886 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.866792+0000", 2024-02-09T17:30:40.886 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8770000000000005e-06 2024-02-09T17:30:40.886 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.886 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.886 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:30:40.886 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.867646+0000", 2024-02-09T17:30:40.886 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00085402700000000004 2024-02-09T17:30:40.886 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.886 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.886 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:30:40.887 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.867775+0000", 2024-02-09T17:30:40.887 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000128792 2024-02-09T17:30:40.887 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.887 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.887 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,6", 2024-02-09T17:30:40.887 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.867927+0000", 2024-02-09T17:30:40.887 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015238799999999999 2024-02-09T17:30:40.887 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:30:40.887 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:30:40.887 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:30:40.887 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.887 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.887 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.24817.0:72516 3.c e214/199)", 2024-02-09T17:30:40.887 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:30:40.869637+0000", 2024-02-09T17:30:40.888 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.00074190200000000001, 2024-02-09T17:30:40.888 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001040911, 2024-02-09T17:30:40.888 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:30:40.888 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "queued for pg", 2024-02-09T17:30:40.888 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:30:40.888 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.888 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:30:40.888 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.869637+0000", 2024-02-09T17:30:40.888 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:30:40.888 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.888 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.888 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:30:40.888 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.869637+0000", 2024-02-09T17:30:40.888 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:30:40.889 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.889 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.889 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:30:40.889 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.869638+0000", 2024-02-09T17:30:40.889 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7000000000000003e-07 2024-02-09T17:30:40.889 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.889 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.889 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:30:40.889 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.869642+0000", 2024-02-09T17:30:40.889 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.16e-06 2024-02-09T17:30:40.889 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.889 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.889 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:30:40.889 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.869654+0000", 2024-02-09T17:30:40.890 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1292e-05 2024-02-09T17:30:40.890 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.890 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.890 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:30:40.890 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.869661+0000", 2024-02-09T17:30:40.890 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6769999999999995e-06 2024-02-09T17:30:40.890 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:30:40.890 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:30:40.890 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:30:40.890 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.890 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.890 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24817.0:72531 3.d e214/199 3:ba5ce56a:::benchmark_data_smithi179_90440_object72530:head v 214'4540, mlcod=214'4538)", 2024-02-09T17:30:40.890 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:30:40.870205+0000", 2024-02-09T17:30:40.890 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.00017432799999999999, 2024-02-09T17:30:40.890 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00052039299999999996, 2024-02-09T17:30:40.891 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:30:40.891 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-09T17:30:40.891 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:30:40.891 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.891 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:30:40.891 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.870205+0000", 2024-02-09T17:30:40.891 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:30:40.891 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.891 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.891 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:30:40.891 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.870205+0000", 2024-02-09T17:30:40.891 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:30:40.891 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.891 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.891 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:30:40.891 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.870206+0000", 2024-02-09T17:30:40.892 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5000000000000001e-07 2024-02-09T17:30:40.892 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.892 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.892 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:30:40.892 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.870234+0000", 2024-02-09T17:30:40.892 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7974e-05 2024-02-09T17:30:40.892 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.892 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.892 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:30:40.892 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.870246+0000", 2024-02-09T17:30:40.892 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2339e-05 2024-02-09T17:30:40.892 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.892 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.892 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:30:40.892 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.870253+0000", 2024-02-09T17:30:40.893 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7360000000000004e-06 2024-02-09T17:30:40.893 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.893 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.893 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:30:40.893 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.870321+0000", 2024-02-09T17:30:40.893 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8151999999999994e-05 2024-02-09T17:30:40.893 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.893 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.893 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:30:40.893 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.870342+0000", 2024-02-09T17:30:40.893 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1197999999999999e-05 2024-02-09T17:30:40.893 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:30:40.893 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:30:40.893 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:30:40.893 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.893 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.894 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:72511 3.c 3:35004941:::benchmark_data_smithi179_90440_object72510:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-02-09T17:30:40.894 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:30:40.853323+0000", 2024-02-09T17:30:40.894 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.017056128, 2024-02-09T17:30:40.894 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.017464066, 2024-02-09T17:30:40.894 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:30:40.894 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-02-09T17:30:40.894 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:30:40.894 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:30:40.894 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:30:40.894 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 72511 2024-02-09T17:30:40.894 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.894 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:30:40.894 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.894 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:30:40.894 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.853323+0000", 2024-02-09T17:30:40.895 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:30:40.895 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.895 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.895 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:30:40.895 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.853323+0000", 2024-02-09T17:30:40.895 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:30:40.895 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.895 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.895 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:30:40.895 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.853325+0000", 2024-02-09T17:30:40.895 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9640000000000002e-06 2024-02-09T17:30:40.895 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.895 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.895 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:30:40.895 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.853505+0000", 2024-02-09T17:30:40.895 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017996599999999999 2024-02-09T17:30:40.896 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.896 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.896 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:30:40.896 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.853518+0000", 2024-02-09T17:30:40.896 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3040000000000001e-05 2024-02-09T17:30:40.896 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.896 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.896 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:30:40.896 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.853525+0000", 2024-02-09T17:30:40.896 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2200000000000003e-06 2024-02-09T17:30:40.896 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.896 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.896 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:30:40.896 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.853571+0000", 2024-02-09T17:30:40.896 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6037e-05 2024-02-09T17:30:40.896 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.897 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.897 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:30:40.897 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.853692+0000", 2024-02-09T17:30:40.897 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012057699999999999 2024-02-09T17:30:40.897 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.897 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.897 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,6", 2024-02-09T17:30:40.897 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.853833+0000", 2024-02-09T17:30:40.897 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014121899999999999 2024-02-09T17:30:40.897 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.897 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.897 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:30:40.897 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.858698+0000", 2024-02-09T17:30:40.897 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0048650450000000001 2024-02-09T17:30:40.897 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.897 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.897 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:30:40.898 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.858711+0000", 2024-02-09T17:30:40.898 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3234999999999999e-05 2024-02-09T17:30:40.898 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:30:40.898 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:30:40.898 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:30:40.898 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.898 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.898 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:72514 3.c 3:3e7b2a95:::benchmark_data_smithi179_90440_object72513:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-02-09T17:30:40.898 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:30:40.854321+0000", 2024-02-09T17:30:40.898 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.016058524000000001, 2024-02-09T17:30:40.898 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.016549899, 2024-02-09T17:30:40.898 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:30:40.898 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-02-09T17:30:40.898 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:30:40.898 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:30:40.899 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:30:40.899 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 72514 2024-02-09T17:30:40.899 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.899 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:30:40.899 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.899 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:30:40.899 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.854321+0000", 2024-02-09T17:30:40.899 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:30:40.899 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.899 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.899 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:30:40.899 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.854321+0000", 2024-02-09T17:30:40.899 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:30:40.899 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.899 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.900 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:30:40.900 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.854322+0000", 2024-02-09T17:30:40.900 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2589999999999999e-06 2024-02-09T17:30:40.900 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.900 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.900 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:30:40.900 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.854423+0000", 2024-02-09T17:30:40.900 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010104500000000001 2024-02-09T17:30:40.900 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.900 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.900 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:30:40.900 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.854435+0000", 2024-02-09T17:30:40.900 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1745e-05 2024-02-09T17:30:40.900 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.901 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.901 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:30:40.901 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.854443+0000", 2024-02-09T17:30:40.901 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.456e-06 2024-02-09T17:30:40.901 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.901 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.901 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:30:40.901 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.856199+0000", 2024-02-09T17:30:40.901 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017557110000000001 2024-02-09T17:30:40.901 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.901 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.901 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:30:40.901 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.856323+0000", 2024-02-09T17:30:40.901 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012380599999999999 2024-02-09T17:30:40.901 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.901 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.902 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,6", 2024-02-09T17:30:40.902 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.856463+0000", 2024-02-09T17:30:40.902 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013985700000000001 2024-02-09T17:30:40.902 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.902 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.902 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:30:40.902 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.863601+0000", 2024-02-09T17:30:40.902 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0071387459999999996 2024-02-09T17:30:40.902 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.902 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.902 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:30:40.902 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.865008+0000", 2024-02-09T17:30:40.902 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001406249 2024-02-09T17:30:40.902 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:30:40.903 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:30:40.903 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:30:40.903 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.903 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.903 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:72516 3.c 3:313309e9:::benchmark_data_smithi179_90440_object72515:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-02-09T17:30:40.903 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:30:40.857239+0000", 2024-02-09T17:30:40.903 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.013140429, 2024-02-09T17:30:40.903 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013717471, 2024-02-09T17:30:40.903 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:30:40.903 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-02-09T17:30:40.903 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:30:40.903 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:30:40.903 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:30:40.903 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 72516 2024-02-09T17:30:40.904 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.904 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:30:40.904 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.904 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:30:40.904 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.857239+0000", 2024-02-09T17:30:40.904 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:30:40.904 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.904 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.904 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:30:40.904 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.857239+0000", 2024-02-09T17:30:40.904 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:30:40.904 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.904 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.904 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:30:40.904 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.857240+0000", 2024-02-09T17:30:40.905 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4529999999999999e-06 2024-02-09T17:30:40.905 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.905 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.905 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:30:40.905 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.857301+0000", 2024-02-09T17:30:40.905 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0809e-05 2024-02-09T17:30:40.905 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.905 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.905 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:30:40.905 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.857319+0000", 2024-02-09T17:30:40.905 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7416000000000001e-05 2024-02-09T17:30:40.905 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.905 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.905 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:30:40.905 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.857327+0000", 2024-02-09T17:30:40.906 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.977e-06 2024-02-09T17:30:40.906 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.906 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.906 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:30:40.906 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.858785+0000", 2024-02-09T17:30:40.906 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014587039999999999 2024-02-09T17:30:40.906 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.906 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.906 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:30:40.906 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.858911+0000", 2024-02-09T17:30:40.906 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012616999999999999 2024-02-09T17:30:40.906 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.906 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.906 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,6", 2024-02-09T17:30:40.906 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.859051+0000", 2024-02-09T17:30:40.906 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000139684 2024-02-09T17:30:40.906 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:30:40.907 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:30:40.907 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:30:40.907 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:40.863608+0000", 2024-02-09T17:30:40.907 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0045570949999999997 2024-02-09T17:30:40.907 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:30:40.907 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:30:40.907 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:30:40.907 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:30:40.907 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-09T17:30:40.907 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 8 2024-02-09T17:30:40.907 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:30:40.915 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:30:40.941 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 35 16 72634 72618 129.664 92.5625 0.0115649 0.00770773 2024-02-09T17:30:40.966 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:41.172 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:30:41.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:41.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:30:41.276 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:30:41.291 INFO:teuthology.orchestra.run.smithi099.stderr:no valid command found; 10 closest matches: 2024-02-09T17:30:41.291 INFO:teuthology.orchestra.run.smithi099.stderr:0 2024-02-09T17:30:41.291 INFO:teuthology.orchestra.run.smithi099.stderr:1 2024-02-09T17:30:41.291 INFO:teuthology.orchestra.run.smithi099.stderr:2 2024-02-09T17:30:41.291 INFO:teuthology.orchestra.run.smithi099.stderr:abort 2024-02-09T17:30:41.291 INFO:teuthology.orchestra.run.smithi099.stderr:assert 2024-02-09T17:30:41.291 INFO:teuthology.orchestra.run.smithi099.stderr:bench [] [] [] [] 2024-02-09T17:30:41.291 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs debug_inject_read_zeros 2024-02-09T17:30:41.291 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs files list 2024-02-09T17:30:41.292 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs stats 2024-02-09T17:30:41.292 INFO:teuthology.orchestra.run.smithi099.stderr:bluestore allocator dump block 2024-02-09T17:30:41.292 INFO:teuthology.orchestra.run.smithi099.stderr:Error EINVAL: invalid command 2024-02-09T17:30:41.328 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:41.669 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-09T17:30:41.694 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:30:41.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:41.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:30:41.803 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:30:41.834 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:41.862 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:30:41.865 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:41 smithi179 ceph-mon[43975]: pgmap v632: 17 pgs: 17 active+clean; 4.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.53k op/s 2024-02-09T17:30:41.865 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:41 smithi179 ceph-mon[43975]: osdmap e215: 12 total, 12 up, 12 in 2024-02-09T17:30:41.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:41.867 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:30:41.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:41 smithi162 ceph-mon[44304]: pgmap v632: 17 pgs: 17 active+clean; 4.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.53k op/s 2024-02-09T17:30:41.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:41 smithi162 ceph-mon[44304]: osdmap e215: 12 total, 12 up, 12 in 2024-02-09T17:30:41.941 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 36 16 74102 74086 128.611 91.75 0.0118373 0.00777142 2024-02-09T17:30:41.961 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:30:41.963 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:41 smithi099 ceph-mon[39420]: pgmap v632: 17 pgs: 17 active+clean; 4.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.53k op/s 2024-02-09T17:30:41.963 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:41 smithi099 ceph-mon[39420]: osdmap e215: 12 total, 12 up, 12 in 2024-02-09T17:30:41.968 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:30:41.999 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:42.010 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-09T17:30:42.011 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-02-09T17:30:42.011 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph tell osd.9 'trim stale osdmaps' 2024-02-09T17:30:42.025 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:30:42.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:42.029 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:30:42.065 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:30:42.130 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:30:42.168 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:42.196 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:30:42.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:42.201 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:30:42.302 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:30:42.361 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:42.941 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 37 16 75470 75454 127.446 85.5 0.00330651 0.00782115 2024-02-09T17:30:43.014 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:30:43.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:43.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:30:43.118 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:30:43.173 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:43.292 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:30:43.534 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:30:43.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:43.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:30:43.643 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:30:43.692 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:43.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:43 smithi099 ceph-mon[39420]: pgmap v634: 17 pgs: 17 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-02-09T17:30:43.759 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:30:43.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:43.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:30:43.866 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:30:43.897 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:43 smithi162 ceph-mon[44304]: pgmap v634: 17 pgs: 17 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-02-09T17:30:43.900 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:43.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:43 smithi179 ceph-mon[43975]: pgmap v634: 17 pgs: 17 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-02-09T17:30:43.941 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 38 16 76818 76802 126.309 84.25 0.00892186 0.0079124 2024-02-09T17:30:44.055 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:30:44.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:44.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:30:44.160 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:30:44.228 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:44.241 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:30:44.241 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-09T17:30:44.241 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-02-09T17:30:44.241 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-02-09T17:30:44.241 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:30:44.265 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:30:44.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:44.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:30:44.369 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:30:44.424 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:44.509 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:30:44.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:44.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:30:44.603 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:44 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1229688111' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:30:44.614 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:30:44.615 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is unset 2024-02-09T17:30:44.646 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:44.668 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:30:44.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:44.678 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:30:44.778 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:30:44.828 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:44.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:44 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1229688111' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:30:44.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:44 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1229688111' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:30:44.943 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 39 16 78215 78199 125.309 87.3125 0.0102557 0.00797555 2024-02-09T17:30:44.943 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:30:44.944 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:30:44.944 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:30:44.944 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:30:44.944 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:30:44.944 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:30:44.944 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:30:44.947 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:30:44.957 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:30:44.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:44.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:30:44.978 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:30:44.979 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-09T17:30:45.006 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:30:45.040 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:30:45.064 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:30:45.065 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:30:45.076 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:30:45.076 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:30:45.076 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:30:45.076 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:30:45.076 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:30:45.076 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:30:45.076 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:30:45.077 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:30:45.077 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:30:45.077 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:30:45.077 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:30:45.077 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:30:45.116 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:45.138 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:30:45.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:45.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:30:45.248 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:30:45.292 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:45.421 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:30:45.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:45.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:30:45.430 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:30:45.525 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:30:45.565 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:45.596 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:30:45.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:45.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:30:45.700 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:30:45.732 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:45.756 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:30:45.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:45.760 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:30:45.861 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:30:45.893 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:45 smithi179 ceph-mon[43975]: pgmap v635: 17 pgs: 17 active+clean; 4.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.53k op/s 2024-02-09T17:30:45.894 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:45 smithi179 ceph-mon[43975]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:30:45.894 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:45 smithi179 ceph-mon[43975]: Cluster is now healthy 2024-02-09T17:30:45.894 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:45 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1229688111' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:30:45.894 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:45 smithi179 ceph-mon[43975]: osdmap e216: 12 total, 12 up, 12 in 2024-02-09T17:30:45.894 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:45 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2079301924' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:30:45.897 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:45.920 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:30:45.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:45.925 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:30:45.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:45 smithi162 ceph-mon[44304]: pgmap v635: 17 pgs: 17 active+clean; 4.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.53k op/s 2024-02-09T17:30:45.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:45 smithi162 ceph-mon[44304]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:30:45.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:45 smithi162 ceph-mon[44304]: Cluster is now healthy 2024-02-09T17:30:45.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:45 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1229688111' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:30:45.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:45 smithi162 ceph-mon[44304]: osdmap e216: 12 total, 12 up, 12 in 2024-02-09T17:30:45.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:45 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2079301924' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:30:45.941 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:2024-02-09T17:30:45.941298+0000 min lat: 0.00110138 max lat: 0.611297 avg lat: 0.0080386 2024-02-09T17:30:45.941 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-09T17:30:45.941 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 40 16 79593 79577 124.329 86.125 0.0141307 0.0080386 2024-02-09T17:30:45.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:45 smithi099 ceph-mon[39420]: pgmap v635: 17 pgs: 17 active+clean; 4.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.53k op/s 2024-02-09T17:30:45.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:45 smithi099 ceph-mon[39420]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:30:45.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:45 smithi099 ceph-mon[39420]: Cluster is now healthy 2024-02-09T17:30:45.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:45 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1229688111' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:30:45.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:45 smithi099 ceph-mon[39420]: osdmap e216: 12 total, 12 up, 12 in 2024-02-09T17:30:45.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:45 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2079301924' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:30:46.026 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:30:46.062 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:46.095 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:30:46.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:46.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:30:46.200 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:30:46.247 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:46.304 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:30:46.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:46.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:30:46.409 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:30:46.450 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:46.478 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:30:46.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:46.483 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:30:46.584 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:30:46.617 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:46.639 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:30:46.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:46.644 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:30:46.745 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:30:46.776 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:46.803 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:30:46.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:46.807 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:30:46.908 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:30:46.941 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 41 16 81049 81033 123.516 91 0.00938802 0.00809241 2024-02-09T17:30:46.946 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:47.221 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:47 smithi099 ceph-mon[39420]: osdmap e217: 12 total, 12 up, 12 in 2024-02-09T17:30:47.222 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:47 smithi099 ceph-mon[39420]: pgmap v638: 17 pgs: 17 active+clean; 4.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.32k op/s 2024-02-09T17:30:47.576 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:30:47.576 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:30:47.606 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:47 smithi162 ceph-mon[44304]: osdmap e217: 12 total, 12 up, 12 in 2024-02-09T17:30:47.606 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:47 smithi162 ceph-mon[44304]: pgmap v638: 17 pgs: 17 active+clean; 4.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.32k op/s 2024-02-09T17:30:47.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:47 smithi179 ceph-mon[43975]: osdmap e217: 12 total, 12 up, 12 in 2024-02-09T17:30:47.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:47 smithi179 ceph-mon[43975]: pgmap v638: 17 pgs: 17 active+clean; 4.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.32k op/s 2024-02-09T17:30:47.941 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 42 16 82255 82239 122.369 75.375 0.00695156 0.00816759 2024-02-09T17:30:48.103 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set noscrub 2024-02-09T17:30:48.232 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:48 smithi162 ceph-mon[44304]: 3.4 scrub starts 2024-02-09T17:30:48.342 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-09T17:30:48.498 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:48 smithi099 ceph-mon[39420]: 3.4 scrub starts 2024-02-09T17:30:48.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:48 smithi179 ceph-mon[43975]: 3.4 scrub starts 2024-02-09T17:30:48.687 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:30:48.764 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:30:48.941 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 43 16 83426 83410 121.226 73.1875 0.0196539 0.00824537 2024-02-09T17:30:49.321 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:49 smithi162 ceph-mon[44304]: pgmap v639: 17 pgs: 17 active+clean; 4.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-02-09T17:30:49.321 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:49 smithi162 ceph-mon[44304]: 3.4 scrub ok 2024-02-09T17:30:49.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:49 smithi099 ceph-mon[39420]: pgmap v639: 17 pgs: 17 active+clean; 4.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-02-09T17:30:49.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:49 smithi099 ceph-mon[39420]: 3.4 scrub ok 2024-02-09T17:30:49.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:49 smithi179 ceph-mon[43975]: pgmap v639: 17 pgs: 17 active+clean; 4.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-02-09T17:30:49.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:49 smithi179 ceph-mon[43975]: 3.4 scrub ok 2024-02-09T17:30:49.942 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 44 16 84646 84630 120.203 76.25 0.0102311 0.00831577 2024-02-09T17:30:50.866 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:30:50.866 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:30:50.942 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 45 16 85932 85916 119.318 80.375 0.0109694 0.00837663 2024-02-09T17:30:51.579 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:51 smithi162 ceph-mon[44304]: pgmap v640: 17 pgs: 17 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.38k op/s 2024-02-09T17:30:51.579 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:51 smithi162 ceph-mon[44304]: osdmap e218: 12 total, 12 up, 12 in 2024-02-09T17:30:51.642 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-09T17:30:51.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:51 smithi179 ceph-mon[43975]: pgmap v640: 17 pgs: 17 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.38k op/s 2024-02-09T17:30:51.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:51 smithi179 ceph-mon[43975]: osdmap e218: 12 total, 12 up, 12 in 2024-02-09T17:30:51.942 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 46 16 87282 87266 118.558 84.375 0.00913082 0.00843039 2024-02-09T17:30:51.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:51 smithi099 ceph-mon[39420]: pgmap v640: 17 pgs: 17 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.38k op/s 2024-02-09T17:30:51.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:51 smithi099 ceph-mon[39420]: osdmap e218: 12 total, 12 up, 12 in 2024-02-09T17:30:52.085 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:30:52.577 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:30:52.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:52.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:30:52.672 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:30:52.679 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:30:52.682 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:30:52.745 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:52.777 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:30:52.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:52.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:30:52.936 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:30:52.942 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 47 16 88580 88564 117.761 81.125 0.00306229 0.00848271 2024-02-09T17:30:52.966 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:53.429 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:30:53.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:53.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:30:53.534 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:30:53.587 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:53.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:53 smithi162 ceph-mon[44304]: pgmap v642: 17 pgs: 17 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.27k op/s 2024-02-09T17:30:53.742 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:30:53.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:53.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:30:53.847 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:30:53.878 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:53 smithi179 ceph-mon[43975]: pgmap v642: 17 pgs: 17 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.27k op/s 2024-02-09T17:30:53.880 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:53.903 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:30:53.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:53.908 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:30:53.942 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 48 16 89830 89814 116.936 78.125 0.00886283 0.0085477 2024-02-09T17:30:53.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:53 smithi099 ceph-mon[39420]: pgmap v642: 17 pgs: 17 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.27k op/s 2024-02-09T17:30:54.010 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:30:54.043 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:54.068 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:30:54.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:54.072 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:30:54.123 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:30:54.123 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:30:54.174 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:30:54.205 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:54.938 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-09T17:30:54.943 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 49 16 91150 91134 116.23 82.5 0.0138279 0.00859896 2024-02-09T17:30:55.263 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:30:55.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:55.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:30:55.287 INFO:teuthology.orchestra.run.smithi099.stderr:no valid command found; 10 closest matches: 2024-02-09T17:30:55.287 INFO:teuthology.orchestra.run.smithi099.stderr:0 2024-02-09T17:30:55.287 INFO:teuthology.orchestra.run.smithi099.stderr:1 2024-02-09T17:30:55.287 INFO:teuthology.orchestra.run.smithi099.stderr:2 2024-02-09T17:30:55.287 INFO:teuthology.orchestra.run.smithi099.stderr:abort 2024-02-09T17:30:55.287 INFO:teuthology.orchestra.run.smithi099.stderr:assert 2024-02-09T17:30:55.287 INFO:teuthology.orchestra.run.smithi099.stderr:bench [] [] [] [] 2024-02-09T17:30:55.287 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs debug_inject_read_zeros 2024-02-09T17:30:55.287 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs files list 2024-02-09T17:30:55.287 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs stats 2024-02-09T17:30:55.287 INFO:teuthology.orchestra.run.smithi099.stderr:bluestore allocator dump block 2024-02-09T17:30:55.287 INFO:teuthology.orchestra.run.smithi099.stderr:Error EINVAL: invalid command 2024-02-09T17:30:55.369 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:30:55.376 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:30:55.442 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:55.664 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:30:55.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:55.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:30:55.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:55 smithi162 ceph-mon[44304]: pgmap v643: 17 pgs: 17 active+clean; 5.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.48k op/s 2024-02-09T17:30:55.768 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:30:55.781 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:55 smithi099 ceph-mon[39420]: pgmap v643: 17 pgs: 17 active+clean; 5.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.48k op/s 2024-02-09T17:30:55.811 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:55.839 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:30:55.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:55.843 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:30:55.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:55 smithi179 ceph-mon[43975]: pgmap v643: 17 pgs: 17 active+clean; 5.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.48k op/s 2024-02-09T17:30:55.943 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 50 16 92343 92327 115.397 74.5625 0.0242122 0.00866126 2024-02-09T17:30:55.944 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:30:56.003 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:56.211 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:30:56.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:56.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:30:56.319 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:30:56.353 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:56.380 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:30:56.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:56.382 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:30:56.483 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:30:56.515 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:56.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:56 smithi179 ceph-mon[43975]: 3.0 scrub starts 2024-02-09T17:30:56.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:56 smithi179 ceph-mon[43975]: osdmap e219: 12 total, 12 up, 12 in 2024-02-09T17:30:56.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:56 smithi162 ceph-mon[44304]: 3.0 scrub starts 2024-02-09T17:30:56.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:56 smithi162 ceph-mon[44304]: osdmap e219: 12 total, 12 up, 12 in 2024-02-09T17:30:56.717 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:30:56.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:56.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:30:56.823 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:30:56.860 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:56.887 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:30:56.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:56.890 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:30:56.943 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 51 15 93434 93419 114.472 68.25 0.00764435 0.0087317 2024-02-09T17:30:56.960 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:56 smithi099 ceph-mon[39420]: 3.0 scrub starts 2024-02-09T17:30:56.961 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:56 smithi099 ceph-mon[39420]: osdmap e219: 12 total, 12 up, 12 in 2024-02-09T17:30:56.991 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:30:57.027 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:57.051 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:30:57.054 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:57.054 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:30:57.155 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:30:57.197 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:57.393 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:30:57.393 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:30:57.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:57 smithi162 ceph-mon[44304]: 3.0 scrub ok 2024-02-09T17:30:57.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:57 smithi162 ceph-mon[44304]: pgmap v645: 17 pgs: 17 active+clean; 5.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.46k op/s 2024-02-09T17:30:57.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:57 smithi099 ceph-mon[39420]: 3.0 scrub ok 2024-02-09T17:30:57.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:57 smithi099 ceph-mon[39420]: pgmap v645: 17 pgs: 17 active+clean; 5.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.46k op/s 2024-02-09T17:30:57.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:57 smithi179 ceph-mon[43975]: 3.0 scrub ok 2024-02-09T17:30:57.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:57 smithi179 ceph-mon[43975]: pgmap v645: 17 pgs: 17 active+clean; 5.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.46k op/s 2024-02-09T17:30:57.943 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 52 16 94694 94678 113.784 78.6875 0.0177307 0.00878491 2024-02-09T17:30:58.060 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:30:58.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:58.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:30:58.168 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:30:58.172 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-09T17:30:58.199 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:58.223 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:30:58.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:58.227 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:30:58.327 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:30:58.363 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:58.385 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:30:58.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:58.389 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:30:58.489 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:30:58.525 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:58.535 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:30:58.551 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:30:58.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:58.555 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:30:58.574 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:30:58.597 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:30:58.607 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:30:58.634 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:30:58.655 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:30:58.665 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:30:58.714 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:58.714 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:30:58.755 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:30:58.793 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:30:58.833 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:30:58.864 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:30:58.871 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:58 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:30:58.871 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:58 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:30:58.871 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:58 smithi162 ceph-mon[44304]: osdmap e220: 12 total, 12 up, 12 in 2024-02-09T17:30:58.871 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:58 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1089944177' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:30:58.899 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:30:58.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:58 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:30:58.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:58 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:30:58.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:58 smithi179 ceph-mon[43975]: osdmap e220: 12 total, 12 up, 12 in 2024-02-09T17:30:58.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:58 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1089944177' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:30:58.938 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:30:58.943 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 53 16 95883 95867 113.039 74.3125 0.0149183 0.00884273 2024-02-09T17:30:58.943 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:30:58.944 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:30:58.944 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:30:58.944 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:30:58.944 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:30:58.944 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:30:58.944 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:30:58.944 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:30:58.944 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:30:58.944 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:30:58.944 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:30:58.944 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:30:58.983 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:58 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:30:58.983 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:58 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:30:58.983 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:58 smithi099 ceph-mon[39420]: osdmap e220: 12 total, 12 up, 12 in 2024-02-09T17:30:58.983 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:58 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1089944177' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:30:59.111 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-09T17:30:59.111 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-02-09T17:30:59.112 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph tell osd.10 'trim stale osdmaps' 2024-02-09T17:30:59.615 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:30:59.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:30:59.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:30:59.721 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:30:59.780 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:30:59.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:59 smithi179 ceph-mon[43975]: pgmap v647: 17 pgs: 17 active+clean; 5.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.19k op/s 2024-02-09T17:30:59.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:30:59 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:30:59.924 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:59 smithi162 ceph-mon[44304]: pgmap v647: 17 pgs: 17 active+clean; 5.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.19k op/s 2024-02-09T17:30:59.925 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:30:59 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:30:59.943 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 54 16 96934 96918 112.162 65.6875 0.0360735 0.00891158 2024-02-09T17:30:59.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:59 smithi099 ceph-mon[39420]: pgmap v647: 17 pgs: 17 active+clean; 5.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.19k op/s 2024-02-09T17:30:59.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:30:59 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:31:00.481 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:31:00.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:00.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:31:00.524 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:31:00.586 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:31:00.636 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:00 smithi162 ceph-mon[44304]: 3.1 scrub starts 2024-02-09T17:31:00.636 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:00 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1544084792' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:31:00.636 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:00 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:31:00.636 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:00 smithi162 ceph-mon[44304]: 3.7 scrub starts 2024-02-09T17:31:00.644 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is set 2024-02-09T17:31:00.661 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:00.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:00 smithi099 ceph-mon[39420]: 3.1 scrub starts 2024-02-09T17:31:00.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:00 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1544084792' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:31:00.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:00 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:31:00.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:00 smithi099 ceph-mon[39420]: 3.7 scrub starts 2024-02-09T17:31:00.764 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:31:00.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:00.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:31:00.863 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:31:00.863 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-02-09T17:31:00.863 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:31:00.863 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24817.0:97878 3.7 e221/199 3:eb6d303a:::benchmark_data_smithi179_90440_object97877:head v 221'6080, mlcod=221'6076)", 2024-02-09T17:31:00.863 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:31:00.844980+0000", 2024-02-09T17:31:00.863 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0011550340000000001, 2024-02-09T17:31:00.863 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001187076, 2024-02-09T17:31:00.863 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:31:00.863 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-09T17:31:00.864 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:31:00.864 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:31:00.864 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:31:00.864 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:00.844980+0000", 2024-02-09T17:31:00.864 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:31:00.864 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:31:00.864 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:31:00.864 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:31:00.864 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:00.844980+0000", 2024-02-09T17:31:00.864 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:31:00.864 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:31:00.865 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:31:00.865 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:31:00.865 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:00.844981+0000", 2024-02-09T17:31:00.865 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3799999999999997e-07 2024-02-09T17:31:00.865 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:31:00.865 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:31:00.865 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:31:00.865 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:00.845008+0000", 2024-02-09T17:31:00.865 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7087999999999999e-05 2024-02-09T17:31:00.865 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:31:00.865 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:31:00.865 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:31:00.865 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:00.845015+0000", 2024-02-09T17:31:00.866 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3889999999999999e-06 2024-02-09T17:31:00.866 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:31:00.866 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:31:00.866 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:31:00.866 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:00.845021+0000", 2024-02-09T17:31:00.866 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9429999999999997e-06 2024-02-09T17:31:00.866 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:31:00.866 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:31:00.866 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:31:00.866 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:00.845070+0000", 2024-02-09T17:31:00.866 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8680000000000001e-05 2024-02-09T17:31:00.866 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:31:00.867 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:31:00.867 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:31:00.867 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:00.845088+0000", 2024-02-09T17:31:00.867 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8215000000000001e-05 2024-02-09T17:31:00.867 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:31:00.867 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:31:00.867 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:31:00.867 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:31:00.867 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:31:00.867 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.24817.0:97872 3.f e221/199 ondisk, result = 0)", 2024-02-09T17:31:00.867 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:31:00.845061+0000", 2024-02-09T17:31:00.867 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0010741800000000001, 2024-02-09T17:31:00.868 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0011855699999999999, 2024-02-09T17:31:00.868 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:31:00.868 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-09T17:31:00.868 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:31:00.868 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:31:00.868 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:31:00.868 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:00.845061+0000", 2024-02-09T17:31:00.868 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:31:00.868 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:31:00.868 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:31:00.868 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:31:00.868 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:00.845061+0000", 2024-02-09T17:31:00.869 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:31:00.869 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:31:00.869 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:31:00.869 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:31:00.869 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:00.845061+0000", 2024-02-09T17:31:00.869 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4000000000000002e-07 2024-02-09T17:31:00.869 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:31:00.869 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:31:00.869 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:31:00.869 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:00.845064+0000", 2024-02-09T17:31:00.869 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7310000000000002e-06 2024-02-09T17:31:00.869 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:31:00.870 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:31:00.870 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:31:00.870 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:00.845071+0000", 2024-02-09T17:31:00.870 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6150000000000002e-06 2024-02-09T17:31:00.870 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:31:00.870 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:31:00.870 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:31:00.870 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:00.845076+0000", 2024-02-09T17:31:00.870 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.079e-06 2024-02-09T17:31:00.870 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:31:00.870 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:31:00.870 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:31:00.871 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:00.845136+0000", 2024-02-09T17:31:00.871 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0075999999999999e-05 2024-02-09T17:31:00.871 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:31:00.871 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:31:00.871 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:31:00.871 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:00.845142+0000", 2024-02-09T17:31:00.871 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6340000000000003e-06 2024-02-09T17:31:00.871 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:31:00.871 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:31:00.871 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:31:00.871 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:31:00.871 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:31:00.871 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24817.0:97864 3.7 e221/199 3:e9d7d3e7:::benchmark_data_smithi179_90440_object97863:head v 221'6078, mlcod=221'6076)", 2024-02-09T17:31:00.872 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:31:00.835245+0000", 2024-02-09T17:31:00.872 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.010889568000000001, 2024-02-09T17:31:00.872 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011068807, 2024-02-09T17:31:00.872 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:31:00.872 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-09T17:31:00.872 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:31:00.872 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:31:00.872 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:31:00.872 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:00.835245+0000", 2024-02-09T17:31:00.872 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:31:00.872 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:31:00.873 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:31:00.873 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:31:00.873 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:00.835245+0000", 2024-02-09T17:31:00.873 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:31:00.873 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:31:00.873 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:31:00.873 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:31:00.873 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:00.835247+0000", 2024-02-09T17:31:00.873 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4449999999999999e-06 2024-02-09T17:31:00.873 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:31:00.873 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:31:00.873 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:31:00.874 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:00.835284+0000", 2024-02-09T17:31:00.874 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6813999999999997e-05 2024-02-09T17:31:00.874 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:31:00.874 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:31:00.874 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:31:00.874 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:00.835297+0000", 2024-02-09T17:31:00.874 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3477e-05 2024-02-09T17:31:00.874 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:31:00.874 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:31:00.874 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:31:00.874 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:00.835304+0000", 2024-02-09T17:31:00.874 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2919999999999998e-06 2024-02-09T17:31:00.874 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:31:00.875 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:31:00.875 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:31:00.875 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:00.835490+0000", 2024-02-09T17:31:00.875 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018553499999999999 2024-02-09T17:31:00.875 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:31:00.875 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:31:00.875 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:31:00.875 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:00.835510+0000", 2024-02-09T17:31:00.875 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9788000000000001e-05 2024-02-09T17:31:00.875 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:31:00.875 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:31:00.875 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:31:00.875 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:31:00.876 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:31:00.876 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24817.0:97865 3.7 e221/199 3:eca940a2:::benchmark_data_smithi179_90440_object97864:head v 221'6079, mlcod=221'6076)", 2024-02-09T17:31:00.876 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:31:00.835515+0000", 2024-02-09T17:31:00.876 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.010619994000000001, 2024-02-09T17:31:00.876 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.01086617, 2024-02-09T17:31:00.876 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:31:00.876 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-09T17:31:00.876 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:31:00.876 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:31:00.876 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:31:00.876 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:00.835515+0000", 2024-02-09T17:31:00.877 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:31:00.877 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:31:00.877 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:31:00.877 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:31:00.877 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:00.835515+0000", 2024-02-09T17:31:00.877 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:31:00.877 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:31:00.877 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:31:00.877 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:31:00.877 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:00.835516+0000", 2024-02-09T17:31:00.877 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8999999999999996e-07 2024-02-09T17:31:00.877 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:31:00.877 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:31:00.878 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:31:00.878 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:00.835567+0000", 2024-02-09T17:31:00.878 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1728999999999999e-05 2024-02-09T17:31:00.878 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:31:00.878 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:31:00.878 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:31:00.878 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:00.835576+0000", 2024-02-09T17:31:00.878 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1500000000000005e-06 2024-02-09T17:31:00.878 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:31:00.878 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:31:00.878 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:31:00.878 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:00.835583+0000", 2024-02-09T17:31:00.878 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.173e-06 2024-02-09T17:31:00.879 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:31:00.879 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:31:00.879 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:31:00.879 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:00.842023+0000", 2024-02-09T17:31:00.879 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0064398850000000002 2024-02-09T17:31:00.879 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:31:00.879 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:31:00.879 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:31:00.879 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:00.842042+0000", 2024-02-09T17:31:00.879 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9415999999999998e-05 2024-02-09T17:31:00.879 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:31:00.879 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:31:00.879 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:31:00.880 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:31:00.880 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:31:00.880 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:97872 3.f 3:f0698518:::benchmark_data_smithi179_90440_object97871:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-02-09T17:31:00.880 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:31:00.835892+0000", 2024-02-09T17:31:00.880 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.010243492, 2024-02-09T17:31:00.880 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010555592000000001, 2024-02-09T17:31:00.880 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:31:00.880 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-02-09T17:31:00.880 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:31:00.880 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:31:00.880 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:31:00.880 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 97872 2024-02-09T17:31:00.881 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:31:00.881 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:31:00.881 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:31:00.881 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:31:00.881 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:00.835892+0000", 2024-02-09T17:31:00.881 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:31:00.881 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:31:00.881 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:31:00.881 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:31:00.881 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:00.835892+0000", 2024-02-09T17:31:00.881 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:31:00.882 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:31:00.882 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:31:00.882 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:31:00.882 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:00.835893+0000", 2024-02-09T17:31:00.882 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.387e-06 2024-02-09T17:31:00.882 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:31:00.882 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:31:00.882 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:31:00.882 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:00.835962+0000", 2024-02-09T17:31:00.882 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9367000000000002e-05 2024-02-09T17:31:00.882 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:31:00.882 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:31:00.883 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:31:00.883 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:00.835978+0000", 2024-02-09T17:31:00.883 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5342999999999999e-05 2024-02-09T17:31:00.883 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:31:00.883 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:31:00.883 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:31:00.883 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:00.835985+0000", 2024-02-09T17:31:00.883 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3989999999999998e-06 2024-02-09T17:31:00.883 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:31:00.883 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:31:00.883 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:31:00.884 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:00.838408+0000", 2024-02-09T17:31:00.884 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0024232260000000001 2024-02-09T17:31:00.884 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:31:00.884 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:31:00.884 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:31:00.884 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:00.838548+0000", 2024-02-09T17:31:00.884 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013954700000000001 2024-02-09T17:31:00.884 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:31:00.884 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:31:00.884 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 8,11", 2024-02-09T17:31:00.884 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:00.838674+0000", 2024-02-09T17:31:00.884 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012637600000000001 2024-02-09T17:31:00.884 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:31:00.885 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:31:00.885 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:00.885 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:00.844953+0000", 2024-02-09T17:31:00.885 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0062790500000000004 2024-02-09T17:31:00.885 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:31:00.885 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:31:00.885 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:00.885 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:00.845148+0000", 2024-02-09T17:31:00.885 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019447099999999999 2024-02-09T17:31:00.885 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:31:00.885 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:31:00.885 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:31:00.885 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:31:00.886 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-09T17:31:00.886 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 5 2024-02-09T17:31:00.886 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:31:00.886 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:31:00.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:00 smithi179 ceph-mon[43975]: 3.1 scrub starts 2024-02-09T17:31:00.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:00 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1544084792' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:31:00.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:00 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:31:00.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:00 smithi179 ceph-mon[43975]: 3.7 scrub starts 2024-02-09T17:31:00.918 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:00.942 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:31:00.943 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 55 16 98004 97988 111.339 66.875 0.0171689 0.00897773 2024-02-09T17:31:00.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:00.948 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:31:01.049 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:31:01.081 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:01.103 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:31:01.106 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:01.106 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:31:01.207 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:31:01.237 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:01.263 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:31:01.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:01.267 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:31:01.368 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:31:01.441 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:01.880 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:01 smithi162 ceph-mon[44304]: 3.9 scrub starts 2024-02-09T17:31:01.881 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:01 smithi162 ceph-mon[44304]: 3.1 scrub ok 2024-02-09T17:31:01.881 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:01 smithi162 ceph-mon[44304]: pgmap v648: 17 pgs: 17 active+clean; 5.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-02-09T17:31:01.881 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:01 smithi162 ceph-mon[44304]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:31:01.881 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:01 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:31:01.881 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:01 smithi162 ceph-mon[44304]: osdmap e221: 12 total, 12 up, 12 in 2024-02-09T17:31:01.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:01 smithi179 ceph-mon[43975]: 3.9 scrub starts 2024-02-09T17:31:01.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:01 smithi179 ceph-mon[43975]: 3.1 scrub ok 2024-02-09T17:31:01.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:01 smithi179 ceph-mon[43975]: pgmap v648: 17 pgs: 17 active+clean; 5.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-02-09T17:31:01.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:01 smithi179 ceph-mon[43975]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:31:01.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:01 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:31:01.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:01 smithi179 ceph-mon[43975]: osdmap e221: 12 total, 12 up, 12 in 2024-02-09T17:31:01.943 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 56 16 99228 99212 110.717 76.5 0.0299651 0.00902747 2024-02-09T17:31:01.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:01 smithi099 ceph-mon[39420]: 3.9 scrub starts 2024-02-09T17:31:01.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:01 smithi099 ceph-mon[39420]: 3.1 scrub ok 2024-02-09T17:31:01.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:01 smithi099 ceph-mon[39420]: pgmap v648: 17 pgs: 17 active+clean; 5.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-02-09T17:31:01.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:01 smithi099 ceph-mon[39420]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:31:01.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:01 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:31:01.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:01 smithi099 ceph-mon[39420]: osdmap e221: 12 total, 12 up, 12 in 2024-02-09T17:31:02.207 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:31:02.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:02.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:31:02.247 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-09T17:31:02.316 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:31:02.348 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:02.507 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:31:02.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:02.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:31:02.517 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:31:02.613 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:31:02.653 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:02.785 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:31:02.913 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:31:02.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:02.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:31:02.943 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 57 16 100301 100285 109.951 67.0625 0.0111416 0.00909047 2024-02-09T17:31:03.016 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:31:03.080 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:03.120 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:31:03.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:03.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:31:03.200 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:31:03.224 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:31:03.255 INFO:teuthology.orchestra.run.smithi099.stderr:no valid command found; 10 closest matches: 2024-02-09T17:31:03.255 INFO:teuthology.orchestra.run.smithi099.stderr:0 2024-02-09T17:31:03.255 INFO:teuthology.orchestra.run.smithi099.stderr:1 2024-02-09T17:31:03.255 INFO:teuthology.orchestra.run.smithi099.stderr:2 2024-02-09T17:31:03.255 INFO:teuthology.orchestra.run.smithi099.stderr:abort 2024-02-09T17:31:03.255 INFO:teuthology.orchestra.run.smithi099.stderr:assert 2024-02-09T17:31:03.255 INFO:teuthology.orchestra.run.smithi099.stderr:bench [] [] [] [] 2024-02-09T17:31:03.255 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs debug_inject_read_zeros 2024-02-09T17:31:03.255 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs files list 2024-02-09T17:31:03.255 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs stats 2024-02-09T17:31:03.255 INFO:teuthology.orchestra.run.smithi099.stderr:bluestore allocator dump block 2024-02-09T17:31:03.255 INFO:teuthology.orchestra.run.smithi099.stderr:Error EINVAL: invalid command 2024-02-09T17:31:03.289 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:03.655 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:31:03.661 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:03 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:31:03.661 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:03 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:31:03.661 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:03 smithi099 ceph-mon[39420]: pgmap v650: 17 pgs: 17 active+clean; 5.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.29k op/s 2024-02-09T17:31:03.661 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:03 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:31:03.661 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:03 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:31:03.661 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:03 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:31:03.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:03.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:31:03.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:03 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:31:03.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:03 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:31:03.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:03 smithi162 ceph-mon[44304]: pgmap v650: 17 pgs: 17 active+clean; 5.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.29k op/s 2024-02-09T17:31:03.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:03 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:31:03.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:03 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:31:03.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:03 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:31:03.763 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:31:03.775 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-09T17:31:03.826 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:03.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:03 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:31:03.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:03 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:31:03.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:03 smithi179 ceph-mon[43975]: pgmap v650: 17 pgs: 17 active+clean; 5.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.29k op/s 2024-02-09T17:31:03.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:03 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:31:03.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:03 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:31:03.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:03 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:31:03.944 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 58 16 101441 101425 109.283 71.25 0.00727968 0.00914637 2024-02-09T17:31:04.457 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-09T17:31:04.457 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-02-09T17:31:04.457 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph tell osd.11 'trim stale osdmaps' 2024-02-09T17:31:04.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:04 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:31:04.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:04 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:31:04.902 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:31:04.902 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-09T17:31:04.902 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-02-09T17:31:04.902 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-02-09T17:31:04.902 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:31:04.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:04 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:31:04.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:04 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:31:04.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:04 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:31:04.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:04 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:31:04.944 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 59 16 102615 102599 108.675 73.375 0.0120729 0.00919778 2024-02-09T17:31:04.947 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:31:04.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:04.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:31:04.977 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:31:05.051 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:31:05.118 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:05.476 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:05 smithi162 ceph-mon[44304]: osdmap e222: 12 total, 12 up, 12 in 2024-02-09T17:31:05.476 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:05 smithi162 ceph-mon[44304]: pgmap v652: 17 pgs: 17 active+clean; 6.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.37k op/s 2024-02-09T17:31:05.476 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:05 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:31:05.476 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:05 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:31:05.507 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:05 smithi099 ceph-mon[39420]: osdmap e222: 12 total, 12 up, 12 in 2024-02-09T17:31:05.508 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:05 smithi099 ceph-mon[39420]: pgmap v652: 17 pgs: 17 active+clean; 6.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.37k op/s 2024-02-09T17:31:05.508 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:05 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:31:05.508 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:05 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:31:05.595 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:31:05.596 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-09T17:31:05.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:05.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:31:05.708 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:31:05.770 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:05.809 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:31:05.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:05.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:31:05.915 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:31:05.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:05 smithi179 ceph-mon[43975]: osdmap e222: 12 total, 12 up, 12 in 2024-02-09T17:31:05.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:05 smithi179 ceph-mon[43975]: pgmap v652: 17 pgs: 17 active+clean; 6.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.37k op/s 2024-02-09T17:31:05.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:05 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:31:05.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:05 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:31:05.944 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:2024-02-09T17:31:05.943998+0000 min lat: 0.00110138 max lat: 0.611297 avg lat: 0.00924994 2024-02-09T17:31:05.944 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-09T17:31:05.944 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 60 15 103761 103746 108.058 71.6875 0.0171502 0.00924994 2024-02-09T17:31:05.959 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:05.999 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:31:06.004 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:06.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:31:06.074 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:31:06.105 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:31:06.134 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:06.158 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:31:06.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:06.163 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:31:06.263 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:31:06.328 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:06.368 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:31:06.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:06.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:31:06.474 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:31:06.531 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:06.600 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:31:06.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:06.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:31:06.604 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:06 smithi162 ceph-mon[44304]: osdmap e223: 12 total, 12 up, 12 in 2024-02-09T17:31:06.605 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:06 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:31:06.605 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:06 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:31:06.705 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:31:06.738 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:06 smithi179 ceph-mon[43975]: osdmap e223: 12 total, 12 up, 12 in 2024-02-09T17:31:06.739 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:06 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:31:06.739 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:06 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:31:06.739 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:06.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:06 smithi099 ceph-mon[39420]: osdmap e223: 12 total, 12 up, 12 in 2024-02-09T17:31:06.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:06 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:31:06.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:06 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:31:06.762 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:31:06.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:06.766 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:31:06.860 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:31:06.867 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:31:06.896 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:06.917 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:31:06.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:06.920 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:31:06.944 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 61 16 104889 104873 107.441 70.4375 0.00607056 0.00930316 2024-02-09T17:31:07.021 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:31:07.074 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:07.636 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:07 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:31:07.636 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:07 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:31:07.636 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:07 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:31:07.636 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:07 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:31:07.637 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:07 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:31:07.637 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:07 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:31:07.637 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:07 smithi099 ceph-mon[39420]: osdmap e224: 12 total, 12 up, 12 in 2024-02-09T17:31:07.637 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:07 smithi099 ceph-mon[39420]: pgmap v655: 17 pgs: 17 active+clean; 6.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s 2024-02-09T17:31:07.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:07 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:31:07.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:07 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:31:07.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:07 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:31:07.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:07 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:31:07.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:07 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:31:07.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:07 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:31:07.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:07 smithi179 ceph-mon[43975]: osdmap e224: 12 total, 12 up, 12 in 2024-02-09T17:31:07.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:07 smithi179 ceph-mon[43975]: pgmap v655: 17 pgs: 17 active+clean; 6.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s 2024-02-09T17:31:07.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:07 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:31:07.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:07 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:31:07.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:07 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:31:07.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:07 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:31:07.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:07 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:31:07.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:07 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:31:07.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:07 smithi162 ceph-mon[44304]: osdmap e224: 12 total, 12 up, 12 in 2024-02-09T17:31:07.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:07 smithi162 ceph-mon[44304]: pgmap v655: 17 pgs: 17 active+clean; 6.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s 2024-02-09T17:31:07.944 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 62 15 106081 106066 106.911 74.5625 0.0151055 0.00934972 2024-02-09T17:31:08.529 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is set 2024-02-09T17:31:08.560 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:08 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3631250903' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:31:08.696 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:31:08.735 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:31:08.761 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:31:08.797 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:31:08.830 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:31:08.873 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:31:08.908 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:31:08.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:08 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3631250903' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:31:08.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:08 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3631250903' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:31:08.941 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:31:08.944 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 63 16 107227 107211 106.35 71.5625 0.00809191 0.00939838 2024-02-09T17:31:08.965 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:31:08.993 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:31:09.024 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:31:09.057 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:31:09.073 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:31:09.073 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:31:09.073 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:31:09.073 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:31:09.073 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:31:09.073 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:31:09.073 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:31:09.073 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:31:09.073 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:31:09.073 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:31:09.073 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:31:09.073 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:31:09.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:09 smithi099 ceph-mon[39420]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:31:09.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:09 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3631250903' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:31:09.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:09 smithi099 ceph-mon[39420]: osdmap e225: 12 total, 12 up, 12 in 2024-02-09T17:31:09.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:09 smithi099 ceph-mon[39420]: pgmap v657: 17 pgs: 17 active+clean; 6.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 713 op/s 2024-02-09T17:31:09.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:09 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1982951836' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:31:09.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:09 smithi179 ceph-mon[43975]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:31:09.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:09 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3631250903' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:31:09.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:09 smithi179 ceph-mon[43975]: osdmap e225: 12 total, 12 up, 12 in 2024-02-09T17:31:09.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:09 smithi179 ceph-mon[43975]: pgmap v657: 17 pgs: 17 active+clean; 6.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 713 op/s 2024-02-09T17:31:09.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:09 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1982951836' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:31:09.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:09 smithi162 ceph-mon[44304]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:31:09.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:09 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3631250903' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:31:09.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:09 smithi162 ceph-mon[44304]: osdmap e225: 12 total, 12 up, 12 in 2024-02-09T17:31:09.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:09 smithi162 ceph-mon[44304]: pgmap v657: 17 pgs: 17 active+clean; 6.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 713 op/s 2024-02-09T17:31:09.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:09 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1982951836' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:31:09.944 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 64 16 108412 108396 105.845 74.0625 0.00751538 0.00944373 2024-02-09T17:31:10.850 INFO:teuthology.orchestra.run.smithi099.stderr:no valid command found; 10 closest matches: 2024-02-09T17:31:10.851 INFO:teuthology.orchestra.run.smithi099.stderr:0 2024-02-09T17:31:10.851 INFO:teuthology.orchestra.run.smithi099.stderr:1 2024-02-09T17:31:10.851 INFO:teuthology.orchestra.run.smithi099.stderr:2 2024-02-09T17:31:10.851 INFO:teuthology.orchestra.run.smithi099.stderr:abort 2024-02-09T17:31:10.851 INFO:teuthology.orchestra.run.smithi099.stderr:assert 2024-02-09T17:31:10.851 INFO:teuthology.orchestra.run.smithi099.stderr:bench [] [] [] [] 2024-02-09T17:31:10.851 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs debug_inject_read_zeros 2024-02-09T17:31:10.851 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs files list 2024-02-09T17:31:10.851 INFO:teuthology.orchestra.run.smithi099.stderr:bluefs stats 2024-02-09T17:31:10.851 INFO:teuthology.orchestra.run.smithi099.stderr:bluestore allocator dump block 2024-02-09T17:31:10.851 INFO:teuthology.orchestra.run.smithi099.stderr:Error EINVAL: invalid command 2024-02-09T17:31:10.944 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 65 16 109545 109529 105.306 70.8125 0.0100129 0.00949206 2024-02-09T17:31:11.478 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset noscrub 2024-02-09T17:31:11.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:11 smithi179 ceph-mon[43975]: pgmap v658: 17 pgs: 17 active+clean; 6.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.08k op/s 2024-02-09T17:31:11.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:11 smithi162 ceph-mon[44304]: pgmap v658: 17 pgs: 17 active+clean; 6.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.08k op/s 2024-02-09T17:31:11.944 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 66 16 110628 110612 104.736 67.6875 0.00479668 0.00954375 2024-02-09T17:31:11.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:11 smithi099 ceph-mon[39420]: pgmap v658: 17 pgs: 17 active+clean; 6.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.08k op/s 2024-02-09T17:31:12.256 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:31:12.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:12.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:31:12.370 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:31:12.404 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:12.425 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:31:12.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:12.429 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:31:12.530 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:31:12.556 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:12.591 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:12 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:31:12.591 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:12 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:31:12.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:12 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:31:12.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:12 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:31:12.945 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 67 16 111701 111685 104.174 67.0625 0.00630834 0.00959571 2024-02-09T17:31:12.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:12 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:31:12.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:12 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:31:13.609 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:31:13.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:13.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:31:13.716 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:31:13.747 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:13.769 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:31:13.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:13.773 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:31:13.874 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:31:13.905 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:13 smithi162 ceph-mon[44304]: pgmap v659: 17 pgs: 17 active+clean; 6.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-02-09T17:31:13.905 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:13 smithi162 ceph-mon[44304]: osdmap e226: 12 total, 12 up, 12 in 2024-02-09T17:31:13.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:13 smithi179 ceph-mon[43975]: pgmap v659: 17 pgs: 17 active+clean; 6.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-02-09T17:31:13.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:13 smithi179 ceph-mon[43975]: osdmap e226: 12 total, 12 up, 12 in 2024-02-09T17:31:13.936 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:13.944 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 68 16 112829 112813 103.678 70.5 0.0170796 0.00964062 2024-02-09T17:31:13.968 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:31:13.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:13.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:31:13.988 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:13 smithi099 ceph-mon[39420]: pgmap v659: 17 pgs: 17 active+clean; 6.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-02-09T17:31:13.988 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:13 smithi099 ceph-mon[39420]: osdmap e226: 12 total, 12 up, 12 in 2024-02-09T17:31:14.076 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:31:14.155 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:14.231 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:31:14.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:14.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:31:14.289 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:31:14.289 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:31:14.338 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:31:14.372 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:14.483 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:31:14.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:14.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:31:14.508 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-09T17:31:14.508 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-02-09T17:31:14.530 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:31:14.590 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:31:14.659 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:14.705 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:31:14.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:14.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:31:14.811 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:31:14.851 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:14.869 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:31:14.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:14.873 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:31:14.944 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 69 16 113923 113907 103.167 68.375 0.00868908 0.00968893 2024-02-09T17:31:14.974 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:31:15.028 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:15.136 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:31:15.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:15.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:31:15.176 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.9 dump_ops_in_flight 2024-02-09T17:31:15.242 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:31:15.300 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:15.331 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:31:15.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:15.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:31:15.437 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:31:15.479 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:15.510 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:31:15.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:15.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:31:15.529 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:31:15.602 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:15 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:31:15.602 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:15 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:31:15.602 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:15 smithi179 ceph-mon[43975]: pgmap v661: 17 pgs: 17 active+clean; 6.8 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s 2024-02-09T17:31:15.602 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:15 smithi179 ceph-mon[43975]: osdmap e227: 12 total, 12 up, 12 in 2024-02-09T17:31:15.611 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:31:15.614 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:31:15.691 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:15.734 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:31:15.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:15.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:31:15.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:15 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:31:15.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:15 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:31:15.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:15 smithi099 ceph-mon[39420]: pgmap v661: 17 pgs: 17 active+clean; 6.8 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s 2024-02-09T17:31:15.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:15 smithi099 ceph-mon[39420]: osdmap e227: 12 total, 12 up, 12 in 2024-02-09T17:31:15.842 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:31:15.891 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:15 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:31:15.892 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:15 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:31:15.892 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:15 smithi162 ceph-mon[44304]: pgmap v661: 17 pgs: 17 active+clean; 6.8 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s 2024-02-09T17:31:15.892 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:15 smithi162 ceph-mon[44304]: osdmap e227: 12 total, 12 up, 12 in 2024-02-09T17:31:15.900 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:15.929 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:31:15.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:15.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:31:15.945 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 70 16 115022 115006 102.674 68.6875 0.0102645 0.00973585 2024-02-09T17:31:16.033 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:31:16.035 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:31:16.079 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:16.945 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 71 16 116147 116131 102.218 70.3125 0.0125833 0.00977925 2024-02-09T17:31:17.045 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:31:17.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:17.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:31:17.150 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:31:17.234 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:17 smithi179 ceph-mon[43975]: pgmap v663: 17 pgs: 17 active+clean; 6.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-02-09T17:31:17.234 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:17 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2730659309' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:31:17.239 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:17.278 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:31:17.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:17.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:31:17.386 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:31:17.433 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:17.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:17 smithi099 ceph-mon[39420]: pgmap v663: 17 pgs: 17 active+clean; 6.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-02-09T17:31:17.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:17 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2730659309' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:31:17.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:17 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:31:17.517 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:31:17.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:17.524 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:17 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:31:17.525 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:31:17.626 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:31:17.675 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:17 smithi162 ceph-mon[44304]: pgmap v663: 17 pgs: 17 active+clean; 6.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-02-09T17:31:17.675 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:17 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2730659309' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:31:17.675 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:17 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:31:17.678 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:17.681 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T17:31:17.681 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:31:17.709 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:31:17.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:17.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:31:17.770 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is unset 2024-02-09T17:31:17.818 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:31:17.885 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:17.923 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:31:17.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:17.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:31:17.945 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 72 16 117252 117236 101.758 69.0625 0.00498037 0.00982328 2024-02-09T17:31:18.030 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:31:18.076 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:18.115 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:31:18.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:18.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:31:18.223 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:31:18.234 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:18 smithi162 ceph-mon[44304]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:31:18.234 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:18 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:31:18.235 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:18 smithi162 ceph-mon[44304]: osdmap e228: 12 total, 12 up, 12 in 2024-02-09T17:31:18.261 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:18.358 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:18 smithi179 ceph-mon[43975]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:31:18.359 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:18 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:31:18.359 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:18 smithi179 ceph-mon[43975]: osdmap e228: 12 total, 12 up, 12 in 2024-02-09T17:31:18.395 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.9 dump_blocked_ops 2024-02-09T17:31:18.450 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:18 smithi099 ceph-mon[39420]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:31:18.450 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:18 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:31:18.450 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:18 smithi099 ceph-mon[39420]: osdmap e228: 12 total, 12 up, 12 in 2024-02-09T17:31:18.744 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:31:18.945 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 73 15 118347 118332 101.302 68.5 0.0204724 0.00986748 2024-02-09T17:31:18.993 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:31:18.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:18.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:31:19.048 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:31:19.074 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:31:19.099 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:31:19.104 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:31:19.136 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:31:19.157 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:19.179 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:31:19.196 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:31:19.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:19.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:31:19.217 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:31:19.259 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:31:19.280 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:19 smithi179 ceph-mon[43975]: pgmap v665: 17 pgs: 17 active+clean; 7.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 920 op/s 2024-02-09T17:31:19.280 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:19 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2876966763' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:31:19.298 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:31:19.304 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:31:19.334 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:31:19.346 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:19 smithi099 ceph-mon[39420]: pgmap v665: 17 pgs: 17 active+clean; 7.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 920 op/s 2024-02-09T17:31:19.347 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:19 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2876966763' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:31:19.355 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:19.363 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:31:19.381 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:31:19.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:19.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:31:19.405 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:31:19.439 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:31:19.440 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:31:19.440 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:31:19.440 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:31:19.440 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:31:19.441 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:31:19.441 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:31:19.441 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:31:19.441 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:31:19.441 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:31:19.441 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:31:19.441 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:31:19.441 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:31:19.488 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:31:19.509 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] 2024-02-09T17:31:19.510 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:31:19.510 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-02-09T17:31:19.510 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph pg dump --format=json 2024-02-09T17:31:19.536 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:19 smithi162 ceph-mon[44304]: pgmap v665: 17 pgs: 17 active+clean; 7.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 920 op/s 2024-02-09T17:31:19.537 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:19 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2876966763' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:31:19.541 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:19.572 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:31:19.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:19.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:31:19.679 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:31:19.733 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:19.771 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:31:19.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:19.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:31:19.875 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:31:19.907 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:19.945 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 74 16 119376 119360 100.801 64.25 0.00759151 0.0099137 2024-02-09T17:31:20.360 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:31:20.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:20.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:31:20.400 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:31:20.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:20 smithi179 ceph-mon[43975]: 3.9 scrub starts 2024-02-09T17:31:20.468 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:31:20.493 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-09T17:31:20.499 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:20.536 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:20 smithi099 ceph-mon[39420]: 3.9 scrub starts 2024-02-09T17:31:20.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:20 smithi162 ceph-mon[44304]: 3.9 scrub starts 2024-02-09T17:31:20.719 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:31:20.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:20.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:31:20.826 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:31:20.870 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:31:20.870 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-02-09T17:31:20.870 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-02-09T17:31:20.870 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-02-09T17:31:20.870 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:31:20.883 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:20.912 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:31:20.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:20.917 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:31:20.946 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 75 16 120274 120258 100.205 56.125 0.017909 0.00997512 2024-02-09T17:31:21.018 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:31:21.075 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:21.111 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:31:21.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:21.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:31:21.216 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:31:21.266 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:21 smithi099 ceph-mon[39420]: 3.8 scrub starts 2024-02-09T17:31:21.266 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:21 smithi099 ceph-mon[39420]: pgmap v666: 17 pgs: 17 active+clean; 7.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.08k op/s 2024-02-09T17:31:21.266 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:21 smithi099 ceph-mon[39420]: 3.9 scrub ok 2024-02-09T17:31:21.278 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:21.314 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:31:21.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:21.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:31:21.322 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:21 smithi162 ceph-mon[44304]: 3.8 scrub starts 2024-02-09T17:31:21.322 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:21 smithi162 ceph-mon[44304]: pgmap v666: 17 pgs: 17 active+clean; 7.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.08k op/s 2024-02-09T17:31:21.322 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:21 smithi162 ceph-mon[44304]: 3.9 scrub ok 2024-02-09T17:31:21.376 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:31:21.420 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:31:21.459 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:21.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:21 smithi179 ceph-mon[43975]: 3.8 scrub starts 2024-02-09T17:31:21.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:21 smithi179 ceph-mon[43975]: pgmap v666: 17 pgs: 17 active+clean; 7.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.08k op/s 2024-02-09T17:31:21.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:21 smithi179 ceph-mon[43975]: 3.9 scrub ok 2024-02-09T17:31:21.920 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:31:21.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:21.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:31:21.945 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 76 16 121221 121205 99.6653 59.1875 0.0144655 0.0100293 2024-02-09T17:31:22.029 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:31:22.068 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:22.196 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:31:22.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:22.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:31:22.308 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:31:22.317 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:31:22.372 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:22.404 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:31:22.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:22.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:31:22.498 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:22 smithi099 ceph-mon[39420]: 3.e scrub starts 2024-02-09T17:31:22.498 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:22 smithi099 ceph-mon[39420]: 3.5 scrub starts 2024-02-09T17:31:22.498 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:22 smithi099 ceph-mon[39420]: 3.8 scrub ok 2024-02-09T17:31:22.518 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:31:22.567 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:22.603 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:31:22.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:22.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:31:22.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:22 smithi179 ceph-mon[43975]: 3.e scrub starts 2024-02-09T17:31:22.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:22 smithi179 ceph-mon[43975]: 3.5 scrub starts 2024-02-09T17:31:22.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:22 smithi179 ceph-mon[43975]: 3.8 scrub ok 2024-02-09T17:31:22.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:22 smithi162 ceph-mon[44304]: 3.e scrub starts 2024-02-09T17:31:22.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:22 smithi162 ceph-mon[44304]: 3.5 scrub starts 2024-02-09T17:31:22.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:22 smithi162 ceph-mon[44304]: 3.8 scrub ok 2024-02-09T17:31:22.711 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:31:22.749 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:22.947 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 77 16 122035 122019 99.0316 50.875 0.0242992 0.0100931 2024-02-09T17:31:23.223 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:31:23.476 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:31:23.477 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-02-09T17:31:23.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:23 smithi179 ceph-mon[43975]: 3.c scrub starts 2024-02-09T17:31:23.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:23 smithi179 ceph-mon[43975]: 3.d scrub starts 2024-02-09T17:31:23.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:23 smithi179 ceph-mon[43975]: pgmap v667: 17 pgs: 17 active+clean; 7.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s 2024-02-09T17:31:23.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:23 smithi162 ceph-mon[44304]: 3.c scrub starts 2024-02-09T17:31:23.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:23 smithi162 ceph-mon[44304]: 3.d scrub starts 2024-02-09T17:31:23.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:23 smithi162 ceph-mon[44304]: pgmap v667: 17 pgs: 17 active+clean; 7.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s 2024-02-09T17:31:23.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:23 smithi099 ceph-mon[39420]: 3.c scrub starts 2024-02-09T17:31:23.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:23 smithi099 ceph-mon[39420]: 3.d scrub starts 2024-02-09T17:31:23.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:23 smithi099 ceph-mon[39420]: pgmap v667: 17 pgs: 17 active+clean; 7.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s 2024-02-09T17:31:23.946 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 78 16 122715 122699 98.3068 42.5 0.0248806 0.0101674 2024-02-09T17:31:24.068 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.9 dump_historic_ops 2024-02-09T17:31:24.108 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:31:24.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:24.114 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:31:24.215 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:31:24.251 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:24.408 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:31:24.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:24.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:31:24.449 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:31:24.520 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:31:24.602 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:24 smithi162 ceph-mon[44304]: 3.e scrub ok 2024-02-09T17:31:24.602 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:24 smithi162 ceph-mon[44304]: 3.a scrub starts 2024-02-09T17:31:24.602 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:24 smithi162 ceph-mon[44304]: 3.5 scrub ok 2024-02-09T17:31:24.602 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:24 smithi162 ceph-mon[44304]: 3.3 scrub starts 2024-02-09T17:31:24.602 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:24 smithi162 ceph-mon[44304]: 3.f scrub starts 2024-02-09T17:31:24.602 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:24 smithi162 ceph-mon[44304]: from='client.24910 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:31:24.603 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:24.643 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:31:24.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:24.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:31:24.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:24 smithi179 ceph-mon[43975]: 3.e scrub ok 2024-02-09T17:31:24.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:24 smithi179 ceph-mon[43975]: 3.a scrub starts 2024-02-09T17:31:24.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:24 smithi179 ceph-mon[43975]: 3.5 scrub ok 2024-02-09T17:31:24.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:24 smithi179 ceph-mon[43975]: 3.3 scrub starts 2024-02-09T17:31:24.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:24 smithi179 ceph-mon[43975]: 3.f scrub starts 2024-02-09T17:31:24.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:24 smithi179 ceph-mon[43975]: from='client.24910 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:31:24.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:24 smithi099 ceph-mon[39420]: 3.e scrub ok 2024-02-09T17:31:24.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:24 smithi099 ceph-mon[39420]: 3.a scrub starts 2024-02-09T17:31:24.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:24 smithi099 ceph-mon[39420]: 3.5 scrub ok 2024-02-09T17:31:24.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:24 smithi099 ceph-mon[39420]: 3.3 scrub starts 2024-02-09T17:31:24.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:24 smithi099 ceph-mon[39420]: 3.f scrub starts 2024-02-09T17:31:24.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:24 smithi099 ceph-mon[39420]: from='client.24910 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:31:24.751 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:31:24.798 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:24.826 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:31:24.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:24.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:31:24.930 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:31:24.945 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 79 16 123408 123392 97.6107 43.3125 0.0147649 0.0102405 2024-02-09T17:31:24.956 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:25.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:25 smithi179 ceph-mon[43975]: 3.6 scrub starts 2024-02-09T17:31:25.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:25 smithi179 ceph-mon[43975]: 3.c scrub ok 2024-02-09T17:31:25.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:25 smithi179 ceph-mon[43975]: pgmap v668: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.4 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.03k op/s 2024-02-09T17:31:25.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:25 smithi179 ceph-mon[43975]: 3.d scrub ok 2024-02-09T17:31:25.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:25 smithi162 ceph-mon[44304]: 3.6 scrub starts 2024-02-09T17:31:25.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:25 smithi162 ceph-mon[44304]: 3.c scrub ok 2024-02-09T17:31:25.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:25 smithi162 ceph-mon[44304]: pgmap v668: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.4 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.03k op/s 2024-02-09T17:31:25.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:25 smithi162 ceph-mon[44304]: 3.d scrub ok 2024-02-09T17:31:25.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:25 smithi099 ceph-mon[39420]: 3.6 scrub starts 2024-02-09T17:31:25.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:25 smithi099 ceph-mon[39420]: 3.c scrub ok 2024-02-09T17:31:25.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:25 smithi099 ceph-mon[39420]: pgmap v668: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.4 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.03k op/s 2024-02-09T17:31:25.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:25 smithi099 ceph-mon[39420]: 3.d scrub ok 2024-02-09T17:31:25.946 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:2024-02-09T17:31:25.945872+0000 min lat: 0.00110138 max lat: 0.611297 avg lat: 0.0103005 2024-02-09T17:31:25.946 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-09T17:31:25.946 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 80 16 124239 124223 97.0397 51.9375 0.00829814 0.0103005 2024-02-09T17:31:26.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:26 smithi179 ceph-mon[43975]: 3.a scrub ok 2024-02-09T17:31:26.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:26 smithi179 ceph-mon[43975]: 3.f scrub ok 2024-02-09T17:31:26.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:26 smithi162 ceph-mon[44304]: 3.a scrub ok 2024-02-09T17:31:26.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:26 smithi162 ceph-mon[44304]: 3.f scrub ok 2024-02-09T17:31:26.708 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:26 smithi099 ceph-mon[39420]: 3.a scrub ok 2024-02-09T17:31:26.708 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:26 smithi099 ceph-mon[39420]: 3.f scrub ok 2024-02-09T17:31:26.722 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:31:26.731 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:26.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:31:26.832 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:31:26.867 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:26.909 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:31:26.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:26.912 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:31:26.950 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 81 16 125311 125295 96.6642 67 0.0149003 0.0103416 2024-02-09T17:31:27.014 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:31:27.048 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:27.078 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:31:27.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:27.083 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:31:27.184 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:31:27.215 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:27.256 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:31:27.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:27.260 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:31:27.362 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:31:27.417 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:27.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:27 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:31:27.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:27 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:31:27.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:27 smithi179 ceph-mon[43975]: 3.6 scrub ok 2024-02-09T17:31:27.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:27 smithi179 ceph-mon[43975]: 3.3 scrub ok 2024-02-09T17:31:27.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:27 smithi179 ceph-mon[43975]: osdmap e229: 12 total, 12 up, 12 in 2024-02-09T17:31:27.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:27 smithi179 ceph-mon[43975]: pgmap v670: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 942 op/s 2024-02-09T17:31:27.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:27 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:31:27.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:27 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:31:27.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:27 smithi162 ceph-mon[44304]: 3.6 scrub ok 2024-02-09T17:31:27.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:27 smithi162 ceph-mon[44304]: 3.3 scrub ok 2024-02-09T17:31:27.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:27 smithi162 ceph-mon[44304]: osdmap e229: 12 total, 12 up, 12 in 2024-02-09T17:31:27.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:27 smithi162 ceph-mon[44304]: pgmap v670: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 942 op/s 2024-02-09T17:31:27.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:27 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:31:27.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:27 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:31:27.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:27 smithi099 ceph-mon[39420]: 3.6 scrub ok 2024-02-09T17:31:27.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:27 smithi099 ceph-mon[39420]: 3.3 scrub ok 2024-02-09T17:31:27.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:27 smithi099 ceph-mon[39420]: osdmap e229: 12 total, 12 up, 12 in 2024-02-09T17:31:27.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:27 smithi099 ceph-mon[39420]: pgmap v670: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 942 op/s 2024-02-09T17:31:27.950 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 82 16 126179 126163 96.1469 54.25 0.00748427 0.0103965 2024-02-09T17:31:28.348 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:28 smithi099 ceph-mon[39420]: 3.7 scrub starts 2024-02-09T17:31:28.348 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:28 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/447713558' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:31:28.348 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:28 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:31:28.352 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is unset 2024-02-09T17:31:28.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:28 smithi179 ceph-mon[43975]: 3.7 scrub starts 2024-02-09T17:31:28.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:28 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/447713558' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:31:28.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:28 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:31:28.593 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":667,"stamp":"2024-02-09T17:31:22.534583+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7837057568,"num_objects":119577,"num_object_clones":0,"num_object_copies":358731,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":119577,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":119756,"num_write_kb":7655738,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119714,"ondisk_log_size":119714,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":24517332,"kb_used_data":22835976,"kb_used_omap":229,"kb_used_meta":1680474,"kb_avail":1100424492,"statfs":{"total":1151940427776,"available":1126834679808,"internally_reserved":0,"allocated":23384039424,"data_stored":23380345620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235127,"internal_metadata":1720805769},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":532873216,"num_objects":8131,"num_object_clones":0,"num_object_copies":24393,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8131,"num_write_kb":520384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003468"},"pg_stats":[{"pgid":"3.f","version":"228'7560","reported_seq":7628,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-09T17:31:21.642520+0000","last_change":"2024-02-09T17:29:47.582551+0000","last_active":"2024-02-09T17:31:21.642520+0000","last_peered":"2024-02-09T17:31:21.642520+0000","last_clean":"2024-02-09T17:31:21.642520+0000","last_became_active":"2024-02-09T17:29:47.582245+0000","last_became_peered":"2024-02-09T17:29:47.582245+0000","last_unstale":"2024-02-09T17:31:21.642520+0000","last_undegraded":"2024-02-09T17:31:21.642520+0000","last_fullsized":"2024-02-09T17:31:21.642520+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:29:46.563947+0000","objects_scrubbed":0,"log_size":7560,"log_dups_size":0,"ondisk_log_size":7560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":495452160,"num_objects":7560,"num_object_clones":0,"num_object_copies":22680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7560,"num_write_kb":483840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"228'7482","reported_seq":7550,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-09T17:31:19.366673+0000","last_change":"2024-02-09T17:29:47.581870+0000","last_active":"2024-02-09T17:31:19.366673+0000","last_peered":"2024-02-09T17:31:19.366673+0000","last_clean":"2024-02-09T17:31:19.366673+0000","last_became_active":"2024-02-09T17:29:47.581515+0000","last_became_peered":"2024-02-09T17:29:47.581515+0000","last_unstale":"2024-02-09T17:31:19.366673+0000","last_undegraded":"2024-02-09T17:31:19.366673+0000","last_fullsized":"2024-02-09T17:31:19.366673+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:29:46.563947+0000","objects_scrubbed":0,"log_size":7482,"log_dups_size":0,"ondisk_log_size":7482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":490340352,"num_objects":7482,"num_object_clones":0,"num_object_copies":22446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7482,"num_write_kb":478848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,0],"acting":[11,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"228'7634","reported_seq":7702,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-09T17:31:21.657044+0000","last_change":"2024-02-09T17:29:47.583552+0000","last_active":"2024-02-09T17:31:21.657044+0000","last_peered":"2024-02-09T17:31:21.657044+0000","last_clean":"2024-02-09T17:31:21.657044+0000","last_became_active":"2024-02-09T17:29:47.583238+0000","last_became_peered":"2024-02-09T17:29:47.583238+0000","last_unstale":"2024-02-09T17:31:21.657044+0000","last_undegraded":"2024-02-09T17:31:21.657044+0000","last_fullsized":"2024-02-09T17:31:21.657044+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:29:46.563947+0000","objects_scrubbed":0,"log_size":7634,"log_dups_size":0,"ondisk_log_size":7634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":500301824,"num_objects":7634,"num_object_clones":0,"num_object_copies":22902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7634,"num_write_kb":488576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,9],"acting":[7,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"228'7392","reported_seq":7458,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-09T17:31:19.147670+0000","last_change":"2024-02-09T17:29:47.583525+0000","last_active":"2024-02-09T17:31:19.147670+0000","last_peered":"2024-02-09T17:31:19.147670+0000","last_clean":"2024-02-09T17:31:19.147670+0000","last_became_active":"2024-02-09T17:29:47.583200+0000","last_became_peered":"2024-02-09T17:29:47.583200+0000","last_unstale":"2024-02-09T17:31:19.147670+0000","last_undegraded":"2024-02-09T17:31:19.147670+0000","last_fullsized":"2024-02-09T17:31:19.147670+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:29:46.563947+0000","objects_scrubbed":0,"log_size":7392,"log_dups_size":0,"ondisk_log_size":7392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:31:20.054575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484442112,"num_objects":7392,"num_object_clones":0,"num_object_copies":22176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7392,"num_write_kb":473088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"228'7503","reported_seq":7571,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-09T17:31:21.414748+0000","last_change":"2024-02-09T17:29:47.583325+0000","last_active":"2024-02-09T17:31:21.414748+0000","last_peered":"2024-02-09T17:31:21.414748+0000","last_clean":"2024-02-09T17:31:21.414748+0000","last_became_active":"2024-02-09T17:29:47.583033+0000","last_became_peered":"2024-02-09T17:29:47.583033+0000","last_unstale":"2024-02-09T17:31:21.414748+0000","last_undegraded":"2024-02-09T17:31:21.414748+0000","last_fullsized":"2024-02-09T17:31:21.414748+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:29:46.563947+0000","objects_scrubbed":0,"log_size":7503,"log_dups_size":0,"ondisk_log_size":7503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":491716608,"num_objects":7503,"num_object_clones":0,"num_object_copies":22509,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7503,"num_write_kb":480192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"228'7468","reported_seq":7535,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-09T17:31:20.523758+0000","last_change":"2024-02-09T17:29:47.581898+0000","last_active":"2024-02-09T17:31:20.523758+0000","last_peered":"2024-02-09T17:31:20.523758+0000","last_clean":"2024-02-09T17:31:20.523758+0000","last_became_active":"2024-02-09T17:29:47.581573+0000","last_became_peered":"2024-02-09T17:29:47.581573+0000","last_unstale":"2024-02-09T17:31:20.523758+0000","last_undegraded":"2024-02-09T17:31:20.523758+0000","last_fullsized":"2024-02-09T17:31:20.523758+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:29:46.563947+0000","objects_scrubbed":0,"log_size":7468,"log_dups_size":0,"ondisk_log_size":7468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:31:20.920209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":489422848,"num_objects":7468,"num_object_clones":0,"num_object_copies":22404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7468,"num_write_kb":477952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"228'7617","reported_seq":7792,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-09T17:31:21.785786+0000","last_change":"2024-02-09T17:31:20.743404+0000","last_active":"2024-02-09T17:31:21.785786+0000","last_peered":"2024-02-09T17:31:21.785786+0000","last_clean":"2024-02-09T17:31:21.785786+0000","last_became_active":"2024-02-09T17:29:47.582488+0000","last_became_peered":"2024-02-09T17:29:47.582488+0000","last_unstale":"2024-02-09T17:31:21.785786+0000","last_undegraded":"2024-02-09T17:31:21.785786+0000","last_fullsized":"2024-02-09T17:31:21.785786+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'7551","last_scrub_stamp":"2024-02-09T17:31:20.743150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:31:20.743150+0000","objects_scrubbed":7492,"log_size":7617,"log_dups_size":0,"ondisk_log_size":7617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:32:34.068254+0000","scrub_duration":1578,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":499187712,"num_objects":7617,"num_object_clones":0,"num_object_copies":22851,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7617,"num_write_kb":487488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.8","version":"228'7463","reported_seq":7531,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-09T17:31:19.299370+0000","last_change":"2024-02-09T17:29:47.581795+0000","last_active":"2024-02-09T17:31:19.299370+0000","last_peered":"2024-02-09T17:31:19.299370+0000","last_clean":"2024-02-09T17:31:19.299370+0000","last_became_active":"2024-02-09T17:29:47.581467+0000","last_became_peered":"2024-02-09T17:29:47.581467+0000","last_unstale":"2024-02-09T17:31:19.299370+0000","last_undegraded":"2024-02-09T17:31:19.299370+0000","last_fullsized":"2024-02-09T17:31:19.299370+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:29:46.563947+0000","objects_scrubbed":0,"log_size":7463,"log_dups_size":0,"ondisk_log_size":7463,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":489095168,"num_objects":7463,"num_object_clones":0,"num_object_copies":22389,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7463,"num_write_kb":477632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.7","version":"228'7512","reported_seq":7608,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-09T17:31:21.442948+0000","last_change":"2024-02-09T17:31:00.664556+0000","last_active":"2024-02-09T17:31:21.442948+0000","last_peered":"2024-02-09T17:31:21.442948+0000","last_clean":"2024-02-09T17:31:21.442948+0000","last_became_active":"2024-02-09T17:29:47.583311+0000","last_became_peered":"2024-02-09T17:29:47.583311+0000","last_unstale":"2024-02-09T17:31:21.442948+0000","last_undegraded":"2024-02-09T17:31:21.442948+0000","last_fullsized":"2024-02-09T17:31:21.442948+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:29:46.563947+0000","objects_scrubbed":2499,"log_size":7512,"log_dups_size":0,"ondisk_log_size":7512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":492306432,"num_objects":7512,"num_object_clones":0,"num_object_copies":22536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7512,"num_write_kb":480768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"228'7276","reported_seq":7343,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-09T17:31:18.621667+0000","last_change":"2024-02-09T17:29:47.580882+0000","last_active":"2024-02-09T17:31:18.621667+0000","last_peered":"2024-02-09T17:31:18.621667+0000","last_clean":"2024-02-09T17:31:18.621667+0000","last_became_active":"2024-02-09T17:29:47.580696+0000","last_became_peered":"2024-02-09T17:29:47.580696+0000","last_unstale":"2024-02-09T17:31:18.621667+0000","last_undegraded":"2024-02-09T17:31:18.621667+0000","last_fullsized":"2024-02-09T17:31:18.621667+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:29:46.563947+0000","objects_scrubbed":0,"log_size":7276,"log_dups_size":0,"ondisk_log_size":7276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:31:20.003483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":476839936,"num_objects":7276,"num_object_clones":0,"num_object_copies":21828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7276,"num_write_kb":465664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"228'7366","reported_seq":7434,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-09T17:31:19.110805+0000","last_change":"2024-02-09T17:29:47.581488+0000","last_active":"2024-02-09T17:31:19.110805+0000","last_peered":"2024-02-09T17:31:19.110805+0000","last_clean":"2024-02-09T17:31:19.110805+0000","last_became_active":"2024-02-09T17:29:47.581119+0000","last_became_peered":"2024-02-09T17:29:47.581119+0000","last_unstale":"2024-02-09T17:31:19.110805+0000","last_undegraded":"2024-02-09T17:31:19.110805+0000","last_fullsized":"2024-02-09T17:31:19.110805+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:29:46.563947+0000","objects_scrubbed":0,"log_size":7366,"log_dups_size":0,"ondisk_log_size":7366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:31:20.054534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482738176,"num_objects":7366,"num_object_clones":0,"num_object_copies":22098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7366,"num_write_kb":471424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"228'7444","reported_seq":7572,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-09T17:31:19.939996+0000","last_change":"2024-02-09T17:30:48.974115+0000","last_active":"2024-02-09T17:31:19.939996+0000","last_peered":"2024-02-09T17:31:19.939996+0000","last_clean":"2024-02-09T17:31:19.939996+0000","last_became_active":"2024-02-09T17:29:47.582922+0000","last_became_peered":"2024-02-09T17:29:47.582922+0000","last_unstale":"2024-02-09T17:31:19.939996+0000","last_undegraded":"2024-02-09T17:31:19.939996+0000","last_fullsized":"2024-02-09T17:31:19.939996+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'5192","last_scrub_stamp":"2024-02-09T17:30:48.973955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:30:48.973955+0000","objects_scrubbed":5138,"log_size":7444,"log_dups_size":0,"ondisk_log_size":7444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:31:57.714286+0000","scrub_duration":1357,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":487849984,"num_objects":7444,"num_object_clones":0,"num_object_copies":22332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7444,"num_write_kb":476416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"228'7477","reported_seq":7545,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-09T17:31:19.168624+0000","last_change":"2024-02-09T17:29:47.581908+0000","last_active":"2024-02-09T17:31:19.168624+0000","last_peered":"2024-02-09T17:31:19.168624+0000","last_clean":"2024-02-09T17:31:19.168624+0000","last_became_active":"2024-02-09T17:29:47.581587+0000","last_became_peered":"2024-02-09T17:29:47.581587+0000","last_unstale":"2024-02-09T17:31:19.168624+0000","last_undegraded":"2024-02-09T17:31:19.168624+0000","last_fullsized":"2024-02-09T17:31:19.168624+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:29:46.563947+0000","objects_scrubbed":0,"log_size":7477,"log_dups_size":0,"ondisk_log_size":7477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:31:19.242974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":490012672,"num_objects":7477,"num_object_clones":0,"num_object_copies":22431,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7477,"num_write_kb":478528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"228'7347","reported_seq":7483,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-09T17:31:18.605733+0000","last_change":"2024-02-09T17:31:00.396464+0000","last_active":"2024-02-09T17:31:18.605733+0000","last_peered":"2024-02-09T17:31:18.605733+0000","last_clean":"2024-02-09T17:31:18.605733+0000","last_became_active":"2024-02-09T17:29:47.789150+0000","last_became_peered":"2024-02-09T17:29:47.789150+0000","last_unstale":"2024-02-09T17:31:18.605733+0000","last_undegraded":"2024-02-09T17:31:18.605733+0000","last_fullsized":"2024-02-09T17:31:18.605733+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"220'6050","last_scrub_stamp":"2024-02-09T17:31:00.396342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:31:00.396342+0000","objects_scrubbed":6003,"log_size":7347,"log_dups_size":0,"ondisk_log_size":7347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:32:19.877511+0000","scrub_duration":1427,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":481492992,"num_objects":7347,"num_object_clones":0,"num_object_copies":22041,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7347,"num_write_kb":470208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,10],"acting":[0,4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"228'7432","reported_seq":7566,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-09T17:31:19.376482+0000","last_change":"2024-02-09T17:30:55.910896+0000","last_active":"2024-02-09T17:31:19.376482+0000","last_peered":"2024-02-09T17:31:19.376482+0000","last_clean":"2024-02-09T17:31:19.376482+0000","last_became_active":"2024-02-09T17:29:47.582887+0000","last_became_peered":"2024-02-09T17:29:47.582887+0000","last_unstale":"2024-02-09T17:31:19.376482+0000","last_undegraded":"2024-02-09T17:31:19.376482+0000","last_fullsized":"2024-02-09T17:31:19.376482+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"218'5796","last_scrub_stamp":"2024-02-09T17:30:55.910807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:30:55.910807+0000","objects_scrubbed":5754,"log_size":7432,"log_dups_size":0,"ondisk_log_size":7432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:32:23.481568+0000","scrub_duration":1081,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":487063552,"num_objects":7432,"num_object_clones":0,"num_object_copies":22296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7432,"num_write_kb":475648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,6],"acting":[11,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"228'7602","reported_seq":7670,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-09T17:31:21.403814+0000","last_change":"2024-02-09T17:29:47.581629+0000","last_active":"2024-02-09T17:31:21.403814+0000","last_peered":"2024-02-09T17:31:21.403814+0000","last_clean":"2024-02-09T17:31:21.403814+0000","last_became_active":"2024-02-09T17:29:47.581341+0000","last_became_peered":"2024-02-09T17:29:47.581341+0000","last_unstale":"2024-02-09T17:31:21.403814+0000","last_undegraded":"2024-02-09T17:31:21.403814+0000","last_fullsized":"2024-02-09T17:31:21.403814+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:29:46.563947+0000","objects_scrubbed":0,"log_size":7602,"log_dups_size":0,"ondisk_log_size":7602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":498204672,"num_objects":7602,"num_object_clones":0,"num_object_copies":22806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7602,"num_write_kb":486528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"63'139","reported_seq":668,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-02-09T17:31:17.242204+0000","last_change":"2024-02-09T17:30:34.216079+0000","last_active":"2024-02-09T17:31:17.242204+0000","last_peered":"2024-02-09T17:31:17.242204+0000","last_clean":"2024-02-09T17:31:17.242204+0000","last_became_active":"2024-02-09T17:22:46.986816+0000","last_became_peered":"2024-02-09T17:22:46.986816+0000","last_unstale":"2024-02-09T17:31:17.242204+0000","last_undegraded":"2024-02-09T17:31:17.242204+0000","last_fullsized":"2024-02-09T17:31:17.242204+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:30:34.215944+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:30:34.215944+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:31:52.241425+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":7836467200,"num_objects":119575,"num_object_clones":0,"num_object_copies":358725,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":119575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":119575,"num_write_kb":7652800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":23476043776,"data_stored":23476043776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119575,"ondisk_log_size":119575,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056747,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3460788,"kb_used_data":3321324,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":90284364,"statfs":{"total":95995035648,"available":92451188736,"internally_reserved":0,"allocated":3401035776,"data_stored":3400725280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":142801348},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:30:55 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":1.238,"15min":0.82699999999999996},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.60599999999999998,"5min":14.913,"15min":14.913},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":1.46,"15min":0.91000000000000003},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":1.77,"5min":14.869,"15min":14.869},"last":3.3290000000000002}]},{"osd":1,"last update":"Fri Feb 9 17:30:55 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":2.274,"15min":1.2210000000000001},"min":{"1min":0.27800000000000002,"5min":0.245,"15min":0.245},"max":{"1min":4.2969999999999997,"5min":32.957999999999998,"15min":32.957999999999998},"last":1.8440000000000001},{"interface":"front","average":{"1min":0.91900000000000004,"5min":2.4620000000000002,"15min":1.2549999999999999},"min":{"1min":0.34899999999999998,"5min":0.26800000000000002,"15min":0.26700000000000002},"max":{"1min":4.3460000000000001,"5min":32.880000000000003,"15min":32.880000000000003},"last":1.8720000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:30:45 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":1.542,"15min":1.728},"min":{"1min":0.27200000000000002,"5min":0.247,"15min":0.247},"max":{"1min":2.2189999999999999,"5min":9.4719999999999995,"15min":9.4719999999999995},"last":3.4990000000000001},{"interface":"front","average":{"1min":0.81299999999999994,"5min":1.516,"15min":1.4399999999999999},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":4.008,"5min":7.96,"15min":7.96},"last":4.8579999999999997}]},{"osd":3,"last update":"Fri Feb 9 17:30:55 2024","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":2.5800000000000001,"15min":1.3069999999999999},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":2.375,"5min":21.315999999999999,"15min":21.315999999999999},"last":4.0990000000000002},{"interface":"front","average":{"1min":1.137,"5min":2.121,"15min":1.204},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":4.3179999999999996,"5min":18.087,"15min":18.087},"last":0.32600000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:30:55 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.81000000000000005,"15min":0.76700000000000002},"min":{"1min":0.317,"5min":0.30499999999999999,"15min":0.30299999999999999},"max":{"1min":2.7519999999999998,"5min":5.694,"15min":5.694},"last":0.40899999999999997},{"interface":"front","average":{"1min":1.254,"5min":1.0389999999999999,"15min":0.85299999999999998},"min":{"1min":0.312,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":5.1360000000000001,"5min":6.5140000000000002,"15min":6.5140000000000002},"last":1.6910000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:30:55 2024","interfaces":[{"interface":"back","average":{"1min":1.6519999999999999,"5min":1.046,"15min":0.85899999999999999},"min":{"1min":0.314,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":13.891,"5min":13.891,"15min":13.891},"last":3.867},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.006,"15min":0.84799999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":2.2879999999999998,"5min":6.5970000000000004,"15min":6.5970000000000004},"last":0.36199999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:30:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.95699999999999996,"15min":0.84699999999999998},"min":{"1min":0.36499999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.5659999999999998,"5min":7.726,"15min":7.726},"last":1.825},{"interface":"front","average":{"1min":1.504,"5min":1.0800000000000001,"15min":0.874},"min":{"1min":0.38400000000000001,"5min":0.379,"15min":0.32300000000000001},"max":{"1min":5.4089999999999998,"5min":6.6219999999999999,"15min":6.6219999999999999},"last":1.794}]},{"osd":7,"last update":"Fri Feb 9 17:30:55 2024","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.198,"15min":0.92500000000000004},"min":{"1min":0.41299999999999998,"5min":0.317,"15min":0.317},"max":{"1min":4.431,"5min":7.7720000000000002,"15min":7.7720000000000002},"last":0.67700000000000005},{"interface":"front","average":{"1min":1.151,"5min":1.071,"15min":0.88400000000000001},"min":{"1min":0.45700000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":3.9809999999999999,"5min":5.0369999999999999,"15min":5.0369999999999999},"last":0.70599999999999996}]},{"osd":8,"last update":"Fri Feb 9 17:25:48 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.71399999999999997,"15min":0.68000000000000005},"min":{"1min":0.40799999999999997,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.9350000000000001,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":2.8180000000000001},{"interface":"front","average":{"1min":1.3029999999999999,"5min":0.80700000000000005,"15min":0.71999999999999997},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":3.7389999999999999,"5min":3.7389999999999999,"15min":3.7389999999999999},"last":6.0659999999999998}]},{"osd":9,"last update":"Fri Feb 9 17:30:42 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":1.0429999999999999,"15min":1.2410000000000001},"min":{"1min":0.33900000000000002,"5min":0.316,"15min":0.316},"max":{"1min":2.1379999999999999,"5min":6.7279999999999998,"15min":6.7279999999999998},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.98199999999999998,"15min":0.89400000000000002},"min":{"1min":0.35199999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.8120000000000001,"5min":6.0860000000000003,"15min":6.0860000000000003},"last":0.41899999999999998}]},{"osd":10,"last update":"Fri Feb 9 17:30:55 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":1.056,"15min":0.83799999999999997},"min":{"1min":0.40000000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.615,"5min":6.7759999999999998,"15min":6.7759999999999998},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.68999999999999995,"5min":1.022,"15min":0.79900000000000004},"min":{"1min":0.33900000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.9990000000000001,"5min":10.675000000000001,"15min":10.675000000000001},"last":0.51300000000000001}]}]},{"osd":10,"up_from":84,"seq":360777252976,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":573288,"kb_used_data":465572,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93171864,"statfs":{"total":95995035648,"available":95407988736,"internally_reserved":0,"allocated":476745728,"data_stored":476438784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":110098880},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:30:48 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":1.681,"15min":1.8600000000000001},"min":{"1min":0.26700000000000002,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":1,"5min":20.120999999999999,"15min":20.120999999999999},"last":1.29},{"interface":"front","average":{"1min":0.63300000000000001,"5min":1.8340000000000001,"15min":2.5379999999999998},"min":{"1min":0.27400000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.361,"5min":14.471,"15min":14.471},"last":2.419}]},{"osd":1,"last update":"Fri Feb 9 17:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":1.6719999999999999,"15min":0.95399999999999996},"min":{"1min":0.29599999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":1.359,"5min":10.557,"15min":10.557},"last":3.8679999999999999},{"interface":"front","average":{"1min":0.55200000000000005,"5min":1.625,"15min":0.96299999999999997},"min":{"1min":0.33200000000000002,"5min":0.245,"15min":0.245},"max":{"1min":1.157,"5min":10.433999999999999,"15min":10.433999999999999},"last":0.30599999999999999}]},{"osd":2,"last update":"Fri Feb 9 17:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":1.847,"15min":1.0349999999999999},"min":{"1min":0.35199999999999998,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":1.3089999999999999,"5min":13.351000000000001,"15min":13.351000000000001},"last":3.149},{"interface":"front","average":{"1min":0.59199999999999997,"5min":1.76,"15min":1.008},"min":{"1min":0.32700000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.391,"5min":10.596,"15min":10.596},"last":4.0629999999999997}]},{"osd":3,"last update":"Fri Feb 9 17:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":1.958,"15min":1.0780000000000001},"min":{"1min":0.29599999999999999,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.034,"5min":9.2650000000000006,"15min":9.2650000000000006},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.65200000000000002,"5min":1.7350000000000001,"15min":1.0089999999999999},"min":{"1min":0.35199999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.071,"5min":8.1080000000000005,"15min":8.1080000000000005},"last":3.9590000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.88900000000000001,"15min":0.77200000000000002},"min":{"1min":0.318,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.9350000000000001,"5min":6.2249999999999996,"15min":6.2249999999999996},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.97799999999999998,"5min":0.98599999999999999,"15min":0.81299999999999994},"min":{"1min":0.52100000000000002,"5min":0.307,"15min":0.307},"max":{"1min":2.5350000000000001,"5min":6.8369999999999997,"15min":6.8369999999999997},"last":0.44400000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:30:38 2024","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.218,"15min":0.91300000000000003},"min":{"1min":0.53300000000000003,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":6.2309999999999999,"5min":7.3339999999999996,"15min":7.3339999999999996},"last":3.976},{"interface":"front","average":{"1min":1.208,"5min":1.266,"15min":0.90600000000000003},"min":{"1min":0.502,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":3.3159999999999998,"5min":8.6159999999999997,"15min":8.6159999999999997},"last":0.39400000000000002}]},{"osd":6,"last update":"Fri Feb 9 17:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":1.0680000000000001,"15min":0.81799999999999995},"min":{"1min":0.33600000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":2.4430000000000001,"5min":6.1230000000000002,"15min":6.1230000000000002},"last":0.60199999999999998},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.165,"15min":0.875},"min":{"1min":0.501,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":3.5790000000000002,"5min":6.8609999999999998,"15min":6.8609999999999998},"last":4.4660000000000002}]},{"osd":7,"last update":"Fri Feb 9 17:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.98999999999999999,"15min":0.80900000000000005},"min":{"1min":0.39300000000000002,"5min":0.312,"15min":0.312},"max":{"1min":1.9119999999999999,"5min":4.3659999999999997,"15min":4.3659999999999997},"last":0.622},{"interface":"front","average":{"1min":0.872,"5min":1.242,"15min":0.91200000000000003},"min":{"1min":0.39400000000000002,"5min":0.316,"15min":0.316},"max":{"1min":1.702,"5min":6.9589999999999996,"15min":6.9589999999999996},"last":0.42399999999999999}]},{"osd":8,"last update":"Fri Feb 9 17:26:28 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.73799999999999999,"15min":0.64000000000000001},"min":{"1min":0.36399999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":3.6509999999999998,"5min":3.6509999999999998,"15min":3.6509999999999998},"last":0.91800000000000004},{"interface":"front","average":{"1min":1.72,"5min":0.89500000000000002,"15min":0.68200000000000005},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":4.173,"5min":4.173,"15min":4.173},"last":0.51100000000000001}]},{"osd":9,"last update":"Fri Feb 9 17:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":1.1319999999999999,"15min":0.82499999999999996},"min":{"1min":0.38800000000000001,"5min":0.314,"15min":0.314},"max":{"1min":1.3260000000000001,"5min":8.7240000000000002,"15min":8.7240000000000002},"last":3.9449999999999998},{"interface":"front","average":{"1min":0.63,"5min":1.456,"15min":0.96299999999999997},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.5820000000000001,"5min":10.1,"15min":10.1},"last":4.0039999999999996}]},{"osd":11,"last update":"Fri Feb 9 17:31:08 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":1.304,"15min":0.97499999999999998},"min":{"1min":0.39200000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":2.8940000000000001,"5min":8.6430000000000007,"15min":8.6430000000000007},"last":4.0330000000000004},{"interface":"front","average":{"1min":0.76600000000000001,"5min":1.242,"15min":0.98399999999999999},"min":{"1min":0.439,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.8640000000000001,"5min":9.298,"15min":9.298},"last":1.153}]}]},{"osd":9,"up_from":77,"seq":330712481911,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1608044,"kb_used_data":1454692,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":92137108,"statfs":{"total":95995035648,"available":94348398592,"internally_reserved":0,"allocated":1489604608,"data_stored":1489297664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":157022656},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:28:03 2024","interfaces":[{"interface":"back","average":{"1min":1.9510000000000001,"5min":1.454,"15min":0.85799999999999998},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":8.4499999999999993,"5min":17.495999999999999,"15min":17.495999999999999},"last":2.7360000000000002},{"interface":"front","average":{"1min":1.905,"5min":1.399,"15min":0.83499999999999996},"min":{"1min":0.32200000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":7.5990000000000002,"5min":15.401999999999999,"15min":15.401999999999999},"last":1.5449999999999999}]},{"osd":1,"last update":"Fri Feb 9 17:31:14 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":1.851,"15min":1.1020000000000001},"min":{"1min":0.24199999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":2.794,"5min":16.527999999999999,"15min":16.527999999999999},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.51500000000000001,"5min":1.6499999999999999,"15min":0.98599999999999999},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":1.863,"5min":17.771999999999998,"15min":17.771999999999998},"last":0.222}]},{"osd":2,"last update":"Fri Feb 9 17:31:14 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":1.514,"15min":0.97599999999999998},"min":{"1min":0.30599999999999999,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":2.5259999999999998,"5min":13.878,"15min":13.878},"last":6.0570000000000004},{"interface":"front","average":{"1min":0.53200000000000003,"5min":1.8,"15min":1.0569999999999999},"min":{"1min":0.28599999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.798,"5min":13.69,"15min":13.69},"last":0.35599999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:31:14 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":1.357,"15min":0.92200000000000004},"min":{"1min":0.27400000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.7749999999999999,"5min":9.2889999999999997,"15min":9.2889999999999997},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":1.8360000000000001,"15min":1.095},"min":{"1min":0.32900000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.964,"5min":11.236000000000001,"15min":11.236000000000001},"last":0.34699999999999998}]},{"osd":4,"last update":"Fri Feb 9 17:31:14 2024","interfaces":[{"interface":"back","average":{"1min":2.2650000000000001,"5min":1.2709999999999999,"15min":0.92200000000000004},"min":{"1min":0.33000000000000002,"5min":0.309,"15min":0.309},"max":{"1min":22.577999999999999,"5min":22.577999999999999,"15min":22.577999999999999},"last":0.433},{"interface":"front","average":{"1min":1.9650000000000001,"5min":1.0720000000000001,"15min":0.84699999999999998},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":22.399999999999999,"5min":22.399999999999999,"15min":22.399999999999999},"last":0.28799999999999998}]},{"osd":5,"last update":"Fri Feb 9 17:31:14 2024","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":1.01,"15min":0.84599999999999997},"min":{"1min":0.32400000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":2.899,"5min":8.2460000000000004,"15min":8.2460000000000004},"last":6.1470000000000002},{"interface":"front","average":{"1min":1.244,"5min":0.95099999999999996,"15min":0.80800000000000005},"min":{"1min":0.34899999999999998,"5min":0.315,"15min":0.315},"max":{"1min":3.0379999999999998,"5min":4.1829999999999998,"15min":5.665},"last":0.32000000000000001}]},{"osd":6,"last update":"Fri Feb 9 17:31:14 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":1.0960000000000001,"15min":0.90200000000000002},"min":{"1min":0.35999999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":2.1549999999999998,"5min":8.2959999999999994,"15min":8.2959999999999994},"last":6.2130000000000001},{"interface":"front","average":{"1min":0.878,"5min":1.1479999999999999,"15min":0.89200000000000002},"min":{"1min":0.38200000000000001,"5min":0.36199999999999999,"15min":0.35099999999999998},"max":{"1min":2.1859999999999999,"5min":8.3130000000000006,"15min":8.3130000000000006},"last":6.242}]},{"osd":7,"last update":"Fri Feb 9 17:31:14 2024","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.083,"15min":0.878},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":7.0780000000000003,"5min":8.0939999999999994,"15min":8.0939999999999994},"last":0.49299999999999999},{"interface":"front","average":{"1min":1.534,"5min":1.1870000000000001,"15min":0.91900000000000004},"min":{"1min":0.41399999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":5.1829999999999998,"5min":5.1829999999999998,"15min":6.056},"last":0.52000000000000002}]},{"osd":8,"last update":"Fri Feb 9 17:30:22 2024","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":1.4019999999999999,"15min":1.3819999999999999},"min":{"1min":0.40699999999999997,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":16.209,"5min":16.209,"15min":16.209},"last":6.1079999999999997},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.4239999999999999,"15min":1.4219999999999999},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":16.263000000000002,"5min":16.263000000000002,"15min":16.263000000000002},"last":0.374}]},{"osd":10,"last update":"Fri Feb 9 17:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":1.206,"15min":0.86299999999999999},"min":{"1min":0.44800000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.0069999999999999,"5min":8.3480000000000008,"15min":8.3810000000000002},"last":9.5090000000000003},{"interface":"front","average":{"1min":0.67000000000000004,"5min":1.349,"15min":0.91000000000000003},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.36099999999999999},"max":{"1min":0.879,"5min":8.0670000000000002,"15min":8.0670000000000002},"last":0.55600000000000005}]},{"osd":11,"last update":"Fri Feb 9 17:30:48 2024","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.278,"15min":1.2609999999999999},"min":{"1min":0.48199999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":3.8679999999999999,"5min":8.2759999999999998,"15min":8.2759999999999998},"last":6.181},{"interface":"front","average":{"1min":0.75800000000000001,"5min":1.034,"15min":1.0249999999999999},"min":{"1min":0.39200000000000002,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":2.1509999999999998,"5min":6.2060000000000004,"15min":6.2060000000000004},"last":0.41699999999999998}]}]},{"osd":8,"up_from":171,"seq":734439407654,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1059244,"kb_used_data":968036,"kb_used_omap":212,"kb_used_meta":90987,"kb_avail":92685908,"statfs":{"total":95995035648,"available":94910369792,"internally_reserved":0,"allocated":991268864,"data_stored":990961940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217579,"internal_metadata":93171221},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:31:18 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":1.3740000000000001,"15min":1.665},"min":{"1min":0.26800000000000002,"5min":0.245,"15min":0.245},"max":{"1min":5.0049999999999999,"5min":6.8840000000000003,"15min":6.8840000000000003},"last":0.28299999999999997},{"interface":"front","average":{"1min":1.4390000000000001,"5min":1.399,"15min":1.5589999999999999},"min":{"1min":0.318,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":7.7969999999999997,"5min":7.7969999999999997,"15min":7.7969999999999997},"last":0.30099999999999999}]},{"osd":1,"last update":"Fri Feb 9 17:31:18 2024","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":1.1559999999999999,"15min":1.3169999999999999},"min":{"1min":0.29599999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":5.1589999999999998,"5min":7.8250000000000002,"15min":7.8250000000000002},"last":0.41399999999999998},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.8400000000000001,"15min":2.278},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":3.093,"5min":11.101000000000001,"15min":11.101000000000001},"last":0.36299999999999999}]},{"osd":2,"last update":"Fri Feb 9 17:31:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.2969999999999999,"15min":1.454},"min":{"1min":0.25600000000000001,"5min":0.255,"15min":0.255},"max":{"1min":4.9189999999999996,"5min":5.4450000000000003,"15min":5.4450000000000003},"last":2.4260000000000002},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.5589999999999999,"15min":1.829},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":4.9050000000000002,"5min":7.1109999999999998,"15min":7.1109999999999998},"last":2.4079999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:31:18 2024","interfaces":[{"interface":"back","average":{"1min":2.113,"5min":2.077,"15min":2.3799999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":10.209,"5min":12.505000000000001,"15min":12.505000000000001},"last":3.3380000000000001},{"interface":"front","average":{"1min":1.72,"5min":2.3370000000000002,"15min":2.8149999999999999},"min":{"1min":0.434,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":5.5140000000000002,"5min":10.92,"15min":10.92},"last":0.39900000000000002}]},{"osd":4,"last update":"Fri Feb 9 17:31:18 2024","interfaces":[{"interface":"back","average":{"1min":2.9729999999999999,"5min":1.1639999999999999,"15min":0.85599999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":37.343000000000004,"5min":37.343000000000004,"15min":37.343000000000004},"last":0.38900000000000001},{"interface":"front","average":{"1min":2.9830000000000001,"5min":1.419,"15min":1.1870000000000001},"min":{"1min":0.374,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":37.875,"5min":37.875,"15min":37.875},"last":0.42899999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:31:18 2024","interfaces":[{"interface":"back","average":{"1min":3.4660000000000002,"5min":1.6579999999999999,"15min":1.4610000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":37.356999999999999,"5min":37.356999999999999,"15min":37.356999999999999},"last":1.292},{"interface":"front","average":{"1min":2.9860000000000002,"5min":1.2250000000000001,"15min":0.94299999999999995},"min":{"1min":0.36499999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":37.25,"5min":37.25,"15min":37.25},"last":3.3730000000000002}]},{"osd":7,"last update":"Fri Feb 9 17:31:18 2024","interfaces":[{"interface":"back","average":{"1min":3.258,"5min":1.4430000000000001,"15min":1.2},"min":{"1min":0.437,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":37.585000000000001,"5min":37.585000000000001,"15min":37.585000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":3.1560000000000001,"5min":1.605,"15min":1.4630000000000001},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":37.918999999999997,"5min":37.918999999999997,"15min":37.918999999999997},"last":0.55500000000000005}]},{"osd":9,"last update":"Fri Feb 9 17:31:18 2024","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":1.075,"15min":1.202},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":3.7970000000000002,"5min":6.9269999999999996,"15min":6.9269999999999996},"last":3.1080000000000001},{"interface":"front","average":{"1min":0.97999999999999998,"5min":1.115,"15min":1.2450000000000001},"min":{"1min":0.33900000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":2.8620000000000001,"5min":7.7409999999999997,"15min":7.7409999999999997},"last":0.48199999999999998}]},{"osd":10,"last update":"Fri Feb 9 17:31:21 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":1.1100000000000001,"15min":1.2609999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":3.0920000000000001,"5min":5.649,"15min":5.649},"last":1.246},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.96599999999999997,"15min":1.0620000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":2.8860000000000001,"5min":5.1070000000000002,"15min":5.1070000000000002},"last":0.56599999999999995}]},{"osd":11,"last update":"Fri Feb 9 17:31:18 2024","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.1890000000000001,"15min":1.3149999999999999},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":4.0860000000000003,"5min":7.6870000000000003,"15min":7.6870000000000003},"last":1.2709999999999999},{"interface":"front","average":{"1min":0.74399999999999999,"5min":1.325,"15min":1.581},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.7829999999999999,"5min":7.6609999999999996,"15min":7.6609999999999996},"last":0.46500000000000002}]}]},{"osd":7,"up_from":57,"seq":244813136000,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2090228,"kb_used_data":1936876,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":91654924,"statfs":{"total":95995035648,"available":93854642176,"internally_reserved":0,"allocated":1983361024,"data_stored":1983050548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":157022666},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:30:46 2024","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.1479999999999999,"15min":0.89000000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.26600000000000001},"max":{"1min":3.9790000000000001,"5min":10.776,"15min":10.776},"last":2.585},{"interface":"front","average":{"1min":0.97799999999999998,"5min":1.5840000000000001,"15min":1.008},"min":{"1min":0.373,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":3.8820000000000001,"5min":18.896000000000001,"15min":18.896000000000001},"last":2.5510000000000002}]},{"osd":1,"last update":"Fri Feb 9 17:27:46 2024","interfaces":[{"interface":"back","average":{"1min":2.262,"5min":2.262,"15min":2.262},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":4.8470000000000004,"5min":4.8470000000000004,"15min":4.8470000000000004},"last":3.9470000000000001},{"interface":"front","average":{"1min":1.8009999999999999,"5min":1.8009999999999999,"15min":1.8009999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.9589999999999996,"5min":6.9589999999999996,"15min":6.9589999999999996},"last":3.6669999999999998}]},{"osd":2,"last update":"Fri Feb 9 17:30:20 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":1.556,"15min":0.92300000000000004},"min":{"1min":0.36199999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.93999999999999995,"5min":11.093999999999999,"15min":11.093999999999999},"last":2.5630000000000002},{"interface":"front","average":{"1min":0.65200000000000002,"5min":1.629,"15min":0.98799999999999999},"min":{"1min":0.38500000000000001,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.236,"5min":11.013,"15min":11.013},"last":2.8319999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:30:20 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":1.7450000000000001,"15min":1.012},"min":{"1min":0.309,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.716,"5min":27.719000000000001,"15min":27.719000000000001},"last":2.6640000000000001},{"interface":"front","average":{"1min":0.65800000000000003,"5min":1.71,"15min":0.98799999999999999},"min":{"1min":0.39400000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.96099999999999997,"5min":8.8629999999999995,"15min":8.8629999999999995},"last":2.5169999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:30:20 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.67300000000000004,"15min":0.58799999999999997},"min":{"1min":0.313,"5min":0.248,"15min":0.248},"max":{"1min":1.653,"5min":5.9880000000000004,"15min":5.9880000000000004},"last":2.9350000000000001},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.67300000000000004,"15min":0.58399999999999996},"min":{"1min":0.32200000000000001,"5min":0.253,"15min":0.22800000000000001},"max":{"1min":1.6759999999999999,"5min":4.7229999999999999,"15min":4.7229999999999999},"last":2.613}]},{"osd":5,"last update":"Fri Feb 9 17:30:20 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.65700000000000003,"15min":0.58099999999999996},"min":{"1min":0.29699999999999999,"5min":0.27900000000000003,"15min":0.21299999999999999},"max":{"1min":1.9930000000000001,"5min":4.4779999999999998,"15min":4.4779999999999998},"last":2.5329999999999999},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.61499999999999999,"15min":0.628},"min":{"1min":0.36899999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.91100000000000003,"5min":3.7730000000000001,"15min":3.7730000000000001},"last":2.7360000000000002}]},{"osd":6,"last update":"Fri Feb 9 17:30:20 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.81000000000000005,"15min":0.67000000000000004},"min":{"1min":0.40400000000000003,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.1759999999999999,"5min":6.0910000000000002,"15min":6.0910000000000002},"last":3.488},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.71099999999999997,"15min":0.65300000000000002},"min":{"1min":0.35799999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.1399999999999999,"5min":3.9510000000000001,"15min":3.9510000000000001},"last":3.4470000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:30:20 2024","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":1.3460000000000001,"15min":1.4199999999999999},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.8029999999999999,"5min":7.9420000000000002,"15min":7.9420000000000002},"last":2.641},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.92200000000000004,"15min":0.93999999999999995},"min":{"1min":0.45800000000000002,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.2889999999999999,"5min":3.6920000000000002,"15min":3.6920000000000002},"last":2.8050000000000002}]},{"osd":9,"last update":"Fri Feb 9 17:31:07 2024","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.5209999999999999,"15min":1.04},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":5.5129999999999999,"5min":41.286999999999999,"15min":41.286999999999999},"last":2.6970000000000001},{"interface":"front","average":{"1min":1.919,"5min":1.5620000000000001,"15min":1.0409999999999999},"min":{"1min":0.46000000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":9.3109999999999999,"5min":41.234999999999999,"15min":41.234999999999999},"last":2.7639999999999998}]},{"osd":10,"last update":"Fri Feb 9 17:30:31 2024","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":1.1100000000000001,"15min":0.94599999999999995},"min":{"1min":0.52600000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.27,"5min":8.4659999999999993,"15min":8.4659999999999993},"last":2.625},{"interface":"front","average":{"1min":0.94999999999999996,"5min":1.1519999999999999,"15min":0.93400000000000005},"min":{"1min":0.52000000000000002,"5min":0.439,"15min":0.439},"max":{"1min":1.8839999999999999,"5min":6.1479999999999997,"15min":6.1479999999999997},"last":2.7210000000000001}]},{"osd":11,"last update":"Fri Feb 9 17:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.5149999999999999,"5min":1.2509999999999999,"15min":1.0089999999999999},"min":{"1min":0.443,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":5.548,"5min":6.173,"15min":6.173},"last":2.9049999999999998},{"interface":"front","average":{"1min":1.101,"5min":1.0880000000000001,"15min":0.94399999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":4.0229999999999997,"5min":8.1799999999999997,"15min":8.1799999999999997},"last":2.6499999999999999}]}]},{"osd":6,"up_from":49,"seq":210453397637,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2504052,"kb_used_data":2392364,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":91241100,"statfs":{"total":95995035648,"available":93430886400,"internally_reserved":0,"allocated":2449780736,"data_stored":2449470260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":114358726},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:30:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.4199999999999999,"15min":0.94599999999999995},"min":{"1min":0.27100000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":4.5179999999999998,"5min":11.582000000000001,"15min":24.544},"last":0.26900000000000002},{"interface":"front","average":{"1min":1.593,"5min":1.3620000000000001,"15min":0.92400000000000004},"min":{"1min":0.248,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":4.4470000000000001,"5min":5.7130000000000001,"15min":5.7130000000000001},"last":0.217}]},{"osd":1,"last update":"Fri Feb 9 17:30:51 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.446,"15min":0.96499999999999997},"min":{"1min":0.28599999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":5.9400000000000004,"5min":9.4469999999999992,"15min":23.954000000000001},"last":2.1259999999999999},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.4990000000000001,"15min":0.995},"min":{"1min":0.38400000000000001,"5min":0.318,"15min":0.318},"max":{"1min":5.9729999999999999,"5min":9.4930000000000003,"15min":23.984999999999999},"last":2.1040000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:30:51 2024","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.706,"15min":1.0649999999999999},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.8769999999999998,"5min":10.682,"15min":24.329999999999998},"last":0.39600000000000002},{"interface":"front","average":{"1min":1.222,"5min":1.7589999999999999,"15min":1.036},"min":{"1min":0.32100000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":4.4630000000000001,"5min":11.553000000000001,"15min":12.906000000000001},"last":0.48899999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:30:51 2024","interfaces":[{"interface":"back","average":{"1min":2.9140000000000001,"5min":2.0049999999999999,"15min":1.107},"min":{"1min":0.33400000000000002,"5min":0.248,"15min":0.248},"max":{"1min":32.545000000000002,"5min":32.545000000000002,"15min":32.545000000000002},"last":4.9020000000000001},{"interface":"front","average":{"1min":2.5350000000000001,"5min":1.681,"15min":1.052},"min":{"1min":0.32100000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":32.670999999999999,"5min":32.670999999999999,"15min":32.670999999999999},"last":0.41499999999999998}]},{"osd":4,"last update":"Fri Feb 9 17:30:51 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":0.79200000000000004,"15min":0.60399999999999998},"min":{"1min":0.27700000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":8.7110000000000003,"5min":8.7110000000000003,"15min":8.7110000000000003},"last":2.0369999999999999},{"interface":"front","average":{"1min":1.468,"5min":1.0129999999999999,"15min":0.68799999999999994},"min":{"1min":0.34499999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":8.8209999999999997,"5min":8.8209999999999997,"15min":8.8209999999999997},"last":1.768}]},{"osd":5,"last update":"Fri Feb 9 17:30:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":0.81799999999999995,"15min":0.63400000000000001},"min":{"1min":0.40400000000000003,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":4.4299999999999997,"5min":5.5720000000000001,"15min":5.5720000000000001},"last":1.4630000000000001},{"interface":"front","average":{"1min":1.2729999999999999,"5min":0.85399999999999998,"15min":0.64300000000000002},"min":{"1min":0.30199999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":4.2519999999999998,"5min":7.601,"15min":7.601},"last":4.9509999999999996}]},{"osd":7,"last update":"Fri Feb 9 17:30:27 2024","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.89100000000000001,"15min":0.69599999999999995},"min":{"1min":0.42399999999999999,"5min":0.30599999999999999,"15min":0.29499999999999998},"max":{"1min":4.8529999999999998,"5min":4.9660000000000002,"15min":4.9660000000000002},"last":0.54500000000000004},{"interface":"front","average":{"1min":1.032,"5min":0.88100000000000001,"15min":0.73999999999999999},"min":{"1min":0.35099999999999998,"5min":0.309,"15min":0.309},"max":{"1min":4.2759999999999998,"5min":11.512,"15min":11.512},"last":0.94399999999999995}]},{"osd":8,"last update":"Fri Feb 9 17:26:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":0.876,"15min":0.79600000000000004},"min":{"1min":0.39900000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":4.0339999999999998,"5min":4.4880000000000004,"15min":4.4880000000000004},"last":0.434},{"interface":"front","average":{"1min":1.0469999999999999,"5min":0.88600000000000001,"15min":0.78300000000000003},"min":{"1min":0.375,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":3.6539999999999999,"5min":4.8570000000000002,"15min":4.8570000000000002},"last":0.45000000000000001}]},{"osd":9,"last update":"Fri Feb 9 17:31:04 2024","interfaces":[{"interface":"back","average":{"1min":1.617,"5min":1.1890000000000001,"15min":0.95699999999999996},"min":{"1min":0.39800000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":6.2110000000000003,"5min":10.803000000000001,"15min":10.803000000000001},"last":0.50800000000000001},{"interface":"front","average":{"1min":1.6659999999999999,"5min":1.2969999999999999,"15min":1.006},"min":{"1min":0.42099999999999999,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":6.3109999999999999,"5min":6.3109999999999999,"15min":6.3109999999999999},"last":0.52400000000000002}]},{"osd":10,"last update":"Fri Feb 9 17:30:27 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":1.3560000000000001,"15min":0.995},"min":{"1min":0.56299999999999994,"5min":0.40699999999999997,"15min":0.35999999999999999},"max":{"1min":1.5309999999999999,"5min":9.5399999999999991,"15min":9.5399999999999991},"last":0.626},{"interface":"front","average":{"1min":1.026,"5min":1.2270000000000001,"15min":0.95899999999999996},"min":{"1min":0.48699999999999999,"5min":0.42299999999999999,"15min":0.39200000000000002},"max":{"1min":3.5339999999999998,"5min":9.5210000000000008,"15min":9.5210000000000008},"last":2.077}]},{"osd":11,"last update":"Fri Feb 9 17:30:51 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":1.282,"15min":1.52},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.6799999999999999,"5min":5.1200000000000001,"15min":5.1200000000000001},"last":0.59399999999999997},{"interface":"front","average":{"1min":1.238,"5min":1.4430000000000001,"15min":1.698},"min":{"1min":0.44700000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":4.4829999999999997,"5min":6.3200000000000003,"15min":6.3200000000000003},"last":1.8340000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575007,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2062636,"kb_used_data":1892644,"kb_used_omap":1,"kb_used_meta":169982,"kb_avail":91682516,"statfs":{"total":95995035648,"available":93882896384,"internally_reserved":0,"allocated":1938067456,"data_stored":1937760532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":174062021},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:31:01 2024","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.869,"15min":0.94499999999999995},"min":{"1min":0.214,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":8.2270000000000003,"5min":12.492000000000001,"15min":12.492000000000001},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.952,"15min":0.97699999999999998},"min":{"1min":0.26800000000000002,"5min":0.214,"15min":0.21299999999999999},"max":{"1min":6.3220000000000001,"5min":9.8439999999999994,"15min":9.8439999999999994},"last":5.2089999999999996}]},{"osd":2,"last update":"Fri Feb 9 17:30:25 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":2.3029999999999999,"15min":1.056},"min":{"1min":0.27100000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":3.504,"5min":12.01,"15min":12.01},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.73899999999999999,"5min":2.706,"15min":1.21},"min":{"1min":0.30099999999999999,"5min":0.255,"15min":0.255},"max":{"1min":4.0229999999999997,"5min":17.459,"15min":17.459},"last":2.2330000000000001}]},{"osd":3,"last update":"Fri Feb 9 17:30:48 2024","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":2.7650000000000001,"15min":1.3160000000000001},"min":{"1min":0.27400000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":8.1899999999999995,"5min":16.088000000000001,"15min":16.088000000000001},"last":5.2359999999999998},{"interface":"front","average":{"1min":1.0609999999999999,"5min":2.1949999999999998,"15min":1.1140000000000001},"min":{"1min":0.37,"5min":0.222,"15min":0.222},"max":{"1min":4.5869999999999997,"5min":12.378,"15min":12.378},"last":1.2210000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:28:03 2024","interfaces":[{"interface":"back","average":{"1min":1.593,"5min":1.323,"15min":0.81899999999999995},"min":{"1min":0.28199999999999997,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":7.3559999999999999,"5min":7.7210000000000001,"15min":7.7210000000000001},"last":0.33000000000000002},{"interface":"front","average":{"1min":2.359,"5min":1.5209999999999999,"15min":0.89200000000000002},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":9.3979999999999997,"5min":9.5280000000000005,"15min":9.5280000000000005},"last":0.47099999999999997}]},{"osd":5,"last update":"Fri Feb 9 17:30:35 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":1.8540000000000001,"15min":1.052},"min":{"1min":0.41099999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":2.645,"5min":7.3760000000000003,"15min":7.7679999999999998},"last":0.505},{"interface":"front","average":{"1min":0.746,"5min":1.258,"15min":0.86099999999999999},"min":{"1min":0.439,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.7869999999999999,"5min":7.1799999999999997,"15min":7.1799999999999997},"last":1.256}]},{"osd":6,"last update":"Fri Feb 9 17:31:01 2024","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.996,"15min":1.1659999999999999},"min":{"1min":0.34200000000000003,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":4.1070000000000002,"5min":10.275,"15min":10.275},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.905,"15min":1.1299999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":2.4569999999999999,"5min":7.0510000000000002,"15min":8.0229999999999997},"last":2.1019999999999999}]},{"osd":7,"last update":"Fri Feb 9 17:30:27 2024","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":2.1190000000000002,"15min":1.1890000000000001},"min":{"1min":0.49199999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":3.7789999999999999,"5min":12.058,"15min":12.058},"last":1.9590000000000001},{"interface":"front","average":{"1min":0.69499999999999995,"5min":2.1219999999999999,"15min":1.22},"min":{"1min":0.38800000000000001,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.94599999999999995,"5min":12.233000000000001,"15min":12.233000000000001},"last":5.1829999999999998}]},{"osd":8,"last update":"Fri Feb 9 17:30:22 2024","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":1.431,"15min":1.5189999999999999},"min":{"1min":0.51900000000000002,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":3.714,"5min":3.919,"15min":3.919},"last":1.3340000000000001},{"interface":"front","average":{"1min":0.81399999999999995,"5min":2.1150000000000002,"15min":2.3319999999999999},"min":{"1min":0.56000000000000005,"5min":0.499,"15min":0.499},"max":{"1min":1.145,"5min":6.7530000000000001,"15min":6.7530000000000001},"last":5.1369999999999996}]},{"osd":9,"last update":"Fri Feb 9 17:31:06 2024","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.9870000000000001,"15min":1.2470000000000001},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":2.4889999999999999,"5min":7.3140000000000001,"15min":7.8010000000000002},"last":0.53400000000000003},{"interface":"front","average":{"1min":1.415,"5min":2.0979999999999999,"15min":1.2789999999999999},"min":{"1min":0.38700000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":7.6849999999999996,"5min":12.117000000000001,"15min":12.117000000000001},"last":2.1349999999999998}]},{"osd":10,"last update":"Fri Feb 9 17:30:35 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":1.5549999999999999,"15min":1.0649999999999999},"min":{"1min":0.501,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":3.1909999999999998,"5min":7.149,"15min":7.149},"last":1.3640000000000001},{"interface":"front","average":{"1min":1.246,"5min":2.2240000000000002,"15min":1.2869999999999999},"min":{"1min":0.40899999999999997,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":3.6920000000000002,"5min":12.087999999999999,"15min":12.087999999999999},"last":1.9199999999999999}]},{"osd":11,"last update":"Fri Feb 9 17:30:46 2024","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":2.3340000000000001,"15min":2.5640000000000001},"min":{"1min":0.437,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":3.5619999999999998,"5min":12.169,"15min":12.169},"last":2.1720000000000002},{"interface":"front","average":{"1min":1.2889999999999999,"5min":2.3010000000000002,"15min":2.3420000000000001},"min":{"1min":0.45200000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":3.5379999999999998,"5min":12.144,"15min":12.144},"last":2.21}]}]},{"osd":0,"up_from":8,"seq":34359738531,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2506916,"kb_used_data":2345060,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":91238236,"statfs":{"total":95995035648,"available":93427953664,"internally_reserved":0,"allocated":2401341440,"data_stored":2401034516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 9 17:30:50 2024","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":2.0720000000000001,"15min":0.95399999999999996},"min":{"1min":0.23100000000000001,"5min":0.221,"15min":0.221},"max":{"1min":4.2190000000000003,"5min":14.166,"15min":14.166},"last":0.29799999999999999},{"interface":"front","average":{"1min":1.4319999999999999,"5min":2.5649999999999999,"15min":1.1779999999999999},"min":{"1min":0.29599999999999999,"5min":0.215,"15min":0.215},"max":{"1min":7.5629999999999997,"5min":20.177,"15min":20.177},"last":0.32900000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:27:44 2024","interfaces":[{"interface":"back","average":{"1min":3.976,"5min":3.976,"15min":3.976},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":13.066000000000001,"5min":13.066000000000001,"15min":13.066000000000001},"last":3.0569999999999999},{"interface":"front","average":{"1min":3.968,"5min":3.968,"15min":3.968},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":13.193,"5min":13.193,"15min":13.193},"last":10.872999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:30:40 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":2.2370000000000001,"15min":1.0900000000000001},"min":{"1min":0.246,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":3.7909999999999999,"5min":13.234,"15min":13.234},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.995,"5min":2.2040000000000002,"15min":1.05},"min":{"1min":0.22700000000000001,"5min":0.223,"15min":0.223},"max":{"1min":7.3810000000000002,"5min":16.984999999999999,"15min":16.984999999999999},"last":0.34999999999999998}]},{"osd":4,"last update":"Fri Feb 9 17:31:11 2024","interfaces":[{"interface":"back","average":{"1min":2.1989999999999998,"5min":1.877,"15min":1.0700000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.253},"max":{"1min":13.803000000000001,"5min":13.803000000000001,"15min":13.803000000000001},"last":4.133},{"interface":"front","average":{"1min":2.2519999999999998,"5min":1.946,"15min":1.1000000000000001},"min":{"1min":0.35199999999999998,"5min":0.35099999999999998,"15min":0.32200000000000001},"max":{"1min":13.861000000000001,"5min":13.861000000000001,"15min":13.861000000000001},"last":4.1619999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:30:29 2024","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":1.8700000000000001,"15min":1.0449999999999999},"min":{"1min":0.34799999999999998,"5min":0.29999999999999999,"15min":0.28999999999999998},"max":{"1min":3.8490000000000002,"5min":9.8569999999999993,"15min":9.8569999999999993},"last":1},{"interface":"front","average":{"1min":0.84199999999999997,"5min":1.958,"15min":1.077},"min":{"1min":0.40899999999999997,"5min":0.314,"15min":0.26700000000000002},"max":{"1min":3.9430000000000001,"5min":9.9049999999999994,"15min":9.9049999999999994},"last":1.536}]},{"osd":6,"last update":"Fri Feb 9 17:30:59 2024","interfaces":[{"interface":"back","average":{"1min":1.78,"5min":2.012,"15min":1.119},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.315},"max":{"1min":6.1200000000000001,"5min":11.933,"15min":11.933},"last":0.35899999999999999},{"interface":"front","average":{"1min":1.7070000000000001,"5min":1.829,"15min":1.099},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":6.157,"5min":10.91,"15min":10.91},"last":0.40699999999999997}]},{"osd":7,"last update":"Fri Feb 9 17:31:17 2024","interfaces":[{"interface":"back","average":{"1min":2.2970000000000002,"5min":1.849,"15min":1.135},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":10.335000000000001,"5min":11.913,"15min":11.913},"last":1.556},{"interface":"front","average":{"1min":2.573,"5min":2.0630000000000002,"15min":1.1739999999999999},"min":{"1min":0.33900000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":10.358000000000001,"5min":16.765999999999998,"15min":16.765999999999998},"last":1.5660000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":1.6559999999999999,"15min":1.8200000000000001},"min":{"1min":0.38100000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.92200000000000004,"5min":8.952,"15min":8.952},"last":0.502},{"interface":"front","average":{"1min":0.68200000000000005,"5min":1.504,"15min":1.641},"min":{"1min":0.45100000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.95599999999999996,"5min":16.41,"15min":16.41},"last":0.60499999999999998}]},{"osd":9,"last update":"Fri Feb 9 17:31:11 2024","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":1.895,"15min":1.1950000000000001},"min":{"1min":0.373,"5min":0.312,"15min":0.312},"max":{"1min":5.508,"5min":10.973000000000001,"15min":10.973000000000001},"last":0.442},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.9259999999999999,"15min":1.171},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36199999999999999},"max":{"1min":2.8980000000000001,"5min":16.82,"15min":16.82},"last":0.53600000000000003}]},{"osd":10,"last update":"Fri Feb 9 17:30:29 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":2.069,"15min":1.238},"min":{"1min":0.45100000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":2.0270000000000001,"5min":16.876000000000001,"15min":16.876000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":1.1739999999999999,"5min":2.0179999999999998,"15min":1.2030000000000001},"min":{"1min":0.432,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":5.0940000000000003,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":0.57199999999999995}]},{"osd":11,"last update":"Fri Feb 9 17:30:59 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":1.7629999999999999,"15min":1.1399999999999999},"min":{"1min":0.45800000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":3.9279999999999999,"5min":10.997,"15min":10.997},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.98799999999999999,"5min":2.1760000000000002,"15min":1.2529999999999999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.443},"max":{"1min":3.9590000000000001,"5min":13.086,"15min":13.086},"last":0.55200000000000005}]}]},{"osd":2,"up_from":18,"seq":77309411481,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":582572,"kb_used_data":475044,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93162580,"statfs":{"total":95995035648,"available":95398481920,"internally_reserved":0,"allocated":486445056,"data_stored":486138132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":2.0009999999999999,"15min":0.92800000000000005},"min":{"1min":0.26100000000000001,"5min":0.23699999999999999,"15min":0.221},"max":{"1min":0.89400000000000002,"5min":17.251000000000001,"15min":17.251000000000001},"last":1.784},{"interface":"front","average":{"1min":0.48299999999999998,"5min":2.391,"15min":1.0569999999999999},"min":{"1min":0.26600000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":1.5429999999999999,"5min":23.097999999999999,"15min":23.097999999999999},"last":0.44800000000000001}]},{"osd":1,"last update":"Fri Feb 9 17:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":2.363,"15min":1.0629999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":1.099,"5min":27.289000000000001,"15min":27.289000000000001},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.47899999999999998,"5min":2.569,"15min":1.141},"min":{"1min":0.254,"5min":0.25,"15min":0.25},"max":{"1min":0.93600000000000005,"5min":37.585999999999999,"15min":37.585999999999999},"last":1.8859999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:30:47 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":2.5470000000000002,"15min":1.1819999999999999},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":1.377,"5min":37.634999999999998,"15min":37.634999999999998},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":2.3759999999999999,"15min":1.091},"min":{"1min":0.26400000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.8580000000000001,"5min":38.924999999999997,"15min":38.924999999999997},"last":0.27500000000000002}]},{"osd":4,"last update":"Fri Feb 9 17:30:53 2024","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.607,"15min":0.98499999999999999},"min":{"1min":0.41799999999999998,"5min":0.30299999999999999,"15min":0.254},"max":{"1min":3.5510000000000002,"5min":7.6399999999999997,"15min":8.8089999999999993},"last":3.9460000000000002},{"interface":"front","average":{"1min":1.425,"5min":1.427,"15min":0.92800000000000005},"min":{"1min":0.39500000000000002,"5min":0.315,"15min":0.30399999999999999},"max":{"1min":9.1120000000000001,"5min":13.305999999999999,"15min":13.305999999999999},"last":1.4350000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:30:27 2024","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":1.724,"15min":1},"min":{"1min":0.47199999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":3.508,"5min":8.6859999999999999,"15min":8.6859999999999999},"last":1.952},{"interface":"front","average":{"1min":0.66800000000000004,"5min":1.379,"15min":0.872},"min":{"1min":0.314,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.798,"5min":13.278,"15min":13.278},"last":4.3890000000000002}]},{"osd":6,"last update":"Fri Feb 9 17:30:53 2024","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.7749999999999999,"15min":1.0780000000000001},"min":{"1min":0.39100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":3.5800000000000001,"5min":7.6159999999999997,"15min":8.8399999999999999},"last":1.9770000000000001},{"interface":"front","average":{"1min":1.21,"5min":1.786,"15min":1.0920000000000001},"min":{"1min":0.53900000000000003,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":3.7650000000000001,"5min":8.2899999999999991,"15min":8.859},"last":3.4700000000000002}]},{"osd":7,"last update":"Fri Feb 9 17:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":1.8240000000000001,"15min":1.0509999999999999},"min":{"1min":0.46600000000000003,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.22,"5min":14.541,"15min":14.541},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.78900000000000003,"5min":2.0369999999999999,"15min":1.141},"min":{"1min":0.504,"5min":0.41299999999999998,"15min":0.40500000000000003},"max":{"1min":1.266,"5min":8.7870000000000008,"15min":8.7870000000000008},"last":2.0049999999999999}]},{"osd":8,"last update":"Fri Feb 9 17:26:31 2024","interfaces":[{"interface":"back","average":{"1min":1.7529999999999999,"5min":1,"15min":0.84599999999999997},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":8.8989999999999991,"5min":8.8989999999999991,"15min":8.8989999999999991},"last":5.2350000000000003},{"interface":"front","average":{"1min":2.4969999999999999,"5min":1.1080000000000001,"15min":0.90600000000000003},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":9.4190000000000005,"5min":9.4190000000000005,"15min":9.4190000000000005},"last":0.57499999999999996}]},{"osd":9,"last update":"Fri Feb 9 17:31:10 2024","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.8080000000000001,"15min":1.1639999999999999},"min":{"1min":0.51300000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":6.0039999999999996,"5min":7.7089999999999996,"15min":8.9190000000000005},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.77000000000000002,"5min":1.49,"15min":1.0649999999999999},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":2.0350000000000001,"5min":13.218,"15min":13.218},"last":0.71199999999999997}]},{"osd":10,"last update":"Fri Feb 9 17:30:35 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":1.6439999999999999,"15min":1.0620000000000001},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.5,"5min":13.244999999999999,"15min":13.244999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.81200000000000006,"5min":1.752,"15min":1.0940000000000001},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42099999999999999},"max":{"1min":1.4710000000000001,"5min":14.499000000000001,"15min":14.499000000000001},"last":0.621}]},{"osd":11,"last update":"Fri Feb 9 17:30:49 2024","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":1.6759999999999999,"15min":2.0790000000000002},"min":{"1min":0.53200000000000003,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":3.605,"5min":13.186,"15min":13.186},"last":1.9199999999999999},{"interface":"front","average":{"1min":0.86399999999999999,"5min":1.8420000000000001,"15min":2.0099999999999998},"min":{"1min":0.51300000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.611,"5min":7.7919999999999998,"15min":7.7919999999999998},"last":0.503}]}]},{"osd":3,"up_from":27,"seq":115964117141,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2645228,"kb_used_data":2403044,"kb_used_omap":1,"kb_used_meta":242174,"kb_avail":91099924,"statfs":{"total":95995035648,"available":93286322176,"internally_reserved":0,"allocated":2460717056,"data_stored":2460410132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":247986631},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:30:59 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":2.7189999999999999,"15min":3.4990000000000001},"min":{"1min":0.24199999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":3.3839999999999999,"5min":19.475000000000001,"15min":19.475000000000001},"last":0.248},{"interface":"front","average":{"1min":0.874,"5min":2.754,"15min":3.476},"min":{"1min":0.31,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":3.419,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.27900000000000003}]},{"osd":1,"last update":"Fri Feb 9 17:30:35 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":2.7050000000000001,"15min":1.276},"min":{"1min":0.27000000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.347,"5min":19.638999999999999,"15min":19.638999999999999},"last":0.314},{"interface":"front","average":{"1min":0.57599999999999996,"5min":2.0739999999999998,"15min":0.98499999999999999},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.1659999999999999,"5min":14.661,"15min":14.661},"last":1.8919999999999999}]},{"osd":2,"last update":"Fri Feb 9 17:30:35 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":2.2040000000000002,"15min":1.091},"min":{"1min":0.28799999999999998,"5min":0.23599999999999999,"15min":0.224},"max":{"1min":2.2109999999999999,"5min":14.914999999999999,"15min":14.914999999999999},"last":1.9199999999999999},{"interface":"front","average":{"1min":0.626,"5min":2.0640000000000001,"15min":1},"min":{"1min":0.29199999999999998,"5min":0.218,"15min":0.218},"max":{"1min":2.1869999999999998,"5min":8.4730000000000008,"15min":8.4730000000000008},"last":1.6339999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:30:59 2024","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":2.1850000000000001,"15min":1.1779999999999999},"min":{"1min":0.26900000000000002,"5min":0.254,"15min":0.254},"max":{"1min":3.5819999999999999,"5min":10.762,"15min":10.762},"last":0.441},{"interface":"front","average":{"1min":1.4870000000000001,"5min":1.8819999999999999,"15min":1.0549999999999999},"min":{"1min":0.28699999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":5.5149999999999997,"5min":10.441000000000001,"15min":10.441000000000001},"last":6.7249999999999996}]},{"osd":5,"last update":"Fri Feb 9 17:30:26 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":1.2450000000000001,"15min":0.83199999999999996},"min":{"1min":0.40500000000000003,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":2.254,"5min":7.0110000000000001,"15min":7.0110000000000001},"last":3.5649999999999999},{"interface":"front","average":{"1min":0.81999999999999995,"5min":1.3049999999999999,"15min":0.84499999999999997},"min":{"1min":0.43099999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":2.2850000000000001,"5min":7.1600000000000001,"15min":7.1600000000000001},"last":3.6000000000000001}]},{"osd":6,"last update":"Fri Feb 9 17:30:54 2024","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":1.8520000000000001,"15min":1.071},"min":{"1min":0.42799999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":2.5609999999999999,"5min":19.582999999999998,"15min":19.582999999999998},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.93999999999999995,"5min":1.306,"15min":0.879},"min":{"1min":0.42099999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":2.5489999999999999,"5min":7.0469999999999997,"15min":7.0469999999999997},"last":4.4219999999999997}]},{"osd":7,"last update":"Fri Feb 9 17:30:19 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":2.4159999999999999,"15min":1.2749999999999999},"min":{"1min":0.435,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.3819999999999999,"5min":10.791,"15min":10.791},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.69199999999999995,"5min":1.9970000000000001,"15min":1.125},"min":{"1min":0.443,"5min":0.37,"15min":0.37},"max":{"1min":1.284,"5min":19.550999999999998,"15min":19.550999999999998},"last":0.38500000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:26:30 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":3.7629999999999999,"5min":3.7629999999999999,"15min":3.7629999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":2.327,"5min":1.22,"15min":0.93000000000000005},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":10.167,"5min":10.167,"15min":10.167},"last":0.49099999999999999}]},{"osd":9,"last update":"Fri Feb 9 17:31:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":1.3839999999999999,"15min":1.014},"min":{"1min":0.45900000000000002,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":2.4889999999999999,"5min":7.0810000000000004,"15min":7.0810000000000004},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.78400000000000003,"5min":2.234,"15min":1.3460000000000001},"min":{"1min":0.379,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":2.597,"5min":13.717000000000001,"15min":13.717000000000001},"last":1.9410000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:30:35 2024","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":1.944,"15min":1.228},"min":{"1min":0.502,"5min":0.40200000000000002,"15min":0.39400000000000002},"max":{"1min":2.5569999999999999,"5min":19.611000000000001,"15min":19.611000000000001},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.93500000000000005,"5min":1.74,"15min":1.0940000000000001},"min":{"1min":0.53100000000000003,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":3.4510000000000001,"5min":10.036,"15min":10.036},"last":2.0049999999999999}]},{"osd":11,"last update":"Fri Feb 9 17:30:59 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.6299999999999999,"15min":1.0600000000000001},"min":{"1min":0.45000000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":3.2229999999999999,"5min":7.1360000000000001,"15min":7.1360000000000001},"last":1.976},{"interface":"front","average":{"1min":0.72899999999999998,"5min":2.218,"15min":1.333},"min":{"1min":0.39300000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":2.3999999999999999,"5min":10.882999999999999,"15min":10.882999999999999},"last":0.64000000000000001}]}]},{"osd":4,"up_from":32,"seq":137438953615,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3432612,"kb_used_data":3288804,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":90312540,"statfs":{"total":95995035648,"available":92480040960,"internally_reserved":0,"allocated":3367735296,"data_stored":3367428372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:30:50 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.4450000000000001,"5min":1.4450000000000001,"15min":1.4450000000000001},"last":0.249},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":2.6030000000000002,"5min":2.6030000000000002,"15min":2.6030000000000002},"last":3.2610000000000001}]},{"osd":1,"last update":"Fri Feb 9 17:30:57 2024","interfaces":[{"interface":"back","average":{"1min":1.4399999999999999,"5min":1.546,"15min":1.1060000000000001},"min":{"1min":0.35899999999999999,"5min":0.246,"15min":0.246},"max":{"1min":6.0919999999999996,"5min":14.221,"15min":42.542999999999999},"last":3.1949999999999998},{"interface":"front","average":{"1min":1.407,"5min":1.4410000000000001,"15min":1.0629999999999999},"min":{"1min":0.30099999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":4.3300000000000001,"5min":7.7249999999999996,"15min":41.813000000000002},"last":3.645}]},{"osd":2,"last update":"Fri Feb 9 17:28:51 2024","interfaces":[{"interface":"back","average":{"1min":2.536,"5min":1.546,"15min":0.91400000000000003},"min":{"1min":0.29999999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":8.8819999999999997,"5min":8.8819999999999997,"15min":8.8819999999999997},"last":0.622},{"interface":"front","average":{"1min":2.214,"5min":1.7330000000000001,"15min":0.97799999999999998},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":9.5039999999999996,"5min":9.5039999999999996,"15min":9.5039999999999996},"last":0.53800000000000003}]},{"osd":3,"last update":"Fri Feb 9 17:30:57 2024","interfaces":[{"interface":"back","average":{"1min":3.016,"5min":2.0830000000000002,"15min":1.139},"min":{"1min":0.36299999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":33.037999999999997,"5min":33.037999999999997,"15min":33.037999999999997},"last":0.72199999999999998},{"interface":"front","average":{"1min":2.956,"5min":1.984,"15min":1.099},"min":{"1min":0.40600000000000003,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":33.054000000000002,"5min":33.054000000000002,"15min":33.054000000000002},"last":3.2240000000000002}]},{"osd":5,"last update":"Fri Feb 9 17:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.78000000000000003,"15min":0.57199999999999995},"min":{"1min":0.25600000000000001,"5min":0.222,"15min":0.222},"max":{"1min":3.8599999999999999,"5min":5.101,"15min":5.101},"last":3.8050000000000002},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.88600000000000001,"15min":0.64800000000000002},"min":{"1min":0.42999999999999999,"5min":0.29199999999999998,"15min":0.26000000000000001},"max":{"1min":3.9180000000000001,"5min":5.2220000000000004,"15min":5.2220000000000004},"last":3.8220000000000001}]},{"osd":6,"last update":"Fri Feb 9 17:30:57 2024","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.1319999999999999,"15min":0.71999999999999997},"min":{"1min":0.28699999999999998,"5min":0.247,"15min":0.247},"max":{"1min":6.0389999999999997,"5min":6.0389999999999997,"15min":6.0389999999999997},"last":3.5049999999999999},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.01,"15min":0.67200000000000004},"min":{"1min":0.32400000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":6.125,"5min":9.4109999999999996,"15min":9.4109999999999996},"last":3.1589999999999998}]},{"osd":7,"last update":"Fri Feb 9 17:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":1.0780000000000001,"15min":0.76600000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":3.968,"5min":15.223000000000001,"15min":15.223000000000001},"last":3.75},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.93899999999999995,"15min":0.71799999999999997},"min":{"1min":0.313,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":4.6669999999999998,"5min":15.579000000000001,"15min":15.579000000000001},"last":0.29699999999999999}]},{"osd":8,"last update":"Fri Feb 9 17:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":1.167,"15min":1.2030000000000001},"min":{"1min":0.46800000000000003,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":3.7570000000000001,"5min":4.1970000000000001,"15min":4.1970000000000001},"last":3.6800000000000002},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.99199999999999999,"15min":1.0029999999999999},"min":{"1min":0.40000000000000002,"5min":0.308,"15min":0.308},"max":{"1min":3.8330000000000002,"5min":3.9289999999999998,"15min":3.9289999999999998},"last":3.71}]},{"osd":9,"last update":"Fri Feb 9 17:31:10 2024","interfaces":[{"interface":"back","average":{"1min":1.794,"5min":1.2889999999999999,"15min":0.97799999999999998},"min":{"1min":0.37,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":4.6580000000000004,"5min":5.1929999999999996,"15min":5.1929999999999996},"last":3.7309999999999999},{"interface":"front","average":{"1min":1.6120000000000001,"5min":1.0900000000000001,"15min":0.90300000000000002},"min":{"1min":0.49299999999999999,"5min":0.371,"15min":0.371},"max":{"1min":6.6120000000000001,"5min":6.6120000000000001,"15min":6.6120000000000001},"last":0.36899999999999999}]},{"osd":10,"last update":"Fri Feb 9 17:30:39 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.163,"15min":0.95699999999999996},"min":{"1min":0.40000000000000002,"5min":0.378,"15min":0.378},"max":{"1min":3.891,"5min":5.2469999999999999,"15min":5.2469999999999999},"last":3.7679999999999998},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.091,"15min":0.93300000000000005},"min":{"1min":0.51900000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":2.6320000000000001,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.40500000000000003}]},{"osd":11,"last update":"Fri Feb 9 17:31:07 2024","interfaces":[{"interface":"back","average":{"1min":1.782,"5min":1.4390000000000001,"15min":1.3560000000000001},"min":{"1min":0.52400000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":4.5990000000000002,"5min":6.3689999999999998,"15min":6.3689999999999998},"last":3.786},{"interface":"front","average":{"1min":1.7969999999999999,"5min":1.232,"15min":1.1639999999999999},"min":{"1min":0.51600000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":6.6399999999999997,"5min":6.6399999999999997,"15min":6.6399999999999997},"last":0.42799999999999999}]}]},{"osd":5,"up_from":40,"seq":171798691977,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1991724,"kb_used_data":1892516,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":91753428,"statfs":{"total":95995035648,"available":93955510272,"internally_reserved":0,"allocated":1937936384,"data_stored":1937629460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:30:24 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":1.4850000000000001,"15min":0.872},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":2.4359999999999999,"5min":11.923,"15min":11.923},"last":1.9410000000000001},{"interface":"front","average":{"1min":0.57899999999999996,"5min":1.357,"15min":0.82099999999999995},"min":{"1min":0.33900000000000002,"5min":0.224,"15min":0.224},"max":{"1min":1.327,"5min":9.4239999999999995,"15min":9.4239999999999995},"last":0.51500000000000001}]},{"osd":1,"last update":"Fri Feb 9 17:30:24 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":2.0289999999999999,"15min":1.056},"min":{"1min":0.27000000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":2.4780000000000002,"5min":29.715,"15min":29.715},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.66000000000000003,"5min":2.0150000000000001,"15min":1.044},"min":{"1min":0.315,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":3.0910000000000002,"5min":29.969999999999999,"15min":29.969999999999999},"last":0.65300000000000002}]},{"osd":2,"last update":"Fri Feb 9 17:30:24 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":1.738,"15min":0.97199999999999998},"min":{"1min":0.32200000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":3.1179999999999999,"5min":9.8219999999999992,"15min":9.8219999999999992},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.79600000000000004,"5min":1.671,"15min":0.97899999999999998},"min":{"1min":0.42299999999999999,"5min":0.318,"15min":0.318},"max":{"1min":2.5089999999999999,"5min":11.978,"15min":11.978},"last":1.286}]},{"osd":3,"last update":"Fri Feb 9 17:30:24 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":1.7549999999999999,"15min":0.96599999999999997},"min":{"1min":0.35599999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":3.1699999999999999,"5min":10.587,"15min":10.587},"last":2.452},{"interface":"front","average":{"1min":0.83399999999999996,"5min":1.4370000000000001,"15min":0.878},"min":{"1min":0.45300000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":2.532,"5min":7.1859999999999999,"15min":7.1859999999999999},"last":1.333}]},{"osd":4,"last update":"Fri Feb 9 17:30:34 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.70199999999999996,"15min":0.51200000000000001},"min":{"1min":0.35899999999999999,"5min":0.246,"15min":0.24199999999999999},"max":{"1min":3.169,"5min":3.7949999999999999,"15min":3.7949999999999999},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.81000000000000005,"15min":0.56899999999999995},"min":{"1min":0.28100000000000003,"5min":0.27900000000000003,"15min":0.27000000000000002},"max":{"1min":3.2189999999999999,"5min":4.8010000000000002,"15min":4.8010000000000002},"last":0.68999999999999995}]},{"osd":6,"last update":"Fri Feb 9 17:30:54 2024","interfaces":[{"interface":"back","average":{"1min":1.548,"5min":1.0049999999999999,"15min":0.69499999999999995},"min":{"1min":0.38600000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":8.9250000000000007,"5min":8.9250000000000007,"15min":8.9250000000000007},"last":2.3820000000000001},{"interface":"front","average":{"1min":2.1269999999999998,"5min":1.0109999999999999,"15min":0.70099999999999996},"min":{"1min":0.374,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":9.141,"5min":9.141,"15min":9.141},"last":1.458}]},{"osd":7,"last update":"Fri Feb 9 17:30:24 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":1.006,"15min":0.747},"min":{"1min":0.36599999999999999,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":2.3889999999999998,"5min":18.626999999999999,"15min":18.626999999999999},"last":0.621},{"interface":"front","average":{"1min":0.88300000000000001,"5min":1.0569999999999999,"15min":0.75},"min":{"1min":0.52100000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":2.7549999999999999,"5min":18.701000000000001,"15min":18.701000000000001},"last":1.3580000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:26:36 2024","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":0.91000000000000003,"15min":0.78000000000000003},"min":{"1min":0.39400000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":5.556,"5min":5.556,"15min":5.556},"last":0.40300000000000002},{"interface":"front","average":{"1min":1.3720000000000001,"5min":0.92600000000000005,"15min":0.82999999999999996},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":5.9000000000000004,"5min":5.9000000000000004,"15min":5.9000000000000004},"last":0.36799999999999999}]},{"osd":9,"last update":"Fri Feb 9 17:31:02 2024","interfaces":[{"interface":"back","average":{"1min":1.548,"5min":1.0780000000000001,"15min":0.90600000000000003},"min":{"1min":0.371,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":6.298,"5min":6.298,"15min":6.298},"last":2.4300000000000002},{"interface":"front","average":{"1min":1.587,"5min":1.038,"15min":0.92300000000000004},"min":{"1min":0.372,"5min":0.312,"15min":0.312},"max":{"1min":6.3949999999999996,"5min":6.3949999999999996,"15min":6.3949999999999996},"last":2.407}]},{"osd":10,"last update":"Fri Feb 9 17:30:34 2024","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":0.99099999999999999,"15min":0.91800000000000004},"min":{"1min":0.624,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":2.8159999999999998,"5min":5.7510000000000003,"15min":13.071999999999999},"last":1.3859999999999999},{"interface":"front","average":{"1min":1.018,"5min":1.095,"15min":0.93200000000000005},"min":{"1min":0.63100000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":2.867,"5min":5.734,"15min":11.587},"last":1.411}]},{"osd":11,"last update":"Fri Feb 9 17:31:07 2024","interfaces":[{"interface":"back","average":{"1min":1.653,"5min":1.1739999999999999,"15min":1.218},"min":{"1min":0.46100000000000002,"5min":0.441,"15min":0.441},"max":{"1min":6.3639999999999999,"5min":6.3639999999999999,"15min":6.3639999999999999},"last":2.3500000000000001},{"interface":"front","average":{"1min":1.8899999999999999,"5min":1.079,"15min":0.91600000000000004},"min":{"1min":0.55800000000000005,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":6.8559999999999999,"5min":6.8559999999999999,"15min":6.8559999999999999},"last":1.4350000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2414542848,"data_stored":2414542848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1947336704,"data_stored":1947336704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":487194624,"data_stored":487194624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2467495936,"data_stored":2467495936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3397124096,"data_stored":3397124096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1940127744,"data_stored":1940127744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2460418048,"data_stored":2460418048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1987903488,"data_stored":1987903488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":995295232,"data_stored":995295232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1488977920,"data_stored":1488977920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":476577792,"data_stored":476577792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":3413049344,"data_stored":3413049344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-09T17:31:28.597 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd pool set .mgr pgp_num 1 2024-02-09T17:31:28.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:28 smithi162 ceph-mon[44304]: 3.7 scrub starts 2024-02-09T17:31:28.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:28 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/447713558' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:31:28.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:28 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:31:28.837 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:31:28.837 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-02-09T17:31:28.837 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-02-09T17:31:28.837 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-02-09T17:31:28.837 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.837 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:80118 2.d 2:b464a328:::benchmark_data_smithi179_77138_object80117:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-02-09T17:31:28.837 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:25:28.551378+0000", 2024-02-09T17:31:28.837 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 360.28189001499999, 2024-02-09T17:31:28.837 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.060053192999999998, 2024-02-09T17:31:28.837 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:31:28.838 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:31:28.838 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:31:28.838 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:31:28.838 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:31:28.838 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 80118 2024-02-09T17:31:28.838 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.838 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:31:28.838 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.838 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:31:28.838 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.551378+0000", 2024-02-09T17:31:28.838 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:28.838 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.838 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.839 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:31:28.839 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.551378+0000", 2024-02-09T17:31:28.839 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:28.839 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.839 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.839 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:31:28.839 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.551380+0000", 2024-02-09T17:31:28.839 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5740000000000001e-06 2024-02-09T17:31:28.839 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.839 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.839 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:31:28.839 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.551399+0000", 2024-02-09T17:31:28.840 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9151e-05 2024-02-09T17:31:28.840 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.840 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.840 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:31:28.840 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.551409+0000", 2024-02-09T17:31:28.840 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.781e-06 2024-02-09T17:31:28.840 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.840 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.840 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:31:28.840 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.551415+0000", 2024-02-09T17:31:28.840 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7270000000000004e-06 2024-02-09T17:31:28.840 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.840 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.841 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:31:28.841 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.551468+0000", 2024-02-09T17:31:28.841 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3041000000000003e-05 2024-02-09T17:31:28.841 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.841 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.841 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:31:28.841 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.551556+0000", 2024-02-09T17:31:28.841 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8195999999999994e-05 2024-02-09T17:31:28.841 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.841 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.841 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,4", 2024-02-09T17:31:28.841 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.551659+0000", 2024-02-09T17:31:28.841 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000103038 2024-02-09T17:31:28.841 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.842 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.842 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:28.842 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.610316+0000", 2024-02-09T17:31:28.842 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.058657448000000001 2024-02-09T17:31:28.842 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.842 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.842 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:28.842 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.610360+0000", 2024-02-09T17:31:28.842 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3344999999999999e-05 2024-02-09T17:31:28.842 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.842 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.842 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:31:28.843 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.611336+0000", 2024-02-09T17:31:28.843 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00097672399999999999 2024-02-09T17:31:28.843 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.843 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.843 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:31:28.843 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.611394+0000", 2024-02-09T17:31:28.843 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7376000000000001e-05 2024-02-09T17:31:28.843 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.843 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.843 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:31:28.843 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.611432+0000", 2024-02-09T17:31:28.843 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7792e-05 2024-02-09T17:31:28.843 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:28.844 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:31:28.844 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:28.844 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.844 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.844 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.14790.0:80117 2.a e129/111 2:564ef782:::benchmark_data_smithi179_77138_object80116:head v 129'4917, mlcod=129'4915)", 2024-02-09T17:31:28.844 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:25:28.551479+0000", 2024-02-09T17:31:28.844 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 360.28178919700002, 2024-02-09T17:31:28.844 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.055548806999999999, 2024-02-09T17:31:28.844 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:31:28.844 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:31:28.844 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:31:28.844 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.844 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:31:28.845 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.551479+0000", 2024-02-09T17:31:28.845 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:28.845 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.845 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.845 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:31:28.845 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.551479+0000", 2024-02-09T17:31:28.845 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:28.845 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.845 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.845 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:31:28.845 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.551480+0000", 2024-02-09T17:31:28.845 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.13e-07 2024-02-09T17:31:28.845 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.846 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.846 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:31:28.846 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.551495+0000", 2024-02-09T17:31:28.846 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5406000000000001e-05 2024-02-09T17:31:28.846 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.846 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.846 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:31:28.846 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.551502+0000", 2024-02-09T17:31:28.846 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3110000000000002e-06 2024-02-09T17:31:28.846 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.846 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.846 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:31:28.846 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.551505+0000", 2024-02-09T17:31:28.846 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6820000000000001e-06 2024-02-09T17:31:28.846 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.847 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.847 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:31:28.847 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.551552+0000", 2024-02-09T17:31:28.847 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6622999999999997e-05 2024-02-09T17:31:28.847 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.847 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.847 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:31:28.847 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.551570+0000", 2024-02-09T17:31:28.847 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8652999999999999e-05 2024-02-09T17:31:28.847 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.847 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.847 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:31:28.847 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.606983+0000", 2024-02-09T17:31:28.847 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.055412241000000001 2024-02-09T17:31:28.847 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.848 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.848 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:31:28.848 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.607028+0000", 2024-02-09T17:31:28.848 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5377999999999998e-05 2024-02-09T17:31:28.848 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:28.848 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:31:28.848 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:28.848 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.848 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.848 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop_reply(client.14790.0:80118 2.d e129/111 ondisk, result = 0)", 2024-02-09T17:31:28.848 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:25:28.554106+0000", 2024-02-09T17:31:28.848 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 360.279162687, 2024-02-09T17:31:28.848 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.056260956000000001, 2024-02-09T17:31:28.848 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:31:28.849 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-02-09T17:31:28.849 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:31:28.849 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.849 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:31:28.849 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.554106+0000", 2024-02-09T17:31:28.849 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:28.849 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.849 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.849 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:31:28.849 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.554106+0000", 2024-02-09T17:31:28.849 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:28.849 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.849 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.849 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:31:28.849 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.554107+0000", 2024-02-09T17:31:28.849 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8100000000000002e-07 2024-02-09T17:31:28.850 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.850 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.850 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:31:28.850 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.554113+0000", 2024-02-09T17:31:28.850 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.888e-06 2024-02-09T17:31:28.850 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.850 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.850 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:31:28.850 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.607102+0000", 2024-02-09T17:31:28.850 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.052988406000000002 2024-02-09T17:31:28.850 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.850 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.850 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:31:28.850 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.607110+0000", 2024-02-09T17:31:28.850 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7680000000000008e-06 2024-02-09T17:31:28.851 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.851 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.851 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:31:28.851 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.610349+0000", 2024-02-09T17:31:28.851 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.003239114 2024-02-09T17:31:28.851 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.851 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.851 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:31:28.851 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.610355+0000", 2024-02-09T17:31:28.851 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.976e-06 2024-02-09T17:31:28.851 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.851 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.851 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:31:28.852 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.610367+0000", 2024-02-09T17:31:28.852 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2023e-05 2024-02-09T17:31:28.852 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:28.852 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:31:28.852 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:28.852 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.852 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.852 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop_reply(client.14790.0:80118 2.d e129/111 ondisk, result = 0)", 2024-02-09T17:31:28.852 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:25:28.554107+0000", 2024-02-09T17:31:28.852 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 360.27916187099999, 2024-02-09T17:31:28.852 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.056217660000000003, 2024-02-09T17:31:28.852 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:31:28.852 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-02-09T17:31:28.853 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:31:28.853 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.853 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:31:28.853 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.554107+0000", 2024-02-09T17:31:28.853 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:28.853 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.853 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.853 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:31:28.853 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.554107+0000", 2024-02-09T17:31:28.853 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:28.853 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.853 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.853 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:31:28.853 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.554107+0000", 2024-02-09T17:31:28.854 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7300000000000005e-07 2024-02-09T17:31:28.854 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.854 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.854 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:31:28.854 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.554112+0000", 2024-02-09T17:31:28.854 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1799999999999998e-06 2024-02-09T17:31:28.854 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.854 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.854 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:31:28.854 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.606920+0000", 2024-02-09T17:31:28.854 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.052808709000000002 2024-02-09T17:31:28.854 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.854 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.854 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:31:28.855 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.606931+0000", 2024-02-09T17:31:28.855 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0978000000000001e-05 2024-02-09T17:31:28.855 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.855 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.855 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:31:28.855 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.610302+0000", 2024-02-09T17:31:28.855 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0033709899999999999 2024-02-09T17:31:28.855 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.855 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.855 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:31:28.855 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.610311+0000", 2024-02-09T17:31:28.855 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7029999999999995e-06 2024-02-09T17:31:28.855 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.855 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.856 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:31:28.856 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.610324+0000", 2024-02-09T17:31:28.856 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3327e-05 2024-02-09T17:31:28.856 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:28.856 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:31:28.856 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:28.856 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.856 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.856 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.14790.0:80121 2.0 e129/111 2:0886889a:::benchmark_data_smithi179_77138_object80120:head v 129'5026, mlcod=129'5024)", 2024-02-09T17:31:28.856 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:25:28.559396+0000", 2024-02-09T17:31:28.856 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 360.27387279800001, 2024-02-09T17:31:28.856 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.052018096999999999, 2024-02-09T17:31:28.856 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:31:28.856 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:31:28.857 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:31:28.857 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.857 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:31:28.857 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.559396+0000", 2024-02-09T17:31:28.857 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:28.857 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.857 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.857 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:31:28.857 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.559396+0000", 2024-02-09T17:31:28.857 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:28.857 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.857 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.857 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:31:28.857 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.559396+0000", 2024-02-09T17:31:28.858 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2799999999999996e-07 2024-02-09T17:31:28.858 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.858 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.858 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:31:28.858 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.559443+0000", 2024-02-09T17:31:28.858 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6338000000000003e-05 2024-02-09T17:31:28.858 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.858 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.858 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:31:28.858 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.607509+0000", 2024-02-09T17:31:28.858 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.048066580999999997 2024-02-09T17:31:28.858 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.859 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.859 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:31:28.859 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.607519+0000", 2024-02-09T17:31:28.859 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.9569999999999997e-06 2024-02-09T17:31:28.859 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.859 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.859 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:31:28.859 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.607592+0000", 2024-02-09T17:31:28.859 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2952000000000002e-05 2024-02-09T17:31:28.859 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.859 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.859 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:31:28.859 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.607614+0000", 2024-02-09T17:31:28.859 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1444e-05 2024-02-09T17:31:28.859 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.860 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.860 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:31:28.860 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.611364+0000", 2024-02-09T17:31:28.860 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0037505120000000001 2024-02-09T17:31:28.860 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.860 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.860 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:31:28.860 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.611414+0000", 2024-02-09T17:31:28.860 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9685000000000003e-05 2024-02-09T17:31:28.860 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:28.860 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:31:28.860 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:28.860 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.860 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.861 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:133740 2.d 2:bb74f4b7:::benchmark_data_smithi179_77138_object332:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e143)", 2024-02-09T17:31:28.861 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:26:19.838295+0000", 2024-02-09T17:31:28.861 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 308.99497302399999, 2024-02-09T17:31:28.861 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.049442864000000003, 2024-02-09T17:31:28.861 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:31:28.861 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:31:28.861 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:31:28.861 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:31:28.861 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:31:28.861 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 133740 2024-02-09T17:31:28.861 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.861 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:31:28.861 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.861 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:31:28.862 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:19.838295+0000", 2024-02-09T17:31:28.862 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:28.862 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.862 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.862 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:31:28.862 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:19.838295+0000", 2024-02-09T17:31:28.862 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:28.862 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.862 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.862 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:31:28.862 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:19.838296+0000", 2024-02-09T17:31:28.862 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3300000000000001e-07 2024-02-09T17:31:28.862 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.862 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.862 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:31:28.863 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:19.838300+0000", 2024-02-09T17:31:28.863 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.5260000000000002e-06 2024-02-09T17:31:28.863 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.863 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.863 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:31:28.863 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:19.838310+0000", 2024-02-09T17:31:28.863 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.8099999999999992e-06 2024-02-09T17:31:28.863 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.863 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.863 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:31:28.863 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:19.838317+0000", 2024-02-09T17:31:28.863 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.3780000000000003e-06 2024-02-09T17:31:28.863 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.863 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.864 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:31:28.864 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:19.838367+0000", 2024-02-09T17:31:28.864 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0019999999999999e-05 2024-02-09T17:31:28.864 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.864 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.864 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:31:28.864 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:19.838484+0000", 2024-02-09T17:31:28.864 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000116999 2024-02-09T17:31:28.864 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.864 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.864 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,4", 2024-02-09T17:31:28.864 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:19.838591+0000", 2024-02-09T17:31:28.864 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010672500000000001 2024-02-09T17:31:28.864 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.865 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.865 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:28.865 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:19.845672+0000", 2024-02-09T17:31:28.865 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0070815330000000001 2024-02-09T17:31:28.865 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.865 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.865 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:31:28.865 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:19.845685+0000", 2024-02-09T17:31:28.865 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2661e-05 2024-02-09T17:31:28.865 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.865 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.865 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:28.865 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:19.887652+0000", 2024-02-09T17:31:28.865 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.041967260999999999 2024-02-09T17:31:28.866 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.866 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.866 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:31:28.866 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:19.887702+0000", 2024-02-09T17:31:28.866 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9944999999999997e-05 2024-02-09T17:31:28.866 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.866 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.866 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:31:28.866 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:19.887738+0000", 2024-02-09T17:31:28.866 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6173000000000001e-05 2024-02-09T17:31:28.866 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:28.866 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:31:28.866 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:28.866 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.866 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.867 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:155521 2.d 2:b2596dcc:::benchmark_data_smithi179_77138_object22113:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-02-09T17:31:28.867 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:26:44.114894+0000", 2024-02-09T17:31:28.867 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 284.71837494699997, 2024-02-09T17:31:28.867 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.048923050000000003, 2024-02-09T17:31:28.867 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:31:28.867 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:31:28.867 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:31:28.867 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:31:28.867 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:31:28.867 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 155521 2024-02-09T17:31:28.867 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.867 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:31:28.867 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.868 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:31:28.868 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:44.114894+0000", 2024-02-09T17:31:28.868 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:28.868 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.868 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.868 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:31:28.868 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:44.114894+0000", 2024-02-09T17:31:28.868 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:28.868 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.868 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.868 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:31:28.868 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:44.114895+0000", 2024-02-09T17:31:28.868 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1000000000000001e-06 2024-02-09T17:31:28.868 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.869 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.869 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:31:28.869 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:44.114898+0000", 2024-02-09T17:31:28.869 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8089999999999999e-06 2024-02-09T17:31:28.869 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.869 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.869 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:31:28.869 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:44.114908+0000", 2024-02-09T17:31:28.869 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7189999999999996e-06 2024-02-09T17:31:28.869 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.869 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.869 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:31:28.869 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:44.114915+0000", 2024-02-09T17:31:28.869 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2049999999999996e-06 2024-02-09T17:31:28.870 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.870 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.870 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:31:28.870 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:44.115675+0000", 2024-02-09T17:31:28.870 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00075982999999999997 2024-02-09T17:31:28.870 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.870 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.870 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:31:28.870 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:44.115795+0000", 2024-02-09T17:31:28.870 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000119776 2024-02-09T17:31:28.870 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.870 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.870 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,4", 2024-02-09T17:31:28.870 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:44.115860+0000", 2024-02-09T17:31:28.870 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4611999999999997e-05 2024-02-09T17:31:28.871 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.871 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.871 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:31:28.871 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:44.119404+0000", 2024-02-09T17:31:28.871 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0035447230000000001 2024-02-09T17:31:28.871 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.871 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.871 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:28.871 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:44.125815+0000", 2024-02-09T17:31:28.871 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0064109850000000001 2024-02-09T17:31:28.871 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.871 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.871 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:28.871 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:44.163748+0000", 2024-02-09T17:31:28.872 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.037932403000000003 2024-02-09T17:31:28.872 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.872 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.872 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:31:28.872 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:44.163789+0000", 2024-02-09T17:31:28.872 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0945999999999997e-05 2024-02-09T17:31:28.872 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.872 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.872 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:31:28.872 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:44.163817+0000", 2024-02-09T17:31:28.872 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7942000000000001e-05 2024-02-09T17:31:28.872 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:28.873 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:31:28.873 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:28.873 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.873 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.873 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.14790.0:173778 2.a e151/111 2:5f827783:::benchmark_data_smithi179_77138_object40370:head v 151'10756, mlcod=151'10744)", 2024-02-09T17:31:28.873 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:04.453964+0000", 2024-02-09T17:31:28.873 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 264.37930462499997, 2024-02-09T17:31:28.873 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.051630930999999998, 2024-02-09T17:31:28.873 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:31:28.873 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:31:28.873 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:31:28.873 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.873 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:31:28.873 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.453964+0000", 2024-02-09T17:31:28.874 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:28.874 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.874 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.874 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:31:28.874 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.453964+0000", 2024-02-09T17:31:28.874 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:28.874 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.874 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.874 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:31:28.874 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.453964+0000", 2024-02-09T17:31:28.874 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4999999999999999e-07 2024-02-09T17:31:28.874 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.874 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.874 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:31:28.874 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.453971+0000", 2024-02-09T17:31:28.874 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.613e-06 2024-02-09T17:31:28.875 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.875 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.875 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:31:28.875 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.453976+0000", 2024-02-09T17:31:28.875 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1410000000000004e-06 2024-02-09T17:31:28.875 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.875 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.875 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:31:28.875 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.453979+0000", 2024-02-09T17:31:28.875 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.726e-06 2024-02-09T17:31:28.875 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.875 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.875 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:31:28.875 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.490546+0000", 2024-02-09T17:31:28.875 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.036566492999999999 2024-02-09T17:31:28.876 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.876 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.876 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:31:28.876 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.490566+0000", 2024-02-09T17:31:28.876 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0395999999999999e-05 2024-02-09T17:31:28.876 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.876 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.876 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:31:28.876 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.505548+0000", 2024-02-09T17:31:28.876 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014981828000000001 2024-02-09T17:31:28.876 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.876 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.876 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:31:28.876 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.505595+0000", 2024-02-09T17:31:28.877 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6584e-05 2024-02-09T17:31:28.877 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:28.877 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:31:28.877 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:28.877 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.877 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.877 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.14790.0:173780 2.a e151/111 2:5f6bf9e0:::benchmark_data_smithi179_77138_object40372:head v 151'10757, mlcod=151'10744)", 2024-02-09T17:31:28.877 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:04.453984+0000", 2024-02-09T17:31:28.877 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 264.37928417500001, 2024-02-09T17:31:28.877 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.058307537999999999, 2024-02-09T17:31:28.877 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:31:28.877 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:31:28.877 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:31:28.877 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.878 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:31:28.878 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.453984+0000", 2024-02-09T17:31:28.878 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:28.878 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.878 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.878 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:31:28.878 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.453984+0000", 2024-02-09T17:31:28.878 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:28.878 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.878 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.878 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:31:28.878 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.453984+0000", 2024-02-09T17:31:28.878 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7100000000000001e-07 2024-02-09T17:31:28.878 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.879 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.879 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:31:28.879 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.453987+0000", 2024-02-09T17:31:28.879 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5790000000000002e-06 2024-02-09T17:31:28.879 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.879 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.879 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:31:28.879 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.453993+0000", 2024-02-09T17:31:28.879 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6030000000000004e-06 2024-02-09T17:31:28.879 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.879 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.879 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:31:28.879 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.453996+0000", 2024-02-09T17:31:28.879 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7969999999999998e-06 2024-02-09T17:31:28.880 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.880 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.880 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:31:28.880 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.494808+0000", 2024-02-09T17:31:28.880 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.040811543999999998 2024-02-09T17:31:28.880 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.880 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.880 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:31:28.880 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.494835+0000", 2024-02-09T17:31:28.880 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6744999999999999e-05 2024-02-09T17:31:28.880 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.880 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.880 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:31:28.880 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.512249+0000", 2024-02-09T17:31:28.881 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017414190999999999 2024-02-09T17:31:28.881 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.881 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.881 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:31:28.881 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.512292+0000", 2024-02-09T17:31:28.881 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2908000000000003e-05 2024-02-09T17:31:28.881 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:28.881 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:31:28.881 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:28.881 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.881 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.881 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.14790.0:173783 2.a e151/111 2:5bec6a3b:::benchmark_data_smithi179_77138_object40375:head v 151'10758, mlcod=151'10744)", 2024-02-09T17:31:28.881 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:04.454001+0000", 2024-02-09T17:31:28.882 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 264.37926752999999, 2024-02-09T17:31:28.882 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.058321306000000003, 2024-02-09T17:31:28.882 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:31:28.882 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:31:28.882 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:31:28.882 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.882 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:31:28.882 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.454001+0000", 2024-02-09T17:31:28.882 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:28.882 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.882 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.882 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:31:28.882 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.454001+0000", 2024-02-09T17:31:28.882 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:28.883 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.883 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.883 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:31:28.883 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.454001+0000", 2024-02-09T17:31:28.883 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.74e-07 2024-02-09T17:31:28.883 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.883 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.883 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:31:28.883 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.454004+0000", 2024-02-09T17:31:28.883 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.593e-06 2024-02-09T17:31:28.883 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.883 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.883 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:31:28.883 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.454009+0000", 2024-02-09T17:31:28.884 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7030000000000003e-06 2024-02-09T17:31:28.884 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.884 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.884 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:31:28.884 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.454013+0000", 2024-02-09T17:31:28.884 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.878e-06 2024-02-09T17:31:28.884 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.884 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.884 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:31:28.884 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.498749+0000", 2024-02-09T17:31:28.884 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.044736154 2024-02-09T17:31:28.884 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.884 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.884 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:31:28.885 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.498774+0000", 2024-02-09T17:31:28.885 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4893e-05 2024-02-09T17:31:28.885 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.885 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.885 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:31:28.885 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.512294+0000", 2024-02-09T17:31:28.885 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013519869 2024-02-09T17:31:28.885 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.885 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.885 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:31:28.885 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.512322+0000", 2024-02-09T17:31:28.885 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.8042e-05 2024-02-09T17:31:28.885 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:28.886 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:31:28.886 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:28.886 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.886 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.886 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.14790.0:173785 2.a e151/111 2:50bf97e6:::benchmark_data_smithi179_77138_object40377:head v 151'10759, mlcod=151'10744)", 2024-02-09T17:31:28.886 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:04.454018+0000", 2024-02-09T17:31:28.886 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 264.37925061099997, 2024-02-09T17:31:28.886 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.058335370999999997, 2024-02-09T17:31:28.886 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:31:28.886 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:31:28.886 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:31:28.886 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.886 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:31:28.886 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.454018+0000", 2024-02-09T17:31:28.886 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:28.887 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.887 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.887 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:31:28.887 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.454018+0000", 2024-02-09T17:31:28.887 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:28.887 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.887 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.887 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:31:28.887 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.454018+0000", 2024-02-09T17:31:28.887 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.48e-07 2024-02-09T17:31:28.887 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.887 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.887 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:31:28.887 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.454023+0000", 2024-02-09T17:31:28.887 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1220000000000003e-06 2024-02-09T17:31:28.888 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.888 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.888 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:31:28.888 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.454029+0000", 2024-02-09T17:31:28.888 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.728e-06 2024-02-09T17:31:28.888 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.888 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.888 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:31:28.888 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.454039+0000", 2024-02-09T17:31:28.888 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0316000000000001e-05 2024-02-09T17:31:28.888 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.888 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.888 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:31:28.888 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.502223+0000", 2024-02-09T17:31:28.889 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.048184044000000002 2024-02-09T17:31:28.889 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.889 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.889 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:31:28.889 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.502242+0000", 2024-02-09T17:31:28.889 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8544000000000001e-05 2024-02-09T17:31:28.889 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.889 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.890 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:31:28.891 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.512326+0000", 2024-02-09T17:31:28.891 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010084253 2024-02-09T17:31:28.891 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.891 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.891 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:31:28.891 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.512353+0000", 2024-02-09T17:31:28.891 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7216000000000002e-05 2024-02-09T17:31:28.891 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:28.891 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:31:28.891 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:28.891 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.891 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.891 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.14790.0:173792 2.a e151/111 2:591697cf:::benchmark_data_smithi179_77138_object40384:head v 151'10760, mlcod=151'10744)", 2024-02-09T17:31:28.891 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:04.454044+0000", 2024-02-09T17:31:28.892 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 264.37922440400001, 2024-02-09T17:31:28.892 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.058332667999999997, 2024-02-09T17:31:28.892 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:31:28.892 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:31:28.892 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:31:28.892 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.892 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:31:28.892 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.454044+0000", 2024-02-09T17:31:28.892 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:28.892 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.892 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.892 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:31:28.892 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.454044+0000", 2024-02-09T17:31:28.892 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:28.893 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.893 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.893 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:31:28.893 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.454044+0000", 2024-02-09T17:31:28.893 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8799999999999999e-07 2024-02-09T17:31:28.893 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.893 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.893 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:31:28.893 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.454047+0000", 2024-02-09T17:31:28.893 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.785e-06 2024-02-09T17:31:28.893 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.893 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.893 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:31:28.893 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.454053+0000", 2024-02-09T17:31:28.893 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7880000000000003e-06 2024-02-09T17:31:28.894 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.894 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.894 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:31:28.894 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.454057+0000", 2024-02-09T17:31:28.894 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7790000000000002e-06 2024-02-09T17:31:28.894 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.894 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.894 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:31:28.894 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.505641+0000", 2024-02-09T17:31:28.894 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.051584818999999997 2024-02-09T17:31:28.894 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.894 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.894 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:31:28.895 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.505661+0000", 2024-02-09T17:31:28.895 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9888e-05 2024-02-09T17:31:28.895 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.895 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.895 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:31:28.895 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.512355+0000", 2024-02-09T17:31:28.895 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0066934439999999998 2024-02-09T17:31:28.895 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.895 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.895 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:31:28.895 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.512377+0000", 2024-02-09T17:31:28.895 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1977e-05 2024-02-09T17:31:28.895 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:28.896 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:31:28.896 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:28.896 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.896 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.896 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:181916 2.d 2:b5c83daf:::benchmark_data_smithi179_77138_object48508:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-02-09T17:31:28.896 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:13.475564+0000", 2024-02-09T17:31:28.896 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 255.35770473400001, 2024-02-09T17:31:28.896 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.052357841000000002, 2024-02-09T17:31:28.896 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:31:28.896 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:31:28.896 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:31:28.896 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:31:28.896 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:31:28.896 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 181916 2024-02-09T17:31:28.897 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.897 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:31:28.897 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.897 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:31:28.897 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.475564+0000", 2024-02-09T17:31:28.897 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:28.897 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.897 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.897 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:31:28.897 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.475564+0000", 2024-02-09T17:31:28.897 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:28.897 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.897 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.897 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:31:28.898 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.475565+0000", 2024-02-09T17:31:28.898 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0470000000000001e-06 2024-02-09T17:31:28.898 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.898 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.898 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:31:28.898 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.475570+0000", 2024-02-09T17:31:28.898 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2700000000000004e-06 2024-02-09T17:31:28.898 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.898 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.898 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:31:28.898 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.475583+0000", 2024-02-09T17:31:28.898 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2823000000000001e-05 2024-02-09T17:31:28.898 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.899 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.899 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:31:28.899 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.475591+0000", 2024-02-09T17:31:28.899 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1920000000000005e-06 2024-02-09T17:31:28.899 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.899 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.899 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:31:28.899 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.482368+0000", 2024-02-09T17:31:28.899 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0067773829999999997 2024-02-09T17:31:28.899 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.899 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.899 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:31:28.899 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.482488+0000", 2024-02-09T17:31:28.899 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000119066 2024-02-09T17:31:28.900 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.900 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.900 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,4", 2024-02-09T17:31:28.900 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.482604+0000", 2024-02-09T17:31:28.900 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000116335 2024-02-09T17:31:28.900 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.900 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.900 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:31:28.900 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.487370+0000", 2024-02-09T17:31:28.900 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0047663820000000004 2024-02-09T17:31:28.900 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.900 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.900 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:28.901 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.498528+0000", 2024-02-09T17:31:28.901 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011157602000000001 2024-02-09T17:31:28.901 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.901 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.901 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:28.901 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.527823+0000", 2024-02-09T17:31:28.901 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.029295307999999999 2024-02-09T17:31:28.901 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.901 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.901 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:31:28.901 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.527881+0000", 2024-02-09T17:31:28.901 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8088999999999999e-05 2024-02-09T17:31:28.901 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.901 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.902 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:31:28.902 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.527922+0000", 2024-02-09T17:31:28.902 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0343999999999999e-05 2024-02-09T17:31:28.902 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:28.902 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:31:28.902 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:28.902 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.902 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.902 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:181917 2.d 2:b69d7600:::benchmark_data_smithi179_77138_object48509:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-02-09T17:31:28.902 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:13.477060+0000", 2024-02-09T17:31:28.902 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 255.356208049, 2024-02-09T17:31:28.902 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.050989665000000003, 2024-02-09T17:31:28.902 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:31:28.902 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:31:28.902 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:31:28.903 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:31:28.903 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:31:28.903 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 181917 2024-02-09T17:31:28.903 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.903 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:31:28.903 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.903 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:31:28.903 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.477060+0000", 2024-02-09T17:31:28.903 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:28.903 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.903 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.903 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:31:28.903 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.477060+0000", 2024-02-09T17:31:28.903 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:28.904 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.904 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.904 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:31:28.904 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.477062+0000", 2024-02-09T17:31:28.904 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1319999999999999e-06 2024-02-09T17:31:28.904 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.904 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.904 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:31:28.904 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.477066+0000", 2024-02-09T17:31:28.904 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9089999999999999e-06 2024-02-09T17:31:28.904 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.904 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.904 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:31:28.904 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.477078+0000", 2024-02-09T17:31:28.905 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1199000000000001e-05 2024-02-09T17:31:28.905 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.905 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.905 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:31:28.905 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.477086+0000", 2024-02-09T17:31:28.905 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8509999999999999e-06 2024-02-09T17:31:28.905 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.905 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.905 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:31:28.905 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.487389+0000", 2024-02-09T17:31:28.905 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010303102999999999 2024-02-09T17:31:28.905 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.905 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.905 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:31:28.906 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.490188+0000", 2024-02-09T17:31:28.906 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0027996200000000001 2024-02-09T17:31:28.906 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.906 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.906 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,4", 2024-02-09T17:31:28.906 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.490313+0000", 2024-02-09T17:31:28.906 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012458400000000001 2024-02-09T17:31:28.906 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.906 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.906 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:31:28.906 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.498535+0000", 2024-02-09T17:31:28.906 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0082224059999999998 2024-02-09T17:31:28.906 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.906 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.906 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:28.907 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.498636+0000", 2024-02-09T17:31:28.907 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000100929 2024-02-09T17:31:28.907 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.907 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.907 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:28.907 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.527967+0000", 2024-02-09T17:31:28.907 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.029331327000000001 2024-02-09T17:31:28.907 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.907 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.907 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:31:28.907 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.528017+0000", 2024-02-09T17:31:28.907 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0008e-05 2024-02-09T17:31:28.907 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.907 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.908 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:31:28.908 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.528050+0000", 2024-02-09T17:31:28.908 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2597e-05 2024-02-09T17:31:28.908 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:28.908 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:31:28.908 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:28.908 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.908 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.908 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:181920 2.d 2:b2590050:::benchmark_data_smithi179_77138_object48512:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-02-09T17:31:28.908 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:13.477491+0000", 2024-02-09T17:31:28.908 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 255.35577703800001, 2024-02-09T17:31:28.908 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.050686139999999998, 2024-02-09T17:31:28.908 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:31:28.909 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:31:28.909 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:31:28.909 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:31:28.909 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:31:28.909 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 181920 2024-02-09T17:31:28.909 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.909 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:31:28.909 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.909 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:31:28.909 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.477491+0000", 2024-02-09T17:31:28.909 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:28.909 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.909 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.909 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:31:28.910 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.477491+0000", 2024-02-09T17:31:28.910 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:28.910 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.910 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.910 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:31:28.910 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.477492+0000", 2024-02-09T17:31:28.910 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7300000000000005e-07 2024-02-09T17:31:28.910 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.910 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.910 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:31:28.910 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.477496+0000", 2024-02-09T17:31:28.910 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6509999999999999e-06 2024-02-09T17:31:28.910 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.911 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.911 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:31:28.911 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.477504+0000", 2024-02-09T17:31:28.911 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1689999999999992e-06 2024-02-09T17:31:28.911 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.911 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.911 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:31:28.911 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.477510+0000", 2024-02-09T17:31:28.911 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4409999999999998e-06 2024-02-09T17:31:28.911 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.911 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.911 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:31:28.911 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.494373+0000", 2024-02-09T17:31:28.912 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.016862460999999999 2024-02-09T17:31:28.912 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.912 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.912 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:31:28.912 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.494488+0000", 2024-02-09T17:31:28.912 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000114688 2024-02-09T17:31:28.912 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.912 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.912 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,4", 2024-02-09T17:31:28.912 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.494560+0000", 2024-02-09T17:31:28.912 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2496000000000005e-05 2024-02-09T17:31:28.912 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.912 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.913 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:28.913 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.498662+0000", 2024-02-09T17:31:28.913 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0041014759999999997 2024-02-09T17:31:28.913 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.913 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.913 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:31:28.913 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.502436+0000", 2024-02-09T17:31:28.913 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.003773906 2024-02-09T17:31:28.913 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.913 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.913 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:28.913 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.528101+0000", 2024-02-09T17:31:28.913 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.025665652000000001 2024-02-09T17:31:28.913 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.914 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.914 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:31:28.914 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.528145+0000", 2024-02-09T17:31:28.914 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3537e-05 2024-02-09T17:31:28.914 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.914 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.914 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:31:28.914 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.528178+0000", 2024-02-09T17:31:28.914 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2889999999999999e-05 2024-02-09T17:31:28.914 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:28.914 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:31:28.914 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:28.914 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.914 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.914 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:204811 2.d 2:bac4c2f9:::benchmark_data_smithi179_77138_object71403:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-02-09T17:31:28.914 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:41.275956+0000", 2024-02-09T17:31:28.915 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 227.55731248999999, 2024-02-09T17:31:28.915 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.050306099999999999, 2024-02-09T17:31:28.915 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:31:28.915 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:31:28.915 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:31:28.915 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:31:28.915 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:31:28.915 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 204811 2024-02-09T17:31:28.915 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.915 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:31:28.915 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.915 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:31:28.915 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:41.275956+0000", 2024-02-09T17:31:28.915 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:28.915 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.915 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.916 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:31:28.916 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:41.275956+0000", 2024-02-09T17:31:28.916 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:28.916 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.916 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.916 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:31:28.916 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:41.275957+0000", 2024-02-09T17:31:28.916 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0249999999999999e-06 2024-02-09T17:31:28.916 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.916 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.916 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:31:28.916 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:41.275962+0000", 2024-02-09T17:31:28.917 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0849999999999996e-06 2024-02-09T17:31:28.917 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.917 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.917 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:31:28.917 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:41.275972+0000", 2024-02-09T17:31:28.917 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5060000000000001e-06 2024-02-09T17:31:28.917 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.917 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.917 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:31:28.917 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:41.275978+0000", 2024-02-09T17:31:28.917 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3249999999999996e-06 2024-02-09T17:31:28.917 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.917 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.917 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:31:28.917 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:41.276019+0000", 2024-02-09T17:31:28.918 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0661999999999998e-05 2024-02-09T17:31:28.918 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.918 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.918 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:31:28.918 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:41.276103+0000", 2024-02-09T17:31:28.918 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4039e-05 2024-02-09T17:31:28.918 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.918 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.918 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,4", 2024-02-09T17:31:28.918 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:41.312999+0000", 2024-02-09T17:31:28.918 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.036896067999999997 2024-02-09T17:31:28.918 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.918 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.918 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:31:28.918 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:41.326063+0000", 2024-02-09T17:31:28.919 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013064015999999999 2024-02-09T17:31:28.919 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.919 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.919 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:28.919 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:41.326123+0000", 2024-02-09T17:31:28.919 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0340000000000002e-05 2024-02-09T17:31:28.919 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.919 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.919 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:28.919 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:41.326201+0000", 2024-02-09T17:31:28.919 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7773000000000002e-05 2024-02-09T17:31:28.919 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.919 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.919 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:31:28.920 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:41.326235+0000", 2024-02-09T17:31:28.920 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4499000000000003e-05 2024-02-09T17:31:28.920 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.920 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.920 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:31:28.920 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:41.326262+0000", 2024-02-09T17:31:28.920 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6761999999999999e-05 2024-02-09T17:31:28.920 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:28.920 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:31:28.920 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:28.920 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.920 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.920 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:218135 2.d 2:b725d747:::benchmark_data_smithi179_77138_object84727:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-02-09T17:31:28.920 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:56.815902+0000", 2024-02-09T17:31:28.921 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 212.017366659, 2024-02-09T17:31:28.921 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.055060459999999999, 2024-02-09T17:31:28.921 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:31:28.921 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:31:28.921 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:31:28.921 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:31:28.921 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:31:28.921 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 218135 2024-02-09T17:31:28.921 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.921 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:31:28.921 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.921 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:31:28.921 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:56.815902+0000", 2024-02-09T17:31:28.922 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:28.922 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.922 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.922 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:31:28.922 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:56.815902+0000", 2024-02-09T17:31:28.922 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:28.922 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.922 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.922 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:31:28.922 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:56.815903+0000", 2024-02-09T17:31:28.922 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.122e-06 2024-02-09T17:31:28.922 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.922 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.922 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:31:28.923 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:56.815908+0000", 2024-02-09T17:31:28.923 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8489999999999997e-06 2024-02-09T17:31:28.923 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.923 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.923 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:31:28.923 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:56.815919+0000", 2024-02-09T17:31:28.923 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1589e-05 2024-02-09T17:31:28.923 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.923 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.923 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:31:28.923 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:56.815926+0000", 2024-02-09T17:31:28.923 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.0190000000000001e-06 2024-02-09T17:31:28.923 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.923 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.923 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:31:28.924 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:56.815978+0000", 2024-02-09T17:31:28.924 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1600000000000001e-05 2024-02-09T17:31:28.924 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.924 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.924 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:31:28.924 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:56.816077+0000", 2024-02-09T17:31:28.924 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.8690000000000005e-05 2024-02-09T17:31:28.924 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.924 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.924 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,4", 2024-02-09T17:31:28.924 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:56.816165+0000", 2024-02-09T17:31:28.924 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8376999999999998e-05 2024-02-09T17:31:28.924 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.924 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.925 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:28.925 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:56.820079+0000", 2024-02-09T17:31:28.925 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0039135849999999998 2024-02-09T17:31:28.925 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.925 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.925 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:31:28.925 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:56.820175+0000", 2024-02-09T17:31:28.925 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5941999999999997e-05 2024-02-09T17:31:28.925 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.925 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.925 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:28.925 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:56.870897+0000", 2024-02-09T17:31:28.925 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.050722738000000003 2024-02-09T17:31:28.925 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.926 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.926 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:31:28.926 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:56.870934+0000", 2024-02-09T17:31:28.926 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6653999999999998e-05 2024-02-09T17:31:28.926 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.926 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.926 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:31:28.926 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:56.870962+0000", 2024-02-09T17:31:28.926 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.8294999999999999e-05 2024-02-09T17:31:28.926 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:28.926 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:31:28.926 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:28.926 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.926 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.926 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:227092 2.d 2:b61c2b3c:::benchmark_data_smithi179_77138_object93684:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-02-09T17:31:28.927 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:28:06.766657+0000", 2024-02-09T17:31:28.927 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 202.066611618, 2024-02-09T17:31:28.927 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.050998014000000001, 2024-02-09T17:31:28.927 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:31:28.927 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:31:28.927 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:31:28.927 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:31:28.927 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:31:28.927 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 227092 2024-02-09T17:31:28.927 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.927 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:31:28.927 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.927 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:31:28.927 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:06.766657+0000", 2024-02-09T17:31:28.927 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:28.927 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.927 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.928 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:31:28.928 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:06.766657+0000", 2024-02-09T17:31:28.928 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:28.928 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.928 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.928 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:31:28.928 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:06.766659+0000", 2024-02-09T17:31:28.928 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.767e-06 2024-02-09T17:31:28.928 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.928 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.928 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:31:28.928 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:06.766663+0000", 2024-02-09T17:31:28.928 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7400000000000004e-06 2024-02-09T17:31:28.928 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.929 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.929 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:31:28.929 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:06.766679+0000", 2024-02-09T17:31:28.929 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5690999999999999e-05 2024-02-09T17:31:28.929 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.929 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.929 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:31:28.929 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:06.766687+0000", 2024-02-09T17:31:28.929 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.32e-06 2024-02-09T17:31:28.929 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.929 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.929 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:31:28.929 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:06.766746+0000", 2024-02-09T17:31:28.929 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8371000000000001e-05 2024-02-09T17:31:28.929 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.929 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.929 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:31:28.930 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:06.766851+0000", 2024-02-09T17:31:28.930 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010524800000000001 2024-02-09T17:31:28.930 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.930 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.930 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,4", 2024-02-09T17:31:28.930 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:06.766979+0000", 2024-02-09T17:31:28.930 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012756900000000001 2024-02-09T17:31:28.930 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.930 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.930 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:31:28.930 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:06.770957+0000", 2024-02-09T17:31:28.930 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0039787520000000003 2024-02-09T17:31:28.930 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.930 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.930 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:28.931 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:06.774094+0000", 2024-02-09T17:31:28.931 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0031361819999999999 2024-02-09T17:31:28.931 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.931 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.931 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:28.931 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:06.817564+0000", 2024-02-09T17:31:28.931 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.043470921000000003 2024-02-09T17:31:28.931 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.931 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.931 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:31:28.931 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:06.817617+0000", 2024-02-09T17:31:28.931 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2417000000000003e-05 2024-02-09T17:31:28.931 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.931 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.931 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:31:28.931 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:06.817655+0000", 2024-02-09T17:31:28.932 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8036e-05 2024-02-09T17:31:28.932 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:28.932 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:31:28.932 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:28.932 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.932 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.932 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.14790.0:239185 2.0 e174/111 2:0429c31f:::benchmark_data_smithi179_77138_object105777:head v 174'14898, mlcod=174'14895)", 2024-02-09T17:31:28.932 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:28:24.211292+0000", 2024-02-09T17:31:28.932 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 184.621976942, 2024-02-09T17:31:28.932 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.048589489, 2024-02-09T17:31:28.932 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:31:28.932 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:31:28.932 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:31:28.932 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.932 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:31:28.933 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:24.211292+0000", 2024-02-09T17:31:28.933 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:28.933 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.933 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.933 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:31:28.933 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:24.211292+0000", 2024-02-09T17:31:28.933 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:28.933 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.933 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.933 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:31:28.933 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:24.211292+0000", 2024-02-09T17:31:28.933 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8100000000000003e-07 2024-02-09T17:31:28.933 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.933 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.934 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:31:28.934 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:24.211297+0000", 2024-02-09T17:31:28.934 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1959999999999999e-06 2024-02-09T17:31:28.934 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.934 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.934 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:31:28.934 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:24.211303+0000", 2024-02-09T17:31:28.934 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4e-06 2024-02-09T17:31:28.934 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.934 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.934 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:31:28.934 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:24.211308+0000", 2024-02-09T17:31:28.934 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1449999999999999e-06 2024-02-09T17:31:28.934 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.935 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.935 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:31:28.935 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:24.211355+0000", 2024-02-09T17:31:28.935 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6678000000000003e-05 2024-02-09T17:31:28.935 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.935 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.935 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:31:28.935 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:24.211373+0000", 2024-02-09T17:31:28.935 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8958e-05 2024-02-09T17:31:28.935 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.935 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.935 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:31:28.935 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:24.259835+0000", 2024-02-09T17:31:28.936 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.048461379999999998 2024-02-09T17:31:28.936 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.936 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.936 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:31:28.936 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:24.259881+0000", 2024-02-09T17:31:28.936 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6150999999999999e-05 2024-02-09T17:31:28.936 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:28.936 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:31:28.936 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:28.936 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.936 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.936 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.14790.0:239189 2.8 e174/111 2:1048378c:::benchmark_data_smithi179_77138_object105781:head v 174'14801, mlcod=174'14799)", 2024-02-09T17:31:28.936 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:28:24.218178+0000", 2024-02-09T17:31:28.936 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 184.61509065600001, 2024-02-09T17:31:28.937 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.056142144999999997, 2024-02-09T17:31:28.937 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:31:28.937 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:31:28.937 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:31:28.937 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.937 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:31:28.937 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:24.218178+0000", 2024-02-09T17:31:28.937 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:28.937 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.937 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.937 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:31:28.937 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:24.218178+0000", 2024-02-09T17:31:28.937 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:28.938 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.938 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.938 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:31:28.938 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:24.218179+0000", 2024-02-09T17:31:28.938 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.09e-07 2024-02-09T17:31:28.938 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.938 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.938 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:31:28.938 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:24.218185+0000", 2024-02-09T17:31:28.938 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3509999999999998e-06 2024-02-09T17:31:28.938 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.938 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.938 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:31:28.939 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:24.259826+0000", 2024-02-09T17:31:28.939 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.041641055000000003 2024-02-09T17:31:28.939 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.939 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.939 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:31:28.939 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:24.259838+0000", 2024-02-09T17:31:28.939 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2235999999999999e-05 2024-02-09T17:31:28.939 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.939 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.939 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:31:28.939 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:24.259885+0000", 2024-02-09T17:31:28.939 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6424999999999997e-05 2024-02-09T17:31:28.939 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.939 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.940 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:31:28.940 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:24.259907+0000", 2024-02-09T17:31:28.940 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2654999999999999e-05 2024-02-09T17:31:28.940 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.940 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.940 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:31:28.940 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:24.274289+0000", 2024-02-09T17:31:28.940 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014381669 2024-02-09T17:31:28.940 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:28.940 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:28.940 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:31:28.940 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:24.274320+0000", 2024-02-09T17:31:28.940 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0945000000000003e-05 2024-02-09T17:31:28.940 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:28.941 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:31:28.941 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:28.941 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:28.941 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:31:28.941 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:31:28.950 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 83 16 127147 127131 95.7174 60.5 0.00525895 0.0104429 2024-02-09T17:31:29.219 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:31:29.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:29.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:31:29.327 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:31:29.360 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:29.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:29 smithi099 ceph-mon[39420]: 3.2 scrub starts 2024-02-09T17:31:29.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:29 smithi099 ceph-mon[39420]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:31:29.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:29 smithi099 ceph-mon[39420]: Cluster is now healthy 2024-02-09T17:31:29.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:29 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:31:29.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:29 smithi099 ceph-mon[39420]: osdmap e230: 12 total, 12 up, 12 in 2024-02-09T17:31:29.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:29 smithi099 ceph-mon[39420]: pgmap v672: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.6 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s wr, 803 op/s 2024-02-09T17:31:29.591 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:29 smithi179 ceph-mon[43975]: 3.2 scrub starts 2024-02-09T17:31:29.591 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:29 smithi179 ceph-mon[43975]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:31:29.591 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:29 smithi179 ceph-mon[43975]: Cluster is now healthy 2024-02-09T17:31:29.591 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:29 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:31:29.591 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:29 smithi179 ceph-mon[43975]: osdmap e230: 12 total, 12 up, 12 in 2024-02-09T17:31:29.591 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:29 smithi179 ceph-mon[43975]: pgmap v672: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.6 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s wr, 803 op/s 2024-02-09T17:31:29.608 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:31:29.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:29.614 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:31:29.626 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.10 dump_ops_in_flight 2024-02-09T17:31:29.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:29 smithi162 ceph-mon[44304]: 3.2 scrub starts 2024-02-09T17:31:29.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:29 smithi162 ceph-mon[44304]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:31:29.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:29 smithi162 ceph-mon[44304]: Cluster is now healthy 2024-02-09T17:31:29.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:29 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:31:29.683 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:29 smithi162 ceph-mon[44304]: osdmap e230: 12 total, 12 up, 12 in 2024-02-09T17:31:29.683 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:29 smithi162 ceph-mon[44304]: pgmap v672: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.6 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s wr, 803 op/s 2024-02-09T17:31:29.714 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:31:29.746 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:29.780 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:31:29.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:29.787 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:31:29.887 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:31:29.893 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:31:29.950 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 84 16 127929 127913 95.1598 48.875 0.0094717 0.0105036 2024-02-09T17:31:29.965 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:30.016 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:31:30.243 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:31:30.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:30.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:31:30.349 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:31:30.380 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:30.408 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:31:30.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:30.417 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:31:30.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:30 smithi099 ceph-mon[39420]: 3.7 scrub ok 2024-02-09T17:31:30.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:30 smithi099 ceph-mon[39420]: 3.b deep-scrub starts 2024-02-09T17:31:30.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:30 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:31:30.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:30 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:31:30.517 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:31:30.549 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:30.573 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:30 smithi179 ceph-mon[43975]: 3.7 scrub ok 2024-02-09T17:31:30.573 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:30 smithi179 ceph-mon[43975]: 3.b deep-scrub starts 2024-02-09T17:31:30.573 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:30 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:31:30.573 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:30 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:31:30.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:30 smithi162 ceph-mon[44304]: 3.7 scrub ok 2024-02-09T17:31:30.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:30 smithi162 ceph-mon[44304]: 3.b deep-scrub starts 2024-02-09T17:31:30.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:30 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:31:30.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:30 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:31:30.950 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 85 15 128705 128690 94.6116 48.5625 0.0229382 0.010565 2024-02-09T17:31:31.262 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set noscrub 2024-02-09T17:31:31.482 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:31:31.506 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:31:31.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:31.511 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:31:31.513 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:31:31.549 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:31:31.599 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:31:31.612 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:31:31.638 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:31:31.645 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:31 smithi179 ceph-mon[43975]: 3.2 scrub ok 2024-02-09T17:31:31.646 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:31 smithi179 ceph-mon[43975]: osdmap e231: 12 total, 12 up, 12 in 2024-02-09T17:31:31.646 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:31 smithi179 ceph-mon[43975]: pgmap v674: 17 pgs: 17 active+clean; 7.7 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 753 op/s 2024-02-09T17:31:31.649 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:31.670 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:31:31.673 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:31:31.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:31.674 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:31:31.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:31 smithi162 ceph-mon[44304]: 3.2 scrub ok 2024-02-09T17:31:31.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:31 smithi162 ceph-mon[44304]: osdmap e231: 12 total, 12 up, 12 in 2024-02-09T17:31:31.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:31 smithi162 ceph-mon[44304]: pgmap v674: 17 pgs: 17 active+clean; 7.7 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 753 op/s 2024-02-09T17:31:31.712 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:31:31.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:31 smithi099 ceph-mon[39420]: 3.2 scrub ok 2024-02-09T17:31:31.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:31 smithi099 ceph-mon[39420]: osdmap e231: 12 total, 12 up, 12 in 2024-02-09T17:31:31.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:31 smithi099 ceph-mon[39420]: pgmap v674: 17 pgs: 17 active+clean; 7.7 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 753 op/s 2024-02-09T17:31:31.750 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:31:31.775 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:31:31.788 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:31:31.818 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:31.838 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:31:31.847 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:31:31.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:31.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:31:31.856 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:31:31.889 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:31:31.894 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:31:31.894 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:31:31.894 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:31:31.895 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:31:31.895 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:31:31.895 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:31:31.895 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:31:31.895 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:31:31.895 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:31:31.895 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:31:31.895 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:31:31.895 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:31:31.950 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 86 16 129536 129520 94.1146 51.875 0.0123354 0.0106194 2024-02-09T17:31:31.953 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:31:31.990 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:32.094 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:31:32.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:32.097 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:31:32.116 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T17:31:32.116 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:31:32.198 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:31:32.233 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:32.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:32 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2665180534' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:31:32.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:32 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:31:32.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:32 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:31:32.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:32 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:31:32.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:32 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:31:32.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:32 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:31:32.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:32 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:31:32.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:32 smithi179 ceph-mon[43975]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:31:32.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:32 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2665180534' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:31:32.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:32 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:31:32.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:32 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:31:32.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:32 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:31:32.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:32 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:31:32.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:32 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:31:32.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:32 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:31:32.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:32 smithi162 ceph-mon[44304]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:31:32.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:32 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2665180534' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:31:32.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:32 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:31:32.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:32 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:31:32.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:32 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:31:32.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:32 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:31:32.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:32 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:31:32.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:32 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:31:32.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:32 smithi099 ceph-mon[39420]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:31:32.784 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:31:32.789 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:32.790 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:31:32.825 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.10 dump_blocked_ops 2024-02-09T17:31:32.891 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:31:32.939 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:32.950 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 87 16 130268 130252 93.5587 45.75 0.1097 0.0106832 2024-02-09T17:31:32.958 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:31:32.970 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:31:32.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:32.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:31:33.075 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:31:33.119 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:33.145 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:31:33.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:33.148 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:31:33.222 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:31:33.250 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:31:33.303 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:33.385 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:33 smithi162 ceph-mon[44304]: pgmap v675: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.8 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 951 op/s 2024-02-09T17:31:33.451 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:33 smithi099 ceph-mon[39420]: pgmap v675: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.8 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 951 op/s 2024-02-09T17:31:33.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:33 smithi179 ceph-mon[43975]: pgmap v675: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.8 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 951 op/s 2024-02-09T17:31:33.677 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:31:33.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:33.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:31:33.790 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:31:33.848 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:33.875 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:31:33.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:33.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:31:33.950 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 88 15 131108 131093 93.0928 52.5625 0.0156333 0.0107344 2024-02-09T17:31:33.982 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:31:34.024 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:34.408 INFO:teuthology.orchestra.run.smithi099.stderr:set pool 1 pgp_num to 1 2024-02-09T17:31:34.478 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:31:34.483 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:34.484 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:31:34.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:34 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3197650611' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-02-09T17:31:34.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:34 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-02-09T17:31:34.584 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:31:34.615 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:34.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:34 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3197650611' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-02-09T17:31:34.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:34 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-02-09T17:31:34.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:34 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3197650611' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-02-09T17:31:34.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:34 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-02-09T17:31:34.824 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:31:34.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:34.827 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:31:34.857 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:31:34.927 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:31:34.950 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 89 16 131917 131901 92.6142 50.5 0.00846896 0.0107817 2024-02-09T17:31:34.954 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:34.974 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:31:34.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:34.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:31:35.081 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:31:35.110 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:35.134 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:31:35.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:35.138 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:31:35.142 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T17:31:35.143 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:31:35.240 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:31:35.267 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:35.391 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:35 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-02-09T17:31:35.392 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:35 smithi099 ceph-mon[39420]: osdmap e232: 12 total, 12 up, 12 in 2024-02-09T17:31:35.392 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:35 smithi099 ceph-mon[39420]: pgmap v677: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.9 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s 2024-02-09T17:31:35.415 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:31:35.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:35.418 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:31:35.519 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:31:35.546 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:35.564 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:31:35.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:35.567 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:31:35.668 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:31:35.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:35 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-02-09T17:31:35.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:35 smithi179 ceph-mon[43975]: osdmap e232: 12 total, 12 up, 12 in 2024-02-09T17:31:35.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:35 smithi179 ceph-mon[43975]: pgmap v677: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.9 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s 2024-02-09T17:31:35.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:35 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-02-09T17:31:35.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:35 smithi162 ceph-mon[44304]: osdmap e232: 12 total, 12 up, 12 in 2024-02-09T17:31:35.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:35 smithi162 ceph-mon[44304]: pgmap v677: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.9 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s 2024-02-09T17:31:35.695 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:35.713 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:31:35.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:35.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:31:35.817 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:31:35.846 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:35.851 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.10 dump_historic_ops 2024-02-09T17:31:35.950 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 90 16 131956 131940 91.6123 2.4375 0.00546656 0.0107818 2024-02-09T17:31:36.046 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:31:36.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:36.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:31:36.061 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:31:36.106 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:31:36.150 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:31:36.179 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:36.305 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:31:36.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:36.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:31:36.410 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:31:36.437 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:36.455 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:31:36.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:36.458 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:31:36.559 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:31:36.586 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:36.951 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 91 16 131956 131940 90.6056 0 - 0.0107818 2024-02-09T17:31:37.232 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is set 2024-02-09T17:31:37.342 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:37 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3111032935' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:31:37.342 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:37 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:31:37.342 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:37 smithi179 ceph-mon[43975]: pgmap v678: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.0 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 858 op/s 2024-02-09T17:31:37.409 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:37 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3111032935' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:31:37.410 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:37 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:31:37.410 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:37 smithi099 ceph-mon[39420]: pgmap v678: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.0 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 858 op/s 2024-02-09T17:31:37.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:37 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3111032935' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:31:37.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:37 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:31:37.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:37 smithi162 ceph-mon[44304]: pgmap v678: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.0 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 858 op/s 2024-02-09T17:31:37.935 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:31:37.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:37.938 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:31:37.949 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T17:31:37.949 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:31:37.951 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 92 16 131956 131940 89.6204 0 - 0.0107818 2024-02-09T17:31:38.039 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:31:38.067 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:38.166 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:31:38.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:38.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:31:38.270 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:31:38.298 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:38.461 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:38 smithi099 ceph-mon[39420]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:31:38.461 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:38 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:31:38.461 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:38 smithi099 ceph-mon[39420]: osdmap e233: 12 total, 12 up, 12 in 2024-02-09T17:31:38.499 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:31:38.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:38.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:31:38.603 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:31:38.630 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:38 smithi179 ceph-mon[43975]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:31:38.630 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:38 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:31:38.630 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:38 smithi179 ceph-mon[43975]: osdmap e233: 12 total, 12 up, 12 in 2024-02-09T17:31:38.632 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:38.659 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:31:38.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:38.662 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:31:38.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:38 smithi162 ceph-mon[44304]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:31:38.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:38 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:31:38.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:38 smithi162 ceph-mon[44304]: osdmap e233: 12 total, 12 up, 12 in 2024-02-09T17:31:38.685 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.11 dump_ops_in_flight 2024-02-09T17:31:38.763 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:31:38.794 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:38.796 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:31:38.814 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:31:38.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:38.816 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:31:38.825 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:31:38.846 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:31:38.867 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:31:38.890 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:31:38.911 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:31:38.917 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:31:38.933 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:31:38.944 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:38.951 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 93 16 131956 131940 88.6568 0 - 0.0107818 2024-02-09T17:31:38.957 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:31:38.963 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:31:38.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:38.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:31:38.979 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:31:38.987 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:31:39.000 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:31:39.021 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:31:39.042 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:31:39.044 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:31:39.044 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:31:39.044 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:31:39.044 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:31:39.044 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:31:39.044 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:31:39.044 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:31:39.044 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:31:39.044 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:31:39.044 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:31:39.044 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:31:39.044 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:31:39.067 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:31:39.093 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:39.112 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:31:39.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:39.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:31:39.215 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:31:39.241 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:39.262 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:31:39.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:39.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:31:39.366 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:31:39.392 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:39.411 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:31:39.413 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:39.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:31:39.471 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:39 smithi099 ceph-mon[39420]: pgmap v680: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.0 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 42 MiB/s wr, 669 op/s 2024-02-09T17:31:39.471 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:39 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/407691892' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:31:39.490 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:39 smithi179 ceph-mon[43975]: pgmap v680: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.0 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 42 MiB/s wr, 669 op/s 2024-02-09T17:31:39.491 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:39 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/407691892' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:31:39.514 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:31:39.544 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:39.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:39 smithi162 ceph-mon[44304]: pgmap v680: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.0 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 42 MiB/s wr, 669 op/s 2024-02-09T17:31:39.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:39 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/407691892' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:31:39.951 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 94 16 131956 131940 87.7137 0 - 0.0107818 2024-02-09T17:31:39.993 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:31:39.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:39.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:31:40.098 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:31:40.125 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:40.147 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:31:40.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:40.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:31:40.185 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-09T17:31:40.251 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:31:40.281 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:40.302 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:31:40.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:40.305 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:31:40.407 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:31:40.433 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:40.452 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:31:40.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:40.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:31:40.477 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:31:40.551 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] 2024-02-09T17:31:40.551 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:31:40.551 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-02-09T17:31:40.552 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 0.609628 2024-02-09T17:31:40.552 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd primary-affinity 0 0.6096282681189776 2024-02-09T17:31:40.555 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:31:40.582 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:40.600 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:31:40.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:40.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:31:40.704 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:31:40.735 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:40.770 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:31:40.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:40.774 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:31:40.875 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:31:40.903 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:40.903 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:31:40.903 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-02-09T17:31:40.903 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 0 2024-02-09T17:31:40.903 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:31:40.923 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:31:40.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:40.926 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:31:40.951 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 95 16 131956 131940 86.7904 0 - 0.0107818 2024-02-09T17:31:41.027 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:31:41.055 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:41.203 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:31:41.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:41.206 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:31:41.307 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:31:41.332 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:41.342 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:31:41.580 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:31:41.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:41.583 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:31:41.598 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.11 dump_blocked_ops 2024-02-09T17:31:41.685 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:31:41.713 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:41.840 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:31:41.848 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:41 smithi099 ceph-mon[39420]: pgmap v681: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 30 MiB/s wr, 483 op/s 2024-02-09T17:31:41.848 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:41.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:31:41.894 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:31:41.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:41 smithi179 ceph-mon[43975]: pgmap v681: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 30 MiB/s wr, 483 op/s 2024-02-09T17:31:41.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:41 smithi162 ceph-mon[44304]: pgmap v681: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 30 MiB/s wr, 483 op/s 2024-02-09T17:31:41.949 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:31:41.950 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:31:41.952 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 96 16 131956 131940 85.8864 0 - 0.0107818 2024-02-09T17:31:41.975 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:42.131 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:31:42.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:42.135 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:31:42.236 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:31:42.264 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:42.363 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:31:42.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:42.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:31:42.476 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:31:42.503 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:42.651 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:31:42.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:42.654 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:31:42.755 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:31:42.784 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:42.952 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 97 16 131956 131940 85.001 0 - 0.0107818 2024-02-09T17:31:43.233 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:31:43.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:43.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:31:43.338 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:31:43.364 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:43.383 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:31:43.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:43.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:31:43.488 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:31:43.517 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:43.615 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:31:43.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:43.618 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:31:43.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:43 smithi162 ceph-mon[44304]: pgmap v682: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s wr, 321 op/s 2024-02-09T17:31:43.718 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:31:43.745 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:43.763 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:31:43.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:43.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:31:43.855 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:31:43.855 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-02-09T17:31:43.855 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-02-09T17:31:43.855 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-02-09T17:31:43.856 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:31:43.867 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:31:43.879 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:43 smithi179 ceph-mon[43975]: pgmap v682: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s wr, 321 op/s 2024-02-09T17:31:43.897 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:43.901 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:43 smithi099 ceph-mon[39420]: pgmap v682: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s wr, 321 op/s 2024-02-09T17:31:43.915 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:31:43.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:43.917 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:31:43.951 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 98 16 131956 131940 84.1337 0 - 0.0107818 2024-02-09T17:31:43.999 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:31:44.018 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:31:44.050 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:44.126 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:31:44.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:44.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:31:44.231 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:31:44.257 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:44.277 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:31:44.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:44.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:31:44.381 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:31:44.411 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:44.468 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:31:44.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:44.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:31:44.572 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:31:44.598 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:44.605 INFO:teuthology.orchestra.run.smithi099.stderr:set osd.0 primary-affinity to 0.609628 (8399522) 2024-02-09T17:31:44.617 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is set 2024-02-09T17:31:44.619 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:31:44.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:44.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:31:44.722 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:31:44.733 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:44 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2507808021' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:31:44.733 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:44 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1173903714' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.6096282681189776}]: dispatch 2024-02-09T17:31:44.741 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.11 dump_historic_ops 2024-02-09T17:31:44.748 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:44.773 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:31:44.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:44.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:31:44.876 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:31:44.900 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:44 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2507808021' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:31:44.900 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:44 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1173903714' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.6096282681189776}]: dispatch 2024-02-09T17:31:44.902 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:44.924 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:31:44.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:44 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2507808021' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:31:44.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:44 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1173903714' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.6096282681189776}]: dispatch 2024-02-09T17:31:44.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:44.927 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:31:44.952 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 99 16 131956 131940 83.2838 0 - 0.0107818 2024-02-09T17:31:45.027 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:31:45.053 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:31:45.055 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:45.077 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:31:45.079 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:45.080 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:31:45.181 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:31:45.207 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:45.307 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:31:45.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:45.310 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:31:45.411 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:31:45.439 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:45.732 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:45 smithi099 ceph-mon[39420]: pgmap v683: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 188 op/s 2024-02-09T17:31:45.732 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:45 smithi099 ceph-mon[39420]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:31:45.733 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:45 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2507808021' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:31:45.733 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:45 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1173903714' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.6096282681189776}]': finished 2024-02-09T17:31:45.733 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:45 smithi099 ceph-mon[39420]: osdmap e234: 12 total, 12 up, 12 in 2024-02-09T17:31:45.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:45 smithi179 ceph-mon[43975]: pgmap v683: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 188 op/s 2024-02-09T17:31:45.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:45 smithi179 ceph-mon[43975]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:31:45.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:45 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2507808021' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:31:45.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:45 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1173903714' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.6096282681189776}]': finished 2024-02-09T17:31:45.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:45 smithi179 ceph-mon[43975]: osdmap e234: 12 total, 12 up, 12 in 2024-02-09T17:31:45.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:45 smithi162 ceph-mon[44304]: pgmap v683: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 188 op/s 2024-02-09T17:31:45.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:45 smithi162 ceph-mon[44304]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:31:45.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:45 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2507808021' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:31:45.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:45 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1173903714' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.6096282681189776}]': finished 2024-02-09T17:31:45.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:45 smithi162 ceph-mon[44304]: osdmap e234: 12 total, 12 up, 12 in 2024-02-09T17:31:45.952 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:2024-02-09T17:31:45.952000+0000 min lat: 0.00110138 max lat: 0.611297 avg lat: 0.0107818 2024-02-09T17:31:45.952 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-09T17:31:45.952 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 100 16 131956 131940 82.451 0 - 0.0107818 2024-02-09T17:31:46.888 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:31:46.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:46.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:31:46.952 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 101 16 131956 131940 81.6347 0 - 0.0107818 2024-02-09T17:31:46.991 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:31:46.999 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:31:46.999 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-02-09T17:31:46.999 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-02-09T17:31:46.999 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-02-09T17:31:47.000 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.000 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:133709 2.c 2:36d46332:::benchmark_data_smithi179_77138_object301:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-02-09T17:31:47.000 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:26:19.805806+0000", 2024-02-09T17:31:47.000 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 327.19054436599998, 2024-02-09T17:31:47.000 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.109725028, 2024-02-09T17:31:47.000 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:31:47.000 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:31:47.000 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:31:47.000 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:31:47.000 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:31:47.000 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 133709 2024-02-09T17:31:47.000 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.001 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:31:47.001 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.001 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:31:47.001 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:19.805806+0000", 2024-02-09T17:31:47.001 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:47.001 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.001 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.001 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:31:47.001 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:19.805806+0000", 2024-02-09T17:31:47.001 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:47.001 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.001 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.001 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:31:47.001 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:19.805807+0000", 2024-02-09T17:31:47.001 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7570000000000001e-06 2024-02-09T17:31:47.002 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.002 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.002 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:31:47.002 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:19.805814+0000", 2024-02-09T17:31:47.002 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5740000000000001e-06 2024-02-09T17:31:47.002 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.002 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.002 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:31:47.002 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:19.805827+0000", 2024-02-09T17:31:47.002 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2795e-05 2024-02-09T17:31:47.002 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.002 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.002 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:31:47.002 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:19.805835+0000", 2024-02-09T17:31:47.002 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.0940000000000008e-06 2024-02-09T17:31:47.002 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.003 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.003 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:31:47.003 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:19.812536+0000", 2024-02-09T17:31:47.003 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0067011859999999996 2024-02-09T17:31:47.003 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.003 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.003 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:31:47.003 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:19.812647+0000", 2024-02-09T17:31:47.003 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000110392 2024-02-09T17:31:47.003 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.003 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.003 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,1", 2024-02-09T17:31:47.003 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:19.812749+0000", 2024-02-09T17:31:47.004 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000102409 2024-02-09T17:31:47.004 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.004 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.004 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:31:47.004 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:19.824100+0000", 2024-02-09T17:31:47.004 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011351228999999999 2024-02-09T17:31:47.004 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.004 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.004 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:47.004 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:19.824141+0000", 2024-02-09T17:31:47.004 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0970999999999998e-05 2024-02-09T17:31:47.004 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.004 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.005 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:47.005 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:19.915446+0000", 2024-02-09T17:31:47.005 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.091304838999999999 2024-02-09T17:31:47.005 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.005 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.005 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:31:47.005 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:19.915483+0000", 2024-02-09T17:31:47.005 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6977999999999997e-05 2024-02-09T17:31:47.005 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.005 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.005 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:31:47.005 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:19.915531+0000", 2024-02-09T17:31:47.006 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7803999999999998e-05 2024-02-09T17:31:47.006 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:47.006 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:31:47.006 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:47.006 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.006 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.006 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:136243 2.c 2:3be08b8b:::benchmark_data_smithi179_77138_object2835:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-02-09T17:31:47.006 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:26:22.618859+0000", 2024-02-09T17:31:47.006 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 324.37749123200001, 2024-02-09T17:31:47.006 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.108967218, 2024-02-09T17:31:47.006 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:31:47.006 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:31:47.006 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:31:47.006 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:31:47.007 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:31:47.007 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 136243 2024-02-09T17:31:47.007 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.007 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:31:47.007 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.007 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:31:47.007 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:22.618859+0000", 2024-02-09T17:31:47.007 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:47.007 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.007 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.007 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:31:47.007 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:22.618859+0000", 2024-02-09T17:31:47.007 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:47.007 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.008 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.008 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:31:47.008 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:22.618860+0000", 2024-02-09T17:31:47.008 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.012e-06 2024-02-09T17:31:47.008 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.008 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.008 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:31:47.008 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:22.618864+0000", 2024-02-09T17:31:47.008 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1189999999999999e-06 2024-02-09T17:31:47.008 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.008 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.008 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:31:47.009 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:22.618874+0000", 2024-02-09T17:31:47.009 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0071000000000001e-05 2024-02-09T17:31:47.009 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.009 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.009 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:31:47.009 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:22.618880+0000", 2024-02-09T17:31:47.009 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5690000000000004e-06 2024-02-09T17:31:47.009 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.009 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.009 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:31:47.009 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:22.619336+0000", 2024-02-09T17:31:47.009 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00045622700000000002 2024-02-09T17:31:47.009 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.009 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.009 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:31:47.010 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:22.619422+0000", 2024-02-09T17:31:47.010 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5951000000000004e-05 2024-02-09T17:31:47.010 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.010 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.010 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,1", 2024-02-09T17:31:47.010 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:22.619494+0000", 2024-02-09T17:31:47.010 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1946000000000005e-05 2024-02-09T17:31:47.010 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.010 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.010 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:31:47.010 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:22.623895+0000", 2024-02-09T17:31:47.010 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0044010389999999998 2024-02-09T17:31:47.010 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.010 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.011 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:47.011 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:22.628717+0000", 2024-02-09T17:31:47.011 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0048221119999999999 2024-02-09T17:31:47.011 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.011 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.011 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:47.011 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:22.727714+0000", 2024-02-09T17:31:47.011 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.098997598000000006 2024-02-09T17:31:47.011 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.011 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.011 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:31:47.011 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:22.727781+0000", 2024-02-09T17:31:47.011 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6029000000000001e-05 2024-02-09T17:31:47.011 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.011 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.012 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:31:47.012 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:26:22.727826+0000", 2024-02-09T17:31:47.012 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5544999999999998e-05 2024-02-09T17:31:47.012 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:47.012 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:31:47.012 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:47.012 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.012 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.012 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:170253 2.c 2:33b79755:::benchmark_data_smithi179_77138_object36845:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-02-09T17:31:47.012 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:00.372563+0000", 2024-02-09T17:31:47.012 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 286.62378685800002, 2024-02-09T17:31:47.012 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.11809602900000001, 2024-02-09T17:31:47.012 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:31:47.013 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:31:47.013 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:31:47.013 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:31:47.013 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:31:47.013 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 170253 2024-02-09T17:31:47.013 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.013 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:31:47.013 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.013 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:31:47.013 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:00.372563+0000", 2024-02-09T17:31:47.013 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:47.013 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.013 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.013 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:31:47.014 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:00.372563+0000", 2024-02-09T17:31:47.014 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:47.014 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.014 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.014 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:31:47.014 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:00.372564+0000", 2024-02-09T17:31:47.014 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0720000000000001e-06 2024-02-09T17:31:47.014 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.014 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.014 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:31:47.014 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:00.372569+0000", 2024-02-09T17:31:47.014 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2889999999999999e-06 2024-02-09T17:31:47.014 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.014 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.015 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:31:47.015 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:00.372580+0000", 2024-02-09T17:31:47.015 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1225e-05 2024-02-09T17:31:47.015 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.015 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.015 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:31:47.015 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:00.372588+0000", 2024-02-09T17:31:47.015 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7100000000000007e-06 2024-02-09T17:31:47.015 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.015 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.015 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:31:47.015 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:00.372926+0000", 2024-02-09T17:31:47.015 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00033813799999999998 2024-02-09T17:31:47.015 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.015 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.015 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:31:47.016 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:00.373013+0000", 2024-02-09T17:31:47.016 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7814999999999999e-05 2024-02-09T17:31:47.016 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.016 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.016 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,1", 2024-02-09T17:31:47.016 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:00.373100+0000", 2024-02-09T17:31:47.016 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.6187999999999995e-05 2024-02-09T17:31:47.016 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.016 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.016 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:47.016 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:00.376947+0000", 2024-02-09T17:31:47.016 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.003847329 2024-02-09T17:31:47.016 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.016 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.017 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:31:47.017 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:00.377062+0000", 2024-02-09T17:31:47.017 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011547799999999999 2024-02-09T17:31:47.017 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.017 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.017 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:47.017 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:00.490604+0000", 2024-02-09T17:31:47.017 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.11354122899999999 2024-02-09T17:31:47.017 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.017 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.017 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:31:47.017 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:00.490635+0000", 2024-02-09T17:31:47.017 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1216000000000001e-05 2024-02-09T17:31:47.017 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.017 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.017 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:31:47.018 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:00.490659+0000", 2024-02-09T17:31:47.018 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4340000000000001e-05 2024-02-09T17:31:47.018 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:47.018 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:31:47.018 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:47.018 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.018 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.018 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:173669 2.a 2:5049bf51:::benchmark_data_smithi179_77138_object40261:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-02-09T17:31:47.018 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:04.251927+0000", 2024-02-09T17:31:47.018 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 282.74442318500002, 2024-02-09T17:31:47.018 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.21543948900000001, 2024-02-09T17:31:47.018 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:31:47.018 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:31:47.018 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:31:47.019 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:31:47.019 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:31:47.019 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 173669 2024-02-09T17:31:47.019 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.019 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:31:47.019 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.019 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:31:47.019 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.251927+0000", 2024-02-09T17:31:47.019 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:47.019 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.019 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.019 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:31:47.019 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.251927+0000", 2024-02-09T17:31:47.019 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:47.019 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.020 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.020 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:31:47.020 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.251929+0000", 2024-02-09T17:31:47.020 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7579999999999999e-06 2024-02-09T17:31:47.020 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.020 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.020 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:31:47.020 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.251935+0000", 2024-02-09T17:31:47.020 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9329999999999998e-06 2024-02-09T17:31:47.020 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.020 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.020 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:31:47.020 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.251949+0000", 2024-02-09T17:31:47.020 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4268e-05 2024-02-09T17:31:47.021 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.021 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.021 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:31:47.021 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.251957+0000", 2024-02-09T17:31:47.021 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5290000000000008e-06 2024-02-09T17:31:47.021 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.021 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.021 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:31:47.021 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.252013+0000", 2024-02-09T17:31:47.021 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5619999999999999e-05 2024-02-09T17:31:47.021 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.021 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.021 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:31:47.021 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.252115+0000", 2024-02-09T17:31:47.022 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000102315 2024-02-09T17:31:47.022 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.022 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.022 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,9", 2024-02-09T17:31:47.022 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.252237+0000", 2024-02-09T17:31:47.022 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000121951 2024-02-09T17:31:47.022 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.022 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.022 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:31:47.022 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.256532+0000", 2024-02-09T17:31:47.022 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0042949540000000001 2024-02-09T17:31:47.022 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.022 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.023 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:47.023 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.259361+0000", 2024-02-09T17:31:47.023 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002829163 2024-02-09T17:31:47.023 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.023 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.023 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:47.023 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.467273+0000", 2024-02-09T17:31:47.023 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.207911866 2024-02-09T17:31:47.023 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.023 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.023 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:31:47.023 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.467324+0000", 2024-02-09T17:31:47.023 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0763000000000002e-05 2024-02-09T17:31:47.023 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.023 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.024 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:31:47.024 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.467366+0000", 2024-02-09T17:31:47.024 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2369e-05 2024-02-09T17:31:47.024 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:47.024 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:31:47.024 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:47.024 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.024 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.024 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:173689 2.a 2:55e666c2:::benchmark_data_smithi179_77138_object40281:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-02-09T17:31:47.024 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:04.273390+0000", 2024-02-09T17:31:47.024 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 282.72295987400003, 2024-02-09T17:31:47.024 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.201714904, 2024-02-09T17:31:47.024 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:31:47.024 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:31:47.025 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:31:47.025 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:31:47.025 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:31:47.025 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 173689 2024-02-09T17:31:47.025 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.025 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:31:47.025 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.025 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:31:47.025 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.273390+0000", 2024-02-09T17:31:47.025 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:47.025 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.025 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.025 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:31:47.025 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.273390+0000", 2024-02-09T17:31:47.025 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:47.026 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.026 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.026 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:31:47.026 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.273392+0000", 2024-02-09T17:31:47.026 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3340000000000001e-06 2024-02-09T17:31:47.026 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.026 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.026 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:31:47.026 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.273396+0000", 2024-02-09T17:31:47.026 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2640000000000002e-06 2024-02-09T17:31:47.026 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.026 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.026 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:31:47.026 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.273406+0000", 2024-02-09T17:31:47.027 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0135999999999999e-05 2024-02-09T17:31:47.027 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.027 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.027 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:31:47.027 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.273412+0000", 2024-02-09T17:31:47.027 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8289999999999996e-06 2024-02-09T17:31:47.027 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.027 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.027 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:31:47.027 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.273457+0000", 2024-02-09T17:31:47.027 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5225000000000002e-05 2024-02-09T17:31:47.027 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.027 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.027 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:31:47.027 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.273543+0000", 2024-02-09T17:31:47.028 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5606999999999995e-05 2024-02-09T17:31:47.028 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.028 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.028 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,9", 2024-02-09T17:31:47.028 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.273618+0000", 2024-02-09T17:31:47.028 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5615e-05 2024-02-09T17:31:47.028 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.028 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.028 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:31:47.028 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.277650+0000", 2024-02-09T17:31:47.028 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0040314699999999997 2024-02-09T17:31:47.028 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.028 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.028 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:47.029 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.288200+0000", 2024-02-09T17:31:47.029 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010550744000000001 2024-02-09T17:31:47.029 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.029 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.029 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:47.029 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.475013+0000", 2024-02-09T17:31:47.029 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.186812582 2024-02-09T17:31:47.029 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.029 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.029 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:31:47.029 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.475063+0000", 2024-02-09T17:31:47.029 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0197000000000001e-05 2024-02-09T17:31:47.029 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.029 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.030 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:31:47.030 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.475105+0000", 2024-02-09T17:31:47.030 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1900999999999997e-05 2024-02-09T17:31:47.030 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:47.030 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:31:47.030 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:47.030 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.030 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.030 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:173697 2.a 2:55ecfd26:::benchmark_data_smithi179_77138_object40289:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-02-09T17:31:47.030 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:04.282923+0000", 2024-02-09T17:31:47.030 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 282.71342666300001, 2024-02-09T17:31:47.030 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.19232663799999999, 2024-02-09T17:31:47.030 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:31:47.030 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:31:47.031 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:31:47.031 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:31:47.031 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:31:47.031 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 173697 2024-02-09T17:31:47.031 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.031 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:31:47.031 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.031 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:31:47.031 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.282923+0000", 2024-02-09T17:31:47.031 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:47.031 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.031 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.031 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:31:47.032 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.282923+0000", 2024-02-09T17:31:47.032 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:47.032 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.032 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.032 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:31:47.032 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.282925+0000", 2024-02-09T17:31:47.032 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4190000000000001e-06 2024-02-09T17:31:47.032 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.032 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.032 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:31:47.032 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.282930+0000", 2024-02-09T17:31:47.032 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9880000000000004e-06 2024-02-09T17:31:47.032 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.033 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.033 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:31:47.033 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.282952+0000", 2024-02-09T17:31:47.033 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1787999999999999e-05 2024-02-09T17:31:47.033 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.033 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.033 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:31:47.033 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.282960+0000", 2024-02-09T17:31:47.033 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7280000000000001e-06 2024-02-09T17:31:47.033 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.033 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.033 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:31:47.033 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.283139+0000", 2024-02-09T17:31:47.033 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00017882299999999999 2024-02-09T17:31:47.034 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.034 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.034 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:31:47.034 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.283270+0000", 2024-02-09T17:31:47.034 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013041300000000001 2024-02-09T17:31:47.034 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.034 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.034 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,9", 2024-02-09T17:31:47.034 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.283394+0000", 2024-02-09T17:31:47.034 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012484099999999999 2024-02-09T17:31:47.034 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.034 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.034 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:47.034 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.288244+0000", 2024-02-09T17:31:47.034 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0048494940000000002 2024-02-09T17:31:47.035 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.035 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.035 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:31:47.035 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.288278+0000", 2024-02-09T17:31:47.035 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.3902000000000002e-05 2024-02-09T17:31:47.035 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.035 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.035 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:47.035 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.475158+0000", 2024-02-09T17:31:47.035 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.18688034100000001 2024-02-09T17:31:47.035 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.035 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.035 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:31:47.035 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.475212+0000", 2024-02-09T17:31:47.036 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3863000000000003e-05 2024-02-09T17:31:47.036 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.036 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.036 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:31:47.036 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.475250+0000", 2024-02-09T17:31:47.036 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8037999999999997e-05 2024-02-09T17:31:47.036 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:47.036 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:31:47.036 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:47.036 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.036 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.036 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:173701 2.a 2:54bdf07e:::benchmark_data_smithi179_77138_object40293:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-02-09T17:31:47.036 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:04.288307+0000", 2024-02-09T17:31:47.036 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 282.70804344999999, 2024-02-09T17:31:47.036 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.19479152299999999, 2024-02-09T17:31:47.037 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:31:47.037 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:31:47.037 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:31:47.037 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:31:47.037 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:31:47.037 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 173701 2024-02-09T17:31:47.037 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.037 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:31:47.037 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.037 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:31:47.037 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.288307+0000", 2024-02-09T17:31:47.037 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:47.037 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.037 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.038 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:31:47.038 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.288307+0000", 2024-02-09T17:31:47.038 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:47.038 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.038 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.038 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:31:47.038 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.288308+0000", 2024-02-09T17:31:47.038 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.376e-06 2024-02-09T17:31:47.038 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.038 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.038 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:31:47.038 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.288314+0000", 2024-02-09T17:31:47.038 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8309999999999997e-06 2024-02-09T17:31:47.038 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.039 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.039 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:31:47.039 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.288330+0000", 2024-02-09T17:31:47.039 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5869999999999999e-05 2024-02-09T17:31:47.039 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.039 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.039 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:31:47.039 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.288338+0000", 2024-02-09T17:31:47.039 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7390000000000005e-06 2024-02-09T17:31:47.039 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.039 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.039 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:31:47.039 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.288385+0000", 2024-02-09T17:31:47.039 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6099e-05 2024-02-09T17:31:47.039 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.039 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.040 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:31:47.040 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.288505+0000", 2024-02-09T17:31:47.040 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000120623 2024-02-09T17:31:47.040 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.040 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.040 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,9", 2024-02-09T17:31:47.040 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.288636+0000", 2024-02-09T17:31:47.040 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000131013 2024-02-09T17:31:47.040 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.040 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.040 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:47.040 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.293939+0000", 2024-02-09T17:31:47.040 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0053033109999999998 2024-02-09T17:31:47.040 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.041 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.041 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:31:47.041 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.297980+0000", 2024-02-09T17:31:47.041 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00404004 2024-02-09T17:31:47.041 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.041 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.041 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:47.041 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.482999+0000", 2024-02-09T17:31:47.041 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.18501925899999999 2024-02-09T17:31:47.041 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.041 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.041 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:31:47.041 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.483057+0000", 2024-02-09T17:31:47.041 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8619000000000002e-05 2024-02-09T17:31:47.042 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.042 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.042 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:31:47.042 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.483098+0000", 2024-02-09T17:31:47.042 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0742999999999999e-05 2024-02-09T17:31:47.042 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:47.042 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:31:47.042 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:47.042 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.042 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.042 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:173707 2.a 2:5f5eaec0:::benchmark_data_smithi179_77138_object40299:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-02-09T17:31:47.042 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:04.293626+0000", 2024-02-09T17:31:47.042 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 282.70272386200003, 2024-02-09T17:31:47.042 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.189622768, 2024-02-09T17:31:47.042 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:31:47.042 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:31:47.043 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:31:47.043 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:31:47.043 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:31:47.043 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 173707 2024-02-09T17:31:47.043 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.043 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:31:47.043 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.043 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:31:47.043 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.293626+0000", 2024-02-09T17:31:47.043 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:47.043 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.043 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.043 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:31:47.043 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.293626+0000", 2024-02-09T17:31:47.043 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:47.044 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.044 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.044 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:31:47.044 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.293627+0000", 2024-02-09T17:31:47.044 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.19e-06 2024-02-09T17:31:47.044 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.044 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.044 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:31:47.044 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.293633+0000", 2024-02-09T17:31:47.044 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1549999999999998e-06 2024-02-09T17:31:47.044 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.044 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.044 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:31:47.044 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.293643+0000", 2024-02-09T17:31:47.045 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0394000000000001e-05 2024-02-09T17:31:47.045 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.045 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.045 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:31:47.045 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.293649+0000", 2024-02-09T17:31:47.045 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4119999999999998e-06 2024-02-09T17:31:47.045 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.045 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.045 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:31:47.045 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.293974+0000", 2024-02-09T17:31:47.045 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00032459299999999997 2024-02-09T17:31:47.045 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.045 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.045 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:31:47.046 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.294093+0000", 2024-02-09T17:31:47.046 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000119394 2024-02-09T17:31:47.046 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.046 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.046 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,9", 2024-02-09T17:31:47.046 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.294245+0000", 2024-02-09T17:31:47.046 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000151333 2024-02-09T17:31:47.046 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.046 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.046 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:31:47.046 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.305358+0000", 2024-02-09T17:31:47.046 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011113125999999999 2024-02-09T17:31:47.046 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.046 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.046 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:47.047 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.307269+0000", 2024-02-09T17:31:47.047 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0019107779999999999 2024-02-09T17:31:47.047 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.047 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.047 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:47.047 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.483159+0000", 2024-02-09T17:31:47.047 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.17589016900000001 2024-02-09T17:31:47.047 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.047 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.047 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:31:47.047 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.483213+0000", 2024-02-09T17:31:47.047 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4048999999999997e-05 2024-02-09T17:31:47.047 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.047 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.047 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:31:47.047 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.483249+0000", 2024-02-09T17:31:47.048 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6174999999999998e-05 2024-02-09T17:31:47.048 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:47.048 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:31:47.048 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:47.048 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.048 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.048 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:173712 2.a 2:5a5d0456:::benchmark_data_smithi179_77138_object40304:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-02-09T17:31:47.048 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:04.297430+0000", 2024-02-09T17:31:47.048 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 282.69892013399999, 2024-02-09T17:31:47.048 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.19325016, 2024-02-09T17:31:47.048 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:31:47.048 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:31:47.048 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:31:47.049 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:31:47.049 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:31:47.049 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 173712 2024-02-09T17:31:47.049 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.049 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:31:47.049 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.049 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:31:47.049 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.297430+0000", 2024-02-09T17:31:47.049 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:47.049 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.049 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.049 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:31:47.049 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.297430+0000", 2024-02-09T17:31:47.049 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:47.050 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.050 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.050 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:31:47.050 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.297432+0000", 2024-02-09T17:31:47.050 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8419999999999999e-06 2024-02-09T17:31:47.050 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.050 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.050 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:31:47.050 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.297439+0000", 2024-02-09T17:31:47.050 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7959999999999998e-06 2024-02-09T17:31:47.050 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.050 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.050 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:31:47.050 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.297470+0000", 2024-02-09T17:31:47.051 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1498000000000002e-05 2024-02-09T17:31:47.051 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.051 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.051 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:31:47.051 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.297480+0000", 2024-02-09T17:31:47.051 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.8740000000000007e-06 2024-02-09T17:31:47.051 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.051 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.051 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:31:47.051 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.298020+0000", 2024-02-09T17:31:47.051 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00054042499999999998 2024-02-09T17:31:47.051 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.051 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.051 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:31:47.051 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.298155+0000", 2024-02-09T17:31:47.052 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000134881 2024-02-09T17:31:47.052 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.052 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.052 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,9", 2024-02-09T17:31:47.052 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.298250+0000", 2024-02-09T17:31:47.052 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5222000000000004e-05 2024-02-09T17:31:47.052 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.052 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.052 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:31:47.052 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.305364+0000", 2024-02-09T17:31:47.052 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0071130430000000003 2024-02-09T17:31:47.052 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.052 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.052 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:47.052 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.308096+0000", 2024-02-09T17:31:47.053 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002732557 2024-02-09T17:31:47.053 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.053 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.053 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:47.053 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.490616+0000", 2024-02-09T17:31:47.053 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.18251959600000001 2024-02-09T17:31:47.053 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.053 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.053 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:31:47.053 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.490651+0000", 2024-02-09T17:31:47.053 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.5692999999999999e-05 2024-02-09T17:31:47.053 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.053 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.053 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:31:47.053 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.490680+0000", 2024-02-09T17:31:47.054 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.8733000000000001e-05 2024-02-09T17:31:47.054 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:47.054 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:31:47.054 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:47.054 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.054 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.054 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:173713 2.a 2:5c91d2d5:::benchmark_data_smithi179_77138_object40305:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-02-09T17:31:47.054 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:04.297886+0000", 2024-02-09T17:31:47.054 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 282.69846374100001, 2024-02-09T17:31:47.054 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.19289321800000001, 2024-02-09T17:31:47.054 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:31:47.054 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:31:47.054 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:31:47.054 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:31:47.054 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:31:47.055 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 173713 2024-02-09T17:31:47.055 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.055 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:31:47.055 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.055 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:31:47.055 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.297886+0000", 2024-02-09T17:31:47.055 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:47.055 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.055 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.055 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:31:47.055 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.297886+0000", 2024-02-09T17:31:47.055 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:47.055 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.055 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.055 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:31:47.056 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.297887+0000", 2024-02-09T17:31:47.056 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1340000000000001e-06 2024-02-09T17:31:47.056 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.056 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.056 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:31:47.056 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.297893+0000", 2024-02-09T17:31:47.056 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3700000000000003e-06 2024-02-09T17:31:47.056 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.056 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.056 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:31:47.056 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.297908+0000", 2024-02-09T17:31:47.056 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.504e-05 2024-02-09T17:31:47.056 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.057 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.057 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:31:47.057 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.297915+0000", 2024-02-09T17:31:47.057 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6020000000000002e-06 2024-02-09T17:31:47.057 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.057 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.057 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:31:47.057 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.301953+0000", 2024-02-09T17:31:47.057 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0040373529999999996 2024-02-09T17:31:47.057 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.057 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.057 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:31:47.057 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.302039+0000", 2024-02-09T17:31:47.057 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.6116e-05 2024-02-09T17:31:47.057 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.058 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.058 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,9", 2024-02-09T17:31:47.058 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.302130+0000", 2024-02-09T17:31:47.058 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.0859000000000005e-05 2024-02-09T17:31:47.058 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.058 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.058 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:31:47.058 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.305481+0000", 2024-02-09T17:31:47.058 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0033510129999999999 2024-02-09T17:31:47.058 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.058 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.058 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:47.058 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.320854+0000", 2024-02-09T17:31:47.059 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.015373658 2024-02-09T17:31:47.059 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.059 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.059 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:47.059 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.490718+0000", 2024-02-09T17:31:47.059 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.16986306000000001 2024-02-09T17:31:47.059 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.059 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.059 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:31:47.059 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.490760+0000", 2024-02-09T17:31:47.059 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2525999999999999e-05 2024-02-09T17:31:47.059 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.059 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.059 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:31:47.060 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.490780+0000", 2024-02-09T17:31:47.060 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9487000000000002e-05 2024-02-09T17:31:47.060 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:47.060 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:31:47.060 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:47.060 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.060 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.060 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:173721 2.a 2:526418d3:::benchmark_data_smithi179_77138_object40313:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-02-09T17:31:47.060 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:04.311408+0000", 2024-02-09T17:31:47.060 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 282.68494230099998, 2024-02-09T17:31:47.060 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.183574246, 2024-02-09T17:31:47.060 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:31:47.060 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:31:47.061 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:31:47.061 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:31:47.061 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:31:47.061 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 173721 2024-02-09T17:31:47.061 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.061 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:31:47.061 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.061 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:31:47.061 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.311408+0000", 2024-02-09T17:31:47.061 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:47.061 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.061 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.061 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:31:47.061 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.311408+0000", 2024-02-09T17:31:47.062 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:47.062 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.062 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.062 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:31:47.062 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.311409+0000", 2024-02-09T17:31:47.062 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0049999999999999e-06 2024-02-09T17:31:47.062 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.062 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.062 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:31:47.062 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.311413+0000", 2024-02-09T17:31:47.062 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4859999999999999e-06 2024-02-09T17:31:47.062 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.062 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.062 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:31:47.063 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.311424+0000", 2024-02-09T17:31:47.063 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0403e-05 2024-02-09T17:31:47.063 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.063 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.063 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:31:47.063 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.311430+0000", 2024-02-09T17:31:47.063 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1330000000000004e-06 2024-02-09T17:31:47.063 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.063 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.063 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:31:47.063 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.311479+0000", 2024-02-09T17:31:47.063 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8979e-05 2024-02-09T17:31:47.063 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.063 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.064 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:31:47.064 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.311557+0000", 2024-02-09T17:31:47.064 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7886000000000006e-05 2024-02-09T17:31:47.064 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.064 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.064 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,9", 2024-02-09T17:31:47.064 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.311638+0000", 2024-02-09T17:31:47.064 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1574000000000002e-05 2024-02-09T17:31:47.064 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.064 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.064 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:31:47.064 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.315588+0000", 2024-02-09T17:31:47.064 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0039498340000000002 2024-02-09T17:31:47.065 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.065 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.065 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:47.065 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.320939+0000", 2024-02-09T17:31:47.065 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0053514269999999997 2024-02-09T17:31:47.065 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.065 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.067 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:47.067 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.494887+0000", 2024-02-09T17:31:47.067 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.17394747799999999 2024-02-09T17:31:47.067 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.067 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.067 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:31:47.067 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.494942+0000", 2024-02-09T17:31:47.067 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5154000000000001e-05 2024-02-09T17:31:47.067 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.067 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.067 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:31:47.067 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.494982+0000", 2024-02-09T17:31:47.067 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9886999999999999e-05 2024-02-09T17:31:47.068 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:47.068 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:31:47.068 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:47.068 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.068 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.068 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:173735 2.a 2:56d1d430:::benchmark_data_smithi179_77138_object40327:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-02-09T17:31:47.068 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:04.331216+0000", 2024-02-09T17:31:47.068 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 282.665134034, 2024-02-09T17:31:47.068 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.16389076299999999, 2024-02-09T17:31:47.068 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:31:47.068 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:31:47.068 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:31:47.069 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:31:47.069 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:31:47.069 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 173735 2024-02-09T17:31:47.069 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.069 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:31:47.069 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.069 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:31:47.069 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.331216+0000", 2024-02-09T17:31:47.069 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:47.069 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.069 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.069 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:31:47.069 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.331216+0000", 2024-02-09T17:31:47.069 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:47.070 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.070 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.070 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:31:47.070 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.331218+0000", 2024-02-09T17:31:47.070 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9520000000000001e-06 2024-02-09T17:31:47.070 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.070 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.070 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:31:47.070 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.331230+0000", 2024-02-09T17:31:47.070 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2106000000000001e-05 2024-02-09T17:31:47.070 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.070 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.070 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:31:47.071 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.331245+0000", 2024-02-09T17:31:47.071 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5373e-05 2024-02-09T17:31:47.071 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.071 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.071 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:31:47.071 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.331255+0000", 2024-02-09T17:31:47.071 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.8269999999999993e-06 2024-02-09T17:31:47.071 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.071 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.071 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:31:47.071 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.331302+0000", 2024-02-09T17:31:47.071 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6808999999999999e-05 2024-02-09T17:31:47.071 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.071 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.071 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:31:47.072 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.331430+0000", 2024-02-09T17:31:47.072 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012804400000000001 2024-02-09T17:31:47.072 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.072 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.072 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,9", 2024-02-09T17:31:47.072 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.331521+0000", 2024-02-09T17:31:47.072 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.0412000000000001e-05 2024-02-09T17:31:47.072 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.072 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.072 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:31:47.072 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.335423+0000", 2024-02-09T17:31:47.072 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0039019829999999999 2024-02-09T17:31:47.072 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.073 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.073 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:47.073 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.335458+0000", 2024-02-09T17:31:47.073 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.5023e-05 2024-02-09T17:31:47.073 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.073 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.073 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:47.073 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.495032+0000", 2024-02-09T17:31:47.073 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15957394699999999 2024-02-09T17:31:47.073 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.073 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.073 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:31:47.073 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.495074+0000", 2024-02-09T17:31:47.073 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2463000000000003e-05 2024-02-09T17:31:47.074 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.074 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.074 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:31:47.074 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.495107+0000", 2024-02-09T17:31:47.074 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2824000000000003e-05 2024-02-09T17:31:47.074 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:47.074 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:31:47.074 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:47.074 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.074 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.074 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:173761 2.a 2:5adb3f55:::benchmark_data_smithi179_77138_object40353:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-02-09T17:31:47.074 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:04.370802+0000", 2024-02-09T17:31:47.074 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 282.62554766800002, 2024-02-09T17:31:47.074 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.128114384, 2024-02-09T17:31:47.075 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:31:47.075 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:31:47.075 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:31:47.075 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:31:47.075 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:31:47.075 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 173761 2024-02-09T17:31:47.075 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.075 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:31:47.075 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.075 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:31:47.075 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.370802+0000", 2024-02-09T17:31:47.075 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:47.075 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.075 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.075 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:31:47.076 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.370802+0000", 2024-02-09T17:31:47.076 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:47.076 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.076 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.076 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:31:47.076 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.370803+0000", 2024-02-09T17:31:47.076 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4600000000000004e-07 2024-02-09T17:31:47.076 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.076 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.076 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:31:47.076 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.370807+0000", 2024-02-09T17:31:47.076 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0300000000000004e-06 2024-02-09T17:31:47.076 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.076 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.077 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:31:47.077 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.370815+0000", 2024-02-09T17:31:47.077 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1149999999999994e-06 2024-02-09T17:31:47.077 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.077 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.077 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:31:47.077 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.370822+0000", 2024-02-09T17:31:47.077 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9280000000000004e-06 2024-02-09T17:31:47.077 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.077 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.077 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:31:47.077 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.370875+0000", 2024-02-09T17:31:47.077 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2380000000000003e-05 2024-02-09T17:31:47.077 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.077 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.078 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:31:47.078 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.370985+0000", 2024-02-09T17:31:47.078 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000109993 2024-02-09T17:31:47.078 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.078 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.078 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,9", 2024-02-09T17:31:47.078 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.371108+0000", 2024-02-09T17:31:47.078 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000123091 2024-02-09T17:31:47.078 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.078 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.078 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:31:47.078 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.375928+0000", 2024-02-09T17:31:47.078 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0048205840000000002 2024-02-09T17:31:47.078 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.079 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.079 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:47.079 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.376700+0000", 2024-02-09T17:31:47.079 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00077203700000000005 2024-02-09T17:31:47.079 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.079 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.079 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:47.079 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.498837+0000", 2024-02-09T17:31:47.079 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.122136288 2024-02-09T17:31:47.079 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.079 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.079 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:31:47.079 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.498889+0000", 2024-02-09T17:31:47.079 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1965999999999997e-05 2024-02-09T17:31:47.080 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.080 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.080 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:31:47.080 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.498917+0000", 2024-02-09T17:31:47.080 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.8226e-05 2024-02-09T17:31:47.080 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:47.080 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:31:47.080 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:47.080 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.080 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.080 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:173780 2.a 2:5f6bf9e0:::benchmark_data_smithi179_77138_object40372:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-02-09T17:31:47.080 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:04.403512+0000", 2024-02-09T17:31:47.080 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 282.59283764600002, 2024-02-09T17:31:47.080 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.109091655, 2024-02-09T17:31:47.080 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:31:47.081 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:31:47.081 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:31:47.081 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:31:47.081 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:31:47.081 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 173780 2024-02-09T17:31:47.081 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.081 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:31:47.081 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.081 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:31:47.081 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.403512+0000", 2024-02-09T17:31:47.081 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:47.081 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.081 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.081 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:31:47.082 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.403512+0000", 2024-02-09T17:31:47.082 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:47.082 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.082 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.082 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:31:47.082 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.403513+0000", 2024-02-09T17:31:47.082 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.068e-06 2024-02-09T17:31:47.082 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.082 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.082 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:31:47.082 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.403518+0000", 2024-02-09T17:31:47.082 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4909999999999999e-06 2024-02-09T17:31:47.082 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.082 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.083 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:31:47.083 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.403528+0000", 2024-02-09T17:31:47.083 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5210000000000008e-06 2024-02-09T17:31:47.083 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.083 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.083 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:31:47.083 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.403535+0000", 2024-02-09T17:31:47.083 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1400000000000002e-06 2024-02-09T17:31:47.083 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.083 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.083 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:31:47.083 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.403591+0000", 2024-02-09T17:31:47.083 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5965999999999999e-05 2024-02-09T17:31:47.083 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.084 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.084 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:31:47.084 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.403706+0000", 2024-02-09T17:31:47.084 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000115307 2024-02-09T17:31:47.084 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.084 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.084 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,9", 2024-02-09T17:31:47.084 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.403811+0000", 2024-02-09T17:31:47.084 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000105507 2024-02-09T17:31:47.084 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.084 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.084 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:47.084 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.412034+0000", 2024-02-09T17:31:47.084 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0082223490000000003 2024-02-09T17:31:47.085 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.085 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.085 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:31:47.085 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.415359+0000", 2024-02-09T17:31:47.085 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0033255400000000001 2024-02-09T17:31:47.085 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.085 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.085 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:47.085 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.512501+0000", 2024-02-09T17:31:47.085 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.097141874000000003 2024-02-09T17:31:47.085 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.085 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.085 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:31:47.085 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.512561+0000", 2024-02-09T17:31:47.085 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9851e-05 2024-02-09T17:31:47.086 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.086 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.086 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:31:47.086 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.512604+0000", 2024-02-09T17:31:47.086 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3040999999999997e-05 2024-02-09T17:31:47.086 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:47.086 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:31:47.086 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:47.086 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.086 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.086 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24817.0:17926 3.8 3:16bd42ec:::benchmark_data_smithi179_90440_object17925:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e208)", 2024-02-09T17:31:47.086 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:30:10.952580+0000", 2024-02-09T17:31:47.086 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 96.043770338000002, 2024-02-09T17:31:47.086 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.20727911800000001, 2024-02-09T17:31:47.087 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:31:47.087 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:31:47.087 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:31:47.087 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24817", 2024-02-09T17:31:47.087 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:31:47.087 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 17926 2024-02-09T17:31:47.087 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.087 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:31:47.087 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.087 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:31:47.087 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.952580+0000", 2024-02-09T17:31:47.087 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:47.087 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.087 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.088 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:31:47.088 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.952580+0000", 2024-02-09T17:31:47.088 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:47.088 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.088 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.088 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:31:47.088 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.952581+0000", 2024-02-09T17:31:47.088 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0419999999999999e-06 2024-02-09T17:31:47.088 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.088 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.088 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:31:47.088 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.952603+0000", 2024-02-09T17:31:47.088 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2453e-05 2024-02-09T17:31:47.088 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.088 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.089 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:31:47.089 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.952615+0000", 2024-02-09T17:31:47.089 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1567e-05 2024-02-09T17:31:47.089 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.089 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.089 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:31:47.089 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.952623+0000", 2024-02-09T17:31:47.089 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7200000000000006e-06 2024-02-09T17:31:47.089 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.089 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.089 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:31:47.089 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.952723+0000", 2024-02-09T17:31:47.089 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000100237 2024-02-09T17:31:47.089 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.089 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.089 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:31:47.090 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.952835+0000", 2024-02-09T17:31:47.090 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011260900000000001 2024-02-09T17:31:47.090 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.090 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.090 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:31:47.090 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.952954+0000", 2024-02-09T17:31:47.090 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000119127 2024-02-09T17:31:47.090 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.090 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.090 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:31:47.090 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.954299+0000", 2024-02-09T17:31:47.090 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001344114 2024-02-09T17:31:47.090 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.090 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.091 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:47.091 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.954781+0000", 2024-02-09T17:31:47.091 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00048271399999999997 2024-02-09T17:31:47.091 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.091 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.091 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:47.091 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:11.159746+0000", 2024-02-09T17:31:47.091 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.204964435 2024-02-09T17:31:47.091 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.091 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.091 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:31:47.091 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:11.159814+0000", 2024-02-09T17:31:47.091 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7871000000000001e-05 2024-02-09T17:31:47.091 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.091 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.092 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:31:47.092 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:11.159859+0000", 2024-02-09T17:31:47.092 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5228999999999997e-05 2024-02-09T17:31:47.092 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:47.092 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:31:47.092 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:47.092 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.092 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.092 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24817.0:17930 3.8 3:1fad3ea4:::benchmark_data_smithi179_90440_object17929:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e208)", 2024-02-09T17:31:47.092 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:30:10.954694+0000", 2024-02-09T17:31:47.092 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 96.041656110999995, 2024-02-09T17:31:47.092 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.20531634600000001, 2024-02-09T17:31:47.092 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:31:47.092 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:31:47.092 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:31:47.093 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24817", 2024-02-09T17:31:47.093 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:31:47.093 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 17930 2024-02-09T17:31:47.093 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.093 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:31:47.093 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.093 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:31:47.093 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.954694+0000", 2024-02-09T17:31:47.093 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:47.093 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.093 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.093 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:31:47.093 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.954694+0000", 2024-02-09T17:31:47.093 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:47.094 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.094 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.094 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:31:47.094 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.954695+0000", 2024-02-09T17:31:47.094 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.11e-06 2024-02-09T17:31:47.094 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.094 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.094 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:31:47.094 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.954714+0000", 2024-02-09T17:31:47.094 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9247000000000001e-05 2024-02-09T17:31:47.094 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.094 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.094 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:31:47.094 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.954726+0000", 2024-02-09T17:31:47.094 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.173e-05 2024-02-09T17:31:47.095 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.095 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.095 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:31:47.095 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.954734+0000", 2024-02-09T17:31:47.095 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6249999999999998e-06 2024-02-09T17:31:47.095 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.095 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.095 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:31:47.095 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.954829+0000", 2024-02-09T17:31:47.095 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5816999999999994e-05 2024-02-09T17:31:47.095 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.095 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.095 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:31:47.095 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.954946+0000", 2024-02-09T17:31:47.095 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011635500000000001 2024-02-09T17:31:47.096 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.096 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.096 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:31:47.096 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.955086+0000", 2024-02-09T17:31:47.096 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000140049 2024-02-09T17:31:47.096 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.096 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.096 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:31:47.096 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.955975+0000", 2024-02-09T17:31:47.096 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00088879900000000001 2024-02-09T17:31:47.096 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.096 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.096 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:47.096 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.959205+0000", 2024-02-09T17:31:47.097 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0032298769999999999 2024-02-09T17:31:47.097 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.097 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.097 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:47.097 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:11.159904+0000", 2024-02-09T17:31:47.097 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.20069910199999999 2024-02-09T17:31:47.097 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.097 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.097 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:31:47.097 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:11.159970+0000", 2024-02-09T17:31:47.097 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6291999999999995e-05 2024-02-09T17:31:47.097 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.097 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.097 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:31:47.097 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:11.160010+0000", 2024-02-09T17:31:47.097 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0343000000000003e-05 2024-02-09T17:31:47.098 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:47.098 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:31:47.098 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:47.098 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.098 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.098 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24817.0:17971 3.8 3:16a53f45:::benchmark_data_smithi179_90440_object17970:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e208)", 2024-02-09T17:31:47.098 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:30:10.965725+0000", 2024-02-09T17:31:47.098 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 96.030624611999997, 2024-02-09T17:31:47.098 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.194499441, 2024-02-09T17:31:47.098 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:31:47.098 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:31:47.098 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:31:47.098 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24817", 2024-02-09T17:31:47.098 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:31:47.099 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 17971 2024-02-09T17:31:47.099 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.099 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:31:47.099 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.099 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:31:47.099 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.965725+0000", 2024-02-09T17:31:47.099 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:47.099 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.099 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.099 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:31:47.099 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.965725+0000", 2024-02-09T17:31:47.099 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:47.099 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.099 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.100 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:31:47.100 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.965726+0000", 2024-02-09T17:31:47.100 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7000000000000003e-07 2024-02-09T17:31:47.100 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.100 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.100 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:31:47.100 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.965744+0000", 2024-02-09T17:31:47.100 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7935e-05 2024-02-09T17:31:47.100 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.100 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.100 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:31:47.100 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.965754+0000", 2024-02-09T17:31:47.101 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0190000000000001e-05 2024-02-09T17:31:47.101 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.101 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.101 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:31:47.101 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.965760+0000", 2024-02-09T17:31:47.101 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5550000000000002e-06 2024-02-09T17:31:47.101 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.101 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.101 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:31:47.101 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.965801+0000", 2024-02-09T17:31:47.101 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0980999999999999e-05 2024-02-09T17:31:47.101 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.101 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.101 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:31:47.101 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.965893+0000", 2024-02-09T17:31:47.102 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.1951000000000001e-05 2024-02-09T17:31:47.102 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.102 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.102 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:31:47.102 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.965984+0000", 2024-02-09T17:31:47.102 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.1308000000000001e-05 2024-02-09T17:31:47.102 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.102 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.102 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:31:47.102 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.966711+0000", 2024-02-09T17:31:47.102 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00072709599999999999 2024-02-09T17:31:47.102 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.102 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.102 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:47.102 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.967233+0000", 2024-02-09T17:31:47.103 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000521194 2024-02-09T17:31:47.103 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.103 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.103 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:47.103 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:11.160160+0000", 2024-02-09T17:31:47.103 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.19292753000000001 2024-02-09T17:31:47.103 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.103 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.103 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:31:47.103 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:11.160201+0000", 2024-02-09T17:31:47.103 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0998999999999998e-05 2024-02-09T17:31:47.103 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.103 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.103 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:31:47.104 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:11.160225+0000", 2024-02-09T17:31:47.104 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3832000000000002e-05 2024-02-09T17:31:47.104 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:47.104 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:31:47.104 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:47.104 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.104 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.104 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24817.0:17974 3.8 3:13dfb358:::benchmark_data_smithi179_90440_object17973:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e208)", 2024-02-09T17:31:47.104 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:30:10.967264+0000", 2024-02-09T17:31:47.104 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 96.029085838, 2024-02-09T17:31:47.104 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.193095355, 2024-02-09T17:31:47.104 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:31:47.104 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:31:47.104 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:31:47.105 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24817", 2024-02-09T17:31:47.105 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:31:47.105 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 17974 2024-02-09T17:31:47.105 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.105 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:31:47.105 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.105 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:31:47.105 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.967264+0000", 2024-02-09T17:31:47.105 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:47.105 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.105 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.105 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:31:47.105 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.967264+0000", 2024-02-09T17:31:47.105 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:47.106 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.106 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.106 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:31:47.106 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.967265+0000", 2024-02-09T17:31:47.106 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2200000000000003e-07 2024-02-09T17:31:47.106 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.106 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.106 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:31:47.106 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.967286+0000", 2024-02-09T17:31:47.106 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0499000000000001e-05 2024-02-09T17:31:47.106 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.106 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.106 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:31:47.106 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.967295+0000", 2024-02-09T17:31:47.107 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.3719999999999994e-06 2024-02-09T17:31:47.107 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.107 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.107 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:31:47.107 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.967302+0000", 2024-02-09T17:31:47.107 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2590000000000003e-06 2024-02-09T17:31:47.107 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.107 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.107 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:31:47.107 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.967341+0000", 2024-02-09T17:31:47.107 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9288000000000001e-05 2024-02-09T17:31:47.107 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.107 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.107 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:31:47.107 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.967433+0000", 2024-02-09T17:31:47.108 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.1874999999999997e-05 2024-02-09T17:31:47.108 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.108 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.108 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:31:47.108 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.967546+0000", 2024-02-09T17:31:47.108 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011241199999999999 2024-02-09T17:31:47.108 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.108 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.108 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:31:47.108 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.968153+0000", 2024-02-09T17:31:47.108 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00060681999999999999 2024-02-09T17:31:47.108 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.108 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.108 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:47.109 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.969239+0000", 2024-02-09T17:31:47.109 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0010867419999999999 2024-02-09T17:31:47.109 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.109 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.109 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:47.109 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:11.160277+0000", 2024-02-09T17:31:47.109 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.191037811 2024-02-09T17:31:47.109 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.109 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.109 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:31:47.109 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:11.160324+0000", 2024-02-09T17:31:47.109 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7222000000000003e-05 2024-02-09T17:31:47.109 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.109 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.110 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:31:47.110 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:11.160360+0000", 2024-02-09T17:31:47.110 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.5233e-05 2024-02-09T17:31:47.110 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:47.110 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:31:47.110 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:47.110 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.110 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.110 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24817.0:17983 3.8 3:10bcfb25:::benchmark_data_smithi179_90440_object17982:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e208)", 2024-02-09T17:31:47.110 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:30:10.969671+0000", 2024-02-09T17:31:47.110 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 96.026679014999999, 2024-02-09T17:31:47.110 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.19078999999999999, 2024-02-09T17:31:47.110 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:31:47.111 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:31:47.111 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:31:47.111 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24817", 2024-02-09T17:31:47.111 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:31:47.111 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 17983 2024-02-09T17:31:47.111 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.111 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:31:47.111 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.111 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:31:47.111 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.969671+0000", 2024-02-09T17:31:47.111 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:47.111 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.111 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.111 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:31:47.111 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.969671+0000", 2024-02-09T17:31:47.112 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:47.112 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.112 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.112 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:31:47.112 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.969672+0000", 2024-02-09T17:31:47.112 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.9100000000000003e-07 2024-02-09T17:31:47.112 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.112 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.112 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:31:47.112 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.969688+0000", 2024-02-09T17:31:47.112 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6336000000000001e-05 2024-02-09T17:31:47.112 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.112 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.112 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:31:47.112 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.969697+0000", 2024-02-09T17:31:47.113 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.6179999999999995e-06 2024-02-09T17:31:47.113 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.113 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.113 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:31:47.113 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.969704+0000", 2024-02-09T17:31:47.113 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8220000000000001e-06 2024-02-09T17:31:47.113 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.113 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.113 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:31:47.113 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.969741+0000", 2024-02-09T17:31:47.113 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7357000000000002e-05 2024-02-09T17:31:47.113 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.113 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.114 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:31:47.114 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.969844+0000", 2024-02-09T17:31:47.114 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010334400000000001 2024-02-09T17:31:47.114 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.114 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.114 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:31:47.114 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.969931+0000", 2024-02-09T17:31:47.114 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.6804000000000004e-05 2024-02-09T17:31:47.114 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.114 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.114 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:31:47.114 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.970409+0000", 2024-02-09T17:31:47.114 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00047774500000000001 2024-02-09T17:31:47.114 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.114 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.115 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:47.115 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.971159+0000", 2024-02-09T17:31:47.115 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00074971900000000004 2024-02-09T17:31:47.115 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.115 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.115 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:47.115 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:11.160388+0000", 2024-02-09T17:31:47.115 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.18922963200000001 2024-02-09T17:31:47.115 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.115 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.115 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:31:47.115 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:11.160431+0000", 2024-02-09T17:31:47.115 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2663999999999997e-05 2024-02-09T17:31:47.115 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.116 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.116 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:31:47.116 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:11.160461+0000", 2024-02-09T17:31:47.116 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0168e-05 2024-02-09T17:31:47.116 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:47.116 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:31:47.116 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:47.116 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.116 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.116 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24817.0:17988 3.8 3:1ea28970:::benchmark_data_smithi179_90440_object17987:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e208)", 2024-02-09T17:31:47.116 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:30:10.973067+0000", 2024-02-09T17:31:47.116 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 96.023282726999994, 2024-02-09T17:31:47.116 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.187513281, 2024-02-09T17:31:47.117 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:31:47.117 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:31:47.117 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:31:47.117 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24817", 2024-02-09T17:31:47.117 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:31:47.117 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 17988 2024-02-09T17:31:47.117 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.117 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:31:47.117 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.117 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:31:47.117 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.973067+0000", 2024-02-09T17:31:47.117 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:47.117 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.117 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.117 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:31:47.118 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.973067+0000", 2024-02-09T17:31:47.118 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:31:47.118 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.118 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.118 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:31:47.118 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.973068+0000", 2024-02-09T17:31:47.118 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7700000000000004e-07 2024-02-09T17:31:47.118 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.118 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.118 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:31:47.118 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.973090+0000", 2024-02-09T17:31:47.118 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1441e-05 2024-02-09T17:31:47.118 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.118 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.119 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:31:47.119 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.973099+0000", 2024-02-09T17:31:47.119 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.3989999999999993e-06 2024-02-09T17:31:47.119 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.119 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.119 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:31:47.119 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.973107+0000", 2024-02-09T17:31:47.119 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2989999999999997e-06 2024-02-09T17:31:47.119 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.119 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.119 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:31:47.119 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.973156+0000", 2024-02-09T17:31:47.119 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8733000000000002e-05 2024-02-09T17:31:47.119 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.119 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.120 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:31:47.120 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.973257+0000", 2024-02-09T17:31:47.120 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000101297 2024-02-09T17:31:47.120 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.120 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.120 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:31:47.120 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.973350+0000", 2024-02-09T17:31:47.120 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2287999999999995e-05 2024-02-09T17:31:47.120 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.120 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.120 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:31:47.120 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.973973+0000", 2024-02-09T17:31:47.120 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00062315299999999999 2024-02-09T17:31:47.120 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.120 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.121 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:47.121 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.974502+0000", 2024-02-09T17:31:47.121 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00052891400000000001 2024-02-09T17:31:47.121 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.121 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.121 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:31:47.121 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:11.160502+0000", 2024-02-09T17:31:47.121 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.186000729 2024-02-09T17:31:47.121 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.121 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.121 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:31:47.121 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:11.160549+0000", 2024-02-09T17:31:47.121 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6536999999999998e-05 2024-02-09T17:31:47.121 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:31:47.121 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:31:47.122 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:31:47.122 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:11.160581+0000", 2024-02-09T17:31:47.122 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1714000000000002e-05 2024-02-09T17:31:47.122 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:47.122 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:31:47.122 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:47.122 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:31:47.122 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:31:47.122 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:31:47.123 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:47.222 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:31:47.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:47.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:31:47.326 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:31:47.354 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:47.452 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:31:47.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:47.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:31:47.491 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:31:47.510 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:31:47.529 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:31:47.548 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:31:47.556 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:31:47.568 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:31:47.587 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:31:47.588 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:47.608 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:31:47.610 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:31:47.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:47.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:31:47.628 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:31:47.650 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:31:47.670 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:31:47.690 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:31:47.709 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:31:47.711 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:31:47.711 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:31:47.711 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:31:47.711 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:31:47.711 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:31:47.711 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:31:47.711 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:31:47.711 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:31:47.711 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:31:47.711 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:31:47.711 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:31:47.714 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:31:47.742 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:47.759 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset noscrub 2024-02-09T17:31:47.762 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:31:47.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:47.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:31:47.856 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.8 dump_ops_in_flight 2024-02-09T17:31:47.866 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:31:47.883 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:47 smithi179 ceph-mon[43975]: pgmap v685: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s wr, 37 op/s 2024-02-09T17:31:47.883 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:47 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2211164132' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:31:47.890 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:47 smithi099 ceph-mon[39420]: pgmap v685: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s wr, 37 op/s 2024-02-09T17:31:47.890 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:47 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2211164132' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:31:47.891 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:47.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:47 smithi162 ceph-mon[44304]: pgmap v685: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s wr, 37 op/s 2024-02-09T17:31:47.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:47 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2211164132' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:31:47.952 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 102 16 131956 131940 80.8341 0 - 0.0107818 2024-02-09T17:31:48.014 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:31:48.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:48.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:31:48.116 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:31:48.118 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:31:48.147 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:48.167 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:31:48.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:48.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:31:48.271 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:31:48.299 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:48.396 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:31:48.399 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:48.399 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:31:48.500 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:31:48.531 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:48.953 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 103 16 131956 131940 80.0493 0 - 0.0107818 2024-02-09T17:31:49.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:49 smithi179 ceph-mon[43975]: pgmap v686: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s wr, 35 op/s 2024-02-09T17:31:49.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:49 smithi162 ceph-mon[44304]: pgmap v686: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s wr, 35 op/s 2024-02-09T17:31:49.953 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Total time run: 103.818 2024-02-09T17:31:49.953 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Total writes made: 131956 2024-02-09T17:31:49.953 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Write size: 65536 2024-02-09T17:31:49.953 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Object size: 65536 2024-02-09T17:31:49.954 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Bandwidth (MB/sec): 79.4397 2024-02-09T17:31:49.954 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Stddev Bandwidth: 52.8371 2024-02-09T17:31:49.954 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Max bandwidth (MB/sec): 282.812 2024-02-09T17:31:49.954 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Min bandwidth (MB/sec): 0 2024-02-09T17:31:49.954 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Average IOPS: 1271 2024-02-09T17:31:49.954 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Stddev IOPS: 845.394 2024-02-09T17:31:49.954 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Max IOPS: 4525 2024-02-09T17:31:49.954 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Min IOPS: 0 2024-02-09T17:31:49.954 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Average Latency(s): 0.0125847 2024-02-09T17:31:49.954 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Stddev Latency(s): 0.164457 2024-02-09T17:31:49.954 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Max latency(s): 15.0213 2024-02-09T17:31:49.954 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Min latency(s): 0.00110138 2024-02-09T17:31:49.962 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Cleaning up (deleting benchmark objects) 2024-02-09T17:31:49.991 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T17:31:49.991 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:31:49.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:49 smithi099 ceph-mon[39420]: pgmap v686: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s wr, 35 op/s 2024-02-09T17:31:50.744 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.8 dump_blocked_ops 2024-02-09T17:31:51.093 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:31:51.501 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:31:51.779 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:51 smithi099 ceph-mon[39420]: pgmap v687: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-02-09T17:31:51.786 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:31:51.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:51.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:31:51.890 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] 2024-02-09T17:31:51.890 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:31:51.890 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-02-09T17:31:51.891 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd out 0 2024-02-09T17:31:51.893 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:31:51.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:51 smithi179 ceph-mon[43975]: pgmap v687: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-02-09T17:31:51.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:51 smithi162 ceph-mon[44304]: pgmap v687: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-02-09T17:31:51.960 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:51.988 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:31:51.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:51.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:31:52.096 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:31:52.139 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:52.345 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:31:52.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:52.351 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:31:52.452 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:31:52.484 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:52.945 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:31:52.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:52.952 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:31:53.053 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:31:53.087 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:53.247 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:31:53.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:53.250 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:31:53.255 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T17:31:53.256 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:31:53.351 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:31:53.398 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:53.506 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:31:53.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:53.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:31:53.547 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:31:53.612 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:31:53.644 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:53.646 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:31:53.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:53 smithi179 ceph-mon[43975]: pgmap v688: 17 pgs: 17 active+clean; 8.0 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 102 B/s wr, 73 op/s 2024-02-09T17:31:53.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:53 smithi162 ceph-mon[44304]: pgmap v688: 17 pgs: 17 active+clean; 8.0 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 102 B/s wr, 73 op/s 2024-02-09T17:31:53.936 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:53 smithi099 ceph-mon[39420]: pgmap v688: 17 pgs: 17 active+clean; 8.0 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 102 B/s wr, 73 op/s 2024-02-09T17:31:53.994 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:31:53.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:53.999 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:31:54.011 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.8 dump_historic_ops 2024-02-09T17:31:54.100 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:31:54.145 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:54.164 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:31:54.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:54.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:31:54.275 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:31:54.307 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:54.341 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:31:54.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:54.348 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:31:54.415 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:31:54.450 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:31:54.504 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:54.531 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:31:54.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:54.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:31:54.575 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:31:54.641 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:31:54.693 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:54.696 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is unset 2024-02-09T17:31:54.907 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:31:54.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:54.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:31:54.912 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:54 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:31:54.912 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:54 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:31:54.912 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:54 smithi099 ceph-mon[39420]: osdmap e235: 12 total, 12 up, 12 in 2024-02-09T17:31:54.912 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:54 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2180991548' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:31:54.912 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:54 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:31:54.996 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:54 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:31:54.996 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:54 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:31:54.996 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:54 smithi179 ceph-mon[43975]: osdmap e235: 12 total, 12 up, 12 in 2024-02-09T17:31:54.996 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:54 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2180991548' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:31:54.996 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:54 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:31:55.011 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:31:55.050 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:55.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:54 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:31:55.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:54 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:31:55.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:54 smithi162 ceph-mon[44304]: osdmap e235: 12 total, 12 up, 12 in 2024-02-09T17:31:55.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:54 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2180991548' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:31:55.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:54 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:31:55.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:55 smithi179 ceph-mon[43975]: pgmap v690: 17 pgs: 17 active+clean; 7.9 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 103 B/s rd, 103 B/s wr, 292 op/s 2024-02-09T17:31:55.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:55 smithi179 ceph-mon[43975]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:31:55.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:55 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:31:55.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:55 smithi179 ceph-mon[43975]: osdmap e236: 12 total, 12 up, 12 in 2024-02-09T17:31:55.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:55 smithi099 ceph-mon[39420]: pgmap v690: 17 pgs: 17 active+clean; 7.9 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 103 B/s rd, 103 B/s wr, 292 op/s 2024-02-09T17:31:55.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:55 smithi099 ceph-mon[39420]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:31:55.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:55 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:31:55.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:55 smithi099 ceph-mon[39420]: osdmap e236: 12 total, 12 up, 12 in 2024-02-09T17:31:56.117 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:31:56.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:56.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:31:56.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:55 smithi162 ceph-mon[44304]: pgmap v690: 17 pgs: 17 active+clean; 7.9 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 103 B/s rd, 103 B/s wr, 292 op/s 2024-02-09T17:31:56.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:55 smithi162 ceph-mon[44304]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:31:56.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:55 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:31:56.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:55 smithi162 ceph-mon[44304]: osdmap e236: 12 total, 12 up, 12 in 2024-02-09T17:31:56.222 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:31:56.280 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:56.317 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:31:56.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:56.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:31:56.425 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:31:56.476 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:56.589 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:31:56.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:56.593 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:31:56.628 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T17:31:56.628 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:31:56.694 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:31:56.733 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:56.868 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:31:56.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:56.873 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:31:56.974 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:31:57.019 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:57.052 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:31:57.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:57.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:31:57.159 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:31:57.200 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:57.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:57 smithi162 ceph-mon[44304]: pgmap v692: 17 pgs: 17 active+clean; 7.8 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 127 B/s wr, 445 op/s 2024-02-09T17:31:57.457 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:57 smithi099 ceph-mon[39420]: pgmap v692: 17 pgs: 17 active+clean; 7.8 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 127 B/s wr, 445 op/s 2024-02-09T17:31:57.462 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:31:57.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:57.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:31:57.494 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:57 smithi179 ceph-mon[43975]: pgmap v692: 17 pgs: 17 active+clean; 7.8 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 127 B/s wr, 445 op/s 2024-02-09T17:31:57.495 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-09T17:31:57.569 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:31:57.624 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:57.853 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:31:57.881 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:31:57.903 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:31:57.940 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:31:57.987 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:31:57.993 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:31:57.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:57.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:31:58.030 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:31:58.058 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:31:58.085 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:31:58.099 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:31:58.113 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:31:58.129 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-09T17:31:58.144 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:31:58.158 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:58.185 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:31:58.193 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:31:58.202 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:58.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:31:58.219 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:31:58.222 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:31:58.222 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:31:58.222 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:31:58.223 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:31:58.223 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:31:58.223 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:31:58.223 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:31:58.223 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:31:58.223 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:31:58.223 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:31:58.223 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:31:58.223 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:31:58.303 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:31:58.379 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:58.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:58 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3867949319' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:31:58.606 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:31:58.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:58.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:31:58.612 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:58 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3867949319' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:31:58.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:58 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3867949319' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:31:58.713 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:31:58.773 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:58.805 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:31:58.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:58.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:31:58.917 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:31:58.953 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:58 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mon-a[39397]: 2024-02-09T17:31:58.718+0000 7feb57590700 -1 mon.a@0(leader).osd e236 definitely_dead 0 2024-02-09T17:31:58.956 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:59.081 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:31:59.100 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:31:59.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:59.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:31:59.125 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:31:59.212 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:31:59.246 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:59.277 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:31:59.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:59.281 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:31:59.287 INFO:teuthology.orchestra.run.smithi099.stderr:marked out osd.0. 2024-02-09T17:31:59.381 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:31:59.408 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:59.428 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:31:59.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:59.431 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:31:59.532 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:31:59.556 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:59 smithi099 ceph-mon[39420]: 3.b scrub starts 2024-02-09T17:31:59.556 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:59 smithi099 ceph-mon[39420]: 3.4 scrub starts 2024-02-09T17:31:59.556 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:59 smithi099 ceph-mon[39420]: pgmap v693: 17 pgs: 17 active+clean; 7.7 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 127 B/s wr, 682 op/s 2024-02-09T17:31:59.556 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:59 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1221737630' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-02-09T17:31:59.557 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:59 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-02-09T17:31:59.557 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:31:59 smithi099 ceph-mon[39420]: Client client.admin marked osd.0 out, while it was still marked up 2024-02-09T17:31:59.559 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:59.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:59 smithi179 ceph-mon[43975]: 3.b scrub starts 2024-02-09T17:31:59.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:59 smithi179 ceph-mon[43975]: 3.4 scrub starts 2024-02-09T17:31:59.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:59 smithi179 ceph-mon[43975]: pgmap v693: 17 pgs: 17 active+clean; 7.7 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 127 B/s wr, 682 op/s 2024-02-09T17:31:59.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:59 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1221737630' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-02-09T17:31:59.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:59 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-02-09T17:31:59.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:31:59 smithi179 ceph-mon[43975]: Client client.admin marked osd.0 out, while it was still marked up 2024-02-09T17:31:59.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:59 smithi162 ceph-mon[44304]: 3.b scrub starts 2024-02-09T17:31:59.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:59 smithi162 ceph-mon[44304]: 3.4 scrub starts 2024-02-09T17:31:59.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:59 smithi162 ceph-mon[44304]: pgmap v693: 17 pgs: 17 active+clean; 7.7 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 127 B/s wr, 682 op/s 2024-02-09T17:31:59.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:59 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1221737630' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-02-09T17:31:59.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:59 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-02-09T17:31:59.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:31:59 smithi162 ceph-mon[44304]: Client client.admin marked osd.0 out, while it was still marked up 2024-02-09T17:31:59.706 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:31:59.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:59.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:31:59.810 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:31:59.837 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:31:59.855 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:31:59.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:31:59.858 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:31:59.959 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:31:59.986 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:00.013 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:32:00.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:00.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:32:00.117 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:32:00.143 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:00.196 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:32:00.199 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:00.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:32:00.301 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:32:00.332 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:00.355 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:32:00.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:00.357 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:32:00.458 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:32:00.486 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:00.505 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:32:00.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:00.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:32:00.609 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:32:00.625 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:00 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-02-09T17:32:00.625 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:00 smithi099 ceph-mon[39420]: osdmap e237: 12 total, 12 up, 11 in 2024-02-09T17:32:00.625 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:00 smithi099 ceph-mon[39420]: 1.0 scrub starts 2024-02-09T17:32:00.625 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:00 smithi099 ceph-mon[39420]: 1.0 scrub ok 2024-02-09T17:32:00.626 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:00 smithi099 ceph-mon[39420]: 3.4 scrub ok 2024-02-09T17:32:00.639 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:00.660 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:32:00.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:00.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:32:00.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:00 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-02-09T17:32:00.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:00 smithi179 ceph-mon[43975]: osdmap e237: 12 total, 12 up, 11 in 2024-02-09T17:32:00.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:00 smithi179 ceph-mon[43975]: 1.0 scrub starts 2024-02-09T17:32:00.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:00 smithi179 ceph-mon[43975]: 1.0 scrub ok 2024-02-09T17:32:00.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:00 smithi179 ceph-mon[43975]: 3.4 scrub ok 2024-02-09T17:32:00.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:00 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-02-09T17:32:00.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:00 smithi162 ceph-mon[44304]: osdmap e237: 12 total, 12 up, 11 in 2024-02-09T17:32:00.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:00 smithi162 ceph-mon[44304]: 1.0 scrub starts 2024-02-09T17:32:00.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:00 smithi162 ceph-mon[44304]: 1.0 scrub ok 2024-02-09T17:32:00.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:00 smithi162 ceph-mon[44304]: 3.4 scrub ok 2024-02-09T17:32:00.765 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:32:00.793 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:00.812 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:32:00.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:00.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:32:00.916 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:32:00.943 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:00.962 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:32:00.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:00.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:32:01.066 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:32:01.093 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:01.111 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:32:01.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:01.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:32:01.116 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:32:01.214 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:32:01.242 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:01.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:01 smithi162 ceph-mon[44304]: osdmap e238: 12 total, 12 up, 11 in 2024-02-09T17:32:01.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:01 smithi162 ceph-mon[44304]: pgmap v696: 17 pgs: 1 peering, 1 unknown, 15 active+clean; 7.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 867 op/s 2024-02-09T17:32:01.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:01 smithi099 ceph-mon[39420]: osdmap e238: 12 total, 12 up, 11 in 2024-02-09T17:32:01.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:01 smithi099 ceph-mon[39420]: pgmap v696: 17 pgs: 1 peering, 1 unknown, 15 active+clean; 7.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 867 op/s 2024-02-09T17:32:01.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:01 smithi179 ceph-mon[43975]: osdmap e238: 12 total, 12 up, 11 in 2024-02-09T17:32:01.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:01 smithi179 ceph-mon[43975]: pgmap v696: 17 pgs: 1 peering, 1 unknown, 15 active+clean; 7.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 867 op/s 2024-02-09T17:32:02.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:02 smithi179 ceph-mon[43975]: 3.b starting backfill to osd.11 from (0'0,0'0] MAX to 236'12111 2024-02-09T17:32:02.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:02 smithi179 ceph-mon[43975]: osdmap e239: 12 total, 12 up, 11 in 2024-02-09T17:32:02.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:02 smithi162 ceph-mon[44304]: 3.b starting backfill to osd.11 from (0'0,0'0] MAX to 236'12111 2024-02-09T17:32:02.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:02 smithi162 ceph-mon[44304]: osdmap e239: 12 total, 12 up, 11 in 2024-02-09T17:32:02.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:02 smithi099 ceph-mon[39420]: 3.b starting backfill to osd.11 from (0'0,0'0] MAX to 236'12111 2024-02-09T17:32:02.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:02 smithi099 ceph-mon[39420]: osdmap e239: 12 total, 12 up, 11 in 2024-02-09T17:32:03.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:03 smithi179 ceph-mon[43975]: pgmap v698: 17 pgs: 1 peering, 1 active+remapped+backfilling, 1 unknown, 14 active+clean; 7.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 962 op/s; 7607/367854 objects misplaced (2.068%); 11 KiB/s, 0 objects/s recovering 2024-02-09T17:32:03.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:03 smithi162 ceph-mon[44304]: pgmap v698: 17 pgs: 1 peering, 1 active+remapped+backfilling, 1 unknown, 14 active+clean; 7.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 962 op/s; 7607/367854 objects misplaced (2.068%); 11 KiB/s, 0 objects/s recovering 2024-02-09T17:32:03.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:03 smithi099 ceph-mon[39420]: pgmap v698: 17 pgs: 1 peering, 1 active+remapped+backfilling, 1 unknown, 14 active+clean; 7.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 962 op/s; 7607/367854 objects misplaced (2.068%); 11 KiB/s, 0 objects/s recovering 2024-02-09T17:32:04.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:04 smithi179 ceph-mon[43975]: Health check failed: Degraded data redundancy: 128/365409 objects degraded (0.035%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:32:04.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:04 smithi162 ceph-mon[44304]: Health check failed: Degraded data redundancy: 128/365409 objects degraded (0.035%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:32:04.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:04 smithi099 ceph-mon[39420]: Health check failed: Degraded data redundancy: 128/365409 objects degraded (0.035%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:32:05.912 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:05 smithi099 ceph-mon[39420]: pgmap v699: 17 pgs: 1 active+remapped+backfilling, 1 unknown, 3 active+recovering+undersized+degraded+remapped, 12 active+clean; 7.4 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 781 op/s; 128/365409 objects degraded (0.035%); 30419/365409 objects misplaced (8.325%); 2.0 MiB/s, 32 objects/s recovering 2024-02-09T17:32:05.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:05 smithi179 ceph-mon[43975]: pgmap v699: 17 pgs: 1 active+remapped+backfilling, 1 unknown, 3 active+recovering+undersized+degraded+remapped, 12 active+clean; 7.4 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 781 op/s; 128/365409 objects degraded (0.035%); 30419/365409 objects misplaced (8.325%); 2.0 MiB/s, 32 objects/s recovering 2024-02-09T17:32:05.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:05 smithi162 ceph-mon[44304]: pgmap v699: 17 pgs: 1 active+remapped+backfilling, 1 unknown, 3 active+recovering+undersized+degraded+remapped, 12 active+clean; 7.4 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 781 op/s; 128/365409 objects degraded (0.035%); 30419/365409 objects misplaced (8.325%); 2.0 MiB/s, 32 objects/s recovering 2024-02-09T17:32:05.971 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:32:05.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:05.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:32:06.080 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:32:06.114 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:06.297 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:32:06.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:06.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:32:06.315 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:32:06.401 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:32:06.441 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:06.995 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:32:06.995 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [], 2024-02-09T17:32:06.995 INFO:teuthology.orchestra.run.smithi099.stdout: "num_ops": 0 2024-02-09T17:32:06.996 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:32:07.494 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:32:07.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:07.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:32:07.602 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:32:07.641 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:07.675 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:32:07.679 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is unset 2024-02-09T17:32:07.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:07.681 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:32:07.782 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:32:07.836 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:07.869 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:32:07.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:07.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:32:07.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:07 smithi179 ceph-mon[43975]: pgmap v700: 17 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 7.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 488 op/s; 145/364257 objects degraded (0.040%); 37841/364257 objects misplaced (10.389%); 2.7 MiB/s, 43 objects/s recovering 2024-02-09T17:32:07.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:07 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1927988898' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:32:07.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:07 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:32:07.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:07 smithi162 ceph-mon[44304]: pgmap v700: 17 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 7.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 488 op/s; 145/364257 objects degraded (0.040%); 37841/364257 objects misplaced (10.389%); 2.7 MiB/s, 43 objects/s recovering 2024-02-09T17:32:07.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:07 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1927988898' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:32:07.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:07 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:32:07.975 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:32:07.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:07 smithi099 ceph-mon[39420]: pgmap v700: 17 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 7.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 488 op/s; 145/364257 objects degraded (0.040%); 37841/364257 objects misplaced (10.389%); 2.7 MiB/s, 43 objects/s recovering 2024-02-09T17:32:07.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:07 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1927988898' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:32:07.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:07 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:32:08.022 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:08.048 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:32:08.051 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:08.051 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:32:08.152 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:32:08.179 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:08.204 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-09T17:32:08.207 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:32:08.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:08.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:32:08.317 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:32:08.376 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:08.408 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:32:08.421 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:08.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:32:08.523 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:32:08.559 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:08.584 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:32:08.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:08.589 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:32:08.690 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:32:08.732 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:08.771 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:32:08.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:08.781 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:32:08.857 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:08 smithi099 ceph-mon[39420]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:32:08.857 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:08 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:32:08.857 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:08 smithi099 ceph-mon[39420]: osdmap e240: 12 total, 12 up, 11 in 2024-02-09T17:32:08.857 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:08 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:32:08.857 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:08 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:32:08.882 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:32:08.913 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:08.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:08 smithi179 ceph-mon[43975]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:32:08.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:08 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:32:08.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:08 smithi179 ceph-mon[43975]: osdmap e240: 12 total, 12 up, 11 in 2024-02-09T17:32:08.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:08 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:32:08.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:08 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:32:09.164 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:32:09.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:09.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:32:09.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:08 smithi162 ceph-mon[44304]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:32:09.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:08 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:32:09.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:08 smithi162 ceph-mon[44304]: osdmap e240: 12 total, 12 up, 11 in 2024-02-09T17:32:09.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:08 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:32:09.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:08 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:32:09.271 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:32:09.303 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:09.754 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:32:09.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:09.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:32:09.858 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:32:09.900 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:09.931 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:32:09.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:09.935 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:32:09.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:09 smithi099 ceph-mon[39420]: pgmap v702: 17 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 7.3 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 448 op/s; 145/360777 objects degraded (0.040%); 36672/360777 objects misplaced (10.165%); 10 MiB/s, 161 objects/s recovering 2024-02-09T17:32:09.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:09 smithi099 ceph-mon[39420]: osdmap e241: 12 total, 12 up, 11 in 2024-02-09T17:32:10.036 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:32:10.073 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:09 smithi179 ceph-mon[43975]: pgmap v702: 17 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 7.3 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 448 op/s; 145/360777 objects degraded (0.040%); 36672/360777 objects misplaced (10.165%); 10 MiB/s, 161 objects/s recovering 2024-02-09T17:32:10.073 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:09 smithi179 ceph-mon[43975]: osdmap e241: 12 total, 12 up, 11 in 2024-02-09T17:32:10.076 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:10.117 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:32:10.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:10.121 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:32:10.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:09 smithi162 ceph-mon[44304]: pgmap v702: 17 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 7.3 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 448 op/s; 145/360777 objects degraded (0.040%); 36672/360777 objects misplaced (10.165%); 10 MiB/s, 161 objects/s recovering 2024-02-09T17:32:10.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:09 smithi162 ceph-mon[44304]: osdmap e241: 12 total, 12 up, 11 in 2024-02-09T17:32:10.222 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:32:10.253 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:10.376 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:32:10.383 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:32:10.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:10.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:32:10.413 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:32:10.446 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:32:10.468 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:32:10.486 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:32:10.494 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:32:10.518 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:10.534 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:32:10.559 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:32:10.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:10.567 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:32:10.571 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:32:10.608 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:32:10.637 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:32:10.665 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:32:10.667 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:32:10.693 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:32:10.725 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:32:10.726 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:10.752 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:32:10.756 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:32:10.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:10.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:32:10.769 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:32:10.769 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:32:10.769 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:32:10.769 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:32:10.769 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:32:10.769 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:32:10.769 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:32:10.769 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:32:10.770 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:32:10.770 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:32:10.770 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:32:10.770 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:32:10.857 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:32:10.890 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set noscrub 2024-02-09T17:32:10.895 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:10.914 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:32:10.918 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:10.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:32:10.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:10 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3027399964' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:32:10.959 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] 2024-02-09T17:32:10.960 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:32:10.960 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-02-09T17:32:10.960 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph pg dump --format=json 2024-02-09T17:32:10.990 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:10 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3027399964' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:32:11.020 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:32:11.079 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:10 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3027399964' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:32:11.090 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:11.130 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:32:11.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:11.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:32:11.237 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:32:11.297 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:11.332 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:32:11.339 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:11.339 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:32:11.440 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:32:11.502 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:11.538 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:32:11.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:11.545 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:32:11.647 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:32:11.685 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:11.895 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:32:11.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:11.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:32:11.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:11 smithi179 ceph-mon[43975]: pgmap v704: 17 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 7.2 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 498 op/s; 353/355881 objects degraded (0.099%); 35947/355881 objects misplaced (10.101%); 14 MiB/s, 225 objects/s recovering 2024-02-09T17:32:11.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:11 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 353/355881 objects degraded (0.099%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:32:11.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:11 smithi099 ceph-mon[39420]: pgmap v704: 17 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 7.2 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 498 op/s; 353/355881 objects degraded (0.099%); 35947/355881 objects misplaced (10.101%); 14 MiB/s, 225 objects/s recovering 2024-02-09T17:32:11.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:11 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 353/355881 objects degraded (0.099%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:32:11.999 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:32:12.031 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:12.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:11 smithi162 ceph-mon[44304]: pgmap v704: 17 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 7.2 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 498 op/s; 353/355881 objects degraded (0.099%); 35947/355881 objects misplaced (10.101%); 14 MiB/s, 225 objects/s recovering 2024-02-09T17:32:12.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:11 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 353/355881 objects degraded (0.099%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:32:12.237 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:32:12.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:12.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:32:12.341 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:32:12.381 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:12.420 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:32:12.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:12.423 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:32:12.523 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:32:12.580 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:12.624 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:32:12.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:12.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:32:12.733 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:32:12.782 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:12.833 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:32:12.837 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:12 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:32:12.837 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:12 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:32:12.837 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:12 smithi162 ceph-mon[44304]: osdmap e242: 12 total, 12 up, 11 in 2024-02-09T17:32:12.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:12.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:32:12.911 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:32:12.917 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:32:12.938 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:32:12.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:12 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:32:12.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:12 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:32:12.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:12 smithi099 ceph-mon[39420]: osdmap e242: 12 total, 12 up, 11 in 2024-02-09T17:32:12.996 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:12 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:32:12.996 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:12 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:32:12.996 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:12 smithi179 ceph-mon[43975]: osdmap e242: 12 total, 12 up, 11 in 2024-02-09T17:32:12.998 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:13.024 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:32:13.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:13.033 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:32:13.134 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:32:13.176 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:13.206 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:32:13.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:13.211 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:32:13.312 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:32:13.377 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:13.430 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:32:13.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:13.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:32:13.535 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:32:13.576 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:13.724 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:13 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:32:13.724 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:13 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:32:13.724 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:13 smithi099 ceph-mon[39420]: pgmap v706: 17 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 7.2 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 675 op/s; 353/352095 objects degraded (0.100%); 35003/352095 objects misplaced (9.941%); 24 MiB/s, 385 objects/s recovering 2024-02-09T17:32:13.724 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:13 smithi099 ceph-mon[39420]: osdmap e243: 12 total, 12 up, 11 in 2024-02-09T17:32:13.789 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:32:13.789 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:32:13.883 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:32:14.031 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:32:14.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:14.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:32:14.135 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:32:14.177 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:13 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:32:14.177 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:13 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:32:14.177 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:13 smithi179 ceph-mon[43975]: pgmap v706: 17 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 7.2 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 675 op/s; 353/352095 objects degraded (0.100%); 35003/352095 objects misplaced (9.941%); 24 MiB/s, 385 objects/s recovering 2024-02-09T17:32:14.177 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:13 smithi179 ceph-mon[43975]: osdmap e243: 12 total, 12 up, 11 in 2024-02-09T17:32:14.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:13 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:32:14.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:13 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:32:14.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:13 smithi162 ceph-mon[44304]: pgmap v706: 17 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 7.2 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 675 op/s; 353/352095 objects degraded (0.100%); 35003/352095 objects misplaced (9.941%); 24 MiB/s, 385 objects/s recovering 2024-02-09T17:32:14.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:13 smithi162 ceph-mon[44304]: osdmap e243: 12 total, 12 up, 11 in 2024-02-09T17:32:14.190 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:14.226 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:32:14.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:14.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:32:14.336 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:32:14.366 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:14.573 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:32:14.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:14.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:32:14.601 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-09T17:32:14.678 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:32:14.718 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:14.805 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:32:14.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:14.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:32:14.910 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:32:14.966 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:15.015 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:32:15.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:15.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:32:15.120 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:32:15.176 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:15.220 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:32:15.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:15.226 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:32:15.327 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:32:15.378 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:15.413 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:32:15.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:15.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:32:15.516 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:32:15.557 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:15.611 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:32:15.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:15.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:32:15.719 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:32:15.729 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:15 smithi099 ceph-mon[39420]: pgmap v708: 17 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 7.1 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 714 op/s; 647/347916 objects degraded (0.186%); 34391/347916 objects misplaced (9.885%); 17 MiB/s, 276 objects/s recovering 2024-02-09T17:32:15.769 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:15.803 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:32:15.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:15.820 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:32:15.921 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:32:15.989 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:15.990 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:15 smithi179 ceph-mon[43975]: pgmap v708: 17 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 7.1 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 714 op/s; 647/347916 objects degraded (0.186%); 34391/347916 objects misplaced (9.885%); 17 MiB/s, 276 objects/s recovering 2024-02-09T17:32:16.014 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:32:16.025 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:16.025 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:32:16.126 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:32:16.157 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:16.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:15 smithi162 ceph-mon[44304]: pgmap v708: 17 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 7.1 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 714 op/s; 647/347916 objects degraded (0.186%); 34391/347916 objects misplaced (9.885%); 17 MiB/s, 276 objects/s recovering 2024-02-09T17:32:16.513 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:32:16.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:16.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:32:16.621 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:32:16.668 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:16.716 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:32:16.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:16.719 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:32:16.773 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:32:16.820 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:32:16.879 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:16.903 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:32:16.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:16.911 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:32:16.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:16 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 647/347916 objects degraded (0.186%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:32:16.943 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:16 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 647/347916 objects degraded (0.186%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:32:17.012 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:32:17.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:16 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 647/347916 objects degraded (0.186%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:32:17.062 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:17.099 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:32:17.101 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-02-09T17:32:17.115 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:32:17.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:17.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:32:17.222 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:32:17.281 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:17.306 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:32:17.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:17.313 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:32:17.414 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:32:17.471 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:17.505 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:32:17.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:17.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:32:17.613 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:32:17.646 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:17.911 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:32:17.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:17.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:32:18.015 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:32:18.062 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:18.063 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:17 smithi179 ceph-mon[43975]: pgmap v709: 17 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 7.0 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 617 op/s; 647/346200 objects degraded (0.187%); 34238/346200 objects misplaced (9.890%); 14 MiB/s, 230 objects/s recovering 2024-02-09T17:32:18.093 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:32:18.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:18.096 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:32:18.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:17 smithi162 ceph-mon[44304]: pgmap v709: 17 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 7.0 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 617 op/s; 647/346200 objects degraded (0.187%); 34238/346200 objects misplaced (9.890%); 14 MiB/s, 230 objects/s recovering 2024-02-09T17:32:18.197 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:32:18.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:17 smithi099 ceph-mon[39420]: pgmap v709: 17 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 7.0 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 617 op/s; 647/346200 objects degraded (0.187%); 34238/346200 objects misplaced (9.890%); 14 MiB/s, 230 objects/s recovering 2024-02-09T17:32:18.257 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:18.294 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:32:18.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:18.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:32:18.404 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:32:18.488 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:18.531 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:32:18.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:18.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:32:18.644 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:32:18.697 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:18.742 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:32:18.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:18.747 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:32:18.848 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:32:18.891 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:18 smithi162 ceph-mon[44304]: from='client.15237 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:32:18.894 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:18.924 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:32:18.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:18.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:32:18.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:18 smithi099 ceph-mon[39420]: from='client.15237 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:32:19.031 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:32:19.070 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:19.070 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:18 smithi179 ceph-mon[43975]: from='client.15237 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:32:19.094 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:32:19.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:19.098 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:32:19.155 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:32:19.198 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:32:19.274 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:19.305 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:32:19.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:19.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:32:19.413 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:32:19.463 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:19.488 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:32:19.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:19.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:32:19.592 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:32:19.608 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":709,"stamp":"2024-02-09T17:32:16.549646+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7563248184,"num_objects":115400,"num_object_clones":0,"num_object_copies":346200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":647,"num_objects_misplaced":34238,"num_objects_unfound":0,"num_objects_dirty":115400,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":148697,"num_write_kb":8448123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3112,"num_bytes_recovered":205740704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149576,"ondisk_log_size":149576,"up":51,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":21631520,"kb_used_data":19991604,"kb_used_omap":228,"kb_used_meta":1639579,"kb_avail":1009565152,"statfs":{"total":1055945392128,"available":1033794715648,"internally_reserved":0,"allocated":20471402496,"data_stored":20468507371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":233532,"internal_metadata":1678929860},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,4,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-318439424,"num_objects":-4859,"num_object_clones":0,"num_object_copies":-14577,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":502,"num_objects_misplaced":-2434,"num_objects_unfound":0,"num_objects_dirty":-4859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4859,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1813,"num_bytes_recovered":118620160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.862857"},"pg_stats":[{"pgid":"3.f","version":"241'9188","reported_seq":9367,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-02-09T17:32:11.701503+0000","last_change":"2024-02-09T17:31:25.731053+0000","last_active":"2024-02-09T17:32:11.701503+0000","last_peered":"2024-02-09T17:32:11.701503+0000","last_clean":"2024-02-09T17:32:11.701503+0000","last_became_active":"2024-02-09T17:29:47.582245+0000","last_became_peered":"2024-02-09T17:29:47.582245+0000","last_unstale":"2024-02-09T17:32:11.701503+0000","last_undegraded":"2024-02-09T17:32:11.701503+0000","last_fullsized":"2024-02-09T17:32:11.701503+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'7772","last_scrub_stamp":"2024-02-09T17:31:25.730848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:31:25.730848+0000","objects_scrubbed":7721,"log_size":9188,"log_dups_size":0,"ondisk_log_size":9188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:32:33.308605+0000","scrub_duration":2441,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":475791360,"num_objects":7260,"num_object_clones":0,"num_object_copies":21780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9188,"num_write_kb":526336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"243'9364","reported_seq":10143,"reported_epoch":243,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-02-09T17:32:14.474666+0000","last_change":"2024-02-09T17:32:01.597065+0000","last_active":"2024-02-09T17:32:14.474666+0000","last_peered":"2024-02-09T17:32:14.474666+0000","last_clean":"2024-02-09T17:31:59.261441+0000","last_became_active":"2024-02-09T17:32:01.541864+0000","last_became_peered":"2024-02-09T17:32:01.541864+0000","last_unstale":"2024-02-09T17:32:14.474666+0000","last_undegraded":"2024-02-09T17:32:01.324655+0000","last_fullsized":"2024-02-09T17:32:01.278077+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'7702","last_scrub_stamp":"2024-02-09T17:31:22.871435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:31:22.871435+0000","objects_scrubbed":7650,"log_size":9364,"log_dups_size":0,"ondisk_log_size":9364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1911,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":475398144,"num_objects":7254,"num_object_clones":0,"num_object_copies":21762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":198,"num_objects_misplaced":7253,"num_objects_unfound":0,"num_objects_dirty":7254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9364,"num_write_kb":531776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":296,"num_bytes_recovered":19267584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9],"acting":[11,4],"avail_no_missing":["11","0","4"],"object_location_counts":[{"shards":"0,4,11","objects":7253},{"shards":"4,11","objects":-197}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"241'9287","reported_seq":9468,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-02-09T17:32:11.686980+0000","last_change":"2024-02-09T17:31:24.785058+0000","last_active":"2024-02-09T17:32:11.686980+0000","last_peered":"2024-02-09T17:32:11.686980+0000","last_clean":"2024-02-09T17:32:11.686980+0000","last_became_active":"2024-02-09T17:29:47.583238+0000","last_became_peered":"2024-02-09T17:29:47.583238+0000","last_unstale":"2024-02-09T17:32:11.686980+0000","last_undegraded":"2024-02-09T17:32:11.686980+0000","last_fullsized":"2024-02-09T17:32:11.686980+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'7776","last_scrub_stamp":"2024-02-09T17:31:24.784873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:31:24.784873+0000","objects_scrubbed":7733,"log_size":9287,"log_dups_size":0,"ondisk_log_size":9287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:32:51.351383+0000","scrub_duration":2469,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484966424,"num_objects":7401,"num_object_clones":0,"num_object_copies":22203,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7401,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9287,"num_write_kb":533953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,9],"acting":[7,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"243'9258","reported_seq":9441,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-09T17:32:14.125515+0000","last_change":"2024-02-09T17:31:24.234560+0000","last_active":"2024-02-09T17:32:14.125515+0000","last_peered":"2024-02-09T17:32:14.125515+0000","last_clean":"2024-02-09T17:32:14.125515+0000","last_became_active":"2024-02-09T17:29:47.583200+0000","last_became_peered":"2024-02-09T17:29:47.583200+0000","last_unstale":"2024-02-09T17:32:14.125515+0000","last_undegraded":"2024-02-09T17:32:14.125515+0000","last_fullsized":"2024-02-09T17:32:14.125515+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'7647","last_scrub_stamp":"2024-02-09T17:31:24.234420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:31:24.234420+0000","objects_scrubbed":7605,"log_size":9258,"log_dups_size":0,"ondisk_log_size":9258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:32:42.521549+0000","scrub_duration":2190,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":468451328,"num_objects":7148,"num_object_clones":0,"num_object_copies":21444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9258,"num_write_kb":524992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"241'12471","reported_seq":16368,"reported_epoch":241,"state":"active+remapped+backfilling","last_fresh":"2024-02-09T17:32:11.596479+0000","last_change":"2024-02-09T17:32:01.600444+0000","last_active":"2024-02-09T17:32:11.596479+0000","last_peered":"2024-02-09T17:32:11.596479+0000","last_clean":"2024-02-09T17:31:59.257931+0000","last_became_active":"2024-02-09T17:32:01.577152+0000","last_became_peered":"2024-02-09T17:32:01.577152+0000","last_unstale":"2024-02-09T17:32:11.596479+0000","last_undegraded":"2024-02-09T17:32:11.596479+0000","last_fullsized":"2024-02-09T17:32:11.596479+0000","mapping_epoch":238,"log_start":"209'2400","ondisk_log_start":"209'2400","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:29:46.563947+0000","objects_scrubbed":2799,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":475136000,"num_objects":7250,"num_object_clones":0,"num_object_copies":21750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5494,"num_objects_unfound":0,"num_objects_dirty":7250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9150,"num_write_kb":524800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1799,"num_bytes_recovered":117899264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,11],"acting":[3,4,0],"avail_no_missing":["3","0","4"],"object_location_counts":[{"shards":"0,3,4","objects":7250}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"243'9355","reported_seq":9539,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-09T17:32:15.509410+0000","last_change":"2024-02-09T17:31:25.346078+0000","last_active":"2024-02-09T17:32:15.509410+0000","last_peered":"2024-02-09T17:32:15.509410+0000","last_clean":"2024-02-09T17:32:15.509410+0000","last_became_active":"2024-02-09T17:29:47.581573+0000","last_became_peered":"2024-02-09T17:29:47.581573+0000","last_unstale":"2024-02-09T17:32:15.509410+0000","last_undegraded":"2024-02-09T17:32:15.509410+0000","last_fullsized":"2024-02-09T17:32:15.509410+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'7710","last_scrub_stamp":"2024-02-09T17:31:25.345946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:31:25.345946+0000","objects_scrubbed":7649,"log_size":9355,"log_dups_size":0,"ondisk_log_size":9355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:32:29.117168+0000","scrub_duration":2400,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":466157568,"num_objects":7113,"num_object_clones":0,"num_object_copies":21339,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9355,"num_write_kb":526976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"242'9354","reported_seq":9558,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-02-09T17:32:11.794662+0000","last_change":"2024-02-09T17:31:20.743404+0000","last_active":"2024-02-09T17:32:11.794662+0000","last_peered":"2024-02-09T17:32:11.794662+0000","last_clean":"2024-02-09T17:32:11.794662+0000","last_became_active":"2024-02-09T17:29:47.582488+0000","last_became_peered":"2024-02-09T17:29:47.582488+0000","last_unstale":"2024-02-09T17:32:11.794662+0000","last_undegraded":"2024-02-09T17:32:11.794662+0000","last_fullsized":"2024-02-09T17:32:11.794662+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'7551","last_scrub_stamp":"2024-02-09T17:31:20.743150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:31:20.743150+0000","objects_scrubbed":7492,"log_size":9354,"log_dups_size":0,"ondisk_log_size":9354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:32:34.068254+0000","scrub_duration":1578,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":476053504,"num_objects":7264,"num_object_clones":0,"num_object_copies":21792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9354,"num_write_kb":531776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.8","version":"243'9288","reported_seq":9471,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-09T17:32:14.366955+0000","last_change":"2024-02-09T17:31:21.687596+0000","last_active":"2024-02-09T17:32:14.366955+0000","last_peered":"2024-02-09T17:32:14.366955+0000","last_clean":"2024-02-09T17:32:14.366955+0000","last_became_active":"2024-02-09T17:29:47.581467+0000","last_became_peered":"2024-02-09T17:29:47.581467+0000","last_unstale":"2024-02-09T17:32:14.366955+0000","last_undegraded":"2024-02-09T17:32:14.366955+0000","last_fullsized":"2024-02-09T17:32:14.366955+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'7591","last_scrub_stamp":"2024-02-09T17:31:21.687400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:31:21.687400+0000","objects_scrubbed":7543,"log_size":9288,"log_dups_size":0,"ondisk_log_size":9288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:32:44.240138+0000","scrub_duration":1706,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":475529216,"num_objects":7256,"num_object_clones":0,"num_object_copies":21768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9288,"num_write_kb":529408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.7","version":"241'9148","reported_seq":9357,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-02-09T17:32:11.569600+0000","last_change":"2024-02-09T17:31:29.003929+0000","last_active":"2024-02-09T17:32:11.569600+0000","last_peered":"2024-02-09T17:32:11.569600+0000","last_clean":"2024-02-09T17:32:11.569600+0000","last_became_active":"2024-02-09T17:29:47.583311+0000","last_became_peered":"2024-02-09T17:29:47.583311+0000","last_unstale":"2024-02-09T17:32:11.569600+0000","last_undegraded":"2024-02-09T17:32:11.569600+0000","last_fullsized":"2024-02-09T17:32:11.569600+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'7925","last_scrub_stamp":"2024-02-09T17:31:29.003795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:31:29.003795+0000","objects_scrubbed":7867,"log_size":9148,"log_dups_size":0,"ondisk_log_size":9148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:32:34.119914+0000","scrub_duration":1984,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":475791360,"num_objects":7260,"num_object_clones":0,"num_object_copies":21780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9148,"num_write_kb":525056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"243'9148","reported_seq":10233,"reported_epoch":243,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-02-09T17:32:15.015981+0000","last_change":"2024-02-09T17:32:01.583215+0000","last_active":"2024-02-09T17:32:15.015981+0000","last_peered":"2024-02-09T17:32:15.015981+0000","last_clean":"2024-02-09T17:31:59.217197+0000","last_became_active":"2024-02-09T17:32:01.532489+0000","last_became_peered":"2024-02-09T17:32:01.532489+0000","last_unstale":"2024-02-09T17:32:15.015981+0000","last_undegraded":"2024-02-09T17:32:01.315621+0000","last_fullsized":"2024-02-09T17:32:01.278340+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'7634","last_scrub_stamp":"2024-02-09T17:31:25.792160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:31:25.792160+0000","objects_scrubbed":7589,"log_size":9148,"log_dups_size":0,"ondisk_log_size":9148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1716,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":465698816,"num_objects":7106,"num_object_clones":0,"num_object_copies":21318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17,"num_objects_misplaced":7095,"num_objects_unfound":0,"num_objects_dirty":7106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9148,"num_write_kb":520128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":450,"num_bytes_recovered":29491200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4],"avail_no_missing":["1","0","4"],"object_location_counts":[{"shards":"0,1,4","objects":7095},{"shards":"1,4","objects":-6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"243'9259","reported_seq":9441,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-09T17:32:14.125532+0000","last_change":"2024-02-09T17:31:23.024422+0000","last_active":"2024-02-09T17:32:14.125532+0000","last_peered":"2024-02-09T17:32:14.125532+0000","last_clean":"2024-02-09T17:32:14.125532+0000","last_became_active":"2024-02-09T17:29:47.581119+0000","last_became_peered":"2024-02-09T17:29:47.581119+0000","last_unstale":"2024-02-09T17:32:14.125532+0000","last_undegraded":"2024-02-09T17:32:14.125532+0000","last_fullsized":"2024-02-09T17:32:14.125532+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'7587","last_scrub_stamp":"2024-02-09T17:31:23.024268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:31:23.024268+0000","objects_scrubbed":7544,"log_size":9259,"log_dups_size":0,"ondisk_log_size":9259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:32:34.908941+0000","scrub_duration":1988,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":466157568,"num_objects":7113,"num_object_clones":0,"num_object_copies":21339,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9259,"num_write_kb":523904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"243'9340","reported_seq":9584,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-09T17:32:14.997853+0000","last_change":"2024-02-09T17:31:59.706801+0000","last_active":"2024-02-09T17:32:14.997853+0000","last_peered":"2024-02-09T17:32:14.997853+0000","last_clean":"2024-02-09T17:32:14.997853+0000","last_became_active":"2024-02-09T17:29:47.582922+0000","last_became_peered":"2024-02-09T17:29:47.582922+0000","last_unstale":"2024-02-09T17:32:14.997853+0000","last_undegraded":"2024-02-09T17:32:14.997853+0000","last_fullsized":"2024-02-09T17:32:14.997853+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'8794","last_scrub_stamp":"2024-02-09T17:31:59.706688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:31:59.706688+0000","objects_scrubbed":7672,"log_size":9340,"log_dups_size":0,"ondisk_log_size":9340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:33:10.369313+0000","scrub_duration":1299,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":466354176,"num_objects":7116,"num_object_clones":0,"num_object_copies":21348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9340,"num_write_kb":526592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"243'9470","reported_seq":10166,"reported_epoch":243,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-02-09T17:32:14.471340+0000","last_change":"2024-02-09T17:32:01.643212+0000","last_active":"2024-02-09T17:32:14.471340+0000","last_peered":"2024-02-09T17:32:14.471340+0000","last_clean":"2024-02-09T17:31:59.234884+0000","last_became_active":"2024-02-09T17:32:01.587380+0000","last_became_peered":"2024-02-09T17:32:01.587380+0000","last_unstale":"2024-02-09T17:32:14.471340+0000","last_undegraded":"2024-02-09T17:32:01.320756+0000","last_fullsized":"2024-02-09T17:32:01.276816+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'7835","last_scrub_stamp":"2024-02-09T17:31:26.001797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:31:26.001797+0000","objects_scrubbed":7767,"log_size":9470,"log_dups_size":0,"ondisk_log_size":9470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2765,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":475922432,"num_objects":7262,"num_object_clones":0,"num_object_copies":21786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":240,"num_objects_misplaced":7262,"num_objects_unfound":0,"num_objects_dirty":7262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9470,"num_write_kb":535424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":254,"num_bytes_recovered":16646144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,11],"acting":[4,6],"avail_no_missing":["4","0","6"],"object_location_counts":[{"shards":"0,4,6","objects":7262},{"shards":"4,6","objects":-240}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"243'9294","reported_seq":10070,"reported_epoch":243,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-02-09T17:32:14.430059+0000","last_change":"2024-02-09T17:32:01.551242+0000","last_active":"2024-02-09T17:32:14.430059+0000","last_peered":"2024-02-09T17:32:14.430059+0000","last_clean":"2024-02-09T17:31:59.232029+0000","last_became_active":"2024-02-09T17:32:01.494982+0000","last_became_peered":"2024-02-09T17:32:01.494982+0000","last_unstale":"2024-02-09T17:32:14.430059+0000","last_undegraded":"2024-02-09T17:32:01.319667+0000","last_fullsized":"2024-02-09T17:32:01.276864+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"220'6050","last_scrub_stamp":"2024-02-09T17:31:00.396342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:31:00.396342+0000","objects_scrubbed":6003,"log_size":9294,"log_dups_size":0,"ondisk_log_size":9294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1427,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":467533824,"num_objects":7134,"num_object_clones":0,"num_object_copies":21402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":192,"num_objects_misplaced":7134,"num_objects_unfound":0,"num_objects_dirty":7134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9294,"num_write_kb":525696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":303,"num_bytes_recovered":19791872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,11],"acting":[4,10],"avail_no_missing":["4","0","10"],"object_location_counts":[{"shards":"0,4,10","objects":7134},{"shards":"4,10","objects":-192}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"243'9349","reported_seq":9514,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-09T17:32:14.422870+0000","last_change":"2024-02-09T17:30:55.910896+0000","last_active":"2024-02-09T17:32:14.422870+0000","last_peered":"2024-02-09T17:32:14.422870+0000","last_clean":"2024-02-09T17:32:14.422870+0000","last_became_active":"2024-02-09T17:29:47.582887+0000","last_became_peered":"2024-02-09T17:29:47.582887+0000","last_unstale":"2024-02-09T17:32:14.422870+0000","last_undegraded":"2024-02-09T17:32:14.422870+0000","last_fullsized":"2024-02-09T17:32:14.422870+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"218'5796","last_scrub_stamp":"2024-02-09T17:30:55.910807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:30:55.910807+0000","objects_scrubbed":5754,"log_size":9349,"log_dups_size":0,"ondisk_log_size":9349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:32:23.481568+0000","scrub_duration":1081,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":469434368,"num_objects":7163,"num_object_clones":0,"num_object_copies":21489,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9349,"num_write_kb":528384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,6],"acting":[11,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"241'9264","reported_seq":9447,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-02-09T17:32:11.565944+0000","last_change":"2024-02-09T17:31:29.902726+0000","last_active":"2024-02-09T17:32:11.565944+0000","last_peered":"2024-02-09T17:32:11.565944+0000","last_clean":"2024-02-09T17:32:11.565944+0000","last_became_active":"2024-02-09T17:29:47.581341+0000","last_became_peered":"2024-02-09T17:29:47.581341+0000","last_unstale":"2024-02-09T17:32:11.565944+0000","last_undegraded":"2024-02-09T17:32:11.565944+0000","last_fullsized":"2024-02-09T17:32:11.565944+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'8039","last_scrub_stamp":"2024-02-09T17:31:29.902558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:31:29.902558+0000","objects_scrubbed":7987,"log_size":9264,"log_dups_size":0,"ondisk_log_size":9264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:32:45.488168+0000","scrub_duration":1893,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":478281728,"num_objects":7298,"num_object_clones":0,"num_object_copies":21894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9264,"num_write_kb":529984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"63'139","reported_seq":703,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-02-09T17:32:08.692722+0000","last_change":"2024-02-09T17:31:59.409457+0000","last_active":"2024-02-09T17:32:08.692722+0000","last_peered":"2024-02-09T17:32:08.692722+0000","last_clean":"2024-02-09T17:32:08.692722+0000","last_became_active":"2024-02-09T17:22:46.986816+0000","last_became_peered":"2024-02-09T17:22:46.986816+0000","last_unstale":"2024-02-09T17:32:08.692722+0000","last_undegraded":"2024-02-09T17:32:08.692722+0000","last_fullsized":"2024-02-09T17:32:08.692722+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:31:59.409353+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:31:59.409353+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:33:01.421976+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":7562657816,"num_objects":115398,"num_object_clones":0,"num_object_copies":346194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":647,"num_objects_misplaced":34238,"num_objects_unfound":0,"num_objects_dirty":115398,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148516,"num_write_kb":8445185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3102,"num_bytes_recovered":203096064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":22893047808,"data_stored":22893035592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149437,"ondisk_log_size":149437,"up":48,"acting":44,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056758,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3545916,"kb_used_data":3362036,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":90199236,"statfs":{"total":95995035648,"available":92364017664,"internally_reserved":0,"allocated":3442724864,"data_stored":3442457416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":188283332},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:32:00 2024","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":1.101,"15min":0.85599999999999998},"min":{"1min":0.26500000000000001,"5min":0.219,"15min":0.219},"max":{"1min":4.3079999999999998,"5min":14.913,"15min":14.913},"last":0.28000000000000003},{"interface":"front","average":{"1min":1.069,"5min":1.272,"15min":0.94299999999999995},"min":{"1min":0.28599999999999998,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":3.8039999999999998,"5min":14.869,"15min":14.869},"last":4.7249999999999996}]},{"osd":1,"last update":"Fri Feb 9 17:32:00 2024","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":1.97,"15min":1.244},"min":{"1min":0.30199999999999999,"5min":0.245,"15min":0.245},"max":{"1min":3.7429999999999999,"5min":32.957999999999998,"15min":32.957999999999998},"last":4.3419999999999996},{"interface":"front","average":{"1min":0.99299999999999999,"5min":2.1619999999999999,"15min":1.282},"min":{"1min":0.39800000000000002,"5min":0.26800000000000002,"15min":0.26700000000000002},"max":{"1min":3.823,"5min":32.880000000000003,"15min":32.880000000000003},"last":4.367}]},{"osd":2,"last update":"Fri Feb 9 17:31:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.415,"15min":1.6859999999999999},"min":{"1min":0.378,"5min":0.247,"15min":0.247},"max":{"1min":4.4189999999999996,"5min":9.4719999999999995,"15min":9.4719999999999995},"last":4.7949999999999999},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.5149999999999999,"15min":1.4399999999999999},"min":{"1min":0.378,"5min":0.252,"15min":0.252},"max":{"1min":4.8579999999999997,"5min":7.96,"15min":7.96},"last":0.33400000000000002}]},{"osd":3,"last update":"Fri Feb 9 17:32:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":2.3220000000000001,"15min":1.3480000000000001},"min":{"1min":0.34399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":4.0990000000000002,"5min":21.315999999999999,"15min":21.315999999999999},"last":4.3150000000000004},{"interface":"front","average":{"1min":0.85899999999999999,"5min":1.867,"15min":1.218},"min":{"1min":0.29599999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":3.722,"5min":18.087,"15min":18.087},"last":4.4029999999999996}]},{"osd":4,"last update":"Fri Feb 9 17:32:00 2024","interfaces":[{"interface":"back","average":{"1min":1.7430000000000001,"5min":0.97199999999999998,"15min":0.83199999999999996},"min":{"1min":0.32700000000000001,"5min":0.30499999999999999,"15min":0.30299999999999999},"max":{"1min":6.8040000000000003,"5min":6.8040000000000003,"15min":6.8040000000000003},"last":4.7640000000000002},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.0700000000000001,"15min":0.88900000000000001},"min":{"1min":0.39800000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":3.673,"5min":6.5140000000000002,"15min":6.5140000000000002},"last":4.2759999999999998}]},{"osd":5,"last update":"Fri Feb 9 17:32:00 2024","interfaces":[{"interface":"back","average":{"1min":1.72,"5min":1.1659999999999999,"15min":0.92200000000000004},"min":{"1min":0.438,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":4.6959999999999997,"5min":13.891,"15min":13.891},"last":13.946999999999999},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.014,"15min":0.87},"min":{"1min":0.36199999999999999,"5min":0.313,"15min":0.313},"max":{"1min":3.6400000000000001,"5min":6.5970000000000004,"15min":6.5970000000000004},"last":4.9269999999999996}]},{"osd":6,"last update":"Fri Feb 9 17:32:00 2024","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.018,"15min":0.88900000000000001},"min":{"1min":0.40400000000000003,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":5.0460000000000003,"5min":7.726,"15min":7.726},"last":0.39900000000000002},{"interface":"front","average":{"1min":1.454,"5min":1.1879999999999999,"15min":0.92000000000000004},"min":{"1min":0.39000000000000001,"5min":0.379,"15min":0.32300000000000001},"max":{"1min":5.7160000000000002,"5min":6.6219999999999999,"15min":6.6219999999999999},"last":4.4249999999999998}]},{"osd":7,"last update":"Fri Feb 9 17:32:00 2024","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.212,"15min":0.95899999999999996},"min":{"1min":0.442,"5min":0.317,"15min":0.317},"max":{"1min":4.3959999999999999,"5min":7.7720000000000002,"15min":7.7720000000000002},"last":14.462},{"interface":"front","average":{"1min":1.526,"5min":1.149,"15min":0.93300000000000005},"min":{"1min":0.53100000000000003,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":4.4969999999999999,"5min":5.0369999999999999,"15min":5.0369999999999999},"last":14.398999999999999}]},{"osd":8,"last update":"Fri Feb 9 17:25:48 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.71399999999999997,"15min":0.68000000000000005},"min":{"1min":0.40799999999999997,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.9350000000000001,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":2.8180000000000001},{"interface":"front","average":{"1min":1.3029999999999999,"5min":0.80700000000000005,"15min":0.71999999999999997},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":3.7389999999999999,"5min":3.7389999999999999,"15min":3.7389999999999999},"last":6.0659999999999998}]},{"osd":9,"last update":"Fri Feb 9 17:31:45 2024","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":1.0680000000000001,"15min":1.2170000000000001},"min":{"1min":0.46300000000000002,"5min":0.316,"15min":0.316},"max":{"1min":4.556,"5min":6.7279999999999998,"15min":6.7279999999999998},"last":5.8680000000000003},{"interface":"front","average":{"1min":0.752,"5min":0.90000000000000002,"15min":0.88800000000000001},"min":{"1min":0.36099999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":4.109,"5min":6.0860000000000003,"15min":6.0860000000000003},"last":5.8540000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:32:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.095,"15min":0.86899999999999999},"min":{"1min":0.434,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":4.218,"5min":6.7759999999999998,"15min":6.7759999999999998},"last":5.9470000000000001},{"interface":"front","average":{"1min":1.101,"5min":0.90900000000000003,"15min":0.82699999999999996},"min":{"1min":0.41699999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":4.4539999999999997,"5min":10.675000000000001,"15min":10.675000000000001},"last":5.8239999999999998}]}]},{"osd":10,"up_from":84,"seq":360777252987,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":561448,"kb_used_data":462240,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93183704,"statfs":{"total":95995035648,"available":95420112896,"internally_reserved":0,"allocated":473333760,"data_stored":473074213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":101579200},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:31:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":1.387,"15min":1.762},"min":{"1min":0.23899999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":1.29,"5min":20.120999999999999,"15min":20.120999999999999},"last":1.4870000000000001},{"interface":"front","average":{"1min":0.74099999999999999,"5min":1.4039999999999999,"15min":2.3940000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":3.173,"5min":14.471,"15min":14.471},"last":1.5860000000000001}]},{"osd":1,"last update":"Fri Feb 9 17:31:39 2024","interfaces":[{"interface":"back","average":{"1min":3.4129999999999998,"5min":2.0569999999999999,"15min":1.143},"min":{"1min":0.24199999999999999,"5min":0.23899999999999999,"15min":0.23499999999999999},"max":{"1min":37.716000000000001,"5min":37.716000000000001,"15min":37.716000000000001},"last":0.33200000000000002},{"interface":"front","average":{"1min":2.9910000000000001,"5min":1.819,"15min":1.1240000000000001},"min":{"1min":0.30499999999999999,"5min":0.245,"15min":0.245},"max":{"1min":38.039999999999999,"5min":38.039999999999999,"15min":38.039999999999999},"last":1.526}]},{"osd":2,"last update":"Fri Feb 9 17:31:39 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":1.512,"15min":1.0580000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":3.149,"5min":13.351000000000001,"15min":13.351000000000001},"last":0.43099999999999999},{"interface":"front","average":{"1min":1.137,"5min":1.6459999999999999,"15min":1.042},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":4.0629999999999997,"5min":10.596,"15min":10.596},"last":1.6970000000000001}]},{"osd":3,"last update":"Fri Feb 9 17:31:39 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":1.452,"15min":1.0920000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.26200000000000001},"max":{"1min":2.3359999999999999,"5min":6.9809999999999999,"15min":9.2650000000000006},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.95199999999999996,"5min":1.53,"15min":1.034},"min":{"1min":0.371,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":3.9590000000000001,"5min":8.1080000000000005,"15min":8.1080000000000005},"last":1.615}]},{"osd":4,"last update":"Fri Feb 9 17:31:39 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":0.90700000000000003,"15min":0.82199999999999995},"min":{"1min":0.32600000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":4.4850000000000003,"5min":4.4850000000000003,"15min":6.2249999999999996},"last":3.2890000000000001},{"interface":"front","average":{"1min":1.167,"5min":0.92500000000000004,"15min":0.84399999999999997},"min":{"1min":0.34200000000000003,"5min":0.307,"15min":0.307},"max":{"1min":3.7719999999999998,"5min":6.8369999999999997,"15min":6.8369999999999997},"last":2.774}]},{"osd":5,"last update":"Fri Feb 9 17:31:39 2024","interfaces":[{"interface":"back","average":{"1min":2.2469999999999999,"5min":1.3440000000000001,"15min":1.014},"min":{"1min":0.35799999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":6.0270000000000001,"5min":6.9160000000000004,"15min":7.3339999999999996},"last":0.45800000000000002},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.371,"15min":0.95799999999999996},"min":{"1min":0.39400000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":4.2130000000000001,"5min":8.6159999999999997,"15min":8.6159999999999997},"last":1.669}]},{"osd":6,"last update":"Fri Feb 9 17:31:39 2024","interfaces":[{"interface":"back","average":{"1min":1.7649999999999999,"5min":1.1850000000000001,"15min":0.89000000000000001},"min":{"1min":0.39100000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":6.8799999999999999,"5min":6.8799999999999999,"15min":6.8799999999999999},"last":1.792},{"interface":"front","average":{"1min":1.927,"5min":1.238,"15min":0.95699999999999996},"min":{"1min":0.43099999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":4.9820000000000002,"5min":6.8609999999999998,"15min":6.8609999999999998},"last":1.5600000000000001}]},{"osd":7,"last update":"Fri Feb 9 17:31:39 2024","interfaces":[{"interface":"back","average":{"1min":1.7130000000000001,"5min":1.089,"15min":0.877},"min":{"1min":0.45600000000000002,"5min":0.312,"15min":0.312},"max":{"1min":4.335,"5min":4.3659999999999997,"15min":4.3659999999999997},"last":1.073},{"interface":"front","average":{"1min":1.165,"5min":1.1359999999999999,"15min":0.94199999999999995},"min":{"1min":0.42399999999999999,"5min":0.316,"15min":0.316},"max":{"1min":3.2810000000000001,"5min":6.9589999999999996,"15min":6.9589999999999996},"last":1.724}]},{"osd":8,"last update":"Fri Feb 9 17:26:28 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.73799999999999999,"15min":0.64000000000000001},"min":{"1min":0.36399999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":3.6509999999999998,"5min":3.6509999999999998,"15min":3.6509999999999998},"last":0.91800000000000004},{"interface":"front","average":{"1min":1.72,"5min":0.89500000000000002,"15min":0.68200000000000005},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":4.173,"5min":4.173,"15min":4.173},"last":0.51100000000000001}]},{"osd":9,"last update":"Fri Feb 9 17:31:39 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":1.036,"15min":0.83199999999999996},"min":{"1min":0.34399999999999997,"5min":0.314,"15min":0.314},"max":{"1min":3.9449999999999998,"5min":8.7240000000000002,"15min":8.7240000000000002},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.753,"5min":1.3,"15min":0.96799999999999997},"min":{"1min":0.42099999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":4.0039999999999996,"5min":10.1,"15min":10.1},"last":0.88200000000000001}]},{"osd":11,"last update":"Fri Feb 9 17:32:09 2024","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":1.1220000000000001,"15min":0.98899999999999999},"min":{"1min":0.45700000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":4.0330000000000004,"5min":8.6430000000000007,"15min":8.6430000000000007},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.94599999999999995,"5min":1.151,"15min":0.99199999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.33400000000000002},"max":{"1min":3.7170000000000001,"5min":9.298,"15min":9.298},"last":1.6419999999999999}]}]},{"osd":9,"up_from":77,"seq":330712481922,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1558204,"kb_used_data":1378480,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":92186948,"statfs":{"total":95995035648,"available":94399434752,"internally_reserved":0,"allocated":1411563520,"data_stored":1411295552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":184023488},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:28:03 2024","interfaces":[{"interface":"back","average":{"1min":1.9510000000000001,"5min":1.454,"15min":0.85799999999999998},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":8.4499999999999993,"5min":17.495999999999999,"15min":17.495999999999999},"last":2.7360000000000002},{"interface":"front","average":{"1min":1.905,"5min":1.399,"15min":0.83499999999999996},"min":{"1min":0.32200000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":7.5990000000000002,"5min":15.401999999999999,"15min":15.401999999999999},"last":1.5449999999999999}]},{"osd":1,"last update":"Fri Feb 9 17:31:14 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":1.851,"15min":1.1020000000000001},"min":{"1min":0.24199999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":2.794,"5min":16.527999999999999,"15min":16.527999999999999},"last":0.5},{"interface":"front","average":{"1min":0.51500000000000001,"5min":1.6499999999999999,"15min":0.98599999999999999},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":1.863,"5min":17.771999999999998,"15min":17.771999999999998},"last":1.5249999999999999}]},{"osd":2,"last update":"Fri Feb 9 17:31:14 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":1.514,"15min":0.97599999999999998},"min":{"1min":0.30599999999999999,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":2.5259999999999998,"5min":13.878,"15min":13.878},"last":4.6319999999999997},{"interface":"front","average":{"1min":0.53200000000000003,"5min":1.8,"15min":1.0569999999999999},"min":{"1min":0.28599999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.798,"5min":13.69,"15min":13.69},"last":0.46200000000000002}]},{"osd":3,"last update":"Fri Feb 9 17:31:14 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":1.357,"15min":0.92200000000000004},"min":{"1min":0.27400000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.7749999999999999,"5min":9.2889999999999997,"15min":9.2889999999999997},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":1.8360000000000001,"15min":1.095},"min":{"1min":0.32900000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.964,"5min":11.236000000000001,"15min":11.236000000000001},"last":1.4830000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:31:14 2024","interfaces":[{"interface":"back","average":{"1min":2.2650000000000001,"5min":1.2709999999999999,"15min":0.92200000000000004},"min":{"1min":0.33000000000000002,"5min":0.309,"15min":0.309},"max":{"1min":22.577999999999999,"5min":22.577999999999999,"15min":22.577999999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":1.9650000000000001,"5min":1.0720000000000001,"15min":0.84699999999999998},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":22.399999999999999,"5min":22.399999999999999,"15min":22.399999999999999},"last":1.508}]},{"osd":5,"last update":"Fri Feb 9 17:31:14 2024","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":1.01,"15min":0.84599999999999997},"min":{"1min":0.32400000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":2.899,"5min":8.2460000000000004,"15min":8.2460000000000004},"last":4.6989999999999998},{"interface":"front","average":{"1min":1.244,"5min":0.95099999999999996,"15min":0.80800000000000005},"min":{"1min":0.34899999999999998,"5min":0.315,"15min":0.315},"max":{"1min":3.0379999999999998,"5min":4.1829999999999998,"15min":5.665},"last":1.454}]},{"osd":6,"last update":"Fri Feb 9 17:31:14 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":1.0960000000000001,"15min":0.90200000000000002},"min":{"1min":0.35999999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":2.1549999999999998,"5min":8.2959999999999994,"15min":8.2959999999999994},"last":4.7830000000000004},{"interface":"front","average":{"1min":0.878,"5min":1.1479999999999999,"15min":0.89200000000000002},"min":{"1min":0.38200000000000001,"5min":0.36199999999999999,"15min":0.35099999999999998},"max":{"1min":2.1859999999999999,"5min":8.3130000000000006,"15min":8.3130000000000006},"last":4.8040000000000003}]},{"osd":7,"last update":"Fri Feb 9 17:31:14 2024","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.083,"15min":0.878},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":7.0780000000000003,"5min":8.0939999999999994,"15min":8.0939999999999994},"last":0.60699999999999998},{"interface":"front","average":{"1min":1.534,"5min":1.1870000000000001,"15min":0.91900000000000004},"min":{"1min":0.41399999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":5.1829999999999998,"5min":5.1829999999999998,"15min":6.056},"last":0.63}]},{"osd":8,"last update":"Fri Feb 9 17:31:25 2024","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":1.3080000000000001,"15min":1.351},"min":{"1min":0.42999999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":6.1079999999999997,"5min":16.209,"15min":16.209},"last":4.7240000000000002},{"interface":"front","average":{"1min":0.94999999999999996,"5min":1.3300000000000001,"15min":1.391},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":2.7759999999999998,"5min":16.263000000000002,"15min":16.263000000000002},"last":1.5600000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:31:42 2024","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.052,"15min":0.90300000000000002},"min":{"1min":0.36799999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":9.5090000000000003,"5min":9.5090000000000003,"15min":9.5090000000000003},"last":4.7450000000000001},{"interface":"front","average":{"1min":0.70799999999999996,"5min":1.095,"15min":0.91500000000000004},"min":{"1min":0.46300000000000002,"5min":0.38200000000000001,"15min":0.36099999999999999},"max":{"1min":1.972,"5min":8.0670000000000002,"15min":8.0670000000000002},"last":0.56999999999999995}]},{"osd":11,"last update":"Fri Feb 9 17:31:48 2024","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.278,"15min":1.2609999999999999},"min":{"1min":0.49099999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":6.181,"5min":8.2759999999999998,"15min":8.2759999999999998},"last":4.7649999999999997},{"interface":"front","average":{"1min":0.92800000000000005,"5min":1.0149999999999999,"15min":1.0189999999999999},"min":{"1min":0.41699999999999998,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":2.9780000000000002,"5min":6.2060000000000004,"15min":6.2060000000000004},"last":1.542}]}]},{"osd":8,"up_from":171,"seq":734439407664,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1036316,"kb_used_data":932628,"kb_used_omap":212,"kb_used_meta":103467,"kb_avail":92708836,"statfs":{"total":95995035648,"available":94933848064,"internally_reserved":0,"allocated":955011072,"data_stored":954755894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217579,"internal_metadata":105950741},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:31:18 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":1.3740000000000001,"15min":1.665},"min":{"1min":0.26800000000000002,"5min":0.245,"15min":0.245},"max":{"1min":5.0049999999999999,"5min":6.8840000000000003,"15min":6.8840000000000003},"last":0.45400000000000001},{"interface":"front","average":{"1min":1.4390000000000001,"5min":1.399,"15min":1.5589999999999999},"min":{"1min":0.318,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":7.7969999999999997,"5min":7.7969999999999997,"15min":7.7969999999999997},"last":0.30099999999999999}]},{"osd":1,"last update":"Fri Feb 9 17:31:18 2024","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":1.1559999999999999,"15min":1.3169999999999999},"min":{"1min":0.29599999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":5.1589999999999998,"5min":7.8250000000000002,"15min":7.8250000000000002},"last":0.41399999999999998},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.8400000000000001,"15min":2.278},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":3.093,"5min":11.101000000000001,"15min":11.101000000000001},"last":0.56100000000000005}]},{"osd":2,"last update":"Fri Feb 9 17:31:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.2969999999999999,"15min":1.454},"min":{"1min":0.25600000000000001,"5min":0.255,"15min":0.255},"max":{"1min":4.9189999999999996,"5min":5.4450000000000003,"15min":5.4450000000000003},"last":0.378},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.5589999999999999,"15min":1.829},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":4.9050000000000002,"5min":7.1109999999999998,"15min":7.1109999999999998},"last":0.57599999999999996}]},{"osd":3,"last update":"Fri Feb 9 17:31:18 2024","interfaces":[{"interface":"back","average":{"1min":2.113,"5min":2.077,"15min":2.3799999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":10.209,"5min":12.505000000000001,"15min":12.505000000000001},"last":0.39400000000000002},{"interface":"front","average":{"1min":1.72,"5min":2.3370000000000002,"15min":2.8149999999999999},"min":{"1min":0.434,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":5.5140000000000002,"5min":10.92,"15min":10.92},"last":0.51800000000000002}]},{"osd":4,"last update":"Fri Feb 9 17:31:18 2024","interfaces":[{"interface":"back","average":{"1min":2.9729999999999999,"5min":1.1639999999999999,"15min":0.85599999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":37.343000000000004,"5min":37.343000000000004,"15min":37.343000000000004},"last":0.5},{"interface":"front","average":{"1min":2.9830000000000001,"5min":1.419,"15min":1.1870000000000001},"min":{"1min":0.374,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":37.875,"5min":37.875,"15min":37.875},"last":2.1840000000000002}]},{"osd":6,"last update":"Fri Feb 9 17:31:18 2024","interfaces":[{"interface":"back","average":{"1min":3.4660000000000002,"5min":1.6579999999999999,"15min":1.4610000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":37.356999999999999,"5min":37.356999999999999,"15min":37.356999999999999},"last":0.47899999999999998},{"interface":"front","average":{"1min":2.9860000000000002,"5min":1.2250000000000001,"15min":0.94299999999999995},"min":{"1min":0.36499999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":37.25,"5min":37.25,"15min":37.25},"last":0.53700000000000003}]},{"osd":7,"last update":"Fri Feb 9 17:31:18 2024","interfaces":[{"interface":"back","average":{"1min":3.258,"5min":1.4430000000000001,"15min":1.2},"min":{"1min":0.437,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":37.585000000000001,"5min":37.585000000000001,"15min":37.585000000000001},"last":2.117},{"interface":"front","average":{"1min":3.1560000000000001,"5min":1.605,"15min":1.4630000000000001},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":37.918999999999997,"5min":37.918999999999997,"15min":37.918999999999997},"last":2.1509999999999998}]},{"osd":9,"last update":"Fri Feb 9 17:31:18 2024","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":1.075,"15min":1.202},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":3.7970000000000002,"5min":6.9269999999999996,"15min":6.9269999999999996},"last":0.63},{"interface":"front","average":{"1min":0.97999999999999998,"5min":1.115,"15min":1.2450000000000001},"min":{"1min":0.33900000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":2.8620000000000001,"5min":7.7409999999999997,"15min":7.7409999999999997},"last":0.59799999999999998}]},{"osd":10,"last update":"Fri Feb 9 17:31:21 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":1.1100000000000001,"15min":1.2609999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":3.0920000000000001,"5min":5.649,"15min":5.649},"last":1.7090000000000001},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.96599999999999997,"15min":1.0620000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":2.8860000000000001,"5min":5.1070000000000002,"15min":5.1070000000000002},"last":0.60599999999999998}]},{"osd":11,"last update":"Fri Feb 9 17:31:18 2024","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.1890000000000001,"15min":1.3149999999999999},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":4.0860000000000003,"5min":7.6870000000000003,"15min":7.6870000000000003},"last":0.434},{"interface":"front","average":{"1min":0.74399999999999999,"5min":1.325,"15min":1.581},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.7829999999999999,"5min":7.6609999999999996,"15min":7.6609999999999996},"last":0.64900000000000002}]}]},{"osd":7,"up_from":57,"seq":244813136010,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2052712,"kb_used_data":1872992,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":91692440,"statfs":{"total":95995035648,"available":93893058560,"internally_reserved":0,"allocated":1917943808,"data_stored":1917681006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":184023498},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:31:49 2024","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.2410000000000001,"15min":0.93300000000000005},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.6710000000000003,"5min":10.776,"15min":10.776},"last":0.27300000000000002},{"interface":"front","average":{"1min":1.335,"5min":1.5940000000000001,"15min":1.0489999999999999},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":5.734,"5min":18.896000000000001,"15min":18.896000000000001},"last":5.7759999999999998}]},{"osd":1,"last update":"Fri Feb 9 17:27:46 2024","interfaces":[{"interface":"back","average":{"1min":2.262,"5min":2.262,"15min":2.262},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":4.8470000000000004,"5min":4.8470000000000004,"15min":4.8470000000000004},"last":3.9470000000000001},{"interface":"front","average":{"1min":1.8009999999999999,"5min":1.8009999999999999,"15min":1.8009999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.9589999999999996,"5min":6.9589999999999996,"15min":6.9589999999999996},"last":3.6669999999999998}]},{"osd":2,"last update":"Fri Feb 9 17:31:22 2024","interfaces":[{"interface":"back","average":{"1min":2.6869999999999998,"5min":1.881,"15min":1.0609999999999999},"min":{"1min":0.315,"5min":0.30399999999999999,"15min":0.26000000000000001},"max":{"1min":9.2420000000000009,"5min":11.093999999999999,"15min":11.093999999999999},"last":0.35199999999999998},{"interface":"front","average":{"1min":2.0720000000000001,"5min":1.6579999999999999,"15min":1.083},"min":{"1min":0.34000000000000002,"5min":0.32100000000000001,"15min":0.28299999999999997},"max":{"1min":9.2040000000000006,"5min":9.2040000000000006,"15min":11.013},"last":0.38900000000000001}]},{"osd":3,"last update":"Fri Feb 9 17:31:22 2024","interfaces":[{"interface":"back","average":{"1min":1.601,"5min":1.871,"15min":1.0760000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":5.8780000000000001,"5min":27.719000000000001,"15min":27.719000000000001},"last":0.31},{"interface":"front","average":{"1min":1.6859999999999999,"5min":1.7390000000000001,"15min":1.0569999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.27600000000000002},"max":{"1min":5.6980000000000004,"5min":8.8629999999999995,"15min":8.8629999999999995},"last":5.8689999999999998}]},{"osd":4,"last update":"Fri Feb 9 17:31:22 2024","interfaces":[{"interface":"back","average":{"1min":2.1059999999999999,"5min":0.97799999999999998,"15min":0.69199999999999995},"min":{"1min":0.27200000000000002,"5min":0.25900000000000001,"15min":0.248},"max":{"1min":7.4740000000000002,"5min":7.4740000000000002,"15min":7.4740000000000002},"last":0.73099999999999998},{"interface":"front","average":{"1min":1.77,"5min":0.85199999999999998,"15min":0.66600000000000004},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.22800000000000001},"max":{"1min":9.3650000000000002,"5min":9.3650000000000002,"15min":9.3650000000000002},"last":0.75700000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:31:22 2024","interfaces":[{"interface":"back","average":{"1min":2.46,"5min":0.98299999999999998,"15min":0.70799999999999996},"min":{"1min":0.28599999999999998,"5min":0.28199999999999997,"15min":0.21299999999999999},"max":{"1min":10.923,"5min":10.923,"15min":10.923},"last":1.1970000000000001},{"interface":"front","average":{"1min":2.0219999999999998,"5min":0.88100000000000001,"15min":0.72099999999999997},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":7.0789999999999997,"5min":7.0789999999999997,"15min":7.0789999999999997},"last":6.016}]},{"osd":6,"last update":"Fri Feb 9 17:31:22 2024","interfaces":[{"interface":"back","average":{"1min":2.4119999999999999,"5min":1.1220000000000001,"15min":0.78900000000000003},"min":{"1min":0.35999999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":12.496,"5min":12.496,"15min":12.496},"last":0.97399999999999998},{"interface":"front","average":{"1min":1.8580000000000001,"5min":0.93200000000000005,"15min":0.73399999999999999},"min":{"1min":0.33300000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.54,"5min":6.54,"15min":6.54},"last":5.9909999999999997}]},{"osd":8,"last update":"Fri Feb 9 17:31:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.337,"15min":1.417},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":5.4589999999999996,"5min":7.9420000000000002,"15min":7.9420000000000002},"last":0.63400000000000001},{"interface":"front","average":{"1min":2.2109999999999999,"5min":1.1739999999999999,"15min":1.024},"min":{"1min":0.48799999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":9.2829999999999995,"5min":9.2829999999999995,"15min":9.2829999999999995},"last":0.58699999999999997}]},{"osd":9,"last update":"Fri Feb 9 17:32:08 2024","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.5529999999999999,"15min":1.075},"min":{"1min":0.48899999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":3.2029999999999998,"5min":41.286999999999999,"15min":41.286999999999999},"last":1.3120000000000001},{"interface":"front","average":{"1min":1.5920000000000001,"5min":1.6970000000000001,"15min":1.0980000000000001},"min":{"1min":0.44700000000000001,"5min":0.40000000000000002,"15min":0.35399999999999998},"max":{"1min":7.9180000000000001,"5min":41.234999999999999,"15min":41.234999999999999},"last":1.028}]},{"osd":10,"last update":"Fri Feb 9 17:31:33 2024","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.081,"15min":0.98699999999999999},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":5.7690000000000001,"5min":5.7690000000000001,"15min":8.4659999999999993},"last":5.8959999999999999},{"interface":"front","average":{"1min":1.476,"5min":1.23,"15min":0.97699999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":5.5810000000000004,"5min":6.1479999999999997,"15min":6.1479999999999997},"last":0.69499999999999995}]},{"osd":11,"last update":"Fri Feb 9 17:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.6399999999999999,"5min":1.3759999999999999,"15min":1.0589999999999999},"min":{"1min":0.41999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":6.3899999999999997,"5min":6.3899999999999997,"15min":6.3899999999999997},"last":0.67000000000000004},{"interface":"front","average":{"1min":1.8240000000000001,"5min":1.24,"15min":1.01},"min":{"1min":0.42699999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":6.9580000000000002,"5min":8.1799999999999997,"15min":8.1799999999999997},"last":5.9169999999999998}]}]},{"osd":6,"up_from":49,"seq":210453397648,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2422912,"kb_used_data":2293172,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":91322240,"statfs":{"total":95995035648,"available":93513973760,"internally_reserved":0,"allocated":2348208128,"data_stored":2347940700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":132839878},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:31:54 2024","interfaces":[{"interface":"back","average":{"1min":1.96,"5min":1.579,"15min":1.036},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.24299999999999999},"max":{"1min":9.5570000000000004,"5min":11.582000000000001,"15min":24.544},"last":0.33300000000000002},{"interface":"front","average":{"1min":1.51,"5min":1.3720000000000001,"15min":0.98199999999999998},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":6.1210000000000004,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.78200000000000003}]},{"osd":1,"last update":"Fri Feb 9 17:31:54 2024","interfaces":[{"interface":"back","average":{"1min":1.641,"5min":1.5209999999999999,"15min":1.036},"min":{"1min":0.308,"5min":0.27400000000000002,"15min":0.27000000000000002},"max":{"1min":9.2739999999999991,"5min":9.4469999999999992,"15min":23.954000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":1.7010000000000001,"5min":1.5780000000000001,"15min":1.069},"min":{"1min":0.379,"5min":0.318,"15min":0.318},"max":{"1min":9.3079999999999998,"5min":9.4930000000000003,"15min":23.984999999999999},"last":0.56799999999999995}]},{"osd":2,"last update":"Fri Feb 9 17:31:54 2024","interfaces":[{"interface":"back","average":{"1min":1.542,"5min":1.6699999999999999,"15min":1.127},"min":{"1min":0.32700000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":6.7199999999999998,"5min":10.682,"15min":24.329999999999998},"last":0.39000000000000001},{"interface":"front","average":{"1min":1.96,"5min":1.821,"15min":1.1279999999999999},"min":{"1min":0.42999999999999999,"5min":0.32100000000000001,"15min":0.28000000000000003},"max":{"1min":6.8819999999999997,"5min":11.553000000000001,"15min":12.906000000000001},"last":0.45900000000000002}]},{"osd":3,"last update":"Fri Feb 9 17:31:54 2024","interfaces":[{"interface":"back","average":{"1min":1.7010000000000001,"5min":2.0910000000000002,"15min":1.1819999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.248},"max":{"1min":9.6920000000000002,"5min":32.545000000000002,"15min":32.545000000000002},"last":1.032},{"interface":"front","average":{"1min":1.645,"5min":1.6779999999999999,"15min":1.1220000000000001},"min":{"1min":0.35099999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":6.6310000000000002,"5min":32.670999999999999,"15min":32.670999999999999},"last":0.89700000000000002}]},{"osd":4,"last update":"Fri Feb 9 17:31:54 2024","interfaces":[{"interface":"back","average":{"1min":1.6140000000000001,"5min":0.995,"15min":0.68000000000000005},"min":{"1min":0.314,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":6.2409999999999997,"5min":8.7110000000000003,"15min":8.7110000000000003},"last":0.53800000000000003},{"interface":"front","average":{"1min":1.8140000000000001,"5min":1.1970000000000001,"15min":0.77700000000000002},"min":{"1min":0.41599999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":6.2859999999999996,"5min":8.8209999999999997,"15min":8.8209999999999997},"last":0.42799999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:31:54 2024","interfaces":[{"interface":"back","average":{"1min":1.887,"5min":1.0449999999999999,"15min":0.72699999999999998},"min":{"1min":0.40999999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":6.8040000000000003,"5min":6.8040000000000003,"15min":6.8040000000000003},"last":0.496},{"interface":"front","average":{"1min":1.7050000000000001,"5min":1.0660000000000001,"15min":0.72399999999999998},"min":{"1min":0.39500000000000002,"5min":0.29299999999999998,"15min":0.27400000000000002},"max":{"1min":4.9509999999999996,"5min":7.601,"15min":7.601},"last":0.64200000000000002}]},{"osd":7,"last update":"Fri Feb 9 17:31:28 2024","interfaces":[{"interface":"back","average":{"1min":1.6459999999999999,"5min":1.0409999999999999,"15min":0.76600000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":6.6820000000000004,"5min":6.6820000000000004,"15min":6.6820000000000004},"last":0.57899999999999996},{"interface":"front","average":{"1min":2.6579999999999999,"5min":1.268,"15min":0.872},"min":{"1min":0.55400000000000005,"5min":0.309,"15min":0.309},"max":{"1min":8.9130000000000003,"5min":11.512,"15min":11.512},"last":0.88500000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:26:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":0.876,"15min":0.79600000000000004},"min":{"1min":0.39900000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":4.0339999999999998,"5min":4.4880000000000004,"15min":4.4880000000000004},"last":0.434},{"interface":"front","average":{"1min":1.0469999999999999,"5min":0.88600000000000001,"15min":0.78300000000000003},"min":{"1min":0.375,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":3.6539999999999999,"5min":4.8570000000000002,"15min":4.8570000000000002},"last":0.45000000000000001}]},{"osd":9,"last update":"Fri Feb 9 17:32:07 2024","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.286,"15min":0.97599999999999998},"min":{"1min":0.50800000000000001,"5min":0.36899999999999999,"15min":0.35999999999999999},"max":{"1min":3.6880000000000002,"5min":10.803000000000001,"15min":10.803000000000001},"last":0.625},{"interface":"front","average":{"1min":1.22,"5min":1.3029999999999999,"15min":1.0309999999999999},"min":{"1min":0.52400000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":3.7440000000000002,"5min":6.3109999999999999,"15min":6.3109999999999999},"last":4.359}]},{"osd":10,"last update":"Fri Feb 9 17:31:28 2024","interfaces":[{"interface":"back","average":{"1min":1.5429999999999999,"5min":1.4490000000000001,"15min":1.0449999999999999},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":6.875,"5min":9.5399999999999991,"15min":9.5399999999999991},"last":0.55200000000000005},{"interface":"front","average":{"1min":1.657,"5min":1.337,"15min":1.0149999999999999},"min":{"1min":0.42499999999999999,"5min":0.42299999999999999,"15min":0.39200000000000002},"max":{"1min":4.3419999999999996,"5min":9.5210000000000008,"15min":9.5210000000000008},"last":2.782}]},{"osd":11,"last update":"Fri Feb 9 17:31:54 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.278,"15min":1.5189999999999999},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":6.8230000000000004,"5min":6.8230000000000004,"15min":6.8230000000000004},"last":0.51200000000000001},{"interface":"front","average":{"1min":1.6240000000000001,"5min":1.403,"15min":1.6850000000000001},"min":{"1min":0.56000000000000005,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":6.2350000000000003,"5min":6.3200000000000003,"15min":6.3200000000000003},"last":2.3719999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575018,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2001332,"kb_used_data":1835500,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":91743820,"statfs":{"total":95995035648,"available":93945671680,"internally_reserved":0,"allocated":1879552000,"data_stored":1879288124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":169802181},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.264,"5min":1.7090000000000001,"15min":1.0089999999999999},"min":{"1min":0.255,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":4.3010000000000002,"5min":12.492000000000001,"15min":12.492000000000001},"last":0.309},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.8919999999999999,"15min":1.0329999999999999},"min":{"1min":0.218,"5min":0.214,"15min":0.21299999999999999},"max":{"1min":5.2089999999999996,"5min":9.8439999999999994,"15min":9.8439999999999994},"last":0.627}]},{"osd":2,"last update":"Fri Feb 9 17:31:28 2024","interfaces":[{"interface":"back","average":{"1min":1.496,"5min":2.0169999999999999,"15min":1.1299999999999999},"min":{"1min":0.26700000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.0490000000000004,"5min":12.01,"15min":12.01},"last":0.40500000000000003},{"interface":"front","average":{"1min":1.216,"5min":2.5150000000000001,"15min":1.2649999999999999},"min":{"1min":0.25600000000000001,"5min":0.255,"15min":0.255},"max":{"1min":7.9770000000000003,"5min":17.459,"15min":17.459},"last":0.34499999999999997}]},{"osd":3,"last update":"Fri Feb 9 17:31:48 2024","interfaces":[{"interface":"back","average":{"1min":1.571,"5min":2.0590000000000002,"15min":1.3899999999999999},"min":{"1min":0.32300000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":5.2359999999999998,"5min":16.088000000000001,"15min":16.088000000000001},"last":4.0679999999999996},{"interface":"front","average":{"1min":0.98999999999999999,"5min":1.927,"15min":1.1479999999999999},"min":{"1min":0.245,"5min":0.222,"15min":0.222},"max":{"1min":3.5630000000000002,"5min":12.378,"15min":12.378},"last":4.1239999999999997}]},{"osd":4,"last update":"Fri Feb 9 17:28:03 2024","interfaces":[{"interface":"back","average":{"1min":1.593,"5min":1.323,"15min":0.81899999999999995},"min":{"1min":0.28199999999999997,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":7.3559999999999999,"5min":7.7210000000000001,"15min":7.7210000000000001},"last":0.64400000000000002},{"interface":"front","average":{"1min":2.359,"5min":1.5209999999999999,"15min":0.89200000000000002},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":9.3979999999999997,"5min":9.5280000000000005,"15min":9.5280000000000005},"last":0.51200000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:31:36 2024","interfaces":[{"interface":"back","average":{"1min":1.73,"5min":1.8380000000000001,"15min":1.127},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":4.742,"5min":7.3760000000000003,"15min":7.7679999999999998},"last":0.67800000000000005},{"interface":"front","average":{"1min":1.8859999999999999,"5min":1.4419999999999999,"15min":0.94599999999999995},"min":{"1min":0.34699999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":7.9009999999999998,"5min":7.9009999999999998,"15min":7.9009999999999998},"last":0.60099999999999998}]},{"osd":6,"last update":"Fri Feb 9 17:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.6319999999999999,"5min":1.718,"15min":1.2310000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":5.7709999999999999,"5min":10.275,"15min":10.275},"last":0.73799999999999999},{"interface":"front","average":{"1min":1.377,"5min":1.587,"15min":1.1759999999999999},"min":{"1min":0.36699999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":4.1890000000000001,"5min":7.0510000000000002,"15min":8.0229999999999997},"last":0.44800000000000001}]},{"osd":7,"last update":"Fri Feb 9 17:31:28 2024","interfaces":[{"interface":"back","average":{"1min":2.387,"5min":2.1099999999999999,"15min":1.2989999999999999},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":15.148999999999999,"5min":15.148999999999999,"15min":15.148999999999999},"last":0.39100000000000001},{"interface":"front","average":{"1min":2.02,"5min":2.0409999999999999,"15min":1.3029999999999999},"min":{"1min":0.41399999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":13.834,"5min":13.834,"15min":13.834},"last":0.56000000000000005}]},{"osd":8,"last update":"Fri Feb 9 17:31:23 2024","interfaces":[{"interface":"back","average":{"1min":1.0609999999999999,"5min":1.3300000000000001,"15min":1.486},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":4.1929999999999996,"5min":4.1929999999999996,"15min":4.1929999999999996},"last":4.851},{"interface":"front","average":{"1min":1.504,"5min":1.9279999999999999,"15min":2.27},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":5.1369999999999996,"5min":6.7530000000000001,"15min":6.7530000000000001},"last":5.0659999999999998}]},{"osd":9,"last update":"Fri Feb 9 17:32:08 2024","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.621,"15min":1.2769999999999999},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":4.3799999999999999,"5min":7.3140000000000001,"15min":7.8010000000000002},"last":0.85799999999999998},{"interface":"front","average":{"1min":1.1850000000000001,"5min":1.752,"15min":1.3069999999999999},"min":{"1min":0.40400000000000003,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":4.1230000000000002,"5min":12.117000000000001,"15min":12.117000000000001},"last":2.0680000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:31:36 2024","interfaces":[{"interface":"back","average":{"1min":1.488,"5min":1.5329999999999999,"15min":1.111},"min":{"1min":0.44600000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":4.2439999999999998,"5min":7.149,"15min":7.149},"last":0.752},{"interface":"front","average":{"1min":1.673,"5min":2.0499999999999998,"15min":1.345},"min":{"1min":0.501,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":7.8550000000000004,"5min":12.087999999999999,"15min":12.087999999999999},"last":0.69299999999999995}]},{"osd":11,"last update":"Fri Feb 9 17:31:47 2024","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":2.097,"15min":2.4849999999999999},"min":{"1min":0.52900000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":7.7919999999999998,"5min":12.169,"15min":12.169},"last":0.48299999999999998},{"interface":"front","average":{"1min":1.5149999999999999,"5min":2.1320000000000001,"15min":2.286},"min":{"1min":0.44600000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":7.8220000000000001,"5min":12.144,"15min":12.144},"last":0.54400000000000004}]}]},{"osd":0,"up_from":0,"seq":34359738542,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411492,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":553848,"kb_used_data":454636,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93191304,"statfs":{"total":95995035648,"available":95427895296,"internally_reserved":0,"allocated":465547264,"data_stored":465283388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:31:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.881,"15min":0.98199999999999998},"min":{"1min":0.24399999999999999,"5min":0.23699999999999999,"15min":0.221},"max":{"1min":6.0350000000000001,"5min":17.251000000000001,"15min":17.251000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.81599999999999995,"5min":2.1120000000000001,"15min":1.0880000000000001},"min":{"1min":0.251,"5min":0.23100000000000001,"15min":0.22900000000000001},"max":{"1min":3.7170000000000001,"5min":23.097999999999999,"15min":23.097999999999999},"last":0.504}]},{"osd":1,"last update":"Fri Feb 9 17:31:24 2024","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":1.9299999999999999,"15min":1.1000000000000001},"min":{"1min":0.26700000000000002,"5min":0.249,"15min":0.249},"max":{"1min":3.6789999999999998,"5min":27.289000000000001,"15min":27.289000000000001},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.95799999999999996,"5min":2.1749999999999998,"15min":1.1799999999999999},"min":{"1min":0.26700000000000002,"5min":0.25,"15min":0.25},"max":{"1min":3.7330000000000001,"5min":37.585999999999999,"15min":37.585999999999999},"last":0.57299999999999995}]},{"osd":3,"last update":"Fri Feb 9 17:31:52 2024","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":2.339,"15min":1.246},"min":{"1min":0.33700000000000002,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":5.4340000000000002,"5min":37.634999999999998,"15min":37.634999999999998},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.85299999999999998,"5min":2.0640000000000001,"15min":1.119},"min":{"1min":0.27500000000000002,"5min":0.255,"15min":0.23100000000000001},"max":{"1min":3.5459999999999998,"5min":38.924999999999997,"15min":38.924999999999997},"last":3.8119999999999998}]},{"osd":4,"last update":"Fri Feb 9 17:31:58 2024","interfaces":[{"interface":"back","average":{"1min":2.0339999999999998,"5min":1.7370000000000001,"15min":1.0820000000000001},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.254},"max":{"1min":5.8380000000000001,"5min":7.6399999999999997,"15min":8.8089999999999993},"last":3.5920000000000001},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.347,"15min":0.97299999999999998},"min":{"1min":0.33700000000000002,"5min":0.315,"15min":0.30399999999999999},"max":{"1min":4.0350000000000001,"5min":13.305999999999999,"15min":13.305999999999999},"last":0.54700000000000004}]},{"osd":5,"last update":"Fri Feb 9 17:31:31 2024","interfaces":[{"interface":"back","average":{"1min":2.1040000000000001,"5min":1.871,"15min":1.099},"min":{"1min":0.32300000000000001,"5min":0.315,"15min":0.28999999999999998},"max":{"1min":6.4980000000000002,"5min":7.5839999999999996,"15min":8.6859999999999999},"last":3.73},{"interface":"front","average":{"1min":1.7509999999999999,"5min":1.397,"15min":0.94799999999999995},"min":{"1min":0.32300000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":6.5069999999999997,"5min":13.278,"15min":13.278},"last":3.0459999999999998}]},{"osd":6,"last update":"Fri Feb 9 17:31:58 2024","interfaces":[{"interface":"back","average":{"1min":1.915,"5min":1.849,"15min":1.1619999999999999},"min":{"1min":0.36699999999999999,"5min":0.35199999999999998,"15min":0.32100000000000001},"max":{"1min":6.077,"5min":7.6159999999999997,"15min":8.8399999999999999},"last":3.5569999999999999},{"interface":"front","average":{"1min":2.1299999999999999,"5min":1.905,"15min":1.1879999999999999},"min":{"1min":0.39900000000000002,"5min":0.35599999999999998,"15min":0.34999999999999998},"max":{"1min":9.1080000000000005,"5min":9.1080000000000005,"15min":9.1080000000000005},"last":2.9089999999999998}]},{"osd":7,"last update":"Fri Feb 9 17:31:24 2024","interfaces":[{"interface":"back","average":{"1min":1.577,"5min":1.778,"15min":1.1100000000000001},"min":{"1min":0.40100000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":6.6210000000000004,"5min":14.541,"15min":14.541},"last":2.5409999999999999},{"interface":"front","average":{"1min":2.5339999999999998,"5min":2.2450000000000001,"15min":1.2629999999999999},"min":{"1min":0.5,"5min":0.41299999999999998,"15min":0.40500000000000003},"max":{"1min":6.6340000000000003,"5min":8.4169999999999998,"15min":8.7870000000000008},"last":2.5259999999999998}]},{"osd":8,"last update":"Fri Feb 9 17:26:31 2024","interfaces":[{"interface":"back","average":{"1min":1.7529999999999999,"5min":1,"15min":0.84599999999999997},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":8.8989999999999991,"5min":8.8989999999999991,"15min":8.8989999999999991},"last":5.2350000000000003},{"interface":"front","average":{"1min":2.4969999999999999,"5min":1.1080000000000001,"15min":0.90600000000000003},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":9.4190000000000005,"5min":9.4190000000000005,"15min":9.4190000000000005},"last":0.57499999999999996}]},{"osd":9,"last update":"Fri Feb 9 17:32:12 2024","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.6879999999999999,"15min":1.2070000000000001},"min":{"1min":0.45900000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":4.6529999999999996,"5min":7.7089999999999996,"15min":8.9190000000000005},"last":0.626},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.355,"15min":1.093},"min":{"1min":0.44500000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":3.8090000000000002,"5min":13.218,"15min":13.218},"last":0.59099999999999997}]},{"osd":10,"last update":"Fri Feb 9 17:31:40 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":1.4259999999999999,"15min":1.0680000000000001},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":2.3559999999999999,"5min":13.244999999999999,"15min":13.244999999999999},"last":1.3919999999999999},{"interface":"front","average":{"1min":0.98099999999999998,"5min":1.571,"15min":1.1100000000000001},"min":{"1min":0.42899999999999999,"5min":0.42499999999999999,"15min":0.42099999999999999},"max":{"1min":3.133,"5min":14.499000000000001,"15min":14.499000000000001},"last":1.4199999999999999}]},{"osd":11,"last update":"Fri Feb 9 17:31:52 2024","interfaces":[{"interface":"back","average":{"1min":1.6719999999999999,"5min":1.554,"15min":2.0379999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":5.931,"5min":13.186,"15min":13.186},"last":0.60299999999999998},{"interface":"front","average":{"1min":1.214,"5min":1.6659999999999999,"15min":1.9510000000000001},"min":{"1min":0.503,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":4.3230000000000004,"5min":7.7919999999999998,"15min":7.7919999999999998},"last":2.9580000000000002}]}]},{"osd":3,"up_from":27,"seq":115964117151,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2558944,"kb_used_data":2325076,"kb_used_omap":1,"kb_used_meta":233854,"kb_avail":91186208,"statfs":{"total":95995035648,"available":93374676992,"internally_reserved":0,"allocated":2380877824,"data_stored":2380622646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":239466951},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:31:59 2024","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":2.1160000000000001,"15min":3.298},"min":{"1min":0.247,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":3.8879999999999999,"5min":19.475000000000001,"15min":19.475000000000001},"last":0.64300000000000002},{"interface":"front","average":{"1min":1.129,"5min":2.2130000000000001,"15min":3.2959999999999998},"min":{"1min":0.27900000000000003,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":6.7859999999999996,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.36299999999999999}]},{"osd":1,"last update":"Fri Feb 9 17:31:39 2024","interfaces":[{"interface":"back","average":{"1min":1.5349999999999999,"5min":2.456,"15min":1.349},"min":{"1min":0.314,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":9.7400000000000002,"5min":19.638999999999999,"15min":19.638999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":1.9830000000000001,"5min":2.0819999999999999,"15min":1.0880000000000001},"min":{"1min":0.23499999999999999,"5min":0.22500000000000001,"15min":0.20399999999999999},"max":{"1min":12.664999999999999,"5min":14.661,"15min":14.661},"last":1.744}]},{"osd":2,"last update":"Fri Feb 9 17:31:39 2024","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":2.0619999999999998,"15min":1.1359999999999999},"min":{"1min":0.27700000000000002,"5min":0.247,"15min":0.224},"max":{"1min":6.766,"5min":14.914999999999999,"15min":14.914999999999999},"last":0.45400000000000001},{"interface":"front","average":{"1min":1.175,"5min":1.972,"15min":1.0489999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.218},"max":{"1min":9.3089999999999993,"5min":9.3089999999999993,"15min":9.3089999999999993},"last":2.153}]},{"osd":4,"last update":"Fri Feb 9 17:31:59 2024","interfaces":[{"interface":"back","average":{"1min":1.806,"5min":2.0249999999999999,"15min":1.26},"min":{"1min":0.29099999999999998,"5min":0.26900000000000002,"15min":0.254},"max":{"1min":9.3510000000000009,"5min":10.762,"15min":10.762},"last":0.50700000000000001},{"interface":"front","average":{"1min":2.1030000000000002,"5min":1.964,"15min":1.157},"min":{"1min":0.30599999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":12.602,"5min":12.602,"15min":12.602},"last":5.6890000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:31:26 2024","interfaces":[{"interface":"back","average":{"1min":2.0379999999999998,"5min":1.4930000000000001,"15min":0.92500000000000004},"min":{"1min":0.34499999999999997,"5min":0.30499999999999999,"15min":0.27300000000000002},"max":{"1min":5.1449999999999996,"5min":7.0110000000000001,"15min":7.0110000000000001},"last":1.976},{"interface":"front","average":{"1min":2.0819999999999999,"5min":1.548,"15min":0.94299999999999995},"min":{"1min":0.40200000000000002,"5min":0.36699999999999999,"15min":0.33300000000000002},"max":{"1min":5.1790000000000003,"5min":7.1600000000000001,"15min":7.1600000000000001},"last":2.117}]},{"osd":6,"last update":"Fri Feb 9 17:31:55 2024","interfaces":[{"interface":"back","average":{"1min":1.6140000000000001,"5min":1.8480000000000001,"15min":1.1339999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.32200000000000001},"max":{"1min":7.9169999999999998,"5min":19.582999999999998,"15min":19.582999999999998},"last":5.5739999999999998},{"interface":"front","average":{"1min":1.6399999999999999,"5min":1.446,"15min":0.94299999999999995},"min":{"1min":0.32200000000000001,"5min":0.30199999999999999,"15min":0.29499999999999998},"max":{"1min":7.9009999999999998,"5min":7.9009999999999998,"15min":7.9009999999999998},"last":6.2009999999999996}]},{"osd":7,"last update":"Fri Feb 9 17:31:23 2024","interfaces":[{"interface":"back","average":{"1min":1.7410000000000001,"5min":2.3879999999999999,"15min":1.3440000000000001},"min":{"1min":0.46400000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":9.327,"5min":10.791,"15min":10.791},"last":2.4399999999999999},{"interface":"front","average":{"1min":1.7509999999999999,"5min":2.0819999999999999,"15min":1.1950000000000001},"min":{"1min":0.38500000000000001,"5min":0.375,"15min":0.37},"max":{"1min":9.3399999999999999,"5min":19.550999999999998,"15min":19.550999999999998},"last":1.548}]},{"osd":8,"last update":"Fri Feb 9 17:26:30 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":3.7629999999999999,"5min":3.7629999999999999,"15min":3.7629999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":2.327,"5min":1.22,"15min":0.93000000000000005},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":10.167,"5min":10.167,"15min":10.167},"last":0.49099999999999999}]},{"osd":9,"last update":"Fri Feb 9 17:31:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":1.3839999999999999,"15min":1.014},"min":{"1min":0.45900000000000002,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":2.4889999999999999,"5min":7.0810000000000004,"15min":7.0810000000000004},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.78400000000000003,"5min":2.234,"15min":1.3460000000000001},"min":{"1min":0.379,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":2.597,"5min":13.717000000000001,"15min":13.717000000000001},"last":2.085}]},{"osd":10,"last update":"Fri Feb 9 17:31:39 2024","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.931,"15min":1.25},"min":{"1min":0.40899999999999997,"5min":0.40200000000000002,"15min":0.39400000000000002},"max":{"1min":3.9980000000000002,"5min":19.611000000000001,"15min":19.611000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":1.048,"5min":1.637,"15min":1.1120000000000001},"min":{"1min":0.48599999999999999,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":3.137,"5min":10.036,"15min":10.036},"last":2.0139999999999998}]},{"osd":11,"last update":"Fri Feb 9 17:31:59 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.5669999999999999,"15min":1.0820000000000001},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.38800000000000001},"max":{"1min":2.3319999999999999,"5min":7.1360000000000001,"15min":7.1360000000000001},"last":2.0529999999999999},{"interface":"front","average":{"1min":1.9299999999999999,"5min":2.0329999999999999,"15min":1.4099999999999999},"min":{"1min":0.443,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":9.4169999999999998,"5min":10.882999999999999,"15min":10.882999999999999},"last":0.54100000000000004}]}]},{"osd":4,"up_from":32,"seq":137438953626,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3380020,"kb_used_data":3226668,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":90365132,"statfs":{"total":95995035648,"available":92533895168,"internally_reserved":0,"allocated":3304108032,"data_stored":3303844156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":157022661},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:31:53 2024","interfaces":[{"interface":"back","average":{"1min":2.286,"5min":1.075,"15min":0.873},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":10.48,"5min":10.48,"15min":10.48},"last":4.1900000000000004},{"interface":"front","average":{"1min":1.5740000000000001,"5min":1.01,"15min":0.91600000000000004},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":6.0049999999999999,"5min":6.0049999999999999,"15min":6.0049999999999999},"last":0.65700000000000003}]},{"osd":1,"last update":"Fri Feb 9 17:31:59 2024","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.6200000000000001,"15min":1.1599999999999999},"min":{"1min":0.316,"5min":0.255,"15min":0.246},"max":{"1min":4.6980000000000004,"5min":14.221,"15min":42.542999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":1.96,"5min":1.5580000000000001,"15min":1.155},"min":{"1min":0.30299999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":9.2769999999999992,"5min":9.2769999999999992,"15min":41.813000000000002},"last":0.35299999999999998}]},{"osd":2,"last update":"Fri Feb 9 17:28:51 2024","interfaces":[{"interface":"back","average":{"1min":2.536,"5min":1.546,"15min":0.91400000000000003},"min":{"1min":0.29999999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":8.8819999999999997,"5min":8.8819999999999997,"15min":8.8819999999999997},"last":0.622},{"interface":"front","average":{"1min":2.214,"5min":1.7330000000000001,"15min":0.97799999999999998},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":9.5039999999999996,"5min":9.5039999999999996,"15min":9.5039999999999996},"last":0.53800000000000003}]},{"osd":3,"last update":"Fri Feb 9 17:31:59 2024","interfaces":[{"interface":"back","average":{"1min":2.1949999999999998,"5min":2.1850000000000001,"15min":1.2450000000000001},"min":{"1min":0.30399999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":10.526,"5min":33.037999999999997,"15min":33.037999999999997},"last":4.149},{"interface":"front","average":{"1min":1.373,"5min":2.0590000000000002,"15min":1.151},"min":{"1min":0.33500000000000002,"5min":0.314,"15min":0.28199999999999997},"max":{"1min":3.4089999999999998,"5min":33.054000000000002,"15min":33.054000000000002},"last":0.71899999999999997}]},{"osd":5,"last update":"Fri Feb 9 17:31:25 2024","interfaces":[{"interface":"back","average":{"1min":1.9490000000000001,"5min":1.0580000000000001,"15min":0.67200000000000004},"min":{"1min":0.28199999999999997,"5min":0.222,"15min":0.222},"max":{"1min":4.3730000000000002,"5min":5.101,"15min":5.101},"last":0.38600000000000001},{"interface":"front","average":{"1min":2.0190000000000001,"5min":1.157,"15min":0.753},"min":{"1min":0.38,"5min":0.29199999999999998,"15min":0.26000000000000001},"max":{"1min":4.3920000000000003,"5min":5.2220000000000004,"15min":5.2220000000000004},"last":0.42999999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:31:59 2024","interfaces":[{"interface":"back","average":{"1min":2.3330000000000002,"5min":1.3280000000000001,"15min":0.84499999999999997},"min":{"1min":0.28000000000000003,"5min":0.247,"15min":0.247},"max":{"1min":18.09,"5min":18.09,"15min":18.09},"last":4.5599999999999996},{"interface":"front","average":{"1min":2.4300000000000002,"5min":1.294,"15min":0.80300000000000005},"min":{"1min":0.33800000000000002,"5min":0.30099999999999999,"15min":0.28000000000000003},"max":{"1min":18.126000000000001,"5min":18.126000000000001,"15min":18.126000000000001},"last":4.5279999999999996}]},{"osd":7,"last update":"Fri Feb 9 17:31:25 2024","interfaces":[{"interface":"back","average":{"1min":2.5169999999999999,"5min":1.4339999999999999,"15min":0.89400000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":7.4790000000000001,"5min":15.223000000000001,"15min":15.223000000000001},"last":7.8040000000000003},{"interface":"front","average":{"1min":3.1400000000000001,"5min":1.4370000000000001,"15min":0.88700000000000001},"min":{"1min":0.29699999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":19.047000000000001,"5min":19.047000000000001,"15min":19.047000000000001},"last":4.3220000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:31:25 2024","interfaces":[{"interface":"back","average":{"1min":1.877,"5min":1.298,"15min":1.2470000000000001},"min":{"1min":0.34499999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":4.5650000000000004,"5min":4.5650000000000004,"15min":4.5650000000000004},"last":0.67100000000000004},{"interface":"front","average":{"1min":1.954,"5min":1.181,"15min":1.0660000000000001},"min":{"1min":0.41699999999999998,"5min":0.308,"15min":0.308},"max":{"1min":4.6180000000000003,"5min":4.6180000000000003,"15min":4.6180000000000003},"last":0.75600000000000001}]},{"osd":9,"last update":"Fri Feb 9 17:32:11 2024","interfaces":[{"interface":"back","average":{"1min":1.101,"5min":1.294,"15min":0.997},"min":{"1min":0.45000000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":3.7309999999999999,"5min":5.1929999999999996,"15min":5.1929999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":2.25,"5min":1.3100000000000001,"15min":1},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":10.555,"5min":10.555,"15min":10.555},"last":4.2169999999999996}]},{"osd":10,"last update":"Fri Feb 9 17:31:43 2024","interfaces":[{"interface":"back","average":{"1min":2.1389999999999998,"5min":1.3979999999999999,"15min":1.0429999999999999},"min":{"1min":0.46600000000000003,"5min":0.378,"15min":0.378},"max":{"1min":4.6890000000000001,"5min":5.2469999999999999,"15min":5.2469999999999999},"last":0.73299999999999998},{"interface":"front","average":{"1min":2.149,"5min":1.3260000000000001,"15min":1.0189999999999999},"min":{"1min":0.40500000000000003,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":10.597,"5min":10.597,"15min":10.597},"last":0.78100000000000003}]},{"osd":11,"last update":"Fri Feb 9 17:32:09 2024","interfaces":[{"interface":"back","average":{"1min":1.514,"5min":1.4790000000000001,"15min":1.369},"min":{"1min":0.48199999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":3.786,"5min":6.3689999999999998,"15min":6.3689999999999998},"last":0.80500000000000005},{"interface":"front","average":{"1min":1.964,"5min":1.399,"15min":1.2190000000000001},"min":{"1min":0.42799999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":10.625,"5min":10.625,"15min":10.625},"last":4.2439999999999998}]}]},{"osd":5,"up_from":40,"seq":171798691988,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1959868,"kb_used_data":1848176,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":91785284,"statfs":{"total":95995035648,"available":93988130816,"internally_reserved":0,"allocated":1892532224,"data_stored":1892264276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:31:24 2024","interfaces":[{"interface":"back","average":{"1min":2.4809999999999999,"5min":1.76,"15min":1.002},"min":{"1min":0.32300000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":7.5270000000000001,"5min":11.923,"15min":11.923},"last":6.8780000000000001},{"interface":"front","average":{"1min":1.782,"5min":1.4530000000000001,"15min":0.90600000000000003},"min":{"1min":0.28399999999999997,"5min":0.224,"15min":0.224},"max":{"1min":7.0659999999999998,"5min":9.4239999999999995,"15min":9.4239999999999995},"last":3.4350000000000001}]},{"osd":1,"last update":"Fri Feb 9 17:31:24 2024","interfaces":[{"interface":"back","average":{"1min":2.5600000000000001,"5min":2.0009999999999999,"15min":1.1930000000000001},"min":{"1min":0.34699999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":7.3369999999999997,"5min":16.896000000000001,"15min":29.715},"last":3.3180000000000001},{"interface":"front","average":{"1min":2.5150000000000001,"5min":1.843,"15min":1.1759999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26600000000000001},"max":{"1min":8.2940000000000005,"5min":16.878,"15min":29.969999999999999},"last":0.371}]},{"osd":2,"last update":"Fri Feb 9 17:31:24 2024","interfaces":[{"interface":"back","average":{"1min":2.5659999999999998,"5min":2,"15min":1.107},"min":{"1min":0.30499999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":8.5099999999999998,"5min":9.8219999999999992,"15min":9.8219999999999992},"last":0.38700000000000001},{"interface":"front","average":{"1min":2.4990000000000001,"5min":1.921,"15min":1.107},"min":{"1min":0.35799999999999998,"5min":0.318,"15min":0.318},"max":{"1min":7.3730000000000002,"5min":11.978,"15min":11.978},"last":3.407}]},{"osd":3,"last update":"Fri Feb 9 17:31:24 2024","interfaces":[{"interface":"back","average":{"1min":2.734,"5min":2.036,"15min":1.1120000000000001},"min":{"1min":0.30399999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":8.3360000000000003,"5min":10.587,"15min":10.587},"last":0.32800000000000001},{"interface":"front","average":{"1min":2.8340000000000001,"5min":1.7230000000000001,"15min":1.03},"min":{"1min":0.42799999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":8.1859999999999999,"5min":8.1859999999999999,"15min":8.1859999999999999},"last":3.359}]},{"osd":4,"last update":"Fri Feb 9 17:31:35 2024","interfaces":[{"interface":"back","average":{"1min":2.2320000000000002,"5min":0.99099999999999999,"15min":0.63700000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":7.8460000000000001,"5min":7.8460000000000001,"15min":7.8460000000000001},"last":8.7260000000000009},{"interface":"front","average":{"1min":2.9449999999999998,"5min":1.2090000000000001,"15min":0.73999999999999999},"min":{"1min":0.34100000000000003,"5min":0.27900000000000003,"15min":0.27000000000000002},"max":{"1min":8.6349999999999998,"5min":8.6349999999999998,"15min":8.6349999999999998},"last":4.8209999999999997}]},{"osd":6,"last update":"Fri Feb 9 17:31:58 2024","interfaces":[{"interface":"back","average":{"1min":2.895,"5min":1.3720000000000001,"15min":0.85599999999999998},"min":{"1min":0.46700000000000003,"5min":0.30499999999999999,"15min":0.28999999999999998},"max":{"1min":8.6080000000000005,"5min":8.9250000000000007,"15min":8.9250000000000007},"last":0.29699999999999999},{"interface":"front","average":{"1min":2.371,"5min":1.3680000000000001,"15min":0.82699999999999996},"min":{"1min":0.41699999999999998,"5min":0.29399999999999998,"15min":0.27200000000000002},"max":{"1min":13.859999999999999,"5min":13.859999999999999,"15min":13.859999999999999},"last":3.5950000000000002}]},{"osd":7,"last update":"Fri Feb 9 17:31:24 2024","interfaces":[{"interface":"back","average":{"1min":2.1070000000000002,"5min":1.2430000000000001,"15min":0.84699999999999998},"min":{"1min":0.443,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":7.1369999999999996,"5min":18.626999999999999,"15min":18.626999999999999},"last":6.9279999999999999},{"interface":"front","average":{"1min":3.48,"5min":1.536,"15min":0.94299999999999995},"min":{"1min":0.38900000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":13.821999999999999,"5min":18.701000000000001,"15min":18.701000000000001},"last":3.3839999999999999}]},{"osd":8,"last update":"Fri Feb 9 17:26:36 2024","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":0.91000000000000003,"15min":0.78000000000000003},"min":{"1min":0.39400000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":5.556,"5min":5.556,"15min":5.556},"last":0.40300000000000002},{"interface":"front","average":{"1min":1.3720000000000001,"5min":0.92600000000000005,"15min":0.82999999999999996},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":5.9000000000000004,"5min":5.9000000000000004,"15min":5.9000000000000004},"last":0.36799999999999999}]},{"osd":9,"last update":"Fri Feb 9 17:32:04 2024","interfaces":[{"interface":"back","average":{"1min":2.1549999999999998,"5min":1.306,"15min":0.998},"min":{"1min":0.53800000000000003,"5min":0.371,"15min":0.35999999999999999},"max":{"1min":8.4280000000000008,"5min":8.4280000000000008,"15min":8.4280000000000008},"last":0.58899999999999997},{"interface":"front","average":{"1min":2.1299999999999999,"5min":1.214,"15min":1.0089999999999999},"min":{"1min":0.54500000000000004,"5min":0.312,"15min":0.312},"max":{"1min":8.3689999999999998,"5min":8.3689999999999998,"15min":8.3689999999999998},"last":0.624}]},{"osd":10,"last update":"Fri Feb 9 17:31:35 2024","interfaces":[{"interface":"back","average":{"1min":2.5630000000000002,"5min":1.2450000000000001,"15min":1.0349999999999999},"min":{"1min":0.58199999999999996,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":7.4480000000000004,"5min":7.4480000000000004,"15min":13.071999999999999},"last":3.4590000000000001},{"interface":"front","average":{"1min":2.8460000000000001,"5min":1.4259999999999999,"15min":1.0700000000000001},"min":{"1min":0.57599999999999996,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":7.4720000000000004,"5min":7.4720000000000004,"15min":11.587},"last":3.5059999999999998}]},{"osd":11,"last update":"Fri Feb 9 17:32:08 2024","interfaces":[{"interface":"back","average":{"1min":2.056,"5min":1.337,"15min":1.2729999999999999},"min":{"1min":0.46500000000000002,"5min":0.441,"15min":0.441},"max":{"1min":8.3979999999999997,"5min":8.3979999999999997,"15min":8.3979999999999997},"last":0.65700000000000003},{"interface":"front","average":{"1min":1.9159999999999999,"5min":1.2949999999999999,"15min":0.98799999999999999},"min":{"1min":0.61099999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":7.4249999999999998,"5min":7.4249999999999998,"15min":7.4249999999999998},"last":3.4830000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2480603136,"data_stored":2480603136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1873281024,"data_stored":1873281024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":463667200,"data_stored":463667200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2376728576,"data_stored":2376728576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3297443840,"data_stored":3297443840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1881411584,"data_stored":1881407512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2337210368,"data_stored":2337210368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1914114048,"data_stored":1914109976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":951517184,"data_stored":951517184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1404178432,"data_stored":1404174360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":472317952,"data_stored":472317952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":3440574464,"data_stored":3440574464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-09T17:32:19.611 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-02-09T17:32:19.612 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd pool set .mgr pg_num 11 2024-02-09T17:32:19.647 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:19.679 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:32:19.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:19.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:32:19.777 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is set 2024-02-09T17:32:19.787 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:32:19.825 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:19.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:19 smithi162 ceph-mon[44304]: pgmap v710: 17 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 7.0 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 548 op/s; 647/342720 objects degraded (0.189%); 33307/342720 objects misplaced (9.718%); 16 MiB/s, 261 objects/s recovering 2024-02-09T17:32:19.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:19 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1539964656' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:32:19.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:19 smithi099 ceph-mon[39420]: pgmap v710: 17 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 7.0 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 548 op/s; 647/342720 objects degraded (0.189%); 33307/342720 objects misplaced (9.718%); 16 MiB/s, 261 objects/s recovering 2024-02-09T17:32:19.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:19 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1539964656' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:32:20.176 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:19 smithi179 ceph-mon[43975]: pgmap v710: 17 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 7.0 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 548 op/s; 647/342720 objects degraded (0.189%); 33307/342720 objects misplaced (9.718%); 16 MiB/s, 261 objects/s recovering 2024-02-09T17:32:20.176 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:19 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1539964656' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:32:20.965 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:20 smithi099 ceph-mon[39420]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:32:20.966 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:20 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1539964656' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:32:20.966 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:20 smithi099 ceph-mon[39420]: osdmap e244: 12 total, 12 up, 11 in 2024-02-09T17:32:21.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:20 smithi179 ceph-mon[43975]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:32:21.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:20 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1539964656' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:32:21.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:20 smithi179 ceph-mon[43975]: osdmap e244: 12 total, 12 up, 11 in 2024-02-09T17:32:21.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:20 smithi162 ceph-mon[44304]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:32:21.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:20 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1539964656' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:32:21.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:20 smithi162 ceph-mon[44304]: osdmap e244: 12 total, 12 up, 11 in 2024-02-09T17:32:21.504 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:32:21.504 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:32:21.504 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:32:21.504 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:32:21.504 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:32:21.505 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:32:21.505 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:32:21.507 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:32:21.536 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:32:21.560 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:32:21.586 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:32:21.616 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:32:21.617 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:32:21.618 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:32:21.618 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:32:21.618 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:32:21.618 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:32:21.618 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:32:21.618 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:32:21.618 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:32:21.618 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:32:21.618 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:32:21.618 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:32:21.618 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:32:21.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:21 smithi099 ceph-mon[39420]: pgmap v712: 17 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.9 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 609 op/s; 963/337464 objects degraded (0.285%); 32518/337464 objects misplaced (9.636%); 14 MiB/s, 217 objects/s recovering 2024-02-09T17:32:21.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:21 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 963/337464 objects degraded (0.285%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:32:21.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:21 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1883975803' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:32:22.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:21 smithi179 ceph-mon[43975]: pgmap v712: 17 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.9 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 609 op/s; 963/337464 objects degraded (0.285%); 32518/337464 objects misplaced (9.636%); 14 MiB/s, 217 objects/s recovering 2024-02-09T17:32:22.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:21 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 963/337464 objects degraded (0.285%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:32:22.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:21 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1883975803' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:32:22.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:21 smithi162 ceph-mon[44304]: pgmap v712: 17 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.9 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 609 op/s; 963/337464 objects degraded (0.285%); 32518/337464 objects misplaced (9.636%); 14 MiB/s, 217 objects/s recovering 2024-02-09T17:32:22.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:21 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 963/337464 objects degraded (0.285%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:32:22.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:21 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1883975803' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:32:22.242 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:32:22.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:22.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:32:22.348 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:32:22.390 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:22.707 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-09T17:32:23.242 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:32:23.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:23.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:32:23.346 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:32:23.415 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:23.463 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:32:23.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:23.470 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:32:23.571 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:32:23.618 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:23.652 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:32:23.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:23.661 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:32:23.762 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:32:23.794 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:23 smithi099 ceph-mon[39420]: pgmap v713: 17 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.8 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 632 op/s; 963/333465 objects degraded (0.289%); 31494/333465 objects misplaced (9.444%); 17 MiB/s, 267 objects/s recovering 2024-02-09T17:32:23.801 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:23.847 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:32:23.852 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:32:23.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:23.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:32:23.902 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:32:23.902 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 20, 2024-02-09T17:32:23.902 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 600, 2024-02-09T17:32:23.902 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [ 2024-02-09T17:32:23.902 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.902 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:152747 2.5 2:ad43f63e:::benchmark_data_smithi179_77138_object19339:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-02-09T17:32:23.902 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:26:40.344141+0000", 2024-02-09T17:32:23.903 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 343.55408067299999, 2024-02-09T17:32:23.903 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.80258039800000003, 2024-02-09T17:32:23.903 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:32:23.903 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:32:23.903 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:32:23.903 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:32:23.903 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:32:23.903 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 152747 2024-02-09T17:32:23.903 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.903 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:32:23.903 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.903 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:32:23.903 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:40.344141+0000", 2024-02-09T17:32:23.904 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:23.904 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.904 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.904 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:32:23.904 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:40.344141+0000", 2024-02-09T17:32:23.904 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:23.904 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.904 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.904 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:32:23.904 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:40.344142+0000", 2024-02-09T17:32:23.904 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.9199999999999999e-07 2024-02-09T17:32:23.904 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.905 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.905 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:32:23.905 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:40.344148+0000", 2024-02-09T17:32:23.905 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2329999999999997e-06 2024-02-09T17:32:23.905 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.905 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.905 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:32:23.905 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:40.344156+0000", 2024-02-09T17:32:23.905 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.5550000000000003e-06 2024-02-09T17:32:23.905 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.905 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.905 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:32:23.905 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:40.344165+0000", 2024-02-09T17:32:23.905 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.5690000000000005e-06 2024-02-09T17:32:23.906 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.906 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.906 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:23.906 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:40.344220+0000", 2024-02-09T17:32:23.906 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.5000999999999997e-05 2024-02-09T17:32:23.906 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.906 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.906 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:32:23.906 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:40.344227+0000", 2024-02-09T17:32:23.906 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.6529999999999994e-06 2024-02-09T17:32:23.906 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.906 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.906 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:23.906 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:41.136569+0000", 2024-02-09T17:32:23.906 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.79234206399999996 2024-02-09T17:32:23.907 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.907 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.907 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:32:23.907 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:41.138526+0000", 2024-02-09T17:32:23.907 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0019561890000000001 2024-02-09T17:32:23.907 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.907 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.907 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 4", 2024-02-09T17:32:23.907 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:41.138618+0000", 2024-02-09T17:32:23.907 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.2074e-05 2024-02-09T17:32:23.907 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.907 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.907 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:32:23.907 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:41.141673+0000", 2024-02-09T17:32:23.908 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0030553669999999998 2024-02-09T17:32:23.908 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.908 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.908 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:23.908 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:41.146591+0000", 2024-02-09T17:32:23.908 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00491764 2024-02-09T17:32:23.908 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.908 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.908 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:32:23.908 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:41.146633+0000", 2024-02-09T17:32:23.908 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2574999999999998e-05 2024-02-09T17:32:23.908 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.908 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.908 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:32:23.909 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:41.146722+0000", 2024-02-09T17:32:23.909 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.8585999999999995e-05 2024-02-09T17:32:23.909 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:23.909 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:32:23.909 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:23.909 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.909 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.909 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:152763 2.5 2:a71711e0:::benchmark_data_smithi179_77138_object19355:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-02-09T17:32:23.909 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:26:40.344182+0000", 2024-02-09T17:32:23.909 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 343.55403976100001, 2024-02-09T17:32:23.909 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.80284695500000003, 2024-02-09T17:32:23.909 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:32:23.909 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:32:23.909 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:32:23.910 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:32:23.910 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:32:23.910 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 152763 2024-02-09T17:32:23.910 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.910 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:32:23.910 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.910 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:32:23.910 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:40.344182+0000", 2024-02-09T17:32:23.910 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:23.910 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.910 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.910 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:32:23.910 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:40.344182+0000", 2024-02-09T17:32:23.911 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:23.911 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.911 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.911 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:32:23.911 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:40.344183+0000", 2024-02-09T17:32:23.911 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.3000000000000002e-07 2024-02-09T17:32:23.911 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.911 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.911 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:32:23.911 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:40.344185+0000", 2024-02-09T17:32:23.911 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.8169999999999999e-06 2024-02-09T17:32:23.911 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.911 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.911 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:32:23.912 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:40.344189+0000", 2024-02-09T17:32:23.912 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.6940000000000003e-06 2024-02-09T17:32:23.912 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.912 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.912 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:32:23.912 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:40.344193+0000", 2024-02-09T17:32:23.912 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.501e-06 2024-02-09T17:32:23.912 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.912 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.912 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:23.912 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:40.344246+0000", 2024-02-09T17:32:23.912 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2828999999999999e-05 2024-02-09T17:32:23.912 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.913 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.913 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:32:23.913 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:40.344252+0000", 2024-02-09T17:32:23.913 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.9959999999999999e-06 2024-02-09T17:32:23.913 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.913 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.913 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:23.913 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:41.141718+0000", 2024-02-09T17:32:23.913 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.79746603699999996 2024-02-09T17:32:23.913 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.913 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.913 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:32:23.913 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:41.141870+0000", 2024-02-09T17:32:23.913 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00015232700000000001 2024-02-09T17:32:23.913 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.914 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.914 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 4", 2024-02-09T17:32:23.914 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:41.141996+0000", 2024-02-09T17:32:23.914 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000126112 2024-02-09T17:32:23.914 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.914 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.914 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:32:23.914 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:41.145302+0000", 2024-02-09T17:32:23.914 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0033059690000000002 2024-02-09T17:32:23.914 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.914 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.914 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:23.914 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:41.146882+0000", 2024-02-09T17:32:23.914 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.001579621 2024-02-09T17:32:23.915 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.915 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.915 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:32:23.915 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:41.146939+0000", 2024-02-09T17:32:23.915 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7719000000000001e-05 2024-02-09T17:32:23.915 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.915 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.915 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:32:23.915 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:26:41.147029+0000", 2024-02-09T17:32:23.915 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.0002999999999999e-05 2024-02-09T17:32:23.915 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:23.915 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:32:23.915 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:23.915 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.916 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.916 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.24817.0:18699 3.6 3:621f053a:::benchmark_data_smithi179_90440_object18698:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e208)", 2024-02-09T17:32:23.916 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:30:11.348148+0000", 2024-02-09T17:32:23.916 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 132.55007367499999, 2024-02-09T17:32:23.916 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.099549631, 2024-02-09T17:32:23.916 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:32:23.916 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:32:23.916 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:32:23.916 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.24817", 2024-02-09T17:32:23.916 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:32:23.916 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 18699 2024-02-09T17:32:23.916 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.916 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:32:23.917 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.917 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:32:23.917 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.348148+0000", 2024-02-09T17:32:23.917 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:23.917 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.917 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.917 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:32:23.917 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.348148+0000", 2024-02-09T17:32:23.917 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:23.917 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.917 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.917 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:32:23.917 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.348150+0000", 2024-02-09T17:32:23.917 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3230000000000001e-06 2024-02-09T17:32:23.918 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.918 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.918 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:32:23.918 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.348227+0000", 2024-02-09T17:32:23.918 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.6837999999999999e-05 2024-02-09T17:32:23.918 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.918 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.918 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:32:23.918 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.348236+0000", 2024-02-09T17:32:23.918 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.6660000000000003e-06 2024-02-09T17:32:23.918 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.918 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.918 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:32:23.918 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.348243+0000", 2024-02-09T17:32:23.918 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.8569999999999997e-06 2024-02-09T17:32:23.919 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.919 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.919 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:23.919 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.348286+0000", 2024-02-09T17:32:23.919 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.3071999999999997e-05 2024-02-09T17:32:23.919 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.919 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.919 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:32:23.919 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.348385+0000", 2024-02-09T17:32:23.919 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.8684999999999995e-05 2024-02-09T17:32:23.919 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.919 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.919 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:32:23.920 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.348491+0000", 2024-02-09T17:32:23.920 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00010642299999999999 2024-02-09T17:32:23.920 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.920 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.920 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:32:23.920 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.349246+0000", 2024-02-09T17:32:23.920 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00075498000000000004 2024-02-09T17:32:23.920 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.920 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.920 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:23.920 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.349686+0000", 2024-02-09T17:32:23.920 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00043964399999999999 2024-02-09T17:32:23.920 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.920 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.921 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:23.921 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.447609+0000", 2024-02-09T17:32:23.921 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.097922912000000001 2024-02-09T17:32:23.921 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.921 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.921 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:32:23.921 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.447663+0000", 2024-02-09T17:32:23.921 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3942e-05 2024-02-09T17:32:23.921 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.921 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.921 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:32:23.921 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.447698+0000", 2024-02-09T17:32:23.921 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.5289e-05 2024-02-09T17:32:23.921 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:23.922 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:32:23.922 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:23.922 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.922 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.922 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.24817.0:18704 3.1 3:81fd704f:::benchmark_data_smithi179_90440_object18703:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e208)", 2024-02-09T17:32:23.922 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:30:11.349249+0000", 2024-02-09T17:32:23.922 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 132.54897300299999, 2024-02-09T17:32:23.922 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.097922830000000002, 2024-02-09T17:32:23.922 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:32:23.922 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:32:23.922 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:32:23.922 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.24817", 2024-02-09T17:32:23.922 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:32:23.922 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 18704 2024-02-09T17:32:23.923 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.923 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:32:23.923 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.923 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:32:23.923 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.349249+0000", 2024-02-09T17:32:23.923 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:23.923 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.923 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.923 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:32:23.923 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.349249+0000", 2024-02-09T17:32:23.923 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:23.923 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.923 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.923 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:32:23.924 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.349250+0000", 2024-02-09T17:32:23.924 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.091e-06 2024-02-09T17:32:23.924 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.924 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.924 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:32:23.924 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.349276+0000", 2024-02-09T17:32:23.924 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.6015000000000001e-05 2024-02-09T17:32:23.924 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.924 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.924 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:32:23.924 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.349298+0000", 2024-02-09T17:32:23.924 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.1594999999999999e-05 2024-02-09T17:32:23.924 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.925 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.925 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:32:23.925 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.349304+0000", 2024-02-09T17:32:23.925 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.5440000000000004e-06 2024-02-09T17:32:23.925 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.925 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.925 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:23.925 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.349353+0000", 2024-02-09T17:32:23.925 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.8288000000000003e-05 2024-02-09T17:32:23.925 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.925 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.925 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:32:23.925 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.349466+0000", 2024-02-09T17:32:23.926 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000113542 2024-02-09T17:32:23.926 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.926 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.926 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 4,10", 2024-02-09T17:32:23.926 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.349588+0000", 2024-02-09T17:32:23.926 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000121493 2024-02-09T17:32:23.926 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.926 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.926 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:32:23.926 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.350857+0000", 2024-02-09T17:32:23.926 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.001268871 2024-02-09T17:32:23.926 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.926 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.926 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:23.927 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.446853+0000", 2024-02-09T17:32:23.927 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.095996385000000004 2024-02-09T17:32:23.927 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.927 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.927 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:23.927 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.447062+0000", 2024-02-09T17:32:23.927 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00020928599999999999 2024-02-09T17:32:23.927 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.927 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.927 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:32:23.927 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.447123+0000", 2024-02-09T17:32:23.927 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.0328000000000002e-05 2024-02-09T17:32:23.927 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.927 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.927 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:32:23.927 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.447172+0000", 2024-02-09T17:32:23.928 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.9391999999999997e-05 2024-02-09T17:32:23.928 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:23.928 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:32:23.928 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:23.928 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.928 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.928 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.24817.0:18705 3.1 3:8d4b7f68:::benchmark_data_smithi179_90440_object18704:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e208)", 2024-02-09T17:32:23.928 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:30:11.349318+0000", 2024-02-09T17:32:23.928 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 132.548904061, 2024-02-09T17:32:23.928 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.097991940999999999, 2024-02-09T17:32:23.928 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:32:23.928 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:32:23.928 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:32:23.928 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.24817", 2024-02-09T17:32:23.929 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:32:23.929 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 18705 2024-02-09T17:32:23.929 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.929 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:32:23.929 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.929 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:32:23.929 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.349318+0000", 2024-02-09T17:32:23.929 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:23.929 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.929 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.929 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:32:23.929 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.349318+0000", 2024-02-09T17:32:23.929 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:23.929 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.929 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.930 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:32:23.930 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.349319+0000", 2024-02-09T17:32:23.930 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2200000000000004e-07 2024-02-09T17:32:23.930 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.930 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.930 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:32:23.930 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.349427+0000", 2024-02-09T17:32:23.930 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000108084 2024-02-09T17:32:23.930 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.930 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.930 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:32:23.930 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.349439+0000", 2024-02-09T17:32:23.930 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2529e-05 2024-02-09T17:32:23.930 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.930 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.931 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:32:23.931 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.349447+0000", 2024-02-09T17:32:23.931 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.7810000000000005e-06 2024-02-09T17:32:23.931 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.931 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.931 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:23.931 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.350223+0000", 2024-02-09T17:32:23.931 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00077561700000000004 2024-02-09T17:32:23.931 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.931 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.931 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:32:23.931 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.350307+0000", 2024-02-09T17:32:23.931 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.4715999999999993e-05 2024-02-09T17:32:23.931 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.932 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.932 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 4,10", 2024-02-09T17:32:23.932 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.350394+0000", 2024-02-09T17:32:23.932 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.6823999999999994e-05 2024-02-09T17:32:23.932 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.932 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.932 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:32:23.932 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.350957+0000", 2024-02-09T17:32:23.932 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00056278000000000003 2024-02-09T17:32:23.932 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.932 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.932 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:23.932 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.446912+0000", 2024-02-09T17:32:23.932 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.095954950999999997 2024-02-09T17:32:23.933 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.933 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.933 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:23.933 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.447212+0000", 2024-02-09T17:32:23.933 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00029981499999999999 2024-02-09T17:32:23.933 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.933 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.933 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:32:23.933 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.447273+0000", 2024-02-09T17:32:23.933 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.1654000000000003e-05 2024-02-09T17:32:23.933 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.933 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.933 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:32:23.933 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.447310+0000", 2024-02-09T17:32:23.934 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.6668000000000002e-05 2024-02-09T17:32:23.934 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:23.934 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:32:23.934 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:23.934 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.934 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.934 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.24817.0:18709 3.6 3:6309abd0:::benchmark_data_smithi179_90440_object18708:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e208)", 2024-02-09T17:32:23.934 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:30:11.350735+0000", 2024-02-09T17:32:23.934 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 132.54748698200001, 2024-02-09T17:32:23.934 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.098570109000000003, 2024-02-09T17:32:23.934 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:32:23.934 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:32:23.934 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:32:23.934 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.24817", 2024-02-09T17:32:23.935 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:32:23.935 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 18709 2024-02-09T17:32:23.935 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.935 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:32:23.935 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.935 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:32:23.935 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.350735+0000", 2024-02-09T17:32:23.935 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:23.935 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.935 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.935 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:32:23.935 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.350735+0000", 2024-02-09T17:32:23.935 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:23.935 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.936 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.936 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:32:23.936 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.350736+0000", 2024-02-09T17:32:23.936 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.0200000000000001e-07 2024-02-09T17:32:23.936 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.936 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.936 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:32:23.936 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.350804+0000", 2024-02-09T17:32:23.936 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.8602999999999994e-05 2024-02-09T17:32:23.936 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.936 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.936 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:32:23.936 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.350813+0000", 2024-02-09T17:32:23.936 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.7560000000000005e-06 2024-02-09T17:32:23.937 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.937 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.937 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:32:23.937 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.350818+0000", 2024-02-09T17:32:23.937 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2549999999999997e-06 2024-02-09T17:32:23.937 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.937 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.937 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:23.937 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.350851+0000", 2024-02-09T17:32:23.937 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.256e-05 2024-02-09T17:32:23.937 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.937 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.938 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:32:23.938 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.350938+0000", 2024-02-09T17:32:23.938 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.6725999999999996e-05 2024-02-09T17:32:23.938 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.938 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.938 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:32:23.938 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.351019+0000", 2024-02-09T17:32:23.938 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.1372000000000006e-05 2024-02-09T17:32:23.938 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.938 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.938 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:23.938 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.447753+0000", 2024-02-09T17:32:23.938 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.096734028 2024-02-09T17:32:23.939 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.939 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.939 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:23.939 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.448688+0000", 2024-02-09T17:32:23.939 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00093440999999999999 2024-02-09T17:32:23.939 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.939 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.939 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:32:23.939 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.449249+0000", 2024-02-09T17:32:23.939 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00056104699999999996 2024-02-09T17:32:23.939 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.939 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.939 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:32:23.939 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.449284+0000", 2024-02-09T17:32:23.940 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.5862999999999999e-05 2024-02-09T17:32:23.940 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.940 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.940 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:32:23.940 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.449305+0000", 2024-02-09T17:32:23.940 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0786999999999999e-05 2024-02-09T17:32:23.940 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:23.940 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:32:23.940 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:23.940 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.941 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.941 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.24817.0:18711 3.e e208/199 3:78c8d768:::benchmark_data_smithi179_90440_object18710:head v 208'1163, mlcod=208'1161)", 2024-02-09T17:32:23.941 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:30:11.351159+0000", 2024-02-09T17:32:23.941 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 132.54706351199999, 2024-02-09T17:32:23.941 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.098631736999999997, 2024-02-09T17:32:23.941 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:32:23.941 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:32:23.941 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:32:23.941 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.941 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:32:23.941 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.351159+0000", 2024-02-09T17:32:23.941 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:23.942 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.942 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.942 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:32:23.942 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.351159+0000", 2024-02-09T17:32:23.942 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:23.942 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.942 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.942 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:32:23.942 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.351159+0000", 2024-02-09T17:32:23.942 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.5000000000000003e-07 2024-02-09T17:32:23.942 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.942 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.942 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:32:23.942 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.351277+0000", 2024-02-09T17:32:23.942 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000118016 2024-02-09T17:32:23.943 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.943 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.943 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:32:23.943 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.446690+0000", 2024-02-09T17:32:23.943 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.095413170000000005 2024-02-09T17:32:23.943 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.943 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.943 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:32:23.943 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.446731+0000", 2024-02-09T17:32:23.943 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.0499e-05 2024-02-09T17:32:23.943 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.943 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.943 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:23.943 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.447812+0000", 2024-02-09T17:32:23.944 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0010815009999999999 2024-02-09T17:32:23.944 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.944 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.944 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:32:23.944 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.447839+0000", 2024-02-09T17:32:23.944 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.7076999999999998e-05 2024-02-09T17:32:23.944 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.944 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.944 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:32:23.944 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.449759+0000", 2024-02-09T17:32:23.944 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0019190780000000001 2024-02-09T17:32:23.944 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.944 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.944 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:32:23.945 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.449790+0000", 2024-02-09T17:32:23.945 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.1846e-05 2024-02-09T17:32:23.945 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:23.945 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:32:23.945 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:23.945 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.945 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.945 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop_reply(client.24817.0:18699 3.6 e208/199 ondisk, result = 0)", 2024-02-09T17:32:23.945 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:30:11.351299+0000", 2024-02-09T17:32:23.945 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 132.54692318599999, 2024-02-09T17:32:23.945 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.096409972999999996, 2024-02-09T17:32:23.945 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:32:23.945 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T17:32:23.945 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:32:23.946 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.946 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:32:23.946 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.351299+0000", 2024-02-09T17:32:23.946 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:23.946 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.946 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.946 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:32:23.946 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.351299+0000", 2024-02-09T17:32:23.946 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:23.946 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.946 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.946 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:32:23.946 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.351300+0000", 2024-02-09T17:32:23.946 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.1999999999999999e-07 2024-02-09T17:32:23.947 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.947 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.947 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:32:23.947 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.351303+0000", 2024-02-09T17:32:23.947 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.3400000000000002e-06 2024-02-09T17:32:23.947 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.947 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.947 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:32:23.947 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.446696+0000", 2024-02-09T17:32:23.947 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.095393033000000002 2024-02-09T17:32:23.947 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.947 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.947 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:32:23.947 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.446731+0000", 2024-02-09T17:32:23.948 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.4632000000000003e-05 2024-02-09T17:32:23.948 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.948 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.948 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:23.948 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.447596+0000", 2024-02-09T17:32:23.948 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00086589900000000005 2024-02-09T17:32:23.948 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.948 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.948 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:32:23.948 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.447604+0000", 2024-02-09T17:32:23.948 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.289e-06 2024-02-09T17:32:23.948 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.948 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.948 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:32:23.949 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.447709+0000", 2024-02-09T17:32:23.949 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00010516 2024-02-09T17:32:23.949 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:23.949 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:32:23.949 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:23.949 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.949 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.949 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.24817.0:22172 3.e e209/199 3:7be95ab9:::benchmark_data_smithi179_90440_object22171:head v 209'1393, mlcod=209'1390)", 2024-02-09T17:32:23.949 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:30:13.000844+0000", 2024-02-09T17:32:23.949 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 130.89737790300001, 2024-02-09T17:32:23.949 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.077101845000000002, 2024-02-09T17:32:23.949 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:32:23.949 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:32:23.949 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:32:23.950 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.950 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:32:23.950 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:13.000844+0000", 2024-02-09T17:32:23.950 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:23.950 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.950 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.950 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:32:23.950 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:13.000844+0000", 2024-02-09T17:32:23.950 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:23.950 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.950 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.950 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:32:23.950 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:13.000845+0000", 2024-02-09T17:32:23.950 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.0699999999999998e-07 2024-02-09T17:32:23.950 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.950 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.951 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:32:23.951 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:13.000986+0000", 2024-02-09T17:32:23.951 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014124 2024-02-09T17:32:23.951 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.951 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.951 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:32:23.951 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:13.073319+0000", 2024-02-09T17:32:23.951 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.072333433000000003 2024-02-09T17:32:23.951 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.951 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.951 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:32:23.951 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:13.073335+0000", 2024-02-09T17:32:23.951 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5855999999999999e-05 2024-02-09T17:32:23.951 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.952 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.952 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:23.952 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:13.074284+0000", 2024-02-09T17:32:23.952 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00094875600000000004 2024-02-09T17:32:23.952 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.952 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.952 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:32:23.952 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:13.074303+0000", 2024-02-09T17:32:23.952 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.9479000000000001e-05 2024-02-09T17:32:23.952 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.952 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.952 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:32:23.952 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:13.077903+0000", 2024-02-09T17:32:23.952 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0035991339999999999 2024-02-09T17:32:23.953 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.953 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.953 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:32:23.953 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:13.077946+0000", 2024-02-09T17:32:23.953 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.354e-05 2024-02-09T17:32:23.953 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:23.953 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:32:23.953 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:23.953 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.953 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.953 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.24817.0:25398 3.6 3:6e133f36:::benchmark_data_smithi179_90440_object25397:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-02-09T17:32:23.953 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:30:14.110393+0000", 2024-02-09T17:32:23.953 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 129.787829351, 2024-02-09T17:32:23.953 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.092345608999999995, 2024-02-09T17:32:23.954 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:32:23.954 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:32:23.954 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:32:23.954 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.24817", 2024-02-09T17:32:23.954 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:32:23.954 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 25398 2024-02-09T17:32:23.954 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.954 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:32:23.954 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.954 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:32:23.954 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.110393+0000", 2024-02-09T17:32:23.954 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:23.954 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.954 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.955 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:32:23.955 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.110393+0000", 2024-02-09T17:32:23.955 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:23.955 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.955 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.955 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:32:23.955 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.110393+0000", 2024-02-09T17:32:23.955 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.9800000000000003e-07 2024-02-09T17:32:23.955 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.955 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.955 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:32:23.955 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.110450+0000", 2024-02-09T17:32:23.955 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6780000000000002e-05 2024-02-09T17:32:23.956 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.956 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.956 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:32:23.956 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.110459+0000", 2024-02-09T17:32:23.956 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.5220000000000007e-06 2024-02-09T17:32:23.956 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.956 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.956 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:32:23.956 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.110465+0000", 2024-02-09T17:32:23.956 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.5980000000000002e-06 2024-02-09T17:32:23.956 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.956 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.956 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:23.956 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.112526+0000", 2024-02-09T17:32:23.957 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0020603679999999999 2024-02-09T17:32:23.957 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.957 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.957 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:32:23.957 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.112649+0000", 2024-02-09T17:32:23.957 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012294000000000001 2024-02-09T17:32:23.957 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.957 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.957 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:32:23.957 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.112775+0000", 2024-02-09T17:32:23.957 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012598000000000001 2024-02-09T17:32:23.957 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.957 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.957 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:23.958 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.115333+0000", 2024-02-09T17:32:23.958 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0025582339999999999 2024-02-09T17:32:23.958 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.958 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.958 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:23.958 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.115367+0000", 2024-02-09T17:32:23.958 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.4415999999999997e-05 2024-02-09T17:32:23.958 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.958 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.958 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:32:23.958 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.202641+0000", 2024-02-09T17:32:23.958 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.087273476000000003 2024-02-09T17:32:23.958 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.959 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.959 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:32:23.959 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.202696+0000", 2024-02-09T17:32:23.959 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.5563000000000003e-05 2024-02-09T17:32:23.959 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.959 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.959 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:32:23.959 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.202738+0000", 2024-02-09T17:32:23.959 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2134e-05 2024-02-09T17:32:23.959 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:23.959 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:32:23.959 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:23.959 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.959 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.960 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.24817.0:25399 3.6 3:64f47d7b:::benchmark_data_smithi179_90440_object25398:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-02-09T17:32:23.960 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:30:14.110512+0000", 2024-02-09T17:32:23.960 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 129.787710081, 2024-02-09T17:32:23.960 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.092312228999999996, 2024-02-09T17:32:23.960 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:32:23.960 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:32:23.960 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:32:23.960 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.24817", 2024-02-09T17:32:23.960 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:32:23.960 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 25399 2024-02-09T17:32:23.960 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.960 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:32:23.960 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.960 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:32:23.961 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.110512+0000", 2024-02-09T17:32:23.961 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:23.961 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.961 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.961 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:32:23.961 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.110512+0000", 2024-02-09T17:32:23.961 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:23.961 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.961 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.961 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:32:23.961 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.110513+0000", 2024-02-09T17:32:23.961 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.1799999999999995e-07 2024-02-09T17:32:23.961 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.961 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.962 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:32:23.962 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.110573+0000", 2024-02-09T17:32:23.962 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.0086e-05 2024-02-09T17:32:23.962 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.962 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.962 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:32:23.962 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.110579+0000", 2024-02-09T17:32:23.962 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.3550000000000001e-06 2024-02-09T17:32:23.962 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.962 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.962 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:32:23.962 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.110585+0000", 2024-02-09T17:32:23.962 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7150000000000003e-06 2024-02-09T17:32:23.962 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.963 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.963 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:23.963 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.113693+0000", 2024-02-09T17:32:23.963 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0031083159999999999 2024-02-09T17:32:23.963 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.963 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.963 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:32:23.963 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.113759+0000", 2024-02-09T17:32:23.963 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.5554000000000003e-05 2024-02-09T17:32:23.963 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.963 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.963 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:32:23.963 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.113836+0000", 2024-02-09T17:32:23.964 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.7359000000000003e-05 2024-02-09T17:32:23.964 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.964 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.964 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:23.964 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.115408+0000", 2024-02-09T17:32:23.964 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.001572197 2024-02-09T17:32:23.964 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.964 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.964 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:23.964 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.202575+0000", 2024-02-09T17:32:23.964 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.087166304999999999 2024-02-09T17:32:23.964 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.964 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.965 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:32:23.965 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.202750+0000", 2024-02-09T17:32:23.965 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00017564499999999999 2024-02-09T17:32:23.965 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.965 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.965 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:32:23.965 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.202791+0000", 2024-02-09T17:32:23.965 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.1261999999999999e-05 2024-02-09T17:32:23.965 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.965 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.965 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:32:23.965 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.202824+0000", 2024-02-09T17:32:23.965 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.2817000000000001e-05 2024-02-09T17:32:23.965 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:23.966 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:32:23.966 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:23.966 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.966 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.966 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.24817.0:25406 3.6 3:67639d95:::benchmark_data_smithi179_90440_object25405:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-02-09T17:32:23.966 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:30:14.112439+0000", 2024-02-09T17:32:23.966 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 129.78578337299999, 2024-02-09T17:32:23.966 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.090566478000000006, 2024-02-09T17:32:23.966 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:32:23.966 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:32:23.966 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:32:23.966 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.24817", 2024-02-09T17:32:23.966 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:32:23.966 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 25406 2024-02-09T17:32:23.966 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.966 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:32:23.967 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.967 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:32:23.967 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.112439+0000", 2024-02-09T17:32:23.967 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:23.967 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.967 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.967 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:32:23.967 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.112439+0000", 2024-02-09T17:32:23.967 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:23.967 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.967 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.967 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:32:23.967 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.112440+0000", 2024-02-09T17:32:23.967 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1179999999999999e-06 2024-02-09T17:32:23.968 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.968 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.968 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:32:23.968 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.112505+0000", 2024-02-09T17:32:23.968 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.5559e-05 2024-02-09T17:32:23.968 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.968 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.968 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:32:23.968 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.112517+0000", 2024-02-09T17:32:23.968 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1943000000000001e-05 2024-02-09T17:32:23.968 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.968 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.968 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:32:23.969 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.112526+0000", 2024-02-09T17:32:23.969 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.2959999999999999e-06 2024-02-09T17:32:23.969 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.969 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.969 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:23.969 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.114478+0000", 2024-02-09T17:32:23.969 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.001952217 2024-02-09T17:32:23.969 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.969 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.969 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:32:23.969 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.114553+0000", 2024-02-09T17:32:23.969 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.4840999999999997e-05 2024-02-09T17:32:23.969 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.969 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.969 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:32:23.970 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.114656+0000", 2024-02-09T17:32:23.970 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000103379 2024-02-09T17:32:23.970 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.970 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.970 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:23.970 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.202624+0000", 2024-02-09T17:32:23.970 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.087968214000000003 2024-02-09T17:32:23.970 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.970 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.970 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:32:23.970 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.202834+0000", 2024-02-09T17:32:23.970 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00021013 2024-02-09T17:32:23.970 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.970 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.971 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:23.971 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.202924+0000", 2024-02-09T17:32:23.971 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.9542000000000004e-05 2024-02-09T17:32:23.971 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.971 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.971 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:32:23.971 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.202972+0000", 2024-02-09T17:32:23.971 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.7920000000000002e-05 2024-02-09T17:32:23.971 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.971 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.971 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:32:23.971 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.203005+0000", 2024-02-09T17:32:23.971 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.3318999999999997e-05 2024-02-09T17:32:23.971 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:23.972 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:32:23.972 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:23.972 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.972 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.972 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.24817.0:25407 3.6 3:6a6f6c71:::benchmark_data_smithi179_90440_object25406:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-02-09T17:32:23.972 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:30:14.112590+0000", 2024-02-09T17:32:23.972 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 129.7856319, 2024-02-09T17:32:23.972 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.091444870999999997, 2024-02-09T17:32:23.972 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:32:23.972 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:32:23.972 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:32:23.972 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.24817", 2024-02-09T17:32:23.972 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:32:23.972 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 25407 2024-02-09T17:32:23.973 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.973 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:32:23.973 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.973 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:32:23.973 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.112590+0000", 2024-02-09T17:32:23.973 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:23.973 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.973 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.973 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:32:23.973 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.112590+0000", 2024-02-09T17:32:23.973 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:23.973 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.973 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.973 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:32:23.973 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.112591+0000", 2024-02-09T17:32:23.974 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.2900000000000002e-07 2024-02-09T17:32:23.974 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.974 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.974 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:32:23.974 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.112715+0000", 2024-02-09T17:32:23.974 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012376299999999999 2024-02-09T17:32:23.974 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.974 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.974 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:32:23.974 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.112725+0000", 2024-02-09T17:32:23.974 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0268999999999999e-05 2024-02-09T17:32:23.974 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.974 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.974 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:32:23.975 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.112732+0000", 2024-02-09T17:32:23.975 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.2099999999999996e-06 2024-02-09T17:32:23.975 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.975 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.975 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:23.975 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.115437+0000", 2024-02-09T17:32:23.975 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0027045070000000001 2024-02-09T17:32:23.975 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.975 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.975 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:32:23.975 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.200282+0000", 2024-02-09T17:32:23.975 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.084844615999999998 2024-02-09T17:32:23.975 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.975 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.975 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:32:23.976 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.200438+0000", 2024-02-09T17:32:23.976 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000156427 2024-02-09T17:32:23.976 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.976 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.976 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:32:23.976 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.202840+0000", 2024-02-09T17:32:23.976 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.002402216 2024-02-09T17:32:23.976 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.976 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.976 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:23.976 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.203903+0000", 2024-02-09T17:32:23.976 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.001063327 2024-02-09T17:32:23.976 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.976 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.977 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:23.977 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.203943+0000", 2024-02-09T17:32:23.977 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9619000000000001e-05 2024-02-09T17:32:23.977 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.977 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.977 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:32:23.977 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.203999+0000", 2024-02-09T17:32:23.977 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.5427000000000003e-05 2024-02-09T17:32:23.977 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.977 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.977 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:32:23.977 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.204035+0000", 2024-02-09T17:32:23.977 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.6560999999999998e-05 2024-02-09T17:32:23.977 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:23.978 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:32:23.978 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:23.978 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.978 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.978 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.24817.0:25412 3.1 3:8bba1a7a:::benchmark_data_smithi179_90440_object25411:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-02-09T17:32:23.978 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:30:14.114505+0000", 2024-02-09T17:32:23.978 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 129.78371724100001, 2024-02-09T17:32:23.978 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.088494283000000007, 2024-02-09T17:32:23.978 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:32:23.978 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:32:23.978 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:32:23.978 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.24817", 2024-02-09T17:32:23.978 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:32:23.978 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 25412 2024-02-09T17:32:23.978 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.979 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:32:23.979 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.979 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:32:23.979 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.114505+0000", 2024-02-09T17:32:23.979 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:23.979 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.979 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.979 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:32:23.979 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.114505+0000", 2024-02-09T17:32:23.979 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:23.979 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.979 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.979 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:32:23.979 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.114506+0000", 2024-02-09T17:32:23.980 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0589999999999999e-06 2024-02-09T17:32:23.980 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.980 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.980 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:32:23.980 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.114571+0000", 2024-02-09T17:32:23.980 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.4579000000000006e-05 2024-02-09T17:32:23.980 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.980 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.980 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:32:23.980 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.114582+0000", 2024-02-09T17:32:23.980 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1482e-05 2024-02-09T17:32:23.980 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.980 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.980 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:32:23.981 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.114590+0000", 2024-02-09T17:32:23.981 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.4989999999999997e-06 2024-02-09T17:32:23.981 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.981 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.981 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:23.981 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.114639+0000", 2024-02-09T17:32:23.981 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.9478000000000003e-05 2024-02-09T17:32:23.981 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.981 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.981 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:32:23.981 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.114751+0000", 2024-02-09T17:32:23.981 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000111786 2024-02-09T17:32:23.981 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.981 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.981 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 4,10", 2024-02-09T17:32:23.982 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.114872+0000", 2024-02-09T17:32:23.982 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000121381 2024-02-09T17:32:23.982 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.982 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.982 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:32:23.982 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.202830+0000", 2024-02-09T17:32:23.982 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.087957840999999995 2024-02-09T17:32:23.982 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.982 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.982 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:23.982 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.202867+0000", 2024-02-09T17:32:23.982 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.7481999999999998e-05 2024-02-09T17:32:23.982 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.982 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.982 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:23.983 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.202910+0000", 2024-02-09T17:32:23.983 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2817e-05 2024-02-09T17:32:23.983 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.983 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.983 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:32:23.983 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.202965+0000", 2024-02-09T17:32:23.983 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.4815000000000003e-05 2024-02-09T17:32:23.983 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.983 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.983 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:32:23.983 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.202999+0000", 2024-02-09T17:32:23.983 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.4063999999999998e-05 2024-02-09T17:32:23.983 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:23.983 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:32:23.983 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:23.984 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.984 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.984 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.24817.0:25414 3.1 3:897931b5:::benchmark_data_smithi179_90440_object25413:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-02-09T17:32:23.984 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:30:14.114608+0000", 2024-02-09T17:32:23.984 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 129.78361443899999, 2024-02-09T17:32:23.984 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.088560491000000005, 2024-02-09T17:32:23.984 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:32:23.984 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:32:23.984 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:32:23.984 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.24817", 2024-02-09T17:32:23.984 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:32:23.984 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 25414 2024-02-09T17:32:23.984 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.984 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:32:23.985 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.985 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:32:23.985 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.114608+0000", 2024-02-09T17:32:23.985 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:23.985 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.985 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.985 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:32:23.985 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.114608+0000", 2024-02-09T17:32:23.985 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:23.985 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.985 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.985 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:32:23.985 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.114608+0000", 2024-02-09T17:32:23.985 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.9699999999999996e-07 2024-02-09T17:32:23.985 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.986 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.986 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:32:23.986 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.114654+0000", 2024-02-09T17:32:23.986 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.6085999999999999e-05 2024-02-09T17:32:23.986 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.986 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.986 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:32:23.986 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.114662+0000", 2024-02-09T17:32:23.986 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.3590000000000002e-06 2024-02-09T17:32:23.986 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.986 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.986 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:32:23.986 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.114667+0000", 2024-02-09T17:32:23.986 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7100000000000004e-06 2024-02-09T17:32:23.986 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.987 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.987 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:23.987 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.200410+0000", 2024-02-09T17:32:23.987 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.085742928999999996 2024-02-09T17:32:23.987 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.987 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.987 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:32:23.987 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.200548+0000", 2024-02-09T17:32:23.987 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000137546 2024-02-09T17:32:23.987 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.987 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.987 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 4,10", 2024-02-09T17:32:23.987 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.200689+0000", 2024-02-09T17:32:23.987 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014156300000000001 2024-02-09T17:32:23.988 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.988 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.988 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:32:23.988 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.202878+0000", 2024-02-09T17:32:23.988 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0021890619999999999 2024-02-09T17:32:23.988 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.988 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.988 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:23.988 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.203039+0000", 2024-02-09T17:32:23.988 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000160576 2024-02-09T17:32:23.988 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.988 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.988 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:23.988 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.203083+0000", 2024-02-09T17:32:23.988 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.4064999999999999e-05 2024-02-09T17:32:23.989 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.989 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.989 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:32:23.989 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.203135+0000", 2024-02-09T17:32:23.989 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1786000000000002e-05 2024-02-09T17:32:23.989 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.989 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.989 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:32:23.989 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.203168+0000", 2024-02-09T17:32:23.989 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.3312000000000002e-05 2024-02-09T17:32:23.989 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:23.989 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:32:23.989 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:23.989 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.990 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.990 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.24817.0:25415 3.6 3:656e5106:::benchmark_data_smithi179_90440_object25414:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-02-09T17:32:23.990 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:30:14.115162+0000", 2024-02-09T17:32:23.990 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 129.783060135, 2024-02-09T17:32:23.990 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.089027325000000004, 2024-02-09T17:32:23.990 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:32:23.990 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:32:23.990 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:32:23.990 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.24817", 2024-02-09T17:32:23.990 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:32:23.990 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 25415 2024-02-09T17:32:23.990 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.990 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:32:23.990 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.990 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:32:23.991 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.115162+0000", 2024-02-09T17:32:23.991 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:23.991 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.991 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.991 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:32:23.991 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.115162+0000", 2024-02-09T17:32:23.991 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:23.991 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.991 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.991 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:32:23.991 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.115163+0000", 2024-02-09T17:32:23.991 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.4199999999999995e-07 2024-02-09T17:32:23.991 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.991 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.992 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:32:23.992 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.115208+0000", 2024-02-09T17:32:23.992 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.5629e-05 2024-02-09T17:32:23.992 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.992 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.992 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:32:23.992 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.115219+0000", 2024-02-09T17:32:23.992 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.075e-05 2024-02-09T17:32:23.992 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.992 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.992 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:32:23.992 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.115226+0000", 2024-02-09T17:32:23.992 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.3610000000000003e-06 2024-02-09T17:32:23.992 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.992 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.993 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:23.993 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.201344+0000", 2024-02-09T17:32:23.993 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.086117970000000002 2024-02-09T17:32:23.993 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.993 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.993 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:32:23.993 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.201478+0000", 2024-02-09T17:32:23.993 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013350100000000001 2024-02-09T17:32:23.993 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.993 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.993 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:32:23.993 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.201631+0000", 2024-02-09T17:32:23.993 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00015296799999999999 2024-02-09T17:32:23.993 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.993 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.994 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:32:23.994 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.203842+0000", 2024-02-09T17:32:23.994 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0022111269999999998 2024-02-09T17:32:23.994 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.994 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.994 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:23.994 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.204071+0000", 2024-02-09T17:32:23.994 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000228464 2024-02-09T17:32:23.994 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.994 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.994 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:23.994 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.204106+0000", 2024-02-09T17:32:23.994 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.5243999999999997e-05 2024-02-09T17:32:23.994 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.994 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.994 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:32:23.995 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.204158+0000", 2024-02-09T17:32:23.995 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1916000000000002e-05 2024-02-09T17:32:23.995 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.995 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.995 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:32:23.995 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.204189+0000", 2024-02-09T17:32:23.995 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.1652999999999997e-05 2024-02-09T17:32:23.995 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:23.995 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:32:23.995 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:23.995 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.995 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.995 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.24817.0:25413 3.3 e209/199 3:c6084140:::benchmark_data_smithi179_90440_object25412:head v 209'1591, mlcod=209'1589)", 2024-02-09T17:32:23.995 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:30:14.115284+0000", 2024-02-09T17:32:23.995 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 129.782938618, 2024-02-09T17:32:23.995 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.085173436000000005, 2024-02-09T17:32:23.996 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:32:23.996 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:32:23.996 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:32:23.996 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.996 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:32:23.996 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.115284+0000", 2024-02-09T17:32:23.996 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:23.996 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.996 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.996 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:32:23.996 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.115284+0000", 2024-02-09T17:32:23.996 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:23.996 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.996 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.997 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:32:23.997 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.115284+0000", 2024-02-09T17:32:23.997 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.0199999999999998e-07 2024-02-09T17:32:23.997 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.997 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.997 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:32:23.997 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.115356+0000", 2024-02-09T17:32:23.997 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.2197e-05 2024-02-09T17:32:23.997 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.997 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.997 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:32:23.997 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.115361+0000", 2024-02-09T17:32:23.997 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.378e-06 2024-02-09T17:32:23.997 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.998 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.998 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:32:23.998 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.115365+0000", 2024-02-09T17:32:23.998 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.3859999999999999e-06 2024-02-09T17:32:23.998 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.998 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.998 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:23.998 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.115416+0000", 2024-02-09T17:32:23.998 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1493000000000003e-05 2024-02-09T17:32:23.998 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.998 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.998 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:32:23.998 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.115432+0000", 2024-02-09T17:32:23.998 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5321000000000001e-05 2024-02-09T17:32:23.999 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.999 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.999 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:32:23.999 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.200414+0000", 2024-02-09T17:32:23.999 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.084982210000000002 2024-02-09T17:32:23.999 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:23.999 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:23.999 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:32:23.999 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.200457+0000", 2024-02-09T17:32:23.999 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.3149000000000003e-05 2024-02-09T17:32:23.999 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:23.999 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:32:23.999 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:23.999 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:24.000 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:24.000 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop_reply(client.24817.0:25399 3.6 e209/199 ondisk, result = 0)", 2024-02-09T17:32:24.000 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:30:14.115635+0000", 2024-02-09T17:32:24.000 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 129.782586796, 2024-02-09T17:32:24.000 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.086950554999999999, 2024-02-09T17:32:24.000 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:32:24.000 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T17:32:24.000 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:32:24.000 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:24.000 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:32:24.000 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.115635+0000", 2024-02-09T17:32:24.000 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:24.000 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:24.001 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:24.001 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:32:24.001 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.115635+0000", 2024-02-09T17:32:24.001 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:24.001 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:24.001 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:24.001 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:32:24.001 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.115636+0000", 2024-02-09T17:32:24.001 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.2200000000000004e-07 2024-02-09T17:32:24.001 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:24.001 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:24.001 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:32:24.001 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.115640+0000", 2024-02-09T17:32:24.001 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.7330000000000001e-06 2024-02-09T17:32:24.002 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:24.002 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:24.002 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:32:24.002 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.200276+0000", 2024-02-09T17:32:24.002 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.084636246999999998 2024-02-09T17:32:24.002 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:24.002 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:24.002 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:32:24.002 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.200284+0000", 2024-02-09T17:32:24.002 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.1100000000000003e-06 2024-02-09T17:32:24.002 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:24.002 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:24.002 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:24.002 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.202561+0000", 2024-02-09T17:32:24.002 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0022765699999999999 2024-02-09T17:32:24.003 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:24.003 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:24.003 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:32:24.003 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.202568+0000", 2024-02-09T17:32:24.003 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.8099999999999998e-06 2024-02-09T17:32:24.003 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:24.003 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:24.003 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:32:24.003 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.202586+0000", 2024-02-09T17:32:24.003 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.7462999999999999e-05 2024-02-09T17:32:24.003 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:24.003 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:32:24.003 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:24.003 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:24.004 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:24.004 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.24817.0:25424 3.e e209/199 3:7abcbc48:::benchmark_data_smithi179_90440_object25423:head v 209'1598, mlcod=209'1596)", 2024-02-09T17:32:24.004 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:30:14.119179+0000", 2024-02-09T17:32:24.004 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 129.779043253, 2024-02-09T17:32:24.004 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.085506429999999994, 2024-02-09T17:32:24.004 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:32:24.004 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:32:24.004 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:32:24.004 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:24.004 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:32:24.004 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.119179+0000", 2024-02-09T17:32:24.004 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:24.004 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:24.005 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:24.005 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:32:24.005 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.119179+0000", 2024-02-09T17:32:24.005 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:24.005 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:24.005 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:24.005 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:32:24.005 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.119180+0000", 2024-02-09T17:32:24.005 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.3099999999999996e-07 2024-02-09T17:32:24.005 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:24.005 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:24.005 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:32:24.005 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.119246+0000", 2024-02-09T17:32:24.005 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.6356e-05 2024-02-09T17:32:24.005 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:24.006 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:24.006 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:32:24.006 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.200247+0000", 2024-02-09T17:32:24.006 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.081001291000000003 2024-02-09T17:32:24.006 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:24.006 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:24.006 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:32:24.006 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.200258+0000", 2024-02-09T17:32:24.006 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0740999999999999e-05 2024-02-09T17:32:24.006 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:24.006 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:24.006 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:24.006 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.202866+0000", 2024-02-09T17:32:24.006 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0026077240000000001 2024-02-09T17:32:24.007 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:24.007 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:24.007 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:32:24.007 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.202886+0000", 2024-02-09T17:32:24.007 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0590000000000001e-05 2024-02-09T17:32:24.007 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:24.007 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:24.007 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:32:24.007 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.204639+0000", 2024-02-09T17:32:24.007 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0017526379999999999 2024-02-09T17:32:24.007 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:24.007 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:24.007 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:32:24.007 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:14.204685+0000", 2024-02-09T17:32:24.008 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.6258999999999999e-05 2024-02-09T17:32:24.008 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:24.008 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:32:24.008 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:24.008 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:24.008 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:24.008 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.24817.0:29380 3.e e209/199 3:70c25e5f:::benchmark_data_smithi179_90440_object29379:head v 209'1826, mlcod=209'1824)", 2024-02-09T17:32:24.008 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:30:15.634318+0000", 2024-02-09T17:32:24.008 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 128.263904475, 2024-02-09T17:32:24.008 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.078561633000000006, 2024-02-09T17:32:24.008 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:32:24.008 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:32:24.008 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:32:24.008 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:24.009 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:32:24.009 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:15.634318+0000", 2024-02-09T17:32:24.009 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:24.009 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:24.009 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:24.009 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:32:24.009 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:15.634318+0000", 2024-02-09T17:32:24.009 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:24.009 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:24.009 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:24.009 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:32:24.009 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:15.634318+0000", 2024-02-09T17:32:24.009 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.0999999999999999e-07 2024-02-09T17:32:24.009 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:24.009 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:24.010 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:32:24.010 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:15.634362+0000", 2024-02-09T17:32:24.010 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.3747e-05 2024-02-09T17:32:24.010 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:24.010 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:24.010 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:32:24.010 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:15.634369+0000", 2024-02-09T17:32:24.010 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.0489999999999998e-06 2024-02-09T17:32:24.010 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:24.010 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:24.010 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:32:24.010 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:15.634372+0000", 2024-02-09T17:32:24.010 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.23e-06 2024-02-09T17:32:24.010 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:24.010 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:24.011 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:24.011 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:15.709695+0000", 2024-02-09T17:32:24.011 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.075323246999999996 2024-02-09T17:32:24.011 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:24.011 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:24.011 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:32:24.011 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:15.709720+0000", 2024-02-09T17:32:24.011 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.4944e-05 2024-02-09T17:32:24.011 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:24.011 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:24.011 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:32:24.011 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:15.712838+0000", 2024-02-09T17:32:24.011 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0031174110000000001 2024-02-09T17:32:24.011 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:24.012 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:24.012 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:32:24.012 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:15.712879+0000", 2024-02-09T17:32:24.012 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.1594999999999997e-05 2024-02-09T17:32:24.012 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:24.012 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:32:24.012 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:24.012 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:24.012 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:32:24.012 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:32:24.012 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:32:24.013 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:23 smithi179 ceph-mon[43975]: pgmap v713: 17 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.8 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 632 op/s; 963/333465 objects degraded (0.289%); 31494/333465 objects misplaced (9.444%); 17 MiB/s, 267 objects/s recovering 2024-02-09T17:32:24.060 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:24.093 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:32:24.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:24.097 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:32:24.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:23 smithi162 ceph-mon[44304]: pgmap v713: 17 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.8 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 632 op/s; 963/333465 objects degraded (0.289%); 31494/333465 objects misplaced (9.444%); 17 MiB/s, 267 objects/s recovering 2024-02-09T17:32:24.197 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:32:24.259 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:24.284 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:32:24.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:24.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:32:24.397 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:32:24.430 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:24.475 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:32:24.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:24.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:32:24.589 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:32:24.636 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:24.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:24.664 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:32:24.664 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:32:24.765 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:32:24.820 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:24.867 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:32:24.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:24.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:32:24.973 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:32:24.974 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-09T17:32:25.022 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:25.063 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:32:25.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:25.069 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:32:25.170 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:32:25.212 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:25.261 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:32:25.261 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:32:25.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:25.265 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:32:25.366 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:32:25.394 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:25.446 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:32:25.453 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:32:25.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:25.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:32:25.562 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:32:25.598 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:25.929 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:25 smithi099 ceph-mon[39420]: pgmap v714: 17 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.8 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 522 op/s; 963/332235 objects degraded (0.290%); 31494/332235 objects misplaced (9.479%); 14 MiB/s, 231 objects/s recovering 2024-02-09T17:32:26.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:25 smithi179 ceph-mon[43975]: pgmap v714: 17 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.8 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 522 op/s; 963/332235 objects degraded (0.290%); 31494/332235 objects misplaced (9.479%); 14 MiB/s, 231 objects/s recovering 2024-02-09T17:32:26.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:25 smithi162 ceph-mon[44304]: pgmap v714: 17 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.8 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 522 op/s; 963/332235 objects degraded (0.290%); 31494/332235 objects misplaced (9.479%); 14 MiB/s, 231 objects/s recovering 2024-02-09T17:32:26.758 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:32:26.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:26.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:32:26.868 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:32:26.938 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:26.973 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:32:26.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:26.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:32:26.989 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:26 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 963/332235 objects degraded (0.290%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:32:27.083 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:32:27.169 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:26 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 963/332235 objects degraded (0.290%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:32:27.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:26 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 963/332235 objects degraded (0.290%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:32:27.171 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:27.202 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:32:27.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:27.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:32:27.319 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:32:27.361 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:27.387 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:32:27.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:27.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:32:27.495 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:32:27.540 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:27.564 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:32:27.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:27.569 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:32:27.670 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:32:27.738 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:27.767 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:32:27.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:27.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:32:27.873 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:32:27.915 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:27.950 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:32:27.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:27.954 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:32:27.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:27 smithi099 ceph-mon[39420]: pgmap v715: 17 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.7 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 625 op/s; 1270/327420 objects degraded (0.388%); 30695/327420 objects misplaced (9.375%); 17 MiB/s, 265 objects/s recovering 2024-02-09T17:32:27.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:27 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1323679347' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-02-09T17:32:28.055 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:32:28.085 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:28.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:27 smithi179 ceph-mon[43975]: pgmap v715: 17 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.7 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 625 op/s; 1270/327420 objects degraded (0.388%); 30695/327420 objects misplaced (9.375%); 17 MiB/s, 265 objects/s recovering 2024-02-09T17:32:28.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:27 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1323679347' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-02-09T17:32:28.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:27 smithi162 ceph-mon[44304]: pgmap v715: 17 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.7 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 625 op/s; 1270/327420 objects degraded (0.388%); 30695/327420 objects misplaced (9.375%); 17 MiB/s, 265 objects/s recovering 2024-02-09T17:32:28.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:27 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1323679347' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-02-09T17:32:28.251 INFO:teuthology.orchestra.run.smithi099.stderr:set pool 1 pg_num to 11 2024-02-09T17:32:28.579 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:32:28.586 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:32:29.035 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:32:29.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:29.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:32:29.141 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:32:29.172 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:29.313 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:29 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1323679347' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-02-09T17:32:29.314 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:29 smithi099 ceph-mon[39420]: osdmap e245: 12 total, 12 up, 11 in 2024-02-09T17:32:29.314 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:29 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:32:29.314 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:29 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:32:29.314 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=2629703955,config=2679774077,health=506382104,kv=2891707644}) 2024-02-09T17:32:29.314 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {kv=5,logm=95} crc {kv=1660393686,logm=1210453227}) 2024-02-09T17:32:29.314 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3451923082}) 2024-02-09T17:32:29.314 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3545785881}) 2024-02-09T17:32:29.314 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1472237423}) 2024-02-09T17:32:29.314 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3258113875}) 2024-02-09T17:32:29.314 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1209253103}) 2024-02-09T17:32:29.314 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3406121943}) 2024-02-09T17:32:29.314 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=618692712}) 2024-02-09T17:32:29.314 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=145037479}) 2024-02-09T17:32:29.315 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3386044008}) 2024-02-09T17:32:29.315 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2606405199}) 2024-02-09T17:32:29.315 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1785375383}) 2024-02-09T17:32:29.315 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2180019715}) 2024-02-09T17:32:29.315 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=608904271}) 2024-02-09T17:32:29.315 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=570814298}) 2024-02-09T17:32:29.315 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=705248428}) 2024-02-09T17:32:29.315 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3841946239}) 2024-02-09T17:32:29.315 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1372389079}) 2024-02-09T17:32:29.315 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1826978956}) 2024-02-09T17:32:29.315 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3945031885}) 2024-02-09T17:32:29.315 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=322221387}) 2024-02-09T17:32:29.315 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4128342744}) 2024-02-09T17:32:29.315 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2847936266}) 2024-02-09T17:32:29.316 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=592380767}) 2024-02-09T17:32:29.316 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1781072784}) 2024-02-09T17:32:29.316 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2618589201}) 2024-02-09T17:32:29.316 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346713826}) 2024-02-09T17:32:29.316 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=62094551}) 2024-02-09T17:32:29.316 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=656166217}) 2024-02-09T17:32:29.316 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=41,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=21} crc {logm=3397500138,mdsmap=3984425268,mgr=1708625883,mgr_command_descs=708342470,mgr_metadata=670763949,mgrstat=1481882663,mon_config_key=2309916072}) 2024-02-09T17:32:29.316 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2652902535}) 2024-02-09T17:32:29.316 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2926131313}) 2024-02-09T17:32:29.316 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=60,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=22} crc {mon_config_key=140227100,monmap=423865542,osd_metadata=545325961,osd_pg_creating=633564157,osdmap=1072216855}) 2024-02-09T17:32:29.316 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=97} crc {osdmap=1923828369}) 2024-02-09T17:32:29.317 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:29 smithi099 ceph-mon[39420]: pgmap v717: 17 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.6 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 551 op/s; 1270/324057 objects degraded (0.392%); 29837/324057 objects misplaced (9.207%); 16 MiB/s, 262 objects/s recovering 2024-02-09T17:32:29.317 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:29 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-02-09T17:32:29.317 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:29 smithi099 ceph-mon[39420]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-02-09T17:32:29.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:29 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1323679347' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-02-09T17:32:29.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:29 smithi179 ceph-mon[43975]: osdmap e245: 12 total, 12 up, 11 in 2024-02-09T17:32:29.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:29 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:32:29.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:29 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:32:29.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=2629703955,config=2679774077,health=506382104,kv=2891707644}) 2024-02-09T17:32:29.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {kv=5,logm=95} crc {kv=1660393686,logm=1210453227}) 2024-02-09T17:32:29.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3451923082}) 2024-02-09T17:32:29.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3545785881}) 2024-02-09T17:32:29.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1472237423}) 2024-02-09T17:32:29.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3258113875}) 2024-02-09T17:32:29.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1209253103}) 2024-02-09T17:32:29.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3406121943}) 2024-02-09T17:32:29.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=618692712}) 2024-02-09T17:32:29.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=145037479}) 2024-02-09T17:32:29.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3386044008}) 2024-02-09T17:32:29.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2606405199}) 2024-02-09T17:32:29.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1785375383}) 2024-02-09T17:32:29.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2180019715}) 2024-02-09T17:32:29.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=608904271}) 2024-02-09T17:32:29.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=570814298}) 2024-02-09T17:32:29.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=705248428}) 2024-02-09T17:32:29.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3841946239}) 2024-02-09T17:32:29.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1372389079}) 2024-02-09T17:32:29.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1826978956}) 2024-02-09T17:32:29.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3945031885}) 2024-02-09T17:32:29.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=322221387}) 2024-02-09T17:32:29.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4128342744}) 2024-02-09T17:32:29.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2847936266}) 2024-02-09T17:32:29.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=592380767}) 2024-02-09T17:32:29.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1781072784}) 2024-02-09T17:32:29.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2618589201}) 2024-02-09T17:32:29.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346713826}) 2024-02-09T17:32:29.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=62094551}) 2024-02-09T17:32:29.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=656166217}) 2024-02-09T17:32:29.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=41,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=21} crc {logm=3397500138,mdsmap=3984425268,mgr=1708625883,mgr_command_descs=708342470,mgr_metadata=670763949,mgrstat=1481882663,mon_config_key=2309916072}) 2024-02-09T17:32:29.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2652902535}) 2024-02-09T17:32:29.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2926131313}) 2024-02-09T17:32:29.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=60,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=22} crc {mon_config_key=140227100,monmap=423865542,osd_metadata=545325961,osd_pg_creating=633564157,osdmap=1072216855}) 2024-02-09T17:32:29.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=97} crc {osdmap=1923828369}) 2024-02-09T17:32:29.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:29 smithi179 ceph-mon[43975]: pgmap v717: 17 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.6 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 551 op/s; 1270/324057 objects degraded (0.392%); 29837/324057 objects misplaced (9.207%); 16 MiB/s, 262 objects/s recovering 2024-02-09T17:32:29.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:29 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-02-09T17:32:29.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:29 smithi179 ceph-mon[43975]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-02-09T17:32:29.684 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:29 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1323679347' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-02-09T17:32:29.685 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:29 smithi162 ceph-mon[44304]: osdmap e245: 12 total, 12 up, 11 in 2024-02-09T17:32:29.685 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:29 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:32:29.685 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:29 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:32:29.685 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=2629703955,config=2679774077,health=506382104,kv=2891707644}) 2024-02-09T17:32:29.685 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {kv=5,logm=95} crc {kv=1660393686,logm=1210453227}) 2024-02-09T17:32:29.685 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3451923082}) 2024-02-09T17:32:29.685 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3545785881}) 2024-02-09T17:32:29.685 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1472237423}) 2024-02-09T17:32:29.685 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3258113875}) 2024-02-09T17:32:29.685 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1209253103}) 2024-02-09T17:32:29.685 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3406121943}) 2024-02-09T17:32:29.685 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=618692712}) 2024-02-09T17:32:29.685 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=145037479}) 2024-02-09T17:32:29.685 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3386044008}) 2024-02-09T17:32:29.686 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2606405199}) 2024-02-09T17:32:29.686 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1785375383}) 2024-02-09T17:32:29.686 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2180019715}) 2024-02-09T17:32:29.686 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=608904271}) 2024-02-09T17:32:29.686 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=570814298}) 2024-02-09T17:32:29.686 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=705248428}) 2024-02-09T17:32:29.686 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3841946239}) 2024-02-09T17:32:29.686 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1372389079}) 2024-02-09T17:32:29.686 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1826978956}) 2024-02-09T17:32:29.686 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3945031885}) 2024-02-09T17:32:29.686 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=322221387}) 2024-02-09T17:32:29.686 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4128342744}) 2024-02-09T17:32:29.686 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2847936266}) 2024-02-09T17:32:29.686 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=592380767}) 2024-02-09T17:32:29.686 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1781072784}) 2024-02-09T17:32:29.686 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2618589201}) 2024-02-09T17:32:29.687 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1346713826}) 2024-02-09T17:32:29.687 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=62094551}) 2024-02-09T17:32:29.687 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=656166217}) 2024-02-09T17:32:29.687 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=41,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=21} crc {logm=3397500138,mdsmap=3984425268,mgr=1708625883,mgr_command_descs=708342470,mgr_metadata=670763949,mgrstat=1481882663,mon_config_key=2309916072}) 2024-02-09T17:32:29.687 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2652902535}) 2024-02-09T17:32:29.687 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2926131313}) 2024-02-09T17:32:29.687 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=60,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=22} crc {mon_config_key=140227100,monmap=423865542,osd_metadata=545325961,osd_pg_creating=633564157,osdmap=1072216855}) 2024-02-09T17:32:29.687 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=97} crc {osdmap=1923828369}) 2024-02-09T17:32:29.687 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:29 smithi162 ceph-mon[44304]: pgmap v717: 17 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.6 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 551 op/s; 1270/324057 objects degraded (0.392%); 29837/324057 objects misplaced (9.207%); 16 MiB/s, 262 objects/s recovering 2024-02-09T17:32:29.687 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:29 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-02-09T17:32:29.687 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:29 smithi162 ceph-mon[44304]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-02-09T17:32:30.024 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:32:30.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:30.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:32:30.128 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:32:30.204 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:30.265 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:32:30.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:30.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:32:30.319 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:30 smithi099 ceph-mon[39420]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-02-09T17:32:30.319 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:30 smithi099 ceph-mon[39420]: osdmap e246: 12 total, 12 up, 11 in 2024-02-09T17:32:30.369 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:32:30.431 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:30.474 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:32:30.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:30.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:32:30.483 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:30 smithi162 ceph-mon[44304]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-02-09T17:32:30.483 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:30 smithi162 ceph-mon[44304]: osdmap e246: 12 total, 12 up, 11 in 2024-02-09T17:32:30.583 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:32:30.616 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:30.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:30 smithi179 ceph-mon[43975]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-02-09T17:32:30.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:30 smithi179 ceph-mon[43975]: osdmap e246: 12 total, 12 up, 11 in 2024-02-09T17:32:30.691 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:32:30.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:30.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:32:30.799 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:32:30.832 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:31.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:31 smithi099 ceph-mon[39420]: osdmap e247: 12 total, 12 up, 11 in 2024-02-09T17:32:31.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:31 smithi099 ceph-mon[39420]: pgmap v720: 27 pgs: 10 unknown, 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.5 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 740 op/s; 1536/318900 objects degraded (0.482%); 29078/318900 objects misplaced (9.118%); 18 MiB/s, 291 objects/s recovering 2024-02-09T17:32:31.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:31 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-09T17:32:31.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:31 smithi099 ceph-mon[39420]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-09T17:32:31.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:31 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 1270/324057 objects degraded (0.392%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:32:31.586 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:32:31.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:31.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:32:31.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:31 smithi179 ceph-mon[43975]: osdmap e247: 12 total, 12 up, 11 in 2024-02-09T17:32:31.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:31 smithi179 ceph-mon[43975]: pgmap v720: 27 pgs: 10 unknown, 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.5 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 740 op/s; 1536/318900 objects degraded (0.482%); 29078/318900 objects misplaced (9.118%); 18 MiB/s, 291 objects/s recovering 2024-02-09T17:32:31.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:31 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-09T17:32:31.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:31 smithi179 ceph-mon[43975]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-09T17:32:31.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:31 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 1270/324057 objects degraded (0.392%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:32:31.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:31 smithi162 ceph-mon[44304]: osdmap e247: 12 total, 12 up, 11 in 2024-02-09T17:32:31.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:31 smithi162 ceph-mon[44304]: pgmap v720: 27 pgs: 10 unknown, 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.5 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 740 op/s; 1536/318900 objects degraded (0.482%); 29078/318900 objects misplaced (9.118%); 18 MiB/s, 291 objects/s recovering 2024-02-09T17:32:31.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:31 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-09T17:32:31.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:31 smithi162 ceph-mon[44304]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-09T17:32:31.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:31 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 1270/324057 objects degraded (0.392%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:32:31.691 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:32:31.724 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:31.882 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:32:31.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:31.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:32:31.988 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:32:32.018 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:32.042 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:32:32.044 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:32.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:32:32.146 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:32:32.191 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:32.402 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:32 smithi099 ceph-mon[39420]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-02-09T17:32:32.402 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:32 smithi099 ceph-mon[39420]: osdmap e248: 12 total, 12 up, 11 in 2024-02-09T17:32:32.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:32 smithi179 ceph-mon[43975]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-02-09T17:32:32.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:32 smithi179 ceph-mon[43975]: osdmap e248: 12 total, 12 up, 11 in 2024-02-09T17:32:32.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:32 smithi162 ceph-mon[44304]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-02-09T17:32:32.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:32 smithi162 ceph-mon[44304]: osdmap e248: 12 total, 12 up, 11 in 2024-02-09T17:32:32.682 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:32:32.682 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [ 2024-02-09T17:32:32.683 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:32.683 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "MOSDPGPushReply(3.6 249/238 [PushReplyOp(3:6dcef751:::benchmark_data_smithi179_90440_object28561:head)])", 2024-02-09T17:32:32.683 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:32:32.674634+0000", 2024-02-09T17:32:32.683 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.0039076759999999997, 2024-02-09T17:32:32.683 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0039431429999999996, 2024-02-09T17:32:32.683 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:32:32.683 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "reached pg", 2024-02-09T17:32:32.683 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:32:32.683 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:32.683 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:32:32.684 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:32.674634+0000", 2024-02-09T17:32:32.684 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:32.684 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:32.684 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:32.684 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:32:32.684 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:32.674634+0000", 2024-02-09T17:32:32.684 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:32.684 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:32.684 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:32.684 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:32:32.685 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:32.674635+0000", 2024-02-09T17:32:32.685 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.16e-07 2024-02-09T17:32:32.685 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:32.685 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:32.685 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:32:32.685 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:32.674642+0000", 2024-02-09T17:32:32.685 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.3570000000000003e-06 2024-02-09T17:32:32.685 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:32.685 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:32.685 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:32:32.685 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:32.674653+0000", 2024-02-09T17:32:32.685 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1114000000000001e-05 2024-02-09T17:32:32.685 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:32.686 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:32.686 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:32:32.686 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:32.674661+0000", 2024-02-09T17:32:32.686 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.1580000000000005e-06 2024-02-09T17:32:32.686 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:32.686 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:32.686 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:32.686 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:32.674734+0000", 2024-02-09T17:32:32.686 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.3040000000000005e-05 2024-02-09T17:32:32.686 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:32.686 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:32:32.687 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:32.687 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:32.687 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:32.687 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.24817.0:160211 3.8 e249/199 3:1254028b:::benchmark_data_smithi179_90440_object28252:head v 249'9997, mlcod=249'9995)", 2024-02-09T17:32:32.687 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:32:32.664882+0000", 2024-02-09T17:32:32.687 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.013660349, 2024-02-09T17:32:32.687 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.013760847, 2024-02-09T17:32:32.687 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:32:32.687 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T17:32:32.687 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:32:32.687 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:32.687 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:32:32.688 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:32.664882+0000", 2024-02-09T17:32:32.688 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:32.688 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:32.688 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:32.688 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:32:32.688 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:32.664882+0000", 2024-02-09T17:32:32.688 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:32.688 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:32.688 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:32.688 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:32:32.688 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:32.664882+0000", 2024-02-09T17:32:32.689 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.85e-07 2024-02-09T17:32:32.689 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:32.689 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:32.689 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:32:32.689 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:32.664889+0000", 2024-02-09T17:32:32.689 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.8750000000000002e-06 2024-02-09T17:32:32.689 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:32.689 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:32.689 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:32:32.689 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:32.664898+0000", 2024-02-09T17:32:32.689 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.0049999999999993e-06 2024-02-09T17:32:32.690 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:32.690 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:32.690 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:32:32.690 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:32.664905+0000", 2024-02-09T17:32:32.690 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.9290000000000001e-06 2024-02-09T17:32:32.690 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:32.690 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:32.690 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:32.690 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:32.664955+0000", 2024-02-09T17:32:32.691 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.0021000000000001e-05 2024-02-09T17:32:32.691 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:32.691 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:32.691 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:32:32.691 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:32.664977+0000", 2024-02-09T17:32:32.691 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.1229999999999998e-05 2024-02-09T17:32:32.691 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:32.691 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:32:32.691 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:32.691 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:32.691 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:32.692 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.24817.0:160212 3.8 e249/199 3:1cfea7d7:::benchmark_data_smithi179_90440_object28253:head v 249'9998, mlcod=249'9995)", 2024-02-09T17:32:32.692 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:32:32.665534+0000", 2024-02-09T17:32:32.692 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.013007973000000001, 2024-02-09T17:32:32.692 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.013169086999999999, 2024-02-09T17:32:32.692 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:32:32.693 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T17:32:32.693 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:32:32.693 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:32.693 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:32:32.693 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:32.665534+0000", 2024-02-09T17:32:32.693 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:32.694 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:32.694 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:32.694 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:32:32.694 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:32.665534+0000", 2024-02-09T17:32:32.694 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:32.694 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:32.695 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:32.695 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:32:32.695 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:32.665534+0000", 2024-02-09T17:32:32.695 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9400000000000001e-07 2024-02-09T17:32:32.695 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:32.695 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:32.695 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:32:32.695 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:32.665538+0000", 2024-02-09T17:32:32.695 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.4750000000000002e-06 2024-02-09T17:32:32.696 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:32.696 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:32.696 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:32:32.696 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:32.665544+0000", 2024-02-09T17:32:32.696 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.3779999999999998e-06 2024-02-09T17:32:32.696 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:32.696 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:32.696 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:32:32.696 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:32.665549+0000", 2024-02-09T17:32:32.697 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.5859999999999998e-06 2024-02-09T17:32:32.697 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:32.697 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:32.697 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:32.697 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:32.669658+0000", 2024-02-09T17:32:32.697 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.004109494 2024-02-09T17:32:32.697 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:32.697 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:32.697 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:32:32.697 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:32.669677+0000", 2024-02-09T17:32:32.697 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.8804000000000002e-05 2024-02-09T17:32:32.697 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:32.698 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:32:32.698 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:32.698 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:32.698 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:32.698 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.24817.0:160215 3.8 e249/199 3:104e8a1d:::benchmark_data_smithi179_90440_object28256:head v 249'9999, mlcod=249'9995)", 2024-02-09T17:32:32.698 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:32:32.670821+0000", 2024-02-09T17:32:32.698 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.0077213150000000003, 2024-02-09T17:32:32.698 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0079402770000000008, 2024-02-09T17:32:32.698 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:32:32.698 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T17:32:32.698 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:32:32.698 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:32.698 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:32:32.698 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:32.670821+0000", 2024-02-09T17:32:32.699 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:32.699 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:32.699 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:32.699 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:32:32.699 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:32.670821+0000", 2024-02-09T17:32:32.699 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:32.699 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:32.699 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:32.699 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:32:32.699 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:32.670821+0000", 2024-02-09T17:32:32.699 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.3499999999999996e-07 2024-02-09T17:32:32.699 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:32.699 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:32.700 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:32:32.700 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:32.670827+0000", 2024-02-09T17:32:32.700 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.241e-06 2024-02-09T17:32:32.700 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:32.700 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:32.700 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:32:32.700 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:32.670837+0000", 2024-02-09T17:32:32.700 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.4010000000000003e-06 2024-02-09T17:32:32.700 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:32.700 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:32.700 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:32:32.701 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:32.670841+0000", 2024-02-09T17:32:32.701 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.5469999999999998e-06 2024-02-09T17:32:32.701 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:32.701 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:32.701 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:32.701 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:32.674338+0000", 2024-02-09T17:32:32.701 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0034968239999999999 2024-02-09T17:32:32.701 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:32.701 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:32.701 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:32:32.701 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:32.674364+0000", 2024-02-09T17:32:32.701 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.6067e-05 2024-02-09T17:32:32.701 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:32.702 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:32:32.702 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:32.702 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:32.702 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T17:32:32.702 INFO:teuthology.orchestra.run.smithi099.stdout: "num_ops": 4 2024-02-09T17:32:32.702 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:32:32.834 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:32:32.856 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:32:32.883 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:32:32.907 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:32:32.931 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:32:32.968 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:32:32.992 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:32:33.027 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:32:33.051 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:32:33.076 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:32:33.114 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:32:33.141 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:32:33.142 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:32:33.142 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:32:33.142 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:32:33.143 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:32:33.143 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:32:33.143 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:32:33.143 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:32:33.143 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:32:33.143 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:32:33.143 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:32:33.143 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:32:33.143 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:32:33.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:33 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:32:33.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:33 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:32:33.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:33 smithi099 ceph-mon[39420]: osdmap e249: 12 total, 12 up, 11 in 2024-02-09T17:32:33.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:33 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:32:33.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:33 smithi099 ceph-mon[39420]: pgmap v723: 27 pgs: 1 active+remapped+backfilling, 10 unknown, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.4 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 704 op/s; 1536/315204 objects degraded (0.487%); 28176/315204 objects misplaced (8.939%); 21 MiB/s, 333 objects/s recovering 2024-02-09T17:32:33.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:33 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3067433863' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:32:33.619 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-09T17:32:33.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:33 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:32:33.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:33 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:32:33.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:33 smithi179 ceph-mon[43975]: osdmap e249: 12 total, 12 up, 11 in 2024-02-09T17:32:33.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:33 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:32:33.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:33 smithi179 ceph-mon[43975]: pgmap v723: 27 pgs: 1 active+remapped+backfilling, 10 unknown, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.4 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 704 op/s; 1536/315204 objects degraded (0.487%); 28176/315204 objects misplaced (8.939%); 21 MiB/s, 333 objects/s recovering 2024-02-09T17:32:33.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:33 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3067433863' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:32:33.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:33 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:32:33.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:33 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:32:33.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:33 smithi162 ceph-mon[44304]: osdmap e249: 12 total, 12 up, 11 in 2024-02-09T17:32:33.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:33 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:32:33.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:33 smithi162 ceph-mon[44304]: pgmap v723: 27 pgs: 1 active+remapped+backfilling, 10 unknown, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.4 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 704 op/s; 1536/315204 objects degraded (0.487%); 28176/315204 objects misplaced (8.939%); 21 MiB/s, 333 objects/s recovering 2024-02-09T17:32:33.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:33 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3067433863' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:32:33.956 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:32:33.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:33.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:32:34.065 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:32:34.078 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] 2024-02-09T17:32:34.078 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:32:34.078 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-02-09T17:32:34.079 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph pg dump --format=json 2024-02-09T17:32:34.138 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:34.164 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:32:34.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:34.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:32:34.271 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:32:34.305 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:34.373 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is set 2024-02-09T17:32:34.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:34 smithi162 ceph-mon[44304]: osdmap e250: 12 total, 12 up, 11 in 2024-02-09T17:32:34.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:34 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:32:34.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:34 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:32:34.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:34 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/469344929' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:32:34.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:34 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:32:34.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:34 smithi179 ceph-mon[43975]: osdmap e250: 12 total, 12 up, 11 in 2024-02-09T17:32:34.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:34 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:32:34.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:34 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:32:34.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:34 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/469344929' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:32:34.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:34 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:32:34.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:34 smithi099 ceph-mon[39420]: osdmap e250: 12 total, 12 up, 11 in 2024-02-09T17:32:34.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:34 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:32:34.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:34 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:32:34.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:34 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/469344929' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:32:34.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:34 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:32:35.263 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:32:35.263 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:32:35.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:35.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:32:35.302 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:32:35.370 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:32:35.436 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:35.470 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:32:35.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:35.474 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:32:35.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:35 smithi099 ceph-mon[39420]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:32:35.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:35 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:32:35.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:35 smithi099 ceph-mon[39420]: osdmap e251: 12 total, 12 up, 11 in 2024-02-09T17:32:35.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:35 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:32:35.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:35 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:32:35.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:35 smithi099 ceph-mon[39420]: pgmap v726: 27 pgs: 1 active+remapped+backfilling, 10 unknown, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.4 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 372 op/s; 1536/314031 objects degraded (0.489%); 28176/314031 objects misplaced (8.972%); 13 MiB/s, 210 objects/s recovering 2024-02-09T17:32:35.575 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:32:35.607 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:35 smithi179 ceph-mon[43975]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:32:35.608 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:35 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:32:35.608 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:35 smithi179 ceph-mon[43975]: osdmap e251: 12 total, 12 up, 11 in 2024-02-09T17:32:35.608 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:35 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:32:35.608 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:35 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:32:35.608 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:35 smithi179 ceph-mon[43975]: pgmap v726: 27 pgs: 1 active+remapped+backfilling, 10 unknown, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.4 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 372 op/s; 1536/314031 objects degraded (0.489%); 28176/314031 objects misplaced (8.972%); 13 MiB/s, 210 objects/s recovering 2024-02-09T17:32:35.612 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:35.638 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:32:35.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:35.642 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:32:35.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:35 smithi162 ceph-mon[44304]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:32:35.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:35 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:32:35.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:35 smithi162 ceph-mon[44304]: osdmap e251: 12 total, 12 up, 11 in 2024-02-09T17:32:35.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:35 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:32:35.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:35 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:32:35.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:35 smithi162 ceph-mon[44304]: pgmap v726: 27 pgs: 1 active+remapped+backfilling, 10 unknown, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.4 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 372 op/s; 1536/314031 objects degraded (0.489%); 28176/314031 objects misplaced (8.972%); 13 MiB/s, 210 objects/s recovering 2024-02-09T17:32:35.743 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:32:35.793 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:35.825 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:32:35.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:35.829 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:32:35.930 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:32:35.975 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:36.006 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:32:36.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:36.011 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:32:36.112 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:32:36.169 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:36.198 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:32:36.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:36.204 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:32:36.306 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:32:36.365 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:36.382 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:36 smithi179 ceph-mon[43975]: osdmap e252: 12 total, 12 up, 11 in 2024-02-09T17:32:36.382 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:36 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:32:36.382 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:36 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:32:36.382 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:36 smithi179 ceph-mon[43975]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:32:36.382 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:36 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 1536/314031 objects degraded (0.489%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:32:36.400 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:32:36.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:36.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:32:36.511 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:32:36.543 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:32:36.560 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:36.597 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:32:36.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:36.604 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:32:36.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:36 smithi162 ceph-mon[44304]: osdmap e252: 12 total, 12 up, 11 in 2024-02-09T17:32:36.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:36 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:32:36.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:36 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:32:36.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:36 smithi162 ceph-mon[44304]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:32:36.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:36 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 1536/314031 objects degraded (0.489%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:32:36.705 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:32:36.741 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:36 smithi099 ceph-mon[39420]: osdmap e252: 12 total, 12 up, 11 in 2024-02-09T17:32:36.741 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:36 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:32:36.741 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:36 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:32:36.741 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:36 smithi099 ceph-mon[39420]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:32:36.741 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:36 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 1536/314031 objects degraded (0.489%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:32:36.741 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:37.093 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:32:37.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:37.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:32:37.198 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:32:37.226 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset noscrub 2024-02-09T17:32:37.262 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:37.292 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:32:37.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:37.304 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:32:37.405 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:32:37.453 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:37 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:32:37.453 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:37 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:32:37.453 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:37 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:32:37.453 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:37 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:32:37.453 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:37 smithi099 ceph-mon[39420]: osdmap e253: 12 total, 12 up, 11 in 2024-02-09T17:32:37.453 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:37 smithi099 ceph-mon[39420]: pgmap v729: 27 pgs: 2 peering, 1 active+remapped+backfilling, 6 unknown, 4 active+recovering+undersized+degraded+remapped, 14 active+clean; 6.3 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 10 KiB/s rd, 267 KiB/s wr, 509 op/s; 1749/309420 objects degraded (0.565%); 27461/309420 objects misplaced (8.875%); 8.0 MiB/s, 126 objects/s recovering 2024-02-09T17:32:37.462 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:37.504 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:32:37.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:37.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:32:37.620 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:32:37.660 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:37 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:32:37.660 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:37 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:32:37.660 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:37 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:32:37.660 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:37 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:32:37.660 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:37 smithi179 ceph-mon[43975]: osdmap e253: 12 total, 12 up, 11 in 2024-02-09T17:32:37.660 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:37 smithi179 ceph-mon[43975]: pgmap v729: 27 pgs: 2 peering, 1 active+remapped+backfilling, 6 unknown, 4 active+recovering+undersized+degraded+remapped, 14 active+clean; 6.3 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 10 KiB/s rd, 267 KiB/s wr, 509 op/s; 1749/309420 objects degraded (0.565%); 27461/309420 objects misplaced (8.875%); 8.0 MiB/s, 126 objects/s recovering 2024-02-09T17:32:37.662 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:37.689 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:32:37.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:37.697 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:32:37.799 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:32:37.827 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:37.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:37 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:32:37.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:37 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:32:37.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:37 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:32:37.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:37 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:32:37.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:37 smithi162 ceph-mon[44304]: osdmap e253: 12 total, 12 up, 11 in 2024-02-09T17:32:37.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:37 smithi162 ceph-mon[44304]: pgmap v729: 27 pgs: 2 peering, 1 active+remapped+backfilling, 6 unknown, 4 active+recovering+undersized+degraded+remapped, 14 active+clean; 6.3 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 10 KiB/s rd, 267 KiB/s wr, 509 op/s; 1749/309420 objects degraded (0.565%); 27461/309420 objects misplaced (8.875%); 8.0 MiB/s, 126 objects/s recovering 2024-02-09T17:32:38.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:38 smithi099 ceph-mon[39420]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-09T17:32:38.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:38 smithi179 ceph-mon[43975]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-09T17:32:38.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:38 smithi162 ceph-mon[44304]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-09T17:32:39.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:39 smithi099 ceph-mon[39420]: pgmap v730: 27 pgs: 2 peering, 1 active+remapped+backfilling, 1 unknown, 4 active+recovering+undersized+degraded+remapped, 19 active+clean; 6.2 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 13 KiB/s rd, 359 KiB/s wr, 614 op/s; 1749/306201 objects degraded (0.571%); 26715/306201 objects misplaced (8.725%); 15 MiB/s, 234 objects/s recovering 2024-02-09T17:32:39.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:39 smithi179 ceph-mon[43975]: pgmap v730: 27 pgs: 2 peering, 1 active+remapped+backfilling, 1 unknown, 4 active+recovering+undersized+degraded+remapped, 19 active+clean; 6.2 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 13 KiB/s rd, 359 KiB/s wr, 614 op/s; 1749/306201 objects degraded (0.571%); 26715/306201 objects misplaced (8.725%); 15 MiB/s, 234 objects/s recovering 2024-02-09T17:32:39.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:39 smithi162 ceph-mon[44304]: pgmap v730: 27 pgs: 2 peering, 1 active+remapped+backfilling, 1 unknown, 4 active+recovering+undersized+degraded+remapped, 19 active+clean; 6.2 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 13 KiB/s rd, 359 KiB/s wr, 614 op/s; 1749/306201 objects degraded (0.571%); 26715/306201 objects misplaced (8.725%); 15 MiB/s, 234 objects/s recovering 2024-02-09T17:32:40.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:40 smithi179 ceph-mon[43975]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-09T17:32:40.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:40 smithi162 ceph-mon[44304]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-09T17:32:40.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:40 smithi099 ceph-mon[39420]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-09T17:32:41.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:41 smithi179 ceph-mon[43975]: pgmap v731: 27 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 22 active+clean; 6.1 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 13 KiB/s rd, 345 KiB/s wr, 720 op/s; 1980/301317 objects degraded (0.657%); 25960/301317 objects misplaced (8.616%); 18 MiB/s, 281 objects/s recovering 2024-02-09T17:32:41.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:41 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 1980/301317 objects degraded (0.657%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:32:41.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:41 smithi179 ceph-mon[43975]: osdmap e254: 12 total, 12 up, 11 in 2024-02-09T17:32:41.936 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:41 smithi162 ceph-mon[44304]: pgmap v731: 27 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 22 active+clean; 6.1 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 13 KiB/s rd, 345 KiB/s wr, 720 op/s; 1980/301317 objects degraded (0.657%); 25960/301317 objects misplaced (8.616%); 18 MiB/s, 281 objects/s recovering 2024-02-09T17:32:41.936 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:41 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 1980/301317 objects degraded (0.657%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:32:41.936 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:41 smithi162 ceph-mon[44304]: osdmap e254: 12 total, 12 up, 11 in 2024-02-09T17:32:41.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:41 smithi099 ceph-mon[39420]: pgmap v731: 27 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 22 active+clean; 6.1 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 13 KiB/s rd, 345 KiB/s wr, 720 op/s; 1980/301317 objects degraded (0.657%); 25960/301317 objects misplaced (8.616%); 18 MiB/s, 281 objects/s recovering 2024-02-09T17:32:41.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:41 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 1980/301317 objects degraded (0.657%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:32:41.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:41 smithi099 ceph-mon[39420]: osdmap e254: 12 total, 12 up, 11 in 2024-02-09T17:32:43.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:43 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:32:43.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:43 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:32:43.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:43 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:32:43.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:43 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:32:43.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:43 smithi179 ceph-mon[43975]: osdmap e255: 12 total, 12 up, 11 in 2024-02-09T17:32:43.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:43 smithi179 ceph-mon[43975]: pgmap v734: 27 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 22 active+clean; 6.1 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 6.5 KiB/s rd, 175 KiB/s wr, 675 op/s; 1980/297378 objects degraded (0.666%); 25008/297378 objects misplaced (8.409%); 22 MiB/s, 357 objects/s recovering 2024-02-09T17:32:43.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:43 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:32:43.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:43 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:32:43.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:43 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:32:43.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:43 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:32:43.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:43 smithi162 ceph-mon[44304]: osdmap e255: 12 total, 12 up, 11 in 2024-02-09T17:32:43.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:43 smithi162 ceph-mon[44304]: pgmap v734: 27 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 22 active+clean; 6.1 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 6.5 KiB/s rd, 175 KiB/s wr, 675 op/s; 1980/297378 objects degraded (0.666%); 25008/297378 objects misplaced (8.409%); 22 MiB/s, 357 objects/s recovering 2024-02-09T17:32:43.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:43 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:32:43.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:43 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:32:43.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:43 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:32:43.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:43 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:32:43.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:43 smithi099 ceph-mon[39420]: osdmap e255: 12 total, 12 up, 11 in 2024-02-09T17:32:43.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:43 smithi099 ceph-mon[39420]: pgmap v734: 27 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 22 active+clean; 6.1 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 6.5 KiB/s rd, 175 KiB/s wr, 675 op/s; 1980/297378 objects degraded (0.666%); 25008/297378 objects misplaced (8.409%); 22 MiB/s, 357 objects/s recovering 2024-02-09T17:32:43.816 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:32:43.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:43.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:32:43.923 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:32:43.938 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:32:43.987 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:32:43.994 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:44.024 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:32:44.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:44.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:32:44.138 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:32:44.167 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:44.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:44 smithi099 ceph-mon[39420]: osdmap e256: 12 total, 12 up, 11 in 2024-02-09T17:32:44.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:44 smithi179 ceph-mon[43975]: osdmap e256: 12 total, 12 up, 11 in 2024-02-09T17:32:44.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:44 smithi162 ceph-mon[44304]: osdmap e256: 12 total, 12 up, 11 in 2024-02-09T17:32:45.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:45 smithi179 ceph-mon[43975]: pgmap v736: 27 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 22 active+clean; 6.0 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 558 op/s; 1980/296235 objects degraded (0.668%); 25008/296235 objects misplaced (8.442%); 15 MiB/s, 244 objects/s recovering 2024-02-09T17:32:45.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:45 smithi162 ceph-mon[44304]: pgmap v736: 27 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 22 active+clean; 6.0 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 558 op/s; 1980/296235 objects degraded (0.668%); 25008/296235 objects misplaced (8.442%); 15 MiB/s, 244 objects/s recovering 2024-02-09T17:32:45.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:45 smithi099 ceph-mon[39420]: pgmap v736: 27 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 22 active+clean; 6.0 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 558 op/s; 1980/296235 objects degraded (0.668%); 25008/296235 objects misplaced (8.442%); 15 MiB/s, 244 objects/s recovering 2024-02-09T17:32:46.430 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:32:46.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:46.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:32:46.531 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:46 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 1980/296235 objects degraded (0.668%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:32:46.531 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:46 smithi099 ceph-mon[39420]: osdmap e257: 12 total, 12 up, 11 in 2024-02-09T17:32:46.535 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:32:46.572 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:46 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 1980/296235 objects degraded (0.668%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:32:46.572 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:46 smithi162 ceph-mon[44304]: osdmap e257: 12 total, 12 up, 11 in 2024-02-09T17:32:46.574 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:46.600 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:32:46.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:46.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:32:46.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:46 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 1980/296235 objects degraded (0.668%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:32:46.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:46 smithi179 ceph-mon[43975]: osdmap e257: 12 total, 12 up, 11 in 2024-02-09T17:32:46.705 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:32:46.738 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:46.813 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:32:46.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:46.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:32:46.917 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:32:46.980 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:47.022 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:32:47.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:47.032 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:32:47.133 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:32:47.183 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:47.214 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:32:47.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:47.225 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:32:47.326 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:32:47.366 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:47.393 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:32:47.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:47.398 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:32:47.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:47 smithi099 ceph-mon[39420]: pgmap v738: 27 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 22 active+clean; 5.9 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 533 op/s; 2091/292263 objects degraded (0.715%); 24297/292263 objects misplaced (8.313%); 18 MiB/s, 290 objects/s recovering 2024-02-09T17:32:47.498 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:32:47.586 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:47 smithi162 ceph-mon[44304]: pgmap v738: 27 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 22 active+clean; 5.9 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 533 op/s; 2091/292263 objects degraded (0.715%); 24297/292263 objects misplaced (8.313%); 18 MiB/s, 290 objects/s recovering 2024-02-09T17:32:47.590 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:47.626 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:32:47.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:47.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:32:47.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:47 smithi179 ceph-mon[43975]: pgmap v738: 27 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded+remapped, 22 active+clean; 5.9 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 533 op/s; 2091/292263 objects degraded (0.715%); 24297/292263 objects misplaced (8.313%); 18 MiB/s, 290 objects/s recovering 2024-02-09T17:32:47.732 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:32:47.790 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:47.821 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:32:47.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:47.834 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:32:47.839 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:32:47.840 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-02-09T17:32:47.935 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:32:47.969 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:48.000 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:32:48.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:48.006 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:32:48.107 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:32:48.144 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:48.175 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:32:48.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:48.179 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:32:48.197 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:32:48.198 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:32:48.280 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:32:48.318 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:48.419 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:32:48.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:48 smithi179 ceph-mon[43975]: from='client.15261 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:32:48.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:48.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:32:48.524 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:32:48.556 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:48.591 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:32:48.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:48.599 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:32:48.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:48 smithi162 ceph-mon[44304]: from='client.15261 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:32:48.700 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:32:48.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:48 smithi099 ceph-mon[39420]: from='client.15261 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:32:48.743 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:48.769 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:32:48.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:48.776 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:32:48.877 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:32:48.905 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:48.949 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:32:48.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:48.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:32:48.984 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":738,"stamp":"2024-02-09T17:32:46.555827+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6384976440,"num_objects":97421,"num_object_clones":0,"num_object_copies":292263,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2091,"num_objects_misplaced":24297,"num_objects_unfound":0,"num_objects_dirty":97421,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":166676,"num_write_kb":8448123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11151,"num_bytes_recovered":731011375,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162445,"ondisk_log_size":162445,"up":81,"acting":77,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":81,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":18967964,"kb_used_data":17169756,"kb_used_omap":228,"kb_used_meta":1797851,"kb_avail":1012228708,"statfs":{"total":1055945392128,"available":1036522196992,"internally_reserved":0,"allocated":17581830144,"data_stored":17578340666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":233532,"internal_metadata":1841000388},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-185663488,"num_objects":-2833,"num_object_clones":0,"num_object_copies":-8499,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":111,"num_objects_misplaced":-1663,"num_objects_unfound":0,"num_objects_dirty":-2833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2833,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1542,"num_bytes_recovered":100663296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.311270"},"pg_stats":[{"pgid":"3.f","version":"254'10302","reported_seq":10508,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-02-09T17:32:41.618437+0000","last_change":"2024-02-09T17:31:25.731053+0000","last_active":"2024-02-09T17:32:41.618437+0000","last_peered":"2024-02-09T17:32:41.618437+0000","last_clean":"2024-02-09T17:32:41.618437+0000","last_became_active":"2024-02-09T17:29:47.582245+0000","last_became_peered":"2024-02-09T17:29:47.582245+0000","last_unstale":"2024-02-09T17:32:41.618437+0000","last_undegraded":"2024-02-09T17:32:41.618437+0000","last_fullsized":"2024-02-09T17:32:41.618437+0000","mapping_epoch":199,"log_start":"208'300","ondisk_log_start":"208'300","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'7772","last_scrub_stamp":"2024-02-09T17:31:25.730848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:31:25.730848+0000","objects_scrubbed":7721,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2441,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402784256,"num_objects":6146,"num_object_clones":0,"num_object_copies":18438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10302,"num_write_kb":526336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"256'10470","reported_seq":12756,"reported_epoch":256,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-02-09T17:32:44.658405+0000","last_change":"2024-02-09T17:32:01.597065+0000","last_active":"2024-02-09T17:32:44.658405+0000","last_peered":"2024-02-09T17:32:44.658405+0000","last_clean":"2024-02-09T17:31:59.261441+0000","last_became_active":"2024-02-09T17:32:01.541864+0000","last_became_peered":"2024-02-09T17:32:01.541864+0000","last_unstale":"2024-02-09T17:32:44.658405+0000","last_undegraded":"2024-02-09T17:32:01.324655+0000","last_fullsized":"2024-02-09T17:32:01.278077+0000","mapping_epoch":238,"log_start":"208'400","ondisk_log_start":"208'400","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'7702","last_scrub_stamp":"2024-02-09T17:31:22.871435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:31:22.871435+0000","objects_scrubbed":7650,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1911,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402915328,"num_objects":6148,"num_object_clones":0,"num_object_copies":18444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":595,"num_objects_misplaced":6114,"num_objects_unfound":0,"num_objects_dirty":6148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10470,"num_write_kb":531776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1038,"num_bytes_recovered":67502080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9],"acting":[11,4],"avail_no_missing":["11","0","4"],"object_location_counts":[{"shards":"0,4,11","objects":6114},{"shards":"4,11","objects":-561}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"254'10397","reported_seq":10604,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-02-09T17:32:41.699030+0000","last_change":"2024-02-09T17:31:24.785058+0000","last_active":"2024-02-09T17:32:41.699030+0000","last_peered":"2024-02-09T17:32:41.699030+0000","last_clean":"2024-02-09T17:32:41.699030+0000","last_became_active":"2024-02-09T17:29:47.583238+0000","last_became_peered":"2024-02-09T17:29:47.583238+0000","last_unstale":"2024-02-09T17:32:41.699030+0000","last_undegraded":"2024-02-09T17:32:41.699030+0000","last_fullsized":"2024-02-09T17:32:41.699030+0000","mapping_epoch":199,"log_start":"208'300","ondisk_log_start":"208'300","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'7776","last_scrub_stamp":"2024-02-09T17:31:24.784873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:31:24.784873+0000","objects_scrubbed":7733,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:32:51.351383+0000","scrub_duration":2469,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":412221464,"num_objects":6291,"num_object_clones":0,"num_object_copies":18873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6291,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10397,"num_write_kb":533953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,9],"acting":[7,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"256'10368","reported_seq":10577,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-02-09T17:32:44.209288+0000","last_change":"2024-02-09T17:31:24.234560+0000","last_active":"2024-02-09T17:32:44.209288+0000","last_peered":"2024-02-09T17:32:44.209288+0000","last_clean":"2024-02-09T17:32:44.209288+0000","last_became_active":"2024-02-09T17:29:47.583200+0000","last_became_peered":"2024-02-09T17:29:47.583200+0000","last_unstale":"2024-02-09T17:32:44.209288+0000","last_undegraded":"2024-02-09T17:32:44.209288+0000","last_fullsized":"2024-02-09T17:32:44.209288+0000","mapping_epoch":199,"log_start":"208'300","ondisk_log_start":"208'300","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'7647","last_scrub_stamp":"2024-02-09T17:31:24.234420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:31:24.234420+0000","objects_scrubbed":7605,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:32:49.079885+0000","scrub_duration":2190,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395706368,"num_objects":6038,"num_object_clones":0,"num_object_copies":18114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10368,"num_write_kb":524992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"63'139","reported_seq":746,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-02-09T17:32:41.265379+0000","last_change":"2024-02-09T17:32:35.402431+0000","last_active":"2024-02-09T17:32:41.265379+0000","last_peered":"2024-02-09T17:32:41.265379+0000","last_clean":"2024-02-09T17:32:41.265379+0000","last_became_active":"2024-02-09T17:32:35.402025+0000","last_became_peered":"2024-02-09T17:32:35.402025+0000","last_unstale":"2024-02-09T17:32:41.265379+0000","last_undegraded":"2024-02-09T17:32:41.265379+0000","last_fullsized":"2024-02-09T17:32:41.265379+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:31:59.409353+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:31:59.409353+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:32:46.176335+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":16,"num_write_kb":267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":240422,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"254'13597","reported_seq":27431,"reported_epoch":254,"state":"active+remapped+backfilling","last_fresh":"2024-02-09T17:32:41.677674+0000","last_change":"2024-02-09T17:32:01.600444+0000","last_active":"2024-02-09T17:32:41.677674+0000","last_peered":"2024-02-09T17:32:41.677674+0000","last_clean":"2024-02-09T17:31:59.257931+0000","last_became_active":"2024-02-09T17:32:01.577152+0000","last_became_peered":"2024-02-09T17:32:01.577152+0000","last_unstale":"2024-02-09T17:32:41.677674+0000","last_undegraded":"2024-02-09T17:32:41.677674+0000","last_fullsized":"2024-02-09T17:32:41.677674+0000","mapping_epoch":238,"log_start":"211'3500","ondisk_log_start":"211'3500","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:29:46.563947+0000","objects_scrubbed":2799,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":401342464,"num_objects":6124,"num_object_clones":0,"num_object_copies":18372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":81,"num_objects_unfound":0,"num_objects_dirty":6124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10276,"num_write_kb":524800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6756,"num_bytes_recovered":442761216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,11],"acting":[3,4,0],"avail_no_missing":["3","0","4"],"object_location_counts":[{"shards":"0,3,4","objects":6124}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"63'139","reported_seq":266,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-02-09T17:32:43.307065+0000","last_change":"2024-02-09T17:32:35.384968+0000","last_active":"2024-02-09T17:32:43.307065+0000","last_peered":"2024-02-09T17:32:43.307065+0000","last_clean":"2024-02-09T17:32:43.307065+0000","last_became_active":"2024-02-09T17:32:35.384617+0000","last_became_peered":"2024-02-09T17:32:35.384617+0000","last_unstale":"2024-02-09T17:32:43.307065+0000","last_undegraded":"2024-02-09T17:32:43.307065+0000","last_fullsized":"2024-02-09T17:32:43.307065+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:31:59.409353+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:31:59.409353+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:32:47.084710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":16,"num_write_kb":267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":144253,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"256'10480","reported_seq":10690,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-02-09T17:32:45.541351+0000","last_change":"2024-02-09T17:31:25.346078+0000","last_active":"2024-02-09T17:32:45.541351+0000","last_peered":"2024-02-09T17:32:45.541351+0000","last_clean":"2024-02-09T17:32:45.541351+0000","last_became_active":"2024-02-09T17:29:47.581573+0000","last_became_peered":"2024-02-09T17:29:47.581573+0000","last_unstale":"2024-02-09T17:32:45.541351+0000","last_undegraded":"2024-02-09T17:32:45.541351+0000","last_fullsized":"2024-02-09T17:32:45.541351+0000","mapping_epoch":199,"log_start":"208'400","ondisk_log_start":"208'400","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'7710","last_scrub_stamp":"2024-02-09T17:31:25.345946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:31:25.345946+0000","objects_scrubbed":7649,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2400,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":392429568,"num_objects":5988,"num_object_clones":0,"num_object_copies":17964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10480,"num_write_kb":526976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"254'10419","reported_seq":10647,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-02-09T17:32:41.787994+0000","last_change":"2024-02-09T17:31:20.743404+0000","last_active":"2024-02-09T17:32:41.787994+0000","last_peered":"2024-02-09T17:32:41.787994+0000","last_clean":"2024-02-09T17:32:41.787994+0000","last_became_active":"2024-02-09T17:29:47.582488+0000","last_became_peered":"2024-02-09T17:29:47.582488+0000","last_unstale":"2024-02-09T17:32:41.787994+0000","last_undegraded":"2024-02-09T17:32:41.787994+0000","last_fullsized":"2024-02-09T17:32:41.787994+0000","mapping_epoch":199,"log_start":"208'400","ondisk_log_start":"208'400","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'7551","last_scrub_stamp":"2024-02-09T17:31:20.743150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:31:20.743150+0000","objects_scrubbed":7492,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:32:45.305620+0000","scrub_duration":1578,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406257664,"num_objects":6199,"num_object_clones":0,"num_object_copies":18597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10419,"num_write_kb":531776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.a","version":"63'139","reported_seq":266,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-02-09T17:32:43.287652+0000","last_change":"2024-02-09T17:32:35.418523+0000","last_active":"2024-02-09T17:32:43.287652+0000","last_peered":"2024-02-09T17:32:43.287652+0000","last_clean":"2024-02-09T17:32:43.287652+0000","last_became_active":"2024-02-09T17:32:35.418200+0000","last_became_peered":"2024-02-09T17:32:35.418200+0000","last_unstale":"2024-02-09T17:32:43.287652+0000","last_undegraded":"2024-02-09T17:32:43.287652+0000","last_fullsized":"2024-02-09T17:32:43.287652+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:31:59.409353+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:31:59.409353+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:32:46.301279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":16,"num_write_kb":267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":144253,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.8","version":"256'10401","reported_seq":10610,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-02-09T17:32:44.489682+0000","last_change":"2024-02-09T17:31:21.687596+0000","last_active":"2024-02-09T17:32:44.489682+0000","last_peered":"2024-02-09T17:32:44.489682+0000","last_clean":"2024-02-09T17:32:44.489682+0000","last_became_active":"2024-02-09T17:29:47.581467+0000","last_became_peered":"2024-02-09T17:29:47.581467+0000","last_unstale":"2024-02-09T17:32:44.489682+0000","last_undegraded":"2024-02-09T17:32:44.489682+0000","last_fullsized":"2024-02-09T17:32:44.489682+0000","mapping_epoch":199,"log_start":"208'400","ondisk_log_start":"208'400","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'7591","last_scrub_stamp":"2024-02-09T17:31:21.687400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:31:21.687400+0000","objects_scrubbed":7543,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1706,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402587648,"num_objects":6143,"num_object_clones":0,"num_object_copies":18429,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10401,"num_write_kb":529408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.5","version":"63'139","reported_seq":264,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-02-09T17:32:42.283050+0000","last_change":"2024-02-09T17:32:35.399428+0000","last_active":"2024-02-09T17:32:42.283050+0000","last_peered":"2024-02-09T17:32:42.283050+0000","last_clean":"2024-02-09T17:32:42.283050+0000","last_became_active":"2024-02-09T17:32:35.399106+0000","last_became_peered":"2024-02-09T17:32:35.399106+0000","last_unstale":"2024-02-09T17:32:42.283050+0000","last_undegraded":"2024-02-09T17:32:42.283050+0000","last_fullsized":"2024-02-09T17:32:42.283050+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:31:59.409353+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:31:59.409353+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:32:45.692862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":17,"num_write_kb":267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":144253,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.7","version":"254'10268","reported_seq":10504,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-02-09T17:32:41.574062+0000","last_change":"2024-02-09T17:31:29.003929+0000","last_active":"2024-02-09T17:32:41.574062+0000","last_peered":"2024-02-09T17:32:41.574062+0000","last_clean":"2024-02-09T17:32:41.574062+0000","last_became_active":"2024-02-09T17:29:47.583311+0000","last_became_peered":"2024-02-09T17:29:47.583311+0000","last_unstale":"2024-02-09T17:32:41.574062+0000","last_undegraded":"2024-02-09T17:32:41.574062+0000","last_fullsized":"2024-02-09T17:32:41.574062+0000","mapping_epoch":199,"log_start":"208'200","ondisk_log_start":"208'200","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'7925","last_scrub_stamp":"2024-02-09T17:31:29.003795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:31:29.003795+0000","objects_scrubbed":7867,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1984,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402391040,"num_objects":6140,"num_object_clones":0,"num_object_copies":18420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10268,"num_write_kb":525056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"63'139","reported_seq":358,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-02-09T17:32:43.267984+0000","last_change":"2024-02-09T17:32:35.394597+0000","last_active":"2024-02-09T17:32:43.267984+0000","last_peered":"2024-02-09T17:32:43.267984+0000","last_clean":"2024-02-09T17:32:43.267984+0000","last_became_active":"2024-02-09T17:32:35.394302+0000","last_became_peered":"2024-02-09T17:32:35.394302+0000","last_unstale":"2024-02-09T17:32:43.267984+0000","last_undegraded":"2024-02-09T17:32:43.267984+0000","last_fullsized":"2024-02-09T17:32:43.267984+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:31:59.409353+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:31:59.409353+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:32:46.773196+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":17,"num_write_kb":267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":520018,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.6","version":"256'10279","reported_seq":13379,"reported_epoch":256,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-02-09T17:32:45.065072+0000","last_change":"2024-02-09T17:32:01.583215+0000","last_active":"2024-02-09T17:32:45.065072+0000","last_peered":"2024-02-09T17:32:45.065072+0000","last_clean":"2024-02-09T17:31:59.217197+0000","last_became_active":"2024-02-09T17:32:01.532489+0000","last_became_peered":"2024-02-09T17:32:01.532489+0000","last_unstale":"2024-02-09T17:32:45.065072+0000","last_undegraded":"2024-02-09T17:32:01.315621+0000","last_fullsized":"2024-02-09T17:32:01.278340+0000","mapping_epoch":238,"log_start":"208'200","ondisk_log_start":"208'200","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'7634","last_scrub_stamp":"2024-02-09T17:31:25.792160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:31:25.792160+0000","objects_scrubbed":7589,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1716,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":391577600,"num_objects":5975,"num_object_clones":0,"num_object_copies":17925,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":154,"num_objects_misplaced":5962,"num_objects_unfound":0,"num_objects_dirty":5975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10279,"num_write_kb":520128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":94765056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4],"avail_no_missing":["1","0","4"],"object_location_counts":[{"shards":"0,1,4","objects":5962},{"shards":"1,4","objects":-141}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"63'139","reported_seq":272,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-02-09T17:32:43.290237+0000","last_change":"2024-02-09T17:32:35.396449+0000","last_active":"2024-02-09T17:32:43.290237+0000","last_peered":"2024-02-09T17:32:43.290237+0000","last_clean":"2024-02-09T17:32:43.290237+0000","last_became_active":"2024-02-09T17:32:35.396115+0000","last_became_peered":"2024-02-09T17:32:35.396115+0000","last_unstale":"2024-02-09T17:32:43.290237+0000","last_undegraded":"2024-02-09T17:32:43.290237+0000","last_fullsized":"2024-02-09T17:32:43.290237+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:31:59.409353+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:31:59.409353+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:32:45.850323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":16,"num_write_kb":267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":144253,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"256'10395","reported_seq":10603,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-02-09T17:32:44.198205+0000","last_change":"2024-02-09T17:31:23.024422+0000","last_active":"2024-02-09T17:32:44.198205+0000","last_peered":"2024-02-09T17:32:44.198205+0000","last_clean":"2024-02-09T17:32:44.198205+0000","last_became_active":"2024-02-09T17:29:47.581119+0000","last_became_peered":"2024-02-09T17:29:47.581119+0000","last_unstale":"2024-02-09T17:32:44.198205+0000","last_undegraded":"2024-02-09T17:32:44.198205+0000","last_fullsized":"2024-02-09T17:32:44.198205+0000","mapping_epoch":199,"log_start":"208'300","ondisk_log_start":"208'300","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'7587","last_scrub_stamp":"2024-02-09T17:31:23.024268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:31:23.024268+0000","objects_scrubbed":7544,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:32:45.021970+0000","scrub_duration":1988,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":391708672,"num_objects":5977,"num_object_clones":0,"num_object_copies":17931,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10395,"num_write_kb":523904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"63'139","reported_seq":264,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-02-09T17:32:42.283059+0000","last_change":"2024-02-09T17:32:35.395074+0000","last_active":"2024-02-09T17:32:42.283059+0000","last_peered":"2024-02-09T17:32:42.283059+0000","last_clean":"2024-02-09T17:32:42.283059+0000","last_became_active":"2024-02-09T17:32:35.393886+0000","last_became_peered":"2024-02-09T17:32:35.393886+0000","last_unstale":"2024-02-09T17:32:42.283059+0000","last_undegraded":"2024-02-09T17:32:42.283059+0000","last_fullsized":"2024-02-09T17:32:42.283059+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:31:59.409353+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:31:59.409353+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:32:45.692842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":16,"num_write_kb":267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":144253,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.4","version":"256'10436","reported_seq":10706,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-02-09T17:32:44.984747+0000","last_change":"2024-02-09T17:31:59.706801+0000","last_active":"2024-02-09T17:32:44.984747+0000","last_peered":"2024-02-09T17:32:44.984747+0000","last_clean":"2024-02-09T17:32:44.984747+0000","last_became_active":"2024-02-09T17:29:47.582922+0000","last_became_peered":"2024-02-09T17:29:47.582922+0000","last_unstale":"2024-02-09T17:32:44.984747+0000","last_undegraded":"2024-02-09T17:32:44.984747+0000","last_fullsized":"2024-02-09T17:32:44.984747+0000","mapping_epoch":199,"log_start":"208'400","ondisk_log_start":"208'400","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'8794","last_scrub_stamp":"2024-02-09T17:31:59.706688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:31:59.706688+0000","objects_scrubbed":7672,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:33:10.369313+0000","scrub_duration":1299,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394526720,"num_objects":6020,"num_object_clones":0,"num_object_copies":18060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10436,"num_write_kb":526592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"63'139","reported_seq":264,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-02-09T17:32:42.283024+0000","last_change":"2024-02-09T17:32:35.391938+0000","last_active":"2024-02-09T17:32:42.283024+0000","last_peered":"2024-02-09T17:32:42.283024+0000","last_clean":"2024-02-09T17:32:42.283024+0000","last_became_active":"2024-02-09T17:32:35.391663+0000","last_became_peered":"2024-02-09T17:32:35.391663+0000","last_unstale":"2024-02-09T17:32:42.283024+0000","last_undegraded":"2024-02-09T17:32:42.283024+0000","last_fullsized":"2024-02-09T17:32:42.283024+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:31:59.409353+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:31:59.409353+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:32:45.692818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":17,"num_write_kb":268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":144254,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,2],"acting":[10,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3","version":"256'10589","reported_seq":12544,"reported_epoch":256,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-02-09T17:32:44.701297+0000","last_change":"2024-02-09T17:32:01.643212+0000","last_active":"2024-02-09T17:32:44.701297+0000","last_peered":"2024-02-09T17:32:44.701297+0000","last_clean":"2024-02-09T17:31:59.234884+0000","last_became_active":"2024-02-09T17:32:01.587380+0000","last_became_peered":"2024-02-09T17:32:01.587380+0000","last_unstale":"2024-02-09T17:32:44.701297+0000","last_undegraded":"2024-02-09T17:32:01.320756+0000","last_fullsized":"2024-02-09T17:32:01.276816+0000","mapping_epoch":238,"log_start":"208'500","ondisk_log_start":"208'500","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'7835","last_scrub_stamp":"2024-02-09T17:31:26.001797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:31:26.001797+0000","objects_scrubbed":7767,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2765,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402587648,"num_objects":6143,"num_object_clones":0,"num_object_copies":18429,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":744,"num_objects_misplaced":6142,"num_objects_unfound":0,"num_objects_dirty":6143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10589,"num_write_kb":535424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":870,"num_bytes_recovered":56557568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,11],"acting":[4,6],"avail_no_missing":["4","0","6"],"object_location_counts":[{"shards":"0,4,6","objects":6142},{"shards":"4,6","objects":-743}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"63'139","reported_seq":265,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-02-09T17:32:43.270302+0000","last_change":"2024-02-09T17:32:35.424894+0000","last_active":"2024-02-09T17:32:43.270302+0000","last_peered":"2024-02-09T17:32:43.270302+0000","last_clean":"2024-02-09T17:32:43.270302+0000","last_became_active":"2024-02-09T17:32:35.424475+0000","last_became_peered":"2024-02-09T17:32:35.424475+0000","last_unstale":"2024-02-09T17:32:43.270302+0000","last_undegraded":"2024-02-09T17:32:43.270302+0000","last_fullsized":"2024-02-09T17:32:43.270302+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:31:59.409353+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:31:59.409353+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:32:46.688981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":17,"num_write_kb":267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":144253,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"256'10413","reported_seq":12675,"reported_epoch":256,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-02-09T17:32:44.705900+0000","last_change":"2024-02-09T17:32:01.551242+0000","last_active":"2024-02-09T17:32:44.705900+0000","last_peered":"2024-02-09T17:32:44.705900+0000","last_clean":"2024-02-09T17:31:59.232029+0000","last_became_active":"2024-02-09T17:32:01.494982+0000","last_became_peered":"2024-02-09T17:32:01.494982+0000","last_unstale":"2024-02-09T17:32:44.705900+0000","last_undegraded":"2024-02-09T17:32:01.319667+0000","last_fullsized":"2024-02-09T17:32:01.276864+0000","mapping_epoch":238,"log_start":"208'400","ondisk_log_start":"208'400","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"220'6050","last_scrub_stamp":"2024-02-09T17:31:00.396342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:31:00.396342+0000","objects_scrubbed":6003,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1427,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394199040,"num_objects":6015,"num_object_clones":0,"num_object_copies":18045,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":598,"num_objects_misplaced":5998,"num_objects_unfound":0,"num_objects_dirty":6015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10413,"num_write_kb":525696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1033,"num_bytes_recovered":67174400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,11],"acting":[4,10],"avail_no_missing":["4","0","10"],"object_location_counts":[{"shards":"0,4,10","objects":5998},{"shards":"4,10","objects":-581}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"63'139","reported_seq":746,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-02-09T17:32:41.265809+0000","last_change":"2024-02-09T17:32:35.407595+0000","last_active":"2024-02-09T17:32:41.265809+0000","last_peered":"2024-02-09T17:32:41.265809+0000","last_clean":"2024-02-09T17:32:41.265809+0000","last_became_active":"2024-02-09T17:32:35.407238+0000","last_became_peered":"2024-02-09T17:32:35.407238+0000","last_unstale":"2024-02-09T17:32:41.265809+0000","last_undegraded":"2024-02-09T17:32:41.265809+0000","last_fullsized":"2024-02-09T17:32:41.265809+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:31:59.409353+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:31:59.409353+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:32:46.176359+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":17,"num_write_kb":267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":240422,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,8],"acting":[7,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"256'10591","reported_seq":10782,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-02-09T17:32:44.561055+0000","last_change":"2024-02-09T17:30:55.910896+0000","last_active":"2024-02-09T17:32:44.561055+0000","last_peered":"2024-02-09T17:32:44.561055+0000","last_clean":"2024-02-09T17:32:44.561055+0000","last_became_active":"2024-02-09T17:29:47.582887+0000","last_became_peered":"2024-02-09T17:29:47.582887+0000","last_unstale":"2024-02-09T17:32:44.561055+0000","last_undegraded":"2024-02-09T17:32:44.561055+0000","last_fullsized":"2024-02-09T17:32:44.561055+0000","mapping_epoch":199,"log_start":"208'500","ondisk_log_start":"208'500","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"218'5796","last_scrub_stamp":"2024-02-09T17:30:55.910807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:30:55.910807+0000","objects_scrubbed":5754,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1081,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":388038656,"num_objects":5921,"num_object_clones":0,"num_object_copies":17763,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10591,"num_write_kb":528384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,6],"acting":[11,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"254'10411","reported_seq":10619,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-02-09T17:32:41.573920+0000","last_change":"2024-02-09T17:31:29.902726+0000","last_active":"2024-02-09T17:32:41.573920+0000","last_peered":"2024-02-09T17:32:41.573920+0000","last_clean":"2024-02-09T17:32:41.573920+0000","last_became_active":"2024-02-09T17:29:47.581341+0000","last_became_peered":"2024-02-09T17:29:47.581341+0000","last_unstale":"2024-02-09T17:32:41.573920+0000","last_undegraded":"2024-02-09T17:32:41.573920+0000","last_fullsized":"2024-02-09T17:32:41.573920+0000","mapping_epoch":199,"log_start":"208'400","ondisk_log_start":"208'400","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'8039","last_scrub_stamp":"2024-02-09T17:31:29.902558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:31:29.902558+0000","objects_scrubbed":7987,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:32:45.488168+0000","scrub_duration":1893,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403111936,"num_objects":6151,"num_object_clones":0,"num_object_copies":18453,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10411,"num_write_kb":529984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"63'139","reported_seq":738,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-02-09T17:32:41.262912+0000","last_change":"2024-02-09T17:32:30.296662+0000","last_active":"2024-02-09T17:32:41.262912+0000","last_peered":"2024-02-09T17:32:41.262912+0000","last_clean":"2024-02-09T17:32:41.262912+0000","last_became_active":"2024-02-09T17:32:30.296454+0000","last_became_peered":"2024-02-09T17:32:30.296454+0000","last_unstale":"2024-02-09T17:32:41.262912+0000","last_undegraded":"2024-02-09T17:32:41.262912+0000","last_fullsized":"2024-02-09T17:32:41.262912+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:31:59.409353+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:31:59.409353+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:32:42.153759+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":16,"num_write_kb":267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":240421,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":6384386072,"num_objects":97419,"num_object_clones":0,"num_object_copies":292257,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2091,"num_objects_misplaced":24297,"num_objects_unfound":0,"num_objects_dirty":97419,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166495,"num_write_kb":8445185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11143,"num_bytes_recovered":728760320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19910307840,"data_stored":19910295624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160916,"ondisk_log_size":160916,"up":48,"acting":44,"num_store_stats":12},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2251055,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1529,"ondisk_log_size":1529,"up":33,"acting":33,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056764,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3387156,"kb_used_data":3125512,"kb_used_omap":1,"kb_used_meta":261630,"kb_avail":90357996,"statfs":{"total":95995035648,"available":92526587904,"internally_reserved":0,"allocated":3200524288,"data_stored":3200205954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":267909572},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:32:00 2024","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":1.101,"15min":0.85599999999999998},"min":{"1min":0.26500000000000001,"5min":0.219,"15min":0.219},"max":{"1min":4.3079999999999998,"5min":14.913,"15min":14.913},"last":0.32800000000000001},{"interface":"front","average":{"1min":1.069,"5min":1.272,"15min":0.94299999999999995},"min":{"1min":0.28599999999999998,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":3.8039999999999998,"5min":14.869,"15min":14.869},"last":0.47299999999999998}]},{"osd":1,"last update":"Fri Feb 9 17:32:00 2024","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":1.97,"15min":1.244},"min":{"1min":0.30199999999999999,"5min":0.245,"15min":0.245},"max":{"1min":3.7429999999999999,"5min":32.957999999999998,"15min":32.957999999999998},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.99299999999999999,"5min":2.1619999999999999,"15min":1.282},"min":{"1min":0.39800000000000002,"5min":0.26800000000000002,"15min":0.26700000000000002},"max":{"1min":3.823,"5min":32.880000000000003,"15min":32.880000000000003},"last":0.5}]},{"osd":2,"last update":"Fri Feb 9 17:31:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.415,"15min":1.6859999999999999},"min":{"1min":0.378,"5min":0.247,"15min":0.247},"max":{"1min":4.4189999999999996,"5min":9.4719999999999995,"15min":9.4719999999999995},"last":5.4420000000000002},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.5149999999999999,"15min":1.4399999999999999},"min":{"1min":0.378,"5min":0.252,"15min":0.252},"max":{"1min":4.8579999999999997,"5min":7.96,"15min":7.96},"last":3.6160000000000001}]},{"osd":3,"last update":"Fri Feb 9 17:32:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":2.3220000000000001,"15min":1.3480000000000001},"min":{"1min":0.34399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":4.0990000000000002,"5min":21.315999999999999,"15min":21.315999999999999},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.85899999999999999,"5min":1.867,"15min":1.218},"min":{"1min":0.29599999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":3.722,"5min":18.087,"15min":18.087},"last":0.312}]},{"osd":4,"last update":"Fri Feb 9 17:32:00 2024","interfaces":[{"interface":"back","average":{"1min":1.7430000000000001,"5min":0.97199999999999998,"15min":0.83199999999999996},"min":{"1min":0.32700000000000001,"5min":0.30499999999999999,"15min":0.30299999999999999},"max":{"1min":6.8040000000000003,"5min":6.8040000000000003,"15min":6.8040000000000003},"last":0.51700000000000002},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.0700000000000001,"15min":0.88900000000000001},"min":{"1min":0.39800000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":3.673,"5min":6.5140000000000002,"15min":6.5140000000000002},"last":0.64900000000000002}]},{"osd":5,"last update":"Fri Feb 9 17:32:00 2024","interfaces":[{"interface":"back","average":{"1min":1.72,"5min":1.1659999999999999,"15min":0.92200000000000004},"min":{"1min":0.438,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":4.6959999999999997,"5min":13.891,"15min":13.891},"last":0.38800000000000001},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.014,"15min":0.87},"min":{"1min":0.36199999999999999,"5min":0.313,"15min":0.313},"max":{"1min":3.6400000000000001,"5min":6.5970000000000004,"15min":6.5970000000000004},"last":0.54800000000000004}]},{"osd":6,"last update":"Fri Feb 9 17:32:00 2024","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.018,"15min":0.88900000000000001},"min":{"1min":0.40400000000000003,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":5.0460000000000003,"5min":7.726,"15min":7.726},"last":5.2640000000000002},{"interface":"front","average":{"1min":1.454,"5min":1.1879999999999999,"15min":0.92000000000000004},"min":{"1min":0.39000000000000001,"5min":0.379,"15min":0.32300000000000001},"max":{"1min":5.7160000000000002,"5min":6.6219999999999999,"15min":6.6219999999999999},"last":0.56200000000000006}]},{"osd":7,"last update":"Fri Feb 9 17:32:00 2024","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.212,"15min":0.95899999999999996},"min":{"1min":0.442,"5min":0.317,"15min":0.317},"max":{"1min":4.3959999999999999,"5min":7.7720000000000002,"15min":7.7720000000000002},"last":0.61799999999999999},{"interface":"front","average":{"1min":1.526,"5min":1.149,"15min":0.93300000000000005},"min":{"1min":0.53100000000000003,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":4.4969999999999999,"5min":5.0369999999999999,"15min":5.0369999999999999},"last":0.57399999999999995}]},{"osd":8,"last update":"Fri Feb 9 17:25:48 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.71399999999999997,"15min":0.68000000000000005},"min":{"1min":0.40799999999999997,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.9350000000000001,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":0.40400000000000003},{"interface":"front","average":{"1min":1.3029999999999999,"5min":0.80700000000000005,"15min":0.71999999999999997},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":3.7389999999999999,"5min":3.7389999999999999,"15min":3.7389999999999999},"last":0.53500000000000003}]},{"osd":9,"last update":"Fri Feb 9 17:31:45 2024","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":1.0680000000000001,"15min":1.2170000000000001},"min":{"1min":0.46300000000000002,"5min":0.316,"15min":0.316},"max":{"1min":4.556,"5min":6.7279999999999998,"15min":6.7279999999999998},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.752,"5min":0.90000000000000002,"15min":0.88800000000000001},"min":{"1min":0.36099999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":4.109,"5min":6.0860000000000003,"15min":6.0860000000000003},"last":0.58699999999999997}]},{"osd":10,"last update":"Fri Feb 9 17:32:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.095,"15min":0.86899999999999999},"min":{"1min":0.434,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":4.218,"5min":6.7759999999999998,"15min":6.7759999999999998},"last":0.67600000000000005},{"interface":"front","average":{"1min":1.101,"5min":0.90900000000000003,"15min":0.82699999999999996},"min":{"1min":0.41699999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":4.4539999999999997,"5min":10.675000000000001,"15min":10.675000000000001},"last":0.73899999999999999}]}]},{"osd":10,"up_from":84,"seq":360777252993,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":496904,"kb_used_data":389372,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93248248,"statfs":{"total":95995035648,"available":95486205952,"internally_reserved":0,"allocated":398716928,"data_stored":398403018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":110098880},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:31:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":1.387,"15min":1.762},"min":{"1min":0.23899999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":1.29,"5min":20.120999999999999,"15min":20.120999999999999},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.74099999999999999,"5min":1.4039999999999999,"15min":2.3940000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":3.173,"5min":14.471,"15min":14.471},"last":0.44500000000000001}]},{"osd":1,"last update":"Fri Feb 9 17:31:39 2024","interfaces":[{"interface":"back","average":{"1min":3.4129999999999998,"5min":2.0569999999999999,"15min":1.143},"min":{"1min":0.24199999999999999,"5min":0.23899999999999999,"15min":0.23499999999999999},"max":{"1min":37.716000000000001,"5min":37.716000000000001,"15min":37.716000000000001},"last":0.35199999999999998},{"interface":"front","average":{"1min":2.9910000000000001,"5min":1.819,"15min":1.1240000000000001},"min":{"1min":0.30499999999999999,"5min":0.245,"15min":0.245},"max":{"1min":38.039999999999999,"5min":38.039999999999999,"15min":38.039999999999999},"last":0.374}]},{"osd":2,"last update":"Fri Feb 9 17:31:39 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":1.512,"15min":1.0580000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":3.149,"5min":13.351000000000001,"15min":13.351000000000001},"last":0.45300000000000001},{"interface":"front","average":{"1min":1.137,"5min":1.6459999999999999,"15min":1.042},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":4.0629999999999997,"5min":10.596,"15min":10.596},"last":0.48999999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:31:39 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":1.452,"15min":1.0920000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.26200000000000001},"max":{"1min":2.3359999999999999,"5min":6.9809999999999999,"15min":9.2650000000000006},"last":0.623},{"interface":"front","average":{"1min":0.95199999999999996,"5min":1.53,"15min":1.034},"min":{"1min":0.371,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":3.9590000000000001,"5min":8.1080000000000005,"15min":8.1080000000000005},"last":0.46300000000000002}]},{"osd":4,"last update":"Fri Feb 9 17:31:39 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":0.90700000000000003,"15min":0.82199999999999995},"min":{"1min":0.32600000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":4.4850000000000003,"5min":4.4850000000000003,"15min":6.2249999999999996},"last":1.05},{"interface":"front","average":{"1min":1.167,"5min":0.92500000000000004,"15min":0.84399999999999997},"min":{"1min":0.34200000000000003,"5min":0.307,"15min":0.307},"max":{"1min":3.7719999999999998,"5min":6.8369999999999997,"15min":6.8369999999999997},"last":0.53400000000000003}]},{"osd":5,"last update":"Fri Feb 9 17:31:39 2024","interfaces":[{"interface":"back","average":{"1min":2.2469999999999999,"5min":1.3440000000000001,"15min":1.014},"min":{"1min":0.35799999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":6.0270000000000001,"5min":6.9160000000000004,"15min":7.3339999999999996},"last":8.8290000000000006},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.371,"15min":0.95799999999999996},"min":{"1min":0.39400000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":4.2130000000000001,"5min":8.6159999999999997,"15min":8.6159999999999997},"last":0.56100000000000005}]},{"osd":6,"last update":"Fri Feb 9 17:31:39 2024","interfaces":[{"interface":"back","average":{"1min":1.7649999999999999,"5min":1.1850000000000001,"15min":0.89000000000000001},"min":{"1min":0.39100000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":6.8799999999999999,"5min":6.8799999999999999,"15min":6.8799999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":1.927,"5min":1.238,"15min":0.95699999999999996},"min":{"1min":0.43099999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":4.9820000000000002,"5min":6.8609999999999998,"15min":6.8609999999999998},"last":0.58499999999999996}]},{"osd":7,"last update":"Fri Feb 9 17:31:39 2024","interfaces":[{"interface":"back","average":{"1min":1.7130000000000001,"5min":1.089,"15min":0.877},"min":{"1min":0.45600000000000002,"5min":0.312,"15min":0.312},"max":{"1min":4.335,"5min":4.3659999999999997,"15min":4.3659999999999997},"last":0.81699999999999995},{"interface":"front","average":{"1min":1.165,"5min":1.1359999999999999,"15min":0.94199999999999995},"min":{"1min":0.42399999999999999,"5min":0.316,"15min":0.316},"max":{"1min":3.2810000000000001,"5min":6.9589999999999996,"15min":6.9589999999999996},"last":0.60899999999999999}]},{"osd":8,"last update":"Fri Feb 9 17:26:28 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.73799999999999999,"15min":0.64000000000000001},"min":{"1min":0.36399999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":3.6509999999999998,"5min":3.6509999999999998,"15min":3.6509999999999998},"last":0.73399999999999999},{"interface":"front","average":{"1min":1.72,"5min":0.89500000000000002,"15min":0.68200000000000005},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":4.173,"5min":4.173,"15min":4.173},"last":0.78200000000000003}]},{"osd":9,"last update":"Fri Feb 9 17:31:39 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":1.036,"15min":0.83199999999999996},"min":{"1min":0.34399999999999997,"5min":0.314,"15min":0.314},"max":{"1min":3.9449999999999998,"5min":8.7240000000000002,"15min":8.7240000000000002},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.753,"5min":1.3,"15min":0.96799999999999997},"min":{"1min":0.42099999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":4.0039999999999996,"5min":10.1,"15min":10.1},"last":0.77000000000000002}]},{"osd":11,"last update":"Fri Feb 9 17:32:09 2024","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":1.1220000000000001,"15min":0.98899999999999999},"min":{"1min":0.45700000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":4.0330000000000004,"5min":8.6430000000000007,"15min":8.6430000000000007},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.94599999999999995,"5min":1.151,"15min":0.99199999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.33400000000000002},"max":{"1min":3.7170000000000001,"5min":9.298,"15min":9.298},"last":0.97999999999999998}]}]},{"osd":9,"up_from":77,"seq":330712481928,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1343768,"kb_used_data":1159884,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":92401384,"statfs":{"total":95995035648,"available":94619017216,"internally_reserved":0,"allocated":1187721216,"data_stored":1187398810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":188283328},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:28:03 2024","interfaces":[{"interface":"back","average":{"1min":1.9510000000000001,"5min":1.454,"15min":0.85799999999999998},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":8.4499999999999993,"5min":17.495999999999999,"15min":17.495999999999999},"last":2.7360000000000002},{"interface":"front","average":{"1min":1.905,"5min":1.399,"15min":0.83499999999999996},"min":{"1min":0.32200000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":7.5990000000000002,"5min":15.401999999999999,"15min":15.401999999999999},"last":1.5449999999999999}]},{"osd":1,"last update":"Fri Feb 9 17:32:17 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":1.4339999999999999,"15min":1.1180000000000001},"min":{"1min":0.23699999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":4.1799999999999997,"5min":16.527999999999999,"15min":16.527999999999999},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.65600000000000003,"5min":1.276,"15min":0.99299999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":2.9990000000000001,"5min":8.6769999999999996,"15min":17.771999999999998},"last":0.39800000000000002}]},{"osd":2,"last update":"Fri Feb 9 17:32:17 2024","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.3029999999999999,"15min":1.02},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":6.0570000000000004,"5min":8.3550000000000004,"15min":13.878},"last":0.437},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.4239999999999999,"15min":1.0900000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":4.0970000000000004,"5min":9.3849999999999998,"15min":13.69},"last":0.58799999999999997}]},{"osd":3,"last update":"Fri Feb 9 17:32:17 2024","interfaces":[{"interface":"back","average":{"1min":3.0859999999999999,"5min":1.5469999999999999,"15min":1.089},"min":{"1min":0.309,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":40.18,"5min":40.18,"15min":40.18},"last":0.315},{"interface":"front","average":{"1min":3.085,"5min":1.9590000000000001,"15min":1.264},"min":{"1min":0.32300000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":40.158000000000001,"5min":40.158000000000001,"15min":40.158000000000001},"last":0.32600000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:32:17 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.393,"15min":0.99299999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":7.2969999999999997,"5min":22.577999999999999,"15min":22.577999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":1.514,"5min":1.1739999999999999,"15min":0.90400000000000003},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":10.403,"5min":22.399999999999999,"15min":22.399999999999999},"last":4.0330000000000004}]},{"osd":5,"last update":"Fri Feb 9 17:32:17 2024","interfaces":[{"interface":"back","average":{"1min":1.496,"5min":1.0880000000000001,"15min":0.90300000000000002},"min":{"1min":0.32100000000000001,"5min":0.31900000000000001,"15min":0.29299999999999998},"max":{"1min":6.1470000000000002,"5min":8.2460000000000004,"15min":8.2460000000000004},"last":2.359},{"interface":"front","average":{"1min":1.6040000000000001,"5min":1.0720000000000001,"15min":0.871},"min":{"1min":0.32000000000000001,"5min":0.315,"15min":0.315},"max":{"1min":4.3520000000000003,"5min":4.3520000000000003,"15min":5.665},"last":0.36299999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:32:17 2024","interfaces":[{"interface":"back","average":{"1min":2.169,"5min":1.2509999999999999,"15min":1.002},"min":{"1min":0.46000000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":7.3650000000000002,"5min":8.2959999999999994,"15min":8.2959999999999994},"last":2.4980000000000002},{"interface":"front","average":{"1min":2.2229999999999999,"5min":1.3520000000000001,"15min":0.996},"min":{"1min":0.48499999999999999,"5min":0.36199999999999999,"15min":0.35099999999999998},"max":{"1min":7.4000000000000004,"5min":8.3130000000000006,"15min":8.3130000000000006},"last":2.5270000000000001}]},{"osd":7,"last update":"Fri Feb 9 17:32:17 2024","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.0409999999999999,"15min":0.90100000000000002},"min":{"1min":0.38500000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":3.4300000000000002,"5min":8.0939999999999994,"15min":8.0939999999999994},"last":4.1929999999999996},{"interface":"front","average":{"1min":1.2110000000000001,"5min":1.218,"15min":0.95599999999999996},"min":{"1min":0.44800000000000001,"5min":0.36499999999999999,"15min":0.34899999999999998},"max":{"1min":4.4630000000000001,"5min":5.1829999999999998,"15min":6.056},"last":4.2229999999999999}]},{"osd":8,"last update":"Fri Feb 9 17:32:28 2024","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.2470000000000001,"15min":1.331},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":4.7240000000000002,"5min":16.209,"15min":16.209},"last":0.46000000000000002},{"interface":"front","average":{"1min":1.5369999999999999,"5min":1.353,"15min":1.399},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":10.241,"5min":16.263000000000002,"15min":16.263000000000002},"last":0.52300000000000002}]},{"osd":10,"last update":"Fri Feb 9 17:31:42 2024","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.052,"15min":0.90300000000000002},"min":{"1min":0.36799999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":9.5090000000000003,"5min":9.5090000000000003,"15min":9.5090000000000003},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.70799999999999996,"5min":1.095,"15min":0.91500000000000004},"min":{"1min":0.46300000000000002,"5min":0.38200000000000001,"15min":0.36099999999999999},"max":{"1min":1.972,"5min":8.0670000000000002,"15min":8.0670000000000002},"last":0.86799999999999999}]},{"osd":11,"last update":"Fri Feb 9 17:31:48 2024","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.278,"15min":1.2609999999999999},"min":{"1min":0.49099999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":6.181,"5min":8.2759999999999998,"15min":8.2759999999999998},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.92800000000000005,"5min":1.0149999999999999,"15min":1.0189999999999999},"min":{"1min":0.41699999999999998,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":2.9780000000000002,"5min":6.2060000000000004,"15min":6.2060000000000004},"last":0.54800000000000004}]}]},{"osd":8,"up_from":171,"seq":734439407670,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":906556,"kb_used_data":793136,"kb_used_omap":212,"kb_used_meta":113195,"kb_avail":92838596,"statfs":{"total":95995035648,"available":95066722304,"internally_reserved":0,"allocated":812171264,"data_stored":811861665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217579,"internal_metadata":115912213},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:32:23 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":1.1559999999999999,"15min":1.5920000000000001},"min":{"1min":0.26300000000000001,"5min":0.245,"15min":0.245},"max":{"1min":2.7970000000000002,"5min":6.8840000000000003,"15min":6.8840000000000003},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.95599999999999996,"5min":1.262,"15min":1.514},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":3.278,"5min":7.7969999999999997,"15min":7.7969999999999997},"last":0.50600000000000001}]},{"osd":1,"last update":"Fri Feb 9 17:32:23 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.0900000000000001,"15min":1.2949999999999999},"min":{"1min":0.28799999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":7.8719999999999999,"5min":7.8719999999999999,"15min":7.8719999999999999},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.90200000000000002,"5min":1.5209999999999999,"15min":2.1709999999999998},"min":{"1min":0.316,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":4.8609999999999998,"5min":11.101000000000001,"15min":11.101000000000001},"last":0.29299999999999998}]},{"osd":2,"last update":"Fri Feb 9 17:32:23 2024","interfaces":[{"interface":"back","average":{"1min":1.5449999999999999,"5min":1.2989999999999999,"15min":1.4550000000000001},"min":{"1min":0.28499999999999998,"5min":0.255,"15min":0.255},"max":{"1min":9.1820000000000004,"5min":9.1820000000000004,"15min":9.1820000000000004},"last":0.48599999999999999},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.407,"15min":1.7789999999999999},"min":{"1min":0.34399999999999997,"5min":0.313,"15min":0.313},"max":{"1min":7.7709999999999999,"5min":7.7709999999999999,"15min":7.7709999999999999},"last":0.55600000000000005}]},{"osd":3,"last update":"Fri Feb 9 17:32:23 2024","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.833,"15min":2.2989999999999999},"min":{"1min":0.36399999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.3129999999999997,"5min":12.505000000000001,"15min":12.505000000000001},"last":24.030999999999999},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.9610000000000001,"15min":2.6899999999999999},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":5.6689999999999996,"5min":10.92,"15min":10.92},"last":24.318999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:32:23 2024","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":1.194,"15min":0.86599999999999999},"min":{"1min":0.307,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":3.1739999999999999,"5min":37.343000000000004,"15min":37.343000000000004},"last":0.63},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.4299999999999999,"15min":1.1899999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":6.2670000000000003,"5min":37.875,"15min":37.875},"last":0.374}]},{"osd":6,"last update":"Fri Feb 9 17:32:23 2024","interfaces":[{"interface":"back","average":{"1min":1.9359999999999999,"5min":1.7729999999999999,"15min":1.4990000000000001},"min":{"1min":0.376,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":9.2959999999999994,"5min":37.356999999999999,"15min":37.356999999999999},"last":0.52500000000000002},{"interface":"front","average":{"1min":1.361,"5min":1.337,"15min":0.98099999999999998},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":4.5549999999999997,"5min":37.25,"15min":37.25},"last":0.61299999999999999}]},{"osd":7,"last update":"Fri Feb 9 17:32:23 2024","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.506,"15min":1.2210000000000001},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":7.9450000000000003,"5min":37.585000000000001,"15min":37.585000000000001},"last":1.2190000000000001},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.619,"15min":1.4670000000000001},"min":{"1min":0.45300000000000001,"5min":0.312,"15min":0.312},"max":{"1min":8.0180000000000007,"5min":37.918999999999997,"15min":37.918999999999997},"last":1.2430000000000001}]},{"osd":9,"last update":"Fri Feb 9 17:32:23 2024","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.0389999999999999,"15min":1.1899999999999999},"min":{"1min":0.34100000000000003,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":3.3450000000000002,"5min":6.9269999999999996,"15min":6.9269999999999996},"last":0.64600000000000002},{"interface":"front","average":{"1min":1.052,"5min":1.0629999999999999,"15min":1.2270000000000001},"min":{"1min":0.35399999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":5.1580000000000004,"5min":7.7409999999999997,"15min":7.7409999999999997},"last":0.54000000000000004}]},{"osd":10,"last update":"Fri Feb 9 17:32:23 2024","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":1.1240000000000001,"15min":1.266},"min":{"1min":0.376,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":9.2609999999999992,"5min":9.2609999999999992,"15min":9.2609999999999992},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.92500000000000004,"15min":1.0489999999999999},"min":{"1min":0.39300000000000002,"5min":0.37,"15min":0.37},"max":{"1min":3.8969999999999998,"5min":5.1070000000000002,"15min":5.1070000000000002},"last":0.66000000000000003}]},{"osd":11,"last update":"Fri Feb 9 17:32:23 2024","interfaces":[{"interface":"back","average":{"1min":1.6830000000000001,"5min":1.2490000000000001,"15min":1.335},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":9.234,"5min":9.234,"15min":9.234},"last":0.56899999999999995},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.2170000000000001,"15min":1.5449999999999999},"min":{"1min":0.45500000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":5.4829999999999997,"5min":7.6609999999999996,"15min":7.6609999999999996},"last":0.46999999999999997}]}]},{"osd":7,"up_from":57,"seq":244813136016,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1781744,"kb_used_data":1592296,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":91963408,"statfs":{"total":95995035648,"available":94170529792,"internally_reserved":0,"allocated":1630511104,"data_stored":1630201832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":193984970},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:31:49 2024","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.2410000000000001,"15min":0.93300000000000005},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.6710000000000003,"5min":10.776,"15min":10.776},"last":1.5149999999999999},{"interface":"front","average":{"1min":1.335,"5min":1.5940000000000001,"15min":1.0489999999999999},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":5.734,"5min":18.896000000000001,"15min":18.896000000000001},"last":1.5680000000000001}]},{"osd":1,"last update":"Fri Feb 9 17:27:46 2024","interfaces":[{"interface":"back","average":{"1min":2.262,"5min":2.262,"15min":2.262},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":4.8470000000000004,"5min":4.8470000000000004,"15min":4.8470000000000004},"last":3.9470000000000001},{"interface":"front","average":{"1min":1.8009999999999999,"5min":1.8009999999999999,"15min":1.8009999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.9589999999999996,"5min":6.9589999999999996,"15min":6.9589999999999996},"last":3.6669999999999998}]},{"osd":2,"last update":"Fri Feb 9 17:32:26 2024","interfaces":[{"interface":"back","average":{"1min":1.964,"5min":2.0230000000000001,"15min":1.151},"min":{"1min":0.31900000000000001,"5min":0.30399999999999999,"15min":0.26000000000000001},"max":{"1min":11.324,"5min":11.324,"15min":11.324},"last":0.55400000000000005},{"interface":"front","average":{"1min":2.2410000000000001,"5min":1.746,"15min":1.1890000000000001},"min":{"1min":0.309,"5min":0.309,"15min":0.28299999999999997},"max":{"1min":11.455,"5min":11.455,"15min":11.455},"last":0.61499999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:32:26 2024","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.8819999999999999,"15min":1.1160000000000001},"min":{"1min":0.28899999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":7.6529999999999996,"5min":27.719000000000001,"15min":27.719000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":2.5030000000000001,"5min":1.881,"15min":1.1799999999999999},"min":{"1min":0.34499999999999997,"5min":0.311,"15min":0.27600000000000002},"max":{"1min":6.9240000000000004,"5min":6.9240000000000004,"15min":8.8629999999999995},"last":1.4810000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:32:26 2024","interfaces":[{"interface":"back","average":{"1min":1.6890000000000001,"5min":1.196,"15min":0.76800000000000002},"min":{"1min":0.29399999999999998,"5min":0.27200000000000002,"15min":0.248},"max":{"1min":8.6150000000000002,"5min":8.6150000000000002,"15min":8.6150000000000002},"last":6.2889999999999997},{"interface":"front","average":{"1min":1.7629999999999999,"5min":1.0960000000000001,"15min":0.748},"min":{"1min":0.35299999999999998,"5min":0.23300000000000001,"15min":0.22800000000000001},"max":{"1min":8.6419999999999995,"5min":9.3650000000000002,"15min":9.3650000000000002},"last":6.3220000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:32:26 2024","interfaces":[{"interface":"back","average":{"1min":1.8129999999999999,"5min":1.2490000000000001,"15min":0.79200000000000004},"min":{"1min":0.33200000000000002,"5min":0.28599999999999998,"15min":0.21299999999999999},"max":{"1min":11.548999999999999,"5min":11.548999999999999,"15min":11.548999999999999},"last":5.9880000000000004},{"interface":"front","average":{"1min":2.2749999999999999,"5min":1.22,"15min":0.82999999999999996},"min":{"1min":0.317,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":11.598000000000001,"5min":11.598000000000001,"15min":11.598000000000001},"last":6.4820000000000002}]},{"osd":6,"last update":"Fri Feb 9 17:32:26 2024","interfaces":[{"interface":"back","average":{"1min":2.0800000000000001,"5min":1.391,"15min":0.88700000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":11.369,"5min":12.496,"15min":12.496},"last":0.70599999999999996},{"interface":"front","average":{"1min":2.1400000000000001,"5min":1.252,"15min":0.83399999999999996},"min":{"1min":0.377,"5min":0.32500000000000001,"15min":0.28699999999999998},"max":{"1min":6.8399999999999999,"5min":6.8399999999999999,"15min":6.8399999999999999},"last":1.5429999999999999}]},{"osd":8,"last update":"Fri Feb 9 17:32:26 2024","interfaces":[{"interface":"back","average":{"1min":1.45,"5min":1.335,"15min":1.417},"min":{"1min":0.46300000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":7.7839999999999998,"5min":7.9420000000000002,"15min":7.9420000000000002},"last":0.72699999999999998},{"interface":"front","average":{"1min":1.7749999999999999,"5min":1.339,"15min":1.079},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":11.481,"5min":11.481,"15min":11.481},"last":0.76400000000000001}]},{"osd":9,"last update":"Fri Feb 9 17:32:08 2024","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.5529999999999999,"15min":1.075},"min":{"1min":0.48899999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":3.2029999999999998,"5min":41.286999999999999,"15min":41.286999999999999},"last":0.754},{"interface":"front","average":{"1min":1.5920000000000001,"5min":1.6970000000000001,"15min":1.0980000000000001},"min":{"1min":0.44700000000000001,"5min":0.40000000000000002,"15min":0.35399999999999998},"max":{"1min":7.9180000000000001,"5min":41.234999999999999,"15min":41.234999999999999},"last":0.80000000000000004}]},{"osd":10,"last update":"Fri Feb 9 17:32:36 2024","interfaces":[{"interface":"back","average":{"1min":2.1909999999999998,"5min":1.357,"15min":1.075},"min":{"1min":0.43099999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":6.9950000000000001,"5min":6.9950000000000001,"15min":8.4659999999999993},"last":1.593},{"interface":"front","average":{"1min":1.778,"5min":1.3100000000000001,"15min":1.0409999999999999},"min":{"1min":0.45600000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":7.8099999999999996,"5min":7.8099999999999996,"15min":7.8099999999999996},"last":3.4580000000000002}]},{"osd":11,"last update":"Fri Feb 9 17:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.6399999999999999,"5min":1.3759999999999999,"15min":1.0589999999999999},"min":{"1min":0.41999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":6.3899999999999997,"5min":6.3899999999999997,"15min":6.3899999999999997},"last":3.496},{"interface":"front","average":{"1min":1.8240000000000001,"5min":1.24,"15min":1.01},"min":{"1min":0.42699999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":6.9580000000000002,"5min":8.1799999999999997,"15min":8.1799999999999997},"last":2.274}]}]},{"osd":6,"up_from":49,"seq":210453397654,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2044240,"kb_used_data":1926984,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":91700912,"statfs":{"total":95995035648,"available":93901733888,"internally_reserved":0,"allocated":1973231616,"data_stored":1972913290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":120060358},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:31:54 2024","interfaces":[{"interface":"back","average":{"1min":1.96,"5min":1.579,"15min":1.036},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.24299999999999999},"max":{"1min":9.5570000000000004,"5min":11.582000000000001,"15min":24.544},"last":0.503},{"interface":"front","average":{"1min":1.51,"5min":1.3720000000000001,"15min":0.98199999999999998},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":6.1210000000000004,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.54500000000000004}]},{"osd":1,"last update":"Fri Feb 9 17:31:54 2024","interfaces":[{"interface":"back","average":{"1min":1.641,"5min":1.5209999999999999,"15min":1.036},"min":{"1min":0.308,"5min":0.27400000000000002,"15min":0.27000000000000002},"max":{"1min":9.2739999999999991,"5min":9.4469999999999992,"15min":23.954000000000001},"last":0.308},{"interface":"front","average":{"1min":1.7010000000000001,"5min":1.5780000000000001,"15min":1.069},"min":{"1min":0.379,"5min":0.318,"15min":0.318},"max":{"1min":9.3079999999999998,"5min":9.4930000000000003,"15min":23.984999999999999},"last":0.33200000000000002}]},{"osd":2,"last update":"Fri Feb 9 17:31:54 2024","interfaces":[{"interface":"back","average":{"1min":1.542,"5min":1.6699999999999999,"15min":1.127},"min":{"1min":0.32700000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":6.7199999999999998,"5min":10.682,"15min":24.329999999999998},"last":3.948},{"interface":"front","average":{"1min":1.96,"5min":1.821,"15min":1.1279999999999999},"min":{"1min":0.42999999999999999,"5min":0.32100000000000001,"15min":0.28000000000000003},"max":{"1min":6.8819999999999997,"5min":11.553000000000001,"15min":12.906000000000001},"last":0.52800000000000002}]},{"osd":3,"last update":"Fri Feb 9 17:31:54 2024","interfaces":[{"interface":"back","average":{"1min":1.7010000000000001,"5min":2.0910000000000002,"15min":1.1819999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.248},"max":{"1min":9.6920000000000002,"5min":32.545000000000002,"15min":32.545000000000002},"last":4.0250000000000004},{"interface":"front","average":{"1min":1.645,"5min":1.6779999999999999,"15min":1.1220000000000001},"min":{"1min":0.35099999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":6.6310000000000002,"5min":32.670999999999999,"15min":32.670999999999999},"last":3.9750000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:31:54 2024","interfaces":[{"interface":"back","average":{"1min":1.6140000000000001,"5min":0.995,"15min":0.68000000000000005},"min":{"1min":0.314,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":6.2409999999999997,"5min":8.7110000000000003,"15min":8.7110000000000003},"last":0.27200000000000002},{"interface":"front","average":{"1min":1.8140000000000001,"5min":1.1970000000000001,"15min":0.77700000000000002},"min":{"1min":0.41599999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":6.2859999999999996,"5min":8.8209999999999997,"15min":8.8209999999999997},"last":0.56299999999999994}]},{"osd":5,"last update":"Fri Feb 9 17:31:54 2024","interfaces":[{"interface":"back","average":{"1min":1.887,"5min":1.0449999999999999,"15min":0.72699999999999998},"min":{"1min":0.40999999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":6.8040000000000003,"5min":6.8040000000000003,"15min":6.8040000000000003},"last":0.621},{"interface":"front","average":{"1min":1.7050000000000001,"5min":1.0660000000000001,"15min":0.72399999999999998},"min":{"1min":0.39500000000000002,"5min":0.29299999999999998,"15min":0.27400000000000002},"max":{"1min":4.9509999999999996,"5min":7.601,"15min":7.601},"last":0.46100000000000002}]},{"osd":7,"last update":"Fri Feb 9 17:32:34 2024","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":1.1459999999999999,"15min":0.82199999999999995},"min":{"1min":0.316,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":6.8280000000000003,"5min":6.8280000000000003,"15min":6.8280000000000003},"last":3.895},{"interface":"front","average":{"1min":1.038,"5min":1.365,"15min":0.89600000000000002},"min":{"1min":0.36599999999999999,"5min":0.309,"15min":0.309},"max":{"1min":3.746,"5min":11.512,"15min":11.512},"last":0.57799999999999996}]},{"osd":8,"last update":"Fri Feb 9 17:26:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":0.876,"15min":0.79600000000000004},"min":{"1min":0.39900000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":4.0339999999999998,"5min":4.4880000000000004,"15min":4.4880000000000004},"last":4.3840000000000003},{"interface":"front","average":{"1min":1.0469999999999999,"5min":0.88600000000000001,"15min":0.78300000000000003},"min":{"1min":0.375,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":3.6539999999999999,"5min":4.8570000000000002,"15min":4.8570000000000002},"last":4.4210000000000003}]},{"osd":9,"last update":"Fri Feb 9 17:32:07 2024","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.286,"15min":0.97599999999999998},"min":{"1min":0.50800000000000001,"5min":0.36899999999999999,"15min":0.35999999999999999},"max":{"1min":3.6880000000000002,"5min":10.803000000000001,"15min":10.803000000000001},"last":0.59299999999999997},{"interface":"front","average":{"1min":1.22,"5min":1.3029999999999999,"15min":1.0309999999999999},"min":{"1min":0.52400000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":3.7440000000000002,"5min":6.3109999999999999,"15min":6.3109999999999999},"last":0.60699999999999998}]},{"osd":10,"last update":"Fri Feb 9 17:32:34 2024","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.319,"15min":1.077},"min":{"1min":0.40699999999999997,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":6.194,"5min":6.875,"15min":9.5399999999999991},"last":4.0010000000000003},{"interface":"front","average":{"1min":1.8360000000000001,"5min":1.306,"15min":1.0840000000000001},"min":{"1min":0.51000000000000001,"5min":0.42499999999999999,"15min":0.39200000000000002},"max":{"1min":5.6139999999999999,"5min":5.6139999999999999,"15min":9.5210000000000008},"last":2.819}]},{"osd":11,"last update":"Fri Feb 9 17:31:54 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.278,"15min":1.5189999999999999},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":6.8230000000000004,"5min":6.8230000000000004,"15min":6.8230000000000004},"last":4.0469999999999997},{"interface":"front","average":{"1min":1.6240000000000001,"5min":1.403,"15min":1.6850000000000001},"min":{"1min":0.56000000000000005,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":6.2350000000000003,"5min":6.3200000000000003,"15min":6.3200000000000003},"last":0.78800000000000003}]}]},{"osd":1,"up_from":13,"seq":55834575024,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1714708,"kb_used_data":1548872,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":92030444,"statfs":{"total":95995035648,"available":94239174656,"internally_reserved":0,"allocated":1586044928,"data_stored":1585726602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":169802181},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.264,"5min":1.7090000000000001,"15min":1.0089999999999999},"min":{"1min":0.255,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":4.3010000000000002,"5min":12.492000000000001,"15min":12.492000000000001},"last":0.32600000000000001},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.8919999999999999,"15min":1.0329999999999999},"min":{"1min":0.218,"5min":0.214,"15min":0.21299999999999999},"max":{"1min":5.2089999999999996,"5min":9.8439999999999994,"15min":9.8439999999999994},"last":0.51300000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:32:28 2024","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":1.623,"15min":1.1659999999999999},"min":{"1min":0.23899999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.4740000000000002,"5min":12.01,"15min":12.01},"last":3.3879999999999999},{"interface":"front","average":{"1min":1.2989999999999999,"5min":2.085,"15min":1.325},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":5.4480000000000004,"5min":17.459,"15min":17.459},"last":0.36199999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:31:48 2024","interfaces":[{"interface":"back","average":{"1min":1.571,"5min":2.0590000000000002,"15min":1.3899999999999999},"min":{"1min":0.32300000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":5.2359999999999998,"5min":16.088000000000001,"15min":16.088000000000001},"last":4.0679999999999996},{"interface":"front","average":{"1min":0.98999999999999999,"5min":1.927,"15min":1.1479999999999999},"min":{"1min":0.245,"5min":0.222,"15min":0.222},"max":{"1min":3.5630000000000002,"5min":12.378,"15min":12.378},"last":4.1239999999999997}]},{"osd":4,"last update":"Fri Feb 9 17:28:03 2024","interfaces":[{"interface":"back","average":{"1min":1.593,"5min":1.323,"15min":0.81899999999999995},"min":{"1min":0.28199999999999997,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":7.3559999999999999,"5min":7.7210000000000001,"15min":7.7210000000000001},"last":1.7050000000000001},{"interface":"front","average":{"1min":2.359,"5min":1.5209999999999999,"15min":0.89200000000000002},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":9.3979999999999997,"5min":9.5280000000000005,"15min":9.5280000000000005},"last":1.6759999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:32:41 2024","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.6359999999999999,"15min":1.181},"min":{"1min":0.28599999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":5.1950000000000003,"5min":7.077,"15min":7.7679999999999998},"last":0.45600000000000002},{"interface":"front","average":{"1min":1.272,"5min":1.3759999999999999,"15min":0.98999999999999999},"min":{"1min":0.35299999999999998,"5min":0.317,"15min":0.26000000000000001},"max":{"1min":5.2169999999999996,"5min":7.9009999999999998,"15min":7.9009999999999998},"last":0.42799999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:32:01 2024","interfaces":[{"interface":"back","average":{"1min":1.6319999999999999,"5min":1.718,"15min":1.2310000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":5.7709999999999999,"5min":10.275,"15min":10.275},"last":0.504},{"interface":"front","average":{"1min":1.377,"5min":1.587,"15min":1.1759999999999999},"min":{"1min":0.36699999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":4.1890000000000001,"5min":7.0510000000000002,"15min":8.0229999999999997},"last":0.56899999999999995}]},{"osd":7,"last update":"Fri Feb 9 17:32:28 2024","interfaces":[{"interface":"back","average":{"1min":1.962,"5min":2.0920000000000001,"15min":1.381},"min":{"1min":0.32400000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":9.0700000000000003,"5min":15.148999999999999,"15min":15.148999999999999},"last":1.9139999999999999},{"interface":"front","average":{"1min":1.506,"5min":1.929,"15min":1.351},"min":{"1min":0.495,"5min":0.32600000000000001,"15min":0.28299999999999997},"max":{"1min":4.2119999999999997,"5min":13.834,"15min":13.834},"last":0.54600000000000004}]},{"osd":8,"last update":"Fri Feb 9 17:32:23 2024","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.2569999999999999,"15min":1.4610000000000001},"min":{"1min":0.45300000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":4.851,"5min":4.851,"15min":4.851},"last":0.67200000000000004},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.7090000000000001,"15min":2.1970000000000001},"min":{"1min":0.48099999999999998,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":5.0659999999999998,"5min":6.7530000000000001,"15min":6.7530000000000001},"last":0.75}]},{"osd":9,"last update":"Fri Feb 9 17:32:08 2024","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.621,"15min":1.2769999999999999},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":4.3799999999999999,"5min":7.3140000000000001,"15min":7.8010000000000002},"last":0.53200000000000003},{"interface":"front","average":{"1min":1.1850000000000001,"5min":1.752,"15min":1.3069999999999999},"min":{"1min":0.40400000000000003,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":4.1230000000000002,"5min":12.117000000000001,"15min":12.117000000000001},"last":0.60199999999999998}]},{"osd":10,"last update":"Fri Feb 9 17:32:41 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":1.355,"15min":1.1080000000000001},"min":{"1min":0.47999999999999998,"5min":0.39600000000000002,"15min":0.35999999999999999},"max":{"1min":1.7110000000000001,"5min":6.5650000000000004,"15min":7.149},"last":0.71599999999999997},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.8320000000000001,"15min":1.3759999999999999},"min":{"1min":0.49399999999999999,"5min":0.40899999999999997,"15min":0.33400000000000002},"max":{"1min":5.5430000000000001,"5min":12.087999999999999,"15min":12.087999999999999},"last":0.63800000000000001}]},{"osd":11,"last update":"Fri Feb 9 17:31:47 2024","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":2.097,"15min":2.4849999999999999},"min":{"1min":0.52900000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":7.7919999999999998,"5min":12.169,"15min":12.169},"last":0.69199999999999995},{"interface":"front","average":{"1min":1.5149999999999999,"5min":2.1320000000000001,"15min":2.286},"min":{"1min":0.44600000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":7.8220000000000001,"5min":12.144,"15min":12.144},"last":0.73499999999999999}]}]},{"osd":0,"up_from":0,"seq":34359738548,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411498,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":486108,"kb_used_data":382736,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93259044,"statfs":{"total":95995035648,"available":95497261056,"internally_reserved":0,"allocated":391921664,"data_stored":391599786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":1.4510000000000001,"15min":1.006},"min":{"1min":0.29999999999999999,"5min":0.23699999999999999,"15min":0.221},"max":{"1min":4.9089999999999998,"5min":17.251000000000001,"15min":17.251000000000001},"last":0.437},{"interface":"front","average":{"1min":0.74399999999999999,"5min":1.611,"15min":1.113},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.22900000000000001},"max":{"1min":4.7720000000000002,"5min":23.097999999999999,"15min":23.097999999999999},"last":0.33200000000000002}]},{"osd":1,"last update":"Fri Feb 9 17:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":1.6259999999999999,"15min":1.129},"min":{"1min":0.27900000000000003,"5min":0.249,"15min":0.249},"max":{"1min":2.8460000000000001,"5min":27.289000000000001,"15min":27.289000000000001},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.85599999999999998,"5min":1.8540000000000001,"15min":1.212},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":4.4379999999999997,"5min":37.585999999999999,"15min":37.585999999999999},"last":0.308}]},{"osd":3,"last update":"Fri Feb 9 17:31:52 2024","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":2.339,"15min":1.246},"min":{"1min":0.33700000000000002,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":5.4340000000000002,"5min":37.634999999999998,"15min":37.634999999999998},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.85299999999999998,"5min":2.0640000000000001,"15min":1.119},"min":{"1min":0.27500000000000002,"5min":0.255,"15min":0.23100000000000001},"max":{"1min":3.5459999999999998,"5min":38.924999999999997,"15min":38.924999999999997},"last":0.42299999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:31:58 2024","interfaces":[{"interface":"back","average":{"1min":2.0339999999999998,"5min":1.7370000000000001,"15min":1.0820000000000001},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.254},"max":{"1min":5.8380000000000001,"5min":7.6399999999999997,"15min":8.8089999999999993},"last":0.53600000000000003},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.347,"15min":0.97299999999999998},"min":{"1min":0.33700000000000002,"5min":0.315,"15min":0.30399999999999999},"max":{"1min":4.0350000000000001,"5min":13.305999999999999,"15min":13.305999999999999},"last":0.51800000000000002}]},{"osd":5,"last update":"Fri Feb 9 17:32:33 2024","interfaces":[{"interface":"back","average":{"1min":2.004,"5min":1.772,"15min":1.1919999999999999},"min":{"1min":0.44800000000000001,"5min":0.32300000000000001,"15min":0.28999999999999998},"max":{"1min":5.3419999999999996,"5min":6.7140000000000004,"15min":8.6859999999999999},"last":0.505},{"interface":"front","average":{"1min":1.51,"5min":1.4019999999999999,"15min":1.008},"min":{"1min":0.32100000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":4.7519999999999998,"5min":13.278,"15min":13.278},"last":0.57199999999999995}]},{"osd":6,"last update":"Fri Feb 9 17:31:58 2024","interfaces":[{"interface":"back","average":{"1min":1.915,"5min":1.849,"15min":1.1619999999999999},"min":{"1min":0.36699999999999999,"5min":0.35199999999999998,"15min":0.32100000000000001},"max":{"1min":6.077,"5min":7.6159999999999997,"15min":8.8399999999999999},"last":1.1619999999999999},{"interface":"front","average":{"1min":2.1299999999999999,"5min":1.905,"15min":1.1879999999999999},"min":{"1min":0.39900000000000002,"5min":0.35599999999999998,"15min":0.34999999999999998},"max":{"1min":9.1080000000000005,"5min":9.1080000000000005,"15min":9.1080000000000005},"last":1.2030000000000001}]},{"osd":7,"last update":"Fri Feb 9 17:32:27 2024","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.607,"15min":1.1499999999999999},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":4.2599999999999998,"5min":14.541,"15min":14.541},"last":0.40000000000000002},{"interface":"front","average":{"1min":1.653,"5min":1.9870000000000001,"15min":1.3260000000000001},"min":{"1min":0.5,"5min":0.41299999999999998,"15min":0.40500000000000003},"max":{"1min":4.4669999999999996,"5min":8.4169999999999998,"15min":8.7870000000000008},"last":0.58399999999999996}]},{"osd":8,"last update":"Fri Feb 9 17:26:31 2024","interfaces":[{"interface":"back","average":{"1min":1.7529999999999999,"5min":1,"15min":0.84599999999999997},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":8.8989999999999991,"5min":8.8989999999999991,"15min":8.8989999999999991},"last":5.2350000000000003},{"interface":"front","average":{"1min":2.4969999999999999,"5min":1.1080000000000001,"15min":0.90600000000000003},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":9.4190000000000005,"5min":9.4190000000000005,"15min":9.4190000000000005},"last":0.57499999999999996}]},{"osd":9,"last update":"Fri Feb 9 17:32:12 2024","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.6879999999999999,"15min":1.2070000000000001},"min":{"1min":0.45900000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":4.6529999999999996,"5min":7.7089999999999996,"15min":8.9190000000000005},"last":0.64900000000000002},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.355,"15min":1.093},"min":{"1min":0.44500000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":3.8090000000000002,"5min":13.218,"15min":13.218},"last":1.1000000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:32:42 2024","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.2210000000000001,"15min":1.105},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":6.1299999999999999,"5min":7.7709999999999999,"15min":13.244999999999999},"last":2.1970000000000001},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.397,"15min":1.1539999999999999},"min":{"1min":0.47399999999999998,"5min":0.42499999999999999,"15min":0.42099999999999999},"max":{"1min":6.1539999999999999,"5min":14.499000000000001,"15min":14.499000000000001},"last":2.2330000000000001}]},{"osd":11,"last update":"Fri Feb 9 17:31:52 2024","interfaces":[{"interface":"back","average":{"1min":1.6719999999999999,"5min":1.554,"15min":2.0379999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":5.931,"5min":13.186,"15min":13.186},"last":2.085},{"interface":"front","average":{"1min":1.214,"5min":1.6659999999999999,"15min":1.9510000000000001},"min":{"1min":0.503,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":4.3230000000000004,"5min":7.7919999999999998,"15min":7.7919999999999998},"last":2.2869999999999999}]}]},{"osd":3,"up_from":27,"seq":115964117157,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2207428,"kb_used_data":1965240,"kb_used_omap":1,"kb_used_meta":242174,"kb_avail":91537724,"statfs":{"total":95995035648,"available":93734629376,"internally_reserved":0,"allocated":2012405760,"data_stored":2012092609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":247986631},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:31:59 2024","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":2.1160000000000001,"15min":3.298},"min":{"1min":0.247,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":3.8879999999999999,"5min":19.475000000000001,"15min":19.475000000000001},"last":1.7649999999999999},{"interface":"front","average":{"1min":1.129,"5min":2.2130000000000001,"15min":3.2959999999999998},"min":{"1min":0.27900000000000003,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":6.7859999999999996,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.35399999999999998}]},{"osd":1,"last update":"Fri Feb 9 17:32:41 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":1.994,"15min":1.379},"min":{"1min":0.375,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":3.581,"5min":19.638999999999999,"15min":19.638999999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.65800000000000003,"5min":1.637,"15min":1.1020000000000001},"min":{"1min":0.27100000000000002,"5min":0.22500000000000001,"15min":0.20399999999999999},"max":{"1min":2.1219999999999999,"5min":14.661,"15min":14.661},"last":0.29999999999999999}]},{"osd":2,"last update":"Fri Feb 9 17:32:41 2024","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.6220000000000001,"15min":1.1950000000000001},"min":{"1min":0.315,"5min":0.247,"15min":0.224},"max":{"1min":5.968,"5min":12.651999999999999,"15min":14.914999999999999},"last":0.56000000000000005},{"interface":"front","average":{"1min":1.7170000000000001,"5min":1.6719999999999999,"15min":1.1339999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.218},"max":{"1min":7.9989999999999997,"5min":9.3089999999999993,"15min":9.3089999999999993},"last":0.46899999999999997}]},{"osd":4,"last update":"Fri Feb 9 17:31:59 2024","interfaces":[{"interface":"back","average":{"1min":1.806,"5min":2.0249999999999999,"15min":1.26},"min":{"1min":0.29099999999999998,"5min":0.26900000000000002,"15min":0.254},"max":{"1min":9.3510000000000009,"5min":10.762,"15min":10.762},"last":0.48799999999999999},{"interface":"front","average":{"1min":2.1030000000000002,"5min":1.964,"15min":1.157},"min":{"1min":0.30599999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":12.602,"5min":12.602,"15min":12.602},"last":0.54100000000000004}]},{"osd":5,"last update":"Fri Feb 9 17:32:27 2024","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":1.508,"15min":0.98799999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.27300000000000002},"max":{"1min":6.016,"5min":7.0110000000000001,"15min":7.0110000000000001},"last":3.964},{"interface":"front","average":{"1min":1.623,"5min":1.5609999999999999,"15min":1.0109999999999999},"min":{"1min":0.42899999999999999,"5min":0.36699999999999999,"15min":0.33300000000000002},"max":{"1min":6.0549999999999997,"5min":7.1600000000000001,"15min":7.1600000000000001},"last":3.9969999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:31:55 2024","interfaces":[{"interface":"back","average":{"1min":1.6140000000000001,"5min":1.8480000000000001,"15min":1.1339999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.32200000000000001},"max":{"1min":7.9169999999999998,"5min":19.582999999999998,"15min":19.582999999999998},"last":4.7119999999999997},{"interface":"front","average":{"1min":1.6399999999999999,"5min":1.446,"15min":0.94299999999999995},"min":{"1min":0.32200000000000001,"5min":0.30199999999999999,"15min":0.29499999999999998},"max":{"1min":7.9009999999999998,"5min":7.9009999999999998,"15min":7.9009999999999998},"last":4.702}]},{"osd":7,"last update":"Fri Feb 9 17:32:25 2024","interfaces":[{"interface":"back","average":{"1min":3.3559999999999999,"5min":2.2509999999999999,"15min":1.5209999999999999},"min":{"1min":0.36599999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":9.3770000000000007,"5min":10.416,"15min":10.791},"last":0.51800000000000002},{"interface":"front","average":{"1min":1.99,"5min":2.0659999999999998,"15min":1.2809999999999999},"min":{"1min":0.40300000000000002,"5min":0.375,"15min":0.37},"max":{"1min":7.8490000000000002,"5min":19.550999999999998,"15min":19.550999999999998},"last":4.1710000000000003}]},{"osd":8,"last update":"Fri Feb 9 17:26:30 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":3.7629999999999999,"5min":3.7629999999999999,"15min":3.7629999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":2.327,"5min":1.22,"15min":0.93000000000000005},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":10.167,"5min":10.167,"15min":10.167},"last":0.49099999999999999}]},{"osd":9,"last update":"Fri Feb 9 17:32:16 2024","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.4399999999999999,"15min":1.05},"min":{"1min":0.36299999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":3.9670000000000001,"5min":7.0810000000000004,"15min":7.0810000000000004},"last":0.64400000000000002},{"interface":"front","average":{"1min":1.0089999999999999,"5min":1.7729999999999999,"15min":1.3600000000000001},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":2.2759999999999998,"5min":13.717000000000001,"15min":13.717000000000001},"last":0.42899999999999999}]},{"osd":10,"last update":"Fri Feb 9 17:32:41 2024","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.762,"15min":1.276},"min":{"1min":0.48899999999999999,"5min":0.40899999999999997,"15min":0.39400000000000002},"max":{"1min":3.6739999999999999,"5min":19.611000000000001,"15min":19.611000000000001},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.88700000000000001,"5min":1.4490000000000001,"15min":1.1200000000000001},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":2.2389999999999999,"5min":7.1109999999999998,"15min":10.036},"last":0.52800000000000002}]},{"osd":11,"last update":"Fri Feb 9 17:31:59 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.5669999999999999,"15min":1.0820000000000001},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.38800000000000001},"max":{"1min":2.3319999999999999,"5min":7.1360000000000001,"15min":7.1360000000000001},"last":0.50600000000000001},{"interface":"front","average":{"1min":1.9299999999999999,"5min":2.0329999999999999,"15min":1.4099999999999999},"min":{"1min":0.443,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":9.4169999999999998,"5min":10.882999999999999,"15min":10.882999999999999},"last":0.69499999999999995}]}]},{"osd":4,"up_from":32,"seq":137438953632,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2931608,"kb_used_data":2729672,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":90813544,"statfs":{"total":95995035648,"available":92993069056,"internally_reserved":0,"allocated":2795184128,"data_stored":2794865802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":206764485},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:31:53 2024","interfaces":[{"interface":"back","average":{"1min":2.286,"5min":1.075,"15min":0.873},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":10.48,"5min":10.48,"15min":10.48},"last":0.374},{"interface":"front","average":{"1min":1.5740000000000001,"5min":1.01,"15min":0.91600000000000004},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":6.0049999999999999,"5min":6.0049999999999999,"15min":6.0049999999999999},"last":4.8289999999999997}]},{"osd":1,"last update":"Fri Feb 9 17:31:59 2024","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.6200000000000001,"15min":1.1599999999999999},"min":{"1min":0.316,"5min":0.255,"15min":0.246},"max":{"1min":4.6980000000000004,"5min":14.221,"15min":42.542999999999999},"last":4.8609999999999998},{"interface":"front","average":{"1min":1.96,"5min":1.5580000000000001,"15min":1.155},"min":{"1min":0.30299999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":9.2769999999999992,"5min":9.2769999999999992,"15min":41.813000000000002},"last":1.681}]},{"osd":2,"last update":"Fri Feb 9 17:28:51 2024","interfaces":[{"interface":"back","average":{"1min":2.536,"5min":1.546,"15min":0.91400000000000003},"min":{"1min":0.29999999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":8.8819999999999997,"5min":8.8819999999999997,"15min":8.8819999999999997},"last":0.622},{"interface":"front","average":{"1min":2.214,"5min":1.7330000000000001,"15min":0.97799999999999998},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":9.5039999999999996,"5min":9.5039999999999996,"15min":9.5039999999999996},"last":0.53800000000000003}]},{"osd":3,"last update":"Fri Feb 9 17:31:59 2024","interfaces":[{"interface":"back","average":{"1min":2.1949999999999998,"5min":2.1850000000000001,"15min":1.2450000000000001},"min":{"1min":0.30399999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":10.526,"5min":33.037999999999997,"15min":33.037999999999997},"last":0.34100000000000003},{"interface":"front","average":{"1min":1.373,"5min":2.0590000000000002,"15min":1.151},"min":{"1min":0.33500000000000002,"5min":0.314,"15min":0.28199999999999997},"max":{"1min":3.4089999999999998,"5min":33.054000000000002,"15min":33.054000000000002},"last":4.9400000000000004}]},{"osd":5,"last update":"Fri Feb 9 17:32:26 2024","interfaces":[{"interface":"back","average":{"1min":1.5960000000000001,"5min":1.196,"15min":0.749},"min":{"1min":0.27900000000000003,"5min":0.222,"15min":0.222},"max":{"1min":8.5660000000000007,"5min":8.5660000000000007,"15min":8.5660000000000007},"last":1.6459999999999999},{"interface":"front","average":{"1min":1.673,"5min":1.294,"15min":0.83399999999999996},"min":{"1min":0.377,"5min":0.29199999999999998,"15min":0.26000000000000001},"max":{"1min":8.5879999999999992,"5min":8.5879999999999992,"15min":8.5879999999999992},"last":1.7370000000000001}]},{"osd":6,"last update":"Fri Feb 9 17:31:59 2024","interfaces":[{"interface":"back","average":{"1min":2.3330000000000002,"5min":1.3280000000000001,"15min":0.84499999999999997},"min":{"1min":0.28000000000000003,"5min":0.247,"15min":0.247},"max":{"1min":18.09,"5min":18.09,"15min":18.09},"last":4.8810000000000002},{"interface":"front","average":{"1min":2.4300000000000002,"5min":1.294,"15min":0.80300000000000005},"min":{"1min":0.33800000000000002,"5min":0.30099999999999999,"15min":0.28000000000000003},"max":{"1min":18.126000000000001,"5min":18.126000000000001,"15min":18.126000000000001},"last":4.9029999999999996}]},{"osd":7,"last update":"Fri Feb 9 17:32:26 2024","interfaces":[{"interface":"back","average":{"1min":1.663,"5min":1.361,"15min":0.96499999999999997},"min":{"1min":0.35099999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":8.3439999999999994,"5min":8.3439999999999994,"15min":15.223000000000001},"last":1.7050000000000001},{"interface":"front","average":{"1min":2.234,"5min":1.5489999999999999,"15min":0.996},"min":{"1min":0.379,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":5.2699999999999996,"5min":19.047000000000001,"15min":19.047000000000001},"last":0.60299999999999998}]},{"osd":8,"last update":"Fri Feb 9 17:32:26 2024","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.3480000000000001,"15min":1.264},"min":{"1min":0.41799999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":8.4160000000000004,"5min":8.4160000000000004,"15min":8.4160000000000004},"last":3.8519999999999999},{"interface":"front","average":{"1min":1.325,"5min":1.2450000000000001,"15min":1.087},"min":{"1min":0.45100000000000001,"5min":0.308,"15min":0.308},"max":{"1min":8.5169999999999995,"5min":8.5169999999999995,"15min":8.5169999999999995},"last":1.784}]},{"osd":9,"last update":"Fri Feb 9 17:32:11 2024","interfaces":[{"interface":"back","average":{"1min":1.101,"5min":1.294,"15min":0.997},"min":{"1min":0.45000000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":3.7309999999999999,"5min":5.1929999999999996,"15min":5.1929999999999996},"last":1.7609999999999999},{"interface":"front","average":{"1min":2.25,"5min":1.3100000000000001,"15min":1},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":10.555,"5min":10.555,"15min":10.555},"last":0.67200000000000004}]},{"osd":10,"last update":"Fri Feb 9 17:31:43 2024","interfaces":[{"interface":"back","average":{"1min":2.1389999999999998,"5min":1.3979999999999999,"15min":1.0429999999999999},"min":{"1min":0.46600000000000003,"5min":0.378,"15min":0.378},"max":{"1min":4.6890000000000001,"5min":5.2469999999999999,"15min":5.2469999999999999},"last":1.8069999999999999},{"interface":"front","average":{"1min":2.149,"5min":1.3260000000000001,"15min":1.0189999999999999},"min":{"1min":0.40500000000000003,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":10.597,"5min":10.597,"15min":10.597},"last":1.8580000000000001}]},{"osd":11,"last update":"Fri Feb 9 17:32:09 2024","interfaces":[{"interface":"back","average":{"1min":1.514,"5min":1.4790000000000001,"15min":1.369},"min":{"1min":0.48199999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":3.786,"5min":6.3689999999999998,"15min":6.3689999999999998},"last":1.833},{"interface":"front","average":{"1min":1.964,"5min":1.399,"15min":1.2190000000000001},"min":{"1min":0.42799999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":10.625,"5min":10.625,"15min":10.625},"last":0.64600000000000002}]}]},{"osd":5,"up_from":40,"seq":171798691994,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1667744,"kb_used_data":1556052,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":92077408,"statfs":{"total":95995035648,"available":94287265792,"internally_reserved":0,"allocated":1593397248,"data_stored":1593071298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:32:25 2024","interfaces":[{"interface":"back","average":{"1min":1.8540000000000001,"5min":1.653,"15min":1.0900000000000001},"min":{"1min":0.318,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":7.7160000000000002,"5min":8.7690000000000001,"15min":11.923},"last":4.2249999999999996},{"interface":"front","average":{"1min":2.4089999999999998,"5min":1.8260000000000001,"15min":1.032},"min":{"1min":0.34499999999999997,"5min":0.24299999999999999,"15min":0.224},"max":{"1min":14.082000000000001,"5min":14.082000000000001,"15min":14.082000000000001},"last":0.56299999999999994}]},{"osd":1,"last update":"Fri Feb 9 17:32:25 2024","interfaces":[{"interface":"back","average":{"1min":3.1659999999999999,"5min":2.1339999999999999,"15min":1.371},"min":{"1min":0.38,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":14.032999999999999,"5min":16.896000000000001,"15min":29.715},"last":0.49099999999999999},{"interface":"front","average":{"1min":1.145,"5min":1.8220000000000001,"15min":1.2170000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":7.2750000000000004,"5min":16.878,"15min":29.969999999999999},"last":4.4889999999999999}]},{"osd":2,"last update":"Fri Feb 9 17:32:25 2024","interfaces":[{"interface":"back","average":{"1min":1.4370000000000001,"5min":1.734,"15min":1.167},"min":{"1min":0.33100000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":10.5,"5min":10.5,"15min":10.5},"last":4.5640000000000001},{"interface":"front","average":{"1min":2.6269999999999998,"5min":1.907,"15min":1.244},"min":{"1min":0.376,"5min":0.318,"15min":0.318},"max":{"1min":14.122,"5min":14.122,"15min":14.122},"last":0.443}]},{"osd":3,"last update":"Fri Feb 9 17:32:25 2024","interfaces":[{"interface":"back","average":{"1min":1.885,"5min":1.843,"15min":1.202},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":7.2270000000000003,"5min":10.587,"15min":10.587},"last":4.5300000000000002},{"interface":"front","average":{"1min":2.4769999999999999,"5min":1.976,"15min":1.1579999999999999},"min":{"1min":0.30499999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":14.154999999999999,"5min":14.154999999999999,"15min":14.154999999999999},"last":0.54100000000000004}]},{"osd":4,"last update":"Fri Feb 9 17:32:38 2024","interfaces":[{"interface":"back","average":{"1min":1.9990000000000001,"5min":1.2949999999999999,"15min":0.746},"min":{"1min":0.31900000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":8.7260000000000009,"5min":8.7260000000000009,"15min":8.7260000000000009},"last":6.25},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.298,"15min":0.80700000000000005},"min":{"1min":0.38500000000000001,"5min":0.28100000000000003,"15min":0.27000000000000002},"max":{"1min":4.8209999999999997,"5min":8.6349999999999998,"15min":8.6349999999999998},"last":5.306}]},{"osd":6,"last update":"Fri Feb 9 17:31:58 2024","interfaces":[{"interface":"back","average":{"1min":2.895,"5min":1.3720000000000001,"15min":0.85599999999999998},"min":{"1min":0.46700000000000003,"5min":0.30499999999999999,"15min":0.28999999999999998},"max":{"1min":8.6080000000000005,"5min":8.9250000000000007,"15min":8.9250000000000007},"last":6.1399999999999997},{"interface":"front","average":{"1min":2.371,"5min":1.3680000000000001,"15min":0.82699999999999996},"min":{"1min":0.41699999999999998,"5min":0.29399999999999998,"15min":0.27200000000000002},"max":{"1min":13.859999999999999,"5min":13.859999999999999,"15min":13.859999999999999},"last":0.51700000000000002}]},{"osd":7,"last update":"Fri Feb 9 17:32:25 2024","interfaces":[{"interface":"back","average":{"1min":2.0670000000000002,"5min":1.339,"15min":0.94399999999999995},"min":{"1min":0.40300000000000002,"5min":0.32200000000000001,"15min":0.32100000000000001},"max":{"1min":8.8100000000000005,"5min":8.8100000000000005,"15min":18.626999999999999},"last":4.5800000000000001},{"interface":"front","average":{"1min":2.867,"5min":1.7809999999999999,"15min":1.095},"min":{"1min":0.34499999999999997,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":19.044,"5min":19.044,"15min":19.044},"last":0.63200000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:26:36 2024","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":0.91000000000000003,"15min":0.78000000000000003},"min":{"1min":0.39400000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":5.556,"5min":5.556,"15min":5.556},"last":0.40300000000000002},{"interface":"front","average":{"1min":1.3720000000000001,"5min":0.92600000000000005,"15min":0.82999999999999996},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":5.9000000000000004,"5min":5.9000000000000004,"15min":5.9000000000000004},"last":0.36799999999999999}]},{"osd":9,"last update":"Fri Feb 9 17:32:04 2024","interfaces":[{"interface":"back","average":{"1min":2.1549999999999998,"5min":1.306,"15min":0.998},"min":{"1min":0.53800000000000003,"5min":0.371,"15min":0.35999999999999999},"max":{"1min":8.4280000000000008,"5min":8.4280000000000008,"15min":8.4280000000000008},"last":4.6310000000000002},{"interface":"front","average":{"1min":2.1299999999999999,"5min":1.214,"15min":1.0089999999999999},"min":{"1min":0.54500000000000004,"5min":0.312,"15min":0.312},"max":{"1min":8.3689999999999998,"5min":8.3689999999999998,"15min":8.3689999999999998},"last":4.601}]},{"osd":10,"last update":"Fri Feb 9 17:32:38 2024","interfaces":[{"interface":"back","average":{"1min":2.5670000000000002,"5min":1.579,"15min":1.1519999999999999},"min":{"1min":0.58599999999999997,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":14.218,"5min":14.218,"15min":14.218},"last":0.58599999999999997},{"interface":"front","average":{"1min":2.5960000000000001,"5min":1.637,"15min":1.1899999999999999},"min":{"1min":0.57499999999999996,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":14.186999999999999,"5min":14.186999999999999,"15min":14.186999999999999},"last":0.60699999999999998}]},{"osd":11,"last update":"Fri Feb 9 17:32:08 2024","interfaces":[{"interface":"back","average":{"1min":2.056,"5min":1.337,"15min":1.2729999999999999},"min":{"1min":0.46500000000000002,"5min":0.441,"15min":0.441},"max":{"1min":8.3979999999999997,"5min":8.3979999999999997,"15min":8.3979999999999997},"last":4.6559999999999997},{"interface":"front","average":{"1min":1.9159999999999999,"5min":1.2949999999999999,"15min":0.98799999999999999},"min":{"1min":0.61099999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":7.4249999999999998,"5min":7.4249999999999998,"15min":7.4249999999999998},"last":5.1260000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2409562112,"data_stored":2409562112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1581121536,"data_stored":1581121536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":389349376,"data_stored":389349376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2005598208,"data_stored":2005598208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2783707136,"data_stored":2783707136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1590431744,"data_stored":1590427672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1963261952,"data_stored":1963261952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1624248320,"data_stored":1624244248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":808714240,"data_stored":808714240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1177227264,"data_stored":1177223192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":397803520,"data_stored":397803520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":3179282432,"data_stored":3179282432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-09T17:32:48.988 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-02-09T17:32:48.988 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd pool set .mgr pg_num 21 2024-02-09T17:32:49.056 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:32:49.127 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:49.148 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:32:49.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:49.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:32:49.255 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:32:49.305 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:49.336 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:32:49.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:49.346 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:32:49.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:49 smithi162 ceph-mon[44304]: pgmap v739: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 23 active+clean; 5.9 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 442 op/s; 2091/289017 objects degraded (0.723%); 24216/289017 objects misplaced (8.379%); 6.7 MiB/s, 108 objects/s recovering 2024-02-09T17:32:49.447 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:32:49.511 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:49.539 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:32:49.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:49.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:32:49.646 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:32:49.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:49 smithi179 ceph-mon[43975]: pgmap v739: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 23 active+clean; 5.9 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 442 op/s; 2091/289017 objects degraded (0.723%); 24216/289017 objects misplaced (8.379%); 6.7 MiB/s, 108 objects/s recovering 2024-02-09T17:32:49.715 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:49.736 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:49 smithi099 ceph-mon[39420]: pgmap v739: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 23 active+clean; 5.9 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 442 op/s; 2091/289017 objects degraded (0.723%); 24216/289017 objects misplaced (8.379%); 6.7 MiB/s, 108 objects/s recovering 2024-02-09T17:32:49.753 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:32:49.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:49.767 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:32:49.868 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:32:49.912 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:49.943 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:32:49.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:49.946 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:32:50.047 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:32:50.101 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:50.126 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:32:50.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:50.140 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:32:50.241 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:32:50.273 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:50.416 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:32:50.442 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:32:50.468 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:32:50.492 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:32:50.521 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:32:50.561 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:32:50.601 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:32:50.629 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:32:50.658 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:32:50.686 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:32:50.711 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:32:50.733 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:32:50.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:50.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:32:50.739 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:50 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1799359724' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:32:50.743 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:32:50.746 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:32:50.746 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:32:50.746 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:32:50.746 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:32:50.746 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:32:50.746 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:32:50.746 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:32:50.746 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:32:50.746 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:32:50.746 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:32:50.746 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:32:50.746 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:32:50.840 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:32:50.897 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:50 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1799359724' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:32:50.902 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:50.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:50 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1799359724' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:32:50.949 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:32:50.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:50.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:32:51.053 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:32:51.113 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:51.145 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:32:51.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:51.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:32:51.254 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:32:51.292 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:51.743 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:32:51.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:51 smithi099 ceph-mon[39420]: pgmap v740: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 23 active+clean; 5.8 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 566 op/s; 2247/283791 objects degraded (0.792%); 23541/283791 objects misplaced (8.295%); 9.3 MiB/s, 149 objects/s recovering 2024-02-09T17:32:51.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:51 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 2091/289017 objects degraded (0.723%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:32:51.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:51 smithi099 ceph-mon[39420]: osdmap e258: 12 total, 12 up, 11 in 2024-02-09T17:32:51.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:51.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:32:51.783 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-09T17:32:51.849 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:32:51.901 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:51 smithi162 ceph-mon[44304]: pgmap v740: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 23 active+clean; 5.8 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 566 op/s; 2247/283791 objects degraded (0.792%); 23541/283791 objects misplaced (8.295%); 9.3 MiB/s, 149 objects/s recovering 2024-02-09T17:32:51.901 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:51 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 2091/289017 objects degraded (0.723%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:32:51.901 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:51 smithi162 ceph-mon[44304]: osdmap e258: 12 total, 12 up, 11 in 2024-02-09T17:32:51.903 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:51.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:51 smithi179 ceph-mon[43975]: pgmap v740: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 23 active+clean; 5.8 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 566 op/s; 2247/283791 objects degraded (0.792%); 23541/283791 objects misplaced (8.295%); 9.3 MiB/s, 149 objects/s recovering 2024-02-09T17:32:51.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:51 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 2091/289017 objects degraded (0.723%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:32:51.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:51 smithi179 ceph-mon[43975]: osdmap e258: 12 total, 12 up, 11 in 2024-02-09T17:32:51.955 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:32:51.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:51.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:32:52.063 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:32:52.129 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:52.156 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:32:52.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:52.162 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:32:52.263 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:32:52.325 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:52.355 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:32:52.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:52.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:32:52.465 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:32:52.501 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:52.854 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:32:52.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:52.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:32:52.963 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:32:52.993 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:53.030 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:32:53.030 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:32:53.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:53.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:32:53.070 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:32:53.139 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:32:53.167 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:53.369 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:32:53.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:53.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:32:53.473 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:32:53.540 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:53.574 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:32:53.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:53.577 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:32:53.633 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is unset 2024-02-09T17:32:53.678 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:32:53.712 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:53.768 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:32:53.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:53.772 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:32:53.873 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:32:53.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:53 smithi179 ceph-mon[43975]: pgmap v742: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 23 active+clean; 5.7 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 685 op/s; 2247/279768 objects degraded (0.803%); 23541/279768 objects misplaced (8.414%); 9.3 MiB/s, 149 objects/s recovering 2024-02-09T17:32:53.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:53 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1179873203' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:32:53.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:53 smithi162 ceph-mon[44304]: pgmap v742: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 23 active+clean; 5.7 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 685 op/s; 2247/279768 objects degraded (0.803%); 23541/279768 objects misplaced (8.414%); 9.3 MiB/s, 149 objects/s recovering 2024-02-09T17:32:53.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:53 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1179873203' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:32:53.949 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:53.973 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:32:53.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:53.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:32:53.976 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:53 smithi099 ceph-mon[39420]: pgmap v742: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 23 active+clean; 5.7 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 685 op/s; 2247/279768 objects degraded (0.803%); 23541/279768 objects misplaced (8.414%); 9.3 MiB/s, 149 objects/s recovering 2024-02-09T17:32:53.976 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:53 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1179873203' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:32:54.076 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:32:54.106 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:54.307 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:32:54.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:54.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:32:54.412 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:32:54.461 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:54.493 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:32:54.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:54.497 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:32:54.598 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:32:54.696 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:54.718 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:32:54.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:54.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:32:54.826 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:32:54.874 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:32:54.875 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:54.899 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:54 smithi099 ceph-mon[39420]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:32:54.899 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:54 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1179873203' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:32:54.899 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:54 smithi099 ceph-mon[39420]: osdmap e259: 12 total, 12 up, 11 in 2024-02-09T17:32:54.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:54 smithi179 ceph-mon[43975]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:32:54.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:54 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1179873203' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:32:54.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:54 smithi179 ceph-mon[43975]: osdmap e259: 12 total, 12 up, 11 in 2024-02-09T17:32:54.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:54 smithi162 ceph-mon[44304]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:32:54.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:54 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1179873203' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:32:54.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:54 smithi162 ceph-mon[44304]: osdmap e259: 12 total, 12 up, 11 in 2024-02-09T17:32:54.927 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:32:54.936 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:54.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:32:55.037 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:32:55.110 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:55.148 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:32:55.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:55.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:32:55.252 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:32:55.281 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:55.300 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:32:55.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:55.304 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:32:55.404 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:32:55.450 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:55.478 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:32:55.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:55.482 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:32:55.583 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:32:55.629 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:55.662 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:32:55.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:55.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:32:55.769 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:32:55.796 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:55 smithi099 ceph-mon[39420]: pgmap v744: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 23 active+clean; 5.7 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 565 op/s; 2247/278691 objects degraded (0.806%); 23541/278691 objects misplaced (8.447%); 4.7 MiB/s, 74 objects/s recovering 2024-02-09T17:32:55.799 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:55.926 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:32:55.927 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:55 smithi179 ceph-mon[43975]: pgmap v744: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 23 active+clean; 5.7 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 565 op/s; 2247/278691 objects degraded (0.806%); 23541/278691 objects misplaced (8.447%); 4.7 MiB/s, 74 objects/s recovering 2024-02-09T17:32:55.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:55.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:32:55.934 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:55 smithi162 ceph-mon[44304]: pgmap v744: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 23 active+clean; 5.7 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 565 op/s; 2247/278691 objects degraded (0.806%); 23541/278691 objects misplaced (8.447%); 4.7 MiB/s, 74 objects/s recovering 2024-02-09T17:32:56.030 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:32:56.075 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:56.117 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:32:56.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:56.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:32:56.224 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:32:56.254 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:56.279 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:32:56.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:56.283 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:32:56.383 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:32:56.435 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:56.462 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:32:56.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:56.474 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:32:56.551 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:32:56.574 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:32:56.621 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:56.642 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:56 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 2247/278691 objects degraded (0.806%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:32:56.777 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:32:56.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:56.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:32:56.780 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:56 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 2247/278691 objects degraded (0.806%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:32:56.880 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:32:56.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:56 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 2247/278691 objects degraded (0.806%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:32:56.930 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:56.958 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:32:56.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:56.962 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:32:57.063 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:32:57.097 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:57.469 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:32:57.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:57.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:32:57.581 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:32:57.615 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:57.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:57 smithi099 ceph-mon[39420]: 3.5 scrub starts 2024-02-09T17:32:57.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:57 smithi099 ceph-mon[39420]: pgmap v745: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 23 active+clean; 5.6 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 620 op/s; 2505/274128 objects degraded (0.914%); 22776/274128 objects misplaced (8.309%); 8.0 MiB/s, 128 objects/s recovering 2024-02-09T17:32:57.871 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:32:57.872 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 20, 2024-02-09T17:32:57.872 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 600, 2024-02-09T17:32:57.872 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [ 2024-02-09T17:32:57.872 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.872 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:68200 2.4 2:2679e32a:::benchmark_data_smithi179_77138_object68199:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-02-09T17:32:57.872 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:25:20.117653+0000", 2024-02-09T17:32:57.872 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 457.75002263300001, 2024-02-09T17:32:57.872 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.21735884, 2024-02-09T17:32:57.872 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:32:57.872 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:32:57.872 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:32:57.872 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:32:57.872 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:32:57.873 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 68200 2024-02-09T17:32:57.873 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.873 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:32:57.873 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.873 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:32:57.873 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.117653+0000", 2024-02-09T17:32:57.873 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:57.873 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.873 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.873 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:32:57.873 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.117653+0000", 2024-02-09T17:32:57.873 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:57.873 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.874 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.874 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:32:57.874 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.117654+0000", 2024-02-09T17:32:57.874 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.2399999999999998e-07 2024-02-09T17:32:57.874 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.874 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.874 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:32:57.874 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.117749+0000", 2024-02-09T17:32:57.874 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.5290000000000004e-05 2024-02-09T17:32:57.874 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.874 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.874 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:32:57.874 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.117759+0000", 2024-02-09T17:32:57.875 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.6889999999999999e-06 2024-02-09T17:32:57.875 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.875 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.875 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:32:57.875 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.117766+0000", 2024-02-09T17:32:57.875 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.3780000000000003e-06 2024-02-09T17:32:57.875 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.875 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.875 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:57.875 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.118175+0000", 2024-02-09T17:32:57.875 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000408238 2024-02-09T17:32:57.875 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.875 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.875 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:32:57.876 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.118278+0000", 2024-02-09T17:32:57.876 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000102823 2024-02-09T17:32:57.876 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.876 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.876 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,10", 2024-02-09T17:32:57.876 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.118405+0000", 2024-02-09T17:32:57.876 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000127632 2024-02-09T17:32:57.876 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.876 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.876 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:57.876 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.123073+0000", 2024-02-09T17:32:57.876 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.004668312 2024-02-09T17:32:57.876 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.876 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.877 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:32:57.877 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.123123+0000", 2024-02-09T17:32:57.877 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.9954999999999999e-05 2024-02-09T17:32:57.877 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.877 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.877 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:57.877 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.334940+0000", 2024-02-09T17:32:57.877 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.211817015 2024-02-09T17:32:57.877 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.877 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.877 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:32:57.877 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.334976+0000", 2024-02-09T17:32:57.877 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.5244999999999999e-05 2024-02-09T17:32:57.878 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.878 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.878 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:32:57.878 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.335012+0000", 2024-02-09T17:32:57.878 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.6638999999999999e-05 2024-02-09T17:32:57.878 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:57.878 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:32:57.878 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:57.878 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.878 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.878 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:68207 2.4 2:2ed4664d:::benchmark_data_smithi179_77138_object68206:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-02-09T17:32:57.878 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:25:20.122888+0000", 2024-02-09T17:32:57.878 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 457.74478822700002, 2024-02-09T17:32:57.878 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.212263546, 2024-02-09T17:32:57.879 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:32:57.879 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:32:57.879 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:32:57.879 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:32:57.879 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:32:57.879 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 68207 2024-02-09T17:32:57.879 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.879 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:32:57.879 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.879 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:32:57.879 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.122888+0000", 2024-02-09T17:32:57.879 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:57.879 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.879 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.880 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:32:57.880 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.122888+0000", 2024-02-09T17:32:57.880 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:57.880 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.880 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.880 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:32:57.880 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.122889+0000", 2024-02-09T17:32:57.880 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4619999999999999e-06 2024-02-09T17:32:57.880 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.880 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.880 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:32:57.880 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.122973+0000", 2024-02-09T17:32:57.880 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.3955999999999994e-05 2024-02-09T17:32:57.880 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.880 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.881 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:32:57.881 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.122984+0000", 2024-02-09T17:32:57.881 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1109e-05 2024-02-09T17:32:57.881 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.881 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.881 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:32:57.881 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.122990+0000", 2024-02-09T17:32:57.881 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3140000000000003e-06 2024-02-09T17:32:57.881 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.881 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.881 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:57.881 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.123150+0000", 2024-02-09T17:32:57.881 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00016015499999999999 2024-02-09T17:32:57.881 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.882 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.882 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:32:57.882 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.123277+0000", 2024-02-09T17:32:57.882 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012703399999999999 2024-02-09T17:32:57.882 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.882 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.882 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,10", 2024-02-09T17:32:57.882 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.123415+0000", 2024-02-09T17:32:57.882 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013789999999999999 2024-02-09T17:32:57.882 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.882 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.882 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:32:57.882 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.125510+0000", 2024-02-09T17:32:57.883 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.002095501 2024-02-09T17:32:57.883 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.883 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.883 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:57.883 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.133075+0000", 2024-02-09T17:32:57.883 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0075643739999999996 2024-02-09T17:32:57.883 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.883 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.883 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:57.883 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.335060+0000", 2024-02-09T17:32:57.883 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.201985306 2024-02-09T17:32:57.883 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.883 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.884 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:32:57.884 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.335117+0000", 2024-02-09T17:32:57.884 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6991999999999999e-05 2024-02-09T17:32:57.884 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.884 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.884 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:32:57.884 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.335151+0000", 2024-02-09T17:32:57.884 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.4443000000000002e-05 2024-02-09T17:32:57.884 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:57.884 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:32:57.884 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:57.884 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.884 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.884 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:68223 2.4 2:2e2ca52e:::benchmark_data_smithi179_77138_object68222:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-02-09T17:32:57.885 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:25:20.133614+0000", 2024-02-09T17:32:57.885 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 457.73406228099998, 2024-02-09T17:32:57.885 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20952131800000001, 2024-02-09T17:32:57.885 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:32:57.885 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:32:57.885 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:32:57.885 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:32:57.885 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:32:57.885 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 68223 2024-02-09T17:32:57.885 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.885 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:32:57.885 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.885 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:32:57.885 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.133614+0000", 2024-02-09T17:32:57.885 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:57.886 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.886 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.886 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:32:57.886 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.133614+0000", 2024-02-09T17:32:57.886 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:57.886 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.886 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.886 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:32:57.886 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.133615+0000", 2024-02-09T17:32:57.886 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.457e-06 2024-02-09T17:32:57.886 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.886 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.886 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:32:57.887 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.133647+0000", 2024-02-09T17:32:57.887 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.1962000000000003e-05 2024-02-09T17:32:57.887 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.887 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.887 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:32:57.887 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.133657+0000", 2024-02-09T17:32:57.887 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.6770000000000007e-06 2024-02-09T17:32:57.887 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.887 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.887 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:32:57.887 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.133664+0000", 2024-02-09T17:32:57.887 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.7889999999999997e-06 2024-02-09T17:32:57.887 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.887 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.888 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:57.888 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.136487+0000", 2024-02-09T17:32:57.888 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.002823187 2024-02-09T17:32:57.888 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.888 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.888 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:32:57.888 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.136596+0000", 2024-02-09T17:32:57.888 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00010904999999999999 2024-02-09T17:32:57.888 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.888 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.888 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,10", 2024-02-09T17:32:57.888 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.136693+0000", 2024-02-09T17:32:57.888 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.6935000000000006e-05 2024-02-09T17:32:57.888 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.889 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.889 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:32:57.889 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.139831+0000", 2024-02-09T17:32:57.889 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0031380140000000002 2024-02-09T17:32:57.889 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.889 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.889 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:57.889 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.140618+0000", 2024-02-09T17:32:57.889 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00078751199999999998 2024-02-09T17:32:57.889 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.889 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.889 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:57.889 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.343024+0000", 2024-02-09T17:32:57.890 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20240518599999999 2024-02-09T17:32:57.890 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.890 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.890 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:32:57.890 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.343090+0000", 2024-02-09T17:32:57.890 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.6347999999999996e-05 2024-02-09T17:32:57.890 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.890 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.890 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:32:57.890 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.343135+0000", 2024-02-09T17:32:57.890 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.5201000000000003e-05 2024-02-09T17:32:57.890 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:57.890 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:32:57.890 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:57.890 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.891 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.891 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:68269 2.4 2:242c6c34:::benchmark_data_smithi179_77138_object68268:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-02-09T17:32:57.891 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:25:20.161075+0000", 2024-02-09T17:32:57.891 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 457.70660075400002, 2024-02-09T17:32:57.891 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.18840509699999999, 2024-02-09T17:32:57.891 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:32:57.891 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:32:57.891 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:32:57.891 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:32:57.891 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:32:57.891 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 68269 2024-02-09T17:32:57.891 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.891 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:32:57.891 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.892 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:32:57.892 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.161075+0000", 2024-02-09T17:32:57.892 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:57.892 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.892 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.892 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:32:57.892 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.161075+0000", 2024-02-09T17:32:57.892 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:57.892 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.892 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.892 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:32:57.892 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.161076+0000", 2024-02-09T17:32:57.892 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.82e-07 2024-02-09T17:32:57.893 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.893 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.893 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:32:57.893 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.161104+0000", 2024-02-09T17:32:57.893 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.7895999999999999e-05 2024-02-09T17:32:57.893 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.893 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.893 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:32:57.893 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.161111+0000", 2024-02-09T17:32:57.893 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.9909999999999996e-06 2024-02-09T17:32:57.893 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.893 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.893 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:32:57.893 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.161117+0000", 2024-02-09T17:32:57.893 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7389999999999996e-06 2024-02-09T17:32:57.894 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.894 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.894 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:57.894 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.161166+0000", 2024-02-09T17:32:57.894 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.9446000000000001e-05 2024-02-09T17:32:57.894 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.894 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.894 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:32:57.894 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.161275+0000", 2024-02-09T17:32:57.894 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00010888200000000001 2024-02-09T17:32:57.894 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.894 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.894 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,10", 2024-02-09T17:32:57.894 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.163797+0000", 2024-02-09T17:32:57.894 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0025216909999999999 2024-02-09T17:32:57.895 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.895 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.895 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:32:57.895 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.167743+0000", 2024-02-09T17:32:57.895 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0039468070000000001 2024-02-09T17:32:57.895 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.895 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.895 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:57.895 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.167796+0000", 2024-02-09T17:32:57.895 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3047000000000002e-05 2024-02-09T17:32:57.895 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.895 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.895 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:57.895 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.349342+0000", 2024-02-09T17:32:57.896 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.18154573700000001 2024-02-09T17:32:57.896 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.896 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.896 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:32:57.896 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.349417+0000", 2024-02-09T17:32:57.896 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.4997e-05 2024-02-09T17:32:57.896 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.896 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.896 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:32:57.896 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.349480+0000", 2024-02-09T17:32:57.896 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.3281999999999995e-05 2024-02-09T17:32:57.896 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:57.896 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:32:57.897 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:57.897 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.897 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.897 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:68274 2.4 2:28e1b85d:::benchmark_data_smithi179_77138_object68273:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-02-09T17:32:57.897 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:25:20.162287+0000", 2024-02-09T17:32:57.897 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 457.70538908999998, 2024-02-09T17:32:57.897 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.187309533, 2024-02-09T17:32:57.897 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:32:57.897 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:32:57.897 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:32:57.897 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:32:57.897 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:32:57.897 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 68274 2024-02-09T17:32:57.898 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.898 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:32:57.898 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.898 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:32:57.898 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.162287+0000", 2024-02-09T17:32:57.898 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:57.898 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.898 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.898 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:32:57.898 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.162287+0000", 2024-02-09T17:32:57.898 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:57.898 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.898 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.898 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:32:57.899 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.162288+0000", 2024-02-09T17:32:57.899 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.4799999999999997e-07 2024-02-09T17:32:57.899 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.899 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.899 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:32:57.899 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.162403+0000", 2024-02-09T17:32:57.899 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000115096 2024-02-09T17:32:57.899 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.899 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.899 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:32:57.899 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.162413+0000", 2024-02-09T17:32:57.899 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.5210000000000008e-06 2024-02-09T17:32:57.899 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.900 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.900 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:32:57.900 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.162418+0000", 2024-02-09T17:32:57.900 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6230000000000002e-06 2024-02-09T17:32:57.900 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.900 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.900 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:57.900 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.165950+0000", 2024-02-09T17:32:57.900 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0035316620000000001 2024-02-09T17:32:57.900 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.900 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.900 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:32:57.900 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.166048+0000", 2024-02-09T17:32:57.901 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.8302999999999997e-05 2024-02-09T17:32:57.901 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.901 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.901 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,10", 2024-02-09T17:32:57.901 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.166136+0000", 2024-02-09T17:32:57.901 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.7918999999999996e-05 2024-02-09T17:32:57.901 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.901 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.901 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:32:57.901 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.169470+0000", 2024-02-09T17:32:57.901 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0033337789999999998 2024-02-09T17:32:57.901 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.901 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.901 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:57.901 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.170179+0000", 2024-02-09T17:32:57.902 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00070950899999999999 2024-02-09T17:32:57.902 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.902 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.902 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:57.902 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.349526+0000", 2024-02-09T17:32:57.902 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.17934718099999999 2024-02-09T17:32:57.902 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.902 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.902 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:32:57.902 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.349577+0000", 2024-02-09T17:32:57.902 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.0039e-05 2024-02-09T17:32:57.902 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.902 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.902 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:32:57.903 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.349597+0000", 2024-02-09T17:32:57.903 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0052999999999999e-05 2024-02-09T17:32:57.903 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:57.903 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:32:57.903 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:57.903 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.903 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.903 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:68281 2.4 2:276f5379:::benchmark_data_smithi179_77138_object68280:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-02-09T17:32:57.903 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:25:20.166266+0000", 2024-02-09T17:32:57.903 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 457.70140989200002, 2024-02-09T17:32:57.903 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.184377706, 2024-02-09T17:32:57.903 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:32:57.903 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:32:57.903 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:32:57.904 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:32:57.904 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:32:57.904 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 68281 2024-02-09T17:32:57.904 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.904 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:32:57.904 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.904 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:32:57.904 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.166266+0000", 2024-02-09T17:32:57.904 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:57.904 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.904 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.904 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:32:57.904 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.166266+0000", 2024-02-09T17:32:57.904 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:57.905 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.905 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.905 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:32:57.905 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.166267+0000", 2024-02-09T17:32:57.905 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1000000000000001e-06 2024-02-09T17:32:57.905 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.905 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.905 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:32:57.905 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.166369+0000", 2024-02-09T17:32:57.905 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00010189200000000001 2024-02-09T17:32:57.905 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.905 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.905 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:32:57.905 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.166378+0000", 2024-02-09T17:32:57.906 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.1309999999999995e-06 2024-02-09T17:32:57.906 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.906 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.906 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:32:57.906 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.166384+0000", 2024-02-09T17:32:57.906 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7189999999999998e-06 2024-02-09T17:32:57.906 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.906 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.906 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:57.906 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.167828+0000", 2024-02-09T17:32:57.906 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.001444243 2024-02-09T17:32:57.906 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.906 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.906 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:32:57.906 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.167938+0000", 2024-02-09T17:32:57.907 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000109234 2024-02-09T17:32:57.907 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.907 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.907 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,10", 2024-02-09T17:32:57.907 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.168026+0000", 2024-02-09T17:32:57.907 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.8317000000000002e-05 2024-02-09T17:32:57.907 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.907 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.907 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:32:57.907 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.169600+0000", 2024-02-09T17:32:57.907 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.001574102 2024-02-09T17:32:57.907 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.907 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.907 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:57.908 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.170413+0000", 2024-02-09T17:32:57.908 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00081325600000000005 2024-02-09T17:32:57.908 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.908 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.908 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:57.908 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.350557+0000", 2024-02-09T17:32:57.908 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.18014343499999999 2024-02-09T17:32:57.908 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.908 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.908 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:32:57.908 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.350610+0000", 2024-02-09T17:32:57.908 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3353000000000002e-05 2024-02-09T17:32:57.908 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.908 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.909 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:32:57.909 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.350644+0000", 2024-02-09T17:32:57.909 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.3924000000000002e-05 2024-02-09T17:32:57.909 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:57.909 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:32:57.909 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:57.909 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.909 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.909 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.14790.0:68287 2.4 2:24b8d849:::benchmark_data_smithi179_77138_object68286:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-02-09T17:32:57.909 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:25:20.172962+0000", 2024-02-09T17:32:57.909 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 457.69471384100001, 2024-02-09T17:32:57.909 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.177793757, 2024-02-09T17:32:57.909 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:32:57.909 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:32:57.910 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:32:57.910 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.14790", 2024-02-09T17:32:57.910 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:32:57.910 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 68287 2024-02-09T17:32:57.910 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.910 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:32:57.910 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.910 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:32:57.910 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.172962+0000", 2024-02-09T17:32:57.910 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:57.910 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.910 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.910 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:32:57.910 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.172962+0000", 2024-02-09T17:32:57.910 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:57.911 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.911 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.911 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:32:57.911 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.172964+0000", 2024-02-09T17:32:57.911 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3400000000000001e-06 2024-02-09T17:32:57.911 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.911 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.911 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:32:57.911 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.173049+0000", 2024-02-09T17:32:57.911 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.5029000000000002e-05 2024-02-09T17:32:57.911 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.911 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.911 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:32:57.911 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.173072+0000", 2024-02-09T17:32:57.911 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.3683999999999999e-05 2024-02-09T17:32:57.911 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.912 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.912 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:32:57.912 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.173079+0000", 2024-02-09T17:32:57.912 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.7569999999999998e-06 2024-02-09T17:32:57.912 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.912 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.912 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:57.912 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.173133+0000", 2024-02-09T17:32:57.912 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3897999999999998e-05 2024-02-09T17:32:57.912 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.912 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.912 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:32:57.912 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.173277+0000", 2024-02-09T17:32:57.912 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000143969 2024-02-09T17:32:57.913 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.913 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.913 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,10", 2024-02-09T17:32:57.913 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.173451+0000", 2024-02-09T17:32:57.913 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00017415599999999999 2024-02-09T17:32:57.913 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.913 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.913 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:32:57.913 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.175861+0000", 2024-02-09T17:32:57.913 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0024101959999999999 2024-02-09T17:32:57.913 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.913 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.913 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:57.913 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.176234+0000", 2024-02-09T17:32:57.914 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00037265200000000001 2024-02-09T17:32:57.914 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.914 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.914 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:57.914 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.350681+0000", 2024-02-09T17:32:57.914 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.174447358 2024-02-09T17:32:57.914 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.914 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.914 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:32:57.914 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.350724+0000", 2024-02-09T17:32:57.914 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.3096000000000002e-05 2024-02-09T17:32:57.914 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.914 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.914 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:32:57.915 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:25:20.350756+0000", 2024-02-09T17:32:57.915 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.1622000000000003e-05 2024-02-09T17:32:57.915 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:57.915 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:32:57.915 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:57.915 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.915 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.915 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.24817.0:17925 3.4 3:2ae260fc:::benchmark_data_smithi179_90440_object17924:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e208)", 2024-02-09T17:32:57.915 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:30:10.952607+0000", 2024-02-09T17:32:57.915 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 166.915069004, 2024-02-09T17:32:57.915 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20464811999999999, 2024-02-09T17:32:57.915 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:32:57.915 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:32:57.915 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:32:57.915 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.24817", 2024-02-09T17:32:57.916 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:32:57.916 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 17925 2024-02-09T17:32:57.916 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.916 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:32:57.916 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.916 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:32:57.916 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.952607+0000", 2024-02-09T17:32:57.916 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:57.916 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.916 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.916 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:32:57.916 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.952607+0000", 2024-02-09T17:32:57.916 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:57.916 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.916 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.917 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:32:57.917 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.952608+0000", 2024-02-09T17:32:57.917 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2899999999999999e-06 2024-02-09T17:32:57.917 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.917 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.917 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:32:57.917 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.952710+0000", 2024-02-09T17:32:57.917 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00010126799999999999 2024-02-09T17:32:57.917 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.917 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.917 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:32:57.917 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.952721+0000", 2024-02-09T17:32:57.917 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0858e-05 2024-02-09T17:32:57.918 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.918 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.918 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:32:57.918 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.952727+0000", 2024-02-09T17:32:57.918 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.4130000000000003e-06 2024-02-09T17:32:57.918 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.918 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.918 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:57.918 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.952864+0000", 2024-02-09T17:32:57.918 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013676899999999999 2024-02-09T17:32:57.918 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.918 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.918 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:32:57.918 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.952982+0000", 2024-02-09T17:32:57.918 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000117798 2024-02-09T17:32:57.919 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.919 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.919 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 5,11", 2024-02-09T17:32:57.919 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.953112+0000", 2024-02-09T17:32:57.919 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000130734 2024-02-09T17:32:57.919 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.919 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.919 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:32:57.919 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.954366+0000", 2024-02-09T17:32:57.919 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.001253384 2024-02-09T17:32:57.919 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.919 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.919 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:57.919 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.955037+0000", 2024-02-09T17:32:57.919 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00067162000000000005 2024-02-09T17:32:57.920 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.920 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.920 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:57.920 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.157181+0000", 2024-02-09T17:32:57.920 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20214359400000001 2024-02-09T17:32:57.920 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.920 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.920 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:32:57.920 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.157221+0000", 2024-02-09T17:32:57.920 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.0432000000000002e-05 2024-02-09T17:32:57.920 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.920 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.920 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:32:57.921 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.157255+0000", 2024-02-09T17:32:57.921 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.396e-05 2024-02-09T17:32:57.921 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:57.921 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:32:57.921 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:57.921 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.921 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.921 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.24817.0:17928 3.4 3:23581cc5:::benchmark_data_smithi179_90440_object17927:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e208)", 2024-02-09T17:32:57.921 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:30:10.952916+0000", 2024-02-09T17:32:57.921 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 166.91476029, 2024-02-09T17:32:57.921 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.206896735, 2024-02-09T17:32:57.921 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:32:57.921 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:32:57.922 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:32:57.922 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.24817", 2024-02-09T17:32:57.922 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:32:57.922 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 17928 2024-02-09T17:32:57.922 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.922 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:32:57.922 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.922 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:32:57.922 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.952916+0000", 2024-02-09T17:32:57.922 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:57.922 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.922 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.922 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:32:57.923 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.952916+0000", 2024-02-09T17:32:57.923 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:57.923 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.923 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.923 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:32:57.923 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.952916+0000", 2024-02-09T17:32:57.923 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.5700000000000002e-07 2024-02-09T17:32:57.923 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.923 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.923 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:32:57.923 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.952992+0000", 2024-02-09T17:32:57.923 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.5897999999999997e-05 2024-02-09T17:32:57.923 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.923 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.924 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:32:57.924 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.953000+0000", 2024-02-09T17:32:57.924 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.2119999999999997e-06 2024-02-09T17:32:57.924 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.924 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.924 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:32:57.924 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.953004+0000", 2024-02-09T17:32:57.924 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.7339999999999999e-06 2024-02-09T17:32:57.924 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.924 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.924 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:57.924 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.953799+0000", 2024-02-09T17:32:57.924 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00079476600000000003 2024-02-09T17:32:57.924 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.925 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.925 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:32:57.925 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.953879+0000", 2024-02-09T17:32:57.925 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.0234999999999993e-05 2024-02-09T17:32:57.925 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.925 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.925 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 5,11", 2024-02-09T17:32:57.925 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.953960+0000", 2024-02-09T17:32:57.925 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.0556000000000006e-05 2024-02-09T17:32:57.925 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.925 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.925 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:32:57.925 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.954694+0000", 2024-02-09T17:32:57.925 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00073438099999999999 2024-02-09T17:32:57.925 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.926 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.926 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:57.926 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.955127+0000", 2024-02-09T17:32:57.926 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00043234300000000002 2024-02-09T17:32:57.926 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.926 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.926 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:57.926 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.159739+0000", 2024-02-09T17:32:57.926 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20461217600000001 2024-02-09T17:32:57.926 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.926 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.926 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:32:57.926 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.159790+0000", 2024-02-09T17:32:57.926 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1161e-05 2024-02-09T17:32:57.926 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.927 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.927 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:32:57.927 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.159813+0000", 2024-02-09T17:32:57.927 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2716000000000001e-05 2024-02-09T17:32:57.927 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:57.927 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:32:57.927 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:57.927 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.927 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.927 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.24817.0:17945 3.4 3:21bbe1ad:::benchmark_data_smithi179_90440_object17944:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e208)", 2024-02-09T17:32:57.927 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:30:10.958576+0000", 2024-02-09T17:32:57.927 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 166.909099677, 2024-02-09T17:32:57.927 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.202096796, 2024-02-09T17:32:57.927 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:32:57.927 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:32:57.927 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:32:57.928 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.24817", 2024-02-09T17:32:57.928 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:32:57.928 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 17945 2024-02-09T17:32:57.928 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.928 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:32:57.928 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.928 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:32:57.928 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.958576+0000", 2024-02-09T17:32:57.928 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:57.928 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.928 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.928 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:32:57.928 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.958576+0000", 2024-02-09T17:32:57.928 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:57.928 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.929 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.929 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:32:57.929 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.958578+0000", 2024-02-09T17:32:57.929 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2750000000000001e-06 2024-02-09T17:32:57.929 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.929 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.929 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:32:57.929 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.958633+0000", 2024-02-09T17:32:57.929 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.5692000000000001e-05 2024-02-09T17:32:57.929 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.929 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.929 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:32:57.929 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.958646+0000", 2024-02-09T17:32:57.929 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2954e-05 2024-02-09T17:32:57.930 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.930 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.930 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:32:57.930 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.958653+0000", 2024-02-09T17:32:57.930 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.9399999999999996e-06 2024-02-09T17:32:57.930 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.930 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.930 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:57.930 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.958711+0000", 2024-02-09T17:32:57.930 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7537999999999997e-05 2024-02-09T17:32:57.930 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.930 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.930 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:32:57.930 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.958810+0000", 2024-02-09T17:32:57.931 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.891e-05 2024-02-09T17:32:57.931 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.931 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.931 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 5,11", 2024-02-09T17:32:57.931 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.958903+0000", 2024-02-09T17:32:57.931 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.3463999999999998e-05 2024-02-09T17:32:57.931 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.931 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.931 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:32:57.931 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.960340+0000", 2024-02-09T17:32:57.931 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0014367609999999999 2024-02-09T17:32:57.931 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.931 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.932 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:57.932 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.961154+0000", 2024-02-09T17:32:57.932 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00081422799999999996 2024-02-09T17:32:57.932 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.932 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.932 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:57.932 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.160587+0000", 2024-02-09T17:32:57.932 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.199432517 2024-02-09T17:32:57.932 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.932 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.932 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:32:57.932 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.160641+0000", 2024-02-09T17:32:57.932 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.4178000000000002e-05 2024-02-09T17:32:57.932 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.932 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.933 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:32:57.933 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.160673+0000", 2024-02-09T17:32:57.933 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.2339000000000003e-05 2024-02-09T17:32:57.933 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:57.933 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:32:57.933 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:57.933 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.933 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.933 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.24817.0:17967 3.4 3:2bd714e9:::benchmark_data_smithi179_90440_object17966:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e208)", 2024-02-09T17:32:57.933 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:30:10.964691+0000", 2024-02-09T17:32:57.933 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 166.902984882, 2024-02-09T17:32:57.933 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.19612623000000001, 2024-02-09T17:32:57.933 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:32:57.934 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:32:57.934 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:32:57.934 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.24817", 2024-02-09T17:32:57.934 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:32:57.934 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 17967 2024-02-09T17:32:57.934 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.934 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:32:57.934 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.934 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:32:57.934 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.964691+0000", 2024-02-09T17:32:57.934 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:57.935 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.935 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.935 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:32:57.935 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.964691+0000", 2024-02-09T17:32:57.935 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:57.935 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.935 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.935 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:32:57.935 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.964693+0000", 2024-02-09T17:32:57.935 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.8789999999999999e-06 2024-02-09T17:32:57.935 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.935 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.935 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:32:57.936 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.964780+0000", 2024-02-09T17:32:57.936 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.6452000000000005e-05 2024-02-09T17:32:57.936 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.936 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.936 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:32:57.936 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.964790+0000", 2024-02-09T17:32:57.936 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0199e-05 2024-02-09T17:32:57.936 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.936 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.936 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:32:57.936 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.964797+0000", 2024-02-09T17:32:57.936 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.058e-06 2024-02-09T17:32:57.936 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.937 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.937 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:57.937 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.964851+0000", 2024-02-09T17:32:57.937 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.4543000000000003e-05 2024-02-09T17:32:57.937 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.937 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.937 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:32:57.937 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.964936+0000", 2024-02-09T17:32:57.937 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.4188000000000001e-05 2024-02-09T17:32:57.937 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.937 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.937 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 5,11", 2024-02-09T17:32:57.937 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.965029+0000", 2024-02-09T17:32:57.937 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.3482999999999999e-05 2024-02-09T17:32:57.937 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.937 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.938 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:32:57.938 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.965851+0000", 2024-02-09T17:32:57.938 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00082165299999999999 2024-02-09T17:32:57.938 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.938 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.938 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:57.938 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.966786+0000", 2024-02-09T17:32:57.938 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00093539199999999995 2024-02-09T17:32:57.938 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.938 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.938 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:57.938 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.160728+0000", 2024-02-09T17:32:57.938 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.19394203099999999 2024-02-09T17:32:57.938 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.939 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.939 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:32:57.939 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.160782+0000", 2024-02-09T17:32:57.939 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3378000000000003e-05 2024-02-09T17:32:57.939 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.939 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.939 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:32:57.939 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.160817+0000", 2024-02-09T17:32:57.939 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.5973999999999998e-05 2024-02-09T17:32:57.939 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:57.939 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:32:57.939 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:57.939 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.940 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.940 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.24817.0:17975 3.4 3:250c10bd:::benchmark_data_smithi179_90440_object17974:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e208)", 2024-02-09T17:32:57.940 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:30:10.967504+0000", 2024-02-09T17:32:57.940 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 166.90017235100001, 2024-02-09T17:32:57.940 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.19349122699999999, 2024-02-09T17:32:57.940 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:32:57.940 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:32:57.940 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:32:57.940 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.24817", 2024-02-09T17:32:57.940 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:32:57.940 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 17975 2024-02-09T17:32:57.940 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.940 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:32:57.941 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.941 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:32:57.941 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.967504+0000", 2024-02-09T17:32:57.941 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:57.941 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.941 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.941 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:32:57.941 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.967504+0000", 2024-02-09T17:32:57.941 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:57.941 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.941 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.941 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:32:57.941 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.967505+0000", 2024-02-09T17:32:57.942 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.8200000000000008e-07 2024-02-09T17:32:57.942 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.942 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.942 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:32:57.942 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.967556+0000", 2024-02-09T17:32:57.942 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1267999999999998e-05 2024-02-09T17:32:57.942 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.942 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.942 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:32:57.942 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.967567+0000", 2024-02-09T17:32:57.942 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.081e-05 2024-02-09T17:32:57.942 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.942 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.942 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:32:57.943 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.967573+0000", 2024-02-09T17:32:57.943 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7259999999999999e-06 2024-02-09T17:32:57.943 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.943 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.943 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:57.943 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.967617+0000", 2024-02-09T17:32:57.943 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.3952999999999997e-05 2024-02-09T17:32:57.943 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.943 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.943 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:32:57.943 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.967701+0000", 2024-02-09T17:32:57.943 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.4358999999999996e-05 2024-02-09T17:32:57.943 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.943 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.944 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 5,11", 2024-02-09T17:32:57.944 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.967830+0000", 2024-02-09T17:32:57.944 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012950700000000001 2024-02-09T17:32:57.944 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.944 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.944 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:32:57.944 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.968547+0000", 2024-02-09T17:32:57.944 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00071674900000000003 2024-02-09T17:32:57.944 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.944 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.944 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:57.944 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.970351+0000", 2024-02-09T17:32:57.945 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.001803806 2024-02-09T17:32:57.945 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.945 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.945 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:57.945 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.160873+0000", 2024-02-09T17:32:57.945 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.19052237999999999 2024-02-09T17:32:57.945 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.945 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.945 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:32:57.945 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.160955+0000", 2024-02-09T17:32:57.945 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.1267999999999995e-05 2024-02-09T17:32:57.945 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.945 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.945 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:32:57.946 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.160995+0000", 2024-02-09T17:32:57.946 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.0419e-05 2024-02-09T17:32:57.946 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:57.946 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:32:57.946 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:57.946 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.946 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.946 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.24817.0:17979 3.4 3:2994c410:::benchmark_data_smithi179_90440_object17978:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e208)", 2024-02-09T17:32:57.946 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:30:10.968478+0000", 2024-02-09T17:32:57.946 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 166.89919787900001, 2024-02-09T17:32:57.946 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.192659571, 2024-02-09T17:32:57.946 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:32:57.946 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:32:57.946 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:32:57.946 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.24817", 2024-02-09T17:32:57.947 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:32:57.947 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 17979 2024-02-09T17:32:57.947 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.947 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:32:57.947 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.947 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:32:57.947 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.968478+0000", 2024-02-09T17:32:57.947 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:57.947 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.947 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.947 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:32:57.947 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.968478+0000", 2024-02-09T17:32:57.947 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:57.947 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.948 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.948 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:32:57.948 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.968479+0000", 2024-02-09T17:32:57.948 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1999999999999999e-06 2024-02-09T17:32:57.948 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.948 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.948 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:32:57.948 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.968567+0000", 2024-02-09T17:32:57.948 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.7250999999999995e-05 2024-02-09T17:32:57.948 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.948 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.948 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:32:57.948 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.968581+0000", 2024-02-09T17:32:57.949 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4112e-05 2024-02-09T17:32:57.949 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.949 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.949 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:32:57.949 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.968592+0000", 2024-02-09T17:32:57.949 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0708e-05 2024-02-09T17:32:57.949 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.949 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.949 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:57.949 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.968643+0000", 2024-02-09T17:32:57.949 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1955e-05 2024-02-09T17:32:57.949 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.949 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.949 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:32:57.949 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.968776+0000", 2024-02-09T17:32:57.950 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013224299999999999 2024-02-09T17:32:57.950 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.950 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.950 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 5,11", 2024-02-09T17:32:57.950 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.968926+0000", 2024-02-09T17:32:57.950 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000150204 2024-02-09T17:32:57.950 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.950 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.950 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:32:57.950 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.970318+0000", 2024-02-09T17:32:57.950 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00139192 2024-02-09T17:32:57.950 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.950 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.950 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:57.951 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.971018+0000", 2024-02-09T17:32:57.951 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00070059099999999997 2024-02-09T17:32:57.951 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.951 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.951 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:57.951 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.161050+0000", 2024-02-09T17:32:57.951 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.190031599 2024-02-09T17:32:57.951 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.951 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.951 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:32:57.951 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.161103+0000", 2024-02-09T17:32:57.951 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2852999999999997e-05 2024-02-09T17:32:57.951 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.951 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.951 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:32:57.952 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.161138+0000", 2024-02-09T17:32:57.952 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.4934999999999997e-05 2024-02-09T17:32:57.952 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:57.952 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:32:57.952 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:57.952 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.952 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.952 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.24817.0:17980 3.4 3:25888b13:::benchmark_data_smithi179_90440_object17979:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e208)", 2024-02-09T17:32:57.952 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:30:10.968692+0000", 2024-02-09T17:32:57.952 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 166.89898453399999, 2024-02-09T17:32:57.952 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.19258343999999999, 2024-02-09T17:32:57.952 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:32:57.952 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:32:57.952 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:32:57.953 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.24817", 2024-02-09T17:32:57.953 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:32:57.953 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 17980 2024-02-09T17:32:57.953 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.953 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:32:57.953 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.953 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:32:57.953 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.968692+0000", 2024-02-09T17:32:57.953 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:57.953 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.953 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.953 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:32:57.953 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.968692+0000", 2024-02-09T17:32:57.953 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:57.954 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.954 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.954 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:32:57.954 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.968692+0000", 2024-02-09T17:32:57.954 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.9899999999999999e-07 2024-02-09T17:32:57.954 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.954 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.954 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:32:57.954 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.968768+0000", 2024-02-09T17:32:57.954 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.5127000000000001e-05 2024-02-09T17:32:57.954 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.954 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.954 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:32:57.954 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.968779+0000", 2024-02-09T17:32:57.955 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1691e-05 2024-02-09T17:32:57.955 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.955 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.955 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:32:57.955 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.968788+0000", 2024-02-09T17:32:57.955 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.2989999999999997e-06 2024-02-09T17:32:57.955 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.955 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.955 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:57.955 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.969654+0000", 2024-02-09T17:32:57.955 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00086646599999999998 2024-02-09T17:32:57.955 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.955 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.955 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:32:57.955 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.969779+0000", 2024-02-09T17:32:57.956 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012484 2024-02-09T17:32:57.956 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.956 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.956 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 5,11", 2024-02-09T17:32:57.956 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.969881+0000", 2024-02-09T17:32:57.956 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000101647 2024-02-09T17:32:57.956 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.956 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.956 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:32:57.956 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.970407+0000", 2024-02-09T17:32:57.956 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00052639699999999995 2024-02-09T17:32:57.956 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.956 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.956 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:57.957 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.971067+0000", 2024-02-09T17:32:57.957 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00065963100000000004 2024-02-09T17:32:57.957 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.957 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.957 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:57.957 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.161188+0000", 2024-02-09T17:32:57.957 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.19012095800000001 2024-02-09T17:32:57.957 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.957 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.957 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:32:57.957 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.161238+0000", 2024-02-09T17:32:57.957 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.0164000000000003e-05 2024-02-09T17:32:57.957 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.957 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.958 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:32:57.958 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.161275+0000", 2024-02-09T17:32:57.958 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.7320999999999997e-05 2024-02-09T17:32:57.958 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:57.958 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:32:57.958 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:57.958 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.958 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.958 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.24817.0:18011 3.4 3:211e0f97:::benchmark_data_smithi179_90440_object18010:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e208)", 2024-02-09T17:32:57.958 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:30:10.985882+0000", 2024-02-09T17:32:57.958 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 166.88179430700001, 2024-02-09T17:32:57.958 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.17555878599999999, 2024-02-09T17:32:57.958 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:32:57.958 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:32:57.958 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:32:57.958 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.24817", 2024-02-09T17:32:57.959 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:32:57.959 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 18011 2024-02-09T17:32:57.959 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.959 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:32:57.959 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.959 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:32:57.959 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.985882+0000", 2024-02-09T17:32:57.959 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:57.959 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.959 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.959 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:32:57.959 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.985882+0000", 2024-02-09T17:32:57.959 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:57.959 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.959 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.960 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:32:57.960 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.985883+0000", 2024-02-09T17:32:57.960 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.395e-06 2024-02-09T17:32:57.960 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.960 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.960 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:32:57.960 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.985932+0000", 2024-02-09T17:32:57.960 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.8433000000000002e-05 2024-02-09T17:32:57.960 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.960 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.960 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:32:57.960 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.985943+0000", 2024-02-09T17:32:57.960 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1382e-05 2024-02-09T17:32:57.960 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.961 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.961 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:32:57.961 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.985949+0000", 2024-02-09T17:32:57.961 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.083e-06 2024-02-09T17:32:57.961 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.961 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.961 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:57.961 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.985998+0000", 2024-02-09T17:32:57.961 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.8949000000000002e-05 2024-02-09T17:32:57.961 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.961 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.961 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:32:57.961 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.986081+0000", 2024-02-09T17:32:57.962 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.3256000000000004e-05 2024-02-09T17:32:57.962 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.962 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.962 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 5,11", 2024-02-09T17:32:57.962 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.986169+0000", 2024-02-09T17:32:57.962 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.7194999999999994e-05 2024-02-09T17:32:57.962 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.962 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.962 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:32:57.962 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.986949+0000", 2024-02-09T17:32:57.962 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00078056499999999999 2024-02-09T17:32:57.962 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.962 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.962 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:57.962 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.987452+0000", 2024-02-09T17:32:57.963 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00050306199999999998 2024-02-09T17:32:57.963 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.963 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.963 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:57.963 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.161333+0000", 2024-02-09T17:32:57.963 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.173880964 2024-02-09T17:32:57.963 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.963 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.963 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:32:57.963 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.161401+0000", 2024-02-09T17:32:57.963 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.7797000000000002e-05 2024-02-09T17:32:57.963 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.963 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.963 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:32:57.963 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.161441+0000", 2024-02-09T17:32:57.964 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9705e-05 2024-02-09T17:32:57.964 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:57.964 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:32:57.964 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:57.964 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.964 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.964 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.24817.0:18012 3.4 3:2a908264:::benchmark_data_smithi179_90440_object18011:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e208)", 2024-02-09T17:32:57.964 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:30:10.987352+0000", 2024-02-09T17:32:57.964 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 166.88032408999999, 2024-02-09T17:32:57.964 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.17466900199999999, 2024-02-09T17:32:57.964 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:32:57.964 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:32:57.964 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:32:57.965 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.24817", 2024-02-09T17:32:57.965 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:32:57.965 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 18012 2024-02-09T17:32:57.965 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.965 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:32:57.965 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.965 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:32:57.965 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.987352+0000", 2024-02-09T17:32:57.965 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:57.965 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.965 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.965 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:32:57.965 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.987352+0000", 2024-02-09T17:32:57.966 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:57.966 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.966 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.966 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:32:57.966 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.987353+0000", 2024-02-09T17:32:57.966 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.16e-07 2024-02-09T17:32:57.966 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.966 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.966 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:32:57.966 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.987450+0000", 2024-02-09T17:32:57.966 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.7262000000000005e-05 2024-02-09T17:32:57.966 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.966 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.967 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:32:57.967 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.987458+0000", 2024-02-09T17:32:57.967 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.8180000000000004e-06 2024-02-09T17:32:57.967 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.967 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.967 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:32:57.967 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.987463+0000", 2024-02-09T17:32:57.967 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.8930000000000004e-06 2024-02-09T17:32:57.967 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.967 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.967 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:57.967 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.987491+0000", 2024-02-09T17:32:57.967 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.7789000000000001e-05 2024-02-09T17:32:57.967 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.967 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.968 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:32:57.968 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.987588+0000", 2024-02-09T17:32:57.968 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.6966999999999995e-05 2024-02-09T17:32:57.968 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.968 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.968 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 5,11", 2024-02-09T17:32:57.968 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.987676+0000", 2024-02-09T17:32:57.968 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.8090000000000005e-05 2024-02-09T17:32:57.968 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.968 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.968 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:32:57.968 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.988405+0000", 2024-02-09T17:32:57.968 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00072901400000000005 2024-02-09T17:32:57.969 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.969 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.969 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:57.969 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:10.988618+0000", 2024-02-09T17:32:57.969 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000213425 2024-02-09T17:32:57.969 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.969 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.969 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:57.969 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.161913+0000", 2024-02-09T17:32:57.969 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.17329457500000001 2024-02-09T17:32:57.969 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.969 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.969 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:32:57.969 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.161974+0000", 2024-02-09T17:32:57.969 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.1194999999999999e-05 2024-02-09T17:32:57.970 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.970 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.970 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:32:57.970 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:30:11.162021+0000", 2024-02-09T17:32:57.970 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.7058000000000002e-05 2024-02-09T17:32:57.970 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:57.970 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:32:57.970 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:57.970 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.970 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.970 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.24817.0:141304 3.6 3:63334779:::benchmark_data_smithi179_90440_object9345:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-02-09T17:32:57.970 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:31:59.273717+0000", 2024-02-09T17:32:57.970 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 58.593958909999998, 2024-02-09T17:32:57.970 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.303239354, 2024-02-09T17:32:57.971 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:32:57.971 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:32:57.971 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:32:57.971 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.24817", 2024-02-09T17:32:57.971 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:32:57.971 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 141304 2024-02-09T17:32:57.971 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.971 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:32:57.971 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.971 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:32:57.971 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.273717+0000", 2024-02-09T17:32:57.971 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:57.971 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.971 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.971 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:32:57.972 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.273717+0000", 2024-02-09T17:32:57.972 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:57.972 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.972 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.972 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:32:57.972 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.273718+0000", 2024-02-09T17:32:57.972 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.8100000000000002e-07 2024-02-09T17:32:57.972 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.972 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.972 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:32:57.972 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.273721+0000", 2024-02-09T17:32:57.972 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.1489999999999998e-06 2024-02-09T17:32:57.972 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.973 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.973 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:32:57.973 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.273729+0000", 2024-02-09T17:32:57.973 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.481e-06 2024-02-09T17:32:57.973 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.973 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.973 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:32:57.973 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.273734+0000", 2024-02-09T17:32:57.973 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.0289999999999996e-06 2024-02-09T17:32:57.973 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.973 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.973 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:57.973 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.273821+0000", 2024-02-09T17:32:57.974 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.7763999999999995e-05 2024-02-09T17:32:57.974 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.974 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.974 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op must wait for map", 2024-02-09T17:32:57.974 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.273826+0000", 2024-02-09T17:32:57.974 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.3259999999999997e-06 2024-02-09T17:32:57.974 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.974 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.974 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:57.974 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.281374+0000", 2024-02-09T17:32:57.974 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0075486549999999996 2024-02-09T17:32:57.974 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.974 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.974 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:32:57.975 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.281384+0000", 2024-02-09T17:32:57.975 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.3349999999999996e-06 2024-02-09T17:32:57.975 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.975 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.975 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:57.975 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:00.272447+0000", 2024-02-09T17:32:57.975 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.99106366400000001 2024-02-09T17:32:57.975 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.975 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.975 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:32:57.975 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:00.272457+0000", 2024-02-09T17:32:57.975 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.1759999999999999e-06 2024-02-09T17:32:57.975 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.975 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.975 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:57.976 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.543600+0000", 2024-02-09T17:32:57.976 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.27114342 2024-02-09T17:32:57.976 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.976 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.976 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:32:57.976 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.546128+0000", 2024-02-09T17:32:57.976 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0025278420000000002 2024-02-09T17:32:57.976 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.976 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.976 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 4,7", 2024-02-09T17:32:57.976 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.546287+0000", 2024-02-09T17:32:57.976 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0001593 2024-02-09T17:32:57.976 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.976 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.977 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:32:57.977 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.556995+0000", 2024-02-09T17:32:57.977 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.010707816 2024-02-09T17:32:57.977 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.977 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.977 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:57.977 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.557048+0000", 2024-02-09T17:32:57.977 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2862999999999999e-05 2024-02-09T17:32:57.977 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.977 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.977 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:57.977 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.574525+0000", 2024-02-09T17:32:57.977 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.017476744999999998 2024-02-09T17:32:57.977 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.978 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.978 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:32:57.978 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.574594+0000", 2024-02-09T17:32:57.978 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.9626e-05 2024-02-09T17:32:57.978 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.978 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.978 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:32:57.978 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.576957+0000", 2024-02-09T17:32:57.978 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0023623820000000001 2024-02-09T17:32:57.978 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:57.978 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:32:57.978 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:57.978 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.978 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.978 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.24817.0:141305 3.6 3:6b0ab8bf:::benchmark_data_smithi179_90440_object9346:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-02-09T17:32:57.979 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:31:59.273750+0000", 2024-02-09T17:32:57.979 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 58.593925728000002, 2024-02-09T17:32:57.979 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.3052010909999998, 2024-02-09T17:32:57.979 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:32:57.979 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:32:57.979 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:32:57.979 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.24817", 2024-02-09T17:32:57.979 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:32:57.979 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 141305 2024-02-09T17:32:57.979 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.979 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:32:57.979 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.979 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:32:57.979 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.273750+0000", 2024-02-09T17:32:57.980 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:57.980 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.980 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.980 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:32:57.980 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.273750+0000", 2024-02-09T17:32:57.980 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:57.980 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.980 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.980 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:32:57.980 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.273751+0000", 2024-02-09T17:32:57.980 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9299999999999999e-07 2024-02-09T17:32:57.980 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.980 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.980 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:32:57.981 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.273753+0000", 2024-02-09T17:32:57.981 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.367e-06 2024-02-09T17:32:57.981 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.981 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.981 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:32:57.981 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.273759+0000", 2024-02-09T17:32:57.981 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.9699999999999996e-06 2024-02-09T17:32:57.981 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.981 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.981 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:32:57.981 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.273765+0000", 2024-02-09T17:32:57.981 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7470000000000002e-06 2024-02-09T17:32:57.981 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.981 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.981 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:57.982 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.273849+0000", 2024-02-09T17:32:57.982 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.3770999999999995e-05 2024-02-09T17:32:57.982 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.982 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.982 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting_for_map not empty", 2024-02-09T17:32:57.982 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.273854+0000", 2024-02-09T17:32:57.982 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.4729999999999999e-06 2024-02-09T17:32:57.982 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.982 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.982 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:57.982 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.281503+0000", 2024-02-09T17:32:57.982 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0076486519999999997 2024-02-09T17:32:57.982 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.982 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.983 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:32:57.983 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.281510+0000", 2024-02-09T17:32:57.983 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.2799999999999998e-06 2024-02-09T17:32:57.983 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.983 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.983 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:57.983 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:00.272552+0000", 2024-02-09T17:32:57.983 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.99104202100000005 2024-02-09T17:32:57.983 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.983 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.983 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:32:57.983 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:00.272558+0000", 2024-02-09T17:32:57.983 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.8719999999999999e-06 2024-02-09T17:32:57.983 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.983 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.984 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:57.984 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.551349+0000", 2024-02-09T17:32:57.984 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2787907810000001 2024-02-09T17:32:57.984 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.984 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.984 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:32:57.984 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.551501+0000", 2024-02-09T17:32:57.984 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00015254400000000001 2024-02-09T17:32:57.984 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.984 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.984 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 4,7", 2024-02-09T17:32:57.984 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.551599+0000", 2024-02-09T17:32:57.984 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.8162e-05 2024-02-09T17:32:57.985 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.985 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.985 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:32:57.985 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.557125+0000", 2024-02-09T17:32:57.985 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0055251789999999999 2024-02-09T17:32:57.985 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.985 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.985 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:57.985 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.562614+0000", 2024-02-09T17:32:57.985 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.005489837 2024-02-09T17:32:57.985 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.985 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.985 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:57.985 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.577014+0000", 2024-02-09T17:32:57.986 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.014399095000000001 2024-02-09T17:32:57.986 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.986 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.986 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:32:57.986 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.577068+0000", 2024-02-09T17:32:57.986 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.4475999999999998e-05 2024-02-09T17:32:57.986 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.986 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.986 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:32:57.986 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.578951+0000", 2024-02-09T17:32:57.986 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0018834710000000001 2024-02-09T17:32:57.986 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:57.986 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:32:57.986 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:57.986 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.987 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.987 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.24817.0:141328 3.6 3:636c2a21:::benchmark_data_smithi179_90440_object9369:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-02-09T17:32:57.987 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:31:59.299967+0000", 2024-02-09T17:32:57.987 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 58.567709502, 2024-02-09T17:32:57.987 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2807691910000001, 2024-02-09T17:32:57.987 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:32:57.987 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:32:57.987 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:32:57.987 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.24817", 2024-02-09T17:32:57.987 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:32:57.987 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 141328 2024-02-09T17:32:57.987 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.987 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:32:57.987 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.988 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:32:57.988 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.299967+0000", 2024-02-09T17:32:57.988 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:57.988 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.988 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.988 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:32:57.988 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.299967+0000", 2024-02-09T17:32:57.988 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:57.988 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.988 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.988 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:32:57.988 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.299968+0000", 2024-02-09T17:32:57.988 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3859999999999999e-06 2024-02-09T17:32:57.988 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.989 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.989 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:32:57.989 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.299972+0000", 2024-02-09T17:32:57.989 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.8999999999999999e-06 2024-02-09T17:32:57.989 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.989 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.989 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:32:57.989 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.299981+0000", 2024-02-09T17:32:57.989 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.7900000000000005e-06 2024-02-09T17:32:57.989 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.989 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.989 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:32:57.989 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.299987+0000", 2024-02-09T17:32:57.989 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.4060000000000002e-06 2024-02-09T17:32:57.989 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.989 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.990 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:57.990 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.300037+0000", 2024-02-09T17:32:57.990 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.9691999999999998e-05 2024-02-09T17:32:57.990 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.990 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.990 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:32:57.990 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.300048+0000", 2024-02-09T17:32:57.990 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0815999999999999e-05 2024-02-09T17:32:57.990 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.990 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.990 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:57.990 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:00.272580+0000", 2024-02-09T17:32:57.990 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.97253208999999996 2024-02-09T17:32:57.990 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.990 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.991 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:32:57.991 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:00.272588+0000", 2024-02-09T17:32:57.991 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.0979999999999994e-06 2024-02-09T17:32:57.991 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.991 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.991 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:57.991 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.554498+0000", 2024-02-09T17:32:57.991 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.281910208 2024-02-09T17:32:57.991 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.991 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.991 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:32:57.991 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.554600+0000", 2024-02-09T17:32:57.991 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00010242999999999999 2024-02-09T17:32:57.991 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.992 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.992 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 4,7", 2024-02-09T17:32:57.992 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.554694+0000", 2024-02-09T17:32:57.992 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.3293000000000003e-05 2024-02-09T17:32:57.992 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.992 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.992 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:32:57.992 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.559220+0000", 2024-02-09T17:32:57.992 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0045263150000000004 2024-02-09T17:32:57.992 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.992 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.992 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:57.992 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.562669+0000", 2024-02-09T17:32:57.993 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.003448965 2024-02-09T17:32:57.993 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.993 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.993 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:57.993 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.579019+0000", 2024-02-09T17:32:57.993 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.016349902999999999 2024-02-09T17:32:57.993 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.993 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.993 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:32:57.993 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.579083+0000", 2024-02-09T17:32:57.993 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.3901999999999999e-05 2024-02-09T17:32:57.993 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.993 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.993 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:32:57.994 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.580736+0000", 2024-02-09T17:32:57.994 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.001652997 2024-02-09T17:32:57.994 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:57.994 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:32:57.994 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:57.994 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.994 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.994 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.24817.0:141331 3.6 3:6c1a36e8:::benchmark_data_smithi179_90440_object9372:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-02-09T17:32:57.994 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:31:59.304058+0000", 2024-02-09T17:32:57.994 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 58.563618015000003, 2024-02-09T17:32:57.994 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2790567400000001, 2024-02-09T17:32:57.994 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:32:57.994 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:32:57.994 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:32:57.994 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.24817", 2024-02-09T17:32:57.995 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:32:57.995 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 141331 2024-02-09T17:32:57.995 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.995 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:32:57.995 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.995 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:32:57.995 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.304058+0000", 2024-02-09T17:32:57.995 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:57.995 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.995 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.995 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:32:57.995 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.304058+0000", 2024-02-09T17:32:57.995 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:32:57.995 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.995 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.996 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:32:57.996 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.304059+0000", 2024-02-09T17:32:57.996 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2669999999999999e-06 2024-02-09T17:32:57.996 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.996 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.996 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:32:57.996 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.304064+0000", 2024-02-09T17:32:57.996 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.0830000000000003e-06 2024-02-09T17:32:57.996 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.996 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.996 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:32:57.996 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.304075+0000", 2024-02-09T17:32:57.996 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0659000000000001e-05 2024-02-09T17:32:57.996 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.997 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.997 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:32:57.997 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.304082+0000", 2024-02-09T17:32:57.997 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.3220000000000004e-06 2024-02-09T17:32:57.997 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.997 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.997 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:57.997 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.304134+0000", 2024-02-09T17:32:57.997 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1638999999999998e-05 2024-02-09T17:32:57.997 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.997 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.997 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:32:57.997 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.304141+0000", 2024-02-09T17:32:57.997 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.1570000000000003e-06 2024-02-09T17:32:57.998 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.998 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.998 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:57.998 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:00.272625+0000", 2024-02-09T17:32:57.998 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.96848358400000001 2024-02-09T17:32:57.998 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.998 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.998 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:32:57.998 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:00.272629+0000", 2024-02-09T17:32:57.998 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.4309999999999996e-06 2024-02-09T17:32:57.998 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.998 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.998 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:32:57.998 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.557153+0000", 2024-02-09T17:32:57.999 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2845239560000001 2024-02-09T17:32:57.999 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.999 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.999 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:32:57.999 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.557307+0000", 2024-02-09T17:32:57.999 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00015353 2024-02-09T17:32:57.999 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.999 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.999 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 4,7", 2024-02-09T17:32:57.999 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.557462+0000", 2024-02-09T17:32:57.999 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00015571000000000001 2024-02-09T17:32:57.999 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:57.999 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:57.999 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:32:57.999 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.559377+0000", 2024-02-09T17:32:58.000 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0019150180000000001 2024-02-09T17:32:58.000 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:58.000 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:58.000 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:58.000 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.562762+0000", 2024-02-09T17:32:58.000 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0033848089999999999 2024-02-09T17:32:58.000 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:58.000 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:58.000 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:32:58.000 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.580802+0000", 2024-02-09T17:32:58.000 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.018039415999999999 2024-02-09T17:32:58.000 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:58.000 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:58.000 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:32:58.000 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.580862+0000", 2024-02-09T17:32:58.001 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.0405999999999997e-05 2024-02-09T17:32:58.001 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:32:58.001 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:32:58.001 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:32:58.001 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.583115+0000", 2024-02-09T17:32:58.001 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0022527530000000001 2024-02-09T17:32:58.001 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:58.001 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:32:58.001 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:58.001 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:32:58.001 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:32:58.001 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:32:58.002 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-09T17:32:58.003 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:57 smithi162 ceph-mon[44304]: 3.5 scrub starts 2024-02-09T17:32:58.003 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:57 smithi162 ceph-mon[44304]: pgmap v745: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 23 active+clean; 5.6 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 620 op/s; 2505/274128 objects degraded (0.914%); 22776/274128 objects misplaced (8.309%); 8.0 MiB/s, 128 objects/s recovering 2024-02-09T17:32:58.003 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:57 smithi179 ceph-mon[43975]: 3.5 scrub starts 2024-02-09T17:32:58.003 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:57 smithi179 ceph-mon[43975]: pgmap v745: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 23 active+clean; 5.6 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 620 op/s; 2505/274128 objects degraded (0.914%); 22776/274128 objects misplaced (8.309%); 8.0 MiB/s, 128 objects/s recovering 2024-02-09T17:32:58.276 INFO:teuthology.orchestra.run.smithi099.stderr:set pool 1 pg_num to 21 2024-02-09T17:32:58.875 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:32:58.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:58.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:32:58.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:58 smithi179 ceph-mon[43975]: 1.8 scrub starts 2024-02-09T17:32:58.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:58 smithi179 ceph-mon[43975]: 1.8 scrub ok 2024-02-09T17:32:58.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:58 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:32:58.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:58 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:32:58.926 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:58 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/385594984' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-02-09T17:32:58.926 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:58 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-02-09T17:32:58.926 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:58 smithi179 ceph-mon[43975]: 3.5 scrub ok 2024-02-09T17:32:58.926 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:58 smithi179 ceph-mon[43975]: 3.b scrub starts 2024-02-09T17:32:58.926 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:58 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-02-09T17:32:58.926 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:58 smithi179 ceph-mon[43975]: osdmap e260: 12 total, 12 up, 11 in 2024-02-09T17:32:58.926 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:58 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-09T17:32:58.926 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:58 smithi179 ceph-mon[43975]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-09T17:32:58.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:58 smithi162 ceph-mon[44304]: 1.8 scrub starts 2024-02-09T17:32:58.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:58 smithi162 ceph-mon[44304]: 1.8 scrub ok 2024-02-09T17:32:58.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:58 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:32:58.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:58 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:32:58.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:58 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/385594984' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-02-09T17:32:58.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:58 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-02-09T17:32:58.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:58 smithi162 ceph-mon[44304]: 3.5 scrub ok 2024-02-09T17:32:58.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:58 smithi162 ceph-mon[44304]: 3.b scrub starts 2024-02-09T17:32:58.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:58 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-02-09T17:32:58.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:58 smithi162 ceph-mon[44304]: osdmap e260: 12 total, 12 up, 11 in 2024-02-09T17:32:58.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:58 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-09T17:32:58.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:58 smithi162 ceph-mon[44304]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-09T17:32:58.977 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:58 smithi099 ceph-mon[39420]: 1.8 scrub starts 2024-02-09T17:32:58.978 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:58 smithi099 ceph-mon[39420]: 1.8 scrub ok 2024-02-09T17:32:58.978 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:58 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:32:58.978 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:58 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:32:58.978 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:58 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/385594984' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-02-09T17:32:58.978 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:58 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-02-09T17:32:58.978 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:58 smithi099 ceph-mon[39420]: 3.5 scrub ok 2024-02-09T17:32:58.978 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:58 smithi099 ceph-mon[39420]: 3.b scrub starts 2024-02-09T17:32:58.978 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:58 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-02-09T17:32:58.978 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:58 smithi099 ceph-mon[39420]: osdmap e260: 12 total, 12 up, 11 in 2024-02-09T17:32:58.978 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:58 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-09T17:32:58.978 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:58 smithi099 ceph-mon[39420]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-09T17:32:58.981 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:32:59.022 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:59.047 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:32:59.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:59.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:32:59.154 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:32:59.188 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:59.314 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:32:59.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:59.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:32:59.422 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:32:59.466 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:59.500 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:32:59.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:59.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:32:59.607 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:32:59.654 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:59.690 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:32:59.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:59.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:32:59.800 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:32:59.868 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:32:59.915 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:32:59.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:32:59.920 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:32:59.920 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:59 smithi179 ceph-mon[43975]: pgmap v747: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 23 active+clean; 5.5 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 567 op/s; 2505/270753 objects degraded (0.925%); 22776/270753 objects misplaced (8.412%); 4.3 MiB/s, 69 objects/s recovering 2024-02-09T17:32:59.920 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:59 smithi179 ceph-mon[43975]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-02-09T17:32:59.920 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:32:59 smithi179 ceph-mon[43975]: osdmap e261: 12 total, 12 up, 11 in 2024-02-09T17:32:59.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:59 smithi162 ceph-mon[44304]: pgmap v747: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 23 active+clean; 5.5 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 567 op/s; 2505/270753 objects degraded (0.925%); 22776/270753 objects misplaced (8.412%); 4.3 MiB/s, 69 objects/s recovering 2024-02-09T17:32:59.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:59 smithi162 ceph-mon[44304]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-02-09T17:32:59.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:32:59 smithi162 ceph-mon[44304]: osdmap e261: 12 total, 12 up, 11 in 2024-02-09T17:32:59.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:59 smithi099 ceph-mon[39420]: pgmap v747: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 23 active+clean; 5.5 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 567 op/s; 2505/270753 objects degraded (0.925%); 22776/270753 objects misplaced (8.412%); 4.3 MiB/s, 69 objects/s recovering 2024-02-09T17:32:59.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:59 smithi099 ceph-mon[39420]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-02-09T17:32:59.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:32:59 smithi099 ceph-mon[39420]: osdmap e261: 12 total, 12 up, 11 in 2024-02-09T17:33:00.020 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:33:00.050 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:00.395 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:33:00.417 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:33:00.442 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:33:00.468 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:33:00.512 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:33:00.540 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:33:00.567 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:33:00.602 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:33:00.627 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:33:00.650 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:33:00.674 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:33:00.698 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:33:00.699 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:33:00.699 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:33:00.700 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:33:00.700 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:33:00.700 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:33:00.700 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:33:00.700 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:33:00.700 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:33:00.700 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:33:00.700 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:33:00.700 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:33:00.700 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:33:00.845 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-09T17:33:00.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:00 smithi099 ceph-mon[39420]: 3.9 scrub starts 2024-02-09T17:33:00.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:00 smithi099 ceph-mon[39420]: osdmap e262: 12 total, 12 up, 11 in 2024-02-09T17:33:00.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:00 smithi099 ceph-mon[39420]: 3.b scrub ok 2024-02-09T17:33:00.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:00 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/101843238' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:33:00.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:00 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-02-09T17:33:00.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:00 smithi099 ceph-mon[39420]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-02-09T17:33:01.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:00 smithi179 ceph-mon[43975]: 3.9 scrub starts 2024-02-09T17:33:01.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:00 smithi179 ceph-mon[43975]: osdmap e262: 12 total, 12 up, 11 in 2024-02-09T17:33:01.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:00 smithi179 ceph-mon[43975]: 3.b scrub ok 2024-02-09T17:33:01.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:00 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/101843238' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:33:01.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:00 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-02-09T17:33:01.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:00 smithi179 ceph-mon[43975]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-02-09T17:33:01.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:00 smithi162 ceph-mon[44304]: 3.9 scrub starts 2024-02-09T17:33:01.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:00 smithi162 ceph-mon[44304]: osdmap e262: 12 total, 12 up, 11 in 2024-02-09T17:33:01.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:00 smithi162 ceph-mon[44304]: 3.b scrub ok 2024-02-09T17:33:01.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:00 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/101843238' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:33:01.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:00 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-02-09T17:33:01.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:00 smithi162 ceph-mon[44304]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-02-09T17:33:01.603 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:33:01.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:01.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:33:01.652 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:33:01.672 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:33:01.708 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:33:01.742 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:01.802 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:33:01.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:01.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:33:01.916 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:33:01.924 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:01 smithi099 ceph-mon[39420]: 3.c scrub starts 2024-02-09T17:33:01.924 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:01 smithi099 ceph-mon[39420]: pgmap v750: 37 pgs: 2 peering, 10 unknown, 4 active+recovering+undersized+degraded+remapped, 21 active+clean; 5.4 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 707 op/s; 2708/265902 objects degraded (1.018%); 22029/265902 objects misplaced (8.285%); 11 MiB/s, 175 objects/s recovering 2024-02-09T17:33:01.924 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:01 smithi099 ceph-mon[39420]: 3.9 scrub ok 2024-02-09T17:33:01.924 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:01 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 2505/270753 objects degraded (0.925%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:33:01.924 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:01 smithi099 ceph-mon[39420]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-02-09T17:33:01.924 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:01 smithi099 ceph-mon[39420]: osdmap e263: 12 total, 12 up, 11 in 2024-02-09T17:33:01.949 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:02.153 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:33:02.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:02.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:33:02.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:01 smithi179 ceph-mon[43975]: 3.c scrub starts 2024-02-09T17:33:02.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:01 smithi179 ceph-mon[43975]: pgmap v750: 37 pgs: 2 peering, 10 unknown, 4 active+recovering+undersized+degraded+remapped, 21 active+clean; 5.4 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 707 op/s; 2708/265902 objects degraded (1.018%); 22029/265902 objects misplaced (8.285%); 11 MiB/s, 175 objects/s recovering 2024-02-09T17:33:02.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:01 smithi179 ceph-mon[43975]: 3.9 scrub ok 2024-02-09T17:33:02.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:01 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 2505/270753 objects degraded (0.925%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:33:02.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:01 smithi179 ceph-mon[43975]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-02-09T17:33:02.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:01 smithi179 ceph-mon[43975]: osdmap e263: 12 total, 12 up, 11 in 2024-02-09T17:33:02.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:01 smithi162 ceph-mon[44304]: 3.c scrub starts 2024-02-09T17:33:02.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:01 smithi162 ceph-mon[44304]: pgmap v750: 37 pgs: 2 peering, 10 unknown, 4 active+recovering+undersized+degraded+remapped, 21 active+clean; 5.4 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 707 op/s; 2708/265902 objects degraded (1.018%); 22029/265902 objects misplaced (8.285%); 11 MiB/s, 175 objects/s recovering 2024-02-09T17:33:02.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:01 smithi162 ceph-mon[44304]: 3.9 scrub ok 2024-02-09T17:33:02.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:01 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 2505/270753 objects degraded (0.925%), 4 pgs degraded (PG_DEGRADED) 2024-02-09T17:33:02.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:01 smithi162 ceph-mon[44304]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-02-09T17:33:02.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:01 smithi162 ceph-mon[44304]: osdmap e263: 12 total, 12 up, 11 in 2024-02-09T17:33:02.261 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:33:02.327 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:02.357 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:33:02.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:02.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:33:02.465 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:33:02.508 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:02.547 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:33:02.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:02.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:33:02.655 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:33:02.710 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:02.747 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:33:02.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:02.751 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:33:02.852 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:33:02.884 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:03.038 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:33:03.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:03.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:33:03.144 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:33:03.204 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:03.244 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:33:03.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:03.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:33:03.352 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:33:03.392 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:03.419 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:33:03.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:03.424 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:33:03.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:03 smithi099 ceph-mon[39420]: 3.c scrub ok 2024-02-09T17:33:03.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:03 smithi099 ceph-mon[39420]: osdmap e264: 12 total, 12 up, 11 in 2024-02-09T17:33:03.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:03 smithi099 ceph-mon[39420]: pgmap v753: 37 pgs: 2 peering, 9 unknown, 4 active+recovering+undersized+degraded+remapped, 22 active+clean; 5.3 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 712 op/s; 2708/262080 objects degraded (1.033%); 22029/262080 objects misplaced (8.405%); 8.3 MiB/s, 135 objects/s recovering 2024-02-09T17:33:03.525 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:33:03.560 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:03.595 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:33:03.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:03.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:33:03.607 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:33:03.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:03 smithi179 ceph-mon[43975]: 3.c scrub ok 2024-02-09T17:33:03.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:03 smithi179 ceph-mon[43975]: osdmap e264: 12 total, 12 up, 11 in 2024-02-09T17:33:03.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:03 smithi179 ceph-mon[43975]: pgmap v753: 37 pgs: 2 peering, 9 unknown, 4 active+recovering+undersized+degraded+remapped, 22 active+clean; 5.3 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 712 op/s; 2708/262080 objects degraded (1.033%); 22029/262080 objects misplaced (8.405%); 8.3 MiB/s, 135 objects/s recovering 2024-02-09T17:33:03.687 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:03 smithi162 ceph-mon[44304]: 3.c scrub ok 2024-02-09T17:33:03.687 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:03 smithi162 ceph-mon[44304]: osdmap e264: 12 total, 12 up, 11 in 2024-02-09T17:33:03.687 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:03 smithi162 ceph-mon[44304]: pgmap v753: 37 pgs: 2 peering, 9 unknown, 4 active+recovering+undersized+degraded+remapped, 22 active+clean; 5.3 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 712 op/s; 2708/262080 objects degraded (1.033%); 22029/262080 objects misplaced (8.405%); 8.3 MiB/s, 135 objects/s recovering 2024-02-09T17:33:03.699 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:33:03.745 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:03.783 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:33:03.789 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:03.789 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:33:03.890 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:33:03.933 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:03.956 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:33:03.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:03.964 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:33:04.048 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:33:04.065 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:33:04.117 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:04.142 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:33:04.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:04.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:33:04.253 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:33:04.262 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:04 smithi099 ceph-mon[39420]: osdmap e265: 12 total, 12 up, 11 in 2024-02-09T17:33:04.323 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:04.354 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:33:04.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:04.365 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:33:04.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:04 smithi162 ceph-mon[44304]: osdmap e265: 12 total, 12 up, 11 in 2024-02-09T17:33:04.465 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:33:04.496 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:04.556 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:33:04.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:04.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:33:04.664 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:33:04.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:04 smithi179 ceph-mon[43975]: osdmap e265: 12 total, 12 up, 11 in 2024-02-09T17:33:04.729 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:04.756 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:33:04.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:04.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:33:04.805 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] 2024-02-09T17:33:04.805 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:33:04.806 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_1 2024-02-09T17:33:04.806 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph pg dump --format=json 2024-02-09T17:33:04.860 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:33:04.892 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:05.146 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:33:05.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:05.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:33:05.232 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:33:05.232 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:33:05.251 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:33:05.287 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:05.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:05 smithi099 ceph-mon[39420]: 3.7 scrub starts 2024-02-09T17:33:05.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:05 smithi099 ceph-mon[39420]: osdmap e266: 12 total, 12 up, 11 in 2024-02-09T17:33:05.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:05 smithi099 ceph-mon[39420]: pgmap v756: 37 pgs: 2 peering, 9 unknown, 4 active+recovering+undersized+degraded+remapped, 22 active+clean; 5.3 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 401 op/s; 2708/260931 objects degraded (1.038%); 22029/260931 objects misplaced (8.442%) 2024-02-09T17:33:05.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:05 smithi179 ceph-mon[43975]: 3.7 scrub starts 2024-02-09T17:33:05.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:05 smithi179 ceph-mon[43975]: osdmap e266: 12 total, 12 up, 11 in 2024-02-09T17:33:05.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:05 smithi179 ceph-mon[43975]: pgmap v756: 37 pgs: 2 peering, 9 unknown, 4 active+recovering+undersized+degraded+remapped, 22 active+clean; 5.3 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 401 op/s; 2708/260931 objects degraded (1.038%); 22029/260931 objects misplaced (8.442%) 2024-02-09T17:33:05.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:05 smithi162 ceph-mon[44304]: 3.7 scrub starts 2024-02-09T17:33:05.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:05 smithi162 ceph-mon[44304]: osdmap e266: 12 total, 12 up, 11 in 2024-02-09T17:33:05.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:05 smithi162 ceph-mon[44304]: pgmap v756: 37 pgs: 2 peering, 9 unknown, 4 active+recovering+undersized+degraded+remapped, 22 active+clean; 5.3 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 401 op/s; 2708/260931 objects degraded (1.038%); 22029/260931 objects misplaced (8.442%) 2024-02-09T17:33:06.049 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:33:06.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:06.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:33:06.157 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:33:06.209 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:06.255 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:33:06.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:06.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:33:06.371 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:33:06.406 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:06 smithi099 ceph-mon[39420]: osdmap e267: 12 total, 12 up, 11 in 2024-02-09T17:33:06.406 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:06 smithi099 ceph-mon[39420]: 1.14 scrub starts 2024-02-09T17:33:06.406 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:06 smithi099 ceph-mon[39420]: 1.14 scrub ok 2024-02-09T17:33:06.406 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:06 smithi099 ceph-mon[39420]: 1.11 scrub starts 2024-02-09T17:33:06.406 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:06 smithi099 ceph-mon[39420]: 3.7 scrub ok 2024-02-09T17:33:06.406 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:06 smithi099 ceph-mon[39420]: 1.11 scrub ok 2024-02-09T17:33:06.406 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:06 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 2708/260931 objects degraded (1.038%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:33:06.408 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:06.430 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:33:06.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:06.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:33:06.537 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:33:06.589 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:06 smithi179 ceph-mon[43975]: osdmap e267: 12 total, 12 up, 11 in 2024-02-09T17:33:06.589 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:06 smithi179 ceph-mon[43975]: 1.14 scrub starts 2024-02-09T17:33:06.590 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:06 smithi179 ceph-mon[43975]: 1.14 scrub ok 2024-02-09T17:33:06.590 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:06 smithi179 ceph-mon[43975]: 1.11 scrub starts 2024-02-09T17:33:06.590 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:06 smithi179 ceph-mon[43975]: 3.7 scrub ok 2024-02-09T17:33:06.590 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:06 smithi179 ceph-mon[43975]: 1.11 scrub ok 2024-02-09T17:33:06.590 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:06 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 2708/260931 objects degraded (1.038%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:33:06.592 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:06.623 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:33:06.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:06.630 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:33:06.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:06 smithi162 ceph-mon[44304]: osdmap e267: 12 total, 12 up, 11 in 2024-02-09T17:33:06.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:06 smithi162 ceph-mon[44304]: 1.14 scrub starts 2024-02-09T17:33:06.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:06 smithi162 ceph-mon[44304]: 1.14 scrub ok 2024-02-09T17:33:06.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:06 smithi162 ceph-mon[44304]: 1.11 scrub starts 2024-02-09T17:33:06.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:06 smithi162 ceph-mon[44304]: 3.7 scrub ok 2024-02-09T17:33:06.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:06 smithi162 ceph-mon[44304]: 1.11 scrub ok 2024-02-09T17:33:06.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:06 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 2708/260931 objects degraded (1.038%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:33:06.731 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:33:06.790 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:06.823 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:33:06.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:06.830 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:33:06.931 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:33:06.983 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:07.042 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:33:07.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:07.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:33:07.161 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:33:07.190 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:07.267 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:33:07.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:07.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:33:07.301 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-09T17:33:07.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:07 smithi162 ceph-mon[44304]: 1.c scrub starts 2024-02-09T17:33:07.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:07 smithi162 ceph-mon[44304]: 1.c scrub ok 2024-02-09T17:33:07.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:07 smithi162 ceph-mon[44304]: 1.12 scrub starts 2024-02-09T17:33:07.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:07 smithi162 ceph-mon[44304]: 1.12 scrub ok 2024-02-09T17:33:07.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:07 smithi162 ceph-mon[44304]: pgmap v758: 37 pgs: 4 peering, 5 unknown, 4 active+recovering+undersized+degraded+remapped, 24 active+clean; 5.2 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 421 op/s; 2878/256593 objects degraded (1.122%); 21313/256593 objects misplaced (8.306%); 7.9 MiB/s, 126 objects/s recovering 2024-02-09T17:33:07.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:07 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2576675000' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:33:07.312 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is unset 2024-02-09T17:33:07.375 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:33:07.424 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:07.475 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:33:07.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:07.478 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:33:07.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:07 smithi099 ceph-mon[39420]: 1.c scrub starts 2024-02-09T17:33:07.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:07 smithi099 ceph-mon[39420]: 1.c scrub ok 2024-02-09T17:33:07.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:07 smithi099 ceph-mon[39420]: 1.12 scrub starts 2024-02-09T17:33:07.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:07 smithi099 ceph-mon[39420]: 1.12 scrub ok 2024-02-09T17:33:07.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:07 smithi099 ceph-mon[39420]: pgmap v758: 37 pgs: 4 peering, 5 unknown, 4 active+recovering+undersized+degraded+remapped, 24 active+clean; 5.2 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 421 op/s; 2878/256593 objects degraded (1.122%); 21313/256593 objects misplaced (8.306%); 7.9 MiB/s, 126 objects/s recovering 2024-02-09T17:33:07.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:07 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2576675000' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:33:07.579 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:33:07.617 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:07.636 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:33:07.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:07.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:33:07.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:07 smithi179 ceph-mon[43975]: 1.c scrub starts 2024-02-09T17:33:07.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:07 smithi179 ceph-mon[43975]: 1.c scrub ok 2024-02-09T17:33:07.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:07 smithi179 ceph-mon[43975]: 1.12 scrub starts 2024-02-09T17:33:07.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:07 smithi179 ceph-mon[43975]: 1.12 scrub ok 2024-02-09T17:33:07.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:07 smithi179 ceph-mon[43975]: pgmap v758: 37 pgs: 4 peering, 5 unknown, 4 active+recovering+undersized+degraded+remapped, 24 active+clean; 5.2 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 421 op/s; 2878/256593 objects degraded (1.122%); 21313/256593 objects misplaced (8.306%); 7.9 MiB/s, 126 objects/s recovering 2024-02-09T17:33:07.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:07 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2576675000' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:33:07.743 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:33:07.776 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:07.808 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:33:07.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:07.811 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:33:07.912 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:33:07.999 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:08.049 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:33:08.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:08.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:33:08.160 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:33:08.208 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:08.239 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:33:08.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:08.244 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:33:08.344 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:33:08.381 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:08.440 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:33:08.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:08.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:33:08.551 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:33:08.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:08 smithi179 ceph-mon[43975]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-09T17:33:08.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:08 smithi179 ceph-mon[43975]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:33:08.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:08 smithi179 ceph-mon[43975]: 1.10 scrub starts 2024-02-09T17:33:08.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:08 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2576675000' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:33:08.587 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:08 smithi179 ceph-mon[43975]: osdmap e268: 12 total, 12 up, 11 in 2024-02-09T17:33:08.587 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:08 smithi179 ceph-mon[43975]: 1.10 scrub ok 2024-02-09T17:33:08.590 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:08.620 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:33:08.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:08.624 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:33:08.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:08 smithi162 ceph-mon[44304]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-09T17:33:08.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:08 smithi162 ceph-mon[44304]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:33:08.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:08 smithi162 ceph-mon[44304]: 1.10 scrub starts 2024-02-09T17:33:08.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:08 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2576675000' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:33:08.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:08 smithi162 ceph-mon[44304]: osdmap e268: 12 total, 12 up, 11 in 2024-02-09T17:33:08.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:08 smithi162 ceph-mon[44304]: 1.10 scrub ok 2024-02-09T17:33:08.725 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:33:08.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:08 smithi099 ceph-mon[39420]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-09T17:33:08.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:08 smithi099 ceph-mon[39420]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:33:08.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:08 smithi099 ceph-mon[39420]: 1.10 scrub starts 2024-02-09T17:33:08.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:08 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2576675000' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:33:08.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:08 smithi099 ceph-mon[39420]: osdmap e268: 12 total, 12 up, 11 in 2024-02-09T17:33:08.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:08 smithi099 ceph-mon[39420]: 1.10 scrub ok 2024-02-09T17:33:08.754 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:33:08.783 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:33:08.787 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:08.804 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:33:08.821 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:33:08.823 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:33:08.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:08.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:33:08.861 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:33:08.885 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:33:08.922 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:33:08.929 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:33:08.967 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:08.974 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:33:09.009 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:33:09.041 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:33:09.076 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:33:09.080 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:33:09.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:09.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:33:09.121 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:33:09.144 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:33:09.187 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:33:09.198 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:33:09.200 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:33:09.200 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:33:09.200 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:33:09.201 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:33:09.201 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:33:09.201 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:33:09.201 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:33:09.201 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:33:09.201 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:33:09.201 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:33:09.201 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:33:09.225 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:09.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:09 smithi099 ceph-mon[39420]: 3.2 scrub starts 2024-02-09T17:33:09.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:09 smithi099 ceph-mon[39420]: pgmap v760: 37 pgs: 4 peering, 2 unknown, 4 active+recovering+undersized+degraded+remapped, 27 active+clean; 5.2 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.9 KiB/s rd, 51 KiB/s wr, 566 op/s; 2878/253149 objects degraded (1.137%); 21313/253149 objects misplaced (8.419%); 6.4 MiB/s, 103 objects/s recovering 2024-02-09T17:33:09.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:09 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/327750374' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:33:09.581 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:33:09.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:09.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:33:09.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:09 smithi179 ceph-mon[43975]: 3.2 scrub starts 2024-02-09T17:33:09.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:09 smithi179 ceph-mon[43975]: pgmap v760: 37 pgs: 4 peering, 2 unknown, 4 active+recovering+undersized+degraded+remapped, 27 active+clean; 5.2 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.9 KiB/s rd, 51 KiB/s wr, 566 op/s; 2878/253149 objects degraded (1.137%); 21313/253149 objects misplaced (8.419%); 6.4 MiB/s, 103 objects/s recovering 2024-02-09T17:33:09.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:09 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/327750374' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:33:09.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:09 smithi162 ceph-mon[44304]: 3.2 scrub starts 2024-02-09T17:33:09.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:09 smithi162 ceph-mon[44304]: pgmap v760: 37 pgs: 4 peering, 2 unknown, 4 active+recovering+undersized+degraded+remapped, 27 active+clean; 5.2 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 1.9 KiB/s rd, 51 KiB/s wr, 566 op/s; 2878/253149 objects degraded (1.137%); 21313/253149 objects misplaced (8.419%); 6.4 MiB/s, 103 objects/s recovering 2024-02-09T17:33:09.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:09 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/327750374' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:33:09.685 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:33:09.737 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:09.761 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:33:09.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:09.765 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:33:09.865 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:33:09.896 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:10.048 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:33:10.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:10.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:33:10.152 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:33:10.189 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:10.470 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set noscrub 2024-02-09T17:33:10.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:10 smithi099 ceph-mon[39420]: 3.2 scrub ok 2024-02-09T17:33:10.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:10 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:33:10.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:10 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:33:10.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:10 smithi179 ceph-mon[43975]: 3.2 scrub ok 2024-02-09T17:33:10.677 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:10 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:33:10.677 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:10 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:33:10.684 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:10 smithi162 ceph-mon[44304]: 3.2 scrub ok 2024-02-09T17:33:10.684 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:10 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:33:10.684 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:10 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:33:11.250 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:33:11.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:11.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:33:11.257 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:33:11.355 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:33:11.384 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:11.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:11 smithi179 ceph-mon[43975]: osdmap e269: 12 total, 12 up, 11 in 2024-02-09T17:33:11.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:11 smithi179 ceph-mon[43975]: pgmap v762: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 5.0 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.2 KiB/s rd, 30 KiB/s wr, 717 op/s; 3102/248070 objects degraded (1.250%); 20557/248070 objects misplaced (8.287%); 11 MiB/s, 179 objects/s recovering 2024-02-09T17:33:11.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:11 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 2878/253149 objects degraded (1.137%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:33:11.688 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:11 smithi162 ceph-mon[44304]: osdmap e269: 12 total, 12 up, 11 in 2024-02-09T17:33:11.688 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:11 smithi162 ceph-mon[44304]: pgmap v762: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 5.0 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.2 KiB/s rd, 30 KiB/s wr, 717 op/s; 3102/248070 objects degraded (1.250%); 20557/248070 objects misplaced (8.287%); 11 MiB/s, 179 objects/s recovering 2024-02-09T17:33:11.688 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:11 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 2878/253149 objects degraded (1.137%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:33:11.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:11 smithi099 ceph-mon[39420]: osdmap e269: 12 total, 12 up, 11 in 2024-02-09T17:33:11.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:11 smithi099 ceph-mon[39420]: pgmap v762: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 5.0 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 1.2 KiB/s rd, 30 KiB/s wr, 717 op/s; 3102/248070 objects degraded (1.250%); 20557/248070 objects misplaced (8.287%); 11 MiB/s, 179 objects/s recovering 2024-02-09T17:33:11.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:11 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 2878/253149 objects degraded (1.137%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:33:12.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:12 smithi179 ceph-mon[43975]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-09T17:33:12.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:12 smithi162 ceph-mon[44304]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-09T17:33:12.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:12 smithi099 ceph-mon[39420]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-09T17:33:12.774 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:33:13.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:13 smithi179 ceph-mon[43975]: pgmap v763: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 5.0 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 281 B/s rd, 6.2 KiB/s wr, 602 op/s; 3102/244023 objects degraded (1.271%); 20557/244023 objects misplaced (8.424%); 4.5 MiB/s, 73 objects/s recovering 2024-02-09T17:33:13.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:13 smithi162 ceph-mon[44304]: pgmap v763: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 5.0 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 281 B/s rd, 6.2 KiB/s wr, 602 op/s; 3102/244023 objects degraded (1.271%); 20557/244023 objects misplaced (8.424%); 4.5 MiB/s, 73 objects/s recovering 2024-02-09T17:33:13.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:13 smithi099 ceph-mon[39420]: pgmap v763: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 5.0 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 281 B/s rd, 6.2 KiB/s wr, 602 op/s; 3102/244023 objects degraded (1.271%); 20557/244023 objects misplaced (8.424%); 4.5 MiB/s, 73 objects/s recovering 2024-02-09T17:33:13.835 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:33:13.839 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:13.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:33:13.941 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:33:13.997 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:14.030 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:33:14.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:14.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:33:14.134 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:33:14.196 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:14.237 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:33:14.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:14.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:33:14.345 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:33:14.383 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:14.411 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:33:14.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:14.416 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:33:14.517 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:33:14.557 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:14.582 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:33:14.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:14.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:33:14.621 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:33:14.621 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [], 2024-02-09T17:33:14.621 INFO:teuthology.orchestra.run.smithi099.stdout: "complaint_time": 30, 2024-02-09T17:33:14.621 INFO:teuthology.orchestra.run.smithi099.stdout: "num_blocked_ops": 0 2024-02-09T17:33:14.621 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:33:14.690 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:33:14.721 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:15.586 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:33:15.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:15.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:33:15.691 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:33:15.761 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:15.788 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:33:15.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:15.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:33:15.844 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:15 smithi099 ceph-mon[39420]: pgmap v764: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 4.9 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 255 B/s rd, 5.6 KiB/s wr, 570 op/s; 3102/242913 objects degraded (1.277%); 20557/242913 objects misplaced (8.463%); 4.1 MiB/s, 66 objects/s recovering 2024-02-09T17:33:15.901 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:33:15.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:15 smithi179 ceph-mon[43975]: pgmap v764: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 4.9 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 255 B/s rd, 5.6 KiB/s wr, 570 op/s; 3102/242913 objects degraded (1.277%); 20557/242913 objects misplaced (8.463%); 4.1 MiB/s, 66 objects/s recovering 2024-02-09T17:33:15.933 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:15.934 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:15 smithi162 ceph-mon[44304]: pgmap v764: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 4.9 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 255 B/s rd, 5.6 KiB/s wr, 570 op/s; 3102/242913 objects degraded (1.277%); 20557/242913 objects misplaced (8.463%); 4.1 MiB/s, 66 objects/s recovering 2024-02-09T17:33:16.638 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-09T17:33:16.865 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:16 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 3102/242913 objects degraded (1.277%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:33:16.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:16 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 3102/242913 objects degraded (1.277%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:33:16.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:16 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 3102/242913 objects degraded (1.277%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:33:17.131 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:33:17.437 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:33:17.438 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-02-09T17:33:17.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:17 smithi099 ceph-mon[39420]: pgmap v765: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 4.8 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 533 op/s; 3314/238311 objects degraded (1.391%); 19787/238311 objects misplaced (8.303%); 7.3 MiB/s, 117 objects/s recovering 2024-02-09T17:33:17.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:17 smithi179 ceph-mon[43975]: pgmap v765: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 4.8 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 533 op/s; 3314/238311 objects degraded (1.391%); 19787/238311 objects misplaced (8.303%); 7.3 MiB/s, 117 objects/s recovering 2024-02-09T17:33:17.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:17 smithi162 ceph-mon[44304]: pgmap v765: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 4.8 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 533 op/s; 3314/238311 objects degraded (1.391%); 19787/238311 objects misplaced (8.303%); 7.3 MiB/s, 117 objects/s recovering 2024-02-09T17:33:18.019 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:33:18.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:18.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:33:18.124 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:33:18.154 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:18.281 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:33:18.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:18.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:33:18.385 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:33:18.426 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:18.466 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:33:18.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:18.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:33:18.571 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:33:18.636 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:18.658 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:18 smithi099 ceph-mon[39420]: from='client.15300 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:33:18.659 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:18 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4101055188' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:33:18.659 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:18 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:33:18.662 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:33:18.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:18.665 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:33:18.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:18 smithi162 ceph-mon[44304]: from='client.15300 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:33:18.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:18 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4101055188' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:33:18.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:18 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:33:18.766 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:33:18.798 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:18.854 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:33:18.860 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:18.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:33:18.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:18 smithi179 ceph-mon[43975]: from='client.15300 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:33:18.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:18 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4101055188' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:33:18.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:18 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:33:18.961 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:33:19.032 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:19.074 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:33:19.079 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:19.079 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:33:19.180 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:33:19.212 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:19.243 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:33:19.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:19.256 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:33:19.356 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:33:19.397 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:19.433 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:33:19.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:19.436 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:33:19.537 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:33:19.567 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:19.663 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:19 smithi179 ceph-mon[43975]: pgmap v766: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 4.8 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 619 op/s; 3314/234546 objects degraded (1.413%); 19787/234546 objects misplaced (8.436%); 6.8 MiB/s, 108 objects/s recovering 2024-02-09T17:33:19.663 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:19 smithi179 ceph-mon[43975]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:33:19.663 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:19 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:33:19.663 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:19 smithi179 ceph-mon[43975]: osdmap e270: 12 total, 12 up, 11 in 2024-02-09T17:33:19.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:19 smithi162 ceph-mon[44304]: pgmap v766: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 4.8 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 619 op/s; 3314/234546 objects degraded (1.413%); 19787/234546 objects misplaced (8.436%); 6.8 MiB/s, 108 objects/s recovering 2024-02-09T17:33:19.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:19 smithi162 ceph-mon[44304]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:33:19.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:19 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:33:19.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:19 smithi162 ceph-mon[44304]: osdmap e270: 12 total, 12 up, 11 in 2024-02-09T17:33:19.958 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:19 smithi099 ceph-mon[39420]: pgmap v766: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 4.8 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 619 op/s; 3314/234546 objects degraded (1.413%); 19787/234546 objects misplaced (8.436%); 6.8 MiB/s, 108 objects/s recovering 2024-02-09T17:33:19.958 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:19 smithi099 ceph-mon[39420]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:33:19.958 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:19 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:33:19.958 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:19 smithi099 ceph-mon[39420]: osdmap e270: 12 total, 12 up, 11 in 2024-02-09T17:33:19.991 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":765,"stamp":"2024-02-09T17:33:16.562352+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5206797429,"num_objects":79437,"num_object_clones":0,"num_object_copies":238311,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3314,"num_objects_misplaced":19787,"num_objects_unfound":0,"num_objects_dirty":79437,"num_whiteouts":0,"num_read":90,"num_read_kb":78,"num_write":184606,"num_write_kb":8447276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14433,"num_bytes_recovered":944838294,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":163819,"ondisk_log_size":163819,"up":111,"acting":107,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":111,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":15900624,"kb_used_data":13981424,"kb_used_omap":228,"kb_used_meta":1918619,"kb_avail":1015296048,"statfs":{"total":1055945392128,"available":1039663153152,"internally_reserved":0,"allocated":14316978176,"data_stored":14312848276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":233532,"internal_metadata":1964666820},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,10,2,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-324158946,"num_objects":-4946,"num_object_clones":0,"num_object_copies":-14838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":436,"num_objects_misplaced":-1526,"num_objects_unfound":0,"num_objects_dirty":-4946,"num_whiteouts":0,"num_read":-4,"num_read_kb":-3,"num_write":4940,"num_write_kb":-89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1090,"num_bytes_recovered":71091955,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.252680"},"pg_stats":[{"pgid":"1.14","version":"63'139","reported_seq":747,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-02-09T17:33:10.329370+0000","last_change":"2024-02-09T17:33:05.385797+0000","last_active":"2024-02-09T17:33:10.329370+0000","last_peered":"2024-02-09T17:33:10.329370+0000","last_clean":"2024-02-09T17:33:10.329370+0000","last_became_active":"2024-02-09T17:33:05.309643+0000","last_became_peered":"2024-02-09T17:33:05.309643+0000","last_unstale":"2024-02-09T17:33:10.329370+0000","last_undegraded":"2024-02-09T17:33:10.329370+0000","last_fullsized":"2024-02-09T17:33:10.329370+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:33:05.385701+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:33:05.385701+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:34:11.311998+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"63'139","reported_seq":777,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-02-09T17:33:10.346540+0000","last_change":"2024-02-09T17:33:00.284769+0000","last_active":"2024-02-09T17:33:10.346540+0000","last_peered":"2024-02-09T17:33:10.346540+0000","last_clean":"2024-02-09T17:33:10.346540+0000","last_became_active":"2024-02-09T17:33:00.284421+0000","last_became_peered":"2024-02-09T17:33:00.284421+0000","last_unstale":"2024-02-09T17:33:10.346540+0000","last_undegraded":"2024-02-09T17:33:10.346540+0000","last_fullsized":"2024-02-09T17:33:10.346540+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:31:59.409353+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:31:59.409353+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:33:13.109937+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26834,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":8,"num_write_kb":133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":120210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"269'11488","reported_seq":11786,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-02-09T17:33:11.684607+0000","last_change":"2024-02-09T17:33:09.200350+0000","last_active":"2024-02-09T17:33:11.684607+0000","last_peered":"2024-02-09T17:33:11.684607+0000","last_clean":"2024-02-09T17:33:11.684607+0000","last_became_active":"2024-02-09T17:29:47.581341+0000","last_became_peered":"2024-02-09T17:29:47.581341+0000","last_unstale":"2024-02-09T17:33:11.684607+0000","last_undegraded":"2024-02-09T17:33:11.684607+0000","last_fullsized":"2024-02-09T17:33:11.684607+0000","mapping_epoch":199,"log_start":"209'1400","ondisk_log_start":"209'1400","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'11391","last_scrub_stamp":"2024-02-09T17:33:09.200126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:33:09.200126+0000","objects_scrubbed":5182,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:34:26.693863+0000","scrub_duration":1026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332529664,"num_objects":5074,"num_object_clones":0,"num_object_copies":15222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11488,"num_write_kb":529984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"269'11742","reported_seq":11959,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-02-09T17:33:14.717760+0000","last_change":"2024-02-09T17:30:55.910896+0000","last_active":"2024-02-09T17:33:14.717760+0000","last_peered":"2024-02-09T17:33:14.717760+0000","last_clean":"2024-02-09T17:33:14.717760+0000","last_became_active":"2024-02-09T17:29:47.582887+0000","last_became_peered":"2024-02-09T17:29:47.582887+0000","last_unstale":"2024-02-09T17:33:14.717760+0000","last_undegraded":"2024-02-09T17:33:14.717760+0000","last_fullsized":"2024-02-09T17:33:14.717760+0000","mapping_epoch":199,"log_start":"209'1700","ondisk_log_start":"209'1700","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"218'5796","last_scrub_stamp":"2024-02-09T17:30:55.910807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:30:55.910807+0000","objects_scrubbed":5754,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1081,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":312606720,"num_objects":4770,"num_object_clones":0,"num_object_copies":14310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11742,"num_write_kb":528384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,6],"acting":[11,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.2","version":"63'139","reported_seq":784,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-02-09T17:33:10.346831+0000","last_change":"2024-02-09T17:33:00.281314+0000","last_active":"2024-02-09T17:33:10.346831+0000","last_peered":"2024-02-09T17:33:10.346831+0000","last_clean":"2024-02-09T17:33:10.346831+0000","last_became_active":"2024-02-09T17:33:00.280651+0000","last_became_peered":"2024-02-09T17:33:00.280651+0000","last_unstale":"2024-02-09T17:33:10.346831+0000","last_undegraded":"2024-02-09T17:33:10.346831+0000","last_fullsized":"2024-02-09T17:33:10.346831+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:31:59.409353+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:31:59.409353+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:33:13.109986+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":8,"num_write_kb":133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":120211,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,8],"acting":[7,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"269'11552","reported_seq":15312,"reported_epoch":269,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-02-09T17:33:14.958865+0000","last_change":"2024-02-09T17:32:01.551242+0000","last_active":"2024-02-09T17:33:14.958865+0000","last_peered":"2024-02-09T17:33:14.958865+0000","last_clean":"2024-02-09T17:31:59.232029+0000","last_became_active":"2024-02-09T17:32:01.494982+0000","last_became_peered":"2024-02-09T17:32:01.494982+0000","last_unstale":"2024-02-09T17:33:14.958865+0000","last_undegraded":"2024-02-09T17:32:01.319667+0000","last_fullsized":"2024-02-09T17:32:01.276864+0000","mapping_epoch":238,"log_start":"209'1500","ondisk_log_start":"209'1500","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"220'6050","last_scrub_stamp":"2024-02-09T17:31:00.396342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:31:00.396342+0000","objects_scrubbed":6003,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1427,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319553536,"num_objects":4876,"num_object_clones":0,"num_object_copies":14628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":983,"num_objects_misplaced":4876,"num_objects_unfound":0,"num_objects_dirty":4876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11552,"num_write_kb":525696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1770,"num_bytes_recovered":115343360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,11],"acting":[4,10],"avail_no_missing":["4","0","10"],"object_location_counts":[{"shards":"0,4,10","objects":4876},{"shards":"4,10","objects":-983}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"63'139","reported_seq":299,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-02-09T17:33:10.341493+0000","last_change":"2024-02-09T17:33:00.279728+0000","last_active":"2024-02-09T17:33:10.341493+0000","last_peered":"2024-02-09T17:33:10.341493+0000","last_clean":"2024-02-09T17:33:10.341493+0000","last_became_active":"2024-02-09T17:33:00.279369+0000","last_became_peered":"2024-02-09T17:33:00.279369+0000","last_unstale":"2024-02-09T17:33:10.341493+0000","last_undegraded":"2024-02-09T17:33:10.341493+0000","last_fullsized":"2024-02-09T17:33:10.341493+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:31:59.409353+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:31:59.409353+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:33:11.697400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17890,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":5,"num_write_kb":89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":48084,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"269'11726","reported_seq":14971,"reported_epoch":269,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-02-09T17:33:14.943733+0000","last_change":"2024-02-09T17:32:01.643212+0000","last_active":"2024-02-09T17:33:14.943733+0000","last_peered":"2024-02-09T17:33:14.943733+0000","last_clean":"2024-02-09T17:31:59.234884+0000","last_became_active":"2024-02-09T17:32:01.587380+0000","last_became_peered":"2024-02-09T17:32:01.587380+0000","last_unstale":"2024-02-09T17:33:14.943733+0000","last_undegraded":"2024-02-09T17:32:01.320756+0000","last_fullsized":"2024-02-09T17:32:01.276816+0000","mapping_epoch":238,"log_start":"209'1700","ondisk_log_start":"209'1700","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'7835","last_scrub_stamp":"2024-02-09T17:31:26.001797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:31:26.001797+0000","objects_scrubbed":7767,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2765,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328073216,"num_objects":5006,"num_object_clones":0,"num_object_copies":15018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1249,"num_objects_misplaced":5005,"num_objects_unfound":0,"num_objects_dirty":5006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11726,"num_write_kb":535424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1502,"num_bytes_recovered":97648640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,11],"acting":[4,6],"avail_no_missing":["4","0","6"],"object_location_counts":[{"shards":"0,4,6","objects":5005},{"shards":"4,6","objects":-1248}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"63'139","reported_seq":300,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-02-09T17:33:10.379846+0000","last_change":"2024-02-09T17:33:00.291337+0000","last_active":"2024-02-09T17:33:10.379846+0000","last_peered":"2024-02-09T17:33:10.379846+0000","last_clean":"2024-02-09T17:33:10.379846+0000","last_became_active":"2024-02-09T17:33:00.290506+0000","last_became_peered":"2024-02-09T17:33:00.290506+0000","last_unstale":"2024-02-09T17:33:10.379846+0000","last_undegraded":"2024-02-09T17:33:10.379846+0000","last_fullsized":"2024-02-09T17:33:10.379846+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:31:59.409353+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:31:59.409353+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:33:11.741023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":8,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":72127,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,2],"acting":[10,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.4","version":"269'11536","reported_seq":11832,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-02-09T17:33:15.033866+0000","last_change":"2024-02-09T17:31:59.706801+0000","last_active":"2024-02-09T17:33:15.033866+0000","last_peered":"2024-02-09T17:33:15.033866+0000","last_clean":"2024-02-09T17:33:15.033866+0000","last_became_active":"2024-02-09T17:29:47.582922+0000","last_became_peered":"2024-02-09T17:29:47.582922+0000","last_unstale":"2024-02-09T17:33:15.033866+0000","last_undegraded":"2024-02-09T17:33:15.033866+0000","last_fullsized":"2024-02-09T17:33:15.033866+0000","mapping_epoch":199,"log_start":"209'1500","ondisk_log_start":"209'1500","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'8794","last_scrub_stamp":"2024-02-09T17:31:59.706688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:31:59.706688+0000","objects_scrubbed":7672,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1299,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":322437120,"num_objects":4920,"num_object_clones":0,"num_object_copies":14760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11536,"num_write_kb":526592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"63'139","reported_seq":300,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-02-09T17:33:10.377107+0000","last_change":"2024-02-09T17:33:00.285366+0000","last_active":"2024-02-09T17:33:10.377107+0000","last_peered":"2024-02-09T17:33:10.377107+0000","last_clean":"2024-02-09T17:33:10.377107+0000","last_became_active":"2024-02-09T17:33:00.284702+0000","last_became_peered":"2024-02-09T17:33:00.284702+0000","last_unstale":"2024-02-09T17:33:10.377107+0000","last_undegraded":"2024-02-09T17:33:10.377107+0000","last_fullsized":"2024-02-09T17:33:10.377107+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:31:59.409353+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:31:59.409353+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:33:11.741052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":8,"num_write_kb":133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":72126,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.5","version":"269'11503","reported_seq":11800,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-02-09T17:33:14.265013+0000","last_change":"2024-02-09T17:32:57.719632+0000","last_active":"2024-02-09T17:33:14.265013+0000","last_peered":"2024-02-09T17:33:14.265013+0000","last_clean":"2024-02-09T17:33:14.265013+0000","last_became_active":"2024-02-09T17:29:47.581119+0000","last_became_peered":"2024-02-09T17:29:47.581119+0000","last_unstale":"2024-02-09T17:33:14.265013+0000","last_undegraded":"2024-02-09T17:33:14.265013+0000","last_fullsized":"2024-02-09T17:33:14.265013+0000","mapping_epoch":199,"log_start":"209'1500","ondisk_log_start":"209'1500","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'10869","last_scrub_stamp":"2024-02-09T17:32:57.719380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:32:57.719380+0000","objects_scrubbed":5523,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:34:22.231484+0000","scrub_duration":1424,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319094784,"num_objects":4869,"num_object_clones":0,"num_object_copies":14607,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11503,"num_write_kb":523904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"63'139","reported_seq":306,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-02-09T17:33:10.334645+0000","last_change":"2024-02-09T17:33:00.279791+0000","last_active":"2024-02-09T17:33:10.334645+0000","last_peered":"2024-02-09T17:33:10.334645+0000","last_clean":"2024-02-09T17:33:10.334645+0000","last_became_active":"2024-02-09T17:33:00.278966+0000","last_became_peered":"2024-02-09T17:33:00.278966+0000","last_unstale":"2024-02-09T17:33:10.334645+0000","last_undegraded":"2024-02-09T17:33:10.334645+0000","last_fullsized":"2024-02-09T17:33:10.334645+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:31:59.409353+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:31:59.409353+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:33:12.716066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":8,"num_write_kb":133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":72126,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"269'11365","reported_seq":16576,"reported_epoch":269,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-02-09T17:33:15.096199+0000","last_change":"2024-02-09T17:32:01.583215+0000","last_active":"2024-02-09T17:33:15.096199+0000","last_peered":"2024-02-09T17:33:15.096199+0000","last_clean":"2024-02-09T17:31:59.217197+0000","last_became_active":"2024-02-09T17:32:01.532489+0000","last_became_peered":"2024-02-09T17:32:01.532489+0000","last_unstale":"2024-02-09T17:33:15.096199+0000","last_undegraded":"2024-02-09T17:32:01.315621+0000","last_fullsized":"2024-02-09T17:32:01.278340+0000","mapping_epoch":238,"log_start":"209'1300","ondisk_log_start":"209'1300","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'7634","last_scrub_stamp":"2024-02-09T17:31:25.792160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:31:25.792160+0000","objects_scrubbed":7589,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1716,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320405504,"num_objects":4889,"num_object_clones":0,"num_object_copies":14667,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":190,"num_objects_misplaced":4885,"num_objects_unfound":0,"num_objects_dirty":4889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11365,"num_write_kb":520128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2487,"num_bytes_recovered":162988032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4],"avail_no_missing":["1","0","4"],"object_location_counts":[{"shards":"0,1,4","objects":4885},{"shards":"1,4","objects":-186}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"63'139","reported_seq":392,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-02-09T17:33:10.346092+0000","last_change":"2024-02-09T17:33:00.285213+0000","last_active":"2024-02-09T17:33:10.346092+0000","last_peered":"2024-02-09T17:33:10.346092+0000","last_clean":"2024-02-09T17:33:10.346092+0000","last_became_active":"2024-02-09T17:33:00.284924+0000","last_became_peered":"2024-02-09T17:33:00.284924+0000","last_unstale":"2024-02-09T17:33:10.346092+0000","last_undegraded":"2024-02-09T17:33:10.346092+0000","last_fullsized":"2024-02-09T17:33:10.346092+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:31:59.409353+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:31:59.409353+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:33:12.463393+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17890,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":5,"num_write_kb":89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":173339,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.7","version":"269'11362","reported_seq":11689,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-02-09T17:33:11.668934+0000","last_change":"2024-02-09T17:33:05.405937+0000","last_active":"2024-02-09T17:33:11.668934+0000","last_peered":"2024-02-09T17:33:11.668934+0000","last_clean":"2024-02-09T17:33:11.668934+0000","last_became_active":"2024-02-09T17:29:47.583311+0000","last_became_peered":"2024-02-09T17:29:47.583311+0000","last_unstale":"2024-02-09T17:33:11.668934+0000","last_undegraded":"2024-02-09T17:33:11.668934+0000","last_fullsized":"2024-02-09T17:33:11.668934+0000","mapping_epoch":199,"log_start":"208'1300","ondisk_log_start":"208'1300","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'11099","last_scrub_stamp":"2024-02-09T17:33:05.405781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:33:05.405781+0000","objects_scrubbed":5338,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:34:13.708592+0000","scrub_duration":1218,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330694656,"num_objects":5046,"num_object_clones":0,"num_object_copies":15138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11362,"num_write_kb":525056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"63'139","reported_seq":300,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-02-09T17:33:10.376986+0000","last_change":"2024-02-09T17:33:00.274940+0000","last_active":"2024-02-09T17:33:10.376986+0000","last_peered":"2024-02-09T17:33:10.376986+0000","last_clean":"2024-02-09T17:33:10.376986+0000","last_became_active":"2024-02-09T17:33:00.274636+0000","last_became_peered":"2024-02-09T17:33:00.274636+0000","last_unstale":"2024-02-09T17:33:10.376986+0000","last_undegraded":"2024-02-09T17:33:10.376986+0000","last_fullsized":"2024-02-09T17:33:10.376986+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:31:59.409353+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:31:59.409353+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:33:11.740999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":8,"num_write_kb":133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":72126,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.8","version":"269'11559","reported_seq":11794,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-02-09T17:33:14.736580+0000","last_change":"2024-02-09T17:31:21.687596+0000","last_active":"2024-02-09T17:33:14.736580+0000","last_peered":"2024-02-09T17:33:14.736580+0000","last_clean":"2024-02-09T17:33:14.736580+0000","last_became_active":"2024-02-09T17:29:47.581467+0000","last_became_peered":"2024-02-09T17:29:47.581467+0000","last_unstale":"2024-02-09T17:33:14.736580+0000","last_undegraded":"2024-02-09T17:33:14.736580+0000","last_fullsized":"2024-02-09T17:33:14.736580+0000","mapping_epoch":199,"log_start":"209'1500","ondisk_log_start":"209'1500","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'7591","last_scrub_stamp":"2024-02-09T17:31:21.687400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:31:21.687400+0000","objects_scrubbed":7543,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1706,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326696960,"num_objects":4985,"num_object_clones":0,"num_object_copies":14955,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11559,"num_write_kb":529408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"63'139","reported_seq":295,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-02-09T17:33:10.335401+0000","last_change":"2024-02-09T17:32:35.418523+0000","last_active":"2024-02-09T17:33:10.335401+0000","last_peered":"2024-02-09T17:33:10.335401+0000","last_clean":"2024-02-09T17:33:10.335401+0000","last_became_active":"2024-02-09T17:32:35.418200+0000","last_became_peered":"2024-02-09T17:32:35.418200+0000","last_unstale":"2024-02-09T17:33:10.335401+0000","last_undegraded":"2024-02-09T17:33:10.335401+0000","last_fullsized":"2024-02-09T17:33:10.335401+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:31:59.409353+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:31:59.409353+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:33:11.362507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":16,"num_write_kb":267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":144253,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"269'11526","reported_seq":11847,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-02-09T17:33:11.827505+0000","last_change":"2024-02-09T17:33:00.629232+0000","last_active":"2024-02-09T17:33:11.827505+0000","last_peered":"2024-02-09T17:33:11.827505+0000","last_clean":"2024-02-09T17:33:11.827505+0000","last_became_active":"2024-02-09T17:29:47.582488+0000","last_became_peered":"2024-02-09T17:29:47.582488+0000","last_unstale":"2024-02-09T17:33:11.827505+0000","last_undegraded":"2024-02-09T17:33:11.827505+0000","last_fullsized":"2024-02-09T17:33:11.827505+0000","mapping_epoch":199,"log_start":"209'1500","ondisk_log_start":"209'1500","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'11120","last_scrub_stamp":"2024-02-09T17:33:00.628967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:33:00.628967+0000","objects_scrubbed":5528,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:34:04.464922+0000","scrub_duration":1394,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333709312,"num_objects":5092,"num_object_clones":0,"num_object_copies":15276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11526,"num_write_kb":531776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.b","version":"63'139","reported_seq":302,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-02-09T17:33:10.346149+0000","last_change":"2024-02-09T17:33:05.313527+0000","last_active":"2024-02-09T17:33:10.346149+0000","last_peered":"2024-02-09T17:33:10.346149+0000","last_clean":"2024-02-09T17:33:10.346149+0000","last_became_active":"2024-02-09T17:33:05.313189+0000","last_became_peered":"2024-02-09T17:33:05.313189+0000","last_unstale":"2024-02-09T17:33:10.346149+0000","last_undegraded":"2024-02-09T17:33:10.346149+0000","last_fullsized":"2024-02-09T17:33:10.346149+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:31:59.409353+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:31:59.409353+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:33:10.444737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17890,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":6,"num_write_kb":89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":48085,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,2],"acting":[11,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"269'11589","reported_seq":11825,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-02-09T17:33:15.578527+0000","last_change":"2024-02-09T17:31:25.346078+0000","last_active":"2024-02-09T17:33:15.578527+0000","last_peered":"2024-02-09T17:33:15.578527+0000","last_clean":"2024-02-09T17:33:15.578527+0000","last_became_active":"2024-02-09T17:29:47.581573+0000","last_became_peered":"2024-02-09T17:29:47.581573+0000","last_unstale":"2024-02-09T17:33:15.578527+0000","last_undegraded":"2024-02-09T17:33:15.578527+0000","last_fullsized":"2024-02-09T17:33:15.578527+0000","mapping_epoch":199,"log_start":"209'1500","ondisk_log_start":"209'1500","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'7710","last_scrub_stamp":"2024-02-09T17:31:25.345946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:31:25.345946+0000","objects_scrubbed":7649,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2400,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319750144,"num_objects":4879,"num_object_clones":0,"num_object_copies":14637,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11589,"num_write_kb":526976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"63'139","reported_seq":303,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-02-09T17:33:10.332307+0000","last_change":"2024-02-09T17:32:57.337095+0000","last_active":"2024-02-09T17:33:10.332307+0000","last_peered":"2024-02-09T17:33:10.332307+0000","last_clean":"2024-02-09T17:33:10.332307+0000","last_became_active":"2024-02-09T17:32:35.384617+0000","last_became_peered":"2024-02-09T17:32:35.384617+0000","last_unstale":"2024-02-09T17:33:10.332307+0000","last_undegraded":"2024-02-09T17:33:10.332307+0000","last_fullsized":"2024-02-09T17:33:10.332307+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:32:57.336980+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:32:57.336980+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:34:18.421998+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"269'14704","reported_seq":28802,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-02-09T17:33:11.676031+0000","last_change":"2024-02-09T17:33:00.367723+0000","last_active":"2024-02-09T17:33:11.676031+0000","last_peered":"2024-02-09T17:33:11.676031+0000","last_clean":"2024-02-09T17:33:11.676031+0000","last_became_active":"2024-02-09T17:32:43.289753+0000","last_became_peered":"2024-02-09T17:32:43.289753+0000","last_unstale":"2024-02-09T17:33:11.676031+0000","last_undegraded":"2024-02-09T17:33:11.676031+0000","last_fullsized":"2024-02-09T17:33:11.676031+0000","mapping_epoch":255,"log_start":"215'4700","ondisk_log_start":"215'4700","created":199,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'14281","last_scrub_stamp":"2024-02-09T17:33:00.367554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:33:00.367554+0000","objects_scrubbed":5471,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:34:29.482546+0000","scrub_duration":2217,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328794112,"num_objects":5017,"num_object_clones":0,"num_object_copies":15051,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11383,"num_write_kb":524800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6836,"num_bytes_recovered":448004096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,11],"acting":[3,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"63'139","reported_seq":779,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-02-09T17:33:10.346465+0000","last_change":"2024-02-09T17:32:35.402431+0000","last_active":"2024-02-09T17:33:10.346465+0000","last_peered":"2024-02-09T17:33:10.346465+0000","last_clean":"2024-02-09T17:33:10.346465+0000","last_became_active":"2024-02-09T17:32:35.402025+0000","last_became_peered":"2024-02-09T17:32:35.402025+0000","last_unstale":"2024-02-09T17:33:10.346465+0000","last_undegraded":"2024-02-09T17:33:10.346465+0000","last_fullsized":"2024-02-09T17:33:10.346465+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:31:59.409353+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:31:59.409353+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:33:13.109964+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":16,"num_write_kb":267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":240422,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"269'11489","reported_seq":11786,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-02-09T17:33:14.195999+0000","last_change":"2024-02-09T17:33:01.842594+0000","last_active":"2024-02-09T17:33:14.195999+0000","last_peered":"2024-02-09T17:33:14.195999+0000","last_clean":"2024-02-09T17:33:14.195999+0000","last_became_active":"2024-02-09T17:29:47.583200+0000","last_became_peered":"2024-02-09T17:29:47.583200+0000","last_unstale":"2024-02-09T17:33:14.195999+0000","last_undegraded":"2024-02-09T17:33:14.195999+0000","last_fullsized":"2024-02-09T17:33:14.195999+0000","mapping_epoch":199,"log_start":"209'1400","ondisk_log_start":"209'1400","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'11030","last_scrub_stamp":"2024-02-09T17:33:01.842463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:33:01.842463+0000","objects_scrubbed":5411,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:34:19.220446+0000","scrub_duration":1494,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":322240512,"num_objects":4917,"num_object_clones":0,"num_object_copies":14751,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11489,"num_write_kb":524992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"63'139","reported_seq":264,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-02-09T17:33:10.347492+0000","last_change":"2024-02-09T17:33:05.310194+0000","last_active":"2024-02-09T17:33:10.347492+0000","last_peered":"2024-02-09T17:33:10.347492+0000","last_clean":"2024-02-09T17:33:10.347492+0000","last_became_active":"2024-02-09T17:33:05.309824+0000","last_became_peered":"2024-02-09T17:33:05.309824+0000","last_unstale":"2024-02-09T17:33:10.347492+0000","last_undegraded":"2024-02-09T17:33:10.347492+0000","last_fullsized":"2024-02-09T17:33:10.347492+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:31:59.409353+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:31:59.409353+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:33:10.870562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":8,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":72127,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"269'11578","reported_seq":11815,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-02-09T17:33:11.788695+0000","last_change":"2024-02-09T17:31:24.785058+0000","last_active":"2024-02-09T17:33:11.788695+0000","last_peered":"2024-02-09T17:33:11.788695+0000","last_clean":"2024-02-09T17:33:11.788695+0000","last_became_active":"2024-02-09T17:29:47.583238+0000","last_became_peered":"2024-02-09T17:29:47.583238+0000","last_unstale":"2024-02-09T17:33:11.788695+0000","last_undegraded":"2024-02-09T17:33:11.788695+0000","last_fullsized":"2024-02-09T17:33:11.788695+0000","mapping_epoch":199,"log_start":"209'1500","ondisk_log_start":"209'1500","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'7776","last_scrub_stamp":"2024-02-09T17:31:24.784873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:31:24.784873+0000","objects_scrubbed":7733,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2469,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334823448,"num_objects":5110,"num_object_clones":0,"num_object_copies":15330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5110,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11578,"num_write_kb":533953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,9],"acting":[7,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"63'139","reported_seq":314,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-02-09T17:33:10.334542+0000","last_change":"2024-02-09T17:33:05.316508+0000","last_active":"2024-02-09T17:33:10.334542+0000","last_peered":"2024-02-09T17:33:10.334542+0000","last_clean":"2024-02-09T17:33:10.334542+0000","last_became_active":"2024-02-09T17:33:05.316100+0000","last_became_peered":"2024-02-09T17:33:05.316100+0000","last_unstale":"2024-02-09T17:33:10.334542+0000","last_undegraded":"2024-02-09T17:33:10.334542+0000","last_fullsized":"2024-02-09T17:33:10.334542+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:31:59.409353+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:31:59.409353+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:33:10.724439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":8,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":72127,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"269'11594","reported_seq":15500,"reported_epoch":269,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-02-09T17:33:14.783357+0000","last_change":"2024-02-09T17:32:01.597065+0000","last_active":"2024-02-09T17:33:14.783357+0000","last_peered":"2024-02-09T17:33:14.783357+0000","last_clean":"2024-02-09T17:31:59.261441+0000","last_became_active":"2024-02-09T17:32:01.541864+0000","last_became_peered":"2024-02-09T17:32:01.541864+0000","last_unstale":"2024-02-09T17:33:14.783357+0000","last_undegraded":"2024-02-09T17:32:01.324655+0000","last_fullsized":"2024-02-09T17:32:01.278077+0000","mapping_epoch":238,"log_start":"209'1500","ondisk_log_start":"209'1500","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'7702","last_scrub_stamp":"2024-02-09T17:31:22.871435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:31:22.871435+0000","objects_scrubbed":7650,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1911,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":329252864,"num_objects":5024,"num_object_clones":0,"num_object_copies":15072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":892,"num_objects_misplaced":5021,"num_objects_unfound":0,"num_objects_dirty":5024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11594,"num_write_kb":531776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1834,"num_bytes_recovered":119406592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9],"acting":[11,4],"avail_no_missing":["11","0","4"],"object_location_counts":[{"shards":"0,4,11","objects":5021},{"shards":"4,11","objects":-889}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"63'139","reported_seq":746,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-02-09T17:33:10.334208+0000","last_change":"2024-02-09T17:33:06.266559+0000","last_active":"2024-02-09T17:33:10.334208+0000","last_peered":"2024-02-09T17:33:10.334208+0000","last_clean":"2024-02-09T17:33:10.334208+0000","last_became_active":"2024-02-09T17:33:05.305259+0000","last_became_peered":"2024-02-09T17:33:05.305259+0000","last_unstale":"2024-02-09T17:33:10.334208+0000","last_undegraded":"2024-02-09T17:33:10.334208+0000","last_fullsized":"2024-02-09T17:33:10.334208+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:33:06.266459+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:33:06.266459+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:34:20.563670+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,9],"acting":[3,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"269'11487","reported_seq":11723,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-02-09T17:33:11.735721+0000","last_change":"2024-02-09T17:31:25.731053+0000","last_active":"2024-02-09T17:33:11.735721+0000","last_peered":"2024-02-09T17:33:11.735721+0000","last_clean":"2024-02-09T17:33:11.735721+0000","last_became_active":"2024-02-09T17:29:47.582245+0000","last_became_peered":"2024-02-09T17:29:47.582245+0000","last_unstale":"2024-02-09T17:33:11.735721+0000","last_undegraded":"2024-02-09T17:33:11.735721+0000","last_fullsized":"2024-02-09T17:33:11.735721+0000","mapping_epoch":199,"log_start":"209'1400","ondisk_log_start":"209'1400","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'7772","last_scrub_stamp":"2024-02-09T17:31:25.730848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:29:46.563947+0000","last_clean_scrub_stamp":"2024-02-09T17:31:25.730848+0000","objects_scrubbed":7721,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2441,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":325124096,"num_objects":4961,"num_object_clones":0,"num_object_copies":14883,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11487,"num_write_kb":526336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"63'139","reported_seq":263,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-02-09T17:33:10.347279+0000","last_change":"2024-02-09T17:33:05.320350+0000","last_active":"2024-02-09T17:33:10.347279+0000","last_peered":"2024-02-09T17:33:10.347279+0000","last_clean":"2024-02-09T17:33:10.347279+0000","last_became_active":"2024-02-09T17:33:05.319668+0000","last_became_peered":"2024-02-09T17:33:05.319668+0000","last_unstale":"2024-02-09T17:33:10.347279+0000","last_undegraded":"2024-02-09T17:33:10.347279+0000","last_fullsized":"2024-02-09T17:33:10.347279+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:31:59.409353+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:31:59.409353+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:33:11.097131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":9,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":72127,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"63'139","reported_seq":270,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-02-09T17:33:10.332175+0000","last_change":"2024-02-09T17:33:07.313242+0000","last_active":"2024-02-09T17:33:10.332175+0000","last_peered":"2024-02-09T17:33:10.332175+0000","last_clean":"2024-02-09T17:33:10.332175+0000","last_became_active":"2024-02-09T17:33:05.310104+0000","last_became_peered":"2024-02-09T17:33:05.310104+0000","last_unstale":"2024-02-09T17:33:10.332175+0000","last_undegraded":"2024-02-09T17:33:10.332175+0000","last_fullsized":"2024-02-09T17:33:10.332175+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:33:07.313108+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:33:07.313108+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:34:20.286945+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"63'139","reported_seq":271,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-02-09T17:33:10.344974+0000","last_change":"2024-02-09T17:33:05.449121+0000","last_active":"2024-02-09T17:33:10.344974+0000","last_peered":"2024-02-09T17:33:10.344974+0000","last_clean":"2024-02-09T17:33:10.344974+0000","last_became_active":"2024-02-09T17:33:05.303472+0000","last_became_peered":"2024-02-09T17:33:05.303472+0000","last_unstale":"2024-02-09T17:33:10.344974+0000","last_undegraded":"2024-02-09T17:33:10.344974+0000","last_fullsized":"2024-02-09T17:33:10.344974+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:33:05.448998+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:33:05.448998+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:34:08.025880+0000","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,3],"acting":[8,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.12","version":"63'139","reported_seq":752,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-02-09T17:33:10.332190+0000","last_change":"2024-02-09T17:33:06.284895+0000","last_active":"2024-02-09T17:33:10.332190+0000","last_peered":"2024-02-09T17:33:10.332190+0000","last_clean":"2024-02-09T17:33:10.332190+0000","last_became_active":"2024-02-09T17:33:05.308268+0000","last_became_peered":"2024-02-09T17:33:05.308268+0000","last_unstale":"2024-02-09T17:33:10.332190+0000","last_undegraded":"2024-02-09T17:33:10.332190+0000","last_fullsized":"2024-02-09T17:33:10.332190+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:33:06.284809+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:33:06.284809+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:34:24.831495+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"63'139","reported_seq":307,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-02-09T17:33:10.341416+0000","last_change":"2024-02-09T17:33:05.309889+0000","last_active":"2024-02-09T17:33:10.341416+0000","last_peered":"2024-02-09T17:33:10.341416+0000","last_clean":"2024-02-09T17:33:10.341416+0000","last_became_active":"2024-02-09T17:33:05.309527+0000","last_became_peered":"2024-02-09T17:33:05.309527+0000","last_unstale":"2024-02-09T17:33:10.341416+0000","last_undegraded":"2024-02-09T17:33:10.341416+0000","last_fullsized":"2024-02-09T17:33:10.341416+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:31:59.409353+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:31:59.409353+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:33:10.703955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17890,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":6,"num_write_kb":89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":48084,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":5205786648,"num_objects":79435,"num_object_clones":0,"num_object_copies":238305,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3314,"num_objects_misplaced":19787,"num_objects_unfound":0,"num_objects_dirty":79435,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":184479,"num_write_kb":8445185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14429,"num_bytes_recovered":943390720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16596480000,"data_stored":16596467784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":48,"acting":44,"num_store_stats":12},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":1010781,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":89,"num_read_kb":77,"num_write":127,"num_write_kb":2091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1447574,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2919,"ondisk_log_size":2919,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056770,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2820972,"kb_used_data":2524452,"kb_used_omap":1,"kb_used_meta":296318,"kb_avail":90924180,"statfs":{"total":95995035648,"available":93106360320,"internally_reserved":0,"allocated":2585038848,"data_stored":2584665476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":303430084},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:33:03 2024","interfaces":[{"interface":"back","average":{"1min":1.7789999999999999,"5min":1.232,"15min":0.93700000000000006},"min":{"1min":0.222,"5min":0.219,"15min":0.219},"max":{"1min":8.6240000000000006,"5min":14.913,"15min":14.913},"last":12.836},{"interface":"front","average":{"1min":1.4299999999999999,"5min":1.2509999999999999,"15min":1.0009999999999999},"min":{"1min":0.26500000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":7.7400000000000002,"5min":14.869,"15min":14.869},"last":6.5640000000000001}]},{"osd":1,"last update":"Fri Feb 9 17:33:03 2024","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.6220000000000001,"15min":1.302},"min":{"1min":0.27900000000000003,"5min":0.25900000000000001,"15min":0.245},"max":{"1min":7.0170000000000003,"5min":17.832999999999998,"15min":32.957999999999998},"last":0.28799999999999998},{"interface":"front","average":{"1min":1.764,"5min":1.821,"15min":1.361},"min":{"1min":0.38100000000000001,"5min":0.26800000000000002,"15min":0.26700000000000002},"max":{"1min":7.0419999999999998,"5min":17.789000000000001,"15min":32.880000000000003},"last":0.42899999999999999}]},{"osd":2,"last update":"Fri Feb 9 17:31:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.415,"15min":1.6859999999999999},"min":{"1min":0.378,"5min":0.247,"15min":0.247},"max":{"1min":4.4189999999999996,"5min":9.4719999999999995,"15min":9.4719999999999995},"last":12.728},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.5149999999999999,"15min":1.4399999999999999},"min":{"1min":0.378,"5min":0.252,"15min":0.252},"max":{"1min":4.8579999999999997,"5min":7.96,"15min":7.96},"last":0.40799999999999997}]},{"osd":3,"last update":"Fri Feb 9 17:33:03 2024","interfaces":[{"interface":"back","average":{"1min":1.6140000000000001,"5min":2.0510000000000002,"15min":1.417},"min":{"1min":0.29299999999999998,"5min":0.247,"15min":0.247},"max":{"1min":5.0860000000000003,"5min":21.315999999999999,"15min":21.315999999999999},"last":0.38200000000000001},{"interface":"front","average":{"1min":2.1309999999999998,"5min":1.873,"15min":1.3169999999999999},"min":{"1min":0.312,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":7.2439999999999998,"5min":18.087,"15min":18.087},"last":0.45500000000000002}]},{"osd":4,"last update":"Fri Feb 9 17:33:03 2024","interfaces":[{"interface":"back","average":{"1min":2.6160000000000001,"5min":1.355,"15min":0.95499999999999996},"min":{"1min":0.311,"5min":0.30499999999999999,"15min":0.30299999999999999},"max":{"1min":10.577999999999999,"5min":10.577999999999999,"15min":10.577999999999999},"last":4.7069999999999999},{"interface":"front","average":{"1min":2.0840000000000001,"5min":1.323,"15min":0.97699999999999998},"min":{"1min":0.36599999999999999,"5min":0.28799999999999998,"15min":0.27700000000000002},"max":{"1min":7.3129999999999997,"5min":7.3129999999999997,"15min":7.3129999999999997},"last":0.47599999999999998}]},{"osd":5,"last update":"Fri Feb 9 17:33:03 2024","interfaces":[{"interface":"back","average":{"1min":3.2869999999999999,"5min":1.6830000000000001,"15min":1.089},"min":{"1min":0.38800000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":13.946999999999999,"5min":13.946999999999999,"15min":13.946999999999999},"last":4.7729999999999997},{"interface":"front","average":{"1min":3.4550000000000001,"5min":1.5549999999999999,"15min":1.048},"min":{"1min":0.441,"5min":0.313,"15min":0.313},"max":{"1min":13.414,"5min":13.414,"15min":13.414},"last":12.771000000000001}]},{"osd":6,"last update":"Fri Feb 9 17:33:03 2024","interfaces":[{"interface":"back","average":{"1min":3.1429999999999998,"5min":1.484,"15min":1.046},"min":{"1min":0.34000000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":8.7070000000000007,"5min":8.7070000000000007,"15min":8.7070000000000007},"last":12.798999999999999},{"interface":"front","average":{"1min":2.6459999999999999,"5min":1.554,"15min":1.046},"min":{"1min":0.44900000000000001,"5min":0.38400000000000001,"15min":0.32300000000000001},"max":{"1min":10.782999999999999,"5min":10.782999999999999,"15min":10.782999999999999},"last":4.0019999999999998}]},{"osd":7,"last update":"Fri Feb 9 17:33:03 2024","interfaces":[{"interface":"back","average":{"1min":3.1520000000000001,"5min":1.669,"15min":1.117},"min":{"1min":0.61099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":14.462,"5min":14.462,"15min":14.462},"last":3.9620000000000002},{"interface":"front","average":{"1min":3.7690000000000001,"5min":1.696,"15min":1.131},"min":{"1min":0.439,"5min":0.42599999999999999,"15min":0.36499999999999999},"max":{"1min":14.398999999999999,"5min":14.398999999999999,"15min":14.398999999999999},"last":4.75}]},{"osd":8,"last update":"Fri Feb 9 17:25:48 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.71399999999999997,"15min":0.68000000000000005},"min":{"1min":0.40799999999999997,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.9350000000000001,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":13.199999999999999},{"interface":"front","average":{"1min":1.3029999999999999,"5min":0.80700000000000005,"15min":0.71999999999999997},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":3.7389999999999999,"5min":3.7389999999999999,"15min":3.7389999999999999},"last":0.5}]},{"osd":9,"last update":"Fri Feb 9 17:32:50 2024","interfaces":[{"interface":"back","average":{"1min":2.3559999999999999,"5min":1.3520000000000001,"15min":1.2849999999999999},"min":{"1min":0.34200000000000003,"5min":0.33700000000000002,"15min":0.316},"max":{"1min":7.859,"5min":7.859,"15min":7.859},"last":12.821},{"interface":"front","average":{"1min":2.258,"5min":1.1930000000000001,"15min":0.98099999999999998},"min":{"1min":0.47399999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":7.6989999999999998,"5min":7.6989999999999998,"15min":7.6989999999999998},"last":4.6390000000000002}]},{"osd":10,"last update":"Fri Feb 9 17:33:03 2024","interfaces":[{"interface":"back","average":{"1min":2.0430000000000001,"5min":1.3009999999999999,"15min":0.95799999999999996},"min":{"1min":0.505,"5min":0.376,"15min":0.33900000000000002},"max":{"1min":8.5679999999999996,"5min":8.5679999999999996,"15min":8.5679999999999996},"last":4.4630000000000001},{"interface":"front","average":{"1min":2.7160000000000002,"5min":1.3200000000000001,"15min":0.96199999999999997},"min":{"1min":0.40300000000000002,"5min":0.33900000000000002,"15min":0.33700000000000002},"max":{"1min":10.622,"5min":10.675000000000001,"15min":10.675000000000001},"last":12.856}]}]},{"osd":10,"up_from":84,"seq":360777253000,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":430700,"kb_used_data":319012,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93314452,"statfs":{"total":95995035648,"available":95553998848,"internally_reserved":0,"allocated":326668288,"data_stored":326294916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":114358720},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:31:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":1.387,"15min":1.762},"min":{"1min":0.23899999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":1.29,"5min":20.120999999999999,"15min":20.120999999999999},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.74099999999999999,"5min":1.4039999999999999,"15min":2.3940000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":3.173,"5min":14.471,"15min":14.471},"last":0.44500000000000001}]},{"osd":1,"last update":"Fri Feb 9 17:32:43 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":1.7749999999999999,"15min":1.1539999999999999},"min":{"1min":0.27300000000000002,"5min":0.23899999999999999,"15min":0.23499999999999999},"max":{"1min":5.008,"5min":37.716000000000001,"15min":37.716000000000001},"last":0.38400000000000001},{"interface":"front","average":{"1min":1.081,"5min":1.5960000000000001,"15min":1.1579999999999999},"min":{"1min":0.36099999999999999,"5min":0.245,"15min":0.245},"max":{"1min":5.4119999999999999,"5min":38.039999999999999,"15min":38.039999999999999},"last":0.308}]},{"osd":2,"last update":"Fri Feb 9 17:32:43 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.997,"15min":1.0600000000000001},"min":{"1min":0.33800000000000002,"5min":0.222,"15min":0.222},"max":{"1min":2.3839999999999999,"5min":9.2200000000000006,"15min":13.351000000000001},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.878,"5min":1.3240000000000001,"15min":1.0600000000000001},"min":{"1min":0.40100000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":4.6079999999999997,"5min":10.596,"15min":10.596},"last":0.46800000000000003}]},{"osd":3,"last update":"Fri Feb 9 17:32:43 2024","interfaces":[{"interface":"back","average":{"1min":1.5189999999999999,"5min":1.2450000000000001,"15min":1.155},"min":{"1min":0.29299999999999998,"5min":0.27700000000000002,"15min":0.26200000000000001},"max":{"1min":9.3829999999999991,"5min":9.3829999999999991,"15min":9.3829999999999991},"last":0.33000000000000002},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.4490000000000001,"15min":1.0880000000000001},"min":{"1min":0.42199999999999999,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":9.2590000000000003,"5min":9.2590000000000003,"15min":9.2590000000000003},"last":0.35299999999999998}]},{"osd":4,"last update":"Fri Feb 9 17:32:43 2024","interfaces":[{"interface":"back","average":{"1min":1.75,"5min":1.069,"15min":0.89100000000000001},"min":{"1min":0.34100000000000003,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":6.9960000000000004,"5min":6.9960000000000004,"15min":6.9960000000000004},"last":0.374},{"interface":"front","average":{"1min":1.345,"5min":0.996,"15min":0.88600000000000001},"min":{"1min":0.33300000000000002,"5min":0.307,"15min":0.307},"max":{"1min":4.6890000000000001,"5min":6.8369999999999997,"15min":6.8369999999999997},"last":0.42999999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:32:43 2024","interfaces":[{"interface":"back","average":{"1min":1.849,"5min":1.506,"15min":1.089},"min":{"1min":0.34399999999999997,"5min":0.33100000000000002,"15min":0.31900000000000001},"max":{"1min":9.2349999999999994,"5min":9.2349999999999994,"15min":9.2349999999999994},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.27,"5min":1.2689999999999999,"15min":0.996},"min":{"1min":0.443,"5min":0.33900000000000002,"15min":0.32200000000000001},"max":{"1min":4.7460000000000004,"5min":8.6159999999999997,"15min":8.6159999999999997},"last":0.53300000000000003}]},{"osd":6,"last update":"Fri Feb 9 17:32:43 2024","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.077,"15min":0.92300000000000004},"min":{"1min":0.39000000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":4.4340000000000002,"5min":6.8799999999999999,"15min":6.8799999999999999},"last":0.48699999999999999},{"interface":"front","average":{"1min":1.714,"5min":1.3360000000000001,"15min":1.0249999999999999},"min":{"1min":0.52800000000000002,"5min":0.38400000000000001,"15min":0.36399999999999999},"max":{"1min":9.4350000000000005,"5min":9.4350000000000005,"15min":9.4350000000000005},"last":0.501}]},{"osd":7,"last update":"Fri Feb 9 17:32:43 2024","interfaces":[{"interface":"back","average":{"1min":1.8220000000000001,"5min":1.198,"15min":0.95299999999999996},"min":{"1min":0.48599999999999999,"5min":0.312,"15min":0.312},"max":{"1min":6.7110000000000003,"5min":6.7110000000000003,"15min":6.7110000000000003},"last":0.626},{"interface":"front","average":{"1min":2.5169999999999999,"5min":1.327,"15min":1.0629999999999999},"min":{"1min":0.503,"5min":0.316,"15min":0.316},"max":{"1min":10.849,"5min":10.849,"15min":10.849},"last":5.5289999999999999}]},{"osd":8,"last update":"Fri Feb 9 17:26:28 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.73799999999999999,"15min":0.64000000000000001},"min":{"1min":0.36399999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":3.6509999999999998,"5min":3.6509999999999998,"15min":3.6509999999999998},"last":0.41099999999999998},{"interface":"front","average":{"1min":1.72,"5min":0.89500000000000002,"15min":0.68200000000000005},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":4.173,"5min":4.173,"15min":4.173},"last":0.44600000000000001}]},{"osd":9,"last update":"Fri Feb 9 17:32:43 2024","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.93500000000000005,"15min":0.84699999999999998},"min":{"1min":0.39700000000000002,"5min":0.314,"15min":0.314},"max":{"1min":3.5019999999999998,"5min":8.7240000000000002,"15min":8.7240000000000002},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.83499999999999996,"5min":1.0880000000000001,"15min":0.97899999999999998},"min":{"1min":0.44500000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":2.512,"5min":10.1,"15min":10.1},"last":0.61399999999999999}]},{"osd":11,"last update":"Fri Feb 9 17:33:10 2024","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.167,"15min":1.0229999999999999},"min":{"1min":0.46500000000000002,"5min":0.39200000000000002,"15min":0.36399999999999999},"max":{"1min":3.681,"5min":8.6430000000000007,"15min":8.6430000000000007},"last":0.64700000000000002},{"interface":"front","average":{"1min":1.7110000000000001,"5min":1.3,"15min":1.0509999999999999},"min":{"1min":0.47899999999999998,"5min":0.377,"15min":0.33400000000000002},"max":{"1min":7.3019999999999996,"5min":9.298,"15min":9.298},"last":0.57499999999999996}]}]},{"osd":9,"up_from":77,"seq":330712481934,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1112368,"kb_used_data":936808,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":92632784,"statfs":{"total":95995035648,"available":94855970816,"internally_reserved":0,"allocated":959291392,"data_stored":958913948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":179763648},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:28:03 2024","interfaces":[{"interface":"back","average":{"1min":1.9510000000000001,"5min":1.454,"15min":0.85799999999999998},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":8.4499999999999993,"5min":17.495999999999999,"15min":17.495999999999999},"last":2.7360000000000002},{"interface":"front","average":{"1min":1.905,"5min":1.399,"15min":0.83499999999999996},"min":{"1min":0.32200000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":7.5990000000000002,"5min":15.401999999999999,"15min":15.401999999999999},"last":1.5449999999999999}]},{"osd":1,"last update":"Fri Feb 9 17:32:17 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":1.4339999999999999,"15min":1.1180000000000001},"min":{"1min":0.23699999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":4.1799999999999997,"5min":16.527999999999999,"15min":16.527999999999999},"last":2.444},{"interface":"front","average":{"1min":0.65600000000000003,"5min":1.276,"15min":0.99299999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":2.9990000000000001,"5min":8.6769999999999996,"15min":17.771999999999998},"last":1.4350000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:32:17 2024","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.3029999999999999,"15min":1.02},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":6.0570000000000004,"5min":8.3550000000000004,"15min":13.878},"last":0.308},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.4239999999999999,"15min":1.0900000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":4.0970000000000004,"5min":9.3849999999999998,"15min":13.69},"last":2.4769999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:32:17 2024","interfaces":[{"interface":"back","average":{"1min":3.0859999999999999,"5min":1.5469999999999999,"15min":1.089},"min":{"1min":0.309,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":40.18,"5min":40.18,"15min":40.18},"last":2.4020000000000001},{"interface":"front","average":{"1min":3.085,"5min":1.9590000000000001,"15min":1.264},"min":{"1min":0.32300000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":40.158000000000001,"5min":40.158000000000001,"15min":40.158000000000001},"last":1.2949999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:32:17 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.393,"15min":0.99299999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":7.2969999999999997,"5min":22.577999999999999,"15min":22.577999999999999},"last":4.3289999999999997},{"interface":"front","average":{"1min":1.514,"5min":1.1739999999999999,"15min":0.90400000000000003},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":10.403,"5min":22.399999999999999,"15min":22.399999999999999},"last":1.2589999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:32:17 2024","interfaces":[{"interface":"back","average":{"1min":1.496,"5min":1.0880000000000001,"15min":0.90300000000000002},"min":{"1min":0.32100000000000001,"5min":0.31900000000000001,"15min":0.29299999999999998},"max":{"1min":6.1470000000000002,"5min":8.2460000000000004,"15min":8.2460000000000004},"last":7.4989999999999997},{"interface":"front","average":{"1min":1.6040000000000001,"5min":1.0720000000000001,"15min":0.871},"min":{"1min":0.32000000000000001,"5min":0.315,"15min":0.315},"max":{"1min":4.3520000000000003,"5min":4.3520000000000003,"15min":5.665},"last":3.0219999999999998}]},{"osd":6,"last update":"Fri Feb 9 17:32:17 2024","interfaces":[{"interface":"back","average":{"1min":2.169,"5min":1.2509999999999999,"15min":1.002},"min":{"1min":0.46000000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":7.3650000000000002,"5min":8.2959999999999994,"15min":8.2959999999999994},"last":0.36599999999999999},{"interface":"front","average":{"1min":2.2229999999999999,"5min":1.3520000000000001,"15min":0.996},"min":{"1min":0.48499999999999999,"5min":0.36199999999999999,"15min":0.35099999999999998},"max":{"1min":7.4000000000000004,"5min":8.3130000000000006,"15min":8.3130000000000006},"last":0.39500000000000002}]},{"osd":7,"last update":"Fri Feb 9 17:32:17 2024","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.0409999999999999,"15min":0.90100000000000002},"min":{"1min":0.38500000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":3.4300000000000002,"5min":8.0939999999999994,"15min":8.0939999999999994},"last":7.3710000000000004},{"interface":"front","average":{"1min":1.2110000000000001,"5min":1.218,"15min":0.95599999999999996},"min":{"1min":0.44800000000000001,"5min":0.36499999999999999,"15min":0.34899999999999998},"max":{"1min":4.4630000000000001,"5min":5.1829999999999998,"15min":6.056},"last":7.4050000000000002}]},{"osd":8,"last update":"Fri Feb 9 17:32:28 2024","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.2470000000000001,"15min":1.331},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":4.7240000000000002,"5min":16.209,"15min":16.209},"last":2.5470000000000002},{"interface":"front","average":{"1min":1.5369999999999999,"5min":1.353,"15min":1.399},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":10.241,"5min":16.263000000000002,"15min":16.263000000000002},"last":1.3169999999999999}]},{"osd":10,"last update":"Fri Feb 9 17:32:47 2024","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.1339999999999999,"15min":0.95099999999999996},"min":{"1min":0.38,"5min":0.36799999999999999,"15min":0.33300000000000002},"max":{"1min":4.7450000000000001,"5min":9.5090000000000003,"15min":9.5090000000000003},"last":2.6480000000000001},{"interface":"front","average":{"1min":0.83299999999999996,"5min":1,"15min":0.92700000000000005},"min":{"1min":0.38500000000000001,"5min":0.38200000000000001,"15min":0.36099999999999999},"max":{"1min":3.4049999999999998,"5min":8.0670000000000002,"15min":8.0670000000000002},"last":1.341}]},{"osd":11,"last update":"Fri Feb 9 17:32:51 2024","interfaces":[{"interface":"back","average":{"1min":2.0219999999999998,"5min":1.431,"15min":1.3120000000000001},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.34699999999999998},"max":{"1min":11.541,"5min":11.541,"15min":11.541},"last":4.431},{"interface":"front","average":{"1min":1.6879999999999999,"5min":1.1479999999999999,"15min":1.0629999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":10.32,"5min":10.32,"15min":10.32},"last":4.4699999999999998}]}]},{"osd":8,"up_from":171,"seq":734439407676,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":768624,"kb_used_data":646884,"kb_used_omap":212,"kb_used_meta":121515,"kb_avail":92976528,"statfs":{"total":95995035648,"available":95207964672,"internally_reserved":0,"allocated":662409216,"data_stored":662035852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217579,"internal_metadata":124431893},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:32:23 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":1.1559999999999999,"15min":1.5920000000000001},"min":{"1min":0.26300000000000001,"5min":0.245,"15min":0.245},"max":{"1min":2.7970000000000002,"5min":6.8840000000000003,"15min":6.8840000000000003},"last":1.9850000000000001},{"interface":"front","average":{"1min":0.95599999999999996,"5min":1.262,"15min":1.514},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":3.278,"5min":7.7969999999999997,"15min":7.7969999999999997},"last":2.347}]},{"osd":1,"last update":"Fri Feb 9 17:32:23 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.0900000000000001,"15min":1.2949999999999999},"min":{"1min":0.28799999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":7.8719999999999999,"5min":7.8719999999999999,"15min":7.8719999999999999},"last":2.331},{"interface":"front","average":{"1min":0.90200000000000002,"5min":1.5209999999999999,"15min":2.1709999999999998},"min":{"1min":0.316,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":4.8609999999999998,"5min":11.101000000000001,"15min":11.101000000000001},"last":1.8620000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:32:23 2024","interfaces":[{"interface":"back","average":{"1min":1.5449999999999999,"5min":1.2989999999999999,"15min":1.4550000000000001},"min":{"1min":0.28499999999999998,"5min":0.255,"15min":0.255},"max":{"1min":9.1820000000000004,"5min":9.1820000000000004,"15min":9.1820000000000004},"last":10.423},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.407,"15min":1.7789999999999999},"min":{"1min":0.34399999999999997,"5min":0.313,"15min":0.313},"max":{"1min":7.7709999999999999,"5min":7.7709999999999999,"15min":7.7709999999999999},"last":10.504}]},{"osd":3,"last update":"Fri Feb 9 17:32:23 2024","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.833,"15min":2.2989999999999999},"min":{"1min":0.36399999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.3129999999999997,"5min":12.505000000000001,"15min":12.505000000000001},"last":2.044},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.9610000000000001,"15min":2.6899999999999999},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":5.6689999999999996,"5min":10.92,"15min":10.92},"last":1.954}]},{"osd":4,"last update":"Fri Feb 9 17:32:23 2024","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":1.194,"15min":0.86599999999999999},"min":{"1min":0.307,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":3.1739999999999999,"5min":37.343000000000004,"15min":37.343000000000004},"last":2.2530000000000001},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.4299999999999999,"15min":1.1899999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":6.2670000000000003,"5min":37.875,"15min":37.875},"last":1.9279999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:32:23 2024","interfaces":[{"interface":"back","average":{"1min":1.9359999999999999,"5min":1.7729999999999999,"15min":1.4990000000000001},"min":{"1min":0.376,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":9.2959999999999994,"5min":37.356999999999999,"15min":37.356999999999999},"last":10.619999999999999},{"interface":"front","average":{"1min":1.361,"5min":1.337,"15min":0.98099999999999998},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":4.5549999999999997,"5min":37.25,"15min":37.25},"last":2.29}]},{"osd":7,"last update":"Fri Feb 9 17:32:23 2024","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.506,"15min":1.2210000000000001},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":7.9450000000000003,"5min":37.585000000000001,"15min":37.585000000000001},"last":2.0819999999999999},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.619,"15min":1.4670000000000001},"min":{"1min":0.45300000000000001,"5min":0.312,"15min":0.312},"max":{"1min":8.0180000000000007,"5min":37.918999999999997,"15min":37.918999999999997},"last":2.1139999999999999}]},{"osd":9,"last update":"Fri Feb 9 17:32:23 2024","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.0389999999999999,"15min":1.1899999999999999},"min":{"1min":0.34100000000000003,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":3.3450000000000002,"5min":6.9269999999999996,"15min":6.9269999999999996},"last":2.3650000000000002},{"interface":"front","average":{"1min":1.052,"5min":1.0629999999999999,"15min":1.2270000000000001},"min":{"1min":0.35399999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":5.1580000000000004,"5min":7.7409999999999997,"15min":7.7409999999999997},"last":1.8999999999999999}]},{"osd":10,"last update":"Fri Feb 9 17:32:23 2024","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":1.1240000000000001,"15min":1.266},"min":{"1min":0.376,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":9.2609999999999992,"5min":9.2609999999999992,"15min":9.2609999999999992},"last":10.65},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.92500000000000004,"15min":1.0489999999999999},"min":{"1min":0.39300000000000002,"5min":0.37,"15min":0.37},"max":{"1min":3.8969999999999998,"5min":5.1070000000000002,"15min":5.1070000000000002},"last":2.3130000000000002}]},{"osd":11,"last update":"Fri Feb 9 17:32:23 2024","interfaces":[{"interface":"back","average":{"1min":1.6830000000000001,"5min":1.2490000000000001,"15min":1.335},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":9.234,"5min":9.234,"15min":9.234},"last":10.712},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.2170000000000001,"15min":1.5449999999999999},"min":{"1min":0.45500000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":5.4829999999999997,"5min":7.6609999999999996,"15min":7.6609999999999996},"last":11.622}]}]},{"osd":7,"up_from":57,"seq":244813136022,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1495088,"kb_used_data":1301480,"kb_used_omap":1,"kb_used_meta":193598,"kb_avail":92250064,"statfs":{"total":95995035648,"available":94464065536,"internally_reserved":0,"allocated":1332715520,"data_stored":1332338084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":198244810},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:32:51 2024","interfaces":[{"interface":"back","average":{"1min":1.8480000000000001,"5min":1.3540000000000001,"15min":1.0049999999999999},"min":{"1min":0.27300000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.8890000000000002,"5min":10.776,"15min":10.776},"last":0.35499999999999998},{"interface":"front","average":{"1min":2.0430000000000001,"5min":1.6899999999999999,"15min":1.1379999999999999},"min":{"1min":0.28899999999999998,"5min":0.252,"15min":0.252},"max":{"1min":7.0279999999999996,"5min":18.896000000000001,"15min":18.896000000000001},"last":0.42399999999999999}]},{"osd":1,"last update":"Fri Feb 9 17:27:46 2024","interfaces":[{"interface":"back","average":{"1min":2.262,"5min":2.262,"15min":2.262},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":4.8470000000000004,"5min":4.8470000000000004,"15min":4.8470000000000004},"last":5.0609999999999999},{"interface":"front","average":{"1min":1.8009999999999999,"5min":1.8009999999999999,"15min":1.8009999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.9589999999999996,"5min":6.9589999999999996,"15min":6.9589999999999996},"last":0.26500000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:32:26 2024","interfaces":[{"interface":"back","average":{"1min":1.964,"5min":2.0230000000000001,"15min":1.151},"min":{"1min":0.31900000000000001,"5min":0.30399999999999999,"15min":0.26000000000000001},"max":{"1min":11.324,"5min":11.324,"15min":11.324},"last":0.312},{"interface":"front","average":{"1min":2.2410000000000001,"5min":1.746,"15min":1.1890000000000001},"min":{"1min":0.309,"5min":0.309,"15min":0.28299999999999997},"max":{"1min":11.455,"5min":11.455,"15min":11.455},"last":0.36699999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:32:26 2024","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.8819999999999999,"15min":1.1160000000000001},"min":{"1min":0.28899999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":7.6529999999999996,"5min":27.719000000000001,"15min":27.719000000000001},"last":5.109},{"interface":"front","average":{"1min":2.5030000000000001,"5min":1.881,"15min":1.1799999999999999},"min":{"1min":0.34499999999999997,"5min":0.311,"15min":0.27600000000000002},"max":{"1min":6.9240000000000004,"5min":6.9240000000000004,"15min":8.8629999999999995},"last":0.28699999999999998}]},{"osd":4,"last update":"Fri Feb 9 17:32:26 2024","interfaces":[{"interface":"back","average":{"1min":1.6890000000000001,"5min":1.196,"15min":0.76800000000000002},"min":{"1min":0.29399999999999998,"5min":0.27200000000000002,"15min":0.248},"max":{"1min":8.6150000000000002,"5min":8.6150000000000002,"15min":8.6150000000000002},"last":5.8639999999999999},{"interface":"front","average":{"1min":1.7629999999999999,"5min":1.0960000000000001,"15min":0.748},"min":{"1min":0.35299999999999998,"5min":0.23300000000000001,"15min":0.22800000000000001},"max":{"1min":8.6419999999999995,"5min":9.3650000000000002,"15min":9.3650000000000002},"last":5.8899999999999997}]},{"osd":5,"last update":"Fri Feb 9 17:32:26 2024","interfaces":[{"interface":"back","average":{"1min":1.8129999999999999,"5min":1.2490000000000001,"15min":0.79200000000000004},"min":{"1min":0.33200000000000002,"5min":0.28599999999999998,"15min":0.21299999999999999},"max":{"1min":11.548999999999999,"5min":11.548999999999999,"15min":11.548999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":2.2749999999999999,"5min":1.22,"15min":0.82999999999999996},"min":{"1min":0.317,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":11.598000000000001,"5min":11.598000000000001,"15min":11.598000000000001},"last":0.42899999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:32:26 2024","interfaces":[{"interface":"back","average":{"1min":2.0800000000000001,"5min":1.391,"15min":0.88700000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":11.369,"5min":12.496,"15min":12.496},"last":2.5110000000000001},{"interface":"front","average":{"1min":2.1400000000000001,"5min":1.252,"15min":0.83399999999999996},"min":{"1min":0.377,"5min":0.32500000000000001,"15min":0.28699999999999998},"max":{"1min":6.8399999999999999,"5min":6.8399999999999999,"15min":6.8399999999999999},"last":2.5550000000000002}]},{"osd":8,"last update":"Fri Feb 9 17:32:26 2024","interfaces":[{"interface":"back","average":{"1min":1.45,"5min":1.335,"15min":1.417},"min":{"1min":0.46300000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":7.7839999999999998,"5min":7.9420000000000002,"15min":7.9420000000000002},"last":5.7670000000000003},{"interface":"front","average":{"1min":1.7749999999999999,"5min":1.339,"15min":1.079},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":11.481,"5min":11.481,"15min":11.481},"last":0.52900000000000003}]},{"osd":9,"last update":"Fri Feb 9 17:33:08 2024","interfaces":[{"interface":"back","average":{"1min":3.4910000000000001,"5min":2.0539999999999998,"15min":1.2549999999999999},"min":{"1min":0.46400000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":8.9900000000000002,"5min":41.286999999999999,"15min":41.286999999999999},"last":5.1699999999999999},{"interface":"front","average":{"1min":3.3180000000000001,"5min":2.23,"15min":1.2689999999999999},"min":{"1min":0.50600000000000001,"5min":0.42799999999999999,"15min":0.35399999999999998},"max":{"1min":11.417999999999999,"5min":41.234999999999999,"15min":41.234999999999999},"last":0.50600000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:32:36 2024","interfaces":[{"interface":"back","average":{"1min":2.1909999999999998,"5min":1.357,"15min":1.075},"min":{"1min":0.43099999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":6.9950000000000001,"5min":6.9950000000000001,"15min":8.4659999999999993},"last":0.45500000000000002},{"interface":"front","average":{"1min":1.778,"5min":1.3100000000000001,"15min":1.0409999999999999},"min":{"1min":0.45600000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":7.8099999999999996,"5min":7.8099999999999996,"15min":7.8099999999999996},"last":5.806}]},{"osd":11,"last update":"Fri Feb 9 17:33:02 2024","interfaces":[{"interface":"back","average":{"1min":3.411,"5min":1.736,"15min":1.228},"min":{"1min":0.52700000000000002,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":12.885999999999999,"5min":12.885999999999999,"15min":12.885999999999999},"last":5.835},{"interface":"front","average":{"1min":2.3599999999999999,"5min":1.4950000000000001,"15min":1.1120000000000001},"min":{"1min":0.41299999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":5.9169999999999998,"5min":6.9580000000000002,"15min":8.1799999999999997},"last":0.47699999999999998}]}]},{"osd":6,"up_from":49,"seq":210453397660,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1711980,"kb_used_data":1572452,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":92033172,"statfs":{"total":95995035648,"available":94241968128,"internally_reserved":0,"allocated":1610190848,"data_stored":1609817484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":142801350},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:32:56 2024","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.373,"15min":1.077},"min":{"1min":0.33300000000000002,"5min":0.26000000000000001,"15min":0.24299999999999999},"max":{"1min":4.5170000000000003,"5min":9.5570000000000004,"15min":24.544},"last":0.42599999999999999},{"interface":"front","average":{"1min":1.222,"5min":1.3029999999999999,"15min":1.02},"min":{"1min":0.26500000000000001,"5min":0.217,"15min":0.217},"max":{"1min":4.5330000000000004,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.46100000000000002}]},{"osd":1,"last update":"Fri Feb 9 17:32:56 2024","interfaces":[{"interface":"back","average":{"1min":2.3290000000000002,"5min":1.5860000000000001,"15min":1.153},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27000000000000002},"max":{"1min":10.35,"5min":10.35,"15min":23.954000000000001},"last":0.624},{"interface":"front","average":{"1min":2.379,"5min":1.643,"15min":1.1879999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.318},"max":{"1min":10.385999999999999,"5min":10.385999999999999,"15min":23.984999999999999},"last":0.64500000000000002}]},{"osd":2,"last update":"Fri Feb 9 17:32:56 2024","interfaces":[{"interface":"back","average":{"1min":2.0779999999999998,"5min":1.6279999999999999,"15min":1.224},"min":{"1min":0.28999999999999998,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":8.0239999999999991,"5min":10.682,"15min":24.329999999999998},"last":0.441},{"interface":"front","average":{"1min":1.373,"5min":1.6319999999999999,"15min":1.181},"min":{"1min":0.317,"5min":0.317,"15min":0.28000000000000003},"max":{"1min":4.5,"5min":10.66,"15min":12.906000000000001},"last":0.38900000000000001}]},{"osd":3,"last update":"Fri Feb 9 17:32:56 2024","interfaces":[{"interface":"back","average":{"1min":2.5070000000000001,"5min":1.9359999999999999,"15min":1.3109999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.248},"max":{"1min":13.468999999999999,"5min":32.545000000000002,"15min":32.545000000000002},"last":0.47999999999999998},{"interface":"front","average":{"1min":2.5659999999999998,"5min":1.8160000000000001,"15min":1.252},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.26600000000000001},"max":{"1min":13.56,"5min":32.670999999999999,"15min":32.670999999999999},"last":0.52600000000000002}]},{"osd":4,"last update":"Fri Feb 9 17:32:56 2024","interfaces":[{"interface":"back","average":{"1min":2.6040000000000001,"5min":1.351,"15min":0.82199999999999995},"min":{"1min":0.27200000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":8.3559999999999999,"5min":8.7110000000000003,"15min":8.7110000000000003},"last":6.7439999999999998},{"interface":"front","average":{"1min":1.496,"5min":1.2270000000000001,"15min":0.84499999999999997},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.26000000000000001},"max":{"1min":8.2149999999999999,"5min":8.8209999999999997,"15min":8.8209999999999997},"last":4.9199999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:32:56 2024","interfaces":[{"interface":"back","average":{"1min":2.613,"5min":1.3720000000000001,"15min":0.86799999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.29199999999999998},"max":{"1min":10.968999999999999,"5min":10.968999999999999,"15min":10.968999999999999},"last":2.153},{"interface":"front","average":{"1min":2.98,"5min":1.4399999999999999,"15min":0.88900000000000001},"min":{"1min":0.46100000000000002,"5min":0.29299999999999998,"15min":0.27400000000000002},"max":{"1min":8.3979999999999997,"5min":8.3979999999999997,"15min":8.3979999999999997},"last":6.6520000000000001}]},{"osd":7,"last update":"Fri Feb 9 17:32:34 2024","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":1.1459999999999999,"15min":0.82199999999999995},"min":{"1min":0.316,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":6.8280000000000003,"5min":6.8280000000000003,"15min":6.8280000000000003},"last":10.859},{"interface":"front","average":{"1min":1.038,"5min":1.365,"15min":0.89600000000000002},"min":{"1min":0.36599999999999999,"5min":0.309,"15min":0.309},"max":{"1min":3.746,"5min":11.512,"15min":11.512},"last":15.458}]},{"osd":8,"last update":"Fri Feb 9 17:26:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":0.876,"15min":0.79600000000000004},"min":{"1min":0.39900000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":4.0339999999999998,"5min":4.4880000000000004,"15min":4.4880000000000004},"last":4.3840000000000003},{"interface":"front","average":{"1min":1.0469999999999999,"5min":0.88600000000000001,"15min":0.78300000000000003},"min":{"1min":0.375,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":3.6539999999999999,"5min":4.8570000000000002,"15min":4.8570000000000002},"last":4.4210000000000003}]},{"osd":9,"last update":"Fri Feb 9 17:33:09 2024","interfaces":[{"interface":"back","average":{"1min":1.4610000000000001,"5min":1.3169999999999999,"15min":1.0169999999999999},"min":{"1min":0.42199999999999999,"5min":0.39800000000000002,"15min":0.35999999999999999},"max":{"1min":4.8949999999999996,"5min":10.803000000000001,"15min":10.803000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":1.766,"5min":1.335,"15min":1.0920000000000001},"min":{"1min":0.53000000000000003,"5min":0.42099999999999999,"15min":0.40300000000000002},"max":{"1min":4.4480000000000004,"5min":6.3109999999999999,"15min":6.3109999999999999},"last":0.73399999999999999}]},{"osd":10,"last update":"Fri Feb 9 17:32:34 2024","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.319,"15min":1.077},"min":{"1min":0.40699999999999997,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":6.194,"5min":6.875,"15min":9.5399999999999991},"last":0.55800000000000005},{"interface":"front","average":{"1min":1.8360000000000001,"5min":1.306,"15min":1.0840000000000001},"min":{"1min":0.51000000000000001,"5min":0.42499999999999999,"15min":0.39200000000000002},"max":{"1min":5.6139999999999999,"5min":5.6139999999999999,"15min":9.5210000000000008},"last":0.59899999999999998}]},{"osd":11,"last update":"Fri Feb 9 17:32:56 2024","interfaces":[{"interface":"back","average":{"1min":2.5630000000000002,"5min":1.4630000000000001,"15min":1.581},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":10.273999999999999,"5min":10.273999999999999,"15min":10.273999999999999},"last":0.504},{"interface":"front","average":{"1min":2.7749999999999999,"5min":1.593,"15min":1.748},"min":{"1min":0.499,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":8.2569999999999997,"5min":8.2569999999999997,"15min":8.2569999999999997},"last":0.70499999999999996}]}]},{"osd":1,"up_from":13,"seq":55834575030,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1446700,"kb_used_data":1262820,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":92298452,"statfs":{"total":95995035648,"available":94513614848,"internally_reserved":0,"allocated":1293127680,"data_stored":1292754316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:33:01 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":1.2,"15min":1.014},"min":{"1min":0.223,"5min":0.214,"15min":0.19500000000000001},"max":{"1min":1.615,"5min":9.8140000000000001,"15min":12.492000000000001},"last":5.8579999999999997},{"interface":"front","average":{"1min":0.81499999999999995,"5min":1.54,"15min":1.0660000000000001},"min":{"1min":0.245,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":3.5910000000000002,"5min":9.8439999999999994,"15min":9.8439999999999994},"last":5.7859999999999996}]},{"osd":2,"last update":"Fri Feb 9 17:32:28 2024","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":1.623,"15min":1.1659999999999999},"min":{"1min":0.23899999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.4740000000000002,"5min":12.01,"15min":12.01},"last":0.23400000000000001},{"interface":"front","average":{"1min":1.2989999999999999,"5min":2.085,"15min":1.325},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":5.4480000000000004,"5min":17.459,"15min":17.459},"last":1.333}]},{"osd":3,"last update":"Fri Feb 9 17:31:48 2024","interfaces":[{"interface":"back","average":{"1min":1.571,"5min":2.0590000000000002,"15min":1.3899999999999999},"min":{"1min":0.32300000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":5.2359999999999998,"5min":16.088000000000001,"15min":16.088000000000001},"last":4.0679999999999996},{"interface":"front","average":{"1min":0.98999999999999999,"5min":1.927,"15min":1.1479999999999999},"min":{"1min":0.245,"5min":0.222,"15min":0.222},"max":{"1min":3.5630000000000002,"5min":12.378,"15min":12.378},"last":4.1239999999999997}]},{"osd":4,"last update":"Fri Feb 9 17:33:01 2024","interfaces":[{"interface":"back","average":{"1min":2.1970000000000001,"5min":2.1970000000000001,"15min":2.1970000000000001},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":11.766999999999999,"5min":11.766999999999999,"15min":11.766999999999999},"last":0.28000000000000003},{"interface":"front","average":{"1min":1.9370000000000001,"5min":1.9370000000000001,"15min":1.9370000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":7.2560000000000002,"5min":7.2560000000000002,"15min":7.2560000000000002},"last":5.7560000000000002}]},{"osd":5,"last update":"Fri Feb 9 17:32:41 2024","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.6359999999999999,"15min":1.181},"min":{"1min":0.28599999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":5.1950000000000003,"5min":7.077,"15min":7.7679999999999998},"last":6.4729999999999999},{"interface":"front","average":{"1min":1.272,"5min":1.3759999999999999,"15min":0.98999999999999999},"min":{"1min":0.35299999999999998,"5min":0.317,"15min":0.26000000000000001},"max":{"1min":5.2169999999999996,"5min":7.9009999999999998,"15min":7.9009999999999998},"last":5.7140000000000004}]},{"osd":6,"last update":"Fri Feb 9 17:33:01 2024","interfaces":[{"interface":"back","average":{"1min":1.804,"5min":1.7250000000000001,"15min":1.3069999999999999},"min":{"1min":0.34699999999999998,"5min":0.317,"15min":0.317},"max":{"1min":8.7210000000000001,"5min":10.275,"15min":10.275},"last":0.29899999999999999},{"interface":"front","average":{"1min":1.748,"5min":1.5640000000000001,"15min":1.246},"min":{"1min":0.36899999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":6.4299999999999997,"5min":7.0510000000000002,"15min":8.0229999999999997},"last":1.369}]},{"osd":7,"last update":"Fri Feb 9 17:32:28 2024","interfaces":[{"interface":"back","average":{"1min":1.962,"5min":2.0920000000000001,"15min":1.381},"min":{"1min":0.32400000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":9.0700000000000003,"5min":15.148999999999999,"15min":15.148999999999999},"last":6.109},{"interface":"front","average":{"1min":1.506,"5min":1.929,"15min":1.351},"min":{"1min":0.495,"5min":0.32600000000000001,"15min":0.28299999999999997},"max":{"1min":4.2119999999999997,"5min":13.834,"15min":13.834},"last":5.8730000000000002}]},{"osd":8,"last update":"Fri Feb 9 17:32:23 2024","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.2569999999999999,"15min":1.4610000000000001},"min":{"1min":0.45300000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":4.851,"5min":4.851,"15min":4.851},"last":5.8029999999999999},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.7090000000000001,"15min":2.1970000000000001},"min":{"1min":0.48099999999999998,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":5.0659999999999998,"5min":6.7530000000000001,"15min":6.7530000000000001},"last":5.8419999999999996}]},{"osd":9,"last update":"Fri Feb 9 17:33:08 2024","interfaces":[{"interface":"back","average":{"1min":2.024,"5min":1.6759999999999999,"15min":1.359},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":9.5839999999999996,"5min":9.5839999999999996,"15min":9.5839999999999996},"last":0.435},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.4319999999999999,"15min":1.327},"min":{"1min":0.39800000000000002,"5min":0.38700000000000001,"15min":0.30099999999999999},"max":{"1min":5.5709999999999997,"5min":12.117000000000001,"15min":12.117000000000001},"last":1.3939999999999999}]},{"osd":10,"last update":"Fri Feb 9 17:32:41 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":1.355,"15min":1.1080000000000001},"min":{"1min":0.47999999999999998,"5min":0.39600000000000002,"15min":0.35999999999999999},"max":{"1min":1.7110000000000001,"5min":6.5650000000000004,"15min":7.149},"last":5.8200000000000003},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.8320000000000001,"15min":1.3759999999999999},"min":{"1min":0.49399999999999999,"5min":0.40899999999999997,"15min":0.33400000000000002},"max":{"1min":5.5430000000000001,"5min":12.087999999999999,"15min":12.087999999999999},"last":1.5369999999999999}]},{"osd":11,"last update":"Fri Feb 9 17:32:51 2024","interfaces":[{"interface":"back","average":{"1min":1.5209999999999999,"5min":1.865,"15min":2.407},"min":{"1min":0.48299999999999998,"5min":0.437,"15min":0.32000000000000001},"max":{"1min":5.609,"5min":12.169,"15min":12.169},"last":1.6180000000000001},{"interface":"front","average":{"1min":1.5509999999999999,"5min":1.9690000000000001,"15min":2.2309999999999999},"min":{"1min":0.54100000000000004,"5min":0.44600000000000001,"15min":0.36599999999999999},"max":{"1min":5.633,"5min":12.144,"15min":12.144},"last":1.6439999999999999}]}]},{"osd":0,"up_from":0,"seq":34359738554,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411504,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":419828,"kb_used_data":312300,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93325324,"statfs":{"total":95995035648,"available":95565131776,"internally_reserved":0,"allocated":319795200,"data_stored":319418284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":1.4510000000000001,"15min":1.006},"min":{"1min":0.29999999999999999,"5min":0.23699999999999999,"15min":0.221},"max":{"1min":4.9089999999999998,"5min":17.251000000000001,"15min":17.251000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.74399999999999999,"5min":1.611,"15min":1.113},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.22900000000000001},"max":{"1min":4.7720000000000002,"5min":23.097999999999999,"15min":23.097999999999999},"last":0.875}]},{"osd":1,"last update":"Fri Feb 9 17:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":1.6259999999999999,"15min":1.129},"min":{"1min":0.27900000000000003,"5min":0.249,"15min":0.249},"max":{"1min":2.8460000000000001,"5min":27.289000000000001,"15min":27.289000000000001},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.85599999999999998,"5min":1.8540000000000001,"15min":1.212},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":4.4379999999999997,"5min":37.585999999999999,"15min":37.585999999999999},"last":0.26200000000000001}]},{"osd":3,"last update":"Fri Feb 9 17:32:52 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":1.915,"15min":1.2649999999999999},"min":{"1min":0.24199999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.887,"5min":37.634999999999998,"15min":37.634999999999998},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.89500000000000002,"5min":1.625,"15min":1.149},"min":{"1min":0.251,"5min":0.251,"15min":0.23100000000000001},"max":{"1min":3.8119999999999998,"5min":38.924999999999997,"15min":38.924999999999997},"last":0.29999999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:32:59 2024","interfaces":[{"interface":"back","average":{"1min":2.4620000000000002,"5min":1.77,"15min":1.208},"min":{"1min":0.41699999999999998,"5min":0.28399999999999997,"15min":0.254},"max":{"1min":8.5280000000000005,"5min":8.5280000000000005,"15min":8.8089999999999993},"last":3.2799999999999998},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.3049999999999999,"15min":1.0369999999999999},"min":{"1min":0.372,"5min":0.33700000000000002,"15min":0.30399999999999999},"max":{"1min":9.6509999999999998,"5min":9.6509999999999998,"15min":13.305999999999999},"last":4.024}]},{"osd":5,"last update":"Fri Feb 9 17:32:33 2024","interfaces":[{"interface":"back","average":{"1min":2.004,"5min":1.772,"15min":1.1919999999999999},"min":{"1min":0.44800000000000001,"5min":0.32300000000000001,"15min":0.28999999999999998},"max":{"1min":5.3419999999999996,"5min":6.7140000000000004,"15min":8.6859999999999999},"last":0.93899999999999995},{"interface":"front","average":{"1min":1.51,"5min":1.4019999999999999,"15min":1.008},"min":{"1min":0.32100000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":4.7519999999999998,"5min":13.278,"15min":13.278},"last":2.3719999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:32:59 2024","interfaces":[{"interface":"back","average":{"1min":1.931,"5min":1.718,"15min":1.246},"min":{"1min":0.51400000000000001,"5min":0.36699999999999999,"15min":0.32100000000000001},"max":{"1min":5.5869999999999997,"5min":6.7839999999999998,"15min":8.8399999999999999},"last":3.2549999999999999},{"interface":"front","average":{"1min":2.3900000000000001,"5min":1.897,"15min":1.3009999999999999},"min":{"1min":0.59499999999999997,"5min":0.39900000000000002,"15min":0.34999999999999998},"max":{"1min":4.9039999999999999,"5min":9.1080000000000005,"15min":9.1080000000000005},"last":1.548}]},{"osd":7,"last update":"Fri Feb 9 17:32:27 2024","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.607,"15min":1.1499999999999999},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":4.2599999999999998,"5min":14.541,"15min":14.541},"last":1.3},{"interface":"front","average":{"1min":1.653,"5min":1.9870000000000001,"15min":1.3260000000000001},"min":{"1min":0.5,"5min":0.41299999999999998,"15min":0.40500000000000003},"max":{"1min":4.4669999999999996,"5min":8.4169999999999998,"15min":8.7870000000000008},"last":1.2889999999999999}]},{"osd":8,"last update":"Fri Feb 9 17:26:31 2024","interfaces":[{"interface":"back","average":{"1min":1.7529999999999999,"5min":1,"15min":0.84599999999999997},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":8.8989999999999991,"5min":8.8989999999999991,"15min":8.8989999999999991},"last":2.9260000000000002},{"interface":"front","average":{"1min":2.4969999999999999,"5min":1.1080000000000001,"15min":0.90600000000000003},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":9.4190000000000005,"5min":9.4190000000000005,"15min":9.4190000000000005},"last":0.53900000000000003}]},{"osd":9,"last update":"Fri Feb 9 17:33:12 2024","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.4850000000000001,"15min":1.2290000000000001},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.38600000000000001},"max":{"1min":4.2930000000000001,"5min":6.8220000000000001,"15min":8.9190000000000005},"last":0.51700000000000002},{"interface":"front","average":{"1min":1.468,"5min":1.258,"15min":1.1399999999999999},"min":{"1min":0.49199999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":4.5830000000000002,"5min":7.7480000000000002,"15min":13.218},"last":0.68200000000000005}]},{"osd":10,"last update":"Fri Feb 9 17:32:42 2024","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.2210000000000001,"15min":1.105},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":6.1299999999999999,"5min":7.7709999999999999,"15min":13.244999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.397,"15min":1.1539999999999999},"min":{"1min":0.47399999999999998,"5min":0.42499999999999999,"15min":0.42099999999999999},"max":{"1min":6.1539999999999999,"5min":14.499000000000001,"15min":14.499000000000001},"last":0.59499999999999997}]},{"osd":11,"last update":"Fri Feb 9 17:32:52 2024","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.341,"15min":1.9670000000000001},"min":{"1min":0.44800000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":4.3529999999999998,"5min":7.7249999999999996,"15min":13.186},"last":3.2189999999999999},{"interface":"front","average":{"1min":1.766,"5min":1.601,"15min":1.929},"min":{"1min":0.48399999999999999,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":4.3879999999999999,"5min":6.8860000000000001,"15min":7.7919999999999998},"last":0.55000000000000004}]}]},{"osd":3,"up_from":27,"seq":115964117163,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1864120,"kb_used_data":1616364,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":91881032,"statfs":{"total":95995035648,"available":94086176768,"internally_reserved":0,"allocated":1655156736,"data_stored":1654779820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":253688263},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:31:59 2024","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":2.1160000000000001,"15min":3.298},"min":{"1min":0.247,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":3.8879999999999999,"5min":19.475000000000001,"15min":19.475000000000001},"last":0.255},{"interface":"front","average":{"1min":1.129,"5min":2.2130000000000001,"15min":3.2959999999999998},"min":{"1min":0.27900000000000003,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":6.7859999999999996,"5min":19.513000000000002,"15min":19.513000000000002},"last":0.30199999999999999}]},{"osd":1,"last update":"Fri Feb 9 17:32:41 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":1.994,"15min":1.379},"min":{"1min":0.375,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":3.581,"5min":19.638999999999999,"15min":19.638999999999999},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.65800000000000003,"5min":1.637,"15min":1.1020000000000001},"min":{"1min":0.27100000000000002,"5min":0.22500000000000001,"15min":0.20399999999999999},"max":{"1min":2.1219999999999999,"5min":14.661,"15min":14.661},"last":0.309}]},{"osd":2,"last update":"Fri Feb 9 17:32:41 2024","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.6220000000000001,"15min":1.1950000000000001},"min":{"1min":0.315,"5min":0.247,"15min":0.224},"max":{"1min":5.968,"5min":12.651999999999999,"15min":14.914999999999999},"last":1.298},{"interface":"front","average":{"1min":1.7170000000000001,"5min":1.6719999999999999,"15min":1.1339999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.218},"max":{"1min":7.9989999999999997,"5min":9.3089999999999993,"15min":9.3089999999999993},"last":0.63900000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:33:01 2024","interfaces":[{"interface":"back","average":{"1min":1.54,"5min":1.635,"15min":1.3240000000000001},"min":{"1min":0.29899999999999999,"5min":0.26900000000000002,"15min":0.254},"max":{"1min":11.178000000000001,"5min":11.178000000000001,"15min":11.178000000000001},"last":9.2059999999999995},{"interface":"front","average":{"1min":3.0169999999999999,"5min":2.0619999999999998,"15min":1.319},"min":{"1min":0.374,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":7.8319999999999999,"5min":12.602,"15min":12.602},"last":1.246}]},{"osd":5,"last update":"Fri Feb 9 17:32:27 2024","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":1.508,"15min":0.98799999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.27300000000000002},"max":{"1min":6.016,"5min":7.0110000000000001,"15min":7.0110000000000001},"last":5.665},{"interface":"front","average":{"1min":1.623,"5min":1.5609999999999999,"15min":1.0109999999999999},"min":{"1min":0.42899999999999999,"5min":0.36699999999999999,"15min":0.33300000000000002},"max":{"1min":6.0549999999999997,"5min":7.1600000000000001,"15min":7.1600000000000001},"last":5.702}]},{"osd":6,"last update":"Fri Feb 9 17:32:58 2024","interfaces":[{"interface":"back","average":{"1min":2.125,"5min":1.7949999999999999,"15min":1.232},"min":{"1min":0.38600000000000001,"5min":0.33900000000000002,"15min":0.32200000000000001},"max":{"1min":6.8700000000000001,"5min":19.582999999999998,"15min":19.582999999999998},"last":4.7709999999999999},{"interface":"front","average":{"1min":2.024,"5min":1.4710000000000001,"15min":1.032},"min":{"1min":0.34599999999999997,"5min":0.30199999999999999,"15min":0.29499999999999998},"max":{"1min":6.8529999999999998,"5min":7.9009999999999998,"15min":7.9009999999999998},"last":4.7489999999999997}]},{"osd":7,"last update":"Fri Feb 9 17:32:25 2024","interfaces":[{"interface":"back","average":{"1min":3.3559999999999999,"5min":2.2509999999999999,"15min":1.5209999999999999},"min":{"1min":0.36599999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":9.3770000000000007,"5min":10.416,"15min":10.791},"last":1.524},{"interface":"front","average":{"1min":1.99,"5min":2.0659999999999998,"15min":1.2809999999999999},"min":{"1min":0.40300000000000002,"5min":0.375,"15min":0.37},"max":{"1min":7.8490000000000002,"5min":19.550999999999998,"15min":19.550999999999998},"last":1.5529999999999999}]},{"osd":8,"last update":"Fri Feb 9 17:26:30 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":3.7629999999999999,"5min":3.7629999999999999,"15min":3.7629999999999999},"last":0.39100000000000001},{"interface":"front","average":{"1min":2.327,"5min":1.22,"15min":0.93000000000000005},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":10.167,"5min":10.167,"15min":10.167},"last":2.0819999999999999}]},{"osd":9,"last update":"Fri Feb 9 17:32:16 2024","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.4399999999999999,"15min":1.05},"min":{"1min":0.36299999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":3.9670000000000001,"5min":7.0810000000000004,"15min":7.0810000000000004},"last":0.627},{"interface":"front","average":{"1min":1.0089999999999999,"5min":1.7729999999999999,"15min":1.3600000000000001},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":2.2759999999999998,"5min":13.717000000000001,"15min":13.717000000000001},"last":2.1680000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:32:41 2024","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.762,"15min":1.276},"min":{"1min":0.48899999999999999,"5min":0.40899999999999997,"15min":0.39400000000000002},"max":{"1min":3.6739999999999999,"5min":19.611000000000001,"15min":19.611000000000001},"last":0.98399999999999999},{"interface":"front","average":{"1min":0.88700000000000001,"5min":1.4490000000000001,"15min":1.1200000000000001},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":2.2389999999999999,"5min":7.1109999999999998,"15min":10.036},"last":0.438}]},{"osd":11,"last update":"Fri Feb 9 17:33:01 2024","interfaces":[{"interface":"back","average":{"1min":1.788,"5min":1.4430000000000001,"15min":1.1499999999999999},"min":{"1min":0.495,"5min":0.40400000000000003,"15min":0.38800000000000001},"max":{"1min":8.6929999999999996,"5min":8.6929999999999996,"15min":8.6929999999999996},"last":0.57199999999999995},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.613,"15min":1.4450000000000001},"min":{"1min":0.44,"5min":0.39300000000000002,"15min":0.36899999999999999},"max":{"1min":4.0979999999999999,"5min":10.465,"15min":10.882999999999999},"last":1.4970000000000001}]}]},{"osd":4,"up_from":32,"seq":137438953638,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2444012,"kb_used_data":2214308,"kb_used_omap":1,"kb_used_meta":229694,"kb_avail":91301140,"statfs":{"total":95995035648,"available":93492367360,"internally_reserved":0,"allocated":2267451392,"data_stored":2267078028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":235207109},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:32:57 2024","interfaces":[{"interface":"back","average":{"1min":2.7669999999999999,"5min":1.474,"15min":1.006},"min":{"1min":0.28899999999999998,"5min":0.249,"15min":0.249},"max":{"1min":6.3739999999999997,"5min":10.48,"15min":10.48},"last":0.84699999999999998},{"interface":"front","average":{"1min":1.607,"5min":1.157,"15min":0.96499999999999997},"min":{"1min":0.29199999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":4.8289999999999997,"5min":6.0049999999999999,"15min":6.0049999999999999},"last":2.3370000000000002}]},{"osd":1,"last update":"Fri Feb 9 17:33:03 2024","interfaces":[{"interface":"back","average":{"1min":1.97,"5min":1.403,"15min":1.2529999999999999},"min":{"1min":0.29499999999999998,"5min":0.255,"15min":0.246},"max":{"1min":5.9930000000000003,"5min":6.0919999999999996,"15min":42.542999999999999},"last":2.3740000000000001},{"interface":"front","average":{"1min":2.0139999999999998,"5min":1.472,"15min":1.25},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":8.3870000000000005,"5min":9.2769999999999992,"15min":41.813000000000002},"last":2.4180000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:28:51 2024","interfaces":[{"interface":"back","average":{"1min":2.536,"5min":1.546,"15min":0.91400000000000003},"min":{"1min":0.29999999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":8.8819999999999997,"5min":8.8819999999999997,"15min":8.8819999999999997},"last":0.622},{"interface":"front","average":{"1min":2.214,"5min":1.7330000000000001,"15min":0.97799999999999998},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":9.5039999999999996,"5min":9.5039999999999996,"15min":9.5039999999999996},"last":0.53800000000000003}]},{"osd":3,"last update":"Fri Feb 9 17:33:03 2024","interfaces":[{"interface":"back","average":{"1min":3.2309999999999999,"5min":2.3100000000000001,"15min":1.419},"min":{"1min":0.34100000000000003,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":6.2619999999999996,"5min":33.037999999999997,"15min":33.037999999999997},"last":0.81799999999999995},{"interface":"front","average":{"1min":2.145,"5min":1.9630000000000001,"15min":1.254},"min":{"1min":0.34000000000000002,"5min":0.314,"15min":0.28199999999999997},"max":{"1min":5.9539999999999997,"5min":33.054000000000002,"15min":33.054000000000002},"last":2.3959999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:32:26 2024","interfaces":[{"interface":"back","average":{"1min":1.5960000000000001,"5min":1.196,"15min":0.749},"min":{"1min":0.27900000000000003,"5min":0.222,"15min":0.222},"max":{"1min":8.5660000000000007,"5min":8.5660000000000007,"15min":8.5660000000000007},"last":0.34300000000000003},{"interface":"front","average":{"1min":1.673,"5min":1.294,"15min":0.83399999999999996},"min":{"1min":0.377,"5min":0.29199999999999998,"15min":0.26000000000000001},"max":{"1min":8.5879999999999992,"5min":8.5879999999999992,"15min":8.5879999999999992},"last":0.36699999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:33:03 2024","interfaces":[{"interface":"back","average":{"1min":2.657,"5min":1.5669999999999999,"15min":0.99199999999999999},"min":{"1min":0.26200000000000001,"5min":0.247,"15min":0.247},"max":{"1min":8.0440000000000005,"5min":18.09,"15min":18.09},"last":6.7320000000000002},{"interface":"front","average":{"1min":3.0219999999999998,"5min":1.6639999999999999,"15min":0.97299999999999998},"min":{"1min":0.36299999999999999,"5min":0.30099999999999999,"15min":0.28000000000000003},"max":{"1min":10.505000000000001,"5min":18.126000000000001,"15min":18.126000000000001},"last":2.7639999999999998}]},{"osd":7,"last update":"Fri Feb 9 17:32:26 2024","interfaces":[{"interface":"back","average":{"1min":1.663,"5min":1.361,"15min":0.96499999999999997},"min":{"1min":0.35099999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":8.3439999999999994,"5min":8.3439999999999994,"15min":15.223000000000001},"last":2.4420000000000002},{"interface":"front","average":{"1min":2.234,"5min":1.5489999999999999,"15min":0.996},"min":{"1min":0.379,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":5.2699999999999996,"5min":19.047000000000001,"15min":19.047000000000001},"last":5.5640000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:32:26 2024","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.3480000000000001,"15min":1.264},"min":{"1min":0.41799999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":8.4160000000000004,"5min":8.4160000000000004,"15min":8.4160000000000004},"last":2.4620000000000002},{"interface":"front","average":{"1min":1.325,"5min":1.2450000000000001,"15min":1.087},"min":{"1min":0.45100000000000001,"5min":0.308,"15min":0.308},"max":{"1min":8.5169999999999995,"5min":8.5169999999999995,"15min":8.5169999999999995},"last":0.38900000000000001}]},{"osd":9,"last update":"Fri Feb 9 17:33:13 2024","interfaces":[{"interface":"back","average":{"1min":2.6970000000000001,"5min":1.5960000000000001,"15min":1.123},"min":{"1min":0.39200000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":8.4920000000000009,"5min":8.4920000000000009,"15min":8.4920000000000009},"last":3.7429999999999999},{"interface":"front","average":{"1min":3.0230000000000001,"5min":1.7030000000000001,"15min":1.1479999999999999},"min":{"1min":0.54400000000000004,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":9.9770000000000003,"5min":10.555,"15min":10.555},"last":0.872}]},{"osd":10,"last update":"Fri Feb 9 17:32:47 2024","interfaces":[{"interface":"back","average":{"1min":2.2919999999999998,"5min":1.613,"15min":1.139},"min":{"1min":0.56200000000000006,"5min":0.40000000000000002,"15min":0.378},"max":{"1min":8.468,"5min":8.468,"15min":8.468},"last":0.63300000000000001},{"interface":"front","average":{"1min":2.0419999999999998,"5min":1.5329999999999999,"15min":1.097},"min":{"1min":0.54400000000000004,"5min":0.39200000000000002,"15min":0.38100000000000001},"max":{"1min":8.4440000000000008,"5min":10.597,"15min":10.597},"last":3.6379999999999999}]},{"osd":11,"last update":"Fri Feb 9 17:33:09 2024","interfaces":[{"interface":"back","average":{"1min":3.8919999999999999,"5min":1.994,"15min":1.5409999999999999},"min":{"1min":0.53500000000000003,"5min":0.42099999999999999,"15min":0.40500000000000003},"max":{"1min":18.327999999999999,"5min":18.327999999999999,"15min":18.327999999999999},"last":0.59099999999999997},{"interface":"front","average":{"1min":4.2530000000000001,"5min":2.024,"15min":1.4279999999999999},"min":{"1min":0.60299999999999998,"5min":0.42799999999999999,"15min":0.34899999999999998},"max":{"1min":18.574999999999999,"5min":18.574999999999999,"15min":18.574999999999999},"last":0.77800000000000002}]}]},{"osd":5,"up_from":40,"seq":171798692000,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1386232,"kb_used_data":1274544,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":92358920,"statfs":{"total":95995035648,"available":94575534080,"internally_reserved":0,"allocated":1305133056,"data_stored":1304752068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:32:25 2024","interfaces":[{"interface":"back","average":{"1min":1.8540000000000001,"5min":1.653,"15min":1.0900000000000001},"min":{"1min":0.318,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":7.7160000000000002,"5min":8.7690000000000001,"15min":11.923},"last":3.8450000000000002},{"interface":"front","average":{"1min":2.4089999999999998,"5min":1.8260000000000001,"15min":1.032},"min":{"1min":0.34499999999999997,"5min":0.24299999999999999,"15min":0.224},"max":{"1min":14.082000000000001,"5min":14.082000000000001,"15min":14.082000000000001},"last":0.79400000000000004}]},{"osd":1,"last update":"Fri Feb 9 17:32:25 2024","interfaces":[{"interface":"back","average":{"1min":3.1659999999999999,"5min":2.1339999999999999,"15min":1.371},"min":{"1min":0.38,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":14.032999999999999,"5min":16.896000000000001,"15min":29.715},"last":2.302},{"interface":"front","average":{"1min":1.145,"5min":1.8220000000000001,"15min":1.2170000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":7.2750000000000004,"5min":16.878,"15min":29.969999999999999},"last":1.4450000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:32:25 2024","interfaces":[{"interface":"back","average":{"1min":1.4370000000000001,"5min":1.734,"15min":1.167},"min":{"1min":0.33100000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":10.5,"5min":10.5,"15min":10.5},"last":1.47},{"interface":"front","average":{"1min":2.6269999999999998,"5min":1.907,"15min":1.244},"min":{"1min":0.376,"5min":0.318,"15min":0.318},"max":{"1min":14.122,"5min":14.122,"15min":14.122},"last":2.3610000000000002}]},{"osd":3,"last update":"Fri Feb 9 17:32:25 2024","interfaces":[{"interface":"back","average":{"1min":1.885,"5min":1.843,"15min":1.202},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":7.2270000000000003,"5min":10.587,"15min":10.587},"last":1.494},{"interface":"front","average":{"1min":2.4769999999999999,"5min":1.976,"15min":1.1579999999999999},"min":{"1min":0.30499999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":14.154999999999999,"5min":14.154999999999999,"15min":14.154999999999999},"last":2.335}]},{"osd":4,"last update":"Fri Feb 9 17:32:38 2024","interfaces":[{"interface":"back","average":{"1min":1.9990000000000001,"5min":1.2949999999999999,"15min":0.746},"min":{"1min":0.31900000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":8.7260000000000009,"5min":8.7260000000000009,"15min":8.7260000000000009},"last":0.75},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.298,"15min":0.80700000000000005},"min":{"1min":0.38500000000000001,"5min":0.28100000000000003,"15min":0.27000000000000002},"max":{"1min":4.8209999999999997,"5min":8.6349999999999998,"15min":8.6349999999999998},"last":1.409}]},{"osd":6,"last update":"Fri Feb 9 17:33:01 2024","interfaces":[{"interface":"back","average":{"1min":2.9710000000000001,"5min":1.7909999999999999,"15min":1.022},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.28999999999999998},"max":{"1min":15.375999999999999,"5min":15.375999999999999,"15min":15.375999999999999},"last":2.6560000000000001},{"interface":"front","average":{"1min":3.1840000000000002,"5min":1.819,"15min":1.0089999999999999},"min":{"1min":0.46000000000000002,"5min":0.29399999999999998,"15min":0.27200000000000002},"max":{"1min":14.250999999999999,"5min":14.250999999999999,"15min":14.250999999999999},"last":2.4350000000000001}]},{"osd":7,"last update":"Fri Feb 9 17:32:25 2024","interfaces":[{"interface":"back","average":{"1min":2.0670000000000002,"5min":1.339,"15min":0.94399999999999995},"min":{"1min":0.40300000000000002,"5min":0.32200000000000001,"15min":0.32100000000000001},"max":{"1min":8.8100000000000005,"5min":8.8100000000000005,"15min":18.626999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":2.867,"5min":1.7809999999999999,"15min":1.095},"min":{"1min":0.34499999999999997,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":19.044,"5min":19.044,"15min":19.044},"last":2.5059999999999998}]},{"osd":8,"last update":"Fri Feb 9 17:26:36 2024","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":0.91000000000000003,"15min":0.78000000000000003},"min":{"1min":0.39400000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":5.556,"5min":5.556,"15min":5.556},"last":1.5580000000000001},{"interface":"front","average":{"1min":1.3720000000000001,"5min":0.92600000000000005,"15min":0.82999999999999996},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":5.9000000000000004,"5min":5.9000000000000004,"15min":5.9000000000000004},"last":2.3849999999999998}]},{"osd":9,"last update":"Fri Feb 9 17:33:06 2024","interfaces":[{"interface":"back","average":{"1min":2.024,"5min":1.4259999999999999,"15min":1.0800000000000001},"min":{"1min":0.38900000000000001,"5min":0.371,"15min":0.35999999999999999},"max":{"1min":6.4139999999999997,"5min":8.4280000000000008,"15min":8.4280000000000008},"last":1.5189999999999999},{"interface":"front","average":{"1min":2.452,"5min":1.5529999999999999,"15min":1.1160000000000001},"min":{"1min":0.36699999999999999,"5min":0.35599999999999998,"15min":0.312},"max":{"1min":8.343,"5min":8.3689999999999998,"15min":8.3689999999999998},"last":5.5259999999999998}]},{"osd":10,"last update":"Fri Feb 9 17:32:38 2024","interfaces":[{"interface":"back","average":{"1min":2.5670000000000002,"5min":1.579,"15min":1.1519999999999999},"min":{"1min":0.58599999999999997,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":14.218,"5min":14.218,"15min":14.218},"last":2.4830000000000001},{"interface":"front","average":{"1min":2.5960000000000001,"5min":1.637,"15min":1.1899999999999999},"min":{"1min":0.57499999999999996,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":14.186999999999999,"5min":14.186999999999999,"15min":14.186999999999999},"last":2.4100000000000001}]},{"osd":11,"last update":"Fri Feb 9 17:33:11 2024","interfaces":[{"interface":"back","average":{"1min":2.5590000000000002,"5min":1.6000000000000001,"15min":1.3600000000000001},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":8.3849999999999998,"5min":8.3979999999999997,"15min":8.3979999999999997},"last":1.5900000000000001},{"interface":"front","average":{"1min":4.5609999999999999,"5min":2.04,"15min":1.236},"min":{"1min":0.56000000000000005,"5min":0.437,"15min":0.38900000000000001},"max":{"1min":14.316000000000001,"5min":14.316000000000001,"15min":14.316000000000001},"last":2.4580000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2352545792,"data_stored":2352545792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1289093120,"data_stored":1289093120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":317063168,"data_stored":317063168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1645215744,"data_stored":1645215744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2264006656,"data_stored":2264006656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1293881344,"data_stored":1293877272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1597308928,"data_stored":1597308928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1324290048,"data_stored":1324285976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":658702336,"data_stored":658702336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":951324672,"data_stored":951320600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":325648384,"data_stored":325648384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2577399808,"data_stored":2577399808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-09T17:33:19.995 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd pool set unique_pool_1 pgp_num 16 2024-02-09T17:33:20.100 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is set 2024-02-09T17:33:20.320 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:33:20.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:20.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:33:20.433 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:33:20.503 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:20.543 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:33:20.552 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:20.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:33:20.654 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:33:20.721 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:20.753 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:33:20.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:20.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:33:20.795 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:33:20.861 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:33:20.892 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:20.949 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:33:20.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:20.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:33:21.055 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:33:21.097 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:21.123 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:33:21.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:21.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:33:21.241 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:33:21.304 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:21.339 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:33:21.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:21.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:33:21.447 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:33:21.486 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:21.667 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:21 smithi099 ceph-mon[39420]: pgmap v768: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 4.7 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 629 op/s; 3576/229173 objects degraded (1.560%); 18980/229173 objects misplaced (8.282%); 6.9 MiB/s, 110 objects/s recovering 2024-02-09T17:33:21.668 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:21 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 3576/229173 objects degraded (1.560%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:33:21.668 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:21 smithi099 ceph-mon[39420]: osdmap e271: 12 total, 12 up, 11 in 2024-02-09T17:33:21.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:21 smithi179 ceph-mon[43975]: pgmap v768: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 4.7 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 629 op/s; 3576/229173 objects degraded (1.560%); 18980/229173 objects misplaced (8.282%); 6.9 MiB/s, 110 objects/s recovering 2024-02-09T17:33:21.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:21 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 3576/229173 objects degraded (1.560%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:33:21.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:21 smithi179 ceph-mon[43975]: osdmap e271: 12 total, 12 up, 11 in 2024-02-09T17:33:21.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:21 smithi162 ceph-mon[44304]: pgmap v768: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 4.7 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 629 op/s; 3576/229173 objects degraded (1.560%); 18980/229173 objects misplaced (8.282%); 6.9 MiB/s, 110 objects/s recovering 2024-02-09T17:33:21.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:21 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 3576/229173 objects degraded (1.560%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:33:21.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:21 smithi162 ceph-mon[44304]: osdmap e271: 12 total, 12 up, 11 in 2024-02-09T17:33:22.037 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:33:22.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:22.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:33:22.089 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:33:22.113 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:33:22.138 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:33:22.141 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:33:22.162 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:33:22.192 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:22.211 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:33:22.254 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:33:22.256 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:33:22.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:22.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:33:22.289 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:33:22.328 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:33:22.355 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:33:22.361 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:33:22.387 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:33:22.412 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:33:22.413 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:22.434 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:33:22.435 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:33:22.435 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:33:22.435 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:33:22.435 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:33:22.435 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:33:22.436 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:33:22.436 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:33:22.436 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:33:22.436 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:33:22.436 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:33:22.436 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:33:22.436 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:33:22.449 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:33:22.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:22.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:33:22.572 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:33:22.605 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:22.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:22 smithi099 ceph-mon[39420]: from='client.15309 172.21.15.99:0/3020589020' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:33:22.816 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:33:22.820 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:33:22.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:22.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:33:22.822 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-09T17:33:22.921 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:33:22.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:22 smithi162 ceph-mon[44304]: from='client.15309 172.21.15.99:0/3020589020' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:33:22.951 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:23.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:22 smithi179 ceph-mon[43975]: from='client.15309 172.21.15.99:0/3020589020' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:33:23.609 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:33:23.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:23.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:33:23.714 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:33:23.788 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:23.822 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:33:23.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:23.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:33:23.892 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:23 smithi099 ceph-mon[39420]: pgmap v770: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 4.6 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 738 op/s; 3576/225189 objects degraded (1.588%); 18980/225189 objects misplaced (8.428%); 8.6 MiB/s, 137 objects/s recovering 2024-02-09T17:33:23.937 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:33:23.986 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:23 smithi162 ceph-mon[44304]: pgmap v770: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 4.6 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 738 op/s; 3576/225189 objects degraded (1.588%); 18980/225189 objects misplaced (8.428%); 8.6 MiB/s, 137 objects/s recovering 2024-02-09T17:33:23.991 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:24.040 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:33:24.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:24.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:33:24.149 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:33:24.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:23 smithi179 ceph-mon[43975]: pgmap v770: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 4.6 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 738 op/s; 3576/225189 objects degraded (1.588%); 18980/225189 objects misplaced (8.428%); 8.6 MiB/s, 137 objects/s recovering 2024-02-09T17:33:24.198 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:24.235 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:33:24.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:24.241 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:33:24.342 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:33:24.380 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:24.411 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:33:24.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:24.419 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:33:24.520 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:33:24.557 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:24.671 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:33:24.671 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:33:25.413 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:33:25.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:25 smithi099 ceph-mon[39420]: pgmap v771: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 4.6 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 592 op/s; 3576/224088 objects degraded (1.596%); 18980/224088 objects misplaced (8.470%); 4.3 MiB/s, 68 objects/s recovering 2024-02-09T17:33:26.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:25 smithi179 ceph-mon[43975]: pgmap v771: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 4.6 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 592 op/s; 3576/224088 objects degraded (1.596%); 18980/224088 objects misplaced (8.470%); 4.3 MiB/s, 68 objects/s recovering 2024-02-09T17:33:26.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:25 smithi162 ceph-mon[44304]: pgmap v771: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 4.6 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 592 op/s; 3576/224088 objects degraded (1.596%); 18980/224088 objects misplaced (8.470%); 4.3 MiB/s, 68 objects/s recovering 2024-02-09T17:33:26.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:26 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 3576/224088 objects degraded (1.596%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:33:27.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:26 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 3576/224088 objects degraded (1.596%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:33:27.177 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:26 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 3576/224088 objects degraded (1.596%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:33:27.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:27 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:33:27.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:27 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:33:27.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:27 smithi099 ceph-mon[39420]: pgmap v772: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 4.5 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 635 op/s; 3838/219288 objects degraded (1.750%); 18150/219288 objects misplaced (8.277%); 8.7 MiB/s, 139 objects/s recovering 2024-02-09T17:33:27.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:27 smithi099 ceph-mon[39420]: osdmap e272: 12 total, 12 up, 11 in 2024-02-09T17:33:28.175 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:27 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:33:28.176 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:27 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:33:28.176 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:27 smithi179 ceph-mon[43975]: pgmap v772: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 4.5 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 635 op/s; 3838/219288 objects degraded (1.750%); 18150/219288 objects misplaced (8.277%); 8.7 MiB/s, 139 objects/s recovering 2024-02-09T17:33:28.176 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:27 smithi179 ceph-mon[43975]: osdmap e272: 12 total, 12 up, 11 in 2024-02-09T17:33:28.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:27 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:33:28.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:27 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:33:28.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:27 smithi162 ceph-mon[44304]: pgmap v772: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 4.5 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 635 op/s; 3838/219288 objects degraded (1.750%); 18150/219288 objects misplaced (8.277%); 8.7 MiB/s, 139 objects/s recovering 2024-02-09T17:33:28.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:27 smithi162 ceph-mon[44304]: osdmap e272: 12 total, 12 up, 11 in 2024-02-09T17:33:29.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:29 smithi099 ceph-mon[39420]: pgmap v774: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 4.4 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 565 op/s; 3838/215607 objects degraded (1.780%); 18150/215607 objects misplaced (8.418%); 4.4 MiB/s, 70 objects/s recovering 2024-02-09T17:33:30.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:29 smithi179 ceph-mon[43975]: pgmap v774: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 4.4 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 565 op/s; 3838/215607 objects degraded (1.780%); 18150/215607 objects misplaced (8.418%); 4.4 MiB/s, 70 objects/s recovering 2024-02-09T17:33:30.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:29 smithi162 ceph-mon[44304]: pgmap v774: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 4.4 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 565 op/s; 3838/215607 objects degraded (1.780%); 18150/215607 objects misplaced (8.418%); 4.4 MiB/s, 70 objects/s recovering 2024-02-09T17:33:31.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:31 smithi099 ceph-mon[39420]: pgmap v775: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 4.3 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 661 op/s; 4062/210078 objects degraded (1.934%); 17340/210078 objects misplaced (8.254%); 7.7 MiB/s, 123 objects/s recovering 2024-02-09T17:33:31.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:31 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 4062/210078 objects degraded (1.934%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:33:31.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:31 smithi099 ceph-mon[39420]: osdmap e273: 12 total, 12 up, 11 in 2024-02-09T17:33:32.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:31 smithi179 ceph-mon[43975]: pgmap v775: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 4.3 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 661 op/s; 4062/210078 objects degraded (1.934%); 17340/210078 objects misplaced (8.254%); 7.7 MiB/s, 123 objects/s recovering 2024-02-09T17:33:32.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:31 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 4062/210078 objects degraded (1.934%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:33:32.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:31 smithi179 ceph-mon[43975]: osdmap e273: 12 total, 12 up, 11 in 2024-02-09T17:33:32.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:31 smithi162 ceph-mon[44304]: pgmap v775: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 4.3 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 661 op/s; 4062/210078 objects degraded (1.934%); 17340/210078 objects misplaced (8.254%); 7.7 MiB/s, 123 objects/s recovering 2024-02-09T17:33:32.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:31 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 4062/210078 objects degraded (1.934%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:33:32.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:31 smithi162 ceph-mon[44304]: osdmap e273: 12 total, 12 up, 11 in 2024-02-09T17:33:33.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:33 smithi099 ceph-mon[39420]: pgmap v777: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 4.2 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 762 op/s; 4062/205773 objects degraded (1.974%); 17340/205773 objects misplaced (8.427%); 9.0 MiB/s, 144 objects/s recovering 2024-02-09T17:33:34.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:33 smithi179 ceph-mon[43975]: pgmap v777: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 4.2 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 762 op/s; 4062/205773 objects degraded (1.974%); 17340/205773 objects misplaced (8.427%); 9.0 MiB/s, 144 objects/s recovering 2024-02-09T17:33:34.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:33 smithi162 ceph-mon[44304]: pgmap v777: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 4.2 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 762 op/s; 4062/205773 objects degraded (1.974%); 17340/205773 objects misplaced (8.427%); 9.0 MiB/s, 144 objects/s recovering 2024-02-09T17:33:35.001 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-09T17:33:36.037 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:33:36.039 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:35 smithi099 ceph-mon[39420]: pgmap v778: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 4.2 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 608 op/s; 4062/204690 objects degraded (1.984%); 17340/204690 objects misplaced (8.471%); 4.6 MiB/s, 73 objects/s recovering 2024-02-09T17:33:36.039 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:35 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:33:36.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:36.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:33:36.062 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:33:36.141 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:33:36.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:35 smithi179 ceph-mon[43975]: pgmap v778: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 4.2 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 608 op/s; 4062/204690 objects degraded (1.984%); 17340/204690 objects misplaced (8.471%); 4.6 MiB/s, 73 objects/s recovering 2024-02-09T17:33:36.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:35 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:33:36.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:35 smithi162 ceph-mon[44304]: pgmap v778: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 4.2 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 608 op/s; 4062/204690 objects degraded (1.984%); 17340/204690 objects misplaced (8.471%); 4.6 MiB/s, 73 objects/s recovering 2024-02-09T17:33:36.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:35 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:33:36.197 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:33:36.224 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:36.257 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:33:36.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:36.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:33:36.369 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:33:36.428 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:36.599 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:33:36.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:36.602 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:33:36.703 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:33:36.751 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:36.781 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:33:36.789 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:36.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:33:36.890 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:33:36.944 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:36.995 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:33:37.011 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:36 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:33:37.011 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:36 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:33:37.011 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:36 smithi162 ceph-mon[44304]: osdmap e274: 12 total, 12 up, 11 in 2024-02-09T17:33:37.011 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:36 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 4062/204690 objects degraded (1.984%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:33:37.011 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:36 smithi162 ceph-mon[44304]: osdmap e275: 12 total, 12 up, 11 in 2024-02-09T17:33:37.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:37.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:33:37.113 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:33:37.164 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:36 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:33:37.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:36 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:33:37.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:36 smithi179 ceph-mon[43975]: osdmap e274: 12 total, 12 up, 11 in 2024-02-09T17:33:37.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:36 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 4062/204690 objects degraded (1.984%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:33:37.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:36 smithi179 ceph-mon[43975]: osdmap e275: 12 total, 12 up, 11 in 2024-02-09T17:33:37.171 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:37.197 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:33:37.202 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:37.203 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:33:37.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:36 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:33:37.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:36 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:33:37.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:36 smithi099 ceph-mon[39420]: osdmap e274: 12 total, 12 up, 11 in 2024-02-09T17:33:37.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:36 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 4062/204690 objects degraded (1.984%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:33:37.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:36 smithi099 ceph-mon[39420]: osdmap e275: 12 total, 12 up, 11 in 2024-02-09T17:33:37.304 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:33:37.371 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:37.407 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:33:37.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:37.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:33:37.519 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:33:37.583 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:37.638 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:33:37.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:37.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:33:37.742 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:33:37.828 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:37.855 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:33:37.859 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:37.860 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:33:37.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:37 smithi162 ceph-mon[44304]: pgmap v781: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 4.1 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 571 op/s; 4329/199791 objects degraded (2.167%); 16469/199791 objects misplaced (8.243%); 6.3 MiB/s, 100 objects/s recovering 2024-02-09T17:33:37.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:37 smithi162 ceph-mon[44304]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:33:37.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:37 smithi162 ceph-mon[44304]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:33:37.961 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:33:38.009 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:38.053 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:33:38.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:38.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:33:38.170 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:33:38.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:37 smithi179 ceph-mon[43975]: pgmap v781: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 4.1 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 571 op/s; 4329/199791 objects degraded (2.167%); 16469/199791 objects misplaced (8.243%); 6.3 MiB/s, 100 objects/s recovering 2024-02-09T17:33:38.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:37 smithi179 ceph-mon[43975]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:33:38.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:37 smithi179 ceph-mon[43975]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:33:38.214 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:38.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:37 smithi099 ceph-mon[39420]: pgmap v781: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 4.1 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 571 op/s; 4329/199791 objects degraded (2.167%); 16469/199791 objects misplaced (8.243%); 6.3 MiB/s, 100 objects/s recovering 2024-02-09T17:33:38.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:37 smithi099 ceph-mon[39420]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:33:38.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:37 smithi099 ceph-mon[39420]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:33:38.251 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:33:38.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:38.258 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:33:38.359 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:33:38.401 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:38.445 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:33:38.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:38.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:33:38.551 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:33:38.614 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:38.665 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:33:38.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:38.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:33:38.768 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:33:38.807 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:38.907 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:33:38.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:38.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:33:38.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:38 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:33:38.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:38 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:33:38.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:38 smithi162 ceph-mon[44304]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:33:39.012 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:33:39.044 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:38 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:33:39.045 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:38 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:33:39.045 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:38 smithi099 ceph-mon[39420]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:33:39.047 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:39.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:38 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:33:39.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:38 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:33:39.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:38 smithi179 ceph-mon[43975]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:33:39.263 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:33:39.501 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:33:39.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:39.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:33:39.607 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:33:39.641 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:39.987 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:39 smithi099 ceph-mon[39420]: pgmap v782: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 4.0 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 601 op/s; 4329/196242 objects degraded (2.206%); 16469/196242 objects misplaced (8.392%); 5.2 MiB/s, 82 objects/s recovering 2024-02-09T17:33:40.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:39 smithi179 ceph-mon[43975]: pgmap v782: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 4.0 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 601 op/s; 4329/196242 objects degraded (2.206%); 16469/196242 objects misplaced (8.392%); 5.2 MiB/s, 82 objects/s recovering 2024-02-09T17:33:40.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:39 smithi162 ceph-mon[44304]: pgmap v782: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 4.0 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 601 op/s; 4329/196242 objects degraded (2.206%); 16469/196242 objects misplaced (8.392%); 5.2 MiB/s, 82 objects/s recovering 2024-02-09T17:33:40.397 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:33:40.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:40.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:33:40.505 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:33:40.535 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:40.799 INFO:teuthology.orchestra.run.smithi099.stderr:set pool 3 pgp_num to 16 2024-02-09T17:33:40.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:40 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/822508465' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]: dispatch 2024-02-09T17:33:41.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:40 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/822508465' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]: dispatch 2024-02-09T17:33:41.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:40 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/822508465' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]: dispatch 2024-02-09T17:33:41.490 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:33:41.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:41.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:33:41.596 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:33:41.632 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:41.659 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:33:41.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:41.664 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:33:41.765 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:33:41.811 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:41 smithi099 ceph-mon[39420]: pgmap v783: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 3.9 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 613 op/s; 4527/191037 objects degraded (2.370%); 15683/191037 objects misplaced (8.209%); 9.3 MiB/s, 148 objects/s recovering 2024-02-09T17:33:41.811 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:41 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/822508465' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]': finished 2024-02-09T17:33:41.811 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:41 smithi099 ceph-mon[39420]: osdmap e276: 12 total, 12 up, 11 in 2024-02-09T17:33:41.811 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:41 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 4527/191037 objects degraded (2.370%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:33:41.811 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:41 smithi099 ceph-mon[39420]: osdmap e277: 12 total, 12 up, 11 in 2024-02-09T17:33:41.842 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:41.879 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:33:41.888 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:41.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:33:41.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:41 smithi179 ceph-mon[43975]: pgmap v783: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 3.9 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 613 op/s; 4527/191037 objects degraded (2.370%); 15683/191037 objects misplaced (8.209%); 9.3 MiB/s, 148 objects/s recovering 2024-02-09T17:33:41.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:41 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/822508465' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]': finished 2024-02-09T17:33:41.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:41 smithi179 ceph-mon[43975]: osdmap e276: 12 total, 12 up, 11 in 2024-02-09T17:33:41.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:41 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 4527/191037 objects degraded (2.370%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:33:41.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:41 smithi179 ceph-mon[43975]: osdmap e277: 12 total, 12 up, 11 in 2024-02-09T17:33:41.990 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:33:42.019 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:42.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:41 smithi162 ceph-mon[44304]: pgmap v783: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 3.9 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 613 op/s; 4527/191037 objects degraded (2.370%); 15683/191037 objects misplaced (8.209%); 9.3 MiB/s, 148 objects/s recovering 2024-02-09T17:33:42.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:41 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/822508465' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]': finished 2024-02-09T17:33:42.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:41 smithi162 ceph-mon[44304]: osdmap e276: 12 total, 12 up, 11 in 2024-02-09T17:33:42.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:41 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 4527/191037 objects degraded (2.370%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:33:42.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:41 smithi162 ceph-mon[44304]: osdmap e277: 12 total, 12 up, 11 in 2024-02-09T17:33:42.873 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:33:42.876 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:42.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:33:42.977 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:33:42.989 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:33:42.989 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:33:43.017 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:43.051 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:33:43.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:43.062 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:33:43.162 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:33:43.210 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:43.242 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:33:43.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:43.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:33:43.350 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:33:43.416 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:43.446 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:33:43.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:43.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:33:43.552 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:33:43.588 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:43.644 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:33:43.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:43.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:33:43.669 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-09T17:33:43.755 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:33:43.787 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:43.932 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:43 smithi099 ceph-mon[39420]: pgmap v786: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 3.8 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 674 op/s; 4527/187023 objects degraded (2.421%); 15683/187023 objects misplaced (8.386%); 5.8 MiB/s, 93 objects/s recovering 2024-02-09T17:33:43.940 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:33:43.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:43.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:33:44.051 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:33:44.091 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:43 smithi179 ceph-mon[43975]: pgmap v786: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 3.8 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 674 op/s; 4527/187023 objects degraded (2.421%); 15683/187023 objects misplaced (8.386%); 5.8 MiB/s, 93 objects/s recovering 2024-02-09T17:33:44.094 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:44.138 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:33:44.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:44.144 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:33:44.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:43 smithi162 ceph-mon[44304]: pgmap v786: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 3.8 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 674 op/s; 4527/187023 objects degraded (2.421%); 15683/187023 objects misplaced (8.386%); 5.8 MiB/s, 93 objects/s recovering 2024-02-09T17:33:44.231 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:33:44.245 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:33:44.259 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:33:44.283 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:33:44.298 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:44.314 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:33:44.354 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:33:44.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:44.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:33:44.364 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:33:44.391 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:33:44.428 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:33:44.460 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:33:44.462 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:33:44.496 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:33:44.528 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:33:44.532 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:44.551 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:33:44.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:44.558 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:33:44.558 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:33:44.586 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:33:44.588 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:33:44.588 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:33:44.588 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:33:44.588 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:33:44.588 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:33:44.588 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:33:44.588 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:33:44.588 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:33:44.588 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:33:44.588 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:33:44.588 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:33:44.588 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:33:44.658 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:33:44.693 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:44.826 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:44 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/558584107' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:33:44.906 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:33:44.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:44.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:33:44.912 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:44 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/558584107' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:33:45.012 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:33:45.072 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:44 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/558584107' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:33:45.077 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:45.105 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:33:45.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:45.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:33:45.215 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:33:45.260 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:45.300 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:33:45.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:45.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:33:45.405 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:33:45.438 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:45.462 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:33:45.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:45.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:33:45.567 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:33:45.595 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:45.748 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:33:45.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:45.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:33:45.853 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:33:45.884 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:45.941 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:33:45.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:45.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:33:46.049 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:33:46.083 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:45 smithi099 ceph-mon[39420]: pgmap v787: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 3.8 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 577 op/s; 4527/185922 objects degraded (2.435%); 15683/185922 objects misplaced (8.435%); 4.6 MiB/s, 73 objects/s recovering 2024-02-09T17:33:46.085 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:46.113 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:33:46.119 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:46.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:33:46.143 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:33:46.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:45 smithi179 ceph-mon[43975]: pgmap v787: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 3.8 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 577 op/s; 4527/185922 objects degraded (2.435%); 15683/185922 objects misplaced (8.435%); 4.6 MiB/s, 73 objects/s recovering 2024-02-09T17:33:46.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:45 smithi162 ceph-mon[44304]: pgmap v787: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 3.8 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 577 op/s; 4527/185922 objects degraded (2.435%); 15683/185922 objects misplaced (8.435%); 4.6 MiB/s, 73 objects/s recovering 2024-02-09T17:33:46.220 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:33:46.272 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:46.312 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:33:46.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:46.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:33:46.425 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:33:46.466 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:46.498 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:33:46.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:46.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:33:46.605 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:33:46.647 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:46.683 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:33:46.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:46.706 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:33:46.807 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:33:46.880 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is set 2024-02-09T17:33:46.881 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:46.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:46 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 4527/185922 objects degraded (2.435%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:33:46.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:46 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3086525561' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:33:46.938 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:33:46.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:46.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:33:46.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:46 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 4527/185922 objects degraded (2.435%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:33:46.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:46 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3086525561' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:33:47.042 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:33:47.123 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:46 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 4527/185922 objects degraded (2.435%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:33:47.123 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:46 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3086525561' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:33:47.124 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:47.153 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:33:47.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:47.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:33:47.258 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:33:47.321 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:47.359 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:33:47.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:47.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:33:47.445 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:33:47.471 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:33:47.506 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:47.548 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:33:47.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:47.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:33:47.655 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:33:47.692 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:47.727 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:33:47.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:47.730 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:33:47.831 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:33:47.882 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:47 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:33:47.882 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:47 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:33:47.882 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:47 smithi162 ceph-mon[44304]: pgmap v788: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 3.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 620 op/s; 4665/181338 objects degraded (2.573%); 14893/181338 objects misplaced (8.213%); 9.6 MiB/s, 154 objects/s recovering 2024-02-09T17:33:47.882 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:47 smithi162 ceph-mon[44304]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:33:47.882 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:47 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3086525561' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:33:47.882 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:47 smithi162 ceph-mon[44304]: osdmap e278: 12 total, 12 up, 11 in 2024-02-09T17:33:47.884 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:47.901 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] 2024-02-09T17:33:47.901 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:33:47.902 INFO:tasks.thrashosds.thrasher:Adding osd 0 2024-02-09T17:33:47.902 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd in 0 2024-02-09T17:33:47.942 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:47 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:33:47.942 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:47 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:33:47.942 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:47 smithi179 ceph-mon[43975]: pgmap v788: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 3.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 620 op/s; 4665/181338 objects degraded (2.573%); 14893/181338 objects misplaced (8.213%); 9.6 MiB/s, 154 objects/s recovering 2024-02-09T17:33:47.942 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:47 smithi179 ceph-mon[43975]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:33:47.943 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:47 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3086525561' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:33:47.943 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:47 smithi179 ceph-mon[43975]: osdmap e278: 12 total, 12 up, 11 in 2024-02-09T17:33:47.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:47.943 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:33:47.944 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:33:47.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:47 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:33:47.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:47 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:33:47.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:47 smithi099 ceph-mon[39420]: pgmap v788: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 3.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 620 op/s; 4665/181338 objects degraded (2.573%); 14893/181338 objects misplaced (8.213%); 9.6 MiB/s, 154 objects/s recovering 2024-02-09T17:33:47.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:47 smithi099 ceph-mon[39420]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:33:47.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:47 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3086525561' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:33:47.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:47 smithi099 ceph-mon[39420]: osdmap e278: 12 total, 12 up, 11 in 2024-02-09T17:33:48.045 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:33:48.093 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:48.118 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:33:48.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:48.123 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:33:48.224 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:33:48.299 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:48.359 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:33:48.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:48.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:33:48.466 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:33:48.497 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:49.045 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:33:49.045 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [], 2024-02-09T17:33:49.045 INFO:teuthology.orchestra.run.smithi099.stdout: "complaint_time": 30, 2024-02-09T17:33:49.045 INFO:teuthology.orchestra.run.smithi099.stdout: "num_blocked_ops": 0 2024-02-09T17:33:49.045 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:33:49.735 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-09T17:33:49.749 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:33:49.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:49.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:33:49.765 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:33:49.858 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:33:49.919 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:49.968 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:33:49.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:49.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:33:49.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:49 smithi099 ceph-mon[39420]: pgmap v790: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 3.6 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 537 op/s; 4665/177933 objects degraded (2.622%); 14893/177933 objects misplaced (8.370%); 5.2 MiB/s, 83 objects/s recovering 2024-02-09T17:33:50.076 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:33:50.123 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:50.164 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:33:50.170 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:49 smithi162 ceph-mon[44304]: pgmap v790: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 3.6 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 537 op/s; 4665/177933 objects degraded (2.622%); 14893/177933 objects misplaced (8.370%); 5.2 MiB/s, 83 objects/s recovering 2024-02-09T17:33:50.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:50.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:33:50.176 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:49 smithi179 ceph-mon[43975]: pgmap v790: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 3.6 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 537 op/s; 4665/177933 objects degraded (2.622%); 14893/177933 objects misplaced (8.370%); 5.2 MiB/s, 83 objects/s recovering 2024-02-09T17:33:50.271 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:33:50.332 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:50.383 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:33:50.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:50.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:33:50.492 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:33:50.526 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:50.573 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:33:50.576 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:50.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:33:50.577 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:33:50.616 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:33:50.678 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:33:50.707 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:50.795 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset noscrub 2024-02-09T17:33:50.836 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:33:50.839 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:50.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:33:50.940 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:33:50.998 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:51.023 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:33:51.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:51.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:33:51.128 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:33:51.193 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:51.246 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:33:51.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:51.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:33:51.351 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:33:51.409 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:51.443 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:33:51.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:51.448 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:33:51.549 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:33:51.600 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:51.641 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:33:51.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:51.648 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:33:51.749 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:33:51.784 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:51.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:51 smithi099 ceph-mon[39420]: pgmap v791: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 3.5 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 630 op/s; 4770/172866 objects degraded (2.759%); 14129/172866 objects misplaced (8.173%); 8.7 MiB/s, 140 objects/s recovering 2024-02-09T17:33:51.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:51 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 4770/172866 objects degraded (2.759%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:33:51.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:51 smithi099 ceph-mon[39420]: osdmap e279: 12 total, 12 up, 11 in 2024-02-09T17:33:52.045 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:33:52.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:52.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:33:52.149 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:33:52.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:51 smithi179 ceph-mon[43975]: pgmap v791: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 3.5 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 630 op/s; 4770/172866 objects degraded (2.759%); 14129/172866 objects misplaced (8.173%); 8.7 MiB/s, 140 objects/s recovering 2024-02-09T17:33:52.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:51 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 4770/172866 objects degraded (2.759%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:33:52.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:51 smithi179 ceph-mon[43975]: osdmap e279: 12 total, 12 up, 11 in 2024-02-09T17:33:52.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:51 smithi162 ceph-mon[44304]: pgmap v791: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 3.5 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 630 op/s; 4770/172866 objects degraded (2.759%); 14129/172866 objects misplaced (8.173%); 8.7 MiB/s, 140 objects/s recovering 2024-02-09T17:33:52.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:51 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 4770/172866 objects degraded (2.759%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:33:52.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:51 smithi162 ceph-mon[44304]: osdmap e279: 12 total, 12 up, 11 in 2024-02-09T17:33:52.181 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:52.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:52 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:33:52.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:52 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:33:52.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:52 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:33:52.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:52 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:33:53.132 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:33:53.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:53.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:33:53.158 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:33:53.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:52 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:33:53.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:52 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:33:53.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:52 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:33:53.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:52 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:33:53.236 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:33:53.273 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:52 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:33:53.273 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:52 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:33:53.273 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:52 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:33:53.273 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:52 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:33:53.281 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:53.314 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:33:53.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:53.322 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:33:53.423 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:33:53.478 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:53.513 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:33:53.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:53.519 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:33:53.620 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:33:53.658 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:53.689 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:33:53.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:53.692 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:33:53.793 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:33:53.827 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:53.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:53.871 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:33:53.871 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:33:53.972 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:33:54.018 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:54.058 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:33:54.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:54.059 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:33:54.161 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:33:54.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:53 smithi162 ceph-mon[44304]: pgmap v793: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 3.4 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 716 op/s; 4770/168732 objects degraded (2.827%); 14129/168732 objects misplaced (8.374%); 10 MiB/s, 163 objects/s recovering 2024-02-09T17:33:54.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:53 smithi162 ceph-mon[44304]: osdmap e280: 12 total, 12 up, 11 in 2024-02-09T17:33:54.190 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:53 smithi099 ceph-mon[39420]: pgmap v793: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 3.4 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 716 op/s; 4770/168732 objects degraded (2.827%); 14129/168732 objects misplaced (8.374%); 10 MiB/s, 163 objects/s recovering 2024-02-09T17:33:54.190 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:53 smithi099 ceph-mon[39420]: osdmap e280: 12 total, 12 up, 11 in 2024-02-09T17:33:54.193 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:54.345 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:33:54.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:54.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:33:54.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:53 smithi179 ceph-mon[43975]: pgmap v793: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 3.4 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 716 op/s; 4770/168732 objects degraded (2.827%); 14129/168732 objects misplaced (8.374%); 10 MiB/s, 163 objects/s recovering 2024-02-09T17:33:54.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:53 smithi179 ceph-mon[43975]: osdmap e280: 12 total, 12 up, 11 in 2024-02-09T17:33:54.450 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:33:54.517 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:54.555 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:33:54.555 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:33:54.555 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:33:54.556 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:33:54.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:54.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:33:54.566 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:33:54.598 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:33:54.631 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:33:54.662 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:33:54.662 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:33:54.688 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:33:54.704 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:54.711 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:33:54.732 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:33:54.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:54.737 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:33:54.740 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:33:54.766 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:33:54.795 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:33:54.797 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:33:54.797 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:33:54.797 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:33:54.797 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:33:54.797 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:33:54.797 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:33:54.797 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:33:54.797 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:33:54.798 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:33:54.798 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:33:54.798 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:33:54.798 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:33:54.837 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:33:54.906 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:54.943 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:33:54.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:54.951 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:33:54.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:54 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3121629415' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:33:55.052 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:33:55.131 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:55.167 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:33:55.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:55.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:33:55.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:54 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3121629415' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:33:55.272 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:33:55.335 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:54 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3121629415' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:33:55.344 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:55.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:55.381 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:33:55.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:33:55.482 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:33:55.555 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:55.603 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:33:55.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:55.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:33:55.710 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:33:55.755 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:55.788 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:33:55.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:55.795 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:33:55.896 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:33:55.928 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:56.149 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:55 smithi099 ceph-mon[39420]: pgmap v795: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 3.4 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 503 op/s; 4770/167502 objects degraded (2.848%); 14129/167502 objects misplaced (8.435%); 5.3 MiB/s, 85 objects/s recovering 2024-02-09T17:33:56.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:55 smithi179 ceph-mon[43975]: pgmap v795: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 3.4 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 503 op/s; 4770/167502 objects degraded (2.848%); 14129/167502 objects misplaced (8.435%); 5.3 MiB/s, 85 objects/s recovering 2024-02-09T17:33:56.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:55 smithi162 ceph-mon[44304]: pgmap v795: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 3.4 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 503 op/s; 4770/167502 objects degraded (2.848%); 14129/167502 objects misplaced (8.435%); 5.3 MiB/s, 85 objects/s recovering 2024-02-09T17:33:56.945 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:56 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mon-a[39397]: 2024-02-09T17:33:56.773+0000 7feb57590700 -1 mon.a@0(leader).osd e281 definitely_dead 0 2024-02-09T17:33:57.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:56 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 4770/167502 objects degraded (2.848%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:33:57.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:56 smithi099 ceph-mon[39420]: osdmap e281: 12 total, 12 up, 11 in 2024-02-09T17:33:57.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:56 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1160116576' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-02-09T17:33:57.298 INFO:teuthology.orchestra.run.smithi099.stderr:marked in osd.0. 2024-02-09T17:33:57.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:56 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 4770/167502 objects degraded (2.848%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:33:57.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:56 smithi179 ceph-mon[43975]: osdmap e281: 12 total, 12 up, 11 in 2024-02-09T17:33:57.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:56 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1160116576' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-02-09T17:33:57.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:56 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 4770/167502 objects degraded (2.848%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:33:57.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:56 smithi162 ceph-mon[44304]: osdmap e281: 12 total, 12 up, 11 in 2024-02-09T17:33:57.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:56 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1160116576' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-02-09T17:33:57.868 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:33:57.869 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:33:57.869 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:33:58.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:57 smithi099 ceph-mon[39420]: pgmap v797: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 3.3 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 551 op/s; 4921/162933 objects degraded (3.020%); 13316/162933 objects misplaced (8.173%); 6.8 MiB/s, 110 objects/s recovering 2024-02-09T17:33:58.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:57 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1160116576' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-02-09T17:33:58.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:57 smithi099 ceph-mon[39420]: osdmap e282: 12 total, 12 up, 12 in 2024-02-09T17:33:58.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:57 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/646058846' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:33:58.287 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:33:58.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:58.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:33:58.299 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is unset 2024-02-09T17:33:58.392 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:33:58.417 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:57 smithi162 ceph-mon[44304]: pgmap v797: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 3.3 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 551 op/s; 4921/162933 objects degraded (3.020%); 13316/162933 objects misplaced (8.173%); 6.8 MiB/s, 110 objects/s recovering 2024-02-09T17:33:58.418 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:57 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1160116576' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-02-09T17:33:58.418 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:57 smithi162 ceph-mon[44304]: osdmap e282: 12 total, 12 up, 12 in 2024-02-09T17:33:58.418 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:57 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/646058846' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:33:58.419 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:58.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:57 smithi179 ceph-mon[43975]: pgmap v797: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 3.3 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 551 op/s; 4921/162933 objects degraded (3.020%); 13316/162933 objects misplaced (8.173%); 6.8 MiB/s, 110 objects/s recovering 2024-02-09T17:33:58.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:57 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1160116576' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-02-09T17:33:58.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:57 smithi179 ceph-mon[43975]: osdmap e282: 12 total, 12 up, 12 in 2024-02-09T17:33:58.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:57 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/646058846' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:33:58.442 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:33:58.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:33:58.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:33:58.546 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:33:58.584 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:33:59.004 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-09T17:33:59.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:58 smithi099 ceph-mon[39420]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:33:59.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:58 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/646058846' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:33:59.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:58 smithi099 ceph-mon[39420]: osdmap e283: 12 total, 12 up, 12 in 2024-02-09T17:33:59.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:58 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:33:59.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:58 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:33:59.288 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:33:59.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:58 smithi179 ceph-mon[43975]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:33:59.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:58 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/646058846' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:33:59.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:58 smithi179 ceph-mon[43975]: osdmap e283: 12 total, 12 up, 12 in 2024-02-09T17:33:59.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:58 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:33:59.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:58 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:33:59.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:58 smithi162 ceph-mon[44304]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:33:59.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:58 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/646058846' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:33:59.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:58 smithi162 ceph-mon[44304]: osdmap e283: 12 total, 12 up, 12 in 2024-02-09T17:33:59.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:58 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:33:59.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:58 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:33:59.624 INFO:tasks.thrashosds.thrasher:Added osd 0 2024-02-09T17:33:59.951 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:59 smithi099 ceph-mon[39420]: pgmap v800: 37 pgs: 1 peering, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 553 op/s; 4921/159357 objects degraded (3.088%); 13316/159357 objects misplaced (8.356%); 7.3 MiB/s, 117 objects/s recovering 2024-02-09T17:33:59.951 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:59 smithi099 ceph-mon[39420]: 3.b starting backfill to osd.0 from (212'3600,254'13619] MIN to 281'16444 2024-02-09T17:33:59.951 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:59 smithi099 ceph-mon[39420]: osdmap e284: 12 total, 12 up, 12 in 2024-02-09T17:33:59.951 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:59 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:33:59.951 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:33:59 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:34:00.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:59 smithi162 ceph-mon[44304]: pgmap v800: 37 pgs: 1 peering, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 553 op/s; 4921/159357 objects degraded (3.088%); 13316/159357 objects misplaced (8.356%); 7.3 MiB/s, 117 objects/s recovering 2024-02-09T17:34:00.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:59 smithi162 ceph-mon[44304]: 3.b starting backfill to osd.0 from (212'3600,254'13619] MIN to 281'16444 2024-02-09T17:34:00.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:59 smithi162 ceph-mon[44304]: osdmap e284: 12 total, 12 up, 12 in 2024-02-09T17:34:00.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:59 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:34:00.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:33:59 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:34:00.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:59 smithi179 ceph-mon[43975]: pgmap v800: 37 pgs: 1 peering, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 553 op/s; 4921/159357 objects degraded (3.088%); 13316/159357 objects misplaced (8.356%); 7.3 MiB/s, 117 objects/s recovering 2024-02-09T17:34:00.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:59 smithi179 ceph-mon[43975]: 3.b starting backfill to osd.0 from (212'3600,254'13619] MIN to 281'16444 2024-02-09T17:34:00.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:59 smithi179 ceph-mon[43975]: osdmap e284: 12 total, 12 up, 12 in 2024-02-09T17:34:00.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:59 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:34:00.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:33:59 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:34:00.873 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:34:00.948 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:34:00.957 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:00.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:34:01.059 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:34:01.127 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:00 smithi162 ceph-mon[44304]: 1.7 scrub starts 2024-02-09T17:34:01.128 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:00 smithi162 ceph-mon[44304]: 1.7 scrub ok 2024-02-09T17:34:01.128 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:00 smithi162 ceph-mon[44304]: osdmap e285: 12 total, 12 up, 12 in 2024-02-09T17:34:01.135 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:01.196 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:00 smithi099 ceph-mon[39420]: 1.7 scrub starts 2024-02-09T17:34:01.196 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:00 smithi099 ceph-mon[39420]: 1.7 scrub ok 2024-02-09T17:34:01.196 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:00 smithi099 ceph-mon[39420]: osdmap e285: 12 total, 12 up, 12 in 2024-02-09T17:34:01.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:00 smithi179 ceph-mon[43975]: 1.7 scrub starts 2024-02-09T17:34:01.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:00 smithi179 ceph-mon[43975]: 1.7 scrub ok 2024-02-09T17:34:01.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:00 smithi179 ceph-mon[43975]: osdmap e285: 12 total, 12 up, 12 in 2024-02-09T17:34:02.108 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:34:02.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:02.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:34:02.220 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:34:02.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:01 smithi099 ceph-mon[39420]: pgmap v803: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+undersized+remapped, 1 peering, 31 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 610 op/s; 4436/155610 objects degraded (2.851%); 4.4 MiB/s, 133 objects/s recovering 2024-02-09T17:34:02.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:01 smithi099 ceph-mon[39420]: 1.f scrub starts 2024-02-09T17:34:02.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:01 smithi099 ceph-mon[39420]: 1.a scrub starts 2024-02-09T17:34:02.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:01 smithi099 ceph-mon[39420]: 1.f scrub ok 2024-02-09T17:34:02.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:01 smithi099 ceph-mon[39420]: 1.a scrub ok 2024-02-09T17:34:02.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:01 smithi099 ceph-mon[39420]: 3.d scrub starts 2024-02-09T17:34:02.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:01 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 4921/159357 objects degraded (3.088%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:34:02.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:01 smithi099 ceph-mon[39420]: osdmap e286: 12 total, 12 up, 12 in 2024-02-09T17:34:02.256 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:01 smithi162 ceph-mon[44304]: pgmap v803: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+undersized+remapped, 1 peering, 31 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 610 op/s; 4436/155610 objects degraded (2.851%); 4.4 MiB/s, 133 objects/s recovering 2024-02-09T17:34:02.256 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:01 smithi162 ceph-mon[44304]: 1.f scrub starts 2024-02-09T17:34:02.256 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:01 smithi162 ceph-mon[44304]: 1.a scrub starts 2024-02-09T17:34:02.256 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:01 smithi162 ceph-mon[44304]: 1.f scrub ok 2024-02-09T17:34:02.257 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:01 smithi162 ceph-mon[44304]: 1.a scrub ok 2024-02-09T17:34:02.257 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:01 smithi162 ceph-mon[44304]: 3.d scrub starts 2024-02-09T17:34:02.257 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:01 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 4921/159357 objects degraded (3.088%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:34:02.257 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:01 smithi162 ceph-mon[44304]: osdmap e286: 12 total, 12 up, 12 in 2024-02-09T17:34:02.266 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:02.369 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:34:02.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:02.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:34:02.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:01 smithi179 ceph-mon[43975]: pgmap v803: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+undersized+remapped, 1 peering, 31 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 610 op/s; 4436/155610 objects degraded (2.851%); 4.4 MiB/s, 133 objects/s recovering 2024-02-09T17:34:02.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:01 smithi179 ceph-mon[43975]: 1.f scrub starts 2024-02-09T17:34:02.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:01 smithi179 ceph-mon[43975]: 1.a scrub starts 2024-02-09T17:34:02.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:01 smithi179 ceph-mon[43975]: 1.f scrub ok 2024-02-09T17:34:02.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:01 smithi179 ceph-mon[43975]: 1.a scrub ok 2024-02-09T17:34:02.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:01 smithi179 ceph-mon[43975]: 3.d scrub starts 2024-02-09T17:34:02.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:01 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 4921/159357 objects degraded (3.088%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:34:02.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:01 smithi179 ceph-mon[43975]: osdmap e286: 12 total, 12 up, 12 in 2024-02-09T17:34:02.481 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:34:02.561 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:02.609 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:34:02.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:02.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:34:02.629 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:34:02.630 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:34:02.712 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:34:02.743 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:02.770 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:34:02.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:02.773 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:34:02.874 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:34:02.928 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-09T17:34:02.946 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:03.003 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:03 smithi179 ceph-mon[43975]: 1.d scrub starts 2024-02-09T17:34:03.003 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:03 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:34:03.003 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:03 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:34:03.003 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:03 smithi179 ceph-mon[43975]: 3.d scrub ok 2024-02-09T17:34:03.003 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:03 smithi179 ceph-mon[43975]: 1.d scrub ok 2024-02-09T17:34:03.003 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:03 smithi179 ceph-mon[43975]: pgmap v805: 37 pgs: 1 active+remapped+backfill_toofull, 27 active+clean, 1 peering, 3 active+recovering+undersized+remapped, 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+clean+scrubbing; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 511 op/s; 4437/152721 objects degraded (2.905%); 3158/152721 objects misplaced (2.068%); 4.0 MiB/s, 124 objects/s recovering 2024-02-09T17:34:03.032 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:34:03.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:03.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:34:03.140 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:34:03.146 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:34:03.180 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:03.199 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:34:03.350 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:03 smithi162 ceph-mon[44304]: 1.d scrub starts 2024-02-09T17:34:03.350 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:03 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:34:03.350 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:03 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:34:03.350 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:03 smithi162 ceph-mon[44304]: 3.d scrub ok 2024-02-09T17:34:03.350 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:03 smithi162 ceph-mon[44304]: 1.d scrub ok 2024-02-09T17:34:03.350 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:03 smithi162 ceph-mon[44304]: pgmap v805: 37 pgs: 1 active+remapped+backfill_toofull, 27 active+clean, 1 peering, 3 active+recovering+undersized+remapped, 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+clean+scrubbing; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 511 op/s; 4437/152721 objects degraded (2.905%); 3158/152721 objects misplaced (2.068%); 4.0 MiB/s, 124 objects/s recovering 2024-02-09T17:34:03.357 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:34:03.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:03.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:34:03.393 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-09T17:34:03.439 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:34:03.448 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:34:03.472 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:34:03.485 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:34:03.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:03 smithi099 ceph-mon[39420]: 1.d scrub starts 2024-02-09T17:34:03.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:03 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:34:03.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:03 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:34:03.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:03 smithi099 ceph-mon[39420]: 3.d scrub ok 2024-02-09T17:34:03.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:03 smithi099 ceph-mon[39420]: 1.d scrub ok 2024-02-09T17:34:03.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:03 smithi099 ceph-mon[39420]: pgmap v805: 37 pgs: 1 active+remapped+backfill_toofull, 27 active+clean, 1 peering, 3 active+recovering+undersized+remapped, 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+clean+scrubbing; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 511 op/s; 4437/152721 objects degraded (2.905%); 3158/152721 objects misplaced (2.068%); 4.0 MiB/s, 124 objects/s recovering 2024-02-09T17:34:03.501 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:03.523 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:34:03.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:03.524 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:34:03.525 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:34:03.565 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:34:03.611 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:34:03.625 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:34:03.649 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:34:03.656 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:03.680 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:34:03.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:03.686 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:34:03.696 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:34:03.724 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:34:03.759 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:34:03.787 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:34:03.801 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:34:03.821 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:34:03.821 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:34:03.821 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:34:03.821 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:34:03.821 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:34:03.821 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:34:03.822 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:34:03.822 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:34:03.822 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:34:03.822 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:34:03.822 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:34:03.822 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:34:03.845 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:03.878 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:34:03.880 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:34:03.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:03.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:34:03.984 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:34:04.027 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:04.055 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:34:04.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:04.059 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:34:04.160 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:34:04.195 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:04.232 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:34:04.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:04.240 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:34:04.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:04 smithi099 ceph-mon[39420]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:34:04.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:04 smithi099 ceph-mon[39420]: osdmap e287: 12 total, 12 up, 12 in 2024-02-09T17:34:04.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:04 smithi099 ceph-mon[39420]: 1.0 scrub starts 2024-02-09T17:34:04.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:04 smithi099 ceph-mon[39420]: 1.0 scrub ok 2024-02-09T17:34:04.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:04 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1098050726' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:34:04.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:04 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:34:04.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:04 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:34:04.270 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:04 smithi162 ceph-mon[44304]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:34:04.270 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:04 smithi162 ceph-mon[44304]: osdmap e287: 12 total, 12 up, 12 in 2024-02-09T17:34:04.270 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:04 smithi162 ceph-mon[44304]: 1.0 scrub starts 2024-02-09T17:34:04.270 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:04 smithi162 ceph-mon[44304]: 1.0 scrub ok 2024-02-09T17:34:04.270 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:04 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1098050726' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:34:04.270 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:04 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:34:04.270 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:04 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:34:04.341 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:34:04.397 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:04.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:04 smithi179 ceph-mon[43975]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:34:04.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:04 smithi179 ceph-mon[43975]: osdmap e287: 12 total, 12 up, 12 in 2024-02-09T17:34:04.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:04 smithi179 ceph-mon[43975]: 1.0 scrub starts 2024-02-09T17:34:04.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:04 smithi179 ceph-mon[43975]: 1.0 scrub ok 2024-02-09T17:34:04.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:04 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1098050726' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:34:04.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:04 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:34:04.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:04 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:34:04.456 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:34:04.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:04.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:34:04.563 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:34:04.625 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] 2024-02-09T17:34:04.625 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:34:04.625 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-02-09T17:34:04.625 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-09T17:34:04.625 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-09T17:34:04.625 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-02-09T17:34:04.628 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:04.731 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:34:04.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:04.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:34:04.851 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:34:04.885 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:04.907 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:34:04.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:04.913 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:34:05.014 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:34:05.109 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:05.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:05 smithi179 ceph-mon[43975]: osdmap e288: 12 total, 12 up, 12 in 2024-02-09T17:34:05.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:05 smithi179 ceph-mon[43975]: 1.9 scrub starts 2024-02-09T17:34:05.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:05 smithi179 ceph-mon[43975]: 1.9 scrub ok 2024-02-09T17:34:05.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:05 smithi179 ceph-mon[43975]: pgmap v808: 37 pgs: 1 active+remapped+backfill_toofull, 28 active+clean, 3 active+recovering+undersized+remapped, 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+clean+scrubbing; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 377 op/s; 4437/151476 objects degraded (2.929%); 3158/151476 objects misplaced (2.085%); 1.5 MiB/s, 53 objects/s recovering 2024-02-09T17:34:05.332 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:34:05.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:05.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:34:05.339 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:05 smithi162 ceph-mon[44304]: osdmap e288: 12 total, 12 up, 12 in 2024-02-09T17:34:05.339 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:05 smithi162 ceph-mon[44304]: 1.9 scrub starts 2024-02-09T17:34:05.339 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:05 smithi162 ceph-mon[44304]: 1.9 scrub ok 2024-02-09T17:34:05.340 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:05 smithi162 ceph-mon[44304]: pgmap v808: 37 pgs: 1 active+remapped+backfill_toofull, 28 active+clean, 3 active+recovering+undersized+remapped, 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+clean+scrubbing; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 377 op/s; 4437/151476 objects degraded (2.929%); 3158/151476 objects misplaced (2.085%); 1.5 MiB/s, 53 objects/s recovering 2024-02-09T17:34:05.348 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:05 smithi099 ceph-mon[39420]: osdmap e288: 12 total, 12 up, 12 in 2024-02-09T17:34:05.348 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:05 smithi099 ceph-mon[39420]: 1.9 scrub starts 2024-02-09T17:34:05.348 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:05 smithi099 ceph-mon[39420]: 1.9 scrub ok 2024-02-09T17:34:05.348 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:05 smithi099 ceph-mon[39420]: pgmap v808: 37 pgs: 1 active+remapped+backfill_toofull, 28 active+clean, 3 active+recovering+undersized+remapped, 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+clean+scrubbing; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 377 op/s; 4437/151476 objects degraded (2.929%); 3158/151476 objects misplaced (2.085%); 1.5 MiB/s, 53 objects/s recovering 2024-02-09T17:34:05.440 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:34:05.472 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:34:05.477 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:05.737 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:34:05.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:05.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:34:05.842 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:34:05.879 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:05.935 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:34:05.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:05.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:34:06.045 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:34:06.090 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:06.293 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:34:06.294 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-09T17:34:06.294 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-02-09T17:34:06.294 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-02-09T17:34:06.294 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:34:06.313 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:06 smithi162 ceph-mon[44304]: 3.a scrub starts 2024-02-09T17:34:06.313 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:06 smithi162 ceph-mon[44304]: 1.2 scrub starts 2024-02-09T17:34:06.313 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:06 smithi162 ceph-mon[44304]: 1.2 scrub ok 2024-02-09T17:34:06.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:06 smithi179 ceph-mon[43975]: 3.a scrub starts 2024-02-09T17:34:06.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:06 smithi179 ceph-mon[43975]: 1.2 scrub starts 2024-02-09T17:34:06.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:06 smithi179 ceph-mon[43975]: 1.2 scrub ok 2024-02-09T17:34:06.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:06 smithi099 ceph-mon[39420]: 3.a scrub starts 2024-02-09T17:34:06.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:06 smithi099 ceph-mon[39420]: 1.2 scrub starts 2024-02-09T17:34:06.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:06 smithi099 ceph-mon[39420]: 1.2 scrub ok 2024-02-09T17:34:07.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:07 smithi162 ceph-mon[44304]: 3.a scrub ok 2024-02-09T17:34:07.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:07 smithi162 ceph-mon[44304]: 3.f scrub starts 2024-02-09T17:34:07.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:07 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 4437/151476 objects degraded (2.929%), 2 pgs degraded (PG_DEGRADED) 2024-02-09T17:34:07.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:07 smithi162 ceph-mon[44304]: osdmap e289: 12 total, 12 up, 12 in 2024-02-09T17:34:07.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:07 smithi162 ceph-mon[44304]: pgmap v810: 37 pgs: 1 active+remapped+backfill_toofull, 29 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+clean+scrubbing; 3.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 579 op/s; 4437/146100 objects degraded (3.037%); 3158/146100 objects misplaced (2.162%); 0 B/s, 91 objects/s recovering 2024-02-09T17:34:07.196 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:34:07.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:07 smithi179 ceph-mon[43975]: 3.a scrub ok 2024-02-09T17:34:07.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:07 smithi179 ceph-mon[43975]: 3.f scrub starts 2024-02-09T17:34:07.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:07 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 4437/151476 objects degraded (2.929%), 2 pgs degraded (PG_DEGRADED) 2024-02-09T17:34:07.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:07 smithi179 ceph-mon[43975]: osdmap e289: 12 total, 12 up, 12 in 2024-02-09T17:34:07.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:07 smithi179 ceph-mon[43975]: pgmap v810: 37 pgs: 1 active+remapped+backfill_toofull, 29 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+clean+scrubbing; 3.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 579 op/s; 4437/146100 objects degraded (3.037%); 3158/146100 objects misplaced (2.162%); 0 B/s, 91 objects/s recovering 2024-02-09T17:34:07.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:07 smithi099 ceph-mon[39420]: 3.a scrub ok 2024-02-09T17:34:07.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:07 smithi099 ceph-mon[39420]: 3.f scrub starts 2024-02-09T17:34:07.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:07 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 4437/151476 objects degraded (2.929%), 2 pgs degraded (PG_DEGRADED) 2024-02-09T17:34:07.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:07 smithi099 ceph-mon[39420]: osdmap e289: 12 total, 12 up, 12 in 2024-02-09T17:34:07.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:07 smithi099 ceph-mon[39420]: pgmap v810: 37 pgs: 1 active+remapped+backfill_toofull, 29 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+clean+scrubbing; 3.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 579 op/s; 4437/146100 objects degraded (3.037%); 3158/146100 objects misplaced (2.162%); 0 B/s, 91 objects/s recovering 2024-02-09T17:34:07.961 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:34:07.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:07.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:34:08.066 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:34:08.102 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:08.109 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-09T17:34:08.206 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:34:08.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:08.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:34:08.314 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:34:08.332 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:08 smithi162 ceph-mon[44304]: 3.f scrub ok 2024-02-09T17:34:08.378 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:08.399 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:34:08.423 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:34:08.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:08.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:34:08.455 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:34:08.455 INFO:teuthology.orchestra.run.smithi162.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-02-09T17:34:08.455 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:34:08.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:08 smithi099 ceph-mon[39420]: 3.f scrub ok 2024-02-09T17:34:08.528 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:34:08.561 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:08.589 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:34:08.591 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:08 smithi179 ceph-mon[43975]: 3.f scrub ok 2024-02-09T17:34:08.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:08.592 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:34:08.693 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:34:08.725 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:34:08.732 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:08.761 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:34:08.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:08.764 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:34:08.865 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:34:08.923 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:08.999 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:34:09.004 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:09.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:34:09.106 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:34:09.142 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:09.166 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:34:09.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:09.170 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:34:09.271 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:34:09.324 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:09 smithi099 ceph-mon[39420]: pgmap v811: 37 pgs: 1 active+remapped+backfill_toofull, 29 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+clean+scrubbing; 2.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 586 op/s; 4437/142014 objects degraded (3.124%); 2923/142014 objects misplaced (2.058%); 0 B/s, 79 objects/s recovering 2024-02-09T17:34:09.324 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:09 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3256510491' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:34:09.324 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:09 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:34:09.335 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:09.346 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is unset 2024-02-09T17:34:09.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:09 smithi179 ceph-mon[43975]: pgmap v811: 37 pgs: 1 active+remapped+backfill_toofull, 29 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+clean+scrubbing; 2.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 586 op/s; 4437/142014 objects degraded (3.124%); 2923/142014 objects misplaced (2.058%); 0 B/s, 79 objects/s recovering 2024-02-09T17:34:09.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:09 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3256510491' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:34:09.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:09 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:34:09.501 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:34:09.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:09.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:34:09.511 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:09 smithi162 ceph-mon[44304]: pgmap v811: 37 pgs: 1 active+remapped+backfill_toofull, 29 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+clean+scrubbing; 2.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 586 op/s; 4437/142014 objects degraded (3.124%); 2923/142014 objects misplaced (2.058%); 0 B/s, 79 objects/s recovering 2024-02-09T17:34:09.511 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:09 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3256510491' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:34:09.511 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:09 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:34:09.609 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:34:09.664 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:09.753 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:34:09.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:09.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:34:09.868 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:34:09.936 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:10.100 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:34:10.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:10.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:34:10.202 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:34:10.230 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:10.255 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:34:10.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:10.261 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:34:10.361 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:34:10.413 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:10.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:10 smithi162 ceph-mon[44304]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:34:10.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:10 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:34:10.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:10 smithi162 ceph-mon[44304]: osdmap e290: 12 total, 12 up, 12 in 2024-02-09T17:34:10.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:10 smithi099 ceph-mon[39420]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:34:10.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:10 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:34:10.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:10 smithi099 ceph-mon[39420]: osdmap e290: 12 total, 12 up, 12 in 2024-02-09T17:34:10.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:10 smithi179 ceph-mon[43975]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:34:10.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:10 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:34:10.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:10 smithi179 ceph-mon[43975]: osdmap e290: 12 total, 12 up, 12 in 2024-02-09T17:34:10.779 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:34:10.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:10.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:34:10.884 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:34:10.933 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:10.993 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:34:11.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:11.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:34:11.104 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:34:11.171 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:11.279 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:34:11.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:11.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:34:11.380 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:34:11.380 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-02-09T17:34:11.380 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-02-09T17:34:11.380 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-02-09T17:34:11.380 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.380 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:141301 3.3 3:c21b7a5c:::benchmark_data_smithi179_90440_object9342:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-02-09T17:34:11.380 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:31:59.267938+0000", 2024-02-09T17:34:11.381 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 132.10774302199999, 2024-02-09T17:34:11.381 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3659083160000001, 2024-02-09T17:34:11.381 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:34:11.381 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:34:11.381 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:34:11.381 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:34:11.381 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:34:11.381 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 141301 2024-02-09T17:34:11.381 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.381 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:34:11.381 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.382 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:34:11.382 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.267938+0000", 2024-02-09T17:34:11.382 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:11.382 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.382 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.382 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:34:11.382 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.267938+0000", 2024-02-09T17:34:11.382 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:11.382 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.382 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.382 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:34:11.382 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.267940+0000", 2024-02-09T17:34:11.382 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6160000000000001e-06 2024-02-09T17:34:11.383 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.383 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.383 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:34:11.383 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.267945+0000", 2024-02-09T17:34:11.383 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6319999999999996e-06 2024-02-09T17:34:11.383 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.383 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.383 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:34:11.383 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.267959+0000", 2024-02-09T17:34:11.383 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3616e-05 2024-02-09T17:34:11.383 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.383 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.383 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:34:11.384 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.267966+0000", 2024-02-09T17:34:11.384 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.002e-06 2024-02-09T17:34:11.384 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.384 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.384 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:11.384 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.279726+0000", 2024-02-09T17:34:11.384 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011759591999999999 2024-02-09T17:34:11.384 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.384 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.384 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:34:11.384 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.279735+0000", 2024-02-09T17:34:11.384 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8050000000000001e-06 2024-02-09T17:34:11.385 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.385 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.385 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:11.385 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:00.271890+0000", 2024-02-09T17:34:11.385 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99215431499999995 2024-02-09T17:34:11.385 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.385 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.385 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:34:11.385 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:00.271903+0000", 2024-02-09T17:34:11.385 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3284e-05 2024-02-09T17:34:11.385 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.385 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.386 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:11.386 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.608947+0000", 2024-02-09T17:34:11.386 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.337043838 2024-02-09T17:34:11.386 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.386 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.386 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:34:11.386 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.609112+0000", 2024-02-09T17:34:11.386 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016556599999999999 2024-02-09T17:34:11.386 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.386 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.386 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-02-09T17:34:11.386 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.609285+0000", 2024-02-09T17:34:11.386 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017263599999999999 2024-02-09T17:34:11.387 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.387 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.387 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:11.387 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.619077+0000", 2024-02-09T17:34:11.387 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0097920879999999991 2024-02-09T17:34:11.387 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.387 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.387 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:34:11.387 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.619097+0000", 2024-02-09T17:34:11.387 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9426999999999999e-05 2024-02-09T17:34:11.387 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.387 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.387 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:11.387 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.627918+0000", 2024-02-09T17:34:11.388 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0088214560000000001 2024-02-09T17:34:11.388 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.388 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.388 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:34:11.388 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.627979+0000", 2024-02-09T17:34:11.388 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1205999999999996e-05 2024-02-09T17:34:11.388 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.388 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.388 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:34:11.388 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.633847+0000", 2024-02-09T17:34:11.388 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.005867237 2024-02-09T17:34:11.388 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:11.388 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:34:11.389 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:11.389 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.389 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.389 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:141303 3.3 3:c41f7f3f:::benchmark_data_smithi179_90440_object9344:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-02-09T17:34:11.389 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:31:59.274155+0000", 2024-02-09T17:34:11.389 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 132.10152650399999, 2024-02-09T17:34:11.389 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.411265985, 2024-02-09T17:34:11.389 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:34:11.389 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:34:11.389 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:34:11.389 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:34:11.389 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:34:11.389 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 141303 2024-02-09T17:34:11.390 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.390 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:34:11.390 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.390 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:34:11.390 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.274155+0000", 2024-02-09T17:34:11.390 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:11.390 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.390 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.390 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:34:11.390 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.274155+0000", 2024-02-09T17:34:11.390 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:11.390 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.390 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.390 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:34:11.391 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.274156+0000", 2024-02-09T17:34:11.391 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3659999999999999e-06 2024-02-09T17:34:11.391 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.391 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.391 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:34:11.391 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.274161+0000", 2024-02-09T17:34:11.391 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5469999999999998e-06 2024-02-09T17:34:11.391 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.391 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.391 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:34:11.391 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.274172+0000", 2024-02-09T17:34:11.391 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1411e-05 2024-02-09T17:34:11.391 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.391 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.392 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:34:11.392 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.274181+0000", 2024-02-09T17:34:11.392 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7660000000000004e-06 2024-02-09T17:34:11.392 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.392 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.392 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:11.392 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.280072+0000", 2024-02-09T17:34:11.392 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.005891444 2024-02-09T17:34:11.392 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.392 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.392 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:34:11.392 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.280079+0000", 2024-02-09T17:34:11.392 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6359999999999997e-06 2024-02-09T17:34:11.392 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.393 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.393 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:11.393 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:00.271993+0000", 2024-02-09T17:34:11.393 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99191438700000001 2024-02-09T17:34:11.393 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.393 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.393 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:34:11.393 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:00.272003+0000", 2024-02-09T17:34:11.393 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5089999999999999e-06 2024-02-09T17:34:11.393 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.393 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.393 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:11.393 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.633918+0000", 2024-02-09T17:34:11.393 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.361915142 2024-02-09T17:34:11.393 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.394 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.394 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:34:11.394 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.634089+0000", 2024-02-09T17:34:11.394 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000170552 2024-02-09T17:34:11.394 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.394 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.394 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-02-09T17:34:11.394 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.634254+0000", 2024-02-09T17:34:11.394 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016508699999999999 2024-02-09T17:34:11.394 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.394 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.394 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:11.394 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.647777+0000", 2024-02-09T17:34:11.394 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013523795 2024-02-09T17:34:11.395 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.395 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.395 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:34:11.395 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.647858+0000", 2024-02-09T17:34:11.395 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0488999999999995e-05 2024-02-09T17:34:11.395 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.395 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.395 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:11.395 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.680537+0000", 2024-02-09T17:34:11.395 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.032679407000000001 2024-02-09T17:34:11.395 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.395 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.395 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:34:11.395 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.680600+0000", 2024-02-09T17:34:11.396 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3167e-05 2024-02-09T17:34:11.396 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.396 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.396 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:34:11.396 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.685421+0000", 2024-02-09T17:34:11.396 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0048202799999999997 2024-02-09T17:34:11.396 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:11.396 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:34:11.396 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:11.396 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.396 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.396 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:141326 3.1 3:81768e06:::benchmark_data_smithi179_90440_object9367:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-02-09T17:34:11.396 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:31:59.299353+0000", 2024-02-09T17:34:11.397 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 132.07632790400001, 2024-02-09T17:34:11.397 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2261435930000002, 2024-02-09T17:34:11.397 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:34:11.397 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:34:11.397 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:34:11.397 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:34:11.397 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:34:11.397 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 141326 2024-02-09T17:34:11.397 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.397 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:34:11.397 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.397 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:34:11.397 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.299353+0000", 2024-02-09T17:34:11.397 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:11.398 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.398 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.398 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:34:11.398 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.299353+0000", 2024-02-09T17:34:11.398 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:11.398 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.398 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.398 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:34:11.398 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.299355+0000", 2024-02-09T17:34:11.398 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.903e-06 2024-02-09T17:34:11.398 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.398 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.398 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:34:11.399 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.299363+0000", 2024-02-09T17:34:11.399 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5440000000000001e-06 2024-02-09T17:34:11.399 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.399 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.399 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:34:11.399 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.299382+0000", 2024-02-09T17:34:11.399 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9511e-05 2024-02-09T17:34:11.399 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.399 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.399 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:34:11.399 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.299393+0000", 2024-02-09T17:34:11.399 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0428000000000001e-05 2024-02-09T17:34:11.399 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.399 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.400 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:11.400 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.299463+0000", 2024-02-09T17:34:11.400 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0080000000000007e-05 2024-02-09T17:34:11.400 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.400 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.400 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:34:11.400 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.299474+0000", 2024-02-09T17:34:11.400 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.149e-05 2024-02-09T17:34:11.400 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.400 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.400 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:11.400 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:00.271334+0000", 2024-02-09T17:34:11.401 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.97185930200000004 2024-02-09T17:34:11.401 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.401 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.401 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:34:11.401 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:00.271349+0000", 2024-02-09T17:34:11.402 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5636000000000001e-05 2024-02-09T17:34:11.402 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.402 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.402 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:11.402 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.509536+0000", 2024-02-09T17:34:11.402 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.23818688 2024-02-09T17:34:11.402 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.402 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.402 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:34:11.402 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.511667+0000", 2024-02-09T17:34:11.402 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021310349999999999 2024-02-09T17:34:11.402 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.403 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.403 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 10,11", 2024-02-09T17:34:11.403 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.511783+0000", 2024-02-09T17:34:11.403 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011539099999999999 2024-02-09T17:34:11.403 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.403 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.403 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:11.403 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.521854+0000", 2024-02-09T17:34:11.403 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010071663999999999 2024-02-09T17:34:11.403 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.403 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.403 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:11.403 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.521897+0000", 2024-02-09T17:34:11.403 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2874999999999999e-05 2024-02-09T17:34:11.404 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.404 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.404 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:34:11.404 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.521910+0000", 2024-02-09T17:34:11.404 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2609000000000001e-05 2024-02-09T17:34:11.404 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.404 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.404 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:34:11.404 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.521952+0000", 2024-02-09T17:34:11.404 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2703000000000001e-05 2024-02-09T17:34:11.404 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.404 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.404 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:34:11.404 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.525497+0000", 2024-02-09T17:34:11.405 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0035445419999999999 2024-02-09T17:34:11.405 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:11.405 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:34:11.405 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:11.405 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.405 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.405 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:141334 3.1 3:83b69ab0:::benchmark_data_smithi179_90440_object9375:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-02-09T17:34:11.405 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:31:59.308786+0000", 2024-02-09T17:34:11.405 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 132.06689520399999, 2024-02-09T17:34:11.405 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2267313619999998, 2024-02-09T17:34:11.405 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:34:11.405 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:34:11.405 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:34:11.405 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:34:11.406 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:34:11.406 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 141334 2024-02-09T17:34:11.406 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.406 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:34:11.406 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.406 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:34:11.406 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.308786+0000", 2024-02-09T17:34:11.406 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:11.406 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.406 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.406 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:34:11.406 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.308786+0000", 2024-02-09T17:34:11.406 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:11.406 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.406 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.407 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:34:11.407 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.308788+0000", 2024-02-09T17:34:11.407 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0470000000000001e-06 2024-02-09T17:34:11.407 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.407 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.407 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:34:11.407 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.308800+0000", 2024-02-09T17:34:11.407 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.163e-05 2024-02-09T17:34:11.407 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.408 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.408 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:34:11.408 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.308816+0000", 2024-02-09T17:34:11.408 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6378000000000001e-05 2024-02-09T17:34:11.408 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.408 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.408 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:34:11.408 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.308827+0000", 2024-02-09T17:34:11.408 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1005000000000001e-05 2024-02-09T17:34:11.408 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.408 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.408 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:11.408 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.308897+0000", 2024-02-09T17:34:11.409 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0285999999999998e-05 2024-02-09T17:34:11.409 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.409 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.409 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:34:11.409 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.308906+0000", 2024-02-09T17:34:11.409 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2020000000000004e-06 2024-02-09T17:34:11.409 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.409 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.409 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:11.409 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:00.271454+0000", 2024-02-09T17:34:11.409 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.96254888100000002 2024-02-09T17:34:11.409 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.409 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.409 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:34:11.410 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:00.271464+0000", 2024-02-09T17:34:11.410 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.499e-06 2024-02-09T17:34:11.410 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.410 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.410 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:11.410 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.518145+0000", 2024-02-09T17:34:11.410 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.246681012 2024-02-09T17:34:11.410 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.410 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.410 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:34:11.410 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.518279+0000", 2024-02-09T17:34:11.410 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000133807 2024-02-09T17:34:11.410 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.410 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.411 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 10,11", 2024-02-09T17:34:11.411 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.518394+0000", 2024-02-09T17:34:11.411 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000114863 2024-02-09T17:34:11.411 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.411 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.411 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:11.411 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.525633+0000", 2024-02-09T17:34:11.411 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0072390129999999999 2024-02-09T17:34:11.411 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.411 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.411 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:34:11.411 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.525649+0000", 2024-02-09T17:34:11.411 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5899000000000001e-05 2024-02-09T17:34:11.411 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.412 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.412 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:11.412 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.531872+0000", 2024-02-09T17:34:11.412 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.006223302 2024-02-09T17:34:11.412 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.412 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.412 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:34:11.412 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.531911+0000", 2024-02-09T17:34:11.412 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8834999999999997e-05 2024-02-09T17:34:11.412 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.412 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.412 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:34:11.412 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.535517+0000", 2024-02-09T17:34:11.413 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0036067030000000002 2024-02-09T17:34:11.413 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:11.413 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:34:11.413 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:11.413 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.413 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.413 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:141335 3.3 3:c1bb3de7:::benchmark_data_smithi179_90440_object9376:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-02-09T17:34:11.413 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:31:59.308929+0000", 2024-02-09T17:34:11.413 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 132.06675194600001, 2024-02-09T17:34:11.413 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3862364280000001, 2024-02-09T17:34:11.413 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:34:11.413 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:34:11.413 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:34:11.414 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:34:11.414 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:34:11.414 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 141335 2024-02-09T17:34:11.414 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.414 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:34:11.414 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.414 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:34:11.414 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.308929+0000", 2024-02-09T17:34:11.414 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:11.414 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.414 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.414 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:34:11.414 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.308929+0000", 2024-02-09T17:34:11.414 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:11.414 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.415 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.415 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:34:11.415 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.308930+0000", 2024-02-09T17:34:11.415 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.75e-07 2024-02-09T17:34:11.415 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.415 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.415 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:34:11.415 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.308934+0000", 2024-02-09T17:34:11.415 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5360000000000001e-06 2024-02-09T17:34:11.415 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.415 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.415 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:34:11.415 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.308941+0000", 2024-02-09T17:34:11.415 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9459999999999998e-06 2024-02-09T17:34:11.415 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.416 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.416 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:34:11.416 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.308947+0000", 2024-02-09T17:34:11.416 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0980000000000001e-06 2024-02-09T17:34:11.416 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.416 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.416 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:11.416 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.309001+0000", 2024-02-09T17:34:11.416 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4840999999999999e-05 2024-02-09T17:34:11.416 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.416 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.416 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:34:11.416 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.309011+0000", 2024-02-09T17:34:11.416 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7329999999999998e-06 2024-02-09T17:34:11.417 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.417 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.417 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:11.417 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:00.272028+0000", 2024-02-09T17:34:11.417 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.963016647 2024-02-09T17:34:11.417 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.417 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.417 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:34:11.417 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:00.272042+0000", 2024-02-09T17:34:11.417 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3910999999999999e-05 2024-02-09T17:34:11.417 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.417 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.417 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:11.417 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.638427+0000", 2024-02-09T17:34:11.418 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.366385634 2024-02-09T17:34:11.418 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.418 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.418 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:34:11.418 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.638566+0000", 2024-02-09T17:34:11.418 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000139021 2024-02-09T17:34:11.418 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.418 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.418 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-02-09T17:34:11.418 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.638703+0000", 2024-02-09T17:34:11.418 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013641599999999999 2024-02-09T17:34:11.418 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.418 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.418 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:11.419 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.647842+0000", 2024-02-09T17:34:11.419 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0091392599999999997 2024-02-09T17:34:11.419 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.419 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.419 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:34:11.419 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.647864+0000", 2024-02-09T17:34:11.419 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2294e-05 2024-02-09T17:34:11.419 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.419 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.419 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:11.419 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.690323+0000", 2024-02-09T17:34:11.419 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.042458900000000001 2024-02-09T17:34:11.419 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.419 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.420 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:34:11.420 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.690393+0000", 2024-02-09T17:34:11.420 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9597000000000005e-05 2024-02-09T17:34:11.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.420 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.420 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:34:11.420 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.695166+0000", 2024-02-09T17:34:11.420 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0047729189999999996 2024-02-09T17:34:11.420 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:11.420 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:34:11.420 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:11.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.420 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.420 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:141338 3.1 3:8fb935f2:::benchmark_data_smithi179_90440_object9379:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-02-09T17:34:11.421 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:31:59.309478+0000", 2024-02-09T17:34:11.421 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 132.06620331100001, 2024-02-09T17:34:11.421 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2380588619999999, 2024-02-09T17:34:11.421 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:34:11.421 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:34:11.421 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:34:11.421 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:34:11.421 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:34:11.421 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 141338 2024-02-09T17:34:11.421 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.421 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:34:11.421 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.421 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:34:11.421 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.309478+0000", 2024-02-09T17:34:11.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:11.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.422 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:34:11.422 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.309478+0000", 2024-02-09T17:34:11.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:11.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.422 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:34:11.422 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.309479+0000", 2024-02-09T17:34:11.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2099999999999996e-07 2024-02-09T17:34:11.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.422 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:34:11.423 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.309482+0000", 2024-02-09T17:34:11.423 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5429999999999998e-06 2024-02-09T17:34:11.423 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.423 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.423 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:34:11.423 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.309492+0000", 2024-02-09T17:34:11.423 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3249999999999997e-06 2024-02-09T17:34:11.423 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.423 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.423 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:34:11.423 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.309497+0000", 2024-02-09T17:34:11.423 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6119999999999998e-06 2024-02-09T17:34:11.423 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.423 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.424 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:11.424 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.309536+0000", 2024-02-09T17:34:11.424 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9208000000000001e-05 2024-02-09T17:34:11.424 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.424 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.424 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:34:11.424 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.309543+0000", 2024-02-09T17:34:11.424 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8990000000000004e-06 2024-02-09T17:34:11.424 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.424 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.424 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:11.424 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:00.271494+0000", 2024-02-09T17:34:11.424 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.96195103400000004 2024-02-09T17:34:11.424 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.424 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.425 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:34:11.425 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:00.271501+0000", 2024-02-09T17:34:11.425 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2430000000000002e-06 2024-02-09T17:34:11.425 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.425 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.425 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:11.425 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.525677+0000", 2024-02-09T17:34:11.425 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2541769249999999 2024-02-09T17:34:11.425 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.425 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.425 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:34:11.425 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.525798+0000", 2024-02-09T17:34:11.425 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000120346 2024-02-09T17:34:11.426 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.426 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.426 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 10,11", 2024-02-09T17:34:11.426 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.525908+0000", 2024-02-09T17:34:11.426 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000109718 2024-02-09T17:34:11.426 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.426 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.426 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:34:11.426 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.535532+0000", 2024-02-09T17:34:11.426 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0096243200000000004 2024-02-09T17:34:11.426 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.426 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.426 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:11.427 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.535569+0000", 2024-02-09T17:34:11.427 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7517e-05 2024-02-09T17:34:11.427 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.427 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.427 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:11.427 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.544823+0000", 2024-02-09T17:34:11.427 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0092536679999999996 2024-02-09T17:34:11.427 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.427 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.427 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:34:11.427 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.544862+0000", 2024-02-09T17:34:11.427 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9372000000000002e-05 2024-02-09T17:34:11.427 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.427 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.428 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:34:11.428 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.547537+0000", 2024-02-09T17:34:11.428 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0026744109999999998 2024-02-09T17:34:11.428 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:11.428 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:34:11.428 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:11.428 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.428 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.428 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:141341 3.1 3:87f265c7:::benchmark_data_smithi179_90440_object9382:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-02-09T17:34:11.428 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:31:59.322821+0000", 2024-02-09T17:34:11.428 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 132.05286011199999, 2024-02-09T17:34:11.428 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2283184889999998, 2024-02-09T17:34:11.428 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:34:11.428 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:34:11.429 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:34:11.429 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:34:11.429 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:34:11.429 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 141341 2024-02-09T17:34:11.429 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.429 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:34:11.429 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.429 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:34:11.429 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.322821+0000", 2024-02-09T17:34:11.429 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:11.429 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.429 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.429 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:34:11.430 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.322821+0000", 2024-02-09T17:34:11.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:11.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.430 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:34:11.430 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.322824+0000", 2024-02-09T17:34:11.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8260000000000001e-06 2024-02-09T17:34:11.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.430 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:34:11.430 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.322836+0000", 2024-02-09T17:34:11.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2057e-05 2024-02-09T17:34:11.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.430 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:34:11.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.322867+0000", 2024-02-09T17:34:11.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1186000000000003e-05 2024-02-09T17:34:11.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.431 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:34:11.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.322884+0000", 2024-02-09T17:34:11.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6676000000000001e-05 2024-02-09T17:34:11.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.431 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:11.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.322999+0000", 2024-02-09T17:34:11.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011505099999999999 2024-02-09T17:34:11.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.431 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:34:11.432 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.323020+0000", 2024-02-09T17:34:11.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1120000000000001e-05 2024-02-09T17:34:11.432 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.432 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.432 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:11.432 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:00.271533+0000", 2024-02-09T17:34:11.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.94851338600000001 2024-02-09T17:34:11.432 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.432 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.432 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:34:11.432 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:00.271543+0000", 2024-02-09T17:34:11.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0599999999999997e-06 2024-02-09T17:34:11.432 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.433 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.433 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:11.433 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.529155+0000", 2024-02-09T17:34:11.433 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2576121200000001 2024-02-09T17:34:11.433 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.433 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.433 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:34:11.433 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.529283+0000", 2024-02-09T17:34:11.433 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012811800000000001 2024-02-09T17:34:11.433 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.433 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.433 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 10,11", 2024-02-09T17:34:11.433 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.529381+0000", 2024-02-09T17:34:11.433 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7850000000000007e-05 2024-02-09T17:34:11.433 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.434 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.434 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:11.434 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.535605+0000", 2024-02-09T17:34:11.434 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0062243070000000001 2024-02-09T17:34:11.434 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.434 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.434 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:34:11.434 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.535613+0000", 2024-02-09T17:34:11.434 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2400000000000007e-06 2024-02-09T17:34:11.434 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.434 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.434 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:11.434 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.547603+0000", 2024-02-09T17:34:11.435 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011989506 2024-02-09T17:34:11.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.435 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.435 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:34:11.435 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.547666+0000", 2024-02-09T17:34:11.435 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3336000000000005e-05 2024-02-09T17:34:11.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.435 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.435 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:34:11.435 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.551140+0000", 2024-02-09T17:34:11.435 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00347365 2024-02-09T17:34:11.435 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:11.435 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:34:11.435 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:11.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.435 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.436 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:145191 3.3 3:cad5c6a9:::benchmark_data_smithi179_90440_object13232:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-02-09T17:34:11.436 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:32:07.951037+0000", 2024-02-09T17:34:11.436 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 123.424644297, 2024-02-09T17:34:11.436 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29727604600000002, 2024-02-09T17:34:11.436 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:34:11.436 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:34:11.436 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:34:11.436 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:34:11.436 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:34:11.436 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 145191 2024-02-09T17:34:11.436 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.436 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:34:11.436 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.436 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:34:11.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.951037+0000", 2024-02-09T17:34:11.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:11.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.437 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:34:11.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.951037+0000", 2024-02-09T17:34:11.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:11.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.437 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:34:11.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.951038+0000", 2024-02-09T17:34:11.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5120000000000001e-06 2024-02-09T17:34:11.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:34:11.438 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.951043+0000", 2024-02-09T17:34:11.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4689999999999999e-06 2024-02-09T17:34:11.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:34:11.438 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.951056+0000", 2024-02-09T17:34:11.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3112000000000001e-05 2024-02-09T17:34:11.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:34:11.438 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.951066+0000", 2024-02-09T17:34:11.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8190000000000003e-06 2024-02-09T17:34:11.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:11.439 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.969513+0000", 2024-02-09T17:34:11.439 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018447468000000002 2024-02-09T17:34:11.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.439 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for rw locks", 2024-02-09T17:34:11.439 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.969630+0000", 2024-02-09T17:34:11.439 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000116783 2024-02-09T17:34:11.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.439 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:11.439 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:08.209923+0000", 2024-02-09T17:34:11.439 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.240293012 2024-02-09T17:34:11.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.439 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:34:11.440 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:08.210020+0000", 2024-02-09T17:34:11.440 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7258999999999998e-05 2024-02-09T17:34:11.440 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.440 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.440 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-02-09T17:34:11.440 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:08.210179+0000", 2024-02-09T17:34:11.440 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015853300000000001 2024-02-09T17:34:11.440 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.440 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.440 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:11.440 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:08.234698+0000", 2024-02-09T17:34:11.440 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.024519374 2024-02-09T17:34:11.440 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.441 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.441 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:34:11.441 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:08.234717+0000", 2024-02-09T17:34:11.441 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.906e-05 2024-02-09T17:34:11.441 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.441 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.441 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:11.441 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:08.244438+0000", 2024-02-09T17:34:11.441 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0097207460000000006 2024-02-09T17:34:11.441 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.441 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.441 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:34:11.441 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:08.244515+0000", 2024-02-09T17:34:11.441 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6779999999999993e-05 2024-02-09T17:34:11.441 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.442 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.442 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:34:11.442 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:08.248313+0000", 2024-02-09T17:34:11.442 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003798119 2024-02-09T17:34:11.442 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:11.442 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:34:11.442 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:11.442 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.442 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.442 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:145196 3.3 3:c8843776:::benchmark_data_smithi179_90440_object13237:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-02-09T17:34:11.442 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:32:07.956283+0000", 2024-02-09T17:34:11.442 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 123.419398191, 2024-02-09T17:34:11.442 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23580672, 2024-02-09T17:34:11.442 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:34:11.443 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:34:11.443 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:34:11.443 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:34:11.443 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:34:11.443 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 145196 2024-02-09T17:34:11.443 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.443 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:34:11.443 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.443 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:34:11.443 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.956283+0000", 2024-02-09T17:34:11.444 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:11.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.444 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.444 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:34:11.444 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.956283+0000", 2024-02-09T17:34:11.444 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:11.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.444 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.444 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:34:11.445 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.956285+0000", 2024-02-09T17:34:11.445 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9690000000000001e-06 2024-02-09T17:34:11.445 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.445 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.445 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:34:11.445 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.956292+0000", 2024-02-09T17:34:11.445 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0480000000000001e-06 2024-02-09T17:34:11.445 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.445 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.445 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:34:11.445 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.956311+0000", 2024-02-09T17:34:11.445 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8859e-05 2024-02-09T17:34:11.445 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.445 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.446 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:34:11.446 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.956321+0000", 2024-02-09T17:34:11.446 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0199e-05 2024-02-09T17:34:11.446 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.446 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.446 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:11.446 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.969660+0000", 2024-02-09T17:34:11.446 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013339177000000001 2024-02-09T17:34:11.446 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.446 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.446 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:34:11.446 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.969809+0000", 2024-02-09T17:34:11.446 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000148462 2024-02-09T17:34:11.447 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.447 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.447 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-02-09T17:34:11.447 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.969975+0000", 2024-02-09T17:34:11.447 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016594900000000001 2024-02-09T17:34:11.447 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.447 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.447 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:34:11.447 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.984771+0000", 2024-02-09T17:34:11.447 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014796444000000001 2024-02-09T17:34:11.447 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.447 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.448 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:11.448 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.986213+0000", 2024-02-09T17:34:11.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001441634 2024-02-09T17:34:11.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.448 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:11.448 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:08.187179+0000", 2024-02-09T17:34:11.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20096615100000001 2024-02-09T17:34:11.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.448 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:34:11.448 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:08.187259+0000", 2024-02-09T17:34:11.449 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9784999999999996e-05 2024-02-09T17:34:11.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.449 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:34:11.449 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:08.192090+0000", 2024-02-09T17:34:11.449 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0048310430000000001 2024-02-09T17:34:11.449 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:11.449 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:34:11.449 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:11.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.449 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:145204 3.3 3:c8983e63:::benchmark_data_smithi179_90440_object13245:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-02-09T17:34:11.449 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:32:07.964287+0000", 2024-02-09T17:34:11.449 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 123.411393813, 2024-02-09T17:34:11.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23277460699999999, 2024-02-09T17:34:11.450 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:34:11.450 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:34:11.450 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:34:11.450 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:34:11.450 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:34:11.450 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 145204 2024-02-09T17:34:11.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.450 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:34:11.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:34:11.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.964287+0000", 2024-02-09T17:34:11.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:11.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.451 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:34:11.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.964287+0000", 2024-02-09T17:34:11.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:11.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.451 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:34:11.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.964289+0000", 2024-02-09T17:34:11.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.889e-06 2024-02-09T17:34:11.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.451 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:34:11.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.964294+0000", 2024-02-09T17:34:11.452 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.028e-06 2024-02-09T17:34:11.452 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.452 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.452 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:34:11.452 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.964306+0000", 2024-02-09T17:34:11.452 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1761e-05 2024-02-09T17:34:11.452 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.452 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.452 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:34:11.452 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.964315+0000", 2024-02-09T17:34:11.452 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9290000000000004e-06 2024-02-09T17:34:11.452 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.452 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.453 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:11.453 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.974776+0000", 2024-02-09T17:34:11.453 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010460661 2024-02-09T17:34:11.453 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.453 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.453 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:34:11.453 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.974939+0000", 2024-02-09T17:34:11.453 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016313699999999999 2024-02-09T17:34:11.453 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.453 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.453 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-02-09T17:34:11.453 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.975094+0000", 2024-02-09T17:34:11.453 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015559699999999999 2024-02-09T17:34:11.453 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.454 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:34:11.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.984825+0000", 2024-02-09T17:34:11.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0097302139999999992 2024-02-09T17:34:11.454 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.454 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:11.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.986252+0000", 2024-02-09T17:34:11.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014271380000000001 2024-02-09T17:34:11.454 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.454 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:11.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:08.192160+0000", 2024-02-09T17:34:11.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20590793499999999 2024-02-09T17:34:11.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:34:11.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:08.192226+0000", 2024-02-09T17:34:11.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6156999999999997e-05 2024-02-09T17:34:11.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:34:11.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:08.197062+0000", 2024-02-09T17:34:11.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0048361610000000003 2024-02-09T17:34:11.455 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:11.455 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:34:11.455 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:11.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.456 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.456 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:145210 3.3 3:ceedd519:::benchmark_data_smithi179_90440_object13251:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-02-09T17:34:11.456 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:32:07.974637+0000", 2024-02-09T17:34:11.456 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 123.40104440499999, 2024-02-09T17:34:11.456 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.227001813, 2024-02-09T17:34:11.456 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:34:11.456 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:34:11.456 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:34:11.456 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:34:11.456 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:34:11.456 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 145210 2024-02-09T17:34:11.456 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.456 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:34:11.456 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.457 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:34:11.457 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.974637+0000", 2024-02-09T17:34:11.457 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:11.457 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.457 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.457 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:34:11.457 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.974637+0000", 2024-02-09T17:34:11.457 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:11.457 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.457 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.457 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:34:11.457 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.974638+0000", 2024-02-09T17:34:11.457 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3400000000000001e-06 2024-02-09T17:34:11.457 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.457 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.458 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:34:11.458 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.974645+0000", 2024-02-09T17:34:11.458 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1980000000000004e-06 2024-02-09T17:34:11.458 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.458 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.458 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:34:11.458 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.974660+0000", 2024-02-09T17:34:11.458 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5028000000000001e-05 2024-02-09T17:34:11.458 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.458 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.458 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:34:11.458 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.974669+0000", 2024-02-09T17:34:11.458 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5499999999999995e-06 2024-02-09T17:34:11.458 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.458 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.459 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:11.459 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.979748+0000", 2024-02-09T17:34:11.459 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0050785989999999996 2024-02-09T17:34:11.459 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.459 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.459 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:34:11.459 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.979931+0000", 2024-02-09T17:34:11.459 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018370300000000001 2024-02-09T17:34:11.459 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.459 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.459 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-02-09T17:34:11.459 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.980100+0000", 2024-02-09T17:34:11.459 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016823399999999999 2024-02-09T17:34:11.459 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.460 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.460 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:34:11.460 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.984894+0000", 2024-02-09T17:34:11.460 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0047940969999999998 2024-02-09T17:34:11.460 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.460 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.460 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:11.460 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.986286+0000", 2024-02-09T17:34:11.460 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001392503 2024-02-09T17:34:11.460 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.460 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.460 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:11.460 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:08.197144+0000", 2024-02-09T17:34:11.461 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.210857987 2024-02-09T17:34:11.461 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.461 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.461 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:34:11.461 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:08.197216+0000", 2024-02-09T17:34:11.461 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1845000000000001e-05 2024-02-09T17:34:11.461 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.461 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.461 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:34:11.461 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:08.201639+0000", 2024-02-09T17:34:11.461 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0044227290000000002 2024-02-09T17:34:11.461 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:11.461 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:34:11.461 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:11.461 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.462 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.462 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:145211 3.1 3:8971f3c3:::benchmark_data_smithi179_90440_object13252:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-02-09T17:34:11.462 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:32:07.975326+0000", 2024-02-09T17:34:11.462 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 123.400355581, 2024-02-09T17:34:11.462 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23558217200000001, 2024-02-09T17:34:11.462 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:34:11.462 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:34:11.462 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:34:11.462 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:34:11.462 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:34:11.462 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 145211 2024-02-09T17:34:11.462 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.462 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:34:11.463 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.463 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:34:11.463 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.975326+0000", 2024-02-09T17:34:11.463 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:11.463 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.463 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.463 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:34:11.463 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.975326+0000", 2024-02-09T17:34:11.463 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:11.463 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.463 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.463 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:34:11.463 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.975326+0000", 2024-02-09T17:34:11.463 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1999999999999998e-07 2024-02-09T17:34:11.464 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.464 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.464 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:34:11.464 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.975330+0000", 2024-02-09T17:34:11.464 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2040000000000002e-06 2024-02-09T17:34:11.464 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.464 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.464 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:34:11.464 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.975339+0000", 2024-02-09T17:34:11.464 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6379999999999999e-06 2024-02-09T17:34:11.464 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.464 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.464 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:34:11.464 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.975346+0000", 2024-02-09T17:34:11.465 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6830000000000002e-06 2024-02-09T17:34:11.465 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.465 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.465 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:11.465 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.982174+0000", 2024-02-09T17:34:11.465 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0068282500000000001 2024-02-09T17:34:11.465 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.465 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.465 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:34:11.465 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.982281+0000", 2024-02-09T17:34:11.465 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010708 2024-02-09T17:34:11.465 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.465 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.466 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 10,11", 2024-02-09T17:34:11.466 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.982401+0000", 2024-02-09T17:34:11.466 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000119327 2024-02-09T17:34:11.466 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.466 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.466 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:11.466 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:08.001265+0000", 2024-02-09T17:34:11.466 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018864715000000001 2024-02-09T17:34:11.466 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.466 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.466 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:34:11.466 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:08.004990+0000", 2024-02-09T17:34:11.467 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0037250740000000001 2024-02-09T17:34:11.467 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.467 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.467 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:11.467 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:08.207271+0000", 2024-02-09T17:34:11.467 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20228028300000001 2024-02-09T17:34:11.467 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.467 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.467 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:34:11.467 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:08.207313+0000", 2024-02-09T17:34:11.467 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2364000000000003e-05 2024-02-09T17:34:11.467 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.467 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.468 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:34:11.468 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:08.210908+0000", 2024-02-09T17:34:11.468 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003594734 2024-02-09T17:34:11.468 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:11.468 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:34:11.468 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:11.468 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.468 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.468 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:145219 3.1 3:84a03c16:::benchmark_data_smithi179_90440_object13260:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-02-09T17:34:11.468 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:32:07.993271+0000", 2024-02-09T17:34:11.468 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 123.382410421, 2024-02-09T17:34:11.468 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22164267600000001, 2024-02-09T17:34:11.468 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:34:11.469 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:34:11.469 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:34:11.469 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:34:11.469 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:34:11.469 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 145219 2024-02-09T17:34:11.469 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.469 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:34:11.469 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.469 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:34:11.469 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.993271+0000", 2024-02-09T17:34:11.469 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:11.469 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.469 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.470 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:34:11.470 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.993271+0000", 2024-02-09T17:34:11.470 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:11.470 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.470 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.470 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:34:11.470 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.993272+0000", 2024-02-09T17:34:11.470 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4929999999999999e-06 2024-02-09T17:34:11.470 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.470 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.470 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:34:11.470 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.993278+0000", 2024-02-09T17:34:11.470 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.276e-06 2024-02-09T17:34:11.470 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.470 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.471 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:34:11.471 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.993289+0000", 2024-02-09T17:34:11.471 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1482e-05 2024-02-09T17:34:11.471 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.471 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.471 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:34:11.471 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.993296+0000", 2024-02-09T17:34:11.471 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4250000000000003e-06 2024-02-09T17:34:11.471 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.471 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.471 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:11.471 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.996862+0000", 2024-02-09T17:34:11.472 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0035664469999999999 2024-02-09T17:34:11.472 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.472 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.472 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:34:11.472 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.996967+0000", 2024-02-09T17:34:11.472 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000104548 2024-02-09T17:34:11.472 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.472 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.472 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 10,11", 2024-02-09T17:34:11.472 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.997106+0000", 2024-02-09T17:34:11.472 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013929 2024-02-09T17:34:11.472 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.472 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.472 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:11.473 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:08.004972+0000", 2024-02-09T17:34:11.473 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.007866662 2024-02-09T17:34:11.473 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.473 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.473 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:34:11.473 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:08.005001+0000", 2024-02-09T17:34:11.473 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8365999999999999e-05 2024-02-09T17:34:11.473 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.473 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.473 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:11.473 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:08.210974+0000", 2024-02-09T17:34:11.473 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.205973291 2024-02-09T17:34:11.473 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.473 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.474 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:34:11.474 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:08.211040+0000", 2024-02-09T17:34:11.474 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6285999999999995e-05 2024-02-09T17:34:11.474 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.474 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.474 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:34:11.474 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:08.214914+0000", 2024-02-09T17:34:11.474 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0038731099999999999 2024-02-09T17:34:11.474 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:11.474 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:34:11.474 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:11.474 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.474 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.474 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:157800 3.1 3:8df3bfb7:::benchmark_data_smithi179_90440_object25841:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-02-09T17:34:11.475 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:32:28.517351+0000", 2024-02-09T17:34:11.475 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 102.85833021099999, 2024-02-09T17:34:11.475 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27107035499999999, 2024-02-09T17:34:11.475 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:34:11.475 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:34:11.475 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:34:11.475 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:34:11.475 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:34:11.475 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 157800 2024-02-09T17:34:11.475 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.475 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:34:11.475 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.475 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:34:11.475 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.517351+0000", 2024-02-09T17:34:11.475 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:11.475 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.476 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.476 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:34:11.476 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.517351+0000", 2024-02-09T17:34:11.476 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:11.476 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.476 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.476 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:34:11.476 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.517353+0000", 2024-02-09T17:34:11.476 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7e-06 2024-02-09T17:34:11.476 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.476 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.476 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:34:11.476 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.517359+0000", 2024-02-09T17:34:11.476 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9390000000000002e-06 2024-02-09T17:34:11.477 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.477 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.477 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:34:11.477 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.517374+0000", 2024-02-09T17:34:11.477 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5179e-05 2024-02-09T17:34:11.477 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.477 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.477 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:34:11.477 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.517384+0000", 2024-02-09T17:34:11.477 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0013000000000001e-05 2024-02-09T17:34:11.477 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.477 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.477 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:11.477 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.525824+0000", 2024-02-09T17:34:11.478 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.008440421 2024-02-09T17:34:11.478 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.478 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.478 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for rw locks", 2024-02-09T17:34:11.478 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.525954+0000", 2024-02-09T17:34:11.478 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000130078 2024-02-09T17:34:11.478 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.478 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.478 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:11.478 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.554541+0000", 2024-02-09T17:34:11.478 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.028586126999999999 2024-02-09T17:34:11.478 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.478 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.479 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:34:11.479 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.554619+0000", 2024-02-09T17:34:11.479 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8422000000000002e-05 2024-02-09T17:34:11.479 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.479 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.479 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 10,11", 2024-02-09T17:34:11.479 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.554742+0000", 2024-02-09T17:34:11.479 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012343799999999999 2024-02-09T17:34:11.479 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.479 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.479 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:34:11.479 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.573831+0000", 2024-02-09T17:34:11.479 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019088766 2024-02-09T17:34:11.479 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.480 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.480 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:11.480 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.573874+0000", 2024-02-09T17:34:11.480 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2917000000000003e-05 2024-02-09T17:34:11.480 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.480 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.480 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:11.480 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.784939+0000", 2024-02-09T17:34:11.480 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21106476599999999 2024-02-09T17:34:11.480 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.480 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.480 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:34:11.480 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.784983+0000", 2024-02-09T17:34:11.480 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4435999999999999e-05 2024-02-09T17:34:11.481 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.481 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.481 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:34:11.481 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.788421+0000", 2024-02-09T17:34:11.481 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034381530000000002 2024-02-09T17:34:11.481 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:11.481 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:34:11.481 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:11.481 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.481 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.481 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:157808 3.1 3:8cbdb1cd:::benchmark_data_smithi179_90440_object25849:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-02-09T17:34:11.481 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:32:28.531471+0000", 2024-02-09T17:34:11.481 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 102.844210561, 2024-02-09T17:34:11.481 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.31521497300000001, 2024-02-09T17:34:11.482 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:34:11.482 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:34:11.482 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:34:11.482 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:34:11.482 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:34:11.482 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 157808 2024-02-09T17:34:11.482 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.482 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:34:11.482 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.482 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:34:11.482 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.531471+0000", 2024-02-09T17:34:11.482 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:11.482 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.482 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.483 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:34:11.483 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.531471+0000", 2024-02-09T17:34:11.483 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:11.483 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.483 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.483 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:34:11.483 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.531472+0000", 2024-02-09T17:34:11.483 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.46e-06 2024-02-09T17:34:11.483 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.483 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.483 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:34:11.483 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.531476+0000", 2024-02-09T17:34:11.483 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5540000000000002e-06 2024-02-09T17:34:11.483 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.483 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.484 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:34:11.484 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.531484+0000", 2024-02-09T17:34:11.484 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0730000000000005e-06 2024-02-09T17:34:11.484 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.484 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.484 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:34:11.484 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.531490+0000", 2024-02-09T17:34:11.484 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5490000000000003e-06 2024-02-09T17:34:11.484 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.484 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.484 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:11.484 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.559900+0000", 2024-02-09T17:34:11.484 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.028409363 2024-02-09T17:34:11.484 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.484 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.485 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for rw locks", 2024-02-09T17:34:11.485 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.560049+0000", 2024-02-09T17:34:11.485 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014909600000000001 2024-02-09T17:34:11.485 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.485 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.485 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:11.485 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.809501+0000", 2024-02-09T17:34:11.485 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24945214700000001 2024-02-09T17:34:11.485 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.485 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.485 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:34:11.485 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.809563+0000", 2024-02-09T17:34:11.485 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2045999999999995e-05 2024-02-09T17:34:11.485 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.486 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.486 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 10,11", 2024-02-09T17:34:11.486 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.809645+0000", 2024-02-09T17:34:11.486 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1823999999999995e-05 2024-02-09T17:34:11.486 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.486 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.486 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:34:11.486 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.831085+0000", 2024-02-09T17:34:11.486 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.021439877999999999 2024-02-09T17:34:11.486 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.486 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.486 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:11.486 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.842911+0000", 2024-02-09T17:34:11.486 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011826346999999999 2024-02-09T17:34:11.486 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.487 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.487 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:11.487 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.842962+0000", 2024-02-09T17:34:11.487 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0525999999999997e-05 2024-02-09T17:34:11.487 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.487 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.487 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:34:11.487 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.843030+0000", 2024-02-09T17:34:11.487 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8524999999999999e-05 2024-02-09T17:34:11.487 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.487 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.487 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:34:11.487 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.846686+0000", 2024-02-09T17:34:11.487 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0036555849999999998 2024-02-09T17:34:11.488 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:11.488 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:34:11.488 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:11.488 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.488 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.488 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:157812 3.3 3:cf0cb945:::benchmark_data_smithi179_90440_object25853:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-02-09T17:34:11.488 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:32:28.531498+0000", 2024-02-09T17:34:11.488 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 102.844183387, 2024-02-09T17:34:11.488 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25599428899999999, 2024-02-09T17:34:11.488 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:34:11.488 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:34:11.488 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:34:11.488 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:34:11.488 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:34:11.489 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 157812 2024-02-09T17:34:11.489 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.489 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:34:11.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.489 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:34:11.489 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.531498+0000", 2024-02-09T17:34:11.489 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:11.489 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.489 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:34:11.489 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.531498+0000", 2024-02-09T17:34:11.489 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:11.489 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.490 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:34:11.490 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.531498+0000", 2024-02-09T17:34:11.490 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7599999999999998e-07 2024-02-09T17:34:11.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.490 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:34:11.490 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.531500+0000", 2024-02-09T17:34:11.490 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1150000000000001e-06 2024-02-09T17:34:11.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.490 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:34:11.490 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.531505+0000", 2024-02-09T17:34:11.490 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0309999999999998e-06 2024-02-09T17:34:11.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.491 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:34:11.491 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.531510+0000", 2024-02-09T17:34:11.491 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2400000000000001e-06 2024-02-09T17:34:11.491 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.491 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.491 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:11.491 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.536272+0000", 2024-02-09T17:34:11.491 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0047623359999999998 2024-02-09T17:34:11.491 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.491 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.491 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for rw locks", 2024-02-09T17:34:11.491 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.536414+0000", 2024-02-09T17:34:11.491 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014161600000000001 2024-02-09T17:34:11.492 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.492 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:11.492 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.548492+0000", 2024-02-09T17:34:11.492 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012077925999999999 2024-02-09T17:34:11.492 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.492 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:34:11.492 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.548597+0000", 2024-02-09T17:34:11.492 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010580400000000001 2024-02-09T17:34:11.492 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.492 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-02-09T17:34:11.495 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.548752+0000", 2024-02-09T17:34:11.495 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015489599999999999 2024-02-09T17:34:11.495 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.495 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.495 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:11.495 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.570524+0000", 2024-02-09T17:34:11.495 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.021771526999999999 2024-02-09T17:34:11.495 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.495 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.495 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:34:11.496 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.570547+0000", 2024-02-09T17:34:11.496 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3326e-05 2024-02-09T17:34:11.496 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.496 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.496 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:11.496 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.782860+0000", 2024-02-09T17:34:11.496 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.212313375 2024-02-09T17:34:11.496 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.496 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.496 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:34:11.496 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.782933+0000", 2024-02-09T17:34:11.496 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3035999999999997e-05 2024-02-09T17:34:11.496 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.497 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.497 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:34:11.497 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.787492+0000", 2024-02-09T17:34:11.497 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0045586849999999998 2024-02-09T17:34:11.497 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:11.497 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:34:11.497 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:11.497 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.497 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.497 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:157846 3.3 3:ca04de7c:::benchmark_data_smithi179_90440_object25887:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-02-09T17:34:11.497 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:32:28.592701+0000", 2024-02-09T17:34:11.497 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 102.78298029, 2024-02-09T17:34:11.497 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.242762859, 2024-02-09T17:34:11.497 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:34:11.497 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:34:11.498 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:34:11.498 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:34:11.498 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:34:11.498 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 157846 2024-02-09T17:34:11.498 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.498 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:34:11.498 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.498 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:34:11.498 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.592701+0000", 2024-02-09T17:34:11.498 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:11.498 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.498 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.498 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:34:11.499 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.592701+0000", 2024-02-09T17:34:11.499 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:11.499 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.499 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.499 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:34:11.499 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.592703+0000", 2024-02-09T17:34:11.499 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.59e-06 2024-02-09T17:34:11.499 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.499 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.499 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:34:11.499 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.592708+0000", 2024-02-09T17:34:11.499 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6239999999999999e-06 2024-02-09T17:34:11.499 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.499 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.500 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:34:11.500 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.592723+0000", 2024-02-09T17:34:11.500 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4554e-05 2024-02-09T17:34:11.500 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.500 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.500 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:34:11.500 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.592732+0000", 2024-02-09T17:34:11.500 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3899999999999999e-06 2024-02-09T17:34:11.500 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.500 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.500 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:11.500 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.592796+0000", 2024-02-09T17:34:11.500 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3545000000000002e-05 2024-02-09T17:34:11.500 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.501 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for rw locks", 2024-02-09T17:34:11.501 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.592939+0000", 2024-02-09T17:34:11.501 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014344 2024-02-09T17:34:11.501 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.501 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:11.501 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.805305+0000", 2024-02-09T17:34:11.501 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21236548599999999 2024-02-09T17:34:11.501 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.501 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:34:11.501 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.805421+0000", 2024-02-09T17:34:11.501 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000116709 2024-02-09T17:34:11.501 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.502 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.502 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-02-09T17:34:11.502 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.805590+0000", 2024-02-09T17:34:11.502 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000169135 2024-02-09T17:34:11.502 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.502 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.502 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:11.502 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.830531+0000", 2024-02-09T17:34:11.502 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.024940424999999999 2024-02-09T17:34:11.502 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.502 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.502 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:11.502 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.830595+0000", 2024-02-09T17:34:11.503 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3744000000000005e-05 2024-02-09T17:34:11.503 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.503 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.503 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:34:11.503 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.830773+0000", 2024-02-09T17:34:11.503 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017874500000000001 2024-02-09T17:34:11.503 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.503 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.503 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:34:11.503 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.830825+0000", 2024-02-09T17:34:11.503 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1660999999999999e-05 2024-02-09T17:34:11.503 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.503 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.504 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:34:11.504 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.835464+0000", 2024-02-09T17:34:11.504 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0046388109999999996 2024-02-09T17:34:11.504 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:11.504 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:34:11.504 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:11.504 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.504 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.504 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:173744 3.1 3:810a7512:::benchmark_data_smithi179_90440_object41785:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e259)", 2024-02-09T17:34:11.504 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:32:55.609893+0000", 2024-02-09T17:34:11.504 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 75.765787764999999, 2024-02-09T17:34:11.504 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.52845357100000001, 2024-02-09T17:34:11.504 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:34:11.504 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-09T17:34:11.505 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:34:11.505 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:34:11.505 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:34:11.505 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 173744 2024-02-09T17:34:11.505 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.505 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:34:11.505 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.505 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:34:11.505 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:55.609893+0000", 2024-02-09T17:34:11.505 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:11.505 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.505 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.505 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:34:11.505 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:55.609893+0000", 2024-02-09T17:34:11.506 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:11.506 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.506 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.506 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:34:11.506 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:55.609894+0000", 2024-02-09T17:34:11.506 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0699999999999999e-07 2024-02-09T17:34:11.506 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.506 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.506 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:34:11.506 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:55.609898+0000", 2024-02-09T17:34:11.506 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9790000000000004e-06 2024-02-09T17:34:11.506 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.506 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.506 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:34:11.507 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:55.609901+0000", 2024-02-09T17:34:11.507 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3529999999999999e-06 2024-02-09T17:34:11.507 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.507 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.507 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:34:11.507 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:55.609904+0000", 2024-02-09T17:34:11.507 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8990000000000001e-06 2024-02-09T17:34:11.507 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.507 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.507 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:11.507 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:55.624985+0000", 2024-02-09T17:34:11.507 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015081099000000001 2024-02-09T17:34:11.507 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.507 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.507 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for missing object", 2024-02-09T17:34:11.508 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:55.625090+0000", 2024-02-09T17:34:11.508 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010458 2024-02-09T17:34:11.508 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.508 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.508 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:11.508 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:56.138330+0000", 2024-02-09T17:34:11.508 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.51324086800000002 2024-02-09T17:34:11.508 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.508 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.508 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:34:11.508 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:56.138347+0000", 2024-02-09T17:34:11.508 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6586e-05 2024-02-09T17:34:11.508 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:11.509 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:34:11.509 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:11.509 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.509 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.509 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:204179 3.1 3:8372c17c:::benchmark_data_smithi179_90440_object72220:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e277)", 2024-02-09T17:34:11.509 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:33:44.992761+0000", 2024-02-09T17:34:11.509 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 26.382920679000001, 2024-02-09T17:34:11.509 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6720821499999996, 2024-02-09T17:34:11.509 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:34:11.509 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-09T17:34:11.509 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:34:11.509 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:34:11.509 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:34:11.509 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 204179 2024-02-09T17:34:11.510 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.510 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:34:11.510 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.510 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:34:11.510 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:44.992761+0000", 2024-02-09T17:34:11.510 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:11.510 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.510 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.510 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:34:11.510 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:44.992761+0000", 2024-02-09T17:34:11.510 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:11.510 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.510 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.511 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:34:11.511 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:44.992761+0000", 2024-02-09T17:34:11.511 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0800000000000001e-07 2024-02-09T17:34:11.511 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.511 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.511 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:34:11.511 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:44.992762+0000", 2024-02-09T17:34:11.511 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6020000000000001e-06 2024-02-09T17:34:11.511 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.511 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.511 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:34:11.511 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:44.992766+0000", 2024-02-09T17:34:11.512 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1540000000000002e-06 2024-02-09T17:34:11.512 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.512 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:34:11.512 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:44.992768+0000", 2024-02-09T17:34:11.512 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7379999999999999e-06 2024-02-09T17:34:11.512 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.512 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:11.512 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:45.004640+0000", 2024-02-09T17:34:11.512 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011871246 2024-02-09T17:34:11.512 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.513 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for missing object", 2024-02-09T17:34:11.513 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:45.004713+0000", 2024-02-09T17:34:11.513 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3804e-05 2024-02-09T17:34:11.513 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.513 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.513 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:11.513 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:49.664827+0000", 2024-02-09T17:34:11.513 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6601140020000003 2024-02-09T17:34:11.513 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.513 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.513 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:34:11.513 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:49.664843+0000", 2024-02-09T17:34:11.514 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5396e-05 2024-02-09T17:34:11.514 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:11.514 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:34:11.514 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:11.514 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.514 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.514 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:204180 3.3 3:c0ceb3da:::benchmark_data_smithi179_90440_object72221:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e277)", 2024-02-09T17:34:11.514 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:33:44.992772+0000", 2024-02-09T17:34:11.514 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 26.382909694999999, 2024-02-09T17:34:11.514 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6746478590000002, 2024-02-09T17:34:11.514 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:34:11.514 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-09T17:34:11.515 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:34:11.515 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:34:11.515 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:34:11.515 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 204180 2024-02-09T17:34:11.515 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.515 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:34:11.515 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.515 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:34:11.515 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:44.992772+0000", 2024-02-09T17:34:11.515 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:11.515 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.515 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.515 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:34:11.516 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:44.992772+0000", 2024-02-09T17:34:11.516 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:11.516 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.516 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.516 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:34:11.516 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:44.992772+0000", 2024-02-09T17:34:11.516 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.04e-07 2024-02-09T17:34:11.516 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.516 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.516 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:34:11.516 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:44.992776+0000", 2024-02-09T17:34:11.516 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8600000000000003e-06 2024-02-09T17:34:11.517 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.517 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.517 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:34:11.517 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:44.992780+0000", 2024-02-09T17:34:11.517 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3549999999999998e-06 2024-02-09T17:34:11.517 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.517 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.517 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:34:11.517 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:44.992783+0000", 2024-02-09T17:34:11.517 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.399e-06 2024-02-09T17:34:11.517 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.517 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.517 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:11.517 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:45.004367+0000", 2024-02-09T17:34:11.518 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011583515000000001 2024-02-09T17:34:11.518 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.518 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.518 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for missing object", 2024-02-09T17:34:11.518 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:45.004446+0000", 2024-02-09T17:34:11.518 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9504000000000003e-05 2024-02-09T17:34:11.518 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.518 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.518 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:11.518 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:49.667412+0000", 2024-02-09T17:34:11.518 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6629655210000003 2024-02-09T17:34:11.518 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:11.518 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:11.518 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:34:11.518 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:49.667419+0000", 2024-02-09T17:34:11.518 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5009999999999998e-06 2024-02-09T17:34:11.519 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:11.519 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:34:11.519 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:11.519 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:11.519 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:34:11.519 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:34:11.519 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:34:11.520 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:11 smithi099 ceph-mon[39420]: pgmap v813: 37 pgs: 1 active+remapped+backfill_toofull, 29 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+clean+scrubbing; 2.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 846 op/s; 4437/135909 objects degraded (3.265%); 2923/135909 objects misplaced (2.151%); 0 B/s, 155 objects/s recovering 2024-02-09T17:34:11.521 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:11 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 4437/142014 objects degraded (3.124%), 2 pgs degraded (PG_DEGRADED) 2024-02-09T17:34:11.521 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:11 smithi099 ceph-mon[39420]: osdmap e291: 12 total, 12 up, 12 in 2024-02-09T17:34:11.533 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:34:11.553 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:11.564 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:34:11.574 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:34:11.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:11.581 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:34:11.610 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:34:11.651 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:34:11.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:11 smithi179 ceph-mon[43975]: pgmap v813: 37 pgs: 1 active+remapped+backfill_toofull, 29 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+clean+scrubbing; 2.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 846 op/s; 4437/135909 objects degraded (3.265%); 2923/135909 objects misplaced (2.151%); 0 B/s, 155 objects/s recovering 2024-02-09T17:34:11.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:11 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 4437/142014 objects degraded (3.124%), 2 pgs degraded (PG_DEGRADED) 2024-02-09T17:34:11.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:11 smithi179 ceph-mon[43975]: osdmap e291: 12 total, 12 up, 12 in 2024-02-09T17:34:11.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:11 smithi162 ceph-mon[44304]: pgmap v813: 37 pgs: 1 active+remapped+backfill_toofull, 29 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+clean+scrubbing; 2.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 846 op/s; 4437/135909 objects degraded (3.265%); 2923/135909 objects misplaced (2.151%); 0 B/s, 155 objects/s recovering 2024-02-09T17:34:11.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:11 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 4437/142014 objects degraded (3.124%), 2 pgs degraded (PG_DEGRADED) 2024-02-09T17:34:11.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:11 smithi162 ceph-mon[44304]: osdmap e291: 12 total, 12 up, 12 in 2024-02-09T17:34:11.682 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:34:11.701 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:34:11.741 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:11.745 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:34:11.803 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:34:11.826 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:34:11.836 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:11.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:34:11.845 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:34:11.875 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:34:11.917 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:34:11.938 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:34:11.953 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:34:11.989 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:34:12.004 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:34:12.004 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:34:12.004 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:34:12.004 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:34:12.004 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:34:12.004 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:34:12.004 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:34:12.004 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:34:12.004 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:34:12.004 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:34:12.004 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:34:12.005 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:34:12.020 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:12.158 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:34:12.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:12.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:34:12.197 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-09T17:34:12.270 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:34:12.271 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set noscrub 2024-02-09T17:34:12.305 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:12.330 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:34:12.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:12.337 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:34:12.438 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:34:12.512 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:12.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:12 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3095664659' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:34:12.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:12 smithi179 ceph-mon[43975]: 3.9 scrub starts 2024-02-09T17:34:12.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:12 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:34:12.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:12 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:34:12.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:12 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3095664659' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:34:12.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:12 smithi162 ceph-mon[44304]: 3.9 scrub starts 2024-02-09T17:34:12.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:12 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:34:12.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:12 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:34:12.683 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:34:12.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:12 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3095664659' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:34:12.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:12 smithi099 ceph-mon[39420]: 3.9 scrub starts 2024-02-09T17:34:12.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:12 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:34:12.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:12 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:34:12.800 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:34:12.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:12.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:34:12.911 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:34:12.952 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:34:12.956 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:13.553 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:34:13.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:13.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:34:13.661 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:34:13.665 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:13 smithi099 ceph-mon[39420]: 3.9 scrub ok 2024-02-09T17:34:13.665 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:13 smithi099 ceph-mon[39420]: osdmap e292: 12 total, 12 up, 12 in 2024-02-09T17:34:13.665 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:13 smithi099 ceph-mon[39420]: pgmap v816: 37 pgs: 1 active+remapped+backfill_toofull, 30 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 828 op/s; 4437/131046 objects degraded (3.386%); 2689/131046 objects misplaced (2.052%); 0 B/s, 89 objects/s recovering 2024-02-09T17:34:13.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:13 smithi179 ceph-mon[43975]: 3.9 scrub ok 2024-02-09T17:34:13.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:13 smithi179 ceph-mon[43975]: osdmap e292: 12 total, 12 up, 12 in 2024-02-09T17:34:13.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:13 smithi179 ceph-mon[43975]: pgmap v816: 37 pgs: 1 active+remapped+backfill_toofull, 30 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 828 op/s; 4437/131046 objects degraded (3.386%); 2689/131046 objects misplaced (2.052%); 0 B/s, 89 objects/s recovering 2024-02-09T17:34:13.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:13 smithi162 ceph-mon[44304]: 3.9 scrub ok 2024-02-09T17:34:13.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:13 smithi162 ceph-mon[44304]: osdmap e292: 12 total, 12 up, 12 in 2024-02-09T17:34:13.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:13 smithi162 ceph-mon[44304]: pgmap v816: 37 pgs: 1 active+remapped+backfill_toofull, 30 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 828 op/s; 4437/131046 objects degraded (3.386%); 2689/131046 objects misplaced (2.052%); 0 B/s, 89 objects/s recovering 2024-02-09T17:34:13.711 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:13.764 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:34:13.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:13.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:34:13.879 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:34:13.912 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:13.944 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:34:13.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:13.955 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:34:14.056 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:34:14.090 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:14.118 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:34:14.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:14.120 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:34:14.221 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:34:14.256 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] 2024-02-09T17:34:14.256 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 1 1 2024-02-09T17:34:14.294 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:14.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:14 smithi179 ceph-mon[43975]: 1.14 deep-scrub starts 2024-02-09T17:34:14.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:14 smithi179 ceph-mon[43975]: 1.14 deep-scrub ok 2024-02-09T17:34:14.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:14 smithi162 ceph-mon[44304]: 1.14 deep-scrub starts 2024-02-09T17:34:14.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:14 smithi162 ceph-mon[44304]: 1.14 deep-scrub ok 2024-02-09T17:34:14.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:14 smithi099 ceph-mon[39420]: 1.14 deep-scrub starts 2024-02-09T17:34:14.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:14 smithi099 ceph-mon[39420]: 1.14 deep-scrub ok 2024-02-09T17:34:14.853 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:34:14.866 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:34:14.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:14.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:34:14.974 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:34:15.005 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:15.061 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:34:15.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:15.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:34:15.164 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:34:15.165 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:34:15.257 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:15.397 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:15 smithi162 ceph-mon[44304]: pgmap v817: 37 pgs: 1 active+remapped+backfill_toofull, 30 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 688 op/s; 4437/129609 objects degraded (3.423%); 2689/129609 objects misplaced (2.075%); 0 B/s, 89 objects/s recovering 2024-02-09T17:34:15.403 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:15 smithi099 ceph-mon[39420]: pgmap v817: 37 pgs: 1 active+remapped+backfill_toofull, 30 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 688 op/s; 4437/129609 objects degraded (3.423%); 2689/129609 objects misplaced (2.075%); 0 B/s, 89 objects/s recovering 2024-02-09T17:34:15.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:15 smithi179 ceph-mon[43975]: pgmap v817: 37 pgs: 1 active+remapped+backfill_toofull, 30 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 688 op/s; 4437/129609 objects degraded (3.423%); 2689/129609 objects misplaced (2.075%); 0 B/s, 89 objects/s recovering 2024-02-09T17:34:16.317 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is set 2024-02-09T17:34:16.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:16 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/677067717' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:34:16.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:16 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 4437/129609 objects degraded (3.423%), 2 pgs degraded (PG_DEGRADED) 2024-02-09T17:34:16.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:16 smithi179 ceph-mon[43975]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:34:16.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:16 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/677067717' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:34:16.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:16 smithi179 ceph-mon[43975]: osdmap e293: 12 total, 12 up, 12 in 2024-02-09T17:34:16.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:16 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/677067717' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:34:16.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:16 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 4437/129609 objects degraded (3.423%), 2 pgs degraded (PG_DEGRADED) 2024-02-09T17:34:16.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:16 smithi162 ceph-mon[44304]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:34:16.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:16 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/677067717' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:34:16.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:16 smithi162 ceph-mon[44304]: osdmap e293: 12 total, 12 up, 12 in 2024-02-09T17:34:16.678 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:34:16.678 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:34:16.689 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:16 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/677067717' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:34:16.689 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:16 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 4437/129609 objects degraded (3.423%), 2 pgs degraded (PG_DEGRADED) 2024-02-09T17:34:16.689 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:16 smithi099 ceph-mon[39420]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:34:16.689 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:16 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/677067717' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:34:16.689 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:16 smithi099 ceph-mon[39420]: osdmap e293: 12 total, 12 up, 12 in 2024-02-09T17:34:16.738 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:34:16.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:16.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:34:16.849 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:34:16.904 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:16.947 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:34:16.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:16.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:34:16.969 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:34:17.051 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:34:17.120 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:17.480 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-09T17:34:17.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:17 smithi099 ceph-mon[39420]: pgmap v819: 37 pgs: 1 active+remapped+backfill_toofull, 30 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 643 op/s; 4437/124314 objects degraded (3.569%); 2689/124314 objects misplaced (2.163%); 0 B/s, 93 objects/s recovering 2024-02-09T17:34:17.497 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:34:17.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:17.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:34:17.610 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:34:17.662 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:17.699 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:34:17.708 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:17 smithi162 ceph-mon[44304]: pgmap v819: 37 pgs: 1 active+remapped+backfill_toofull, 30 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 643 op/s; 4437/124314 objects degraded (3.569%); 2689/124314 objects misplaced (2.163%); 0 B/s, 93 objects/s recovering 2024-02-09T17:34:17.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:17.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:34:17.810 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:34:17.878 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:17.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:17 smithi179 ceph-mon[43975]: pgmap v819: 37 pgs: 1 active+remapped+backfill_toofull, 30 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 643 op/s; 4437/124314 objects degraded (3.569%); 2689/124314 objects misplaced (2.163%); 0 B/s, 93 objects/s recovering 2024-02-09T17:34:17.924 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:34:17.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:17.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:34:17.938 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:34:18.034 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:34:18.100 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:18.133 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:34:18.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:18.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:34:18.241 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:34:18.287 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:18.435 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:18 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:34:18.435 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:18 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:34:18.436 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:18 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4181757763' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-09T17:34:18.436 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:18 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-09T17:34:18.461 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.1 to 1 (10000) 2024-02-09T17:34:18.528 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:18 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:34:18.528 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:18 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:34:18.528 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:18 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4181757763' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-09T17:34:18.529 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:18 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-09T17:34:18.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:18 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:34:18.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:18 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:34:18.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:18 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4181757763' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-09T17:34:18.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:18 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-09T17:34:19.657 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:19 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-09T17:34:19.657 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:19 smithi162 ceph-mon[44304]: osdmap e294: 12 total, 12 up, 12 in 2024-02-09T17:34:19.657 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:19 smithi162 ceph-mon[44304]: pgmap v821: 37 pgs: 1 active+remapped+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 2.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 576 op/s; 4437/120273 objects degraded (3.689%); 2457/120273 objects misplaced (2.043%); 0 B/s, 90 objects/s recovering 2024-02-09T17:34:19.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:19 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-09T17:34:19.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:19 smithi099 ceph-mon[39420]: osdmap e294: 12 total, 12 up, 12 in 2024-02-09T17:34:19.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:19 smithi099 ceph-mon[39420]: pgmap v821: 37 pgs: 1 active+remapped+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 2.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 576 op/s; 4437/120273 objects degraded (3.689%); 2457/120273 objects misplaced (2.043%); 0 B/s, 90 objects/s recovering 2024-02-09T17:34:19.838 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-09T17:34:19.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:19 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-09T17:34:19.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:19 smithi179 ceph-mon[43975]: osdmap e294: 12 total, 12 up, 12 in 2024-02-09T17:34:19.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:19 smithi179 ceph-mon[43975]: pgmap v821: 37 pgs: 1 active+remapped+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 2.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 576 op/s; 4437/120273 objects degraded (3.689%); 2457/120273 objects misplaced (2.043%); 0 B/s, 90 objects/s recovering 2024-02-09T17:34:20.029 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:34:20.029 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:34:20.365 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:34:20.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:20.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:34:20.471 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:34:20.503 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:20.525 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:34:20.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:20.533 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:34:20.634 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:34:20.677 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:20.795 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-09T17:34:21.297 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:34:21.549 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:34:21.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:21.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:34:21.658 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:34:21.708 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:21.733 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:34:21.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:21.736 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:34:21.837 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:34:21.875 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:21.890 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:21 smithi162 ceph-mon[44304]: pgmap v822: 37 pgs: 1 active+remapped+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 2.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 704 op/s; 4437/114117 objects degraded (3.888%); 2457/114117 objects misplaced (2.153%); 0 B/s, 135 objects/s recovering 2024-02-09T17:34:21.890 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:21 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 4437/114117 objects degraded (3.888%), 2 pgs degraded (PG_DEGRADED) 2024-02-09T17:34:21.890 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:21 smithi162 ceph-mon[44304]: osdmap e295: 12 total, 12 up, 12 in 2024-02-09T17:34:21.898 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:34:21.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:21.901 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:34:21.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:21 smithi179 ceph-mon[43975]: pgmap v822: 37 pgs: 1 active+remapped+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 2.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 704 op/s; 4437/114117 objects degraded (3.888%); 2457/114117 objects misplaced (2.153%); 0 B/s, 135 objects/s recovering 2024-02-09T17:34:21.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:21 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 4437/114117 objects degraded (3.888%), 2 pgs degraded (PG_DEGRADED) 2024-02-09T17:34:21.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:21 smithi179 ceph-mon[43975]: osdmap e295: 12 total, 12 up, 12 in 2024-02-09T17:34:21.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:21 smithi099 ceph-mon[39420]: pgmap v822: 37 pgs: 1 active+remapped+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 2.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 704 op/s; 4437/114117 objects degraded (3.888%); 2457/114117 objects misplaced (2.153%); 0 B/s, 135 objects/s recovering 2024-02-09T17:34:21.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:21 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 4437/114117 objects degraded (3.888%), 2 pgs degraded (PG_DEGRADED) 2024-02-09T17:34:21.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:21 smithi099 ceph-mon[39420]: osdmap e295: 12 total, 12 up, 12 in 2024-02-09T17:34:22.001 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:34:22.064 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:23.439 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:34:23.439 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:34:23.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:23 smithi162 ceph-mon[44304]: pgmap v824: 37 pgs: 1 active+remapped+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 2.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 801 op/s; 4437/109152 objects degraded (4.065%); 2226/109152 objects misplaced (2.039%); 0 B/s, 82 objects/s recovering 2024-02-09T17:34:23.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:23 smithi179 ceph-mon[43975]: pgmap v824: 37 pgs: 1 active+remapped+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 2.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 801 op/s; 4437/109152 objects degraded (4.065%); 2226/109152 objects misplaced (2.039%); 0 B/s, 82 objects/s recovering 2024-02-09T17:34:24.000 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:23 smithi099 ceph-mon[39420]: pgmap v824: 37 pgs: 1 active+remapped+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 2.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 801 op/s; 4437/109152 objects degraded (4.065%); 2226/109152 objects misplaced (2.039%); 0 B/s, 82 objects/s recovering 2024-02-09T17:34:24.178 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-09T17:34:24.656 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:34:25.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:25 smithi162 ceph-mon[44304]: pgmap v825: 37 pgs: 1 active+remapped+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 2.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 694 op/s; 4437/107649 objects degraded (4.122%); 2226/107649 objects misplaced (2.068%); 0 B/s, 64 objects/s recovering 2024-02-09T17:34:25.998 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:25 smithi099 ceph-mon[39420]: pgmap v825: 37 pgs: 1 active+remapped+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 2.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 694 op/s; 4437/107649 objects degraded (4.122%); 2226/107649 objects misplaced (2.068%); 0 B/s, 64 objects/s recovering 2024-02-09T17:34:26.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:25 smithi179 ceph-mon[43975]: pgmap v825: 37 pgs: 1 active+remapped+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 2.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 694 op/s; 4437/107649 objects degraded (4.122%); 2226/107649 objects misplaced (2.068%); 0 B/s, 64 objects/s recovering 2024-02-09T17:34:26.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:26 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 4437/107649 objects degraded (4.122%), 2 pgs degraded (PG_DEGRADED) 2024-02-09T17:34:27.093 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:34:27.093 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:34:27.106 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:26 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 4437/107649 objects degraded (4.122%), 2 pgs degraded (PG_DEGRADED) 2024-02-09T17:34:27.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:26 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 4437/107649 objects degraded (4.122%), 2 pgs degraded (PG_DEGRADED) 2024-02-09T17:34:27.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:27 smithi162 ceph-mon[44304]: pgmap v826: 37 pgs: 1 active+remapped+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 2.1 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 760 op/s; 4437/101718 objects degraded (4.362%); 2226/101718 objects misplaced (2.188%); 0 B/s, 136 objects/s recovering 2024-02-09T17:34:27.851 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-09T17:34:27.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:27 smithi099 ceph-mon[39420]: pgmap v826: 37 pgs: 1 active+remapped+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 2.1 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 760 op/s; 4437/101718 objects degraded (4.362%); 2226/101718 objects misplaced (2.188%); 0 B/s, 136 objects/s recovering 2024-02-09T17:34:28.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:27 smithi179 ceph-mon[43975]: pgmap v826: 37 pgs: 1 active+remapped+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 2.1 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 760 op/s; 4437/101718 objects degraded (4.362%); 2226/101718 objects misplaced (2.188%); 0 B/s, 136 objects/s recovering 2024-02-09T17:34:28.317 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:34:29.551 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:34:29.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:29.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:34:29.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:29 smithi162 ceph-mon[44304]: pgmap v827: 37 pgs: 1 active+remapped+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 2.0 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 757 op/s; 4437/97539 objects degraded (4.549%); 1983/97539 objects misplaced (2.033%); 0 B/s, 110 objects/s recovering 2024-02-09T17:34:29.825 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:34:29.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:29 smithi099 ceph-mon[39420]: pgmap v827: 37 pgs: 1 active+remapped+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 2.0 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 757 op/s; 4437/97539 objects degraded (4.549%); 1983/97539 objects misplaced (2.033%); 0 B/s, 110 objects/s recovering 2024-02-09T17:34:30.167 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:30.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:29 smithi179 ceph-mon[43975]: pgmap v827: 37 pgs: 1 active+remapped+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 2.0 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 757 op/s; 4437/97539 objects degraded (4.549%); 1983/97539 objects misplaced (2.033%); 0 B/s, 110 objects/s recovering 2024-02-09T17:34:30.442 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:34:30.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:30.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:34:30.455 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 2 1 2024-02-09T17:34:30.548 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:34:30.709 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:30.740 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:34:30.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:30.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:34:30.845 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:34:31.307 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:34:31.575 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:31.714 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:34:31.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:31.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:34:31.902 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:34:31.983 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:34:31.983 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:34:31.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:31 smithi099 ceph-mon[39420]: pgmap v828: 37 pgs: 1 active+remapped+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 1.9 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 762 op/s; 4437/91230 objects degraded (4.864%); 1983/91230 objects misplaced (2.174%); 0 B/s, 122 objects/s recovering 2024-02-09T17:34:31.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:31 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 4437/91230 objects degraded (4.864%), 2 pgs degraded (PG_DEGRADED) 2024-02-09T17:34:32.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:31 smithi179 ceph-mon[43975]: pgmap v828: 37 pgs: 1 active+remapped+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 1.9 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 762 op/s; 4437/91230 objects degraded (4.864%); 1983/91230 objects misplaced (2.174%); 0 B/s, 122 objects/s recovering 2024-02-09T17:34:32.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:31 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 4437/91230 objects degraded (4.864%), 2 pgs degraded (PG_DEGRADED) 2024-02-09T17:34:32.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:31 smithi162 ceph-mon[44304]: pgmap v828: 37 pgs: 1 active+remapped+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 1.9 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 762 op/s; 4437/91230 objects degraded (4.864%); 1983/91230 objects misplaced (2.174%); 0 B/s, 122 objects/s recovering 2024-02-09T17:34:32.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:31 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 4437/91230 objects degraded (4.864%), 2 pgs degraded (PG_DEGRADED) 2024-02-09T17:34:32.839 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:32.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:32 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:34:33.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:32 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:34:33.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:32 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:34:33.820 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:34:33.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:33.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:34:33.928 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:34:33.988 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:33.988 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:33 smithi162 ceph-mon[44304]: pgmap v829: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 1.8 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 803 op/s; 4437/86115 objects degraded (5.152%); 1735/86115 objects misplaced (2.015%); 0 B/s, 108 objects/s recovering 2024-02-09T17:34:33.988 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:33 smithi162 ceph-mon[44304]: osdmap e296: 12 total, 12 up, 12 in 2024-02-09T17:34:33.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:33 smithi099 ceph-mon[39420]: pgmap v829: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 1.8 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 803 op/s; 4437/86115 objects degraded (5.152%); 1735/86115 objects misplaced (2.015%); 0 B/s, 108 objects/s recovering 2024-02-09T17:34:33.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:33 smithi099 ceph-mon[39420]: osdmap e296: 12 total, 12 up, 12 in 2024-02-09T17:34:34.025 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:34:34.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:34.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:34:34.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:33 smithi179 ceph-mon[43975]: pgmap v829: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 1.8 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 803 op/s; 4437/86115 objects degraded (5.152%); 1735/86115 objects misplaced (2.015%); 0 B/s, 108 objects/s recovering 2024-02-09T17:34:34.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:33 smithi179 ceph-mon[43975]: osdmap e296: 12 total, 12 up, 12 in 2024-02-09T17:34:34.216 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:34:34.713 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:36.087 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:35 smithi099 ceph-mon[39420]: pgmap v831: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 1.7 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 764 op/s; 4437/84708 objects degraded (5.238%); 1735/84708 objects misplaced (2.048%); 0 B/s, 122 objects/s recovering 2024-02-09T17:34:36.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:35 smithi179 ceph-mon[43975]: pgmap v831: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 1.7 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 764 op/s; 4437/84708 objects degraded (5.238%); 1735/84708 objects misplaced (2.048%); 0 B/s, 122 objects/s recovering 2024-02-09T17:34:36.225 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:35 smithi162 ceph-mon[44304]: pgmap v831: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 1.7 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 764 op/s; 4437/84708 objects degraded (5.238%); 1735/84708 objects misplaced (2.048%); 0 B/s, 122 objects/s recovering 2024-02-09T17:34:36.622 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:34:36.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:36.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:34:37.037 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:34:37.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:36 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 4437/84708 objects degraded (5.238%), 2 pgs degraded (PG_DEGRADED) 2024-02-09T17:34:37.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:36 smithi179 ceph-mon[43975]: osdmap e297: 12 total, 12 up, 12 in 2024-02-09T17:34:37.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:36 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 4437/84708 objects degraded (5.238%), 2 pgs degraded (PG_DEGRADED) 2024-02-09T17:34:37.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:36 smithi162 ceph-mon[44304]: osdmap e297: 12 total, 12 up, 12 in 2024-02-09T17:34:37.252 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:36 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 4437/84708 objects degraded (5.238%), 2 pgs degraded (PG_DEGRADED) 2024-02-09T17:34:37.252 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:36 smithi099 ceph-mon[39420]: osdmap e297: 12 total, 12 up, 12 in 2024-02-09T17:34:38.001 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:38.164 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:37 smithi162 ceph-mon[44304]: pgmap v833: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 774 op/s; 4437/78930 objects degraded (5.621%); 1735/78930 objects misplaced (2.198%); 0 B/s, 156 objects/s recovering 2024-02-09T17:34:38.164 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:37 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1390623729' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:34:38.171 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:34:38.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:37 smithi179 ceph-mon[43975]: pgmap v833: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 774 op/s; 4437/78930 objects degraded (5.621%); 1735/78930 objects misplaced (2.198%); 0 B/s, 156 objects/s recovering 2024-02-09T17:34:38.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:37 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1390623729' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:34:38.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:38.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:34:38.202 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-09T17:34:38.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:37 smithi099 ceph-mon[39420]: pgmap v833: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 774 op/s; 4437/78930 objects degraded (5.621%); 1735/78930 objects misplaced (2.198%); 0 B/s, 156 objects/s recovering 2024-02-09T17:34:38.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:37 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1390623729' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:34:38.283 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:34:38.311 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:38.330 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:34:38.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:38.333 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:34:38.434 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:34:38.481 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:34:38.481 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:34:38.481 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:34:38.481 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:34:38.481 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:34:38.481 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:34:38.481 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:34:38.481 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:34:38.481 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:34:38.481 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:34:38.481 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:34:38.482 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:34:38.482 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:34:38.482 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:34:38.482 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:34:38.482 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:34:38.482 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:34:38.482 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:34:38.482 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:34:38.482 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:34:38.482 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:34:38.482 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:34:38.483 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:34:38.483 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:34:38.491 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:38.521 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:34:38.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:38.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:34:38.626 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:34:38.758 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:38 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:34:38.758 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:38 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:34:38.758 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:38 smithi162 ceph-mon[44304]: osdmap e298: 12 total, 12 up, 12 in 2024-02-09T17:34:38.758 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:38 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:34:38.799 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:39.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:38 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:34:39.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:38 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:34:39.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:38 smithi179 ceph-mon[43975]: osdmap e298: 12 total, 12 up, 12 in 2024-02-09T17:34:39.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:38 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:34:39.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:38 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:34:39.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:38 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:34:39.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:38 smithi099 ceph-mon[39420]: osdmap e298: 12 total, 12 up, 12 in 2024-02-09T17:34:39.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:38 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:34:40.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:39 smithi179 ceph-mon[43975]: pgmap v835: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 1.5 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 643 op/s; 4437/74532 objects degraded (5.953%); 1487/74532 objects misplaced (1.995%); 0 B/s, 101 objects/s recovering 2024-02-09T17:34:40.174 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:34:40.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:39 smithi162 ceph-mon[44304]: pgmap v835: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 1.5 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 643 op/s; 4437/74532 objects degraded (5.953%); 1487/74532 objects misplaced (1.995%); 0 B/s, 101 objects/s recovering 2024-02-09T17:34:40.179 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:39 smithi099 ceph-mon[39420]: pgmap v835: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 1.5 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 643 op/s; 4437/74532 objects degraded (5.953%); 1487/74532 objects misplaced (1.995%); 0 B/s, 101 objects/s recovering 2024-02-09T17:34:40.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:40.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:34:40.281 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:34:40.322 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:42.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:41 smithi179 ceph-mon[43975]: pgmap v836: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 767 op/s; 4437/68007 objects degraded (6.524%); 1487/68007 objects misplaced (2.187%); 0 B/s, 157 objects/s recovering 2024-02-09T17:34:42.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:41 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 4437/68007 objects degraded (6.524%), 2 pgs degraded (PG_DEGRADED) 2024-02-09T17:34:42.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:41 smithi179 ceph-mon[43975]: osdmap e299: 12 total, 12 up, 12 in 2024-02-09T17:34:42.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:41 smithi162 ceph-mon[44304]: pgmap v836: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 767 op/s; 4437/68007 objects degraded (6.524%); 1487/68007 objects misplaced (2.187%); 0 B/s, 157 objects/s recovering 2024-02-09T17:34:42.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:41 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 4437/68007 objects degraded (6.524%), 2 pgs degraded (PG_DEGRADED) 2024-02-09T17:34:42.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:41 smithi162 ceph-mon[44304]: osdmap e299: 12 total, 12 up, 12 in 2024-02-09T17:34:42.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:41 smithi099 ceph-mon[39420]: pgmap v836: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 767 op/s; 4437/68007 objects degraded (6.524%); 1487/68007 objects misplaced (2.187%); 0 B/s, 157 objects/s recovering 2024-02-09T17:34:42.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:41 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 4437/68007 objects degraded (6.524%), 2 pgs degraded (PG_DEGRADED) 2024-02-09T17:34:42.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:41 smithi099 ceph-mon[39420]: osdmap e299: 12 total, 12 up, 12 in 2024-02-09T17:34:43.294 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:34:43.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:43.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:34:43.406 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:34:43.436 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:43.462 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:34:43.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:43.468 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:34:43.568 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:34:43.611 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:43.913 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:43 smithi099 ceph-mon[39420]: pgmap v838: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 1.3 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 850 op/s; 4437/62823 objects degraded (7.063%); 1232/62823 objects misplaced (1.961%); 0 B/s, 99 objects/s recovering 2024-02-09T17:34:43.914 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:34:43.933 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:34:43.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:43.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:34:44.037 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:34:44.079 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:44.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:43 smithi179 ceph-mon[43975]: pgmap v838: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 1.3 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 850 op/s; 4437/62823 objects degraded (7.063%); 1232/62823 objects misplaced (1.961%); 0 B/s, 99 objects/s recovering 2024-02-09T17:34:44.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:43 smithi162 ceph-mon[44304]: pgmap v838: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 1.3 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 850 op/s; 4437/62823 objects degraded (7.063%); 1232/62823 objects misplaced (1.961%); 0 B/s, 99 objects/s recovering 2024-02-09T17:34:44.226 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:34:44.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:44.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:34:44.330 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:34:44.359 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:44.380 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:34:44.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:44.383 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:34:44.484 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:34:44.516 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:44.545 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:34:44.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:44.549 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:34:44.650 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:34:44.681 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:44.697 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:34:44.708 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:34:44.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:44.716 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:34:44.817 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:34:44.871 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:45.088 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:34:45.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:45.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:34:45.193 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:34:45.223 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:45.249 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:34:45.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:45.252 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:34:45.352 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:34:45.394 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:45.956 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:34:45.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:45 smithi099 ceph-mon[39420]: pgmap v839: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 1.2 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 734 op/s; 4437/61308 objects degraded (7.237%); 1232/61308 objects misplaced (2.010%); 0 B/s, 78 objects/s recovering 2024-02-09T17:34:45.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:45 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:34:45.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:45 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:34:45.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:45 smithi099 ceph-mon[39420]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:34:45.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:45 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3277791248' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:34:45.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:45 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:34:46.110 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is set 2024-02-09T17:34:46.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:45 smithi179 ceph-mon[43975]: pgmap v839: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 1.2 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 734 op/s; 4437/61308 objects degraded (7.237%); 1232/61308 objects misplaced (2.010%); 0 B/s, 78 objects/s recovering 2024-02-09T17:34:46.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:45 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:34:46.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:45 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:34:46.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:45 smithi179 ceph-mon[43975]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:34:46.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:45 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3277791248' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:34:46.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:45 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:34:46.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:45 smithi162 ceph-mon[44304]: pgmap v839: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 1.2 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 734 op/s; 4437/61308 objects degraded (7.237%); 1232/61308 objects misplaced (2.010%); 0 B/s, 78 objects/s recovering 2024-02-09T17:34:46.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:45 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:34:46.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:45 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:34:46.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:45 smithi162 ceph-mon[44304]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:34:46.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:45 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3277791248' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:34:46.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:45 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:34:46.891 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:34:46.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:46.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:34:46.999 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:34:47.034 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:47.054 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:34:47.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:47.060 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:34:47.161 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:34:47.162 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:46 smithi162 ceph-mon[44304]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:34:47.162 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:46 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:34:47.163 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:46 smithi162 ceph-mon[44304]: osdmap e300: 12 total, 12 up, 12 in 2024-02-09T17:34:47.163 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:46 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 4437/61308 objects degraded (7.237%), 2 pgs degraded (PG_DEGRADED) 2024-02-09T17:34:47.163 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:46 smithi162 ceph-mon[44304]: osdmap e301: 12 total, 12 up, 12 in 2024-02-09T17:34:47.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:46 smithi179 ceph-mon[43975]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:34:47.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:46 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:34:47.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:46 smithi179 ceph-mon[43975]: osdmap e300: 12 total, 12 up, 12 in 2024-02-09T17:34:47.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:46 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 4437/61308 objects degraded (7.237%), 2 pgs degraded (PG_DEGRADED) 2024-02-09T17:34:47.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:46 smithi179 ceph-mon[43975]: osdmap e301: 12 total, 12 up, 12 in 2024-02-09T17:34:47.199 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:47.229 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:34:47.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:47.232 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:34:47.333 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:34:47.334 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:46 smithi099 ceph-mon[39420]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:34:47.334 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:46 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:34:47.334 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:46 smithi099 ceph-mon[39420]: osdmap e300: 12 total, 12 up, 12 in 2024-02-09T17:34:47.334 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:46 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 4437/61308 objects degraded (7.237%), 2 pgs degraded (PG_DEGRADED) 2024-02-09T17:34:47.334 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:46 smithi099 ceph-mon[39420]: osdmap e301: 12 total, 12 up, 12 in 2024-02-09T17:34:47.388 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:47.438 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:34:47.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:47.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:34:47.550 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:34:47.602 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:48.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:47 smithi179 ceph-mon[43975]: pgmap v842: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 702 op/s; 4437/55350 objects degraded (8.016%); 1232/55350 objects misplaced (2.226%); 0 B/s, 103 objects/s recovering 2024-02-09T17:34:48.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:47 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:34:48.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:47 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:34:48.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:47 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2624998632' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-09T17:34:48.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:47 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-09T17:34:48.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:47 smithi179 ceph-mon[43975]: osdmap e302: 12 total, 12 up, 12 in 2024-02-09T17:34:48.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:47 smithi162 ceph-mon[44304]: pgmap v842: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 702 op/s; 4437/55350 objects degraded (8.016%); 1232/55350 objects misplaced (2.226%); 0 B/s, 103 objects/s recovering 2024-02-09T17:34:48.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:47 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:34:48.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:47 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:34:48.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:47 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2624998632' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-09T17:34:48.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:47 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-09T17:34:48.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:47 smithi162 ceph-mon[44304]: osdmap e302: 12 total, 12 up, 12 in 2024-02-09T17:34:48.264 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:47 smithi099 ceph-mon[39420]: pgmap v842: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 702 op/s; 4437/55350 objects degraded (8.016%); 1232/55350 objects misplaced (2.226%); 0 B/s, 103 objects/s recovering 2024-02-09T17:34:48.264 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:47 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:34:48.264 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:47 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:34:48.264 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:47 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2624998632' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-09T17:34:48.264 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:47 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-09T17:34:48.265 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:47 smithi099 ceph-mon[39420]: osdmap e302: 12 total, 12 up, 12 in 2024-02-09T17:34:48.623 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.2 to 1 (10000) 2024-02-09T17:34:49.414 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:34:49.429 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:49.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:34:49.531 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:34:49.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:49 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-09T17:34:49.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:49 smithi179 ceph-mon[43975]: osdmap e303: 12 total, 12 up, 12 in 2024-02-09T17:34:49.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:49 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:34:49.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:49 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:34:49.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:49 smithi179 ceph-mon[43975]: pgmap v845: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 1.0 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 853 op/s; 4437/51063 objects degraded (8.689%); 1023/51063 objects misplaced (2.003%); 0 B/s, 154 objects/s recovering 2024-02-09T17:34:49.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:49 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-09T17:34:49.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:49 smithi162 ceph-mon[44304]: osdmap e303: 12 total, 12 up, 12 in 2024-02-09T17:34:49.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:49 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:34:49.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:49 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:34:49.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:49 smithi162 ceph-mon[44304]: pgmap v845: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 1.0 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 853 op/s; 4437/51063 objects degraded (8.689%); 1023/51063 objects misplaced (2.003%); 0 B/s, 154 objects/s recovering 2024-02-09T17:34:49.710 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:49 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-09T17:34:49.711 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:49 smithi099 ceph-mon[39420]: osdmap e303: 12 total, 12 up, 12 in 2024-02-09T17:34:49.711 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:49 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:34:49.711 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:49 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:34:49.711 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:49 smithi099 ceph-mon[39420]: pgmap v845: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 1.0 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 853 op/s; 4437/51063 objects degraded (8.689%); 1023/51063 objects misplaced (2.003%); 0 B/s, 154 objects/s recovering 2024-02-09T17:34:49.721 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:50.188 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:34:50.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:50.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:34:50.228 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 3 1 2024-02-09T17:34:50.296 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:34:50.351 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:50.382 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:34:50.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:50.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:34:50.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:50 smithi162 ceph-mon[44304]: osdmap e304: 12 total, 12 up, 12 in 2024-02-09T17:34:50.459 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:34:50.460 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:50 smithi099 ceph-mon[39420]: osdmap e304: 12 total, 12 up, 12 in 2024-02-09T17:34:50.486 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:34:50.582 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:50.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:50 smithi179 ceph-mon[43975]: osdmap e304: 12 total, 12 up, 12 in 2024-02-09T17:34:51.137 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:34:51.137 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-02-09T17:34:51.137 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-02-09T17:34:51.137 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-02-09T17:34:51.137 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.138 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14790.0:146946 2.7 2:e490c70e:::benchmark_data_smithi179_77138_object13538:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-02-09T17:34:51.138 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:26:33.894273+0000", 2024-02-09T17:34:51.138 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 496.68818016300003, 2024-02-09T17:34:51.138 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.053209737, 2024-02-09T17:34:51.138 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:34:51.138 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:34:51.138 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:34:51.138 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14790", 2024-02-09T17:34:51.138 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:34:51.138 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 146946 2024-02-09T17:34:51.138 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.138 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:34:51.138 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.138 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:34:51.139 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.894273+0000", 2024-02-09T17:34:51.139 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:51.139 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.139 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.139 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:34:51.139 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.894273+0000", 2024-02-09T17:34:51.139 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:51.139 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.139 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.139 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:34:51.139 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.894273+0000", 2024-02-09T17:34:51.139 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9999999999999998e-07 2024-02-09T17:34:51.139 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.139 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.139 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:34:51.140 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.894276+0000", 2024-02-09T17:34:51.140 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5809999999999999e-06 2024-02-09T17:34:51.140 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.140 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.140 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:34:51.140 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.894283+0000", 2024-02-09T17:34:51.140 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6279999999999999e-06 2024-02-09T17:34:51.140 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.140 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.140 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:34:51.140 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.894288+0000", 2024-02-09T17:34:51.140 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4530000000000001e-06 2024-02-09T17:34:51.140 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.141 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.141 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:51.141 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.898941+0000", 2024-02-09T17:34:51.141 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0046531790000000003 2024-02-09T17:34:51.141 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.141 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.141 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:34:51.141 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.899054+0000", 2024-02-09T17:34:51.141 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000112633 2024-02-09T17:34:51.141 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.141 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.141 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,10", 2024-02-09T17:34:51.141 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.899154+0000", 2024-02-09T17:34:51.141 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9679999999999994e-05 2024-02-09T17:34:51.141 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.142 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.142 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:34:51.142 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.903397+0000", 2024-02-09T17:34:51.142 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042431109999999999 2024-02-09T17:34:51.142 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.142 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.142 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:51.142 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.904895+0000", 2024-02-09T17:34:51.142 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014981040000000001 2024-02-09T17:34:51.142 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.142 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.142 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:51.142 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.925296+0000", 2024-02-09T17:34:51.142 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020401633999999998 2024-02-09T17:34:51.143 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.143 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.143 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:34:51.143 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.947434+0000", 2024-02-09T17:34:51.143 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.022137093 2024-02-09T17:34:51.143 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.143 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.143 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:34:51.143 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:26:33.947483+0000", 2024-02-09T17:34:51.143 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9240999999999998e-05 2024-02-09T17:34:51.143 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:51.143 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:34:51.143 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:51.144 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.144 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.144 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14790.0:238247 2.7 2:e7d6d383:::benchmark_data_smithi179_77138_object104839:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-02-09T17:34:51.144 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:28:22.825302+0000", 2024-02-09T17:34:51.144 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 387.75715098400002, 2024-02-09T17:34:51.144 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.073699464000000006, 2024-02-09T17:34:51.144 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:34:51.144 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:34:51.144 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:34:51.144 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14790", 2024-02-09T17:34:51.144 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:34:51.144 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 238247 2024-02-09T17:34:51.144 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.145 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:34:51.145 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.145 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:34:51.145 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:28:22.825302+0000", 2024-02-09T17:34:51.145 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:51.145 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.145 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.145 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:34:51.145 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:28:22.825302+0000", 2024-02-09T17:34:51.145 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:51.145 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.145 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.145 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:34:51.145 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:28:22.825303+0000", 2024-02-09T17:34:51.146 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2550000000000001e-06 2024-02-09T17:34:51.146 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.146 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.146 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:34:51.146 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:28:22.825309+0000", 2024-02-09T17:34:51.146 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1270000000000002e-06 2024-02-09T17:34:51.146 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.146 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.146 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:34:51.146 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:28:22.825320+0000", 2024-02-09T17:34:51.146 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1652e-05 2024-02-09T17:34:51.146 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.146 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.147 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:34:51.147 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:28:22.825327+0000", 2024-02-09T17:34:51.147 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.075e-06 2024-02-09T17:34:51.147 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.147 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.147 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:51.147 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:28:22.825380+0000", 2024-02-09T17:34:51.147 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3031000000000001e-05 2024-02-09T17:34:51.147 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.147 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.147 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:34:51.147 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:28:22.825469+0000", 2024-02-09T17:34:51.147 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8284999999999999e-05 2024-02-09T17:34:51.148 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.148 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.148 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,10", 2024-02-09T17:34:51.148 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:28:22.825542+0000", 2024-02-09T17:34:51.148 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3060999999999997e-05 2024-02-09T17:34:51.148 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.148 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.148 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:34:51.148 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:28:22.829347+0000", 2024-02-09T17:34:51.148 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0038056380000000001 2024-02-09T17:34:51.148 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.148 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.148 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:51.149 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:28:22.833994+0000", 2024-02-09T17:34:51.149 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0046462700000000001 2024-02-09T17:34:51.149 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.149 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.149 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:51.149 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:28:22.898898+0000", 2024-02-09T17:34:51.149 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.064904544999999994 2024-02-09T17:34:51.149 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.149 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.149 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:34:51.149 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:28:22.898959+0000", 2024-02-09T17:34:51.149 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0427000000000003e-05 2024-02-09T17:34:51.149 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.149 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.150 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:34:51.150 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:28:22.899002+0000", 2024-02-09T17:34:51.150 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3098e-05 2024-02-09T17:34:51.150 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:51.150 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:34:51.150 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:51.150 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.150 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.150 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14790.0:238250 2.7 2:e6fac0c4:::benchmark_data_smithi179_77138_object104842:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-02-09T17:34:51.150 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:28:22.829751+0000", 2024-02-09T17:34:51.150 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 387.75270223799998, 2024-02-09T17:34:51.150 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.092457308000000002, 2024-02-09T17:34:51.150 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:34:51.150 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:34:51.151 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:34:51.151 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14790", 2024-02-09T17:34:51.151 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:34:51.151 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 238250 2024-02-09T17:34:51.151 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.151 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:34:51.151 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.151 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:34:51.151 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:28:22.829751+0000", 2024-02-09T17:34:51.151 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:51.151 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.151 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.151 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:34:51.152 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:28:22.829751+0000", 2024-02-09T17:34:51.152 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:51.152 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.152 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.152 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:34:51.152 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:28:22.829752+0000", 2024-02-09T17:34:51.152 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2669999999999999e-06 2024-02-09T17:34:51.152 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.152 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.152 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:34:51.152 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:28:22.829758+0000", 2024-02-09T17:34:51.152 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3149999999999999e-06 2024-02-09T17:34:51.152 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.152 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.153 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:34:51.153 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:28:22.829771+0000", 2024-02-09T17:34:51.153 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3185e-05 2024-02-09T17:34:51.153 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.153 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.153 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:34:51.153 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:28:22.829780+0000", 2024-02-09T17:34:51.153 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1209999999999996e-06 2024-02-09T17:34:51.153 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.153 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.153 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:51.153 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:28:22.829847+0000", 2024-02-09T17:34:51.153 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7089999999999996e-05 2024-02-09T17:34:51.153 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.153 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.154 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:34:51.154 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:28:22.829926+0000", 2024-02-09T17:34:51.154 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9201000000000003e-05 2024-02-09T17:34:51.154 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.154 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.154 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,10", 2024-02-09T17:34:51.154 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:28:22.830444+0000", 2024-02-09T17:34:51.154 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000518075 2024-02-09T17:34:51.154 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.154 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.154 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:34:51.154 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:28:22.834182+0000", 2024-02-09T17:34:51.154 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0037381110000000001 2024-02-09T17:34:51.155 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.155 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.155 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:51.155 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:28:22.839430+0000", 2024-02-09T17:34:51.155 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0052479400000000004 2024-02-09T17:34:51.155 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.155 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.155 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:51.155 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:28:22.922128+0000", 2024-02-09T17:34:51.155 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.082697963999999999 2024-02-09T17:34:51.155 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.155 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.155 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:34:51.155 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:28:22.922169+0000", 2024-02-09T17:34:51.155 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0309999999999999e-05 2024-02-09T17:34:51.156 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.156 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.156 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:34:51.156 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:28:22.922208+0000", 2024-02-09T17:34:51.156 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9728999999999999e-05 2024-02-09T17:34:51.156 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:51.156 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:34:51.156 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:51.156 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.156 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.156 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.14790.0:238256 2.7 2:e73e60d4:::benchmark_data_smithi179_77138_object104848:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-02-09T17:34:51.156 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:28:22.834343+0000", 2024-02-09T17:34:51.157 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 387.74810986699998, 2024-02-09T17:34:51.157 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.088015511000000005, 2024-02-09T17:34:51.157 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:34:51.157 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:34:51.157 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:34:51.157 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.14790", 2024-02-09T17:34:51.157 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:34:51.157 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 238256 2024-02-09T17:34:51.157 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.157 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:34:51.157 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.157 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:34:51.157 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:28:22.834343+0000", 2024-02-09T17:34:51.158 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:51.158 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.158 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.158 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:34:51.158 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:28:22.834343+0000", 2024-02-09T17:34:51.158 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:51.158 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.158 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.158 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:34:51.158 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:28:22.834345+0000", 2024-02-09T17:34:51.158 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9190000000000002e-06 2024-02-09T17:34:51.158 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.158 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.159 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:34:51.159 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:28:22.834351+0000", 2024-02-09T17:34:51.159 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4739999999999996e-06 2024-02-09T17:34:51.159 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.159 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.159 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:34:51.159 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:28:22.834363+0000", 2024-02-09T17:34:51.159 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2186e-05 2024-02-09T17:34:51.159 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.159 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.159 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:34:51.159 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:28:22.834372+0000", 2024-02-09T17:34:51.159 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7450000000000001e-06 2024-02-09T17:34:51.159 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.160 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.160 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:51.160 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:28:22.834438+0000", 2024-02-09T17:34:51.160 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6173000000000005e-05 2024-02-09T17:34:51.160 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.160 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.160 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:34:51.160 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:28:22.834556+0000", 2024-02-09T17:34:51.160 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000117883 2024-02-09T17:34:51.160 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.160 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.160 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,10", 2024-02-09T17:34:51.160 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:28:22.834666+0000", 2024-02-09T17:34:51.160 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000110466 2024-02-09T17:34:51.161 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.161 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.161 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:34:51.161 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:28:22.839450+0000", 2024-02-09T17:34:51.161 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0047833579999999997 2024-02-09T17:34:51.161 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.161 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.161 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:51.161 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:28:22.843714+0000", 2024-02-09T17:34:51.161 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042643469999999999 2024-02-09T17:34:51.161 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.161 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.161 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:51.162 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:28:22.922278+0000", 2024-02-09T17:34:51.162 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.078564070999999999 2024-02-09T17:34:51.162 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.162 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.162 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:34:51.162 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:28:22.922321+0000", 2024-02-09T17:34:51.162 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2741000000000003e-05 2024-02-09T17:34:51.162 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.162 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.162 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:34:51.162 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:28:22.922359+0000", 2024-02-09T17:34:51.162 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8148000000000001e-05 2024-02-09T17:34:51.162 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:51.162 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:34:51.162 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:51.163 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.163 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.163 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:64949 3.a 3:55889991:::benchmark_data_smithi179_90440_object64948:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-02-09T17:34:51.163 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:30:34.845336+0000", 2024-02-09T17:34:51.163 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 255.73711767399999, 2024-02-09T17:34:51.163 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.055809939000000003, 2024-02-09T17:34:51.163 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:34:51.163 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:34:51.163 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:34:51.163 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:34:51.163 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:34:51.163 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 64949 2024-02-09T17:34:51.163 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.163 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:34:51.163 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.164 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:34:51.164 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:34.845336+0000", 2024-02-09T17:34:51.164 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:51.164 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.164 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.164 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:34:51.164 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:34.845336+0000", 2024-02-09T17:34:51.164 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:51.164 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.164 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.164 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:34:51.164 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:34.845337+0000", 2024-02-09T17:34:51.164 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.44e-06 2024-02-09T17:34:51.164 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.164 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.165 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:34:51.165 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:34.845372+0000", 2024-02-09T17:34:51.165 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5046999999999998e-05 2024-02-09T17:34:51.165 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.165 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.165 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:34:51.165 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:34.845383+0000", 2024-02-09T17:34:51.165 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1355999999999999e-05 2024-02-09T17:34:51.165 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.165 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.165 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:34:51.165 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:34.845391+0000", 2024-02-09T17:34:51.165 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7749999999999993e-06 2024-02-09T17:34:51.165 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.165 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.166 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:51.166 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:34.845448+0000", 2024-02-09T17:34:51.166 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6382000000000003e-05 2024-02-09T17:34:51.166 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.166 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.166 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:34:51.166 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:34.845572+0000", 2024-02-09T17:34:51.166 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000124826 2024-02-09T17:34:51.166 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.166 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.166 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:34:51.166 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:34.845674+0000", 2024-02-09T17:34:51.166 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000101921 2024-02-09T17:34:51.166 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.166 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.167 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:34:51.167 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:34.847457+0000", 2024-02-09T17:34:51.167 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017822479999999999 2024-02-09T17:34:51.167 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.167 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.167 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:51.167 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:34.850566+0000", 2024-02-09T17:34:51.167 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0031098639999999999 2024-02-09T17:34:51.167 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.167 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.167 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:51.167 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:34.901050+0000", 2024-02-09T17:34:51.167 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.050483282999999997 2024-02-09T17:34:51.167 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.167 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.168 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:34:51.168 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:34.901105+0000", 2024-02-09T17:34:51.168 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5238999999999997e-05 2024-02-09T17:34:51.168 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.168 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.168 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:34:51.168 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:34.901145+0000", 2024-02-09T17:34:51.168 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0558e-05 2024-02-09T17:34:51.168 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:51.168 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:34:51.168 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:51.168 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.168 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.168 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:64951 3.a 3:57e5ecc9:::benchmark_data_smithi179_90440_object64950:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-02-09T17:34:51.169 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:30:34.847534+0000", 2024-02-09T17:34:51.169 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 255.734919013, 2024-02-09T17:34:51.169 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.053837844000000003, 2024-02-09T17:34:51.169 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:34:51.169 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:34:51.169 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:34:51.169 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:34:51.169 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:34:51.169 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 64951 2024-02-09T17:34:51.169 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.169 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:34:51.169 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.169 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:34:51.169 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:34.847534+0000", 2024-02-09T17:34:51.169 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:51.170 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.170 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.170 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:34:51.170 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:34.847534+0000", 2024-02-09T17:34:51.170 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:51.170 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.170 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.170 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:34:51.170 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:34.847535+0000", 2024-02-09T17:34:51.170 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.147e-06 2024-02-09T17:34:51.170 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.170 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.170 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:34:51.171 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:34.847576+0000", 2024-02-09T17:34:51.171 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0389000000000002e-05 2024-02-09T17:34:51.171 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.171 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.171 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:34:51.171 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:34.847585+0000", 2024-02-09T17:34:51.171 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2129999999999997e-06 2024-02-09T17:34:51.171 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.171 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.171 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:34:51.171 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:34.847591+0000", 2024-02-09T17:34:51.171 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6409999999999999e-06 2024-02-09T17:34:51.171 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.172 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.172 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:51.172 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:34.847635+0000", 2024-02-09T17:34:51.172 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4067999999999999e-05 2024-02-09T17:34:51.172 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.172 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.172 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:34:51.172 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:34.847757+0000", 2024-02-09T17:34:51.172 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000122185 2024-02-09T17:34:51.172 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.172 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.172 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:34:51.172 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:34.847885+0000", 2024-02-09T17:34:51.173 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012774999999999999 2024-02-09T17:34:51.173 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.173 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.173 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:34:51.173 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:34.851878+0000", 2024-02-09T17:34:51.173 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0039933829999999997 2024-02-09T17:34:51.173 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.173 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.173 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:51.173 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:34.852372+0000", 2024-02-09T17:34:51.173 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00049374100000000001 2024-02-09T17:34:51.173 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.174 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.174 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:51.174 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:34.901276+0000", 2024-02-09T17:34:51.174 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.048904156999999997 2024-02-09T17:34:51.174 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.174 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.174 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:34:51.174 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:34.901335+0000", 2024-02-09T17:34:51.174 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8931999999999998e-05 2024-02-09T17:34:51.174 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.174 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.174 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:34:51.174 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:34.901372+0000", 2024-02-09T17:34:51.175 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7237999999999998e-05 2024-02-09T17:34:51.175 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:51.175 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:34:51.175 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:51.175 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.175 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.175 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:68330 3.a 3:5e3269b9:::benchmark_data_smithi179_90440_object68329:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-02-09T17:34:51.175 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:30:37.203724+0000", 2024-02-09T17:34:51.175 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 253.37872919599999, 2024-02-09T17:34:51.175 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.053426699000000001, 2024-02-09T17:34:51.175 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:34:51.175 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:34:51.175 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:34:51.176 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:34:51.176 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:34:51.176 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 68330 2024-02-09T17:34:51.176 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.176 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:34:51.176 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.176 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:34:51.176 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:37.203724+0000", 2024-02-09T17:34:51.176 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:51.176 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.176 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.176 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:34:51.176 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:37.203724+0000", 2024-02-09T17:34:51.176 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:51.177 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.177 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.177 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:34:51.177 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:37.203725+0000", 2024-02-09T17:34:51.177 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2699999999999998e-07 2024-02-09T17:34:51.177 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.177 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.177 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:34:51.177 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:37.203784+0000", 2024-02-09T17:34:51.177 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8817999999999998e-05 2024-02-09T17:34:51.177 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.177 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.177 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:34:51.177 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:37.203793+0000", 2024-02-09T17:34:51.178 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7359999999999996e-06 2024-02-09T17:34:51.178 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.178 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.178 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:34:51.178 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:37.203801+0000", 2024-02-09T17:34:51.178 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5490000000000001e-06 2024-02-09T17:34:51.178 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.178 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.178 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:51.178 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:37.205994+0000", 2024-02-09T17:34:51.178 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002193029 2024-02-09T17:34:51.178 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.178 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.178 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:34:51.179 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:37.206103+0000", 2024-02-09T17:34:51.179 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000108756 2024-02-09T17:34:51.179 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.179 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.179 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:34:51.179 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:37.206234+0000", 2024-02-09T17:34:51.179 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013149599999999999 2024-02-09T17:34:51.179 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.179 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.179 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:34:51.179 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:37.209138+0000", 2024-02-09T17:34:51.179 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0029040390000000002 2024-02-09T17:34:51.179 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.180 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.180 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:51.180 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:37.211062+0000", 2024-02-09T17:34:51.180 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0019236069999999999 2024-02-09T17:34:51.180 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.180 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.180 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:51.180 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:37.257051+0000", 2024-02-09T17:34:51.180 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.045988686000000001 2024-02-09T17:34:51.180 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.180 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.180 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:34:51.180 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:37.257111+0000", 2024-02-09T17:34:51.180 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0170999999999997e-05 2024-02-09T17:34:51.181 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.181 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.181 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:34:51.181 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:30:37.257151+0000", 2024-02-09T17:34:51.181 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9984999999999997e-05 2024-02-09T17:34:51.181 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:51.181 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:34:51.181 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:51.181 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.181 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.181 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:122107 3.a 3:5cfa21ab:::benchmark_data_smithi179_90440_object122106:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-02-09T17:34:51.181 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:31:23.038513+0000", 2024-02-09T17:34:51.181 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 207.54393977399999, 2024-02-09T17:34:51.181 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.053888977999999997, 2024-02-09T17:34:51.181 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:34:51.182 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:34:51.182 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:34:51.182 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:34:51.182 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:34:51.182 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 122107 2024-02-09T17:34:51.182 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.182 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:34:51.182 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.182 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:34:51.182 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.038513+0000", 2024-02-09T17:34:51.182 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:51.182 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.182 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.182 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:34:51.183 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.038513+0000", 2024-02-09T17:34:51.183 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:51.183 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.183 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.183 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:34:51.183 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.038515+0000", 2024-02-09T17:34:51.183 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.544e-06 2024-02-09T17:34:51.183 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.183 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.183 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:34:51.183 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.038564+0000", 2024-02-09T17:34:51.183 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9391000000000002e-05 2024-02-09T17:34:51.183 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.183 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.183 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:34:51.184 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.038576+0000", 2024-02-09T17:34:51.184 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1834999999999999e-05 2024-02-09T17:34:51.184 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.184 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.184 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:34:51.184 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.038584+0000", 2024-02-09T17:34:51.184 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4730000000000003e-06 2024-02-09T17:34:51.184 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.184 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.184 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:51.184 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.051678+0000", 2024-02-09T17:34:51.184 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013094058 2024-02-09T17:34:51.184 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.185 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.185 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:34:51.185 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.051805+0000", 2024-02-09T17:34:51.185 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012710900000000001 2024-02-09T17:34:51.185 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.185 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.185 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:34:51.185 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.052025+0000", 2024-02-09T17:34:51.185 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021998100000000001 2024-02-09T17:34:51.185 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.185 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.185 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:51.185 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.066561+0000", 2024-02-09T17:34:51.186 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0145361 2024-02-09T17:34:51.186 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.186 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.186 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:51.186 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.075035+0000", 2024-02-09T17:34:51.186 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0084742370000000008 2024-02-09T17:34:51.186 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.186 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.186 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:34:51.186 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.092278+0000", 2024-02-09T17:34:51.186 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.017242644000000001 2024-02-09T17:34:51.186 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.186 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.187 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:34:51.187 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.092344+0000", 2024-02-09T17:34:51.187 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5937000000000002e-05 2024-02-09T17:34:51.187 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.187 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.187 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:34:51.187 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.092402+0000", 2024-02-09T17:34:51.187 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8668999999999997e-05 2024-02-09T17:34:51.187 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:51.187 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:34:51.187 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:51.187 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.187 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.188 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:122112 3.a 3:58a5b51f:::benchmark_data_smithi179_90440_object122111:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-02-09T17:34:51.188 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:31:23.038598+0000", 2024-02-09T17:34:51.188 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 207.54385518999999, 2024-02-09T17:34:51.188 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.054077954999999997, 2024-02-09T17:34:51.188 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:34:51.188 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:34:51.188 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:34:51.188 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:34:51.188 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:34:51.188 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 122112 2024-02-09T17:34:51.188 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.188 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:34:51.189 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.189 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:34:51.189 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.038598+0000", 2024-02-09T17:34:51.189 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:51.189 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.189 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.189 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:34:51.189 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.038598+0000", 2024-02-09T17:34:51.189 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:51.189 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.189 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.189 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:34:51.189 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.038598+0000", 2024-02-09T17:34:51.189 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6499999999999999e-07 2024-02-09T17:34:51.189 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.190 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.190 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:34:51.190 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.038723+0000", 2024-02-09T17:34:51.190 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012455300000000001 2024-02-09T17:34:51.190 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.190 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.190 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:34:51.190 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.038739+0000", 2024-02-09T17:34:51.190 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5897999999999999e-05 2024-02-09T17:34:51.190 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.190 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.190 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:34:51.190 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.038745+0000", 2024-02-09T17:34:51.190 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3169999999999998e-06 2024-02-09T17:34:51.190 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.191 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.191 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:51.191 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.062443+0000", 2024-02-09T17:34:51.191 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.023697304999999998 2024-02-09T17:34:51.191 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.191 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.191 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:34:51.191 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.062578+0000", 2024-02-09T17:34:51.191 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013568 2024-02-09T17:34:51.191 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.191 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.191 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:34:51.191 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.062777+0000", 2024-02-09T17:34:51.192 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000198872 2024-02-09T17:34:51.192 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.192 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.192 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:51.192 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.074989+0000", 2024-02-09T17:34:51.192 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.01221186 2024-02-09T17:34:51.192 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.192 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.192 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:51.192 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.075074+0000", 2024-02-09T17:34:51.192 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5451000000000006e-05 2024-02-09T17:34:51.192 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.192 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.192 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:34:51.193 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.092567+0000", 2024-02-09T17:34:51.193 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.017492278 2024-02-09T17:34:51.193 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.193 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.193 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:34:51.193 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.092622+0000", 2024-02-09T17:34:51.193 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4917000000000003e-05 2024-02-09T17:34:51.193 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.193 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.193 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:34:51.193 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.092676+0000", 2024-02-09T17:34:51.193 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4358999999999999e-05 2024-02-09T17:34:51.193 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:51.193 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:34:51.194 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:51.194 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.194 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.194 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:122123 3.a 3:5cda48e6:::benchmark_data_smithi179_90440_object122122:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-02-09T17:34:51.194 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:31:23.052971+0000", 2024-02-09T17:34:51.194 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 207.529482632, 2024-02-09T17:34:51.194 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.060830658000000003, 2024-02-09T17:34:51.194 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:34:51.194 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:34:51.194 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:34:51.194 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:34:51.194 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:34:51.194 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 122123 2024-02-09T17:34:51.195 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.195 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:34:51.195 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.195 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:34:51.195 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.052971+0000", 2024-02-09T17:34:51.195 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:51.195 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.195 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.195 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:34:51.195 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.052971+0000", 2024-02-09T17:34:51.195 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:51.195 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.195 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.195 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:34:51.196 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.052972+0000", 2024-02-09T17:34:51.196 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5460000000000001e-06 2024-02-09T17:34:51.196 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.196 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.196 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:34:51.196 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.053025+0000", 2024-02-09T17:34:51.196 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3371000000000001e-05 2024-02-09T17:34:51.196 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.196 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.196 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:34:51.196 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.053038+0000", 2024-02-09T17:34:51.196 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2567e-05 2024-02-09T17:34:51.196 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.196 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.197 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:34:51.197 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.053047+0000", 2024-02-09T17:34:51.197 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0399999999999998e-06 2024-02-09T17:34:51.197 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.197 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.197 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:51.197 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.075110+0000", 2024-02-09T17:34:51.197 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.022062604999999999 2024-02-09T17:34:51.197 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.197 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.197 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:34:51.197 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.075342+0000", 2024-02-09T17:34:51.197 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023209500000000001 2024-02-09T17:34:51.197 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.197 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.197 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:34:51.198 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.075513+0000", 2024-02-09T17:34:51.198 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000170908 2024-02-09T17:34:51.198 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.198 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.198 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:51.198 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.087832+0000", 2024-02-09T17:34:51.198 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012318811000000001 2024-02-09T17:34:51.198 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.198 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.198 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:34:51.198 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.092772+0000", 2024-02-09T17:34:51.198 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0049404649999999998 2024-02-09T17:34:51.198 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.198 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.199 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:51.199 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.113718+0000", 2024-02-09T17:34:51.199 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020945584 2024-02-09T17:34:51.199 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.199 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.199 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:34:51.199 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.113760+0000", 2024-02-09T17:34:51.199 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2583000000000002e-05 2024-02-09T17:34:51.199 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.199 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.199 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:34:51.199 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.113801+0000", 2024-02-09T17:34:51.199 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1083e-05 2024-02-09T17:34:51.199 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:51.199 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:34:51.200 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:51.200 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.200 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.200 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:122125 3.a 3:538e0ca3:::benchmark_data_smithi179_90440_object122124:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-02-09T17:34:51.200 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:31:23.054723+0000", 2024-02-09T17:34:51.200 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 207.52773060600001, 2024-02-09T17:34:51.200 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.059231450999999997, 2024-02-09T17:34:51.200 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:34:51.200 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:34:51.200 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:34:51.200 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:34:51.200 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:34:51.200 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 122125 2024-02-09T17:34:51.200 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.201 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:34:51.201 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.201 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:34:51.201 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.054723+0000", 2024-02-09T17:34:51.201 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:51.201 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.201 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.201 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:34:51.201 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.054723+0000", 2024-02-09T17:34:51.201 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:51.201 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.201 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.201 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:34:51.201 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.054724+0000", 2024-02-09T17:34:51.201 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6670000000000001e-06 2024-02-09T17:34:51.202 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.202 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.202 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:34:51.202 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.054756+0000", 2024-02-09T17:34:51.202 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1962999999999999e-05 2024-02-09T17:34:51.202 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.202 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.202 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:34:51.202 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.054774+0000", 2024-02-09T17:34:51.202 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7566000000000001e-05 2024-02-09T17:34:51.202 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.202 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.202 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:34:51.203 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.054783+0000", 2024-02-09T17:34:51.203 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4930000000000004e-06 2024-02-09T17:34:51.203 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.203 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.203 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:51.203 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.079291+0000", 2024-02-09T17:34:51.203 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.024507536999999999 2024-02-09T17:34:51.203 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.203 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.203 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:34:51.203 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.079452+0000", 2024-02-09T17:34:51.203 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000161416 2024-02-09T17:34:51.203 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.203 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.203 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:34:51.204 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.079610+0000", 2024-02-09T17:34:51.204 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015788799999999999 2024-02-09T17:34:51.204 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.204 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.204 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:51.204 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.087864+0000", 2024-02-09T17:34:51.204 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0082539429999999997 2024-02-09T17:34:51.204 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.204 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.204 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:34:51.204 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.092778+0000", 2024-02-09T17:34:51.204 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0049136789999999998 2024-02-09T17:34:51.204 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.205 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.205 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:51.205 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.113851+0000", 2024-02-09T17:34:51.205 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.021073229999999998 2024-02-09T17:34:51.205 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.205 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.205 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:34:51.205 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.113905+0000", 2024-02-09T17:34:51.205 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4117999999999999e-05 2024-02-09T17:34:51.205 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.205 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.205 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:34:51.205 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:23.113954+0000", 2024-02-09T17:34:51.205 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8950999999999999e-05 2024-02-09T17:34:51.206 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:51.206 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:34:51.206 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:51.206 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.206 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.206 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:140334 3.a 3:57a58b5b:::benchmark_data_smithi179_90440_object8375:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-02-09T17:34:51.206 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:31:58.130916+0000", 2024-02-09T17:34:51.206 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 172.45153752499999, 2024-02-09T17:34:51.206 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.057673296999999998, 2024-02-09T17:34:51.206 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:34:51.206 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:34:51.206 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:34:51.206 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:34:51.206 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:34:51.207 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 140334 2024-02-09T17:34:51.207 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.207 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:34:51.207 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.207 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:34:51.207 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:58.130916+0000", 2024-02-09T17:34:51.207 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:51.207 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.207 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.207 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:34:51.207 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:58.130916+0000", 2024-02-09T17:34:51.207 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:51.207 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.207 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.208 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:34:51.208 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:58.130917+0000", 2024-02-09T17:34:51.208 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8139999999999999e-06 2024-02-09T17:34:51.208 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.208 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.208 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:34:51.208 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:58.130924+0000", 2024-02-09T17:34:51.208 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1709999999999999e-06 2024-02-09T17:34:51.208 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.208 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.208 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:34:51.208 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:58.130937+0000", 2024-02-09T17:34:51.208 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2886e-05 2024-02-09T17:34:51.208 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.209 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.209 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:34:51.209 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:58.130945+0000", 2024-02-09T17:34:51.209 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3990000000000004e-06 2024-02-09T17:34:51.209 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.209 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.209 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:51.209 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:58.130999+0000", 2024-02-09T17:34:51.209 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3940999999999997e-05 2024-02-09T17:34:51.209 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.209 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.209 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:34:51.209 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:58.131130+0000", 2024-02-09T17:34:51.209 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013112900000000001 2024-02-09T17:34:51.210 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.210 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.210 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:34:51.210 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:58.131250+0000", 2024-02-09T17:34:51.210 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000119759 2024-02-09T17:34:51.210 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.210 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.210 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:34:51.210 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:58.139661+0000", 2024-02-09T17:34:51.210 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.008411066 2024-02-09T17:34:51.210 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.210 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.210 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:51.210 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:58.146987+0000", 2024-02-09T17:34:51.210 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0073262620000000001 2024-02-09T17:34:51.211 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.211 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.211 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:51.211 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:58.188482+0000", 2024-02-09T17:34:51.211 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.041494910000000003 2024-02-09T17:34:51.211 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.211 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.211 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:34:51.211 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:58.188546+0000", 2024-02-09T17:34:51.211 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3738000000000005e-05 2024-02-09T17:34:51.211 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.211 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.211 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:34:51.211 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:58.188589+0000", 2024-02-09T17:34:51.211 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3222000000000001e-05 2024-02-09T17:34:51.211 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:51.212 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:34:51.212 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:51.212 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.212 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.212 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:140338 3.a 3:582577a9:::benchmark_data_smithi179_90440_object8379:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-02-09T17:34:51.212 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:31:58.133876+0000", 2024-02-09T17:34:51.212 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 172.448577489, 2024-02-09T17:34:51.212 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.063732062000000006, 2024-02-09T17:34:51.212 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:34:51.212 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:34:51.212 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:34:51.212 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:34:51.212 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:34:51.212 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 140338 2024-02-09T17:34:51.213 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.213 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:34:51.213 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.213 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:34:51.213 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:58.133876+0000", 2024-02-09T17:34:51.213 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:51.213 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.213 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.213 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:34:51.213 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:58.133876+0000", 2024-02-09T17:34:51.213 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:51.213 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.213 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.213 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:34:51.213 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:58.133878+0000", 2024-02-09T17:34:51.214 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.069e-06 2024-02-09T17:34:51.214 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.214 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.214 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:34:51.214 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:58.133889+0000", 2024-02-09T17:34:51.214 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1342000000000001e-05 2024-02-09T17:34:51.214 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.214 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.214 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:34:51.214 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:58.133905+0000", 2024-02-09T17:34:51.214 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6160000000000001e-05 2024-02-09T17:34:51.214 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.214 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.214 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:34:51.214 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:58.133915+0000", 2024-02-09T17:34:51.215 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0049e-05 2024-02-09T17:34:51.215 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.215 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.215 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:51.215 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:58.134824+0000", 2024-02-09T17:34:51.215 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00090873899999999999 2024-02-09T17:34:51.215 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.215 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.215 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:34:51.215 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:58.134942+0000", 2024-02-09T17:34:51.215 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000117691 2024-02-09T17:34:51.215 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.215 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.215 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:34:51.215 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:58.135055+0000", 2024-02-09T17:34:51.215 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000112957 2024-02-09T17:34:51.216 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.216 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.216 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:34:51.216 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:58.143271+0000", 2024-02-09T17:34:51.216 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0082164600000000001 2024-02-09T17:34:51.216 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.216 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.216 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:51.216 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:58.152541+0000", 2024-02-09T17:34:51.216 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0092703219999999992 2024-02-09T17:34:51.216 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.216 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.216 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:51.216 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:58.197513+0000", 2024-02-09T17:34:51.217 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.044971480000000001 2024-02-09T17:34:51.217 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.217 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.217 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:34:51.217 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:58.197567+0000", 2024-02-09T17:34:51.217 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4373000000000003e-05 2024-02-09T17:34:51.217 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.217 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.217 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:34:51.217 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:58.197608+0000", 2024-02-09T17:34:51.217 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0420000000000003e-05 2024-02-09T17:34:51.217 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:51.217 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:34:51.217 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:51.218 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.218 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.218 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:205590 3.a 3:56eeeadb:::benchmark_data_smithi179_90440_object73631:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-02-09T17:34:51.218 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:33:47.276996+0000", 2024-02-09T17:34:51.218 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 63.305457464, 2024-02-09T17:34:51.218 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.059109983999999997, 2024-02-09T17:34:51.218 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:34:51.218 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:34:51.218 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:34:51.218 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:34:51.218 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:34:51.218 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 205590 2024-02-09T17:34:51.218 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.218 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:34:51.218 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.219 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:34:51.219 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:47.276996+0000", 2024-02-09T17:34:51.219 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:51.219 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.219 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.219 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:34:51.219 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:47.276996+0000", 2024-02-09T17:34:51.219 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:51.219 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.219 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.219 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:34:51.219 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:47.276997+0000", 2024-02-09T17:34:51.219 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.482e-06 2024-02-09T17:34:51.219 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.220 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.220 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:34:51.220 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:47.277002+0000", 2024-02-09T17:34:51.220 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0440000000000003e-06 2024-02-09T17:34:51.220 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.220 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.220 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:34:51.220 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:47.277012+0000", 2024-02-09T17:34:51.220 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9070000000000002e-06 2024-02-09T17:34:51.220 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.220 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.220 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:34:51.221 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:47.277020+0000", 2024-02-09T17:34:51.221 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4270000000000002e-06 2024-02-09T17:34:51.221 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.221 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.221 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:51.221 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:47.309086+0000", 2024-02-09T17:34:51.221 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.032066168999999999 2024-02-09T17:34:51.221 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.221 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.221 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:34:51.221 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:47.309214+0000", 2024-02-09T17:34:51.221 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012833699999999999 2024-02-09T17:34:51.221 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.222 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.222 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:34:51.222 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:47.309345+0000", 2024-02-09T17:34:51.222 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000131268 2024-02-09T17:34:51.222 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.222 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.222 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:34:51.222 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:47.324891+0000", 2024-02-09T17:34:51.222 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015545750000000001 2024-02-09T17:34:51.222 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.222 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.222 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:51.222 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:47.329964+0000", 2024-02-09T17:34:51.222 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0050732310000000001 2024-02-09T17:34:51.223 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.223 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.223 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:51.223 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:47.335998+0000", 2024-02-09T17:34:51.223 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.006033208 2024-02-09T17:34:51.223 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.223 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.223 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:34:51.223 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:47.336058+0000", 2024-02-09T17:34:51.223 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0708999999999998e-05 2024-02-09T17:34:51.223 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.223 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.223 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:34:51.223 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:47.336106+0000", 2024-02-09T17:34:51.224 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7451999999999999e-05 2024-02-09T17:34:51.224 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:51.224 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:34:51.224 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:51.224 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.224 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.224 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:216001 3.a 3:5f34e339:::benchmark_data_smithi179_90440_object84042:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-02-09T17:34:51.224 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:34:04.899802+0000", 2024-02-09T17:34:51.224 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 45.682651229000001, 2024-02-09T17:34:51.224 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.062031625999999999, 2024-02-09T17:34:51.224 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:34:51.224 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:34:51.224 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:34:51.224 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:34:51.225 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:34:51.225 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 216001 2024-02-09T17:34:51.225 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.225 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:34:51.225 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.225 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:34:51.225 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:04.899802+0000", 2024-02-09T17:34:51.225 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:51.225 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.225 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.225 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:34:51.225 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:04.899802+0000", 2024-02-09T17:34:51.225 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:51.226 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.226 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.226 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:34:51.226 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:04.899804+0000", 2024-02-09T17:34:51.226 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9640000000000002e-06 2024-02-09T17:34:51.226 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.226 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.226 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:34:51.226 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:04.899811+0000", 2024-02-09T17:34:51.226 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6120000000000004e-06 2024-02-09T17:34:51.226 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.226 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.226 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:34:51.227 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:04.899823+0000", 2024-02-09T17:34:51.227 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2918000000000001e-05 2024-02-09T17:34:51.227 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.227 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.227 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:34:51.227 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:04.899832+0000", 2024-02-09T17:34:51.227 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9849999999999995e-06 2024-02-09T17:34:51.227 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.227 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.227 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:51.227 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:04.907558+0000", 2024-02-09T17:34:51.227 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0077254569999999998 2024-02-09T17:34:51.227 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.228 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.228 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:34:51.228 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:04.907641+0000", 2024-02-09T17:34:51.228 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3325999999999995e-05 2024-02-09T17:34:51.228 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.228 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.228 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:34:51.228 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:04.907721+0000", 2024-02-09T17:34:51.228 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9401999999999996e-05 2024-02-09T17:34:51.228 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.228 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.228 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:51.228 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:04.913194+0000", 2024-02-09T17:34:51.229 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0054731989999999998 2024-02-09T17:34:51.229 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.229 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.229 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:51.229 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:04.936476+0000", 2024-02-09T17:34:51.229 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.023282251 2024-02-09T17:34:51.229 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.229 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.229 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:34:51.229 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:04.961710+0000", 2024-02-09T17:34:51.229 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.025233498 2024-02-09T17:34:51.229 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.229 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.229 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:34:51.230 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:04.961774+0000", 2024-02-09T17:34:51.230 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4554000000000006e-05 2024-02-09T17:34:51.230 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.230 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.230 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:34:51.230 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:04.961834+0000", 2024-02-09T17:34:51.230 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9460000000000003e-05 2024-02-09T17:34:51.230 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:51.230 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:34:51.230 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:51.230 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.230 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.231 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:216127 3.a 3:51c0fcf6:::benchmark_data_smithi179_90440_object84168:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-02-09T17:34:51.231 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:34:05.090000+0000", 2024-02-09T17:34:51.231 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 45.492452948, 2024-02-09T17:34:51.231 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.059167605999999998, 2024-02-09T17:34:51.231 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:34:51.231 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:34:51.231 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:34:51.231 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:34:51.231 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:34:51.231 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 216127 2024-02-09T17:34:51.231 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.231 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:34:51.231 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.232 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:34:51.232 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:05.090000+0000", 2024-02-09T17:34:51.232 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:51.232 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.232 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.232 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:34:51.232 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:05.090000+0000", 2024-02-09T17:34:51.232 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:51.232 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.232 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.232 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:34:51.232 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:05.090002+0000", 2024-02-09T17:34:51.232 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0719999999999998e-06 2024-02-09T17:34:51.232 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.233 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.233 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:34:51.233 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:05.090009+0000", 2024-02-09T17:34:51.233 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3080000000000004e-06 2024-02-09T17:34:51.233 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.233 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.233 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:34:51.233 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:05.090021+0000", 2024-02-09T17:34:51.233 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2495e-05 2024-02-09T17:34:51.233 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.234 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.234 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:34:51.234 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:05.090029+0000", 2024-02-09T17:34:51.234 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3110000000000006e-06 2024-02-09T17:34:51.234 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.234 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.234 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:51.234 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:05.104292+0000", 2024-02-09T17:34:51.234 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014262169999999999 2024-02-09T17:34:51.234 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.234 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.234 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:34:51.234 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:05.104430+0000", 2024-02-09T17:34:51.235 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000138629 2024-02-09T17:34:51.235 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.235 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.235 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:34:51.235 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:05.104571+0000", 2024-02-09T17:34:51.235 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014070599999999999 2024-02-09T17:34:51.235 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.235 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.235 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:51.235 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:05.117295+0000", 2024-02-09T17:34:51.235 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012724249999999999 2024-02-09T17:34:51.235 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.235 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.235 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:51.236 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:05.122306+0000", 2024-02-09T17:34:51.236 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0050103300000000003 2024-02-09T17:34:51.236 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.236 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.236 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:34:51.236 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:05.149046+0000", 2024-02-09T17:34:51.236 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.026740501 2024-02-09T17:34:51.236 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.236 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.236 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:34:51.236 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:05.149107+0000", 2024-02-09T17:34:51.236 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1198999999999995e-05 2024-02-09T17:34:51.236 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.236 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.237 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:34:51.237 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:05.149168+0000", 2024-02-09T17:34:51.237 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0634999999999998e-05 2024-02-09T17:34:51.237 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:51.237 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:34:51.237 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:51.237 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.237 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.237 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24817.0:216360 3.2 e288/199 3:432d3abf:::benchmark_data_smithi179_90440_object84401:head v 288'13631, mlcod=288'13626)", 2024-02-09T17:34:51.237 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:34:05.428350+0000", 2024-02-09T17:34:51.237 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 45.154103210000002, 2024-02-09T17:34:51.237 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.058082162, 2024-02-09T17:34:51.237 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:34:51.237 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:34:51.237 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:34:51.238 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.238 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:34:51.238 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:05.428350+0000", 2024-02-09T17:34:51.238 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:51.238 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.238 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.238 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:34:51.238 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:05.428350+0000", 2024-02-09T17:34:51.238 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:51.238 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.238 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.238 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:34:51.238 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:05.428350+0000", 2024-02-09T17:34:51.238 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5800000000000001e-07 2024-02-09T17:34:51.238 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.239 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.239 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:34:51.239 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:05.428354+0000", 2024-02-09T17:34:51.239 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7440000000000001e-06 2024-02-09T17:34:51.239 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.239 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.239 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:34:51.239 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:05.428361+0000", 2024-02-09T17:34:51.239 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7950000000000002e-06 2024-02-09T17:34:51.239 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.239 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.239 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:34:51.239 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:05.428366+0000", 2024-02-09T17:34:51.239 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0599999999999998e-06 2024-02-09T17:34:51.240 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.240 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.240 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:51.240 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:05.437544+0000", 2024-02-09T17:34:51.240 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0091779670000000004 2024-02-09T17:34:51.240 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.240 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.240 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:34:51.240 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:05.437570+0000", 2024-02-09T17:34:51.240 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6610000000000001e-05 2024-02-09T17:34:51.240 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.240 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.240 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:34:51.240 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:05.486381+0000", 2024-02-09T17:34:51.241 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.048810655000000001 2024-02-09T17:34:51.241 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.241 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.241 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:34:51.241 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:05.486432+0000", 2024-02-09T17:34:51.241 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1072999999999997e-05 2024-02-09T17:34:51.241 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:51.241 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:34:51.241 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:51.241 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.241 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.241 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:216380 3.a 3:552f2350:::benchmark_data_smithi179_90440_object84421:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-02-09T17:34:51.241 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:34:05.454885+0000", 2024-02-09T17:34:51.241 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 45.127568445000001, 2024-02-09T17:34:51.242 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.054765411, 2024-02-09T17:34:51.242 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:34:51.242 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:34:51.242 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:34:51.242 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:34:51.242 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:34:51.242 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 216380 2024-02-09T17:34:51.242 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.242 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:34:51.242 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.242 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:34:51.242 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:05.454885+0000", 2024-02-09T17:34:51.242 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:51.242 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.242 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.243 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:34:51.243 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:05.454885+0000", 2024-02-09T17:34:51.243 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:51.243 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.243 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.243 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:34:51.243 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:05.454886+0000", 2024-02-09T17:34:51.243 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7489999999999999e-06 2024-02-09T17:34:51.243 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.243 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.243 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:34:51.243 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:05.454892+0000", 2024-02-09T17:34:51.243 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8769999999999998e-06 2024-02-09T17:34:51.243 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.244 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.244 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:34:51.244 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:05.454906+0000", 2024-02-09T17:34:51.244 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3307e-05 2024-02-09T17:34:51.244 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.244 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.244 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:34:51.244 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:05.454914+0000", 2024-02-09T17:34:51.244 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5820000000000002e-06 2024-02-09T17:34:51.244 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.244 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.244 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:51.244 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:05.467091+0000", 2024-02-09T17:34:51.244 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012176592999999999 2024-02-09T17:34:51.244 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.245 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.245 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:34:51.245 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:05.467238+0000", 2024-02-09T17:34:51.245 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000147441 2024-02-09T17:34:51.245 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.245 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.245 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:34:51.245 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:05.467379+0000", 2024-02-09T17:34:51.245 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014093600000000001 2024-02-09T17:34:51.245 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.245 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.245 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:51.245 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:05.479691+0000", 2024-02-09T17:34:51.245 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012312251 2024-02-09T17:34:51.245 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.246 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.246 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:51.246 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:05.479728+0000", 2024-02-09T17:34:51.246 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6684999999999999e-05 2024-02-09T17:34:51.246 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.246 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.246 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:34:51.246 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:05.509525+0000", 2024-02-09T17:34:51.246 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.029796349999999999 2024-02-09T17:34:51.246 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.246 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.246 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:34:51.246 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:05.509587+0000", 2024-02-09T17:34:51.246 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2496000000000006e-05 2024-02-09T17:34:51.246 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.247 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.247 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:34:51.247 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:05.509650+0000", 2024-02-09T17:34:51.247 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3144000000000004e-05 2024-02-09T17:34:51.247 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:51.247 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:34:51.247 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:51.247 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.247 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.247 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:216977 3.a 3:5b72b66c:::benchmark_data_smithi179_90440_object85018:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-02-09T17:34:51.247 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:34:06.305292+0000", 2024-02-09T17:34:51.247 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 44.277161534999998, 2024-02-09T17:34:51.247 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.052574325999999998, 2024-02-09T17:34:51.248 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:34:51.248 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:34:51.248 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:34:51.248 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:34:51.248 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:34:51.248 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 216977 2024-02-09T17:34:51.248 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.248 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:34:51.248 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.248 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:34:51.248 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:06.305292+0000", 2024-02-09T17:34:51.248 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:51.248 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.248 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.249 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:34:51.249 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:06.305292+0000", 2024-02-09T17:34:51.249 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:51.249 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.249 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.249 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:34:51.249 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:06.305294+0000", 2024-02-09T17:34:51.249 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.638e-06 2024-02-09T17:34:51.249 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.249 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.249 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:34:51.249 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:06.305304+0000", 2024-02-09T17:34:51.249 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3039999999999994e-06 2024-02-09T17:34:51.249 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.249 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.250 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:34:51.250 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:06.305326+0000", 2024-02-09T17:34:51.250 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1923e-05 2024-02-09T17:34:51.250 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.250 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.250 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:34:51.250 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:06.305340+0000", 2024-02-09T17:34:51.250 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.471e-05 2024-02-09T17:34:51.250 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.250 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.250 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:51.250 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:06.307219+0000", 2024-02-09T17:34:51.250 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001878268 2024-02-09T17:34:51.251 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.251 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.251 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:34:51.251 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:06.307358+0000", 2024-02-09T17:34:51.251 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013926499999999999 2024-02-09T17:34:51.251 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.251 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.251 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:34:51.251 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:06.307485+0000", 2024-02-09T17:34:51.251 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000127331 2024-02-09T17:34:51.251 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.251 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.251 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:34:51.251 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:06.313847+0000", 2024-02-09T17:34:51.252 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0063617129999999997 2024-02-09T17:34:51.252 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.252 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.252 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:51.252 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:06.332451+0000", 2024-02-09T17:34:51.252 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018604404000000001 2024-02-09T17:34:51.252 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.252 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.252 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:51.252 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:06.357776+0000", 2024-02-09T17:34:51.252 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.025325251 2024-02-09T17:34:51.252 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.252 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.252 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:34:51.253 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:06.357830+0000", 2024-02-09T17:34:51.253 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3099000000000001e-05 2024-02-09T17:34:51.253 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.253 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.253 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:34:51.253 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:06.357866+0000", 2024-02-09T17:34:51.253 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.642e-05 2024-02-09T17:34:51.253 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:51.253 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:34:51.253 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:51.253 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.253 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.253 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:241041 3.a 3:5d702660:::benchmark_data_smithi179_90440_object109082:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-02-09T17:34:51.254 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:34:38.279302+0000", 2024-02-09T17:34:51.254 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 12.303150743, 2024-02-09T17:34:51.254 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.068551030999999998, 2024-02-09T17:34:51.254 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:34:51.254 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:34:51.254 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:34:51.254 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:34:51.254 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:34:51.254 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 241041 2024-02-09T17:34:51.254 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.254 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:34:51.254 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.254 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:34:51.255 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:38.279302+0000", 2024-02-09T17:34:51.255 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:51.255 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.255 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.255 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:34:51.255 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:38.279302+0000", 2024-02-09T17:34:51.255 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:34:51.255 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.255 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.255 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:34:51.255 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:38.279303+0000", 2024-02-09T17:34:51.255 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.41e-07 2024-02-09T17:34:51.255 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.255 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.255 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:34:51.256 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:38.279305+0000", 2024-02-09T17:34:51.256 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3089999999999998e-06 2024-02-09T17:34:51.256 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.256 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.256 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:34:51.256 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:38.279311+0000", 2024-02-09T17:34:51.256 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9750000000000004e-06 2024-02-09T17:34:51.256 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.256 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.256 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:34:51.256 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:38.279316+0000", 2024-02-09T17:34:51.256 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1209999999999998e-06 2024-02-09T17:34:51.256 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.256 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.257 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:34:51.257 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:38.288536+0000", 2024-02-09T17:34:51.257 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0092198320000000007 2024-02-09T17:34:51.257 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.257 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.257 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:34:51.257 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:38.288647+0000", 2024-02-09T17:34:51.257 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000110969 2024-02-09T17:34:51.257 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.257 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.257 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:34:51.257 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:38.288758+0000", 2024-02-09T17:34:51.257 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000110659 2024-02-09T17:34:51.257 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.258 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.258 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:34:51.258 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:38.309781+0000", 2024-02-09T17:34:51.258 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.021023276 2024-02-09T17:34:51.258 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.258 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.258 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:51.258 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:38.310137+0000", 2024-02-09T17:34:51.258 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00035563899999999999 2024-02-09T17:34:51.258 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.258 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.258 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:34:51.258 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:38.347756+0000", 2024-02-09T17:34:51.259 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.037619756999999997 2024-02-09T17:34:51.259 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.259 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.259 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:34:51.259 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:38.347818+0000", 2024-02-09T17:34:51.259 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1933999999999994e-05 2024-02-09T17:34:51.259 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:34:51.259 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:34:51.259 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:34:51.259 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:38.347853+0000", 2024-02-09T17:34:51.259 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5219000000000003e-05 2024-02-09T17:34:51.259 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:51.259 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:34:51.259 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:51.260 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:34:51.260 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:34:51.260 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:34:51.465 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:34:51.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:51.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:34:51.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:51 smithi099 ceph-mon[39420]: pgmap v847: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 930 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 827 op/s; 4437/44628 objects degraded (9.942%); 1023/44628 objects misplaced (2.292%); 0 B/s, 149 objects/s recovering 2024-02-09T17:34:51.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:51 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 4437/51063 objects degraded (8.689%), 2 pgs degraded (PG_DEGRADED) 2024-02-09T17:34:51.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:51 smithi099 ceph-mon[39420]: osdmap e305: 12 total, 12 up, 12 in 2024-02-09T17:34:51.517 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:34:51.571 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:34:51.602 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:51 smithi179 ceph-mon[43975]: pgmap v847: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 930 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 827 op/s; 4437/44628 objects degraded (9.942%); 1023/44628 objects misplaced (2.292%); 0 B/s, 149 objects/s recovering 2024-02-09T17:34:51.602 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:51 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 4437/51063 objects degraded (8.689%), 2 pgs degraded (PG_DEGRADED) 2024-02-09T17:34:51.602 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:51 smithi179 ceph-mon[43975]: osdmap e305: 12 total, 12 up, 12 in 2024-02-09T17:34:51.604 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:51.618 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:51 smithi162 ceph-mon[44304]: pgmap v847: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 930 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 827 op/s; 4437/44628 objects degraded (9.942%); 1023/44628 objects misplaced (2.292%); 0 B/s, 149 objects/s recovering 2024-02-09T17:34:51.618 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:51 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 4437/51063 objects degraded (8.689%), 2 pgs degraded (PG_DEGRADED) 2024-02-09T17:34:51.618 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:51 smithi162 ceph-mon[44304]: osdmap e305: 12 total, 12 up, 12 in 2024-02-09T17:34:51.636 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:34:51.647 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:51.647 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:34:51.748 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:34:51.804 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:51.835 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:34:51.840 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:51.840 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:34:51.941 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:34:52.023 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:52.049 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:34:52.051 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:52.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:34:52.059 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-09T17:34:52.153 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:34:52.184 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:52.204 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:34:52.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:52.212 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:34:52.313 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:34:52.367 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:52.463 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset noscrub 2024-02-09T17:34:52.547 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:34:52.558 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:34:52.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:52.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:34:52.672 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:34:52.748 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:52.817 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:34:52.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:52.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:34:52.927 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:34:52.982 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:53.666 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:53 smithi099 ceph-mon[39420]: pgmap v849: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 821 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 735 op/s; 4437/39384 objects degraded (11.266%); 766/39384 objects misplaced (1.945%); 0 B/s, 122 objects/s recovering 2024-02-09T17:34:53.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:53 smithi179 ceph-mon[43975]: pgmap v849: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 821 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 735 op/s; 4437/39384 objects degraded (11.266%); 766/39384 objects misplaced (1.945%); 0 B/s, 122 objects/s recovering 2024-02-09T17:34:53.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:53 smithi162 ceph-mon[44304]: pgmap v849: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 821 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 735 op/s; 4437/39384 objects degraded (11.266%); 766/39384 objects misplaced (1.945%); 0 B/s, 122 objects/s recovering 2024-02-09T17:34:54.553 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:34:54.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:54.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:34:54.660 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:34:54.716 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:54.987 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:34:55.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:55.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:34:55.107 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:34:55.186 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:55.472 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:34:55.472 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:34:55.476 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:34:55.514 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:34:55.551 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:34:55.587 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:34:55.630 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:34:55.670 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:34:55.703 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:34:55.743 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:34:55.782 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:34:55.816 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:34:55.819 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:34:55.819 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:34:55.819 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:34:55.819 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:34:55.819 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:34:55.819 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:34:55.819 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:34:55.819 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:34:55.819 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:34:55.820 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:34:55.820 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:34:55.820 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:34:55.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:55 smithi179 ceph-mon[43975]: pgmap v850: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 790 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 698 op/s; 4437/37899 objects degraded (11.707%); 766/37899 objects misplaced (2.021%); 0 B/s, 102 objects/s recovering 2024-02-09T17:34:55.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:55 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2629301261' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:34:55.932 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:55 smithi162 ceph-mon[44304]: pgmap v850: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 790 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 698 op/s; 4437/37899 objects degraded (11.707%); 766/37899 objects misplaced (2.021%); 0 B/s, 102 objects/s recovering 2024-02-09T17:34:55.932 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:55 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2629301261' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:34:55.970 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:34:55.970 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:34:55.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:55 smithi099 ceph-mon[39420]: pgmap v850: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 790 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 698 op/s; 4437/37899 objects degraded (11.707%); 766/37899 objects misplaced (2.021%); 0 B/s, 102 objects/s recovering 2024-02-09T17:34:55.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:55 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2629301261' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:34:56.353 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:34:56.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:56.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:34:56.463 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:34:56.511 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:56.675 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:56 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 4437/37899 objects degraded (11.707%), 2 pgs degraded (PG_DEGRADED) 2024-02-09T17:34:57.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:56 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 4437/37899 objects degraded (11.707%), 2 pgs degraded (PG_DEGRADED) 2024-02-09T17:34:57.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:56 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 4437/37899 objects degraded (11.707%), 2 pgs degraded (PG_DEGRADED) 2024-02-09T17:34:57.371 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:34:57.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:57.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:34:57.383 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:34:57.475 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:34:57.509 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:57.539 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:34:57.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:57.542 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:34:57.643 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:34:57.692 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:57.811 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-09T17:34:57.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:57 smithi179 ceph-mon[43975]: pgmap v851: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 668 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 792 op/s; 4437/32043 objects degraded (13.847%); 766/32043 objects misplaced (2.391%); 0 B/s, 164 objects/s recovering 2024-02-09T17:34:57.937 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:57 smithi099 ceph-mon[39420]: pgmap v851: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 668 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 792 op/s; 4437/32043 objects degraded (13.847%); 766/32043 objects misplaced (2.391%); 0 B/s, 164 objects/s recovering 2024-02-09T17:34:57.962 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:34:57.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:57.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:34:58.076 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:34:58.103 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:58.131 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:34:58.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:58.134 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:34:58.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:57 smithi162 ceph-mon[44304]: pgmap v851: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 668 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 792 op/s; 4437/32043 objects degraded (13.847%); 766/32043 objects misplaced (2.391%); 0 B/s, 164 objects/s recovering 2024-02-09T17:34:58.235 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:34:58.315 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:58.328 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:34:58.349 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:34:58.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:58.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:34:58.453 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:34:58.485 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:58.514 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:34:58.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:58.518 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:34:58.619 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:34:58.677 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:58.712 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:34:58.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:58.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:34:58.725 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.3 to 1 (10000) 2024-02-09T17:34:58.814 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:34:58.863 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:58.903 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:34:58.909 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:58.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:34:58.912 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:58 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/246936243' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-09T17:34:58.912 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:58 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-09T17:34:58.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:58 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/246936243' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-09T17:34:58.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:58 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-09T17:34:58.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:58 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/246936243' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-09T17:34:58.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:58 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-09T17:34:59.010 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:34:59.049 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:59.218 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:34:59.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:34:59.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:34:59.322 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:34:59.369 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:34:59.402 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:34:59.724 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:59 smithi099 ceph-mon[39420]: pgmap v852: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 574 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 844 op/s; 4437/27537 objects degraded (16.113%); 524/27537 objects misplaced (1.903%); 0 B/s, 142 objects/s recovering 2024-02-09T17:34:59.724 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:59 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-09T17:34:59.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:34:59 smithi099 ceph-mon[39420]: osdmap e306: 12 total, 12 up, 12 in 2024-02-09T17:35:00.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:59 smithi162 ceph-mon[44304]: pgmap v852: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 574 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 844 op/s; 4437/27537 objects degraded (16.113%); 524/27537 objects misplaced (1.903%); 0 B/s, 142 objects/s recovering 2024-02-09T17:35:00.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:59 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-09T17:35:00.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:34:59 smithi162 ceph-mon[44304]: osdmap e306: 12 total, 12 up, 12 in 2024-02-09T17:35:00.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:59 smithi179 ceph-mon[43975]: pgmap v852: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 574 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 844 op/s; 4437/27537 objects degraded (16.113%); 524/27537 objects misplaced (1.903%); 0 B/s, 142 objects/s recovering 2024-02-09T17:35:00.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:59 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-09T17:35:00.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:34:59 smithi179 ceph-mon[43975]: osdmap e306: 12 total, 12 up, 12 in 2024-02-09T17:35:00.479 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:35:00.479 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:35:01.209 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:35:01.215 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:01.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:35:01.238 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-09T17:35:01.316 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:35:01.381 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:01.450 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:35:01.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:01.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:35:01.569 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:35:01.604 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:01.627 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:35:01.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:01.630 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:35:01.705 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:35:01.730 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:35:01.764 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:01.794 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:35:01.803 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:01.804 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:35:01.905 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:35:01.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:01 smithi179 ceph-mon[43975]: pgmap v854: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 435 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 824 op/s; 4437/20871 objects degraded (21.259%); 524/20871 objects misplaced (2.511%); 0 B/s, 145 objects/s recovering 2024-02-09T17:35:01.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:01 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 4437/20871 objects degraded (21.259%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-09T17:35:01.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:01 smithi179 ceph-mon[43975]: osdmap e307: 12 total, 12 up, 12 in 2024-02-09T17:35:01.957 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 4 1 2024-02-09T17:35:01.969 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:01.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:01 smithi099 ceph-mon[39420]: pgmap v854: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 435 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 824 op/s; 4437/20871 objects degraded (21.259%); 524/20871 objects misplaced (2.511%); 0 B/s, 145 objects/s recovering 2024-02-09T17:35:01.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:01 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 4437/20871 objects degraded (21.259%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-09T17:35:01.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:01 smithi099 ceph-mon[39420]: osdmap e307: 12 total, 12 up, 12 in 2024-02-09T17:35:01.998 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:35:02.005 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:01 smithi162 ceph-mon[44304]: pgmap v854: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 435 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 824 op/s; 4437/20871 objects degraded (21.259%); 524/20871 objects misplaced (2.511%); 0 B/s, 145 objects/s recovering 2024-02-09T17:35:02.005 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:01 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 4437/20871 objects degraded (21.259%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-09T17:35:02.005 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:01 smithi162 ceph-mon[44304]: osdmap e307: 12 total, 12 up, 12 in 2024-02-09T17:35:02.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:02.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:35:02.107 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:35:02.150 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:02.179 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:35:02.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:02.187 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:35:02.288 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:35:02.322 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:02.344 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:35:02.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:02.350 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:35:02.450 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:35:02.510 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:02.626 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:35:02.685 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:35:03.781 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is unset 2024-02-09T17:35:03.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:03 smithi099 ceph-mon[39420]: pgmap v856: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 332 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 915 op/s; 4437/15918 objects degraded (27.874%); 271/15918 objects misplaced (1.702%); 0 B/s, 169 objects/s recovering 2024-02-09T17:35:03.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:03 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3190327635' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:35:04.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:03 smithi179 ceph-mon[43975]: pgmap v856: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 332 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 915 op/s; 4437/15918 objects degraded (27.874%); 271/15918 objects misplaced (1.702%); 0 B/s, 169 objects/s recovering 2024-02-09T17:35:04.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:03 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3190327635' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:35:04.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:03 smithi162 ceph-mon[44304]: pgmap v856: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 332 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 915 op/s; 4437/15918 objects degraded (27.874%); 271/15918 objects misplaced (1.702%); 0 B/s, 169 objects/s recovering 2024-02-09T17:35:04.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:03 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3190327635' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:35:04.316 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:35:04.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:04.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:35:04.421 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:35:04.475 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:04.530 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:35:04.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:04.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:35:04.642 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:35:04.671 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:04.706 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:35:04.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:04.713 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:35:04.814 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:35:04.849 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:04.870 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:35:04.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:04.872 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:35:04.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:04 smithi179 ceph-mon[43975]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:35:04.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:04 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3190327635' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:35:04.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:04 smithi179 ceph-mon[43975]: osdmap e308: 12 total, 12 up, 12 in 2024-02-09T17:35:04.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:04 smithi179 ceph-mon[43975]: 1.11 scrub starts 2024-02-09T17:35:04.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:04 smithi179 ceph-mon[43975]: 1.11 scrub ok 2024-02-09T17:35:04.973 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:35:05.009 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:05.038 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:35:05.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:05.041 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:35:05.142 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:35:05.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:04 smithi162 ceph-mon[44304]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:35:05.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:04 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3190327635' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:35:05.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:04 smithi162 ceph-mon[44304]: osdmap e308: 12 total, 12 up, 12 in 2024-02-09T17:35:05.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:04 smithi162 ceph-mon[44304]: 1.11 scrub starts 2024-02-09T17:35:05.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:04 smithi162 ceph-mon[44304]: 1.11 scrub ok 2024-02-09T17:35:05.190 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:05.218 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:35:05.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:05.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:35:05.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:04 smithi099 ceph-mon[39420]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:35:05.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:04 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3190327635' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:35:05.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:04 smithi099 ceph-mon[39420]: osdmap e308: 12 total, 12 up, 12 in 2024-02-09T17:35:05.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:04 smithi099 ceph-mon[39420]: 1.11 scrub starts 2024-02-09T17:35:05.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:04 smithi099 ceph-mon[39420]: 1.11 scrub ok 2024-02-09T17:35:05.323 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:35:05.362 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:05.393 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:35:05.399 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:05.400 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:35:05.501 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:35:05.580 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:35:05.580 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:35:05.598 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:05.636 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:35:05.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:05.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:35:05.741 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:35:05.772 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:05.801 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:35:05.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:05.804 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:35:05.905 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:35:05.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:05 smithi162 ceph-mon[44304]: pgmap v858: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 303 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 722 op/s; 4437/14520 objects degraded (30.558%); 271/14520 objects misplaced (1.866%); 0 B/s, 113 objects/s recovering 2024-02-09T17:35:05.937 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:05.962 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:35:05.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:05.968 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:35:06.069 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:35:06.118 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:05 smithi099 ceph-mon[39420]: pgmap v858: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 303 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 722 op/s; 4437/14520 objects degraded (30.558%); 271/14520 objects misplaced (1.866%); 0 B/s, 113 objects/s recovering 2024-02-09T17:35:06.123 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:06.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:05 smithi179 ceph-mon[43975]: pgmap v858: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 303 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 722 op/s; 4437/14520 objects degraded (30.558%); 271/14520 objects misplaced (1.866%); 0 B/s, 113 objects/s recovering 2024-02-09T17:35:06.266 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:35:06.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:06.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:35:06.375 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:35:06.379 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.9 dump_ops_in_flight 2024-02-09T17:35:06.411 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:06.499 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:35:06.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:06.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:35:06.605 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:35:06.645 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:06.685 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:35:06.694 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:06.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:35:06.741 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:35:06.795 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:35:06.854 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:06.893 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:35:06.937 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:35:06.973 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:35:06.986 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:35:06.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:06 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 4437/14520 objects degraded (30.558%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-09T17:35:06.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:06 smithi099 ceph-mon[39420]: osdmap e309: 12 total, 12 up, 12 in 2024-02-09T17:35:06.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:06.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:35:07.040 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:35:07.075 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:35:07.096 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:35:07.123 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:35:07.165 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:35:07.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:06 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 4437/14520 objects degraded (30.558%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-09T17:35:07.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:06 smithi179 ceph-mon[43975]: osdmap e309: 12 total, 12 up, 12 in 2024-02-09T17:35:07.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:06 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 4437/14520 objects degraded (30.558%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-09T17:35:07.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:06 smithi162 ceph-mon[44304]: osdmap e309: 12 total, 12 up, 12 in 2024-02-09T17:35:07.195 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:07.220 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:35:07.266 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:35:07.313 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:35:07.379 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:35:07.395 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:35:07.413 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:35:07.414 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:35:07.414 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:35:07.414 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:35:07.414 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:35:07.414 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:35:07.414 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:35:07.414 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:35:07.414 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:35:07.414 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:35:07.414 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:35:07.414 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:35:07.647 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Removed 131956 objects 2024-02-09T17:35:07.647 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Clean up completed and total clean up time :197.685 2024-02-09T17:35:08.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:07 smithi179 ceph-mon[43975]: pgmap v860: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 182 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 675 op/s; 4437/8715 objects degraded (50.912%); 271/8715 objects misplaced (3.110%); 0 B/s, 125 objects/s recovering 2024-02-09T17:35:08.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:07 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3905080853' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:35:08.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:07 smithi162 ceph-mon[44304]: pgmap v860: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 182 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 675 op/s; 4437/8715 objects degraded (50.912%); 271/8715 objects misplaced (3.110%); 0 B/s, 125 objects/s recovering 2024-02-09T17:35:08.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:07 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3905080853' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:35:08.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:07 smithi099 ceph-mon[39420]: pgmap v860: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 182 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 675 op/s; 4437/8715 objects degraded (50.912%); 271/8715 objects misplaced (3.110%); 0 B/s, 125 objects/s recovering 2024-02-09T17:35:08.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:07 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3905080853' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:35:08.291 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-09T17:35:08.885 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:35:08.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:08.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:35:08.991 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:35:09.019 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:09.039 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:35:09.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:09.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:35:09.143 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:35:09.167 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:08 smithi162 ceph-mon[44304]: 1.8 scrub starts 2024-02-09T17:35:09.167 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:08 smithi162 ceph-mon[44304]: 1.8 scrub ok 2024-02-09T17:35:09.169 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:09.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:08 smithi179 ceph-mon[43975]: 1.8 scrub starts 2024-02-09T17:35:09.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:08 smithi179 ceph-mon[43975]: 1.8 scrub ok 2024-02-09T17:35:09.188 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:35:09.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:09.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:35:09.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:08 smithi099 ceph-mon[39420]: 1.8 scrub starts 2024-02-09T17:35:09.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:08 smithi099 ceph-mon[39420]: 1.8 scrub ok 2024-02-09T17:35:09.291 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:35:09.318 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:09.338 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:35:09.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:09.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:35:09.442 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:35:09.471 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:09.492 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:35:09.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:09.495 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:35:09.596 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:35:09.622 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:09.641 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:35:09.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:09.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:35:09.745 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:35:09.776 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:09.803 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:35:09.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:09.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:35:09.907 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:35:09.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:09 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:35:09.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:09 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:35:09.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:09 smithi179 ceph-mon[43975]: pgmap v861: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 92 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 717 op/s; 4437/4398 objects degraded (100.887%); 42/4398 objects misplaced (0.955%); 0 B/s, 103 objects/s recovering 2024-02-09T17:35:09.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:09 smithi179 ceph-mon[43975]: osdmap e310: 12 total, 12 up, 12 in 2024-02-09T17:35:09.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:09 smithi179 ceph-mon[43975]: 1.7 scrub starts 2024-02-09T17:35:09.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:09 smithi179 ceph-mon[43975]: 1.7 scrub ok 2024-02-09T17:35:09.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:09 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:35:09.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:09 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:35:09.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:09 smithi162 ceph-mon[44304]: pgmap v861: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 92 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 717 op/s; 4437/4398 objects degraded (100.887%); 42/4398 objects misplaced (0.955%); 0 B/s, 103 objects/s recovering 2024-02-09T17:35:09.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:09 smithi162 ceph-mon[44304]: osdmap e310: 12 total, 12 up, 12 in 2024-02-09T17:35:09.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:09 smithi162 ceph-mon[44304]: 1.7 scrub starts 2024-02-09T17:35:09.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:09 smithi162 ceph-mon[44304]: 1.7 scrub ok 2024-02-09T17:35:09.934 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:09.953 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:35:09.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:09.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:35:10.057 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:35:10.083 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:10.103 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:35:10.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:10.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:35:10.164 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:09 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:35:10.164 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:09 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:35:10.164 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:09 smithi099 ceph-mon[39420]: pgmap v861: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 92 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 717 op/s; 4437/4398 objects degraded (100.887%); 42/4398 objects misplaced (0.955%); 0 B/s, 103 objects/s recovering 2024-02-09T17:35:10.164 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:09 smithi099 ceph-mon[39420]: osdmap e310: 12 total, 12 up, 12 in 2024-02-09T17:35:10.164 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:09 smithi099 ceph-mon[39420]: 1.7 scrub starts 2024-02-09T17:35:10.164 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:09 smithi099 ceph-mon[39420]: 1.7 scrub ok 2024-02-09T17:35:10.207 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:35:10.236 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:10.683 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:35:10.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:10.687 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:35:10.788 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:35:10.819 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:10.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:10 smithi179 ceph-mon[43975]: 1.f scrub starts 2024-02-09T17:35:10.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:10 smithi179 ceph-mon[43975]: 1.f scrub ok 2024-02-09T17:35:11.042 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-02-09T17:35:11.043 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-02-09T17:35:11.071 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:10 smithi099 ceph-mon[39420]: 1.f scrub starts 2024-02-09T17:35:11.071 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:10 smithi099 ceph-mon[39420]: 1.f scrub ok 2024-02-09T17:35:11.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:10 smithi162 ceph-mon[44304]: 1.f scrub starts 2024-02-09T17:35:11.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:10 smithi162 ceph-mon[44304]: 1.f scrub ok 2024-02-09T17:35:11.270 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:35:11.271 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-02-09T17:35:11.271 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 0 2024-02-09T17:35:11.271 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:35:11.832 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.4 to 1 (10000) 2024-02-09T17:35:11.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:11 smithi099 ceph-mon[39420]: pgmap v863: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 20 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 729 op/s; 4437/948 objects degraded (468.038%); 42/948 objects misplaced (4.430%); 0 B/s, 371 objects/s recovering 2024-02-09T17:35:11.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:11 smithi099 ceph-mon[39420]: osdmap e311: 12 total, 12 up, 12 in 2024-02-09T17:35:11.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:11 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2371137540' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-09T17:35:11.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:11 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 4437/948 objects degraded (468.038%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-09T17:35:12.027 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:11 smithi179 ceph-mon[43975]: pgmap v863: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 20 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 729 op/s; 4437/948 objects degraded (468.038%); 42/948 objects misplaced (4.430%); 0 B/s, 371 objects/s recovering 2024-02-09T17:35:12.027 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:11 smithi179 ceph-mon[43975]: osdmap e311: 12 total, 12 up, 12 in 2024-02-09T17:35:12.027 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:11 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2371137540' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-09T17:35:12.027 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:11 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 4437/948 objects degraded (468.038%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-09T17:35:12.054 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.9 dump_blocked_ops 2024-02-09T17:35:12.171 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:35:12.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:11 smithi162 ceph-mon[44304]: pgmap v863: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 20 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 729 op/s; 4437/948 objects degraded (468.038%); 42/948 objects misplaced (4.430%); 0 B/s, 371 objects/s recovering 2024-02-09T17:35:12.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:11 smithi162 ceph-mon[44304]: osdmap e311: 12 total, 12 up, 12 in 2024-02-09T17:35:12.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:11 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2371137540' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-09T17:35:12.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:11 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 4437/948 objects degraded (468.038%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-09T17:35:12.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:12.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:35:12.279 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:35:12.291 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:35:12.312 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:12.322 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:35:12.637 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 5 1 2024-02-09T17:35:12.639 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:35:12.869 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:35:12.869 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:12 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2371137540' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-09T17:35:12.869 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:12 smithi099 ceph-mon[39420]: osdmap e312: 12 total, 12 up, 12 in 2024-02-09T17:35:12.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:12.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:35:12.894 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:12 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2371137540' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-09T17:35:12.894 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:12 smithi179 ceph-mon[43975]: osdmap e312: 12 total, 12 up, 12 in 2024-02-09T17:35:12.974 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:35:13.004 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:13.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:12 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2371137540' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-09T17:35:13.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:12 smithi162 ceph-mon[44304]: osdmap e312: 12 total, 12 up, 12 in 2024-02-09T17:35:13.203 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:35:13.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:13.206 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:35:13.307 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:35:13.340 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:13.770 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:35:13.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:13 smithi179 ceph-mon[43975]: pgmap v866: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 751 KiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 483 op/s; 4437/6 objects degraded (73950.000%); 0 B/s, 297 objects/s recovering 2024-02-09T17:35:13.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:13 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:35:13.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:13 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:35:14.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:13 smithi162 ceph-mon[44304]: pgmap v866: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 751 KiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 483 op/s; 4437/6 objects degraded (73950.000%); 0 B/s, 297 objects/s recovering 2024-02-09T17:35:14.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:13 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:35:14.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:13 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:35:14.190 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:35:14.192 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:13 smithi099 ceph-mon[39420]: pgmap v866: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 31 active+clean, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 751 KiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 483 op/s; 4437/6 objects degraded (73950.000%); 0 B/s, 297 objects/s recovering 2024-02-09T17:35:14.192 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:13 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:35:14.192 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:13 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:35:14.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:14.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:35:14.259 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:35:14.259 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-02-09T17:35:14.259 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-02-09T17:35:14.259 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-02-09T17:35:14.259 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:35:14.294 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:35:14.321 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:14.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:14 smithi179 ceph-mon[43975]: 1.10 scrub starts 2024-02-09T17:35:14.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:14 smithi179 ceph-mon[43975]: 1.10 scrub ok 2024-02-09T17:35:14.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:14 smithi179 ceph-mon[43975]: 1.5 scrub starts 2024-02-09T17:35:14.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:14 smithi179 ceph-mon[43975]: osdmap e313: 12 total, 12 up, 12 in 2024-02-09T17:35:14.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:14 smithi179 ceph-mon[43975]: 1.5 scrub ok 2024-02-09T17:35:15.027 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.9 dump_historic_ops 2024-02-09T17:35:15.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:14 smithi162 ceph-mon[44304]: 1.10 scrub starts 2024-02-09T17:35:15.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:14 smithi162 ceph-mon[44304]: 1.10 scrub ok 2024-02-09T17:35:15.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:14 smithi162 ceph-mon[44304]: 1.5 scrub starts 2024-02-09T17:35:15.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:14 smithi162 ceph-mon[44304]: osdmap e313: 12 total, 12 up, 12 in 2024-02-09T17:35:15.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:14 smithi162 ceph-mon[44304]: 1.5 scrub ok 2024-02-09T17:35:15.186 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:14 smithi099 ceph-mon[39420]: 1.10 scrub starts 2024-02-09T17:35:15.186 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:14 smithi099 ceph-mon[39420]: 1.10 scrub ok 2024-02-09T17:35:15.186 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:14 smithi099 ceph-mon[39420]: 1.5 scrub starts 2024-02-09T17:35:15.186 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:14 smithi099 ceph-mon[39420]: osdmap e313: 12 total, 12 up, 12 in 2024-02-09T17:35:15.187 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:14 smithi099 ceph-mon[39420]: 1.5 scrub ok 2024-02-09T17:35:15.293 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:35:15.571 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:35:15.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:15.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:35:15.675 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:35:15.704 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:15.724 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:35:15.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:15.727 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:35:15.828 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:35:15.865 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:15 smithi179 ceph-mon[43975]: 3.5 scrub starts 2024-02-09T17:35:15.865 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:15 smithi179 ceph-mon[43975]: 3.5 scrub ok 2024-02-09T17:35:15.865 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:15 smithi179 ceph-mon[43975]: pgmap v868: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 32 active+clean, 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 751 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 252 op/s; 4437/6 objects degraded (73950.000%); 0 B/s, 310 objects/s recovering 2024-02-09T17:35:15.865 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:15 smithi179 ceph-mon[43975]: 1.6 scrub starts 2024-02-09T17:35:15.865 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:15 smithi179 ceph-mon[43975]: 1.6 scrub ok 2024-02-09T17:35:15.865 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:15.994 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:35:16.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:16.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:35:16.111 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:35:16.141 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:15 smithi099 ceph-mon[39420]: 3.5 scrub starts 2024-02-09T17:35:16.141 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:15 smithi099 ceph-mon[39420]: 3.5 scrub ok 2024-02-09T17:35:16.141 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:15 smithi099 ceph-mon[39420]: pgmap v868: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 32 active+clean, 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 751 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 252 op/s; 4437/6 objects degraded (73950.000%); 0 B/s, 310 objects/s recovering 2024-02-09T17:35:16.141 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:15 smithi099 ceph-mon[39420]: 1.6 scrub starts 2024-02-09T17:35:16.141 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:15 smithi099 ceph-mon[39420]: 1.6 scrub ok 2024-02-09T17:35:16.143 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:16.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:15 smithi162 ceph-mon[44304]: 3.5 scrub starts 2024-02-09T17:35:16.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:15 smithi162 ceph-mon[44304]: 3.5 scrub ok 2024-02-09T17:35:16.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:15 smithi162 ceph-mon[44304]: pgmap v868: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 32 active+clean, 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped; 751 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 252 op/s; 4437/6 objects degraded (73950.000%); 0 B/s, 310 objects/s recovering 2024-02-09T17:35:16.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:15 smithi162 ceph-mon[44304]: 1.6 scrub starts 2024-02-09T17:35:16.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:15 smithi162 ceph-mon[44304]: 1.6 scrub ok 2024-02-09T17:35:16.677 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:35:16.693 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:35:16.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:16.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:35:16.699 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:35:16.799 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:35:16.832 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:16 smithi099 ceph-mon[39420]: 1.1 scrub starts 2024-02-09T17:35:16.832 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:16 smithi099 ceph-mon[39420]: 1.1 scrub ok 2024-02-09T17:35:16.832 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:16 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 4437/6 objects degraded (73950.000%), 2 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:35:16.832 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:16 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3366324012' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:35:16.842 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:16.871 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:35:16.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:16.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:35:16.975 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:35:16.999 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:16 smithi179 ceph-mon[43975]: 1.1 scrub starts 2024-02-09T17:35:17.000 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:16 smithi179 ceph-mon[43975]: 1.1 scrub ok 2024-02-09T17:35:17.000 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:16 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 4437/6 objects degraded (73950.000%), 2 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:35:17.000 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:16 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3366324012' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:35:17.002 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:17.024 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:35:17.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:17.027 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:35:17.128 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:35:17.159 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:17.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:16 smithi162 ceph-mon[44304]: 1.1 scrub starts 2024-02-09T17:35:17.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:16 smithi162 ceph-mon[44304]: 1.1 scrub ok 2024-02-09T17:35:17.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:16 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 4437/6 objects degraded (73950.000%), 2 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:35:17.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:16 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3366324012' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:35:17.186 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:35:17.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:17.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:35:17.291 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:35:17.303 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is unset 2024-02-09T17:35:17.323 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:17.342 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:35:17.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:17.348 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:35:17.358 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:35:17.358 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-02-09T17:35:17.358 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-02-09T17:35:17.358 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-02-09T17:35:17.358 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.358 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:80118 2.d 2:b464a328:::benchmark_data_smithi179_77138_object80117:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-02-09T17:35:17.358 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:25:28.551378+0000", 2024-02-09T17:35:17.358 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 588.80426274700005, 2024-02-09T17:35:17.358 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.060053192999999998, 2024-02-09T17:35:17.359 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:35:17.359 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:35:17.359 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:35:17.359 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:35:17.359 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:35:17.359 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 80118 2024-02-09T17:35:17.359 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.359 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:35:17.359 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.359 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:35:17.359 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.551378+0000", 2024-02-09T17:35:17.359 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:17.359 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.360 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.360 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:35:17.360 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.551378+0000", 2024-02-09T17:35:17.360 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:17.360 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.360 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.360 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:35:17.360 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.551380+0000", 2024-02-09T17:35:17.360 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5740000000000001e-06 2024-02-09T17:35:17.360 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.360 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.360 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:35:17.361 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.551399+0000", 2024-02-09T17:35:17.361 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9151e-05 2024-02-09T17:35:17.361 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.361 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.361 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:35:17.361 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.551409+0000", 2024-02-09T17:35:17.361 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.781e-06 2024-02-09T17:35:17.361 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.361 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.361 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:35:17.361 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.551415+0000", 2024-02-09T17:35:17.361 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7270000000000004e-06 2024-02-09T17:35:17.361 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.361 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.362 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:35:17.362 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.551468+0000", 2024-02-09T17:35:17.362 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3041000000000003e-05 2024-02-09T17:35:17.362 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.362 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.362 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:35:17.362 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.551556+0000", 2024-02-09T17:35:17.362 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8195999999999994e-05 2024-02-09T17:35:17.362 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.362 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.362 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,4", 2024-02-09T17:35:17.362 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.551659+0000", 2024-02-09T17:35:17.362 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000103038 2024-02-09T17:35:17.363 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.363 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.363 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:35:17.363 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.610316+0000", 2024-02-09T17:35:17.363 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.058657448000000001 2024-02-09T17:35:17.363 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.363 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.363 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:35:17.363 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.610360+0000", 2024-02-09T17:35:17.363 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3344999999999999e-05 2024-02-09T17:35:17.363 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.363 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.363 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:35:17.364 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.611336+0000", 2024-02-09T17:35:17.364 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00097672399999999999 2024-02-09T17:35:17.364 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.364 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.364 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:35:17.364 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.611394+0000", 2024-02-09T17:35:17.364 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7376000000000001e-05 2024-02-09T17:35:17.364 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.364 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.364 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:35:17.364 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.611432+0000", 2024-02-09T17:35:17.364 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7792e-05 2024-02-09T17:35:17.365 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:17.365 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:35:17.365 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:17.365 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.365 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.365 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.14790.0:80117 2.a e129/111 2:564ef782:::benchmark_data_smithi179_77138_object80116:head v 129'4917, mlcod=129'4915)", 2024-02-09T17:35:17.365 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:25:28.551479+0000", 2024-02-09T17:35:17.365 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 588.80416192899997, 2024-02-09T17:35:17.365 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.055548806999999999, 2024-02-09T17:35:17.365 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:35:17.365 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:35:17.365 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:35:17.365 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.365 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:35:17.366 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.551479+0000", 2024-02-09T17:35:17.366 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:17.366 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.366 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.366 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:35:17.366 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.551479+0000", 2024-02-09T17:35:17.366 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:17.366 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.366 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.366 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:35:17.366 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.551480+0000", 2024-02-09T17:35:17.366 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.13e-07 2024-02-09T17:35:17.366 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.366 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.366 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:35:17.367 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.551495+0000", 2024-02-09T17:35:17.367 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5406000000000001e-05 2024-02-09T17:35:17.367 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.367 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.367 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:35:17.367 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.551502+0000", 2024-02-09T17:35:17.367 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3110000000000002e-06 2024-02-09T17:35:17.367 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.367 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.367 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:35:17.367 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.551505+0000", 2024-02-09T17:35:17.367 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6820000000000001e-06 2024-02-09T17:35:17.367 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.368 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.368 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:35:17.368 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.551552+0000", 2024-02-09T17:35:17.368 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6622999999999997e-05 2024-02-09T17:35:17.368 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.368 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.368 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:35:17.368 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.551570+0000", 2024-02-09T17:35:17.368 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8652999999999999e-05 2024-02-09T17:35:17.368 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.368 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.368 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:35:17.368 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.606983+0000", 2024-02-09T17:35:17.368 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.055412241000000001 2024-02-09T17:35:17.369 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.369 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.369 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:35:17.369 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.607028+0000", 2024-02-09T17:35:17.369 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5377999999999998e-05 2024-02-09T17:35:17.369 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:17.369 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:35:17.369 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:17.369 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.369 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.369 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop_reply(client.14790.0:80118 2.d e129/111 ondisk, result = 0)", 2024-02-09T17:35:17.369 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:25:28.554106+0000", 2024-02-09T17:35:17.369 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 588.80153541899995, 2024-02-09T17:35:17.369 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.056260956000000001, 2024-02-09T17:35:17.369 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:35:17.370 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-02-09T17:35:17.370 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:35:17.370 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.370 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:35:17.370 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.554106+0000", 2024-02-09T17:35:17.370 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:17.370 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.370 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.370 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:35:17.370 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.554106+0000", 2024-02-09T17:35:17.370 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:17.370 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.370 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.370 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:35:17.371 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.554107+0000", 2024-02-09T17:35:17.371 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8100000000000002e-07 2024-02-09T17:35:17.371 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.371 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.371 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:35:17.371 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.554113+0000", 2024-02-09T17:35:17.371 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.888e-06 2024-02-09T17:35:17.371 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.371 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.371 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:35:17.371 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.607102+0000", 2024-02-09T17:35:17.371 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.052988406000000002 2024-02-09T17:35:17.371 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.371 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.371 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:35:17.371 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.607110+0000", 2024-02-09T17:35:17.371 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7680000000000008e-06 2024-02-09T17:35:17.372 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.372 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.372 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:35:17.372 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.610349+0000", 2024-02-09T17:35:17.372 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.003239114 2024-02-09T17:35:17.372 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.372 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.372 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:35:17.372 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.610355+0000", 2024-02-09T17:35:17.372 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.976e-06 2024-02-09T17:35:17.372 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.372 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.372 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:35:17.373 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.610367+0000", 2024-02-09T17:35:17.373 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2023e-05 2024-02-09T17:35:17.373 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:17.373 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:35:17.373 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:17.373 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.373 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.373 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop_reply(client.14790.0:80118 2.d e129/111 ondisk, result = 0)", 2024-02-09T17:35:17.373 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:25:28.554107+0000", 2024-02-09T17:35:17.373 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 588.80153460300005, 2024-02-09T17:35:17.373 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.056217660000000003, 2024-02-09T17:35:17.373 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:35:17.373 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-02-09T17:35:17.374 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:35:17.374 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.374 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:35:17.374 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.554107+0000", 2024-02-09T17:35:17.374 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:17.374 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.374 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.374 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:35:17.374 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.554107+0000", 2024-02-09T17:35:17.374 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:17.374 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.374 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.374 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:35:17.374 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.554107+0000", 2024-02-09T17:35:17.375 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7300000000000005e-07 2024-02-09T17:35:17.375 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.375 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.375 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:35:17.375 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.554112+0000", 2024-02-09T17:35:17.375 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1799999999999998e-06 2024-02-09T17:35:17.375 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.375 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.375 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:35:17.375 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.606920+0000", 2024-02-09T17:35:17.375 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.052808709000000002 2024-02-09T17:35:17.375 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.375 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.375 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:35:17.376 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.606931+0000", 2024-02-09T17:35:17.376 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0978000000000001e-05 2024-02-09T17:35:17.376 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.376 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.376 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:35:17.376 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.610302+0000", 2024-02-09T17:35:17.376 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0033709899999999999 2024-02-09T17:35:17.376 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.376 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.376 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:35:17.376 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.610311+0000", 2024-02-09T17:35:17.376 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7029999999999995e-06 2024-02-09T17:35:17.376 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.376 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.377 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:35:17.377 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.610324+0000", 2024-02-09T17:35:17.377 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3327e-05 2024-02-09T17:35:17.377 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:17.377 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:35:17.377 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:17.377 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.377 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.377 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.14790.0:80121 2.0 e129/111 2:0886889a:::benchmark_data_smithi179_77138_object80120:head v 129'5026, mlcod=129'5024)", 2024-02-09T17:35:17.377 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:25:28.559396+0000", 2024-02-09T17:35:17.377 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 588.79624552999996, 2024-02-09T17:35:17.377 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.052018096999999999, 2024-02-09T17:35:17.377 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:35:17.378 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:35:17.378 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:35:17.378 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.378 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:35:17.378 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.559396+0000", 2024-02-09T17:35:17.378 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:17.378 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.378 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.378 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:35:17.378 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.559396+0000", 2024-02-09T17:35:17.378 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:17.378 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.378 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.379 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:35:17.379 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.559396+0000", 2024-02-09T17:35:17.379 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2799999999999996e-07 2024-02-09T17:35:17.379 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.379 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.379 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:35:17.379 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.559443+0000", 2024-02-09T17:35:17.379 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6338000000000003e-05 2024-02-09T17:35:17.379 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.379 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.379 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:35:17.379 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.607509+0000", 2024-02-09T17:35:17.379 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.048066580999999997 2024-02-09T17:35:17.379 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.380 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.380 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:35:17.380 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.607519+0000", 2024-02-09T17:35:17.380 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.9569999999999997e-06 2024-02-09T17:35:17.380 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.380 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.380 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:35:17.380 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.607592+0000", 2024-02-09T17:35:17.380 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2952000000000002e-05 2024-02-09T17:35:17.380 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.380 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.380 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:35:17.380 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.607614+0000", 2024-02-09T17:35:17.381 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1444e-05 2024-02-09T17:35:17.381 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.381 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.381 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:35:17.381 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.611364+0000", 2024-02-09T17:35:17.381 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0037505120000000001 2024-02-09T17:35:17.381 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.381 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.381 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:35:17.381 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:25:28.611414+0000", 2024-02-09T17:35:17.381 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9685000000000003e-05 2024-02-09T17:35:17.382 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:17.382 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:35:17.382 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:17.382 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.382 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.382 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.14790.0:173778 2.a e151/111 2:5f827783:::benchmark_data_smithi179_77138_object40370:head v 151'10756, mlcod=151'10744)", 2024-02-09T17:35:17.382 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:04.453964+0000", 2024-02-09T17:35:17.382 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 492.90167735699998, 2024-02-09T17:35:17.382 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.051630930999999998, 2024-02-09T17:35:17.382 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:35:17.382 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:35:17.382 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:35:17.382 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.382 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:35:17.383 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.453964+0000", 2024-02-09T17:35:17.383 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:17.383 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.383 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.383 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:35:17.383 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.453964+0000", 2024-02-09T17:35:17.383 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:17.383 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.383 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.383 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:35:17.383 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.453964+0000", 2024-02-09T17:35:17.383 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4999999999999999e-07 2024-02-09T17:35:17.383 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.383 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.384 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:35:17.384 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.453971+0000", 2024-02-09T17:35:17.384 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.613e-06 2024-02-09T17:35:17.384 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.384 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.384 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:35:17.384 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.453976+0000", 2024-02-09T17:35:17.384 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1410000000000004e-06 2024-02-09T17:35:17.384 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.384 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.384 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:35:17.384 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.453979+0000", 2024-02-09T17:35:17.384 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.726e-06 2024-02-09T17:35:17.385 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.385 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.385 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:35:17.385 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.490546+0000", 2024-02-09T17:35:17.385 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.036566492999999999 2024-02-09T17:35:17.385 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.385 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.385 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:35:17.385 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.490566+0000", 2024-02-09T17:35:17.385 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0395999999999999e-05 2024-02-09T17:35:17.385 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.385 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.385 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:35:17.385 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.505548+0000", 2024-02-09T17:35:17.386 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014981828000000001 2024-02-09T17:35:17.386 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.386 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.386 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:35:17.386 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.505595+0000", 2024-02-09T17:35:17.386 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6584e-05 2024-02-09T17:35:17.386 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:17.386 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:35:17.386 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:17.386 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.386 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.386 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.14790.0:173780 2.a e151/111 2:5f6bf9e0:::benchmark_data_smithi179_77138_object40372:head v 151'10757, mlcod=151'10744)", 2024-02-09T17:35:17.387 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:04.453984+0000", 2024-02-09T17:35:17.387 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 492.90165690700002, 2024-02-09T17:35:17.387 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.058307537999999999, 2024-02-09T17:35:17.387 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:35:17.387 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:35:17.387 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:35:17.387 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.387 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:35:17.387 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.453984+0000", 2024-02-09T17:35:17.387 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:17.387 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.387 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.387 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:35:17.388 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.453984+0000", 2024-02-09T17:35:17.388 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:17.388 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.388 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.388 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:35:17.388 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.453984+0000", 2024-02-09T17:35:17.388 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7100000000000001e-07 2024-02-09T17:35:17.388 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.388 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.388 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:35:17.388 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.453987+0000", 2024-02-09T17:35:17.388 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5790000000000002e-06 2024-02-09T17:35:17.389 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.389 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.389 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:35:17.389 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.453993+0000", 2024-02-09T17:35:17.389 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6030000000000004e-06 2024-02-09T17:35:17.389 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.389 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.389 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:35:17.389 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.453996+0000", 2024-02-09T17:35:17.389 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7969999999999998e-06 2024-02-09T17:35:17.389 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.389 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.389 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:35:17.390 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.494808+0000", 2024-02-09T17:35:17.390 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.040811543999999998 2024-02-09T17:35:17.390 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.390 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.390 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:35:17.390 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.494835+0000", 2024-02-09T17:35:17.390 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6744999999999999e-05 2024-02-09T17:35:17.390 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.390 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.390 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:35:17.390 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.512249+0000", 2024-02-09T17:35:17.390 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017414190999999999 2024-02-09T17:35:17.390 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.390 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.391 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:35:17.391 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.512292+0000", 2024-02-09T17:35:17.391 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2908000000000003e-05 2024-02-09T17:35:17.391 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:17.391 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:35:17.391 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:17.391 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.391 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.391 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.14790.0:173783 2.a e151/111 2:5bec6a3b:::benchmark_data_smithi179_77138_object40375:head v 151'10758, mlcod=151'10744)", 2024-02-09T17:35:17.391 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:04.454001+0000", 2024-02-09T17:35:17.391 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 492.901640262, 2024-02-09T17:35:17.391 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.058321306000000003, 2024-02-09T17:35:17.391 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:35:17.391 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:35:17.392 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:35:17.392 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.392 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:35:17.392 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.454001+0000", 2024-02-09T17:35:17.392 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:17.392 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.392 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.392 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:35:17.392 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.454001+0000", 2024-02-09T17:35:17.392 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:17.392 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.392 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.392 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:35:17.392 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.454001+0000", 2024-02-09T17:35:17.393 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.74e-07 2024-02-09T17:35:17.393 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.393 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.393 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:35:17.393 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.454004+0000", 2024-02-09T17:35:17.393 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.593e-06 2024-02-09T17:35:17.393 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.393 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.393 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:35:17.393 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.454009+0000", 2024-02-09T17:35:17.393 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7030000000000003e-06 2024-02-09T17:35:17.393 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.393 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.393 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:35:17.393 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.454013+0000", 2024-02-09T17:35:17.394 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.878e-06 2024-02-09T17:35:17.394 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.394 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.394 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:35:17.394 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.498749+0000", 2024-02-09T17:35:17.394 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.044736154 2024-02-09T17:35:17.394 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.394 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.394 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:35:17.394 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.498774+0000", 2024-02-09T17:35:17.394 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4893e-05 2024-02-09T17:35:17.394 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.394 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.394 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:35:17.395 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.512294+0000", 2024-02-09T17:35:17.395 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013519869 2024-02-09T17:35:17.395 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.395 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.395 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:35:17.395 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.512322+0000", 2024-02-09T17:35:17.395 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.8042e-05 2024-02-09T17:35:17.395 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:17.395 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:35:17.395 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:17.395 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.395 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.395 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.14790.0:173785 2.a e151/111 2:50bf97e6:::benchmark_data_smithi179_77138_object40377:head v 151'10759, mlcod=151'10744)", 2024-02-09T17:35:17.395 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:04.454018+0000", 2024-02-09T17:35:17.395 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 492.90162334299998, 2024-02-09T17:35:17.396 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.058335370999999997, 2024-02-09T17:35:17.396 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:35:17.396 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:35:17.396 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:35:17.396 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.396 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:35:17.396 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.454018+0000", 2024-02-09T17:35:17.396 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:17.396 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.396 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.396 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:35:17.396 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.454018+0000", 2024-02-09T17:35:17.396 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:17.397 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.397 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.397 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:35:17.397 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.454018+0000", 2024-02-09T17:35:17.397 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.48e-07 2024-02-09T17:35:17.397 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.397 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.397 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:35:17.397 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.454023+0000", 2024-02-09T17:35:17.397 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1220000000000003e-06 2024-02-09T17:35:17.397 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.397 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.397 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:35:17.397 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.454029+0000", 2024-02-09T17:35:17.398 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.728e-06 2024-02-09T17:35:17.398 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.398 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.398 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:35:17.398 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.454039+0000", 2024-02-09T17:35:17.398 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0316000000000001e-05 2024-02-09T17:35:17.398 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.398 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.398 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:35:17.398 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.502223+0000", 2024-02-09T17:35:17.398 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.048184044000000002 2024-02-09T17:35:17.398 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.398 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.398 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:35:17.398 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.502242+0000", 2024-02-09T17:35:17.398 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8544000000000001e-05 2024-02-09T17:35:17.399 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.399 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.399 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:35:17.399 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.512326+0000", 2024-02-09T17:35:17.399 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010084253 2024-02-09T17:35:17.399 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.399 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.399 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:35:17.399 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.512353+0000", 2024-02-09T17:35:17.399 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7216000000000002e-05 2024-02-09T17:35:17.399 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:17.399 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:35:17.399 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:17.399 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.400 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.400 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.14790.0:173792 2.a e151/111 2:591697cf:::benchmark_data_smithi179_77138_object40384:head v 151'10760, mlcod=151'10744)", 2024-02-09T17:35:17.400 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:04.454044+0000", 2024-02-09T17:35:17.400 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 492.90159713600002, 2024-02-09T17:35:17.400 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.058332667999999997, 2024-02-09T17:35:17.400 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:35:17.400 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:35:17.400 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:35:17.400 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.400 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:35:17.400 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.454044+0000", 2024-02-09T17:35:17.400 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:17.400 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.400 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.400 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:35:17.401 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.454044+0000", 2024-02-09T17:35:17.401 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:17.401 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.401 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.401 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:35:17.401 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.454044+0000", 2024-02-09T17:35:17.401 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8799999999999999e-07 2024-02-09T17:35:17.401 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.401 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.401 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:35:17.401 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.454047+0000", 2024-02-09T17:35:17.401 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.785e-06 2024-02-09T17:35:17.401 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.402 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.402 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:35:17.402 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.454053+0000", 2024-02-09T17:35:17.402 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7880000000000003e-06 2024-02-09T17:35:17.402 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.402 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.402 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:35:17.402 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.454057+0000", 2024-02-09T17:35:17.402 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7790000000000002e-06 2024-02-09T17:35:17.402 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.402 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.402 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:35:17.402 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.505641+0000", 2024-02-09T17:35:17.403 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.051584818999999997 2024-02-09T17:35:17.403 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.403 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.403 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:35:17.403 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.505661+0000", 2024-02-09T17:35:17.403 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9888e-05 2024-02-09T17:35:17.403 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.403 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.403 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:35:17.403 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.512355+0000", 2024-02-09T17:35:17.403 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0066934439999999998 2024-02-09T17:35:17.403 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.403 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.403 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:35:17.404 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.512377+0000", 2024-02-09T17:35:17.404 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1977e-05 2024-02-09T17:35:17.404 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:17.404 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:35:17.404 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:17.404 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.404 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.404 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:181916 2.d 2:b5c83daf:::benchmark_data_smithi179_77138_object48508:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-02-09T17:35:17.404 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:13.475564+0000", 2024-02-09T17:35:17.404 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 483.88007746599999, 2024-02-09T17:35:17.404 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.052357841000000002, 2024-02-09T17:35:17.404 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:35:17.405 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:35:17.405 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:35:17.405 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:35:17.405 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:35:17.405 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 181916 2024-02-09T17:35:17.405 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.405 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:35:17.405 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.405 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:35:17.405 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.475564+0000", 2024-02-09T17:35:17.405 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:17.405 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.405 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.405 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:35:17.406 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.475564+0000", 2024-02-09T17:35:17.406 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:17.406 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.406 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.406 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:35:17.406 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.475565+0000", 2024-02-09T17:35:17.406 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0470000000000001e-06 2024-02-09T17:35:17.406 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.406 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.406 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:35:17.406 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.475570+0000", 2024-02-09T17:35:17.406 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2700000000000004e-06 2024-02-09T17:35:17.406 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.406 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.407 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:35:17.407 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.475583+0000", 2024-02-09T17:35:17.407 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2823000000000001e-05 2024-02-09T17:35:17.407 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.407 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.407 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:35:17.407 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.475591+0000", 2024-02-09T17:35:17.407 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1920000000000005e-06 2024-02-09T17:35:17.407 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.407 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.407 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:35:17.407 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.482368+0000", 2024-02-09T17:35:17.407 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0067773829999999997 2024-02-09T17:35:17.407 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.408 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.408 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:35:17.408 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.482488+0000", 2024-02-09T17:35:17.408 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000119066 2024-02-09T17:35:17.408 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.408 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.408 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,4", 2024-02-09T17:35:17.408 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.482604+0000", 2024-02-09T17:35:17.408 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000116335 2024-02-09T17:35:17.408 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.408 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.408 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:35:17.408 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.487370+0000", 2024-02-09T17:35:17.408 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0047663820000000004 2024-02-09T17:35:17.409 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.409 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.409 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:35:17.409 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.498528+0000", 2024-02-09T17:35:17.409 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011157602000000001 2024-02-09T17:35:17.409 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.409 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.409 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:35:17.409 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.527823+0000", 2024-02-09T17:35:17.409 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.029295307999999999 2024-02-09T17:35:17.409 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.409 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.409 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:35:17.409 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.527881+0000", 2024-02-09T17:35:17.410 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8088999999999999e-05 2024-02-09T17:35:17.410 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.410 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.410 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:35:17.410 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.527922+0000", 2024-02-09T17:35:17.410 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0343999999999999e-05 2024-02-09T17:35:17.410 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:17.410 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:35:17.410 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:17.410 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.410 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.410 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:181917 2.d 2:b69d7600:::benchmark_data_smithi179_77138_object48509:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-02-09T17:35:17.410 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:13.477060+0000", 2024-02-09T17:35:17.410 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 483.87858078099998, 2024-02-09T17:35:17.411 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.050989665000000003, 2024-02-09T17:35:17.411 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:35:17.411 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:35:17.411 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:35:17.411 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:35:17.411 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:35:17.411 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 181917 2024-02-09T17:35:17.411 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.411 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:35:17.411 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.411 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:35:17.411 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.477060+0000", 2024-02-09T17:35:17.411 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:17.411 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.411 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.412 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:35:17.412 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.477060+0000", 2024-02-09T17:35:17.412 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:17.412 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.412 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.412 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:35:17.412 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.477062+0000", 2024-02-09T17:35:17.412 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1319999999999999e-06 2024-02-09T17:35:17.412 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.412 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.412 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:35:17.412 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.477066+0000", 2024-02-09T17:35:17.412 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9089999999999999e-06 2024-02-09T17:35:17.413 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.413 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.413 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:35:17.413 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.477078+0000", 2024-02-09T17:35:17.413 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1199000000000001e-05 2024-02-09T17:35:17.413 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.413 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.413 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:35:17.413 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.477086+0000", 2024-02-09T17:35:17.413 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8509999999999999e-06 2024-02-09T17:35:17.413 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.413 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.413 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:35:17.413 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.487389+0000", 2024-02-09T17:35:17.413 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010303102999999999 2024-02-09T17:35:17.414 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.414 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.414 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:35:17.414 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.490188+0000", 2024-02-09T17:35:17.414 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0027996200000000001 2024-02-09T17:35:17.414 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.414 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.414 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,4", 2024-02-09T17:35:17.414 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.490313+0000", 2024-02-09T17:35:17.414 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012458400000000001 2024-02-09T17:35:17.414 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.414 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.414 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:35:17.415 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.498535+0000", 2024-02-09T17:35:17.415 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0082224059999999998 2024-02-09T17:35:17.415 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.415 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.415 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:35:17.415 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.498636+0000", 2024-02-09T17:35:17.415 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000100929 2024-02-09T17:35:17.415 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.415 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.415 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:35:17.415 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.527967+0000", 2024-02-09T17:35:17.415 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.029331327000000001 2024-02-09T17:35:17.416 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.416 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.416 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:35:17.416 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.528017+0000", 2024-02-09T17:35:17.416 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0008e-05 2024-02-09T17:35:17.416 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.416 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.416 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:35:17.416 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:13.528050+0000", 2024-02-09T17:35:17.416 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2597e-05 2024-02-09T17:35:17.416 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:17.416 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:35:17.416 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:17.417 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.417 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.417 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:218135 2.d 2:b725d747:::benchmark_data_smithi179_77138_object84727:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-02-09T17:35:17.417 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:56.815902+0000", 2024-02-09T17:35:17.417 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 440.53973939100001, 2024-02-09T17:35:17.417 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.055060459999999999, 2024-02-09T17:35:17.417 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:35:17.417 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:35:17.417 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:35:17.417 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:35:17.417 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:35:17.417 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 218135 2024-02-09T17:35:17.417 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.417 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:35:17.418 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.418 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:35:17.418 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:56.815902+0000", 2024-02-09T17:35:17.418 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:17.418 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.418 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.418 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:35:17.418 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:56.815902+0000", 2024-02-09T17:35:17.418 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:17.418 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.418 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.418 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:35:17.418 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:56.815903+0000", 2024-02-09T17:35:17.419 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.122e-06 2024-02-09T17:35:17.419 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.419 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.419 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:35:17.419 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:56.815908+0000", 2024-02-09T17:35:17.419 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8489999999999997e-06 2024-02-09T17:35:17.419 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.419 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.419 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:35:17.419 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:56.815919+0000", 2024-02-09T17:35:17.419 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1589e-05 2024-02-09T17:35:17.419 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.420 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.420 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:35:17.420 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:56.815926+0000", 2024-02-09T17:35:17.420 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.0190000000000001e-06 2024-02-09T17:35:17.420 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.420 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.420 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:35:17.420 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:56.815978+0000", 2024-02-09T17:35:17.420 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1600000000000001e-05 2024-02-09T17:35:17.420 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.420 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.421 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:35:17.421 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:56.816077+0000", 2024-02-09T17:35:17.421 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.8690000000000005e-05 2024-02-09T17:35:17.421 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.421 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.421 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,4", 2024-02-09T17:35:17.421 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:56.816165+0000", 2024-02-09T17:35:17.421 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8376999999999998e-05 2024-02-09T17:35:17.421 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.421 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.421 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:35:17.422 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:56.820079+0000", 2024-02-09T17:35:17.422 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0039135849999999998 2024-02-09T17:35:17.422 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.422 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.422 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:35:17.422 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:56.820175+0000", 2024-02-09T17:35:17.422 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5941999999999997e-05 2024-02-09T17:35:17.422 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.422 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.422 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:35:17.422 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:56.870897+0000", 2024-02-09T17:35:17.422 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.050722738000000003 2024-02-09T17:35:17.422 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.422 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.423 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:35:17.423 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:56.870934+0000", 2024-02-09T17:35:17.423 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6653999999999998e-05 2024-02-09T17:35:17.423 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.423 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.423 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:35:17.423 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:56.870962+0000", 2024-02-09T17:35:17.423 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.8294999999999999e-05 2024-02-09T17:35:17.423 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:17.423 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:35:17.423 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:17.423 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.423 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.424 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:227092 2.d 2:b61c2b3c:::benchmark_data_smithi179_77138_object93684:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-02-09T17:35:17.424 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:28:06.766657+0000", 2024-02-09T17:35:17.424 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 430.58898434999998, 2024-02-09T17:35:17.424 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.050998014000000001, 2024-02-09T17:35:17.424 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:35:17.424 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:35:17.424 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:35:17.424 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:35:17.424 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:35:17.424 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 227092 2024-02-09T17:35:17.424 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.424 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:35:17.424 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.425 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:35:17.425 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:06.766657+0000", 2024-02-09T17:35:17.425 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:17.425 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.425 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.425 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:35:17.425 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:06.766657+0000", 2024-02-09T17:35:17.425 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:17.425 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.425 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.425 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:35:17.425 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:06.766659+0000", 2024-02-09T17:35:17.425 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.767e-06 2024-02-09T17:35:17.426 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.426 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.426 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:35:17.426 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:06.766663+0000", 2024-02-09T17:35:17.426 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7400000000000004e-06 2024-02-09T17:35:17.426 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.426 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.426 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:35:17.426 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:06.766679+0000", 2024-02-09T17:35:17.426 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5690999999999999e-05 2024-02-09T17:35:17.426 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.426 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.426 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:35:17.426 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:06.766687+0000", 2024-02-09T17:35:17.427 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.32e-06 2024-02-09T17:35:17.427 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.427 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.427 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:35:17.427 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:06.766746+0000", 2024-02-09T17:35:17.427 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8371000000000001e-05 2024-02-09T17:35:17.427 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.427 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.427 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:35:17.427 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:06.766851+0000", 2024-02-09T17:35:17.427 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010524800000000001 2024-02-09T17:35:17.427 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.427 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.427 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,4", 2024-02-09T17:35:17.428 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:06.766979+0000", 2024-02-09T17:35:17.428 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012756900000000001 2024-02-09T17:35:17.428 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.428 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.428 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:35:17.428 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:06.770957+0000", 2024-02-09T17:35:17.428 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0039787520000000003 2024-02-09T17:35:17.428 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.428 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.428 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:35:17.428 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:06.774094+0000", 2024-02-09T17:35:17.428 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0031361819999999999 2024-02-09T17:35:17.428 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.429 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.429 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:35:17.429 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:06.817564+0000", 2024-02-09T17:35:17.429 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.043470921000000003 2024-02-09T17:35:17.429 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.429 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.429 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:35:17.429 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:06.817617+0000", 2024-02-09T17:35:17.429 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2417000000000003e-05 2024-02-09T17:35:17.429 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.429 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.429 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:35:17.429 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:06.817655+0000", 2024-02-09T17:35:17.429 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8036e-05 2024-02-09T17:35:17.430 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:17.430 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:35:17.430 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:17.430 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.430 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.430 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.14790.0:239189 2.8 e174/111 2:1048378c:::benchmark_data_smithi179_77138_object105781:head v 174'14801, mlcod=174'14799)", 2024-02-09T17:35:17.430 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:28:24.218178+0000", 2024-02-09T17:35:17.430 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 413.13746338800001, 2024-02-09T17:35:17.430 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.056142144999999997, 2024-02-09T17:35:17.430 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:35:17.430 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:35:17.430 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:35:17.430 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.431 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:35:17.431 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:24.218178+0000", 2024-02-09T17:35:17.431 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:17.431 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.431 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.431 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:35:17.431 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:24.218178+0000", 2024-02-09T17:35:17.431 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:17.431 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.431 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.431 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:35:17.431 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:24.218179+0000", 2024-02-09T17:35:17.431 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.09e-07 2024-02-09T17:35:17.431 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.432 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.432 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:35:17.432 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:24.218185+0000", 2024-02-09T17:35:17.432 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3509999999999998e-06 2024-02-09T17:35:17.432 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.432 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.432 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:35:17.432 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:24.259826+0000", 2024-02-09T17:35:17.432 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.041641055000000003 2024-02-09T17:35:17.432 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.432 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.432 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:35:17.432 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:24.259838+0000", 2024-02-09T17:35:17.432 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2235999999999999e-05 2024-02-09T17:35:17.433 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.433 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.433 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:35:17.433 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:24.259885+0000", 2024-02-09T17:35:17.433 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6424999999999997e-05 2024-02-09T17:35:17.433 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.433 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.433 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:35:17.433 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:24.259907+0000", 2024-02-09T17:35:17.433 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2654999999999999e-05 2024-02-09T17:35:17.433 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.433 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.433 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:35:17.433 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:24.274289+0000", 2024-02-09T17:35:17.434 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014381669 2024-02-09T17:35:17.434 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.434 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.434 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:35:17.434 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:28:24.274320+0000", 2024-02-09T17:35:17.434 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0945000000000003e-05 2024-02-09T17:35:17.434 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:17.434 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:35:17.434 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:17.434 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.434 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.434 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24817.0:154466 3.e e244/238 3:7e0367ee:::benchmark_data_smithi179_90440_object22507:head v 244'9727, mlcod=209'1415)", 2024-02-09T17:35:17.434 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:32:23.126145+0000", 2024-02-09T17:35:17.434 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 174.22949603800001, 2024-02-09T17:35:17.434 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.055461469999999999, 2024-02-09T17:35:17.435 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:35:17.435 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:35:17.435 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:35:17.435 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.435 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:35:17.435 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:23.126145+0000", 2024-02-09T17:35:17.435 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:17.435 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.435 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.435 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:35:17.435 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:23.126145+0000", 2024-02-09T17:35:17.435 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:17.435 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.435 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.436 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:35:17.436 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:23.126146+0000", 2024-02-09T17:35:17.436 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1480000000000001e-06 2024-02-09T17:35:17.436 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.436 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.436 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:35:17.436 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:23.126156+0000", 2024-02-09T17:35:17.436 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7839999999999998e-06 2024-02-09T17:35:17.436 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.436 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.436 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:35:17.436 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:23.126170+0000", 2024-02-09T17:35:17.436 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3949e-05 2024-02-09T17:35:17.436 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.436 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.437 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:35:17.437 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:23.126180+0000", 2024-02-09T17:35:17.437 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6889999999999999e-06 2024-02-09T17:35:17.437 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.437 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.437 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:35:17.437 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:23.136725+0000", 2024-02-09T17:35:17.437 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010545262 2024-02-09T17:35:17.437 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.437 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.437 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:35:17.437 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:23.136756+0000", 2024-02-09T17:35:17.437 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1208999999999999e-05 2024-02-09T17:35:17.437 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.437 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.438 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:35:17.438 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:23.181553+0000", 2024-02-09T17:35:17.438 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.044797030000000002 2024-02-09T17:35:17.438 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.438 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.438 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:35:17.438 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:23.181607+0000", 2024-02-09T17:35:17.438 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3399000000000002e-05 2024-02-09T17:35:17.438 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:17.438 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:35:17.438 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:17.438 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.438 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.438 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24817.0:154468 3.e e244/238 3:79737029:::benchmark_data_smithi179_90440_object22509:head v 244'9728, mlcod=209'1415)", 2024-02-09T17:35:17.438 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:32:23.130657+0000", 2024-02-09T17:35:17.439 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 174.22498440199999, 2024-02-09T17:35:17.439 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.050984070999999999, 2024-02-09T17:35:17.439 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:35:17.439 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:35:17.439 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:35:17.439 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.439 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:35:17.439 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:23.130657+0000", 2024-02-09T17:35:17.439 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:17.439 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.439 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.439 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:35:17.439 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:23.130657+0000", 2024-02-09T17:35:17.439 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:17.439 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.439 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.440 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:35:17.440 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:23.130658+0000", 2024-02-09T17:35:17.440 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8999999999999995e-07 2024-02-09T17:35:17.440 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.440 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.440 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:35:17.440 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:23.130664+0000", 2024-02-09T17:35:17.440 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.891e-06 2024-02-09T17:35:17.440 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.440 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.440 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:35:17.440 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:23.130674+0000", 2024-02-09T17:35:17.440 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0382e-05 2024-02-09T17:35:17.440 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.440 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.440 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:35:17.441 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:23.130683+0000", 2024-02-09T17:35:17.441 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8750000000000006e-06 2024-02-09T17:35:17.441 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.441 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.441 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:35:17.441 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:23.147378+0000", 2024-02-09T17:35:17.441 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.016695504999999999 2024-02-09T17:35:17.441 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.441 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.441 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:35:17.441 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:23.147403+0000", 2024-02-09T17:35:17.441 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5089e-05 2024-02-09T17:35:17.441 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.441 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.441 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:35:17.441 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:23.181613+0000", 2024-02-09T17:35:17.442 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.034209563999999998 2024-02-09T17:35:17.442 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.442 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.442 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:35:17.442 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:23.181641+0000", 2024-02-09T17:35:17.442 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7875e-05 2024-02-09T17:35:17.442 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:17.442 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:35:17.442 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:17.442 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.442 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.442 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24817.0:175370 3.e e260/238 3:773b4c0d:::benchmark_data_smithi179_90440_object43411:head v 260'10987, mlcod=210'2674)", 2024-02-09T17:35:17.442 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:32:58.483184+0000", 2024-02-09T17:35:17.442 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 138.87245674100001, 2024-02-09T17:35:17.442 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.052519464000000002, 2024-02-09T17:35:17.443 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:35:17.443 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:35:17.443 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:35:17.443 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.443 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:35:17.443 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:58.483184+0000", 2024-02-09T17:35:17.443 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:17.443 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.443 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.443 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:35:17.443 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:58.483184+0000", 2024-02-09T17:35:17.443 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:17.443 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.443 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.444 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:35:17.444 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:58.483186+0000", 2024-02-09T17:35:17.444 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.756e-06 2024-02-09T17:35:17.444 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.444 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.444 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:35:17.444 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:58.483199+0000", 2024-02-09T17:35:17.444 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2469999999999999e-05 2024-02-09T17:35:17.444 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.444 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.444 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:35:17.444 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:58.483214+0000", 2024-02-09T17:35:17.444 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4857e-05 2024-02-09T17:35:17.444 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.444 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.445 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:35:17.445 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:58.483225+0000", 2024-02-09T17:35:17.445 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1003e-05 2024-02-09T17:35:17.445 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.445 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.445 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:35:17.445 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:58.487567+0000", 2024-02-09T17:35:17.445 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0043421279999999998 2024-02-09T17:35:17.445 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.445 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.445 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:35:17.445 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:58.487596+0000", 2024-02-09T17:35:17.445 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.8833e-05 2024-02-09T17:35:17.445 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.445 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.446 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:35:17.446 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:58.535642+0000", 2024-02-09T17:35:17.446 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.048046012999999999 2024-02-09T17:35:17.446 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.446 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.446 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:35:17.446 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:58.535704+0000", 2024-02-09T17:35:17.446 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2403999999999994e-05 2024-02-09T17:35:17.446 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:17.446 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:35:17.446 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:17.446 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.446 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.446 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24817.0:175389 3.e e260/238 3:7d444f99:::benchmark_data_smithi179_90440_object43430:head v 260'10988, mlcod=210'2674)", 2024-02-09T17:35:17.447 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:32:58.483779+0000", 2024-02-09T17:35:17.447 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 138.87186251700001, 2024-02-09T17:35:17.447 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.051957446999999997, 2024-02-09T17:35:17.447 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:35:17.447 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:35:17.447 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:35:17.447 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.447 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:35:17.447 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:58.483779+0000", 2024-02-09T17:35:17.447 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:17.447 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.447 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.447 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:35:17.447 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:58.483779+0000", 2024-02-09T17:35:17.447 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:17.448 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.448 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.448 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:35:17.448 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:58.483779+0000", 2024-02-09T17:35:17.448 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4499999999999996e-07 2024-02-09T17:35:17.448 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.448 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.448 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:35:17.448 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:58.483786+0000", 2024-02-09T17:35:17.448 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7179999999999999e-06 2024-02-09T17:35:17.448 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.448 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.448 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:35:17.449 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:58.483799+0000", 2024-02-09T17:35:17.449 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2581e-05 2024-02-09T17:35:17.449 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.449 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.449 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:35:17.449 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:58.483807+0000", 2024-02-09T17:35:17.449 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7999999999999999e-06 2024-02-09T17:35:17.449 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.449 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.449 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:35:17.449 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:58.497301+0000", 2024-02-09T17:35:17.449 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013494257000000001 2024-02-09T17:35:17.449 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.449 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.449 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:35:17.450 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:58.497340+0000", 2024-02-09T17:35:17.450 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9452000000000001e-05 2024-02-09T17:35:17.450 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.450 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.450 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:35:17.450 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:58.535707+0000", 2024-02-09T17:35:17.450 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.038367218000000002 2024-02-09T17:35:17.450 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.450 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.450 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:35:17.451 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:58.535736+0000", 2024-02-09T17:35:17.451 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.8676000000000001e-05 2024-02-09T17:35:17.451 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:17.451 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:35:17.451 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:17.451 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.451 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.451 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24817.0:182401 3.e e269/238 3:730d2496:::benchmark_data_smithi179_90440_object50442:head v 269'11426, mlcod=210'3112)", 2024-02-09T17:35:17.451 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:33:10.434332+0000", 2024-02-09T17:35:17.451 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 126.921309198, 2024-02-09T17:35:17.451 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.055432381000000003, 2024-02-09T17:35:17.451 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:35:17.451 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:35:17.451 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:35:17.452 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.452 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:35:17.452 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:33:10.434332+0000", 2024-02-09T17:35:17.452 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:17.452 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.452 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.452 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:35:17.452 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:33:10.434332+0000", 2024-02-09T17:35:17.452 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:17.452 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.452 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.452 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:35:17.452 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:33:10.434334+0000", 2024-02-09T17:35:17.453 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.578e-06 2024-02-09T17:35:17.453 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.453 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.453 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:35:17.453 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:33:10.434342+0000", 2024-02-09T17:35:17.453 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7979999999999995e-06 2024-02-09T17:35:17.453 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.453 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.453 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:35:17.453 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:33:10.434358+0000", 2024-02-09T17:35:17.453 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5747e-05 2024-02-09T17:35:17.453 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.453 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.454 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:35:17.454 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:33:10.434367+0000", 2024-02-09T17:35:17.454 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8699999999999998e-06 2024-02-09T17:35:17.454 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.454 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.454 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:35:17.454 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:33:10.453860+0000", 2024-02-09T17:35:17.454 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.019492678999999999 2024-02-09T17:35:17.454 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.454 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.454 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:35:17.454 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:33:10.453880+0000", 2024-02-09T17:35:17.454 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0590000000000001e-05 2024-02-09T17:35:17.454 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.454 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.455 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:35:17.455 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:33:10.489707+0000", 2024-02-09T17:35:17.455 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.035826597000000002 2024-02-09T17:35:17.455 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:17.455 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:17.455 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:35:17.455 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:33:10.489764+0000", 2024-02-09T17:35:17.455 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7522000000000003e-05 2024-02-09T17:35:17.455 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:17.455 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:35:17.455 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:17.455 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:17.455 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:35:17.455 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:35:17.456 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:35:17.483 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:18.066 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:17 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:35:18.066 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:17 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:35:18.066 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:17 smithi179 ceph-mon[43975]: pgmap v869: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 32 active+clean, 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped; 751 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 52 op/s; 1/6 objects degraded (16.667%); 0 B/s, 383 objects/s recovering 2024-02-09T17:35:18.066 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:17 smithi179 ceph-mon[43975]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:35:18.066 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:17 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3366324012' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:35:18.066 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:17 smithi179 ceph-mon[43975]: osdmap e314: 12 total, 12 up, 12 in 2024-02-09T17:35:18.067 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:17 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/456877894' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-09T17:35:18.091 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.10 dump_ops_in_flight 2024-02-09T17:35:18.131 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:35:18.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:18.134 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:35:18.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:17 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:35:18.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:17 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:35:18.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:17 smithi162 ceph-mon[44304]: pgmap v869: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 32 active+clean, 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped; 751 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 52 op/s; 1/6 objects degraded (16.667%); 0 B/s, 383 objects/s recovering 2024-02-09T17:35:18.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:17 smithi162 ceph-mon[44304]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:35:18.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:17 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3366324012' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:35:18.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:17 smithi162 ceph-mon[44304]: osdmap e314: 12 total, 12 up, 12 in 2024-02-09T17:35:18.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:17 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/456877894' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-09T17:35:18.234 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:35:18.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:17 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:35:18.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:17 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:35:18.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:17 smithi099 ceph-mon[39420]: pgmap v869: 37 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 32 active+clean, 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped; 751 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 52 op/s; 1/6 objects degraded (16.667%); 0 B/s, 383 objects/s recovering 2024-02-09T17:35:18.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:17 smithi099 ceph-mon[39420]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:35:18.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:17 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3366324012' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:35:18.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:17 smithi099 ceph-mon[39420]: osdmap e314: 12 total, 12 up, 12 in 2024-02-09T17:35:18.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:17 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/456877894' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-09T17:35:18.261 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:18.289 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:35:18.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:18.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:35:18.308 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'unique_pool_1' removed 2024-02-09T17:35:18.394 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:35:18.434 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:18.464 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:35:18.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:18.467 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:35:18.479 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:35:18.568 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:35:18.596 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:18.856 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:18 smithi099 ceph-mon[39420]: 1.12 scrub starts 2024-02-09T17:35:18.857 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:18 smithi099 ceph-mon[39420]: 1.12 scrub ok 2024-02-09T17:35:19.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:18 smithi179 ceph-mon[43975]: 1.12 scrub starts 2024-02-09T17:35:19.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:18 smithi179 ceph-mon[43975]: 1.12 scrub ok 2024-02-09T17:35:19.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:18 smithi179 ceph-mon[43975]: 1.d scrub starts 2024-02-09T17:35:19.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:18 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/456877894' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-02-09T17:35:19.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:18 smithi179 ceph-mon[43975]: osdmap e315: 12 total, 12 up, 12 in 2024-02-09T17:35:19.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:18 smithi162 ceph-mon[44304]: 1.12 scrub starts 2024-02-09T17:35:19.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:18 smithi162 ceph-mon[44304]: 1.12 scrub ok 2024-02-09T17:35:19.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:18 smithi162 ceph-mon[44304]: 1.d scrub starts 2024-02-09T17:35:19.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:18 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/456877894' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-02-09T17:35:19.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:18 smithi162 ceph-mon[44304]: osdmap e315: 12 total, 12 up, 12 in 2024-02-09T17:35:19.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:18 smithi099 ceph-mon[39420]: 1.d scrub starts 2024-02-09T17:35:19.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:18 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/456877894' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-02-09T17:35:19.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:18 smithi099 ceph-mon[39420]: osdmap e315: 12 total, 12 up, 12 in 2024-02-09T17:35:19.445 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:35:19.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:19.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:35:19.549 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:35:19.576 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:19.594 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:35:19.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:19.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:35:19.698 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:35:19.726 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:19.988 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:19 smithi179 ceph-mon[43975]: 1.d scrub ok 2024-02-09T17:35:19.988 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:19 smithi179 ceph-mon[43975]: pgmap v872: 21 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 20 active+clean; 673 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1/6 objects degraded (16.667%) 2024-02-09T17:35:19.988 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:19 smithi179 ceph-mon[43975]: 1.14 scrub starts 2024-02-09T17:35:19.988 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:19 smithi179 ceph-mon[43975]: 1.14 scrub ok 2024-02-09T17:35:19.988 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:19 smithi179 ceph-mon[43975]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-09T17:35:19.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:19 smithi099 ceph-mon[39420]: 1.d scrub ok 2024-02-09T17:35:19.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:19 smithi099 ceph-mon[39420]: pgmap v872: 21 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 20 active+clean; 673 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1/6 objects degraded (16.667%) 2024-02-09T17:35:19.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:19 smithi099 ceph-mon[39420]: 1.14 scrub starts 2024-02-09T17:35:19.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:19 smithi099 ceph-mon[39420]: 1.14 scrub ok 2024-02-09T17:35:19.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:19 smithi099 ceph-mon[39420]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-09T17:35:20.078 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:35:20.081 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:20.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:35:20.086 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-02-09T17:35:20.086 INFO:tasks.radosbench:Beginning radosbench... 2024-02-09T17:35:20.087 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-02-09T17:35:20.087 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd pool create unique_pool_2 16 2024-02-09T17:35:20.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:19 smithi162 ceph-mon[44304]: 1.d scrub ok 2024-02-09T17:35:20.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:19 smithi162 ceph-mon[44304]: pgmap v872: 21 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 20 active+clean; 673 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1/6 objects degraded (16.667%) 2024-02-09T17:35:20.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:19 smithi162 ceph-mon[44304]: 1.14 scrub starts 2024-02-09T17:35:20.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:19 smithi162 ceph-mon[44304]: 1.14 scrub ok 2024-02-09T17:35:20.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:19 smithi162 ceph-mon[44304]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-09T17:35:20.183 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:35:20.213 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:20.232 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:35:20.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:20.236 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:35:20.253 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:35:20.253 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-02-09T17:35:20.253 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 0 2024-02-09T17:35:20.253 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:35:20.337 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:35:20.364 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:20.913 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:20 smithi179 ceph-mon[43975]: 1.e scrub starts 2024-02-09T17:35:20.913 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:20 smithi179 ceph-mon[43975]: 1.e scrub ok 2024-02-09T17:35:20.913 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:20 smithi179 ceph-mon[43975]: osdmap e316: 12 total, 12 up, 12 in 2024-02-09T17:35:20.913 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:20 smithi179 ceph-mon[43975]: 1.4 scrub starts 2024-02-09T17:35:20.914 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:20 smithi179 ceph-mon[43975]: 1.4 scrub ok 2024-02-09T17:35:20.939 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.10 dump_blocked_ops 2024-02-09T17:35:20.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:20 smithi099 ceph-mon[39420]: 1.e scrub starts 2024-02-09T17:35:20.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:20 smithi099 ceph-mon[39420]: 1.e scrub ok 2024-02-09T17:35:20.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:20 smithi099 ceph-mon[39420]: osdmap e316: 12 total, 12 up, 12 in 2024-02-09T17:35:20.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:20 smithi099 ceph-mon[39420]: 1.4 scrub starts 2024-02-09T17:35:20.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:20 smithi099 ceph-mon[39420]: 1.4 scrub ok 2024-02-09T17:35:21.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:20 smithi162 ceph-mon[44304]: 1.e scrub starts 2024-02-09T17:35:21.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:20 smithi162 ceph-mon[44304]: 1.e scrub ok 2024-02-09T17:35:21.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:20 smithi162 ceph-mon[44304]: osdmap e316: 12 total, 12 up, 12 in 2024-02-09T17:35:21.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:20 smithi162 ceph-mon[44304]: 1.4 scrub starts 2024-02-09T17:35:21.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:20 smithi162 ceph-mon[44304]: 1.4 scrub ok 2024-02-09T17:35:21.239 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:35:22.132 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:21 smithi099 ceph-mon[39420]: pgmap v874: 21 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 20 active+clean; 629 KiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1/6 objects degraded (16.667%) 2024-02-09T17:35:22.132 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:21 smithi099 ceph-mon[39420]: 1.b scrub starts 2024-02-09T17:35:22.132 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:21 smithi099 ceph-mon[39420]: osdmap e317: 12 total, 12 up, 12 in 2024-02-09T17:35:22.132 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:21 smithi099 ceph-mon[39420]: 1.b scrub ok 2024-02-09T17:35:22.132 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:21 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 1/6 objects degraded (16.667%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-02-09T17:35:22.138 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:21 smithi179 ceph-mon[43975]: pgmap v874: 21 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 20 active+clean; 629 KiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1/6 objects degraded (16.667%) 2024-02-09T17:35:22.139 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:21 smithi179 ceph-mon[43975]: 1.b scrub starts 2024-02-09T17:35:22.139 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:21 smithi179 ceph-mon[43975]: osdmap e317: 12 total, 12 up, 12 in 2024-02-09T17:35:22.139 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:21 smithi179 ceph-mon[43975]: 1.b scrub ok 2024-02-09T17:35:22.139 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:21 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 1/6 objects degraded (16.667%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-02-09T17:35:22.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:21 smithi162 ceph-mon[44304]: pgmap v874: 21 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 20 active+clean; 629 KiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1/6 objects degraded (16.667%) 2024-02-09T17:35:22.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:21 smithi162 ceph-mon[44304]: 1.b scrub starts 2024-02-09T17:35:22.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:21 smithi162 ceph-mon[44304]: osdmap e317: 12 total, 12 up, 12 in 2024-02-09T17:35:22.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:21 smithi162 ceph-mon[44304]: 1.b scrub ok 2024-02-09T17:35:22.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:21 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 1/6 objects degraded (16.667%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-02-09T17:35:22.608 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:35:22.627 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:35:22.646 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:35:22.665 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:35:22.686 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:35:22.706 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:35:22.727 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:35:22.741 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set noscrub 2024-02-09T17:35:22.748 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:35:22.772 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:35:22.812 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:35:22.832 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:35:22.862 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:35:22.864 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:35:22.864 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:35:22.864 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:35:22.864 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:35:22.864 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:35:22.864 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:35:22.864 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:35:22.864 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:35:22.864 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:35:22.864 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:35:22.864 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:35:22.865 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:35:23.109 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T17:35:23.109 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:35:23.128 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:22 smithi179 ceph-mon[43975]: 1.3 scrub starts 2024-02-09T17:35:23.128 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:22 smithi179 ceph-mon[43975]: 1.3 scrub ok 2024-02-09T17:35:23.128 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:22 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/389391652' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:35:23.128 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:22 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:35:23.129 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:22 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:35:23.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:22 smithi162 ceph-mon[44304]: 1.3 scrub starts 2024-02-09T17:35:23.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:22 smithi162 ceph-mon[44304]: 1.3 scrub ok 2024-02-09T17:35:23.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:22 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/389391652' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:35:23.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:22 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:35:23.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:22 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:35:23.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:22 smithi099 ceph-mon[39420]: 1.3 scrub starts 2024-02-09T17:35:23.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:22 smithi099 ceph-mon[39420]: 1.3 scrub ok 2024-02-09T17:35:23.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:22 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/389391652' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:35:23.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:22 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:35:23.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:22 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:35:23.692 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:35:23.735 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.10 dump_historic_ops 2024-02-09T17:35:23.863 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.5 to 1 (10000) 2024-02-09T17:35:23.958 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:23 smithi099 ceph-mon[39420]: pgmap v876: 21 pgs: 21 active+clean; 629 KiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:35:23.958 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:23 smithi099 ceph-mon[39420]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/6 objects degraded (16.667%), 1 pg degraded, 1 pg undersized) 2024-02-09T17:35:23.958 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:23 smithi099 ceph-mon[39420]: Cluster is now healthy 2024-02-09T17:35:23.958 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:23 smithi099 ceph-mon[39420]: osdmap e318: 12 total, 12 up, 12 in 2024-02-09T17:35:23.959 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:23 smithi099 ceph-mon[39420]: 1.13 scrub starts 2024-02-09T17:35:23.959 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:23 smithi099 ceph-mon[39420]: 1.13 scrub ok 2024-02-09T17:35:23.959 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:23 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2289899534' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-09T17:35:23.999 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:35:24.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:23 smithi179 ceph-mon[43975]: pgmap v876: 21 pgs: 21 active+clean; 629 KiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:35:24.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:23 smithi179 ceph-mon[43975]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/6 objects degraded (16.667%), 1 pg degraded, 1 pg undersized) 2024-02-09T17:35:24.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:23 smithi179 ceph-mon[43975]: Cluster is now healthy 2024-02-09T17:35:24.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:23 smithi179 ceph-mon[43975]: osdmap e318: 12 total, 12 up, 12 in 2024-02-09T17:35:24.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:23 smithi179 ceph-mon[43975]: 1.13 scrub starts 2024-02-09T17:35:24.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:23 smithi179 ceph-mon[43975]: 1.13 scrub ok 2024-02-09T17:35:24.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:23 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2289899534' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-09T17:35:24.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:23 smithi162 ceph-mon[44304]: pgmap v876: 21 pgs: 21 active+clean; 629 KiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:35:24.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:23 smithi162 ceph-mon[44304]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/6 objects degraded (16.667%), 1 pg degraded, 1 pg undersized) 2024-02-09T17:35:24.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:23 smithi162 ceph-mon[44304]: Cluster is now healthy 2024-02-09T17:35:24.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:23 smithi162 ceph-mon[44304]: osdmap e318: 12 total, 12 up, 12 in 2024-02-09T17:35:24.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:23 smithi162 ceph-mon[44304]: 1.13 scrub starts 2024-02-09T17:35:24.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:23 smithi162 ceph-mon[44304]: 1.13 scrub ok 2024-02-09T17:35:24.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:23 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2289899534' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-09T17:35:24.919 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:24 smithi099 ceph-mon[39420]: 1.0 scrub starts 2024-02-09T17:35:24.919 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:24 smithi099 ceph-mon[39420]: 1.0 scrub ok 2024-02-09T17:35:24.920 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:24 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2289899534' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-09T17:35:24.920 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:24 smithi099 ceph-mon[39420]: osdmap e319: 12 total, 12 up, 12 in 2024-02-09T17:35:24.920 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:24 smithi099 ceph-mon[39420]: 1.c deep-scrub starts 2024-02-09T17:35:24.920 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:24 smithi099 ceph-mon[39420]: 1.c deep-scrub ok 2024-02-09T17:35:24.945 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:35:24.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:24.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:35:24.952 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 6 1 2024-02-09T17:35:24.961 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:35:25.049 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:35:25.075 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:25.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:24 smithi179 ceph-mon[43975]: 1.0 scrub starts 2024-02-09T17:35:25.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:24 smithi179 ceph-mon[43975]: 1.0 scrub ok 2024-02-09T17:35:25.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:24 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2289899534' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-09T17:35:25.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:24 smithi179 ceph-mon[43975]: osdmap e319: 12 total, 12 up, 12 in 2024-02-09T17:35:25.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:24 smithi179 ceph-mon[43975]: 1.c deep-scrub starts 2024-02-09T17:35:25.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:24 smithi179 ceph-mon[43975]: 1.c deep-scrub ok 2024-02-09T17:35:25.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:24 smithi162 ceph-mon[44304]: 1.0 scrub starts 2024-02-09T17:35:25.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:24 smithi162 ceph-mon[44304]: 1.0 scrub ok 2024-02-09T17:35:25.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:24 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2289899534' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-09T17:35:25.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:24 smithi162 ceph-mon[44304]: osdmap e319: 12 total, 12 up, 12 in 2024-02-09T17:35:25.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:24 smithi162 ceph-mon[44304]: 1.c deep-scrub starts 2024-02-09T17:35:25.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:24 smithi162 ceph-mon[44304]: 1.c deep-scrub ok 2024-02-09T17:35:25.274 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:35:25.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:25.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:35:25.378 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:35:25.411 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:25.434 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:35:25.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:25.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:35:25.539 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:35:25.569 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:25.588 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:35:25.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:25.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:35:25.678 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:35:25.693 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:35:25.747 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:25.864 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T17:35:25.864 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:35:26.135 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:25 smithi099 ceph-mon[39420]: pgmap v879: 21 pgs: 21 active+clean; 629 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s, 0 objects/s recovering 2024-02-09T17:35:26.135 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:25 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:35:26.136 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:25 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:35:26.150 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:35:26.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:25 smithi179 ceph-mon[43975]: pgmap v879: 21 pgs: 21 active+clean; 629 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s, 0 objects/s recovering 2024-02-09T17:35:26.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:25 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:35:26.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:25 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:35:26.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:25 smithi162 ceph-mon[44304]: pgmap v879: 21 pgs: 21 active+clean; 629 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s, 0 objects/s recovering 2024-02-09T17:35:26.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:25 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:35:26.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:25 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:35:26.549 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.11 dump_ops_in_flight 2024-02-09T17:35:26.809 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:35:27.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:26 smithi179 ceph-mon[43975]: osdmap e320: 12 total, 12 up, 12 in 2024-02-09T17:35:27.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:26 smithi179 ceph-mon[43975]: osdmap e321: 12 total, 12 up, 12 in 2024-02-09T17:35:27.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:26 smithi162 ceph-mon[44304]: osdmap e320: 12 total, 12 up, 12 in 2024-02-09T17:35:27.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:26 smithi162 ceph-mon[44304]: osdmap e321: 12 total, 12 up, 12 in 2024-02-09T17:35:27.197 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:35:27.199 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:26 smithi099 ceph-mon[39420]: osdmap e320: 12 total, 12 up, 12 in 2024-02-09T17:35:27.199 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:26 smithi099 ceph-mon[39420]: osdmap e321: 12 total, 12 up, 12 in 2024-02-09T17:35:27.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:27.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:35:27.302 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:35:27.330 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:27.578 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:35:27.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:27.581 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:35:27.682 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:35:27.709 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:28.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:27 smithi179 ceph-mon[43975]: pgmap v882: 21 pgs: 21 active+clean; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:35:28.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:27 smithi162 ceph-mon[44304]: pgmap v882: 21 pgs: 21 active+clean; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:35:28.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:27 smithi099 ceph-mon[39420]: pgmap v882: 21 pgs: 21 active+clean; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:35:28.555 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T17:35:28.555 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:35:28.556 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:35:28.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:28.560 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:35:28.661 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:35:28.693 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:28.714 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:35:28.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:28.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:35:28.736 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:35:28.818 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:35:28.849 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:28.869 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:35:28.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:28.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:35:28.889 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:28 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/142052707' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-02-09T17:35:28.897 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'unique_pool_2' created 2024-02-09T17:35:28.973 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:35:29.003 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:29.042 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:35:29.044 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:29.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:35:29.145 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:35:29.152 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:28 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/142052707' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-02-09T17:35:29.173 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:28 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/142052707' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-02-09T17:35:29.175 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:29.178 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.11 dump_blocked_ops 2024-02-09T17:35:29.195 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:35:29.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:29.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:35:29.299 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:35:29.325 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:29.346 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:35:29.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:29.349 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:35:29.442 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:35:29.450 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:35:29.478 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:29.726 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:35:29.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:29.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:35:29.830 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:35:29.856 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:29.894 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is set 2024-02-09T17:35:29.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:29 smithi179 ceph-mon[43975]: pgmap v883: 21 pgs: 21 active+clean; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:35:29.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:29 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/142052707' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-02-09T17:35:29.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:29 smithi179 ceph-mon[43975]: osdmap e322: 12 total, 12 up, 12 in 2024-02-09T17:35:29.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:29 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2937292046' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:35:29.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:29 smithi099 ceph-mon[39420]: pgmap v883: 21 pgs: 21 active+clean; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:35:29.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:29 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/142052707' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-02-09T17:35:29.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:29 smithi099 ceph-mon[39420]: osdmap e322: 12 total, 12 up, 12 in 2024-02-09T17:35:29.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:29 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2937292046' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:35:30.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:29 smithi162 ceph-mon[44304]: pgmap v883: 21 pgs: 21 active+clean; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:35:30.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:29 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/142052707' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-02-09T17:35:30.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:29 smithi162 ceph-mon[44304]: osdmap e322: 12 total, 12 up, 12 in 2024-02-09T17:35:30.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:29 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2937292046' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:35:30.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:30 smithi179 ceph-mon[43975]: 1.2 scrub starts 2024-02-09T17:35:30.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:30 smithi179 ceph-mon[43975]: 1.2 scrub ok 2024-02-09T17:35:30.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:30 smithi179 ceph-mon[43975]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:35:30.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:30 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2937292046' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:35:30.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:30 smithi179 ceph-mon[43975]: osdmap e323: 12 total, 12 up, 12 in 2024-02-09T17:35:30.934 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:35:30.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:30.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:35:31.038 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:35:31.067 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:31.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:30 smithi162 ceph-mon[44304]: 1.2 scrub starts 2024-02-09T17:35:31.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:30 smithi162 ceph-mon[44304]: 1.2 scrub ok 2024-02-09T17:35:31.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:30 smithi162 ceph-mon[44304]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:35:31.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:30 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2937292046' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:35:31.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:30 smithi162 ceph-mon[44304]: osdmap e323: 12 total, 12 up, 12 in 2024-02-09T17:35:31.217 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T17:35:31.217 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:35:31.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:30 smithi099 ceph-mon[39420]: 1.2 scrub starts 2024-02-09T17:35:31.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:30 smithi099 ceph-mon[39420]: 1.2 scrub ok 2024-02-09T17:35:31.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:30 smithi099 ceph-mon[39420]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:35:31.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:30 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2937292046' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:35:31.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:30 smithi099 ceph-mon[39420]: osdmap e323: 12 total, 12 up, 12 in 2024-02-09T17:35:31.298 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.6 to 1 (10000) 2024-02-09T17:35:31.944 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.11 dump_historic_ops 2024-02-09T17:35:32.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:31 smithi179 ceph-mon[43975]: 1.9 deep-scrub starts 2024-02-09T17:35:32.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:31 smithi179 ceph-mon[43975]: 1.9 deep-scrub ok 2024-02-09T17:35:32.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:31 smithi179 ceph-mon[43975]: pgmap v886: 37 pgs: 3 creating+activating, 6 unknown, 28 active+clean; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:35:32.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:31 smithi179 ceph-mon[43975]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-09T17:35:32.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:31 smithi179 ceph-mon[43975]: osdmap e324: 12 total, 12 up, 12 in 2024-02-09T17:35:32.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:31 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1477163510' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-09T17:35:32.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:31 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1477163510' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-09T17:35:32.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:31 smithi179 ceph-mon[43975]: osdmap e325: 12 total, 12 up, 12 in 2024-02-09T17:35:32.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:31 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:35:32.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:31 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:35:32.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:31 smithi162 ceph-mon[44304]: 1.9 deep-scrub starts 2024-02-09T17:35:32.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:31 smithi162 ceph-mon[44304]: 1.9 deep-scrub ok 2024-02-09T17:35:32.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:31 smithi162 ceph-mon[44304]: pgmap v886: 37 pgs: 3 creating+activating, 6 unknown, 28 active+clean; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:35:32.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:31 smithi162 ceph-mon[44304]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-09T17:35:32.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:31 smithi162 ceph-mon[44304]: osdmap e324: 12 total, 12 up, 12 in 2024-02-09T17:35:32.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:31 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1477163510' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-09T17:35:32.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:31 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1477163510' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-09T17:35:32.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:31 smithi162 ceph-mon[44304]: osdmap e325: 12 total, 12 up, 12 in 2024-02-09T17:35:32.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:31 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:35:32.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:31 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:35:32.204 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:35:32.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:31 smithi099 ceph-mon[39420]: 1.9 deep-scrub starts 2024-02-09T17:35:32.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:31 smithi099 ceph-mon[39420]: 1.9 deep-scrub ok 2024-02-09T17:35:32.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:31 smithi099 ceph-mon[39420]: pgmap v886: 37 pgs: 3 creating+activating, 6 unknown, 28 active+clean; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:35:32.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:31 smithi099 ceph-mon[39420]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-09T17:35:32.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:31 smithi099 ceph-mon[39420]: osdmap e324: 12 total, 12 up, 12 in 2024-02-09T17:35:32.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:31 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1477163510' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-09T17:35:32.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:31 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1477163510' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-09T17:35:32.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:31 smithi099 ceph-mon[39420]: osdmap e325: 12 total, 12 up, 12 in 2024-02-09T17:35:32.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:31 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:35:32.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:31 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:35:32.410 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-02-09T17:35:32.646 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:35:32.651 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:35:32.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:32.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:35:32.674 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:35:32.693 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:35:32.712 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:35:32.733 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:35:32.753 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:35:32.754 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:35:32.774 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:35:32.783 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:32.794 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:35:32.814 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:35:32.835 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:35:32.855 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:35:32.874 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:35:32.876 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:35:32.876 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:35:32.876 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:35:32.876 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:35:32.876 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:35:32.876 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:35:32.876 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:35:32.876 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:35:32.876 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:35:32.876 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:35:32.876 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:35:32.877 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:35:32.981 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:35:32.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:32.983 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:35:33.084 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:35:33.111 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:33.140 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 7 1 2024-02-09T17:35:33.341 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:35:33.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:33 smithi179 ceph-mon[43975]: osdmap e326: 12 total, 12 up, 12 in 2024-02-09T17:35:33.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:33 smithi179 ceph-mon[43975]: pgmap v890: 37 pgs: 3 creating+activating, 1 unknown, 33 active+clean; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:35:33.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:33 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4240320635' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:35:33.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:33 smithi099 ceph-mon[39420]: osdmap e326: 12 total, 12 up, 12 in 2024-02-09T17:35:33.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:33 smithi099 ceph-mon[39420]: pgmap v890: 37 pgs: 3 creating+activating, 1 unknown, 33 active+clean; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:35:33.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:33 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4240320635' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:35:33.560 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:35:33.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:33.563 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:35:33.615 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-09T17:35:33.664 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:35:33.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:33 smithi162 ceph-mon[44304]: osdmap e326: 12 total, 12 up, 12 in 2024-02-09T17:35:33.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:33 smithi162 ceph-mon[44304]: pgmap v890: 37 pgs: 3 creating+activating, 1 unknown, 33 active+clean; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:35:33.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:33 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4240320635' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:35:33.692 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:33.791 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:35:33.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:33.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:35:33.903 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:35:33.935 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:33.957 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:35:33.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:33.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:35:34.049 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:35:34.049 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-02-09T17:35:34.049 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-02-09T17:35:34.049 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-02-09T17:35:34.049 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.049 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:173669 2.a 2:5049bf51:::benchmark_data_smithi179_77138_object40261:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-02-09T17:35:34.050 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:04.251927+0000", 2024-02-09T17:35:34.050 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 509.79410714400001, 2024-02-09T17:35:34.050 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.21543948900000001, 2024-02-09T17:35:34.050 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:35:34.050 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:35:34.050 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:35:34.050 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:35:34.050 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:35:34.050 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 173669 2024-02-09T17:35:34.050 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.050 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:35:34.050 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.051 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:35:34.051 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.251927+0000", 2024-02-09T17:35:34.051 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:34.051 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.051 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.051 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:35:34.051 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.251927+0000", 2024-02-09T17:35:34.051 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:34.051 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.051 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.051 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:35:34.051 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.251929+0000", 2024-02-09T17:35:34.052 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7579999999999999e-06 2024-02-09T17:35:34.052 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.052 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.052 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:35:34.052 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.251935+0000", 2024-02-09T17:35:34.052 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9329999999999998e-06 2024-02-09T17:35:34.052 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.052 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.052 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:35:34.052 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.251949+0000", 2024-02-09T17:35:34.052 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4268e-05 2024-02-09T17:35:34.053 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.053 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.053 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:35:34.053 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.251957+0000", 2024-02-09T17:35:34.053 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5290000000000008e-06 2024-02-09T17:35:34.053 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.053 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.053 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:35:34.053 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.252013+0000", 2024-02-09T17:35:34.053 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5619999999999999e-05 2024-02-09T17:35:34.053 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.053 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.053 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:35:34.054 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.252115+0000", 2024-02-09T17:35:34.054 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000102315 2024-02-09T17:35:34.054 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.054 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.054 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,9", 2024-02-09T17:35:34.054 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.252237+0000", 2024-02-09T17:35:34.054 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000121951 2024-02-09T17:35:34.054 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.054 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.054 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:35:34.054 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.256532+0000", 2024-02-09T17:35:34.055 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0042949540000000001 2024-02-09T17:35:34.055 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.055 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.055 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:35:34.055 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.259361+0000", 2024-02-09T17:35:34.055 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002829163 2024-02-09T17:35:34.055 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.055 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.055 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:35:34.055 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.467273+0000", 2024-02-09T17:35:34.055 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.207911866 2024-02-09T17:35:34.055 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.056 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.056 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:35:34.056 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.467324+0000", 2024-02-09T17:35:34.056 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0763000000000002e-05 2024-02-09T17:35:34.056 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.056 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.056 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:35:34.056 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.467366+0000", 2024-02-09T17:35:34.056 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2369e-05 2024-02-09T17:35:34.056 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:34.056 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:35:34.057 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:34.057 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.057 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.057 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:173689 2.a 2:55e666c2:::benchmark_data_smithi179_77138_object40281:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-02-09T17:35:34.057 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:04.273390+0000", 2024-02-09T17:35:34.057 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 509.77264383300002, 2024-02-09T17:35:34.057 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.201714904, 2024-02-09T17:35:34.057 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:35:34.057 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:35:34.057 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:35:34.057 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:35:34.058 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:35:34.058 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 173689 2024-02-09T17:35:34.058 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.058 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:35:34.058 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.058 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:35:34.058 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.273390+0000", 2024-02-09T17:35:34.058 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:34.058 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.058 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.059 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:35:34.059 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.273390+0000", 2024-02-09T17:35:34.059 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:34.059 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.059 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.059 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:35:34.059 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.273392+0000", 2024-02-09T17:35:34.059 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3340000000000001e-06 2024-02-09T17:35:34.059 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.059 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.059 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:35:34.060 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.273396+0000", 2024-02-09T17:35:34.060 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2640000000000002e-06 2024-02-09T17:35:34.060 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.060 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.060 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:35:34.060 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.273406+0000", 2024-02-09T17:35:34.060 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0135999999999999e-05 2024-02-09T17:35:34.060 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.060 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.060 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:35:34.060 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.273412+0000", 2024-02-09T17:35:34.060 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8289999999999996e-06 2024-02-09T17:35:34.061 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.061 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.061 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:35:34.061 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.273457+0000", 2024-02-09T17:35:34.061 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5225000000000002e-05 2024-02-09T17:35:34.061 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.061 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.061 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:35:34.061 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.273543+0000", 2024-02-09T17:35:34.061 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5606999999999995e-05 2024-02-09T17:35:34.061 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.062 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.062 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,9", 2024-02-09T17:35:34.062 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.273618+0000", 2024-02-09T17:35:34.062 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5615e-05 2024-02-09T17:35:34.062 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.062 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.062 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:35:34.062 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.277650+0000", 2024-02-09T17:35:34.062 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0040314699999999997 2024-02-09T17:35:34.062 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.062 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.062 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:35:34.063 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.288200+0000", 2024-02-09T17:35:34.063 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010550744000000001 2024-02-09T17:35:34.063 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.063 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.063 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:35:34.063 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.475013+0000", 2024-02-09T17:35:34.063 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.186812582 2024-02-09T17:35:34.063 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.063 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.063 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:35:34.063 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.475063+0000", 2024-02-09T17:35:34.063 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0197000000000001e-05 2024-02-09T17:35:34.063 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.064 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.064 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:35:34.064 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.475105+0000", 2024-02-09T17:35:34.064 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1900999999999997e-05 2024-02-09T17:35:34.064 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:34.064 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:35:34.064 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:34.064 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.064 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.064 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:173701 2.a 2:54bdf07e:::benchmark_data_smithi179_77138_object40293:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-02-09T17:35:34.064 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:04.288307+0000", 2024-02-09T17:35:34.064 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 509.75772740899998, 2024-02-09T17:35:34.065 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.19479152299999999, 2024-02-09T17:35:34.065 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:35:34.065 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:35:34.065 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:35:34.065 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:35:34.065 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:35:34.065 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 173701 2024-02-09T17:35:34.065 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.065 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:35:34.065 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.065 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:35:34.065 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.288307+0000", 2024-02-09T17:35:34.065 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:34.066 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.066 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.066 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:35:34.066 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.288307+0000", 2024-02-09T17:35:34.066 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:34.066 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.066 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.066 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:35:34.066 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.288308+0000", 2024-02-09T17:35:34.066 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.376e-06 2024-02-09T17:35:34.066 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.066 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.066 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:35:34.067 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.288314+0000", 2024-02-09T17:35:34.067 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8309999999999997e-06 2024-02-09T17:35:34.067 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.067 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.067 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:35:34.067 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.288330+0000", 2024-02-09T17:35:34.067 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5869999999999999e-05 2024-02-09T17:35:34.067 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.067 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.067 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:35:34.067 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.288338+0000", 2024-02-09T17:35:34.067 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7390000000000005e-06 2024-02-09T17:35:34.067 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.068 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.068 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:35:34.068 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.288385+0000", 2024-02-09T17:35:34.068 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6099e-05 2024-02-09T17:35:34.068 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.068 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.068 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:35:34.068 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.288505+0000", 2024-02-09T17:35:34.068 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000120623 2024-02-09T17:35:34.068 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.069 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.069 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,9", 2024-02-09T17:35:34.069 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.288636+0000", 2024-02-09T17:35:34.069 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000131013 2024-02-09T17:35:34.069 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.069 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.069 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:35:34.069 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.293939+0000", 2024-02-09T17:35:34.069 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0053033109999999998 2024-02-09T17:35:34.069 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.069 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.069 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:35:34.069 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.297980+0000", 2024-02-09T17:35:34.070 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00404004 2024-02-09T17:35:34.070 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.070 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.070 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:35:34.070 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.482999+0000", 2024-02-09T17:35:34.070 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.18501925899999999 2024-02-09T17:35:34.070 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.070 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.070 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:35:34.070 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.483057+0000", 2024-02-09T17:35:34.070 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8619000000000002e-05 2024-02-09T17:35:34.070 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.070 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.071 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:35:34.071 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.483098+0000", 2024-02-09T17:35:34.071 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0742999999999999e-05 2024-02-09T17:35:34.071 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:34.071 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:35:34.071 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:34.071 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.071 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.071 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14790.0:173712 2.a 2:5a5d0456:::benchmark_data_smithi179_77138_object40304:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-02-09T17:35:34.071 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:27:04.297430+0000", 2024-02-09T17:35:34.071 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 509.74860409299998, 2024-02-09T17:35:34.071 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.19325016, 2024-02-09T17:35:34.072 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:35:34.072 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:35:34.072 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:35:34.072 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14790", 2024-02-09T17:35:34.072 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/516962761", 2024-02-09T17:35:34.072 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 173712 2024-02-09T17:35:34.072 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.072 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:35:34.072 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.072 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:35:34.072 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.297430+0000", 2024-02-09T17:35:34.072 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:34.072 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.073 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.073 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:35:34.073 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.297430+0000", 2024-02-09T17:35:34.073 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:34.073 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.073 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.073 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:35:34.073 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.297432+0000", 2024-02-09T17:35:34.073 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8419999999999999e-06 2024-02-09T17:35:34.073 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.073 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.074 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:35:34.074 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.297439+0000", 2024-02-09T17:35:34.074 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7959999999999998e-06 2024-02-09T17:35:34.074 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.074 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.074 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:35:34.074 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.297470+0000", 2024-02-09T17:35:34.074 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1498000000000002e-05 2024-02-09T17:35:34.074 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.074 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.074 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:35:34.074 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.297480+0000", 2024-02-09T17:35:34.074 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.8740000000000007e-06 2024-02-09T17:35:34.075 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.075 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.075 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:35:34.075 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.298020+0000", 2024-02-09T17:35:34.075 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00054042499999999998 2024-02-09T17:35:34.075 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.075 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.075 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:35:34.075 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.298155+0000", 2024-02-09T17:35:34.075 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000134881 2024-02-09T17:35:34.075 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.076 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.076 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,9", 2024-02-09T17:35:34.076 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.298250+0000", 2024-02-09T17:35:34.076 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5222000000000004e-05 2024-02-09T17:35:34.076 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.076 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.076 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:35:34.076 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.305364+0000", 2024-02-09T17:35:34.076 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0071130430000000003 2024-02-09T17:35:34.076 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.076 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.076 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:35:34.077 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.308096+0000", 2024-02-09T17:35:34.077 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002732557 2024-02-09T17:35:34.077 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.077 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.077 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:35:34.077 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.490616+0000", 2024-02-09T17:35:34.077 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.18251959600000001 2024-02-09T17:35:34.077 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.077 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.077 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:35:34.078 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.490651+0000", 2024-02-09T17:35:34.078 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.5692999999999999e-05 2024-02-09T17:35:34.078 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.078 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.078 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:35:34.078 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:27:04.490680+0000", 2024-02-09T17:35:34.078 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.8733000000000001e-05 2024-02-09T17:35:34.078 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:34.078 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:35:34.078 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:34.078 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.078 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.078 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24817.0:17926 3.8 3:16bd42ec:::benchmark_data_smithi179_90440_object17925:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e208)", 2024-02-09T17:35:34.079 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:30:10.952580+0000", 2024-02-09T17:35:34.079 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 323.09345429699999, 2024-02-09T17:35:34.079 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.20727911800000001, 2024-02-09T17:35:34.079 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:35:34.079 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:35:34.079 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:35:34.079 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24817", 2024-02-09T17:35:34.079 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:35:34.079 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 17926 2024-02-09T17:35:34.079 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.079 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:35:34.079 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.079 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:35:34.080 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.952580+0000", 2024-02-09T17:35:34.080 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:34.080 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.080 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.080 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:35:34.080 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.952580+0000", 2024-02-09T17:35:34.080 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:34.080 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.080 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.081 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:35:34.081 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.952581+0000", 2024-02-09T17:35:34.081 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0419999999999999e-06 2024-02-09T17:35:34.081 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.081 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.081 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:35:34.081 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.952603+0000", 2024-02-09T17:35:34.081 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2453e-05 2024-02-09T17:35:34.081 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.081 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.081 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:35:34.081 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.952615+0000", 2024-02-09T17:35:34.082 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1567e-05 2024-02-09T17:35:34.082 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.082 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.082 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:35:34.082 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.952623+0000", 2024-02-09T17:35:34.082 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7200000000000006e-06 2024-02-09T17:35:34.082 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.082 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.082 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:35:34.082 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.952723+0000", 2024-02-09T17:35:34.082 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000100237 2024-02-09T17:35:34.082 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.082 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.083 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:35:34.083 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.952835+0000", 2024-02-09T17:35:34.083 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011260900000000001 2024-02-09T17:35:34.083 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.083 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.083 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:35:34.083 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.952954+0000", 2024-02-09T17:35:34.083 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000119127 2024-02-09T17:35:34.083 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.083 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.083 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:35:34.083 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.954299+0000", 2024-02-09T17:35:34.084 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001344114 2024-02-09T17:35:34.084 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.084 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.084 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:35:34.084 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.954781+0000", 2024-02-09T17:35:34.084 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00048271399999999997 2024-02-09T17:35:34.084 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.084 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.084 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:35:34.085 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:11.159746+0000", 2024-02-09T17:35:34.085 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.204964435 2024-02-09T17:35:34.085 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.085 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.085 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:35:34.085 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:11.159814+0000", 2024-02-09T17:35:34.085 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7871000000000001e-05 2024-02-09T17:35:34.085 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.085 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.085 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:35:34.085 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:11.159859+0000", 2024-02-09T17:35:34.085 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5228999999999997e-05 2024-02-09T17:35:34.085 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:34.086 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:35:34.086 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:34.086 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.086 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.086 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24817.0:17930 3.8 3:1fad3ea4:::benchmark_data_smithi179_90440_object17929:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e208)", 2024-02-09T17:35:34.086 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:30:10.954694+0000", 2024-02-09T17:35:34.086 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 323.09134007, 2024-02-09T17:35:34.086 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.20531634600000001, 2024-02-09T17:35:34.086 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:35:34.086 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:35:34.086 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:35:34.086 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24817", 2024-02-09T17:35:34.087 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:35:34.087 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 17930 2024-02-09T17:35:34.087 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.087 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:35:34.087 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.087 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:35:34.087 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.954694+0000", 2024-02-09T17:35:34.087 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:34.087 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.087 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.087 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:35:34.088 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.954694+0000", 2024-02-09T17:35:34.088 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:34.088 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.088 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.088 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:35:34.088 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.954695+0000", 2024-02-09T17:35:34.088 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.11e-06 2024-02-09T17:35:34.088 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.088 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.089 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:35:34.089 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.954714+0000", 2024-02-09T17:35:34.089 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9247000000000001e-05 2024-02-09T17:35:34.089 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.089 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.089 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:35:34.089 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.954726+0000", 2024-02-09T17:35:34.089 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.173e-05 2024-02-09T17:35:34.089 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.089 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.089 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:35:34.090 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.954734+0000", 2024-02-09T17:35:34.090 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6249999999999998e-06 2024-02-09T17:35:34.090 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.090 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.090 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:35:34.090 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.954829+0000", 2024-02-09T17:35:34.090 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5816999999999994e-05 2024-02-09T17:35:34.090 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.090 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.090 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:35:34.090 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.954946+0000", 2024-02-09T17:35:34.090 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011635500000000001 2024-02-09T17:35:34.091 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.091 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.091 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:35:34.091 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.955086+0000", 2024-02-09T17:35:34.091 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000140049 2024-02-09T17:35:34.091 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.091 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.091 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:35:34.091 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.955975+0000", 2024-02-09T17:35:34.091 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00088879900000000001 2024-02-09T17:35:34.091 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.091 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.092 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:35:34.092 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.959205+0000", 2024-02-09T17:35:34.092 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0032298769999999999 2024-02-09T17:35:34.092 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.092 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.092 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:35:34.092 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:11.159904+0000", 2024-02-09T17:35:34.092 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.20069910199999999 2024-02-09T17:35:34.092 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.092 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.092 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:35:34.093 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:11.159970+0000", 2024-02-09T17:35:34.093 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6291999999999995e-05 2024-02-09T17:35:34.093 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.093 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.093 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:35:34.093 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:11.160010+0000", 2024-02-09T17:35:34.093 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0343000000000003e-05 2024-02-09T17:35:34.093 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:34.093 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:35:34.093 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:34.093 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.093 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.094 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24817.0:17971 3.8 3:16a53f45:::benchmark_data_smithi179_90440_object17970:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e208)", 2024-02-09T17:35:34.094 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:30:10.965725+0000", 2024-02-09T17:35:34.094 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 323.08030857099999, 2024-02-09T17:35:34.094 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.194499441, 2024-02-09T17:35:34.094 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:35:34.094 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:35:34.094 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:35:34.094 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24817", 2024-02-09T17:35:34.094 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:35:34.094 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 17971 2024-02-09T17:35:34.094 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.095 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:35:34.095 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.095 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:35:34.095 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.965725+0000", 2024-02-09T17:35:34.095 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:34.095 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.095 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.095 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:35:34.095 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.965725+0000", 2024-02-09T17:35:34.095 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:34.095 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.095 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.096 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:35:34.096 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.965726+0000", 2024-02-09T17:35:34.096 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7000000000000003e-07 2024-02-09T17:35:34.096 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.096 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.096 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:35:34.096 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.965744+0000", 2024-02-09T17:35:34.096 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7935e-05 2024-02-09T17:35:34.096 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.096 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.097 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:35:34.097 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.965754+0000", 2024-02-09T17:35:34.097 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0190000000000001e-05 2024-02-09T17:35:34.097 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.097 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.097 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:35:34.097 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.965760+0000", 2024-02-09T17:35:34.097 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5550000000000002e-06 2024-02-09T17:35:34.097 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.097 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.097 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:35:34.097 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.965801+0000", 2024-02-09T17:35:34.098 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0980999999999999e-05 2024-02-09T17:35:34.098 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.098 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.098 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:35:34.098 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.965893+0000", 2024-02-09T17:35:34.098 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.1951000000000001e-05 2024-02-09T17:35:34.098 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.098 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.098 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:35:34.098 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.965984+0000", 2024-02-09T17:35:34.098 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.1308000000000001e-05 2024-02-09T17:35:34.098 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.099 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.099 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:35:34.099 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.966711+0000", 2024-02-09T17:35:34.099 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00072709599999999999 2024-02-09T17:35:34.099 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.099 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.099 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:35:34.099 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.967233+0000", 2024-02-09T17:35:34.099 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000521194 2024-02-09T17:35:34.099 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.099 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.099 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:35:34.100 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:11.160160+0000", 2024-02-09T17:35:34.100 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.19292753000000001 2024-02-09T17:35:34.100 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.100 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.100 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:35:34.100 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:11.160201+0000", 2024-02-09T17:35:34.100 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0998999999999998e-05 2024-02-09T17:35:34.100 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.100 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.101 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:35:34.101 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:11.160225+0000", 2024-02-09T17:35:34.101 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3832000000000002e-05 2024-02-09T17:35:34.101 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:34.101 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:35:34.101 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:34.101 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.101 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.101 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24817.0:17974 3.8 3:13dfb358:::benchmark_data_smithi179_90440_object17973:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e208)", 2024-02-09T17:35:34.101 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:30:10.967264+0000", 2024-02-09T17:35:34.102 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 323.07876979700001, 2024-02-09T17:35:34.102 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.193095355, 2024-02-09T17:35:34.102 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:35:34.102 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:35:34.102 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:35:34.102 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24817", 2024-02-09T17:35:34.102 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:35:34.102 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 17974 2024-02-09T17:35:34.102 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.102 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:35:34.102 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.103 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:35:34.103 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.967264+0000", 2024-02-09T17:35:34.103 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:34.103 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.103 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.103 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:35:34.103 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.967264+0000", 2024-02-09T17:35:34.103 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:34.103 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.103 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.103 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:35:34.103 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.967265+0000", 2024-02-09T17:35:34.104 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2200000000000003e-07 2024-02-09T17:35:34.104 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.104 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.104 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:35:34.104 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.967286+0000", 2024-02-09T17:35:34.104 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0499000000000001e-05 2024-02-09T17:35:34.104 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.104 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.104 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:35:34.104 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.967295+0000", 2024-02-09T17:35:34.104 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.3719999999999994e-06 2024-02-09T17:35:34.104 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.105 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.105 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:35:34.105 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.967302+0000", 2024-02-09T17:35:34.105 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2590000000000003e-06 2024-02-09T17:35:34.105 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.105 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.105 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:35:34.105 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.967341+0000", 2024-02-09T17:35:34.105 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9288000000000001e-05 2024-02-09T17:35:34.105 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.105 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.106 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:35:34.106 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.967433+0000", 2024-02-09T17:35:34.106 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.1874999999999997e-05 2024-02-09T17:35:34.106 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.106 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.106 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:35:34.106 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.967546+0000", 2024-02-09T17:35:34.106 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011241199999999999 2024-02-09T17:35:34.106 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.106 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.106 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:35:34.106 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.968153+0000", 2024-02-09T17:35:34.107 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00060681999999999999 2024-02-09T17:35:34.107 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.107 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.107 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:35:34.107 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:10.969239+0000", 2024-02-09T17:35:34.107 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0010867419999999999 2024-02-09T17:35:34.107 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.107 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.107 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:35:34.107 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:11.160277+0000", 2024-02-09T17:35:34.107 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.191037811 2024-02-09T17:35:34.108 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.108 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.108 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:35:34.108 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:11.160324+0000", 2024-02-09T17:35:34.108 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7222000000000003e-05 2024-02-09T17:35:34.108 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.108 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.108 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:35:34.108 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:11.160360+0000", 2024-02-09T17:35:34.108 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.5233e-05 2024-02-09T17:35:34.109 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:34.109 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:35:34.109 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:34.109 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.109 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.109 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24817.0:141310 3.e 3:72167b5f:::benchmark_data_smithi179_90440_object9351:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-02-09T17:35:34.109 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:31:59.277289+0000", 2024-02-09T17:35:34.109 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 214.768744965, 2024-02-09T17:35:34.109 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2998382479999999, 2024-02-09T17:35:34.109 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:35:34.109 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:35:34.109 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:35:34.110 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24817", 2024-02-09T17:35:34.110 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:35:34.110 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 141310 2024-02-09T17:35:34.110 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.110 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:35:34.110 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.110 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:35:34.110 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:31:59.277289+0000", 2024-02-09T17:35:34.110 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:34.110 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.110 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.111 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:35:34.111 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:31:59.277289+0000", 2024-02-09T17:35:34.111 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:34.111 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.111 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.111 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:35:34.111 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:31:59.277291+0000", 2024-02-09T17:35:34.111 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8509999999999999e-06 2024-02-09T17:35:34.111 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.111 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.111 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:35:34.111 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:31:59.277296+0000", 2024-02-09T17:35:34.112 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.502e-06 2024-02-09T17:35:34.112 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.112 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.112 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:35:34.112 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:31:59.277310+0000", 2024-02-09T17:35:34.112 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.346e-05 2024-02-09T17:35:34.112 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.112 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.112 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:35:34.112 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:31:59.277318+0000", 2024-02-09T17:35:34.112 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5599999999999994e-06 2024-02-09T17:35:34.113 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.113 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.113 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:35:34.113 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:31:59.277374+0000", 2024-02-09T17:35:34.113 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5682e-05 2024-02-09T17:35:34.113 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.113 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.113 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:35:34.113 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:31:59.277387+0000", 2024-02-09T17:35:34.113 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3253e-05 2024-02-09T17:35:34.113 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.113 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.114 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:35:34.114 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:00.271193+0000", 2024-02-09T17:35:34.114 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.99380549500000004 2024-02-09T17:35:34.114 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.114 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.114 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:35:34.114 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:00.271215+0000", 2024-02-09T17:35:34.114 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2432000000000001e-05 2024-02-09T17:35:34.114 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.114 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.114 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:35:34.114 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:01.559145+0000", 2024-02-09T17:35:34.115 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2879295260000001 2024-02-09T17:35:34.115 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.115 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.115 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:35:34.115 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:01.559278+0000", 2024-02-09T17:35:34.115 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000132972 2024-02-09T17:35:34.115 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.115 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.115 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 4,9", 2024-02-09T17:35:34.115 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:01.559429+0000", 2024-02-09T17:35:34.115 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015140600000000001 2024-02-09T17:35:34.115 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.116 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.116 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:35:34.116 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:01.571889+0000", 2024-02-09T17:35:34.116 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.012459359 2024-02-09T17:35:34.116 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.116 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.116 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:35:34.116 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:01.571931+0000", 2024-02-09T17:35:34.116 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2147000000000002e-05 2024-02-09T17:35:34.116 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.117 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.117 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:35:34.117 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:01.571944+0000", 2024-02-09T17:35:34.117 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3195e-05 2024-02-09T17:35:34.117 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.117 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.117 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:35:34.117 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:01.571987+0000", 2024-02-09T17:35:34.117 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.303e-05 2024-02-09T17:35:34.117 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.117 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.117 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:35:34.118 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:01.577127+0000", 2024-02-09T17:35:34.118 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0051403780000000001 2024-02-09T17:35:34.118 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:34.118 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:35:34.118 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:34.118 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.118 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.118 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24817.0:141324 3.e 3:7394bf3e:::benchmark_data_smithi179_90440_object9365:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-02-09T17:35:34.118 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:31:59.298805+0000", 2024-02-09T17:35:34.118 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 214.747228826, 2024-02-09T17:35:34.118 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2933926279999999, 2024-02-09T17:35:34.119 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:35:34.119 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:35:34.119 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:35:34.119 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24817", 2024-02-09T17:35:34.119 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:35:34.119 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 141324 2024-02-09T17:35:34.119 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.119 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:35:34.119 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.119 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:35:34.119 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:31:59.298805+0000", 2024-02-09T17:35:34.120 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:34.120 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.120 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.120 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:35:34.120 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:31:59.298805+0000", 2024-02-09T17:35:34.120 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:34.120 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.120 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.120 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:35:34.121 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:31:59.298806+0000", 2024-02-09T17:35:34.121 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.47e-07 2024-02-09T17:35:34.121 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.121 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.121 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:35:34.121 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:31:59.298810+0000", 2024-02-09T17:35:34.121 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.783e-06 2024-02-09T17:35:34.121 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.121 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.121 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:35:34.121 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:31:59.298820+0000", 2024-02-09T17:35:34.122 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7189999999999996e-06 2024-02-09T17:35:34.122 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.122 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.122 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:35:34.122 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:31:59.298825+0000", 2024-02-09T17:35:34.122 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6679999999999998e-06 2024-02-09T17:35:34.122 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.122 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.122 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:35:34.122 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:31:59.298875+0000", 2024-02-09T17:35:34.122 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.952e-05 2024-02-09T17:35:34.123 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.123 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.123 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:35:34.123 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:31:59.298882+0000", 2024-02-09T17:35:34.123 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.3799999999999996e-06 2024-02-09T17:35:34.123 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.123 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.123 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:35:34.123 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:00.271680+0000", 2024-02-09T17:35:34.123 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.97279783099999995 2024-02-09T17:35:34.123 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.123 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.124 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:35:34.124 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:00.271695+0000", 2024-02-09T17:35:34.124 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4562999999999999e-05 2024-02-09T17:35:34.124 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.124 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.124 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:35:34.124 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:01.567826+0000", 2024-02-09T17:35:34.124 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2961313429999999 2024-02-09T17:35:34.124 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.124 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.124 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:35:34.125 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:01.567943+0000", 2024-02-09T17:35:34.125 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011716400000000001 2024-02-09T17:35:34.125 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.125 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.125 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 4,9", 2024-02-09T17:35:34.125 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:01.568052+0000", 2024-02-09T17:35:34.125 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010929699999999999 2024-02-09T17:35:34.125 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.125 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.125 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:35:34.125 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:01.577328+0000", 2024-02-09T17:35:34.125 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0092758000000000007 2024-02-09T17:35:34.126 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.126 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.126 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:35:34.126 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:01.577408+0000", 2024-02-09T17:35:34.126 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.983e-05 2024-02-09T17:35:34.126 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.126 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.126 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:35:34.126 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:01.586920+0000", 2024-02-09T17:35:34.126 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0095125339999999996 2024-02-09T17:35:34.126 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.126 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.127 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:35:34.127 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:01.586972+0000", 2024-02-09T17:35:34.127 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1502000000000003e-05 2024-02-09T17:35:34.127 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.127 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.127 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:35:34.127 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:01.592198+0000", 2024-02-09T17:35:34.127 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0052258469999999996 2024-02-09T17:35:34.127 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:34.127 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:35:34.127 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:34.127 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.127 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.128 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24817.0:142079 3.e 3:7505dc96:::benchmark_data_smithi179_90440_object10120:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e239)", 2024-02-09T17:35:34.128 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:32:02.698578+0000", 2024-02-09T17:35:34.128 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 211.34745624000001, 2024-02-09T17:35:34.128 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3525411920000003, 2024-02-09T17:35:34.128 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:35:34.128 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-02-09T17:35:34.128 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:35:34.128 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24817", 2024-02-09T17:35:34.128 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:35:34.128 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 142079 2024-02-09T17:35:34.128 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.129 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:35:34.129 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.129 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:35:34.129 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:02.698578+0000", 2024-02-09T17:35:34.129 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:34.129 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.129 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.129 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:35:34.129 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:02.698578+0000", 2024-02-09T17:35:34.129 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:34.129 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.129 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.130 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:35:34.130 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:02.698578+0000", 2024-02-09T17:35:34.130 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1899999999999998e-07 2024-02-09T17:35:34.130 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.130 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.130 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:35:34.130 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:02.698580+0000", 2024-02-09T17:35:34.130 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6199999999999999e-06 2024-02-09T17:35:34.130 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.130 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.130 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:35:34.131 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:02.698585+0000", 2024-02-09T17:35:34.131 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8770000000000001e-06 2024-02-09T17:35:34.131 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.131 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.131 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:35:34.131 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:02.698589+0000", 2024-02-09T17:35:34.131 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4700000000000004e-06 2024-02-09T17:35:34.131 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.131 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.131 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:35:34.132 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:02.703298+0000", 2024-02-09T17:35:34.132 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0047084550000000003 2024-02-09T17:35:34.132 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.132 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.132 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for missing object", 2024-02-09T17:35:34.132 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:02.703371+0000", 2024-02-09T17:35:34.132 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.3894999999999996e-05 2024-02-09T17:35:34.132 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.132 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.132 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:35:34.132 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:07.051111+0000", 2024-02-09T17:35:34.133 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3477393749999997 2024-02-09T17:35:34.133 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.133 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.133 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:35:34.133 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:07.051119+0000", 2024-02-09T17:35:34.133 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1810000000000001e-06 2024-02-09T17:35:34.133 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:34.133 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:35:34.133 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:34.133 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.133 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.133 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24817.0:142087 3.e 3:71a5efe2:::benchmark_data_smithi179_90440_object10128:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e239)", 2024-02-09T17:35:34.134 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:32:02.698594+0000", 2024-02-09T17:35:34.134 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 211.34744024, 2024-02-09T17:35:34.134 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3619114699999999, 2024-02-09T17:35:34.134 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:35:34.134 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-02-09T17:35:34.134 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:35:34.134 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24817", 2024-02-09T17:35:34.134 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:35:34.134 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 142087 2024-02-09T17:35:34.134 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.134 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:35:34.135 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.135 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:35:34.135 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:02.698594+0000", 2024-02-09T17:35:34.135 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:34.135 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.135 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.135 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:35:34.135 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:02.698594+0000", 2024-02-09T17:35:34.135 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:34.135 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.135 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.136 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:35:34.136 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:02.698594+0000", 2024-02-09T17:35:34.136 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4499999999999998e-07 2024-02-09T17:35:34.136 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.136 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.136 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:35:34.136 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:02.698596+0000", 2024-02-09T17:35:34.136 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.824e-06 2024-02-09T17:35:34.136 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.136 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.136 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:35:34.137 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:02.698601+0000", 2024-02-09T17:35:34.137 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0030000000000002e-06 2024-02-09T17:35:34.137 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.137 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.137 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:35:34.137 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:02.698605+0000", 2024-02-09T17:35:34.137 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4329999999999997e-06 2024-02-09T17:35:34.137 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.137 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.137 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:35:34.137 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:02.703399+0000", 2024-02-09T17:35:34.138 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0047936970000000004 2024-02-09T17:35:34.138 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.138 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.138 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for missing object", 2024-02-09T17:35:34.138 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:02.703468+0000", 2024-02-09T17:35:34.138 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9261e-05 2024-02-09T17:35:34.138 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.138 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.138 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:35:34.138 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:07.060495+0000", 2024-02-09T17:35:34.138 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3570272430000001 2024-02-09T17:35:34.138 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.138 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.139 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:35:34.139 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:07.060505+0000", 2024-02-09T17:35:34.139 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.764e-06 2024-02-09T17:35:34.139 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:34.139 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:35:34.139 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:34.139 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.139 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.139 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24817.0:145186 3.e 3:790ea800:::benchmark_data_smithi179_90440_object13227:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-02-09T17:35:34.139 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:32:07.939458+0000", 2024-02-09T17:35:34.139 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 206.10657578600001, 2024-02-09T17:35:34.139 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.26007465000000002, 2024-02-09T17:35:34.139 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:35:34.140 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:35:34.140 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:35:34.140 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24817", 2024-02-09T17:35:34.140 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:35:34.140 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 145186 2024-02-09T17:35:34.140 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.140 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:35:34.140 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.140 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:35:34.140 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:07.939458+0000", 2024-02-09T17:35:34.141 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:34.141 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.141 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.141 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:35:34.141 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:07.939458+0000", 2024-02-09T17:35:34.141 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:34.141 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.141 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.141 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:35:34.141 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:07.939460+0000", 2024-02-09T17:35:34.142 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.296e-06 2024-02-09T17:35:34.142 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.142 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.142 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:35:34.142 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:07.939464+0000", 2024-02-09T17:35:34.142 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9189999999999998e-06 2024-02-09T17:35:34.142 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.142 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.142 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:35:34.142 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:07.939476+0000", 2024-02-09T17:35:34.143 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1202e-05 2024-02-09T17:35:34.143 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.143 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.143 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:35:34.143 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:07.939483+0000", 2024-02-09T17:35:34.143 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.3529999999999997e-06 2024-02-09T17:35:34.143 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.143 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.143 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:35:34.143 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:07.963500+0000", 2024-02-09T17:35:34.143 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.024016493999999999 2024-02-09T17:35:34.143 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.144 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.144 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for rw locks", 2024-02-09T17:35:34.144 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:07.963626+0000", 2024-02-09T17:35:34.144 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000126809 2024-02-09T17:35:34.144 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.144 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.144 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:35:34.144 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:07.978591+0000", 2024-02-09T17:35:34.144 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014964244 2024-02-09T17:35:34.144 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.144 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.145 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:35:34.145 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:07.978722+0000", 2024-02-09T17:35:34.145 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013103100000000001 2024-02-09T17:35:34.145 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.145 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.145 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 4,9", 2024-02-09T17:35:34.145 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:07.978875+0000", 2024-02-09T17:35:34.145 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015352700000000001 2024-02-09T17:35:34.145 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.145 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.145 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:35:34.145 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:08.003034+0000", 2024-02-09T17:35:34.145 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.024158777999999999 2024-02-09T17:35:34.146 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.146 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.146 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:35:34.146 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:08.003218+0000", 2024-02-09T17:35:34.146 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00018436599999999999 2024-02-09T17:35:34.146 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.146 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.146 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:35:34.146 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:08.194478+0000", 2024-02-09T17:35:34.146 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.19125986 2024-02-09T17:35:34.146 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.146 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.146 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:35:34.147 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:08.194521+0000", 2024-02-09T17:35:34.147 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3356999999999998e-05 2024-02-09T17:35:34.147 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.147 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.147 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:35:34.147 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:08.199533+0000", 2024-02-09T17:35:34.147 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0050114139999999996 2024-02-09T17:35:34.147 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:34.147 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:35:34.147 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:34.147 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.147 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.147 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24817.0:145217 3.e 3:744ba7dc:::benchmark_data_smithi179_90440_object13258:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-02-09T17:35:34.148 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:32:07.995634+0000", 2024-02-09T17:35:34.148 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 206.050399602, 2024-02-09T17:35:34.148 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.22444186099999999, 2024-02-09T17:35:34.148 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:35:34.148 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:35:34.148 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:35:34.148 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24817", 2024-02-09T17:35:34.148 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:35:34.148 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 145217 2024-02-09T17:35:34.148 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.148 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:35:34.149 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.149 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:35:34.149 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:07.995634+0000", 2024-02-09T17:35:34.149 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:34.149 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.149 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.149 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:35:34.149 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:07.995634+0000", 2024-02-09T17:35:34.149 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:34.149 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.149 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.150 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:35:34.150 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:07.995636+0000", 2024-02-09T17:35:34.150 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6619999999999999e-06 2024-02-09T17:35:34.150 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.150 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.150 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:35:34.150 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:07.995643+0000", 2024-02-09T17:35:34.150 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7309999999999996e-06 2024-02-09T17:35:34.150 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.150 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.150 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:35:34.150 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:07.995656+0000", 2024-02-09T17:35:34.151 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.278e-05 2024-02-09T17:35:34.151 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.151 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.151 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:35:34.151 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:07.995664+0000", 2024-02-09T17:35:34.151 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3790000000000006e-06 2024-02-09T17:35:34.151 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.151 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.151 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:35:34.151 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:08.003070+0000", 2024-02-09T17:35:34.151 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0074062959999999997 2024-02-09T17:35:34.152 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.152 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.152 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for rw locks", 2024-02-09T17:35:34.152 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:08.003205+0000", 2024-02-09T17:35:34.152 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013469099999999999 2024-02-09T17:35:34.152 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.152 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.152 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:35:34.152 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:08.023610+0000", 2024-02-09T17:35:34.152 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.020404839000000001 2024-02-09T17:35:34.152 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.153 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.153 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:35:34.153 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:08.023721+0000", 2024-02-09T17:35:34.153 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000111537 2024-02-09T17:35:34.153 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.153 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.153 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 4,9", 2024-02-09T17:35:34.153 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:08.023922+0000", 2024-02-09T17:35:34.153 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00020108499999999999 2024-02-09T17:35:34.153 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.153 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.153 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:35:34.154 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:08.044498+0000", 2024-02-09T17:35:34.154 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.020576065000000001 2024-02-09T17:35:34.154 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.154 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.154 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:35:34.154 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:08.044622+0000", 2024-02-09T17:35:34.154 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000123366 2024-02-09T17:35:34.154 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.154 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.154 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:35:34.154 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:08.215009+0000", 2024-02-09T17:35:34.154 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.17038697799999999 2024-02-09T17:35:34.154 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.155 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.155 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:35:34.155 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:08.215050+0000", 2024-02-09T17:35:34.155 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1371000000000001e-05 2024-02-09T17:35:34.155 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.155 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.155 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:35:34.155 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:08.220076+0000", 2024-02-09T17:35:34.155 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.005026081 2024-02-09T17:35:34.155 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:34.155 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:35:34.155 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:34.156 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.156 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.156 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24817.0:145227 3.e 3:7bff3351:::benchmark_data_smithi179_90440_object13268:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-02-09T17:35:34.156 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:32:08.008335+0000", 2024-02-09T17:35:34.156 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 206.03769918, 2024-02-09T17:35:34.156 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.20661220699999999, 2024-02-09T17:35:34.156 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:35:34.156 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:35:34.156 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:35:34.156 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24817", 2024-02-09T17:35:34.156 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:35:34.156 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 145227 2024-02-09T17:35:34.157 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.157 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:35:34.157 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.157 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:35:34.157 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:08.008335+0000", 2024-02-09T17:35:34.157 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:34.157 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.157 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.157 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:35:34.157 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:08.008335+0000", 2024-02-09T17:35:34.157 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:34.158 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.158 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.158 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:35:34.158 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:08.008337+0000", 2024-02-09T17:35:34.158 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6619999999999999e-06 2024-02-09T17:35:34.158 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.158 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.158 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:35:34.158 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:08.008342+0000", 2024-02-09T17:35:34.158 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3369999999999999e-06 2024-02-09T17:35:34.158 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.158 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.159 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:35:34.159 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:08.008355+0000", 2024-02-09T17:35:34.159 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3283e-05 2024-02-09T17:35:34.159 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.159 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.159 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:35:34.159 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:08.008365+0000", 2024-02-09T17:35:34.159 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0315e-05 2024-02-09T17:35:34.159 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.159 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.159 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:35:34.159 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:08.008420+0000", 2024-02-09T17:35:34.160 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4178999999999997e-05 2024-02-09T17:35:34.160 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.160 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.160 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:35:34.160 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:08.008594+0000", 2024-02-09T17:35:34.160 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00017437099999999999 2024-02-09T17:35:34.160 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.160 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.160 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 4,9", 2024-02-09T17:35:34.160 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:08.008750+0000", 2024-02-09T17:35:34.161 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000156322 2024-02-09T17:35:34.161 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.161 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.161 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:35:34.161 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:08.029094+0000", 2024-02-09T17:35:34.161 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.020344101999999999 2024-02-09T17:35:34.161 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.161 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.161 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:35:34.161 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:08.029161+0000", 2024-02-09T17:35:34.161 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6681999999999996e-05 2024-02-09T17:35:34.161 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.162 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.162 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:35:34.162 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:08.210032+0000", 2024-02-09T17:35:34.162 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.18087120400000001 2024-02-09T17:35:34.162 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.162 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.162 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:35:34.162 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:08.210092+0000", 2024-02-09T17:35:34.162 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9197000000000003e-05 2024-02-09T17:35:34.162 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.162 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.163 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:35:34.163 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:08.214947+0000", 2024-02-09T17:35:34.163 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0048555530000000003 2024-02-09T17:35:34.163 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:34.163 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:35:34.163 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:34.163 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.163 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.163 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24817.0:157817 3.8 3:186a603d:::benchmark_data_smithi179_90440_object25858:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-02-09T17:35:34.163 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:32:28.538602+0000", 2024-02-09T17:35:34.163 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 185.50743220999999, 2024-02-09T17:35:34.163 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.22345542500000001, 2024-02-09T17:35:34.163 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:35:34.164 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:35:34.164 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:35:34.164 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24817", 2024-02-09T17:35:34.164 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:35:34.164 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 157817 2024-02-09T17:35:34.164 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.164 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:35:34.164 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.164 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:35:34.164 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.538602+0000", 2024-02-09T17:35:34.165 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:34.165 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.165 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.165 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:35:34.165 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.538602+0000", 2024-02-09T17:35:34.165 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:34.165 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.165 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.165 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:35:34.165 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.538604+0000", 2024-02-09T17:35:34.165 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3659999999999999e-06 2024-02-09T17:35:34.165 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.165 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.166 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:35:34.166 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.538611+0000", 2024-02-09T17:35:34.166 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5230000000000001e-06 2024-02-09T17:35:34.166 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.166 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.166 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:35:34.166 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.538628+0000", 2024-02-09T17:35:34.166 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7079999999999999e-05 2024-02-09T17:35:34.166 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.166 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.166 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:35:34.166 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.538646+0000", 2024-02-09T17:35:34.166 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8601e-05 2024-02-09T17:35:34.167 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.167 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.167 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:35:34.167 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.542049+0000", 2024-02-09T17:35:34.167 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0034029770000000002 2024-02-09T17:35:34.167 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.167 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.167 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:35:34.167 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.542172+0000", 2024-02-09T17:35:34.167 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012233 2024-02-09T17:35:34.167 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.168 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.168 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:35:34.168 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.542283+0000", 2024-02-09T17:35:34.168 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000111445 2024-02-09T17:35:34.168 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.168 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.168 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:35:34.168 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.546600+0000", 2024-02-09T17:35:34.168 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0043170429999999996 2024-02-09T17:35:34.168 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.168 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.169 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:35:34.169 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.554234+0000", 2024-02-09T17:35:34.169 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0076341400000000002 2024-02-09T17:35:34.169 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.169 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.169 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:35:34.169 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.761951+0000", 2024-02-09T17:35:34.169 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.207716228 2024-02-09T17:35:34.169 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.169 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.169 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:35:34.170 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.762011+0000", 2024-02-09T17:35:34.170 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0498000000000003e-05 2024-02-09T17:35:34.170 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.170 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.170 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:35:34.170 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.762057+0000", 2024-02-09T17:35:34.170 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6193999999999999e-05 2024-02-09T17:35:34.170 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:34.170 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:35:34.170 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:34.170 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.170 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.171 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24817.0:157821 3.8 3:182e34a8:::benchmark_data_smithi179_90440_object25862:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-02-09T17:35:34.171 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:32:28.548696+0000", 2024-02-09T17:35:34.171 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 185.49733821500001, 2024-02-09T17:35:34.171 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.21352769399999999, 2024-02-09T17:35:34.171 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:35:34.171 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:35:34.171 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:35:34.171 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24817", 2024-02-09T17:35:34.171 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:35:34.171 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 157821 2024-02-09T17:35:34.171 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.172 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:35:34.172 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.172 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:35:34.172 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.548696+0000", 2024-02-09T17:35:34.172 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:34.172 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.172 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.172 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:35:34.172 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.548696+0000", 2024-02-09T17:35:34.172 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:34.172 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.172 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.173 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:35:34.173 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.548697+0000", 2024-02-09T17:35:34.173 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2470000000000001e-06 2024-02-09T17:35:34.173 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.173 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.173 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:35:34.173 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.548701+0000", 2024-02-09T17:35:34.173 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6229999999999999e-06 2024-02-09T17:35:34.173 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.173 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.173 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:35:34.173 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.548709+0000", 2024-02-09T17:35:34.174 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.6419999999999996e-06 2024-02-09T17:35:34.174 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.174 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.174 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:35:34.174 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.548714+0000", 2024-02-09T17:35:34.174 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1429999999999997e-06 2024-02-09T17:35:34.174 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.174 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.174 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:35:34.174 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.548767+0000", 2024-02-09T17:35:34.174 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2713000000000002e-05 2024-02-09T17:35:34.175 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.175 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.175 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:35:34.175 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.548903+0000", 2024-02-09T17:35:34.175 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000135325 2024-02-09T17:35:34.175 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.175 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.175 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:35:34.175 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.549025+0000", 2024-02-09T17:35:34.175 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012276799999999999 2024-02-09T17:35:34.175 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.175 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.176 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:35:34.176 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.554055+0000", 2024-02-09T17:35:34.176 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0050293109999999999 2024-02-09T17:35:34.176 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.176 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.176 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:35:34.176 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.554278+0000", 2024-02-09T17:35:34.176 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00022313199999999999 2024-02-09T17:35:34.176 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.176 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.176 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:35:34.177 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.762110+0000", 2024-02-09T17:35:34.177 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.207831873 2024-02-09T17:35:34.177 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.177 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.177 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:35:34.177 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.762185+0000", 2024-02-09T17:35:34.177 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5199999999999998e-05 2024-02-09T17:35:34.177 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.177 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.177 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:35:34.177 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.762224+0000", 2024-02-09T17:35:34.178 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8717000000000002e-05 2024-02-09T17:35:34.178 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:34.178 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:35:34.178 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:34.178 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.178 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.178 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24817.0:157826 3.e 3:7459c87d:::benchmark_data_smithi179_90440_object25867:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-02-09T17:35:34.178 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:32:28.554030+0000", 2024-02-09T17:35:34.178 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 185.49200389399999, 2024-02-09T17:35:34.178 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.23630322500000001, 2024-02-09T17:35:34.178 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:35:34.178 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:35:34.178 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:35:34.178 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24817", 2024-02-09T17:35:34.179 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:35:34.179 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 157826 2024-02-09T17:35:34.179 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.179 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:35:34.179 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.179 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:35:34.179 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.554030+0000", 2024-02-09T17:35:34.179 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:34.179 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.179 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.179 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:35:34.179 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.554030+0000", 2024-02-09T17:35:34.180 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:34.180 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.180 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.180 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:35:34.180 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.554032+0000", 2024-02-09T17:35:34.180 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8640000000000001e-06 2024-02-09T17:35:34.180 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.180 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.180 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:35:34.180 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.554037+0000", 2024-02-09T17:35:34.180 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0880000000000002e-06 2024-02-09T17:35:34.180 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.181 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.181 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:35:34.181 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.554053+0000", 2024-02-09T17:35:34.181 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6283999999999999e-05 2024-02-09T17:35:34.181 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.181 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.181 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:35:34.181 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.554062+0000", 2024-02-09T17:35:34.181 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3760000000000008e-06 2024-02-09T17:35:34.181 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.181 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.181 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:35:34.182 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.557927+0000", 2024-02-09T17:35:34.182 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0038654900000000001 2024-02-09T17:35:34.182 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.182 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.182 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:35:34.182 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.558011+0000", 2024-02-09T17:35:34.182 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4136999999999998e-05 2024-02-09T17:35:34.182 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.182 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.182 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 4,9", 2024-02-09T17:35:34.182 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.558106+0000", 2024-02-09T17:35:34.183 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.4945000000000007e-05 2024-02-09T17:35:34.183 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.183 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.183 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:35:34.183 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.575148+0000", 2024-02-09T17:35:34.183 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017041639000000001 2024-02-09T17:35:34.183 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.183 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.183 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:35:34.183 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.587046+0000", 2024-02-09T17:35:34.184 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011898432 2024-02-09T17:35:34.184 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.184 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.184 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:35:34.184 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.785646+0000", 2024-02-09T17:35:34.184 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.19860003500000001 2024-02-09T17:35:34.184 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.184 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.184 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:35:34.184 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.785717+0000", 2024-02-09T17:35:34.184 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.0568999999999995e-05 2024-02-09T17:35:34.185 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.185 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.185 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:35:34.185 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.790333+0000", 2024-02-09T17:35:34.185 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0046163660000000002 2024-02-09T17:35:34.185 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:34.185 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:35:34.185 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:34.185 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.185 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.186 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24817.0:157831 3.8 3:1fc80b22:::benchmark_data_smithi179_90440_object25872:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-02-09T17:35:34.186 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:32:28.561103+0000", 2024-02-09T17:35:34.186 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 185.48493138399999, 2024-02-09T17:35:34.186 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.214470253, 2024-02-09T17:35:34.186 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:35:34.186 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:35:34.186 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:35:34.186 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24817", 2024-02-09T17:35:34.186 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:35:34.186 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 157831 2024-02-09T17:35:34.186 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.186 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:35:34.187 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.187 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:35:34.187 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.561103+0000", 2024-02-09T17:35:34.187 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:34.187 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.187 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.187 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:35:34.187 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.561103+0000", 2024-02-09T17:35:34.187 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:34.187 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.187 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.187 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:35:34.188 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.561104+0000", 2024-02-09T17:35:34.188 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.271e-06 2024-02-09T17:35:34.188 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.188 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.188 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:35:34.188 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.561110+0000", 2024-02-09T17:35:34.188 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.054e-06 2024-02-09T17:35:34.188 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.188 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.188 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:35:34.189 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.561123+0000", 2024-02-09T17:35:34.189 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2840000000000001e-05 2024-02-09T17:35:34.189 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.189 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.189 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:35:34.189 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.561129+0000", 2024-02-09T17:35:34.189 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3110000000000002e-06 2024-02-09T17:35:34.189 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.189 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.189 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:35:34.189 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.561196+0000", 2024-02-09T17:35:34.189 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6528999999999999e-05 2024-02-09T17:35:34.189 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.190 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.190 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:35:34.190 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.561340+0000", 2024-02-09T17:35:34.190 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014477500000000001 2024-02-09T17:35:34.190 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.190 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.190 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:35:34.190 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.561468+0000", 2024-02-09T17:35:34.190 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000127761 2024-02-09T17:35:34.190 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.190 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.190 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:35:34.191 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.565549+0000", 2024-02-09T17:35:34.191 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0040806879999999999 2024-02-09T17:35:34.191 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.191 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.191 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:35:34.191 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.565822+0000", 2024-02-09T17:35:34.191 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00027355599999999999 2024-02-09T17:35:34.191 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.191 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.191 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:35:34.191 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.775459+0000", 2024-02-09T17:35:34.191 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.20963662299999999 2024-02-09T17:35:34.191 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.192 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.192 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:35:34.192 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.775524+0000", 2024-02-09T17:35:34.192 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4985000000000002e-05 2024-02-09T17:35:34.192 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.192 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.192 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:35:34.192 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.775573+0000", 2024-02-09T17:35:34.192 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8860000000000003e-05 2024-02-09T17:35:34.192 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:34.192 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:35:34.193 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:34.193 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.193 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.193 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24817.0:157851 3.e 3:791b1c8b:::benchmark_data_smithi179_90440_object25892:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-02-09T17:35:34.193 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:32:28.599332+0000", 2024-02-09T17:35:34.193 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 185.44670187099999, 2024-02-09T17:35:34.193 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.19560749899999999, 2024-02-09T17:35:34.193 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:35:34.193 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:35:34.193 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:35:34.193 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24817", 2024-02-09T17:35:34.193 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:35:34.193 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 157851 2024-02-09T17:35:34.194 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.194 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:35:34.194 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.194 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:35:34.194 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.599332+0000", 2024-02-09T17:35:34.194 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:34.194 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.194 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.194 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:35:34.194 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.599332+0000", 2024-02-09T17:35:34.195 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:35:34.195 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.195 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.195 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:35:34.195 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.599334+0000", 2024-02-09T17:35:34.195 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2220000000000001e-06 2024-02-09T17:35:34.195 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.195 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.195 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:35:34.195 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.599342+0000", 2024-02-09T17:35:34.196 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6699999999999994e-06 2024-02-09T17:35:34.196 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.196 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.196 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:35:34.196 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.599356+0000", 2024-02-09T17:35:34.196 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4211e-05 2024-02-09T17:35:34.196 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.196 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.196 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:35:34.196 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.599365+0000", 2024-02-09T17:35:34.197 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.6649999999999992e-06 2024-02-09T17:35:34.197 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.197 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.197 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:35:34.197 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.611213+0000", 2024-02-09T17:35:34.197 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011847669999999999 2024-02-09T17:35:34.197 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.197 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.197 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:35:34.197 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.611293+0000", 2024-02-09T17:35:34.197 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.0479e-05 2024-02-09T17:35:34.198 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.198 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.198 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 4,9", 2024-02-09T17:35:34.198 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.611437+0000", 2024-02-09T17:35:34.198 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014405 2024-02-09T17:35:34.198 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.198 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.198 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:35:34.198 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.616367+0000", 2024-02-09T17:35:34.198 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0049300680000000001 2024-02-09T17:35:34.198 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.198 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.198 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:35:34.198 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.634185+0000", 2024-02-09T17:35:34.199 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017817603000000001 2024-02-09T17:35:34.199 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.199 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.199 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:35:34.199 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.790402+0000", 2024-02-09T17:35:34.199 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15621753499999999 2024-02-09T17:35:34.199 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.199 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.199 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:35:34.199 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.790468+0000", 2024-02-09T17:35:34.199 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5456000000000005e-05 2024-02-09T17:35:34.199 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:35:34.199 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:35:34.199 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:35:34.200 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:32:28.794940+0000", 2024-02-09T17:35:34.200 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0044718700000000002 2024-02-09T17:35:34.200 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:34.200 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:35:34.200 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:34.200 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:35:34.200 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:35:34.200 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:35:34.200 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:35:34.232 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:34.254 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:35:34.256 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:34.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:35:34.358 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:35:34.385 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:34.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:34 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:35:34.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:34 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:35:34.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:34 smithi179 ceph-mon[43975]: osdmap e327: 12 total, 12 up, 12 in 2024-02-09T17:35:34.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:34 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:35:34.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:34 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:35:34.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:34 smithi099 ceph-mon[39420]: osdmap e327: 12 total, 12 up, 12 in 2024-02-09T17:35:34.507 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:35:34.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:34.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:35:34.612 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:35:34.638 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:34.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:34 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:35:34.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:34 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:35:34.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:34 smithi162 ceph-mon[44304]: osdmap e327: 12 total, 12 up, 12 in 2024-02-09T17:35:34.738 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.8 dump_ops_in_flight 2024-02-09T17:35:34.928 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:35:34.997 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:35:35.186 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:35:35.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:35.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:35:35.291 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:35:35.320 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:35.338 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:35:35.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:35.341 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:35:35.442 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:35:35.444 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:35 smithi099 ceph-mon[39420]: pgmap v892: 37 pgs: 3 creating+activating, 34 active+clean; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:35:35.474 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:35.495 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:35:35.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:35.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:35:35.524 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:35 smithi179 ceph-mon[43975]: pgmap v892: 37 pgs: 3 creating+activating, 34 active+clean; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:35:35.599 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:35:35.628 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:35.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:35 smithi162 ceph-mon[44304]: pgmap v892: 37 pgs: 3 creating+activating, 34 active+clean; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:35:36.121 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:35:36.254 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:35:36.377 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:35:36.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:36.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:35:36.481 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:35:36.513 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:36.534 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:35:36.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:36.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:35:36.638 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:35:36.666 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:36.829 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:35:36.829 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-02-09T17:35:36.829 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 0 2024-02-09T17:35:36.829 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:35:37.305 INFO:teuthology.orchestra.run.smithi099.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-02-09T17:35:37.414 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:35:37.416 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:37 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2814951581' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-09T17:35:37.416 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:37 smithi099 ceph-mon[39420]: pgmap v893: 37 pgs: 37 active+clean; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:35:37.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:37.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:35:37.435 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.8 dump_blocked_ops 2024-02-09T17:35:37.518 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:35:37.548 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:37 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2814951581' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-09T17:35:37.548 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:37 smithi162 ceph-mon[44304]: pgmap v893: 37 pgs: 37 active+clean; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:35:37.551 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:37.571 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:35:37.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:37.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:35:37.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:37 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2814951581' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-09T17:35:37.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:37 smithi179 ceph-mon[43975]: pgmap v893: 37 pgs: 37 active+clean; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:35:37.675 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:35:37.695 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:35:37.705 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:37.725 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:35:37.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:37.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:35:37.829 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:35:37.859 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:37.878 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:35:37.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:37.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:35:37.982 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:35:38.010 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:38.029 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:35:38.031 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:38.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:35:38.133 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:35:38.160 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:38.187 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:35:38.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:38.190 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:35:38.291 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:35:38.318 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:38.566 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:35:38.567 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:38 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2814951581' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-09T17:35:38.567 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:38 smithi179 ceph-mon[43975]: osdmap e328: 12 total, 12 up, 12 in 2024-02-09T17:35:38.568 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:38.568 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:35:38.631 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:38 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2814951581' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-09T17:35:38.631 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:38 smithi099 ceph-mon[39420]: osdmap e328: 12 total, 12 up, 12 in 2024-02-09T17:35:38.669 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:35:38.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:38 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2814951581' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-09T17:35:38.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:38 smithi162 ceph-mon[44304]: osdmap e328: 12 total, 12 up, 12 in 2024-02-09T17:35:38.698 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:38.735 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:35:38.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:38.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:35:38.840 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:35:38.867 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:38.990 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:35:38.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:38.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:35:39.018 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:35:39.097 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:35:39.129 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:39.150 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:35:39.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:39.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:35:39.254 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:35:39.283 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:39.307 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:35:39.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:39.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:35:39.315 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is set 2024-02-09T17:35:39.358 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:39 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2172618843' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:35:39.384 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:39 smithi179 ceph-mon[43975]: pgmap v895: 37 pgs: 37 active+clean; 577 KiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:35:39.413 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:35:39.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:39 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2172618843' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:35:39.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:39 smithi162 ceph-mon[44304]: pgmap v895: 37 pgs: 37 active+clean; 577 KiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:35:39.440 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:39.563 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:39 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2172618843' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:35:39.563 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:39 smithi099 ceph-mon[39420]: pgmap v895: 37 pgs: 37 active+clean; 577 KiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:35:39.565 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:35:39.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:39.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:35:39.650 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:35:39.650 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-02-09T17:35:39.650 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-02-09T17:35:39.650 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-02-09T17:35:39.650 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:35:39.655 DEBUG:teuthology.orchestra.run.smithi179:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-02-09T17:35:39.658 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-02-09T17:35:39.658 INFO:teuthology.orchestra.run:waiting for 3000 2024-02-09T17:35:39.672 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:35:39.704 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:39.708 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:hints = 1 2024-02-09T17:35:39.708 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-02-09T17:35:39.708 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Object prefix: benchmark_data_smithi179_103809 2024-02-09T17:35:39.708 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-09T17:35:39.708 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 0 0 0 0 0 0 - 0 2024-02-09T17:35:39.737 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:35:39.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:39.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:35:39.842 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:35:39.887 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:39.917 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:35:39.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:39.920 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:35:40.021 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:35:40.062 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:40.426 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:35:40.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:40.432 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:35:40.462 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.8 dump_historic_ops 2024-02-09T17:35:40.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:40 smithi099 ceph-mon[39420]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-09T17:35:40.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:40 smithi099 ceph-mon[39420]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:35:40.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:40 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2172618843' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:35:40.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:40 smithi099 ceph-mon[39420]: osdmap e329: 12 total, 12 up, 12 in 2024-02-09T17:35:40.533 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:35:40.571 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:40 smithi162 ceph-mon[44304]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-09T17:35:40.571 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:40 smithi162 ceph-mon[44304]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:35:40.571 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:40 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2172618843' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:35:40.571 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:40 smithi162 ceph-mon[44304]: osdmap e329: 12 total, 12 up, 12 in 2024-02-09T17:35:40.574 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:40.597 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:35:40.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:40.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:35:40.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:40 smithi179 ceph-mon[43975]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-09T17:35:40.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:40 smithi179 ceph-mon[43975]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:35:40.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:40 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2172618843' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:35:40.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:40 smithi179 ceph-mon[43975]: osdmap e329: 12 total, 12 up, 12 in 2024-02-09T17:35:40.701 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:35:40.708 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 1 16 4538 4522 282.609 282.625 0.00610711 0.00353199 2024-02-09T17:35:40.739 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:40.765 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:35:40.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:40.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:35:40.869 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:35:40.916 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:40.927 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:35:40.959 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:35:40.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:40.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:35:41.063 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:35:41.101 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:41.129 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:35:41.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:41.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:35:41.234 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:35:41.267 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:41.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:41 smithi162 ceph-mon[44304]: pgmap v897: 37 pgs: 37 active+clean; 99 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 216 op/s 2024-02-09T17:35:41.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:41 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1923454615' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-09T17:35:41.595 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.7 to 1 (10000) 2024-02-09T17:35:41.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:41 smithi179 ceph-mon[43975]: pgmap v897: 37 pgs: 37 active+clean; 99 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 216 op/s 2024-02-09T17:35:41.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:41 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1923454615' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-09T17:35:41.708 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 2 16 8197 8181 255.628 228.688 0.00147835 0.00388508 2024-02-09T17:35:41.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:41 smithi099 ceph-mon[39420]: pgmap v897: 37 pgs: 37 active+clean; 99 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 216 op/s 2024-02-09T17:35:41.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:41 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1923454615' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-09T17:35:42.467 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 8 1 2024-02-09T17:35:42.659 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:42 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:35:42.659 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:42 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:35:42.659 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:42 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1923454615' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-09T17:35:42.659 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:42 smithi099 ceph-mon[39420]: osdmap e330: 12 total, 12 up, 12 in 2024-02-09T17:35:42.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:42 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:35:42.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:42 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:35:42.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:42 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1923454615' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-09T17:35:42.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:42 smithi179 ceph-mon[43975]: osdmap e330: 12 total, 12 up, 12 in 2024-02-09T17:35:42.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:42 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:35:42.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:42 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:35:42.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:42 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1923454615' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-09T17:35:42.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:42 smithi162 ceph-mon[44304]: osdmap e330: 12 total, 12 up, 12 in 2024-02-09T17:35:42.708 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 3 16 11875 11859 247.038 229.875 0.0130945 0.00403696 2024-02-09T17:35:42.969 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset noscrub 2024-02-09T17:35:43.017 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:35:43.058 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:35:43.063 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:35:43.098 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:35:43.132 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:35:43.169 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:35:43.222 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:35:43.267 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:35:43.306 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:35:43.336 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:43 smithi099 ceph-mon[39420]: pgmap v899: 37 pgs: 37 active+clean; 223 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 592 op/s 2024-02-09T17:35:43.336 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:43 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2198903389' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:35:43.353 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:35:43.383 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:35:43.417 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:35:43.460 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:35:43.476 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:35:43.477 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:35:43.477 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:35:43.477 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:35:43.477 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:35:43.477 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:35:43.477 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:35:43.477 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:35:43.477 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:35:43.477 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:35:43.477 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:35:43.477 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:35:43.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:43 smithi179 ceph-mon[43975]: pgmap v899: 37 pgs: 37 active+clean; 223 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 592 op/s 2024-02-09T17:35:43.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:43 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2198903389' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:35:43.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:43 smithi162 ceph-mon[44304]: pgmap v899: 37 pgs: 37 active+clean; 223 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 592 op/s 2024-02-09T17:35:43.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:43 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2198903389' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:35:43.708 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 4 16 14895 14879 232.462 188.75 0.00430401 0.004296 2024-02-09T17:35:43.819 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:35:43.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:43.823 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:35:43.923 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:35:43.970 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:43.996 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:35:44.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:44.000 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:35:44.101 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:35:44.140 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:44.167 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:35:44.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:44.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:35:44.272 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:35:44.313 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:44.340 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:35:44.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:44.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:35:44.444 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:35:44.520 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:44.709 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 5 16 18167 18151 226.867 204.5 0.00476103 0.00440361 2024-02-09T17:35:45.542 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T17:35:45.543 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:35:45.709 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 6 16 20290 20274 211.168 132.688 0.00315733 0.00473073 2024-02-09T17:35:45.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:45 smithi179 ceph-mon[43975]: pgmap v900: 37 pgs: 37 active+clean; 279 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 38 MiB/s wr, 610 op/s 2024-02-09T17:35:45.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:45 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:35:45.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:45 smithi162 ceph-mon[44304]: pgmap v900: 37 pgs: 37 active+clean; 279 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 38 MiB/s wr, 610 op/s 2024-02-09T17:35:45.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:45 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:35:45.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:45 smithi099 ceph-mon[39420]: pgmap v900: 37 pgs: 37 active+clean; 279 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 38 MiB/s wr, 610 op/s 2024-02-09T17:35:45.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:45 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:35:46.323 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-09T17:35:46.488 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:35:46.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:46.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:35:46.593 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:35:46.670 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:46.699 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:35:46.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:46.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:35:46.708 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 7 15 22974 22959 204.973 167.812 0.00438806 0.00487439 2024-02-09T17:35:46.803 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:35:46.834 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:46.853 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:35:46.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:46.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:35:46.956 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:35:47.017 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:47.310 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:47 smithi099 ceph-mon[39420]: osdmap e331: 12 total, 12 up, 12 in 2024-02-09T17:35:47.310 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:47 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:35:47.311 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:47 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:35:47.311 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:47 smithi099 ceph-mon[39420]: pgmap v902: 37 pgs: 37 active+clean; 1.2 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 163 MiB/s wr, 2.60k op/s 2024-02-09T17:35:47.311 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:47 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:35:47.311 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:47 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:35:47.311 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:47 smithi099 ceph-mon[39420]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:35:47.571 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:35:47.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:47.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:35:47.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:47 smithi179 ceph-mon[43975]: osdmap e331: 12 total, 12 up, 12 in 2024-02-09T17:35:47.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:47 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:35:47.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:47 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:35:47.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:47 smithi179 ceph-mon[43975]: pgmap v902: 37 pgs: 37 active+clean; 1.2 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 163 MiB/s wr, 2.60k op/s 2024-02-09T17:35:47.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:47 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:35:47.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:47 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:35:47.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:47 smithi179 ceph-mon[43975]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:35:47.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:47 smithi162 ceph-mon[44304]: osdmap e331: 12 total, 12 up, 12 in 2024-02-09T17:35:47.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:47 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:35:47.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:47 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:35:47.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:47 smithi162 ceph-mon[44304]: pgmap v902: 37 pgs: 37 active+clean; 1.2 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 163 MiB/s wr, 2.60k op/s 2024-02-09T17:35:47.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:47 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:35:47.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:47 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:35:47.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:47 smithi162 ceph-mon[44304]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:35:47.703 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:35:47.708 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 8 16 25753 25737 201.053 173.625 0.00735711 0.00496837 2024-02-09T17:35:47.745 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:47.774 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:35:47.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:47.779 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:35:47.880 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:35:47.922 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:47.955 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:35:47.957 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:47.958 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:35:48.059 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:35:48.100 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:48.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:48 smithi099 ceph-mon[39420]: osdmap e332: 12 total, 12 up, 12 in 2024-02-09T17:35:48.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:48 smithi179 ceph-mon[43975]: osdmap e332: 12 total, 12 up, 12 in 2024-02-09T17:35:48.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:48 smithi162 ceph-mon[44304]: osdmap e332: 12 total, 12 up, 12 in 2024-02-09T17:35:48.709 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 9 16 28491 28475 197.725 171.125 0.00172703 0.00503529 2024-02-09T17:35:49.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:49 smithi099 ceph-mon[39420]: pgmap v904: 37 pgs: 37 active+clean; 1.3 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 160 MiB/s wr, 2.56k op/s 2024-02-09T17:35:49.570 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:35:49.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:49.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:35:49.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:49 smithi179 ceph-mon[43975]: pgmap v904: 37 pgs: 37 active+clean; 1.3 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 160 MiB/s wr, 2.56k op/s 2024-02-09T17:35:49.675 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:35:49.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:49 smithi162 ceph-mon[44304]: pgmap v904: 37 pgs: 37 active+clean; 1.3 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 160 MiB/s wr, 2.56k op/s 2024-02-09T17:35:49.709 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 10 16 31178 31162 194.745 167.938 0.00521112 0.00512853 2024-02-09T17:35:49.718 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:49.744 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:35:49.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:49.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:35:49.851 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:35:49.881 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:49.985 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:35:49.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:49.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:35:50.090 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:35:50.128 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:50.158 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:35:50.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:50.163 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:35:50.264 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:35:50.308 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:50.338 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:35:50.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:50.342 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:35:50.419 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:35:50.420 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:35:50.443 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:35:50.474 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:50.500 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:35:50.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:50.504 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:35:50.605 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:35:50.659 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:50.706 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:35:50.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:50.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:35:50.709 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 11 16 33186 33170 188.449 125.5 0.00284777 0.00530222 2024-02-09T17:35:50.810 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:35:50.857 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:51.320 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.8 to 1 (10000) 2024-02-09T17:35:51.674 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:51 smithi099 ceph-mon[39420]: pgmap v905: 37 pgs: 37 active+clean; 1.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 195 MiB/s wr, 3.12k op/s 2024-02-09T17:35:51.709 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 12 16 35509 35493 184.841 145.188 0.00190341 0.00539516 2024-02-09T17:35:51.778 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:35:51.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:51 smithi162 ceph-mon[44304]: pgmap v905: 37 pgs: 37 active+clean; 1.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 195 MiB/s wr, 3.12k op/s 2024-02-09T17:35:51.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:51 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2555253286' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-02-09T17:35:51.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:51 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-02-09T17:35:51.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:51 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-02-09T17:35:51.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:51 smithi162 ceph-mon[44304]: osdmap e333: 12 total, 12 up, 12 in 2024-02-09T17:35:51.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:51 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2555253286' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-02-09T17:35:51.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:51 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-02-09T17:35:51.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:51 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-02-09T17:35:51.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:51 smithi099 ceph-mon[39420]: osdmap e333: 12 total, 12 up, 12 in 2024-02-09T17:35:52.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:51 smithi179 ceph-mon[43975]: pgmap v905: 37 pgs: 37 active+clean; 1.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 195 MiB/s wr, 3.12k op/s 2024-02-09T17:35:52.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:51 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2555253286' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-02-09T17:35:52.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:51 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-02-09T17:35:52.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:51 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-02-09T17:35:52.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:51 smithi179 ceph-mon[43975]: osdmap e333: 12 total, 12 up, 12 in 2024-02-09T17:35:52.709 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 13 16 37848 37832 181.867 146.188 0.0072644 0.00549375 2024-02-09T17:35:52.728 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:35:52.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:52.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:35:52.837 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:35:52.894 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:53.709 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 14 16 40209 40193 179.416 147.562 0.00679348 0.00556926 2024-02-09T17:35:53.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:53 smithi099 ceph-mon[39420]: pgmap v907: 37 pgs: 37 active+clean; 2.0 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 152 MiB/s wr, 2.44k op/s 2024-02-09T17:35:54.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:53 smithi179 ceph-mon[43975]: pgmap v907: 37 pgs: 37 active+clean; 2.0 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 152 MiB/s wr, 2.44k op/s 2024-02-09T17:35:54.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:53 smithi162 ceph-mon[44304]: pgmap v907: 37 pgs: 37 active+clean; 2.0 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 152 MiB/s wr, 2.44k op/s 2024-02-09T17:35:54.688 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:54 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:35:54.709 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 15 16 42585 42569 177.354 148.5 0.0033295 0.00563419 2024-02-09T17:35:54.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:54 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:35:54.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:54 smithi099 ceph-mon[39420]: osdmap e334: 12 total, 12 up, 12 in 2024-02-09T17:35:55.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:54 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:35:55.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:54 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:35:55.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:54 smithi179 ceph-mon[43975]: osdmap e334: 12 total, 12 up, 12 in 2024-02-09T17:35:55.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:54 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:35:55.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:54 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:35:55.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:54 smithi162 ceph-mon[44304]: osdmap e334: 12 total, 12 up, 12 in 2024-02-09T17:35:55.668 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:35:55.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:55.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:35:55.710 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 16 16 44459 44443 173.589 117.125 0.00423088 0.00575344 2024-02-09T17:35:55.778 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:35:55.815 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:55.844 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:35:55.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:55.850 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:35:55.948 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:55 smithi099 ceph-mon[39420]: pgmap v909: 37 pgs: 37 active+clean; 2.0 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 114 MiB/s wr, 1.82k op/s 2024-02-09T17:35:55.948 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:55 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:35:55.948 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:55 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:35:55.951 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:35:55.981 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:55 smithi162 ceph-mon[44304]: pgmap v909: 37 pgs: 37 active+clean; 2.0 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 114 MiB/s wr, 1.82k op/s 2024-02-09T17:35:55.981 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:55 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:35:55.981 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:55 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:35:55.986 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:56.056 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:35:56.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:56.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:35:56.160 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:35:56.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:55 smithi179 ceph-mon[43975]: pgmap v909: 37 pgs: 37 active+clean; 2.0 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 114 MiB/s wr, 1.82k op/s 2024-02-09T17:35:56.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:55 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:35:56.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:55 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:35:56.196 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:56.222 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:35:56.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:56.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:35:56.327 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:35:56.459 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:56.709 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:56 smithi099 ceph-mon[39420]: osdmap e335: 12 total, 12 up, 12 in 2024-02-09T17:35:56.710 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 17 16 46493 46477 170.855 127.125 0.00271899 0.00584814 2024-02-09T17:35:57.067 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:35:57.068 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:35:57.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:56 smithi179 ceph-mon[43975]: osdmap e335: 12 total, 12 up, 12 in 2024-02-09T17:35:57.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:56 smithi162 ceph-mon[44304]: osdmap e335: 12 total, 12 up, 12 in 2024-02-09T17:35:57.710 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 18 15 48636 48621 168.807 134 0.00802794 0.00591978 2024-02-09T17:35:57.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:57 smithi099 ceph-mon[39420]: pgmap v911: 37 pgs: 37 active+clean; 2.6 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 154 MiB/s wr, 2.46k op/s 2024-02-09T17:35:58.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:57 smithi179 ceph-mon[43975]: pgmap v911: 37 pgs: 37 active+clean; 2.6 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 154 MiB/s wr, 2.46k op/s 2024-02-09T17:35:58.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:57 smithi162 ceph-mon[44304]: pgmap v911: 37 pgs: 37 active+clean; 2.6 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 154 MiB/s wr, 2.46k op/s 2024-02-09T17:35:58.660 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:35:58.662 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:35:58.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:58.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:35:58.696 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 9 1 2024-02-09T17:35:58.710 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 19 16 50514 50498 166.096 117.312 0.0046623 0.00601616 2024-02-09T17:35:58.765 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:35:58.842 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:59.005 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:35:59.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:59.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:35:59.109 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:35:59.146 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:59.172 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:35:59.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:59.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:35:59.277 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:35:59.311 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:59.344 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:35:59.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:59.350 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:35:59.452 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:35:59.513 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:59.679 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:35:59.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:59.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:35:59.701 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-09T17:35:59.710 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:2024-02-09T17:35:59.710319+0000 min lat: 0.00117111 max lat: 0.220727 avg lat: 0.00611467 2024-02-09T17:35:59.710 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-09T17:35:59.710 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 20 16 52161 52145 162.938 102.938 0.00256574 0.00611467 2024-02-09T17:35:59.716 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is unset 2024-02-09T17:35:59.783 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:35:59.813 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:59.845 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:35:59.848 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:35:59.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:35:59.949 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:35:59.981 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:59 smithi179 ceph-mon[43975]: pgmap v912: 37 pgs: 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 123 MiB/s wr, 1.97k op/s 2024-02-09T17:35:59.981 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:35:59 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3471777169' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:35:59.984 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:35:59.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:59 smithi099 ceph-mon[39420]: pgmap v912: 37 pgs: 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 123 MiB/s wr, 1.97k op/s 2024-02-09T17:35:59.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:35:59 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3471777169' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:36:00.008 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:36:00.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:00.012 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:36:00.113 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:36:00.147 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:00.175 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:36:00.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:59 smithi162 ceph-mon[44304]: pgmap v912: 37 pgs: 37 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 123 MiB/s wr, 1.97k op/s 2024-02-09T17:36:00.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:35:59 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3471777169' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:36:00.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:00.179 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:36:00.280 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:36:00.319 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:00.342 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:36:00.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:00.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:36:00.446 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:36:00.482 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:00.509 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:36:00.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:00.512 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:36:00.613 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:36:00.653 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:00.681 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:36:00.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:00.683 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:36:00.710 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 21 16 53937 53921 160.464 111 0.00780007 0.00622783 2024-02-09T17:36:00.785 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:36:00.835 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:00 smithi099 ceph-mon[39420]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:36:00.835 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:00 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3471777169' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:36:00.835 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:00 smithi099 ceph-mon[39420]: osdmap e336: 12 total, 12 up, 12 in 2024-02-09T17:36:00.839 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:00.878 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:36:00.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:00.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:36:00.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:00 smithi179 ceph-mon[43975]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:36:00.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:00 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3471777169' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:36:00.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:00 smithi179 ceph-mon[43975]: osdmap e336: 12 total, 12 up, 12 in 2024-02-09T17:36:00.983 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:36:01.013 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:00 smithi162 ceph-mon[44304]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:36:01.013 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:00 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3471777169' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:36:01.013 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:00 smithi162 ceph-mon[44304]: osdmap e336: 12 total, 12 up, 12 in 2024-02-09T17:36:01.015 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:01.043 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:36:01.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:01.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:36:01.149 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:36:01.191 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:01.710 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 22 16 55849 55833 158.602 119.5 0.00561151 0.00630067 2024-02-09T17:36:01.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:01 smithi099 ceph-mon[39420]: pgmap v914: 37 pgs: 37 active+clean; 3.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 155 MiB/s wr, 2.48k op/s 2024-02-09T17:36:01.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:01 smithi099 ceph-mon[39420]: osdmap e337: 12 total, 12 up, 12 in 2024-02-09T17:36:02.056 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:36:02.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:02.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:36:02.162 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:36:02.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:01 smithi179 ceph-mon[43975]: pgmap v914: 37 pgs: 37 active+clean; 3.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 155 MiB/s wr, 2.48k op/s 2024-02-09T17:36:02.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:01 smithi179 ceph-mon[43975]: osdmap e337: 12 total, 12 up, 12 in 2024-02-09T17:36:02.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:01 smithi162 ceph-mon[44304]: pgmap v914: 37 pgs: 37 active+clean; 3.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 155 MiB/s wr, 2.48k op/s 2024-02-09T17:36:02.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:01 smithi162 ceph-mon[44304]: osdmap e337: 12 total, 12 up, 12 in 2024-02-09T17:36:02.199 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:02.226 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:36:02.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:02.232 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:36:02.333 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:36:02.374 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:02.403 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:36:02.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:02.407 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:36:02.508 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:36:02.535 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:36:02.538 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:02.562 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:36:02.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:02.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:36:02.595 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:36:02.669 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:36:02.711 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 23 16 57665 57649 156.641 113.5 0.00743315 0.00637951 2024-02-09T17:36:02.716 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:03.524 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:36:03.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:03.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:36:03.630 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:36:03.656 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:03.681 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:36:03.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:03.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:36:03.710 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 24 16 59370 59354 154.554 106.562 0.0137315 0.00646642 2024-02-09T17:36:03.785 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:36:03.827 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:03.855 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:36:03.859 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:03.860 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:36:03.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:03 smithi162 ceph-mon[44304]: pgmap v916: 37 pgs: 37 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 116 MiB/s wr, 1.85k op/s 2024-02-09T17:36:03.960 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:36:03.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:03 smithi099 ceph-mon[39420]: pgmap v916: 37 pgs: 37 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 116 MiB/s wr, 1.85k op/s 2024-02-09T17:36:04.015 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:04.100 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-09T17:36:04.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:03 smithi179 ceph-mon[43975]: pgmap v916: 37 pgs: 37 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 116 MiB/s wr, 1.85k op/s 2024-02-09T17:36:04.710 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 25 16 61144 61128 152.806 110.875 0.00583262 0.00654005 2024-02-09T17:36:05.711 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 26 16 62790 62774 150.885 102.875 0.00247735 0.00661157 2024-02-09T17:36:06.140 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:05 smithi099 ceph-mon[39420]: pgmap v917: 37 pgs: 37 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.33k op/s 2024-02-09T17:36:06.141 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:05 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3649948826' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:36:06.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:05 smithi179 ceph-mon[43975]: pgmap v917: 37 pgs: 37 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.33k op/s 2024-02-09T17:36:06.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:05 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3649948826' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:36:06.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:05 smithi162 ceph-mon[44304]: pgmap v917: 37 pgs: 37 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.33k op/s 2024-02-09T17:36:06.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:05 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3649948826' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:36:06.446 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:36:06.446 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:36:06.447 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:36:06.447 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:36:06.447 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:36:06.447 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:36:06.447 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:36:06.447 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:36:06.447 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:36:06.447 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:36:06.447 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:36:06.447 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:36:06.448 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:36:06.448 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:36:06.448 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:36:06.448 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:36:06.448 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:36:06.448 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:36:06.448 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:36:06.448 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:36:06.448 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:36:06.448 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:36:06.449 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:36:06.449 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:36:06.683 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:36:06.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:06.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:36:06.710 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 27 16 64518 64502 149.297 108 0.0159795 0.00669395 2024-02-09T17:36:06.788 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:36:06.830 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:06.855 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:36:06.859 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:06.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:36:06.961 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:36:06.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:06 smithi099 ceph-mon[39420]: 1.a scrub starts 2024-02-09T17:36:06.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:06 smithi099 ceph-mon[39420]: 1.a scrub ok 2024-02-09T17:36:07.006 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:06 smithi162 ceph-mon[44304]: 1.a scrub starts 2024-02-09T17:36:07.006 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:06 smithi162 ceph-mon[44304]: 1.a scrub ok 2024-02-09T17:36:07.011 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:07.037 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:36:07.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:07.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:36:07.141 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:36:07.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:06 smithi179 ceph-mon[43975]: 1.a scrub starts 2024-02-09T17:36:07.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:06 smithi179 ceph-mon[43975]: 1.a scrub ok 2024-02-09T17:36:07.194 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:07.710 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 28 16 65985 65969 147.239 91.6875 0.0221947 0.00678511 2024-02-09T17:36:08.112 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:07 smithi099 ceph-mon[39420]: pgmap v918: 37 pgs: 37 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 134 MiB/s wr, 2.15k op/s 2024-02-09T17:36:08.145 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:36:08.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:08.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:36:08.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:07 smithi179 ceph-mon[43975]: pgmap v918: 37 pgs: 37 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 134 MiB/s wr, 2.15k op/s 2024-02-09T17:36:08.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:07 smithi162 ceph-mon[44304]: pgmap v918: 37 pgs: 37 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 134 MiB/s wr, 2.15k op/s 2024-02-09T17:36:08.249 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:36:08.292 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:08.323 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:36:08.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:08.327 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:36:08.428 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:36:08.472 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:08.493 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:36:08.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:08.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:36:08.603 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:36:08.638 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:08.669 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:36:08.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:08.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:36:08.711 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 29 16 67634 67618 145.715 103.062 0.0114095 0.00685805 2024-02-09T17:36:08.773 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:36:08.800 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:08.825 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:36:08.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:08.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:36:08.930 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:36:08.971 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:08.999 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:36:09.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:09.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:36:09.104 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:36:09.142 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:09.166 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:36:09.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:09.170 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:36:09.271 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:36:09.308 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:09.331 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:36:09.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:09.335 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:36:09.436 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:36:09.475 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:09.506 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:36:09.509 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:09.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:36:09.610 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:36:09.649 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:09.682 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:36:09.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:09.688 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:36:09.711 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 30 16 69187 69171 144.093 97.0625 0.0085063 0.00693553 2024-02-09T17:36:09.789 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:36:09.824 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:09.847 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:36:09.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:09.852 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:36:09.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:09 smithi179 ceph-mon[43975]: pgmap v919: 37 pgs: 37 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 128 MiB/s wr, 2.05k op/s 2024-02-09T17:36:09.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:09 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4178217508' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-02-09T17:36:09.931 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.9 to 1 (10000) 2024-02-09T17:36:09.953 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:36:10.007 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:10.113 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:36:10.113 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:36:10.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:10.121 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:09 smithi099 ceph-mon[39420]: pgmap v919: 37 pgs: 37 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 128 MiB/s wr, 2.05k op/s 2024-02-09T17:36:10.121 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:09 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4178217508' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-02-09T17:36:10.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:36:10.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:09 smithi162 ceph-mon[44304]: pgmap v919: 37 pgs: 37 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 128 MiB/s wr, 2.05k op/s 2024-02-09T17:36:10.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:09 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4178217508' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-02-09T17:36:10.222 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:36:10.261 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:10.291 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:36:10.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:10.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:36:10.397 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:36:10.437 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:10.467 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:36:10.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:10.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:36:10.572 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:36:10.608 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:10.634 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:36:10.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:10.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:36:10.711 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 31 16 70654 70638 142.403 91.6875 0.0104441 0.00700922 2024-02-09T17:36:10.740 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:36:10.796 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:10.936 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:36:10.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:10.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:36:10.941 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:10 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4178217508' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-02-09T17:36:10.941 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:10 smithi099 ceph-mon[39420]: osdmap e338: 12 total, 12 up, 12 in 2024-02-09T17:36:11.041 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:36:11.075 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:10 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4178217508' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-02-09T17:36:11.075 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:10 smithi179 ceph-mon[43975]: osdmap e338: 12 total, 12 up, 12 in 2024-02-09T17:36:11.076 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:11.103 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:36:11.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:11.108 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:36:11.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:10 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4178217508' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-02-09T17:36:11.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:10 smithi162 ceph-mon[44304]: osdmap e338: 12 total, 12 up, 12 in 2024-02-09T17:36:11.209 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:36:11.250 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:11.283 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:36:11.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:11.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:36:11.386 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:36:11.429 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:11.455 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:36:11.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:11.460 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:36:11.561 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:36:11.616 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:11.711 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 32 16 72262 72246 141.093 100.5 0.00937425 0.00708335 2024-02-09T17:36:11.858 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:36:12.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:11 smithi179 ceph-mon[43975]: pgmap v921: 37 pgs: 37 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 108 MiB/s wr, 1.73k op/s 2024-02-09T17:36:12.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:11 smithi179 ceph-mon[43975]: osdmap e339: 12 total, 12 up, 12 in 2024-02-09T17:36:12.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:11 smithi162 ceph-mon[44304]: pgmap v921: 37 pgs: 37 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 108 MiB/s wr, 1.73k op/s 2024-02-09T17:36:12.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:11 smithi162 ceph-mon[44304]: osdmap e339: 12 total, 12 up, 12 in 2024-02-09T17:36:12.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:11 smithi099 ceph-mon[39420]: pgmap v921: 37 pgs: 37 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 108 MiB/s wr, 1.73k op/s 2024-02-09T17:36:12.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:11 smithi099 ceph-mon[39420]: osdmap e339: 12 total, 12 up, 12 in 2024-02-09T17:36:12.698 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 10 1 2024-02-09T17:36:12.711 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 33 16 73853 73837 139.83 99.4375 0.00774671 0.00714746 2024-02-09T17:36:12.988 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:36:12.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:12.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:36:13.094 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:36:13.123 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:13.149 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:36:13.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:13.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:36:13.255 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:36:13.292 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:13.320 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:36:13.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:13.326 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:36:13.427 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:36:13.465 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:13.478 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:36:13.478 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [], 2024-02-09T17:36:13.478 INFO:teuthology.orchestra.run.smithi099.stdout: "complaint_time": 30, 2024-02-09T17:36:13.478 INFO:teuthology.orchestra.run.smithi099.stdout: "num_blocked_ops": 0 2024-02-09T17:36:13.479 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:36:13.489 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:36:13.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:13.493 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:36:13.594 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:36:13.625 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:13.652 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:36:13.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:13.656 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:36:13.711 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 34 16 75251 75235 138.287 87.375 0.00747 0.00722701 2024-02-09T17:36:13.758 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:36:13.793 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:13.825 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:36:13.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:13.828 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:36:13.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:13 smithi179 ceph-mon[43975]: pgmap v923: 37 pgs: 37 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 128 MiB/s wr, 2.04k op/s 2024-02-09T17:36:13.929 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:36:13.980 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:14.032 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:36:14.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:14.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:36:14.137 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:36:14.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:13 smithi162 ceph-mon[44304]: pgmap v923: 37 pgs: 37 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 128 MiB/s wr, 2.04k op/s 2024-02-09T17:36:14.185 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:14.212 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:36:14.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:14.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:36:14.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:13 smithi099 ceph-mon[39420]: pgmap v923: 37 pgs: 37 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 128 MiB/s wr, 2.04k op/s 2024-02-09T17:36:14.318 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:36:14.355 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:14.372 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:36:14.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:14.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:36:14.477 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:36:14.519 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:14.542 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:36:14.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:14.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:36:14.647 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:36:14.686 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:14.711 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 35 16 76700 76684 136.924 90.5625 0.00929004 0.00729966 2024-02-09T17:36:14.718 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:36:14.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:14.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:36:14.825 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:36:14.849 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:14 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:36:14.849 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:14 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:36:14.849 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:14 smithi162 ceph-mon[44304]: osdmap e340: 12 total, 12 up, 12 in 2024-02-09T17:36:14.849 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:14 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1898327461' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:36:14.849 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:14 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:36:14.853 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is unset 2024-02-09T17:36:14.863 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:36:14.866 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:14.893 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:36:14.896 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:14.896 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:36:14.997 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:36:15.051 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:15.089 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:36:15.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:15.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:36:15.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:14 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:36:15.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:14 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:36:15.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:14 smithi179 ceph-mon[43975]: osdmap e340: 12 total, 12 up, 12 in 2024-02-09T17:36:15.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:14 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1898327461' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:36:15.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:14 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:36:15.194 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:36:15.239 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:14 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:36:15.239 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:14 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:36:15.239 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:14 smithi099 ceph-mon[39420]: osdmap e340: 12 total, 12 up, 12 in 2024-02-09T17:36:15.239 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:14 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1898327461' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:36:15.239 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:14 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:36:15.241 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:15.308 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:36:15.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:15.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:36:15.416 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:36:15.456 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:15.490 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:36:15.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:15.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:36:15.594 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:36:15.657 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:15.712 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 36 16 78156 78140 135.648 91 0.00988718 0.00736824 2024-02-09T17:36:15.852 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:15 smithi162 ceph-mon[44304]: pgmap v925: 37 pgs: 37 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s 2024-02-09T17:36:15.852 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:15 smithi162 ceph-mon[44304]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:36:15.852 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:15 smithi162 ceph-mon[44304]: Cluster is now healthy 2024-02-09T17:36:15.852 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:15 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:36:15.852 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:15 smithi162 ceph-mon[44304]: osdmap e341: 12 total, 12 up, 12 in 2024-02-09T17:36:15.954 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:15 smithi099 ceph-mon[39420]: pgmap v925: 37 pgs: 37 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s 2024-02-09T17:36:15.954 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:15 smithi099 ceph-mon[39420]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:36:15.954 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:15 smithi099 ceph-mon[39420]: Cluster is now healthy 2024-02-09T17:36:15.954 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:15 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:36:15.954 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:15 smithi099 ceph-mon[39420]: osdmap e341: 12 total, 12 up, 12 in 2024-02-09T17:36:15.957 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:36:15.985 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:36:15.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:15.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:36:16.007 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-09T17:36:16.091 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:36:16.122 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:16.141 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:36:16.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:16.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:36:16.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:15 smithi179 ceph-mon[43975]: pgmap v925: 37 pgs: 37 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s 2024-02-09T17:36:16.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:15 smithi179 ceph-mon[43975]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:36:16.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:15 smithi179 ceph-mon[43975]: Cluster is now healthy 2024-02-09T17:36:16.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:15 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:36:16.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:15 smithi179 ceph-mon[43975]: osdmap e341: 12 total, 12 up, 12 in 2024-02-09T17:36:16.245 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:36:16.310 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:16.711 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 37 16 79283 79267 133.885 70.4375 0.00794581 0.00746529 2024-02-09T17:36:17.319 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:17 smithi099 ceph-mon[39420]: pgmap v927: 37 pgs: 37 active+clean; 4.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 98 MiB/s wr, 1.57k op/s 2024-02-09T17:36:17.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:17 smithi179 ceph-mon[43975]: pgmap v927: 37 pgs: 37 active+clean; 4.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 98 MiB/s wr, 1.57k op/s 2024-02-09T17:36:17.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:17 smithi162 ceph-mon[44304]: pgmap v927: 37 pgs: 37 active+clean; 4.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 98 MiB/s wr, 1.57k op/s 2024-02-09T17:36:17.711 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 38 16 80653 80637 132.615 85.625 0.00602746 0.00753682 2024-02-09T17:36:17.771 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:36:17.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:17.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:36:17.879 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:36:17.943 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:18.302 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:36:18.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:18.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:36:18.409 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:36:18.452 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:18.483 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:36:18.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:18.486 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:36:18.587 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:36:18.620 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:18 smithi179 ceph-mon[43975]: 1.10 scrub starts 2024-02-09T17:36:18.620 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:18 smithi179 ceph-mon[43975]: 1.10 scrub ok 2024-02-09T17:36:18.623 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:18.646 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:36:18.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:18.653 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:36:18.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:18 smithi162 ceph-mon[44304]: 1.10 scrub starts 2024-02-09T17:36:18.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:18 smithi162 ceph-mon[44304]: 1.10 scrub ok 2024-02-09T17:36:18.712 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 39 16 81955 81939 131.301 81.375 0.00907203 0.007612 2024-02-09T17:36:18.752 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:18 smithi099 ceph-mon[39420]: 1.10 scrub starts 2024-02-09T17:36:18.752 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:18 smithi099 ceph-mon[39420]: 1.10 scrub ok 2024-02-09T17:36:18.755 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:36:18.820 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:18.902 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:36:18.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:18.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:36:19.015 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:36:19.041 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:19.058 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:36:19.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:19.061 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:36:19.162 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:36:19.209 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:19.311 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:36:19.315 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:36:19.317 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:19.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:36:19.353 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:19 smithi179 ceph-mon[43975]: pgmap v928: 37 pgs: 37 active+clean; 4.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.27k op/s 2024-02-09T17:36:19.353 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:19 smithi179 ceph-mon[43975]: 1.11 scrub starts 2024-02-09T17:36:19.353 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:19 smithi179 ceph-mon[43975]: 1.11 scrub ok 2024-02-09T17:36:19.418 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:36:19.467 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:19.482 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set noscrub 2024-02-09T17:36:19.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:19 smithi099 ceph-mon[39420]: pgmap v928: 37 pgs: 37 active+clean; 4.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.27k op/s 2024-02-09T17:36:19.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:19 smithi099 ceph-mon[39420]: 1.11 scrub starts 2024-02-09T17:36:19.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:19 smithi099 ceph-mon[39420]: 1.11 scrub ok 2024-02-09T17:36:19.497 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:36:19.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:19.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:36:19.603 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:36:19.640 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:19.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:19.671 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:36:19.671 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:36:19.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:19 smithi162 ceph-mon[44304]: pgmap v928: 37 pgs: 37 active+clean; 4.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.27k op/s 2024-02-09T17:36:19.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:19 smithi162 ceph-mon[44304]: 1.11 scrub starts 2024-02-09T17:36:19.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:19 smithi162 ceph-mon[44304]: 1.11 scrub ok 2024-02-09T17:36:19.712 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:2024-02-09T17:36:19.711964+0000 min lat: 0.00117111 max lat: 0.220727 avg lat: 0.00770356 2024-02-09T17:36:19.712 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-09T17:36:19.712 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 40 16 83060 83044 129.745 69.0625 0.0141052 0.00770356 2024-02-09T17:36:19.771 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:36:19.833 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:20.712 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 41 16 84456 84440 128.708 87.25 0.00278661 0.0077603 2024-02-09T17:36:21.003 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:36:21.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:21.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:36:21.108 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:36:21.141 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:21.166 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:36:21.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:21.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:36:21.273 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:36:21.307 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:21.340 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:36:21.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:21.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:36:21.447 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:36:21.487 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:21.519 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:36:21.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:21.525 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:36:21.566 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:36:21.566 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:36:21.566 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:36:21.625 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:36:21.642 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:36:21.643 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:36:21.654 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:21.677 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:36:21.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:21.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:36:21.696 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:36:21.712 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 42 16 85721 85705 127.526 79.0625 0.0033593 0.00783623 2024-02-09T17:36:21.745 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:36:21.777 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:36:21.781 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:36:21.819 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:21.839 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:36:21.839 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:36:21.859 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:36:21.901 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:36:21.901 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:36:21.901 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:36:21.901 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:36:21.901 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:36:21.901 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:36:21.901 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:36:21.901 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:36:21.902 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:36:21.902 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:36:21.902 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:36:21.902 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:36:21.902 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:36:21.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:21 smithi179 ceph-mon[43975]: 1.8 scrub starts 2024-02-09T17:36:21.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:21 smithi179 ceph-mon[43975]: 1.8 scrub ok 2024-02-09T17:36:21.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:21 smithi179 ceph-mon[43975]: pgmap v929: 37 pgs: 37 active+clean; 5.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.48k op/s 2024-02-09T17:36:21.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:21 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3407525434' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:36:21.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:21 smithi162 ceph-mon[44304]: 1.8 scrub starts 2024-02-09T17:36:21.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:21 smithi162 ceph-mon[44304]: 1.8 scrub ok 2024-02-09T17:36:21.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:21 smithi162 ceph-mon[44304]: pgmap v929: 37 pgs: 37 active+clean; 5.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.48k op/s 2024-02-09T17:36:21.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:21 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3407525434' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:36:21.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:21 smithi099 ceph-mon[39420]: 1.8 scrub starts 2024-02-09T17:36:21.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:21 smithi099 ceph-mon[39420]: 1.8 scrub ok 2024-02-09T17:36:21.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:21 smithi099 ceph-mon[39420]: pgmap v929: 37 pgs: 37 active+clean; 5.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.48k op/s 2024-02-09T17:36:21.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:21 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3407525434' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:36:22.277 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:36:22.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:22.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:36:22.384 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:36:22.419 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:22.454 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:36:22.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:22.460 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:36:22.561 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:36:22.629 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:22.712 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 43 16 87024 87008 126.454 81.4375 0.0122352 0.00790308 2024-02-09T17:36:22.734 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.10 to 1 (10000) 2024-02-09T17:36:22.968 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:22 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3667329401' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-02-09T17:36:22.969 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:22 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-02-09T17:36:23.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:22 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3667329401' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-02-09T17:36:23.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:22 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-02-09T17:36:23.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:22 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3667329401' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-02-09T17:36:23.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:22 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-02-09T17:36:23.293 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:36:23.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:23.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:36:23.397 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:36:23.428 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:23.467 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:36:23.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:23.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:36:23.573 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:36:23.620 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:23.647 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:36:23.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:23.651 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:36:23.712 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 44 16 88269 88253 125.348 77.8125 0.00821796 0.00797325 2024-02-09T17:36:23.752 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:36:23.810 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:23.876 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:36:23.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:23.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:36:23.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:23 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:36:23.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:23 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:36:23.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:23 smithi179 ceph-mon[43975]: 1.f deep-scrub starts 2024-02-09T17:36:23.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:23 smithi179 ceph-mon[43975]: 1.f deep-scrub ok 2024-02-09T17:36:23.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:23 smithi179 ceph-mon[43975]: pgmap v930: 37 pgs: 37 active+clean; 5.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.52k op/s 2024-02-09T17:36:23.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:23 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-02-09T17:36:23.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:23 smithi179 ceph-mon[43975]: osdmap e342: 12 total, 12 up, 12 in 2024-02-09T17:36:23.982 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:36:23.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:23 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:36:23.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:23 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:36:23.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:23 smithi099 ceph-mon[39420]: 1.f deep-scrub starts 2024-02-09T17:36:23.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:23 smithi099 ceph-mon[39420]: 1.f deep-scrub ok 2024-02-09T17:36:23.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:23 smithi099 ceph-mon[39420]: pgmap v930: 37 pgs: 37 active+clean; 5.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.52k op/s 2024-02-09T17:36:23.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:23 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-02-09T17:36:23.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:23 smithi099 ceph-mon[39420]: osdmap e342: 12 total, 12 up, 12 in 2024-02-09T17:36:24.017 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:23 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:36:24.017 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:23 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:36:24.017 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:23 smithi162 ceph-mon[44304]: 1.f deep-scrub starts 2024-02-09T17:36:24.017 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:23 smithi162 ceph-mon[44304]: 1.f deep-scrub ok 2024-02-09T17:36:24.017 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:23 smithi162 ceph-mon[44304]: pgmap v930: 37 pgs: 37 active+clean; 5.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.52k op/s 2024-02-09T17:36:24.018 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:23 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-02-09T17:36:24.018 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:23 smithi162 ceph-mon[44304]: osdmap e342: 12 total, 12 up, 12 in 2024-02-09T17:36:24.018 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:24.044 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:36:24.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:24.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:36:24.149 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:36:24.187 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:24.230 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:36:24.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:24.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:36:24.296 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:36:24.337 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:36:24.368 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:24.404 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:36:24.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:24.409 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:36:24.510 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:36:24.555 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:24.584 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:36:24.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:24.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:36:24.691 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:36:24.712 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 45 16 89611 89595 124.426 83.875 0.0121003 0.00803245 2024-02-09T17:36:24.729 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:24.758 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:36:24.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:24.761 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:36:24.862 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:36:24.899 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:24.923 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:36:24.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:24.928 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:36:25.029 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:36:25.069 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:25.104 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:36:25.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:25.109 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:36:25.210 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:36:25.248 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:25.252 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 11 1 2024-02-09T17:36:25.280 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:36:25.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:25.283 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:36:25.384 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:36:25.415 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:25.438 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:36:25.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:25.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:36:25.542 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:36:25.579 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:25.607 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:36:25.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:25.611 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:36:25.711 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:36:25.712 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 46 16 90906 90890 123.481 80.9375 0.00398925 0.00809411 2024-02-09T17:36:25.792 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:25.810 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:36:25.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:25 smithi179 ceph-mon[43975]: pgmap v932: 37 pgs: 37 active+clean; 5.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.41k op/s 2024-02-09T17:36:25.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:25 smithi099 ceph-mon[39420]: pgmap v932: 37 pgs: 37 active+clean; 5.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.41k op/s 2024-02-09T17:36:26.006 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:36:26.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:26.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:36:26.114 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:36:26.153 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:26.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:25 smithi162 ceph-mon[44304]: pgmap v932: 37 pgs: 37 active+clean; 5.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.41k op/s 2024-02-09T17:36:26.181 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:36:26.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:26.184 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:36:26.285 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:36:26.346 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:26.714 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 47 16 92154 92138 122.509 78 0.0110063 0.00815888 2024-02-09T17:36:27.316 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:36:27.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:27.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:36:27.423 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:36:27.495 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:27.530 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:36:27.530 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:36:27.618 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:27 smithi099 ceph-mon[39420]: osdmap e343: 12 total, 12 up, 12 in 2024-02-09T17:36:27.618 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:27 smithi099 ceph-mon[39420]: pgmap v934: 37 pgs: 37 active+clean; 5.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 94 MiB/s wr, 1.50k op/s 2024-02-09T17:36:27.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:27 smithi179 ceph-mon[43975]: osdmap e343: 12 total, 12 up, 12 in 2024-02-09T17:36:27.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:27 smithi179 ceph-mon[43975]: pgmap v934: 37 pgs: 37 active+clean; 5.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 94 MiB/s wr, 1.50k op/s 2024-02-09T17:36:27.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:27 smithi162 ceph-mon[44304]: osdmap e343: 12 total, 12 up, 12 in 2024-02-09T17:36:27.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:27 smithi162 ceph-mon[44304]: pgmap v934: 37 pgs: 37 active+clean; 5.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 94 MiB/s wr, 1.50k op/s 2024-02-09T17:36:27.714 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 48 16 93386 93370 121.561 77 0.00330814 0.00821426 2024-02-09T17:36:28.162 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:36:28.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:28.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:36:28.268 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:36:28.337 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:28.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:28 smithi099 ceph-mon[39420]: 1.13 scrub starts 2024-02-09T17:36:28.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:28 smithi099 ceph-mon[39420]: 1.13 scrub ok 2024-02-09T17:36:28.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:28 smithi179 ceph-mon[43975]: 1.13 scrub starts 2024-02-09T17:36:28.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:28 smithi179 ceph-mon[43975]: 1.13 scrub ok 2024-02-09T17:36:28.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:28 smithi162 ceph-mon[44304]: 1.13 scrub starts 2024-02-09T17:36:28.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:28 smithi162 ceph-mon[44304]: 1.13 scrub ok 2024-02-09T17:36:28.714 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 49 16 94620 94604 120.654 77.125 0.00748813 0.00828435 2024-02-09T17:36:29.103 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:36:29.105 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:36:29.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:29.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:36:29.122 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:36:29.145 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-09T17:36:29.211 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:36:29.255 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:29.284 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:36:29.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:29.289 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:36:29.369 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:29 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:36:29.369 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:29 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:36:29.369 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:29 smithi099 ceph-mon[39420]: osdmap e344: 12 total, 12 up, 12 in 2024-02-09T17:36:29.369 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:29 smithi099 ceph-mon[39420]: pgmap v936: 37 pgs: 37 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-02-09T17:36:29.391 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:36:29.464 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:29.522 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:36:29.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:29.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:36:29.642 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:36:29.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:29 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:36:29.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:29 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:36:29.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:29 smithi179 ceph-mon[43975]: osdmap e344: 12 total, 12 up, 12 in 2024-02-09T17:36:29.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:29 smithi179 ceph-mon[43975]: pgmap v936: 37 pgs: 37 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-02-09T17:36:29.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:29 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:36:29.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:29 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:36:29.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:29 smithi162 ceph-mon[44304]: osdmap e344: 12 total, 12 up, 12 in 2024-02-09T17:36:29.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:29 smithi162 ceph-mon[44304]: pgmap v936: 37 pgs: 37 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-02-09T17:36:29.698 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:29.714 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 50 16 95743 95727 119.644 70.1875 0.00718821 0.0083544 2024-02-09T17:36:29.752 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:36:29.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:29.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:36:29.857 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:36:29.890 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:29.918 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:36:29.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:29.925 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:36:30.026 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:36:30.060 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:30.094 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:36:30.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:30.097 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:36:30.198 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:36:30.224 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:30.242 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:36:30.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:30.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:36:30.346 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:36:30.417 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:30.630 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:36:30.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:30.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:36:30.714 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 51 16 96758 96742 118.542 63.4375 0.00420948 0.00843107 2024-02-09T17:36:30.743 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:36:30.905 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:31.353 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is set 2024-02-09T17:36:31.700 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:31 smithi099 ceph-mon[39420]: pgmap v937: 37 pgs: 37 active+clean; 5.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s 2024-02-09T17:36:31.700 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:31 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3139069693' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:36:31.700 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:31 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:36:31.701 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:31 smithi099 ceph-mon[39420]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:36:31.701 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:31 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:36:31.701 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:31 smithi099 ceph-mon[39420]: osdmap e345: 12 total, 12 up, 12 in 2024-02-09T17:36:31.714 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 52 15 97979 97964 117.731 76.375 0.00984014 0.00849022 2024-02-09T17:36:32.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:31 smithi179 ceph-mon[43975]: pgmap v937: 37 pgs: 37 active+clean; 5.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s 2024-02-09T17:36:32.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:31 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3139069693' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:36:32.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:31 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:36:32.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:31 smithi179 ceph-mon[43975]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:36:32.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:31 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:36:32.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:31 smithi179 ceph-mon[43975]: osdmap e345: 12 total, 12 up, 12 in 2024-02-09T17:36:32.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:31 smithi162 ceph-mon[44304]: pgmap v937: 37 pgs: 37 active+clean; 5.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s 2024-02-09T17:36:32.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:31 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3139069693' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:36:32.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:31 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:36:32.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:31 smithi162 ceph-mon[44304]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:36:32.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:31 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:36:32.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:31 smithi162 ceph-mon[44304]: osdmap e345: 12 total, 12 up, 12 in 2024-02-09T17:36:32.665 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:36:32.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:32.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:36:32.716 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 53 16 99352 99336 117.125 85.75 0.0118424 0.0085341 2024-02-09T17:36:32.769 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:36:32.843 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:32.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:32 smithi099 ceph-mon[39420]: 4.1 scrub starts 2024-02-09T17:36:32.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:32 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:36:32.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:32 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:36:32.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:32 smithi099 ceph-mon[39420]: osdmap e346: 12 total, 12 up, 12 in 2024-02-09T17:36:33.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:32 smithi179 ceph-mon[43975]: 4.1 scrub starts 2024-02-09T17:36:33.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:32 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:36:33.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:32 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:36:33.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:32 smithi179 ceph-mon[43975]: osdmap e346: 12 total, 12 up, 12 in 2024-02-09T17:36:33.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:32 smithi162 ceph-mon[44304]: 4.1 scrub starts 2024-02-09T17:36:33.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:32 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:36:33.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:32 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:36:33.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:32 smithi162 ceph-mon[44304]: osdmap e346: 12 total, 12 up, 12 in 2024-02-09T17:36:33.702 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:33 smithi099 ceph-mon[39420]: pgmap v940: 37 pgs: 37 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.06k op/s 2024-02-09T17:36:33.716 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 54 16 100641 100625 116.448 80.5625 0.00789576 0.00858257 2024-02-09T17:36:34.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:33 smithi179 ceph-mon[43975]: pgmap v940: 37 pgs: 37 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.06k op/s 2024-02-09T17:36:34.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:33 smithi162 ceph-mon[44304]: pgmap v940: 37 pgs: 37 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.06k op/s 2024-02-09T17:36:34.716 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 55 15 101845 101830 115.7 75.3125 0.00969464 0.00863864 2024-02-09T17:36:35.716 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 56 16 102963 102947 114.88 69.8125 0.00640808 0.00870115 2024-02-09T17:36:35.765 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:36:35.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:35.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:36:35.869 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:36:35.934 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:35 smithi099 ceph-mon[39420]: pgmap v941: 37 pgs: 37 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s 2024-02-09T17:36:35.939 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:36.028 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:36:36.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:36.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:36:36.136 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:36:36.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:35 smithi179 ceph-mon[43975]: pgmap v941: 37 pgs: 37 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s 2024-02-09T17:36:36.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:35 smithi162 ceph-mon[44304]: pgmap v941: 37 pgs: 37 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s 2024-02-09T17:36:36.188 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:36:36.238 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:36.264 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:36:36.318 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:36:36.358 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:36:36.377 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:36:36.379 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.11 to 1 (10000) 2024-02-09T17:36:36.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:36.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:36:36.392 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:36:36.419 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:36:36.448 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:36:36.476 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:36:36.485 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:36:36.512 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:36.514 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:36:36.535 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:36:36.539 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:36:36.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:36.540 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:36:36.570 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:36:36.607 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:36:36.640 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:36:36.642 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:36:36.648 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:36:36.648 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:36:36.648 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:36:36.648 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:36:36.649 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:36:36.649 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:36:36.649 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:36:36.649 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:36:36.649 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:36:36.649 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:36:36.649 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:36:36.649 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:36:36.681 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:36.709 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:36:36.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:36.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:36:36.716 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 57 16 103964 103948 113.962 62.5625 0.0129962 0.00877102 2024-02-09T17:36:36.815 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:36:36.856 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:36 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4150771547' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:36:36.856 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:36 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2737292395' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-02-09T17:36:36.856 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:36 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2737292395' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-02-09T17:36:36.856 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:36 smithi099 ceph-mon[39420]: osdmap e347: 12 total, 12 up, 12 in 2024-02-09T17:36:36.879 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:36.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:36 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4150771547' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:36:36.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:36 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2737292395' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-02-09T17:36:36.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:36 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2737292395' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-02-09T17:36:36.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:36 smithi179 ceph-mon[43975]: osdmap e347: 12 total, 12 up, 12 in 2024-02-09T17:36:37.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:36 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4150771547' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:36:37.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:36 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2737292395' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-02-09T17:36:37.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:36 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2737292395' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-02-09T17:36:37.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:36 smithi162 ceph-mon[44304]: osdmap e347: 12 total, 12 up, 12 in 2024-02-09T17:36:37.678 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-09T17:36:37.716 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 58 16 105096 105080 113.217 70.75 0.00986762 0.00882888 2024-02-09T17:36:37.944 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:36:37.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:37.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:36:37.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:37 smithi099 ceph-mon[39420]: pgmap v943: 37 pgs: 37 active+clean; 6.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-02-09T17:36:38.050 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:36:38.082 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:37 smithi162 ceph-mon[44304]: pgmap v943: 37 pgs: 37 active+clean; 6.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-02-09T17:36:38.090 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:38.112 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:36:38.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:38.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:36:38.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:37 smithi179 ceph-mon[43975]: pgmap v943: 37 pgs: 37 active+clean; 6.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-02-09T17:36:38.217 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:36:38.265 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:38.680 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 0 1 2024-02-09T17:36:38.690 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:36:38.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:38.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:36:38.716 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 59 16 106298 106282 112.571 75.125 0.0176562 0.00887889 2024-02-09T17:36:38.801 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:36:38.837 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:38.867 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:36:38.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:38.874 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:36:38.975 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:36:39.027 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:39.057 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:36:39.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:39.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:36:39.165 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:36:39.197 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:39.225 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:36:39.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:39.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:36:39.330 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:36:39.384 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:39.448 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:36:39.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:39.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:36:39.554 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:36:39.602 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:39.716 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:2024-02-09T17:36:39.716664+0000 min lat: 0.00117111 max lat: 0.234475 avg lat: 0.0089314 2024-02-09T17:36:39.716 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-09T17:36:39.717 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 60 16 107469 107453 111.915 73.1875 0.0144601 0.0089314 2024-02-09T17:36:39.878 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:39 smithi099 ceph-mon[39420]: pgmap v944: 37 pgs: 37 active+clean; 6.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-02-09T17:36:40.127 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:36:40.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:40.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:36:40.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:39 smithi179 ceph-mon[43975]: pgmap v944: 37 pgs: 37 active+clean; 6.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-02-09T17:36:40.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:39 smithi162 ceph-mon[44304]: pgmap v944: 37 pgs: 37 active+clean; 6.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-02-09T17:36:40.234 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:36:40.268 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:40.294 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:36:40.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:40.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:36:40.403 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:36:40.406 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:36:40.429 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:40.449 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:36:40.449 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [ 2024-02-09T17:36:40.449 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:36:40.449 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:108212 4.4 4:2c9b2f25:::benchmark_data_smithi179_103809_object108211:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e347)", 2024-02-09T17:36:40.449 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:36:40.432147+0000", 2024-02-09T17:36:40.449 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.015982946000000001, 2024-02-09T17:36:40.449 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.016024439000000001, 2024-02-09T17:36:40.449 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:36:40.449 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "waiting for sub ops", 2024-02-09T17:36:40.449 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:36:40.449 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:36:40.450 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:36:40.450 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 108212 2024-02-09T17:36:40.450 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:36:40.450 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:36:40.450 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:36:40.450 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:36:40.450 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:36:40.432147+0000", 2024-02-09T17:36:40.450 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:36:40.450 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:36:40.450 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:36:40.450 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:36:40.450 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:36:40.432147+0000", 2024-02-09T17:36:40.450 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:36:40.451 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:36:40.451 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:36:40.451 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:36:40.451 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:36:40.432148+0000", 2024-02-09T17:36:40.451 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.7880000000000001e-06 2024-02-09T17:36:40.451 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:36:40.451 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:36:40.451 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:36:40.451 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:36:40.432212+0000", 2024-02-09T17:36:40.451 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.3701000000000006e-05 2024-02-09T17:36:40.451 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:36:40.451 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:36:40.451 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:36:40.452 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:36:40.432229+0000", 2024-02-09T17:36:40.452 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.6781000000000001e-05 2024-02-09T17:36:40.452 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:36:40.452 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:36:40.452 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:36:40.452 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:36:40.432238+0000", 2024-02-09T17:36:40.452 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.8710000000000003e-06 2024-02-09T17:36:40.452 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:36:40.452 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:36:40.452 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:36:40.452 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:36:40.432301+0000", 2024-02-09T17:36:40.452 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.2873000000000006e-05 2024-02-09T17:36:40.453 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:36:40.453 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:36:40.453 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:36:40.453 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:36:40.432432+0000", 2024-02-09T17:36:40.453 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013167800000000001 2024-02-09T17:36:40.453 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:36:40.453 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:36:40.453 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 2,3", 2024-02-09T17:36:40.453 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:36:40.432531+0000", 2024-02-09T17:36:40.453 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.8952999999999999e-05 2024-02-09T17:36:40.453 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:36:40.453 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:36:40.453 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:36:40.453 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:36:40.438955+0000", 2024-02-09T17:36:40.454 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0064233060000000002 2024-02-09T17:36:40.454 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:36:40.454 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:36:40.454 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:36:40.454 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:36:40.439646+0000", 2024-02-09T17:36:40.454 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00069119199999999998 2024-02-09T17:36:40.454 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:36:40.454 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:36:40.454 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:36:40.454 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:36:40.454 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:36:40.454 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:108213 4.4 4:23ae504a:::benchmark_data_smithi179_103809_object108212:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e347)", 2024-02-09T17:36:40.454 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:36:40.432257+0000", 2024-02-09T17:36:40.454 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.015872654999999999, 2024-02-09T17:36:40.455 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.016013254000000001, 2024-02-09T17:36:40.455 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:36:40.455 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "waiting for sub ops", 2024-02-09T17:36:40.455 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:36:40.455 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:36:40.455 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:36:40.455 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 108213 2024-02-09T17:36:40.455 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:36:40.455 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:36:40.455 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:36:40.455 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:36:40.455 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:36:40.432257+0000", 2024-02-09T17:36:40.455 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:36:40.456 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:36:40.456 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:36:40.456 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:36:40.456 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:36:40.432257+0000", 2024-02-09T17:36:40.456 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:36:40.456 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:36:40.456 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:36:40.456 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:36:40.456 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:36:40.432257+0000", 2024-02-09T17:36:40.456 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2300000000000001e-07 2024-02-09T17:36:40.456 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:36:40.456 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:36:40.456 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:36:40.457 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:36:40.432289+0000", 2024-02-09T17:36:40.457 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.1250000000000001e-05 2024-02-09T17:36:40.457 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:36:40.457 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:36:40.457 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:36:40.457 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:36:40.432296+0000", 2024-02-09T17:36:40.457 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.4290000000000004e-06 2024-02-09T17:36:40.457 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:36:40.457 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:36:40.457 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:36:40.457 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:36:40.432302+0000", 2024-02-09T17:36:40.457 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.66e-06 2024-02-09T17:36:40.457 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:36:40.457 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:36:40.458 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:36:40.458 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:36:40.435707+0000", 2024-02-09T17:36:40.458 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0034056239999999999 2024-02-09T17:36:40.458 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:36:40.458 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:36:40.458 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:36:40.458 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:36:40.435850+0000", 2024-02-09T17:36:40.458 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000142247 2024-02-09T17:36:40.458 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:36:40.458 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:36:40.458 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 2,3", 2024-02-09T17:36:40.458 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:36:40.435991+0000", 2024-02-09T17:36:40.458 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000141531 2024-02-09T17:36:40.458 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:36:40.458 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:36:40.459 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:36:40.459 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:36:40.439090+0000", 2024-02-09T17:36:40.459 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0030991270000000001 2024-02-09T17:36:40.459 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:36:40.459 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:36:40.459 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:36:40.459 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:36:40.447583+0000", 2024-02-09T17:36:40.459 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0084923810000000002 2024-02-09T17:36:40.459 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:36:40.459 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:36:40.459 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:36:40.459 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:36:40.459 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:36:40.459 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:108216 4.4 4:2100ff67:::benchmark_data_smithi179_103809_object108215:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e347)", 2024-02-09T17:36:40.460 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:36:40.444159+0000", 2024-02-09T17:36:40.460 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.0039706089999999999, 2024-02-09T17:36:40.460 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.004199805, 2024-02-09T17:36:40.460 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:36:40.460 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "waiting for sub ops", 2024-02-09T17:36:40.460 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:36:40.460 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:36:40.460 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:36:40.460 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 108216 2024-02-09T17:36:40.460 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:36:40.460 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:36:40.460 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:36:40.460 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:36:40.460 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:36:40.444159+0000", 2024-02-09T17:36:40.461 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:36:40.461 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:36:40.461 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:36:40.461 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:36:40.461 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:36:40.444159+0000", 2024-02-09T17:36:40.461 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:36:40.461 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:36:40.461 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:36:40.461 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:36:40.461 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:36:40.444161+0000", 2024-02-09T17:36:40.461 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.77e-06 2024-02-09T17:36:40.461 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:36:40.461 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:36:40.461 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:36:40.461 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:36:40.444200+0000", 2024-02-09T17:36:40.461 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9156999999999998e-05 2024-02-09T17:36:40.462 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:36:40.462 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:36:40.462 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:36:40.462 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:36:40.444216+0000", 2024-02-09T17:36:40.462 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.6198e-05 2024-02-09T17:36:40.462 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:36:40.462 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:36:40.462 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:36:40.462 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:36:40.444227+0000", 2024-02-09T17:36:40.462 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0883999999999999e-05 2024-02-09T17:36:40.462 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:36:40.462 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:36:40.462 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:36:40.462 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:36:40.444296+0000", 2024-02-09T17:36:40.463 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.8895999999999999e-05 2024-02-09T17:36:40.463 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:36:40.463 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:36:40.463 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:36:40.463 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:36:40.444466+0000", 2024-02-09T17:36:40.463 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00016969700000000001 2024-02-09T17:36:40.463 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:36:40.463 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:36:40.463 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 2,3", 2024-02-09T17:36:40.463 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:36:40.444631+0000", 2024-02-09T17:36:40.463 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000165052 2024-02-09T17:36:40.463 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:36:40.463 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:36:40.463 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:36:40.463 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:36:40.447620+0000", 2024-02-09T17:36:40.464 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.002989528 2024-02-09T17:36:40.464 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:36:40.464 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:36:40.464 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:36:40.464 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:36:40.464 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T17:36:40.464 INFO:teuthology.orchestra.run.smithi099.stdout: "num_ops": 3 2024-02-09T17:36:40.464 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:36:40.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:40.465 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:36:40.465 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:36:40.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:36:40.567 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:36:40.628 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:40.688 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:36:40.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:40.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:36:40.716 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 61 16 108505 108489 111.142 64.75 0.00748851 0.00899333 2024-02-09T17:36:40.794 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:36:40.839 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:40.875 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:36:40.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:40.881 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:36:40.981 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:36:41.023 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:41.047 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:36:41.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:41.055 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:36:41.156 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:36:41.209 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:41.717 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 62 16 109673 109657 110.526 73 0.0145431 0.00904302 2024-02-09T17:36:41.878 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:36:42.067 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:41 smithi099 ceph-mon[39420]: pgmap v945: 37 pgs: 37 active+clean; 6.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-02-09T17:36:42.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:41 smithi179 ceph-mon[43975]: pgmap v945: 37 pgs: 37 active+clean; 6.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-02-09T17:36:42.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:41 smithi162 ceph-mon[44304]: pgmap v945: 37 pgs: 37 active+clean; 6.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-02-09T17:36:42.717 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 63 16 110831 110815 109.921 72.375 0.00748785 0.00909372 2024-02-09T17:36:42.756 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:42 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:36:42.756 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:42 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:36:42.756 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:42 smithi179 ceph-mon[43975]: osdmap e348: 12 total, 12 up, 12 in 2024-02-09T17:36:42.834 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:36:42.840 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:42.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:36:42.941 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:36:42.983 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:43.025 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-09T17:36:43.026 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:42 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:36:43.026 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:42 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:36:43.026 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:42 smithi099 ceph-mon[39420]: osdmap e348: 12 total, 12 up, 12 in 2024-02-09T17:36:43.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:42 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:36:43.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:42 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:36:43.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:42 smithi162 ceph-mon[44304]: osdmap e348: 12 total, 12 up, 12 in 2024-02-09T17:36:43.215 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:36:43.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:43.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:36:43.323 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:36:43.355 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:43.382 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:36:43.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:43.387 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:36:43.488 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:36:43.526 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:43.554 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:36:43.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:43.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:36:43.659 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:36:43.697 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:43.718 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 64 16 111943 111927 109.287 69.5 0.0216551 0.00914592 2024-02-09T17:36:43.728 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:36:43.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:43.734 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:36:43.835 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:36:43.909 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:43.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:43 smithi162 ceph-mon[44304]: pgmap v947: 37 pgs: 37 active+clean; 6.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.42k op/s 2024-02-09T17:36:43.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:43 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:36:43.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:43 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:36:43.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:43 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:36:43.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:43 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:36:44.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:43 smithi179 ceph-mon[43975]: pgmap v947: 37 pgs: 37 active+clean; 6.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.42k op/s 2024-02-09T17:36:44.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:43 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:36:44.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:43 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:36:44.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:43 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:36:44.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:43 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:36:44.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:43 smithi099 ceph-mon[39420]: pgmap v947: 37 pgs: 37 active+clean; 6.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.42k op/s 2024-02-09T17:36:44.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:43 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:36:44.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:43 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:36:44.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:43 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:36:44.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:43 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:36:44.507 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:36:44.514 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:36:44.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:44.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:36:44.621 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:36:44.653 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:44.676 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:36:44.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:44.684 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:36:44.718 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 65 16 113051 113035 108.671 69.25 0.0120341 0.00919796 2024-02-09T17:36:44.784 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:36:44.819 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:44.841 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:36:44.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:44.849 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:36:44.950 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:36:44.984 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:45.020 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:36:45.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:45.027 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:36:45.128 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:36:45.168 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:45.193 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:36:45.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:45.196 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:36:45.297 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:36:45.334 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:45.367 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:36:45.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:45.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:36:45.473 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:36:45.501 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:45.532 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:36:45.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:45.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:36:45.640 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:36:45.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:45 smithi179 ceph-mon[43975]: osdmap e349: 12 total, 12 up, 12 in 2024-02-09T17:36:45.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:45 smithi179 ceph-mon[43975]: pgmap v949: 37 pgs: 37 active+clean; 6.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s wr, 800 op/s 2024-02-09T17:36:45.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:45 smithi162 ceph-mon[44304]: osdmap e349: 12 total, 12 up, 12 in 2024-02-09T17:36:45.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:45 smithi162 ceph-mon[44304]: pgmap v949: 37 pgs: 37 active+clean; 6.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s wr, 800 op/s 2024-02-09T17:36:45.684 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:45 smithi099 ceph-mon[39420]: osdmap e349: 12 total, 12 up, 12 in 2024-02-09T17:36:45.684 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:45 smithi099 ceph-mon[39420]: pgmap v949: 37 pgs: 37 active+clean; 6.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s wr, 800 op/s 2024-02-09T17:36:45.690 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:45.718 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 66 16 114135 114119 108.051 67.75 0.0269915 0.00925026 2024-02-09T17:36:46.402 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is set 2024-02-09T17:36:46.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:46 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2170257029' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:36:46.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:46 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:36:46.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:46 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2170257029' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:36:46.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:46 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:36:46.718 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 67 16 115315 115299 107.539 73.75 0.0121619 0.00929531 2024-02-09T17:36:46.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:46 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2170257029' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:36:46.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:46 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:36:47.164 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:36:47.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:47.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:36:47.272 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:36:47.316 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:47.340 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:36:47.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:47.345 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:36:47.381 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:47 smithi162 ceph-mon[44304]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:36:47.381 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:47 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:36:47.381 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:47 smithi162 ceph-mon[44304]: osdmap e350: 12 total, 12 up, 12 in 2024-02-09T17:36:47.381 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:47 smithi162 ceph-mon[44304]: pgmap v951: 37 pgs: 37 active+clean; 6.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-02-09T17:36:47.445 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:36:47.476 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:47.499 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:36:47.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:47.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:36:47.603 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:36:47.648 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:47 smithi099 ceph-mon[39420]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:36:47.648 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:47 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:36:47.648 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:47 smithi099 ceph-mon[39420]: osdmap e350: 12 total, 12 up, 12 in 2024-02-09T17:36:47.648 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:47 smithi099 ceph-mon[39420]: pgmap v951: 37 pgs: 37 active+clean; 6.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-02-09T17:36:47.648 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:47 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:36:47.656 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:47.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:47 smithi179 ceph-mon[43975]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:36:47.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:47 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:36:47.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:47 smithi179 ceph-mon[43975]: osdmap e350: 12 total, 12 up, 12 in 2024-02-09T17:36:47.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:47 smithi179 ceph-mon[43975]: pgmap v951: 37 pgs: 37 active+clean; 6.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-02-09T17:36:47.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:47 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:36:47.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:47 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:36:47.718 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 68 16 116336 116320 106.896 63.8125 0.00821989 0.00935036 2024-02-09T17:36:47.820 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:36:47.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:47.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:36:47.930 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:36:47.998 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:48.024 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:36:48.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:48.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:36:48.128 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:36:48.167 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:48.194 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:36:48.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:48.197 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:36:48.298 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:36:48.340 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:48.383 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:36:48.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:48.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:36:48.489 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:36:48.565 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:48.681 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:36:48.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:48.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:36:48.718 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 69 16 117427 117411 106.335 68.1875 0.005631 0.00939978 2024-02-09T17:36:48.792 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:36:48.828 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:48.850 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:36:48.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:48.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:36:48.856 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:48 smithi162 ceph-mon[44304]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:36:48.856 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:48 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:36:48.856 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:48 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:36:48.856 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:48 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:36:48.856 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:48 smithi162 ceph-mon[44304]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:36:48.857 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:48 smithi162 ceph-mon[44304]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:36:48.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:48 smithi179 ceph-mon[43975]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:36:48.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:48 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:36:48.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:48 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:36:48.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:48 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:36:48.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:48 smithi179 ceph-mon[43975]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:36:48.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:48 smithi179 ceph-mon[43975]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:36:48.957 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:36:49.001 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:48 smithi099 ceph-mon[39420]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:36:49.002 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:48 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:36:49.002 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:48 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:36:49.002 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:48 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:36:49.002 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:48 smithi099 ceph-mon[39420]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:36:49.002 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:48 smithi099 ceph-mon[39420]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:36:49.023 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:49.383 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:36:49.643 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:49 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:36:49.644 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:49 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:36:49.644 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:49 smithi099 ceph-mon[39420]: osdmap e351: 12 total, 12 up, 12 in 2024-02-09T17:36:49.644 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:49 smithi099 ceph-mon[39420]: pgmap v953: 37 pgs: 37 active+clean; 7.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 980 op/s 2024-02-09T17:36:49.644 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:49 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:36:49.644 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:49 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:36:49.644 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:49 smithi099 ceph-mon[39420]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:36:49.644 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:49 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3379982769' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-09T17:36:49.644 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:49 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-09T17:36:49.644 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:49 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:36:49.644 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:49 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:36:49.644 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:49 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:36:49.718 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 70 16 118385 118369 105.671 59.875 0.00838217 0.00945956 2024-02-09T17:36:49.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:49 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:36:49.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:49 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:36:49.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:49 smithi179 ceph-mon[43975]: osdmap e351: 12 total, 12 up, 12 in 2024-02-09T17:36:49.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:49 smithi179 ceph-mon[43975]: pgmap v953: 37 pgs: 37 active+clean; 7.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 980 op/s 2024-02-09T17:36:49.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:49 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:36:49.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:49 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:36:49.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:49 smithi179 ceph-mon[43975]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:36:49.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:49 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3379982769' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-09T17:36:49.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:49 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-09T17:36:49.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:49 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:36:49.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:49 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:36:49.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:49 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:36:49.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:49 smithi179 ceph-mon[43975]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:36:49.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:49 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:36:49.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:49 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:36:49.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:49 smithi162 ceph-mon[44304]: osdmap e351: 12 total, 12 up, 12 in 2024-02-09T17:36:49.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:49 smithi162 ceph-mon[44304]: pgmap v953: 37 pgs: 37 active+clean; 7.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 980 op/s 2024-02-09T17:36:49.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:49 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:36:49.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:49 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:36:49.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:49 smithi162 ceph-mon[44304]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:36:49.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:49 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3379982769' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-09T17:36:49.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:49 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-09T17:36:49.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:49 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:36:49.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:49 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:36:49.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:49 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:36:49.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:49 smithi162 ceph-mon[44304]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:36:49.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:49 smithi099 ceph-mon[39420]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:36:50.019 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:36:50.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:50.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:36:50.045 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.0 to 1 (10000) 2024-02-09T17:36:50.127 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:36:50.188 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:50.487 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:36:50.526 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:36:50.553 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:36:50.593 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:36:50.626 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:36:50.667 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:36:50.701 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:36:50.718 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 71 16 119443 119427 105.114 66.125 0.008475 0.00950847 2024-02-09T17:36:50.740 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:36:50.781 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:36:50.810 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:36:50.853 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:36:50.905 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:36:50.932 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:36:50.932 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:36:50.932 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:36:50.932 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:36:50.932 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:36:50.932 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:36:50.932 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:36:50.932 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:36:50.933 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:36:50.933 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:36:50.933 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:36:50.933 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:36:51.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:51 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-09T17:36:51.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:51 smithi099 ceph-mon[39420]: osdmap e352: 12 total, 12 up, 12 in 2024-02-09T17:36:51.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:51 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2491421536' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:36:51.289 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:36:51.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:51.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:36:51.400 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:36:51.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:51 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-09T17:36:51.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:51 smithi179 ceph-mon[43975]: osdmap e352: 12 total, 12 up, 12 in 2024-02-09T17:36:51.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:51 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2491421536' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:36:51.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:51 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-09T17:36:51.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:51 smithi162 ceph-mon[44304]: osdmap e352: 12 total, 12 up, 12 in 2024-02-09T17:36:51.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:51 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2491421536' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:36:51.437 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:51.458 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:36:51.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:51.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:36:51.562 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:36:51.597 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:51.718 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 72 16 120293 120277 104.392 53.125 0.0246452 0.00957519 2024-02-09T17:36:51.873 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:36:51.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:51.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:36:51.983 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:36:52.009 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:52.028 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:36:52.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:52.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:36:52.132 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:36:52.167 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:52.196 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:36:52.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:52.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:36:52.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:52 smithi099 ceph-mon[39420]: pgmap v955: 37 pgs: 37 active+clean; 7.2 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.48k op/s 2024-02-09T17:36:52.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:52 smithi099 ceph-mon[39420]: osdmap e353: 12 total, 12 up, 12 in 2024-02-09T17:36:52.306 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:36:52.361 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:52.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:52 smithi179 ceph-mon[43975]: pgmap v955: 37 pgs: 37 active+clean; 7.2 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.48k op/s 2024-02-09T17:36:52.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:52 smithi179 ceph-mon[43975]: osdmap e353: 12 total, 12 up, 12 in 2024-02-09T17:36:52.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:52 smithi162 ceph-mon[44304]: pgmap v955: 37 pgs: 37 active+clean; 7.2 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.48k op/s 2024-02-09T17:36:52.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:52 smithi162 ceph-mon[44304]: osdmap e353: 12 total, 12 up, 12 in 2024-02-09T17:36:52.456 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:36:52.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:52.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:36:52.563 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:36:52.598 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:52.620 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:36:52.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:52.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:36:52.718 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 73 16 121268 121252 103.797 60.9375 0.00728302 0.00963019 2024-02-09T17:36:52.724 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:36:52.777 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:52.808 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:36:52.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:52.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:36:52.914 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:36:52.945 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:52.974 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:36:52.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:52.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:36:53.082 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:36:53.137 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:53.298 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:36:53.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:53.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:36:53.415 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:36:53.453 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:53.634 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:36:53.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:53.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:36:53.675 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-02-09T17:36:53.676 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph pg dump --format=json 2024-02-09T17:36:53.718 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 74 16 122346 122330 103.305 67.375 0.024378 0.00967633 2024-02-09T17:36:53.740 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:36:53.780 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:53.810 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:36:53.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:53.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:36:53.914 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:36:53.917 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset noscrub 2024-02-09T17:36:53.944 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:53.966 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:36:53.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:53.969 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:36:54.037 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:54 smithi162 ceph-mon[44304]: pgmap v957: 37 pgs: 37 active+clean; 7.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 935 op/s 2024-02-09T17:36:54.069 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:36:54.106 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:54.136 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:36:54.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:54.144 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:36:54.245 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:36:54.274 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:54.301 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:36:54.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:54.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:36:54.408 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:36:54.416 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:36:54.416 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:36:54.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:54 smithi179 ceph-mon[43975]: pgmap v957: 37 pgs: 37 active+clean; 7.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 935 op/s 2024-02-09T17:36:54.423 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:36:54.441 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:54.457 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:36:54.459 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:54 smithi099 ceph-mon[39420]: pgmap v957: 37 pgs: 37 active+clean; 7.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 935 op/s 2024-02-09T17:36:54.462 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:36:54.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:36:54.465 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:36:54.491 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:36:54.566 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:36:54.631 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:36:54.719 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 75 16 123245 123229 102.676 56.1875 0.0189296 0.00973454 2024-02-09T17:36:55.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:55 smithi179 ceph-mon[43975]: pgmap v958: 37 pgs: 37 active+clean; 7.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 876 op/s 2024-02-09T17:36:55.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:55 smithi162 ceph-mon[44304]: pgmap v958: 37 pgs: 37 active+clean; 7.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 876 op/s 2024-02-09T17:36:55.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:55 smithi099 ceph-mon[39420]: pgmap v958: 37 pgs: 37 active+clean; 7.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 876 op/s 2024-02-09T17:36:55.719 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 76 16 124172 124156 102.088 57.9375 0.0264639 0.00979095 2024-02-09T17:36:56.719 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 77 16 125249 125233 101.636 67.3125 0.0135087 0.0098351 2024-02-09T17:36:57.264 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-09T17:36:57.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:57 smithi099 ceph-mon[39420]: pgmap v959: 37 pgs: 37 active+clean; 7.5 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-02-09T17:36:57.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:57 smithi179 ceph-mon[43975]: pgmap v959: 37 pgs: 37 active+clean; 7.5 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-02-09T17:36:57.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:57 smithi162 ceph-mon[44304]: pgmap v959: 37 pgs: 37 active+clean; 7.5 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-02-09T17:36:57.719 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 78 16 126272 126256 101.153 63.9375 0.0191575 0.00988148 2024-02-09T17:36:58.722 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 79 16 127361 127345 100.73 68.0625 0.0138171 0.00992376 2024-02-09T17:36:59.722 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:2024-02-09T17:36:59.722669+0000 min lat: 0.00117111 max lat: 0.234475 avg lat: 0.00996319 2024-02-09T17:36:59.723 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-09T17:36:59.723 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 80 16 128466 128450 100.334 69.0625 0.0186427 0.00996319 2024-02-09T17:36:59.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:36:59 smithi099 ceph-mon[39420]: pgmap v960: 37 pgs: 37 active+clean; 7.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-02-09T17:37:00.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:36:59 smithi179 ceph-mon[43975]: pgmap v960: 37 pgs: 37 active+clean; 7.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-02-09T17:37:00.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:36:59 smithi162 ceph-mon[44304]: pgmap v960: 37 pgs: 37 active+clean; 7.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-02-09T17:37:00.722 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 81 16 129460 129444 99.862 62.125 0.0129316 0.0100085 2024-02-09T17:37:00.793 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:37:00.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:00.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:37:00.875 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:37:00.888 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-02-09T17:37:00.898 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:37:00.969 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:01.726 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 82 16 130399 130383 99.3559 58.6875 0.0054204 0.010061 2024-02-09T17:37:01.799 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is unset 2024-02-09T17:37:01.998 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:01 smithi099 ceph-mon[39420]: pgmap v961: 37 pgs: 37 active+clean; 7.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 998 op/s 2024-02-09T17:37:01.998 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:01 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:37:01.998 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:01 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:37:01.998 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:01 smithi099 ceph-mon[39420]: osdmap e354: 12 total, 12 up, 12 in 2024-02-09T17:37:01.998 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:01 smithi099 ceph-mon[39420]: from='client.25270 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:37:01.999 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:01 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1328096594' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:37:01.999 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:01 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:37:02.063 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:37:02.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:02.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:37:02.169 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:37:02.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:01 smithi179 ceph-mon[43975]: pgmap v961: 37 pgs: 37 active+clean; 7.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 998 op/s 2024-02-09T17:37:02.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:01 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:37:02.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:01 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:37:02.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:01 smithi179 ceph-mon[43975]: osdmap e354: 12 total, 12 up, 12 in 2024-02-09T17:37:02.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:01 smithi179 ceph-mon[43975]: from='client.25270 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:37:02.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:01 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1328096594' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:37:02.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:01 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:37:02.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:01 smithi162 ceph-mon[44304]: pgmap v961: 37 pgs: 37 active+clean; 7.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 998 op/s 2024-02-09T17:37:02.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:01 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:37:02.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:01 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:37:02.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:01 smithi162 ceph-mon[44304]: osdmap e354: 12 total, 12 up, 12 in 2024-02-09T17:37:02.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:01 smithi162 ceph-mon[44304]: from='client.25270 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:37:02.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:01 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1328096594' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:37:02.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:01 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:37:02.234 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:02.726 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 83 16 131409 131393 98.9193 63.125 0.00527271 0.0101045 2024-02-09T17:37:02.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:02 smithi099 ceph-mon[39420]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:37:02.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:02 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:37:02.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:02 smithi099 ceph-mon[39420]: osdmap e355: 12 total, 12 up, 12 in 2024-02-09T17:37:03.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:02 smithi179 ceph-mon[43975]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:37:03.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:02 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:37:03.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:02 smithi179 ceph-mon[43975]: osdmap e355: 12 total, 12 up, 12 in 2024-02-09T17:37:03.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:02 smithi162 ceph-mon[44304]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:37:03.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:02 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:37:03.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:02 smithi162 ceph-mon[44304]: osdmap e355: 12 total, 12 up, 12 in 2024-02-09T17:37:03.219 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:37:03.224 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:37:03.268 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:37:03.269 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":962,"stamp":"2024-02-09T17:37:00.724557+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8360952352,"num_objects":127571,"num_object_clones":0,"num_object_copies":382713,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":127571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":127569,"num_write_kb":8164416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130488,"ondisk_log_size":130488,"up":111,"acting":111,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":113,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":27582684,"kb_used_data":24377528,"kb_used_omap":229,"kb_used_meta":3204826,"kb_avail":1097359140,"statfs":{"total":1151940427776,"available":1123695759360,"internally_reserved":0,"allocated":24962588672,"data_stored":24954244808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235127,"internal_metadata":3281742217},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":613220352,"num_objects":9357,"num_object_clones":0,"num_object_copies":28071,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9357,"num_write_kb":598848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.387011"},"pg_stats":[{"pgid":"1.14","version":"63'139","reported_seq":932,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-02-09T17:36:51.341990+0000","last_change":"2024-02-09T17:35:19.250490+0000","last_active":"2024-02-09T17:36:51.341990+0000","last_peered":"2024-02-09T17:36:51.341990+0000","last_clean":"2024-02-09T17:36:51.341990+0000","last_became_active":"2024-02-09T17:33:05.309643+0000","last_became_peered":"2024-02-09T17:33:05.309643+0000","last_unstale":"2024-02-09T17:36:51.341990+0000","last_undegraded":"2024-02-09T17:36:51.341990+0000","last_fullsized":"2024-02-09T17:36:51.341990+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:35:19.250349+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:34:13.228622+0000","last_clean_scrub_stamp":"2024-02-09T17:35:19.250349+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"353'7711","reported_seq":7783,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-02-09T17:36:55.696088+0000","last_change":"2024-02-09T17:35:30.391405+0000","last_active":"2024-02-09T17:36:55.696088+0000","last_peered":"2024-02-09T17:36:55.696088+0000","last_clean":"2024-02-09T17:36:55.696088+0000","last_became_active":"2024-02-09T17:35:30.390786+0000","last_became_peered":"2024-02-09T17:35:30.390786+0000","last_unstale":"2024-02-09T17:36:55.696088+0000","last_undegraded":"2024-02-09T17:36:55.696088+0000","last_fullsized":"2024-02-09T17:36:55.696088+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:35:28.884607+0000","objects_scrubbed":0,"log_size":7711,"log_dups_size":0,"ondisk_log_size":7711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":505348096,"num_objects":7711,"num_object_clones":0,"num_object_copies":23133,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7711,"num_write_kb":493504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,11],"acting":[6,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"63'139","reported_seq":962,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-02-09T17:36:51.342137+0000","last_change":"2024-02-09T17:35:23.087780+0000","last_active":"2024-02-09T17:36:51.342137+0000","last_peered":"2024-02-09T17:36:51.342137+0000","last_clean":"2024-02-09T17:36:51.342137+0000","last_became_active":"2024-02-09T17:33:00.284421+0000","last_became_peered":"2024-02-09T17:33:00.284421+0000","last_unstale":"2024-02-09T17:36:51.342137+0000","last_undegraded":"2024-02-09T17:36:51.342137+0000","last_fullsized":"2024-02-09T17:36:51.342137+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:35:23.087651+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:35:23.087651+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:36:53.394855+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"353'8133","reported_seq":8205,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-02-09T17:37:00.243613+0000","last_change":"2024-02-09T17:35:30.390671+0000","last_active":"2024-02-09T17:37:00.243613+0000","last_peered":"2024-02-09T17:37:00.243613+0000","last_clean":"2024-02-09T17:37:00.243613+0000","last_became_active":"2024-02-09T17:35:30.390424+0000","last_became_peered":"2024-02-09T17:35:30.390424+0000","last_unstale":"2024-02-09T17:37:00.243613+0000","last_undegraded":"2024-02-09T17:37:00.243613+0000","last_fullsized":"2024-02-09T17:37:00.243613+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:35:28.884607+0000","objects_scrubbed":0,"log_size":8133,"log_dups_size":0,"ondisk_log_size":8133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:37:04.465140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":533004288,"num_objects":8133,"num_object_clones":0,"num_object_copies":24399,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8133,"num_write_kb":520512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,0],"acting":[1,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"63'139","reported_seq":979,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-02-09T17:36:51.341796+0000","last_change":"2024-02-09T17:35:29.115817+0000","last_active":"2024-02-09T17:36:51.341796+0000","last_peered":"2024-02-09T17:36:51.341796+0000","last_clean":"2024-02-09T17:36:51.341796+0000","last_became_active":"2024-02-09T17:34:01.410594+0000","last_became_peered":"2024-02-09T17:34:01.410594+0000","last_unstale":"2024-02-09T17:36:51.341796+0000","last_undegraded":"2024-02-09T17:36:51.341796+0000","last_fullsized":"2024-02-09T17:36:51.341796+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:35:29.115704+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:35:29.115704+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:36:52.409578+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"353'7830","reported_seq":7902,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-02-09T17:36:56.381575+0000","last_change":"2024-02-09T17:35:30.166964+0000","last_active":"2024-02-09T17:36:56.381575+0000","last_peered":"2024-02-09T17:36:56.381575+0000","last_clean":"2024-02-09T17:36:56.381575+0000","last_became_active":"2024-02-09T17:35:30.166669+0000","last_became_peered":"2024-02-09T17:35:30.166669+0000","last_unstale":"2024-02-09T17:36:56.381575+0000","last_undegraded":"2024-02-09T17:36:56.381575+0000","last_fullsized":"2024-02-09T17:36:56.381575+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:35:28.884607+0000","objects_scrubbed":0,"log_size":7830,"log_dups_size":0,"ondisk_log_size":7830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:37:00.083037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":513146880,"num_objects":7830,"num_object_clones":0,"num_object_copies":23490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7830,"num_write_kb":501120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,2],"acting":[9,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.3","version":"63'139","reported_seq":476,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-02-09T17:36:51.339383+0000","last_change":"2024-02-09T17:35:22.061109+0000","last_active":"2024-02-09T17:36:51.339383+0000","last_peered":"2024-02-09T17:36:51.339383+0000","last_clean":"2024-02-09T17:36:51.339383+0000","last_became_active":"2024-02-09T17:33:00.279369+0000","last_became_peered":"2024-02-09T17:33:00.279369+0000","last_unstale":"2024-02-09T17:36:51.339383+0000","last_undegraded":"2024-02-09T17:36:51.339383+0000","last_fullsized":"2024-02-09T17:36:51.339383+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:35:22.060998+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:35:22.060998+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:36:52.329226+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"353'8028","reported_seq":8098,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-02-09T17:37:00.273962+0000","last_change":"2024-02-09T17:35:30.166581+0000","last_active":"2024-02-09T17:37:00.273962+0000","last_peered":"2024-02-09T17:37:00.273962+0000","last_clean":"2024-02-09T17:37:00.273962+0000","last_became_active":"2024-02-09T17:35:30.166273+0000","last_became_peered":"2024-02-09T17:35:30.166273+0000","last_unstale":"2024-02-09T17:37:00.273962+0000","last_undegraded":"2024-02-09T17:37:00.273962+0000","last_fullsized":"2024-02-09T17:37:00.273962+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:35:28.884607+0000","objects_scrubbed":0,"log_size":8028,"log_dups_size":0,"ondisk_log_size":8028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:37:04.465178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526123008,"num_objects":8028,"num_object_clones":0,"num_object_copies":24084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8028,"num_write_kb":513792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"63'139","reported_seq":487,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-02-09T17:36:51.364526+0000","last_change":"2024-02-09T17:35:15.862358+0000","last_active":"2024-02-09T17:36:51.364526+0000","last_peered":"2024-02-09T17:36:51.364526+0000","last_clean":"2024-02-09T17:36:51.364526+0000","last_became_active":"2024-02-09T17:34:01.342607+0000","last_became_peered":"2024-02-09T17:34:01.342607+0000","last_unstale":"2024-02-09T17:36:51.364526+0000","last_undegraded":"2024-02-09T17:36:51.364526+0000","last_fullsized":"2024-02-09T17:36:51.364526+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:35:15.862253+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:35:15.862253+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:36:51.662201+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.3","version":"353'8027","reported_seq":8099,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-02-09T17:37:00.122264+0000","last_change":"2024-02-09T17:35:30.134316+0000","last_active":"2024-02-09T17:37:00.122264+0000","last_peered":"2024-02-09T17:37:00.122264+0000","last_clean":"2024-02-09T17:37:00.122264+0000","last_became_active":"2024-02-09T17:35:30.134014+0000","last_became_peered":"2024-02-09T17:35:30.134014+0000","last_unstale":"2024-02-09T17:37:00.122264+0000","last_undegraded":"2024-02-09T17:37:00.122264+0000","last_fullsized":"2024-02-09T17:37:00.122264+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:35:28.884607+0000","objects_scrubbed":0,"log_size":8027,"log_dups_size":0,"ondisk_log_size":8027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:37:00.275439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526057472,"num_objects":8027,"num_object_clones":0,"num_object_copies":24081,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8027,"num_write_kb":513728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,1],"acting":[11,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"63'139","reported_seq":478,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-02-09T17:36:51.364682+0000","last_change":"2024-02-09T17:35:14.866242+0000","last_active":"2024-02-09T17:36:51.364682+0000","last_peered":"2024-02-09T17:36:51.364682+0000","last_clean":"2024-02-09T17:36:51.364682+0000","last_became_active":"2024-02-09T17:33:00.284702+0000","last_became_peered":"2024-02-09T17:33:00.284702+0000","last_unstale":"2024-02-09T17:36:51.364682+0000","last_undegraded":"2024-02-09T17:36:51.364682+0000","last_fullsized":"2024-02-09T17:36:51.364682+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:35:14.866130+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:35:14.866130+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2","version":"353'7771","reported_seq":7842,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-02-09T17:36:56.392291+0000","last_change":"2024-02-09T17:35:29.900248+0000","last_active":"2024-02-09T17:36:56.392291+0000","last_peered":"2024-02-09T17:36:56.392291+0000","last_clean":"2024-02-09T17:36:56.392291+0000","last_became_active":"2024-02-09T17:35:29.899914+0000","last_became_peered":"2024-02-09T17:35:29.899914+0000","last_unstale":"2024-02-09T17:36:56.392291+0000","last_undegraded":"2024-02-09T17:36:56.392291+0000","last_fullsized":"2024-02-09T17:36:56.392291+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:35:28.884607+0000","objects_scrubbed":0,"log_size":7771,"log_dups_size":0,"ondisk_log_size":7771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:37:00.087247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":509280256,"num_objects":7771,"num_object_clones":0,"num_object_copies":23313,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7771,"num_write_kb":497344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.7","version":"63'139","reported_seq":491,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-02-09T17:36:51.344630+0000","last_change":"2024-02-09T17:35:09.011634+0000","last_active":"2024-02-09T17:36:51.344630+0000","last_peered":"2024-02-09T17:36:51.344630+0000","last_clean":"2024-02-09T17:36:51.344630+0000","last_became_active":"2024-02-09T17:33:00.278966+0000","last_became_peered":"2024-02-09T17:33:00.278966+0000","last_unstale":"2024-02-09T17:36:51.344630+0000","last_undegraded":"2024-02-09T17:36:51.344630+0000","last_fullsized":"2024-02-09T17:36:51.344630+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:35:09.011504+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:35:09.011504+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:36:59.477488+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"353'8030","reported_seq":8107,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-02-09T17:37:00.122993+0000","last_change":"2024-02-09T17:36:31.371028+0000","last_active":"2024-02-09T17:37:00.122993+0000","last_peered":"2024-02-09T17:37:00.122993+0000","last_clean":"2024-02-09T17:37:00.122993+0000","last_became_active":"2024-02-09T17:35:29.900032+0000","last_became_peered":"2024-02-09T17:35:29.900032+0000","last_unstale":"2024-02-09T17:37:00.122993+0000","last_undegraded":"2024-02-09T17:37:00.122993+0000","last_fullsized":"2024-02-09T17:37:00.122993+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:35:28.884607+0000","objects_scrubbed":199,"log_size":8030,"log_dups_size":0,"ondisk_log_size":8030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:37:00.275369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526254080,"num_objects":8030,"num_object_clones":0,"num_object_copies":24090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8030,"num_write_kb":513920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"63'139","reported_seq":569,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-02-09T17:36:51.340482+0000","last_change":"2024-02-09T17:35:19.886268+0000","last_active":"2024-02-09T17:36:51.340482+0000","last_peered":"2024-02-09T17:36:51.340482+0000","last_clean":"2024-02-09T17:36:51.340482+0000","last_became_active":"2024-02-09T17:33:00.284924+0000","last_became_peered":"2024-02-09T17:33:00.284924+0000","last_unstale":"2024-02-09T17:36:51.340482+0000","last_undegraded":"2024-02-09T17:36:51.340482+0000","last_fullsized":"2024-02-09T17:36:51.340482+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:35:19.886117+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:35:19.886117+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:36:53.208807+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.0","version":"353'7862","reported_seq":7934,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-02-09T17:36:56.857153+0000","last_change":"2024-02-09T17:35:30.134707+0000","last_active":"2024-02-09T17:36:56.857153+0000","last_peered":"2024-02-09T17:36:56.857153+0000","last_clean":"2024-02-09T17:36:56.857153+0000","last_became_active":"2024-02-09T17:35:30.134347+0000","last_became_peered":"2024-02-09T17:35:30.134347+0000","last_unstale":"2024-02-09T17:36:56.857153+0000","last_undegraded":"2024-02-09T17:36:56.857153+0000","last_fullsized":"2024-02-09T17:36:56.857153+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:35:28.884607+0000","objects_scrubbed":0,"log_size":7862,"log_dups_size":0,"ondisk_log_size":7862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:37:01.016848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":515244032,"num_objects":7862,"num_object_clones":0,"num_object_copies":23586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7862,"num_write_kb":503168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"63'139","reported_seq":478,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-02-09T17:36:51.364571+0000","last_change":"2024-02-09T17:35:13.847319+0000","last_active":"2024-02-09T17:36:51.364571+0000","last_peered":"2024-02-09T17:36:51.364571+0000","last_clean":"2024-02-09T17:36:51.364571+0000","last_became_active":"2024-02-09T17:33:00.274636+0000","last_became_peered":"2024-02-09T17:33:00.274636+0000","last_unstale":"2024-02-09T17:36:51.364571+0000","last_undegraded":"2024-02-09T17:36:51.364571+0000","last_fullsized":"2024-02-09T17:36:51.364571+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:35:13.847223+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:35:13.847223+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:36:51.662163+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.f","version":"353'7935","reported_seq":8007,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-02-09T17:37:00.251879+0000","last_change":"2024-02-09T17:35:29.905704+0000","last_active":"2024-02-09T17:37:00.251879+0000","last_peered":"2024-02-09T17:37:00.251879+0000","last_clean":"2024-02-09T17:37:00.251879+0000","last_became_active":"2024-02-09T17:35:29.905385+0000","last_became_peered":"2024-02-09T17:35:29.905385+0000","last_unstale":"2024-02-09T17:37:00.251879+0000","last_undegraded":"2024-02-09T17:37:00.251879+0000","last_fullsized":"2024-02-09T17:37:00.251879+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:35:28.884607+0000","objects_scrubbed":0,"log_size":7935,"log_dups_size":0,"ondisk_log_size":7935,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:37:04.465399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520028160,"num_objects":7935,"num_object_clones":0,"num_object_copies":23805,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7935,"num_write_kb":507840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,11],"acting":[1,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"63'139","reported_seq":480,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-02-09T17:36:51.344377+0000","last_change":"2024-02-09T17:36:06.172514+0000","last_active":"2024-02-09T17:36:51.344377+0000","last_peered":"2024-02-09T17:36:51.344377+0000","last_clean":"2024-02-09T17:36:51.344377+0000","last_became_active":"2024-02-09T17:32:35.418200+0000","last_became_peered":"2024-02-09T17:32:35.418200+0000","last_unstale":"2024-02-09T17:36:51.344377+0000","last_undegraded":"2024-02-09T17:36:51.344377+0000","last_fullsized":"2024-02-09T17:36:51.344377+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:36:06.172377+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:36:06.172377+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:37:35.221695+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.e","version":"353'8075","reported_seq":8147,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-02-09T17:37:00.508290+0000","last_change":"2024-02-09T17:35:30.391216+0000","last_active":"2024-02-09T17:37:00.508290+0000","last_peered":"2024-02-09T17:37:00.508290+0000","last_clean":"2024-02-09T17:37:00.508290+0000","last_became_active":"2024-02-09T17:35:30.390749+0000","last_became_peered":"2024-02-09T17:35:30.390749+0000","last_unstale":"2024-02-09T17:37:00.508290+0000","last_undegraded":"2024-02-09T17:37:00.508290+0000","last_fullsized":"2024-02-09T17:37:00.508290+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:35:28.884607+0000","objects_scrubbed":0,"log_size":8075,"log_dups_size":0,"ondisk_log_size":8075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":529203200,"num_objects":8075,"num_object_clones":0,"num_object_copies":24225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8075,"num_write_kb":516800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,6],"acting":[4,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"63'139","reported_seq":487,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-02-09T17:36:51.340651+0000","last_change":"2024-02-09T17:35:20.873649+0000","last_active":"2024-02-09T17:36:51.340651+0000","last_peered":"2024-02-09T17:36:51.340651+0000","last_clean":"2024-02-09T17:36:51.340651+0000","last_became_active":"2024-02-09T17:34:01.388313+0000","last_became_peered":"2024-02-09T17:34:01.388313+0000","last_unstale":"2024-02-09T17:36:51.340651+0000","last_undegraded":"2024-02-09T17:36:51.340651+0000","last_fullsized":"2024-02-09T17:36:51.340651+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:35:20.873540+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:35:20.873540+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:36:53.205184+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.d","version":"353'8056","reported_seq":8128,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-02-09T17:37:00.476135+0000","last_change":"2024-02-09T17:35:29.906111+0000","last_active":"2024-02-09T17:37:00.476135+0000","last_peered":"2024-02-09T17:37:00.476135+0000","last_clean":"2024-02-09T17:37:00.476135+0000","last_became_active":"2024-02-09T17:35:29.905824+0000","last_became_peered":"2024-02-09T17:35:29.905824+0000","last_unstale":"2024-02-09T17:37:00.476135+0000","last_undegraded":"2024-02-09T17:37:00.476135+0000","last_fullsized":"2024-02-09T17:37:00.476135+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:35:28.884607+0000","objects_scrubbed":0,"log_size":8056,"log_dups_size":0,"ondisk_log_size":8056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":527958016,"num_objects":8056,"num_object_clones":0,"num_object_copies":24168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8056,"num_write_kb":515584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,1],"acting":[4,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"63'139","reported_seq":489,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-02-09T17:36:51.338524+0000","last_change":"2024-02-09T17:36:20.412325+0000","last_active":"2024-02-09T17:36:51.338524+0000","last_peered":"2024-02-09T17:36:51.338524+0000","last_clean":"2024-02-09T17:36:51.338524+0000","last_became_active":"2024-02-09T17:32:35.384617+0000","last_became_peered":"2024-02-09T17:32:35.384617+0000","last_unstale":"2024-02-09T17:36:51.338524+0000","last_undegraded":"2024-02-09T17:36:51.338524+0000","last_fullsized":"2024-02-09T17:36:51.338524+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:36:20.412161+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:36:20.412161+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:37:20.924824+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"353'8247","reported_seq":8317,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-02-09T17:37:00.433987+0000","last_change":"2024-02-09T17:35:29.903975+0000","last_active":"2024-02-09T17:37:00.433987+0000","last_peered":"2024-02-09T17:37:00.433987+0000","last_clean":"2024-02-09T17:37:00.433987+0000","last_became_active":"2024-02-09T17:35:29.903553+0000","last_became_peered":"2024-02-09T17:35:29.903553+0000","last_unstale":"2024-02-09T17:37:00.433987+0000","last_undegraded":"2024-02-09T17:37:00.433987+0000","last_fullsized":"2024-02-09T17:37:00.433987+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:35:28.884607+0000","objects_scrubbed":0,"log_size":8247,"log_dups_size":0,"ondisk_log_size":8247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:37:00.989373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540475392,"num_objects":8247,"num_object_clones":0,"num_object_copies":24741,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8247,"num_write_kb":527808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"63'139","reported_seq":965,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-02-09T17:36:51.341765+0000","last_change":"2024-02-09T17:35:30.193355+0000","last_active":"2024-02-09T17:36:51.341765+0000","last_peered":"2024-02-09T17:36:51.341765+0000","last_clean":"2024-02-09T17:36:51.341765+0000","last_became_active":"2024-02-09T17:32:35.402025+0000","last_became_peered":"2024-02-09T17:32:35.402025+0000","last_unstale":"2024-02-09T17:36:51.341765+0000","last_undegraded":"2024-02-09T17:36:51.341765+0000","last_fullsized":"2024-02-09T17:36:51.341765+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:35:30.193194+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:30.193194+0000","last_clean_scrub_stamp":"2024-02-09T17:35:30.193194+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:36:52.681919+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"353'8079","reported_seq":8151,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-02-09T17:36:58.572374+0000","last_change":"2024-02-09T17:35:29.907065+0000","last_active":"2024-02-09T17:36:58.572374+0000","last_peered":"2024-02-09T17:36:58.572374+0000","last_clean":"2024-02-09T17:36:58.572374+0000","last_became_active":"2024-02-09T17:35:29.906768+0000","last_became_peered":"2024-02-09T17:35:29.906768+0000","last_unstale":"2024-02-09T17:36:58.572374+0000","last_undegraded":"2024-02-09T17:36:58.572374+0000","last_fullsized":"2024-02-09T17:36:58.572374+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:35:28.884607+0000","objects_scrubbed":0,"log_size":8079,"log_dups_size":0,"ondisk_log_size":8079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:37:01.567260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":529465344,"num_objects":8079,"num_object_clones":0,"num_object_copies":24237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8079,"num_write_kb":517056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"63'139","reported_seq":440,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-02-09T17:36:51.340754+0000","last_change":"2024-02-09T17:35:19.291657+0000","last_active":"2024-02-09T17:36:51.340754+0000","last_peered":"2024-02-09T17:36:51.340754+0000","last_clean":"2024-02-09T17:36:51.340754+0000","last_became_active":"2024-02-09T17:33:05.309824+0000","last_became_peered":"2024-02-09T17:33:05.309824+0000","last_unstale":"2024-02-09T17:36:51.340754+0000","last_undegraded":"2024-02-09T17:36:51.340754+0000","last_fullsized":"2024-02-09T17:36:51.340754+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:35:19.291548+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:35:19.291548+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"353'7648","reported_seq":7719,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-02-09T17:36:55.715432+0000","last_change":"2024-02-09T17:35:30.391286+0000","last_active":"2024-02-09T17:36:55.715432+0000","last_peered":"2024-02-09T17:36:55.715432+0000","last_clean":"2024-02-09T17:36:55.715432+0000","last_became_active":"2024-02-09T17:35:30.390843+0000","last_became_peered":"2024-02-09T17:35:30.390843+0000","last_unstale":"2024-02-09T17:36:55.715432+0000","last_undegraded":"2024-02-09T17:36:55.715432+0000","last_fullsized":"2024-02-09T17:36:55.715432+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:35:28.884607+0000","objects_scrubbed":0,"log_size":7648,"log_dups_size":0,"ondisk_log_size":7648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:36:59.477509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":501219328,"num_objects":7648,"num_object_clones":0,"num_object_copies":22944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7648,"num_write_kb":489472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.f","version":"63'139","reported_seq":507,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-02-09T17:36:51.344488+0000","last_change":"2024-02-09T17:36:22.364212+0000","last_active":"2024-02-09T17:36:51.344488+0000","last_peered":"2024-02-09T17:36:51.344488+0000","last_clean":"2024-02-09T17:36:51.344488+0000","last_became_active":"2024-02-09T17:33:05.316100+0000","last_became_peered":"2024-02-09T17:33:05.316100+0000","last_unstale":"2024-02-09T17:36:51.344488+0000","last_undegraded":"2024-02-09T17:36:51.344488+0000","last_fullsized":"2024-02-09T17:36:51.344488+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:36:22.364130+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:36:22.364130+0000","last_clean_scrub_stamp":"2024-02-09T17:36:22.364130+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:37:41.044157+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"353'8060","reported_seq":8131,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-02-09T17:37:00.122577+0000","last_change":"2024-02-09T17:35:29.906181+0000","last_active":"2024-02-09T17:37:00.122577+0000","last_peered":"2024-02-09T17:37:00.122577+0000","last_clean":"2024-02-09T17:37:00.122577+0000","last_became_active":"2024-02-09T17:35:29.905885+0000","last_became_peered":"2024-02-09T17:35:29.905885+0000","last_unstale":"2024-02-09T17:37:00.122577+0000","last_undegraded":"2024-02-09T17:37:00.122577+0000","last_fullsized":"2024-02-09T17:37:00.122577+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:35:28.884607+0000","objects_scrubbed":0,"log_size":8060,"log_dups_size":0,"ondisk_log_size":8060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:37:00.275534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528220160,"num_objects":8060,"num_object_clones":0,"num_object_copies":24180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8060,"num_write_kb":515840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,3],"acting":[11,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"63'139","reported_seq":936,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-02-09T17:36:51.354400+0000","last_change":"2024-02-09T17:35:23.912610+0000","last_active":"2024-02-09T17:36:51.354400+0000","last_peered":"2024-02-09T17:36:51.354400+0000","last_clean":"2024-02-09T17:36:51.354400+0000","last_became_active":"2024-02-09T17:35:20.840386+0000","last_became_peered":"2024-02-09T17:35:20.840386+0000","last_unstale":"2024-02-09T17:36:51.354400+0000","last_undegraded":"2024-02-09T17:36:51.354400+0000","last_fullsized":"2024-02-09T17:36:51.354400+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:35:23.912504+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:23.912504+0000","last_clean_scrub_stamp":"2024-02-09T17:35:23.912504+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:36:55.951274+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"353'8077","reported_seq":8149,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-02-09T17:37:00.063565+0000","last_change":"2024-02-09T17:35:30.134361+0000","last_active":"2024-02-09T17:37:00.063565+0000","last_peered":"2024-02-09T17:37:00.063565+0000","last_clean":"2024-02-09T17:37:00.063565+0000","last_became_active":"2024-02-09T17:35:30.134041+0000","last_became_peered":"2024-02-09T17:35:30.134041+0000","last_unstale":"2024-02-09T17:37:00.063565+0000","last_undegraded":"2024-02-09T17:37:00.063565+0000","last_fullsized":"2024-02-09T17:37:00.063565+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:35:28.884607+0000","objects_scrubbed":0,"log_size":8077,"log_dups_size":0,"ondisk_log_size":8077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:37:00.275472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":529334272,"num_objects":8077,"num_object_clones":0,"num_object_copies":24231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8077,"num_write_kb":516928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"63'139","reported_seq":448,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-02-09T17:36:51.342059+0000","last_change":"2024-02-09T17:35:18.023180+0000","last_active":"2024-02-09T17:36:51.342059+0000","last_peered":"2024-02-09T17:36:51.342059+0000","last_clean":"2024-02-09T17:36:51.342059+0000","last_became_active":"2024-02-09T17:33:05.319668+0000","last_became_peered":"2024-02-09T17:33:05.319668+0000","last_unstale":"2024-02-09T17:36:51.342059+0000","last_undegraded":"2024-02-09T17:36:51.342059+0000","last_fullsized":"2024-02-09T17:36:51.342059+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:35:18.023078+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:35:18.023078+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:36:53.394892+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"63'139","reported_seq":455,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-02-09T17:36:51.338395+0000","last_change":"2024-02-09T17:36:17.357034+0000","last_active":"2024-02-09T17:36:51.338395+0000","last_peered":"2024-02-09T17:36:51.338395+0000","last_clean":"2024-02-09T17:36:51.338395+0000","last_became_active":"2024-02-09T17:33:05.310104+0000","last_became_peered":"2024-02-09T17:33:05.310104+0000","last_unstale":"2024-02-09T17:36:51.338395+0000","last_undegraded":"2024-02-09T17:36:51.338395+0000","last_fullsized":"2024-02-09T17:36:51.338395+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:36:17.356909+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:36:17.356909+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:37:45.339428+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"63'139","reported_seq":463,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-02-09T17:36:51.658251+0000","last_change":"2024-02-09T17:36:18.779412+0000","last_active":"2024-02-09T17:36:51.658251+0000","last_peered":"2024-02-09T17:36:51.658251+0000","last_clean":"2024-02-09T17:36:51.658251+0000","last_became_active":"2024-02-09T17:34:01.353171+0000","last_became_peered":"2024-02-09T17:34:01.353171+0000","last_unstale":"2024-02-09T17:36:51.658251+0000","last_undegraded":"2024-02-09T17:36:51.658251+0000","last_fullsized":"2024-02-09T17:36:51.658251+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:36:18.779304+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:36:18.779304+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:37:42.122054+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.12","version":"63'139","reported_seq":929,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-02-09T17:36:51.339385+0000","last_change":"2024-02-09T17:35:17.274311+0000","last_active":"2024-02-09T17:36:51.339385+0000","last_peered":"2024-02-09T17:36:51.339385+0000","last_clean":"2024-02-09T17:36:51.339385+0000","last_became_active":"2024-02-09T17:33:05.308268+0000","last_became_peered":"2024-02-09T17:33:05.308268+0000","last_unstale":"2024-02-09T17:36:51.339385+0000","last_undegraded":"2024-02-09T17:36:51.339385+0000","last_fullsized":"2024-02-09T17:36:51.339385+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:35:17.274175+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:35:17.274175+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"63'139","reported_seq":491,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-02-09T17:36:51.339282+0000","last_change":"2024-02-09T17:36:27.497000+0000","last_active":"2024-02-09T17:36:51.339282+0000","last_peered":"2024-02-09T17:36:51.339282+0000","last_clean":"2024-02-09T17:36:51.339282+0000","last_became_active":"2024-02-09T17:33:05.309527+0000","last_became_peered":"2024-02-09T17:33:05.309527+0000","last_unstale":"2024-02-09T17:36:51.339282+0000","last_undegraded":"2024-02-09T17:36:51.339282+0000","last_fullsized":"2024-02-09T17:36:51.339282+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:36:27.496900+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:36:27.496900+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:37:56.755524+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":8360361984,"num_objects":127569,"num_object_clones":0,"num_object_copies":382707,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":127569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":127569,"num_write_kb":8164416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":25004605440,"data_stored":25004605440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127569,"ondisk_log_size":127569,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2919,"ondisk_log_size":2919,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056815,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3959184,"kb_used_data":3567048,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":89785968,"statfs":{"total":95995035648,"available":91940831232,"internally_reserved":0,"allocated":3652657152,"data_stored":3651962712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":401537476},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:36:14 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":1.2010000000000001,"15min":1.0409999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":1.9159999999999999,"5min":12.836,"15min":14.913},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.42299999999999999,"5min":1.2869999999999999,"15min":1.149},"min":{"1min":0.27200000000000002,"5min":0.25700000000000001,"15min":0.23000000000000001},"max":{"1min":0.91200000000000003,"5min":9.7639999999999993,"15min":14.869},"last":0.68500000000000005}]},{"osd":1,"last update":"Fri Feb 9 17:36:14 2024","interfaces":[{"interface":"back","average":{"1min":1.284,"5min":1.1080000000000001,"15min":1.3959999999999999},"min":{"1min":0.25700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":9.0079999999999991,"5min":9.0079999999999991,"15min":32.957999999999998},"last":0.34300000000000003},{"interface":"front","average":{"1min":1.304,"5min":1.2290000000000001,"15min":1.472},"min":{"1min":0.36699999999999999,"5min":0.34999999999999998,"15min":0.26700000000000002},"max":{"1min":9.0280000000000005,"5min":9.0280000000000005,"15min":32.880000000000003},"last":0.42299999999999999}]},{"osd":2,"last update":"Fri Feb 9 17:31:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.415,"15min":1.6859999999999999},"min":{"1min":0.378,"5min":0.247,"15min":0.247},"max":{"1min":4.4189999999999996,"5min":9.4719999999999995,"15min":9.4719999999999995},"last":0.35399999999999998},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.5149999999999999,"15min":1.4399999999999999},"min":{"1min":0.378,"5min":0.252,"15min":0.252},"max":{"1min":4.8579999999999997,"5min":7.96,"15min":7.96},"last":0.60399999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:36:14 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":1.0409999999999999,"15min":1.4610000000000001},"min":{"1min":0.28399999999999997,"5min":0.25700000000000001,"15min":0.247},"max":{"1min":1.1319999999999999,"5min":5.0860000000000003,"15min":21.315999999999999},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.2649999999999999,"15min":1.409},"min":{"1min":0.33400000000000002,"5min":0.29599999999999999,"15min":0.26400000000000001},"max":{"1min":2.2850000000000001,"5min":7.2439999999999998,"15min":18.087},"last":3.7320000000000002}]},{"osd":4,"last update":"Fri Feb 9 17:36:14 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":2.4239999999999999,"15min":1.319},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":16.125,"5min":16.125,"15min":16.125},"last":0.29999999999999999},{"interface":"front","average":{"1min":1.28,"5min":1.6479999999999999,"15min":1.143},"min":{"1min":0.33600000000000002,"5min":0.33300000000000002,"15min":0.27700000000000002},"max":{"1min":16.091999999999999,"5min":16.091999999999999,"15min":16.091999999999999},"last":0.372}]},{"osd":5,"last update":"Fri Feb 9 17:36:14 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":1.9259999999999999,"15min":1.2410000000000001},"min":{"1min":0.371,"5min":0.312,"15min":0.28899999999999998},"max":{"1min":3.2240000000000002,"5min":13.946999999999999,"15min":13.946999999999999},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.50700000000000001,"5min":2.133,"15min":1.298},"min":{"1min":0.31900000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.88400000000000001,"5min":13.414,"15min":13.414},"last":0.38900000000000001}]},{"osd":6,"last update":"Fri Feb 9 17:36:14 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":1.986,"15min":1.2450000000000001},"min":{"1min":0.30199999999999999,"5min":0.30099999999999999,"15min":0.29899999999999999},"max":{"1min":1.1839999999999999,"5min":12.798999999999999,"15min":12.798999999999999},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.59699999999999998,"5min":1.9410000000000001,"15min":1.2669999999999999},"min":{"1min":0.33800000000000002,"5min":0.33000000000000002,"15min":0.32300000000000001},"max":{"1min":0.998,"5min":10.782999999999999,"15min":10.782999999999999},"last":0.65700000000000003}]},{"osd":7,"last update":"Fri Feb 9 17:36:14 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":2.274,"15min":1.4239999999999999},"min":{"1min":0.38900000000000001,"5min":0.312,"15min":0.312},"max":{"1min":1.4730000000000001,"5min":14.462,"15min":14.462},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.68200000000000005,"5min":2.3809999999999998,"15min":1.413},"min":{"1min":0.42399999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.3640000000000001,"5min":14.398999999999999,"15min":14.398999999999999},"last":0.80600000000000005}]},{"osd":8,"last update":"Fri Feb 9 17:36:46 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":1.472,"15min":2.1040000000000001},"min":{"1min":0.34799999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":2.8069999999999999,"5min":13.199999999999999,"15min":13.199999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.86099999999999999,"5min":1.26,"15min":1.7090000000000001},"min":{"1min":0.36899999999999999,"5min":0.318,"15min":0.318},"max":{"1min":3.1859999999999999,"5min":7.1520000000000001,"15min":7.1520000000000001},"last":0.72299999999999998}]},{"osd":9,"last update":"Fri Feb 9 17:36:00 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":1.446,"15min":1.2769999999999999},"min":{"1min":0.33300000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.216,"5min":12.821,"15min":12.821},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.68000000000000005,"5min":1.2769999999999999,"15min":1.036},"min":{"1min":0.38500000000000001,"5min":0.36099999999999999,"15min":0.34100000000000003},"max":{"1min":1.232,"5min":7.6989999999999998,"15min":7.6989999999999998},"last":0.85799999999999998}]},{"osd":10,"last update":"Fri Feb 9 17:36:14 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":1.333,"15min":1.05},"min":{"1min":0.41699999999999998,"5min":0.36199999999999999,"15min":0.33900000000000002},"max":{"1min":0.79600000000000004,"5min":9.6199999999999992,"15min":9.6199999999999992},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.66200000000000003,"5min":1.498,"15min":1.069},"min":{"1min":0.45900000000000002,"5min":0.35799999999999998,"15min":0.33700000000000002},"max":{"1min":1.298,"5min":12.856,"15min":12.856},"last":0.83499999999999996}]}]},{"osd":10,"up_from":84,"seq":360777253045,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2169552,"kb_used_data":2012040,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":91575600,"statfs":{"total":95995035648,"available":93773414400,"internally_reserved":0,"allocated":2060328960,"data_stored":2059634520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":161282496},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:36:04 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":1.6839999999999999,"15min":1.821},"min":{"1min":0.27000000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":5.1139999999999999,"5min":8.5239999999999991,"15min":8.5239999999999991},"last":1.8420000000000001},{"interface":"front","average":{"1min":0.68799999999999994,"5min":1.04,"15min":1.0980000000000001},"min":{"1min":0.28899999999999998,"5min":0.253,"15min":0.253},"max":{"1min":2.177,"5min":5.2060000000000004,"15min":5.2060000000000004},"last":0.79100000000000004}]},{"osd":1,"last update":"Fri Feb 9 17:33:46 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":1.345,"15min":1.1599999999999999},"min":{"1min":0.25900000000000001,"5min":0.24199999999999999,"15min":0.23499999999999999},"max":{"1min":3.1709999999999998,"5min":37.716000000000001,"15min":37.716000000000001},"last":0.26000000000000001},{"interface":"front","average":{"1min":1.121,"5min":1.3660000000000001,"15min":1.194},"min":{"1min":0.251,"5min":0.251,"15min":0.245},"max":{"1min":5.3049999999999997,"5min":38.039999999999999,"15min":38.039999999999999},"last":0.27800000000000002}]},{"osd":2,"last update":"Fri Feb 9 17:36:51 2024","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":0.98799999999999999,"15min":1.1910000000000001},"min":{"1min":0.29799999999999999,"5min":0.27100000000000002,"15min":0.222},"max":{"1min":3.7440000000000002,"5min":6.4199999999999999,"15min":13.351000000000001},"last":3.9769999999999999},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.089,"15min":1.2},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.26100000000000001},"max":{"1min":3.7770000000000001,"5min":6.4829999999999997,"15min":10.596},"last":3.9620000000000002}]},{"osd":3,"last update":"Fri Feb 9 17:36:51 2024","interfaces":[{"interface":"back","average":{"1min":1.6439999999999999,"5min":2.0179999999999998,"15min":1.573},"min":{"1min":0.28699999999999998,"5min":0.26800000000000002,"15min":0.26200000000000001},"max":{"1min":8.8800000000000008,"5min":25.972999999999999,"15min":25.972999999999999},"last":0.44600000000000001},{"interface":"front","average":{"1min":1.4390000000000001,"5min":1.7829999999999999,"15min":1.4390000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":8.2509999999999994,"5min":25.882000000000001,"15min":25.882000000000001},"last":1.7609999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:36:55 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":1.274,"15min":1.012},"min":{"1min":0.307,"5min":0.307,"15min":0.26100000000000001},"max":{"1min":3.589,"5min":7.1360000000000001,"15min":7.1360000000000001},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.83599999999999997,"5min":1.2949999999999999,"15min":1.0409999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":3.5419999999999998,"5min":6.0119999999999996,"15min":6.8369999999999997},"last":0.51100000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:36:51 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":1.282,"15min":1.2},"min":{"1min":0.30199999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.9809999999999999,"5min":9.2349999999999994,"15min":9.2349999999999994},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.95399999999999996,"5min":1.3029999999999999,"15min":1.157},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.32200000000000001},"max":{"1min":3.8959999999999999,"5min":5.3079999999999998,"15min":8.6159999999999997},"last":0.57599999999999996}]},{"osd":6,"last update":"Fri Feb 9 17:36:51 2024","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":1.331,"15min":1.107},"min":{"1min":0.36899999999999999,"5min":0.29999999999999999,"15min":0.29099999999999998},"max":{"1min":5.6550000000000002,"5min":6.3760000000000003,"15min":6.8799999999999999},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.93600000000000005,"5min":1.7749999999999999,"15min":1.3169999999999999},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":3.1389999999999998,"5min":9.4350000000000005,"15min":9.4350000000000005},"last":0.63600000000000001}]},{"osd":7,"last update":"Fri Feb 9 17:36:51 2024","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":1.778,"15min":1.242},"min":{"1min":0.39600000000000002,"5min":0.33500000000000002,"15min":0.312},"max":{"1min":3.3180000000000001,"5min":17.754000000000001,"15min":17.754000000000001},"last":0.498},{"interface":"front","average":{"1min":1.03,"5min":1.958,"15min":1.3580000000000001},"min":{"1min":0.45100000000000001,"5min":0.42599999999999999,"15min":0.316},"max":{"1min":3.665,"5min":11.308999999999999,"15min":11.308999999999999},"last":0.60999999999999999}]},{"osd":8,"last update":"Fri Feb 9 17:36:35 2024","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":1.081,"15min":1.1950000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":2.3130000000000002,"5min":6.4630000000000001,"15min":6.4630000000000001},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.84499999999999997,"15min":0.94999999999999996},"min":{"1min":0.38900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":3.794,"5min":5.3079999999999998,"15min":5.3079999999999998},"last":0.67800000000000005}]},{"osd":9,"last update":"Fri Feb 9 17:36:51 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.97899999999999998,"15min":0.94899999999999995},"min":{"1min":0.372,"5min":0.36099999999999999,"15min":0.314},"max":{"1min":3.1070000000000002,"5min":6.5049999999999999,"15min":8.7240000000000002},"last":0.435},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.048,"15min":1.0940000000000001},"min":{"1min":0.40300000000000002,"5min":0.35899999999999999,"15min":0.29799999999999999},"max":{"1min":4.1029999999999998,"5min":6.9640000000000004,"15min":10.1},"last":0.59499999999999997}]},{"osd":11,"last update":"Fri Feb 9 17:36:15 2024","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":1.1379999999999999,"15min":1.1020000000000001},"min":{"1min":0.39800000000000002,"5min":0.379,"15min":0.36399999999999999},"max":{"1min":2.7909999999999999,"5min":7.383,"15min":8.6430000000000007},"last":0.54800000000000004},{"interface":"front","average":{"1min":1.002,"5min":1.196,"15min":1.1080000000000001},"min":{"1min":0.45000000000000001,"5min":0.377,"15min":0.33400000000000002},"max":{"1min":2.831,"5min":7.3019999999999996,"15min":9.298},"last":0.65900000000000003}]}]},{"osd":9,"up_from":77,"seq":330712481978,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1787472,"kb_used_data":1495304,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":91957680,"statfs":{"total":95995035648,"available":94164664320,"internally_reserved":0,"allocated":1531191296,"data_stored":1530496856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":299170240},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:28:03 2024","interfaces":[{"interface":"back","average":{"1min":1.9510000000000001,"5min":1.454,"15min":0.85799999999999998},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":8.4499999999999993,"5min":17.495999999999999,"15min":17.495999999999999},"last":2.7360000000000002},{"interface":"front","average":{"1min":1.905,"5min":1.399,"15min":0.83499999999999996},"min":{"1min":0.32200000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":7.5990000000000002,"5min":15.401999999999999,"15min":15.401999999999999},"last":1.5449999999999999}]},{"osd":1,"last update":"Fri Feb 9 17:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.88400000000000001,"15min":1.2070000000000001},"min":{"1min":0.27100000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":4.601,"5min":5.2549999999999999,"15min":16.527999999999999},"last":2.581},{"interface":"front","average":{"1min":1.0209999999999999,"5min":0.88500000000000001,"15min":1.095},"min":{"1min":0.255,"5min":0.222,"15min":0.222},"max":{"1min":4.282,"5min":4.282,"15min":17.771999999999998},"last":2.7160000000000002}]},{"osd":2,"last update":"Fri Feb 9 17:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":1.038,"15min":1.131},"min":{"1min":0.28399999999999997,"5min":0.249,"15min":0.249},"max":{"1min":1.8320000000000001,"5min":6.0570000000000004,"15min":13.878},"last":1.288},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.997,"15min":1.2},"min":{"1min":0.26100000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":4.0599999999999996,"5min":5.5590000000000002,"15min":13.69},"last":2.6139999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:36:28 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":1.4199999999999999,"15min":1.202},"min":{"1min":0.307,"5min":0.29699999999999999,"15min":0.26000000000000001},"max":{"1min":2.0950000000000002,"5min":40.18,"15min":40.18},"last":1.405},{"interface":"front","average":{"1min":0.75900000000000001,"5min":1.4370000000000001,"15min":1.391},"min":{"1min":0.35699999999999998,"5min":0.30599999999999999,"15min":0.27900000000000003},"max":{"1min":3.4420000000000002,"5min":40.158000000000001,"15min":40.158000000000001},"last":2.2709999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":1.7070000000000001,"15min":1.2649999999999999},"min":{"1min":0.33800000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":2.2450000000000001,"5min":9.0519999999999996,"15min":22.577999999999999},"last":0.5},{"interface":"front","average":{"1min":0.63400000000000001,"5min":1.54,"15min":1.1399999999999999},"min":{"1min":0.33300000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":2.2389999999999999,"5min":12.651,"15min":22.399999999999999},"last":2.238}]},{"osd":5,"last update":"Fri Feb 9 17:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":1.7949999999999999,"15min":1.2310000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.8049999999999999,"5min":10.689,"15min":10.689},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.71499999999999997,"5min":1.3580000000000001,"15min":1.042},"min":{"1min":0.378,"5min":0.311,"15min":0.311},"max":{"1min":2.202,"5min":6.4470000000000001,"15min":6.4470000000000001},"last":2.3730000000000002}]},{"osd":6,"last update":"Fri Feb 9 17:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":1.734,"15min":1.2589999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.32200000000000001},"max":{"1min":3.0590000000000002,"5min":9.1699999999999999,"15min":9.1699999999999999},"last":1.3580000000000001},{"interface":"front","average":{"1min":0.93100000000000005,"5min":1.7729999999999999,"15min":1.2589999999999999},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.35099999999999998},"max":{"1min":3.1110000000000002,"5min":9.2080000000000002,"15min":9.2080000000000002},"last":1.3899999999999999}]},{"osd":7,"last update":"Fri Feb 9 17:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":1.73,"15min":1.224},"min":{"1min":0.40999999999999998,"5min":0.38500000000000001,"15min":0.34300000000000003},"max":{"1min":3.2320000000000002,"5min":7.3710000000000004,"15min":8.0939999999999994},"last":1.498},{"interface":"front","average":{"1min":0.98099999999999998,"5min":1.7709999999999999,"15min":1.2909999999999999},"min":{"1min":0.38400000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":3.262,"5min":8.0530000000000008,"15min":8.0530000000000008},"last":1.5249999999999999}]},{"osd":8,"last update":"Fri Feb 9 17:36:46 2024","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":1.157,"15min":1.2789999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":4.141,"5min":5.5069999999999997,"15min":16.209},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.999,"5min":1.1180000000000001,"15min":1.29},"min":{"1min":0.41299999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":2.492,"5min":10.241,"15min":16.263000000000002},"last":2.347}]},{"osd":10,"last update":"Fri Feb 9 17:35:57 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":1.1870000000000001,"15min":1.056},"min":{"1min":0.44700000000000001,"5min":0.33900000000000002,"15min":0.33300000000000002},"max":{"1min":3.274,"5min":9.5090000000000003,"15min":9.5090000000000003},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.88500000000000001,"15min":0.99099999999999999},"min":{"1min":0.41399999999999998,"5min":0.38500000000000001,"15min":0.36099999999999999},"max":{"1min":1.7809999999999999,"5min":4.0759999999999996,"15min":8.0670000000000002},"last":2.2989999999999999}]},{"osd":11,"last update":"Fri Feb 9 17:35:57 2024","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":1.464,"15min":1.3320000000000001},"min":{"1min":0.46700000000000003,"5min":0.42399999999999999,"15min":0.34699999999999998},"max":{"1min":2.657,"5min":11.541,"15min":11.541},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.95499999999999996,"5min":1.3129999999999999,"15min":1.1220000000000001},"min":{"1min":0.43099999999999999,"5min":0.374,"15min":0.374},"max":{"1min":3.4889999999999999,"5min":10.32,"15min":10.32},"last":2.3239999999999998}]}]},{"osd":8,"up_from":171,"seq":734439407721,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":133332,"kb_used_data":1864,"kb_used_omap":212,"kb_used_meta":131243,"kb_avail":93611820,"statfs":{"total":95995035648,"available":95858503680,"internally_reserved":0,"allocated":1908736,"data_stored":1214304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217579,"internal_metadata":134393365},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:36:31 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":0.98499999999999999,"15min":1.3899999999999999},"min":{"1min":0.246,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":3.0129999999999999,"5min":8.0289999999999999,"15min":8.0289999999999999},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.89200000000000002,"5min":1.0589999999999999,"15min":1.3660000000000001},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":3.0960000000000001,"5min":7.5990000000000002,"15min":7.7969999999999997},"last":0.68400000000000005}]},{"osd":1,"last update":"Fri Feb 9 17:36:31 2024","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.2470000000000001,"15min":1.2669999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.0810000000000004,"5min":7.8719999999999999,"15min":7.8719999999999999},"last":2.5409999999999999},{"interface":"front","average":{"1min":1.05,"5min":1.1879999999999999,"15min":1.841},"min":{"1min":0.23400000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":3.6680000000000001,"5min":8.5519999999999996,"15min":11.101000000000001},"last":3.5649999999999999}]},{"osd":2,"last update":"Fri Feb 9 17:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":1.522,"15min":1.45},"min":{"1min":0.29399999999999998,"5min":0.28199999999999997,"15min":0.255},"max":{"1min":3.8239999999999998,"5min":10.423,"15min":10.423},"last":0.442},{"interface":"front","average":{"1min":0.93400000000000005,"5min":1.3080000000000001,"15min":1.611},"min":{"1min":0.373,"5min":0.34300000000000003,"15min":0.313},"max":{"1min":3.847,"5min":10.504,"15min":10.504},"last":0.52000000000000002}]},{"osd":3,"last update":"Fri Feb 9 17:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":1.4790000000000001,"15min":2.0289999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.28699999999999998},"max":{"1min":3.7210000000000001,"5min":24.030999999999999,"15min":24.030999999999999},"last":2.6549999999999998},{"interface":"front","average":{"1min":1.163,"5min":1.5880000000000001,"15min":2.327},"min":{"1min":0.45500000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":3.1110000000000002,"5min":24.318999999999999,"15min":24.318999999999999},"last":4.1529999999999996}]},{"osd":4,"last update":"Fri Feb 9 17:36:31 2024","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":1.8500000000000001,"15min":1.238},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.27500000000000002},"max":{"1min":3.2010000000000001,"5min":53.060000000000002,"15min":53.060000000000002},"last":0.495},{"interface":"front","average":{"1min":0.60999999999999999,"5min":2.0760000000000001,"15min":1.522},"min":{"1min":0.32900000000000001,"5min":0.313,"15min":0.313},"max":{"1min":1.4430000000000001,"5min":53.311999999999998,"15min":53.311999999999998},"last":3.3380000000000001}]},{"osd":6,"last update":"Fri Feb 9 17:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":1.7470000000000001,"15min":1.589},"min":{"1min":0.36399999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":2.8980000000000001,"5min":10.619999999999999,"15min":37.356999999999999},"last":0.62},{"interface":"front","average":{"1min":0.96999999999999997,"5min":1.3999999999999999,"15min":1.143},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":3.0739999999999998,"5min":4.984,"15min":37.25},"last":0.77900000000000003}]},{"osd":7,"last update":"Fri Feb 9 17:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":1.9039999999999999,"15min":1.4750000000000001},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":3.4399999999999999,"5min":12.731999999999999,"15min":37.585000000000001},"last":0.93999999999999995},{"interface":"front","average":{"1min":0.83399999999999996,"5min":1.9430000000000001,"15min":1.6459999999999999},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.312},"max":{"1min":3.4710000000000001,"5min":12.779,"15min":37.918999999999997},"last":0.95799999999999996}]},{"osd":9,"last update":"Fri Feb 9 17:36:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.2190000000000001,"15min":1.1870000000000001},"min":{"1min":0.44900000000000001,"5min":0.34100000000000003,"15min":0.29699999999999999},"max":{"1min":3.2440000000000002,"5min":9.4900000000000002,"15min":9.4900000000000002},"last":3.359},{"interface":"front","average":{"1min":1.006,"5min":1.266,"15min":1.23},"min":{"1min":0.36599999999999999,"5min":0.35399999999999998,"15min":0.32000000000000001},"max":{"1min":4.101,"5min":17.434999999999999,"15min":17.434999999999999},"last":3.1749999999999998}]},{"osd":10,"last update":"Fri Feb 9 17:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":1.127,"15min":1.1910000000000001},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.33700000000000002},"max":{"1min":1.3979999999999999,"5min":10.65,"15min":10.65},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.99199999999999999,"5min":1.1699999999999999,"15min":1.0820000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":3.024,"5min":8.2799999999999994,"15min":8.2799999999999994},"last":0.82599999999999996}]},{"osd":11,"last update":"Fri Feb 9 17:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":1.5760000000000001,"15min":1.381},"min":{"1min":0.42899999999999999,"5min":0.375,"15min":0.375},"max":{"1min":5.4980000000000002,"5min":14.433999999999999,"15min":14.433999999999999},"last":3.4630000000000001},{"interface":"front","average":{"1min":1.012,"5min":1.3500000000000001,"15min":1.4610000000000001},"min":{"1min":0.48399999999999999,"5min":0.45000000000000001,"15min":0.38100000000000001},"max":{"1min":5.3239999999999998,"5min":11.622,"15min":11.622},"last":3.4119999999999999}]}]},{"osd":7,"up_from":57,"seq":244813136067,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1798544,"kb_used_data":1500808,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":91946608,"statfs":{"total":95995035648,"available":94153326592,"internally_reserved":0,"allocated":1536827392,"data_stored":1536132960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":304871882},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:35:59 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":3.109,"15min":3.4590000000000001},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":8.5609999999999999,"5min":36.753,"15min":36.753},"last":2.008},{"interface":"front","average":{"1min":1.2070000000000001,"5min":3.3260000000000001,"15min":3.6789999999999998},"min":{"1min":0.27000000000000002,"5min":0.253,"15min":0.253},"max":{"1min":9.1649999999999991,"5min":36.918999999999997,"15min":36.918999999999997},"last":0.84999999999999998}]},{"osd":1,"last update":"Fri Feb 9 17:36:06 2024","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":2.073,"15min":1.972},"min":{"1min":0.28399999999999997,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":6.4240000000000004,"5min":20.553999999999998,"15min":20.553999999999998},"last":1.9630000000000001},{"interface":"front","average":{"1min":0.80800000000000005,"5min":1.782,"15min":1.581},"min":{"1min":0.35999999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":2.7690000000000001,"5min":20.568000000000001,"15min":20.568000000000001},"last":2.4089999999999998}]},{"osd":2,"last update":"Fri Feb 9 17:36:29 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.8029999999999999,"15min":1.456},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.26000000000000001},"max":{"1min":2.722,"5min":11.324,"15min":11.324},"last":0.92500000000000004},{"interface":"front","average":{"1min":1.0569999999999999,"5min":2.004,"15min":1.5349999999999999},"min":{"1min":0.32100000000000001,"5min":0.309,"15min":0.28299999999999997},"max":{"1min":2.742,"5min":19.361999999999998,"15min":19.361999999999998},"last":0.95899999999999996}]},{"osd":3,"last update":"Fri Feb 9 17:36:29 2024","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":2.077,"15min":1.554},"min":{"1min":0.36399999999999999,"5min":0.28899999999999998,"15min":0.26900000000000002},"max":{"1min":2.7010000000000001,"5min":20.754000000000001,"15min":27.719000000000001},"last":1.9330000000000001},{"interface":"front","average":{"1min":0.79700000000000004,"5min":2.2149999999999999,"15min":1.5780000000000001},"min":{"1min":0.34799999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":1.958,"5min":24.457000000000001,"15min":24.457000000000001},"last":2.2290000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:36:29 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":2.3319999999999999,"15min":1.286},"min":{"1min":0.28000000000000003,"5min":0.25700000000000001,"15min":0.248},"max":{"1min":2.544,"5min":14.372999999999999,"15min":14.372999999999999},"last":1.8919999999999999},{"interface":"front","average":{"1min":0.77700000000000002,"5min":2.415,"15min":1.2909999999999999},"min":{"1min":0.379,"5min":0.34699999999999998,"15min":0.22800000000000001},"max":{"1min":2.8599999999999999,"5min":14.401,"15min":14.401},"last":2.0449999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:35:28 2024","interfaces":[{"interface":"back","average":{"1min":2.073,"5min":2.2999999999999998,"15min":1.1619999999999999},"min":{"1min":0.36899999999999999,"5min":0.28599999999999998,"15min":0.21299999999999999},"max":{"1min":13.409000000000001,"5min":13.409000000000001,"15min":13.409000000000001},"last":0.53300000000000003},{"interface":"front","average":{"1min":1.946,"5min":2.1230000000000002,"15min":1.1240000000000001},"min":{"1min":0.30599999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":10.553000000000001,"5min":11.598000000000001,"15min":11.598000000000001},"last":0.48999999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:36:29 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":2.3559999999999999,"15min":1.3700000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":2.6909999999999998,"5min":25.965,"15min":25.965},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.61599999999999999,"5min":1.8959999999999999,"15min":1.155},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.28699999999999998},"max":{"1min":1.6279999999999999,"5min":9.4090000000000007,"15min":9.4090000000000007},"last":2.254}]},{"osd":8,"last update":"Fri Feb 9 17:36:29 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":1.6970000000000001,"15min":1.4970000000000001},"min":{"1min":0.442,"5min":0.36599999999999999,"15min":0.32300000000000001},"max":{"1min":2.641,"5min":9.0809999999999995,"15min":9.0809999999999995},"last":2.125},{"interface":"front","average":{"1min":0.80000000000000004,"5min":1.635,"15min":1.2529999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":2.5070000000000001,"5min":11.481,"15min":11.481},"last":0.88700000000000001}]},{"osd":9,"last update":"Fri Feb 9 17:36:16 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":1.847,"15min":1.395},"min":{"1min":0.438,"5min":0.40899999999999997,"15min":0.39600000000000002},"max":{"1min":2.7690000000000001,"5min":8.9900000000000002,"15min":41.286999999999999},"last":2.093},{"interface":"front","average":{"1min":1.127,"5min":1.9259999999999999,"15min":1.4330000000000001},"min":{"1min":0.41399999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":7.3579999999999997,"5min":11.417999999999999,"15min":41.234999999999999},"last":2.1539999999999999}]},{"osd":10,"last update":"Fri Feb 9 17:36:44 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":1.8109999999999999,"15min":1.3009999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.35899999999999999},"max":{"1min":1.6399999999999999,"5min":9.4830000000000005,"15min":9.4830000000000005},"last":2.2869999999999999},{"interface":"front","average":{"1min":0.95299999999999996,"5min":2.044,"15min":1.385},"min":{"1min":0.52200000000000002,"5min":0.45600000000000002,"15min":0.35599999999999998},"max":{"1min":2.5289999999999999,"5min":9.109,"15min":9.109},"last":2.181}]},{"osd":11,"last update":"Fri Feb 9 17:36:06 2024","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":2.3260000000000001,"15min":1.4910000000000001},"min":{"1min":0.438,"5min":0.41299999999999998,"15min":0.40999999999999998},"max":{"1min":2.23,"5min":12.885999999999999,"15min":12.885999999999999},"last":2.2719999999999998},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.9590000000000001,"15min":1.321},"min":{"1min":0.53800000000000003,"5min":0.41299999999999998,"15min":0.39500000000000002},"max":{"1min":5.2359999999999998,"5min":9.9489999999999998,"15min":9.9489999999999998},"last":5.6230000000000002}]}]},{"osd":6,"up_from":49,"seq":210453397704,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3261264,"kb_used_data":2973256,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":90483888,"statfs":{"total":95995035648,"available":92655501312,"internally_reserved":0,"allocated":3044614144,"data_stored":3043919712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":294910406},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:36:04 2024","interfaces":[{"interface":"back","average":{"1min":1.5289999999999999,"5min":1.7929999999999999,"15min":1.3420000000000001},"min":{"1min":0.25700000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":12.544,"5min":12.544,"15min":24.544},"last":0.28199999999999997},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.649,"15min":1.2589999999999999},"min":{"1min":0.26700000000000002,"5min":0.217,"15min":0.217},"max":{"1min":7.7599999999999998,"5min":8.3689999999999998,"15min":8.3689999999999998},"last":2.3239999999999998}]},{"osd":1,"last update":"Fri Feb 9 17:36:04 2024","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":2.1920000000000002,"15min":1.504},"min":{"1min":0.26700000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":9.0340000000000007,"5min":17.762,"15min":23.954000000000001},"last":2.9009999999999998},{"interface":"front","average":{"1min":1.3140000000000001,"5min":2.2589999999999999,"15min":1.55},"min":{"1min":0.33700000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":9.0609999999999999,"5min":17.734999999999999,"15min":23.984999999999999},"last":2.9300000000000002}]},{"osd":2,"last update":"Fri Feb 9 17:36:04 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":1.6930000000000001,"15min":1.4239999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.22600000000000001},"max":{"1min":1.736,"5min":8.0239999999999991,"15min":24.329999999999998},"last":2.621},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.742,"15min":1.423},"min":{"1min":0.375,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":7.7999999999999998,"5min":10.156000000000001,"15min":12.906000000000001},"last":2.6419999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:36:04 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":2.2549999999999999,"15min":1.667},"min":{"1min":0.28199999999999997,"5min":0.25600000000000001,"15min":0.248},"max":{"1min":4.9960000000000004,"5min":16.931999999999999,"15min":32.545000000000002},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.79400000000000004,"5min":2.3769999999999998,"15min":1.643},"min":{"1min":0.41899999999999998,"5min":0.29599999999999999,"15min":0.26600000000000001},"max":{"1min":3.165,"5min":17.047000000000001,"15min":32.670999999999999},"last":0.32600000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:36:04 2024","interfaces":[{"interface":"back","average":{"1min":1.091,"5min":2.452,"15min":1.264},"min":{"1min":0.34699999999999998,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":5.6550000000000002,"5min":19.798999999999999,"15min":19.798999999999999},"last":0.255},{"interface":"front","average":{"1min":1.343,"5min":1.988,"15min":1.1930000000000001},"min":{"1min":0.316,"5min":0.28000000000000003,"15min":0.26000000000000001},"max":{"1min":7.976,"5min":10.702999999999999,"15min":10.702999999999999},"last":0.40400000000000003}]},{"osd":5,"last update":"Fri Feb 9 17:36:04 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":2.149,"15min":1.1850000000000001},"min":{"1min":0.41699999999999998,"5min":0.29499999999999998,"15min":0.29199999999999998},"max":{"1min":8.6370000000000005,"5min":10.968999999999999,"15min":10.968999999999999},"last":0.376},{"interface":"front","average":{"1min":0.89400000000000002,"5min":2.298,"15min":1.244},"min":{"1min":0.39100000000000001,"5min":0.35299999999999998,"15min":0.27400000000000002},"max":{"1min":5.5090000000000003,"5min":13.91,"15min":13.91},"last":0.35999999999999999}]},{"osd":7,"last update":"Fri Feb 9 17:36:39 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":2.1680000000000001,"15min":1.2869999999999999},"min":{"1min":0.32200000000000001,"5min":0.311,"15min":0.28000000000000003},"max":{"1min":5.0579999999999998,"5min":14.022,"15min":14.022},"last":0.311},{"interface":"front","average":{"1min":0.90300000000000002,"5min":2.2730000000000001,"15min":1.405},"min":{"1min":0.36699999999999999,"5min":0.35699999999999998,"15min":0.309},"max":{"1min":2.9420000000000002,"5min":15.458,"15min":15.458},"last":0.42999999999999999}]},{"osd":8,"last update":"Fri Feb 9 17:26:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":0.876,"15min":0.79600000000000004},"min":{"1min":0.39900000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":4.0339999999999998,"5min":4.4880000000000004,"15min":4.4880000000000004},"last":4.3840000000000003},{"interface":"front","average":{"1min":1.0469999999999999,"5min":0.88600000000000001,"15min":0.78300000000000003},"min":{"1min":0.375,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":3.6539999999999999,"5min":4.8570000000000002,"15min":4.8570000000000002},"last":4.4210000000000003}]},{"osd":9,"last update":"Fri Feb 9 17:36:12 2024","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":1.5820000000000001,"15min":1.204},"min":{"1min":0.54800000000000004,"5min":0.374,"15min":0.35999999999999999},"max":{"1min":2.7149999999999999,"5min":8.3409999999999993,"15min":10.803000000000001},"last":1.101},{"interface":"front","average":{"1min":0.97399999999999998,"5min":1.7310000000000001,"15min":1.3},"min":{"1min":0.58599999999999997,"5min":0.40400000000000003,"15min":0.40300000000000002},"max":{"1min":2.8530000000000002,"5min":8.1630000000000003,"15min":8.1630000000000003},"last":0.495}]},{"osd":10,"last update":"Fri Feb 9 17:36:39 2024","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":1.631,"15min":1.325},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.35399999999999998},"max":{"1min":5.5469999999999997,"5min":7.734,"15min":9.5399999999999991},"last":0.55500000000000005},{"interface":"front","average":{"1min":1.0269999999999999,"5min":2.0129999999999999,"15min":1.417},"min":{"1min":0.41999999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":4.8819999999999997,"5min":13.836,"15min":13.836},"last":0.46200000000000002}]},{"osd":11,"last update":"Fri Feb 9 17:36:04 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.8600000000000001,"15min":1.5940000000000001},"min":{"1min":0.40000000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":4.9539999999999997,"5min":10.273999999999999,"15min":10.273999999999999},"last":0.441},{"interface":"front","average":{"1min":1.5,"5min":2.0680000000000001,"15min":1.7789999999999999},"min":{"1min":0.56399999999999995,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":7.8650000000000002,"5min":8.2739999999999991,"15min":8.2739999999999991},"last":1.016}]}]},{"osd":1,"up_from":13,"seq":55834575075,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4951760,"kb_used_data":4609608,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":88793392,"statfs":{"total":95995035648,"available":90924433408,"internally_reserved":0,"allocated":4720238592,"data_stored":4719544160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":350353861},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:36:08 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":1.109,"15min":1.2050000000000001},"min":{"1min":0.26200000000000001,"5min":0.223,"15min":0.214},"max":{"1min":2.605,"5min":9.2609999999999992,"15min":12.492000000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.94999999999999996,"5min":1.21,"15min":1.264},"min":{"1min":0.33200000000000002,"5min":0.218,"15min":0.214},"max":{"1min":4.9349999999999996,"5min":7.5810000000000004,"15min":9.8439999999999994},"last":1.226}]},{"osd":2,"last update":"Fri Feb 9 17:36:39 2024","interfaces":[{"interface":"back","average":{"1min":2.4430000000000001,"5min":1.423,"15min":1.474},"min":{"1min":0.27600000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":26.361999999999998,"5min":26.361999999999998,"15min":26.361999999999998},"last":9.5039999999999996},{"interface":"front","average":{"1min":2.3650000000000002,"5min":1.397,"15min":1.597},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":26.087,"5min":26.087,"15min":26.087},"last":1.2609999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:36:32 2024","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":8.5030000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":2.0600000000000001},{"interface":"front","average":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.3480000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":4.0250000000000004,"5min":4.0250000000000004,"15min":4.0250000000000004},"last":0.24099999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:36:08 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":1.774,"15min":2.056},"min":{"1min":0.32100000000000001,"5min":0.253,"15min":0.253},"max":{"1min":3.4529999999999998,"5min":11.766999999999999,"15min":11.766999999999999},"last":4.9459999999999997},{"interface":"front","average":{"1min":0.95299999999999996,"5min":1.849,"15min":1.907},"min":{"1min":0.39500000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":3.415,"5min":8.0519999999999996,"15min":8.0519999999999996},"last":0.309}]},{"osd":5,"last update":"Fri Feb 9 17:36:50 2024","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.774,"15min":1.518},"min":{"1min":0.374,"5min":0.28599999999999998,"15min":0.27000000000000002},"max":{"1min":8.2859999999999996,"5min":13.544,"15min":13.544},"last":5.0750000000000002},{"interface":"front","average":{"1min":2.0089999999999999,"5min":1.839,"15min":1.353},"min":{"1min":0.36699999999999999,"5min":0.26800000000000002,"15min":0.26000000000000001},"max":{"1min":4.843,"5min":12.117000000000001,"15min":12.117000000000001},"last":0.39800000000000002}]},{"osd":6,"last update":"Fri Feb 9 17:36:08 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":1.679,"15min":1.506},"min":{"1min":0.41999999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":3.5259999999999998,"5min":8.7210000000000001,"15min":10.275},"last":5.0199999999999996},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.792,"15min":1.4970000000000001},"min":{"1min":0.36499999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":5.1020000000000003,"5min":9.0069999999999997,"15min":9.0069999999999997},"last":1.3069999999999999}]},{"osd":7,"last update":"Fri Feb 9 17:34:33 2024","interfaces":[{"interface":"back","average":{"1min":2.1560000000000001,"5min":1.9099999999999999,"15min":1.5609999999999999},"min":{"1min":0.35699999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":9.0489999999999995,"5min":15.148999999999999,"15min":15.148999999999999},"last":0.76600000000000001},{"interface":"front","average":{"1min":1.419,"5min":1.3740000000000001,"15min":1.425},"min":{"1min":0.31,"5min":0.31,"15min":0.28299999999999997},"max":{"1min":7.3380000000000001,"5min":13.834,"15min":13.834},"last":0.61499999999999999}]},{"osd":8,"last update":"Fri Feb 9 17:36:32 2024","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.2929999999999999,"15min":1.3959999999999999},"min":{"1min":0.39200000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":3.8079999999999998,"5min":7.4349999999999996,"15min":7.4349999999999996},"last":0.45500000000000002},{"interface":"front","average":{"1min":1.333,"5min":1.569,"15min":1.9790000000000001},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":8.3780000000000001,"5min":10.433999999999999,"15min":10.433999999999999},"last":4.9889999999999999}]},{"osd":9,"last update":"Fri Feb 9 17:36:17 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":1.464,"15min":1.4710000000000001},"min":{"1min":0.41199999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":3.548,"5min":9.5839999999999996,"15min":9.5839999999999996},"last":5.048},{"interface":"front","average":{"1min":0.99199999999999999,"5min":1.288,"15min":1.4530000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.30099999999999999},"max":{"1min":4.5389999999999997,"5min":8.4079999999999995,"15min":12.117000000000001},"last":1.3560000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:36:50 2024","interfaces":[{"interface":"back","average":{"1min":1.9099999999999999,"5min":1.425,"15min":1.321},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.35999999999999999},"max":{"1min":4.8810000000000002,"5min":7.9809999999999999,"15min":7.9809999999999999},"last":0.43099999999999999},{"interface":"front","average":{"1min":1.958,"5min":1.512,"15min":1.581},"min":{"1min":0.48099999999999998,"5min":0.44400000000000001,"15min":0.33400000000000002},"max":{"1min":4.9080000000000004,"5min":8.0020000000000007,"15min":12.087999999999999},"last":0.48199999999999998}]},{"osd":11,"last update":"Fri Feb 9 17:36:56 2024","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":1.585,"15min":2.1200000000000001},"min":{"1min":0.437,"5min":0.42199999999999999,"15min":0.32000000000000001},"max":{"1min":5.4640000000000004,"5min":8.3079999999999998,"15min":12.169},"last":1.4019999999999999},{"interface":"front","average":{"1min":1.5509999999999999,"5min":1.619,"15min":2.0379999999999998},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.36599999999999999},"max":{"1min":5.4880000000000004,"5min":8.4339999999999993,"15min":12.144},"last":1.431}]}]},{"osd":0,"up_from":8,"seq":34359738599,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1920848,"kb_used_data":1592584,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":91824304,"statfs":{"total":95995035648,"available":94028087296,"internally_reserved":0,"allocated":1630806016,"data_stored":1630111584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":336132549},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 9 17:36:56 2024","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.1919999999999999,"15min":1.2809999999999999},"min":{"1min":0.22800000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":3.5030000000000001,"5min":11.236000000000001,"15min":14.166},"last":0.432},{"interface":"front","average":{"1min":2.4870000000000001,"5min":1.3380000000000001,"15min":1.5209999999999999},"min":{"1min":0.32000000000000001,"5min":0.26800000000000002,"15min":0.215},"max":{"1min":7.8579999999999997,"5min":9.5649999999999995,"15min":20.177},"last":0.66300000000000003}]},{"osd":2,"last update":"Fri Feb 9 17:27:44 2024","interfaces":[{"interface":"back","average":{"1min":3.976,"5min":3.976,"15min":3.976},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":13.066000000000001,"5min":13.066000000000001,"15min":13.066000000000001},"last":3.0569999999999999},{"interface":"front","average":{"1min":3.968,"5min":3.968,"15min":3.968},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":13.193,"5min":13.193,"15min":13.193},"last":10.872999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:36:50 2024","interfaces":[{"interface":"back","average":{"1min":2.21,"5min":1.7290000000000001,"15min":1.5489999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":6.1139999999999999,"5min":16.472000000000001,"15min":16.472000000000001},"last":3.0369999999999999},{"interface":"front","average":{"1min":2.1070000000000002,"5min":1.3919999999999999,"15min":1.399},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.223},"max":{"1min":15.132999999999999,"5min":15.132999999999999,"15min":16.984999999999999},"last":0.73699999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:36:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":1.8979999999999999,"15min":1.504},"min":{"1min":0.30299999999999999,"5min":0.29999999999999999,"15min":0.253},"max":{"1min":3.4630000000000001,"5min":14.419,"15min":14.419},"last":0.629},{"interface":"front","average":{"1min":0.56000000000000005,"5min":2.137,"15min":1.6120000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.5720000000000001,"5min":14.465999999999999,"15min":14.465999999999999},"last":0.47399999999999998}]},{"osd":5,"last update":"Fri Feb 9 17:36:40 2024","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.6539999999999999,"15min":1.5029999999999999},"min":{"1min":0.36499999999999999,"5min":0.32400000000000001,"15min":0.28999999999999998},"max":{"1min":3.5259999999999998,"5min":9.532,"15min":9.8569999999999993},"last":0.92100000000000004},{"interface":"front","average":{"1min":1.506,"5min":1.9370000000000001,"15min":1.6399999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.26700000000000002},"max":{"1min":5.7919999999999998,"5min":9.5640000000000001,"15min":9.9049999999999994},"last":0.96599999999999997}]},{"osd":6,"last update":"Fri Feb 9 17:36:13 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":2.0550000000000002,"15min":1.5840000000000001},"min":{"1min":0.34499999999999997,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.6339999999999999,"5min":13.102,"15min":13.102},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.78600000000000003,"5min":2.1299999999999999,"15min":1.583},"min":{"1min":0.38200000000000001,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":2.798,"5min":13.129,"15min":13.129},"last":0.60699999999999998}]},{"osd":7,"last update":"Fri Feb 9 17:36:30 2024","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":2.2879999999999998,"15min":1.663},"min":{"1min":0.42999999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":2.528,"5min":13.063000000000001,"15min":13.063000000000001},"last":0.54900000000000004},{"interface":"front","average":{"1min":1.141,"5min":2.0190000000000001,"15min":1.6140000000000001},"min":{"1min":0.36099999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":3.0939999999999999,"5min":12.959,"15min":16.765999999999998},"last":3.0859999999999999}]},{"osd":8,"last update":"Fri Feb 9 17:36:34 2024","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.3300000000000001,"15min":1.571},"min":{"1min":0.40999999999999998,"5min":0.36599999999999999,"15min":0.35799999999999998},"max":{"1min":3.2429999999999999,"5min":10.207000000000001,"15min":10.207000000000001},"last":3.1859999999999999},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.5429999999999999,"15min":1.552},"min":{"1min":0.40400000000000003,"5min":0.38600000000000001,"15min":0.36799999999999999},"max":{"1min":3.2719999999999998,"5min":10.231999999999999,"15min":16.41},"last":3.137}]},{"osd":9,"last update":"Fri Feb 9 17:36:20 2024","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":1.589,"15min":1.472},"min":{"1min":0.436,"5min":0.34699999999999998,"15min":0.312},"max":{"1min":2.847,"5min":13.02,"15min":13.02},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.548,"15min":1.4319999999999999},"min":{"1min":0.42099999999999999,"5min":0.373,"15min":0.36199999999999999},"max":{"1min":3.1800000000000002,"5min":10.180999999999999,"15min":16.82},"last":3.1120000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:36:40 2024","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.51,"15min":1.5089999999999999},"min":{"1min":0.45400000000000001,"5min":0.42199999999999999,"15min":0.38500000000000001},"max":{"1min":3.2639999999999998,"5min":13.888,"15min":16.876000000000001},"last":3.1589999999999998},{"interface":"front","average":{"1min":1.264,"5min":1.3580000000000001,"15min":1.4370000000000001},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.39800000000000002},"max":{"1min":3.5750000000000002,"5min":9.3350000000000009,"15min":9.7970000000000006},"last":0.88700000000000001}]},{"osd":11,"last update":"Fri Feb 9 17:36:13 2024","interfaces":[{"interface":"back","average":{"1min":3.3730000000000002,"5min":2.1080000000000001,"15min":1.603},"min":{"1min":0.53100000000000003,"5min":0.501,"15min":0.42299999999999999},"max":{"1min":40.021000000000001,"5min":40.021000000000001,"15min":40.021000000000001},"last":0.86299999999999999},{"interface":"front","average":{"1min":3.3980000000000001,"5min":2.0590000000000002,"15min":1.6950000000000001},"min":{"1min":0.55600000000000005,"5min":0.46000000000000002,"15min":0.443},"max":{"1min":40.055999999999997,"5min":40.055999999999997,"15min":40.055999999999997},"last":0.94299999999999995}]}]},{"osd":2,"up_from":18,"seq":77309411549,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1173080,"kb_used_data":1029456,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":92572072,"statfs":{"total":95995035648,"available":94793801728,"internally_reserved":0,"allocated":1054162944,"data_stored":1053464960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":1.4510000000000001,"15min":1.006},"min":{"1min":0.29999999999999999,"5min":0.23699999999999999,"15min":0.221},"max":{"1min":4.9089999999999998,"5min":17.251000000000001,"15min":17.251000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.74399999999999999,"5min":1.611,"15min":1.113},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.22900000000000001},"max":{"1min":4.7720000000000002,"5min":23.097999999999999,"15min":23.097999999999999},"last":0.875}]},{"osd":1,"last update":"Fri Feb 9 17:36:29 2024","interfaces":[{"interface":"back","average":{"1min":2.2330000000000001,"5min":1.3600000000000001,"15min":1.425},"min":{"1min":0.26100000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":9.5199999999999996,"5min":22.306000000000001,"15min":27.289000000000001},"last":0.311},{"interface":"front","average":{"1min":1.347,"5min":1.5900000000000001,"15min":1.581},"min":{"1min":0.27100000000000002,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":7.008,"5min":44.838999999999999,"15min":44.838999999999999},"last":2.6619999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:36:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.488,"15min":1.532},"min":{"1min":0.31900000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":7.5670000000000002,"5min":24.353000000000002,"15min":37.634999999999998},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.67300000000000004,"5min":1.1100000000000001,"15min":1.3140000000000001},"min":{"1min":0.28299999999999997,"5min":0.251,"15min":0.23100000000000001},"max":{"1min":1.869,"5min":9.4860000000000007,"15min":38.924999999999997},"last":1.643}]},{"osd":4,"last update":"Fri Feb 9 17:36:05 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":1.8300000000000001,"15min":1.403},"min":{"1min":0.29799999999999999,"5min":0.28399999999999997,"15min":0.254},"max":{"1min":1.9079999999999999,"5min":8.5280000000000005,"15min":8.8089999999999993},"last":1.5629999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":1.754,"15min":1.3120000000000001},"min":{"1min":0.34599999999999997,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.4199999999999999,"5min":9.6509999999999998,"15min":13.305999999999999},"last":0.33700000000000002}]},{"osd":5,"last update":"Fri Feb 9 17:36:42 2024","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.613,"15min":1.4330000000000001},"min":{"1min":0.379,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":3.7189999999999999,"5min":7.7539999999999996,"15min":8.6859999999999999},"last":0.35999999999999999},{"interface":"front","average":{"1min":2.5209999999999999,"5min":2.036,"15min":1.423},"min":{"1min":0.375,"5min":0.30199999999999999,"15min":0.29899999999999999},"max":{"1min":10.781000000000001,"5min":10.781000000000001,"15min":13.278},"last":1.605}]},{"osd":6,"last update":"Fri Feb 9 17:36:05 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":1.798,"15min":1.4570000000000001},"min":{"1min":0.38200000000000001,"5min":0.36399999999999999,"15min":0.32100000000000001},"max":{"1min":1.6399999999999999,"5min":6.931,"15min":8.8399999999999999},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.71199999999999997,"5min":2.1600000000000001,"15min":1.581},"min":{"1min":0.40300000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.456,"5min":11.269,"15min":11.269},"last":0.73999999999999999}]},{"osd":7,"last update":"Fri Feb 9 17:36:29 2024","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":1.851,"15min":1.4990000000000001},"min":{"1min":0.40999999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":3.6640000000000001,"5min":11.365,"15min":14.541},"last":0.41299999999999998},{"interface":"front","average":{"1min":1.1140000000000001,"5min":2.0830000000000002,"15min":1.7210000000000001},"min":{"1min":0.46100000000000002,"5min":0.371,"15min":0.371},"max":{"1min":4.8230000000000004,"5min":22.359999999999999,"15min":22.359999999999999},"last":0.46899999999999997}]},{"osd":8,"last update":"Fri Feb 9 17:36:05 2024","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.3540000000000001,"15min":1.208},"min":{"1min":0.437,"5min":0.38,"15min":0.38},"max":{"1min":4.8550000000000004,"5min":22.393999999999998,"15min":22.393999999999998},"last":1.5109999999999999},{"interface":"front","average":{"1min":0.76100000000000001,"5min":1.343,"15min":1.4199999999999999},"min":{"1min":0.50700000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.4910000000000001,"5min":9.9260000000000002,"15min":9.9260000000000002},"last":2.6960000000000002}]},{"osd":9,"last update":"Fri Feb 9 17:36:16 2024","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.2549999999999999,"15min":1.333},"min":{"1min":0.48199999999999998,"5min":0.371,"15min":0.371},"max":{"1min":2.7130000000000001,"5min":9.5440000000000005,"15min":9.5440000000000005},"last":6.7919999999999998},{"interface":"front","average":{"1min":1.024,"5min":1.3720000000000001,"15min":1.266},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.39800000000000002},"max":{"1min":3.8119999999999998,"5min":9.6509999999999998,"15min":13.218},"last":6.758}]},{"osd":10,"last update":"Fri Feb 9 17:36:54 2024","interfaces":[{"interface":"back","average":{"1min":1.984,"5min":1.579,"15min":1.333},"min":{"1min":0.501,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":8.2609999999999992,"5min":22.425999999999998,"15min":22.425999999999998},"last":3.8300000000000001},{"interface":"front","average":{"1min":2.0209999999999999,"5min":1.617,"15min":1.4019999999999999},"min":{"1min":0.54800000000000004,"5min":0.47099999999999997,"15min":0.42099999999999999},"max":{"1min":8.2970000000000006,"5min":22.454999999999998,"15min":22.454999999999998},"last":3.8610000000000002}]},{"osd":11,"last update":"Fri Feb 9 17:36:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.4970000000000001,"15min":1.8180000000000001},"min":{"1min":0.54400000000000004,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":3.6829999999999998,"5min":6.7709999999999999,"15min":13.186},"last":3.9289999999999998},{"interface":"front","average":{"1min":0.97499999999999998,"5min":1.536,"15min":1.8240000000000001},"min":{"1min":0.48799999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":2.1970000000000001,"5min":9.5730000000000004,"15min":9.5730000000000004},"last":4.0659999999999998}]}]},{"osd":3,"up_from":27,"seq":115964117209,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2820824,"kb_used_data":2496720,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":90924328,"statfs":{"total":95995035648,"available":93106511872,"internally_reserved":0,"allocated":2556641280,"data_stored":2555943296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":331872711},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:35:59 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":1.571,"15min":1.7030000000000001},"min":{"1min":0.23599999999999999,"5min":0.224,"15min":0.224},"max":{"1min":4.2000000000000002,"5min":7.5110000000000001,"15min":7.5110000000000001},"last":2.4380000000000002},{"interface":"front","average":{"1min":0.79000000000000004,"5min":1.6180000000000001,"15min":1.756},"min":{"1min":0.29499999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":3.754,"5min":15.167999999999999,"15min":15.167999999999999},"last":0.42199999999999999}]},{"osd":1,"last update":"Fri Feb 9 17:33:41 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":1.1399999999999999,"15min":1.3939999999999999},"min":{"1min":0.26100000000000001,"5min":0.23400000000000001,"15min":0.23300000000000001},"max":{"1min":3.8759999999999999,"5min":17.384,"15min":19.638999999999999},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":1.0249999999999999,"15min":1.115},"min":{"1min":0.27500000000000002,"5min":0.22500000000000001,"15min":0.20399999999999999},"max":{"1min":3.8980000000000001,"5min":12.664999999999999,"15min":14.661},"last":0.314}]},{"osd":2,"last update":"Fri Feb 9 17:36:50 2024","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.2849999999999999,"15min":1.4079999999999999},"min":{"1min":0.33500000000000002,"5min":0.22600000000000001,"15min":0.224},"max":{"1min":4.2670000000000003,"5min":14.888999999999999,"15min":14.914999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.583,"15min":1.4299999999999999},"min":{"1min":0.249,"5min":0.22500000000000001,"15min":0.218},"max":{"1min":6.9139999999999997,"5min":17.613,"15min":17.613},"last":4.4859999999999998}]},{"osd":4,"last update":"Fri Feb 9 17:36:04 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":2.0270000000000001,"15min":1.661},"min":{"1min":0.33400000000000002,"5min":0.28799999999999998,"15min":0.254},"max":{"1min":0.99199999999999999,"5min":34.898000000000003,"15min":34.898000000000003},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.622,"5min":1.925,"15min":1.5029999999999999},"min":{"1min":0.38700000000000001,"5min":0.30599999999999999,"15min":0.27900000000000003},"max":{"1min":1.1899999999999999,"5min":14.061999999999999,"15min":14.061999999999999},"last":0.65700000000000003}]},{"osd":5,"last update":"Fri Feb 9 17:36:35 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":1.8100000000000001,"15min":1.3180000000000001},"min":{"1min":0.36799999999999999,"5min":0.27800000000000002,"15min":0.27300000000000002},"max":{"1min":1.77,"5min":17.651,"15min":17.651},"last":4.391},{"interface":"front","average":{"1min":0.86199999999999999,"5min":1.853,"15min":1.3580000000000001},"min":{"1min":0.38900000000000001,"5min":0.34100000000000003,"15min":0.33300000000000002},"max":{"1min":1.8080000000000001,"5min":17.675999999999998,"15min":17.675999999999998},"last":4.4210000000000003}]},{"osd":6,"last update":"Fri Feb 9 17:36:04 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":1.873,"15min":1.474},"min":{"1min":0.33200000000000002,"5min":0.32900000000000001,"15min":0.32200000000000001},"max":{"1min":1.4670000000000001,"5min":20.262,"15min":20.262},"last":10.394},{"interface":"front","average":{"1min":0.71399999999999997,"5min":1.6950000000000001,"15min":1.216},"min":{"1min":0.38,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":2.9470000000000001,"5min":7.9009999999999998,"15min":7.9009999999999998},"last":0.51800000000000002}]},{"osd":7,"last update":"Fri Feb 9 17:36:35 2024","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":2.4209999999999998,"15min":1.9179999999999999},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.27200000000000002},"max":{"1min":4.9820000000000002,"5min":21.018999999999998,"15min":21.018999999999998},"last":2.7280000000000002},{"interface":"front","average":{"1min":1.236,"5min":1.9950000000000001,"15min":1.6259999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.37},"max":{"1min":4.7329999999999997,"5min":14.186,"15min":19.550999999999998},"last":0.58199999999999996}]},{"osd":8,"last update":"Fri Feb 9 17:36:04 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":1.329,"15min":1.323},"min":{"1min":0.46700000000000003,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":3.0310000000000001,"5min":7.7569999999999997,"15min":7.7569999999999997},"last":2.1339999999999999},{"interface":"front","average":{"1min":0.95799999999999996,"5min":1.046,"15min":0.82699999999999996},"min":{"1min":0.49199999999999999,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":3.9350000000000001,"5min":14.129,"15min":14.129},"last":0.54800000000000004}]},{"osd":9,"last update":"Fri Feb 9 17:36:19 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":1.5189999999999999,"15min":1.2589999999999999},"min":{"1min":0.371,"5min":0.36299999999999999,"15min":0.35499999999999998},"max":{"1min":2.173,"5min":7.7370000000000001,"15min":7.7370000000000001},"last":2.0990000000000002},{"interface":"front","average":{"1min":0.89500000000000002,"5min":1.3049999999999999,"15min":1.516},"min":{"1min":0.438,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":2.9649999999999999,"5min":12.598000000000001,"15min":13.717000000000001},"last":2.7469999999999999}]},{"osd":10,"last update":"Fri Feb 9 17:36:50 2024","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.355,"15min":1.431},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":2.1429999999999998,"5min":7.7889999999999997,"15min":19.611000000000001},"last":2.1680000000000001},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.401,"15min":1.3200000000000001},"min":{"1min":0.41999999999999998,"5min":0.38,"15min":0.38},"max":{"1min":6.8179999999999996,"5min":12.638,"15min":12.638},"last":2.798}]},{"osd":11,"last update":"Fri Feb 9 17:36:04 2024","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":1.585,"15min":1.331},"min":{"1min":0.42499999999999999,"5min":0.40400000000000003,"15min":0.38800000000000001},"max":{"1min":2.52,"5min":12.69,"15min":12.69},"last":2.6880000000000002},{"interface":"front","average":{"1min":0.995,"5min":1.464,"15min":1.5640000000000001},"min":{"1min":0.501,"5min":0.42499999999999999,"15min":0.36899999999999999},"max":{"1min":3.9079999999999999,"5min":14.157999999999999,"15min":14.157999999999999},"last":0.63100000000000001}]}]},{"osd":4,"up_from":32,"seq":137438953683,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2404880,"kb_used_data":2080776,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":91340272,"statfs":{"total":95995035648,"available":93532438528,"internally_reserved":0,"allocated":2130714624,"data_stored":2130020192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":331872709},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:36:02 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":2.6779999999999999,"15min":1.4079999999999999},"min":{"1min":0.35499999999999998,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":2.5489999999999999,"5min":50.356999999999999,"15min":50.356999999999999},"last":2.7589999999999999},{"interface":"front","average":{"1min":1.851,"5min":2.2250000000000001,"15min":1.321},"min":{"1min":0.29699999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":15.731999999999999,"5min":50.569000000000003,"15min":50.569000000000003},"last":7.0190000000000001}]},{"osd":1,"last update":"Fri Feb 9 17:36:11 2024","interfaces":[{"interface":"back","average":{"1min":1.9490000000000001,"5min":2.2730000000000001,"15min":1.673},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.246},"max":{"1min":18.353999999999999,"5min":50.473999999999997,"15min":50.473999999999997},"last":7.5099999999999998},{"interface":"front","average":{"1min":2.0489999999999999,"5min":2.532,"15min":1.712},"min":{"1min":0.376,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":18.385999999999999,"5min":50.499000000000002,"15min":50.499000000000002},"last":7.5350000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:28:51 2024","interfaces":[{"interface":"back","average":{"1min":2.536,"5min":1.546,"15min":0.91400000000000003},"min":{"1min":0.29999999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":8.8819999999999997,"5min":8.8819999999999997,"15min":8.8819999999999997},"last":0.622},{"interface":"front","average":{"1min":2.214,"5min":1.7330000000000001,"15min":0.97799999999999998},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":9.5039999999999996,"5min":9.5039999999999996,"15min":9.5039999999999996},"last":0.53800000000000003}]},{"osd":3,"last update":"Fri Feb 9 17:36:11 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":2.5920000000000001,"15min":1.8},"min":{"1min":0.41199999999999998,"5min":0.25,"15min":0.25},"max":{"1min":4.1479999999999997,"5min":50.462000000000003,"15min":50.462000000000003},"last":7.1820000000000004},{"interface":"front","average":{"1min":0.96899999999999997,"5min":2.1499999999999999,"15min":1.6180000000000001},"min":{"1min":0.34300000000000003,"5min":0.255,"15min":0.255},"max":{"1min":2.0070000000000001,"5min":50.529000000000003,"15min":50.529000000000003},"last":4.1260000000000003}]},{"osd":5,"last update":"Fri Feb 9 17:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":1.915,"15min":1.1639999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.222},"max":{"1min":2.0339999999999998,"5min":9.5630000000000006,"15min":9.5630000000000006},"last":1.629},{"interface":"front","average":{"1min":0.79200000000000004,"5min":2.5579999999999998,"15min":1.4550000000000001},"min":{"1min":0.314,"5min":0.314,"15min":0.29199999999999998},"max":{"1min":2.073,"5min":50.509999999999998,"15min":50.509999999999998},"last":1.7090000000000001}]},{"osd":6,"last update":"Fri Feb 9 17:36:11 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":1.9570000000000001,"15min":1.2190000000000001},"min":{"1min":0.39600000000000002,"5min":0.26200000000000001,"15min":0.247},"max":{"1min":3.2250000000000001,"5min":18.09,"15min":18.09},"last":4.2220000000000004},{"interface":"front","average":{"1min":1.028,"5min":2.6989999999999998,"15min":1.415},"min":{"1min":0.40400000000000003,"5min":0.33600000000000002,"15min":0.28000000000000003},"max":{"1min":4.0060000000000002,"5min":50.402999999999999,"15min":50.402999999999999},"last":0.34100000000000003}]},{"osd":7,"last update":"Fri Feb 9 17:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":2.25,"15min":1.4450000000000001},"min":{"1min":0.40699999999999997,"5min":0.33800000000000002,"15min":0.28399999999999997},"max":{"1min":3.266,"5min":14.919,"15min":15.223000000000001},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.99399999999999999,"5min":2.169,"15min":1.4099999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.28100000000000003},"max":{"1min":4.0419999999999998,"5min":9.343,"15min":19.047000000000001},"last":1.698}]},{"osd":8,"last update":"Fri Feb 9 17:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":1.869,"15min":1.4630000000000001},"min":{"1min":0.36199999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":2.2000000000000002,"5min":8.4160000000000004,"15min":8.4160000000000004},"last":0.496},{"interface":"front","average":{"1min":0.80800000000000005,"5min":2.3839999999999999,"15min":1.5249999999999999},"min":{"1min":0.40999999999999998,"5min":0.34699999999999998,"15min":0.308},"max":{"1min":1.4830000000000001,"5min":50.594000000000001,"15min":50.594000000000001},"last":0.56399999999999995}]},{"osd":9,"last update":"Fri Feb 9 17:36:16 2024","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":2.468,"15min":1.53},"min":{"1min":0.436,"5min":0.39200000000000002,"15min":0.34599999999999997},"max":{"1min":2.319,"5min":50.442999999999998,"15min":50.442999999999998},"last":0.44900000000000001},{"interface":"front","average":{"1min":1.1339999999999999,"5min":2.0049999999999999,"15min":1.3049999999999999},"min":{"1min":0.52900000000000003,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":4.0709999999999997,"5min":10.555,"15min":10.555},"last":1.6739999999999999}]},{"osd":10,"last update":"Fri Feb 9 17:36:53 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":2.6680000000000001,"15min":1.649},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.378},"max":{"1min":3.137,"5min":51.223999999999997,"15min":51.223999999999997},"last":0.59399999999999997},{"interface":"front","average":{"1min":1.294,"5min":2.9430000000000001,"15min":1.7130000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.38100000000000001},"max":{"1min":5.117,"5min":50.552,"15min":50.552},"last":0.67500000000000004}]},{"osd":11,"last update":"Fri Feb 9 17:36:16 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":2.899,"15min":1.8839999999999999},"min":{"1min":0.48599999999999999,"5min":0.443,"15min":0.40500000000000003},"max":{"1min":3.5139999999999998,"5min":50.484999999999999,"15min":50.484999999999999},"last":0.61799999999999999},{"interface":"front","average":{"1min":1.403,"5min":2.2829999999999999,"15min":1.548},"min":{"1min":0.55700000000000005,"5min":0.42499999999999999,"15min":0.34899999999999998},"max":{"1min":5.5599999999999996,"5min":18.574999999999999,"15min":18.574999999999999},"last":1.6479999999999999}]}]},{"osd":5,"up_from":40,"seq":171798692045,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1201944,"kb_used_data":1018064,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":92543208,"statfs":{"total":95995035648,"available":94764244992,"internally_reserved":0,"allocated":1042497536,"data_stored":1041799552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:32:25 2024","interfaces":[{"interface":"back","average":{"1min":1.8540000000000001,"5min":1.653,"15min":1.0900000000000001},"min":{"1min":0.318,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":7.7160000000000002,"5min":8.7690000000000001,"15min":11.923},"last":3.8450000000000002},{"interface":"front","average":{"1min":2.4089999999999998,"5min":1.8260000000000001,"15min":1.032},"min":{"1min":0.34499999999999997,"5min":0.24299999999999999,"15min":0.224},"max":{"1min":14.082000000000001,"5min":14.082000000000001,"15min":14.082000000000001},"last":0.79400000000000004}]},{"osd":1,"last update":"Fri Feb 9 17:36:32 2024","interfaces":[{"interface":"back","average":{"1min":1.502,"5min":2.1920000000000002,"15min":1.758},"min":{"1min":0.308,"5min":0.23899999999999999,"15min":0.23200000000000001},"max":{"1min":9.2690000000000001,"5min":14.032999999999999,"15min":29.715},"last":3.2269999999999999},{"interface":"front","average":{"1min":1.359,"5min":1.7969999999999999,"15min":1.597},"min":{"1min":0.26000000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":8.6920000000000002,"5min":11.148999999999999,"15min":29.969999999999999},"last":0.32900000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:36:32 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":1.496,"15min":1.425},"min":{"1min":0.28799999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.7729999999999999,"5min":10.5,"15min":10.5},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.71699999999999997,"5min":1.907,"15min":1.5509999999999999},"min":{"1min":0.371,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":2.472,"5min":14.122,"15min":14.122},"last":0.58799999999999997}]},{"osd":3,"last update":"Fri Feb 9 17:36:32 2024","interfaces":[{"interface":"back","average":{"1min":4.1070000000000002,"5min":2.7250000000000001,"15min":1.8400000000000001},"min":{"1min":0.33900000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":64.150999999999996,"5min":64.150999999999996,"15min":64.150999999999996},"last":0.437},{"interface":"front","average":{"1min":4.2119999999999997,"5min":2.8839999999999999,"15min":1.8080000000000001},"min":{"1min":0.41399999999999998,"5min":0.30499999999999999,"15min":0.28000000000000003},"max":{"1min":64.176000000000002,"5min":64.176000000000002,"15min":64.176000000000002},"last":1.1830000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:36:41 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":1.863,"15min":1.139},"min":{"1min":0.28699999999999998,"5min":0.26200000000000001,"15min":0.23400000000000001},"max":{"1min":2.27,"5min":11.036,"15min":11.036},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.81299999999999994,"5min":1.7689999999999999,"15min":1.202},"min":{"1min":0.28599999999999998,"5min":0.28499999999999998,"15min":0.27900000000000003},"max":{"1min":3.7559999999999998,"5min":13.59,"15min":13.59},"last":0.69299999999999995}]},{"osd":6,"last update":"Fri Feb 9 17:36:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":2.0979999999999999,"15min":1.2330000000000001},"min":{"1min":0.308,"5min":0.253,"15min":0.253},"max":{"1min":0.99299999999999999,"5min":15.375999999999999,"15min":15.375999999999999},"last":0.624},{"interface":"front","average":{"1min":0.82999999999999996,"5min":2.2240000000000002,"15min":1.286},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.27200000000000002},"max":{"1min":3.718,"5min":14.250999999999999,"15min":14.250999999999999},"last":0.46700000000000003}]},{"osd":7,"last update":"Fri Feb 9 17:36:32 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":1.778,"15min":1.2370000000000001},"min":{"1min":0.29399999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.726,"5min":9.7140000000000004,"15min":18.626999999999999},"last":1.988},{"interface":"front","average":{"1min":0.93700000000000006,"5min":2.1110000000000002,"15min":1.452},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.28799999999999998},"max":{"1min":3.0030000000000001,"5min":19.044,"15min":19.044},"last":0.81000000000000005}]},{"osd":8,"last update":"Fri Feb 9 17:36:07 2024","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":1.579,"15min":1.7290000000000001},"min":{"1min":0.53900000000000003,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":3.569,"5min":9.3710000000000004,"15min":9.3710000000000004},"last":1.8320000000000001},{"interface":"front","average":{"1min":0.78100000000000003,"5min":2.02,"15min":2.2229999999999999},"min":{"1min":0.47799999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.1220000000000001,"5min":9.6240000000000006,"15min":9.6240000000000006},"last":0.64400000000000002}]},{"osd":9,"last update":"Fri Feb 9 17:36:13 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":1.802,"15min":1.246},"min":{"1min":0.48899999999999999,"5min":0.378,"15min":0.35999999999999999},"max":{"1min":1.5509999999999999,"5min":14.98,"15min":14.98},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.78900000000000003,"5min":1.873,"15min":1.2669999999999999},"min":{"1min":0.47799999999999998,"5min":0.36699999999999999,"15min":0.312},"max":{"1min":1.141,"5min":9.4239999999999995,"15min":9.4239999999999995},"last":2.0720000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:36:41 2024","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":1.9930000000000001,"15min":1.4299999999999999},"min":{"1min":0.48599999999999999,"5min":0.47499999999999998,"15min":0.33800000000000002},"max":{"1min":2.581,"5min":14.218,"15min":14.218},"last":0.66000000000000003},{"interface":"front","average":{"1min":1.01,"5min":2.016,"15min":1.48},"min":{"1min":0.497,"5min":0.45000000000000001,"15min":0.36099999999999999},"max":{"1min":3.3340000000000001,"5min":14.186999999999999,"15min":14.186999999999999},"last":2.2170000000000001}]},{"osd":11,"last update":"Fri Feb 9 17:36:13 2024","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":2.1110000000000002,"15min":1.508},"min":{"1min":0.51900000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":2.5449999999999999,"5min":9.4239999999999995,"15min":9.4239999999999995},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.86399999999999999,"5min":2.4169999999999998,"15min":1.444},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.38900000000000001},"max":{"1min":1.9119999999999999,"5min":14.316000000000001,"15min":14.316000000000001},"last":2.0409999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1578696704,"data_stored":1578696704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4750639104,"data_stored":4750639104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1057161216,"data_stored":1057161216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2572746752,"data_stored":2572746752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2134507520,"data_stored":2134507520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1043529728,"data_stored":1043529728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3050962944,"data_stored":3050962944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1543634944,"data_stored":1543634944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1534001152,"data_stored":1534001152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2060320768,"data_stored":2060320768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":3678404608,"data_stored":3678404608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-09T17:37:03.274 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph pg dump --format=json 2024-02-09T17:37:03.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:03.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:37:03.381 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:37:03.449 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:03.539 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:37:03.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:03.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:37:03.649 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:37:03.681 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:03.700 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:37:03.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:03.704 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:37:03.726 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 84 16 132279 132263 98.3891 54.375 0.0212914 0.0101601 2024-02-09T17:37:03.805 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:37:03.838 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:03.847 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:03 smithi099 ceph-mon[39420]: 4.b scrub starts 2024-02-09T17:37:03.847 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:03 smithi099 ceph-mon[39420]: pgmap v964: 37 pgs: 37 active+clean; 7.9 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-02-09T17:37:03.847 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:03 smithi099 ceph-mon[39420]: 1.5 scrub starts 2024-02-09T17:37:03.847 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:03 smithi099 ceph-mon[39420]: 1.5 scrub ok 2024-02-09T17:37:03.847 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:03 smithi099 ceph-mon[39420]: 1.14 scrub starts 2024-02-09T17:37:03.863 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:37:03.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:03.866 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:37:03.967 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:37:04.040 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:04.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:03 smithi179 ceph-mon[43975]: 4.b scrub starts 2024-02-09T17:37:04.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:03 smithi179 ceph-mon[43975]: pgmap v964: 37 pgs: 37 active+clean; 7.9 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-02-09T17:37:04.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:03 smithi179 ceph-mon[43975]: 1.5 scrub starts 2024-02-09T17:37:04.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:03 smithi179 ceph-mon[43975]: 1.5 scrub ok 2024-02-09T17:37:04.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:03 smithi179 ceph-mon[43975]: 1.14 scrub starts 2024-02-09T17:37:04.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:03 smithi162 ceph-mon[44304]: 4.b scrub starts 2024-02-09T17:37:04.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:03 smithi162 ceph-mon[44304]: pgmap v964: 37 pgs: 37 active+clean; 7.9 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-02-09T17:37:04.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:03 smithi162 ceph-mon[44304]: 1.5 scrub starts 2024-02-09T17:37:04.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:03 smithi162 ceph-mon[44304]: 1.5 scrub ok 2024-02-09T17:37:04.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:03 smithi162 ceph-mon[44304]: 1.14 scrub starts 2024-02-09T17:37:04.726 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 85 16 133172 133156 97.8882 55.8125 0.0103251 0.0102111 2024-02-09T17:37:04.804 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-09T17:37:05.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:04 smithi179 ceph-mon[43975]: 1.1 scrub starts 2024-02-09T17:37:05.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:04 smithi179 ceph-mon[43975]: 1.14 scrub ok 2024-02-09T17:37:05.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:04 smithi179 ceph-mon[43975]: 1.1 scrub ok 2024-02-09T17:37:05.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:04 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:37:05.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:04 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:37:05.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:04 smithi162 ceph-mon[44304]: 1.1 scrub starts 2024-02-09T17:37:05.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:04 smithi162 ceph-mon[44304]: 1.14 scrub ok 2024-02-09T17:37:05.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:04 smithi162 ceph-mon[44304]: 1.1 scrub ok 2024-02-09T17:37:05.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:04 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:37:05.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:04 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:37:05.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:04 smithi099 ceph-mon[39420]: 1.1 scrub starts 2024-02-09T17:37:05.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:04 smithi099 ceph-mon[39420]: 1.14 scrub ok 2024-02-09T17:37:05.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:04 smithi099 ceph-mon[39420]: 1.1 scrub ok 2024-02-09T17:37:05.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:04 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:37:05.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:04 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:37:05.608 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:37:05.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:05.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:37:05.714 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:37:05.726 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 86 16 134060 134044 97.3953 55.5 0.00663694 0.0102633 2024-02-09T17:37:05.765 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:05.795 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:37:05.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:05.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:37:05.900 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:37:05.944 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:05 smithi099 ceph-mon[39420]: 4.1 scrub starts 2024-02-09T17:37:05.944 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:05 smithi099 ceph-mon[39420]: 4.7 scrub starts 2024-02-09T17:37:05.944 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:05 smithi099 ceph-mon[39420]: pgmap v965: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 7.9 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 46 MiB/s wr, 729 op/s 2024-02-09T17:37:05.944 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:05 smithi099 ceph-mon[39420]: osdmap e356: 12 total, 12 up, 12 in 2024-02-09T17:37:05.945 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:05 smithi099 ceph-mon[39420]: 4.b scrub ok 2024-02-09T17:37:05.951 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:06.116 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:37:06.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:06.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:37:06.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:05 smithi179 ceph-mon[43975]: 4.1 scrub starts 2024-02-09T17:37:06.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:05 smithi179 ceph-mon[43975]: 4.7 scrub starts 2024-02-09T17:37:06.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:05 smithi179 ceph-mon[43975]: pgmap v965: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 7.9 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 46 MiB/s wr, 729 op/s 2024-02-09T17:37:06.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:05 smithi179 ceph-mon[43975]: osdmap e356: 12 total, 12 up, 12 in 2024-02-09T17:37:06.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:05 smithi179 ceph-mon[43975]: 4.b scrub ok 2024-02-09T17:37:06.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:05 smithi162 ceph-mon[44304]: 4.1 scrub starts 2024-02-09T17:37:06.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:05 smithi162 ceph-mon[44304]: 4.7 scrub starts 2024-02-09T17:37:06.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:05 smithi162 ceph-mon[44304]: pgmap v965: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 7.9 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 46 MiB/s wr, 729 op/s 2024-02-09T17:37:06.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:05 smithi162 ceph-mon[44304]: osdmap e356: 12 total, 12 up, 12 in 2024-02-09T17:37:06.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:05 smithi162 ceph-mon[44304]: 4.b scrub ok 2024-02-09T17:37:06.221 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:37:06.256 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:06.282 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:37:06.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:06.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:37:06.387 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:37:06.420 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:06.447 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:37:06.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:06.450 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:37:06.551 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:37:06.619 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:06.728 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 87 16 134982 134966 96.9367 57.625 0.017164 0.0103127 2024-02-09T17:37:06.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:06 smithi099 ceph-mon[39420]: 1.b scrub starts 2024-02-09T17:37:06.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:06 smithi099 ceph-mon[39420]: 1.b scrub ok 2024-02-09T17:37:07.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:06 smithi179 ceph-mon[43975]: 1.b scrub starts 2024-02-09T17:37:07.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:06 smithi179 ceph-mon[43975]: 1.b scrub ok 2024-02-09T17:37:07.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:06 smithi162 ceph-mon[44304]: 1.b scrub starts 2024-02-09T17:37:07.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:06 smithi162 ceph-mon[44304]: 1.b scrub ok 2024-02-09T17:37:07.391 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:37:07.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:07.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:37:07.496 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:37:07.528 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:07.551 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:37:07.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:07.559 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:37:07.660 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:37:07.710 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:07.728 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 88 16 135861 135845 96.4595 54.9375 0.0102631 0.0103629 2024-02-09T17:37:07.746 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:37:07.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:07.752 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:37:07.799 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:07 smithi179 ceph-mon[43975]: 4.1 scrub ok 2024-02-09T17:37:07.799 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:07 smithi179 ceph-mon[43975]: 1.e scrub starts 2024-02-09T17:37:07.799 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:07 smithi179 ceph-mon[43975]: 1.e scrub ok 2024-02-09T17:37:07.799 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:07 smithi179 ceph-mon[43975]: 1.7 scrub starts 2024-02-09T17:37:07.799 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:07 smithi179 ceph-mon[43975]: 1.7 scrub ok 2024-02-09T17:37:07.799 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:07 smithi179 ceph-mon[43975]: pgmap v967: 37 pgs: 3 active+clean+scrubbing, 34 active+clean; 8.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1k op/s 2024-02-09T17:37:07.800 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:07 smithi179 ceph-mon[43975]: 1.c scrub starts 2024-02-09T17:37:07.800 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:07 smithi179 ceph-mon[43975]: 4.6 scrub starts 2024-02-09T17:37:07.800 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:07 smithi179 ceph-mon[43975]: 1.c scrub ok 2024-02-09T17:37:07.852 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:37:07.884 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:07.908 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:37:07.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:07.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:37:08.015 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:37:08.049 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:08.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:07 smithi162 ceph-mon[44304]: 4.1 scrub ok 2024-02-09T17:37:08.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:07 smithi162 ceph-mon[44304]: 1.e scrub starts 2024-02-09T17:37:08.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:07 smithi162 ceph-mon[44304]: 1.e scrub ok 2024-02-09T17:37:08.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:07 smithi162 ceph-mon[44304]: 1.7 scrub starts 2024-02-09T17:37:08.053 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:07 smithi162 ceph-mon[44304]: 1.7 scrub ok 2024-02-09T17:37:08.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:07 smithi162 ceph-mon[44304]: pgmap v967: 37 pgs: 3 active+clean+scrubbing, 34 active+clean; 8.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1k op/s 2024-02-09T17:37:08.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:07 smithi162 ceph-mon[44304]: 1.c scrub starts 2024-02-09T17:37:08.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:07 smithi162 ceph-mon[44304]: 4.6 scrub starts 2024-02-09T17:37:08.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:07 smithi162 ceph-mon[44304]: 1.c scrub ok 2024-02-09T17:37:08.080 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:37:08.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:08.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:37:08.185 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:37:08.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:07 smithi099 ceph-mon[39420]: 4.1 scrub ok 2024-02-09T17:37:08.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:07 smithi099 ceph-mon[39420]: 1.e scrub starts 2024-02-09T17:37:08.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:07 smithi099 ceph-mon[39420]: 1.e scrub ok 2024-02-09T17:37:08.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:07 smithi099 ceph-mon[39420]: 1.7 scrub starts 2024-02-09T17:37:08.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:07 smithi099 ceph-mon[39420]: 1.7 scrub ok 2024-02-09T17:37:08.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:07 smithi099 ceph-mon[39420]: pgmap v967: 37 pgs: 3 active+clean+scrubbing, 34 active+clean; 8.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1k op/s 2024-02-09T17:37:08.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:07 smithi099 ceph-mon[39420]: 1.c scrub starts 2024-02-09T17:37:08.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:07 smithi099 ceph-mon[39420]: 4.6 scrub starts 2024-02-09T17:37:08.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:07 smithi099 ceph-mon[39420]: 1.c scrub ok 2024-02-09T17:37:08.253 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:08.274 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:37:08.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:08.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:37:08.383 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:37:08.433 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:08.469 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:37:08.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:08.472 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:37:08.573 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:37:08.623 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:08.665 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:37:08.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:08.670 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:37:08.714 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:37:08.727 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 89 15 136714 136699 95.9754 53.375 0.0272806 0.0104149 2024-02-09T17:37:08.771 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:37:08.817 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:08.817 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:08 smithi179 ceph-mon[43975]: 1.0 scrub starts 2024-02-09T17:37:08.817 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:08 smithi179 ceph-mon[43975]: 1.0 scrub ok 2024-02-09T17:37:08.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:08 smithi099 ceph-mon[39420]: 1.0 scrub starts 2024-02-09T17:37:08.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:08 smithi099 ceph-mon[39420]: 1.0 scrub ok 2024-02-09T17:37:08.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:08 smithi099 ceph-mon[39420]: 4.a scrub starts 2024-02-09T17:37:08.998 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:08 smithi099 ceph-mon[39420]: 1.6 scrub starts 2024-02-09T17:37:08.998 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:08 smithi099 ceph-mon[39420]: 4.7 scrub ok 2024-02-09T17:37:08.998 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:08 smithi099 ceph-mon[39420]: 1.6 scrub ok 2024-02-09T17:37:08.998 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:08 smithi099 ceph-mon[39420]: 1.12 scrub starts 2024-02-09T17:37:08.998 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:08 smithi099 ceph-mon[39420]: 1.12 scrub ok 2024-02-09T17:37:08.998 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:08 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:37:08.998 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:08 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:37:08.998 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:08 smithi099 ceph-mon[39420]: 4.2 scrub starts 2024-02-09T17:37:09.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:08 smithi179 ceph-mon[43975]: 4.a scrub starts 2024-02-09T17:37:09.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:08 smithi179 ceph-mon[43975]: 1.6 scrub starts 2024-02-09T17:37:09.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:08 smithi179 ceph-mon[43975]: 4.7 scrub ok 2024-02-09T17:37:09.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:08 smithi179 ceph-mon[43975]: 1.6 scrub ok 2024-02-09T17:37:09.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:08 smithi179 ceph-mon[43975]: 1.12 scrub starts 2024-02-09T17:37:09.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:08 smithi179 ceph-mon[43975]: 1.12 scrub ok 2024-02-09T17:37:09.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:08 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:37:09.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:08 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:37:09.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:08 smithi179 ceph-mon[43975]: 4.2 scrub starts 2024-02-09T17:37:09.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:08 smithi162 ceph-mon[44304]: 1.0 scrub starts 2024-02-09T17:37:09.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:08 smithi162 ceph-mon[44304]: 1.0 scrub ok 2024-02-09T17:37:09.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:08 smithi162 ceph-mon[44304]: 4.a scrub starts 2024-02-09T17:37:09.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:08 smithi162 ceph-mon[44304]: 1.6 scrub starts 2024-02-09T17:37:09.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:08 smithi162 ceph-mon[44304]: 4.7 scrub ok 2024-02-09T17:37:09.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:08 smithi162 ceph-mon[44304]: 1.6 scrub ok 2024-02-09T17:37:09.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:08 smithi162 ceph-mon[44304]: 1.12 scrub starts 2024-02-09T17:37:09.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:08 smithi162 ceph-mon[44304]: 1.12 scrub ok 2024-02-09T17:37:09.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:08 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:37:09.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:08 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:37:09.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:08 smithi162 ceph-mon[44304]: 4.2 scrub starts 2024-02-09T17:37:09.728 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Total time run: 90.0149 2024-02-09T17:37:09.728 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Total writes made: 137463 2024-02-09T17:37:09.728 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Write size: 65536 2024-02-09T17:37:09.728 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Object size: 65536 2024-02-09T17:37:09.728 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Bandwidth (MB/sec): 95.4446 2024-02-09T17:37:09.728 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Stddev Bandwidth: 44.3985 2024-02-09T17:37:09.728 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Max bandwidth (MB/sec): 282.625 2024-02-09T17:37:09.729 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Min bandwidth (MB/sec): 53.125 2024-02-09T17:37:09.729 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Average IOPS: 1527 2024-02-09T17:37:09.729 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Stddev IOPS: 710.375 2024-02-09T17:37:09.729 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Max IOPS: 4522 2024-02-09T17:37:09.729 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Min IOPS: 850 2024-02-09T17:37:09.729 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Average Latency(s): 0.0104738 2024-02-09T17:37:09.729 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Stddev Latency(s): 0.0117078 2024-02-09T17:37:09.729 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Max latency(s): 0.234475 2024-02-09T17:37:09.729 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Min latency(s): 0.00117111 2024-02-09T17:37:09.734 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Cleaning up (deleting benchmark objects) 2024-02-09T17:37:09.775 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:37:09.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:09.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:37:09.881 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:37:09.912 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:09.936 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:37:09.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:09.941 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:37:10.005 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:37:10.005 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:37:10.041 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:37:10.072 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:09 smithi179 ceph-mon[43975]: 4.5 scrub starts 2024-02-09T17:37:10.072 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:09 smithi179 ceph-mon[43975]: pgmap v968: 37 pgs: 2 active+clean+scrubbing, 35 active+clean; 8.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 824 op/s 2024-02-09T17:37:10.072 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:09 smithi179 ceph-mon[43975]: osdmap e357: 12 total, 12 up, 12 in 2024-02-09T17:37:10.072 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:09 smithi179 ceph-mon[43975]: 4.6 scrub ok 2024-02-09T17:37:10.074 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:10.096 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:37:10.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:10.101 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:37:10.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:09 smithi162 ceph-mon[44304]: 4.5 scrub starts 2024-02-09T17:37:10.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:09 smithi162 ceph-mon[44304]: pgmap v968: 37 pgs: 2 active+clean+scrubbing, 35 active+clean; 8.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 824 op/s 2024-02-09T17:37:10.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:09 smithi162 ceph-mon[44304]: osdmap e357: 12 total, 12 up, 12 in 2024-02-09T17:37:10.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:09 smithi162 ceph-mon[44304]: 4.6 scrub ok 2024-02-09T17:37:10.202 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:37:10.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:09 smithi099 ceph-mon[39420]: 4.5 scrub starts 2024-02-09T17:37:10.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:09 smithi099 ceph-mon[39420]: pgmap v968: 37 pgs: 2 active+clean+scrubbing, 35 active+clean; 8.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 824 op/s 2024-02-09T17:37:10.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:09 smithi099 ceph-mon[39420]: osdmap e357: 12 total, 12 up, 12 in 2024-02-09T17:37:10.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:09 smithi099 ceph-mon[39420]: 4.6 scrub ok 2024-02-09T17:37:10.248 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:10.272 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:37:10.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:10.276 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:37:10.377 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:37:10.420 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:10.450 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:37:10.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:10.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:37:10.553 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:37:10.597 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:10.630 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:37:10.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:10.633 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:37:10.734 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:37:10.789 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:10.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:10 smithi179 ceph-mon[43975]: 4.a scrub ok 2024-02-09T17:37:10.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:10 smithi179 ceph-mon[43975]: 1.d scrub starts 2024-02-09T17:37:10.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:10 smithi179 ceph-mon[43975]: 1.d scrub ok 2024-02-09T17:37:10.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:10 smithi179 ceph-mon[43975]: 4.4 scrub starts 2024-02-09T17:37:10.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:10 smithi179 ceph-mon[43975]: 4.2 scrub ok 2024-02-09T17:37:10.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:10 smithi179 ceph-mon[43975]: 4.5 scrub ok 2024-02-09T17:37:11.063 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:37:11.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:11.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:37:11.073 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:10 smithi099 ceph-mon[39420]: 4.a scrub ok 2024-02-09T17:37:11.073 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:10 smithi099 ceph-mon[39420]: 1.d scrub starts 2024-02-09T17:37:11.074 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:10 smithi099 ceph-mon[39420]: 1.d scrub ok 2024-02-09T17:37:11.074 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:10 smithi099 ceph-mon[39420]: 4.4 scrub starts 2024-02-09T17:37:11.074 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:10 smithi099 ceph-mon[39420]: 4.2 scrub ok 2024-02-09T17:37:11.074 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:10 smithi099 ceph-mon[39420]: 4.5 scrub ok 2024-02-09T17:37:11.169 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:37:11.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:10 smithi162 ceph-mon[44304]: 4.a scrub ok 2024-02-09T17:37:11.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:10 smithi162 ceph-mon[44304]: 1.d scrub starts 2024-02-09T17:37:11.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:10 smithi162 ceph-mon[44304]: 1.d scrub ok 2024-02-09T17:37:11.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:10 smithi162 ceph-mon[44304]: 4.4 scrub starts 2024-02-09T17:37:11.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:10 smithi162 ceph-mon[44304]: 4.2 scrub ok 2024-02-09T17:37:11.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:10 smithi162 ceph-mon[44304]: 4.5 scrub ok 2024-02-09T17:37:11.198 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:11.219 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:37:11.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:11.222 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:37:11.323 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:37:11.377 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:11.407 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:37:11.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:11.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:37:11.515 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:37:11.614 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:11.641 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:37:11.642 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:37:11.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:11.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:37:11.685 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-09T17:37:11.750 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:37:11.850 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:11.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:11 smithi162 ceph-mon[44304]: 4.e scrub starts 2024-02-09T17:37:11.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:11 smithi162 ceph-mon[44304]: 1.2 scrub starts 2024-02-09T17:37:11.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:11 smithi162 ceph-mon[44304]: 1.2 scrub ok 2024-02-09T17:37:11.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:11 smithi162 ceph-mon[44304]: pgmap v970: 37 pgs: 2 active+clean+scrubbing, 35 active+clean; 8.3 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 57 MiB/s wr, 952 op/s 2024-02-09T17:37:11.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:11 smithi162 ceph-mon[44304]: 4.0 scrub starts 2024-02-09T17:37:11.929 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:37:11.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:11.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:37:11.932 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:11 smithi099 ceph-mon[39420]: 4.e scrub starts 2024-02-09T17:37:11.933 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:11 smithi099 ceph-mon[39420]: 1.2 scrub starts 2024-02-09T17:37:11.933 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:11 smithi099 ceph-mon[39420]: 1.2 scrub ok 2024-02-09T17:37:11.933 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:11 smithi099 ceph-mon[39420]: pgmap v970: 37 pgs: 2 active+clean+scrubbing, 35 active+clean; 8.3 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 57 MiB/s wr, 952 op/s 2024-02-09T17:37:11.933 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:11 smithi099 ceph-mon[39420]: 4.0 scrub starts 2024-02-09T17:37:12.033 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:37:12.080 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:12.111 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:37:12.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:12.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:37:12.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:11 smithi179 ceph-mon[43975]: 4.e scrub starts 2024-02-09T17:37:12.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:11 smithi179 ceph-mon[43975]: 1.2 scrub starts 2024-02-09T17:37:12.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:11 smithi179 ceph-mon[43975]: 1.2 scrub ok 2024-02-09T17:37:12.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:11 smithi179 ceph-mon[43975]: pgmap v970: 37 pgs: 2 active+clean+scrubbing, 35 active+clean; 8.3 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 57 MiB/s wr, 952 op/s 2024-02-09T17:37:12.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:11 smithi179 ceph-mon[43975]: 4.0 scrub starts 2024-02-09T17:37:12.216 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:37:12.258 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:12.294 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:37:12.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:12.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:37:12.396 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:37:12.473 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:13.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:12 smithi179 ceph-mon[43975]: 4.d scrub starts 2024-02-09T17:37:13.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:12 smithi179 ceph-mon[43975]: 4.3 scrub starts 2024-02-09T17:37:13.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:12 smithi179 ceph-mon[43975]: 1.9 scrub starts 2024-02-09T17:37:13.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:12 smithi179 ceph-mon[43975]: 1.9 scrub ok 2024-02-09T17:37:13.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:12 smithi179 ceph-mon[43975]: 4.4 scrub ok 2024-02-09T17:37:13.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:12 smithi179 ceph-mon[43975]: 4.e scrub ok 2024-02-09T17:37:13.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:12 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4122709319' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:37:13.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:12 smithi162 ceph-mon[44304]: 4.d scrub starts 2024-02-09T17:37:13.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:12 smithi162 ceph-mon[44304]: 4.3 scrub starts 2024-02-09T17:37:13.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:12 smithi162 ceph-mon[44304]: 1.9 scrub starts 2024-02-09T17:37:13.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:12 smithi162 ceph-mon[44304]: 1.9 scrub ok 2024-02-09T17:37:13.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:12 smithi162 ceph-mon[44304]: 4.4 scrub ok 2024-02-09T17:37:13.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:12 smithi162 ceph-mon[44304]: 4.e scrub ok 2024-02-09T17:37:13.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:12 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4122709319' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:37:13.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:12 smithi099 ceph-mon[39420]: 4.d scrub starts 2024-02-09T17:37:13.248 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:12 smithi099 ceph-mon[39420]: 4.3 scrub starts 2024-02-09T17:37:13.248 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:12 smithi099 ceph-mon[39420]: 1.9 scrub starts 2024-02-09T17:37:13.248 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:12 smithi099 ceph-mon[39420]: 1.9 scrub ok 2024-02-09T17:37:13.248 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:12 smithi099 ceph-mon[39420]: 4.4 scrub ok 2024-02-09T17:37:13.248 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:12 smithi099 ceph-mon[39420]: 4.e scrub ok 2024-02-09T17:37:13.248 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:12 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4122709319' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:37:13.574 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:37:13.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:13.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:37:13.687 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:37:13.700 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:37:13.700 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:37:13.701 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:37:13.701 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:37:13.701 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:37:13.701 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:37:13.701 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:37:13.701 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:37:13.701 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:37:13.701 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:37:13.701 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:37:13.701 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:37:13.701 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:37:13.701 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:37:13.702 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:37:13.702 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:37:13.702 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:37:13.702 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:37:13.702 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:37:13.702 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:37:13.702 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:37:13.702 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:37:13.702 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:37:13.702 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:37:13.723 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:13.749 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:37:13.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:13.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:37:13.854 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:37:13.927 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:14.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:13 smithi179 ceph-mon[43975]: pgmap v971: 37 pgs: 3 active+clean+scrubbing, 34 active+clean; 8.3 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 62 MiB/s wr, 1.08k op/s 2024-02-09T17:37:14.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:13 smithi162 ceph-mon[44304]: pgmap v971: 37 pgs: 3 active+clean+scrubbing, 34 active+clean; 8.3 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 62 MiB/s wr, 1.08k op/s 2024-02-09T17:37:14.240 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:13 smithi099 ceph-mon[39420]: pgmap v971: 37 pgs: 3 active+clean+scrubbing, 34 active+clean; 8.3 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 62 MiB/s wr, 1.08k op/s 2024-02-09T17:37:14.820 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:37:14.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:14.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:37:14.925 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:37:14.995 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:15.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:14 smithi179 ceph-mon[43975]: 4.3 scrub ok 2024-02-09T17:37:15.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:14 smithi179 ceph-mon[43975]: 4.d scrub ok 2024-02-09T17:37:15.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:14 smithi179 ceph-mon[43975]: 4.0 scrub ok 2024-02-09T17:37:15.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:14 smithi162 ceph-mon[44304]: 4.3 scrub ok 2024-02-09T17:37:15.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:14 smithi162 ceph-mon[44304]: 4.d scrub ok 2024-02-09T17:37:15.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:14 smithi162 ceph-mon[44304]: 4.0 scrub ok 2024-02-09T17:37:15.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:14 smithi099 ceph-mon[39420]: 4.3 scrub ok 2024-02-09T17:37:15.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:14 smithi099 ceph-mon[39420]: 4.d scrub ok 2024-02-09T17:37:15.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:14 smithi099 ceph-mon[39420]: 4.0 scrub ok 2024-02-09T17:37:15.303 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:37:15.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:15.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:37:15.411 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:37:15.478 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:16.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:15 smithi179 ceph-mon[43975]: pgmap v972: 37 pgs: 3 active+clean+scrubbing, 34 active+clean; 8.3 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 103 B/s rd, 50 MiB/s wr, 902 op/s 2024-02-09T17:37:16.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:15 smithi179 ceph-mon[43975]: 1.3 scrub starts 2024-02-09T17:37:16.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:15 smithi179 ceph-mon[43975]: 1.3 scrub ok 2024-02-09T17:37:16.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:15 smithi162 ceph-mon[44304]: pgmap v972: 37 pgs: 3 active+clean+scrubbing, 34 active+clean; 8.3 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 103 B/s rd, 50 MiB/s wr, 902 op/s 2024-02-09T17:37:16.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:15 smithi162 ceph-mon[44304]: 1.3 scrub starts 2024-02-09T17:37:16.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:15 smithi162 ceph-mon[44304]: 1.3 scrub ok 2024-02-09T17:37:16.212 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:15 smithi099 ceph-mon[39420]: pgmap v972: 37 pgs: 3 active+clean+scrubbing, 34 active+clean; 8.3 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 103 B/s rd, 50 MiB/s wr, 902 op/s 2024-02-09T17:37:16.212 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:15 smithi099 ceph-mon[39420]: 1.3 scrub starts 2024-02-09T17:37:16.213 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:15 smithi099 ceph-mon[39420]: 1.3 scrub ok 2024-02-09T17:37:17.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:17 smithi179 ceph-mon[43975]: 4.8 scrub starts 2024-02-09T17:37:17.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:17 smithi179 ceph-mon[43975]: pgmap v973: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 8.1 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 24 MiB/s wr, 864 op/s 2024-02-09T17:37:17.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:17 smithi179 ceph-mon[43975]: 4.f scrub starts 2024-02-09T17:37:17.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:17 smithi179 ceph-mon[43975]: from='client.15552 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:37:17.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:17 smithi162 ceph-mon[44304]: 4.8 scrub starts 2024-02-09T17:37:17.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:17 smithi162 ceph-mon[44304]: pgmap v973: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 8.1 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 24 MiB/s wr, 864 op/s 2024-02-09T17:37:17.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:17 smithi162 ceph-mon[44304]: 4.f scrub starts 2024-02-09T17:37:17.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:17 smithi162 ceph-mon[44304]: from='client.15552 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:37:17.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:17 smithi099 ceph-mon[39420]: 4.8 scrub starts 2024-02-09T17:37:17.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:17 smithi099 ceph-mon[39420]: pgmap v973: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 8.1 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 24 MiB/s wr, 864 op/s 2024-02-09T17:37:17.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:17 smithi099 ceph-mon[39420]: 4.f scrub starts 2024-02-09T17:37:17.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:17 smithi099 ceph-mon[39420]: from='client.15552 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:37:18.300 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:37:18.301 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:37:18.305 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:37:18.307 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-02-09T17:37:18.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:18.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:37:18.410 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:37:18.445 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:18.473 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:37:18.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:18.478 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:37:18.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:18 smithi099 ceph-mon[39420]: 4.8 scrub ok 2024-02-09T17:37:18.580 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:37:18.610 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:18.632 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:37:18.634 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:18 smithi162 ceph-mon[44304]: 4.8 scrub ok 2024-02-09T17:37:18.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:18.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:37:18.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:18 smithi179 ceph-mon[43975]: 4.8 scrub ok 2024-02-09T17:37:18.736 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:37:18.781 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:18.893 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:37:18.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:18.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:37:19.004 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:37:19.046 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:19.072 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:37:19.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:19.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:37:19.178 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:37:19.236 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:19.387 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:19 smithi162 ceph-mon[44304]: 4.c scrub starts 2024-02-09T17:37:19.387 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:19 smithi162 ceph-mon[44304]: pgmap v974: 37 pgs: 37 active+clean; 8.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 21 MiB/s wr, 871 op/s 2024-02-09T17:37:19.387 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:19 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:37:19.387 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:19 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:37:19.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:19 smithi099 ceph-mon[39420]: 4.c scrub starts 2024-02-09T17:37:19.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:19 smithi099 ceph-mon[39420]: pgmap v974: 37 pgs: 37 active+clean; 8.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 21 MiB/s wr, 871 op/s 2024-02-09T17:37:19.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:19 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:37:19.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:19 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:37:19.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:19 smithi179 ceph-mon[43975]: 4.c scrub starts 2024-02-09T17:37:19.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:19 smithi179 ceph-mon[43975]: pgmap v974: 37 pgs: 37 active+clean; 8.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 21 MiB/s wr, 871 op/s 2024-02-09T17:37:19.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:19 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:37:19.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:19 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:37:20.218 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:37:20.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:20 smithi179 ceph-mon[43975]: 1.4 scrub starts 2024-02-09T17:37:20.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:20 smithi179 ceph-mon[43975]: 1.4 scrub ok 2024-02-09T17:37:20.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:20 smithi179 ceph-mon[43975]: osdmap e358: 12 total, 12 up, 12 in 2024-02-09T17:37:20.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:20 smithi179 ceph-mon[43975]: 4.f scrub ok 2024-02-09T17:37:20.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:20 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/603714186' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:37:20.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:20 smithi162 ceph-mon[44304]: 1.4 scrub starts 2024-02-09T17:37:20.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:20 smithi162 ceph-mon[44304]: 1.4 scrub ok 2024-02-09T17:37:20.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:20 smithi162 ceph-mon[44304]: osdmap e358: 12 total, 12 up, 12 in 2024-02-09T17:37:20.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:20 smithi162 ceph-mon[44304]: 4.f scrub ok 2024-02-09T17:37:20.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:20 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/603714186' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:37:20.696 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is unset 2024-02-09T17:37:20.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:20 smithi099 ceph-mon[39420]: 1.4 scrub starts 2024-02-09T17:37:20.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:20 smithi099 ceph-mon[39420]: 1.4 scrub ok 2024-02-09T17:37:20.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:20 smithi099 ceph-mon[39420]: osdmap e358: 12 total, 12 up, 12 in 2024-02-09T17:37:20.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:20 smithi099 ceph-mon[39420]: 4.f scrub ok 2024-02-09T17:37:20.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:20 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/603714186' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:37:21.415 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:37:21.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:21.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:37:21.542 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:37:21.590 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:21.612 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:37:21.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:21.617 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:37:21.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:21 smithi179 ceph-mon[43975]: 4.c scrub ok 2024-02-09T17:37:21.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:21 smithi179 ceph-mon[43975]: 4.9 scrub starts 2024-02-09T17:37:21.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:21 smithi179 ceph-mon[43975]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:37:21.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:21 smithi179 ceph-mon[43975]: Cluster is now healthy 2024-02-09T17:37:21.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:21 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/603714186' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:37:21.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:21 smithi179 ceph-mon[43975]: osdmap e359: 12 total, 12 up, 12 in 2024-02-09T17:37:21.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:21 smithi179 ceph-mon[43975]: pgmap v977: 37 pgs: 37 active+clean; 7.9 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s wr, 923 op/s 2024-02-09T17:37:21.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:21 smithi162 ceph-mon[44304]: 4.c scrub ok 2024-02-09T17:37:21.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:21 smithi162 ceph-mon[44304]: 4.9 scrub starts 2024-02-09T17:37:21.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:21 smithi162 ceph-mon[44304]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:37:21.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:21 smithi162 ceph-mon[44304]: Cluster is now healthy 2024-02-09T17:37:21.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:21 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/603714186' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:37:21.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:21 smithi162 ceph-mon[44304]: osdmap e359: 12 total, 12 up, 12 in 2024-02-09T17:37:21.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:21 smithi162 ceph-mon[44304]: pgmap v977: 37 pgs: 37 active+clean; 7.9 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s wr, 923 op/s 2024-02-09T17:37:21.719 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:37:21.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:21 smithi099 ceph-mon[39420]: 4.c scrub ok 2024-02-09T17:37:21.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:21 smithi099 ceph-mon[39420]: 4.9 scrub starts 2024-02-09T17:37:21.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:21 smithi099 ceph-mon[39420]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:37:21.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:21 smithi099 ceph-mon[39420]: Cluster is now healthy 2024-02-09T17:37:21.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:21 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/603714186' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:37:21.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:21 smithi099 ceph-mon[39420]: osdmap e359: 12 total, 12 up, 12 in 2024-02-09T17:37:21.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:21 smithi099 ceph-mon[39420]: pgmap v977: 37 pgs: 37 active+clean; 7.9 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s wr, 923 op/s 2024-02-09T17:37:21.757 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:21.794 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:37:21.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:21.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:37:21.897 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:37:21.945 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:21.973 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:37:21.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:21.977 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:37:22.078 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:37:22.125 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:22.166 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:37:22.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:22.172 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:37:22.273 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:37:22.311 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:22.333 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:37:22.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:22.338 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:37:22.439 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:37:22.468 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:22.492 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:37:22.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:22.497 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:37:22.509 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":973,"stamp":"2024-02-09T17:37:16.653847+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8697807416,"num_objects":132712,"num_object_clones":0,"num_object_copies":398136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":132712,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142218,"num_write_kb":8797633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145137,"ondisk_log_size":145137,"up":111,"acting":111,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":112,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":29014484,"kb_used_data":25742276,"kb_used_omap":229,"kb_used_meta":3271450,"kb_avail":1095927340,"statfs":{"total":1151940427776,"available":1122229596160,"internally_reserved":0,"allocated":26360090624,"data_stored":26354690564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235127,"internal_metadata":3349965193},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,1,0,3,9,7,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-56557544,"num_objects":-862,"num_object_clones":0,"num_object_copies":-2586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-862,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8646,"num_write_kb":249025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004776"},"pg_stats":[{"pgid":"1.14","version":"63'139","reported_seq":948,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-09T17:37:08.823804+0000","last_change":"2024-02-09T17:37:03.748841+0000","last_active":"2024-02-09T17:37:08.823804+0000","last_peered":"2024-02-09T17:37:08.823804+0000","last_clean":"2024-02-09T17:37:08.823804+0000","last_became_active":"2024-02-09T17:33:05.309643+0000","last_became_peered":"2024-02-09T17:33:05.309643+0000","last_unstale":"2024-02-09T17:37:08.823804+0000","last_undegraded":"2024-02-09T17:37:08.823804+0000","last_fullsized":"2024-02-09T17:37:08.823804+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:37:03.748701+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:34:13.228622+0000","last_clean_scrub_stamp":"2024-02-09T17:37:03.748701+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:22.979456+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"357'8842","reported_seq":9015,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-09T17:37:15.769759+0000","last_change":"2024-02-09T17:37:10.153953+0000","last_active":"2024-02-09T17:37:15.769759+0000","last_peered":"2024-02-09T17:37:15.769759+0000","last_clean":"2024-02-09T17:37:15.769759+0000","last_became_active":"2024-02-09T17:35:30.390786+0000","last_became_peered":"2024-02-09T17:35:30.390786+0000","last_unstale":"2024-02-09T17:37:15.769759+0000","last_undegraded":"2024-02-09T17:37:15.769759+0000","last_fullsized":"2024-02-09T17:37:15.769759+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'8529","last_scrub_stamp":"2024-02-09T17:37:10.153802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:37:10.153802+0000","objects_scrubbed":8484,"log_size":8842,"log_dups_size":0,"ondisk_log_size":8842,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:34.665727+0000","scrub_duration":1669,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535429120,"num_objects":8170,"num_object_clones":0,"num_object_copies":24510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8842,"num_write_kb":544384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,11],"acting":[6,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"63'139","reported_seq":979,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-09T17:37:08.826447+0000","last_change":"2024-02-09T17:37:07.369195+0000","last_active":"2024-02-09T17:37:08.826447+0000","last_peered":"2024-02-09T17:37:08.826447+0000","last_clean":"2024-02-09T17:37:08.826447+0000","last_became_active":"2024-02-09T17:33:00.284421+0000","last_became_peered":"2024-02-09T17:33:00.284421+0000","last_unstale":"2024-02-09T17:37:08.826447+0000","last_undegraded":"2024-02-09T17:37:08.826447+0000","last_fullsized":"2024-02-09T17:37:08.826447+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:37:07.369056+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:37:07.369056+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:14.417376+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"357'8985","reported_seq":9157,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-09T17:37:15.329421+0000","last_change":"2024-02-09T17:37:07.695049+0000","last_active":"2024-02-09T17:37:15.329421+0000","last_peered":"2024-02-09T17:37:15.329421+0000","last_clean":"2024-02-09T17:37:15.329421+0000","last_became_active":"2024-02-09T17:35:30.390424+0000","last_became_peered":"2024-02-09T17:35:30.390424+0000","last_unstale":"2024-02-09T17:37:15.329421+0000","last_undegraded":"2024-02-09T17:37:15.329421+0000","last_fullsized":"2024-02-09T17:37:15.329421+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"356'8575","last_scrub_stamp":"2024-02-09T17:37:07.694769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:37:07.694769+0000","objects_scrubbed":8497,"log_size":8985,"log_dups_size":0,"ondisk_log_size":8985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:25.342399+0000","scrub_duration":3173,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":548470784,"num_objects":8369,"num_object_clones":0,"num_object_copies":25107,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8985,"num_write_kb":555328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,0],"acting":[1,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"63'139","reported_seq":995,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-09T17:37:10.429117+0000","last_change":"2024-02-09T17:37:10.429162+0000","last_active":"2024-02-09T17:37:10.429117+0000","last_peered":"2024-02-09T17:37:10.429117+0000","last_clean":"2024-02-09T17:37:10.429117+0000","last_became_active":"2024-02-09T17:34:01.410594+0000","last_became_peered":"2024-02-09T17:34:01.410594+0000","last_unstale":"2024-02-09T17:37:10.429117+0000","last_undegraded":"2024-02-09T17:37:10.429117+0000","last_fullsized":"2024-02-09T17:37:10.429117+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:37:10.429033+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:37:10.429033+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:17.314134+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"357'9021","reported_seq":9195,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-09T17:37:16.388629+0000","last_change":"2024-02-09T17:37:09.133954+0000","last_active":"2024-02-09T17:37:16.388629+0000","last_peered":"2024-02-09T17:37:16.388629+0000","last_clean":"2024-02-09T17:37:16.388629+0000","last_became_active":"2024-02-09T17:35:30.166669+0000","last_became_peered":"2024-02-09T17:35:30.166669+0000","last_unstale":"2024-02-09T17:37:16.388629+0000","last_undegraded":"2024-02-09T17:37:16.388629+0000","last_fullsized":"2024-02-09T17:37:16.388629+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'8615","last_scrub_stamp":"2024-02-09T17:37:09.133803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:37:09.133803+0000","objects_scrubbed":8567,"log_size":9021,"log_dups_size":0,"ondisk_log_size":9021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:13.853601+0000","scrub_duration":2038,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540606464,"num_objects":8249,"num_object_clones":0,"num_object_copies":24747,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9021,"num_write_kb":552640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,2],"acting":[9,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.3","version":"63'139","reported_seq":492,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-09T17:37:14.844912+0000","last_change":"2024-02-09T17:37:14.844939+0000","last_active":"2024-02-09T17:37:14.844912+0000","last_peered":"2024-02-09T17:37:14.844912+0000","last_clean":"2024-02-09T17:37:14.844912+0000","last_became_active":"2024-02-09T17:33:00.279369+0000","last_became_peered":"2024-02-09T17:33:00.279369+0000","last_unstale":"2024-02-09T17:37:14.844912+0000","last_undegraded":"2024-02-09T17:37:14.844912+0000","last_fullsized":"2024-02-09T17:37:14.844912+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:37:14.844841+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:37:14.844841+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:15.289072+0000","scrub_duration":98,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"357'8788","reported_seq":8961,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-09T17:37:15.327704+0000","last_change":"2024-02-09T17:37:11.857576+0000","last_active":"2024-02-09T17:37:15.327704+0000","last_peered":"2024-02-09T17:37:15.327704+0000","last_clean":"2024-02-09T17:37:15.327704+0000","last_became_active":"2024-02-09T17:35:30.166273+0000","last_became_peered":"2024-02-09T17:35:30.166273+0000","last_unstale":"2024-02-09T17:37:15.327704+0000","last_undegraded":"2024-02-09T17:37:15.327704+0000","last_fullsized":"2024-02-09T17:37:15.327704+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'8608","last_scrub_stamp":"2024-02-09T17:37:11.857362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:37:11.857362+0000","objects_scrubbed":8467,"log_size":8788,"log_dups_size":0,"ondisk_log_size":8788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:38.710861+0000","scrub_duration":2226,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":539492352,"num_objects":8232,"num_object_clones":0,"num_object_copies":24696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8788,"num_write_kb":544640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"63'139","reported_seq":503,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-09T17:37:08.829400+0000","last_change":"2024-02-09T17:37:03.758658+0000","last_active":"2024-02-09T17:37:08.829400+0000","last_peered":"2024-02-09T17:37:08.829400+0000","last_clean":"2024-02-09T17:37:08.829400+0000","last_became_active":"2024-02-09T17:34:01.342607+0000","last_became_peered":"2024-02-09T17:34:01.342607+0000","last_unstale":"2024-02-09T17:37:08.829400+0000","last_undegraded":"2024-02-09T17:37:08.829400+0000","last_fullsized":"2024-02-09T17:37:08.829400+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:37:03.758546+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:37:03.758546+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:12.219105+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.3","version":"357'8858","reported_seq":9031,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-09T17:37:15.143921+0000","last_change":"2024-02-09T17:37:13.488486+0000","last_active":"2024-02-09T17:37:15.143921+0000","last_peered":"2024-02-09T17:37:15.143921+0000","last_clean":"2024-02-09T17:37:15.143921+0000","last_became_active":"2024-02-09T17:35:30.134014+0000","last_became_peered":"2024-02-09T17:35:30.134014+0000","last_unstale":"2024-02-09T17:37:15.143921+0000","last_undegraded":"2024-02-09T17:37:15.143921+0000","last_fullsized":"2024-02-09T17:37:15.143921+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'8765","last_scrub_stamp":"2024-02-09T17:37:13.488343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:37:13.488343+0000","objects_scrubbed":8429,"log_size":8858,"log_dups_size":0,"ondisk_log_size":8858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:28.392245+0000","scrub_duration":2180,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":543293440,"num_objects":8290,"num_object_clones":0,"num_object_copies":24870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8858,"num_write_kb":548736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,1],"acting":[11,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"63'139","reported_seq":494,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-09T17:37:08.829455+0000","last_change":"2024-02-09T17:37:07.707766+0000","last_active":"2024-02-09T17:37:08.829455+0000","last_peered":"2024-02-09T17:37:08.829455+0000","last_clean":"2024-02-09T17:37:08.829455+0000","last_became_active":"2024-02-09T17:33:00.284702+0000","last_became_peered":"2024-02-09T17:33:00.284702+0000","last_unstale":"2024-02-09T17:37:08.829455+0000","last_undegraded":"2024-02-09T17:37:08.829455+0000","last_fullsized":"2024-02-09T17:37:08.829455+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:37:07.707633+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:37:07.707633+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:28.410525+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2","version":"357'8910","reported_seq":9083,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-09T17:37:16.388367+0000","last_change":"2024-02-09T17:37:09.776219+0000","last_active":"2024-02-09T17:37:16.388367+0000","last_peered":"2024-02-09T17:37:16.388367+0000","last_clean":"2024-02-09T17:37:16.388367+0000","last_became_active":"2024-02-09T17:35:29.899914+0000","last_became_peered":"2024-02-09T17:35:29.899914+0000","last_unstale":"2024-02-09T17:37:16.388367+0000","last_undegraded":"2024-02-09T17:37:16.388367+0000","last_fullsized":"2024-02-09T17:37:16.388367+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'8538","last_scrub_stamp":"2024-02-09T17:37:09.776086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:37:09.776086+0000","objects_scrubbed":8510,"log_size":8910,"log_dups_size":0,"ondisk_log_size":8910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:34.505426+0000","scrub_duration":1666,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":534904832,"num_objects":8162,"num_object_clones":0,"num_object_copies":24486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8910,"num_write_kb":546304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.7","version":"63'139","reported_seq":507,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-09T17:37:08.833885+0000","last_change":"2024-02-09T17:37:06.599735+0000","last_active":"2024-02-09T17:37:08.833885+0000","last_peered":"2024-02-09T17:37:08.833885+0000","last_clean":"2024-02-09T17:37:08.833885+0000","last_became_active":"2024-02-09T17:33:00.278966+0000","last_became_peered":"2024-02-09T17:33:00.278966+0000","last_unstale":"2024-02-09T17:37:08.833885+0000","last_undegraded":"2024-02-09T17:37:08.833885+0000","last_fullsized":"2024-02-09T17:37:08.833885+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:37:06.599635+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:37:06.599635+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:25.470941+0000","scrub_duration":102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"357'8913","reported_seq":9091,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-09T17:37:15.115321+0000","last_change":"2024-02-09T17:37:05.983264+0000","last_active":"2024-02-09T17:37:15.115321+0000","last_peered":"2024-02-09T17:37:15.115321+0000","last_clean":"2024-02-09T17:37:15.115321+0000","last_became_active":"2024-02-09T17:35:29.900032+0000","last_became_peered":"2024-02-09T17:35:29.900032+0000","last_unstale":"2024-02-09T17:37:15.115321+0000","last_undegraded":"2024-02-09T17:37:15.115321+0000","last_fullsized":"2024-02-09T17:37:15.115321+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"356'8379","last_scrub_stamp":"2024-02-09T17:37:05.983122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:37:05.983122+0000","objects_scrubbed":8324,"log_size":8913,"log_dups_size":0,"ondisk_log_size":8913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:28.414791+0000","scrub_duration":1701,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":543752192,"num_objects":8297,"num_object_clones":0,"num_object_copies":24891,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8913,"num_write_kb":550720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"63'139","reported_seq":577,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-09T17:37:08.823650+0000","last_change":"2024-02-09T17:35:19.886268+0000","last_active":"2024-02-09T17:37:08.823650+0000","last_peered":"2024-02-09T17:37:08.823650+0000","last_clean":"2024-02-09T17:37:08.823650+0000","last_became_active":"2024-02-09T17:33:00.284924+0000","last_became_peered":"2024-02-09T17:33:00.284924+0000","last_unstale":"2024-02-09T17:37:08.823650+0000","last_undegraded":"2024-02-09T17:37:08.823650+0000","last_fullsized":"2024-02-09T17:37:08.823650+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:35:19.886117+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:35:19.886117+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.0","version":"357'8721","reported_seq":8830,"reported_epoch":357,"state":"active+clean+scrubbing","last_fresh":"2024-02-09T17:37:11.924663+0000","last_change":"2024-02-09T17:37:11.045319+0000","last_active":"2024-02-09T17:37:11.924663+0000","last_peered":"2024-02-09T17:37:11.924663+0000","last_clean":"2024-02-09T17:37:11.924663+0000","last_became_active":"2024-02-09T17:35:30.134347+0000","last_became_peered":"2024-02-09T17:35:30.134347+0000","last_unstale":"2024-02-09T17:37:11.924663+0000","last_undegraded":"2024-02-09T17:37:11.924663+0000","last_fullsized":"2024-02-09T17:37:11.924663+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:35:28.884607+0000","objects_scrubbed":2599,"log_size":8721,"log_dups_size":0,"ondisk_log_size":8721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":555941888,"num_objects":8483,"num_object_clones":0,"num_object_copies":25449,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8721,"num_write_kb":550528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"63'139","reported_seq":494,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-09T17:37:08.829737+0000","last_change":"2024-02-09T17:37:02.738763+0000","last_active":"2024-02-09T17:37:08.829737+0000","last_peered":"2024-02-09T17:37:08.829737+0000","last_clean":"2024-02-09T17:37:08.829737+0000","last_became_active":"2024-02-09T17:33:00.274636+0000","last_became_peered":"2024-02-09T17:33:00.274636+0000","last_unstale":"2024-02-09T17:37:08.829737+0000","last_undegraded":"2024-02-09T17:37:08.829737+0000","last_fullsized":"2024-02-09T17:37:08.829737+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:37:02.738611+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:37:02.738611+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:28.854048+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.f","version":"357'8814","reported_seq":8894,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-09T17:37:15.284377+0000","last_change":"2024-02-09T17:35:29.905704+0000","last_active":"2024-02-09T17:37:15.284377+0000","last_peered":"2024-02-09T17:37:15.284377+0000","last_clean":"2024-02-09T17:37:15.284377+0000","last_became_active":"2024-02-09T17:35:29.905385+0000","last_became_peered":"2024-02-09T17:35:29.905385+0000","last_unstale":"2024-02-09T17:37:15.284377+0000","last_undegraded":"2024-02-09T17:37:15.284377+0000","last_fullsized":"2024-02-09T17:37:15.284377+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:35:28.884607+0000","objects_scrubbed":0,"log_size":8814,"log_dups_size":0,"ondisk_log_size":8814,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540540928,"num_objects":8248,"num_object_clones":0,"num_object_copies":24744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8814,"num_write_kb":545984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,11],"acting":[1,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"63'139","reported_seq":488,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-09T17:37:08.823036+0000","last_change":"2024-02-09T17:36:06.172514+0000","last_active":"2024-02-09T17:37:08.823036+0000","last_peered":"2024-02-09T17:37:08.823036+0000","last_clean":"2024-02-09T17:37:08.823036+0000","last_became_active":"2024-02-09T17:32:35.418200+0000","last_became_peered":"2024-02-09T17:32:35.418200+0000","last_unstale":"2024-02-09T17:37:08.823036+0000","last_undegraded":"2024-02-09T17:37:08.823036+0000","last_fullsized":"2024-02-09T17:37:08.823036+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:36:06.172377+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:36:06.172377+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:37:35.221695+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.e","version":"357'8866","reported_seq":9040,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-09T17:37:15.509017+0000","last_change":"2024-02-09T17:37:11.874433+0000","last_active":"2024-02-09T17:37:15.509017+0000","last_peered":"2024-02-09T17:37:15.509017+0000","last_clean":"2024-02-09T17:37:15.509017+0000","last_became_active":"2024-02-09T17:35:30.390749+0000","last_became_peered":"2024-02-09T17:35:30.390749+0000","last_unstale":"2024-02-09T17:37:15.509017+0000","last_undegraded":"2024-02-09T17:37:15.509017+0000","last_fullsized":"2024-02-09T17:37:15.509017+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'8651","last_scrub_stamp":"2024-02-09T17:37:11.874252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:37:11.874252+0000","objects_scrubbed":8474,"log_size":8866,"log_dups_size":0,"ondisk_log_size":8866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:29.860070+0000","scrub_duration":1689,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":538574848,"num_objects":8218,"num_object_clones":0,"num_object_copies":24654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8866,"num_write_kb":546688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,6],"acting":[4,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"63'139","reported_seq":503,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-09T17:37:08.823848+0000","last_change":"2024-02-09T17:37:05.469016+0000","last_active":"2024-02-09T17:37:08.823848+0000","last_peered":"2024-02-09T17:37:08.823848+0000","last_clean":"2024-02-09T17:37:08.823848+0000","last_became_active":"2024-02-09T17:34:01.388313+0000","last_became_peered":"2024-02-09T17:34:01.388313+0000","last_unstale":"2024-02-09T17:37:08.823848+0000","last_undegraded":"2024-02-09T17:37:08.823848+0000","last_fullsized":"2024-02-09T17:37:08.823848+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:37:05.468894+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:37:05.468894+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:34.390980+0000","scrub_duration":138,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.d","version":"357'8883","reported_seq":9056,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-09T17:37:15.493851+0000","last_change":"2024-02-09T17:37:13.543612+0000","last_active":"2024-02-09T17:37:15.493851+0000","last_peered":"2024-02-09T17:37:15.493851+0000","last_clean":"2024-02-09T17:37:15.493851+0000","last_became_active":"2024-02-09T17:35:29.905824+0000","last_became_peered":"2024-02-09T17:35:29.905824+0000","last_unstale":"2024-02-09T17:37:15.493851+0000","last_undegraded":"2024-02-09T17:37:15.493851+0000","last_fullsized":"2024-02-09T17:37:15.493851+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'8767","last_scrub_stamp":"2024-02-09T17:37:13.543492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:37:13.543492+0000","objects_scrubbed":8445,"log_size":8883,"log_dups_size":0,"ondisk_log_size":8883,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:22.127226+0000","scrub_duration":2388,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":541982744,"num_objects":8271,"num_object_clones":0,"num_object_copies":24813,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8271,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8883,"num_write_kb":548865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,1],"acting":[4,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"63'139","reported_seq":497,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-09T17:37:08.825857+0000","last_change":"2024-02-09T17:36:20.412325+0000","last_active":"2024-02-09T17:37:08.825857+0000","last_peered":"2024-02-09T17:37:08.825857+0000","last_clean":"2024-02-09T17:37:08.825857+0000","last_became_active":"2024-02-09T17:32:35.384617+0000","last_became_peered":"2024-02-09T17:32:35.384617+0000","last_unstale":"2024-02-09T17:37:08.825857+0000","last_undegraded":"2024-02-09T17:37:08.825857+0000","last_fullsized":"2024-02-09T17:37:08.825857+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:36:20.412161+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:36:20.412161+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:37:20.924824+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"357'9101","reported_seq":9181,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-09T17:37:15.480277+0000","last_change":"2024-02-09T17:35:29.903975+0000","last_active":"2024-02-09T17:37:15.480277+0000","last_peered":"2024-02-09T17:37:15.480277+0000","last_clean":"2024-02-09T17:37:15.480277+0000","last_became_active":"2024-02-09T17:35:29.903553+0000","last_became_peered":"2024-02-09T17:35:29.903553+0000","last_unstale":"2024-02-09T17:37:15.480277+0000","last_undegraded":"2024-02-09T17:37:15.480277+0000","last_fullsized":"2024-02-09T17:37:15.480277+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:35:28.884607+0000","objects_scrubbed":0,"log_size":9101,"log_dups_size":0,"ondisk_log_size":9101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":551747584,"num_objects":8419,"num_object_clones":0,"num_object_copies":25257,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9101,"num_write_kb":560640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"63'139","reported_seq":981,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-09T17:37:11.520678+0000","last_change":"2024-02-09T17:37:11.520715+0000","last_active":"2024-02-09T17:37:11.520678+0000","last_peered":"2024-02-09T17:37:11.520678+0000","last_clean":"2024-02-09T17:37:11.520678+0000","last_became_active":"2024-02-09T17:32:35.402025+0000","last_became_peered":"2024-02-09T17:32:35.402025+0000","last_unstale":"2024-02-09T17:37:11.520678+0000","last_undegraded":"2024-02-09T17:37:11.520678+0000","last_fullsized":"2024-02-09T17:37:11.520678+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:37:11.520601+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:30.193194+0000","last_clean_scrub_stamp":"2024-02-09T17:37:11.520601+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:38.007919+0000","scrub_duration":102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"357'8899","reported_seq":9072,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-09T17:37:13.565575+0000","last_change":"2024-02-09T17:37:05.121978+0000","last_active":"2024-02-09T17:37:13.565575+0000","last_peered":"2024-02-09T17:37:13.565575+0000","last_clean":"2024-02-09T17:37:13.565575+0000","last_became_active":"2024-02-09T17:35:29.906768+0000","last_became_peered":"2024-02-09T17:35:29.906768+0000","last_unstale":"2024-02-09T17:37:13.565575+0000","last_undegraded":"2024-02-09T17:37:13.565575+0000","last_fullsized":"2024-02-09T17:37:13.565575+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"356'8446","last_scrub_stamp":"2024-02-09T17:37:05.121787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:37:05.121787+0000","objects_scrubbed":8387,"log_size":8899,"log_dups_size":0,"ondisk_log_size":8899,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:30.011025+0000","scrub_duration":2496,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":557645824,"num_objects":8509,"num_object_clones":0,"num_object_copies":25527,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8899,"num_write_kb":557056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"63'139","reported_seq":456,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-09T17:37:08.827193+0000","last_change":"2024-02-09T17:37:06.250206+0000","last_active":"2024-02-09T17:37:08.827193+0000","last_peered":"2024-02-09T17:37:08.827193+0000","last_clean":"2024-02-09T17:37:08.827193+0000","last_became_active":"2024-02-09T17:33:05.309824+0000","last_became_peered":"2024-02-09T17:33:05.309824+0000","last_unstale":"2024-02-09T17:37:08.827193+0000","last_undegraded":"2024-02-09T17:37:08.827193+0000","last_fullsized":"2024-02-09T17:37:08.827193+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:37:06.250017+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:37:06.250017+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:24.561623+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"357'8817","reported_seq":8988,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-09T17:37:15.775726+0000","last_change":"2024-02-09T17:37:09.380478+0000","last_active":"2024-02-09T17:37:15.775726+0000","last_peered":"2024-02-09T17:37:15.775726+0000","last_clean":"2024-02-09T17:37:15.775726+0000","last_became_active":"2024-02-09T17:35:30.390843+0000","last_became_peered":"2024-02-09T17:35:30.390843+0000","last_unstale":"2024-02-09T17:37:15.775726+0000","last_undegraded":"2024-02-09T17:37:15.775726+0000","last_fullsized":"2024-02-09T17:37:15.775726+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'8477","last_scrub_stamp":"2024-02-09T17:37:09.380351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:37:09.380351+0000","objects_scrubbed":8431,"log_size":8817,"log_dups_size":0,"ondisk_log_size":8817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:24.285734+0000","scrub_duration":1888,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535232512,"num_objects":8167,"num_object_clones":0,"num_object_copies":24501,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8817,"num_write_kb":543488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.f","version":"63'139","reported_seq":515,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-09T17:37:08.833779+0000","last_change":"2024-02-09T17:36:22.364212+0000","last_active":"2024-02-09T17:37:08.833779+0000","last_peered":"2024-02-09T17:37:08.833779+0000","last_clean":"2024-02-09T17:37:08.833779+0000","last_became_active":"2024-02-09T17:33:05.316100+0000","last_became_peered":"2024-02-09T17:33:05.316100+0000","last_unstale":"2024-02-09T17:37:08.833779+0000","last_undegraded":"2024-02-09T17:37:08.833779+0000","last_fullsized":"2024-02-09T17:37:08.833779+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:36:22.364130+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:36:22.364130+0000","last_clean_scrub_stamp":"2024-02-09T17:36:22.364130+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:37:41.044157+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"357'8875","reported_seq":8955,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-09T17:37:15.150947+0000","last_change":"2024-02-09T17:35:29.906181+0000","last_active":"2024-02-09T17:37:15.150947+0000","last_peered":"2024-02-09T17:37:15.150947+0000","last_clean":"2024-02-09T17:37:15.150947+0000","last_became_active":"2024-02-09T17:35:29.905885+0000","last_became_peered":"2024-02-09T17:35:29.905885+0000","last_unstale":"2024-02-09T17:37:15.150947+0000","last_undegraded":"2024-02-09T17:37:15.150947+0000","last_fullsized":"2024-02-09T17:37:15.150947+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:35:28.884607+0000","objects_scrubbed":0,"log_size":8875,"log_dups_size":0,"ondisk_log_size":8875,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":544145408,"num_objects":8303,"num_object_clones":0,"num_object_copies":24909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8875,"num_write_kb":549696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,3],"acting":[11,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"63'139","reported_seq":952,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-09T17:37:08.837354+0000","last_change":"2024-02-09T17:37:07.137667+0000","last_active":"2024-02-09T17:37:08.837354+0000","last_peered":"2024-02-09T17:37:08.837354+0000","last_clean":"2024-02-09T17:37:08.837354+0000","last_became_active":"2024-02-09T17:35:20.840386+0000","last_became_peered":"2024-02-09T17:35:20.840386+0000","last_unstale":"2024-02-09T17:37:08.837354+0000","last_undegraded":"2024-02-09T17:37:08.837354+0000","last_fullsized":"2024-02-09T17:37:08.837354+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:37:07.137544+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:23.912504+0000","last_clean_scrub_stamp":"2024-02-09T17:37:07.137544+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:21.263740+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"357'8925","reported_seq":9005,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-09T17:37:15.129833+0000","last_change":"2024-02-09T17:35:30.134361+0000","last_active":"2024-02-09T17:37:15.129833+0000","last_peered":"2024-02-09T17:37:15.129833+0000","last_clean":"2024-02-09T17:37:15.129833+0000","last_became_active":"2024-02-09T17:35:30.134041+0000","last_became_peered":"2024-02-09T17:35:30.134041+0000","last_unstale":"2024-02-09T17:37:15.129833+0000","last_undegraded":"2024-02-09T17:37:15.129833+0000","last_fullsized":"2024-02-09T17:37:15.129833+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:35:28.884607+0000","objects_scrubbed":0,"log_size":8925,"log_dups_size":0,"ondisk_log_size":8925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":545456128,"num_objects":8323,"num_object_clones":0,"num_object_copies":24969,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8925,"num_write_kb":551936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"63'139","reported_seq":464,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-09T17:37:09.476210+0000","last_change":"2024-02-09T17:37:09.476252+0000","last_active":"2024-02-09T17:37:09.476210+0000","last_peered":"2024-02-09T17:37:09.476210+0000","last_clean":"2024-02-09T17:37:09.476210+0000","last_became_active":"2024-02-09T17:33:05.319668+0000","last_became_peered":"2024-02-09T17:33:05.319668+0000","last_unstale":"2024-02-09T17:37:09.476210+0000","last_undegraded":"2024-02-09T17:37:09.476210+0000","last_fullsized":"2024-02-09T17:37:09.476210+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:37:09.476147+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:37:09.476147+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:14.845386+0000","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"63'139","reported_seq":463,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-09T17:37:08.825710+0000","last_change":"2024-02-09T17:36:17.357034+0000","last_active":"2024-02-09T17:37:08.825710+0000","last_peered":"2024-02-09T17:37:08.825710+0000","last_clean":"2024-02-09T17:37:08.825710+0000","last_became_active":"2024-02-09T17:33:05.310104+0000","last_became_peered":"2024-02-09T17:33:05.310104+0000","last_unstale":"2024-02-09T17:37:08.825710+0000","last_undegraded":"2024-02-09T17:37:08.825710+0000","last_fullsized":"2024-02-09T17:37:08.825710+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:36:17.356909+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:36:17.356909+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:37:45.339428+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"63'139","reported_seq":471,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-09T17:37:09.136873+0000","last_change":"2024-02-09T17:36:18.779412+0000","last_active":"2024-02-09T17:37:09.136873+0000","last_peered":"2024-02-09T17:37:09.136873+0000","last_clean":"2024-02-09T17:37:09.136873+0000","last_became_active":"2024-02-09T17:34:01.353171+0000","last_became_peered":"2024-02-09T17:34:01.353171+0000","last_unstale":"2024-02-09T17:37:09.136873+0000","last_undegraded":"2024-02-09T17:37:09.136873+0000","last_fullsized":"2024-02-09T17:37:09.136873+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:36:18.779304+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:36:18.779304+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:37:42.122054+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.12","version":"63'139","reported_seq":945,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-09T17:37:08.826393+0000","last_change":"2024-02-09T17:37:07.797486+0000","last_active":"2024-02-09T17:37:08.826393+0000","last_peered":"2024-02-09T17:37:08.826393+0000","last_clean":"2024-02-09T17:37:08.826393+0000","last_became_active":"2024-02-09T17:33:05.308268+0000","last_became_peered":"2024-02-09T17:33:05.308268+0000","last_unstale":"2024-02-09T17:37:08.826393+0000","last_undegraded":"2024-02-09T17:37:08.826393+0000","last_fullsized":"2024-02-09T17:37:08.826393+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:37:07.797354+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:37:07.797354+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:29.748476+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"63'139","reported_seq":499,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-09T17:37:08.822328+0000","last_change":"2024-02-09T17:36:27.497000+0000","last_active":"2024-02-09T17:37:08.822328+0000","last_peered":"2024-02-09T17:37:08.822328+0000","last_clean":"2024-02-09T17:37:08.822328+0000","last_became_active":"2024-02-09T17:33:05.309527+0000","last_became_peered":"2024-02-09T17:33:05.309527+0000","last_unstale":"2024-02-09T17:37:08.822328+0000","last_undegraded":"2024-02-09T17:37:08.822328+0000","last_fullsized":"2024-02-09T17:37:08.822328+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:36:27.496900+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:36:27.496900+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:37:56.755524+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":8697217048,"num_objects":132710,"num_object_clones":0,"num_object_copies":398130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":132710,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142218,"num_write_kb":8797633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26213822464,"data_stored":26213810248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142218,"ondisk_log_size":142218,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2919,"ondisk_log_size":2919,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056819,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4144432,"kb_used_data":3730088,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":89600720,"statfs":{"total":95995035648,"available":91751137280,"internally_reserved":0,"allocated":3819610112,"data_stored":3819162013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":424278468},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:36:14 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":1.2010000000000001,"15min":1.0409999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":1.9159999999999999,"5min":12.836,"15min":14.913},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.42299999999999999,"5min":1.2869999999999999,"15min":1.149},"min":{"1min":0.27200000000000002,"5min":0.25700000000000001,"15min":0.23000000000000001},"max":{"1min":0.91200000000000003,"5min":9.7639999999999993,"15min":14.869},"last":3.7069999999999999}]},{"osd":1,"last update":"Fri Feb 9 17:36:14 2024","interfaces":[{"interface":"back","average":{"1min":1.284,"5min":1.1080000000000001,"15min":1.3959999999999999},"min":{"1min":0.25700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":9.0079999999999991,"5min":9.0079999999999991,"15min":32.957999999999998},"last":1.0960000000000001},{"interface":"front","average":{"1min":1.304,"5min":1.2290000000000001,"15min":1.472},"min":{"1min":0.36699999999999999,"5min":0.34999999999999998,"15min":0.26700000000000002},"max":{"1min":9.0280000000000005,"5min":9.0280000000000005,"15min":32.880000000000003},"last":1.107}]},{"osd":2,"last update":"Fri Feb 9 17:31:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.415,"15min":1.6859999999999999},"min":{"1min":0.378,"5min":0.247,"15min":0.247},"max":{"1min":4.4189999999999996,"5min":9.4719999999999995,"15min":9.4719999999999995},"last":0.35399999999999998},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.5149999999999999,"15min":1.4399999999999999},"min":{"1min":0.378,"5min":0.252,"15min":0.252},"max":{"1min":4.8579999999999997,"5min":7.96,"15min":7.96},"last":0.60399999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:36:14 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":1.0409999999999999,"15min":1.4610000000000001},"min":{"1min":0.28399999999999997,"5min":0.25700000000000001,"15min":0.247},"max":{"1min":1.1319999999999999,"5min":5.0860000000000003,"15min":21.315999999999999},"last":0.439},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.2649999999999999,"15min":1.409},"min":{"1min":0.33400000000000002,"5min":0.29599999999999999,"15min":0.26400000000000001},"max":{"1min":2.2850000000000001,"5min":7.2439999999999998,"15min":18.087},"last":3.8839999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:36:14 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":2.4239999999999999,"15min":1.319},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":16.125,"5min":16.125,"15min":16.125},"last":0.5},{"interface":"front","average":{"1min":1.28,"5min":1.6479999999999999,"15min":1.143},"min":{"1min":0.33600000000000002,"5min":0.33300000000000002,"15min":0.27700000000000002},"max":{"1min":16.091999999999999,"5min":16.091999999999999,"15min":16.091999999999999},"last":0.32600000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:36:14 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":1.9259999999999999,"15min":1.2410000000000001},"min":{"1min":0.371,"5min":0.312,"15min":0.28899999999999998},"max":{"1min":3.2240000000000002,"5min":13.946999999999999,"15min":13.946999999999999},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.50700000000000001,"5min":2.133,"15min":1.298},"min":{"1min":0.31900000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.88400000000000001,"5min":13.414,"15min":13.414},"last":0.52800000000000002}]},{"osd":6,"last update":"Fri Feb 9 17:36:14 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":1.986,"15min":1.2450000000000001},"min":{"1min":0.30199999999999999,"5min":0.30099999999999999,"15min":0.29899999999999999},"max":{"1min":1.1839999999999999,"5min":12.798999999999999,"15min":12.798999999999999},"last":3.6760000000000002},{"interface":"front","average":{"1min":0.59699999999999998,"5min":1.9410000000000001,"15min":1.2669999999999999},"min":{"1min":0.33800000000000002,"5min":0.33000000000000002,"15min":0.32300000000000001},"max":{"1min":0.998,"5min":10.782999999999999,"15min":10.782999999999999},"last":0.42799999999999999}]},{"osd":7,"last update":"Fri Feb 9 17:36:14 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":2.274,"15min":1.4239999999999999},"min":{"1min":0.38900000000000001,"5min":0.312,"15min":0.312},"max":{"1min":1.4730000000000001,"5min":14.462,"15min":14.462},"last":1.984},{"interface":"front","average":{"1min":0.68200000000000005,"5min":2.3809999999999998,"15min":1.413},"min":{"1min":0.42399999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.3640000000000001,"5min":14.398999999999999,"15min":14.398999999999999},"last":0.53800000000000003}]},{"osd":8,"last update":"Fri Feb 9 17:36:46 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":1.472,"15min":2.1040000000000001},"min":{"1min":0.34799999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":2.8069999999999999,"5min":13.199999999999999,"15min":13.199999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.86099999999999999,"5min":1.26,"15min":1.7090000000000001},"min":{"1min":0.36899999999999999,"5min":0.318,"15min":0.318},"max":{"1min":3.1859999999999999,"5min":7.1520000000000001,"15min":7.1520000000000001},"last":0.55400000000000005}]},{"osd":9,"last update":"Fri Feb 9 17:37:04 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":1.417,"15min":1.2430000000000001},"min":{"1min":0.39300000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":2.839,"5min":12.821,"15min":12.821},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.747,"5min":1.276,"15min":1.0289999999999999},"min":{"1min":0.45400000000000001,"5min":0.38500000000000001,"15min":0.34100000000000003},"max":{"1min":2.153,"5min":7.6989999999999998,"15min":7.6989999999999998},"last":0.54700000000000004}]},{"osd":10,"last update":"Fri Feb 9 17:36:14 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":1.333,"15min":1.05},"min":{"1min":0.41699999999999998,"5min":0.36199999999999999,"15min":0.33900000000000002},"max":{"1min":0.79600000000000004,"5min":9.6199999999999992,"15min":9.6199999999999992},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.66200000000000003,"5min":1.498,"15min":1.069},"min":{"1min":0.45900000000000002,"5min":0.35799999999999998,"15min":0.33700000000000002},"max":{"1min":1.298,"5min":12.856,"15min":12.856},"last":0.56699999999999995}]}]},{"osd":10,"up_from":84,"seq":360777253048,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2322480,"kb_used_data":2152488,"kb_used_omap":1,"kb_used_meta":169982,"kb_avail":91422672,"statfs":{"total":95995035648,"available":93616816128,"internally_reserved":0,"allocated":2204147712,"data_stored":2203699613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":174062016},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:37:04 2024","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.5640000000000001,"15min":1.7809999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":8.5779999999999994,"5min":8.5779999999999994,"15min":8.5779999999999994},"last":9.2810000000000006},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.05,"15min":1.1020000000000001},"min":{"1min":0.27800000000000002,"5min":0.253,"15min":0.253},"max":{"1min":5.923,"5min":5.923,"15min":5.923},"last":2.1960000000000002}]},{"osd":1,"last update":"Fri Feb 9 17:33:46 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":1.345,"15min":1.1599999999999999},"min":{"1min":0.25900000000000001,"5min":0.24199999999999999,"15min":0.23499999999999999},"max":{"1min":3.1709999999999998,"5min":37.716000000000001,"15min":37.716000000000001},"last":0.26000000000000001},{"interface":"front","average":{"1min":1.121,"5min":1.3660000000000001,"15min":1.194},"min":{"1min":0.251,"5min":0.251,"15min":0.245},"max":{"1min":5.3049999999999997,"5min":38.039999999999999,"15min":38.039999999999999},"last":0.27800000000000002}]},{"osd":2,"last update":"Fri Feb 9 17:36:51 2024","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":0.98799999999999999,"15min":1.1910000000000001},"min":{"1min":0.29799999999999999,"5min":0.27100000000000002,"15min":0.222},"max":{"1min":3.7440000000000002,"5min":6.4199999999999999,"15min":13.351000000000001},"last":5.6900000000000004},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.089,"15min":1.2},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.26100000000000001},"max":{"1min":3.7770000000000001,"5min":6.4829999999999997,"15min":10.596},"last":5.6740000000000004}]},{"osd":3,"last update":"Fri Feb 9 17:36:51 2024","interfaces":[{"interface":"back","average":{"1min":1.6439999999999999,"5min":2.0179999999999998,"15min":1.573},"min":{"1min":0.28699999999999998,"5min":0.26800000000000002,"15min":0.26200000000000001},"max":{"1min":8.8800000000000008,"5min":25.972999999999999,"15min":25.972999999999999},"last":0.53000000000000003},{"interface":"front","average":{"1min":1.4390000000000001,"5min":1.7829999999999999,"15min":1.4390000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":8.2509999999999994,"5min":25.882000000000001,"15min":25.882000000000001},"last":0.67300000000000004}]},{"osd":4,"last update":"Fri Feb 9 17:36:55 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":1.274,"15min":1.012},"min":{"1min":0.307,"5min":0.307,"15min":0.26100000000000001},"max":{"1min":3.589,"5min":7.1360000000000001,"15min":7.1360000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.83599999999999997,"5min":1.2949999999999999,"15min":1.0409999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":3.5419999999999998,"5min":6.0119999999999996,"15min":6.8369999999999997},"last":0.32400000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:36:51 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":1.282,"15min":1.2},"min":{"1min":0.30199999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.9809999999999999,"5min":9.2349999999999994,"15min":9.2349999999999994},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.95399999999999996,"5min":1.3029999999999999,"15min":1.157},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.32200000000000001},"max":{"1min":3.8959999999999999,"5min":5.3079999999999998,"15min":8.6159999999999997},"last":0.374}]},{"osd":6,"last update":"Fri Feb 9 17:36:51 2024","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":1.331,"15min":1.107},"min":{"1min":0.36899999999999999,"5min":0.29999999999999999,"15min":0.29099999999999998},"max":{"1min":5.6550000000000002,"5min":6.3760000000000003,"15min":6.8799999999999999},"last":0.502},{"interface":"front","average":{"1min":0.93600000000000005,"5min":1.7749999999999999,"15min":1.3169999999999999},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":3.1389999999999998,"5min":9.4350000000000005,"15min":9.4350000000000005},"last":0.43099999999999999}]},{"osd":7,"last update":"Fri Feb 9 17:36:51 2024","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":1.778,"15min":1.242},"min":{"1min":0.39600000000000002,"5min":0.33500000000000002,"15min":0.312},"max":{"1min":3.3180000000000001,"5min":17.754000000000001,"15min":17.754000000000001},"last":0.61399999999999999},{"interface":"front","average":{"1min":1.03,"5min":1.958,"15min":1.3580000000000001},"min":{"1min":0.45100000000000001,"5min":0.42599999999999999,"15min":0.316},"max":{"1min":3.665,"5min":11.308999999999999,"15min":11.308999999999999},"last":0.56899999999999995}]},{"osd":8,"last update":"Fri Feb 9 17:36:35 2024","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":1.081,"15min":1.1950000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":2.3130000000000002,"5min":6.4630000000000001,"15min":6.4630000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.84499999999999997,"15min":0.94999999999999996},"min":{"1min":0.38900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":3.794,"5min":5.3079999999999998,"15min":5.3079999999999998},"last":0.55500000000000005}]},{"osd":9,"last update":"Fri Feb 9 17:36:51 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.97899999999999998,"15min":0.94899999999999995},"min":{"1min":0.372,"5min":0.36099999999999999,"15min":0.314},"max":{"1min":3.1070000000000002,"5min":6.5049999999999999,"15min":8.7240000000000002},"last":0.65400000000000003},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.048,"15min":1.0940000000000001},"min":{"1min":0.40300000000000002,"5min":0.35899999999999999,"15min":0.29799999999999999},"max":{"1min":4.1029999999999998,"5min":6.9640000000000004,"15min":10.1},"last":0.59999999999999998}]},{"osd":11,"last update":"Fri Feb 9 17:36:15 2024","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":1.1379999999999999,"15min":1.1020000000000001},"min":{"1min":0.39800000000000002,"5min":0.379,"15min":0.36399999999999999},"max":{"1min":2.7909999999999999,"5min":7.383,"15min":8.6430000000000007},"last":0.40600000000000003},{"interface":"front","average":{"1min":1.002,"5min":1.196,"15min":1.1080000000000001},"min":{"1min":0.45000000000000001,"5min":0.377,"15min":0.33400000000000002},"max":{"1min":2.831,"5min":7.3019999999999996,"15min":9.298},"last":0.45700000000000002}]}]},{"osd":9,"up_from":77,"seq":330712481982,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1873716,"kb_used_data":1581548,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":91871436,"statfs":{"total":95995035648,"available":94076350464,"internally_reserved":0,"allocated":1619505152,"data_stored":1619052981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":299170240},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:28:03 2024","interfaces":[{"interface":"back","average":{"1min":1.9510000000000001,"5min":1.454,"15min":0.85799999999999998},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":8.4499999999999993,"5min":17.495999999999999,"15min":17.495999999999999},"last":2.7360000000000002},{"interface":"front","average":{"1min":1.905,"5min":1.399,"15min":0.83499999999999996},"min":{"1min":0.32200000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":7.5990000000000002,"5min":15.401999999999999,"15min":15.401999999999999},"last":1.5449999999999999}]},{"osd":1,"last update":"Fri Feb 9 17:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.88400000000000001,"15min":1.2070000000000001},"min":{"1min":0.27100000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":4.601,"5min":5.2549999999999999,"15min":16.527999999999999},"last":0.48399999999999999},{"interface":"front","average":{"1min":1.0209999999999999,"5min":0.88500000000000001,"15min":1.095},"min":{"1min":0.255,"5min":0.222,"15min":0.222},"max":{"1min":4.282,"5min":4.282,"15min":17.771999999999998},"last":4.1379999999999999}]},{"osd":2,"last update":"Fri Feb 9 17:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":1.038,"15min":1.131},"min":{"1min":0.28399999999999997,"5min":0.249,"15min":0.249},"max":{"1min":1.8320000000000001,"5min":6.0570000000000004,"15min":13.878},"last":0.311},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.997,"15min":1.2},"min":{"1min":0.26100000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":4.0599999999999996,"5min":5.5590000000000002,"15min":13.69},"last":0.44600000000000001}]},{"osd":3,"last update":"Fri Feb 9 17:36:28 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":1.4199999999999999,"15min":1.202},"min":{"1min":0.307,"5min":0.29699999999999999,"15min":0.26000000000000001},"max":{"1min":2.0950000000000002,"5min":40.18,"15min":40.18},"last":4.6970000000000001},{"interface":"front","average":{"1min":0.75900000000000001,"5min":1.4370000000000001,"15min":1.391},"min":{"1min":0.35699999999999998,"5min":0.30599999999999999,"15min":0.27900000000000003},"max":{"1min":3.4420000000000002,"5min":40.158000000000001,"15min":40.158000000000001},"last":4.7949999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":1.7070000000000001,"15min":1.2649999999999999},"min":{"1min":0.33800000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":2.2450000000000001,"5min":9.0519999999999996,"15min":22.577999999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":1.54,"15min":1.1399999999999999},"min":{"1min":0.33300000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":2.2389999999999999,"5min":12.651,"15min":22.399999999999999},"last":0.59799999999999998}]},{"osd":5,"last update":"Fri Feb 9 17:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":1.7949999999999999,"15min":1.2310000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.8049999999999999,"5min":10.689,"15min":10.689},"last":0.379},{"interface":"front","average":{"1min":0.71499999999999997,"5min":1.3580000000000001,"15min":1.042},"min":{"1min":0.378,"5min":0.311,"15min":0.311},"max":{"1min":2.202,"5min":6.4470000000000001,"15min":6.4470000000000001},"last":0.57099999999999995}]},{"osd":6,"last update":"Fri Feb 9 17:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":1.734,"15min":1.2589999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.32200000000000001},"max":{"1min":3.0590000000000002,"5min":9.1699999999999999,"15min":9.1699999999999999},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.93100000000000005,"5min":1.7729999999999999,"15min":1.2589999999999999},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.35099999999999998},"max":{"1min":3.1110000000000002,"5min":9.2080000000000002,"15min":9.2080000000000002},"last":0.40300000000000002}]},{"osd":7,"last update":"Fri Feb 9 17:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":1.73,"15min":1.224},"min":{"1min":0.40999999999999998,"5min":0.38500000000000001,"15min":0.34300000000000003},"max":{"1min":3.2320000000000002,"5min":7.3710000000000004,"15min":8.0939999999999994},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.98099999999999998,"5min":1.7709999999999999,"15min":1.2909999999999999},"min":{"1min":0.38400000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":3.262,"5min":8.0530000000000008,"15min":8.0530000000000008},"last":0.55700000000000005}]},{"osd":8,"last update":"Fri Feb 9 17:36:46 2024","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":1.157,"15min":1.2789999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":4.141,"5min":5.5069999999999997,"15min":16.209},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.999,"5min":1.1180000000000001,"15min":1.29},"min":{"1min":0.41299999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":2.492,"5min":10.241,"15min":16.263000000000002},"last":4.4770000000000003}]},{"osd":10,"last update":"Fri Feb 9 17:36:58 2024","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":1.111,"15min":1.071},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":2.7189999999999999,"5min":6.0659999999999998,"15min":9.5090000000000003},"last":4.5039999999999996},{"interface":"front","average":{"1min":1.022,"5min":0.94799999999999995,"15min":1.0169999999999999},"min":{"1min":0.40000000000000002,"5min":0.38500000000000001,"15min":0.36099999999999999},"max":{"1min":2.395,"5min":4.0759999999999996,"15min":8.0670000000000002},"last":5.1609999999999996}]},{"osd":11,"last update":"Fri Feb 9 17:36:58 2024","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.4710000000000001,"15min":1.3340000000000001},"min":{"1min":0.42599999999999999,"5min":0.42399999999999999,"15min":0.34699999999999998},"max":{"1min":7.2190000000000003,"5min":11.541,"15min":11.541},"last":4.718},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.405,"15min":1.147},"min":{"1min":0.51100000000000001,"5min":0.374,"15min":0.374},"max":{"1min":3.1600000000000001,"5min":10.32,"15min":10.32},"last":4.7599999999999998}]}]},{"osd":8,"up_from":171,"seq":734439407724,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":132840,"kb_used_data":1192,"kb_used_omap":212,"kb_used_meta":131243,"kb_avail":93612312,"statfs":{"total":95995035648,"available":95859007488,"internally_reserved":0,"allocated":1220608,"data_stored":772509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217579,"internal_metadata":134393365},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:36:31 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":0.98499999999999999,"15min":1.3899999999999999},"min":{"1min":0.246,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":3.0129999999999999,"5min":8.0289999999999999,"15min":8.0289999999999999},"last":1.171},{"interface":"front","average":{"1min":0.89200000000000002,"5min":1.0589999999999999,"15min":1.3660000000000001},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":3.0960000000000001,"5min":7.5990000000000002,"15min":7.7969999999999997},"last":1.609}]},{"osd":1,"last update":"Fri Feb 9 17:36:31 2024","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.2470000000000001,"15min":1.2669999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.0810000000000004,"5min":7.8719999999999999,"15min":7.8719999999999999},"last":3.5059999999999998},{"interface":"front","average":{"1min":1.05,"5min":1.1879999999999999,"15min":1.841},"min":{"1min":0.23400000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":3.6680000000000001,"5min":8.5519999999999996,"15min":11.101000000000001},"last":0.93700000000000006}]},{"osd":2,"last update":"Fri Feb 9 17:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":1.522,"15min":1.45},"min":{"1min":0.29399999999999998,"5min":0.28199999999999997,"15min":0.255},"max":{"1min":3.8239999999999998,"5min":10.423,"15min":10.423},"last":2.3260000000000001},{"interface":"front","average":{"1min":0.93400000000000005,"5min":1.3080000000000001,"15min":1.611},"min":{"1min":0.373,"5min":0.34300000000000003,"15min":0.313},"max":{"1min":3.847,"5min":10.504,"15min":10.504},"last":2.359}]},{"osd":3,"last update":"Fri Feb 9 17:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":1.4790000000000001,"15min":2.0289999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.28699999999999998},"max":{"1min":3.7210000000000001,"5min":24.030999999999999,"15min":24.030999999999999},"last":1.341},{"interface":"front","average":{"1min":1.163,"5min":1.5880000000000001,"15min":2.327},"min":{"1min":0.45500000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":3.1110000000000002,"5min":24.318999999999999,"15min":24.318999999999999},"last":3.254}]},{"osd":4,"last update":"Fri Feb 9 17:36:31 2024","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":1.8500000000000001,"15min":1.238},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.27500000000000002},"max":{"1min":3.2010000000000001,"5min":53.060000000000002,"15min":53.060000000000002},"last":0.495},{"interface":"front","average":{"1min":0.60999999999999999,"5min":2.0760000000000001,"15min":1.522},"min":{"1min":0.32900000000000001,"5min":0.313,"15min":0.313},"max":{"1min":1.4430000000000001,"5min":53.311999999999998,"15min":53.311999999999998},"last":0.52000000000000002}]},{"osd":6,"last update":"Fri Feb 9 17:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":1.7470000000000001,"15min":1.589},"min":{"1min":0.36399999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":2.8980000000000001,"5min":10.619999999999999,"15min":37.356999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.96999999999999997,"5min":1.3999999999999999,"15min":1.143},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":3.0739999999999998,"5min":4.984,"15min":37.25},"last":0.58999999999999997}]},{"osd":7,"last update":"Fri Feb 9 17:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":1.9039999999999999,"15min":1.4750000000000001},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":3.4399999999999999,"5min":12.731999999999999,"15min":37.585000000000001},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.83399999999999996,"5min":1.9430000000000001,"15min":1.6459999999999999},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.312},"max":{"1min":3.4710000000000001,"5min":12.779,"15min":37.918999999999997},"last":0.60699999999999998}]},{"osd":9,"last update":"Fri Feb 9 17:36:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.2190000000000001,"15min":1.1870000000000001},"min":{"1min":0.44900000000000001,"5min":0.34100000000000003,"15min":0.29699999999999999},"max":{"1min":3.2440000000000002,"5min":9.4900000000000002,"15min":9.4900000000000002},"last":0.56200000000000006},{"interface":"front","average":{"1min":1.006,"5min":1.266,"15min":1.23},"min":{"1min":0.36599999999999999,"5min":0.35399999999999998,"15min":0.32000000000000001},"max":{"1min":4.101,"5min":17.434999999999999,"15min":17.434999999999999},"last":0.57899999999999996}]},{"osd":10,"last update":"Fri Feb 9 17:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":1.127,"15min":1.1910000000000001},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.33700000000000002},"max":{"1min":1.3979999999999999,"5min":10.65,"15min":10.65},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.99199999999999999,"5min":1.1699999999999999,"15min":1.0820000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":3.024,"5min":8.2799999999999994,"15min":8.2799999999999994},"last":0.85899999999999999}]},{"osd":11,"last update":"Fri Feb 9 17:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":1.5760000000000001,"15min":1.381},"min":{"1min":0.42899999999999999,"5min":0.375,"15min":0.375},"max":{"1min":5.4980000000000002,"5min":14.433999999999999,"15min":14.433999999999999},"last":0.72299999999999998},{"interface":"front","average":{"1min":1.012,"5min":1.3500000000000001,"15min":1.4610000000000001},"min":{"1min":0.48399999999999999,"5min":0.45000000000000001,"15min":0.38100000000000001},"max":{"1min":5.3239999999999998,"5min":11.622,"15min":11.622},"last":0.75600000000000001}]}]},{"osd":7,"up_from":57,"seq":244813136070,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1961968,"kb_used_data":1633704,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":91783184,"statfs":{"total":95995035648,"available":93985980416,"internally_reserved":0,"allocated":1672912896,"data_stored":1672464797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":336132554},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:37:04 2024","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":2.6120000000000001,"15min":3.294},"min":{"1min":0.26200000000000001,"5min":0.254,"15min":0.254},"max":{"1min":4.1719999999999997,"5min":36.753,"15min":36.753},"last":4.3920000000000003},{"interface":"front","average":{"1min":1.913,"5min":2.9369999999999998,"15min":3.5489999999999999},"min":{"1min":0.29199999999999998,"5min":0.253,"15min":0.253},"max":{"1min":9.0640000000000001,"5min":36.918999999999997,"15min":36.918999999999997},"last":7.3070000000000004}]},{"osd":1,"last update":"Fri Feb 9 17:37:08 2024","interfaces":[{"interface":"back","average":{"1min":1.3600000000000001,"5min":1.9610000000000001,"15min":1.9350000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":6.3920000000000003,"5min":20.553999999999998,"15min":20.553999999999998},"last":6.8230000000000004},{"interface":"front","average":{"1min":2.6219999999999999,"5min":2.0099999999999998,"15min":1.657},"min":{"1min":0.28399999999999997,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":8.6430000000000007,"5min":20.568000000000001,"15min":20.568000000000001},"last":14.007999999999999}]},{"osd":2,"last update":"Fri Feb 9 17:36:29 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.8029999999999999,"15min":1.456},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.26000000000000001},"max":{"1min":2.722,"5min":11.324,"15min":11.324},"last":7.4109999999999996},{"interface":"front","average":{"1min":1.0569999999999999,"5min":2.004,"15min":1.5349999999999999},"min":{"1min":0.32100000000000001,"5min":0.309,"15min":0.28299999999999997},"max":{"1min":2.742,"5min":19.361999999999998,"15min":19.361999999999998},"last":7.3819999999999997}]},{"osd":3,"last update":"Fri Feb 9 17:36:29 2024","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":2.077,"15min":1.554},"min":{"1min":0.36399999999999999,"5min":0.28899999999999998,"15min":0.26900000000000002},"max":{"1min":2.7010000000000001,"5min":20.754000000000001,"15min":27.719000000000001},"last":6.923},{"interface":"front","average":{"1min":0.79700000000000004,"5min":2.2149999999999999,"15min":1.5780000000000001},"min":{"1min":0.34799999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":1.958,"5min":24.457000000000001,"15min":24.457000000000001},"last":5.6479999999999997}]},{"osd":4,"last update":"Fri Feb 9 17:36:29 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":2.3319999999999999,"15min":1.286},"min":{"1min":0.28000000000000003,"5min":0.25700000000000001,"15min":0.248},"max":{"1min":2.544,"5min":14.372999999999999,"15min":14.372999999999999},"last":4.952},{"interface":"front","average":{"1min":0.77700000000000002,"5min":2.415,"15min":1.2909999999999999},"min":{"1min":0.379,"5min":0.34699999999999998,"15min":0.22800000000000001},"max":{"1min":2.8599999999999999,"5min":14.401,"15min":14.401},"last":4.9900000000000002}]},{"osd":5,"last update":"Fri Feb 9 17:35:28 2024","interfaces":[{"interface":"back","average":{"1min":2.073,"5min":2.2999999999999998,"15min":1.1619999999999999},"min":{"1min":0.36899999999999999,"5min":0.28599999999999998,"15min":0.21299999999999999},"max":{"1min":13.409000000000001,"5min":13.409000000000001,"15min":13.409000000000001},"last":0.53300000000000003},{"interface":"front","average":{"1min":1.946,"5min":2.1230000000000002,"15min":1.1240000000000001},"min":{"1min":0.30599999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":10.553000000000001,"5min":11.598000000000001,"15min":11.598000000000001},"last":0.48999999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:36:29 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":2.3559999999999999,"15min":1.3700000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":2.6909999999999998,"5min":25.965,"15min":25.965},"last":7.3479999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":1.8959999999999999,"15min":1.155},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.28699999999999998},"max":{"1min":1.6279999999999999,"5min":9.4090000000000007,"15min":9.4090000000000007},"last":4.875}]},{"osd":8,"last update":"Fri Feb 9 17:36:29 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":1.6970000000000001,"15min":1.4970000000000001},"min":{"1min":0.442,"5min":0.36599999999999999,"15min":0.32300000000000001},"max":{"1min":2.641,"5min":9.0809999999999995,"15min":9.0809999999999995},"last":4.4340000000000002},{"interface":"front","average":{"1min":0.80000000000000004,"5min":1.635,"15min":1.2529999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":2.5070000000000001,"5min":11.481,"15min":11.481},"last":0.499}]},{"osd":9,"last update":"Fri Feb 9 17:36:16 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":1.847,"15min":1.395},"min":{"1min":0.438,"5min":0.40899999999999997,"15min":0.39600000000000002},"max":{"1min":2.7690000000000001,"5min":8.9900000000000002,"15min":41.286999999999999},"last":4.5119999999999996},{"interface":"front","average":{"1min":1.127,"5min":1.9259999999999999,"15min":1.4330000000000001},"min":{"1min":0.41399999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":7.3579999999999997,"5min":11.417999999999999,"15min":41.234999999999999},"last":4.633}]},{"osd":10,"last update":"Fri Feb 9 17:36:44 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":1.8109999999999999,"15min":1.3009999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.35899999999999999},"max":{"1min":1.6399999999999999,"5min":9.4830000000000005,"15min":9.4830000000000005},"last":4.8490000000000002},{"interface":"front","average":{"1min":0.95299999999999996,"5min":2.044,"15min":1.385},"min":{"1min":0.52200000000000002,"5min":0.45600000000000002,"15min":0.35599999999999998},"max":{"1min":2.5289999999999999,"5min":9.109,"15min":9.109},"last":4.4809999999999999}]},{"osd":11,"last update":"Fri Feb 9 17:37:08 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":2.2519999999999998,"15min":1.5169999999999999},"min":{"1min":0.44500000000000001,"5min":0.41299999999999998,"15min":0.40999999999999998},"max":{"1min":6.6360000000000001,"5min":12.885999999999999,"15min":12.885999999999999},"last":4.8129999999999997},{"interface":"front","average":{"1min":1.444,"5min":1.883,"15min":1.3620000000000001},"min":{"1min":0.5,"5min":0.41299999999999998,"15min":0.39500000000000002},"max":{"1min":5.6230000000000002,"5min":9.9489999999999998,"15min":9.9489999999999998},"last":4.6050000000000004}]}]},{"osd":6,"up_from":49,"seq":210453397708,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3459184,"kb_used_data":3175336,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":90285968,"statfs":{"total":95995035648,"available":92452831232,"internally_reserved":0,"allocated":3251544064,"data_stored":3251095965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":290650566},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:37:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.6020000000000001,"15min":1.369},"min":{"1min":0.26300000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.4510000000000001,"5min":12.544,"15min":24.544},"last":4.077},{"interface":"front","average":{"1min":1.296,"5min":1.6060000000000001,"15min":1.302},"min":{"1min":0.35499999999999998,"5min":0.26500000000000001,"15min":0.217},"max":{"1min":4.4950000000000001,"5min":8.3689999999999998,"15min":8.3689999999999998},"last":4.1120000000000001}]},{"osd":1,"last update":"Fri Feb 9 17:37:04 2024","interfaces":[{"interface":"back","average":{"1min":1.6950000000000001,"5min":2.2029999999999998,"15min":1.579},"min":{"1min":0.30399999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":6.8380000000000001,"5min":17.762,"15min":23.954000000000001},"last":4.4160000000000004},{"interface":"front","average":{"1min":1.7250000000000001,"5min":2.2639999999999998,"15min":1.625},"min":{"1min":0.34499999999999997,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":6.8929999999999998,"5min":17.734999999999999,"15min":23.984999999999999},"last":4.484}]},{"osd":2,"last update":"Fri Feb 9 17:37:04 2024","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.623,"15min":1.462},"min":{"1min":0.34000000000000002,"5min":0.28999999999999998,"15min":0.22600000000000001},"max":{"1min":4.0830000000000002,"5min":8.0239999999999991,"15min":24.329999999999998},"last":4.3979999999999997},{"interface":"front","average":{"1min":1.268,"5min":1.6040000000000001,"15min":1.4690000000000001},"min":{"1min":0.377,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":4.1159999999999997,"5min":10.156000000000001,"15min":12.906000000000001},"last":4.4299999999999997}]},{"osd":3,"last update":"Fri Feb 9 17:37:04 2024","interfaces":[{"interface":"back","average":{"1min":1.8069999999999999,"5min":2.2759999999999998,"15min":1.7490000000000001},"min":{"1min":0.30099999999999999,"5min":0.25600000000000001,"15min":0.248},"max":{"1min":8.7780000000000005,"5min":16.931999999999999,"15min":32.545000000000002},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.82299999999999995,"5min":2.2120000000000002,"15min":1.657},"min":{"1min":0.31900000000000001,"5min":0.29599999999999999,"15min":0.26600000000000001},"max":{"1min":4.2510000000000003,"5min":17.047000000000001,"15min":32.670999999999999},"last":4.4429999999999996}]},{"osd":4,"last update":"Fri Feb 9 17:37:04 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":2.2879999999999998,"15min":1.286},"min":{"1min":0.255,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":5.944,"5min":19.798999999999999,"15min":19.798999999999999},"last":4.3840000000000003},{"interface":"front","average":{"1min":0.92500000000000004,"5min":1.8100000000000001,"15min":1.2230000000000001},"min":{"1min":0.377,"5min":0.28000000000000003,"15min":0.26000000000000001},"max":{"1min":5.9260000000000002,"5min":10.702999999999999,"15min":10.702999999999999},"last":4.1840000000000002}]},{"osd":5,"last update":"Fri Feb 9 17:37:04 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":1.909,"15min":1.198},"min":{"1min":0.318,"5min":0.29499999999999998,"15min":0.29199999999999998},"max":{"1min":2.2090000000000001,"5min":10.968999999999999,"15min":10.968999999999999},"last":4.1550000000000002},{"interface":"front","average":{"1min":0.73699999999999999,"5min":2.105,"15min":1.26},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.27400000000000002},"max":{"1min":2.2370000000000001,"5min":13.91,"15min":13.91},"last":4.3360000000000003}]},{"osd":7,"last update":"Fri Feb 9 17:36:39 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":2.1680000000000001,"15min":1.2869999999999999},"min":{"1min":0.32200000000000001,"5min":0.311,"15min":0.28000000000000003},"max":{"1min":5.0579999999999998,"5min":14.022,"15min":14.022},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.90300000000000002,"5min":2.2730000000000001,"15min":1.405},"min":{"1min":0.36699999999999999,"5min":0.35699999999999998,"15min":0.309},"max":{"1min":2.9420000000000002,"5min":15.458,"15min":15.458},"last":4.4690000000000003}]},{"osd":8,"last update":"Fri Feb 9 17:26:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":0.876,"15min":0.79600000000000004},"min":{"1min":0.39900000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":4.0339999999999998,"5min":4.4880000000000004,"15min":4.4880000000000004},"last":4.3840000000000003},{"interface":"front","average":{"1min":1.0469999999999999,"5min":0.88600000000000001,"15min":0.78300000000000003},"min":{"1min":0.375,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":3.6539999999999999,"5min":4.8570000000000002,"15min":4.8570000000000002},"last":4.4210000000000003}]},{"osd":9,"last update":"Fri Feb 9 17:37:12 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":1.494,"15min":1.1930000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.1779999999999999,"5min":8.3409999999999993,"15min":10.803000000000001},"last":4.3140000000000001},{"interface":"front","average":{"1min":0.70999999999999996,"5min":1.629,"15min":1.2909999999999999},"min":{"1min":0.40999999999999998,"5min":0.40400000000000003,"15min":0.40300000000000002},"max":{"1min":1.7190000000000001,"5min":8.1630000000000003,"15min":8.1630000000000003},"last":4.2110000000000003}]},{"osd":10,"last update":"Fri Feb 9 17:36:39 2024","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":1.631,"15min":1.325},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.35399999999999998},"max":{"1min":5.5469999999999997,"5min":7.734,"15min":9.5399999999999991},"last":4.5119999999999996},{"interface":"front","average":{"1min":1.0269999999999999,"5min":2.0129999999999999,"15min":1.417},"min":{"1min":0.41999999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":4.8819999999999997,"5min":13.836,"15min":13.836},"last":0.70399999999999996}]},{"osd":11,"last update":"Fri Feb 9 17:37:04 2024","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.774,"15min":1.5640000000000001},"min":{"1min":0.40799999999999997,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":4.1609999999999996,"5min":10.273999999999999,"15min":10.273999999999999},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.746,"5min":1.893,"15min":1.7070000000000001},"min":{"1min":0.45100000000000001,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.9279999999999999,"5min":8.2739999999999991,"15min":8.2739999999999991},"last":4.2359999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575078,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5143668,"kb_used_data":4797356,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":88601484,"statfs":{"total":95995035648,"available":90727919616,"internally_reserved":0,"allocated":4912492544,"data_stored":4912040373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,5,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:37:10 2024","interfaces":[{"interface":"back","average":{"1min":3.6230000000000002,"5min":1.581,"15min":1.4199999999999999},"min":{"1min":0.26300000000000001,"5min":0.223,"15min":0.214},"max":{"1min":14.917,"5min":14.917,"15min":14.917},"last":3.8809999999999998},{"interface":"front","average":{"1min":2.3279999999999998,"5min":1.4430000000000001,"15min":1.387},"min":{"1min":0.30099999999999999,"5min":0.245,"15min":0.214},"max":{"1min":8.7260000000000009,"5min":8.7260000000000009,"15min":9.8439999999999994},"last":0.19400000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:36:39 2024","interfaces":[{"interface":"back","average":{"1min":2.4430000000000001,"5min":1.423,"15min":1.474},"min":{"1min":0.27600000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":26.361999999999998,"5min":26.361999999999998,"15min":26.361999999999998},"last":0.83699999999999997},{"interface":"front","average":{"1min":2.3650000000000002,"5min":1.397,"15min":1.597},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":26.087,"5min":26.087,"15min":26.087},"last":4.125}]},{"osd":3,"last update":"Fri Feb 9 17:36:32 2024","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":8.5030000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":3.7090000000000001},{"interface":"front","average":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.3480000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":4.0250000000000004,"5min":4.0250000000000004,"15min":4.0250000000000004},"last":3.8439999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:37:10 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":1.6950000000000001,"15min":2.0289999999999999},"min":{"1min":0.317,"5min":0.253,"15min":0.253},"max":{"1min":8.3460000000000001,"5min":11.766999999999999,"15min":11.766999999999999},"last":0.97899999999999998},{"interface":"front","average":{"1min":2.3069999999999999,"5min":1.923,"15min":1.9319999999999999},"min":{"1min":0.309,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":10.512,"5min":10.512,"15min":10.512},"last":0.374}]},{"osd":5,"last update":"Fri Feb 9 17:36:50 2024","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.774,"15min":1.518},"min":{"1min":0.374,"5min":0.28599999999999998,"15min":0.27000000000000002},"max":{"1min":8.2859999999999996,"5min":13.544,"15min":13.544},"last":0.95199999999999996},{"interface":"front","average":{"1min":2.0089999999999999,"5min":1.839,"15min":1.353},"min":{"1min":0.36699999999999999,"5min":0.26800000000000002,"15min":0.26000000000000001},"max":{"1min":4.843,"5min":12.117000000000001,"15min":12.117000000000001},"last":0.41299999999999998}]},{"osd":6,"last update":"Fri Feb 9 17:37:10 2024","interfaces":[{"interface":"back","average":{"1min":1.8959999999999999,"5min":1.732,"15min":1.5880000000000001},"min":{"1min":0.35999999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":8.25,"5min":8.7210000000000001,"15min":10.275},"last":0.92700000000000005},{"interface":"front","average":{"1min":2.0209999999999999,"5min":1.921,"15min":1.5860000000000001},"min":{"1min":0.41799999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":8.5999999999999996,"5min":9.0069999999999997,"15min":9.0069999999999997},"last":0.35799999999999998}]},{"osd":7,"last update":"Fri Feb 9 17:34:33 2024","interfaces":[{"interface":"back","average":{"1min":2.1560000000000001,"5min":1.9099999999999999,"15min":1.5609999999999999},"min":{"1min":0.35699999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":9.0489999999999995,"5min":15.148999999999999,"15min":15.148999999999999},"last":0.76600000000000001},{"interface":"front","average":{"1min":1.419,"5min":1.3740000000000001,"15min":1.425},"min":{"1min":0.31,"5min":0.31,"15min":0.28299999999999997},"max":{"1min":7.3380000000000001,"5min":13.834,"15min":13.834},"last":0.61499999999999999}]},{"osd":8,"last update":"Fri Feb 9 17:36:32 2024","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.2929999999999999,"15min":1.3959999999999999},"min":{"1min":0.39200000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":3.8079999999999998,"5min":7.4349999999999996,"15min":7.4349999999999996},"last":0.433},{"interface":"front","average":{"1min":1.333,"5min":1.569,"15min":1.9790000000000001},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":8.3780000000000001,"5min":10.433999999999999,"15min":10.433999999999999},"last":1.0009999999999999}]},{"osd":9,"last update":"Fri Feb 9 17:36:17 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":1.464,"15min":1.4710000000000001},"min":{"1min":0.41199999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":3.548,"5min":9.5839999999999996,"15min":9.5839999999999996},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.99199999999999999,"5min":1.288,"15min":1.4530000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.30099999999999999},"max":{"1min":4.5389999999999997,"5min":8.4079999999999995,"15min":12.117000000000001},"last":0.40100000000000002}]},{"osd":10,"last update":"Fri Feb 9 17:36:50 2024","interfaces":[{"interface":"back","average":{"1min":1.9099999999999999,"5min":1.425,"15min":1.321},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.35999999999999999},"max":{"1min":4.8810000000000002,"5min":7.9809999999999999,"15min":7.9809999999999999},"last":0.46200000000000002},{"interface":"front","average":{"1min":1.958,"5min":1.512,"15min":1.581},"min":{"1min":0.48099999999999998,"5min":0.44400000000000001,"15min":0.33400000000000002},"max":{"1min":4.9080000000000004,"5min":8.0020000000000007,"15min":12.087999999999999},"last":0.49399999999999999}]},{"osd":11,"last update":"Fri Feb 9 17:36:56 2024","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":1.585,"15min":2.1200000000000001},"min":{"1min":0.437,"5min":0.42199999999999999,"15min":0.32000000000000001},"max":{"1min":5.4640000000000004,"5min":8.3079999999999998,"15min":12.169},"last":0.45000000000000001},{"interface":"front","average":{"1min":1.5509999999999999,"5min":1.619,"15min":2.0379999999999998},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.36599999999999999},"max":{"1min":5.4880000000000004,"5min":8.4339999999999993,"15min":12.144},"last":0.47999999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738602,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1985392,"kb_used_data":1665448,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":91759760,"statfs":{"total":95995035648,"available":93961994240,"internally_reserved":0,"allocated":1705418752,"data_stored":1704970653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":327612869},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 9 17:36:56 2024","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.1919999999999999,"15min":1.2809999999999999},"min":{"1min":0.22800000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":3.5030000000000001,"5min":11.236000000000001,"15min":14.166},"last":1.341},{"interface":"front","average":{"1min":2.4870000000000001,"5min":1.3380000000000001,"15min":1.5209999999999999},"min":{"1min":0.32000000000000001,"5min":0.26800000000000002,"15min":0.215},"max":{"1min":7.8579999999999997,"5min":9.5649999999999995,"15min":20.177},"last":2.3170000000000002}]},{"osd":2,"last update":"Fri Feb 9 17:27:44 2024","interfaces":[{"interface":"back","average":{"1min":3.976,"5min":3.976,"15min":3.976},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":13.066000000000001,"5min":13.066000000000001,"15min":13.066000000000001},"last":3.0569999999999999},{"interface":"front","average":{"1min":3.968,"5min":3.968,"15min":3.968},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":13.193,"5min":13.193,"15min":13.193},"last":10.872999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:36:50 2024","interfaces":[{"interface":"back","average":{"1min":2.21,"5min":1.7290000000000001,"15min":1.5489999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":6.1139999999999999,"5min":16.472000000000001,"15min":16.472000000000001},"last":0.27000000000000002},{"interface":"front","average":{"1min":2.1070000000000002,"5min":1.3919999999999999,"15min":1.399},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.223},"max":{"1min":15.132999999999999,"5min":15.132999999999999,"15min":16.984999999999999},"last":1.379}]},{"osd":4,"last update":"Fri Feb 9 17:36:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":1.8979999999999999,"15min":1.504},"min":{"1min":0.30299999999999999,"5min":0.29999999999999999,"15min":0.253},"max":{"1min":3.4630000000000001,"5min":14.419,"15min":14.419},"last":1.4079999999999999},{"interface":"front","average":{"1min":0.56000000000000005,"5min":2.137,"15min":1.6120000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.5720000000000001,"5min":14.465999999999999,"15min":14.465999999999999},"last":0.28499999999999998}]},{"osd":5,"last update":"Fri Feb 9 17:36:40 2024","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.6539999999999999,"15min":1.5029999999999999},"min":{"1min":0.36499999999999999,"5min":0.32400000000000001,"15min":0.28999999999999998},"max":{"1min":3.5259999999999998,"5min":9.532,"15min":9.8569999999999993},"last":1.429},{"interface":"front","average":{"1min":1.506,"5min":1.9370000000000001,"15min":1.6399999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.26700000000000002},"max":{"1min":5.7919999999999998,"5min":9.5640000000000001,"15min":9.9049999999999994},"last":1.4850000000000001}]},{"osd":6,"last update":"Fri Feb 9 17:36:13 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":2.0550000000000002,"15min":1.5840000000000001},"min":{"1min":0.34499999999999997,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.6339999999999999,"5min":13.102,"15min":13.102},"last":1.216},{"interface":"front","average":{"1min":0.78600000000000003,"5min":2.1299999999999999,"15min":1.583},"min":{"1min":0.38200000000000001,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":2.798,"5min":13.129,"15min":13.129},"last":1.248}]},{"osd":7,"last update":"Fri Feb 9 17:36:30 2024","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":2.2879999999999998,"15min":1.663},"min":{"1min":0.42999999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":2.528,"5min":13.063000000000001,"15min":13.063000000000001},"last":0.32700000000000001},{"interface":"front","average":{"1min":1.141,"5min":2.0190000000000001,"15min":1.6140000000000001},"min":{"1min":0.36099999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":3.0939999999999999,"5min":12.959,"15min":16.765999999999998},"last":0.41399999999999998}]},{"osd":8,"last update":"Fri Feb 9 17:36:34 2024","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.3300000000000001,"15min":1.571},"min":{"1min":0.40999999999999998,"5min":0.36599999999999999,"15min":0.35799999999999998},"max":{"1min":3.2429999999999999,"5min":10.207000000000001,"15min":10.207000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.5429999999999999,"15min":1.552},"min":{"1min":0.40400000000000003,"5min":0.38600000000000001,"15min":0.36799999999999999},"max":{"1min":3.2719999999999998,"5min":10.231999999999999,"15min":16.41},"last":0.51000000000000001}]},{"osd":9,"last update":"Fri Feb 9 17:36:20 2024","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":1.589,"15min":1.472},"min":{"1min":0.436,"5min":0.34699999999999998,"15min":0.312},"max":{"1min":2.847,"5min":13.02,"15min":13.02},"last":3.9769999999999999},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.548,"15min":1.4319999999999999},"min":{"1min":0.42099999999999999,"5min":0.373,"15min":0.36199999999999999},"max":{"1min":3.1800000000000002,"5min":10.180999999999999,"15min":16.82},"last":0.46999999999999997}]},{"osd":10,"last update":"Fri Feb 9 17:36:40 2024","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.51,"15min":1.5089999999999999},"min":{"1min":0.45400000000000001,"5min":0.42199999999999999,"15min":0.38500000000000001},"max":{"1min":3.2639999999999998,"5min":13.888,"15min":16.876000000000001},"last":3.859},{"interface":"front","average":{"1min":1.264,"5min":1.3580000000000001,"15min":1.4370000000000001},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.39800000000000002},"max":{"1min":3.5750000000000002,"5min":9.3350000000000009,"15min":9.7970000000000006},"last":1.468}]},{"osd":11,"last update":"Fri Feb 9 17:36:13 2024","interfaces":[{"interface":"back","average":{"1min":3.3730000000000002,"5min":2.1080000000000001,"15min":1.603},"min":{"1min":0.53100000000000003,"5min":0.501,"15min":0.42299999999999999},"max":{"1min":40.021000000000001,"5min":40.021000000000001,"15min":40.021000000000001},"last":1.4470000000000001},{"interface":"front","average":{"1min":3.3980000000000001,"5min":2.0590000000000002,"15min":1.6950000000000001},"min":{"1min":0.55600000000000005,"5min":0.46000000000000002,"15min":0.443},"max":{"1min":40.055999999999997,"5min":40.055999999999997,"15min":40.055999999999997},"last":1.502}]}]},{"osd":2,"up_from":18,"seq":77309411552,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1213560,"kb_used_data":1065776,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":92531592,"statfs":{"total":95995035648,"available":94752350208,"internally_reserved":0,"allocated":1091354624,"data_stored":1090902973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":1.4510000000000001,"15min":1.006},"min":{"1min":0.29999999999999999,"5min":0.23699999999999999,"15min":0.221},"max":{"1min":4.9089999999999998,"5min":17.251000000000001,"15min":17.251000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.74399999999999999,"5min":1.611,"15min":1.113},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.22900000000000001},"max":{"1min":4.7720000000000002,"5min":23.097999999999999,"15min":23.097999999999999},"last":0.875}]},{"osd":1,"last update":"Fri Feb 9 17:36:29 2024","interfaces":[{"interface":"back","average":{"1min":2.2330000000000001,"5min":1.3600000000000001,"15min":1.425},"min":{"1min":0.26100000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":9.5199999999999996,"5min":22.306000000000001,"15min":27.289000000000001},"last":0.72099999999999997},{"interface":"front","average":{"1min":1.347,"5min":1.5900000000000001,"15min":1.581},"min":{"1min":0.27100000000000002,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":7.008,"5min":44.838999999999999,"15min":44.838999999999999},"last":0.59899999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:37:05 2024","interfaces":[{"interface":"back","average":{"1min":2.2599999999999998,"5min":1.659,"15min":1.653},"min":{"1min":0.27300000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":9.5630000000000006,"5min":24.353000000000002,"15min":37.634999999999998},"last":0.42799999999999999},{"interface":"front","average":{"1min":2.984,"5min":1.5369999999999999,"15min":1.484},"min":{"1min":0.27200000000000002,"5min":0.251,"15min":0.23100000000000001},"max":{"1min":10.801,"5min":10.801,"15min":38.924999999999997},"last":4.835}]},{"osd":4,"last update":"Fri Feb 9 17:37:05 2024","interfaces":[{"interface":"back","average":{"1min":2.6709999999999998,"5min":1.9570000000000001,"15min":1.54},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.254},"max":{"1min":10.741,"5min":10.741,"15min":10.741},"last":4.7130000000000001},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.7789999999999999,"15min":1.3600000000000001},"min":{"1min":0.29099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.359,"5min":9.6509999999999998,"15min":13.305999999999999},"last":0.63300000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:36:42 2024","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.613,"15min":1.4330000000000001},"min":{"1min":0.379,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":3.7189999999999999,"5min":7.7539999999999996,"15min":8.6859999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":2.5209999999999999,"5min":2.036,"15min":1.423},"min":{"1min":0.375,"5min":0.30199999999999999,"15min":0.29899999999999999},"max":{"1min":10.781000000000001,"5min":10.781000000000001,"15min":13.278},"last":4.7549999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:37:05 2024","interfaces":[{"interface":"back","average":{"1min":1.6419999999999999,"5min":1.744,"15min":1.522},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.32100000000000001},"max":{"1min":6.4450000000000003,"5min":6.931,"15min":8.8399999999999999},"last":0.748},{"interface":"front","average":{"1min":1.5329999999999999,"5min":2.0409999999999999,"15min":1.637},"min":{"1min":0.433,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":6.5380000000000003,"5min":11.269,"15min":11.269},"last":0.84899999999999998}]},{"osd":7,"last update":"Fri Feb 9 17:36:29 2024","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":1.851,"15min":1.4990000000000001},"min":{"1min":0.40999999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":3.6640000000000001,"5min":11.365,"15min":14.541},"last":0.80300000000000005},{"interface":"front","average":{"1min":1.1140000000000001,"5min":2.0830000000000002,"15min":1.7210000000000001},"min":{"1min":0.46100000000000002,"5min":0.371,"15min":0.371},"max":{"1min":4.8230000000000004,"5min":22.359999999999999,"15min":22.359999999999999},"last":0.47399999999999998}]},{"osd":8,"last update":"Fri Feb 9 17:37:05 2024","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.3380000000000001,"15min":1.2030000000000001},"min":{"1min":0.41199999999999998,"5min":0.38,"15min":0.38},"max":{"1min":3.8919999999999999,"5min":22.393999999999998,"15min":22.393999999999998},"last":0.53900000000000003},{"interface":"front","average":{"1min":1.8080000000000001,"5min":1.413,"15min":1.4430000000000001},"min":{"1min":0.36399999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":6.508,"5min":9.9260000000000002,"15min":9.9260000000000002},"last":0.67500000000000004}]},{"osd":9,"last update":"Fri Feb 9 17:36:16 2024","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.2549999999999999,"15min":1.333},"min":{"1min":0.48199999999999998,"5min":0.371,"15min":0.371},"max":{"1min":2.7130000000000001,"5min":9.5440000000000005,"15min":9.5440000000000005},"last":4.8609999999999998},{"interface":"front","average":{"1min":1.024,"5min":1.3720000000000001,"15min":1.266},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.39800000000000002},"max":{"1min":3.8119999999999998,"5min":9.6509999999999998,"15min":13.218},"last":4.8099999999999996}]},{"osd":10,"last update":"Fri Feb 9 17:36:54 2024","interfaces":[{"interface":"back","average":{"1min":1.984,"5min":1.579,"15min":1.333},"min":{"1min":0.501,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":8.2609999999999992,"5min":22.425999999999998,"15min":22.425999999999998},"last":0.51100000000000001},{"interface":"front","average":{"1min":2.0209999999999999,"5min":1.617,"15min":1.4019999999999999},"min":{"1min":0.54800000000000004,"5min":0.47099999999999997,"15min":0.42099999999999999},"max":{"1min":8.2970000000000006,"5min":22.454999999999998,"15min":22.454999999999998},"last":0.57699999999999996}]},{"osd":11,"last update":"Fri Feb 9 17:37:05 2024","interfaces":[{"interface":"back","average":{"1min":1.798,"5min":1.522,"15min":1.7849999999999999},"min":{"1min":0.39600000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":6.4790000000000001,"5min":6.7709999999999999,"15min":13.186},"last":0.81899999999999995},{"interface":"front","average":{"1min":2.8490000000000002,"5min":1.863,"15min":1.8740000000000001},"min":{"1min":0.45900000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":10.856999999999999,"5min":10.856999999999999,"15min":10.856999999999999},"last":4.7830000000000004}]}]},{"osd":3,"up_from":27,"seq":115964117212,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3055408,"kb_used_data":2731120,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":90689744,"statfs":{"total":95995035648,"available":92866297856,"internally_reserved":0,"allocated":2796666880,"data_stored":2796215229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":331872711},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:37:00 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":1.3999999999999999,"15min":1.6459999999999999},"min":{"1min":0.22600000000000001,"5min":0.224,"15min":0.224},"max":{"1min":2.4380000000000002,"5min":7.5110000000000001,"15min":7.5110000000000001},"last":6.6310000000000002},{"interface":"front","average":{"1min":1.522,"5min":1.5580000000000001,"15min":1.736},"min":{"1min":0.24399999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":6.6269999999999998,"5min":15.167999999999999,"15min":15.167999999999999},"last":14.034000000000001}]},{"osd":1,"last update":"Fri Feb 9 17:33:41 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":1.1399999999999999,"15min":1.3939999999999999},"min":{"1min":0.26100000000000001,"5min":0.23400000000000001,"15min":0.23300000000000001},"max":{"1min":3.8759999999999999,"5min":17.384,"15min":19.638999999999999},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":1.0249999999999999,"15min":1.115},"min":{"1min":0.27500000000000002,"5min":0.22500000000000001,"15min":0.20399999999999999},"max":{"1min":3.8980000000000001,"5min":12.664999999999999,"15min":14.661},"last":0.314}]},{"osd":2,"last update":"Fri Feb 9 17:36:50 2024","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.2849999999999999,"15min":1.4079999999999999},"min":{"1min":0.33500000000000002,"5min":0.22600000000000001,"15min":0.224},"max":{"1min":4.2670000000000003,"5min":14.888999999999999,"15min":14.914999999999999},"last":9.0350000000000001},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.583,"15min":1.4299999999999999},"min":{"1min":0.249,"5min":0.22500000000000001,"15min":0.218},"max":{"1min":6.9139999999999997,"5min":17.613,"15min":17.613},"last":6.7510000000000003}]},{"osd":4,"last update":"Fri Feb 9 17:37:05 2024","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.9259999999999999,"15min":1.7050000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.254},"max":{"1min":5.5860000000000003,"5min":34.898000000000003,"15min":34.898000000000003},"last":13.933},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.756,"15min":1.5449999999999999},"min":{"1min":0.33800000000000002,"5min":0.32300000000000001,"15min":0.27900000000000003},"max":{"1min":4.1760000000000002,"5min":14.061999999999999,"15min":14.061999999999999},"last":9.0830000000000002}]},{"osd":5,"last update":"Fri Feb 9 17:36:35 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":1.8100000000000001,"15min":1.3180000000000001},"min":{"1min":0.36799999999999999,"5min":0.27800000000000002,"15min":0.27300000000000002},"max":{"1min":1.77,"5min":17.651,"15min":17.651},"last":6.7969999999999997},{"interface":"front","average":{"1min":0.86199999999999999,"5min":1.853,"15min":1.3580000000000001},"min":{"1min":0.38900000000000001,"5min":0.34100000000000003,"15min":0.33300000000000002},"max":{"1min":1.8080000000000001,"5min":17.675999999999998,"15min":17.675999999999998},"last":6.8380000000000001}]},{"osd":6,"last update":"Fri Feb 9 17:37:05 2024","interfaces":[{"interface":"back","average":{"1min":2.306,"5min":2.0110000000000001,"15min":1.5840000000000001},"min":{"1min":0.33200000000000002,"5min":0.32900000000000001,"15min":0.32200000000000001},"max":{"1min":10.487,"5min":20.262,"15min":20.262},"last":6.774},{"interface":"front","average":{"1min":1.5980000000000001,"5min":1.6859999999999999,"15min":1.2769999999999999},"min":{"1min":0.372,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":5.157,"5min":7.6959999999999997,"15min":7.9009999999999998},"last":9.1110000000000007}]},{"osd":7,"last update":"Fri Feb 9 17:36:35 2024","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":2.4209999999999998,"15min":1.9179999999999999},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.27200000000000002},"max":{"1min":4.9820000000000002,"5min":21.018999999999998,"15min":21.018999999999998},"last":0.72799999999999998},{"interface":"front","average":{"1min":1.236,"5min":1.9950000000000001,"15min":1.6259999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.37},"max":{"1min":4.7329999999999997,"5min":14.186,"15min":19.550999999999998},"last":0.79000000000000004}]},{"osd":8,"last update":"Fri Feb 9 17:37:05 2024","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.3069999999999999,"15min":1.3149999999999999},"min":{"1min":0.38500000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":4.3739999999999997,"5min":7.7569999999999997,"15min":7.7569999999999997},"last":6.7030000000000003},{"interface":"front","average":{"1min":1.448,"5min":1.1919999999999999,"15min":0.876},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":4.0599999999999996,"5min":14.129,"15min":14.129},"last":13.976000000000001}]},{"osd":9,"last update":"Fri Feb 9 17:36:19 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":1.5189999999999999,"15min":1.2589999999999999},"min":{"1min":0.371,"5min":0.36299999999999999,"15min":0.35499999999999998},"max":{"1min":2.173,"5min":7.7370000000000001,"15min":7.7370000000000001},"last":6.5949999999999998},{"interface":"front","average":{"1min":0.89500000000000002,"5min":1.3049999999999999,"15min":1.516},"min":{"1min":0.438,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":2.9649999999999999,"5min":12.598000000000001,"15min":13.717000000000001},"last":12.815}]},{"osd":10,"last update":"Fri Feb 9 17:36:50 2024","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.355,"15min":1.431},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":2.1429999999999998,"5min":7.7889999999999997,"15min":19.611000000000001},"last":6.6639999999999997},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.401,"15min":1.3200000000000001},"min":{"1min":0.41999999999999998,"5min":0.38,"15min":0.38},"max":{"1min":6.8179999999999996,"5min":12.638,"15min":12.638},"last":12.762}]},{"osd":11,"last update":"Fri Feb 9 17:37:05 2024","interfaces":[{"interface":"back","average":{"1min":1.6539999999999999,"5min":1.6930000000000001,"15min":1.3899999999999999},"min":{"1min":0.46300000000000002,"5min":0.42499999999999999,"15min":0.38800000000000001},"max":{"1min":6.8449999999999998,"5min":12.69,"15min":12.69},"last":6.8170000000000002},{"interface":"front","average":{"1min":1.6899999999999999,"5min":1.4159999999999999,"15min":1.625},"min":{"1min":0.45400000000000001,"5min":0.42499999999999999,"15min":0.36899999999999999},"max":{"1min":4.7729999999999997,"5min":14.157999999999999,"15min":14.157999999999999},"last":14.006}]}]},{"osd":4,"up_from":32,"seq":137438953686,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2476532,"kb_used_data":2138476,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":91268620,"statfs":{"total":95995035648,"available":93459066880,"internally_reserved":0,"allocated":2189799424,"data_stored":2189347253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:37:02 2024","interfaces":[{"interface":"back","average":{"1min":1.954,"5min":2.6120000000000001,"15min":1.4870000000000001},"min":{"1min":0.35999999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":7.9669999999999996,"5min":50.356999999999999,"15min":50.356999999999999},"last":5.1079999999999997},{"interface":"front","average":{"1min":2.0800000000000001,"5min":2.3260000000000001,"15min":1.401},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":8.0440000000000005,"5min":50.569000000000003,"15min":50.569000000000003},"last":4.1040000000000001}]},{"osd":1,"last update":"Fri Feb 9 17:37:13 2024","interfaces":[{"interface":"back","average":{"1min":3.3140000000000001,"5min":2.6619999999999999,"15min":1.853},"min":{"1min":0.27300000000000002,"5min":0.26100000000000001,"15min":0.246},"max":{"1min":28.143000000000001,"5min":50.473999999999997,"15min":50.473999999999997},"last":4.6280000000000001},{"interface":"front","average":{"1min":3.3559999999999999,"5min":2.8109999999999999,"15min":1.8939999999999999},"min":{"1min":0.32200000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":28.103000000000002,"5min":50.499000000000002,"15min":50.499000000000002},"last":4.6609999999999996}]},{"osd":2,"last update":"Fri Feb 9 17:28:51 2024","interfaces":[{"interface":"back","average":{"1min":2.536,"5min":1.546,"15min":0.91400000000000003},"min":{"1min":0.29999999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":8.8819999999999997,"5min":8.8819999999999997,"15min":8.8819999999999997},"last":0.622},{"interface":"front","average":{"1min":2.214,"5min":1.7330000000000001,"15min":0.97799999999999998},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":9.5039999999999996,"5min":9.5039999999999996,"15min":9.5039999999999996},"last":0.53800000000000003}]},{"osd":3,"last update":"Fri Feb 9 17:37:13 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":2.4769999999999999,"15min":1.8620000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":7.1820000000000004,"5min":50.462000000000003,"15min":50.462000000000003},"last":4.1399999999999997},{"interface":"front","average":{"1min":2.0630000000000002,"5min":2.2879999999999998,"15min":1.7110000000000001},"min":{"1min":0.29699999999999999,"5min":0.255,"15min":0.255},"max":{"1min":5.5869999999999997,"5min":50.529000000000003,"15min":50.529000000000003},"last":0.63500000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":1.915,"15min":1.1639999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.222},"max":{"1min":2.0339999999999998,"5min":9.5630000000000006,"15min":9.5630000000000006},"last":0.371},{"interface":"front","average":{"1min":0.79200000000000004,"5min":2.5579999999999998,"15min":1.4550000000000001},"min":{"1min":0.314,"5min":0.314,"15min":0.29199999999999998},"max":{"1min":2.073,"5min":50.509999999999998,"15min":50.509999999999998},"last":0.40799999999999997}]},{"osd":6,"last update":"Fri Feb 9 17:37:13 2024","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.76,"15min":1.2789999999999999},"min":{"1min":0.26600000000000001,"5min":0.26200000000000001,"15min":0.247},"max":{"1min":7.0419999999999998,"5min":11.742000000000001,"15min":18.09},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.999,"5min":2.4129999999999998,"15min":1.4510000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.28000000000000003},"max":{"1min":3.8900000000000001,"5min":50.402999999999999,"15min":50.402999999999999},"last":0.505}]},{"osd":7,"last update":"Fri Feb 9 17:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":2.25,"15min":1.4450000000000001},"min":{"1min":0.40699999999999997,"5min":0.33800000000000002,"15min":0.28399999999999997},"max":{"1min":3.266,"5min":14.919,"15min":15.223000000000001},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.99399999999999999,"5min":2.169,"15min":1.4099999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.28100000000000003},"max":{"1min":4.0419999999999998,"5min":9.343,"15min":19.047000000000001},"last":0.39100000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":1.869,"15min":1.4630000000000001},"min":{"1min":0.36199999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":2.2000000000000002,"5min":8.4160000000000004,"15min":8.4160000000000004},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.80800000000000005,"5min":2.3839999999999999,"15min":1.5249999999999999},"min":{"1min":0.40999999999999998,"5min":0.34699999999999998,"15min":0.308},"max":{"1min":1.4830000000000001,"5min":50.594000000000001,"15min":50.594000000000001},"last":0.53400000000000003}]},{"osd":9,"last update":"Fri Feb 9 17:36:16 2024","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":2.468,"15min":1.53},"min":{"1min":0.436,"5min":0.39200000000000002,"15min":0.34599999999999997},"max":{"1min":2.319,"5min":50.442999999999998,"15min":50.442999999999998},"last":0.49199999999999999},{"interface":"front","average":{"1min":1.1339999999999999,"5min":2.0049999999999999,"15min":1.3049999999999999},"min":{"1min":0.52900000000000003,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":4.0709999999999997,"5min":10.555,"15min":10.555},"last":0.58899999999999997}]},{"osd":10,"last update":"Fri Feb 9 17:36:53 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":2.6680000000000001,"15min":1.649},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.378},"max":{"1min":3.137,"5min":51.223999999999997,"15min":51.223999999999997},"last":0.68300000000000005},{"interface":"front","average":{"1min":1.294,"5min":2.9430000000000001,"15min":1.7130000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.38100000000000001},"max":{"1min":5.117,"5min":50.552,"15min":50.552},"last":0.70199999999999996}]},{"osd":11,"last update":"Fri Feb 9 17:36:16 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":2.899,"15min":1.8839999999999999},"min":{"1min":0.48599999999999999,"5min":0.443,"15min":0.40500000000000003},"max":{"1min":3.5139999999999998,"5min":50.484999999999999,"15min":50.484999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":1.403,"5min":2.2829999999999999,"15min":1.548},"min":{"1min":0.55700000000000005,"5min":0.42499999999999999,"15min":0.34899999999999998},"max":{"1min":5.5599999999999996,"5min":18.574999999999999,"15min":18.574999999999999},"last":0.623}]}]},{"osd":5,"up_from":40,"seq":171798692048,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1245304,"kb_used_data":1069744,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":92499848,"statfs":{"total":95995035648,"available":94719844352,"internally_reserved":0,"allocated":1095417856,"data_stored":1094966205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:32:25 2024","interfaces":[{"interface":"back","average":{"1min":1.8540000000000001,"5min":1.653,"15min":1.0900000000000001},"min":{"1min":0.318,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":7.7160000000000002,"5min":8.7690000000000001,"15min":11.923},"last":3.8450000000000002},{"interface":"front","average":{"1min":2.4089999999999998,"5min":1.8260000000000001,"15min":1.032},"min":{"1min":0.34499999999999997,"5min":0.24299999999999999,"15min":0.224},"max":{"1min":14.082000000000001,"5min":14.082000000000001,"15min":14.082000000000001},"last":0.79400000000000004}]},{"osd":1,"last update":"Fri Feb 9 17:36:32 2024","interfaces":[{"interface":"back","average":{"1min":1.502,"5min":2.1920000000000002,"15min":1.758},"min":{"1min":0.308,"5min":0.23899999999999999,"15min":0.23200000000000001},"max":{"1min":9.2690000000000001,"5min":14.032999999999999,"15min":29.715},"last":0.26000000000000001},{"interface":"front","average":{"1min":1.359,"5min":1.7969999999999999,"15min":1.597},"min":{"1min":0.26000000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":8.6920000000000002,"5min":11.148999999999999,"15min":29.969999999999999},"last":5.1639999999999997}]},{"osd":2,"last update":"Fri Feb 9 17:36:32 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":1.496,"15min":1.425},"min":{"1min":0.28799999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.7729999999999999,"5min":10.5,"15min":10.5},"last":5.21},{"interface":"front","average":{"1min":0.71699999999999997,"5min":1.907,"15min":1.5509999999999999},"min":{"1min":0.371,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":2.472,"5min":14.122,"15min":14.122},"last":0.29299999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:36:32 2024","interfaces":[{"interface":"back","average":{"1min":4.1070000000000002,"5min":2.7250000000000001,"15min":1.8400000000000001},"min":{"1min":0.33900000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":64.150999999999996,"5min":64.150999999999996,"15min":64.150999999999996},"last":5.2430000000000003},{"interface":"front","average":{"1min":4.2119999999999997,"5min":2.8839999999999999,"15min":1.8080000000000001},"min":{"1min":0.41399999999999998,"5min":0.30499999999999999,"15min":0.28000000000000003},"max":{"1min":64.176000000000002,"5min":64.176000000000002,"15min":64.176000000000002},"last":0.46800000000000003}]},{"osd":4,"last update":"Fri Feb 9 17:36:41 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":1.863,"15min":1.139},"min":{"1min":0.28699999999999998,"5min":0.26200000000000001,"15min":0.23400000000000001},"max":{"1min":2.27,"5min":11.036,"15min":11.036},"last":5.2629999999999999},{"interface":"front","average":{"1min":0.81299999999999994,"5min":1.7689999999999999,"15min":1.202},"min":{"1min":0.28599999999999998,"5min":0.28499999999999998,"15min":0.27900000000000003},"max":{"1min":3.7559999999999998,"5min":13.59,"15min":13.59},"last":0.317}]},{"osd":6,"last update":"Fri Feb 9 17:37:09 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":1.6919999999999999,"15min":1.258},"min":{"1min":0.30499999999999999,"5min":0.253,"15min":0.253},"max":{"1min":2.964,"5min":15.375999999999999,"15min":15.375999999999999},"last":0.41599999999999998},{"interface":"front","average":{"1min":1.093,"5min":1.9690000000000001,"15min":1.3280000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.27200000000000002},"max":{"1min":5.4690000000000003,"5min":14.250999999999999,"15min":14.250999999999999},"last":5.4690000000000003}]},{"osd":7,"last update":"Fri Feb 9 17:36:32 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":1.778,"15min":1.2370000000000001},"min":{"1min":0.29399999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.726,"5min":9.7140000000000004,"15min":18.626999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.93700000000000006,"5min":2.1110000000000002,"15min":1.452},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.28799999999999998},"max":{"1min":3.0030000000000001,"5min":19.044,"15min":19.044},"last":1.9550000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:37:09 2024","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.4530000000000001,"15min":1.6870000000000001},"min":{"1min":0.42399999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":5.2960000000000003,"5min":9.3710000000000004,"15min":9.3710000000000004},"last":5.2960000000000003},{"interface":"front","average":{"1min":0.81000000000000005,"5min":1.7170000000000001,"15min":2.1219999999999999},"min":{"1min":0.53500000000000003,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":2.5640000000000001,"5min":9.6240000000000006,"15min":9.6240000000000006},"last":0.57099999999999995}]},{"osd":9,"last update":"Fri Feb 9 17:36:13 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":1.802,"15min":1.246},"min":{"1min":0.48899999999999999,"5min":0.378,"15min":0.35999999999999999},"max":{"1min":1.5509999999999999,"5min":14.98,"15min":14.98},"last":6.5780000000000003},{"interface":"front","average":{"1min":0.78900000000000003,"5min":1.873,"15min":1.2669999999999999},"min":{"1min":0.47799999999999998,"5min":0.36699999999999999,"15min":0.312},"max":{"1min":1.141,"5min":9.4239999999999995,"15min":9.4239999999999995},"last":0.35699999999999998}]},{"osd":10,"last update":"Fri Feb 9 17:36:41 2024","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":1.9930000000000001,"15min":1.4299999999999999},"min":{"1min":0.48599999999999999,"5min":0.47499999999999998,"15min":0.33800000000000002},"max":{"1min":2.581,"5min":14.218,"15min":14.218},"last":0.499},{"interface":"front","average":{"1min":1.01,"5min":2.016,"15min":1.48},"min":{"1min":0.497,"5min":0.45000000000000001,"15min":0.36099999999999999},"max":{"1min":3.3340000000000001,"5min":14.186999999999999,"15min":14.186999999999999},"last":5.2800000000000002}]},{"osd":11,"last update":"Fri Feb 9 17:36:13 2024","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":2.1110000000000002,"15min":1.508},"min":{"1min":0.51900000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":2.5449999999999999,"5min":9.4239999999999995,"15min":9.4239999999999995},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.86399999999999999,"5min":2.4169999999999998,"15min":1.444},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.38900000000000001},"max":{"1min":1.9119999999999999,"5min":14.316000000000001,"15min":14.316000000000001},"last":0.39200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1664090112,"data_stored":1664090112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4890103808,"data_stored":4890099736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1084620800,"data_stored":1084620800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2780168192,"data_stored":2780168192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2182418432,"data_stored":2182414360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1091633152,"data_stored":1091633152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3243835392,"data_stored":3243835392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1666711552,"data_stored":1666711552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1614483456,"data_stored":1614479384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2201550848,"data_stored":2201550848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":3794206720,"data_stored":3794206720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-09T17:37:22.513 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-02-09T17:37:22.514 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:37:22.514 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph pg dump --format=json 2024-02-09T17:37:22.598 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:37:22.632 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:22.652 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:37:22.656 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:22 smithi162 ceph-mon[44304]: 1.8 scrub starts 2024-02-09T17:37:22.656 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:22 smithi162 ceph-mon[44304]: 1.8 scrub ok 2024-02-09T17:37:22.657 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:22.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:37:22.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:22 smithi179 ceph-mon[43975]: 1.8 scrub starts 2024-02-09T17:37:22.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:22 smithi179 ceph-mon[43975]: 1.8 scrub ok 2024-02-09T17:37:22.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:22 smithi099 ceph-mon[39420]: 1.8 scrub starts 2024-02-09T17:37:22.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:22 smithi099 ceph-mon[39420]: 1.8 scrub ok 2024-02-09T17:37:22.757 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:37:22.885 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:23.031 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:37:23.039 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:23.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:37:23.140 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:37:23.184 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:23.208 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:37:23.215 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:23.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:37:23.317 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:37:23.364 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:23.393 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:37:23.393 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [ 2024-02-09T17:37:23.394 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:23.394 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25201.0:149227 4.6 e359/322 4:6dd86a12:::benchmark_data_smithi179_103809_object11761:head v 359'9408, mlcod=359'9404)", 2024-02-09T17:37:23.394 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:37:23.207343+0000", 2024-02-09T17:37:23.394 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.003271857, 2024-02-09T17:37:23.394 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0032946770000000002, 2024-02-09T17:37:23.394 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:37:23.394 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T17:37:23.394 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:37:23.394 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:23.394 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:37:23.394 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:23.207343+0000", 2024-02-09T17:37:23.395 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:37:23.395 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:23.395 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:23.395 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:37:23.395 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:23.207343+0000", 2024-02-09T17:37:23.395 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:37:23.395 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:23.395 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:23.395 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:37:23.395 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:23.207344+0000", 2024-02-09T17:37:23.395 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.8400000000000002e-07 2024-02-09T17:37:23.395 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:23.396 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:23.396 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:37:23.396 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:23.207352+0000", 2024-02-09T17:37:23.396 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.644e-06 2024-02-09T17:37:23.396 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:23.396 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:23.396 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:37:23.396 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:23.207363+0000", 2024-02-09T17:37:23.396 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1113e-05 2024-02-09T17:37:23.396 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:23.396 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:23.396 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:37:23.397 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:23.207372+0000", 2024-02-09T17:37:23.397 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.3750000000000003e-06 2024-02-09T17:37:23.397 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:23.397 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:23.397 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:37:23.397 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:23.207440+0000", 2024-02-09T17:37:23.397 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.8059999999999996e-05 2024-02-09T17:37:23.397 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:23.397 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:23.397 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:37:23.397 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:23.207459+0000", 2024-02-09T17:37:23.398 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.9606000000000002e-05 2024-02-09T17:37:23.398 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:37:23.398 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:37:23.398 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:37:23.398 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:37:23.398 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T17:37:23.398 INFO:teuthology.orchestra.run.smithi099.stdout: "num_ops": 1 2024-02-09T17:37:23.398 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:37:23.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:23 smithi099 ceph-mon[39420]: 4.9 scrub ok 2024-02-09T17:37:23.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:23 smithi099 ceph-mon[39420]: pgmap v978: 37 pgs: 37 active+clean; 7.8 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s 2024-02-09T17:37:23.596 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:37:23.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:23.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:37:23.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:23 smithi179 ceph-mon[43975]: 4.9 scrub ok 2024-02-09T17:37:23.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:23 smithi179 ceph-mon[43975]: pgmap v978: 37 pgs: 37 active+clean; 7.8 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s 2024-02-09T17:37:23.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:23 smithi162 ceph-mon[44304]: 4.9 scrub ok 2024-02-09T17:37:23.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:23 smithi162 ceph-mon[44304]: pgmap v978: 37 pgs: 37 active+clean; 7.8 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s 2024-02-09T17:37:23.701 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:37:23.758 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:24.338 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:37:24.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:24.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:37:24.372 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-09T17:37:24.396 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:37:24.403 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:37:24.442 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:37:24.508 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:24.682 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:37:24.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:24.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:37:24.792 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:37:24.848 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:24.944 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:37:24.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:24.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:37:25.052 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:37:25.125 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:25.595 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:37:25.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:25.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:37:25.705 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:37:25.739 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:25.762 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:37:25.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:25.768 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:37:25.869 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:37:25.898 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:25.934 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:37:25.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:25.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:37:25.955 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set noscrub 2024-02-09T17:37:25.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:25 smithi099 ceph-mon[39420]: pgmap v979: 37 pgs: 37 active+clean; 7.8 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 622 op/s 2024-02-09T17:37:26.043 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:37:26.141 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:26.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:25 smithi179 ceph-mon[43975]: pgmap v979: 37 pgs: 37 active+clean; 7.8 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 622 op/s 2024-02-09T17:37:26.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:25 smithi162 ceph-mon[44304]: pgmap v979: 37 pgs: 37 active+clean; 7.8 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 622 op/s 2024-02-09T17:37:27.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:27 smithi179 ceph-mon[43975]: pgmap v980: 37 pgs: 37 active+clean; 7.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s 2024-02-09T17:37:27.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:27 smithi162 ceph-mon[44304]: pgmap v980: 37 pgs: 37 active+clean; 7.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s 2024-02-09T17:37:27.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:27 smithi099 ceph-mon[39420]: pgmap v980: 37 pgs: 37 active+clean; 7.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s 2024-02-09T17:37:27.865 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:37:27.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:27.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:37:27.971 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:37:28.003 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:28.266 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:37:28.531 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {auth=31,config=2,health=12,kv=53,logm=2} crc {auth=72462687,config=2679774077,health=3897446751,kv=2952948609,logm=2662608713}) 2024-02-09T17:37:28.531 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1513323602}) 2024-02-09T17:37:28.531 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=999063565}) 2024-02-09T17:37:28.531 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1278526977}) 2024-02-09T17:37:28.531 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1833009765}) 2024-02-09T17:37:28.531 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3793322314}) 2024-02-09T17:37:28.531 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=347783529}) 2024-02-09T17:37:28.531 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1513240262}) 2024-02-09T17:37:28.532 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1632552427}) 2024-02-09T17:37:28.532 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1444447996}) 2024-02-09T17:37:28.532 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3111509304}) 2024-02-09T17:37:28.532 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1242739506}) 2024-02-09T17:37:28.532 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4158196959}) 2024-02-09T17:37:28.532 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=456274968}) 2024-02-09T17:37:28.533 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=405982345}) 2024-02-09T17:37:28.533 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=657370132}) 2024-02-09T17:37:28.533 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1996570940}) 2024-02-09T17:37:28.533 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=734601101}) 2024-02-09T17:37:28.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {auth=31,config=2,health=12,kv=53,logm=2} crc {auth=72462687,config=2679774077,health=3897446751,kv=2952948609,logm=2662608713}) 2024-02-09T17:37:28.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1513323602}) 2024-02-09T17:37:28.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=999063565}) 2024-02-09T17:37:28.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1278526977}) 2024-02-09T17:37:28.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1833009765}) 2024-02-09T17:37:28.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3793322314}) 2024-02-09T17:37:28.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=347783529}) 2024-02-09T17:37:28.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1513240262}) 2024-02-09T17:37:28.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1632552427}) 2024-02-09T17:37:28.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1444447996}) 2024-02-09T17:37:28.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3111509304}) 2024-02-09T17:37:28.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1242739506}) 2024-02-09T17:37:28.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4158196959}) 2024-02-09T17:37:28.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=456274968}) 2024-02-09T17:37:28.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=405982345}) 2024-02-09T17:37:28.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=657370132}) 2024-02-09T17:37:28.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1996570940}) 2024-02-09T17:37:28.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=734601101}) 2024-02-09T17:37:28.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {auth=31,config=2,health=12,kv=53,logm=2} crc {auth=72462687,config=2679774077,health=3897446751,kv=2952948609,logm=2662608713}) 2024-02-09T17:37:28.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1513323602}) 2024-02-09T17:37:28.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=999063565}) 2024-02-09T17:37:28.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1278526977}) 2024-02-09T17:37:28.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1833009765}) 2024-02-09T17:37:28.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3793322314}) 2024-02-09T17:37:28.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=347783529}) 2024-02-09T17:37:28.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1513240262}) 2024-02-09T17:37:28.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1632552427}) 2024-02-09T17:37:28.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1444447996}) 2024-02-09T17:37:28.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3111509304}) 2024-02-09T17:37:28.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1242739506}) 2024-02-09T17:37:28.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4158196959}) 2024-02-09T17:37:28.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=456274968}) 2024-02-09T17:37:28.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=405982345}) 2024-02-09T17:37:28.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=657370132}) 2024-02-09T17:37:28.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1996570940}) 2024-02-09T17:37:28.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=734601101}) 2024-02-09T17:37:29.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3345804313}) 2024-02-09T17:37:29.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3588037136}) 2024-02-09T17:37:29.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3742394748}) 2024-02-09T17:37:29.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1260412706}) 2024-02-09T17:37:29.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1119010726}) 2024-02-09T17:37:29.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3218556636}) 2024-02-09T17:37:29.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3284728121}) 2024-02-09T17:37:29.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2532259430}) 2024-02-09T17:37:29.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760573858}) 2024-02-09T17:37:29.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2178323666}) 2024-02-09T17:37:29.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=92983727}) 2024-02-09T17:37:29.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=877779374}) 2024-02-09T17:37:29.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3906325730}) 2024-02-09T17:37:29.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4287636561}) 2024-02-09T17:37:29.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4005913447}) 2024-02-09T17:37:29.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2657847637}) 2024-02-09T17:37:29.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=965691274}) 2024-02-09T17:37:29.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1924146550}) 2024-02-09T17:37:29.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2418917660}) 2024-02-09T17:37:29.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3870368783}) 2024-02-09T17:37:29.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=39,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=23} crc {logm=4246461307,mdsmap=3984425268,mgr=1708625883,mgr_command_descs=708342470,mgr_metadata=670763949,mgrstat=478054412,mon_config_key=3567809231}) 2024-02-09T17:37:29.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=270705568}) 2024-02-09T17:37:29.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3179563362}) 2024-02-09T17:37:29.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=58,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=24} crc {mon_config_key=1459206162,monmap=423865542,osd_metadata=545325961,osd_pg_creating=3665861355,osdmap=3887856233}) 2024-02-09T17:37:29.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=93} crc {osdmap=366440326}) 2024-02-09T17:37:29.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:29 smithi099 ceph-mon[39420]: pgmap v981: 37 pgs: 37 active+clean; 7.5 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 947 op/s 2024-02-09T17:37:29.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3345804313}) 2024-02-09T17:37:29.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3588037136}) 2024-02-09T17:37:29.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3742394748}) 2024-02-09T17:37:29.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1260412706}) 2024-02-09T17:37:29.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1119010726}) 2024-02-09T17:37:29.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3218556636}) 2024-02-09T17:37:29.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3284728121}) 2024-02-09T17:37:29.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2532259430}) 2024-02-09T17:37:29.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760573858}) 2024-02-09T17:37:29.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2178323666}) 2024-02-09T17:37:29.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=92983727}) 2024-02-09T17:37:29.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=877779374}) 2024-02-09T17:37:29.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3906325730}) 2024-02-09T17:37:29.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4287636561}) 2024-02-09T17:37:29.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4005913447}) 2024-02-09T17:37:29.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2657847637}) 2024-02-09T17:37:29.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=965691274}) 2024-02-09T17:37:29.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1924146550}) 2024-02-09T17:37:29.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2418917660}) 2024-02-09T17:37:29.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3870368783}) 2024-02-09T17:37:29.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=39,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=23} crc {logm=4246461307,mdsmap=3984425268,mgr=1708625883,mgr_command_descs=708342470,mgr_metadata=670763949,mgrstat=478054412,mon_config_key=3567809231}) 2024-02-09T17:37:29.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=270705568}) 2024-02-09T17:37:29.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3179563362}) 2024-02-09T17:37:29.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=58,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=24} crc {mon_config_key=1459206162,monmap=423865542,osd_metadata=545325961,osd_pg_creating=3665861355,osdmap=3887856233}) 2024-02-09T17:37:29.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=93} crc {osdmap=366440326}) 2024-02-09T17:37:29.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:29 smithi179 ceph-mon[43975]: pgmap v981: 37 pgs: 37 active+clean; 7.5 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 947 op/s 2024-02-09T17:37:29.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3345804313}) 2024-02-09T17:37:29.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3588037136}) 2024-02-09T17:37:29.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3742394748}) 2024-02-09T17:37:29.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1260412706}) 2024-02-09T17:37:29.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1119010726}) 2024-02-09T17:37:29.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3218556636}) 2024-02-09T17:37:29.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3284728121}) 2024-02-09T17:37:29.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2532259430}) 2024-02-09T17:37:29.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1760573858}) 2024-02-09T17:37:29.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2178323666}) 2024-02-09T17:37:29.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=92983727}) 2024-02-09T17:37:29.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=877779374}) 2024-02-09T17:37:29.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3906325730}) 2024-02-09T17:37:29.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4287636561}) 2024-02-09T17:37:29.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4005913447}) 2024-02-09T17:37:29.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2657847637}) 2024-02-09T17:37:29.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=965691274}) 2024-02-09T17:37:29.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1924146550}) 2024-02-09T17:37:29.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2418917660}) 2024-02-09T17:37:29.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3870368783}) 2024-02-09T17:37:29.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=39,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=23} crc {logm=4246461307,mdsmap=3984425268,mgr=1708625883,mgr_command_descs=708342470,mgr_metadata=670763949,mgrstat=478054412,mon_config_key=3567809231}) 2024-02-09T17:37:29.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=270705568}) 2024-02-09T17:37:29.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3179563362}) 2024-02-09T17:37:29.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=58,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=24} crc {mon_config_key=1459206162,monmap=423865542,osd_metadata=545325961,osd_pg_creating=3665861355,osdmap=3887856233}) 2024-02-09T17:37:29.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=93} crc {osdmap=366440326}) 2024-02-09T17:37:29.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:29 smithi162 ceph-mon[44304]: pgmap v981: 37 pgs: 37 active+clean; 7.5 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 947 op/s 2024-02-09T17:37:29.999 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:37:30.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:30.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:37:30.115 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:37:30.151 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:30.172 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:37:30.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:30.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:37:30.279 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:37:30.319 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:30.360 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:37:30.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:30.367 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:37:30.468 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:37:30.515 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:30.549 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:37:30.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:30.554 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:37:30.613 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:37:30.613 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:37:30.614 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:37:30.614 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:37:30.614 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:37:30.614 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:37:30.614 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:37:30.614 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:37:30.614 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:37:30.614 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:37:30.614 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:37:30.614 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:37:30.614 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:37:30.614 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:37:30.615 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:37:30.615 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:37:30.615 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:37:30.615 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:37:30.615 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:37:30.615 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:37:30.615 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:37:30.615 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:37:30.615 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:37:30.615 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:37:30.655 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:37:30.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:30 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1769295581' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:37:30.698 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:30.727 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:37:30.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:30.731 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:37:30.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:30 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1769295581' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:37:30.831 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:37:30.862 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:30 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1769295581' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:37:30.865 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:30.889 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:37:30.896 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:30.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:37:30.998 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:37:31.039 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:31.070 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:37:31.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:31.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:37:31.175 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:37:31.229 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:31.580 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:37:31.621 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-02-09T17:37:31.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:31 smithi099 ceph-mon[39420]: pgmap v982: 37 pgs: 37 active+clean; 7.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 941 op/s 2024-02-09T17:37:31.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:31 smithi179 ceph-mon[43975]: pgmap v982: 37 pgs: 37 active+clean; 7.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 941 op/s 2024-02-09T17:37:31.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:31 smithi162 ceph-mon[44304]: pgmap v982: 37 pgs: 37 active+clean; 7.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 941 op/s 2024-02-09T17:37:32.215 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:37:32.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:32.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:37:32.321 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:37:32.349 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:32.371 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:37:32.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:32.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:37:32.477 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:37:32.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:32 smithi099 ceph-mon[39420]: from='client.25297 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:37:32.523 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:32.549 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:37:32.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:32.552 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:37:32.653 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:37:32.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:32 smithi162 ceph-mon[44304]: from='client.25297 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:37:32.691 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:32.713 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:37:32.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:32.718 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:37:32.820 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:37:32.885 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:32.886 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":982,"stamp":"2024-02-09T17:37:30.669833+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7897481784,"num_objects":120500,"num_object_clones":0,"num_object_copies":361500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":120500,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154430,"num_write_kb":8797633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157349,"ondisk_log_size":157349,"up":111,"acting":111,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":112,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":26595868,"kb_used_data":23300868,"kb_used_omap":229,"kb_used_meta":3293722,"kb_avail":1098345956,"statfs":{"total":1151940427776,"available":1124706258944,"internally_reserved":0,"allocated":23860088832,"data_stored":23856120060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235127,"internal_metadata":3372771721},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,2,8,6,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-633798656,"num_objects":-9671,"num_object_clones":0,"num_object_copies":-29013,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9671,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.272123"},"pg_stats":[{"pgid":"1.14","version":"63'139","reported_seq":952,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-02-09T17:37:20.404083+0000","last_change":"2024-02-09T17:37:03.748841+0000","last_active":"2024-02-09T17:37:20.404083+0000","last_peered":"2024-02-09T17:37:20.404083+0000","last_clean":"2024-02-09T17:37:20.404083+0000","last_became_active":"2024-02-09T17:33:05.309643+0000","last_became_peered":"2024-02-09T17:33:05.309643+0000","last_unstale":"2024-02-09T17:37:20.404083+0000","last_undegraded":"2024-02-09T17:37:20.404083+0000","last_fullsized":"2024-02-09T17:37:20.404083+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:37:03.748701+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:34:13.228622+0000","last_clean_scrub_stamp":"2024-02-09T17:37:03.748701+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:22.979456+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"359'9396","reported_seq":9573,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-02-09T17:37:25.779235+0000","last_change":"2024-02-09T17:37:10.153953+0000","last_active":"2024-02-09T17:37:25.779235+0000","last_peered":"2024-02-09T17:37:25.779235+0000","last_clean":"2024-02-09T17:37:25.779235+0000","last_became_active":"2024-02-09T17:35:30.390786+0000","last_became_peered":"2024-02-09T17:35:30.390786+0000","last_unstale":"2024-02-09T17:37:25.779235+0000","last_undegraded":"2024-02-09T17:37:25.779235+0000","last_fullsized":"2024-02-09T17:37:25.779235+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'8529","last_scrub_stamp":"2024-02-09T17:37:10.153802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:37:10.153802+0000","objects_scrubbed":8484,"log_size":9396,"log_dups_size":0,"ondisk_log_size":9396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:34.665727+0000","scrub_duration":1669,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":499122176,"num_objects":7616,"num_object_clones":0,"num_object_copies":22848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9396,"num_write_kb":544384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,11],"acting":[6,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"63'139","reported_seq":983,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-02-09T17:37:20.405705+0000","last_change":"2024-02-09T17:37:07.369195+0000","last_active":"2024-02-09T17:37:20.405705+0000","last_peered":"2024-02-09T17:37:20.405705+0000","last_clean":"2024-02-09T17:37:20.405705+0000","last_became_active":"2024-02-09T17:33:00.284421+0000","last_became_peered":"2024-02-09T17:33:00.284421+0000","last_unstale":"2024-02-09T17:37:20.405705+0000","last_undegraded":"2024-02-09T17:37:20.405705+0000","last_fullsized":"2024-02-09T17:37:20.405705+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:37:07.369056+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:37:07.369056+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:14.417376+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"359'9790","reported_seq":9967,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-02-09T17:37:30.394500+0000","last_change":"2024-02-09T17:37:07.695049+0000","last_active":"2024-02-09T17:37:30.394500+0000","last_peered":"2024-02-09T17:37:30.394500+0000","last_clean":"2024-02-09T17:37:30.394500+0000","last_became_active":"2024-02-09T17:35:30.390424+0000","last_became_peered":"2024-02-09T17:35:30.390424+0000","last_unstale":"2024-02-09T17:37:30.394500+0000","last_undegraded":"2024-02-09T17:37:30.394500+0000","last_fullsized":"2024-02-09T17:37:30.394500+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"356'8575","last_scrub_stamp":"2024-02-09T17:37:07.694769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:37:07.694769+0000","objects_scrubbed":8497,"log_size":9790,"log_dups_size":0,"ondisk_log_size":9790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:25.342399+0000","scrub_duration":3173,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":495714304,"num_objects":7564,"num_object_clones":0,"num_object_copies":22692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9790,"num_write_kb":555328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,0],"acting":[1,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"63'139","reported_seq":999,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-02-09T17:37:20.405718+0000","last_change":"2024-02-09T17:37:10.429162+0000","last_active":"2024-02-09T17:37:20.405718+0000","last_peered":"2024-02-09T17:37:20.405718+0000","last_clean":"2024-02-09T17:37:20.405718+0000","last_became_active":"2024-02-09T17:34:01.410594+0000","last_became_peered":"2024-02-09T17:34:01.410594+0000","last_unstale":"2024-02-09T17:37:20.405718+0000","last_undegraded":"2024-02-09T17:37:20.405718+0000","last_fullsized":"2024-02-09T17:37:20.405718+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:37:10.429033+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:37:10.429033+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:17.314134+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"359'9572","reported_seq":9750,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-02-09T17:37:26.390972+0000","last_change":"2024-02-09T17:37:09.133954+0000","last_active":"2024-02-09T17:37:26.390972+0000","last_peered":"2024-02-09T17:37:26.390972+0000","last_clean":"2024-02-09T17:37:26.390972+0000","last_became_active":"2024-02-09T17:35:30.166669+0000","last_became_peered":"2024-02-09T17:35:30.166669+0000","last_unstale":"2024-02-09T17:37:26.390972+0000","last_undegraded":"2024-02-09T17:37:26.390972+0000","last_fullsized":"2024-02-09T17:37:26.390972+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'8615","last_scrub_stamp":"2024-02-09T17:37:09.133803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:37:09.133803+0000","objects_scrubbed":8567,"log_size":9572,"log_dups_size":0,"ondisk_log_size":9572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:13.853601+0000","scrub_duration":2038,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":504496128,"num_objects":7698,"num_object_clones":0,"num_object_copies":23094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9572,"num_write_kb":552640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,2],"acting":[9,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.3","version":"63'139","reported_seq":496,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-02-09T17:37:20.403604+0000","last_change":"2024-02-09T17:37:14.844939+0000","last_active":"2024-02-09T17:37:20.403604+0000","last_peered":"2024-02-09T17:37:20.403604+0000","last_clean":"2024-02-09T17:37:20.403604+0000","last_became_active":"2024-02-09T17:33:00.279369+0000","last_became_peered":"2024-02-09T17:33:00.279369+0000","last_unstale":"2024-02-09T17:37:20.403604+0000","last_undegraded":"2024-02-09T17:37:20.403604+0000","last_fullsized":"2024-02-09T17:37:20.403604+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:37:14.844841+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:37:14.844841+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:15.289072+0000","scrub_duration":98,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"359'9654","reported_seq":9831,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-02-09T17:37:30.378771+0000","last_change":"2024-02-09T17:37:11.857576+0000","last_active":"2024-02-09T17:37:30.378771+0000","last_peered":"2024-02-09T17:37:30.378771+0000","last_clean":"2024-02-09T17:37:30.378771+0000","last_became_active":"2024-02-09T17:35:30.166273+0000","last_became_peered":"2024-02-09T17:35:30.166273+0000","last_unstale":"2024-02-09T17:37:30.378771+0000","last_undegraded":"2024-02-09T17:37:30.378771+0000","last_fullsized":"2024-02-09T17:37:30.378771+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'8608","last_scrub_stamp":"2024-02-09T17:37:11.857362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:37:11.857362+0000","objects_scrubbed":8467,"log_size":9654,"log_dups_size":0,"ondisk_log_size":9654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:38.710861+0000","scrub_duration":2226,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482738176,"num_objects":7366,"num_object_clones":0,"num_object_copies":22098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9654,"num_write_kb":544640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"63'139","reported_seq":507,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-02-09T17:37:20.406430+0000","last_change":"2024-02-09T17:37:03.758658+0000","last_active":"2024-02-09T17:37:20.406430+0000","last_peered":"2024-02-09T17:37:20.406430+0000","last_clean":"2024-02-09T17:37:20.406430+0000","last_became_active":"2024-02-09T17:34:01.342607+0000","last_became_peered":"2024-02-09T17:34:01.342607+0000","last_unstale":"2024-02-09T17:37:20.406430+0000","last_undegraded":"2024-02-09T17:37:20.406430+0000","last_fullsized":"2024-02-09T17:37:20.406430+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:37:03.758546+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:37:03.758546+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:12.219105+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.3","version":"359'9690","reported_seq":9867,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-02-09T17:37:30.097351+0000","last_change":"2024-02-09T17:37:13.488486+0000","last_active":"2024-02-09T17:37:30.097351+0000","last_peered":"2024-02-09T17:37:30.097351+0000","last_clean":"2024-02-09T17:37:30.097351+0000","last_became_active":"2024-02-09T17:35:30.134014+0000","last_became_peered":"2024-02-09T17:35:30.134014+0000","last_unstale":"2024-02-09T17:37:30.097351+0000","last_undegraded":"2024-02-09T17:37:30.097351+0000","last_fullsized":"2024-02-09T17:37:30.097351+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'8765","last_scrub_stamp":"2024-02-09T17:37:13.488343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:37:13.488343+0000","objects_scrubbed":8429,"log_size":9690,"log_dups_size":0,"ondisk_log_size":9690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:28.392245+0000","scrub_duration":2180,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":488767488,"num_objects":7458,"num_object_clones":0,"num_object_copies":22374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9690,"num_write_kb":548736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,1],"acting":[11,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"63'139","reported_seq":498,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-02-09T17:37:20.408435+0000","last_change":"2024-02-09T17:37:07.707766+0000","last_active":"2024-02-09T17:37:20.408435+0000","last_peered":"2024-02-09T17:37:20.408435+0000","last_clean":"2024-02-09T17:37:20.408435+0000","last_became_active":"2024-02-09T17:33:00.284702+0000","last_became_peered":"2024-02-09T17:33:00.284702+0000","last_unstale":"2024-02-09T17:37:20.408435+0000","last_undegraded":"2024-02-09T17:37:20.408435+0000","last_fullsized":"2024-02-09T17:37:20.408435+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:37:07.707633+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:37:07.707633+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:28.410525+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2","version":"359'9429","reported_seq":9605,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-02-09T17:37:26.397651+0000","last_change":"2024-02-09T17:37:09.776219+0000","last_active":"2024-02-09T17:37:26.397651+0000","last_peered":"2024-02-09T17:37:26.397651+0000","last_clean":"2024-02-09T17:37:26.397651+0000","last_became_active":"2024-02-09T17:35:29.899914+0000","last_became_peered":"2024-02-09T17:35:29.899914+0000","last_unstale":"2024-02-09T17:37:26.397651+0000","last_undegraded":"2024-02-09T17:37:26.397651+0000","last_fullsized":"2024-02-09T17:37:26.397651+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'8538","last_scrub_stamp":"2024-02-09T17:37:09.776086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:37:09.776086+0000","objects_scrubbed":8510,"log_size":9429,"log_dups_size":0,"ondisk_log_size":9429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:34.505426+0000","scrub_duration":1666,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":500891648,"num_objects":7643,"num_object_clones":0,"num_object_copies":22929,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9429,"num_write_kb":546304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.7","version":"63'139","reported_seq":511,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-02-09T17:37:20.402090+0000","last_change":"2024-02-09T17:37:06.599735+0000","last_active":"2024-02-09T17:37:20.402090+0000","last_peered":"2024-02-09T17:37:20.402090+0000","last_clean":"2024-02-09T17:37:20.402090+0000","last_became_active":"2024-02-09T17:33:00.278966+0000","last_became_peered":"2024-02-09T17:33:00.278966+0000","last_unstale":"2024-02-09T17:37:20.402090+0000","last_undegraded":"2024-02-09T17:37:20.402090+0000","last_fullsized":"2024-02-09T17:37:20.402090+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:37:06.599635+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:37:06.599635+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:25.470941+0000","scrub_duration":102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"359'9743","reported_seq":9925,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-02-09T17:37:30.111521+0000","last_change":"2024-02-09T17:37:05.983264+0000","last_active":"2024-02-09T17:37:30.111521+0000","last_peered":"2024-02-09T17:37:30.111521+0000","last_clean":"2024-02-09T17:37:30.111521+0000","last_became_active":"2024-02-09T17:35:29.900032+0000","last_became_peered":"2024-02-09T17:35:29.900032+0000","last_unstale":"2024-02-09T17:37:30.111521+0000","last_undegraded":"2024-02-09T17:37:30.111521+0000","last_fullsized":"2024-02-09T17:37:30.111521+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"356'8379","last_scrub_stamp":"2024-02-09T17:37:05.983122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:37:05.983122+0000","objects_scrubbed":8324,"log_size":9743,"log_dups_size":0,"ondisk_log_size":9743,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:28.414791+0000","scrub_duration":1701,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":489357312,"num_objects":7467,"num_object_clones":0,"num_object_copies":22401,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9743,"num_write_kb":550720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"63'139","reported_seq":589,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-02-09T17:37:20.403670+0000","last_change":"2024-02-09T17:37:19.223274+0000","last_active":"2024-02-09T17:37:20.403670+0000","last_peered":"2024-02-09T17:37:20.403670+0000","last_clean":"2024-02-09T17:37:20.403670+0000","last_became_active":"2024-02-09T17:33:00.284924+0000","last_became_peered":"2024-02-09T17:33:00.284924+0000","last_unstale":"2024-02-09T17:37:20.403670+0000","last_undegraded":"2024-02-09T17:37:20.403670+0000","last_fullsized":"2024-02-09T17:37:20.403670+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:37:19.223119+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:37:19.223119+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:43.320825+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.0","version":"359'9531","reported_seq":9708,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-02-09T17:37:26.916097+0000","last_change":"2024-02-09T17:37:13.908630+0000","last_active":"2024-02-09T17:37:26.916097+0000","last_peered":"2024-02-09T17:37:26.916097+0000","last_clean":"2024-02-09T17:37:26.916097+0000","last_became_active":"2024-02-09T17:35:30.134347+0000","last_became_peered":"2024-02-09T17:35:30.134347+0000","last_unstale":"2024-02-09T17:37:26.916097+0000","last_undegraded":"2024-02-09T17:37:26.916097+0000","last_fullsized":"2024-02-09T17:37:26.916097+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'8832","last_scrub_stamp":"2024-02-09T17:37:13.908444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:37:13.908444+0000","objects_scrubbed":8446,"log_size":9531,"log_dups_size":0,"ondisk_log_size":9531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:33.081607+0000","scrub_duration":2861,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502857728,"num_objects":7673,"num_object_clones":0,"num_object_copies":23019,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9531,"num_write_kb":550528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"63'139","reported_seq":498,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-02-09T17:37:20.406521+0000","last_change":"2024-02-09T17:37:02.738763+0000","last_active":"2024-02-09T17:37:20.406521+0000","last_peered":"2024-02-09T17:37:20.406521+0000","last_clean":"2024-02-09T17:37:20.406521+0000","last_became_active":"2024-02-09T17:33:00.274636+0000","last_became_peered":"2024-02-09T17:33:00.274636+0000","last_unstale":"2024-02-09T17:37:20.406521+0000","last_undegraded":"2024-02-09T17:37:20.406521+0000","last_fullsized":"2024-02-09T17:37:20.406521+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:37:02.738611+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:37:02.738611+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:28.854048+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.f","version":"359'9655","reported_seq":9828,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-02-09T17:37:30.375283+0000","last_change":"2024-02-09T17:37:19.398235+0000","last_active":"2024-02-09T17:37:30.375283+0000","last_peered":"2024-02-09T17:37:30.375283+0000","last_clean":"2024-02-09T17:37:30.375283+0000","last_became_active":"2024-02-09T17:35:29.905385+0000","last_became_peered":"2024-02-09T17:35:29.905385+0000","last_unstale":"2024-02-09T17:37:30.375283+0000","last_undegraded":"2024-02-09T17:37:30.375283+0000","last_fullsized":"2024-02-09T17:37:30.375283+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'9026","last_scrub_stamp":"2024-02-09T17:37:19.398062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:37:19.398062+0000","objects_scrubbed":8100,"log_size":9655,"log_dups_size":0,"ondisk_log_size":9655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:46.416390+0000","scrub_duration":2684,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485425152,"num_objects":7407,"num_object_clones":0,"num_object_copies":22221,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9655,"num_write_kb":545984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,11],"acting":[1,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"63'139","reported_seq":492,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-02-09T17:37:20.404406+0000","last_change":"2024-02-09T17:36:06.172514+0000","last_active":"2024-02-09T17:37:20.404406+0000","last_peered":"2024-02-09T17:37:20.404406+0000","last_clean":"2024-02-09T17:37:20.404406+0000","last_became_active":"2024-02-09T17:32:35.418200+0000","last_became_peered":"2024-02-09T17:32:35.418200+0000","last_unstale":"2024-02-09T17:37:20.404406+0000","last_undegraded":"2024-02-09T17:37:20.404406+0000","last_fullsized":"2024-02-09T17:37:20.404406+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:36:06.172377+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:36:06.172377+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:37:35.221695+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.e","version":"359'9693","reported_seq":9871,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-02-09T17:37:30.531479+0000","last_change":"2024-02-09T17:37:11.874433+0000","last_active":"2024-02-09T17:37:30.531479+0000","last_peered":"2024-02-09T17:37:30.531479+0000","last_clean":"2024-02-09T17:37:30.531479+0000","last_became_active":"2024-02-09T17:35:30.390749+0000","last_became_peered":"2024-02-09T17:35:30.390749+0000","last_unstale":"2024-02-09T17:37:30.531479+0000","last_undegraded":"2024-02-09T17:37:30.531479+0000","last_fullsized":"2024-02-09T17:37:30.531479+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'8651","last_scrub_stamp":"2024-02-09T17:37:11.874252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:37:11.874252+0000","objects_scrubbed":8474,"log_size":9693,"log_dups_size":0,"ondisk_log_size":9693,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:29.860070+0000","scrub_duration":1689,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484376576,"num_objects":7391,"num_object_clones":0,"num_object_copies":22173,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9693,"num_write_kb":546688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,6],"acting":[4,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"63'139","reported_seq":507,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-02-09T17:37:20.404167+0000","last_change":"2024-02-09T17:37:05.469016+0000","last_active":"2024-02-09T17:37:20.404167+0000","last_peered":"2024-02-09T17:37:20.404167+0000","last_clean":"2024-02-09T17:37:20.404167+0000","last_became_active":"2024-02-09T17:34:01.388313+0000","last_became_peered":"2024-02-09T17:34:01.388313+0000","last_unstale":"2024-02-09T17:37:20.404167+0000","last_undegraded":"2024-02-09T17:37:20.404167+0000","last_fullsized":"2024-02-09T17:37:20.404167+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:37:05.468894+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:37:05.468894+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:34.390980+0000","scrub_duration":138,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.d","version":"359'9731","reported_seq":9908,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-02-09T17:37:30.494893+0000","last_change":"2024-02-09T17:37:13.543612+0000","last_active":"2024-02-09T17:37:30.494893+0000","last_peered":"2024-02-09T17:37:30.494893+0000","last_clean":"2024-02-09T17:37:30.494893+0000","last_became_active":"2024-02-09T17:35:29.905824+0000","last_became_peered":"2024-02-09T17:35:29.905824+0000","last_unstale":"2024-02-09T17:37:30.494893+0000","last_undegraded":"2024-02-09T17:37:30.494893+0000","last_fullsized":"2024-02-09T17:37:30.494893+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'8767","last_scrub_stamp":"2024-02-09T17:37:13.543492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:37:13.543492+0000","objects_scrubbed":8445,"log_size":9731,"log_dups_size":0,"ondisk_log_size":9731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:22.127226+0000","scrub_duration":2388,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":486408216,"num_objects":7423,"num_object_clones":0,"num_object_copies":22269,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7423,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9731,"num_write_kb":548865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,1],"acting":[4,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"63'139","reported_seq":509,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-02-09T17:37:21.762029+0000","last_change":"2024-02-09T17:37:21.762086+0000","last_active":"2024-02-09T17:37:21.762029+0000","last_peered":"2024-02-09T17:37:21.762029+0000","last_clean":"2024-02-09T17:37:21.762029+0000","last_became_active":"2024-02-09T17:32:35.384617+0000","last_became_peered":"2024-02-09T17:32:35.384617+0000","last_unstale":"2024-02-09T17:37:21.762029+0000","last_undegraded":"2024-02-09T17:37:21.762029+0000","last_fullsized":"2024-02-09T17:37:21.762029+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:37:21.761916+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:37:21.761916+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:49.944236+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"359'9893","reported_seq":10067,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-02-09T17:37:30.510326+0000","last_change":"2024-02-09T17:37:20.108495+0000","last_active":"2024-02-09T17:37:30.510326+0000","last_peered":"2024-02-09T17:37:30.510326+0000","last_clean":"2024-02-09T17:37:30.510326+0000","last_became_active":"2024-02-09T17:35:29.903553+0000","last_became_peered":"2024-02-09T17:35:29.903553+0000","last_unstale":"2024-02-09T17:37:30.510326+0000","last_undegraded":"2024-02-09T17:37:30.510326+0000","last_fullsized":"2024-02-09T17:37:30.510326+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'9333","last_scrub_stamp":"2024-02-09T17:37:20.108372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:37:20.108372+0000","objects_scrubbed":8252,"log_size":9893,"log_dups_size":0,"ondisk_log_size":9893,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:40.001226+0000","scrub_duration":2010,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":499843072,"num_objects":7627,"num_object_clones":0,"num_object_copies":22881,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9893,"num_write_kb":560640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"63'139","reported_seq":985,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-02-09T17:37:20.405607+0000","last_change":"2024-02-09T17:37:11.520715+0000","last_active":"2024-02-09T17:37:20.405607+0000","last_peered":"2024-02-09T17:37:20.405607+0000","last_clean":"2024-02-09T17:37:20.405607+0000","last_became_active":"2024-02-09T17:32:35.402025+0000","last_became_peered":"2024-02-09T17:32:35.402025+0000","last_unstale":"2024-02-09T17:37:20.405607+0000","last_undegraded":"2024-02-09T17:37:20.405607+0000","last_fullsized":"2024-02-09T17:37:20.405607+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:37:11.520601+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:30.193194+0000","last_clean_scrub_stamp":"2024-02-09T17:37:11.520601+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:38.007919+0000","scrub_duration":102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"359'9758","reported_seq":9935,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-02-09T17:37:28.591702+0000","last_change":"2024-02-09T17:37:05.121978+0000","last_active":"2024-02-09T17:37:28.591702+0000","last_peered":"2024-02-09T17:37:28.591702+0000","last_clean":"2024-02-09T17:37:28.591702+0000","last_became_active":"2024-02-09T17:35:29.906768+0000","last_became_peered":"2024-02-09T17:35:29.906768+0000","last_unstale":"2024-02-09T17:37:28.591702+0000","last_undegraded":"2024-02-09T17:37:28.591702+0000","last_fullsized":"2024-02-09T17:37:28.591702+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"356'8446","last_scrub_stamp":"2024-02-09T17:37:05.121787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:37:05.121787+0000","objects_scrubbed":8387,"log_size":9758,"log_dups_size":0,"ondisk_log_size":9758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:30.011025+0000","scrub_duration":2496,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":501350400,"num_objects":7650,"num_object_clones":0,"num_object_copies":22950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9758,"num_write_kb":557056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"63'139","reported_seq":460,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-02-09T17:37:20.403405+0000","last_change":"2024-02-09T17:37:06.250206+0000","last_active":"2024-02-09T17:37:20.403405+0000","last_peered":"2024-02-09T17:37:20.403405+0000","last_clean":"2024-02-09T17:37:20.403405+0000","last_became_active":"2024-02-09T17:33:05.309824+0000","last_became_peered":"2024-02-09T17:33:05.309824+0000","last_unstale":"2024-02-09T17:37:20.403405+0000","last_undegraded":"2024-02-09T17:37:20.403405+0000","last_fullsized":"2024-02-09T17:37:20.403405+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:37:06.250017+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:37:06.250017+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:24.561623+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"359'9390","reported_seq":9567,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-02-09T17:37:25.776909+0000","last_change":"2024-02-09T17:37:09.380478+0000","last_active":"2024-02-09T17:37:25.776909+0000","last_peered":"2024-02-09T17:37:25.776909+0000","last_clean":"2024-02-09T17:37:25.776909+0000","last_became_active":"2024-02-09T17:35:30.390843+0000","last_became_peered":"2024-02-09T17:35:30.390843+0000","last_unstale":"2024-02-09T17:37:25.776909+0000","last_undegraded":"2024-02-09T17:37:25.776909+0000","last_fullsized":"2024-02-09T17:37:25.776909+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'8477","last_scrub_stamp":"2024-02-09T17:37:09.380351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:37:09.380351+0000","objects_scrubbed":8431,"log_size":9390,"log_dups_size":0,"ondisk_log_size":9390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:24.285734+0000","scrub_duration":1888,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":497680384,"num_objects":7594,"num_object_clones":0,"num_object_copies":22782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9390,"num_write_kb":543488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.f","version":"63'139","reported_seq":519,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-02-09T17:37:20.401995+0000","last_change":"2024-02-09T17:36:22.364212+0000","last_active":"2024-02-09T17:37:20.401995+0000","last_peered":"2024-02-09T17:37:20.401995+0000","last_clean":"2024-02-09T17:37:20.401995+0000","last_became_active":"2024-02-09T17:33:05.316100+0000","last_became_peered":"2024-02-09T17:33:05.316100+0000","last_unstale":"2024-02-09T17:37:20.401995+0000","last_undegraded":"2024-02-09T17:37:20.401995+0000","last_fullsized":"2024-02-09T17:37:20.401995+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:36:22.364130+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:36:22.364130+0000","last_clean_scrub_stamp":"2024-02-09T17:36:22.364130+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:37:41.044157+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"359'9763","reported_seq":9934,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-02-09T17:37:30.096214+0000","last_change":"2024-02-09T17:37:22.382066+0000","last_active":"2024-02-09T17:37:30.096214+0000","last_peered":"2024-02-09T17:37:30.096214+0000","last_clean":"2024-02-09T17:37:30.096214+0000","last_became_active":"2024-02-09T17:35:29.905885+0000","last_became_peered":"2024-02-09T17:35:29.905885+0000","last_unstale":"2024-02-09T17:37:30.096214+0000","last_undegraded":"2024-02-09T17:37:30.096214+0000","last_fullsized":"2024-02-09T17:37:30.096214+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"359'9289","last_scrub_stamp":"2024-02-09T17:37:22.381944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:37:22.381944+0000","objects_scrubbed":7938,"log_size":9763,"log_dups_size":0,"ondisk_log_size":9763,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:44.146286+0000","scrub_duration":2196,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485949440,"num_objects":7415,"num_object_clones":0,"num_object_copies":22245,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9763,"num_write_kb":549696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,3],"acting":[11,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"63'139","reported_seq":956,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-02-09T17:37:20.417572+0000","last_change":"2024-02-09T17:37:07.137667+0000","last_active":"2024-02-09T17:37:20.417572+0000","last_peered":"2024-02-09T17:37:20.417572+0000","last_clean":"2024-02-09T17:37:20.417572+0000","last_became_active":"2024-02-09T17:35:20.840386+0000","last_became_peered":"2024-02-09T17:35:20.840386+0000","last_unstale":"2024-02-09T17:37:20.417572+0000","last_undegraded":"2024-02-09T17:37:20.417572+0000","last_fullsized":"2024-02-09T17:37:20.417572+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:37:07.137544+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:23.912504+0000","last_clean_scrub_stamp":"2024-02-09T17:37:07.137544+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:21.263740+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"359'9742","reported_seq":9916,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-02-09T17:37:30.153697+0000","last_change":"2024-02-09T17:37:17.682453+0000","last_active":"2024-02-09T17:37:30.153697+0000","last_peered":"2024-02-09T17:37:30.153697+0000","last_clean":"2024-02-09T17:37:30.153697+0000","last_became_active":"2024-02-09T17:35:30.134041+0000","last_became_peered":"2024-02-09T17:35:30.134041+0000","last_unstale":"2024-02-09T17:37:30.153697+0000","last_undegraded":"2024-02-09T17:37:30.153697+0000","last_fullsized":"2024-02-09T17:37:30.153697+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'9078","last_scrub_stamp":"2024-02-09T17:37:17.682321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:37:17.682321+0000","objects_scrubbed":8212,"log_size":9742,"log_dups_size":0,"ondisk_log_size":9742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:38.388985+0000","scrub_duration":1514,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":491913216,"num_objects":7506,"num_object_clones":0,"num_object_copies":22518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9742,"num_write_kb":551936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"63'139","reported_seq":468,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-02-09T17:37:20.406128+0000","last_change":"2024-02-09T17:37:09.476252+0000","last_active":"2024-02-09T17:37:20.406128+0000","last_peered":"2024-02-09T17:37:20.406128+0000","last_clean":"2024-02-09T17:37:20.406128+0000","last_became_active":"2024-02-09T17:33:05.319668+0000","last_became_peered":"2024-02-09T17:33:05.319668+0000","last_unstale":"2024-02-09T17:37:20.406128+0000","last_undegraded":"2024-02-09T17:37:20.406128+0000","last_fullsized":"2024-02-09T17:37:20.406128+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:37:09.476147+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:37:09.476147+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:14.845386+0000","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"63'139","reported_seq":467,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-02-09T17:37:20.401343+0000","last_change":"2024-02-09T17:36:17.357034+0000","last_active":"2024-02-09T17:37:20.401343+0000","last_peered":"2024-02-09T17:37:20.401343+0000","last_clean":"2024-02-09T17:37:20.401343+0000","last_became_active":"2024-02-09T17:33:05.310104+0000","last_became_peered":"2024-02-09T17:33:05.310104+0000","last_unstale":"2024-02-09T17:37:20.401343+0000","last_undegraded":"2024-02-09T17:37:20.401343+0000","last_fullsized":"2024-02-09T17:37:20.401343+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:36:17.356909+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:36:17.356909+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:37:45.339428+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"63'139","reported_seq":475,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-02-09T17:37:20.399321+0000","last_change":"2024-02-09T17:36:18.779412+0000","last_active":"2024-02-09T17:37:20.399321+0000","last_peered":"2024-02-09T17:37:20.399321+0000","last_clean":"2024-02-09T17:37:20.399321+0000","last_became_active":"2024-02-09T17:34:01.353171+0000","last_became_peered":"2024-02-09T17:34:01.353171+0000","last_unstale":"2024-02-09T17:37:20.399321+0000","last_undegraded":"2024-02-09T17:37:20.399321+0000","last_fullsized":"2024-02-09T17:37:20.399321+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:36:18.779304+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:36:18.779304+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:37:42.122054+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.12","version":"63'139","reported_seq":949,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-02-09T17:37:20.401509+0000","last_change":"2024-02-09T17:37:07.797486+0000","last_active":"2024-02-09T17:37:20.401509+0000","last_peered":"2024-02-09T17:37:20.401509+0000","last_clean":"2024-02-09T17:37:20.401509+0000","last_became_active":"2024-02-09T17:33:05.308268+0000","last_became_peered":"2024-02-09T17:33:05.308268+0000","last_unstale":"2024-02-09T17:37:20.401509+0000","last_undegraded":"2024-02-09T17:37:20.401509+0000","last_fullsized":"2024-02-09T17:37:20.401509+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:37:07.797354+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:37:07.797354+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:29.748476+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"63'139","reported_seq":503,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-02-09T17:37:20.403501+0000","last_change":"2024-02-09T17:36:27.497000+0000","last_active":"2024-02-09T17:37:20.403501+0000","last_peered":"2024-02-09T17:37:20.403501+0000","last_clean":"2024-02-09T17:37:20.403501+0000","last_became_active":"2024-02-09T17:33:05.309527+0000","last_became_peered":"2024-02-09T17:33:05.309527+0000","last_unstale":"2024-02-09T17:37:20.403501+0000","last_undegraded":"2024-02-09T17:37:20.403501+0000","last_fullsized":"2024-02-09T17:37:20.403501+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:36:27.496900+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:36:27.496900+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:37:56.755524+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":7896891416,"num_objects":120498,"num_object_clones":0,"num_object_copies":361494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":120498,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154430,"num_write_kb":8797633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":23756550144,"data_stored":23756537928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154430,"ondisk_log_size":154430,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2919,"ondisk_log_size":2919,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056822,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3762400,"kb_used_data":3356376,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":89982752,"statfs":{"total":95995035648,"available":92142338048,"internally_reserved":0,"allocated":3436929024,"data_stored":3436600199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":415758788},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:37:16 2024","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.3009999999999999,"15min":1.103},"min":{"1min":0.246,"5min":0.216,"15min":0.216},"max":{"1min":8.1660000000000004,"5min":12.836,"15min":14.913},"last":0.96499999999999997},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.335,"15min":1.198},"min":{"1min":0.27000000000000002,"5min":0.25700000000000001,"15min":0.23000000000000001},"max":{"1min":7.8499999999999996,"5min":9.7639999999999993,"15min":14.869},"last":1.637}]},{"osd":1,"last update":"Fri Feb 9 17:37:16 2024","interfaces":[{"interface":"back","average":{"1min":1.877,"5min":1.2989999999999999,"15min":1.4830000000000001},"min":{"1min":0.26700000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":7.3209999999999997,"5min":9.0079999999999991,"15min":32.957999999999998},"last":1.151},{"interface":"front","average":{"1min":1.7889999999999999,"5min":1.389,"15min":1.552},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.26700000000000002},"max":{"1min":7.3090000000000002,"5min":9.0280000000000005,"15min":32.880000000000003},"last":1.1870000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:31:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.415,"15min":1.6859999999999999},"min":{"1min":0.378,"5min":0.247,"15min":0.247},"max":{"1min":4.4189999999999996,"5min":9.4719999999999995,"15min":9.4719999999999995},"last":0.35399999999999998},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.5149999999999999,"15min":1.4399999999999999},"min":{"1min":0.378,"5min":0.252,"15min":0.252},"max":{"1min":4.8579999999999997,"5min":7.96,"15min":7.96},"last":0.60399999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:37:16 2024","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.093,"15min":1.52},"min":{"1min":0.27400000000000002,"5min":0.25700000000000001,"15min":0.247},"max":{"1min":5.6980000000000004,"5min":5.6980000000000004,"15min":21.315999999999999},"last":1.8959999999999999},{"interface":"front","average":{"1min":1.8460000000000001,"5min":1.4630000000000001,"15min":1.488},"min":{"1min":0.36599999999999999,"5min":0.312,"15min":0.26400000000000001},"max":{"1min":5.7679999999999998,"5min":7.2439999999999998,"15min":18.087},"last":0.37}]},{"osd":4,"last update":"Fri Feb 9 17:37:16 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":2.2389999999999999,"15min":1.3220000000000001},"min":{"1min":0.27900000000000003,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":3.6509999999999998,"5min":16.125,"15min":16.125},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.78000000000000003,"5min":1.5389999999999999,"15min":1.1439999999999999},"min":{"1min":0.316,"5min":0.316,"15min":0.27700000000000002},"max":{"1min":3.0139999999999998,"5min":16.091999999999999,"15min":16.091999999999999},"last":0.309}]},{"osd":5,"last update":"Fri Feb 9 17:37:16 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.8220000000000001,"15min":1.2689999999999999},"min":{"1min":0.37,"5min":0.312,"15min":0.28899999999999998},"max":{"1min":5.8819999999999997,"5min":13.946999999999999,"15min":13.946999999999999},"last":0.34599999999999997},{"interface":"front","average":{"1min":1.038,"5min":2.1179999999999999,"15min":1.3149999999999999},"min":{"1min":0.315,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":3.1760000000000002,"5min":13.414,"15min":13.414},"last":0.505}]},{"osd":6,"last update":"Fri Feb 9 17:37:16 2024","interfaces":[{"interface":"back","average":{"1min":1.2410000000000001,"5min":1.9490000000000001,"15min":1.2749999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":3.7080000000000002,"5min":12.798999999999999,"15min":12.798999999999999},"last":0.746},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.8740000000000001,"15min":1.29},"min":{"1min":0.379,"5min":0.33000000000000002,"15min":0.32300000000000001},"max":{"1min":4.3310000000000004,"5min":10.782999999999999,"15min":10.782999999999999},"last":3.399}]},{"osd":7,"last update":"Fri Feb 9 17:37:16 2024","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":2.2469999999999999,"15min":1.4490000000000001},"min":{"1min":0.38,"5min":0.312,"15min":0.312},"max":{"1min":3.6139999999999999,"5min":14.462,"15min":14.462},"last":0.53200000000000003},{"interface":"front","average":{"1min":1.0629999999999999,"5min":2.2879999999999998,"15min":1.431},"min":{"1min":0.40100000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":3.2799999999999998,"5min":14.398999999999999,"15min":14.398999999999999},"last":0.71599999999999997}]},{"osd":8,"last update":"Fri Feb 9 17:36:46 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":1.472,"15min":2.1040000000000001},"min":{"1min":0.34799999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":2.8069999999999999,"5min":13.199999999999999,"15min":13.199999999999999},"last":2.9870000000000001},{"interface":"front","average":{"1min":0.86099999999999999,"5min":1.26,"15min":1.7090000000000001},"min":{"1min":0.36899999999999999,"5min":0.318,"15min":0.318},"max":{"1min":3.1859999999999999,"5min":7.1520000000000001,"15min":7.1520000000000001},"last":2.7759999999999998}]},{"osd":9,"last update":"Fri Feb 9 17:37:04 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":1.417,"15min":1.2430000000000001},"min":{"1min":0.39300000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":2.839,"5min":12.821,"15min":12.821},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.747,"5min":1.276,"15min":1.0289999999999999},"min":{"1min":0.45400000000000001,"5min":0.38500000000000001,"15min":0.34100000000000003},"max":{"1min":2.153,"5min":7.6989999999999998,"15min":7.6989999999999998},"last":0.42599999999999999}]},{"osd":10,"last update":"Fri Feb 9 17:37:16 2024","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":1.284,"15min":1.0640000000000001},"min":{"1min":0.41299999999999998,"5min":0.36199999999999999,"15min":0.33900000000000002},"max":{"1min":3.371,"5min":9.6199999999999992,"15min":9.6199999999999992},"last":0.56000000000000005},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.51,"15min":1.101},"min":{"1min":0.40000000000000002,"5min":0.35799999999999998,"15min":0.33700000000000002},"max":{"1min":5.3680000000000003,"5min":12.856,"15min":12.856},"last":2.569}]}]},{"osd":10,"up_from":84,"seq":360777253051,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2106528,"kb_used_data":1940696,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":91638624,"statfs":{"total":95995035648,"available":93837950976,"internally_reserved":0,"allocated":1987272704,"data_stored":1986943879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":169802176},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:37:04 2024","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.5640000000000001,"15min":1.7809999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":8.5779999999999994,"5min":8.5779999999999994,"15min":8.5779999999999994},"last":0.36099999999999999},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.05,"15min":1.1020000000000001},"min":{"1min":0.27800000000000002,"5min":0.253,"15min":0.253},"max":{"1min":5.923,"5min":5.923,"15min":5.923},"last":0.27000000000000002}]},{"osd":1,"last update":"Fri Feb 9 17:33:46 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":1.345,"15min":1.1599999999999999},"min":{"1min":0.25900000000000001,"5min":0.24199999999999999,"15min":0.23499999999999999},"max":{"1min":3.1709999999999998,"5min":37.716000000000001,"15min":37.716000000000001},"last":0.26000000000000001},{"interface":"front","average":{"1min":1.121,"5min":1.3660000000000001,"15min":1.194},"min":{"1min":0.251,"5min":0.251,"15min":0.245},"max":{"1min":5.3049999999999997,"5min":38.039999999999999,"15min":38.039999999999999},"last":0.27800000000000002}]},{"osd":2,"last update":"Fri Feb 9 17:36:51 2024","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":0.98799999999999999,"15min":1.1910000000000001},"min":{"1min":0.29799999999999999,"5min":0.27100000000000002,"15min":0.222},"max":{"1min":3.7440000000000002,"5min":6.4199999999999999,"15min":13.351000000000001},"last":1.3600000000000001},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.089,"15min":1.2},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.26100000000000001},"max":{"1min":3.7770000000000001,"5min":6.4829999999999997,"15min":10.596},"last":1.3720000000000001}]},{"osd":3,"last update":"Fri Feb 9 17:36:51 2024","interfaces":[{"interface":"back","average":{"1min":1.6439999999999999,"5min":2.0179999999999998,"15min":1.573},"min":{"1min":0.28699999999999998,"5min":0.26800000000000002,"15min":0.26200000000000001},"max":{"1min":8.8800000000000008,"5min":25.972999999999999,"15min":25.972999999999999},"last":0.34499999999999997},{"interface":"front","average":{"1min":1.4390000000000001,"5min":1.7829999999999999,"15min":1.4390000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":8.2509999999999994,"5min":25.882000000000001,"15min":25.882000000000001},"last":0.30399999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:36:55 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":1.274,"15min":1.012},"min":{"1min":0.307,"5min":0.307,"15min":0.26100000000000001},"max":{"1min":3.589,"5min":7.1360000000000001,"15min":7.1360000000000001},"last":0.373},{"interface":"front","average":{"1min":0.83599999999999997,"5min":1.2949999999999999,"15min":1.0409999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":3.5419999999999998,"5min":6.0119999999999996,"15min":6.8369999999999997},"last":0.44500000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:36:51 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":1.282,"15min":1.2},"min":{"1min":0.30199999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.9809999999999999,"5min":9.2349999999999994,"15min":9.2349999999999994},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.95399999999999996,"5min":1.3029999999999999,"15min":1.157},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.32200000000000001},"max":{"1min":3.8959999999999999,"5min":5.3079999999999998,"15min":8.6159999999999997},"last":0.40600000000000003}]},{"osd":6,"last update":"Fri Feb 9 17:36:51 2024","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":1.331,"15min":1.107},"min":{"1min":0.36899999999999999,"5min":0.29999999999999999,"15min":0.29099999999999998},"max":{"1min":5.6550000000000002,"5min":6.3760000000000003,"15min":6.8799999999999999},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.93600000000000005,"5min":1.7749999999999999,"15min":1.3169999999999999},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":3.1389999999999998,"5min":9.4350000000000005,"15min":9.4350000000000005},"last":1.5940000000000001}]},{"osd":7,"last update":"Fri Feb 9 17:36:51 2024","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":1.778,"15min":1.242},"min":{"1min":0.39600000000000002,"5min":0.33500000000000002,"15min":0.312},"max":{"1min":3.3180000000000001,"5min":17.754000000000001,"15min":17.754000000000001},"last":1.9299999999999999},{"interface":"front","average":{"1min":1.03,"5min":1.958,"15min":1.3580000000000001},"min":{"1min":0.45100000000000001,"5min":0.42599999999999999,"15min":0.316},"max":{"1min":3.665,"5min":11.308999999999999,"15min":11.308999999999999},"last":1.7090000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:36:35 2024","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":1.081,"15min":1.1950000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":2.3130000000000002,"5min":6.4630000000000001,"15min":6.4630000000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.84499999999999997,"15min":0.94999999999999996},"min":{"1min":0.38900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":3.794,"5min":5.3079999999999998,"15min":5.3079999999999998},"last":0.51200000000000001}]},{"osd":9,"last update":"Fri Feb 9 17:36:51 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.97899999999999998,"15min":0.94899999999999995},"min":{"1min":0.372,"5min":0.36099999999999999,"15min":0.314},"max":{"1min":3.1070000000000002,"5min":6.5049999999999999,"15min":8.7240000000000002},"last":0.38200000000000001},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.048,"15min":1.0940000000000001},"min":{"1min":0.40300000000000002,"5min":0.35899999999999999,"15min":0.29799999999999999},"max":{"1min":4.1029999999999998,"5min":6.9640000000000004,"15min":10.1},"last":0.45700000000000002}]},{"osd":11,"last update":"Fri Feb 9 17:37:16 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":1.071,"15min":1.0940000000000001},"min":{"1min":0.40600000000000003,"5min":0.379,"15min":0.36399999999999999},"max":{"1min":0.93400000000000005,"5min":7.383,"15min":8.6430000000000007},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.68000000000000005,"5min":1.143,"15min":1.0980000000000001},"min":{"1min":0.45700000000000002,"5min":0.44800000000000001,"15min":0.33400000000000002},"max":{"1min":0.95999999999999996,"5min":7.3019999999999996,"15min":9.298},"last":0.497}]}]},{"osd":9,"up_from":77,"seq":330712481984,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1753380,"kb_used_data":1473692,"kb_used_omap":1,"kb_used_meta":279678,"kb_avail":91991772,"statfs":{"total":95995035648,"available":94199574528,"internally_reserved":0,"allocated":1509060608,"data_stored":1508727711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":286390720},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:28:03 2024","interfaces":[{"interface":"back","average":{"1min":1.9510000000000001,"5min":1.454,"15min":0.85799999999999998},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":8.4499999999999993,"5min":17.495999999999999,"15min":17.495999999999999},"last":2.7360000000000002},{"interface":"front","average":{"1min":1.905,"5min":1.399,"15min":0.83499999999999996},"min":{"1min":0.32200000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":7.5990000000000002,"5min":15.401999999999999,"15min":15.401999999999999},"last":1.5449999999999999}]},{"osd":1,"last update":"Fri Feb 9 17:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.88400000000000001,"15min":1.2070000000000001},"min":{"1min":0.27100000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":4.601,"5min":5.2549999999999999,"15min":16.527999999999999},"last":3.9209999999999998},{"interface":"front","average":{"1min":1.0209999999999999,"5min":0.88500000000000001,"15min":1.095},"min":{"1min":0.255,"5min":0.222,"15min":0.222},"max":{"1min":4.282,"5min":4.282,"15min":17.771999999999998},"last":5.2889999999999997}]},{"osd":2,"last update":"Fri Feb 9 17:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":1.038,"15min":1.131},"min":{"1min":0.28399999999999997,"5min":0.249,"15min":0.249},"max":{"1min":1.8320000000000001,"5min":6.0570000000000004,"15min":13.878},"last":2.992},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.997,"15min":1.2},"min":{"1min":0.26100000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":4.0599999999999996,"5min":5.5590000000000002,"15min":13.69},"last":2.101}]},{"osd":3,"last update":"Fri Feb 9 17:36:28 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":1.4199999999999999,"15min":1.202},"min":{"1min":0.307,"5min":0.29699999999999999,"15min":0.26000000000000001},"max":{"1min":2.0950000000000002,"5min":40.18,"15min":40.18},"last":2.0619999999999998},{"interface":"front","average":{"1min":0.75900000000000001,"5min":1.4370000000000001,"15min":1.391},"min":{"1min":0.35699999999999998,"5min":0.30599999999999999,"15min":0.27900000000000003},"max":{"1min":3.4420000000000002,"5min":40.158000000000001,"15min":40.158000000000001},"last":1.329}]},{"osd":4,"last update":"Fri Feb 9 17:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":1.7070000000000001,"15min":1.2649999999999999},"min":{"1min":0.33800000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":2.2450000000000001,"5min":9.0519999999999996,"15min":22.577999999999999},"last":2.1739999999999999},{"interface":"front","average":{"1min":0.63400000000000001,"5min":1.54,"15min":1.1399999999999999},"min":{"1min":0.33300000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":2.2389999999999999,"5min":12.651,"15min":22.399999999999999},"last":1.304}]},{"osd":5,"last update":"Fri Feb 9 17:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":1.7949999999999999,"15min":1.2310000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.8049999999999999,"5min":10.689,"15min":10.689},"last":3.0539999999999998},{"interface":"front","average":{"1min":0.71499999999999997,"5min":1.3580000000000001,"15min":1.042},"min":{"1min":0.378,"5min":0.311,"15min":0.311},"max":{"1min":2.202,"5min":6.4470000000000001,"15min":6.4470000000000001},"last":1.2789999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":1.734,"15min":1.2589999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.32200000000000001},"max":{"1min":3.0590000000000002,"5min":9.1699999999999999,"15min":9.1699999999999999},"last":5.6109999999999998},{"interface":"front","average":{"1min":0.93100000000000005,"5min":1.7729999999999999,"15min":1.2589999999999999},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.35099999999999998},"max":{"1min":3.1110000000000002,"5min":9.2080000000000002,"15min":9.2080000000000002},"last":5.6449999999999996}]},{"osd":7,"last update":"Fri Feb 9 17:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":1.73,"15min":1.224},"min":{"1min":0.40999999999999998,"5min":0.38500000000000001,"15min":0.34300000000000003},"max":{"1min":3.2320000000000002,"5min":7.3710000000000004,"15min":8.0939999999999994},"last":2.2010000000000001},{"interface":"front","average":{"1min":0.98099999999999998,"5min":1.7709999999999999,"15min":1.2909999999999999},"min":{"1min":0.38400000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":3.262,"5min":8.0530000000000008,"15min":8.0530000000000008},"last":2.2250000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:36:46 2024","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":1.157,"15min":1.2789999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":4.141,"5min":5.5069999999999997,"15min":16.209},"last":3.0790000000000002},{"interface":"front","average":{"1min":0.999,"5min":1.1180000000000001,"15min":1.29},"min":{"1min":0.41299999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":2.492,"5min":10.241,"15min":16.263000000000002},"last":1.25}]},{"osd":10,"last update":"Fri Feb 9 17:36:58 2024","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":1.111,"15min":1.071},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":2.7189999999999999,"5min":6.0659999999999998,"15min":9.5090000000000003},"last":2.1459999999999999},{"interface":"front","average":{"1min":1.022,"5min":0.94799999999999995,"15min":1.0169999999999999},"min":{"1min":0.40000000000000002,"5min":0.38500000000000001,"15min":0.36099999999999999},"max":{"1min":2.395,"5min":4.0759999999999996,"15min":8.0670000000000002},"last":1.2130000000000001}]},{"osd":11,"last update":"Fri Feb 9 17:36:58 2024","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.4710000000000001,"15min":1.3340000000000001},"min":{"1min":0.42599999999999999,"5min":0.42399999999999999,"15min":0.34699999999999998},"max":{"1min":7.2190000000000003,"5min":11.541,"15min":11.541},"last":3.0249999999999999},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.405,"15min":1.147},"min":{"1min":0.51100000000000001,"5min":0.374,"15min":0.374},"max":{"1min":3.1600000000000001,"5min":10.32,"15min":10.32},"last":1.913}]}]},{"osd":8,"up_from":171,"seq":734439407727,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":132504,"kb_used_data":856,"kb_used_omap":212,"kb_used_meta":131243,"kb_avail":93612648,"statfs":{"total":95995035648,"available":95859351552,"internally_reserved":0,"allocated":876544,"data_stored":547719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217579,"internal_metadata":134393365},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:36:31 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":0.98499999999999999,"15min":1.3899999999999999},"min":{"1min":0.246,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":3.0129999999999999,"5min":8.0289999999999999,"15min":8.0289999999999999},"last":3.3809999999999998},{"interface":"front","average":{"1min":0.89200000000000002,"5min":1.0589999999999999,"15min":1.3660000000000001},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":3.0960000000000001,"5min":7.5990000000000002,"15min":7.7969999999999997},"last":2.117}]},{"osd":1,"last update":"Fri Feb 9 17:36:31 2024","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.2470000000000001,"15min":1.2669999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.0810000000000004,"5min":7.8719999999999999,"15min":7.8719999999999999},"last":0.80000000000000004},{"interface":"front","average":{"1min":1.05,"5min":1.1879999999999999,"15min":1.841},"min":{"1min":0.23400000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":3.6680000000000001,"5min":8.5519999999999996,"15min":11.101000000000001},"last":3.407}]},{"osd":2,"last update":"Fri Feb 9 17:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":1.522,"15min":1.45},"min":{"1min":0.29399999999999998,"5min":0.28199999999999997,"15min":0.255},"max":{"1min":3.8239999999999998,"5min":10.423,"15min":10.423},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.93400000000000005,"5min":1.3080000000000001,"15min":1.611},"min":{"1min":0.373,"5min":0.34300000000000003,"15min":0.313},"max":{"1min":3.847,"5min":10.504,"15min":10.504},"last":0.747}]},{"osd":3,"last update":"Fri Feb 9 17:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":1.4790000000000001,"15min":2.0289999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.28699999999999998},"max":{"1min":3.7210000000000001,"5min":24.030999999999999,"15min":24.030999999999999},"last":3.3530000000000002},{"interface":"front","average":{"1min":1.163,"5min":1.5880000000000001,"15min":2.327},"min":{"1min":0.45500000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":3.1110000000000002,"5min":24.318999999999999,"15min":24.318999999999999},"last":3.431}]},{"osd":4,"last update":"Fri Feb 9 17:36:31 2024","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":1.8500000000000001,"15min":1.238},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.27500000000000002},"max":{"1min":3.2010000000000001,"5min":53.060000000000002,"15min":53.060000000000002},"last":0.31},{"interface":"front","average":{"1min":0.60999999999999999,"5min":2.0760000000000001,"15min":1.522},"min":{"1min":0.32900000000000001,"5min":0.313,"15min":0.313},"max":{"1min":1.4430000000000001,"5min":53.311999999999998,"15min":53.311999999999998},"last":0.35999999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":1.7470000000000001,"15min":1.589},"min":{"1min":0.36399999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":2.8980000000000001,"5min":10.619999999999999,"15min":37.356999999999999},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.96999999999999997,"5min":1.3999999999999999,"15min":1.143},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":3.0739999999999998,"5min":4.984,"15min":37.25},"last":0.51400000000000001}]},{"osd":7,"last update":"Fri Feb 9 17:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":1.9039999999999999,"15min":1.4750000000000001},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":3.4399999999999999,"5min":12.731999999999999,"15min":37.585000000000001},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.83399999999999996,"5min":1.9430000000000001,"15min":1.6459999999999999},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.312},"max":{"1min":3.4710000000000001,"5min":12.779,"15min":37.918999999999997},"last":0.432}]},{"osd":9,"last update":"Fri Feb 9 17:36:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.2190000000000001,"15min":1.1870000000000001},"min":{"1min":0.44900000000000001,"5min":0.34100000000000003,"15min":0.29699999999999999},"max":{"1min":3.2440000000000002,"5min":9.4900000000000002,"15min":9.4900000000000002},"last":0.57899999999999996},{"interface":"front","average":{"1min":1.006,"5min":1.266,"15min":1.23},"min":{"1min":0.36599999999999999,"5min":0.35399999999999998,"15min":0.32000000000000001},"max":{"1min":4.101,"5min":17.434999999999999,"15min":17.434999999999999},"last":3.3140000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":1.127,"15min":1.1910000000000001},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.33700000000000002},"max":{"1min":1.3979999999999999,"5min":10.65,"15min":10.65},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.99199999999999999,"5min":1.1699999999999999,"15min":1.0820000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":3.024,"5min":8.2799999999999994,"15min":8.2799999999999994},"last":0.45700000000000002}]},{"osd":11,"last update":"Fri Feb 9 17:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":1.5760000000000001,"15min":1.381},"min":{"1min":0.42899999999999999,"5min":0.375,"15min":0.375},"max":{"1min":5.4980000000000002,"5min":14.433999999999999,"15min":14.433999999999999},"last":0.77300000000000002},{"interface":"front","average":{"1min":1.012,"5min":1.3500000000000001,"15min":1.4610000000000001},"min":{"1min":0.48399999999999999,"5min":0.45000000000000001,"15min":0.38100000000000001},"max":{"1min":5.3239999999999998,"5min":11.622,"15min":11.622},"last":0.53300000000000003}]}]},{"osd":7,"up_from":57,"seq":244813136073,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1794272,"kb_used_data":1474328,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":91950880,"statfs":{"total":95995035648,"available":94157701120,"internally_reserved":0,"allocated":1509711872,"data_stored":1509383047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":327612874},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:37:04 2024","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":2.6120000000000001,"15min":3.294},"min":{"1min":0.26200000000000001,"5min":0.254,"15min":0.254},"max":{"1min":4.1719999999999997,"5min":36.753,"15min":36.753},"last":0.26500000000000001},{"interface":"front","average":{"1min":1.913,"5min":2.9369999999999998,"15min":3.5489999999999999},"min":{"1min":0.29199999999999998,"5min":0.253,"15min":0.253},"max":{"1min":9.0640000000000001,"5min":36.918999999999997,"15min":36.918999999999997},"last":0.36499999999999999}]},{"osd":1,"last update":"Fri Feb 9 17:37:08 2024","interfaces":[{"interface":"back","average":{"1min":1.3600000000000001,"5min":1.9610000000000001,"15min":1.9350000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":6.3920000000000003,"5min":20.553999999999998,"15min":20.553999999999998},"last":3.895},{"interface":"front","average":{"1min":2.6219999999999999,"5min":2.0099999999999998,"15min":1.657},"min":{"1min":0.28399999999999997,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":8.6430000000000007,"5min":20.568000000000001,"15min":20.568000000000001},"last":2.5800000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:36:29 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.8029999999999999,"15min":1.456},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.26000000000000001},"max":{"1min":2.722,"5min":11.324,"15min":11.324},"last":2.641},{"interface":"front","average":{"1min":1.0569999999999999,"5min":2.004,"15min":1.5349999999999999},"min":{"1min":0.32100000000000001,"5min":0.309,"15min":0.28299999999999997},"max":{"1min":2.742,"5min":19.361999999999998,"15min":19.361999999999998},"last":2.6629999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:36:29 2024","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":2.077,"15min":1.554},"min":{"1min":0.36399999999999999,"5min":0.28899999999999998,"15min":0.26900000000000002},"max":{"1min":2.7010000000000001,"5min":20.754000000000001,"15min":27.719000000000001},"last":3.9279999999999999},{"interface":"front","average":{"1min":0.79700000000000004,"5min":2.2149999999999999,"15min":1.5780000000000001},"min":{"1min":0.34799999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":1.958,"5min":24.457000000000001,"15min":24.457000000000001},"last":3.6320000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:36:29 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":2.3319999999999999,"15min":1.286},"min":{"1min":0.28000000000000003,"5min":0.25700000000000001,"15min":0.248},"max":{"1min":2.544,"5min":14.372999999999999,"15min":14.372999999999999},"last":0.31},{"interface":"front","average":{"1min":0.77700000000000002,"5min":2.415,"15min":1.2909999999999999},"min":{"1min":0.379,"5min":0.34699999999999998,"15min":0.22800000000000001},"max":{"1min":2.8599999999999999,"5min":14.401,"15min":14.401},"last":0.35199999999999998}]},{"osd":5,"last update":"Fri Feb 9 17:35:28 2024","interfaces":[{"interface":"back","average":{"1min":2.073,"5min":2.2999999999999998,"15min":1.1619999999999999},"min":{"1min":0.36899999999999999,"5min":0.28599999999999998,"15min":0.21299999999999999},"max":{"1min":13.409000000000001,"5min":13.409000000000001,"15min":13.409000000000001},"last":0.53300000000000003},{"interface":"front","average":{"1min":1.946,"5min":2.1230000000000002,"15min":1.1240000000000001},"min":{"1min":0.30599999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":10.553000000000001,"5min":11.598000000000001,"15min":11.598000000000001},"last":0.48999999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:36:29 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":2.3559999999999999,"15min":1.3700000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":2.6909999999999998,"5min":25.965,"15min":25.965},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":1.8959999999999999,"15min":1.155},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.28699999999999998},"max":{"1min":1.6279999999999999,"5min":9.4090000000000007,"15min":9.4090000000000007},"last":0.38}]},{"osd":8,"last update":"Fri Feb 9 17:36:29 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":1.6970000000000001,"15min":1.4970000000000001},"min":{"1min":0.442,"5min":0.36599999999999999,"15min":0.32300000000000001},"max":{"1min":2.641,"5min":9.0809999999999995,"15min":9.0809999999999995},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.80000000000000004,"5min":1.635,"15min":1.2529999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":2.5070000000000001,"5min":11.481,"15min":11.481},"last":0.432}]},{"osd":9,"last update":"Fri Feb 9 17:37:17 2024","interfaces":[{"interface":"back","average":{"1min":1.7410000000000001,"5min":1.9359999999999999,"15min":1.4590000000000001},"min":{"1min":0.50600000000000001,"5min":0.40899999999999997,"15min":0.39600000000000002},"max":{"1min":4.6820000000000004,"5min":8.9900000000000002,"15min":41.286999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":1.671,"5min":1.9419999999999999,"15min":1.4950000000000001},"min":{"1min":0.49199999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":5.7240000000000002,"5min":11.417999999999999,"15min":41.234999999999999},"last":0.46800000000000003}]},{"osd":10,"last update":"Fri Feb 9 17:36:44 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":1.8109999999999999,"15min":1.3009999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.35899999999999999},"max":{"1min":1.6399999999999999,"5min":9.4830000000000005,"15min":9.4830000000000005},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.95299999999999996,"5min":2.044,"15min":1.385},"min":{"1min":0.52200000000000002,"5min":0.45600000000000002,"15min":0.35599999999999998},"max":{"1min":2.5289999999999999,"5min":9.109,"15min":9.109},"last":0.48499999999999999}]},{"osd":11,"last update":"Fri Feb 9 17:37:08 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":2.2519999999999998,"15min":1.5169999999999999},"min":{"1min":0.44500000000000001,"5min":0.41299999999999998,"15min":0.40999999999999998},"max":{"1min":6.6360000000000001,"5min":12.885999999999999,"15min":12.885999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":1.444,"5min":1.883,"15min":1.3620000000000001},"min":{"1min":0.5,"5min":0.41299999999999998,"15min":0.39500000000000002},"max":{"1min":5.6230000000000002,"5min":9.9489999999999998,"15min":9.9489999999999998},"last":2.8159999999999998}]}]},{"osd":6,"up_from":49,"seq":210453397710,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3242272,"kb_used_data":2962392,"kb_used_omap":1,"kb_used_meta":279678,"kb_avail":90502880,"statfs":{"total":95995035648,"available":92674949120,"internally_reserved":0,"allocated":3033489408,"data_stored":3033160583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":286390726},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:37:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.6020000000000001,"15min":1.369},"min":{"1min":0.26300000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.4510000000000001,"5min":12.544,"15min":24.544},"last":0.31900000000000001},{"interface":"front","average":{"1min":1.296,"5min":1.6060000000000001,"15min":1.302},"min":{"1min":0.35499999999999998,"5min":0.26500000000000001,"15min":0.217},"max":{"1min":4.4950000000000001,"5min":8.3689999999999998,"15min":8.3689999999999998},"last":0.35699999999999998}]},{"osd":1,"last update":"Fri Feb 9 17:37:04 2024","interfaces":[{"interface":"back","average":{"1min":1.6950000000000001,"5min":2.2029999999999998,"15min":1.579},"min":{"1min":0.30399999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":6.8380000000000001,"5min":17.762,"15min":23.954000000000001},"last":0.67700000000000005},{"interface":"front","average":{"1min":1.7250000000000001,"5min":2.2639999999999998,"15min":1.625},"min":{"1min":0.34499999999999997,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":6.8929999999999998,"5min":17.734999999999999,"15min":23.984999999999999},"last":0.71499999999999997}]},{"osd":2,"last update":"Fri Feb 9 17:37:04 2024","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.623,"15min":1.462},"min":{"1min":0.34000000000000002,"5min":0.28999999999999998,"15min":0.22600000000000001},"max":{"1min":4.0830000000000002,"5min":8.0239999999999991,"15min":24.329999999999998},"last":0.57499999999999996},{"interface":"front","average":{"1min":1.268,"5min":1.6040000000000001,"15min":1.4690000000000001},"min":{"1min":0.377,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":4.1159999999999997,"5min":10.156000000000001,"15min":12.906000000000001},"last":0.63700000000000001}]},{"osd":3,"last update":"Fri Feb 9 17:37:04 2024","interfaces":[{"interface":"back","average":{"1min":1.8069999999999999,"5min":2.2759999999999998,"15min":1.7490000000000001},"min":{"1min":0.30099999999999999,"5min":0.25600000000000001,"15min":0.248},"max":{"1min":8.7780000000000005,"5min":16.931999999999999,"15min":32.545000000000002},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.82299999999999995,"5min":2.2120000000000002,"15min":1.657},"min":{"1min":0.31900000000000001,"5min":0.29599999999999999,"15min":0.26600000000000001},"max":{"1min":4.2510000000000003,"5min":17.047000000000001,"15min":32.670999999999999},"last":2.9169999999999998}]},{"osd":4,"last update":"Fri Feb 9 17:37:04 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":2.2879999999999998,"15min":1.286},"min":{"1min":0.255,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":5.944,"5min":19.798999999999999,"15min":19.798999999999999},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.92500000000000004,"5min":1.8100000000000001,"15min":1.2230000000000001},"min":{"1min":0.377,"5min":0.28000000000000003,"15min":0.26000000000000001},"max":{"1min":5.9260000000000002,"5min":10.702999999999999,"15min":10.702999999999999},"last":0.48299999999999998}]},{"osd":5,"last update":"Fri Feb 9 17:37:04 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":1.909,"15min":1.198},"min":{"1min":0.318,"5min":0.29499999999999998,"15min":0.29199999999999998},"max":{"1min":2.2090000000000001,"5min":10.968999999999999,"15min":10.968999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.73699999999999999,"5min":2.105,"15min":1.26},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.27400000000000002},"max":{"1min":2.2370000000000001,"5min":13.91,"15min":13.91},"last":0.378}]},{"osd":7,"last update":"Fri Feb 9 17:36:39 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":2.1680000000000001,"15min":1.2869999999999999},"min":{"1min":0.32200000000000001,"5min":0.311,"15min":0.28000000000000003},"max":{"1min":5.0579999999999998,"5min":14.022,"15min":14.022},"last":0.38},{"interface":"front","average":{"1min":0.90300000000000002,"5min":2.2730000000000001,"15min":1.405},"min":{"1min":0.36699999999999999,"5min":0.35699999999999998,"15min":0.309},"max":{"1min":2.9420000000000002,"5min":15.458,"15min":15.458},"last":0.52500000000000002}]},{"osd":8,"last update":"Fri Feb 9 17:26:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":0.876,"15min":0.79600000000000004},"min":{"1min":0.39900000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":4.0339999999999998,"5min":4.4880000000000004,"15min":4.4880000000000004},"last":4.3840000000000003},{"interface":"front","average":{"1min":1.0469999999999999,"5min":0.88600000000000001,"15min":0.78300000000000003},"min":{"1min":0.375,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":3.6539999999999999,"5min":4.8570000000000002,"15min":4.8570000000000002},"last":4.4210000000000003}]},{"osd":9,"last update":"Fri Feb 9 17:37:12 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":1.494,"15min":1.1930000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.1779999999999999,"5min":8.3409999999999993,"15min":10.803000000000001},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.70999999999999996,"5min":1.629,"15min":1.2909999999999999},"min":{"1min":0.40999999999999998,"5min":0.40400000000000003,"15min":0.40300000000000002},"max":{"1min":1.7190000000000001,"5min":8.1630000000000003,"15min":8.1630000000000003},"last":2.5139999999999998}]},{"osd":10,"last update":"Fri Feb 9 17:36:39 2024","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":1.631,"15min":1.325},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.35399999999999998},"max":{"1min":5.5469999999999997,"5min":7.734,"15min":9.5399999999999991},"last":0.61299999999999999},{"interface":"front","average":{"1min":1.0269999999999999,"5min":2.0129999999999999,"15min":1.417},"min":{"1min":0.41999999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":4.8819999999999997,"5min":13.836,"15min":13.836},"last":0.59499999999999997}]},{"osd":11,"last update":"Fri Feb 9 17:37:04 2024","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.774,"15min":1.5640000000000001},"min":{"1min":0.40799999999999997,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":4.1609999999999996,"5min":10.273999999999999,"15min":10.273999999999999},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.746,"5min":1.893,"15min":1.7070000000000001},"min":{"1min":0.45100000000000001,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.9279999999999999,"5min":8.2739999999999991,"15min":8.2739999999999991},"last":0.84699999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575081,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4658852,"kb_used_data":4306524,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":89086300,"statfs":{"total":95995035648,"available":91224371200,"internally_reserved":0,"allocated":4409880576,"data_stored":4409547679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":360315333},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:37:10 2024","interfaces":[{"interface":"back","average":{"1min":3.6230000000000002,"5min":1.581,"15min":1.4199999999999999},"min":{"1min":0.26300000000000001,"5min":0.223,"15min":0.214},"max":{"1min":14.917,"5min":14.917,"15min":14.917},"last":0.23000000000000001},{"interface":"front","average":{"1min":2.3279999999999998,"5min":1.4430000000000001,"15min":1.387},"min":{"1min":0.30099999999999999,"5min":0.245,"15min":0.214},"max":{"1min":8.7260000000000009,"5min":8.7260000000000009,"15min":9.8439999999999994},"last":6.5999999999999996}]},{"osd":2,"last update":"Fri Feb 9 17:36:39 2024","interfaces":[{"interface":"back","average":{"1min":2.4430000000000001,"5min":1.423,"15min":1.474},"min":{"1min":0.27600000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":26.361999999999998,"5min":26.361999999999998,"15min":26.361999999999998},"last":0.28899999999999998},{"interface":"front","average":{"1min":2.3650000000000002,"5min":1.397,"15min":1.597},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":26.087,"5min":26.087,"15min":26.087},"last":1.631}]},{"osd":3,"last update":"Fri Feb 9 17:36:32 2024","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":8.5030000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":5.585},{"interface":"front","average":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.3480000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":4.0250000000000004,"5min":4.0250000000000004,"15min":4.0250000000000004},"last":1.5529999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:37:10 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":1.6950000000000001,"15min":2.0289999999999999},"min":{"1min":0.317,"5min":0.253,"15min":0.253},"max":{"1min":8.3460000000000001,"5min":11.766999999999999,"15min":11.766999999999999},"last":0.378},{"interface":"front","average":{"1min":2.3069999999999999,"5min":1.923,"15min":1.9319999999999999},"min":{"1min":0.309,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":10.512,"5min":10.512,"15min":10.512},"last":0.318}]},{"osd":5,"last update":"Fri Feb 9 17:36:50 2024","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.774,"15min":1.518},"min":{"1min":0.374,"5min":0.28599999999999998,"15min":0.27000000000000002},"max":{"1min":8.2859999999999996,"5min":13.544,"15min":13.544},"last":0.33000000000000002},{"interface":"front","average":{"1min":2.0089999999999999,"5min":1.839,"15min":1.353},"min":{"1min":0.36699999999999999,"5min":0.26800000000000002,"15min":0.26000000000000001},"max":{"1min":4.843,"5min":12.117000000000001,"15min":12.117000000000001},"last":0.34899999999999998}]},{"osd":6,"last update":"Fri Feb 9 17:37:10 2024","interfaces":[{"interface":"back","average":{"1min":1.8959999999999999,"5min":1.732,"15min":1.5880000000000001},"min":{"1min":0.35999999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":8.25,"5min":8.7210000000000001,"15min":10.275},"last":5.6159999999999997},{"interface":"front","average":{"1min":2.0209999999999999,"5min":1.921,"15min":1.5860000000000001},"min":{"1min":0.41799999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":8.5999999999999996,"5min":9.0069999999999997,"15min":9.0069999999999997},"last":6.4089999999999998}]},{"osd":7,"last update":"Fri Feb 9 17:34:33 2024","interfaces":[{"interface":"back","average":{"1min":2.1560000000000001,"5min":1.9099999999999999,"15min":1.5609999999999999},"min":{"1min":0.35699999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":9.0489999999999995,"5min":15.148999999999999,"15min":15.148999999999999},"last":0.76600000000000001},{"interface":"front","average":{"1min":1.419,"5min":1.3740000000000001,"15min":1.425},"min":{"1min":0.31,"5min":0.31,"15min":0.28299999999999997},"max":{"1min":7.3380000000000001,"5min":13.834,"15min":13.834},"last":0.61499999999999999}]},{"osd":8,"last update":"Fri Feb 9 17:36:32 2024","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.2929999999999999,"15min":1.3959999999999999},"min":{"1min":0.39200000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":3.8079999999999998,"5min":7.4349999999999996,"15min":7.4349999999999996},"last":0.439},{"interface":"front","average":{"1min":1.333,"5min":1.569,"15min":1.9790000000000001},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":8.3780000000000001,"5min":10.433999999999999,"15min":10.433999999999999},"last":0.505}]},{"osd":9,"last update":"Fri Feb 9 17:37:19 2024","interfaces":[{"interface":"back","average":{"1min":2.0550000000000002,"5min":1.627,"15min":1.5549999999999999},"min":{"1min":0.45100000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":8.3170000000000002,"5min":9.5839999999999996,"15min":9.5839999999999996},"last":0.53500000000000003},{"interface":"front","average":{"1min":2.6859999999999999,"5min":1.5880000000000001,"15min":1.5820000000000001},"min":{"1min":0.38900000000000001,"5min":0.37,"15min":0.30099999999999999},"max":{"1min":11.366,"5min":11.366,"15min":12.117000000000001},"last":1.7210000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:36:50 2024","interfaces":[{"interface":"back","average":{"1min":1.9099999999999999,"5min":1.425,"15min":1.321},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.35999999999999999},"max":{"1min":4.8810000000000002,"5min":7.9809999999999999,"15min":7.9809999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":1.958,"5min":1.512,"15min":1.581},"min":{"1min":0.48099999999999998,"5min":0.44400000000000001,"15min":0.33400000000000002},"max":{"1min":4.9080000000000004,"5min":8.0020000000000007,"15min":12.087999999999999},"last":0.496}]},{"osd":11,"last update":"Fri Feb 9 17:36:56 2024","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":1.585,"15min":2.1200000000000001},"min":{"1min":0.437,"5min":0.42199999999999999,"15min":0.32000000000000001},"max":{"1min":5.4640000000000004,"5min":8.3079999999999998,"15min":12.169},"last":1.6839999999999999},{"interface":"front","average":{"1min":1.5509999999999999,"5min":1.619,"15min":2.0379999999999998},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.36599999999999999},"max":{"1min":5.4880000000000004,"5min":8.4339999999999993,"15min":12.144},"last":1.6990000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738605,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1822048,"kb_used_data":1475608,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":91923104,"statfs":{"total":95995035648,"available":94129258496,"internally_reserved":0,"allocated":1511022592,"data_stored":1510693767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 9 17:36:56 2024","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.1919999999999999,"15min":1.2809999999999999},"min":{"1min":0.22800000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":3.5030000000000001,"5min":11.236000000000001,"15min":14.166},"last":9.516},{"interface":"front","average":{"1min":2.4870000000000001,"5min":1.3380000000000001,"15min":1.5209999999999999},"min":{"1min":0.32000000000000001,"5min":0.26800000000000002,"15min":0.215},"max":{"1min":7.8579999999999997,"5min":9.5649999999999995,"15min":20.177},"last":4.702}]},{"osd":2,"last update":"Fri Feb 9 17:27:44 2024","interfaces":[{"interface":"back","average":{"1min":3.976,"5min":3.976,"15min":3.976},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":13.066000000000001,"5min":13.066000000000001,"15min":13.066000000000001},"last":3.0569999999999999},{"interface":"front","average":{"1min":3.968,"5min":3.968,"15min":3.968},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":13.193,"5min":13.193,"15min":13.193},"last":10.872999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:36:50 2024","interfaces":[{"interface":"back","average":{"1min":2.21,"5min":1.7290000000000001,"15min":1.5489999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":6.1139999999999999,"5min":16.472000000000001,"15min":16.472000000000001},"last":1.532},{"interface":"front","average":{"1min":2.1070000000000002,"5min":1.3919999999999999,"15min":1.399},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.223},"max":{"1min":15.132999999999999,"5min":15.132999999999999,"15min":16.984999999999999},"last":4.6189999999999998}]},{"osd":4,"last update":"Fri Feb 9 17:37:23 2024","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.8520000000000001,"15min":1.532},"min":{"1min":0.33200000000000002,"5min":0.29999999999999999,"15min":0.253},"max":{"1min":2.823,"5min":14.419,"15min":14.419},"last":4.6550000000000002},{"interface":"front","average":{"1min":0.86599999999999999,"5min":2.0369999999999999,"15min":1.627},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":6.0819999999999999,"5min":14.465999999999999,"15min":14.465999999999999},"last":3.2229999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:36:40 2024","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.6539999999999999,"15min":1.5029999999999999},"min":{"1min":0.36499999999999999,"5min":0.32400000000000001,"15min":0.28999999999999998},"max":{"1min":3.5259999999999998,"5min":9.532,"15min":9.8569999999999993},"last":4.7729999999999997},{"interface":"front","average":{"1min":1.506,"5min":1.9370000000000001,"15min":1.6399999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.26700000000000002},"max":{"1min":5.7919999999999998,"5min":9.5640000000000001,"15min":9.9049999999999994},"last":4.8330000000000002}]},{"osd":6,"last update":"Fri Feb 9 17:37:13 2024","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.883,"15min":1.6240000000000001},"min":{"1min":0.34599999999999997,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":6.1619999999999999,"5min":13.102,"15min":13.102},"last":3.3300000000000001},{"interface":"front","average":{"1min":1.304,"5min":1.956,"15min":1.625},"min":{"1min":0.379,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":6.1900000000000004,"5min":13.129,"15min":13.129},"last":3.3530000000000002}]},{"osd":7,"last update":"Fri Feb 9 17:36:30 2024","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":2.2879999999999998,"15min":1.663},"min":{"1min":0.42999999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":2.528,"5min":13.063000000000001,"15min":13.063000000000001},"last":3.274},{"interface":"front","average":{"1min":1.141,"5min":2.0190000000000001,"15min":1.6140000000000001},"min":{"1min":0.36099999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":3.0939999999999999,"5min":12.959,"15min":16.765999999999998},"last":0.51500000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:36:34 2024","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.3300000000000001,"15min":1.571},"min":{"1min":0.40999999999999998,"5min":0.36599999999999999,"15min":0.35799999999999998},"max":{"1min":3.2429999999999999,"5min":10.207000000000001,"15min":10.207000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.5429999999999999,"15min":1.552},"min":{"1min":0.40400000000000003,"5min":0.38600000000000001,"15min":0.36799999999999999},"max":{"1min":3.2719999999999998,"5min":10.231999999999999,"15min":16.41},"last":0.61299999999999999}]},{"osd":9,"last update":"Fri Feb 9 17:37:23 2024","interfaces":[{"interface":"back","average":{"1min":1.284,"5min":1.5740000000000001,"15min":1.5069999999999999},"min":{"1min":0.36399999999999999,"5min":0.34699999999999998,"15min":0.312},"max":{"1min":6.1280000000000001,"5min":13.02,"15min":13.02},"last":3.306},{"interface":"front","average":{"1min":2.3319999999999999,"5min":1.76,"15min":1.536},"min":{"1min":0.42299999999999999,"5min":0.373,"15min":0.36199999999999999},"max":{"1min":11.295999999999999,"5min":11.295999999999999,"15min":16.82},"last":0.58599999999999997}]},{"osd":10,"last update":"Fri Feb 9 17:36:40 2024","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.51,"15min":1.5089999999999999},"min":{"1min":0.45400000000000001,"5min":0.42199999999999999,"15min":0.38500000000000001},"max":{"1min":3.2639999999999998,"5min":13.888,"15min":16.876000000000001},"last":0.56000000000000005},{"interface":"front","average":{"1min":1.264,"5min":1.3580000000000001,"15min":1.4370000000000001},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.39800000000000002},"max":{"1min":3.5750000000000002,"5min":9.3350000000000009,"15min":9.7970000000000006},"last":4.6790000000000003}]},{"osd":11,"last update":"Fri Feb 9 17:37:13 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":2.1970000000000001,"15min":1.665},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.42299999999999999},"max":{"1min":3.5510000000000002,"5min":40.021000000000001,"15min":40.021000000000001},"last":4.7270000000000003},{"interface":"front","average":{"1min":1.6870000000000001,"5min":2.1739999999999999,"15min":1.7589999999999999},"min":{"1min":0.50600000000000001,"5min":0.46000000000000002,"15min":0.443},"max":{"1min":3.6200000000000001,"5min":40.055999999999997,"15min":40.055999999999997},"last":4.8099999999999996}]}]},{"osd":2,"up_from":18,"seq":77309411555,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1112680,"kb_used_data":955168,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":92632472,"statfs":{"total":95995035648,"available":94855651328,"internally_reserved":0,"allocated":978092032,"data_stored":977759655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":161282501},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":1.4510000000000001,"15min":1.006},"min":{"1min":0.29999999999999999,"5min":0.23699999999999999,"15min":0.221},"max":{"1min":4.9089999999999998,"5min":17.251000000000001,"15min":17.251000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.74399999999999999,"5min":1.611,"15min":1.113},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.22900000000000001},"max":{"1min":4.7720000000000002,"5min":23.097999999999999,"15min":23.097999999999999},"last":0.875}]},{"osd":1,"last update":"Fri Feb 9 17:36:29 2024","interfaces":[{"interface":"back","average":{"1min":2.2330000000000001,"5min":1.3600000000000001,"15min":1.425},"min":{"1min":0.26100000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":9.5199999999999996,"5min":22.306000000000001,"15min":27.289000000000001},"last":0.69099999999999995},{"interface":"front","average":{"1min":1.347,"5min":1.5900000000000001,"15min":1.581},"min":{"1min":0.27100000000000002,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":7.008,"5min":44.838999999999999,"15min":44.838999999999999},"last":2.2160000000000002}]},{"osd":3,"last update":"Fri Feb 9 17:37:05 2024","interfaces":[{"interface":"back","average":{"1min":2.2599999999999998,"5min":1.659,"15min":1.653},"min":{"1min":0.27300000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":9.5630000000000006,"5min":24.353000000000002,"15min":37.634999999999998},"last":0.60299999999999998},{"interface":"front","average":{"1min":2.984,"5min":1.5369999999999999,"15min":1.484},"min":{"1min":0.27200000000000002,"5min":0.251,"15min":0.23100000000000001},"max":{"1min":10.801,"5min":10.801,"15min":38.924999999999997},"last":0.246}]},{"osd":4,"last update":"Fri Feb 9 17:37:05 2024","interfaces":[{"interface":"back","average":{"1min":2.6709999999999998,"5min":1.9570000000000001,"15min":1.54},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.254},"max":{"1min":10.741,"5min":10.741,"15min":10.741},"last":0.31900000000000001},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.7789999999999999,"15min":1.3600000000000001},"min":{"1min":0.29099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.359,"5min":9.6509999999999998,"15min":13.305999999999999},"last":2.0830000000000002}]},{"osd":5,"last update":"Fri Feb 9 17:36:42 2024","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.613,"15min":1.4330000000000001},"min":{"1min":0.379,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":3.7189999999999999,"5min":7.7539999999999996,"15min":8.6859999999999999},"last":2.1400000000000001},{"interface":"front","average":{"1min":2.5209999999999999,"5min":2.036,"15min":1.423},"min":{"1min":0.375,"5min":0.30199999999999999,"15min":0.29899999999999999},"max":{"1min":10.781000000000001,"5min":10.781000000000001,"15min":13.278},"last":0.34899999999999998}]},{"osd":6,"last update":"Fri Feb 9 17:37:05 2024","interfaces":[{"interface":"back","average":{"1min":1.6419999999999999,"5min":1.744,"15min":1.522},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.32100000000000001},"max":{"1min":6.4450000000000003,"5min":6.931,"15min":8.8399999999999999},"last":2.1160000000000001},{"interface":"front","average":{"1min":1.5329999999999999,"5min":2.0409999999999999,"15min":1.637},"min":{"1min":0.433,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":6.5380000000000003,"5min":11.269,"15min":11.269},"last":2.238}]},{"osd":7,"last update":"Fri Feb 9 17:36:29 2024","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":1.851,"15min":1.4990000000000001},"min":{"1min":0.40999999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":3.6640000000000001,"5min":11.365,"15min":14.541},"last":2.1640000000000001},{"interface":"front","average":{"1min":1.1140000000000001,"5min":2.0830000000000002,"15min":1.7210000000000001},"min":{"1min":0.46100000000000002,"5min":0.371,"15min":0.371},"max":{"1min":4.8230000000000004,"5min":22.359999999999999,"15min":22.359999999999999},"last":0.45200000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:37:05 2024","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.3380000000000001,"15min":1.2030000000000001},"min":{"1min":0.41199999999999998,"5min":0.38,"15min":0.38},"max":{"1min":3.8919999999999999,"5min":22.393999999999998,"15min":22.393999999999998},"last":0.56999999999999995},{"interface":"front","average":{"1min":1.8080000000000001,"5min":1.413,"15min":1.4430000000000001},"min":{"1min":0.36399999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":6.508,"5min":9.9260000000000002,"15min":9.9260000000000002},"last":2.1920000000000002}]},{"osd":9,"last update":"Fri Feb 9 17:37:16 2024","interfaces":[{"interface":"back","average":{"1min":3.2709999999999999,"5min":1.631,"15min":1.5009999999999999},"min":{"1min":0.39500000000000002,"5min":0.371,"15min":0.371},"max":{"1min":10.819000000000001,"5min":10.819000000000001,"15min":10.819000000000001},"last":0.52100000000000002},{"interface":"front","average":{"1min":3.2639999999999998,"5min":1.7849999999999999,"15min":1.4319999999999999},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":10.839,"5min":10.839,"15min":13.218},"last":0.46400000000000002}]},{"osd":10,"last update":"Fri Feb 9 17:36:54 2024","interfaces":[{"interface":"back","average":{"1min":1.984,"5min":1.579,"15min":1.333},"min":{"1min":0.501,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":8.2609999999999992,"5min":22.425999999999998,"15min":22.425999999999998},"last":0.63200000000000001},{"interface":"front","average":{"1min":2.0209999999999999,"5min":1.617,"15min":1.4019999999999999},"min":{"1min":0.54800000000000004,"5min":0.47099999999999997,"15min":0.42099999999999999},"max":{"1min":8.2970000000000006,"5min":22.454999999999998,"15min":22.454999999999998},"last":0.66000000000000003}]},{"osd":11,"last update":"Fri Feb 9 17:37:05 2024","interfaces":[{"interface":"back","average":{"1min":1.798,"5min":1.522,"15min":1.7849999999999999},"min":{"1min":0.39600000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":6.4790000000000001,"5min":6.7709999999999999,"15min":13.186},"last":2.2610000000000001},{"interface":"front","average":{"1min":2.8490000000000002,"5min":1.863,"15min":1.8740000000000001},"min":{"1min":0.45900000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":10.856999999999999,"5min":10.856999999999999,"15min":10.856999999999999},"last":0.497}]}]},{"osd":3,"up_from":27,"seq":115964117215,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2802472,"kb_used_data":2464480,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":90942680,"statfs":{"total":95995035648,"available":93125304320,"internally_reserved":0,"allocated":2523627520,"data_stored":2523295143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":346094023},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4],"upper_bound":8},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:37:00 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":1.3999999999999999,"15min":1.6459999999999999},"min":{"1min":0.22600000000000001,"5min":0.224,"15min":0.224},"max":{"1min":2.4380000000000002,"5min":7.5110000000000001,"15min":7.5110000000000001},"last":9.423},{"interface":"front","average":{"1min":1.522,"5min":1.5580000000000001,"15min":1.736},"min":{"1min":0.24399999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":6.6269999999999998,"5min":15.167999999999999,"15min":15.167999999999999},"last":9.3849999999999998}]},{"osd":1,"last update":"Fri Feb 9 17:33:41 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":1.1399999999999999,"15min":1.3939999999999999},"min":{"1min":0.26100000000000001,"5min":0.23400000000000001,"15min":0.23300000000000001},"max":{"1min":3.8759999999999999,"5min":17.384,"15min":19.638999999999999},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":1.0249999999999999,"15min":1.115},"min":{"1min":0.27500000000000002,"5min":0.22500000000000001,"15min":0.20399999999999999},"max":{"1min":3.8980000000000001,"5min":12.664999999999999,"15min":14.661},"last":0.314}]},{"osd":2,"last update":"Fri Feb 9 17:36:50 2024","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.2849999999999999,"15min":1.4079999999999999},"min":{"1min":0.33500000000000002,"5min":0.22600000000000001,"15min":0.224},"max":{"1min":4.2670000000000003,"5min":14.888999999999999,"15min":14.914999999999999},"last":9.3529999999999998},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.583,"15min":1.4299999999999999},"min":{"1min":0.249,"5min":0.22500000000000001,"15min":0.218},"max":{"1min":6.9139999999999997,"5min":17.613,"15min":17.613},"last":6.5830000000000002}]},{"osd":4,"last update":"Fri Feb 9 17:37:05 2024","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.9259999999999999,"15min":1.7050000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.254},"max":{"1min":5.5860000000000003,"5min":34.898000000000003,"15min":34.898000000000003},"last":9.2330000000000005},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.756,"15min":1.5449999999999999},"min":{"1min":0.33800000000000002,"5min":0.32300000000000001,"15min":0.27900000000000003},"max":{"1min":4.1760000000000002,"5min":14.061999999999999,"15min":14.061999999999999},"last":9.3059999999999992}]},{"osd":5,"last update":"Fri Feb 9 17:36:35 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":1.8100000000000001,"15min":1.3180000000000001},"min":{"1min":0.36799999999999999,"5min":0.27800000000000002,"15min":0.27300000000000002},"max":{"1min":1.77,"5min":17.651,"15min":17.651},"last":1.9770000000000001},{"interface":"front","average":{"1min":0.86199999999999999,"5min":1.853,"15min":1.3580000000000001},"min":{"1min":0.38900000000000001,"5min":0.34100000000000003,"15min":0.33300000000000002},"max":{"1min":1.8080000000000001,"5min":17.675999999999998,"15min":17.675999999999998},"last":2.097}]},{"osd":6,"last update":"Fri Feb 9 17:37:05 2024","interfaces":[{"interface":"back","average":{"1min":2.306,"5min":2.0110000000000001,"15min":1.5840000000000001},"min":{"1min":0.33200000000000002,"5min":0.32900000000000001,"15min":0.32200000000000001},"max":{"1min":10.487,"5min":20.262,"15min":20.262},"last":2.0230000000000001},{"interface":"front","average":{"1min":1.5980000000000001,"5min":1.6859999999999999,"15min":1.2769999999999999},"min":{"1min":0.372,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":5.157,"5min":7.6959999999999997,"15min":7.9009999999999998},"last":9.1899999999999995}]},{"osd":7,"last update":"Fri Feb 9 17:36:35 2024","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":2.4209999999999998,"15min":1.9179999999999999},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.27200000000000002},"max":{"1min":4.9820000000000002,"5min":21.018999999999998,"15min":21.018999999999998},"last":9.3290000000000006},{"interface":"front","average":{"1min":1.236,"5min":1.9950000000000001,"15min":1.6259999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.37},"max":{"1min":4.7329999999999997,"5min":14.186,"15min":19.550999999999998},"last":2.0539999999999998}]},{"osd":8,"last update":"Fri Feb 9 17:37:05 2024","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.3069999999999999,"15min":1.3149999999999999},"min":{"1min":0.38500000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":4.3739999999999997,"5min":7.7569999999999997,"15min":7.7569999999999997},"last":3.012},{"interface":"front","average":{"1min":1.448,"5min":1.1919999999999999,"15min":0.876},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":4.0599999999999996,"5min":14.129,"15min":14.129},"last":9.2829999999999995}]},{"osd":9,"last update":"Fri Feb 9 17:37:19 2024","interfaces":[{"interface":"back","average":{"1min":1.879,"5min":1.629,"15min":1.3320000000000001},"min":{"1min":0.42899999999999999,"5min":0.371,"15min":0.35499999999999998},"max":{"1min":6.5949999999999998,"5min":7.7370000000000001,"15min":7.7370000000000001},"last":2.9670000000000001},{"interface":"front","average":{"1min":1.9970000000000001,"5min":1.502,"15min":1.5960000000000001},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":12.815,"5min":12.815,"15min":13.717000000000001},"last":2.145}]},{"osd":10,"last update":"Fri Feb 9 17:36:50 2024","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.355,"15min":1.431},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":2.1429999999999998,"5min":7.7889999999999997,"15min":19.611000000000001},"last":3.0409999999999999},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.401,"15min":1.3200000000000001},"min":{"1min":0.41999999999999998,"5min":0.38,"15min":0.38},"max":{"1min":6.8179999999999996,"5min":12.638,"15min":12.638},"last":2.125}]},{"osd":11,"last update":"Fri Feb 9 17:37:05 2024","interfaces":[{"interface":"back","average":{"1min":1.6539999999999999,"5min":1.6930000000000001,"15min":1.3899999999999999},"min":{"1min":0.46300000000000002,"5min":0.42499999999999999,"15min":0.38800000000000001},"max":{"1min":6.8449999999999998,"5min":12.69,"15min":12.69},"last":2.077},{"interface":"front","average":{"1min":1.6899999999999999,"5min":1.4159999999999999,"15min":1.625},"min":{"1min":0.45400000000000001,"5min":0.42499999999999999,"15min":0.36899999999999999},"max":{"1min":4.7729999999999997,"5min":14.157999999999999,"15min":14.157999999999999},"last":9.2569999999999997}]}]},{"osd":4,"up_from":32,"seq":137438953689,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2259812,"kb_used_data":1925980,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":91485340,"statfs":{"total":95995035648,"available":93680988160,"internally_reserved":0,"allocated":1972203520,"data_stored":1971870623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:37:02 2024","interfaces":[{"interface":"back","average":{"1min":1.954,"5min":2.6120000000000001,"15min":1.4870000000000001},"min":{"1min":0.35999999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":7.9669999999999996,"5min":50.356999999999999,"15min":50.356999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":2.0800000000000001,"5min":2.3260000000000001,"15min":1.401},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":8.0440000000000005,"5min":50.569000000000003,"15min":50.569000000000003},"last":1.831}]},{"osd":1,"last update":"Fri Feb 9 17:37:13 2024","interfaces":[{"interface":"back","average":{"1min":3.3140000000000001,"5min":2.6619999999999999,"15min":1.853},"min":{"1min":0.27300000000000002,"5min":0.26100000000000001,"15min":0.246},"max":{"1min":28.143000000000001,"5min":50.473999999999997,"15min":50.473999999999997},"last":0.67300000000000004},{"interface":"front","average":{"1min":3.3559999999999999,"5min":2.8109999999999999,"15min":1.8939999999999999},"min":{"1min":0.32200000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":28.103000000000002,"5min":50.499000000000002,"15min":50.499000000000002},"last":0.70199999999999996}]},{"osd":2,"last update":"Fri Feb 9 17:28:51 2024","interfaces":[{"interface":"back","average":{"1min":2.536,"5min":1.546,"15min":0.91400000000000003},"min":{"1min":0.29999999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":8.8819999999999997,"5min":8.8819999999999997,"15min":8.8819999999999997},"last":0.622},{"interface":"front","average":{"1min":2.214,"5min":1.7330000000000001,"15min":0.97799999999999998},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":9.5039999999999996,"5min":9.5039999999999996,"15min":9.5039999999999996},"last":0.53800000000000003}]},{"osd":3,"last update":"Fri Feb 9 17:37:13 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":2.4769999999999999,"15min":1.8620000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":7.1820000000000004,"5min":50.462000000000003,"15min":50.462000000000003},"last":1.4259999999999999},{"interface":"front","average":{"1min":2.0630000000000002,"5min":2.2879999999999998,"15min":1.7110000000000001},"min":{"1min":0.29699999999999999,"5min":0.255,"15min":0.255},"max":{"1min":5.5869999999999997,"5min":50.529000000000003,"15min":50.529000000000003},"last":3.5209999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":1.915,"15min":1.1639999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.222},"max":{"1min":2.0339999999999998,"5min":9.5630000000000006,"15min":9.5630000000000006},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.79200000000000004,"5min":2.5579999999999998,"15min":1.4550000000000001},"min":{"1min":0.314,"5min":0.314,"15min":0.29199999999999998},"max":{"1min":2.073,"5min":50.509999999999998,"15min":50.509999999999998},"last":0.36899999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:37:13 2024","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.76,"15min":1.2789999999999999},"min":{"1min":0.26600000000000001,"5min":0.26200000000000001,"15min":0.247},"max":{"1min":7.0419999999999998,"5min":11.742000000000001,"15min":18.09},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.999,"5min":2.4129999999999998,"15min":1.4510000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.28000000000000003},"max":{"1min":3.8900000000000001,"5min":50.402999999999999,"15min":50.402999999999999},"last":0.68400000000000005}]},{"osd":7,"last update":"Fri Feb 9 17:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":2.25,"15min":1.4450000000000001},"min":{"1min":0.40699999999999997,"5min":0.33800000000000002,"15min":0.28399999999999997},"max":{"1min":3.266,"5min":14.919,"15min":15.223000000000001},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.99399999999999999,"5min":2.169,"15min":1.4099999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.28100000000000003},"max":{"1min":4.0419999999999998,"5min":9.343,"15min":19.047000000000001},"last":0.56699999999999995}]},{"osd":8,"last update":"Fri Feb 9 17:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":1.869,"15min":1.4630000000000001},"min":{"1min":0.36199999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":2.2000000000000002,"5min":8.4160000000000004,"15min":8.4160000000000004},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.80800000000000005,"5min":2.3839999999999999,"15min":1.5249999999999999},"min":{"1min":0.40999999999999998,"5min":0.34699999999999998,"15min":0.308},"max":{"1min":1.4830000000000001,"5min":50.594000000000001,"15min":50.594000000000001},"last":0.441}]},{"osd":9,"last update":"Fri Feb 9 17:37:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":2.4969999999999999,"15min":1.5589999999999999},"min":{"1min":0.44400000000000001,"5min":0.39200000000000002,"15min":0.34599999999999997},"max":{"1min":5.157,"5min":50.442999999999998,"15min":50.442999999999998},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.86299999999999999,"5min":1.728,"15min":1.3100000000000001},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":3.2669999999999999,"5min":9.9770000000000003,"15min":10.555},"last":0.83299999999999996}]},{"osd":10,"last update":"Fri Feb 9 17:36:53 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":2.6680000000000001,"15min":1.649},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.378},"max":{"1min":3.137,"5min":51.223999999999997,"15min":51.223999999999997},"last":1.117},{"interface":"front","average":{"1min":1.294,"5min":2.9430000000000001,"15min":1.7130000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.38100000000000001},"max":{"1min":5.117,"5min":50.552,"15min":50.552},"last":0.52100000000000002}]},{"osd":11,"last update":"Fri Feb 9 17:37:18 2024","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":2.8490000000000002,"15min":1.881},"min":{"1min":0.495,"5min":0.443,"15min":0.40500000000000003},"max":{"1min":3.6619999999999999,"5min":50.484999999999999,"15min":50.484999999999999},"last":0.55400000000000005},{"interface":"front","average":{"1min":1.022,"5min":2.0939999999999999,"15min":1.5409999999999999},"min":{"1min":0.435,"5min":0.42499999999999999,"15min":0.34899999999999998},"max":{"1min":3.4300000000000002,"5min":18.574999999999999,"15min":18.574999999999999},"last":1.129}]}]},{"osd":5,"up_from":40,"seq":171798692051,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1148648,"kb_used_data":964768,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":92596504,"statfs":{"total":95995035648,"available":94818820096,"internally_reserved":0,"allocated":987922432,"data_stored":987590055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:32:25 2024","interfaces":[{"interface":"back","average":{"1min":1.8540000000000001,"5min":1.653,"15min":1.0900000000000001},"min":{"1min":0.318,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":7.7160000000000002,"5min":8.7690000000000001,"15min":11.923},"last":3.8450000000000002},{"interface":"front","average":{"1min":2.4089999999999998,"5min":1.8260000000000001,"15min":1.032},"min":{"1min":0.34499999999999997,"5min":0.24299999999999999,"15min":0.224},"max":{"1min":14.082000000000001,"5min":14.082000000000001,"15min":14.082000000000001},"last":0.79400000000000004}]},{"osd":1,"last update":"Fri Feb 9 17:36:32 2024","interfaces":[{"interface":"back","average":{"1min":1.502,"5min":2.1920000000000002,"15min":1.758},"min":{"1min":0.308,"5min":0.23899999999999999,"15min":0.23200000000000001},"max":{"1min":9.2690000000000001,"5min":14.032999999999999,"15min":29.715},"last":0.371},{"interface":"front","average":{"1min":1.359,"5min":1.7969999999999999,"15min":1.597},"min":{"1min":0.26000000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":8.6920000000000002,"5min":11.148999999999999,"15min":29.969999999999999},"last":0.42799999999999999}]},{"osd":2,"last update":"Fri Feb 9 17:36:32 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":1.496,"15min":1.425},"min":{"1min":0.28799999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.7729999999999999,"5min":10.5,"15min":10.5},"last":3.823},{"interface":"front","average":{"1min":0.71699999999999997,"5min":1.907,"15min":1.5509999999999999},"min":{"1min":0.371,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":2.472,"5min":14.122,"15min":14.122},"last":3.8050000000000002}]},{"osd":3,"last update":"Fri Feb 9 17:36:32 2024","interfaces":[{"interface":"back","average":{"1min":4.1070000000000002,"5min":2.7250000000000001,"15min":1.8400000000000001},"min":{"1min":0.33900000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":64.150999999999996,"5min":64.150999999999996,"15min":64.150999999999996},"last":3.8540000000000001},{"interface":"front","average":{"1min":4.2119999999999997,"5min":2.8839999999999999,"15min":1.8080000000000001},"min":{"1min":0.41399999999999998,"5min":0.30499999999999999,"15min":0.28000000000000003},"max":{"1min":64.176000000000002,"5min":64.176000000000002,"15min":64.176000000000002},"last":0.57699999999999996}]},{"osd":4,"last update":"Fri Feb 9 17:36:41 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":1.863,"15min":1.139},"min":{"1min":0.28699999999999998,"5min":0.26200000000000001,"15min":0.23400000000000001},"max":{"1min":2.27,"5min":11.036,"15min":11.036},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.81299999999999994,"5min":1.7689999999999999,"15min":1.202},"min":{"1min":0.28599999999999998,"5min":0.28499999999999998,"15min":0.27900000000000003},"max":{"1min":3.7559999999999998,"5min":13.59,"15min":13.59},"last":0.497}]},{"osd":6,"last update":"Fri Feb 9 17:37:09 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":1.6919999999999999,"15min":1.258},"min":{"1min":0.30499999999999999,"5min":0.253,"15min":0.253},"max":{"1min":2.964,"5min":15.375999999999999,"15min":15.375999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":1.093,"5min":1.9690000000000001,"15min":1.3280000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.27200000000000002},"max":{"1min":5.4690000000000003,"5min":14.250999999999999,"15min":14.250999999999999},"last":0.47599999999999998}]},{"osd":7,"last update":"Fri Feb 9 17:36:32 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":1.778,"15min":1.2370000000000001},"min":{"1min":0.29399999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.726,"5min":9.7140000000000004,"15min":18.626999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.93700000000000006,"5min":2.1110000000000002,"15min":1.452},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.28799999999999998},"max":{"1min":3.0030000000000001,"5min":19.044,"15min":19.044},"last":0.53800000000000003}]},{"osd":8,"last update":"Fri Feb 9 17:37:09 2024","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.4530000000000001,"15min":1.6870000000000001},"min":{"1min":0.42399999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":5.2960000000000003,"5min":9.3710000000000004,"15min":9.3710000000000004},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.81000000000000005,"5min":1.7170000000000001,"15min":2.1219999999999999},"min":{"1min":0.53500000000000003,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":2.5640000000000001,"5min":9.6240000000000006,"15min":9.6240000000000006},"last":0.65800000000000003}]},{"osd":9,"last update":"Fri Feb 9 17:37:14 2024","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.6240000000000001,"15min":1.278},"min":{"1min":0.41799999999999998,"5min":0.378,"15min":0.35999999999999999},"max":{"1min":6.5780000000000003,"5min":14.98,"15min":14.98},"last":0.64600000000000002},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.716,"15min":1.3},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.312},"max":{"1min":7.7779999999999996,"5min":9.4239999999999995,"15min":9.4239999999999995},"last":0.44600000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:36:41 2024","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":1.9930000000000001,"15min":1.4299999999999999},"min":{"1min":0.48599999999999999,"5min":0.47499999999999998,"15min":0.33800000000000002},"max":{"1min":2.581,"5min":14.218,"15min":14.218},"last":0.69899999999999995},{"interface":"front","average":{"1min":1.01,"5min":2.016,"15min":1.48},"min":{"1min":0.497,"5min":0.45000000000000001,"15min":0.36099999999999999},"max":{"1min":3.3340000000000001,"5min":14.186999999999999,"15min":14.186999999999999},"last":0.68999999999999995}]},{"osd":11,"last update":"Fri Feb 9 17:37:14 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":1.881,"15min":1.486},"min":{"1min":0.53600000000000003,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":4.0940000000000003,"5min":9.4239999999999995,"15min":9.4239999999999995},"last":0.60699999999999998},{"interface":"front","average":{"1min":1.377,"5min":2.3090000000000002,"15min":1.48},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.38900000000000001},"max":{"1min":7.734,"5min":14.316000000000001,"15min":14.316000000000001},"last":0.51800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1498152960,"data_stored":1498152960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4389015552,"data_stored":4389011480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":972750848,"data_stored":972750848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2510749696,"data_stored":2510749696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1964314624,"data_stored":1964310552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":985071616,"data_stored":985071616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3026321408,"data_stored":3026321408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1506213888,"data_stored":1506213888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1506742272,"data_stored":1506738200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1983053824,"data_stored":1983053824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":3414163456,"data_stored":3414163456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-09T17:37:32.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:32 smithi179 ceph-mon[43975]: from='client.25297 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:37:33.002 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:37:33.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:33.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:37:33.116 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:37:33.147 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:33.171 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:37:33.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:33.174 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:37:33.275 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:37:33.320 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:33.346 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:37:33.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:33.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:37:33.454 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:37:33.485 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:33.514 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:37:33.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:33.517 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:37:33.618 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:37:33.667 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:33.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:33 smithi162 ceph-mon[44304]: pgmap v983: 37 pgs: 37 active+clean; 7.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 858 op/s 2024-02-09T17:37:33.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:33 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:37:33.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:33 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:37:33.828 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:33 smithi099 ceph-mon[39420]: pgmap v983: 37 pgs: 37 active+clean; 7.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 858 op/s 2024-02-09T17:37:33.828 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:33 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:37:33.828 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:33 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:37:33.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:33 smithi179 ceph-mon[43975]: pgmap v983: 37 pgs: 37 active+clean; 7.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 858 op/s 2024-02-09T17:37:33.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:33 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:37:33.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:33 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:37:33.968 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:37:33.968 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:37:34.255 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:37:34.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:34.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:37:34.274 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:37:34.360 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:37:34.395 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:34.416 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:37:34.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:34.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:37:34.524 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:34 smithi099 ceph-mon[39420]: osdmap e360: 12 total, 12 up, 12 in 2024-02-09T17:37:34.524 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:37:34.565 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:34.593 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:37:34.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:34.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:37:34.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:34 smithi162 ceph-mon[44304]: osdmap e360: 12 total, 12 up, 12 in 2024-02-09T17:37:34.704 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:37:34.736 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:34.760 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:37:34.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:34.766 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:37:34.867 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:37:34.900 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-09T17:37:34.914 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:34.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:34 smithi179 ceph-mon[43975]: osdmap e360: 12 total, 12 up, 12 in 2024-02-09T17:37:34.988 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:37:34.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:34.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:37:35.096 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:37:35.182 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:35.229 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:37:35.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:35.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:37:35.333 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:37:35.375 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:35.399 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:37:35.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:35.406 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:37:35.507 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:37:35.541 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:35.583 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:37:35.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:35.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:37:35.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:35 smithi179 ceph-mon[43975]: pgmap v985: 37 pgs: 37 active+clean; 7.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 894 op/s 2024-02-09T17:37:35.691 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:37:35.744 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:35.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:35 smithi099 ceph-mon[39420]: pgmap v985: 37 pgs: 37 active+clean; 7.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 894 op/s 2024-02-09T17:37:35.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:35 smithi162 ceph-mon[44304]: pgmap v985: 37 pgs: 37 active+clean; 7.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 894 op/s 2024-02-09T17:37:36.215 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:37:36.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:36.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:37:36.303 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:37:36.322 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:37:36.361 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:36.385 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:37:36.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:36.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:37:36.463 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:37:36.493 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:37:36.531 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:36.558 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:37:36.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:36.561 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:37:36.662 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:37:36.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:36 smithi162 ceph-mon[44304]: 1.a scrub starts 2024-02-09T17:37:36.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:36 smithi162 ceph-mon[44304]: 1.a scrub ok 2024-02-09T17:37:36.695 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:36.718 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:37:36.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:36.723 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:37:36.824 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:37:36.871 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:36 smithi099 ceph-mon[39420]: 1.a scrub starts 2024-02-09T17:37:36.871 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:36 smithi099 ceph-mon[39420]: 1.a scrub ok 2024-02-09T17:37:36.875 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:36.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:36 smithi179 ceph-mon[43975]: 1.a scrub starts 2024-02-09T17:37:36.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:36 smithi179 ceph-mon[43975]: 1.a scrub ok 2024-02-09T17:37:36.952 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:37:36.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:36.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:37:37.064 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:37:37.093 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:37.113 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:37:37.119 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:37.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:37:37.221 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:37:37.282 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:37.597 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is set 2024-02-09T17:37:37.891 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] 2024-02-09T17:37:37.891 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:37:37.892 INFO:tasks.thrashosds.thrasher:inject_pause on osd.10 2024-02-09T17:37:37.892 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-09T17:37:37.892 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-09T17:37:37.892 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.10 config set filestore_inject_stall 3 2024-02-09T17:37:37.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:37 smithi179 ceph-mon[43975]: pgmap v986: 37 pgs: 37 active+clean; 7.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 904 op/s 2024-02-09T17:37:37.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:37 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4221555815' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:37:37.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:37 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:37:37.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:37 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:37:37.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:37 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:37:37.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:37 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:37:37.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:37 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:37:37.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:37 smithi162 ceph-mon[44304]: pgmap v986: 37 pgs: 37 active+clean; 7.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 904 op/s 2024-02-09T17:37:37.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:37 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4221555815' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:37:37.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:37 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:37:37.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:37 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:37:37.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:37 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:37:37.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:37 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:37:37.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:37 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:37:37.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:37 smithi099 ceph-mon[39420]: pgmap v986: 37 pgs: 37 active+clean; 7.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 904 op/s 2024-02-09T17:37:37.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:37 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4221555815' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:37:37.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:37 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:37:37.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:37 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:37:37.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:37 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:37:37.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:37 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:37:37.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:37 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:37:38.273 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:37:38.838 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:38 smithi179 ceph-mon[43975]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:37:38.838 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:38 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:37:38.838 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:38 smithi179 ceph-mon[43975]: osdmap e361: 12 total, 12 up, 12 in 2024-02-09T17:37:38.838 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:38 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:37:38.838 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:38 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:37:38.839 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:38 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:37:38.839 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:38 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:37:38.839 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:38 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:37:38.839 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:38 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:37:38.839 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:38 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:37:38.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:38 smithi162 ceph-mon[44304]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:37:38.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:38 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:37:38.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:38 smithi162 ceph-mon[44304]: osdmap e361: 12 total, 12 up, 12 in 2024-02-09T17:37:38.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:38 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:37:38.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:38 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:37:38.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:38 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:37:38.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:38 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:37:38.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:38 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:37:38.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:38 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:37:38.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:38 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:37:38.961 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:37:38.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:38.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:37:38.981 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:38 smithi099 ceph-mon[39420]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:37:38.981 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:38 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:37:38.981 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:38 smithi099 ceph-mon[39420]: osdmap e361: 12 total, 12 up, 12 in 2024-02-09T17:37:38.981 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:38 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:37:38.981 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:38 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:37:38.982 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:38 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:37:38.982 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:38 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:37:38.982 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:38 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:37:38.982 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:38 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:37:38.982 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:38 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:37:39.074 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:37:39.103 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:39.126 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:37:39.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:39.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:37:39.232 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:37:39.283 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:39.650 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:37:39.659 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:39.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:37:39.761 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:37:39.809 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:39.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:39 smithi179 ceph-mon[43975]: pgmap v988: 37 pgs: 37 active+clean; 7.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 782 op/s 2024-02-09T17:37:39.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:39 smithi162 ceph-mon[44304]: pgmap v988: 37 pgs: 37 active+clean; 7.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 782 op/s 2024-02-09T17:37:40.001 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:39 smithi099 ceph-mon[39420]: pgmap v988: 37 pgs: 37 active+clean; 7.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 782 op/s 2024-02-09T17:37:40.082 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:37:40.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:40.086 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:37:40.156 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:37:40.187 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:37:40.218 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:40.423 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:37:40.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:40.426 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:37:40.450 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:37:40.450 INFO:teuthology.orchestra.run.smithi179.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-02-09T17:37:40.450 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:37:40.528 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:37:40.578 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:41.062 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:37:41.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:41.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:37:41.172 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:37:41.228 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:41.641 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-09T17:37:41.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:41 smithi099 ceph-mon[39420]: pgmap v989: 37 pgs: 37 active+clean; 6.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 941 op/s 2024-02-09T17:37:41.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:41 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:37:41.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:41 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:37:42.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:41 smithi179 ceph-mon[43975]: pgmap v989: 37 pgs: 37 active+clean; 6.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 941 op/s 2024-02-09T17:37:42.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:41 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:37:42.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:41 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:37:42.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:41 smithi162 ceph-mon[44304]: pgmap v989: 37 pgs: 37 active+clean; 6.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 941 op/s 2024-02-09T17:37:42.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:41 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:37:42.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:41 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:37:42.505 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:37:42.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:42.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:37:42.614 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:37:42.645 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:42.665 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:37:42.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:42.670 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:37:42.675 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:37:42.675 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:37:42.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:42 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:37:42.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:42 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:37:42.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:42 smithi099 ceph-mon[39420]: osdmap e362: 12 total, 12 up, 12 in 2024-02-09T17:37:42.770 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:37:42.834 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:42.880 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:37:42.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:42.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:37:42.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:42 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:37:42.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:42 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:37:42.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:42 smithi179 ceph-mon[43975]: osdmap e362: 12 total, 12 up, 12 in 2024-02-09T17:37:42.992 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:37:43.057 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:43.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:42 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:37:43.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:42 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:37:43.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:42 smithi162 ceph-mon[44304]: osdmap e362: 12 total, 12 up, 12 in 2024-02-09T17:37:43.725 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:37:43.772 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:37:43.829 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:37:43.864 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:37:43.900 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:37:43.938 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:37:43.977 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:37:44.019 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:37:44.056 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:37:44.084 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:37:44.107 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:37:44.131 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:37:44.133 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:37:44.133 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:37:44.133 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:37:44.133 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:37:44.133 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:37:44.133 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:37:44.133 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:37:44.133 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:37:44.133 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:37:44.134 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:37:44.134 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:37:44.134 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:37:44.151 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:43 smithi099 ceph-mon[39420]: pgmap v991: 37 pgs: 37 active+clean; 6.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s 2024-02-09T17:37:44.151 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:43 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1787153918' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:37:44.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:43 smithi179 ceph-mon[43975]: pgmap v991: 37 pgs: 37 active+clean; 6.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s 2024-02-09T17:37:44.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:43 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1787153918' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:37:44.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:43 smithi162 ceph-mon[44304]: pgmap v991: 37 pgs: 37 active+clean; 6.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s 2024-02-09T17:37:44.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:43 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1787153918' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:37:44.253 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:37:44.258 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:37:44.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:44.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:37:44.275 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-09T17:37:44.364 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:37:44.396 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:44.422 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:37:44.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:44.425 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:37:44.526 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:37:44.554 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:44.577 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:37:44.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:44.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:37:44.681 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:37:44.721 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:44.747 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:37:44.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:44.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:37:44.851 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:37:44.896 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:44.922 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:37:44.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:44.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:37:45.028 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:37:45.065 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:45.097 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:37:45.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:45.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:37:45.205 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:37:45.241 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:45.270 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:37:45.272 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:45.273 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:37:45.374 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:37:45.457 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:45.593 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:37:45.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:45.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:37:45.701 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:37:45.771 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:46.160 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:45 smithi099 ceph-mon[39420]: pgmap v992: 37 pgs: 37 active+clean; 6.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 623 op/s 2024-02-09T17:37:46.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:45 smithi179 ceph-mon[43975]: pgmap v992: 37 pgs: 37 active+clean; 6.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 623 op/s 2024-02-09T17:37:46.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:45 smithi162 ceph-mon[44304]: pgmap v992: 37 pgs: 37 active+clean; 6.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 623 op/s 2024-02-09T17:37:46.214 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] 2024-02-09T17:37:46.214 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:37:46.215 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-02-09T17:37:46.215 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph pg dump --format=json 2024-02-09T17:37:46.510 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:37:46.552 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:37:46.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:46.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:37:46.662 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:37:46.701 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:46.742 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:37:46.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:46.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:37:46.850 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:37:46.890 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:46.916 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:37:46.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:46.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:37:47.023 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:37:47.026 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:46 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:37:47.026 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:46 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:37:47.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:46 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:37:47.054 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:46 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:37:47.056 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:47.095 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:37:47.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:47.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:37:47.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:46 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:37:47.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:46 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:37:47.204 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:37:47.247 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:47.270 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:37:47.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:47.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:37:47.374 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:37:47.408 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:47.434 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:37:47.441 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:47.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:37:47.542 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:37:47.594 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:47.626 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:37:47.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:47 smithi099 ceph-mon[39420]: pgmap v993: 37 pgs: 37 active+clean; 6.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 925 op/s 2024-02-09T17:37:47.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:47 smithi099 ceph-mon[39420]: osdmap e363: 12 total, 12 up, 12 in 2024-02-09T17:37:48.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:47 smithi179 ceph-mon[43975]: pgmap v993: 37 pgs: 37 active+clean; 6.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 925 op/s 2024-02-09T17:37:48.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:47 smithi179 ceph-mon[43975]: osdmap e363: 12 total, 12 up, 12 in 2024-02-09T17:37:48.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:47 smithi162 ceph-mon[44304]: pgmap v993: 37 pgs: 37 active+clean; 6.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 925 op/s 2024-02-09T17:37:48.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:47 smithi162 ceph-mon[44304]: osdmap e363: 12 total, 12 up, 12 in 2024-02-09T17:37:48.204 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:37:48.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:48.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:37:48.312 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:37:48.358 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:48.382 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:37:48.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:48.385 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:37:48.486 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:37:48.520 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:48.542 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:37:48.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:48.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:37:48.649 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:37:48.686 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:48.726 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:37:48.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:48.741 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:37:48.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:48 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4124257674' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:37:48.801 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:48 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:37:48.811 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is set 2024-02-09T17:37:48.841 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:37:48.844 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:48 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4124257674' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:37:48.844 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:48 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:37:48.916 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:49.041 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:37:49.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:49.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:37:49.150 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:37:49.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:48 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4124257674' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:37:49.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:48 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:37:49.224 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:49.313 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:37:49.317 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:49.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:37:49.419 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:37:49.501 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:49.874 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:37:49.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:49.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:37:49.982 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:37:50.023 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:50.044 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:37:50.051 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:50.051 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:37:50.147 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:49 smithi099 ceph-mon[39420]: pgmap v995: 37 pgs: 37 active+clean; 6.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 749 op/s 2024-02-09T17:37:50.148 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:49 smithi099 ceph-mon[39420]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:37:50.148 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:49 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:37:50.148 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:49 smithi099 ceph-mon[39420]: osdmap e364: 12 total, 12 up, 12 in 2024-02-09T17:37:50.152 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:37:50.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:49 smithi179 ceph-mon[43975]: pgmap v995: 37 pgs: 37 active+clean; 6.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 749 op/s 2024-02-09T17:37:50.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:49 smithi179 ceph-mon[43975]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:37:50.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:49 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:37:50.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:49 smithi179 ceph-mon[43975]: osdmap e364: 12 total, 12 up, 12 in 2024-02-09T17:37:50.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:49 smithi162 ceph-mon[44304]: pgmap v995: 37 pgs: 37 active+clean; 6.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 749 op/s 2024-02-09T17:37:50.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:49 smithi162 ceph-mon[44304]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:37:50.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:49 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:37:50.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:49 smithi162 ceph-mon[44304]: osdmap e364: 12 total, 12 up, 12 in 2024-02-09T17:37:50.185 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:50.202 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:37:50.211 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:37:50.215 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:50.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:37:50.222 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:37:50.316 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:37:50.360 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:50.722 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:37:50.722 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [ 2024-02-09T17:37:50.722 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:50.722 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25201.0:173704 4.c e365/322 4:31f4a287:::benchmark_data_smithi179_103809_object36238:head v 365'11004, mlcod=365'11002)", 2024-02-09T17:37:50.722 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:37:50.703547+0000", 2024-02-09T17:37:50.722 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.017331488999999999, 2024-02-09T17:37:50.722 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.017361578999999999, 2024-02-09T17:37:50.722 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:37:50.722 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T17:37:50.723 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:37:50.723 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:50.723 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:37:50.723 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.703547+0000", 2024-02-09T17:37:50.723 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:37:50.723 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:50.723 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:50.723 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:37:50.723 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.703547+0000", 2024-02-09T17:37:50.723 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:37:50.723 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:50.723 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:50.723 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:37:50.724 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.703548+0000", 2024-02-09T17:37:50.724 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.47e-07 2024-02-09T17:37:50.724 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:50.724 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:50.724 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:37:50.724 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.703556+0000", 2024-02-09T17:37:50.724 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.0479999999999999e-06 2024-02-09T17:37:50.724 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:50.724 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:50.724 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:37:50.724 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.703572+0000", 2024-02-09T17:37:50.724 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5985e-05 2024-02-09T17:37:50.724 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:50.724 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:50.725 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:37:50.725 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.703582+0000", 2024-02-09T17:37:50.725 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0054000000000001e-05 2024-02-09T17:37:50.725 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:50.725 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:50.725 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:37:50.725 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.707640+0000", 2024-02-09T17:37:50.725 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0040580979999999996 2024-02-09T17:37:50.725 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:50.725 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:50.725 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:37:50.725 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.707661+0000", 2024-02-09T17:37:50.725 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0737000000000002e-05 2024-02-09T17:37:50.725 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:37:50.725 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:37:50.726 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:37:50.726 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:50.726 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:50.726 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25201.0:173703 4.f e365/322 4:fc5a7f5c:::benchmark_data_smithi179_103809_object36237:head v 365'10773, mlcod=365'10771)", 2024-02-09T17:37:50.726 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:37:50.703605+0000", 2024-02-09T17:37:50.726 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.017274063999999999, 2024-02-09T17:37:50.726 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.01736304, 2024-02-09T17:37:50.726 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:37:50.726 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T17:37:50.726 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:37:50.726 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:50.726 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:37:50.726 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.703605+0000", 2024-02-09T17:37:50.726 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:37:50.727 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:50.727 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:50.727 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:37:50.727 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.703605+0000", 2024-02-09T17:37:50.727 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:37:50.727 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:50.727 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:50.727 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:37:50.727 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.703605+0000", 2024-02-09T17:37:50.727 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9200000000000002e-07 2024-02-09T17:37:50.727 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:50.727 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:50.727 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:37:50.727 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.703609+0000", 2024-02-09T17:37:50.728 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.3810000000000001e-06 2024-02-09T17:37:50.728 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:50.728 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:50.728 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:37:50.728 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.703616+0000", 2024-02-09T17:37:50.728 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.5289999999999997e-06 2024-02-09T17:37:50.728 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:50.728 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:50.728 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:37:50.728 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.703620+0000", 2024-02-09T17:37:50.728 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.4850000000000003e-06 2024-02-09T17:37:50.728 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:50.728 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:50.728 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:37:50.729 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.703666+0000", 2024-02-09T17:37:50.729 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.5852000000000001e-05 2024-02-09T17:37:50.729 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:50.729 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:50.729 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:37:50.729 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.703687+0000", 2024-02-09T17:37:50.729 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0981000000000001e-05 2024-02-09T17:37:50.729 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:37:50.729 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:37:50.729 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:37:50.730 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:50.730 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:50.730 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:173707 4.0 4:00ad2ee0:::benchmark_data_smithi179_103809_object36241:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-02-09T17:37:50.730 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:37:50.704632+0000", 2024-02-09T17:37:50.730 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.016246242000000001, 2024-02-09T17:37:50.730 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.016391840000000001, 2024-02-09T17:37:50.730 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:37:50.730 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "waiting for sub ops", 2024-02-09T17:37:50.730 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:37:50.730 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:37:50.730 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:37:50.730 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 173707 2024-02-09T17:37:50.730 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:50.730 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:37:50.731 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:50.731 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:37:50.731 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.704632+0000", 2024-02-09T17:37:50.731 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:37:50.731 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:50.731 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:50.731 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:37:50.731 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.704632+0000", 2024-02-09T17:37:50.731 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:37:50.732 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:50.732 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:50.732 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:37:50.732 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.704634+0000", 2024-02-09T17:37:50.732 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.6440000000000001e-06 2024-02-09T17:37:50.732 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:50.732 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:50.732 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:37:50.733 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.704640+0000", 2024-02-09T17:37:50.733 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.8200000000000002e-06 2024-02-09T17:37:50.733 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:50.733 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:50.733 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:37:50.733 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.704653+0000", 2024-02-09T17:37:50.733 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.271e-05 2024-02-09T17:37:50.733 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:50.733 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:50.733 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:37:50.733 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.704661+0000", 2024-02-09T17:37:50.733 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.1689999999999992e-06 2024-02-09T17:37:50.733 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:50.733 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:50.734 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:37:50.734 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.704719+0000", 2024-02-09T17:37:50.734 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7936000000000002e-05 2024-02-09T17:37:50.734 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:50.734 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:50.734 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:37:50.734 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.704833+0000", 2024-02-09T17:37:50.734 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00011472399999999999 2024-02-09T17:37:50.734 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:50.734 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:50.734 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,7", 2024-02-09T17:37:50.734 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.704939+0000", 2024-02-09T17:37:50.734 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000105756 2024-02-09T17:37:50.734 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:50.734 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:50.734 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:37:50.734 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.715617+0000", 2024-02-09T17:37:50.735 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.010678244 2024-02-09T17:37:50.735 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:50.735 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:50.735 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:37:50.735 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.718803+0000", 2024-02-09T17:37:50.735 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0031856330000000002 2024-02-09T17:37:50.735 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:37:50.735 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:37:50.735 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:37:50.735 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:50.735 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:50.735 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25201.0:173705 4.c e365/322 4:3b9b10ea:::benchmark_data_smithi179_103809_object36239:head v 365'11005, mlcod=365'11002)", 2024-02-09T17:37:50.735 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:37:50.705780+0000", 2024-02-09T17:37:50.735 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.015098561, 2024-02-09T17:37:50.735 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.015320956, 2024-02-09T17:37:50.736 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:37:50.736 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T17:37:50.736 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:37:50.736 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:50.736 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:37:50.736 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.705780+0000", 2024-02-09T17:37:50.736 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:37:50.736 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:50.736 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:50.736 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:37:50.736 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.705780+0000", 2024-02-09T17:37:50.736 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:37:50.736 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:50.736 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:50.736 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:37:50.737 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.705781+0000", 2024-02-09T17:37:50.737 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.2500000000000004e-07 2024-02-09T17:37:50.737 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:50.737 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:50.737 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:37:50.737 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.705789+0000", 2024-02-09T17:37:50.737 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.6680000000000001e-06 2024-02-09T17:37:50.737 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:50.737 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:50.737 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:37:50.737 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.705799+0000", 2024-02-09T17:37:50.737 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0764999999999999e-05 2024-02-09T17:37:50.737 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:50.737 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:50.737 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:37:50.737 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.705806+0000", 2024-02-09T17:37:50.738 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.9630000000000001e-06 2024-02-09T17:37:50.738 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:50.738 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:50.738 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:37:50.738 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.713246+0000", 2024-02-09T17:37:50.738 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0074397500000000002 2024-02-09T17:37:50.738 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:50.738 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:50.738 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:37:50.738 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.713265+0000", 2024-02-09T17:37:50.738 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.8340999999999999e-05 2024-02-09T17:37:50.738 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:37:50.738 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:37:50.738 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:37:50.738 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:50.738 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:50.739 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25201.0:173710 4.c e365/322 4:31e022a6:::benchmark_data_smithi179_103809_object36244:head v 365'11006, mlcod=365'11002)", 2024-02-09T17:37:50.739 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:37:50.712226+0000", 2024-02-09T17:37:50.739 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.0086522390000000008, 2024-02-09T17:37:50.739 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0089328440000000005, 2024-02-09T17:37:50.739 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:37:50.739 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T17:37:50.739 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:37:50.739 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:50.739 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:37:50.739 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.712226+0000", 2024-02-09T17:37:50.739 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:37:50.739 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:50.739 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:50.739 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:37:50.739 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.712226+0000", 2024-02-09T17:37:50.740 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:37:50.740 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:50.740 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:50.740 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:37:50.740 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.712227+0000", 2024-02-09T17:37:50.740 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.9099999999999991e-07 2024-02-09T17:37:50.740 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:50.740 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:50.740 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:37:50.740 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.712235+0000", 2024-02-09T17:37:50.740 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.8280000000000003e-06 2024-02-09T17:37:50.740 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:50.740 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:50.740 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:37:50.740 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.712252+0000", 2024-02-09T17:37:50.741 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.662e-05 2024-02-09T17:37:50.741 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:50.741 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:50.741 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:37:50.741 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.712261+0000", 2024-02-09T17:37:50.741 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.8429999999999999e-06 2024-02-09T17:37:50.741 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:50.741 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:50.741 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:37:50.741 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.717640+0000", 2024-02-09T17:37:50.741 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0053791660000000003 2024-02-09T17:37:50.741 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:37:50.741 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:37:50.741 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:37:50.741 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.717655+0000", 2024-02-09T17:37:50.741 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5546999999999999e-05 2024-02-09T17:37:50.742 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:37:50.742 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:37:50.742 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:37:50.742 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:37:50.742 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T17:37:50.742 INFO:teuthology.orchestra.run.smithi099.stdout: "num_ops": 5 2024-02-09T17:37:50.743 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:37:50.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:50.744 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:37:50.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:37:50.845 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:37:50.887 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:50.925 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:37:50.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:50.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:37:50.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:50 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:37:50.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:50 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:37:50.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:50 smithi099 ceph-mon[39420]: osdmap e365: 12 total, 12 up, 12 in 2024-02-09T17:37:51.030 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:37:51.062 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:51.089 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:37:51.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:51.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:37:51.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:50 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:37:51.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:50 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:37:51.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:50 smithi179 ceph-mon[43975]: osdmap e365: 12 total, 12 up, 12 in 2024-02-09T17:37:51.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:50 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:37:51.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:50 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:37:51.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:50 smithi162 ceph-mon[44304]: osdmap e365: 12 total, 12 up, 12 in 2024-02-09T17:37:51.197 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:37:51.233 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:51.271 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:37:51.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:51.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:37:51.379 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:37:51.445 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:51.872 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset noscrub 2024-02-09T17:37:52.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:51 smithi179 ceph-mon[43975]: pgmap v998: 37 pgs: 37 active+clean; 6.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s 2024-02-09T17:37:52.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:51 smithi162 ceph-mon[44304]: pgmap v998: 37 pgs: 37 active+clean; 6.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s 2024-02-09T17:37:52.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:51 smithi099 ceph-mon[39420]: pgmap v998: 37 pgs: 37 active+clean; 6.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s 2024-02-09T17:37:52.822 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:37:52.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:52.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:37:52.928 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:37:52.957 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:52.981 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:37:52.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:52.987 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:37:53.088 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:37:53.126 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:53.161 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:37:53.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:53.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:37:53.270 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:37:53.326 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:54.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:53 smithi179 ceph-mon[43975]: pgmap v999: 37 pgs: 37 active+clean; 6.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 786 op/s 2024-02-09T17:37:54.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:53 smithi162 ceph-mon[44304]: pgmap v999: 37 pgs: 37 active+clean; 6.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 786 op/s 2024-02-09T17:37:54.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:53 smithi099 ceph-mon[39420]: pgmap v999: 37 pgs: 37 active+clean; 6.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 786 op/s 2024-02-09T17:37:54.524 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-09T17:37:55.618 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:37:55.627 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-02-09T17:37:55.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:55 smithi099 ceph-mon[39420]: pgmap v1000: 37 pgs: 37 active+clean; 6.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 633 op/s 2024-02-09T17:37:56.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:55 smithi179 ceph-mon[43975]: pgmap v1000: 37 pgs: 37 active+clean; 6.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 633 op/s 2024-02-09T17:37:56.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:55 smithi162 ceph-mon[44304]: pgmap v1000: 37 pgs: 37 active+clean; 6.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 633 op/s 2024-02-09T17:37:56.775 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:37:56.775 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:37:56.776 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:37:56.776 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:37:56.776 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:37:56.776 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:37:56.776 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:37:56.776 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:37:56.776 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:37:56.798 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:37:56.823 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:37:56.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:56.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:37:56.843 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:37:56.887 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:37:56.903 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:37:56.903 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:37:56.903 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:37:56.903 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:37:56.903 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:37:56.903 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:37:56.903 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:37:56.903 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:37:56.903 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:37:56.903 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:37:56.904 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:37:56.904 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:37:56.932 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:37:56.990 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:57.140 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:56 smithi099 ceph-mon[39420]: from='client.15582 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:37:57.140 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:56 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2979022257' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:37:57.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:56 smithi179 ceph-mon[43975]: from='client.15582 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:37:57.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:56 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2979022257' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:37:57.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:56 smithi162 ceph-mon[44304]: from='client.15582 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:37:57.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:56 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2979022257' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:37:58.078 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:37:58.112 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":1000,"stamp":"2024-02-09T17:37:54.688354+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6624772664,"num_objects":101080,"num_object_clones":0,"num_object_copies":303240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101080,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173850,"num_write_kb":8797633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":163669,"ondisk_log_size":163669,"up":111,"acting":111,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":111,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":22806800,"kb_used_data":19420872,"kb_used_omap":229,"kb_used_meta":3385306,"kb_avail":1102135024,"statfs":{"total":1151940427776,"available":1128586264576,"internally_reserved":0,"allocated":19886972928,"data_stored":19883362822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235127,"internal_metadata":3466553737},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,3,4,8,1,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-328335360,"num_objects":-5010,"num_object_clones":0,"num_object_copies":-15030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5010,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.909890"},"pg_stats":[{"pgid":"1.14","version":"63'139","reported_seq":965,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-09T17:37:49.827527+0000","last_change":"2024-02-09T17:37:03.748841+0000","last_active":"2024-02-09T17:37:49.827527+0000","last_peered":"2024-02-09T17:37:49.827527+0000","last_clean":"2024-02-09T17:37:49.827527+0000","last_became_active":"2024-02-09T17:33:05.309643+0000","last_became_peered":"2024-02-09T17:33:05.309643+0000","last_unstale":"2024-02-09T17:37:49.827527+0000","last_undegraded":"2024-02-09T17:37:49.827527+0000","last_fullsized":"2024-02-09T17:37:49.827527+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:37:03.748701+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:34:13.228622+0000","last_clean_scrub_stamp":"2024-02-09T17:37:03.748701+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:22.979456+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"365'10811","reported_seq":11001,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-09T17:37:50.795314+0000","last_change":"2024-02-09T17:37:10.153953+0000","last_active":"2024-02-09T17:37:50.795314+0000","last_peered":"2024-02-09T17:37:50.795314+0000","last_clean":"2024-02-09T17:37:50.795314+0000","last_became_active":"2024-02-09T17:35:30.390786+0000","last_became_peered":"2024-02-09T17:35:30.390786+0000","last_unstale":"2024-02-09T17:37:50.795314+0000","last_undegraded":"2024-02-09T17:37:50.795314+0000","last_fullsized":"2024-02-09T17:37:50.795314+0000","mapping_epoch":322,"log_start":"330'800","ondisk_log_start":"330'800","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'8529","last_scrub_stamp":"2024-02-09T17:37:10.153802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:37:10.153802+0000","objects_scrubbed":8484,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:34.665727+0000","scrub_duration":1669,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406388736,"num_objects":6201,"num_object_clones":0,"num_object_copies":18603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10811,"num_write_kb":544384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,11],"acting":[6,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"63'139","reported_seq":995,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-09T17:37:49.824994+0000","last_change":"2024-02-09T17:37:07.369195+0000","last_active":"2024-02-09T17:37:49.824994+0000","last_peered":"2024-02-09T17:37:49.824994+0000","last_clean":"2024-02-09T17:37:49.824994+0000","last_became_active":"2024-02-09T17:33:00.284421+0000","last_became_peered":"2024-02-09T17:33:00.284421+0000","last_unstale":"2024-02-09T17:37:49.824994+0000","last_undegraded":"2024-02-09T17:37:49.824994+0000","last_fullsized":"2024-02-09T17:37:49.824994+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:37:07.369056+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:37:07.369056+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:14.417376+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"365'10924","reported_seq":11112,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-09T17:37:50.422841+0000","last_change":"2024-02-09T17:37:07.695049+0000","last_active":"2024-02-09T17:37:50.422841+0000","last_peered":"2024-02-09T17:37:50.422841+0000","last_clean":"2024-02-09T17:37:50.422841+0000","last_became_active":"2024-02-09T17:35:30.390424+0000","last_became_peered":"2024-02-09T17:35:30.390424+0000","last_unstale":"2024-02-09T17:37:50.422841+0000","last_undegraded":"2024-02-09T17:37:50.422841+0000","last_fullsized":"2024-02-09T17:37:50.422841+0000","mapping_epoch":322,"log_start":"330'900","ondisk_log_start":"330'900","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"356'8575","last_scrub_stamp":"2024-02-09T17:37:07.694769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:37:07.694769+0000","objects_scrubbed":8497,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:25.342399+0000","scrub_duration":3173,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":421396480,"num_objects":6430,"num_object_clones":0,"num_object_copies":19290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10924,"num_write_kb":555328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,0],"acting":[1,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"63'139","reported_seq":1011,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-09T17:37:49.824918+0000","last_change":"2024-02-09T17:37:10.429162+0000","last_active":"2024-02-09T17:37:49.824918+0000","last_peered":"2024-02-09T17:37:49.824918+0000","last_clean":"2024-02-09T17:37:49.824918+0000","last_became_active":"2024-02-09T17:34:01.410594+0000","last_became_peered":"2024-02-09T17:34:01.410594+0000","last_unstale":"2024-02-09T17:37:49.824918+0000","last_undegraded":"2024-02-09T17:37:49.824918+0000","last_fullsized":"2024-02-09T17:37:49.824918+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:37:10.429033+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:37:10.429033+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:17.314134+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"365'10956","reported_seq":11147,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-09T17:37:51.361323+0000","last_change":"2024-02-09T17:37:09.133954+0000","last_active":"2024-02-09T17:37:51.361323+0000","last_peered":"2024-02-09T17:37:51.361323+0000","last_clean":"2024-02-09T17:37:51.361323+0000","last_became_active":"2024-02-09T17:35:30.166669+0000","last_became_peered":"2024-02-09T17:35:30.166669+0000","last_unstale":"2024-02-09T17:37:51.361323+0000","last_undegraded":"2024-02-09T17:37:51.361323+0000","last_fullsized":"2024-02-09T17:37:51.361323+0000","mapping_epoch":322,"log_start":"330'900","ondisk_log_start":"330'900","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'8615","last_scrub_stamp":"2024-02-09T17:37:09.133803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:37:09.133803+0000","objects_scrubbed":8567,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:13.853601+0000","scrub_duration":2038,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":413794304,"num_objects":6314,"num_object_clones":0,"num_object_copies":18942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10956,"num_write_kb":552640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,2],"acting":[9,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.3","version":"63'139","reported_seq":508,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-09T17:37:49.822183+0000","last_change":"2024-02-09T17:37:14.844939+0000","last_active":"2024-02-09T17:37:49.822183+0000","last_peered":"2024-02-09T17:37:49.822183+0000","last_clean":"2024-02-09T17:37:49.822183+0000","last_became_active":"2024-02-09T17:33:00.279369+0000","last_became_peered":"2024-02-09T17:33:00.279369+0000","last_unstale":"2024-02-09T17:37:49.822183+0000","last_undegraded":"2024-02-09T17:37:49.822183+0000","last_fullsized":"2024-02-09T17:37:49.822183+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:37:14.844841+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:37:14.844841+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:15.289072+0000","scrub_duration":98,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"365'10756","reported_seq":10945,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-09T17:37:50.412706+0000","last_change":"2024-02-09T17:37:11.857576+0000","last_active":"2024-02-09T17:37:50.412706+0000","last_peered":"2024-02-09T17:37:50.412706+0000","last_clean":"2024-02-09T17:37:50.412706+0000","last_became_active":"2024-02-09T17:35:30.166273+0000","last_became_peered":"2024-02-09T17:35:30.166273+0000","last_unstale":"2024-02-09T17:37:50.412706+0000","last_undegraded":"2024-02-09T17:37:50.412706+0000","last_fullsized":"2024-02-09T17:37:50.412706+0000","mapping_epoch":322,"log_start":"330'700","ondisk_log_start":"330'700","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'8608","last_scrub_stamp":"2024-02-09T17:37:11.857362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:37:11.857362+0000","objects_scrubbed":8467,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:38.710861+0000","scrub_duration":2226,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410517504,"num_objects":6264,"num_object_clones":0,"num_object_copies":18792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10756,"num_write_kb":544640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"63'139","reported_seq":519,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-09T17:37:49.815844+0000","last_change":"2024-02-09T17:37:03.758658+0000","last_active":"2024-02-09T17:37:49.815844+0000","last_peered":"2024-02-09T17:37:49.815844+0000","last_clean":"2024-02-09T17:37:49.815844+0000","last_became_active":"2024-02-09T17:34:01.342607+0000","last_became_peered":"2024-02-09T17:34:01.342607+0000","last_unstale":"2024-02-09T17:37:49.815844+0000","last_undegraded":"2024-02-09T17:37:49.815844+0000","last_fullsized":"2024-02-09T17:37:49.815844+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:37:03.758546+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:37:03.758546+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:12.219105+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.3","version":"365'10731","reported_seq":10920,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-09T17:37:50.206071+0000","last_change":"2024-02-09T17:37:13.488486+0000","last_active":"2024-02-09T17:37:50.206071+0000","last_peered":"2024-02-09T17:37:50.206071+0000","last_clean":"2024-02-09T17:37:50.206071+0000","last_became_active":"2024-02-09T17:35:30.134014+0000","last_became_peered":"2024-02-09T17:35:30.134014+0000","last_unstale":"2024-02-09T17:37:50.206071+0000","last_undegraded":"2024-02-09T17:37:50.206071+0000","last_fullsized":"2024-02-09T17:37:50.206071+0000","mapping_epoch":322,"log_start":"330'700","ondisk_log_start":"330'700","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'8765","last_scrub_stamp":"2024-02-09T17:37:13.488343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:37:13.488343+0000","objects_scrubbed":8429,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:28.392245+0000","scrub_duration":2180,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420544512,"num_objects":6417,"num_object_clones":0,"num_object_copies":19251,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10731,"num_write_kb":548736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,1],"acting":[11,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"63'139","reported_seq":510,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-09T17:37:49.817090+0000","last_change":"2024-02-09T17:37:07.707766+0000","last_active":"2024-02-09T17:37:49.817090+0000","last_peered":"2024-02-09T17:37:49.817090+0000","last_clean":"2024-02-09T17:37:49.817090+0000","last_became_active":"2024-02-09T17:33:00.284702+0000","last_became_peered":"2024-02-09T17:33:00.284702+0000","last_unstale":"2024-02-09T17:37:49.817090+0000","last_undegraded":"2024-02-09T17:37:49.817090+0000","last_fullsized":"2024-02-09T17:37:49.817090+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:37:07.707633+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:37:07.707633+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:28.410525+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2","version":"365'10804","reported_seq":10994,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-09T17:37:51.373654+0000","last_change":"2024-02-09T17:37:09.776219+0000","last_active":"2024-02-09T17:37:51.373654+0000","last_peered":"2024-02-09T17:37:51.373654+0000","last_clean":"2024-02-09T17:37:51.373654+0000","last_became_active":"2024-02-09T17:35:29.899914+0000","last_became_peered":"2024-02-09T17:35:29.899914+0000","last_unstale":"2024-02-09T17:37:51.373654+0000","last_undegraded":"2024-02-09T17:37:51.373654+0000","last_fullsized":"2024-02-09T17:37:51.373654+0000","mapping_epoch":322,"log_start":"330'800","ondisk_log_start":"330'800","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'8538","last_scrub_stamp":"2024-02-09T17:37:09.776086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:37:09.776086+0000","objects_scrubbed":8510,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:34.505426+0000","scrub_duration":1666,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410779648,"num_objects":6268,"num_object_clones":0,"num_object_copies":18804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10804,"num_write_kb":546304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.7","version":"63'139","reported_seq":523,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-09T17:37:49.817776+0000","last_change":"2024-02-09T17:37:06.599735+0000","last_active":"2024-02-09T17:37:49.817776+0000","last_peered":"2024-02-09T17:37:49.817776+0000","last_clean":"2024-02-09T17:37:49.817776+0000","last_became_active":"2024-02-09T17:33:00.278966+0000","last_became_peered":"2024-02-09T17:33:00.278966+0000","last_unstale":"2024-02-09T17:37:49.817776+0000","last_undegraded":"2024-02-09T17:37:49.817776+0000","last_fullsized":"2024-02-09T17:37:49.817776+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:37:06.599635+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:37:06.599635+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:25.470941+0000","scrub_duration":102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"365'10833","reported_seq":11027,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-09T17:37:50.137846+0000","last_change":"2024-02-09T17:37:05.983264+0000","last_active":"2024-02-09T17:37:50.137846+0000","last_peered":"2024-02-09T17:37:50.137846+0000","last_clean":"2024-02-09T17:37:50.137846+0000","last_became_active":"2024-02-09T17:35:29.900032+0000","last_became_peered":"2024-02-09T17:35:29.900032+0000","last_unstale":"2024-02-09T17:37:50.137846+0000","last_undegraded":"2024-02-09T17:37:50.137846+0000","last_fullsized":"2024-02-09T17:37:50.137846+0000","mapping_epoch":322,"log_start":"330'800","ondisk_log_start":"330'800","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"356'8379","last_scrub_stamp":"2024-02-09T17:37:05.983122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:37:05.983122+0000","objects_scrubbed":8324,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:28.414791+0000","scrub_duration":1701,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":417923072,"num_objects":6377,"num_object_clones":0,"num_object_copies":19131,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10833,"num_write_kb":550720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"63'139","reported_seq":601,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-09T17:37:49.819521+0000","last_change":"2024-02-09T17:37:19.223274+0000","last_active":"2024-02-09T17:37:49.819521+0000","last_peered":"2024-02-09T17:37:49.819521+0000","last_clean":"2024-02-09T17:37:49.819521+0000","last_became_active":"2024-02-09T17:33:00.284924+0000","last_became_peered":"2024-02-09T17:33:00.284924+0000","last_unstale":"2024-02-09T17:37:49.819521+0000","last_undegraded":"2024-02-09T17:37:49.819521+0000","last_fullsized":"2024-02-09T17:37:49.819521+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:37:19.223119+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:37:19.223119+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:43.320825+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.0","version":"365'10950","reported_seq":11139,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-09T17:37:51.965780+0000","last_change":"2024-02-09T17:37:13.908630+0000","last_active":"2024-02-09T17:37:51.965780+0000","last_peered":"2024-02-09T17:37:51.965780+0000","last_clean":"2024-02-09T17:37:51.965780+0000","last_became_active":"2024-02-09T17:35:30.134347+0000","last_became_peered":"2024-02-09T17:35:30.134347+0000","last_unstale":"2024-02-09T17:37:51.965780+0000","last_undegraded":"2024-02-09T17:37:51.965780+0000","last_fullsized":"2024-02-09T17:37:51.965780+0000","mapping_epoch":322,"log_start":"330'900","ondisk_log_start":"330'900","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'8832","last_scrub_stamp":"2024-02-09T17:37:13.908444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:37:13.908444+0000","objects_scrubbed":8446,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:33.081607+0000","scrub_duration":2861,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":409862144,"num_objects":6254,"num_object_clones":0,"num_object_copies":18762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10950,"num_write_kb":550528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"63'139","reported_seq":510,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-09T17:37:49.816014+0000","last_change":"2024-02-09T17:37:02.738763+0000","last_active":"2024-02-09T17:37:49.816014+0000","last_peered":"2024-02-09T17:37:49.816014+0000","last_clean":"2024-02-09T17:37:49.816014+0000","last_became_active":"2024-02-09T17:33:00.274636+0000","last_became_peered":"2024-02-09T17:33:00.274636+0000","last_unstale":"2024-02-09T17:37:49.816014+0000","last_undegraded":"2024-02-09T17:37:49.816014+0000","last_fullsized":"2024-02-09T17:37:49.816014+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:37:02.738611+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:37:02.738611+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:28.854048+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.f","version":"365'10751","reported_seq":10937,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-09T17:37:50.382711+0000","last_change":"2024-02-09T17:37:19.398235+0000","last_active":"2024-02-09T17:37:50.382711+0000","last_peered":"2024-02-09T17:37:50.382711+0000","last_clean":"2024-02-09T17:37:50.382711+0000","last_became_active":"2024-02-09T17:35:29.905385+0000","last_became_peered":"2024-02-09T17:35:29.905385+0000","last_unstale":"2024-02-09T17:37:50.382711+0000","last_undegraded":"2024-02-09T17:37:50.382711+0000","last_fullsized":"2024-02-09T17:37:50.382711+0000","mapping_epoch":322,"log_start":"330'700","ondisk_log_start":"330'700","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'9026","last_scrub_stamp":"2024-02-09T17:37:19.398062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:37:19.398062+0000","objects_scrubbed":8100,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:46.416390+0000","scrub_duration":2684,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":413597696,"num_objects":6311,"num_object_clones":0,"num_object_copies":18933,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10751,"num_write_kb":545984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,11],"acting":[1,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"63'139","reported_seq":512,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-09T17:37:49.813952+0000","last_change":"2024-02-09T17:37:35.358011+0000","last_active":"2024-02-09T17:37:49.813952+0000","last_peered":"2024-02-09T17:37:49.813952+0000","last_clean":"2024-02-09T17:37:49.813952+0000","last_became_active":"2024-02-09T17:32:35.418200+0000","last_became_peered":"2024-02-09T17:32:35.418200+0000","last_unstale":"2024-02-09T17:37:49.813952+0000","last_undegraded":"2024-02-09T17:37:49.813952+0000","last_fullsized":"2024-02-09T17:37:49.813952+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:37:35.357889+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:37:35.357889+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:44.325409+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.e","version":"365'10825","reported_seq":11015,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-09T17:37:50.520499+0000","last_change":"2024-02-09T17:37:11.874433+0000","last_active":"2024-02-09T17:37:50.520499+0000","last_peered":"2024-02-09T17:37:50.520499+0000","last_clean":"2024-02-09T17:37:50.520499+0000","last_became_active":"2024-02-09T17:35:30.390749+0000","last_became_peered":"2024-02-09T17:35:30.390749+0000","last_unstale":"2024-02-09T17:37:50.520499+0000","last_undegraded":"2024-02-09T17:37:50.520499+0000","last_fullsized":"2024-02-09T17:37:50.520499+0000","mapping_epoch":322,"log_start":"330'800","ondisk_log_start":"330'800","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'8651","last_scrub_stamp":"2024-02-09T17:37:11.874252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:37:11.874252+0000","objects_scrubbed":8474,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:29.860070+0000","scrub_duration":1689,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410189824,"num_objects":6259,"num_object_clones":0,"num_object_copies":18777,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10825,"num_write_kb":546688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,6],"acting":[4,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"63'139","reported_seq":520,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-09T17:37:49.819660+0000","last_change":"2024-02-09T17:37:05.469016+0000","last_active":"2024-02-09T17:37:49.819660+0000","last_peered":"2024-02-09T17:37:49.819660+0000","last_clean":"2024-02-09T17:37:49.819660+0000","last_became_active":"2024-02-09T17:34:01.388313+0000","last_became_peered":"2024-02-09T17:34:01.388313+0000","last_unstale":"2024-02-09T17:37:49.819660+0000","last_undegraded":"2024-02-09T17:37:49.819660+0000","last_fullsized":"2024-02-09T17:37:49.819660+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:37:05.468894+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:37:05.468894+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:34.390980+0000","scrub_duration":138,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.d","version":"365'10819","reported_seq":11009,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-09T17:37:50.519315+0000","last_change":"2024-02-09T17:37:13.543612+0000","last_active":"2024-02-09T17:37:50.519315+0000","last_peered":"2024-02-09T17:37:50.519315+0000","last_clean":"2024-02-09T17:37:50.519315+0000","last_became_active":"2024-02-09T17:35:29.905824+0000","last_became_peered":"2024-02-09T17:35:29.905824+0000","last_unstale":"2024-02-09T17:37:50.519315+0000","last_undegraded":"2024-02-09T17:37:50.519315+0000","last_fullsized":"2024-02-09T17:37:50.519315+0000","mapping_epoch":322,"log_start":"330'800","ondisk_log_start":"330'800","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'8767","last_scrub_stamp":"2024-02-09T17:37:13.543492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:37:13.543492+0000","objects_scrubbed":8445,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:22.127226+0000","scrub_duration":2388,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":415105048,"num_objects":6335,"num_object_clones":0,"num_object_copies":19005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6335,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10819,"num_write_kb":548865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,1],"acting":[4,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"63'139","reported_seq":521,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-09T17:37:49.812600+0000","last_change":"2024-02-09T17:37:21.762086+0000","last_active":"2024-02-09T17:37:49.812600+0000","last_peered":"2024-02-09T17:37:49.812600+0000","last_clean":"2024-02-09T17:37:49.812600+0000","last_became_active":"2024-02-09T17:32:35.384617+0000","last_became_peered":"2024-02-09T17:32:35.384617+0000","last_unstale":"2024-02-09T17:37:49.812600+0000","last_undegraded":"2024-02-09T17:37:49.812600+0000","last_fullsized":"2024-02-09T17:37:49.812600+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:37:21.761916+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:37:21.761916+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:49.944236+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"365'10996","reported_seq":11183,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-09T17:37:50.480556+0000","last_change":"2024-02-09T17:37:20.108495+0000","last_active":"2024-02-09T17:37:50.480556+0000","last_peered":"2024-02-09T17:37:50.480556+0000","last_clean":"2024-02-09T17:37:50.480556+0000","last_became_active":"2024-02-09T17:35:29.903553+0000","last_became_peered":"2024-02-09T17:35:29.903553+0000","last_unstale":"2024-02-09T17:37:50.480556+0000","last_undegraded":"2024-02-09T17:37:50.480556+0000","last_fullsized":"2024-02-09T17:37:50.480556+0000","mapping_epoch":322,"log_start":"330'900","ondisk_log_start":"330'900","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'9333","last_scrub_stamp":"2024-02-09T17:37:20.108372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:37:20.108372+0000","objects_scrubbed":8252,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:40.001226+0000","scrub_duration":2010,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":427556864,"num_objects":6524,"num_object_clones":0,"num_object_copies":19572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10996,"num_write_kb":560640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"63'139","reported_seq":997,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-09T17:37:49.824869+0000","last_change":"2024-02-09T17:37:11.520715+0000","last_active":"2024-02-09T17:37:49.824869+0000","last_peered":"2024-02-09T17:37:49.824869+0000","last_clean":"2024-02-09T17:37:49.824869+0000","last_became_active":"2024-02-09T17:32:35.402025+0000","last_became_peered":"2024-02-09T17:32:35.402025+0000","last_unstale":"2024-02-09T17:37:49.824869+0000","last_undegraded":"2024-02-09T17:37:49.824869+0000","last_fullsized":"2024-02-09T17:37:49.824869+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:37:11.520601+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:30.193194+0000","last_clean_scrub_stamp":"2024-02-09T17:37:11.520601+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:38.007919+0000","scrub_duration":102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"365'11187","reported_seq":11375,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-09T17:37:53.634658+0000","last_change":"2024-02-09T17:37:05.121978+0000","last_active":"2024-02-09T17:37:53.634658+0000","last_peered":"2024-02-09T17:37:53.634658+0000","last_clean":"2024-02-09T17:37:53.634658+0000","last_became_active":"2024-02-09T17:35:29.906768+0000","last_became_peered":"2024-02-09T17:35:29.906768+0000","last_unstale":"2024-02-09T17:37:53.634658+0000","last_undegraded":"2024-02-09T17:37:53.634658+0000","last_fullsized":"2024-02-09T17:37:53.634658+0000","mapping_epoch":322,"log_start":"330'1100","ondisk_log_start":"330'1100","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"356'8446","last_scrub_stamp":"2024-02-09T17:37:05.121787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:37:05.121787+0000","objects_scrubbed":8387,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:30.011025+0000","scrub_duration":2496,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":407699456,"num_objects":6221,"num_object_clones":0,"num_object_copies":18663,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11187,"num_write_kb":557056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"63'139","reported_seq":473,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-09T17:37:49.813345+0000","last_change":"2024-02-09T17:37:06.250206+0000","last_active":"2024-02-09T17:37:49.813345+0000","last_peered":"2024-02-09T17:37:49.813345+0000","last_clean":"2024-02-09T17:37:49.813345+0000","last_became_active":"2024-02-09T17:33:05.309824+0000","last_became_peered":"2024-02-09T17:33:05.309824+0000","last_unstale":"2024-02-09T17:37:49.813345+0000","last_undegraded":"2024-02-09T17:37:49.813345+0000","last_fullsized":"2024-02-09T17:37:49.813345+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:37:06.250017+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:37:06.250017+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:24.561623+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"365'10786","reported_seq":10976,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-09T17:37:50.777023+0000","last_change":"2024-02-09T17:37:09.380478+0000","last_active":"2024-02-09T17:37:50.777023+0000","last_peered":"2024-02-09T17:37:50.777023+0000","last_clean":"2024-02-09T17:37:50.777023+0000","last_became_active":"2024-02-09T17:35:30.390843+0000","last_became_peered":"2024-02-09T17:35:30.390843+0000","last_unstale":"2024-02-09T17:37:50.777023+0000","last_undegraded":"2024-02-09T17:37:50.777023+0000","last_fullsized":"2024-02-09T17:37:50.777023+0000","mapping_epoch":322,"log_start":"330'700","ondisk_log_start":"330'700","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'8477","last_scrub_stamp":"2024-02-09T17:37:09.380351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:37:09.380351+0000","objects_scrubbed":8431,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:24.285734+0000","scrub_duration":1888,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406192128,"num_objects":6198,"num_object_clones":0,"num_object_copies":18594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10786,"num_write_kb":543488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.f","version":"63'139","reported_seq":531,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-09T17:37:49.817708+0000","last_change":"2024-02-09T17:36:22.364212+0000","last_active":"2024-02-09T17:37:49.817708+0000","last_peered":"2024-02-09T17:37:49.817708+0000","last_clean":"2024-02-09T17:37:49.817708+0000","last_became_active":"2024-02-09T17:33:05.316100+0000","last_became_peered":"2024-02-09T17:33:05.316100+0000","last_unstale":"2024-02-09T17:37:49.817708+0000","last_undegraded":"2024-02-09T17:37:49.817708+0000","last_fullsized":"2024-02-09T17:37:49.817708+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:36:22.364130+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:36:22.364130+0000","last_clean_scrub_stamp":"2024-02-09T17:36:22.364130+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:37:53.516468+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"365'10878","reported_seq":11062,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-09T17:37:50.173868+0000","last_change":"2024-02-09T17:37:22.382066+0000","last_active":"2024-02-09T17:37:50.173868+0000","last_peered":"2024-02-09T17:37:50.173868+0000","last_clean":"2024-02-09T17:37:50.173868+0000","last_became_active":"2024-02-09T17:35:29.905885+0000","last_became_peered":"2024-02-09T17:35:29.905885+0000","last_unstale":"2024-02-09T17:37:50.173868+0000","last_undegraded":"2024-02-09T17:37:50.173868+0000","last_fullsized":"2024-02-09T17:37:50.173868+0000","mapping_epoch":322,"log_start":"330'800","ondisk_log_start":"330'800","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"359'9289","last_scrub_stamp":"2024-02-09T17:37:22.381944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:37:22.381944+0000","objects_scrubbed":7938,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:44.146286+0000","scrub_duration":2196,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":412876800,"num_objects":6300,"num_object_clones":0,"num_object_copies":18900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10878,"num_write_kb":549696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,3],"acting":[11,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"63'139","reported_seq":969,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-09T17:37:49.822674+0000","last_change":"2024-02-09T17:37:07.137667+0000","last_active":"2024-02-09T17:37:49.822674+0000","last_peered":"2024-02-09T17:37:49.822674+0000","last_clean":"2024-02-09T17:37:49.822674+0000","last_became_active":"2024-02-09T17:35:20.840386+0000","last_became_peered":"2024-02-09T17:35:20.840386+0000","last_unstale":"2024-02-09T17:37:49.822674+0000","last_undegraded":"2024-02-09T17:37:49.822674+0000","last_fullsized":"2024-02-09T17:37:49.822674+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:37:07.137544+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:23.912504+0000","last_clean_scrub_stamp":"2024-02-09T17:37:07.137544+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:21.263740+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"365'10843","reported_seq":11030,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-09T17:37:50.077662+0000","last_change":"2024-02-09T17:37:17.682453+0000","last_active":"2024-02-09T17:37:50.077662+0000","last_peered":"2024-02-09T17:37:50.077662+0000","last_clean":"2024-02-09T17:37:50.077662+0000","last_became_active":"2024-02-09T17:35:30.134041+0000","last_became_peered":"2024-02-09T17:35:30.134041+0000","last_unstale":"2024-02-09T17:37:50.077662+0000","last_undegraded":"2024-02-09T17:37:50.077662+0000","last_fullsized":"2024-02-09T17:37:50.077662+0000","mapping_epoch":322,"log_start":"330'800","ondisk_log_start":"330'800","created":322,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'9078","last_scrub_stamp":"2024-02-09T17:37:17.682321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:35:28.884607+0000","last_clean_scrub_stamp":"2024-02-09T17:37:17.682321+0000","objects_scrubbed":8212,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:38.388985+0000","scrub_duration":1514,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419758080,"num_objects":6405,"num_object_clones":0,"num_object_copies":19215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10843,"num_write_kb":551936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"63'139","reported_seq":481,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-09T17:37:49.825142+0000","last_change":"2024-02-09T17:37:09.476252+0000","last_active":"2024-02-09T17:37:49.825142+0000","last_peered":"2024-02-09T17:37:49.825142+0000","last_clean":"2024-02-09T17:37:49.825142+0000","last_became_active":"2024-02-09T17:33:05.319668+0000","last_became_peered":"2024-02-09T17:33:05.319668+0000","last_unstale":"2024-02-09T17:37:49.825142+0000","last_undegraded":"2024-02-09T17:37:49.825142+0000","last_fullsized":"2024-02-09T17:37:49.825142+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:37:09.476147+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:37:09.476147+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:14.845386+0000","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"63'139","reported_seq":480,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-09T17:37:49.812495+0000","last_change":"2024-02-09T17:36:17.357034+0000","last_active":"2024-02-09T17:37:49.812495+0000","last_peered":"2024-02-09T17:37:49.812495+0000","last_clean":"2024-02-09T17:37:49.812495+0000","last_became_active":"2024-02-09T17:33:05.310104+0000","last_became_peered":"2024-02-09T17:33:05.310104+0000","last_unstale":"2024-02-09T17:37:49.812495+0000","last_undegraded":"2024-02-09T17:37:49.812495+0000","last_fullsized":"2024-02-09T17:37:49.812495+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:36:17.356909+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:36:17.356909+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:37:51.722159+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"63'139","reported_seq":488,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-09T17:37:49.923693+0000","last_change":"2024-02-09T17:36:18.779412+0000","last_active":"2024-02-09T17:37:49.923693+0000","last_peered":"2024-02-09T17:37:49.923693+0000","last_clean":"2024-02-09T17:37:49.923693+0000","last_became_active":"2024-02-09T17:34:01.353171+0000","last_became_peered":"2024-02-09T17:34:01.353171+0000","last_unstale":"2024-02-09T17:37:49.923693+0000","last_undegraded":"2024-02-09T17:37:49.923693+0000","last_fullsized":"2024-02-09T17:37:49.923693+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:36:18.779304+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:36:18.779304+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.12","version":"63'139","reported_seq":962,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-09T17:37:49.812583+0000","last_change":"2024-02-09T17:37:07.797486+0000","last_active":"2024-02-09T17:37:49.812583+0000","last_peered":"2024-02-09T17:37:49.812583+0000","last_clean":"2024-02-09T17:37:49.812583+0000","last_became_active":"2024-02-09T17:33:05.308268+0000","last_became_peered":"2024-02-09T17:33:05.308268+0000","last_unstale":"2024-02-09T17:37:49.812583+0000","last_undegraded":"2024-02-09T17:37:49.812583+0000","last_fullsized":"2024-02-09T17:37:49.812583+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:37:07.797354+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:37:07.797354+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:38:29.748476+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"63'139","reported_seq":516,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-09T17:37:49.822139+0000","last_change":"2024-02-09T17:36:27.497000+0000","last_active":"2024-02-09T17:37:49.822139+0000","last_peered":"2024-02-09T17:37:49.822139+0000","last_clean":"2024-02-09T17:37:49.822139+0000","last_became_active":"2024-02-09T17:33:05.309527+0000","last_became_peered":"2024-02-09T17:33:05.309527+0000","last_unstale":"2024-02-09T17:37:49.822139+0000","last_undegraded":"2024-02-09T17:37:49.822139+0000","last_fullsized":"2024-02-09T17:37:49.822139+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:36:27.496900+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:36:27.496900+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:37:56.755524+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":6624182296,"num_objects":101078,"num_object_clones":0,"num_object_copies":303234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101078,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173850,"num_write_kb":8797633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19794767872,"data_stored":19794755656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160750,"ondisk_log_size":160750,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2919,"ondisk_log_size":2919,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056826,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3280516,"kb_used_data":2861944,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":90464636,"statfs":{"total":95995035648,"available":92635787264,"internally_reserved":0,"allocated":2930630656,"data_stored":2930335843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":428538308},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:37:16 2024","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.3009999999999999,"15min":1.103},"min":{"1min":0.246,"5min":0.216,"15min":0.216},"max":{"1min":8.1660000000000004,"5min":12.836,"15min":14.913},"last":6.8819999999999997},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.335,"15min":1.198},"min":{"1min":0.27000000000000002,"5min":0.25700000000000001,"15min":0.23000000000000001},"max":{"1min":7.8499999999999996,"5min":9.7639999999999993,"15min":14.869},"last":4.5090000000000003}]},{"osd":1,"last update":"Fri Feb 9 17:37:16 2024","interfaces":[{"interface":"back","average":{"1min":1.877,"5min":1.2989999999999999,"15min":1.4830000000000001},"min":{"1min":0.26700000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":7.3209999999999997,"5min":9.0079999999999991,"15min":32.957999999999998},"last":4.2720000000000002},{"interface":"front","average":{"1min":1.7889999999999999,"5min":1.389,"15min":1.552},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.26700000000000002},"max":{"1min":7.3090000000000002,"5min":9.0280000000000005,"15min":32.880000000000003},"last":6.8029999999999999}]},{"osd":2,"last update":"Fri Feb 9 17:31:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.415,"15min":1.6859999999999999},"min":{"1min":0.378,"5min":0.247,"15min":0.247},"max":{"1min":4.4189999999999996,"5min":9.4719999999999995,"15min":9.4719999999999995},"last":0.35399999999999998},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.5149999999999999,"15min":1.4399999999999999},"min":{"1min":0.378,"5min":0.252,"15min":0.252},"max":{"1min":4.8579999999999997,"5min":7.96,"15min":7.96},"last":0.60399999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:37:16 2024","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.093,"15min":1.52},"min":{"1min":0.27400000000000002,"5min":0.25700000000000001,"15min":0.247},"max":{"1min":5.6980000000000004,"5min":5.6980000000000004,"15min":21.315999999999999},"last":3.6440000000000001},{"interface":"front","average":{"1min":1.8460000000000001,"5min":1.4630000000000001,"15min":1.488},"min":{"1min":0.36599999999999999,"5min":0.312,"15min":0.26400000000000001},"max":{"1min":5.7679999999999998,"5min":7.2439999999999998,"15min":18.087},"last":3.9969999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:37:16 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":2.2389999999999999,"15min":1.3220000000000001},"min":{"1min":0.27900000000000003,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":3.6509999999999998,"5min":16.125,"15min":16.125},"last":2.778},{"interface":"front","average":{"1min":0.78000000000000003,"5min":1.5389999999999999,"15min":1.1439999999999999},"min":{"1min":0.316,"5min":0.316,"15min":0.27700000000000002},"max":{"1min":3.0139999999999998,"5min":16.091999999999999,"15min":16.091999999999999},"last":1.3520000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:37:16 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.8220000000000001,"15min":1.2689999999999999},"min":{"1min":0.37,"5min":0.312,"15min":0.28899999999999998},"max":{"1min":5.8819999999999997,"5min":13.946999999999999,"15min":13.946999999999999},"last":0.316},{"interface":"front","average":{"1min":1.038,"5min":2.1179999999999999,"15min":1.3149999999999999},"min":{"1min":0.315,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":3.1760000000000002,"5min":13.414,"15min":13.414},"last":2.7989999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:37:16 2024","interfaces":[{"interface":"back","average":{"1min":1.2410000000000001,"5min":1.9490000000000001,"15min":1.2749999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":3.7080000000000002,"5min":12.798999999999999,"15min":12.798999999999999},"last":2.0470000000000002},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.8740000000000001,"15min":1.29},"min":{"1min":0.379,"5min":0.33000000000000002,"15min":0.32300000000000001},"max":{"1min":4.3310000000000004,"5min":10.782999999999999,"15min":10.782999999999999},"last":6.6429999999999998}]},{"osd":7,"last update":"Fri Feb 9 17:37:16 2024","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":2.2469999999999999,"15min":1.4490000000000001},"min":{"1min":0.38,"5min":0.312,"15min":0.312},"max":{"1min":3.6139999999999999,"5min":14.462,"15min":14.462},"last":2.827},{"interface":"front","average":{"1min":1.0629999999999999,"5min":2.2879999999999998,"15min":1.431},"min":{"1min":0.40100000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":3.2799999999999998,"5min":14.398999999999999,"15min":14.398999999999999},"last":0.35299999999999998}]},{"osd":8,"last update":"Fri Feb 9 17:37:48 2024","interfaces":[{"interface":"back","average":{"1min":1.879,"5min":1.3640000000000001,"15min":2.0680000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":9.2680000000000007,"5min":13.199999999999999,"15min":13.199999999999999},"last":6.6799999999999997},{"interface":"front","average":{"1min":1.788,"5min":1.2310000000000001,"15min":1.6990000000000001},"min":{"1min":0.34300000000000003,"5min":0.318,"15min":0.318},"max":{"1min":9.2940000000000005,"5min":9.2940000000000005,"15min":9.2940000000000005},"last":6.7789999999999999}]},{"osd":9,"last update":"Fri Feb 9 17:37:04 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":1.417,"15min":1.2430000000000001},"min":{"1min":0.39300000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":2.839,"5min":12.821,"15min":12.821},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.747,"5min":1.276,"15min":1.0289999999999999},"min":{"1min":0.45400000000000001,"5min":0.38500000000000001,"15min":0.34100000000000003},"max":{"1min":2.153,"5min":7.6989999999999998,"15min":7.6989999999999998},"last":2.742}]},{"osd":10,"last update":"Fri Feb 9 17:37:16 2024","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":1.284,"15min":1.0640000000000001},"min":{"1min":0.41299999999999998,"5min":0.36199999999999999,"15min":0.33900000000000002},"max":{"1min":3.371,"5min":9.6199999999999992,"15min":9.6199999999999992},"last":6.7439999999999998},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.51,"15min":1.101},"min":{"1min":0.40000000000000002,"5min":0.35799999999999998,"15min":0.33700000000000002},"max":{"1min":5.3680000000000003,"5min":12.856,"15min":12.856},"last":0.40899999999999997}]}]},{"osd":10,"up_from":84,"seq":360777253056,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1749644,"kb_used_data":1578116,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":91995508,"statfs":{"total":95995035648,"available":94203400192,"internally_reserved":0,"allocated":1615990784,"data_stored":1615691473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":175503808},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:37:04 2024","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.5640000000000001,"15min":1.7809999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":8.5779999999999994,"5min":8.5779999999999994,"15min":8.5779999999999994},"last":0.30399999999999999},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.05,"15min":1.1020000000000001},"min":{"1min":0.27800000000000002,"5min":0.253,"15min":0.253},"max":{"1min":5.923,"5min":5.923,"15min":5.923},"last":2.0339999999999998}]},{"osd":1,"last update":"Fri Feb 9 17:33:46 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":1.345,"15min":1.1599999999999999},"min":{"1min":0.25900000000000001,"5min":0.24199999999999999,"15min":0.23499999999999999},"max":{"1min":3.1709999999999998,"5min":37.716000000000001,"15min":37.716000000000001},"last":0.26000000000000001},{"interface":"front","average":{"1min":1.121,"5min":1.3660000000000001,"15min":1.194},"min":{"1min":0.251,"5min":0.251,"15min":0.245},"max":{"1min":5.3049999999999997,"5min":38.039999999999999,"15min":38.039999999999999},"last":0.27800000000000002}]},{"osd":2,"last update":"Fri Feb 9 17:37:52 2024","interfaces":[{"interface":"back","average":{"1min":2.383,"5min":1.3420000000000001,"15min":1.3109999999999999},"min":{"1min":0.28499999999999998,"5min":0.27100000000000002,"15min":0.222},"max":{"1min":11.384,"5min":11.384,"15min":13.351000000000001},"last":0.28499999999999998},{"interface":"front","average":{"1min":2.2250000000000001,"5min":1.3580000000000001,"15min":1.3069999999999999},"min":{"1min":0.36499999999999999,"5min":0.27200000000000002,"15min":0.26100000000000001},"max":{"1min":11.465,"5min":11.465,"15min":11.465},"last":0.39800000000000002}]},{"osd":3,"last update":"Fri Feb 9 17:37:52 2024","interfaces":[{"interface":"back","average":{"1min":1.591,"5min":2.032,"15min":1.6399999999999999},"min":{"1min":0.309,"5min":0.26800000000000002,"15min":0.26200000000000001},"max":{"1min":5.0800000000000001,"5min":25.972999999999999,"15min":25.972999999999999},"last":2.3490000000000002},{"interface":"front","average":{"1min":1.647,"5min":1.837,"15min":1.5109999999999999},"min":{"1min":0.30399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":5.0970000000000004,"5min":25.882000000000001,"15min":25.882000000000001},"last":2.7999999999999998}]},{"osd":4,"last update":"Fri Feb 9 17:36:55 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":1.274,"15min":1.012},"min":{"1min":0.307,"5min":0.307,"15min":0.26100000000000001},"max":{"1min":3.589,"5min":7.1360000000000001,"15min":7.1360000000000001},"last":2.0710000000000002},{"interface":"front","average":{"1min":0.83599999999999997,"5min":1.2949999999999999,"15min":1.0409999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":3.5419999999999998,"5min":6.0119999999999996,"15min":6.8369999999999997},"last":0.375}]},{"osd":5,"last update":"Fri Feb 9 17:37:52 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.113,"15min":1.2190000000000001},"min":{"1min":0.32200000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":3.7370000000000001,"5min":6.5350000000000001,"15min":9.2349999999999994},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.79900000000000004,"5min":1.2090000000000001,"15min":1.163},"min":{"1min":0.374,"5min":0.33500000000000002,"15min":0.32200000000000001},"max":{"1min":4.548,"5min":5.3079999999999998,"15min":8.6159999999999997},"last":2.8319999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:37:52 2024","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":1.3009999999999999,"15min":1.1299999999999999},"min":{"1min":0.33600000000000002,"5min":0.29999999999999999,"15min":0.29099999999999998},"max":{"1min":6.0419999999999998,"5min":6.3760000000000003,"15min":6.8799999999999999},"last":6.0419999999999998},{"interface":"front","average":{"1min":0.72599999999999998,"5min":1.5780000000000001,"15min":1.3200000000000001},"min":{"1min":0.40799999999999997,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":2.4260000000000002,"5min":7.0960000000000001,"15min":9.4350000000000005},"last":0.42599999999999999}]},{"osd":7,"last update":"Fri Feb 9 17:37:52 2024","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.6140000000000001,"15min":1.2629999999999999},"min":{"1min":0.34200000000000003,"5min":0.33500000000000002,"15min":0.312},"max":{"1min":3.1549999999999998,"5min":17.754000000000001,"15min":17.754000000000001},"last":2.1269999999999998},{"interface":"front","average":{"1min":1.054,"5min":1.665,"15min":1.3799999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.316},"max":{"1min":5.0949999999999998,"5min":11.308999999999999,"15min":11.308999999999999},"last":2.698}]},{"osd":8,"last update":"Fri Feb 9 17:37:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":1.032,"15min":1.1779999999999999},"min":{"1min":0.39600000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":3.7730000000000001,"5min":6.4630000000000001,"15min":6.4630000000000001},"last":0.437},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.78200000000000003,"15min":0.92900000000000005},"min":{"1min":0.42299999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.766,"5min":5.3079999999999998,"15min":5.3079999999999998},"last":2.1000000000000001}]},{"osd":9,"last update":"Fri Feb 9 17:37:52 2024","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":1.008,"15min":0.97399999999999998},"min":{"1min":0.36399999999999999,"5min":0.36099999999999999,"15min":0.314},"max":{"1min":3.794,"5min":6.5049999999999999,"15min":8.7240000000000002},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.085,"15min":1.117},"min":{"1min":0.45700000000000002,"5min":0.35899999999999999,"15min":0.29799999999999999},"max":{"1min":3.8199999999999998,"5min":6.9640000000000004,"15min":10.1},"last":0.496}]},{"osd":11,"last update":"Fri Feb 9 17:37:16 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":1.071,"15min":1.0940000000000001},"min":{"1min":0.40600000000000003,"5min":0.379,"15min":0.36399999999999999},"max":{"1min":0.93400000000000005,"5min":7.383,"15min":8.6430000000000007},"last":2.7440000000000002},{"interface":"front","average":{"1min":0.68000000000000005,"5min":1.143,"15min":1.0980000000000001},"min":{"1min":0.45700000000000002,"5min":0.44800000000000001,"15min":0.33400000000000002},"max":{"1min":0.95999999999999996,"5min":7.3019999999999996,"15min":9.298},"last":2.7709999999999999}]}]},{"osd":9,"up_from":77,"seq":330712481989,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1507088,"kb_used_data":1209352,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":92238064,"statfs":{"total":95995035648,"available":94451777536,"internally_reserved":0,"allocated":1238376448,"data_stored":1238073065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":304871872},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:28:03 2024","interfaces":[{"interface":"back","average":{"1min":1.9510000000000001,"5min":1.454,"15min":0.85799999999999998},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":8.4499999999999993,"5min":17.495999999999999,"15min":17.495999999999999},"last":2.7360000000000002},{"interface":"front","average":{"1min":1.905,"5min":1.399,"15min":0.83499999999999996},"min":{"1min":0.32200000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":7.5990000000000002,"5min":15.401999999999999,"15min":15.401999999999999},"last":1.5449999999999999}]},{"osd":1,"last update":"Fri Feb 9 17:37:29 2024","interfaces":[{"interface":"back","average":{"1min":6.5099999999999998,"5min":2.0230000000000001,"15min":1.603},"min":{"1min":0.33600000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":89.515000000000001,"5min":89.515000000000001,"15min":89.515000000000001},"last":0.41299999999999998},{"interface":"front","average":{"1min":6.891,"5min":2.1320000000000001,"15min":1.518},"min":{"1min":0.26200000000000001,"5min":0.251,"15min":0.222},"max":{"1min":89.388000000000005,"5min":89.388000000000005,"15min":89.388000000000005},"last":3.4910000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:37:29 2024","interfaces":[{"interface":"back","average":{"1min":1.5429999999999999,"5min":1.0980000000000001,"15min":1.196},"min":{"1min":0.29599999999999999,"5min":0.27500000000000002,"15min":0.249},"max":{"1min":4.2779999999999996,"5min":5.9429999999999996,"15min":13.878},"last":1.8540000000000001},{"interface":"front","average":{"1min":1.323,"5min":1.0489999999999999,"15min":1.25},"min":{"1min":0.315,"5min":0.26100000000000001,"15min":0.25600000000000001},"max":{"1min":4.0149999999999997,"5min":5.5590000000000002,"15min":13.69},"last":2.1869999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:37:29 2024","interfaces":[{"interface":"back","average":{"1min":2.0649999999999999,"5min":1.216,"15min":1.3009999999999999},"min":{"1min":0.315,"5min":0.29699999999999999,"15min":0.26000000000000001},"max":{"1min":5.5149999999999997,"5min":5.6479999999999997,"15min":40.18},"last":3.081},{"interface":"front","average":{"1min":2.0249999999999999,"5min":1.2250000000000001,"15min":1.49},"min":{"1min":0.34000000000000002,"5min":0.30599999999999999,"15min":0.27900000000000003},"max":{"1min":5.407,"5min":5.407,"15min":40.158000000000001},"last":3.1059999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:37:29 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.548,"15min":1.2829999999999999},"min":{"1min":0.29599999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":3.992,"5min":9.0519999999999996,"15min":22.577999999999999},"last":1.383},{"interface":"front","average":{"1min":1.204,"5min":1.478,"15min":1.177},"min":{"1min":0.34300000000000003,"5min":0.33300000000000002,"15min":0.28799999999999998},"max":{"1min":5.3289999999999997,"5min":12.651,"15min":22.399999999999999},"last":0.30599999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:37:35 2024","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":1.796,"15min":1.288},"min":{"1min":0.35399999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":4.827,"5min":10.689,"15min":10.689},"last":0.36699999999999999},{"interface":"front","average":{"1min":1.369,"5min":1.3109999999999999,"15min":1.0900000000000001},"min":{"1min":0.32200000000000001,"5min":0.311,"15min":0.311},"max":{"1min":5.4329999999999998,"5min":6.4470000000000001,"15min":6.4470000000000001},"last":0.436}]},{"osd":6,"last update":"Fri Feb 9 17:37:29 2024","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.597,"15min":1.3140000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":5.6109999999999998,"5min":9.1699999999999999,"15min":9.1699999999999999},"last":0.70199999999999996},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.607,"15min":1.3069999999999999},"min":{"1min":0.40300000000000002,"5min":0.38900000000000001,"15min":0.35099999999999998},"max":{"1min":5.6449999999999996,"5min":9.2080000000000002,"15min":9.2080000000000002},"last":0.73499999999999999}]},{"osd":7,"last update":"Fri Feb 9 17:37:29 2024","interfaces":[{"interface":"back","average":{"1min":1.4730000000000001,"5min":1.8180000000000001,"15min":1.276},"min":{"1min":0.47399999999999998,"5min":0.39300000000000002,"15min":0.34300000000000003},"max":{"1min":5.4699999999999998,"5min":7.3710000000000004,"15min":8.0939999999999994},"last":1.4239999999999999},{"interface":"front","average":{"1min":1.512,"5min":1.831,"15min":1.3480000000000001},"min":{"1min":0.49399999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":5.5090000000000003,"5min":8.0530000000000008,"15min":8.0530000000000008},"last":1.4550000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:37:46 2024","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.1870000000000001,"15min":1.2689999999999999},"min":{"1min":0.443,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":4.351,"5min":5.5069999999999997,"15min":16.209},"last":3.7440000000000002},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.0840000000000001,"15min":1.286},"min":{"1min":0.42099999999999999,"5min":0.35099999999999998,"15min":0.34799999999999998},"max":{"1min":5.3819999999999997,"5min":5.5439999999999996,"15min":16.263000000000002},"last":3.706}]},{"osd":10,"last update":"Fri Feb 9 17:36:58 2024","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":1.111,"15min":1.071},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":2.7189999999999999,"5min":6.0659999999999998,"15min":9.5090000000000003},"last":0.56699999999999995},{"interface":"front","average":{"1min":1.022,"5min":0.94799999999999995,"15min":1.0169999999999999},"min":{"1min":0.40000000000000002,"5min":0.38500000000000001,"15min":0.36099999999999999},"max":{"1min":2.395,"5min":4.0759999999999996,"15min":8.0670000000000002},"last":0.39500000000000002}]},{"osd":11,"last update":"Fri Feb 9 17:36:58 2024","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.4710000000000001,"15min":1.3340000000000001},"min":{"1min":0.42599999999999999,"5min":0.42399999999999999,"15min":0.34699999999999998},"max":{"1min":7.2190000000000003,"5min":11.541,"15min":11.541},"last":0.47299999999999998},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.405,"15min":1.147},"min":{"1min":0.51100000000000001,"5min":0.374,"15min":0.374},"max":{"1min":3.1600000000000001,"5min":10.32,"15min":10.32},"last":0.50900000000000001}]}]},{"osd":8,"up_from":171,"seq":734439407732,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":132244,"kb_used_data":772,"kb_used_omap":212,"kb_used_meta":131243,"kb_avail":93612908,"statfs":{"total":95995035648,"available":95859617792,"internally_reserved":0,"allocated":790528,"data_stored":491217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217579,"internal_metadata":134393365},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:37:34 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":1.006,"15min":1.3240000000000001},"min":{"1min":0.25900000000000001,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":3.3809999999999998,"5min":8.0289999999999999,"15min":8.0289999999999999},"last":0.44600000000000001},{"interface":"front","average":{"1min":1.4279999999999999,"5min":1.153,"15min":1.351},"min":{"1min":0.28599999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":5.7060000000000004,"5min":7.5990000000000002,"15min":7.7969999999999997},"last":0.27700000000000002}]},{"osd":1,"last update":"Fri Feb 9 17:37:34 2024","interfaces":[{"interface":"back","average":{"1min":1.794,"5min":1.3919999999999999,"15min":1.2929999999999999},"min":{"1min":0.27400000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":7.7329999999999997,"5min":7.7329999999999997,"15min":7.8719999999999999},"last":0.32200000000000001},{"interface":"front","average":{"1min":1.9530000000000001,"5min":1.3979999999999999,"15min":1.8049999999999999},"min":{"1min":0.255,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":5.4809999999999999,"5min":8.5519999999999996,"15min":11.101000000000001},"last":0.53800000000000003}]},{"osd":2,"last update":"Fri Feb 9 17:37:34 2024","interfaces":[{"interface":"back","average":{"1min":1.446,"5min":1.502,"15min":1.4450000000000001},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.255},"max":{"1min":5.9969999999999999,"5min":10.423,"15min":10.423},"last":0.35599999999999998},{"interface":"front","average":{"1min":1.5049999999999999,"5min":1.369,"15min":1.5800000000000001},"min":{"1min":0.36099999999999999,"5min":0.34300000000000003,"15min":0.313},"max":{"1min":6.024,"5min":10.504,"15min":10.504},"last":0.46999999999999997}]},{"osd":3,"last update":"Fri Feb 9 17:37:34 2024","interfaces":[{"interface":"back","average":{"1min":1.621,"5min":1.54,"15min":1.968},"min":{"1min":0.39100000000000001,"5min":0.307,"15min":0.28699999999999998},"max":{"1min":4.1020000000000003,"5min":24.030999999999999,"15min":24.030999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":2.2170000000000001,"5min":1.7969999999999999,"15min":2.2709999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":10.816000000000001,"5min":24.318999999999999,"15min":24.318999999999999},"last":0.51100000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:37:34 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":1.8440000000000001,"15min":1.2470000000000001},"min":{"1min":0.29099999999999998,"5min":0.28199999999999997,"15min":0.27500000000000002},"max":{"1min":4.1779999999999999,"5min":53.060000000000002,"15min":53.060000000000002},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.67500000000000004,"5min":1.986,"15min":1.4950000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.313},"max":{"1min":3.3380000000000001,"5min":53.311999999999998,"15min":53.311999999999998},"last":0.48299999999999998}]},{"osd":6,"last update":"Fri Feb 9 17:37:34 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.573,"15min":1.569},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":3.9660000000000002,"5min":10.619999999999999,"15min":37.356999999999999},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.79200000000000004,"5min":1.286,"15min":1.1419999999999999},"min":{"1min":0.41099999999999998,"5min":0.311,"15min":0.311},"max":{"1min":3.4569999999999999,"5min":4.984,"15min":37.25},"last":0.52800000000000002}]},{"osd":7,"last update":"Fri Feb 9 17:37:34 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":1.7470000000000001,"15min":1.444},"min":{"1min":0.38100000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.2929999999999999,"5min":12.731999999999999,"15min":37.585000000000001},"last":0.623},{"interface":"front","average":{"1min":0.64100000000000001,"5min":1.778,"15min":1.5960000000000001},"min":{"1min":0.40600000000000003,"5min":0.40200000000000002,"15min":0.312},"max":{"1min":1.323,"5min":12.779,"15min":37.918999999999997},"last":0.65300000000000002}]},{"osd":9,"last update":"Fri Feb 9 17:37:34 2024","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":1.171,"15min":1.159},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.29699999999999999},"max":{"1min":3.359,"5min":9.4900000000000002,"15min":9.4900000000000002},"last":0.497},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.298,"15min":1.224},"min":{"1min":0.377,"5min":0.36599999999999999,"15min":0.32000000000000001},"max":{"1min":3.3140000000000001,"5min":17.434999999999999,"15min":17.434999999999999},"last":0.58099999999999996}]},{"osd":10,"last update":"Fri Feb 9 17:37:34 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":1.0129999999999999,"15min":1.1579999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33700000000000002},"max":{"1min":3.1160000000000001,"5min":10.65,"15min":10.65},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.74099999999999999,"5min":1.137,"15min":1.0569999999999999},"min":{"1min":0.45400000000000001,"5min":0.318,"15min":0.318},"max":{"1min":1.611,"5min":8.2799999999999994,"15min":8.2799999999999994},"last":0.55300000000000005}]},{"osd":11,"last update":"Fri Feb 9 17:37:34 2024","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.464,"15min":1.3640000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":3.9940000000000002,"5min":14.433999999999999,"15min":14.433999999999999},"last":2.8079999999999998},{"interface":"front","average":{"1min":1.093,"5min":1.335,"15min":1.4199999999999999},"min":{"1min":0.47899999999999998,"5min":0.45000000000000001,"15min":0.38100000000000001},"max":{"1min":4.3600000000000003,"5min":11.622,"15min":11.622},"last":2.79}]}]},{"osd":7,"up_from":57,"seq":244813136078,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1534348,"kb_used_data":1214404,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":92210804,"statfs":{"total":95995035648,"available":94423863296,"internally_reserved":0,"allocated":1243549696,"data_stored":1243250385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":327612874},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:37:04 2024","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":2.6120000000000001,"15min":3.294},"min":{"1min":0.26200000000000001,"5min":0.254,"15min":0.254},"max":{"1min":4.1719999999999997,"5min":36.753,"15min":36.753},"last":0.80700000000000005},{"interface":"front","average":{"1min":1.913,"5min":2.9369999999999998,"15min":3.5489999999999999},"min":{"1min":0.29199999999999998,"5min":0.253,"15min":0.253},"max":{"1min":9.0640000000000001,"5min":36.918999999999997,"15min":36.918999999999997},"last":0.68700000000000006}]},{"osd":1,"last update":"Fri Feb 9 17:37:08 2024","interfaces":[{"interface":"back","average":{"1min":1.3600000000000001,"5min":1.9610000000000001,"15min":1.9350000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":6.3920000000000003,"5min":20.553999999999998,"15min":20.553999999999998},"last":2.1349999999999998},{"interface":"front","average":{"1min":2.6219999999999999,"5min":2.0099999999999998,"15min":1.657},"min":{"1min":0.28399999999999997,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":8.6430000000000007,"5min":20.568000000000001,"15min":20.568000000000001},"last":3.6549999999999998}]},{"osd":2,"last update":"Fri Feb 9 17:37:31 2024","interfaces":[{"interface":"back","average":{"1min":2.0249999999999999,"5min":1.8149999999999999,"15min":1.55},"min":{"1min":0.34100000000000003,"5min":0.28499999999999998,"15min":0.26000000000000001},"max":{"1min":7.4109999999999996,"5min":10.622,"15min":11.324},"last":0.50600000000000001},{"interface":"front","average":{"1min":2.0569999999999999,"5min":1.9670000000000001,"15min":1.629},"min":{"1min":0.41599999999999998,"5min":0.32100000000000001,"15min":0.28299999999999997},"max":{"1min":7.3819999999999997,"5min":19.361999999999998,"15min":19.361999999999998},"last":0.55300000000000005}]},{"osd":3,"last update":"Fri Feb 9 17:37:31 2024","interfaces":[{"interface":"back","average":{"1min":1.966,"5min":2.2210000000000001,"15min":1.643},"min":{"1min":0.313,"5min":0.311,"15min":0.26900000000000002},"max":{"1min":6.923,"5min":20.754000000000001,"15min":27.719000000000001},"last":0.73999999999999999},{"interface":"front","average":{"1min":2.294,"5min":2.173,"15min":1.6870000000000001},"min":{"1min":0.308,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":7.0599999999999996,"5min":24.457000000000001,"15min":24.457000000000001},"last":0.85499999999999998}]},{"osd":4,"last update":"Fri Feb 9 17:37:31 2024","interfaces":[{"interface":"back","average":{"1min":1.585,"5min":2.3119999999999998,"15min":1.355},"min":{"1min":0.31,"5min":0.25700000000000001,"15min":0.248},"max":{"1min":5.9950000000000001,"5min":14.372999999999999,"15min":14.372999999999999},"last":0.28999999999999998},{"interface":"front","average":{"1min":1.6399999999999999,"5min":2.3900000000000001,"15min":1.3640000000000001},"min":{"1min":0.35199999999999998,"5min":0.34699999999999998,"15min":0.22800000000000001},"max":{"1min":6.0220000000000002,"5min":14.401,"15min":14.401},"last":0.371}]},{"osd":5,"last update":"Fri Feb 9 17:35:28 2024","interfaces":[{"interface":"back","average":{"1min":2.073,"5min":2.2999999999999998,"15min":1.1619999999999999},"min":{"1min":0.36899999999999999,"5min":0.28599999999999998,"15min":0.21299999999999999},"max":{"1min":13.409000000000001,"5min":13.409000000000001,"15min":13.409000000000001},"last":0.53300000000000003},{"interface":"front","average":{"1min":1.946,"5min":2.1230000000000002,"15min":1.1240000000000001},"min":{"1min":0.30599999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":10.553000000000001,"5min":11.598000000000001,"15min":11.598000000000001},"last":0.48999999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:37:31 2024","interfaces":[{"interface":"back","average":{"1min":2.0070000000000001,"5min":2.3410000000000002,"15min":1.4630000000000001},"min":{"1min":0.29599999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":8.1929999999999996,"5min":25.965,"15min":25.965},"last":0.72899999999999998},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.784,"15min":1.218},"min":{"1min":0.32600000000000001,"5min":0.29799999999999999,"15min":0.28699999999999998},"max":{"1min":4.875,"5min":9.4090000000000007,"15min":9.4090000000000007},"last":0.89600000000000002}]},{"osd":8,"last update":"Fri Feb 9 17:37:31 2024","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.6479999999999999,"15min":1.48},"min":{"1min":0.40600000000000003,"5min":0.36599999999999999,"15min":0.32300000000000001},"max":{"1min":4.4340000000000002,"5min":9.0809999999999995,"15min":9.0809999999999995},"last":0.42899999999999999},{"interface":"front","average":{"1min":1.149,"5min":1.51,"15min":1.266},"min":{"1min":0.39300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":5.0679999999999996,"5min":9.2569999999999997,"15min":11.481},"last":0.77600000000000002}]},{"osd":9,"last update":"Fri Feb 9 17:37:17 2024","interfaces":[{"interface":"back","average":{"1min":1.7410000000000001,"5min":1.9359999999999999,"15min":1.4590000000000001},"min":{"1min":0.50600000000000001,"5min":0.40899999999999997,"15min":0.39600000000000002},"max":{"1min":4.6820000000000004,"5min":8.9900000000000002,"15min":41.286999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":1.671,"5min":1.9419999999999999,"15min":1.4950000000000001},"min":{"1min":0.49199999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":5.7240000000000002,"5min":11.417999999999999,"15min":41.234999999999999},"last":0.70599999999999996}]},{"osd":10,"last update":"Fri Feb 9 17:37:45 2024","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.641,"15min":1.333},"min":{"1min":0.41199999999999998,"5min":0.39100000000000001,"15min":0.35899999999999999},"max":{"1min":4.8490000000000002,"5min":9.4830000000000005,"15min":9.4830000000000005},"last":0.93000000000000005},{"interface":"front","average":{"1min":1.516,"5min":1.992,"15min":1.431},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.35599999999999998},"max":{"1min":5.641,"5min":9.109,"15min":9.109},"last":0.63400000000000001}]},{"osd":11,"last update":"Fri Feb 9 17:37:08 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":2.2519999999999998,"15min":1.5169999999999999},"min":{"1min":0.44500000000000001,"5min":0.41299999999999998,"15min":0.40999999999999998},"max":{"1min":6.6360000000000001,"5min":12.885999999999999,"15min":12.885999999999999},"last":0.95599999999999996},{"interface":"front","average":{"1min":1.444,"5min":1.883,"15min":1.3620000000000001},"min":{"1min":0.5,"5min":0.41299999999999998,"15min":0.39500000000000002},"max":{"1min":5.6230000000000002,"5min":9.9489999999999998,"15min":9.9489999999999998},"last":0.66700000000000004}]}]},{"osd":6,"up_from":49,"seq":210453397715,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2721932,"kb_used_data":2429764,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":91023220,"statfs":{"total":95995035648,"available":93207777280,"internally_reserved":0,"allocated":2488078336,"data_stored":2487779025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":299170246},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:37:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.6020000000000001,"15min":1.369},"min":{"1min":0.26300000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.4510000000000001,"5min":12.544,"15min":24.544},"last":0.27000000000000002},{"interface":"front","average":{"1min":1.296,"5min":1.6060000000000001,"15min":1.302},"min":{"1min":0.35499999999999998,"5min":0.26500000000000001,"15min":0.217},"max":{"1min":4.4950000000000001,"5min":8.3689999999999998,"15min":8.3689999999999998},"last":0.378}]},{"osd":1,"last update":"Fri Feb 9 17:37:04 2024","interfaces":[{"interface":"back","average":{"1min":1.6950000000000001,"5min":2.2029999999999998,"15min":1.579},"min":{"1min":0.30399999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":6.8380000000000001,"5min":17.762,"15min":23.954000000000001},"last":0.48999999999999999},{"interface":"front","average":{"1min":1.7250000000000001,"5min":2.2639999999999998,"15min":1.625},"min":{"1min":0.34499999999999997,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":6.8929999999999998,"5min":17.734999999999999,"15min":23.984999999999999},"last":0.51200000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:37:04 2024","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.623,"15min":1.462},"min":{"1min":0.34000000000000002,"5min":0.28999999999999998,"15min":0.22600000000000001},"max":{"1min":4.0830000000000002,"5min":8.0239999999999991,"15min":24.329999999999998},"last":0.42899999999999999},{"interface":"front","average":{"1min":1.268,"5min":1.6040000000000001,"15min":1.4690000000000001},"min":{"1min":0.377,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":4.1159999999999997,"5min":10.156000000000001,"15min":12.906000000000001},"last":0.47199999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:37:04 2024","interfaces":[{"interface":"back","average":{"1min":1.8069999999999999,"5min":2.2759999999999998,"15min":1.7490000000000001},"min":{"1min":0.30099999999999999,"5min":0.25600000000000001,"15min":0.248},"max":{"1min":8.7780000000000005,"5min":16.931999999999999,"15min":32.545000000000002},"last":3.915},{"interface":"front","average":{"1min":0.82299999999999995,"5min":2.2120000000000002,"15min":1.657},"min":{"1min":0.31900000000000001,"5min":0.29599999999999999,"15min":0.26600000000000001},"max":{"1min":4.2510000000000003,"5min":17.047000000000001,"15min":32.670999999999999},"last":0.82399999999999995}]},{"osd":4,"last update":"Fri Feb 9 17:37:04 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":2.2879999999999998,"15min":1.286},"min":{"1min":0.255,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":5.944,"5min":19.798999999999999,"15min":19.798999999999999},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.92500000000000004,"5min":1.8100000000000001,"15min":1.2230000000000001},"min":{"1min":0.377,"5min":0.28000000000000003,"15min":0.26000000000000001},"max":{"1min":5.9260000000000002,"5min":10.702999999999999,"15min":10.702999999999999},"last":0.53000000000000003}]},{"osd":5,"last update":"Fri Feb 9 17:37:04 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":1.909,"15min":1.198},"min":{"1min":0.318,"5min":0.29499999999999998,"15min":0.29199999999999998},"max":{"1min":2.2090000000000001,"5min":10.968999999999999,"15min":10.968999999999999},"last":0.502},{"interface":"front","average":{"1min":0.73699999999999999,"5min":2.105,"15min":1.26},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.27400000000000002},"max":{"1min":2.2370000000000001,"5min":13.91,"15min":13.91},"last":0.40699999999999997}]},{"osd":7,"last update":"Fri Feb 9 17:37:44 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":2.1040000000000001,"15min":1.3220000000000001},"min":{"1min":0.311,"5min":0.311,"15min":0.28000000000000003},"max":{"1min":4.8760000000000003,"5min":14.022,"15min":14.022},"last":0.55000000000000004},{"interface":"front","average":{"1min":1.038,"5min":2.2730000000000001,"15min":1.4299999999999999},"min":{"1min":0.36099999999999999,"5min":0.35699999999999998,"15min":0.309},"max":{"1min":4.4690000000000003,"5min":15.458,"15min":15.458},"last":0.57199999999999995}]},{"osd":8,"last update":"Fri Feb 9 17:26:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":0.876,"15min":0.79600000000000004},"min":{"1min":0.39900000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":4.0339999999999998,"5min":4.4880000000000004,"15min":4.4880000000000004},"last":4.3840000000000003},{"interface":"front","average":{"1min":1.0469999999999999,"5min":0.88600000000000001,"15min":0.78300000000000003},"min":{"1min":0.375,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":3.6539999999999999,"5min":4.8570000000000002,"15min":4.8570000000000002},"last":4.4210000000000003}]},{"osd":9,"last update":"Fri Feb 9 17:37:12 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":1.494,"15min":1.1930000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.1779999999999999,"5min":8.3409999999999993,"15min":10.803000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.70999999999999996,"5min":1.629,"15min":1.2909999999999999},"min":{"1min":0.40999999999999998,"5min":0.40400000000000003,"15min":0.40300000000000002},"max":{"1min":1.7190000000000001,"5min":8.1630000000000003,"15min":8.1630000000000003},"last":0.65100000000000002}]},{"osd":10,"last update":"Fri Feb 9 17:37:44 2024","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.611,"15min":1.349},"min":{"1min":0.40799999999999997,"5min":0.38500000000000001,"15min":0.35399999999999998},"max":{"1min":4.5119999999999996,"5min":7.734,"15min":9.5399999999999991},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.83599999999999997,"5min":1.8129999999999999,"15min":1.419},"min":{"1min":0.38,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":2.2349999999999999,"5min":13.836,"15min":13.836},"last":0.58799999999999997}]},{"osd":11,"last update":"Fri Feb 9 17:37:04 2024","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.774,"15min":1.5640000000000001},"min":{"1min":0.40799999999999997,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":4.1609999999999996,"5min":10.273999999999999,"15min":10.273999999999999},"last":1.522},{"interface":"front","average":{"1min":0.746,"5min":1.893,"15min":1.7070000000000001},"min":{"1min":0.45100000000000001,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.9279999999999999,"5min":8.2739999999999991,"15min":8.2739999999999991},"last":0.624}]}]},{"osd":1,"up_from":13,"seq":55834575085,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4047568,"kb_used_data":3665160,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":89697584,"statfs":{"total":95995035648,"available":91850326016,"internally_reserved":0,"allocated":3753123840,"data_stored":3752820457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,1,0,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:37:10 2024","interfaces":[{"interface":"back","average":{"1min":3.6230000000000002,"5min":1.581,"15min":1.4199999999999999},"min":{"1min":0.26300000000000001,"5min":0.223,"15min":0.214},"max":{"1min":14.917,"5min":14.917,"15min":14.917},"last":4.1829999999999998},{"interface":"front","average":{"1min":2.3279999999999998,"5min":1.4430000000000001,"15min":1.387},"min":{"1min":0.30099999999999999,"5min":0.245,"15min":0.214},"max":{"1min":8.7260000000000009,"5min":8.7260000000000009,"15min":9.8439999999999994},"last":0.439}]},{"osd":2,"last update":"Fri Feb 9 17:37:41 2024","interfaces":[{"interface":"back","average":{"1min":2.3519999999999999,"5min":1.708,"15min":1.6040000000000001},"min":{"1min":0.28899999999999998,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":9.5039999999999996,"5min":26.361999999999998,"15min":26.361999999999998},"last":3.9790000000000001},{"interface":"front","average":{"1min":4.1609999999999996,"5min":1.97,"15min":1.8440000000000001},"min":{"1min":0.375,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":11.346,"5min":26.087,"15min":26.087},"last":4.4870000000000001}]},{"osd":3,"last update":"Fri Feb 9 17:37:33 2024","interfaces":[{"interface":"back","average":{"1min":2.7770000000000001,"5min":1.575,"15min":1.375},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":8.4800000000000004,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":9.7140000000000004},{"interface":"front","average":{"1min":3.3100000000000001,"5min":1.74,"15min":1.478},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":14.68,"5min":14.68,"15min":14.68},"last":1.1839999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:37:10 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":1.6950000000000001,"15min":2.0289999999999999},"min":{"1min":0.317,"5min":0.253,"15min":0.253},"max":{"1min":8.3460000000000001,"5min":11.766999999999999,"15min":11.766999999999999},"last":4.5270000000000001},{"interface":"front","average":{"1min":2.3069999999999999,"5min":1.923,"15min":1.9319999999999999},"min":{"1min":0.309,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":10.512,"5min":10.512,"15min":10.512},"last":1.232}]},{"osd":5,"last update":"Fri Feb 9 17:36:50 2024","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.774,"15min":1.518},"min":{"1min":0.374,"5min":0.28599999999999998,"15min":0.27000000000000002},"max":{"1min":8.2859999999999996,"5min":13.544,"15min":13.544},"last":4.5789999999999997},{"interface":"front","average":{"1min":2.0089999999999999,"5min":1.839,"15min":1.353},"min":{"1min":0.36699999999999999,"5min":0.26800000000000002,"15min":0.26000000000000001},"max":{"1min":4.843,"5min":12.117000000000001,"15min":12.117000000000001},"last":1.282}]},{"osd":6,"last update":"Fri Feb 9 17:37:10 2024","interfaces":[{"interface":"back","average":{"1min":1.8959999999999999,"5min":1.732,"15min":1.5880000000000001},"min":{"1min":0.35999999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":8.25,"5min":8.7210000000000001,"15min":10.275},"last":4.5529999999999999},{"interface":"front","average":{"1min":2.0209999999999999,"5min":1.921,"15min":1.5860000000000001},"min":{"1min":0.41799999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":8.5999999999999996,"5min":9.0069999999999997,"15min":9.0069999999999997},"last":0.49299999999999999}]},{"osd":7,"last update":"Fri Feb 9 17:34:33 2024","interfaces":[{"interface":"back","average":{"1min":2.1560000000000001,"5min":1.9099999999999999,"15min":1.5609999999999999},"min":{"1min":0.35699999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":9.0489999999999995,"5min":15.148999999999999,"15min":15.148999999999999},"last":0.76600000000000001},{"interface":"front","average":{"1min":1.419,"5min":1.3740000000000001,"15min":1.425},"min":{"1min":0.31,"5min":0.31,"15min":0.28299999999999997},"max":{"1min":7.3380000000000001,"5min":13.834,"15min":13.834},"last":0.61499999999999999}]},{"osd":8,"last update":"Fri Feb 9 17:37:33 2024","interfaces":[{"interface":"back","average":{"1min":1.9419999999999999,"5min":1.4419999999999999,"15min":1.421},"min":{"1min":0.40999999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":10.486000000000001,"5min":10.486000000000001,"15min":10.486000000000001},"last":1.2649999999999999},{"interface":"front","average":{"1min":1.7210000000000001,"5min":1.6439999999999999,"15min":1.931},"min":{"1min":0.47899999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":5.2240000000000002,"5min":10.433999999999999,"15min":10.433999999999999},"last":4.6260000000000003}]},{"osd":9,"last update":"Fri Feb 9 17:37:19 2024","interfaces":[{"interface":"back","average":{"1min":2.0550000000000002,"5min":1.627,"15min":1.5549999999999999},"min":{"1min":0.45100000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":8.3170000000000002,"5min":9.5839999999999996,"15min":9.5839999999999996},"last":4.6020000000000003},{"interface":"front","average":{"1min":2.6859999999999999,"5min":1.5880000000000001,"15min":1.5820000000000001},"min":{"1min":0.38900000000000001,"5min":0.37,"15min":0.30099999999999999},"max":{"1min":11.366,"5min":11.366,"15min":12.117000000000001},"last":0.46999999999999997}]},{"osd":10,"last update":"Fri Feb 9 17:36:50 2024","interfaces":[{"interface":"back","average":{"1min":1.9099999999999999,"5min":1.425,"15min":1.321},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.35999999999999999},"max":{"1min":4.8810000000000002,"5min":7.9809999999999999,"15min":7.9809999999999999},"last":1.25},{"interface":"front","average":{"1min":1.958,"5min":1.512,"15min":1.581},"min":{"1min":0.48099999999999998,"5min":0.44400000000000001,"15min":0.33400000000000002},"max":{"1min":4.9080000000000004,"5min":8.0020000000000007,"15min":12.087999999999999},"last":1.2989999999999999}]},{"osd":11,"last update":"Fri Feb 9 17:36:56 2024","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":1.585,"15min":2.1200000000000001},"min":{"1min":0.437,"5min":0.42199999999999999,"15min":0.32000000000000001},"max":{"1min":5.4640000000000004,"5min":8.3079999999999998,"15min":12.169},"last":0.51800000000000002},{"interface":"front","average":{"1min":1.5509999999999999,"5min":1.619,"15min":2.0379999999999998},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.36599999999999999},"max":{"1min":5.4880000000000004,"5min":8.4339999999999993,"15min":12.144},"last":0.53900000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738610,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1548560,"kb_used_data":1202244,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":92196592,"statfs":{"total":95995035648,"available":94409310208,"internally_reserved":0,"allocated":1231097856,"data_stored":1230798545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 9 17:36:56 2024","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.1919999999999999,"15min":1.2809999999999999},"min":{"1min":0.22800000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":3.5030000000000001,"5min":11.236000000000001,"15min":14.166},"last":2.0099999999999998},{"interface":"front","average":{"1min":2.4870000000000001,"5min":1.3380000000000001,"15min":1.5209999999999999},"min":{"1min":0.32000000000000001,"5min":0.26800000000000002,"15min":0.215},"max":{"1min":7.8579999999999997,"5min":9.5649999999999995,"15min":20.177},"last":0.371}]},{"osd":2,"last update":"Fri Feb 9 17:27:44 2024","interfaces":[{"interface":"back","average":{"1min":3.976,"5min":3.976,"15min":3.976},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":13.066000000000001,"5min":13.066000000000001,"15min":13.066000000000001},"last":3.0569999999999999},{"interface":"front","average":{"1min":3.968,"5min":3.968,"15min":3.968},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":13.193,"5min":13.193,"15min":13.193},"last":10.872999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:37:50 2024","interfaces":[{"interface":"back","average":{"1min":1.8540000000000001,"5min":1.75,"15min":1.6399999999999999},"min":{"1min":0.219,"5min":0.218,"15min":0.218},"max":{"1min":8.7789999999999999,"5min":16.472000000000001,"15min":16.472000000000001},"last":0.35299999999999998},{"interface":"front","average":{"1min":2.8149999999999999,"5min":1.8049999999999999,"15min":1.556},"min":{"1min":0.35899999999999999,"5min":0.26900000000000002,"15min":0.223},"max":{"1min":8.2919999999999998,"5min":15.132999999999999,"15min":16.984999999999999},"last":8.2919999999999998}]},{"osd":4,"last update":"Fri Feb 9 17:37:23 2024","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.8520000000000001,"15min":1.532},"min":{"1min":0.33200000000000002,"5min":0.29999999999999999,"15min":0.253},"max":{"1min":2.823,"5min":14.419,"15min":14.419},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.86599999999999999,"5min":2.0369999999999999,"15min":1.627},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":6.0819999999999999,"5min":14.465999999999999,"15min":14.465999999999999},"last":1.3340000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:37:41 2024","interfaces":[{"interface":"back","average":{"1min":1.726,"5min":1.8049999999999999,"15min":1.575},"min":{"1min":0.37,"5min":0.32400000000000001,"15min":0.28999999999999998},"max":{"1min":7.8470000000000004,"5min":9.532,"15min":9.8569999999999993},"last":0.48899999999999999},{"interface":"front","average":{"1min":2.218,"5min":2.0489999999999999,"15min":1.7430000000000001},"min":{"1min":0.41599999999999998,"5min":0.32400000000000001,"15min":0.26700000000000002},"max":{"1min":7.8239999999999998,"5min":9.5640000000000001,"15min":9.9049999999999994},"last":0.51500000000000001}]},{"osd":6,"last update":"Fri Feb 9 17:37:13 2024","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.883,"15min":1.6240000000000001},"min":{"1min":0.34599999999999997,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":6.1619999999999999,"5min":13.102,"15min":13.102},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.304,"5min":1.956,"15min":1.625},"min":{"1min":0.379,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":6.1900000000000004,"5min":13.129,"15min":13.129},"last":1.3799999999999999}]},{"osd":7,"last update":"Fri Feb 9 17:37:34 2024","interfaces":[{"interface":"back","average":{"1min":1.841,"5min":2.2599999999999998,"15min":1.7390000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":6.2190000000000003,"5min":13.063000000000001,"15min":13.063000000000001},"last":1.4490000000000001},{"interface":"front","average":{"1min":2.4860000000000002,"5min":2.1139999999999999,"15min":1.7330000000000001},"min":{"1min":0.36399999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":8.6259999999999994,"5min":12.959,"15min":16.765999999999998},"last":0.47799999999999998}]},{"osd":8,"last update":"Fri Feb 9 17:37:36 2024","interfaces":[{"interface":"back","average":{"1min":2.0859999999999999,"5min":1.498,"15min":1.583},"min":{"1min":0.41799999999999998,"5min":0.36699999999999999,"15min":0.35799999999999998},"max":{"1min":8.7550000000000008,"5min":10.207000000000001,"15min":10.207000000000001},"last":0.623},{"interface":"front","average":{"1min":2.319,"5min":1.704,"15min":1.593},"min":{"1min":0.40999999999999998,"5min":0.38600000000000001,"15min":0.36799999999999999},"max":{"1min":8.7309999999999999,"5min":10.231999999999999,"15min":16.41},"last":0.65200000000000002}]},{"osd":9,"last update":"Fri Feb 9 17:37:23 2024","interfaces":[{"interface":"back","average":{"1min":1.284,"5min":1.5740000000000001,"15min":1.5069999999999999},"min":{"1min":0.36399999999999999,"5min":0.34699999999999998,"15min":0.312},"max":{"1min":6.1280000000000001,"5min":13.02,"15min":13.02},"last":1.302},{"interface":"front","average":{"1min":2.3319999999999999,"5min":1.76,"15min":1.536},"min":{"1min":0.42299999999999999,"5min":0.373,"15min":0.36199999999999999},"max":{"1min":11.295999999999999,"5min":11.295999999999999,"15min":16.82},"last":0.50600000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:37:41 2024","interfaces":[{"interface":"back","average":{"1min":2.4950000000000001,"5min":1.726,"15min":1.6220000000000001},"min":{"1min":0.47299999999999998,"5min":0.42199999999999999,"15min":0.38500000000000001},"max":{"1min":8.7040000000000006,"5min":13.888,"15min":16.876000000000001},"last":0.52900000000000003},{"interface":"front","average":{"1min":2.355,"5min":1.635,"15min":1.542},"min":{"1min":0.47799999999999998,"5min":0.45700000000000002,"15min":0.39800000000000002},"max":{"1min":11.384,"5min":11.384,"15min":11.384},"last":0.54700000000000004}]},{"osd":11,"last update":"Fri Feb 9 17:37:13 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":2.1970000000000001,"15min":1.665},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.42299999999999999},"max":{"1min":3.5510000000000002,"5min":40.021000000000001,"15min":40.021000000000001},"last":0.57799999999999996},{"interface":"front","average":{"1min":1.6870000000000001,"5min":2.1739999999999999,"15min":1.7589999999999999},"min":{"1min":0.50600000000000001,"5min":0.46000000000000002,"15min":0.443},"max":{"1min":3.6200000000000001,"5min":40.055999999999997,"15min":40.055999999999997},"last":0.63500000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411559,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":979548,"kb_used_data":813708,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":92765604,"statfs":{"total":95995035648,"available":94991978496,"internally_reserved":0,"allocated":833236992,"data_stored":832934129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":169802181},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":1.4510000000000001,"15min":1.006},"min":{"1min":0.29999999999999999,"5min":0.23699999999999999,"15min":0.221},"max":{"1min":4.9089999999999998,"5min":17.251000000000001,"15min":17.251000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.74399999999999999,"5min":1.611,"15min":1.113},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.22900000000000001},"max":{"1min":4.7720000000000002,"5min":23.097999999999999,"15min":23.097999999999999},"last":0.875}]},{"osd":1,"last update":"Fri Feb 9 17:37:30 2024","interfaces":[{"interface":"back","average":{"1min":2.1520000000000001,"5min":1.6279999999999999,"15min":1.538},"min":{"1min":0.23899999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":8.3780000000000001,"5min":22.306000000000001,"15min":27.289000000000001},"last":3.7330000000000001},{"interface":"front","average":{"1min":2.1000000000000001,"5min":1.839,"15min":1.6879999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":6.5659999999999998,"5min":44.838999999999999,"15min":44.838999999999999},"last":2.6139999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:37:05 2024","interfaces":[{"interface":"back","average":{"1min":2.2599999999999998,"5min":1.659,"15min":1.653},"min":{"1min":0.27300000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":9.5630000000000006,"5min":24.353000000000002,"15min":37.634999999999998},"last":2.843},{"interface":"front","average":{"1min":2.984,"5min":1.5369999999999999,"15min":1.484},"min":{"1min":0.27200000000000002,"5min":0.251,"15min":0.23100000000000001},"max":{"1min":10.801,"5min":10.801,"15min":38.924999999999997},"last":13.640000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:37:05 2024","interfaces":[{"interface":"back","average":{"1min":2.6709999999999998,"5min":1.9570000000000001,"15min":1.54},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.254},"max":{"1min":10.741,"5min":10.741,"15min":10.741},"last":13.526},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.7789999999999999,"15min":1.3600000000000001},"min":{"1min":0.29099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.359,"5min":9.6509999999999998,"15min":13.305999999999999},"last":2.7080000000000002}]},{"osd":5,"last update":"Fri Feb 9 17:37:45 2024","interfaces":[{"interface":"back","average":{"1min":2.7189999999999999,"5min":1.756,"15min":1.5660000000000001},"min":{"1min":0.35999999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":7.0410000000000004,"5min":7.7539999999999996,"15min":8.6859999999999999},"last":2.7469999999999999},{"interface":"front","average":{"1min":2.7509999999999999,"5min":2.2839999999999998,"15min":1.5589999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.29899999999999999},"max":{"1min":13.571999999999999,"5min":13.571999999999999,"15min":13.571999999999999},"last":13.571999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:37:05 2024","interfaces":[{"interface":"back","average":{"1min":1.6419999999999999,"5min":1.744,"15min":1.522},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.32100000000000001},"max":{"1min":6.4450000000000003,"5min":6.931,"15min":8.8399999999999999},"last":2.6739999999999999},{"interface":"front","average":{"1min":1.5329999999999999,"5min":2.0409999999999999,"15min":1.637},"min":{"1min":0.433,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":6.5380000000000003,"5min":11.269,"15min":11.269},"last":2.778}]},{"osd":7,"last update":"Fri Feb 9 17:37:30 2024","interfaces":[{"interface":"back","average":{"1min":2.3420000000000001,"5min":2.0640000000000001,"15min":1.6100000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.34599999999999997},"max":{"1min":6.5960000000000001,"5min":11.365,"15min":14.541},"last":3.1299999999999999},{"interface":"front","average":{"1min":1.403,"5min":2.0329999999999999,"15min":1.7669999999999999},"min":{"1min":0.437,"5min":0.371,"15min":0.371},"max":{"1min":4.3810000000000002,"5min":22.359999999999999,"15min":22.359999999999999},"last":0.33700000000000002}]},{"osd":8,"last update":"Fri Feb 9 17:37:05 2024","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.3380000000000001,"15min":1.2030000000000001},"min":{"1min":0.41199999999999998,"5min":0.38,"15min":0.38},"max":{"1min":3.8919999999999999,"5min":22.393999999999998,"15min":22.393999999999998},"last":0.377},{"interface":"front","average":{"1min":1.8080000000000001,"5min":1.413,"15min":1.4430000000000001},"min":{"1min":0.36399999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":6.508,"5min":9.9260000000000002,"15min":9.9260000000000002},"last":2.8170000000000002}]},{"osd":9,"last update":"Fri Feb 9 17:37:16 2024","interfaces":[{"interface":"back","average":{"1min":3.2709999999999999,"5min":1.631,"15min":1.5009999999999999},"min":{"1min":0.39500000000000002,"5min":0.371,"15min":0.371},"max":{"1min":10.819000000000001,"5min":10.819000000000001,"15min":10.819000000000001},"last":13.5},{"interface":"front","average":{"1min":3.2639999999999998,"5min":1.7849999999999999,"15min":1.4319999999999999},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":10.839,"5min":10.839,"15min":13.218},"last":13.589}]},{"osd":10,"last update":"Fri Feb 9 17:36:54 2024","interfaces":[{"interface":"back","average":{"1min":1.984,"5min":1.579,"15min":1.333},"min":{"1min":0.501,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":8.2609999999999992,"5min":22.425999999999998,"15min":22.425999999999998},"last":3.9729999999999999},{"interface":"front","average":{"1min":2.0209999999999999,"5min":1.617,"15min":1.4019999999999999},"min":{"1min":0.54800000000000004,"5min":0.47099999999999997,"15min":0.42099999999999999},"max":{"1min":8.2970000000000006,"5min":22.454999999999998,"15min":22.454999999999998},"last":4.0049999999999999}]},{"osd":11,"last update":"Fri Feb 9 17:37:05 2024","interfaces":[{"interface":"back","average":{"1min":1.798,"5min":1.522,"15min":1.7849999999999999},"min":{"1min":0.39600000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":6.4790000000000001,"5min":6.7709999999999999,"15min":13.186},"last":2.8690000000000002},{"interface":"front","average":{"1min":2.8490000000000002,"5min":1.863,"15min":1.8740000000000001},"min":{"1min":0.45900000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":10.856999999999999,"5min":10.856999999999999,"15min":10.856999999999999},"last":13.552}]}]},{"osd":3,"up_from":27,"seq":115964117220,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2364444,"kb_used_data":2018060,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":91380708,"statfs":{"total":95995035648,"available":93573844992,"internally_reserved":0,"allocated":2066493440,"data_stored":2066190577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":354613703},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:37:00 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":1.3999999999999999,"15min":1.6459999999999999},"min":{"1min":0.22600000000000001,"5min":0.224,"15min":0.224},"max":{"1min":2.4380000000000002,"5min":7.5110000000000001,"15min":7.5110000000000001},"last":3.524},{"interface":"front","average":{"1min":1.522,"5min":1.5580000000000001,"15min":1.736},"min":{"1min":0.24399999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":6.6269999999999998,"5min":15.167999999999999,"15min":15.167999999999999},"last":3.7349999999999999}]},{"osd":1,"last update":"Fri Feb 9 17:33:41 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":1.1399999999999999,"15min":1.3939999999999999},"min":{"1min":0.26100000000000001,"5min":0.23400000000000001,"15min":0.23300000000000001},"max":{"1min":3.8759999999999999,"5min":17.384,"15min":19.638999999999999},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":1.0249999999999999,"15min":1.115},"min":{"1min":0.27500000000000002,"5min":0.22500000000000001,"15min":0.20399999999999999},"max":{"1min":3.8980000000000001,"5min":12.664999999999999,"15min":14.661},"last":0.314}]},{"osd":2,"last update":"Fri Feb 9 17:36:50 2024","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.2849999999999999,"15min":1.4079999999999999},"min":{"1min":0.33500000000000002,"5min":0.22600000000000001,"15min":0.224},"max":{"1min":4.2670000000000003,"5min":14.888999999999999,"15min":14.914999999999999},"last":2.6000000000000001},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.583,"15min":1.4299999999999999},"min":{"1min":0.249,"5min":0.22500000000000001,"15min":0.218},"max":{"1min":6.9139999999999997,"5min":17.613,"15min":17.613},"last":2.6499999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:37:05 2024","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.9259999999999999,"15min":1.7050000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.254},"max":{"1min":5.5860000000000003,"5min":34.898000000000003,"15min":34.898000000000003},"last":2.355},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.756,"15min":1.5449999999999999},"min":{"1min":0.33800000000000002,"5min":0.32300000000000001,"15min":0.27900000000000003},"max":{"1min":4.1760000000000002,"5min":14.061999999999999,"15min":14.061999999999999},"last":2.2120000000000002}]},{"osd":5,"last update":"Fri Feb 9 17:37:38 2024","interfaces":[{"interface":"back","average":{"1min":1.881,"5min":1.871,"15min":1.399},"min":{"1min":0.29899999999999999,"5min":0.27800000000000002,"15min":0.27300000000000002},"max":{"1min":6.8689999999999998,"5min":17.651,"15min":17.651},"last":1.819},{"interface":"front","average":{"1min":1.9259999999999999,"5min":1.9139999999999999,"15min":1.4419999999999999},"min":{"1min":0.38,"5min":0.34100000000000003,"15min":0.33300000000000002},"max":{"1min":6.8929999999999998,"5min":17.675999999999998,"15min":17.675999999999998},"last":1.8580000000000001}]},{"osd":6,"last update":"Fri Feb 9 17:37:05 2024","interfaces":[{"interface":"back","average":{"1min":2.306,"5min":2.0110000000000001,"15min":1.5840000000000001},"min":{"1min":0.33200000000000002,"5min":0.32900000000000001,"15min":0.32200000000000001},"max":{"1min":10.487,"5min":20.262,"15min":20.262},"last":2.3199999999999998},{"interface":"front","average":{"1min":1.5980000000000001,"5min":1.6859999999999999,"15min":1.2769999999999999},"min":{"1min":0.372,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":5.157,"5min":7.6959999999999997,"15min":7.9009999999999998},"last":2.0070000000000001}]},{"osd":7,"last update":"Fri Feb 9 17:37:38 2024","interfaces":[{"interface":"back","average":{"1min":2.2949999999999999,"5min":2.2090000000000001,"15min":2.0249999999999999},"min":{"1min":0.34399999999999997,"5min":0.32200000000000001,"15min":0.27200000000000002},"max":{"1min":9.3290000000000006,"5min":21.018999999999998,"15min":21.018999999999998},"last":2.238},{"interface":"front","average":{"1min":1.7270000000000001,"5min":1.9430000000000001,"15min":1.6950000000000001},"min":{"1min":0.45500000000000002,"5min":0.372,"15min":0.37},"max":{"1min":5.9379999999999997,"5min":14.186,"15min":19.550999999999998},"last":2.1539999999999999}]},{"osd":8,"last update":"Fri Feb 9 17:37:05 2024","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.3069999999999999,"15min":1.3149999999999999},"min":{"1min":0.38500000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":4.3739999999999997,"5min":7.7569999999999997,"15min":7.7569999999999997},"last":3.4929999999999999},{"interface":"front","average":{"1min":1.448,"5min":1.1919999999999999,"15min":0.876},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":4.0599999999999996,"5min":14.129,"15min":14.129},"last":2.2909999999999999}]},{"osd":9,"last update":"Fri Feb 9 17:37:19 2024","interfaces":[{"interface":"back","average":{"1min":1.879,"5min":1.629,"15min":1.3320000000000001},"min":{"1min":0.42899999999999999,"5min":0.371,"15min":0.35499999999999998},"max":{"1min":6.5949999999999998,"5min":7.7370000000000001,"15min":7.7370000000000001},"last":3.464},{"interface":"front","average":{"1min":1.9970000000000001,"5min":1.502,"15min":1.5960000000000001},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":12.815,"5min":12.815,"15min":13.717000000000001},"last":1.952}]},{"osd":10,"last update":"Fri Feb 9 17:36:50 2024","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.355,"15min":1.431},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":2.1429999999999998,"5min":7.7889999999999997,"15min":19.611000000000001},"last":3.4300000000000002},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.401,"15min":1.3200000000000001},"min":{"1min":0.41999999999999998,"5min":0.38,"15min":0.38},"max":{"1min":6.8179999999999996,"5min":12.638,"15min":12.638},"last":2.1230000000000002}]},{"osd":11,"last update":"Fri Feb 9 17:37:05 2024","interfaces":[{"interface":"back","average":{"1min":1.6539999999999999,"5min":1.6930000000000001,"15min":1.3899999999999999},"min":{"1min":0.46300000000000002,"5min":0.42499999999999999,"15min":0.38800000000000001},"max":{"1min":6.8449999999999998,"5min":12.69,"15min":12.69},"last":1.9810000000000001},{"interface":"front","average":{"1min":1.6899999999999999,"5min":1.4159999999999999,"15min":1.625},"min":{"1min":0.45400000000000001,"5min":0.42499999999999999,"15min":0.36899999999999999},"max":{"1min":4.7729999999999997,"5min":14.157999999999999,"15min":14.157999999999999},"last":2.2650000000000001}]}]},{"osd":4,"up_from":32,"seq":137438953693,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1975832,"kb_used_data":1637832,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":91769320,"statfs":{"total":95995035648,"available":93971783680,"internally_reserved":0,"allocated":1677139968,"data_stored":1676836585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:37:02 2024","interfaces":[{"interface":"back","average":{"1min":1.954,"5min":2.6120000000000001,"15min":1.4870000000000001},"min":{"1min":0.35999999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":7.9669999999999996,"5min":50.356999999999999,"15min":50.356999999999999},"last":1.5660000000000001},{"interface":"front","average":{"1min":2.0800000000000001,"5min":2.3260000000000001,"15min":1.401},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":8.0440000000000005,"5min":50.569000000000003,"15min":50.569000000000003},"last":3.133}]},{"osd":1,"last update":"Fri Feb 9 17:37:13 2024","interfaces":[{"interface":"back","average":{"1min":3.3140000000000001,"5min":2.6619999999999999,"15min":1.853},"min":{"1min":0.27300000000000002,"5min":0.26100000000000001,"15min":0.246},"max":{"1min":28.143000000000001,"5min":50.473999999999997,"15min":50.473999999999997},"last":5.9850000000000003},{"interface":"front","average":{"1min":3.3559999999999999,"5min":2.8109999999999999,"15min":1.8939999999999999},"min":{"1min":0.32200000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":28.103000000000002,"5min":50.499000000000002,"15min":50.499000000000002},"last":6.0170000000000003}]},{"osd":2,"last update":"Fri Feb 9 17:28:51 2024","interfaces":[{"interface":"back","average":{"1min":2.536,"5min":1.546,"15min":0.91400000000000003},"min":{"1min":0.29999999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":8.8819999999999997,"5min":8.8819999999999997,"15min":8.8819999999999997},"last":0.622},{"interface":"front","average":{"1min":2.214,"5min":1.7330000000000001,"15min":0.97799999999999998},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":9.5039999999999996,"5min":9.5039999999999996,"15min":9.5039999999999996},"last":0.53800000000000003}]},{"osd":3,"last update":"Fri Feb 9 17:37:13 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":2.4769999999999999,"15min":1.8620000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":7.1820000000000004,"5min":50.462000000000003,"15min":50.462000000000003},"last":3.3279999999999998},{"interface":"front","average":{"1min":2.0630000000000002,"5min":2.2879999999999998,"15min":1.7110000000000001},"min":{"1min":0.29699999999999999,"5min":0.255,"15min":0.255},"max":{"1min":5.5869999999999997,"5min":50.529000000000003,"15min":50.529000000000003},"last":1.7589999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:37:35 2024","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.8340000000000001,"15min":1.2110000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.222},"max":{"1min":4.9619999999999997,"5min":9.5630000000000006,"15min":9.5630000000000006},"last":0.71399999999999997},{"interface":"front","average":{"1min":1.2490000000000001,"5min":2.4740000000000002,"15min":1.4970000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.29199999999999998},"max":{"1min":4.9969999999999999,"5min":50.509999999999998,"15min":50.509999999999998},"last":0.73799999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:37:13 2024","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.76,"15min":1.2789999999999999},"min":{"1min":0.26600000000000001,"5min":0.26200000000000001,"15min":0.247},"max":{"1min":7.0419999999999998,"5min":11.742000000000001,"15min":18.09},"last":2.7679999999999998},{"interface":"front","average":{"1min":0.999,"5min":2.4129999999999998,"15min":1.4510000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.28000000000000003},"max":{"1min":3.8900000000000001,"5min":50.402999999999999,"15min":50.402999999999999},"last":3.5489999999999999}]},{"osd":7,"last update":"Fri Feb 9 17:37:35 2024","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":2.1099999999999999,"15min":1.47},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.28399999999999997},"max":{"1min":4.0620000000000003,"5min":14.919,"15min":15.223000000000001},"last":0.68999999999999995},{"interface":"front","average":{"1min":1.4370000000000001,"5min":2.0089999999999999,"15min":1.466},"min":{"1min":0.36499999999999999,"5min":0.317,"15min":0.28100000000000003},"max":{"1min":6.7640000000000002,"5min":9.343,"15min":19.047000000000001},"last":1.524}]},{"osd":8,"last update":"Fri Feb 9 17:37:35 2024","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":1.7589999999999999,"15min":1.4430000000000001},"min":{"1min":0.35399999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":3.5019999999999998,"5min":8.0050000000000008,"15min":8.4160000000000004},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.94299999999999995,"5min":2.3079999999999998,"15min":1.52},"min":{"1min":0.40500000000000003,"5min":0.34699999999999998,"15min":0.308},"max":{"1min":3.569,"5min":50.594000000000001,"15min":50.594000000000001},"last":0.63300000000000001}]},{"osd":9,"last update":"Fri Feb 9 17:37:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":2.4969999999999999,"15min":1.5589999999999999},"min":{"1min":0.44400000000000001,"5min":0.39200000000000002,"15min":0.34599999999999997},"max":{"1min":5.157,"5min":50.442999999999998,"15min":50.442999999999998},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.86299999999999999,"5min":1.728,"15min":1.3100000000000001},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":3.2669999999999999,"5min":9.9770000000000003,"15min":10.555},"last":1.5920000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:36:53 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":2.6680000000000001,"15min":1.649},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.378},"max":{"1min":3.137,"5min":51.223999999999997,"15min":51.223999999999997},"last":0.54300000000000004},{"interface":"front","average":{"1min":1.294,"5min":2.9430000000000001,"15min":1.7130000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.38100000000000001},"max":{"1min":5.117,"5min":50.552,"15min":50.552},"last":0.76400000000000001}]},{"osd":11,"last update":"Fri Feb 9 17:37:18 2024","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":2.8490000000000002,"15min":1.881},"min":{"1min":0.495,"5min":0.443,"15min":0.40500000000000003},"max":{"1min":3.6619999999999999,"5min":50.484999999999999,"15min":50.484999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":1.022,"5min":2.0939999999999999,"15min":1.5409999999999999},"min":{"1min":0.435,"5min":0.42499999999999999,"15min":0.34899999999999998},"max":{"1min":3.4300000000000002,"5min":18.574999999999999,"15min":18.574999999999999},"last":1.619}]}]},{"osd":5,"up_from":40,"seq":171798692056,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":965076,"kb_used_data":789516,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":92780076,"statfs":{"total":95995035648,"available":95006797824,"internally_reserved":0,"allocated":808464384,"data_stored":808161521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:32:25 2024","interfaces":[{"interface":"back","average":{"1min":1.8540000000000001,"5min":1.653,"15min":1.0900000000000001},"min":{"1min":0.318,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":7.7160000000000002,"5min":8.7690000000000001,"15min":11.923},"last":3.8450000000000002},{"interface":"front","average":{"1min":2.4089999999999998,"5min":1.8260000000000001,"15min":1.032},"min":{"1min":0.34499999999999997,"5min":0.24299999999999999,"15min":0.224},"max":{"1min":14.082000000000001,"5min":14.082000000000001,"15min":14.082000000000001},"last":0.79400000000000004}]},{"osd":1,"last update":"Fri Feb 9 17:37:33 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.833,"15min":1.8069999999999999},"min":{"1min":0.26000000000000001,"5min":0.23899999999999999,"15min":0.23200000000000001},"max":{"1min":4.1189999999999998,"5min":11.208,"15min":29.715},"last":0.34200000000000003},{"interface":"front","average":{"1min":1.5,"5min":1.8680000000000001,"15min":1.657},"min":{"1min":0.26300000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":5.1639999999999997,"5min":11.148999999999999,"15min":29.969999999999999},"last":0.28599999999999998}]},{"osd":2,"last update":"Fri Feb 9 17:37:33 2024","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":1.4950000000000001,"15min":1.4770000000000001},"min":{"1min":0.34499999999999997,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":5.21,"5min":9.2690000000000001,"15min":10.5},"last":0.38},{"interface":"front","average":{"1min":1.327,"5min":1.647,"15min":1.591},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":3.875,"5min":10.680999999999999,"15min":14.122},"last":0.46300000000000002}]},{"osd":3,"last update":"Fri Feb 9 17:37:33 2024","interfaces":[{"interface":"back","average":{"1min":1.73,"5min":2.694,"15min":1.915},"min":{"1min":0.29799999999999999,"5min":0.26800000000000002,"15min":0.26500000000000001},"max":{"1min":5.2430000000000003,"5min":64.150999999999996,"15min":64.150999999999996},"last":4.5410000000000004},{"interface":"front","average":{"1min":1.2070000000000001,"5min":2.6299999999999999,"15min":1.845},"min":{"1min":0.40699999999999997,"5min":0.32800000000000001,"15min":0.28000000000000003},"max":{"1min":3.9870000000000001,"5min":64.176000000000002,"15min":64.176000000000002},"last":3.2090000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:37:44 2024","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.704,"15min":1.1850000000000001},"min":{"1min":0.28100000000000003,"5min":0.26200000000000001,"15min":0.23400000000000001},"max":{"1min":8.9239999999999995,"5min":11.036,"15min":11.036},"last":4.4820000000000002},{"interface":"front","average":{"1min":1.0269999999999999,"5min":1.6950000000000001,"15min":1.236},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":6.1779999999999999,"5min":13.59,"15min":13.59},"last":1.671}]},{"osd":6,"last update":"Fri Feb 9 17:37:09 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":1.6919999999999999,"15min":1.258},"min":{"1min":0.30499999999999999,"5min":0.253,"15min":0.253},"max":{"1min":2.964,"5min":15.375999999999999,"15min":15.375999999999999},"last":8.609},{"interface":"front","average":{"1min":1.093,"5min":1.9690000000000001,"15min":1.3280000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.27200000000000002},"max":{"1min":5.4690000000000003,"5min":14.250999999999999,"15min":14.250999999999999},"last":4.4480000000000004}]},{"osd":7,"last update":"Fri Feb 9 17:37:33 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.6439999999999999,"15min":1.29},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":7.8209999999999997,"5min":9.7140000000000004,"15min":18.626999999999999},"last":1.6379999999999999},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.7809999999999999,"15min":1.494},"min":{"1min":0.38800000000000001,"5min":0.34200000000000003,"15min":0.28799999999999998},"max":{"1min":4.0140000000000002,"5min":10.753,"15min":19.044},"last":0.54000000000000004}]},{"osd":8,"last update":"Fri Feb 9 17:37:09 2024","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.4530000000000001,"15min":1.6870000000000001},"min":{"1min":0.42399999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":5.2960000000000003,"5min":9.3710000000000004,"15min":9.3710000000000004},"last":4.4130000000000003},{"interface":"front","average":{"1min":0.81000000000000005,"5min":1.7170000000000001,"15min":2.1219999999999999},"min":{"1min":0.53500000000000003,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":2.5640000000000001,"5min":9.6240000000000006,"15min":9.6240000000000006},"last":0.59099999999999997}]},{"osd":9,"last update":"Fri Feb 9 17:37:14 2024","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.6240000000000001,"15min":1.278},"min":{"1min":0.41799999999999998,"5min":0.378,"15min":0.35999999999999999},"max":{"1min":6.5780000000000003,"5min":14.98,"15min":14.98},"last":4.367},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.716,"15min":1.3},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.312},"max":{"1min":7.7779999999999996,"5min":9.4239999999999995,"15min":9.4239999999999995},"last":1.536}]},{"osd":10,"last update":"Fri Feb 9 17:37:44 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":1.653,"15min":1.4330000000000001},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.33800000000000002},"max":{"1min":4.04,"5min":10.816000000000001,"15min":14.218},"last":0.63200000000000001},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.73,"15min":1.5049999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.36099999999999999},"max":{"1min":5.2800000000000002,"5min":8.2850000000000001,"15min":14.186999999999999},"last":4.5129999999999999}]},{"osd":11,"last update":"Fri Feb 9 17:37:14 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":1.881,"15min":1.486},"min":{"1min":0.53600000000000003,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":4.0940000000000003,"5min":9.4239999999999995,"15min":9.4239999999999995},"last":1.71},{"interface":"front","average":{"1min":1.377,"5min":2.3090000000000002,"15min":1.48},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.38900000000000001},"max":{"1min":7.734,"5min":14.316000000000001,"15min":14.316000000000001},"last":1.6859999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1220542464,"data_stored":1220542464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3737063424,"data_stored":3737059352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":828309504,"data_stored":828309504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2052194304,"data_stored":2052194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1671499776,"data_stored":1671495704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":806092800,"data_stored":806092800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2482110464,"data_stored":2482110464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1237909504,"data_stored":1237909504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1236340736,"data_stored":1236336664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1614217216,"data_stored":1614217216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2908487680,"data_stored":2908487680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-09T17:37:58.116 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd pool set .mgr pgp_num 21 2024-02-09T17:37:58.159 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:57 smithi099 ceph-mon[39420]: pgmap v1001: 37 pgs: 37 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s 2024-02-09T17:37:58.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:57 smithi179 ceph-mon[43975]: pgmap v1001: 37 pgs: 37 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s 2024-02-09T17:37:58.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:57 smithi162 ceph-mon[44304]: pgmap v1001: 37 pgs: 37 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s 2024-02-09T17:37:58.576 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:37:58.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:58.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:37:58.592 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:37:58.690 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:37:58.724 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:37:58.750 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:37:58.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:37:58.756 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:37:58.857 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:37:58.912 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:00.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:59 smithi179 ceph-mon[43975]: pgmap v1002: 37 pgs: 37 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 881 op/s 2024-02-09T17:38:00.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:59 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:38:00.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:37:59 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:38:00.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:59 smithi162 ceph-mon[44304]: pgmap v1002: 37 pgs: 37 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 881 op/s 2024-02-09T17:38:00.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:59 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:38:00.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:37:59 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:38:00.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:59 smithi099 ceph-mon[39420]: pgmap v1002: 37 pgs: 37 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 881 op/s 2024-02-09T17:38:00.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:59 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:38:00.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:37:59 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:38:00.570 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:38:01.127 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:38:01.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:00 smithi099 ceph-mon[39420]: osdmap e366: 12 total, 12 up, 12 in 2024-02-09T17:38:01.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:00 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:38:01.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:00 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:38:01.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:00 smithi179 ceph-mon[43975]: osdmap e366: 12 total, 12 up, 12 in 2024-02-09T17:38:01.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:00 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:38:01.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:00 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:38:01.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:00 smithi162 ceph-mon[44304]: osdmap e366: 12 total, 12 up, 12 in 2024-02-09T17:38:01.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:00 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:38:01.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:00 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:38:01.582 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:38:01.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:01.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:38:01.694 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:38:01.741 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:01.769 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:38:01.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:01.772 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:38:01.873 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:38:01.900 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:01.921 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:38:01.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:01.924 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:38:02.025 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:38:02.084 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:01 smithi099 ceph-mon[39420]: pgmap v1004: 37 pgs: 37 active+clean; 5.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 872 op/s 2024-02-09T17:38:02.084 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:01 smithi099 ceph-mon[39420]: osdmap e367: 12 total, 12 up, 12 in 2024-02-09T17:38:02.085 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:02.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:01 smithi179 ceph-mon[43975]: pgmap v1004: 37 pgs: 37 active+clean; 5.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 872 op/s 2024-02-09T17:38:02.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:01 smithi179 ceph-mon[43975]: osdmap e367: 12 total, 12 up, 12 in 2024-02-09T17:38:02.259 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:38:02.259 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:38:02.277 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:38:02.281 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:38:02.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:02.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:38:02.388 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:38:02.422 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:02.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:01 smithi162 ceph-mon[44304]: pgmap v1004: 37 pgs: 37 active+clean; 5.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 872 op/s 2024-02-09T17:38:02.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:01 smithi162 ceph-mon[44304]: osdmap e367: 12 total, 12 up, 12 in 2024-02-09T17:38:02.452 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:38:02.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:02.460 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:38:02.561 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:38:02.598 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:02.623 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:38:02.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:02.626 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:38:02.727 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:38:02.802 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:02.996 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is unset 2024-02-09T17:38:03.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:02 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2249324600' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:38:03.373 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:38:03.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:03.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:38:03.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:02 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2249324600' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:38:03.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:02 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2249324600' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:38:03.484 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:38:03.525 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:04.093 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:38:04.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:04.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:38:04.105 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:03 smithi099 ceph-mon[39420]: pgmap v1006: 37 pgs: 37 active+clean; 5.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s 2024-02-09T17:38:04.105 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:03 smithi099 ceph-mon[39420]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:38:04.105 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:03 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2249324600' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:38:04.105 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:03 smithi099 ceph-mon[39420]: osdmap e368: 12 total, 12 up, 12 in 2024-02-09T17:38:04.201 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:38:04.245 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:03 smithi179 ceph-mon[43975]: pgmap v1006: 37 pgs: 37 active+clean; 5.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s 2024-02-09T17:38:04.245 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:03 smithi179 ceph-mon[43975]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:38:04.246 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:03 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2249324600' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:38:04.246 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:03 smithi179 ceph-mon[43975]: osdmap e368: 12 total, 12 up, 12 in 2024-02-09T17:38:04.251 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:04.278 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:38:04.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:04.282 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:38:04.383 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:38:04.422 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:04.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:03 smithi162 ceph-mon[44304]: pgmap v1006: 37 pgs: 37 active+clean; 5.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s 2024-02-09T17:38:04.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:03 smithi162 ceph-mon[44304]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:38:04.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:03 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2249324600' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:38:04.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:03 smithi162 ceph-mon[44304]: osdmap e368: 12 total, 12 up, 12 in 2024-02-09T17:38:04.459 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:38:04.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:04.466 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:38:04.567 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:38:04.603 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:04.630 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:38:04.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:04.636 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:38:04.737 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:38:04.783 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:05.448 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:38:05.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:05.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:38:05.555 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:38:05.624 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:06.089 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:38:06.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:06.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:38:06.196 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:38:06.230 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:05 smithi162 ceph-mon[44304]: pgmap v1008: 37 pgs: 37 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 739 op/s 2024-02-09T17:38:06.239 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:06.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:05 smithi099 ceph-mon[39420]: pgmap v1008: 37 pgs: 37 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 739 op/s 2024-02-09T17:38:06.266 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:38:06.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:06.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:38:06.370 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:38:06.413 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:06.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:05 smithi179 ceph-mon[43975]: pgmap v1008: 37 pgs: 37 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 739 op/s 2024-02-09T17:38:06.439 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:38:06.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:06.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:38:06.547 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:38:06.603 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:06.639 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:38:06.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:06.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:38:06.751 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:38:06.782 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:06.817 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:38:06.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:06.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:38:06.925 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:38:06.973 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:07.001 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:38:07.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:07.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:38:07.108 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:38:07.139 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:07.163 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:38:07.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:07.169 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:38:07.250 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:06 smithi099 ceph-mon[39420]: 1.13 scrub starts 2024-02-09T17:38:07.251 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:06 smithi099 ceph-mon[39420]: 1.13 scrub ok 2024-02-09T17:38:07.270 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:38:07.379 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:07.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:06 smithi179 ceph-mon[43975]: 1.13 scrub starts 2024-02-09T17:38:07.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:06 smithi179 ceph-mon[43975]: 1.13 scrub ok 2024-02-09T17:38:07.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:06 smithi162 ceph-mon[44304]: 1.13 scrub starts 2024-02-09T17:38:07.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:06 smithi162 ceph-mon[44304]: 1.13 scrub ok 2024-02-09T17:38:07.855 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:38:07.855 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:38:07.855 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:38:07.855 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:38:07.855 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:38:07.856 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:38:07.856 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:38:07.856 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:38:07.856 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:38:07.856 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:38:07.878 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:38:07.921 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:38:07.923 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:38:07.923 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:38:07.923 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:38:07.923 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:38:07.924 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:38:07.924 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:38:07.924 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:38:07.924 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:38:07.924 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:38:07.924 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:38:07.924 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:38:07.924 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:38:08.139 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-09T17:38:08.251 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:07 smithi099 ceph-mon[39420]: pgmap v1009: 37 pgs: 37 active+clean; 5.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s 2024-02-09T17:38:08.251 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:07 smithi099 ceph-mon[39420]: 1.11 scrub starts 2024-02-09T17:38:08.251 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:07 smithi099 ceph-mon[39420]: 1.10 scrub starts 2024-02-09T17:38:08.251 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:07 smithi099 ceph-mon[39420]: 1.10 scrub ok 2024-02-09T17:38:08.251 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:07 smithi099 ceph-mon[39420]: 1.11 scrub ok 2024-02-09T17:38:08.251 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:07 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/263463903' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:38:08.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:07 smithi179 ceph-mon[43975]: pgmap v1009: 37 pgs: 37 active+clean; 5.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s 2024-02-09T17:38:08.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:07 smithi179 ceph-mon[43975]: 1.11 scrub starts 2024-02-09T17:38:08.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:07 smithi179 ceph-mon[43975]: 1.10 scrub starts 2024-02-09T17:38:08.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:07 smithi179 ceph-mon[43975]: 1.10 scrub ok 2024-02-09T17:38:08.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:07 smithi179 ceph-mon[43975]: 1.11 scrub ok 2024-02-09T17:38:08.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:07 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/263463903' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:38:08.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:07 smithi162 ceph-mon[44304]: pgmap v1009: 37 pgs: 37 active+clean; 5.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s 2024-02-09T17:38:08.431 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:07 smithi162 ceph-mon[44304]: 1.11 scrub starts 2024-02-09T17:38:08.431 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:07 smithi162 ceph-mon[44304]: 1.10 scrub starts 2024-02-09T17:38:08.431 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:07 smithi162 ceph-mon[44304]: 1.10 scrub ok 2024-02-09T17:38:08.431 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:07 smithi162 ceph-mon[44304]: 1.11 scrub ok 2024-02-09T17:38:08.431 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:07 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/263463903' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:38:09.433 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-09T17:38:09.463 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:38:09.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:09.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:38:09.570 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:38:09.605 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:09.633 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:38:09.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:09.636 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:38:09.737 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:38:09.768 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:09.790 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:38:09.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:09.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:38:09.895 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:38:09.954 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:10.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:10 smithi162 ceph-mon[44304]: pgmap v1010: 37 pgs: 37 active+clean; 5.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 739 op/s 2024-02-09T17:38:10.207 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:38:10.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:10.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:38:10.315 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:38:10.348 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:10.372 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:38:10.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:10.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:38:10.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:10 smithi179 ceph-mon[43975]: pgmap v1010: 37 pgs: 37 active+clean; 5.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 739 op/s 2024-02-09T17:38:10.481 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:38:10.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:10 smithi099 ceph-mon[39420]: pgmap v1010: 37 pgs: 37 active+clean; 5.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 739 op/s 2024-02-09T17:38:10.503 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:38:10.506 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:10.515 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:38:10.524 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:38:10.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:10.527 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:38:10.628 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:38:10.658 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:10.684 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:38:10.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:10.692 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:38:10.793 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:38:10.822 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:10.849 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:38:10.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:10.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:38:10.956 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:38:11.002 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:11.027 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:38:11.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:11.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:38:11.056 INFO:teuthology.orchestra.run.smithi099.stderr:set pool 1 pgp_num to 21 2024-02-09T17:38:11.077 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:11 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3263970878' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-02-09T17:38:11.131 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:38:11.166 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:11 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3263970878' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-02-09T17:38:11.167 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:11.191 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:38:11.199 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:11.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:38:11.300 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:38:11.329 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:11.362 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:38:11.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:11.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:38:11.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:11 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3263970878' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-02-09T17:38:11.473 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:38:11.542 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:11.675 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:38:11.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:11.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:38:11.786 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:38:11.829 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:11.872 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:38:11.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:11.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:38:11.983 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:38:12.036 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:12.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:12 smithi162 ceph-mon[44304]: 1.f scrub starts 2024-02-09T17:38:12.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:12 smithi162 ceph-mon[44304]: 1.f scrub ok 2024-02-09T17:38:12.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:12 smithi162 ceph-mon[44304]: pgmap v1011: 37 pgs: 37 active+clean; 5.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 833 op/s 2024-02-09T17:38:12.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:12 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3263970878' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-02-09T17:38:12.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:12 smithi162 ceph-mon[44304]: osdmap e369: 12 total, 12 up, 12 in 2024-02-09T17:38:12.216 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:38:12.251 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:38:12.253 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:12.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:38:12.354 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:38:12.391 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:12 smithi179 ceph-mon[43975]: 1.f scrub starts 2024-02-09T17:38:12.392 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:12 smithi179 ceph-mon[43975]: 1.f scrub ok 2024-02-09T17:38:12.392 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:12 smithi179 ceph-mon[43975]: pgmap v1011: 37 pgs: 37 active+clean; 5.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 833 op/s 2024-02-09T17:38:12.392 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:12 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3263970878' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-02-09T17:38:12.392 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:12 smithi179 ceph-mon[43975]: osdmap e369: 12 total, 12 up, 12 in 2024-02-09T17:38:12.392 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:12.416 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:38:12.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:12.421 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:38:12.438 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:12 smithi099 ceph-mon[39420]: 1.f scrub starts 2024-02-09T17:38:12.438 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:12 smithi099 ceph-mon[39420]: 1.f scrub ok 2024-02-09T17:38:12.438 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:12 smithi099 ceph-mon[39420]: pgmap v1011: 37 pgs: 37 active+clean; 5.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 833 op/s 2024-02-09T17:38:12.438 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:12 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3263970878' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-02-09T17:38:12.438 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:12 smithi099 ceph-mon[39420]: osdmap e369: 12 total, 12 up, 12 in 2024-02-09T17:38:12.522 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:38:12.571 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:12.606 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:38:12.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:12.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:38:12.712 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:38:12.737 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:13.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:13 smithi179 ceph-mon[43975]: 1.c scrub starts 2024-02-09T17:38:13.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:13 smithi179 ceph-mon[43975]: 1.c scrub ok 2024-02-09T17:38:13.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:13 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:38:13.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:13 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:38:13.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:13 smithi179 ceph-mon[43975]: 1.5 scrub starts 2024-02-09T17:38:13.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:13 smithi179 ceph-mon[43975]: 1.5 scrub ok 2024-02-09T17:38:13.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:13 smithi162 ceph-mon[44304]: 1.c scrub starts 2024-02-09T17:38:13.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:13 smithi162 ceph-mon[44304]: 1.c scrub ok 2024-02-09T17:38:13.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:13 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:38:13.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:13 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:38:13.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:13 smithi162 ceph-mon[44304]: 1.5 scrub starts 2024-02-09T17:38:13.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:13 smithi162 ceph-mon[44304]: 1.5 scrub ok 2024-02-09T17:38:13.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:13 smithi099 ceph-mon[39420]: 1.c scrub starts 2024-02-09T17:38:13.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:13 smithi099 ceph-mon[39420]: 1.c scrub ok 2024-02-09T17:38:13.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:13 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:38:13.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:13 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:38:13.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:13 smithi099 ceph-mon[39420]: 1.5 scrub starts 2024-02-09T17:38:13.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:13 smithi099 ceph-mon[39420]: 1.5 scrub ok 2024-02-09T17:38:13.794 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:38:13.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:13.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:38:13.899 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:38:13.937 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:13.972 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:38:13.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:13.976 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:38:14.077 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:38:14.116 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:14.154 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:38:14.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:14.162 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:38:14.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:14 smithi099 ceph-mon[39420]: pgmap v1013: 37 pgs: 37 active+clean; 5.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 886 op/s 2024-02-09T17:38:14.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:14 smithi099 ceph-mon[39420]: osdmap e370: 12 total, 12 up, 12 in 2024-02-09T17:38:14.263 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:38:14.326 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:14.358 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:38:14.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:14.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:38:14.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:14 smithi179 ceph-mon[43975]: pgmap v1013: 37 pgs: 37 active+clean; 5.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 886 op/s 2024-02-09T17:38:14.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:14 smithi179 ceph-mon[43975]: osdmap e370: 12 total, 12 up, 12 in 2024-02-09T17:38:14.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:14 smithi162 ceph-mon[44304]: pgmap v1013: 37 pgs: 37 active+clean; 5.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 886 op/s 2024-02-09T17:38:14.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:14 smithi162 ceph-mon[44304]: osdmap e370: 12 total, 12 up, 12 in 2024-02-09T17:38:14.464 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:38:14.493 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:14.521 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:38:14.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:14.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:38:14.629 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:38:14.694 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:14.816 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:38:14.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:14.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:38:14.866 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:38:14.920 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:38:14.975 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:15.341 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:38:15.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:15.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:38:15.452 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:38:15.484 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:15.509 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:38:15.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:15.513 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:38:15.588 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:38:15.588 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 20, 2024-02-09T17:38:15.588 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 600, 2024-02-09T17:38:15.588 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [ 2024-02-09T17:38:15.589 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.589 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.24817.0:141283 3.b 3:d2ec9097:::benchmark_data_smithi179_90440_object9324:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-02-09T17:38:15.589 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:31:59.248105+0000", 2024-02-09T17:38:15.589 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 376.28432537200001, 2024-02-09T17:38:15.589 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.3382161039999998, 2024-02-09T17:38:15.589 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:38:15.589 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "reached pg", 2024-02-09T17:38:15.589 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:38:15.589 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.24817", 2024-02-09T17:38:15.589 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:38:15.589 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 141283 2024-02-09T17:38:15.589 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.589 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:38:15.589 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.590 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:38:15.590 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.248105+0000", 2024-02-09T17:38:15.590 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:38:15.590 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.590 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.590 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:38:15.590 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.248105+0000", 2024-02-09T17:38:15.590 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:38:15.590 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.590 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.590 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:38:15.590 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.248106+0000", 2024-02-09T17:38:15.590 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1659999999999999e-06 2024-02-09T17:38:15.591 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.591 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.591 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:38:15.591 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.248111+0000", 2024-02-09T17:38:15.591 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.8219999999999997e-06 2024-02-09T17:38:15.591 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.591 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.591 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:38:15.591 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.248121+0000", 2024-02-09T17:38:15.591 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0370999999999999e-05 2024-02-09T17:38:15.591 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.591 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.591 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:38:15.591 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.248127+0000", 2024-02-09T17:38:15.591 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.5860000000000004e-06 2024-02-09T17:38:15.592 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.592 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.592 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:38:15.592 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.248178+0000", 2024-02-09T17:38:15.592 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1285000000000001e-05 2024-02-09T17:38:15.592 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.592 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.592 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:38:15.592 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.248297+0000", 2024-02-09T17:38:15.592 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000119437 2024-02-09T17:38:15.592 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.592 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.592 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,4", 2024-02-09T17:38:15.593 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.248425+0000", 2024-02-09T17:38:15.593 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012806499999999999 2024-02-09T17:38:15.593 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.593 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.593 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:15.593 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.267275+0000", 2024-02-09T17:38:15.593 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.018849626000000001 2024-02-09T17:38:15.593 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.593 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.593 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:38:15.593 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.267325+0000", 2024-02-09T17:38:15.593 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.0420999999999997e-05 2024-02-09T17:38:15.593 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.593 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.594 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:38:15.594 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.272104+0000", 2024-02-09T17:38:15.595 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0047789879999999996 2024-02-09T17:38:15.595 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.595 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.595 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:38:15.595 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.272120+0000", 2024-02-09T17:38:15.595 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5548000000000001e-05 2024-02-09T17:38:15.595 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.595 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.595 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:38:15.595 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:00.269493+0000", 2024-02-09T17:38:15.595 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.99737326299999995 2024-02-09T17:38:15.595 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.595 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.595 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:38:15.596 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:00.269508+0000", 2024-02-09T17:38:15.596 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5208000000000001e-05 2024-02-09T17:38:15.596 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.596 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.596 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:38:15.596 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.586264+0000", 2024-02-09T17:38:15.596 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.316755348 2024-02-09T17:38:15.596 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.596 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.596 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:38:15.596 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.586321+0000", 2024-02-09T17:38:15.596 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6969999999999998e-05 2024-02-09T17:38:15.596 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:38:15.597 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:38:15.597 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:38:15.597 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.597 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.597 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.24817.0:141286 3.b 3:df788bf0:::benchmark_data_smithi179_90440_object9327:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-02-09T17:38:15.597 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:31:59.250595+0000", 2024-02-09T17:38:15.597 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 376.28183491700003, 2024-02-09T17:38:15.597 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.3359782299999998, 2024-02-09T17:38:15.597 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:38:15.597 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "reached pg", 2024-02-09T17:38:15.597 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:38:15.597 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.24817", 2024-02-09T17:38:15.597 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:38:15.597 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 141286 2024-02-09T17:38:15.598 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.598 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:38:15.598 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.598 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:38:15.598 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.250595+0000", 2024-02-09T17:38:15.598 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:38:15.598 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.598 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.598 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:38:15.598 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.250595+0000", 2024-02-09T17:38:15.598 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:38:15.598 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.598 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.598 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:38:15.599 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.250596+0000", 2024-02-09T17:38:15.599 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.09e-07 2024-02-09T17:38:15.599 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.599 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.599 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:38:15.599 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.250600+0000", 2024-02-09T17:38:15.599 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9509999999999999e-06 2024-02-09T17:38:15.599 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.599 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.599 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:38:15.599 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.250610+0000", 2024-02-09T17:38:15.599 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0329000000000001e-05 2024-02-09T17:38:15.599 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.599 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.599 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:38:15.599 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.250617+0000", 2024-02-09T17:38:15.600 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.041e-06 2024-02-09T17:38:15.600 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.600 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.600 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:38:15.600 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.253427+0000", 2024-02-09T17:38:15.600 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0028099589999999999 2024-02-09T17:38:15.600 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.600 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.600 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:38:15.600 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.253569+0000", 2024-02-09T17:38:15.600 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000141798 2024-02-09T17:38:15.600 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.600 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.600 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,4", 2024-02-09T17:38:15.601 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.253708+0000", 2024-02-09T17:38:15.601 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000139369 2024-02-09T17:38:15.601 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.601 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.601 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:15.601 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.267316+0000", 2024-02-09T17:38:15.601 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.013607161 2024-02-09T17:38:15.601 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.601 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.601 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:38:15.601 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.267329+0000", 2024-02-09T17:38:15.601 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3844e-05 2024-02-09T17:38:15.601 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.601 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.601 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:38:15.602 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.272218+0000", 2024-02-09T17:38:15.602 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.004888697 2024-02-09T17:38:15.602 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.602 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.602 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:38:15.602 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.272229+0000", 2024-02-09T17:38:15.602 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1034e-05 2024-02-09T17:38:15.602 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.602 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.602 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:38:15.602 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:00.269558+0000", 2024-02-09T17:38:15.602 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.997328459 2024-02-09T17:38:15.602 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.602 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.602 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:38:15.603 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:00.269569+0000", 2024-02-09T17:38:15.603 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1207e-05 2024-02-09T17:38:15.603 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.603 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.603 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:38:15.603 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.586531+0000", 2024-02-09T17:38:15.603 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.316962338 2024-02-09T17:38:15.603 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.603 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.603 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:38:15.603 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.586573+0000", 2024-02-09T17:38:15.603 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2134e-05 2024-02-09T17:38:15.603 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:38:15.603 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:38:15.604 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:38:15.604 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.604 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.604 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.24817.0:141337 3.b 3:df1cd110:::benchmark_data_smithi179_90440_object9378:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-02-09T17:38:15.604 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:31:59.309373+0000", 2024-02-09T17:38:15.604 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 376.22305703299998, 2024-02-09T17:38:15.604 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.321910618, 2024-02-09T17:38:15.604 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:38:15.604 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:38:15.604 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:38:15.604 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.24817", 2024-02-09T17:38:15.604 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:38:15.604 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 141337 2024-02-09T17:38:15.604 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.605 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:38:15.605 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.605 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:38:15.605 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.309373+0000", 2024-02-09T17:38:15.605 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:38:15.605 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.605 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.605 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:38:15.605 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.309373+0000", 2024-02-09T17:38:15.605 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:38:15.605 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.605 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.605 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:38:15.605 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.309373+0000", 2024-02-09T17:38:15.605 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.6699999999999999e-07 2024-02-09T17:38:15.606 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.606 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.606 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:38:15.606 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.309376+0000", 2024-02-09T17:38:15.606 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.8930000000000001e-06 2024-02-09T17:38:15.606 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.606 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.606 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:38:15.606 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.309383+0000", 2024-02-09T17:38:15.606 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.207e-06 2024-02-09T17:38:15.606 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.606 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.606 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:38:15.606 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.309386+0000", 2024-02-09T17:38:15.607 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9740000000000004e-06 2024-02-09T17:38:15.607 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.607 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.607 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:38:15.607 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.309460+0000", 2024-02-09T17:38:15.607 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.3609000000000006e-05 2024-02-09T17:38:15.607 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.607 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.607 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:38:15.607 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.309471+0000", 2024-02-09T17:38:15.607 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1129e-05 2024-02-09T17:38:15.607 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.607 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.607 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:38:15.607 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:00.269607+0000", 2024-02-09T17:38:15.608 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.96013575600000001 2024-02-09T17:38:15.608 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.609 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.609 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:38:15.609 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:00.269618+0000", 2024-02-09T17:38:15.609 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0581000000000001e-05 2024-02-09T17:38:15.609 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.609 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.609 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:38:15.609 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.586609+0000", 2024-02-09T17:38:15.609 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3169916719999999 2024-02-09T17:38:15.609 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.609 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.609 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:38:15.609 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.586700+0000", 2024-02-09T17:38:15.609 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.0446999999999996e-05 2024-02-09T17:38:15.609 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.610 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.610 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,4,11", 2024-02-09T17:38:15.610 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.586782+0000", 2024-02-09T17:38:15.610 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.2571000000000006e-05 2024-02-09T17:38:15.610 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.610 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.610 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:15.610 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.596440+0000", 2024-02-09T17:38:15.610 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0096572710000000003 2024-02-09T17:38:15.610 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.610 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.610 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:15.610 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.596485+0000", 2024-02-09T17:38:15.610 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.5444000000000001e-05 2024-02-09T17:38:15.610 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.611 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.611 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:38:15.611 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.596499+0000", 2024-02-09T17:38:15.611 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4136e-05 2024-02-09T17:38:15.611 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.611 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.611 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:15.611 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.631074+0000", 2024-02-09T17:38:15.611 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.034574463 2024-02-09T17:38:15.611 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.611 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.611 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:38:15.611 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.631147+0000", 2024-02-09T17:38:15.611 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.3683000000000006e-05 2024-02-09T17:38:15.611 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.612 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.612 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:38:15.612 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.631284+0000", 2024-02-09T17:38:15.612 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000136315 2024-02-09T17:38:15.612 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:38:15.612 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:38:15.612 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:38:15.612 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.612 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.612 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:108028 4.0 4:06f84d26:::benchmark_data_smithi179_103809_object108027:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e347)", 2024-02-09T17:38:15.612 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:36:40.206437+0000", 2024-02-09T17:38:15.612 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 95.325992752999994, 2024-02-09T17:38:15.612 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.22505144899999999, 2024-02-09T17:38:15.612 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:38:15.612 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:38:15.613 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:38:15.613 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:38:15.613 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:38:15.613 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 108028 2024-02-09T17:38:15.613 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.613 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:38:15.613 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.613 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:38:15.613 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:36:40.206437+0000", 2024-02-09T17:38:15.613 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:38:15.613 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.613 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.613 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:38:15.613 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:36:40.206437+0000", 2024-02-09T17:38:15.613 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:38:15.613 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.614 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.614 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:38:15.614 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:36:40.206439+0000", 2024-02-09T17:38:15.614 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3740000000000001e-06 2024-02-09T17:38:15.614 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.614 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.614 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:38:15.614 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:36:40.206510+0000", 2024-02-09T17:38:15.614 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.1836999999999997e-05 2024-02-09T17:38:15.614 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.614 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.614 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:38:15.614 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:36:40.206521+0000", 2024-02-09T17:38:15.614 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0890000000000001e-05 2024-02-09T17:38:15.614 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.615 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.615 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:38:15.615 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:36:40.206527+0000", 2024-02-09T17:38:15.615 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.942e-06 2024-02-09T17:38:15.615 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.615 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.615 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:38:15.615 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:36:40.207265+0000", 2024-02-09T17:38:15.615 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00073802199999999996 2024-02-09T17:38:15.615 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.615 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.615 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:38:15.615 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:36:40.207353+0000", 2024-02-09T17:38:15.615 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.8115999999999995e-05 2024-02-09T17:38:15.615 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.615 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.615 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,7", 2024-02-09T17:38:15.616 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:36:40.207476+0000", 2024-02-09T17:38:15.616 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000122621 2024-02-09T17:38:15.616 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.616 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.616 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:38:15.616 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:36:40.215610+0000", 2024-02-09T17:38:15.616 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0081337370000000003 2024-02-09T17:38:15.616 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.616 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.616 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:15.616 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:36:40.215653+0000", 2024-02-09T17:38:15.616 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2954999999999998e-05 2024-02-09T17:38:15.616 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.617 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.617 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:15.617 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:36:40.431354+0000", 2024-02-09T17:38:15.617 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.21570145700000001 2024-02-09T17:38:15.617 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.617 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.617 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:38:15.617 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:36:40.431439+0000", 2024-02-09T17:38:15.617 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.5043999999999994e-05 2024-02-09T17:38:15.617 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.617 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.617 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:38:15.617 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:36:40.431489+0000", 2024-02-09T17:38:15.617 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.9453999999999998e-05 2024-02-09T17:38:15.617 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:38:15.617 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:38:15.617 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:38:15.618 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.618 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.618 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:145996 4.0 4:02a71ed0:::benchmark_data_smithi179_103809_object8530:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e358)", 2024-02-09T17:38:15.618 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:37:19.414371+0000", 2024-02-09T17:38:15.618 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 56.118058529000002, 2024-02-09T17:38:15.618 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.058960754999999997, 2024-02-09T17:38:15.618 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:38:15.618 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:38:15.618 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:38:15.618 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:38:15.618 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:38:15.618 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 145996 2024-02-09T17:38:15.618 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.618 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:38:15.618 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.619 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:38:15.619 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:19.414371+0000", 2024-02-09T17:38:15.619 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:38:15.619 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.619 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.619 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:38:15.619 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:19.414371+0000", 2024-02-09T17:38:15.619 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:38:15.619 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.619 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.619 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:38:15.619 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:19.414373+0000", 2024-02-09T17:38:15.619 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4920000000000001e-06 2024-02-09T17:38:15.619 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.619 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.620 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:38:15.620 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:19.414378+0000", 2024-02-09T17:38:15.620 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2050000000000001e-06 2024-02-09T17:38:15.620 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.620 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.620 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:38:15.620 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:19.414401+0000", 2024-02-09T17:38:15.620 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.296e-05 2024-02-09T17:38:15.620 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.620 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.620 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:38:15.620 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:19.414409+0000", 2024-02-09T17:38:15.620 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.5390000000000002e-06 2024-02-09T17:38:15.620 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.620 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.621 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:38:15.621 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:19.414466+0000", 2024-02-09T17:38:15.621 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7394e-05 2024-02-09T17:38:15.621 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.621 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.621 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:38:15.621 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:19.414558+0000", 2024-02-09T17:38:15.621 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.2236000000000003e-05 2024-02-09T17:38:15.621 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.621 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.621 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,7", 2024-02-09T17:38:15.621 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:19.414646+0000", 2024-02-09T17:38:15.621 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.7359000000000002e-05 2024-02-09T17:38:15.621 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.621 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.621 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:15.622 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:19.421603+0000", 2024-02-09T17:38:15.622 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0069571529999999998 2024-02-09T17:38:15.622 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.622 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.622 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:38:15.622 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:19.422840+0000", 2024-02-09T17:38:15.622 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.001236741 2024-02-09T17:38:15.622 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.622 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.622 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:15.622 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:19.473252+0000", 2024-02-09T17:38:15.622 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.050412086000000002 2024-02-09T17:38:15.622 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.622 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.622 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:38:15.622 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:19.473293+0000", 2024-02-09T17:38:15.623 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.1196999999999999e-05 2024-02-09T17:38:15.623 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.623 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.623 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:38:15.623 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:19.473332+0000", 2024-02-09T17:38:15.623 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9393e-05 2024-02-09T17:38:15.623 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:38:15.623 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:38:15.623 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:38:15.623 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.623 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.623 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:167871 4.0 4:08df4aaf:::benchmark_data_smithi179_103809_object30405:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e362)", 2024-02-09T17:38:15.623 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:37:44.045842+0000", 2024-02-09T17:38:15.623 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 31.486587532000001, 2024-02-09T17:38:15.624 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.11452219199999999, 2024-02-09T17:38:15.624 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:38:15.624 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:38:15.624 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:38:15.624 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:38:15.624 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:38:15.624 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 167871 2024-02-09T17:38:15.624 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.624 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:38:15.624 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.624 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:38:15.624 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.045842+0000", 2024-02-09T17:38:15.624 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:38:15.624 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.625 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.625 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:38:15.625 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.045842+0000", 2024-02-09T17:38:15.625 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:38:15.625 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.625 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.625 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:38:15.625 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.045844+0000", 2024-02-09T17:38:15.625 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.8050000000000001e-06 2024-02-09T17:38:15.625 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.625 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.625 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:38:15.625 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.045850+0000", 2024-02-09T17:38:15.625 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.0730000000000001e-06 2024-02-09T17:38:15.626 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.626 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.626 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:38:15.626 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.045861+0000", 2024-02-09T17:38:15.626 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0271e-05 2024-02-09T17:38:15.626 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.626 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.626 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:38:15.626 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.045868+0000", 2024-02-09T17:38:15.626 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.7710000000000006e-06 2024-02-09T17:38:15.626 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.626 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.626 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:38:15.626 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.045915+0000", 2024-02-09T17:38:15.626 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.7040999999999999e-05 2024-02-09T17:38:15.627 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.627 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.627 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:38:15.627 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.046002+0000", 2024-02-09T17:38:15.627 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.6454999999999998e-05 2024-02-09T17:38:15.627 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.627 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.627 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,7", 2024-02-09T17:38:15.627 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.046076+0000", 2024-02-09T17:38:15.627 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.4148999999999998e-05 2024-02-09T17:38:15.627 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.627 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.627 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:38:15.627 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.051439+0000", 2024-02-09T17:38:15.627 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0053624900000000001 2024-02-09T17:38:15.627 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.628 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.628 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:15.628 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.053621+0000", 2024-02-09T17:38:15.628 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.002182977 2024-02-09T17:38:15.628 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.628 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.628 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:15.628 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.160258+0000", 2024-02-09T17:38:15.628 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.106636063 2024-02-09T17:38:15.628 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.628 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.628 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:38:15.628 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.160318+0000", 2024-02-09T17:38:15.628 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.0186999999999998e-05 2024-02-09T17:38:15.629 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.629 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.629 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:38:15.629 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.160365+0000", 2024-02-09T17:38:15.629 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.6910000000000003e-05 2024-02-09T17:38:15.629 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:38:15.629 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:38:15.629 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:38:15.629 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.629 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.629 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25201.0:167875 4.4 e362/322 4:2455adba:::benchmark_data_smithi179_103809_object30409:head v 362'10418, mlcod=362'10416)", 2024-02-09T17:38:15.629 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:37:44.049219+0000", 2024-02-09T17:38:15.629 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 31.483211102999999, 2024-02-09T17:38:15.629 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.11105707400000001, 2024-02-09T17:38:15.629 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:38:15.629 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:38:15.630 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:38:15.630 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.630 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:38:15.630 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.049219+0000", 2024-02-09T17:38:15.630 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:38:15.630 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.630 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.630 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:38:15.630 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.049219+0000", 2024-02-09T17:38:15.630 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:38:15.630 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.630 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.630 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:38:15.630 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.049220+0000", 2024-02-09T17:38:15.630 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.2399999999999997e-07 2024-02-09T17:38:15.630 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.631 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.631 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:38:15.631 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.049226+0000", 2024-02-09T17:38:15.631 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.8939999999999998e-06 2024-02-09T17:38:15.631 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.631 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.631 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:38:15.631 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.049235+0000", 2024-02-09T17:38:15.631 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.6350000000000001e-06 2024-02-09T17:38:15.631 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.631 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.631 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:38:15.632 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.049242+0000", 2024-02-09T17:38:15.632 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.148e-06 2024-02-09T17:38:15.632 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.632 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.632 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:38:15.632 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.049306+0000", 2024-02-09T17:38:15.632 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.3424999999999997e-05 2024-02-09T17:38:15.632 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.632 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.633 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:38:15.633 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.049324+0000", 2024-02-09T17:38:15.633 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.8499999999999999e-05 2024-02-09T17:38:15.633 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.633 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.633 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:38:15.633 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.160219+0000", 2024-02-09T17:38:15.633 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.110894203 2024-02-09T17:38:15.633 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.633 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.633 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:38:15.633 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.160276+0000", 2024-02-09T17:38:15.633 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7445000000000003e-05 2024-02-09T17:38:15.633 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:38:15.634 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:38:15.634 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:38:15.634 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.634 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.634 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop_reply(client.25201.0:167871 4.0 e362/322 ondisk, result = 0)", 2024-02-09T17:38:15.634 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:37:44.053980+0000", 2024-02-09T17:38:15.634 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 31.478449822000002, 2024-02-09T17:38:15.634 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.10639011, 2024-02-09T17:38:15.634 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:38:15.634 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T17:38:15.634 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:38:15.634 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.634 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:38:15.634 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.053980+0000", 2024-02-09T17:38:15.634 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:38:15.635 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.635 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.635 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:38:15.635 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.053980+0000", 2024-02-09T17:38:15.635 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:38:15.635 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.635 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.635 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:38:15.635 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.053981+0000", 2024-02-09T17:38:15.635 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.2500000000000005e-07 2024-02-09T17:38:15.635 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.635 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.635 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:38:15.635 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.053985+0000", 2024-02-09T17:38:15.635 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.6160000000000004e-06 2024-02-09T17:38:15.636 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.636 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.636 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:38:15.636 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.053994+0000", 2024-02-09T17:38:15.636 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.867e-06 2024-02-09T17:38:15.636 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.636 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.636 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:38:15.636 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.054001+0000", 2024-02-09T17:38:15.636 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.472e-06 2024-02-09T17:38:15.636 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.636 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.636 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:38:15.636 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.160240+0000", 2024-02-09T17:38:15.636 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.106239369 2024-02-09T17:38:15.637 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.637 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.637 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:38:15.637 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.160251+0000", 2024-02-09T17:38:15.637 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0604999999999999e-05 2024-02-09T17:38:15.637 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.637 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.637 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:38:15.637 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.160370+0000", 2024-02-09T17:38:15.637 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00011955600000000001 2024-02-09T17:38:15.637 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:38:15.637 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:38:15.637 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:38:15.637 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.637 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.638 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:167885 4.0 4:081e94b1:::benchmark_data_smithi179_103809_object30419:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e362)", 2024-02-09T17:38:15.638 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:37:44.056154+0000", 2024-02-09T17:38:15.638 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 31.476276469999998, 2024-02-09T17:38:15.638 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.11438984100000001, 2024-02-09T17:38:15.638 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:38:15.638 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:38:15.638 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:38:15.638 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:38:15.638 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:38:15.638 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 167885 2024-02-09T17:38:15.638 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.638 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:38:15.638 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.638 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:38:15.638 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.056154+0000", 2024-02-09T17:38:15.639 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:38:15.639 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.639 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.639 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:38:15.639 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.056154+0000", 2024-02-09T17:38:15.639 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:38:15.639 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.639 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.639 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:38:15.639 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.056155+0000", 2024-02-09T17:38:15.639 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.263e-06 2024-02-09T17:38:15.639 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.639 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.639 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:38:15.639 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.056160+0000", 2024-02-09T17:38:15.639 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3460000000000001e-06 2024-02-09T17:38:15.640 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.640 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.640 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:38:15.640 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.160219+0000", 2024-02-09T17:38:15.640 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.10405898800000001 2024-02-09T17:38:15.640 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.640 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.640 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:38:15.640 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.160231+0000", 2024-02-09T17:38:15.640 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1826e-05 2024-02-09T17:38:15.640 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.640 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.640 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:38:15.640 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.160409+0000", 2024-02-09T17:38:15.641 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00017760599999999999 2024-02-09T17:38:15.641 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.641 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.641 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:38:15.641 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.160538+0000", 2024-02-09T17:38:15.641 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000129691 2024-02-09T17:38:15.641 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.641 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.641 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,7", 2024-02-09T17:38:15.641 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.160663+0000", 2024-02-09T17:38:15.641 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012486499999999999 2024-02-09T17:38:15.641 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.641 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.641 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:15.641 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.170317+0000", 2024-02-09T17:38:15.641 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0096537759999999993 2024-02-09T17:38:15.642 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.642 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.642 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:15.642 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.170369+0000", 2024-02-09T17:38:15.642 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2476999999999999e-05 2024-02-09T17:38:15.642 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.642 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.642 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:38:15.642 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.170457+0000", 2024-02-09T17:38:15.642 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.8035999999999995e-05 2024-02-09T17:38:15.642 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.642 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.642 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:38:15.642 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.170498+0000", 2024-02-09T17:38:15.642 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.1017999999999999e-05 2024-02-09T17:38:15.643 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.643 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.643 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:38:15.643 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.170543+0000", 2024-02-09T17:38:15.643 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.4948999999999999e-05 2024-02-09T17:38:15.643 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:38:15.643 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:38:15.643 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:38:15.643 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.643 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.643 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25201.0:167889 4.9 e362/322 4:9c116f3c:::benchmark_data_smithi179_103809_object30423:head v 362'10532, mlcod=362'10530)", 2024-02-09T17:38:15.643 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:37:44.060706+0000", 2024-02-09T17:38:15.643 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 31.471723963999999, 2024-02-09T17:38:15.643 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.109034292, 2024-02-09T17:38:15.643 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:38:15.644 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:38:15.644 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:38:15.644 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.644 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:38:15.644 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.060706+0000", 2024-02-09T17:38:15.644 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:38:15.644 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.644 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.644 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:38:15.644 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.060706+0000", 2024-02-09T17:38:15.644 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:38:15.644 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.644 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.644 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:38:15.645 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.060707+0000", 2024-02-09T17:38:15.645 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1960000000000001e-06 2024-02-09T17:38:15.645 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.645 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.645 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:38:15.645 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.060716+0000", 2024-02-09T17:38:15.645 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.4700000000000002e-06 2024-02-09T17:38:15.645 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.645 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.645 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:38:15.645 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.160207+0000", 2024-02-09T17:38:15.645 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.099491534000000006 2024-02-09T17:38:15.645 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.645 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.645 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:38:15.646 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.160224+0000", 2024-02-09T17:38:15.646 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.6504999999999999e-05 2024-02-09T17:38:15.646 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.646 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.646 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:38:15.646 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.160285+0000", 2024-02-09T17:38:15.646 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.1029000000000001e-05 2024-02-09T17:38:15.646 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.646 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.646 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:38:15.646 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.160310+0000", 2024-02-09T17:38:15.646 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.4743e-05 2024-02-09T17:38:15.646 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.646 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.647 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:38:15.647 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.169697+0000", 2024-02-09T17:38:15.647 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0093877950000000009 2024-02-09T17:38:15.647 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.647 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.647 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:38:15.647 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:44.169740+0000", 2024-02-09T17:38:15.647 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.3019999999999998e-05 2024-02-09T17:38:15.647 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:38:15.647 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:38:15.647 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:38:15.647 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.647 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.647 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:173279 4.0 4:0a77f654:::benchmark_data_smithi179_103809_object35813:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-02-09T17:38:15.647 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:37:50.206474+0000", 2024-02-09T17:38:15.648 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 25.32595602, 2024-02-09T17:38:15.648 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.056854031999999999, 2024-02-09T17:38:15.648 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:38:15.648 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:38:15.648 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:38:15.648 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:38:15.648 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:38:15.648 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 173279 2024-02-09T17:38:15.648 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.648 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:38:15.648 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.648 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:38:15.648 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.206474+0000", 2024-02-09T17:38:15.648 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:38:15.649 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.649 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.649 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:38:15.649 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.206474+0000", 2024-02-09T17:38:15.649 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:38:15.649 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.649 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.649 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:38:15.649 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.206475+0000", 2024-02-09T17:38:15.649 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3570000000000001e-06 2024-02-09T17:38:15.649 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.649 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.649 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:38:15.649 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.206488+0000", 2024-02-09T17:38:15.649 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2432999999999999e-05 2024-02-09T17:38:15.650 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.650 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.650 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:38:15.650 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.206503+0000", 2024-02-09T17:38:15.650 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4871e-05 2024-02-09T17:38:15.650 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.650 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.650 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:38:15.650 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.206511+0000", 2024-02-09T17:38:15.650 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.9140000000000008e-06 2024-02-09T17:38:15.650 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.650 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.650 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:38:15.650 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.206689+0000", 2024-02-09T17:38:15.650 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000178612 2024-02-09T17:38:15.650 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.651 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.651 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:38:15.651 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.206821+0000", 2024-02-09T17:38:15.651 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013212599999999999 2024-02-09T17:38:15.651 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.651 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.651 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,7", 2024-02-09T17:38:15.651 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.206952+0000", 2024-02-09T17:38:15.651 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000131094 2024-02-09T17:38:15.651 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.651 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.651 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:38:15.651 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.219919+0000", 2024-02-09T17:38:15.651 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.012966848 2024-02-09T17:38:15.651 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.652 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.652 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:15.652 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.222171+0000", 2024-02-09T17:38:15.652 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0022519390000000001 2024-02-09T17:38:15.652 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.652 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.652 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:15.652 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.263268+0000", 2024-02-09T17:38:15.652 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.041096475 2024-02-09T17:38:15.652 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.652 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.652 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:38:15.652 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.263303+0000", 2024-02-09T17:38:15.652 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.5296999999999998e-05 2024-02-09T17:38:15.652 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.653 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.653 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:38:15.653 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.263328+0000", 2024-02-09T17:38:15.653 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.5066e-05 2024-02-09T17:38:15.653 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:38:15.653 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:38:15.653 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:38:15.653 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.653 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.653 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:173282 4.0 4:0ae5bcb7:::benchmark_data_smithi179_103809_object35816:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-02-09T17:38:15.653 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:37:50.206531+0000", 2024-02-09T17:38:15.653 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 25.325899303, 2024-02-09T17:38:15.653 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.056881612999999998, 2024-02-09T17:38:15.653 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:38:15.653 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:38:15.654 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:38:15.654 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:38:15.654 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:38:15.654 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 173282 2024-02-09T17:38:15.654 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.654 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:38:15.654 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.654 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:38:15.654 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.206531+0000", 2024-02-09T17:38:15.654 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:38:15.654 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.654 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.654 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:38:15.654 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.206531+0000", 2024-02-09T17:38:15.654 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:38:15.655 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.655 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.655 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:38:15.655 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.206531+0000", 2024-02-09T17:38:15.655 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.8299999999999998e-07 2024-02-09T17:38:15.655 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.655 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.655 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:38:15.655 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.206533+0000", 2024-02-09T17:38:15.655 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.3149999999999999e-06 2024-02-09T17:38:15.655 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.655 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.655 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:38:15.655 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.206537+0000", 2024-02-09T17:38:15.656 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.1130000000000003e-06 2024-02-09T17:38:15.656 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.656 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.656 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:38:15.656 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.206541+0000", 2024-02-09T17:38:15.656 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.0690000000000002e-06 2024-02-09T17:38:15.656 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.656 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.656 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:38:15.656 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.212267+0000", 2024-02-09T17:38:15.656 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0057268270000000003 2024-02-09T17:38:15.656 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.656 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.656 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:38:15.656 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.212406+0000", 2024-02-09T17:38:15.657 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013848500000000001 2024-02-09T17:38:15.657 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.657 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.657 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,7", 2024-02-09T17:38:15.657 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.212532+0000", 2024-02-09T17:38:15.657 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012562200000000001 2024-02-09T17:38:15.657 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.657 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.657 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:38:15.657 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.220074+0000", 2024-02-09T17:38:15.657 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0075423290000000004 2024-02-09T17:38:15.657 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.657 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.657 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:15.657 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.223923+0000", 2024-02-09T17:38:15.658 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0038495280000000001 2024-02-09T17:38:15.658 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.658 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.658 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:15.658 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.263350+0000", 2024-02-09T17:38:15.658 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.039426706999999998 2024-02-09T17:38:15.658 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.658 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.658 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:38:15.658 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.263377+0000", 2024-02-09T17:38:15.658 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.6656e-05 2024-02-09T17:38:15.658 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.658 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.658 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:38:15.658 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:50.263412+0000", 2024-02-09T17:38:15.658 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.5578999999999999e-05 2024-02-09T17:38:15.659 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:38:15.659 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:38:15.659 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:38:15.659 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.659 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.659 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:175619 4.0 4:0d3e4d70:::benchmark_data_smithi179_103809_object38153:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-02-09T17:38:15.659 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:37:52.940028+0000", 2024-02-09T17:38:15.659 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 22.592402382, 2024-02-09T17:38:15.659 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.057155535, 2024-02-09T17:38:15.659 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:38:15.659 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:38:15.659 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:38:15.659 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:38:15.659 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:38:15.660 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 175619 2024-02-09T17:38:15.660 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.660 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:38:15.660 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.660 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:38:15.660 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:52.940028+0000", 2024-02-09T17:38:15.660 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:38:15.660 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.660 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.660 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:38:15.660 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:52.940028+0000", 2024-02-09T17:38:15.660 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:38:15.660 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.660 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.660 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:38:15.661 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:52.940028+0000", 2024-02-09T17:38:15.661 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.0699999999999998e-07 2024-02-09T17:38:15.661 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.661 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.661 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:38:15.661 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:52.940030+0000", 2024-02-09T17:38:15.661 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.6500000000000001e-06 2024-02-09T17:38:15.661 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.661 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.661 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:38:15.661 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:52.940036+0000", 2024-02-09T17:38:15.661 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.0850000000000002e-06 2024-02-09T17:38:15.661 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.661 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.661 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:38:15.661 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:52.940040+0000", 2024-02-09T17:38:15.662 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.7700000000000001e-06 2024-02-09T17:38:15.662 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.662 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.662 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:38:15.662 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:52.979722+0000", 2024-02-09T17:38:15.662 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.039681389999999997 2024-02-09T17:38:15.662 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.662 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.662 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:38:15.662 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:52.979865+0000", 2024-02-09T17:38:15.662 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014274900000000001 2024-02-09T17:38:15.662 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.662 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.662 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,7", 2024-02-09T17:38:15.662 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:52.979976+0000", 2024-02-09T17:38:15.662 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000111329 2024-02-09T17:38:15.663 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.663 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.663 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:38:15.663 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:52.989851+0000", 2024-02-09T17:38:15.663 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0098751889999999995 2024-02-09T17:38:15.663 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.663 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.663 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:15.663 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:52.992918+0000", 2024-02-09T17:38:15.663 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0030668869999999999 2024-02-09T17:38:15.663 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.663 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.663 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:15.663 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:52.997111+0000", 2024-02-09T17:38:15.663 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0041926380000000003 2024-02-09T17:38:15.663 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.664 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.664 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:38:15.664 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:52.997160+0000", 2024-02-09T17:38:15.664 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.914e-05 2024-02-09T17:38:15.664 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.664 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.664 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:38:15.664 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:52.997183+0000", 2024-02-09T17:38:15.664 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.3400999999999998e-05 2024-02-09T17:38:15.664 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:38:15.664 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:38:15.664 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:38:15.664 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.665 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.665 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:175631 4.0 4:0e144415:::benchmark_data_smithi179_103809_object38165:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-02-09T17:38:15.665 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:37:52.940046+0000", 2024-02-09T17:38:15.665 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 22.592384215999999, 2024-02-09T17:38:15.665 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.057223966000000001, 2024-02-09T17:38:15.665 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:38:15.665 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:38:15.665 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:38:15.665 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:38:15.665 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:38:15.665 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 175631 2024-02-09T17:38:15.665 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.665 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:38:15.665 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.665 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:38:15.666 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:52.940046+0000", 2024-02-09T17:38:15.666 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:38:15.666 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.666 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.666 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:38:15.666 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:52.940046+0000", 2024-02-09T17:38:15.666 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:38:15.666 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.666 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.666 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:38:15.666 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:52.940046+0000", 2024-02-09T17:38:15.666 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.03e-07 2024-02-09T17:38:15.666 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.666 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.667 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:38:15.667 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:52.940048+0000", 2024-02-09T17:38:15.667 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.7239999999999999e-06 2024-02-09T17:38:15.667 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.667 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.667 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:38:15.667 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:52.940054+0000", 2024-02-09T17:38:15.667 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.2450000000000003e-06 2024-02-09T17:38:15.667 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.667 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.667 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:38:15.667 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:52.940059+0000", 2024-02-09T17:38:15.667 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.0309999999999998e-06 2024-02-09T17:38:15.667 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.667 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.668 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:38:15.668 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:52.984196+0000", 2024-02-09T17:38:15.668 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.044136821 2024-02-09T17:38:15.668 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.668 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.668 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:38:15.668 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:52.984289+0000", 2024-02-09T17:38:15.668 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.2724000000000002e-05 2024-02-09T17:38:15.668 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.668 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.668 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,7", 2024-02-09T17:38:15.668 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:52.984360+0000", 2024-02-09T17:38:15.668 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.1509999999999998e-05 2024-02-09T17:38:15.668 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.669 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.669 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:15.669 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:52.995257+0000", 2024-02-09T17:38:15.669 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.010897263000000001 2024-02-09T17:38:15.669 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.669 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.669 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:38:15.669 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:52.995265+0000", 2024-02-09T17:38:15.669 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.4189999999999996e-06 2024-02-09T17:38:15.669 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.669 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.669 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:15.669 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:52.997215+0000", 2024-02-09T17:38:15.669 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0019500380000000001 2024-02-09T17:38:15.669 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.670 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.670 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:38:15.670 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:52.997251+0000", 2024-02-09T17:38:15.670 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.5978e-05 2024-02-09T17:38:15.670 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.670 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.670 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:38:15.670 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:52.997270+0000", 2024-02-09T17:38:15.670 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.891e-05 2024-02-09T17:38:15.670 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:38:15.670 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:38:15.670 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:38:15.670 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.670 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.670 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:178018 4.0 4:0450335d:::benchmark_data_smithi179_103809_object40552:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-02-09T17:38:15.671 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:37:55.743267+0000", 2024-02-09T17:38:15.671 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 19.789163468000002, 2024-02-09T17:38:15.671 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.056315891, 2024-02-09T17:38:15.671 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:38:15.671 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:38:15.671 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:38:15.671 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:38:15.671 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:38:15.671 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 178018 2024-02-09T17:38:15.671 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.671 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:38:15.671 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.671 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:38:15.671 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:55.743267+0000", 2024-02-09T17:38:15.671 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:38:15.672 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.672 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.672 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:38:15.672 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:55.743267+0000", 2024-02-09T17:38:15.672 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:38:15.672 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.672 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.672 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:38:15.672 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:55.743268+0000", 2024-02-09T17:38:15.672 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.339e-06 2024-02-09T17:38:15.672 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.672 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.672 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:38:15.672 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:55.743273+0000", 2024-02-09T17:38:15.673 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.9849999999999997e-06 2024-02-09T17:38:15.673 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.673 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.673 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:38:15.673 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:55.743287+0000", 2024-02-09T17:38:15.673 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4297999999999999e-05 2024-02-09T17:38:15.673 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.673 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.673 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:38:15.673 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:55.743295+0000", 2024-02-09T17:38:15.673 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.4220000000000003e-06 2024-02-09T17:38:15.673 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.673 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.673 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:38:15.674 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:55.743342+0000", 2024-02-09T17:38:15.674 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.7295e-05 2024-02-09T17:38:15.674 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.674 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.674 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:38:15.674 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:55.743479+0000", 2024-02-09T17:38:15.674 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000136985 2024-02-09T17:38:15.674 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.674 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.674 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,7", 2024-02-09T17:38:15.674 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:55.743597+0000", 2024-02-09T17:38:15.674 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000118534 2024-02-09T17:38:15.674 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.674 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.674 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:38:15.675 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:55.749649+0000", 2024-02-09T17:38:15.675 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0060519459999999999 2024-02-09T17:38:15.675 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.675 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.675 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:15.675 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:55.753490+0000", 2024-02-09T17:38:15.675 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0038407300000000001 2024-02-09T17:38:15.675 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.675 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.675 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:15.675 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:55.799468+0000", 2024-02-09T17:38:15.675 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.045978038999999998 2024-02-09T17:38:15.675 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.675 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.675 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:38:15.676 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:55.799536+0000", 2024-02-09T17:38:15.676 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.758e-05 2024-02-09T17:38:15.676 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.676 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.676 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:38:15.676 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:55.799582+0000", 2024-02-09T17:38:15.676 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.6737999999999999e-05 2024-02-09T17:38:15.676 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:38:15.676 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:38:15.676 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:38:15.676 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.676 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.676 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:178257 4.0 4:0616d6be:::benchmark_data_smithi179_103809_object40791:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-02-09T17:38:15.676 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:37:55.992201+0000", 2024-02-09T17:38:15.676 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 19.540228967000001, 2024-02-09T17:38:15.677 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.057577110000000001, 2024-02-09T17:38:15.677 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:38:15.677 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:38:15.677 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:38:15.677 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:38:15.677 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:38:15.677 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 178257 2024-02-09T17:38:15.677 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.677 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:38:15.677 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.677 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:38:15.677 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:55.992201+0000", 2024-02-09T17:38:15.677 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:38:15.677 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.677 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.677 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:38:15.678 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:55.992201+0000", 2024-02-09T17:38:15.678 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:38:15.678 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.678 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.678 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:38:15.678 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:55.992203+0000", 2024-02-09T17:38:15.678 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.658e-06 2024-02-09T17:38:15.678 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.678 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.678 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:38:15.678 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:55.992208+0000", 2024-02-09T17:38:15.678 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7679999999999997e-06 2024-02-09T17:38:15.678 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.678 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.678 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:38:15.678 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:55.992220+0000", 2024-02-09T17:38:15.678 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1175000000000001e-05 2024-02-09T17:38:15.679 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.679 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.679 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:38:15.679 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:55.992226+0000", 2024-02-09T17:38:15.679 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.0739999999999998e-06 2024-02-09T17:38:15.679 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.679 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.679 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:38:15.679 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:55.992280+0000", 2024-02-09T17:38:15.679 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.4651999999999998e-05 2024-02-09T17:38:15.679 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.679 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.679 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:38:15.679 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:55.992365+0000", 2024-02-09T17:38:15.679 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.4728999999999995e-05 2024-02-09T17:38:15.680 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.680 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.680 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,7", 2024-02-09T17:38:15.680 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:56.027071+0000", 2024-02-09T17:38:15.680 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.034706186999999999 2024-02-09T17:38:15.680 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.680 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.680 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:15.680 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:56.042715+0000", 2024-02-09T17:38:15.680 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.015643440000000002 2024-02-09T17:38:15.680 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.680 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.680 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:38:15.680 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:56.042870+0000", 2024-02-09T17:38:15.680 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000155508 2024-02-09T17:38:15.681 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.681 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.681 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:15.681 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:56.049683+0000", 2024-02-09T17:38:15.681 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0068130710000000004 2024-02-09T17:38:15.681 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.681 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.681 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:38:15.681 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:56.049740+0000", 2024-02-09T17:38:15.681 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7204000000000003e-05 2024-02-09T17:38:15.681 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.681 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.681 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:38:15.681 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:56.049778+0000", 2024-02-09T17:38:15.681 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.7644000000000001e-05 2024-02-09T17:38:15.682 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:38:15.682 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:38:15.682 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:38:15.682 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.682 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.682 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:189528 4.0 4:0891890b:::benchmark_data_smithi179_103809_object52062:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-02-09T17:38:15.682 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:38:08.843100+0000", 2024-02-09T17:38:15.682 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 6.6893303150000003, 2024-02-09T17:38:15.682 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.076995043999999999, 2024-02-09T17:38:15.682 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:38:15.682 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:38:15.682 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:38:15.682 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:38:15.682 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:38:15.682 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 189528 2024-02-09T17:38:15.682 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.683 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:38:15.683 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.683 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:38:15.683 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:08.843100+0000", 2024-02-09T17:38:15.683 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:38:15.683 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.683 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.683 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:38:15.683 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:08.843100+0000", 2024-02-09T17:38:15.683 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:38:15.683 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.683 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.683 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:38:15.683 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:08.843101+0000", 2024-02-09T17:38:15.683 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5570000000000001e-06 2024-02-09T17:38:15.683 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.684 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.684 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:38:15.684 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:08.843107+0000", 2024-02-09T17:38:15.684 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.519e-06 2024-02-09T17:38:15.684 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.684 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.684 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:38:15.684 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:08.843122+0000", 2024-02-09T17:38:15.684 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5187999999999999e-05 2024-02-09T17:38:15.684 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.684 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.684 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:38:15.684 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:08.843131+0000", 2024-02-09T17:38:15.684 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.3239999999999992e-06 2024-02-09T17:38:15.684 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.685 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.685 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:38:15.685 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:08.843192+0000", 2024-02-09T17:38:15.685 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.0631000000000003e-05 2024-02-09T17:38:15.685 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.685 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.685 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:38:15.685 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:08.843292+0000", 2024-02-09T17:38:15.685 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000100134 2024-02-09T17:38:15.685 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.685 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.686 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,7", 2024-02-09T17:38:15.686 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:08.843409+0000", 2024-02-09T17:38:15.686 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000116864 2024-02-09T17:38:15.686 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.686 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.686 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:38:15.686 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:08.848379+0000", 2024-02-09T17:38:15.686 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0049705870000000003 2024-02-09T17:38:15.686 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.686 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.686 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:15.686 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:08.852451+0000", 2024-02-09T17:38:15.686 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0040712459999999997 2024-02-09T17:38:15.686 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.686 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.687 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:15.687 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:08.919988+0000", 2024-02-09T17:38:15.687 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.067537700000000006 2024-02-09T17:38:15.687 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.687 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.687 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:38:15.687 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:08.920049+0000", 2024-02-09T17:38:15.687 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.084e-05 2024-02-09T17:38:15.687 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.687 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.687 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:38:15.687 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:08.920095+0000", 2024-02-09T17:38:15.687 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.5454000000000002e-05 2024-02-09T17:38:15.687 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:38:15.688 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:38:15.688 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:38:15.688 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.688 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.688 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:193208 4.0 4:0c528ca2:::benchmark_data_smithi179_103809_object55742:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-02-09T17:38:15.688 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:38:13.383003+0000", 2024-02-09T17:38:15.688 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 2.1494271120000001, 2024-02-09T17:38:15.688 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.066517430000000002, 2024-02-09T17:38:15.688 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:38:15.688 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:38:15.688 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:38:15.688 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:38:15.688 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:38:15.688 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 193208 2024-02-09T17:38:15.689 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.689 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:38:15.689 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.689 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:38:15.689 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:13.383003+0000", 2024-02-09T17:38:15.689 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:38:15.689 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.689 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.689 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:38:15.689 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:13.383003+0000", 2024-02-09T17:38:15.689 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:38:15.689 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.689 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.689 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:38:15.690 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:13.383005+0000", 2024-02-09T17:38:15.690 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.804e-06 2024-02-09T17:38:15.690 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.690 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.690 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:38:15.690 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:13.383010+0000", 2024-02-09T17:38:15.690 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2159999999999997e-06 2024-02-09T17:38:15.690 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.690 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.690 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:38:15.690 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:13.383022+0000", 2024-02-09T17:38:15.690 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2505e-05 2024-02-09T17:38:15.690 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.690 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.690 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:38:15.691 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:13.383030+0000", 2024-02-09T17:38:15.691 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.4549999999999998e-06 2024-02-09T17:38:15.691 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.691 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.691 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:38:15.691 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:13.383088+0000", 2024-02-09T17:38:15.691 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7918999999999999e-05 2024-02-09T17:38:15.691 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.691 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.691 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:38:15.691 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:13.383177+0000", 2024-02-09T17:38:15.691 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.9277999999999995e-05 2024-02-09T17:38:15.691 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.691 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.692 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,7", 2024-02-09T17:38:15.692 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:13.383252+0000", 2024-02-09T17:38:15.692 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.5082999999999999e-05 2024-02-09T17:38:15.692 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.692 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.692 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:38:15.692 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:13.386998+0000", 2024-02-09T17:38:15.692 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00374553 2024-02-09T17:38:15.692 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.692 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.692 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:15.692 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:13.387697+0000", 2024-02-09T17:38:15.692 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00069966799999999997 2024-02-09T17:38:15.692 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.692 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.693 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:15.693 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:13.449414+0000", 2024-02-09T17:38:15.693 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.061717012000000002 2024-02-09T17:38:15.693 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.693 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.693 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:38:15.693 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:13.449476+0000", 2024-02-09T17:38:15.693 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.2012000000000002e-05 2024-02-09T17:38:15.693 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.693 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.693 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:38:15.693 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:13.449520+0000", 2024-02-09T17:38:15.693 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.3948e-05 2024-02-09T17:38:15.693 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:38:15.693 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:38:15.694 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:38:15.694 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.694 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.694 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25201.0:193193 4.c e370/322 4:36bc6d84:::benchmark_data_smithi179_103809_object55727:head v 370'12305, mlcod=370'12302)", 2024-02-09T17:38:15.694 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:38:13.388485+0000", 2024-02-09T17:38:15.694 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 2.1439451319999998, 2024-02-09T17:38:15.694 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.077287192000000005, 2024-02-09T17:38:15.694 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:38:15.694 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:38:15.694 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:38:15.694 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.694 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:38:15.694 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:13.388485+0000", 2024-02-09T17:38:15.694 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:38:15.695 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.695 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.695 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:38:15.695 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:13.388485+0000", 2024-02-09T17:38:15.695 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:38:15.695 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.695 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.695 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:38:15.695 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:13.388486+0000", 2024-02-09T17:38:15.695 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.0800000000000004e-07 2024-02-09T17:38:15.695 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.695 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.695 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:38:15.695 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:13.388490+0000", 2024-02-09T17:38:15.695 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.5240000000000002e-06 2024-02-09T17:38:15.696 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.696 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.696 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:38:15.696 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:13.388498+0000", 2024-02-09T17:38:15.696 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.0770000000000008e-06 2024-02-09T17:38:15.696 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.696 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.696 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:38:15.696 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:13.388503+0000", 2024-02-09T17:38:15.697 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.7940000000000002e-06 2024-02-09T17:38:15.697 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.697 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.697 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:38:15.697 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:13.388542+0000", 2024-02-09T17:38:15.697 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9512999999999999e-05 2024-02-09T17:38:15.697 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.697 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.697 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:38:15.697 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:13.388556+0000", 2024-02-09T17:38:15.697 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3113e-05 2024-02-09T17:38:15.697 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.697 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.697 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:38:15.698 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:13.465728+0000", 2024-02-09T17:38:15.698 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.077172555000000004 2024-02-09T17:38:15.698 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.698 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.698 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:38:15.698 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:13.465772+0000", 2024-02-09T17:38:15.698 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.3908e-05 2024-02-09T17:38:15.698 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:38:15.698 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:38:15.698 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:38:15.698 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.698 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.698 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop_reply(client.25201.0:193208 4.0 e370/322 ondisk, result = 0)", 2024-02-09T17:38:15.698 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:38:13.391508+0000", 2024-02-09T17:38:15.698 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 2.1409216130000002, 2024-02-09T17:38:15.699 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.058017742999999997, 2024-02-09T17:38:15.699 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:38:15.699 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T17:38:15.699 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:38:15.699 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.699 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:38:15.699 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:13.391508+0000", 2024-02-09T17:38:15.699 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:38:15.699 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.699 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.699 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:38:15.699 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:13.391508+0000", 2024-02-09T17:38:15.699 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:38:15.699 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.700 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.700 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:38:15.700 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:13.391510+0000", 2024-02-09T17:38:15.700 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.6899999999999999e-06 2024-02-09T17:38:15.700 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.700 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.700 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:38:15.700 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:13.391519+0000", 2024-02-09T17:38:15.700 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.7900000000000005e-06 2024-02-09T17:38:15.700 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.700 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.700 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:38:15.700 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:13.449026+0000", 2024-02-09T17:38:15.700 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.057506651999999998 2024-02-09T17:38:15.700 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.701 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.701 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:38:15.701 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:13.449050+0000", 2024-02-09T17:38:15.701 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.4817e-05 2024-02-09T17:38:15.701 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.701 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.701 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:38:15.701 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:13.449387+0000", 2024-02-09T17:38:15.701 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00033664300000000002 2024-02-09T17:38:15.701 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.701 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.701 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:38:15.701 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:13.449408+0000", 2024-02-09T17:38:15.701 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.1369000000000002e-05 2024-02-09T17:38:15.702 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:38:15.702 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:38:15.702 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:38:15.702 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:13.449526+0000", 2024-02-09T17:38:15.702 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000117782 2024-02-09T17:38:15.702 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:38:15.702 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:38:15.702 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:38:15.702 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:38:15.702 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:38:15.702 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:38:15.702 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:38:15.757 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:15.876 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:38:15.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:15.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:38:15.983 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:38:16.012 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:16.048 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:38:16.049 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:16 smithi099 ceph-mon[39420]: pgmap v1015: 37 pgs: 37 active+clean; 5.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 607 op/s 2024-02-09T17:38:16.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:16.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:38:16.154 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:38:16.185 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:16.210 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:38:16.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:16.214 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:38:16.315 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:38:16.353 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:16 smithi179 ceph-mon[43975]: pgmap v1015: 37 pgs: 37 active+clean; 5.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 607 op/s 2024-02-09T17:38:16.354 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:16.384 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:38:16.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:16.392 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:38:16.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:16 smithi162 ceph-mon[44304]: pgmap v1015: 37 pgs: 37 active+clean; 5.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 607 op/s 2024-02-09T17:38:16.493 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:38:16.538 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:16.562 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:38:16.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:16.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:38:16.667 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:38:16.733 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:17.080 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is unset 2024-02-09T17:38:17.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:17 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3510989778' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:38:17.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:17 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:38:17.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:17 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3510989778' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:38:17.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:17 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:38:17.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:17 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3510989778' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:38:17.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:17 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:38:18.217 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:38:18.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:18.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:38:18.249 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-09T17:38:18.330 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:38:18.363 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:18 smithi179 ceph-mon[43975]: pgmap v1016: 37 pgs: 37 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 996 op/s 2024-02-09T17:38:18.363 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:18 smithi179 ceph-mon[43975]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:38:18.363 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:18 smithi179 ceph-mon[43975]: Cluster is now healthy 2024-02-09T17:38:18.363 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:18 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:38:18.364 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:18 smithi179 ceph-mon[43975]: osdmap e371: 12 total, 12 up, 12 in 2024-02-09T17:38:18.364 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:18 smithi179 ceph-mon[43975]: 1.e scrub starts 2024-02-09T17:38:18.364 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:18 smithi179 ceph-mon[43975]: 1.e scrub ok 2024-02-09T17:38:18.370 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:18.403 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:38:18.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:18.408 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:38:18.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:18 smithi162 ceph-mon[44304]: pgmap v1016: 37 pgs: 37 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 996 op/s 2024-02-09T17:38:18.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:18 smithi162 ceph-mon[44304]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:38:18.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:18 smithi162 ceph-mon[44304]: Cluster is now healthy 2024-02-09T17:38:18.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:18 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:38:18.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:18 smithi162 ceph-mon[44304]: osdmap e371: 12 total, 12 up, 12 in 2024-02-09T17:38:18.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:18 smithi162 ceph-mon[44304]: 1.e scrub starts 2024-02-09T17:38:18.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:18 smithi162 ceph-mon[44304]: 1.e scrub ok 2024-02-09T17:38:18.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:18 smithi099 ceph-mon[39420]: pgmap v1016: 37 pgs: 37 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 996 op/s 2024-02-09T17:38:18.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:18 smithi099 ceph-mon[39420]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:38:18.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:18 smithi099 ceph-mon[39420]: Cluster is now healthy 2024-02-09T17:38:18.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:18 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:38:18.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:18 smithi099 ceph-mon[39420]: osdmap e371: 12 total, 12 up, 12 in 2024-02-09T17:38:18.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:18 smithi099 ceph-mon[39420]: 1.e scrub starts 2024-02-09T17:38:18.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:18 smithi099 ceph-mon[39420]: 1.e scrub ok 2024-02-09T17:38:18.508 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:38:18.547 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:18.612 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:38:18.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:18.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:38:18.652 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:38:18.732 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:38:18.761 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:18.789 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:38:18.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:18.792 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:38:18.893 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:38:18.934 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:18.961 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:38:18.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:18.967 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:38:19.068 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:38:19.129 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:19.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:19 smithi162 ceph-mon[44304]: 1.2 scrub starts 2024-02-09T17:38:19.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:19 smithi162 ceph-mon[44304]: 1.2 scrub ok 2024-02-09T17:38:19.406 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] 2024-02-09T17:38:19.406 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:38:19.407 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-02-09T17:38:19.407 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight-by-utilization 110 0.05 2024-02-09T17:38:19.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:19 smithi179 ceph-mon[43975]: 1.2 scrub starts 2024-02-09T17:38:19.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:19 smithi179 ceph-mon[43975]: 1.2 scrub ok 2024-02-09T17:38:19.445 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:19 smithi099 ceph-mon[39420]: 1.2 scrub starts 2024-02-09T17:38:19.445 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:19 smithi099 ceph-mon[39420]: 1.2 scrub ok 2024-02-09T17:38:19.480 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:38:19.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:19.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:38:19.582 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:38:19.612 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:19.636 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:38:19.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:19.643 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:38:19.744 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:38:19.809 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:19.901 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:38:19.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:19.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:38:20.012 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:38:20.086 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:20.130 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:38:20.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:20.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:38:20.242 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:38:20.301 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:38:20.333 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:20 smithi099 ceph-mon[39420]: pgmap v1018: 37 pgs: 37 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 679 op/s 2024-02-09T17:38:20.333 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:20 smithi099 ceph-mon[39420]: 4.6 scrub starts 2024-02-09T17:38:20.335 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:20.342 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:38:20.401 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:38:20.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:20 smithi179 ceph-mon[43975]: pgmap v1018: 37 pgs: 37 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 679 op/s 2024-02-09T17:38:20.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:20 smithi179 ceph-mon[43975]: 4.6 scrub starts 2024-02-09T17:38:20.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:20 smithi162 ceph-mon[44304]: pgmap v1018: 37 pgs: 37 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 679 op/s 2024-02-09T17:38:20.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:20 smithi162 ceph-mon[44304]: 4.6 scrub starts 2024-02-09T17:38:20.435 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:38:20.472 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:38:20.513 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:38:20.519 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:38:20.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:20.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:38:20.548 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:38:20.601 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:38:20.622 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:38:20.644 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:38:20.678 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:38:20.703 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:20.722 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:38:20.751 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:38:20.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:20.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:38:20.771 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:38:20.815 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:38:20.815 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:38:20.816 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:38:20.848 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:38:20.848 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:38:20.848 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:38:20.848 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:38:20.849 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:38:20.849 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:38:20.849 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:38:20.849 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:38:20.849 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:38:20.849 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:38:20.849 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:38:20.849 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:38:20.862 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:38:20.911 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:21.040 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set noscrub 2024-02-09T17:38:21.078 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:38:21.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:21.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:38:21.123 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:21 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/551596596' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:38:21.124 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:21 smithi162 ceph-mon[44304]: 4.6 scrub ok 2024-02-09T17:38:21.124 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:21 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:38:21.124 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:21 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:38:21.186 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:38:21.233 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:21.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:21 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/551596596' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:38:21.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:21 smithi179 ceph-mon[43975]: 4.6 scrub ok 2024-02-09T17:38:21.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:21 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:38:21.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:21 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:38:21.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:21 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/551596596' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:38:21.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:21 smithi099 ceph-mon[39420]: 4.6 scrub ok 2024-02-09T17:38:21.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:21 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:38:21.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:21 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:38:21.884 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-09T17:38:22.101 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:38:22.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:22.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:38:22.141 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:22 smithi162 ceph-mon[44304]: pgmap v1019: 37 pgs: 37 active+clean; 4.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 894 op/s 2024-02-09T17:38:22.212 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:38:22.274 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:22.306 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:38:22.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:22 smithi179 ceph-mon[43975]: pgmap v1019: 37 pgs: 37 active+clean; 4.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 894 op/s 2024-02-09T17:38:22.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:22 smithi179 ceph-mon[43975]: osdmap e372: 12 total, 12 up, 12 in 2024-02-09T17:38:22.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:22 smithi179 ceph-mon[43975]: 1.0 scrub starts 2024-02-09T17:38:22.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:22 smithi179 ceph-mon[43975]: 1.0 scrub ok 2024-02-09T17:38:22.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:22 smithi162 ceph-mon[44304]: osdmap e372: 12 total, 12 up, 12 in 2024-02-09T17:38:22.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:22 smithi162 ceph-mon[44304]: 1.0 scrub starts 2024-02-09T17:38:22.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:22 smithi162 ceph-mon[44304]: 1.0 scrub ok 2024-02-09T17:38:22.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:22 smithi099 ceph-mon[39420]: pgmap v1019: 37 pgs: 37 active+clean; 4.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 894 op/s 2024-02-09T17:38:22.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:22 smithi099 ceph-mon[39420]: osdmap e372: 12 total, 12 up, 12 in 2024-02-09T17:38:22.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:22 smithi099 ceph-mon[39420]: 1.0 scrub starts 2024-02-09T17:38:22.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:22 smithi099 ceph-mon[39420]: 1.0 scrub ok 2024-02-09T17:38:22.654 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:38:22.659 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:22.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:38:22.760 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:38:22.824 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:23.044 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:38:23.051 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:23.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:38:23.152 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:38:23.198 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:23.228 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:38:23.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:23.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:38:23.338 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:38:23.422 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:23 smithi099 ceph-mon[39420]: pgmap v1021: 37 pgs: 37 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s 2024-02-09T17:38:23.423 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:23.452 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:38:23.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:23.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:38:23.475 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:38:23.567 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:38:23.635 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:23.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:23 smithi179 ceph-mon[43975]: pgmap v1021: 37 pgs: 37 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s 2024-02-09T17:38:23.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:23 smithi162 ceph-mon[44304]: pgmap v1021: 37 pgs: 37 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s 2024-02-09T17:38:23.686 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:38:23.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:23.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:38:23.791 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:38:23.831 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:23.856 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:38:23.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:23.861 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:38:23.962 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:38:24.051 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:24.117 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:38:24.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:24.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:38:24.217 INFO:teuthology.orchestra.run.smithi099.stdout:moved 3 / 111 (2.7027%) 2024-02-09T17:38:24.217 INFO:teuthology.orchestra.run.smithi099.stdout:avg 9.25 2024-02-09T17:38:24.217 INFO:teuthology.orchestra.run.smithi099.stdout:stddev 2.89036 -> 2.71186 (expected baseline 2.9119) 2024-02-09T17:38:24.217 INFO:teuthology.orchestra.run.smithi099.stdout:min osd.8 with 3 -> 3 pgs (0.324324 -> 0.324324 * mean) 2024-02-09T17:38:24.217 INFO:teuthology.orchestra.run.smithi099.stdout:max osd.6 with 14 -> 14 pgs (1.51351 -> 1.51351 * mean) 2024-02-09T17:38:24.217 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:38:24.217 INFO:teuthology.orchestra.run.smithi099.stdout:oload 110 2024-02-09T17:38:24.217 INFO:teuthology.orchestra.run.smithi099.stdout:max_change 0.05 2024-02-09T17:38:24.218 INFO:teuthology.orchestra.run.smithi099.stdout:max_change_osds 4 2024-02-09T17:38:24.218 INFO:teuthology.orchestra.run.smithi099.stdout:average_utilization 0.0160 2024-02-09T17:38:24.218 INFO:teuthology.orchestra.run.smithi099.stdout:overload_utilization 0.0177 2024-02-09T17:38:24.218 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1 weight 1.0000 -> 0.9500 2024-02-09T17:38:24.218 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11 weight 1.0000 -> 0.9500 2024-02-09T17:38:24.218 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6 weight 1.0000 -> 0.9500 2024-02-09T17:38:24.218 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3 weight 1.0000 -> 0.9500 2024-02-09T17:38:24.223 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:38:24.271 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:24.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:24 smithi162 ceph-mon[44304]: 4.d scrub starts 2024-02-09T17:38:24.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:24 smithi162 ceph-mon[44304]: from='client.25348 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:38:24.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:24 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'62260','3':'62260','6':'62260','11':'62260'}"}]: dispatch 2024-02-09T17:38:24.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:24 smithi099 ceph-mon[39420]: 4.d scrub starts 2024-02-09T17:38:24.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:24 smithi099 ceph-mon[39420]: from='client.25348 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:38:24.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:24 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'62260','3':'62260','6':'62260','11':'62260'}"}]: dispatch 2024-02-09T17:38:24.521 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:38:24.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:24.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:38:24.625 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:38:24.650 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:24 smithi179 ceph-mon[43975]: 4.d scrub starts 2024-02-09T17:38:24.650 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:24 smithi179 ceph-mon[43975]: from='client.25348 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:38:24.650 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:24 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'62260','3':'62260','6':'62260','11':'62260'}"}]: dispatch 2024-02-09T17:38:24.652 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:24.658 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:38:24.658 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:38:24.670 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:38:24.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:24.673 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:38:24.694 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:38:24.774 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:38:24.802 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:24.875 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:38:24.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:24.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:38:24.979 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:38:25.005 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:25.289 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-09T17:38:25.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:25 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:38:25.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:25 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:38:25.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:25 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'62260','3':'62260','6':'62260','11':'62260'}"}]': finished 2024-02-09T17:38:25.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:25 smithi099 ceph-mon[39420]: osdmap e373: 12 total, 12 up, 12 in 2024-02-09T17:38:25.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:25 smithi099 ceph-mon[39420]: 4.d scrub ok 2024-02-09T17:38:25.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:25 smithi099 ceph-mon[39420]: pgmap v1023: 37 pgs: 37 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 602 op/s 2024-02-09T17:38:25.561 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:38:25.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:25 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:38:25.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:25 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:38:25.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:25 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'62260','3':'62260','6':'62260','11':'62260'}"}]': finished 2024-02-09T17:38:25.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:25 smithi179 ceph-mon[43975]: osdmap e373: 12 total, 12 up, 12 in 2024-02-09T17:38:25.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:25 smithi179 ceph-mon[43975]: 4.d scrub ok 2024-02-09T17:38:25.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:25 smithi179 ceph-mon[43975]: pgmap v1023: 37 pgs: 37 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 602 op/s 2024-02-09T17:38:25.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:25 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:38:25.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:25 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:38:25.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:25 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'62260','3':'62260','6':'62260','11':'62260'}"}]': finished 2024-02-09T17:38:25.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:25 smithi162 ceph-mon[44304]: osdmap e373: 12 total, 12 up, 12 in 2024-02-09T17:38:25.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:25 smithi162 ceph-mon[44304]: 4.d scrub ok 2024-02-09T17:38:25.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:25 smithi162 ceph-mon[44304]: pgmap v1023: 37 pgs: 37 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 602 op/s 2024-02-09T17:38:26.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:26 smithi162 ceph-mon[44304]: osdmap e374: 12 total, 12 up, 12 in 2024-02-09T17:38:26.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:26 smithi099 ceph-mon[39420]: osdmap e374: 12 total, 12 up, 12 in 2024-02-09T17:38:26.574 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:38:26.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:26.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:38:26.602 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:38:26.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:26 smithi179 ceph-mon[43975]: osdmap e374: 12 total, 12 up, 12 in 2024-02-09T17:38:26.680 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:38:26.737 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:26.791 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:38:26.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:26.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:38:26.899 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:38:26.955 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:27.232 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:27 smithi099 ceph-mon[39420]: 4.a scrub starts 2024-02-09T17:38:27.232 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:27 smithi099 ceph-mon[39420]: 4.a scrub ok 2024-02-09T17:38:27.232 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:27 smithi099 ceph-mon[39420]: 4.3 starting backfill to osd.0 from (0'0,0'0] MAX to 372'12510 2024-02-09T17:38:27.233 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:27 smithi099 ceph-mon[39420]: 4.7 starting backfill to osd.7 from (0'0,0'0] MAX to 372'12811 2024-02-09T17:38:27.233 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:27 smithi099 ceph-mon[39420]: osdmap e375: 12 total, 12 up, 12 in 2024-02-09T17:38:27.233 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:27 smithi099 ceph-mon[39420]: osdmap e376: 12 total, 12 up, 12 in 2024-02-09T17:38:27.233 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:27 smithi099 ceph-mon[39420]: pgmap v1027: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 642 op/s 2024-02-09T17:38:27.233 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:27 smithi162 ceph-mon[44304]: 4.a scrub starts 2024-02-09T17:38:27.233 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:27 smithi162 ceph-mon[44304]: 4.a scrub ok 2024-02-09T17:38:27.233 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:27 smithi162 ceph-mon[44304]: 4.3 starting backfill to osd.0 from (0'0,0'0] MAX to 372'12510 2024-02-09T17:38:27.234 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:27 smithi162 ceph-mon[44304]: 4.7 starting backfill to osd.7 from (0'0,0'0] MAX to 372'12811 2024-02-09T17:38:27.234 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:27 smithi162 ceph-mon[44304]: osdmap e375: 12 total, 12 up, 12 in 2024-02-09T17:38:27.234 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:27 smithi162 ceph-mon[44304]: osdmap e376: 12 total, 12 up, 12 in 2024-02-09T17:38:27.234 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:27 smithi162 ceph-mon[44304]: pgmap v1027: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 642 op/s 2024-02-09T17:38:27.327 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:38:27.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:27.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:38:27.433 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:38:27.473 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:27.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:27 smithi179 ceph-mon[43975]: 4.a scrub starts 2024-02-09T17:38:27.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:27 smithi179 ceph-mon[43975]: 4.a scrub ok 2024-02-09T17:38:27.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:27 smithi179 ceph-mon[43975]: 4.3 starting backfill to osd.0 from (0'0,0'0] MAX to 372'12510 2024-02-09T17:38:27.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:27 smithi179 ceph-mon[43975]: 4.7 starting backfill to osd.7 from (0'0,0'0] MAX to 372'12811 2024-02-09T17:38:27.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:27 smithi179 ceph-mon[43975]: osdmap e375: 12 total, 12 up, 12 in 2024-02-09T17:38:27.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:27 smithi179 ceph-mon[43975]: osdmap e376: 12 total, 12 up, 12 in 2024-02-09T17:38:27.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:27 smithi179 ceph-mon[43975]: pgmap v1027: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 642 op/s 2024-02-09T17:38:28.349 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-02-09T17:38:28.349 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight-by-utilization 110 3.0 2024-02-09T17:38:28.426 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is set 2024-02-09T17:38:28.483 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:28 smithi099 ceph-mon[39420]: 1.14 deep-scrub starts 2024-02-09T17:38:28.483 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:28 smithi099 ceph-mon[39420]: 1.3 deep-scrub starts 2024-02-09T17:38:28.483 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:28 smithi099 ceph-mon[39420]: 1.14 deep-scrub ok 2024-02-09T17:38:28.483 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:28 smithi099 ceph-mon[39420]: 1.3 deep-scrub ok 2024-02-09T17:38:28.483 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:28 smithi099 ceph-mon[39420]: osdmap e377: 12 total, 12 up, 12 in 2024-02-09T17:38:28.483 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:28 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1698177434' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:38:28.483 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:28 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:38:28.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:28 smithi179 ceph-mon[43975]: 1.14 deep-scrub starts 2024-02-09T17:38:28.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:28 smithi179 ceph-mon[43975]: 1.3 deep-scrub starts 2024-02-09T17:38:28.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:28 smithi179 ceph-mon[43975]: 1.14 deep-scrub ok 2024-02-09T17:38:28.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:28 smithi179 ceph-mon[43975]: 1.3 deep-scrub ok 2024-02-09T17:38:28.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:28 smithi179 ceph-mon[43975]: osdmap e377: 12 total, 12 up, 12 in 2024-02-09T17:38:28.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:28 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1698177434' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:38:28.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:28 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:38:28.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:28 smithi162 ceph-mon[44304]: 1.14 deep-scrub starts 2024-02-09T17:38:28.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:28 smithi162 ceph-mon[44304]: 1.3 deep-scrub starts 2024-02-09T17:38:28.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:28 smithi162 ceph-mon[44304]: 1.14 deep-scrub ok 2024-02-09T17:38:28.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:28 smithi162 ceph-mon[44304]: 1.3 deep-scrub ok 2024-02-09T17:38:28.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:28 smithi162 ceph-mon[44304]: osdmap e377: 12 total, 12 up, 12 in 2024-02-09T17:38:28.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:28 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1698177434' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:38:28.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:28 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:38:29.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:29 smithi099 ceph-mon[39420]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:38:29.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:29 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:38:29.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:29 smithi099 ceph-mon[39420]: osdmap e378: 12 total, 12 up, 12 in 2024-02-09T17:38:29.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:29 smithi099 ceph-mon[39420]: pgmap v1030: 37 pgs: 2 active+remapped+backfilling, 1 active+clean+scrubbing, 34 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 25 KiB/s wr, 883 op/s; 8697/216321 objects misplaced (4.020%); 6.8 MiB/s, 108 objects/s recovering 2024-02-09T17:38:29.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:29 smithi179 ceph-mon[43975]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:38:29.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:29 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:38:29.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:29 smithi179 ceph-mon[43975]: osdmap e378: 12 total, 12 up, 12 in 2024-02-09T17:38:29.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:29 smithi179 ceph-mon[43975]: pgmap v1030: 37 pgs: 2 active+remapped+backfilling, 1 active+clean+scrubbing, 34 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 25 KiB/s wr, 883 op/s; 8697/216321 objects misplaced (4.020%); 6.8 MiB/s, 108 objects/s recovering 2024-02-09T17:38:29.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:29 smithi162 ceph-mon[44304]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:38:29.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:29 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:38:29.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:29 smithi162 ceph-mon[44304]: osdmap e378: 12 total, 12 up, 12 in 2024-02-09T17:38:29.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:29 smithi162 ceph-mon[44304]: pgmap v1030: 37 pgs: 2 active+remapped+backfilling, 1 active+clean+scrubbing, 34 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 25 KiB/s wr, 883 op/s; 8697/216321 objects misplaced (4.020%); 6.8 MiB/s, 108 objects/s recovering 2024-02-09T17:38:30.201 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:38:30.201 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:38:30.201 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:38:30.201 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:38:30.201 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:38:30.201 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:38:30.202 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:38:30.202 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:38:30.219 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:38:30.220 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:38:30.256 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:38:30.288 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:38:30.320 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:38:30.323 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:38:30.323 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:38:30.324 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:38:30.324 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:38:30.324 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:38:30.324 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:38:30.324 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:38:30.324 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:38:30.324 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:38:30.324 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:38:30.324 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:38:30.324 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:38:30.602 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:30 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1887521645' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:38:30.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:30 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1887521645' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:38:30.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:30 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1887521645' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:38:30.969 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:38:30.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:30.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:38:31.085 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:38:31.125 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:31.158 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:38:31.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:31.166 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:38:31.267 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:38:31.313 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:31.352 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:38:31.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:31.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:38:31.471 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:38:31.512 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:31.545 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:38:31.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:31.548 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:38:31.649 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:38:31.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:31 smithi162 ceph-mon[44304]: pgmap v1031: 37 pgs: 2 active+remapped+backfilling, 1 active+clean+scrubbing, 34 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 912 B/s rd, 20 KiB/s wr, 602 op/s; 8697/211572 objects misplaced (4.111%); 5.3 MiB/s, 84 objects/s recovering 2024-02-09T17:38:31.686 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:31.708 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:38:31.713 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:31 smithi179 ceph-mon[43975]: pgmap v1031: 37 pgs: 2 active+remapped+backfilling, 1 active+clean+scrubbing, 34 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 912 B/s rd, 20 KiB/s wr, 602 op/s; 8697/211572 objects misplaced (4.111%); 5.3 MiB/s, 84 objects/s recovering 2024-02-09T17:38:31.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:31.714 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:38:31.731 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:31 smithi099 ceph-mon[39420]: pgmap v1031: 37 pgs: 2 active+remapped+backfilling, 1 active+clean+scrubbing, 34 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 912 B/s rd, 20 KiB/s wr, 602 op/s; 8697/211572 objects misplaced (4.111%); 5.3 MiB/s, 84 objects/s recovering 2024-02-09T17:38:31.738 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:38:31.738 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-02-09T17:38:31.738 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-02-09T17:38:31.738 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-02-09T17:38:31.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.738 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:141301 3.3 3:c21b7a5c:::benchmark_data_smithi179_90440_object9342:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-02-09T17:38:31.738 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:31:59.267938+0000", 2024-02-09T17:38:31.738 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 392.46633989499998, 2024-02-09T17:38:31.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3659083160000001, 2024-02-09T17:38:31.739 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:38:31.739 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:38:31.739 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:38:31.739 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:38:31.739 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:38:31.739 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 141301 2024-02-09T17:38:31.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.739 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:38:31.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.739 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:38:31.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.267938+0000", 2024-02-09T17:38:31.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:38:31.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.740 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.740 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:38:31.740 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.267938+0000", 2024-02-09T17:38:31.740 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:38:31.740 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.740 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.740 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:38:31.740 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.267940+0000", 2024-02-09T17:38:31.740 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6160000000000001e-06 2024-02-09T17:38:31.740 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.740 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.740 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:38:31.740 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.267945+0000", 2024-02-09T17:38:31.740 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6319999999999996e-06 2024-02-09T17:38:31.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:38:31.741 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.267959+0000", 2024-02-09T17:38:31.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3616e-05 2024-02-09T17:38:31.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:38:31.741 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.267966+0000", 2024-02-09T17:38:31.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.002e-06 2024-02-09T17:38:31.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:38:31.741 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.279726+0000", 2024-02-09T17:38:31.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011759591999999999 2024-02-09T17:38:31.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:38:31.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.279735+0000", 2024-02-09T17:38:31.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8050000000000001e-06 2024-02-09T17:38:31.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:38:31.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:00.271890+0000", 2024-02-09T17:38:31.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99215431499999995 2024-02-09T17:38:31.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:38:31.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:00.271903+0000", 2024-02-09T17:38:31.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3284e-05 2024-02-09T17:38:31.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.743 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:38:31.743 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.608947+0000", 2024-02-09T17:38:31.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.337043838 2024-02-09T17:38:31.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.743 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:38:31.743 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.609112+0000", 2024-02-09T17:38:31.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016556599999999999 2024-02-09T17:38:31.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.743 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-02-09T17:38:31.744 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.609285+0000", 2024-02-09T17:38:31.744 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017263599999999999 2024-02-09T17:38:31.744 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.744 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.744 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:31.744 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.619077+0000", 2024-02-09T17:38:31.744 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0097920879999999991 2024-02-09T17:38:31.744 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.744 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.744 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:38:31.744 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.619097+0000", 2024-02-09T17:38:31.744 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9426999999999999e-05 2024-02-09T17:38:31.744 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.744 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.745 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:31.745 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.627918+0000", 2024-02-09T17:38:31.745 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0088214560000000001 2024-02-09T17:38:31.745 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.745 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.745 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:38:31.745 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.627979+0000", 2024-02-09T17:38:31.745 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1205999999999996e-05 2024-02-09T17:38:31.745 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.745 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.745 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:38:31.745 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.633847+0000", 2024-02-09T17:38:31.745 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.005867237 2024-02-09T17:38:31.745 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:38:31.746 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:38:31.746 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:38:31.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.746 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:141303 3.3 3:c41f7f3f:::benchmark_data_smithi179_90440_object9344:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-02-09T17:38:31.746 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:31:59.274155+0000", 2024-02-09T17:38:31.746 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 392.460123377, 2024-02-09T17:38:31.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.411265985, 2024-02-09T17:38:31.746 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:38:31.746 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:38:31.746 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:38:31.746 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:38:31.746 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:38:31.746 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 141303 2024-02-09T17:38:31.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.747 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:38:31.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.747 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:38:31.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.274155+0000", 2024-02-09T17:38:31.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:38:31.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.747 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:38:31.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.274155+0000", 2024-02-09T17:38:31.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:38:31.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.747 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:38:31.748 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.274156+0000", 2024-02-09T17:38:31.748 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3659999999999999e-06 2024-02-09T17:38:31.748 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.748 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.748 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:38:31.748 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.274161+0000", 2024-02-09T17:38:31.748 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5469999999999998e-06 2024-02-09T17:38:31.748 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.748 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.748 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:38:31.748 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.274172+0000", 2024-02-09T17:38:31.748 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1411e-05 2024-02-09T17:38:31.748 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.748 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.749 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:38:31.749 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.274181+0000", 2024-02-09T17:38:31.749 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7660000000000004e-06 2024-02-09T17:38:31.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.749 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.749 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:38:31.749 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.280072+0000", 2024-02-09T17:38:31.749 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.005891444 2024-02-09T17:38:31.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.749 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.749 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:38:31.749 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.280079+0000", 2024-02-09T17:38:31.749 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6359999999999997e-06 2024-02-09T17:38:31.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.750 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:38:31.750 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:00.271993+0000", 2024-02-09T17:38:31.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99191438700000001 2024-02-09T17:38:31.750 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.750 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:38:31.750 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:00.272003+0000", 2024-02-09T17:38:31.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5089999999999999e-06 2024-02-09T17:38:31.750 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.750 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:38:31.750 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.633918+0000", 2024-02-09T17:38:31.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.361915142 2024-02-09T17:38:31.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:38:31.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.634089+0000", 2024-02-09T17:38:31.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000170552 2024-02-09T17:38:31.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-02-09T17:38:31.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.634254+0000", 2024-02-09T17:38:31.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016508699999999999 2024-02-09T17:38:31.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:31.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.647777+0000", 2024-02-09T17:38:31.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013523795 2024-02-09T17:38:31.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.752 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:38:31.752 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.647858+0000", 2024-02-09T17:38:31.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0488999999999995e-05 2024-02-09T17:38:31.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.752 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:31.752 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.680537+0000", 2024-02-09T17:38:31.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.032679407000000001 2024-02-09T17:38:31.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.752 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:38:31.752 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.680600+0000", 2024-02-09T17:38:31.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3167e-05 2024-02-09T17:38:31.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.753 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:38:31.753 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.685421+0000", 2024-02-09T17:38:31.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0048202799999999997 2024-02-09T17:38:31.753 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:38:31.753 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:38:31.753 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:38:31.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.753 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.753 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:141326 3.1 3:81768e06:::benchmark_data_smithi179_90440_object9367:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-02-09T17:38:31.753 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:31:59.299353+0000", 2024-02-09T17:38:31.753 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 392.43492477699999, 2024-02-09T17:38:31.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2261435930000002, 2024-02-09T17:38:31.754 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:38:31.754 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:38:31.754 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:38:31.754 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:38:31.754 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:38:31.754 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 141326 2024-02-09T17:38:31.754 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.754 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:38:31.754 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.754 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:38:31.754 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.299353+0000", 2024-02-09T17:38:31.754 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:38:31.754 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.754 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.755 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:38:31.755 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.299353+0000", 2024-02-09T17:38:31.755 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:38:31.755 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.755 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.755 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:38:31.755 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.299355+0000", 2024-02-09T17:38:31.755 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.903e-06 2024-02-09T17:38:31.755 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.755 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.755 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:38:31.755 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.299363+0000", 2024-02-09T17:38:31.755 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5440000000000001e-06 2024-02-09T17:38:31.755 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.755 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.756 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:38:31.756 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.299382+0000", 2024-02-09T17:38:31.756 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9511e-05 2024-02-09T17:38:31.756 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.756 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.756 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:38:31.756 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.299393+0000", 2024-02-09T17:38:31.756 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0428000000000001e-05 2024-02-09T17:38:31.756 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.756 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.756 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:38:31.756 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.299463+0000", 2024-02-09T17:38:31.756 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0080000000000007e-05 2024-02-09T17:38:31.756 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.756 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:38:31.757 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.299474+0000", 2024-02-09T17:38:31.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.149e-05 2024-02-09T17:38:31.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:38:31.757 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:00.271334+0000", 2024-02-09T17:38:31.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.97185930200000004 2024-02-09T17:38:31.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:38:31.757 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:00.271349+0000", 2024-02-09T17:38:31.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5636000000000001e-05 2024-02-09T17:38:31.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.758 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:38:31.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.509536+0000", 2024-02-09T17:38:31.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.23818688 2024-02-09T17:38:31.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.758 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:38:31.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.511667+0000", 2024-02-09T17:38:31.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021310349999999999 2024-02-09T17:38:31.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.758 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 10,11", 2024-02-09T17:38:31.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.511783+0000", 2024-02-09T17:38:31.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011539099999999999 2024-02-09T17:38:31.759 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.759 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:31.759 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.521854+0000", 2024-02-09T17:38:31.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010071663999999999 2024-02-09T17:38:31.759 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.759 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:31.759 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.521897+0000", 2024-02-09T17:38:31.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2874999999999999e-05 2024-02-09T17:38:31.759 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.759 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:38:31.759 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.521910+0000", 2024-02-09T17:38:31.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2609000000000001e-05 2024-02-09T17:38:31.760 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.760 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.760 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:38:31.760 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.521952+0000", 2024-02-09T17:38:31.760 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2703000000000001e-05 2024-02-09T17:38:31.760 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.760 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.760 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:38:31.760 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.525497+0000", 2024-02-09T17:38:31.760 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0035445419999999999 2024-02-09T17:38:31.760 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:38:31.760 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:38:31.760 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:38:31.760 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.761 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.761 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:141334 3.1 3:83b69ab0:::benchmark_data_smithi179_90440_object9375:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-02-09T17:38:31.761 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:31:59.308786+0000", 2024-02-09T17:38:31.761 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 392.425492077, 2024-02-09T17:38:31.761 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2267313619999998, 2024-02-09T17:38:31.761 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:38:31.761 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:38:31.761 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:38:31.761 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:38:31.761 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:38:31.761 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 141334 2024-02-09T17:38:31.761 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.761 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:38:31.761 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.761 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:38:31.762 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.308786+0000", 2024-02-09T17:38:31.762 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:38:31.762 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.762 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.762 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:38:31.762 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.308786+0000", 2024-02-09T17:38:31.762 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:38:31.762 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.762 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.762 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:38:31.762 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.308788+0000", 2024-02-09T17:38:31.762 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0470000000000001e-06 2024-02-09T17:38:31.762 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.762 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.762 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:38:31.763 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.308800+0000", 2024-02-09T17:38:31.763 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.163e-05 2024-02-09T17:38:31.763 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.763 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:38:31.763 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.308816+0000", 2024-02-09T17:38:31.763 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6378000000000001e-05 2024-02-09T17:38:31.763 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.763 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:38:31.763 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.308827+0000", 2024-02-09T17:38:31.763 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1005000000000001e-05 2024-02-09T17:38:31.763 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.763 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:38:31.764 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.308897+0000", 2024-02-09T17:38:31.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0285999999999998e-05 2024-02-09T17:38:31.764 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.764 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.764 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:38:31.764 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.308906+0000", 2024-02-09T17:38:31.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2020000000000004e-06 2024-02-09T17:38:31.764 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.764 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.764 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:38:31.764 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:00.271454+0000", 2024-02-09T17:38:31.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.96254888100000002 2024-02-09T17:38:31.764 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.764 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:38:31.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:00.271464+0000", 2024-02-09T17:38:31.765 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.499e-06 2024-02-09T17:38:31.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:38:31.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.518145+0000", 2024-02-09T17:38:31.765 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.246681012 2024-02-09T17:38:31.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:38:31.766 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.518279+0000", 2024-02-09T17:38:31.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000133807 2024-02-09T17:38:31.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.766 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 10,11", 2024-02-09T17:38:31.766 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.518394+0000", 2024-02-09T17:38:31.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000114863 2024-02-09T17:38:31.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.766 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:31.766 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.525633+0000", 2024-02-09T17:38:31.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0072390129999999999 2024-02-09T17:38:31.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.766 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:38:31.767 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.525649+0000", 2024-02-09T17:38:31.767 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5899000000000001e-05 2024-02-09T17:38:31.767 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.767 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.767 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:31.767 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.531872+0000", 2024-02-09T17:38:31.767 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.006223302 2024-02-09T17:38:31.767 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.767 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.767 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:38:31.767 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.531911+0000", 2024-02-09T17:38:31.767 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8834999999999997e-05 2024-02-09T17:38:31.767 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.767 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.768 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:38:31.768 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.535517+0000", 2024-02-09T17:38:31.768 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0036067030000000002 2024-02-09T17:38:31.768 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:38:31.768 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:38:31.768 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:38:31.769 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.769 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.769 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:141335 3.3 3:c1bb3de7:::benchmark_data_smithi179_90440_object9376:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-02-09T17:38:31.769 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:31:59.308929+0000", 2024-02-09T17:38:31.769 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 392.42534881900002, 2024-02-09T17:38:31.769 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3862364280000001, 2024-02-09T17:38:31.769 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:38:31.769 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:38:31.769 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:38:31.769 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:38:31.769 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:38:31.770 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 141335 2024-02-09T17:38:31.770 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.770 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:38:31.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:38:31.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.308929+0000", 2024-02-09T17:38:31.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:38:31.770 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:38:31.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.308929+0000", 2024-02-09T17:38:31.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:38:31.770 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.771 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:38:31.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.308930+0000", 2024-02-09T17:38:31.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.75e-07 2024-02-09T17:38:31.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.771 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:38:31.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.308934+0000", 2024-02-09T17:38:31.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5360000000000001e-06 2024-02-09T17:38:31.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.771 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:38:31.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.308941+0000", 2024-02-09T17:38:31.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9459999999999998e-06 2024-02-09T17:38:31.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.772 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.772 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:38:31.772 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.308947+0000", 2024-02-09T17:38:31.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0980000000000001e-06 2024-02-09T17:38:31.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.772 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.772 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:38:31.772 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.309001+0000", 2024-02-09T17:38:31.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4840999999999999e-05 2024-02-09T17:38:31.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.772 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.772 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:38:31.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.309011+0000", 2024-02-09T17:38:31.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7329999999999998e-06 2024-02-09T17:38:31.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.773 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:38:31.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:00.272028+0000", 2024-02-09T17:38:31.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.963016647 2024-02-09T17:38:31.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.773 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:38:31.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:00.272042+0000", 2024-02-09T17:38:31.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3910999999999999e-05 2024-02-09T17:38:31.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.774 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:38:31.774 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.638427+0000", 2024-02-09T17:38:31.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.366385634 2024-02-09T17:38:31.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.774 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.774 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:38:31.774 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.638566+0000", 2024-02-09T17:38:31.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000139021 2024-02-09T17:38:31.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.774 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.774 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-02-09T17:38:31.774 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.638703+0000", 2024-02-09T17:38:31.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013641599999999999 2024-02-09T17:38:31.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.775 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.775 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:31.775 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.647842+0000", 2024-02-09T17:38:31.775 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0091392599999999997 2024-02-09T17:38:31.775 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.775 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.775 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:38:31.775 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.647864+0000", 2024-02-09T17:38:31.775 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2294e-05 2024-02-09T17:38:31.775 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.775 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.775 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:31.775 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.690323+0000", 2024-02-09T17:38:31.775 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.042458900000000001 2024-02-09T17:38:31.776 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.776 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.776 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:38:31.776 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.690393+0000", 2024-02-09T17:38:31.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9597000000000005e-05 2024-02-09T17:38:31.776 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.776 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.776 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:38:31.776 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.695166+0000", 2024-02-09T17:38:31.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0047729189999999996 2024-02-09T17:38:31.776 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:38:31.776 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:38:31.776 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:38:31.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.777 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:141338 3.1 3:8fb935f2:::benchmark_data_smithi179_90440_object9379:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-02-09T17:38:31.777 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:31:59.309478+0000", 2024-02-09T17:38:31.777 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 392.42480018399999, 2024-02-09T17:38:31.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2380588619999999, 2024-02-09T17:38:31.777 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:38:31.777 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:38:31.777 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:38:31.777 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:38:31.777 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:38:31.777 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 141338 2024-02-09T17:38:31.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.778 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:38:31.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:38:31.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.309478+0000", 2024-02-09T17:38:31.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:38:31.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:38:31.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.309478+0000", 2024-02-09T17:38:31.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:38:31.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:38:31.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.309479+0000", 2024-02-09T17:38:31.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2099999999999996e-07 2024-02-09T17:38:31.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.779 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:38:31.779 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.309482+0000", 2024-02-09T17:38:31.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5429999999999998e-06 2024-02-09T17:38:31.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.779 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:38:31.779 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.309492+0000", 2024-02-09T17:38:31.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3249999999999997e-06 2024-02-09T17:38:31.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.779 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:38:31.780 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.309497+0000", 2024-02-09T17:38:31.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6119999999999998e-06 2024-02-09T17:38:31.780 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.780 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.780 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:38:31.780 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.309536+0000", 2024-02-09T17:38:31.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9208000000000001e-05 2024-02-09T17:38:31.780 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.780 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.780 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:38:31.780 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.309543+0000", 2024-02-09T17:38:31.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8990000000000004e-06 2024-02-09T17:38:31.780 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.780 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.781 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:38:31.781 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:00.271494+0000", 2024-02-09T17:38:31.781 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.96195103400000004 2024-02-09T17:38:31.781 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.781 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:38:31.781 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:00.271501+0000", 2024-02-09T17:38:31.781 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2430000000000002e-06 2024-02-09T17:38:31.781 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.781 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:38:31.781 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.525677+0000", 2024-02-09T17:38:31.781 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2541769249999999 2024-02-09T17:38:31.782 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.782 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.782 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:38:31.782 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.525798+0000", 2024-02-09T17:38:31.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000120346 2024-02-09T17:38:31.782 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.782 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.782 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 10,11", 2024-02-09T17:38:31.782 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.525908+0000", 2024-02-09T17:38:31.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000109718 2024-02-09T17:38:31.782 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.782 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.782 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:38:31.783 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.535532+0000", 2024-02-09T17:38:31.783 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0096243200000000004 2024-02-09T17:38:31.783 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.783 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:31.783 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.535569+0000", 2024-02-09T17:38:31.783 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7517e-05 2024-02-09T17:38:31.783 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.783 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:31.783 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.544823+0000", 2024-02-09T17:38:31.783 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0092536679999999996 2024-02-09T17:38:31.783 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.784 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.784 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:38:31.784 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.544862+0000", 2024-02-09T17:38:31.784 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9372000000000002e-05 2024-02-09T17:38:31.784 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.784 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.784 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:38:31.784 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.547537+0000", 2024-02-09T17:38:31.784 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0026744109999999998 2024-02-09T17:38:31.784 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:38:31.785 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:38:31.785 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:38:31.785 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.785 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.785 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:141341 3.1 3:87f265c7:::benchmark_data_smithi179_90440_object9382:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-02-09T17:38:31.785 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:31:59.322821+0000", 2024-02-09T17:38:31.785 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 392.41145698499997, 2024-02-09T17:38:31.785 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2283184889999998, 2024-02-09T17:38:31.785 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:38:31.785 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:38:31.785 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:38:31.785 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:38:31.785 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:38:31.785 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 141341 2024-02-09T17:38:31.786 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.786 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:38:31.786 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.786 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:38:31.786 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.322821+0000", 2024-02-09T17:38:31.786 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:38:31.786 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.786 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.786 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:38:31.786 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.322821+0000", 2024-02-09T17:38:31.786 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:38:31.786 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.786 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.786 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:38:31.787 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.322824+0000", 2024-02-09T17:38:31.787 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8260000000000001e-06 2024-02-09T17:38:31.787 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.787 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:38:31.787 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.322836+0000", 2024-02-09T17:38:31.787 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2057e-05 2024-02-09T17:38:31.787 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.787 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:38:31.787 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.322867+0000", 2024-02-09T17:38:31.787 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1186000000000003e-05 2024-02-09T17:38:31.787 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.788 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:38:31.788 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.322884+0000", 2024-02-09T17:38:31.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6676000000000001e-05 2024-02-09T17:38:31.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.788 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:38:31.788 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.322999+0000", 2024-02-09T17:38:31.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011505099999999999 2024-02-09T17:38:31.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.788 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:38:31.788 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:31:59.323020+0000", 2024-02-09T17:38:31.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1120000000000001e-05 2024-02-09T17:38:31.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.789 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:38:31.789 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:00.271533+0000", 2024-02-09T17:38:31.789 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.94851338600000001 2024-02-09T17:38:31.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.789 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:38:31.789 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:00.271543+0000", 2024-02-09T17:38:31.789 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0599999999999997e-06 2024-02-09T17:38:31.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.789 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:38:31.789 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.529155+0000", 2024-02-09T17:38:31.789 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2576121200000001 2024-02-09T17:38:31.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.790 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.790 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:38:31.790 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.529283+0000", 2024-02-09T17:38:31.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012811800000000001 2024-02-09T17:38:31.790 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.790 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.790 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 10,11", 2024-02-09T17:38:31.790 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.529381+0000", 2024-02-09T17:38:31.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7850000000000007e-05 2024-02-09T17:38:31.790 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.790 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.790 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:31.790 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.535605+0000", 2024-02-09T17:38:31.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0062243070000000001 2024-02-09T17:38:31.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:38:31.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.535613+0000", 2024-02-09T17:38:31.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2400000000000007e-06 2024-02-09T17:38:31.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:31.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.547603+0000", 2024-02-09T17:38:31.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011989506 2024-02-09T17:38:31.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:38:31.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.547666+0000", 2024-02-09T17:38:31.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3336000000000005e-05 2024-02-09T17:38:31.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.792 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:38:31.792 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:01.551140+0000", 2024-02-09T17:38:31.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00347365 2024-02-09T17:38:31.792 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:38:31.792 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:38:31.792 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:38:31.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.792 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:145191 3.3 3:cad5c6a9:::benchmark_data_smithi179_90440_object13232:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-02-09T17:38:31.792 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:32:07.951037+0000", 2024-02-09T17:38:31.793 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 383.78324117, 2024-02-09T17:38:31.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29727604600000002, 2024-02-09T17:38:31.793 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:38:31.793 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:38:31.793 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:38:31.793 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:38:31.793 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:38:31.793 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 145191 2024-02-09T17:38:31.793 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.793 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:38:31.793 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.793 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:38:31.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.951037+0000", 2024-02-09T17:38:31.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:38:31.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.794 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:38:31.794 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.951037+0000", 2024-02-09T17:38:31.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:38:31.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.794 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:38:31.794 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.951038+0000", 2024-02-09T17:38:31.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5120000000000001e-06 2024-02-09T17:38:31.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.794 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:38:31.794 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.951043+0000", 2024-02-09T17:38:31.795 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4689999999999999e-06 2024-02-09T17:38:31.795 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.795 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.795 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:38:31.795 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.951056+0000", 2024-02-09T17:38:31.795 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3112000000000001e-05 2024-02-09T17:38:31.795 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.795 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.795 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:38:31.795 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.951066+0000", 2024-02-09T17:38:31.795 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8190000000000003e-06 2024-02-09T17:38:31.795 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.795 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.795 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:38:31.796 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.969513+0000", 2024-02-09T17:38:31.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018447468000000002 2024-02-09T17:38:31.796 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.796 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.796 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for rw locks", 2024-02-09T17:38:31.796 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.969630+0000", 2024-02-09T17:38:31.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000116783 2024-02-09T17:38:31.796 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.796 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.796 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:38:31.796 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:08.209923+0000", 2024-02-09T17:38:31.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.240293012 2024-02-09T17:38:31.796 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.797 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.797 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:38:31.797 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:08.210020+0000", 2024-02-09T17:38:31.797 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7258999999999998e-05 2024-02-09T17:38:31.797 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.797 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.797 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-02-09T17:38:31.797 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:08.210179+0000", 2024-02-09T17:38:31.797 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015853300000000001 2024-02-09T17:38:31.797 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.797 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.797 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:31.797 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:08.234698+0000", 2024-02-09T17:38:31.797 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.024519374 2024-02-09T17:38:31.798 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.798 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.798 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:38:31.798 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:08.234717+0000", 2024-02-09T17:38:31.798 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.906e-05 2024-02-09T17:38:31.798 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.798 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.798 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:31.798 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:08.244438+0000", 2024-02-09T17:38:31.798 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0097207460000000006 2024-02-09T17:38:31.798 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.798 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.798 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:38:31.798 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:08.244515+0000", 2024-02-09T17:38:31.798 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6779999999999993e-05 2024-02-09T17:38:31.799 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.799 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:38:31.799 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:08.248313+0000", 2024-02-09T17:38:31.799 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003798119 2024-02-09T17:38:31.799 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:38:31.799 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:38:31.799 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:38:31.799 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.799 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:145196 3.3 3:c8843776:::benchmark_data_smithi179_90440_object13237:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-02-09T17:38:31.799 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:32:07.956283+0000", 2024-02-09T17:38:31.799 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 383.77799506399998, 2024-02-09T17:38:31.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23580672, 2024-02-09T17:38:31.800 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:38:31.800 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:38:31.800 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:38:31.800 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:38:31.800 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:38:31.800 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 145196 2024-02-09T17:38:31.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.800 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:38:31.800 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.800 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:38:31.800 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.956283+0000", 2024-02-09T17:38:31.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:38:31.801 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.801 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.801 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:38:31.801 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.956283+0000", 2024-02-09T17:38:31.801 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:38:31.801 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.801 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.801 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:38:31.801 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.956285+0000", 2024-02-09T17:38:31.801 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9690000000000001e-06 2024-02-09T17:38:31.801 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.801 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.801 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:38:31.801 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.956292+0000", 2024-02-09T17:38:31.802 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0480000000000001e-06 2024-02-09T17:38:31.802 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.802 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.802 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:38:31.802 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.956311+0000", 2024-02-09T17:38:31.802 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8859e-05 2024-02-09T17:38:31.802 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.802 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.802 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:38:31.802 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.956321+0000", 2024-02-09T17:38:31.802 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0199e-05 2024-02-09T17:38:31.802 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.802 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.802 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:38:31.803 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.969660+0000", 2024-02-09T17:38:31.803 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013339177000000001 2024-02-09T17:38:31.803 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.803 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.803 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:38:31.803 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.969809+0000", 2024-02-09T17:38:31.803 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000148462 2024-02-09T17:38:31.803 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.803 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.803 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-02-09T17:38:31.803 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.969975+0000", 2024-02-09T17:38:31.803 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016594900000000001 2024-02-09T17:38:31.803 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.803 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.804 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:38:31.804 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.984771+0000", 2024-02-09T17:38:31.804 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014796444000000001 2024-02-09T17:38:31.804 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.804 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.804 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:31.804 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.986213+0000", 2024-02-09T17:38:31.804 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001441634 2024-02-09T17:38:31.804 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.804 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.804 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:31.804 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:08.187179+0000", 2024-02-09T17:38:31.804 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20096615100000001 2024-02-09T17:38:31.804 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.804 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.805 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:38:31.805 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:08.187259+0000", 2024-02-09T17:38:31.805 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9784999999999996e-05 2024-02-09T17:38:31.805 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.805 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.805 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:38:31.805 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:08.192090+0000", 2024-02-09T17:38:31.805 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0048310430000000001 2024-02-09T17:38:31.805 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:38:31.805 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:38:31.805 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:38:31.805 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.805 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.805 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:145204 3.3 3:c8983e63:::benchmark_data_smithi179_90440_object13245:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-02-09T17:38:31.805 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:32:07.964287+0000", 2024-02-09T17:38:31.806 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 383.76999068600003, 2024-02-09T17:38:31.806 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23277460699999999, 2024-02-09T17:38:31.806 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:38:31.806 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:38:31.806 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:38:31.806 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:38:31.806 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:38:31.806 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 145204 2024-02-09T17:38:31.806 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.806 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:38:31.806 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.806 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:38:31.806 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.964287+0000", 2024-02-09T17:38:31.806 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:38:31.806 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.807 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.807 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:38:31.807 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.964287+0000", 2024-02-09T17:38:31.807 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:38:31.807 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.807 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.807 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:38:31.807 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.964289+0000", 2024-02-09T17:38:31.807 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.889e-06 2024-02-09T17:38:31.807 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.807 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.807 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:38:31.807 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.964294+0000", 2024-02-09T17:38:31.807 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.028e-06 2024-02-09T17:38:31.808 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.808 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.808 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:38:31.808 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.964306+0000", 2024-02-09T17:38:31.808 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1761e-05 2024-02-09T17:38:31.808 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.808 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.808 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:38:31.808 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.964315+0000", 2024-02-09T17:38:31.808 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9290000000000004e-06 2024-02-09T17:38:31.808 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.808 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.808 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:38:31.808 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.974776+0000", 2024-02-09T17:38:31.809 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010460661 2024-02-09T17:38:31.809 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.809 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.809 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:38:31.809 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.974939+0000", 2024-02-09T17:38:31.809 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016313699999999999 2024-02-09T17:38:31.809 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.809 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.809 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-02-09T17:38:31.809 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.975094+0000", 2024-02-09T17:38:31.809 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015559699999999999 2024-02-09T17:38:31.809 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.809 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.809 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:38:31.809 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.984825+0000", 2024-02-09T17:38:31.810 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0097302139999999992 2024-02-09T17:38:31.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.810 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.810 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:31.810 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.986252+0000", 2024-02-09T17:38:31.810 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014271380000000001 2024-02-09T17:38:31.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.810 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.810 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:31.810 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:08.192160+0000", 2024-02-09T17:38:31.810 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20590793499999999 2024-02-09T17:38:31.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.810 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.810 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:38:31.810 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:08.192226+0000", 2024-02-09T17:38:31.811 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6156999999999997e-05 2024-02-09T17:38:31.811 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.811 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.811 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:38:31.811 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:08.197062+0000", 2024-02-09T17:38:31.811 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0048361610000000003 2024-02-09T17:38:31.811 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:38:31.811 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:38:31.811 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:38:31.811 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.811 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.811 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:145210 3.3 3:ceedd519:::benchmark_data_smithi179_90440_object13251:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-02-09T17:38:31.811 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:32:07.974637+0000", 2024-02-09T17:38:31.811 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 383.759641278, 2024-02-09T17:38:31.812 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.227001813, 2024-02-09T17:38:31.812 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:38:31.812 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:38:31.812 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:38:31.812 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:38:31.812 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:38:31.812 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 145210 2024-02-09T17:38:31.812 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.812 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:38:31.812 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.812 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:38:31.812 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.974637+0000", 2024-02-09T17:38:31.812 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:38:31.812 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.813 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.813 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:38:31.813 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.974637+0000", 2024-02-09T17:38:31.813 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:38:31.813 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.813 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.813 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:38:31.813 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.974638+0000", 2024-02-09T17:38:31.813 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3400000000000001e-06 2024-02-09T17:38:31.813 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.813 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.813 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:38:31.813 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.974645+0000", 2024-02-09T17:38:31.813 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1980000000000004e-06 2024-02-09T17:38:31.814 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.814 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.814 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:38:31.814 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.974660+0000", 2024-02-09T17:38:31.814 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5028000000000001e-05 2024-02-09T17:38:31.814 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.814 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.814 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:38:31.814 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.974669+0000", 2024-02-09T17:38:31.814 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5499999999999995e-06 2024-02-09T17:38:31.814 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.814 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.815 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:38:31.815 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.979748+0000", 2024-02-09T17:38:31.815 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0050785989999999996 2024-02-09T17:38:31.815 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.815 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.815 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:38:31.815 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.979931+0000", 2024-02-09T17:38:31.815 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018370300000000001 2024-02-09T17:38:31.815 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.815 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.815 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-02-09T17:38:31.815 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.980100+0000", 2024-02-09T17:38:31.815 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016823399999999999 2024-02-09T17:38:31.815 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.816 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.816 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:38:31.816 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.984894+0000", 2024-02-09T17:38:31.816 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0047940969999999998 2024-02-09T17:38:31.816 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.816 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.816 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:31.816 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.986286+0000", 2024-02-09T17:38:31.816 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001392503 2024-02-09T17:38:31.816 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.816 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.816 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:31.816 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:08.197144+0000", 2024-02-09T17:38:31.816 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.210857987 2024-02-09T17:38:31.817 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.817 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.817 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:38:31.817 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:08.197216+0000", 2024-02-09T17:38:31.817 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1845000000000001e-05 2024-02-09T17:38:31.817 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.817 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.817 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:38:31.817 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:08.201639+0000", 2024-02-09T17:38:31.817 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0044227290000000002 2024-02-09T17:38:31.817 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:38:31.817 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:38:31.817 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:38:31.817 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.817 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.818 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:145211 3.1 3:8971f3c3:::benchmark_data_smithi179_90440_object13252:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-02-09T17:38:31.818 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:32:07.975326+0000", 2024-02-09T17:38:31.818 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 383.758952454, 2024-02-09T17:38:31.818 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23558217200000001, 2024-02-09T17:38:31.818 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:38:31.818 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:38:31.818 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:38:31.818 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:38:31.818 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:38:31.818 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 145211 2024-02-09T17:38:31.818 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.818 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:38:31.818 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.818 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:38:31.818 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.975326+0000", 2024-02-09T17:38:31.819 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:38:31.819 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.819 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.819 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:38:31.819 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.975326+0000", 2024-02-09T17:38:31.819 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:38:31.819 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.819 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.819 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:38:31.819 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.975326+0000", 2024-02-09T17:38:31.819 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1999999999999998e-07 2024-02-09T17:38:31.819 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.819 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.819 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:38:31.820 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.975330+0000", 2024-02-09T17:38:31.820 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2040000000000002e-06 2024-02-09T17:38:31.820 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.820 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.820 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:38:31.820 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.975339+0000", 2024-02-09T17:38:31.820 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6379999999999999e-06 2024-02-09T17:38:31.820 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.820 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.820 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:38:31.820 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.975346+0000", 2024-02-09T17:38:31.820 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6830000000000002e-06 2024-02-09T17:38:31.821 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.821 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.821 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:38:31.821 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.982174+0000", 2024-02-09T17:38:31.821 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0068282500000000001 2024-02-09T17:38:31.821 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.821 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.821 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:38:31.821 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.982281+0000", 2024-02-09T17:38:31.821 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010708 2024-02-09T17:38:31.821 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.821 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.822 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 10,11", 2024-02-09T17:38:31.822 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:07.982401+0000", 2024-02-09T17:38:31.822 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000119327 2024-02-09T17:38:31.822 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.822 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.822 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:31.822 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:08.001265+0000", 2024-02-09T17:38:31.822 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018864715000000001 2024-02-09T17:38:31.822 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.822 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.822 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:38:31.822 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:08.004990+0000", 2024-02-09T17:38:31.822 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0037250740000000001 2024-02-09T17:38:31.822 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.823 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.823 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:31.823 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:08.207271+0000", 2024-02-09T17:38:31.823 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20228028300000001 2024-02-09T17:38:31.823 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.823 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.823 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:38:31.823 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:08.207313+0000", 2024-02-09T17:38:31.823 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2364000000000003e-05 2024-02-09T17:38:31.823 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.823 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.823 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:38:31.823 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:08.210908+0000", 2024-02-09T17:38:31.823 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003594734 2024-02-09T17:38:31.824 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:38:31.824 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:38:31.824 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:38:31.824 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.824 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.824 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:157800 3.1 3:8df3bfb7:::benchmark_data_smithi179_90440_object25841:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-02-09T17:38:31.824 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:32:28.517351+0000", 2024-02-09T17:38:31.824 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 363.21692708400002, 2024-02-09T17:38:31.824 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27107035499999999, 2024-02-09T17:38:31.824 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:38:31.824 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:38:31.824 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:38:31.824 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:38:31.825 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:38:31.825 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 157800 2024-02-09T17:38:31.825 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.825 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:38:31.825 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.825 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:38:31.825 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.517351+0000", 2024-02-09T17:38:31.825 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:38:31.825 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.825 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.825 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:38:31.825 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.517351+0000", 2024-02-09T17:38:31.825 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:38:31.825 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.826 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.826 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:38:31.826 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.517353+0000", 2024-02-09T17:38:31.826 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7e-06 2024-02-09T17:38:31.826 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.826 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.826 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:38:31.826 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.517359+0000", 2024-02-09T17:38:31.826 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9390000000000002e-06 2024-02-09T17:38:31.826 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.826 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.826 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:38:31.826 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.517374+0000", 2024-02-09T17:38:31.826 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5179e-05 2024-02-09T17:38:31.827 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.827 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.827 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:38:31.827 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.517384+0000", 2024-02-09T17:38:31.827 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0013000000000001e-05 2024-02-09T17:38:31.827 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.827 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.827 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:38:31.827 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.525824+0000", 2024-02-09T17:38:31.827 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.008440421 2024-02-09T17:38:31.827 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.827 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.827 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for rw locks", 2024-02-09T17:38:31.827 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.525954+0000", 2024-02-09T17:38:31.827 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000130078 2024-02-09T17:38:31.828 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.828 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.828 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:38:31.828 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.554541+0000", 2024-02-09T17:38:31.828 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.028586126999999999 2024-02-09T17:38:31.828 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.828 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.828 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:38:31.828 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.554619+0000", 2024-02-09T17:38:31.828 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8422000000000002e-05 2024-02-09T17:38:31.828 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.828 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.828 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 10,11", 2024-02-09T17:38:31.829 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.554742+0000", 2024-02-09T17:38:31.829 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012343799999999999 2024-02-09T17:38:31.829 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.829 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.829 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:38:31.829 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.573831+0000", 2024-02-09T17:38:31.829 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019088766 2024-02-09T17:38:31.829 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.829 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.829 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:31.829 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.573874+0000", 2024-02-09T17:38:31.829 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2917000000000003e-05 2024-02-09T17:38:31.829 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.829 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.829 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:31.830 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.784939+0000", 2024-02-09T17:38:31.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21106476599999999 2024-02-09T17:38:31.830 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.830 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.830 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:38:31.830 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.784983+0000", 2024-02-09T17:38:31.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4435999999999999e-05 2024-02-09T17:38:31.830 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.830 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.830 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:38:31.830 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.788421+0000", 2024-02-09T17:38:31.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034381530000000002 2024-02-09T17:38:31.830 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:38:31.830 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:38:31.830 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:38:31.831 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.831 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.831 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:157808 3.1 3:8cbdb1cd:::benchmark_data_smithi179_90440_object25849:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-02-09T17:38:31.831 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:32:28.531471+0000", 2024-02-09T17:38:31.831 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 363.20280743400002, 2024-02-09T17:38:31.831 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.31521497300000001, 2024-02-09T17:38:31.831 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:38:31.831 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:38:31.831 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:38:31.831 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:38:31.831 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:38:31.831 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 157808 2024-02-09T17:38:31.831 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.832 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:38:31.832 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.832 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:38:31.832 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.531471+0000", 2024-02-09T17:38:31.832 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:38:31.832 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.832 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.832 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:38:31.832 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.531471+0000", 2024-02-09T17:38:31.832 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:38:31.832 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.833 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.833 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:38:31.833 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.531472+0000", 2024-02-09T17:38:31.833 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.46e-06 2024-02-09T17:38:31.833 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.833 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.833 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:38:31.833 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.531476+0000", 2024-02-09T17:38:31.833 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5540000000000002e-06 2024-02-09T17:38:31.833 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.833 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.833 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:38:31.833 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.531484+0000", 2024-02-09T17:38:31.833 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0730000000000005e-06 2024-02-09T17:38:31.834 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.834 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.834 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:38:31.834 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.531490+0000", 2024-02-09T17:38:31.834 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5490000000000003e-06 2024-02-09T17:38:31.834 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.834 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.834 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:38:31.834 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.559900+0000", 2024-02-09T17:38:31.834 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.028409363 2024-02-09T17:38:31.834 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.834 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.835 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for rw locks", 2024-02-09T17:38:31.835 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.560049+0000", 2024-02-09T17:38:31.835 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014909600000000001 2024-02-09T17:38:31.835 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.835 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.835 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:38:31.835 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.809501+0000", 2024-02-09T17:38:31.835 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24945214700000001 2024-02-09T17:38:31.835 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.835 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.835 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:38:31.835 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.809563+0000", 2024-02-09T17:38:31.836 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2045999999999995e-05 2024-02-09T17:38:31.836 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.836 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.836 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 10,11", 2024-02-09T17:38:31.836 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.809645+0000", 2024-02-09T17:38:31.836 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1823999999999995e-05 2024-02-09T17:38:31.836 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.836 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.836 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:38:31.836 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.831085+0000", 2024-02-09T17:38:31.836 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.021439877999999999 2024-02-09T17:38:31.837 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.837 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.837 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:31.837 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.842911+0000", 2024-02-09T17:38:31.837 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011826346999999999 2024-02-09T17:38:31.837 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.837 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.837 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:31.837 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.842962+0000", 2024-02-09T17:38:31.837 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0525999999999997e-05 2024-02-09T17:38:31.838 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.838 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.838 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:38:31.838 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.843030+0000", 2024-02-09T17:38:31.838 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8524999999999999e-05 2024-02-09T17:38:31.838 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.838 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.838 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:38:31.838 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.846686+0000", 2024-02-09T17:38:31.838 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0036555849999999998 2024-02-09T17:38:31.838 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:38:31.838 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:38:31.839 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:38:31.839 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.839 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.839 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:157812 3.3 3:cf0cb945:::benchmark_data_smithi179_90440_object25853:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-02-09T17:38:31.839 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:32:28.531498+0000", 2024-02-09T17:38:31.839 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 363.20278026, 2024-02-09T17:38:31.839 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25599428899999999, 2024-02-09T17:38:31.839 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:38:31.839 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:38:31.839 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:38:31.839 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:38:31.840 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:38:31.840 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 157812 2024-02-09T17:38:31.840 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.840 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:38:31.840 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.840 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:38:31.840 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.531498+0000", 2024-02-09T17:38:31.840 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:38:31.840 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.840 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.840 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:38:31.840 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.531498+0000", 2024-02-09T17:38:31.841 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:38:31.841 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.841 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.841 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:38:31.841 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.531498+0000", 2024-02-09T17:38:31.841 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7599999999999998e-07 2024-02-09T17:38:31.841 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.841 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.841 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:38:31.841 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.531500+0000", 2024-02-09T17:38:31.842 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1150000000000001e-06 2024-02-09T17:38:31.842 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.842 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.842 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:38:31.842 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.531505+0000", 2024-02-09T17:38:31.842 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0309999999999998e-06 2024-02-09T17:38:31.842 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.842 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.842 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:38:31.842 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.531510+0000", 2024-02-09T17:38:31.842 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2400000000000001e-06 2024-02-09T17:38:31.843 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.843 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.843 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:38:31.843 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.536272+0000", 2024-02-09T17:38:31.843 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0047623359999999998 2024-02-09T17:38:31.843 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.843 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.843 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for rw locks", 2024-02-09T17:38:31.843 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.536414+0000", 2024-02-09T17:38:31.843 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014161600000000001 2024-02-09T17:38:31.843 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.843 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.844 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:38:31.844 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.548492+0000", 2024-02-09T17:38:31.844 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012077925999999999 2024-02-09T17:38:31.844 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.844 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.844 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:38:31.844 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.548597+0000", 2024-02-09T17:38:31.844 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010580400000000001 2024-02-09T17:38:31.844 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.844 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.844 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-02-09T17:38:31.845 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.548752+0000", 2024-02-09T17:38:31.845 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015489599999999999 2024-02-09T17:38:31.845 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.845 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.845 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:31.845 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.570524+0000", 2024-02-09T17:38:31.845 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.021771526999999999 2024-02-09T17:38:31.845 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.845 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.845 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:38:31.846 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.570547+0000", 2024-02-09T17:38:31.846 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3326e-05 2024-02-09T17:38:31.846 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.846 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.846 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:31.846 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.782860+0000", 2024-02-09T17:38:31.846 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.212313375 2024-02-09T17:38:31.846 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.846 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.846 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:38:31.846 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.782933+0000", 2024-02-09T17:38:31.847 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3035999999999997e-05 2024-02-09T17:38:31.847 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.847 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.847 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:38:31.847 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.787492+0000", 2024-02-09T17:38:31.847 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0045586849999999998 2024-02-09T17:38:31.847 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:38:31.847 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:38:31.847 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:38:31.847 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.847 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.848 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:157846 3.3 3:ca04de7c:::benchmark_data_smithi179_90440_object25887:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-02-09T17:38:31.848 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:32:28.592701+0000", 2024-02-09T17:38:31.848 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 363.14157716300002, 2024-02-09T17:38:31.848 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.242762859, 2024-02-09T17:38:31.848 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:38:31.848 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:38:31.848 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:38:31.848 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:38:31.848 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:38:31.848 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 157846 2024-02-09T17:38:31.848 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.849 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:38:31.849 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.849 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:38:31.849 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.592701+0000", 2024-02-09T17:38:31.849 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:38:31.849 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.849 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.849 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:38:31.849 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.592701+0000", 2024-02-09T17:38:31.849 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:38:31.849 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.850 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.850 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:38:31.850 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.592703+0000", 2024-02-09T17:38:31.850 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.59e-06 2024-02-09T17:38:31.850 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.850 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.850 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:38:31.850 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.592708+0000", 2024-02-09T17:38:31.850 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6239999999999999e-06 2024-02-09T17:38:31.850 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.850 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.850 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:38:31.850 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.592723+0000", 2024-02-09T17:38:31.851 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4554e-05 2024-02-09T17:38:31.851 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.851 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.851 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:38:31.851 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.592732+0000", 2024-02-09T17:38:31.851 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3899999999999999e-06 2024-02-09T17:38:31.851 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.851 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.851 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:38:31.851 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.592796+0000", 2024-02-09T17:38:31.851 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3545000000000002e-05 2024-02-09T17:38:31.851 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.851 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.851 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for rw locks", 2024-02-09T17:38:31.852 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.592939+0000", 2024-02-09T17:38:31.852 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014344 2024-02-09T17:38:31.852 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.852 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.852 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:38:31.852 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.805305+0000", 2024-02-09T17:38:31.852 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21236548599999999 2024-02-09T17:38:31.852 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.852 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.852 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:38:31.852 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.805421+0000", 2024-02-09T17:38:31.852 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000116709 2024-02-09T17:38:31.853 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.853 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.853 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-02-09T17:38:31.853 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.805590+0000", 2024-02-09T17:38:31.853 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000169135 2024-02-09T17:38:31.853 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.853 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.853 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:31.853 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.830531+0000", 2024-02-09T17:38:31.853 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.024940424999999999 2024-02-09T17:38:31.853 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.853 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.853 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:31.854 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.830595+0000", 2024-02-09T17:38:31.854 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3744000000000005e-05 2024-02-09T17:38:31.854 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.854 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.854 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:38:31.854 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.830773+0000", 2024-02-09T17:38:31.854 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017874500000000001 2024-02-09T17:38:31.854 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.854 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.854 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:38:31.854 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.830825+0000", 2024-02-09T17:38:31.854 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1660999999999999e-05 2024-02-09T17:38:31.854 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.854 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.855 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:38:31.855 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:28.835464+0000", 2024-02-09T17:38:31.855 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0046388109999999996 2024-02-09T17:38:31.855 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:38:31.855 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:38:31.855 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:38:31.855 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.855 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.855 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:173744 3.1 3:810a7512:::benchmark_data_smithi179_90440_object41785:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e259)", 2024-02-09T17:38:31.855 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:32:55.609893+0000", 2024-02-09T17:38:31.855 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 336.12438463799998, 2024-02-09T17:38:31.855 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.52845357100000001, 2024-02-09T17:38:31.855 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:38:31.855 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-09T17:38:31.856 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:38:31.856 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:38:31.856 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:38:31.856 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 173744 2024-02-09T17:38:31.856 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.856 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:38:31.856 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.856 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:38:31.856 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:55.609893+0000", 2024-02-09T17:38:31.856 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:38:31.856 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.856 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.856 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:38:31.856 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:55.609893+0000", 2024-02-09T17:38:31.857 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:38:31.857 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.857 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.857 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:38:31.857 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:55.609894+0000", 2024-02-09T17:38:31.857 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0699999999999999e-07 2024-02-09T17:38:31.857 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.857 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.857 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:38:31.857 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:55.609898+0000", 2024-02-09T17:38:31.857 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9790000000000004e-06 2024-02-09T17:38:31.857 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.857 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.857 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:38:31.858 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:55.609901+0000", 2024-02-09T17:38:31.858 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3529999999999999e-06 2024-02-09T17:38:31.858 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.858 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.858 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:38:31.858 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:55.609904+0000", 2024-02-09T17:38:31.858 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8990000000000001e-06 2024-02-09T17:38:31.858 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.858 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.858 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:38:31.858 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:55.624985+0000", 2024-02-09T17:38:31.858 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015081099000000001 2024-02-09T17:38:31.858 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.858 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.858 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for missing object", 2024-02-09T17:38:31.859 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:55.625090+0000", 2024-02-09T17:38:31.859 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010458 2024-02-09T17:38:31.859 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.859 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.859 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:38:31.859 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:56.138330+0000", 2024-02-09T17:38:31.859 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.51324086800000002 2024-02-09T17:38:31.859 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.859 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.859 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:38:31.859 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:32:56.138347+0000", 2024-02-09T17:38:31.859 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6586e-05 2024-02-09T17:38:31.859 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:38:31.859 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:38:31.860 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:38:31.860 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.860 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.860 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:204179 3.1 3:8372c17c:::benchmark_data_smithi179_90440_object72220:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e277)", 2024-02-09T17:38:31.860 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:33:44.992761+0000", 2024-02-09T17:38:31.860 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 286.741517552, 2024-02-09T17:38:31.860 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6720821499999996, 2024-02-09T17:38:31.860 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:38:31.860 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-09T17:38:31.860 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:38:31.860 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:38:31.860 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:38:31.860 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 204179 2024-02-09T17:38:31.861 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.861 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:38:31.861 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.861 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:38:31.861 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:44.992761+0000", 2024-02-09T17:38:31.861 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:38:31.861 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.861 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.861 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:38:31.861 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:44.992761+0000", 2024-02-09T17:38:31.861 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:38:31.861 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.861 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.861 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:38:31.862 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:44.992761+0000", 2024-02-09T17:38:31.862 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0800000000000001e-07 2024-02-09T17:38:31.862 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.862 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.862 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:38:31.862 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:44.992762+0000", 2024-02-09T17:38:31.862 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6020000000000001e-06 2024-02-09T17:38:31.862 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.862 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.862 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:38:31.862 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:44.992766+0000", 2024-02-09T17:38:31.862 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1540000000000002e-06 2024-02-09T17:38:31.862 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.862 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.862 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:38:31.863 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:44.992768+0000", 2024-02-09T17:38:31.863 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7379999999999999e-06 2024-02-09T17:38:31.863 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.863 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.863 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:38:31.863 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:45.004640+0000", 2024-02-09T17:38:31.863 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011871246 2024-02-09T17:38:31.863 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.863 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.863 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for missing object", 2024-02-09T17:38:31.863 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:45.004713+0000", 2024-02-09T17:38:31.863 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3804e-05 2024-02-09T17:38:31.863 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.864 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.864 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:38:31.864 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:49.664827+0000", 2024-02-09T17:38:31.864 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6601140020000003 2024-02-09T17:38:31.864 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.864 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.864 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:38:31.864 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:49.664843+0000", 2024-02-09T17:38:31.864 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5396e-05 2024-02-09T17:38:31.864 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:38:31.864 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:38:31.864 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:38:31.864 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.865 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.865 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:204180 3.3 3:c0ceb3da:::benchmark_data_smithi179_90440_object72221:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e277)", 2024-02-09T17:38:31.865 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:33:44.992772+0000", 2024-02-09T17:38:31.865 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 286.74150656799998, 2024-02-09T17:38:31.865 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6746478590000002, 2024-02-09T17:38:31.865 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:38:31.865 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-09T17:38:31.865 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:38:31.865 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:38:31.865 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:38:31.865 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 204180 2024-02-09T17:38:31.865 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.865 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:38:31.865 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.866 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:38:31.866 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:44.992772+0000", 2024-02-09T17:38:31.866 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:38:31.866 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.866 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.866 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:38:31.866 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:44.992772+0000", 2024-02-09T17:38:31.866 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:38:31.866 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.866 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.866 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:38:31.866 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:44.992772+0000", 2024-02-09T17:38:31.866 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.04e-07 2024-02-09T17:38:31.866 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.866 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.867 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:38:31.867 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:44.992776+0000", 2024-02-09T17:38:31.867 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8600000000000003e-06 2024-02-09T17:38:31.867 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.867 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.867 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:38:31.867 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:44.992780+0000", 2024-02-09T17:38:31.867 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3549999999999998e-06 2024-02-09T17:38:31.867 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.867 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.867 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:38:31.867 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:44.992783+0000", 2024-02-09T17:38:31.867 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.399e-06 2024-02-09T17:38:31.867 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.868 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.868 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:38:31.868 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:45.004367+0000", 2024-02-09T17:38:31.868 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011583515000000001 2024-02-09T17:38:31.868 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.868 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.868 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for missing object", 2024-02-09T17:38:31.868 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:45.004446+0000", 2024-02-09T17:38:31.868 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9504000000000003e-05 2024-02-09T17:38:31.868 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.868 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.868 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:38:31.868 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:49.667412+0000", 2024-02-09T17:38:31.868 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6629655210000003 2024-02-09T17:38:31.868 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.869 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.869 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:38:31.869 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:49.667419+0000", 2024-02-09T17:38:31.869 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5009999999999998e-06 2024-02-09T17:38:31.869 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:38:31.869 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:38:31.869 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:38:31.869 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.869 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.869 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:245525 3.1 3:8275b376:::benchmark_data_smithi179_90440_object113566:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e299)", 2024-02-09T17:38:31.869 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:34:43.859177+0000", 2024-02-09T17:38:31.869 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 227.875100831, 2024-02-09T17:38:31.869 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.234020112, 2024-02-09T17:38:31.869 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:38:31.870 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:38:31.870 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:38:31.870 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:38:31.870 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:38:31.870 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 245525 2024-02-09T17:38:31.870 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.870 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:38:31.870 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.870 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:38:31.870 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:43.859177+0000", 2024-02-09T17:38:31.870 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:38:31.870 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.870 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.870 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:38:31.870 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:43.859177+0000", 2024-02-09T17:38:31.871 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:38:31.871 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.871 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.871 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:38:31.871 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:43.859179+0000", 2024-02-09T17:38:31.871 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.719e-06 2024-02-09T17:38:31.871 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.871 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.871 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:38:31.871 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:43.859186+0000", 2024-02-09T17:38:31.871 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4989999999999999e-06 2024-02-09T17:38:31.871 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.871 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.871 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:38:31.871 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:43.859199+0000", 2024-02-09T17:38:31.872 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3858e-05 2024-02-09T17:38:31.872 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.872 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.872 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:38:31.872 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:43.859208+0000", 2024-02-09T17:38:31.872 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6819999999999992e-06 2024-02-09T17:38:31.872 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.872 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.872 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:38:31.872 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:43.859299+0000", 2024-02-09T17:38:31.872 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0841999999999995e-05 2024-02-09T17:38:31.872 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.872 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.872 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:38:31.873 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:43.859443+0000", 2024-02-09T17:38:31.873 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000143812 2024-02-09T17:38:31.873 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.873 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.873 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,10", 2024-02-09T17:38:31.873 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:43.859584+0000", 2024-02-09T17:38:31.873 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014136699999999999 2024-02-09T17:38:31.873 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.873 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.873 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:38:31.873 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:43.864718+0000", 2024-02-09T17:38:31.873 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0051343889999999996 2024-02-09T17:38:31.873 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.873 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.873 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:31.874 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:43.864763+0000", 2024-02-09T17:38:31.874 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4972000000000002e-05 2024-02-09T17:38:31.874 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.874 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.874 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:38:31.874 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:44.092888+0000", 2024-02-09T17:38:31.874 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22812454700000001 2024-02-09T17:38:31.874 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.874 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.874 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:38:31.874 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:44.092928+0000", 2024-02-09T17:38:31.874 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9820999999999997e-05 2024-02-09T17:38:31.874 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:38:31.874 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:38:31.874 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:38:31.874 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:44.093197+0000", 2024-02-09T17:38:31.875 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000269604 2024-02-09T17:38:31.875 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:38:31.875 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:38:31.875 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:38:31.875 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:38:31.875 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:38:31.875 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:38:31.875 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:38:31.919 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:31.945 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:38:31.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:31.947 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:38:32.048 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:38:32.049 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-09T17:38:32.089 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:32.118 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:38:32.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:32.122 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:38:32.223 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:38:32.257 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:32.462 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:38:32.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:32.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:38:32.486 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-09T17:38:32.566 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:38:32.627 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:32.675 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:38:32.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:32.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:38:32.783 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:38:32.815 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:32.839 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:38:32.845 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:32.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:38:32.939 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:38:32.946 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:38:32.964 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:38:33.011 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:33.275 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:38:33.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:33.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:38:33.387 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:38:33.435 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:33.592 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:38:33.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:33.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:38:33.699 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:38:33.737 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:33.766 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:38:33.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:33.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:38:33.823 INFO:teuthology.orchestra.run.smithi099.stdout:moved 16 / 111 (14.4144%) 2024-02-09T17:38:33.823 INFO:teuthology.orchestra.run.smithi099.stdout:avg 9.25 2024-02-09T17:38:33.823 INFO:teuthology.orchestra.run.smithi099.stdout:stddev 2.71186 -> 2.74241 (expected baseline 2.9119) 2024-02-09T17:38:33.823 INFO:teuthology.orchestra.run.smithi099.stdout:min osd.8 with 3 -> 6 pgs (0.324324 -> 0.648649 * mean) 2024-02-09T17:38:33.823 INFO:teuthology.orchestra.run.smithi099.stdout:max osd.6 with 14 -> 9 pgs (1.51351 -> 0.972973 * mean) 2024-02-09T17:38:33.823 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:38:33.823 INFO:teuthology.orchestra.run.smithi099.stdout:oload 110 2024-02-09T17:38:33.823 INFO:teuthology.orchestra.run.smithi099.stdout:max_change 3 2024-02-09T17:38:33.824 INFO:teuthology.orchestra.run.smithi099.stdout:max_change_osds 4 2024-02-09T17:38:33.824 INFO:teuthology.orchestra.run.smithi099.stdout:average_utilization 0.0150 2024-02-09T17:38:33.824 INFO:teuthology.orchestra.run.smithi099.stdout:overload_utilization 0.0165 2024-02-09T17:38:33.824 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1 weight 0.9500 -> 0.4649 2024-02-09T17:38:33.824 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11 weight 0.9500 -> 0.5636 2024-02-09T17:38:33.824 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6 weight 0.9500 -> 0.6767 2024-02-09T17:38:33.824 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3 weight 0.9500 -> 0.7671 2024-02-09T17:38:33.873 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:38:33.877 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:38:33.900 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:33 smithi099 ceph-mon[39420]: pgmap v1032: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 645 B/s rd, 14 KiB/s wr, 660 op/s; 7971/207111 objects misplaced (3.849%); 8.4 MiB/s, 135 objects/s recovering 2024-02-09T17:38:33.900 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:33 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'30467','3':'50271','6':'44347','11':'36939'}"}]: dispatch 2024-02-09T17:38:33.903 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:34.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:33 smithi179 ceph-mon[43975]: pgmap v1032: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 645 B/s rd, 14 KiB/s wr, 660 op/s; 7971/207111 objects misplaced (3.849%); 8.4 MiB/s, 135 objects/s recovering 2024-02-09T17:38:34.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:33 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'30467','3':'50271','6':'44347','11':'36939'}"}]: dispatch 2024-02-09T17:38:34.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:33 smithi162 ceph-mon[44304]: pgmap v1032: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 645 B/s rd, 14 KiB/s wr, 660 op/s; 7971/207111 objects misplaced (3.849%); 8.4 MiB/s, 135 objects/s recovering 2024-02-09T17:38:34.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:33 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'30467','3':'50271','6':'44347','11':'36939'}"}]: dispatch 2024-02-09T17:38:34.726 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-02-09T17:38:34.727 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight-by-utilization 110 0.05 2024-02-09T17:38:34.753 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:38:34.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:34.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:38:34.857 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:38:34.893 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:34.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:34 smithi162 ceph-mon[44304]: from='client.15624 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:38:34.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:34 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'30467','3':'50271','6':'44347','11':'36939'}"}]': finished 2024-02-09T17:38:34.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:34 smithi162 ceph-mon[44304]: osdmap e379: 12 total, 12 up, 12 in 2024-02-09T17:38:34.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:34 smithi162 ceph-mon[44304]: osdmap e380: 12 total, 12 up, 12 in 2024-02-09T17:38:34.987 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:38:34.987 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-09T17:38:34.987 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-02-09T17:38:34.987 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:38:34.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:34 smithi099 ceph-mon[39420]: from='client.15624 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:38:34.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:34 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'30467','3':'50271','6':'44347','11':'36939'}"}]': finished 2024-02-09T17:38:34.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:34 smithi099 ceph-mon[39420]: osdmap e379: 12 total, 12 up, 12 in 2024-02-09T17:38:34.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:34 smithi099 ceph-mon[39420]: osdmap e380: 12 total, 12 up, 12 in 2024-02-09T17:38:34.994 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:38:34.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:34.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:38:35.099 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:38:35.126 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:35.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:34 smithi179 ceph-mon[43975]: from='client.15624 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:38:35.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:34 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'30467','3':'50271','6':'44347','11':'36939'}"}]': finished 2024-02-09T17:38:35.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:34 smithi179 ceph-mon[43975]: osdmap e379: 12 total, 12 up, 12 in 2024-02-09T17:38:35.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:34 smithi179 ceph-mon[43975]: osdmap e380: 12 total, 12 up, 12 in 2024-02-09T17:38:35.249 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:38:35.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:35.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:38:35.281 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:38:35.357 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:38:35.383 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:35.403 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:38:35.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:35.406 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:38:35.507 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:38:35.535 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:35.609 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:38:35.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:35.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:38:35.710 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-09T17:38:35.712 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:38:35.740 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:35.759 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:38:35.761 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:35.762 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:38:35.835 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:35 smithi099 ceph-mon[39420]: pgmap v1034: 37 pgs: 1 peering, 2 active+remapped+backfilling, 34 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 560 B/s rd, 12 KiB/s wr, 549 op/s; 6730/205527 objects misplaced (3.275%); 16 MiB/s, 253 objects/s recovering 2024-02-09T17:38:35.835 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:35 smithi099 ceph-mon[39420]: osdmap e381: 12 total, 12 up, 12 in 2024-02-09T17:38:35.863 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:38:35.926 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:36.088 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:38:36.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:36.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:38:36.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:35 smithi179 ceph-mon[43975]: pgmap v1034: 37 pgs: 1 peering, 2 active+remapped+backfilling, 34 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 560 B/s rd, 12 KiB/s wr, 549 op/s; 6730/205527 objects misplaced (3.275%); 16 MiB/s, 253 objects/s recovering 2024-02-09T17:38:36.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:35 smithi179 ceph-mon[43975]: osdmap e381: 12 total, 12 up, 12 in 2024-02-09T17:38:36.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:35 smithi162 ceph-mon[44304]: pgmap v1034: 37 pgs: 1 peering, 2 active+remapped+backfilling, 34 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 560 B/s rd, 12 KiB/s wr, 549 op/s; 6730/205527 objects misplaced (3.275%); 16 MiB/s, 253 objects/s recovering 2024-02-09T17:38:36.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:35 smithi162 ceph-mon[44304]: osdmap e381: 12 total, 12 up, 12 in 2024-02-09T17:38:36.191 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:38:36.216 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:38:36.242 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:36.309 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:38:36.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:36.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:38:36.469 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:38:36.553 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:36.934 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:38:36.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:36.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:38:37.043 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:38:37.110 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:36 smithi099 ceph-mon[39420]: 4.5 starting backfill to osd.0 from (0'0,0'0] MAX to 378'12940 2024-02-09T17:38:37.110 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:36 smithi099 ceph-mon[39420]: 4.9 starting backfill to osd.0 from (0'0,0'0] MAX to 378'13054 2024-02-09T17:38:37.110 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:36 smithi099 ceph-mon[39420]: 4.f starting backfill to osd.9 from (0'0,0'0] MAX to 378'12906 2024-02-09T17:38:37.110 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:36 smithi099 ceph-mon[39420]: 4.a starting backfill to osd.7 from (0'0,0'0] MAX to 378'12938 2024-02-09T17:38:37.110 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:36 smithi099 ceph-mon[39420]: 4.2 starting backfill to osd.8 from (0'0,0'0] MAX to 378'12907 2024-02-09T17:38:37.110 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:36 smithi099 ceph-mon[39420]: 4.4 starting backfill to osd.10 from (0'0,0'0] MAX to 378'12875 2024-02-09T17:38:37.110 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:36 smithi099 ceph-mon[39420]: 4.e starting backfill to osd.3 from (0'0,0'0] MAX to 378'12989 2024-02-09T17:38:37.110 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:36 smithi099 ceph-mon[39420]: 4.8 starting backfill to osd.9 from (0'0,0'0] MAX to 378'13096 2024-02-09T17:38:37.111 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:36 smithi099 ceph-mon[39420]: 4.0 starting backfill to osd.8 from (0'0,0'0] MAX to 378'13004 2024-02-09T17:38:37.111 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:36 smithi099 ceph-mon[39420]: 4.5 starting backfill to osd.7 from (0'0,0'0] MAX to 378'12940 2024-02-09T17:38:37.111 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:36 smithi099 ceph-mon[39420]: 4.9 starting backfill to osd.4 from (0'0,0'0] MAX to 378'13054 2024-02-09T17:38:37.111 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:36 smithi099 ceph-mon[39420]: 4.9 starting backfill to osd.6 from (0'0,0'0] MAX to 378'13054 2024-02-09T17:38:37.111 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:36 smithi099 ceph-mon[39420]: osdmap e382: 12 total, 12 up, 12 in 2024-02-09T17:38:37.111 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:36 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2185127192' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:38:37.111 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:36 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:38:37.119 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:37.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:36 smithi179 ceph-mon[43975]: 4.5 starting backfill to osd.0 from (0'0,0'0] MAX to 378'12940 2024-02-09T17:38:37.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:36 smithi179 ceph-mon[43975]: 4.9 starting backfill to osd.0 from (0'0,0'0] MAX to 378'13054 2024-02-09T17:38:37.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:36 smithi179 ceph-mon[43975]: 4.f starting backfill to osd.9 from (0'0,0'0] MAX to 378'12906 2024-02-09T17:38:37.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:36 smithi179 ceph-mon[43975]: 4.a starting backfill to osd.7 from (0'0,0'0] MAX to 378'12938 2024-02-09T17:38:37.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:36 smithi179 ceph-mon[43975]: 4.2 starting backfill to osd.8 from (0'0,0'0] MAX to 378'12907 2024-02-09T17:38:37.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:36 smithi179 ceph-mon[43975]: 4.4 starting backfill to osd.10 from (0'0,0'0] MAX to 378'12875 2024-02-09T17:38:37.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:36 smithi179 ceph-mon[43975]: 4.e starting backfill to osd.3 from (0'0,0'0] MAX to 378'12989 2024-02-09T17:38:37.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:36 smithi179 ceph-mon[43975]: 4.8 starting backfill to osd.9 from (0'0,0'0] MAX to 378'13096 2024-02-09T17:38:37.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:36 smithi179 ceph-mon[43975]: 4.0 starting backfill to osd.8 from (0'0,0'0] MAX to 378'13004 2024-02-09T17:38:37.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:36 smithi179 ceph-mon[43975]: 4.5 starting backfill to osd.7 from (0'0,0'0] MAX to 378'12940 2024-02-09T17:38:37.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:36 smithi179 ceph-mon[43975]: 4.9 starting backfill to osd.4 from (0'0,0'0] MAX to 378'13054 2024-02-09T17:38:37.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:36 smithi179 ceph-mon[43975]: 4.9 starting backfill to osd.6 from (0'0,0'0] MAX to 378'13054 2024-02-09T17:38:37.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:36 smithi179 ceph-mon[43975]: osdmap e382: 12 total, 12 up, 12 in 2024-02-09T17:38:37.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:36 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2185127192' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:38:37.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:36 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:38:37.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:36 smithi162 ceph-mon[44304]: 4.5 starting backfill to osd.0 from (0'0,0'0] MAX to 378'12940 2024-02-09T17:38:37.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:36 smithi162 ceph-mon[44304]: 4.9 starting backfill to osd.0 from (0'0,0'0] MAX to 378'13054 2024-02-09T17:38:37.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:36 smithi162 ceph-mon[44304]: 4.f starting backfill to osd.9 from (0'0,0'0] MAX to 378'12906 2024-02-09T17:38:37.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:36 smithi162 ceph-mon[44304]: 4.a starting backfill to osd.7 from (0'0,0'0] MAX to 378'12938 2024-02-09T17:38:37.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:36 smithi162 ceph-mon[44304]: 4.2 starting backfill to osd.8 from (0'0,0'0] MAX to 378'12907 2024-02-09T17:38:37.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:36 smithi162 ceph-mon[44304]: 4.4 starting backfill to osd.10 from (0'0,0'0] MAX to 378'12875 2024-02-09T17:38:37.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:36 smithi162 ceph-mon[44304]: 4.e starting backfill to osd.3 from (0'0,0'0] MAX to 378'12989 2024-02-09T17:38:37.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:36 smithi162 ceph-mon[44304]: 4.8 starting backfill to osd.9 from (0'0,0'0] MAX to 378'13096 2024-02-09T17:38:37.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:36 smithi162 ceph-mon[44304]: 4.0 starting backfill to osd.8 from (0'0,0'0] MAX to 378'13004 2024-02-09T17:38:37.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:36 smithi162 ceph-mon[44304]: 4.5 starting backfill to osd.7 from (0'0,0'0] MAX to 378'12940 2024-02-09T17:38:37.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:36 smithi162 ceph-mon[44304]: 4.9 starting backfill to osd.4 from (0'0,0'0] MAX to 378'13054 2024-02-09T17:38:37.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:36 smithi162 ceph-mon[44304]: 4.9 starting backfill to osd.6 from (0'0,0'0] MAX to 378'13054 2024-02-09T17:38:37.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:36 smithi162 ceph-mon[44304]: osdmap e382: 12 total, 12 up, 12 in 2024-02-09T17:38:37.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:36 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2185127192' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:38:37.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:36 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:38:37.206 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:38:37.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:37.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:38:37.319 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:38:37.331 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:38:37.365 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:37.393 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:38:37.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:37.398 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:38:37.499 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:38:37.532 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:37.615 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is set 2024-02-09T17:38:37.852 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:37 smithi162 ceph-mon[44304]: pgmap v1038: 37 pgs: 1 activating+undersized+remapped, 7 remapped+peering, 1 peering, 3 active+remapped+backfilling, 25 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 550 op/s; 10816/200508 objects misplaced (5.394%); 17 MiB/s, 274 objects/s recovering 2024-02-09T17:38:37.852 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:37 smithi162 ceph-mon[44304]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:38:37.852 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:37 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:38:37.852 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:37 smithi162 ceph-mon[44304]: osdmap e383: 12 total, 12 up, 12 in 2024-02-09T17:38:37.962 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:38:37.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:37 smithi099 ceph-mon[39420]: pgmap v1038: 37 pgs: 1 activating+undersized+remapped, 7 remapped+peering, 1 peering, 3 active+remapped+backfilling, 25 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 550 op/s; 10816/200508 objects misplaced (5.394%); 17 MiB/s, 274 objects/s recovering 2024-02-09T17:38:37.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:37 smithi099 ceph-mon[39420]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:38:37.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:37 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:38:37.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:37 smithi099 ceph-mon[39420]: osdmap e383: 12 total, 12 up, 12 in 2024-02-09T17:38:38.019 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:38:38.061 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:38:38.090 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:38:38.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:38.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:38:38.113 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:38:38.151 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:38:38.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:37 smithi179 ceph-mon[43975]: pgmap v1038: 37 pgs: 1 activating+undersized+remapped, 7 remapped+peering, 1 peering, 3 active+remapped+backfilling, 25 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 550 op/s; 10816/200508 objects misplaced (5.394%); 17 MiB/s, 274 objects/s recovering 2024-02-09T17:38:38.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:37 smithi179 ceph-mon[43975]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:38:38.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:37 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:38:38.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:37 smithi179 ceph-mon[43975]: osdmap e383: 12 total, 12 up, 12 in 2024-02-09T17:38:38.196 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:38:38.203 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:38:38.240 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:38.256 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:38:38.270 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:38:38.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:38.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:38:38.311 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:38:38.360 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:38:38.375 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:38:38.411 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:38:38.430 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:38.459 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:38:38.463 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:38:38.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:38.466 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:38:38.522 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:38:38.545 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:38:38.547 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:38:38.547 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:38:38.547 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:38:38.547 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:38:38.547 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:38:38.547 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:38:38.548 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:38:38.548 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:38:38.548 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:38:38.549 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:38:38.549 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:38:38.567 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:38:38.665 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:38.702 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:38:38.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:38.708 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:38:38.809 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:38:38.876 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:38 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:38:38.876 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:38 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:38:38.876 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:38 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/962511975' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:38:38.877 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:38 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:38:38.903 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:39.124 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:38:39.132 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:38 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:38:39.132 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:38 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:38:39.132 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:38 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/962511975' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:38:39.132 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:38 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:38:39.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:39.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:38:39.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:38 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:38:39.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:38 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:38:39.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:38 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/962511975' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:38:39.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:38 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:38:39.233 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:38:39.284 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:39.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:39.401 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:38:39.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:38:39.502 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:38:39.547 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:39.580 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:38:39.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:39.587 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:38:39.688 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:38:39.746 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:39.777 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:38:39.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:39.781 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:38:39.882 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:38:39.886 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:39 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:38:39.886 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:39 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:38:39.887 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:39 smithi179 ceph-mon[43975]: pgmap v1040: 37 pgs: 2 active+remapped+backfill_toofull, 1 activating+undersized+remapped, 7 remapped+peering, 1 peering, 3 active+remapped+backfilling, 23 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 573 op/s; 19552/198645 objects misplaced (9.843%); 53 MiB/s, 850 objects/s recovering 2024-02-09T17:38:39.887 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:39 smithi179 ceph-mon[43975]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:38:39.887 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:39 smithi179 ceph-mon[43975]: osdmap e384: 12 total, 12 up, 12 in 2024-02-09T17:38:39.898 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:39 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:38:39.898 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:39 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:38:39.898 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:39 smithi162 ceph-mon[44304]: pgmap v1040: 37 pgs: 2 active+remapped+backfill_toofull, 1 activating+undersized+remapped, 7 remapped+peering, 1 peering, 3 active+remapped+backfilling, 23 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 573 op/s; 19552/198645 objects misplaced (9.843%); 53 MiB/s, 850 objects/s recovering 2024-02-09T17:38:39.898 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:39 smithi162 ceph-mon[44304]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:38:39.898 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:39 smithi162 ceph-mon[44304]: osdmap e384: 12 total, 12 up, 12 in 2024-02-09T17:38:39.932 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:39.946 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:38:39.946 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:38:39.956 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:38:39.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:39.959 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:38:40.060 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:38:40.103 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:40.220 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:38:40.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:40.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:38:40.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:39 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:38:40.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:39 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:38:40.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:39 smithi099 ceph-mon[39420]: pgmap v1040: 37 pgs: 2 active+remapped+backfill_toofull, 1 activating+undersized+remapped, 7 remapped+peering, 1 peering, 3 active+remapped+backfilling, 23 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 573 op/s; 19552/198645 objects misplaced (9.843%); 53 MiB/s, 850 objects/s recovering 2024-02-09T17:38:40.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:39 smithi099 ceph-mon[39420]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:38:40.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:39 smithi099 ceph-mon[39420]: osdmap e384: 12 total, 12 up, 12 in 2024-02-09T17:38:40.325 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:38:40.371 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:40.463 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:38:40.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:40.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:38:40.568 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:38:40.599 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:40.685 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:38:40.694 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:40.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:38:40.730 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-09T17:38:40.795 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:38:40.829 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset noscrub 2024-02-09T17:38:40.877 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:41.047 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:38:41.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:41.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:38:41.151 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:38:41.153 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:38:41.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:40 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:38:41.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:40 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:38:41.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:40 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:38:41.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:40 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:38:41.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:40 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:38:41.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:40 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:38:41.228 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:40 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:38:41.229 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:40 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:38:41.229 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:40 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:38:41.230 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:41.276 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:38:41.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:41.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:38:41.385 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:38:41.430 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:38:41.445 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:41.483 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:38:41.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:41.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:38:41.586 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:38:41.657 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:41.749 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:38:41.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:41.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:38:41.856 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:38:41.943 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:41 smithi099 ceph-mon[39420]: pgmap v1042: 37 pgs: 3 active+remapped+backfill_toofull, 1 activating+undersized+remapped, 8 active+remapped+backfilling, 25 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 506 op/s; 48395/193812 objects misplaced (24.970%); 79 MiB/s, 1.26k objects/s recovering 2024-02-09T17:38:41.943 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:41 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'27191','3':'46995','6':'41071','11':'33663'}"}]: dispatch 2024-02-09T17:38:41.945 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:42.094 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:38:42.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:41 smithi179 ceph-mon[43975]: pgmap v1042: 37 pgs: 3 active+remapped+backfill_toofull, 1 activating+undersized+remapped, 8 active+remapped+backfilling, 25 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 506 op/s; 48395/193812 objects misplaced (24.970%); 79 MiB/s, 1.26k objects/s recovering 2024-02-09T17:38:42.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:41 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'27191','3':'46995','6':'41071','11':'33663'}"}]: dispatch 2024-02-09T17:38:42.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:41 smithi162 ceph-mon[44304]: pgmap v1042: 37 pgs: 3 active+remapped+backfill_toofull, 1 activating+undersized+remapped, 8 active+remapped+backfilling, 25 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 506 op/s; 48395/193812 objects misplaced (24.970%); 79 MiB/s, 1.26k objects/s recovering 2024-02-09T17:38:42.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:41 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'27191','3':'46995','6':'41071','11':'33663'}"}]: dispatch 2024-02-09T17:38:42.226 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:38:42.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:42.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:38:42.334 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:38:42.410 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:42.427 INFO:teuthology.orchestra.run.smithi099.stdout:moved 1 / 111 (0.900901%) 2024-02-09T17:38:42.427 INFO:teuthology.orchestra.run.smithi099.stdout:avg 9.25 2024-02-09T17:38:42.427 INFO:teuthology.orchestra.run.smithi099.stdout:stddev 2.74241 -> 2.68095 (expected baseline 2.9119) 2024-02-09T17:38:42.427 INFO:teuthology.orchestra.run.smithi099.stdout:min osd.11 with 5 -> 5 pgs (0.540541 -> 0.540541 * mean) 2024-02-09T17:38:42.427 INFO:teuthology.orchestra.run.smithi099.stdout:max osd.7 with 15 -> 15 pgs (1.62162 -> 1.62162 * mean) 2024-02-09T17:38:42.427 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:38:42.427 INFO:teuthology.orchestra.run.smithi099.stdout:oload 110 2024-02-09T17:38:42.427 INFO:teuthology.orchestra.run.smithi099.stdout:max_change 0.05 2024-02-09T17:38:42.428 INFO:teuthology.orchestra.run.smithi099.stdout:max_change_osds 4 2024-02-09T17:38:42.428 INFO:teuthology.orchestra.run.smithi099.stdout:average_utilization 0.0146 2024-02-09T17:38:42.428 INFO:teuthology.orchestra.run.smithi099.stdout:overload_utilization 0.0161 2024-02-09T17:38:42.428 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1 weight 0.4649 -> 0.4149 2024-02-09T17:38:42.428 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11 weight 0.5636 -> 0.5137 2024-02-09T17:38:42.428 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6 weight 0.6767 -> 0.6267 2024-02-09T17:38:42.428 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3 weight 0.7671 -> 0.7171 2024-02-09T17:38:42.865 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:38:42.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:42.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:38:42.970 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:38:43.003 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:43.031 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:38:43.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:43.035 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:38:43.136 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:38:43.167 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:42 smithi099 ceph-mon[39420]: from='client.25364 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:38:43.167 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:42 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:38:43.167 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:42 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:38:43.167 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:42 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'27191','3':'46995','6':'41071','11':'33663'}"}]': finished 2024-02-09T17:38:43.167 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:42 smithi099 ceph-mon[39420]: osdmap e385: 12 total, 12 up, 12 in 2024-02-09T17:38:43.171 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:43.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:42 smithi179 ceph-mon[43975]: from='client.25364 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:38:43.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:42 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:38:43.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:42 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:38:43.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:42 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'27191','3':'46995','6':'41071','11':'33663'}"}]': finished 2024-02-09T17:38:43.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:42 smithi179 ceph-mon[43975]: osdmap e385: 12 total, 12 up, 12 in 2024-02-09T17:38:43.172 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:38:43.172 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:38:43.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:42 smithi162 ceph-mon[44304]: from='client.25364 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:38:43.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:42 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:38:43.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:42 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:38:43.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:42 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'27191','3':'46995','6':'41071','11':'33663'}"}]': finished 2024-02-09T17:38:43.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:42 smithi162 ceph-mon[44304]: osdmap e385: 12 total, 12 up, 12 in 2024-02-09T17:38:43.524 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-02-09T17:38:43.524 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight-by-utilization 110 1.0 2024-02-09T17:38:43.527 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:38:43.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:43.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:38:43.632 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:38:43.675 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:43.909 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:43 smithi162 ceph-mon[44304]: pgmap v1044: 37 pgs: 3 active+remapped+backfill_toofull, 8 active+remapped+backfilling, 26 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 500 op/s; 46535/191499 objects misplaced (24.300%); 78 MiB/s, 1.25k objects/s recovering 2024-02-09T17:38:43.910 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:43 smithi162 ceph-mon[44304]: osdmap e386: 12 total, 12 up, 12 in 2024-02-09T17:38:43.979 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-09T17:38:44.023 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:38:44.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:44.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:38:44.110 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:38:44.131 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:38:44.162 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:43 smithi179 ceph-mon[43975]: pgmap v1044: 37 pgs: 3 active+remapped+backfill_toofull, 8 active+remapped+backfilling, 26 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 500 op/s; 46535/191499 objects misplaced (24.300%); 78 MiB/s, 1.25k objects/s recovering 2024-02-09T17:38:44.163 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:43 smithi179 ceph-mon[43975]: osdmap e386: 12 total, 12 up, 12 in 2024-02-09T17:38:44.165 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:44.191 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:38:44.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:44.195 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:38:44.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:43 smithi099 ceph-mon[39420]: pgmap v1044: 37 pgs: 3 active+remapped+backfill_toofull, 8 active+remapped+backfilling, 26 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 500 op/s; 46535/191499 objects misplaced (24.300%); 78 MiB/s, 1.25k objects/s recovering 2024-02-09T17:38:44.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:43 smithi099 ceph-mon[39420]: osdmap e386: 12 total, 12 up, 12 in 2024-02-09T17:38:44.296 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:38:44.329 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:44.336 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:38:44.371 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:38:44.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:44.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:38:44.476 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:38:44.513 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:44.553 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:38:44.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:44.560 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:38:44.662 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:38:44.697 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:44.728 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:38:44.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:44.733 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:38:44.834 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:38:44.909 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:45.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:44 smithi179 ceph-mon[43975]: 4.9 continuing backfill to osd.0 from (335'3300,384'13310] MIN to 384'13310 2024-02-09T17:38:45.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:44 smithi179 ceph-mon[43975]: 4.9 continuing backfill to osd.4 from (335'3300,384'13310] MIN to 384'13310 2024-02-09T17:38:45.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:44 smithi179 ceph-mon[43975]: 4.9 starting backfill to osd.8 from (0'0,0'0] MAX to 384'13310 2024-02-09T17:38:45.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:44 smithi179 ceph-mon[43975]: osdmap e387: 12 total, 12 up, 12 in 2024-02-09T17:38:45.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:44 smithi162 ceph-mon[44304]: 4.9 continuing backfill to osd.0 from (335'3300,384'13310] MIN to 384'13310 2024-02-09T17:38:45.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:44 smithi162 ceph-mon[44304]: 4.9 continuing backfill to osd.4 from (335'3300,384'13310] MIN to 384'13310 2024-02-09T17:38:45.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:44 smithi162 ceph-mon[44304]: 4.9 starting backfill to osd.8 from (0'0,0'0] MAX to 384'13310 2024-02-09T17:38:45.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:44 smithi162 ceph-mon[44304]: osdmap e387: 12 total, 12 up, 12 in 2024-02-09T17:38:45.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:44 smithi099 ceph-mon[39420]: 4.9 continuing backfill to osd.0 from (335'3300,384'13310] MIN to 384'13310 2024-02-09T17:38:45.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:44 smithi099 ceph-mon[39420]: 4.9 continuing backfill to osd.4 from (335'3300,384'13310] MIN to 384'13310 2024-02-09T17:38:45.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:44 smithi099 ceph-mon[39420]: 4.9 starting backfill to osd.8 from (0'0,0'0] MAX to 384'13310 2024-02-09T17:38:45.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:44 smithi099 ceph-mon[39420]: osdmap e387: 12 total, 12 up, 12 in 2024-02-09T17:38:45.347 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:38:45.391 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:38:45.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:45.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:38:45.496 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:38:45.547 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:45.586 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:38:45.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:45.589 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:38:45.690 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:38:45.735 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:45.783 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:38:45.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:45.796 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:38:45.897 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:38:45.960 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:45.967 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is unset 2024-02-09T17:38:46.066 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:45 smithi162 ceph-mon[44304]: pgmap v1047: 37 pgs: 3 active+remapped+backfill_toofull, 8 active+remapped+backfilling, 26 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 501 op/s; 45194/189837 objects misplaced (23.807%); 54 MiB/s, 865 objects/s recovering 2024-02-09T17:38:46.066 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:45 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1625423436' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:38:46.067 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:45 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:38:46.074 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:38:46.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:46.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:38:46.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:45 smithi179 ceph-mon[43975]: pgmap v1047: 37 pgs: 3 active+remapped+backfill_toofull, 8 active+remapped+backfilling, 26 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 501 op/s; 45194/189837 objects misplaced (23.807%); 54 MiB/s, 865 objects/s recovering 2024-02-09T17:38:46.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:45 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1625423436' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:38:46.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:45 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:38:46.184 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:38:46.232 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:45 smithi099 ceph-mon[39420]: pgmap v1047: 37 pgs: 3 active+remapped+backfill_toofull, 8 active+remapped+backfilling, 26 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 501 op/s; 45194/189837 objects misplaced (23.807%); 54 MiB/s, 865 objects/s recovering 2024-02-09T17:38:46.232 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:45 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1625423436' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:38:46.232 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:45 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:38:46.261 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:46.448 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:38:46.448 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:38:46.948 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:46 smithi099 ceph-mon[39420]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:38:46.948 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:46 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:38:46.949 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:46 smithi099 ceph-mon[39420]: osdmap e388: 12 total, 12 up, 12 in 2024-02-09T17:38:46.949 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:46 smithi099 ceph-mon[39420]: 1.7 scrub starts 2024-02-09T17:38:46.949 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:46 smithi099 ceph-mon[39420]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:38:47.153 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:46 smithi162 ceph-mon[44304]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:38:47.153 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:46 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:38:47.154 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:46 smithi162 ceph-mon[44304]: osdmap e388: 12 total, 12 up, 12 in 2024-02-09T17:38:47.154 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:46 smithi162 ceph-mon[44304]: 1.7 scrub starts 2024-02-09T17:38:47.154 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:46 smithi162 ceph-mon[44304]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:38:47.186 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-09T17:38:47.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:46 smithi179 ceph-mon[43975]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:38:47.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:46 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:38:47.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:46 smithi179 ceph-mon[43975]: osdmap e388: 12 total, 12 up, 12 in 2024-02-09T17:38:47.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:46 smithi179 ceph-mon[43975]: 1.7 scrub starts 2024-02-09T17:38:47.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:46 smithi179 ceph-mon[43975]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:38:47.452 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:38:47.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:47.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:38:47.562 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:38:47.596 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:38:47.608 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:48.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:47 smithi162 ceph-mon[44304]: 1.7 scrub ok 2024-02-09T17:38:48.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:47 smithi162 ceph-mon[44304]: pgmap v1049: 37 pgs: 3 active+remapped+backfill_toofull, 8 active+remapped+backfilling, 26 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 497 op/s; 37532/185058 objects misplaced (20.281%); 109 MiB/s, 1.75k objects/s recovering 2024-02-09T17:38:48.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:47 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'15352','3':'37910','6':'30685','11':'20737'}"}]: dispatch 2024-02-09T17:38:48.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:47 smithi099 ceph-mon[39420]: 1.7 scrub ok 2024-02-09T17:38:48.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:47 smithi099 ceph-mon[39420]: pgmap v1049: 37 pgs: 3 active+remapped+backfill_toofull, 8 active+remapped+backfilling, 26 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 497 op/s; 37532/185058 objects misplaced (20.281%); 109 MiB/s, 1.75k objects/s recovering 2024-02-09T17:38:48.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:47 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'15352','3':'37910','6':'30685','11':'20737'}"}]: dispatch 2024-02-09T17:38:48.381 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:38:48.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:48.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:38:48.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:47 smithi179 ceph-mon[43975]: 1.7 scrub ok 2024-02-09T17:38:48.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:47 smithi179 ceph-mon[43975]: pgmap v1049: 37 pgs: 3 active+remapped+backfill_toofull, 8 active+remapped+backfilling, 26 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 497 op/s; 37532/185058 objects misplaced (20.281%); 109 MiB/s, 1.75k objects/s recovering 2024-02-09T17:38:48.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:47 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'15352','3':'37910','6':'30685','11':'20737'}"}]: dispatch 2024-02-09T17:38:48.435 INFO:teuthology.orchestra.run.smithi099.stdout:moved 8 / 111 (7.20721%) 2024-02-09T17:38:48.435 INFO:teuthology.orchestra.run.smithi099.stdout:avg 9.25 2024-02-09T17:38:48.435 INFO:teuthology.orchestra.run.smithi099.stdout:stddev 2.68095 -> 3.72212 (expected baseline 2.9119) 2024-02-09T17:38:48.435 INFO:teuthology.orchestra.run.smithi099.stdout:min osd.11 with 5 -> 4 pgs (0.540541 -> 0.432432 * mean) 2024-02-09T17:38:48.436 INFO:teuthology.orchestra.run.smithi099.stdout:max osd.7 with 15 -> 17 pgs (1.62162 -> 1.83784 * mean) 2024-02-09T17:38:48.436 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:38:48.436 INFO:teuthology.orchestra.run.smithi099.stdout:oload 110 2024-02-09T17:38:48.436 INFO:teuthology.orchestra.run.smithi099.stdout:max_change 1 2024-02-09T17:38:48.436 INFO:teuthology.orchestra.run.smithi099.stdout:max_change_osds 4 2024-02-09T17:38:48.436 INFO:teuthology.orchestra.run.smithi099.stdout:average_utilization 0.0143 2024-02-09T17:38:48.436 INFO:teuthology.orchestra.run.smithi099.stdout:overload_utilization 0.0157 2024-02-09T17:38:48.436 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1 weight 0.4149 -> 0.2343 2024-02-09T17:38:48.437 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11 weight 0.5137 -> 0.3164 2024-02-09T17:38:48.437 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6 weight 0.6267 -> 0.4682 2024-02-09T17:38:48.437 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3 weight 0.7171 -> 0.5785 2024-02-09T17:38:48.487 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:38:48.524 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:48.550 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:38:48.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:48.554 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:38:48.654 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:38:48.687 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:49.035 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:48 smithi162 ceph-mon[44304]: from='client.25373 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:38:49.035 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:48 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'15352','3':'37910','6':'30685','11':'20737'}"}]': finished 2024-02-09T17:38:49.035 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:48 smithi162 ceph-mon[44304]: osdmap e389: 12 total, 12 up, 12 in 2024-02-09T17:38:49.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:48 smithi099 ceph-mon[39420]: from='client.25373 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:38:49.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:48 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'15352','3':'37910','6':'30685','11':'20737'}"}]': finished 2024-02-09T17:38:49.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:48 smithi099 ceph-mon[39420]: osdmap e389: 12 total, 12 up, 12 in 2024-02-09T17:38:49.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:48 smithi179 ceph-mon[43975]: from='client.25373 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:38:49.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:48 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'15352','3':'37910','6':'30685','11':'20737'}"}]': finished 2024-02-09T17:38:49.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:48 smithi179 ceph-mon[43975]: osdmap e389: 12 total, 12 up, 12 in 2024-02-09T17:38:49.653 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:38:49.653 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:38:50.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:49 smithi099 ceph-mon[39420]: pgmap v1051: 37 pgs: 3 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 27 active+clean; 3.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 497 op/s; 36056/182412 objects misplaced (19.766%); 89 MiB/s, 1.42k objects/s recovering 2024-02-09T17:38:50.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:49 smithi099 ceph-mon[39420]: osdmap e390: 12 total, 12 up, 12 in 2024-02-09T17:38:50.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:49 smithi099 ceph-mon[39420]: 4.4 starting backfill to osd.7 from (0'0,0'0] MAX to 388'13333 2024-02-09T17:38:50.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:49 smithi099 ceph-mon[39420]: 4.4 continuing backfill to osd.10 from (336'3300,388'13333] 4:2810fb8b:::benchmark_data_smithi179_103809_object115717:head to 388'13333 2024-02-09T17:38:50.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:49 smithi099 ceph-mon[39420]: 1.1 scrub starts 2024-02-09T17:38:50.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:49 smithi099 ceph-mon[39420]: 1.1 scrub ok 2024-02-09T17:38:50.377 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:49 smithi162 ceph-mon[44304]: pgmap v1051: 37 pgs: 3 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 27 active+clean; 3.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 497 op/s; 36056/182412 objects misplaced (19.766%); 89 MiB/s, 1.42k objects/s recovering 2024-02-09T17:38:50.377 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:49 smithi162 ceph-mon[44304]: osdmap e390: 12 total, 12 up, 12 in 2024-02-09T17:38:50.377 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:49 smithi162 ceph-mon[44304]: 4.4 starting backfill to osd.7 from (0'0,0'0] MAX to 388'13333 2024-02-09T17:38:50.377 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:49 smithi162 ceph-mon[44304]: 4.4 continuing backfill to osd.10 from (336'3300,388'13333] 4:2810fb8b:::benchmark_data_smithi179_103809_object115717:head to 388'13333 2024-02-09T17:38:50.377 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:49 smithi162 ceph-mon[44304]: 1.1 scrub starts 2024-02-09T17:38:50.377 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:49 smithi162 ceph-mon[44304]: 1.1 scrub ok 2024-02-09T17:38:50.408 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-09T17:38:50.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:49 smithi179 ceph-mon[43975]: pgmap v1051: 37 pgs: 3 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 27 active+clean; 3.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 497 op/s; 36056/182412 objects misplaced (19.766%); 89 MiB/s, 1.42k objects/s recovering 2024-02-09T17:38:50.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:49 smithi179 ceph-mon[43975]: osdmap e390: 12 total, 12 up, 12 in 2024-02-09T17:38:50.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:49 smithi179 ceph-mon[43975]: 4.4 starting backfill to osd.7 from (0'0,0'0] MAX to 388'13333 2024-02-09T17:38:50.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:49 smithi179 ceph-mon[43975]: 4.4 continuing backfill to osd.10 from (336'3300,388'13333] 4:2810fb8b:::benchmark_data_smithi179_103809_object115717:head to 388'13333 2024-02-09T17:38:50.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:49 smithi179 ceph-mon[43975]: 1.1 scrub starts 2024-02-09T17:38:50.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:49 smithi179 ceph-mon[43975]: 1.1 scrub ok 2024-02-09T17:38:50.806 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:38:51.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:51 smithi099 ceph-mon[39420]: 4.b scrub starts 2024-02-09T17:38:51.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:51 smithi099 ceph-mon[39420]: osdmap e391: 12 total, 12 up, 12 in 2024-02-09T17:38:51.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:51 smithi099 ceph-mon[39420]: 4.6 starting backfill to osd.7 from (0'0,0'0] MAX to 388'13555 2024-02-09T17:38:51.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:51 smithi099 ceph-mon[39420]: 4.c starting backfill to osd.8 from (0'0,0'0] MAX to 388'13697 2024-02-09T17:38:51.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:51 smithi099 ceph-mon[39420]: 4.b scrub ok 2024-02-09T17:38:51.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:51 smithi099 ceph-mon[39420]: pgmap v1054: 37 pgs: 1 remapped+peering, 1 peering, 4 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 1 active+recovery_wait+degraded+remapped, 24 active+clean; 3.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 19 KiB/s wr, 506 op/s; 1/178755 objects degraded (0.001%); 37443/178755 objects misplaced (20.947%); 100 MiB/s, 1.60k objects/s recovering 2024-02-09T17:38:51.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:51 smithi099 ceph-mon[39420]: osdmap e392: 12 total, 12 up, 12 in 2024-02-09T17:38:51.744 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:51 smithi162 ceph-mon[44304]: 4.b scrub starts 2024-02-09T17:38:51.744 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:51 smithi162 ceph-mon[44304]: osdmap e391: 12 total, 12 up, 12 in 2024-02-09T17:38:51.744 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:51 smithi162 ceph-mon[44304]: 4.6 starting backfill to osd.7 from (0'0,0'0] MAX to 388'13555 2024-02-09T17:38:51.744 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:51 smithi162 ceph-mon[44304]: 4.c starting backfill to osd.8 from (0'0,0'0] MAX to 388'13697 2024-02-09T17:38:51.745 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:51 smithi162 ceph-mon[44304]: 4.b scrub ok 2024-02-09T17:38:51.745 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:51 smithi162 ceph-mon[44304]: pgmap v1054: 37 pgs: 1 remapped+peering, 1 peering, 4 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 1 active+recovery_wait+degraded+remapped, 24 active+clean; 3.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 19 KiB/s wr, 506 op/s; 1/178755 objects degraded (0.001%); 37443/178755 objects misplaced (20.947%); 100 MiB/s, 1.60k objects/s recovering 2024-02-09T17:38:51.745 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:51 smithi162 ceph-mon[44304]: osdmap e392: 12 total, 12 up, 12 in 2024-02-09T17:38:51.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:51 smithi179 ceph-mon[43975]: 4.b scrub starts 2024-02-09T17:38:51.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:51 smithi179 ceph-mon[43975]: osdmap e391: 12 total, 12 up, 12 in 2024-02-09T17:38:51.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:51 smithi179 ceph-mon[43975]: 4.6 starting backfill to osd.7 from (0'0,0'0] MAX to 388'13555 2024-02-09T17:38:51.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:51 smithi179 ceph-mon[43975]: 4.c starting backfill to osd.8 from (0'0,0'0] MAX to 388'13697 2024-02-09T17:38:51.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:51 smithi179 ceph-mon[43975]: 4.b scrub ok 2024-02-09T17:38:51.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:51 smithi179 ceph-mon[43975]: pgmap v1054: 37 pgs: 1 remapped+peering, 1 peering, 4 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 1 active+recovery_wait+degraded+remapped, 24 active+clean; 3.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 19 KiB/s wr, 506 op/s; 1/178755 objects degraded (0.001%); 37443/178755 objects misplaced (20.947%); 100 MiB/s, 1.60k objects/s recovering 2024-02-09T17:38:51.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:51 smithi179 ceph-mon[43975]: osdmap e392: 12 total, 12 up, 12 in 2024-02-09T17:38:52.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:52 smithi162 ceph-mon[44304]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-09T17:38:52.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:52 smithi162 ceph-mon[44304]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:38:52.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:52 smithi162 ceph-mon[44304]: Health check failed: Degraded data redundancy: 1/178755 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:38:52.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:52 smithi162 ceph-mon[44304]: osdmap e393: 12 total, 12 up, 12 in 2024-02-09T17:38:52.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:52 smithi099 ceph-mon[39420]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-09T17:38:52.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:52 smithi099 ceph-mon[39420]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:38:52.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:52 smithi099 ceph-mon[39420]: Health check failed: Degraded data redundancy: 1/178755 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:38:52.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:52 smithi099 ceph-mon[39420]: osdmap e393: 12 total, 12 up, 12 in 2024-02-09T17:38:52.776 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:38:52.777 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:38:52.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:52 smithi179 ceph-mon[43975]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-09T17:38:52.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:52 smithi179 ceph-mon[43975]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:38:52.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:52 smithi179 ceph-mon[43975]: Health check failed: Degraded data redundancy: 1/178755 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:38:52.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:52 smithi179 ceph-mon[43975]: osdmap e393: 12 total, 12 up, 12 in 2024-02-09T17:38:53.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:53 smithi099 ceph-mon[39420]: 1.13 scrub starts 2024-02-09T17:38:53.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:53 smithi099 ceph-mon[39420]: 1.13 scrub ok 2024-02-09T17:38:53.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:53 smithi099 ceph-mon[39420]: 1.7 scrub starts 2024-02-09T17:38:53.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:53 smithi099 ceph-mon[39420]: 1.7 scrub ok 2024-02-09T17:38:53.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:53 smithi099 ceph-mon[39420]: pgmap v1057: 37 pgs: 1 active+undersized+remapped, 1 remapped+peering, 3 peering, 4 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 1 active+recovery_wait+degraded+remapped, 22 active+clean; 3.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 412 op/s; 1/177513 objects degraded (0.001%); 38377/177513 objects misplaced (21.619%); 125 MiB/s, 2.00k objects/s recovering 2024-02-09T17:38:53.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:53 smithi162 ceph-mon[44304]: 1.13 scrub starts 2024-02-09T17:38:53.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:53 smithi162 ceph-mon[44304]: 1.13 scrub ok 2024-02-09T17:38:53.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:53 smithi162 ceph-mon[44304]: 1.7 scrub starts 2024-02-09T17:38:53.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:53 smithi162 ceph-mon[44304]: 1.7 scrub ok 2024-02-09T17:38:53.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:53 smithi162 ceph-mon[44304]: pgmap v1057: 37 pgs: 1 active+undersized+remapped, 1 remapped+peering, 3 peering, 4 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 1 active+recovery_wait+degraded+remapped, 22 active+clean; 3.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 412 op/s; 1/177513 objects degraded (0.001%); 38377/177513 objects misplaced (21.619%); 125 MiB/s, 2.00k objects/s recovering 2024-02-09T17:38:53.831 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-09T17:38:53.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:53 smithi179 ceph-mon[43975]: 1.13 scrub starts 2024-02-09T17:38:53.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:53 smithi179 ceph-mon[43975]: 1.13 scrub ok 2024-02-09T17:38:53.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:53 smithi179 ceph-mon[43975]: 1.7 scrub starts 2024-02-09T17:38:53.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:53 smithi179 ceph-mon[43975]: 1.7 scrub ok 2024-02-09T17:38:53.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:53 smithi179 ceph-mon[43975]: pgmap v1057: 37 pgs: 1 active+undersized+remapped, 1 remapped+peering, 3 peering, 4 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 1 active+recovery_wait+degraded+remapped, 22 active+clean; 3.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 412 op/s; 1/177513 objects degraded (0.001%); 38377/177513 objects misplaced (21.619%); 125 MiB/s, 2.00k objects/s recovering 2024-02-09T17:38:54.255 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:38:54.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:54 smithi099 ceph-mon[39420]: 1.c scrub ok 2024-02-09T17:38:54.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:54 smithi099 ceph-mon[39420]: 4.1 scrub starts 2024-02-09T17:38:54.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:54 smithi099 ceph-mon[39420]: 4.7 scrub starts 2024-02-09T17:38:54.827 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:54 smithi162 ceph-mon[44304]: 1.c scrub ok 2024-02-09T17:38:54.827 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:54 smithi162 ceph-mon[44304]: 4.1 scrub starts 2024-02-09T17:38:54.827 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:54 smithi162 ceph-mon[44304]: 4.7 scrub starts 2024-02-09T17:38:54.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:54 smithi179 ceph-mon[43975]: 1.c scrub ok 2024-02-09T17:38:54.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:54 smithi179 ceph-mon[43975]: 4.1 scrub starts 2024-02-09T17:38:54.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:54 smithi179 ceph-mon[43975]: 4.7 scrub starts 2024-02-09T17:38:55.476 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:55 smithi162 ceph-mon[44304]: osdmap e394: 12 total, 12 up, 12 in 2024-02-09T17:38:55.476 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:55 smithi162 ceph-mon[44304]: 4.1 scrub ok 2024-02-09T17:38:55.476 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:55 smithi162 ceph-mon[44304]: 4.7 scrub ok 2024-02-09T17:38:55.476 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:55 smithi162 ceph-mon[44304]: pgmap v1059: 37 pgs: 1 active+undersized+remapped, 1 remapped+peering, 2 peering, 4 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 1 active+recovery_wait+degraded+remapped, 23 active+clean; 3.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 951 B/s rd, 21 KiB/s wr, 320 op/s; 1/176898 objects degraded (0.001%); 38259/176898 objects misplaced (21.628%); 91 MiB/s, 1.46k objects/s recovering 2024-02-09T17:38:55.476 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:55 smithi162 ceph-mon[44304]: 4.f scrub starts 2024-02-09T17:38:55.476 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:55 smithi162 ceph-mon[44304]: 4.f scrub ok 2024-02-09T17:38:55.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:55 smithi099 ceph-mon[39420]: osdmap e394: 12 total, 12 up, 12 in 2024-02-09T17:38:55.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:55 smithi099 ceph-mon[39420]: 4.1 scrub ok 2024-02-09T17:38:55.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:55 smithi099 ceph-mon[39420]: 4.7 scrub ok 2024-02-09T17:38:55.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:55 smithi099 ceph-mon[39420]: pgmap v1059: 37 pgs: 1 active+undersized+remapped, 1 remapped+peering, 2 peering, 4 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 1 active+recovery_wait+degraded+remapped, 23 active+clean; 3.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 951 B/s rd, 21 KiB/s wr, 320 op/s; 1/176898 objects degraded (0.001%); 38259/176898 objects misplaced (21.628%); 91 MiB/s, 1.46k objects/s recovering 2024-02-09T17:38:55.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:55 smithi099 ceph-mon[39420]: 4.f scrub starts 2024-02-09T17:38:55.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:55 smithi099 ceph-mon[39420]: 4.f scrub ok 2024-02-09T17:38:55.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:55 smithi179 ceph-mon[43975]: osdmap e394: 12 total, 12 up, 12 in 2024-02-09T17:38:55.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:55 smithi179 ceph-mon[43975]: 4.1 scrub ok 2024-02-09T17:38:55.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:55 smithi179 ceph-mon[43975]: 4.7 scrub ok 2024-02-09T17:38:55.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:55 smithi179 ceph-mon[43975]: pgmap v1059: 37 pgs: 1 active+undersized+remapped, 1 remapped+peering, 2 peering, 4 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 1 active+recovery_wait+degraded+remapped, 23 active+clean; 3.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 951 B/s rd, 21 KiB/s wr, 320 op/s; 1/176898 objects degraded (0.001%); 38259/176898 objects misplaced (21.628%); 91 MiB/s, 1.46k objects/s recovering 2024-02-09T17:38:55.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:55 smithi179 ceph-mon[43975]: 4.f scrub starts 2024-02-09T17:38:55.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:55 smithi179 ceph-mon[43975]: 4.f scrub ok 2024-02-09T17:38:56.162 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:38:56.162 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:38:56.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:56 smithi099 ceph-mon[39420]: 1.8 scrub starts 2024-02-09T17:38:56.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:56 smithi099 ceph-mon[39420]: 1.b scrub starts 2024-02-09T17:38:56.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:56 smithi099 ceph-mon[39420]: 1.8 scrub ok 2024-02-09T17:38:56.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:56 smithi099 ceph-mon[39420]: 1.b scrub ok 2024-02-09T17:38:56.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:56 smithi099 ceph-mon[39420]: osdmap e395: 12 total, 12 up, 12 in 2024-02-09T17:38:56.896 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:56 smithi162 ceph-mon[44304]: 1.8 scrub starts 2024-02-09T17:38:56.896 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:56 smithi162 ceph-mon[44304]: 1.b scrub starts 2024-02-09T17:38:56.896 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:56 smithi162 ceph-mon[44304]: 1.8 scrub ok 2024-02-09T17:38:56.896 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:56 smithi162 ceph-mon[44304]: 1.b scrub ok 2024-02-09T17:38:56.897 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:56 smithi162 ceph-mon[44304]: osdmap e395: 12 total, 12 up, 12 in 2024-02-09T17:38:56.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:56 smithi179 ceph-mon[43975]: 1.8 scrub starts 2024-02-09T17:38:56.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:56 smithi179 ceph-mon[43975]: 1.b scrub starts 2024-02-09T17:38:56.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:56 smithi179 ceph-mon[43975]: 1.8 scrub ok 2024-02-09T17:38:56.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:56 smithi179 ceph-mon[43975]: 1.b scrub ok 2024-02-09T17:38:56.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:56 smithi179 ceph-mon[43975]: osdmap e395: 12 total, 12 up, 12 in 2024-02-09T17:38:56.945 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-09T17:38:57.119 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:38:57.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:57.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:38:57.227 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:38:57.260 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:57.291 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:38:57.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:57.299 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:38:57.400 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:38:57.417 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:38:57.443 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:57.482 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:38:57.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:57.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:38:57.591 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:38:57.636 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:57.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:57 smithi179 ceph-mon[43975]: osdmap e396: 12 total, 12 up, 12 in 2024-02-09T17:38:57.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:57 smithi179 ceph-mon[43975]: pgmap v1062: 37 pgs: 1 active+recovery_wait+degraded, 3 peering, 5 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 27 active+clean; 3.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 332 op/s; 2/173157 objects degraded (0.001%); 31958/173157 objects misplaced (18.456%); 80 MiB/s, 1.28k objects/s recovering 2024-02-09T17:38:57.695 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:38:57.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:57.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:38:57.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:57 smithi099 ceph-mon[39420]: osdmap e396: 12 total, 12 up, 12 in 2024-02-09T17:38:57.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:57 smithi099 ceph-mon[39420]: pgmap v1062: 37 pgs: 1 active+recovery_wait+degraded, 3 peering, 5 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 27 active+clean; 3.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 332 op/s; 2/173157 objects degraded (0.001%); 31958/173157 objects misplaced (18.456%); 80 MiB/s, 1.28k objects/s recovering 2024-02-09T17:38:57.764 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:57 smithi162 ceph-mon[44304]: osdmap e396: 12 total, 12 up, 12 in 2024-02-09T17:38:57.764 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:57 smithi162 ceph-mon[44304]: pgmap v1062: 37 pgs: 1 active+recovery_wait+degraded, 3 peering, 5 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 27 active+clean; 3.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 332 op/s; 2/173157 objects degraded (0.001%); 31958/173157 objects misplaced (18.456%); 80 MiB/s, 1.28k objects/s recovering 2024-02-09T17:38:57.804 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:38:57.840 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:57.872 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:38:57.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:38:57.876 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:38:57.938 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:38:57.976 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:38:57.989 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:38:58.036 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:38:58.041 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:38:58.067 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:38:58.115 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:38:58.155 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:38:58.214 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:38:58.244 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:38:58.280 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:38:58.339 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:38:58.343 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-02-09T17:38:58.343 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight-by-utilization 110 1.0 2024-02-09T17:38:58.393 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:38:58.450 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:38:58.482 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:38:58.482 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:38:58.482 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:38:58.483 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:38:58.483 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:38:58.483 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:38:58.483 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:38:58.483 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:38:58.483 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:38:58.483 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:38:58.483 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:38:58.483 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:38:58.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:58 smithi162 ceph-mon[44304]: 4.e scrub starts 2024-02-09T17:38:58.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:58 smithi162 ceph-mon[44304]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:38:58.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:58 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 2/173157 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:38:58.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:58 smithi162 ceph-mon[44304]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-09T17:38:58.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:58 smithi162 ceph-mon[44304]: 1.6 scrub starts 2024-02-09T17:38:58.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:58 smithi162 ceph-mon[44304]: 1.6 scrub ok 2024-02-09T17:38:58.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:58 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3021471262' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:38:58.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:58 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:38:58.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:58 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:38:58.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:58 smithi162 ceph-mon[44304]: 4.e scrub ok 2024-02-09T17:38:58.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:58 smithi099 ceph-mon[39420]: 4.e scrub starts 2024-02-09T17:38:58.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:58 smithi099 ceph-mon[39420]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:38:58.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:58 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 2/173157 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:38:58.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:58 smithi099 ceph-mon[39420]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-09T17:38:58.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:58 smithi099 ceph-mon[39420]: 1.6 scrub starts 2024-02-09T17:38:58.748 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:58 smithi099 ceph-mon[39420]: 1.6 scrub ok 2024-02-09T17:38:58.748 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:58 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3021471262' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:38:58.748 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:58 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:38:58.748 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:58 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:38:58.748 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:58 smithi099 ceph-mon[39420]: 4.e scrub ok 2024-02-09T17:38:58.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:58 smithi179 ceph-mon[43975]: 4.e scrub starts 2024-02-09T17:38:58.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:58 smithi179 ceph-mon[43975]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:38:58.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:58 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 2/173157 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:38:58.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:58 smithi179 ceph-mon[43975]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-09T17:38:58.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:58 smithi179 ceph-mon[43975]: 1.6 scrub starts 2024-02-09T17:38:58.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:58 smithi179 ceph-mon[43975]: 1.6 scrub ok 2024-02-09T17:38:58.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:58 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3021471262' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:38:58.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:58 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:38:58.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:58 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:38:58.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:58 smithi179 ceph-mon[43975]: 4.e scrub ok 2024-02-09T17:38:59.258 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-09T17:38:59.583 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:38:59.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:59 smithi099 ceph-mon[39420]: osdmap e397: 12 total, 12 up, 12 in 2024-02-09T17:38:59.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:38:59 smithi099 ceph-mon[39420]: pgmap v1064: 37 pgs: 1 active+recovery_wait+degraded, 2 peering, 5 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 28 active+clean; 3.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 453 op/s; 2/171027 objects degraded (0.001%); 31526/171027 objects misplaced (18.433%); 81 MiB/s, 1.30k objects/s recovering 2024-02-09T17:38:59.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:59 smithi179 ceph-mon[43975]: osdmap e397: 12 total, 12 up, 12 in 2024-02-09T17:38:59.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:38:59 smithi179 ceph-mon[43975]: pgmap v1064: 37 pgs: 1 active+recovery_wait+degraded, 2 peering, 5 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 28 active+clean; 3.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 453 op/s; 2/171027 objects degraded (0.001%); 31526/171027 objects misplaced (18.433%); 81 MiB/s, 1.30k objects/s recovering 2024-02-09T17:38:59.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:59 smithi162 ceph-mon[44304]: osdmap e397: 12 total, 12 up, 12 in 2024-02-09T17:38:59.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:38:59 smithi162 ceph-mon[44304]: pgmap v1064: 37 pgs: 1 active+recovery_wait+degraded, 2 peering, 5 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 28 active+clean; 3.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 453 op/s; 2/171027 objects degraded (0.001%); 31526/171027 objects misplaced (18.433%); 81 MiB/s, 1.30k objects/s recovering 2024-02-09T17:39:00.131 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:39:00.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:00.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:39:00.302 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:39:00.353 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:00.513 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:39:00.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:00 smithi179 ceph-mon[43975]: 1.12 scrub starts 2024-02-09T17:39:00.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:00 smithi179 ceph-mon[43975]: 1.12 scrub ok 2024-02-09T17:39:00.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:00 smithi162 ceph-mon[44304]: 1.12 scrub starts 2024-02-09T17:39:00.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:00 smithi162 ceph-mon[44304]: 1.12 scrub ok 2024-02-09T17:39:00.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:00 smithi099 ceph-mon[39420]: 1.12 scrub starts 2024-02-09T17:39:00.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:00 smithi099 ceph-mon[39420]: 1.12 scrub ok 2024-02-09T17:39:01.332 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:39:01.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:01.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:39:01.439 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:39:01.477 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:01.484 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:39:01.556 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:01 smithi162 ceph-mon[44304]: pgmap v1065: 37 pgs: 5 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 31 active+clean; 3.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 619 op/s; 30494/165684 objects misplaced (18.405%); 58 MiB/s, 923 objects/s recovering 2024-02-09T17:39:01.557 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:01 smithi099 ceph-mon[39420]: pgmap v1065: 37 pgs: 5 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 31 active+clean; 3.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 619 op/s; 30494/165684 objects misplaced (18.405%); 58 MiB/s, 923 objects/s recovering 2024-02-09T17:39:01.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:01 smithi179 ceph-mon[43975]: pgmap v1065: 37 pgs: 5 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 31 active+clean; 3.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 619 op/s; 30494/165684 objects misplaced (18.405%); 58 MiB/s, 923 objects/s recovering 2024-02-09T17:39:02.592 INFO:teuthology.orchestra.run.smithi099.stdout:moved 12 / 111 (10.8108%) 2024-02-09T17:39:02.592 INFO:teuthology.orchestra.run.smithi099.stdout:avg 9.25 2024-02-09T17:39:02.592 INFO:teuthology.orchestra.run.smithi099.stdout:stddev 3.72212 -> 4.12563 (expected baseline 2.9119) 2024-02-09T17:39:02.592 INFO:teuthology.orchestra.run.smithi099.stdout:min osd.11 with 4 -> 3 pgs (0.432432 -> 0.324324 * mean) 2024-02-09T17:39:02.592 INFO:teuthology.orchestra.run.smithi099.stdout:max osd.7 with 17 -> 11 pgs (1.83784 -> 1.18919 * mean) 2024-02-09T17:39:02.592 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:39:02.593 INFO:teuthology.orchestra.run.smithi099.stdout:oload 110 2024-02-09T17:39:02.593 INFO:teuthology.orchestra.run.smithi099.stdout:max_change 1 2024-02-09T17:39:02.593 INFO:teuthology.orchestra.run.smithi099.stdout:max_change_osds 4 2024-02-09T17:39:02.593 INFO:teuthology.orchestra.run.smithi099.stdout:average_utilization 0.0141 2024-02-09T17:39:02.593 INFO:teuthology.orchestra.run.smithi099.stdout:overload_utilization 0.0155 2024-02-09T17:39:02.593 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1 weight 0.2343 -> 0.1446 2024-02-09T17:39:02.593 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11 weight 0.3164 -> 0.2113 2024-02-09T17:39:02.593 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3 weight 0.5785 -> 0.4365 2024-02-09T17:39:02.593 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7 weight 1.0000 -> 0.7886 2024-02-09T17:39:02.634 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:02 smithi099 ceph-mon[39420]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/171027 objects degraded (0.001%), 1 pg degraded) 2024-02-09T17:39:02.634 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:02 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'9478','3':'28608','7':'51679','11':'13848'}"}]: dispatch 2024-02-09T17:39:02.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:02 smithi179 ceph-mon[43975]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/171027 objects degraded (0.001%), 1 pg degraded) 2024-02-09T17:39:02.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:02 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'9478','3':'28608','7':'51679','11':'13848'}"}]: dispatch 2024-02-09T17:39:02.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:02 smithi162 ceph-mon[44304]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/171027 objects degraded (0.001%), 1 pg degraded) 2024-02-09T17:39:02.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:02 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'9478','3':'28608','7':'51679','11':'13848'}"}]: dispatch 2024-02-09T17:39:03.739 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:39:03.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:03.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:39:03.845 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:39:03.863 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:03 smithi162 ceph-mon[44304]: from='client.15648 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:39:03.863 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:03 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'9478','3':'28608','7':'51679','11':'13848'}"}]': finished 2024-02-09T17:39:03.863 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:03 smithi162 ceph-mon[44304]: osdmap e398: 12 total, 12 up, 12 in 2024-02-09T17:39:03.863 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:03 smithi162 ceph-mon[44304]: pgmap v1067: 37 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 31 active+clean; 3.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 543 op/s; 29137/162918 objects misplaced (17.884%); 11 MiB/s, 170 objects/s recovering 2024-02-09T17:39:03.870 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:03 smithi179 ceph-mon[43975]: from='client.15648 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:39:03.870 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:03 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'9478','3':'28608','7':'51679','11':'13848'}"}]': finished 2024-02-09T17:39:03.870 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:03 smithi179 ceph-mon[43975]: osdmap e398: 12 total, 12 up, 12 in 2024-02-09T17:39:03.870 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:03 smithi179 ceph-mon[43975]: pgmap v1067: 37 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 31 active+clean; 3.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 543 op/s; 29137/162918 objects misplaced (17.884%); 11 MiB/s, 170 objects/s recovering 2024-02-09T17:39:03.873 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:03.893 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:39:03.896 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:03.896 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:39:03.955 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:03 smithi099 ceph-mon[39420]: from='client.15648 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:39:03.955 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:03 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'9478','3':'28608','7':'51679','11':'13848'}"}]': finished 2024-02-09T17:39:03.956 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:03 smithi099 ceph-mon[39420]: osdmap e398: 12 total, 12 up, 12 in 2024-02-09T17:39:03.956 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:03 smithi099 ceph-mon[39420]: pgmap v1067: 37 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 31 active+clean; 3.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 543 op/s; 29137/162918 objects misplaced (17.884%); 11 MiB/s, 170 objects/s recovering 2024-02-09T17:39:03.997 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:39:04.025 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:04.045 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:39:04.047 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:39:04.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:04.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:39:04.149 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:39:04.197 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:39:04.197 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-09T17:39:04.197 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-02-09T17:39:04.197 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-02-09T17:39:04.197 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:39:04.228 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:04.249 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:39:04.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:04.252 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:39:04.353 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:39:04.386 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:04.440 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:39:04.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:04.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:39:04.544 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:39:04.572 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:04.596 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:39:04.598 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:04 smithi099 ceph-mon[39420]: 4.0 continuing backfill to osd.8 from (337'3800,397'13875] MIN to 397'13875 2024-02-09T17:39:04.598 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:04 smithi099 ceph-mon[39420]: 4.0 starting backfill to osd.9 from (0'0,0'0] MAX to 397'13875 2024-02-09T17:39:04.598 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:04 smithi099 ceph-mon[39420]: osdmap e399: 12 total, 12 up, 12 in 2024-02-09T17:39:04.598 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:04 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/142772254' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:39:04.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:04.600 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:39:04.601 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is unset 2024-02-09T17:39:04.701 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:39:04.731 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:04.787 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:39:04.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:04.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:39:04.896 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:39:04.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:04 smithi179 ceph-mon[43975]: 4.0 continuing backfill to osd.8 from (337'3800,397'13875] MIN to 397'13875 2024-02-09T17:39:04.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:04 smithi179 ceph-mon[43975]: 4.0 starting backfill to osd.9 from (0'0,0'0] MAX to 397'13875 2024-02-09T17:39:04.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:04 smithi179 ceph-mon[43975]: osdmap e399: 12 total, 12 up, 12 in 2024-02-09T17:39:04.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:04 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/142772254' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:39:04.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:04 smithi162 ceph-mon[44304]: 4.0 continuing backfill to osd.8 from (337'3800,397'13875] MIN to 397'13875 2024-02-09T17:39:04.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:04 smithi162 ceph-mon[44304]: 4.0 starting backfill to osd.9 from (0'0,0'0] MAX to 397'13875 2024-02-09T17:39:04.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:04 smithi162 ceph-mon[44304]: osdmap e399: 12 total, 12 up, 12 in 2024-02-09T17:39:04.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:04 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/142772254' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:39:04.958 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:04.996 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:39:04.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:05.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:39:05.005 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-09T17:39:05.100 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:39:05.154 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:05.503 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:39:05.894 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:05 smithi162 ceph-mon[44304]: 1.4 scrub starts 2024-02-09T17:39:05.895 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:05 smithi162 ceph-mon[44304]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:39:05.895 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:05 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/142772254' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:39:05.895 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:05 smithi162 ceph-mon[44304]: osdmap e400: 12 total, 12 up, 12 in 2024-02-09T17:39:05.895 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:05 smithi162 ceph-mon[44304]: 4.f starting backfill to osd.5 from (0'0,0'0] MAX to 397'13775 2024-02-09T17:39:05.895 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:05 smithi162 ceph-mon[44304]: 4.8 starting backfill to osd.10 from (0'0,0'0] MAX to 397'13980 2024-02-09T17:39:05.895 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:05 smithi162 ceph-mon[44304]: 4.e starting backfill to osd.5 from (0'0,0'0] MAX to 397'13840 2024-02-09T17:39:05.895 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:05 smithi162 ceph-mon[44304]: 4.a starting backfill to osd.5 from (0'0,0'0] MAX to 397'13822 2024-02-09T17:39:05.895 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:05 smithi162 ceph-mon[44304]: 4.3 starting backfill to osd.2 from (0'0,0'0] MAX to 397'13829 2024-02-09T17:39:05.895 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:05 smithi162 ceph-mon[44304]: 1.4 scrub ok 2024-02-09T17:39:05.895 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:05 smithi162 ceph-mon[44304]: pgmap v1070: 37 pgs: 3 remapped+peering, 1 unknown, 1 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 27 active+clean; 3.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 37 KiB/s wr, 562 op/s; 28673/160971 objects misplaced (17.813%); 11 MiB/s, 178 objects/s recovering 2024-02-09T17:39:05.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:05 smithi179 ceph-mon[43975]: 1.4 scrub starts 2024-02-09T17:39:05.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:05 smithi179 ceph-mon[43975]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:39:05.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:05 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/142772254' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:39:05.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:05 smithi179 ceph-mon[43975]: osdmap e400: 12 total, 12 up, 12 in 2024-02-09T17:39:05.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:05 smithi179 ceph-mon[43975]: 4.f starting backfill to osd.5 from (0'0,0'0] MAX to 397'13775 2024-02-09T17:39:05.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:05 smithi179 ceph-mon[43975]: 4.8 starting backfill to osd.10 from (0'0,0'0] MAX to 397'13980 2024-02-09T17:39:05.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:05 smithi179 ceph-mon[43975]: 4.e starting backfill to osd.5 from (0'0,0'0] MAX to 397'13840 2024-02-09T17:39:05.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:05 smithi179 ceph-mon[43975]: 4.a starting backfill to osd.5 from (0'0,0'0] MAX to 397'13822 2024-02-09T17:39:05.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:05 smithi179 ceph-mon[43975]: 4.3 starting backfill to osd.2 from (0'0,0'0] MAX to 397'13829 2024-02-09T17:39:05.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:05 smithi179 ceph-mon[43975]: 1.4 scrub ok 2024-02-09T17:39:05.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:05 smithi179 ceph-mon[43975]: pgmap v1070: 37 pgs: 3 remapped+peering, 1 unknown, 1 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 27 active+clean; 3.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 37 KiB/s wr, 562 op/s; 28673/160971 objects misplaced (17.813%); 11 MiB/s, 178 objects/s recovering 2024-02-09T17:39:05.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:05 smithi099 ceph-mon[39420]: 1.4 scrub starts 2024-02-09T17:39:05.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:05 smithi099 ceph-mon[39420]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:39:05.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:05 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/142772254' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:39:05.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:05 smithi099 ceph-mon[39420]: osdmap e400: 12 total, 12 up, 12 in 2024-02-09T17:39:05.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:05 smithi099 ceph-mon[39420]: 4.f starting backfill to osd.5 from (0'0,0'0] MAX to 397'13775 2024-02-09T17:39:05.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:05 smithi099 ceph-mon[39420]: 4.8 starting backfill to osd.10 from (0'0,0'0] MAX to 397'13980 2024-02-09T17:39:05.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:05 smithi099 ceph-mon[39420]: 4.e starting backfill to osd.5 from (0'0,0'0] MAX to 397'13840 2024-02-09T17:39:05.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:05 smithi099 ceph-mon[39420]: 4.a starting backfill to osd.5 from (0'0,0'0] MAX to 397'13822 2024-02-09T17:39:05.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:05 smithi099 ceph-mon[39420]: 4.3 starting backfill to osd.2 from (0'0,0'0] MAX to 397'13829 2024-02-09T17:39:05.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:05 smithi099 ceph-mon[39420]: 1.4 scrub ok 2024-02-09T17:39:05.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:05 smithi099 ceph-mon[39420]: pgmap v1070: 37 pgs: 3 remapped+peering, 1 unknown, 1 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 27 active+clean; 3.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 37 KiB/s wr, 562 op/s; 28673/160971 objects misplaced (17.813%); 11 MiB/s, 178 objects/s recovering 2024-02-09T17:39:06.135 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:39:06.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:06.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:39:06.240 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:39:06.281 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:06.321 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:39:06.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:06.334 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:39:06.435 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:39:06.481 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:06.638 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:06 smithi179 ceph-mon[43975]: Health check failed: Reduced data availability: 2 pgs inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-09T17:39:06.638 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:06 smithi179 ceph-mon[43975]: osdmap e401: 12 total, 12 up, 12 in 2024-02-09T17:39:06.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:06 smithi162 ceph-mon[44304]: Health check failed: Reduced data availability: 2 pgs inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-09T17:39:06.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:06 smithi162 ceph-mon[44304]: osdmap e401: 12 total, 12 up, 12 in 2024-02-09T17:39:06.944 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:39:06.984 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:39:06.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:06 smithi099 ceph-mon[39420]: Health check failed: Reduced data availability: 2 pgs inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-09T17:39:06.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:06 smithi099 ceph-mon[39420]: osdmap e401: 12 total, 12 up, 12 in 2024-02-09T17:39:07.044 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:39:07.064 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:39:07.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:07.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:39:07.090 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:39:07.126 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:39:07.157 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:39:07.171 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:39:07.226 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:07.262 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:39:07.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:07.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:39:07.372 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:39:07.417 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:39:07.418 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:07.445 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:39:07.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:07.452 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:39:07.454 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:39:07.489 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:39:07.544 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:39:07.552 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:39:07.592 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:39:07.607 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set noscrub 2024-02-09T17:39:07.608 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:07.635 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:39:07.640 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:39:07.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:07.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:39:07.666 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:39:07.666 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:39:07.667 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:39:07.667 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:39:07.667 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:39:07.667 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:39:07.667 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:39:07.667 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:39:07.667 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:39:07.667 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:39:07.667 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:39:07.667 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:39:07.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:07 smithi179 ceph-mon[43975]: osdmap e402: 12 total, 12 up, 12 in 2024-02-09T17:39:07.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:07 smithi179 ceph-mon[43975]: pgmap v1073: 37 pgs: 2 active+undersized+remapped, 3 remapped+peering, 1 unknown, 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 23 active+clean; 3.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 111 KiB/s wr, 490 op/s; 37083/157179 objects misplaced (23.593%); 7.8 MiB/s, 123 objects/s recovering 2024-02-09T17:39:07.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:07 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/389504240' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:39:07.762 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:39:07.819 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:07.833 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:39:07.833 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:39:07.856 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:07 smithi162 ceph-mon[44304]: osdmap e402: 12 total, 12 up, 12 in 2024-02-09T17:39:07.856 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:07 smithi162 ceph-mon[44304]: pgmap v1073: 37 pgs: 2 active+undersized+remapped, 3 remapped+peering, 1 unknown, 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 23 active+clean; 3.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 111 KiB/s wr, 490 op/s; 37083/157179 objects misplaced (23.593%); 7.8 MiB/s, 123 objects/s recovering 2024-02-09T17:39:07.856 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:07 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/389504240' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:39:07.941 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:39:07.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:07.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:39:07.956 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:07 smithi099 ceph-mon[39420]: osdmap e402: 12 total, 12 up, 12 in 2024-02-09T17:39:07.956 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:07 smithi099 ceph-mon[39420]: pgmap v1073: 37 pgs: 2 active+undersized+remapped, 3 remapped+peering, 1 unknown, 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 23 active+clean; 3.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 111 KiB/s wr, 490 op/s; 37083/157179 objects misplaced (23.593%); 7.8 MiB/s, 123 objects/s recovering 2024-02-09T17:39:07.956 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:07 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/389504240' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:39:08.052 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:39:08.098 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:08.143 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:39:08.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:08.149 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:39:08.250 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:39:08.316 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:08.469 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:39:08.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:08.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:39:08.535 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:39:08.577 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:39:08.626 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:08.695 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.9 dump_ops_in_flight 2024-02-09T17:39:08.851 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:39:08.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:08.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:39:08.956 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:39:08.980 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] 2024-02-09T17:39:08.980 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:39:08.981 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] 2024-02-09T17:39:08.981 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd out 2 2024-02-09T17:39:09.019 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:09.101 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:39:09.159 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:39:09.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:09.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:39:09.268 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:39:09.317 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:09.481 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:39:09.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:09.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:39:09.590 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:39:09.651 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:09.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:09 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:39:09.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:09 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:39:09.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:09 smithi179 ceph-mon[43975]: osdmap e403: 12 total, 12 up, 12 in 2024-02-09T17:39:09.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:09 smithi179 ceph-mon[43975]: pgmap v1075: 37 pgs: 2 active+undersized+remapped, 2 remapped+peering, 7 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 23 active+clean; 3.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 54 KiB/s wr, 518 op/s; 44266/154587 objects misplaced (28.635%); 27 MiB/s, 431 objects/s recovering 2024-02-09T17:39:09.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:09 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:39:09.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:09 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:39:09.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:09 smithi099 ceph-mon[39420]: osdmap e403: 12 total, 12 up, 12 in 2024-02-09T17:39:09.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:09 smithi099 ceph-mon[39420]: pgmap v1075: 37 pgs: 2 active+undersized+remapped, 2 remapped+peering, 7 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 23 active+clean; 3.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 54 KiB/s wr, 518 op/s; 44266/154587 objects misplaced (28.635%); 27 MiB/s, 431 objects/s recovering 2024-02-09T17:39:10.120 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:39:10.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:10.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:39:10.176 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:39:10.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:09 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:39:10.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:09 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:39:10.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:09 smithi162 ceph-mon[44304]: osdmap e403: 12 total, 12 up, 12 in 2024-02-09T17:39:10.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:09 smithi162 ceph-mon[44304]: pgmap v1075: 37 pgs: 2 active+undersized+remapped, 2 remapped+peering, 7 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 23 active+clean; 3.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 54 KiB/s wr, 518 op/s; 44266/154587 objects misplaced (28.635%); 27 MiB/s, 431 objects/s recovering 2024-02-09T17:39:10.235 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:39:10.287 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:10.331 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:39:10.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:10.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:39:10.435 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:39:10.484 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:10.510 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:39:10.518 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:39:10.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:10.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:39:10.623 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:39:10.680 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:10.706 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:10 smithi099 ceph-mon[39420]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:39:10.706 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:10 smithi099 ceph-mon[39420]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 1 pg peering) 2024-02-09T17:39:10.873 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:10 smithi179 ceph-mon[43975]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:39:10.873 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:10 smithi179 ceph-mon[43975]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 1 pg peering) 2024-02-09T17:39:10.890 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:39:10.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:10.893 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:39:10.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:10 smithi162 ceph-mon[44304]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:39:10.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:10 smithi162 ceph-mon[44304]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 1 pg peering) 2024-02-09T17:39:10.994 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:39:11.040 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:11.172 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:39:11.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:11.177 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:39:11.238 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T17:39:11.238 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:39:11.278 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:39:11.332 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:11.550 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:39:11.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:11.557 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:39:11.658 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:39:11.715 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:11.753 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:39:11.753 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is set 2024-02-09T17:39:11.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:11.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:39:11.858 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:39:11.889 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:11 smithi099 ceph-mon[39420]: pgmap v1076: 37 pgs: 7 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 27 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 37 KiB/s wr, 641 op/s; 40226/149493 objects misplaced (26.908%); 43 MiB/s, 691 objects/s recovering 2024-02-09T17:39:11.889 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:11 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3500948510' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:39:11.909 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:11.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:11 smithi179 ceph-mon[43975]: pgmap v1076: 37 pgs: 7 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 27 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 37 KiB/s wr, 641 op/s; 40226/149493 objects misplaced (26.908%); 43 MiB/s, 691 objects/s recovering 2024-02-09T17:39:11.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:11 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3500948510' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:39:11.938 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:39:11.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:11.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:39:12.042 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:39:12.055 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.9 dump_blocked_ops 2024-02-09T17:39:12.089 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:11 smithi162 ceph-mon[44304]: pgmap v1076: 37 pgs: 7 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 27 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 37 KiB/s wr, 641 op/s; 40226/149493 objects misplaced (26.908%); 43 MiB/s, 691 objects/s recovering 2024-02-09T17:39:12.089 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:11 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3500948510' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:39:12.091 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:12.127 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:39:12.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:12.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:39:12.237 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:39:12.283 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:12.315 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:39:12.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:12.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:39:12.429 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:39:12.475 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:12.481 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:39:12.509 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:39:12.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:12.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:39:12.615 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:39:12.690 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:12.734 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:12 smithi162 ceph-mon[44304]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:39:12.734 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:12 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3500948510' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:39:12.734 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:12 smithi162 ceph-mon[44304]: osdmap e404: 12 total, 12 up, 12 in 2024-02-09T17:39:12.992 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:12 smithi099 ceph-mon[39420]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:39:12.992 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:12 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3500948510' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:39:12.992 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:12 smithi099 ceph-mon[39420]: osdmap e404: 12 total, 12 up, 12 in 2024-02-09T17:39:13.092 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:12 smithi179 ceph-mon[43975]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:39:13.092 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:12 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3500948510' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:39:13.092 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:12 smithi179 ceph-mon[43975]: osdmap e404: 12 total, 12 up, 12 in 2024-02-09T17:39:13.223 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:39:13.259 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:39:13.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:13.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:39:13.363 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:39:13.413 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:13.463 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:39:13.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:13.469 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:39:13.570 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:39:13.663 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:13.709 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:39:13.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:13.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:39:13.782 INFO:teuthology.orchestra.run.smithi099.stderr:marked out osd.2. 2024-02-09T17:39:13.814 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:39:13.859 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:13 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mon-a[39397]: 2024-02-09T17:39:13.661+0000 7feb57590700 -1 mon.a@0(leader).osd e404 definitely_dead 0 2024-02-09T17:39:13.859 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:13 smithi099 ceph-mon[39420]: pgmap v1078: 37 pgs: 7 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 27 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 599 op/s; 37569/146190 objects misplaced (25.699%); 53 MiB/s, 841 objects/s recovering 2024-02-09T17:39:13.859 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:13 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3776813970' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-02-09T17:39:13.859 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:13 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-02-09T17:39:13.859 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:13 smithi099 ceph-mon[39420]: Client client.admin marked osd.2 out, while it was still marked up 2024-02-09T17:39:13.862 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:14.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:13 smithi179 ceph-mon[43975]: pgmap v1078: 37 pgs: 7 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 27 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 599 op/s; 37569/146190 objects misplaced (25.699%); 53 MiB/s, 841 objects/s recovering 2024-02-09T17:39:14.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:13 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3776813970' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-02-09T17:39:14.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:13 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-02-09T17:39:14.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:13 smithi179 ceph-mon[43975]: Client client.admin marked osd.2 out, while it was still marked up 2024-02-09T17:39:14.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:13 smithi162 ceph-mon[44304]: pgmap v1078: 37 pgs: 7 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 27 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 599 op/s; 37569/146190 objects misplaced (25.699%); 53 MiB/s, 841 objects/s recovering 2024-02-09T17:39:14.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:13 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3776813970' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-02-09T17:39:14.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:13 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-02-09T17:39:14.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:13 smithi162 ceph-mon[44304]: Client client.admin marked osd.2 out, while it was still marked up 2024-02-09T17:39:14.630 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T17:39:14.630 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:39:14.815 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:39:14.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:14.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:39:14.919 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:39:14.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:14 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:39:14.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:14 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:39:14.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:14 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-02-09T17:39:14.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:14 smithi179 ceph-mon[43975]: osdmap e405: 12 total, 12 up, 11 in 2024-02-09T17:39:14.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:14 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:39:14.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:14 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:39:14.955 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:14.986 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:39:14.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:14.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:39:14.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:14 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:39:14.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:14 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:39:14.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:14 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-02-09T17:39:14.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:14 smithi099 ceph-mon[39420]: osdmap e405: 12 total, 12 up, 11 in 2024-02-09T17:39:14.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:14 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:39:14.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:14 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:39:15.033 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-09T17:39:15.091 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:39:15.137 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:14 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:39:15.137 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:14 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:39:15.137 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:14 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-02-09T17:39:15.137 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:14 smithi162 ceph-mon[44304]: osdmap e405: 12 total, 12 up, 11 in 2024-02-09T17:39:15.137 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:14 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:39:15.138 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:14 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:39:15.139 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:15.174 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:39:15.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:15.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:39:15.279 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:39:15.321 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:15.356 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:39:15.362 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:15.363 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:39:15.403 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.9 dump_historic_ops 2024-02-09T17:39:15.464 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:39:15.499 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:15.651 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:39:15.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:15.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:39:15.756 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:39:15.776 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:39:15.794 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:15.820 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:39:15.821 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:39:15.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:15.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:39:15.926 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:39:15.968 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:16.028 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:15 smithi099 ceph-mon[39420]: pgmap v1080: 37 pgs: 1 peering, 7 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 26 active+clean; 2.9 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 597 op/s; 36381/144231 objects misplaced (25.224%); 44 MiB/s, 698 objects/s recovering 2024-02-09T17:39:16.028 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:15 smithi099 ceph-mon[39420]: 4.3 starting backfill to osd.5 from (0'0,0'0] MAX to 404'14208 2024-02-09T17:39:16.028 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:15 smithi099 ceph-mon[39420]: 4.4 continuing backfill to osd.7 from (337'4100,404'14154] MIN to 404'14154 2024-02-09T17:39:16.028 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:15 smithi099 ceph-mon[39420]: 4.4 starting backfill to osd.8 from (0'0,0'0] MAX to 404'14154 2024-02-09T17:39:16.028 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:15 smithi099 ceph-mon[39420]: osdmap e406: 12 total, 12 up, 11 in 2024-02-09T17:39:16.029 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:15 smithi099 ceph-mon[39420]: 4.4 continuing backfill to osd.10 from (337'4100,404'14154] 4:2810fb8b:::benchmark_data_smithi179_103809_object115717:head to 404'14154 2024-02-09T17:39:16.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:15 smithi179 ceph-mon[43975]: pgmap v1080: 37 pgs: 1 peering, 7 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 26 active+clean; 2.9 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 597 op/s; 36381/144231 objects misplaced (25.224%); 44 MiB/s, 698 objects/s recovering 2024-02-09T17:39:16.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:15 smithi179 ceph-mon[43975]: 4.3 starting backfill to osd.5 from (0'0,0'0] MAX to 404'14208 2024-02-09T17:39:16.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:15 smithi179 ceph-mon[43975]: 4.4 continuing backfill to osd.7 from (337'4100,404'14154] MIN to 404'14154 2024-02-09T17:39:16.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:15 smithi179 ceph-mon[43975]: 4.4 starting backfill to osd.8 from (0'0,0'0] MAX to 404'14154 2024-02-09T17:39:16.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:15 smithi179 ceph-mon[43975]: osdmap e406: 12 total, 12 up, 11 in 2024-02-09T17:39:16.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:15 smithi179 ceph-mon[43975]: 4.4 continuing backfill to osd.10 from (337'4100,404'14154] 4:2810fb8b:::benchmark_data_smithi179_103809_object115717:head to 404'14154 2024-02-09T17:39:16.135 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:39:16.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:16.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:39:16.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:15 smithi162 ceph-mon[44304]: pgmap v1080: 37 pgs: 1 peering, 7 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 26 active+clean; 2.9 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 597 op/s; 36381/144231 objects misplaced (25.224%); 44 MiB/s, 698 objects/s recovering 2024-02-09T17:39:16.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:15 smithi162 ceph-mon[44304]: 4.3 starting backfill to osd.5 from (0'0,0'0] MAX to 404'14208 2024-02-09T17:39:16.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:15 smithi162 ceph-mon[44304]: 4.4 continuing backfill to osd.7 from (337'4100,404'14154] MIN to 404'14154 2024-02-09T17:39:16.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:15 smithi162 ceph-mon[44304]: 4.4 starting backfill to osd.8 from (0'0,0'0] MAX to 404'14154 2024-02-09T17:39:16.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:15 smithi162 ceph-mon[44304]: osdmap e406: 12 total, 12 up, 11 in 2024-02-09T17:39:16.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:15 smithi162 ceph-mon[44304]: 4.4 continuing backfill to osd.10 from (337'4100,404'14154] 4:2810fb8b:::benchmark_data_smithi179_103809_object115717:head to 404'14154 2024-02-09T17:39:16.241 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:39:16.308 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:16.345 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:39:16.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:16.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:39:16.452 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:39:16.492 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:16.623 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:39:16.675 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:39:16.726 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:39:16.751 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:39:16.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:16.754 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:39:16.778 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:39:16.843 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:39:16.855 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:39:16.888 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:39:16.944 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:16 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:39:16.944 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:16 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:39:16.944 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:16 smithi099 ceph-mon[39420]: osdmap e407: 12 total, 12 up, 11 in 2024-02-09T17:39:16.944 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:16 smithi099 ceph-mon[39420]: 4.6 starting backfill to osd.5 from (0'0,0'0] MAX to 404'14357 2024-02-09T17:39:16.944 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:16 smithi099 ceph-mon[39420]: osdmap e408: 12 total, 12 up, 11 in 2024-02-09T17:39:16.944 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:16 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/159353228' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:39:16.953 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:16.955 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:39:17.013 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:39:17.045 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:39:17.090 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:39:17.145 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:39:17.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:16 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:39:17.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:16 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:39:17.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:16 smithi179 ceph-mon[43975]: osdmap e407: 12 total, 12 up, 11 in 2024-02-09T17:39:17.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:16 smithi179 ceph-mon[43975]: 4.6 starting backfill to osd.5 from (0'0,0'0] MAX to 404'14357 2024-02-09T17:39:17.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:16 smithi179 ceph-mon[43975]: osdmap e408: 12 total, 12 up, 11 in 2024-02-09T17:39:17.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:16 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/159353228' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:39:17.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:16 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:39:17.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:16 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:39:17.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:16 smithi162 ceph-mon[44304]: osdmap e407: 12 total, 12 up, 11 in 2024-02-09T17:39:17.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:16 smithi162 ceph-mon[44304]: 4.6 starting backfill to osd.5 from (0'0,0'0] MAX to 404'14357 2024-02-09T17:39:17.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:16 smithi162 ceph-mon[44304]: osdmap e408: 12 total, 12 up, 11 in 2024-02-09T17:39:17.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:16 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/159353228' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:39:17.178 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:39:17.222 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:39:17.222 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:39:17.223 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:39:17.223 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:39:17.223 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:39:17.223 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:39:17.223 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:39:17.223 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:39:17.223 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:39:17.223 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:39:17.223 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:39:17.223 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:39:17.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:17 smithi179 ceph-mon[43975]: pgmap v1084: 37 pgs: 1 active+remapped+backfill_wait, 1 peering, 8 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 25 active+clean; 2.8 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 532 op/s; 35811/139794 objects misplaced (25.617%); 71 MiB/s, 1.14k objects/s recovering 2024-02-09T17:39:17.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:17 smithi179 ceph-mon[43975]: osdmap e409: 12 total, 12 up, 11 in 2024-02-09T17:39:17.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:17 smithi179 ceph-mon[43975]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:39:17.935 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:39:17.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:17.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:39:18.005 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:39:18.006 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-02-09T17:39:18.006 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-02-09T17:39:18.006 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-02-09T17:39:18.006 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.006 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:78562 4.2 4:413c7ec5:::benchmark_data_smithi179_103809_object78561:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-02-09T17:39:18.006 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:36:15.973879+0000", 2024-02-09T17:39:18.006 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 182.02888382399999, 2024-02-09T17:39:18.006 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.20721125100000001, 2024-02-09T17:39:18.006 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:18.006 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:18.006 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:18.007 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:18.007 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:18.007 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 78562 2024-02-09T17:39:18.007 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.007 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:18.007 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.007 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:18.007 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:15.973879+0000", 2024-02-09T17:39:18.007 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:18.007 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.007 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.007 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:18.007 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:15.973879+0000", 2024-02-09T17:39:18.007 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:18.008 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.008 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.008 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:18.008 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:15.973881+0000", 2024-02-09T17:39:18.008 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4300000000000001e-06 2024-02-09T17:39:18.008 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.008 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.008 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:18.008 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:15.973905+0000", 2024-02-09T17:39:18.008 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4703999999999999e-05 2024-02-09T17:39:18.008 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.008 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.008 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:18.008 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:15.973919+0000", 2024-02-09T17:39:18.009 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3954999999999999e-05 2024-02-09T17:39:18.009 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.009 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.009 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:18.009 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:15.973927+0000", 2024-02-09T17:39:18.009 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.0269999999999995e-06 2024-02-09T17:39:18.009 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.009 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.009 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:18.009 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:15.973988+0000", 2024-02-09T17:39:18.009 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0423000000000001e-05 2024-02-09T17:39:18.009 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.009 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.009 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:18.010 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:15.974108+0000", 2024-02-09T17:39:18.010 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000120241 2024-02-09T17:39:18.010 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.010 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.010 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,11", 2024-02-09T17:39:18.010 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:15.974205+0000", 2024-02-09T17:39:18.010 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6856999999999998e-05 2024-02-09T17:39:18.010 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.010 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.010 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:18.010 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:15.982065+0000", 2024-02-09T17:39:18.010 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0078605649999999999 2024-02-09T17:39:18.010 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.010 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.011 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:18.011 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:15.982127+0000", 2024-02-09T17:39:18.011 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1779999999999995e-05 2024-02-09T17:39:18.011 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.011 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.011 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:18.011 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.181030+0000", 2024-02-09T17:39:18.011 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.19890259199999999 2024-02-09T17:39:18.011 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.011 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.011 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:18.011 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.181067+0000", 2024-02-09T17:39:18.011 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6810000000000002e-05 2024-02-09T17:39:18.011 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.012 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.012 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:18.012 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.181090+0000", 2024-02-09T17:39:18.012 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3867e-05 2024-02-09T17:39:18.012 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:18.012 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:18.012 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:18.012 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.012 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.012 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:78564 4.2 4:4dd3f444:::benchmark_data_smithi179_103809_object78563:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-02-09T17:39:18.012 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:36:15.974111+0000", 2024-02-09T17:39:18.012 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 182.02865226899999, 2024-02-09T17:39:18.012 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.21427249000000001, 2024-02-09T17:39:18.012 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:18.013 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:18.013 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:18.013 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:18.013 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:18.013 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 78564 2024-02-09T17:39:18.013 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.013 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:18.013 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.013 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:18.013 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:15.974111+0000", 2024-02-09T17:39:18.013 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:18.013 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.013 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.014 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:18.014 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:15.974111+0000", 2024-02-09T17:39:18.014 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:18.014 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.014 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.014 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:18.014 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:15.974112+0000", 2024-02-09T17:39:18.014 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.009e-06 2024-02-09T17:39:18.014 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.014 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.014 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:18.014 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:15.974140+0000", 2024-02-09T17:39:18.014 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.8609999999999999e-05 2024-02-09T17:39:18.014 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.014 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.015 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:18.015 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:15.974154+0000", 2024-02-09T17:39:18.015 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3301e-05 2024-02-09T17:39:18.015 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.015 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.015 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:18.015 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:15.974162+0000", 2024-02-09T17:39:18.015 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.0669999999999992e-06 2024-02-09T17:39:18.015 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.015 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.015 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:18.015 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:15.976726+0000", 2024-02-09T17:39:18.015 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002564308 2024-02-09T17:39:18.015 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.016 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.016 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:18.016 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:15.976869+0000", 2024-02-09T17:39:18.016 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000142519 2024-02-09T17:39:18.016 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.016 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.016 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,11", 2024-02-09T17:39:18.016 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:15.977120+0000", 2024-02-09T17:39:18.016 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00025143399999999998 2024-02-09T17:39:18.016 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.016 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.016 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:18.016 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:15.982246+0000", 2024-02-09T17:39:18.016 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0051265060000000003 2024-02-09T17:39:18.017 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.017 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.017 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:18.017 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:15.982259+0000", 2024-02-09T17:39:18.017 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3002e-05 2024-02-09T17:39:18.017 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.017 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.017 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:18.017 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.188293+0000", 2024-02-09T17:39:18.017 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.206033042 2024-02-09T17:39:18.017 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.017 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.017 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:18.017 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.188350+0000", 2024-02-09T17:39:18.018 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7058999999999997e-05 2024-02-09T17:39:18.018 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.018 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.018 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:18.018 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.188383+0000", 2024-02-09T17:39:18.018 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.3633000000000001e-05 2024-02-09T17:39:18.018 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:18.018 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:18.018 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:18.018 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.018 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.018 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:78567 4.2 4:40a39246:::benchmark_data_smithi179_103809_object78566:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-02-09T17:39:18.018 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:36:15.977366+0000", 2024-02-09T17:39:18.018 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 182.02539735400001, 2024-02-09T17:39:18.018 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.21312125100000001, 2024-02-09T17:39:18.019 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:18.019 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:18.019 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:18.019 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:18.020 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:18.021 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 78567 2024-02-09T17:39:18.021 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.021 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:18.021 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.021 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:18.021 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:15.977366+0000", 2024-02-09T17:39:18.021 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:18.021 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.021 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.021 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:18.021 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:15.977366+0000", 2024-02-09T17:39:18.021 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:18.021 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.022 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.022 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:18.022 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:15.977367+0000", 2024-02-09T17:39:18.022 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.373e-06 2024-02-09T17:39:18.022 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.022 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.022 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:18.022 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:15.977390+0000", 2024-02-09T17:39:18.022 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2789000000000002e-05 2024-02-09T17:39:18.022 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.022 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.022 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:18.022 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:15.977401+0000", 2024-02-09T17:39:18.023 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1558e-05 2024-02-09T17:39:18.023 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.023 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.023 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:18.023 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:15.977410+0000", 2024-02-09T17:39:18.023 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.6780000000000006e-06 2024-02-09T17:39:18.023 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.023 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.023 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:18.023 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:15.979561+0000", 2024-02-09T17:39:18.023 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002151175 2024-02-09T17:39:18.023 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.023 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.024 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:18.024 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:15.979703+0000", 2024-02-09T17:39:18.024 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014191799999999999 2024-02-09T17:39:18.024 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.024 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.024 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,11", 2024-02-09T17:39:18.024 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:15.979789+0000", 2024-02-09T17:39:18.024 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5933000000000005e-05 2024-02-09T17:39:18.024 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.024 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.024 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:18.024 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:15.982266+0000", 2024-02-09T17:39:18.024 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0024773180000000001 2024-02-09T17:39:18.025 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.025 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.025 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:18.025 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:15.983653+0000", 2024-02-09T17:39:18.025 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001386682 2024-02-09T17:39:18.025 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.025 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.025 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:18.025 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.190424+0000", 2024-02-09T17:39:18.025 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.20677056499999999 2024-02-09T17:39:18.025 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.025 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.025 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:18.025 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.190459+0000", 2024-02-09T17:39:18.026 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.5172999999999997e-05 2024-02-09T17:39:18.026 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.026 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.026 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:18.026 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.190487+0000", 2024-02-09T17:39:18.026 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.8089000000000002e-05 2024-02-09T17:39:18.026 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:18.026 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:18.026 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:18.026 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.026 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.026 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:78613 4.2 4:4ae969f9:::benchmark_data_smithi179_103809_object78612:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-02-09T17:39:18.026 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:36:16.015426+0000", 2024-02-09T17:39:18.026 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 181.98733656300001, 2024-02-09T17:39:18.026 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.17521065399999999, 2024-02-09T17:39:18.027 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:18.027 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:18.027 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:18.027 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:18.027 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:18.027 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 78613 2024-02-09T17:39:18.027 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.027 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:18.027 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.027 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:18.027 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.015426+0000", 2024-02-09T17:39:18.027 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:18.027 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.027 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.027 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:18.028 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.015426+0000", 2024-02-09T17:39:18.028 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:18.028 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.028 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.028 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:18.028 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.015428+0000", 2024-02-09T17:39:18.028 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.195e-06 2024-02-09T17:39:18.028 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.028 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.028 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:18.028 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.015449+0000", 2024-02-09T17:39:18.028 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1585000000000001e-05 2024-02-09T17:39:18.028 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.028 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.029 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:18.029 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.015465+0000", 2024-02-09T17:39:18.029 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5670000000000001e-05 2024-02-09T17:39:18.029 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.029 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.029 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:18.029 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.015475+0000", 2024-02-09T17:39:18.029 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0015e-05 2024-02-09T17:39:18.029 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.029 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.030 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:18.030 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.015536+0000", 2024-02-09T17:39:18.030 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0986999999999997e-05 2024-02-09T17:39:18.030 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.030 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.030 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:18.030 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.015693+0000", 2024-02-09T17:39:18.030 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015718500000000001 2024-02-09T17:39:18.030 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.030 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.030 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,11", 2024-02-09T17:39:18.030 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.015811+0000", 2024-02-09T17:39:18.030 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000118306 2024-02-09T17:39:18.030 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.031 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.031 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:18.031 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.018489+0000", 2024-02-09T17:39:18.031 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0026774749999999999 2024-02-09T17:39:18.031 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.031 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.031 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:18.031 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.018791+0000", 2024-02-09T17:39:18.031 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00030174599999999998 2024-02-09T17:39:18.031 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.031 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.031 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:18.031 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.190538+0000", 2024-02-09T17:39:18.031 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.17174766499999999 2024-02-09T17:39:18.031 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.032 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.032 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:18.032 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.190599+0000", 2024-02-09T17:39:18.032 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1097000000000001e-05 2024-02-09T17:39:18.032 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.032 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.032 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:18.032 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.190637+0000", 2024-02-09T17:39:18.032 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7728000000000002e-05 2024-02-09T17:39:18.032 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:18.033 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:18.033 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:18.033 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.033 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.033 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:78628 4.2 4:42e97a50:::benchmark_data_smithi179_103809_object78627:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-02-09T17:39:18.033 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:36:16.025492+0000", 2024-02-09T17:39:18.033 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 181.977270514, 2024-02-09T17:39:18.033 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.16805967699999999, 2024-02-09T17:39:18.033 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:18.033 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:18.033 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:18.033 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:18.034 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:18.034 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 78628 2024-02-09T17:39:18.034 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.034 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:18.034 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.034 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:18.034 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.025492+0000", 2024-02-09T17:39:18.034 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:18.034 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.034 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.034 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:18.034 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.025492+0000", 2024-02-09T17:39:18.034 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:18.034 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.035 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.035 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:18.035 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.025494+0000", 2024-02-09T17:39:18.035 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.547e-06 2024-02-09T17:39:18.035 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.035 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.035 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:18.035 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.025521+0000", 2024-02-09T17:39:18.035 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6962999999999999e-05 2024-02-09T17:39:18.035 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.035 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.035 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:18.035 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.025537+0000", 2024-02-09T17:39:18.035 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6487999999999999e-05 2024-02-09T17:39:18.036 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.036 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.036 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:18.036 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.025546+0000", 2024-02-09T17:39:18.036 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2679999999999994e-06 2024-02-09T17:39:18.036 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.036 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.036 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:18.036 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.025599+0000", 2024-02-09T17:39:18.036 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2948000000000002e-05 2024-02-09T17:39:18.036 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.036 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.036 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:18.036 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.025680+0000", 2024-02-09T17:39:18.037 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1639999999999998e-05 2024-02-09T17:39:18.037 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.037 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.037 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,11", 2024-02-09T17:39:18.037 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.025783+0000", 2024-02-09T17:39:18.037 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000102589 2024-02-09T17:39:18.037 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.037 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.037 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:18.037 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.028467+0000", 2024-02-09T17:39:18.037 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0026837369999999998 2024-02-09T17:39:18.037 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.037 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.037 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:18.038 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.028681+0000", 2024-02-09T17:39:18.038 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00021443300000000001 2024-02-09T17:39:18.038 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.038 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.038 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:18.038 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.193461+0000", 2024-02-09T17:39:18.038 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.16478020600000001 2024-02-09T17:39:18.038 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.038 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.038 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:18.038 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.193516+0000", 2024-02-09T17:39:18.038 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4889000000000003e-05 2024-02-09T17:39:18.038 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.038 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.039 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:18.039 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.193552+0000", 2024-02-09T17:39:18.039 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.5969000000000001e-05 2024-02-09T17:39:18.039 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:18.039 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:18.039 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:18.039 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.039 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.039 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:78670 4.2 4:449b23d2:::benchmark_data_smithi179_103809_object78669:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-02-09T17:39:18.039 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:36:16.056733+0000", 2024-02-09T17:39:18.039 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 181.94603002900001, 2024-02-09T17:39:18.039 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.136957947, 2024-02-09T17:39:18.039 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:18.039 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:18.039 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:18.040 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:18.040 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:18.040 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 78670 2024-02-09T17:39:18.040 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.040 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:18.040 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.040 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:18.040 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.056733+0000", 2024-02-09T17:39:18.040 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:18.040 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.040 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.040 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:18.040 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.056733+0000", 2024-02-09T17:39:18.041 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:18.041 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.041 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.041 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:18.041 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.056735+0000", 2024-02-09T17:39:18.041 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.649e-06 2024-02-09T17:39:18.041 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.041 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.041 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:18.041 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.056761+0000", 2024-02-09T17:39:18.041 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6208e-05 2024-02-09T17:39:18.041 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.041 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.042 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:18.042 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.056780+0000", 2024-02-09T17:39:18.042 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8930999999999999e-05 2024-02-09T17:39:18.042 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.042 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.042 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:18.042 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.056789+0000", 2024-02-09T17:39:18.042 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6059999999999992e-06 2024-02-09T17:39:18.042 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.042 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.042 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:18.042 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.056859+0000", 2024-02-09T17:39:18.042 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9560999999999993e-05 2024-02-09T17:39:18.042 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.043 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.043 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:18.043 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.057003+0000", 2024-02-09T17:39:18.043 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014423699999999999 2024-02-09T17:39:18.043 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.043 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.043 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,11", 2024-02-09T17:39:18.043 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.057189+0000", 2024-02-09T17:39:18.043 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00018578999999999999 2024-02-09T17:39:18.043 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.043 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.043 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:18.043 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.059528+0000", 2024-02-09T17:39:18.043 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0023391190000000002 2024-02-09T17:39:18.044 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.044 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.044 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:18.044 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.059970+0000", 2024-02-09T17:39:18.044 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00044243100000000002 2024-02-09T17:39:18.044 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.044 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.044 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:18.044 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.193598+0000", 2024-02-09T17:39:18.044 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.133627616 2024-02-09T17:39:18.044 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.044 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.044 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:18.044 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.193655+0000", 2024-02-09T17:39:18.045 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6861999999999999e-05 2024-02-09T17:39:18.045 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.045 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.045 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:18.045 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.193691+0000", 2024-02-09T17:39:18.045 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.5936999999999998e-05 2024-02-09T17:39:18.045 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:18.045 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:18.045 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:18.045 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.045 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.045 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:78682 4.2 4:44b5306f:::benchmark_data_smithi179_103809_object78681:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-02-09T17:39:18.045 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:36:16.066758+0000", 2024-02-09T17:39:18.045 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 181.93600466800001, 2024-02-09T17:39:18.046 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.13008614099999999, 2024-02-09T17:39:18.046 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:18.046 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:18.046 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:18.046 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:18.046 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:18.046 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 78682 2024-02-09T17:39:18.046 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.046 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:18.046 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.046 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:18.046 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.066758+0000", 2024-02-09T17:39:18.046 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:18.046 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.046 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.047 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:18.047 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.066758+0000", 2024-02-09T17:39:18.047 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:18.047 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.047 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.047 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:18.047 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.066760+0000", 2024-02-09T17:39:18.047 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8619999999999999e-06 2024-02-09T17:39:18.047 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.047 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.047 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:18.047 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.066782+0000", 2024-02-09T17:39:18.047 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1882000000000002e-05 2024-02-09T17:39:18.047 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.047 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.048 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:18.048 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.066795+0000", 2024-02-09T17:39:18.048 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3222e-05 2024-02-09T17:39:18.048 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.048 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.048 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:18.048 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.066802+0000", 2024-02-09T17:39:18.048 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.613e-06 2024-02-09T17:39:18.048 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.048 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.048 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:18.048 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.066857+0000", 2024-02-09T17:39:18.048 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4861000000000002e-05 2024-02-09T17:39:18.048 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.049 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.049 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:18.049 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.066974+0000", 2024-02-09T17:39:18.049 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000117477 2024-02-09T17:39:18.049 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.049 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.049 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,11", 2024-02-09T17:39:18.049 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.067066+0000", 2024-02-09T17:39:18.049 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.1985999999999996e-05 2024-02-09T17:39:18.049 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.049 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.049 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:18.049 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.069616+0000", 2024-02-09T17:39:18.050 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0025499350000000001 2024-02-09T17:39:18.050 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.050 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.050 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:18.050 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.070077+0000", 2024-02-09T17:39:18.050 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00046079900000000002 2024-02-09T17:39:18.050 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.050 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.050 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:18.050 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.196771+0000", 2024-02-09T17:39:18.050 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.12669438099999999 2024-02-09T17:39:18.050 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.050 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.050 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:18.051 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.196805+0000", 2024-02-09T17:39:18.051 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.3785000000000003e-05 2024-02-09T17:39:18.051 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.051 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.051 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:18.051 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:36:16.196844+0000", 2024-02-09T17:39:18.051 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9338000000000002e-05 2024-02-09T17:39:18.051 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:18.051 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:18.051 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:18.051 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.051 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.051 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:193093 4.6 4:6a1f00b3:::benchmark_data_smithi179_103809_object55627:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-02-09T17:39:18.052 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:13.139224+0000", 2024-02-09T17:39:18.052 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 64.863538911999996, 2024-02-09T17:39:18.052 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.153935459, 2024-02-09T17:39:18.052 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:18.052 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:18.052 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:18.052 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:18.052 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:18.052 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 193093 2024-02-09T17:39:18.052 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.052 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:18.052 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.052 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:18.052 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:13.139224+0000", 2024-02-09T17:39:18.053 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:18.053 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.053 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.053 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:18.053 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:13.139224+0000", 2024-02-09T17:39:18.053 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:18.053 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.053 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.053 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:18.053 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:13.139226+0000", 2024-02-09T17:39:18.053 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5740000000000001e-06 2024-02-09T17:39:18.053 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.053 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.053 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:18.054 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:13.139231+0000", 2024-02-09T17:39:18.054 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6559999999999997e-06 2024-02-09T17:39:18.054 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.054 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.054 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:18.054 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:13.139247+0000", 2024-02-09T17:39:18.054 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5974999999999999e-05 2024-02-09T17:39:18.054 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.054 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.054 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:18.054 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:13.139256+0000", 2024-02-09T17:39:18.054 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.9290000000000004e-06 2024-02-09T17:39:18.054 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.054 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.054 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:18.055 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:13.139356+0000", 2024-02-09T17:39:18.055 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.9957000000000005e-05 2024-02-09T17:39:18.055 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.055 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.055 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:18.055 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:13.139482+0000", 2024-02-09T17:39:18.055 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012539699999999999 2024-02-09T17:39:18.055 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.055 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.055 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,2", 2024-02-09T17:39:18.055 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:13.139595+0000", 2024-02-09T17:39:18.055 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000113187 2024-02-09T17:39:18.055 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.055 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.055 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:18.056 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:13.144239+0000", 2024-02-09T17:39:18.056 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0046444549999999996 2024-02-09T17:39:18.056 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.056 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.056 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:18.056 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:13.151721+0000", 2024-02-09T17:39:18.056 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0074815580000000001 2024-02-09T17:39:18.056 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.056 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.056 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:18.056 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:13.293040+0000", 2024-02-09T17:39:18.056 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14131946100000001 2024-02-09T17:39:18.056 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.056 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.057 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:18.057 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:13.293107+0000", 2024-02-09T17:39:18.057 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7230000000000005e-05 2024-02-09T17:39:18.057 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.057 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.057 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:18.057 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:13.293160+0000", 2024-02-09T17:39:18.057 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2080000000000003e-05 2024-02-09T17:39:18.057 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:18.057 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:18.057 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:18.057 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.057 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.057 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:207380 4.2 4:4bb2b55e:::benchmark_data_smithi179_103809_object69914:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-02-09T17:39:18.058 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:32.357399+0000", 2024-02-09T17:39:18.058 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 45.645364178999998, 2024-02-09T17:39:18.058 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.161728278, 2024-02-09T17:39:18.058 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:18.058 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:18.058 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:18.058 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:18.058 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:18.058 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 207380 2024-02-09T17:39:18.058 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.058 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:18.058 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.058 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:18.058 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.357399+0000", 2024-02-09T17:39:18.059 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:18.059 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.059 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.059 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:18.059 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.357399+0000", 2024-02-09T17:39:18.059 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:18.059 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.059 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.059 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:18.059 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.357400+0000", 2024-02-09T17:39:18.059 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.671e-06 2024-02-09T17:39:18.059 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.059 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.059 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:18.059 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.357408+0000", 2024-02-09T17:39:18.060 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2869999999999999e-06 2024-02-09T17:39:18.060 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.060 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.060 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:18.060 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.357425+0000", 2024-02-09T17:39:18.060 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7672e-05 2024-02-09T17:39:18.060 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.060 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.060 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:18.060 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.357436+0000", 2024-02-09T17:39:18.060 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0118e-05 2024-02-09T17:39:18.061 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.061 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.061 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:18.061 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.357500+0000", 2024-02-09T17:39:18.061 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4488999999999999e-05 2024-02-09T17:39:18.061 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.061 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.061 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:18.061 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.357636+0000", 2024-02-09T17:39:18.061 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000136248 2024-02-09T17:39:18.061 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.061 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.061 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,11", 2024-02-09T17:39:18.061 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.357769+0000", 2024-02-09T17:39:18.062 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000133137 2024-02-09T17:39:18.062 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.062 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.062 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:18.062 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.361748+0000", 2024-02-09T17:39:18.062 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0039787859999999998 2024-02-09T17:39:18.062 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.062 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.062 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:18.062 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.363086+0000", 2024-02-09T17:39:18.062 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0013380239999999999 2024-02-09T17:39:18.062 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.062 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.062 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:18.062 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.519028+0000", 2024-02-09T17:39:18.063 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.155941686 2024-02-09T17:39:18.063 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.063 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.063 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:18.063 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.519088+0000", 2024-02-09T17:39:18.063 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9796000000000001e-05 2024-02-09T17:39:18.063 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.063 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.063 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:18.063 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.519127+0000", 2024-02-09T17:39:18.063 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9363999999999998e-05 2024-02-09T17:39:18.063 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:18.063 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:18.063 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:18.064 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.064 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.064 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:207389 4.2 4:415e4b43:::benchmark_data_smithi179_103809_object69923:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-02-09T17:39:18.064 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:32.382219+0000", 2024-02-09T17:39:18.064 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 45.620544316999997, 2024-02-09T17:39:18.064 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.13703472999999999, 2024-02-09T17:39:18.064 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:18.064 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:18.064 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:18.064 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:18.064 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:18.064 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 207389 2024-02-09T17:39:18.064 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.064 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:18.065 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.065 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:18.065 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.382219+0000", 2024-02-09T17:39:18.065 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:18.065 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.065 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.065 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:18.065 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.382219+0000", 2024-02-09T17:39:18.065 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:18.065 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.065 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.065 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:18.065 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.382220+0000", 2024-02-09T17:39:18.065 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5540000000000001e-06 2024-02-09T17:39:18.066 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.066 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.066 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:18.066 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.382226+0000", 2024-02-09T17:39:18.066 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4070000000000001e-06 2024-02-09T17:39:18.066 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.066 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.066 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:18.066 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.382239+0000", 2024-02-09T17:39:18.066 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.363e-05 2024-02-09T17:39:18.066 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.066 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.066 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:18.066 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.382248+0000", 2024-02-09T17:39:18.066 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2900000000000002e-06 2024-02-09T17:39:18.067 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.067 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.067 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:18.067 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.382308+0000", 2024-02-09T17:39:18.067 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0643000000000002e-05 2024-02-09T17:39:18.067 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.067 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.067 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:18.067 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.382448+0000", 2024-02-09T17:39:18.067 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013979199999999999 2024-02-09T17:39:18.067 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.067 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.067 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,11", 2024-02-09T17:39:18.068 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.382588+0000", 2024-02-09T17:39:18.068 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014004800000000001 2024-02-09T17:39:18.068 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.068 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.068 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:18.068 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.386836+0000", 2024-02-09T17:39:18.068 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0042483310000000002 2024-02-09T17:39:18.068 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.068 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.068 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:18.068 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.386854+0000", 2024-02-09T17:39:18.068 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.791e-05 2024-02-09T17:39:18.068 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.068 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.069 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:18.069 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.519175+0000", 2024-02-09T17:39:18.069 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.132320358 2024-02-09T17:39:18.069 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.069 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.069 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:18.069 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.519222+0000", 2024-02-09T17:39:18.069 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6986e-05 2024-02-09T17:39:18.069 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.069 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.069 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:18.069 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.519253+0000", 2024-02-09T17:39:18.069 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1780999999999999e-05 2024-02-09T17:39:18.069 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:18.069 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:18.070 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:18.070 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.070 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.070 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:208400 4.2 4:4b2cafe6:::benchmark_data_smithi179_103809_object70934:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-02-09T17:39:18.070 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:33.808091+0000", 2024-02-09T17:39:18.070 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 44.194671648000003, 2024-02-09T17:39:18.070 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3447615150000001, 2024-02-09T17:39:18.070 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:18.070 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-02-09T17:39:18.070 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:18.070 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:18.070 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:18.070 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 208400 2024-02-09T17:39:18.070 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.071 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:18.071 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.071 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:18.071 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.808091+0000", 2024-02-09T17:39:18.071 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:18.071 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.071 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.071 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:18.071 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.808091+0000", 2024-02-09T17:39:18.071 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:18.071 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.071 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.071 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:18.071 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.808093+0000", 2024-02-09T17:39:18.072 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.48e-06 2024-02-09T17:39:18.072 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.072 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.072 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:18.072 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.808099+0000", 2024-02-09T17:39:18.072 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9329999999999998e-06 2024-02-09T17:39:18.072 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.072 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.072 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:18.072 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.808113+0000", 2024-02-09T17:39:18.072 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4032000000000001e-05 2024-02-09T17:39:18.072 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.072 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.072 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:18.072 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.808120+0000", 2024-02-09T17:39:18.073 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7020000000000001e-06 2024-02-09T17:39:18.073 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.073 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.073 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:18.073 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.808181+0000", 2024-02-09T17:39:18.073 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0825999999999997e-05 2024-02-09T17:39:18.073 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.073 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.073 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:18.073 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.808278+0000", 2024-02-09T17:39:18.073 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6510999999999998e-05 2024-02-09T17:39:18.073 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.073 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.073 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,11", 2024-02-09T17:39:18.074 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.808354+0000", 2024-02-09T17:39:18.074 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6171e-05 2024-02-09T17:39:18.074 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.074 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.074 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:18.074 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.815614+0000", 2024-02-09T17:39:18.074 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0072599580000000004 2024-02-09T17:39:18.074 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.074 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.074 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:18.074 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.821051+0000", 2024-02-09T17:39:18.074 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.005437023 2024-02-09T17:39:18.074 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.075 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.075 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:39:18.075 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.821065+0000", 2024-02-09T17:39:18.075 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4011e-05 2024-02-09T17:39:18.075 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.075 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.075 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:18.075 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:34.812708+0000", 2024-02-09T17:39:18.075 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.99164264599999996 2024-02-09T17:39:18.075 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.075 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.075 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:39:18.075 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:34.812721+0000", 2024-02-09T17:39:18.075 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3475999999999999e-05 2024-02-09T17:39:18.076 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.076 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.076 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:18.076 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.152766+0000", 2024-02-09T17:39:18.076 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3400449459999999 2024-02-09T17:39:18.076 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.076 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.076 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:18.076 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.152853+0000", 2024-02-09T17:39:18.076 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.6799999999999996e-05 2024-02-09T17:39:18.076 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:18.076 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:18.076 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:18.076 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.077 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.077 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:209208 4.2 4:4d9658b8:::benchmark_data_smithi179_103809_object71742:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-02-09T17:39:18.077 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:37.305316+0000", 2024-02-09T17:39:18.077 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 40.69744687, 2024-02-09T17:39:18.077 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.218476264, 2024-02-09T17:39:18.077 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:18.077 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:18.077 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:18.077 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:18.077 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:18.077 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 209208 2024-02-09T17:39:18.077 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.077 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:18.077 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.077 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:18.078 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.305316+0000", 2024-02-09T17:39:18.078 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:18.078 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.078 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.078 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:18.078 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.305316+0000", 2024-02-09T17:39:18.078 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:18.078 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.078 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.078 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:18.078 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.305318+0000", 2024-02-09T17:39:18.078 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7549999999999999e-06 2024-02-09T17:39:18.078 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.078 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.079 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:18.079 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.305324+0000", 2024-02-09T17:39:18.079 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.533e-06 2024-02-09T17:39:18.079 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.079 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.079 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:18.079 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.305338+0000", 2024-02-09T17:39:18.079 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3716999999999999e-05 2024-02-09T17:39:18.079 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.079 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.079 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:18.079 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.305347+0000", 2024-02-09T17:39:18.079 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.9199999999999993e-06 2024-02-09T17:39:18.079 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.080 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.080 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:18.080 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.305412+0000", 2024-02-09T17:39:18.080 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5012999999999996e-05 2024-02-09T17:39:18.080 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.080 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.080 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:18.080 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.305574+0000", 2024-02-09T17:39:18.080 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016153599999999999 2024-02-09T17:39:18.080 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.080 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.080 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,8,11", 2024-02-09T17:39:18.080 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.305726+0000", 2024-02-09T17:39:18.080 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015199199999999999 2024-02-09T17:39:18.081 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.081 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.081 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:18.081 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.310758+0000", 2024-02-09T17:39:18.081 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0050327130000000003 2024-02-09T17:39:18.081 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.081 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.081 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:18.081 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.312487+0000", 2024-02-09T17:39:18.081 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0017286949999999999 2024-02-09T17:39:18.081 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.081 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.081 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:18.081 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.313534+0000", 2024-02-09T17:39:18.081 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001047405 2024-02-09T17:39:18.082 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.082 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.082 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:18.082 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.523599+0000", 2024-02-09T17:39:18.082 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.21006412599999999 2024-02-09T17:39:18.082 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.082 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.082 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:18.082 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.523667+0000", 2024-02-09T17:39:18.082 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8040999999999994e-05 2024-02-09T17:39:18.082 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.082 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.082 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:18.083 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.523792+0000", 2024-02-09T17:39:18.083 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012581800000000001 2024-02-09T17:39:18.083 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:18.083 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:18.083 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:18.083 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.083 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.083 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:209216 4.2 4:4b5c16ca:::benchmark_data_smithi179_103809_object71750:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-02-09T17:39:18.083 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:37.315200+0000", 2024-02-09T17:39:18.083 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 40.687562491000001, 2024-02-09T17:39:18.083 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.20880422400000001, 2024-02-09T17:39:18.083 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:18.083 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:18.083 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:18.084 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:18.084 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:18.084 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 209216 2024-02-09T17:39:18.084 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.084 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:18.084 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.084 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:18.084 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.315200+0000", 2024-02-09T17:39:18.084 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:18.084 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.084 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.084 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:18.084 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.315200+0000", 2024-02-09T17:39:18.084 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:18.085 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.085 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.085 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:18.085 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.315202+0000", 2024-02-09T17:39:18.085 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5260000000000001e-06 2024-02-09T17:39:18.085 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.085 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.085 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:18.085 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.315208+0000", 2024-02-09T17:39:18.085 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7309999999999998e-06 2024-02-09T17:39:18.085 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.085 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.085 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:18.085 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.315220+0000", 2024-02-09T17:39:18.086 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2167000000000001e-05 2024-02-09T17:39:18.086 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.086 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.086 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:18.086 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.315228+0000", 2024-02-09T17:39:18.086 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6969999999999993e-06 2024-02-09T17:39:18.086 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.086 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.086 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:18.086 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.315369+0000", 2024-02-09T17:39:18.086 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014159200000000001 2024-02-09T17:39:18.086 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.086 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.086 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:18.086 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.315496+0000", 2024-02-09T17:39:18.087 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000126477 2024-02-09T17:39:18.087 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.087 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.087 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,8,11", 2024-02-09T17:39:18.087 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.315627+0000", 2024-02-09T17:39:18.087 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000131562 2024-02-09T17:39:18.087 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.087 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.087 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:18.087 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.322165+0000", 2024-02-09T17:39:18.087 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.006538131 2024-02-09T17:39:18.087 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.087 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.087 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:18.087 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.329352+0000", 2024-02-09T17:39:18.088 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.007186947 2024-02-09T17:39:18.088 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.088 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.088 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:18.088 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.332320+0000", 2024-02-09T17:39:18.088 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002968142 2024-02-09T17:39:18.088 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.088 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.088 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:18.088 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.523840+0000", 2024-02-09T17:39:18.089 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.191519561 2024-02-09T17:39:18.089 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.089 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.089 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:18.089 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.523895+0000", 2024-02-09T17:39:18.089 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5000999999999997e-05 2024-02-09T17:39:18.089 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.089 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.089 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:18.089 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.524005+0000", 2024-02-09T17:39:18.089 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010969 2024-02-09T17:39:18.089 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:18.090 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:18.090 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:18.090 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.090 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.090 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:209219 4.2 4:4d2cfdf8:::benchmark_data_smithi179_103809_object71753:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-02-09T17:39:18.090 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:37.324408+0000", 2024-02-09T17:39:18.090 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 40.678354481, 2024-02-09T17:39:18.090 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.19981770700000001, 2024-02-09T17:39:18.090 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:18.090 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:18.090 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:18.090 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:18.090 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:18.091 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 209219 2024-02-09T17:39:18.091 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.091 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:18.091 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.091 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:18.091 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.324408+0000", 2024-02-09T17:39:18.091 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:18.091 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.091 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.091 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:18.091 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.324408+0000", 2024-02-09T17:39:18.091 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:18.091 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.091 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.092 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:18.092 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.324410+0000", 2024-02-09T17:39:18.092 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5510000000000001e-06 2024-02-09T17:39:18.092 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.092 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.092 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:18.092 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.324417+0000", 2024-02-09T17:39:18.092 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1219999999999997e-06 2024-02-09T17:39:18.092 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.092 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.092 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:18.092 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.324429+0000", 2024-02-09T17:39:18.092 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1741e-05 2024-02-09T17:39:18.092 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.093 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.093 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:18.093 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.324438+0000", 2024-02-09T17:39:18.093 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5470000000000002e-06 2024-02-09T17:39:18.093 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.093 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.093 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:18.093 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.324502+0000", 2024-02-09T17:39:18.093 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3635999999999999e-05 2024-02-09T17:39:18.093 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.093 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.093 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:18.094 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.324668+0000", 2024-02-09T17:39:18.094 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016551700000000001 2024-02-09T17:39:18.094 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.094 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.094 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,8,11", 2024-02-09T17:39:18.094 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.324820+0000", 2024-02-09T17:39:18.094 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015196900000000001 2024-02-09T17:39:18.094 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.094 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.094 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:18.094 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.329403+0000", 2024-02-09T17:39:18.094 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0045829399999999998 2024-02-09T17:39:18.094 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.095 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.095 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:18.095 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.332252+0000", 2024-02-09T17:39:18.095 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002849037 2024-02-09T17:39:18.095 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.095 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.095 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:18.095 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.332378+0000", 2024-02-09T17:39:18.095 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000126301 2024-02-09T17:39:18.095 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.095 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.095 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:18.095 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.524048+0000", 2024-02-09T17:39:18.095 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.191670117 2024-02-09T17:39:18.096 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.096 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.096 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:18.096 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.524110+0000", 2024-02-09T17:39:18.096 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2341999999999994e-05 2024-02-09T17:39:18.096 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.096 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.096 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:18.096 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.524226+0000", 2024-02-09T17:39:18.096 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000115887 2024-02-09T17:39:18.096 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:18.096 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:18.096 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:18.096 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.096 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.097 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:209236 4.2 4:4fc96872:::benchmark_data_smithi179_103809_object71770:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-02-09T17:39:18.097 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:37.345768+0000", 2024-02-09T17:39:18.097 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 40.656994959999999, 2024-02-09T17:39:18.097 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.178693559, 2024-02-09T17:39:18.097 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:18.097 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:18.097 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:18.097 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:18.097 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:18.097 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 209236 2024-02-09T17:39:18.097 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.097 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:18.097 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.097 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:18.098 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.345768+0000", 2024-02-09T17:39:18.098 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:18.098 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.098 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.098 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:18.098 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.345768+0000", 2024-02-09T17:39:18.098 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:18.098 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.098 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.098 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:18.098 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.345770+0000", 2024-02-09T17:39:18.098 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7770000000000001e-06 2024-02-09T17:39:18.098 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.098 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.099 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:18.099 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.345776+0000", 2024-02-09T17:39:18.099 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0170000000000002e-06 2024-02-09T17:39:18.099 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.099 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.099 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:18.099 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.345791+0000", 2024-02-09T17:39:18.099 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5641000000000002e-05 2024-02-09T17:39:18.099 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.099 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.099 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:18.099 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.345800+0000", 2024-02-09T17:39:18.099 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1580000000000005e-06 2024-02-09T17:39:18.099 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.099 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.100 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:18.100 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.345863+0000", 2024-02-09T17:39:18.100 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3812999999999994e-05 2024-02-09T17:39:18.100 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.100 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.100 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:18.100 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.345999+0000", 2024-02-09T17:39:18.100 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013552900000000001 2024-02-09T17:39:18.100 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.100 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.101 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,8,11", 2024-02-09T17:39:18.101 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.346096+0000", 2024-02-09T17:39:18.101 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6912999999999999e-05 2024-02-09T17:39:18.101 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.101 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.101 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:18.101 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.351978+0000", 2024-02-09T17:39:18.101 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0058820749999999996 2024-02-09T17:39:18.101 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.101 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.101 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:18.101 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.353224+0000", 2024-02-09T17:39:18.101 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001246521 2024-02-09T17:39:18.101 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.102 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.102 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:18.102 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.358127+0000", 2024-02-09T17:39:18.102 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0049025830000000003 2024-02-09T17:39:18.102 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.102 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.102 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:18.102 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.524271+0000", 2024-02-09T17:39:18.102 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.166144186 2024-02-09T17:39:18.102 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.102 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.102 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:18.102 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.524334+0000", 2024-02-09T17:39:18.102 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3032000000000002e-05 2024-02-09T17:39:18.103 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.103 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.103 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:18.103 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.524462+0000", 2024-02-09T17:39:18.103 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012731400000000001 2024-02-09T17:39:18.103 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:18.103 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:18.103 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:18.103 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.103 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.103 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:209242 4.2 4:48fdfb63:::benchmark_data_smithi179_103809_object71776:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-02-09T17:39:18.103 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:37.352490+0000", 2024-02-09T17:39:18.103 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 40.650272952000002, 2024-02-09T17:39:18.103 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.17222269400000001, 2024-02-09T17:39:18.104 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:18.104 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:18.104 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:18.104 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:18.104 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:18.104 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 209242 2024-02-09T17:39:18.104 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.104 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:18.104 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.104 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:18.104 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.352490+0000", 2024-02-09T17:39:18.104 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:18.104 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.104 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.105 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:18.105 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.352490+0000", 2024-02-09T17:39:18.105 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:18.105 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.105 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.105 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:18.105 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.352492+0000", 2024-02-09T17:39:18.105 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.683e-06 2024-02-09T17:39:18.105 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.105 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.105 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:18.105 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.352498+0000", 2024-02-09T17:39:18.105 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9989999999999997e-06 2024-02-09T17:39:18.105 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.106 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.106 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:18.106 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.352512+0000", 2024-02-09T17:39:18.106 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4012999999999999e-05 2024-02-09T17:39:18.106 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.106 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.106 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:18.106 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.352520+0000", 2024-02-09T17:39:18.106 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.6780000000000006e-06 2024-02-09T17:39:18.106 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.106 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.106 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:18.106 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.353308+0000", 2024-02-09T17:39:18.106 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000787183 2024-02-09T17:39:18.106 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.107 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.107 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:18.107 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.353408+0000", 2024-02-09T17:39:18.107 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000100499 2024-02-09T17:39:18.107 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.107 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.107 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,8,11", 2024-02-09T17:39:18.107 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.353494+0000", 2024-02-09T17:39:18.107 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5866999999999996e-05 2024-02-09T17:39:18.107 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.107 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.107 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:18.107 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.358721+0000", 2024-02-09T17:39:18.107 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0052271369999999998 2024-02-09T17:39:18.108 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.108 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.108 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:18.108 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.360198+0000", 2024-02-09T17:39:18.108 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0014773270000000001 2024-02-09T17:39:18.108 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.108 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.108 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:18.108 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.363499+0000", 2024-02-09T17:39:18.108 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.003300285 2024-02-09T17:39:18.108 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.108 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.108 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:18.109 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.524523+0000", 2024-02-09T17:39:18.109 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.16102476199999999 2024-02-09T17:39:18.109 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.109 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.109 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:18.109 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.524589+0000", 2024-02-09T17:39:18.109 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5259999999999995e-05 2024-02-09T17:39:18.109 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.109 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.109 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:18.109 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.524713+0000", 2024-02-09T17:39:18.109 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012400099999999999 2024-02-09T17:39:18.109 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:18.110 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:18.110 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:18.110 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.110 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.110 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:209256 4.2 4:445129f3:::benchmark_data_smithi179_103809_object71790:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-02-09T17:39:18.110 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:37.387963+0000", 2024-02-09T17:39:18.110 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 40.614800285999998, 2024-02-09T17:39:18.110 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.13698802600000001, 2024-02-09T17:39:18.110 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:18.110 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:18.110 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:18.110 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:18.110 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:18.110 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 209256 2024-02-09T17:39:18.111 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.111 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:18.111 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.111 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:18.111 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.387963+0000", 2024-02-09T17:39:18.111 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:18.111 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.111 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.111 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:18.111 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.387963+0000", 2024-02-09T17:39:18.111 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:18.111 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.111 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.111 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:18.112 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.387964+0000", 2024-02-09T17:39:18.112 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3349999999999999e-06 2024-02-09T17:39:18.112 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.112 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.112 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:18.112 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.387971+0000", 2024-02-09T17:39:18.112 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4749999999999998e-06 2024-02-09T17:39:18.112 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.112 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.112 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:18.112 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.387987+0000", 2024-02-09T17:39:18.112 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6526999999999999e-05 2024-02-09T17:39:18.112 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.112 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.113 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:18.113 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.387997+0000", 2024-02-09T17:39:18.113 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0355e-05 2024-02-09T17:39:18.113 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.113 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.113 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:18.113 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.388078+0000", 2024-02-09T17:39:18.113 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.0594999999999997e-05 2024-02-09T17:39:18.113 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.113 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.113 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:18.113 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.388233+0000", 2024-02-09T17:39:18.114 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015454100000000001 2024-02-09T17:39:18.114 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.114 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.114 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,8,11", 2024-02-09T17:39:18.114 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.388375+0000", 2024-02-09T17:39:18.114 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000142614 2024-02-09T17:39:18.114 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.114 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.114 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:18.114 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.394143+0000", 2024-02-09T17:39:18.114 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0057683220000000002 2024-02-09T17:39:18.114 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.114 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.114 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:18.114 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.397239+0000", 2024-02-09T17:39:18.115 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0030957960000000001 2024-02-09T17:39:18.115 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.115 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.115 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:18.115 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.400285+0000", 2024-02-09T17:39:18.115 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0030453189999999999 2024-02-09T17:39:18.115 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.115 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.115 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:18.115 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.524772+0000", 2024-02-09T17:39:18.115 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.12448747 2024-02-09T17:39:18.115 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.116 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.116 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:18.116 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.524830+0000", 2024-02-09T17:39:18.116 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8417e-05 2024-02-09T17:39:18.116 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.116 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.116 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:18.116 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.524951+0000", 2024-02-09T17:39:18.116 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012026 2024-02-09T17:39:18.116 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:18.116 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:18.116 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:18.116 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.116 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.117 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:222253 4.6 4:62bd8438:::benchmark_data_smithi179_103809_object84787:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e397)", 2024-02-09T17:39:18.117 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:39:02.580398+0000", 2024-02-09T17:39:18.117 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 15.422364783999999, 2024-02-09T17:39:18.117 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0475380700000001, 2024-02-09T17:39:18.117 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:18.117 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-02-09T17:39:18.117 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:18.117 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:18.117 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:18.117 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 222253 2024-02-09T17:39:18.117 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.117 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:18.117 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.117 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:18.118 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.580398+0000", 2024-02-09T17:39:18.118 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:18.118 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.118 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.118 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:18.118 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.580398+0000", 2024-02-09T17:39:18.118 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:18.118 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.118 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.118 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:18.118 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.580399+0000", 2024-02-09T17:39:18.118 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6800000000000001e-07 2024-02-09T17:39:18.118 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.118 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.119 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:18.119 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.580401+0000", 2024-02-09T17:39:18.119 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6010000000000002e-06 2024-02-09T17:39:18.119 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.119 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.119 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:18.119 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.580409+0000", 2024-02-09T17:39:18.119 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1119999999999996e-06 2024-02-09T17:39:18.119 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.119 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.119 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:18.119 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.580415+0000", 2024-02-09T17:39:18.119 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6540000000000004e-06 2024-02-09T17:39:18.119 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.120 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.120 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:18.120 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.580470+0000", 2024-02-09T17:39:18.120 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5068999999999997e-05 2024-02-09T17:39:18.120 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.120 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.120 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:18.120 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.580565+0000", 2024-02-09T17:39:18.120 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.4953999999999999e-05 2024-02-09T17:39:18.120 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.120 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.120 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,2,7", 2024-02-09T17:39:18.120 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.580654+0000", 2024-02-09T17:39:18.121 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8467000000000005e-05 2024-02-09T17:39:18.121 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.121 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.121 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:18.121 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.590354+0000", 2024-02-09T17:39:18.121 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0097001880000000002 2024-02-09T17:39:18.121 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.121 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.121 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:18.121 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.590379+0000", 2024-02-09T17:39:18.121 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5245e-05 2024-02-09T17:39:18.121 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.121 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.121 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:18.122 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.594262+0000", 2024-02-09T17:39:18.122 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0038834389999999998 2024-02-09T17:39:18.122 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.122 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.122 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:39:18.122 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.594278+0000", 2024-02-09T17:39:18.122 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5860999999999999e-05 2024-02-09T17:39:18.122 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.122 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.122 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:18.122 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:03.627880+0000", 2024-02-09T17:39:18.122 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.033601583 2024-02-09T17:39:18.122 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.122 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.122 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:18.123 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:03.627936+0000", 2024-02-09T17:39:18.123 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6428999999999998e-05 2024-02-09T17:39:18.123 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:18.123 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:18.123 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:18.123 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.123 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.123 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:222261 4.6 4:64982b2b:::benchmark_data_smithi179_103809_object84795:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e397)", 2024-02-09T17:39:18.123 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:39:02.584432+0000", 2024-02-09T17:39:18.123 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 15.418330571, 2024-02-09T17:39:18.123 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.044083275, 2024-02-09T17:39:18.123 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:18.123 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-02-09T17:39:18.123 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:18.123 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:18.124 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:18.124 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 222261 2024-02-09T17:39:18.124 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.124 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:18.124 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.124 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:18.124 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.584432+0000", 2024-02-09T17:39:18.124 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:18.124 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.124 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.124 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:18.124 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.584432+0000", 2024-02-09T17:39:18.124 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:18.125 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.125 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.125 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:18.125 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.584434+0000", 2024-02-09T17:39:18.125 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.513e-06 2024-02-09T17:39:18.125 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.125 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.125 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:18.125 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.584440+0000", 2024-02-09T17:39:18.125 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1210000000000003e-06 2024-02-09T17:39:18.125 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.125 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.125 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:18.125 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.584455+0000", 2024-02-09T17:39:18.126 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4980000000000001e-05 2024-02-09T17:39:18.126 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.126 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.126 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:18.126 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.584464+0000", 2024-02-09T17:39:18.126 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8009999999999993e-06 2024-02-09T17:39:18.126 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.126 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.126 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:18.126 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.584927+0000", 2024-02-09T17:39:18.126 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00046321200000000001 2024-02-09T17:39:18.126 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.126 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.126 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:18.126 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.585045+0000", 2024-02-09T17:39:18.127 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000117528 2024-02-09T17:39:18.127 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.127 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.127 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,2,7", 2024-02-09T17:39:18.127 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.585122+0000", 2024-02-09T17:39:18.127 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7447999999999994e-05 2024-02-09T17:39:18.127 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.127 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.127 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:18.127 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.594348+0000", 2024-02-09T17:39:18.127 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0092264579999999999 2024-02-09T17:39:18.127 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.127 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.127 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:39:18.127 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.594361+0000", 2024-02-09T17:39:18.128 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2779000000000001e-05 2024-02-09T17:39:18.128 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.128 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.128 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:18.128 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:03.628471+0000", 2024-02-09T17:39:18.128 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0341101669999999 2024-02-09T17:39:18.128 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.128 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.128 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:18.128 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:03.628516+0000", 2024-02-09T17:39:18.128 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4267999999999997e-05 2024-02-09T17:39:18.128 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:18.128 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:18.128 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:18.129 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.129 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.129 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:222276 4.6 4:6cb938f4:::benchmark_data_smithi179_103809_object84810:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-02-09T17:39:18.129 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:39:02.616343+0000", 2024-02-09T17:39:18.129 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 15.386420371, 2024-02-09T17:39:18.129 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0185018139999999, 2024-02-09T17:39:18.129 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:18.129 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:18.129 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:18.129 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:18.129 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:18.129 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 222276 2024-02-09T17:39:18.129 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.129 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:18.130 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.130 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:18.130 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.616343+0000", 2024-02-09T17:39:18.130 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:18.130 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.130 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.130 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:18.130 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.616343+0000", 2024-02-09T17:39:18.130 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:18.130 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.130 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.130 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:18.130 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.616344+0000", 2024-02-09T17:39:18.130 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.299e-06 2024-02-09T17:39:18.130 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.131 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.131 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:18.131 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.616349+0000", 2024-02-09T17:39:18.131 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7759999999999997e-06 2024-02-09T17:39:18.131 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.131 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.131 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:18.131 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.616361+0000", 2024-02-09T17:39:18.131 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2628e-05 2024-02-09T17:39:18.131 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.131 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.131 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:18.131 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.616369+0000", 2024-02-09T17:39:18.131 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.9049999999999997e-06 2024-02-09T17:39:18.132 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.132 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.132 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:18.132 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.616431+0000", 2024-02-09T17:39:18.132 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2003999999999998e-05 2024-02-09T17:39:18.132 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.132 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.132 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:39:18.132 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.616443+0000", 2024-02-09T17:39:18.132 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1503e-05 2024-02-09T17:39:18.132 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.132 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.132 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:18.132 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:03.628548+0000", 2024-02-09T17:39:18.132 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0121055539999999 2024-02-09T17:39:18.133 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.133 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.133 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:18.133 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:03.628687+0000", 2024-02-09T17:39:18.133 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000138675 2024-02-09T17:39:18.133 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.133 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.133 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,7", 2024-02-09T17:39:18.133 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:03.628832+0000", 2024-02-09T17:39:18.133 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000144967 2024-02-09T17:39:18.133 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.133 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.133 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:18.133 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:03.634518+0000", 2024-02-09T17:39:18.134 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0056860340000000004 2024-02-09T17:39:18.134 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.134 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.134 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:18.134 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:03.634577+0000", 2024-02-09T17:39:18.134 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9253999999999999e-05 2024-02-09T17:39:18.134 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.134 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.134 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:18.134 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:03.634726+0000", 2024-02-09T17:39:18.134 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014919 2024-02-09T17:39:18.134 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.134 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.134 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:18.134 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:03.634790+0000", 2024-02-09T17:39:18.135 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3972000000000003e-05 2024-02-09T17:39:18.135 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:18.135 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:18.135 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:18.135 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:03.634844+0000", 2024-02-09T17:39:18.135 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4052999999999999e-05 2024-02-09T17:39:18.135 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:18.135 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:18.135 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:18.135 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:18.135 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:18.135 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:39:18.136 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:39:18.171 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:18.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:17 smithi162 ceph-mon[44304]: pgmap v1084: 37 pgs: 1 active+remapped+backfill_wait, 1 peering, 8 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 25 active+clean; 2.8 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 532 op/s; 35811/139794 objects misplaced (25.617%); 71 MiB/s, 1.14k objects/s recovering 2024-02-09T17:39:18.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:17 smithi162 ceph-mon[44304]: osdmap e409: 12 total, 12 up, 11 in 2024-02-09T17:39:18.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:17 smithi162 ceph-mon[44304]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:39:18.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:17 smithi099 ceph-mon[39420]: pgmap v1084: 37 pgs: 1 active+remapped+backfill_wait, 1 peering, 8 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 25 active+clean; 2.8 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 532 op/s; 35811/139794 objects misplaced (25.617%); 71 MiB/s, 1.14k objects/s recovering 2024-02-09T17:39:18.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:17 smithi099 ceph-mon[39420]: osdmap e409: 12 total, 12 up, 11 in 2024-02-09T17:39:18.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:17 smithi099 ceph-mon[39420]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:39:18.815 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.10 dump_ops_in_flight 2024-02-09T17:39:18.825 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:39:18.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:18.829 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:39:18.930 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:39:18.992 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:19.031 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:39:19.039 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:19.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:39:19.141 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:39:19.195 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:19.235 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:39:19.238 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:39:19.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:19.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:39:19.340 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:39:19.386 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:19.422 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:39:19.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:19.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:39:19.483 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:19 smithi162 ceph-mon[44304]: osdmap e410: 12 total, 12 up, 11 in 2024-02-09T17:39:19.483 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:19 smithi162 ceph-mon[44304]: pgmap v1087: 37 pgs: 2 active+remapped+backfill_wait, 3 peering, 7 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 23 active+clean; 2.8 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 768 B/s rd, 22 KiB/s wr, 560 op/s; 30922/137544 objects misplaced (22.482%); 117 MiB/s, 1.87k objects/s recovering 2024-02-09T17:39:19.529 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:39:19.583 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:19.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:19 smithi099 ceph-mon[39420]: osdmap e410: 12 total, 12 up, 11 in 2024-02-09T17:39:19.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:19 smithi099 ceph-mon[39420]: pgmap v1087: 37 pgs: 2 active+remapped+backfill_wait, 3 peering, 7 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 23 active+clean; 2.8 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 768 B/s rd, 22 KiB/s wr, 560 op/s; 30922/137544 objects misplaced (22.482%); 117 MiB/s, 1.87k objects/s recovering 2024-02-09T17:39:19.803 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] 2024-02-09T17:39:19.804 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 1 1 2024-02-09T17:39:19.861 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:19 smithi179 ceph-mon[43975]: osdmap e410: 12 total, 12 up, 11 in 2024-02-09T17:39:19.861 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:19 smithi179 ceph-mon[43975]: pgmap v1087: 37 pgs: 2 active+remapped+backfill_wait, 3 peering, 7 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 23 active+clean; 2.8 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 768 B/s rd, 22 KiB/s wr, 560 op/s; 30922/137544 objects misplaced (22.482%); 117 MiB/s, 1.87k objects/s recovering 2024-02-09T17:39:20.190 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:39:20.219 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:39:20.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:20.222 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:39:20.323 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:39:20.355 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:20.497 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is set 2024-02-09T17:39:20.709 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:39:20.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:20.714 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:39:20.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:20 smithi099 ceph-mon[39420]: osdmap e411: 12 total, 12 up, 11 in 2024-02-09T17:39:20.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:20 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1984736622' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:39:20.815 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:39:20.877 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:20.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:20 smithi179 ceph-mon[43975]: osdmap e411: 12 total, 12 up, 11 in 2024-02-09T17:39:20.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:20 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1984736622' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:39:20.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:20 smithi162 ceph-mon[44304]: osdmap e411: 12 total, 12 up, 11 in 2024-02-09T17:39:20.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:20 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1984736622' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:39:20.971 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:39:20.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:20.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:39:21.085 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:39:21.133 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:21.197 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:39:21.453 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:39:21.454 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-02-09T17:39:21.454 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.454 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop_reply(client.25201.0:231638 4.5 e412/380)", 2024-02-09T17:39:21.454 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:39:21.448371+0000", 2024-02-09T17:39:21.454 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 6.1112999999999996e-05, 2024-02-09T17:39:21.454 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.9993999999999993e-05, 2024-02-09T17:39:21.454 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:21.454 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "queued for pg", 2024-02-09T17:39:21.454 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:21.454 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.454 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:21.454 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:21.448371+0000", 2024-02-09T17:39:21.454 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:21.455 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.455 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.455 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:21.455 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:21.448371+0000", 2024-02-09T17:39:21.455 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:21.455 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.455 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.455 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:21.455 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:21.448372+0000", 2024-02-09T17:39:21.455 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4000000000000001e-07 2024-02-09T17:39:21.455 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.455 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.455 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:21.455 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:21.448377+0000", 2024-02-09T17:39:21.456 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4950000000000002e-06 2024-02-09T17:39:21.456 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.456 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.456 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:21.456 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:21.448390+0000", 2024-02-09T17:39:21.456 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2849e-05 2024-02-09T17:39:21.456 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.456 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.456 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:21.456 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:21.448396+0000", 2024-02-09T17:39:21.456 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3770000000000001e-06 2024-02-09T17:39:21.456 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:21.456 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:21.456 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:21.457 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.457 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.457 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.25201.0:231643 4.e e412/411)", 2024-02-09T17:39:21.457 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:39:21.448427+0000", 2024-02-09T17:39:21.457 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 5.6189999999999999e-06, 2024-02-09T17:39:21.457 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5767000000000003e-05, 2024-02-09T17:39:21.457 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:21.457 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "queued for pg", 2024-02-09T17:39:21.457 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:21.457 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.457 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:21.457 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:21.448427+0000", 2024-02-09T17:39:21.457 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:21.457 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.458 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.458 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:21.458 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:21.448427+0000", 2024-02-09T17:39:21.458 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:21.458 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.458 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.458 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:21.458 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:21.448427+0000", 2024-02-09T17:39:21.458 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5299999999999999e-07 2024-02-09T17:39:21.458 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.458 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.458 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:21.458 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:21.448432+0000", 2024-02-09T17:39:21.458 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.977e-06 2024-02-09T17:39:21.459 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.459 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.459 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:21.459 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:21.448439+0000", 2024-02-09T17:39:21.459 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.956e-06 2024-02-09T17:39:21.459 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.459 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.459 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:21.459 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:21.448445+0000", 2024-02-09T17:39:21.459 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3920000000000002e-06 2024-02-09T17:39:21.459 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.459 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.459 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:21.459 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:21.448484+0000", 2024-02-09T17:39:21.460 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9190999999999998e-05 2024-02-09T17:39:21.460 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:21.460 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:21.460 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:21.460 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.460 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.460 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:231638 4.5 4:a02ceb66:::benchmark_data_smithi179_103809_object94172:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-02-09T17:39:21.460 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:39:21.435101+0000", 2024-02-09T17:39:21.460 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.013331882, 2024-02-09T17:39:21.460 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013420369999999999, 2024-02-09T17:39:21.460 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:21.460 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "waiting for sub ops", 2024-02-09T17:39:21.460 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:21.460 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:21.461 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:21.461 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 231638 2024-02-09T17:39:21.461 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.461 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:21.461 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.461 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:21.461 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:21.435101+0000", 2024-02-09T17:39:21.461 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:21.461 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.461 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.461 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:21.461 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:21.435101+0000", 2024-02-09T17:39:21.461 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:21.461 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.461 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.461 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:21.462 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:21.435102+0000", 2024-02-09T17:39:21.462 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5319999999999999e-06 2024-02-09T17:39:21.462 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.462 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.462 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:21.462 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:21.435108+0000", 2024-02-09T17:39:21.462 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0050000000000001e-06 2024-02-09T17:39:21.462 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.462 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.462 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:21.462 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:21.435122+0000", 2024-02-09T17:39:21.462 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3451999999999999e-05 2024-02-09T17:39:21.462 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.462 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.462 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:21.463 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:21.435142+0000", 2024-02-09T17:39:21.463 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0098999999999998e-05 2024-02-09T17:39:21.463 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.463 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.463 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:21.463 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:21.436613+0000", 2024-02-09T17:39:21.463 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00147073 2024-02-09T17:39:21.463 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.463 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.463 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:21.463 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:21.436750+0000", 2024-02-09T17:39:21.463 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013773100000000001 2024-02-09T17:39:21.463 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.463 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.463 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,6,7,11", 2024-02-09T17:39:21.464 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:21.436897+0000", 2024-02-09T17:39:21.464 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014646799999999999 2024-02-09T17:39:21.464 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.464 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.464 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:21.464 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:21.446546+0000", 2024-02-09T17:39:21.464 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0096495869999999994 2024-02-09T17:39:21.464 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.464 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.464 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:21.464 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:21.446794+0000", 2024-02-09T17:39:21.464 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000247531 2024-02-09T17:39:21.464 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.464 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.464 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:21.465 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:21.447576+0000", 2024-02-09T17:39:21.465 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00078167999999999998 2024-02-09T17:39:21.465 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.465 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.465 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:21.465 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:21.447631+0000", 2024-02-09T17:39:21.465 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5402999999999998e-05 2024-02-09T17:39:21.465 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:21.465 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:21.465 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:21.465 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.465 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.465 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:231640 4.7 4:e34f7886:::benchmark_data_smithi179_103809_object94174:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-02-09T17:39:21.465 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:39:21.440214+0000", 2024-02-09T17:39:21.466 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.0082184679999999996, 2024-02-09T17:39:21.466 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0084184120000000001, 2024-02-09T17:39:21.466 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:21.466 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "waiting for sub ops", 2024-02-09T17:39:21.466 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:21.466 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:21.466 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:21.466 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 231640 2024-02-09T17:39:21.466 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.466 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:21.466 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.466 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:21.466 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:21.440214+0000", 2024-02-09T17:39:21.466 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:21.466 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.466 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.467 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:21.467 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:21.440214+0000", 2024-02-09T17:39:21.467 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:21.467 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.467 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.467 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:21.467 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:21.440216+0000", 2024-02-09T17:39:21.467 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.001e-06 2024-02-09T17:39:21.467 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.467 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.467 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:21.467 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:21.440224+0000", 2024-02-09T17:39:21.467 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7540000000000006e-06 2024-02-09T17:39:21.468 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.468 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.468 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:21.468 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:21.440243+0000", 2024-02-09T17:39:21.468 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.946e-05 2024-02-09T17:39:21.468 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.468 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.468 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:21.468 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:21.440254+0000", 2024-02-09T17:39:21.468 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0614999999999999e-05 2024-02-09T17:39:21.468 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.468 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.468 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:21.468 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:21.440328+0000", 2024-02-09T17:39:21.469 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.3716999999999999e-05 2024-02-09T17:39:21.469 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.469 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.469 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:21.469 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:21.440458+0000", 2024-02-09T17:39:21.469 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013021899999999999 2024-02-09T17:39:21.469 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.469 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.469 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,7", 2024-02-09T17:39:21.469 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:21.440581+0000", 2024-02-09T17:39:21.469 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012354099999999999 2024-02-09T17:39:21.469 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.469 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.469 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:21.470 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:21.445637+0000", 2024-02-09T17:39:21.470 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0050556430000000003 2024-02-09T17:39:21.470 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:21.470 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:21.470 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:21.470 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.470 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.470 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:231641 4.5 4:ad50388d:::benchmark_data_smithi179_103809_object94175:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-02-09T17:39:21.470 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:39:21.440280+0000", 2024-02-09T17:39:21.470 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.0081524289999999992, 2024-02-09T17:39:21.470 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0084048379999999995, 2024-02-09T17:39:21.470 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:21.470 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "waiting for sub ops", 2024-02-09T17:39:21.470 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:21.471 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:21.471 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:21.471 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 231641 2024-02-09T17:39:21.471 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.471 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:21.471 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.471 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:21.471 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:21.440280+0000", 2024-02-09T17:39:21.471 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:21.471 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.471 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.471 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:21.471 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:21.440280+0000", 2024-02-09T17:39:21.471 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:21.471 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.472 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.472 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:21.472 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:21.440281+0000", 2024-02-09T17:39:21.472 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0999999999999999e-07 2024-02-09T17:39:21.472 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.472 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.472 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:21.472 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:21.440283+0000", 2024-02-09T17:39:21.472 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7149999999999998e-06 2024-02-09T17:39:21.472 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.472 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.472 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:21.472 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:21.440290+0000", 2024-02-09T17:39:21.472 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3770000000000001e-06 2024-02-09T17:39:21.472 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.473 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.473 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:21.473 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:21.440296+0000", 2024-02-09T17:39:21.473 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1199999999999999e-06 2024-02-09T17:39:21.473 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.473 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.473 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:21.473 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:21.441659+0000", 2024-02-09T17:39:21.473 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0013630339999999999 2024-02-09T17:39:21.473 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.473 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.473 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:21.473 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:21.441772+0000", 2024-02-09T17:39:21.473 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000113276 2024-02-09T17:39:21.473 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.474 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.474 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,6,7,11", 2024-02-09T17:39:21.474 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:21.441864+0000", 2024-02-09T17:39:21.474 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.1948999999999997e-05 2024-02-09T17:39:21.474 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.474 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.474 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:21.474 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:21.446555+0000", 2024-02-09T17:39:21.474 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00469141 2024-02-09T17:39:21.474 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.474 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.474 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:21.474 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:21.446957+0000", 2024-02-09T17:39:21.474 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00040190700000000001 2024-02-09T17:39:21.474 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:21.475 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:21.475 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:21.475 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:21.447683+0000", 2024-02-09T17:39:21.475 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00072608700000000004 2024-02-09T17:39:21.475 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:21.475 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:21.475 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:21.475 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:21.475 INFO:teuthology.orchestra.run.smithi179.stdout: ], 2024-02-09T17:39:21.475 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 5 2024-02-09T17:39:21.475 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:39:21.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:21 smithi099 ceph-mon[39420]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:39:21.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:21 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1984736622' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:39:21.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:21 smithi099 ceph-mon[39420]: osdmap e412: 12 total, 12 up, 11 in 2024-02-09T17:39:21.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:21 smithi099 ceph-mon[39420]: pgmap v1090: 37 pgs: 2 active+remapped+backfill_wait, 2 peering, 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 26 active+clean; 2.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 767 B/s rd, 22 KiB/s wr, 478 op/s; 25666/134079 objects misplaced (19.142%); 98 MiB/s, 1.57k objects/s recovering 2024-02-09T17:39:21.818 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:39:21.822 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:21 smithi179 ceph-mon[43975]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:39:21.822 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:21 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1984736622' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:39:21.822 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:21 smithi179 ceph-mon[43975]: osdmap e412: 12 total, 12 up, 11 in 2024-02-09T17:39:21.823 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:21 smithi179 ceph-mon[43975]: pgmap v1090: 37 pgs: 2 active+remapped+backfill_wait, 2 peering, 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 26 active+clean; 2.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 767 B/s rd, 22 KiB/s wr, 478 op/s; 25666/134079 objects misplaced (19.142%); 98 MiB/s, 1.57k objects/s recovering 2024-02-09T17:39:21.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:21.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:39:21.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:21 smithi162 ceph-mon[44304]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:39:21.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:21 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1984736622' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:39:21.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:21 smithi162 ceph-mon[44304]: osdmap e412: 12 total, 12 up, 11 in 2024-02-09T17:39:21.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:21 smithi162 ceph-mon[44304]: pgmap v1090: 37 pgs: 2 active+remapped+backfill_wait, 2 peering, 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 26 active+clean; 2.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 767 B/s rd, 22 KiB/s wr, 478 op/s; 25666/134079 objects misplaced (19.142%); 98 MiB/s, 1.57k objects/s recovering 2024-02-09T17:39:21.929 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:39:21.986 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:22.237 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.10 dump_blocked_ops 2024-02-09T17:39:22.357 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:39:22.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:22.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:39:22.466 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:39:22.553 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:22.592 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:39:22.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:22.595 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:39:22.642 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:22 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:39:22.642 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:22 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:39:22.642 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:22 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:39:22.642 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:22 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:39:22.677 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:39:22.696 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:39:22.745 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:22.774 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:39:22.777 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:22 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:39:22.777 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:22 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:39:22.777 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:22 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:39:22.777 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:22 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:39:22.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:22.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:39:22.879 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:39:22.913 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:22.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:22 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:39:22.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:22 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:39:22.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:22 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:39:22.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:22 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:39:22.935 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:39:22.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:22.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:39:23.039 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:39:23.071 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:23.095 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:39:23.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:23.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:39:23.199 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:39:23.233 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:23.531 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:23 smithi179 ceph-mon[43975]: osdmap e413: 12 total, 12 up, 11 in 2024-02-09T17:39:23.531 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:23 smithi179 ceph-mon[43975]: pgmap v1092: 37 pgs: 1 active+remapped+backfill_wait, 1 active+clean+remapped, 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 28 active+clean; 2.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 497 op/s; 22103/131481 objects misplaced (16.811%); 78 MiB/s, 1.25k objects/s recovering 2024-02-09T17:39:23.531 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:23 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3771598195' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-09T17:39:23.865 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.1 to 1 (10000) 2024-02-09T17:39:23.871 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:39:23.885 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:39:23.887 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:23 smithi099 ceph-mon[39420]: osdmap e413: 12 total, 12 up, 11 in 2024-02-09T17:39:23.887 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:23 smithi099 ceph-mon[39420]: pgmap v1092: 37 pgs: 1 active+remapped+backfill_wait, 1 active+clean+remapped, 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 28 active+clean; 2.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 497 op/s; 22103/131481 objects misplaced (16.811%); 78 MiB/s, 1.25k objects/s recovering 2024-02-09T17:39:23.887 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:23 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3771598195' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-09T17:39:23.888 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:23.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:39:23.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:23 smithi162 ceph-mon[44304]: osdmap e413: 12 total, 12 up, 11 in 2024-02-09T17:39:23.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:23 smithi162 ceph-mon[44304]: pgmap v1092: 37 pgs: 1 active+remapped+backfill_wait, 1 active+clean+remapped, 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 28 active+clean; 2.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 497 op/s; 22103/131481 objects misplaced (16.811%); 78 MiB/s, 1.25k objects/s recovering 2024-02-09T17:39:23.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:23 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3771598195' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-09T17:39:23.990 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:39:24.016 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:24.036 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:39:24.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:24.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:39:24.139 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:39:24.165 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:24.184 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:39:24.186 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:24.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:39:24.287 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:39:24.319 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:24.337 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:39:24.339 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:24.340 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:39:24.401 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset noscrub 2024-02-09T17:39:24.440 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:39:24.469 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:24.536 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:24 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3771598195' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-09T17:39:24.536 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:24 smithi179 ceph-mon[43975]: osdmap e414: 12 total, 12 up, 11 in 2024-02-09T17:39:24.566 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:39:24.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:24.569 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:39:24.579 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:39:24.579 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-02-09T17:39:24.579 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-02-09T17:39:24.579 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-02-09T17:39:24.579 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:39:24.670 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:39:24.699 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:24 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3771598195' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-09T17:39:24.699 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:24 smithi099 ceph-mon[39420]: osdmap e414: 12 total, 12 up, 11 in 2024-02-09T17:39:24.700 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:24.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:24 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3771598195' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-09T17:39:24.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:24 smithi162 ceph-mon[44304]: osdmap e414: 12 total, 12 up, 11 in 2024-02-09T17:39:25.516 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.10 dump_historic_ops 2024-02-09T17:39:25.865 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:39:25.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:25 smithi179 ceph-mon[43975]: 4.6 starting backfill to osd.1 from (0'0,0'0] MAX to 413'14598 2024-02-09T17:39:25.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:25 smithi179 ceph-mon[43975]: 4.9 continuing backfill to osd.0 from (338'4500,413'14523] 4:99c62d52:::benchmark_data_smithi179_103809_object109957:head to 413'14523 2024-02-09T17:39:25.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:25 smithi179 ceph-mon[43975]: 4.9 continuing backfill to osd.4 from (338'4500,413'14523] 4:99c21988:::benchmark_data_smithi179_103809_object119362:head to 413'14523 2024-02-09T17:39:25.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:25 smithi179 ceph-mon[43975]: osdmap e415: 12 total, 12 up, 11 in 2024-02-09T17:39:25.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:25 smithi179 ceph-mon[43975]: pgmap v1095: 37 pgs: 2 peering, 1 active+remapped+backfill_wait, 1 active+clean+remapped, 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 26 active+clean; 2.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 490 op/s; 20481/129999 objects misplaced (15.755%); 83 MiB/s, 1.33k objects/s recovering 2024-02-09T17:39:25.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:25 smithi179 ceph-mon[43975]: osdmap e416: 12 total, 12 up, 11 in 2024-02-09T17:39:25.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:25 smithi162 ceph-mon[44304]: 4.6 starting backfill to osd.1 from (0'0,0'0] MAX to 413'14598 2024-02-09T17:39:25.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:25 smithi162 ceph-mon[44304]: 4.9 continuing backfill to osd.0 from (338'4500,413'14523] 4:99c62d52:::benchmark_data_smithi179_103809_object109957:head to 413'14523 2024-02-09T17:39:25.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:25 smithi162 ceph-mon[44304]: 4.9 continuing backfill to osd.4 from (338'4500,413'14523] 4:99c21988:::benchmark_data_smithi179_103809_object119362:head to 413'14523 2024-02-09T17:39:25.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:25 smithi162 ceph-mon[44304]: osdmap e415: 12 total, 12 up, 11 in 2024-02-09T17:39:25.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:25 smithi162 ceph-mon[44304]: pgmap v1095: 37 pgs: 2 peering, 1 active+remapped+backfill_wait, 1 active+clean+remapped, 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 26 active+clean; 2.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 490 op/s; 20481/129999 objects misplaced (15.755%); 83 MiB/s, 1.33k objects/s recovering 2024-02-09T17:39:25.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:25 smithi162 ceph-mon[44304]: osdmap e416: 12 total, 12 up, 11 in 2024-02-09T17:39:25.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:25 smithi099 ceph-mon[39420]: 4.6 starting backfill to osd.1 from (0'0,0'0] MAX to 413'14598 2024-02-09T17:39:25.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:25 smithi099 ceph-mon[39420]: 4.9 continuing backfill to osd.0 from (338'4500,413'14523] 4:99c62d52:::benchmark_data_smithi179_103809_object109957:head to 413'14523 2024-02-09T17:39:25.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:25 smithi099 ceph-mon[39420]: 4.9 continuing backfill to osd.4 from (338'4500,413'14523] 4:99c21988:::benchmark_data_smithi179_103809_object119362:head to 413'14523 2024-02-09T17:39:25.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:25 smithi099 ceph-mon[39420]: osdmap e415: 12 total, 12 up, 11 in 2024-02-09T17:39:25.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:25 smithi099 ceph-mon[39420]: pgmap v1095: 37 pgs: 2 peering, 1 active+remapped+backfill_wait, 1 active+clean+remapped, 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 26 active+clean; 2.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 490 op/s; 20481/129999 objects misplaced (15.755%); 83 MiB/s, 1.33k objects/s recovering 2024-02-09T17:39:25.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:25 smithi099 ceph-mon[39420]: osdmap e416: 12 total, 12 up, 11 in 2024-02-09T17:39:26.258 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:39:26.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:26.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:39:26.365 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:39:26.445 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:26.609 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:39:26.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:26.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:39:26.720 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:39:26.759 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:26 smithi179 ceph-mon[43975]: 4.7 starting backfill to osd.1 from (0'0,0'0] MAX to 413'14681 2024-02-09T17:39:26.759 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:26 smithi179 ceph-mon[43975]: 4.e starting backfill to osd.1 from (0'0,0'0] MAX to 413'14467 2024-02-09T17:39:26.759 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:26 smithi179 ceph-mon[43975]: 4.4 continuing backfill to osd.7 from (338'4400,413'14417] MIN to 413'14417 2024-02-09T17:39:26.760 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:26 smithi179 ceph-mon[43975]: 4.4 continuing backfill to osd.10 from (338'4400,413'14417] 4:2810fb8b:::benchmark_data_smithi179_103809_object115717:head to 413'14417 2024-02-09T17:39:26.760 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:26 smithi179 ceph-mon[43975]: osdmap e417: 12 total, 12 up, 11 in 2024-02-09T17:39:26.763 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:26.867 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:39:26.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:26.871 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:39:26.885 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:26 smithi099 ceph-mon[39420]: 4.7 starting backfill to osd.1 from (0'0,0'0] MAX to 413'14681 2024-02-09T17:39:26.885 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:26 smithi099 ceph-mon[39420]: 4.e starting backfill to osd.1 from (0'0,0'0] MAX to 413'14467 2024-02-09T17:39:26.885 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:26 smithi099 ceph-mon[39420]: 4.4 continuing backfill to osd.7 from (338'4400,413'14417] MIN to 413'14417 2024-02-09T17:39:26.885 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:26 smithi099 ceph-mon[39420]: 4.4 continuing backfill to osd.10 from (338'4400,413'14417] 4:2810fb8b:::benchmark_data_smithi179_103809_object115717:head to 413'14417 2024-02-09T17:39:26.885 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:26 smithi099 ceph-mon[39420]: osdmap e417: 12 total, 12 up, 11 in 2024-02-09T17:39:26.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:26 smithi162 ceph-mon[44304]: 4.7 starting backfill to osd.1 from (0'0,0'0] MAX to 413'14681 2024-02-09T17:39:26.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:26 smithi162 ceph-mon[44304]: 4.e starting backfill to osd.1 from (0'0,0'0] MAX to 413'14467 2024-02-09T17:39:26.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:26 smithi162 ceph-mon[44304]: 4.4 continuing backfill to osd.7 from (338'4400,413'14417] MIN to 413'14417 2024-02-09T17:39:26.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:26 smithi162 ceph-mon[44304]: 4.4 continuing backfill to osd.10 from (338'4400,413'14417] 4:2810fb8b:::benchmark_data_smithi179_103809_object115717:head to 413'14417 2024-02-09T17:39:26.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:26 smithi162 ceph-mon[44304]: osdmap e417: 12 total, 12 up, 11 in 2024-02-09T17:39:26.972 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:39:27.002 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:27.206 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:39:27.215 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:27.215 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:39:27.316 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:39:27.355 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:27.387 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:39:27.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:27.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:39:27.496 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:39:27.528 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:27.558 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:39:27.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:27.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:39:27.662 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:39:27.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:27 smithi162 ceph-mon[44304]: pgmap v1098: 37 pgs: 1 active+remapped+backfill_toofull, 1 remapped+peering, 2 peering, 1 active+undersized+remapped, 1 active+clean+remapped, 1 activating+remapped, 4 active+remapped+backfilling, 26 active+clean; 2.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 374 op/s; 11835/126990 objects misplaced (9.320%); 46 MiB/s, 741 objects/s recovering 2024-02-09T17:39:27.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:27 smithi162 ceph-mon[44304]: osdmap e418: 12 total, 12 up, 11 in 2024-02-09T17:39:27.704 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:27.731 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:39:27.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:27.740 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:39:27.841 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:39:27.878 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:27.910 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:39:27.915 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:39:27.916 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-02-09T17:39:27.916 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-02-09T17:39:27.916 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-02-09T17:39:27.916 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.916 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.25201.0:207294 4.5 e378/322 4:aad876f2:::benchmark_data_smithi179_103809_object69828:head v 378'12879, mlcod=378'12876)", 2024-02-09T17:39:27.916 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:32.201583+0000", 2024-02-09T17:39:27.916 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 55.709696147999999, 2024-02-09T17:39:27.916 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.080803220999999995, 2024-02-09T17:39:27.916 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:27.916 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:27.916 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:27.916 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.917 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:27.917 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.201583+0000", 2024-02-09T17:39:27.917 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:27.917 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.917 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.917 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:27.917 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.201583+0000", 2024-02-09T17:39:27.917 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:27.917 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.917 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.917 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:27.917 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.201583+0000", 2024-02-09T17:39:27.917 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.3099999999999997e-07 2024-02-09T17:39:27.918 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.918 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.918 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:27.918 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.201589+0000", 2024-02-09T17:39:27.918 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8189999999999997e-06 2024-02-09T17:39:27.918 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.918 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.918 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:27.918 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.268415+0000", 2024-02-09T17:39:27.918 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.066825438000000001 2024-02-09T17:39:27.918 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.918 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.918 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:27.919 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.268432+0000", 2024-02-09T17:39:27.919 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7125e-05 2024-02-09T17:39:27.919 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.919 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.919 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:27.919 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.268612+0000", 2024-02-09T17:39:27.919 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00018039999999999999 2024-02-09T17:39:27.919 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.919 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.919 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:27.919 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.268632+0000", 2024-02-09T17:39:27.919 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0207000000000002e-05 2024-02-09T17:39:27.920 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.920 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.920 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:27.920 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.282337+0000", 2024-02-09T17:39:27.920 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013705175999999999 2024-02-09T17:39:27.920 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.920 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.920 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:27.920 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.282386+0000", 2024-02-09T17:39:27.920 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8325000000000002e-05 2024-02-09T17:39:27.920 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:27.920 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:27.920 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:27.921 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.921 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.921 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:207298 4.7 4:e19504a0:::benchmark_data_smithi179_103809_object69832:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-02-09T17:39:27.921 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:32.206380+0000", 2024-02-09T17:39:27.921 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 55.704898792000002, 2024-02-09T17:39:27.921 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.07781855, 2024-02-09T17:39:27.921 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:27.921 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:27.921 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:27.921 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:27.921 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:27.921 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 207298 2024-02-09T17:39:27.921 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.922 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:27.922 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.922 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:27.922 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.206380+0000", 2024-02-09T17:39:27.922 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:27.922 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.922 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.922 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:27.922 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.206380+0000", 2024-02-09T17:39:27.922 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:27.922 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.922 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.922 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:27.922 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.206381+0000", 2024-02-09T17:39:27.922 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5009999999999999e-06 2024-02-09T17:39:27.923 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.923 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.923 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:27.923 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.206386+0000", 2024-02-09T17:39:27.923 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6439999999999999e-06 2024-02-09T17:39:27.923 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.923 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.923 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:27.923 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.268875+0000", 2024-02-09T17:39:27.923 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.062488587999999998 2024-02-09T17:39:27.923 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.923 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.923 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:27.923 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.268887+0000", 2024-02-09T17:39:27.924 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2673999999999999e-05 2024-02-09T17:39:27.924 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.924 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.924 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:27.924 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.269731+0000", 2024-02-09T17:39:27.924 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00084322399999999995 2024-02-09T17:39:27.924 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.924 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.924 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:27.924 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.269845+0000", 2024-02-09T17:39:27.924 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011463699999999999 2024-02-09T17:39:27.924 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.924 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.924 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,1,7", 2024-02-09T17:39:27.924 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.269969+0000", 2024-02-09T17:39:27.925 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000123437 2024-02-09T17:39:27.925 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.925 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.925 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:27.925 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.283747+0000", 2024-02-09T17:39:27.925 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013778095000000001 2024-02-09T17:39:27.925 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.925 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.925 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:27.925 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.283801+0000", 2024-02-09T17:39:27.925 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4107000000000002e-05 2024-02-09T17:39:27.925 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.925 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.925 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:27.926 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.283857+0000", 2024-02-09T17:39:27.926 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6483000000000001e-05 2024-02-09T17:39:27.926 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.926 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.926 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:27.926 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.284008+0000", 2024-02-09T17:39:27.926 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000150385 2024-02-09T17:39:27.926 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.926 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.926 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:27.926 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.284063+0000", 2024-02-09T17:39:27.926 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5050999999999999e-05 2024-02-09T17:39:27.926 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.926 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.926 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:27.926 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.284198+0000", 2024-02-09T17:39:27.927 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013572399999999999 2024-02-09T17:39:27.927 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:27.927 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:27.927 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:27.927 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.927 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.927 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:207589 4.7 4:e905c4e5:::benchmark_data_smithi179_103809_object70123:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-02-09T17:39:27.927 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:32.748961+0000", 2024-02-09T17:39:27.927 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 55.162317737000002, 2024-02-09T17:39:27.927 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.216048499, 2024-02-09T17:39:27.927 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:27.927 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:27.927 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:27.927 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:27.927 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:27.928 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 207589 2024-02-09T17:39:27.928 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.928 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:27.928 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.928 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:27.928 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.748961+0000", 2024-02-09T17:39:27.928 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:27.928 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.928 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.928 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:27.928 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.748961+0000", 2024-02-09T17:39:27.928 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:27.928 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.929 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.929 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:27.929 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.748962+0000", 2024-02-09T17:39:27.929 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.4600000000000003e-07 2024-02-09T17:39:27.929 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.929 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.929 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:27.929 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.748966+0000", 2024-02-09T17:39:27.929 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2130000000000002e-06 2024-02-09T17:39:27.929 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.929 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.929 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:27.929 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.748976+0000", 2024-02-09T17:39:27.930 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5740000000000002e-06 2024-02-09T17:39:27.930 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.930 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.930 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:27.930 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.748983+0000", 2024-02-09T17:39:27.930 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4499999999999998e-06 2024-02-09T17:39:27.930 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.930 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.930 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:27.930 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.753831+0000", 2024-02-09T17:39:27.930 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0048474240000000004 2024-02-09T17:39:27.930 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.931 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.931 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:27.931 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.753976+0000", 2024-02-09T17:39:27.931 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014569600000000001 2024-02-09T17:39:27.931 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.931 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.931 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,1,7", 2024-02-09T17:39:27.931 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.754124+0000", 2024-02-09T17:39:27.931 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000148104 2024-02-09T17:39:27.931 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.931 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.931 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:27.931 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.763802+0000", 2024-02-09T17:39:27.931 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.009677527 2024-02-09T17:39:27.932 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.932 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.932 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:27.932 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.763872+0000", 2024-02-09T17:39:27.932 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9945999999999997e-05 2024-02-09T17:39:27.932 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.932 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.932 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:27.932 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.783491+0000", 2024-02-09T17:39:27.932 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.019619533000000001 2024-02-09T17:39:27.932 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.933 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.933 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:27.933 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.964888+0000", 2024-02-09T17:39:27.933 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.18139617699999999 2024-02-09T17:39:27.933 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.933 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.933 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:27.933 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.964933+0000", 2024-02-09T17:39:27.933 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5852000000000001e-05 2024-02-09T17:39:27.933 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.933 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.933 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:27.933 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.965009+0000", 2024-02-09T17:39:27.934 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6056999999999994e-05 2024-02-09T17:39:27.934 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:27.934 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:27.934 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:27.934 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.934 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.934 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:207592 4.7 4:e615ae58:::benchmark_data_smithi179_103809_object70126:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-02-09T17:39:27.934 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:32.749662+0000", 2024-02-09T17:39:27.934 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 55.161616680000002, 2024-02-09T17:39:27.934 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.22551052399999999, 2024-02-09T17:39:27.934 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:27.934 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:27.935 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:27.935 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:27.935 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:27.935 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 207592 2024-02-09T17:39:27.935 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.935 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:27.935 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.935 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:27.935 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.749662+0000", 2024-02-09T17:39:27.935 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:27.935 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.935 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.935 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:27.936 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.749662+0000", 2024-02-09T17:39:27.936 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:27.936 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.936 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.936 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:27.936 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.749663+0000", 2024-02-09T17:39:27.936 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7900000000000007e-07 2024-02-09T17:39:27.936 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.936 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.936 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:27.936 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.749668+0000", 2024-02-09T17:39:27.937 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0159999999999999e-06 2024-02-09T17:39:27.937 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.937 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.937 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:27.937 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.749685+0000", 2024-02-09T17:39:27.937 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6803000000000002e-05 2024-02-09T17:39:27.937 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.937 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.937 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:27.937 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.749702+0000", 2024-02-09T17:39:27.937 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7421000000000001e-05 2024-02-09T17:39:27.937 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.938 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.938 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:27.938 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.758677+0000", 2024-02-09T17:39:27.938 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0089749089999999997 2024-02-09T17:39:27.938 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.938 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.938 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:27.938 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.758809+0000", 2024-02-09T17:39:27.938 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013181900000000001 2024-02-09T17:39:27.938 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.938 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.939 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,1,7", 2024-02-09T17:39:27.939 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.758936+0000", 2024-02-09T17:39:27.939 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012730900000000001 2024-02-09T17:39:27.939 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.939 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.939 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:27.939 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.773609+0000", 2024-02-09T17:39:27.939 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014672810999999999 2024-02-09T17:39:27.939 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.939 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.939 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:27.940 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.773633+0000", 2024-02-09T17:39:27.940 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4340000000000001e-05 2024-02-09T17:39:27.940 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.940 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.940 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:27.940 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.783535+0000", 2024-02-09T17:39:27.940 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0099020099999999993 2024-02-09T17:39:27.940 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.940 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.940 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:27.941 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.975037+0000", 2024-02-09T17:39:27.941 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.19150186399999999 2024-02-09T17:39:27.941 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.941 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.941 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:27.941 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.975080+0000", 2024-02-09T17:39:27.941 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2839000000000001e-05 2024-02-09T17:39:27.941 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.941 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.941 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:27.941 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.975173+0000", 2024-02-09T17:39:27.942 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2404000000000005e-05 2024-02-09T17:39:27.942 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:27.942 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:27.942 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:27.942 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.942 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.942 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:207599 4.7 4:e2960c10:::benchmark_data_smithi179_103809_object70133:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-02-09T17:39:27.942 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:32.754607+0000", 2024-02-09T17:39:27.942 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 55.156671529999997, 2024-02-09T17:39:27.942 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.23920739299999999, 2024-02-09T17:39:27.942 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:27.942 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:27.943 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:27.943 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:27.943 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:27.943 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 207599 2024-02-09T17:39:27.943 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.943 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:27.943 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.943 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:27.943 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.754607+0000", 2024-02-09T17:39:27.943 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:27.943 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.943 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.944 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:27.944 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.754607+0000", 2024-02-09T17:39:27.944 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:27.944 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.944 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.944 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:27.944 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.754608+0000", 2024-02-09T17:39:27.944 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1119999999999999e-06 2024-02-09T17:39:27.944 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.944 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.944 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:27.944 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.754613+0000", 2024-02-09T17:39:27.945 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3980000000000001e-06 2024-02-09T17:39:27.945 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.945 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.945 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:27.945 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.754624+0000", 2024-02-09T17:39:27.945 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1541e-05 2024-02-09T17:39:27.945 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.945 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.945 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:27.945 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.754632+0000", 2024-02-09T17:39:27.945 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6129999999999998e-06 2024-02-09T17:39:27.945 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.946 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.946 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:27.946 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.763892+0000", 2024-02-09T17:39:27.946 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0092598009999999998 2024-02-09T17:39:27.946 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.946 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.946 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:27.946 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.764014+0000", 2024-02-09T17:39:27.946 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012283300000000001 2024-02-09T17:39:27.946 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.946 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.947 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,1,7", 2024-02-09T17:39:27.947 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.764148+0000", 2024-02-09T17:39:27.947 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013364299999999999 2024-02-09T17:39:27.947 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.947 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.947 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:27.947 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.773862+0000", 2024-02-09T17:39:27.947 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0097142159999999995 2024-02-09T17:39:27.947 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.947 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.947 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:27.948 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.787249+0000", 2024-02-09T17:39:27.948 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013387125 2024-02-09T17:39:27.948 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.948 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.948 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:27.948 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.787619+0000", 2024-02-09T17:39:27.948 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00036931799999999999 2024-02-09T17:39:27.948 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.948 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.948 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:27.948 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.993634+0000", 2024-02-09T17:39:27.948 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.20601524800000001 2024-02-09T17:39:27.948 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.948 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.949 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:27.949 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.993696+0000", 2024-02-09T17:39:27.949 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2101999999999996e-05 2024-02-09T17:39:27.949 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.949 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.949 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:27.949 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.993815+0000", 2024-02-09T17:39:27.949 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000118443 2024-02-09T17:39:27.949 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:27.949 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:27.949 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:27.950 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.950 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.950 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:207601 4.7 4:ea9472b7:::benchmark_data_smithi179_103809_object70135:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-02-09T17:39:27.950 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:32.758215+0000", 2024-02-09T17:39:27.950 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 55.153063983000003, 2024-02-09T17:39:27.950 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.23582645499999999, 2024-02-09T17:39:27.950 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:27.950 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:27.950 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:27.950 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:27.950 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:27.950 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 207601 2024-02-09T17:39:27.950 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.950 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:27.951 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.951 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:27.951 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.758215+0000", 2024-02-09T17:39:27.951 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:27.951 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.951 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.951 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:27.951 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.758215+0000", 2024-02-09T17:39:27.951 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:27.951 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.951 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.951 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:27.951 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.758216+0000", 2024-02-09T17:39:27.951 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2720000000000001e-06 2024-02-09T17:39:27.952 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.952 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.952 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:27.952 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.758221+0000", 2024-02-09T17:39:27.952 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.921e-06 2024-02-09T17:39:27.952 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.952 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.952 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:27.952 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.758233+0000", 2024-02-09T17:39:27.952 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1987999999999999e-05 2024-02-09T17:39:27.952 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.952 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.952 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:27.953 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.758241+0000", 2024-02-09T17:39:27.953 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7190000000000001e-06 2024-02-09T17:39:27.953 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.953 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.953 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:27.953 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.768835+0000", 2024-02-09T17:39:27.953 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010594236 2024-02-09T17:39:27.953 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.953 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.953 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:27.953 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.768976+0000", 2024-02-09T17:39:27.953 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000141125 2024-02-09T17:39:27.953 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.953 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.954 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,1,7", 2024-02-09T17:39:27.954 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.769113+0000", 2024-02-09T17:39:27.954 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013665699999999999 2024-02-09T17:39:27.954 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.954 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.954 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:27.954 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.783190+0000", 2024-02-09T17:39:27.954 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014077838000000001 2024-02-09T17:39:27.954 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.954 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.954 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:27.954 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.787290+0000", 2024-02-09T17:39:27.955 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0040999169999999998 2024-02-09T17:39:27.955 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.955 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.955 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:27.955 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.800012+0000", 2024-02-09T17:39:27.955 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.012721193 2024-02-09T17:39:27.955 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.955 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.955 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:27.955 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.993864+0000", 2024-02-09T17:39:27.955 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.19385191299999999 2024-02-09T17:39:27.955 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.955 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.955 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:27.956 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.993924+0000", 2024-02-09T17:39:27.956 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0930000000000001e-05 2024-02-09T17:39:27.956 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.956 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.956 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:27.956 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.994041+0000", 2024-02-09T17:39:27.956 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000116746 2024-02-09T17:39:27.956 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:27.956 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:27.956 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:27.956 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.956 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.956 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:207604 4.7 4:ef4a4ef9:::benchmark_data_smithi179_103809_object70138:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-02-09T17:39:27.956 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:32.764002+0000", 2024-02-09T17:39:27.957 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 55.147277158999998, 2024-02-09T17:39:27.957 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.23024998399999999, 2024-02-09T17:39:27.957 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:27.957 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:27.957 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:27.957 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:27.957 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:27.957 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 207604 2024-02-09T17:39:27.957 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.957 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:27.957 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.957 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:27.958 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.764002+0000", 2024-02-09T17:39:27.958 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:27.958 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.958 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.958 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:27.958 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.764002+0000", 2024-02-09T17:39:27.958 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:27.958 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.958 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.958 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:27.958 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.764002+0000", 2024-02-09T17:39:27.958 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5899999999999995e-07 2024-02-09T17:39:27.958 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.959 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.959 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:27.959 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.764005+0000", 2024-02-09T17:39:27.959 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4439999999999998e-06 2024-02-09T17:39:27.959 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.959 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.959 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:27.959 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.764017+0000", 2024-02-09T17:39:27.959 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2299e-05 2024-02-09T17:39:27.959 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.959 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.959 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:27.959 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.764022+0000", 2024-02-09T17:39:27.960 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6349999999999997e-06 2024-02-09T17:39:27.960 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.960 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.960 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:27.960 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.773909+0000", 2024-02-09T17:39:27.960 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0098870529999999998 2024-02-09T17:39:27.960 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.960 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.960 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:27.960 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.774054+0000", 2024-02-09T17:39:27.960 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000145317 2024-02-09T17:39:27.960 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.961 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.961 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,1,7", 2024-02-09T17:39:27.961 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.774215+0000", 2024-02-09T17:39:27.961 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016044900000000001 2024-02-09T17:39:27.961 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.961 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.961 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:27.961 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.783200+0000", 2024-02-09T17:39:27.961 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0089857329999999992 2024-02-09T17:39:27.961 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.961 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.961 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:27.962 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.787330+0000", 2024-02-09T17:39:27.962 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0041292100000000003 2024-02-09T17:39:27.962 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.962 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.962 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:27.962 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.806313+0000", 2024-02-09T17:39:27.962 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.018983659999999999 2024-02-09T17:39:27.962 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.962 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.962 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:27.962 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.994082+0000", 2024-02-09T17:39:27.963 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.18776861 2024-02-09T17:39:27.963 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.963 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.963 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:27.963 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.994143+0000", 2024-02-09T17:39:27.963 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0942e-05 2024-02-09T17:39:27.963 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.963 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.963 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:27.963 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.994252+0000", 2024-02-09T17:39:27.963 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000108773 2024-02-09T17:39:27.963 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:27.964 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:27.964 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:27.964 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.964 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.964 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:207614 4.7 4:e8e480d8:::benchmark_data_smithi179_103809_object70148:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-02-09T17:39:27.964 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:32.778084+0000", 2024-02-09T17:39:27.964 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 55.133194918000001, 2024-02-09T17:39:27.964 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.21637295300000001, 2024-02-09T17:39:27.964 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:27.964 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:27.965 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:27.965 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:27.965 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:27.965 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 207614 2024-02-09T17:39:27.965 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.965 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:27.965 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.965 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:27.965 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.778084+0000", 2024-02-09T17:39:27.965 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:27.965 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.965 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.966 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:27.966 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.778084+0000", 2024-02-09T17:39:27.966 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:27.966 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.966 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.966 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:27.966 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.778085+0000", 2024-02-09T17:39:27.966 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5230000000000001e-06 2024-02-09T17:39:27.966 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.966 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.966 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:27.966 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.778090+0000", 2024-02-09T17:39:27.967 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8999999999999997e-06 2024-02-09T17:39:27.967 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.967 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.967 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:27.967 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.778102+0000", 2024-02-09T17:39:27.967 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2276000000000001e-05 2024-02-09T17:39:27.967 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.967 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.967 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:27.967 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.778112+0000", 2024-02-09T17:39:27.967 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.1560000000000001e-06 2024-02-09T17:39:27.967 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.968 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.968 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:27.968 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.778551+0000", 2024-02-09T17:39:27.968 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00043917899999999999 2024-02-09T17:39:27.968 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.968 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.968 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:27.968 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.778680+0000", 2024-02-09T17:39:27.968 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012922000000000001 2024-02-09T17:39:27.968 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.968 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.971 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,1,7", 2024-02-09T17:39:27.971 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.778786+0000", 2024-02-09T17:39:27.971 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000106053 2024-02-09T17:39:27.971 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.972 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.972 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:27.972 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.783509+0000", 2024-02-09T17:39:27.972 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0047230220000000003 2024-02-09T17:39:27.972 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.972 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.972 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:27.972 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.787358+0000", 2024-02-09T17:39:27.972 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0038488490000000001 2024-02-09T17:39:27.972 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.972 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.973 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:27.973 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.806355+0000", 2024-02-09T17:39:27.973 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.018997284 2024-02-09T17:39:27.973 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.973 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.973 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:27.973 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.994292+0000", 2024-02-09T17:39:27.973 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.187936361 2024-02-09T17:39:27.973 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.973 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.973 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:27.973 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.994346+0000", 2024-02-09T17:39:27.973 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4518000000000002e-05 2024-02-09T17:39:27.973 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.973 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.974 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:27.974 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.994457+0000", 2024-02-09T17:39:27.974 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000110612 2024-02-09T17:39:27.974 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:27.974 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:27.974 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:27.974 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.974 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.974 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:207627 4.7 4:e1284f13:::benchmark_data_smithi179_103809_object70161:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-02-09T17:39:27.974 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:32.800287+0000", 2024-02-09T17:39:27.974 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 55.110992103000001, 2024-02-09T17:39:27.974 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.194366803, 2024-02-09T17:39:27.974 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:27.974 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:27.975 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:27.975 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:27.975 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:27.975 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 207627 2024-02-09T17:39:27.975 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.975 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:27.975 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.975 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:27.975 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.800287+0000", 2024-02-09T17:39:27.975 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:27.975 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.975 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.975 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:27.975 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.800287+0000", 2024-02-09T17:39:27.975 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:27.976 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.976 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.976 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:27.976 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.800288+0000", 2024-02-09T17:39:27.976 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.669e-06 2024-02-09T17:39:27.976 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.976 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.976 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:27.976 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.800293+0000", 2024-02-09T17:39:27.976 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1950000000000002e-06 2024-02-09T17:39:27.976 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.976 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.976 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:27.976 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.800304+0000", 2024-02-09T17:39:27.976 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0545e-05 2024-02-09T17:39:27.977 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.977 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.977 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:27.977 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.800312+0000", 2024-02-09T17:39:27.977 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8080000000000005e-06 2024-02-09T17:39:27.977 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.977 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.977 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:27.977 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.800371+0000", 2024-02-09T17:39:27.977 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9151000000000003e-05 2024-02-09T17:39:27.977 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.977 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.978 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:27.978 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.800473+0000", 2024-02-09T17:39:27.978 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000102465 2024-02-09T17:39:27.978 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.978 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.978 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,1,7", 2024-02-09T17:39:27.978 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.800564+0000", 2024-02-09T17:39:27.978 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.0450000000000003e-05 2024-02-09T17:39:27.978 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.978 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.978 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:27.978 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.804199+0000", 2024-02-09T17:39:27.978 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.003635283 2024-02-09T17:39:27.978 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.979 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.979 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:27.979 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.813550+0000", 2024-02-09T17:39:27.979 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0093505749999999999 2024-02-09T17:39:27.979 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.979 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.979 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:27.979 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.817741+0000", 2024-02-09T17:39:27.979 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0041907740000000004 2024-02-09T17:39:27.979 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.979 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.979 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:27.979 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.994498+0000", 2024-02-09T17:39:27.979 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.17675744199999999 2024-02-09T17:39:27.980 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.980 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.980 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:27.980 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.994551+0000", 2024-02-09T17:39:27.980 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3470999999999997e-05 2024-02-09T17:39:27.980 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.980 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.980 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:27.980 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.994653+0000", 2024-02-09T17:39:27.980 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000101975 2024-02-09T17:39:27.980 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:27.980 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:27.980 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:27.980 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.981 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.981 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:207664 4.7 4:e9bc5ca9:::benchmark_data_smithi179_103809_object70198:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-02-09T17:39:27.981 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:32.854556+0000", 2024-02-09T17:39:27.981 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 55.056722264999998, 2024-02-09T17:39:27.981 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.140295527, 2024-02-09T17:39:27.981 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:27.981 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:27.981 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:27.981 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:27.981 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:27.981 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 207664 2024-02-09T17:39:27.981 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.981 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:27.981 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.982 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:27.982 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.854556+0000", 2024-02-09T17:39:27.982 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:27.982 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.982 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.982 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:27.982 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.854556+0000", 2024-02-09T17:39:27.982 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:27.982 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.982 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.982 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:27.982 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.854558+0000", 2024-02-09T17:39:27.982 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.46e-06 2024-02-09T17:39:27.982 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.982 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.983 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:27.983 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.854564+0000", 2024-02-09T17:39:27.983 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8300000000000001e-06 2024-02-09T17:39:27.983 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.983 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.983 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:27.983 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.854576+0000", 2024-02-09T17:39:27.983 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2381000000000001e-05 2024-02-09T17:39:27.983 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.983 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.983 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:27.983 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.854585+0000", 2024-02-09T17:39:27.983 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7339999999999997e-06 2024-02-09T17:39:27.983 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.984 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.984 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:27.984 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.854653+0000", 2024-02-09T17:39:27.984 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8150000000000003e-05 2024-02-09T17:39:27.984 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.984 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.984 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:27.984 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.854778+0000", 2024-02-09T17:39:27.984 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012533999999999999 2024-02-09T17:39:27.984 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.984 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.984 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,1,7", 2024-02-09T17:39:27.984 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.854872+0000", 2024-02-09T17:39:27.984 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.4075999999999998e-05 2024-02-09T17:39:27.985 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.985 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.985 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:27.985 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.862265+0000", 2024-02-09T17:39:27.985 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0073921489999999998 2024-02-09T17:39:27.985 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.985 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.985 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:27.985 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.865744+0000", 2024-02-09T17:39:27.985 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0034792600000000001 2024-02-09T17:39:27.985 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.985 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.985 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:27.985 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.866870+0000", 2024-02-09T17:39:27.985 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0011265450000000001 2024-02-09T17:39:27.986 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.986 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.986 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:27.986 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.994692+0000", 2024-02-09T17:39:27.986 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.12782187 2024-02-09T17:39:27.986 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.986 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.986 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:27.986 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.994744+0000", 2024-02-09T17:39:27.986 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1499000000000003e-05 2024-02-09T17:39:27.986 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.986 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.986 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:27.986 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.994852+0000", 2024-02-09T17:39:27.987 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010823300000000001 2024-02-09T17:39:27.987 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:27.987 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:27.987 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:27.987 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.987 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.987 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:207665 4.7 4:ea24a272:::benchmark_data_smithi179_103809_object70199:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-02-09T17:39:27.987 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:32.857672+0000", 2024-02-09T17:39:27.987 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 55.053606342999998, 2024-02-09T17:39:27.987 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.13739986400000001, 2024-02-09T17:39:27.987 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:27.987 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:27.987 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:27.987 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:27.988 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:27.988 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 207665 2024-02-09T17:39:27.988 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.988 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:27.988 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.988 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:27.988 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.857672+0000", 2024-02-09T17:39:27.988 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:27.988 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.988 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.988 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:27.988 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.857672+0000", 2024-02-09T17:39:27.988 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:27.988 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.988 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.989 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:27.989 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.857674+0000", 2024-02-09T17:39:27.989 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.429e-06 2024-02-09T17:39:27.989 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.989 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.989 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:27.989 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.857680+0000", 2024-02-09T17:39:27.989 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0260000000000004e-06 2024-02-09T17:39:27.989 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.989 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.989 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:27.989 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.857694+0000", 2024-02-09T17:39:27.989 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3935e-05 2024-02-09T17:39:27.989 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.990 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.990 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:27.990 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.857703+0000", 2024-02-09T17:39:27.990 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8349999999999993e-06 2024-02-09T17:39:27.990 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.990 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.990 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:27.990 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.858574+0000", 2024-02-09T17:39:27.990 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00087124800000000003 2024-02-09T17:39:27.990 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.990 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.990 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:27.990 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.858671+0000", 2024-02-09T17:39:27.990 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7087000000000001e-05 2024-02-09T17:39:27.991 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.991 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.991 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,1,7", 2024-02-09T17:39:27.991 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.858750+0000", 2024-02-09T17:39:27.991 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8802999999999998e-05 2024-02-09T17:39:27.991 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.991 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.991 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:27.991 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.865766+0000", 2024-02-09T17:39:27.991 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0070160489999999999 2024-02-09T17:39:27.991 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.991 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.991 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:27.991 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.865790+0000", 2024-02-09T17:39:27.992 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4414999999999999e-05 2024-02-09T17:39:27.992 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.992 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.992 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:27.992 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.866908+0000", 2024-02-09T17:39:27.992 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0011182259999999999 2024-02-09T17:39:27.992 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.992 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.992 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:27.992 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.994902+0000", 2024-02-09T17:39:27.992 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.12799374499999999 2024-02-09T17:39:27.992 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.992 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.993 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:27.993 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.994963+0000", 2024-02-09T17:39:27.993 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0674999999999998e-05 2024-02-09T17:39:27.993 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.993 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.993 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:27.993 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.995072+0000", 2024-02-09T17:39:27.993 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000109391 2024-02-09T17:39:27.993 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:27.993 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:27.993 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:27.993 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.993 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.994 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:207669 4.7 4:ee9f4aa6:::benchmark_data_smithi179_103809_object70203:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-02-09T17:39:27.994 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:32.859678+0000", 2024-02-09T17:39:27.994 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 55.051600317999998, 2024-02-09T17:39:27.994 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.13561305100000001, 2024-02-09T17:39:27.994 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:27.994 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:27.994 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:27.994 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:27.994 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:27.994 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 207669 2024-02-09T17:39:27.994 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.995 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:27.995 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.995 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:27.995 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.859678+0000", 2024-02-09T17:39:27.995 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:27.995 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.995 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.995 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:27.995 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.859678+0000", 2024-02-09T17:39:27.995 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:27.995 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.995 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.995 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:27.996 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.859679+0000", 2024-02-09T17:39:27.996 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.004e-06 2024-02-09T17:39:27.996 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.996 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.996 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:27.996 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.859684+0000", 2024-02-09T17:39:27.996 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.634e-06 2024-02-09T17:39:27.996 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.996 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.996 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:27.996 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.859696+0000", 2024-02-09T17:39:27.997 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2166e-05 2024-02-09T17:39:27.997 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.997 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.997 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:27.997 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.859704+0000", 2024-02-09T17:39:27.997 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2919999999999996e-06 2024-02-09T17:39:27.997 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.997 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.997 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:27.997 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.862300+0000", 2024-02-09T17:39:27.997 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002595694 2024-02-09T17:39:27.998 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.998 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.998 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:27.998 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.862409+0000", 2024-02-09T17:39:27.998 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010896 2024-02-09T17:39:27.998 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.998 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.998 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,1,7", 2024-02-09T17:39:27.998 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.862499+0000", 2024-02-09T17:39:27.998 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.9881000000000002e-05 2024-02-09T17:39:27.998 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.998 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.999 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:27.999 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.865880+0000", 2024-02-09T17:39:27.999 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0033812579999999998 2024-02-09T17:39:27.999 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.999 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.999 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:27.999 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.870770+0000", 2024-02-09T17:39:27.999 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0048899540000000002 2024-02-09T17:39:27.999 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:27.999 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:27.999 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:28.000 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.873234+0000", 2024-02-09T17:39:28.000 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0024635360000000001 2024-02-09T17:39:28.000 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.000 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.000 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:28.000 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.995119+0000", 2024-02-09T17:39:28.000 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.12188568299999999 2024-02-09T17:39:28.000 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.000 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.000 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:28.000 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.995182+0000", 2024-02-09T17:39:28.000 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2805999999999995e-05 2024-02-09T17:39:28.001 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.001 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.001 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:28.001 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.995291+0000", 2024-02-09T17:39:28.001 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000109183 2024-02-09T17:39:28.001 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:28.001 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:28.001 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:28.001 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.001 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.001 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:207704 4.7 4:e60f6a61:::benchmark_data_smithi179_103809_object70238:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-02-09T17:39:28.002 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:32.919175+0000", 2024-02-09T17:39:28.002 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 54.992103497000002, 2024-02-09T17:39:28.002 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.078584807000000007, 2024-02-09T17:39:28.002 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:28.002 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:28.002 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:28.002 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:28.002 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:28.002 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 207704 2024-02-09T17:39:28.002 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.002 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:28.002 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.002 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:28.003 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.919175+0000", 2024-02-09T17:39:28.003 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:28.003 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.003 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.003 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:28.003 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.919175+0000", 2024-02-09T17:39:28.003 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:28.003 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.003 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.003 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:28.003 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.919177+0000", 2024-02-09T17:39:28.004 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.815e-06 2024-02-09T17:39:28.004 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.004 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.004 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:28.004 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.919183+0000", 2024-02-09T17:39:28.004 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8220000000000003e-06 2024-02-09T17:39:28.004 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.004 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.004 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:28.004 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.919198+0000", 2024-02-09T17:39:28.004 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5443000000000001e-05 2024-02-09T17:39:28.005 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.005 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.005 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:28.005 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.919208+0000", 2024-02-09T17:39:28.005 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.8390000000000002e-06 2024-02-09T17:39:28.005 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.005 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.005 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:28.005 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.919265+0000", 2024-02-09T17:39:28.005 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7309999999999998e-05 2024-02-09T17:39:28.005 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.006 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.006 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:28.006 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.919374+0000", 2024-02-09T17:39:28.006 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000108989 2024-02-09T17:39:28.006 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.006 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.006 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,1,7", 2024-02-09T17:39:28.006 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.919505+0000", 2024-02-09T17:39:28.006 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013053699999999999 2024-02-09T17:39:28.006 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.006 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.006 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:28.007 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.923310+0000", 2024-02-09T17:39:28.007 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0038046899999999999 2024-02-09T17:39:28.007 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.007 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.007 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:28.007 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.923995+0000", 2024-02-09T17:39:28.007 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00068561700000000002 2024-02-09T17:39:28.007 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.007 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.007 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:28.007 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.929777+0000", 2024-02-09T17:39:28.007 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0057818000000000001 2024-02-09T17:39:28.007 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.007 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.008 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:28.008 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.997570+0000", 2024-02-09T17:39:28.008 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.067793176999999996 2024-02-09T17:39:28.008 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.008 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.008 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:28.008 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.997640+0000", 2024-02-09T17:39:28.008 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9579000000000006e-05 2024-02-09T17:39:28.008 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.008 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.009 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:28.009 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.997760+0000", 2024-02-09T17:39:28.009 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000120189 2024-02-09T17:39:28.009 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:28.009 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:28.009 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:28.009 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.009 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.009 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:208399 4.5 4:a2a3b00a:::benchmark_data_smithi179_103809_object70933:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e379)", 2024-02-09T17:39:28.009 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:33.814732+0000", 2024-02-09T17:39:28.009 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 54.096546584000002, 2024-02-09T17:39:28.009 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2041751440000001, 2024-02-09T17:39:28.010 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:28.010 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-02-09T17:39:28.010 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:28.010 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:28.010 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:28.010 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 208399 2024-02-09T17:39:28.010 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.011 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:28.011 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.011 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:28.011 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.814732+0000", 2024-02-09T17:39:28.011 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:28.011 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.011 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.011 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:28.011 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.814732+0000", 2024-02-09T17:39:28.011 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:28.011 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.012 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.012 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:28.012 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.814734+0000", 2024-02-09T17:39:28.012 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4720000000000001e-06 2024-02-09T17:39:28.012 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.012 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.012 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:28.012 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.814739+0000", 2024-02-09T17:39:28.012 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3929999999999999e-06 2024-02-09T17:39:28.012 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.012 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.012 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:28.013 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.814755+0000", 2024-02-09T17:39:28.013 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5719e-05 2024-02-09T17:39:28.013 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.013 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.013 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:28.013 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.814764+0000", 2024-02-09T17:39:28.013 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.3060000000000003e-06 2024-02-09T17:39:28.013 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.013 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.013 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:28.013 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.814834+0000", 2024-02-09T17:39:28.013 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9986000000000004e-05 2024-02-09T17:39:28.013 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.013 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.014 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op must wait for map", 2024-02-09T17:39:28.014 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.814841+0000", 2024-02-09T17:39:28.014 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7259999999999996e-06 2024-02-09T17:39:28.014 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.014 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.014 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:28.014 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.819052+0000", 2024-02-09T17:39:28.014 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0042111850000000001 2024-02-09T17:39:28.014 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.014 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.014 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:39:28.014 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.819059+0000", 2024-02-09T17:39:28.015 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.3839999999999999e-06 2024-02-09T17:39:28.015 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.015 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.015 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:28.015 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:34.811056+0000", 2024-02-09T17:39:28.015 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.99199702899999997 2024-02-09T17:39:28.015 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.015 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.015 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:39:28.015 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:34.811139+0000", 2024-02-09T17:39:28.015 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2931999999999998e-05 2024-02-09T17:39:28.015 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.015 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.015 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:28.016 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.015456+0000", 2024-02-09T17:39:28.016 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.204317029 2024-02-09T17:39:28.016 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.016 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.016 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:28.016 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.018907+0000", 2024-02-09T17:39:28.016 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0034509829999999999 2024-02-09T17:39:28.016 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:28.016 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:28.016 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:28.016 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.016 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.017 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:208418 4.5 4:a02ae9eb:::benchmark_data_smithi179_103809_object70952:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-02-09T17:39:28.017 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:33.835944+0000", 2024-02-09T17:39:28.017 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 54.075334666000003, 2024-02-09T17:39:28.017 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.197308585, 2024-02-09T17:39:28.017 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:28.017 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:28.017 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:28.017 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:28.017 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:28.017 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 208418 2024-02-09T17:39:28.017 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.017 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:28.017 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.018 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:28.018 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.835944+0000", 2024-02-09T17:39:28.018 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:28.018 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.018 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.018 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:28.018 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.835944+0000", 2024-02-09T17:39:28.018 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:28.018 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.018 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.018 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:28.018 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.835945+0000", 2024-02-09T17:39:28.018 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1650000000000001e-06 2024-02-09T17:39:28.019 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.019 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.019 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:28.019 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.835949+0000", 2024-02-09T17:39:28.019 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4410000000000002e-06 2024-02-09T17:39:28.019 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.019 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.019 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:28.019 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.835958+0000", 2024-02-09T17:39:28.019 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2380000000000003e-06 2024-02-09T17:39:28.019 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.019 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.019 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:28.020 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.835963+0000", 2024-02-09T17:39:28.020 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0350000000000001e-06 2024-02-09T17:39:28.020 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.020 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.020 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:28.020 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.836006+0000", 2024-02-09T17:39:28.020 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3275000000000002e-05 2024-02-09T17:39:28.020 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.020 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.020 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:39:28.020 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.836014+0000", 2024-02-09T17:39:28.020 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1580000000000005e-06 2024-02-09T17:39:28.020 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.020 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.020 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:28.021 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:34.811212+0000", 2024-02-09T17:39:28.021 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.97519715799999995 2024-02-09T17:39:28.021 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.021 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.021 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:39:28.021 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:34.811222+0000", 2024-02-09T17:39:28.021 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0030000000000001e-05 2024-02-09T17:39:28.021 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.021 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.021 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:28.021 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.019231+0000", 2024-02-09T17:39:28.021 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2080094990000001 2024-02-09T17:39:28.021 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.022 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.022 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:28.022 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.020160+0000", 2024-02-09T17:39:28.022 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00092906900000000003 2024-02-09T17:39:28.022 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.022 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.022 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,6,7,11", 2024-02-09T17:39:28.022 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.020254+0000", 2024-02-09T17:39:28.022 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.3797999999999999e-05 2024-02-09T17:39:28.022 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.022 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.022 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:28.022 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.024546+0000", 2024-02-09T17:39:28.023 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0042921720000000003 2024-02-09T17:39:28.023 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.023 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.023 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:28.023 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.026292+0000", 2024-02-09T17:39:28.023 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0017459909999999999 2024-02-09T17:39:28.023 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.023 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.023 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:28.023 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.027748+0000", 2024-02-09T17:39:28.023 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001456176 2024-02-09T17:39:28.023 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.024 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.024 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:28.024 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.029364+0000", 2024-02-09T17:39:28.024 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0016155080000000001 2024-02-09T17:39:28.024 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.024 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.024 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:28.024 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.033044+0000", 2024-02-09T17:39:28.024 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0036805330000000002 2024-02-09T17:39:28.024 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.024 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.024 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:28.025 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.033116+0000", 2024-02-09T17:39:28.025 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1848999999999996e-05 2024-02-09T17:39:28.025 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.025 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.025 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:28.025 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.033253+0000", 2024-02-09T17:39:28.025 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013648999999999999 2024-02-09T17:39:28.025 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:28.025 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:28.025 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:28.025 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.026 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.026 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:209065 4.7 4:e30085c8:::benchmark_data_smithi179_103809_object71599:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-02-09T17:39:28.026 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:37.083719+0000", 2024-02-09T17:39:28.026 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 50.827560073999997, 2024-02-09T17:39:28.026 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.058342169999999999, 2024-02-09T17:39:28.026 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:28.026 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:28.026 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:28.026 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:28.026 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:28.026 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 209065 2024-02-09T17:39:28.027 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.027 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:28.027 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.027 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:28.027 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.083719+0000", 2024-02-09T17:39:28.027 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:28.027 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.027 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.027 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:28.027 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.083719+0000", 2024-02-09T17:39:28.027 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:28.028 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.028 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.028 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:28.028 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.083721+0000", 2024-02-09T17:39:28.028 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.323e-06 2024-02-09T17:39:28.028 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.028 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.028 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:28.028 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.083734+0000", 2024-02-09T17:39:28.028 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2695e-05 2024-02-09T17:39:28.028 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.029 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.029 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:28.029 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.083749+0000", 2024-02-09T17:39:28.029 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5285e-05 2024-02-09T17:39:28.029 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.029 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.029 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:28.029 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.083760+0000", 2024-02-09T17:39:28.029 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0883999999999999e-05 2024-02-09T17:39:28.029 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.029 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.029 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:28.030 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.086382+0000", 2024-02-09T17:39:28.030 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0026216880000000001 2024-02-09T17:39:28.030 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.030 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.030 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:28.030 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.086544+0000", 2024-02-09T17:39:28.030 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016243 2024-02-09T17:39:28.030 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.030 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.030 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,1,7", 2024-02-09T17:39:28.030 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.086703+0000", 2024-02-09T17:39:28.030 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015889800000000001 2024-02-09T17:39:28.031 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.031 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.031 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:28.031 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.095967+0000", 2024-02-09T17:39:28.031 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0092637369999999993 2024-02-09T17:39:28.031 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.031 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.031 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:28.031 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.095982+0000", 2024-02-09T17:39:28.031 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5624000000000002e-05 2024-02-09T17:39:28.031 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.032 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.032 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:28.032 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.106464+0000", 2024-02-09T17:39:28.032 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010482178999999999 2024-02-09T17:39:28.032 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.032 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.032 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:28.032 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.141863+0000", 2024-02-09T17:39:28.032 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.03539871 2024-02-09T17:39:28.032 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.033 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.033 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:28.033 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.141936+0000", 2024-02-09T17:39:28.033 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2435999999999996e-05 2024-02-09T17:39:28.033 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.033 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.033 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:28.033 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.142061+0000", 2024-02-09T17:39:28.033 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012528100000000001 2024-02-09T17:39:28.033 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:28.033 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:28.033 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:28.034 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.034 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.034 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:221016 4.7 4:e90acc43:::benchmark_data_smithi179_103809_object83550:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e397)", 2024-02-09T17:39:28.034 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:39:00.819301+0000", 2024-02-09T17:39:28.034 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 27.091977496999998, 2024-02-09T17:39:28.034 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.088963515000000007, 2024-02-09T17:39:28.034 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:28.034 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:28.034 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:28.035 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:28.035 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:28.035 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 221016 2024-02-09T17:39:28.035 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.035 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:28.035 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.035 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:28.035 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:00.819301+0000", 2024-02-09T17:39:28.035 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:28.035 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.035 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.036 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:28.036 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:00.819301+0000", 2024-02-09T17:39:28.036 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:28.036 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.036 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.036 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:28.036 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:00.819303+0000", 2024-02-09T17:39:28.036 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0899999999999999e-06 2024-02-09T17:39:28.036 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.037 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.037 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:28.037 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:00.819309+0000", 2024-02-09T17:39:28.037 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9950000000000002e-06 2024-02-09T17:39:28.037 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.037 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.037 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:28.037 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:00.819323+0000", 2024-02-09T17:39:28.037 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3835000000000001e-05 2024-02-09T17:39:28.037 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.038 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.038 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:28.038 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:00.819331+0000", 2024-02-09T17:39:28.038 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.174e-06 2024-02-09T17:39:28.038 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.038 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.038 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:28.038 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:00.819391+0000", 2024-02-09T17:39:28.038 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9963999999999997e-05 2024-02-09T17:39:28.038 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.038 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.039 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:28.039 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:00.819534+0000", 2024-02-09T17:39:28.039 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014299899999999999 2024-02-09T17:39:28.039 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.039 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.039 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,7", 2024-02-09T17:39:28.039 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:00.819656+0000", 2024-02-09T17:39:28.039 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000121977 2024-02-09T17:39:28.039 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.039 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.039 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:28.039 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:00.827783+0000", 2024-02-09T17:39:28.040 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0081267839999999997 2024-02-09T17:39:28.040 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.040 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.040 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:28.040 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:00.828012+0000", 2024-02-09T17:39:28.040 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00022934999999999999 2024-02-09T17:39:28.040 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.040 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.040 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:28.040 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:00.908163+0000", 2024-02-09T17:39:28.040 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.080151063999999994 2024-02-09T17:39:28.040 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.040 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.040 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:28.041 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:00.908225+0000", 2024-02-09T17:39:28.041 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1439000000000006e-05 2024-02-09T17:39:28.041 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.041 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.041 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:28.041 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:00.908265+0000", 2024-02-09T17:39:28.041 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9844e-05 2024-02-09T17:39:28.041 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:28.041 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:28.041 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:28.041 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.041 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.041 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:232311 4.7 4:ef68b42c:::benchmark_data_smithi179_103809_object94845:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-02-09T17:39:28.041 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:39:22.353494+0000", 2024-02-09T17:39:28.041 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 5.5577850719999997, 2024-02-09T17:39:28.042 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.064656210000000006, 2024-02-09T17:39:28.042 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:28.042 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:28.042 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:28.042 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:28.042 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:28.042 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 232311 2024-02-09T17:39:28.042 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.042 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:28.042 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.042 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:28.042 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.353494+0000", 2024-02-09T17:39:28.042 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:28.042 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.042 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.043 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:28.043 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.353494+0000", 2024-02-09T17:39:28.043 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:28.043 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.043 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.043 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:28.043 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.353495+0000", 2024-02-09T17:39:28.043 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.322e-06 2024-02-09T17:39:28.043 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.043 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.043 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:28.043 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.353501+0000", 2024-02-09T17:39:28.043 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7710000000000003e-06 2024-02-09T17:39:28.043 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.044 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.044 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:28.044 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.353515+0000", 2024-02-09T17:39:28.044 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3859999999999999e-05 2024-02-09T17:39:28.044 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.044 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.044 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:28.044 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.353524+0000", 2024-02-09T17:39:28.044 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.064e-06 2024-02-09T17:39:28.044 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.044 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.044 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:28.044 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.353579+0000", 2024-02-09T17:39:28.044 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5643999999999998e-05 2024-02-09T17:39:28.045 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.045 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.045 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:28.045 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.353710+0000", 2024-02-09T17:39:28.045 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013103600000000001 2024-02-09T17:39:28.045 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.045 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.045 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,7", 2024-02-09T17:39:28.045 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.353840+0000", 2024-02-09T17:39:28.045 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012951400000000001 2024-02-09T17:39:28.045 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.045 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.045 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:28.045 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.363306+0000", 2024-02-09T17:39:28.045 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0094663279999999995 2024-02-09T17:39:28.046 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.046 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.046 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:28.046 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.363324+0000", 2024-02-09T17:39:28.046 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8063999999999998e-05 2024-02-09T17:39:28.046 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.046 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.046 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:28.046 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.418033+0000", 2024-02-09T17:39:28.046 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.054708344999999999 2024-02-09T17:39:28.046 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.046 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.046 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:28.046 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.418097+0000", 2024-02-09T17:39:28.047 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3952e-05 2024-02-09T17:39:28.047 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.047 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.047 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:28.047 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.418150+0000", 2024-02-09T17:39:28.047 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3310000000000003e-05 2024-02-09T17:39:28.047 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:28.047 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:28.047 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:28.047 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.047 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.047 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:232313 4.7 4:e5adcb4d:::benchmark_data_smithi179_103809_object94847:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-02-09T17:39:28.047 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:39:22.356043+0000", 2024-02-09T17:39:28.047 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 5.5552359899999999, 2024-02-09T17:39:28.048 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.069876890999999997, 2024-02-09T17:39:28.048 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:28.048 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:28.048 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:28.048 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:28.048 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:28.048 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 232313 2024-02-09T17:39:28.048 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.048 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:28.048 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.048 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:28.048 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.356043+0000", 2024-02-09T17:39:28.048 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:28.049 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.049 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.049 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:28.049 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.356043+0000", 2024-02-09T17:39:28.049 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:28.049 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.049 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.049 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:28.049 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.356044+0000", 2024-02-09T17:39:28.049 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.44e-06 2024-02-09T17:39:28.049 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.049 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.049 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:28.050 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.356051+0000", 2024-02-09T17:39:28.050 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5010000000000001e-06 2024-02-09T17:39:28.050 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.050 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.050 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:28.050 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.356066+0000", 2024-02-09T17:39:28.050 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5607000000000002e-05 2024-02-09T17:39:28.050 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.050 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.050 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:28.050 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.356076+0000", 2024-02-09T17:39:28.050 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7820000000000005e-06 2024-02-09T17:39:28.050 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.051 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.051 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:28.051 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.357996+0000", 2024-02-09T17:39:28.051 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001919551 2024-02-09T17:39:28.051 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.051 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.051 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:28.051 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.358113+0000", 2024-02-09T17:39:28.051 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000117033 2024-02-09T17:39:28.051 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.051 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.051 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,7", 2024-02-09T17:39:28.052 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.358236+0000", 2024-02-09T17:39:28.052 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000123106 2024-02-09T17:39:28.052 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.052 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.052 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:28.052 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.363815+0000", 2024-02-09T17:39:28.052 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0055790090000000002 2024-02-09T17:39:28.052 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.052 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.052 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:28.052 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.369557+0000", 2024-02-09T17:39:28.052 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0057423429999999996 2024-02-09T17:39:28.052 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.052 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.053 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:28.053 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.425810+0000", 2024-02-09T17:39:28.053 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.056253351 2024-02-09T17:39:28.053 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.053 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.053 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:28.053 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.425868+0000", 2024-02-09T17:39:28.053 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7581000000000003e-05 2024-02-09T17:39:28.053 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.053 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.053 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:28.053 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.425920+0000", 2024-02-09T17:39:28.053 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1586999999999999e-05 2024-02-09T17:39:28.053 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:28.054 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:28.054 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:28.054 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.054 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.054 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:232581 4.5 4:a36c0d16:::benchmark_data_smithi179_103809_object95115:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-02-09T17:39:28.054 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:39:24.589203+0000", 2024-02-09T17:39:28.054 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 3.3220754490000002, 2024-02-09T17:39:28.054 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0160283800000001, 2024-02-09T17:39:28.054 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:28.054 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:28.054 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:28.054 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:28.055 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:28.055 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 232581 2024-02-09T17:39:28.055 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.055 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:28.055 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.055 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:28.055 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:24.589203+0000", 2024-02-09T17:39:28.055 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:28.055 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.055 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.055 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:28.055 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:24.589203+0000", 2024-02-09T17:39:28.055 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:28.056 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.056 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.056 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:28.056 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:24.589205+0000", 2024-02-09T17:39:28.056 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.756e-06 2024-02-09T17:39:28.056 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.056 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.056 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:28.056 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:24.589213+0000", 2024-02-09T17:39:28.056 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1939999999999998e-06 2024-02-09T17:39:28.056 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.056 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.056 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:28.057 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:24.589232+0000", 2024-02-09T17:39:28.057 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.912e-05 2024-02-09T17:39:28.057 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.057 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.057 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:28.057 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:24.589242+0000", 2024-02-09T17:39:28.057 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.3700000000000001e-06 2024-02-09T17:39:28.057 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.057 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.057 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:28.057 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:24.589307+0000", 2024-02-09T17:39:28.057 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5356000000000002e-05 2024-02-09T17:39:28.058 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.058 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.058 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:39:28.058 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:24.589316+0000", 2024-02-09T17:39:28.058 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.3260000000000002e-06 2024-02-09T17:39:28.058 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.058 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.058 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:28.058 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:25.599093+0000", 2024-02-09T17:39:28.058 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.009776778 2024-02-09T17:39:28.058 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.059 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.059 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:28.059 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:25.599260+0000", 2024-02-09T17:39:28.059 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016675300000000001 2024-02-09T17:39:28.059 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.059 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.059 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,7", 2024-02-09T17:39:28.059 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:25.599410+0000", 2024-02-09T17:39:28.059 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015003099999999999 2024-02-09T17:39:28.059 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.059 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.059 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:28.060 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:25.605045+0000", 2024-02-09T17:39:28.060 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0056348650000000002 2024-02-09T17:39:28.060 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.060 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.060 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:28.060 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:25.605094+0000", 2024-02-09T17:39:28.060 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9413000000000003e-05 2024-02-09T17:39:28.060 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.060 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.060 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:28.060 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:25.605116+0000", 2024-02-09T17:39:28.061 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2272999999999998e-05 2024-02-09T17:39:28.061 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.061 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.061 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:28.061 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:25.605186+0000", 2024-02-09T17:39:28.061 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9356999999999993e-05 2024-02-09T17:39:28.061 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:28.061 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:28.061 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:28.061 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:25.605232+0000", 2024-02-09T17:39:28.061 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5788000000000003e-05 2024-02-09T17:39:28.062 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:28.062 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:28.062 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:28.062 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:28.062 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:28.062 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:39:28.063 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:27 smithi099 ceph-mon[39420]: pgmap v1098: 37 pgs: 1 active+remapped+backfill_toofull, 1 remapped+peering, 2 peering, 1 active+undersized+remapped, 1 active+clean+remapped, 1 activating+remapped, 4 active+remapped+backfilling, 26 active+clean; 2.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 374 op/s; 11835/126990 objects misplaced (9.320%); 46 MiB/s, 741 objects/s recovering 2024-02-09T17:39:28.063 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:27 smithi099 ceph-mon[39420]: osdmap e418: 12 total, 12 up, 11 in 2024-02-09T17:39:28.064 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:27 smithi179 ceph-mon[43975]: pgmap v1098: 37 pgs: 1 active+remapped+backfill_toofull, 1 remapped+peering, 2 peering, 1 active+undersized+remapped, 1 active+clean+remapped, 1 activating+remapped, 4 active+remapped+backfilling, 26 active+clean; 2.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 374 op/s; 11835/126990 objects misplaced (9.320%); 46 MiB/s, 741 objects/s recovering 2024-02-09T17:39:28.064 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:27 smithi179 ceph-mon[43975]: osdmap e418: 12 total, 12 up, 11 in 2024-02-09T17:39:28.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:28.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:39:28.065 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 3 1 2024-02-09T17:39:28.166 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:39:28.223 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:28.283 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:39:28.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:28.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:39:28.392 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:39:28.426 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:28.582 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:39:28.599 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:28 smithi099 ceph-mon[39420]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:39:28.648 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:39:28.677 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.11 dump_ops_in_flight 2024-02-09T17:39:28.681 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:39:28.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:28.685 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:39:28.690 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:39:28.737 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:39:28.783 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:39:28.786 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:39:28.815 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:28.829 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:39:28.836 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:39:28.839 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:28.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:39:28.866 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:39:28.908 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:39:28.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:28 smithi162 ceph-mon[44304]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:39:28.926 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:28 smithi179 ceph-mon[43975]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:39:28.941 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:39:28.950 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:39:28.974 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:29.001 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:39:29.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:29.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:39:29.009 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:39:29.049 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:39:29.054 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:39:29.096 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:39:29.110 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:39:29.139 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:29.151 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:39:29.206 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:39:29.206 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:39:29.207 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:39:29.207 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:39:29.207 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:39:29.207 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:39:29.207 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:39:29.207 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:39:29.207 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:39:29.207 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:39:29.207 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:39:29.207 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:39:29.207 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:39:29.340 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:39:29.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:29.344 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:39:29.445 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:39:29.478 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:29.506 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:39:29.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:29.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:39:29.614 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:39:29.662 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:29.873 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:39:29.875 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:29 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3111823509' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:39:29.875 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:29 smithi179 ceph-mon[43975]: pgmap v1100: 37 pgs: 1 active+remapped+backfill_toofull, 1 remapped+peering, 3 peering, 1 active+undersized+remapped, 1 activating+remapped, 2 active+remapped+backfilling, 28 active+clean; 2.6 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 302 op/s; 8740/126171 objects misplaced (6.927%); 48 MiB/s, 769 objects/s recovering 2024-02-09T17:39:29.876 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:29.876 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:39:29.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:29 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3111823509' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:39:29.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:29 smithi162 ceph-mon[44304]: pgmap v1100: 37 pgs: 1 active+remapped+backfill_toofull, 1 remapped+peering, 3 peering, 1 active+undersized+remapped, 1 activating+remapped, 2 active+remapped+backfilling, 28 active+clean; 2.6 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 302 op/s; 8740/126171 objects misplaced (6.927%); 48 MiB/s, 769 objects/s recovering 2024-02-09T17:39:29.977 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:39:29.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:29 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3111823509' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:39:29.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:29 smithi099 ceph-mon[39420]: pgmap v1100: 37 pgs: 1 active+remapped+backfill_toofull, 1 remapped+peering, 3 peering, 1 active+undersized+remapped, 1 activating+remapped, 2 active+remapped+backfilling, 28 active+clean; 2.6 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 302 op/s; 8740/126171 objects misplaced (6.927%); 48 MiB/s, 769 objects/s recovering 2024-02-09T17:39:30.021 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:30.128 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:39:30.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:30.134 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:39:30.235 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:39:30.272 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:31.232 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:39:31.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:31.237 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:39:31.338 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:39:31.417 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:31.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:31 smithi099 ceph-mon[39420]: pgmap v1101: 37 pgs: 1 active+remapped+backfill_toofull, 1 peering, 5 active+remapped+backfilling, 30 active+clean; 2.4 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 575 op/s; 12477/119646 objects misplaced (10.428%); 50 MiB/s, 792 objects/s recovering 2024-02-09T17:39:31.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:31 smithi099 ceph-mon[39420]: osdmap e419: 12 total, 12 up, 11 in 2024-02-09T17:39:32.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:31 smithi179 ceph-mon[43975]: pgmap v1101: 37 pgs: 1 active+remapped+backfill_toofull, 1 peering, 5 active+remapped+backfilling, 30 active+clean; 2.4 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 575 op/s; 12477/119646 objects misplaced (10.428%); 50 MiB/s, 792 objects/s recovering 2024-02-09T17:39:32.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:31 smithi179 ceph-mon[43975]: osdmap e419: 12 total, 12 up, 11 in 2024-02-09T17:39:32.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:31 smithi162 ceph-mon[44304]: pgmap v1101: 37 pgs: 1 active+remapped+backfill_toofull, 1 peering, 5 active+remapped+backfilling, 30 active+clean; 2.4 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 575 op/s; 12477/119646 objects misplaced (10.428%); 50 MiB/s, 792 objects/s recovering 2024-02-09T17:39:32.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:31 smithi162 ceph-mon[44304]: osdmap e419: 12 total, 12 up, 11 in 2024-02-09T17:39:32.811 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:39:32.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:32.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:39:32.916 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:39:32.957 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:33.348 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:39:33.788 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:39:33.855 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is unset 2024-02-09T17:39:33.878 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.3 to 1 (10000) 2024-02-09T17:39:34.019 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:39:34.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:34.023 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:39:34.108 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:33 smithi179 ceph-mon[43975]: pgmap v1103: 37 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 31 active+clean; 2.4 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 566 op/s; 11996/116874 objects misplaced (10.264%); 39 MiB/s, 619 objects/s recovering 2024-02-09T17:39:34.108 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:33 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2558205220' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:39:34.108 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:33 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:39:34.108 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:33 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3625244618' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-09T17:39:34.108 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:33 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:39:34.109 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:33 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:39:34.124 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:39:34.140 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:33 smithi099 ceph-mon[39420]: pgmap v1103: 37 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 31 active+clean; 2.4 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 566 op/s; 11996/116874 objects misplaced (10.264%); 39 MiB/s, 619 objects/s recovering 2024-02-09T17:39:34.140 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:33 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2558205220' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:39:34.141 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:33 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:39:34.141 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:33 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3625244618' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-09T17:39:34.141 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:33 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:39:34.141 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:33 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:39:34.153 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:34.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:33 smithi162 ceph-mon[44304]: pgmap v1103: 37 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 31 active+clean; 2.4 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 566 op/s; 11996/116874 objects misplaced (10.264%); 39 MiB/s, 619 objects/s recovering 2024-02-09T17:39:34.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:33 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2558205220' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:39:34.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:33 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:39:34.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:33 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3625244618' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-09T17:39:34.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:33 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:39:34.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:33 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:39:34.471 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T17:39:34.472 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:39:34.502 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:39:34.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:34.504 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:39:34.605 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:39:34.639 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:34.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:34 smithi099 ceph-mon[39420]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:39:34.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:34 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:39:34.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:34 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3625244618' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-09T17:39:34.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:34 smithi099 ceph-mon[39420]: osdmap e420: 12 total, 12 up, 11 in 2024-02-09T17:39:34.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:34 smithi099 ceph-mon[39420]: 4.3 scrub starts 2024-02-09T17:39:35.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:34 smithi179 ceph-mon[43975]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:39:35.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:34 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:39:35.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:34 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3625244618' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-09T17:39:35.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:34 smithi179 ceph-mon[43975]: osdmap e420: 12 total, 12 up, 11 in 2024-02-09T17:39:35.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:34 smithi179 ceph-mon[43975]: 4.3 scrub starts 2024-02-09T17:39:35.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:34 smithi162 ceph-mon[44304]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:39:35.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:34 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:39:35.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:34 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3625244618' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-09T17:39:35.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:34 smithi162 ceph-mon[44304]: osdmap e420: 12 total, 12 up, 11 in 2024-02-09T17:39:35.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:34 smithi162 ceph-mon[44304]: 4.3 scrub starts 2024-02-09T17:39:35.219 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.11 dump_blocked_ops 2024-02-09T17:39:35.503 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:39:36.061 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:35 smithi179 ceph-mon[43975]: 4.3 scrub ok 2024-02-09T17:39:36.061 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:35 smithi179 ceph-mon[43975]: 4.8 scrub starts 2024-02-09T17:39:36.061 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:35 smithi179 ceph-mon[43975]: pgmap v1105: 37 pgs: 1 peering, 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 30 active+clean; 2.3 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 539 op/s; 11467/114654 objects misplaced (10.001%); 23 MiB/s, 368 objects/s recovering 2024-02-09T17:39:36.061 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:35 smithi179 ceph-mon[43975]: osdmap e421: 12 total, 12 up, 11 in 2024-02-09T17:39:36.061 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:35 smithi179 ceph-mon[43975]: 4.8 scrub ok 2024-02-09T17:39:36.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:35 smithi162 ceph-mon[44304]: 4.3 scrub ok 2024-02-09T17:39:36.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:35 smithi162 ceph-mon[44304]: 4.8 scrub starts 2024-02-09T17:39:36.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:35 smithi162 ceph-mon[44304]: pgmap v1105: 37 pgs: 1 peering, 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 30 active+clean; 2.3 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 539 op/s; 11467/114654 objects misplaced (10.001%); 23 MiB/s, 368 objects/s recovering 2024-02-09T17:39:36.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:35 smithi162 ceph-mon[44304]: osdmap e421: 12 total, 12 up, 11 in 2024-02-09T17:39:36.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:35 smithi162 ceph-mon[44304]: 4.8 scrub ok 2024-02-09T17:39:36.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:35 smithi099 ceph-mon[39420]: 4.3 scrub ok 2024-02-09T17:39:36.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:35 smithi099 ceph-mon[39420]: 4.8 scrub starts 2024-02-09T17:39:36.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:35 smithi099 ceph-mon[39420]: pgmap v1105: 37 pgs: 1 peering, 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 30 active+clean; 2.3 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 539 op/s; 11467/114654 objects misplaced (10.001%); 23 MiB/s, 368 objects/s recovering 2024-02-09T17:39:36.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:35 smithi099 ceph-mon[39420]: osdmap e421: 12 total, 12 up, 11 in 2024-02-09T17:39:36.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:35 smithi099 ceph-mon[39420]: 4.8 scrub ok 2024-02-09T17:39:37.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:36 smithi179 ceph-mon[43975]: osdmap e422: 12 total, 12 up, 11 in 2024-02-09T17:39:37.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:36 smithi179 ceph-mon[43975]: 4.0 starting backfill to osd.3 from (0'0,0'0] MAX to 419'14924 2024-02-09T17:39:37.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:36 smithi179 ceph-mon[43975]: 4.4 continuing backfill to osd.10 from (340'4700,419'14790] 4:2810fb8b:::benchmark_data_smithi179_103809_object115717:head to 419'14790 2024-02-09T17:39:37.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:36 smithi179 ceph-mon[43975]: 1.9 scrub starts 2024-02-09T17:39:37.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:36 smithi179 ceph-mon[43975]: 1.11 scrub starts 2024-02-09T17:39:37.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:36 smithi179 ceph-mon[43975]: 1.11 scrub ok 2024-02-09T17:39:37.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:36 smithi179 ceph-mon[43975]: 1.9 scrub ok 2024-02-09T17:39:37.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:36 smithi179 ceph-mon[43975]: 1.f scrub starts 2024-02-09T17:39:37.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:36 smithi179 ceph-mon[43975]: 1.f scrub ok 2024-02-09T17:39:37.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:36 smithi179 ceph-mon[43975]: osdmap e423: 12 total, 12 up, 11 in 2024-02-09T17:39:37.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:36 smithi162 ceph-mon[44304]: osdmap e422: 12 total, 12 up, 11 in 2024-02-09T17:39:37.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:36 smithi162 ceph-mon[44304]: 4.0 starting backfill to osd.3 from (0'0,0'0] MAX to 419'14924 2024-02-09T17:39:37.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:36 smithi162 ceph-mon[44304]: 4.4 continuing backfill to osd.10 from (340'4700,419'14790] 4:2810fb8b:::benchmark_data_smithi179_103809_object115717:head to 419'14790 2024-02-09T17:39:37.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:36 smithi162 ceph-mon[44304]: 1.9 scrub starts 2024-02-09T17:39:37.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:36 smithi162 ceph-mon[44304]: 1.11 scrub starts 2024-02-09T17:39:37.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:36 smithi162 ceph-mon[44304]: 1.11 scrub ok 2024-02-09T17:39:37.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:36 smithi162 ceph-mon[44304]: 1.9 scrub ok 2024-02-09T17:39:37.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:36 smithi162 ceph-mon[44304]: 1.f scrub starts 2024-02-09T17:39:37.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:36 smithi162 ceph-mon[44304]: 1.f scrub ok 2024-02-09T17:39:37.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:36 smithi162 ceph-mon[44304]: osdmap e423: 12 total, 12 up, 11 in 2024-02-09T17:39:37.249 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:36 smithi099 ceph-mon[39420]: osdmap e422: 12 total, 12 up, 11 in 2024-02-09T17:39:37.249 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:36 smithi099 ceph-mon[39420]: 4.0 starting backfill to osd.3 from (0'0,0'0] MAX to 419'14924 2024-02-09T17:39:37.249 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:36 smithi099 ceph-mon[39420]: 4.4 continuing backfill to osd.10 from (340'4700,419'14790] 4:2810fb8b:::benchmark_data_smithi179_103809_object115717:head to 419'14790 2024-02-09T17:39:37.249 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:36 smithi099 ceph-mon[39420]: 1.9 scrub starts 2024-02-09T17:39:37.249 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:36 smithi099 ceph-mon[39420]: 1.11 scrub starts 2024-02-09T17:39:37.249 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:36 smithi099 ceph-mon[39420]: 1.11 scrub ok 2024-02-09T17:39:37.249 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:36 smithi099 ceph-mon[39420]: 1.9 scrub ok 2024-02-09T17:39:37.249 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:36 smithi099 ceph-mon[39420]: 1.f scrub starts 2024-02-09T17:39:37.249 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:36 smithi099 ceph-mon[39420]: 1.f scrub ok 2024-02-09T17:39:37.249 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:36 smithi099 ceph-mon[39420]: osdmap e423: 12 total, 12 up, 11 in 2024-02-09T17:39:37.686 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T17:39:37.687 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:39:38.092 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:37 smithi179 ceph-mon[43975]: pgmap v1109: 37 pgs: 1 active+remapped+backfill_toofull, 1 remapped+peering, 1 peering, 3 active+remapped+backfilling, 31 active+clean; 2.2 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 587 op/s; 5272/109818 objects misplaced (4.801%); 53 MiB/s, 854 objects/s recovering 2024-02-09T17:39:38.092 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:37 smithi179 ceph-mon[43975]: osdmap e424: 12 total, 12 up, 11 in 2024-02-09T17:39:38.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:37 smithi162 ceph-mon[44304]: pgmap v1109: 37 pgs: 1 active+remapped+backfill_toofull, 1 remapped+peering, 1 peering, 3 active+remapped+backfilling, 31 active+clean; 2.2 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 587 op/s; 5272/109818 objects misplaced (4.801%); 53 MiB/s, 854 objects/s recovering 2024-02-09T17:39:38.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:37 smithi162 ceph-mon[44304]: osdmap e424: 12 total, 12 up, 11 in 2024-02-09T17:39:38.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:37 smithi099 ceph-mon[39420]: pgmap v1109: 37 pgs: 1 active+remapped+backfill_toofull, 1 remapped+peering, 1 peering, 3 active+remapped+backfilling, 31 active+clean; 2.2 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 587 op/s; 5272/109818 objects misplaced (4.801%); 53 MiB/s, 854 objects/s recovering 2024-02-09T17:39:38.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:37 smithi099 ceph-mon[39420]: osdmap e424: 12 total, 12 up, 11 in 2024-02-09T17:39:38.507 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.11 dump_historic_ops 2024-02-09T17:39:38.896 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:39:39.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:38 smithi179 ceph-mon[43975]: 1.0 scrub starts 2024-02-09T17:39:39.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:38 smithi179 ceph-mon[43975]: 1.0 scrub ok 2024-02-09T17:39:39.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:38 smithi179 ceph-mon[43975]: 1.10 scrub starts 2024-02-09T17:39:39.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:38 smithi179 ceph-mon[43975]: 1.10 scrub ok 2024-02-09T17:39:39.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:38 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:39:39.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:38 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:39:39.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:38 smithi162 ceph-mon[44304]: 1.0 scrub starts 2024-02-09T17:39:39.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:38 smithi162 ceph-mon[44304]: 1.0 scrub ok 2024-02-09T17:39:39.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:38 smithi162 ceph-mon[44304]: 1.10 scrub starts 2024-02-09T17:39:39.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:38 smithi162 ceph-mon[44304]: 1.10 scrub ok 2024-02-09T17:39:39.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:38 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:39:39.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:38 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:39:39.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:38 smithi099 ceph-mon[39420]: 1.0 scrub starts 2024-02-09T17:39:39.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:38 smithi099 ceph-mon[39420]: 1.0 scrub ok 2024-02-09T17:39:39.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:38 smithi099 ceph-mon[39420]: 1.10 scrub starts 2024-02-09T17:39:39.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:38 smithi099 ceph-mon[39420]: 1.10 scrub ok 2024-02-09T17:39:39.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:38 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:39:39.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:38 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:39:40.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:39 smithi179 ceph-mon[43975]: pgmap v1111: 37 pgs: 1 active+remapped+backfill_toofull, 1 remapped+peering, 2 peering, 3 active+remapped+backfilling, 30 active+clean; 2.2 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 471 op/s; 5272/108990 objects misplaced (4.837%); 48 MiB/s, 770 objects/s recovering 2024-02-09T17:39:40.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:39 smithi179 ceph-mon[43975]: 1.14 scrub starts 2024-02-09T17:39:40.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:39 smithi179 ceph-mon[43975]: osdmap e425: 12 total, 12 up, 11 in 2024-02-09T17:39:40.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:39 smithi179 ceph-mon[43975]: 1.14 scrub ok 2024-02-09T17:39:40.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:39 smithi179 ceph-mon[43975]: 4.2 scrub starts 2024-02-09T17:39:40.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:39 smithi162 ceph-mon[44304]: pgmap v1111: 37 pgs: 1 active+remapped+backfill_toofull, 1 remapped+peering, 2 peering, 3 active+remapped+backfilling, 30 active+clean; 2.2 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 471 op/s; 5272/108990 objects misplaced (4.837%); 48 MiB/s, 770 objects/s recovering 2024-02-09T17:39:40.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:39 smithi162 ceph-mon[44304]: 1.14 scrub starts 2024-02-09T17:39:40.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:39 smithi162 ceph-mon[44304]: osdmap e425: 12 total, 12 up, 11 in 2024-02-09T17:39:40.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:39 smithi162 ceph-mon[44304]: 1.14 scrub ok 2024-02-09T17:39:40.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:39 smithi162 ceph-mon[44304]: 4.2 scrub starts 2024-02-09T17:39:40.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:39 smithi099 ceph-mon[39420]: pgmap v1111: 37 pgs: 1 active+remapped+backfill_toofull, 1 remapped+peering, 2 peering, 3 active+remapped+backfilling, 30 active+clean; 2.2 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 471 op/s; 5272/108990 objects misplaced (4.837%); 48 MiB/s, 770 objects/s recovering 2024-02-09T17:39:40.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:39 smithi099 ceph-mon[39420]: 1.14 scrub starts 2024-02-09T17:39:40.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:39 smithi099 ceph-mon[39420]: osdmap e425: 12 total, 12 up, 11 in 2024-02-09T17:39:40.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:39 smithi099 ceph-mon[39420]: 1.14 scrub ok 2024-02-09T17:39:40.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:39 smithi099 ceph-mon[39420]: 4.2 scrub starts 2024-02-09T17:39:40.849 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T17:39:40.849 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:39:41.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:40 smithi179 ceph-mon[43975]: 4.2 scrub ok 2024-02-09T17:39:41.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:40 smithi179 ceph-mon[43975]: osdmap e426: 12 total, 12 up, 11 in 2024-02-09T17:39:41.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:40 smithi162 ceph-mon[44304]: 4.2 scrub ok 2024-02-09T17:39:41.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:40 smithi162 ceph-mon[44304]: osdmap e426: 12 total, 12 up, 11 in 2024-02-09T17:39:41.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:40 smithi099 ceph-mon[39420]: 4.2 scrub ok 2024-02-09T17:39:41.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:40 smithi099 ceph-mon[39420]: osdmap e426: 12 total, 12 up, 11 in 2024-02-09T17:39:41.680 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.8 dump_ops_in_flight 2024-02-09T17:39:41.957 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:39:42.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:41 smithi179 ceph-mon[43975]: 1.d scrub starts 2024-02-09T17:39:42.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:41 smithi179 ceph-mon[43975]: 1.d scrub ok 2024-02-09T17:39:42.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:41 smithi179 ceph-mon[43975]: pgmap v1114: 37 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 1 peering, 2 active+remapped+backfilling, 29 active+clean; 2.1 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 646 op/s; 6246/102297 objects misplaced (6.106%); 46 MiB/s, 729 objects/s recovering 2024-02-09T17:39:42.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:41 smithi179 ceph-mon[43975]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:39:42.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:41 smithi179 ceph-mon[43975]: osdmap e427: 12 total, 12 up, 11 in 2024-02-09T17:39:42.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:41 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:39:42.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:41 smithi179 ceph-mon[43975]: osdmap e428: 12 total, 12 up, 11 in 2024-02-09T17:39:42.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:41 smithi162 ceph-mon[44304]: 1.d scrub starts 2024-02-09T17:39:42.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:41 smithi162 ceph-mon[44304]: 1.d scrub ok 2024-02-09T17:39:42.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:41 smithi162 ceph-mon[44304]: pgmap v1114: 37 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 1 peering, 2 active+remapped+backfilling, 29 active+clean; 2.1 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 646 op/s; 6246/102297 objects misplaced (6.106%); 46 MiB/s, 729 objects/s recovering 2024-02-09T17:39:42.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:41 smithi162 ceph-mon[44304]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:39:42.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:41 smithi162 ceph-mon[44304]: osdmap e427: 12 total, 12 up, 11 in 2024-02-09T17:39:42.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:41 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:39:42.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:41 smithi162 ceph-mon[44304]: osdmap e428: 12 total, 12 up, 11 in 2024-02-09T17:39:42.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:41 smithi099 ceph-mon[39420]: 1.d scrub starts 2024-02-09T17:39:42.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:41 smithi099 ceph-mon[39420]: 1.d scrub ok 2024-02-09T17:39:42.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:41 smithi099 ceph-mon[39420]: pgmap v1114: 37 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 1 peering, 2 active+remapped+backfilling, 29 active+clean; 2.1 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 646 op/s; 6246/102297 objects misplaced (6.106%); 46 MiB/s, 729 objects/s recovering 2024-02-09T17:39:42.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:41 smithi099 ceph-mon[39420]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:39:42.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:41 smithi099 ceph-mon[39420]: osdmap e427: 12 total, 12 up, 11 in 2024-02-09T17:39:42.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:41 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:39:42.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:41 smithi099 ceph-mon[39420]: osdmap e428: 12 total, 12 up, 11 in 2024-02-09T17:39:42.625 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:39:42.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:42.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:39:42.736 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:39:42.796 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:43.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:42 smithi162 ceph-mon[44304]: 4.d scrub starts 2024-02-09T17:39:43.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:42 smithi162 ceph-mon[44304]: 4.d scrub ok 2024-02-09T17:39:43.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:42 smithi162 ceph-mon[44304]: 1.a scrub starts 2024-02-09T17:39:43.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:42 smithi162 ceph-mon[44304]: 1.a scrub ok 2024-02-09T17:39:43.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:42 smithi162 ceph-mon[44304]: osdmap e429: 12 total, 12 up, 11 in 2024-02-09T17:39:43.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:42 smithi099 ceph-mon[39420]: 4.d scrub starts 2024-02-09T17:39:43.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:42 smithi099 ceph-mon[39420]: 4.d scrub ok 2024-02-09T17:39:43.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:42 smithi099 ceph-mon[39420]: 1.a scrub starts 2024-02-09T17:39:43.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:42 smithi099 ceph-mon[39420]: 1.a scrub ok 2024-02-09T17:39:43.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:42 smithi099 ceph-mon[39420]: osdmap e429: 12 total, 12 up, 11 in 2024-02-09T17:39:43.324 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:42 smithi179 ceph-mon[43975]: 4.d scrub starts 2024-02-09T17:39:43.324 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:42 smithi179 ceph-mon[43975]: 4.d scrub ok 2024-02-09T17:39:43.324 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:42 smithi179 ceph-mon[43975]: 1.a scrub starts 2024-02-09T17:39:43.324 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:42 smithi179 ceph-mon[43975]: 1.a scrub ok 2024-02-09T17:39:43.324 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:42 smithi179 ceph-mon[43975]: osdmap e429: 12 total, 12 up, 11 in 2024-02-09T17:39:43.592 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:39:43.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:43.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:39:43.700 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:39:43.735 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:43.764 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:39:43.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:43.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:39:43.868 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:39:43.896 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:44.034 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T17:39:44.034 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:39:44.062 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:43 smithi179 ceph-mon[43975]: pgmap v1118: 37 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 31 active+clean; 2.0 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 593 op/s; 5970/100629 objects misplaced (5.933%); 16 MiB/s, 259 objects/s recovering 2024-02-09T17:39:44.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:43 smithi162 ceph-mon[44304]: pgmap v1118: 37 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 31 active+clean; 2.0 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 593 op/s; 5970/100629 objects misplaced (5.933%); 16 MiB/s, 259 objects/s recovering 2024-02-09T17:39:44.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:43 smithi099 ceph-mon[39420]: pgmap v1118: 37 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 31 active+clean; 2.0 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 593 op/s; 5970/100629 objects misplaced (5.933%); 16 MiB/s, 259 objects/s recovering 2024-02-09T17:39:44.854 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.8 dump_blocked_ops 2024-02-09T17:39:45.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:44 smithi179 ceph-mon[43975]: 1.3 scrub starts 2024-02-09T17:39:45.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:44 smithi179 ceph-mon[43975]: 1.3 scrub ok 2024-02-09T17:39:45.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:44 smithi099 ceph-mon[39420]: 1.3 scrub starts 2024-02-09T17:39:45.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:44 smithi099 ceph-mon[39420]: 1.3 scrub ok 2024-02-09T17:39:45.249 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:39:45.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:44 smithi162 ceph-mon[44304]: 1.3 scrub starts 2024-02-09T17:39:45.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:44 smithi162 ceph-mon[44304]: 1.3 scrub ok 2024-02-09T17:39:45.605 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-09T17:39:45.945 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:45 smithi099 ceph-mon[39420]: pgmap v1119: 37 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 31 active+clean; 2.0 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 565 op/s; 5790/99282 objects misplaced (5.832%); 13 MiB/s, 206 objects/s recovering 2024-02-09T17:39:45.946 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:45 smithi099 ceph-mon[39420]: 1.13 scrub starts 2024-02-09T17:39:45.946 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:45 smithi099 ceph-mon[39420]: 1.13 scrub ok 2024-02-09T17:39:45.946 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:45 smithi099 ceph-mon[39420]: 4.5 scrub starts 2024-02-09T17:39:46.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:45 smithi179 ceph-mon[43975]: pgmap v1119: 37 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 31 active+clean; 2.0 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 565 op/s; 5790/99282 objects misplaced (5.832%); 13 MiB/s, 206 objects/s recovering 2024-02-09T17:39:46.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:45 smithi179 ceph-mon[43975]: 1.13 scrub starts 2024-02-09T17:39:46.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:45 smithi179 ceph-mon[43975]: 1.13 scrub ok 2024-02-09T17:39:46.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:45 smithi179 ceph-mon[43975]: 4.5 scrub starts 2024-02-09T17:39:46.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:45 smithi162 ceph-mon[44304]: pgmap v1119: 37 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 31 active+clean; 2.0 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 565 op/s; 5790/99282 objects misplaced (5.832%); 13 MiB/s, 206 objects/s recovering 2024-02-09T17:39:46.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:45 smithi162 ceph-mon[44304]: 1.13 scrub starts 2024-02-09T17:39:46.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:45 smithi162 ceph-mon[44304]: 1.13 scrub ok 2024-02-09T17:39:46.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:45 smithi162 ceph-mon[44304]: 4.5 scrub starts 2024-02-09T17:39:46.452 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 4 1 2024-02-09T17:39:46.972 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:39:46.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:46.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:39:47.073 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:39:47.081 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:46 smithi179 ceph-mon[43975]: 4.9 scrub starts 2024-02-09T17:39:47.081 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:46 smithi179 ceph-mon[43975]: 4.5 scrub ok 2024-02-09T17:39:47.081 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:46 smithi179 ceph-mon[43975]: 4.6 scrub starts 2024-02-09T17:39:47.081 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:46 smithi179 ceph-mon[43975]: 4.9 scrub ok 2024-02-09T17:39:47.081 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:46 smithi179 ceph-mon[43975]: osdmap e430: 12 total, 12 up, 11 in 2024-02-09T17:39:47.081 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:46 smithi179 ceph-mon[43975]: 1.5 scrub starts 2024-02-09T17:39:47.081 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:46 smithi179 ceph-mon[43975]: 1.5 scrub ok 2024-02-09T17:39:47.083 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:39:47.117 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:47.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:46 smithi099 ceph-mon[39420]: 4.9 scrub starts 2024-02-09T17:39:47.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:46 smithi099 ceph-mon[39420]: 4.5 scrub ok 2024-02-09T17:39:47.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:46 smithi099 ceph-mon[39420]: 4.6 scrub starts 2024-02-09T17:39:47.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:46 smithi099 ceph-mon[39420]: 4.9 scrub ok 2024-02-09T17:39:47.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:46 smithi099 ceph-mon[39420]: osdmap e430: 12 total, 12 up, 11 in 2024-02-09T17:39:47.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:46 smithi099 ceph-mon[39420]: 1.5 scrub starts 2024-02-09T17:39:47.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:46 smithi099 ceph-mon[39420]: 1.5 scrub ok 2024-02-09T17:39:47.330 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:39:47.371 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:39:47.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:47.379 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:39:47.385 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T17:39:47.385 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:39:47.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:46 smithi162 ceph-mon[44304]: 4.9 scrub starts 2024-02-09T17:39:47.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:46 smithi162 ceph-mon[44304]: 4.5 scrub ok 2024-02-09T17:39:47.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:46 smithi162 ceph-mon[44304]: 4.6 scrub starts 2024-02-09T17:39:47.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:46 smithi162 ceph-mon[44304]: 4.9 scrub ok 2024-02-09T17:39:47.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:46 smithi162 ceph-mon[44304]: osdmap e430: 12 total, 12 up, 11 in 2024-02-09T17:39:47.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:46 smithi162 ceph-mon[44304]: 1.5 scrub starts 2024-02-09T17:39:47.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:46 smithi162 ceph-mon[44304]: 1.5 scrub ok 2024-02-09T17:39:47.480 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:39:47.543 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:47.592 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:39:47.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:47.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:39:47.704 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:39:47.761 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:47.847 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:39:47.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:47.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:39:47.958 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:39:47.992 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:48.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:47 smithi179 ceph-mon[43975]: 4.6 scrub ok 2024-02-09T17:39:48.276 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:47 smithi179 ceph-mon[43975]: 4.a scrub starts 2024-02-09T17:39:48.276 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:47 smithi179 ceph-mon[43975]: pgmap v1121: 37 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 33 active+clean; 1.9 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 565 op/s; 4873/92301 objects misplaced (5.279%); 9.3 MiB/s, 149 objects/s recovering 2024-02-09T17:39:48.276 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:47 smithi179 ceph-mon[43975]: 1.2 scrub starts 2024-02-09T17:39:48.276 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:47 smithi179 ceph-mon[43975]: 1.2 scrub ok 2024-02-09T17:39:48.276 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:47 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:39:48.277 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:47 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:39:48.277 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:47 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:39:48.277 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:47 smithi179 ceph-mon[43975]: 4.a scrub ok 2024-02-09T17:39:48.278 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:47 smithi099 ceph-mon[39420]: 4.6 scrub ok 2024-02-09T17:39:48.278 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:47 smithi099 ceph-mon[39420]: 4.a scrub starts 2024-02-09T17:39:48.278 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:47 smithi099 ceph-mon[39420]: pgmap v1121: 37 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 33 active+clean; 1.9 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 565 op/s; 4873/92301 objects misplaced (5.279%); 9.3 MiB/s, 149 objects/s recovering 2024-02-09T17:39:48.278 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:47 smithi099 ceph-mon[39420]: 1.2 scrub starts 2024-02-09T17:39:48.278 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:47 smithi099 ceph-mon[39420]: 1.2 scrub ok 2024-02-09T17:39:48.278 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:47 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:39:48.278 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:47 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:39:48.278 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:47 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:39:48.279 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:47 smithi099 ceph-mon[39420]: 4.a scrub ok 2024-02-09T17:39:48.279 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:39:48.280 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.8 dump_historic_ops 2024-02-09T17:39:48.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:48.281 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:39:48.382 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:39:48.409 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:47 smithi162 ceph-mon[44304]: 4.6 scrub ok 2024-02-09T17:39:48.409 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:47 smithi162 ceph-mon[44304]: 4.a scrub starts 2024-02-09T17:39:48.409 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:47 smithi162 ceph-mon[44304]: pgmap v1121: 37 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 33 active+clean; 1.9 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 565 op/s; 4873/92301 objects misplaced (5.279%); 9.3 MiB/s, 149 objects/s recovering 2024-02-09T17:39:48.409 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:47 smithi162 ceph-mon[44304]: 1.2 scrub starts 2024-02-09T17:39:48.409 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:47 smithi162 ceph-mon[44304]: 1.2 scrub ok 2024-02-09T17:39:48.409 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:47 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:39:48.409 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:47 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:39:48.409 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:47 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:39:48.409 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:47 smithi162 ceph-mon[44304]: 4.a scrub ok 2024-02-09T17:39:48.411 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:48.440 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:39:48.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:48.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:39:48.547 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:39:48.579 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:48.604 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:39:48.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:48.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:39:48.690 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:39:48.707 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:39:48.735 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:48.760 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:39:48.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:48.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:39:48.864 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:39:48.893 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:48.913 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:39:48.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:48.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:39:49.021 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:39:49.088 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:49.666 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:39:49.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:49.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:39:49.755 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:39:49.773 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:39:49.801 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:39:49.826 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:49.838 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:39:49.884 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:39:49.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:49.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:39:49.980 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:49 smithi179 ceph-mon[43975]: 1.e scrub starts 2024-02-09T17:39:49.980 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:49 smithi179 ceph-mon[43975]: 1.e scrub ok 2024-02-09T17:39:49.980 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:49 smithi179 ceph-mon[43975]: pgmap v1122: 37 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 33 active+clean; 1.9 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 512 op/s; 4873/90951 objects misplaced (5.358%); 7.4 MiB/s, 119 objects/s recovering 2024-02-09T17:39:49.980 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:49 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/749821084' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:39:49.991 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:39:50.026 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:50.051 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:39:50.054 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:50.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:39:50.077 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:39:50.115 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:39:50.154 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:39:50.163 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:39:50.196 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:39:50.214 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:50.234 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:39:50.255 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:49 smithi099 ceph-mon[39420]: 1.e scrub starts 2024-02-09T17:39:50.255 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:49 smithi099 ceph-mon[39420]: 1.e scrub ok 2024-02-09T17:39:50.255 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:49 smithi099 ceph-mon[39420]: pgmap v1122: 37 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 33 active+clean; 1.9 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 512 op/s; 4873/90951 objects misplaced (5.358%); 7.4 MiB/s, 119 objects/s recovering 2024-02-09T17:39:50.255 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:49 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/749821084' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:39:50.272 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:39:50.322 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:39:50.374 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:39:50.411 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:39:50.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:49 smithi162 ceph-mon[44304]: 1.e scrub starts 2024-02-09T17:39:50.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:49 smithi162 ceph-mon[44304]: 1.e scrub ok 2024-02-09T17:39:50.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:49 smithi162 ceph-mon[44304]: pgmap v1122: 37 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 33 active+clean; 1.9 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 512 op/s; 4873/90951 objects misplaced (5.358%); 7.4 MiB/s, 119 objects/s recovering 2024-02-09T17:39:50.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:49 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/749821084' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:39:50.456 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:39:50.456 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:39:50.457 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:39:50.457 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:39:50.457 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:39:50.457 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:39:50.457 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:39:50.457 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:39:50.457 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:39:50.457 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:39:50.457 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:39:50.457 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:39:50.466 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:39:50.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:50.469 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:39:50.571 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:39:50.636 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:51.028 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.4 to 1 (10000) 2024-02-09T17:39:51.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:50 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3409591509' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-09T17:39:51.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:50 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-09T17:39:51.196 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:39:51.196 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-02-09T17:39:51.196 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-02-09T17:39:51.197 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-02-09T17:39:51.197 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.197 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24817.0:71091 3.9 3:9632d412:::benchmark_data_smithi179_90440_object71090:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-02-09T17:39:51.197 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:30:39.845879+0000", 2024-02-09T17:39:51.197 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 551.34296473100005, 2024-02-09T17:39:51.197 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.085269534999999994, 2024-02-09T17:39:51.197 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:51.197 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:51.197 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:51.197 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24817", 2024-02-09T17:39:51.197 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:39:51.197 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 71091 2024-02-09T17:39:51.198 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.198 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:51.198 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.198 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:51.198 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:39.845879+0000", 2024-02-09T17:39:51.198 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:51.198 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.198 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.198 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:51.198 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:39.845879+0000", 2024-02-09T17:39:51.198 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:51.198 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.198 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.199 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:51.199 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:39.845881+0000", 2024-02-09T17:39:51.199 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7880000000000001e-06 2024-02-09T17:39:51.199 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.199 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.199 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:51.199 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:39.845901+0000", 2024-02-09T17:39:51.199 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0378999999999999e-05 2024-02-09T17:39:51.199 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.199 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.199 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:51.199 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:39.845916+0000", 2024-02-09T17:39:51.200 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5468000000000002e-05 2024-02-09T17:39:51.200 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.200 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.200 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:51.200 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:39.845924+0000", 2024-02-09T17:39:51.200 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.9999999999999996e-06 2024-02-09T17:39:51.200 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.200 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.200 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:51.200 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:39.846000+0000", 2024-02-09T17:39:51.200 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5995000000000007e-05 2024-02-09T17:39:51.201 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.201 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.201 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:51.201 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:39.846125+0000", 2024-02-09T17:39:51.201 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012463700000000001 2024-02-09T17:39:51.201 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.201 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.201 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 7,11", 2024-02-09T17:39:51.201 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:39.846259+0000", 2024-02-09T17:39:51.201 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013358000000000001 2024-02-09T17:39:51.201 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.202 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.202 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:51.202 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:39.850039+0000", 2024-02-09T17:39:51.202 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0037806010000000002 2024-02-09T17:39:51.202 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.202 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.202 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:51.202 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:39.850359+0000", 2024-02-09T17:39:51.202 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00031984800000000002 2024-02-09T17:39:51.202 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.202 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.202 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:51.203 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:39.931028+0000", 2024-02-09T17:39:51.203 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.080669414999999994 2024-02-09T17:39:51.203 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.203 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.203 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:51.203 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:39.931104+0000", 2024-02-09T17:39:51.203 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5834000000000006e-05 2024-02-09T17:39:51.203 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.203 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.203 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:51.204 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:39.931148+0000", 2024-02-09T17:39:51.204 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3989999999999997e-05 2024-02-09T17:39:51.204 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:51.204 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:51.204 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:51.204 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.204 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.204 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24817.0:71513 3.9 3:96d309dc:::benchmark_data_smithi179_90440_object71512:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-02-09T17:39:51.204 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:30:40.163268+0000", 2024-02-09T17:39:51.204 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 551.02557514199998, 2024-02-09T17:39:51.204 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.071249064000000001, 2024-02-09T17:39:51.205 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:51.205 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:51.205 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:51.205 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24817", 2024-02-09T17:39:51.205 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:39:51.205 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 71513 2024-02-09T17:39:51.205 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.205 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:51.205 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.205 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:51.205 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:40.163268+0000", 2024-02-09T17:39:51.206 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:51.206 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.206 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.206 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:51.206 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:40.163268+0000", 2024-02-09T17:39:51.206 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:51.206 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.206 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.206 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:51.206 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:40.163270+0000", 2024-02-09T17:39:51.207 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2929999999999999e-06 2024-02-09T17:39:51.207 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.207 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.207 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:51.207 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:40.163289+0000", 2024-02-09T17:39:51.207 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9653999999999999e-05 2024-02-09T17:39:51.207 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.207 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.207 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:51.207 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:40.163302+0000", 2024-02-09T17:39:51.207 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2408999999999999e-05 2024-02-09T17:39:51.207 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.208 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.208 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:51.208 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:40.163308+0000", 2024-02-09T17:39:51.208 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6660000000000002e-06 2024-02-09T17:39:51.208 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.208 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.208 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:51.208 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:40.163364+0000", 2024-02-09T17:39:51.208 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5726000000000001e-05 2024-02-09T17:39:51.208 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.208 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.209 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:51.209 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:40.163533+0000", 2024-02-09T17:39:51.209 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016900899999999999 2024-02-09T17:39:51.209 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.209 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.209 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 7,11", 2024-02-09T17:39:51.209 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:40.163715+0000", 2024-02-09T17:39:51.209 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00018148400000000001 2024-02-09T17:39:51.210 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.210 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.210 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:51.210 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:40.167068+0000", 2024-02-09T17:39:51.210 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0033529940000000002 2024-02-09T17:39:51.210 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.210 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.210 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:51.210 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:40.167464+0000", 2024-02-09T17:39:51.210 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00039671699999999999 2024-02-09T17:39:51.210 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.210 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.211 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:51.211 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:40.234405+0000", 2024-02-09T17:39:51.211 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.066940816 2024-02-09T17:39:51.211 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.211 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.211 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:51.211 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:40.234466+0000", 2024-02-09T17:39:51.211 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0822999999999997e-05 2024-02-09T17:39:51.211 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.211 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.211 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:51.211 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:40.234517+0000", 2024-02-09T17:39:51.212 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1473e-05 2024-02-09T17:39:51.212 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:51.212 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:51.212 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:51.212 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.212 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.212 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24817.0:71526 3.9 3:96597252:::benchmark_data_smithi179_90440_object71525:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-02-09T17:39:51.212 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:30:40.174694+0000", 2024-02-09T17:39:51.212 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 551.01414956899998, 2024-02-09T17:39:51.212 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.069582065999999998, 2024-02-09T17:39:51.212 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:51.213 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:51.213 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:51.213 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24817", 2024-02-09T17:39:51.213 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:39:51.213 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 71526 2024-02-09T17:39:51.213 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.213 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:51.213 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.213 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:51.213 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:40.174694+0000", 2024-02-09T17:39:51.213 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:51.214 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.214 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.214 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:51.214 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:40.174694+0000", 2024-02-09T17:39:51.214 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:51.214 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.214 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.214 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:51.214 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:40.174696+0000", 2024-02-09T17:39:51.214 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6130000000000001e-06 2024-02-09T17:39:51.214 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.215 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.215 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:51.215 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:40.174715+0000", 2024-02-09T17:39:51.215 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9788000000000001e-05 2024-02-09T17:39:51.215 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.215 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.215 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:51.215 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:40.174726+0000", 2024-02-09T17:39:51.215 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1031e-05 2024-02-09T17:39:51.215 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.215 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.215 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:51.215 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:40.174732+0000", 2024-02-09T17:39:51.216 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1759999999999998e-06 2024-02-09T17:39:51.216 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.216 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.216 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:51.216 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:40.174779+0000", 2024-02-09T17:39:51.216 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6168999999999998e-05 2024-02-09T17:39:51.216 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.216 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.216 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:51.216 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:40.174890+0000", 2024-02-09T17:39:51.216 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000111116 2024-02-09T17:39:51.217 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.217 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.217 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 7,11", 2024-02-09T17:39:51.217 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:40.174993+0000", 2024-02-09T17:39:51.217 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000103014 2024-02-09T17:39:51.217 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.217 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.217 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:51.217 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:40.176986+0000", 2024-02-09T17:39:51.217 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0019928839999999999 2024-02-09T17:39:51.217 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.218 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.218 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:51.218 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:40.177678+0000", 2024-02-09T17:39:51.218 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00069230500000000005 2024-02-09T17:39:51.218 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.218 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.218 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:51.218 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:40.244196+0000", 2024-02-09T17:39:51.218 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.066517580000000007 2024-02-09T17:39:51.218 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.218 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.218 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:51.218 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:40.244237+0000", 2024-02-09T17:39:51.219 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1826000000000002e-05 2024-02-09T17:39:51.219 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.219 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.219 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:51.219 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:30:40.244276+0000", 2024-02-09T17:39:51.219 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8563999999999999e-05 2024-02-09T17:39:51.219 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:51.219 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:51.219 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:51.219 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.219 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.219 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDPGPush(4.2 382/380 [PushOp(4:40fb736c:::benchmark_data_smithi179_103809_object104074:head, version: 347'6425, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:40fb736c:::benchmark_data_smithi179_103809_object104074:head@347'6425, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-02-09T17:39:51.220 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:37.483000+0000", 2024-02-09T17:39:51.220 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 73.705843578, 2024-02-09T17:39:51.220 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.040765556000000001, 2024-02-09T17:39:51.220 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:51.220 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-02-09T17:39:51.220 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:51.220 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.220 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:51.220 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483000+0000", 2024-02-09T17:39:51.220 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:51.220 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.220 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.221 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:51.221 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483000+0000", 2024-02-09T17:39:51.221 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:51.221 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.221 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.221 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:51.221 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483001+0000", 2024-02-09T17:39:51.221 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.9000000000000005e-07 2024-02-09T17:39:51.221 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.221 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.221 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:51.221 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483096+0000", 2024-02-09T17:39:51.221 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5166000000000003e-05 2024-02-09T17:39:51.221 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.222 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.222 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:51.222 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483111+0000", 2024-02-09T17:39:51.222 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5242000000000001e-05 2024-02-09T17:39:51.222 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.222 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.222 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:51.222 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483124+0000", 2024-02-09T17:39:51.222 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2880999999999999e-05 2024-02-09T17:39:51.222 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.222 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.222 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:51.222 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.523546+0000", 2024-02-09T17:39:51.222 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.040421603 2024-02-09T17:39:51.223 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.223 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.223 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:51.223 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.523561+0000", 2024-02-09T17:39:51.223 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5310000000000001e-05 2024-02-09T17:39:51.223 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.223 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.223 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:51.223 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.523765+0000", 2024-02-09T17:39:51.223 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00020436400000000001 2024-02-09T17:39:51.223 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:51.223 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:51.223 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:51.223 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.224 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.224 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.25201.0:209208 4.2 e382/380 4:4d9658b8:::benchmark_data_smithi179_103809_object71742:head v 382'12944, mlcod=382'12942)", 2024-02-09T17:39:51.224 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:37.483160+0000", 2024-02-09T17:39:51.224 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 73.705683375999996, 2024-02-09T17:39:51.224 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.040098823999999998, 2024-02-09T17:39:51.224 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:51.224 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:51.224 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:51.224 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.224 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:51.224 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483160+0000", 2024-02-09T17:39:51.224 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:51.224 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.225 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.225 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:51.225 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483160+0000", 2024-02-09T17:39:51.225 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:51.225 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.225 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.225 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:51.225 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483161+0000", 2024-02-09T17:39:51.225 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7e-07 2024-02-09T17:39:51.225 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.225 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.225 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:51.225 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483165+0000", 2024-02-09T17:39:51.225 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6870000000000002e-06 2024-02-09T17:39:51.225 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.226 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.226 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:51.226 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483174+0000", 2024-02-09T17:39:51.226 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7700000000000007e-06 2024-02-09T17:39:51.226 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.226 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.226 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:51.226 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483180+0000", 2024-02-09T17:39:51.226 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2009999999999996e-06 2024-02-09T17:39:51.226 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.226 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.226 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:51.226 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483428+0000", 2024-02-09T17:39:51.226 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000247803 2024-02-09T17:39:51.226 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.226 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.227 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:51.227 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483454+0000", 2024-02-09T17:39:51.227 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6291e-05 2024-02-09T17:39:51.227 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.227 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.227 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:51.227 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.523215+0000", 2024-02-09T17:39:51.227 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.039760223999999997 2024-02-09T17:39:51.227 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.227 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.227 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:51.227 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.523259+0000", 2024-02-09T17:39:51.227 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4378000000000001e-05 2024-02-09T17:39:51.227 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:51.228 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:51.228 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:51.228 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.228 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.228 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.25201.0:209216 4.2 e382/380 4:4b5c16ca:::benchmark_data_smithi179_103809_object71750:head v 382'12945, mlcod=382'12942)", 2024-02-09T17:39:51.228 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:37.483190+0000", 2024-02-09T17:39:51.228 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 73.705653249999997, 2024-02-09T17:39:51.228 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.040088927000000003, 2024-02-09T17:39:51.228 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:51.228 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:51.228 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:51.228 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.228 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:51.229 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483190+0000", 2024-02-09T17:39:51.229 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:51.229 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.229 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.229 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:51.229 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483190+0000", 2024-02-09T17:39:51.229 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:51.229 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.229 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.229 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:51.229 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483190+0000", 2024-02-09T17:39:51.229 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.48e-07 2024-02-09T17:39:51.230 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.230 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.230 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:51.230 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483193+0000", 2024-02-09T17:39:51.230 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6280000000000001e-06 2024-02-09T17:39:51.230 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.230 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.230 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:51.230 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483198+0000", 2024-02-09T17:39:51.230 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9869999999999999e-06 2024-02-09T17:39:51.230 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.231 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.231 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:51.231 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483201+0000", 2024-02-09T17:39:51.231 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0479999999999999e-06 2024-02-09T17:39:51.231 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.231 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.231 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:51.231 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.486468+0000", 2024-02-09T17:39:51.231 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00326699 2024-02-09T17:39:51.231 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.232 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.232 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:51.232 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.486488+0000", 2024-02-09T17:39:51.232 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0023000000000001e-05 2024-02-09T17:39:51.232 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.232 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.232 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:51.232 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.523262+0000", 2024-02-09T17:39:51.232 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.036774284999999997 2024-02-09T17:39:51.232 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.232 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.233 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:51.233 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.523279+0000", 2024-02-09T17:39:51.233 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6718000000000002e-05 2024-02-09T17:39:51.233 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:51.233 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:51.233 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:51.233 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.233 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.233 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.25201.0:209219 4.2 e382/380 4:4d2cfdf8:::benchmark_data_smithi179_103809_object71753:head v 382'12946, mlcod=382'12942)", 2024-02-09T17:39:51.233 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:37.483206+0000", 2024-02-09T17:39:51.233 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 73.705637894000006, 2024-02-09T17:39:51.234 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.040090200999999999, 2024-02-09T17:39:51.234 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:51.234 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:51.234 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:51.234 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.234 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:51.234 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483206+0000", 2024-02-09T17:39:51.234 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:51.234 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.234 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.235 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:51.235 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483206+0000", 2024-02-09T17:39:51.235 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:51.235 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.235 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.235 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:51.235 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483206+0000", 2024-02-09T17:39:51.235 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.73e-07 2024-02-09T17:39:51.235 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.235 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.235 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:51.236 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483211+0000", 2024-02-09T17:39:51.236 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2979999999999999e-06 2024-02-09T17:39:51.236 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.236 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.236 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:51.236 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483218+0000", 2024-02-09T17:39:51.236 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4749999999999998e-06 2024-02-09T17:39:51.236 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.236 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.236 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:51.237 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483222+0000", 2024-02-09T17:39:51.237 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4880000000000001e-06 2024-02-09T17:39:51.237 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.237 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.237 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:51.237 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.490100+0000", 2024-02-09T17:39:51.237 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0068782519999999996 2024-02-09T17:39:51.237 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.237 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.237 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:51.237 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.490121+0000", 2024-02-09T17:39:51.238 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0607000000000001e-05 2024-02-09T17:39:51.238 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.238 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.238 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:51.238 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.523282+0000", 2024-02-09T17:39:51.238 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.033160927 2024-02-09T17:39:51.238 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.238 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.238 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:51.238 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.523296+0000", 2024-02-09T17:39:51.239 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3981e-05 2024-02-09T17:39:51.239 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:51.239 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:51.239 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:51.239 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.239 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.239 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.25201.0:209236 4.2 e382/380 4:4fc96872:::benchmark_data_smithi179_103809_object71770:head v 382'12947, mlcod=382'12942)", 2024-02-09T17:39:51.239 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:37.483229+0000", 2024-02-09T17:39:51.239 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 73.705614893000003, 2024-02-09T17:39:51.239 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.040085010999999997, 2024-02-09T17:39:51.239 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:51.239 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:51.239 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:51.239 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.240 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:51.240 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483229+0000", 2024-02-09T17:39:51.240 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:51.240 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.240 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.240 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:51.240 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483229+0000", 2024-02-09T17:39:51.240 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:51.240 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.240 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.240 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:51.240 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483229+0000", 2024-02-09T17:39:51.241 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1899999999999999e-07 2024-02-09T17:39:51.241 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.241 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.241 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:51.241 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483235+0000", 2024-02-09T17:39:51.241 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6100000000000002e-06 2024-02-09T17:39:51.241 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.241 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.241 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:51.241 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483242+0000", 2024-02-09T17:39:51.241 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.0060000000000003e-06 2024-02-09T17:39:51.241 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.241 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.242 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:51.242 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483247+0000", 2024-02-09T17:39:51.242 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7539999999999997e-06 2024-02-09T17:39:51.242 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.242 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.242 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:51.242 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.493265+0000", 2024-02-09T17:39:51.242 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010017979 2024-02-09T17:39:51.242 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.242 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.242 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:51.242 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.493296+0000", 2024-02-09T17:39:51.243 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0490000000000001e-05 2024-02-09T17:39:51.243 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.243 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.243 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:51.243 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.523298+0000", 2024-02-09T17:39:51.243 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.030002267999999999 2024-02-09T17:39:51.243 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.243 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.243 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:51.243 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.523314+0000", 2024-02-09T17:39:51.243 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5685e-05 2024-02-09T17:39:51.244 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:51.244 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:51.244 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:51.244 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.244 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.244 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.25201.0:209275 4.2 e382/380 4:4c202b7a:::benchmark_data_smithi179_103809_object71809:head v 382'12950, mlcod=382'12942)", 2024-02-09T17:39:51.244 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:37.483281+0000", 2024-02-09T17:39:51.244 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 73.705562744000005, 2024-02-09T17:39:51.244 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.040079798999999999, 2024-02-09T17:39:51.244 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:51.245 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:51.245 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:51.245 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.245 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:51.245 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483281+0000", 2024-02-09T17:39:51.245 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:51.245 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.245 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.245 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:51.245 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483281+0000", 2024-02-09T17:39:51.245 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:51.245 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.246 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.246 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:51.246 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483281+0000", 2024-02-09T17:39:51.246 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.11e-07 2024-02-09T17:39:51.246 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.246 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.246 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:51.246 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483285+0000", 2024-02-09T17:39:51.246 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.844e-06 2024-02-09T17:39:51.246 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.246 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.246 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:51.247 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483289+0000", 2024-02-09T17:39:51.247 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0400000000000003e-06 2024-02-09T17:39:51.247 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.247 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.247 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:51.247 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483291+0000", 2024-02-09T17:39:51.247 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3269999999999999e-06 2024-02-09T17:39:51.247 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.247 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.247 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:51.247 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.503008+0000", 2024-02-09T17:39:51.247 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.019716656999999999 2024-02-09T17:39:51.248 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.248 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.248 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:51.248 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.503033+0000", 2024-02-09T17:39:51.248 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4961999999999999e-05 2024-02-09T17:39:51.248 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.248 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.248 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:51.248 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.523347+0000", 2024-02-09T17:39:51.248 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.020314536000000001 2024-02-09T17:39:51.248 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.249 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.249 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:51.249 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.523361+0000", 2024-02-09T17:39:51.249 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3322000000000001e-05 2024-02-09T17:39:51.249 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:51.249 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:51.249 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:51.249 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.249 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.249 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.25201.0:209284 4.2 e382/380 4:47bb47a9:::benchmark_data_smithi179_103809_object71818:head v 382'12951, mlcod=382'12942)", 2024-02-09T17:39:51.249 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:37.483294+0000", 2024-02-09T17:39:51.250 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 73.705549059999996, 2024-02-09T17:39:51.250 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.040081492000000003, 2024-02-09T17:39:51.250 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:51.250 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:51.250 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:51.250 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.250 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:51.250 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483294+0000", 2024-02-09T17:39:51.250 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:51.250 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.250 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.250 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:51.251 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483294+0000", 2024-02-09T17:39:51.251 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:51.251 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.251 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.251 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:51.251 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483295+0000", 2024-02-09T17:39:51.251 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2100000000000001e-07 2024-02-09T17:39:51.251 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.251 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.251 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:51.251 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483297+0000", 2024-02-09T17:39:51.251 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6259999999999999e-06 2024-02-09T17:39:51.251 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.252 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.252 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:51.252 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483302+0000", 2024-02-09T17:39:51.252 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.702e-06 2024-02-09T17:39:51.252 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.252 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.252 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:51.252 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483306+0000", 2024-02-09T17:39:51.252 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9060000000000004e-06 2024-02-09T17:39:51.252 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.252 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.252 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:51.252 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.506214+0000", 2024-02-09T17:39:51.253 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.02290828 2024-02-09T17:39:51.253 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.253 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.253 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:51.253 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.506239+0000", 2024-02-09T17:39:51.253 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5068000000000001e-05 2024-02-09T17:39:51.253 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.253 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.253 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:51.253 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.523363+0000", 2024-02-09T17:39:51.253 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017123482999999998 2024-02-09T17:39:51.253 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.253 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.254 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:51.254 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.523376+0000", 2024-02-09T17:39:51.254 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3305999999999999e-05 2024-02-09T17:39:51.254 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:51.254 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:51.254 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:51.254 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.254 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.254 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.25201.0:209298 4.2 e382/380 4:420a02f2:::benchmark_data_smithi179_103809_object71832:head v 382'12952, mlcod=382'12942)", 2024-02-09T17:39:51.254 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:37.483311+0000", 2024-02-09T17:39:51.254 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 73.705532172000005, 2024-02-09T17:39:51.254 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.040080776999999998, 2024-02-09T17:39:51.254 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:51.254 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:51.255 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:51.255 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.255 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:51.255 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483311+0000", 2024-02-09T17:39:51.255 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:51.255 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.255 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.255 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:51.255 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483311+0000", 2024-02-09T17:39:51.255 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:51.255 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.255 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.255 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:51.256 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483312+0000", 2024-02-09T17:39:51.256 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6800000000000002e-07 2024-02-09T17:39:51.256 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.256 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.256 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:51.256 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483315+0000", 2024-02-09T17:39:51.256 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4790000000000001e-06 2024-02-09T17:39:51.256 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.256 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.256 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:51.256 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483322+0000", 2024-02-09T17:39:51.256 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6499999999999999e-06 2024-02-09T17:39:51.256 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.257 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.257 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:51.257 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483326+0000", 2024-02-09T17:39:51.257 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5549999999999996e-06 2024-02-09T17:39:51.257 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.257 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.257 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:51.257 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.509673+0000", 2024-02-09T17:39:51.257 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.026346815999999999 2024-02-09T17:39:51.257 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.257 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.257 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:51.257 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.509698+0000", 2024-02-09T17:39:51.257 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4933999999999999e-05 2024-02-09T17:39:51.258 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.258 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.258 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:51.258 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.523379+0000", 2024-02-09T17:39:51.258 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013680661 2024-02-09T17:39:51.258 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.258 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.258 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:51.258 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.523392+0000", 2024-02-09T17:39:51.258 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3414000000000001e-05 2024-02-09T17:39:51.258 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:51.258 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:51.258 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:51.258 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.259 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.259 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.25201.0:209300 4.2 e382/380 4:4a042078:::benchmark_data_smithi179_103809_object71834:head v 382'12953, mlcod=382'12942)", 2024-02-09T17:39:51.259 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:37.483332+0000", 2024-02-09T17:39:51.259 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 73.705511082000001, 2024-02-09T17:39:51.259 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.040077496999999997, 2024-02-09T17:39:51.259 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:51.259 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:51.259 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:51.259 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.259 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:51.259 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483332+0000", 2024-02-09T17:39:51.259 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:51.259 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.260 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.260 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:51.260 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483332+0000", 2024-02-09T17:39:51.260 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:51.260 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.260 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.260 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:51.260 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483333+0000", 2024-02-09T17:39:51.260 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.16e-07 2024-02-09T17:39:51.260 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.260 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.260 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:51.260 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483339+0000", 2024-02-09T17:39:51.261 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0530000000000003e-06 2024-02-09T17:39:51.261 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.261 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.261 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:51.261 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483345+0000", 2024-02-09T17:39:51.261 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7360000000000004e-06 2024-02-09T17:39:51.261 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.261 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.261 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:51.261 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.483350+0000", 2024-02-09T17:39:51.261 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6399999999999996e-06 2024-02-09T17:39:51.261 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.262 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.262 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:51.262 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.513077+0000", 2024-02-09T17:39:51.262 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.029727408 2024-02-09T17:39:51.262 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.262 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.262 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:51.262 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.513099+0000", 2024-02-09T17:39:51.262 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1256000000000001e-05 2024-02-09T17:39:51.262 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.262 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.262 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:51.263 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.523396+0000", 2024-02-09T17:39:51.263 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010297028 2024-02-09T17:39:51.263 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.263 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.263 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:51.263 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.523410+0000", 2024-02-09T17:39:51.263 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.416e-05 2024-02-09T17:39:51.263 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:51.263 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:51.263 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:51.263 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.263 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.264 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:232414 4.0 4:062b8f20:::benchmark_data_smithi179_103809_object94948:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e413)", 2024-02-09T17:39:51.264 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:39:22.735659+0000", 2024-02-09T17:39:51.264 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 28.453184937, 2024-02-09T17:39:51.264 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.85919624400000005, 2024-02-09T17:39:51.264 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:51.264 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-02-09T17:39:51.264 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:51.264 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:51.264 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:51.264 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 232414 2024-02-09T17:39:51.264 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.264 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:51.265 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.265 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:51.265 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735659+0000", 2024-02-09T17:39:51.265 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:51.265 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.265 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.265 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:51.265 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735659+0000", 2024-02-09T17:39:51.265 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:51.265 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.265 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.266 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:51.266 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735660+0000", 2024-02-09T17:39:51.266 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.279e-06 2024-02-09T17:39:51.266 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.266 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.266 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:51.266 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735664+0000", 2024-02-09T17:39:51.266 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2010000000000001e-06 2024-02-09T17:39:51.266 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.266 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.266 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:51.266 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735674+0000", 2024-02-09T17:39:51.267 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6360000000000006e-06 2024-02-09T17:39:51.267 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.267 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.267 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:51.267 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735682+0000", 2024-02-09T17:39:51.267 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.541e-06 2024-02-09T17:39:51.267 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.267 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.267 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:51.267 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735749+0000", 2024-02-09T17:39:51.267 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6737999999999997e-05 2024-02-09T17:39:51.268 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.268 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.268 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:39:51.268 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735760+0000", 2024-02-09T17:39:51.268 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.096e-05 2024-02-09T17:39:51.268 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.268 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.268 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:51.268 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.594762+0000", 2024-02-09T17:39:51.268 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.85900255199999997 2024-02-09T17:39:51.268 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.269 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.269 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:51.269 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.594855+0000", 2024-02-09T17:39:51.269 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2336999999999994e-05 2024-02-09T17:39:51.269 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:51.269 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:51.269 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:51.269 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.269 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.269 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:232443 4.0 4:06265949:::benchmark_data_smithi179_103809_object94977:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-02-09T17:39:51.269 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:39:22.735704+0000", 2024-02-09T17:39:51.270 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 28.453139153999999, 2024-02-09T17:39:51.270 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.87793835899999995, 2024-02-09T17:39:51.270 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:51.270 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:51.270 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:51.270 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:51.270 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:51.270 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 232443 2024-02-09T17:39:51.270 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.270 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:51.270 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.271 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:51.271 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735704+0000", 2024-02-09T17:39:51.271 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:51.271 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.271 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.271 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:51.271 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735704+0000", 2024-02-09T17:39:51.271 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:51.271 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.271 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.271 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:51.272 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735705+0000", 2024-02-09T17:39:51.272 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1300000000000001e-07 2024-02-09T17:39:51.272 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.272 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.272 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:51.272 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735707+0000", 2024-02-09T17:39:51.272 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0600000000000002e-06 2024-02-09T17:39:51.272 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.272 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.272 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:51.272 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735713+0000", 2024-02-09T17:39:51.273 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8449999999999999e-06 2024-02-09T17:39:51.273 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.273 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.273 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:51.273 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735719+0000", 2024-02-09T17:39:51.273 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0410000000000003e-06 2024-02-09T17:39:51.273 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.273 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.273 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:51.273 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735792+0000", 2024-02-09T17:39:51.274 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.3719000000000003e-05 2024-02-09T17:39:51.274 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.274 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.274 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:39:51.274 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735798+0000", 2024-02-09T17:39:51.274 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7860000000000002e-06 2024-02-09T17:39:51.274 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.274 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.274 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:51.274 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.595333+0000", 2024-02-09T17:39:51.275 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.85953531500000002 2024-02-09T17:39:51.275 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.275 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.275 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:51.275 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.595449+0000", 2024-02-09T17:39:51.275 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011531600000000001 2024-02-09T17:39:51.275 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.275 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.275 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,9", 2024-02-09T17:39:51.275 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.595565+0000", 2024-02-09T17:39:51.275 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011670100000000001 2024-02-09T17:39:51.275 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.276 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.276 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:51.276 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.603480+0000", 2024-02-09T17:39:51.276 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0079149809999999998 2024-02-09T17:39:51.276 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.276 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.276 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:51.276 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.613425+0000", 2024-02-09T17:39:51.276 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0099445639999999995 2024-02-09T17:39:51.276 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.277 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.277 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:51.277 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.613547+0000", 2024-02-09T17:39:51.277 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012220300000000001 2024-02-09T17:39:51.277 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.277 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.277 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:51.277 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.613607+0000", 2024-02-09T17:39:51.277 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9687e-05 2024-02-09T17:39:51.277 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.277 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.278 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:51.278 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.613643+0000", 2024-02-09T17:39:51.278 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.5728000000000001e-05 2024-02-09T17:39:51.278 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:51.278 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:51.278 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:51.278 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.278 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.278 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:232449 4.0 4:0d4bf3f6:::benchmark_data_smithi179_103809_object94983:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-02-09T17:39:51.278 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:39:22.735743+0000", 2024-02-09T17:39:51.278 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 28.453100854999999, 2024-02-09T17:39:51.279 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.87819052500000006, 2024-02-09T17:39:51.279 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:51.279 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:51.279 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:51.279 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:51.279 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:51.279 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 232449 2024-02-09T17:39:51.279 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.279 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:51.279 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.279 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:51.279 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735743+0000", 2024-02-09T17:39:51.280 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:51.280 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.280 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.280 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:51.280 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735743+0000", 2024-02-09T17:39:51.280 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:51.280 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.280 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.280 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:51.280 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735743+0000", 2024-02-09T17:39:51.280 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0500000000000004e-07 2024-02-09T17:39:51.280 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.280 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.281 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:51.281 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735746+0000", 2024-02-09T17:39:51.281 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4710000000000001e-06 2024-02-09T17:39:51.281 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.281 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.281 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:51.281 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735753+0000", 2024-02-09T17:39:51.281 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.097e-06 2024-02-09T17:39:51.281 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.281 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.281 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:51.281 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735761+0000", 2024-02-09T17:39:51.281 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5860000000000005e-06 2024-02-09T17:39:51.281 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.282 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.282 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:51.282 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735823+0000", 2024-02-09T17:39:51.282 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1563000000000007e-05 2024-02-09T17:39:51.282 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.282 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.282 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:39:51.282 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735830+0000", 2024-02-09T17:39:51.282 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5959999999999998e-06 2024-02-09T17:39:51.282 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.282 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.282 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:51.282 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.599236+0000", 2024-02-09T17:39:51.282 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.86340511399999997 2024-02-09T17:39:51.283 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.283 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.283 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:51.283 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.599367+0000", 2024-02-09T17:39:51.283 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013145100000000001 2024-02-09T17:39:51.283 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.283 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.283 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,9", 2024-02-09T17:39:51.283 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.599476+0000", 2024-02-09T17:39:51.283 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000108902 2024-02-09T17:39:51.283 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.283 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.284 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:51.284 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.613658+0000", 2024-02-09T17:39:51.284 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014181808000000001 2024-02-09T17:39:51.284 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.284 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.284 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:51.284 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.613714+0000", 2024-02-09T17:39:51.284 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6283000000000003e-05 2024-02-09T17:39:51.284 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.284 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.284 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:51.284 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.613840+0000", 2024-02-09T17:39:51.284 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000126248 2024-02-09T17:39:51.285 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.285 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.285 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:51.285 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.613895+0000", 2024-02-09T17:39:51.285 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4435000000000003e-05 2024-02-09T17:39:51.285 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.285 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.285 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:51.285 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.613933+0000", 2024-02-09T17:39:51.285 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8466000000000001e-05 2024-02-09T17:39:51.285 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:51.285 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:51.285 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:51.285 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.286 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.286 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:232450 4.0 4:0d2c10a7:::benchmark_data_smithi179_103809_object94984:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-02-09T17:39:51.286 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:39:22.735785+0000", 2024-02-09T17:39:51.286 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 28.453058618, 2024-02-09T17:39:51.286 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.88694171700000002, 2024-02-09T17:39:51.286 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:51.286 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:51.286 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:51.286 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:51.286 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:51.287 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 232450 2024-02-09T17:39:51.287 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.287 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:51.287 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.287 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:51.287 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735785+0000", 2024-02-09T17:39:51.287 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:51.287 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.287 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.287 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:51.287 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735785+0000", 2024-02-09T17:39:51.287 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:51.287 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.287 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.288 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:51.288 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735785+0000", 2024-02-09T17:39:51.288 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1600000000000001e-07 2024-02-09T17:39:51.288 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.288 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.288 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:51.288 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735788+0000", 2024-02-09T17:39:51.288 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5320000000000001e-06 2024-02-09T17:39:51.288 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.288 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.288 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:51.288 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735795+0000", 2024-02-09T17:39:51.288 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.0770000000000002e-06 2024-02-09T17:39:51.289 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.289 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.289 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:51.289 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735801+0000", 2024-02-09T17:39:51.289 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7930000000000003e-06 2024-02-09T17:39:51.289 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.289 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.289 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:51.289 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735849+0000", 2024-02-09T17:39:51.289 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8727000000000003e-05 2024-02-09T17:39:51.289 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.289 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.290 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:39:51.290 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735853+0000", 2024-02-09T17:39:51.290 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0040000000000001e-06 2024-02-09T17:39:51.290 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.290 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.290 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:51.290 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.603542+0000", 2024-02-09T17:39:51.290 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.86768888 2024-02-09T17:39:51.290 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.290 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.290 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:51.291 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.603668+0000", 2024-02-09T17:39:51.291 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000125714 2024-02-09T17:39:51.291 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.291 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.291 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,9", 2024-02-09T17:39:51.291 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.603811+0000", 2024-02-09T17:39:51.291 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014338799999999999 2024-02-09T17:39:51.291 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.291 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.291 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:51.291 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.613664+0000", 2024-02-09T17:39:51.291 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0098526250000000003 2024-02-09T17:39:51.291 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.292 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.292 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:51.292 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.613770+0000", 2024-02-09T17:39:51.292 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000106165 2024-02-09T17:39:51.292 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.292 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.292 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:51.292 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.622628+0000", 2024-02-09T17:39:51.292 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0088578870000000001 2024-02-09T17:39:51.292 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.292 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.293 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:51.293 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.622686+0000", 2024-02-09T17:39:51.293 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8233999999999998e-05 2024-02-09T17:39:51.293 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.293 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.293 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:51.293 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.622727+0000", 2024-02-09T17:39:51.293 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0175000000000001e-05 2024-02-09T17:39:51.293 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:51.293 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:51.293 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:51.293 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.294 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.294 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:232453 4.0 4:0be208e0:::benchmark_data_smithi179_103809_object94987:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-02-09T17:39:51.294 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:39:22.735817+0000", 2024-02-09T17:39:51.294 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 28.453026264000002, 2024-02-09T17:39:51.294 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.88704940300000001, 2024-02-09T17:39:51.294 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:51.294 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:51.294 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:51.294 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:51.294 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:51.294 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 232453 2024-02-09T17:39:51.295 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.295 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:51.295 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.295 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:51.295 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735817+0000", 2024-02-09T17:39:51.295 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:51.295 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.295 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.295 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:51.295 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735817+0000", 2024-02-09T17:39:51.295 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:51.296 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.296 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.296 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:51.296 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735818+0000", 2024-02-09T17:39:51.296 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4400000000000001e-07 2024-02-09T17:39:51.296 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.296 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.296 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:51.296 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735820+0000", 2024-02-09T17:39:51.296 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.34e-06 2024-02-09T17:39:51.296 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.296 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.297 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:51.297 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735827+0000", 2024-02-09T17:39:51.297 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8299999999999998e-06 2024-02-09T17:39:51.297 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.297 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.297 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:51.297 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735832+0000", 2024-02-09T17:39:51.297 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6679999999999998e-06 2024-02-09T17:39:51.297 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.297 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.297 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:51.298 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735867+0000", 2024-02-09T17:39:51.298 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4746999999999998e-05 2024-02-09T17:39:51.298 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.298 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.298 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:39:51.298 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735871+0000", 2024-02-09T17:39:51.298 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9210000000000002e-06 2024-02-09T17:39:51.298 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.298 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.298 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:51.298 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.606897+0000", 2024-02-09T17:39:51.299 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.87102594600000005 2024-02-09T17:39:51.299 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.299 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.299 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:51.299 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.607026+0000", 2024-02-09T17:39:51.299 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012845000000000001 2024-02-09T17:39:51.299 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.299 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.299 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,9", 2024-02-09T17:39:51.299 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.607107+0000", 2024-02-09T17:39:51.299 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1458000000000005e-05 2024-02-09T17:39:51.300 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.300 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.300 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:51.300 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.613784+0000", 2024-02-09T17:39:51.300 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0066770040000000003 2024-02-09T17:39:51.300 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.300 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.300 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:51.300 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.621146+0000", 2024-02-09T17:39:51.300 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0073621379999999998 2024-02-09T17:39:51.300 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.301 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.301 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:51.301 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.622785+0000", 2024-02-09T17:39:51.303 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001639199 2024-02-09T17:39:51.303 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.303 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.303 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:51.303 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.622829+0000", 2024-02-09T17:39:51.303 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3457000000000001e-05 2024-02-09T17:39:51.303 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.303 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.304 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:51.304 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.622867+0000", 2024-02-09T17:39:51.304 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7800999999999999e-05 2024-02-09T17:39:51.304 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:51.304 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:51.304 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:51.304 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.304 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.304 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:232474 4.0 4:05e1ef6a:::benchmark_data_smithi179_103809_object95008:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-02-09T17:39:51.304 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:39:22.735849+0000", 2024-02-09T17:39:51.304 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 28.452994447000002, 2024-02-09T17:39:51.304 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.88716656000000005, 2024-02-09T17:39:51.305 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:51.305 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:51.305 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:51.305 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:51.305 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:51.305 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 232474 2024-02-09T17:39:51.305 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.305 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:51.305 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.305 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:51.305 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735849+0000", 2024-02-09T17:39:51.305 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:51.305 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.306 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.306 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:51.306 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735849+0000", 2024-02-09T17:39:51.306 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:51.306 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.306 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.306 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:51.306 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735849+0000", 2024-02-09T17:39:51.306 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7399999999999999e-07 2024-02-09T17:39:51.306 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.306 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.306 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:51.306 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735852+0000", 2024-02-09T17:39:51.307 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3549999999999999e-06 2024-02-09T17:39:51.307 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.307 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.307 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:51.307 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735858+0000", 2024-02-09T17:39:51.307 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.319e-06 2024-02-09T17:39:51.307 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.307 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.307 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:51.307 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735864+0000", 2024-02-09T17:39:51.307 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1800000000000001e-06 2024-02-09T17:39:51.307 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.307 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.308 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:51.308 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735886+0000", 2024-02-09T17:39:51.308 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1517000000000001e-05 2024-02-09T17:39:51.308 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.308 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.308 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:39:51.308 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735889+0000", 2024-02-09T17:39:51.308 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.41e-06 2024-02-09T17:39:51.308 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.308 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.308 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:51.309 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.609948+0000", 2024-02-09T17:39:51.309 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.87405859299999999 2024-02-09T17:39:51.309 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.309 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.309 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:51.309 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.610078+0000", 2024-02-09T17:39:51.309 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000130204 2024-02-09T17:39:51.309 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.309 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.309 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,9", 2024-02-09T17:39:51.310 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.610219+0000", 2024-02-09T17:39:51.310 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014117000000000001 2024-02-09T17:39:51.310 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.310 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.310 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:51.310 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.613789+0000", 2024-02-09T17:39:51.311 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0035698560000000002 2024-02-09T17:39:51.311 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.311 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.311 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:51.311 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.621196+0000", 2024-02-09T17:39:51.311 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0074071079999999999 2024-02-09T17:39:51.311 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.311 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.311 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:51.311 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.622922+0000", 2024-02-09T17:39:51.311 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001725667 2024-02-09T17:39:51.312 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.312 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.312 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:51.312 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.622980+0000", 2024-02-09T17:39:51.312 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8508999999999998e-05 2024-02-09T17:39:51.312 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.312 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.312 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:51.312 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.623016+0000", 2024-02-09T17:39:51.312 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.5298e-05 2024-02-09T17:39:51.312 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:51.312 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:51.313 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:51.313 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.313 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.313 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:232516 4.0 4:0c99ad98:::benchmark_data_smithi179_103809_object95050:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-02-09T17:39:51.313 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:39:22.735873+0000", 2024-02-09T17:39:51.313 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 28.452970158999999, 2024-02-09T17:39:51.313 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.88729041500000005, 2024-02-09T17:39:51.313 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:51.313 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:51.313 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:51.313 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:51.314 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:51.314 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 232516 2024-02-09T17:39:51.314 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.314 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:51.314 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.314 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:51.314 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735873+0000", 2024-02-09T17:39:51.314 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:51.314 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.314 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.314 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:51.314 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735873+0000", 2024-02-09T17:39:51.314 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:51.314 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.315 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.315 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:51.315 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735874+0000", 2024-02-09T17:39:51.315 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7800000000000002e-07 2024-02-09T17:39:51.315 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.315 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.315 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:51.315 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735876+0000", 2024-02-09T17:39:51.315 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2069999999999998e-06 2024-02-09T17:39:51.315 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.315 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.315 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:51.315 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735882+0000", 2024-02-09T17:39:51.315 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9220000000000002e-06 2024-02-09T17:39:51.316 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.316 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.316 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:51.316 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735887+0000", 2024-02-09T17:39:51.316 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.31e-06 2024-02-09T17:39:51.316 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.316 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.316 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:51.316 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735909+0000", 2024-02-09T17:39:51.316 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2161e-05 2024-02-09T17:39:51.316 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.316 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.316 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:39:51.316 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735913+0000", 2024-02-09T17:39:51.317 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.495e-06 2024-02-09T17:39:51.317 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.317 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.317 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:51.317 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.613979+0000", 2024-02-09T17:39:51.317 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.87806641699999999 2024-02-09T17:39:51.317 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.317 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.317 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:51.317 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.614061+0000", 2024-02-09T17:39:51.317 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1575000000000004e-05 2024-02-09T17:39:51.317 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.317 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.317 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,9", 2024-02-09T17:39:51.318 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.614181+0000", 2024-02-09T17:39:51.318 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000120658 2024-02-09T17:39:51.318 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.318 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.318 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:51.318 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.621212+0000", 2024-02-09T17:39:51.318 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0070306910000000004 2024-02-09T17:39:51.318 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.318 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.318 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:51.318 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.621274+0000", 2024-02-09T17:39:51.318 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2323000000000006e-05 2024-02-09T17:39:51.319 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.319 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.319 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:51.319 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.623072+0000", 2024-02-09T17:39:51.319 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0017977850000000001 2024-02-09T17:39:51.319 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.319 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.319 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:51.319 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.623123+0000", 2024-02-09T17:39:51.319 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0671000000000003e-05 2024-02-09T17:39:51.319 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.319 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.319 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:51.320 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.623164+0000", 2024-02-09T17:39:51.320 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0822000000000003e-05 2024-02-09T17:39:51.320 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:51.320 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:51.320 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:51.320 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.320 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.320 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:232537 4.0 4:0c405ed9:::benchmark_data_smithi179_103809_object95071:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-02-09T17:39:51.320 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:39:22.800233+0000", 2024-02-09T17:39:51.320 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 28.388610200999999, 2024-02-09T17:39:51.320 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.82809807000000002, 2024-02-09T17:39:51.321 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:39:51.321 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:39:51.321 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:39:51.321 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:39:51.321 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:39:51.321 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 232537 2024-02-09T17:39:51.321 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.321 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:39:51.321 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.321 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:39:51.321 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.800233+0000", 2024-02-09T17:39:51.321 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:51.321 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.322 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.322 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:39:51.322 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.800233+0000", 2024-02-09T17:39:51.322 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:39:51.322 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.322 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.322 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:39:51.322 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.800235+0000", 2024-02-09T17:39:51.322 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.671e-06 2024-02-09T17:39:51.322 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.322 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.322 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:39:51.322 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.800241+0000", 2024-02-09T17:39:51.323 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2450000000000003e-06 2024-02-09T17:39:51.323 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.323 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.323 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:39:51.323 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.800255+0000", 2024-02-09T17:39:51.323 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3689999999999999e-05 2024-02-09T17:39:51.323 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.323 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.323 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:39:51.323 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.800268+0000", 2024-02-09T17:39:51.323 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3546999999999999e-05 2024-02-09T17:39:51.323 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.323 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.324 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:51.324 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.800330+0000", 2024-02-09T17:39:51.324 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1965999999999996e-05 2024-02-09T17:39:51.324 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.324 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.324 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:39:51.324 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.800340+0000", 2024-02-09T17:39:51.324 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2040000000000003e-06 2024-02-09T17:39:51.324 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.324 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.324 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:39:51.324 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.617653+0000", 2024-02-09T17:39:51.324 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.81731293000000005 2024-02-09T17:39:51.325 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.325 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.325 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:39:51.325 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.617775+0000", 2024-02-09T17:39:51.325 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012293800000000001 2024-02-09T17:39:51.325 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.326 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:50 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3409591509' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-09T17:39:51.326 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:50 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-09T17:39:51.326 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.326 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,9", 2024-02-09T17:39:51.326 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.617891+0000", 2024-02-09T17:39:51.326 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011540900000000001 2024-02-09T17:39:51.326 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.327 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.327 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:39:51.327 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.621350+0000", 2024-02-09T17:39:51.327 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0034587469999999999 2024-02-09T17:39:51.327 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.327 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.327 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:51.327 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.623212+0000", 2024-02-09T17:39:51.327 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001862306 2024-02-09T17:39:51.327 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.327 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.327 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:39:51.328 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.628274+0000", 2024-02-09T17:39:51.328 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0050616360000000004 2024-02-09T17:39:51.328 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.328 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.328 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:39:51.328 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.628309+0000", 2024-02-09T17:39:51.328 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.5828000000000003e-05 2024-02-09T17:39:51.328 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:39:51.328 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:39:51.328 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:39:51.328 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.628331+0000", 2024-02-09T17:39:51.328 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1953000000000001e-05 2024-02-09T17:39:51.328 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:51.329 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:51.329 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:51.329 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:39:51.329 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:39:51.329 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:39:51.399 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:39:51.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:51.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:39:51.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:50 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3409591509' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-09T17:39:51.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:50 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-09T17:39:51.514 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:39:51.563 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:51.589 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:39:51.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:51.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:39:51.694 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:39:51.752 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:51.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:51 smithi099 ceph-mon[39420]: pgmap v1123: 37 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 33 active+clean; 1.7 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 730 op/s; 4253/82302 objects misplaced (5.168%); 4.6 MiB/s, 73 objects/s recovering 2024-02-09T17:39:51.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:51 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-09T17:39:51.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:51 smithi099 ceph-mon[39420]: osdmap e431: 12 total, 12 up, 11 in 2024-02-09T17:39:51.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:51 smithi099 ceph-mon[39420]: osdmap e432: 12 total, 12 up, 11 in 2024-02-09T17:39:52.001 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-09T17:39:52.200 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 5 1 2024-02-09T17:39:52.212 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:39:52.223 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:52.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:39:52.325 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:39:52.364 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:51 smithi162 ceph-mon[44304]: pgmap v1123: 37 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 33 active+clean; 1.7 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 730 op/s; 4253/82302 objects misplaced (5.168%); 4.6 MiB/s, 73 objects/s recovering 2024-02-09T17:39:52.364 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:51 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-09T17:39:52.364 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:51 smithi162 ceph-mon[44304]: osdmap e431: 12 total, 12 up, 11 in 2024-02-09T17:39:52.364 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:51 smithi162 ceph-mon[44304]: osdmap e432: 12 total, 12 up, 11 in 2024-02-09T17:39:52.365 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:52.385 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:39:52.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:52.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:39:52.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:51 smithi179 ceph-mon[43975]: pgmap v1123: 37 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 33 active+clean; 1.7 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 730 op/s; 4253/82302 objects misplaced (5.168%); 4.6 MiB/s, 73 objects/s recovering 2024-02-09T17:39:52.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:51 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-09T17:39:52.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:51 smithi179 ceph-mon[43975]: osdmap e431: 12 total, 12 up, 11 in 2024-02-09T17:39:52.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:51 smithi179 ceph-mon[43975]: osdmap e432: 12 total, 12 up, 11 in 2024-02-09T17:39:52.490 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:39:52.525 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:52.553 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:39:52.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:52.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:39:52.659 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:39:52.723 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:52.829 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:39:52.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:52.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:39:52.876 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:39:52.888 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:39:52.939 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:39:52.967 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:52.996 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:39:53.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:53.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:39:53.105 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:39:53.134 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:53.161 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:39:53.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:53.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:39:53.268 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:39:53.297 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:53.319 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:39:53.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:53.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:39:53.428 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:39:53.429 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:39:53.464 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:53.499 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:39:53.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:53.502 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:39:53.603 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:39:53.648 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:53.674 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:39:53.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:53.680 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:39:53.781 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:39:53.813 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:53.846 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:39:53.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:53.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:39:53.957 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:39:54.028 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:54.041 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is unset 2024-02-09T17:39:54.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:54 smithi162 ceph-mon[44304]: pgmap v1126: 37 pgs: 3 active+remapped+backfill_toofull, 34 active+clean; 1.6 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 717 op/s; 4253/79986 objects misplaced (5.317%) 2024-02-09T17:39:54.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:54 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/656898217' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:39:54.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:54 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:39:54.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:54 smithi179 ceph-mon[43975]: pgmap v1126: 37 pgs: 3 active+remapped+backfill_toofull, 34 active+clean; 1.6 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 717 op/s; 4253/79986 objects misplaced (5.317%) 2024-02-09T17:39:54.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:54 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/656898217' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:39:54.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:54 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:39:54.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:54 smithi099 ceph-mon[39420]: pgmap v1126: 37 pgs: 3 active+remapped+backfill_toofull, 34 active+clean; 1.6 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 717 op/s; 4253/79986 objects misplaced (5.317%) 2024-02-09T17:39:54.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:54 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/656898217' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:39:54.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:54 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:39:54.867 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:39:54.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:54.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:39:54.974 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:39:55.017 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:55.048 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:39:55.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:55.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:39:55.153 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:39:55.184 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:55.208 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:39:55.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:55.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:39:55.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:55 smithi099 ceph-mon[39420]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:39:55.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:55 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:39:55.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:55 smithi099 ceph-mon[39420]: osdmap e433: 12 total, 12 up, 11 in 2024-02-09T17:39:55.314 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:39:55.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:55 smithi162 ceph-mon[44304]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:39:55.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:55 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:39:55.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:55 smithi162 ceph-mon[44304]: osdmap e433: 12 total, 12 up, 11 in 2024-02-09T17:39:55.348 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:55.374 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:39:55.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:55.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:39:55.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:55 smithi179 ceph-mon[43975]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:39:55.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:55 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:39:55.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:55 smithi179 ceph-mon[43975]: osdmap e433: 12 total, 12 up, 11 in 2024-02-09T17:39:55.478 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:39:55.523 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:56.166 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:56 smithi099 ceph-mon[39420]: pgmap v1128: 37 pgs: 3 active+remapped+backfill_toofull, 34 active+clean; 1.6 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 727 op/s; 4034/77829 objects misplaced (5.183%) 2024-02-09T17:39:56.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:56 smithi179 ceph-mon[43975]: pgmap v1128: 37 pgs: 3 active+remapped+backfill_toofull, 34 active+clean; 1.6 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 727 op/s; 4034/77829 objects misplaced (5.183%) 2024-02-09T17:39:56.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:56 smithi162 ceph-mon[44304]: pgmap v1128: 37 pgs: 3 active+remapped+backfill_toofull, 34 active+clean; 1.6 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 727 op/s; 4034/77829 objects misplaced (5.183%) 2024-02-09T17:39:57.285 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:39:57.285 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [ 2024-02-09T17:39:57.285 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:39:57.285 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25201.0:253203 4.7 e434/428 4:ef3a8bea:::benchmark_data_smithi179_103809_object115737:head v 434'16001, mlcod=433'15999)", 2024-02-09T17:39:57.285 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:39:56.518563+0000", 2024-02-09T17:39:57.285 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.00674648, 2024-02-09T17:39:57.285 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0067743839999999996, 2024-02-09T17:39:57.285 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:39:57.285 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T17:39:57.285 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:39:57.285 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:39:57.286 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:39:57.286 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:56.518563+0000", 2024-02-09T17:39:57.286 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:39:57.286 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:39:57.286 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:39:57.286 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:39:57.286 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:56.518563+0000", 2024-02-09T17:39:57.286 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:39:57.286 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:39:57.286 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:39:57.286 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:39:57.286 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:56.518564+0000", 2024-02-09T17:39:57.286 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.3600000000000003e-07 2024-02-09T17:39:57.287 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:39:57.287 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:39:57.287 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:39:57.287 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:56.518570+0000", 2024-02-09T17:39:57.287 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.0619999999999997e-06 2024-02-09T17:39:57.287 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:39:57.287 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:39:57.287 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:39:57.287 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:56.518583+0000", 2024-02-09T17:39:57.287 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2850000000000001e-05 2024-02-09T17:39:57.287 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:39:57.287 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:39:57.287 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:39:57.288 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:56.518589+0000", 2024-02-09T17:39:57.288 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.4799999999999998e-06 2024-02-09T17:39:57.288 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:39:57.288 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:39:57.288 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:39:57.288 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:56.518662+0000", 2024-02-09T17:39:57.288 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.2336000000000007e-05 2024-02-09T17:39:57.288 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:39:57.288 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:39:57.288 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:39:57.288 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:56.518681+0000", 2024-02-09T17:39:57.288 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.8830000000000001e-05 2024-02-09T17:39:57.288 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:39:57.289 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:39:57.289 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:39:57.289 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:39:57.289 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T17:39:57.289 INFO:teuthology.orchestra.run.smithi099.stdout: "num_ops": 1 2024-02-09T17:39:57.289 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:39:57.483 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:57 smithi099 ceph-mon[39420]: osdmap e434: 12 total, 12 up, 11 in 2024-02-09T17:39:57.484 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:57 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:39:57.484 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:57 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:39:57.484 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:57 smithi099 ceph-mon[39420]: pgmap v1130: 37 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 34 active+clean; 1.4 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 785 op/s; 2458/68679 objects misplaced (3.579%); 14 MiB/s, 224 objects/s recovering 2024-02-09T17:39:57.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:57 smithi179 ceph-mon[43975]: osdmap e434: 12 total, 12 up, 11 in 2024-02-09T17:39:57.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:57 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:39:57.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:57 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:39:57.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:57 smithi179 ceph-mon[43975]: pgmap v1130: 37 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 34 active+clean; 1.4 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 785 op/s; 2458/68679 objects misplaced (3.579%); 14 MiB/s, 224 objects/s recovering 2024-02-09T17:39:57.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:57 smithi162 ceph-mon[44304]: osdmap e434: 12 total, 12 up, 11 in 2024-02-09T17:39:57.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:57 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:39:57.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:57 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:39:57.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:57 smithi162 ceph-mon[44304]: pgmap v1130: 37 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 34 active+clean; 1.4 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 785 op/s; 2458/68679 objects misplaced (3.579%); 14 MiB/s, 224 objects/s recovering 2024-02-09T17:39:57.962 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-09T17:39:58.166 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:39:58.503 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.5 to 1 (10000) 2024-02-09T17:39:58.509 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:39:58.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:58.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:39:58.614 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:39:58.670 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:58.740 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:39:58.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:58 smithi099 ceph-mon[39420]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:39:58.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:58 smithi099 ceph-mon[39420]: osdmap e435: 12 total, 12 up, 11 in 2024-02-09T17:39:58.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:58 smithi099 ceph-mon[39420]: 4.b scrub starts 2024-02-09T17:39:58.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:58 smithi099 ceph-mon[39420]: 4.b scrub ok 2024-02-09T17:39:58.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:58 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2757425783' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-09T17:39:58.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:58.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:39:58.845 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:39:58.864 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set noscrub 2024-02-09T17:39:58.873 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:58 smithi179 ceph-mon[43975]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:39:58.873 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:58 smithi179 ceph-mon[43975]: osdmap e435: 12 total, 12 up, 11 in 2024-02-09T17:39:58.873 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:58 smithi179 ceph-mon[43975]: 4.b scrub starts 2024-02-09T17:39:58.874 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:58 smithi179 ceph-mon[43975]: 4.b scrub ok 2024-02-09T17:39:58.874 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:58 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2757425783' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-09T17:39:58.876 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:58.905 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:39:58.909 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:58.910 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:39:58.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:58 smithi162 ceph-mon[44304]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:39:58.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:58 smithi162 ceph-mon[44304]: osdmap e435: 12 total, 12 up, 11 in 2024-02-09T17:39:58.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:58 smithi162 ceph-mon[44304]: 4.b scrub starts 2024-02-09T17:39:58.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:58 smithi162 ceph-mon[44304]: 4.b scrub ok 2024-02-09T17:39:58.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:58 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2757425783' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-09T17:39:59.010 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:39:59.075 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:59.249 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:39:59.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:59.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:39:59.333 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:39:59.353 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:39:59.379 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:59.398 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:39:59.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:59.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:39:59.502 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:39:59.533 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:59.557 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:39:59.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:59.560 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:39:59.661 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:39:59.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:59 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2757425783' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-09T17:39:59.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:59 smithi162 ceph-mon[44304]: osdmap e436: 12 total, 12 up, 11 in 2024-02-09T17:39:59.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:59 smithi162 ceph-mon[44304]: 4.c scrub starts 2024-02-09T17:39:59.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:59 smithi162 ceph-mon[44304]: pgmap v1133: 37 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 34 active+clean; 1.4 GiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 744 op/s; 2458/67206 objects misplaced (3.657%); 17 MiB/s, 273 objects/s recovering 2024-02-09T17:39:59.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:39:59 smithi162 ceph-mon[44304]: 4.c scrub ok 2024-02-09T17:39:59.695 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:59.721 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:39:59.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:59.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:39:59.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:59 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2757425783' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-09T17:39:59.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:59 smithi099 ceph-mon[39420]: osdmap e436: 12 total, 12 up, 11 in 2024-02-09T17:39:59.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:59 smithi099 ceph-mon[39420]: 4.c scrub starts 2024-02-09T17:39:59.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:59 smithi099 ceph-mon[39420]: pgmap v1133: 37 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 34 active+clean; 1.4 GiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 744 op/s; 2458/67206 objects misplaced (3.657%); 17 MiB/s, 273 objects/s recovering 2024-02-09T17:39:59.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:39:59 smithi099 ceph-mon[39420]: 4.c scrub ok 2024-02-09T17:39:59.827 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:39:59.870 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:59 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2757425783' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-09T17:39:59.870 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:59 smithi179 ceph-mon[43975]: osdmap e436: 12 total, 12 up, 11 in 2024-02-09T17:39:59.870 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:59 smithi179 ceph-mon[43975]: 4.c scrub starts 2024-02-09T17:39:59.870 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:59 smithi179 ceph-mon[43975]: pgmap v1133: 37 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 34 active+clean; 1.4 GiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 744 op/s; 2458/67206 objects misplaced (3.657%); 17 MiB/s, 273 objects/s recovering 2024-02-09T17:39:59.870 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:39:59 smithi179 ceph-mon[43975]: 4.c scrub ok 2024-02-09T17:39:59.873 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:39:59.903 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:39:59.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:39:59.908 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:39:59.958 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 6 1 2024-02-09T17:40:00.009 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:40:00.062 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:00.132 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:40:00.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:00.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:40:00.238 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:40:00.276 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:00.299 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:40:00.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:00.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:40:00.405 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:40:00.439 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:00.461 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:40:00.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:00.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:40:00.566 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:40:00.610 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:00.625 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:40:00.636 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:40:00.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:00.641 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:40:00.743 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:40:00.778 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:00.824 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:40:00.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:00.826 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:40:00.828 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:00 smithi179 ceph-mon[43975]: Health detail: HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull 2024-02-09T17:40:00.828 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:00 smithi179 ceph-mon[43975]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull 2024-02-09T17:40:00.828 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:00 smithi179 ceph-mon[43975]: pg 4.0 is active+remapped+backfill_toofull, acting [0,8,9] 2024-02-09T17:40:00.828 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:00 smithi179 ceph-mon[43975]: pg 4.4 is active+remapped+backfill_toofull, acting [1,3,2] 2024-02-09T17:40:00.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:00 smithi162 ceph-mon[44304]: Health detail: HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull 2024-02-09T17:40:00.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:00 smithi162 ceph-mon[44304]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull 2024-02-09T17:40:00.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:00 smithi162 ceph-mon[44304]: pg 4.0 is active+remapped+backfill_toofull, acting [0,8,9] 2024-02-09T17:40:00.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:00 smithi162 ceph-mon[44304]: pg 4.4 is active+remapped+backfill_toofull, acting [1,3,2] 2024-02-09T17:40:00.927 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:40:00.978 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:00.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:00 smithi099 ceph-mon[39420]: Health detail: HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull 2024-02-09T17:40:00.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:00 smithi099 ceph-mon[39420]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull 2024-02-09T17:40:00.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:00 smithi099 ceph-mon[39420]: pg 4.0 is active+remapped+backfill_toofull, acting [0,8,9] 2024-02-09T17:40:00.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:00 smithi099 ceph-mon[39420]: pg 4.4 is active+remapped+backfill_toofull, acting [1,3,2] 2024-02-09T17:40:01.006 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:40:01.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:01.010 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:40:01.111 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:40:01.144 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:01.178 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:40:01.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:01.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:40:01.287 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:40:01.321 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:01.678 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:40:01.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:01.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:40:01.685 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:01 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:40:01.685 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:01 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:40:01.685 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:01 smithi099 ceph-mon[39420]: osdmap e437: 12 total, 12 up, 11 in 2024-02-09T17:40:01.685 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:01 smithi099 ceph-mon[39420]: pgmap v1135: 37 pgs: 2 active+remapped+backfill_toofull, 35 active+clean; 1.2 GiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 823 op/s; 1878/59565 objects misplaced (3.153%); 3.8 MiB/s, 61 objects/s recovering 2024-02-09T17:40:01.685 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:01 smithi099 ceph-mon[39420]: osdmap e438: 12 total, 12 up, 11 in 2024-02-09T17:40:01.785 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:40:01.815 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:01 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:40:01.815 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:01 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:40:01.816 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:01 smithi162 ceph-mon[44304]: osdmap e437: 12 total, 12 up, 11 in 2024-02-09T17:40:01.816 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:01 smithi162 ceph-mon[44304]: pgmap v1135: 37 pgs: 2 active+remapped+backfill_toofull, 35 active+clean; 1.2 GiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 823 op/s; 1878/59565 objects misplaced (3.153%); 3.8 MiB/s, 61 objects/s recovering 2024-02-09T17:40:01.816 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:01 smithi162 ceph-mon[44304]: osdmap e438: 12 total, 12 up, 11 in 2024-02-09T17:40:01.818 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:01.843 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:40:01.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:01.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:40:01.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:01 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:40:01.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:01 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:40:01.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:01 smithi179 ceph-mon[43975]: osdmap e437: 12 total, 12 up, 11 in 2024-02-09T17:40:01.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:01 smithi179 ceph-mon[43975]: pgmap v1135: 37 pgs: 2 active+remapped+backfill_toofull, 35 active+clean; 1.2 GiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 823 op/s; 1878/59565 objects misplaced (3.153%); 3.8 MiB/s, 61 objects/s recovering 2024-02-09T17:40:01.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:01 smithi179 ceph-mon[43975]: osdmap e438: 12 total, 12 up, 11 in 2024-02-09T17:40:01.948 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:40:01.961 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:40:02.014 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:02.069 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:40:02.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:02.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:40:02.191 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:40:02.225 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:02.259 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:40:02.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:02.267 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:40:02.368 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:40:02.398 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:02.419 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:40:02.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:02.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:40:02.526 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:40:02.584 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:02.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:02 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:40:02.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:02 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:40:02.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:02 smithi162 ceph-mon[44304]: 1.6 deep-scrub starts 2024-02-09T17:40:02.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:02 smithi162 ceph-mon[44304]: 1.6 deep-scrub ok 2024-02-09T17:40:02.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:02 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1635979000' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:40:02.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:02 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:40:02.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:02 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:40:02.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:02 smithi179 ceph-mon[43975]: 1.6 deep-scrub starts 2024-02-09T17:40:02.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:02 smithi179 ceph-mon[43975]: 1.6 deep-scrub ok 2024-02-09T17:40:02.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:02 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1635979000' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:40:02.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:02 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:40:02.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:02 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:40:02.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:02 smithi099 ceph-mon[39420]: 1.6 deep-scrub starts 2024-02-09T17:40:02.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:02 smithi099 ceph-mon[39420]: 1.6 deep-scrub ok 2024-02-09T17:40:02.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:02 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1635979000' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:40:03.168 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:40:03.168 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:40:03.168 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:40:03.168 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:40:03.168 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:40:03.168 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:40:03.168 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:40:03.168 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:40:03.168 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:40:03.168 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:40:03.169 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:40:03.169 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:40:03.169 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:40:03.169 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:40:03.169 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:40:03.169 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:40:03.169 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:40:03.169 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:40:03.169 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:40:03.169 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:40:03.169 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:40:03.169 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:40:03.170 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:40:03.170 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:40:03.465 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:40:03.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:03.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:40:03.576 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:40:03.610 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:03.632 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:40:03.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:03.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:40:03.739 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:40:03.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:03 smithi099 ceph-mon[39420]: 1.7 scrub starts 2024-02-09T17:40:03.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:03 smithi099 ceph-mon[39420]: 1.7 scrub ok 2024-02-09T17:40:03.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:03 smithi099 ceph-mon[39420]: pgmap v1137: 37 pgs: 2 active+remapped+backfill_toofull, 35 active+clean; 1.2 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 638 op/s; 1878/57810 objects misplaced (3.249%); 3.1 MiB/s, 50 objects/s recovering 2024-02-09T17:40:03.808 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:03.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:03 smithi179 ceph-mon[43975]: 1.7 scrub starts 2024-02-09T17:40:03.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:03 smithi179 ceph-mon[43975]: 1.7 scrub ok 2024-02-09T17:40:03.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:03 smithi179 ceph-mon[43975]: pgmap v1137: 37 pgs: 2 active+remapped+backfill_toofull, 35 active+clean; 1.2 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 638 op/s; 1878/57810 objects misplaced (3.249%); 3.1 MiB/s, 50 objects/s recovering 2024-02-09T17:40:03.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:03 smithi162 ceph-mon[44304]: 1.7 scrub starts 2024-02-09T17:40:03.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:03 smithi162 ceph-mon[44304]: 1.7 scrub ok 2024-02-09T17:40:03.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:03 smithi162 ceph-mon[44304]: pgmap v1137: 37 pgs: 2 active+remapped+backfill_toofull, 35 active+clean; 1.2 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 638 op/s; 1878/57810 objects misplaced (3.249%); 3.1 MiB/s, 50 objects/s recovering 2024-02-09T17:40:04.181 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:40:04.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:04.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:40:04.294 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:40:04.330 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:04.369 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:40:04.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:04.373 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:40:04.473 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:40:04.532 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:04.576 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:40:04.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:04.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:40:04.618 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:04 smithi179 ceph-mon[43975]: 1.c scrub starts 2024-02-09T17:40:04.618 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:04 smithi179 ceph-mon[43975]: 1.c scrub ok 2024-02-09T17:40:04.682 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:40:04.716 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:04.735 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:40:04.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:04.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:40:04.840 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:40:04.872 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:04.897 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:40:04.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:04.901 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:40:04.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:04 smithi162 ceph-mon[44304]: 1.c scrub starts 2024-02-09T17:40:04.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:04 smithi162 ceph-mon[44304]: 1.c scrub ok 2024-02-09T17:40:04.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:04 smithi099 ceph-mon[39420]: 1.c scrub starts 2024-02-09T17:40:04.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:04 smithi099 ceph-mon[39420]: 1.c scrub ok 2024-02-09T17:40:05.002 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:40:05.043 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:05.080 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:40:05.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:05.088 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:40:05.189 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:40:05.246 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:05.268 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:40:05.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:05.274 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:40:05.375 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:40:05.408 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:05.433 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:40:05.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:05.437 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:40:05.538 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:40:05.567 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:05.587 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:40:05.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:05.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:40:05.656 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:05 smithi099 ceph-mon[39420]: pgmap v1138: 37 pgs: 2 active+remapped+backfill_toofull, 35 active+clean; 1.1 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 603 op/s; 1630/55848 objects misplaced (2.919%); 2.6 MiB/s, 42 objects/s recovering 2024-02-09T17:40:05.656 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:05 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1527604962' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:40:05.657 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:05 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:40:05.668 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is set 2024-02-09T17:40:05.691 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:40:05.745 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:05.779 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:40:05.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:05.785 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:40:05.886 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:40:05.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:05 smithi179 ceph-mon[43975]: pgmap v1138: 37 pgs: 2 active+remapped+backfill_toofull, 35 active+clean; 1.1 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 603 op/s; 1630/55848 objects misplaced (2.919%); 2.6 MiB/s, 42 objects/s recovering 2024-02-09T17:40:05.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:05 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1527604962' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:40:05.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:05 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:40:05.917 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:05.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:05 smithi162 ceph-mon[44304]: pgmap v1138: 37 pgs: 2 active+remapped+backfill_toofull, 35 active+clean; 1.1 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 603 op/s; 1630/55848 objects misplaced (2.919%); 2.6 MiB/s, 42 objects/s recovering 2024-02-09T17:40:05.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:05 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1527604962' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:40:05.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:05 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:40:05.939 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:40:05.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:05.942 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:40:06.043 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:40:06.072 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:06.089 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:40:06.089 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:40:06.092 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:40:06.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:06.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:40:06.196 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:40:06.269 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:06.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:06 smithi179 ceph-mon[43975]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:40:06.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:06 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:40:06.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:06 smithi179 ceph-mon[43975]: osdmap e439: 12 total, 12 up, 11 in 2024-02-09T17:40:06.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:06 smithi179 ceph-mon[43975]: osdmap e440: 12 total, 12 up, 11 in 2024-02-09T17:40:06.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:06 smithi162 ceph-mon[44304]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:40:06.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:06 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:40:06.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:06 smithi162 ceph-mon[44304]: osdmap e439: 12 total, 12 up, 11 in 2024-02-09T17:40:06.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:06 smithi162 ceph-mon[44304]: osdmap e440: 12 total, 12 up, 11 in 2024-02-09T17:40:06.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:06 smithi099 ceph-mon[39420]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:40:06.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:06 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:40:06.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:06 smithi099 ceph-mon[39420]: osdmap e439: 12 total, 12 up, 11 in 2024-02-09T17:40:06.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:06 smithi099 ceph-mon[39420]: osdmap e440: 12 total, 12 up, 11 in 2024-02-09T17:40:07.424 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:40:07.666 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:40:07.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:07.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:40:07.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:07 smithi099 ceph-mon[39420]: pgmap v1141: 37 pgs: 2 active+remapped+backfill_toofull, 35 active+clean; 977 MiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 703 op/s; 1484/46890 objects misplaced (3.165%) 2024-02-09T17:40:07.779 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:40:07.808 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:07.834 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:40:07.839 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:07.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:40:07.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:07 smithi179 ceph-mon[43975]: pgmap v1141: 37 pgs: 2 active+remapped+backfill_toofull, 35 active+clean; 977 MiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 703 op/s; 1484/46890 objects misplaced (3.165%) 2024-02-09T17:40:07.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:07 smithi162 ceph-mon[44304]: pgmap v1141: 37 pgs: 2 active+remapped+backfill_toofull, 35 active+clean; 977 MiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 703 op/s; 1484/46890 objects misplaced (3.165%) 2024-02-09T17:40:07.940 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:40:07.968 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:07.989 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:40:07.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:07.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:40:08.095 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:40:08.136 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:08.348 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:40:08.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:08.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:40:08.456 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:40:08.531 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:08.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:08 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:40:08.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:08 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:40:08.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:08 smithi099 ceph-mon[39420]: osdmap e441: 12 total, 12 up, 11 in 2024-02-09T17:40:08.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:08 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:40:08.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:08 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:40:08.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:08 smithi179 ceph-mon[43975]: osdmap e441: 12 total, 12 up, 11 in 2024-02-09T17:40:08.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:08 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:40:08.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:08 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:40:08.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:08 smithi162 ceph-mon[44304]: osdmap e441: 12 total, 12 up, 11 in 2024-02-09T17:40:09.048 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:40:09.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:09.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:40:09.160 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:40:09.192 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:09.215 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:40:09.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:09.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:40:09.319 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:40:09.367 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:09.399 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:40:09.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:09.407 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:40:09.508 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:40:09.547 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:09.573 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:40:09.576 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:09.576 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:40:09.667 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:09 smithi162 ceph-mon[44304]: pgmap v1143: 37 pgs: 2 active+remapped+backfill_toofull, 35 active+clean; 947 MiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 686 op/s; 1484/45444 objects misplaced (3.266%) 2024-02-09T17:40:09.668 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:09 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1603535952' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-09T17:40:09.677 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:40:09.695 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.6 to 1 (10000) 2024-02-09T17:40:09.720 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:09.722 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:09 smithi099 ceph-mon[39420]: pgmap v1143: 37 pgs: 2 active+remapped+backfill_toofull, 35 active+clean; 947 MiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 686 op/s; 1484/45444 objects misplaced (3.266%) 2024-02-09T17:40:09.722 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:09 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1603535952' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-09T17:40:09.740 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:40:09.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:09.742 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:40:09.814 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-09T17:40:09.843 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:40:09.869 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:09.887 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:40:09.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:09.889 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:40:09.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:09 smithi179 ceph-mon[43975]: pgmap v1143: 37 pgs: 2 active+remapped+backfill_toofull, 35 active+clean; 947 MiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 686 op/s; 1484/45444 objects misplaced (3.266%) 2024-02-09T17:40:09.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:09 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1603535952' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-09T17:40:09.990 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:40:10.016 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:10.034 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:40:10.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:10.037 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:40:10.138 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:40:10.166 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:10.185 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:40:10.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:10.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:40:10.289 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:40:10.315 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:10.334 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:40:10.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:10.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:40:10.437 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:40:10.439 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:40:10.465 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:10.471 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 7 1 2024-02-09T17:40:10.484 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:40:10.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:10.486 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:40:10.588 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:40:10.613 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:10.642 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:40:10.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:10.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:40:10.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:10 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1603535952' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-09T17:40:10.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:10 smithi099 ceph-mon[39420]: osdmap e442: 12 total, 12 up, 11 in 2024-02-09T17:40:10.748 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:40:10.773 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:10 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1603535952' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-09T17:40:10.773 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:10 smithi179 ceph-mon[43975]: osdmap e442: 12 total, 12 up, 11 in 2024-02-09T17:40:10.775 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:10.793 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:40:10.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:10.796 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:40:10.897 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:40:10.923 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:10.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:10 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1603535952' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-09T17:40:10.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:10 smithi162 ceph-mon[44304]: osdmap e442: 12 total, 12 up, 11 in 2024-02-09T17:40:10.941 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:40:10.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:10.944 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:40:11.045 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:40:11.073 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:11.090 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-09T17:40:11.093 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:40:11.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:11.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:40:11.197 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:40:11.225 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:11.244 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:40:11.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:11.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:40:11.256 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:40:11.261 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:40:11.347 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:40:11.374 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:11.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:11 smithi099 ceph-mon[39420]: osdmap e443: 12 total, 12 up, 11 in 2024-02-09T17:40:11.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:11 smithi099 ceph-mon[39420]: pgmap v1146: 37 pgs: 1 active+clean+remapped, 5 peering, 1 active+remapped+backfilling, 30 active+clean; 800 MiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 775 op/s; 539/38385 objects misplaced (1.404%); 9.5 MiB/s, 152 objects/s recovering 2024-02-09T17:40:11.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:11 smithi099 ceph-mon[39420]: osdmap e444: 12 total, 12 up, 11 in 2024-02-09T17:40:12.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:11 smithi179 ceph-mon[43975]: osdmap e443: 12 total, 12 up, 11 in 2024-02-09T17:40:12.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:11 smithi179 ceph-mon[43975]: pgmap v1146: 37 pgs: 1 active+clean+remapped, 5 peering, 1 active+remapped+backfilling, 30 active+clean; 800 MiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 775 op/s; 539/38385 objects misplaced (1.404%); 9.5 MiB/s, 152 objects/s recovering 2024-02-09T17:40:12.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:11 smithi179 ceph-mon[43975]: osdmap e444: 12 total, 12 up, 11 in 2024-02-09T17:40:12.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:11 smithi162 ceph-mon[44304]: osdmap e443: 12 total, 12 up, 11 in 2024-02-09T17:40:12.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:11 smithi162 ceph-mon[44304]: pgmap v1146: 37 pgs: 1 active+clean+remapped, 5 peering, 1 active+remapped+backfilling, 30 active+clean; 800 MiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 775 op/s; 539/38385 objects misplaced (1.404%); 9.5 MiB/s, 152 objects/s recovering 2024-02-09T17:40:12.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:11 smithi162 ceph-mon[44304]: osdmap e444: 12 total, 12 up, 11 in 2024-02-09T17:40:12.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:12 smithi099 ceph-mon[39420]: 4.8 starting backfill to osd.6 from (0'0,0'0] MAX to 441'16482 2024-02-09T17:40:12.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:12 smithi099 ceph-mon[39420]: 4.c starting backfill to osd.6 from (0'0,0'0] MAX to 441'16756 2024-02-09T17:40:12.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:12 smithi099 ceph-mon[39420]: 4.e starting backfill to osd.6 from (0'0,0'0] MAX to 441'16377 2024-02-09T17:40:12.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:12 smithi099 ceph-mon[39420]: 4.5 starting backfill to osd.6 from (0'0,0'0] MAX to 441'16269 2024-02-09T17:40:12.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:12 smithi099 ceph-mon[39420]: 4.a starting backfill to osd.6 from (0'0,0'0] MAX to 441'16221 2024-02-09T17:40:12.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:12 smithi099 ceph-mon[39420]: 4.9 starting backfill to osd.6 from (0'0,0'0] MAX to 441'16440 2024-02-09T17:40:12.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:12 smithi099 ceph-mon[39420]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-02-09T17:40:12.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:12 smithi099 ceph-mon[39420]: osdmap e445: 12 total, 12 up, 11 in 2024-02-09T17:40:13.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:12 smithi179 ceph-mon[43975]: 4.8 starting backfill to osd.6 from (0'0,0'0] MAX to 441'16482 2024-02-09T17:40:13.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:12 smithi179 ceph-mon[43975]: 4.c starting backfill to osd.6 from (0'0,0'0] MAX to 441'16756 2024-02-09T17:40:13.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:12 smithi179 ceph-mon[43975]: 4.e starting backfill to osd.6 from (0'0,0'0] MAX to 441'16377 2024-02-09T17:40:13.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:12 smithi179 ceph-mon[43975]: 4.5 starting backfill to osd.6 from (0'0,0'0] MAX to 441'16269 2024-02-09T17:40:13.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:12 smithi179 ceph-mon[43975]: 4.a starting backfill to osd.6 from (0'0,0'0] MAX to 441'16221 2024-02-09T17:40:13.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:12 smithi179 ceph-mon[43975]: 4.9 starting backfill to osd.6 from (0'0,0'0] MAX to 441'16440 2024-02-09T17:40:13.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:12 smithi179 ceph-mon[43975]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-02-09T17:40:13.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:12 smithi179 ceph-mon[43975]: osdmap e445: 12 total, 12 up, 11 in 2024-02-09T17:40:13.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:12 smithi162 ceph-mon[44304]: 4.8 starting backfill to osd.6 from (0'0,0'0] MAX to 441'16482 2024-02-09T17:40:13.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:12 smithi162 ceph-mon[44304]: 4.c starting backfill to osd.6 from (0'0,0'0] MAX to 441'16756 2024-02-09T17:40:13.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:12 smithi162 ceph-mon[44304]: 4.e starting backfill to osd.6 from (0'0,0'0] MAX to 441'16377 2024-02-09T17:40:13.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:12 smithi162 ceph-mon[44304]: 4.5 starting backfill to osd.6 from (0'0,0'0] MAX to 441'16269 2024-02-09T17:40:13.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:12 smithi162 ceph-mon[44304]: 4.a starting backfill to osd.6 from (0'0,0'0] MAX to 441'16221 2024-02-09T17:40:13.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:12 smithi162 ceph-mon[44304]: 4.9 starting backfill to osd.6 from (0'0,0'0] MAX to 441'16440 2024-02-09T17:40:13.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:12 smithi162 ceph-mon[44304]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-02-09T17:40:13.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:12 smithi162 ceph-mon[44304]: osdmap e445: 12 total, 12 up, 11 in 2024-02-09T17:40:13.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:13 smithi099 ceph-mon[39420]: pgmap v1149: 37 pgs: 1 active+undersized+remapped, 1 active+clean+remapped, 5 peering, 30 active+clean; 766 MiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 725 op/s; 18 MiB/s, 293 objects/s recovering 2024-02-09T17:40:13.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:13 smithi099 ceph-mon[39420]: osdmap e446: 12 total, 12 up, 11 in 2024-02-09T17:40:14.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:13 smithi179 ceph-mon[43975]: pgmap v1149: 37 pgs: 1 active+undersized+remapped, 1 active+clean+remapped, 5 peering, 30 active+clean; 766 MiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 725 op/s; 18 MiB/s, 293 objects/s recovering 2024-02-09T17:40:14.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:13 smithi179 ceph-mon[43975]: osdmap e446: 12 total, 12 up, 11 in 2024-02-09T17:40:14.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:13 smithi162 ceph-mon[44304]: pgmap v1149: 37 pgs: 1 active+undersized+remapped, 1 active+clean+remapped, 5 peering, 30 active+clean; 766 MiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 725 op/s; 18 MiB/s, 293 objects/s recovering 2024-02-09T17:40:14.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:13 smithi162 ceph-mon[44304]: osdmap e446: 12 total, 12 up, 11 in 2024-02-09T17:40:15.339 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:40:15.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:15.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:40:15.445 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:40:15.471 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:15.489 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:40:15.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:15.492 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:40:15.593 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:40:15.620 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:15.637 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:40:15.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:15.640 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:40:15.741 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:40:15.775 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:15.799 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:40:15.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:15.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:40:15.905 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:40:15.954 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:15 smithi099 ceph-mon[39420]: osdmap e447: 12 total, 12 up, 11 in 2024-02-09T17:40:15.954 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:15 smithi099 ceph-mon[39420]: pgmap v1152: 37 pgs: 1 active+remapped+backfilling, 1 active+undersized+remapped, 1 active+clean+remapped, 6 peering, 28 active+clean; 745 MiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 221 op/s; 351/35724 objects misplaced (0.983%); 13 MiB/s, 209 objects/s recovering 2024-02-09T17:40:15.957 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:16.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:15 smithi179 ceph-mon[43975]: osdmap e447: 12 total, 12 up, 11 in 2024-02-09T17:40:16.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:15 smithi179 ceph-mon[43975]: pgmap v1152: 37 pgs: 1 active+remapped+backfilling, 1 active+undersized+remapped, 1 active+clean+remapped, 6 peering, 28 active+clean; 745 MiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 221 op/s; 351/35724 objects misplaced (0.983%); 13 MiB/s, 209 objects/s recovering 2024-02-09T17:40:16.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:15 smithi162 ceph-mon[44304]: osdmap e447: 12 total, 12 up, 11 in 2024-02-09T17:40:16.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:15 smithi162 ceph-mon[44304]: pgmap v1152: 37 pgs: 1 active+remapped+backfilling, 1 active+undersized+remapped, 1 active+clean+remapped, 6 peering, 28 active+clean; 745 MiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 221 op/s; 351/35724 objects misplaced (0.983%); 13 MiB/s, 209 objects/s recovering 2024-02-09T17:40:16.381 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:40:16.612 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:40:16.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:16.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:40:16.718 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:40:16.743 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:16.762 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:40:16.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:16.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:40:16.866 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:40:16.879 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:16 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:40:16.879 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:16 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:40:16.879 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:16 smithi099 ceph-mon[39420]: osdmap e448: 12 total, 12 up, 11 in 2024-02-09T17:40:16.879 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:16 smithi099 ceph-mon[39420]: osdmap e449: 12 total, 12 up, 11 in 2024-02-09T17:40:16.892 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:16.911 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:40:16.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:16.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:40:17.016 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:40:17.045 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:17.148 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:40:17.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:17.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:40:17.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:16 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:40:17.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:16 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:40:17.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:16 smithi179 ceph-mon[43975]: osdmap e448: 12 total, 12 up, 11 in 2024-02-09T17:40:17.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:16 smithi179 ceph-mon[43975]: osdmap e449: 12 total, 12 up, 11 in 2024-02-09T17:40:17.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:16 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:40:17.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:16 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:40:17.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:16 smithi162 ceph-mon[44304]: osdmap e448: 12 total, 12 up, 11 in 2024-02-09T17:40:17.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:16 smithi162 ceph-mon[44304]: osdmap e449: 12 total, 12 up, 11 in 2024-02-09T17:40:17.248 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:40:17.248 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:40:17.248 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:40:17.248 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:40:17.248 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:40:17.248 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:40:17.251 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:40:17.270 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:40:17.282 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:17.290 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:40:17.302 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:40:17.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:17.305 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:40:17.310 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:40:17.331 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:40:17.351 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:40:17.371 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:40:17.372 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:40:17.372 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:40:17.372 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:40:17.372 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:40:17.372 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:40:17.372 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:40:17.373 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:40:17.373 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:40:17.373 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:40:17.373 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:40:17.373 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:40:17.373 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:40:17.405 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:40:17.434 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:17.453 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:40:17.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:17.456 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:40:17.557 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:40:17.599 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:17.638 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:40:17.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:17.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:40:17.744 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:40:17.787 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:17.817 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:40:17.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:17.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:40:17.921 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:40:17.932 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:17 smithi162 ceph-mon[44304]: pgmap v1155: 37 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 3 peering, 30 active+clean; 611 MiB data, 7.0 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 617 op/s; 1126/29322 objects misplaced (3.840%); 41 MiB/s, 650 objects/s recovering 2024-02-09T17:40:17.932 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:17 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3888890821' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:40:17.932 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:17 smithi162 ceph-mon[44304]: osdmap e450: 12 total, 12 up, 11 in 2024-02-09T17:40:17.932 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:17 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4268576865' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-09T17:40:17.989 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:17.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:17 smithi099 ceph-mon[39420]: pgmap v1155: 37 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 3 peering, 30 active+clean; 611 MiB data, 7.0 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 617 op/s; 1126/29322 objects misplaced (3.840%); 41 MiB/s, 650 objects/s recovering 2024-02-09T17:40:17.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:17 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3888890821' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:40:17.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:17 smithi099 ceph-mon[39420]: osdmap e450: 12 total, 12 up, 11 in 2024-02-09T17:40:17.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:17 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4268576865' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-09T17:40:18.020 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:40:18.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:18.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:40:18.130 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:40:18.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:17 smithi179 ceph-mon[43975]: pgmap v1155: 37 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 3 peering, 30 active+clean; 611 MiB data, 7.0 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 617 op/s; 1126/29322 objects misplaced (3.840%); 41 MiB/s, 650 objects/s recovering 2024-02-09T17:40:18.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:17 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3888890821' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:40:18.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:17 smithi179 ceph-mon[43975]: osdmap e450: 12 total, 12 up, 11 in 2024-02-09T17:40:18.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:17 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4268576865' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-09T17:40:18.209 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:18.490 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.7 to 1 (10000) 2024-02-09T17:40:18.764 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:40:18.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:18.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:40:18.787 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:18 smithi099 ceph-mon[39420]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:40:18.787 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:18 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4268576865' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-09T17:40:18.787 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:18 smithi099 ceph-mon[39420]: osdmap e451: 12 total, 12 up, 11 in 2024-02-09T17:40:18.868 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:40:18.895 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:18.913 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:40:18.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:18.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:40:19.017 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:40:19.041 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:18 smithi179 ceph-mon[43975]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:40:19.041 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:18 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4268576865' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-09T17:40:19.042 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:18 smithi179 ceph-mon[43975]: osdmap e451: 12 total, 12 up, 11 in 2024-02-09T17:40:19.043 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:19.062 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:40:19.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:19.064 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:40:19.165 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:40:19.350 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:18 smithi162 ceph-mon[44304]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:40:19.350 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:18 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4268576865' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-09T17:40:19.350 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:18 smithi162 ceph-mon[44304]: osdmap e451: 12 total, 12 up, 11 in 2024-02-09T17:40:19.352 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:40:19.352 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 20, 2024-02-09T17:40:19.352 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 600, 2024-02-09T17:40:19.353 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [ 2024-02-09T17:40:19.353 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.353 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:142473 4.b 4:d5341d23:::benchmark_data_smithi179_103809_object5007:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-02-09T17:40:19.353 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:37:15.431448+0000", 2024-02-09T17:40:19.353 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 183.85599841800001, 2024-02-09T17:40:19.353 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.22891070199999999, 2024-02-09T17:40:19.353 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:40:19.353 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:40:19.353 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:40:19.354 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:40:19.354 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:40:19.354 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 142473 2024-02-09T17:40:19.354 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.354 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:40:19.354 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.354 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:40:19.354 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.431448+0000", 2024-02-09T17:40:19.354 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:40:19.354 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.354 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.355 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:40:19.355 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.431448+0000", 2024-02-09T17:40:19.355 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:40:19.355 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.355 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.355 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:40:19.355 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.431450+0000", 2024-02-09T17:40:19.355 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.79e-06 2024-02-09T17:40:19.355 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.355 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.355 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:40:19.356 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.431458+0000", 2024-02-09T17:40:19.356 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.8269999999999998e-06 2024-02-09T17:40:19.356 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.356 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.356 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:40:19.356 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.431479+0000", 2024-02-09T17:40:19.356 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.1160000000000001e-05 2024-02-09T17:40:19.356 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.356 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.356 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:40:19.356 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.431489+0000", 2024-02-09T17:40:19.357 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0558e-05 2024-02-09T17:40:19.357 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.357 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.357 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:40:19.357 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.433160+0000", 2024-02-09T17:40:19.357 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0016701369999999999 2024-02-09T17:40:19.357 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.357 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.357 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:40:19.358 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.433300+0000", 2024-02-09T17:40:19.358 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014065700000000001 2024-02-09T17:40:19.358 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.358 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.358 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:40:19.358 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.433419+0000", 2024-02-09T17:40:19.358 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00011865400000000001 2024-02-09T17:40:19.358 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.358 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.358 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:40:19.358 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.447885+0000", 2024-02-09T17:40:19.358 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.014466494 2024-02-09T17:40:19.359 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.359 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.359 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:40:19.359 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.450722+0000", 2024-02-09T17:40:19.359 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0028368159999999998 2024-02-09T17:40:19.359 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.359 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.359 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:40:19.359 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.660293+0000", 2024-02-09T17:40:19.359 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20957071599999999 2024-02-09T17:40:19.359 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.360 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.360 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:40:19.361 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.660330+0000", 2024-02-09T17:40:19.361 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.7305000000000003e-05 2024-02-09T17:40:19.361 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.361 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.361 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:40:19.361 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.660359+0000", 2024-02-09T17:40:19.361 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.8588000000000001e-05 2024-02-09T17:40:19.361 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:40:19.362 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:40:19.362 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:40:19.362 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.362 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.362 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:142479 4.b 4:dba015d4:::benchmark_data_smithi179_103809_object5013:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-02-09T17:40:19.362 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:37:15.435623+0000", 2024-02-09T17:40:19.362 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 183.85182345300001, 2024-02-09T17:40:19.363 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.22484269500000001, 2024-02-09T17:40:19.363 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:40:19.363 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:40:19.363 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:40:19.363 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:40:19.363 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:40:19.363 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 142479 2024-02-09T17:40:19.363 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.363 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:40:19.363 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.363 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:40:19.363 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.435623+0000", 2024-02-09T17:40:19.363 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:40:19.364 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.364 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.364 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:40:19.364 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.435623+0000", 2024-02-09T17:40:19.364 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:40:19.364 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.364 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.364 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:40:19.364 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.435625+0000", 2024-02-09T17:40:19.364 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.826e-06 2024-02-09T17:40:19.364 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.364 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.364 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:40:19.364 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.435631+0000", 2024-02-09T17:40:19.365 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.998e-06 2024-02-09T17:40:19.365 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.365 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.365 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:40:19.365 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.435649+0000", 2024-02-09T17:40:19.365 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.7958e-05 2024-02-09T17:40:19.365 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.365 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.365 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:40:19.365 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.435659+0000", 2024-02-09T17:40:19.365 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.7100000000000002e-06 2024-02-09T17:40:19.365 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.365 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.365 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:40:19.365 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.437577+0000", 2024-02-09T17:40:19.366 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0019182590000000001 2024-02-09T17:40:19.366 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.366 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.366 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:40:19.366 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.437717+0000", 2024-02-09T17:40:19.366 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000139844 2024-02-09T17:40:19.366 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.366 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.366 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:40:19.366 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.437832+0000", 2024-02-09T17:40:19.366 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00011568100000000001 2024-02-09T17:40:19.366 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.366 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.366 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:40:19.367 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.447892+0000", 2024-02-09T17:40:19.367 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.010059462 2024-02-09T17:40:19.367 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.367 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.367 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:40:19.367 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.450758+0000", 2024-02-09T17:40:19.367 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0028664929999999999 2024-02-09T17:40:19.367 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.367 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.367 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:40:19.367 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.660407+0000", 2024-02-09T17:40:19.367 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.209648365 2024-02-09T17:40:19.367 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.367 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.367 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:40:19.368 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.660442+0000", 2024-02-09T17:40:19.368 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.5188000000000003e-05 2024-02-09T17:40:19.368 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.368 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.368 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:40:19.368 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.660466+0000", 2024-02-09T17:40:19.368 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.3910999999999999e-05 2024-02-09T17:40:19.368 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:40:19.368 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:40:19.368 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:40:19.368 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.368 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.368 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:142480 4.b 4:dadb69b5:::benchmark_data_smithi179_103809_object5014:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-02-09T17:40:19.368 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:37:15.439167+0000", 2024-02-09T17:40:19.369 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 183.848279738, 2024-02-09T17:40:19.369 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.22137198999999999, 2024-02-09T17:40:19.369 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:40:19.369 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:40:19.369 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:40:19.369 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:40:19.369 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:40:19.369 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 142480 2024-02-09T17:40:19.369 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.369 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:40:19.369 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.369 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:40:19.369 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.439167+0000", 2024-02-09T17:40:19.369 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:40:19.370 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.370 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.370 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:40:19.370 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.439167+0000", 2024-02-09T17:40:19.370 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:40:19.370 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.370 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.370 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:40:19.370 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.439168+0000", 2024-02-09T17:40:19.370 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.403e-06 2024-02-09T17:40:19.370 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.370 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.370 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:40:19.370 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.439173+0000", 2024-02-09T17:40:19.370 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.8389999999999997e-06 2024-02-09T17:40:19.371 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.371 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.371 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:40:19.371 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.439185+0000", 2024-02-09T17:40:19.371 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2191000000000001e-05 2024-02-09T17:40:19.371 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.371 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.371 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:40:19.371 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.439194+0000", 2024-02-09T17:40:19.371 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.4970000000000001e-06 2024-02-09T17:40:19.371 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.371 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.371 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:40:19.371 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.441558+0000", 2024-02-09T17:40:19.372 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0023641280000000001 2024-02-09T17:40:19.372 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.372 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.372 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:40:19.372 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.441695+0000", 2024-02-09T17:40:19.372 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000136911 2024-02-09T17:40:19.372 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.372 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.372 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:40:19.372 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.441814+0000", 2024-02-09T17:40:19.372 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000119448 2024-02-09T17:40:19.372 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.372 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.372 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:40:19.372 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.449253+0000", 2024-02-09T17:40:19.373 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0074392319999999996 2024-02-09T17:40:19.373 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.373 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.373 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:40:19.373 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.463169+0000", 2024-02-09T17:40:19.373 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.013915195 2024-02-09T17:40:19.373 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.373 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.373 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:40:19.373 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.660491+0000", 2024-02-09T17:40:19.373 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.19732286199999999 2024-02-09T17:40:19.373 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.373 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.373 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:40:19.374 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.660519+0000", 2024-02-09T17:40:19.374 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.7861e-05 2024-02-09T17:40:19.374 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.374 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.374 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:40:19.374 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.660539+0000", 2024-02-09T17:40:19.374 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.9423e-05 2024-02-09T17:40:19.374 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:40:19.374 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:40:19.374 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:40:19.374 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.374 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.374 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:142502 4.b 4:d9f571e1:::benchmark_data_smithi179_103809_object5036:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-02-09T17:40:19.374 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:37:15.462075+0000", 2024-02-09T17:40:19.374 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 183.82537161400001, 2024-02-09T17:40:19.375 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.21471936999999999, 2024-02-09T17:40:19.375 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:40:19.375 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:40:19.375 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:40:19.375 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:40:19.375 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:40:19.375 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 142502 2024-02-09T17:40:19.375 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.375 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:40:19.375 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.375 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:40:19.375 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.462075+0000", 2024-02-09T17:40:19.375 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:40:19.375 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.376 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.376 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:40:19.376 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.462075+0000", 2024-02-09T17:40:19.376 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:40:19.376 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.376 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.376 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:40:19.376 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.462077+0000", 2024-02-09T17:40:19.376 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.6729999999999999e-06 2024-02-09T17:40:19.376 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.376 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.376 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:40:19.376 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.462082+0000", 2024-02-09T17:40:19.376 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.8749999999999997e-06 2024-02-09T17:40:19.377 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.377 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.377 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:40:19.377 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.462097+0000", 2024-02-09T17:40:19.377 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.437e-05 2024-02-09T17:40:19.377 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.377 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.377 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:40:19.377 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.462106+0000", 2024-02-09T17:40:19.377 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.9800000000000004e-06 2024-02-09T17:40:19.377 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.377 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.377 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:40:19.377 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.463204+0000", 2024-02-09T17:40:19.378 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0010978069999999999 2024-02-09T17:40:19.378 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.378 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.378 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:40:19.378 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.463352+0000", 2024-02-09T17:40:19.378 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014882099999999999 2024-02-09T17:40:19.378 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.378 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.378 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:40:19.378 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.463477+0000", 2024-02-09T17:40:19.378 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012462500000000001 2024-02-09T17:40:19.378 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.378 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.378 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:40:19.378 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.467898+0000", 2024-02-09T17:40:19.379 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0044207159999999999 2024-02-09T17:40:19.379 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.379 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.379 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:40:19.379 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.473913+0000", 2024-02-09T17:40:19.379 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0060150890000000004 2024-02-09T17:40:19.379 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.379 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.379 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:40:19.379 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.676693+0000", 2024-02-09T17:40:19.379 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.202780556 2024-02-09T17:40:19.379 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.379 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.379 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:40:19.379 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.676750+0000", 2024-02-09T17:40:19.380 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7012000000000002e-05 2024-02-09T17:40:19.380 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.380 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.380 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:40:19.380 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.676794+0000", 2024-02-09T17:40:19.380 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.3846e-05 2024-02-09T17:40:19.380 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:40:19.380 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:40:19.380 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:40:19.380 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.380 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.380 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:142521 4.b 4:db739e59:::benchmark_data_smithi179_103809_object5055:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-02-09T17:40:19.380 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:37:15.480646+0000", 2024-02-09T17:40:19.380 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 183.806800643, 2024-02-09T17:40:19.381 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.200893504, 2024-02-09T17:40:19.381 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:40:19.381 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:40:19.381 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:40:19.381 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:40:19.381 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:40:19.381 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 142521 2024-02-09T17:40:19.381 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.381 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:40:19.381 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.381 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:40:19.381 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.480646+0000", 2024-02-09T17:40:19.381 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:40:19.381 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.382 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.382 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:40:19.382 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.480646+0000", 2024-02-09T17:40:19.382 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:40:19.382 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.382 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.382 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:40:19.382 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.480647+0000", 2024-02-09T17:40:19.382 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.607e-06 2024-02-09T17:40:19.382 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.382 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.382 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:40:19.382 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.480655+0000", 2024-02-09T17:40:19.382 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.0260000000000007e-06 2024-02-09T17:40:19.383 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.383 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.383 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:40:19.383 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.480672+0000", 2024-02-09T17:40:19.383 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.6790999999999999e-05 2024-02-09T17:40:19.383 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.383 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.383 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:40:19.383 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.480683+0000", 2024-02-09T17:40:19.383 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.062e-05 2024-02-09T17:40:19.383 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.383 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.383 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:40:19.383 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.480744+0000", 2024-02-09T17:40:19.384 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.1344000000000001e-05 2024-02-09T17:40:19.384 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.384 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.384 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:40:19.384 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.480895+0000", 2024-02-09T17:40:19.384 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00015104100000000001 2024-02-09T17:40:19.384 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.384 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.384 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:40:19.384 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.481014+0000", 2024-02-09T17:40:19.384 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00011828999999999999 2024-02-09T17:40:19.384 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.384 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.384 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:40:19.385 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.484952+0000", 2024-02-09T17:40:19.385 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0039388050000000001 2024-02-09T17:40:19.385 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.385 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.385 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:40:19.385 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.488401+0000", 2024-02-09T17:40:19.385 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0034482369999999998 2024-02-09T17:40:19.385 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.385 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.385 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:40:19.385 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.681462+0000", 2024-02-09T17:40:19.385 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.19306173500000001 2024-02-09T17:40:19.385 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.385 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.386 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:40:19.386 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.681498+0000", 2024-02-09T17:40:19.386 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.5176999999999999e-05 2024-02-09T17:40:19.386 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.386 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.386 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:40:19.386 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.681539+0000", 2024-02-09T17:40:19.386 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.1831e-05 2024-02-09T17:40:19.386 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:40:19.386 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:40:19.386 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:40:19.386 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.386 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.386 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:142536 4.b 4:d405018c:::benchmark_data_smithi179_103809_object5070:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-02-09T17:40:19.386 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:37:15.494013+0000", 2024-02-09T17:40:19.387 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 183.79343307400001, 2024-02-09T17:40:19.387 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.18767062100000001, 2024-02-09T17:40:19.387 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:40:19.387 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:40:19.387 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:40:19.387 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:40:19.387 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:40:19.387 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 142536 2024-02-09T17:40:19.387 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.387 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:40:19.388 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.388 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:40:19.388 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.494013+0000", 2024-02-09T17:40:19.388 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:40:19.388 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.388 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.388 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:40:19.388 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.494013+0000", 2024-02-09T17:40:19.388 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:40:19.388 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.388 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.388 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:40:19.389 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.494015+0000", 2024-02-09T17:40:19.389 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.668e-06 2024-02-09T17:40:19.389 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.389 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.389 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:40:19.389 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.494020+0000", 2024-02-09T17:40:19.389 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1100000000000002e-06 2024-02-09T17:40:19.389 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.389 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.389 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:40:19.389 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.494030+0000", 2024-02-09T17:40:19.389 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.6090000000000007e-06 2024-02-09T17:40:19.389 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.389 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.390 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:40:19.390 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.494036+0000", 2024-02-09T17:40:19.390 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7529999999999998e-06 2024-02-09T17:40:19.390 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.390 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.390 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:40:19.390 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.494089+0000", 2024-02-09T17:40:19.390 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3791e-05 2024-02-09T17:40:19.390 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.390 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.390 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:40:19.390 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.494201+0000", 2024-02-09T17:40:19.390 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000111827 2024-02-09T17:40:19.390 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.390 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.391 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:40:19.391 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.494286+0000", 2024-02-09T17:40:19.391 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.5203000000000004e-05 2024-02-09T17:40:19.391 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.391 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.391 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:40:19.391 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.502425+0000", 2024-02-09T17:40:19.391 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0081390779999999992 2024-02-09T17:40:19.391 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.391 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.391 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:40:19.391 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.505478+0000", 2024-02-09T17:40:19.391 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0030527060000000001 2024-02-09T17:40:19.392 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.392 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.392 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:40:19.392 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.681586+0000", 2024-02-09T17:40:19.392 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.17610826600000001 2024-02-09T17:40:19.392 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.392 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.392 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:40:19.392 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.681643+0000", 2024-02-09T17:40:19.392 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6558999999999998e-05 2024-02-09T17:40:19.392 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.392 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.392 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:40:19.392 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.681684+0000", 2024-02-09T17:40:19.392 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.1050999999999997e-05 2024-02-09T17:40:19.393 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:40:19.393 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:40:19.393 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:40:19.393 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.393 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.393 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:142560 4.b 4:d2b72131:::benchmark_data_smithi179_103809_object5094:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-02-09T17:40:19.393 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:37:15.523291+0000", 2024-02-09T17:40:19.393 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 183.76415524000001, 2024-02-09T17:40:19.393 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.165307342, 2024-02-09T17:40:19.393 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:40:19.393 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:40:19.393 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:40:19.393 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:40:19.394 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:40:19.394 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 142560 2024-02-09T17:40:19.394 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.394 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:40:19.394 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.394 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:40:19.394 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.523291+0000", 2024-02-09T17:40:19.394 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:40:19.394 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.394 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.394 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:40:19.394 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.523291+0000", 2024-02-09T17:40:19.394 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:40:19.394 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.395 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.395 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:40:19.395 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.523293+0000", 2024-02-09T17:40:19.395 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.6139999999999999e-06 2024-02-09T17:40:19.395 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.395 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.395 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:40:19.395 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.523298+0000", 2024-02-09T17:40:19.395 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2780000000000001e-06 2024-02-09T17:40:19.395 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.395 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.395 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:40:19.395 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.523312+0000", 2024-02-09T17:40:19.396 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3987e-05 2024-02-09T17:40:19.396 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.396 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.396 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:40:19.396 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.523321+0000", 2024-02-09T17:40:19.396 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.6449999999999994e-06 2024-02-09T17:40:19.396 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.396 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.396 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:40:19.396 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.523383+0000", 2024-02-09T17:40:19.396 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.1929999999999998e-05 2024-02-09T17:40:19.396 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.396 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.396 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:40:19.397 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.523527+0000", 2024-02-09T17:40:19.397 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000143924 2024-02-09T17:40:19.397 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.397 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.397 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:40:19.397 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.523649+0000", 2024-02-09T17:40:19.397 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012264599999999999 2024-02-09T17:40:19.397 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.397 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.397 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:40:19.397 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.531462+0000", 2024-02-09T17:40:19.397 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0078131569999999994 2024-02-09T17:40:19.397 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.397 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.398 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:40:19.398 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.532213+0000", 2024-02-09T17:40:19.398 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00075038000000000004 2024-02-09T17:40:19.398 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.398 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.398 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:40:19.398 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.688507+0000", 2024-02-09T17:40:19.398 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.156293984 2024-02-09T17:40:19.398 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.398 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.398 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:40:19.398 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.688559+0000", 2024-02-09T17:40:19.398 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2525999999999998e-05 2024-02-09T17:40:19.398 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.398 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.399 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:40:19.399 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.688599+0000", 2024-02-09T17:40:19.399 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9270999999999997e-05 2024-02-09T17:40:19.399 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:40:19.399 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:40:19.399 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:40:19.399 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.399 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.399 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:142574 4.b 4:d276a6a8:::benchmark_data_smithi179_103809_object5108:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-02-09T17:40:19.399 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:37:15.537260+0000", 2024-02-09T17:40:19.399 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 183.750186095, 2024-02-09T17:40:19.399 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.15147265600000001, 2024-02-09T17:40:19.399 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:40:19.399 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:40:19.400 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:40:19.400 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:40:19.400 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:40:19.400 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 142574 2024-02-09T17:40:19.400 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.400 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:40:19.400 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.400 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:40:19.400 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.537260+0000", 2024-02-09T17:40:19.400 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:40:19.400 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.400 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.400 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:40:19.400 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.537260+0000", 2024-02-09T17:40:19.401 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:40:19.401 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.401 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.401 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:40:19.401 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.537262+0000", 2024-02-09T17:40:19.401 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4479999999999999e-06 2024-02-09T17:40:19.401 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.401 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.401 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:40:19.401 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.537266+0000", 2024-02-09T17:40:19.401 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.5199999999999999e-06 2024-02-09T17:40:19.401 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.401 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.401 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:40:19.401 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.537281+0000", 2024-02-09T17:40:19.402 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4245e-05 2024-02-09T17:40:19.402 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.402 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.402 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:40:19.402 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.537289+0000", 2024-02-09T17:40:19.402 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.1689999999999992e-06 2024-02-09T17:40:19.402 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.402 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.402 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:40:19.402 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.537347+0000", 2024-02-09T17:40:19.402 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.8223000000000001e-05 2024-02-09T17:40:19.402 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.402 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.402 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:40:19.403 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.537477+0000", 2024-02-09T17:40:19.403 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000130479 2024-02-09T17:40:19.403 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.403 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.403 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:40:19.403 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.537590+0000", 2024-02-09T17:40:19.403 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000112932 2024-02-09T17:40:19.403 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.403 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.403 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:40:19.403 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.547075+0000", 2024-02-09T17:40:19.403 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0094847660000000004 2024-02-09T17:40:19.403 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.403 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.403 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:40:19.404 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.550800+0000", 2024-02-09T17:40:19.404 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.003724879 2024-02-09T17:40:19.404 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.404 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.404 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:40:19.404 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.688643+0000", 2024-02-09T17:40:19.404 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.13784332099999999 2024-02-09T17:40:19.404 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.404 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.404 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:40:19.404 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.688696+0000", 2024-02-09T17:40:19.404 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2500000000000002e-05 2024-02-09T17:40:19.404 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.404 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.405 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:40:19.405 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.688733+0000", 2024-02-09T17:40:19.405 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.7174e-05 2024-02-09T17:40:19.405 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:40:19.405 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:40:19.405 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:40:19.405 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.405 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.405 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:142579 4.b 4:da85fca5:::benchmark_data_smithi179_103809_object5113:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-02-09T17:40:19.405 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:37:15.542523+0000", 2024-02-09T17:40:19.405 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 183.744923371, 2024-02-09T17:40:19.405 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.15209349699999999, 2024-02-09T17:40:19.405 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:40:19.405 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:40:19.405 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:40:19.406 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:40:19.406 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:40:19.406 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 142579 2024-02-09T17:40:19.406 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.406 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:40:19.406 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.406 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:40:19.406 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.542523+0000", 2024-02-09T17:40:19.406 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:40:19.406 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.406 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.406 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:40:19.406 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.542523+0000", 2024-02-09T17:40:19.406 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:40:19.407 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.407 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.407 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:40:19.407 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.542525+0000", 2024-02-09T17:40:19.407 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.643e-06 2024-02-09T17:40:19.407 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.407 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.407 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:40:19.407 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.542529+0000", 2024-02-09T17:40:19.407 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2969999999999997e-06 2024-02-09T17:40:19.407 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.407 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.408 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:40:19.408 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.542544+0000", 2024-02-09T17:40:19.408 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4474999999999999e-05 2024-02-09T17:40:19.408 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.408 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.408 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:40:19.408 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.542552+0000", 2024-02-09T17:40:19.408 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.7320000000000004e-06 2024-02-09T17:40:19.408 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.408 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.408 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:40:19.408 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.542610+0000", 2024-02-09T17:40:19.409 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7713999999999997e-05 2024-02-09T17:40:19.409 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.409 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.409 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:40:19.409 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.542736+0000", 2024-02-09T17:40:19.409 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012584099999999999 2024-02-09T17:40:19.409 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.409 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.409 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:40:19.409 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.542850+0000", 2024-02-09T17:40:19.409 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000113986 2024-02-09T17:40:19.409 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.409 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.409 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:40:19.409 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.547232+0000", 2024-02-09T17:40:19.410 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0043821950000000002 2024-02-09T17:40:19.410 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.410 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.410 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:40:19.410 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.550927+0000", 2024-02-09T17:40:19.410 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.003695411 2024-02-09T17:40:19.410 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.410 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.410 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:40:19.410 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.694523+0000", 2024-02-09T17:40:19.410 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.143595955 2024-02-09T17:40:19.410 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.410 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.410 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:40:19.411 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.694577+0000", 2024-02-09T17:40:19.411 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3545000000000003e-05 2024-02-09T17:40:19.411 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.411 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.411 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:40:19.411 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.694617+0000", 2024-02-09T17:40:19.411 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9703000000000003e-05 2024-02-09T17:40:19.411 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:40:19.411 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:40:19.411 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:40:19.411 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.411 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.411 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:142585 4.b 4:d9b09195:::benchmark_data_smithi179_103809_object5119:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-02-09T17:40:19.411 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:37:15.551256+0000", 2024-02-09T17:40:19.411 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 183.736190513, 2024-02-09T17:40:19.412 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.143511992, 2024-02-09T17:40:19.412 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:40:19.412 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:40:19.412 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:40:19.412 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:40:19.412 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:40:19.412 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 142585 2024-02-09T17:40:19.412 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.412 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:40:19.412 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.412 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:40:19.412 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.551256+0000", 2024-02-09T17:40:19.412 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:40:19.412 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.413 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.413 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:40:19.413 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.551256+0000", 2024-02-09T17:40:19.413 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:40:19.413 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.413 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.413 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:40:19.413 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.551257+0000", 2024-02-09T17:40:19.413 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.11e-06 2024-02-09T17:40:19.413 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.413 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.413 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:40:19.414 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.551261+0000", 2024-02-09T17:40:19.414 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.4910000000000001e-06 2024-02-09T17:40:19.414 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.414 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.414 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:40:19.414 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.551273+0000", 2024-02-09T17:40:19.414 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2717999999999999e-05 2024-02-09T17:40:19.414 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.414 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.414 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:40:19.414 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.551281+0000", 2024-02-09T17:40:19.414 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.8110000000000003e-06 2024-02-09T17:40:19.415 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.415 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.415 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:40:19.415 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.551325+0000", 2024-02-09T17:40:19.415 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.4338999999999997e-05 2024-02-09T17:40:19.415 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.415 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.415 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:40:19.415 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.551439+0000", 2024-02-09T17:40:19.415 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00011325299999999999 2024-02-09T17:40:19.415 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.415 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.415 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:40:19.415 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.551536+0000", 2024-02-09T17:40:19.415 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.7312000000000006e-05 2024-02-09T17:40:19.416 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.416 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.416 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:40:19.416 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.555946+0000", 2024-02-09T17:40:19.416 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0044101080000000003 2024-02-09T17:40:19.416 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.416 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.416 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:40:19.416 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.556017+0000", 2024-02-09T17:40:19.416 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.1048000000000001e-05 2024-02-09T17:40:19.416 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.416 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.417 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:40:19.417 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.694675+0000", 2024-02-09T17:40:19.417 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.13865815100000001 2024-02-09T17:40:19.417 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.417 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.417 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:40:19.417 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.694730+0000", 2024-02-09T17:40:19.417 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.4379000000000002e-05 2024-02-09T17:40:19.417 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.417 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.417 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:40:19.417 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.694768+0000", 2024-02-09T17:40:19.417 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.8272000000000002e-05 2024-02-09T17:40:19.417 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:40:19.418 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:40:19.418 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:40:19.418 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.418 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.418 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:229736 4.b 4:d699b73b:::benchmark_data_smithi179_103809_object92270:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e409)", 2024-02-09T17:40:19.418 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:39:17.539047+0000", 2024-02-09T17:40:19.418 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 61.748399143999997, 2024-02-09T17:40:19.418 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.25139220699999998, 2024-02-09T17:40:19.418 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:40:19.418 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:40:19.418 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:40:19.418 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:40:19.418 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:40:19.419 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 229736 2024-02-09T17:40:19.419 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.419 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:40:19.419 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.419 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:40:19.419 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.539047+0000", 2024-02-09T17:40:19.419 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:40:19.419 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.419 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.419 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:40:19.419 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.539047+0000", 2024-02-09T17:40:19.419 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:40:19.419 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.419 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.420 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:40:19.420 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.539048+0000", 2024-02-09T17:40:19.420 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.4299999999999998e-07 2024-02-09T17:40:19.420 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.420 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.420 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:40:19.420 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.539051+0000", 2024-02-09T17:40:19.420 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.7499999999999999e-06 2024-02-09T17:40:19.420 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.420 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.420 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:40:19.420 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.539058+0000", 2024-02-09T17:40:19.420 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.4390000000000003e-06 2024-02-09T17:40:19.420 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.421 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.421 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:40:19.421 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.539063+0000", 2024-02-09T17:40:19.421 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.4770000000000002e-06 2024-02-09T17:40:19.421 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.421 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.421 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:40:19.421 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.554843+0000", 2024-02-09T17:40:19.421 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.015779313999999999 2024-02-09T17:40:19.421 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.422 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.422 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:40:19.422 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.554981+0000", 2024-02-09T17:40:19.422 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013780699999999999 2024-02-09T17:40:19.422 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.422 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.422 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:40:19.422 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.555101+0000", 2024-02-09T17:40:19.422 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012060799999999999 2024-02-09T17:40:19.422 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.422 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.422 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:40:19.422 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.562887+0000", 2024-02-09T17:40:19.423 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0077858459999999999 2024-02-09T17:40:19.423 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.423 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.423 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:40:19.423 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.565544+0000", 2024-02-09T17:40:19.423 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0026572309999999999 2024-02-09T17:40:19.423 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.423 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.423 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:40:19.423 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.790321+0000", 2024-02-09T17:40:19.423 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.224776637 2024-02-09T17:40:19.423 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.423 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.423 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:40:19.423 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.790393+0000", 2024-02-09T17:40:19.424 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.2350999999999999e-05 2024-02-09T17:40:19.424 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.424 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.424 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:40:19.424 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.790440+0000", 2024-02-09T17:40:19.424 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.6304000000000003e-05 2024-02-09T17:40:19.424 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:40:19.424 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:40:19.424 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:40:19.424 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.424 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.424 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:229764 4.b 4:dd3428d9:::benchmark_data_smithi179_103809_object92298:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e409)", 2024-02-09T17:40:19.424 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:39:17.585994+0000", 2024-02-09T17:40:19.424 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 61.701452560999996, 2024-02-09T17:40:19.425 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.204602479, 2024-02-09T17:40:19.425 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:40:19.425 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:40:19.425 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:40:19.425 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:40:19.425 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:40:19.425 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 229764 2024-02-09T17:40:19.425 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.425 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:40:19.425 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.425 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:40:19.425 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.585994+0000", 2024-02-09T17:40:19.425 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:40:19.425 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.425 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.426 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:40:19.426 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.585994+0000", 2024-02-09T17:40:19.426 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:40:19.426 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.426 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.426 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:40:19.426 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.585996+0000", 2024-02-09T17:40:19.426 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.702e-06 2024-02-09T17:40:19.426 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.426 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.426 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:40:19.426 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.586002+0000", 2024-02-09T17:40:19.426 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.4659999999999996e-06 2024-02-09T17:40:19.426 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.426 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.427 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:40:19.427 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.586017+0000", 2024-02-09T17:40:19.427 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4928e-05 2024-02-09T17:40:19.427 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.427 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.427 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:40:19.427 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.586025+0000", 2024-02-09T17:40:19.427 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.96e-06 2024-02-09T17:40:19.427 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.427 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.427 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:40:19.427 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.586077+0000", 2024-02-09T17:40:19.427 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1788e-05 2024-02-09T17:40:19.427 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.427 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.428 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:40:19.428 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.586206+0000", 2024-02-09T17:40:19.428 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012875899999999999 2024-02-09T17:40:19.428 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.428 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.428 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:40:19.428 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.586324+0000", 2024-02-09T17:40:19.428 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000118486 2024-02-09T17:40:19.428 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.428 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.428 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:40:19.428 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.591097+0000", 2024-02-09T17:40:19.428 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0047726829999999998 2024-02-09T17:40:19.428 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.429 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.429 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:40:19.429 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.591679+0000", 2024-02-09T17:40:19.429 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000582747 2024-02-09T17:40:19.429 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.429 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.429 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:40:19.429 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.790499+0000", 2024-02-09T17:40:19.429 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.19881927199999999 2024-02-09T17:40:19.429 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.429 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.429 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:40:19.429 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.790557+0000", 2024-02-09T17:40:19.429 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.8235000000000001e-05 2024-02-09T17:40:19.429 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.430 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.430 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:40:19.430 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.790596+0000", 2024-02-09T17:40:19.430 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9453000000000003e-05 2024-02-09T17:40:19.430 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:40:19.430 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:40:19.430 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:40:19.430 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.430 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.430 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:229767 4.b 4:d03aef11:::benchmark_data_smithi179_103809_object92301:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e409)", 2024-02-09T17:40:19.430 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:39:17.593089+0000", 2024-02-09T17:40:19.430 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 61.694357124, 2024-02-09T17:40:19.430 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.19766324599999999, 2024-02-09T17:40:19.430 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:40:19.431 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:40:19.431 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:40:19.431 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:40:19.431 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:40:19.431 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 229767 2024-02-09T17:40:19.431 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.431 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:40:19.431 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.431 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:40:19.431 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.593089+0000", 2024-02-09T17:40:19.431 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:40:19.431 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.431 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.431 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:40:19.431 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.593089+0000", 2024-02-09T17:40:19.432 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:40:19.432 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.432 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.432 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:40:19.432 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.593091+0000", 2024-02-09T17:40:19.432 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.412e-06 2024-02-09T17:40:19.432 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.432 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.432 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:40:19.432 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.593095+0000", 2024-02-09T17:40:19.433 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.386e-06 2024-02-09T17:40:19.433 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.433 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.433 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:40:19.433 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.593107+0000", 2024-02-09T17:40:19.433 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.15e-05 2024-02-09T17:40:19.433 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.433 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.433 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:40:19.433 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.593113+0000", 2024-02-09T17:40:19.434 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.7599999999999997e-06 2024-02-09T17:40:19.434 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.434 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.434 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:40:19.434 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.593167+0000", 2024-02-09T17:40:19.434 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3773999999999997e-05 2024-02-09T17:40:19.434 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.434 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.434 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:40:19.434 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.593287+0000", 2024-02-09T17:40:19.434 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000119323 2024-02-09T17:40:19.434 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.434 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.434 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:40:19.435 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.593364+0000", 2024-02-09T17:40:19.435 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.7312999999999997e-05 2024-02-09T17:40:19.435 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.435 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.435 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:40:19.435 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.597528+0000", 2024-02-09T17:40:19.435 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00416447 2024-02-09T17:40:19.435 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.435 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.435 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:40:19.435 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.603409+0000", 2024-02-09T17:40:19.435 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0058809559999999997 2024-02-09T17:40:19.436 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.436 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.436 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:40:19.436 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.790660+0000", 2024-02-09T17:40:19.436 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.18725065199999999 2024-02-09T17:40:19.436 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.436 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.436 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:40:19.436 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.790714+0000", 2024-02-09T17:40:19.436 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.4014000000000002e-05 2024-02-09T17:40:19.436 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.436 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.436 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:40:19.436 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.790753+0000", 2024-02-09T17:40:19.436 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.8686000000000002e-05 2024-02-09T17:40:19.437 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:40:19.437 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:40:19.437 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:40:19.437 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.437 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.437 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:229801 4.b 4:db62d972:::benchmark_data_smithi179_103809_object92335:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e409)", 2024-02-09T17:40:19.437 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:39:17.651766+0000", 2024-02-09T17:40:19.437 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 61.635680440000002, 2024-02-09T17:40:19.437 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.13929277200000001, 2024-02-09T17:40:19.437 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:40:19.437 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:40:19.437 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:40:19.437 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:40:19.437 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:40:19.437 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 229801 2024-02-09T17:40:19.438 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.438 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:40:19.438 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.438 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:40:19.438 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.651766+0000", 2024-02-09T17:40:19.438 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:40:19.438 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.438 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.438 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:40:19.438 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.651766+0000", 2024-02-09T17:40:19.438 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:40:19.438 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.438 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.439 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:40:19.439 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.651767+0000", 2024-02-09T17:40:19.439 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.4600000000000004e-07 2024-02-09T17:40:19.439 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.439 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.439 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:40:19.439 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.651769+0000", 2024-02-09T17:40:19.439 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.5289999999999998e-06 2024-02-09T17:40:19.439 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.439 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.439 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:40:19.439 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.651775+0000", 2024-02-09T17:40:19.439 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6359999999999999e-06 2024-02-09T17:40:19.439 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.439 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.440 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:40:19.440 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.651778+0000", 2024-02-09T17:40:19.440 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.4649999999999999e-06 2024-02-09T17:40:19.440 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.440 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.440 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:40:19.440 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.656787+0000", 2024-02-09T17:40:19.440 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0050083769999999996 2024-02-09T17:40:19.440 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.440 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.440 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:40:19.440 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.656862+0000", 2024-02-09T17:40:19.440 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.5763e-05 2024-02-09T17:40:19.440 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.441 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.441 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:40:19.441 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.656973+0000", 2024-02-09T17:40:19.441 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00011006300000000001 2024-02-09T17:40:19.441 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.441 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.441 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:40:19.441 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.661536+0000", 2024-02-09T17:40:19.441 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0045634999999999998 2024-02-09T17:40:19.441 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.441 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.441 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:40:19.441 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.661879+0000", 2024-02-09T17:40:19.441 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00034313399999999998 2024-02-09T17:40:19.442 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.442 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.442 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:40:19.442 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.790972+0000", 2024-02-09T17:40:19.442 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.12909321000000001 2024-02-09T17:40:19.442 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.442 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.442 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:40:19.442 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.791026+0000", 2024-02-09T17:40:19.442 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.4128000000000001e-05 2024-02-09T17:40:19.442 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.442 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.442 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:40:19.442 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.791059+0000", 2024-02-09T17:40:19.442 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.2320999999999998e-05 2024-02-09T17:40:19.443 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:40:19.443 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:40:19.443 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:40:19.443 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.443 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.443 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:232553 4.7 4:e6318b68:::benchmark_data_smithi179_103809_object95087:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e415)", 2024-02-09T17:40:19.443 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:39:24.529349+0000", 2024-02-09T17:40:19.443 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 54.758097554999999, 2024-02-09T17:40:19.443 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.259789582, 2024-02-09T17:40:19.443 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:40:19.443 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:40:19.443 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:40:19.443 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:40:19.443 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:40:19.443 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 232553 2024-02-09T17:40:19.444 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.444 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:40:19.444 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.444 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:40:19.444 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529349+0000", 2024-02-09T17:40:19.444 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:40:19.444 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.444 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.444 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:40:19.444 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529349+0000", 2024-02-09T17:40:19.444 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:40:19.444 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.444 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.444 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:40:19.445 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529351+0000", 2024-02-09T17:40:19.445 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.9400000000000001e-06 2024-02-09T17:40:19.445 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.445 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.445 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:40:19.445 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529360+0000", 2024-02-09T17:40:19.445 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.6640000000000004e-06 2024-02-09T17:40:19.445 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.445 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.445 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:40:19.445 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529381+0000", 2024-02-09T17:40:19.445 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.1449000000000001e-05 2024-02-09T17:40:19.445 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.445 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.445 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:40:19.446 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529404+0000", 2024-02-09T17:40:19.446 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.3328000000000001e-05 2024-02-09T17:40:19.446 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.446 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.446 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:40:19.446 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529502+0000", 2024-02-09T17:40:19.446 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.7256999999999994e-05 2024-02-09T17:40:19.446 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.446 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.446 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op must wait for map", 2024-02-09T17:40:19.446 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529509+0000", 2024-02-09T17:40:19.446 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.7610000000000007e-06 2024-02-09T17:40:19.446 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.446 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.446 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:40:19.447 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.531763+0000", 2024-02-09T17:40:19.447 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0022537740000000001 2024-02-09T17:40:19.447 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.447 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.447 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:40:19.447 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.531774+0000", 2024-02-09T17:40:19.447 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0967e-05 2024-02-09T17:40:19.447 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.447 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.447 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:40:19.447 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.773951+0000", 2024-02-09T17:40:19.447 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2421772010000001 2024-02-09T17:40:19.447 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.447 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.448 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:40:19.448 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.777956+0000", 2024-02-09T17:40:19.448 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0040046150000000004 2024-02-09T17:40:19.448 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.448 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.448 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,7,10", 2024-02-09T17:40:19.448 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.778105+0000", 2024-02-09T17:40:19.448 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014957499999999999 2024-02-09T17:40:19.448 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.448 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.448 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:40:19.448 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.783408+0000", 2024-02-09T17:40:19.448 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.005302106 2024-02-09T17:40:19.448 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.448 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.449 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:40:19.449 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.783425+0000", 2024-02-09T17:40:19.449 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.7003e-05 2024-02-09T17:40:19.449 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.449 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.449 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:40:19.449 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.783488+0000", 2024-02-09T17:40:19.449 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.3232999999999996e-05 2024-02-09T17:40:19.449 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.449 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.449 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:40:19.449 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.789011+0000", 2024-02-09T17:40:19.449 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.005523315 2024-02-09T17:40:19.449 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.450 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.450 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:40:19.450 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.789066+0000", 2024-02-09T17:40:19.450 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.4656e-05 2024-02-09T17:40:19.450 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.450 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.450 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:40:19.450 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.789139+0000", 2024-02-09T17:40:19.450 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.2737999999999994e-05 2024-02-09T17:40:19.450 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:40:19.450 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:40:19.450 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:40:19.450 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.451 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.451 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:232579 4.7 4:eafda101:::benchmark_data_smithi179_103809_object95113:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e415)", 2024-02-09T17:40:19.451 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:39:24.529431+0000", 2024-02-09T17:40:19.451 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 54.758015792000002, 2024-02-09T17:40:19.451 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2695841749999999, 2024-02-09T17:40:19.451 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:40:19.451 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:40:19.451 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:40:19.451 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:40:19.451 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:40:19.451 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 232579 2024-02-09T17:40:19.451 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.451 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:40:19.451 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.451 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:40:19.452 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529431+0000", 2024-02-09T17:40:19.452 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:40:19.452 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.452 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.452 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:40:19.452 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529431+0000", 2024-02-09T17:40:19.452 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:40:19.452 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.452 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.452 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:40:19.452 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529431+0000", 2024-02-09T17:40:19.452 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6400000000000002e-07 2024-02-09T17:40:19.452 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.452 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.453 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:40:19.453 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529435+0000", 2024-02-09T17:40:19.453 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.3739999999999998e-06 2024-02-09T17:40:19.453 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.453 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.453 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:40:19.453 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529443+0000", 2024-02-09T17:40:19.453 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.4279999999999996e-06 2024-02-09T17:40:19.453 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.453 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.453 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:40:19.453 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529450+0000", 2024-02-09T17:40:19.453 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.5400000000000001e-06 2024-02-09T17:40:19.453 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.454 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.454 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:40:19.454 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529563+0000", 2024-02-09T17:40:19.454 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0001131 2024-02-09T17:40:19.454 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.454 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.454 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting_for_map not empty", 2024-02-09T17:40:19.454 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529568+0000", 2024-02-09T17:40:19.454 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2920000000000003e-06 2024-02-09T17:40:19.454 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.454 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.454 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:40:19.454 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.532793+0000", 2024-02-09T17:40:19.454 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0032253260000000001 2024-02-09T17:40:19.454 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.455 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.455 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:40:19.455 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.532803+0000", 2024-02-09T17:40:19.455 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.2010000000000005e-06 2024-02-09T17:40:19.455 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.455 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.455 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:40:19.455 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.783535+0000", 2024-02-09T17:40:19.455 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2507324470000001 2024-02-09T17:40:19.455 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.455 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.455 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:40:19.455 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.783654+0000", 2024-02-09T17:40:19.455 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00011907400000000001 2024-02-09T17:40:19.455 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.456 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.456 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,7,10", 2024-02-09T17:40:19.456 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.783808+0000", 2024-02-09T17:40:19.456 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000153976 2024-02-09T17:40:19.456 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.456 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.456 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:40:19.456 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.789175+0000", 2024-02-09T17:40:19.456 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0053669319999999996 2024-02-09T17:40:19.456 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.456 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.456 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:40:19.456 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.789183+0000", 2024-02-09T17:40:19.456 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.9359999999999999e-06 2024-02-09T17:40:19.457 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.457 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.457 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:40:19.457 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.798870+0000", 2024-02-09T17:40:19.457 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0096870970000000004 2024-02-09T17:40:19.457 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.457 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.457 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:40:19.457 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.798910+0000", 2024-02-09T17:40:19.457 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.0018999999999997e-05 2024-02-09T17:40:19.457 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.457 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.457 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:40:19.458 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.798951+0000", 2024-02-09T17:40:19.458 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.1349e-05 2024-02-09T17:40:19.458 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.458 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.458 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:40:19.458 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.799015+0000", 2024-02-09T17:40:19.458 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.3520000000000002e-05 2024-02-09T17:40:19.458 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:40:19.458 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:40:19.458 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:40:19.458 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.458 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.458 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:232588 4.7 4:e414a3eb:::benchmark_data_smithi179_103809_object95122:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-02-09T17:40:19.458 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:39:24.612490+0000", 2024-02-09T17:40:19.458 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 54.674955994000001, 2024-02-09T17:40:19.459 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.196787614, 2024-02-09T17:40:19.459 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:40:19.459 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:40:19.459 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:40:19.459 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:40:19.459 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:40:19.459 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 232588 2024-02-09T17:40:19.459 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.459 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:40:19.459 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.459 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:40:19.459 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.612490+0000", 2024-02-09T17:40:19.459 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:40:19.459 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.459 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.460 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:40:19.460 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.612490+0000", 2024-02-09T17:40:19.460 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:40:19.460 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.460 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.460 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:40:19.460 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.612492+0000", 2024-02-09T17:40:19.460 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.234e-06 2024-02-09T17:40:19.460 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.460 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.460 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:40:19.460 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.612497+0000", 2024-02-09T17:40:19.460 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1959999999999999e-06 2024-02-09T17:40:19.460 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.461 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.461 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:40:19.461 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.612507+0000", 2024-02-09T17:40:19.461 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.04e-05 2024-02-09T17:40:19.461 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.461 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.461 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:40:19.461 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.612513+0000", 2024-02-09T17:40:19.461 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.0340000000000002e-06 2024-02-09T17:40:19.461 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.461 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.461 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:40:19.461 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.612566+0000", 2024-02-09T17:40:19.461 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2352000000000003e-05 2024-02-09T17:40:19.461 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.462 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.462 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:40:19.462 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.612577+0000", 2024-02-09T17:40:19.462 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0886e-05 2024-02-09T17:40:19.462 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.462 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.462 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:40:19.462 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.789202+0000", 2024-02-09T17:40:19.462 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1766251320000001 2024-02-09T17:40:19.462 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.462 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.462 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:40:19.462 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.789317+0000", 2024-02-09T17:40:19.462 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000115123 2024-02-09T17:40:19.462 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.463 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.463 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,7,10", 2024-02-09T17:40:19.463 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.789408+0000", 2024-02-09T17:40:19.463 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.1421000000000004e-05 2024-02-09T17:40:19.463 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.463 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.463 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:40:19.463 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.799027+0000", 2024-02-09T17:40:19.463 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0096182330000000003 2024-02-09T17:40:19.463 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.463 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.463 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:40:19.463 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.799152+0000", 2024-02-09T17:40:19.463 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012501999999999999 2024-02-09T17:40:19.463 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.464 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.464 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:40:19.464 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.809067+0000", 2024-02-09T17:40:19.464 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.009915119 2024-02-09T17:40:19.464 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.464 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.464 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:40:19.464 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.809128+0000", 2024-02-09T17:40:19.464 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.1671e-05 2024-02-09T17:40:19.464 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.464 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.464 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:40:19.464 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.809164+0000", 2024-02-09T17:40:19.464 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.5988000000000002e-05 2024-02-09T17:40:19.465 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.465 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.465 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:40:19.465 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.809278+0000", 2024-02-09T17:40:19.465 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000113805 2024-02-09T17:40:19.465 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:40:19.465 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:40:19.465 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:40:19.465 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.465 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.465 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:232589 4.7 4:e37473dd:::benchmark_data_smithi179_103809_object95123:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-02-09T17:40:19.465 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:39:24.619917+0000", 2024-02-09T17:40:19.465 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 54.667529530000003, 2024-02-09T17:40:19.465 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.189582264, 2024-02-09T17:40:19.466 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:40:19.466 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:40:19.466 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:40:19.466 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:40:19.466 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:40:19.466 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 232589 2024-02-09T17:40:19.466 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.466 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:40:19.466 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.466 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:40:19.466 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.619917+0000", 2024-02-09T17:40:19.466 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:40:19.466 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.466 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.466 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:40:19.467 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.619917+0000", 2024-02-09T17:40:19.467 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:40:19.467 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.467 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.467 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:40:19.467 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.619919+0000", 2024-02-09T17:40:19.467 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.4099999999999998e-06 2024-02-09T17:40:19.467 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.467 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.467 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:40:19.467 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.619928+0000", 2024-02-09T17:40:19.467 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.3219999999999993e-06 2024-02-09T17:40:19.467 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.467 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.468 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:40:19.468 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.619949+0000", 2024-02-09T17:40:19.468 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.1160000000000001e-05 2024-02-09T17:40:19.468 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.468 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.468 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:40:19.468 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.619960+0000", 2024-02-09T17:40:19.468 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1287000000000001e-05 2024-02-09T17:40:19.468 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.468 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.468 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:40:19.468 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.620034+0000", 2024-02-09T17:40:19.468 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.4089000000000002e-05 2024-02-09T17:40:19.468 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.468 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.469 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:40:19.469 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.620048+0000", 2024-02-09T17:40:19.469 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3447e-05 2024-02-09T17:40:19.469 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.469 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.469 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:40:19.469 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.794239+0000", 2024-02-09T17:40:19.469 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.174191516 2024-02-09T17:40:19.469 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.469 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.469 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:40:19.469 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.794361+0000", 2024-02-09T17:40:19.469 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000122256 2024-02-09T17:40:19.469 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.469 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.470 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,7,10", 2024-02-09T17:40:19.470 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.794542+0000", 2024-02-09T17:40:19.470 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00018047899999999999 2024-02-09T17:40:19.470 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.470 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.470 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:40:19.470 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.799034+0000", 2024-02-09T17:40:19.470 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0044917710000000003 2024-02-09T17:40:19.470 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.470 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.470 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:40:19.470 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.809098+0000", 2024-02-09T17:40:19.470 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.010064619 2024-02-09T17:40:19.470 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.470 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.471 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:40:19.471 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.809308+0000", 2024-02-09T17:40:19.471 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00020926500000000001 2024-02-09T17:40:19.471 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.471 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.471 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:40:19.471 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.809355+0000", 2024-02-09T17:40:19.471 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.7741999999999998e-05 2024-02-09T17:40:19.471 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.471 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.471 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:40:19.471 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.809385+0000", 2024-02-09T17:40:19.471 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.9709e-05 2024-02-09T17:40:19.471 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.471 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.472 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:40:19.472 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.809499+0000", 2024-02-09T17:40:19.472 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00011419199999999999 2024-02-09T17:40:19.472 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:40:19.472 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:40:19.472 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:40:19.472 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.472 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.472 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:232593 4.7 4:e255490f:::benchmark_data_smithi179_103809_object95127:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-02-09T17:40:19.472 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:39:25.605375+0000", 2024-02-09T17:40:19.472 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 53.682071979, 2024-02-09T17:40:19.472 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.22923427800000001, 2024-02-09T17:40:19.472 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:40:19.472 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:40:19.473 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:40:19.473 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:40:19.473 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:40:19.473 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 232593 2024-02-09T17:40:19.473 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.473 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:40:19.473 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.473 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:40:19.473 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.605375+0000", 2024-02-09T17:40:19.473 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:40:19.473 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.473 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.473 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:40:19.473 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.605375+0000", 2024-02-09T17:40:19.473 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:40:19.474 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.474 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.474 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:40:19.474 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.605376+0000", 2024-02-09T17:40:19.474 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.719e-06 2024-02-09T17:40:19.474 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.474 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.474 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:40:19.474 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.605382+0000", 2024-02-09T17:40:19.474 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3149999999999999e-06 2024-02-09T17:40:19.474 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.474 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.474 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:40:19.474 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.605407+0000", 2024-02-09T17:40:19.474 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.4998999999999999e-05 2024-02-09T17:40:19.474 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.475 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.475 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:40:19.475 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.605416+0000", 2024-02-09T17:40:19.475 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.4069999999999999e-06 2024-02-09T17:40:19.475 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.475 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.475 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:40:19.475 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.606684+0000", 2024-02-09T17:40:19.475 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0012678240000000001 2024-02-09T17:40:19.475 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.475 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.475 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:40:19.475 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.606697+0000", 2024-02-09T17:40:19.475 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2877999999999999e-05 2024-02-09T17:40:19.475 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.476 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.476 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:40:19.476 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.799193+0000", 2024-02-09T17:40:19.476 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.19249639599999999 2024-02-09T17:40:19.476 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.476 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.476 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:40:19.476 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.799327+0000", 2024-02-09T17:40:19.476 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013401999999999999 2024-02-09T17:40:19.476 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.476 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.476 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,7,10", 2024-02-09T17:40:19.476 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.799498+0000", 2024-02-09T17:40:19.476 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000171137 2024-02-09T17:40:19.476 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.477 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.477 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:40:19.477 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.809075+0000", 2024-02-09T17:40:19.477 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0095770730000000002 2024-02-09T17:40:19.477 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.477 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.477 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:40:19.477 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.809331+0000", 2024-02-09T17:40:19.477 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00025584099999999998 2024-02-09T17:40:19.477 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.477 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.477 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:40:19.477 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.820212+0000", 2024-02-09T17:40:19.477 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.010881165 2024-02-09T17:40:19.478 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.478 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.478 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:40:19.478 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.834245+0000", 2024-02-09T17:40:19.478 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.014032263999999999 2024-02-09T17:40:19.478 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.478 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.478 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:40:19.478 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.834492+0000", 2024-02-09T17:40:19.478 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00024776699999999997 2024-02-09T17:40:19.478 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.478 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.478 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:40:19.478 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.834609+0000", 2024-02-09T17:40:19.478 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00011647299999999999 2024-02-09T17:40:19.479 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:40:19.479 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:40:19.479 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:40:19.479 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.479 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.479 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:232606 4.7 4:eec97097:::benchmark_data_smithi179_103809_object95140:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-02-09T17:40:19.479 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:39:25.642341+0000", 2024-02-09T17:40:19.479 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 53.645105205999997, 2024-02-09T17:40:19.479 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.19250051600000001, 2024-02-09T17:40:19.479 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:40:19.479 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:40:19.479 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:40:19.479 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:40:19.479 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:40:19.480 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 232606 2024-02-09T17:40:19.480 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.480 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:40:19.480 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.480 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:40:19.480 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.642341+0000", 2024-02-09T17:40:19.480 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:40:19.480 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.480 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.480 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:40:19.480 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.642341+0000", 2024-02-09T17:40:19.480 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:40:19.480 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.480 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.481 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:40:19.481 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.642343+0000", 2024-02-09T17:40:19.481 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.863e-06 2024-02-09T17:40:19.481 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.481 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.481 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:40:19.481 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.642350+0000", 2024-02-09T17:40:19.481 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.2860000000000002e-06 2024-02-09T17:40:19.481 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.481 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.481 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:40:19.481 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.642368+0000", 2024-02-09T17:40:19.481 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.7841000000000001e-05 2024-02-09T17:40:19.481 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.481 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.482 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:40:19.482 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.642379+0000", 2024-02-09T17:40:19.482 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1088e-05 2024-02-09T17:40:19.482 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.482 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.482 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:40:19.482 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.643440+0000", 2024-02-09T17:40:19.482 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.001060213 2024-02-09T17:40:19.482 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.482 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.482 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:40:19.482 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.643451+0000", 2024-02-09T17:40:19.482 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1416999999999999e-05 2024-02-09T17:40:19.482 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.482 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.482 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:40:19.483 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.804558+0000", 2024-02-09T17:40:19.483 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.16110685799999999 2024-02-09T17:40:19.483 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.483 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.483 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:40:19.483 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.804663+0000", 2024-02-09T17:40:19.483 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000105327 2024-02-09T17:40:19.483 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.483 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.483 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,7,10", 2024-02-09T17:40:19.483 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.804806+0000", 2024-02-09T17:40:19.483 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000143251 2024-02-09T17:40:19.483 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.483 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.483 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:40:19.484 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.820076+0000", 2024-02-09T17:40:19.484 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.015270064999999999 2024-02-09T17:40:19.484 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.484 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.484 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:40:19.484 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.820143+0000", 2024-02-09T17:40:19.484 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.6710000000000003e-05 2024-02-09T17:40:19.484 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.484 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.484 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:40:19.484 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.820327+0000", 2024-02-09T17:40:19.484 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00018374099999999999 2024-02-09T17:40:19.484 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.484 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.485 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:40:19.485 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.834665+0000", 2024-02-09T17:40:19.485 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.014338404000000001 2024-02-09T17:40:19.485 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.485 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.485 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:40:19.485 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.834721+0000", 2024-02-09T17:40:19.485 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.5779999999999998e-05 2024-02-09T17:40:19.485 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:40:19.485 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:40:19.485 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:40:19.485 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.834842+0000", 2024-02-09T17:40:19.485 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000120672 2024-02-09T17:40:19.485 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:40:19.485 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:40:19.485 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:40:19.486 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:40:19.486 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:40:19.486 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:40:19.514 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:19.532 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:40:19.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:19.535 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:40:19.636 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:40:19.661 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:19.684 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:40:19.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:19.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:40:19.791 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:40:19.818 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:19 smithi179 ceph-mon[43975]: pgmap v1158: 37 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 peering, 31 active+clean; 591 MiB data, 6.8 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 614 op/s; 1126/28353 objects misplaced (3.971%); 36 MiB/s, 570 objects/s recovering 2024-02-09T17:40:19.818 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:19 smithi179 ceph-mon[43975]: osdmap e452: 12 total, 12 up, 11 in 2024-02-09T17:40:19.821 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:19.845 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:40:19.848 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:19.848 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:40:19.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:19 smithi162 ceph-mon[44304]: pgmap v1158: 37 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 peering, 31 active+clean; 591 MiB data, 6.8 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 614 op/s; 1126/28353 objects misplaced (3.971%); 36 MiB/s, 570 objects/s recovering 2024-02-09T17:40:19.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:19 smithi162 ceph-mon[44304]: osdmap e452: 12 total, 12 up, 11 in 2024-02-09T17:40:19.949 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:40:19.980 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:20.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:19 smithi099 ceph-mon[39420]: pgmap v1158: 37 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 peering, 31 active+clean; 591 MiB data, 6.8 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 614 op/s; 1126/28353 objects misplaced (3.971%); 36 MiB/s, 570 objects/s recovering 2024-02-09T17:40:20.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:19 smithi099 ceph-mon[39420]: osdmap e452: 12 total, 12 up, 11 in 2024-02-09T17:40:20.309 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 8 1 2024-02-09T17:40:20.789 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:40:21.175 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:20 smithi179 ceph-mon[43975]: 4.8 continuing backfill to osd.6 from (347'6700,450'16721] MIN to 450'16721 2024-02-09T17:40:21.175 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:20 smithi179 ceph-mon[43975]: 4.8 starting backfill to osd.7 from (338'4400,408'14401] MIN to 450'16721 2024-02-09T17:40:21.175 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:20 smithi179 ceph-mon[43975]: osdmap e453: 12 total, 12 up, 11 in 2024-02-09T17:40:21.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:20 smithi162 ceph-mon[44304]: 4.8 continuing backfill to osd.6 from (347'6700,450'16721] MIN to 450'16721 2024-02-09T17:40:21.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:20 smithi162 ceph-mon[44304]: 4.8 starting backfill to osd.7 from (338'4400,408'14401] MIN to 450'16721 2024-02-09T17:40:21.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:20 smithi162 ceph-mon[44304]: osdmap e453: 12 total, 12 up, 11 in 2024-02-09T17:40:21.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:20 smithi099 ceph-mon[39420]: 4.8 continuing backfill to osd.6 from (347'6700,450'16721] MIN to 450'16721 2024-02-09T17:40:21.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:20 smithi099 ceph-mon[39420]: 4.8 starting backfill to osd.7 from (338'4400,408'14401] MIN to 450'16721 2024-02-09T17:40:21.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:20 smithi099 ceph-mon[39420]: osdmap e453: 12 total, 12 up, 11 in 2024-02-09T17:40:22.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:21 smithi179 ceph-mon[43975]: 4.0 starting backfill to osd.7 from (339'4500,412'14542] MIN to 450'16722 2024-02-09T17:40:22.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:21 smithi179 ceph-mon[43975]: pgmap v1161: 37 pgs: 1 active+undersized+remapped, 2 active+remapped+backfilling, 1 peering, 33 active+clean; 536 MiB data, 6.6 GiB used, 977 GiB / 983 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 304 op/s; 981/25707 objects misplaced (3.816%); 5.2 MiB/s, 82 objects/s recovering 2024-02-09T17:40:22.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:21 smithi179 ceph-mon[43975]: osdmap e454: 12 total, 12 up, 11 in 2024-02-09T17:40:22.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:21 smithi162 ceph-mon[44304]: 4.0 starting backfill to osd.7 from (339'4500,412'14542] MIN to 450'16722 2024-02-09T17:40:22.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:21 smithi162 ceph-mon[44304]: pgmap v1161: 37 pgs: 1 active+undersized+remapped, 2 active+remapped+backfilling, 1 peering, 33 active+clean; 536 MiB data, 6.6 GiB used, 977 GiB / 983 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 304 op/s; 981/25707 objects misplaced (3.816%); 5.2 MiB/s, 82 objects/s recovering 2024-02-09T17:40:22.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:21 smithi162 ceph-mon[44304]: osdmap e454: 12 total, 12 up, 11 in 2024-02-09T17:40:22.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:21 smithi099 ceph-mon[39420]: 4.0 starting backfill to osd.7 from (339'4500,412'14542] MIN to 450'16722 2024-02-09T17:40:22.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:21 smithi099 ceph-mon[39420]: pgmap v1161: 37 pgs: 1 active+undersized+remapped, 2 active+remapped+backfilling, 1 peering, 33 active+clean; 536 MiB data, 6.6 GiB used, 977 GiB / 983 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 304 op/s; 981/25707 objects misplaced (3.816%); 5.2 MiB/s, 82 objects/s recovering 2024-02-09T17:40:22.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:21 smithi099 ceph-mon[39420]: osdmap e454: 12 total, 12 up, 11 in 2024-02-09T17:40:22.504 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:40:22.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:22.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:40:22.609 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:40:22.669 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:22.713 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:40:22.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:22.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:40:22.823 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:40:22.880 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:22.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:22 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:40:22.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:22 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:40:22.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:22 smithi162 ceph-mon[44304]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-09T17:40:22.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:22 smithi162 ceph-mon[44304]: osdmap e455: 12 total, 12 up, 11 in 2024-02-09T17:40:23.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:22 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:40:23.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:22 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:40:23.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:22 smithi179 ceph-mon[43975]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-09T17:40:23.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:22 smithi179 ceph-mon[43975]: osdmap e455: 12 total, 12 up, 11 in 2024-02-09T17:40:23.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:22 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:40:23.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:22 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:40:23.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:22 smithi099 ceph-mon[39420]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-09T17:40:23.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:22 smithi099 ceph-mon[39420]: osdmap e455: 12 total, 12 up, 11 in 2024-02-09T17:40:23.273 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:40:23.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:23.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:40:23.305 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-09T17:40:23.379 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:40:23.415 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:23.441 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:40:23.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:23.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:40:23.546 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:40:23.596 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:23.628 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:40:23.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:23.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:40:23.734 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:40:23.770 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:23.795 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:40:23.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:23.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:40:23.899 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:40:23.992 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:24.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:23 smithi179 ceph-mon[43975]: pgmap v1164: 37 pgs: 1 active+remapped+backfill_toofull, 2 peering, 1 active+remapped+backfilling, 33 active+clean; 496 MiB data, 6.4 GiB used, 977 GiB / 983 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 384 op/s; 990/23775 objects misplaced (4.164%); 12 MiB/s, 185 objects/s recovering 2024-02-09T17:40:24.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:23 smithi162 ceph-mon[44304]: pgmap v1164: 37 pgs: 1 active+remapped+backfill_toofull, 2 peering, 1 active+remapped+backfilling, 33 active+clean; 496 MiB data, 6.4 GiB used, 977 GiB / 983 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 384 op/s; 990/23775 objects misplaced (4.164%); 12 MiB/s, 185 objects/s recovering 2024-02-09T17:40:24.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:23 smithi099 ceph-mon[39420]: pgmap v1164: 37 pgs: 1 active+remapped+backfill_toofull, 2 peering, 1 active+remapped+backfilling, 33 active+clean; 496 MiB data, 6.4 GiB used, 977 GiB / 983 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 384 op/s; 990/23775 objects misplaced (4.164%); 12 MiB/s, 185 objects/s recovering 2024-02-09T17:40:24.355 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:40:24.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:24.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:40:24.374 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:40:24.385 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:40:24.458 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:40:24.491 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:24.517 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:40:24.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:24.521 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:40:24.622 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:40:24.658 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:24.679 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:40:24.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:24.682 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:40:24.783 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:40:24.824 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:24.850 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:40:24.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:24.853 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:40:24.944 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:24 smithi099 ceph-mon[39420]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:40:24.953 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:40:25.009 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:25.100 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:40:25.112 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:25.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:40:25.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:24 smithi179 ceph-mon[43975]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:40:25.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:24 smithi162 ceph-mon[44304]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:40:25.213 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:40:25.257 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:25.544 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:40:25.828 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is set 2024-02-09T17:40:26.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:25 smithi179 ceph-mon[43975]: pgmap v1165: 37 pgs: 1 active+remapped+backfill_toofull, 2 peering, 1 active+remapped+backfilling, 33 active+clean; 488 MiB data, 6.3 GiB used, 977 GiB / 983 GiB avail; 965 B/s rd, 25 KiB/s wr, 292 op/s; 990/23400 objects misplaced (4.231%); 8.7 MiB/s, 139 objects/s recovering 2024-02-09T17:40:26.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:25 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3441419303' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:40:26.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:25 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:40:26.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:25 smithi162 ceph-mon[44304]: pgmap v1165: 37 pgs: 1 active+remapped+backfill_toofull, 2 peering, 1 active+remapped+backfilling, 33 active+clean; 488 MiB data, 6.3 GiB used, 977 GiB / 983 GiB avail; 965 B/s rd, 25 KiB/s wr, 292 op/s; 990/23400 objects misplaced (4.231%); 8.7 MiB/s, 139 objects/s recovering 2024-02-09T17:40:26.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:25 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3441419303' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:40:26.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:25 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:40:26.234 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:40:26.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:26.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:40:26.240 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:25 smithi099 ceph-mon[39420]: pgmap v1165: 37 pgs: 1 active+remapped+backfill_toofull, 2 peering, 1 active+remapped+backfilling, 33 active+clean; 488 MiB data, 6.3 GiB used, 977 GiB / 983 GiB avail; 965 B/s rd, 25 KiB/s wr, 292 op/s; 990/23400 objects misplaced (4.231%); 8.7 MiB/s, 139 objects/s recovering 2024-02-09T17:40:26.240 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:25 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3441419303' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:40:26.241 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:25 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:40:26.341 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:40:26.401 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:26.442 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:40:26.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:26.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:40:26.550 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:40:26.600 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:26.877 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:40:26.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:26.887 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:26 smithi099 ceph-mon[39420]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:40:26.887 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:26 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:40:26.887 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:26 smithi099 ceph-mon[39420]: osdmap e456: 12 total, 12 up, 11 in 2024-02-09T17:40:26.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:40:26.988 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:40:27.030 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:27.060 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:40:27.064 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:26 smithi179 ceph-mon[43975]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:40:27.064 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:26 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:40:27.064 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:26 smithi179 ceph-mon[43975]: osdmap e456: 12 total, 12 up, 11 in 2024-02-09T17:40:27.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:27.065 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:40:27.166 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:40:27.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:26 smithi162 ceph-mon[44304]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:40:27.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:26 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:40:27.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:26 smithi162 ceph-mon[44304]: osdmap e456: 12 total, 12 up, 11 in 2024-02-09T17:40:27.222 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:27.612 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:40:27.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:27.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:40:27.726 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:40:27.780 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:27.817 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:40:27.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:27.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:40:27.875 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:27 smithi099 ceph-mon[39420]: pgmap v1167: 37 pgs: 1 active+remapped+backfill_toofull, 2 peering, 34 active+clean; 314 MiB data, 5.9 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 591 op/s; 424/15054 objects misplaced (2.817%); 7.2 MiB/s, 114 objects/s recovering 2024-02-09T17:40:27.927 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:40:27.988 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:28.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:27 smithi179 ceph-mon[43975]: pgmap v1167: 37 pgs: 1 active+remapped+backfill_toofull, 2 peering, 34 active+clean; 314 MiB data, 5.9 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 591 op/s; 424/15054 objects misplaced (2.817%); 7.2 MiB/s, 114 objects/s recovering 2024-02-09T17:40:28.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:27 smithi162 ceph-mon[44304]: pgmap v1167: 37 pgs: 1 active+remapped+backfill_toofull, 2 peering, 34 active+clean; 314 MiB data, 5.9 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 591 op/s; 424/15054 objects misplaced (2.817%); 7.2 MiB/s, 114 objects/s recovering 2024-02-09T17:40:29.598 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:40:29.619 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:40:29.620 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:40:29.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:29.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:40:29.637 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset noscrub 2024-02-09T17:40:29.662 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:40:29.697 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:40:29.728 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:40:29.735 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:40:29.795 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:40:29.823 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:29.825 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:40:29.865 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.8 to 1 (10000) 2024-02-09T17:40:29.875 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:40:29.919 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:40:29.967 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:40:30.013 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:40:30.041 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:40:30.050 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:40:30.050 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:40:30.050 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:40:30.050 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:40:30.050 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:40:30.051 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:40:30.051 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:40:30.051 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:40:30.051 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:40:30.051 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:40:30.051 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:40:30.051 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:40:30.082 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:29 smithi099 ceph-mon[39420]: pgmap v1168: 37 pgs: 1 active+remapped+backfill_toofull, 36 active+clean; 284 MiB data, 5.6 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 529 op/s; 201/13593 objects misplaced (1.479%); 2.4 MiB/s, 38 objects/s recovering 2024-02-09T17:40:30.082 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:29 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3161105053' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-02-09T17:40:30.082 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:29 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-02-09T17:40:30.082 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:29 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/720883257' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:40:30.175 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:29 smithi179 ceph-mon[43975]: pgmap v1168: 37 pgs: 1 active+remapped+backfill_toofull, 36 active+clean; 284 MiB data, 5.6 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 529 op/s; 201/13593 objects misplaced (1.479%); 2.4 MiB/s, 38 objects/s recovering 2024-02-09T17:40:30.175 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:29 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3161105053' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-02-09T17:40:30.176 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:29 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-02-09T17:40:30.176 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:29 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/720883257' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:40:30.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:29 smithi162 ceph-mon[44304]: pgmap v1168: 37 pgs: 1 active+remapped+backfill_toofull, 36 active+clean; 284 MiB data, 5.6 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 529 op/s; 201/13593 objects misplaced (1.479%); 2.4 MiB/s, 38 objects/s recovering 2024-02-09T17:40:30.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:29 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3161105053' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-02-09T17:40:30.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:29 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-02-09T17:40:30.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:29 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/720883257' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:40:30.955 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Removed 137463 objects 2024-02-09T17:40:30.955 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Clean up completed and total clean up time :201.22 2024-02-09T17:40:31.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:30 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-02-09T17:40:31.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:30 smithi179 ceph-mon[43975]: osdmap e457: 12 total, 12 up, 11 in 2024-02-09T17:40:31.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:30 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-02-09T17:40:31.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:30 smithi162 ceph-mon[44304]: osdmap e457: 12 total, 12 up, 11 in 2024-02-09T17:40:31.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:30 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-02-09T17:40:31.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:30 smithi099 ceph-mon[39420]: osdmap e457: 12 total, 12 up, 11 in 2024-02-09T17:40:31.520 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:40:31.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:31.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:40:31.624 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:40:31.651 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:31.671 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:40:31.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:31.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:40:31.775 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:40:31.807 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:31.826 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:40:31.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:31.829 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:40:31.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:31 smithi162 ceph-mon[44304]: pgmap v1170: 37 pgs: 1 active+remapped+backfill_toofull, 36 active+clean; 111 MiB data, 5.2 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 769 op/s; 201/5304 objects misplaced (3.790%); 2.2 MiB/s, 34 objects/s recovering 2024-02-09T17:40:31.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:31 smithi162 ceph-mon[44304]: osdmap e458: 12 total, 12 up, 11 in 2024-02-09T17:40:31.930 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:40:31.931 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 9 1 2024-02-09T17:40:31.958 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:32.154 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:40:32.155 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:40:32.157 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:40:32.158 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:31 smithi099 ceph-mon[39420]: pgmap v1170: 37 pgs: 1 active+remapped+backfill_toofull, 36 active+clean; 111 MiB data, 5.2 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 769 op/s; 201/5304 objects misplaced (3.790%); 2.2 MiB/s, 34 objects/s recovering 2024-02-09T17:40:32.158 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:31 smithi099 ceph-mon[39420]: osdmap e458: 12 total, 12 up, 11 in 2024-02-09T17:40:32.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:32.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:40:32.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:31 smithi179 ceph-mon[43975]: pgmap v1170: 37 pgs: 1 active+remapped+backfill_toofull, 36 active+clean; 111 MiB data, 5.2 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 769 op/s; 201/5304 objects misplaced (3.790%); 2.2 MiB/s, 34 objects/s recovering 2024-02-09T17:40:32.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:31 smithi179 ceph-mon[43975]: osdmap e458: 12 total, 12 up, 11 in 2024-02-09T17:40:32.261 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:40:32.291 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:32.310 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:40:32.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:32.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:40:32.414 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:40:32.446 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:32.467 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:40:32.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:32.470 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:40:32.530 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-02-09T17:40:32.530 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-02-09T17:40:32.571 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:40:32.604 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:32.625 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:40:32.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:32.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:40:32.729 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:40:32.760 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:32.781 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:40:32.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:32.784 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:40:32.835 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:40:32.885 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:40:32.911 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:33.575 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:40:33.610 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-09T17:40:34.108 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:33 smithi099 ceph-mon[39420]: pgmap v1172: 37 pgs: 1 active+remapped+backfill_toofull, 36 active+clean; 23 MiB data, 4.9 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 796 op/s 2024-02-09T17:40:34.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:33 smithi179 ceph-mon[43975]: pgmap v1172: 37 pgs: 1 active+remapped+backfill_toofull, 36 active+clean; 23 MiB data, 4.9 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 796 op/s 2024-02-09T17:40:34.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:33 smithi162 ceph-mon[44304]: pgmap v1172: 37 pgs: 1 active+remapped+backfill_toofull, 36 active+clean; 23 MiB data, 4.9 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 796 op/s 2024-02-09T17:40:34.825 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:40:35.662 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:40:35.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:35.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:40:35.766 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:40:35.799 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:35.820 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:40:35.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:35.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:40:35.925 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:40:35.955 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:35.976 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:40:35.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:35.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:40:36.080 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:40:36.112 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:36.133 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:40:36.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:36.137 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:40:36.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:35 smithi179 ceph-mon[43975]: pgmap v1173: 37 pgs: 1 active+remapped+backfill_toofull, 36 active+clean; 15 MiB data, 4.7 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 598 op/s 2024-02-09T17:40:36.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:35 smithi162 ceph-mon[44304]: pgmap v1173: 37 pgs: 1 active+remapped+backfill_toofull, 36 active+clean; 15 MiB data, 4.7 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 598 op/s 2024-02-09T17:40:36.238 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:40:36.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:35 smithi099 ceph-mon[39420]: pgmap v1173: 37 pgs: 1 active+remapped+backfill_toofull, 36 active+clean; 15 MiB data, 4.7 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 598 op/s 2024-02-09T17:40:36.270 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:36.289 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:40:36.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:36.292 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:40:36.393 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:40:36.423 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:36.483 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'unique_pool_2' removed 2024-02-09T17:40:36.541 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:40:36.543 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:40:36.548 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:40:36.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:36.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:40:36.654 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:40:36.685 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:36.705 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:40:36.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:36.709 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:40:36.810 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:40:36.811 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:40:36.840 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:36.860 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:40:36.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:36.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:40:36.965 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:40:36.994 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:36 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3666203873' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-09T17:40:36.994 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:36 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3666203873' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-02-09T17:40:36.994 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:36 smithi179 ceph-mon[43975]: osdmap e459: 12 total, 12 up, 11 in 2024-02-09T17:40:36.996 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:37.017 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:40:37.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:37.021 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:40:37.122 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:40:37.151 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:36 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3666203873' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-09T17:40:37.151 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:36 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3666203873' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-02-09T17:40:37.151 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:36 smithi099 ceph-mon[39420]: osdmap e459: 12 total, 12 up, 11 in 2024-02-09T17:40:37.153 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:37.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:36 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3666203873' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-09T17:40:37.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:36 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3666203873' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-02-09T17:40:37.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:36 smithi162 ceph-mon[44304]: osdmap e459: 12 total, 12 up, 11 in 2024-02-09T17:40:37.365 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-02-09T17:40:37.365 INFO:tasks.radosbench:Beginning radosbench... 2024-02-09T17:40:37.366 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-02-09T17:40:37.366 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd pool create unique_pool_3 16 2024-02-09T17:40:37.955 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:37 smithi099 ceph-mon[39420]: pgmap v1175: 21 pgs: 21 active+clean; 603 KiB data, 4.6 GiB used, 979 GiB / 983 GiB avail 2024-02-09T17:40:37.955 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:37 smithi099 ceph-mon[39420]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-09T17:40:38.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:37 smithi179 ceph-mon[43975]: pgmap v1175: 21 pgs: 21 active+clean; 603 KiB data, 4.6 GiB used, 979 GiB / 983 GiB avail 2024-02-09T17:40:38.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:37 smithi179 ceph-mon[43975]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-09T17:40:38.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:37 smithi162 ceph-mon[44304]: pgmap v1175: 21 pgs: 21 active+clean; 603 KiB data, 4.6 GiB used, 979 GiB / 983 GiB avail 2024-02-09T17:40:38.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:37 smithi162 ceph-mon[44304]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-09T17:40:39.962 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:40:39.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:39.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:40:40.066 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:40:40.098 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:40.118 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:40:40.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:40.121 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:40:40.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:39 smithi179 ceph-mon[43975]: pgmap v1176: 21 pgs: 21 active+clean; 603 KiB data, 4.5 GiB used, 979 GiB / 983 GiB avail 2024-02-09T17:40:40.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:39 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3601865502' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:40:40.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:39 smithi162 ceph-mon[44304]: pgmap v1176: 21 pgs: 21 active+clean; 603 KiB data, 4.5 GiB used, 979 GiB / 983 GiB avail 2024-02-09T17:40:40.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:39 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3601865502' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:40:40.223 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:40:40.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:39 smithi099 ceph-mon[39420]: pgmap v1176: 21 pgs: 21 active+clean; 603 KiB data, 4.5 GiB used, 979 GiB / 983 GiB avail 2024-02-09T17:40:40.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:39 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3601865502' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:40:40.252 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:41.073 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is unset 2024-02-09T17:40:41.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:40 smithi179 ceph-mon[43975]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:40:41.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:40 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3601865502' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:40:41.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:40 smithi179 ceph-mon[43975]: osdmap e460: 12 total, 12 up, 11 in 2024-02-09T17:40:41.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:40 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:40:41.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:40 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:40:41.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:40 smithi162 ceph-mon[44304]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:40:41.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:40 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3601865502' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:40:41.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:40 smithi162 ceph-mon[44304]: osdmap e460: 12 total, 12 up, 11 in 2024-02-09T17:40:41.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:40 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:40:41.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:40 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:40:41.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:40 smithi099 ceph-mon[39420]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:40:41.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:40 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3601865502' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:40:41.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:40 smithi099 ceph-mon[39420]: osdmap e460: 12 total, 12 up, 11 in 2024-02-09T17:40:41.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:40 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:40:41.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:40 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:40:41.902 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:40:41.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:41.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:40:42.010 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:40:42.035 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:41 smithi099 ceph-mon[39420]: pgmap v1178: 21 pgs: 21 active+clean; 603 KiB data, 4.5 GiB used, 979 GiB / 983 GiB avail 2024-02-09T17:40:42.035 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:41 smithi099 ceph-mon[39420]: osdmap e461: 12 total, 12 up, 11 in 2024-02-09T17:40:42.035 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:41 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:40:42.035 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:41 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:40:42.035 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:41 smithi099 ceph-mon[39420]: osdmap e462: 12 total, 12 up, 11 in 2024-02-09T17:40:42.037 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:42.110 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:40:42.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:42.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:40:42.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:41 smithi179 ceph-mon[43975]: pgmap v1178: 21 pgs: 21 active+clean; 603 KiB data, 4.5 GiB used, 979 GiB / 983 GiB avail 2024-02-09T17:40:42.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:41 smithi179 ceph-mon[43975]: osdmap e461: 12 total, 12 up, 11 in 2024-02-09T17:40:42.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:41 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:40:42.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:41 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:40:42.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:41 smithi179 ceph-mon[43975]: osdmap e462: 12 total, 12 up, 11 in 2024-02-09T17:40:42.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:41 smithi162 ceph-mon[44304]: pgmap v1178: 21 pgs: 21 active+clean; 603 KiB data, 4.5 GiB used, 979 GiB / 983 GiB avail 2024-02-09T17:40:42.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:41 smithi162 ceph-mon[44304]: osdmap e461: 12 total, 12 up, 11 in 2024-02-09T17:40:42.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:41 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:40:42.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:41 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:40:42.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:41 smithi162 ceph-mon[44304]: osdmap e462: 12 total, 12 up, 11 in 2024-02-09T17:40:42.214 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:40:42.246 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:42.267 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:40:42.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:42.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:40:42.371 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:40:42.403 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:42.424 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:40:42.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:42.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:40:42.529 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:40:42.558 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:42.755 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:40:42.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:42.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:40:42.859 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:40:42.886 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:42.970 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.9 to 1 (10000) 2024-02-09T17:40:42.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:42 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:40:42.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:42 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:40:42.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:42 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2546726519' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-02-09T17:40:43.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:42 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:40:43.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:42 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:40:43.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:42 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2546726519' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-02-09T17:40:43.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:42 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:40:43.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:42 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:40:43.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:42 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2546726519' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-02-09T17:40:43.412 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:40:43.453 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:40:43.453 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:40:43.776 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-09T17:40:44.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:43 smithi179 ceph-mon[43975]: pgmap v1181: 21 pgs: 21 active+clean; 603 KiB data, 4.5 GiB used, 979 GiB / 983 GiB avail 2024-02-09T17:40:44.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:43 smithi179 ceph-mon[43975]: 1.b scrub starts 2024-02-09T17:40:44.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:43 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2546726519' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-02-09T17:40:44.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:43 smithi179 ceph-mon[43975]: osdmap e463: 12 total, 12 up, 11 in 2024-02-09T17:40:44.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:43 smithi179 ceph-mon[43975]: 1.b scrub ok 2024-02-09T17:40:44.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:43 smithi162 ceph-mon[44304]: pgmap v1181: 21 pgs: 21 active+clean; 603 KiB data, 4.5 GiB used, 979 GiB / 983 GiB avail 2024-02-09T17:40:44.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:43 smithi162 ceph-mon[44304]: 1.b scrub starts 2024-02-09T17:40:44.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:43 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2546726519' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-02-09T17:40:44.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:43 smithi162 ceph-mon[44304]: osdmap e463: 12 total, 12 up, 11 in 2024-02-09T17:40:44.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:43 smithi162 ceph-mon[44304]: 1.b scrub ok 2024-02-09T17:40:44.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:43 smithi099 ceph-mon[39420]: pgmap v1181: 21 pgs: 21 active+clean; 603 KiB data, 4.5 GiB used, 979 GiB / 983 GiB avail 2024-02-09T17:40:44.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:43 smithi099 ceph-mon[39420]: 1.b scrub starts 2024-02-09T17:40:44.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:43 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2546726519' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-02-09T17:40:44.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:43 smithi099 ceph-mon[39420]: osdmap e463: 12 total, 12 up, 11 in 2024-02-09T17:40:44.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:43 smithi099 ceph-mon[39420]: 1.b scrub ok 2024-02-09T17:40:44.415 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 10 1 2024-02-09T17:40:44.440 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:40:44.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:44.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:40:44.544 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:40:44.571 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:45.226 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:40:45.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:45.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:40:45.329 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:40:45.346 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:40:45.361 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:45.366 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:40:45.383 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:40:45.384 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:40:45.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:45.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:40:45.404 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:40:45.423 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:40:45.443 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:40:45.466 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:40:45.486 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:40:45.488 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:40:45.508 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:40:45.516 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:45.530 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:40:45.550 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:40:45.572 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:40:45.574 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:40:45.574 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:40:45.574 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:40:45.574 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:40:45.574 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:40:45.574 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:40:45.574 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:40:45.574 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:40:45.574 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:40:45.575 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:40:45.575 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:40:45.575 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:40:46.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:45 smithi162 ceph-mon[44304]: 1.12 scrub starts 2024-02-09T17:40:46.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:45 smithi162 ceph-mon[44304]: 1.12 scrub ok 2024-02-09T17:40:46.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:45 smithi162 ceph-mon[44304]: 1.4 scrub starts 2024-02-09T17:40:46.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:45 smithi162 ceph-mon[44304]: 1.4 scrub ok 2024-02-09T17:40:46.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:45 smithi162 ceph-mon[44304]: pgmap v1183: 21 pgs: 21 active+clean; 603 KiB data, 4.4 GiB used, 979 GiB / 983 GiB avail 2024-02-09T17:40:46.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:45 smithi162 ceph-mon[44304]: 1.d scrub starts 2024-02-09T17:40:46.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:45 smithi162 ceph-mon[44304]: 1.d scrub ok 2024-02-09T17:40:46.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:45 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1670276139' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:40:46.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:45 smithi099 ceph-mon[39420]: 1.12 scrub starts 2024-02-09T17:40:46.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:45 smithi099 ceph-mon[39420]: 1.12 scrub ok 2024-02-09T17:40:46.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:45 smithi099 ceph-mon[39420]: 1.4 scrub starts 2024-02-09T17:40:46.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:45 smithi099 ceph-mon[39420]: 1.4 scrub ok 2024-02-09T17:40:46.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:45 smithi099 ceph-mon[39420]: pgmap v1183: 21 pgs: 21 active+clean; 603 KiB data, 4.4 GiB used, 979 GiB / 983 GiB avail 2024-02-09T17:40:46.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:45 smithi099 ceph-mon[39420]: 1.d scrub starts 2024-02-09T17:40:46.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:45 smithi099 ceph-mon[39420]: 1.d scrub ok 2024-02-09T17:40:46.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:45 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1670276139' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:40:46.364 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:40:46.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:46.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:40:46.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:45 smithi179 ceph-mon[43975]: 1.12 scrub starts 2024-02-09T17:40:46.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:45 smithi179 ceph-mon[43975]: 1.12 scrub ok 2024-02-09T17:40:46.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:45 smithi179 ceph-mon[43975]: 1.4 scrub starts 2024-02-09T17:40:46.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:45 smithi179 ceph-mon[43975]: 1.4 scrub ok 2024-02-09T17:40:46.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:45 smithi179 ceph-mon[43975]: pgmap v1183: 21 pgs: 21 active+clean; 603 KiB data, 4.4 GiB used, 979 GiB / 983 GiB avail 2024-02-09T17:40:46.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:45 smithi179 ceph-mon[43975]: 1.d scrub starts 2024-02-09T17:40:46.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:45 smithi179 ceph-mon[43975]: 1.d scrub ok 2024-02-09T17:40:46.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:45 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1670276139' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:40:46.469 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:40:46.502 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:46.525 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:40:46.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:46.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:40:46.629 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:40:46.659 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:46.680 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:40:46.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:46.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:40:46.784 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:40:46.810 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:40:46.812 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:46.833 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:40:46.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:46.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:40:46.937 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:40:46.969 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:46.989 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:40:46.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:46.993 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:40:47.094 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:40:47.122 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:47.142 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:40:47.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:47.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:40:47.247 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:40:47.277 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:47.771 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'unique_pool_3' created 2024-02-09T17:40:47.830 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:40:47.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:47.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:40:47.857 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-09T17:40:47.934 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:40:47.967 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:47.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:47 smithi099 ceph-mon[39420]: 1.8 scrub starts 2024-02-09T17:40:47.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:47 smithi099 ceph-mon[39420]: 1.8 scrub ok 2024-02-09T17:40:47.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:47 smithi099 ceph-mon[39420]: osdmap e464: 12 total, 12 up, 11 in 2024-02-09T17:40:47.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:47 smithi099 ceph-mon[39420]: 1.1 scrub starts 2024-02-09T17:40:47.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:47 smithi099 ceph-mon[39420]: 1.1 scrub ok 2024-02-09T17:40:47.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:47 smithi099 ceph-mon[39420]: pgmap v1185: 21 pgs: 21 active+clean; 603 KiB data, 4.4 GiB used, 979 GiB / 983 GiB avail 2024-02-09T17:40:47.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:47 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:40:48.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:47 smithi179 ceph-mon[43975]: 1.8 scrub starts 2024-02-09T17:40:48.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:47 smithi179 ceph-mon[43975]: 1.8 scrub ok 2024-02-09T17:40:48.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:47 smithi179 ceph-mon[43975]: osdmap e464: 12 total, 12 up, 11 in 2024-02-09T17:40:48.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:47 smithi179 ceph-mon[43975]: 1.1 scrub starts 2024-02-09T17:40:48.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:47 smithi179 ceph-mon[43975]: 1.1 scrub ok 2024-02-09T17:40:48.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:47 smithi179 ceph-mon[43975]: pgmap v1185: 21 pgs: 21 active+clean; 603 KiB data, 4.4 GiB used, 979 GiB / 983 GiB avail 2024-02-09T17:40:48.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:47 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:40:48.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:47 smithi162 ceph-mon[44304]: 1.8 scrub starts 2024-02-09T17:40:48.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:47 smithi162 ceph-mon[44304]: 1.8 scrub ok 2024-02-09T17:40:48.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:47 smithi162 ceph-mon[44304]: osdmap e464: 12 total, 12 up, 11 in 2024-02-09T17:40:48.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:47 smithi162 ceph-mon[44304]: 1.1 scrub starts 2024-02-09T17:40:48.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:47 smithi162 ceph-mon[44304]: 1.1 scrub ok 2024-02-09T17:40:48.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:47 smithi162 ceph-mon[44304]: pgmap v1185: 21 pgs: 21 active+clean; 603 KiB data, 4.4 GiB used, 979 GiB / 983 GiB avail 2024-02-09T17:40:48.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:47 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:40:48.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:48 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3115669760' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-02-09T17:40:48.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:48 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3115669760' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-02-09T17:40:48.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:48 smithi099 ceph-mon[39420]: osdmap e465: 12 total, 12 up, 11 in 2024-02-09T17:40:49.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:48 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3115669760' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-02-09T17:40:49.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:48 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3115669760' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-02-09T17:40:49.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:48 smithi179 ceph-mon[43975]: osdmap e465: 12 total, 12 up, 11 in 2024-02-09T17:40:49.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:48 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3115669760' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-02-09T17:40:49.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:48 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3115669760' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-02-09T17:40:49.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:48 smithi162 ceph-mon[44304]: osdmap e465: 12 total, 12 up, 11 in 2024-02-09T17:40:50.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:49 smithi179 ceph-mon[43975]: osdmap e466: 12 total, 12 up, 11 in 2024-02-09T17:40:50.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:49 smithi179 ceph-mon[43975]: pgmap v1188: 37 pgs: 2 creating+peering, 14 unknown, 21 active+clean; 577 KiB data, 4.4 GiB used, 979 GiB / 983 GiB avail 2024-02-09T17:40:50.212 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:49 smithi099 ceph-mon[39420]: osdmap e466: 12 total, 12 up, 11 in 2024-02-09T17:40:50.212 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:49 smithi099 ceph-mon[39420]: pgmap v1188: 37 pgs: 2 creating+peering, 14 unknown, 21 active+clean; 577 KiB data, 4.4 GiB used, 979 GiB / 983 GiB avail 2024-02-09T17:40:50.212 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:49 smithi162 ceph-mon[44304]: osdmap e466: 12 total, 12 up, 11 in 2024-02-09T17:40:50.212 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:49 smithi162 ceph-mon[44304]: pgmap v1188: 37 pgs: 2 creating+peering, 14 unknown, 21 active+clean; 577 KiB data, 4.4 GiB used, 979 GiB / 983 GiB avail 2024-02-09T17:40:50.643 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:40:51.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:50 smithi179 ceph-mon[43975]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-09T17:40:51.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:50 smithi179 ceph-mon[43975]: osdmap e467: 12 total, 12 up, 11 in 2024-02-09T17:40:51.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:50 smithi162 ceph-mon[44304]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-09T17:40:51.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:50 smithi162 ceph-mon[44304]: osdmap e467: 12 total, 12 up, 11 in 2024-02-09T17:40:51.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:50 smithi099 ceph-mon[39420]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-09T17:40:51.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:50 smithi099 ceph-mon[39420]: osdmap e467: 12 total, 12 up, 11 in 2024-02-09T17:40:51.318 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:40:51.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:51.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:40:51.423 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:40:51.452 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:51.859 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:40:51.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:51 smithi099 ceph-mon[39420]: pgmap v1190: 37 pgs: 2 creating+peering, 6 unknown, 29 active+clean; 577 KiB data, 4.3 GiB used, 979 GiB / 983 GiB avail 2024-02-09T17:40:51.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:51 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:40:51.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:51 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:40:51.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:51 smithi099 ceph-mon[39420]: osdmap e468: 12 total, 12 up, 11 in 2024-02-09T17:40:52.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:51 smithi179 ceph-mon[43975]: pgmap v1190: 37 pgs: 2 creating+peering, 6 unknown, 29 active+clean; 577 KiB data, 4.3 GiB used, 979 GiB / 983 GiB avail 2024-02-09T17:40:52.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:51 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:40:52.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:51 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:40:52.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:51 smithi179 ceph-mon[43975]: osdmap e468: 12 total, 12 up, 11 in 2024-02-09T17:40:52.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:51 smithi162 ceph-mon[44304]: pgmap v1190: 37 pgs: 2 creating+peering, 6 unknown, 29 active+clean; 577 KiB data, 4.3 GiB used, 979 GiB / 983 GiB avail 2024-02-09T17:40:52.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:51 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:40:52.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:51 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:40:52.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:51 smithi162 ceph-mon[44304]: osdmap e468: 12 total, 12 up, 11 in 2024-02-09T17:40:52.229 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:40:52.301 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:40:52.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:52.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:40:52.410 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:40:52.443 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:52.465 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:40:52.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:52.468 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:40:52.569 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:40:52.598 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:52.617 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:40:52.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:52.621 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:40:52.722 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:40:52.749 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-02-09T17:40:52.754 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:52.777 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:40:52.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:52.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:40:52.881 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:40:52.911 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:52.930 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:40:52.933 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:52.934 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:40:52.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:52 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:40:52.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:52 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:40:53.035 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:40:53.063 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:52 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:40:53.063 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:52 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:40:53.065 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:53.087 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:40:53.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:53.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:40:53.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:52 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:40:53.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:52 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:40:53.193 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:40:53.223 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:53.792 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is unset 2024-02-09T17:40:54.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:53 smithi179 ceph-mon[43975]: pgmap v1192: 37 pgs: 2 creating+peering, 35 active+clean; 577 KiB data, 4.3 GiB used, 979 GiB / 983 GiB avail 2024-02-09T17:40:54.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:53 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4021231570' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:40:54.176 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:53 smithi099 ceph-mon[39420]: pgmap v1192: 37 pgs: 2 creating+peering, 35 active+clean; 577 KiB data, 4.3 GiB used, 979 GiB / 983 GiB avail 2024-02-09T17:40:54.176 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:53 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4021231570' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:40:54.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:53 smithi162 ceph-mon[44304]: pgmap v1192: 37 pgs: 2 creating+peering, 35 active+clean; 577 KiB data, 4.3 GiB used, 979 GiB / 983 GiB avail 2024-02-09T17:40:54.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:53 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4021231570' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:40:54.973 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:40:54.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:54.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:40:55.078 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:40:55.107 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:54 smithi179 ceph-mon[43975]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:40:55.108 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:54 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4021231570' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:40:55.108 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:54 smithi179 ceph-mon[43975]: osdmap e469: 12 total, 12 up, 11 in 2024-02-09T17:40:55.110 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:55.131 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:40:55.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:55.135 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:40:55.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:54 smithi162 ceph-mon[44304]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:40:55.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:54 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4021231570' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:40:55.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:54 smithi162 ceph-mon[44304]: osdmap e469: 12 total, 12 up, 11 in 2024-02-09T17:40:55.185 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:54 smithi099 ceph-mon[39420]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:40:55.185 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:54 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4021231570' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:40:55.185 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:54 smithi099 ceph-mon[39420]: osdmap e469: 12 total, 12 up, 11 in 2024-02-09T17:40:55.235 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:40:55.264 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:55.362 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:40:55.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:55.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:40:55.467 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:40:55.499 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:55.519 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:40:55.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:55.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:40:55.624 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:40:55.652 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:55.672 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:40:55.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:55.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:40:55.776 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:40:55.791 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:55 smithi099 ceph-mon[39420]: pgmap v1194: 37 pgs: 37 active+clean; 577 KiB data, 4.2 GiB used, 979 GiB / 983 GiB avail 2024-02-09T17:40:55.791 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:55 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4173360957' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-02-09T17:40:55.800 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.10 to 1 (10000) 2024-02-09T17:40:55.805 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:56.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:55 smithi179 ceph-mon[43975]: pgmap v1194: 37 pgs: 37 active+clean; 577 KiB data, 4.2 GiB used, 979 GiB / 983 GiB avail 2024-02-09T17:40:56.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:55 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4173360957' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-02-09T17:40:56.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:55 smithi162 ceph-mon[44304]: pgmap v1194: 37 pgs: 37 active+clean; 577 KiB data, 4.2 GiB used, 979 GiB / 983 GiB avail 2024-02-09T17:40:56.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:55 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4173360957' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-02-09T17:40:56.918 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:56 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:40:56.918 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:56 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:40:56.918 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:56 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4173360957' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-02-09T17:40:56.918 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:56 smithi099 ceph-mon[39420]: osdmap e470: 12 total, 12 up, 11 in 2024-02-09T17:40:56.943 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 11 1 2024-02-09T17:40:57.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:56 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:40:57.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:56 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:40:57.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:56 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4173360957' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-02-09T17:40:57.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:56 smithi179 ceph-mon[43975]: osdmap e470: 12 total, 12 up, 11 in 2024-02-09T17:40:57.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:56 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:40:57.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:56 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:40:57.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:56 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4173360957' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-02-09T17:40:57.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:56 smithi162 ceph-mon[44304]: osdmap e470: 12 total, 12 up, 11 in 2024-02-09T17:40:57.522 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:40:57.885 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:40:57.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:57.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:40:57.893 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:40:57.990 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:40:58.019 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:58.039 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:40:58.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:58.042 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:40:58.042 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set noscrub 2024-02-09T17:40:58.068 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:57 smithi099 ceph-mon[39420]: 1.5 scrub starts 2024-02-09T17:40:58.069 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:57 smithi099 ceph-mon[39420]: 1.5 scrub ok 2024-02-09T17:40:58.069 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:57 smithi099 ceph-mon[39420]: pgmap v1196: 37 pgs: 37 active+clean; 577 KiB data, 4.2 GiB used, 979 GiB / 983 GiB avail 2024-02-09T17:40:58.069 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:57 smithi099 ceph-mon[39420]: 1.3 scrub starts 2024-02-09T17:40:58.069 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:57 smithi099 ceph-mon[39420]: 1.3 scrub ok 2024-02-09T17:40:58.143 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:40:58.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:57 smithi179 ceph-mon[43975]: 1.5 scrub starts 2024-02-09T17:40:58.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:57 smithi179 ceph-mon[43975]: 1.5 scrub ok 2024-02-09T17:40:58.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:57 smithi179 ceph-mon[43975]: pgmap v1196: 37 pgs: 37 active+clean; 577 KiB data, 4.2 GiB used, 979 GiB / 983 GiB avail 2024-02-09T17:40:58.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:57 smithi179 ceph-mon[43975]: 1.3 scrub starts 2024-02-09T17:40:58.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:57 smithi179 ceph-mon[43975]: 1.3 scrub ok 2024-02-09T17:40:58.171 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:58.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:57 smithi162 ceph-mon[44304]: 1.5 scrub starts 2024-02-09T17:40:58.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:57 smithi162 ceph-mon[44304]: 1.5 scrub ok 2024-02-09T17:40:58.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:57 smithi162 ceph-mon[44304]: pgmap v1196: 37 pgs: 37 active+clean; 577 KiB data, 4.2 GiB used, 979 GiB / 983 GiB avail 2024-02-09T17:40:58.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:57 smithi162 ceph-mon[44304]: 1.3 scrub starts 2024-02-09T17:40:58.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:57 smithi162 ceph-mon[44304]: 1.3 scrub ok 2024-02-09T17:40:58.190 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:40:58.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:58.193 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:40:58.294 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:40:58.321 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:58.341 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:40:58.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:58.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:40:58.446 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:40:58.474 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:58.493 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:40:58.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:58.495 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:40:58.597 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:40:58.624 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:58.845 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:58 smithi099 ceph-mon[39420]: 1.9 scrub starts 2024-02-09T17:40:58.845 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:58 smithi099 ceph-mon[39420]: 1.9 scrub ok 2024-02-09T17:40:58.845 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:58 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:40:58.845 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:58 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:40:59.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:58 smithi179 ceph-mon[43975]: 1.9 scrub starts 2024-02-09T17:40:59.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:58 smithi179 ceph-mon[43975]: 1.9 scrub ok 2024-02-09T17:40:59.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:58 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:40:59.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:58 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:40:59.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:58 smithi162 ceph-mon[44304]: 1.9 scrub starts 2024-02-09T17:40:59.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:58 smithi162 ceph-mon[44304]: 1.9 scrub ok 2024-02-09T17:40:59.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:58 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:40:59.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:58 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:40:59.774 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:40:59.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:59.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:40:59.880 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:40:59.913 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:40:59.934 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:40:59.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:40:59.937 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:41:00.038 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:41:00.068 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:59 smithi162 ceph-mon[44304]: pgmap v1197: 37 pgs: 37 active+clean; 577 KiB data, 4.2 GiB used, 979 GiB / 983 GiB avail 2024-02-09T17:41:00.069 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:59 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:41:00.069 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:59 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:41:00.069 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:40:59 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:41:00.071 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:00.092 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:41:00.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:00.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:41:00.143 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:59 smithi099 ceph-mon[39420]: pgmap v1197: 37 pgs: 37 active+clean; 577 KiB data, 4.2 GiB used, 979 GiB / 983 GiB avail 2024-02-09T17:41:00.143 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:59 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:41:00.143 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:59 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:41:00.143 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:40:59 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:41:00.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:59 smithi179 ceph-mon[43975]: pgmap v1197: 37 pgs: 37 active+clean; 577 KiB data, 4.2 GiB used, 979 GiB / 983 GiB avail 2024-02-09T17:41:00.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:59 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:41:00.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:59 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:41:00.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:40:59 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:41:00.196 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:41:00.224 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:00.737 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:41:00.766 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:41:00.767 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 20, 2024-02-09T17:41:00.767 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 600, 2024-02-09T17:41:00.767 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [ 2024-02-09T17:41:00.767 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.767 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.24817.0:141304 3.6 3:63334779:::benchmark_data_smithi179_90440_object9345:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-02-09T17:41:00.767 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:31:59.273717+0000", 2024-02-09T17:41:00.767 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 541.47638912299999, 2024-02-09T17:41:00.767 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.303239354, 2024-02-09T17:41:00.767 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:41:00.767 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:41:00.767 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:41:00.768 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.24817", 2024-02-09T17:41:00.768 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:41:00.768 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 141304 2024-02-09T17:41:00.768 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.768 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:41:00.768 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.768 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:41:00.768 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.273717+0000", 2024-02-09T17:41:00.768 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:41:00.768 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.768 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.768 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:41:00.768 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.273717+0000", 2024-02-09T17:41:00.768 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:41:00.769 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.769 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.769 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:41:00.769 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.273718+0000", 2024-02-09T17:41:00.769 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.8100000000000002e-07 2024-02-09T17:41:00.769 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.769 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.769 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:41:00.769 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.273721+0000", 2024-02-09T17:41:00.769 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.1489999999999998e-06 2024-02-09T17:41:00.769 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.769 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.769 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:41:00.769 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.273729+0000", 2024-02-09T17:41:00.769 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.481e-06 2024-02-09T17:41:00.770 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.770 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.770 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:41:00.770 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.273734+0000", 2024-02-09T17:41:00.770 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.0289999999999996e-06 2024-02-09T17:41:00.770 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.770 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.770 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.770 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.273821+0000", 2024-02-09T17:41:00.770 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.7763999999999995e-05 2024-02-09T17:41:00.770 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.770 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.770 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op must wait for map", 2024-02-09T17:41:00.770 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.273826+0000", 2024-02-09T17:41:00.771 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.3259999999999997e-06 2024-02-09T17:41:00.771 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.771 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.771 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.771 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.281374+0000", 2024-02-09T17:41:00.771 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0075486549999999996 2024-02-09T17:41:00.771 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.771 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.771 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:41:00.771 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.281384+0000", 2024-02-09T17:41:00.771 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.3349999999999996e-06 2024-02-09T17:41:00.771 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.771 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.771 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.772 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:00.272447+0000", 2024-02-09T17:41:00.772 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.99106366400000001 2024-02-09T17:41:00.772 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.772 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.772 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:41:00.772 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:00.272457+0000", 2024-02-09T17:41:00.772 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.1759999999999999e-06 2024-02-09T17:41:00.772 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.772 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.772 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.772 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.543600+0000", 2024-02-09T17:41:00.772 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.27114342 2024-02-09T17:41:00.772 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.772 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.773 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:41:00.773 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.546128+0000", 2024-02-09T17:41:00.773 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0025278420000000002 2024-02-09T17:41:00.773 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.773 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.773 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 4,7", 2024-02-09T17:41:00.773 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.546287+0000", 2024-02-09T17:41:00.773 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0001593 2024-02-09T17:41:00.773 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.773 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.773 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:41:00.773 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.556995+0000", 2024-02-09T17:41:00.773 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.010707816 2024-02-09T17:41:00.773 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.774 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.774 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.774 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.557048+0000", 2024-02-09T17:41:00.774 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2862999999999999e-05 2024-02-09T17:41:00.774 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.774 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.774 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.774 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.574525+0000", 2024-02-09T17:41:00.774 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.017476744999999998 2024-02-09T17:41:00.774 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.774 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.774 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:41:00.774 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.574594+0000", 2024-02-09T17:41:00.774 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.9626e-05 2024-02-09T17:41:00.775 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.775 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.775 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:41:00.775 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.576957+0000", 2024-02-09T17:41:00.775 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0023623820000000001 2024-02-09T17:41:00.775 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:41:00.775 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:41:00.775 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:41:00.775 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.775 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.775 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.24817.0:141305 3.6 3:6b0ab8bf:::benchmark_data_smithi179_90440_object9346:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-02-09T17:41:00.775 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:31:59.273750+0000", 2024-02-09T17:41:00.775 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 541.47635594099995, 2024-02-09T17:41:00.775 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.3052010909999998, 2024-02-09T17:41:00.775 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:41:00.776 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:41:00.776 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:41:00.776 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.24817", 2024-02-09T17:41:00.776 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:41:00.776 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 141305 2024-02-09T17:41:00.776 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.776 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:41:00.776 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.776 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:41:00.776 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.273750+0000", 2024-02-09T17:41:00.776 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:41:00.776 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.776 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.776 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:41:00.777 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.273750+0000", 2024-02-09T17:41:00.777 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:41:00.777 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.777 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.777 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:41:00.777 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.273751+0000", 2024-02-09T17:41:00.777 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9299999999999999e-07 2024-02-09T17:41:00.777 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.777 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.777 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:41:00.777 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.273753+0000", 2024-02-09T17:41:00.777 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.367e-06 2024-02-09T17:41:00.777 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.777 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.777 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:41:00.778 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.273759+0000", 2024-02-09T17:41:00.778 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.9699999999999996e-06 2024-02-09T17:41:00.778 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.778 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.778 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:41:00.778 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.273765+0000", 2024-02-09T17:41:00.778 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7470000000000002e-06 2024-02-09T17:41:00.778 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.778 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.778 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.778 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.273849+0000", 2024-02-09T17:41:00.778 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.3770999999999995e-05 2024-02-09T17:41:00.778 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.778 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.778 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting_for_map not empty", 2024-02-09T17:41:00.779 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.273854+0000", 2024-02-09T17:41:00.779 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.4729999999999999e-06 2024-02-09T17:41:00.779 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.779 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.779 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.779 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.281503+0000", 2024-02-09T17:41:00.779 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0076486519999999997 2024-02-09T17:41:00.779 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.779 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.779 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:41:00.779 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.281510+0000", 2024-02-09T17:41:00.779 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.2799999999999998e-06 2024-02-09T17:41:00.779 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.779 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.779 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.780 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:00.272552+0000", 2024-02-09T17:41:00.780 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.99104202100000005 2024-02-09T17:41:00.780 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.780 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.780 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:41:00.780 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:00.272558+0000", 2024-02-09T17:41:00.780 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.8719999999999999e-06 2024-02-09T17:41:00.780 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.780 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.780 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.780 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.551349+0000", 2024-02-09T17:41:00.780 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2787907810000001 2024-02-09T17:41:00.780 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.781 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.781 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:41:00.781 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.551501+0000", 2024-02-09T17:41:00.781 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00015254400000000001 2024-02-09T17:41:00.781 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.781 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.781 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 4,7", 2024-02-09T17:41:00.781 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.551599+0000", 2024-02-09T17:41:00.781 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.8162e-05 2024-02-09T17:41:00.781 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.781 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.781 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:41:00.781 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.557125+0000", 2024-02-09T17:41:00.781 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0055251789999999999 2024-02-09T17:41:00.782 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.782 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.782 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.782 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.562614+0000", 2024-02-09T17:41:00.782 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.005489837 2024-02-09T17:41:00.782 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.782 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.782 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.782 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.577014+0000", 2024-02-09T17:41:00.782 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.014399095000000001 2024-02-09T17:41:00.782 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.782 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.782 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:41:00.782 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.577068+0000", 2024-02-09T17:41:00.782 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.4475999999999998e-05 2024-02-09T17:41:00.783 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.783 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.783 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:41:00.783 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.578951+0000", 2024-02-09T17:41:00.783 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0018834710000000001 2024-02-09T17:41:00.783 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:41:00.783 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:41:00.783 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:41:00.783 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.783 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.783 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.24817.0:141328 3.6 3:636c2a21:::benchmark_data_smithi179_90440_object9369:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-02-09T17:41:00.783 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:31:59.299967+0000", 2024-02-09T17:41:00.783 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 541.45013971499998, 2024-02-09T17:41:00.783 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2807691910000001, 2024-02-09T17:41:00.783 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:41:00.784 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:41:00.784 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:41:00.784 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.24817", 2024-02-09T17:41:00.784 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:41:00.784 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 141328 2024-02-09T17:41:00.784 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.784 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:41:00.784 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.784 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:41:00.784 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.299967+0000", 2024-02-09T17:41:00.784 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:41:00.784 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.784 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.785 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:41:00.785 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.299967+0000", 2024-02-09T17:41:00.785 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:41:00.785 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.785 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.785 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:41:00.785 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.299968+0000", 2024-02-09T17:41:00.785 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3859999999999999e-06 2024-02-09T17:41:00.785 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.785 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.785 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:41:00.785 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.299972+0000", 2024-02-09T17:41:00.785 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.8999999999999999e-06 2024-02-09T17:41:00.785 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.786 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.786 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:41:00.786 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.299981+0000", 2024-02-09T17:41:00.786 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.7900000000000005e-06 2024-02-09T17:41:00.786 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.786 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.786 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:41:00.786 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.299987+0000", 2024-02-09T17:41:00.786 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.4060000000000002e-06 2024-02-09T17:41:00.786 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.786 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.786 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.786 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.300037+0000", 2024-02-09T17:41:00.786 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.9691999999999998e-05 2024-02-09T17:41:00.786 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.787 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.787 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:41:00.787 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.300048+0000", 2024-02-09T17:41:00.787 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0815999999999999e-05 2024-02-09T17:41:00.787 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.787 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.787 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.787 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:00.272580+0000", 2024-02-09T17:41:00.787 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.97253208999999996 2024-02-09T17:41:00.787 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.787 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.787 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:41:00.787 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:00.272588+0000", 2024-02-09T17:41:00.787 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.0979999999999994e-06 2024-02-09T17:41:00.787 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.788 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.788 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.788 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.554498+0000", 2024-02-09T17:41:00.788 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.281910208 2024-02-09T17:41:00.788 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.788 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.788 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:41:00.788 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.554600+0000", 2024-02-09T17:41:00.788 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00010242999999999999 2024-02-09T17:41:00.788 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.788 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.788 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 4,7", 2024-02-09T17:41:00.788 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.554694+0000", 2024-02-09T17:41:00.788 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.3293000000000003e-05 2024-02-09T17:41:00.789 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.789 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.789 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:41:00.789 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.559220+0000", 2024-02-09T17:41:00.789 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0045263150000000004 2024-02-09T17:41:00.789 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.789 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.789 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.789 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.562669+0000", 2024-02-09T17:41:00.789 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.003448965 2024-02-09T17:41:00.789 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.789 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.789 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.789 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.579019+0000", 2024-02-09T17:41:00.789 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.016349902999999999 2024-02-09T17:41:00.790 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.790 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.790 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:41:00.790 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.579083+0000", 2024-02-09T17:41:00.790 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.3901999999999999e-05 2024-02-09T17:41:00.790 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.790 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.790 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:41:00.790 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.580736+0000", 2024-02-09T17:41:00.790 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.001652997 2024-02-09T17:41:00.790 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:41:00.790 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:41:00.790 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:41:00.790 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.790 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.791 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.24817.0:141331 3.6 3:6c1a36e8:::benchmark_data_smithi179_90440_object9372:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-02-09T17:41:00.791 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:31:59.304058+0000", 2024-02-09T17:41:00.791 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 541.446048228, 2024-02-09T17:41:00.791 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2790567400000001, 2024-02-09T17:41:00.791 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:41:00.791 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:41:00.791 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:41:00.791 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.24817", 2024-02-09T17:41:00.791 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:41:00.791 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 141331 2024-02-09T17:41:00.791 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.791 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:41:00.791 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.791 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:41:00.791 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.304058+0000", 2024-02-09T17:41:00.792 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:41:00.792 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.792 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.792 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:41:00.792 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.304058+0000", 2024-02-09T17:41:00.792 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:41:00.792 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.792 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.792 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:41:00.792 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.304059+0000", 2024-02-09T17:41:00.792 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2669999999999999e-06 2024-02-09T17:41:00.792 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.792 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.792 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:41:00.792 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.304064+0000", 2024-02-09T17:41:00.793 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.0830000000000003e-06 2024-02-09T17:41:00.793 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.793 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.793 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:41:00.793 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.304075+0000", 2024-02-09T17:41:00.793 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0659000000000001e-05 2024-02-09T17:41:00.793 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.793 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.793 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:41:00.793 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.304082+0000", 2024-02-09T17:41:00.793 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.3220000000000004e-06 2024-02-09T17:41:00.793 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.793 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.793 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.794 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.304134+0000", 2024-02-09T17:41:00.794 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1638999999999998e-05 2024-02-09T17:41:00.794 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.794 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.794 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:41:00.794 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:31:59.304141+0000", 2024-02-09T17:41:00.794 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.1570000000000003e-06 2024-02-09T17:41:00.794 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.794 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.794 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.794 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:00.272625+0000", 2024-02-09T17:41:00.794 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.96848358400000001 2024-02-09T17:41:00.794 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.794 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.794 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:41:00.795 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:00.272629+0000", 2024-02-09T17:41:00.795 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.4309999999999996e-06 2024-02-09T17:41:00.795 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.795 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.795 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.795 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.557153+0000", 2024-02-09T17:41:00.795 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2845239560000001 2024-02-09T17:41:00.795 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.795 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.795 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:41:00.795 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.557307+0000", 2024-02-09T17:41:00.795 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00015353 2024-02-09T17:41:00.795 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.795 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.795 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 4,7", 2024-02-09T17:41:00.796 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.557462+0000", 2024-02-09T17:41:00.796 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00015571000000000001 2024-02-09T17:41:00.796 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.796 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.796 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:41:00.796 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.559377+0000", 2024-02-09T17:41:00.796 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0019150180000000001 2024-02-09T17:41:00.796 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.796 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.796 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.796 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.562762+0000", 2024-02-09T17:41:00.796 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0033848089999999999 2024-02-09T17:41:00.796 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.796 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.797 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.797 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.580802+0000", 2024-02-09T17:41:00.797 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.018039415999999999 2024-02-09T17:41:00.797 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.797 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.797 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:41:00.797 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.580862+0000", 2024-02-09T17:41:00.797 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.0405999999999997e-05 2024-02-09T17:41:00.797 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.797 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.797 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:41:00.797 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:32:01.583115+0000", 2024-02-09T17:41:00.797 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0022527530000000001 2024-02-09T17:41:00.797 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:41:00.797 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:41:00.797 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:41:00.798 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.798 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.798 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:208395 4.f 4:fb1bb549:::benchmark_data_smithi179_103809_object70929:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-02-09T17:41:00.798 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:38:33.803932+0000", 2024-02-09T17:41:00.798 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 146.94617415600001, 2024-02-09T17:41:00.798 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2222837719999999, 2024-02-09T17:41:00.798 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:41:00.798 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "reached pg", 2024-02-09T17:41:00.798 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:41:00.798 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:41:00.798 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:41:00.798 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 208395 2024-02-09T17:41:00.798 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.798 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:41:00.799 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.799 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:41:00.799 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.803932+0000", 2024-02-09T17:41:00.799 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:41:00.799 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.799 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.799 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:41:00.799 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.803932+0000", 2024-02-09T17:41:00.799 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:41:00.799 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.799 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.799 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:41:00.799 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.803933+0000", 2024-02-09T17:41:00.799 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.2800000000000005e-07 2024-02-09T17:41:00.799 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.800 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.800 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:41:00.800 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.803937+0000", 2024-02-09T17:41:00.800 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.4589999999999998e-06 2024-02-09T17:41:00.800 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.800 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.800 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:41:00.800 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.803946+0000", 2024-02-09T17:41:00.800 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.465e-06 2024-02-09T17:41:00.800 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.800 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.800 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:41:00.800 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.803952+0000", 2024-02-09T17:41:00.800 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.9959999999999999e-06 2024-02-09T17:41:00.800 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.801 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.801 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.801 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.803987+0000", 2024-02-09T17:41:00.801 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.4887e-05 2024-02-09T17:41:00.801 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.801 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.801 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:41:00.801 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.804067+0000", 2024-02-09T17:41:00.801 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.9674999999999999e-05 2024-02-09T17:41:00.801 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.801 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.801 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,11", 2024-02-09T17:41:00.801 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.804135+0000", 2024-02-09T17:41:00.801 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.8201000000000007e-05 2024-02-09T17:41:00.801 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.802 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.802 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:41:00.802 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.809243+0000", 2024-02-09T17:41:00.802 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0051077320000000002 2024-02-09T17:41:00.802 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.802 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.802 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.802 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.812530+0000", 2024-02-09T17:41:00.802 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0032871210000000001 2024-02-09T17:41:00.802 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.802 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.802 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.802 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.831775+0000", 2024-02-09T17:41:00.802 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.019245038999999999 2024-02-09T17:41:00.802 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.802 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.803 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:41:00.803 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.831785+0000", 2024-02-09T17:41:00.803 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.8740000000000007e-06 2024-02-09T17:41:00.803 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.803 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.803 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.803 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.810674+0000", 2024-02-09T17:41:00.803 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.97888962099999999 2024-02-09T17:41:00.803 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.803 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.803 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:41:00.803 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.810686+0000", 2024-02-09T17:41:00.803 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1963999999999999e-05 2024-02-09T17:41:00.803 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.803 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.804 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.804 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.026131+0000", 2024-02-09T17:41:00.804 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2154448579999999 2024-02-09T17:41:00.804 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.804 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.804 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:41:00.804 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.026216+0000", 2024-02-09T17:41:00.804 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.4951999999999996e-05 2024-02-09T17:41:00.804 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:41:00.804 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:41:00.804 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:41:00.804 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.804 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.804 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:208404 4.a 4:527668e2:::benchmark_data_smithi179_103809_object70938:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e379)", 2024-02-09T17:41:00.804 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:38:33.818190+0000", 2024-02-09T17:41:00.805 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 146.93191607700001, 2024-02-09T17:41:00.805 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2784034790000001, 2024-02-09T17:41:00.805 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:41:00.805 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:41:00.805 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:41:00.805 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:41:00.805 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:41:00.805 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 208404 2024-02-09T17:41:00.805 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.805 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:41:00.805 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.805 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:41:00.805 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.818190+0000", 2024-02-09T17:41:00.805 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:41:00.805 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.806 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.806 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:41:00.806 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.818190+0000", 2024-02-09T17:41:00.806 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:41:00.806 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.806 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.806 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:41:00.806 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.818192+0000", 2024-02-09T17:41:00.806 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.358e-06 2024-02-09T17:41:00.806 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.806 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.806 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:41:00.806 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.818195+0000", 2024-02-09T17:41:00.806 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.1020000000000001e-06 2024-02-09T17:41:00.806 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.807 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.807 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:41:00.807 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.818203+0000", 2024-02-09T17:41:00.807 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.585e-06 2024-02-09T17:41:00.807 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.807 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.807 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:41:00.807 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.818210+0000", 2024-02-09T17:41:00.807 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.545e-06 2024-02-09T17:41:00.807 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.807 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.807 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.807 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.820340+0000", 2024-02-09T17:41:00.807 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0021302629999999999 2024-02-09T17:41:00.807 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.808 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.808 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:41:00.808 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.820351+0000", 2024-02-09T17:41:00.808 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0514e-05 2024-02-09T17:41:00.808 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.808 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.808 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.808 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.810548+0000", 2024-02-09T17:41:00.808 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.99019749999999995 2024-02-09T17:41:00.808 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.808 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.808 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:41:00.808 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.810555+0000", 2024-02-09T17:41:00.808 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.0149999999999998e-06 2024-02-09T17:41:00.809 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.809 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.809 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.809 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.063853+0000", 2024-02-09T17:41:00.809 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2532982509999999 2024-02-09T17:41:00.809 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.809 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.809 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:41:00.809 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.067093+0000", 2024-02-09T17:41:00.809 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0032394160000000002 2024-02-09T17:41:00.809 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.809 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.809 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 6,7,10", 2024-02-09T17:41:00.809 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.067178+0000", 2024-02-09T17:41:00.809 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.5220000000000001e-05 2024-02-09T17:41:00.810 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.810 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.810 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.810 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.071071+0000", 2024-02-09T17:41:00.810 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0038930480000000001 2024-02-09T17:41:00.810 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.810 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.810 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:41:00.810 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.074920+0000", 2024-02-09T17:41:00.810 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0038487450000000002 2024-02-09T17:41:00.810 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.810 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.810 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.810 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.079243+0000", 2024-02-09T17:41:00.810 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0043232460000000002 2024-02-09T17:41:00.811 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.811 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.811 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.811 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.096462+0000", 2024-02-09T17:41:00.811 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.017218444999999999 2024-02-09T17:41:00.811 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.811 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.811 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:41:00.811 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.096526+0000", 2024-02-09T17:41:00.811 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.4646000000000004e-05 2024-02-09T17:41:00.811 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.811 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.811 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:41:00.811 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.096594+0000", 2024-02-09T17:41:00.812 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.758e-05 2024-02-09T17:41:00.812 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:41:00.812 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:41:00.812 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:41:00.812 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.812 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.812 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:208412 4.f 4:f71ab7a9:::benchmark_data_smithi179_103809_object70946:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-02-09T17:41:00.812 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:38:33.828165+0000", 2024-02-09T17:41:00.812 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 146.92194141600001, 2024-02-09T17:41:00.812 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2159022140000002, 2024-02-09T17:41:00.812 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:41:00.812 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:41:00.812 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:41:00.812 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:41:00.812 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:41:00.813 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 208412 2024-02-09T17:41:00.813 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.813 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:41:00.813 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.813 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:41:00.813 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.828165+0000", 2024-02-09T17:41:00.813 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:41:00.813 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.813 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.813 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:41:00.813 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.828165+0000", 2024-02-09T17:41:00.813 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:41:00.813 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.813 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.813 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:41:00.814 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.828166+0000", 2024-02-09T17:41:00.814 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0249999999999999e-06 2024-02-09T17:41:00.814 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.814 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.814 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:41:00.814 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.828169+0000", 2024-02-09T17:41:00.814 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.039e-06 2024-02-09T17:41:00.814 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.814 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.814 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:41:00.814 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.828178+0000", 2024-02-09T17:41:00.814 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.1349999999999998e-06 2024-02-09T17:41:00.814 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.814 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.814 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:41:00.815 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.828185+0000", 2024-02-09T17:41:00.815 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.1620000000000002e-06 2024-02-09T17:41:00.815 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.815 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.815 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.815 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.831810+0000", 2024-02-09T17:41:00.815 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.003625133 2024-02-09T17:41:00.815 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.815 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.815 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:41:00.815 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.831817+0000", 2024-02-09T17:41:00.815 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.5830000000000003e-06 2024-02-09T17:41:00.815 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.815 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.815 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.816 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.810755+0000", 2024-02-09T17:41:00.816 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.97893847300000003 2024-02-09T17:41:00.816 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.816 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.816 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:41:00.816 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.810766+0000", 2024-02-09T17:41:00.816 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0362e-05 2024-02-09T17:41:00.816 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.816 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.816 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.816 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.026526+0000", 2024-02-09T17:41:00.816 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2157602970000001 2024-02-09T17:41:00.816 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.816 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.816 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:41:00.817 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.026642+0000", 2024-02-09T17:41:00.817 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00011571899999999999 2024-02-09T17:41:00.817 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.817 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.817 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,9,11", 2024-02-09T17:41:00.817 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.026770+0000", 2024-02-09T17:41:00.817 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012810600000000001 2024-02-09T17:41:00.817 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.817 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.817 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.817 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.037750+0000", 2024-02-09T17:41:00.817 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.010980492999999999 2024-02-09T17:41:00.817 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.817 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.817 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:41:00.818 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.038050+0000", 2024-02-09T17:41:00.818 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00029930899999999999 2024-02-09T17:41:00.818 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.818 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.818 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.818 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.043224+0000", 2024-02-09T17:41:00.818 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.005174344 2024-02-09T17:41:00.818 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.818 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.818 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.818 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.043952+0000", 2024-02-09T17:41:00.818 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00072777900000000001 2024-02-09T17:41:00.818 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.818 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.818 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:41:00.819 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.043996+0000", 2024-02-09T17:41:00.819 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.4234999999999999e-05 2024-02-09T17:41:00.819 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.819 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.819 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:41:00.819 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.044067+0000", 2024-02-09T17:41:00.819 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.1019999999999994e-05 2024-02-09T17:41:00.819 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:41:00.819 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:41:00.819 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:41:00.819 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.819 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.819 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:208420 4.f 4:f8390c68:::benchmark_data_smithi179_103809_object70954:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-02-09T17:41:00.819 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:38:33.837086+0000", 2024-02-09T17:41:00.820 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 146.913020751, 2024-02-09T17:41:00.820 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2070928369999998, 2024-02-09T17:41:00.820 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:41:00.820 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:41:00.820 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:41:00.820 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:41:00.820 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:41:00.820 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 208420 2024-02-09T17:41:00.820 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.820 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:41:00.820 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.820 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:41:00.820 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.837086+0000", 2024-02-09T17:41:00.820 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:41:00.821 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.821 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.821 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:41:00.821 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.837086+0000", 2024-02-09T17:41:00.821 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:41:00.821 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.821 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.821 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:41:00.821 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.837087+0000", 2024-02-09T17:41:00.821 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.158e-06 2024-02-09T17:41:00.821 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.821 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.821 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:41:00.821 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.837091+0000", 2024-02-09T17:41:00.821 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.0169999999999998e-06 2024-02-09T17:41:00.821 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.822 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.822 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:41:00.822 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.837100+0000", 2024-02-09T17:41:00.822 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.2390000000000008e-06 2024-02-09T17:41:00.822 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.822 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.822 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:41:00.822 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.837107+0000", 2024-02-09T17:41:00.822 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.6959999999999999e-06 2024-02-09T17:41:00.822 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.822 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.822 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.822 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.837156+0000", 2024-02-09T17:41:00.822 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.8928999999999999e-05 2024-02-09T17:41:00.822 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.823 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.823 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:41:00.823 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.837163+0000", 2024-02-09T17:41:00.823 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.306e-06 2024-02-09T17:41:00.823 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.823 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.823 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.823 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.810843+0000", 2024-02-09T17:41:00.823 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.97367960899999995 2024-02-09T17:41:00.823 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.823 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.823 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:41:00.823 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.810852+0000", 2024-02-09T17:41:00.823 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.944e-06 2024-02-09T17:41:00.823 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.823 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.824 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.824 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.032038+0000", 2024-02-09T17:41:00.824 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2211856670000001 2024-02-09T17:41:00.824 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.824 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.824 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:41:00.824 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.032189+0000", 2024-02-09T17:41:00.824 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00015130499999999999 2024-02-09T17:41:00.824 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.824 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.824 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,9,11", 2024-02-09T17:41:00.824 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.032342+0000", 2024-02-09T17:41:00.824 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00015267300000000001 2024-02-09T17:41:00.824 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.824 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.825 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.825 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.038032+0000", 2024-02-09T17:41:00.825 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0056899239999999998 2024-02-09T17:41:00.825 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.825 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.825 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:41:00.825 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.038056+0000", 2024-02-09T17:41:00.825 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.3737e-05 2024-02-09T17:41:00.825 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.825 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.826 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.826 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.043294+0000", 2024-02-09T17:41:00.826 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0052377689999999998 2024-02-09T17:41:00.826 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.826 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.826 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.826 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.044091+0000", 2024-02-09T17:41:00.826 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00079718400000000004 2024-02-09T17:41:00.826 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.826 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.826 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:41:00.826 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.044121+0000", 2024-02-09T17:41:00.826 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.0623000000000001e-05 2024-02-09T17:41:00.826 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.826 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.827 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:41:00.827 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.044178+0000", 2024-02-09T17:41:00.827 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7056999999999999e-05 2024-02-09T17:41:00.827 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:41:00.827 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:41:00.827 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:41:00.827 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.827 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.827 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:208424 4.a 4:57643a18:::benchmark_data_smithi179_103809_object70958:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-02-09T17:41:00.827 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:38:33.852457+0000", 2024-02-09T17:41:00.827 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 146.89764973800001, 2024-02-09T17:41:00.827 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.24435845, 2024-02-09T17:41:00.827 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:41:00.827 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:41:00.827 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:41:00.828 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:41:00.828 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:41:00.828 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 208424 2024-02-09T17:41:00.828 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.828 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:41:00.828 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.828 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:41:00.828 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.852457+0000", 2024-02-09T17:41:00.828 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:41:00.828 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.828 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.828 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:41:00.828 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.852457+0000", 2024-02-09T17:41:00.828 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:41:00.828 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.829 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.829 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:41:00.829 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.852458+0000", 2024-02-09T17:41:00.829 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2780000000000001e-06 2024-02-09T17:41:00.829 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.829 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.829 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:41:00.829 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.852463+0000", 2024-02-09T17:41:00.829 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.0189999999999997e-06 2024-02-09T17:41:00.829 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.829 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.829 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:41:00.829 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.852474+0000", 2024-02-09T17:41:00.829 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0637e-05 2024-02-09T17:41:00.829 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.829 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.830 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:41:00.830 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.852480+0000", 2024-02-09T17:41:00.830 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.5069999999999997e-06 2024-02-09T17:41:00.830 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.830 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.830 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.830 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.852536+0000", 2024-02-09T17:41:00.830 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6042000000000003e-05 2024-02-09T17:41:00.830 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.830 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.830 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:41:00.830 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.852545+0000", 2024-02-09T17:41:00.830 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.8019999999999998e-06 2024-02-09T17:41:00.830 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.831 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.831 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.831 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.810592+0000", 2024-02-09T17:41:00.831 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.95804690800000003 2024-02-09T17:41:00.831 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.831 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.831 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:41:00.831 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.810601+0000", 2024-02-09T17:41:00.831 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.132e-06 2024-02-09T17:41:00.831 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.831 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.831 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.831 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.071108+0000", 2024-02-09T17:41:00.831 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2605074979999999 2024-02-09T17:41:00.831 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.832 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.832 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:41:00.832 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.071189+0000", 2024-02-09T17:41:00.832 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.0723000000000006e-05 2024-02-09T17:41:00.832 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.832 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.832 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 6,7,10", 2024-02-09T17:41:00.832 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.071301+0000", 2024-02-09T17:41:00.832 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00011174 2024-02-09T17:41:00.832 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.832 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.832 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.832 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.079299+0000", 2024-02-09T17:41:00.832 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0079979620000000008 2024-02-09T17:41:00.833 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.833 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.833 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:41:00.833 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.079314+0000", 2024-02-09T17:41:00.833 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4751e-05 2024-02-09T17:41:00.833 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.833 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.833 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.833 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.096314+0000", 2024-02-09T17:41:00.833 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.017000767 2024-02-09T17:41:00.833 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.833 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.833 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.833 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.096648+0000", 2024-02-09T17:41:00.833 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00033350399999999998 2024-02-09T17:41:00.834 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.834 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.834 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:41:00.834 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.096704+0000", 2024-02-09T17:41:00.834 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6137999999999997e-05 2024-02-09T17:41:00.834 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.834 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.834 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:41:00.834 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.096815+0000", 2024-02-09T17:41:00.834 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000111042 2024-02-09T17:41:00.834 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:41:00.834 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:41:00.834 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:41:00.834 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.835 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.835 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:208410 4.9 4:96199963:::benchmark_data_smithi179_103809_object70944:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e380)", 2024-02-09T17:41:00.835 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:38:34.811330+0000", 2024-02-09T17:41:00.835 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 145.93877608700001, 2024-02-09T17:41:00.835 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.339120402, 2024-02-09T17:41:00.835 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:41:00.835 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:41:00.835 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:41:00.835 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:41:00.835 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:41:00.835 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 208410 2024-02-09T17:41:00.835 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.835 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:41:00.835 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.836 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:41:00.836 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.811330+0000", 2024-02-09T17:41:00.836 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:41:00.836 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.836 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.836 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:41:00.836 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.811330+0000", 2024-02-09T17:41:00.836 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:41:00.836 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.836 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.836 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:41:00.836 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.811332+0000", 2024-02-09T17:41:00.836 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2559999999999999e-06 2024-02-09T17:41:00.836 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.836 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.837 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:41:00.837 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.811336+0000", 2024-02-09T17:41:00.837 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.9599999999999999e-06 2024-02-09T17:41:00.837 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.837 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.837 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:41:00.837 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.811346+0000", 2024-02-09T17:41:00.837 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.5340000000000005e-06 2024-02-09T17:41:00.837 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.837 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.837 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:41:00.837 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.811355+0000", 2024-02-09T17:41:00.837 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.2599999999999994e-06 2024-02-09T17:41:00.837 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.838 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.838 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.838 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.811423+0000", 2024-02-09T17:41:00.838 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.7872000000000003e-05 2024-02-09T17:41:00.838 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.838 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.838 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:41:00.838 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.811433+0000", 2024-02-09T17:41:00.838 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.8940000000000005e-06 2024-02-09T17:41:00.838 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.838 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.838 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.838 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.120834+0000", 2024-02-09T17:41:00.838 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.309400841 2024-02-09T17:41:00.838 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.839 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.839 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:41:00.839 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.124192+0000", 2024-02-09T17:41:00.839 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0033576330000000001 2024-02-09T17:41:00.839 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.839 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.839 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,3,4,6,11", 2024-02-09T17:41:00.839 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.124373+0000", 2024-02-09T17:41:00.839 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000181187 2024-02-09T17:41:00.839 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.839 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.839 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.839 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.138957+0000", 2024-02-09T17:41:00.839 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.014584456000000001 2024-02-09T17:41:00.840 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.840 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.840 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.840 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.139006+0000", 2024-02-09T17:41:00.840 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.8989999999999997e-05 2024-02-09T17:41:00.840 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.840 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.840 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.840 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.139139+0000", 2024-02-09T17:41:00.840 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000132597 2024-02-09T17:41:00.840 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.840 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.840 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:41:00.840 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.144076+0000", 2024-02-09T17:41:00.841 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.004937617 2024-02-09T17:41:00.841 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.841 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.841 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.841 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.150149+0000", 2024-02-09T17:41:00.841 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0060727510000000004 2024-02-09T17:41:00.841 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.841 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.841 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.841 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.150251+0000", 2024-02-09T17:41:00.841 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000102126 2024-02-09T17:41:00.841 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.841 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.841 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:41:00.841 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.150308+0000", 2024-02-09T17:41:00.842 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7197000000000001e-05 2024-02-09T17:41:00.842 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.842 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.842 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:41:00.842 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.150451+0000", 2024-02-09T17:41:00.842 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014223099999999999 2024-02-09T17:41:00.842 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:41:00.842 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:41:00.842 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:41:00.842 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.842 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.842 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:208415 4.9 4:9dcaa6ee:::benchmark_data_smithi179_103809_object70949:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e380)", 2024-02-09T17:41:00.842 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:38:34.811375+0000", 2024-02-09T17:41:00.842 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 145.93873159099999, 2024-02-09T17:41:00.842 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.339417547, 2024-02-09T17:41:00.843 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:41:00.843 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:41:00.843 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:41:00.843 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:41:00.843 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:41:00.843 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 208415 2024-02-09T17:41:00.843 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.843 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:41:00.843 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.843 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:41:00.843 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.811375+0000", 2024-02-09T17:41:00.843 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:41:00.843 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.843 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.843 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:41:00.843 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.811375+0000", 2024-02-09T17:41:00.844 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:41:00.844 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.844 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.844 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:41:00.844 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.811375+0000", 2024-02-09T17:41:00.844 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2e-07 2024-02-09T17:41:00.844 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.844 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.844 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:41:00.844 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.811378+0000", 2024-02-09T17:41:00.844 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.4420000000000001e-06 2024-02-09T17:41:00.844 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.844 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.844 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:41:00.844 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.811385+0000", 2024-02-09T17:41:00.845 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.9160000000000004e-06 2024-02-09T17:41:00.845 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.845 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.845 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:41:00.845 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.811403+0000", 2024-02-09T17:41:00.845 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.8108999999999999e-05 2024-02-09T17:41:00.845 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.845 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.845 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.845 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.811464+0000", 2024-02-09T17:41:00.845 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.1223999999999995e-05 2024-02-09T17:41:00.845 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.845 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.845 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:41:00.845 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.811472+0000", 2024-02-09T17:41:00.846 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.5950000000000001e-06 2024-02-09T17:41:00.846 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.846 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.846 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.846 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.128953+0000", 2024-02-09T17:41:00.846 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3174819369999999 2024-02-09T17:41:00.846 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.846 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.846 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:41:00.846 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.129041+0000", 2024-02-09T17:41:00.846 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.7145000000000007e-05 2024-02-09T17:41:00.846 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.846 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.846 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,3,4,6,11", 2024-02-09T17:41:00.846 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.129125+0000", 2024-02-09T17:41:00.846 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.4670000000000001e-05 2024-02-09T17:41:00.847 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.847 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.847 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.847 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.139049+0000", 2024-02-09T17:41:00.847 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0099241659999999999 2024-02-09T17:41:00.847 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.847 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.847 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:41:00.847 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.150099+0000", 2024-02-09T17:41:00.847 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.011049188999999999 2024-02-09T17:41:00.847 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.847 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.847 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.847 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.150198+0000", 2024-02-09T17:41:00.847 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.946e-05 2024-02-09T17:41:00.848 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.848 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.848 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.848 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.150494+0000", 2024-02-09T17:41:00.848 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00029567299999999999 2024-02-09T17:41:00.848 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.848 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.848 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.848 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.150587+0000", 2024-02-09T17:41:00.848 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.3043999999999999e-05 2024-02-09T17:41:00.848 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.848 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.848 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.848 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.150626+0000", 2024-02-09T17:41:00.848 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9113999999999999e-05 2024-02-09T17:41:00.849 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.849 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.849 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:41:00.849 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.150678+0000", 2024-02-09T17:41:00.849 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2358999999999998e-05 2024-02-09T17:41:00.849 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.849 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.849 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:41:00.849 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.150792+0000", 2024-02-09T17:41:00.849 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000113984 2024-02-09T17:41:00.849 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:41:00.849 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:41:00.849 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:41:00.849 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.849 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.850 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:208419 4.9 4:94ab8e49:::benchmark_data_smithi179_103809_object70953:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e380)", 2024-02-09T17:41:00.850 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:38:34.811412+0000", 2024-02-09T17:41:00.850 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 145.93869411899999, 2024-02-09T17:41:00.850 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3399166849999999, 2024-02-09T17:41:00.850 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:41:00.850 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:41:00.850 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:41:00.850 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:41:00.850 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:41:00.850 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 208419 2024-02-09T17:41:00.850 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.850 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:41:00.850 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.850 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:41:00.850 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.811412+0000", 2024-02-09T17:41:00.850 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:41:00.851 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.851 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.851 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:41:00.851 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.811412+0000", 2024-02-09T17:41:00.851 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:41:00.851 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.851 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.851 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:41:00.851 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.811413+0000", 2024-02-09T17:41:00.851 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.8599999999999999e-07 2024-02-09T17:41:00.851 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.851 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.851 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:41:00.851 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.811415+0000", 2024-02-09T17:41:00.851 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.5009999999999999e-06 2024-02-09T17:41:00.852 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.852 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.852 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:41:00.852 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.811422+0000", 2024-02-09T17:41:00.852 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.5710000000000003e-06 2024-02-09T17:41:00.852 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.852 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.852 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:41:00.852 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.811434+0000", 2024-02-09T17:41:00.852 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2656999999999999e-05 2024-02-09T17:41:00.852 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.852 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.852 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.852 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.811496+0000", 2024-02-09T17:41:00.852 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.1896999999999994e-05 2024-02-09T17:41:00.853 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.853 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.853 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:41:00.853 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.811503+0000", 2024-02-09T17:41:00.853 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.17e-06 2024-02-09T17:41:00.853 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.853 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.853 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.853 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.133540+0000", 2024-02-09T17:41:00.853 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.322036762 2024-02-09T17:41:00.853 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.853 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.853 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:41:00.853 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.133742+0000", 2024-02-09T17:41:00.853 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00020171300000000001 2024-02-09T17:41:00.853 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.854 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.854 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,3,4,6,11", 2024-02-09T17:41:00.854 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.133931+0000", 2024-02-09T17:41:00.854 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00018926500000000001 2024-02-09T17:41:00.854 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.854 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.854 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.854 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.139093+0000", 2024-02-09T17:41:00.854 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0051615929999999999 2024-02-09T17:41:00.854 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.854 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.854 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:41:00.854 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.150106+0000", 2024-02-09T17:41:00.854 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.011012815 2024-02-09T17:41:00.854 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.855 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.855 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.855 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.150821+0000", 2024-02-09T17:41:00.855 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00071553100000000004 2024-02-09T17:41:00.855 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.855 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.855 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.855 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.151065+0000", 2024-02-09T17:41:00.855 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000244007 2024-02-09T17:41:00.855 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.855 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.855 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.855 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.151155+0000", 2024-02-09T17:41:00.855 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.0369000000000002e-05 2024-02-09T17:41:00.855 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.855 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.856 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.856 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.151198+0000", 2024-02-09T17:41:00.856 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2556999999999999e-05 2024-02-09T17:41:00.856 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.856 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.856 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:41:00.856 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.151232+0000", 2024-02-09T17:41:00.856 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.4082999999999999e-05 2024-02-09T17:41:00.856 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.856 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.856 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:41:00.856 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.151329+0000", 2024-02-09T17:41:00.856 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.6807999999999999e-05 2024-02-09T17:41:00.856 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:41:00.856 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:41:00.857 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:41:00.857 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.857 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.857 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:222268 4.a 4:5f76f052:::benchmark_data_smithi179_103809_object84802:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-02-09T17:41:00.857 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:39:02.603039+0000", 2024-02-09T17:41:00.857 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 118.147067245, 2024-02-09T17:41:00.857 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2423865329999999, 2024-02-09T17:41:00.857 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:41:00.857 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:41:00.857 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:41:00.857 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:41:00.857 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:41:00.857 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 222268 2024-02-09T17:41:00.857 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.857 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:41:00.858 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.858 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:41:00.858 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:02.603039+0000", 2024-02-09T17:41:00.858 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:41:00.858 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.858 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.858 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:41:00.858 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:02.603039+0000", 2024-02-09T17:41:00.858 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:41:00.858 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.858 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.858 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:41:00.858 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:02.603041+0000", 2024-02-09T17:41:00.858 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5510000000000001e-06 2024-02-09T17:41:00.859 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.859 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.859 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:41:00.859 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:02.603045+0000", 2024-02-09T17:41:00.859 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2119999999999997e-06 2024-02-09T17:41:00.859 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.859 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.859 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:41:00.859 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:02.603054+0000", 2024-02-09T17:41:00.859 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.2490000000000007e-06 2024-02-09T17:41:00.859 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.859 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.859 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:41:00.859 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:02.603061+0000", 2024-02-09T17:41:00.859 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.3629999999999996e-06 2024-02-09T17:41:00.860 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.860 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.860 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.860 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:02.603222+0000", 2024-02-09T17:41:00.860 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00016069599999999999 2024-02-09T17:41:00.860 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.860 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.860 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:41:00.860 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:02.603229+0000", 2024-02-09T17:41:00.860 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.8770000000000004e-06 2024-02-09T17:41:00.860 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.860 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.860 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.860 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:03.593353+0000", 2024-02-09T17:41:00.861 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.99012438400000002 2024-02-09T17:41:00.861 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.861 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.861 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:41:00.861 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:03.593372+0000", 2024-02-09T17:41:00.861 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.8258e-05 2024-02-09T17:41:00.861 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.861 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.861 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.861 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:04.822274+0000", 2024-02-09T17:41:00.861 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.228901875 2024-02-09T17:41:00.861 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.861 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.861 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:41:00.861 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:04.822386+0000", 2024-02-09T17:41:00.861 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000112428 2024-02-09T17:41:00.862 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.862 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.862 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 5,7,10", 2024-02-09T17:41:00.862 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:04.822515+0000", 2024-02-09T17:41:00.862 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000129338 2024-02-09T17:41:00.862 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.862 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.862 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:41:00.862 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:04.836316+0000", 2024-02-09T17:41:00.862 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.013800442 2024-02-09T17:41:00.862 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.862 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.862 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.862 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:04.836359+0000", 2024-02-09T17:41:00.862 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.3720000000000002e-05 2024-02-09T17:41:00.863 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.863 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.863 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.863 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:04.845056+0000", 2024-02-09T17:41:00.863 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0086968759999999992 2024-02-09T17:41:00.863 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.863 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.863 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.863 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:04.845224+0000", 2024-02-09T17:41:00.863 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000167889 2024-02-09T17:41:00.863 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.863 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.863 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:41:00.863 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:04.845291+0000", 2024-02-09T17:41:00.863 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.7212000000000006e-05 2024-02-09T17:41:00.864 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.864 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.864 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:41:00.864 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:04.845426+0000", 2024-02-09T17:41:00.864 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000134163 2024-02-09T17:41:00.864 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:41:00.864 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:41:00.864 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:41:00.864 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.864 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.864 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:222305 4.a 4:5aeef130:::benchmark_data_smithi179_103809_object84839:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e399)", 2024-02-09T17:41:00.864 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:39:03.789959+0000", 2024-02-09T17:41:00.864 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 116.960147791, 2024-02-09T17:41:00.864 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.055702921, 2024-02-09T17:41:00.864 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:41:00.865 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:41:00.865 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:41:00.865 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:41:00.865 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:41:00.865 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 222305 2024-02-09T17:41:00.865 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.865 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:41:00.865 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.865 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:41:00.865 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:03.789959+0000", 2024-02-09T17:41:00.865 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:41:00.865 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.865 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.865 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:41:00.865 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:03.789959+0000", 2024-02-09T17:41:00.866 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:41:00.866 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.866 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.866 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:41:00.866 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:03.789960+0000", 2024-02-09T17:41:00.866 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0389999999999999e-06 2024-02-09T17:41:00.866 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.866 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.866 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:41:00.866 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:03.789964+0000", 2024-02-09T17:41:00.866 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2440000000000004e-06 2024-02-09T17:41:00.866 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.866 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.866 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:41:00.866 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:03.789974+0000", 2024-02-09T17:41:00.867 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0047999999999999e-05 2024-02-09T17:41:00.867 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.867 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.867 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:41:00.867 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:03.789980+0000", 2024-02-09T17:41:00.867 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7590000000000003e-06 2024-02-09T17:41:00.867 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.867 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.867 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.867 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:03.790042+0000", 2024-02-09T17:41:00.867 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.2090999999999999e-05 2024-02-09T17:41:00.867 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.867 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.867 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:41:00.867 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:03.790055+0000", 2024-02-09T17:41:00.868 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.288e-05 2024-02-09T17:41:00.868 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.868 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.868 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.868 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:04.827439+0000", 2024-02-09T17:41:00.868 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0373842870000001 2024-02-09T17:41:00.868 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.868 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.868 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:41:00.868 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:04.827551+0000", 2024-02-09T17:41:00.868 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000112283 2024-02-09T17:41:00.868 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.868 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.868 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 5,7,10", 2024-02-09T17:41:00.869 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:04.827688+0000", 2024-02-09T17:41:00.869 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013701499999999999 2024-02-09T17:41:00.869 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.869 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.869 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:41:00.869 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:04.840580+0000", 2024-02-09T17:41:00.869 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.012891767 2024-02-09T17:41:00.869 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.869 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.869 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.869 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:04.845116+0000", 2024-02-09T17:41:00.869 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0045357540000000003 2024-02-09T17:41:00.869 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.869 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.869 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.869 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:04.845175+0000", 2024-02-09T17:41:00.870 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.9490999999999997e-05 2024-02-09T17:41:00.870 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.870 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.870 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.870 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:04.845474+0000", 2024-02-09T17:41:00.870 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00029928399999999998 2024-02-09T17:41:00.870 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.870 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.870 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:41:00.870 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:04.845541+0000", 2024-02-09T17:41:00.870 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.6828999999999993e-05 2024-02-09T17:41:00.870 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.870 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.870 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:41:00.870 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:04.845661+0000", 2024-02-09T17:41:00.871 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012015000000000001 2024-02-09T17:41:00.871 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:41:00.871 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:41:00.871 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:41:00.871 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.871 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.871 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:232427 4.a 4:58636665:::benchmark_data_smithi179_103809_object94961:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-02-09T17:41:00.871 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:39:22.538697+0000", 2024-02-09T17:41:00.871 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 98.211409098999994, 2024-02-09T17:41:00.871 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.040289537, 2024-02-09T17:41:00.871 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:41:00.871 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:41:00.871 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:41:00.871 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:41:00.871 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:41:00.871 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 232427 2024-02-09T17:41:00.872 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.872 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:41:00.872 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.872 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:41:00.872 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:22.538697+0000", 2024-02-09T17:41:00.872 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:41:00.872 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.872 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.872 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:41:00.872 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:22.538697+0000", 2024-02-09T17:41:00.872 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:41:00.872 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.872 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.872 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:41:00.873 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:22.538699+0000", 2024-02-09T17:41:00.873 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.9989999999999998e-06 2024-02-09T17:41:00.873 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.873 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.873 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:41:00.873 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:22.538706+0000", 2024-02-09T17:41:00.873 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.3980000000000004e-06 2024-02-09T17:41:00.873 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.873 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.873 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:41:00.873 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:22.538720+0000", 2024-02-09T17:41:00.873 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4402e-05 2024-02-09T17:41:00.873 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.873 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.873 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:41:00.874 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:22.538728+0000", 2024-02-09T17:41:00.874 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.191e-06 2024-02-09T17:41:00.874 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.874 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.874 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.874 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:22.538800+0000", 2024-02-09T17:41:00.874 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.1508999999999996e-05 2024-02-09T17:41:00.874 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.874 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.874 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:41:00.874 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:22.538809+0000", 2024-02-09T17:41:00.874 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.9970000000000004e-06 2024-02-09T17:41:00.874 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.874 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.874 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.875 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.568239+0000", 2024-02-09T17:41:00.875 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.029430676 2024-02-09T17:41:00.875 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.875 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.875 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:41:00.875 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.568349+0000", 2024-02-09T17:41:00.875 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00010967 2024-02-09T17:41:00.875 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.875 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.875 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 5,10", 2024-02-09T17:41:00.875 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.568441+0000", 2024-02-09T17:41:00.875 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.2266000000000001e-05 2024-02-09T17:41:00.875 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.875 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.875 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.876 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.572628+0000", 2024-02-09T17:41:00.876 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0041869070000000001 2024-02-09T17:41:00.876 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.876 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.876 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:41:00.876 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.575783+0000", 2024-02-09T17:41:00.876 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0031545499999999999 2024-02-09T17:41:00.876 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.876 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.876 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.876 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.578929+0000", 2024-02-09T17:41:00.876 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0031464000000000002 2024-02-09T17:41:00.876 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.876 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.876 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:41:00.877 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.578962+0000", 2024-02-09T17:41:00.877 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.2634e-05 2024-02-09T17:41:00.877 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.877 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.877 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:41:00.877 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.578987+0000", 2024-02-09T17:41:00.877 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.4938000000000001e-05 2024-02-09T17:41:00.877 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:41:00.877 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:41:00.877 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:41:00.877 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.877 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.877 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:232455 4.a 4:5c3729de:::benchmark_data_smithi179_103809_object94989:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-02-09T17:41:00.877 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:39:22.579430+0000", 2024-02-09T17:41:00.877 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 98.170676412999995, 2024-02-09T17:41:00.878 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.011227925, 2024-02-09T17:41:00.878 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:41:00.878 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:41:00.878 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:41:00.878 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:41:00.878 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:41:00.878 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 232455 2024-02-09T17:41:00.878 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.878 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:41:00.878 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.878 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:41:00.878 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:22.579430+0000", 2024-02-09T17:41:00.878 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:41:00.878 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.878 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.879 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:41:00.879 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:22.579430+0000", 2024-02-09T17:41:00.879 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:41:00.879 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.879 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.879 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:41:00.879 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:22.579431+0000", 2024-02-09T17:41:00.879 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.22e-06 2024-02-09T17:41:00.879 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.879 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.879 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:41:00.879 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:22.579436+0000", 2024-02-09T17:41:00.879 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2689999999999999e-06 2024-02-09T17:41:00.879 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.879 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.880 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:41:00.880 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:22.579447+0000", 2024-02-09T17:41:00.880 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0338e-05 2024-02-09T17:41:00.880 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.880 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.880 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:41:00.880 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:22.579453+0000", 2024-02-09T17:41:00.880 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.9599999999999997e-06 2024-02-09T17:41:00.880 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.880 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.880 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.880 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:22.579511+0000", 2024-02-09T17:41:00.880 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7861e-05 2024-02-09T17:41:00.880 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.880 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.881 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:41:00.881 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:22.579521+0000", 2024-02-09T17:41:00.881 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0033e-05 2024-02-09T17:41:00.884 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.884 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.884 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.884 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.572668+0000", 2024-02-09T17:41:00.884 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.99314702600000004 2024-02-09T17:41:00.884 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.884 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.884 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:41:00.884 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.572758+0000", 2024-02-09T17:41:00.885 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.0660000000000003e-05 2024-02-09T17:41:00.885 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.885 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.885 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 5,10", 2024-02-09T17:41:00.885 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.572828+0000", 2024-02-09T17:41:00.885 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.9357999999999996e-05 2024-02-09T17:41:00.885 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.885 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.885 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.885 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.578834+0000", 2024-02-09T17:41:00.885 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0060068020000000003 2024-02-09T17:41:00.885 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.885 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.885 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:41:00.885 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.579059+0000", 2024-02-09T17:41:00.886 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00022447299999999999 2024-02-09T17:41:00.886 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.886 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.886 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.886 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.590550+0000", 2024-02-09T17:41:00.886 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.011491533 2024-02-09T17:41:00.886 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.886 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.886 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:41:00.886 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.590611+0000", 2024-02-09T17:41:00.886 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.0819999999999997e-05 2024-02-09T17:41:00.886 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.886 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.886 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:41:00.886 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.590658+0000", 2024-02-09T17:41:00.887 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.6572000000000001e-05 2024-02-09T17:41:00.887 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:41:00.887 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:41:00.887 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:41:00.887 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.887 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.887 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:232552 4.9 4:9d8f33fb:::benchmark_data_smithi179_103809_object95086:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e414)", 2024-02-09T17:41:00.887 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:39:23.607410+0000", 2024-02-09T17:41:00.887 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 97.142696727000001, 2024-02-09T17:41:00.887 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.99336764799999999, 2024-02-09T17:41:00.887 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:41:00.887 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:41:00.887 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:41:00.887 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:41:00.887 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:41:00.888 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 232552 2024-02-09T17:41:00.888 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.888 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:41:00.888 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.888 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:41:00.888 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.607410+0000", 2024-02-09T17:41:00.888 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:41:00.888 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.888 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.888 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:41:00.888 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.607410+0000", 2024-02-09T17:41:00.888 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:41:00.888 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.888 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.888 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:41:00.889 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.607410+0000", 2024-02-09T17:41:00.889 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2300000000000002e-07 2024-02-09T17:41:00.889 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.889 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.889 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:41:00.889 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.607413+0000", 2024-02-09T17:41:00.889 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.1999999999999999e-06 2024-02-09T17:41:00.889 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.889 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.889 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:41:00.889 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.607425+0000", 2024-02-09T17:41:00.889 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1827e-05 2024-02-09T17:41:00.889 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.889 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.889 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:41:00.890 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.607431+0000", 2024-02-09T17:41:00.890 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.9229999999999999e-06 2024-02-09T17:41:00.890 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.890 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.890 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.890 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.607475+0000", 2024-02-09T17:41:00.890 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.4502000000000002e-05 2024-02-09T17:41:00.890 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.890 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.890 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:41:00.890 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.607481+0000", 2024-02-09T17:41:00.890 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6840000000000001e-06 2024-02-09T17:41:00.890 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.890 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.890 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.891 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.584768+0000", 2024-02-09T17:41:00.891 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.97728727800000004 2024-02-09T17:41:00.891 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.891 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.891 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:41:00.891 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.584863+0000", 2024-02-09T17:41:00.891 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.4399000000000002e-05 2024-02-09T17:41:00.891 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.891 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.891 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,3,4,11", 2024-02-09T17:41:00.891 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.584978+0000", 2024-02-09T17:41:00.891 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000115016 2024-02-09T17:41:00.891 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.891 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.891 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:41:00.892 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.589764+0000", 2024-02-09T17:41:00.892 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0047862499999999997 2024-02-09T17:41:00.892 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.892 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.892 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.892 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.590123+0000", 2024-02-09T17:41:00.892 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00035844000000000002 2024-02-09T17:41:00.892 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.892 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.892 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.892 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.590166+0000", 2024-02-09T17:41:00.892 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.3826999999999999e-05 2024-02-09T17:41:00.892 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.892 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.893 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.893 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.590208+0000", 2024-02-09T17:41:00.893 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.1360999999999999e-05 2024-02-09T17:41:00.893 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.893 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.893 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.893 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.600581+0000", 2024-02-09T17:41:00.893 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.010373746 2024-02-09T17:41:00.893 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.893 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.893 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:41:00.893 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.600650+0000", 2024-02-09T17:41:00.893 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.86e-05 2024-02-09T17:41:00.893 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.893 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.894 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:41:00.894 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.600777+0000", 2024-02-09T17:41:00.894 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012717199999999999 2024-02-09T17:41:00.894 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:41:00.894 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:41:00.894 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:41:00.894 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.894 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.894 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:232568 4.4 4:2cb129ac:::benchmark_data_smithi179_103809_object95102:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e415)", 2024-02-09T17:41:00.894 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:39:24.529753+0000", 2024-02-09T17:41:00.894 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 96.220352930999994, 2024-02-09T17:41:00.894 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.100915619, 2024-02-09T17:41:00.894 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:41:00.894 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:41:00.894 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:41:00.895 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:41:00.895 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:41:00.895 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 232568 2024-02-09T17:41:00.895 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.895 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:41:00.895 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.895 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:41:00.895 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529753+0000", 2024-02-09T17:41:00.895 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:41:00.895 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.895 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.895 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:41:00.895 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529753+0000", 2024-02-09T17:41:00.895 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:41:00.895 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.895 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.896 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:41:00.896 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529755+0000", 2024-02-09T17:41:00.896 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.389e-06 2024-02-09T17:41:00.896 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.896 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.896 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:41:00.896 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529760+0000", 2024-02-09T17:41:00.896 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.4840000000000003e-06 2024-02-09T17:41:00.896 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.896 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.896 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:41:00.896 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529771+0000", 2024-02-09T17:41:00.896 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0529e-05 2024-02-09T17:41:00.896 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.896 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.897 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:41:00.897 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529780+0000", 2024-02-09T17:41:00.897 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.1200000000000008e-06 2024-02-09T17:41:00.897 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.897 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.897 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.897 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529952+0000", 2024-02-09T17:41:00.897 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000172323 2024-02-09T17:41:00.897 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.897 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.897 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:41:00.897 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529964+0000", 2024-02-09T17:41:00.897 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1834000000000001e-05 2024-02-09T17:41:00.897 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.897 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.897 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.898 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.601463+0000", 2024-02-09T17:41:00.898 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0714987389999999 2024-02-09T17:41:00.898 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.898 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.898 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:41:00.898 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.601595+0000", 2024-02-09T17:41:00.898 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013218999999999999 2024-02-09T17:41:00.898 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.898 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.898 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 2,3,7,10", 2024-02-09T17:41:00.898 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.601736+0000", 2024-02-09T17:41:00.898 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014145800000000001 2024-02-09T17:41:00.898 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.898 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.899 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.899 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.619078+0000", 2024-02-09T17:41:00.899 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.017341235999999999 2024-02-09T17:41:00.899 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.899 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.899 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.899 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.619119+0000", 2024-02-09T17:41:00.899 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.1523999999999997e-05 2024-02-09T17:41:00.899 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.899 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.899 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.899 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.619159+0000", 2024-02-09T17:41:00.899 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9576000000000002e-05 2024-02-09T17:41:00.899 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.899 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.899 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:41:00.900 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.625012+0000", 2024-02-09T17:41:00.900 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0058534930000000004 2024-02-09T17:41:00.900 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.900 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.900 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.900 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.630457+0000", 2024-02-09T17:41:00.900 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0054445379999999996 2024-02-09T17:41:00.900 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.900 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.900 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:41:00.900 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.630519+0000", 2024-02-09T17:41:00.900 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.2483999999999993e-05 2024-02-09T17:41:00.900 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.900 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.900 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:41:00.901 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.630669+0000", 2024-02-09T17:41:00.901 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014970200000000001 2024-02-09T17:41:00.901 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:41:00.901 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:41:00.901 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:41:00.901 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.901 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.901 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:232574 4.4 4:2e59b7f1:::benchmark_data_smithi179_103809_object95108:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e415)", 2024-02-09T17:41:00.901 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:39:24.529801+0000", 2024-02-09T17:41:00.901 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 96.220305038000006, 2024-02-09T17:41:00.901 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.10112791, 2024-02-09T17:41:00.901 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:41:00.901 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:41:00.901 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:41:00.902 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:41:00.902 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:41:00.902 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 232574 2024-02-09T17:41:00.902 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.902 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:41:00.902 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.902 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:41:00.902 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529801+0000", 2024-02-09T17:41:00.902 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:41:00.902 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.902 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.902 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:41:00.902 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529801+0000", 2024-02-09T17:41:00.902 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:41:00.902 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.903 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.903 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:41:00.903 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529802+0000", 2024-02-09T17:41:00.903 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3099999999999998e-07 2024-02-09T17:41:00.903 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.903 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.903 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:41:00.903 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529804+0000", 2024-02-09T17:41:00.903 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.5689999999999999e-06 2024-02-09T17:41:00.903 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.903 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.903 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:41:00.903 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529811+0000", 2024-02-09T17:41:00.903 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.8689999999999998e-06 2024-02-09T17:41:00.904 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.904 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.904 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:41:00.904 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529817+0000", 2024-02-09T17:41:00.904 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.9410000000000004e-06 2024-02-09T17:41:00.904 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.904 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.904 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.904 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529995+0000", 2024-02-09T17:41:00.904 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000177842 2024-02-09T17:41:00.904 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.904 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.904 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:41:00.904 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.530004+0000", 2024-02-09T17:41:00.904 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.0510000000000002e-06 2024-02-09T17:41:00.905 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.905 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.905 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.905 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.608017+0000", 2024-02-09T17:41:00.905 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0780130729999999 2024-02-09T17:41:00.905 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.905 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.905 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:41:00.905 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.608153+0000", 2024-02-09T17:41:00.905 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013630000000000001 2024-02-09T17:41:00.905 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.905 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.905 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 2,3,7,10", 2024-02-09T17:41:00.905 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.608294+0000", 2024-02-09T17:41:00.906 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014021699999999999 2024-02-09T17:41:00.906 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.906 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.906 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.906 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.619198+0000", 2024-02-09T17:41:00.906 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.010904173 2024-02-09T17:41:00.906 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.906 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.906 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.906 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.619240+0000", 2024-02-09T17:41:00.906 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2422999999999997e-05 2024-02-09T17:41:00.906 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.906 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.906 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:41:00.906 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.625019+0000", 2024-02-09T17:41:00.907 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0057783959999999999 2024-02-09T17:41:00.907 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.907 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.907 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.907 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.625096+0000", 2024-02-09T17:41:00.907 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.6874999999999998e-05 2024-02-09T17:41:00.907 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.907 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.907 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.907 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.630718+0000", 2024-02-09T17:41:00.907 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0056226269999999998 2024-02-09T17:41:00.907 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.907 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.907 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:41:00.907 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.630786+0000", 2024-02-09T17:41:00.907 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.7532000000000003e-05 2024-02-09T17:41:00.908 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.908 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.908 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:41:00.908 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.630929+0000", 2024-02-09T17:41:00.908 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014349100000000001 2024-02-09T17:41:00.908 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:41:00.908 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:41:00.908 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:41:00.908 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.908 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.908 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:232585 4.4 4:26ea6e8c:::benchmark_data_smithi179_103809_object95119:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-02-09T17:41:00.908 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:39:24.601261+0000", 2024-02-09T17:41:00.908 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 96.148845151000003, 2024-02-09T17:41:00.908 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0298792859999999, 2024-02-09T17:41:00.909 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:41:00.909 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:41:00.909 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:41:00.909 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:41:00.909 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:41:00.909 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 232585 2024-02-09T17:41:00.909 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.909 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:41:00.909 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.909 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:41:00.909 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.601261+0000", 2024-02-09T17:41:00.909 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:41:00.909 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.909 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.909 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:41:00.910 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.601261+0000", 2024-02-09T17:41:00.910 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:41:00.910 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.910 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.910 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:41:00.910 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.601263+0000", 2024-02-09T17:41:00.910 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3170000000000001e-06 2024-02-09T17:41:00.910 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.910 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.910 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:41:00.910 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.601268+0000", 2024-02-09T17:41:00.910 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.13e-06 2024-02-09T17:41:00.910 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.910 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.910 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:41:00.911 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.601276+0000", 2024-02-09T17:41:00.911 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.7399999999999993e-06 2024-02-09T17:41:00.911 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.911 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.911 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:41:00.911 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.601284+0000", 2024-02-09T17:41:00.911 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.7409999999999992e-06 2024-02-09T17:41:00.911 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.911 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.911 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.911 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.601343+0000", 2024-02-09T17:41:00.911 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.9208e-05 2024-02-09T17:41:00.911 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.911 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.911 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:41:00.912 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.601353+0000", 2024-02-09T17:41:00.912 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.2629999999999992e-06 2024-02-09T17:41:00.912 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.912 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.912 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:41:00.912 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.613145+0000", 2024-02-09T17:41:00.912 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0117919879999999 2024-02-09T17:41:00.912 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.912 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.912 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:41:00.912 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.613289+0000", 2024-02-09T17:41:00.912 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000144883 2024-02-09T17:41:00.912 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.912 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.913 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 2,3,7,10", 2024-02-09T17:41:00.913 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.613455+0000", 2024-02-09T17:41:00.913 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000165854 2024-02-09T17:41:00.913 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.913 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.913 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.913 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.624998+0000", 2024-02-09T17:41:00.913 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.011542327 2024-02-09T17:41:00.913 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.913 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.913 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:41:00.913 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.625025+0000", 2024-02-09T17:41:00.913 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.7263999999999998e-05 2024-02-09T17:41:00.913 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.913 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.914 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.914 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.625058+0000", 2024-02-09T17:41:00.914 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.3288999999999999e-05 2024-02-09T17:41:00.914 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.914 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.914 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.914 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.625193+0000", 2024-02-09T17:41:00.914 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000134442 2024-02-09T17:41:00.914 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.914 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.914 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:41:00.914 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.630977+0000", 2024-02-09T17:41:00.914 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0057848250000000004 2024-02-09T17:41:00.914 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.914 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.915 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:41:00.915 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.631011+0000", 2024-02-09T17:41:00.915 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.3411000000000002e-05 2024-02-09T17:41:00.915 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:41:00.915 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:41:00.915 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:41:00.915 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.631140+0000", 2024-02-09T17:41:00.915 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012960399999999999 2024-02-09T17:41:00.915 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:41:00.915 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:41:00.915 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:41:00.915 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:41:00.915 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:41:00.915 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:41:00.974 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:41:00.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:00.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:41:01.078 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:41:01.113 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:01.135 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:41:01.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:01.138 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:41:01.239 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:41:01.269 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:01.291 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:41:01.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:01.293 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:41:01.394 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:41:01.423 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:01.442 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:41:01.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:01.445 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:41:01.448 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:41:01.448 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:41:01.448 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:41:01.448 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:41:01.448 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:41:01.448 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:41:01.459 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:41:01.479 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:41:01.500 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:41:01.521 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:41:01.541 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:41:01.547 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:41:01.562 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:41:01.563 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:41:01.564 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:41:01.564 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:41:01.564 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:41:01.564 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:41:01.564 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:41:01.564 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:41:01.564 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:41:01.565 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:41:01.565 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:41:01.565 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:41:01.565 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:41:01.576 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:01.594 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:41:01.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:01.598 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:41:01.699 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:41:01.727 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:01.745 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:41:01.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:01.748 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:41:01.849 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:41:01.882 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:01.904 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:41:01.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:01.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:41:01.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:01 smithi099 ceph-mon[39420]: pgmap v1198: 37 pgs: 37 active+clean; 577 KiB data, 4.1 GiB used, 979 GiB / 983 GiB avail 2024-02-09T17:41:01.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:01 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/583653380' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:41:01.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:01 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2540713637' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-09T17:41:02.007 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:41:02.039 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:02.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:01 smithi179 ceph-mon[43975]: pgmap v1198: 37 pgs: 37 active+clean; 577 KiB data, 4.1 GiB used, 979 GiB / 983 GiB avail 2024-02-09T17:41:02.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:01 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/583653380' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:41:02.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:01 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2540713637' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-09T17:41:02.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:01 smithi162 ceph-mon[44304]: pgmap v1198: 37 pgs: 37 active+clean; 577 KiB data, 4.1 GiB used, 979 GiB / 983 GiB avail 2024-02-09T17:41:02.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:01 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/583653380' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:41:02.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:01 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2540713637' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-09T17:41:02.493 INFO:teuthology.orchestra.run.smithi099.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-02-09T17:41:02.727 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-09T17:41:02.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:02 smithi099 ceph-mon[39420]: 1.f scrub starts 2024-02-09T17:41:02.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:02 smithi099 ceph-mon[39420]: 1.f scrub ok 2024-02-09T17:41:02.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:02 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2540713637' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-09T17:41:02.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:02 smithi099 ceph-mon[39420]: osdmap e471: 12 total, 12 up, 11 in 2024-02-09T17:41:03.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:02 smithi179 ceph-mon[43975]: 1.f scrub starts 2024-02-09T17:41:03.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:02 smithi179 ceph-mon[43975]: 1.f scrub ok 2024-02-09T17:41:03.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:02 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2540713637' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-09T17:41:03.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:02 smithi179 ceph-mon[43975]: osdmap e471: 12 total, 12 up, 11 in 2024-02-09T17:41:03.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:02 smithi162 ceph-mon[44304]: 1.f scrub starts 2024-02-09T17:41:03.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:02 smithi162 ceph-mon[44304]: 1.f scrub ok 2024-02-09T17:41:03.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:02 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2540713637' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-09T17:41:03.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:02 smithi162 ceph-mon[44304]: osdmap e471: 12 total, 12 up, 11 in 2024-02-09T17:41:04.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:03 smithi179 ceph-mon[43975]: pgmap v1200: 37 pgs: 37 active+clean; 577 KiB data, 4.1 GiB used, 979 GiB / 983 GiB avail 2024-02-09T17:41:04.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:03 smithi179 ceph-mon[43975]: 1.13 scrub starts 2024-02-09T17:41:04.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:03 smithi179 ceph-mon[43975]: 1.13 scrub ok 2024-02-09T17:41:04.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:03 smithi179 ceph-mon[43975]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-09T17:41:04.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:03 smithi179 ceph-mon[43975]: Cluster is now healthy 2024-02-09T17:41:04.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:03 smithi162 ceph-mon[44304]: pgmap v1200: 37 pgs: 37 active+clean; 577 KiB data, 4.1 GiB used, 979 GiB / 983 GiB avail 2024-02-09T17:41:04.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:03 smithi162 ceph-mon[44304]: 1.13 scrub starts 2024-02-09T17:41:04.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:03 smithi162 ceph-mon[44304]: 1.13 scrub ok 2024-02-09T17:41:04.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:03 smithi162 ceph-mon[44304]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-09T17:41:04.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:03 smithi162 ceph-mon[44304]: Cluster is now healthy 2024-02-09T17:41:04.232 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:41:04.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:03 smithi099 ceph-mon[39420]: pgmap v1200: 37 pgs: 37 active+clean; 577 KiB data, 4.1 GiB used, 979 GiB / 983 GiB avail 2024-02-09T17:41:04.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:03 smithi099 ceph-mon[39420]: 1.13 scrub starts 2024-02-09T17:41:04.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:03 smithi099 ceph-mon[39420]: 1.13 scrub ok 2024-02-09T17:41:04.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:03 smithi099 ceph-mon[39420]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-09T17:41:04.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:03 smithi099 ceph-mon[39420]: Cluster is now healthy 2024-02-09T17:41:04.368 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:41:04.654 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:41:04.697 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:41:04.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:04.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:41:04.803 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:41:04.834 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:04.855 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:41:04.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:04.858 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:41:04.868 DEBUG:teuthology.orchestra.run.smithi179:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-02-09T17:41:04.901 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-02-09T17:41:04.901 INFO:teuthology.orchestra.run:waiting for 3000 2024-02-09T17:41:04.927 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:41:04.951 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:hints = 1 2024-02-09T17:41:04.951 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-02-09T17:41:04.951 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Object prefix: benchmark_data_smithi179_115199 2024-02-09T17:41:04.951 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-09T17:41:04.951 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 0 0 0 0 0 0 - 0 2024-02-09T17:41:04.959 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:41:04.996 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:05.154 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:41:05.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:05.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:41:05.259 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:41:05.305 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:05.331 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:41:05.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:05.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:41:05.436 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:41:05.476 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:05.506 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:41:05.509 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:05.510 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:41:05.611 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:41:05.658 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:05.684 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:41:05.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:05.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:41:05.792 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:41:05.818 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:05.837 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:41:05.839 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:05.840 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:41:05.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:05 smithi162 ceph-mon[44304]: pgmap v1201: 37 pgs: 37 active+clean; 577 KiB data, 4.1 GiB used, 979 GiB / 983 GiB avail 2024-02-09T17:41:05.941 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:41:05.951 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 1 16 3647 3631 226.925 226.938 0.00350073 0.00439238 2024-02-09T17:41:05.979 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:06.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:05 smithi179 ceph-mon[43975]: pgmap v1201: 37 pgs: 37 active+clean; 577 KiB data, 4.1 GiB used, 979 GiB / 983 GiB avail 2024-02-09T17:41:06.182 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:41:06.184 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:05 smithi099 ceph-mon[39420]: pgmap v1201: 37 pgs: 37 active+clean; 577 KiB data, 4.1 GiB used, 979 GiB / 983 GiB avail 2024-02-09T17:41:06.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:06.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:41:06.294 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:41:06.330 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:06.459 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:41:06.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:06.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:41:06.566 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:41:06.606 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:06.629 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:41:06.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:06.632 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:41:06.732 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:41:06.759 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:06.791 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:41:06.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:06.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:41:06.897 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:41:06.942 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:06.951 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 2 15 7198 7183 224.453 222 0.00446626 0.00444455 2024-02-09T17:41:06.973 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:41:06.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:06.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:41:07.080 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:41:07.114 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:07.469 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:41:07.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:07.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:41:07.574 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:41:07.612 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:07.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:07 smithi099 ceph-mon[39420]: osdmap e472: 12 total, 12 up, 11 in 2024-02-09T17:41:07.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:07 smithi099 ceph-mon[39420]: 1.11 scrub starts 2024-02-09T17:41:07.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:07 smithi099 ceph-mon[39420]: 1.e scrub starts 2024-02-09T17:41:07.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:07 smithi099 ceph-mon[39420]: pgmap v1203: 37 pgs: 37 active+clean; 100 MiB data, 4.3 GiB used, 979 GiB / 983 GiB avail; 12 MiB/s wr, 198 op/s 2024-02-09T17:41:07.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:07 smithi099 ceph-mon[39420]: 1.11 scrub ok 2024-02-09T17:41:07.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:07 smithi099 ceph-mon[39420]: 1.e scrub ok 2024-02-09T17:41:07.870 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:41:07.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:07.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:41:07.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:07 smithi179 ceph-mon[43975]: osdmap e472: 12 total, 12 up, 11 in 2024-02-09T17:41:07.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:07 smithi179 ceph-mon[43975]: 1.11 scrub starts 2024-02-09T17:41:07.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:07 smithi179 ceph-mon[43975]: 1.e scrub starts 2024-02-09T17:41:07.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:07 smithi179 ceph-mon[43975]: pgmap v1203: 37 pgs: 37 active+clean; 100 MiB data, 4.3 GiB used, 979 GiB / 983 GiB avail; 12 MiB/s wr, 198 op/s 2024-02-09T17:41:07.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:07 smithi179 ceph-mon[43975]: 1.11 scrub ok 2024-02-09T17:41:07.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:07 smithi179 ceph-mon[43975]: 1.e scrub ok 2024-02-09T17:41:07.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:07 smithi162 ceph-mon[44304]: osdmap e472: 12 total, 12 up, 11 in 2024-02-09T17:41:07.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:07 smithi162 ceph-mon[44304]: 1.11 scrub starts 2024-02-09T17:41:07.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:07 smithi162 ceph-mon[44304]: 1.e scrub starts 2024-02-09T17:41:07.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:07 smithi162 ceph-mon[44304]: pgmap v1203: 37 pgs: 37 active+clean; 100 MiB data, 4.3 GiB used, 979 GiB / 983 GiB avail; 12 MiB/s wr, 198 op/s 2024-02-09T17:41:07.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:07 smithi162 ceph-mon[44304]: 1.11 scrub ok 2024-02-09T17:41:07.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:07 smithi162 ceph-mon[44304]: 1.e scrub ok 2024-02-09T17:41:07.951 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 3 16 11073 11057 230.337 242.125 0.00504956 0.0043343 2024-02-09T17:41:07.974 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:41:08.020 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:08.048 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:41:08.051 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:08.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:41:08.152 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:41:08.154 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:41:08.154 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:41:08.194 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:08.224 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:41:08.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:08.228 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:41:08.329 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:41:08.369 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:08.401 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:41:08.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:08.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:41:08.506 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:41:08.551 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:08.575 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:41:08.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:08.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:41:08.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:08 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:41:08.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:08 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:41:08.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:08 smithi162 ceph-mon[44304]: 1.14 scrub starts 2024-02-09T17:41:08.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:08 smithi162 ceph-mon[44304]: 1.14 scrub ok 2024-02-09T17:41:08.679 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:41:08.705 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:08.723 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:41:08.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:08.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:41:08.826 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:41:08.853 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:08.871 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:41:08.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:08.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:41:08.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:08 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:41:08.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:08 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:41:08.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:08 smithi179 ceph-mon[43975]: 1.14 scrub starts 2024-02-09T17:41:08.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:08 smithi179 ceph-mon[43975]: 1.14 scrub ok 2024-02-09T17:41:08.951 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 4 16 13579 13563 211.901 156.625 0.00169328 0.00454775 2024-02-09T17:41:08.975 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:41:08.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:08 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:41:08.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:08 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:41:08.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:08 smithi099 ceph-mon[39420]: 1.14 scrub starts 2024-02-09T17:41:08.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:08 smithi099 ceph-mon[39420]: 1.14 scrub ok 2024-02-09T17:41:09.003 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:09.033 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:41:09.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:09.038 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:41:09.139 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:41:09.179 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:09.734 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:41:09.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:09.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:41:09.741 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:09 smithi099 ceph-mon[39420]: osdmap e473: 12 total, 12 up, 11 in 2024-02-09T17:41:09.741 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:09 smithi099 ceph-mon[39420]: pgmap v1205: 37 pgs: 37 active+clean; 240 MiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 38 MiB/s wr, 608 op/s 2024-02-09T17:41:09.840 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:41:09.888 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:09.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:09 smithi179 ceph-mon[43975]: osdmap e473: 12 total, 12 up, 11 in 2024-02-09T17:41:09.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:09 smithi179 ceph-mon[43975]: pgmap v1205: 37 pgs: 37 active+clean; 240 MiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 38 MiB/s wr, 608 op/s 2024-02-09T17:41:09.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:09 smithi162 ceph-mon[44304]: osdmap e473: 12 total, 12 up, 11 in 2024-02-09T17:41:09.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:09 smithi162 ceph-mon[44304]: pgmap v1205: 37 pgs: 37 active+clean; 240 MiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 38 MiB/s wr, 608 op/s 2024-02-09T17:41:09.951 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 5 16 16853 16837 210.442 204.625 0.00287897 0.00474761 2024-02-09T17:41:09.967 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:41:09.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:09.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:41:10.072 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:41:10.108 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:10.528 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.11 to 1 (10000) 2024-02-09T17:41:10.547 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is set 2024-02-09T17:41:10.567 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-09T17:41:10.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:10 smithi179 ceph-mon[43975]: 1.10 scrub starts 2024-02-09T17:41:10.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:10 smithi179 ceph-mon[43975]: 1.10 scrub ok 2024-02-09T17:41:10.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:10 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1071186021' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:41:10.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:10 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:41:10.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:10 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/285282286' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-02-09T17:41:10.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:10 smithi179 ceph-mon[43975]: 1.0 scrub starts 2024-02-09T17:41:10.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:10 smithi179 ceph-mon[43975]: 1.0 scrub ok 2024-02-09T17:41:10.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:10 smithi162 ceph-mon[44304]: 1.10 scrub starts 2024-02-09T17:41:10.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:10 smithi162 ceph-mon[44304]: 1.10 scrub ok 2024-02-09T17:41:10.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:10 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1071186021' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:41:10.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:10 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:41:10.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:10 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/285282286' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-02-09T17:41:10.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:10 smithi162 ceph-mon[44304]: 1.0 scrub starts 2024-02-09T17:41:10.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:10 smithi162 ceph-mon[44304]: 1.0 scrub ok 2024-02-09T17:41:10.952 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 6 16 18868 18852 196.354 125.938 0.00183043 0.00474244 2024-02-09T17:41:10.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:10 smithi099 ceph-mon[39420]: 1.10 scrub starts 2024-02-09T17:41:10.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:10 smithi099 ceph-mon[39420]: 1.10 scrub ok 2024-02-09T17:41:10.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:10 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1071186021' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:41:10.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:10 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:41:10.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:10 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/285282286' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-02-09T17:41:10.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:10 smithi099 ceph-mon[39420]: 1.0 scrub starts 2024-02-09T17:41:10.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:10 smithi099 ceph-mon[39420]: 1.0 scrub ok 2024-02-09T17:41:11.481 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 0 1 2024-02-09T17:41:11.491 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:41:11.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:11.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:41:11.597 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:41:11.623 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:11.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:11 smithi099 ceph-mon[39420]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:41:11.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:11 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:41:11.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:11 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/285282286' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-02-09T17:41:11.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:11 smithi099 ceph-mon[39420]: osdmap e474: 12 total, 12 up, 11 in 2024-02-09T17:41:11.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:11 smithi099 ceph-mon[39420]: pgmap v1207: 37 pgs: 1 peering, 36 active+clean; 688 MiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 115 MiB/s wr, 1.83k op/s 2024-02-09T17:41:11.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:11 smithi099 ceph-mon[39420]: osdmap e475: 12 total, 12 up, 11 in 2024-02-09T17:41:11.820 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:41:11.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:11.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:41:11.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:11 smithi179 ceph-mon[43975]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:41:11.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:11 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:41:11.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:11 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/285282286' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-02-09T17:41:11.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:11 smithi179 ceph-mon[43975]: osdmap e474: 12 total, 12 up, 11 in 2024-02-09T17:41:11.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:11 smithi179 ceph-mon[43975]: pgmap v1207: 37 pgs: 1 peering, 36 active+clean; 688 MiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 115 MiB/s wr, 1.83k op/s 2024-02-09T17:41:11.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:11 smithi179 ceph-mon[43975]: osdmap e475: 12 total, 12 up, 11 in 2024-02-09T17:41:11.924 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:41:11.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:11 smithi162 ceph-mon[44304]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:41:11.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:11 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:41:11.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:11 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/285282286' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-02-09T17:41:11.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:11 smithi162 ceph-mon[44304]: osdmap e474: 12 total, 12 up, 11 in 2024-02-09T17:41:11.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:11 smithi162 ceph-mon[44304]: pgmap v1207: 37 pgs: 1 peering, 36 active+clean; 688 MiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 115 MiB/s wr, 1.83k op/s 2024-02-09T17:41:11.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:11 smithi162 ceph-mon[44304]: osdmap e475: 12 total, 12 up, 11 in 2024-02-09T17:41:11.951 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:11.952 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 7 16 18868 18852 168.302 0 - 0.00474244 2024-02-09T17:41:12.199 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:41:12.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:12.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:41:12.304 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:41:12.331 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:12.351 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:41:12.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:12.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:41:12.455 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:41:12.487 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:12.512 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:41:12.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:12.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:41:12.616 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:41:12.645 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:41:12.646 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:12.666 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:41:12.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:12.669 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:41:12.770 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:41:12.803 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:12.824 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:41:12.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:12.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:41:12.835 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:41:12.854 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:41:12.876 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:41:12.906 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:41:12.929 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:41:12.946 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:41:12.952 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 8 16 19073 19057 148.866 6.40625 0.00766773 0.00670997 2024-02-09T17:41:12.971 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:12.982 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:41:13.003 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:41:13.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:13.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:41:13.030 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:41:13.061 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:41:13.093 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:41:13.108 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:41:13.127 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:41:13.152 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:13.171 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:41:13.179 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:41:13.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:13.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:41:13.210 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:41:13.216 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:41:13.216 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:41:13.216 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:41:13.216 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:41:13.216 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:41:13.216 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:41:13.216 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:41:13.216 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:41:13.216 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:41:13.216 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:41:13.216 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:41:13.216 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:41:13.287 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:41:13.325 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:13.349 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:41:13.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:13.352 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:41:13.453 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:41:13.495 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:13.522 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:41:13.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:13.527 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:41:13.627 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:41:13.668 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:13 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:41:13.668 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:13 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:41:13.668 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:13 smithi179 ceph-mon[43975]: osdmap e476: 12 total, 12 up, 11 in 2024-02-09T17:41:13.668 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:13 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/657224437' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:41:13.668 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:13 smithi179 ceph-mon[43975]: pgmap v1210: 37 pgs: 1 remapped+peering, 1 peering, 35 active+clean; 997 MiB data, 6.9 GiB used, 977 GiB / 983 GiB avail; 176 MiB/s wr, 2.82k op/s 2024-02-09T17:41:13.671 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:13.698 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:41:13.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:13.704 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:41:13.774 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:41:13.805 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:41:13.843 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:13 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:41:13.843 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:13 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:41:13.843 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:13 smithi099 ceph-mon[39420]: osdmap e476: 12 total, 12 up, 11 in 2024-02-09T17:41:13.843 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:13 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/657224437' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:41:13.843 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:13 smithi099 ceph-mon[39420]: pgmap v1210: 37 pgs: 1 remapped+peering, 1 peering, 35 active+clean; 997 MiB data, 6.9 GiB used, 977 GiB / 983 GiB avail; 176 MiB/s wr, 2.82k op/s 2024-02-09T17:41:13.850 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:13.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:13 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:41:13.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:13 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:41:13.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:13 smithi162 ceph-mon[44304]: osdmap e476: 12 total, 12 up, 11 in 2024-02-09T17:41:13.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:13 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/657224437' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:41:13.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:13 smithi162 ceph-mon[44304]: pgmap v1210: 37 pgs: 1 remapped+peering, 1 peering, 35 active+clean; 997 MiB data, 6.9 GiB used, 977 GiB / 983 GiB avail; 176 MiB/s wr, 2.82k op/s 2024-02-09T17:41:13.952 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 9 16 21544 21528 149.484 154.438 0.0078521 0.00668324 2024-02-09T17:41:14.213 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-09T17:41:14.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:14 smithi179 ceph-mon[43975]: osdmap e477: 12 total, 12 up, 11 in 2024-02-09T17:41:14.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:14 smithi162 ceph-mon[44304]: osdmap e477: 12 total, 12 up, 11 in 2024-02-09T17:41:14.952 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 10 16 23356 23340 145.86 113.25 0.00304467 0.00677989 2024-02-09T17:41:14.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:14 smithi099 ceph-mon[39420]: osdmap e477: 12 total, 12 up, 11 in 2024-02-09T17:41:15.088 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:41:15.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:15.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:41:15.197 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:41:15.242 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:15.270 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:41:15.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:15.273 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:41:15.375 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:41:15.418 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:15.452 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:41:15.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:15.456 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:41:15.557 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:41:15.604 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:15.641 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:41:15.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:15.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:41:15.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:15 smithi099 ceph-mon[39420]: osdmap e478: 12 total, 12 up, 11 in 2024-02-09T17:41:15.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:15 smithi099 ceph-mon[39420]: pgmap v1213: 37 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 activating+undersized+remapped, 1 remapped+peering, 1 peering, 31 active+clean; 1.2 GiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 139 MiB/s wr, 2.22k op/s; 3249/59667 objects misplaced (5.445%); 5.9 MiB/s, 93 objects/s recovering 2024-02-09T17:41:15.748 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:41:15.786 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:15 smithi179 ceph-mon[43975]: osdmap e478: 12 total, 12 up, 11 in 2024-02-09T17:41:15.786 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:15 smithi179 ceph-mon[43975]: pgmap v1213: 37 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 activating+undersized+remapped, 1 remapped+peering, 1 peering, 31 active+clean; 1.2 GiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 139 MiB/s wr, 2.22k op/s; 3249/59667 objects misplaced (5.445%); 5.9 MiB/s, 93 objects/s recovering 2024-02-09T17:41:15.788 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:15.822 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:41:15.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:15.826 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:41:15.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:15 smithi162 ceph-mon[44304]: osdmap e478: 12 total, 12 up, 11 in 2024-02-09T17:41:15.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:15 smithi162 ceph-mon[44304]: pgmap v1213: 37 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 activating+undersized+remapped, 1 remapped+peering, 1 peering, 31 active+clean; 1.2 GiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 139 MiB/s wr, 2.22k op/s; 3249/59667 objects misplaced (5.445%); 5.9 MiB/s, 93 objects/s recovering 2024-02-09T17:41:15.928 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:41:15.952 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 11 15 25607 25592 145.393 140.75 0.00906309 0.0068725 2024-02-09T17:41:15.970 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:15.997 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:41:16.004 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:16.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:41:16.105 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:41:16.132 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:16.151 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:41:16.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:16.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:41:16.254 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:41:16.293 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:16.355 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:41:16.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:16.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:41:16.459 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:41:16.499 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:16.536 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.0 to 1 (10000) 2024-02-09T17:41:16.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:16 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1279662103' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-09T17:41:16.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:16 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-09T17:41:16.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:16 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1279662103' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-09T17:41:16.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:16 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-09T17:41:16.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:16 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1279662103' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-09T17:41:16.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:16 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-09T17:41:16.954 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 12 16 27153 27137 141.324 96.5625 0.00536422 0.0070704 2024-02-09T17:41:17.055 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:41:17.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:17.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:41:17.166 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:41:17.209 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:17.236 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:41:17.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:17.241 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:41:17.342 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:41:17.387 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:17.416 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:41:17.421 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:17.421 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:41:17.516 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:41:17.523 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:41:17.539 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-02-09T17:41:17.539 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph pg dump --format=json 2024-02-09T17:41:17.555 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:17.575 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:41:17.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:17.578 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:41:17.679 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:41:17.705 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:17.742 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:41:17.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:17 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-09T17:41:17.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:17 smithi099 ceph-mon[39420]: osdmap e479: 12 total, 12 up, 11 in 2024-02-09T17:41:17.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:17 smithi099 ceph-mon[39420]: pgmap v1215: 37 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 activating+undersized+remapped, 1 remapped+peering, 32 active+clean; 1.4 GiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 94 MiB/s wr, 1.51k op/s; 3249/67290 objects misplaced (4.828%); 5.5 MiB/s, 87 objects/s recovering 2024-02-09T17:41:17.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:17.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:41:17.849 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:41:17.891 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:17 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-09T17:41:17.892 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:17 smithi162 ceph-mon[44304]: osdmap e479: 12 total, 12 up, 11 in 2024-02-09T17:41:17.892 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:17 smithi162 ceph-mon[44304]: pgmap v1215: 37 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 activating+undersized+remapped, 1 remapped+peering, 32 active+clean; 1.4 GiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 94 MiB/s wr, 1.51k op/s; 3249/67290 objects misplaced (4.828%); 5.5 MiB/s, 87 objects/s recovering 2024-02-09T17:41:17.894 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:17.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:17 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-09T17:41:17.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:17 smithi179 ceph-mon[43975]: osdmap e479: 12 total, 12 up, 11 in 2024-02-09T17:41:17.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:17 smithi179 ceph-mon[43975]: pgmap v1215: 37 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 activating+undersized+remapped, 1 remapped+peering, 32 active+clean; 1.4 GiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 94 MiB/s wr, 1.51k op/s; 3249/67290 objects misplaced (4.828%); 5.5 MiB/s, 87 objects/s recovering 2024-02-09T17:41:17.922 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:41:17.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:17.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:41:17.952 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 13 16 29044 29028 139.543 118.188 0.00968401 0.00716166 2024-02-09T17:41:18.028 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:41:18.069 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:18.132 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:41:18.280 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:41:18.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:18.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:41:18.384 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:41:18.422 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:18.450 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:41:18.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:18.455 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:41:18.556 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:41:18.593 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:18.629 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:41:18.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:18.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:41:18.735 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:41:18.773 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:18.815 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:41:18.815 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [], 2024-02-09T17:41:18.815 INFO:teuthology.orchestra.run.smithi099.stdout: "complaint_time": 30, 2024-02-09T17:41:18.815 INFO:teuthology.orchestra.run.smithi099.stdout: "num_blocked_ops": 0 2024-02-09T17:41:18.815 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:41:18.953 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 14 16 31349 31333 139.865 144.062 0.00348497 0.00714651 2024-02-09T17:41:19.652 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:41:19.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:19.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:41:19.756 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:41:19.790 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:19.857 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:19 smithi099 ceph-mon[39420]: pgmap v1216: 37 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 34 active+clean; 1.5 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 91 MiB/s wr, 1.46k op/s; 3249/74133 objects misplaced (4.383%); 3.9 MiB/s, 62 objects/s recovering 2024-02-09T17:41:19.857 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:19 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:41:19.857 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:19 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:41:19.953 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 15 16 33230 33214 138.377 117.562 0.00188454 0.00719687 2024-02-09T17:41:20.123 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:41:20.125 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:41:20.159 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-09T17:41:20.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:19 smithi179 ceph-mon[43975]: pgmap v1216: 37 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 34 active+clean; 1.5 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 91 MiB/s wr, 1.46k op/s; 3249/74133 objects misplaced (4.383%); 3.9 MiB/s, 62 objects/s recovering 2024-02-09T17:41:20.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:19 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:41:20.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:19 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:41:20.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:19 smithi162 ceph-mon[44304]: pgmap v1216: 37 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 34 active+clean; 1.5 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 91 MiB/s wr, 1.46k op/s; 3249/74133 objects misplaced (4.383%); 3.9 MiB/s, 62 objects/s recovering 2024-02-09T17:41:20.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:19 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:41:20.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:19 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:41:20.750 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:41:20.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:20.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:41:20.854 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:41:20.885 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:20.953 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 16 16 33373 33357 130.287 8.9375 0.00595571 0.00766399 2024-02-09T17:41:21.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:20 smithi179 ceph-mon[43975]: osdmap e480: 12 total, 12 up, 11 in 2024-02-09T17:41:21.183 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:20 smithi162 ceph-mon[44304]: osdmap e480: 12 total, 12 up, 11 in 2024-02-09T17:41:21.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:20 smithi099 ceph-mon[39420]: osdmap e480: 12 total, 12 up, 11 in 2024-02-09T17:41:21.953 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 17 16 34389 34373 126.358 63.5 0.00204363 0.00785268 2024-02-09T17:41:22.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:21 smithi179 ceph-mon[43975]: pgmap v1218: 37 pgs: 1 creating+peering, 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 34 active+clean; 1.8 GiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 96 MiB/s wr, 1.53k op/s; 358/88569 objects misplaced (0.404%); 29 MiB/s, 460 objects/s recovering 2024-02-09T17:41:22.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:21 smithi179 ceph-mon[43975]: osdmap e481: 12 total, 12 up, 11 in 2024-02-09T17:41:22.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:21 smithi179 ceph-mon[43975]: osdmap e482: 12 total, 12 up, 11 in 2024-02-09T17:41:22.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:21 smithi162 ceph-mon[44304]: pgmap v1218: 37 pgs: 1 creating+peering, 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 34 active+clean; 1.8 GiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 96 MiB/s wr, 1.53k op/s; 358/88569 objects misplaced (0.404%); 29 MiB/s, 460 objects/s recovering 2024-02-09T17:41:22.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:21 smithi162 ceph-mon[44304]: osdmap e481: 12 total, 12 up, 11 in 2024-02-09T17:41:22.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:21 smithi162 ceph-mon[44304]: osdmap e482: 12 total, 12 up, 11 in 2024-02-09T17:41:22.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:21 smithi099 ceph-mon[39420]: pgmap v1218: 37 pgs: 1 creating+peering, 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 34 active+clean; 1.8 GiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 96 MiB/s wr, 1.53k op/s; 358/88569 objects misplaced (0.404%); 29 MiB/s, 460 objects/s recovering 2024-02-09T17:41:22.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:21 smithi099 ceph-mon[39420]: osdmap e481: 12 total, 12 up, 11 in 2024-02-09T17:41:22.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:21 smithi099 ceph-mon[39420]: osdmap e482: 12 total, 12 up, 11 in 2024-02-09T17:41:22.953 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 18 15 36642 36627 127.164 140.875 0.00488908 0.00785974 2024-02-09T17:41:23.953 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 19 16 38895 38879 127.878 140.75 0.00397119 0.00781585 2024-02-09T17:41:24.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:23 smithi179 ceph-mon[43975]: pgmap v1221: 37 pgs: 1 creating+peering, 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 34 active+clean; 2.0 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 103 MiB/s wr, 1.65k op/s; 358/97062 objects misplaced (0.369%); 30 MiB/s, 481 objects/s recovering 2024-02-09T17:41:24.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:23 smithi162 ceph-mon[44304]: pgmap v1221: 37 pgs: 1 creating+peering, 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 34 active+clean; 2.0 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 103 MiB/s wr, 1.65k op/s; 358/97062 objects misplaced (0.369%); 30 MiB/s, 481 objects/s recovering 2024-02-09T17:41:24.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:23 smithi099 ceph-mon[39420]: pgmap v1221: 37 pgs: 1 creating+peering, 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 34 active+clean; 2.0 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 103 MiB/s wr, 1.65k op/s; 358/97062 objects misplaced (0.369%); 30 MiB/s, 481 objects/s recovering 2024-02-09T17:41:24.953 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:2024-02-09T17:41:24.953071+0000 min lat: 0.00110706 max lat: 2.34848 avg lat: 0.00777387 2024-02-09T17:41:24.953 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-09T17:41:24.953 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 20 16 41160 41144 128.562 141.562 0.0100776 0.00777387 2024-02-09T17:41:25.167 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is set 2024-02-09T17:41:25.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:24 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3467327572' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:41:25.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:24 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3467327572' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:41:25.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:24 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3467327572' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:41:25.953 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 21 16 43318 43302 128.862 134.875 0.00483806 0.00774166 2024-02-09T17:41:26.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:25 smithi179 ceph-mon[43975]: pgmap v1222: 37 pgs: 1 creating+peering, 1 active+recovering+remapped, 35 active+clean; 2.1 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 109 MiB/s wr, 1.74k op/s; 271/105480 objects misplaced (0.257%); 31 MiB/s, 496 objects/s recovering 2024-02-09T17:41:26.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:25 smithi179 ceph-mon[43975]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:41:26.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:25 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3467327572' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:41:26.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:25 smithi179 ceph-mon[43975]: osdmap e483: 12 total, 12 up, 11 in 2024-02-09T17:41:26.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:25 smithi162 ceph-mon[44304]: pgmap v1222: 37 pgs: 1 creating+peering, 1 active+recovering+remapped, 35 active+clean; 2.1 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 109 MiB/s wr, 1.74k op/s; 271/105480 objects misplaced (0.257%); 31 MiB/s, 496 objects/s recovering 2024-02-09T17:41:26.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:25 smithi162 ceph-mon[44304]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:41:26.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:25 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3467327572' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:41:26.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:25 smithi162 ceph-mon[44304]: osdmap e483: 12 total, 12 up, 11 in 2024-02-09T17:41:26.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:25 smithi099 ceph-mon[39420]: pgmap v1222: 37 pgs: 1 creating+peering, 1 active+recovering+remapped, 35 active+clean; 2.1 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 109 MiB/s wr, 1.74k op/s; 271/105480 objects misplaced (0.257%); 31 MiB/s, 496 objects/s recovering 2024-02-09T17:41:26.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:25 smithi099 ceph-mon[39420]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:41:26.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:25 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3467327572' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:41:26.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:25 smithi099 ceph-mon[39420]: osdmap e483: 12 total, 12 up, 11 in 2024-02-09T17:41:26.953 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 22 16 45301 45285 128.638 123.938 0.00898677 0.00776956 2024-02-09T17:41:27.347 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:41:27.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:27.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:41:27.455 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:41:27.497 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:27.523 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:41:27.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:27.527 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:41:27.628 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:41:27.657 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:27.675 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:41:27.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:27.678 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:41:27.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:27 smithi099 ceph-mon[39420]: osdmap e484: 12 total, 12 up, 11 in 2024-02-09T17:41:27.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:27 smithi099 ceph-mon[39420]: pgmap v1225: 37 pgs: 37 active+clean; 2.5 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 113 MiB/s wr, 1.80k op/s; 3.8 MiB/s, 60 objects/s recovering 2024-02-09T17:41:27.779 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:41:27.821 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:27 smithi162 ceph-mon[44304]: osdmap e484: 12 total, 12 up, 11 in 2024-02-09T17:41:27.821 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:27 smithi162 ceph-mon[44304]: pgmap v1225: 37 pgs: 37 active+clean; 2.5 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 113 MiB/s wr, 1.80k op/s; 3.8 MiB/s, 60 objects/s recovering 2024-02-09T17:41:27.827 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:27.857 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:41:27.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:27.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:41:27.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:27 smithi179 ceph-mon[43975]: osdmap e484: 12 total, 12 up, 11 in 2024-02-09T17:41:27.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:27 smithi179 ceph-mon[43975]: pgmap v1225: 37 pgs: 37 active+clean; 2.5 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 113 MiB/s wr, 1.80k op/s; 3.8 MiB/s, 60 objects/s recovering 2024-02-09T17:41:27.953 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 23 16 47031 47015 127.746 108.125 0.00472542 0.0078245 2024-02-09T17:41:27.962 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:41:28.002 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:28.031 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:41:28.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:28.035 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:41:28.136 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:41:28.176 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:28.207 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:41:28.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:28.212 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:41:28.313 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:41:28.353 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:28.378 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:41:28.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:28.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:41:28.486 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:41:28.523 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:28.953 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 24 16 49027 49011 127.62 124.75 0.00239235 0.00781543 2024-02-09T17:41:29.314 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:41:29.880 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:41:29.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:29.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:41:29.953 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 25 16 50845 50829 127.06 113.625 0.0116975 0.00786517 2024-02-09T17:41:29.962 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset noscrub 2024-02-09T17:41:29.985 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:41:30.031 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:30.062 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:41:30.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:30.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:41:30.168 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:41:30.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:29 smithi179 ceph-mon[43975]: pgmap v1226: 37 pgs: 37 active+clean; 2.6 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 106 MiB/s wr, 1.69k op/s; 3.1 MiB/s, 49 objects/s recovering 2024-02-09T17:41:30.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:29 smithi162 ceph-mon[44304]: pgmap v1226: 37 pgs: 37 active+clean; 2.6 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 106 MiB/s wr, 1.69k op/s; 3.1 MiB/s, 49 objects/s recovering 2024-02-09T17:41:30.204 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:29 smithi099 ceph-mon[39420]: pgmap v1226: 37 pgs: 37 active+clean; 2.6 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 106 MiB/s wr, 1.69k op/s; 3.1 MiB/s, 49 objects/s recovering 2024-02-09T17:41:30.207 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:30.773 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:41:30.774 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-02-09T17:41:30.870 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:30 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:41:30.870 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:30 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:41:30.870 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:30 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:41:30.870 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:30 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:41:30.870 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:30 smithi099 ceph-mon[39420]: osdmap e485: 12 total, 12 up, 11 in 2024-02-09T17:41:30.870 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:30 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1603719396' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:41:30.954 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 26 16 52634 52618 126.47 111.812 0.00610486 0.00790292 2024-02-09T17:41:31.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:30 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:41:31.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:30 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:41:31.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:30 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:41:31.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:30 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:41:31.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:30 smithi179 ceph-mon[43975]: osdmap e485: 12 total, 12 up, 11 in 2024-02-09T17:41:31.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:30 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1603719396' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:41:31.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:30 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:41:31.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:30 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:41:31.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:30 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:41:31.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:30 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:41:31.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:30 smithi162 ceph-mon[44304]: osdmap e485: 12 total, 12 up, 11 in 2024-02-09T17:41:31.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:30 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1603719396' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:41:31.189 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:41:31.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:31.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:41:31.294 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:41:31.327 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:31.840 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:41:31.840 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:41:31.840 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:41:31.841 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:41:31.841 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:41:31.841 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:41:31.841 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:41:31.841 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:41:31.841 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:41:31.841 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:41:31.841 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:41:31.841 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:41:31.841 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:41:31.842 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:41:31.842 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:41:31.842 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:41:31.842 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:41:31.842 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:41:31.842 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:41:31.842 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:41:31.842 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:41:31.842 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:41:31.842 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:41:31.842 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:41:31.954 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 27 16 54570 54554 126.267 121 0.00798643 0.00791543 2024-02-09T17:41:32.082 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:41:32.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:32.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:41:32.090 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:31 smithi099 ceph-mon[39420]: from='client.25532 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:41:32.090 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:31 smithi099 ceph-mon[39420]: pgmap v1228: 37 pgs: 37 active+clean; 2.9 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 131 MiB/s wr, 2.09k op/s; 2.8 MiB/s, 45 objects/s recovering 2024-02-09T17:41:32.090 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:31 smithi099 ceph-mon[39420]: osdmap e486: 12 total, 12 up, 11 in 2024-02-09T17:41:32.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:31 smithi179 ceph-mon[43975]: from='client.25532 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:41:32.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:31 smithi179 ceph-mon[43975]: pgmap v1228: 37 pgs: 37 active+clean; 2.9 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 131 MiB/s wr, 2.09k op/s; 2.8 MiB/s, 45 objects/s recovering 2024-02-09T17:41:32.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:31 smithi179 ceph-mon[43975]: osdmap e486: 12 total, 12 up, 11 in 2024-02-09T17:41:32.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:31 smithi162 ceph-mon[44304]: from='client.25532 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:41:32.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:31 smithi162 ceph-mon[44304]: pgmap v1228: 37 pgs: 37 active+clean; 2.9 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 131 MiB/s wr, 2.09k op/s; 2.8 MiB/s, 45 objects/s recovering 2024-02-09T17:41:32.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:31 smithi162 ceph-mon[44304]: osdmap e486: 12 total, 12 up, 11 in 2024-02-09T17:41:32.189 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:41:32.232 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:32.336 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:41:32.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:32.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:41:32.443 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:41:32.490 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:32.514 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:41:32.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:32.518 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:41:32.619 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:41:32.658 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:32.954 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 28 16 56429 56413 125.906 116.188 0.00773969 0.00793769 2024-02-09T17:41:33.065 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":1227,"stamp":"2024-02-09T17:41:29.858884+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2972385824,"num_objects":45348,"num_object_clones":0,"num_object_copies":136044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45346,"num_write_kb":2902144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3629,"num_bytes_recovered":238499424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":48265,"ondisk_log_size":48265,"up":111,"acting":111,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":113,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":13163356,"kb_used_data":8738452,"kb_used_omap":228,"kb_used_meta":4424539,"kb_avail":1018033316,"statfs":{"total":1055945392128,"available":1042466115584,"internally_reserved":0,"allocated":8948174848,"data_stored":8942476728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":233532,"internal_metadata":4530728900},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,3,11,2,3,2,0,0,3,13],"upper_bound":1024},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":851574784,"num_objects":12994,"num_object_clones":0,"num_object_copies":38982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-358,"num_objects_unfound":0,"num_objects_dirty":12994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12994,"num_write_kb":831616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":358,"num_bytes_recovered":23461888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.058801"},"pg_stats":[{"pgid":"1.14","version":"63'139","reported_seq":925,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-02-09T17:41:26.501332+0000","last_change":"2024-02-09T17:41:08.077049+0000","last_active":"2024-02-09T17:41:26.501332+0000","last_peered":"2024-02-09T17:41:26.501332+0000","last_clean":"2024-02-09T17:41:26.501332+0000","last_became_active":"2024-02-09T17:39:27.455644+0000","last_became_peered":"2024-02-09T17:39:27.455644+0000","last_unstale":"2024-02-09T17:41:26.501332+0000","last_undegraded":"2024-02-09T17:41:26.501332+0000","last_fullsized":"2024-02-09T17:41:26.501332+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:08.076894+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:38:26.968892+0000","last_clean_scrub_stamp":"2024-02-09T17:41:08.076894+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:16.380706+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"484'2851","reported_seq":2899,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-02-09T17:41:27.852926+0000","last_change":"2024-02-09T17:40:48.780201+0000","last_active":"2024-02-09T17:41:27.852926+0000","last_peered":"2024-02-09T17:41:27.852926+0000","last_clean":"2024-02-09T17:41:27.852926+0000","last_became_active":"2024-02-09T17:40:48.779925+0000","last_became_peered":"2024-02-09T17:40:48.779925+0000","last_unstale":"2024-02-09T17:41:27.852926+0000","last_undegraded":"2024-02-09T17:41:27.852926+0000","last_fullsized":"2024-02-09T17:41:27.852926+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:40:47.763277+0000","objects_scrubbed":0,"log_size":2851,"log_dups_size":0,"ondisk_log_size":2851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:10.602974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186843136,"num_objects":2851,"num_object_clones":0,"num_object_copies":8553,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2851,"num_write_kb":182464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,8],"acting":[10,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"63'139","reported_seq":1278,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-02-09T17:41:26.501148+0000","last_change":"2024-02-09T17:41:14.521470+0000","last_active":"2024-02-09T17:41:26.501148+0000","last_peered":"2024-02-09T17:41:26.501148+0000","last_clean":"2024-02-09T17:41:26.501148+0000","last_became_active":"2024-02-09T17:41:14.521079+0000","last_became_peered":"2024-02-09T17:41:14.521079+0000","last_unstale":"2024-02-09T17:41:26.501148+0000","last_undegraded":"2024-02-09T17:41:26.501148+0000","last_fullsized":"2024-02-09T17:41:26.501148+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:10.115197+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:41:10.115197+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:18.769273+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"484'3038","reported_seq":3086,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-02-09T17:41:28.991162+0000","last_change":"2024-02-09T17:40:48.783610+0000","last_active":"2024-02-09T17:41:28.991162+0000","last_peered":"2024-02-09T17:41:28.991162+0000","last_clean":"2024-02-09T17:41:28.991162+0000","last_became_active":"2024-02-09T17:40:48.783218+0000","last_became_peered":"2024-02-09T17:40:48.783218+0000","last_unstale":"2024-02-09T17:41:28.991162+0000","last_undegraded":"2024-02-09T17:41:28.991162+0000","last_fullsized":"2024-02-09T17:41:28.991162+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:40:47.763277+0000","objects_scrubbed":0,"log_size":3038,"log_dups_size":0,"ondisk_log_size":3038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:02.568964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199098368,"num_objects":3038,"num_object_clones":0,"num_object_copies":9114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3038,"num_write_kb":194432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"63'139","reported_seq":1007,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-02-09T17:41:26.501265+0000","last_change":"2024-02-09T17:40:22.485379+0000","last_active":"2024-02-09T17:41:26.501265+0000","last_peered":"2024-02-09T17:41:26.501265+0000","last_clean":"2024-02-09T17:41:26.501265+0000","last_became_active":"2024-02-09T17:40:22.484952+0000","last_became_peered":"2024-02-09T17:40:22.484952+0000","last_unstale":"2024-02-09T17:41:26.501265+0000","last_undegraded":"2024-02-09T17:41:26.501265+0000","last_fullsized":"2024-02-09T17:41:26.501265+0000","mapping_epoch":454,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":455,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:39:46.807673+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:39:46.807673+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:41:27.268056+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"484'2997","reported_seq":3045,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-02-09T17:41:29.014948+0000","last_change":"2024-02-09T17:40:48.781806+0000","last_active":"2024-02-09T17:41:29.014948+0000","last_peered":"2024-02-09T17:41:29.014948+0000","last_clean":"2024-02-09T17:41:29.014948+0000","last_became_active":"2024-02-09T17:40:48.781472+0000","last_became_peered":"2024-02-09T17:40:48.781472+0000","last_unstale":"2024-02-09T17:41:29.014948+0000","last_undegraded":"2024-02-09T17:41:29.014948+0000","last_fullsized":"2024-02-09T17:41:29.014948+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:40:47.763277+0000","objects_scrubbed":0,"log_size":2997,"log_dups_size":0,"ondisk_log_size":2997,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:13.563605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196411392,"num_objects":2997,"num_object_clones":0,"num_object_copies":8991,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2997,"num_write_kb":191808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"63'139","reported_seq":457,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-09T17:41:24.895033+0000","last_change":"2024-02-09T17:40:57.059982+0000","last_active":"2024-02-09T17:41:24.895033+0000","last_peered":"2024-02-09T17:41:24.895033+0000","last_clean":"2024-02-09T17:41:24.895033+0000","last_became_active":"2024-02-09T17:40:13.501711+0000","last_became_peered":"2024-02-09T17:40:13.501711+0000","last_unstale":"2024-02-09T17:41:24.895033+0000","last_undegraded":"2024-02-09T17:41:24.895033+0000","last_fullsized":"2024-02-09T17:41:24.895033+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:40:57.059847+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:38:27.045816+0000","last_clean_scrub_stamp":"2024-02-09T17:40:57.059847+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:04.876510+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"484'2997","reported_seq":3043,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-02-09T17:41:28.845689+0000","last_change":"2024-02-09T17:40:48.780916+0000","last_active":"2024-02-09T17:41:28.845689+0000","last_peered":"2024-02-09T17:41:28.845689+0000","last_clean":"2024-02-09T17:41:28.845689+0000","last_became_active":"2024-02-09T17:40:48.780579+0000","last_became_peered":"2024-02-09T17:40:48.780579+0000","last_unstale":"2024-02-09T17:41:28.845689+0000","last_undegraded":"2024-02-09T17:41:28.845689+0000","last_fullsized":"2024-02-09T17:41:28.845689+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:40:47.763277+0000","objects_scrubbed":0,"log_size":2997,"log_dups_size":0,"ondisk_log_size":2997,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:03.096001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196411392,"num_objects":2997,"num_object_clones":0,"num_object_copies":8991,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2997,"num_write_kb":191808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"63'139","reported_seq":776,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-02-09T17:41:26.501027+0000","last_change":"2024-02-09T17:40:46.690262+0000","last_active":"2024-02-09T17:41:26.501027+0000","last_peered":"2024-02-09T17:41:26.501027+0000","last_clean":"2024-02-09T17:41:26.501027+0000","last_became_active":"2024-02-09T17:34:01.342607+0000","last_became_peered":"2024-02-09T17:34:01.342607+0000","last_unstale":"2024-02-09T17:41:26.501027+0000","last_undegraded":"2024-02-09T17:41:26.501027+0000","last_fullsized":"2024-02-09T17:41:26.501027+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:40:46.690100+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:40:46.690100+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:03.472698+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.2","version":"483'2767","reported_seq":2813,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-09T17:41:25.825599+0000","last_change":"2024-02-09T17:40:48.781838+0000","last_active":"2024-02-09T17:41:25.825599+0000","last_peered":"2024-02-09T17:41:25.825599+0000","last_clean":"2024-02-09T17:41:25.825599+0000","last_became_active":"2024-02-09T17:40:48.781532+0000","last_became_peered":"2024-02-09T17:40:48.781532+0000","last_unstale":"2024-02-09T17:41:25.825599+0000","last_undegraded":"2024-02-09T17:41:25.825599+0000","last_fullsized":"2024-02-09T17:41:25.825599+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:40:47.763277+0000","objects_scrubbed":0,"log_size":2767,"log_dups_size":0,"ondisk_log_size":2767,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:10.569691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181338112,"num_objects":2767,"num_object_clones":0,"num_object_copies":8301,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2767,"num_write_kb":177088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"63'139","reported_seq":767,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-02-09T17:41:26.502241+0000","last_change":"2024-02-09T17:40:01.416463+0000","last_active":"2024-02-09T17:41:26.502241+0000","last_peered":"2024-02-09T17:41:26.502241+0000","last_clean":"2024-02-09T17:41:26.502241+0000","last_became_active":"2024-02-09T17:33:00.284702+0000","last_became_peered":"2024-02-09T17:33:00.284702+0000","last_unstale":"2024-02-09T17:41:26.502241+0000","last_undegraded":"2024-02-09T17:41:26.502241+0000","last_fullsized":"2024-02-09T17:41:26.502241+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:40:01.416303+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:40:01.416303+0000","last_clean_scrub_stamp":"2024-02-09T17:40:01.416303+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.3","version":"483'2657","reported_seq":2703,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-09T17:41:25.872077+0000","last_change":"2024-02-09T17:40:48.783695+0000","last_active":"2024-02-09T17:41:25.872077+0000","last_peered":"2024-02-09T17:41:25.872077+0000","last_clean":"2024-02-09T17:41:25.872077+0000","last_became_active":"2024-02-09T17:40:48.783369+0000","last_became_peered":"2024-02-09T17:40:48.783369+0000","last_unstale":"2024-02-09T17:41:25.872077+0000","last_undegraded":"2024-02-09T17:41:25.872077+0000","last_fullsized":"2024-02-09T17:41:25.872077+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:40:47.763277+0000","objects_scrubbed":0,"log_size":2657,"log_dups_size":0,"ondisk_log_size":2657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:41:48.151059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174129152,"num_objects":2657,"num_object_clones":0,"num_object_copies":7971,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2657,"num_write_kb":170048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"63'139","reported_seq":436,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-09T17:41:24.894366+0000","last_change":"2024-02-09T17:40:13.498076+0000","last_active":"2024-02-09T17:41:24.894366+0000","last_peered":"2024-02-09T17:41:24.894366+0000","last_clean":"2024-02-09T17:41:24.894366+0000","last_became_active":"2024-02-09T17:40:13.497710+0000","last_became_peered":"2024-02-09T17:40:13.497710+0000","last_unstale":"2024-02-09T17:41:24.894366+0000","last_undegraded":"2024-02-09T17:41:24.894366+0000","last_fullsized":"2024-02-09T17:41:24.894366+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:40:02.483295+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:40:02.483295+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"484'2967","reported_seq":3015,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-02-09T17:41:27.854667+0000","last_change":"2024-02-09T17:40:48.780689+0000","last_active":"2024-02-09T17:41:27.854667+0000","last_peered":"2024-02-09T17:41:27.854667+0000","last_clean":"2024-02-09T17:41:27.854667+0000","last_became_active":"2024-02-09T17:40:48.780290+0000","last_became_peered":"2024-02-09T17:40:48.780290+0000","last_unstale":"2024-02-09T17:41:27.854667+0000","last_undegraded":"2024-02-09T17:41:27.854667+0000","last_fullsized":"2024-02-09T17:41:27.854667+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:40:47.763277+0000","objects_scrubbed":0,"log_size":2967,"log_dups_size":0,"ondisk_log_size":2967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:00.604584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194445312,"num_objects":2967,"num_object_clones":0,"num_object_copies":8901,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2967,"num_write_kb":189888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"63'139","reported_seq":856,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-09T17:41:24.896797+0000","last_change":"2024-02-09T17:40:44.545620+0000","last_active":"2024-02-09T17:41:24.896797+0000","last_peered":"2024-02-09T17:41:24.896797+0000","last_clean":"2024-02-09T17:41:24.896797+0000","last_became_active":"2024-02-09T17:33:00.284924+0000","last_became_peered":"2024-02-09T17:33:00.284924+0000","last_unstale":"2024-02-09T17:41:24.896797+0000","last_undegraded":"2024-02-09T17:41:24.896797+0000","last_fullsized":"2024-02-09T17:41:24.896797+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:40:44.545494+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:40:44.545494+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:08.678934+0000","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.1","version":"483'2696","reported_seq":2742,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-09T17:41:25.682130+0000","last_change":"2024-02-09T17:40:49.661020+0000","last_active":"2024-02-09T17:41:25.682130+0000","last_peered":"2024-02-09T17:41:25.682130+0000","last_clean":"2024-02-09T17:41:25.682130+0000","last_became_active":"2024-02-09T17:40:49.660617+0000","last_became_peered":"2024-02-09T17:40:49.660617+0000","last_unstale":"2024-02-09T17:41:25.682130+0000","last_undegraded":"2024-02-09T17:41:25.682130+0000","last_fullsized":"2024-02-09T17:41:25.682130+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:40:47.763277+0000","objects_scrubbed":0,"log_size":2696,"log_dups_size":0,"ondisk_log_size":2696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:13.075794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176685056,"num_objects":2696,"num_object_clones":0,"num_object_copies":8088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2696,"num_write_kb":172544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"63'139","reported_seq":811,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-02-09T17:41:26.501216+0000","last_change":"2024-02-09T17:41:14.521510+0000","last_active":"2024-02-09T17:41:26.501216+0000","last_peered":"2024-02-09T17:41:26.501216+0000","last_clean":"2024-02-09T17:41:26.501216+0000","last_became_active":"2024-02-09T17:41:14.521098+0000","last_became_peered":"2024-02-09T17:41:14.521098+0000","last_unstale":"2024-02-09T17:41:26.501216+0000","last_undegraded":"2024-02-09T17:41:26.501216+0000","last_fullsized":"2024-02-09T17:41:26.501216+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:40:56.752599+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:40:56.752599+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:14.187085+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11],"acting":[10,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.e","version":"484'2749","reported_seq":2797,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-02-09T17:41:26.527843+0000","last_change":"2024-02-09T17:40:48.779687+0000","last_active":"2024-02-09T17:41:26.527843+0000","last_peered":"2024-02-09T17:41:26.527843+0000","last_clean":"2024-02-09T17:41:26.527843+0000","last_became_active":"2024-02-09T17:40:48.779366+0000","last_became_peered":"2024-02-09T17:40:48.779366+0000","last_unstale":"2024-02-09T17:41:26.527843+0000","last_undegraded":"2024-02-09T17:41:26.527843+0000","last_fullsized":"2024-02-09T17:41:26.527843+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:40:47.763277+0000","objects_scrubbed":0,"log_size":2749,"log_dups_size":0,"ondisk_log_size":2749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:41:52.402214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180158464,"num_objects":2749,"num_object_clones":0,"num_object_copies":8247,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2749,"num_write_kb":175936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"63'139","reported_seq":760,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-02-09T17:41:26.503704+0000","last_change":"2024-02-09T17:39:42.324945+0000","last_active":"2024-02-09T17:41:26.503704+0000","last_peered":"2024-02-09T17:41:26.503704+0000","last_clean":"2024-02-09T17:41:26.503704+0000","last_became_active":"2024-02-09T17:32:35.418200+0000","last_became_peered":"2024-02-09T17:32:35.418200+0000","last_unstale":"2024-02-09T17:41:26.503704+0000","last_undegraded":"2024-02-09T17:41:26.503704+0000","last_fullsized":"2024-02-09T17:41:26.503704+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:39:42.324851+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:39:42.324851+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:41:27.597316+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.f","version":"484'3051","reported_seq":5473,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-02-09T17:41:29.078972+0000","last_change":"2024-02-09T17:41:20.878405+0000","last_active":"2024-02-09T17:41:29.078972+0000","last_peered":"2024-02-09T17:41:29.078972+0000","last_clean":"2024-02-09T17:41:29.078972+0000","last_became_active":"2024-02-09T17:41:20.876180+0000","last_became_peered":"2024-02-09T17:41:20.876180+0000","last_unstale":"2024-02-09T17:41:29.078972+0000","last_undegraded":"2024-02-09T17:41:29.078972+0000","last_fullsized":"2024-02-09T17:41:29.078972+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:40:47.763277+0000","objects_scrubbed":0,"log_size":3051,"log_dups_size":0,"ondisk_log_size":3051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:06.473578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199950336,"num_objects":3051,"num_object_clones":0,"num_object_copies":9153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3051,"num_write_kb":195264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1182,"num_bytes_recovered":77463552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,11],"acting":[5,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"63'139","reported_seq":600,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-09T17:41:24.896122+0000","last_change":"2024-02-09T17:41:14.522753+0000","last_active":"2024-02-09T17:41:24.896122+0000","last_peered":"2024-02-09T17:41:24.896122+0000","last_clean":"2024-02-09T17:41:24.896122+0000","last_became_active":"2024-02-09T17:41:14.522410+0000","last_became_peered":"2024-02-09T17:41:14.522410+0000","last_unstale":"2024-02-09T17:41:24.896122+0000","last_undegraded":"2024-02-09T17:41:24.896122+0000","last_fullsized":"2024-02-09T17:41:24.896122+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:40:42.968753+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:40:42.968753+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:41:45.066546+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"483'2726","reported_seq":2772,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-09T17:41:25.640147+0000","last_change":"2024-02-09T17:40:48.781690+0000","last_active":"2024-02-09T17:41:25.640147+0000","last_peered":"2024-02-09T17:41:25.640147+0000","last_clean":"2024-02-09T17:41:25.640147+0000","last_became_active":"2024-02-09T17:40:48.781384+0000","last_became_peered":"2024-02-09T17:40:48.781384+0000","last_unstale":"2024-02-09T17:41:25.640147+0000","last_undegraded":"2024-02-09T17:41:25.640147+0000","last_fullsized":"2024-02-09T17:41:25.640147+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:40:47.763277+0000","objects_scrubbed":0,"log_size":2726,"log_dups_size":0,"ondisk_log_size":2726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:41:59.692496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178651136,"num_objects":2726,"num_object_clones":0,"num_object_copies":8178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2726,"num_write_kb":174464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"63'139","reported_seq":817,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-02-09T17:41:26.498173+0000","last_change":"2024-02-09T17:40:46.233294+0000","last_active":"2024-02-09T17:41:26.498173+0000","last_peered":"2024-02-09T17:41:26.498173+0000","last_clean":"2024-02-09T17:41:26.498173+0000","last_became_active":"2024-02-09T17:39:37.460615+0000","last_became_peered":"2024-02-09T17:39:37.460615+0000","last_unstale":"2024-02-09T17:41:26.498173+0000","last_undegraded":"2024-02-09T17:41:26.498173+0000","last_fullsized":"2024-02-09T17:41:26.498173+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:40:46.233161+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:40:46.233161+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:41:50.354159+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"483'2689","reported_seq":5235,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-09T17:41:25.301884+0000","last_change":"2024-02-09T17:41:21.502193+0000","last_active":"2024-02-09T17:41:25.301884+0000","last_peered":"2024-02-09T17:41:25.301884+0000","last_clean":"2024-02-09T17:41:25.301884+0000","last_became_active":"2024-02-09T17:41:21.499187+0000","last_became_peered":"2024-02-09T17:41:21.499187+0000","last_unstale":"2024-02-09T17:41:25.301884+0000","last_undegraded":"2024-02-09T17:41:25.301884+0000","last_fullsized":"2024-02-09T17:41:25.301884+0000","mapping_epoch":481,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:40:47.763277+0000","objects_scrubbed":0,"log_size":2689,"log_dups_size":0,"ondisk_log_size":2689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:41:51.463528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176226304,"num_objects":2689,"num_object_clones":0,"num_object_copies":8067,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2689,"num_write_kb":172096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1248,"num_bytes_recovered":81788928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"63'139","reported_seq":1263,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-02-09T17:41:26.501526+0000","last_change":"2024-02-09T17:40:57.958820+0000","last_active":"2024-02-09T17:41:26.501526+0000","last_peered":"2024-02-09T17:41:26.501526+0000","last_clean":"2024-02-09T17:41:26.501526+0000","last_became_active":"2024-02-09T17:39:17.456739+0000","last_became_peered":"2024-02-09T17:39:17.456739+0000","last_unstale":"2024-02-09T17:41:26.501526+0000","last_undegraded":"2024-02-09T17:41:26.501526+0000","last_fullsized":"2024-02-09T17:41:26.501526+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:40:57.958641+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:30.193194+0000","last_clean_scrub_stamp":"2024-02-09T17:40:57.958641+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:11.460211+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"484'2799","reported_seq":2846,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-02-09T17:41:26.895254+0000","last_change":"2024-02-09T17:40:48.781919+0000","last_active":"2024-02-09T17:41:26.895254+0000","last_peered":"2024-02-09T17:41:26.895254+0000","last_clean":"2024-02-09T17:41:26.895254+0000","last_became_active":"2024-02-09T17:40:48.781591+0000","last_became_peered":"2024-02-09T17:40:48.781591+0000","last_unstale":"2024-02-09T17:41:26.895254+0000","last_undegraded":"2024-02-09T17:41:26.895254+0000","last_fullsized":"2024-02-09T17:41:26.895254+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:40:47.763277+0000","objects_scrubbed":0,"log_size":2799,"log_dups_size":0,"ondisk_log_size":2799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:06.277557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183435264,"num_objects":2799,"num_object_clones":0,"num_object_copies":8397,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2799,"num_write_kb":179136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,4],"acting":[8,10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.e","version":"63'139","reported_seq":734,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-09T17:41:24.897455+0000","last_change":"2024-02-09T17:41:06.857849+0000","last_active":"2024-02-09T17:41:24.897455+0000","last_peered":"2024-02-09T17:41:24.897455+0000","last_clean":"2024-02-09T17:41:24.897455+0000","last_became_active":"2024-02-09T17:33:05.309824+0000","last_became_peered":"2024-02-09T17:33:05.309824+0000","last_unstale":"2024-02-09T17:41:24.897455+0000","last_undegraded":"2024-02-09T17:41:24.897455+0000","last_fullsized":"2024-02-09T17:41:24.897455+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:06.857716+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:41:06.857716+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:29.895394+0000","scrub_duration":190,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"484'2789","reported_seq":2837,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-02-09T17:41:26.542786+0000","last_change":"2024-02-09T17:40:48.783627+0000","last_active":"2024-02-09T17:41:26.542786+0000","last_peered":"2024-02-09T17:41:26.542786+0000","last_clean":"2024-02-09T17:41:26.542786+0000","last_became_active":"2024-02-09T17:40:48.783235+0000","last_became_peered":"2024-02-09T17:40:48.783235+0000","last_unstale":"2024-02-09T17:41:26.542786+0000","last_undegraded":"2024-02-09T17:41:26.542786+0000","last_fullsized":"2024-02-09T17:41:26.542786+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:40:47.763277+0000","objects_scrubbed":0,"log_size":2789,"log_dups_size":0,"ondisk_log_size":2789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:14.909299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182779904,"num_objects":2789,"num_object_clones":0,"num_object_copies":8367,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2789,"num_write_kb":178496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.f","version":"63'139","reported_seq":793,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-09T17:41:24.894453+0000","last_change":"2024-02-09T17:41:02.126077+0000","last_active":"2024-02-09T17:41:24.894453+0000","last_peered":"2024-02-09T17:41:24.894453+0000","last_clean":"2024-02-09T17:41:24.894453+0000","last_became_active":"2024-02-09T17:33:05.316100+0000","last_became_peered":"2024-02-09T17:33:05.316100+0000","last_unstale":"2024-02-09T17:41:24.894453+0000","last_undegraded":"2024-02-09T17:41:24.894453+0000","last_fullsized":"2024-02-09T17:41:24.894453+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:02.125974+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:36:22.364130+0000","last_clean_scrub_stamp":"2024-02-09T17:41:02.125974+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:09.528604+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"483'2647","reported_seq":5093,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-09T17:41:25.295911+0000","last_change":"2024-02-09T17:41:20.879764+0000","last_active":"2024-02-09T17:41:25.295911+0000","last_peered":"2024-02-09T17:41:25.295911+0000","last_clean":"2024-02-09T17:41:25.295911+0000","last_became_active":"2024-02-09T17:41:20.876434+0000","last_became_peered":"2024-02-09T17:41:20.876434+0000","last_unstale":"2024-02-09T17:41:25.295911+0000","last_undegraded":"2024-02-09T17:41:25.295911+0000","last_fullsized":"2024-02-09T17:41:25.295911+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:40:47.763277+0000","objects_scrubbed":0,"log_size":2647,"log_dups_size":0,"ondisk_log_size":2647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:01.223195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173473792,"num_objects":2647,"num_object_clones":0,"num_object_copies":7941,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2647,"num_write_kb":169408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1195,"num_bytes_recovered":78315520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"63'139","reported_seq":915,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-02-09T17:41:26.498611+0000","last_change":"2024-02-09T17:40:03.690279+0000","last_active":"2024-02-09T17:41:26.498611+0000","last_peered":"2024-02-09T17:41:26.498611+0000","last_clean":"2024-02-09T17:41:26.498611+0000","last_became_active":"2024-02-09T17:39:37.471954+0000","last_became_peered":"2024-02-09T17:39:37.471954+0000","last_unstale":"2024-02-09T17:41:26.498611+0000","last_undegraded":"2024-02-09T17:41:26.498611+0000","last_fullsized":"2024-02-09T17:41:26.498611+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:40:03.690165+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:23.912504+0000","last_clean_scrub_stamp":"2024-02-09T17:40:03.690165+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":118,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"484'2926","reported_seq":2972,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-02-09T17:41:27.155937+0000","last_change":"2024-02-09T17:40:49.436637+0000","last_active":"2024-02-09T17:41:27.155937+0000","last_peered":"2024-02-09T17:41:27.155937+0000","last_clean":"2024-02-09T17:41:27.155937+0000","last_became_active":"2024-02-09T17:40:49.436285+0000","last_became_peered":"2024-02-09T17:40:49.436285+0000","last_unstale":"2024-02-09T17:41:27.155937+0000","last_undegraded":"2024-02-09T17:41:27.155937+0000","last_fullsized":"2024-02-09T17:41:27.155937+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:40:47.763277+0000","objects_scrubbed":0,"log_size":2926,"log_dups_size":0,"ondisk_log_size":2926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:41:50.232026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191758336,"num_objects":2926,"num_object_clones":0,"num_object_copies":8778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2926,"num_write_kb":187264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"63'139","reported_seq":328,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-02-09T17:41:26.501151+0000","last_change":"2024-02-09T17:40:44.903454+0000","last_active":"2024-02-09T17:41:26.501151+0000","last_peered":"2024-02-09T17:41:26.501151+0000","last_clean":"2024-02-09T17:41:26.501151+0000","last_became_active":"2024-02-09T17:40:22.485257+0000","last_became_peered":"2024-02-09T17:40:22.485257+0000","last_unstale":"2024-02-09T17:41:26.501151+0000","last_undegraded":"2024-02-09T17:41:26.501151+0000","last_fullsized":"2024-02-09T17:41:26.501151+0000","mapping_epoch":454,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":455,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:40:44.903308+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:40:44.903308+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:13.814742+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"63'139","reported_seq":744,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-02-09T17:41:26.498168+0000","last_change":"2024-02-09T17:41:09.323335+0000","last_active":"2024-02-09T17:41:26.498168+0000","last_peered":"2024-02-09T17:41:26.498168+0000","last_clean":"2024-02-09T17:41:26.498168+0000","last_became_active":"2024-02-09T17:33:05.310104+0000","last_became_peered":"2024-02-09T17:33:05.310104+0000","last_unstale":"2024-02-09T17:41:26.498168+0000","last_undegraded":"2024-02-09T17:41:26.498168+0000","last_fullsized":"2024-02-09T17:41:26.498168+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:09.323236+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:41:09.323236+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:13.566799+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"63'139","reported_seq":751,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-02-09T17:41:26.509592+0000","last_change":"2024-02-09T17:41:06.851766+0000","last_active":"2024-02-09T17:41:26.509592+0000","last_peered":"2024-02-09T17:41:26.509592+0000","last_clean":"2024-02-09T17:41:26.509592+0000","last_became_active":"2024-02-09T17:34:01.353171+0000","last_became_peered":"2024-02-09T17:34:01.353171+0000","last_unstale":"2024-02-09T17:41:26.509592+0000","last_undegraded":"2024-02-09T17:41:26.509592+0000","last_fullsized":"2024-02-09T17:41:26.509592+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:06.851643+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:41:06.851643+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:34.858536+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.12","version":"63'139","reported_seq":1263,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-02-09T17:41:26.498286+0000","last_change":"2024-02-09T17:40:44.208044+0000","last_active":"2024-02-09T17:41:26.498286+0000","last_peered":"2024-02-09T17:41:26.498286+0000","last_clean":"2024-02-09T17:41:26.498286+0000","last_became_active":"2024-02-09T17:40:13.499071+0000","last_became_peered":"2024-02-09T17:40:13.499071+0000","last_unstale":"2024-02-09T17:41:26.498286+0000","last_undegraded":"2024-02-09T17:41:26.498286+0000","last_fullsized":"2024-02-09T17:41:26.498286+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:40:44.207923+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:40:44.207923+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:13.625252+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"63'139","reported_seq":460,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-02-09T17:41:24.895173+0000","last_change":"2024-02-09T17:41:14.521999+0000","last_active":"2024-02-09T17:41:24.895173+0000","last_peered":"2024-02-09T17:41:24.895173+0000","last_clean":"2024-02-09T17:41:24.895173+0000","last_became_active":"2024-02-09T17:41:14.521672+0000","last_became_peered":"2024-02-09T17:41:14.521672+0000","last_unstale":"2024-02-09T17:41:24.895173+0000","last_undegraded":"2024-02-09T17:41:24.895173+0000","last_fullsized":"2024-02-09T17:41:24.895173+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:03.023657+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:41:03.023657+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:28.356260+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":2971795456,"num_objects":45346,"num_object_clones":0,"num_object_copies":136038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45346,"num_write_kb":2902144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3625,"num_bytes_recovered":237568000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9096855552,"data_stored":9096855552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45346,"ondisk_log_size":45346,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":931424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2919,"ondisk_log_size":2919,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056869,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1169856,"kb_used_data":673592,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":92575296,"statfs":{"total":95995035648,"available":94797103104,"internally_reserved":0,"allocated":689758208,"data_stored":689244060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":508164548},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,3,13],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:40:24 2024","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.6180000000000001,"15min":1.3999999999999999},"min":{"1min":0.252,"5min":0.216,"15min":0.216},"max":{"1min":5.8470000000000004,"5min":9.5709999999999997,"15min":14.913},"last":0.379},{"interface":"front","average":{"1min":1.6679999999999999,"5min":1.341,"15min":1.4179999999999999},"min":{"1min":0.28799999999999998,"5min":0.27000000000000002,"15min":0.23000000000000001},"max":{"1min":9.8219999999999992,"5min":9.8219999999999992,"15min":14.869},"last":0.436}]},{"osd":1,"last update":"Fri Feb 9 17:40:24 2024","interfaces":[{"interface":"back","average":{"1min":1.819,"5min":1.7909999999999999,"15min":1.752},"min":{"1min":0.27600000000000002,"5min":0.24299999999999999,"15min":0.22700000000000001},"max":{"1min":9.4789999999999992,"5min":9.7710000000000008,"15min":32.957999999999998},"last":0.45300000000000001},{"interface":"front","average":{"1min":1.9850000000000001,"5min":1.948,"15min":1.881},"min":{"1min":0.26500000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":9.5009999999999994,"5min":9.718,"15min":32.880000000000003},"last":0.46400000000000002}]},{"osd":2,"last update":"Fri Feb 9 17:31:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.415,"15min":1.6859999999999999},"min":{"1min":0.378,"5min":0.247,"15min":0.247},"max":{"1min":4.4189999999999996,"5min":9.4719999999999995,"15min":9.4719999999999995},"last":0.35399999999999998},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.5149999999999999,"15min":1.4399999999999999},"min":{"1min":0.378,"5min":0.252,"15min":0.252},"max":{"1min":4.8579999999999997,"5min":7.96,"15min":7.96},"last":0.60399999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:40:24 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":1.333,"15min":1.7170000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.247},"max":{"1min":1.5549999999999999,"5min":9.5299999999999994,"15min":21.315999999999999},"last":0.42599999999999999},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.873,"15min":1.8180000000000001},"min":{"1min":0.33000000000000002,"5min":0.29799999999999999,"15min":0.26400000000000001},"max":{"1min":4.9939999999999998,"5min":12.706,"15min":18.087},"last":0.40300000000000002}]},{"osd":4,"last update":"Fri Feb 9 17:40:24 2024","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":1.302,"15min":1.47},"min":{"1min":0.26800000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":3.8999999999999999,"5min":21.030999999999999,"15min":21.030999999999999},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":1.371,"15min":1.3109999999999999},"min":{"1min":0.317,"5min":0.309,"15min":0.27700000000000002},"max":{"1min":1.6859999999999999,"5min":20.876999999999999,"15min":20.876999999999999},"last":0.65700000000000003}]},{"osd":5,"last update":"Fri Feb 9 17:40:24 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":1.0469999999999999,"15min":1.3360000000000001},"min":{"1min":0.32900000000000001,"5min":0.316,"15min":0.28899999999999998},"max":{"1min":2.2050000000000001,"5min":6.4630000000000001,"15min":13.946999999999999},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.71399999999999997,"5min":1.139,"15min":1.4379999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.30199999999999999},"max":{"1min":3.8439999999999999,"5min":7.1349999999999998,"15min":13.414},"last":0.51200000000000001}]},{"osd":6,"last update":"Fri Feb 9 17:40:24 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":1.262,"15min":1.4219999999999999},"min":{"1min":0.39200000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":4.2030000000000003,"5min":9.4779999999999998,"15min":12.798999999999999},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.71999999999999997,"5min":1.1839999999999999,"15min":1.4179999999999999},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32300000000000001},"max":{"1min":3.5299999999999998,"5min":9.2420000000000009,"15min":10.782999999999999},"last":0.53800000000000003}]},{"osd":7,"last update":"Fri Feb 9 17:40:24 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":1.3520000000000001,"15min":1.633},"min":{"1min":0.40899999999999997,"5min":0.38,"15min":0.312},"max":{"1min":3.8719999999999999,"5min":20.489999999999998,"15min":20.489999999999998},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.64100000000000001,"5min":1.2629999999999999,"15min":1.579},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.34200000000000003},"max":{"1min":1.5309999999999999,"5min":20.712,"15min":20.712},"last":0.52800000000000002}]},{"osd":8,"last update":"Fri Feb 9 17:40:52 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":1.0680000000000001,"15min":1.7569999999999999},"min":{"1min":0.34300000000000003,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":3.5619999999999998,"5min":9.2680000000000007,"15min":13.199999999999999},"last":0.316},{"interface":"front","average":{"1min":0.79400000000000004,"5min":1.0760000000000001,"15min":1.4950000000000001},"min":{"1min":0.443,"5min":0.34300000000000003,"15min":0.318},"max":{"1min":3.6539999999999999,"5min":9.2940000000000005,"15min":9.2940000000000005},"last":0.41199999999999998}]},{"osd":9,"last update":"Fri Feb 9 17:41:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":1.79,"15min":1.4259999999999999},"min":{"1min":0.36799999999999999,"5min":0.36499999999999999,"15min":0.29399999999999998},"max":{"1min":1.5629999999999999,"5min":62.518000000000001,"15min":62.518000000000001},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.84299999999999997,"5min":1.8480000000000001,"15min":1.369},"min":{"1min":0.46300000000000002,"5min":0.379,"15min":0.34100000000000003},"max":{"1min":3.8159999999999998,"5min":63.241999999999997,"15min":63.241999999999997},"last":0.47899999999999998}]},{"osd":10,"last update":"Fri Feb 9 17:40:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":1.1619999999999999,"15min":1.2130000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.33900000000000002},"max":{"1min":3.677,"5min":9.3109999999999999,"15min":9.6199999999999992},"last":0.499},{"interface":"front","average":{"1min":0.96999999999999997,"5min":1.2370000000000001,"15min":1.2549999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.33700000000000002},"max":{"1min":5.7720000000000002,"5min":9.593,"15min":12.856},"last":0.49099999999999999}]}]},{"osd":10,"up_from":84,"seq":360777253099,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1412236,"kb_used_data":1120068,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":92332916,"statfs":{"total":95995035648,"available":94548905984,"internally_reserved":0,"allocated":1146949632,"data_stored":1146430936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":299170240},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:41:14 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":1.54,"15min":1.7050000000000001},"min":{"1min":0.25700000000000001,"5min":0.222,"15min":0.222},"max":{"1min":1.6559999999999999,"5min":9.343,"15min":9.343},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.58899999999999997,"5min":1.365,"15min":1.177},"min":{"1min":0.27600000000000002,"5min":0.26500000000000001,"15min":0.253},"max":{"1min":1.484,"5min":6.3399999999999999,"15min":6.3399999999999999},"last":0.29199999999999998}]},{"osd":1,"last update":"Fri Feb 9 17:39:30 2024","interfaces":[{"interface":"back","average":{"1min":1.804,"5min":1.804,"15min":1.804},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":7.673,"5min":7.673,"15min":7.673},"last":0.73899999999999999},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":3.9649999999999999,"5min":3.9649999999999999,"15min":3.9649999999999999},"last":0.26100000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:40:57 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":1.8600000000000001,"15min":1.554},"min":{"1min":0.28999999999999998,"5min":0.28499999999999998,"15min":0.222},"max":{"1min":2.177,"5min":11.384,"15min":13.351000000000001},"last":0.68000000000000005},{"interface":"front","average":{"1min":1.129,"5min":1.474,"15min":1.444},"min":{"1min":0.42799999999999999,"5min":0.27200000000000002,"15min":0.26100000000000001},"max":{"1min":9.0850000000000009,"5min":11.465,"15min":11.465},"last":0.32600000000000001}]},{"osd":3,"last update":"Fri Feb 9 17:40:39 2024","interfaces":[{"interface":"back","average":{"1min":2.048,"5min":2.048,"15min":2.048},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":9.1649999999999991,"5min":9.1649999999999991,"15min":9.1649999999999991},"last":0.35099999999999998},{"interface":"front","average":{"1min":2.1059999999999999,"5min":2.1059999999999999,"15min":2.1059999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":8.9670000000000005,"5min":8.9670000000000005,"15min":8.9670000000000005},"last":0.78700000000000003}]},{"osd":4,"last update":"Fri Feb 9 17:41:05 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.90000000000000002,"15min":1.071},"min":{"1min":0.314,"5min":0.29199999999999998,"15min":0.26100000000000001},"max":{"1min":1.0609999999999999,"5min":5.8840000000000003,"15min":7.1360000000000001},"last":0.375},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.86899999999999999,"15min":1.0720000000000001},"min":{"1min":0.41999999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.038,"5min":4.6139999999999999,"15min":6.8369999999999997},"last":0.40100000000000002}]},{"osd":5,"last update":"Fri Feb 9 17:40:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":1.012,"15min":1.2789999999999999},"min":{"1min":0.33600000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.92100000000000004,"5min":7.5179999999999998,"15min":9.2349999999999994},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.98299999999999998,"15min":1.22},"min":{"1min":0.44900000000000001,"5min":0.33500000000000002,"15min":0.32200000000000001},"max":{"1min":4.181,"5min":6.8940000000000001,"15min":8.6159999999999997},"last":1.2609999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:40:57 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":1.244,"15min":1.268},"min":{"1min":0.32400000000000001,"5min":0.308,"15min":0.29099999999999998},"max":{"1min":0.85499999999999998,"5min":8.6869999999999994,"15min":8.6869999999999994},"last":1.3180000000000001},{"interface":"front","average":{"1min":0.86799999999999999,"5min":1.0329999999999999,"15min":1.391},"min":{"1min":0.41099999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":2.339,"5min":4.9720000000000004,"15min":9.4350000000000005},"last":1.0760000000000001}]},{"osd":7,"last update":"Fri Feb 9 17:39:52 2024","interfaces":[{"interface":"back","average":{"1min":1.6000000000000001,"5min":1.367,"15min":1.415},"min":{"1min":0.42099999999999999,"5min":0.34200000000000003,"15min":0.312},"max":{"1min":4.0430000000000001,"5min":6.7590000000000003,"15min":17.754000000000001},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.498,"5min":1.4359999999999999,"15min":1.5189999999999999},"min":{"1min":0.47399999999999998,"5min":0.38300000000000001,"15min":0.316},"max":{"1min":5.8170000000000002,"5min":11.308999999999999,"15min":11.308999999999999},"last":0.78800000000000003}]},{"osd":8,"last update":"Fri Feb 9 17:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":1.095,"15min":1.1639999999999999},"min":{"1min":0.35499999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":3.3679999999999999,"5min":7.4359999999999999,"15min":7.4359999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.96899999999999997,"5min":1.0309999999999999,"15min":0.97399999999999998},"min":{"1min":0.39400000000000002,"5min":0.37,"15min":0.32900000000000001},"max":{"1min":4.0389999999999997,"5min":7.7060000000000004,"15min":7.7060000000000004},"last":1.143}]},{"osd":9,"last update":"Fri Feb 9 17:40:57 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":1.2130000000000001,"15min":1.103},"min":{"1min":0.377,"5min":0.36399999999999999,"15min":0.314},"max":{"1min":1.093,"5min":7.4850000000000003,"15min":8.7240000000000002},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.216,"15min":1.2230000000000001},"min":{"1min":0.441,"5min":0.40300000000000002,"15min":0.29799999999999999},"max":{"1min":1.371,"5min":7.891,"15min":10.1},"last":0.83399999999999996}]},{"osd":11,"last update":"Fri Feb 9 17:41:21 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.89600000000000002,"15min":1.113},"min":{"1min":0.45800000000000002,"5min":0.40600000000000003,"15min":0.36399999999999999},"max":{"1min":0.98399999999999999,"5min":4.2560000000000002,"15min":8.6430000000000007},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.95199999999999996,"15min":1.1299999999999999},"min":{"1min":0.60199999999999998,"5min":0.45700000000000002,"15min":0.33400000000000002},"max":{"1min":1.0269999999999999,"5min":4.2800000000000002,"15min":9.298},"last":0.45200000000000001}]}]},{"osd":9,"up_from":77,"seq":330712482032,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":943316,"kb_used_data":528964,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":92801836,"statfs":{"total":95995035648,"available":95029080064,"internally_reserved":0,"allocated":541659136,"data_stored":541140440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":424278464},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:28:03 2024","interfaces":[{"interface":"back","average":{"1min":1.9510000000000001,"5min":1.454,"15min":0.85799999999999998},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":8.4499999999999993,"5min":17.495999999999999,"15min":17.495999999999999},"last":0.441},{"interface":"front","average":{"1min":1.905,"5min":1.399,"15min":0.83499999999999996},"min":{"1min":0.32200000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":7.5990000000000002,"5min":15.401999999999999,"15min":15.401999999999999},"last":1.9570000000000001}]},{"osd":1,"last update":"Fri Feb 9 17:40:35 2024","interfaces":[{"interface":"back","average":{"1min":2.0779999999999998,"5min":2.802,"15min":1.931},"min":{"1min":0.29599999999999999,"5min":0.23499999999999999,"15min":0.23100000000000001},"max":{"1min":6.9409999999999998,"5min":89.515000000000001,"15min":89.515000000000001},"last":0.36099999999999999},{"interface":"front","average":{"1min":2.2749999999999999,"5min":2.9540000000000002,"15min":1.851},"min":{"1min":0.249,"5min":0.249,"15min":0.222},"max":{"1min":11.598000000000001,"5min":89.388000000000005,"15min":89.388000000000005},"last":0.52900000000000003}]},{"osd":2,"last update":"Fri Feb 9 17:40:39 2024","interfaces":[{"interface":"back","average":{"1min":1.742,"5min":1.766,"15min":1.514},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":5.6630000000000003,"5min":14.121,"15min":14.121},"last":0.63400000000000001},{"interface":"front","average":{"1min":1.6910000000000001,"5min":1.52,"15min":1.4870000000000001},"min":{"1min":0.34100000000000003,"5min":0.26100000000000001,"15min":0.25600000000000001},"max":{"1min":5.7030000000000003,"5min":8.8040000000000003,"15min":13.69},"last":0.73899999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:40:35 2024","interfaces":[{"interface":"back","average":{"1min":2.5550000000000002,"5min":2.1259999999999999,"15min":1.6990000000000001},"min":{"1min":0.38100000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":9.6959999999999997,"5min":15.414999999999999,"15min":40.18},"last":1.0669999999999999},{"interface":"front","average":{"1min":2.2810000000000001,"5min":2.1890000000000001,"15min":1.915},"min":{"1min":0.433,"5min":0.317,"15min":0.27900000000000003},"max":{"1min":9.7140000000000004,"5min":15.372,"15min":40.158000000000001},"last":1.087}]},{"osd":4,"last update":"Fri Feb 9 17:40:35 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":1.2470000000000001,"15min":1.4399999999999999},"min":{"1min":0.309,"5min":0.29599999999999999,"15min":0.28399999999999997},"max":{"1min":2.0499999999999998,"5min":27.381,"15min":27.381},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":1.3939999999999999,"15min":1.371},"min":{"1min":0.313,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.651,"5min":27.355,"15min":27.355},"last":0.57399999999999995}]},{"osd":5,"last update":"Fri Feb 9 17:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":1.1870000000000001,"15min":1.3939999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.613,"5min":8.6110000000000007,"15min":10.689},"last":1.4139999999999999},{"interface":"front","average":{"1min":0.96999999999999997,"5min":1.131,"15min":1.179},"min":{"1min":0.42399999999999999,"5min":0.32200000000000001,"15min":0.311},"max":{"1min":5.5090000000000003,"5min":13.978999999999999,"15min":13.978999999999999},"last":1.9750000000000001}]},{"osd":6,"last update":"Fri Feb 9 17:40:35 2024","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.3169999999999999,"15min":1.452},"min":{"1min":0.40899999999999997,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":5.4640000000000004,"5min":14.069000000000001,"15min":14.069000000000001},"last":0.41499999999999998},{"interface":"front","average":{"1min":1.421,"5min":1.3340000000000001,"15min":1.446},"min":{"1min":0.47899999999999998,"5min":0.37,"15min":0.35099999999999998},"max":{"1min":5.6079999999999997,"5min":14.097,"15min":14.097},"last":0.45500000000000002}]},{"osd":7,"last update":"Fri Feb 9 17:40:35 2024","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.306,"15min":1.395},"min":{"1min":0.40200000000000002,"5min":0.34499999999999997,"15min":0.34300000000000003},"max":{"1min":4.8159999999999998,"5min":7.4800000000000004,"15min":8.0939999999999994},"last":0.69799999999999995},{"interface":"front","average":{"1min":1.135,"5min":1.355,"15min":1.4890000000000001},"min":{"1min":0.433,"5min":0.38400000000000001,"15min":0.32200000000000001},"max":{"1min":4.8799999999999999,"5min":7.5039999999999996,"15min":8.0530000000000008},"last":0.73399999999999999}]},{"osd":8,"last update":"Fri Feb 9 17:40:54 2024","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":1.3460000000000001,"15min":1.286},"min":{"1min":0.35799999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":5.1440000000000001,"5min":14.045999999999999,"15min":16.209},"last":0.873},{"interface":"front","average":{"1min":1.085,"5min":1.353,"15min":1.2949999999999999},"min":{"1min":0.41799999999999998,"5min":0.41299999999999998,"15min":0.34799999999999998},"max":{"1min":5.548,"5min":5.548,"15min":16.263000000000002},"last":1.9330000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:41:05 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":1.2010000000000001,"15min":1.198},"min":{"1min":0.373,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":2.254,"5min":7.4089999999999998,"15min":9.5090000000000003},"last":0.97599999999999998},{"interface":"front","average":{"1min":0.71299999999999997,"5min":1.377,"15min":1.2030000000000001},"min":{"1min":0.437,"5min":0.39500000000000002,"15min":0.38200000000000001},"max":{"1min":1.2210000000000001,"5min":9.7569999999999997,"15min":9.7569999999999997},"last":0.83999999999999997}]},{"osd":11,"last update":"Fri Feb 9 17:41:05 2024","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":1.542,"15min":1.4279999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.34699999999999998},"max":{"1min":5.6349999999999998,"5min":14.02,"15min":14.02},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.76100000000000001,"5min":1.357,"15min":1.2350000000000001},"min":{"1min":0.46800000000000003,"5min":0.39400000000000002,"15min":0.374},"max":{"1min":1.25,"5min":7.29,"15min":10.32},"last":0.61199999999999999}]}]},{"osd":8,"up_from":171,"seq":734439407775,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":727580,"kb_used_data":533644,"kb_used_omap":212,"kb_used_meta":193707,"kb_avail":93017572,"statfs":{"total":95995035648,"available":95249993728,"internally_reserved":0,"allocated":546451456,"data_stored":545929208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217579,"internal_metadata":198356501},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:40:38 2024","interfaces":[{"interface":"back","average":{"1min":1.5529999999999999,"5min":1.1950000000000001,"15min":1.228},"min":{"1min":0.28599999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":5.827,"5min":8.0920000000000005,"15min":8.0920000000000005},"last":0.39000000000000001},{"interface":"front","average":{"1min":2.1110000000000002,"5min":1.48,"15min":1.3620000000000001},"min":{"1min":0.38600000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":7.2949999999999999,"5min":10.256,"15min":10.256},"last":1.3380000000000001}]},{"osd":1,"last update":"Fri Feb 9 17:40:38 2024","interfaces":[{"interface":"back","average":{"1min":1.7609999999999999,"5min":1.6080000000000001,"15min":1.351},"min":{"1min":0.26400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":6.351,"5min":7.867,"15min":7.8719999999999999},"last":1.367},{"interface":"front","average":{"1min":3.1859999999999999,"5min":2.105,"15min":1.8069999999999999},"min":{"1min":0.318,"5min":0.23400000000000001,"15min":0.23300000000000001},"max":{"1min":25.027000000000001,"5min":25.027000000000001,"15min":25.027000000000001},"last":0.313}]},{"osd":2,"last update":"Fri Feb 9 17:40:38 2024","interfaces":[{"interface":"back","average":{"1min":1.512,"5min":1.536,"15min":1.494},"min":{"1min":0.33300000000000002,"5min":0.27500000000000002,"15min":0.255},"max":{"1min":5.891,"5min":8.5649999999999995,"15min":10.423},"last":0.69499999999999995},{"interface":"front","average":{"1min":1.5649999999999999,"5min":1.5840000000000001,"15min":1.552},"min":{"1min":0.36599999999999999,"5min":0.35899999999999999,"15min":0.313},"max":{"1min":5.9470000000000001,"5min":8.5980000000000008,"15min":10.504},"last":0.73299999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:40:38 2024","interfaces":[{"interface":"back","average":{"1min":2.0870000000000002,"5min":1.7809999999999999,"15min":1.885},"min":{"1min":0.39100000000000001,"5min":0.29499999999999998,"15min":0.28699999999999998},"max":{"1min":8.9060000000000006,"5min":15.228999999999999,"15min":24.030999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":2.621,"5min":2.2149999999999999,"15min":2.173},"min":{"1min":0.51100000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":8.8710000000000004,"5min":14.988,"15min":24.318999999999999},"last":0.68200000000000005}]},{"osd":4,"last update":"Fri Feb 9 17:40:38 2024","interfaces":[{"interface":"back","average":{"1min":1.583,"5min":1.2949999999999999,"15min":1.413},"min":{"1min":0.33400000000000002,"5min":0.28199999999999997,"15min":0.27500000000000002},"max":{"1min":7.383,"5min":12.936,"15min":53.060000000000002},"last":0.53700000000000003},{"interface":"front","average":{"1min":1.841,"5min":1.1279999999999999,"15min":1.5720000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":7.46,"5min":7.46,"15min":53.311999999999998},"last":0.75}]},{"osd":6,"last update":"Fri Feb 9 17:40:38 2024","interfaces":[{"interface":"back","average":{"1min":1.774,"5min":1.147,"15min":1.5589999999999999},"min":{"1min":0.39100000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":7.4870000000000001,"5min":7.4870000000000001,"15min":37.356999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.204,"15min":1.2649999999999999},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":5.9930000000000003,"5min":7.702,"15min":37.25},"last":0.87}]},{"osd":7,"last update":"Fri Feb 9 17:40:38 2024","interfaces":[{"interface":"back","average":{"1min":1.859,"5min":1.25,"15min":1.55},"min":{"1min":0.48199999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":8.5120000000000005,"5min":8.5120000000000005,"15min":37.585000000000001},"last":0.60199999999999998},{"interface":"front","average":{"1min":1.9119999999999999,"5min":1.2889999999999999,"15min":1.649},"min":{"1min":0.51500000000000001,"5min":0.38500000000000001,"15min":0.312},"max":{"1min":8.548,"5min":8.548,"15min":37.918999999999997},"last":0.64600000000000002}]},{"osd":9,"last update":"Fri Feb 9 17:40:38 2024","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.98299999999999998,"15min":1.105},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.29699999999999999},"max":{"1min":3.5099999999999998,"5min":5.3330000000000002,"15min":9.4900000000000002},"last":2.2890000000000001},{"interface":"front","average":{"1min":1.474,"5min":1.24,"15min":1.2270000000000001},"min":{"1min":0.40000000000000002,"5min":0.36599999999999999,"15min":0.32000000000000001},"max":{"1min":7.4189999999999996,"5min":7.4189999999999996,"15min":17.434999999999999},"last":0.58499999999999996}]},{"osd":10,"last update":"Fri Feb 9 17:40:38 2024","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":0.92200000000000004,"15min":1.099},"min":{"1min":0.375,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":4.5620000000000003,"5min":4.8170000000000002,"15min":10.65},"last":3.194},{"interface":"front","average":{"1min":1.407,"5min":1.254,"15min":1.1379999999999999},"min":{"1min":0.34799999999999998,"5min":0.318,"15min":0.318},"max":{"1min":6.694,"5min":11.071,"15min":11.071},"last":2.7519999999999998}]},{"osd":11,"last update":"Fri Feb 9 17:40:38 2024","interfaces":[{"interface":"back","average":{"1min":1.542,"5min":1.292,"15min":1.3779999999999999},"min":{"1min":0.38800000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":6.734,"5min":8.5519999999999996,"15min":14.433999999999999},"last":0.77100000000000002},{"interface":"front","average":{"1min":1.5760000000000001,"5min":1.2749999999999999,"15min":1.363},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.38100000000000001},"max":{"1min":7.4359999999999999,"5min":7.4359999999999999,"15min":11.622},"last":0.71399999999999997}]}]},{"osd":7,"up_from":57,"seq":244813136121,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1286804,"kb_used_data":773892,"kb_used_omap":1,"kb_used_meta":512894,"kb_avail":92458348,"statfs":{"total":95995035648,"available":94677348352,"internally_reserved":0,"allocated":792465408,"data_stored":791946712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":525203914},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:41:21 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":1.4139999999999999,"15min":2.7189999999999999},"min":{"1min":0.41299999999999998,"5min":0.252,"15min":0.252},"max":{"1min":3.472,"5min":14.348000000000001,"15min":36.753},"last":1.6459999999999999},{"interface":"front","average":{"1min":0.69099999999999995,"5min":1.4359999999999999,"15min":2.8719999999999999},"min":{"1min":0.31900000000000001,"5min":0.29099999999999998,"15min":0.253},"max":{"1min":1.2290000000000001,"5min":9.0640000000000001,"15min":36.918999999999997},"last":2.2250000000000001}]},{"osd":1,"last update":"Fri Feb 9 17:41:21 2024","interfaces":[{"interface":"back","average":{"1min":1.6060000000000001,"5min":2.242,"15min":2.0790000000000002},"min":{"1min":0.50700000000000001,"5min":0.249,"15min":0.249},"max":{"1min":15.509,"5min":21.704999999999998,"15min":21.704999999999998},"last":1.718},{"interface":"front","average":{"1min":1.5469999999999999,"5min":2.2280000000000002,"15min":1.8300000000000001},"min":{"1min":0.38100000000000001,"5min":0.26600000000000001,"15min":0.26500000000000001},"max":{"1min":15.276,"5min":21.858000000000001,"15min":21.858000000000001},"last":1.7709999999999999}]},{"osd":2,"last update":"Fri Feb 9 17:39:34 2024","interfaces":[{"interface":"back","average":{"1min":1.8520000000000001,"5min":1.7470000000000001,"15min":1.7030000000000001},"min":{"1min":0.315,"5min":0.28499999999999998,"15min":0.26000000000000001},"max":{"1min":6.6470000000000002,"5min":10.622,"15min":11.324},"last":0.76900000000000002},{"interface":"front","average":{"1min":1.8859999999999999,"5min":1.855,"15min":1.7769999999999999},"min":{"1min":0.34699999999999998,"5min":0.32100000000000001,"15min":0.28299999999999997},"max":{"1min":6.6870000000000003,"5min":19.361999999999998,"15min":19.361999999999998},"last":0.82299999999999995}]},{"osd":3,"last update":"Fri Feb 9 17:40:37 2024","interfaces":[{"interface":"back","average":{"1min":1.5009999999999999,"5min":1.704,"15min":1.881},"min":{"1min":0.33600000000000002,"5min":0.307,"15min":0.26900000000000002},"max":{"1min":5.8470000000000004,"5min":7.3899999999999997,"15min":27.719000000000001},"last":2.258},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.6180000000000001,"15min":1.907},"min":{"1min":0.32600000000000001,"5min":0.308,"15min":0.26500000000000001},"max":{"1min":5.8310000000000004,"5min":11.945,"15min":24.457000000000001},"last":1.744}]},{"osd":4,"last update":"Fri Feb 9 17:41:21 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":1.6839999999999999},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":1.448,"5min":1.448,"15min":1.448},"last":0.68700000000000006}]},{"osd":5,"last update":"Fri Feb 9 17:41:21 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.64900000000000002,"15min":0.64500000000000002},"min":{"1min":0.37,"5min":0.249,"15min":0.249},"max":{"1min":1.5209999999999999,"5min":3.0539999999999998,"15min":3.0539999999999998},"last":3.403},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.56299999999999994,"15min":0.52600000000000002},"min":{"1min":0.39700000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":2.3740000000000001}]},{"osd":6,"last update":"Fri Feb 9 17:40:37 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":1.056,"15min":1.5169999999999999},"min":{"1min":0.28100000000000003,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.341,"5min":8.1929999999999996,"15min":25.965},"last":2.1930000000000001},{"interface":"front","average":{"1min":0.66700000000000004,"5min":1.036,"15min":1.2969999999999999},"min":{"1min":0.33100000000000002,"5min":0.29799999999999999,"15min":0.28699999999999998},"max":{"1min":3.0960000000000001,"5min":7.8810000000000002,"15min":9.4090000000000007},"last":0.45900000000000002}]},{"osd":8,"last update":"Fri Feb 9 17:40:37 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":1.0509999999999999,"15min":1.409},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":3.7280000000000002,"5min":6.4420000000000002,"15min":9.0809999999999995},"last":2.7349999999999999},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.97599999999999998,"15min":1.272},"min":{"1min":0.36599999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":4.0810000000000004,"5min":6.0339999999999998,"15min":11.481},"last":2.5760000000000001}]},{"osd":9,"last update":"Fri Feb 9 17:40:27 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":1.1950000000000001,"15min":1.536},"min":{"1min":0.35199999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":3.7010000000000001,"5min":6.4749999999999996,"15min":41.286999999999999},"last":2.7850000000000001},{"interface":"front","average":{"1min":0.89200000000000002,"5min":1.2869999999999999,"15min":1.5840000000000001},"min":{"1min":0.379,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":3.7869999999999999,"5min":7.3579999999999997,"15min":41.234999999999999},"last":2.29}]},{"osd":10,"last update":"Fri Feb 9 17:40:51 2024","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":1.179,"15min":1.419},"min":{"1min":0.40100000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":5.0869999999999997,"5min":12.050000000000001,"15min":12.050000000000001},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.93300000000000005,"5min":1.2050000000000001,"15min":1.502},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.35599999999999998},"max":{"1min":3.7599999999999998,"5min":6.5129999999999999,"15min":9.109},"last":2.319}]},{"osd":11,"last update":"Fri Feb 9 17:41:21 2024","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":1.528,"15min":1.7010000000000001},"min":{"1min":0.46100000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":2.0670000000000002,"5min":21.945,"15min":21.945},"last":3.4319999999999999},{"interface":"front","average":{"1min":0.96899999999999997,"5min":1.5940000000000001,"15min":1.5469999999999999},"min":{"1min":0.502,"5min":0.40699999999999997,"15min":0.39500000000000002},"max":{"1min":2.3050000000000002,"5min":21.731999999999999,"15min":21.731999999999999},"last":3.371}]}]},{"osd":6,"up_from":49,"seq":210453397758,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":878852,"kb_used_data":500600,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":92866300,"statfs":{"total":95995035648,"available":95095091200,"internally_reserved":0,"allocated":512614400,"data_stored":512100252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":387316166},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:41:09 2024","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":1.593,"15min":1.6020000000000001},"min":{"1min":0.33200000000000002,"5min":0.24199999999999999,"15min":0.21299999999999999},"max":{"1min":5.6989999999999998,"5min":9.5739999999999998,"15min":12.544},"last":1.6479999999999999},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.5820000000000001,"15min":1.5309999999999999},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.217},"max":{"1min":5.4349999999999996,"5min":9.2520000000000007,"15min":9.2520000000000007},"last":0.60799999999999998}]},{"osd":1,"last update":"Fri Feb 9 17:41:09 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":1.9379999999999999,"15min":1.859},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.25900000000000001},"max":{"1min":3.6560000000000001,"5min":8.9689999999999994,"15min":17.762},"last":1.522},{"interface":"front","average":{"1min":0.79500000000000004,"5min":1.984,"15min":1.9139999999999999},"min":{"1min":0.41799999999999998,"5min":0.34499999999999997,"15min":0.29899999999999999},"max":{"1min":3.6909999999999998,"5min":9.0090000000000003,"15min":17.734999999999999},"last":1.54}]},{"osd":2,"last update":"Fri Feb 9 17:41:09 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":1.484,"15min":1.6279999999999999},"min":{"1min":0.32500000000000001,"5min":0.29399999999999998,"15min":0.22600000000000001},"max":{"1min":5.601,"5min":9.8019999999999996,"15min":10.682},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.89500000000000002,"5min":1.5329999999999999,"15min":1.6779999999999999},"min":{"1min":0.39100000000000001,"5min":0.35599999999999998,"15min":0.28000000000000003},"max":{"1min":5.6299999999999999,"5min":9.8439999999999994,"15min":11.553000000000001},"last":0.84899999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:41:09 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":1.8700000000000001,"15min":2.0430000000000001},"min":{"1min":0.36499999999999999,"5min":0.26700000000000002,"15min":0.248},"max":{"1min":5.1500000000000004,"5min":11.276,"15min":32.545000000000002},"last":1.5069999999999999},{"interface":"front","average":{"1min":0.67700000000000005,"5min":1.7,"15min":1.919},"min":{"1min":0.34200000000000003,"5min":0.307,"15min":0.26600000000000001},"max":{"1min":1.8759999999999999,"5min":11.122,"15min":32.670999999999999},"last":0.372}]},{"osd":4,"last update":"Fri Feb 9 17:41:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.82999999999999996,"15min":1.3580000000000001},"min":{"1min":0.35599999999999998,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.96899999999999997,"5min":6.1509999999999998,"15min":19.798999999999999},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.90300000000000002,"5min":1.0189999999999999,"15min":1.3400000000000001},"min":{"1min":0.34300000000000003,"5min":0.28499999999999998,"15min":0.26000000000000001},"max":{"1min":5.5140000000000002,"5min":5.9260000000000002,"15min":10.702999999999999},"last":0.35099999999999998}]},{"osd":5,"last update":"Fri Feb 9 17:41:09 2024","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":1.1020000000000001,"15min":1.3560000000000001},"min":{"1min":0.41499999999999998,"5min":0.318,"15min":0.29199999999999998},"max":{"1min":5.6680000000000001,"5min":6.7679999999999998,"15min":10.968999999999999},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.998,"15min":1.383},"min":{"1min":0.317,"5min":0.316,"15min":0.27400000000000002},"max":{"1min":1.9259999999999999,"5min":6.0350000000000001,"15min":13.91},"last":0.41699999999999998}]},{"osd":7,"last update":"Fri Feb 9 17:40:50 2024","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.0720000000000001,"15min":1.4199999999999999},"min":{"1min":0.32100000000000001,"5min":0.29399999999999998,"15min":0.28000000000000003},"max":{"1min":5.1799999999999997,"5min":5.4029999999999996,"15min":14.022},"last":0.47899999999999998},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.2370000000000001,"15min":1.581},"min":{"1min":0.41099999999999998,"5min":0.36099999999999999,"15min":0.309},"max":{"1min":5.5609999999999999,"5min":12.441000000000001,"15min":15.458},"last":0.39100000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:26:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":0.876,"15min":0.79600000000000004},"min":{"1min":0.39900000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":4.0339999999999998,"5min":4.4880000000000004,"15min":4.4880000000000004},"last":4.3840000000000003},{"interface":"front","average":{"1min":1.0469999999999999,"5min":0.88600000000000001,"15min":0.78300000000000003},"min":{"1min":0.375,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":3.6539999999999999,"5min":4.8570000000000002,"15min":4.8570000000000002},"last":4.4210000000000003}]},{"osd":9,"last update":"Fri Feb 9 17:41:16 2024","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.089,"15min":1.2869999999999999},"min":{"1min":0.499,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":5.5789999999999997,"5min":10.779999999999999,"15min":10.803000000000001},"last":0.438},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.218,"15min":1.415},"min":{"1min":0.61299999999999999,"5min":0.40999999999999998,"15min":0.40300000000000002},"max":{"1min":5.5419999999999998,"5min":6.3650000000000002,"15min":8.1630000000000003},"last":0.40999999999999998}]},{"osd":10,"last update":"Fri Feb 9 17:40:50 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":1.0700000000000001,"15min":1.391},"min":{"1min":0.46400000000000002,"5min":0.38500000000000001,"15min":0.35399999999999998},"max":{"1min":1.6970000000000001,"5min":10.677,"15min":10.677},"last":0.52000000000000002},{"interface":"front","average":{"1min":1.0409999999999999,"5min":1.0680000000000001,"15min":1.478},"min":{"1min":0.48599999999999999,"5min":0.38,"15min":0.35899999999999999},"max":{"1min":5.0979999999999999,"5min":5.9509999999999996,"15min":13.836},"last":0.53000000000000003}]},{"osd":11,"last update":"Fri Feb 9 17:41:09 2024","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.2789999999999999,"15min":1.4730000000000001},"min":{"1min":0.63600000000000001,"5min":0.40799999999999997,"15min":0.36099999999999999},"max":{"1min":7.6040000000000001,"5min":7.7450000000000001,"15min":10.273999999999999},"last":0.46200000000000002},{"interface":"front","average":{"1min":1.026,"5min":1.0589999999999999,"15min":1.5229999999999999},"min":{"1min":0.44500000000000001,"5min":0.442,"15min":0.41199999999999998},"max":{"1min":5.2140000000000004,"5min":5.2140000000000004,"15min":8.2739999999999991},"last":0.56699999999999995}]}]},{"osd":1,"up_from":13,"seq":55834575128,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1175172,"kb_used_data":660856,"kb_used_omap":1,"kb_used_meta":514302,"kb_avail":92569980,"statfs":{"total":95995035648,"available":94791659520,"internally_reserved":0,"allocated":676716544,"data_stored":676202396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":526645701},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:40:24 2024","interfaces":[{"interface":"back","average":{"1min":2.2549999999999999,"5min":2.6880000000000002,"15min":1.9419999999999999},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":7.0220000000000002,"5min":19.079000000000001,"15min":19.079000000000001},"last":0.25},{"interface":"front","average":{"1min":2.21,"5min":2.3650000000000002,"15min":1.8640000000000001},"min":{"1min":0.218,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":8.8100000000000005,"5min":23.573,"15min":23.573},"last":0.26800000000000002}]},{"osd":2,"last update":"Fri Feb 9 17:40:47 2024","interfaces":[{"interface":"back","average":{"1min":2.1720000000000002,"5min":2.4590000000000001,"15min":1.998},"min":{"1min":0.30199999999999999,"5min":0.23200000000000001,"15min":0.22600000000000001},"max":{"1min":9.5609999999999999,"5min":26.361999999999998,"15min":26.361999999999998},"last":0.317},{"interface":"front","average":{"1min":1.9099999999999999,"5min":2.6859999999999999,"15min":2.1859999999999999},"min":{"1min":0.27100000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":6.2610000000000001,"5min":26.087,"15min":26.087},"last":0.34100000000000003}]},{"osd":3,"last update":"Fri Feb 9 17:40:36 2024","interfaces":[{"interface":"back","average":{"1min":1.8959999999999999,"5min":2.2690000000000001,"15min":1.6060000000000001},"min":{"1min":0.26200000000000001,"5min":0.245,"15min":0.245},"max":{"1min":6.9400000000000004,"5min":10.657,"15min":10.657},"last":0.30299999999999999},{"interface":"front","average":{"1min":2.2349999999999999,"5min":2.508,"15min":1.734},"min":{"1min":0.36599999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":7.1029999999999998,"5min":14.68,"15min":14.68},"last":1.25}]},{"osd":4,"last update":"Fri Feb 9 17:41:21 2024","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.9259999999999999,"15min":1.966},"min":{"1min":0.34699999999999998,"5min":0.25800000000000001,"15min":0.253},"max":{"1min":4.694,"5min":9.4649999999999999,"15min":11.766999999999999},"last":4.0060000000000002},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.927,"15min":1.9039999999999999},"min":{"1min":0.38800000000000001,"5min":0.309,"15min":0.29999999999999999},"max":{"1min":4.5730000000000004,"5min":10.512,"15min":10.512},"last":3.9500000000000002}]},{"osd":5,"last update":"Fri Feb 9 17:41:05 2024","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.8819999999999999,"15min":1.867},"min":{"1min":0.36599999999999999,"5min":0.28999999999999998,"15min":0.27000000000000002},"max":{"1min":3.9380000000000002,"5min":9.4299999999999997,"15min":13.544},"last":0.29199999999999998},{"interface":"front","average":{"1min":1.149,"5min":2.1789999999999998,"15min":1.75},"min":{"1min":0.442,"5min":0.30499999999999999,"15min":0.26000000000000001},"max":{"1min":4.0800000000000001,"5min":10.531000000000001,"15min":12.117000000000001},"last":0.49199999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:41:10 2024","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":5.0119999999999996,"5min":5.0119999999999996,"15min":5.0119999999999996},"last":0.45600000000000002},{"interface":"front","average":{"1min":1.175,"5min":1.175,"15min":1.175},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":3.831,"5min":3.831,"15min":3.831},"last":0.54700000000000004}]},{"osd":7,"last update":"Fri Feb 9 17:41:11 2024","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":1.157,"15min":1.157},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":4.0439999999999996,"5min":4.0439999999999996,"15min":4.0439999999999996},"last":0.51600000000000001},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":4.9820000000000002,"5min":4.9820000000000002,"15min":4.9820000000000002},"last":0.58699999999999997}]},{"osd":8,"last update":"Fri Feb 9 17:41:21 2024","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":2.121,"5min":2.121,"15min":2.121},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":2.1379999999999999,"5min":2.1379999999999999,"15min":2.1379999999999999},"last":0.622}]},{"osd":9,"last update":"Fri Feb 9 17:40:24 2024","interfaces":[{"interface":"back","average":{"1min":2.2189999999999999,"5min":2.004,"15min":1.8779999999999999},"min":{"1min":0.38300000000000001,"5min":0.33400000000000002,"15min":0.32400000000000001},"max":{"1min":6.8819999999999997,"5min":9.4960000000000004,"15min":9.5839999999999996},"last":1.083},{"interface":"front","average":{"1min":2.2370000000000001,"5min":2.117,"15min":1.8919999999999999},"min":{"1min":0.41399999999999998,"5min":0.32900000000000001,"15min":0.30099999999999999},"max":{"1min":8.7780000000000005,"5min":11.366,"15min":12.117000000000001},"last":0.88}]},{"osd":10,"last update":"Fri Feb 9 17:41:05 2024","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":2.246,"15min":1.714},"min":{"1min":0.57499999999999996,"5min":0.41899999999999998,"15min":0.35999999999999999},"max":{"1min":2.9729999999999999,"5min":15.083,"15min":15.083},"last":0.96599999999999997},{"interface":"front","average":{"1min":1.1799999999999999,"5min":2.29,"15min":1.99},"min":{"1min":0.55100000000000005,"5min":0.48099999999999998,"15min":0.33400000000000002},"max":{"1min":3.137,"5min":15.121,"15min":15.121},"last":1.0009999999999999}]},{"osd":11,"last update":"Fri Feb 9 17:40:01 2024","interfaces":[{"interface":"back","average":{"1min":2.9849999999999999,"5min":2.2050000000000001,"15min":2.1269999999999998},"min":{"1min":0.45200000000000001,"5min":0.38300000000000001,"15min":0.32000000000000001},"max":{"1min":8.7379999999999995,"5min":11.404999999999999,"15min":12.169},"last":0.41399999999999998},{"interface":"front","average":{"1min":2.3660000000000001,"5min":2.1040000000000001,"15min":2.0699999999999998},"min":{"1min":0.47099999999999997,"5min":0.42399999999999999,"15min":0.36599999999999999},"max":{"1min":7.0099999999999998,"5min":11.387,"15min":12.144},"last":0.56699999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738653,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1909648,"kb_used_data":1441156,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":91835504,"statfs":{"total":95995035648,"available":94039556096,"internally_reserved":0,"allocated":1475743744,"data_stored":1475225048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 9 17:41:08 2024","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":1.911,"15min":1.7170000000000001},"min":{"1min":0.23599999999999999,"5min":0.22800000000000001,"15min":0.20499999999999999},"max":{"1min":7.202,"5min":13.842000000000001,"15min":14.166},"last":0.56699999999999995},{"interface":"front","average":{"1min":1.4339999999999999,"5min":2.4289999999999998,"15min":1.992},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.215},"max":{"1min":7.1669999999999998,"5min":14.122,"15min":20.177},"last":0.59199999999999997}]},{"osd":2,"last update":"Fri Feb 9 17:39:25 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":5.5339999999999998,"5min":5.5339999999999998,"15min":5.5339999999999998},"last":1.744},{"interface":"front","average":{"1min":1.593,"5min":1.593,"15min":1.593},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":13.473000000000001,"5min":13.473000000000001,"15min":13.473000000000001},"last":3.109}]},{"osd":3,"last update":"Fri Feb 9 17:40:39 2024","interfaces":[{"interface":"back","average":{"1min":3.0230000000000001,"5min":3.0230000000000001,"15min":3.0230000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":16.756,"5min":16.756,"15min":16.756},"last":0.27500000000000002},{"interface":"front","average":{"1min":1.806,"5min":1.806,"15min":1.806},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":6.0570000000000004,"5min":6.0570000000000004,"15min":6.0570000000000004},"last":0.253}]},{"osd":4,"last update":"Fri Feb 9 17:40:27 2024","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":1.252,"15min":1.6879999999999999},"min":{"1min":0.29999999999999999,"5min":0.29399999999999998,"15min":0.253},"max":{"1min":5.625,"5min":7.7560000000000002,"15min":14.419},"last":0.311},{"interface":"front","average":{"1min":2.3769999999999998,"5min":1.597,"15min":1.9330000000000001},"min":{"1min":0.378,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":5.7050000000000001,"5min":13.308999999999999,"15min":14.465999999999999},"last":1.4179999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:40:48 2024","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":1.4910000000000001,"15min":1.7310000000000001},"min":{"1min":0.379,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":4.3079999999999998,"5min":8.4079999999999995,"15min":9.8569999999999993},"last":0.373},{"interface":"front","average":{"1min":0.84199999999999997,"5min":1.468,"15min":1.8500000000000001},"min":{"1min":0.41299999999999998,"5min":0.32400000000000001,"15min":0.312},"max":{"1min":3.407,"5min":7.8239999999999998,"15min":9.9049999999999994},"last":0.39900000000000002}]},{"osd":6,"last update":"Fri Feb 9 17:41:18 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":1.673,"15min":1.913},"min":{"1min":0.374,"5min":0.33300000000000002,"15min":0.27800000000000002},"max":{"1min":2.7810000000000001,"5min":13.407,"15min":13.407},"last":1.907},{"interface":"front","average":{"1min":0.80600000000000005,"5min":1.7250000000000001,"15min":1.8939999999999999},"min":{"1min":0.39200000000000002,"5min":0.379,"15min":0.33000000000000002},"max":{"1min":2.8079999999999998,"5min":13.448,"15min":13.448},"last":1.9319999999999999}]},{"osd":7,"last update":"Fri Feb 9 17:40:39 2024","interfaces":[{"interface":"back","average":{"1min":2.149,"5min":2.0529999999999999,"15min":2.1150000000000002},"min":{"1min":0.35299999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":5.8490000000000002,"5min":13.349,"15min":13.349},"last":1.0009999999999999},{"interface":"front","average":{"1min":2.4089999999999998,"5min":2.1539999999999999,"15min":2.0800000000000001},"min":{"1min":0.41699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":8.0009999999999994,"5min":14.451000000000001,"15min":16.765999999999998},"last":1.401}]},{"osd":8,"last update":"Fri Feb 9 17:40:40 2024","interfaces":[{"interface":"back","average":{"1min":2.0670000000000002,"5min":1.992,"15min":1.6519999999999999},"min":{"1min":0.496,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":6.6390000000000002,"5min":14.323,"15min":14.323},"last":1.3620000000000001},{"interface":"front","average":{"1min":2.1099999999999999,"5min":2.1070000000000002,"15min":1.7190000000000001},"min":{"1min":0.58499999999999996,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":6.6959999999999997,"5min":14.481999999999999,"15min":16.41},"last":1.474}]},{"osd":9,"last update":"Fri Feb 9 17:40:27 2024","interfaces":[{"interface":"back","average":{"1min":3.7869999999999999,"5min":2.1080000000000001,"15min":1.9159999999999999},"min":{"1min":0.35799999999999998,"5min":0.32500000000000001,"15min":0.312},"max":{"1min":17.952000000000002,"5min":17.952000000000002,"15min":17.952000000000002},"last":1.446},{"interface":"front","average":{"1min":3.806,"5min":2.2200000000000002,"15min":1.905},"min":{"1min":0.46500000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":17.913,"5min":17.913,"15min":17.913},"last":1.504}]},{"osd":10,"last update":"Fri Feb 9 17:40:48 2024","interfaces":[{"interface":"back","average":{"1min":1.752,"5min":2.2010000000000001,"15min":1.9319999999999999},"min":{"1min":0.52800000000000002,"5min":0.42399999999999999,"15min":0.38500000000000001},"max":{"1min":6.6689999999999996,"5min":14.271000000000001,"15min":16.876000000000001},"last":3.2040000000000002},{"interface":"front","average":{"1min":0.82799999999999996,"5min":1.3879999999999999,"15min":1.593},"min":{"1min":0.46600000000000003,"5min":0.40699999999999997,"15min":0.39800000000000002},"max":{"1min":3.3780000000000001,"5min":11.384,"15min":11.384},"last":0.47799999999999998}]},{"osd":11,"last update":"Fri Feb 9 17:41:18 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":1.3819999999999999,"15min":1.7509999999999999},"min":{"1min":0.41899999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":2.5430000000000001,"5min":7.8739999999999997,"15min":40.021000000000001},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.4790000000000001,"15min":1.905},"min":{"1min":0.5,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":2.702,"5min":7.899,"15min":40.055999999999997},"last":0.53400000000000003}]}]},{"osd":2,"up_from":0,"seq":77309411602,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":27,"seq":115964117263,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":571804,"kb_used_data":183820,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":93173348,"statfs":{"total":95995035648,"available":95409508352,"internally_reserved":0,"allocated":188231680,"data_stored":187709432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":397277639},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:41:07 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":2.355,"15min":1.8979999999999999},"min":{"1min":0.27100000000000002,"5min":0.186,"15min":0.186},"max":{"1min":2.1840000000000002,"5min":38.984999999999999,"15min":38.984999999999999},"last":15.657999999999999},{"interface":"front","average":{"1min":0.45500000000000002,"5min":2.556,"15min":2},"min":{"1min":0.27600000000000002,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":1.417,"5min":41.497999999999998,"15min":41.497999999999998},"last":16.637}]},{"osd":1,"last update":"Fri Feb 9 17:40:06 2024","interfaces":[{"interface":"back","average":{"1min":4.5670000000000002,"5min":4.5670000000000002,"15min":4.5670000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":25.965,"5min":25.965,"15min":25.965},"last":0.35299999999999998},{"interface":"front","average":{"1min":4.5129999999999999,"5min":4.5129999999999999,"15min":4.5129999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":31.454000000000001,"5min":31.454000000000001,"15min":31.454000000000001},"last":0.375}]},{"osd":2,"last update":"Fri Feb 9 17:41:00 2024","interfaces":[{"interface":"back","average":{"1min":1.9119999999999999,"5min":2.1240000000000001,"15min":1.8740000000000001},"min":{"1min":0.29099999999999998,"5min":0.255,"15min":0.22600000000000001},"max":{"1min":11,"5min":11.663,"15min":14.914999999999999},"last":0.33200000000000002},{"interface":"front","average":{"1min":1.5249999999999999,"5min":2.0529999999999999,"15min":1.891},"min":{"1min":0.32200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.071,"5min":12.347,"15min":17.613},"last":0.371}]},{"osd":4,"last update":"Fri Feb 9 17:41:10 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":1.5940000000000001,"15min":1.9350000000000001},"min":{"1min":0.40000000000000002,"5min":0.26800000000000002,"15min":0.254},"max":{"1min":2.2639999999999998,"5min":13.933,"15min":34.898000000000003},"last":0.57599999999999996},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.6830000000000001,"15min":1.8300000000000001},"min":{"1min":0.437,"5min":0.28000000000000003,"15min":0.27900000000000003},"max":{"1min":3.4380000000000002,"5min":9.3059999999999992,"15min":14.061999999999999},"last":0.65700000000000003}]},{"osd":5,"last update":"Fri Feb 9 17:40:45 2024","interfaces":[{"interface":"back","average":{"1min":2.504,"5min":1.71,"15min":1.6699999999999999},"min":{"1min":0.33700000000000002,"5min":0.29899999999999999,"15min":0.27300000000000002},"max":{"1min":11.007999999999999,"5min":11.007999999999999,"15min":17.651},"last":0.749},{"interface":"front","average":{"1min":2.5619999999999998,"5min":1.7689999999999999,"15min":1.724},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.33300000000000002},"max":{"1min":11.036,"5min":11.036,"15min":17.675999999999998},"last":0.71599999999999997}]},{"osd":6,"last update":"Fri Feb 9 17:40:34 2024","interfaces":[{"interface":"back","average":{"1min":2.7450000000000001,"5min":2.7450000000000001,"15min":2.7450000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":12.717000000000001,"5min":12.717000000000001,"15min":12.717000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":8.7360000000000007,"5min":8.7360000000000007,"15min":8.7360000000000007},"last":0.60699999999999998}]},{"osd":7,"last update":"Fri Feb 9 17:41:20 2024","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.872,"15min":0.872},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":3.3999999999999999,"5min":3.3999999999999999,"15min":3.3999999999999999},"last":0.626},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":2.7989999999999999,"5min":2.7989999999999999,"15min":2.7989999999999999},"last":0.72599999999999998}]},{"osd":8,"last update":"Fri Feb 9 17:41:10 2024","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.6759999999999999,"15min":1.4410000000000001},"min":{"1min":0.54100000000000004,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":6.2229999999999999,"5min":8.6940000000000008,"15min":8.6940000000000008},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.78700000000000003,"5min":1.851,"15min":1.2050000000000001},"min":{"1min":0.442,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":2.3079999999999998,"5min":13.976000000000001,"15min":14.129},"last":0.59599999999999997}]},{"osd":9,"last update":"Fri Feb 9 17:40:28 2024","interfaces":[{"interface":"back","average":{"1min":1.617,"5min":1.7150000000000001,"15min":1.573},"min":{"1min":0.36199999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":8.8160000000000007,"5min":9.7929999999999993,"15min":9.7929999999999993},"last":0.58599999999999997},{"interface":"front","average":{"1min":1.8169999999999999,"5min":1.6850000000000001,"15min":1.7989999999999999},"min":{"1min":0.497,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":8.7899999999999991,"5min":12.815,"15min":13.717000000000001},"last":0.51400000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:41:16 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":2.2850000000000001,"5min":2.2850000000000001,"15min":2.2850000000000001},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.248,"5min":1.248,"15min":1.248},"last":0.49299999999999999}]},{"osd":11,"last update":"Fri Feb 9 17:41:10 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":1.7490000000000001,"15min":1.6539999999999999},"min":{"1min":0.48399999999999999,"5min":0.40400000000000003,"15min":0.38800000000000001},"max":{"1min":1.613,"5min":14.300000000000001,"15min":14.300000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.73399999999999999,"5min":2.3500000000000001,"15min":2.0099999999999998},"min":{"1min":0.495,"5min":0.442,"15min":0.36899999999999999},"max":{"1min":1.2769999999999999,"5min":17.145,"15min":17.145},"last":0.61399999999999999}]}]},{"osd":4,"up_from":32,"seq":137438953736,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1402116,"kb_used_data":991928,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":92343036,"statfs":{"total":95995035648,"available":94559268864,"internally_reserved":0,"allocated":1015734272,"data_stored":1015220124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":420018629},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:41:14 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":1.6930000000000001,"15min":1.7150000000000001},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23599999999999999},"max":{"1min":2.073,"5min":7.9669999999999996,"15min":50.356999999999999},"last":2.7389999999999999},{"interface":"front","average":{"1min":0.77800000000000002,"5min":1.851,"15min":1.6479999999999999},"min":{"1min":0.34599999999999997,"5min":0.247,"15min":0.247},"max":{"1min":2.633,"5min":8.0440000000000005,"15min":50.569000000000003},"last":2.8119999999999998}]},{"osd":1,"last update":"Fri Feb 9 17:41:18 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":2.3420000000000001,"15min":2.0539999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":4.0540000000000003,"5min":28.143000000000001,"15min":50.473999999999997},"last":0.45800000000000002},{"interface":"front","average":{"1min":1.073,"5min":2.3799999999999999,"15min":2.1179999999999999},"min":{"1min":0.35899999999999999,"5min":0.31,"15min":0.28100000000000003},"max":{"1min":4.0679999999999996,"5min":28.103000000000002,"15min":50.499000000000002},"last":3.1699999999999999}]},{"osd":2,"last update":"Fri Feb 9 17:28:51 2024","interfaces":[{"interface":"back","average":{"1min":2.536,"5min":1.546,"15min":0.91400000000000003},"min":{"1min":0.29999999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":8.8819999999999997,"5min":8.8819999999999997,"15min":8.8819999999999997},"last":0.622},{"interface":"front","average":{"1min":2.214,"5min":1.7330000000000001,"15min":0.97799999999999998},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":9.5039999999999996,"5min":9.5039999999999996,"15min":9.5039999999999996},"last":0.53800000000000003}]},{"osd":3,"last update":"Fri Feb 9 17:41:18 2024","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":1.8109999999999999,"15min":2.1619999999999999},"min":{"1min":0.311,"5min":0.245,"15min":0.245},"max":{"1min":3.7000000000000002,"5min":7.9349999999999996,"15min":50.462000000000003},"last":3.2050000000000001},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.8700000000000001,"15min":2.0009999999999999},"min":{"1min":0.40699999999999997,"5min":0.29699999999999999,"15min":0.255},"max":{"1min":4.8330000000000002,"5min":9.1579999999999995,"15min":50.529000000000003},"last":2.7799999999999998}]},{"osd":5,"last update":"Fri Feb 9 17:40:44 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.93999999999999995,"15min":1.294},"min":{"1min":0.26100000000000001,"5min":0.22500000000000001,"15min":0.222},"max":{"1min":0.96099999999999997,"5min":6.3760000000000003,"15min":9.5630000000000006},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.56699999999999995,"5min":1.0209999999999999,"15min":1.5700000000000001},"min":{"1min":0.34000000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.995,"5min":6.4080000000000004,"15min":50.509999999999998},"last":0.498}]},{"osd":6,"last update":"Fri Feb 9 17:41:18 2024","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.333,"15min":1.474},"min":{"1min":0.28999999999999998,"5min":0.246,"15min":0.246},"max":{"1min":7.4249999999999998,"5min":8.0039999999999996,"15min":18.09},"last":3.843},{"interface":"front","average":{"1min":1.137,"5min":1.137,"15min":1.6160000000000001},"min":{"1min":0.442,"5min":0.245,"15min":0.245},"max":{"1min":5.524,"5min":6.7329999999999997,"15min":50.402999999999999},"last":3.1150000000000002}]},{"osd":7,"last update":"Fri Feb 9 17:40:49 2024","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.042,"15min":1.575},"min":{"1min":0.32200000000000001,"5min":0.29899999999999999,"15min":0.28399999999999997},"max":{"1min":7.4640000000000004,"5min":7.4640000000000004,"15min":15.223000000000001},"last":3.536},{"interface":"front","average":{"1min":0.64300000000000002,"5min":1.1060000000000001,"15min":1.5469999999999999},"min":{"1min":0.40999999999999998,"5min":0.28199999999999997,"15min":0.28100000000000003},"max":{"1min":1.0249999999999999,"5min":6.7640000000000002,"15min":19.047000000000001},"last":3.2400000000000002}]},{"osd":8,"last update":"Fri Feb 9 17:40:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":1.0109999999999999,"15min":1.4159999999999999},"min":{"1min":0.39100000000000001,"5min":0.34799999999999998,"15min":0.34399999999999997},"max":{"1min":2.6749999999999998,"5min":6.1079999999999997,"15min":8.4160000000000004},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.998,"15min":1.5349999999999999},"min":{"1min":0.377,"5min":0.36599999999999999,"15min":0.308},"max":{"1min":1.821,"5min":6,"15min":50.594000000000001},"last":2.9209999999999998}]},{"osd":9,"last update":"Fri Feb 9 17:40:23 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":1.1240000000000001,"15min":1.621},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.5840000000000001,"5min":6.9699999999999998,"15min":50.442999999999998},"last":0.58999999999999997},{"interface":"front","average":{"1min":1.302,"5min":1.286,"15min":1.446},"min":{"1min":0.46800000000000003,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":7.5999999999999996,"5min":7.5999999999999996,"15min":10.555},"last":3.7679999999999998}]},{"osd":10,"last update":"Fri Feb 9 17:40:59 2024","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.482,"15min":1.8600000000000001},"min":{"1min":0.40300000000000002,"5min":0.39400000000000002,"15min":0.378},"max":{"1min":5.7629999999999999,"5min":7.0030000000000001,"15min":51.223999999999997},"last":0.56399999999999995},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.474,"15min":1.893},"min":{"1min":0.47699999999999998,"5min":0.41999999999999998,"15min":0.38100000000000001},"max":{"1min":5.7000000000000002,"5min":6.3860000000000001,"15min":50.552},"last":0.79800000000000004}]},{"osd":11,"last update":"Fri Feb 9 17:40:23 2024","interfaces":[{"interface":"back","average":{"1min":1.5489999999999999,"5min":1.4950000000000001,"15min":1.954},"min":{"1min":0.49099999999999999,"5min":0.48599999999999999,"15min":0.40500000000000003},"max":{"1min":5.3550000000000004,"5min":6.9379999999999997,"15min":50.484999999999999},"last":0.624},{"interface":"front","average":{"1min":1.365,"5min":1.4530000000000001,"15min":1.6379999999999999},"min":{"1min":0.53000000000000003,"5min":0.435,"15min":0.34899999999999998},"max":{"1min":7.694,"5min":7.694,"15min":18.574999999999999},"last":2.859}]}]},{"osd":5,"up_from":40,"seq":171798692099,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1685972,"kb_used_data":1329932,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":92059180,"statfs":{"total":95995035648,"available":94268600320,"internally_reserved":0,"allocated":1361850368,"data_stored":1361328120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:41:09 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.995,"15min":1.0329999999999999},"min":{"1min":0.28100000000000003,"5min":0.252,"15min":0.252},"max":{"1min":4.4560000000000004,"5min":4.4560000000000004,"15min":4.4560000000000004},"last":1.357},{"interface":"front","average":{"1min":1.075,"5min":1.6659999999999999,"15min":1.764},"min":{"1min":0.313,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":6.4939999999999998,"5min":6.4939999999999998,"15min":6.4939999999999998},"last":0.46400000000000002}]},{"osd":1,"last update":"Fri Feb 9 17:40:42 2024","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.9750000000000001,"15min":2.1360000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.23200000000000001},"max":{"1min":7.6559999999999997,"5min":48.161999999999999,"15min":48.161999999999999},"last":0.38100000000000001},{"interface":"front","average":{"1min":2.02,"5min":2.1080000000000001,"15min":2.0499999999999998},"min":{"1min":0.28000000000000003,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":7.4390000000000001,"5min":48.218000000000004,"15min":48.218000000000004},"last":1.3939999999999999}]},{"osd":2,"last update":"Fri Feb 9 17:40:42 2024","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":1.345,"15min":1.6539999999999999},"min":{"1min":0.32000000000000001,"5min":0.27900000000000003,"15min":0.25600000000000001},"max":{"1min":4.4930000000000003,"5min":11.143000000000001,"15min":11.143000000000001},"last":2.6560000000000001},{"interface":"front","average":{"1min":1.796,"5min":1.218,"15min":1.718},"min":{"1min":0.39100000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":6.1970000000000001,"5min":6.1970000000000001,"15min":14.122},"last":1.5649999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:40:42 2024","interfaces":[{"interface":"back","average":{"1min":1.8300000000000001,"5min":2.3239999999999998,"15min":2.1760000000000002},"min":{"1min":0.32300000000000001,"5min":0.29799999999999999,"15min":0.26500000000000001},"max":{"1min":11.081,"5min":64.150999999999996,"15min":64.150999999999996},"last":1.4139999999999999},{"interface":"front","average":{"1min":2.3149999999999999,"5min":2.3679999999999999,"15min":2.1379999999999999},"min":{"1min":0.36299999999999999,"5min":0.247,"15min":0.247},"max":{"1min":12.468,"5min":64.176000000000002,"15min":64.176000000000002},"last":0.44400000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:40:45 2024","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":1.5209999999999999,"15min":1.472},"min":{"1min":0.35799999999999998,"5min":0.28100000000000003,"15min":0.23400000000000001},"max":{"1min":6.4619999999999997,"5min":49.313000000000002,"15min":49.313000000000002},"last":1.575},{"interface":"front","average":{"1min":0.97499999999999998,"5min":1.4359999999999999,"15min":1.4810000000000001},"min":{"1min":0.38100000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":6.4260000000000002,"5min":50.505000000000003,"15min":50.505000000000003},"last":0.48099999999999998}]},{"osd":6,"last update":"Fri Feb 9 17:41:21 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":1.2010000000000001,"15min":1.4350000000000001},"min":{"1min":0.32100000000000001,"5min":0.26400000000000001,"15min":0.253},"max":{"1min":2.1930000000000001,"5min":8.609,"15min":15.375999999999999},"last":1.8500000000000001},{"interface":"front","average":{"1min":0.95399999999999996,"5min":1.29,"15min":1.508},"min":{"1min":0.40799999999999997,"5min":0.28899999999999998,"15min":0.27200000000000002},"max":{"1min":2.7530000000000001,"5min":11.038,"15min":14.250999999999999},"last":1.603}]},{"osd":7,"last update":"Fri Feb 9 17:40:42 2024","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":1.5329999999999999,"15min":1.536},"min":{"1min":0.33000000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":3.8380000000000001,"5min":50.545000000000002,"15min":50.545000000000002},"last":0.495},{"interface":"front","average":{"1min":1.1879999999999999,"5min":2.5739999999999998,"15min":2.0830000000000002},"min":{"1min":0.36399999999999999,"5min":0.32100000000000001,"15min":0.28799999999999998},"max":{"1min":6.968,"5min":102.526,"15min":102.526},"last":0.32400000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:41:21 2024","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":1.361,"15min":1.581},"min":{"1min":0.40899999999999997,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":14.542,"5min":14.542,"15min":14.542},"last":1.675},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.94699999999999995,"15min":1.764},"min":{"1min":0.40000000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.524,"5min":5.71,"15min":9.6240000000000006},"last":1.1279999999999999}]},{"osd":9,"last update":"Fri Feb 9 17:41:21 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":1.23,"15min":1.3700000000000001},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.35999999999999999},"max":{"1min":1.4179999999999999,"5min":6.8600000000000003,"15min":14.98},"last":1.653},{"interface":"front","average":{"1min":0.995,"5min":1.1950000000000001,"15min":1.369},"min":{"1min":0.45300000000000001,"5min":0.35699999999999998,"15min":0.312},"max":{"1min":5.0439999999999996,"5min":7.7779999999999996,"15min":9.4239999999999995},"last":0.40600000000000003}]},{"osd":10,"last update":"Fri Feb 9 17:40:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.002,"15min":1.4430000000000001},"min":{"1min":0.54200000000000004,"5min":0.35199999999999998,"15min":0.33800000000000002},"max":{"1min":3.6589999999999998,"5min":5.7400000000000002,"15min":14.218},"last":0.51800000000000002},{"interface":"front","average":{"1min":1.452,"5min":1.319,"15min":1.599},"min":{"1min":0.442,"5min":0.42399999999999999,"15min":0.36099999999999999},"max":{"1min":7.1159999999999997,"5min":10.606,"15min":14.186999999999999},"last":0.434}]},{"osd":11,"last update":"Fri Feb 9 17:38:19 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":1.54,"15min":1.46},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":2.6579999999999999,"5min":9.4239999999999995,"15min":9.4239999999999995},"last":1.756},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.6639999999999999,"15min":1.5129999999999999},"min":{"1min":0.41399999999999998,"5min":0.39200000000000002,"15min":0.38900000000000001},"max":{"1min":3.4870000000000001,"5min":8.125,"15min":14.316000000000001},"last":2.012}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1529413632,"data_stored":1529413632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":697106432,"data_stored":697106432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":188547072,"data_stored":188547072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1054212096,"data_stored":1054212096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1396441088,"data_stored":1396441088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":536805376,"data_stored":536805376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":756088832,"data_stored":756088832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":546242560,"data_stored":546242560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":540934144,"data_stored":540934144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1153630208,"data_stored":1153630208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":697434112,"data_stored":697434112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-09T17:41:33.070 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph pg dump --format=json 2024-02-09T17:41:33.318 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:41:33.319 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:41:33.615 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:41:33.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:33.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:41:33.722 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:41:33.759 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:33.790 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:41:33.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:33.795 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:41:33.896 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:41:33.929 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:33.954 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 29 16 57965 57949 124.875 96 0.00837009 0.00800376 2024-02-09T17:41:33.971 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:41:33.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:33.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:41:34.077 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:41:34.120 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:33 smithi179 ceph-mon[43975]: pgmap v1230: 37 pgs: 37 active+clean; 3.1 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-02-09T17:41:34.122 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:34.155 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:41:34.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:34.158 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:41:34.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:33 smithi162 ceph-mon[44304]: pgmap v1230: 37 pgs: 37 active+clean; 3.1 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-02-09T17:41:34.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:33 smithi099 ceph-mon[39420]: pgmap v1230: 37 pgs: 37 active+clean; 3.1 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-02-09T17:41:34.259 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:41:34.305 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:34.328 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:41:34.330 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:41:34.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:34.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:41:34.344 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-09T17:41:34.439 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:41:34.495 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:34.528 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:41:34.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:34.532 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:41:34.633 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:41:34.690 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:34.724 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:41:34.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:34.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:41:34.830 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:41:34.869 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:34.955 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 30 16 59727 59711 124.383 110.125 0.0073227 0.00803468 2024-02-09T17:41:35.227 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:41:35.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:35.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:41:35.250 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:41:35.257 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:41:35.334 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:41:35.388 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:35.421 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:41:35.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:35.428 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:41:35.529 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:41:35.573 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:35.650 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:41:35.955 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 31 16 61092 61076 123.122 85.3125 0.00269588 0.00804365 2024-02-09T17:41:36.133 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:41:36.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:36.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:41:36.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:35 smithi179 ceph-mon[43975]: pgmap v1231: 37 pgs: 37 active+clean; 3.3 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-02-09T17:41:36.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:35 smithi162 ceph-mon[44304]: pgmap v1231: 37 pgs: 37 active+clean; 3.3 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-02-09T17:41:36.237 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:41:36.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:35 smithi099 ceph-mon[39420]: pgmap v1231: 37 pgs: 37 active+clean; 3.3 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-02-09T17:41:36.274 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:36.374 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:41:36.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:36.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:41:36.479 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:41:36.523 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:36.625 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:41:36.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:36.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:41:36.730 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:41:36.765 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:36.961 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 32 16 62445 62429 121.912 84.5625 0.00882996 0.00819845 2024-02-09T17:41:36.965 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:41:36.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:36.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:41:37.069 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:41:37.112 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:37.142 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:41:37.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:37.147 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:41:37.251 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:41:37.289 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:37.317 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:41:37.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:37.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:41:37.422 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:41:37.461 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:37.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:37 smithi162 ceph-mon[44304]: pgmap v1232: 37 pgs: 37 active+clean; 3.6 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 126 MiB/s wr, 2.01k op/s 2024-02-09T17:41:37.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:37 smithi099 ceph-mon[39420]: pgmap v1232: 37 pgs: 37 active+clean; 3.6 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 126 MiB/s wr, 2.01k op/s 2024-02-09T17:41:37.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:37 smithi179 ceph-mon[43975]: pgmap v1232: 37 pgs: 37 active+clean; 3.6 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 126 MiB/s wr, 2.01k op/s 2024-02-09T17:41:37.956 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 33 16 63887 63871 120.948 90.125 0.00938691 0.00826411 2024-02-09T17:41:38.623 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:41:38.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:38.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:41:38.728 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:41:38.769 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:38.793 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:41:38.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:38.796 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:41:38.861 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:41:38.897 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:41:38.931 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:38.956 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 34 16 65551 65535 120.449 104 0.00739697 0.00829796 2024-02-09T17:41:39.153 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:41:39.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:39.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:41:39.258 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:41:39.316 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:39.365 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:41:39.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:39.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:41:39.470 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:41:39.506 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:39.595 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:41:39.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:39.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:41:39.700 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:41:39.754 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:39.785 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:41:39.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:39.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:41:39.877 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is unset 2024-02-09T17:41:39.889 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:41:39.895 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:41:39.895 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:41:39.934 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:41:39.935 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:39.937 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-02-09T17:41:39.957 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 35 16 67153 67137 119.868 100.125 0.0162844 0.00833772 2024-02-09T17:41:39.959 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:41:39.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:39.964 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:41:40.065 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:41:40.106 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:39 smithi162 ceph-mon[44304]: pgmap v1233: 37 pgs: 37 active+clean; 3.7 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-02-09T17:41:40.106 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:39 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3798914215' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:41:40.107 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:40.147 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:41:40.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:40.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:41:40.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:39 smithi179 ceph-mon[43975]: pgmap v1233: 37 pgs: 37 active+clean; 3.7 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-02-09T17:41:40.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:39 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3798914215' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:41:40.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:39 smithi099 ceph-mon[39420]: pgmap v1233: 37 pgs: 37 active+clean; 3.7 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-02-09T17:41:40.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:39 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3798914215' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:41:40.251 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:41:40.287 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:40.332 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:41:40.339 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:40.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:41:40.440 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:41:40.482 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:40.509 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:41:40.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:40.514 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:41:40.615 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:41:40.657 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:40.685 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:41:40.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:40.688 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:41:40.789 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:41:40.837 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:40.877 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:41:40.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:40.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:41:40.957 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 36 16 68738 68722 119.29 99.0625 0.00965024 0.00837911 2024-02-09T17:41:40.985 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:41:41.029 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:41.049 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:40 smithi099 ceph-mon[39420]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:41:41.049 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:40 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3798914215' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:41:41.049 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:40 smithi099 ceph-mon[39420]: osdmap e487: 12 total, 12 up, 11 in 2024-02-09T17:41:41.049 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:40 smithi099 ceph-mon[39420]: from='client.15831 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:41:41.050 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":1234,"stamp":"2024-02-09T17:41:39.872843+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4115661344,"num_objects":62793,"num_object_clones":0,"num_object_copies":188379,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62791,"num_write_kb":4018624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3629,"num_bytes_recovered":238499424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65710,"ondisk_log_size":65710,"up":111,"acting":111,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":113,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":16588004,"kb_used_data":12049292,"kb_used_omap":228,"kb_used_meta":4538395,"kb_avail":1014608668,"statfs":{"total":1055945392128,"available":1038959276032,"internally_reserved":0,"allocated":12338475008,"data_stored":12332659740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":233532,"internal_metadata":4647317444},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":987889664,"num_objects":15074,"num_object_clones":0,"num_object_copies":45222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15074,"num_write_kb":964736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.071282"},"pg_stats":[{"pgid":"1.14","version":"63'139","reported_seq":930,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-02-09T17:41:31.507955+0000","last_change":"2024-02-09T17:41:08.077049+0000","last_active":"2024-02-09T17:41:31.507955+0000","last_peered":"2024-02-09T17:41:31.507955+0000","last_clean":"2024-02-09T17:41:31.507955+0000","last_became_active":"2024-02-09T17:39:27.455644+0000","last_became_peered":"2024-02-09T17:39:27.455644+0000","last_unstale":"2024-02-09T17:41:31.507955+0000","last_undegraded":"2024-02-09T17:41:31.507955+0000","last_fullsized":"2024-02-09T17:41:31.507955+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:08.076894+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:38:26.968892+0000","last_clean_scrub_stamp":"2024-02-09T17:41:08.076894+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:16.380706+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"486'3911","reported_seq":3963,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-02-09T17:41:37.873591+0000","last_change":"2024-02-09T17:40:48.780201+0000","last_active":"2024-02-09T17:41:37.873591+0000","last_peered":"2024-02-09T17:41:37.873591+0000","last_clean":"2024-02-09T17:41:37.873591+0000","last_became_active":"2024-02-09T17:40:48.779925+0000","last_became_peered":"2024-02-09T17:40:48.779925+0000","last_unstale":"2024-02-09T17:41:37.873591+0000","last_undegraded":"2024-02-09T17:41:37.873591+0000","last_fullsized":"2024-02-09T17:41:37.873591+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:40:47.763277+0000","objects_scrubbed":0,"log_size":3911,"log_dups_size":0,"ondisk_log_size":3911,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:10.602974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256311296,"num_objects":3911,"num_object_clones":0,"num_object_copies":11733,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3911,"num_write_kb":250304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,8],"acting":[10,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"63'139","reported_seq":1282,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-02-09T17:41:31.507616+0000","last_change":"2024-02-09T17:41:14.521470+0000","last_active":"2024-02-09T17:41:31.507616+0000","last_peered":"2024-02-09T17:41:31.507616+0000","last_clean":"2024-02-09T17:41:31.507616+0000","last_became_active":"2024-02-09T17:41:14.521079+0000","last_became_peered":"2024-02-09T17:41:14.521079+0000","last_unstale":"2024-02-09T17:41:31.507616+0000","last_undegraded":"2024-02-09T17:41:31.507616+0000","last_fullsized":"2024-02-09T17:41:31.507616+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:10.115197+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:41:10.115197+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:18.769273+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"486'4079","reported_seq":4131,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-02-09T17:41:39.074585+0000","last_change":"2024-02-09T17:40:48.783610+0000","last_active":"2024-02-09T17:41:39.074585+0000","last_peered":"2024-02-09T17:41:39.074585+0000","last_clean":"2024-02-09T17:41:39.074585+0000","last_became_active":"2024-02-09T17:40:48.783218+0000","last_became_peered":"2024-02-09T17:40:48.783218+0000","last_unstale":"2024-02-09T17:41:39.074585+0000","last_undegraded":"2024-02-09T17:41:39.074585+0000","last_fullsized":"2024-02-09T17:41:39.074585+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:40:47.763277+0000","objects_scrubbed":0,"log_size":4079,"log_dups_size":0,"ondisk_log_size":4079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:02.568964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":267321344,"num_objects":4079,"num_object_clones":0,"num_object_copies":12237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4079,"num_write_kb":261056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"63'139","reported_seq":1011,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-02-09T17:41:31.507722+0000","last_change":"2024-02-09T17:40:22.485379+0000","last_active":"2024-02-09T17:41:31.507722+0000","last_peered":"2024-02-09T17:41:31.507722+0000","last_clean":"2024-02-09T17:41:31.507722+0000","last_became_active":"2024-02-09T17:40:22.484952+0000","last_became_peered":"2024-02-09T17:40:22.484952+0000","last_unstale":"2024-02-09T17:41:31.507722+0000","last_undegraded":"2024-02-09T17:41:31.507722+0000","last_fullsized":"2024-02-09T17:41:31.507722+0000","mapping_epoch":454,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":455,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:39:46.807673+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:39:46.807673+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"486'4008","reported_seq":4060,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-02-09T17:41:39.089445+0000","last_change":"2024-02-09T17:40:48.781806+0000","last_active":"2024-02-09T17:41:39.089445+0000","last_peered":"2024-02-09T17:41:39.089445+0000","last_clean":"2024-02-09T17:41:39.089445+0000","last_became_active":"2024-02-09T17:40:48.781472+0000","last_became_peered":"2024-02-09T17:40:48.781472+0000","last_unstale":"2024-02-09T17:41:39.089445+0000","last_undegraded":"2024-02-09T17:41:39.089445+0000","last_fullsized":"2024-02-09T17:41:39.089445+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:40:47.763277+0000","objects_scrubbed":0,"log_size":4008,"log_dups_size":0,"ondisk_log_size":4008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:13.563605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262668288,"num_objects":4008,"num_object_clones":0,"num_object_copies":12024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4008,"num_write_kb":256512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"63'139","reported_seq":464,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-02-09T17:41:31.503800+0000","last_change":"2024-02-09T17:40:57.059982+0000","last_active":"2024-02-09T17:41:31.503800+0000","last_peered":"2024-02-09T17:41:31.503800+0000","last_clean":"2024-02-09T17:41:31.503800+0000","last_became_active":"2024-02-09T17:40:13.501711+0000","last_became_peered":"2024-02-09T17:40:13.501711+0000","last_unstale":"2024-02-09T17:41:31.503800+0000","last_undegraded":"2024-02-09T17:41:31.503800+0000","last_fullsized":"2024-02-09T17:41:31.503800+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:40:57.059847+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:38:27.045816+0000","last_clean_scrub_stamp":"2024-02-09T17:40:57.059847+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:04.876510+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"486'4073","reported_seq":4125,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-02-09T17:41:38.843461+0000","last_change":"2024-02-09T17:40:48.780916+0000","last_active":"2024-02-09T17:41:38.843461+0000","last_peered":"2024-02-09T17:41:38.843461+0000","last_clean":"2024-02-09T17:41:38.843461+0000","last_became_active":"2024-02-09T17:40:48.780579+0000","last_became_peered":"2024-02-09T17:40:48.780579+0000","last_unstale":"2024-02-09T17:41:38.843461+0000","last_undegraded":"2024-02-09T17:41:38.843461+0000","last_fullsized":"2024-02-09T17:41:38.843461+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:40:47.763277+0000","objects_scrubbed":0,"log_size":4073,"log_dups_size":0,"ondisk_log_size":4073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:03.096001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266928128,"num_objects":4073,"num_object_clones":0,"num_object_copies":12219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4073,"num_write_kb":260672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"63'139","reported_seq":780,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-02-09T17:41:31.505292+0000","last_change":"2024-02-09T17:40:46.690262+0000","last_active":"2024-02-09T17:41:31.505292+0000","last_peered":"2024-02-09T17:41:31.505292+0000","last_clean":"2024-02-09T17:41:31.505292+0000","last_became_active":"2024-02-09T17:34:01.342607+0000","last_became_peered":"2024-02-09T17:34:01.342607+0000","last_unstale":"2024-02-09T17:41:31.505292+0000","last_undegraded":"2024-02-09T17:41:31.505292+0000","last_fullsized":"2024-02-09T17:41:31.505292+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:40:46.690100+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:40:46.690100+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:03.472698+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.2","version":"486'3902","reported_seq":3954,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-02-09T17:41:35.771554+0000","last_change":"2024-02-09T17:40:48.781838+0000","last_active":"2024-02-09T17:41:35.771554+0000","last_peered":"2024-02-09T17:41:35.771554+0000","last_clean":"2024-02-09T17:41:35.771554+0000","last_became_active":"2024-02-09T17:40:48.781532+0000","last_became_peered":"2024-02-09T17:40:48.781532+0000","last_unstale":"2024-02-09T17:41:35.771554+0000","last_undegraded":"2024-02-09T17:41:35.771554+0000","last_fullsized":"2024-02-09T17:41:35.771554+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:40:47.763277+0000","objects_scrubbed":0,"log_size":3902,"log_dups_size":0,"ondisk_log_size":3902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:10.569691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255721472,"num_objects":3902,"num_object_clones":0,"num_object_copies":11706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3902,"num_write_kb":249728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"63'139","reported_seq":771,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-02-09T17:41:31.507229+0000","last_change":"2024-02-09T17:40:01.416463+0000","last_active":"2024-02-09T17:41:31.507229+0000","last_peered":"2024-02-09T17:41:31.507229+0000","last_clean":"2024-02-09T17:41:31.507229+0000","last_became_active":"2024-02-09T17:33:00.284702+0000","last_became_peered":"2024-02-09T17:33:00.284702+0000","last_unstale":"2024-02-09T17:41:31.507229+0000","last_undegraded":"2024-02-09T17:41:31.507229+0000","last_fullsized":"2024-02-09T17:41:31.507229+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:40:01.416303+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:40:01.416303+0000","last_clean_scrub_stamp":"2024-02-09T17:40:01.416303+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:41:32.600839+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.3","version":"486'3713","reported_seq":3765,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-02-09T17:41:35.768755+0000","last_change":"2024-02-09T17:40:48.783695+0000","last_active":"2024-02-09T17:41:35.768755+0000","last_peered":"2024-02-09T17:41:35.768755+0000","last_clean":"2024-02-09T17:41:35.768755+0000","last_became_active":"2024-02-09T17:40:48.783369+0000","last_became_peered":"2024-02-09T17:40:48.783369+0000","last_unstale":"2024-02-09T17:41:35.768755+0000","last_undegraded":"2024-02-09T17:41:35.768755+0000","last_fullsized":"2024-02-09T17:41:35.768755+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:40:47.763277+0000","objects_scrubbed":0,"log_size":3713,"log_dups_size":0,"ondisk_log_size":3713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:41:48.151059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243335168,"num_objects":3713,"num_object_clones":0,"num_object_copies":11139,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3713,"num_write_kb":237632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"63'139","reported_seq":442,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-02-09T17:41:31.506568+0000","last_change":"2024-02-09T17:40:13.498076+0000","last_active":"2024-02-09T17:41:31.506568+0000","last_peered":"2024-02-09T17:41:31.506568+0000","last_clean":"2024-02-09T17:41:31.506568+0000","last_became_active":"2024-02-09T17:40:13.497710+0000","last_became_peered":"2024-02-09T17:40:13.497710+0000","last_unstale":"2024-02-09T17:41:31.506568+0000","last_undegraded":"2024-02-09T17:41:31.506568+0000","last_fullsized":"2024-02-09T17:41:31.506568+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:40:02.483295+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:40:02.483295+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:41:36.087141+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"486'4135","reported_seq":4186,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-02-09T17:41:37.866403+0000","last_change":"2024-02-09T17:40:48.780689+0000","last_active":"2024-02-09T17:41:37.866403+0000","last_peered":"2024-02-09T17:41:37.866403+0000","last_clean":"2024-02-09T17:41:37.866403+0000","last_became_active":"2024-02-09T17:40:48.780290+0000","last_became_peered":"2024-02-09T17:40:48.780290+0000","last_unstale":"2024-02-09T17:41:37.866403+0000","last_undegraded":"2024-02-09T17:41:37.866403+0000","last_fullsized":"2024-02-09T17:41:37.866403+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:40:47.763277+0000","objects_scrubbed":0,"log_size":4135,"log_dups_size":0,"ondisk_log_size":4135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:00.604584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":270991360,"num_objects":4135,"num_object_clones":0,"num_object_copies":12405,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4135,"num_write_kb":264640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"63'139","reported_seq":862,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-02-09T17:41:31.507771+0000","last_change":"2024-02-09T17:40:44.545620+0000","last_active":"2024-02-09T17:41:31.507771+0000","last_peered":"2024-02-09T17:41:31.507771+0000","last_clean":"2024-02-09T17:41:31.507771+0000","last_became_active":"2024-02-09T17:33:00.284924+0000","last_became_peered":"2024-02-09T17:33:00.284924+0000","last_unstale":"2024-02-09T17:41:31.507771+0000","last_undegraded":"2024-02-09T17:41:31.507771+0000","last_fullsized":"2024-02-09T17:41:31.507771+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:40:44.545494+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:40:44.545494+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:08.678934+0000","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.1","version":"486'3802","reported_seq":3854,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-02-09T17:41:35.659999+0000","last_change":"2024-02-09T17:40:49.661020+0000","last_active":"2024-02-09T17:41:35.659999+0000","last_peered":"2024-02-09T17:41:35.659999+0000","last_clean":"2024-02-09T17:41:35.659999+0000","last_became_active":"2024-02-09T17:40:49.660617+0000","last_became_peered":"2024-02-09T17:40:49.660617+0000","last_unstale":"2024-02-09T17:41:35.659999+0000","last_undegraded":"2024-02-09T17:41:35.659999+0000","last_fullsized":"2024-02-09T17:41:35.659999+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:40:47.763277+0000","objects_scrubbed":0,"log_size":3802,"log_dups_size":0,"ondisk_log_size":3802,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:13.075794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249167872,"num_objects":3802,"num_object_clones":0,"num_object_copies":11406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3802,"num_write_kb":243328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"63'139","reported_seq":815,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-02-09T17:41:31.505423+0000","last_change":"2024-02-09T17:41:14.521510+0000","last_active":"2024-02-09T17:41:31.505423+0000","last_peered":"2024-02-09T17:41:31.505423+0000","last_clean":"2024-02-09T17:41:31.505423+0000","last_became_active":"2024-02-09T17:41:14.521098+0000","last_became_peered":"2024-02-09T17:41:14.521098+0000","last_unstale":"2024-02-09T17:41:31.505423+0000","last_undegraded":"2024-02-09T17:41:31.505423+0000","last_fullsized":"2024-02-09T17:41:31.505423+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:40:56.752599+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:40:56.752599+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:14.187085+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11],"acting":[10,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.e","version":"486'3840","reported_seq":3892,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-02-09T17:41:36.557410+0000","last_change":"2024-02-09T17:40:48.779687+0000","last_active":"2024-02-09T17:41:36.557410+0000","last_peered":"2024-02-09T17:41:36.557410+0000","last_clean":"2024-02-09T17:41:36.557410+0000","last_became_active":"2024-02-09T17:40:48.779366+0000","last_became_peered":"2024-02-09T17:40:48.779366+0000","last_unstale":"2024-02-09T17:41:36.557410+0000","last_undegraded":"2024-02-09T17:41:36.557410+0000","last_fullsized":"2024-02-09T17:41:36.557410+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:40:47.763277+0000","objects_scrubbed":0,"log_size":3840,"log_dups_size":0,"ondisk_log_size":3840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:41:52.402214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251658240,"num_objects":3840,"num_object_clones":0,"num_object_copies":11520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3840,"num_write_kb":245760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"63'139","reported_seq":764,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-02-09T17:41:31.505556+0000","last_change":"2024-02-09T17:39:42.324945+0000","last_active":"2024-02-09T17:41:31.505556+0000","last_peered":"2024-02-09T17:41:31.505556+0000","last_clean":"2024-02-09T17:41:31.505556+0000","last_became_active":"2024-02-09T17:32:35.418200+0000","last_became_peered":"2024-02-09T17:32:35.418200+0000","last_unstale":"2024-02-09T17:41:31.505556+0000","last_undegraded":"2024-02-09T17:41:31.505556+0000","last_fullsized":"2024-02-09T17:41:31.505556+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:39:42.324851+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:39:42.324851+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:41:35.593301+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.f","version":"486'4110","reported_seq":6536,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-02-09T17:41:39.071720+0000","last_change":"2024-02-09T17:41:20.878405+0000","last_active":"2024-02-09T17:41:39.071720+0000","last_peered":"2024-02-09T17:41:39.071720+0000","last_clean":"2024-02-09T17:41:39.071720+0000","last_became_active":"2024-02-09T17:41:20.876180+0000","last_became_peered":"2024-02-09T17:41:20.876180+0000","last_unstale":"2024-02-09T17:41:39.071720+0000","last_undegraded":"2024-02-09T17:41:39.071720+0000","last_fullsized":"2024-02-09T17:41:39.071720+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:40:47.763277+0000","objects_scrubbed":0,"log_size":4110,"log_dups_size":0,"ondisk_log_size":4110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:06.473578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269352960,"num_objects":4110,"num_object_clones":0,"num_object_copies":12330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4110,"num_write_kb":263040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1182,"num_bytes_recovered":77463552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,11],"acting":[5,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"63'139","reported_seq":606,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-02-09T17:41:31.507659+0000","last_change":"2024-02-09T17:41:14.522753+0000","last_active":"2024-02-09T17:41:31.507659+0000","last_peered":"2024-02-09T17:41:31.507659+0000","last_clean":"2024-02-09T17:41:31.507659+0000","last_became_active":"2024-02-09T17:41:14.522410+0000","last_became_peered":"2024-02-09T17:41:14.522410+0000","last_unstale":"2024-02-09T17:41:31.507659+0000","last_undegraded":"2024-02-09T17:41:31.507659+0000","last_fullsized":"2024-02-09T17:41:31.507659+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:40:42.968753+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:40:42.968753+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:41:45.066546+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"486'3873","reported_seq":3925,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-02-09T17:41:35.612578+0000","last_change":"2024-02-09T17:40:48.781690+0000","last_active":"2024-02-09T17:41:35.612578+0000","last_peered":"2024-02-09T17:41:35.612578+0000","last_clean":"2024-02-09T17:41:35.612578+0000","last_became_active":"2024-02-09T17:40:48.781384+0000","last_became_peered":"2024-02-09T17:40:48.781384+0000","last_unstale":"2024-02-09T17:41:35.612578+0000","last_undegraded":"2024-02-09T17:41:35.612578+0000","last_fullsized":"2024-02-09T17:41:35.612578+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:40:47.763277+0000","objects_scrubbed":0,"log_size":3873,"log_dups_size":0,"ondisk_log_size":3873,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:41:59.692496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":253820928,"num_objects":3873,"num_object_clones":0,"num_object_copies":11619,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3873,"num_write_kb":247872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"63'139","reported_seq":821,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-02-09T17:41:31.506553+0000","last_change":"2024-02-09T17:40:46.233294+0000","last_active":"2024-02-09T17:41:31.506553+0000","last_peered":"2024-02-09T17:41:31.506553+0000","last_clean":"2024-02-09T17:41:31.506553+0000","last_became_active":"2024-02-09T17:39:37.460615+0000","last_became_peered":"2024-02-09T17:39:37.460615+0000","last_unstale":"2024-02-09T17:41:31.506553+0000","last_undegraded":"2024-02-09T17:41:31.506553+0000","last_fullsized":"2024-02-09T17:41:31.506553+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:40:46.233161+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:40:46.233161+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:41:50.354159+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"486'3842","reported_seq":6395,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-02-09T17:41:35.300355+0000","last_change":"2024-02-09T17:41:21.502193+0000","last_active":"2024-02-09T17:41:35.300355+0000","last_peered":"2024-02-09T17:41:35.300355+0000","last_clean":"2024-02-09T17:41:35.300355+0000","last_became_active":"2024-02-09T17:41:21.499187+0000","last_became_peered":"2024-02-09T17:41:21.499187+0000","last_unstale":"2024-02-09T17:41:35.300355+0000","last_undegraded":"2024-02-09T17:41:35.300355+0000","last_fullsized":"2024-02-09T17:41:35.300355+0000","mapping_epoch":481,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:40:47.763277+0000","objects_scrubbed":0,"log_size":3842,"log_dups_size":0,"ondisk_log_size":3842,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:41:51.463528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251789312,"num_objects":3842,"num_object_clones":0,"num_object_copies":11526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3842,"num_write_kb":245888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1248,"num_bytes_recovered":81788928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"63'139","reported_seq":1267,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-02-09T17:41:31.507642+0000","last_change":"2024-02-09T17:40:57.958820+0000","last_active":"2024-02-09T17:41:31.507642+0000","last_peered":"2024-02-09T17:41:31.507642+0000","last_clean":"2024-02-09T17:41:31.507642+0000","last_became_active":"2024-02-09T17:39:17.456739+0000","last_became_peered":"2024-02-09T17:39:17.456739+0000","last_unstale":"2024-02-09T17:41:31.507642+0000","last_undegraded":"2024-02-09T17:41:31.507642+0000","last_fullsized":"2024-02-09T17:41:31.507642+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:40:57.958641+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:30.193194+0000","last_clean_scrub_stamp":"2024-02-09T17:40:57.958641+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:11.460211+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"486'3869","reported_seq":3921,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-02-09T17:41:36.910744+0000","last_change":"2024-02-09T17:40:48.781919+0000","last_active":"2024-02-09T17:41:36.910744+0000","last_peered":"2024-02-09T17:41:36.910744+0000","last_clean":"2024-02-09T17:41:36.910744+0000","last_became_active":"2024-02-09T17:40:48.781591+0000","last_became_peered":"2024-02-09T17:40:48.781591+0000","last_unstale":"2024-02-09T17:41:36.910744+0000","last_undegraded":"2024-02-09T17:41:36.910744+0000","last_fullsized":"2024-02-09T17:41:36.910744+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:40:47.763277+0000","objects_scrubbed":0,"log_size":3869,"log_dups_size":0,"ondisk_log_size":3869,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:06.277557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":253558784,"num_objects":3869,"num_object_clones":0,"num_object_copies":11607,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3869,"num_write_kb":247616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,4],"acting":[8,10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.e","version":"63'139","reported_seq":740,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-02-09T17:41:31.500213+0000","last_change":"2024-02-09T17:41:06.857849+0000","last_active":"2024-02-09T17:41:31.500213+0000","last_peered":"2024-02-09T17:41:31.500213+0000","last_clean":"2024-02-09T17:41:31.500213+0000","last_became_active":"2024-02-09T17:33:05.309824+0000","last_became_peered":"2024-02-09T17:33:05.309824+0000","last_unstale":"2024-02-09T17:41:31.500213+0000","last_undegraded":"2024-02-09T17:41:31.500213+0000","last_fullsized":"2024-02-09T17:41:31.500213+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:06.857716+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:41:06.857716+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:29.895394+0000","scrub_duration":190,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"486'3854","reported_seq":3906,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-02-09T17:41:36.561553+0000","last_change":"2024-02-09T17:40:48.783627+0000","last_active":"2024-02-09T17:41:36.561553+0000","last_peered":"2024-02-09T17:41:36.561553+0000","last_clean":"2024-02-09T17:41:36.561553+0000","last_became_active":"2024-02-09T17:40:48.783235+0000","last_became_peered":"2024-02-09T17:40:48.783235+0000","last_unstale":"2024-02-09T17:41:36.561553+0000","last_undegraded":"2024-02-09T17:41:36.561553+0000","last_fullsized":"2024-02-09T17:41:36.561553+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:40:47.763277+0000","objects_scrubbed":0,"log_size":3854,"log_dups_size":0,"ondisk_log_size":3854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:14.909299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252575744,"num_objects":3854,"num_object_clones":0,"num_object_copies":11562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3854,"num_write_kb":246656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.f","version":"63'139","reported_seq":799,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-02-09T17:41:31.506679+0000","last_change":"2024-02-09T17:41:02.126077+0000","last_active":"2024-02-09T17:41:31.506679+0000","last_peered":"2024-02-09T17:41:31.506679+0000","last_clean":"2024-02-09T17:41:31.506679+0000","last_became_active":"2024-02-09T17:33:05.316100+0000","last_became_peered":"2024-02-09T17:33:05.316100+0000","last_unstale":"2024-02-09T17:41:31.506679+0000","last_undegraded":"2024-02-09T17:41:31.506679+0000","last_fullsized":"2024-02-09T17:41:31.506679+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:02.125974+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:36:22.364130+0000","last_clean_scrub_stamp":"2024-02-09T17:41:02.125974+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:09.528604+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"486'3837","reported_seq":6289,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-02-09T17:41:35.310003+0000","last_change":"2024-02-09T17:41:20.879764+0000","last_active":"2024-02-09T17:41:35.310003+0000","last_peered":"2024-02-09T17:41:35.310003+0000","last_clean":"2024-02-09T17:41:35.310003+0000","last_became_active":"2024-02-09T17:41:20.876434+0000","last_became_peered":"2024-02-09T17:41:20.876434+0000","last_unstale":"2024-02-09T17:41:35.310003+0000","last_undegraded":"2024-02-09T17:41:35.310003+0000","last_fullsized":"2024-02-09T17:41:35.310003+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:40:47.763277+0000","objects_scrubbed":0,"log_size":3837,"log_dups_size":0,"ondisk_log_size":3837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:01.223195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251461632,"num_objects":3837,"num_object_clones":0,"num_object_copies":11511,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3837,"num_write_kb":245568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1195,"num_bytes_recovered":78315520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"63'139","reported_seq":919,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-02-09T17:41:31.517101+0000","last_change":"2024-02-09T17:40:03.690279+0000","last_active":"2024-02-09T17:41:31.517101+0000","last_peered":"2024-02-09T17:41:31.517101+0000","last_clean":"2024-02-09T17:41:31.517101+0000","last_became_active":"2024-02-09T17:39:37.471954+0000","last_became_peered":"2024-02-09T17:39:37.471954+0000","last_unstale":"2024-02-09T17:41:31.517101+0000","last_undegraded":"2024-02-09T17:41:31.517101+0000","last_fullsized":"2024-02-09T17:41:31.517101+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:40:03.690165+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:23.912504+0000","last_clean_scrub_stamp":"2024-02-09T17:40:03.690165+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:41:33.782192+0000","scrub_duration":118,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"486'3943","reported_seq":3995,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-02-09T17:41:37.167737+0000","last_change":"2024-02-09T17:40:49.436637+0000","last_active":"2024-02-09T17:41:37.167737+0000","last_peered":"2024-02-09T17:41:37.167737+0000","last_clean":"2024-02-09T17:41:37.167737+0000","last_became_active":"2024-02-09T17:40:49.436285+0000","last_became_peered":"2024-02-09T17:40:49.436285+0000","last_unstale":"2024-02-09T17:41:37.167737+0000","last_undegraded":"2024-02-09T17:41:37.167737+0000","last_fullsized":"2024-02-09T17:41:37.167737+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:40:47.763277+0000","objects_scrubbed":0,"log_size":3943,"log_dups_size":0,"ondisk_log_size":3943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:41:50.232026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":258408448,"num_objects":3943,"num_object_clones":0,"num_object_copies":11829,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3943,"num_write_kb":252352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"63'139","reported_seq":332,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-02-09T17:41:31.507729+0000","last_change":"2024-02-09T17:40:44.903454+0000","last_active":"2024-02-09T17:41:31.507729+0000","last_peered":"2024-02-09T17:41:31.507729+0000","last_clean":"2024-02-09T17:41:31.507729+0000","last_became_active":"2024-02-09T17:40:22.485257+0000","last_became_peered":"2024-02-09T17:40:22.485257+0000","last_unstale":"2024-02-09T17:41:31.507729+0000","last_undegraded":"2024-02-09T17:41:31.507729+0000","last_fullsized":"2024-02-09T17:41:31.507729+0000","mapping_epoch":454,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":455,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:40:44.903308+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:40:44.903308+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:13.814742+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"63'139","reported_seq":748,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-02-09T17:41:31.506448+0000","last_change":"2024-02-09T17:41:09.323335+0000","last_active":"2024-02-09T17:41:31.506448+0000","last_peered":"2024-02-09T17:41:31.506448+0000","last_clean":"2024-02-09T17:41:31.506448+0000","last_became_active":"2024-02-09T17:33:05.310104+0000","last_became_peered":"2024-02-09T17:33:05.310104+0000","last_unstale":"2024-02-09T17:41:31.506448+0000","last_undegraded":"2024-02-09T17:41:31.506448+0000","last_fullsized":"2024-02-09T17:41:31.506448+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:09.323236+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:41:09.323236+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:13.566799+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"63'139","reported_seq":755,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-02-09T17:41:31.510568+0000","last_change":"2024-02-09T17:41:06.851766+0000","last_active":"2024-02-09T17:41:31.510568+0000","last_peered":"2024-02-09T17:41:31.510568+0000","last_clean":"2024-02-09T17:41:31.510568+0000","last_became_active":"2024-02-09T17:34:01.353171+0000","last_became_peered":"2024-02-09T17:34:01.353171+0000","last_unstale":"2024-02-09T17:41:31.510568+0000","last_undegraded":"2024-02-09T17:41:31.510568+0000","last_fullsized":"2024-02-09T17:41:31.510568+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:06.851643+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:41:06.851643+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:34.858536+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.12","version":"63'139","reported_seq":1267,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-02-09T17:41:31.506669+0000","last_change":"2024-02-09T17:40:44.208044+0000","last_active":"2024-02-09T17:41:31.506669+0000","last_peered":"2024-02-09T17:41:31.506669+0000","last_clean":"2024-02-09T17:41:31.506669+0000","last_became_active":"2024-02-09T17:40:13.499071+0000","last_became_peered":"2024-02-09T17:40:13.499071+0000","last_unstale":"2024-02-09T17:41:31.506669+0000","last_undegraded":"2024-02-09T17:41:31.506669+0000","last_fullsized":"2024-02-09T17:41:31.506669+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:40:44.207923+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:40:44.207923+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:13.625252+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"63'139","reported_seq":466,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-02-09T17:41:31.503856+0000","last_change":"2024-02-09T17:41:14.521999+0000","last_active":"2024-02-09T17:41:31.503856+0000","last_peered":"2024-02-09T17:41:31.503856+0000","last_clean":"2024-02-09T17:41:31.503856+0000","last_became_active":"2024-02-09T17:41:14.521672+0000","last_became_peered":"2024-02-09T17:41:14.521672+0000","last_unstale":"2024-02-09T17:41:31.503856+0000","last_undegraded":"2024-02-09T17:41:31.503856+0000","last_fullsized":"2024-02-09T17:41:31.503856+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:03.023657+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:41:03.023657+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:28.356260+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":4115070976,"num_objects":62791,"num_object_clones":0,"num_object_copies":188373,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62791,"num_write_kb":4018624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3625,"num_bytes_recovered":237568000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12480937984,"data_stored":12480937984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62791,"ondisk_log_size":62791,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":931424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2919,"ondisk_log_size":2919,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056871,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1466596,"kb_used_data":970332,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":92278556,"statfs":{"total":95995035648,"available":94493241344,"internally_reserved":0,"allocated":993619968,"data_stored":993092276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":508164548},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:41:25 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":1.6399999999999999,"15min":1.359},"min":{"1min":0.33100000000000002,"5min":0.246,"15min":0.216},"max":{"1min":1.4410000000000001,"5min":9.5709999999999997,"15min":14.913},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":1.369,"15min":1.3720000000000001},"min":{"1min":0.33800000000000002,"5min":0.27000000000000002,"15min":0.23000000000000001},"max":{"1min":1.849,"5min":9.8219999999999992,"15min":14.869},"last":0.372}]},{"osd":1,"last update":"Fri Feb 9 17:41:25 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":1.6739999999999999,"15min":1.6850000000000001},"min":{"1min":0.45300000000000001,"5min":0.24299999999999999,"15min":0.22700000000000001},"max":{"1min":1.7070000000000001,"5min":9.7710000000000008,"15min":32.957999999999998},"last":1.546},{"interface":"front","average":{"1min":0.58999999999999997,"5min":1.8049999999999999,"15min":1.8320000000000001},"min":{"1min":0.29099999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.97299999999999998,"5min":9.718,"15min":32.880000000000003},"last":0.64800000000000002}]},{"osd":2,"last update":"Fri Feb 9 17:31:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.415,"15min":1.6859999999999999},"min":{"1min":0.378,"5min":0.247,"15min":0.247},"max":{"1min":4.4189999999999996,"5min":9.4719999999999995,"15min":9.4719999999999995},"last":0.35399999999999998},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.5149999999999999,"15min":1.4399999999999999},"min":{"1min":0.378,"5min":0.252,"15min":0.252},"max":{"1min":4.8579999999999997,"5min":7.96,"15min":7.96},"last":0.60399999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:41:25 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":1.355,"15min":1.659},"min":{"1min":0.28699999999999998,"5min":0.27100000000000002,"15min":0.247},"max":{"1min":1.7629999999999999,"5min":9.5299999999999994,"15min":21.315999999999999},"last":1.621},{"interface":"front","average":{"1min":0.66600000000000004,"5min":1.869,"15min":1.752},"min":{"1min":0.40300000000000002,"5min":0.29799999999999999,"15min":0.26400000000000001},"max":{"1min":1.7929999999999999,"5min":12.706,"15min":18.087},"last":1.7010000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:41:25 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":1.1919999999999999,"15min":1.4750000000000001},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.25600000000000001},"max":{"1min":1.522,"5min":21.030999999999999,"15min":21.030999999999999},"last":1.7689999999999999},{"interface":"front","average":{"1min":0.76300000000000001,"5min":1.2669999999999999,"15min":1.3180000000000001},"min":{"1min":0.40200000000000002,"5min":0.309,"15min":0.27700000000000002},"max":{"1min":1.5109999999999999,"5min":20.876999999999999,"15min":20.876999999999999},"last":1.7869999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:41:25 2024","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":1.0640000000000001,"15min":1.345},"min":{"1min":0.441,"5min":0.316,"15min":0.28899999999999998},"max":{"1min":1.663,"5min":6.4630000000000001,"15min":13.946999999999999},"last":1.6479999999999999},{"interface":"front","average":{"1min":0.65800000000000003,"5min":1.169,"15min":1.4359999999999999},"min":{"1min":0.378,"5min":0.311,"15min":0.30199999999999999},"max":{"1min":1.1519999999999999,"5min":7.1349999999999998,"15min":13.414},"last":0.47199999999999998}]},{"osd":6,"last update":"Fri Feb 9 17:41:25 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":1.3200000000000001,"15min":1.421},"min":{"1min":0.48899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.74,"5min":9.4779999999999998,"15min":12.798999999999999},"last":1.6739999999999999},{"interface":"front","average":{"1min":0.74099999999999999,"5min":1.212,"15min":1.411},"min":{"1min":0.38500000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.474,"5min":9.2420000000000009,"15min":10.782999999999999},"last":4.4829999999999997}]},{"osd":7,"last update":"Fri Feb 9 17:41:25 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":1.375,"15min":1.615},"min":{"1min":0.503,"5min":0.38,"15min":0.312},"max":{"1min":1.409,"5min":20.489999999999998,"15min":20.489999999999998},"last":0.629},{"interface":"front","average":{"1min":0.877,"5min":1.302,"15min":1.585},"min":{"1min":0.52800000000000002,"5min":0.35299999999999998,"15min":0.34200000000000003},"max":{"1min":1.841,"5min":20.712,"15min":20.712},"last":1.73}]},{"osd":8,"last update":"Fri Feb 9 17:40:52 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":1.0680000000000001,"15min":1.7569999999999999},"min":{"1min":0.34300000000000003,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":3.5619999999999998,"5min":9.2680000000000007,"15min":13.199999999999999},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.79400000000000004,"5min":1.0760000000000001,"15min":1.4950000000000001},"min":{"1min":0.443,"5min":0.34300000000000003,"15min":0.318},"max":{"1min":3.6539999999999999,"5min":9.2940000000000005,"15min":9.2940000000000005},"last":0.77700000000000002}]},{"osd":9,"last update":"Fri Feb 9 17:41:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":1.79,"15min":1.4259999999999999},"min":{"1min":0.36799999999999999,"5min":0.36499999999999999,"15min":0.29399999999999998},"max":{"1min":1.5629999999999999,"5min":62.518000000000001,"15min":62.518000000000001},"last":1.587},{"interface":"front","average":{"1min":0.84299999999999997,"5min":1.8480000000000001,"15min":1.369},"min":{"1min":0.46300000000000002,"5min":0.379,"15min":0.34100000000000003},"max":{"1min":3.8159999999999998,"5min":63.241999999999997,"15min":63.241999999999997},"last":0.66200000000000003}]},{"osd":10,"last update":"Fri Feb 9 17:41:25 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":1.1990000000000001,"15min":1.196},"min":{"1min":0.47199999999999998,"5min":0.36799999999999999,"15min":0.33900000000000002},"max":{"1min":1.214,"5min":9.3109999999999999,"15min":9.6199999999999992},"last":0.747},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.306,"15min":1.2749999999999999},"min":{"1min":0.44700000000000001,"5min":0.39100000000000001,"15min":0.33700000000000002},"max":{"1min":4.0279999999999996,"5min":9.593,"15min":12.856},"last":2.5419999999999998}]}]},{"osd":10,"up_from":84,"seq":360777253101,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1825508,"kb_used_data":1523612,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":91919644,"statfs":{"total":95995035648,"available":94125715456,"internally_reserved":0,"allocated":1560178688,"data_stored":1559650996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":309131712},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:41:14 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":1.54,"15min":1.7050000000000001},"min":{"1min":0.25700000000000001,"5min":0.222,"15min":0.222},"max":{"1min":1.6559999999999999,"5min":9.343,"15min":9.343},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.58899999999999997,"5min":1.365,"15min":1.177},"min":{"1min":0.27600000000000002,"5min":0.26500000000000001,"15min":0.253},"max":{"1min":1.484,"5min":6.3399999999999999,"15min":6.3399999999999999},"last":0.71299999999999997}]},{"osd":1,"last update":"Fri Feb 9 17:39:30 2024","interfaces":[{"interface":"back","average":{"1min":1.804,"5min":1.804,"15min":1.804},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":7.673,"5min":7.673,"15min":7.673},"last":0.78400000000000003},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":3.9649999999999999,"5min":3.9649999999999999,"15min":3.9649999999999999},"last":0.42799999999999999}]},{"osd":2,"last update":"Fri Feb 9 17:40:57 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":1.8600000000000001,"15min":1.554},"min":{"1min":0.28999999999999998,"5min":0.28499999999999998,"15min":0.222},"max":{"1min":2.177,"5min":11.384,"15min":13.351000000000001},"last":0.58299999999999996},{"interface":"front","average":{"1min":1.129,"5min":1.474,"15min":1.444},"min":{"1min":0.42799999999999999,"5min":0.27200000000000002,"15min":0.26100000000000001},"max":{"1min":9.0850000000000009,"5min":11.465,"15min":11.465},"last":0.67200000000000004}]},{"osd":3,"last update":"Fri Feb 9 17:40:39 2024","interfaces":[{"interface":"back","average":{"1min":2.048,"5min":2.048,"15min":2.048},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":9.1649999999999991,"5min":9.1649999999999991,"15min":9.1649999999999991},"last":0.60499999999999998},{"interface":"front","average":{"1min":2.1059999999999999,"5min":2.1059999999999999,"15min":2.1059999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":8.9670000000000005,"5min":8.9670000000000005,"15min":8.9670000000000005},"last":0.72399999999999998}]},{"osd":4,"last update":"Fri Feb 9 17:41:05 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.90000000000000002,"15min":1.071},"min":{"1min":0.314,"5min":0.29199999999999998,"15min":0.26100000000000001},"max":{"1min":1.0609999999999999,"5min":5.8840000000000003,"15min":7.1360000000000001},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.86899999999999999,"15min":1.0720000000000001},"min":{"1min":0.41999999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.038,"5min":4.6139999999999999,"15min":6.8369999999999997},"last":0.79600000000000004}]},{"osd":5,"last update":"Fri Feb 9 17:40:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":1.012,"15min":1.2789999999999999},"min":{"1min":0.33600000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.92100000000000004,"5min":7.5179999999999998,"15min":9.2349999999999994},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.98299999999999998,"15min":1.22},"min":{"1min":0.44900000000000001,"5min":0.33500000000000002,"15min":0.32200000000000001},"max":{"1min":4.181,"5min":6.8940000000000001,"15min":8.6159999999999997},"last":0.82599999999999996}]},{"osd":6,"last update":"Fri Feb 9 17:40:57 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":1.244,"15min":1.268},"min":{"1min":0.32400000000000001,"5min":0.308,"15min":0.29099999999999998},"max":{"1min":0.85499999999999998,"5min":8.6869999999999994,"15min":8.6869999999999994},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.86799999999999999,"5min":1.0329999999999999,"15min":1.391},"min":{"1min":0.41099999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":2.339,"5min":4.9720000000000004,"15min":9.4350000000000005},"last":0.84999999999999998}]},{"osd":7,"last update":"Fri Feb 9 17:39:52 2024","interfaces":[{"interface":"back","average":{"1min":1.6000000000000001,"5min":1.367,"15min":1.415},"min":{"1min":0.42099999999999999,"5min":0.34200000000000003,"15min":0.312},"max":{"1min":4.0430000000000001,"5min":6.7590000000000003,"15min":17.754000000000001},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.498,"5min":1.4359999999999999,"15min":1.5189999999999999},"min":{"1min":0.47399999999999998,"5min":0.38300000000000001,"15min":0.316},"max":{"1min":5.8170000000000002,"5min":11.308999999999999,"15min":11.308999999999999},"last":0.78800000000000003}]},{"osd":8,"last update":"Fri Feb 9 17:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":1.095,"15min":1.1639999999999999},"min":{"1min":0.35499999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":3.3679999999999999,"5min":7.4359999999999999,"15min":7.4359999999999999},"last":0.624},{"interface":"front","average":{"1min":0.96899999999999997,"5min":1.0309999999999999,"15min":0.97399999999999998},"min":{"1min":0.39400000000000002,"5min":0.37,"15min":0.32900000000000001},"max":{"1min":4.0389999999999997,"5min":7.7060000000000004,"15min":7.7060000000000004},"last":0.69599999999999995}]},{"osd":9,"last update":"Fri Feb 9 17:40:57 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":1.2130000000000001,"15min":1.103},"min":{"1min":0.377,"5min":0.36399999999999999,"15min":0.314},"max":{"1min":1.093,"5min":7.4850000000000003,"15min":8.7240000000000002},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.216,"15min":1.2230000000000001},"min":{"1min":0.441,"5min":0.40300000000000002,"15min":0.29799999999999999},"max":{"1min":1.371,"5min":7.891,"15min":10.1},"last":0.65300000000000002}]},{"osd":11,"last update":"Fri Feb 9 17:41:21 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.89600000000000002,"15min":1.113},"min":{"1min":0.45800000000000002,"5min":0.40600000000000003,"15min":0.36399999999999999},"max":{"1min":0.98399999999999999,"5min":4.2560000000000002,"15min":8.6430000000000007},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.95199999999999996,"15min":1.1299999999999999},"min":{"1min":0.60199999999999998,"5min":0.45700000000000002,"15min":0.33400000000000002},"max":{"1min":1.0269999999999999,"5min":4.2800000000000002,"15min":9.298},"last":0.73799999999999999}]}]},{"osd":9,"up_from":77,"seq":330712482034,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1134052,"kb_used_data":723868,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":92611100,"statfs":{"total":95995035648,"available":94833766400,"internally_reserved":0,"allocated":741240832,"data_stored":740713140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":420018624},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:28:03 2024","interfaces":[{"interface":"back","average":{"1min":1.9510000000000001,"5min":1.454,"15min":0.85799999999999998},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":8.4499999999999993,"5min":17.495999999999999,"15min":17.495999999999999},"last":1.5860000000000001},{"interface":"front","average":{"1min":1.905,"5min":1.399,"15min":0.83499999999999996},"min":{"1min":0.32200000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":7.5990000000000002,"5min":15.401999999999999,"15min":15.401999999999999},"last":1.0760000000000001}]},{"osd":1,"last update":"Fri Feb 9 17:40:35 2024","interfaces":[{"interface":"back","average":{"1min":2.0779999999999998,"5min":2.802,"15min":1.931},"min":{"1min":0.29599999999999999,"5min":0.23499999999999999,"15min":0.23100000000000001},"max":{"1min":6.9409999999999998,"5min":89.515000000000001,"15min":89.515000000000001},"last":0.44400000000000001},{"interface":"front","average":{"1min":2.2749999999999999,"5min":2.9540000000000002,"15min":1.851},"min":{"1min":0.249,"5min":0.249,"15min":0.222},"max":{"1min":11.598000000000001,"5min":89.388000000000005,"15min":89.388000000000005},"last":0.39000000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:40:39 2024","interfaces":[{"interface":"back","average":{"1min":1.742,"5min":1.766,"15min":1.514},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":5.6630000000000003,"5min":14.121,"15min":14.121},"last":0.63400000000000001},{"interface":"front","average":{"1min":1.6910000000000001,"5min":1.52,"15min":1.4870000000000001},"min":{"1min":0.34100000000000003,"5min":0.26100000000000001,"15min":0.25600000000000001},"max":{"1min":5.7030000000000003,"5min":8.8040000000000003,"15min":13.69},"last":0.73899999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:40:35 2024","interfaces":[{"interface":"back","average":{"1min":2.5550000000000002,"5min":2.1259999999999999,"15min":1.6990000000000001},"min":{"1min":0.38100000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":9.6959999999999997,"5min":15.414999999999999,"15min":40.18},"last":0.57899999999999996},{"interface":"front","average":{"1min":2.2810000000000001,"5min":2.1890000000000001,"15min":1.915},"min":{"1min":0.433,"5min":0.317,"15min":0.27900000000000003},"max":{"1min":9.7140000000000004,"5min":15.372,"15min":40.158000000000001},"last":0.32000000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:40:35 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":1.2470000000000001,"15min":1.4399999999999999},"min":{"1min":0.309,"5min":0.29599999999999999,"15min":0.28399999999999997},"max":{"1min":2.0499999999999998,"5min":27.381,"15min":27.381},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":1.3939999999999999,"15min":1.371},"min":{"1min":0.313,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.651,"5min":27.355,"15min":27.355},"last":0.42699999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":1.1870000000000001,"15min":1.3939999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.613,"5min":8.6110000000000007,"15min":10.689},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.96999999999999997,"5min":1.131,"15min":1.179},"min":{"1min":0.42399999999999999,"5min":0.32200000000000001,"15min":0.311},"max":{"1min":5.5090000000000003,"5min":13.978999999999999,"15min":13.978999999999999},"last":0.89700000000000002}]},{"osd":6,"last update":"Fri Feb 9 17:40:35 2024","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.3169999999999999,"15min":1.452},"min":{"1min":0.40899999999999997,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":5.4640000000000004,"5min":14.069000000000001,"15min":14.069000000000001},"last":0.93300000000000005},{"interface":"front","average":{"1min":1.421,"5min":1.3340000000000001,"15min":1.446},"min":{"1min":0.47899999999999998,"5min":0.37,"15min":0.35099999999999998},"max":{"1min":5.6079999999999997,"5min":14.097,"15min":14.097},"last":0.95899999999999996}]},{"osd":7,"last update":"Fri Feb 9 17:40:35 2024","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.306,"15min":1.395},"min":{"1min":0.40200000000000002,"5min":0.34499999999999997,"15min":0.34300000000000003},"max":{"1min":4.8159999999999998,"5min":7.4800000000000004,"15min":8.0939999999999994},"last":0.55000000000000004},{"interface":"front","average":{"1min":1.135,"5min":1.355,"15min":1.4890000000000001},"min":{"1min":0.433,"5min":0.38400000000000001,"15min":0.32200000000000001},"max":{"1min":4.8799999999999999,"5min":7.5039999999999996,"15min":8.0530000000000008},"last":0.60199999999999998}]},{"osd":8,"last update":"Fri Feb 9 17:40:54 2024","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":1.3460000000000001,"15min":1.286},"min":{"1min":0.35799999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":5.1440000000000001,"5min":14.045999999999999,"15min":16.209},"last":1.623},{"interface":"front","average":{"1min":1.085,"5min":1.353,"15min":1.2949999999999999},"min":{"1min":0.41799999999999998,"5min":0.41299999999999998,"15min":0.34799999999999998},"max":{"1min":5.548,"5min":5.548,"15min":16.263000000000002},"last":1.425}]},{"osd":10,"last update":"Fri Feb 9 17:41:05 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":1.2010000000000001,"15min":1.198},"min":{"1min":0.373,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":2.254,"5min":7.4089999999999998,"15min":9.5090000000000003},"last":0.62},{"interface":"front","average":{"1min":0.71299999999999997,"5min":1.377,"15min":1.2030000000000001},"min":{"1min":0.437,"5min":0.39500000000000002,"15min":0.38200000000000001},"max":{"1min":1.2210000000000001,"5min":9.7569999999999997,"15min":9.7569999999999997},"last":0.47099999999999997}]},{"osd":11,"last update":"Fri Feb 9 17:41:05 2024","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":1.542,"15min":1.4279999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.34699999999999998},"max":{"1min":5.6349999999999998,"5min":14.02,"15min":14.02},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.76100000000000001,"5min":1.357,"15min":1.2350000000000001},"min":{"1min":0.46800000000000003,"5min":0.39400000000000002,"15min":0.374},"max":{"1min":1.25,"5min":7.29,"15min":10.32},"last":0.503}]}]},{"osd":8,"up_from":171,"seq":734439407777,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":945708,"kb_used_data":737892,"kb_used_omap":212,"kb_used_meta":207595,"kb_avail":92799444,"statfs":{"total":95995035648,"available":95026630656,"internally_reserved":0,"allocated":755601408,"data_stored":755070164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217579,"internal_metadata":212577813},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:40:38 2024","interfaces":[{"interface":"back","average":{"1min":1.5529999999999999,"5min":1.1950000000000001,"15min":1.228},"min":{"1min":0.28599999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":5.827,"5min":8.0920000000000005,"15min":8.0920000000000005},"last":0.28000000000000003},{"interface":"front","average":{"1min":2.1110000000000002,"5min":1.48,"15min":1.3620000000000001},"min":{"1min":0.38600000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":7.2949999999999999,"5min":10.256,"15min":10.256},"last":0.30399999999999999}]},{"osd":1,"last update":"Fri Feb 9 17:40:38 2024","interfaces":[{"interface":"back","average":{"1min":1.7609999999999999,"5min":1.6080000000000001,"15min":1.351},"min":{"1min":0.26400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":6.351,"5min":7.867,"15min":7.8719999999999999},"last":0.377},{"interface":"front","average":{"1min":3.1859999999999999,"5min":2.105,"15min":1.8069999999999999},"min":{"1min":0.318,"5min":0.23400000000000001,"15min":0.23300000000000001},"max":{"1min":25.027000000000001,"5min":25.027000000000001,"15min":25.027000000000001},"last":3.2370000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:40:38 2024","interfaces":[{"interface":"back","average":{"1min":1.512,"5min":1.536,"15min":1.494},"min":{"1min":0.33300000000000002,"5min":0.27500000000000002,"15min":0.255},"max":{"1min":5.891,"5min":8.5649999999999995,"15min":10.423},"last":0.45200000000000001},{"interface":"front","average":{"1min":1.5649999999999999,"5min":1.5840000000000001,"15min":1.552},"min":{"1min":0.36599999999999999,"5min":0.35899999999999999,"15min":0.313},"max":{"1min":5.9470000000000001,"5min":8.5980000000000008,"15min":10.504},"last":0.502}]},{"osd":3,"last update":"Fri Feb 9 17:40:38 2024","interfaces":[{"interface":"back","average":{"1min":2.0870000000000002,"5min":1.7809999999999999,"15min":1.885},"min":{"1min":0.39100000000000001,"5min":0.29499999999999998,"15min":0.28699999999999998},"max":{"1min":8.9060000000000006,"5min":15.228999999999999,"15min":24.030999999999999},"last":3.2799999999999998},{"interface":"front","average":{"1min":2.621,"5min":2.2149999999999999,"15min":2.173},"min":{"1min":0.51100000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":8.8710000000000004,"5min":14.988,"15min":24.318999999999999},"last":3.343}]},{"osd":4,"last update":"Fri Feb 9 17:40:38 2024","interfaces":[{"interface":"back","average":{"1min":1.583,"5min":1.2949999999999999,"15min":1.413},"min":{"1min":0.33400000000000002,"5min":0.28199999999999997,"15min":0.27500000000000002},"max":{"1min":7.383,"5min":12.936,"15min":53.060000000000002},"last":3.262},{"interface":"front","average":{"1min":1.841,"5min":1.1279999999999999,"15min":1.5720000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":7.46,"5min":7.46,"15min":53.311999999999998},"last":3.2959999999999998}]},{"osd":6,"last update":"Fri Feb 9 17:40:38 2024","interfaces":[{"interface":"back","average":{"1min":1.774,"5min":1.147,"15min":1.5589999999999999},"min":{"1min":0.39100000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":7.4870000000000001,"5min":7.4870000000000001,"15min":37.356999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.204,"15min":1.2649999999999999},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":5.9930000000000003,"5min":7.702,"15min":37.25},"last":0.52600000000000002}]},{"osd":7,"last update":"Fri Feb 9 17:40:38 2024","interfaces":[{"interface":"back","average":{"1min":1.859,"5min":1.25,"15min":1.55},"min":{"1min":0.48199999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":8.5120000000000005,"5min":8.5120000000000005,"15min":37.585000000000001},"last":3.3119999999999998},{"interface":"front","average":{"1min":1.9119999999999999,"5min":1.2889999999999999,"15min":1.649},"min":{"1min":0.51500000000000001,"5min":0.38500000000000001,"15min":0.312},"max":{"1min":8.548,"5min":8.548,"15min":37.918999999999997},"last":3.327}]},{"osd":9,"last update":"Fri Feb 9 17:40:38 2024","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.98299999999999998,"15min":1.105},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.29699999999999999},"max":{"1min":3.5099999999999998,"5min":5.3330000000000002,"15min":9.4900000000000002},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.474,"5min":1.24,"15min":1.2270000000000001},"min":{"1min":0.40000000000000002,"5min":0.36599999999999999,"15min":0.32000000000000001},"max":{"1min":7.4189999999999996,"5min":7.4189999999999996,"15min":17.434999999999999},"last":1.286}]},{"osd":10,"last update":"Fri Feb 9 17:40:38 2024","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":0.92200000000000004,"15min":1.099},"min":{"1min":0.375,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":4.5620000000000003,"5min":4.8170000000000002,"15min":10.65},"last":0.56799999999999995},{"interface":"front","average":{"1min":1.407,"5min":1.254,"15min":1.1379999999999999},"min":{"1min":0.34799999999999998,"5min":0.318,"15min":0.318},"max":{"1min":6.694,"5min":11.071,"15min":11.071},"last":0.55400000000000005}]},{"osd":11,"last update":"Fri Feb 9 17:40:38 2024","interfaces":[{"interface":"back","average":{"1min":1.542,"5min":1.292,"15min":1.3779999999999999},"min":{"1min":0.38800000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":6.734,"5min":8.5519999999999996,"15min":14.433999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":1.5760000000000001,"5min":1.2749999999999999,"15min":1.363},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.38100000000000001},"max":{"1min":7.4359999999999999,"5min":7.4359999999999999,"15min":11.622},"last":3.1800000000000002}]}]},{"osd":7,"up_from":57,"seq":244813136123,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1535140,"kb_used_data":1026396,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":92210012,"statfs":{"total":95995035648,"available":94423052288,"internally_reserved":0,"allocated":1051029504,"data_stored":1050501812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":520944074},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:41:21 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":1.4139999999999999,"15min":2.7189999999999999},"min":{"1min":0.41299999999999998,"5min":0.252,"15min":0.252},"max":{"1min":3.472,"5min":14.348000000000001,"15min":36.753},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.69099999999999995,"5min":1.4359999999999999,"15min":2.8719999999999999},"min":{"1min":0.31900000000000001,"5min":0.29099999999999998,"15min":0.253},"max":{"1min":1.2290000000000001,"5min":9.0640000000000001,"15min":36.918999999999997},"last":2.4199999999999999}]},{"osd":1,"last update":"Fri Feb 9 17:41:21 2024","interfaces":[{"interface":"back","average":{"1min":1.6060000000000001,"5min":2.242,"15min":2.0790000000000002},"min":{"1min":0.50700000000000001,"5min":0.249,"15min":0.249},"max":{"1min":15.509,"5min":21.704999999999998,"15min":21.704999999999998},"last":0.50900000000000001},{"interface":"front","average":{"1min":1.5469999999999999,"5min":2.2280000000000002,"15min":1.8300000000000001},"min":{"1min":0.38100000000000001,"5min":0.26600000000000001,"15min":0.26500000000000001},"max":{"1min":15.276,"5min":21.858000000000001,"15min":21.858000000000001},"last":0.53300000000000003}]},{"osd":2,"last update":"Fri Feb 9 17:39:34 2024","interfaces":[{"interface":"back","average":{"1min":1.8520000000000001,"5min":1.7470000000000001,"15min":1.7030000000000001},"min":{"1min":0.315,"5min":0.28499999999999998,"15min":0.26000000000000001},"max":{"1min":6.6470000000000002,"5min":10.622,"15min":11.324},"last":0.76900000000000002},{"interface":"front","average":{"1min":1.8859999999999999,"5min":1.855,"15min":1.7769999999999999},"min":{"1min":0.34699999999999998,"5min":0.32100000000000001,"15min":0.28299999999999997},"max":{"1min":6.6870000000000003,"5min":19.361999999999998,"15min":19.361999999999998},"last":0.82299999999999995}]},{"osd":3,"last update":"Fri Feb 9 17:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":1.6539999999999999,"15min":1.867},"min":{"1min":0.35899999999999999,"5min":0.307,"15min":0.26900000000000002},"max":{"1min":2.4510000000000001,"5min":7.3899999999999997,"15min":27.719000000000001},"last":2.4510000000000001},{"interface":"front","average":{"1min":0.84599999999999997,"5min":1.627,"15min":1.8600000000000001},"min":{"1min":0.41999999999999998,"5min":0.308,"15min":0.26500000000000001},"max":{"1min":1.861,"5min":11.945,"15min":24.457000000000001},"last":0.56100000000000005}]},{"osd":4,"last update":"Fri Feb 9 17:41:21 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.373},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":1.448,"5min":1.448,"15min":1.448},"last":2.3799999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:41:21 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.64900000000000002,"15min":0.64500000000000002},"min":{"1min":0.37,"5min":0.249,"15min":0.249},"max":{"1min":1.5209999999999999,"5min":3.0539999999999998,"15min":3.0539999999999998},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.56299999999999994,"15min":0.52600000000000002},"min":{"1min":0.39700000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.61599999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":1.091,"15min":1.5229999999999999},"min":{"1min":0.36599999999999999,"5min":0.28100000000000003,"15min":0.26000000000000001},"max":{"1min":2.4830000000000001,"5min":8.1929999999999996,"15min":25.965},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.84999999999999998,"5min":1.083,"15min":1.3029999999999999},"min":{"1min":0.44600000000000001,"5min":0.32600000000000001,"15min":0.28699999999999998},"max":{"1min":2.0659999999999998,"5min":7.8810000000000002,"15min":9.4090000000000007},"last":0.44600000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:41:37 2024","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.1140000000000001,"15min":1.383},"min":{"1min":0.52900000000000003,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":2.7349999999999999,"5min":6.4420000000000002,"15min":9.0809999999999995},"last":2.5129999999999999},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.0309999999999999,"15min":1.28},"min":{"1min":0.48599999999999999,"5min":0.36599999999999999,"15min":0.33300000000000002},"max":{"1min":2.5760000000000001,"5min":6.0339999999999998,"15min":11.481},"last":0.64100000000000001}]},{"osd":9,"last update":"Fri Feb 9 17:41:28 2024","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":1.226,"15min":1.5309999999999999},"min":{"1min":0.47799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":2.7850000000000001,"5min":6.4749999999999996,"15min":41.286999999999999},"last":2.544},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.286,"15min":1.5920000000000001},"min":{"1min":0.56000000000000005,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":4.8440000000000003,"5min":6.5359999999999996,"15min":41.234999999999999},"last":2.8410000000000002}]},{"osd":10,"last update":"Fri Feb 9 17:40:51 2024","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":1.179,"15min":1.419},"min":{"1min":0.40100000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":5.0869999999999997,"5min":12.050000000000001,"15min":12.050000000000001},"last":1.7949999999999999},{"interface":"front","average":{"1min":0.93300000000000005,"5min":1.2050000000000001,"15min":1.502},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.35599999999999998},"max":{"1min":3.7599999999999998,"5min":6.5129999999999999,"15min":9.109},"last":2.4830000000000001}]},{"osd":11,"last update":"Fri Feb 9 17:41:21 2024","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":1.528,"15min":1.7010000000000001},"min":{"1min":0.46100000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":2.0670000000000002,"5min":21.945,"15min":21.945},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.96899999999999997,"5min":1.5940000000000001,"15min":1.5469999999999999},"min":{"1min":0.502,"5min":0.40699999999999997,"15min":0.39500000000000002},"max":{"1min":2.3050000000000002,"5min":21.731999999999999,"15min":21.731999999999999},"last":1.2170000000000001}]}]},{"osd":6,"up_from":49,"seq":210453397760,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1097252,"kb_used_data":723164,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":92647900,"statfs":{"total":95995035648,"available":94871449600,"internally_reserved":0,"allocated":740519936,"data_stored":739992244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":383056326},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:41:09 2024","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":1.593,"15min":1.6020000000000001},"min":{"1min":0.33200000000000002,"5min":0.24199999999999999,"15min":0.21299999999999999},"max":{"1min":5.6989999999999998,"5min":9.5739999999999998,"15min":12.544},"last":0.33800000000000002},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.5820000000000001,"15min":1.5309999999999999},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.217},"max":{"1min":5.4349999999999996,"5min":9.2520000000000007,"15min":9.2520000000000007},"last":0.27800000000000002}]},{"osd":1,"last update":"Fri Feb 9 17:41:09 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":1.9379999999999999,"15min":1.859},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.25900000000000001},"max":{"1min":3.6560000000000001,"5min":8.9689999999999994,"15min":17.762},"last":0.44},{"interface":"front","average":{"1min":0.79500000000000004,"5min":1.984,"15min":1.9139999999999999},"min":{"1min":0.41799999999999998,"5min":0.34499999999999997,"15min":0.29899999999999999},"max":{"1min":3.6909999999999998,"5min":9.0090000000000003,"15min":17.734999999999999},"last":0.46100000000000002}]},{"osd":2,"last update":"Fri Feb 9 17:41:09 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":1.484,"15min":1.6279999999999999},"min":{"1min":0.32500000000000001,"5min":0.29399999999999998,"15min":0.22600000000000001},"max":{"1min":5.601,"5min":9.8019999999999996,"15min":10.682},"last":0.495},{"interface":"front","average":{"1min":0.89500000000000002,"5min":1.5329999999999999,"15min":1.6779999999999999},"min":{"1min":0.39100000000000001,"5min":0.35599999999999998,"15min":0.28000000000000003},"max":{"1min":5.6299999999999999,"5min":9.8439999999999994,"15min":11.553000000000001},"last":0.52300000000000002}]},{"osd":3,"last update":"Fri Feb 9 17:41:09 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":1.8700000000000001,"15min":2.0430000000000001},"min":{"1min":0.36499999999999999,"5min":0.26700000000000002,"15min":0.248},"max":{"1min":5.1500000000000004,"5min":11.276,"15min":32.545000000000002},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.67700000000000005,"5min":1.7,"15min":1.919},"min":{"1min":0.34200000000000003,"5min":0.307,"15min":0.26600000000000001},"max":{"1min":1.8759999999999999,"5min":11.122,"15min":32.670999999999999},"last":0.47699999999999998}]},{"osd":4,"last update":"Fri Feb 9 17:41:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.82999999999999996,"15min":1.3580000000000001},"min":{"1min":0.35599999999999998,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.96899999999999997,"5min":6.1509999999999998,"15min":19.798999999999999},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.90300000000000002,"5min":1.0189999999999999,"15min":1.3400000000000001},"min":{"1min":0.34300000000000003,"5min":0.28499999999999998,"15min":0.26000000000000001},"max":{"1min":5.5140000000000002,"5min":5.9260000000000002,"15min":10.702999999999999},"last":0.63500000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:41:09 2024","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":1.1020000000000001,"15min":1.3560000000000001},"min":{"1min":0.41499999999999998,"5min":0.318,"15min":0.29199999999999998},"max":{"1min":5.6680000000000001,"5min":6.7679999999999998,"15min":10.968999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.998,"15min":1.383},"min":{"1min":0.317,"5min":0.316,"15min":0.27400000000000002},"max":{"1min":1.9259999999999999,"5min":6.0350000000000001,"15min":13.91},"last":0.50600000000000001}]},{"osd":7,"last update":"Fri Feb 9 17:40:50 2024","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.0720000000000001,"15min":1.4199999999999999},"min":{"1min":0.32100000000000001,"5min":0.29399999999999998,"15min":0.28000000000000003},"max":{"1min":5.1799999999999997,"5min":5.4029999999999996,"15min":14.022},"last":0.91400000000000003},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.2370000000000001,"15min":1.581},"min":{"1min":0.41099999999999998,"5min":0.36099999999999999,"15min":0.309},"max":{"1min":5.5609999999999999,"5min":12.441000000000001,"15min":15.458},"last":0.55900000000000005}]},{"osd":8,"last update":"Fri Feb 9 17:26:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":0.876,"15min":0.79600000000000004},"min":{"1min":0.39900000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":4.0339999999999998,"5min":4.4880000000000004,"15min":4.4880000000000004},"last":4.3840000000000003},{"interface":"front","average":{"1min":1.0469999999999999,"5min":0.88600000000000001,"15min":0.78300000000000003},"min":{"1min":0.375,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":3.6539999999999999,"5min":4.8570000000000002,"15min":4.8570000000000002},"last":4.4210000000000003}]},{"osd":9,"last update":"Fri Feb 9 17:41:16 2024","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.089,"15min":1.2869999999999999},"min":{"1min":0.499,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":5.5789999999999997,"5min":10.779999999999999,"15min":10.803000000000001},"last":0.75800000000000001},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.218,"15min":1.415},"min":{"1min":0.61299999999999999,"5min":0.40999999999999998,"15min":0.40300000000000002},"max":{"1min":5.5419999999999998,"5min":6.3650000000000002,"15min":8.1630000000000003},"last":0.66800000000000004}]},{"osd":10,"last update":"Fri Feb 9 17:40:50 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":1.0700000000000001,"15min":1.391},"min":{"1min":0.46400000000000002,"5min":0.38500000000000001,"15min":0.35399999999999998},"max":{"1min":1.6970000000000001,"5min":10.677,"15min":10.677},"last":0.73899999999999999},{"interface":"front","average":{"1min":1.0409999999999999,"5min":1.0680000000000001,"15min":1.478},"min":{"1min":0.48599999999999999,"5min":0.38,"15min":0.35899999999999999},"max":{"1min":5.0979999999999999,"5min":5.9509999999999996,"15min":13.836},"last":1.0549999999999999}]},{"osd":11,"last update":"Fri Feb 9 17:41:09 2024","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.2789999999999999,"15min":1.4730000000000001},"min":{"1min":0.63600000000000001,"5min":0.40799999999999997,"15min":0.36099999999999999},"max":{"1min":7.6040000000000001,"5min":7.7450000000000001,"15min":10.273999999999999},"last":0.70999999999999996},{"interface":"front","average":{"1min":1.026,"5min":1.0589999999999999,"15min":1.5229999999999999},"min":{"1min":0.44500000000000001,"5min":0.442,"15min":0.41199999999999998},"max":{"1min":5.2140000000000004,"5min":5.2140000000000004,"15min":8.2739999999999991},"last":0.84699999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575130,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1513764,"kb_used_data":963356,"kb_used_omap":1,"kb_used_meta":550398,"kb_avail":92231388,"statfs":{"total":95995035648,"available":94444941312,"internally_reserved":0,"allocated":986476544,"data_stored":985948852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":563608005},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:41:27 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":2.6509999999999998,"15min":1.8759999999999999},"min":{"1min":0.25,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":1.9450000000000001,"5min":19.079000000000001,"15min":19.079000000000001},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.44700000000000001,"5min":2.2650000000000001,"15min":1.8089999999999999},"min":{"1min":0.26300000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.99299999999999999,"5min":23.573,"15min":23.573},"last":0.34799999999999998}]},{"osd":2,"last update":"Fri Feb 9 17:40:47 2024","interfaces":[{"interface":"back","average":{"1min":2.1720000000000002,"5min":2.4590000000000001,"15min":1.998},"min":{"1min":0.30199999999999999,"5min":0.23200000000000001,"15min":0.22600000000000001},"max":{"1min":9.5609999999999999,"5min":26.361999999999998,"15min":26.361999999999998},"last":0.32300000000000001},{"interface":"front","average":{"1min":1.9099999999999999,"5min":2.6859999999999999,"15min":2.1859999999999999},"min":{"1min":0.27100000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":6.2610000000000001,"5min":26.087,"15min":26.087},"last":0.28499999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:40:36 2024","interfaces":[{"interface":"back","average":{"1min":1.8959999999999999,"5min":2.2690000000000001,"15min":1.6060000000000001},"min":{"1min":0.26200000000000001,"5min":0.245,"15min":0.245},"max":{"1min":6.9400000000000004,"5min":10.657,"15min":10.657},"last":0.30299999999999999},{"interface":"front","average":{"1min":2.2349999999999999,"5min":2.508,"15min":1.734},"min":{"1min":0.36599999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":7.1029999999999998,"5min":14.68,"15min":14.68},"last":1.25}]},{"osd":4,"last update":"Fri Feb 9 17:41:21 2024","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.9259999999999999,"15min":1.966},"min":{"1min":0.34699999999999998,"5min":0.25800000000000001,"15min":0.253},"max":{"1min":4.694,"5min":9.4649999999999999,"15min":11.766999999999999},"last":0.443},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.927,"15min":1.9039999999999999},"min":{"1min":0.38800000000000001,"5min":0.309,"15min":0.29999999999999999},"max":{"1min":4.5730000000000004,"5min":10.512,"15min":10.512},"last":0.36299999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:41:05 2024","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.8819999999999999,"15min":1.867},"min":{"1min":0.36599999999999999,"5min":0.28999999999999998,"15min":0.27000000000000002},"max":{"1min":3.9380000000000002,"5min":9.4299999999999997,"15min":13.544},"last":0.40999999999999998},{"interface":"front","average":{"1min":1.149,"5min":2.1789999999999998,"15min":1.75},"min":{"1min":0.442,"5min":0.30499999999999999,"15min":0.26000000000000001},"max":{"1min":4.0800000000000001,"5min":10.531000000000001,"15min":12.117000000000001},"last":0.378}]},{"osd":6,"last update":"Fri Feb 9 17:41:10 2024","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":5.0119999999999996,"5min":5.0119999999999996,"15min":5.0119999999999996},"last":0.96099999999999997},{"interface":"front","average":{"1min":1.175,"5min":1.175,"15min":1.175},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":3.831,"5min":3.831,"15min":3.831},"last":0.97899999999999998}]},{"osd":7,"last update":"Fri Feb 9 17:41:11 2024","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":1.157,"15min":1.157},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":4.0439999999999996,"5min":4.0439999999999996,"15min":4.0439999999999996},"last":1.611},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":4.9820000000000002,"5min":4.9820000000000002,"15min":4.9820000000000002},"last":0.39600000000000002}]},{"osd":8,"last update":"Fri Feb 9 17:41:21 2024","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":2.121,"5min":2.121,"15min":2.121},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":2.1379999999999999,"5min":2.1379999999999999,"15min":2.1379999999999999},"last":0.45400000000000001}]},{"osd":9,"last update":"Fri Feb 9 17:41:27 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":1.9810000000000001,"15min":1.8109999999999999},"min":{"1min":0.42999999999999999,"5min":0.33400000000000002,"15min":0.32400000000000001},"max":{"1min":1.48,"5min":9.4960000000000004,"15min":9.5839999999999996},"last":0.52900000000000003},{"interface":"front","average":{"1min":1.1100000000000001,"5min":2.1400000000000001,"15min":1.8420000000000001},"min":{"1min":0.38700000000000001,"5min":0.32900000000000001,"15min":0.30099999999999999},"max":{"1min":3.9710000000000001,"5min":11.366,"15min":12.117000000000001},"last":3.4569999999999999}]},{"osd":10,"last update":"Fri Feb 9 17:41:05 2024","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":2.246,"15min":1.714},"min":{"1min":0.57499999999999996,"5min":0.41899999999999998,"15min":0.35999999999999999},"max":{"1min":2.9729999999999999,"5min":15.083,"15min":15.083},"last":0.875},{"interface":"front","average":{"1min":1.1799999999999999,"5min":2.29,"15min":1.99},"min":{"1min":0.55100000000000005,"5min":0.48099999999999998,"15min":0.33400000000000002},"max":{"1min":3.137,"5min":15.121,"15min":15.121},"last":0.89200000000000002}]},{"osd":11,"last update":"Fri Feb 9 17:40:01 2024","interfaces":[{"interface":"back","average":{"1min":2.9849999999999999,"5min":2.2050000000000001,"15min":2.1269999999999998},"min":{"1min":0.45200000000000001,"5min":0.38300000000000001,"15min":0.32000000000000001},"max":{"1min":8.7379999999999995,"5min":11.404999999999999,"15min":12.169},"last":0.504},{"interface":"front","average":{"1min":2.3660000000000001,"5min":2.1040000000000001,"15min":2.0699999999999998},"min":{"1min":0.47099999999999997,"5min":0.42399999999999999,"15min":0.36599999999999999},"max":{"1min":7.0099999999999998,"5min":11.387,"15min":12.144},"last":3.0910000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738655,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2374628,"kb_used_data":1892252,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":91370524,"statfs":{"total":95995035648,"available":93563416576,"internally_reserved":0,"allocated":1937666048,"data_stored":1937138356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":493943237},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 9 17:41:08 2024","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":1.911,"15min":1.7170000000000001},"min":{"1min":0.23599999999999999,"5min":0.22800000000000001,"15min":0.20499999999999999},"max":{"1min":7.202,"5min":13.842000000000001,"15min":14.166},"last":0.23699999999999999},{"interface":"front","average":{"1min":1.4339999999999999,"5min":2.4289999999999998,"15min":1.992},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.215},"max":{"1min":7.1669999999999998,"5min":14.122,"15min":20.177},"last":0.33900000000000002}]},{"osd":2,"last update":"Fri Feb 9 17:39:25 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":5.5339999999999998,"5min":5.5339999999999998,"15min":5.5339999999999998},"last":1.744},{"interface":"front","average":{"1min":1.593,"5min":1.593,"15min":1.593},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":13.473000000000001,"5min":13.473000000000001,"15min":13.473000000000001},"last":3.109}]},{"osd":3,"last update":"Fri Feb 9 17:40:39 2024","interfaces":[{"interface":"back","average":{"1min":3.0230000000000001,"5min":3.0230000000000001,"15min":3.0230000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":16.756,"5min":16.756,"15min":16.756},"last":0.308},{"interface":"front","average":{"1min":1.806,"5min":1.806,"15min":1.806},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":6.0570000000000004,"5min":6.0570000000000004,"15min":6.0570000000000004},"last":0.25600000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:41:30 2024","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":1.2450000000000001,"15min":1.673},"min":{"1min":0.311,"5min":0.29399999999999998,"15min":0.26400000000000001},"max":{"1min":2.9169999999999998,"5min":7.7560000000000002,"15min":14.419},"last":1.2869999999999999},{"interface":"front","average":{"1min":0.85899999999999999,"5min":1.657,"15min":1.913},"min":{"1min":0.39700000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":2.5129999999999999,"5min":13.308999999999999,"15min":14.465999999999999},"last":0.36899999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:40:48 2024","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":1.4910000000000001,"15min":1.7310000000000001},"min":{"1min":0.379,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":4.3079999999999998,"5min":8.4079999999999995,"15min":9.8569999999999993},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.84199999999999997,"5min":1.468,"15min":1.8500000000000001},"min":{"1min":0.41299999999999998,"5min":0.32400000000000001,"15min":0.312},"max":{"1min":3.407,"5min":7.8239999999999998,"15min":9.9049999999999994},"last":0.45800000000000002}]},{"osd":6,"last update":"Fri Feb 9 17:41:18 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":1.673,"15min":1.913},"min":{"1min":0.374,"5min":0.33300000000000002,"15min":0.27800000000000002},"max":{"1min":2.7810000000000001,"5min":13.407,"15min":13.407},"last":2.802},{"interface":"front","average":{"1min":0.80600000000000005,"5min":1.7250000000000001,"15min":1.8939999999999999},"min":{"1min":0.39200000000000002,"5min":0.379,"15min":0.33000000000000002},"max":{"1min":2.8079999999999998,"5min":13.448,"15min":13.448},"last":2.835}]},{"osd":7,"last update":"Fri Feb 9 17:40:39 2024","interfaces":[{"interface":"back","average":{"1min":2.149,"5min":2.0529999999999999,"15min":2.1150000000000002},"min":{"1min":0.35299999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":5.8490000000000002,"5min":13.349,"15min":13.349},"last":0.39900000000000002},{"interface":"front","average":{"1min":2.4089999999999998,"5min":2.1539999999999999,"15min":2.0800000000000001},"min":{"1min":0.41699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":8.0009999999999994,"5min":14.451000000000001,"15min":16.765999999999998},"last":0.61799999999999999}]},{"osd":8,"last update":"Fri Feb 9 17:40:40 2024","interfaces":[{"interface":"back","average":{"1min":2.0670000000000002,"5min":1.992,"15min":1.6519999999999999},"min":{"1min":0.496,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":6.6390000000000002,"5min":14.323,"15min":14.323},"last":0.48999999999999999},{"interface":"front","average":{"1min":2.1099999999999999,"5min":2.1070000000000002,"15min":1.7190000000000001},"min":{"1min":0.58499999999999996,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":6.6959999999999997,"5min":14.481999999999999,"15min":16.41},"last":0.59099999999999997}]},{"osd":9,"last update":"Fri Feb 9 17:41:30 2024","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":2.1120000000000001,"15min":1.865},"min":{"1min":0.433,"5min":0.32500000000000001,"15min":0.312},"max":{"1min":2.407,"5min":17.952000000000002,"15min":17.952000000000002},"last":0.51400000000000001},{"interface":"front","average":{"1min":1.0129999999999999,"5min":2.2250000000000001,"15min":1.899},"min":{"1min":0.57299999999999995,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":2.3290000000000002,"5min":17.913,"15min":17.913},"last":1.6180000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:40:48 2024","interfaces":[{"interface":"back","average":{"1min":1.752,"5min":2.2010000000000001,"15min":1.9319999999999999},"min":{"1min":0.52800000000000002,"5min":0.42399999999999999,"15min":0.38500000000000001},"max":{"1min":6.6689999999999996,"5min":14.271000000000001,"15min":16.876000000000001},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.82799999999999996,"5min":1.3879999999999999,"15min":1.593},"min":{"1min":0.46600000000000003,"5min":0.40699999999999997,"15min":0.39800000000000002},"max":{"1min":3.3780000000000001,"5min":11.384,"15min":11.384},"last":0.503}]},{"osd":11,"last update":"Fri Feb 9 17:41:18 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":1.3819999999999999,"15min":1.7509999999999999},"min":{"1min":0.41899999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":2.5430000000000001,"5min":7.8739999999999997,"15min":40.021000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.4790000000000001,"15min":1.905},"min":{"1min":0.5,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":2.702,"5min":7.899,"15min":40.055999999999997},"last":0.53300000000000003}]}]},{"osd":2,"up_from":0,"seq":77309411604,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":27,"seq":115964117265,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":658796,"kb_used_data":248612,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":93086356,"statfs":{"total":95995035648,"available":95320428544,"internally_reserved":0,"allocated":254578688,"data_stored":254047444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":420018631},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:41:07 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":2.355,"15min":1.8979999999999999},"min":{"1min":0.27100000000000002,"5min":0.186,"15min":0.186},"max":{"1min":2.1840000000000002,"5min":38.984999999999999,"15min":38.984999999999999},"last":1.325},{"interface":"front","average":{"1min":0.45500000000000002,"5min":2.556,"15min":2},"min":{"1min":0.27600000000000002,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":1.417,"5min":41.497999999999998,"15min":41.497999999999998},"last":2.0139999999999998}]},{"osd":1,"last update":"Fri Feb 9 17:40:06 2024","interfaces":[{"interface":"back","average":{"1min":4.5670000000000002,"5min":4.5670000000000002,"15min":4.5670000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":25.965,"5min":25.965,"15min":25.965},"last":0.35299999999999998},{"interface":"front","average":{"1min":4.5129999999999999,"5min":4.5129999999999999,"15min":4.5129999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":31.454000000000001,"5min":31.454000000000001,"15min":31.454000000000001},"last":0.375}]},{"osd":2,"last update":"Fri Feb 9 17:41:00 2024","interfaces":[{"interface":"back","average":{"1min":1.9119999999999999,"5min":2.1240000000000001,"15min":1.8740000000000001},"min":{"1min":0.29099999999999998,"5min":0.255,"15min":0.22600000000000001},"max":{"1min":11,"5min":11.663,"15min":14.914999999999999},"last":3.7959999999999998},{"interface":"front","average":{"1min":1.5249999999999999,"5min":2.0529999999999999,"15min":1.891},"min":{"1min":0.32200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.071,"5min":12.347,"15min":17.613},"last":1.9370000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:41:10 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":1.5940000000000001,"15min":1.9350000000000001},"min":{"1min":0.40000000000000002,"5min":0.26800000000000002,"15min":0.254},"max":{"1min":2.2639999999999998,"5min":13.933,"15min":34.898000000000003},"last":0.30399999999999999},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.6830000000000001,"15min":1.8300000000000001},"min":{"1min":0.437,"5min":0.28000000000000003,"15min":0.27900000000000003},"max":{"1min":3.4380000000000002,"5min":9.3059999999999992,"15min":14.061999999999999},"last":1.3640000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:40:45 2024","interfaces":[{"interface":"back","average":{"1min":2.504,"5min":1.71,"15min":1.6699999999999999},"min":{"1min":0.33700000000000002,"5min":0.29899999999999999,"15min":0.27300000000000002},"max":{"1min":11.007999999999999,"5min":11.007999999999999,"15min":17.651},"last":0.501},{"interface":"front","average":{"1min":2.5619999999999998,"5min":1.7689999999999999,"15min":1.724},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.33300000000000002},"max":{"1min":11.036,"5min":11.036,"15min":17.675999999999998},"last":0.36299999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:40:34 2024","interfaces":[{"interface":"back","average":{"1min":2.7450000000000001,"5min":2.7450000000000001,"15min":2.7450000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":12.717000000000001,"5min":12.717000000000001,"15min":12.717000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":8.7360000000000007,"5min":8.7360000000000007,"15min":8.7360000000000007},"last":1.395}]},{"osd":7,"last update":"Fri Feb 9 17:41:20 2024","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.872,"15min":0.872},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":3.3999999999999999,"5min":3.3999999999999999,"15min":3.3999999999999999},"last":1.4970000000000001},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":2.7989999999999999,"5min":2.7989999999999999,"15min":2.7989999999999999},"last":1.556}]},{"osd":8,"last update":"Fri Feb 9 17:41:10 2024","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.6759999999999999,"15min":1.4410000000000001},"min":{"1min":0.54100000000000004,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":6.2229999999999999,"5min":8.6940000000000008,"15min":8.6940000000000008},"last":1.4450000000000001},{"interface":"front","average":{"1min":0.78700000000000003,"5min":1.851,"15min":1.2050000000000001},"min":{"1min":0.442,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":2.3079999999999998,"5min":13.976000000000001,"15min":14.129},"last":0.42799999999999999}]},{"osd":9,"last update":"Fri Feb 9 17:41:30 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":1.7250000000000001,"15min":1.5429999999999999},"min":{"1min":0.53500000000000003,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.821,"5min":9.7929999999999993,"15min":9.7929999999999993},"last":1.421},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.7090000000000001,"15min":1.7490000000000001},"min":{"1min":0.51400000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":2.6160000000000001,"5min":12.815,"15min":13.717000000000001},"last":1.47}]},{"osd":10,"last update":"Fri Feb 9 17:41:16 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":2.2850000000000001,"5min":2.2850000000000001,"15min":2.2850000000000001},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.248,"5min":1.248,"15min":1.248},"last":0.53900000000000003}]},{"osd":11,"last update":"Fri Feb 9 17:41:10 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":1.7490000000000001,"15min":1.6539999999999999},"min":{"1min":0.48399999999999999,"5min":0.40400000000000003,"15min":0.38800000000000001},"max":{"1min":1.613,"5min":14.300000000000001,"15min":14.300000000000001},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.73399999999999999,"5min":2.3500000000000001,"15min":2.0099999999999998},"min":{"1min":0.495,"5min":0.442,"15min":0.36899999999999999},"max":{"1min":1.2769999999999999,"5min":17.145,"15min":17.145},"last":0.94499999999999995}]}]},{"osd":4,"up_from":32,"seq":137438953738,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1880228,"kb_used_data":1443676,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":91864924,"statfs":{"total":95995035648,"available":94069682176,"internally_reserved":0,"allocated":1478324224,"data_stored":1477796532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:41:14 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":1.6930000000000001,"15min":1.7150000000000001},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23599999999999999},"max":{"1min":2.073,"5min":7.9669999999999996,"15min":50.356999999999999},"last":2.1579999999999999},{"interface":"front","average":{"1min":0.77800000000000002,"5min":1.851,"15min":1.6479999999999999},"min":{"1min":0.34599999999999997,"5min":0.247,"15min":0.247},"max":{"1min":2.633,"5min":8.0440000000000005,"15min":50.569000000000003},"last":2.3769999999999998}]},{"osd":1,"last update":"Fri Feb 9 17:41:18 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":2.3420000000000001,"15min":2.0539999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":4.0540000000000003,"5min":28.143000000000001,"15min":50.473999999999997},"last":2.0190000000000001},{"interface":"front","average":{"1min":1.073,"5min":2.3799999999999999,"15min":2.1179999999999999},"min":{"1min":0.35899999999999999,"5min":0.31,"15min":0.28100000000000003},"max":{"1min":4.0679999999999996,"5min":28.103000000000002,"15min":50.499000000000002},"last":0.28299999999999997}]},{"osd":2,"last update":"Fri Feb 9 17:28:51 2024","interfaces":[{"interface":"back","average":{"1min":2.536,"5min":1.546,"15min":0.91400000000000003},"min":{"1min":0.29999999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":8.8819999999999997,"5min":8.8819999999999997,"15min":8.8819999999999997},"last":0.622},{"interface":"front","average":{"1min":2.214,"5min":1.7330000000000001,"15min":0.97799999999999998},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":9.5039999999999996,"5min":9.5039999999999996,"15min":9.5039999999999996},"last":0.53800000000000003}]},{"osd":3,"last update":"Fri Feb 9 17:41:18 2024","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":1.8109999999999999,"15min":2.1619999999999999},"min":{"1min":0.311,"5min":0.245,"15min":0.245},"max":{"1min":3.7000000000000002,"5min":7.9349999999999996,"15min":50.462000000000003},"last":0.34499999999999997},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.8700000000000001,"15min":2.0009999999999999},"min":{"1min":0.40699999999999997,"5min":0.29699999999999999,"15min":0.255},"max":{"1min":4.8330000000000002,"5min":9.1579999999999995,"15min":50.529000000000003},"last":2.194}]},{"osd":5,"last update":"Fri Feb 9 17:40:44 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.93999999999999995,"15min":1.294},"min":{"1min":0.26100000000000001,"5min":0.22500000000000001,"15min":0.222},"max":{"1min":0.96099999999999997,"5min":6.3760000000000003,"15min":9.5630000000000006},"last":2.222},{"interface":"front","average":{"1min":0.56699999999999995,"5min":1.0209999999999999,"15min":1.5700000000000001},"min":{"1min":0.34000000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.995,"5min":6.4080000000000004,"15min":50.509999999999998},"last":2.266}]},{"osd":6,"last update":"Fri Feb 9 17:41:18 2024","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.333,"15min":1.474},"min":{"1min":0.28999999999999998,"5min":0.246,"15min":0.246},"max":{"1min":7.4249999999999998,"5min":8.0039999999999996,"15min":18.09},"last":2.4689999999999999},{"interface":"front","average":{"1min":1.137,"5min":1.137,"15min":1.6160000000000001},"min":{"1min":0.442,"5min":0.245,"15min":0.245},"max":{"1min":5.524,"5min":6.7329999999999997,"15min":50.402999999999999},"last":0.317}]},{"osd":7,"last update":"Fri Feb 9 17:40:49 2024","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.042,"15min":1.575},"min":{"1min":0.32200000000000001,"5min":0.29899999999999999,"15min":0.28399999999999997},"max":{"1min":7.4640000000000004,"5min":7.4640000000000004,"15min":15.223000000000001},"last":2.4870000000000001},{"interface":"front","average":{"1min":0.64300000000000002,"5min":1.1060000000000001,"15min":1.5469999999999999},"min":{"1min":0.40999999999999998,"5min":0.28199999999999997,"15min":0.28100000000000003},"max":{"1min":1.0249999999999999,"5min":6.7640000000000002,"15min":19.047000000000001},"last":1.536}]},{"osd":8,"last update":"Fri Feb 9 17:40:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":1.0109999999999999,"15min":1.4159999999999999},"min":{"1min":0.39100000000000001,"5min":0.34799999999999998,"15min":0.34399999999999997},"max":{"1min":2.6749999999999998,"5min":6.1079999999999997,"15min":8.4160000000000004},"last":1.946},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.998,"15min":1.5349999999999999},"min":{"1min":0.377,"5min":0.36599999999999999,"15min":0.308},"max":{"1min":1.821,"5min":6,"15min":50.594000000000001},"last":2.4049999999999998}]},{"osd":9,"last update":"Fri Feb 9 17:41:28 2024","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":1.171,"15min":1.6419999999999999},"min":{"1min":0.443,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":3.5449999999999999,"5min":6.9699999999999998,"15min":50.442999999999998},"last":1.976},{"interface":"front","average":{"1min":0.92900000000000005,"5min":1.2450000000000001,"15min":1.446},"min":{"1min":0.42499999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":3.7679999999999998,"5min":7.5999999999999996,"15min":10.555},"last":2.4399999999999999}]},{"osd":10,"last update":"Fri Feb 9 17:40:59 2024","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.482,"15min":1.8600000000000001},"min":{"1min":0.40300000000000002,"5min":0.39400000000000002,"15min":0.378},"max":{"1min":5.7629999999999999,"5min":7.0030000000000001,"15min":51.223999999999997},"last":1.9119999999999999},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.474,"15min":1.893},"min":{"1min":0.47699999999999998,"5min":0.41999999999999998,"15min":0.38100000000000001},"max":{"1min":5.7000000000000002,"5min":6.3860000000000001,"15min":50.552},"last":2.0600000000000001}]},{"osd":11,"last update":"Fri Feb 9 17:41:28 2024","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.4910000000000001,"15min":1.9430000000000001},"min":{"1min":0.59699999999999998,"5min":0.49099999999999999,"15min":0.40500000000000003},"max":{"1min":3.6469999999999998,"5min":6.9379999999999997,"15min":50.484999999999999},"last":2.1120000000000001},{"interface":"front","average":{"1min":0.89400000000000002,"5min":1.351,"15min":1.6220000000000001},"min":{"1min":0.46500000000000002,"5min":0.435,"15min":0.34899999999999998},"max":{"1min":2.859,"5min":7.694,"15min":18.574999999999999},"last":2.423}]}]},{"osd":5,"up_from":40,"seq":171798692101,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2156332,"kb_used_data":1796132,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":91588820,"statfs":{"total":95995035648,"available":93786951680,"internally_reserved":0,"allocated":1839239168,"data_stored":1838707924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:41:09 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.995,"15min":1.0329999999999999},"min":{"1min":0.28100000000000003,"5min":0.252,"15min":0.252},"max":{"1min":4.4560000000000004,"5min":4.4560000000000004,"15min":4.4560000000000004},"last":0.31900000000000001},{"interface":"front","average":{"1min":1.075,"5min":1.6659999999999999,"15min":1.764},"min":{"1min":0.313,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":6.4939999999999998,"5min":6.4939999999999998,"15min":6.4939999999999998},"last":2.5659999999999998}]},{"osd":1,"last update":"Fri Feb 9 17:40:42 2024","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.9750000000000001,"15min":2.1360000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.23200000000000001},"max":{"1min":7.6559999999999997,"5min":48.161999999999999,"15min":48.161999999999999},"last":2.254},{"interface":"front","average":{"1min":2.02,"5min":2.1080000000000001,"15min":2.0499999999999998},"min":{"1min":0.28000000000000003,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":7.4390000000000001,"5min":48.218000000000004,"15min":48.218000000000004},"last":6.2290000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:40:42 2024","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":1.345,"15min":1.6539999999999999},"min":{"1min":0.32000000000000001,"5min":0.27900000000000003,"15min":0.25600000000000001},"max":{"1min":4.4930000000000003,"5min":11.143000000000001,"15min":11.143000000000001},"last":2.6560000000000001},{"interface":"front","average":{"1min":1.796,"5min":1.218,"15min":1.718},"min":{"1min":0.39100000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":6.1970000000000001,"5min":6.1970000000000001,"15min":14.122},"last":1.5649999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:40:42 2024","interfaces":[{"interface":"back","average":{"1min":1.8300000000000001,"5min":2.3239999999999998,"15min":2.1760000000000002},"min":{"1min":0.32300000000000001,"5min":0.29799999999999999,"15min":0.26500000000000001},"max":{"1min":11.081,"5min":64.150999999999996,"15min":64.150999999999996},"last":1.8100000000000001},{"interface":"front","average":{"1min":2.3149999999999999,"5min":2.3679999999999999,"15min":2.1379999999999999},"min":{"1min":0.36299999999999999,"5min":0.247,"15min":0.247},"max":{"1min":12.468,"5min":64.176000000000002,"15min":64.176000000000002},"last":2.2170000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:40:45 2024","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":1.5209999999999999,"15min":1.472},"min":{"1min":0.35799999999999998,"5min":0.28100000000000003,"15min":0.23400000000000001},"max":{"1min":6.4619999999999997,"5min":49.313000000000002,"15min":49.313000000000002},"last":1.778},{"interface":"front","average":{"1min":0.97499999999999998,"5min":1.4359999999999999,"15min":1.4810000000000001},"min":{"1min":0.38100000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":6.4260000000000002,"5min":50.505000000000003,"15min":50.505000000000003},"last":2.399}]},{"osd":6,"last update":"Fri Feb 9 17:41:21 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":1.2010000000000001,"15min":1.4350000000000001},"min":{"1min":0.32100000000000001,"5min":0.26400000000000001,"15min":0.253},"max":{"1min":2.1930000000000001,"5min":8.609,"15min":15.375999999999999},"last":2.3740000000000001},{"interface":"front","average":{"1min":0.95399999999999996,"5min":1.29,"15min":1.508},"min":{"1min":0.40799999999999997,"5min":0.28899999999999998,"15min":0.27200000000000002},"max":{"1min":2.7530000000000001,"5min":11.038,"15min":14.250999999999999},"last":1.103}]},{"osd":7,"last update":"Fri Feb 9 17:40:42 2024","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":1.5329999999999999,"15min":1.536},"min":{"1min":0.33000000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":3.8380000000000001,"5min":50.545000000000002,"15min":50.545000000000002},"last":2.3039999999999998},{"interface":"front","average":{"1min":1.1879999999999999,"5min":2.5739999999999998,"15min":2.0830000000000002},"min":{"1min":0.36399999999999999,"5min":0.32100000000000001,"15min":0.28799999999999998},"max":{"1min":6.968,"5min":102.526,"15min":102.526},"last":1.756}]},{"osd":8,"last update":"Fri Feb 9 17:41:21 2024","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":1.361,"15min":1.581},"min":{"1min":0.40899999999999997,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":14.542,"5min":14.542,"15min":14.542},"last":2.3279999999999998},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.94699999999999995,"15min":1.764},"min":{"1min":0.40000000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.524,"5min":5.71,"15min":9.6240000000000006},"last":0.46400000000000002}]},{"osd":9,"last update":"Fri Feb 9 17:41:21 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":1.23,"15min":1.3700000000000001},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.35999999999999999},"max":{"1min":1.4179999999999999,"5min":6.8600000000000003,"15min":14.98},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.995,"5min":1.1950000000000001,"15min":1.369},"min":{"1min":0.45300000000000001,"5min":0.35699999999999998,"15min":0.312},"max":{"1min":5.0439999999999996,"5min":7.7779999999999996,"15min":9.4239999999999995},"last":0.435}]},{"osd":10,"last update":"Fri Feb 9 17:40:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.002,"15min":1.4430000000000001},"min":{"1min":0.54200000000000004,"5min":0.35199999999999998,"15min":0.33800000000000002},"max":{"1min":3.6589999999999998,"5min":5.7400000000000002,"15min":14.218},"last":2.2789999999999999},{"interface":"front","average":{"1min":1.452,"5min":1.319,"15min":1.599},"min":{"1min":0.442,"5min":0.42399999999999999,"15min":0.36099999999999999},"max":{"1min":7.1159999999999997,"5min":10.606,"15min":14.186999999999999},"last":0.57099999999999995}]},{"osd":11,"last update":"Fri Feb 9 17:38:19 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":1.54,"15min":1.46},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":2.6579999999999999,"5min":9.4239999999999995,"15min":9.4239999999999995},"last":0.49099999999999999},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.6639999999999999,"15min":1.5129999999999999},"min":{"1min":0.41399999999999998,"5min":0.39200000000000002,"15min":0.38900000000000001},"max":{"1min":3.4870000000000001,"5min":8.125,"15min":14.316000000000001},"last":2.3519999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1981808640,"data_stored":1981808640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1002176512,"data_stored":1002176512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":256770048,"data_stored":256770048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1503395840,"data_stored":1503395840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1870331904,"data_stored":1870331904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":750387200,"data_stored":750387200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1032978432,"data_stored":1032978432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":758120448,"data_stored":758120448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":751435776,"data_stored":751435776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1576468480,"data_stored":1576468480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":997064704,"data_stored":997064704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-09T17:41:41.054 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-02-09T17:41:41.054 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:41:41.055 INFO:tasks.thrashosds.thrasher:inject_pause on osd.8 2024-02-09T17:41:41.055 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-09T17:41:41.055 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-09T17:41:41.055 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.8 config set filestore_inject_stall 3 2024-02-09T17:41:41.059 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:41:41.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:41.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:41:41.167 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:41:41.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:40 smithi179 ceph-mon[43975]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:41:41.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:40 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3798914215' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:41:41.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:40 smithi179 ceph-mon[43975]: osdmap e487: 12 total, 12 up, 11 in 2024-02-09T17:41:41.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:40 smithi179 ceph-mon[43975]: from='client.15831 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:41:41.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:40 smithi162 ceph-mon[44304]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:41:41.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:40 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3798914215' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:41:41.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:40 smithi162 ceph-mon[44304]: osdmap e487: 12 total, 12 up, 11 in 2024-02-09T17:41:41.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:40 smithi162 ceph-mon[44304]: from='client.15831 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:41:41.199 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:41.449 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:41:41.455 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:41:41.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:41.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:41:41.484 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-09T17:41:41.557 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:41:41.602 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:41.638 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:41:41.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:41.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:41:41.747 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:41:41.786 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:41.815 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:41:41.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:41.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:41:41.882 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:41 smithi162 ceph-mon[44304]: 1.2 scrub starts 2024-02-09T17:41:41.882 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:41 smithi162 ceph-mon[44304]: 1.2 scrub ok 2024-02-09T17:41:41.882 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:41 smithi162 ceph-mon[44304]: pgmap v1235: 37 pgs: 37 active+clean; 3.9 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-02-09T17:41:41.882 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:41 smithi162 ceph-mon[44304]: osdmap e488: 12 total, 12 up, 11 in 2024-02-09T17:41:41.921 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:41:41.957 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 37 16 70217 70201 118.564 92.4375 0.0122059 0.00842943 2024-02-09T17:41:41.983 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:42.015 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:41:42.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:42.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:41:42.030 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:41 smithi179 ceph-mon[43975]: 1.2 scrub starts 2024-02-09T17:41:42.030 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:41 smithi179 ceph-mon[43975]: 1.2 scrub ok 2024-02-09T17:41:42.030 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:41 smithi179 ceph-mon[43975]: pgmap v1235: 37 pgs: 37 active+clean; 3.9 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-02-09T17:41:42.030 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:41 smithi179 ceph-mon[43975]: osdmap e488: 12 total, 12 up, 11 in 2024-02-09T17:41:42.122 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:41:42.165 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:42.191 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:41:42.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:42.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:41:42.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:41 smithi099 ceph-mon[39420]: 1.2 scrub starts 2024-02-09T17:41:42.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:41 smithi099 ceph-mon[39420]: 1.2 scrub ok 2024-02-09T17:41:42.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:41 smithi099 ceph-mon[39420]: pgmap v1235: 37 pgs: 37 active+clean; 3.9 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-02-09T17:41:42.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:41 smithi099 ceph-mon[39420]: osdmap e488: 12 total, 12 up, 11 in 2024-02-09T17:41:42.297 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:41:42.333 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:42.460 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:41:42.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:42.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:41:42.565 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:41:42.620 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:42.878 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:41:42.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:42.887 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:41:42.957 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 38 16 71730 71714 117.932 94.5625 0.00406815 0.00847582 2024-02-09T17:41:42.988 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:41:43.035 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:43.071 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:41:43.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:43.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:41:43.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:42 smithi179 ceph-mon[43975]: 1.a scrub starts 2024-02-09T17:41:43.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:42 smithi179 ceph-mon[43975]: 1.a scrub ok 2024-02-09T17:41:43.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:42 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:41:43.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:42 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:41:43.176 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:41:43.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:42 smithi162 ceph-mon[44304]: 1.a scrub starts 2024-02-09T17:41:43.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:42 smithi162 ceph-mon[44304]: 1.a scrub ok 2024-02-09T17:41:43.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:42 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:41:43.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:42 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:41:43.182 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:42 smithi099 ceph-mon[39420]: 1.a scrub starts 2024-02-09T17:41:43.182 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:42 smithi099 ceph-mon[39420]: 1.a scrub ok 2024-02-09T17:41:43.182 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:42 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:41:43.183 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:42 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:41:43.220 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:43.254 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:41:43.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:43.259 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:41:43.360 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:41:43.404 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:43.577 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:41:43.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:43.581 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:41:43.594 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:41:43.594 INFO:teuthology.orchestra.run.smithi179.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-02-09T17:41:43.594 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:41:43.682 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:41:43.757 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:43.783 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:41:43.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:43.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:41:43.824 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:41:43.877 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:41:43.888 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:41:43.900 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:43 smithi099 ceph-mon[39420]: pgmap v1237: 37 pgs: 37 active+clean; 4.1 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-02-09T17:41:43.900 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:43 smithi099 ceph-mon[39420]: osdmap e489: 12 total, 12 up, 11 in 2024-02-09T17:41:43.900 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:43 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1061406645' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:41:43.900 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:43 smithi179 ceph-mon[43975]: pgmap v1237: 37 pgs: 37 active+clean; 4.1 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-02-09T17:41:43.900 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:43 smithi179 ceph-mon[43975]: osdmap e489: 12 total, 12 up, 11 in 2024-02-09T17:41:43.900 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:43 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1061406645' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:41:43.903 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:41:43.929 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:41:43.941 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:43.957 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 39 15 73161 73146 117.203 89.5 0.0132395 0.00852763 2024-02-09T17:41:43.958 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:41:43.986 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:41:43.988 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:41:43.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:43.999 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:41:44.010 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:41:44.035 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:41:44.060 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:41:44.085 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:41:44.100 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:41:44.110 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:41:44.129 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:44.132 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:41:44.148 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:41:44.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:44.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:41:44.154 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:41:44.155 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:41:44.155 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:41:44.155 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:41:44.156 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:41:44.156 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:41:44.156 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:41:44.156 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:41:44.156 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:41:44.156 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:41:44.156 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:41:44.156 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:41:44.156 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:41:44.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:43 smithi162 ceph-mon[44304]: pgmap v1237: 37 pgs: 37 active+clean; 4.1 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-02-09T17:41:44.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:43 smithi162 ceph-mon[44304]: osdmap e489: 12 total, 12 up, 11 in 2024-02-09T17:41:44.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:43 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1061406645' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:41:44.253 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:41:44.312 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:44.340 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:41:44.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:44.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:41:44.444 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:41:44.476 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:44.483 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-09T17:41:44.957 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:2024-02-09T17:41:44.957082+0000 min lat: 0.00110706 max lat: 2.34848 avg lat: 0.00859066 2024-02-09T17:41:44.957 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-09T17:41:44.957 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 40 16 74488 74472 116.345 82.875 0.0177792 0.00859066 2024-02-09T17:41:45.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:44 smithi179 ceph-mon[43975]: 1.c scrub starts 2024-02-09T17:41:45.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:44 smithi179 ceph-mon[43975]: 1.c scrub ok 2024-02-09T17:41:45.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:44 smithi162 ceph-mon[44304]: 1.c scrub starts 2024-02-09T17:41:45.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:44 smithi162 ceph-mon[44304]: 1.c scrub ok 2024-02-09T17:41:45.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:44 smithi099 ceph-mon[39420]: 1.c scrub starts 2024-02-09T17:41:45.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:44 smithi099 ceph-mon[39420]: 1.c scrub ok 2024-02-09T17:41:45.936 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:41:45.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:45.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:41:45.958 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 41 16 75821 75805 115.536 83.3125 0.0206265 0.00865148 2024-02-09T17:41:45.983 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:41:46.041 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:41:46.095 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:46.131 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:41:46.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:46.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:41:46.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:45 smithi179 ceph-mon[43975]: 1.6 scrub starts 2024-02-09T17:41:46.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:45 smithi179 ceph-mon[43975]: 1.6 scrub ok 2024-02-09T17:41:46.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:45 smithi179 ceph-mon[43975]: pgmap v1239: 37 pgs: 37 active+clean; 4.3 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-02-09T17:41:46.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:45 smithi162 ceph-mon[44304]: 1.6 scrub starts 2024-02-09T17:41:46.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:45 smithi162 ceph-mon[44304]: 1.6 scrub ok 2024-02-09T17:41:46.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:45 smithi162 ceph-mon[44304]: pgmap v1239: 37 pgs: 37 active+clean; 4.3 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-02-09T17:41:46.243 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:41:46.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:45 smithi099 ceph-mon[39420]: 1.6 scrub starts 2024-02-09T17:41:46.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:45 smithi099 ceph-mon[39420]: 1.6 scrub ok 2024-02-09T17:41:46.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:45 smithi099 ceph-mon[39420]: pgmap v1239: 37 pgs: 37 active+clean; 4.3 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-02-09T17:41:46.287 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:46.315 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:41:46.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:46.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:41:46.424 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:41:46.465 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:46.487 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:41:46.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:46.491 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:41:46.592 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:41:46.640 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:46.671 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:41:46.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:46.674 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:41:46.775 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:41:46.825 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:46.855 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:41:46.859 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:46.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:41:46.925 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:46 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:41:46.925 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:46 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:41:46.925 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:46 smithi099 ceph-mon[39420]: osdmap e490: 12 total, 12 up, 11 in 2024-02-09T17:41:46.958 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 42 16 77220 77204 114.867 87.4375 0.00546988 0.00870121 2024-02-09T17:41:46.961 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:41:46.987 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:41:46.987 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [], 2024-02-09T17:41:46.987 INFO:teuthology.orchestra.run.smithi099.stdout: "complaint_time": 30, 2024-02-09T17:41:46.987 INFO:teuthology.orchestra.run.smithi099.stdout: "num_blocked_ops": 0 2024-02-09T17:41:46.987 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:41:47.002 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:47.036 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:41:47.039 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:47.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:41:47.140 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:41:47.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:46 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:41:47.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:46 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:41:47.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:46 smithi162 ceph-mon[44304]: osdmap e490: 12 total, 12 up, 11 in 2024-02-09T17:41:47.192 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:47.219 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:41:47.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:47.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:41:47.324 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:41:47.369 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:47.402 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:41:47.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:47.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:41:47.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:46 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:41:47.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:46 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:41:47.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:46 smithi179 ceph-mon[43975]: osdmap e490: 12 total, 12 up, 11 in 2024-02-09T17:41:47.481 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:41:47.506 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:41:47.554 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:47.591 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:41:47.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:47.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:41:47.705 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:41:47.742 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:47.771 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:41:47.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:47.776 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:41:47.877 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:41:47.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:47 smithi162 ceph-mon[44304]: 1.b scrub starts 2024-02-09T17:41:47.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:47 smithi162 ceph-mon[44304]: 1.b scrub ok 2024-02-09T17:41:47.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:47 smithi162 ceph-mon[44304]: pgmap v1241: 37 pgs: 37 active+clean; 4.5 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 96 MiB/s wr, 1.54k op/s 2024-02-09T17:41:47.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:47 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:41:47.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:47 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:41:47.938 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:47.958 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 43 16 78580 78564 114.172 85 0.00348762 0.00875362 2024-02-09T17:41:47.972 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:41:47.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:47.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:41:48.072 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-09T17:41:48.078 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:41:48.155 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:48.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:47 smithi179 ceph-mon[43975]: 1.b scrub starts 2024-02-09T17:41:48.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:47 smithi179 ceph-mon[43975]: 1.b scrub ok 2024-02-09T17:41:48.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:47 smithi179 ceph-mon[43975]: pgmap v1241: 37 pgs: 37 active+clean; 4.5 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 96 MiB/s wr, 1.54k op/s 2024-02-09T17:41:48.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:47 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:41:48.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:47 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:41:48.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:47 smithi099 ceph-mon[39420]: 1.b scrub starts 2024-02-09T17:41:48.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:47 smithi099 ceph-mon[39420]: 1.b scrub ok 2024-02-09T17:41:48.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:47 smithi099 ceph-mon[39420]: pgmap v1241: 37 pgs: 37 active+clean; 4.5 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 96 MiB/s wr, 1.54k op/s 2024-02-09T17:41:48.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:47 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:41:48.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:47 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:41:48.410 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:41:48.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:48.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:41:48.515 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:41:48.559 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:48.584 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:41:48.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:48.589 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:41:48.690 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:41:48.720 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:48.959 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 44 16 80030 80014 113.637 90.625 0.00842773 0.00879588 2024-02-09T17:41:48.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:48 smithi099 ceph-mon[39420]: osdmap e491: 12 total, 12 up, 11 in 2024-02-09T17:41:49.173 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:41:49.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:49.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:41:49.210 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:41:49.222 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:41:49.282 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:41:49.327 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:48 smithi162 ceph-mon[44304]: osdmap e491: 12 total, 12 up, 11 in 2024-02-09T17:41:49.334 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:49.355 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:41:49.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:49.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:41:49.376 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] 2024-02-09T17:41:49.376 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:41:49.376 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-02-09T17:41:49.376 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-09T17:41:49.376 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-09T17:41:49.377 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-02-09T17:41:49.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:48 smithi179 ceph-mon[43975]: osdmap e491: 12 total, 12 up, 11 in 2024-02-09T17:41:49.459 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:41:49.493 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:49.959 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 45 16 81458 81442 113.095 89.25 0.0155455 0.00883861 2024-02-09T17:41:50.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:49 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:41:50.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:49 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:41:50.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:49 smithi099 ceph-mon[39420]: pgmap v1243: 37 pgs: 37 active+clean; 4.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 85 MiB/s wr, 1.35k op/s 2024-02-09T17:41:50.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:49 smithi099 ceph-mon[39420]: osdmap e492: 12 total, 12 up, 11 in 2024-02-09T17:41:50.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:49 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:41:50.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:49 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:41:50.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:49 smithi179 ceph-mon[43975]: pgmap v1243: 37 pgs: 37 active+clean; 4.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 85 MiB/s wr, 1.35k op/s 2024-02-09T17:41:50.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:49 smithi179 ceph-mon[43975]: osdmap e492: 12 total, 12 up, 11 in 2024-02-09T17:41:50.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:49 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:41:50.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:49 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:41:50.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:49 smithi162 ceph-mon[44304]: pgmap v1243: 37 pgs: 37 active+clean; 4.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 85 MiB/s wr, 1.35k op/s 2024-02-09T17:41:50.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:49 smithi162 ceph-mon[44304]: osdmap e492: 12 total, 12 up, 11 in 2024-02-09T17:41:50.959 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 46 16 82848 82832 112.525 86.875 0.0103315 0.00888294 2024-02-09T17:41:51.959 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 47 16 84142 84126 111.851 80.875 0.0162008 0.00893301 2024-02-09T17:41:52.190 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:51 smithi099 ceph-mon[39420]: pgmap v1245: 37 pgs: 37 active+clean; 4.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-02-09T17:41:52.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:51 smithi179 ceph-mon[43975]: pgmap v1245: 37 pgs: 37 active+clean; 4.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-02-09T17:41:52.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:51 smithi162 ceph-mon[44304]: pgmap v1245: 37 pgs: 37 active+clean; 4.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-02-09T17:41:52.959 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 48 16 85292 85276 111.019 71.875 0.0107546 0.00900284 2024-02-09T17:41:52.960 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is unset 2024-02-09T17:41:53.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:52 smithi099 ceph-mon[39420]: 5.9 scrub starts 2024-02-09T17:41:53.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:52 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/358408163' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:41:53.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:52 smithi179 ceph-mon[43975]: 5.9 scrub starts 2024-02-09T17:41:53.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:52 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/358408163' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:41:53.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:52 smithi162 ceph-mon[44304]: 5.9 scrub starts 2024-02-09T17:41:53.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:52 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/358408163' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:41:53.959 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 49 16 86464 86448 110.248 73.25 0.0105163 0.00906596 2024-02-09T17:41:54.053 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:41:54.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:54.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:41:54.160 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:41:54.211 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:54.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:53 smithi099 ceph-mon[39420]: 5.9 scrub ok 2024-02-09T17:41:54.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:53 smithi099 ceph-mon[39420]: 1.7 scrub starts 2024-02-09T17:41:54.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:53 smithi099 ceph-mon[39420]: 1.7 scrub ok 2024-02-09T17:41:54.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:53 smithi099 ceph-mon[39420]: pgmap v1246: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.0 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-02-09T17:41:54.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:53 smithi099 ceph-mon[39420]: 5.e scrub starts 2024-02-09T17:41:54.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:53 smithi099 ceph-mon[39420]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:41:54.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:53 smithi099 ceph-mon[39420]: Cluster is now healthy 2024-02-09T17:41:54.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:53 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/358408163' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:41:54.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:53 smithi099 ceph-mon[39420]: osdmap e493: 12 total, 12 up, 11 in 2024-02-09T17:41:54.245 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:41:54.248 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:53 smithi162 ceph-mon[44304]: 5.9 scrub ok 2024-02-09T17:41:54.249 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:53 smithi162 ceph-mon[44304]: 1.7 scrub starts 2024-02-09T17:41:54.249 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:53 smithi162 ceph-mon[44304]: 1.7 scrub ok 2024-02-09T17:41:54.249 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:53 smithi162 ceph-mon[44304]: pgmap v1246: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.0 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-02-09T17:41:54.249 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:53 smithi162 ceph-mon[44304]: 5.e scrub starts 2024-02-09T17:41:54.249 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:53 smithi162 ceph-mon[44304]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:41:54.249 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:53 smithi162 ceph-mon[44304]: Cluster is now healthy 2024-02-09T17:41:54.249 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:53 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/358408163' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:41:54.249 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:53 smithi162 ceph-mon[44304]: osdmap e493: 12 total, 12 up, 11 in 2024-02-09T17:41:54.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:54.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:41:54.353 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:41:54.392 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:54.418 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:41:54.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:54.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:41:54.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:53 smithi179 ceph-mon[43975]: 5.9 scrub ok 2024-02-09T17:41:54.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:53 smithi179 ceph-mon[43975]: 1.7 scrub starts 2024-02-09T17:41:54.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:53 smithi179 ceph-mon[43975]: 1.7 scrub ok 2024-02-09T17:41:54.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:53 smithi179 ceph-mon[43975]: pgmap v1246: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.0 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-02-09T17:41:54.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:53 smithi179 ceph-mon[43975]: 5.e scrub starts 2024-02-09T17:41:54.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:53 smithi179 ceph-mon[43975]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:41:54.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:53 smithi179 ceph-mon[43975]: Cluster is now healthy 2024-02-09T17:41:54.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:53 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/358408163' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:41:54.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:53 smithi179 ceph-mon[43975]: osdmap e493: 12 total, 12 up, 11 in 2024-02-09T17:41:54.522 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:41:54.552 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:54.937 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:41:54.959 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 50 16 87679 87663 109.561 75.9375 0.0182149 0.00912248 2024-02-09T17:41:55.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:54 smithi099 ceph-mon[39420]: 5.d scrub starts 2024-02-09T17:41:55.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:54 smithi099 ceph-mon[39420]: 5.e scrub ok 2024-02-09T17:41:55.302 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:41:55.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:55.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:41:55.408 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:41:55.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:54 smithi179 ceph-mon[43975]: 5.d scrub starts 2024-02-09T17:41:55.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:54 smithi179 ceph-mon[43975]: 5.e scrub ok 2024-02-09T17:41:55.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:54 smithi162 ceph-mon[44304]: 5.d scrub starts 2024-02-09T17:41:55.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:54 smithi162 ceph-mon[44304]: 5.e scrub ok 2024-02-09T17:41:55.479 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:55.514 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:41:55.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:55.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:41:55.620 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:41:55.658 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:55.688 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:41:55.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:55.692 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:41:55.793 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:41:55.840 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:55.875 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:41:55.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:55.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:41:55.959 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 51 16 88891 88875 108.898 75.75 0.010882 0.00917904 2024-02-09T17:41:55.980 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:41:56.014 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:56.021 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set noscrub 2024-02-09T17:41:56.053 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:55 smithi099 ceph-mon[39420]: 5.3 scrub starts 2024-02-09T17:41:56.053 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:55 smithi099 ceph-mon[39420]: 5.d scrub ok 2024-02-09T17:41:56.053 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:55 smithi099 ceph-mon[39420]: pgmap v1248: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.1 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 77 MiB/s wr, 1.24k op/s 2024-02-09T17:41:56.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:55 smithi162 ceph-mon[44304]: 5.3 scrub starts 2024-02-09T17:41:56.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:55 smithi162 ceph-mon[44304]: 5.d scrub ok 2024-02-09T17:41:56.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:55 smithi162 ceph-mon[44304]: pgmap v1248: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.1 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 77 MiB/s wr, 1.24k op/s 2024-02-09T17:41:56.274 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:41:56.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:56.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:41:56.379 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:41:56.410 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:56.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:55 smithi179 ceph-mon[43975]: 5.3 scrub starts 2024-02-09T17:41:56.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:55 smithi179 ceph-mon[43975]: 5.d scrub ok 2024-02-09T17:41:56.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:55 smithi179 ceph-mon[43975]: pgmap v1248: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.1 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 77 MiB/s wr, 1.24k op/s 2024-02-09T17:41:56.559 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:41:56.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:56.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:41:56.665 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:41:56.702 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:56.729 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:41:56.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:56.732 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:41:56.833 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:41:56.859 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:41:56.881 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:41:56.896 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:56.907 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:41:56.921 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:41:56.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:56.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:41:56.939 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:41:56.961 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 52 16 90192 90176 108.364 81.3125 0.00775422 0.00922416 2024-02-09T17:41:56.972 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:41:57.001 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:41:57.031 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:41:57.032 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:41:57.060 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:41:57.079 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:57.090 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:41:57.110 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:41:57.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:57.119 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:41:57.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:41:57.149 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:41:57.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:56 smithi162 ceph-mon[44304]: 1.8 scrub starts 2024-02-09T17:41:57.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:56 smithi162 ceph-mon[44304]: 1.8 scrub ok 2024-02-09T17:41:57.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:56 smithi162 ceph-mon[44304]: 5.3 scrub ok 2024-02-09T17:41:57.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:56 smithi162 ceph-mon[44304]: osdmap e494: 12 total, 12 up, 11 in 2024-02-09T17:41:57.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:56 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/332963482' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:41:57.177 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:41:57.178 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:41:57.178 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:41:57.178 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:41:57.178 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:41:57.178 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:41:57.178 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:41:57.178 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:41:57.179 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:41:57.179 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:41:57.179 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:41:57.179 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:41:57.179 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:41:57.220 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:41:57.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:56 smithi099 ceph-mon[39420]: 1.8 scrub starts 2024-02-09T17:41:57.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:56 smithi099 ceph-mon[39420]: 1.8 scrub ok 2024-02-09T17:41:57.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:56 smithi099 ceph-mon[39420]: 5.3 scrub ok 2024-02-09T17:41:57.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:56 smithi099 ceph-mon[39420]: osdmap e494: 12 total, 12 up, 11 in 2024-02-09T17:41:57.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:56 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/332963482' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:41:57.281 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:57.311 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:41:57.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:57.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:41:57.416 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:41:57.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:56 smithi179 ceph-mon[43975]: 1.8 scrub starts 2024-02-09T17:41:57.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:56 smithi179 ceph-mon[43975]: 1.8 scrub ok 2024-02-09T17:41:57.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:56 smithi179 ceph-mon[43975]: 5.3 scrub ok 2024-02-09T17:41:57.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:56 smithi179 ceph-mon[43975]: osdmap e494: 12 total, 12 up, 11 in 2024-02-09T17:41:57.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:56 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/332963482' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:41:57.455 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:57.478 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:41:57.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:57.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:41:57.582 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:41:57.608 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:57.627 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:41:57.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:41:57.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:41:57.731 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:41:57.762 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:41:57.961 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 53 16 90838 90822 107.081 40.375 0.0074589 0.00933481 2024-02-09T17:41:58.170 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:57 smithi099 ceph-mon[39420]: pgmap v1250: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.3 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-02-09T17:41:58.210 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:41:58.210 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:41:58.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:57 smithi179 ceph-mon[43975]: pgmap v1250: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.3 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-02-09T17:41:58.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:57 smithi162 ceph-mon[44304]: pgmap v1250: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.3 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-02-09T17:41:58.961 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 54 16 92164 92148 106.633 82.875 0.0063124 0.00937353 2024-02-09T17:41:59.322 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:41:59.322 INFO:teuthology.orchestra.run.smithi099.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-02-09T17:41:59.322 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:41:59.961 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 55 16 93312 93296 105.999 71.75 0.00828509 0.00942996 2024-02-09T17:42:00.015 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:42:00.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:00.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:42:00.119 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:42:00.165 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:00.207 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:42:00.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:00.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:42:00.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:59 smithi099 ceph-mon[39420]: pgmap v1251: 37 pgs: 37 active+clean; 5.4 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-02-09T17:42:00.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:59 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:42:00.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:59 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:42:00.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:41:59 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:42:00.313 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:42:00.345 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:59 smithi179 ceph-mon[43975]: pgmap v1251: 37 pgs: 37 active+clean; 5.4 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-02-09T17:42:00.345 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:59 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:42:00.345 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:59 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:42:00.346 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:41:59 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:42:00.348 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:59 smithi162 ceph-mon[44304]: pgmap v1251: 37 pgs: 37 active+clean; 5.4 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-02-09T17:42:00.348 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:59 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:42:00.348 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:59 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:42:00.348 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:41:59 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:42:00.351 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:00.374 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:42:00.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:00.378 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:42:00.479 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:42:00.525 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:00.584 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:42:00.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:00.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:42:00.593 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:42:00.624 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-09T17:42:00.689 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:42:00.725 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:00.747 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:42:00.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:00.755 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:42:00.856 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:42:00.887 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:00.914 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:42:00.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:00.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:42:00.961 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 56 16 94426 94410 105.349 69.625 0.0222084 0.00948807 2024-02-09T17:42:01.021 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:42:01.053 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:01.124 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:42:01.161 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:01 smithi099 ceph-mon[39420]: 5.c scrub starts 2024-02-09T17:42:01.161 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:01 smithi099 ceph-mon[39420]: osdmap e495: 12 total, 12 up, 11 in 2024-02-09T17:42:01.276 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:42:01.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:01 smithi179 ceph-mon[43975]: 5.c scrub starts 2024-02-09T17:42:01.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:01 smithi179 ceph-mon[43975]: osdmap e495: 12 total, 12 up, 11 in 2024-02-09T17:42:01.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:01 smithi162 ceph-mon[44304]: 5.c scrub starts 2024-02-09T17:42:01.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:01 smithi162 ceph-mon[44304]: osdmap e495: 12 total, 12 up, 11 in 2024-02-09T17:42:01.961 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 57 16 95517 95501 104.697 68.1875 0.00912606 0.00954728 2024-02-09T17:42:02.105 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:42:02.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:02.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:42:02.152 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:42:02.216 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:42:02.265 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:02.420 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:02 smithi099 ceph-mon[39420]: 5.c scrub ok 2024-02-09T17:42:02.420 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:02 smithi099 ceph-mon[39420]: pgmap v1253: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.6 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-02-09T17:42:02.420 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:02 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:42:02.420 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:02 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:42:02.420 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:02 smithi099 ceph-mon[39420]: osdmap e496: 12 total, 12 up, 11 in 2024-02-09T17:42:02.420 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:02 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:42:02.421 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:02 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:42:02.421 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:02 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:42:02.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:02 smithi179 ceph-mon[43975]: 5.c scrub ok 2024-02-09T17:42:02.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:02 smithi179 ceph-mon[43975]: pgmap v1253: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.6 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-02-09T17:42:02.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:02 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:42:02.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:02 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:42:02.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:02 smithi179 ceph-mon[43975]: osdmap e496: 12 total, 12 up, 11 in 2024-02-09T17:42:02.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:02 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:42:02.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:02 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:42:02.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:02 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:42:02.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:02 smithi162 ceph-mon[44304]: 5.c scrub ok 2024-02-09T17:42:02.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:02 smithi162 ceph-mon[44304]: pgmap v1253: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.6 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-02-09T17:42:02.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:02 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:42:02.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:02 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:42:02.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:02 smithi162 ceph-mon[44304]: osdmap e496: 12 total, 12 up, 11 in 2024-02-09T17:42:02.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:02 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:42:02.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:02 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:42:02.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:02 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:42:02.744 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:42:02.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:02.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:42:02.853 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:42:02.899 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:02.935 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:42:02.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:02.940 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:42:02.961 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 58 16 96724 96708 104.193 75.4375 0.0129821 0.00959326 2024-02-09T17:42:03.041 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:42:03.094 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:03.215 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:03 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2470137268' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:42:03.216 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is set 2024-02-09T17:42:03.228 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:42:03.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:03.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:42:03.337 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:42:03.369 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:03.396 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:42:03.396 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:42:03.408 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:42:03.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:03.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:42:03.420 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:03 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2470137268' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:42:03.519 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:42:03.549 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:03.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:03 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2470137268' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:42:03.961 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 59 16 97818 97802 103.585 68.375 0.0142922 0.00964979 2024-02-09T17:42:04.001 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:42:04.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:04.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:42:04.108 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:42:04.164 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-09T17:42:04.172 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:04.208 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:42:04.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:04.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:42:04.315 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:42:04.356 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:04.385 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:42:04.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:04.394 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:42:04.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:04 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:42:04.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:04 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:42:04.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:04 smithi162 ceph-mon[44304]: pgmap v1255: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-02-09T17:42:04.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:04 smithi162 ceph-mon[44304]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:42:04.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:04 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2470137268' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:42:04.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:04 smithi162 ceph-mon[44304]: osdmap e497: 12 total, 12 up, 11 in 2024-02-09T17:42:04.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:04 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:42:04.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:04 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:42:04.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:04 smithi099 ceph-mon[39420]: pgmap v1255: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-02-09T17:42:04.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:04 smithi099 ceph-mon[39420]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:42:04.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:04 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2470137268' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:42:04.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:04 smithi099 ceph-mon[39420]: osdmap e497: 12 total, 12 up, 11 in 2024-02-09T17:42:04.495 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:42:04.563 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:04.611 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:42:04.615 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:04.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:42:04.655 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:42:04.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:04 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:42:04.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:04 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:42:04.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:04 smithi179 ceph-mon[43975]: pgmap v1255: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-02-09T17:42:04.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:04 smithi179 ceph-mon[43975]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:42:04.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:04 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2470137268' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:42:04.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:04 smithi179 ceph-mon[43975]: osdmap e497: 12 total, 12 up, 11 in 2024-02-09T17:42:04.716 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:42:04.762 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:04.791 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:42:04.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:04.794 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:42:04.895 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:42:04.930 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:04.962 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:2024-02-09T17:42:04.961608+0000 min lat: 0.00110706 max lat: 2.34848 avg lat: 0.00970243 2024-02-09T17:42:04.962 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-09T17:42:04.962 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 60 16 98938 98922 103.026 70 0.00972681 0.00970243 2024-02-09T17:42:04.985 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] 2024-02-09T17:42:04.985 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:42:04.986 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-02-09T17:42:04.986 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-09T17:42:04.986 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-09T17:42:04.986 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-02-09T17:42:05.037 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:42:05.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:05.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:42:05.142 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:42:05.183 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:05.207 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:42:05.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:05.212 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:42:05.221 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:42:05.227 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:05 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2809097791' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:42:05.260 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:42:05.298 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:42:05.312 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:42:05.324 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:42:05.362 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:42:05.367 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:42:05.385 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:05.405 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:42:05.437 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:42:05.474 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:42:05.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:05 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2809097791' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:42:05.500 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:42:05.525 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:42:05.525 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:42:05.529 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:05.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:42:05.550 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:42:05.577 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:42:05.580 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:42:05.580 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:42:05.581 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:42:05.581 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:42:05.581 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:42:05.581 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:42:05.581 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:42:05.581 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:42:05.581 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:42:05.581 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:42:05.581 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:42:05.581 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:42:05.630 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:42:05.663 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:05.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:05 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2809097791' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:42:05.962 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 61 16 100131 100115 102.559 74.5625 0.0128692 0.0097464 2024-02-09T17:42:06.026 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:42:06.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:06.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:42:06.134 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:42:06.174 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:06.379 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:06 smithi162 ceph-mon[44304]: pgmap v1257: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.8 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-02-09T17:42:06.429 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:42:06.431 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:06 smithi099 ceph-mon[39420]: pgmap v1257: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.8 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-02-09T17:42:06.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:06.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:42:06.533 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:42:06.562 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:06.597 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-09T17:42:06.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:06 smithi179 ceph-mon[43975]: pgmap v1257: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.8 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-02-09T17:42:06.724 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:42:06.724 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-09T17:42:06.725 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-02-09T17:42:06.725 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-02-09T17:42:06.725 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:42:06.962 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 62 16 101264 101248 102.047 70.8125 0.00575583 0.00979531 2024-02-09T17:42:07.114 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:42:07.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:07.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:42:07.218 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:42:07.286 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:07.444 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:42:07.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:07.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:42:07.467 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-09T17:42:07.554 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:42:07.562 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:42:07.597 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:07.628 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:42:07.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:07.632 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:42:07.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:07 smithi162 ceph-mon[44304]: osdmap e498: 12 total, 12 up, 11 in 2024-02-09T17:42:07.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:07 smithi162 ceph-mon[44304]: pgmap v1259: 37 pgs: 37 active+clean; 6.0 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-02-09T17:42:07.733 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:42:07.787 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:07.834 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:07 smithi099 ceph-mon[39420]: osdmap e498: 12 total, 12 up, 11 in 2024-02-09T17:42:07.834 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:07 smithi099 ceph-mon[39420]: pgmap v1259: 37 pgs: 37 active+clean; 6.0 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-02-09T17:42:07.835 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:42:07.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:07.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:42:07.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:07 smithi179 ceph-mon[43975]: osdmap e498: 12 total, 12 up, 11 in 2024-02-09T17:42:07.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:07 smithi179 ceph-mon[43975]: pgmap v1259: 37 pgs: 37 active+clean; 6.0 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-02-09T17:42:07.945 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:42:07.953 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:42:07.962 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 63 16 102355 102339 101.509 68.1875 0.00528668 0.00984681 2024-02-09T17:42:07.980 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:08.935 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:42:08.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:08.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:42:08.963 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 64 16 103510 103494 101.051 72.1875 0.0067521 0.00989202 2024-02-09T17:42:08.964 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:42:08.964 INFO:teuthology.orchestra.run.smithi099.stdout: "success": "" 2024-02-09T17:42:08.964 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:42:09.040 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:42:09.075 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:42:09.082 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:09.339 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:42:09.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:09.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:42:09.448 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:42:09.488 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:09.520 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:42:09.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:09.529 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:42:09.630 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:42:09.672 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:09.701 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:42:09.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:09.707 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:42:09.808 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:42:09.841 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:09.962 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 65 16 104650 104634 100.593 71.25 0.0126554 0.00993712 2024-02-09T17:42:09.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:09 smithi099 ceph-mon[39420]: pgmap v1260: 37 pgs: 37 active+clean; 6.1 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-02-09T17:42:10.094 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:42:10.095 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-02-09T17:42:10.095 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-02-09T17:42:10.095 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-02-09T17:42:10.095 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.095 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:204179 3.1 3:8372c17c:::benchmark_data_smithi179_90440_object72220:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e277)", 2024-02-09T17:42:10.095 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:33:44.992761+0000", 2024-02-09T17:42:10.095 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 505.09655538700002, 2024-02-09T17:42:10.095 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6720821499999996, 2024-02-09T17:42:10.095 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:10.095 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-09T17:42:10.095 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:10.096 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:42:10.096 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:42:10.096 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 204179 2024-02-09T17:42:10.096 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.096 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:10.096 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.096 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:10.096 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:44.992761+0000", 2024-02-09T17:42:10.096 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:10.096 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.096 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.097 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:10.097 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:44.992761+0000", 2024-02-09T17:42:10.097 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:10.097 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.097 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.097 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:10.097 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:44.992761+0000", 2024-02-09T17:42:10.097 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0800000000000001e-07 2024-02-09T17:42:10.097 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.097 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.097 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:10.097 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:44.992762+0000", 2024-02-09T17:42:10.097 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6020000000000001e-06 2024-02-09T17:42:10.098 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.098 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.098 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:10.098 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:44.992766+0000", 2024-02-09T17:42:10.098 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1540000000000002e-06 2024-02-09T17:42:10.098 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.098 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.098 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:10.098 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:44.992768+0000", 2024-02-09T17:42:10.098 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7379999999999999e-06 2024-02-09T17:42:10.098 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.098 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.098 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:10.098 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:45.004640+0000", 2024-02-09T17:42:10.098 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011871246 2024-02-09T17:42:10.099 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.099 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.099 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for missing object", 2024-02-09T17:42:10.099 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:45.004713+0000", 2024-02-09T17:42:10.099 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3804e-05 2024-02-09T17:42:10.099 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.099 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.099 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:10.099 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:49.664827+0000", 2024-02-09T17:42:10.099 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6601140020000003 2024-02-09T17:42:10.099 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.099 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.099 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:10.100 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:49.664843+0000", 2024-02-09T17:42:10.100 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5396e-05 2024-02-09T17:42:10.100 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:10.100 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:10.100 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:10.100 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.100 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.100 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:204180 3.3 3:c0ceb3da:::benchmark_data_smithi179_90440_object72221:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e277)", 2024-02-09T17:42:10.100 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:33:44.992772+0000", 2024-02-09T17:42:10.100 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 505.096544403, 2024-02-09T17:42:10.100 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6746478590000002, 2024-02-09T17:42:10.100 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:10.100 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-09T17:42:10.101 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:10.101 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:42:10.101 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:42:10.101 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 204180 2024-02-09T17:42:10.101 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.101 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:10.101 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.101 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:10.101 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:44.992772+0000", 2024-02-09T17:42:10.101 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:10.101 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.101 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.101 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:10.101 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:44.992772+0000", 2024-02-09T17:42:10.102 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:10.102 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.102 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.102 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:10.102 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:44.992772+0000", 2024-02-09T17:42:10.102 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.04e-07 2024-02-09T17:42:10.102 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.102 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.102 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:10.102 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:44.992776+0000", 2024-02-09T17:42:10.102 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8600000000000003e-06 2024-02-09T17:42:10.102 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.102 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.102 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:10.103 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:44.992780+0000", 2024-02-09T17:42:10.103 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3549999999999998e-06 2024-02-09T17:42:10.103 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.103 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.103 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:10.103 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:44.992783+0000", 2024-02-09T17:42:10.103 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.399e-06 2024-02-09T17:42:10.103 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.103 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.103 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:10.103 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:45.004367+0000", 2024-02-09T17:42:10.103 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011583515000000001 2024-02-09T17:42:10.103 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.103 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.104 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for missing object", 2024-02-09T17:42:10.104 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:45.004446+0000", 2024-02-09T17:42:10.104 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9504000000000003e-05 2024-02-09T17:42:10.104 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.104 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.104 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:10.104 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:49.667412+0000", 2024-02-09T17:42:10.104 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6629655210000003 2024-02-09T17:42:10.104 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.104 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.104 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:10.104 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:33:49.667419+0000", 2024-02-09T17:42:10.104 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5009999999999998e-06 2024-02-09T17:42:10.105 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:10.105 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:10.105 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:10.105 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.105 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.105 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25201.0:208407 4.e 4:7bdf9d69:::benchmark_data_smithi179_103809_object70941:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-02-09T17:42:10.105 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:38:33.813701+0000", 2024-02-09T17:42:10.105 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 216.275614491, 2024-02-09T17:42:10.105 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2211754350000001, 2024-02-09T17:42:10.105 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:10.105 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-09T17:42:10.105 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:10.105 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25201", 2024-02-09T17:42:10.106 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:42:10.106 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 208407 2024-02-09T17:42:10.106 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.106 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:10.106 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.106 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:10.106 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:33.813701+0000", 2024-02-09T17:42:10.106 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:10.106 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.106 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.106 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:10.106 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:33.813701+0000", 2024-02-09T17:42:10.106 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:10.107 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.107 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.107 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:10.107 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:33.813703+0000", 2024-02-09T17:42:10.107 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6479999999999999e-06 2024-02-09T17:42:10.107 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.107 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.107 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:10.107 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:33.813709+0000", 2024-02-09T17:42:10.107 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9800000000000003e-06 2024-02-09T17:42:10.107 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.107 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.107 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:10.107 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:33.813726+0000", 2024-02-09T17:42:10.107 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6872999999999999e-05 2024-02-09T17:42:10.108 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.108 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.108 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:10.108 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:33.813736+0000", 2024-02-09T17:42:10.108 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8600000000000005e-06 2024-02-09T17:42:10.108 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.108 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.108 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:10.108 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:33.814238+0000", 2024-02-09T17:42:10.108 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00050189199999999998 2024-02-09T17:42:10.108 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.108 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.108 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:10.108 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:33.814346+0000", 2024-02-09T17:42:10.109 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000108315 2024-02-09T17:42:10.109 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.109 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.109 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,10", 2024-02-09T17:42:10.109 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:33.814458+0000", 2024-02-09T17:42:10.109 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011234 2024-02-09T17:42:10.109 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.109 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.109 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:10.109 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:33.820137+0000", 2024-02-09T17:42:10.109 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0056781569999999996 2024-02-09T17:42:10.109 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.109 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.109 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:10.109 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:33.833664+0000", 2024-02-09T17:42:10.110 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013527617 2024-02-09T17:42:10.110 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.110 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.110 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:42:10.110 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:33.833682+0000", 2024-02-09T17:42:10.110 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7796999999999999e-05 2024-02-09T17:42:10.110 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.110 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.110 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:10.110 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:34.815121+0000", 2024-02-09T17:42:10.110 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.98143867200000001 2024-02-09T17:42:10.110 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.110 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.110 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:42:10.110 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:34.815135+0000", 2024-02-09T17:42:10.111 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4551e-05 2024-02-09T17:42:10.111 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.111 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.111 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:10.111 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:36.017611+0000", 2024-02-09T17:42:10.111 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.202475483 2024-02-09T17:42:10.111 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.111 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.111 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for missing object", 2024-02-09T17:42:10.111 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:36.017774+0000", 2024-02-09T17:42:10.111 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016380999999999999 2024-02-09T17:42:10.111 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.111 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.111 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:10.112 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:36.034802+0000", 2024-02-09T17:42:10.112 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.017027706 2024-02-09T17:42:10.112 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.112 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.112 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:10.112 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:36.034877+0000", 2024-02-09T17:42:10.112 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4734000000000006e-05 2024-02-09T17:42:10.112 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:10.112 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:10.112 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:10.112 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.112 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.112 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25201.0:222259 4.e 4:79e2d474:::benchmark_data_smithi179_103809_object84793:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e397)", 2024-02-09T17:42:10.112 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:39:02.584331+0000", 2024-02-09T17:42:10.113 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 187.50498488599999, 2024-02-09T17:42:10.113 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2140263610000002, 2024-02-09T17:42:10.113 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:10.113 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-09T17:42:10.113 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:10.113 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25201", 2024-02-09T17:42:10.113 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:42:10.113 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 222259 2024-02-09T17:42:10.113 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.113 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:10.113 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.114 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:10.114 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.584331+0000", 2024-02-09T17:42:10.114 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:10.114 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.114 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.114 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:10.114 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.584331+0000", 2024-02-09T17:42:10.114 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:10.115 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.115 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.115 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:10.115 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.584332+0000", 2024-02-09T17:42:10.115 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9900000000000001e-07 2024-02-09T17:42:10.115 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.115 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.116 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:10.116 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.584335+0000", 2024-02-09T17:42:10.116 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.151e-06 2024-02-09T17:42:10.116 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.116 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.116 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:10.116 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.584343+0000", 2024-02-09T17:42:10.116 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0409999999999998e-06 2024-02-09T17:42:10.116 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.116 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.116 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:10.116 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.584349+0000", 2024-02-09T17:42:10.116 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6030000000000001e-06 2024-02-09T17:42:10.117 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.117 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.117 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:10.117 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.584413+0000", 2024-02-09T17:42:10.117 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3984000000000003e-05 2024-02-09T17:42:10.117 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.117 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.117 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:10.117 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.584567+0000", 2024-02-09T17:42:10.117 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000153969 2024-02-09T17:42:10.117 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.117 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.117 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,10", 2024-02-09T17:42:10.117 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.584688+0000", 2024-02-09T17:42:10.118 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012019399999999999 2024-02-09T17:42:10.118 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.118 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.118 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:10.118 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.590409+0000", 2024-02-09T17:42:10.118 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0057209849999999996 2024-02-09T17:42:10.118 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.118 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.118 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:42:10.118 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.590418+0000", 2024-02-09T17:42:10.118 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7999999999999993e-06 2024-02-09T17:42:10.118 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.118 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.118 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:10.119 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.598122+0000", 2024-02-09T17:42:10.119 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.007703708 2024-02-09T17:42:10.119 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.119 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.119 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:42:10.119 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.598142+0000", 2024-02-09T17:42:10.119 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9797e-05 2024-02-09T17:42:10.119 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.119 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.119 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:10.119 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.798282+0000", 2024-02-09T17:42:10.119 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2001401380000001 2024-02-09T17:42:10.119 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.119 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.120 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:10.120 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.798357+0000", 2024-02-09T17:42:10.120 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5492000000000002e-05 2024-02-09T17:42:10.120 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:10.120 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:10.120 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:10.120 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.120 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.120 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25201.0:222287 4.e 4:7c7bef54:::benchmark_data_smithi179_103809_object84821:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e399)", 2024-02-09T17:42:10.120 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:39:03.641509+0000", 2024-02-09T17:42:10.120 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 186.44780702, 2024-02-09T17:42:10.120 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.168818505, 2024-02-09T17:42:10.120 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:10.120 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:42:10.120 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:10.121 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25201", 2024-02-09T17:42:10.121 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:42:10.121 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 222287 2024-02-09T17:42:10.121 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.121 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:10.121 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.121 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:10.121 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.641509+0000", 2024-02-09T17:42:10.121 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:10.121 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.121 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.121 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:10.121 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.641509+0000", 2024-02-09T17:42:10.121 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:10.121 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.122 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.122 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:10.122 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.641510+0000", 2024-02-09T17:42:10.122 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4890000000000001e-06 2024-02-09T17:42:10.122 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.122 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.122 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:10.122 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.641515+0000", 2024-02-09T17:42:10.122 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7809999999999996e-06 2024-02-09T17:42:10.122 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.122 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.122 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:10.122 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.641526+0000", 2024-02-09T17:42:10.123 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1060000000000001e-05 2024-02-09T17:42:10.123 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.123 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.123 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:10.123 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.641535+0000", 2024-02-09T17:42:10.123 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5180000000000004e-06 2024-02-09T17:42:10.123 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.123 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.123 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:10.123 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.641605+0000", 2024-02-09T17:42:10.123 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0267999999999999e-05 2024-02-09T17:42:10.123 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.123 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.123 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:42:10.124 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.641618+0000", 2024-02-09T17:42:10.124 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2862e-05 2024-02-09T17:42:10.124 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.124 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.124 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:10.124 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.798602+0000", 2024-02-09T17:42:10.124 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.15698368 2024-02-09T17:42:10.124 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.124 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.124 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:10.124 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.798717+0000", 2024-02-09T17:42:10.124 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000115429 2024-02-09T17:42:10.124 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.125 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.125 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,5,10", 2024-02-09T17:42:10.125 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.798851+0000", 2024-02-09T17:42:10.125 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013354299999999999 2024-02-09T17:42:10.125 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.125 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.125 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:10.125 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.809916+0000", 2024-02-09T17:42:10.125 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011065042000000001 2024-02-09T17:42:10.125 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.125 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.125 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:10.125 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.809959+0000", 2024-02-09T17:42:10.126 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3517999999999999e-05 2024-02-09T17:42:10.126 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.126 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.126 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:10.126 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.810001+0000", 2024-02-09T17:42:10.126 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1732e-05 2024-02-09T17:42:10.126 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.126 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.126 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:10.126 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.810144+0000", 2024-02-09T17:42:10.126 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014349400000000001 2024-02-09T17:42:10.126 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.126 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.126 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:42:10.126 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.810204+0000", 2024-02-09T17:42:10.126 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9463000000000003e-05 2024-02-09T17:42:10.127 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.127 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.127 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:10.127 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.810327+0000", 2024-02-09T17:42:10.127 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000123626 2024-02-09T17:42:10.127 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:10.127 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:10.127 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:10.127 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.127 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.127 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25201.0:222290 4.e 4:7ebcc15e:::benchmark_data_smithi179_103809_object84824:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e399)", 2024-02-09T17:42:10.127 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:39:03.647256+0000", 2024-02-09T17:42:10.127 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 186.44205976399999, 2024-02-09T17:42:10.127 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.163461198, 2024-02-09T17:42:10.127 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:10.128 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:42:10.128 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:10.128 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25201", 2024-02-09T17:42:10.128 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:42:10.128 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 222290 2024-02-09T17:42:10.128 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.128 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:10.128 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.128 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:10.128 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.647256+0000", 2024-02-09T17:42:10.128 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:10.128 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.128 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.128 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:10.129 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.647256+0000", 2024-02-09T17:42:10.129 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:10.129 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.129 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.129 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:10.129 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.647257+0000", 2024-02-09T17:42:10.129 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0359999999999999e-06 2024-02-09T17:42:10.129 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.129 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.129 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:10.129 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.647262+0000", 2024-02-09T17:42:10.129 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8049999999999997e-06 2024-02-09T17:42:10.129 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.130 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.130 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:10.130 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.647275+0000", 2024-02-09T17:42:10.130 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2788e-05 2024-02-09T17:42:10.130 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.130 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.130 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:10.130 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.647284+0000", 2024-02-09T17:42:10.130 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9819999999999997e-06 2024-02-09T17:42:10.130 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.130 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.130 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:10.130 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.647330+0000", 2024-02-09T17:42:10.130 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5924999999999998e-05 2024-02-09T17:42:10.130 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.131 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.131 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:42:10.131 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.647337+0000", 2024-02-09T17:42:10.131 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3209999999999999e-06 2024-02-09T17:42:10.131 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.131 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.131 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:10.131 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.804329+0000", 2024-02-09T17:42:10.131 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.156992231 2024-02-09T17:42:10.131 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.131 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.131 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:10.131 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.804477+0000", 2024-02-09T17:42:10.132 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014802899999999999 2024-02-09T17:42:10.132 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.132 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.132 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,5,10", 2024-02-09T17:42:10.132 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.804646+0000", 2024-02-09T17:42:10.132 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000168853 2024-02-09T17:42:10.132 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.132 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.132 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:10.132 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.810042+0000", 2024-02-09T17:42:10.132 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0053958519999999996 2024-02-09T17:42:10.132 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.132 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.132 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:10.133 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.810083+0000", 2024-02-09T17:42:10.133 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0639000000000002e-05 2024-02-09T17:42:10.133 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.133 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.133 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:10.133 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.810126+0000", 2024-02-09T17:42:10.133 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2905999999999999e-05 2024-02-09T17:42:10.133 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.133 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.133 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:10.133 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.810565+0000", 2024-02-09T17:42:10.133 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00043988300000000002 2024-02-09T17:42:10.133 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.134 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.134 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:42:10.134 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.810617+0000", 2024-02-09T17:42:10.134 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1560000000000001e-05 2024-02-09T17:42:10.134 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.134 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.134 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:10.134 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.810717+0000", 2024-02-09T17:42:10.134 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000100388 2024-02-09T17:42:10.134 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:10.134 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:10.134 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:10.134 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.134 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.134 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25201.0:230624 4.e 4:7f23c367:::benchmark_data_smithi179_103809_object93158:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e411)", 2024-02-09T17:42:10.135 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:39:19.517268+0000", 2024-02-09T17:42:10.135 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 170.572047914, 2024-02-09T17:42:10.135 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0345313949999999, 2024-02-09T17:42:10.135 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:10.135 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:42:10.135 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:10.135 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25201", 2024-02-09T17:42:10.135 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:42:10.135 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 230624 2024-02-09T17:42:10.135 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.135 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:10.135 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.135 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:10.135 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:19.517268+0000", 2024-02-09T17:42:10.136 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:10.136 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.136 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.136 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:10.136 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:19.517268+0000", 2024-02-09T17:42:10.136 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:10.136 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.136 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.136 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:10.136 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:19.517269+0000", 2024-02-09T17:42:10.136 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2079999999999999e-06 2024-02-09T17:42:10.136 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.136 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.136 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:10.137 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:19.517274+0000", 2024-02-09T17:42:10.137 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1000000000000003e-06 2024-02-09T17:42:10.137 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.137 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.137 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:10.137 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:19.517285+0000", 2024-02-09T17:42:10.137 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0173000000000001e-05 2024-02-09T17:42:10.137 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.137 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.137 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:10.137 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:19.517292+0000", 2024-02-09T17:42:10.137 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5240000000000003e-06 2024-02-09T17:42:10.137 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.138 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.138 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:10.138 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:19.517350+0000", 2024-02-09T17:42:10.138 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7605999999999997e-05 2024-02-09T17:42:10.138 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.138 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.138 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:42:10.138 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:19.517356+0000", 2024-02-09T17:42:10.138 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0290000000000002e-06 2024-02-09T17:42:10.138 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.138 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.138 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:10.138 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:20.536023+0000", 2024-02-09T17:42:10.138 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.01866758 2024-02-09T17:42:10.139 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.139 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.139 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:10.139 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:20.536138+0000", 2024-02-09T17:42:10.139 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000114258 2024-02-09T17:42:10.139 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.139 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.139 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5,10", 2024-02-09T17:42:10.139 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:20.536264+0000", 2024-02-09T17:42:10.139 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000126663 2024-02-09T17:42:10.139 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.139 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.139 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:10.139 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:20.551552+0000", 2024-02-09T17:42:10.140 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015288047000000001 2024-02-09T17:42:10.140 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.140 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.140 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:10.140 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:20.551567+0000", 2024-02-09T17:42:10.140 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4739999999999999e-05 2024-02-09T17:42:10.140 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.140 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.140 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:10.140 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:20.551689+0000", 2024-02-09T17:42:10.140 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012160899999999999 2024-02-09T17:42:10.140 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.140 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.141 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:42:10.141 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:20.551749+0000", 2024-02-09T17:42:10.141 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0665000000000003e-05 2024-02-09T17:42:10.141 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.141 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.141 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:10.141 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:20.551799+0000", 2024-02-09T17:42:10.141 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0192999999999999e-05 2024-02-09T17:42:10.141 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:10.141 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:10.141 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:10.141 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.142 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.142 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25201.0:230657 4.e 4:7f594855:::benchmark_data_smithi179_103809_object93191:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e411)", 2024-02-09T17:42:10.142 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:39:19.565283+0000", 2024-02-09T17:42:10.142 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 170.524032611, 2024-02-09T17:42:10.142 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99722966499999999, 2024-02-09T17:42:10.142 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:10.142 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:42:10.142 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:10.142 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25201", 2024-02-09T17:42:10.142 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:42:10.142 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 230657 2024-02-09T17:42:10.142 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.142 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:10.142 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.142 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:10.143 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:19.565283+0000", 2024-02-09T17:42:10.143 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:10.143 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.143 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.143 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:10.143 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:19.565283+0000", 2024-02-09T17:42:10.143 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:10.143 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.143 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.143 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:10.143 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:19.565284+0000", 2024-02-09T17:42:10.143 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3200000000000004e-07 2024-02-09T17:42:10.144 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.144 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.144 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:10.144 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:19.565289+0000", 2024-02-09T17:42:10.144 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5299999999999998e-06 2024-02-09T17:42:10.144 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.144 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.144 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:10.144 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:19.565300+0000", 2024-02-09T17:42:10.144 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0797e-05 2024-02-09T17:42:10.144 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.144 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.144 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:10.145 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:19.565307+0000", 2024-02-09T17:42:10.145 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7300000000000005e-06 2024-02-09T17:42:10.145 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.145 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.145 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:10.145 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:19.565421+0000", 2024-02-09T17:42:10.145 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000114135 2024-02-09T17:42:10.145 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.145 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.145 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:42:10.145 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:19.565432+0000", 2024-02-09T17:42:10.145 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0125999999999999e-05 2024-02-09T17:42:10.145 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.145 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.146 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:10.146 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:20.541301+0000", 2024-02-09T17:42:10.146 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.975869498 2024-02-09T17:42:10.146 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.146 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.146 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:10.146 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:20.541428+0000", 2024-02-09T17:42:10.146 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012745899999999999 2024-02-09T17:42:10.146 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.146 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.146 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5,10", 2024-02-09T17:42:10.146 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:20.541554+0000", 2024-02-09T17:42:10.146 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000125322 2024-02-09T17:42:10.147 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.147 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.147 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:10.147 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:20.551647+0000", 2024-02-09T17:42:10.147 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010093098999999999 2024-02-09T17:42:10.147 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.147 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.147 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:10.147 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:20.551850+0000", 2024-02-09T17:42:10.147 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020347200000000001 2024-02-09T17:42:10.147 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.147 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.147 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:10.147 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:20.562452+0000", 2024-02-09T17:42:10.148 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010601413000000001 2024-02-09T17:42:10.148 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.148 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.148 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:42:10.148 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:20.562487+0000", 2024-02-09T17:42:10.148 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4958e-05 2024-02-09T17:42:10.148 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.148 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.148 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:10.148 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:20.562513+0000", 2024-02-09T17:42:10.148 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6293999999999999e-05 2024-02-09T17:42:10.148 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:10.148 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:10.149 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:10.149 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.149 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.149 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25201.0:232563 4.e 4:7b844ab9:::benchmark_data_smithi179_103809_object95097:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e414)", 2024-02-09T17:42:10.149 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:39:23.623236+0000", 2024-02-09T17:42:10.149 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 166.466079756, 2024-02-09T17:42:10.149 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1374754500000002, 2024-02-09T17:42:10.149 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:10.149 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:42:10.149 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:10.149 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25201", 2024-02-09T17:42:10.149 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:42:10.149 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 232563 2024-02-09T17:42:10.149 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.149 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:10.149 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.150 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:10.150 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.623236+0000", 2024-02-09T17:42:10.150 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:10.150 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.150 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.150 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:10.150 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.623236+0000", 2024-02-09T17:42:10.150 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:10.150 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.150 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.150 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:10.150 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.623237+0000", 2024-02-09T17:42:10.150 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7599999999999996e-07 2024-02-09T17:42:10.150 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.151 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.151 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:10.151 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.623244+0000", 2024-02-09T17:42:10.151 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5339999999999996e-06 2024-02-09T17:42:10.151 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.151 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.151 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:10.151 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.623253+0000", 2024-02-09T17:42:10.151 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6639999999999993e-06 2024-02-09T17:42:10.151 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.151 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.151 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:10.151 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.623260+0000", 2024-02-09T17:42:10.152 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9e-06 2024-02-09T17:42:10.152 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.152 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.152 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:10.152 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.623328+0000", 2024-02-09T17:42:10.152 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8103999999999997e-05 2024-02-09T17:42:10.152 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.152 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.152 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:42:10.152 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.623340+0000", 2024-02-09T17:42:10.152 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1479e-05 2024-02-09T17:42:10.152 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.152 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.153 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:10.153 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:24.532124+0000", 2024-02-09T17:42:10.153 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.90878441499999996 2024-02-09T17:42:10.153 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.153 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.153 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:42:10.153 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:24.532137+0000", 2024-02-09T17:42:10.153 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2692e-05 2024-02-09T17:42:10.153 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.153 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.153 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:10.153 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.756022+0000", 2024-02-09T17:42:10.153 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2238848529999999 2024-02-09T17:42:10.153 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.154 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.154 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:10.154 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.756142+0000", 2024-02-09T17:42:10.154 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000120562 2024-02-09T17:42:10.154 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.154 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.154 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,5,10", 2024-02-09T17:42:10.154 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.756238+0000", 2024-02-09T17:42:10.154 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5424e-05 2024-02-09T17:42:10.154 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.154 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.154 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:10.154 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.760515+0000", 2024-02-09T17:42:10.154 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042770830000000001 2024-02-09T17:42:10.155 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.155 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.155 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:10.155 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.760566+0000", 2024-02-09T17:42:10.155 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1172999999999999e-05 2024-02-09T17:42:10.155 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.155 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.155 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:10.155 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.760578+0000", 2024-02-09T17:42:10.155 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1693e-05 2024-02-09T17:42:10.155 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.155 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.155 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:10.155 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.760606+0000", 2024-02-09T17:42:10.155 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8138999999999999e-05 2024-02-09T17:42:10.156 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.156 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.156 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:42:10.156 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.760644+0000", 2024-02-09T17:42:10.156 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7966000000000002e-05 2024-02-09T17:42:10.156 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.156 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.156 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:10.156 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.760712+0000", 2024-02-09T17:42:10.156 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7992999999999998e-05 2024-02-09T17:42:10.156 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:10.156 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:10.156 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:10.156 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.157 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.157 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25201.0:232572 4.e 4:7679dda5:::benchmark_data_smithi179_103809_object95106:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e414)", 2024-02-09T17:42:10.157 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:39:23.637905+0000", 2024-02-09T17:42:10.157 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 166.451410837, 2024-02-09T17:42:10.157 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1455606629999999, 2024-02-09T17:42:10.157 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:10.157 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:42:10.157 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:10.157 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25201", 2024-02-09T17:42:10.157 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:42:10.157 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 232572 2024-02-09T17:42:10.157 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.157 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:10.158 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.158 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:10.158 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.637905+0000", 2024-02-09T17:42:10.158 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:10.158 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.158 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.158 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:10.158 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.637905+0000", 2024-02-09T17:42:10.158 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:10.158 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.158 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.158 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:10.158 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.637906+0000", 2024-02-09T17:42:10.158 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8000000000000004e-07 2024-02-09T17:42:10.159 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.159 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.159 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:10.159 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.637910+0000", 2024-02-09T17:42:10.159 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2209999999999999e-06 2024-02-09T17:42:10.159 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.159 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.159 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:10.159 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.637923+0000", 2024-02-09T17:42:10.159 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2279999999999999e-05 2024-02-09T17:42:10.159 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.159 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.159 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:10.159 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.637930+0000", 2024-02-09T17:42:10.160 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5379999999999997e-06 2024-02-09T17:42:10.160 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.160 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.160 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:10.160 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.637990+0000", 2024-02-09T17:42:10.160 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0269999999999997e-05 2024-02-09T17:42:10.160 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.160 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.160 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:42:10.160 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.638000+0000", 2024-02-09T17:42:10.160 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6350000000000001e-06 2024-02-09T17:42:10.160 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.160 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.160 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:10.161 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:24.532226+0000", 2024-02-09T17:42:10.161 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.89422645700000003 2024-02-09T17:42:10.161 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.161 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.161 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:42:10.161 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:24.532242+0000", 2024-02-09T17:42:10.161 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5114e-05 2024-02-09T17:42:10.161 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.161 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.161 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:10.161 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.760842+0000", 2024-02-09T17:42:10.161 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2286009120000001 2024-02-09T17:42:10.161 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.161 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.162 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:10.162 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.760940+0000", 2024-02-09T17:42:10.162 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7262000000000005e-05 2024-02-09T17:42:10.162 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.162 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.162 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,5,10", 2024-02-09T17:42:10.162 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.761020+0000", 2024-02-09T17:42:10.162 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0433999999999996e-05 2024-02-09T17:42:10.162 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.162 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.162 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:10.162 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.769371+0000", 2024-02-09T17:42:10.162 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0083504789999999992 2024-02-09T17:42:10.162 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.163 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.163 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:10.163 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.769381+0000", 2024-02-09T17:42:10.163 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9199999999999999e-06 2024-02-09T17:42:10.163 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.163 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.163 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:10.163 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.774049+0000", 2024-02-09T17:42:10.163 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0046689549999999998 2024-02-09T17:42:10.163 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.163 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.163 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:10.163 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.783345+0000", 2024-02-09T17:42:10.163 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0092952850000000004 2024-02-09T17:42:10.164 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.164 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.164 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:42:10.164 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.783385+0000", 2024-02-09T17:42:10.164 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0299999999999997e-05 2024-02-09T17:42:10.164 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.164 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.164 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:10.164 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.783466+0000", 2024-02-09T17:42:10.164 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0721000000000002e-05 2024-02-09T17:42:10.164 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:10.164 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:10.164 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:10.164 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.165 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.165 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25201.0:232591 4.e 4:74b9959f:::benchmark_data_smithi179_103809_object95125:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-02-09T17:42:10.165 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:39:24.727541+0000", 2024-02-09T17:42:10.165 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 165.361775005, 2024-02-09T17:42:10.165 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0664996879999999, 2024-02-09T17:42:10.165 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:10.165 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:42:10.165 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:10.165 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25201", 2024-02-09T17:42:10.165 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:42:10.165 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 232591 2024-02-09T17:42:10.165 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.165 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:10.165 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.166 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:10.166 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:24.727541+0000", 2024-02-09T17:42:10.166 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:10.166 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.166 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.166 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:10.166 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:24.727541+0000", 2024-02-09T17:42:10.166 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:10.166 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.166 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.166 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:10.166 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:24.727542+0000", 2024-02-09T17:42:10.166 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3510000000000001e-06 2024-02-09T17:42:10.166 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.166 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.167 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:10.167 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:24.727548+0000", 2024-02-09T17:42:10.167 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5659999999999997e-06 2024-02-09T17:42:10.167 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.167 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.167 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:10.167 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:24.727560+0000", 2024-02-09T17:42:10.167 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1902000000000001e-05 2024-02-09T17:42:10.167 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.167 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.167 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:10.167 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:24.727569+0000", 2024-02-09T17:42:10.167 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7790000000000001e-06 2024-02-09T17:42:10.168 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.168 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.168 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:10.168 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:24.727624+0000", 2024-02-09T17:42:10.168 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5831000000000001e-05 2024-02-09T17:42:10.168 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.168 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.168 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:42:10.168 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:24.727632+0000", 2024-02-09T17:42:10.168 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9319999999999996e-06 2024-02-09T17:42:10.168 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.168 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.168 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:10.168 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.764987+0000", 2024-02-09T17:42:10.169 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0373542650000001 2024-02-09T17:42:10.169 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.169 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.169 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:10.169 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.765082+0000", 2024-02-09T17:42:10.169 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4952999999999997e-05 2024-02-09T17:42:10.169 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.169 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.169 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,5,10", 2024-02-09T17:42:10.169 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.765165+0000", 2024-02-09T17:42:10.169 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3306000000000005e-05 2024-02-09T17:42:10.169 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.170 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.170 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:10.170 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.769420+0000", 2024-02-09T17:42:10.170 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042556549999999997 2024-02-09T17:42:10.170 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.170 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.170 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:10.170 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.774008+0000", 2024-02-09T17:42:10.170 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0045875550000000001 2024-02-09T17:42:10.170 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.170 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.170 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:10.170 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.774092+0000", 2024-02-09T17:42:10.171 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4079000000000006e-05 2024-02-09T17:42:10.171 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.171 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.171 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:10.171 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.793926+0000", 2024-02-09T17:42:10.171 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019833871999999999 2024-02-09T17:42:10.171 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.171 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.171 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:42:10.171 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.793968+0000", 2024-02-09T17:42:10.171 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1757e-05 2024-02-09T17:42:10.171 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.171 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.171 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:10.171 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.794041+0000", 2024-02-09T17:42:10.172 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2885000000000004e-05 2024-02-09T17:42:10.172 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:10.172 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:10.172 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:10.172 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.172 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.172 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25201.0:254074 4.c 4.79812fac (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-02-09T17:42:10.172 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:39:57.530922+0000", 2024-02-09T17:42:10.172 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 132.55839427800001, 2024-02-09T17:42:10.172 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0069641279999999, 2024-02-09T17:42:10.172 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:10.172 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-09T17:42:10.173 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:10.173 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25201", 2024-02-09T17:42:10.173 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:42:10.173 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 254074 2024-02-09T17:42:10.173 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.173 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:10.173 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.173 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:10.173 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:57.530922+0000", 2024-02-09T17:42:10.173 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:10.173 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.173 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.173 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:10.173 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:57.530922+0000", 2024-02-09T17:42:10.174 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:10.174 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.174 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.174 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:10.174 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:57.530923+0000", 2024-02-09T17:42:10.174 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.406e-06 2024-02-09T17:42:10.174 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.174 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.174 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:10.174 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:57.530928+0000", 2024-02-09T17:42:10.174 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.164e-06 2024-02-09T17:42:10.174 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.175 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.175 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:10.175 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:57.530939+0000", 2024-02-09T17:42:10.175 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1208e-05 2024-02-09T17:42:10.175 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.175 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.175 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:10.175 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:57.530946+0000", 2024-02-09T17:42:10.175 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0330000000000003e-06 2024-02-09T17:42:10.175 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.175 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.175 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:10.175 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:57.530997+0000", 2024-02-09T17:42:10.175 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0145999999999997e-05 2024-02-09T17:42:10.176 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.176 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.176 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:42:10.176 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:57.531006+0000", 2024-02-09T17:42:10.176 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0219999999999993e-06 2024-02-09T17:42:10.176 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.176 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.176 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:10.176 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:58.537872+0000", 2024-02-09T17:42:10.176 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.006866217 2024-02-09T17:42:10.176 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.176 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.176 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:10.176 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:58.537886+0000", 2024-02-09T17:42:10.177 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3932e-05 2024-02-09T17:42:10.177 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:10.177 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:10.177 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:10.177 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.177 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.177 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25201.0:254123 4.c 4.882473c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-02-09T17:42:10.177 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:39:57.580994+0000", 2024-02-09T17:42:10.177 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 132.508321999, 2024-02-09T17:42:10.177 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.95720749500000002, 2024-02-09T17:42:10.177 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:10.177 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-09T17:42:10.177 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:10.177 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25201", 2024-02-09T17:42:10.177 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:42:10.178 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 254123 2024-02-09T17:42:10.178 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.178 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:10.178 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.178 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:10.178 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:57.580994+0000", 2024-02-09T17:42:10.178 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:10.178 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.178 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.178 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:10.178 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:57.580994+0000", 2024-02-09T17:42:10.178 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:10.178 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.178 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.179 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:10.179 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:57.580995+0000", 2024-02-09T17:42:10.179 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2900000000000003e-07 2024-02-09T17:42:10.179 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.179 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.179 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:10.179 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:57.580997+0000", 2024-02-09T17:42:10.179 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8720000000000002e-06 2024-02-09T17:42:10.179 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.179 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.179 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:10.179 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:57.581008+0000", 2024-02-09T17:42:10.179 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0393e-05 2024-02-09T17:42:10.179 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.180 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.180 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:10.180 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:57.581013+0000", 2024-02-09T17:42:10.180 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.631e-06 2024-02-09T17:42:10.180 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.180 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.180 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:10.180 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:57.581069+0000", 2024-02-09T17:42:10.180 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5433000000000003e-05 2024-02-09T17:42:10.180 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.181 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.181 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:42:10.181 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:57.581079+0000", 2024-02-09T17:42:10.181 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0287e-05 2024-02-09T17:42:10.181 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.181 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.181 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:10.181 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:58.538193+0000", 2024-02-09T17:42:10.181 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.95711387699999995 2024-02-09T17:42:10.181 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.181 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.181 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:10.181 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:58.538201+0000", 2024-02-09T17:42:10.181 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3729999999999993e-06 2024-02-09T17:42:10.182 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:10.182 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:10.182 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:10.182 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.182 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.182 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:100592 5.1 5:8fae0b51:::benchmark_data_smithi179_115199_object100591:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e497)", 2024-02-09T17:42:10.182 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:42:06.355428+0000", 2024-02-09T17:42:10.182 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 3.7338884110000001, 2024-02-09T17:42:10.182 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.035283621000000001, 2024-02-09T17:42:10.182 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:10.182 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:42:10.182 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:10.182 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:42:10.182 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:42:10.183 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 100592 2024-02-09T17:42:10.183 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.183 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:10.183 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.183 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:10.183 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:06.355428+0000", 2024-02-09T17:42:10.183 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:10.183 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.183 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.183 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:10.183 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:06.355428+0000", 2024-02-09T17:42:10.183 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:10.183 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.183 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.183 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:10.184 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:06.355429+0000", 2024-02-09T17:42:10.184 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.114e-06 2024-02-09T17:42:10.184 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.184 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.184 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:10.184 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:06.355480+0000", 2024-02-09T17:42:10.184 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1193000000000003e-05 2024-02-09T17:42:10.184 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.184 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.184 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:10.184 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:06.355489+0000", 2024-02-09T17:42:10.184 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5310000000000007e-06 2024-02-09T17:42:10.184 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.185 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.185 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:10.185 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:06.355504+0000", 2024-02-09T17:42:10.185 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4904e-05 2024-02-09T17:42:10.185 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.185 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.185 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:10.185 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:06.355587+0000", 2024-02-09T17:42:10.185 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2528999999999995e-05 2024-02-09T17:42:10.185 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.185 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.185 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:10.185 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:06.355722+0000", 2024-02-09T17:42:10.185 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000135306 2024-02-09T17:42:10.186 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.186 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.186 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,7", 2024-02-09T17:42:10.186 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:06.355863+0000", 2024-02-09T17:42:10.186 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014065 2024-02-09T17:42:10.186 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.186 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.186 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:10.186 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:06.358857+0000", 2024-02-09T17:42:10.186 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0029942079999999999 2024-02-09T17:42:10.186 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.186 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.186 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:10.186 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:06.362887+0000", 2024-02-09T17:42:10.186 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0040297750000000002 2024-02-09T17:42:10.187 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.187 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.187 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:10.187 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:06.390625+0000", 2024-02-09T17:42:10.187 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.027738382999999998 2024-02-09T17:42:10.187 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.187 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.187 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:42:10.187 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:06.390672+0000", 2024-02-09T17:42:10.187 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6511999999999998e-05 2024-02-09T17:42:10.187 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.187 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.187 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:10.187 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:06.390711+0000", 2024-02-09T17:42:10.187 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9515999999999999e-05 2024-02-09T17:42:10.188 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:10.188 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:10.188 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:10.188 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.188 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.188 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:100952 5.1 5:8827816c:::benchmark_data_smithi179_115199_object100951:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e498)", 2024-02-09T17:42:10.188 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:42:06.694099+0000", 2024-02-09T17:42:10.188 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 3.395216928, 2024-02-09T17:42:10.188 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.037467378000000003, 2024-02-09T17:42:10.188 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:10.188 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:42:10.188 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:10.188 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:42:10.188 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:42:10.189 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 100952 2024-02-09T17:42:10.189 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.189 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:10.189 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.189 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:10.189 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:06.694099+0000", 2024-02-09T17:42:10.189 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:10.189 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.189 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.189 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:10.189 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:06.694099+0000", 2024-02-09T17:42:10.189 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:10.189 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.189 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.190 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:10.190 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:06.694101+0000", 2024-02-09T17:42:10.190 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8309999999999999e-06 2024-02-09T17:42:10.190 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.190 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.190 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:10.190 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:06.694158+0000", 2024-02-09T17:42:10.190 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7077000000000003e-05 2024-02-09T17:42:10.190 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.190 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.190 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:10.190 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:06.694171+0000", 2024-02-09T17:42:10.190 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2592000000000001e-05 2024-02-09T17:42:10.190 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.190 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.191 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:10.191 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:06.694179+0000", 2024-02-09T17:42:10.191 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2430000000000005e-06 2024-02-09T17:42:10.191 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.191 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.191 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:10.191 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:06.694219+0000", 2024-02-09T17:42:10.191 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0642999999999997e-05 2024-02-09T17:42:10.191 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.191 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.191 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:10.191 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:06.694338+0000", 2024-02-09T17:42:10.191 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000118603 2024-02-09T17:42:10.191 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.191 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.192 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,7", 2024-02-09T17:42:10.192 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:06.694435+0000", 2024-02-09T17:42:10.192 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6667000000000001e-05 2024-02-09T17:42:10.192 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.192 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.192 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:10.192 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:06.700678+0000", 2024-02-09T17:42:10.192 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0062437070000000002 2024-02-09T17:42:10.192 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.192 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.192 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:10.192 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:06.705468+0000", 2024-02-09T17:42:10.192 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0047895389999999998 2024-02-09T17:42:10.192 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.193 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.193 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:10.193 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:06.731467+0000", 2024-02-09T17:42:10.193 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.025998601999999999 2024-02-09T17:42:10.193 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.193 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.193 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:42:10.193 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:06.731523+0000", 2024-02-09T17:42:10.193 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6885000000000002e-05 2024-02-09T17:42:10.193 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.193 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.193 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:10.193 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:06.731566+0000", 2024-02-09T17:42:10.193 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2988999999999998e-05 2024-02-09T17:42:10.194 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:10.194 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:10.194 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:10.194 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.194 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.194 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:101583 5.1 5:8429c2cf:::benchmark_data_smithi179_115199_object101582:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e498)", 2024-02-09T17:42:10.194 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:42:07.228444+0000", 2024-02-09T17:42:10.194 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 2.8608720860000001, 2024-02-09T17:42:10.194 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.041170087000000001, 2024-02-09T17:42:10.194 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:10.194 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:42:10.194 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:10.194 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:42:10.195 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:42:10.195 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 101583 2024-02-09T17:42:10.195 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.195 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:10.195 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.195 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:10.195 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.228444+0000", 2024-02-09T17:42:10.195 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:10.195 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.195 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.195 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:10.195 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.228444+0000", 2024-02-09T17:42:10.196 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:10.196 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.196 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.196 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:10.196 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.228444+0000", 2024-02-09T17:42:10.196 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5000000000000003e-07 2024-02-09T17:42:10.196 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.196 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.196 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:10.196 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.228561+0000", 2024-02-09T17:42:10.196 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000116554 2024-02-09T17:42:10.196 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.196 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.196 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:10.197 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.228569+0000", 2024-02-09T17:42:10.197 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6229999999999997e-06 2024-02-09T17:42:10.197 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.197 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.197 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:10.197 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.228574+0000", 2024-02-09T17:42:10.197 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7629999999999997e-06 2024-02-09T17:42:10.197 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.197 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.199 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:10.199 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.232098+0000", 2024-02-09T17:42:10.199 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0035238219999999998 2024-02-09T17:42:10.199 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.199 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.199 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:10.199 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.232206+0000", 2024-02-09T17:42:10.200 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000107378 2024-02-09T17:42:10.200 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.200 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.200 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,7", 2024-02-09T17:42:10.200 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.232340+0000", 2024-02-09T17:42:10.200 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013467199999999999 2024-02-09T17:42:10.200 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.200 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.200 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:10.200 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.235284+0000", 2024-02-09T17:42:10.200 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0029436129999999999 2024-02-09T17:42:10.200 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.200 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.201 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:10.201 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.241281+0000", 2024-02-09T17:42:10.201 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0059968510000000001 2024-02-09T17:42:10.201 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.201 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.201 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:10.201 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.269524+0000", 2024-02-09T17:42:10.201 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.028243536 2024-02-09T17:42:10.201 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.201 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.201 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:42:10.201 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.269579+0000", 2024-02-09T17:42:10.201 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4561999999999997e-05 2024-02-09T17:42:10.202 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.202 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.202 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:10.202 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.269614+0000", 2024-02-09T17:42:10.202 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5163000000000002e-05 2024-02-09T17:42:10.202 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:10.202 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:10.202 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:10.202 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.202 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.202 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:101587 5.1 5:886e88df:::benchmark_data_smithi179_115199_object101586:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e498)", 2024-02-09T17:42:10.202 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:42:07.234466+0000", 2024-02-09T17:42:10.202 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 2.8548503730000001, 2024-02-09T17:42:10.203 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.035289369000000001, 2024-02-09T17:42:10.203 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:10.203 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:42:10.203 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:10.203 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:42:10.203 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:42:10.203 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 101587 2024-02-09T17:42:10.203 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.203 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:10.203 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.203 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:10.203 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.234466+0000", 2024-02-09T17:42:10.203 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:10.203 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.204 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.204 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:10.204 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.234466+0000", 2024-02-09T17:42:10.204 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:10.204 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.204 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.204 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:10.204 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.234467+0000", 2024-02-09T17:42:10.204 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.39e-06 2024-02-09T17:42:10.204 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.204 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.204 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:10.204 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.234501+0000", 2024-02-09T17:42:10.205 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3929e-05 2024-02-09T17:42:10.205 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.205 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.205 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:10.205 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.234521+0000", 2024-02-09T17:42:10.205 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0482000000000001e-05 2024-02-09T17:42:10.205 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.205 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.205 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:10.205 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.234530+0000", 2024-02-09T17:42:10.205 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3250000000000008e-06 2024-02-09T17:42:10.205 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.205 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.205 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:10.206 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.237587+0000", 2024-02-09T17:42:10.206 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0030573150000000001 2024-02-09T17:42:10.206 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.206 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.206 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:10.206 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.237715+0000", 2024-02-09T17:42:10.206 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000128019 2024-02-09T17:42:10.206 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.206 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.206 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,7", 2024-02-09T17:42:10.206 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.237872+0000", 2024-02-09T17:42:10.206 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015720400000000001 2024-02-09T17:42:10.206 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.206 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.207 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:10.207 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.241643+0000", 2024-02-09T17:42:10.207 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0037708120000000001 2024-02-09T17:42:10.207 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.207 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.207 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:10.207 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.242132+0000", 2024-02-09T17:42:10.207 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00048856799999999999 2024-02-09T17:42:10.207 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.207 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.207 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:10.207 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.269683+0000", 2024-02-09T17:42:10.207 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.027550959 2024-02-09T17:42:10.207 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.207 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.208 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:42:10.208 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.269725+0000", 2024-02-09T17:42:10.208 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2716000000000002e-05 2024-02-09T17:42:10.208 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.208 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.208 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:10.208 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.269755+0000", 2024-02-09T17:42:10.208 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9649999999999999e-05 2024-02-09T17:42:10.208 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:10.208 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:10.208 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:10.208 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.208 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.209 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:101643 5.1 5:82018690:::benchmark_data_smithi179_115199_object101642:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e498)", 2024-02-09T17:42:10.209 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:42:07.289957+0000", 2024-02-09T17:42:10.209 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 2.7993586960000001, 2024-02-09T17:42:10.209 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.040346298000000003, 2024-02-09T17:42:10.209 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:10.209 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:42:10.209 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:10.209 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:42:10.209 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:42:10.209 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 101643 2024-02-09T17:42:10.209 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.209 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:10.209 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.209 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:10.209 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.289957+0000", 2024-02-09T17:42:10.210 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:10.210 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.210 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.210 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:10.210 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.289957+0000", 2024-02-09T17:42:10.210 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:10.210 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.210 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.210 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:10.210 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.289959+0000", 2024-02-09T17:42:10.210 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0420000000000001e-06 2024-02-09T17:42:10.210 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.210 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.210 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:10.210 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.290004+0000", 2024-02-09T17:42:10.211 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4274999999999999e-05 2024-02-09T17:42:10.211 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.211 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.211 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:10.211 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.290025+0000", 2024-02-09T17:42:10.211 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1182000000000001e-05 2024-02-09T17:42:10.211 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.211 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.211 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:10.211 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.290036+0000", 2024-02-09T17:42:10.211 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1572e-05 2024-02-09T17:42:10.211 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.211 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.211 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:10.212 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.300274+0000", 2024-02-09T17:42:10.212 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010237688 2024-02-09T17:42:10.212 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.212 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.212 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:10.212 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.300387+0000", 2024-02-09T17:42:10.212 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000113294 2024-02-09T17:42:10.212 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.212 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.212 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,7", 2024-02-09T17:42:10.212 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.300519+0000", 2024-02-09T17:42:10.212 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013142199999999999 2024-02-09T17:42:10.212 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.212 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.212 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:10.213 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.307951+0000", 2024-02-09T17:42:10.213 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.007432719 2024-02-09T17:42:10.213 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.213 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.213 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:10.213 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.312055+0000", 2024-02-09T17:42:10.213 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041038769999999997 2024-02-09T17:42:10.213 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.213 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.213 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:10.213 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.330202+0000", 2024-02-09T17:42:10.213 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018146829 2024-02-09T17:42:10.213 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.213 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.214 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:42:10.214 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.330267+0000", 2024-02-09T17:42:10.214 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4367000000000002e-05 2024-02-09T17:42:10.214 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.214 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.214 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:10.214 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.330304+0000", 2024-02-09T17:42:10.214 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7030999999999998e-05 2024-02-09T17:42:10.214 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:10.214 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:10.214 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:10.214 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.214 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.214 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:101793 5.1 5:8ee18d9a:::benchmark_data_smithi179_115199_object101792:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e498)", 2024-02-09T17:42:10.215 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:42:07.435208+0000", 2024-02-09T17:42:10.215 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 2.654107963, 2024-02-09T17:42:10.215 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.038755841999999999, 2024-02-09T17:42:10.215 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:10.215 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:42:10.215 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:10.215 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:42:10.215 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:42:10.215 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 101793 2024-02-09T17:42:10.215 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.215 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:10.215 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.215 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:10.215 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.435208+0000", 2024-02-09T17:42:10.216 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:10.216 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.216 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.216 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:10.216 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.435208+0000", 2024-02-09T17:42:10.216 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:10.216 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.216 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.216 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:10.216 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.435210+0000", 2024-02-09T17:42:10.216 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1229999999999998e-06 2024-02-09T17:42:10.216 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.216 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.217 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:10.217 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.435252+0000", 2024-02-09T17:42:10.217 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2086000000000003e-05 2024-02-09T17:42:10.217 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.217 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.217 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:10.217 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.435265+0000", 2024-02-09T17:42:10.217 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.294e-05 2024-02-09T17:42:10.217 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.217 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.217 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:10.217 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.435275+0000", 2024-02-09T17:42:10.217 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3850000000000008e-06 2024-02-09T17:42:10.217 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.217 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.218 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:10.218 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.435424+0000", 2024-02-09T17:42:10.218 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014960599999999999 2024-02-09T17:42:10.218 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.218 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.218 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:10.218 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.435563+0000", 2024-02-09T17:42:10.218 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013899100000000001 2024-02-09T17:42:10.218 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.218 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.218 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,7", 2024-02-09T17:42:10.218 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.435710+0000", 2024-02-09T17:42:10.218 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000146556 2024-02-09T17:42:10.218 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.218 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.219 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:10.219 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.442296+0000", 2024-02-09T17:42:10.219 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0065858189999999997 2024-02-09T17:42:10.219 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.219 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.219 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:10.219 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.443730+0000", 2024-02-09T17:42:10.219 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001434891 2024-02-09T17:42:10.219 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.219 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.219 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:10.219 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.473861+0000", 2024-02-09T17:42:10.219 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.030130724000000001 2024-02-09T17:42:10.219 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.220 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.220 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:42:10.220 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.473922+0000", 2024-02-09T17:42:10.220 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0785000000000002e-05 2024-02-09T17:42:10.220 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.220 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.220 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:10.220 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.473964+0000", 2024-02-09T17:42:10.220 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1936e-05 2024-02-09T17:42:10.220 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:10.220 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:10.220 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:10.220 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.220 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.220 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:102259 5.1 5:868c8951:::benchmark_data_smithi179_115199_object102258:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e498)", 2024-02-09T17:42:10.221 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:42:07.868720+0000", 2024-02-09T17:42:10.221 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 2.2205962320000001, 2024-02-09T17:42:10.221 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.037250710999999999, 2024-02-09T17:42:10.221 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:10.221 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:42:10.221 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:10.221 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:42:10.221 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:42:10.221 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 102259 2024-02-09T17:42:10.221 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.221 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:10.221 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.221 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:10.221 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.868720+0000", 2024-02-09T17:42:10.222 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:10.222 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.222 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.222 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:10.222 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.868720+0000", 2024-02-09T17:42:10.222 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:10.222 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.222 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.222 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:10.222 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.868721+0000", 2024-02-09T17:42:10.222 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6500000000000001e-06 2024-02-09T17:42:10.222 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.222 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.222 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:10.223 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.868774+0000", 2024-02-09T17:42:10.223 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2287000000000003e-05 2024-02-09T17:42:10.223 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.223 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.223 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:10.223 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.868787+0000", 2024-02-09T17:42:10.223 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2979999999999999e-05 2024-02-09T17:42:10.223 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.223 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.223 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:10.223 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.868795+0000", 2024-02-09T17:42:10.223 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3969999999999994e-06 2024-02-09T17:42:10.223 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.223 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.224 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:10.224 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.868849+0000", 2024-02-09T17:42:10.224 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3631999999999997e-05 2024-02-09T17:42:10.224 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.224 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.224 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:10.224 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.869003+0000", 2024-02-09T17:42:10.224 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000154031 2024-02-09T17:42:10.224 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.224 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.224 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,7", 2024-02-09T17:42:10.224 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.869151+0000", 2024-02-09T17:42:10.224 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014875199999999999 2024-02-09T17:42:10.225 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.225 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.225 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:10.225 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.877371+0000", 2024-02-09T17:42:10.225 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0082199520000000009 2024-02-09T17:42:10.225 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.225 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.225 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:10.225 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.878034+0000", 2024-02-09T17:42:10.225 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00066282000000000005 2024-02-09T17:42:10.225 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.225 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.225 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:10.225 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.905850+0000", 2024-02-09T17:42:10.226 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.027815329999999999 2024-02-09T17:42:10.226 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.226 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.226 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:42:10.226 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.905910+0000", 2024-02-09T17:42:10.226 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0476000000000002e-05 2024-02-09T17:42:10.226 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:10.226 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:10.226 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:10.226 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:07.905970+0000", 2024-02-09T17:42:10.226 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0404e-05 2024-02-09T17:42:10.226 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:10.226 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:10.226 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:10.226 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:10.227 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:10.227 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:42:10.228 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:09 smithi179 ceph-mon[43975]: pgmap v1260: 37 pgs: 37 active+clean; 6.1 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-02-09T17:42:10.228 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:09 smithi162 ceph-mon[44304]: pgmap v1260: 37 pgs: 37 active+clean; 6.1 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-02-09T17:42:10.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:10.229 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:42:10.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:42:10.330 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:42:10.360 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:10.390 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:42:10.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:10.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:42:10.497 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:42:10.534 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:10.559 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:42:10.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:10.563 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:42:10.664 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:42:10.685 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:42:10.729 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:10.897 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-09T17:42:10.962 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 66 16 105763 105747 100.122 69.5625 0.0158768 0.00998322 2024-02-09T17:42:10.979 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:42:10.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:10.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:42:11.085 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:42:11.125 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:11.150 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:42:11.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:11.156 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:42:11.256 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:42:11.343 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:11.351 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:42:11.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:11.391 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:42:11.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:42:11.493 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:42:11.558 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:11.614 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:42:11.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:11.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:42:11.721 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:42:11.758 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:11.892 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is set 2024-02-09T17:42:11.893 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:11 smithi162 ceph-mon[44304]: pgmap v1261: 37 pgs: 37 active+clean; 6.3 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-02-09T17:42:11.893 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:11 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/708013379' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:42:11.893 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:11 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:42:11.962 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 67 16 106819 106803 99.6132 66 0.00456006 0.0100346 2024-02-09T17:42:12.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:11 smithi179 ceph-mon[43975]: pgmap v1261: 37 pgs: 37 active+clean; 6.3 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-02-09T17:42:12.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:11 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/708013379' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:42:12.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:11 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:42:12.214 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:11 smithi099 ceph-mon[39420]: pgmap v1261: 37 pgs: 37 active+clean; 6.3 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-02-09T17:42:12.215 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:11 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/708013379' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:42:12.215 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:11 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:42:12.239 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:42:12.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:12.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:42:12.348 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:42:12.390 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:12.418 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:42:12.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:12.424 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:42:12.525 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:42:12.562 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:12.904 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:12 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:42:12.904 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:12 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:42:12.904 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:12 smithi099 ceph-mon[39420]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:42:12.904 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:12 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:42:12.904 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:12 smithi099 ceph-mon[39420]: osdmap e499: 12 total, 12 up, 11 in 2024-02-09T17:42:12.966 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 68 16 107945 107929 99.1832 70.375 0.0105829 0.010079 2024-02-09T17:42:13.029 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:12 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:42:13.029 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:12 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:42:13.029 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:12 smithi162 ceph-mon[44304]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:42:13.029 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:12 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:42:13.029 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:12 smithi162 ceph-mon[44304]: osdmap e499: 12 total, 12 up, 11 in 2024-02-09T17:42:13.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:12 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:42:13.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:12 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:42:13.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:12 smithi179 ceph-mon[43975]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:42:13.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:12 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:42:13.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:12 smithi179 ceph-mon[43975]: osdmap e499: 12 total, 12 up, 11 in 2024-02-09T17:42:13.701 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:42:13.716 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:42:13.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:13.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:42:13.724 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:42:13.746 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:42:13.746 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-02-09T17:42:13.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.746 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:108756 5.6 5:61480fa5:::benchmark_data_smithi179_115199_object108755:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-02-09T17:42:13.746 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:42:13.736475+0000", 2024-02-09T17:42:13.746 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0081244660000000003, 2024-02-09T17:42:13.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0081492289999999992, 2024-02-09T17:42:13.746 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:13.746 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-02-09T17:42:13.746 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:13.746 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:42:13.746 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:42:13.747 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 108756 2024-02-09T17:42:13.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.747 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:13.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.747 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:13.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.736475+0000", 2024-02-09T17:42:13.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:13.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.747 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:13.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.736475+0000", 2024-02-09T17:42:13.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:13.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.748 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:13.748 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.736476+0000", 2024-02-09T17:42:13.748 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.59e-06 2024-02-09T17:42:13.748 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.748 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.748 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:13.748 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.739906+0000", 2024-02-09T17:42:13.748 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034294740000000001 2024-02-09T17:42:13.748 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.748 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.748 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:13.748 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.739922+0000", 2024-02-09T17:42:13.748 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6218999999999998e-05 2024-02-09T17:42:13.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.749 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.749 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:13.749 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.739945+0000", 2024-02-09T17:42:13.749 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2932999999999999e-05 2024-02-09T17:42:13.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.749 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.749 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:13.749 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.740114+0000", 2024-02-09T17:42:13.749 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016959599999999999 2024-02-09T17:42:13.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.749 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.749 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:13.749 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.740267+0000", 2024-02-09T17:42:13.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015286499999999999 2024-02-09T17:42:13.750 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.750 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7,10", 2024-02-09T17:42:13.750 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.740415+0000", 2024-02-09T17:42:13.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000147529 2024-02-09T17:42:13.750 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:13.750 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:13.750 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:13.750 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.750 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25528.0:108759 5.4 e499/465 5:27be7e20:::benchmark_data_smithi179_115199_object108758:head v 499'6716, mlcod=499'6714)", 2024-02-09T17:42:13.750 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:42:13.739855+0000", 2024-02-09T17:42:13.750 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.004743588, 2024-02-09T17:42:13.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0048181350000000003, 2024-02-09T17:42:13.751 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:13.751 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-09T17:42:13.751 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:13.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:13.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.739855+0000", 2024-02-09T17:42:13.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:13.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:13.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.739855+0000", 2024-02-09T17:42:13.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:13.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.752 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:13.752 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.739856+0000", 2024-02-09T17:42:13.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7700000000000003e-07 2024-02-09T17:42:13.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.752 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:13.752 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.739928+0000", 2024-02-09T17:42:13.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1954999999999998e-05 2024-02-09T17:42:13.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.752 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:13.752 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.739951+0000", 2024-02-09T17:42:13.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.23e-05 2024-02-09T17:42:13.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.753 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:13.753 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.739961+0000", 2024-02-09T17:42:13.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.91e-06 2024-02-09T17:42:13.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.753 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:13.753 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.740033+0000", 2024-02-09T17:42:13.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2553999999999997e-05 2024-02-09T17:42:13.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.753 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:13.753 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.740055+0000", 2024-02-09T17:42:13.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1449000000000001e-05 2024-02-09T17:42:13.753 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:13.754 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:13.754 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:13.754 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.754 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.754 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:108757 5.6 5:65bfe895:::benchmark_data_smithi179_115199_object108756:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-02-09T17:42:13.754 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:42:13.739969+0000", 2024-02-09T17:42:13.754 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0046295729999999997, 2024-02-09T17:42:13.754 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0047423120000000003, 2024-02-09T17:42:13.754 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:13.754 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-02-09T17:42:13.754 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:13.754 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:42:13.754 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:42:13.755 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 108757 2024-02-09T17:42:13.755 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.755 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:13.755 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.755 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:13.755 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.739969+0000", 2024-02-09T17:42:13.755 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:13.755 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.755 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.755 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:13.755 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.739969+0000", 2024-02-09T17:42:13.755 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:13.755 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.755 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.755 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:13.756 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.739971+0000", 2024-02-09T17:42:13.756 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0270000000000001e-06 2024-02-09T17:42:13.756 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.756 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.756 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:13.756 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.740012+0000", 2024-02-09T17:42:13.756 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1041000000000002e-05 2024-02-09T17:42:13.756 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.756 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:13.757 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.740020+0000", 2024-02-09T17:42:13.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1140000000000006e-06 2024-02-09T17:42:13.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:13.757 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.740025+0000", 2024-02-09T17:42:13.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7100000000000004e-06 2024-02-09T17:42:13.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:13.757 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.743211+0000", 2024-02-09T17:42:13.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0031859480000000001 2024-02-09T17:42:13.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.758 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:13.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.743350+0000", 2024-02-09T17:42:13.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013833699999999999 2024-02-09T17:42:13.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.758 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7,10", 2024-02-09T17:42:13.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.743455+0000", 2024-02-09T17:42:13.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010505099999999999 2024-02-09T17:42:13.758 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:13.758 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:13.758 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:13.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.759 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:108760 5.6 5.7e394ad6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-02-09T17:42:13.759 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:42:13.740034+0000", 2024-02-09T17:42:13.759 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0045651140000000003, 2024-02-09T17:42:13.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0047181339999999997, 2024-02-09T17:42:13.759 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:13.759 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "queued for pg", 2024-02-09T17:42:13.759 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:13.759 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:42:13.759 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:42:13.759 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 108760 2024-02-09T17:42:13.759 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.759 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:13.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.760 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:13.760 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.740034+0000", 2024-02-09T17:42:13.760 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:13.760 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.760 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.760 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:13.760 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.740034+0000", 2024-02-09T17:42:13.760 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:13.760 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.760 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.760 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:13.761 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.740034+0000", 2024-02-09T17:42:13.761 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3099999999999998e-07 2024-02-09T17:42:13.761 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.761 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.761 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:13.761 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.740060+0000", 2024-02-09T17:42:13.761 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5292000000000001e-05 2024-02-09T17:42:13.761 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.761 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.761 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:13.761 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.740066+0000", 2024-02-09T17:42:13.762 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5570000000000001e-06 2024-02-09T17:42:13.762 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.762 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.762 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:13.762 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.740071+0000", 2024-02-09T17:42:13.762 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1220000000000003e-06 2024-02-09T17:42:13.762 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:13.762 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:13.762 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:13.762 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.762 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.763 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25528.0:108762 5.4 e499/465 5:20230213:::benchmark_data_smithi179_115199_object108761:head v 499'6717, mlcod=499'6714)", 2024-02-09T17:42:13.763 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:42:13.743369+0000", 2024-02-09T17:42:13.763 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0012302680000000001, 2024-02-09T17:42:13.763 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001416856, 2024-02-09T17:42:13.763 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:13.763 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-09T17:42:13.763 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:13.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.763 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:13.763 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.743369+0000", 2024-02-09T17:42:13.763 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:13.764 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.764 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.764 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:13.764 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.743369+0000", 2024-02-09T17:42:13.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:13.764 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.764 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.764 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:13.764 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.743370+0000", 2024-02-09T17:42:13.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5000000000000001e-07 2024-02-09T17:42:13.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:13.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.743436+0000", 2024-02-09T17:42:13.765 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6171000000000001e-05 2024-02-09T17:42:13.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:13.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.743446+0000", 2024-02-09T17:42:13.765 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7399999999999999e-06 2024-02-09T17:42:13.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:13.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.743455+0000", 2024-02-09T17:42:13.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3179999999999996e-06 2024-02-09T17:42:13.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.766 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:13.766 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.743510+0000", 2024-02-09T17:42:13.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5479999999999997e-05 2024-02-09T17:42:13.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.766 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:13.766 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.743530+0000", 2024-02-09T17:42:13.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0029000000000001e-05 2024-02-09T17:42:13.766 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:13.766 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:13.767 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:13.767 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.767 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.767 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25528.0:108761 5.b e499/465 5:d41528d3:::benchmark_data_smithi179_115199_object108760:head v 499'6775, mlcod=499'6773)", 2024-02-09T17:42:13.767 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:42:13.744046+0000", 2024-02-09T17:42:13.767 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.00055301099999999995, 2024-02-09T17:42:13.767 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00077290200000000001, 2024-02-09T17:42:13.767 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:13.767 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-09T17:42:13.767 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:13.768 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.768 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:13.768 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.744046+0000", 2024-02-09T17:42:13.768 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:13.768 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.768 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.768 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:13.768 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.744046+0000", 2024-02-09T17:42:13.768 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:13.768 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.768 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.769 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:13.769 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.744047+0000", 2024-02-09T17:42:13.769 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8999999999999996e-07 2024-02-09T17:42:13.769 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.769 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.769 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:13.769 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.744070+0000", 2024-02-09T17:42:13.769 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3399000000000001e-05 2024-02-09T17:42:13.769 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.769 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.769 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:13.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.744080+0000", 2024-02-09T17:42:13.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8369999999999992e-06 2024-02-09T17:42:13.770 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:13.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.744085+0000", 2024-02-09T17:42:13.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0629999999999996e-06 2024-02-09T17:42:13.770 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:13.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.744134+0000", 2024-02-09T17:42:13.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8563000000000002e-05 2024-02-09T17:42:13.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.771 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:13.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.744154+0000", 2024-02-09T17:42:13.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0540999999999999e-05 2024-02-09T17:42:13.771 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:13.771 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:13.771 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:13.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.771 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.25528.0:108756 5.6 e499/465)", 2024-02-09T17:42:13.772 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:42:13.744326+0000", 2024-02-09T17:42:13.772 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.00027350299999999997, 2024-02-09T17:42:13.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00052487699999999998, 2024-02-09T17:42:13.772 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:13.772 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "queued for pg", 2024-02-09T17:42:13.772 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:13.772 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.772 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:13.772 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.744326+0000", 2024-02-09T17:42:13.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:13.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.773 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:13.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.744326+0000", 2024-02-09T17:42:13.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:13.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.773 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:13.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.744326+0000", 2024-02-09T17:42:13.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0999999999999998e-07 2024-02-09T17:42:13.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.773 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:13.774 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.744330+0000", 2024-02-09T17:42:13.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5030000000000002e-06 2024-02-09T17:42:13.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.774 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.774 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:13.774 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.744342+0000", 2024-02-09T17:42:13.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2148e-05 2024-02-09T17:42:13.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.774 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.774 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:13.774 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.744348+0000", 2024-02-09T17:42:13.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9560000000000002e-06 2024-02-09T17:42:13.775 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:13.775 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:13.775 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:13.775 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.775 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.775 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25528.0:108763 5.b e499/465)", 2024-02-09T17:42:13.775 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:42:13.744449+0000", 2024-02-09T17:42:13.775 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.00014959399999999999, 2024-02-09T17:42:13.775 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00042602200000000002, 2024-02-09T17:42:13.775 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:13.775 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "queued for pg", 2024-02-09T17:42:13.776 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:13.776 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.776 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:13.776 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.744449+0000", 2024-02-09T17:42:13.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:13.776 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.776 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.776 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:13.776 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.744449+0000", 2024-02-09T17:42:13.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:13.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:13.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.744450+0000", 2024-02-09T17:42:13.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0100000000000005e-07 2024-02-09T17:42:13.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:13.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.744525+0000", 2024-02-09T17:42:13.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4603000000000004e-05 2024-02-09T17:42:13.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:13.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.744533+0000", 2024-02-09T17:42:13.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7110000000000001e-06 2024-02-09T17:42:13.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:13.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:13.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:13.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.744539+0000", 2024-02-09T17:42:13.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6230000000000002e-06 2024-02-09T17:42:13.778 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:13.778 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:13.778 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:13.779 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:13.779 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-09T17:42:13.779 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 8 2024-02-09T17:42:13.779 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:42:13.779 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:42:13.780 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:42:13.795 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:42:13.820 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:42:13.821 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:42:13.854 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:13.856 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:42:13.881 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:42:13.888 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:13.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:42:13.889 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:42:13.926 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:42:13.960 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:42:13.962 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 69 16 108993 108977 98.695 65.5 0.0112206 0.0101278 2024-02-09T17:42:13.984 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:42:13.990 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:42:13.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:13 smithi099 ceph-mon[39420]: pgmap v1263: 37 pgs: 37 active+clean; 6.4 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-02-09T17:42:13.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:13 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3234909141' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:42:14.008 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:42:14.009 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:42:14.009 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:42:14.009 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:42:14.009 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:42:14.010 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:42:14.010 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:42:14.010 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:42:14.010 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:42:14.010 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:42:14.010 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:42:14.010 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:42:14.010 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:42:14.063 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:13 smithi162 ceph-mon[44304]: pgmap v1263: 37 pgs: 37 active+clean; 6.4 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-02-09T17:42:14.063 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:13 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3234909141' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:42:14.065 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:14.155 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:42:14.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:14.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:42:14.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:13 smithi179 ceph-mon[43975]: pgmap v1263: 37 pgs: 37 active+clean; 6.4 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-02-09T17:42:14.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:13 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3234909141' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:42:14.269 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:42:14.308 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:14.340 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:42:14.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:14.344 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:42:14.445 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:42:14.483 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:14.511 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-09T17:42:14.511 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:42:14.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:14.516 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:42:14.617 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:42:14.655 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:14.687 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:42:14.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:14.694 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:42:14.795 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:42:14.824 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:14.854 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:42:14.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:14.858 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:42:14.943 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:42:14.959 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:42:14.962 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 70 16 110078 110062 98.2537 67.8125 0.0182666 0.0101732 2024-02-09T17:42:14.987 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:15.010 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:42:15.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:15.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:42:15.084 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset noscrub 2024-02-09T17:42:15.115 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:42:15.163 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:15.208 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:42:15.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:15.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:42:15.230 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] 2024-02-09T17:42:15.230 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:42:15.230 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-02-09T17:42:15.231 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph pg dump --format=json 2024-02-09T17:42:15.315 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:42:15.351 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:15.382 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:42:15.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:15.385 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:42:15.422 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:42:15.486 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:42:15.539 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:15.914 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:15 smithi162 ceph-mon[44304]: pgmap v1264: 37 pgs: 37 active+clean; 6.5 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-02-09T17:42:15.962 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 71 16 111161 111145 97.8232 67.6875 0.0150947 0.0102183 2024-02-09T17:42:16.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:15 smithi179 ceph-mon[43975]: pgmap v1264: 37 pgs: 37 active+clean; 6.5 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-02-09T17:42:16.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:15 smithi099 ceph-mon[39420]: pgmap v1264: 37 pgs: 37 active+clean; 6.5 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-02-09T17:42:16.283 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:42:16.310 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:42:16.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:16.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:42:16.419 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:42:16.451 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:16.479 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:42:16.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:16.483 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:42:16.583 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:42:16.645 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:16.707 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:42:16.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:16.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:42:16.816 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:42:16.849 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:16.892 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:42:16.963 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 72 16 112232 112216 97.3942 66.9375 0.00688437 0.0102639 2024-02-09T17:42:17.093 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:42:17.093 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:42:17.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:17 smithi179 ceph-mon[43975]: pgmap v1265: 37 pgs: 37 active+clean; 6.7 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-02-09T17:42:17.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:17 smithi162 ceph-mon[44304]: pgmap v1265: 37 pgs: 37 active+clean; 6.7 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-02-09T17:42:17.963 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 73 15 113291 113276 96.9675 66.25 0.021535 0.0103092 2024-02-09T17:42:17.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:17 smithi099 ceph-mon[39420]: pgmap v1265: 37 pgs: 37 active+clean; 6.7 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-02-09T17:42:18.005 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:42:18.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:18.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:42:18.112 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:42:18.144 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:18.174 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:42:18.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:18.180 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:42:18.281 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:42:18.347 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:18.511 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:42:18.963 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 74 16 114336 114320 96.5389 65.25 0.036289 0.0103545 2024-02-09T17:42:19.881 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:42:19.886 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-02-09T17:42:19.963 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 75 16 115374 115358 96.1166 64.875 0.0125298 0.0103999 2024-02-09T17:42:20.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:19 smithi179 ceph-mon[43975]: pgmap v1266: 37 pgs: 37 active+clean; 6.8 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-02-09T17:42:20.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:19 smithi162 ceph-mon[44304]: pgmap v1266: 37 pgs: 37 active+clean; 6.8 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-02-09T17:42:20.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:19 smithi099 ceph-mon[39420]: pgmap v1266: 37 pgs: 37 active+clean; 6.8 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-02-09T17:42:20.256 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-09T17:42:20.285 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:42:20.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:20.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:42:20.390 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:42:20.431 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:20.723 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:42:20.963 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 76 16 116368 116352 95.6693 62.125 0.00906616 0.0104485 2024-02-09T17:42:21.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:20 smithi179 ceph-mon[43975]: from='client.15873 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:42:21.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:20 smithi162 ceph-mon[44304]: from='client.15873 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:42:21.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:20 smithi099 ceph-mon[39420]: from='client.15873 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:42:21.963 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 77 16 117361 117345 95.2329 62.0625 0.00796598 0.0104965 2024-02-09T17:42:22.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:21 smithi179 ceph-mon[43975]: pgmap v1267: 37 pgs: 37 active+clean; 6.9 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-02-09T17:42:22.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:21 smithi162 ceph-mon[44304]: pgmap v1267: 37 pgs: 37 active+clean; 6.9 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-02-09T17:42:22.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:21 smithi099 ceph-mon[39420]: pgmap v1267: 37 pgs: 37 active+clean; 6.9 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-02-09T17:42:22.963 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 78 16 118406 118390 94.8492 65.3125 0.0154437 0.0105389 2024-02-09T17:42:23.963 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 79 16 119472 119456 94.4919 66.625 0.0119417 0.0105793 2024-02-09T17:42:24.067 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:23 smithi162 ceph-mon[44304]: pgmap v1268: 37 pgs: 37 active+clean; 7.1 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-02-09T17:42:24.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:23 smithi179 ceph-mon[43975]: pgmap v1268: 37 pgs: 37 active+clean; 7.1 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-02-09T17:42:24.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:23 smithi099 ceph-mon[39420]: pgmap v1268: 37 pgs: 37 active+clean; 7.1 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-02-09T17:42:24.771 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:42:24.771 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:42:24.963 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:2024-02-09T17:42:24.963153+0000 min lat: 0.00110706 max lat: 2.34848 avg lat: 0.0106317 2024-02-09T17:42:24.963 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-09T17:42:24.964 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 80 16 120382 120366 94.0217 56.875 0.0201903 0.0106317 2024-02-09T17:42:25.543 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":1266,"stamp":"2024-02-09T17:42:18.812107+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7275020832,"num_objects":111001,"num_object_clones":0,"num_object_copies":333003,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":111001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":110999,"num_write_kb":7103936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3629,"num_bytes_recovered":238499424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113918,"ondisk_log_size":113918,"up":111,"acting":111,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":112,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":26017556,"kb_used_data":21244036,"kb_used_omap":228,"kb_used_meta":4773019,"kb_avail":1005179116,"statfs":{"total":1055945392128,"available":1029303414784,"internally_reserved":0,"allocated":21753892864,"data_stored":21748137579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":233532,"internal_metadata":4887572420},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,6,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":724828160,"num_objects":11060,"num_object_clones":0,"num_object_copies":33180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11060,"num_write_kb":707840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002749"},"pg_stats":[{"pgid":"1.14","version":"63'139","reported_seq":956,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-02-09T17:42:11.889001+0000","last_change":"2024-02-09T17:41:08.077049+0000","last_active":"2024-02-09T17:42:11.889001+0000","last_peered":"2024-02-09T17:42:11.889001+0000","last_clean":"2024-02-09T17:42:11.889001+0000","last_became_active":"2024-02-09T17:39:27.455644+0000","last_became_peered":"2024-02-09T17:39:27.455644+0000","last_unstale":"2024-02-09T17:42:11.889001+0000","last_undegraded":"2024-02-09T17:42:11.889001+0000","last_fullsized":"2024-02-09T17:42:11.889001+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:08.076894+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:38:26.968892+0000","last_clean_scrub_stamp":"2024-02-09T17:41:08.076894+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:16.380706+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"499'6997","reported_seq":7075,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-02-09T17:42:17.891477+0000","last_change":"2024-02-09T17:40:48.780201+0000","last_active":"2024-02-09T17:42:17.891477+0000","last_peered":"2024-02-09T17:42:17.891477+0000","last_clean":"2024-02-09T17:42:17.891477+0000","last_became_active":"2024-02-09T17:40:48.779925+0000","last_became_peered":"2024-02-09T17:40:48.779925+0000","last_unstale":"2024-02-09T17:42:17.891477+0000","last_undegraded":"2024-02-09T17:42:17.891477+0000","last_fullsized":"2024-02-09T17:42:17.891477+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:40:47.763277+0000","objects_scrubbed":0,"log_size":6997,"log_dups_size":0,"ondisk_log_size":6997,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:19.499252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":458555392,"num_objects":6997,"num_object_clones":0,"num_object_copies":20991,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6997,"num_write_kb":447808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,8],"acting":[10,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"63'139","reported_seq":1308,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-02-09T17:42:11.888607+0000","last_change":"2024-02-09T17:41:14.521470+0000","last_active":"2024-02-09T17:42:11.888607+0000","last_peered":"2024-02-09T17:42:11.888607+0000","last_clean":"2024-02-09T17:42:11.888607+0000","last_became_active":"2024-02-09T17:41:14.521079+0000","last_became_peered":"2024-02-09T17:41:14.521079+0000","last_unstale":"2024-02-09T17:42:11.888607+0000","last_undegraded":"2024-02-09T17:42:11.888607+0000","last_fullsized":"2024-02-09T17:42:11.888607+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:10.115197+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:41:10.115197+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:18.769273+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"499'6783","reported_seq":6861,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-02-09T17:42:14.090321+0000","last_change":"2024-02-09T17:40:48.783610+0000","last_active":"2024-02-09T17:42:14.090321+0000","last_peered":"2024-02-09T17:42:14.090321+0000","last_clean":"2024-02-09T17:42:14.090321+0000","last_became_active":"2024-02-09T17:40:48.783218+0000","last_became_peered":"2024-02-09T17:40:48.783218+0000","last_unstale":"2024-02-09T17:42:14.090321+0000","last_undegraded":"2024-02-09T17:42:14.090321+0000","last_fullsized":"2024-02-09T17:42:14.090321+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:40:47.763277+0000","objects_scrubbed":0,"log_size":6783,"log_dups_size":0,"ondisk_log_size":6783,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:16.467939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":444530688,"num_objects":6783,"num_object_clones":0,"num_object_copies":20349,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6783,"num_write_kb":434112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"63'139","reported_seq":1045,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-02-09T17:42:11.888892+0000","last_change":"2024-02-09T17:41:40.216180+0000","last_active":"2024-02-09T17:42:11.888892+0000","last_peered":"2024-02-09T17:42:11.888892+0000","last_clean":"2024-02-09T17:42:11.888892+0000","last_became_active":"2024-02-09T17:40:22.484952+0000","last_became_peered":"2024-02-09T17:40:22.484952+0000","last_unstale":"2024-02-09T17:42:11.888892+0000","last_undegraded":"2024-02-09T17:42:11.888892+0000","last_fullsized":"2024-02-09T17:42:11.888892+0000","mapping_epoch":454,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":455,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:40.216046+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:41:40.216046+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:51.445943+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"499'6759","reported_seq":6836,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-02-09T17:42:14.115429+0000","last_change":"2024-02-09T17:40:48.781806+0000","last_active":"2024-02-09T17:42:14.115429+0000","last_peered":"2024-02-09T17:42:14.115429+0000","last_clean":"2024-02-09T17:42:14.115429+0000","last_became_active":"2024-02-09T17:40:48.781472+0000","last_became_peered":"2024-02-09T17:40:48.781472+0000","last_unstale":"2024-02-09T17:42:14.115429+0000","last_undegraded":"2024-02-09T17:42:14.115429+0000","last_fullsized":"2024-02-09T17:42:14.115429+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:40:47.763277+0000","objects_scrubbed":0,"log_size":6759,"log_dups_size":0,"ondisk_log_size":6759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":442957824,"num_objects":6759,"num_object_clones":0,"num_object_copies":20277,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6759,"num_write_kb":432576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"63'139","reported_seq":490,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-02-09T17:42:11.885850+0000","last_change":"2024-02-09T17:40:57.059982+0000","last_active":"2024-02-09T17:42:11.885850+0000","last_peered":"2024-02-09T17:42:11.885850+0000","last_clean":"2024-02-09T17:42:11.885850+0000","last_became_active":"2024-02-09T17:40:13.501711+0000","last_became_peered":"2024-02-09T17:40:13.501711+0000","last_unstale":"2024-02-09T17:42:11.885850+0000","last_undegraded":"2024-02-09T17:42:11.885850+0000","last_fullsized":"2024-02-09T17:42:11.885850+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:40:57.059847+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:38:27.045816+0000","last_clean_scrub_stamp":"2024-02-09T17:40:57.059847+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:12.781614+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"499'6720","reported_seq":6797,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-02-09T17:42:13.836859+0000","last_change":"2024-02-09T17:40:48.780916+0000","last_active":"2024-02-09T17:42:13.836859+0000","last_peered":"2024-02-09T17:42:13.836859+0000","last_clean":"2024-02-09T17:42:13.836859+0000","last_became_active":"2024-02-09T17:40:48.780579+0000","last_became_peered":"2024-02-09T17:40:48.780579+0000","last_unstale":"2024-02-09T17:42:13.836859+0000","last_undegraded":"2024-02-09T17:42:13.836859+0000","last_fullsized":"2024-02-09T17:42:13.836859+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:40:47.763277+0000","objects_scrubbed":0,"log_size":6720,"log_dups_size":0,"ondisk_log_size":6720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":440401920,"num_objects":6720,"num_object_clones":0,"num_object_copies":20160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6720,"num_write_kb":430080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"63'139","reported_seq":806,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-02-09T17:42:11.885394+0000","last_change":"2024-02-09T17:40:46.690262+0000","last_active":"2024-02-09T17:42:11.885394+0000","last_peered":"2024-02-09T17:42:11.885394+0000","last_clean":"2024-02-09T17:42:11.885394+0000","last_became_active":"2024-02-09T17:34:01.342607+0000","last_became_peered":"2024-02-09T17:34:01.342607+0000","last_unstale":"2024-02-09T17:42:11.885394+0000","last_undegraded":"2024-02-09T17:42:11.885394+0000","last_fullsized":"2024-02-09T17:42:11.885394+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:40:46.690100+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:40:46.690100+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.2","version":"499'7030","reported_seq":7108,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-02-09T17:42:15.886449+0000","last_change":"2024-02-09T17:40:48.781838+0000","last_active":"2024-02-09T17:42:15.886449+0000","last_peered":"2024-02-09T17:42:15.886449+0000","last_clean":"2024-02-09T17:42:15.886449+0000","last_became_active":"2024-02-09T17:40:48.781532+0000","last_became_peered":"2024-02-09T17:40:48.781532+0000","last_unstale":"2024-02-09T17:42:15.886449+0000","last_undegraded":"2024-02-09T17:42:15.886449+0000","last_fullsized":"2024-02-09T17:42:15.886449+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:40:47.763277+0000","objects_scrubbed":0,"log_size":7030,"log_dups_size":0,"ondisk_log_size":7030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:17.274798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":460718080,"num_objects":7030,"num_object_clones":0,"num_object_copies":21090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7030,"num_write_kb":449920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"63'139","reported_seq":805,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-02-09T17:42:11.886057+0000","last_change":"2024-02-09T17:41:44.649428+0000","last_active":"2024-02-09T17:42:11.886057+0000","last_peered":"2024-02-09T17:42:11.886057+0000","last_clean":"2024-02-09T17:42:11.886057+0000","last_became_active":"2024-02-09T17:33:00.284702+0000","last_became_peered":"2024-02-09T17:33:00.284702+0000","last_unstale":"2024-02-09T17:42:11.886057+0000","last_undegraded":"2024-02-09T17:42:11.886057+0000","last_fullsized":"2024-02-09T17:42:11.886057+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:44.649313+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:40:01.416303+0000","last_clean_scrub_stamp":"2024-02-09T17:41:44.649313+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:06.594818+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.3","version":"499'6827","reported_seq":6967,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-02-09T17:42:15.866717+0000","last_change":"2024-02-09T17:41:55.628782+0000","last_active":"2024-02-09T17:42:15.866717+0000","last_peered":"2024-02-09T17:42:15.866717+0000","last_clean":"2024-02-09T17:42:15.866717+0000","last_became_active":"2024-02-09T17:40:48.783369+0000","last_became_peered":"2024-02-09T17:40:48.783369+0000","last_unstale":"2024-02-09T17:42:15.866717+0000","last_undegraded":"2024-02-09T17:42:15.866717+0000","last_fullsized":"2024-02-09T17:42:15.866717+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"493'5422","last_scrub_stamp":"2024-02-09T17:41:55.628642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:41:55.628642+0000","objects_scrubbed":5380,"log_size":6827,"log_dups_size":0,"ondisk_log_size":6827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:59.171152+0000","scrub_duration":1305,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":447414272,"num_objects":6827,"num_object_clones":0,"num_object_copies":20481,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6827,"num_write_kb":436928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"63'139","reported_seq":476,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-02-09T17:42:11.883750+0000","last_change":"2024-02-09T17:41:52.484254+0000","last_active":"2024-02-09T17:42:11.883750+0000","last_peered":"2024-02-09T17:42:11.883750+0000","last_clean":"2024-02-09T17:42:11.883750+0000","last_became_active":"2024-02-09T17:40:13.497710+0000","last_became_peered":"2024-02-09T17:40:13.497710+0000","last_unstale":"2024-02-09T17:42:11.883750+0000","last_undegraded":"2024-02-09T17:42:11.883750+0000","last_fullsized":"2024-02-09T17:42:11.883750+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:52.484109+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:41:52.484109+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:09.431307+0000","scrub_duration":125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"499'7286","reported_seq":7364,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-02-09T17:42:17.881714+0000","last_change":"2024-02-09T17:40:48.780689+0000","last_active":"2024-02-09T17:42:17.881714+0000","last_peered":"2024-02-09T17:42:17.881714+0000","last_clean":"2024-02-09T17:42:17.881714+0000","last_became_active":"2024-02-09T17:40:48.780290+0000","last_became_peered":"2024-02-09T17:40:48.780290+0000","last_unstale":"2024-02-09T17:42:17.881714+0000","last_undegraded":"2024-02-09T17:42:17.881714+0000","last_fullsized":"2024-02-09T17:42:17.881714+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:40:47.763277+0000","objects_scrubbed":0,"log_size":7286,"log_dups_size":0,"ondisk_log_size":7286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:19.496598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":477495296,"num_objects":7286,"num_object_clones":0,"num_object_copies":21858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7286,"num_write_kb":466304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"63'139","reported_seq":888,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-02-09T17:42:11.889478+0000","last_change":"2024-02-09T17:40:44.545620+0000","last_active":"2024-02-09T17:42:11.889478+0000","last_peered":"2024-02-09T17:42:11.889478+0000","last_clean":"2024-02-09T17:42:11.889478+0000","last_became_active":"2024-02-09T17:33:00.284924+0000","last_became_peered":"2024-02-09T17:33:00.284924+0000","last_unstale":"2024-02-09T17:42:11.889478+0000","last_undegraded":"2024-02-09T17:42:11.889478+0000","last_fullsized":"2024-02-09T17:42:11.889478+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:40:44.545494+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:40:44.545494+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:14.078645+0000","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.1","version":"499'6838","reported_seq":6916,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-02-09T17:42:15.675341+0000","last_change":"2024-02-09T17:40:49.661020+0000","last_active":"2024-02-09T17:42:15.675341+0000","last_peered":"2024-02-09T17:42:15.675341+0000","last_clean":"2024-02-09T17:42:15.675341+0000","last_became_active":"2024-02-09T17:40:49.660617+0000","last_became_peered":"2024-02-09T17:40:49.660617+0000","last_unstale":"2024-02-09T17:42:15.675341+0000","last_undegraded":"2024-02-09T17:42:15.675341+0000","last_fullsized":"2024-02-09T17:42:15.675341+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:40:47.763277+0000","objects_scrubbed":0,"log_size":6838,"log_dups_size":0,"ondisk_log_size":6838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:19.104685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448135168,"num_objects":6838,"num_object_clones":0,"num_object_copies":20514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6838,"num_write_kb":437632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"63'139","reported_seq":841,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-02-09T17:42:11.885451+0000","last_change":"2024-02-09T17:41:14.521510+0000","last_active":"2024-02-09T17:42:11.885451+0000","last_peered":"2024-02-09T17:42:11.885451+0000","last_clean":"2024-02-09T17:42:11.885451+0000","last_became_active":"2024-02-09T17:41:14.521098+0000","last_became_peered":"2024-02-09T17:41:14.521098+0000","last_unstale":"2024-02-09T17:42:11.885451+0000","last_undegraded":"2024-02-09T17:42:11.885451+0000","last_fullsized":"2024-02-09T17:42:11.885451+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:40:56.752599+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:40:56.752599+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:14.187085+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11],"acting":[10,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.e","version":"499'7051","reported_seq":7191,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-02-09T17:42:16.527795+0000","last_change":"2024-02-09T17:41:54.022262+0000","last_active":"2024-02-09T17:42:16.527795+0000","last_peered":"2024-02-09T17:42:16.527795+0000","last_clean":"2024-02-09T17:42:16.527795+0000","last_became_active":"2024-02-09T17:40:48.779366+0000","last_became_peered":"2024-02-09T17:40:48.779366+0000","last_unstale":"2024-02-09T17:42:16.527795+0000","last_undegraded":"2024-02-09T17:42:16.527795+0000","last_fullsized":"2024-02-09T17:42:16.527795+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"493'5450","last_scrub_stamp":"2024-02-09T17:41:54.019230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:41:54.019230+0000","objects_scrubbed":5409,"log_size":7051,"log_dups_size":0,"ondisk_log_size":7051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:19.376786+0000","scrub_duration":1190,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":462094336,"num_objects":7051,"num_object_clones":0,"num_object_copies":21153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7051,"num_write_kb":451264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"63'139","reported_seq":798,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-02-09T17:42:11.886604+0000","last_change":"2024-02-09T17:41:41.681560+0000","last_active":"2024-02-09T17:42:11.886604+0000","last_peered":"2024-02-09T17:42:11.886604+0000","last_clean":"2024-02-09T17:42:11.886604+0000","last_became_active":"2024-02-09T17:32:35.418200+0000","last_became_peered":"2024-02-09T17:32:35.418200+0000","last_unstale":"2024-02-09T17:42:11.886604+0000","last_undegraded":"2024-02-09T17:42:11.886604+0000","last_fullsized":"2024-02-09T17:42:11.886604+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:41.681434+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:41:41.681434+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:48.245978+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.f","version":"499'6863","reported_seq":9315,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-02-09T17:42:14.106398+0000","last_change":"2024-02-09T17:41:20.878405+0000","last_active":"2024-02-09T17:42:14.106398+0000","last_peered":"2024-02-09T17:42:14.106398+0000","last_clean":"2024-02-09T17:42:14.106398+0000","last_became_active":"2024-02-09T17:41:20.876180+0000","last_became_peered":"2024-02-09T17:41:20.876180+0000","last_unstale":"2024-02-09T17:42:14.106398+0000","last_undegraded":"2024-02-09T17:42:14.106398+0000","last_fullsized":"2024-02-09T17:42:14.106398+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:40:47.763277+0000","objects_scrubbed":0,"log_size":6863,"log_dups_size":0,"ondisk_log_size":6863,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:14.436622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":449773568,"num_objects":6863,"num_object_clones":0,"num_object_copies":20589,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6863,"num_write_kb":439232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1182,"num_bytes_recovered":77463552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,11],"acting":[5,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"63'139","reported_seq":640,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-02-09T17:42:11.889372+0000","last_change":"2024-02-09T17:41:46.320889+0000","last_active":"2024-02-09T17:42:11.889372+0000","last_peered":"2024-02-09T17:42:11.889372+0000","last_clean":"2024-02-09T17:42:11.889372+0000","last_became_active":"2024-02-09T17:41:14.522410+0000","last_became_peered":"2024-02-09T17:41:14.522410+0000","last_unstale":"2024-02-09T17:42:11.889372+0000","last_undegraded":"2024-02-09T17:42:11.889372+0000","last_fullsized":"2024-02-09T17:42:11.889372+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:46.320807+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:41:46.320807+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:05.512272+0000","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"499'6936","reported_seq":7082,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-02-09T17:42:15.639120+0000","last_change":"2024-02-09T17:42:00.784129+0000","last_active":"2024-02-09T17:42:15.639120+0000","last_peered":"2024-02-09T17:42:15.639120+0000","last_clean":"2024-02-09T17:42:15.639120+0000","last_became_active":"2024-02-09T17:40:48.781384+0000","last_became_peered":"2024-02-09T17:40:48.781384+0000","last_unstale":"2024-02-09T17:42:15.639120+0000","last_undegraded":"2024-02-09T17:42:15.639120+0000","last_fullsized":"2024-02-09T17:42:15.639120+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"495'5883","last_scrub_stamp":"2024-02-09T17:42:00.784027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:42:00.784027+0000","objects_scrubbed":5858,"log_size":6936,"log_dups_size":0,"ondisk_log_size":6936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:15.646893+0000","scrub_duration":1061,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":454557696,"num_objects":6936,"num_object_clones":0,"num_object_copies":20808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6936,"num_write_kb":443904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"63'139","reported_seq":855,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-02-09T17:42:11.885489+0000","last_change":"2024-02-09T17:41:55.371903+0000","last_active":"2024-02-09T17:42:11.885489+0000","last_peered":"2024-02-09T17:42:11.885489+0000","last_clean":"2024-02-09T17:42:11.885489+0000","last_became_active":"2024-02-09T17:39:37.460615+0000","last_became_peered":"2024-02-09T17:39:37.460615+0000","last_unstale":"2024-02-09T17:42:11.885489+0000","last_undegraded":"2024-02-09T17:42:11.885489+0000","last_fullsized":"2024-02-09T17:42:11.885489+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:55.371768+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:41:55.371768+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:12.227887+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"499'7046","reported_seq":9688,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-02-09T17:42:15.311799+0000","last_change":"2024-02-09T17:41:54.334501+0000","last_active":"2024-02-09T17:42:15.311799+0000","last_peered":"2024-02-09T17:42:15.311799+0000","last_clean":"2024-02-09T17:42:15.311799+0000","last_became_active":"2024-02-09T17:41:21.499187+0000","last_became_peered":"2024-02-09T17:41:21.499187+0000","last_unstale":"2024-02-09T17:42:15.311799+0000","last_undegraded":"2024-02-09T17:42:15.311799+0000","last_fullsized":"2024-02-09T17:42:15.311799+0000","mapping_epoch":481,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"493'5534","last_scrub_stamp":"2024-02-09T17:41:54.334404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:41:54.334404+0000","objects_scrubbed":5491,"log_size":7046,"log_dups_size":0,"ondisk_log_size":7046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:03.963347+0000","scrub_duration":1279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":461766656,"num_objects":7046,"num_object_clones":0,"num_object_copies":21138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7046,"num_write_kb":450944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1248,"num_bytes_recovered":81788928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"63'139","reported_seq":1293,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-02-09T17:42:11.892733+0000","last_change":"2024-02-09T17:40:57.958820+0000","last_active":"2024-02-09T17:42:11.892733+0000","last_peered":"2024-02-09T17:42:11.892733+0000","last_clean":"2024-02-09T17:42:11.892733+0000","last_became_active":"2024-02-09T17:39:17.456739+0000","last_became_peered":"2024-02-09T17:39:17.456739+0000","last_unstale":"2024-02-09T17:42:11.892733+0000","last_undegraded":"2024-02-09T17:42:11.892733+0000","last_fullsized":"2024-02-09T17:42:11.892733+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:40:57.958641+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:30.193194+0000","last_clean_scrub_stamp":"2024-02-09T17:40:57.958641+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"499'6901","reported_seq":6979,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-02-09T17:42:16.893834+0000","last_change":"2024-02-09T17:40:48.781919+0000","last_active":"2024-02-09T17:42:16.893834+0000","last_peered":"2024-02-09T17:42:16.893834+0000","last_clean":"2024-02-09T17:42:16.893834+0000","last_became_active":"2024-02-09T17:40:48.781591+0000","last_became_peered":"2024-02-09T17:40:48.781591+0000","last_unstale":"2024-02-09T17:42:16.893834+0000","last_undegraded":"2024-02-09T17:42:16.893834+0000","last_fullsized":"2024-02-09T17:42:16.893834+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:40:47.763277+0000","objects_scrubbed":0,"log_size":6901,"log_dups_size":0,"ondisk_log_size":6901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:19.764744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452263936,"num_objects":6901,"num_object_clones":0,"num_object_copies":20703,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6901,"num_write_kb":441664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,4],"acting":[8,10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.e","version":"63'139","reported_seq":767,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-02-09T17:42:11.898502+0000","last_change":"2024-02-09T17:41:06.857849+0000","last_active":"2024-02-09T17:42:11.898502+0000","last_peered":"2024-02-09T17:42:11.898502+0000","last_clean":"2024-02-09T17:42:11.898502+0000","last_became_active":"2024-02-09T17:33:05.309824+0000","last_became_peered":"2024-02-09T17:33:05.309824+0000","last_unstale":"2024-02-09T17:42:11.898502+0000","last_undegraded":"2024-02-09T17:42:11.898502+0000","last_fullsized":"2024-02-09T17:42:11.898502+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:06.857716+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:41:06.857716+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:29.895394+0000","scrub_duration":190,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"499'6960","reported_seq":7038,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-02-09T17:42:16.527973+0000","last_change":"2024-02-09T17:40:48.783627+0000","last_active":"2024-02-09T17:42:16.527973+0000","last_peered":"2024-02-09T17:42:16.527973+0000","last_clean":"2024-02-09T17:42:16.527973+0000","last_became_active":"2024-02-09T17:40:48.783235+0000","last_became_peered":"2024-02-09T17:40:48.783235+0000","last_unstale":"2024-02-09T17:42:16.527973+0000","last_undegraded":"2024-02-09T17:42:16.527973+0000","last_fullsized":"2024-02-09T17:42:16.527973+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:40:47.763277+0000","objects_scrubbed":0,"log_size":6960,"log_dups_size":0,"ondisk_log_size":6960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":456130560,"num_objects":6960,"num_object_clones":0,"num_object_copies":20880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6960,"num_write_kb":445440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.f","version":"63'139","reported_seq":825,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-02-09T17:42:11.883798+0000","last_change":"2024-02-09T17:41:02.126077+0000","last_active":"2024-02-09T17:42:11.883798+0000","last_peered":"2024-02-09T17:42:11.883798+0000","last_clean":"2024-02-09T17:42:11.883798+0000","last_became_active":"2024-02-09T17:33:05.316100+0000","last_became_peered":"2024-02-09T17:33:05.316100+0000","last_unstale":"2024-02-09T17:42:11.883798+0000","last_undegraded":"2024-02-09T17:42:11.883798+0000","last_fullsized":"2024-02-09T17:42:11.883798+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:02.125974+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:36:22.364130+0000","last_clean_scrub_stamp":"2024-02-09T17:41:02.125974+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"499'6951","reported_seq":9429,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-02-09T17:42:15.317977+0000","last_change":"2024-02-09T17:41:20.879764+0000","last_active":"2024-02-09T17:42:15.317977+0000","last_peered":"2024-02-09T17:42:15.317977+0000","last_clean":"2024-02-09T17:42:15.317977+0000","last_became_active":"2024-02-09T17:41:20.876434+0000","last_became_peered":"2024-02-09T17:41:20.876434+0000","last_unstale":"2024-02-09T17:42:15.317977+0000","last_undegraded":"2024-02-09T17:42:15.317977+0000","last_fullsized":"2024-02-09T17:42:15.317977+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:40:47.763277+0000","objects_scrubbed":0,"log_size":6951,"log_dups_size":0,"ondisk_log_size":6951,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":455540736,"num_objects":6951,"num_object_clones":0,"num_object_copies":20853,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6951,"num_write_kb":444864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1195,"num_bytes_recovered":78315520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"63'139","reported_seq":954,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-02-09T17:42:11.881434+0000","last_change":"2024-02-09T17:41:43.719325+0000","last_active":"2024-02-09T17:42:11.881434+0000","last_peered":"2024-02-09T17:42:11.881434+0000","last_clean":"2024-02-09T17:42:11.881434+0000","last_became_active":"2024-02-09T17:39:37.471954+0000","last_became_peered":"2024-02-09T17:39:37.471954+0000","last_unstale":"2024-02-09T17:42:11.881434+0000","last_undegraded":"2024-02-09T17:42:11.881434+0000","last_fullsized":"2024-02-09T17:42:11.881434+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:43.719190+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:23.912504+0000","last_clean_scrub_stamp":"2024-02-09T17:41:43.719190+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:03.687255+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"499'7051","reported_seq":7190,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-02-09T17:42:17.194862+0000","last_change":"2024-02-09T17:41:52.315160+0000","last_active":"2024-02-09T17:42:17.194862+0000","last_peered":"2024-02-09T17:42:17.194862+0000","last_clean":"2024-02-09T17:42:17.194862+0000","last_became_active":"2024-02-09T17:40:49.436285+0000","last_became_peered":"2024-02-09T17:40:49.436285+0000","last_unstale":"2024-02-09T17:42:17.194862+0000","last_undegraded":"2024-02-09T17:42:17.194862+0000","last_fullsized":"2024-02-09T17:42:17.194862+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"492'5271","last_scrub_stamp":"2024-02-09T17:41:52.314972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:41:52.314972+0000","objects_scrubbed":5238,"log_size":7051,"log_dups_size":0,"ondisk_log_size":7051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:03.342519+0000","scrub_duration":1131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":462094336,"num_objects":7051,"num_object_clones":0,"num_object_copies":21153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7051,"num_write_kb":451264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"63'139","reported_seq":358,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-02-09T17:42:11.890675+0000","last_change":"2024-02-09T17:40:44.903454+0000","last_active":"2024-02-09T17:42:11.890675+0000","last_peered":"2024-02-09T17:42:11.890675+0000","last_clean":"2024-02-09T17:42:11.890675+0000","last_became_active":"2024-02-09T17:40:22.485257+0000","last_became_peered":"2024-02-09T17:40:22.485257+0000","last_unstale":"2024-02-09T17:42:11.890675+0000","last_undegraded":"2024-02-09T17:42:11.890675+0000","last_fullsized":"2024-02-09T17:42:11.890675+0000","mapping_epoch":454,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":455,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:40:44.903308+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:40:44.903308+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:13.814742+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"63'139","reported_seq":774,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-02-09T17:42:11.885391+0000","last_change":"2024-02-09T17:41:09.323335+0000","last_active":"2024-02-09T17:42:11.885391+0000","last_peered":"2024-02-09T17:42:11.885391+0000","last_clean":"2024-02-09T17:42:11.885391+0000","last_became_active":"2024-02-09T17:33:05.310104+0000","last_became_peered":"2024-02-09T17:33:05.310104+0000","last_unstale":"2024-02-09T17:42:11.885391+0000","last_undegraded":"2024-02-09T17:42:11.885391+0000","last_fullsized":"2024-02-09T17:42:11.885391+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:09.323236+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:41:09.323236+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:13.566799+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"63'139","reported_seq":781,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-02-09T17:42:11.890921+0000","last_change":"2024-02-09T17:41:06.851766+0000","last_active":"2024-02-09T17:42:11.890921+0000","last_peered":"2024-02-09T17:42:11.890921+0000","last_clean":"2024-02-09T17:42:11.890921+0000","last_became_active":"2024-02-09T17:34:01.353171+0000","last_became_peered":"2024-02-09T17:34:01.353171+0000","last_unstale":"2024-02-09T17:42:11.890921+0000","last_undegraded":"2024-02-09T17:42:11.890921+0000","last_fullsized":"2024-02-09T17:42:11.890921+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:06.851643+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:41:06.851643+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:34.858536+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.12","version":"63'139","reported_seq":1293,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-02-09T17:42:11.885442+0000","last_change":"2024-02-09T17:40:44.208044+0000","last_active":"2024-02-09T17:42:11.885442+0000","last_peered":"2024-02-09T17:42:11.885442+0000","last_clean":"2024-02-09T17:42:11.885442+0000","last_became_active":"2024-02-09T17:40:13.499071+0000","last_became_peered":"2024-02-09T17:40:13.499071+0000","last_unstale":"2024-02-09T17:42:11.885442+0000","last_undegraded":"2024-02-09T17:42:11.885442+0000","last_fullsized":"2024-02-09T17:42:11.885442+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:40:44.207923+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:40:44.207923+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:13.625252+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"63'139","reported_seq":493,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-02-09T17:42:11.885915+0000","last_change":"2024-02-09T17:41:14.521999+0000","last_active":"2024-02-09T17:42:11.885915+0000","last_peered":"2024-02-09T17:42:11.885915+0000","last_clean":"2024-02-09T17:42:11.885915+0000","last_became_active":"2024-02-09T17:41:14.521672+0000","last_became_peered":"2024-02-09T17:41:14.521672+0000","last_unstale":"2024-02-09T17:42:11.885915+0000","last_undegraded":"2024-02-09T17:42:11.885915+0000","last_fullsized":"2024-02-09T17:42:11.885915+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:03.023657+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:41:03.023657+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:28.356260+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":7274430464,"num_objects":110999,"num_object_clones":0,"num_object_copies":332997,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":110999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":110999,"num_write_kb":7103936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3625,"num_bytes_recovered":237568000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21843345408,"data_stored":21843345408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110999,"ondisk_log_size":110999,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":931424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2919,"ondisk_log_size":2919,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056879,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2305164,"kb_used_data":1778372,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":91439988,"statfs":{"total":95995035648,"available":93634547712,"internally_reserved":0,"allocated":1821052928,"data_stored":1820530689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":539425220},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:41:25 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":1.6399999999999999,"15min":1.359},"min":{"1min":0.33100000000000002,"5min":0.246,"15min":0.216},"max":{"1min":1.4410000000000001,"5min":9.5709999999999997,"15min":14.913},"last":2.431},{"interface":"front","average":{"1min":0.56399999999999995,"5min":1.369,"15min":1.3720000000000001},"min":{"1min":0.33800000000000002,"5min":0.27000000000000002,"15min":0.23000000000000001},"max":{"1min":1.849,"5min":9.8219999999999992,"15min":14.869},"last":0.28999999999999998}]},{"osd":1,"last update":"Fri Feb 9 17:41:25 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":1.6739999999999999,"15min":1.6850000000000001},"min":{"1min":0.45300000000000001,"5min":0.24299999999999999,"15min":0.22700000000000001},"max":{"1min":1.7070000000000001,"5min":9.7710000000000008,"15min":32.957999999999998},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":1.8049999999999999,"15min":1.8320000000000001},"min":{"1min":0.29099999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.97299999999999998,"5min":9.718,"15min":32.880000000000003},"last":0.36399999999999999}]},{"osd":2,"last update":"Fri Feb 9 17:31:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.415,"15min":1.6859999999999999},"min":{"1min":0.378,"5min":0.247,"15min":0.247},"max":{"1min":4.4189999999999996,"5min":9.4719999999999995,"15min":9.4719999999999995},"last":0.35399999999999998},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.5149999999999999,"15min":1.4399999999999999},"min":{"1min":0.378,"5min":0.252,"15min":0.252},"max":{"1min":4.8579999999999997,"5min":7.96,"15min":7.96},"last":0.60399999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:41:25 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":1.355,"15min":1.659},"min":{"1min":0.28699999999999998,"5min":0.27100000000000002,"15min":0.247},"max":{"1min":1.7629999999999999,"5min":9.5299999999999994,"15min":21.315999999999999},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.66600000000000004,"5min":1.869,"15min":1.752},"min":{"1min":0.40300000000000002,"5min":0.29799999999999999,"15min":0.26400000000000001},"max":{"1min":1.7929999999999999,"5min":12.706,"15min":18.087},"last":0.48799999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:41:25 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":1.1919999999999999,"15min":1.4750000000000001},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.25600000000000001},"max":{"1min":1.522,"5min":21.030999999999999,"15min":21.030999999999999},"last":0.246},{"interface":"front","average":{"1min":0.76300000000000001,"5min":1.2669999999999999,"15min":1.3180000000000001},"min":{"1min":0.40200000000000002,"5min":0.309,"15min":0.27700000000000002},"max":{"1min":1.5109999999999999,"5min":20.876999999999999,"15min":20.876999999999999},"last":0.32600000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:41:25 2024","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":1.0640000000000001,"15min":1.345},"min":{"1min":0.441,"5min":0.316,"15min":0.28899999999999998},"max":{"1min":1.663,"5min":6.4630000000000001,"15min":13.946999999999999},"last":2.8109999999999999},{"interface":"front","average":{"1min":0.65800000000000003,"5min":1.169,"15min":1.4359999999999999},"min":{"1min":0.378,"5min":0.311,"15min":0.30199999999999999},"max":{"1min":1.1519999999999999,"5min":7.1349999999999998,"15min":13.414},"last":2.2679999999999998}]},{"osd":6,"last update":"Fri Feb 9 17:41:25 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":1.3200000000000001,"15min":1.421},"min":{"1min":0.48899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.74,"5min":9.4779999999999998,"15min":12.798999999999999},"last":1.6579999999999999},{"interface":"front","average":{"1min":0.74099999999999999,"5min":1.212,"15min":1.411},"min":{"1min":0.38500000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.474,"5min":9.2420000000000009,"15min":10.782999999999999},"last":2.23}]},{"osd":7,"last update":"Fri Feb 9 17:41:25 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":1.375,"15min":1.615},"min":{"1min":0.503,"5min":0.38,"15min":0.312},"max":{"1min":1.409,"5min":20.489999999999998,"15min":20.489999999999998},"last":5.54},{"interface":"front","average":{"1min":0.877,"5min":1.302,"15min":1.585},"min":{"1min":0.52800000000000002,"5min":0.35299999999999998,"15min":0.34200000000000003},"max":{"1min":1.841,"5min":20.712,"15min":20.712},"last":1.536}]},{"osd":8,"last update":"Fri Feb 9 17:41:54 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":1.0780000000000001,"15min":1.657},"min":{"1min":0.316,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":4.5739999999999998,"5min":9.2680000000000007,"15min":13.199999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.149,"15min":1.448},"min":{"1min":0.40999999999999998,"5min":0.34300000000000003,"15min":0.318},"max":{"1min":6.1219999999999999,"5min":9.2940000000000005,"15min":9.2940000000000005},"last":0.43099999999999999}]},{"osd":9,"last update":"Fri Feb 9 17:41:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":1.79,"15min":1.4259999999999999},"min":{"1min":0.36799999999999999,"5min":0.36499999999999999,"15min":0.29399999999999998},"max":{"1min":1.5629999999999999,"5min":62.518000000000001,"15min":62.518000000000001},"last":1.095},{"interface":"front","average":{"1min":0.84299999999999997,"5min":1.8480000000000001,"15min":1.369},"min":{"1min":0.46300000000000002,"5min":0.379,"15min":0.34100000000000003},"max":{"1min":3.8159999999999998,"5min":63.241999999999997,"15min":63.241999999999997},"last":1.1930000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:41:25 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":1.1990000000000001,"15min":1.196},"min":{"1min":0.47199999999999998,"5min":0.36799999999999999,"15min":0.33900000000000002},"max":{"1min":1.214,"5min":9.3109999999999999,"15min":9.6199999999999992},"last":0.40899999999999997},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.306,"15min":1.2749999999999999},"min":{"1min":0.44700000000000001,"5min":0.39100000000000001,"15min":0.33700000000000002},"max":{"1min":4.0279999999999996,"5min":9.593,"15min":12.856},"last":0.96599999999999997}]}]},{"osd":10,"up_from":84,"seq":360777253109,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3035084,"kb_used_data":2715140,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":90710068,"statfs":{"total":95995035648,"available":92887109632,"internally_reserved":0,"allocated":2780303360,"data_stored":2779781121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":327612864},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:41:14 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":1.54,"15min":1.7050000000000001},"min":{"1min":0.25700000000000001,"5min":0.222,"15min":0.222},"max":{"1min":1.6559999999999999,"5min":9.343,"15min":9.343},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":1.365,"15min":1.177},"min":{"1min":0.27600000000000002,"5min":0.26500000000000001,"15min":0.253},"max":{"1min":1.484,"5min":6.3399999999999999,"15min":6.3399999999999999},"last":0.28899999999999998}]},{"osd":1,"last update":"Fri Feb 9 17:41:50 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":2.3460000000000001,"5min":2.3460000000000001,"15min":2.3460000000000001},"last":0.307},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":3.5049999999999999,"5min":3.5049999999999999,"15min":3.5049999999999999},"last":0.434}]},{"osd":2,"last update":"Fri Feb 9 17:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":1.774,"15min":1.425},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.222},"max":{"1min":2.4140000000000001,"5min":11.384,"15min":13.351000000000001},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.83699999999999997,"5min":1.425,"15min":1.3859999999999999},"min":{"1min":0.32500000000000001,"5min":0.28799999999999998,"15min":0.26100000000000001},"max":{"1min":3.254,"5min":11.465,"15min":11.465},"last":0.45200000000000001}]},{"osd":3,"last update":"Fri Feb 9 17:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":1.7509999999999999,"15min":1.9490000000000001},"min":{"1min":0.34699999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.5029999999999999,"5min":9.1649999999999991,"15min":9.1649999999999991},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.77900000000000003,"5min":1.8400000000000001,"15min":2.0169999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":2.492,"5min":8.9670000000000005,"15min":8.9670000000000005},"last":0.41499999999999998}]},{"osd":4,"last update":"Fri Feb 9 17:42:05 2024","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":0.97199999999999998,"15min":1.0509999999999999},"min":{"1min":0.36899999999999999,"5min":0.29199999999999998,"15min":0.26100000000000001},"max":{"1min":6.4930000000000003,"5min":6.4930000000000003,"15min":7.1360000000000001},"last":0.47899999999999998},{"interface":"front","average":{"1min":1.1080000000000001,"5min":0.92300000000000004,"15min":1.048},"min":{"1min":0.39300000000000002,"5min":0.31,"15min":0.29999999999999999},"max":{"1min":6.5339999999999998,"5min":6.5339999999999998,"15min":6.8369999999999997},"last":0.502}]},{"osd":5,"last update":"Fri Feb 9 17:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":1.085,"15min":1.2370000000000001},"min":{"1min":0.35999999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":2.9460000000000002,"5min":7.5179999999999998,"15min":9.2349999999999994},"last":1.3859999999999999},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.0740000000000001,"15min":1.2490000000000001},"min":{"1min":0.47399999999999998,"5min":0.374,"15min":0.32200000000000001},"max":{"1min":6.569,"5min":6.8940000000000001,"15min":8.6159999999999997},"last":1.8420000000000001}]},{"osd":6,"last update":"Fri Feb 9 17:42:00 2024","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.3049999999999999,"15min":1.274},"min":{"1min":0.42399999999999999,"5min":0.308,"15min":0.29099999999999998},"max":{"1min":4.21,"5min":8.6869999999999994,"15min":8.6869999999999994},"last":1.198},{"interface":"front","average":{"1min":1.411,"5min":1.1279999999999999,"15min":1.3799999999999999},"min":{"1min":0.497,"5min":0.38600000000000001,"15min":0.35399999999999998},"max":{"1min":6.6040000000000001,"5min":6.6040000000000001,"15min":9.4350000000000005},"last":1.673}]},{"osd":7,"last update":"Fri Feb 9 17:39:52 2024","interfaces":[{"interface":"back","average":{"1min":1.6000000000000001,"5min":1.367,"15min":1.415},"min":{"1min":0.42099999999999999,"5min":0.34200000000000003,"15min":0.312},"max":{"1min":4.0430000000000001,"5min":6.7590000000000003,"15min":17.754000000000001},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.498,"5min":1.4359999999999999,"15min":1.5189999999999999},"min":{"1min":0.47399999999999998,"5min":0.38300000000000001,"15min":0.316},"max":{"1min":5.8170000000000002,"5min":11.308999999999999,"15min":11.308999999999999},"last":0.78800000000000003}]},{"osd":8,"last update":"Fri Feb 9 17:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":1.097,"15min":1.143},"min":{"1min":0.37,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":2.444,"5min":7.4359999999999999,"15min":7.4359999999999999},"last":2.4529999999999998},{"interface":"front","average":{"1min":0.998,"5min":1.075,"15min":0.97399999999999998},"min":{"1min":0.41899999999999998,"5min":0.37,"15min":0.32900000000000001},"max":{"1min":4.343,"5min":7.7060000000000004,"15min":7.7060000000000004},"last":0.39700000000000002}]},{"osd":9,"last update":"Fri Feb 9 17:42:00 2024","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":1.2729999999999999,"15min":1.0960000000000001},"min":{"1min":0.36799999999999999,"5min":0.36399999999999999,"15min":0.314},"max":{"1min":4.766,"5min":7.4850000000000003,"15min":8.7240000000000002},"last":0.59499999999999997},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.2370000000000001,"15min":1.1950000000000001},"min":{"1min":0.46000000000000002,"5min":0.441,"15min":0.29799999999999999},"max":{"1min":3.8090000000000002,"5min":7.891,"15min":10.1},"last":0.621}]},{"osd":11,"last update":"Fri Feb 9 17:41:21 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.89600000000000002,"15min":1.113},"min":{"1min":0.45800000000000002,"5min":0.40600000000000003,"15min":0.36399999999999999},"max":{"1min":0.98399999999999999,"5min":4.2560000000000002,"15min":8.6430000000000007},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.95199999999999996,"15min":1.1299999999999999},"min":{"1min":0.60199999999999998,"5min":0.45700000000000002,"15min":0.33400000000000002},"max":{"1min":1.0269999999999999,"5min":4.2800000000000002,"15min":9.298},"last":0.57299999999999995}]}]},{"osd":9,"up_from":77,"seq":330712482042,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1765836,"kb_used_data":1333444,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":91979316,"statfs":{"total":95995035648,"available":94186819584,"internally_reserved":0,"allocated":1365446656,"data_stored":1364924417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":442759616},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:41:50 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.9790000000000001,"5min":1.9790000000000001,"15min":1.9790000000000001},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":4.3339999999999996,"5min":4.3339999999999996,"15min":4.3339999999999996},"last":3.2509999999999999}]},{"osd":1,"last update":"Fri Feb 9 17:41:37 2024","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":2.8159999999999998,"15min":1.8500000000000001},"min":{"1min":0.32700000000000001,"5min":0.23499999999999999,"15min":0.23100000000000001},"max":{"1min":9.0449999999999999,"5min":89.515000000000001,"15min":89.515000000000001},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.72599999999999998,"5min":2.895,"15min":1.8100000000000001},"min":{"1min":0.38500000000000001,"5min":0.249,"15min":0.222},"max":{"1min":2.0609999999999999,"5min":89.388000000000005,"15min":89.388000000000005},"last":0.54100000000000004}]},{"osd":2,"last update":"Fri Feb 9 17:40:39 2024","interfaces":[{"interface":"back","average":{"1min":1.742,"5min":1.766,"15min":1.514},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":5.6630000000000003,"5min":14.121,"15min":14.121},"last":0.63400000000000001},{"interface":"front","average":{"1min":1.6910000000000001,"5min":1.52,"15min":1.4870000000000001},"min":{"1min":0.34100000000000003,"5min":0.26100000000000001,"15min":0.25600000000000001},"max":{"1min":5.7030000000000003,"5min":8.8040000000000003,"15min":13.69},"last":0.73899999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:41:37 2024","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":2.1880000000000002,"15min":1.655},"min":{"1min":0.371,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":8.9939999999999998,"5min":15.414999999999999,"15min":40.18},"last":0.92700000000000005},{"interface":"front","average":{"1min":0.69399999999999995,"5min":2.1760000000000002,"15min":1.8160000000000001},"min":{"1min":0.32000000000000001,"5min":0.317,"15min":0.27900000000000003},"max":{"1min":2.2679999999999998,"5min":15.372,"15min":40.158000000000001},"last":0.60199999999999998}]},{"osd":4,"last update":"Fri Feb 9 17:41:37 2024","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.331,"15min":1.4359999999999999},"min":{"1min":0.40899999999999997,"5min":0.29599999999999999,"15min":0.28399999999999997},"max":{"1min":9.0920000000000005,"5min":27.381,"15min":27.381},"last":3.802},{"interface":"front","average":{"1min":0.76100000000000001,"5min":1.419,"15min":1.3440000000000001},"min":{"1min":0.32800000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.571,"5min":27.355,"15min":27.355},"last":3.9990000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":1.2090000000000001,"15min":1.3380000000000001},"min":{"1min":0.41499999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.4790000000000001,"5min":8.6110000000000007,"15min":10.689},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.879,"5min":1.1639999999999999,"15min":1.157},"min":{"1min":0.39300000000000002,"5min":0.32200000000000001,"15min":0.311},"max":{"1min":2.032,"5min":13.978999999999999,"15min":13.978999999999999},"last":4.7050000000000001}]},{"osd":6,"last update":"Fri Feb 9 17:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":1.3080000000000001,"15min":1.379},"min":{"1min":0.41499999999999998,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":2.0659999999999998,"5min":14.069000000000001,"15min":14.069000000000001},"last":4.6689999999999996},{"interface":"front","average":{"1min":0.94699999999999995,"5min":1.3380000000000001,"15min":1.419},"min":{"1min":0.45500000000000002,"5min":0.37,"15min":0.36199999999999999},"max":{"1min":2.133,"5min":14.097,"15min":14.097},"last":3.2909999999999999}]},{"osd":7,"last update":"Fri Feb 9 17:41:37 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.3580000000000001,"15min":1.3899999999999999},"min":{"1min":0.45100000000000001,"5min":0.34499999999999997,"15min":0.34300000000000003},"max":{"1min":9.125,"5min":9.125,"15min":9.125},"last":0.97099999999999997},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.403,"15min":1.454},"min":{"1min":0.49299999999999999,"5min":0.39600000000000002,"15min":0.32200000000000001},"max":{"1min":9.157,"5min":9.157,"15min":9.157},"last":0.98999999999999999}]},{"osd":8,"last update":"Fri Feb 9 17:41:57 2024","interfaces":[{"interface":"back","average":{"1min":1.2410000000000001,"5min":1.3680000000000001,"15min":1.2769999999999999},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.34999999999999998},"max":{"1min":4.1299999999999999,"5min":14.045999999999999,"15min":16.209},"last":0.50900000000000001},{"interface":"front","average":{"1min":1.452,"5min":1.444,"15min":1.2969999999999999},"min":{"1min":0.432,"5min":0.41799999999999998,"15min":0.34799999999999998},"max":{"1min":4.6639999999999997,"5min":5.548,"15min":16.263000000000002},"last":3.1739999999999999}]},{"osd":10,"last update":"Fri Feb 9 17:42:06 2024","interfaces":[{"interface":"back","average":{"1min":1.724,"5min":1.3779999999999999,"15min":1.181},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":9.1869999999999994,"5min":9.1869999999999994,"15min":9.5090000000000003},"last":0.94799999999999995},{"interface":"front","average":{"1min":1.129,"5min":1.3979999999999999,"15min":1.147},"min":{"1min":0.40999999999999998,"5min":0.39500000000000002,"15min":0.38200000000000001},"max":{"1min":4.1779999999999999,"5min":9.7569999999999997,"15min":9.7569999999999997},"last":0.91100000000000003}]},{"osd":11,"last update":"Fri Feb 9 17:42:06 2024","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.5660000000000001,"15min":1.4379999999999999},"min":{"1min":0.434,"5min":0.39000000000000001,"15min":0.34699999999999998},"max":{"1min":6.8070000000000004,"5min":14.02,"15min":14.02},"last":3.2160000000000002},{"interface":"front","average":{"1min":1.0029999999999999,"5min":1.28,"15min":1.2330000000000001},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.374},"max":{"1min":3.4460000000000002,"5min":7.29,"15min":10.32},"last":0.57099999999999995}]}]},{"osd":8,"up_from":171,"seq":734439407785,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1566740,"kb_used_data":1336716,"kb_used_omap":212,"kb_used_meta":229803,"kb_avail":92178412,"statfs":{"total":95995035648,"available":94390693888,"internally_reserved":0,"allocated":1368797184,"data_stored":1368271393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217579,"internal_metadata":235318805},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":1.0880000000000001,"15min":1.149},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":4.7370000000000001,"5min":8.0920000000000005,"15min":8.0920000000000005},"last":0.28100000000000003},{"interface":"front","average":{"1min":1.024,"5min":1.506,"15min":1.321},"min":{"1min":0.28699999999999998,"5min":0.26200000000000001,"15min":0.23999999999999999},"max":{"1min":4.1360000000000001,"5min":10.256,"15min":10.256},"last":0.307}]},{"osd":1,"last update":"Fri Feb 9 17:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":1.5469999999999999,"15min":1.3160000000000001},"min":{"1min":0.25,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.8959999999999999,"5min":7.867,"15min":7.8719999999999999},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.60199999999999998,"5min":2.0150000000000001,"15min":1.681},"min":{"1min":0.27900000000000003,"5min":0.255,"15min":0.23300000000000001},"max":{"1min":3.2370000000000001,"5min":25.027000000000001,"15min":25.027000000000001},"last":0.46300000000000002}]},{"osd":2,"last update":"Fri Feb 9 17:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":1.4630000000000001,"15min":1.427},"min":{"1min":0.308,"5min":0.27500000000000002,"15min":0.255},"max":{"1min":0.73099999999999998,"5min":8.5649999999999995,"15min":10.423},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.59399999999999997,"5min":1.516,"15min":1.4610000000000001},"min":{"1min":0.39200000000000002,"5min":0.35899999999999999,"15min":0.313},"max":{"1min":0.871,"5min":8.5980000000000008,"15min":10.504},"last":0.73599999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":1.7270000000000001,"15min":1.7609999999999999},"min":{"1min":0.33000000000000002,"5min":0.29499999999999998,"15min":0.28699999999999998},"max":{"1min":3.2799999999999998,"5min":15.228999999999999,"15min":24.030999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.71599999999999997,"5min":2.1259999999999999,"15min":2.0169999999999999},"min":{"1min":0.39300000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":3.343,"5min":14.988,"15min":24.318999999999999},"last":0.52000000000000002}]},{"osd":4,"last update":"Fri Feb 9 17:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":1.2569999999999999,"15min":1.4239999999999999},"min":{"1min":0.45900000000000002,"5min":0.29099999999999998,"15min":0.27500000000000002},"max":{"1min":3.262,"5min":12.936,"15min":53.060000000000002},"last":0.32700000000000001},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.2130000000000001,"15min":1.569},"min":{"1min":0.46700000000000003,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":3.2959999999999998,"5min":7.46,"15min":53.311999999999998},"last":0.49399999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":1.1479999999999999,"15min":1.518},"min":{"1min":0.42799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.4410000000000001,"5min":7.4870000000000001,"15min":37.356999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.73199999999999998,"5min":1.157,"15min":1.2609999999999999},"min":{"1min":0.39500000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.5129999999999999,"5min":7.702,"15min":37.25},"last":0.436}]},{"osd":7,"last update":"Fri Feb 9 17:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":1.2849999999999999,"15min":1.544},"min":{"1min":0.39600000000000002,"5min":0.38100000000000001,"15min":0.32800000000000001},"max":{"1min":3.3119999999999998,"5min":8.5120000000000005,"15min":37.585000000000001},"last":0.54600000000000004},{"interface":"front","average":{"1min":1.0269999999999999,"5min":1.3280000000000001,"15min":1.625},"min":{"1min":0.54700000000000004,"5min":0.38500000000000001,"15min":0.312},"max":{"1min":3.327,"5min":8.548,"15min":37.918999999999997},"last":0.56699999999999995}]},{"osd":9,"last update":"Fri Feb 9 17:41:38 2024","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":0.98799999999999999,"15min":1.0940000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":3.9239999999999999,"5min":5.3330000000000002,"15min":9.4900000000000002},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.81200000000000006,"5min":1.2010000000000001,"15min":1.194},"min":{"1min":0.40400000000000003,"5min":0.376,"15min":0.32000000000000001},"max":{"1min":3.423,"5min":7.4189999999999996,"15min":17.434999999999999},"last":0.77100000000000002}]},{"osd":10,"last update":"Fri Feb 9 17:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":0.98899999999999999,"15min":1.075},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":3.194,"5min":4.8170000000000002,"15min":10.65},"last":0.80100000000000005},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.274,"15min":1.1359999999999999},"min":{"1min":0.35899999999999999,"5min":0.34799999999999998,"15min":0.318},"max":{"1min":4.093,"5min":11.071,"15min":11.071},"last":0.504}]},{"osd":11,"last update":"Fri Feb 9 17:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":1.2629999999999999,"15min":1.3420000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":2.464,"5min":8.5519999999999996,"15min":14.433999999999999},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.97599999999999998,"5min":1.268,"15min":1.3140000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":3.1800000000000002,"5min":7.4359999999999999,"15min":11.622},"last":0.625}]}]},{"osd":7,"up_from":57,"seq":244813136131,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2308812,"kb_used_data":1790340,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":91436340,"statfs":{"total":95995035648,"available":93630812160,"internally_reserved":0,"allocated":1833308160,"data_stored":1832785921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":530905546},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:41:21 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":1.4139999999999999,"15min":2.7189999999999999},"min":{"1min":0.41299999999999998,"5min":0.252,"15min":0.252},"max":{"1min":3.472,"5min":14.348000000000001,"15min":36.753},"last":2.2749999999999999},{"interface":"front","average":{"1min":0.69099999999999995,"5min":1.4359999999999999,"15min":2.8719999999999999},"min":{"1min":0.31900000000000001,"5min":0.29099999999999998,"15min":0.253},"max":{"1min":1.2290000000000001,"5min":9.0640000000000001,"15min":36.918999999999997},"last":0.37}]},{"osd":1,"last update":"Fri Feb 9 17:41:21 2024","interfaces":[{"interface":"back","average":{"1min":1.6060000000000001,"5min":2.242,"15min":2.0790000000000002},"min":{"1min":0.50700000000000001,"5min":0.249,"15min":0.249},"max":{"1min":15.509,"5min":21.704999999999998,"15min":21.704999999999998},"last":2.3290000000000002},{"interface":"front","average":{"1min":1.5469999999999999,"5min":2.2280000000000002,"15min":1.8300000000000001},"min":{"1min":0.38100000000000001,"5min":0.26600000000000001,"15min":0.26500000000000001},"max":{"1min":15.276,"5min":21.858000000000001,"15min":21.858000000000001},"last":2.3820000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:39:34 2024","interfaces":[{"interface":"back","average":{"1min":1.8520000000000001,"5min":1.7470000000000001,"15min":1.7030000000000001},"min":{"1min":0.315,"5min":0.28499999999999998,"15min":0.26000000000000001},"max":{"1min":6.6470000000000002,"5min":10.622,"15min":11.324},"last":0.76900000000000002},{"interface":"front","average":{"1min":1.8859999999999999,"5min":1.855,"15min":1.7769999999999999},"min":{"1min":0.34699999999999998,"5min":0.32100000000000001,"15min":0.28299999999999997},"max":{"1min":6.6870000000000003,"5min":19.361999999999998,"15min":19.361999999999998},"last":0.82299999999999995}]},{"osd":3,"last update":"Fri Feb 9 17:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":1.6539999999999999,"15min":1.867},"min":{"1min":0.35899999999999999,"5min":0.307,"15min":0.26900000000000002},"max":{"1min":2.4510000000000001,"5min":7.3899999999999997,"15min":27.719000000000001},"last":2.2160000000000002},{"interface":"front","average":{"1min":0.84599999999999997,"5min":1.627,"15min":1.8600000000000001},"min":{"1min":0.41999999999999998,"5min":0.308,"15min":0.26500000000000001},"max":{"1min":1.861,"5min":11.945,"15min":24.457000000000001},"last":2.3570000000000002}]},{"osd":4,"last update":"Fri Feb 9 17:41:21 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":4.2510000000000003},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":1.448,"5min":1.448,"15min":1.448},"last":7.9509999999999996}]},{"osd":5,"last update":"Fri Feb 9 17:41:21 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.64900000000000002,"15min":0.64500000000000002},"min":{"1min":0.37,"5min":0.249,"15min":0.249},"max":{"1min":1.5209999999999999,"5min":3.0539999999999998,"15min":3.0539999999999998},"last":8.3689999999999998},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.56299999999999994,"15min":0.52600000000000002},"min":{"1min":0.39700000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":8.0069999999999997}]},{"osd":6,"last update":"Fri Feb 9 17:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":1.091,"15min":1.5229999999999999},"min":{"1min":0.36599999999999999,"5min":0.28100000000000003,"15min":0.26000000000000001},"max":{"1min":2.4830000000000001,"5min":8.1929999999999996,"15min":25.965},"last":3.1349999999999998},{"interface":"front","average":{"1min":0.84999999999999998,"5min":1.083,"15min":1.3029999999999999},"min":{"1min":0.44600000000000001,"5min":0.32600000000000001,"15min":0.28699999999999998},"max":{"1min":2.0659999999999998,"5min":7.8810000000000002,"15min":9.4090000000000007},"last":3.298}]},{"osd":8,"last update":"Fri Feb 9 17:41:37 2024","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.1140000000000001,"15min":1.383},"min":{"1min":0.52900000000000003,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":2.7349999999999999,"5min":6.4420000000000002,"15min":9.0809999999999995},"last":3.569},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.0309999999999999,"15min":1.28},"min":{"1min":0.48599999999999999,"5min":0.36599999999999999,"15min":0.33300000000000002},"max":{"1min":2.5760000000000001,"5min":6.0339999999999998,"15min":11.481},"last":3.5880000000000001}]},{"osd":9,"last update":"Fri Feb 9 17:41:28 2024","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":1.226,"15min":1.5309999999999999},"min":{"1min":0.47799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":2.7850000000000001,"5min":6.4749999999999996,"15min":41.286999999999999},"last":3.0619999999999998},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.286,"15min":1.5920000000000001},"min":{"1min":0.56000000000000005,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":4.8440000000000003,"5min":6.5359999999999996,"15min":41.234999999999999},"last":2.2519999999999998}]},{"osd":10,"last update":"Fri Feb 9 17:41:53 2024","interfaces":[{"interface":"back","average":{"1min":1.871,"5min":1.413,"15min":1.4350000000000001},"min":{"1min":0.68799999999999994,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":9.3130000000000006,"5min":12.050000000000001,"15min":12.050000000000001},"last":3.423},{"interface":"front","average":{"1min":1.673,"5min":1.349,"15min":1.5409999999999999},"min":{"1min":0.63900000000000001,"5min":0.39800000000000002,"15min":0.35599999999999998},"max":{"1min":8.8520000000000003,"5min":8.8520000000000003,"15min":9.109},"last":2.3980000000000001}]},{"osd":11,"last update":"Fri Feb 9 17:41:21 2024","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":1.528,"15min":1.7010000000000001},"min":{"1min":0.46100000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":2.0670000000000002,"5min":21.945,"15min":21.945},"last":3.3540000000000001},{"interface":"front","average":{"1min":0.96899999999999997,"5min":1.5940000000000001,"15min":1.5469999999999999},"min":{"1min":0.502,"5min":0.40699999999999997,"15min":0.39500000000000002},"max":{"1min":2.3050000000000002,"5min":21.731999999999999,"15min":21.731999999999999},"last":3.3929999999999998}]}]},{"osd":6,"up_from":49,"seq":210453397768,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1717644,"kb_used_data":1325508,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":92027508,"statfs":{"total":95995035648,"available":94236168192,"internally_reserved":0,"allocated":1357320192,"data_stored":1356797953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":401537478},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:42:13 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":1.579,"15min":1.587},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.21299999999999999},"max":{"1min":4.2519999999999998,"5min":9.5739999999999998,"15min":12.544},"last":6.2549999999999999},{"interface":"front","average":{"1min":0.97899999999999998,"5min":1.5189999999999999,"15min":1.4990000000000001},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.217},"max":{"1min":4.9950000000000001,"5min":9.2520000000000007,"15min":9.2520000000000007},"last":6.3440000000000003}]},{"osd":1,"last update":"Fri Feb 9 17:42:13 2024","interfaces":[{"interface":"back","average":{"1min":1.748,"5min":1.9490000000000001,"15min":1.891},"min":{"1min":0.307,"5min":0.28599999999999998,"15min":0.25900000000000001},"max":{"1min":8.3260000000000005,"5min":8.9689999999999994,"15min":17.762},"last":0.58299999999999996},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.869,"15min":1.903},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":4.5270000000000001,"5min":9.0090000000000003,"15min":17.734999999999999},"last":6.298}]},{"osd":2,"last update":"Fri Feb 9 17:42:13 2024","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":1.472,"15min":1.5880000000000001},"min":{"1min":0.35699999999999998,"5min":0.29399999999999998,"15min":0.22600000000000001},"max":{"1min":4.2160000000000002,"5min":9.8019999999999996,"15min":10.682},"last":6.407},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.5149999999999999,"15min":1.6459999999999999},"min":{"1min":0.372,"5min":0.35599999999999998,"15min":0.28000000000000003},"max":{"1min":4.2850000000000001,"5min":9.8439999999999994,"15min":11.553000000000001},"last":6.5}]},{"osd":3,"last update":"Fri Feb 9 17:42:13 2024","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.7969999999999999,"15min":2.0550000000000002},"min":{"1min":0.28399999999999997,"5min":0.26700000000000002,"15min":0.25600000000000001},"max":{"1min":9.6389999999999993,"5min":11.276,"15min":32.545000000000002},"last":2.4199999999999999},{"interface":"front","average":{"1min":1.6419999999999999,"5min":1.8640000000000001,"15min":1.9179999999999999},"min":{"1min":0.36599999999999999,"5min":0.307,"15min":0.26600000000000001},"max":{"1min":8.2520000000000007,"5min":11.122,"15min":32.670999999999999},"last":0.61899999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:42:13 2024","interfaces":[{"interface":"back","average":{"1min":1.796,"5min":1.03,"15min":1.4379999999999999},"min":{"1min":0.252,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":8.2959999999999994,"5min":8.2959999999999994,"15min":19.798999999999999},"last":1.4530000000000001},{"interface":"front","average":{"1min":1.5660000000000001,"5min":1.147,"15min":1.385},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26000000000000001},"max":{"1min":4.8109999999999999,"5min":5.7270000000000003,"15min":10.702999999999999},"last":7.476}]},{"osd":5,"last update":"Fri Feb 9 17:42:13 2024","interfaces":[{"interface":"back","average":{"1min":1.7629999999999999,"5min":1.3180000000000001,"15min":1.4239999999999999},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29199999999999998},"max":{"1min":5.6790000000000003,"5min":6.7679999999999998,"15min":10.968999999999999},"last":6.3760000000000003},{"interface":"front","average":{"1min":1.879,"5min":1.226,"15min":1.4650000000000001},"min":{"1min":0.33700000000000002,"5min":0.316,"15min":0.29299999999999998},"max":{"1min":8.3989999999999991,"5min":8.3989999999999991,"15min":13.91},"last":0.64700000000000002}]},{"osd":7,"last update":"Fri Feb 9 17:41:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.0960000000000001,"15min":1.4350000000000001},"min":{"1min":0.30199999999999999,"5min":0.29399999999999998,"15min":0.28000000000000003},"max":{"1min":2.8250000000000002,"5min":5.4029999999999996,"15min":14.022},"last":6.4359999999999999},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.288,"15min":1.6100000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.309},"max":{"1min":4.234,"5min":12.441000000000001,"15min":15.458},"last":2.6850000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:26:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":0.876,"15min":0.79600000000000004},"min":{"1min":0.39900000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":4.0339999999999998,"5min":4.4880000000000004,"15min":4.4880000000000004},"last":4.3840000000000003},{"interface":"front","average":{"1min":1.0469999999999999,"5min":0.88600000000000001,"15min":0.78300000000000003},"min":{"1min":0.375,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":3.6539999999999999,"5min":4.8570000000000002,"15min":4.8570000000000002},"last":4.4210000000000003}]},{"osd":9,"last update":"Fri Feb 9 17:41:16 2024","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.089,"15min":1.2869999999999999},"min":{"1min":0.499,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":5.5789999999999997,"5min":10.779999999999999,"15min":10.803000000000001},"last":6.4710000000000001},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.218,"15min":1.415},"min":{"1min":0.61299999999999999,"5min":0.40999999999999998,"15min":0.40300000000000002},"max":{"1min":5.5419999999999998,"5min":6.3650000000000002,"15min":8.1630000000000003},"last":6.5640000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:41:55 2024","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":1.1619999999999999,"15min":1.4139999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":8.3580000000000005,"5min":10.677,"15min":10.677},"last":0.67600000000000005},{"interface":"front","average":{"1min":1.214,"5min":1.1060000000000001,"15min":1.4850000000000001},"min":{"1min":0.40899999999999997,"5min":0.38,"15min":0.35899999999999999},"max":{"1min":3.9420000000000002,"5min":5.9509999999999996,"15min":13.836},"last":2.4809999999999999}]},{"osd":11,"last update":"Fri Feb 9 17:42:13 2024","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.2889999999999999,"15min":1.4470000000000001},"min":{"1min":0.46200000000000002,"5min":0.45500000000000002,"15min":0.36099999999999999},"max":{"1min":4.3179999999999996,"5min":7.7450000000000001,"15min":10.273999999999999},"last":6.5289999999999999},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.171,"15min":1.4890000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":4.0620000000000003,"5min":5.2140000000000004,"15min":8.2739999999999991},"last":2.52}]}]},{"osd":1,"up_from":13,"seq":55834575138,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2314828,"kb_used_data":1764420,"kb_used_omap":1,"kb_used_meta":550398,"kb_avail":91430324,"statfs":{"total":95995035648,"available":93624651776,"internally_reserved":0,"allocated":1806766080,"data_stored":1806243841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":563608005},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:41:27 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":2.6509999999999998,"15min":1.8759999999999999},"min":{"1min":0.25,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":1.9450000000000001,"5min":19.079000000000001,"15min":19.079000000000001},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.44700000000000001,"5min":2.2650000000000001,"15min":1.8089999999999999},"min":{"1min":0.26300000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.99299999999999999,"5min":23.573,"15min":23.573},"last":0.25900000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:41:50 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":2.0470000000000002,"15min":1.829},"min":{"1min":0.28999999999999998,"5min":0.23200000000000001,"15min":0.22600000000000001},"max":{"1min":0.54300000000000004,"5min":12.52,"15min":26.361999999999998},"last":0.98499999999999999},{"interface":"front","average":{"1min":0.63200000000000001,"5min":2.339,"15min":2.0840000000000001},"min":{"1min":0.27400000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":1.6299999999999999,"5min":11.346,"15min":26.087},"last":0.28100000000000003}]},{"osd":3,"last update":"Fri Feb 9 17:40:36 2024","interfaces":[{"interface":"back","average":{"1min":1.8959999999999999,"5min":2.2690000000000001,"15min":1.6060000000000001},"min":{"1min":0.26200000000000001,"5min":0.245,"15min":0.245},"max":{"1min":6.9400000000000004,"5min":10.657,"15min":10.657},"last":0.30299999999999999},{"interface":"front","average":{"1min":2.2349999999999999,"5min":2.508,"15min":1.734},"min":{"1min":0.36599999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":7.1029999999999998,"5min":14.68,"15min":14.68},"last":1.25}]},{"osd":4,"last update":"Fri Feb 9 17:41:21 2024","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.9259999999999999,"15min":1.966},"min":{"1min":0.34699999999999998,"5min":0.25800000000000001,"15min":0.253},"max":{"1min":4.694,"5min":9.4649999999999999,"15min":11.766999999999999},"last":1.6819999999999999},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.927,"15min":1.9039999999999999},"min":{"1min":0.38800000000000001,"5min":0.309,"15min":0.29999999999999999},"max":{"1min":4.5730000000000004,"5min":10.512,"15min":10.512},"last":0.36799999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:42:09 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.9319999999999999,"15min":1.8480000000000001},"min":{"1min":0.29199999999999998,"5min":0.28999999999999998,"15min":0.27000000000000002},"max":{"1min":5.9429999999999996,"5min":9.4299999999999997,"15min":13.544},"last":0.67400000000000004},{"interface":"front","average":{"1min":1.323,"5min":2.0419999999999998,"15min":1.774},"min":{"1min":0.32300000000000001,"5min":0.30499999999999999,"15min":0.26000000000000001},"max":{"1min":5.9569999999999999,"5min":10.531000000000001,"15min":12.117000000000001},"last":0.71199999999999997}]},{"osd":6,"last update":"Fri Feb 9 17:42:13 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.2609999999999999,"15min":1.2689999999999999},"min":{"1min":0.436,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":4.9160000000000004,"5min":5.0119999999999996,"15min":5.0119999999999996},"last":1.1479999999999999},{"interface":"front","average":{"1min":1.238,"5min":1.1870000000000001,"15min":1.179},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":4.952,"5min":4.952,"15min":4.952},"last":1.1759999999999999}]},{"osd":7,"last update":"Fri Feb 9 17:42:13 2024","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":1.147,"15min":1.153},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":3.8700000000000001,"5min":4.0439999999999996,"15min":4.0439999999999996},"last":0.68500000000000005},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.21,"15min":1.228},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":4.0389999999999997,"5min":4.9820000000000002,"15min":4.9820000000000002},"last":0.51300000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:41:21 2024","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":2.121,"5min":2.121,"15min":2.121},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":2.1379999999999999,"5min":2.1379999999999999,"15min":2.1379999999999999},"last":0.59299999999999997}]},{"osd":9,"last update":"Fri Feb 9 17:41:27 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":1.9810000000000001,"15min":1.8109999999999999},"min":{"1min":0.42999999999999999,"5min":0.33400000000000002,"15min":0.32400000000000001},"max":{"1min":1.48,"5min":9.4960000000000004,"15min":9.5839999999999996},"last":0.52900000000000003},{"interface":"front","average":{"1min":1.1100000000000001,"5min":2.1400000000000001,"15min":1.8420000000000001},"min":{"1min":0.38700000000000001,"5min":0.32900000000000001,"15min":0.30099999999999999},"max":{"1min":3.9710000000000001,"5min":11.366,"15min":12.117000000000001},"last":0.57299999999999995}]},{"osd":10,"last update":"Fri Feb 9 17:42:09 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":2.0099999999999998,"15min":1.6559999999999999},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.35999999999999999},"max":{"1min":1.1699999999999999,"5min":15.083,"15min":15.083},"last":0.621},{"interface":"front","average":{"1min":0.77400000000000002,"5min":2.0529999999999999,"15min":1.871},"min":{"1min":0.51200000000000001,"5min":0.48199999999999998,"15min":0.33400000000000002},"max":{"1min":1.268,"5min":15.121,"15min":15.121},"last":0.64500000000000002}]},{"osd":11,"last update":"Fri Feb 9 17:42:13 2024","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":2.1509999999999998,"5min":2.1509999999999998,"15min":2.1509999999999998},"last":0.497},{"interface":"front","average":{"1min":1.226,"5min":1.226,"15min":1.226},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":3.0910000000000002,"5min":3.0910000000000002,"15min":3.0910000000000002},"last":2.4079999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738662,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3644612,"kb_used_data":3081540,"kb_used_omap":1,"kb_used_meta":562878,"kb_avail":90100540,"statfs":{"total":95995035648,"available":92262952960,"internally_reserved":0,"allocated":3155496960,"data_stored":3154974721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":576387525},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 9 17:42:11 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":1.833,"15min":1.6000000000000001},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.20499999999999999},"max":{"1min":3.6600000000000001,"5min":13.842000000000001,"15min":14.166},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.77100000000000002,"5min":2.0859999999999999,"15min":1.833},"min":{"1min":0.32000000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.7050000000000001,"5min":14.122,"15min":20.177},"last":0.45600000000000002}]},{"osd":2,"last update":"Fri Feb 9 17:39:25 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":5.5339999999999998,"5min":5.5339999999999998,"15min":5.5339999999999998},"last":1.744},{"interface":"front","average":{"1min":1.593,"5min":1.593,"15min":1.593},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":13.473000000000001,"5min":13.473000000000001,"15min":13.473000000000001},"last":3.109}]},{"osd":3,"last update":"Fri Feb 9 17:41:40 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":2.52,"15min":2.855},"min":{"1min":0.27200000000000002,"5min":0.247,"15min":0.247},"max":{"1min":2.4729999999999999,"5min":16.756,"15min":16.756},"last":2.3780000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":1.538,"15min":1.716},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":1.7689999999999999,"5min":6.0570000000000004,"15min":6.0570000000000004},"last":1.109}]},{"osd":4,"last update":"Fri Feb 9 17:41:30 2024","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":1.2450000000000001,"15min":1.673},"min":{"1min":0.311,"5min":0.29399999999999998,"15min":0.26400000000000001},"max":{"1min":2.9169999999999998,"5min":7.7560000000000002,"15min":14.419},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.85899999999999999,"5min":1.657,"15min":1.913},"min":{"1min":0.39700000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":2.5129999999999999,"5min":13.308999999999999,"15min":14.465999999999999},"last":0.91800000000000004}]},{"osd":5,"last update":"Fri Feb 9 17:41:51 2024","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.4370000000000001,"15min":1.6419999999999999},"min":{"1min":0.32200000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":3.976,"5min":8.4079999999999995,"15min":9.8569999999999993},"last":1.319},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.363,"15min":1.76},"min":{"1min":0.39900000000000002,"5min":0.371,"15min":0.312},"max":{"1min":3.3900000000000001,"5min":7.8239999999999998,"15min":9.9049999999999994},"last":2.1320000000000001}]},{"osd":6,"last update":"Fri Feb 9 17:41:18 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":1.673,"15min":1.913},"min":{"1min":0.374,"5min":0.33300000000000002,"15min":0.27800000000000002},"max":{"1min":2.7810000000000001,"5min":13.407,"15min":13.407},"last":2.5489999999999999},{"interface":"front","average":{"1min":0.80600000000000005,"5min":1.7250000000000001,"15min":1.8939999999999999},"min":{"1min":0.39200000000000002,"5min":0.379,"15min":0.33000000000000002},"max":{"1min":2.8079999999999998,"5min":13.448,"15min":13.448},"last":2.5800000000000001}]},{"osd":7,"last update":"Fri Feb 9 17:41:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":2.0299999999999998,"15min":2.056},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":2.3279999999999998,"5min":13.349,"15min":13.349},"last":1.865},{"interface":"front","average":{"1min":0.94699999999999995,"5min":2.1150000000000002,"15min":2.0659999999999998},"min":{"1min":0.35899999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":2.097,"5min":14.451000000000001,"15min":16.765999999999998},"last":2.3959999999999999}]},{"osd":8,"last update":"Fri Feb 9 17:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":1.9419999999999999,"15min":1.5840000000000001},"min":{"1min":0.371,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":2.145,"5min":14.323,"15min":14.323},"last":0.441},{"interface":"front","average":{"1min":1.016,"5min":2.0720000000000001,"15min":1.673},"min":{"1min":0.372,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":3.0099999999999998,"5min":14.481999999999999,"15min":16.41},"last":0.41199999999999998}]},{"osd":9,"last update":"Fri Feb 9 17:41:30 2024","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":2.1120000000000001,"15min":1.865},"min":{"1min":0.433,"5min":0.32500000000000001,"15min":0.312},"max":{"1min":2.407,"5min":17.952000000000002,"15min":17.952000000000002},"last":0.33800000000000002},{"interface":"front","average":{"1min":1.0129999999999999,"5min":2.2250000000000001,"15min":1.899},"min":{"1min":0.57299999999999995,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":2.3290000000000002,"5min":17.913,"15min":17.913},"last":0.46600000000000003}]},{"osd":10,"last update":"Fri Feb 9 17:41:51 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":2.137,"15min":1.8280000000000001},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.40899999999999997},"max":{"1min":3.2040000000000002,"5min":14.271000000000001,"15min":16.876000000000001},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.73399999999999999,"5min":1.282,"15min":1.4830000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":2.5659999999999998,"5min":11.384,"15min":11.384},"last":0.42199999999999999}]},{"osd":11,"last update":"Fri Feb 9 17:41:18 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":1.3819999999999999,"15min":1.7509999999999999},"min":{"1min":0.41899999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":2.5430000000000001,"5min":7.8739999999999997,"15min":40.021000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.4790000000000001,"15min":1.905},"min":{"1min":0.5,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":2.702,"5min":7.899,"15min":40.055999999999997},"last":1.653}]}]},{"osd":2,"up_from":0,"seq":77309411612,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":27,"seq":115964117273,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":850196,"kb_used_data":444172,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":92894956,"statfs":{"total":95995035648,"available":95124434944,"internally_reserved":0,"allocated":454832128,"data_stored":454306337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":415758791},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:42:09 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":2.423,"15min":1.8640000000000001},"min":{"1min":0.23300000000000001,"5min":0.186,"15min":0.186},"max":{"1min":15.657999999999999,"5min":38.984999999999999,"15min":38.984999999999999},"last":0.61799999999999999},{"interface":"front","average":{"1min":1.3839999999999999,"5min":2.528,"15min":1.97},"min":{"1min":0.25800000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":16.637,"5min":41.497999999999998,"15min":41.497999999999998},"last":0.28599999999999998}]},{"osd":1,"last update":"Fri Feb 9 17:40:06 2024","interfaces":[{"interface":"back","average":{"1min":4.5670000000000002,"5min":4.5670000000000002,"15min":4.5670000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":25.965,"5min":25.965,"15min":25.965},"last":0.35299999999999998},{"interface":"front","average":{"1min":4.5129999999999999,"5min":4.5129999999999999,"15min":4.5129999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":31.454000000000001,"5min":31.454000000000001,"15min":31.454000000000001},"last":0.375}]},{"osd":2,"last update":"Fri Feb 9 17:42:04 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":2.0600000000000001,"15min":1.802},"min":{"1min":0.32200000000000001,"5min":0.255,"15min":0.22600000000000001},"max":{"1min":4.2409999999999997,"5min":11.663,"15min":14.914999999999999},"last":0.307},{"interface":"front","average":{"1min":0.67300000000000004,"5min":1.9239999999999999,"15min":1.8260000000000001},"min":{"1min":0.27600000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":4.258,"5min":12.347,"15min":17.613},"last":0.32600000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:42:11 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":1.4670000000000001,"15min":1.806},"min":{"1min":0.30399999999999999,"5min":0.29099999999999998,"15min":0.26800000000000002},"max":{"1min":2.556,"5min":13.933,"15min":34.898000000000003},"last":0.34699999999999998},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.645,"15min":1.788},"min":{"1min":0.32800000000000001,"5min":0.28000000000000003,"15min":0.27900000000000003},"max":{"1min":4.415,"5min":9.3059999999999992,"15min":14.061999999999999},"last":0.63300000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:41:47 2024","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":1.72,"15min":1.6739999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.27800000000000002},"max":{"1min":3.1669999999999998,"5min":11.007999999999999,"15min":17.651},"last":3.4569999999999999},{"interface":"front","average":{"1min":0.93700000000000006,"5min":1.784,"15min":1.728},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.34100000000000003},"max":{"1min":3.1299999999999999,"5min":11.036,"15min":17.675999999999998},"last":3.4689999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":2.3370000000000002,"15min":2.609},"min":{"1min":0.39400000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.669,"5min":12.717000000000001,"15min":12.717000000000001},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.86099999999999999,"5min":1.2969999999999999,"15min":1.369},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":2.9409999999999998,"5min":8.7360000000000007,"15min":8.7360000000000007},"last":0.40799999999999997}]},{"osd":7,"last update":"Fri Feb 9 17:41:20 2024","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.872,"15min":0.872},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":3.3999999999999999,"5min":3.3999999999999999,"15min":3.3999999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":2.7989999999999999,"5min":2.7989999999999999,"15min":2.7989999999999999},"last":2.9830000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:42:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.653,"15min":1.427},"min":{"1min":0.41299999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":4.3289999999999997,"5min":8.6940000000000008,"15min":8.6940000000000008},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.7569999999999999,"15min":1.222},"min":{"1min":0.42799999999999999,"5min":0.36599999999999999,"15min":0.34499999999999997},"max":{"1min":3.6309999999999998,"5min":13.976000000000001,"15min":14.129},"last":0.45500000000000002}]},{"osd":9,"last update":"Fri Feb 9 17:41:30 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":1.7250000000000001,"15min":1.5429999999999999},"min":{"1min":0.53500000000000003,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.821,"5min":9.7929999999999993,"15min":9.7929999999999993},"last":0.64600000000000002},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.7090000000000001,"15min":1.7490000000000001},"min":{"1min":0.51400000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":2.6160000000000001,"5min":12.815,"15min":13.717000000000001},"last":0.67400000000000004}]},{"osd":10,"last update":"Fri Feb 9 17:41:16 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":2.2850000000000001,"5min":2.2850000000000001,"15min":2.2850000000000001},"last":0.437},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.248,"5min":1.248,"15min":1.248},"last":0.39500000000000002}]},{"osd":11,"last update":"Fri Feb 9 17:42:11 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":1.5680000000000001,"15min":1.609},"min":{"1min":0.42699999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.7410000000000001,"5min":14.300000000000001,"15min":14.300000000000001},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.87,"5min":2.1859999999999999,"15min":1.8779999999999999},"min":{"1min":0.50700000000000001,"5min":0.442,"15min":0.36899999999999999},"max":{"1min":1.917,"5min":17.145,"15min":17.145},"last":0.54800000000000004}]}]},{"osd":4,"up_from":32,"seq":137438953746,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3091340,"kb_used_data":2645060,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":90653812,"statfs":{"total":95995035648,"available":92829503488,"internally_reserved":0,"allocated":2708541440,"data_stored":2708019201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":456980933},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:41:14 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":1.6930000000000001,"15min":1.7150000000000001},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23599999999999999},"max":{"1min":2.073,"5min":7.9669999999999996,"15min":50.356999999999999},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.77800000000000002,"5min":1.851,"15min":1.6479999999999999},"min":{"1min":0.34599999999999997,"5min":0.247,"15min":0.247},"max":{"1min":2.633,"5min":8.0440000000000005,"15min":50.569000000000003},"last":0.36299999999999999}]},{"osd":1,"last update":"Fri Feb 9 17:41:18 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":2.3420000000000001,"15min":2.0539999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":4.0540000000000003,"5min":28.143000000000001,"15min":50.473999999999997},"last":1.851},{"interface":"front","average":{"1min":1.073,"5min":2.3799999999999999,"15min":2.1179999999999999},"min":{"1min":0.35899999999999999,"5min":0.31,"15min":0.28100000000000003},"max":{"1min":4.0679999999999996,"5min":28.103000000000002,"15min":50.499000000000002},"last":2.1749999999999998}]},{"osd":2,"last update":"Fri Feb 9 17:28:51 2024","interfaces":[{"interface":"back","average":{"1min":2.536,"5min":1.546,"15min":0.91400000000000003},"min":{"1min":0.29999999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":8.8819999999999997,"5min":8.8819999999999997,"15min":8.8819999999999997},"last":0.622},{"interface":"front","average":{"1min":2.214,"5min":1.7330000000000001,"15min":0.97799999999999998},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":9.5039999999999996,"5min":9.5039999999999996,"15min":9.5039999999999996},"last":0.53800000000000003}]},{"osd":3,"last update":"Fri Feb 9 17:41:18 2024","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":1.8109999999999999,"15min":2.1619999999999999},"min":{"1min":0.311,"5min":0.245,"15min":0.245},"max":{"1min":3.7000000000000002,"5min":7.9349999999999996,"15min":50.462000000000003},"last":2.2130000000000001},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.8700000000000001,"15min":2.0009999999999999},"min":{"1min":0.40699999999999997,"5min":0.29699999999999999,"15min":0.255},"max":{"1min":4.8330000000000002,"5min":9.1579999999999995,"15min":50.529000000000003},"last":0.39800000000000002}]},{"osd":5,"last update":"Fri Feb 9 17:41:47 2024","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.0249999999999999,"15min":1.333},"min":{"1min":0.248,"5min":0.22500000000000001,"15min":0.222},"max":{"1min":3.351,"5min":6.3760000000000003,"15min":9.5630000000000006},"last":2.3159999999999998},{"interface":"front","average":{"1min":1.216,"5min":1.1060000000000001,"15min":1.607},"min":{"1min":0.39400000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":3.4350000000000001,"5min":6.4080000000000004,"15min":50.509999999999998},"last":2.3439999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:41:18 2024","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.333,"15min":1.474},"min":{"1min":0.28999999999999998,"5min":0.246,"15min":0.246},"max":{"1min":7.4249999999999998,"5min":8.0039999999999996,"15min":18.09},"last":0.44900000000000001},{"interface":"front","average":{"1min":1.137,"5min":1.137,"15min":1.6160000000000001},"min":{"1min":0.442,"5min":0.245,"15min":0.245},"max":{"1min":5.524,"5min":6.7329999999999997,"15min":50.402999999999999},"last":7.6619999999999999}]},{"osd":7,"last update":"Fri Feb 9 17:41:53 2024","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":1.173,"15min":1.619},"min":{"1min":0.45900000000000002,"5min":0.29899999999999999,"15min":0.28399999999999997},"max":{"1min":3.536,"5min":7.4640000000000004,"15min":15.223000000000001},"last":0.42399999999999999},{"interface":"front","average":{"1min":1.6120000000000001,"5min":1.2290000000000001,"15min":1.6120000000000001},"min":{"1min":0.37,"5min":0.28199999999999997,"15min":0.28100000000000003},"max":{"1min":4.0119999999999996,"5min":6.7640000000000002,"15min":19.047000000000001},"last":12.696}]},{"osd":8,"last update":"Fri Feb 9 17:41:47 2024","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.073,"15min":1.413},"min":{"1min":0.47199999999999998,"5min":0.34799999999999998,"15min":0.34399999999999997},"max":{"1min":3.5880000000000001,"5min":6.1079999999999997,"15min":8.4160000000000004},"last":1.913},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.056,"15min":1.5409999999999999},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.308},"max":{"1min":3.214,"5min":6,"15min":50.594000000000001},"last":0.51000000000000001}]},{"osd":9,"last update":"Fri Feb 9 17:41:28 2024","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":1.171,"15min":1.6419999999999999},"min":{"1min":0.443,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":3.5449999999999999,"5min":6.9699999999999998,"15min":50.442999999999998},"last":1.8819999999999999},{"interface":"front","average":{"1min":0.92900000000000005,"5min":1.2450000000000001,"15min":1.446},"min":{"1min":0.42499999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":3.7679999999999998,"5min":7.5999999999999996,"15min":10.555},"last":0.47899999999999998}]},{"osd":10,"last update":"Fri Feb 9 17:42:03 2024","interfaces":[{"interface":"back","average":{"1min":1.4359999999999999,"5min":1.607,"15min":1.891},"min":{"1min":0.52500000000000002,"5min":0.40300000000000002,"15min":0.378},"max":{"1min":3.6160000000000001,"5min":7.0030000000000001,"15min":51.223999999999997},"last":1.9650000000000001},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.4610000000000001,"15min":1.9099999999999999},"min":{"1min":0.42899999999999999,"5min":0.41999999999999998,"15min":0.38100000000000001},"max":{"1min":3.617,"5min":6.3860000000000001,"15min":50.552},"last":2.25}]},{"osd":11,"last update":"Fri Feb 9 17:41:28 2024","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.4910000000000001,"15min":1.9430000000000001},"min":{"1min":0.59699999999999998,"5min":0.49099999999999999,"15min":0.40500000000000003},"max":{"1min":3.6469999999999998,"5min":6.9379999999999997,"15min":50.484999999999999},"last":1.9410000000000001},{"interface":"front","average":{"1min":0.89400000000000002,"5min":1.351,"15min":1.6220000000000001},"min":{"1min":0.46500000000000002,"5min":0.435,"15min":0.34899999999999998},"max":{"1min":2.859,"5min":7.694,"15min":18.574999999999999},"last":0.57599999999999996}]}]},{"osd":5,"up_from":40,"seq":171798692108,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3417300,"kb_used_data":3029324,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":90327852,"statfs":{"total":95995035648,"available":92495720448,"internally_reserved":0,"allocated":3102027776,"data_stored":3101501985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":397277637},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:42:11 2024","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.046,"15min":1.05},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":7.3730000000000002,"5min":7.3730000000000002,"15min":7.3730000000000002},"last":5.5410000000000004},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.51,"15min":1.712},"min":{"1min":0.28699999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":3.2589999999999999,"5min":6.4939999999999998,"15min":6.4939999999999998},"last":5.9080000000000004}]},{"osd":1,"last update":"Fri Feb 9 17:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":1.8260000000000001,"15min":2.0059999999999998},"min":{"1min":0.245,"5min":0.24199999999999999,"15min":0.23200000000000001},"max":{"1min":2.254,"5min":48.161999999999999,"15min":48.161999999999999},"last":5.9820000000000002},{"interface":"front","average":{"1min":1.0980000000000001,"5min":2.056,"15min":1.8979999999999999},"min":{"1min":0.25900000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":6.2290000000000001,"5min":48.218000000000004,"15min":48.218000000000004},"last":7.569}]},{"osd":2,"last update":"Fri Feb 9 17:40:42 2024","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":1.345,"15min":1.6539999999999999},"min":{"1min":0.32000000000000001,"5min":0.27900000000000003,"15min":0.25600000000000001},"max":{"1min":4.4930000000000003,"5min":11.143000000000001,"15min":11.143000000000001},"last":2.6560000000000001},{"interface":"front","average":{"1min":1.796,"5min":1.218,"15min":1.718},"min":{"1min":0.39100000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":6.1970000000000001,"5min":6.1970000000000001,"15min":14.122},"last":1.5649999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":1.6850000000000001,"15min":2.149},"min":{"1min":0.32800000000000001,"5min":0.29799999999999999,"15min":0.26500000000000001},"max":{"1min":2.9980000000000002,"5min":16.59,"15min":64.150999999999996},"last":5.6059999999999999},{"interface":"front","average":{"1min":0.85699999999999998,"5min":1.6970000000000001,"15min":2.101},"min":{"1min":0.44400000000000001,"5min":0.247,"15min":0.247},"max":{"1min":2.2170000000000001,"5min":12.468,"15min":64.176000000000002},"last":6.0129999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:41:46 2024","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.6659999999999999,"15min":1.5069999999999999},"min":{"1min":0.33600000000000002,"5min":0.28100000000000003,"15min":0.23400000000000001},"max":{"1min":8.0169999999999995,"5min":49.313000000000002,"15min":49.313000000000002},"last":5.5209999999999999},{"interface":"front","average":{"1min":1.2,"5min":1.514,"15min":1.4970000000000001},"min":{"1min":0.35399999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.5960000000000001,"5min":50.505000000000003,"15min":50.505000000000003},"last":5.8659999999999997}]},{"osd":6,"last update":"Fri Feb 9 17:41:21 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":1.2010000000000001,"15min":1.4350000000000001},"min":{"1min":0.32100000000000001,"5min":0.26400000000000001,"15min":0.253},"max":{"1min":2.1930000000000001,"5min":8.609,"15min":15.375999999999999},"last":5.9480000000000004},{"interface":"front","average":{"1min":0.95399999999999996,"5min":1.29,"15min":1.508},"min":{"1min":0.40799999999999997,"5min":0.28899999999999998,"15min":0.27200000000000002},"max":{"1min":2.7530000000000001,"5min":11.038,"15min":14.250999999999999},"last":5.4960000000000004}]},{"osd":7,"last update":"Fri Feb 9 17:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":1.589,"15min":1.536},"min":{"1min":0.38500000000000001,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":2.3039999999999998,"5min":50.545000000000002,"15min":50.545000000000002},"last":6.1230000000000002},{"interface":"front","average":{"1min":0.96999999999999997,"5min":2.5800000000000001,"15min":2.0760000000000001},"min":{"1min":0.32400000000000001,"5min":0.32100000000000001,"15min":0.28799999999999998},"max":{"1min":2.8730000000000002,"5min":102.526,"15min":102.526},"last":0.42099999999999999}]},{"osd":8,"last update":"Fri Feb 9 17:41:21 2024","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":1.361,"15min":1.581},"min":{"1min":0.40899999999999997,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":14.542,"5min":14.542,"15min":14.542},"last":6.0659999999999998},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.94699999999999995,"15min":1.764},"min":{"1min":0.40000000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.524,"5min":5.71,"15min":9.6240000000000006},"last":0.378}]},{"osd":9,"last update":"Fri Feb 9 17:41:21 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":1.23,"15min":1.3700000000000001},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.35999999999999999},"max":{"1min":1.4179999999999999,"5min":6.8600000000000003,"15min":14.98},"last":5.5590000000000002},{"interface":"front","average":{"1min":0.995,"5min":1.1950000000000001,"15min":1.369},"min":{"1min":0.45300000000000001,"5min":0.35699999999999998,"15min":0.312},"max":{"1min":5.0439999999999996,"5min":7.7779999999999996,"15min":9.4239999999999995},"last":0.40400000000000003}]},{"osd":10,"last update":"Fri Feb 9 17:41:46 2024","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.069,"15min":1.4359999999999999},"min":{"1min":0.51800000000000002,"5min":0.35199999999999998,"15min":0.33800000000000002},"max":{"1min":3.843,"5min":5.7400000000000002,"15min":14.218},"last":0.438},{"interface":"front","average":{"1min":0.89900000000000002,"5min":1.2969999999999999,"15min":1.579},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.36099999999999999},"max":{"1min":2.8239999999999998,"5min":10.606,"15min":14.186999999999999},"last":6.0339999999999998}]},{"osd":11,"last update":"Fri Feb 9 17:42:11 2024","interfaces":[{"interface":"back","average":{"1min":1.5369999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":3.597,"5min":3.597,"15min":3.597},"last":5.4550000000000001},{"interface":"front","average":{"1min":1.653,"5min":1.653,"15min":1.653},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":5.593,"5min":5.593,"15min":5.593},"last":6.0940000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3156082688,"data_stored":3156082688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1821900800,"data_stored":1821900800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":453705728,"data_stored":453705728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2723414016,"data_stored":2723414016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3121938432,"data_stored":3121938432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1364721664,"data_stored":1364721664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1844707328,"data_stored":1844707328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1369899008,"data_stored":1369899008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1372717056,"data_stored":1372717056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2789343232,"data_stored":2789343232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1824915456,"data_stored":1824915456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-09T17:42:25.548 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd pool set .mgr pgp_num 21 2024-02-09T17:42:25.556 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-09T17:42:25.888 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:42:25.891 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:25.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:42:25.907 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:25 smithi162 ceph-mon[44304]: pgmap v1269: 37 pgs: 37 active+clean; 7.2 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-02-09T17:42:25.907 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:25 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:42:25.907 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:25 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:42:25.907 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:25 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/153113385' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:42:25.907 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:25 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:42:25.914 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is unset 2024-02-09T17:42:25.963 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 81 16 121367 121351 93.6209 61.5625 0.0198217 0.010677 2024-02-09T17:42:25.983 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:42:25.992 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:42:26.030 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:26.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:25 smithi179 ceph-mon[43975]: pgmap v1269: 37 pgs: 37 active+clean; 7.2 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-02-09T17:42:26.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:25 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:42:26.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:25 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:42:26.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:25 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/153113385' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:42:26.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:25 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:42:26.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:25 smithi099 ceph-mon[39420]: pgmap v1269: 37 pgs: 37 active+clean; 7.2 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-02-09T17:42:26.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:25 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:42:26.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:25 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:42:26.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:25 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/153113385' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:42:26.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:25 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:42:26.392 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:42:26.392 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:42:26.392 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:42:26.392 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:42:26.392 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:42:26.398 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:42:26.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:26.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:42:26.418 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:42:26.457 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:42:26.495 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:42:26.513 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:42:26.531 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:42:26.548 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:26.563 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:42:26.596 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:42:26.629 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:42:26.631 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:42:26.631 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:42:26.632 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:42:26.632 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:42:26.632 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:42:26.632 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:42:26.632 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:42:26.632 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:42:26.632 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:42:26.632 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:42:26.633 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:42:26.633 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:42:26.920 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:42:26.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:26.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:42:26.963 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 82 16 122367 122351 93.2414 62.5 0.0125973 0.0107206 2024-02-09T17:42:27.027 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:42:27.098 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:27.135 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:42:27.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:27.139 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:42:27.239 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:42:27.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:26 smithi099 ceph-mon[39420]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:42:27.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:26 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:42:27.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:26 smithi099 ceph-mon[39420]: osdmap e500: 12 total, 12 up, 11 in 2024-02-09T17:42:27.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:26 smithi099 ceph-mon[39420]: 5.5 scrub starts 2024-02-09T17:42:27.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:26 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/114144259' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:42:27.270 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:26 smithi162 ceph-mon[44304]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:42:27.270 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:26 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:42:27.270 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:26 smithi162 ceph-mon[44304]: osdmap e500: 12 total, 12 up, 11 in 2024-02-09T17:42:27.270 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:26 smithi162 ceph-mon[44304]: 5.5 scrub starts 2024-02-09T17:42:27.270 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:26 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/114144259' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:42:27.272 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:27.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:26 smithi179 ceph-mon[43975]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:42:27.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:26 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:42:27.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:26 smithi179 ceph-mon[43975]: osdmap e500: 12 total, 12 up, 11 in 2024-02-09T17:42:27.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:26 smithi179 ceph-mon[43975]: 5.5 scrub starts 2024-02-09T17:42:27.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:26 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/114144259' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:42:27.633 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:42:27.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:27.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:42:27.686 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:42:27.739 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:42:27.797 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:27.963 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 83 16 123357 123341 92.8634 61.875 0.0134784 0.0107636 2024-02-09T17:42:28.070 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:42:28.077 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:27 smithi162 ceph-mon[44304]: pgmap v1271: 37 pgs: 37 active+clean; 7.3 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-02-09T17:42:28.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:28.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:42:28.128 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:42:28.128 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-02-09T17:42:28.128 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:28.128 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25528.0:123480 5.9 e500/465 5:9177709f:::benchmark_data_smithi179_115199_object123479:head v 500'7784, mlcod=500'7781)", 2024-02-09T17:42:28.128 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:42:28.106064+0000", 2024-02-09T17:42:28.128 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.018828437, 2024-02-09T17:42:28.128 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018859476, 2024-02-09T17:42:28.128 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:28.128 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-09T17:42:28.128 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:28.128 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:28.128 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:28.129 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:28.106064+0000", 2024-02-09T17:42:28.129 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:28.129 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:28.129 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:28.129 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:28.129 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:28.106064+0000", 2024-02-09T17:42:28.129 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:28.129 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:28.129 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:28.129 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:28.129 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:28.106065+0000", 2024-02-09T17:42:28.129 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9999999999999999e-07 2024-02-09T17:42:28.129 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:28.129 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:28.129 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:28.130 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:28.106087+0000", 2024-02-09T17:42:28.130 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2337999999999999e-05 2024-02-09T17:42:28.130 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:28.130 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:28.130 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:28.130 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:28.106093+0000", 2024-02-09T17:42:28.130 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1109999999999996e-06 2024-02-09T17:42:28.130 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:28.130 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:28.130 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:28.130 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:28.106098+0000", 2024-02-09T17:42:28.130 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0860000000000001e-06 2024-02-09T17:42:28.130 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:28.130 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:28.130 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:28.131 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:28.110162+0000", 2024-02-09T17:42:28.131 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.004064247 2024-02-09T17:42:28.131 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:28.131 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:28.131 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:28.131 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:28.110183+0000", 2024-02-09T17:42:28.131 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0965e-05 2024-02-09T17:42:28.131 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:28.131 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:28.131 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:28.131 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:28.131 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:28.131 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25528.0:123485 5.9 e500/465 5:9478055a:::benchmark_data_smithi179_115199_object123484:head v 500'7785, mlcod=500'7781)", 2024-02-09T17:42:28.131 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:42:28.107862+0000", 2024-02-09T17:42:28.131 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.017030751, 2024-02-09T17:42:28.131 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.017137378000000002, 2024-02-09T17:42:28.132 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:28.132 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-09T17:42:28.132 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:28.132 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:28.132 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:28.132 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:28.107862+0000", 2024-02-09T17:42:28.132 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:28.132 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:28.132 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:28.132 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:28.132 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:28.107862+0000", 2024-02-09T17:42:28.132 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:28.132 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:28.132 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:28.133 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:28.133 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:28.107863+0000", 2024-02-09T17:42:28.133 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9400000000000004e-07 2024-02-09T17:42:28.133 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:28.133 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:28.133 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:28.133 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:28.107886+0000", 2024-02-09T17:42:28.133 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3552000000000001e-05 2024-02-09T17:42:28.133 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:28.133 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:28.133 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:28.133 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:28.107898+0000", 2024-02-09T17:42:28.133 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1408e-05 2024-02-09T17:42:28.133 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:28.134 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:28.134 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:28.134 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:28.107905+0000", 2024-02-09T17:42:28.134 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0199999999999997e-06 2024-02-09T17:42:28.134 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:28.134 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:28.134 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:28.134 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:28.113920+0000", 2024-02-09T17:42:28.134 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0060148709999999998 2024-02-09T17:42:28.134 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:28.134 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:28.134 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:28.134 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:28.113942+0000", 2024-02-09T17:42:28.134 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2399e-05 2024-02-09T17:42:28.134 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:28.135 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:28.135 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:28.135 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:28.135 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:28.135 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25528.0:123490 5.9 e500/465 5:9f54157d:::benchmark_data_smithi179_115199_object123489:head v 500'7786, mlcod=500'7781)", 2024-02-09T17:42:28.135 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:42:28.112202+0000", 2024-02-09T17:42:28.135 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.012691040000000001, 2024-02-09T17:42:28.135 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012868618, 2024-02-09T17:42:28.135 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:28.135 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-09T17:42:28.135 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:28.135 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:28.135 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:28.135 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:28.112202+0000", 2024-02-09T17:42:28.135 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:28.135 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:28.136 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:28.136 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:28.136 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:28.112202+0000", 2024-02-09T17:42:28.136 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:28.136 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:28.136 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:28.136 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:28.136 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:28.112203+0000", 2024-02-09T17:42:28.136 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8400000000000003e-07 2024-02-09T17:42:28.136 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:28.136 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:28.136 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:28.136 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:28.112234+0000", 2024-02-09T17:42:28.136 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1013000000000003e-05 2024-02-09T17:42:28.137 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:28.137 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:28.137 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:28.137 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:28.112247+0000", 2024-02-09T17:42:28.137 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3091000000000001e-05 2024-02-09T17:42:28.137 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:28.137 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:28.137 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:28.137 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:28.112255+0000", 2024-02-09T17:42:28.137 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8779999999999998e-06 2024-02-09T17:42:28.137 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:28.137 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:28.137 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:28.137 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:28.117740+0000", 2024-02-09T17:42:28.137 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0054852119999999997 2024-02-09T17:42:28.137 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:28.138 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:28.138 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:28.138 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:28.117754+0000", 2024-02-09T17:42:28.138 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4253000000000001e-05 2024-02-09T17:42:28.138 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:28.138 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:28.138 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:28.138 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:28.138 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:28.138 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25528.0:123498 5.9 e500/465 5:9955590d:::benchmark_data_smithi179_115199_object123497:head v 500'7787, mlcod=500'7781)", 2024-02-09T17:42:28.138 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:42:28.121429+0000", 2024-02-09T17:42:28.138 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0034634430000000001, 2024-02-09T17:42:28.138 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0037073520000000001, 2024-02-09T17:42:28.138 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:28.138 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-09T17:42:28.139 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:28.139 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:28.139 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:28.139 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:28.121429+0000", 2024-02-09T17:42:28.139 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:28.139 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:28.139 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:28.139 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:28.139 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:28.121429+0000", 2024-02-09T17:42:28.139 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:28.139 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:28.139 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:28.139 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:28.139 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:28.121430+0000", 2024-02-09T17:42:28.139 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4e-07 2024-02-09T17:42:28.139 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:28.140 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:28.140 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:28.140 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:28.121454+0000", 2024-02-09T17:42:28.140 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3351999999999999e-05 2024-02-09T17:42:28.140 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:28.140 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:28.140 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:28.140 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:28.121471+0000", 2024-02-09T17:42:28.140 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7039999999999999e-05 2024-02-09T17:42:28.140 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:28.140 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:28.140 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:28.140 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:28.121479+0000", 2024-02-09T17:42:28.140 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6850000000000007e-06 2024-02-09T17:42:28.140 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:28.141 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:28.141 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:28.141 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:28.122022+0000", 2024-02-09T17:42:28.141 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00054261900000000004 2024-02-09T17:42:28.141 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:28.141 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:28.141 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:28.141 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:28.122041+0000", 2024-02-09T17:42:28.141 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8940000000000002e-05 2024-02-09T17:42:28.141 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:28.141 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:28.141 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:28.141 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:28.141 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-09T17:42:28.141 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 4 2024-02-09T17:42:28.142 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:42:28.179 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:42:28.212 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:27 smithi099 ceph-mon[39420]: pgmap v1271: 37 pgs: 37 active+clean; 7.3 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-02-09T17:42:28.213 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:28.426 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:27 smithi179 ceph-mon[43975]: pgmap v1271: 37 pgs: 37 active+clean; 7.3 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-02-09T17:42:28.570 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:42:28.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:28.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:42:28.674 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:42:28.707 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:28.899 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-09T17:42:28.934 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-09T17:42:28.963 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 84 16 124319 124303 92.4737 60.125 0.0163961 0.0108096 2024-02-09T17:42:29.158 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:42:29.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:29.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:42:29.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: 5.5 scrub ok 2024-02-09T17:42:29.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: 5.0 scrub starts 2024-02-09T17:42:29.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=3033316106,config=2679774077,health=1997216375,kv=2968647132}) 2024-02-09T17:42:29.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {kv=2,logm=98} crc {kv=3875779517,logm=3453059370}) 2024-02-09T17:42:29.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2863933594}) 2024-02-09T17:42:29.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=586301621}) 2024-02-09T17:42:29.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=200870694}) 2024-02-09T17:42:29.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=819928243}) 2024-02-09T17:42:29.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4073353696}) 2024-02-09T17:42:29.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1845165907}) 2024-02-09T17:42:29.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3167805455}) 2024-02-09T17:42:29.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=476646788}) 2024-02-09T17:42:29.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=519232390}) 2024-02-09T17:42:29.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=297388794}) 2024-02-09T17:42:29.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3836324796}) 2024-02-09T17:42:29.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1885180089}) 2024-02-09T17:42:29.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2339918444}) 2024-02-09T17:42:29.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3175166919}) 2024-02-09T17:42:29.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3264493094}) 2024-02-09T17:42:29.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1294190254}) 2024-02-09T17:42:29.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=545214313}) 2024-02-09T17:42:29.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=628237582}) 2024-02-09T17:42:29.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1984976354}) 2024-02-09T17:42:29.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2352746884}) 2024-02-09T17:42:29.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4088601730}) 2024-02-09T17:42:29.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4263762307}) 2024-02-09T17:42:29.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=581302038}) 2024-02-09T17:42:29.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619598327}) 2024-02-09T17:42:29.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=416928964}) 2024-02-09T17:42:29.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3687733313}) 2024-02-09T17:42:29.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4020578130}) 2024-02-09T17:42:29.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2515609529}) 2024-02-09T17:42:29.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1887696960}) 2024-02-09T17:42:29.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3091240892}) 2024-02-09T17:42:29.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4021140910}) 2024-02-09T17:42:29.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3504756644}) 2024-02-09T17:42:29.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1443545845}) 2024-02-09T17:42:29.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2668039629}) 2024-02-09T17:42:29.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2951948489}) 2024-02-09T17:42:29.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2836408576}) 2024-02-09T17:42:29.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=687334421}) 2024-02-09T17:42:29.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3711150489}) 2024-02-09T17:42:29.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2814992639}) 2024-02-09T17:42:29.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2798316833}) 2024-02-09T17:42:29.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=648492125}) 2024-02-09T17:42:29.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3994701922}) 2024-02-09T17:42:29.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1971360991}) 2024-02-09T17:42:29.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3956854456}) 2024-02-09T17:42:29.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=773076517}) 2024-02-09T17:42:29.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=3,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=59} crc {logm=3601257412,mdsmap=3984425268,mgr=1708625883,mgr_command_descs=708342470,mgr_metadata=670763949,mgrstat=3339151042,mon_config_key=329113217}) 2024-02-09T17:42:29.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2117787269}) 2024-02-09T17:42:29.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=200357539}) 2024-02-09T17:42:29.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=22,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=60} crc {mon_config_key=738307068,monmap=423865542,osd_metadata=545325961,osd_pg_creating=1244709281,osdmap=450398693}) 2024-02-09T17:42:29.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=49} crc {osdmap=2147024994}) 2024-02-09T17:42:29.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: 5.5 scrub ok 2024-02-09T17:42:29.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: 5.0 scrub starts 2024-02-09T17:42:29.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=3033316106,config=2679774077,health=1997216375,kv=2968647132}) 2024-02-09T17:42:29.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {kv=2,logm=98} crc {kv=3875779517,logm=3453059370}) 2024-02-09T17:42:29.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2863933594}) 2024-02-09T17:42:29.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=586301621}) 2024-02-09T17:42:29.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=200870694}) 2024-02-09T17:42:29.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=819928243}) 2024-02-09T17:42:29.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4073353696}) 2024-02-09T17:42:29.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1845165907}) 2024-02-09T17:42:29.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3167805455}) 2024-02-09T17:42:29.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=476646788}) 2024-02-09T17:42:29.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=519232390}) 2024-02-09T17:42:29.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=297388794}) 2024-02-09T17:42:29.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3836324796}) 2024-02-09T17:42:29.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1885180089}) 2024-02-09T17:42:29.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2339918444}) 2024-02-09T17:42:29.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3175166919}) 2024-02-09T17:42:29.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3264493094}) 2024-02-09T17:42:29.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1294190254}) 2024-02-09T17:42:29.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=545214313}) 2024-02-09T17:42:29.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=628237582}) 2024-02-09T17:42:29.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1984976354}) 2024-02-09T17:42:29.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2352746884}) 2024-02-09T17:42:29.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4088601730}) 2024-02-09T17:42:29.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4263762307}) 2024-02-09T17:42:29.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=581302038}) 2024-02-09T17:42:29.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619598327}) 2024-02-09T17:42:29.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=416928964}) 2024-02-09T17:42:29.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3687733313}) 2024-02-09T17:42:29.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4020578130}) 2024-02-09T17:42:29.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2515609529}) 2024-02-09T17:42:29.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1887696960}) 2024-02-09T17:42:29.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3091240892}) 2024-02-09T17:42:29.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4021140910}) 2024-02-09T17:42:29.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3504756644}) 2024-02-09T17:42:29.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1443545845}) 2024-02-09T17:42:29.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2668039629}) 2024-02-09T17:42:29.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2951948489}) 2024-02-09T17:42:29.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2836408576}) 2024-02-09T17:42:29.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=687334421}) 2024-02-09T17:42:29.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3711150489}) 2024-02-09T17:42:29.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2814992639}) 2024-02-09T17:42:29.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2798316833}) 2024-02-09T17:42:29.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=648492125}) 2024-02-09T17:42:29.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3994701922}) 2024-02-09T17:42:29.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1971360991}) 2024-02-09T17:42:29.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3956854456}) 2024-02-09T17:42:29.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=773076517}) 2024-02-09T17:42:29.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=3,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=59} crc {logm=3601257412,mdsmap=3984425268,mgr=1708625883,mgr_command_descs=708342470,mgr_metadata=670763949,mgrstat=3339151042,mon_config_key=329113217}) 2024-02-09T17:42:29.248 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2117787269}) 2024-02-09T17:42:29.248 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=200357539}) 2024-02-09T17:42:29.248 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=22,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=60} crc {mon_config_key=738307068,monmap=423865542,osd_metadata=545325961,osd_pg_creating=1244709281,osdmap=450398693}) 2024-02-09T17:42:29.248 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=49} crc {osdmap=2147024994}) 2024-02-09T17:42:29.264 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:42:29.324 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:42:29.390 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:29.392 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: 5.5 scrub ok 2024-02-09T17:42:29.392 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: 5.0 scrub starts 2024-02-09T17:42:29.392 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=3033316106,config=2679774077,health=1997216375,kv=2968647132}) 2024-02-09T17:42:29.392 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {kv=2,logm=98} crc {kv=3875779517,logm=3453059370}) 2024-02-09T17:42:29.392 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2863933594}) 2024-02-09T17:42:29.392 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=586301621}) 2024-02-09T17:42:29.392 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=200870694}) 2024-02-09T17:42:29.393 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=819928243}) 2024-02-09T17:42:29.393 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4073353696}) 2024-02-09T17:42:29.393 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1845165907}) 2024-02-09T17:42:29.393 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3167805455}) 2024-02-09T17:42:29.393 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=476646788}) 2024-02-09T17:42:29.393 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=519232390}) 2024-02-09T17:42:29.393 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=297388794}) 2024-02-09T17:42:29.393 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3836324796}) 2024-02-09T17:42:29.393 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1885180089}) 2024-02-09T17:42:29.393 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2339918444}) 2024-02-09T17:42:29.393 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3175166919}) 2024-02-09T17:42:29.393 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3264493094}) 2024-02-09T17:42:29.393 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1294190254}) 2024-02-09T17:42:29.393 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=545214313}) 2024-02-09T17:42:29.394 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=628237582}) 2024-02-09T17:42:29.394 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1984976354}) 2024-02-09T17:42:29.394 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2352746884}) 2024-02-09T17:42:29.394 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4088601730}) 2024-02-09T17:42:29.394 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4263762307}) 2024-02-09T17:42:29.394 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=581302038}) 2024-02-09T17:42:29.394 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1619598327}) 2024-02-09T17:42:29.394 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=416928964}) 2024-02-09T17:42:29.394 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3687733313}) 2024-02-09T17:42:29.394 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4020578130}) 2024-02-09T17:42:29.394 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2515609529}) 2024-02-09T17:42:29.394 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1887696960}) 2024-02-09T17:42:29.394 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3091240892}) 2024-02-09T17:42:29.394 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4021140910}) 2024-02-09T17:42:29.395 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3504756644}) 2024-02-09T17:42:29.395 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1443545845}) 2024-02-09T17:42:29.395 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2668039629}) 2024-02-09T17:42:29.395 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2951948489}) 2024-02-09T17:42:29.395 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2836408576}) 2024-02-09T17:42:29.395 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=687334421}) 2024-02-09T17:42:29.395 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3711150489}) 2024-02-09T17:42:29.395 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2814992639}) 2024-02-09T17:42:29.395 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2798316833}) 2024-02-09T17:42:29.395 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=648492125}) 2024-02-09T17:42:29.396 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3994701922}) 2024-02-09T17:42:29.396 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1971360991}) 2024-02-09T17:42:29.396 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3956854456}) 2024-02-09T17:42:29.396 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=773076517}) 2024-02-09T17:42:29.396 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=3,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=59} crc {logm=3601257412,mdsmap=3984425268,mgr=1708625883,mgr_command_descs=708342470,mgr_metadata=670763949,mgrstat=3339151042,mon_config_key=329113217}) 2024-02-09T17:42:29.396 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2117787269}) 2024-02-09T17:42:29.396 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=200357539}) 2024-02-09T17:42:29.396 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=22,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=60} crc {mon_config_key=738307068,monmap=423865542,osd_metadata=545325961,osd_pg_creating=1244709281,osdmap=450398693}) 2024-02-09T17:42:29.396 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=49} crc {osdmap=2147024994}) 2024-02-09T17:42:29.450 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:42:29.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:29.462 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:42:29.563 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:42:29.596 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:29.647 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:42:29.725 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:42:29.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:29.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:42:29.763 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:42:29.830 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:42:29.877 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:29.912 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:42:29.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:29.922 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:42:29.963 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 85 16 125136 125120 91.9864 51.0625 0.0138812 0.0108659 2024-02-09T17:42:29.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:29 smithi099 ceph-mon[39420]: pgmap v1272: 37 pgs: 37 active+clean; 7.4 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-02-09T17:42:29.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:29 smithi099 ceph-mon[39420]: 5.b scrub starts 2024-02-09T17:42:30.023 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:42:30.060 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:30.089 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:42:30.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:30.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:42:30.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:29 smithi162 ceph-mon[44304]: pgmap v1272: 37 pgs: 37 active+clean; 7.4 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-02-09T17:42:30.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:29 smithi162 ceph-mon[44304]: 5.b scrub starts 2024-02-09T17:42:30.183 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:29 smithi179 ceph-mon[43975]: pgmap v1272: 37 pgs: 37 active+clean; 7.4 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-02-09T17:42:30.183 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:29 smithi179 ceph-mon[43975]: 5.b scrub starts 2024-02-09T17:42:30.197 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:42:30.228 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:30.267 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:42:30.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:30.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:42:30.372 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:42:30.428 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:30.644 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:42:30.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:30.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:42:30.659 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:42:30.753 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:42:30.869 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:30.889 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:42:30.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:30.892 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:42:30.964 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 86 16 125744 125728 91.3587 38 0.0115574 0.010941 2024-02-09T17:42:30.969 INFO:teuthology.orchestra.run.smithi099.stderr:set pool 1 pgp_num to 21 2024-02-09T17:42:30.993 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:42:31.064 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:31.153 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:42:31.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:31.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:42:31.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:30 smithi179 ceph-mon[43975]: 1.f scrub starts 2024-02-09T17:42:31.175 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:30 smithi179 ceph-mon[43975]: 1.f scrub ok 2024-02-09T17:42:31.175 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:30 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4274582192' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-02-09T17:42:31.175 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:30 smithi179 ceph-mon[43975]: 1.1 scrub starts 2024-02-09T17:42:31.175 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:30 smithi179 ceph-mon[43975]: 1.1 scrub ok 2024-02-09T17:42:31.175 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:30 smithi179 ceph-mon[43975]: 5.0 scrub ok 2024-02-09T17:42:31.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:30 smithi099 ceph-mon[39420]: 1.f scrub starts 2024-02-09T17:42:31.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:30 smithi099 ceph-mon[39420]: 1.f scrub ok 2024-02-09T17:42:31.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:30 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4274582192' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-02-09T17:42:31.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:30 smithi099 ceph-mon[39420]: 1.1 scrub starts 2024-02-09T17:42:31.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:30 smithi099 ceph-mon[39420]: 1.1 scrub ok 2024-02-09T17:42:31.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:30 smithi099 ceph-mon[39420]: 5.0 scrub ok 2024-02-09T17:42:31.258 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:42:31.290 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:31.355 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:42:31.355 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-09T17:42:31.356 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-02-09T17:42:31.356 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-02-09T17:42:31.356 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:42:31.382 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:30 smithi162 ceph-mon[44304]: 1.f scrub starts 2024-02-09T17:42:31.382 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:30 smithi162 ceph-mon[44304]: 1.f scrub ok 2024-02-09T17:42:31.383 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:30 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4274582192' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-02-09T17:42:31.383 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:30 smithi162 ceph-mon[44304]: 1.1 scrub starts 2024-02-09T17:42:31.383 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:30 smithi162 ceph-mon[44304]: 1.1 scrub ok 2024-02-09T17:42:31.383 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:30 smithi162 ceph-mon[44304]: 5.0 scrub ok 2024-02-09T17:42:31.964 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 87 15 126510 126495 90.8596 47.9375 0.0146529 0.011002 2024-02-09T17:42:31.966 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:31 smithi099 ceph-mon[39420]: 5.6 scrub starts 2024-02-09T17:42:31.966 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:31 smithi099 ceph-mon[39420]: 5.b scrub ok 2024-02-09T17:42:31.966 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:31 smithi099 ceph-mon[39420]: pgmap v1273: 37 pgs: 37 active+clean; 7.5 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 62 MiB/s wr, 994 op/s 2024-02-09T17:42:31.966 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:31 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4274582192' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-02-09T17:42:31.966 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:31 smithi099 ceph-mon[39420]: osdmap e501: 12 total, 12 up, 11 in 2024-02-09T17:42:32.099 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:31 smithi162 ceph-mon[44304]: 5.6 scrub starts 2024-02-09T17:42:32.099 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:31 smithi162 ceph-mon[44304]: 5.b scrub ok 2024-02-09T17:42:32.100 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:31 smithi162 ceph-mon[44304]: pgmap v1273: 37 pgs: 37 active+clean; 7.5 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 62 MiB/s wr, 994 op/s 2024-02-09T17:42:32.100 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:31 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4274582192' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-02-09T17:42:32.100 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:31 smithi162 ceph-mon[44304]: osdmap e501: 12 total, 12 up, 11 in 2024-02-09T17:42:32.129 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-09T17:42:32.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:31 smithi179 ceph-mon[43975]: 5.6 scrub starts 2024-02-09T17:42:32.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:31 smithi179 ceph-mon[43975]: 5.b scrub ok 2024-02-09T17:42:32.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:31 smithi179 ceph-mon[43975]: pgmap v1273: 37 pgs: 37 active+clean; 7.5 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 62 MiB/s wr, 994 op/s 2024-02-09T17:42:32.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:31 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4274582192' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-02-09T17:42:32.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:31 smithi179 ceph-mon[43975]: osdmap e501: 12 total, 12 up, 11 in 2024-02-09T17:42:32.556 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:42:32.964 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 88 16 127362 127346 90.4315 53.1875 0.0213029 0.0110547 2024-02-09T17:42:33.115 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:32 smithi162 ceph-mon[44304]: 1.4 scrub starts 2024-02-09T17:42:33.115 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:32 smithi162 ceph-mon[44304]: 1.4 scrub ok 2024-02-09T17:42:33.115 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:32 smithi162 ceph-mon[44304]: 1.d scrub starts 2024-02-09T17:42:33.115 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:32 smithi162 ceph-mon[44304]: 1.12 scrub starts 2024-02-09T17:42:33.115 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:32 smithi162 ceph-mon[44304]: 1.12 scrub ok 2024-02-09T17:42:33.115 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:32 smithi162 ceph-mon[44304]: 1.d scrub ok 2024-02-09T17:42:33.115 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:32 smithi162 ceph-mon[44304]: 5.a scrub starts 2024-02-09T17:42:33.115 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:32 smithi162 ceph-mon[44304]: 5.6 scrub ok 2024-02-09T17:42:33.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:32 smithi099 ceph-mon[39420]: 1.4 scrub starts 2024-02-09T17:42:33.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:32 smithi099 ceph-mon[39420]: 1.4 scrub ok 2024-02-09T17:42:33.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:32 smithi099 ceph-mon[39420]: 1.d scrub starts 2024-02-09T17:42:33.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:32 smithi099 ceph-mon[39420]: 1.12 scrub starts 2024-02-09T17:42:33.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:32 smithi099 ceph-mon[39420]: 1.12 scrub ok 2024-02-09T17:42:33.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:32 smithi099 ceph-mon[39420]: 1.d scrub ok 2024-02-09T17:42:33.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:32 smithi099 ceph-mon[39420]: 5.a scrub starts 2024-02-09T17:42:33.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:32 smithi099 ceph-mon[39420]: 5.6 scrub ok 2024-02-09T17:42:33.489 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:32 smithi179 ceph-mon[43975]: 1.4 scrub starts 2024-02-09T17:42:33.489 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:32 smithi179 ceph-mon[43975]: 1.4 scrub ok 2024-02-09T17:42:33.490 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:32 smithi179 ceph-mon[43975]: 1.d scrub starts 2024-02-09T17:42:33.490 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:32 smithi179 ceph-mon[43975]: 1.12 scrub starts 2024-02-09T17:42:33.490 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:32 smithi179 ceph-mon[43975]: 1.12 scrub ok 2024-02-09T17:42:33.490 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:32 smithi179 ceph-mon[43975]: 1.d scrub ok 2024-02-09T17:42:33.490 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:32 smithi179 ceph-mon[43975]: 5.a scrub starts 2024-02-09T17:42:33.490 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:32 smithi179 ceph-mon[43975]: 5.6 scrub ok 2024-02-09T17:42:33.746 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:42:33.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:33.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:42:33.850 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:42:33.896 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:33.950 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:42:33.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:33.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:42:33.974 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 89 16 128092 128076 89.928 45.625 0.0385096 0.0111154 2024-02-09T17:42:34.006 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is unset 2024-02-09T17:42:34.054 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:42:34.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:33 smithi162 ceph-mon[44304]: 5.8 scrub starts 2024-02-09T17:42:34.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:33 smithi162 ceph-mon[44304]: pgmap v1275: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 7.6 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 59 MiB/s wr, 936 op/s 2024-02-09T17:42:34.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:33 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/933913829' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:42:34.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:33 smithi162 ceph-mon[44304]: 5.a scrub ok 2024-02-09T17:42:34.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:33 smithi162 ceph-mon[44304]: 1.11 scrub starts 2024-02-09T17:42:34.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:33 smithi162 ceph-mon[44304]: 5.4 scrub starts 2024-02-09T17:42:34.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:33 smithi162 ceph-mon[44304]: 1.11 scrub ok 2024-02-09T17:42:34.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:33 smithi099 ceph-mon[39420]: 5.8 scrub starts 2024-02-09T17:42:34.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:33 smithi099 ceph-mon[39420]: pgmap v1275: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 7.6 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 59 MiB/s wr, 936 op/s 2024-02-09T17:42:34.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:33 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/933913829' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:42:34.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:33 smithi099 ceph-mon[39420]: 5.a scrub ok 2024-02-09T17:42:34.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:33 smithi099 ceph-mon[39420]: 1.11 scrub starts 2024-02-09T17:42:34.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:33 smithi099 ceph-mon[39420]: 5.4 scrub starts 2024-02-09T17:42:34.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:33 smithi099 ceph-mon[39420]: 1.11 scrub ok 2024-02-09T17:42:34.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:33 smithi179 ceph-mon[43975]: 5.8 scrub starts 2024-02-09T17:42:34.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:33 smithi179 ceph-mon[43975]: pgmap v1275: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 7.6 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 59 MiB/s wr, 936 op/s 2024-02-09T17:42:34.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:33 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/933913829' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:42:34.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:33 smithi179 ceph-mon[43975]: 5.a scrub ok 2024-02-09T17:42:34.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:33 smithi179 ceph-mon[43975]: 1.11 scrub starts 2024-02-09T17:42:34.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:33 smithi179 ceph-mon[43975]: 5.4 scrub starts 2024-02-09T17:42:34.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:33 smithi179 ceph-mon[43975]: 1.11 scrub ok 2024-02-09T17:42:34.547 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:42:34.569 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:42:34.595 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:42:34.617 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:42:34.652 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:42:34.686 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:42:34.720 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:42:34.720 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-02-09T17:42:34.720 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-02-09T17:42:34.720 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-02-09T17:42:34.720 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.720 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25201.0:80430 4.a 4:51c835cc:::benchmark_data_smithi179_103809_object80429:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-02-09T17:42:34.721 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:36:17.452429+0000", 2024-02-09T17:42:34.721 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 377.25877205299997, 2024-02-09T17:42:34.721 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21455295199999999, 2024-02-09T17:42:34.721 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:34.721 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:42:34.721 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:34.721 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25201", 2024-02-09T17:42:34.721 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:42:34.721 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 80430 2024-02-09T17:42:34.721 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.721 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:34.721 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.721 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:34.721 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.452429+0000", 2024-02-09T17:42:34.721 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:34.722 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:34.722 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.452429+0000", 2024-02-09T17:42:34.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:34.722 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:34.722 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.452430+0000", 2024-02-09T17:42:34.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.142e-06 2024-02-09T17:42:34.722 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:34.722 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.452487+0000", 2024-02-09T17:42:34.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7500000000000002e-05 2024-02-09T17:42:34.723 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.723 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.723 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:34.723 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.452496+0000", 2024-02-09T17:42:34.723 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7080000000000003e-06 2024-02-09T17:42:34.723 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.723 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.723 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:34.723 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.452501+0000", 2024-02-09T17:42:34.723 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8609999999999997e-06 2024-02-09T17:42:34.723 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.723 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.723 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:34.723 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.452549+0000", 2024-02-09T17:42:34.723 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8192999999999998e-05 2024-02-09T17:42:34.724 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.724 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.724 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:34.724 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.452690+0000", 2024-02-09T17:42:34.724 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000140638 2024-02-09T17:42:34.724 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.724 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.724 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,10", 2024-02-09T17:42:34.724 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.452845+0000", 2024-02-09T17:42:34.724 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015569100000000001 2024-02-09T17:42:34.724 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.724 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.724 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:34.724 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.457687+0000", 2024-02-09T17:42:34.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0048420110000000002 2024-02-09T17:42:34.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:34.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.457720+0000", 2024-02-09T17:42:34.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3232000000000003e-05 2024-02-09T17:42:34.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:34.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.666915+0000", 2024-02-09T17:42:34.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.209194355 2024-02-09T17:42:34.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:42:34.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.666954+0000", 2024-02-09T17:42:34.726 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9264000000000002e-05 2024-02-09T17:42:34.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.726 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.726 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:34.726 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.666981+0000", 2024-02-09T17:42:34.726 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7356999999999999e-05 2024-02-09T17:42:34.726 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:34.726 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:34.726 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:34.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.726 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.726 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25201.0:80431 4.a 4:515ff9b3:::benchmark_data_smithi179_103809_object80430:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-02-09T17:42:34.726 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:36:17.452513+0000", 2024-02-09T17:42:34.726 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 377.25868770099999, 2024-02-09T17:42:34.726 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21460262499999999, 2024-02-09T17:42:34.727 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:34.727 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:42:34.727 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:34.727 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25201", 2024-02-09T17:42:34.727 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:42:34.727 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 80431 2024-02-09T17:42:34.727 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.727 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:34.727 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.727 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:34.727 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.452513+0000", 2024-02-09T17:42:34.727 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:34.727 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.727 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.727 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:34.728 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.452513+0000", 2024-02-09T17:42:34.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:34.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.728 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.728 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:34.728 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.452513+0000", 2024-02-09T17:42:34.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1900000000000003e-07 2024-02-09T17:42:34.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.728 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.728 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:34.728 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.452559+0000", 2024-02-09T17:42:34.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5762e-05 2024-02-09T17:42:34.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.728 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.728 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:34.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.452564+0000", 2024-02-09T17:42:34.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0780000000000003e-06 2024-02-09T17:42:34.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:34.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.452568+0000", 2024-02-09T17:42:34.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8759999999999998e-06 2024-02-09T17:42:34.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:34.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.454922+0000", 2024-02-09T17:42:34.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023536030000000001 2024-02-09T17:42:34.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:34.730 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.455059+0000", 2024-02-09T17:42:34.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013749500000000001 2024-02-09T17:42:34.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.730 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,10", 2024-02-09T17:42:34.730 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.455208+0000", 2024-02-09T17:42:34.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014833500000000001 2024-02-09T17:42:34.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.730 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:34.730 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.459649+0000", 2024-02-09T17:42:34.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0044419109999999998 2024-02-09T17:42:34.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.730 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:34.730 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.459982+0000", 2024-02-09T17:42:34.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00033270400000000002 2024-02-09T17:42:34.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:34.731 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.667026+0000", 2024-02-09T17:42:34.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20704413499999999 2024-02-09T17:42:34.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:42:34.731 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.667080+0000", 2024-02-09T17:42:34.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3866999999999998e-05 2024-02-09T17:42:34.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:34.731 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.667115+0000", 2024-02-09T17:42:34.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5339999999999997e-05 2024-02-09T17:42:34.732 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:34.732 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:34.732 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:34.732 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.732 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25201.0:80432 4.a 4:527ee64a:::benchmark_data_smithi179_103809_object80431:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-02-09T17:42:34.732 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:36:17.452576+0000", 2024-02-09T17:42:34.732 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 377.25862467500002, 2024-02-09T17:42:34.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.214621268, 2024-02-09T17:42:34.732 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:34.732 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:42:34.732 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:34.732 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25201", 2024-02-09T17:42:34.732 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:42:34.733 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 80432 2024-02-09T17:42:34.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.733 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:34.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:34.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.452576+0000", 2024-02-09T17:42:34.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:34.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:34.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.452576+0000", 2024-02-09T17:42:34.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:34.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:34.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.452576+0000", 2024-02-09T17:42:34.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5200000000000002e-07 2024-02-09T17:42:34.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:34.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.452625+0000", 2024-02-09T17:42:34.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8380000000000001e-05 2024-02-09T17:42:34.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:34.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.452635+0000", 2024-02-09T17:42:34.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0566e-05 2024-02-09T17:42:34.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:34.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.452640+0000", 2024-02-09T17:42:34.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7169999999999999e-06 2024-02-09T17:42:34.735 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.735 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.735 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:34.735 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.457738+0000", 2024-02-09T17:42:34.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0050982140000000002 2024-02-09T17:42:34.735 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.735 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.735 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:34.735 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.457816+0000", 2024-02-09T17:42:34.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7704e-05 2024-02-09T17:42:34.735 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.735 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.735 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,10", 2024-02-09T17:42:34.735 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.457904+0000", 2024-02-09T17:42:34.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7954000000000005e-05 2024-02-09T17:42:34.736 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.736 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.736 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:34.736 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.459796+0000", 2024-02-09T17:42:34.736 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001891773 2024-02-09T17:42:34.736 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.736 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.736 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:34.736 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.462483+0000", 2024-02-09T17:42:34.736 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0026875789999999998 2024-02-09T17:42:34.736 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.736 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.736 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:34.736 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.667142+0000", 2024-02-09T17:42:34.736 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.204659177 2024-02-09T17:42:34.737 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.737 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.737 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:42:34.737 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.667175+0000", 2024-02-09T17:42:34.737 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3052999999999997e-05 2024-02-09T17:42:34.737 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.737 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.737 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:34.737 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.667197+0000", 2024-02-09T17:42:34.737 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1699e-05 2024-02-09T17:42:34.737 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:34.737 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:34.737 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:34.737 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.737 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.737 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25201.0:80451 4.a 4:5f65eccf:::benchmark_data_smithi179_103809_object80450:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-02-09T17:42:34.738 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:36:17.465120+0000", 2024-02-09T17:42:34.738 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 377.24608087500002, 2024-02-09T17:42:34.738 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.208945241, 2024-02-09T17:42:34.738 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:34.738 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:42:34.738 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:34.738 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25201", 2024-02-09T17:42:34.738 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:42:34.738 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 80451 2024-02-09T17:42:34.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.738 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:34.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:34.738 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.465120+0000", 2024-02-09T17:42:34.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:34.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.739 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:34.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.465120+0000", 2024-02-09T17:42:34.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:34.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.739 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:34.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.465121+0000", 2024-02-09T17:42:34.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3850000000000001e-06 2024-02-09T17:42:34.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.739 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:34.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.465173+0000", 2024-02-09T17:42:34.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1888000000000002e-05 2024-02-09T17:42:34.740 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.740 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.740 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:34.740 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.465187+0000", 2024-02-09T17:42:34.740 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3837e-05 2024-02-09T17:42:34.740 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.740 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.740 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:34.740 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.465195+0000", 2024-02-09T17:42:34.740 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8040000000000002e-06 2024-02-09T17:42:34.740 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.740 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.740 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:34.740 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.465256+0000", 2024-02-09T17:42:34.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0899000000000001e-05 2024-02-09T17:42:34.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:34.741 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.465347+0000", 2024-02-09T17:42:34.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1416999999999996e-05 2024-02-09T17:42:34.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,10", 2024-02-09T17:42:34.741 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.465440+0000", 2024-02-09T17:42:34.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2676999999999994e-05 2024-02-09T17:42:34.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:34.741 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.467656+0000", 2024-02-09T17:42:34.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0022164709999999998 2024-02-09T17:42:34.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:34.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.470003+0000", 2024-02-09T17:42:34.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002346814 2024-02-09T17:42:34.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:34.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.673967+0000", 2024-02-09T17:42:34.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.203964597 2024-02-09T17:42:34.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:42:34.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.674026+0000", 2024-02-09T17:42:34.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8105e-05 2024-02-09T17:42:34.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.743 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:34.743 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:36:17.674065+0000", 2024-02-09T17:42:34.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9347000000000001e-05 2024-02-09T17:42:34.743 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:34.743 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:34.743 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:34.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.743 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25201.0:208410 4.9 4.c6999869 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-02-09T17:42:34.743 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:38:33.821409+0000", 2024-02-09T17:42:34.743 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 240.889791368, 2024-02-09T17:42:34.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99082882000000005, 2024-02-09T17:42:34.744 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:34.744 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-09T17:42:34.744 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:34.744 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25201", 2024-02-09T17:42:34.744 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:42:34.744 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 208410 2024-02-09T17:42:34.744 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.744 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:34.744 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.744 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:34.744 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:33.821409+0000", 2024-02-09T17:42:34.744 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:34.744 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.744 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.744 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:34.745 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:33.821409+0000", 2024-02-09T17:42:34.745 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:34.745 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.745 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.745 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:34.745 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:33.821410+0000", 2024-02-09T17:42:34.745 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1209999999999999e-06 2024-02-09T17:42:34.745 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.745 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.745 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:34.745 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:33.821413+0000", 2024-02-09T17:42:34.745 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7080000000000002e-06 2024-02-09T17:42:34.745 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.745 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.746 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:34.746 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:33.821422+0000", 2024-02-09T17:42:34.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4829999999999999e-06 2024-02-09T17:42:34.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.746 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:34.746 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:33.821428+0000", 2024-02-09T17:42:34.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6220000000000003e-06 2024-02-09T17:42:34.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.746 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:34.746 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:33.824432+0000", 2024-02-09T17:42:34.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0030041019999999998 2024-02-09T17:42:34.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.747 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:42:34.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:33.824440+0000", 2024-02-09T17:42:34.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1519999999999992e-06 2024-02-09T17:42:34.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.747 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:34.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:34.812221+0000", 2024-02-09T17:42:34.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.98778070399999995 2024-02-09T17:42:34.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.747 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:34.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:34.812238+0000", 2024-02-09T17:42:34.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6928000000000001e-05 2024-02-09T17:42:34.747 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:34.748 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:34.748 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:34.748 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.748 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.748 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25201.0:208415 4.9 4.776553b9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-02-09T17:42:34.748 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:38:33.830876+0000", 2024-02-09T17:42:34.748 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 240.880324992, 2024-02-09T17:42:34.748 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.981399252, 2024-02-09T17:42:34.748 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:34.748 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-09T17:42:34.748 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:34.748 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25201", 2024-02-09T17:42:34.748 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:42:34.748 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 208415 2024-02-09T17:42:34.748 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.749 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:34.749 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.749 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:34.749 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:33.830876+0000", 2024-02-09T17:42:34.749 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:34.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.749 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.749 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:34.749 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:33.830876+0000", 2024-02-09T17:42:34.749 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:34.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.749 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.749 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:34.749 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:33.830877+0000", 2024-02-09T17:42:34.749 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6819999999999999e-06 2024-02-09T17:42:34.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.750 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:34.750 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:33.830882+0000", 2024-02-09T17:42:34.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0499999999999999e-06 2024-02-09T17:42:34.750 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.750 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:34.750 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:33.830894+0000", 2024-02-09T17:42:34.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1888e-05 2024-02-09T17:42:34.750 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.750 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:34.750 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:33.830903+0000", 2024-02-09T17:42:34.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2059999999999996e-06 2024-02-09T17:42:34.750 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:34.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:33.830954+0000", 2024-02-09T17:42:34.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0838999999999999e-05 2024-02-09T17:42:34.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:42:34.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:33.830961+0000", 2024-02-09T17:42:34.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1500000000000002e-06 2024-02-09T17:42:34.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:34.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:34.812267+0000", 2024-02-09T17:42:34.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.98130531700000001 2024-02-09T17:42:34.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.752 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:34.752 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:34.812275+0000", 2024-02-09T17:42:34.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1200000000000002e-06 2024-02-09T17:42:34.752 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:34.752 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:34.752 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:34.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.752 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25201.0:208419 4.9 4.9271d529 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-02-09T17:42:34.752 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:38:33.836354+0000", 2024-02-09T17:42:34.752 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 240.874846797, 2024-02-09T17:42:34.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.97600734200000006, 2024-02-09T17:42:34.752 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:34.752 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-09T17:42:34.753 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:34.753 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25201", 2024-02-09T17:42:34.753 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:42:34.753 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 208419 2024-02-09T17:42:34.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.753 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:34.753 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:34.753 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:33.836354+0000", 2024-02-09T17:42:34.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:34.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.753 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:34.753 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:33.836354+0000", 2024-02-09T17:42:34.754 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:34.754 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.754 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.754 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:34.754 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:33.836355+0000", 2024-02-09T17:42:34.754 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7179999999999999e-06 2024-02-09T17:42:34.754 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.754 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.754 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:34.754 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:33.836362+0000", 2024-02-09T17:42:34.754 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2079999999999997e-06 2024-02-09T17:42:34.754 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.754 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.754 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:34.755 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:33.836375+0000", 2024-02-09T17:42:34.755 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3532e-05 2024-02-09T17:42:34.755 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.755 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.755 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:34.755 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:33.836382+0000", 2024-02-09T17:42:34.755 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.148e-06 2024-02-09T17:42:34.755 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.755 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.755 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:34.755 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:33.836432+0000", 2024-02-09T17:42:34.755 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9679000000000003e-05 2024-02-09T17:42:34.755 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.755 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.755 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:42:34.755 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:33.836438+0000", 2024-02-09T17:42:34.756 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1020000000000002e-06 2024-02-09T17:42:34.756 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.756 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.756 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:34.756 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:34.812351+0000", 2024-02-09T17:42:34.756 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.97591331699999995 2024-02-09T17:42:34.756 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.756 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.756 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:34.756 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:34.812361+0000", 2024-02-09T17:42:34.756 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6379999999999999e-06 2024-02-09T17:42:34.756 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:34.756 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:34.756 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:34.756 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.757 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:22592 5.3 5:c8c232c5:::benchmark_data_smithi179_115199_object22591:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e477)", 2024-02-09T17:42:34.757 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:41:14.357863+0000", 2024-02-09T17:42:34.757 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 80.353337742999997, 2024-02-09T17:42:34.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.204502774, 2024-02-09T17:42:34.757 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:34.757 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:42:34.757 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:34.757 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:42:34.757 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:42:34.757 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 22592 2024-02-09T17:42:34.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.757 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:34.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:34.757 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.357863+0000", 2024-02-09T17:42:34.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:34.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.758 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:34.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.357863+0000", 2024-02-09T17:42:34.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:34.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.758 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:34.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.357863+0000", 2024-02-09T17:42:34.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3900000000000005e-07 2024-02-09T17:42:34.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.758 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:34.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.357905+0000", 2024-02-09T17:42:34.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2095000000000003e-05 2024-02-09T17:42:34.759 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.759 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:34.759 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.357916+0000", 2024-02-09T17:42:34.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0869999999999999e-05 2024-02-09T17:42:34.759 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.759 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:34.759 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.357922+0000", 2024-02-09T17:42:34.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1210000000000003e-06 2024-02-09T17:42:34.759 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.759 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:34.759 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.359087+0000", 2024-02-09T17:42:34.760 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001164469 2024-02-09T17:42:34.760 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.760 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.760 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:34.760 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.359220+0000", 2024-02-09T17:42:34.760 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013342700000000001 2024-02-09T17:42:34.760 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.760 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.760 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,5", 2024-02-09T17:42:34.760 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.359370+0000", 2024-02-09T17:42:34.760 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000149345 2024-02-09T17:42:34.760 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.760 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.760 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:34.760 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.363501+0000", 2024-02-09T17:42:34.760 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041311500000000001 2024-02-09T17:42:34.761 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.761 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.761 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:34.761 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.363538+0000", 2024-02-09T17:42:34.761 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6695e-05 2024-02-09T17:42:34.761 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.761 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.761 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:34.761 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.562265+0000", 2024-02-09T17:42:34.761 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.198727449 2024-02-09T17:42:34.761 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.761 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.761 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:42:34.761 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.562325+0000", 2024-02-09T17:42:34.761 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9867999999999997e-05 2024-02-09T17:42:34.762 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.762 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.762 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:34.762 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.562366+0000", 2024-02-09T17:42:34.762 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0745999999999999e-05 2024-02-09T17:42:34.762 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:34.762 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:34.762 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:34.762 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.762 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.762 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:22597 5.3 5:c17e0158:::benchmark_data_smithi179_115199_object22596:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e477)", 2024-02-09T17:42:34.762 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:41:14.358366+0000", 2024-02-09T17:42:34.762 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 80.352834176000002, 2024-02-09T17:42:34.762 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20612446700000001, 2024-02-09T17:42:34.762 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:34.763 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:42:34.763 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:34.763 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:42:34.763 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:42:34.763 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 22597 2024-02-09T17:42:34.763 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.763 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:34.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.763 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:34.763 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.358366+0000", 2024-02-09T17:42:34.763 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:34.763 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.763 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:34.763 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.358366+0000", 2024-02-09T17:42:34.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:34.764 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.764 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.764 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:34.764 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.358367+0000", 2024-02-09T17:42:34.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0529999999999999e-06 2024-02-09T17:42:34.764 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.764 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.764 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:34.764 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.358428+0000", 2024-02-09T17:42:34.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0590000000000001e-05 2024-02-09T17:42:34.764 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.764 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.764 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:34.764 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.358440+0000", 2024-02-09T17:42:34.765 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2038999999999999e-05 2024-02-09T17:42:34.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:34.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.358449+0000", 2024-02-09T17:42:34.765 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6249999999999996e-06 2024-02-09T17:42:34.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:34.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.360118+0000", 2024-02-09T17:42:34.765 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016695779999999999 2024-02-09T17:42:34.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:34.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.360311+0000", 2024-02-09T17:42:34.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019232599999999999 2024-02-09T17:42:34.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.766 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,5", 2024-02-09T17:42:34.766 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.360466+0000", 2024-02-09T17:42:34.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000155224 2024-02-09T17:42:34.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.766 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:34.766 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.363549+0000", 2024-02-09T17:42:34.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003083499 2024-02-09T17:42:34.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.766 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:34.767 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.363578+0000", 2024-02-09T17:42:34.767 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8929e-05 2024-02-09T17:42:34.767 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.767 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.767 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:34.767 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.564426+0000", 2024-02-09T17:42:34.767 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20084755100000001 2024-02-09T17:42:34.767 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.767 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.767 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:42:34.767 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.564465+0000", 2024-02-09T17:42:34.767 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9609e-05 2024-02-09T17:42:34.767 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.767 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.767 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:34.768 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.564491+0000", 2024-02-09T17:42:34.768 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5443999999999999e-05 2024-02-09T17:42:34.768 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:34.768 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:34.768 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:34.768 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.768 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.768 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:22601 5.3 5:c3f24201:::benchmark_data_smithi179_115199_object22600:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e477)", 2024-02-09T17:42:34.768 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:41:14.360281+0000", 2024-02-09T17:42:34.768 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 80.350919368000007, 2024-02-09T17:42:34.768 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20523081600000001, 2024-02-09T17:42:34.768 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:34.768 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:42:34.768 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:34.768 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:42:34.769 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:42:34.769 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 22601 2024-02-09T17:42:34.769 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.769 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:34.769 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.769 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:34.769 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.360281+0000", 2024-02-09T17:42:34.769 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:34.769 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.769 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.769 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:34.769 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.360281+0000", 2024-02-09T17:42:34.769 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:34.769 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.769 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:34.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.360282+0000", 2024-02-09T17:42:34.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.068e-06 2024-02-09T17:42:34.770 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:34.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.360392+0000", 2024-02-09T17:42:34.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000109269 2024-02-09T17:42:34.770 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:34.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.360401+0000", 2024-02-09T17:42:34.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6600000000000007e-06 2024-02-09T17:42:34.770 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:34.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.360407+0000", 2024-02-09T17:42:34.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9789999999999998e-06 2024-02-09T17:42:34.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.771 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:34.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.362497+0000", 2024-02-09T17:42:34.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0020897310000000001 2024-02-09T17:42:34.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.771 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:34.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.362625+0000", 2024-02-09T17:42:34.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012762299999999999 2024-02-09T17:42:34.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.772 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,5", 2024-02-09T17:42:34.772 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.362722+0000", 2024-02-09T17:42:34.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6995000000000002e-05 2024-02-09T17:42:34.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.772 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.772 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:34.772 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.364192+0000", 2024-02-09T17:42:34.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001470907 2024-02-09T17:42:34.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.772 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.772 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:34.772 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.366958+0000", 2024-02-09T17:42:34.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0027656299999999998 2024-02-09T17:42:34.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.772 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.773 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:34.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.565425+0000", 2024-02-09T17:42:34.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.198467172 2024-02-09T17:42:34.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.773 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:42:34.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.565478+0000", 2024-02-09T17:42:34.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2419000000000001e-05 2024-02-09T17:42:34.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.773 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:34.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.565512+0000", 2024-02-09T17:42:34.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4363000000000003e-05 2024-02-09T17:42:34.773 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:34.773 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:34.774 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:34.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.774 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.774 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:22610 5.3 5:c60fda14:::benchmark_data_smithi179_115199_object22609:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e477)", 2024-02-09T17:42:34.774 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:41:14.364687+0000", 2024-02-09T17:42:34.774 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 80.346513950000002, 2024-02-09T17:42:34.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20096886, 2024-02-09T17:42:34.774 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:34.774 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:42:34.774 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:34.774 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:42:34.774 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:42:34.774 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 22610 2024-02-09T17:42:34.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.774 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:34.775 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.775 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:34.775 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.364687+0000", 2024-02-09T17:42:34.775 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:34.775 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.775 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.775 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:34.775 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.364687+0000", 2024-02-09T17:42:34.775 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:34.775 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.775 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.775 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:34.775 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.364688+0000", 2024-02-09T17:42:34.775 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2389999999999999e-06 2024-02-09T17:42:34.776 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.776 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.776 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:34.776 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.364742+0000", 2024-02-09T17:42:34.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3779999999999997e-05 2024-02-09T17:42:34.776 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.776 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.776 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:34.776 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.364753+0000", 2024-02-09T17:42:34.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1121000000000001e-05 2024-02-09T17:42:34.776 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.776 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.776 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:34.776 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.364759+0000", 2024-02-09T17:42:34.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1999999999999999e-06 2024-02-09T17:42:34.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:34.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.364822+0000", 2024-02-09T17:42:34.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3472000000000005e-05 2024-02-09T17:42:34.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:34.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.364968+0000", 2024-02-09T17:42:34.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000145628 2024-02-09T17:42:34.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,5", 2024-02-09T17:42:34.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.365174+0000", 2024-02-09T17:42:34.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020566900000000001 2024-02-09T17:42:34.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:34.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.366962+0000", 2024-02-09T17:42:34.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017886950000000001 2024-02-09T17:42:34.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:34.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.368246+0000", 2024-02-09T17:42:34.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0012837300000000001 2024-02-09T17:42:34.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:34.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.565557+0000", 2024-02-09T17:42:34.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19731054200000001 2024-02-09T17:42:34.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.779 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:42:34.779 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.565620+0000", 2024-02-09T17:42:34.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3672999999999998e-05 2024-02-09T17:42:34.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.779 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:34.779 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.565655+0000", 2024-02-09T17:42:34.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5111000000000003e-05 2024-02-09T17:42:34.779 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:34.779 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:34.779 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:34.780 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.780 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.780 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:22612 5.3 5:c49ee808:::benchmark_data_smithi179_115199_object22611:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e477)", 2024-02-09T17:42:34.780 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:41:14.365060+0000", 2024-02-09T17:42:34.780 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 80.346140168000005, 2024-02-09T17:42:34.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.201005463, 2024-02-09T17:42:34.780 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:34.780 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:42:34.780 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:34.780 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:42:34.780 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:42:34.780 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 22612 2024-02-09T17:42:34.780 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.781 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:34.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.781 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:34.781 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.365060+0000", 2024-02-09T17:42:34.781 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:34.781 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.781 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:34.781 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.365060+0000", 2024-02-09T17:42:34.781 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:34.781 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.782 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:34.782 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.365061+0000", 2024-02-09T17:42:34.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0109999999999999e-06 2024-02-09T17:42:34.782 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.782 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.782 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:34.782 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.365121+0000", 2024-02-09T17:42:34.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9905999999999999e-05 2024-02-09T17:42:34.782 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.782 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.782 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:34.782 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.365134+0000", 2024-02-09T17:42:34.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2187999999999999e-05 2024-02-09T17:42:34.782 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.782 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.783 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:34.783 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.365142+0000", 2024-02-09T17:42:34.783 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4619999999999996e-06 2024-02-09T17:42:34.783 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.783 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:34.783 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.366064+0000", 2024-02-09T17:42:34.783 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000921885 2024-02-09T17:42:34.783 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.783 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:34.783 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.366198+0000", 2024-02-09T17:42:34.783 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013434499999999999 2024-02-09T17:42:34.783 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.784 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,5", 2024-02-09T17:42:34.784 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.366337+0000", 2024-02-09T17:42:34.784 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013868499999999999 2024-02-09T17:42:34.784 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.784 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.784 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:34.784 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.367092+0000", 2024-02-09T17:42:34.784 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00075555000000000001 2024-02-09T17:42:34.784 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.784 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.784 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:34.784 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.368396+0000", 2024-02-09T17:42:34.784 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001303794 2024-02-09T17:42:34.784 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.785 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.785 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:34.785 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.565980+0000", 2024-02-09T17:42:34.785 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19758384900000001 2024-02-09T17:42:34.785 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.785 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.785 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:42:34.785 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.566041+0000", 2024-02-09T17:42:34.785 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0980999999999998e-05 2024-02-09T17:42:34.785 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.785 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.785 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:34.785 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.566066+0000", 2024-02-09T17:42:34.786 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4807000000000002e-05 2024-02-09T17:42:34.786 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:34.786 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:34.786 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:34.786 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.786 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.786 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:23262 5.2 5:4e4c7e18:::benchmark_data_smithi179_115199_object23261:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e478)", 2024-02-09T17:42:34.786 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:41:14.820360+0000", 2024-02-09T17:42:34.786 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 79.890841023999997, 2024-02-09T17:42:34.786 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20603479199999999, 2024-02-09T17:42:34.786 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:34.786 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:42:34.787 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:34.787 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:42:34.787 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:42:34.787 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 23262 2024-02-09T17:42:34.787 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.787 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:34.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.787 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:34.787 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.820360+0000", 2024-02-09T17:42:34.787 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:34.787 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.787 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:34.787 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.820360+0000", 2024-02-09T17:42:34.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:34.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.788 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:34.788 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.820361+0000", 2024-02-09T17:42:34.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.446e-06 2024-02-09T17:42:34.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.788 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:34.788 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.820443+0000", 2024-02-09T17:42:34.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1799999999999996e-05 2024-02-09T17:42:34.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.788 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:34.789 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.820456+0000", 2024-02-09T17:42:34.789 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2979999999999999e-05 2024-02-09T17:42:34.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.789 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:34.789 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.820463+0000", 2024-02-09T17:42:34.789 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7309999999999996e-06 2024-02-09T17:42:34.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.789 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:34.789 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.820514+0000", 2024-02-09T17:42:34.789 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1109999999999997e-05 2024-02-09T17:42:34.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.789 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:34.790 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.820620+0000", 2024-02-09T17:42:34.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000106705 2024-02-09T17:42:34.790 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.790 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.790 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,10", 2024-02-09T17:42:34.790 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.820739+0000", 2024-02-09T17:42:34.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011826 2024-02-09T17:42:34.790 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.790 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.790 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:34.790 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.821747+0000", 2024-02-09T17:42:34.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0010087500000000001 2024-02-09T17:42:34.790 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.790 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.790 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:34.790 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.822460+0000", 2024-02-09T17:42:34.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00071277700000000005 2024-02-09T17:42:34.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:34.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:15.026311+0000", 2024-02-09T17:42:34.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20385081599999999 2024-02-09T17:42:34.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:42:34.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:15.026354+0000", 2024-02-09T17:42:34.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2812999999999998e-05 2024-02-09T17:42:34.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:34.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:15.026394+0000", 2024-02-09T17:42:34.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0604e-05 2024-02-09T17:42:34.792 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:34.792 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:34.792 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:34.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.792 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:23271 5.3 5:cf6180d6:::benchmark_data_smithi179_115199_object23270:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e478)", 2024-02-09T17:42:34.792 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:41:14.823341+0000", 2024-02-09T17:42:34.792 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 79.887860067000005, 2024-02-09T17:42:34.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20541593999999999, 2024-02-09T17:42:34.792 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:34.792 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:42:34.792 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:34.792 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:42:34.793 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:42:34.793 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 23271 2024-02-09T17:42:34.793 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.793 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:34.793 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.793 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:34.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.823341+0000", 2024-02-09T17:42:34.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:34.793 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.793 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.793 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:34.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.823341+0000", 2024-02-09T17:42:34.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:34.793 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.794 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:34.794 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.823341+0000", 2024-02-09T17:42:34.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8000000000000005e-07 2024-02-09T17:42:34.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.794 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:34.794 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.823406+0000", 2024-02-09T17:42:34.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4789000000000006e-05 2024-02-09T17:42:34.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.794 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:34.794 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.823417+0000", 2024-02-09T17:42:34.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1121000000000001e-05 2024-02-09T17:42:34.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.795 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.795 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:34.795 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.823425+0000", 2024-02-09T17:42:34.795 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4730000000000003e-06 2024-02-09T17:42:34.795 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.795 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.795 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:34.795 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.823477+0000", 2024-02-09T17:42:34.795 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2590000000000003e-05 2024-02-09T17:42:34.795 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.795 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.795 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:34.795 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.823593+0000", 2024-02-09T17:42:34.795 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000116018 2024-02-09T17:42:34.796 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.796 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.796 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,5", 2024-02-09T17:42:34.796 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.823740+0000", 2024-02-09T17:42:34.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000147066 2024-02-09T17:42:34.796 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.796 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.796 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:34.796 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.826396+0000", 2024-02-09T17:42:34.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002656088 2024-02-09T17:42:34.796 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.796 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.796 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:34.796 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.829444+0000", 2024-02-09T17:42:34.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0030480020000000002 2024-02-09T17:42:34.797 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.797 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.797 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:34.797 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:15.028671+0000", 2024-02-09T17:42:34.797 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.199226661 2024-02-09T17:42:34.797 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.797 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.797 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:42:34.797 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:15.028713+0000", 2024-02-09T17:42:34.797 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1421000000000002e-05 2024-02-09T17:42:34.797 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.797 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.797 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:34.798 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:15.028756+0000", 2024-02-09T17:42:34.798 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3930999999999997e-05 2024-02-09T17:42:34.798 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:34.798 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:34.798 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:34.798 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.798 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.798 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:23275 5.3 5:c9b325cc:::benchmark_data_smithi179_115199_object23274:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e478)", 2024-02-09T17:42:34.798 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:41:14.825009+0000", 2024-02-09T17:42:34.798 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 79.886191416000003, 2024-02-09T17:42:34.798 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20389706899999999, 2024-02-09T17:42:34.798 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:34.798 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:42:34.798 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:34.798 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:42:34.798 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:42:34.799 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 23275 2024-02-09T17:42:34.799 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.799 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:34.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.799 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:34.799 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.825009+0000", 2024-02-09T17:42:34.799 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:34.799 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.799 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:34.799 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.825009+0000", 2024-02-09T17:42:34.799 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:34.799 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.799 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:34.800 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.825010+0000", 2024-02-09T17:42:34.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.099e-06 2024-02-09T17:42:34.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.800 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.800 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:34.800 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.825070+0000", 2024-02-09T17:42:34.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9953e-05 2024-02-09T17:42:34.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.800 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.800 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:34.800 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.825081+0000", 2024-02-09T17:42:34.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1199999999999999e-05 2024-02-09T17:42:34.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.800 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.800 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:34.801 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.825089+0000", 2024-02-09T17:42:34.801 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8900000000000007e-06 2024-02-09T17:42:34.801 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.801 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.801 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:34.801 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.825147+0000", 2024-02-09T17:42:34.801 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8084000000000002e-05 2024-02-09T17:42:34.801 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.801 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.801 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:34.801 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.825283+0000", 2024-02-09T17:42:34.801 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013611299999999999 2024-02-09T17:42:34.801 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.801 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.801 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,5", 2024-02-09T17:42:34.802 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.825422+0000", 2024-02-09T17:42:34.802 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013818700000000001 2024-02-09T17:42:34.802 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.802 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.802 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:34.802 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.826588+0000", 2024-02-09T17:42:34.802 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0011660139999999999 2024-02-09T17:42:34.802 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.802 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.802 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:34.802 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:14.830833+0000", 2024-02-09T17:42:34.802 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042452000000000002 2024-02-09T17:42:34.802 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.802 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.802 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:34.802 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:15.028800+0000", 2024-02-09T17:42:34.803 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19796697199999999 2024-02-09T17:42:34.803 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.803 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.803 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:42:34.803 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:15.028850+0000", 2024-02-09T17:42:34.803 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0226999999999999e-05 2024-02-09T17:42:34.803 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.803 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.803 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:34.803 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:15.028906+0000", 2024-02-09T17:42:34.803 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.613e-05 2024-02-09T17:42:34.803 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:34.803 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:34.803 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:34.803 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.804 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.804 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:43129 5.2 5:42d2a849:::benchmark_data_smithi179_115199_object43128:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e483)", 2024-02-09T17:42:34.804 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:41:25.852375+0000", 2024-02-09T17:42:34.804 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 68.858825128000007, 2024-02-09T17:42:34.804 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20656733199999999, 2024-02-09T17:42:34.804 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:34.804 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:42:34.804 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:34.804 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:42:34.804 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:42:34.804 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 43129 2024-02-09T17:42:34.804 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.804 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:34.804 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.805 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:34.805 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:25.852375+0000", 2024-02-09T17:42:34.805 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:34.805 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.805 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.805 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:34.805 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:25.852375+0000", 2024-02-09T17:42:34.805 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:34.805 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.805 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.805 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:34.805 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:25.852376+0000", 2024-02-09T17:42:34.805 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0999999999999997e-07 2024-02-09T17:42:34.805 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.805 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.806 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:34.806 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:25.852493+0000", 2024-02-09T17:42:34.806 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000116665 2024-02-09T17:42:34.806 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.806 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.806 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:34.806 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:25.852507+0000", 2024-02-09T17:42:34.806 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.412e-05 2024-02-09T17:42:34.806 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.806 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.806 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:34.806 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:25.852516+0000", 2024-02-09T17:42:34.806 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6740000000000003e-06 2024-02-09T17:42:34.806 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.806 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.807 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:34.807 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:25.852578+0000", 2024-02-09T17:42:34.807 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1829999999999996e-05 2024-02-09T17:42:34.807 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.807 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.807 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:34.807 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:25.852729+0000", 2024-02-09T17:42:34.807 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015105699999999999 2024-02-09T17:42:34.807 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.807 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.807 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,10", 2024-02-09T17:42:34.807 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:25.852866+0000", 2024-02-09T17:42:34.807 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013740399999999999 2024-02-09T17:42:34.807 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.807 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.808 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:34.808 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:25.854754+0000", 2024-02-09T17:42:34.808 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018883960000000001 2024-02-09T17:42:34.808 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.808 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.808 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:34.808 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:25.855736+0000", 2024-02-09T17:42:34.808 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00098122199999999991 2024-02-09T17:42:34.808 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.808 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.808 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:34.808 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:26.058859+0000", 2024-02-09T17:42:34.808 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.203123734 2024-02-09T17:42:34.808 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.809 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.809 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:42:34.809 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:26.058907+0000", 2024-02-09T17:42:34.809 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7701999999999998e-05 2024-02-09T17:42:34.809 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.809 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.809 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:34.809 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:26.058943+0000", 2024-02-09T17:42:34.809 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5717999999999999e-05 2024-02-09T17:42:34.809 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:34.809 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:34.809 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:34.809 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.809 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.809 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:108578 5.3 5:c3dd9fad:::benchmark_data_smithi179_115199_object108577:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-02-09T17:42:34.810 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:42:13.544522+0000", 2024-02-09T17:42:34.810 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 21.166678739000002, 2024-02-09T17:42:34.810 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22786947099999999, 2024-02-09T17:42:34.810 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:34.810 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:42:34.810 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:34.810 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:42:34.810 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:42:34.810 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 108578 2024-02-09T17:42:34.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.810 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:34.810 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.810 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:34.810 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.544522+0000", 2024-02-09T17:42:34.810 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:34.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.811 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.811 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:34.811 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.544522+0000", 2024-02-09T17:42:34.811 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:34.811 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.811 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.811 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:34.811 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.544522+0000", 2024-02-09T17:42:34.811 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.63e-07 2024-02-09T17:42:34.811 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.811 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.811 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:34.811 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.544569+0000", 2024-02-09T17:42:34.811 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6227000000000003e-05 2024-02-09T17:42:34.812 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.812 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.812 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:34.812 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.544575+0000", 2024-02-09T17:42:34.812 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5549999999999999e-06 2024-02-09T17:42:34.812 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.812 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.812 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:34.812 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.544580+0000", 2024-02-09T17:42:34.812 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0239999999999997e-06 2024-02-09T17:42:34.812 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.812 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.812 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:34.812 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.552329+0000", 2024-02-09T17:42:34.812 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0077490119999999996 2024-02-09T17:42:34.813 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.813 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.813 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:34.813 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.552442+0000", 2024-02-09T17:42:34.813 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000112934 2024-02-09T17:42:34.813 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.813 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.813 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,5", 2024-02-09T17:42:34.813 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.552570+0000", 2024-02-09T17:42:34.813 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012795800000000001 2024-02-09T17:42:34.813 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.813 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.813 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:34.813 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.558341+0000", 2024-02-09T17:42:34.814 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.005770631 2024-02-09T17:42:34.814 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.814 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.814 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:34.814 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.569543+0000", 2024-02-09T17:42:34.814 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011202287 2024-02-09T17:42:34.814 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.814 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.814 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:34.814 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.772295+0000", 2024-02-09T17:42:34.814 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.202751868 2024-02-09T17:42:34.814 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.814 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.814 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:42:34.814 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.772349+0000", 2024-02-09T17:42:34.815 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4318999999999999e-05 2024-02-09T17:42:34.815 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.815 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.815 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:34.815 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.772391+0000", 2024-02-09T17:42:34.815 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2193000000000001e-05 2024-02-09T17:42:34.815 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:34.815 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:34.815 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:34.815 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.815 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.815 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:108591 5.3 5:cc4f4e47:::benchmark_data_smithi179_115199_object108590:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-02-09T17:42:34.815 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:42:13.558537+0000", 2024-02-09T17:42:34.815 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 21.152663432000001, 2024-02-09T17:42:34.815 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.229785869, 2024-02-09T17:42:34.816 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:34.816 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:42:34.816 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:34.816 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:42:34.816 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:42:34.816 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 108591 2024-02-09T17:42:34.816 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.816 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:34.816 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.816 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:34.816 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.558537+0000", 2024-02-09T17:42:34.816 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:34.816 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.816 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.817 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:34.817 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.558537+0000", 2024-02-09T17:42:34.817 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:34.817 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.817 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.817 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:34.817 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.558539+0000", 2024-02-09T17:42:34.817 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5939999999999999e-06 2024-02-09T17:42:34.817 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.817 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.817 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:34.817 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.558607+0000", 2024-02-09T17:42:34.817 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8172999999999999e-05 2024-02-09T17:42:34.817 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.817 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.818 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:34.818 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.558621+0000", 2024-02-09T17:42:34.818 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4361e-05 2024-02-09T17:42:34.818 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.818 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.818 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:34.818 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.558629+0000", 2024-02-09T17:42:34.818 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.123e-06 2024-02-09T17:42:34.818 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.818 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.818 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:34.818 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.569572+0000", 2024-02-09T17:42:34.818 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010942999 2024-02-09T17:42:34.819 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.819 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.819 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:34.819 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.569672+0000", 2024-02-09T17:42:34.819 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9982000000000006e-05 2024-02-09T17:42:34.819 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.819 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.819 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,5", 2024-02-09T17:42:34.819 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.569798+0000", 2024-02-09T17:42:34.819 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012534800000000001 2024-02-09T17:42:34.819 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.819 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.819 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:34.819 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.573124+0000", 2024-02-09T17:42:34.819 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0033262669999999999 2024-02-09T17:42:34.820 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.820 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.820 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:34.820 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.574307+0000", 2024-02-09T17:42:34.820 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0011831809999999999 2024-02-09T17:42:34.820 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.820 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.820 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:34.820 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.788199+0000", 2024-02-09T17:42:34.820 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21389181400000001 2024-02-09T17:42:34.820 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.820 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.820 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:42:34.820 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.788273+0000", 2024-02-09T17:42:34.821 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4396999999999999e-05 2024-02-09T17:42:34.821 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.821 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.821 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:34.821 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.788323+0000", 2024-02-09T17:42:34.821 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9629999999999997e-05 2024-02-09T17:42:34.821 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:34.821 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:34.821 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:34.821 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.821 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.821 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:108607 5.3 5:c761f34f:::benchmark_data_smithi179_115199_object108606:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-02-09T17:42:34.821 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:42:13.575058+0000", 2024-02-09T17:42:34.821 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 21.136142144000001, 2024-02-09T17:42:34.822 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22974799600000001, 2024-02-09T17:42:34.822 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:34.822 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:42:34.822 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:34.822 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:42:34.822 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:42:34.822 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 108607 2024-02-09T17:42:34.822 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.822 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:34.822 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.822 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:34.822 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.575058+0000", 2024-02-09T17:42:34.822 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:34.822 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.822 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.823 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:34.823 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.575058+0000", 2024-02-09T17:42:34.823 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:34.823 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.823 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.823 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:34.823 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.575060+0000", 2024-02-09T17:42:34.823 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.745e-06 2024-02-09T17:42:34.823 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.823 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.823 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:34.823 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.575164+0000", 2024-02-09T17:42:34.823 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000103955 2024-02-09T17:42:34.823 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.823 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.824 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:34.824 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.575183+0000", 2024-02-09T17:42:34.824 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8782000000000001e-05 2024-02-09T17:42:34.824 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.824 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.824 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:34.824 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.575191+0000", 2024-02-09T17:42:34.824 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.028e-06 2024-02-09T17:42:34.824 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.824 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.824 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:34.824 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.575252+0000", 2024-02-09T17:42:34.824 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.109e-05 2024-02-09T17:42:34.824 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.824 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.825 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:34.825 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.575389+0000", 2024-02-09T17:42:34.825 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013661599999999999 2024-02-09T17:42:34.825 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.825 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.825 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,5", 2024-02-09T17:42:34.825 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.575528+0000", 2024-02-09T17:42:34.825 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013976 2024-02-09T17:42:34.825 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.825 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.825 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:34.825 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.582986+0000", 2024-02-09T17:42:34.825 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0074577130000000004 2024-02-09T17:42:34.825 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.825 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.825 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:34.826 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.590357+0000", 2024-02-09T17:42:34.826 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0073711360000000004 2024-02-09T17:42:34.826 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.826 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.826 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:34.826 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.804692+0000", 2024-02-09T17:42:34.826 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21433513200000001 2024-02-09T17:42:34.826 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.826 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.826 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:42:34.826 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.804759+0000", 2024-02-09T17:42:34.826 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6550000000000005e-05 2024-02-09T17:42:34.826 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.826 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.826 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:34.827 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.804806+0000", 2024-02-09T17:42:34.827 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7488999999999999e-05 2024-02-09T17:42:34.827 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:34.827 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:34.827 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:34.827 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.827 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.827 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:108629 5.3 5:cf57fb68:::benchmark_data_smithi179_115199_object108628:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-02-09T17:42:34.827 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:42:13.599802+0000", 2024-02-09T17:42:34.827 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 21.111399046999999, 2024-02-09T17:42:34.827 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20514634600000001, 2024-02-09T17:42:34.827 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:34.827 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:42:34.827 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:34.827 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:42:34.828 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:42:34.828 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 108629 2024-02-09T17:42:34.828 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.828 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:34.828 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.828 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:34.828 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.599802+0000", 2024-02-09T17:42:34.828 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:34.828 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.828 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.828 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:34.828 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.599802+0000", 2024-02-09T17:42:34.828 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:34.828 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.829 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.829 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:34.829 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.599807+0000", 2024-02-09T17:42:34.829 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7479999999999999e-06 2024-02-09T17:42:34.829 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.829 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.829 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:34.829 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.599861+0000", 2024-02-09T17:42:34.829 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3711000000000001e-05 2024-02-09T17:42:34.829 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.829 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.829 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:34.829 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.599874+0000", 2024-02-09T17:42:34.829 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.347e-05 2024-02-09T17:42:34.829 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.830 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.830 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:34.830 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.599881+0000", 2024-02-09T17:42:34.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.657e-06 2024-02-09T17:42:34.830 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.830 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.830 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:34.830 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.602968+0000", 2024-02-09T17:42:34.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0030871029999999999 2024-02-09T17:42:34.830 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.830 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.830 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:34.830 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.603107+0000", 2024-02-09T17:42:34.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013906300000000001 2024-02-09T17:42:34.830 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.830 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.831 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,5", 2024-02-09T17:42:34.831 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.603265+0000", 2024-02-09T17:42:34.831 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015803599999999999 2024-02-09T17:42:34.831 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.831 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.831 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:34.831 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.608392+0000", 2024-02-09T17:42:34.831 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0051267939999999996 2024-02-09T17:42:34.831 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.831 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.831 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:34.831 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.619288+0000", 2024-02-09T17:42:34.831 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.01089621 2024-02-09T17:42:34.831 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.831 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.831 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:34.832 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.804856+0000", 2024-02-09T17:42:34.832 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18556797899999999 2024-02-09T17:42:34.832 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.832 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.832 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:42:34.832 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.804911+0000", 2024-02-09T17:42:34.832 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4987000000000001e-05 2024-02-09T17:42:34.832 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:34.832 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:34.832 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:34.832 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:13.804948+0000", 2024-02-09T17:42:34.832 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6588000000000003e-05 2024-02-09T17:42:34.832 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:34.832 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:34.832 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:34.833 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:34.833 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:34.833 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:42:34.834 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:42:34.834 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:42:34.834 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:42:34.834 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:42:34.838 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:42:34.868 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:42:34.873 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:42:34.873 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:42:34.873 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:42:34.873 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:42:34.873 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:42:34.873 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:42:34.873 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:42:34.873 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:42:34.873 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:42:34.874 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:42:34.874 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:42:34.874 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:42:34.910 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:34.936 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:42:34.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:34.941 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:42:34.975 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Total time run: 90.0041 2024-02-09T17:42:34.975 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Total writes made: 128903 2024-02-09T17:42:34.975 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Write size: 65536 2024-02-09T17:42:34.975 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Object size: 65536 2024-02-09T17:42:34.975 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Bandwidth (MB/sec): 89.5119 2024-02-09T17:42:34.975 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Stddev Bandwidth: 42.3724 2024-02-09T17:42:34.975 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Max bandwidth (MB/sec): 242.125 2024-02-09T17:42:34.975 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Min bandwidth (MB/sec): 0 2024-02-09T17:42:34.975 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Average IOPS: 1432 2024-02-09T17:42:34.975 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Stddev IOPS: 677.969 2024-02-09T17:42:34.975 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Max IOPS: 3874 2024-02-09T17:42:34.975 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Min IOPS: 0 2024-02-09T17:42:34.975 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Average Latency(s): 0.0111684 2024-02-09T17:42:34.976 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Stddev Latency(s): 0.0323144 2024-02-09T17:42:34.976 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Max latency(s): 2.34848 2024-02-09T17:42:34.976 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Min latency(s): 0.00110706 2024-02-09T17:42:34.981 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Cleaning up (deleting benchmark objects) 2024-02-09T17:42:35.010 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:35 smithi162 ceph-mon[44304]: 5.2 scrub starts 2024-02-09T17:42:35.010 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:35 smithi162 ceph-mon[44304]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:42:35.010 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:35 smithi162 ceph-mon[44304]: Cluster is now healthy 2024-02-09T17:42:35.010 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:35 smithi162 ceph-mon[44304]: 5.8 scrub ok 2024-02-09T17:42:35.010 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:35 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/933913829' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:42:35.010 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:35 smithi162 ceph-mon[44304]: osdmap e502: 12 total, 12 up, 11 in 2024-02-09T17:42:35.010 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:35 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1835864473' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:42:35.010 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:35 smithi162 ceph-mon[44304]: 1.5 scrub starts 2024-02-09T17:42:35.010 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:35 smithi162 ceph-mon[44304]: 1.5 scrub ok 2024-02-09T17:42:35.043 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:42:35.104 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:35.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:34 smithi179 ceph-mon[43975]: 5.2 scrub starts 2024-02-09T17:42:35.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:34 smithi179 ceph-mon[43975]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:42:35.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:34 smithi179 ceph-mon[43975]: Cluster is now healthy 2024-02-09T17:42:35.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:34 smithi179 ceph-mon[43975]: 5.8 scrub ok 2024-02-09T17:42:35.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:34 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/933913829' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:42:35.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:34 smithi179 ceph-mon[43975]: osdmap e502: 12 total, 12 up, 11 in 2024-02-09T17:42:35.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:34 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1835864473' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:42:35.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:34 smithi179 ceph-mon[43975]: 1.5 scrub starts 2024-02-09T17:42:35.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:34 smithi179 ceph-mon[43975]: 1.5 scrub ok 2024-02-09T17:42:35.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:34 smithi099 ceph-mon[39420]: 5.2 scrub starts 2024-02-09T17:42:35.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:34 smithi099 ceph-mon[39420]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:42:35.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:34 smithi099 ceph-mon[39420]: Cluster is now healthy 2024-02-09T17:42:35.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:34 smithi099 ceph-mon[39420]: 5.8 scrub ok 2024-02-09T17:42:35.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:34 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/933913829' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:42:35.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:34 smithi099 ceph-mon[39420]: osdmap e502: 12 total, 12 up, 11 in 2024-02-09T17:42:35.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:34 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1835864473' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:42:35.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:34 smithi099 ceph-mon[39420]: 1.5 scrub starts 2024-02-09T17:42:35.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:34 smithi099 ceph-mon[39420]: 1.5 scrub ok 2024-02-09T17:42:35.486 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:42:35.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:35.496 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-09T17:42:35.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:42:35.598 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:42:35.630 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:35.779 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:42:35.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:35.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:42:35.884 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:42:35.965 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:42:35.996 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:36.029 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:42:36.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:36.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:42:36.133 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:42:36.167 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:36.196 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:42:36.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:36.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:42:36.301 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:42:36.313 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:36 smithi162 ceph-mon[44304]: pgmap v1277: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 7.7 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 54 MiB/s wr, 864 op/s 2024-02-09T17:42:36.313 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:36 smithi162 ceph-mon[44304]: 5.1 scrub starts 2024-02-09T17:42:36.313 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:36 smithi162 ceph-mon[44304]: 5.4 scrub ok 2024-02-09T17:42:36.364 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:36.414 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:42:36.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:36 smithi179 ceph-mon[43975]: pgmap v1277: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 7.7 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 54 MiB/s wr, 864 op/s 2024-02-09T17:42:36.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:36 smithi179 ceph-mon[43975]: 5.1 scrub starts 2024-02-09T17:42:36.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:36 smithi179 ceph-mon[43975]: 5.4 scrub ok 2024-02-09T17:42:36.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:36.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:42:36.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:36 smithi099 ceph-mon[39420]: pgmap v1277: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 7.7 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 54 MiB/s wr, 864 op/s 2024-02-09T17:42:36.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:36 smithi099 ceph-mon[39420]: 5.1 scrub starts 2024-02-09T17:42:36.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:36 smithi099 ceph-mon[39420]: 5.4 scrub ok 2024-02-09T17:42:36.523 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:42:36.588 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:36.975 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:42:36.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:36.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:42:36.991 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set noscrub 2024-02-09T17:42:37.082 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:42:37.115 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:37.144 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:42:37.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:37.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:42:37.222 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] 2024-02-09T17:42:37.222 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:42:37.223 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_3 2024-02-09T17:42:37.223 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph pg dump --format=json 2024-02-09T17:42:37.250 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:42:37.330 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:42:37.336 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:37 smithi162 ceph-mon[44304]: 5.2 scrub ok 2024-02-09T17:42:37.336 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:37 smithi162 ceph-mon[44304]: 1.9 scrub starts 2024-02-09T17:42:37.336 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:37 smithi162 ceph-mon[44304]: 5.f scrub starts 2024-02-09T17:42:37.336 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:37 smithi162 ceph-mon[44304]: 1.9 scrub ok 2024-02-09T17:42:37.336 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:37 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:42:37.336 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:37 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:42:37.341 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:37.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:37 smithi179 ceph-mon[43975]: 5.2 scrub ok 2024-02-09T17:42:37.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:37 smithi179 ceph-mon[43975]: 1.9 scrub starts 2024-02-09T17:42:37.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:37 smithi179 ceph-mon[43975]: 5.f scrub starts 2024-02-09T17:42:37.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:37 smithi179 ceph-mon[43975]: 1.9 scrub ok 2024-02-09T17:42:37.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:37 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:42:37.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:37 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:42:37.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:37 smithi099 ceph-mon[39420]: 5.2 scrub ok 2024-02-09T17:42:37.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:37 smithi099 ceph-mon[39420]: 1.9 scrub starts 2024-02-09T17:42:37.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:37 smithi099 ceph-mon[39420]: 5.f scrub starts 2024-02-09T17:42:37.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:37 smithi099 ceph-mon[39420]: 1.9 scrub ok 2024-02-09T17:42:37.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:37 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:42:37.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:37 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:42:37.577 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:42:37.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:37.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:42:37.685 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:42:37.739 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:37.795 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:42:37.908 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:42:37.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:37.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:42:38.016 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:42:38.084 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:38.239 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:42:38.241 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:42:38.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:38 smithi099 ceph-mon[39420]: 5.7 scrub starts 2024-02-09T17:42:38.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:38 smithi099 ceph-mon[39420]: pgmap v1278: 37 pgs: 2 active+clean+scrubbing, 35 active+clean; 7.8 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 127 B/s rd, 55 MiB/s wr, 927 op/s 2024-02-09T17:42:38.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:38 smithi099 ceph-mon[39420]: osdmap e503: 12 total, 12 up, 11 in 2024-02-09T17:42:38.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:38.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:42:38.314 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:42:38.314 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:42:38.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:38 smithi162 ceph-mon[44304]: 5.7 scrub starts 2024-02-09T17:42:38.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:38 smithi162 ceph-mon[44304]: pgmap v1278: 37 pgs: 2 active+clean+scrubbing, 35 active+clean; 7.8 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 127 B/s rd, 55 MiB/s wr, 927 op/s 2024-02-09T17:42:38.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:38 smithi162 ceph-mon[44304]: osdmap e503: 12 total, 12 up, 11 in 2024-02-09T17:42:38.350 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:42:38.384 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:38.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:38 smithi179 ceph-mon[43975]: 5.7 scrub starts 2024-02-09T17:42:38.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:38 smithi179 ceph-mon[43975]: pgmap v1278: 37 pgs: 2 active+clean+scrubbing, 35 active+clean; 7.8 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 127 B/s rd, 55 MiB/s wr, 927 op/s 2024-02-09T17:42:38.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:38 smithi179 ceph-mon[43975]: osdmap e503: 12 total, 12 up, 11 in 2024-02-09T17:42:38.515 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:42:38.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:38.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:42:38.626 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:42:38.680 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:38.706 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:42:38.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:38.711 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:42:38.813 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:42:38.863 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:39.068 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:39 smithi162 ceph-mon[44304]: 5.1 scrub ok 2024-02-09T17:42:39.068 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:39 smithi162 ceph-mon[44304]: 1.3 scrub starts 2024-02-09T17:42:39.068 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:39 smithi162 ceph-mon[44304]: 1.3 scrub ok 2024-02-09T17:42:39.091 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:42:39.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:39.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:42:39.110 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-09T17:42:39.197 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:42:39.238 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:39.274 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:42:39.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:39.283 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:42:39.384 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:42:39.413 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:39 smithi099 ceph-mon[39420]: 5.1 scrub ok 2024-02-09T17:42:39.413 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:39 smithi099 ceph-mon[39420]: 1.3 scrub starts 2024-02-09T17:42:39.413 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:39 smithi099 ceph-mon[39420]: 1.3 scrub ok 2024-02-09T17:42:39.415 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:39.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:39 smithi179 ceph-mon[43975]: 5.1 scrub ok 2024-02-09T17:42:39.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:39 smithi179 ceph-mon[43975]: 1.3 scrub starts 2024-02-09T17:42:39.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:39 smithi179 ceph-mon[43975]: 1.3 scrub ok 2024-02-09T17:42:39.448 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:42:39.452 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:42:39.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:39.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:42:39.554 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:42:39.578 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:42:39.592 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:39.618 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:42:39.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:39.623 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:42:39.724 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:42:39.757 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:39.880 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:42:39.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:39.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:42:39.984 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:42:40.034 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:40.151 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:42:40.160 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:40 smithi162 ceph-mon[44304]: pgmap v1280: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 7.8 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 130 B/s rd, 36 MiB/s wr, 687 op/s 2024-02-09T17:42:40.160 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:40 smithi162 ceph-mon[44304]: 5.f scrub ok 2024-02-09T17:42:40.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:40.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:42:40.262 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:42:40.316 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:40 smithi179 ceph-mon[43975]: pgmap v1280: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 7.8 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 130 B/s rd, 36 MiB/s wr, 687 op/s 2024-02-09T17:42:40.316 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:40 smithi179 ceph-mon[43975]: 5.f scrub ok 2024-02-09T17:42:40.319 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:40.361 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:42:40.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:40.367 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:42:40.468 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:42:40.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:40 smithi099 ceph-mon[39420]: pgmap v1280: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 7.8 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 130 B/s rd, 36 MiB/s wr, 687 op/s 2024-02-09T17:42:40.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:40 smithi099 ceph-mon[39420]: 5.f scrub ok 2024-02-09T17:42:40.506 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:40.533 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:42:40.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:40.545 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:42:40.646 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:42:40.695 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:40.719 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:42:40.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:40.726 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:42:40.827 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:42:40.871 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:41.025 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:42:41.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:41.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:42:41.067 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:41 smithi179 ceph-mon[43975]: 5.7 scrub ok 2024-02-09T17:42:41.067 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:41 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/178841324' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:42:41.067 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:41 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:42:41.085 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is set 2024-02-09T17:42:41.134 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:42:41.171 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:41.368 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:41 smithi162 ceph-mon[44304]: 5.7 scrub ok 2024-02-09T17:42:41.368 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:41 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/178841324' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:42:41.368 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:41 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:42:41.398 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:42:41.399 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-02-09T17:42:41.430 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:41 smithi099 ceph-mon[39420]: 5.7 scrub ok 2024-02-09T17:42:41.430 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:41 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/178841324' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:42:41.430 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:41 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:42:41.769 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:42:41.769 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:42:42.138 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:42:42.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:42.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:42:42.245 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:42:42.307 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:42.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:42 smithi179 ceph-mon[43975]: 1.e scrub starts 2024-02-09T17:42:42.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:42 smithi179 ceph-mon[43975]: 1.e scrub ok 2024-02-09T17:42:42.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:42 smithi179 ceph-mon[43975]: pgmap v1281: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 7.7 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 127 B/s rd, 29 MiB/s wr, 807 op/s 2024-02-09T17:42:42.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:42 smithi179 ceph-mon[43975]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:42:42.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:42 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:42:42.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:42 smithi179 ceph-mon[43975]: osdmap e504: 12 total, 12 up, 11 in 2024-02-09T17:42:42.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:42 smithi162 ceph-mon[44304]: 1.e scrub starts 2024-02-09T17:42:42.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:42 smithi162 ceph-mon[44304]: 1.e scrub ok 2024-02-09T17:42:42.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:42 smithi162 ceph-mon[44304]: pgmap v1281: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 7.7 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 127 B/s rd, 29 MiB/s wr, 807 op/s 2024-02-09T17:42:42.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:42 smithi162 ceph-mon[44304]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:42:42.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:42 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:42:42.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:42 smithi162 ceph-mon[44304]: osdmap e504: 12 total, 12 up, 11 in 2024-02-09T17:42:42.486 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:42:42.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:42 smithi099 ceph-mon[39420]: 1.e scrub starts 2024-02-09T17:42:42.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:42 smithi099 ceph-mon[39420]: 1.e scrub ok 2024-02-09T17:42:42.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:42 smithi099 ceph-mon[39420]: pgmap v1281: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 7.7 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 127 B/s rd, 29 MiB/s wr, 807 op/s 2024-02-09T17:42:42.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:42 smithi099 ceph-mon[39420]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:42:42.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:42 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:42:42.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:42 smithi099 ceph-mon[39420]: osdmap e504: 12 total, 12 up, 11 in 2024-02-09T17:42:42.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:42.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:42:42.537 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-09T17:42:42.600 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:42:42.715 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:42.738 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:42:42.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:42.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:42:42.842 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:42:42.912 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:42.958 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:42:42.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:42.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:42:43.064 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:42:43.077 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:42:43.128 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:43.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:43 smithi162 ceph-mon[44304]: from='client.15906 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:42:43.395 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:42:43.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:43.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:42:43.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:43 smithi179 ceph-mon[43975]: from='client.15906 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:42:43.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:43 smithi099 ceph-mon[39420]: from='client.15906 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:42:43.502 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:42:43.530 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:43.654 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:42:43.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:43.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:42:43.671 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":1282,"stamp":"2024-02-09T17:42:41.068788+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8229290552,"num_objects":125563,"num_object_clones":0,"num_object_copies":376689,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":125563,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132247,"num_write_kb":8249793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3629,"num_bytes_recovered":238499424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135166,"ondisk_log_size":135166,"up":111,"acting":111,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":111,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":29114040,"kb_used_data":24218768,"kb_used_omap":228,"kb_used_meta":4893787,"kb_avail":1002082632,"statfs":{"total":1055945392128,"available":1026132615168,"internally_reserved":0,"allocated":24800018432,"data_stored":24796585830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":233532,"internal_metadata":5011238852},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,4,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26279912,"num_objects":-400,"num_object_clones":0,"num_object_copies":-1200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-400,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6286,"num_write_kb":188289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.067599"},"pg_stats":[{"pgid":"1.14","version":"63'139","reported_seq":963,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-09T17:42:34.012676+0000","last_change":"2024-02-09T17:41:08.077049+0000","last_active":"2024-02-09T17:42:34.012676+0000","last_peered":"2024-02-09T17:42:34.012676+0000","last_clean":"2024-02-09T17:42:34.012676+0000","last_became_active":"2024-02-09T17:39:27.455644+0000","last_became_peered":"2024-02-09T17:39:27.455644+0000","last_unstale":"2024-02-09T17:42:34.012676+0000","last_undegraded":"2024-02-09T17:42:34.012676+0000","last_fullsized":"2024-02-09T17:42:34.012676+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:08.076894+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:38:26.968892+0000","last_clean_scrub_stamp":"2024-02-09T17:41:08.076894+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"503'8052","reported_seq":8224,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-09T17:42:37.892672+0000","last_change":"2024-02-09T17:42:35.239915+0000","last_active":"2024-02-09T17:42:37.892672+0000","last_peered":"2024-02-09T17:42:37.892672+0000","last_clean":"2024-02-09T17:42:37.892672+0000","last_became_active":"2024-02-09T17:40:48.779925+0000","last_became_peered":"2024-02-09T17:40:48.779925+0000","last_unstale":"2024-02-09T17:42:37.892672+0000","last_undegraded":"2024-02-09T17:42:37.892672+0000","last_fullsized":"2024-02-09T17:42:37.892672+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'7928","last_scrub_stamp":"2024-02-09T17:42:35.239746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:42:35.239746+0000","objects_scrubbed":7892,"log_size":8052,"log_dups_size":0,"ondisk_log_size":8052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:54.830845+0000","scrub_duration":1665,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":509739008,"num_objects":7778,"num_object_clones":0,"num_object_copies":23334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8052,"num_write_kb":506560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,8],"acting":[10,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"63'139","reported_seq":1315,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-09T17:42:34.012535+0000","last_change":"2024-02-09T17:41:14.521470+0000","last_active":"2024-02-09T17:42:34.012535+0000","last_peered":"2024-02-09T17:42:34.012535+0000","last_clean":"2024-02-09T17:42:34.012535+0000","last_became_active":"2024-02-09T17:41:14.521079+0000","last_became_peered":"2024-02-09T17:41:14.521079+0000","last_unstale":"2024-02-09T17:42:34.012535+0000","last_undegraded":"2024-02-09T17:42:34.012535+0000","last_fullsized":"2024-02-09T17:42:34.012535+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:10.115197+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:41:10.115197+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"503'8195","reported_seq":8367,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-09T17:42:39.156650+0000","last_change":"2024-02-09T17:42:31.833863+0000","last_active":"2024-02-09T17:42:39.156650+0000","last_peered":"2024-02-09T17:42:39.156650+0000","last_clean":"2024-02-09T17:42:39.156650+0000","last_became_active":"2024-02-09T17:40:48.783218+0000","last_became_peered":"2024-02-09T17:40:48.783218+0000","last_unstale":"2024-02-09T17:42:39.156650+0000","last_undegraded":"2024-02-09T17:42:39.156650+0000","last_fullsized":"2024-02-09T17:42:39.156650+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'7820","last_scrub_stamp":"2024-02-09T17:42:31.833770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:42:31.833770+0000","objects_scrubbed":7802,"log_size":8195,"log_dups_size":0,"ondisk_log_size":8195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:44.391158+0000","scrub_duration":1362,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":506265600,"num_objects":7725,"num_object_clones":0,"num_object_copies":23175,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8195,"num_write_kb":509440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"63'139","reported_seq":1053,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-09T17:42:34.012559+0000","last_change":"2024-02-09T17:41:40.216180+0000","last_active":"2024-02-09T17:42:34.012559+0000","last_peered":"2024-02-09T17:42:34.012559+0000","last_clean":"2024-02-09T17:42:34.012559+0000","last_became_active":"2024-02-09T17:40:22.484952+0000","last_became_peered":"2024-02-09T17:40:22.484952+0000","last_unstale":"2024-02-09T17:42:34.012559+0000","last_undegraded":"2024-02-09T17:42:34.012559+0000","last_fullsized":"2024-02-09T17:42:34.012559+0000","mapping_epoch":454,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":455,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:40.216046+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:41:40.216046+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:02.783875+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"503'8179","reported_seq":8339,"reported_epoch":503,"state":"active+clean+scrubbing","last_fresh":"2024-02-09T17:42:39.167585+0000","last_change":"2024-02-09T17:42:36.477820+0000","last_active":"2024-02-09T17:42:39.167585+0000","last_peered":"2024-02-09T17:42:39.167585+0000","last_clean":"2024-02-09T17:42:39.167585+0000","last_became_active":"2024-02-09T17:40:48.781472+0000","last_became_peered":"2024-02-09T17:40:48.781472+0000","last_unstale":"2024-02-09T17:42:39.167585+0000","last_undegraded":"2024-02-09T17:42:39.167585+0000","last_fullsized":"2024-02-09T17:42:39.167585+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:40:47.763277+0000","objects_scrubbed":6874,"log_size":8179,"log_dups_size":0,"ondisk_log_size":8179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 2s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":509149184,"num_objects":7769,"num_object_clones":0,"num_object_copies":23307,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8179,"num_write_kb":510336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"63'139","reported_seq":507,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-09T17:42:38.047835+0000","last_change":"2024-02-09T17:42:38.047873+0000","last_active":"2024-02-09T17:42:38.047835+0000","last_peered":"2024-02-09T17:42:38.047835+0000","last_clean":"2024-02-09T17:42:38.047835+0000","last_became_active":"2024-02-09T17:40:13.501711+0000","last_became_peered":"2024-02-09T17:40:13.501711+0000","last_unstale":"2024-02-09T17:42:38.047835+0000","last_undegraded":"2024-02-09T17:42:38.047835+0000","last_fullsized":"2024-02-09T17:42:38.047835+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:42:38.047749+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:38:27.045816+0000","last_clean_scrub_stamp":"2024-02-09T17:42:38.047749+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:54.530419+0000","scrub_duration":149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"503'8245","reported_seq":8416,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-09T17:42:38.886108+0000","last_change":"2024-02-09T17:42:27.680013+0000","last_active":"2024-02-09T17:42:38.886108+0000","last_peered":"2024-02-09T17:42:38.886108+0000","last_clean":"2024-02-09T17:42:38.886108+0000","last_became_active":"2024-02-09T17:40:48.780579+0000","last_became_peered":"2024-02-09T17:40:48.780579+0000","last_unstale":"2024-02-09T17:42:38.886108+0000","last_undegraded":"2024-02-09T17:42:38.886108+0000","last_fullsized":"2024-02-09T17:42:38.886108+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'7631","last_scrub_stamp":"2024-02-09T17:42:27.679861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:42:27.679861+0000","objects_scrubbed":7571,"log_size":8245,"log_dups_size":0,"ondisk_log_size":8245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:54.590985+0000","scrub_duration":1709,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":509149184,"num_objects":7769,"num_object_clones":0,"num_object_copies":23307,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8245,"num_write_kb":512448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"63'139","reported_seq":823,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-09T17:42:37.551557+0000","last_change":"2024-02-09T17:42:30.576428+0000","last_active":"2024-02-09T17:42:37.551557+0000","last_peered":"2024-02-09T17:42:37.551557+0000","last_clean":"2024-02-09T17:42:37.551557+0000","last_became_active":"2024-02-09T17:34:01.342607+0000","last_became_peered":"2024-02-09T17:34:01.342607+0000","last_unstale":"2024-02-09T17:42:37.551557+0000","last_undegraded":"2024-02-09T17:42:37.551557+0000","last_fullsized":"2024-02-09T17:42:37.551557+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:42:30.576340+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:42:30.576340+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:33.730137+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.2","version":"503'8514","reported_seq":8690,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-09T17:42:40.875521+0000","last_change":"2024-02-09T17:42:35.385816+0000","last_active":"2024-02-09T17:42:40.875521+0000","last_peered":"2024-02-09T17:42:40.875521+0000","last_clean":"2024-02-09T17:42:40.875521+0000","last_became_active":"2024-02-09T17:40:48.781532+0000","last_became_peered":"2024-02-09T17:40:48.781532+0000","last_unstale":"2024-02-09T17:42:40.875521+0000","last_undegraded":"2024-02-09T17:42:40.875521+0000","last_fullsized":"2024-02-09T17:42:40.875521+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'8226","last_scrub_stamp":"2024-02-09T17:42:35.385605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:42:35.385605+0000","objects_scrubbed":8178,"log_size":8514,"log_dups_size":0,"ondisk_log_size":8514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:49.651467+0000","scrub_duration":1906,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":516816896,"num_objects":7886,"num_object_clones":0,"num_object_copies":23658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8514,"num_write_kb":524800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"63'139","reported_seq":814,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-09T17:42:37.551669+0000","last_change":"2024-02-09T17:41:44.649428+0000","last_active":"2024-02-09T17:42:37.551669+0000","last_peered":"2024-02-09T17:42:37.551669+0000","last_clean":"2024-02-09T17:42:37.551669+0000","last_became_active":"2024-02-09T17:33:00.284702+0000","last_became_peered":"2024-02-09T17:33:00.284702+0000","last_unstale":"2024-02-09T17:42:37.551669+0000","last_undegraded":"2024-02-09T17:42:37.551669+0000","last_fullsized":"2024-02-09T17:42:37.551669+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:44.649313+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:40:01.416303+0000","last_clean_scrub_stamp":"2024-02-09T17:41:44.649313+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:11.533734+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.3","version":"503'8306","reported_seq":8454,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-09T17:42:40.921454+0000","last_change":"2024-02-09T17:41:55.628782+0000","last_active":"2024-02-09T17:42:40.921454+0000","last_peered":"2024-02-09T17:42:40.921454+0000","last_clean":"2024-02-09T17:42:40.921454+0000","last_became_active":"2024-02-09T17:40:48.783369+0000","last_became_peered":"2024-02-09T17:40:48.783369+0000","last_unstale":"2024-02-09T17:42:40.921454+0000","last_undegraded":"2024-02-09T17:42:40.921454+0000","last_fullsized":"2024-02-09T17:42:40.921454+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"493'5422","last_scrub_stamp":"2024-02-09T17:41:55.628642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:41:55.628642+0000","objects_scrubbed":5380,"log_size":8306,"log_dups_size":0,"ondisk_log_size":8306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:59.171152+0000","scrub_duration":1305,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":500170752,"num_objects":7632,"num_object_clones":0,"num_object_copies":22896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8306,"num_write_kb":510016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"63'139","reported_seq":486,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-09T17:42:37.555662+0000","last_change":"2024-02-09T17:41:52.484254+0000","last_active":"2024-02-09T17:42:37.555662+0000","last_peered":"2024-02-09T17:42:37.555662+0000","last_clean":"2024-02-09T17:42:37.555662+0000","last_became_active":"2024-02-09T17:40:13.497710+0000","last_became_peered":"2024-02-09T17:40:13.497710+0000","last_unstale":"2024-02-09T17:42:37.555662+0000","last_undegraded":"2024-02-09T17:42:37.555662+0000","last_fullsized":"2024-02-09T17:42:37.555662+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:52.484109+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:41:52.484109+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:17.603694+0000","scrub_duration":125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"503'8353","reported_seq":8527,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-09T17:42:37.889298+0000","last_change":"2024-02-09T17:42:30.704487+0000","last_active":"2024-02-09T17:42:37.889298+0000","last_peered":"2024-02-09T17:42:37.889298+0000","last_clean":"2024-02-09T17:42:37.889298+0000","last_became_active":"2024-02-09T17:40:48.780290+0000","last_became_peered":"2024-02-09T17:40:48.780290+0000","last_unstale":"2024-02-09T17:42:37.889298+0000","last_undegraded":"2024-02-09T17:42:37.889298+0000","last_fullsized":"2024-02-09T17:42:37.889298+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'8033","last_scrub_stamp":"2024-02-09T17:42:30.704387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:42:30.704387+0000","objects_scrubbed":7963,"log_size":8353,"log_dups_size":0,"ondisk_log_size":8353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:40.575044+0000","scrub_duration":2172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528416768,"num_objects":8063,"num_object_clones":0,"num_object_copies":24189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8353,"num_write_kb":525312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"63'139","reported_seq":905,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-09T17:42:37.563335+0000","last_change":"2024-02-09T17:42:31.121705+0000","last_active":"2024-02-09T17:42:37.563335+0000","last_peered":"2024-02-09T17:42:37.563335+0000","last_clean":"2024-02-09T17:42:37.563335+0000","last_became_active":"2024-02-09T17:33:00.284924+0000","last_became_peered":"2024-02-09T17:33:00.284924+0000","last_unstale":"2024-02-09T17:42:37.563335+0000","last_undegraded":"2024-02-09T17:42:37.563335+0000","last_fullsized":"2024-02-09T17:42:37.563335+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:42:31.121560+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:42:31.121560+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:53.381458+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.1","version":"503'8248","reported_seq":8421,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-09T17:42:40.683814+0000","last_change":"2024-02-09T17:42:37.561624+0000","last_active":"2024-02-09T17:42:40.683814+0000","last_peered":"2024-02-09T17:42:40.683814+0000","last_clean":"2024-02-09T17:42:40.683814+0000","last_became_active":"2024-02-09T17:40:49.660617+0000","last_became_peered":"2024-02-09T17:40:49.660617+0000","last_unstale":"2024-02-09T17:42:40.683814+0000","last_undegraded":"2024-02-09T17:42:40.683814+0000","last_fullsized":"2024-02-09T17:42:40.683814+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'8068","last_scrub_stamp":"2024-02-09T17:42:37.561384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:42:37.561384+0000","objects_scrubbed":7885,"log_size":8248,"log_dups_size":0,"ondisk_log_size":8248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:03.679026+0000","scrub_duration":2520,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":501219328,"num_objects":7648,"num_object_clones":0,"num_object_copies":22944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8248,"num_write_kb":508672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"63'139","reported_seq":858,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-09T17:42:37.551649+0000","last_change":"2024-02-09T17:42:34.665192+0000","last_active":"2024-02-09T17:42:37.551649+0000","last_peered":"2024-02-09T17:42:37.551649+0000","last_clean":"2024-02-09T17:42:37.551649+0000","last_became_active":"2024-02-09T17:41:14.521098+0000","last_became_peered":"2024-02-09T17:41:14.521098+0000","last_unstale":"2024-02-09T17:42:37.551649+0000","last_undegraded":"2024-02-09T17:42:37.551649+0000","last_fullsized":"2024-02-09T17:42:37.551649+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:42:34.665067+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:42:34.665067+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:58.504994+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11],"acting":[10,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.e","version":"502'8171","reported_seq":8317,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-09T17:42:36.526882+0000","last_change":"2024-02-09T17:41:54.022262+0000","last_active":"2024-02-09T17:42:36.526882+0000","last_peered":"2024-02-09T17:42:36.526882+0000","last_clean":"2024-02-09T17:42:36.526882+0000","last_became_active":"2024-02-09T17:40:48.779366+0000","last_became_peered":"2024-02-09T17:40:48.779366+0000","last_unstale":"2024-02-09T17:42:36.526882+0000","last_undegraded":"2024-02-09T17:42:36.526882+0000","last_fullsized":"2024-02-09T17:42:36.526882+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"493'5450","last_scrub_stamp":"2024-02-09T17:41:54.019230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:41:54.019230+0000","objects_scrubbed":5409,"log_size":8171,"log_dups_size":0,"ondisk_log_size":8171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:19.376786+0000","scrub_duration":1190,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":524484608,"num_objects":8003,"num_object_clones":0,"num_object_copies":24009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8171,"num_write_kb":517568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"63'139","reported_seq":805,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-09T17:42:34.005625+0000","last_change":"2024-02-09T17:41:41.681560+0000","last_active":"2024-02-09T17:42:34.005625+0000","last_peered":"2024-02-09T17:42:34.005625+0000","last_clean":"2024-02-09T17:42:34.005625+0000","last_became_active":"2024-02-09T17:32:35.418200+0000","last_became_peered":"2024-02-09T17:32:35.418200+0000","last_unstale":"2024-02-09T17:42:34.005625+0000","last_undegraded":"2024-02-09T17:42:34.005625+0000","last_fullsized":"2024-02-09T17:42:34.005625+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:41.681434+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:41:41.681434+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:51.414785+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.f","version":"503'8249","reported_seq":10796,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-09T17:42:39.156482+0000","last_change":"2024-02-09T17:42:38.866203+0000","last_active":"2024-02-09T17:42:39.156482+0000","last_peered":"2024-02-09T17:42:39.156482+0000","last_clean":"2024-02-09T17:42:39.156482+0000","last_became_active":"2024-02-09T17:41:20.876180+0000","last_became_peered":"2024-02-09T17:41:20.876180+0000","last_unstale":"2024-02-09T17:42:39.156482+0000","last_undegraded":"2024-02-09T17:42:39.156482+0000","last_fullsized":"2024-02-09T17:42:39.156482+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"503'8226","last_scrub_stamp":"2024-02-09T17:42:38.866022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:42:38.866022+0000","objects_scrubbed":7951,"log_size":8249,"log_dups_size":0,"ondisk_log_size":8249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:52.644777+0000","scrub_duration":3408,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":513998848,"num_objects":7843,"num_object_clones":0,"num_object_copies":23529,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8249,"num_write_kb":514944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1182,"num_bytes_recovered":77463552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,11],"acting":[5,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"63'139","reported_seq":649,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-09T17:42:37.563424+0000","last_change":"2024-02-09T17:41:46.320889+0000","last_active":"2024-02-09T17:42:37.563424+0000","last_peered":"2024-02-09T17:42:37.563424+0000","last_clean":"2024-02-09T17:42:37.563424+0000","last_became_active":"2024-02-09T17:41:14.522410+0000","last_became_peered":"2024-02-09T17:41:14.522410+0000","last_unstale":"2024-02-09T17:42:37.563424+0000","last_undegraded":"2024-02-09T17:42:37.563424+0000","last_fullsized":"2024-02-09T17:42:37.563424+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:46.320807+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:41:46.320807+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:07.977725+0000","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"503'8307","reported_seq":8461,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-09T17:42:39.676334+0000","last_change":"2024-02-09T17:42:00.784129+0000","last_active":"2024-02-09T17:42:39.676334+0000","last_peered":"2024-02-09T17:42:39.676334+0000","last_clean":"2024-02-09T17:42:39.676334+0000","last_became_active":"2024-02-09T17:40:48.781384+0000","last_became_peered":"2024-02-09T17:40:48.781384+0000","last_unstale":"2024-02-09T17:42:39.676334+0000","last_undegraded":"2024-02-09T17:42:39.676334+0000","last_fullsized":"2024-02-09T17:42:39.676334+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"495'5883","last_scrub_stamp":"2024-02-09T17:42:00.784027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:42:00.784027+0000","objects_scrubbed":5858,"log_size":8307,"log_dups_size":0,"ondisk_log_size":8307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:15.646893+0000","scrub_duration":1061,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":513998848,"num_objects":7843,"num_object_clones":0,"num_object_copies":23529,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8307,"num_write_kb":516800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"63'139","reported_seq":864,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-09T17:42:37.559459+0000","last_change":"2024-02-09T17:41:55.371903+0000","last_active":"2024-02-09T17:42:37.559459+0000","last_peered":"2024-02-09T17:42:37.559459+0000","last_clean":"2024-02-09T17:42:37.559459+0000","last_became_active":"2024-02-09T17:39:37.460615+0000","last_became_peered":"2024-02-09T17:39:37.460615+0000","last_unstale":"2024-02-09T17:42:37.559459+0000","last_undegraded":"2024-02-09T17:42:37.559459+0000","last_fullsized":"2024-02-09T17:42:37.559459+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:55.371768+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:41:55.371768+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:56.313752+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"503'8535","reported_seq":11187,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-09T17:42:40.323444+0000","last_change":"2024-02-09T17:41:54.334501+0000","last_active":"2024-02-09T17:42:40.323444+0000","last_peered":"2024-02-09T17:42:40.323444+0000","last_clean":"2024-02-09T17:42:40.323444+0000","last_became_active":"2024-02-09T17:41:21.499187+0000","last_became_peered":"2024-02-09T17:41:21.499187+0000","last_unstale":"2024-02-09T17:42:40.323444+0000","last_undegraded":"2024-02-09T17:42:40.323444+0000","last_fullsized":"2024-02-09T17:42:40.323444+0000","mapping_epoch":481,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"493'5534","last_scrub_stamp":"2024-02-09T17:41:54.334404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:41:54.334404+0000","objects_scrubbed":5491,"log_size":8535,"log_dups_size":0,"ondisk_log_size":8535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:03.963347+0000","scrub_duration":1279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":518258712,"num_objects":7909,"num_object_clones":0,"num_object_copies":23727,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7909,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8535,"num_write_kb":526145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1248,"num_bytes_recovered":81788928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"63'139","reported_seq":1308,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-09T17:42:35.605546+0000","last_change":"2024-02-09T17:42:35.605590+0000","last_active":"2024-02-09T17:42:35.605546+0000","last_peered":"2024-02-09T17:42:35.605546+0000","last_clean":"2024-02-09T17:42:35.605546+0000","last_became_active":"2024-02-09T17:39:17.456739+0000","last_became_peered":"2024-02-09T17:39:17.456739+0000","last_unstale":"2024-02-09T17:42:35.605546+0000","last_undegraded":"2024-02-09T17:42:35.605546+0000","last_fullsized":"2024-02-09T17:42:35.605546+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:42:35.605475+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:30.193194+0000","last_clean_scrub_stamp":"2024-02-09T17:42:35.605475+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:49.365305+0000","scrub_duration":222,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"502'8079","reported_seq":8250,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-09T17:42:36.946751+0000","last_change":"2024-02-09T17:42:33.501160+0000","last_active":"2024-02-09T17:42:36.946751+0000","last_peered":"2024-02-09T17:42:36.946751+0000","last_clean":"2024-02-09T17:42:36.946751+0000","last_became_active":"2024-02-09T17:40:48.781591+0000","last_became_peered":"2024-02-09T17:40:48.781591+0000","last_unstale":"2024-02-09T17:42:36.946751+0000","last_undegraded":"2024-02-09T17:42:36.946751+0000","last_fullsized":"2024-02-09T17:42:36.946751+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'7912","last_scrub_stamp":"2024-02-09T17:42:33.501032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:42:33.501032+0000","objects_scrubbed":7852,"log_size":8079,"log_dups_size":0,"ondisk_log_size":8079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:39.582111+0000","scrub_duration":1903,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":516489216,"num_objects":7881,"num_object_clones":0,"num_object_copies":23643,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8079,"num_write_kb":510720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,4],"acting":[8,10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.e","version":"63'139","reported_seq":784,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-09T17:42:40.250714+0000","last_change":"2024-02-09T17:42:40.250751+0000","last_active":"2024-02-09T17:42:40.250714+0000","last_peered":"2024-02-09T17:42:40.250714+0000","last_clean":"2024-02-09T17:42:40.250714+0000","last_became_active":"2024-02-09T17:33:05.309824+0000","last_became_peered":"2024-02-09T17:33:05.309824+0000","last_unstale":"2024-02-09T17:42:40.250714+0000","last_undegraded":"2024-02-09T17:42:40.250714+0000","last_fullsized":"2024-02-09T17:42:40.250714+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:42:40.250649+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:42:40.250649+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:53.976568+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"502'8117","reported_seq":8287,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-09T17:42:36.524384+0000","last_change":"2024-02-09T17:42:30.664783+0000","last_active":"2024-02-09T17:42:36.524384+0000","last_peered":"2024-02-09T17:42:36.524384+0000","last_clean":"2024-02-09T17:42:36.524384+0000","last_became_active":"2024-02-09T17:40:48.783235+0000","last_became_peered":"2024-02-09T17:40:48.783235+0000","last_unstale":"2024-02-09T17:42:36.524384+0000","last_undegraded":"2024-02-09T17:42:36.524384+0000","last_fullsized":"2024-02-09T17:42:36.524384+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'7838","last_scrub_stamp":"2024-02-09T17:42:30.664657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:42:30.664657+0000","objects_scrubbed":7804,"log_size":8117,"log_dups_size":0,"ondisk_log_size":8117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:43.843631+0000","scrub_duration":1694,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":522256384,"num_objects":7969,"num_object_clones":0,"num_object_copies":23907,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8117,"num_write_kb":514752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.f","version":"63'139","reported_seq":842,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-09T17:42:37.555767+0000","last_change":"2024-02-09T17:42:29.589223+0000","last_active":"2024-02-09T17:42:37.555767+0000","last_peered":"2024-02-09T17:42:37.555767+0000","last_clean":"2024-02-09T17:42:37.555767+0000","last_became_active":"2024-02-09T17:33:05.316100+0000","last_became_peered":"2024-02-09T17:33:05.316100+0000","last_unstale":"2024-02-09T17:42:37.555767+0000","last_undegraded":"2024-02-09T17:42:37.555767+0000","last_fullsized":"2024-02-09T17:42:37.555767+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:42:29.589083+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:36:22.364130+0000","last_clean_scrub_stamp":"2024-02-09T17:42:29.589083+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:39.671033+0000","scrub_duration":136,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"503'8433","reported_seq":11008,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-09T17:42:40.337025+0000","last_change":"2024-02-09T17:42:33.997317+0000","last_active":"2024-02-09T17:42:40.337025+0000","last_peered":"2024-02-09T17:42:40.337025+0000","last_clean":"2024-02-09T17:42:40.337025+0000","last_became_active":"2024-02-09T17:41:20.876434+0000","last_became_peered":"2024-02-09T17:41:20.876434+0000","last_unstale":"2024-02-09T17:42:40.337025+0000","last_undegraded":"2024-02-09T17:42:40.337025+0000","last_fullsized":"2024-02-09T17:42:40.337025+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'8076","last_scrub_stamp":"2024-02-09T17:42:33.997206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:42:33.997206+0000","objects_scrubbed":8030,"log_size":8433,"log_dups_size":0,"ondisk_log_size":8433,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:48.378624+0000","scrub_duration":1898,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":512425984,"num_objects":7819,"num_object_clones":0,"num_object_copies":23457,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8433,"num_write_kb":520064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1195,"num_bytes_recovered":78315520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"63'139","reported_seq":961,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-09T17:42:34.014806+0000","last_change":"2024-02-09T17:41:43.719325+0000","last_active":"2024-02-09T17:42:34.014806+0000","last_peered":"2024-02-09T17:42:34.014806+0000","last_clean":"2024-02-09T17:42:34.014806+0000","last_became_active":"2024-02-09T17:39:37.471954+0000","last_became_peered":"2024-02-09T17:39:37.471954+0000","last_unstale":"2024-02-09T17:42:34.014806+0000","last_undegraded":"2024-02-09T17:42:34.014806+0000","last_fullsized":"2024-02-09T17:42:34.014806+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:43.719190+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:23.912504+0000","last_clean_scrub_stamp":"2024-02-09T17:41:43.719190+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:10.334978+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"502'8264","reported_seq":8409,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-09T17:42:37.203468+0000","last_change":"2024-02-09T17:41:52.315160+0000","last_active":"2024-02-09T17:42:37.203468+0000","last_peered":"2024-02-09T17:42:37.203468+0000","last_clean":"2024-02-09T17:42:37.203468+0000","last_became_active":"2024-02-09T17:40:49.436285+0000","last_became_peered":"2024-02-09T17:40:49.436285+0000","last_unstale":"2024-02-09T17:42:37.203468+0000","last_undegraded":"2024-02-09T17:42:37.203468+0000","last_fullsized":"2024-02-09T17:42:37.203468+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"492'5271","last_scrub_stamp":"2024-02-09T17:41:52.314972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:41:52.314972+0000","objects_scrubbed":5238,"log_size":8264,"log_dups_size":0,"ondisk_log_size":8264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:03.342519+0000","scrub_duration":1131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":525860864,"num_objects":8024,"num_object_clones":0,"num_object_copies":24072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8264,"num_write_kb":521216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"63'139","reported_seq":374,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-09T17:42:34.015861+0000","last_change":"2024-02-09T17:42:31.580840+0000","last_active":"2024-02-09T17:42:34.015861+0000","last_peered":"2024-02-09T17:42:34.015861+0000","last_clean":"2024-02-09T17:42:34.015861+0000","last_became_active":"2024-02-09T17:40:22.485257+0000","last_became_peered":"2024-02-09T17:40:22.485257+0000","last_unstale":"2024-02-09T17:42:34.015861+0000","last_undegraded":"2024-02-09T17:42:34.015861+0000","last_fullsized":"2024-02-09T17:42:34.015861+0000","mapping_epoch":454,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":455,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:42:31.580611+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:42:31.580611+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:41.436655+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"63'139","reported_seq":784,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-09T17:42:37.559342+0000","last_change":"2024-02-09T17:41:09.323335+0000","last_active":"2024-02-09T17:42:37.559342+0000","last_peered":"2024-02-09T17:42:37.559342+0000","last_clean":"2024-02-09T17:42:37.559342+0000","last_became_active":"2024-02-09T17:33:05.310104+0000","last_became_peered":"2024-02-09T17:33:05.310104+0000","last_unstale":"2024-02-09T17:42:37.559342+0000","last_undegraded":"2024-02-09T17:42:37.559342+0000","last_fullsized":"2024-02-09T17:42:37.559342+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:09.323236+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:41:09.323236+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:37.737049+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"63'139","reported_seq":797,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-09T17:42:34.008488+0000","last_change":"2024-02-09T17:42:33.646446+0000","last_active":"2024-02-09T17:42:34.008488+0000","last_peered":"2024-02-09T17:42:34.008488+0000","last_clean":"2024-02-09T17:42:34.008488+0000","last_became_active":"2024-02-09T17:34:01.353171+0000","last_became_peered":"2024-02-09T17:34:01.353171+0000","last_unstale":"2024-02-09T17:42:34.008488+0000","last_undegraded":"2024-02-09T17:42:34.008488+0000","last_fullsized":"2024-02-09T17:42:34.008488+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:42:33.646309+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:42:33.646309+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:57.573164+0000","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.12","version":"63'139","reported_seq":1311,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-09T17:42:37.560274+0000","last_change":"2024-02-09T17:42:31.533609+0000","last_active":"2024-02-09T17:42:37.560274+0000","last_peered":"2024-02-09T17:42:37.560274+0000","last_clean":"2024-02-09T17:42:37.560274+0000","last_became_active":"2024-02-09T17:40:13.499071+0000","last_became_peered":"2024-02-09T17:40:13.499071+0000","last_unstale":"2024-02-09T17:42:37.560274+0000","last_undegraded":"2024-02-09T17:42:37.560274+0000","last_fullsized":"2024-02-09T17:42:37.560274+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:42:31.533484+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:42:31.533484+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:36.556705+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"63'139","reported_seq":502,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-09T17:42:37.550889+0000","last_change":"2024-02-09T17:41:14.521999+0000","last_active":"2024-02-09T17:42:37.550889+0000","last_peered":"2024-02-09T17:42:37.550889+0000","last_clean":"2024-02-09T17:42:37.550889+0000","last_became_active":"2024-02-09T17:41:14.521672+0000","last_became_peered":"2024-02-09T17:41:14.521672+0000","last_unstale":"2024-02-09T17:42:37.550889+0000","last_undegraded":"2024-02-09T17:42:37.550889+0000","last_fullsized":"2024-02-09T17:42:37.550889+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:03.023657+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:41:03.023657+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":8228700184,"num_objects":125561,"num_object_clones":0,"num_object_copies":376683,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":125561,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132247,"num_write_kb":8249793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3625,"num_bytes_recovered":237568000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24686833664,"data_stored":24686821448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132247,"ondisk_log_size":132247,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":931424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2919,"ondisk_log_size":2919,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056884,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2529192,"kb_used_data":2010536,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":91215960,"statfs":{"total":95995035648,"available":93405143040,"internally_reserved":0,"allocated":2058788864,"data_stored":2058497142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":530905540},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:42:28 2024","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":1.5069999999999999,"15min":1.3029999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.216},"max":{"1min":3.2280000000000002,"5min":9.5709999999999997,"15min":14.913},"last":0.315},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.321,"15min":1.3089999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.23000000000000001},"max":{"1min":5.367,"5min":9.8219999999999992,"15min":14.869},"last":0.34499999999999997}]},{"osd":1,"last update":"Fri Feb 9 17:42:28 2024","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":1.468,"15min":1.579},"min":{"1min":0.28999999999999998,"5min":0.24299999999999999,"15min":0.22700000000000001},"max":{"1min":5.415,"5min":9.7710000000000008,"15min":32.957999999999998},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.82699999999999996,"5min":1.613,"15min":1.7210000000000001},"min":{"1min":0.31900000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":3.3170000000000002,"5min":9.718,"15min":32.880000000000003},"last":0.432}]},{"osd":2,"last update":"Fri Feb 9 17:31:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.415,"15min":1.6859999999999999},"min":{"1min":0.378,"5min":0.247,"15min":0.247},"max":{"1min":4.4189999999999996,"5min":9.4719999999999995,"15min":9.4719999999999995},"last":0.35399999999999998},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.5149999999999999,"15min":1.4399999999999999},"min":{"1min":0.378,"5min":0.252,"15min":0.252},"max":{"1min":4.8579999999999997,"5min":7.96,"15min":7.96},"last":0.60399999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:42:28 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":1.24,"15min":1.552},"min":{"1min":0.29799999999999999,"5min":0.27100000000000002,"15min":0.247},"max":{"1min":5.4539999999999997,"5min":9.5299999999999994,"15min":21.315999999999999},"last":0.44800000000000001},{"interface":"front","average":{"1min":1.175,"5min":1.734,"15min":1.6879999999999999},"min":{"1min":0.38500000000000001,"5min":0.29799999999999999,"15min":0.26400000000000001},"max":{"1min":5.5499999999999998,"5min":12.706,"15min":18.087},"last":0.48099999999999998}]},{"osd":4,"last update":"Fri Feb 9 17:42:28 2024","interfaces":[{"interface":"back","average":{"1min":2.1110000000000002,"5min":1.45,"15min":1.554},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":6.1790000000000003,"5min":21.030999999999999,"15min":21.030999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":1.498,"5min":1.411,"15min":1.3400000000000001},"min":{"1min":0.32600000000000001,"5min":0.309,"15min":0.27700000000000002},"max":{"1min":6.9509999999999996,"5min":20.876999999999999,"15min":20.876999999999999},"last":0.46300000000000002}]},{"osd":5,"last update":"Fri Feb 9 17:42:28 2024","interfaces":[{"interface":"back","average":{"1min":1.6679999999999999,"5min":1.157,"15min":1.3819999999999999},"min":{"1min":0.36199999999999999,"5min":0.316,"15min":0.28899999999999998},"max":{"1min":5.4800000000000004,"5min":6.4630000000000001,"15min":13.946999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":1.75,"5min":1.3109999999999999,"15min":1.4810000000000001},"min":{"1min":0.317,"5min":0.311,"15min":0.30199999999999999},"max":{"1min":5.4610000000000003,"5min":7.1349999999999998,"15min":13.414},"last":2.0569999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:42:28 2024","interfaces":[{"interface":"back","average":{"1min":2.0129999999999999,"5min":1.4750000000000001,"15min":1.4810000000000001},"min":{"1min":0.442,"5min":0.39200000000000002,"15min":0.29899999999999999},"max":{"1min":6.8330000000000002,"5min":9.4779999999999998,"15min":12.798999999999999},"last":0.38600000000000001},{"interface":"front","average":{"1min":2.7970000000000002,"5min":1.548,"15min":1.536},"min":{"1min":0.46500000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":6.9249999999999998,"5min":9.2420000000000009,"15min":10.782999999999999},"last":1.7}]},{"osd":7,"last update":"Fri Feb 9 17:42:28 2024","interfaces":[{"interface":"back","average":{"1min":1.875,"5min":1.5189999999999999,"15min":1.659},"min":{"1min":0.41599999999999998,"5min":0.38200000000000001,"15min":0.312},"max":{"1min":7.5830000000000002,"5min":20.489999999999998,"15min":20.489999999999998},"last":2.5859999999999999},{"interface":"front","average":{"1min":2.0750000000000002,"5min":1.504,"15min":1.647},"min":{"1min":0.52200000000000002,"5min":0.35299999999999998,"15min":0.34200000000000003},"max":{"1min":10.276999999999999,"5min":20.712,"15min":20.712},"last":0.57799999999999996}]},{"osd":8,"last update":"Fri Feb 9 17:41:54 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":1.0780000000000001,"15min":1.657},"min":{"1min":0.316,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":4.5739999999999998,"5min":9.2680000000000007,"15min":13.199999999999999},"last":0.621},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.149,"15min":1.448},"min":{"1min":0.40999999999999998,"5min":0.34300000000000003,"15min":0.318},"max":{"1min":6.1219999999999999,"5min":9.2940000000000005,"15min":9.2940000000000005},"last":0.59199999999999997}]},{"osd":9,"last update":"Fri Feb 9 17:42:16 2024","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.887,"15min":1.4570000000000001},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29399999999999998},"max":{"1min":7.3860000000000001,"5min":62.518000000000001,"15min":62.518000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":1.1990000000000001,"5min":1.9379999999999999,"15min":1.371},"min":{"1min":0.44700000000000001,"5min":0.379,"15min":0.34100000000000003},"max":{"1min":5.4089999999999998,"5min":63.241999999999997,"15min":63.241999999999997},"last":0.64400000000000002}]},{"osd":10,"last update":"Fri Feb 9 17:42:28 2024","interfaces":[{"interface":"back","average":{"1min":1.5600000000000001,"5min":1.329,"15min":1.236},"min":{"1min":0.40899999999999997,"5min":0.36799999999999999,"15min":0.33900000000000002},"max":{"1min":9.2460000000000004,"5min":9.3109999999999999,"15min":9.6199999999999992},"last":0.72199999999999998},{"interface":"front","average":{"1min":1.72,"5min":1.417,"15min":1.2789999999999999},"min":{"1min":0.47299999999999998,"5min":0.39100000000000001,"15min":0.33700000000000002},"max":{"1min":7.9329999999999998,"5min":9.593,"15min":12.856},"last":0.51400000000000001}]}]},{"osd":10,"up_from":84,"seq":360777253113,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3393316,"kb_used_data":3046820,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":90351836,"statfs":{"total":95995035648,"available":92520280064,"internally_reserved":0,"allocated":3119943680,"data_stored":3119656030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":354613696},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:42:18 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":1.4019999999999999,"15min":1.6180000000000001},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":3.2149999999999999,"5min":9.343,"15min":9.343},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.67000000000000004,"5min":1.2629999999999999,"15min":1.147},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":3.3159999999999998,"5min":6.3399999999999999,"15min":6.3399999999999999},"last":0.79600000000000004}]},{"osd":1,"last update":"Fri Feb 9 17:41:50 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":2.3460000000000001,"5min":2.3460000000000001,"15min":2.3460000000000001},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":3.5049999999999999,"5min":3.5049999999999999,"15min":3.5049999999999999},"last":0.39300000000000002}]},{"osd":2,"last update":"Fri Feb 9 17:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":1.774,"15min":1.425},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.222},"max":{"1min":2.4140000000000001,"5min":11.384,"15min":13.351000000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.83699999999999997,"5min":1.425,"15min":1.3859999999999999},"min":{"1min":0.32500000000000001,"5min":0.28799999999999998,"15min":0.26100000000000001},"max":{"1min":3.254,"5min":11.465,"15min":11.465},"last":0.84699999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":1.7509999999999999,"15min":1.9490000000000001},"min":{"1min":0.34699999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.5029999999999999,"5min":9.1649999999999991,"15min":9.1649999999999991},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.77900000000000003,"5min":1.8400000000000001,"15min":2.0169999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":2.492,"5min":8.9670000000000005,"15min":8.9670000000000005},"last":0.49099999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:42:05 2024","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":0.97199999999999998,"15min":1.0509999999999999},"min":{"1min":0.36899999999999999,"5min":0.29199999999999998,"15min":0.26100000000000001},"max":{"1min":6.4930000000000003,"5min":6.4930000000000003,"15min":7.1360000000000001},"last":0.89100000000000001},{"interface":"front","average":{"1min":1.1080000000000001,"5min":0.92300000000000004,"15min":1.048},"min":{"1min":0.39300000000000002,"5min":0.31,"15min":0.29999999999999999},"max":{"1min":6.5339999999999998,"5min":6.5339999999999998,"15min":6.8369999999999997},"last":0.91200000000000003}]},{"osd":5,"last update":"Fri Feb 9 17:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":1.085,"15min":1.2370000000000001},"min":{"1min":0.35999999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":2.9460000000000002,"5min":7.5179999999999998,"15min":9.2349999999999994},"last":0.46600000000000003},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.0740000000000001,"15min":1.2490000000000001},"min":{"1min":0.47399999999999998,"5min":0.374,"15min":0.32200000000000001},"max":{"1min":6.569,"5min":6.8940000000000001,"15min":8.6159999999999997},"last":0.86899999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:42:00 2024","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.3049999999999999,"15min":1.274},"min":{"1min":0.42399999999999999,"5min":0.308,"15min":0.29099999999999998},"max":{"1min":4.21,"5min":8.6869999999999994,"15min":8.6869999999999994},"last":4.0679999999999996},{"interface":"front","average":{"1min":1.411,"5min":1.1279999999999999,"15min":1.3799999999999999},"min":{"1min":0.497,"5min":0.38600000000000001,"15min":0.35399999999999998},"max":{"1min":6.6040000000000001,"5min":6.6040000000000001,"15min":9.4350000000000005},"last":1.8919999999999999}]},{"osd":7,"last update":"Fri Feb 9 17:39:52 2024","interfaces":[{"interface":"back","average":{"1min":1.6000000000000001,"5min":1.367,"15min":1.415},"min":{"1min":0.42099999999999999,"5min":0.34200000000000003,"15min":0.312},"max":{"1min":4.0430000000000001,"5min":6.7590000000000003,"15min":17.754000000000001},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.498,"5min":1.4359999999999999,"15min":1.5189999999999999},"min":{"1min":0.47399999999999998,"5min":0.38300000000000001,"15min":0.316},"max":{"1min":5.8170000000000002,"5min":11.308999999999999,"15min":11.308999999999999},"last":0.78800000000000003}]},{"osd":8,"last update":"Fri Feb 9 17:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":1.097,"15min":1.143},"min":{"1min":0.37,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":2.444,"5min":7.4359999999999999,"15min":7.4359999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.998,"5min":1.075,"15min":0.97399999999999998},"min":{"1min":0.41899999999999998,"5min":0.37,"15min":0.32900000000000001},"max":{"1min":4.343,"5min":7.7060000000000004,"15min":7.7060000000000004},"last":0.59599999999999997}]},{"osd":9,"last update":"Fri Feb 9 17:42:00 2024","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":1.2729999999999999,"15min":1.0960000000000001},"min":{"1min":0.36799999999999999,"5min":0.36399999999999999,"15min":0.314},"max":{"1min":4.766,"5min":7.4850000000000003,"15min":8.7240000000000002},"last":0.68100000000000005},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.2370000000000001,"15min":1.1950000000000001},"min":{"1min":0.46000000000000002,"5min":0.441,"15min":0.29799999999999999},"max":{"1min":3.8090000000000002,"5min":7.891,"15min":10.1},"last":0.71099999999999997}]},{"osd":11,"last update":"Fri Feb 9 17:42:21 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":0.98899999999999999,"15min":1.0609999999999999},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.36399999999999999},"max":{"1min":4.9630000000000001,"5min":4.9630000000000001,"15min":8.6430000000000007},"last":0.93500000000000005},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.0469999999999999,"15min":1.1140000000000001},"min":{"1min":0.441,"5min":0.441,"15min":0.377},"max":{"1min":4.9960000000000004,"5min":4.9960000000000004,"15min":9.298},"last":0.95699999999999996}]}]},{"osd":9,"up_from":77,"seq":330712482046,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1961740,"kb_used_data":1533324,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":91783412,"statfs":{"total":95995035648,"available":93986213888,"internally_reserved":0,"allocated":1570123776,"data_stored":1569774731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":438499776},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:41:50 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.9790000000000001,"5min":1.9790000000000001,"15min":1.9790000000000001},"last":7.2199999999999998},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":4.3339999999999996,"5min":4.3339999999999996,"15min":4.3339999999999996},"last":0.56899999999999995}]},{"osd":1,"last update":"Fri Feb 9 17:41:37 2024","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":2.8159999999999998,"15min":1.8500000000000001},"min":{"1min":0.32700000000000001,"5min":0.23499999999999999,"15min":0.23100000000000001},"max":{"1min":9.0449999999999999,"5min":89.515000000000001,"15min":89.515000000000001},"last":6.8769999999999998},{"interface":"front","average":{"1min":0.72599999999999998,"5min":2.895,"15min":1.8100000000000001},"min":{"1min":0.38500000000000001,"5min":0.249,"15min":0.222},"max":{"1min":2.0609999999999999,"5min":89.388000000000005,"15min":89.388000000000005},"last":7.3230000000000004}]},{"osd":2,"last update":"Fri Feb 9 17:40:39 2024","interfaces":[{"interface":"back","average":{"1min":1.742,"5min":1.766,"15min":1.514},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":5.6630000000000003,"5min":14.121,"15min":14.121},"last":0.63400000000000001},{"interface":"front","average":{"1min":1.6910000000000001,"5min":1.52,"15min":1.4870000000000001},"min":{"1min":0.34100000000000003,"5min":0.26100000000000001,"15min":0.25600000000000001},"max":{"1min":5.7030000000000003,"5min":8.8040000000000003,"15min":13.69},"last":0.73899999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:41:37 2024","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":2.1880000000000002,"15min":1.655},"min":{"1min":0.371,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":8.9939999999999998,"5min":15.414999999999999,"15min":40.18},"last":6.915},{"interface":"front","average":{"1min":0.69399999999999995,"5min":2.1760000000000002,"15min":1.8160000000000001},"min":{"1min":0.32000000000000001,"5min":0.317,"15min":0.27900000000000003},"max":{"1min":2.2679999999999998,"5min":15.372,"15min":40.158000000000001},"last":7.2519999999999998}]},{"osd":4,"last update":"Fri Feb 9 17:41:37 2024","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.331,"15min":1.4359999999999999},"min":{"1min":0.40899999999999997,"5min":0.29599999999999999,"15min":0.28399999999999997},"max":{"1min":9.0920000000000005,"5min":27.381,"15min":27.381},"last":6.9409999999999998},{"interface":"front","average":{"1min":0.76100000000000001,"5min":1.419,"15min":1.3440000000000001},"min":{"1min":0.32800000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.571,"5min":27.355,"15min":27.355},"last":7.3010000000000002}]},{"osd":5,"last update":"Fri Feb 9 17:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":1.2090000000000001,"15min":1.3380000000000001},"min":{"1min":0.41499999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.4790000000000001,"5min":8.6110000000000007,"15min":10.689},"last":7.367},{"interface":"front","average":{"1min":0.879,"5min":1.1639999999999999,"15min":1.157},"min":{"1min":0.39300000000000002,"5min":0.32200000000000001,"15min":0.311},"max":{"1min":2.032,"5min":13.978999999999999,"15min":13.978999999999999},"last":0.82199999999999995}]},{"osd":6,"last update":"Fri Feb 9 17:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":1.3080000000000001,"15min":1.379},"min":{"1min":0.41499999999999998,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":2.0659999999999998,"5min":14.069000000000001,"15min":14.069000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.94699999999999995,"5min":1.3380000000000001,"15min":1.419},"min":{"1min":0.45500000000000002,"5min":0.37,"15min":0.36199999999999999},"max":{"1min":2.133,"5min":14.097,"15min":14.097},"last":0.59399999999999997}]},{"osd":7,"last update":"Fri Feb 9 17:41:37 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.3580000000000001,"15min":1.3899999999999999},"min":{"1min":0.45100000000000001,"5min":0.34499999999999997,"15min":0.34300000000000003},"max":{"1min":9.125,"5min":9.125,"15min":9.125},"last":6.9640000000000004},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.403,"15min":1.454},"min":{"1min":0.49299999999999999,"5min":0.39600000000000002,"15min":0.32200000000000001},"max":{"1min":9.157,"5min":9.157,"15min":9.157},"last":6.9880000000000004}]},{"osd":8,"last update":"Fri Feb 9 17:41:57 2024","interfaces":[{"interface":"back","average":{"1min":1.2410000000000001,"5min":1.3680000000000001,"15min":1.2769999999999999},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.34999999999999998},"max":{"1min":4.1299999999999999,"5min":14.045999999999999,"15min":16.209},"last":7.2759999999999998},{"interface":"front","average":{"1min":1.452,"5min":1.444,"15min":1.2969999999999999},"min":{"1min":0.432,"5min":0.41799999999999998,"15min":0.34799999999999998},"max":{"1min":4.6639999999999997,"5min":5.548,"15min":16.263000000000002},"last":2.4820000000000002}]},{"osd":10,"last update":"Fri Feb 9 17:42:06 2024","interfaces":[{"interface":"back","average":{"1min":1.724,"5min":1.3779999999999999,"15min":1.181},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":9.1869999999999994,"5min":9.1869999999999994,"15min":9.5090000000000003},"last":7.0250000000000004},{"interface":"front","average":{"1min":1.129,"5min":1.3979999999999999,"15min":1.147},"min":{"1min":0.40999999999999998,"5min":0.39500000000000002,"15min":0.38200000000000001},"max":{"1min":4.1779999999999999,"5min":9.7569999999999997,"15min":9.7569999999999997},"last":7.3449999999999998}]},{"osd":11,"last update":"Fri Feb 9 17:42:06 2024","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.5660000000000001,"15min":1.4379999999999999},"min":{"1min":0.434,"5min":0.39000000000000001,"15min":0.34699999999999998},"max":{"1min":6.8070000000000004,"5min":14.02,"15min":14.02},"last":0.79500000000000004},{"interface":"front","average":{"1min":1.0029999999999999,"5min":1.28,"15min":1.2330000000000001},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.374},"max":{"1min":3.4460000000000002,"5min":7.29,"15min":10.32},"last":10.760999999999999}]}]},{"osd":8,"up_from":171,"seq":734439407789,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1746780,"kb_used_data":1520916,"kb_used_omap":212,"kb_used_meta":225643,"kb_avail":91998372,"statfs":{"total":95995035648,"available":94206332928,"internally_reserved":0,"allocated":1557417984,"data_stored":1557065387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217579,"internal_metadata":231058965},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":1.0880000000000001,"15min":1.149},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":4.7370000000000001,"5min":8.0920000000000005,"15min":8.0920000000000005},"last":0.85499999999999998},{"interface":"front","average":{"1min":1.024,"5min":1.506,"15min":1.321},"min":{"1min":0.28699999999999998,"5min":0.26200000000000001,"15min":0.23999999999999999},"max":{"1min":4.1360000000000001,"5min":10.256,"15min":10.256},"last":0.94299999999999995}]},{"osd":1,"last update":"Fri Feb 9 17:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":1.5469999999999999,"15min":1.3160000000000001},"min":{"1min":0.25,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.8959999999999999,"5min":7.867,"15min":7.8719999999999999},"last":0.877},{"interface":"front","average":{"1min":0.60199999999999998,"5min":2.0150000000000001,"15min":1.681},"min":{"1min":0.27900000000000003,"5min":0.255,"15min":0.23300000000000001},"max":{"1min":3.2370000000000001,"5min":25.027000000000001,"15min":25.027000000000001},"last":0.377}]},{"osd":2,"last update":"Fri Feb 9 17:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":1.4630000000000001,"15min":1.427},"min":{"1min":0.308,"5min":0.27500000000000002,"15min":0.255},"max":{"1min":0.73099999999999998,"5min":8.5649999999999995,"15min":10.423},"last":0.96599999999999997},{"interface":"front","average":{"1min":0.59399999999999997,"5min":1.516,"15min":1.4610000000000001},"min":{"1min":0.39200000000000002,"5min":0.35899999999999999,"15min":0.313},"max":{"1min":0.871,"5min":8.5980000000000008,"15min":10.504},"last":1.048}]},{"osd":3,"last update":"Fri Feb 9 17:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":1.7270000000000001,"15min":1.7609999999999999},"min":{"1min":0.33000000000000002,"5min":0.29499999999999998,"15min":0.28699999999999998},"max":{"1min":3.2799999999999998,"5min":15.228999999999999,"15min":24.030999999999999},"last":0.44},{"interface":"front","average":{"1min":0.71599999999999997,"5min":2.1259999999999999,"15min":2.0169999999999999},"min":{"1min":0.39300000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":3.343,"5min":14.988,"15min":24.318999999999999},"last":0.504}]},{"osd":4,"last update":"Fri Feb 9 17:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":1.2569999999999999,"15min":1.4239999999999999},"min":{"1min":0.45900000000000002,"5min":0.29099999999999998,"15min":0.27500000000000002},"max":{"1min":3.262,"5min":12.936,"15min":53.060000000000002},"last":0.47599999999999998},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.2130000000000001,"15min":1.569},"min":{"1min":0.46700000000000003,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":3.2959999999999998,"5min":7.46,"15min":53.311999999999998},"last":0.59199999999999997}]},{"osd":6,"last update":"Fri Feb 9 17:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":1.1479999999999999,"15min":1.518},"min":{"1min":0.42799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.4410000000000001,"5min":7.4870000000000001,"15min":37.356999999999999},"last":1.022},{"interface":"front","average":{"1min":0.73199999999999998,"5min":1.157,"15min":1.2609999999999999},"min":{"1min":0.39500000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.5129999999999999,"5min":7.702,"15min":37.25},"last":1.004}]},{"osd":7,"last update":"Fri Feb 9 17:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":1.2849999999999999,"15min":1.544},"min":{"1min":0.39600000000000002,"5min":0.38100000000000001,"15min":0.32800000000000001},"max":{"1min":3.3119999999999998,"5min":8.5120000000000005,"15min":37.585000000000001},"last":0.55600000000000005},{"interface":"front","average":{"1min":1.0269999999999999,"5min":1.3280000000000001,"15min":1.625},"min":{"1min":0.54700000000000004,"5min":0.38500000000000001,"15min":0.312},"max":{"1min":3.327,"5min":8.548,"15min":37.918999999999997},"last":0.57399999999999995}]},{"osd":9,"last update":"Fri Feb 9 17:41:38 2024","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":0.98799999999999999,"15min":1.0940000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":3.9239999999999999,"5min":5.3330000000000002,"15min":9.4900000000000002},"last":1.069},{"interface":"front","average":{"1min":0.81200000000000006,"5min":1.2010000000000001,"15min":1.194},"min":{"1min":0.40400000000000003,"5min":0.376,"15min":0.32000000000000001},"max":{"1min":3.423,"5min":7.4189999999999996,"15min":17.434999999999999},"last":1.1000000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":0.98899999999999999,"15min":1.075},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":3.194,"5min":4.8170000000000002,"15min":10.65},"last":4.4500000000000002},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.274,"15min":1.1359999999999999},"min":{"1min":0.35899999999999999,"5min":0.34799999999999998,"15min":0.318},"max":{"1min":4.093,"5min":11.071,"15min":11.071},"last":1.0880000000000001}]},{"osd":11,"last update":"Fri Feb 9 17:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":1.2629999999999999,"15min":1.3420000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":2.464,"5min":8.5519999999999996,"15min":14.433999999999999},"last":1.1259999999999999},{"interface":"front","average":{"1min":0.97599999999999998,"5min":1.268,"15min":1.3140000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":3.1800000000000002,"5min":7.4359999999999999,"15min":11.622},"last":0.69699999999999995}]}]},{"osd":7,"up_from":57,"seq":244813136135,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2566488,"kb_used_data":2047952,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":91178664,"statfs":{"total":95995035648,"available":93366951936,"internally_reserved":0,"allocated":2097102848,"data_stored":2096749731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":530905546},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:42:21 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":1.3360000000000001,"15min":2.528},"min":{"1min":0.26000000000000001,"5min":0.252,"15min":0.252},"max":{"1min":2.5899999999999999,"5min":14.348000000000001,"15min":36.753},"last":9.1910000000000007},{"interface":"front","average":{"1min":1.4750000000000001,"5min":1.349,"15min":2.714},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.253},"max":{"1min":6.7960000000000003,"5min":8.2289999999999992,"15min":36.918999999999997},"last":9.3309999999999995}]},{"osd":1,"last update":"Fri Feb 9 17:42:21 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":2.1230000000000002,"15min":2.0019999999999998},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":2.6669999999999998,"5min":21.704999999999998,"15min":21.704999999999998},"last":4.6070000000000002},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.9590000000000001,"15min":1.8160000000000001},"min":{"1min":0.33300000000000002,"5min":0.26600000000000001,"15min":0.26500000000000001},"max":{"1min":7.2800000000000002,"5min":21.858000000000001,"15min":21.858000000000001},"last":4.5469999999999997}]},{"osd":2,"last update":"Fri Feb 9 17:39:34 2024","interfaces":[{"interface":"back","average":{"1min":1.8520000000000001,"5min":1.7470000000000001,"15min":1.7030000000000001},"min":{"1min":0.315,"5min":0.28499999999999998,"15min":0.26000000000000001},"max":{"1min":6.6470000000000002,"5min":10.622,"15min":11.324},"last":0.76900000000000002},{"interface":"front","average":{"1min":1.8859999999999999,"5min":1.855,"15min":1.7769999999999999},"min":{"1min":0.34699999999999998,"5min":0.32100000000000001,"15min":0.28299999999999997},"max":{"1min":6.6870000000000003,"5min":19.361999999999998,"15min":19.361999999999998},"last":0.82299999999999995}]},{"osd":3,"last update":"Fri Feb 9 17:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":1.6539999999999999,"15min":1.867},"min":{"1min":0.35899999999999999,"5min":0.307,"15min":0.26900000000000002},"max":{"1min":2.4510000000000001,"5min":7.3899999999999997,"15min":27.719000000000001},"last":9.2300000000000004},{"interface":"front","average":{"1min":0.84599999999999997,"5min":1.627,"15min":1.8600000000000001},"min":{"1min":0.41999999999999998,"5min":0.308,"15min":0.26500000000000001},"max":{"1min":1.861,"5min":11.945,"15min":24.457000000000001},"last":4.6429999999999998}]},{"osd":4,"last update":"Fri Feb 9 17:42:21 2024","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":0.84299999999999997,"15min":0.747},"min":{"1min":0.27700000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":4.5789999999999997,"5min":4.5789999999999997,"15min":4.5789999999999997},"last":0.48799999999999999},{"interface":"front","average":{"1min":2.1509999999999998,"5min":0.91500000000000004,"15min":0.70899999999999996},"min":{"1min":0.312,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":7.9939999999999998,"5min":7.9939999999999998,"15min":7.9939999999999998},"last":9.4160000000000004}]},{"osd":5,"last update":"Fri Feb 9 17:42:21 2024","interfaces":[{"interface":"back","average":{"1min":2.492,"5min":1.018,"15min":0.76800000000000002},"min":{"1min":0.32100000000000001,"5min":0.249,"15min":0.249},"max":{"1min":8.3689999999999998,"5min":8.3689999999999998,"15min":8.3689999999999998},"last":3.641},{"interface":"front","average":{"1min":1.8580000000000001,"5min":0.83299999999999996,"15min":0.61599999999999999},"min":{"1min":0.35999999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":8.0069999999999997,"5min":8.0069999999999997,"15min":8.0069999999999997},"last":4.7199999999999998}]},{"osd":6,"last update":"Fri Feb 9 17:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":1.091,"15min":1.5229999999999999},"min":{"1min":0.36599999999999999,"5min":0.28100000000000003,"15min":0.26000000000000001},"max":{"1min":2.4830000000000001,"5min":8.1929999999999996,"15min":25.965},"last":4.6799999999999997},{"interface":"front","average":{"1min":0.84999999999999998,"5min":1.083,"15min":1.3029999999999999},"min":{"1min":0.44600000000000001,"5min":0.32600000000000001,"15min":0.28699999999999998},"max":{"1min":2.0659999999999998,"5min":7.8810000000000002,"15min":9.4090000000000007},"last":0.442}]},{"osd":8,"last update":"Fri Feb 9 17:41:37 2024","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.1140000000000001,"15min":1.383},"min":{"1min":0.52900000000000003,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":2.7349999999999999,"5min":6.4420000000000002,"15min":9.0809999999999995},"last":9.2680000000000007},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.0309999999999999,"15min":1.28},"min":{"1min":0.48599999999999999,"5min":0.36599999999999999,"15min":0.33300000000000002},"max":{"1min":2.5760000000000001,"5min":6.0339999999999998,"15min":11.481},"last":4.5810000000000004}]},{"osd":9,"last update":"Fri Feb 9 17:42:30 2024","interfaces":[{"interface":"back","average":{"1min":1.8220000000000001,"5min":1.242,"15min":1.577},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":6.7380000000000004,"5min":6.7380000000000004,"15min":41.286999999999999},"last":9.3000000000000007},{"interface":"front","average":{"1min":2.105,"5min":1.373,"15min":1.671},"min":{"1min":0.42099999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":8.0399999999999991,"5min":8.0399999999999991,"15min":41.234999999999999},"last":9.3610000000000007}]},{"osd":10,"last update":"Fri Feb 9 17:41:53 2024","interfaces":[{"interface":"back","average":{"1min":1.871,"5min":1.413,"15min":1.4350000000000001},"min":{"1min":0.68799999999999994,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":9.3130000000000006,"5min":12.050000000000001,"15min":12.050000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":1.673,"5min":1.349,"15min":1.5409999999999999},"min":{"1min":0.63900000000000001,"5min":0.39800000000000002,"15min":0.35599999999999998},"max":{"1min":8.8520000000000003,"5min":8.8520000000000003,"15min":9.109},"last":9.3879999999999999}]},{"osd":11,"last update":"Fri Feb 9 17:42:21 2024","interfaces":[{"interface":"back","average":{"1min":2.0390000000000001,"5min":1.681,"15min":1.77},"min":{"1min":0.42399999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":9.2490000000000006,"5min":21.945,"15min":21.945},"last":0.64700000000000002},{"interface":"front","average":{"1min":1.998,"5min":1.7050000000000001,"15min":1.609},"min":{"1min":0.45500000000000002,"5min":0.40699999999999997,"15min":0.39500000000000002},"max":{"1min":9.3420000000000005,"5min":21.731999999999999,"15min":21.731999999999999},"last":0.67600000000000005}]}]},{"osd":6,"up_from":49,"seq":210453397773,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1892836,"kb_used_data":1500516,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":91852316,"statfs":{"total":95995035648,"available":94056771584,"internally_reserved":0,"allocated":1536528384,"data_stored":1536240734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":401537478},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:42:13 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":1.579,"15min":1.587},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.21299999999999999},"max":{"1min":4.2519999999999998,"5min":9.5739999999999998,"15min":12.544},"last":3.484},{"interface":"front","average":{"1min":0.97899999999999998,"5min":1.5189999999999999,"15min":1.4990000000000001},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.217},"max":{"1min":4.9950000000000001,"5min":9.2520000000000007,"15min":9.2520000000000007},"last":3.5419999999999998}]},{"osd":1,"last update":"Fri Feb 9 17:42:13 2024","interfaces":[{"interface":"back","average":{"1min":1.748,"5min":1.9490000000000001,"15min":1.891},"min":{"1min":0.307,"5min":0.28599999999999998,"15min":0.25900000000000001},"max":{"1min":8.3260000000000005,"5min":8.9689999999999994,"15min":17.762},"last":3.0619999999999998},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.869,"15min":1.903},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":4.5270000000000001,"5min":9.0090000000000003,"15min":17.734999999999999},"last":3.5129999999999999}]},{"osd":2,"last update":"Fri Feb 9 17:42:13 2024","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":1.472,"15min":1.5880000000000001},"min":{"1min":0.35699999999999998,"5min":0.29399999999999998,"15min":0.22600000000000001},"max":{"1min":4.2160000000000002,"5min":9.8019999999999996,"15min":10.682},"last":3.5649999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.5149999999999999,"15min":1.6459999999999999},"min":{"1min":0.372,"5min":0.35599999999999998,"15min":0.28000000000000003},"max":{"1min":4.2850000000000001,"5min":9.8439999999999994,"15min":11.553000000000001},"last":3.665}]},{"osd":3,"last update":"Fri Feb 9 17:42:13 2024","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.7969999999999999,"15min":2.0550000000000002},"min":{"1min":0.28399999999999997,"5min":0.26700000000000002,"15min":0.25600000000000001},"max":{"1min":9.6389999999999993,"5min":11.276,"15min":32.545000000000002},"last":4.71},{"interface":"front","average":{"1min":1.6419999999999999,"5min":1.8640000000000001,"15min":1.9179999999999999},"min":{"1min":0.36599999999999999,"5min":0.307,"15min":0.26600000000000001},"max":{"1min":8.2520000000000007,"5min":11.122,"15min":32.670999999999999},"last":3.097}]},{"osd":4,"last update":"Fri Feb 9 17:42:13 2024","interfaces":[{"interface":"back","average":{"1min":1.796,"5min":1.03,"15min":1.4379999999999999},"min":{"1min":0.252,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":8.2959999999999994,"5min":8.2959999999999994,"15min":19.798999999999999},"last":5.3010000000000002},{"interface":"front","average":{"1min":1.5660000000000001,"5min":1.147,"15min":1.385},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26000000000000001},"max":{"1min":4.8109999999999999,"5min":5.7270000000000003,"15min":10.702999999999999},"last":8.5069999999999997}]},{"osd":5,"last update":"Fri Feb 9 17:42:13 2024","interfaces":[{"interface":"back","average":{"1min":1.7629999999999999,"5min":1.3180000000000001,"15min":1.4239999999999999},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29199999999999998},"max":{"1min":5.6790000000000003,"5min":6.7679999999999998,"15min":10.968999999999999},"last":2.8100000000000001},{"interface":"front","average":{"1min":1.879,"5min":1.226,"15min":1.4650000000000001},"min":{"1min":0.33700000000000002,"5min":0.316,"15min":0.29299999999999998},"max":{"1min":8.3989999999999991,"5min":8.3989999999999991,"15min":13.91},"last":2.778}]},{"osd":7,"last update":"Fri Feb 9 17:41:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.0960000000000001,"15min":1.4350000000000001},"min":{"1min":0.30199999999999999,"5min":0.29399999999999998,"15min":0.28000000000000003},"max":{"1min":2.8250000000000002,"5min":5.4029999999999996,"15min":14.022},"last":3.7170000000000001},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.288,"15min":1.6100000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.309},"max":{"1min":4.234,"5min":12.441000000000001,"15min":15.458},"last":4.8079999999999998}]},{"osd":8,"last update":"Fri Feb 9 17:26:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":0.876,"15min":0.79600000000000004},"min":{"1min":0.39900000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":4.0339999999999998,"5min":4.4880000000000004,"15min":4.4880000000000004},"last":4.3840000000000003},{"interface":"front","average":{"1min":1.0469999999999999,"5min":0.88600000000000001,"15min":0.78300000000000003},"min":{"1min":0.375,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":3.6539999999999999,"5min":4.8570000000000002,"15min":4.8570000000000002},"last":4.4210000000000003}]},{"osd":9,"last update":"Fri Feb 9 17:42:19 2024","interfaces":[{"interface":"back","average":{"1min":1.6830000000000001,"5min":1.2889999999999999,"15min":1.3560000000000001},"min":{"1min":0.438,"5min":0.38800000000000001,"15min":0.35899999999999999},"max":{"1min":6.9029999999999996,"5min":10.779999999999999,"15min":10.803000000000001},"last":3.6920000000000002},{"interface":"front","average":{"1min":1.8420000000000001,"5min":1.444,"15min":1.4590000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40300000000000002},"max":{"1min":6.5640000000000001,"5min":6.5640000000000001,"15min":8.1630000000000003},"last":3.6200000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:41:55 2024","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":1.1619999999999999,"15min":1.4139999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":8.3580000000000005,"5min":10.677,"15min":10.677},"last":3.2519999999999998},{"interface":"front","average":{"1min":1.214,"5min":1.1060000000000001,"15min":1.4850000000000001},"min":{"1min":0.40899999999999997,"5min":0.38,"15min":0.35899999999999999},"max":{"1min":3.9420000000000002,"5min":5.9509999999999996,"15min":13.836},"last":4.7859999999999996}]},{"osd":11,"last update":"Fri Feb 9 17:42:13 2024","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.2889999999999999,"15min":1.4470000000000001},"min":{"1min":0.46200000000000002,"5min":0.45500000000000002,"15min":0.36099999999999999},"max":{"1min":4.3179999999999996,"5min":7.7450000000000001,"15min":10.273999999999999},"last":3.6429999999999998},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.171,"15min":1.4890000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":4.0620000000000003,"5min":5.2140000000000004,"15min":8.2739999999999991},"last":4.7510000000000003}]}]},{"osd":1,"up_from":13,"seq":55834575143,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2585644,"kb_used_data":2013028,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":91159508,"statfs":{"total":95995035648,"available":93347336192,"internally_reserved":0,"allocated":2061340672,"data_stored":2061053022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":586348997},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:42:28 2024","interfaces":[{"interface":"back","average":{"1min":0.28999999999999998,"5min":1.9850000000000001,"15min":1.758},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.51600000000000001,"5min":19.079000000000001,"15min":19.079000000000001},"last":0.312},{"interface":"front","average":{"1min":0.40999999999999998,"5min":1.881,"15min":1.7390000000000001},"min":{"1min":0.23100000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":1.075,"5min":23.573,"15min":23.573},"last":0.48699999999999999}]},{"osd":2,"last update":"Fri Feb 9 17:41:50 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":2.0470000000000002,"15min":1.829},"min":{"1min":0.28999999999999998,"5min":0.23200000000000001,"15min":0.22600000000000001},"max":{"1min":0.54300000000000004,"5min":12.52,"15min":26.361999999999998},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.63200000000000001,"5min":2.339,"15min":2.0840000000000001},"min":{"1min":0.27400000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":1.6299999999999999,"5min":11.346,"15min":26.087},"last":3.0859999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:40:36 2024","interfaces":[{"interface":"back","average":{"1min":1.8959999999999999,"5min":2.2690000000000001,"15min":1.6060000000000001},"min":{"1min":0.26200000000000001,"5min":0.245,"15min":0.245},"max":{"1min":6.9400000000000004,"5min":10.657,"15min":10.657},"last":0.30299999999999999},{"interface":"front","average":{"1min":2.2349999999999999,"5min":2.508,"15min":1.734},"min":{"1min":0.36599999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":7.1029999999999998,"5min":14.68,"15min":14.68},"last":1.25}]},{"osd":4,"last update":"Fri Feb 9 17:42:22 2024","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.8360000000000001,"15min":1.909},"min":{"1min":0.27700000000000002,"5min":0.25800000000000001,"15min":0.253},"max":{"1min":6.5140000000000002,"5min":9.4649999999999999,"15min":11.766999999999999},"last":1.7210000000000001},{"interface":"front","average":{"1min":0.86099999999999999,"5min":1.6379999999999999,"15min":1.8320000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":5.0179999999999998,"5min":10.228,"15min":10.512},"last":0.61899999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:42:09 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.9319999999999999,"15min":1.8480000000000001},"min":{"1min":0.29199999999999998,"5min":0.28999999999999998,"15min":0.27000000000000002},"max":{"1min":5.9429999999999996,"5min":9.4299999999999997,"15min":13.544},"last":1.754},{"interface":"front","average":{"1min":1.323,"5min":2.0419999999999998,"15min":1.774},"min":{"1min":0.32300000000000001,"5min":0.30499999999999999,"15min":0.26000000000000001},"max":{"1min":5.9569999999999999,"5min":10.531000000000001,"15min":12.117000000000001},"last":0.60199999999999998}]},{"osd":6,"last update":"Fri Feb 9 17:42:13 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.2609999999999999,"15min":1.2689999999999999},"min":{"1min":0.436,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":4.9160000000000004,"5min":5.0119999999999996,"15min":5.0119999999999996},"last":0.71099999999999997},{"interface":"front","average":{"1min":1.238,"5min":1.1870000000000001,"15min":1.179},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":4.952,"5min":4.952,"15min":4.952},"last":0.75800000000000001}]},{"osd":7,"last update":"Fri Feb 9 17:42:13 2024","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":1.147,"15min":1.153},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":3.8700000000000001,"5min":4.0439999999999996,"15min":4.0439999999999996},"last":0.93300000000000005},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.21,"15min":1.228},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":4.0389999999999997,"5min":4.9820000000000002,"15min":4.9820000000000002},"last":0.77900000000000003}]},{"osd":8,"last update":"Fri Feb 9 17:42:22 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.83299999999999996,"15min":0.872},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.296,"5min":2.121,"15min":2.121},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.81000000000000005,"15min":0.83699999999999997},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.325,"5min":2.1379999999999999,"15min":2.1379999999999999},"last":0.68600000000000005}]},{"osd":9,"last update":"Fri Feb 9 17:42:28 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":1.7250000000000001,"15min":1.6579999999999999},"min":{"1min":0.33900000000000002,"5min":0.33400000000000002,"15min":0.32400000000000001},"max":{"1min":2.516,"5min":9.4960000000000004,"15min":9.5839999999999996},"last":0.58799999999999997},{"interface":"front","average":{"1min":1.331,"5min":1.869,"15min":1.736},"min":{"1min":0.375,"5min":0.32900000000000001,"15min":0.30099999999999999},"max":{"1min":4.3179999999999996,"5min":10.548,"15min":12.117000000000001},"last":1.802}]},{"osd":10,"last update":"Fri Feb 9 17:42:09 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":2.0099999999999998,"15min":1.6559999999999999},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.35999999999999999},"max":{"1min":1.1699999999999999,"5min":15.083,"15min":15.083},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.77400000000000002,"5min":2.0529999999999999,"15min":1.871},"min":{"1min":0.51200000000000001,"5min":0.48199999999999998,"15min":0.33400000000000002},"max":{"1min":1.268,"5min":15.121,"15min":15.121},"last":0.73299999999999998}]},{"osd":11,"last update":"Fri Feb 9 17:42:13 2024","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":2.1509999999999998,"5min":2.1509999999999998,"15min":2.1509999999999998},"last":1.78},{"interface":"front","average":{"1min":1.226,"5min":1.226,"15min":1.226},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":3.0910000000000002,"5min":3.0910000000000002,"15min":3.0910000000000002},"last":0.89900000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738667,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4093604,"kb_used_data":3543012,"kb_used_omap":1,"kb_used_meta":550398,"kb_avail":89651548,"statfs":{"total":95995035648,"available":91803185152,"internally_reserved":0,"allocated":3628044288,"data_stored":3627756638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":563608005},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 9 17:42:11 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":1.833,"15min":1.6000000000000001},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.20499999999999999},"max":{"1min":3.6600000000000001,"5min":13.842000000000001,"15min":14.166},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.77100000000000002,"5min":2.0859999999999999,"15min":1.833},"min":{"1min":0.32000000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.7050000000000001,"5min":14.122,"15min":20.177},"last":0.35899999999999999}]},{"osd":2,"last update":"Fri Feb 9 17:39:25 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":5.5339999999999998,"5min":5.5339999999999998,"15min":5.5339999999999998},"last":1.744},{"interface":"front","average":{"1min":1.593,"5min":1.593,"15min":1.593},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":13.473000000000001,"5min":13.473000000000001,"15min":13.473000000000001},"last":3.109}]},{"osd":3,"last update":"Fri Feb 9 17:41:40 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":2.52,"15min":2.855},"min":{"1min":0.27200000000000002,"5min":0.247,"15min":0.247},"max":{"1min":2.4729999999999999,"5min":16.756,"15min":16.756},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.46899999999999997,"5min":1.538,"15min":1.716},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":1.7689999999999999,"5min":6.0570000000000004,"15min":6.0570000000000004},"last":3.0470000000000002}]},{"osd":4,"last update":"Fri Feb 9 17:42:31 2024","interfaces":[{"interface":"back","average":{"1min":1.6830000000000001,"5min":1.3660000000000001,"15min":1.655},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.26400000000000001},"max":{"1min":6.5609999999999999,"5min":7.7560000000000002,"15min":14.419},"last":0.443},{"interface":"front","average":{"1min":1.651,"5min":1.8140000000000001,"15min":1.8879999999999999},"min":{"1min":0.35899999999999999,"5min":0.28000000000000003,"15min":0.27200000000000002},"max":{"1min":6.5919999999999996,"5min":13.308999999999999,"15min":14.465999999999999},"last":0.502}]},{"osd":5,"last update":"Fri Feb 9 17:41:51 2024","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.4370000000000001,"15min":1.6419999999999999},"min":{"1min":0.32200000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":3.976,"5min":8.4079999999999995,"15min":9.8569999999999993},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.363,"15min":1.76},"min":{"1min":0.39900000000000002,"5min":0.371,"15min":0.312},"max":{"1min":3.3900000000000001,"5min":7.8239999999999998,"15min":9.9049999999999994},"last":1.73}]},{"osd":6,"last update":"Fri Feb 9 17:42:21 2024","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.722,"15min":1.8480000000000001},"min":{"1min":0.55000000000000004,"5min":0.33300000000000002,"15min":0.27800000000000002},"max":{"1min":3.411,"5min":13.407,"15min":13.407},"last":3.7599999999999998},{"interface":"front","average":{"1min":1.538,"5min":1.772,"15min":1.8799999999999999},"min":{"1min":0.57899999999999996,"5min":0.39200000000000002,"15min":0.33000000000000002},"max":{"1min":3.4460000000000002,"5min":13.448,"15min":13.448},"last":3.7890000000000001}]},{"osd":7,"last update":"Fri Feb 9 17:41:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":2.0299999999999998,"15min":2.056},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":2.3279999999999998,"5min":13.349,"15min":13.349},"last":3.7160000000000002},{"interface":"front","average":{"1min":0.94699999999999995,"5min":2.1150000000000002,"15min":2.0659999999999998},"min":{"1min":0.35899999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":2.097,"5min":14.451000000000001,"15min":16.765999999999998},"last":3.3719999999999999}]},{"osd":8,"last update":"Fri Feb 9 17:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":1.9419999999999999,"15min":1.5840000000000001},"min":{"1min":0.371,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":2.145,"5min":14.323,"15min":14.323},"last":0.45600000000000002},{"interface":"front","average":{"1min":1.016,"5min":2.0720000000000001,"15min":1.673},"min":{"1min":0.372,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":3.0099999999999998,"5min":14.481999999999999,"15min":16.41},"last":0.54000000000000004}]},{"osd":9,"last update":"Fri Feb 9 17:42:31 2024","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":2.0630000000000002,"15min":1.7709999999999999},"min":{"1min":0.33800000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":4.1139999999999999,"5min":17.952000000000002,"15min":17.952000000000002},"last":3.6539999999999999},{"interface":"front","average":{"1min":1.6379999999999999,"5min":2.0859999999999999,"15min":1.8120000000000001},"min":{"1min":0.40999999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":4.7839999999999998,"5min":17.913,"15min":17.913},"last":3.5339999999999998}]},{"osd":10,"last update":"Fri Feb 9 17:41:51 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":2.137,"15min":1.8280000000000001},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.40899999999999997},"max":{"1min":3.2040000000000002,"5min":14.271000000000001,"15min":16.876000000000001},"last":3.4079999999999999},{"interface":"front","average":{"1min":0.73399999999999999,"5min":1.282,"15min":1.4830000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":2.5659999999999998,"5min":11.384,"15min":11.384},"last":0.54300000000000004}]},{"osd":11,"last update":"Fri Feb 9 17:42:21 2024","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.2729999999999999,"15min":1.6739999999999999},"min":{"1min":0.44600000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":4.4550000000000001,"5min":7.8739999999999997,"15min":40.021000000000001},"last":0.51500000000000001},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.397,"15min":1.7549999999999999},"min":{"1min":0.48199999999999998,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":3.9630000000000001,"5min":7.899,"15min":40.055999999999997},"last":0.69099999999999995}]}]},{"osd":2,"up_from":0,"seq":77309411617,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":27,"seq":115964117277,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":914268,"kb_used_data":504084,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":92830884,"statfs":{"total":95995035648,"available":95058825216,"internally_reserved":0,"allocated":516182016,"data_stored":515829419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":420018631},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:42:09 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":2.423,"15min":1.8640000000000001},"min":{"1min":0.23300000000000001,"5min":0.186,"15min":0.186},"max":{"1min":15.657999999999999,"5min":38.984999999999999,"15min":38.984999999999999},"last":0.40200000000000002},{"interface":"front","average":{"1min":1.3839999999999999,"5min":2.528,"15min":1.97},"min":{"1min":0.25800000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":16.637,"5min":41.497999999999998,"15min":41.497999999999998},"last":0.249}]},{"osd":1,"last update":"Fri Feb 9 17:40:06 2024","interfaces":[{"interface":"back","average":{"1min":4.5670000000000002,"5min":4.5670000000000002,"15min":4.5670000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":25.965,"5min":25.965,"15min":25.965},"last":0.35299999999999998},{"interface":"front","average":{"1min":4.5129999999999999,"5min":4.5129999999999999,"15min":4.5129999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":31.454000000000001,"5min":31.454000000000001,"15min":31.454000000000001},"last":0.375}]},{"osd":2,"last update":"Fri Feb 9 17:42:04 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":2.0600000000000001,"15min":1.802},"min":{"1min":0.32200000000000001,"5min":0.255,"15min":0.22600000000000001},"max":{"1min":4.2409999999999997,"5min":11.663,"15min":14.914999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.67300000000000004,"5min":1.9239999999999999,"15min":1.8260000000000001},"min":{"1min":0.27600000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":4.258,"5min":12.347,"15min":17.613},"last":0.42399999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:42:11 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":1.4670000000000001,"15min":1.806},"min":{"1min":0.30399999999999999,"5min":0.29099999999999998,"15min":0.26800000000000002},"max":{"1min":2.556,"5min":13.933,"15min":34.898000000000003},"last":1.5469999999999999},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.645,"15min":1.788},"min":{"1min":0.32800000000000001,"5min":0.28000000000000003,"15min":0.27900000000000003},"max":{"1min":4.415,"5min":9.3059999999999992,"15min":14.061999999999999},"last":2.1160000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:41:47 2024","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":1.72,"15min":1.6739999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.27800000000000002},"max":{"1min":3.1669999999999998,"5min":11.007999999999999,"15min":17.651},"last":0.96399999999999997},{"interface":"front","average":{"1min":0.93700000000000006,"5min":1.784,"15min":1.728},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.34100000000000003},"max":{"1min":3.1299999999999999,"5min":11.036,"15min":17.675999999999998},"last":0.998}]},{"osd":6,"last update":"Fri Feb 9 17:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":2.3370000000000002,"15min":2.609},"min":{"1min":0.39400000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.669,"5min":12.717000000000001,"15min":12.717000000000001},"last":1.1499999999999999},{"interface":"front","average":{"1min":0.86099999999999999,"5min":1.2969999999999999,"15min":1.369},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":2.9409999999999998,"5min":8.7360000000000007,"15min":8.7360000000000007},"last":1.718}]},{"osd":7,"last update":"Fri Feb 9 17:42:20 2024","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":0.96299999999999997,"15min":0.90200000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":3.2320000000000002,"5min":3.3999999999999999,"15min":3.3999999999999999},"last":0.92400000000000004},{"interface":"front","average":{"1min":1.0489999999999999,"5min":0.81299999999999994,"15min":0.77300000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":3.8610000000000002,"5min":3.8610000000000002,"15min":3.8610000000000002},"last":2.2930000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:42:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.653,"15min":1.427},"min":{"1min":0.41299999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":4.3289999999999997,"5min":8.6940000000000008,"15min":8.6940000000000008},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.7569999999999999,"15min":1.222},"min":{"1min":0.42799999999999999,"5min":0.36599999999999999,"15min":0.34499999999999997},"max":{"1min":3.6309999999999998,"5min":13.976000000000001,"15min":14.129},"last":0.36199999999999999}]},{"osd":9,"last update":"Fri Feb 9 17:42:34 2024","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":1.5369999999999999,"15min":1.536},"min":{"1min":0.46000000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":3.698,"5min":9.7929999999999993,"15min":9.7929999999999993},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.97399999999999998,"5min":1.5049999999999999,"15min":1.593},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.32800000000000001},"max":{"1min":3.2650000000000001,"5min":8.7899999999999991,"15min":13.717000000000001},"last":0.57499999999999996}]},{"osd":10,"last update":"Fri Feb 9 17:42:18 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.88500000000000001,"15min":0.86899999999999999},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":3.2120000000000002,"5min":3.2120000000000002,"15min":3.2120000000000002},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.75800000000000001,"15min":0.753},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":2.6179999999999999,"5min":2.6179999999999999,"15min":2.6179999999999999},"last":0.52300000000000002}]},{"osd":11,"last update":"Fri Feb 9 17:42:11 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":1.5680000000000001,"15min":1.609},"min":{"1min":0.42699999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.7410000000000001,"5min":14.300000000000001,"15min":14.300000000000001},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.87,"5min":2.1859999999999999,"15min":1.8779999999999999},"min":{"1min":0.50700000000000001,"5min":0.442,"15min":0.36899999999999999},"max":{"1min":1.917,"5min":17.145,"15min":17.145},"last":0.63}]}]},{"osd":4,"up_from":32,"seq":137438953751,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3516012,"kb_used_data":2985060,"kb_used_omap":1,"kb_used_meta":530942,"kb_avail":90229140,"statfs":{"total":95995035648,"available":92394639360,"internally_reserved":0,"allocated":3056701440,"data_stored":3056413790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":543685061},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":1.4530000000000001,"15min":1.7130000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":3.1600000000000001,"5min":7.7199999999999998,"15min":50.356999999999999},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.83299999999999996,"5min":1.6020000000000001,"15min":1.6459999999999999},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.247},"max":{"1min":3.137,"5min":7.8220000000000001,"15min":50.569000000000003},"last":0.57399999999999995}]},{"osd":1,"last update":"Fri Feb 9 17:42:18 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.9850000000000001,"15min":2.089},"min":{"1min":0.27000000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":7.3920000000000003,"5min":8.0609999999999999,"15min":50.473999999999997},"last":4.8609999999999998},{"interface":"front","average":{"1min":1.5660000000000001,"5min":2.0219999999999998,"15min":2.1299999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28100000000000003},"max":{"1min":6.4939999999999998,"5min":8.0990000000000002,"15min":50.499000000000002},"last":0.81499999999999995}]},{"osd":2,"last update":"Fri Feb 9 17:28:51 2024","interfaces":[{"interface":"back","average":{"1min":2.536,"5min":1.546,"15min":0.91400000000000003},"min":{"1min":0.29999999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":8.8819999999999997,"5min":8.8819999999999997,"15min":8.8819999999999997},"last":0.622},{"interface":"front","average":{"1min":2.214,"5min":1.7330000000000001,"15min":0.97799999999999998},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":9.5039999999999996,"5min":9.5039999999999996,"15min":9.5039999999999996},"last":0.53800000000000003}]},{"osd":3,"last update":"Fri Feb 9 17:42:18 2024","interfaces":[{"interface":"back","average":{"1min":1.591,"5min":1.8049999999999999,"15min":2.1560000000000001},"min":{"1min":0.317,"5min":0.28199999999999997,"15min":0.245},"max":{"1min":6.548,"5min":7.9349999999999996,"15min":50.462000000000003},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.75,"5min":1.607,"15min":1.9850000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.255},"max":{"1min":2.7799999999999998,"5min":9.1579999999999995,"15min":50.529000000000003},"last":0.52400000000000002}]},{"osd":5,"last update":"Fri Feb 9 17:41:47 2024","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.0249999999999999,"15min":1.333},"min":{"1min":0.248,"5min":0.22500000000000001,"15min":0.222},"max":{"1min":3.351,"5min":6.3760000000000003,"15min":9.5630000000000006},"last":4.6829999999999998},{"interface":"front","average":{"1min":1.216,"5min":1.1060000000000001,"15min":1.607},"min":{"1min":0.39400000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":3.4350000000000001,"5min":6.4080000000000004,"15min":50.509999999999998},"last":4.8380000000000001}]},{"osd":6,"last update":"Fri Feb 9 17:42:18 2024","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.3260000000000001,"15min":1.4710000000000001},"min":{"1min":0.35999999999999999,"5min":0.246,"15min":0.246},"max":{"1min":4.2329999999999997,"5min":8.0039999999999996,"15min":18.09},"last":3.8860000000000001},{"interface":"front","average":{"1min":2.415,"5min":1.421,"15min":1.7090000000000001},"min":{"1min":0.312,"5min":0.245,"15min":0.245},"max":{"1min":7.6619999999999999,"5min":7.6619999999999999,"15min":50.402999999999999},"last":1.0129999999999999}]},{"osd":7,"last update":"Fri Feb 9 17:41:53 2024","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":1.173,"15min":1.619},"min":{"1min":0.45900000000000002,"5min":0.29899999999999999,"15min":0.28399999999999997},"max":{"1min":3.536,"5min":7.4640000000000004,"15min":15.223000000000001},"last":3.7850000000000001},{"interface":"front","average":{"1min":1.6120000000000001,"5min":1.2290000000000001,"15min":1.6120000000000001},"min":{"1min":0.37,"5min":0.28199999999999997,"15min":0.28100000000000003},"max":{"1min":4.0119999999999996,"5min":6.7640000000000002,"15min":19.047000000000001},"last":4.2140000000000004}]},{"osd":8,"last update":"Fri Feb 9 17:41:47 2024","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.073,"15min":1.413},"min":{"1min":0.47199999999999998,"5min":0.34799999999999998,"15min":0.34399999999999997},"max":{"1min":3.5880000000000001,"5min":6.1079999999999997,"15min":8.4160000000000004},"last":4.7279999999999998},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.056,"15min":1.5409999999999999},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.308},"max":{"1min":3.214,"5min":6,"15min":50.594000000000001},"last":3.8540000000000001}]},{"osd":9,"last update":"Fri Feb 9 17:42:33 2024","interfaces":[{"interface":"back","average":{"1min":2.6589999999999998,"5min":1.4530000000000001,"15min":1.748},"min":{"1min":0.371,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":9.2370000000000001,"5min":9.2370000000000001,"15min":50.442999999999998},"last":4.7830000000000004},{"interface":"front","average":{"1min":1.403,"5min":1.353,"15min":1.4630000000000001},"min":{"1min":0.435,"5min":0.39000000000000001,"15min":0.36199999999999999},"max":{"1min":8.3580000000000005,"5min":8.3580000000000005,"15min":10.555},"last":0.54900000000000004}]},{"osd":10,"last update":"Fri Feb 9 17:42:03 2024","interfaces":[{"interface":"back","average":{"1min":1.4359999999999999,"5min":1.607,"15min":1.891},"min":{"1min":0.52500000000000002,"5min":0.40300000000000002,"15min":0.378},"max":{"1min":3.6160000000000001,"5min":7.0030000000000001,"15min":51.223999999999997},"last":4.758},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.4610000000000001,"15min":1.9099999999999999},"min":{"1min":0.42899999999999999,"5min":0.41999999999999998,"15min":0.38100000000000001},"max":{"1min":3.617,"5min":6.3860000000000001,"15min":50.552},"last":4.1760000000000002}]},{"osd":11,"last update":"Fri Feb 9 17:42:33 2024","interfaces":[{"interface":"back","average":{"1min":2.3959999999999999,"5min":1.718,"15min":2.0150000000000001},"min":{"1min":0.50900000000000001,"5min":0.49099999999999999,"15min":0.40500000000000003},"max":{"1min":9.4060000000000006,"5min":9.4060000000000006,"15min":50.484999999999999},"last":4.8150000000000004},{"interface":"front","average":{"1min":1.135,"5min":1.3740000000000001,"15min":1.6220000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.34899999999999998},"max":{"1min":3.4409999999999998,"5min":7.694,"15min":18.574999999999999},"last":3.8220000000000001}]}]},{"osd":5,"up_from":40,"seq":171798692113,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3914160,"kb_used_data":3513520,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":89830992,"statfs":{"total":95995035648,"available":91986935808,"internally_reserved":0,"allocated":3597844480,"data_stored":3597549206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:42:11 2024","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.046,"15min":1.05},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":7.3730000000000002,"5min":7.3730000000000002,"15min":7.3730000000000002},"last":2.327},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.51,"15min":1.712},"min":{"1min":0.28699999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":3.2589999999999999,"5min":6.4939999999999998,"15min":6.4939999999999998},"last":0.54700000000000004}]},{"osd":1,"last update":"Fri Feb 9 17:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":1.8260000000000001,"15min":2.0059999999999998},"min":{"1min":0.245,"5min":0.24199999999999999,"15min":0.23200000000000001},"max":{"1min":2.254,"5min":48.161999999999999,"15min":48.161999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":1.0980000000000001,"5min":2.056,"15min":1.8979999999999999},"min":{"1min":0.25900000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":6.2290000000000001,"5min":48.218000000000004,"15min":48.218000000000004},"last":2.3660000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:40:42 2024","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":1.345,"15min":1.6539999999999999},"min":{"1min":0.32000000000000001,"5min":0.27900000000000003,"15min":0.25600000000000001},"max":{"1min":4.4930000000000003,"5min":11.143000000000001,"15min":11.143000000000001},"last":2.6560000000000001},{"interface":"front","average":{"1min":1.796,"5min":1.218,"15min":1.718},"min":{"1min":0.39100000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":6.1970000000000001,"5min":6.1970000000000001,"15min":14.122},"last":1.5649999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":1.6850000000000001,"15min":2.149},"min":{"1min":0.32800000000000001,"5min":0.29799999999999999,"15min":0.26500000000000001},"max":{"1min":2.9980000000000002,"5min":16.59,"15min":64.150999999999996},"last":2.3929999999999998},{"interface":"front","average":{"1min":0.85699999999999998,"5min":1.6970000000000001,"15min":2.101},"min":{"1min":0.44400000000000001,"5min":0.247,"15min":0.247},"max":{"1min":2.2170000000000001,"5min":12.468,"15min":64.176000000000002},"last":0.60799999999999998}]},{"osd":4,"last update":"Fri Feb 9 17:41:46 2024","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.6659999999999999,"15min":1.5069999999999999},"min":{"1min":0.33600000000000002,"5min":0.28100000000000003,"15min":0.23400000000000001},"max":{"1min":8.0169999999999995,"5min":49.313000000000002,"15min":49.313000000000002},"last":8.1980000000000004},{"interface":"front","average":{"1min":1.2,"5min":1.514,"15min":1.4970000000000001},"min":{"1min":0.35399999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.5960000000000001,"5min":50.505000000000003,"15min":50.505000000000003},"last":6.1639999999999997}]},{"osd":6,"last update":"Fri Feb 9 17:42:22 2024","interfaces":[{"interface":"back","average":{"1min":2.8479999999999999,"5min":1.597,"15min":1.554},"min":{"1min":0.28199999999999997,"5min":0.26400000000000001,"15min":0.253},"max":{"1min":12.425000000000001,"5min":12.425000000000001,"15min":15.375999999999999},"last":3.7050000000000001},{"interface":"front","average":{"1min":3.121,"5min":1.696,"15min":1.677},"min":{"1min":0.374,"5min":0.32800000000000001,"15min":0.28899999999999998},"max":{"1min":11.199,"5min":11.199,"15min":14.250999999999999},"last":8.4169999999999998}]},{"osd":7,"last update":"Fri Feb 9 17:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":1.589,"15min":1.536},"min":{"1min":0.38500000000000001,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":2.3039999999999998,"5min":50.545000000000002,"15min":50.545000000000002},"last":3.742},{"interface":"front","average":{"1min":0.96999999999999997,"5min":2.5800000000000001,"15min":2.0760000000000001},"min":{"1min":0.32400000000000001,"5min":0.32100000000000001,"15min":0.28799999999999998},"max":{"1min":2.8730000000000002,"5min":102.526,"15min":102.526},"last":3.7909999999999999}]},{"osd":8,"last update":"Fri Feb 9 17:42:22 2024","interfaces":[{"interface":"back","average":{"1min":2.2629999999999999,"5min":1.579,"15min":1.6120000000000001},"min":{"1min":0.433,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":9.8520000000000003,"5min":14.542,"15min":14.542},"last":0.68100000000000005},{"interface":"front","average":{"1min":1.9770000000000001,"5min":1.181,"15min":1.7410000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":7.734,"5min":7.734,"15min":9.6240000000000006},"last":9.8350000000000009}]},{"osd":9,"last update":"Fri Feb 9 17:42:22 2024","interfaces":[{"interface":"back","average":{"1min":2.2930000000000001,"5min":1.4359999999999999,"15min":1.4550000000000001},"min":{"1min":0.40300000000000002,"5min":0.38600000000000001,"15min":0.371},"max":{"1min":7.4180000000000001,"5min":7.4180000000000001,"15min":14.98},"last":2.4169999999999998},{"interface":"front","average":{"1min":1.972,"5min":1.3200000000000001,"15min":1.417},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.312},"max":{"1min":7.7910000000000004,"5min":7.7910000000000004,"15min":9.4239999999999995},"last":3.8220000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:41:46 2024","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.069,"15min":1.4359999999999999},"min":{"1min":0.51800000000000002,"5min":0.35199999999999998,"15min":0.33800000000000002},"max":{"1min":3.843,"5min":5.7400000000000002,"15min":14.218},"last":9.8829999999999991},{"interface":"front","average":{"1min":0.89900000000000002,"5min":1.2969999999999999,"15min":1.579},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.36099999999999999},"max":{"1min":2.8239999999999998,"5min":10.606,"15min":14.186999999999999},"last":0.63500000000000001}]},{"osd":11,"last update":"Fri Feb 9 17:42:11 2024","interfaces":[{"interface":"back","average":{"1min":1.5369999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":3.597,"5min":3.597,"15min":3.597},"last":9.9250000000000007},{"interface":"front","average":{"1min":1.653,"5min":1.653,"15min":1.653},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":5.593,"5min":5.593,"15min":5.593},"last":0.65900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3607756800,"data_stored":3607756800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2048589824,"data_stored":2048589824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":514260992,"data_stored":514260992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3042181120,"data_stored":3042181120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3578859520,"data_stored":3578855448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1528365056,"data_stored":1528365056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2084114432,"data_stored":2084110360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1551499264,"data_stored":1551499264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1564016640,"data_stored":1564016640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":3112501248,"data_stored":3112501248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2054688768,"data_stored":2054684696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-09T17:42:43.675 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd pool set unique_pool_3 pgp_num 16 2024-02-09T17:42:43.760 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:42:43.796 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:44.001 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:42:44.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:44.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:42:44.081 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:44 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:42:44.081 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:44 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:42:44.081 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:44 smithi099 ceph-mon[39420]: pgmap v1283: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 7.6 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 127 B/s rd, 16 MiB/s wr, 812 op/s 2024-02-09T17:42:44.081 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:44 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:42:44.081 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:44 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:42:44.081 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:44 smithi099 ceph-mon[39420]: osdmap e505: 12 total, 12 up, 11 in 2024-02-09T17:42:44.081 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:44 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:42:44.081 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:44 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:42:44.081 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:44 smithi162 ceph-mon[44304]: pgmap v1283: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 7.6 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 127 B/s rd, 16 MiB/s wr, 812 op/s 2024-02-09T17:42:44.081 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:44 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:42:44.081 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:44 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:42:44.082 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:44 smithi162 ceph-mon[44304]: osdmap e505: 12 total, 12 up, 11 in 2024-02-09T17:42:44.111 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:42:44.142 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:44.172 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:42:44.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:44.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:42:44.281 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:42:44.318 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:42:44.345 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:42:44.346 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:44.388 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:42:44.393 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:42:44.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:44.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:42:44.419 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:42:44.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:44 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:42:44.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:44 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:42:44.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:44 smithi179 ceph-mon[43975]: pgmap v1283: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 7.6 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 127 B/s rd, 16 MiB/s wr, 812 op/s 2024-02-09T17:42:44.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:44 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:42:44.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:44 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:42:44.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:44 smithi179 ceph-mon[43975]: osdmap e505: 12 total, 12 up, 11 in 2024-02-09T17:42:44.441 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:42:44.474 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:42:44.495 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:42:44.504 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:42:44.530 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:42:44.535 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:44.565 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:42:44.580 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:42:44.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:44.583 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:42:44.599 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:42:44.628 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:42:44.659 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-09T17:42:44.665 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:42:44.684 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:42:44.695 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:42:44.699 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:42:44.699 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:42:44.699 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:42:44.700 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:42:44.700 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:42:44.700 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:42:44.700 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:42:44.700 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:42:44.700 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:42:44.700 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:42:44.700 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:42:44.700 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:42:44.735 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:44.823 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:42:44.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:44.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:42:44.928 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:42:44.957 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:44.986 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:42:44.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:44.994 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:42:45.095 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:42:45.130 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:45.137 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:45 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1804630540' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:42:45.173 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:42:45.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:45 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1804630540' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:42:45.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:45.181 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:42:45.282 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:42:45.327 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:45 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1804630540' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:42:45.333 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:45.368 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:42:45.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:45.372 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:42:45.473 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:42:45.507 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:45.530 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:42:45.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:45.533 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:42:45.633 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:42:45.651 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:42:45.651 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-02-09T17:42:45.651 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-02-09T17:42:45.652 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-02-09T17:42:45.652 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.652 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:218267 3.f 3:fc3d9bb7:::benchmark_data_smithi179_90440_object86308:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-02-09T17:42:45.652 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:34:08.019138+0000", 2024-02-09T17:42:45.652 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 517.62859265999998, 2024-02-09T17:42:45.652 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21643954900000001, 2024-02-09T17:42:45.652 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:45.652 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:42:45.652 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:45.652 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:42:45.652 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:42:45.652 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 218267 2024-02-09T17:42:45.652 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.653 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:45.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.653 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:45.653 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:08.019138+0000", 2024-02-09T17:42:45.653 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:45.653 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.653 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:45.653 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:08.019138+0000", 2024-02-09T17:42:45.653 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:45.653 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.653 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:45.653 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:08.019139+0000", 2024-02-09T17:42:45.654 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0079999999999999e-06 2024-02-09T17:42:45.654 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.654 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.654 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:45.654 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:08.019142+0000", 2024-02-09T17:42:45.654 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0180000000000002e-06 2024-02-09T17:42:45.654 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.654 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.654 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:45.654 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:08.019150+0000", 2024-02-09T17:42:45.654 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1200000000000002e-06 2024-02-09T17:42:45.654 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.654 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.654 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:45.655 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:08.019157+0000", 2024-02-09T17:42:45.655 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5519999999999999e-06 2024-02-09T17:42:45.655 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.655 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.655 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:45.655 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:08.019205+0000", 2024-02-09T17:42:45.655 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7827000000000001e-05 2024-02-09T17:42:45.655 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.655 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.655 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:45.655 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:08.019296+0000", 2024-02-09T17:42:45.655 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0422999999999998e-05 2024-02-09T17:42:45.655 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.656 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 8,11", 2024-02-09T17:42:45.656 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:08.019366+0000", 2024-02-09T17:42:45.656 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0469000000000006e-05 2024-02-09T17:42:45.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.656 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:45.656 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:08.027160+0000", 2024-02-09T17:42:45.656 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0077935319999999997 2024-02-09T17:42:45.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.656 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:45.656 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:08.039241+0000", 2024-02-09T17:42:45.656 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012081273999999999 2024-02-09T17:42:45.657 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.657 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.657 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:45.657 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:08.235457+0000", 2024-02-09T17:42:45.657 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.196216684 2024-02-09T17:42:45.657 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.657 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.657 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:42:45.657 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:08.235526+0000", 2024-02-09T17:42:45.657 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8942999999999994e-05 2024-02-09T17:42:45.657 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.657 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.657 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:45.657 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:08.235577+0000", 2024-02-09T17:42:45.657 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0698999999999997e-05 2024-02-09T17:42:45.658 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:45.658 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:45.658 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:45.658 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.658 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.658 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24817.0:218269 3.f 3:fc4a6248:::benchmark_data_smithi179_90440_object86310:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-02-09T17:42:45.658 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:34:08.019316+0000", 2024-02-09T17:42:45.658 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 517.62841420300003, 2024-02-09T17:42:45.658 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22402417799999999, 2024-02-09T17:42:45.658 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:45.658 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:42:45.658 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:45.658 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24817", 2024-02-09T17:42:45.658 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/531700086", 2024-02-09T17:42:45.659 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 218269 2024-02-09T17:42:45.659 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.659 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:45.659 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.659 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:45.659 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:08.019316+0000", 2024-02-09T17:42:45.659 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:45.659 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.659 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.659 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:45.659 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:08.019316+0000", 2024-02-09T17:42:45.659 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:45.659 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.659 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.659 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:45.660 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:08.019317+0000", 2024-02-09T17:42:45.660 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4e-07 2024-02-09T17:42:45.660 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.660 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.660 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:45.660 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:08.019321+0000", 2024-02-09T17:42:45.660 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0980000000000004e-06 2024-02-09T17:42:45.660 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.660 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.660 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:45.660 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:08.019330+0000", 2024-02-09T17:42:45.660 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3789999999999995e-06 2024-02-09T17:42:45.660 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.660 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.661 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:45.661 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:08.019337+0000", 2024-02-09T17:42:45.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6390000000000003e-06 2024-02-09T17:42:45.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.661 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:45.661 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:08.023188+0000", 2024-02-09T17:42:45.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0038511190000000001 2024-02-09T17:42:45.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.661 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:45.661 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:08.023292+0000", 2024-02-09T17:42:45.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000103457 2024-02-09T17:42:45.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.662 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 8,11", 2024-02-09T17:42:45.662 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:08.023369+0000", 2024-02-09T17:42:45.662 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7615999999999997e-05 2024-02-09T17:42:45.662 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.662 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.662 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:45.662 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:08.027288+0000", 2024-02-09T17:42:45.662 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003919255 2024-02-09T17:42:45.662 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.662 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.662 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:45.662 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:08.039286+0000", 2024-02-09T17:42:45.662 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011998015000000001 2024-02-09T17:42:45.662 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.662 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.663 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:45.663 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:08.243247+0000", 2024-02-09T17:42:45.663 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20396029299999999 2024-02-09T17:42:45.663 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.663 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.663 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:42:45.663 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:08.243302+0000", 2024-02-09T17:42:45.663 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5560000000000003e-05 2024-02-09T17:42:45.663 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.663 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.663 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:45.663 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:34:08.243340+0000", 2024-02-09T17:42:45.663 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7907000000000001e-05 2024-02-09T17:42:45.663 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:45.663 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:45.663 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:45.664 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.664 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.664 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25201.0:222237 4.0 4:0512c488:::benchmark_data_smithi179_103809_object84771:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e397)", 2024-02-09T17:42:45.664 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:39:02.559207+0000", 2024-02-09T17:42:45.664 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 223.08852308600001, 2024-02-09T17:42:45.664 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1982094000000001, 2024-02-09T17:42:45.664 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:45.664 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-09T17:42:45.664 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:45.664 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25201", 2024-02-09T17:42:45.664 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:42:45.664 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 222237 2024-02-09T17:42:45.664 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.664 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:45.665 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.665 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:45.665 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.559207+0000", 2024-02-09T17:42:45.665 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:45.665 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.665 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.665 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:45.665 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.559207+0000", 2024-02-09T17:42:45.665 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:45.665 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.665 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.665 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:45.665 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.559209+0000", 2024-02-09T17:42:45.665 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8220000000000001e-06 2024-02-09T17:42:45.665 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.666 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.666 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:45.666 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.559214+0000", 2024-02-09T17:42:45.666 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8409999999999999e-06 2024-02-09T17:42:45.666 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.666 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.666 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:45.666 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.559225+0000", 2024-02-09T17:42:45.666 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1483e-05 2024-02-09T17:42:45.666 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.666 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.666 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:45.666 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.559237+0000", 2024-02-09T17:42:45.666 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1851999999999999e-05 2024-02-09T17:42:45.666 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.667 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.667 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:45.667 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.559292+0000", 2024-02-09T17:42:45.667 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4528999999999999e-05 2024-02-09T17:42:45.667 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.667 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.667 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:45.667 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.559409+0000", 2024-02-09T17:42:45.667 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000117732 2024-02-09T17:42:45.667 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.667 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.667 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,3,8", 2024-02-09T17:42:45.667 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.559511+0000", 2024-02-09T17:42:45.667 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000101975 2024-02-09T17:42:45.667 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.667 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.668 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:45.668 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.569156+0000", 2024-02-09T17:42:45.668 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0096444830000000006 2024-02-09T17:42:45.668 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.668 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.668 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:45.668 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.574637+0000", 2024-02-09T17:42:45.668 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0054813609999999997 2024-02-09T17:42:45.668 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.668 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.668 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:45.668 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.599257+0000", 2024-02-09T17:42:45.668 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.024619704999999999 2024-02-09T17:42:45.668 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.668 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.669 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:42:45.669 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.599267+0000", 2024-02-09T17:42:45.669 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0009e-05 2024-02-09T17:42:45.669 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.669 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.669 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:45.669 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.757344+0000", 2024-02-09T17:42:45.669 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.158076592 2024-02-09T17:42:45.669 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.669 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.669 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:45.669 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.757417+0000", 2024-02-09T17:42:45.669 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3015999999999993e-05 2024-02-09T17:42:45.669 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:45.669 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:45.670 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:45.670 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.670 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.670 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25201.0:222239 4.0 4:07795178:::benchmark_data_smithi179_103809_object84773:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e397)", 2024-02-09T17:42:45.670 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:39:02.560782+0000", 2024-02-09T17:42:45.670 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 223.08694819799999, 2024-02-09T17:42:45.670 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.196921535, 2024-02-09T17:42:45.670 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:45.670 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-09T17:42:45.670 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:45.670 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25201", 2024-02-09T17:42:45.670 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:42:45.670 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 222239 2024-02-09T17:42:45.670 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.670 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:45.671 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.671 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:45.671 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.560782+0000", 2024-02-09T17:42:45.671 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:45.671 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.671 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.671 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:45.671 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.560782+0000", 2024-02-09T17:42:45.671 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:45.671 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.671 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.671 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:45.671 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.560783+0000", 2024-02-09T17:42:45.671 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6099999999999999e-07 2024-02-09T17:42:45.671 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.672 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.672 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:45.672 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.560787+0000", 2024-02-09T17:42:45.672 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0389999999999998e-06 2024-02-09T17:42:45.672 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.672 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.672 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:45.672 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.560798+0000", 2024-02-09T17:42:45.672 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1222e-05 2024-02-09T17:42:45.672 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.672 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.672 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:45.672 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.560805+0000", 2024-02-09T17:42:45.672 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2019999999999998e-06 2024-02-09T17:42:45.672 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.673 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.673 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:45.673 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.563901+0000", 2024-02-09T17:42:45.673 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0030953539999999998 2024-02-09T17:42:45.673 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.673 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.673 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:45.673 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.564041+0000", 2024-02-09T17:42:45.673 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013989199999999999 2024-02-09T17:42:45.673 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.673 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.673 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,3,8", 2024-02-09T17:42:45.673 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.564180+0000", 2024-02-09T17:42:45.673 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013952699999999999 2024-02-09T17:42:45.673 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.674 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.674 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:45.674 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.574699+0000", 2024-02-09T17:42:45.674 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.01051903 2024-02-09T17:42:45.674 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.674 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.674 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:45.674 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.579807+0000", 2024-02-09T17:42:45.674 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0051073780000000001 2024-02-09T17:42:45.674 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.674 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.674 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:45.674 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.600194+0000", 2024-02-09T17:42:45.674 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020387845000000002 2024-02-09T17:42:45.674 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.675 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.675 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:42:45.675 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.600203+0000", 2024-02-09T17:42:45.675 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6049999999999998e-06 2024-02-09T17:42:45.675 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.675 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.675 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:45.675 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.757657+0000", 2024-02-09T17:42:45.675 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.157454231 2024-02-09T17:42:45.675 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.675 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.675 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:45.675 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.757704+0000", 2024-02-09T17:42:45.675 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6248999999999997e-05 2024-02-09T17:42:45.675 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:45.675 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:45.675 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:45.676 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.676 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.676 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25201.0:222241 4.0 4:01207b42:::benchmark_data_smithi179_103809_object84775:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e397)", 2024-02-09T17:42:45.676 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:39:02.561102+0000", 2024-02-09T17:42:45.676 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 223.086628214, 2024-02-09T17:42:45.676 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.19671661, 2024-02-09T17:42:45.676 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:45.676 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-09T17:42:45.676 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:45.676 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25201", 2024-02-09T17:42:45.676 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:42:45.676 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 222241 2024-02-09T17:42:45.676 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.676 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:45.676 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.677 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:45.677 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.561102+0000", 2024-02-09T17:42:45.677 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:45.677 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.677 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.677 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:45.677 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.561102+0000", 2024-02-09T17:42:45.677 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:45.677 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.677 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.677 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:45.678 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.561102+0000", 2024-02-09T17:42:45.678 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5499999999999998e-07 2024-02-09T17:42:45.678 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.678 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.678 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:45.678 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.561105+0000", 2024-02-09T17:42:45.678 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3659999999999999e-06 2024-02-09T17:42:45.678 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.678 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.678 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:45.678 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.561111+0000", 2024-02-09T17:42:45.678 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0190000000000003e-06 2024-02-09T17:42:45.678 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.678 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.679 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:45.679 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.561116+0000", 2024-02-09T17:42:45.679 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1329999999999998e-06 2024-02-09T17:42:45.679 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.679 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.679 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:45.679 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.569194+0000", 2024-02-09T17:42:45.679 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0080776679999999997 2024-02-09T17:42:45.679 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.679 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.679 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:45.679 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.569346+0000", 2024-02-09T17:42:45.679 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015229100000000001 2024-02-09T17:42:45.679 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.679 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.679 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,3,8", 2024-02-09T17:42:45.679 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.569499+0000", 2024-02-09T17:42:45.680 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015261800000000001 2024-02-09T17:42:45.680 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.680 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.680 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:45.680 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.574756+0000", 2024-02-09T17:42:45.680 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0052577450000000003 2024-02-09T17:42:45.680 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.680 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.680 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:45.680 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.579861+0000", 2024-02-09T17:42:45.680 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0051049820000000001 2024-02-09T17:42:45.680 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.680 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.680 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:45.680 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.600256+0000", 2024-02-09T17:42:45.681 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020394366000000001 2024-02-09T17:42:45.681 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.681 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.681 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:42:45.681 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.600266+0000", 2024-02-09T17:42:45.681 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0159e-05 2024-02-09T17:42:45.681 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.681 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.681 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:45.681 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.757744+0000", 2024-02-09T17:42:45.681 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1574786500000001 2024-02-09T17:42:45.681 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.681 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.681 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:45.681 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.757819+0000", 2024-02-09T17:42:45.682 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4158000000000004e-05 2024-02-09T17:42:45.682 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:45.682 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:45.682 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:45.682 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.682 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.682 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25201.0:222249 4.0 4:027cf6cc:::benchmark_data_smithi179_103809_object84783:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e397)", 2024-02-09T17:42:45.682 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:39:02.574349+0000", 2024-02-09T17:42:45.682 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 223.07338168300001, 2024-02-09T17:42:45.682 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1835428379999999, 2024-02-09T17:42:45.682 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:45.682 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-09T17:42:45.682 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:45.682 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25201", 2024-02-09T17:42:45.682 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:42:45.682 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 222249 2024-02-09T17:42:45.683 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.683 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:45.683 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.683 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:45.683 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.574349+0000", 2024-02-09T17:42:45.683 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:45.683 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.683 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.683 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:45.683 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.574349+0000", 2024-02-09T17:42:45.683 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:45.683 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.683 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.683 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:45.684 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.574350+0000", 2024-02-09T17:42:45.684 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.285e-06 2024-02-09T17:42:45.684 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.684 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.684 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:45.684 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.574354+0000", 2024-02-09T17:42:45.684 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.143e-06 2024-02-09T17:42:45.684 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.684 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.684 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:45.684 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.574363+0000", 2024-02-09T17:42:45.684 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8480000000000007e-06 2024-02-09T17:42:45.684 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.684 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.685 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:45.685 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.574368+0000", 2024-02-09T17:42:45.685 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9439999999999996e-06 2024-02-09T17:42:45.685 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.685 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.685 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:45.685 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.574792+0000", 2024-02-09T17:42:45.685 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00042435299999999998 2024-02-09T17:42:45.685 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.685 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.685 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:45.685 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.574903+0000", 2024-02-09T17:42:45.685 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000110776 2024-02-09T17:42:45.685 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.685 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.685 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,3,8", 2024-02-09T17:42:45.686 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.574995+0000", 2024-02-09T17:42:45.686 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2085999999999999e-05 2024-02-09T17:42:45.686 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.686 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.686 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:45.686 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.579847+0000", 2024-02-09T17:42:45.686 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0048517150000000004 2024-02-09T17:42:45.686 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.686 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.686 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:45.686 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.579904+0000", 2024-02-09T17:42:45.686 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7757000000000003e-05 2024-02-09T17:42:45.686 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.686 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.686 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:45.687 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.600297+0000", 2024-02-09T17:42:45.687 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020392484999999998 2024-02-09T17:42:45.687 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.687 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.687 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:42:45.687 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.600306+0000", 2024-02-09T17:42:45.687 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9789999999999999e-06 2024-02-09T17:42:45.687 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.687 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.687 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:45.687 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.757848+0000", 2024-02-09T17:42:45.687 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.157542262 2024-02-09T17:42:45.687 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.687 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.687 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:45.688 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.757891+0000", 2024-02-09T17:42:45.688 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3204999999999997e-05 2024-02-09T17:42:45.688 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:45.688 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:45.688 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:45.688 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.688 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.688 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25201.0:222256 4.3 4:c3fd71ca:::benchmark_data_smithi179_103809_object84790:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e397)", 2024-02-09T17:42:45.688 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:39:02.584411+0000", 2024-02-09T17:42:45.688 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 223.06331970100001, 2024-02-09T17:42:45.688 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1717092849999999, 2024-02-09T17:42:45.688 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:45.688 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-09T17:42:45.688 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:45.688 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25201", 2024-02-09T17:42:45.689 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:42:45.689 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 222256 2024-02-09T17:42:45.689 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.689 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:45.689 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.689 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:45.689 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.584411+0000", 2024-02-09T17:42:45.689 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:45.689 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.689 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.689 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:45.689 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.584411+0000", 2024-02-09T17:42:45.689 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:45.689 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.689 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.690 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:45.690 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.584412+0000", 2024-02-09T17:42:45.690 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5e-06 2024-02-09T17:42:45.690 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.690 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.690 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:45.690 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.584417+0000", 2024-02-09T17:42:45.690 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8180000000000003e-06 2024-02-09T17:42:45.690 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.690 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.690 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:45.690 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.584431+0000", 2024-02-09T17:42:45.690 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3862e-05 2024-02-09T17:42:45.690 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.690 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.691 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:45.691 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.584439+0000", 2024-02-09T17:42:45.691 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6780000000000006e-06 2024-02-09T17:42:45.691 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.691 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.691 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:45.691 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.584498+0000", 2024-02-09T17:42:45.691 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8272e-05 2024-02-09T17:42:45.691 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.691 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.691 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:45.691 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.584614+0000", 2024-02-09T17:42:45.691 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011599699999999999 2024-02-09T17:42:45.691 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.691 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.691 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,1", 2024-02-09T17:42:45.692 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.584683+0000", 2024-02-09T17:42:45.692 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9062999999999999e-05 2024-02-09T17:42:45.692 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.692 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.692 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:45.692 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.593639+0000", 2024-02-09T17:42:45.692 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0089563309999999997 2024-02-09T17:42:45.692 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.692 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.692 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:45.692 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.598755+0000", 2024-02-09T17:42:45.692 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0051158380000000002 2024-02-09T17:42:45.692 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.692 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.693 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:42:45.693 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.598769+0000", 2024-02-09T17:42:45.693 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4105e-05 2024-02-09T17:42:45.693 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.693 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.693 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:45.693 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.591680+0000", 2024-02-09T17:42:45.693 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99291052099999999 2024-02-09T17:42:45.693 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.693 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.693 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:42:45.693 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.591696+0000", 2024-02-09T17:42:45.693 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6966999999999999e-05 2024-02-09T17:42:45.693 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.693 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.694 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:45.694 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.756030+0000", 2024-02-09T17:42:45.694 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.16433393 2024-02-09T17:42:45.694 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.694 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.694 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:45.694 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.756120+0000", 2024-02-09T17:42:45.694 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9402999999999998e-05 2024-02-09T17:42:45.694 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:45.694 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:45.694 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:45.694 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.694 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.694 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25201.0:222267 4.3 4:ce2ff2fd:::benchmark_data_smithi179_103809_object84801:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-02-09T17:42:45.694 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:39:02.602612+0000", 2024-02-09T17:42:45.695 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 223.045118006, 2024-02-09T17:42:45.695 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.181128046, 2024-02-09T17:42:45.695 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:45.695 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:42:45.695 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:45.695 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25201", 2024-02-09T17:42:45.695 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:42:45.695 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 222267 2024-02-09T17:42:45.695 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.695 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:45.695 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.695 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:45.695 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.602612+0000", 2024-02-09T17:42:45.695 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:45.695 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.696 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.696 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:45.696 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.602612+0000", 2024-02-09T17:42:45.696 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:45.696 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.696 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.696 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:45.696 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.602614+0000", 2024-02-09T17:42:45.696 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9e-06 2024-02-09T17:42:45.696 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.696 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.696 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:45.696 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.602618+0000", 2024-02-09T17:42:45.696 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3590000000000001e-06 2024-02-09T17:42:45.697 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.697 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.697 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:45.697 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.602630+0000", 2024-02-09T17:42:45.697 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1249e-05 2024-02-09T17:42:45.697 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.697 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.697 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:45.697 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.602638+0000", 2024-02-09T17:42:45.697 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8739999999999995e-06 2024-02-09T17:42:45.697 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.697 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.697 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:45.697 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.602686+0000", 2024-02-09T17:42:45.697 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8755000000000003e-05 2024-02-09T17:42:45.698 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.698 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.698 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:42:45.698 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.602694+0000", 2024-02-09T17:42:45.698 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6550000000000004e-06 2024-02-09T17:42:45.698 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.698 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.698 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:45.698 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.591737+0000", 2024-02-09T17:42:45.698 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.98904337399999998 2024-02-09T17:42:45.698 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.698 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.698 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:42:45.698 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.591748+0000", 2024-02-09T17:42:45.699 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0465000000000001e-05 2024-02-09T17:42:45.699 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.699 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.699 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:45.699 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.756361+0000", 2024-02-09T17:42:45.699 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.164612765 2024-02-09T17:42:45.699 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.699 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.699 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:45.699 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.756495+0000", 2024-02-09T17:42:45.699 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013414000000000001 2024-02-09T17:42:45.699 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.699 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.699 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,1,2", 2024-02-09T17:42:45.699 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.756646+0000", 2024-02-09T17:42:45.700 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000151261 2024-02-09T17:42:45.700 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.700 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.700 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:45.700 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.773391+0000", 2024-02-09T17:42:45.700 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.016744886 2024-02-09T17:42:45.700 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.700 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.700 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:45.700 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.773598+0000", 2024-02-09T17:42:45.700 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020660700000000001 2024-02-09T17:42:45.700 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.700 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.700 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:45.701 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.773681+0000", 2024-02-09T17:42:45.701 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3466000000000004e-05 2024-02-09T17:42:45.701 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.701 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.701 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:45.701 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.783618+0000", 2024-02-09T17:42:45.701 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0099368900000000003 2024-02-09T17:42:45.701 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.701 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.701 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:42:45.701 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.783658+0000", 2024-02-09T17:42:45.701 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9817000000000002e-05 2024-02-09T17:42:45.701 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.701 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.702 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:45.702 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.783740+0000", 2024-02-09T17:42:45.702 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2583000000000005e-05 2024-02-09T17:42:45.702 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:45.702 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:45.702 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:45.702 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.702 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.702 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25201.0:222271 4.3 4:c7c890ae:::benchmark_data_smithi179_103809_object84805:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-02-09T17:42:45.702 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:39:02.609626+0000", 2024-02-09T17:42:45.702 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 223.03810465399999, 2024-02-09T17:42:45.702 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.187357123, 2024-02-09T17:42:45.702 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:45.702 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:42:45.702 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:45.703 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25201", 2024-02-09T17:42:45.703 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:42:45.703 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 222271 2024-02-09T17:42:45.703 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.703 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:45.703 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.703 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:45.703 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.609626+0000", 2024-02-09T17:42:45.703 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:45.703 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.703 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.703 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:45.703 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.609626+0000", 2024-02-09T17:42:45.703 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:45.703 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.704 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.704 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:45.704 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.609627+0000", 2024-02-09T17:42:45.704 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.73e-06 2024-02-09T17:42:45.704 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.704 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.704 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:45.704 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.609634+0000", 2024-02-09T17:42:45.704 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3899999999999998e-06 2024-02-09T17:42:45.704 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.704 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.704 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:45.704 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.609645+0000", 2024-02-09T17:42:45.704 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1280999999999999e-05 2024-02-09T17:42:45.705 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.705 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.705 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:45.705 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.609653+0000", 2024-02-09T17:42:45.705 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1359999999999997e-06 2024-02-09T17:42:45.705 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.705 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.705 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:45.705 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.609708+0000", 2024-02-09T17:42:45.705 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4938000000000002e-05 2024-02-09T17:42:45.705 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.705 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.705 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:42:45.705 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.609720+0000", 2024-02-09T17:42:45.705 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1721e-05 2024-02-09T17:42:45.706 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.706 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.706 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:45.706 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.591779+0000", 2024-02-09T17:42:45.706 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.98205883800000005 2024-02-09T17:42:45.706 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.706 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.706 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:42:45.706 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.591788+0000", 2024-02-09T17:42:45.706 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9360000000000005e-06 2024-02-09T17:42:45.706 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.706 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.706 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:45.706 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.761893+0000", 2024-02-09T17:42:45.706 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.170105886 2024-02-09T17:42:45.707 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.707 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.707 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:45.707 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.762041+0000", 2024-02-09T17:42:45.707 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014795300000000001 2024-02-09T17:42:45.707 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.707 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.707 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,1,2", 2024-02-09T17:42:45.707 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.762214+0000", 2024-02-09T17:42:45.707 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017221700000000001 2024-02-09T17:42:45.707 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.707 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.707 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:45.707 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.773636+0000", 2024-02-09T17:42:45.708 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011422453000000001 2024-02-09T17:42:45.708 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.708 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.708 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:45.708 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.783450+0000", 2024-02-09T17:42:45.708 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0098142679999999993 2024-02-09T17:42:45.708 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.708 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.708 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:45.708 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.783582+0000", 2024-02-09T17:42:45.708 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013120100000000001 2024-02-09T17:42:45.708 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.708 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.708 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:45.708 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.796855+0000", 2024-02-09T17:42:45.709 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.01327303 2024-02-09T17:42:45.709 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.709 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.709 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:42:45.709 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.796901+0000", 2024-02-09T17:42:45.709 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6372000000000003e-05 2024-02-09T17:42:45.709 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.709 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.709 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:45.709 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.796983+0000", 2024-02-09T17:42:45.709 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1773000000000005e-05 2024-02-09T17:42:45.709 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:45.709 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:45.709 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:45.709 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.710 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.710 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25201.0:222272 4.3 4:c47dff37:::benchmark_data_smithi179_103809_object84806:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-02-09T17:42:45.710 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:39:02.609785+0000", 2024-02-09T17:42:45.710 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 223.03794480900001, 2024-02-09T17:42:45.710 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1955497909999999, 2024-02-09T17:42:45.710 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:45.710 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:42:45.710 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:45.710 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25201", 2024-02-09T17:42:45.710 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:42:45.710 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 222272 2024-02-09T17:42:45.710 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.710 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:45.710 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.710 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:45.711 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.609785+0000", 2024-02-09T17:42:45.711 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:45.711 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.711 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.711 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:45.711 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.609785+0000", 2024-02-09T17:42:45.711 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:45.711 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.711 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.711 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:45.711 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.609786+0000", 2024-02-09T17:42:45.711 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6000000000000003e-07 2024-02-09T17:42:45.711 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.711 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.711 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:45.711 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.609789+0000", 2024-02-09T17:42:45.712 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3500000000000001e-06 2024-02-09T17:42:45.712 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.712 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.712 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:45.712 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.609796+0000", 2024-02-09T17:42:45.712 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9419999999999998e-06 2024-02-09T17:42:45.712 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.712 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.712 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:45.712 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.609803+0000", 2024-02-09T17:42:45.712 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5400000000000001e-06 2024-02-09T17:42:45.712 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.712 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.712 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:45.713 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.609844+0000", 2024-02-09T17:42:45.713 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1131999999999999e-05 2024-02-09T17:42:45.713 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.713 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.713 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:42:45.713 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.609849+0000", 2024-02-09T17:42:45.713 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4260000000000002e-06 2024-02-09T17:42:45.713 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.713 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.713 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:45.713 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.591818+0000", 2024-02-09T17:42:45.713 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.98196833100000003 2024-02-09T17:42:45.713 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.713 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.713 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:42:45.714 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.591826+0000", 2024-02-09T17:42:45.714 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4020000000000002e-06 2024-02-09T17:42:45.714 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.714 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.714 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:45.714 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.773712+0000", 2024-02-09T17:42:45.714 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1818854089999999 2024-02-09T17:42:45.714 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.714 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.714 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:45.714 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.773857+0000", 2024-02-09T17:42:45.714 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014581699999999999 2024-02-09T17:42:45.714 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.714 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.714 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,1,2", 2024-02-09T17:42:45.715 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.774041+0000", 2024-02-09T17:42:45.715 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018403000000000001 2024-02-09T17:42:45.715 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.715 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.715 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:45.715 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.783586+0000", 2024-02-09T17:42:45.715 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0095448789999999992 2024-02-09T17:42:45.715 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.715 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.715 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:45.715 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.783776+0000", 2024-02-09T17:42:45.715 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000189629 2024-02-09T17:42:45.715 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.715 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.715 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:45.715 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.796812+0000", 2024-02-09T17:42:45.716 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013036030000000001 2024-02-09T17:42:45.716 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.716 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.716 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:45.716 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.805127+0000", 2024-02-09T17:42:45.716 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0083153889999999994 2024-02-09T17:42:45.716 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.716 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.716 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:42:45.716 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.805195+0000", 2024-02-09T17:42:45.716 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7984999999999994e-05 2024-02-09T17:42:45.716 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.716 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.717 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:45.717 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.805335+0000", 2024-02-09T17:42:45.717 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013983999999999999 2024-02-09T17:42:45.717 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:45.717 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:45.717 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:45.717 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.717 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.717 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25201.0:222279 4.0 4:01a3b9da:::benchmark_data_smithi179_103809_object84813:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-02-09T17:42:45.717 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:39:02.621042+0000", 2024-02-09T17:42:45.717 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 223.02668788599999, 2024-02-09T17:42:45.717 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1469982990000001, 2024-02-09T17:42:45.717 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:45.717 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:42:45.717 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:45.717 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25201", 2024-02-09T17:42:45.718 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:42:45.718 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 222279 2024-02-09T17:42:45.718 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.718 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:45.718 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.718 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:45.718 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.621042+0000", 2024-02-09T17:42:45.718 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:45.718 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.718 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.718 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:45.718 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.621042+0000", 2024-02-09T17:42:45.718 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:45.718 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.718 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.719 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:45.719 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.621044+0000", 2024-02-09T17:42:45.719 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5030000000000001e-06 2024-02-09T17:42:45.719 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.719 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.719 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:45.719 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.621051+0000", 2024-02-09T17:42:45.719 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.109e-06 2024-02-09T17:42:45.719 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.719 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.719 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:45.719 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.621064+0000", 2024-02-09T17:42:45.719 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3125999999999999e-05 2024-02-09T17:42:45.719 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.720 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.720 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:45.720 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.621072+0000", 2024-02-09T17:42:45.720 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1159999999999999e-06 2024-02-09T17:42:45.720 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.720 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.720 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:45.720 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.621130+0000", 2024-02-09T17:42:45.720 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8037e-05 2024-02-09T17:42:45.720 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.720 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.720 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:42:45.720 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.621142+0000", 2024-02-09T17:42:45.720 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1532000000000001e-05 2024-02-09T17:42:45.721 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.721 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.721 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:45.721 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.758043+0000", 2024-02-09T17:42:45.721 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.136901436 2024-02-09T17:42:45.721 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.721 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.721 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:45.721 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.758117+0000", 2024-02-09T17:42:45.721 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3453000000000003e-05 2024-02-09T17:42:45.721 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.721 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.721 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,3,8,9", 2024-02-09T17:42:45.721 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.758211+0000", 2024-02-09T17:42:45.721 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4186999999999998e-05 2024-02-09T17:42:45.722 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:45.722 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.762702+0000", 2024-02-09T17:42:45.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.004491554 2024-02-09T17:42:45.722 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:45.722 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.762744+0000", 2024-02-09T17:42:45.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1489000000000002e-05 2024-02-09T17:42:45.722 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:45.722 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.762767+0000", 2024-02-09T17:42:45.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3238999999999999e-05 2024-02-09T17:42:45.723 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.723 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.723 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:45.723 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.767870+0000", 2024-02-09T17:42:45.723 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0051030479999999998 2024-02-09T17:42:45.723 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.723 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.723 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:45.723 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.767911+0000", 2024-02-09T17:42:45.723 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0330000000000002e-05 2024-02-09T17:42:45.723 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.723 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.723 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:42:45.723 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.767952+0000", 2024-02-09T17:42:45.724 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1142e-05 2024-02-09T17:42:45.724 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.724 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.724 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:45.724 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.768041+0000", 2024-02-09T17:42:45.724 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8998000000000004e-05 2024-02-09T17:42:45.724 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:45.724 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:45.724 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:45.724 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.724 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.724 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25201.0:222281 4.3 4:c47355be:::benchmark_data_smithi179_103809_object84815:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-02-09T17:42:45.724 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:39:02.629982+0000", 2024-02-09T17:42:45.724 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 223.01774786799999, 2024-02-09T17:42:45.724 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.175573108, 2024-02-09T17:42:45.725 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:45.725 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:42:45.725 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:45.725 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25201", 2024-02-09T17:42:45.725 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:42:45.725 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 222281 2024-02-09T17:42:45.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.725 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:45.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:45.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.629982+0000", 2024-02-09T17:42:45.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:45.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:45.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.629982+0000", 2024-02-09T17:42:45.726 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:45.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.726 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.726 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:45.726 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.629983+0000", 2024-02-09T17:42:45.726 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1900000000000001e-07 2024-02-09T17:42:45.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.726 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.726 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:45.726 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.629987+0000", 2024-02-09T17:42:45.726 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.433e-06 2024-02-09T17:42:45.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.726 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.726 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:45.726 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.629997+0000", 2024-02-09T17:42:45.727 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0378e-05 2024-02-09T17:42:45.727 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.727 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.727 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:45.727 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.630004+0000", 2024-02-09T17:42:45.727 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.199e-06 2024-02-09T17:42:45.727 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.727 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.727 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:45.727 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.630082+0000", 2024-02-09T17:42:45.727 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7980999999999998e-05 2024-02-09T17:42:45.727 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.727 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.727 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:42:45.727 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.630089+0000", 2024-02-09T17:42:45.727 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0049999999999998e-06 2024-02-09T17:42:45.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.728 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.728 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:45.728 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.591871+0000", 2024-02-09T17:42:45.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.96178198000000004 2024-02-09T17:42:45.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.728 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.728 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:42:45.728 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.591884+0000", 2024-02-09T17:42:45.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2564e-05 2024-02-09T17:42:45.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.728 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.728 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:45.728 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.779201+0000", 2024-02-09T17:42:45.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1873176409999999 2024-02-09T17:42:45.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:45.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.779373+0000", 2024-02-09T17:42:45.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017127099999999999 2024-02-09T17:42:45.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,1,2", 2024-02-09T17:42:45.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.779494+0000", 2024-02-09T17:42:45.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012153400000000001 2024-02-09T17:42:45.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:45.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.783784+0000", 2024-02-09T17:42:45.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042902870000000003 2024-02-09T17:42:45.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.730 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:45.730 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.783815+0000", 2024-02-09T17:42:45.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0593000000000003e-05 2024-02-09T17:42:45.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.730 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:45.730 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.797043+0000", 2024-02-09T17:42:45.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013227756 2024-02-09T17:42:45.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.730 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:45.730 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.805380+0000", 2024-02-09T17:42:45.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0083370270000000003 2024-02-09T17:42:45.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:42:45.731 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.805434+0000", 2024-02-09T17:42:45.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4137e-05 2024-02-09T17:42:45.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:45.731 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.805555+0000", 2024-02-09T17:42:45.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012150300000000001 2024-02-09T17:42:45.731 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:45.731 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:45.731 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:45.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.732 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25201.0:222289 4.3 4:c1824169:::benchmark_data_smithi179_103809_object84823:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e399)", 2024-02-09T17:42:45.732 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:39:03.642378+0000", 2024-02-09T17:42:45.732 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 222.00535255599999, 2024-02-09T17:42:45.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.163390887, 2024-02-09T17:42:45.732 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:45.732 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:42:45.732 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:45.732 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25201", 2024-02-09T17:42:45.732 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:42:45.732 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 222289 2024-02-09T17:42:45.732 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.732 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:45.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.732 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:45.732 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.642378+0000", 2024-02-09T17:42:45.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:45.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:45.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.642378+0000", 2024-02-09T17:42:45.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:45.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:45.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.642379+0000", 2024-02-09T17:42:45.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7179999999999999e-06 2024-02-09T17:42:45.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:45.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.642385+0000", 2024-02-09T17:42:45.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4870000000000002e-06 2024-02-09T17:42:45.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:45.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.642399+0000", 2024-02-09T17:42:45.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3694e-05 2024-02-09T17:42:45.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:45.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.642406+0000", 2024-02-09T17:42:45.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2779999999999996e-06 2024-02-09T17:42:45.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:45.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.642471+0000", 2024-02-09T17:42:45.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5007999999999998e-05 2024-02-09T17:42:45.735 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.735 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.735 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:42:45.735 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.642479+0000", 2024-02-09T17:42:45.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4889999999999995e-06 2024-02-09T17:42:45.735 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.735 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.735 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:45.735 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.783839+0000", 2024-02-09T17:42:45.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1413593099999999 2024-02-09T17:42:45.735 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.735 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.735 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:45.735 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.783920+0000", 2024-02-09T17:42:45.736 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0987000000000002e-05 2024-02-09T17:42:45.736 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.736 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.736 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,1,2", 2024-02-09T17:42:45.736 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.784036+0000", 2024-02-09T17:42:45.736 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000116282 2024-02-09T17:42:45.736 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.736 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.736 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:45.736 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.796755+0000", 2024-02-09T17:42:45.736 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012719212000000001 2024-02-09T17:42:45.736 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.736 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.737 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:45.737 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.796993+0000", 2024-02-09T17:42:45.737 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000237655 2024-02-09T17:42:45.737 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.737 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.737 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:45.737 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.797077+0000", 2024-02-09T17:42:45.737 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3941000000000002e-05 2024-02-09T17:42:45.737 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.737 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.737 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:45.737 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.805594+0000", 2024-02-09T17:42:45.737 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0085175530000000006 2024-02-09T17:42:45.737 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.737 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:42:45.738 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.805647+0000", 2024-02-09T17:42:45.738 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2697e-05 2024-02-09T17:42:45.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:45.738 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.805769+0000", 2024-02-09T17:42:45.738 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000121576 2024-02-09T17:42:45.738 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:45.738 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:45.738 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:45.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.738 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25201.0:232546 4.3 4:c59348e0:::benchmark_data_smithi179_103809_object95080:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e414)", 2024-02-09T17:42:45.738 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:39:23.603335+0000", 2024-02-09T17:42:45.738 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 202.04439519900001, 2024-02-09T17:42:45.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0045989610000001, 2024-02-09T17:42:45.739 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:45.739 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:42:45.739 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:45.739 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25201", 2024-02-09T17:42:45.739 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:42:45.739 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 232546 2024-02-09T17:42:45.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.739 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:45.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.739 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:45.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.603335+0000", 2024-02-09T17:42:45.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:45.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.740 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.740 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:45.740 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.603335+0000", 2024-02-09T17:42:45.740 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:45.740 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.740 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.740 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:45.740 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.603336+0000", 2024-02-09T17:42:45.740 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.381e-06 2024-02-09T17:42:45.740 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.740 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.740 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:45.740 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.603340+0000", 2024-02-09T17:42:45.740 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7740000000000002e-06 2024-02-09T17:42:45.740 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.740 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:45.741 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.603350+0000", 2024-02-09T17:42:45.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5410000000000006e-06 2024-02-09T17:42:45.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:45.741 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.603357+0000", 2024-02-09T17:42:45.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4089999999999997e-06 2024-02-09T17:42:45.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:45.741 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.603404+0000", 2024-02-09T17:42:45.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6493999999999999e-05 2024-02-09T17:42:45.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:42:45.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.603410+0000", 2024-02-09T17:42:45.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8090000000000004e-06 2024-02-09T17:42:45.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:45.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:24.529677+0000", 2024-02-09T17:42:45.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.92626684500000001 2024-02-09T17:42:45.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:42:45.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:24.529688+0000", 2024-02-09T17:42:45.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0865e-05 2024-02-09T17:42:45.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:45.743 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.580256+0000", 2024-02-09T17:42:45.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.050567866 2024-02-09T17:42:45.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.743 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:45.743 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.580352+0000", 2024-02-09T17:42:45.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5659e-05 2024-02-09T17:42:45.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.743 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,1", 2024-02-09T17:42:45.743 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.580433+0000", 2024-02-09T17:42:45.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1678e-05 2024-02-09T17:42:45.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.744 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:45.744 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.591780+0000", 2024-02-09T17:42:45.744 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011347036 2024-02-09T17:42:45.744 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.744 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.744 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:45.744 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.593996+0000", 2024-02-09T17:42:45.744 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0022152280000000001 2024-02-09T17:42:45.744 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.744 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.744 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:45.744 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.607827+0000", 2024-02-09T17:42:45.744 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013831256 2024-02-09T17:42:45.744 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.745 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.745 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:42:45.745 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.607889+0000", 2024-02-09T17:42:45.745 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2318999999999997e-05 2024-02-09T17:42:45.745 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.745 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.745 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:45.745 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.607934+0000", 2024-02-09T17:42:45.745 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4801e-05 2024-02-09T17:42:45.745 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:45.745 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:45.745 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:45.745 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.745 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.746 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25201.0:232547 4.3 4:cbba7e36:::benchmark_data_smithi179_103809_object95081:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e414)", 2024-02-09T17:42:45.746 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:39:23.603392+0000", 2024-02-09T17:42:45.746 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 202.044338009, 2024-02-09T17:42:45.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0046948050000002, 2024-02-09T17:42:45.746 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:45.746 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:42:45.746 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:45.746 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25201", 2024-02-09T17:42:45.746 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:42:45.746 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 232547 2024-02-09T17:42:45.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.746 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:45.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.746 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:45.746 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.603392+0000", 2024-02-09T17:42:45.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:45.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.747 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:45.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.603392+0000", 2024-02-09T17:42:45.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:45.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.747 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:45.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.603393+0000", 2024-02-09T17:42:45.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1999999999999999e-07 2024-02-09T17:42:45.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.747 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:45.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.603396+0000", 2024-02-09T17:42:45.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.354e-06 2024-02-09T17:42:45.748 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.748 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.748 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:45.748 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.603404+0000", 2024-02-09T17:42:45.748 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.011e-06 2024-02-09T17:42:45.748 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.748 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.748 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:45.748 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.603411+0000", 2024-02-09T17:42:45.748 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8979999999999999e-06 2024-02-09T17:42:45.748 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.748 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.748 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:45.748 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.603445+0000", 2024-02-09T17:42:45.749 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4242999999999997e-05 2024-02-09T17:42:45.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.749 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.749 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:42:45.749 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.603451+0000", 2024-02-09T17:42:45.749 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7470000000000002e-06 2024-02-09T17:42:45.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.749 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.749 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:45.749 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:24.529775+0000", 2024-02-09T17:42:45.749 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.92632368899999995 2024-02-09T17:42:45.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.749 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.749 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:42:45.749 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:24.529783+0000", 2024-02-09T17:42:45.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1089999999999998e-06 2024-02-09T17:42:45.750 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.750 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:45.750 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.584396+0000", 2024-02-09T17:42:45.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0546128969999999 2024-02-09T17:42:45.750 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.750 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:45.750 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.584474+0000", 2024-02-09T17:42:45.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8423000000000005e-05 2024-02-09T17:42:45.750 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.750 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,1", 2024-02-09T17:42:45.750 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.584546+0000", 2024-02-09T17:42:45.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1859000000000004e-05 2024-02-09T17:42:45.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:45.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.594072+0000", 2024-02-09T17:42:45.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0095256960000000002 2024-02-09T17:42:45.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:45.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.603750+0000", 2024-02-09T17:42:45.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0096781599999999999 2024-02-09T17:42:45.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:45.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.607989+0000", 2024-02-09T17:42:45.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042386430000000003 2024-02-09T17:42:45.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.752 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:42:45.752 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.608047+0000", 2024-02-09T17:42:45.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8656999999999998e-05 2024-02-09T17:42:45.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.752 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:45.752 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.608087+0000", 2024-02-09T17:42:45.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9699000000000001e-05 2024-02-09T17:42:45.752 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:45.752 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:45.752 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:45.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.753 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.753 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25201.0:232571 4.3 4:c89fdab4:::benchmark_data_smithi179_103809_object95105:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e414)", 2024-02-09T17:42:45.753 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:39:23.637814+0000", 2024-02-09T17:42:45.753 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 202.00991634499999, 2024-02-09T17:42:45.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9704033760000002, 2024-02-09T17:42:45.753 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:45.753 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:42:45.753 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:45.753 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25201", 2024-02-09T17:42:45.753 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:42:45.753 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 232571 2024-02-09T17:42:45.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.753 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:45.753 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:45.754 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.637814+0000", 2024-02-09T17:42:45.754 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:45.754 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.754 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.754 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:45.754 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.637814+0000", 2024-02-09T17:42:45.754 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:45.754 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.754 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.754 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:45.754 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.637815+0000", 2024-02-09T17:42:45.754 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3969999999999999e-06 2024-02-09T17:42:45.754 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.754 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.754 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:45.755 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.637819+0000", 2024-02-09T17:42:45.755 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4759999999999998e-06 2024-02-09T17:42:45.755 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.755 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.755 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:45.755 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.637830+0000", 2024-02-09T17:42:45.755 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0916e-05 2024-02-09T17:42:45.755 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.755 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.755 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:45.755 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.637837+0000", 2024-02-09T17:42:45.755 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2030000000000003e-06 2024-02-09T17:42:45.755 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.755 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.755 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:45.756 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.637887+0000", 2024-02-09T17:42:45.756 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0065000000000003e-05 2024-02-09T17:42:45.756 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.756 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.756 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:42:45.756 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.637898+0000", 2024-02-09T17:42:45.756 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1039000000000001e-05 2024-02-09T17:42:45.756 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.756 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.756 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:45.756 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:24.529809+0000", 2024-02-09T17:42:45.756 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.891911385 2024-02-09T17:42:45.756 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.756 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:42:45.757 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:24.529817+0000", 2024-02-09T17:42:45.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1840000000000002e-06 2024-02-09T17:42:45.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:45.757 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.587165+0000", 2024-02-09T17:42:45.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0573482510000001 2024-02-09T17:42:45.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:45.757 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.587281+0000", 2024-02-09T17:42:45.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000116487 2024-02-09T17:42:45.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.758 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,1", 2024-02-09T17:42:45.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.587352+0000", 2024-02-09T17:42:45.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1019000000000006e-05 2024-02-09T17:42:45.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.758 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:45.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.594075+0000", 2024-02-09T17:42:45.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0067228770000000004 2024-02-09T17:42:45.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.758 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:45.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.603789+0000", 2024-02-09T17:42:45.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.009713579 2024-02-09T17:42:45.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.759 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:45.759 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.608128+0000", 2024-02-09T17:42:45.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0043391560000000003 2024-02-09T17:42:45.759 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.759 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:42:45.759 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.608182+0000", 2024-02-09T17:42:45.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4265000000000003e-05 2024-02-09T17:42:45.759 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.759 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:45.759 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.608217+0000", 2024-02-09T17:42:45.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5077000000000003e-05 2024-02-09T17:42:45.759 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:45.759 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:45.760 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:45.760 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.760 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.760 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25201.0:232587 4.3 4:cd317105:::benchmark_data_smithi179_103809_object95121:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-02-09T17:42:45.760 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:39:24.611809+0000", 2024-02-09T17:42:45.760 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 201.035921696, 2024-02-09T17:42:45.760 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0016254849999999, 2024-02-09T17:42:45.760 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:45.760 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:42:45.760 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:45.760 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25201", 2024-02-09T17:42:45.760 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:42:45.760 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 232587 2024-02-09T17:42:45.760 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.761 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:45.761 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.761 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:45.761 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:24.611809+0000", 2024-02-09T17:42:45.761 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:45.761 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.761 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.761 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:45.761 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:24.611809+0000", 2024-02-09T17:42:45.761 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:45.761 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.761 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.761 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:45.761 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:24.611811+0000", 2024-02-09T17:42:45.761 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.774e-06 2024-02-09T17:42:45.761 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.762 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.762 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:45.762 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:24.611823+0000", 2024-02-09T17:42:45.762 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1912999999999999e-05 2024-02-09T17:42:45.762 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.762 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.762 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:45.762 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:24.611855+0000", 2024-02-09T17:42:45.762 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1776000000000002e-05 2024-02-09T17:42:45.762 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.762 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.762 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:45.762 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:24.611872+0000", 2024-02-09T17:42:45.762 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6840000000000001e-05 2024-02-09T17:42:45.762 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.763 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:45.763 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:24.612018+0000", 2024-02-09T17:42:45.763 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014583 2024-02-09T17:42:45.763 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.763 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:42:45.763 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:24.612051+0000", 2024-02-09T17:42:45.763 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3589e-05 2024-02-09T17:42:45.763 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.763 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:45.763 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.589562+0000", 2024-02-09T17:42:45.763 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.97751064200000004 2024-02-09T17:42:45.763 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.764 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:45.764 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.589653+0000", 2024-02-09T17:42:45.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1230000000000006e-05 2024-02-09T17:42:45.764 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.764 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.764 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,1", 2024-02-09T17:42:45.764 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.589727+0000", 2024-02-09T17:42:45.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3543999999999999e-05 2024-02-09T17:42:45.764 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.764 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.764 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:45.764 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.594078+0000", 2024-02-09T17:42:45.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.004351382 2024-02-09T17:42:45.764 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.764 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:45.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.603975+0000", 2024-02-09T17:42:45.765 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0098968300000000006 2024-02-09T17:42:45.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:45.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.613332+0000", 2024-02-09T17:42:45.765 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0093571030000000003 2024-02-09T17:42:45.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:42:45.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.613375+0000", 2024-02-09T17:42:45.765 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2622e-05 2024-02-09T17:42:45.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.766 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:45.766 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.613434+0000", 2024-02-09T17:42:45.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9410000000000002e-05 2024-02-09T17:42:45.766 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:45.766 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:45.766 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:45.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.766 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25201.0:232592 4.3 4:c7fda831:::benchmark_data_smithi179_103809_object95126:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-02-09T17:42:45.766 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:39:24.728619+0000", 2024-02-09T17:42:45.766 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 200.91911146699999, 2024-02-09T17:42:45.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.89123529499999998, 2024-02-09T17:42:45.766 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:45.766 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:42:45.766 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:45.767 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25201", 2024-02-09T17:42:45.767 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:45.768 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:42:45.768 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 232592 2024-02-09T17:42:45.768 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.768 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:45.768 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.768 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:45.768 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:24.728619+0000", 2024-02-09T17:42:45.768 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:45.768 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.768 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.768 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:45.768 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:24.728619+0000", 2024-02-09T17:42:45.768 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:45.768 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.769 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.769 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:45.769 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:24.728620+0000", 2024-02-09T17:42:45.769 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3629999999999999e-06 2024-02-09T17:42:45.769 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.769 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.769 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:45.769 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:24.728625+0000", 2024-02-09T17:42:45.769 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1669999999999998e-06 2024-02-09T17:42:45.769 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.769 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.769 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:45.769 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:24.728637+0000", 2024-02-09T17:42:45.769 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1803e-05 2024-02-09T17:42:45.769 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:45.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:24.728660+0000", 2024-02-09T17:42:45.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3323e-05 2024-02-09T17:42:45.770 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:45.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:24.728711+0000", 2024-02-09T17:42:45.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0268000000000001e-05 2024-02-09T17:42:45.770 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:42:45.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:24.728718+0000", 2024-02-09T17:42:45.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4220000000000003e-06 2024-02-09T17:42:45.770 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.771 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:45.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.591812+0000", 2024-02-09T17:42:45.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.86309382099999998 2024-02-09T17:42:45.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.771 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:45.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.591885+0000", 2024-02-09T17:42:45.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3403000000000002e-05 2024-02-09T17:42:45.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.771 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,1", 2024-02-09T17:42:45.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.591953+0000", 2024-02-09T17:42:45.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7949999999999998e-05 2024-02-09T17:42:45.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.771 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:45.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.594142+0000", 2024-02-09T17:42:45.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021890770000000002 2024-02-09T17:42:45.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.772 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.772 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:45.772 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.604012+0000", 2024-02-09T17:42:45.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0098694030000000006 2024-02-09T17:42:45.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.772 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.772 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:45.772 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.619792+0000", 2024-02-09T17:42:45.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015779850000000002 2024-02-09T17:42:45.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.772 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.772 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:42:45.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.619828+0000", 2024-02-09T17:42:45.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6284e-05 2024-02-09T17:42:45.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.773 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:45.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.619854+0000", 2024-02-09T17:42:45.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6160999999999999e-05 2024-02-09T17:42:45.773 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:45.773 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:45.773 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:45.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.773 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:126294 5.9 5:96d777d6:::benchmark_data_smithi179_115199_object126293:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e501)", 2024-02-09T17:42:45.773 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:42:31.641173+0000", 2024-02-09T17:42:45.773 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 14.006557233000001, 2024-02-09T17:42:45.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.226279324, 2024-02-09T17:42:45.774 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:45.774 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:42:45.774 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:45.774 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:42:45.774 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:42:45.774 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 126294 2024-02-09T17:42:45.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.774 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:45.774 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.774 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:45.774 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.641173+0000", 2024-02-09T17:42:45.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:45.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.774 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.775 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:45.775 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.641173+0000", 2024-02-09T17:42:45.775 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:45.775 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.775 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.775 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:45.775 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.641174+0000", 2024-02-09T17:42:45.775 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.438e-06 2024-02-09T17:42:45.775 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.775 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.775 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:45.775 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.641245+0000", 2024-02-09T17:42:45.775 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1007000000000006e-05 2024-02-09T17:42:45.775 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.775 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.775 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:45.775 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.641263+0000", 2024-02-09T17:42:45.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7717e-05 2024-02-09T17:42:45.776 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.776 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.776 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:45.776 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.641273+0000", 2024-02-09T17:42:45.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9410000000000002e-06 2024-02-09T17:42:45.776 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.776 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.776 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:45.776 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.641392+0000", 2024-02-09T17:42:45.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011917300000000001 2024-02-09T17:42:45.776 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.776 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.776 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:45.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.641540+0000", 2024-02-09T17:42:45.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000147415 2024-02-09T17:42:45.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-02-09T17:42:45.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.641654+0000", 2024-02-09T17:42:45.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000113904 2024-02-09T17:42:45.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:45.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.649067+0000", 2024-02-09T17:42:45.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0074131860000000004 2024-02-09T17:42:45.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:45.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.654523+0000", 2024-02-09T17:42:45.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0054562480000000003 2024-02-09T17:42:45.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:45.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.867341+0000", 2024-02-09T17:42:45.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21281793500000001 2024-02-09T17:42:45.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:42:45.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.867407+0000", 2024-02-09T17:42:45.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5939999999999995e-05 2024-02-09T17:42:45.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:45.779 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.867452+0000", 2024-02-09T17:42:45.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5420000000000002e-05 2024-02-09T17:42:45.779 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:45.779 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:45.779 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:45.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.779 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:126295 5.9 5:99c08477:::benchmark_data_smithi179_115199_object126294:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e501)", 2024-02-09T17:42:45.779 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:42:31.641296+0000", 2024-02-09T17:42:45.779 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 14.006434675, 2024-02-09T17:42:45.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22630713299999999, 2024-02-09T17:42:45.779 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:42:45.779 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:42:45.779 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:42:45.779 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:42:45.780 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:42:45.780 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 126295 2024-02-09T17:42:45.780 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.780 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:42:45.780 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.780 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:42:45.780 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.641296+0000", 2024-02-09T17:42:45.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:45.780 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.780 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.780 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:42:45.780 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.641296+0000", 2024-02-09T17:42:45.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:42:45.780 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.780 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.781 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:42:45.781 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.641296+0000", 2024-02-09T17:42:45.781 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1900000000000001e-07 2024-02-09T17:42:45.781 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.781 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:42:45.781 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.641330+0000", 2024-02-09T17:42:45.781 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3914000000000001e-05 2024-02-09T17:42:45.781 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.781 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:42:45.781 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.641339+0000", 2024-02-09T17:42:45.781 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.456e-06 2024-02-09T17:42:45.781 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.782 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:42:45.782 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.641345+0000", 2024-02-09T17:42:45.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5830000000000003e-06 2024-02-09T17:42:45.782 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.782 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.782 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:42:45.782 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.645645+0000", 2024-02-09T17:42:45.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0043000649999999996 2024-02-09T17:42:45.782 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.782 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.782 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:42:45.782 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.645792+0000", 2024-02-09T17:42:45.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014633699999999999 2024-02-09T17:42:45.782 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.783 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-02-09T17:42:45.783 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.645886+0000", 2024-02-09T17:42:45.783 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4126e-05 2024-02-09T17:42:45.783 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.783 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:42:45.783 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.649221+0000", 2024-02-09T17:42:45.783 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003335438 2024-02-09T17:42:45.783 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.783 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:45.783 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.669407+0000", 2024-02-09T17:42:45.783 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020185446999999999 2024-02-09T17:42:45.783 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.784 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:45.784 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.867504+0000", 2024-02-09T17:42:45.784 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19809727499999999 2024-02-09T17:42:45.784 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.784 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.784 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:42:45.784 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.867565+0000", 2024-02-09T17:42:45.784 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0692000000000001e-05 2024-02-09T17:42:45.784 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:42:45.784 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:42:45.784 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:42:45.784 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.867603+0000", 2024-02-09T17:42:45.784 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7981000000000001e-05 2024-02-09T17:42:45.784 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:45.785 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:45.785 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:45.785 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:42:45.785 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:42:45.785 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:42:46.011 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:42:46.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:46.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:42:46.120 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:42:46.150 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:46.351 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:46 smithi162 ceph-mon[44304]: pgmap v1285: 37 pgs: 37 active+clean; 7.5 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 3.0 MiB/s wr, 868 op/s 2024-02-09T17:42:46.395 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.9 dump_ops_in_flight 2024-02-09T17:42:46.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:46 smithi179 ceph-mon[43975]: pgmap v1285: 37 pgs: 37 active+clean; 7.5 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 3.0 MiB/s wr, 868 op/s 2024-02-09T17:42:46.443 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:46 smithi099 ceph-mon[39420]: pgmap v1285: 37 pgs: 37 active+clean; 7.5 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 3.0 MiB/s wr, 868 op/s 2024-02-09T17:42:46.476 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:42:46.503 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:42:46.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:46.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:42:46.618 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:42:46.651 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:46.681 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:42:46.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:46.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:42:46.786 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:42:46.814 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:46.852 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:42:46.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:46.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:42:46.959 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:42:46.985 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:42:47.019 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:47.043 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:42:47.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:47.046 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:42:47.147 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:42:47.203 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:47.248 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:42:47.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:47.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:42:47.355 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:42:47.415 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:47.440 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:42:47.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:47.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:42:47.549 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:42:47.602 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:47.604 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:42:47.634 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:42:47.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:47.638 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:42:47.739 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:42:47.800 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:48.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:48 smithi179 ceph-mon[43975]: pgmap v1286: 37 pgs: 37 active+clean; 7.3 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 1.4 MiB/s wr, 996 op/s 2024-02-09T17:42:48.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:48 smithi162 ceph-mon[44304]: pgmap v1286: 37 pgs: 37 active+clean; 7.3 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 1.4 MiB/s wr, 996 op/s 2024-02-09T17:42:48.459 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:48 smithi099 ceph-mon[39420]: pgmap v1286: 37 pgs: 37 active+clean; 7.3 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 1.4 MiB/s wr, 996 op/s 2024-02-09T17:42:48.662 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:42:48.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:48.668 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:42:48.769 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:42:48.818 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:48.944 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:42:48.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:48.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:42:49.048 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:42:49.077 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:42:49.083 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:49.109 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:42:49.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:49.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:42:49.140 INFO:teuthology.orchestra.run.smithi099.stderr:set pool 5 pgp_num to 16 2024-02-09T17:42:49.223 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:42:49.295 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:49.325 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:42:49.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:49.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:42:49.397 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:49 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:42:49.397 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:49 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:42:49.397 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:49 smithi099 ceph-mon[39420]: osdmap e506: 12 total, 12 up, 11 in 2024-02-09T17:42:49.397 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:49 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:42:49.397 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:49 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:42:49.397 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:49 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3786361203' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]: dispatch 2024-02-09T17:42:49.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:49 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:42:49.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:49 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:42:49.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:49 smithi179 ceph-mon[43975]: osdmap e506: 12 total, 12 up, 11 in 2024-02-09T17:42:49.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:49 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:42:49.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:49 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:42:49.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:49 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3786361203' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]: dispatch 2024-02-09T17:42:49.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:49 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:42:49.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:49 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:42:49.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:49 smithi162 ceph-mon[44304]: osdmap e506: 12 total, 12 up, 11 in 2024-02-09T17:42:49.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:49 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:42:49.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:49 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:42:49.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:49 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3786361203' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]: dispatch 2024-02-09T17:42:49.435 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:42:49.574 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:49.729 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:42:49.731 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:49.732 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:42:49.833 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:42:49.870 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:50.131 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:50 smithi179 ceph-mon[43975]: pgmap v1288: 37 pgs: 37 active+clean; 7.3 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 833 op/s 2024-02-09T17:42:50.131 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:50 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:42:50.131 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:50 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:42:50.131 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:50 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3786361203' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]': finished 2024-02-09T17:42:50.131 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:50 smithi179 ceph-mon[43975]: osdmap e507: 12 total, 12 up, 11 in 2024-02-09T17:42:50.132 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:50 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3383262472' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:42:50.132 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:50 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:42:50.132 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:50 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:42:50.132 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:50 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:42:50.157 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is set 2024-02-09T17:42:50.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:50 smithi162 ceph-mon[44304]: pgmap v1288: 37 pgs: 37 active+clean; 7.3 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 833 op/s 2024-02-09T17:42:50.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:50 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:42:50.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:50 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:42:50.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:50 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3786361203' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]': finished 2024-02-09T17:42:50.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:50 smithi162 ceph-mon[44304]: osdmap e507: 12 total, 12 up, 11 in 2024-02-09T17:42:50.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:50 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3383262472' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:42:50.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:50 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:42:50.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:50 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:42:50.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:50 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:42:50.431 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:42:50.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:50.433 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:42:50.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:50 smithi099 ceph-mon[39420]: pgmap v1288: 37 pgs: 37 active+clean; 7.3 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 833 op/s 2024-02-09T17:42:50.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:50 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:42:50.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:50 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:42:50.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:50 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3786361203' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]': finished 2024-02-09T17:42:50.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:50 smithi099 ceph-mon[39420]: osdmap e507: 12 total, 12 up, 11 in 2024-02-09T17:42:50.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:50 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3383262472' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:42:50.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:50 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:42:50.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:50 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:42:50.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:50 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:42:50.534 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:42:50.610 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:50.646 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:42:50.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:50.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:42:50.751 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:42:50.787 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:50.938 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:42:50.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:50.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:42:51.044 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:42:51.098 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:42:51.099 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-02-09T17:42:51.099 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:42:51.099 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop_reply(client.25528.0:143643 5.b e508/465)", 2024-02-09T17:42:51.099 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:42:51.087014+0000", 2024-02-09T17:42:51.099 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 6.9428e-05, 2024-02-09T17:42:51.099 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.3164000000000005e-05, 2024-02-09T17:42:51.099 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:42:51.099 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-02-09T17:42:51.099 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:42:51.099 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:42:51.099 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:42:51.099 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:51.087014+0000", 2024-02-09T17:42:51.100 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:42:51.100 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:42:51.100 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:42:51.100 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:42:51.100 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:51.087014+0000", 2024-02-09T17:42:51.100 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:42:51.100 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:42:51.100 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:42:51.100 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:42:51.100 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:51.087015+0000", 2024-02-09T17:42:51.100 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4099999999999997e-07 2024-02-09T17:42:51.100 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:42:51.100 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:42:51.100 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:42:51.100 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:51.087020+0000", 2024-02-09T17:42:51.101 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9690000000000002e-06 2024-02-09T17:42:51.101 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:42:51.101 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:42:51.101 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:42:51.101 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:51.087034+0000", 2024-02-09T17:42:51.101 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3791999999999999e-05 2024-02-09T17:42:51.101 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:42:51.101 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:42:51.101 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:42:51.101 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:51.087041+0000", 2024-02-09T17:42:51.101 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1899999999999998e-06 2024-02-09T17:42:51.101 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:42:51.101 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:42:51.101 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:42:51.101 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:51.087110+0000", 2024-02-09T17:42:51.102 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8875999999999996e-05 2024-02-09T17:42:51.102 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:42:51.102 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:42:51.102 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:42:51.102 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:42:51.102 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:42:51.102 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:143641 5.e 5:75291be1:::benchmark_data_smithi179_115199_object14735:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e508)", 2024-02-09T17:42:51.102 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:42:51.076255+0000", 2024-02-09T17:42:51.102 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.010828508000000001, 2024-02-09T17:42:51.102 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010931883, 2024-02-09T17:42:51.102 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:42:51.102 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "waiting for sub ops", 2024-02-09T17:42:51.102 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:42:51.103 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:42:51.103 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:42:51.103 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 143641 2024-02-09T17:42:51.103 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:42:51.103 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:42:51.103 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:42:51.103 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:42:51.103 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:51.076255+0000", 2024-02-09T17:42:51.103 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:42:51.103 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:42:51.103 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:42:51.103 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:42:51.103 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:51.076255+0000", 2024-02-09T17:42:51.103 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:42:51.103 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:42:51.104 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:42:51.104 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:42:51.104 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:51.076257+0000", 2024-02-09T17:42:51.104 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.714e-06 2024-02-09T17:42:51.104 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:42:51.104 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:42:51.104 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:42:51.104 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:51.076262+0000", 2024-02-09T17:42:51.104 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2190000000000003e-06 2024-02-09T17:42:51.104 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:42:51.104 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:42:51.104 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:42:51.104 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:51.076274+0000", 2024-02-09T17:42:51.104 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2278e-05 2024-02-09T17:42:51.105 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:42:51.105 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:42:51.105 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:42:51.105 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:51.076283+0000", 2024-02-09T17:42:51.105 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4230000000000005e-06 2024-02-09T17:42:51.105 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:42:51.105 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:42:51.105 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:42:51.105 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:51.076444+0000", 2024-02-09T17:42:51.105 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000161144 2024-02-09T17:42:51.105 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:42:51.105 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:42:51.105 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:42:51.105 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:51.076563+0000", 2024-02-09T17:42:51.106 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011893800000000001 2024-02-09T17:42:51.106 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:42:51.106 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:42:51.106 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,8", 2024-02-09T17:42:51.106 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:51.076672+0000", 2024-02-09T17:42:51.106 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000109004 2024-02-09T17:42:51.106 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:42:51.106 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:42:51.106 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:42:51.106 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:51.081130+0000", 2024-02-09T17:42:51.106 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0044583210000000003 2024-02-09T17:42:51.106 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:42:51.106 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:42:51.106 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:42:51.106 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:42:51.106 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:42:51.107 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:143643 5.b 5:de79c605:::benchmark_data_smithi179_115199_object14737:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e508)", 2024-02-09T17:42:51.107 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:42:51.080701+0000", 2024-02-09T17:42:51.107 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.0063827279999999998, 2024-02-09T17:42:51.107 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0065689390000000002, 2024-02-09T17:42:51.107 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:42:51.107 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "waiting for sub ops", 2024-02-09T17:42:51.107 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:42:51.107 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:42:51.107 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:42:51.107 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 143643 2024-02-09T17:42:51.107 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:42:51.107 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:42:51.107 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:42:51.107 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:42:51.108 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:51.080701+0000", 2024-02-09T17:42:51.108 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:42:51.108 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:42:51.108 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:42:51.108 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:42:51.108 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:51.080701+0000", 2024-02-09T17:42:51.108 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:42:51.108 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:42:51.108 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:42:51.108 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:42:51.108 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:51.080703+0000", 2024-02-09T17:42:51.108 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9740000000000001e-06 2024-02-09T17:42:51.108 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:42:51.108 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:42:51.108 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:42:51.108 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:51.080708+0000", 2024-02-09T17:42:51.109 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1429999999999997e-06 2024-02-09T17:42:51.109 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:42:51.109 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:42:51.109 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:42:51.109 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:51.080720+0000", 2024-02-09T17:42:51.109 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1966e-05 2024-02-09T17:42:51.109 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:42:51.109 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:42:51.109 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:42:51.109 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:51.080728+0000", 2024-02-09T17:42:51.109 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7200000000000006e-06 2024-02-09T17:42:51.110 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:42:51.110 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:42:51.110 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:42:51.110 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:51.080829+0000", 2024-02-09T17:42:51.110 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000100886 2024-02-09T17:42:51.110 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:42:51.110 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:42:51.110 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:42:51.110 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:51.080959+0000", 2024-02-09T17:42:51.110 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000130168 2024-02-09T17:42:51.110 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:42:51.110 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:42:51.110 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,5", 2024-02-09T17:42:51.110 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:51.081075+0000", 2024-02-09T17:42:51.110 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000116418 2024-02-09T17:42:51.111 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:42:51.111 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:42:51.111 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:42:51.111 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:42:51.111 INFO:teuthology.orchestra.run.smithi179.stdout: ], 2024-02-09T17:42:51.111 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 3 2024-02-09T17:42:51.111 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:42:51.111 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:51.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:51 smithi179 ceph-mon[43975]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:42:51.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:51 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:42:51.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:51 smithi179 ceph-mon[43975]: osdmap e508: 12 total, 12 up, 11 in 2024-02-09T17:42:51.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:51 smithi162 ceph-mon[44304]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:42:51.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:51 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:42:51.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:51 smithi162 ceph-mon[44304]: osdmap e508: 12 total, 12 up, 11 in 2024-02-09T17:42:51.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:51 smithi099 ceph-mon[39420]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:42:51.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:51 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:42:51.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:51 smithi099 ceph-mon[39420]: osdmap e508: 12 total, 12 up, 11 in 2024-02-09T17:42:52.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:52 smithi179 ceph-mon[43975]: pgmap v1291: 37 pgs: 37 active+clean; 7.1 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 946 op/s 2024-02-09T17:42:52.281 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:42:52.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:52.289 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:42:52.323 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.9 dump_blocked_ops 2024-02-09T17:42:52.374 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:42:52.390 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:42:52.404 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:42:52.421 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:52 smithi099 ceph-mon[39420]: pgmap v1291: 37 pgs: 37 active+clean; 7.1 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 946 op/s 2024-02-09T17:42:52.425 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:52.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:52 smithi162 ceph-mon[44304]: pgmap v1291: 37 pgs: 37 active+clean; 7.1 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 946 op/s 2024-02-09T17:42:52.436 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:42:52.448 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:42:52.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:52.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:42:52.461 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:42:52.491 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:42:52.518 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:42:52.550 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:42:52.552 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:42:52.579 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:42:52.595 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:52.601 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:42:52.626 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:42:52.627 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:42:52.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:52.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:42:52.652 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:42:52.677 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:42:52.678 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:42:52.678 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:42:52.678 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:42:52.678 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:42:52.678 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:42:52.678 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:42:52.678 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:42:52.679 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:42:52.679 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:42:52.679 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:42:52.679 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:42:52.679 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:42:52.730 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:42:52.756 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:42:52.771 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:52.810 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:42:52.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:52.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:42:52.918 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:42:52.966 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:53.045 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:42:53.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:53.049 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:42:53.150 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:42:53.186 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:53.360 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:53 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2442963628' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:42:53.447 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:53 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2442963628' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:42:53.538 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:42:53.542 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:53.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:42:53.644 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:42:53.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:53 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2442963628' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:42:53.710 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:53.738 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:42:53.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:53.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:42:53.847 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:42:53.911 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:53.953 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:42:53.957 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:53.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:42:54.059 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:42:54.084 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset noscrub 2024-02-09T17:42:54.120 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:54.163 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:42:54.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:54.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:42:54.267 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:42:54.319 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:54.416 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:42:54.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:54 smithi162 ceph-mon[44304]: pgmap v1292: 37 pgs: 37 active+clean; 7.0 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 881 op/s 2024-02-09T17:42:54.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:54 smithi099 ceph-mon[39420]: pgmap v1292: 37 pgs: 37 active+clean; 7.0 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 881 op/s 2024-02-09T17:42:54.581 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:42:54.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:54.588 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:42:54.588 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:54 smithi179 ceph-mon[43975]: pgmap v1292: 37 pgs: 37 active+clean; 7.0 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 881 op/s 2024-02-09T17:42:54.689 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:42:54.750 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:54.794 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:42:54.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:54.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:42:54.903 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:42:54.924 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T17:42:54.925 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:42:54.956 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:54.993 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:42:55.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:55.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:42:55.102 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:42:55.136 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:55.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:55 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:42:55.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:55 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:42:55.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:55 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:42:55.434 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:55 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:42:55.434 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:55 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:42:55.434 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:55 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:42:55.482 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:42:55.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:55 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:42:55.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:55 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:42:55.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:55 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:42:56.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:56 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:42:56.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:56 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:42:56.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:56 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:42:56.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:56 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:42:56.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:56 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:42:56.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:56 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:42:56.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:56 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:42:56.491 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.9 dump_historic_ops 2024-02-09T17:42:56.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:56 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:42:56.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:56 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:42:56.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:56 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:42:56.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:56 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:42:56.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:56 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:42:56.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:56 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:42:56.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:56 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:42:56.555 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] 2024-02-09T17:42:56.555 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:42:56.555 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-02-09T17:42:56.556 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph pg dump --format=json 2024-02-09T17:42:56.681 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:56 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:42:56.681 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:56 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:42:56.681 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:56 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:42:56.681 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:56 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:42:56.681 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:56 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:42:56.681 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:56 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:42:56.681 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:56 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:42:56.906 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:42:57.871 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:57 smithi179 ceph-mon[43975]: pgmap v1294: 37 pgs: 37 active+clean; 6.8 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 1.02k op/s 2024-02-09T17:42:57.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:57 smithi162 ceph-mon[44304]: pgmap v1294: 37 pgs: 37 active+clean; 6.8 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 1.02k op/s 2024-02-09T17:42:57.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:57 smithi099 ceph-mon[39420]: pgmap v1294: 37 pgs: 37 active+clean; 6.8 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 1.02k op/s 2024-02-09T17:42:58.903 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:42:58.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:58.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:42:59.011 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:42:59.042 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:59.065 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:42:59.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:59.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:42:59.179 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:42:59.217 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:42:59.365 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T17:42:59.365 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:42:59.564 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:42:59.568 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:42:59.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:42:59.624 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:42:59.670 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:42:59.781 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:00.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:42:59 smithi179 ceph-mon[43975]: pgmap v1295: 37 pgs: 37 active+clean; 6.7 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 932 op/s 2024-02-09T17:43:00.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:42:59 smithi162 ceph-mon[44304]: pgmap v1295: 37 pgs: 37 active+clean; 6.7 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 932 op/s 2024-02-09T17:43:00.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:42:59 smithi099 ceph-mon[39420]: pgmap v1295: 37 pgs: 37 active+clean; 6.7 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 932 op/s 2024-02-09T17:43:00.283 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:43:00.346 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:43:00.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:00.359 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:43:00.460 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:43:00.514 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:00.555 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:43:00.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:00.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:43:00.663 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:43:00.705 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:00.741 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.10 dump_ops_in_flight 2024-02-09T17:43:00.743 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:43:00.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:00.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:43:00.847 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:43:00.882 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:00.898 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:00 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3775387857' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:43:00.898 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:00 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:43:00.910 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is unset 2024-02-09T17:43:01.118 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:43:01.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:00 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3775387857' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:43:01.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:00 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:43:01.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:00 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3775387857' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:43:01.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:00 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:43:01.862 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:43:01.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:01.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:43:01.973 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:43:02.011 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:02.153 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:01 smithi179 ceph-mon[43975]: pgmap v1296: 37 pgs: 37 active+clean; 6.6 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 950 op/s 2024-02-09T17:43:02.153 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:01 smithi179 ceph-mon[43975]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:43:02.153 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:01 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:43:02.153 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:01 smithi179 ceph-mon[43975]: osdmap e509: 12 total, 12 up, 11 in 2024-02-09T17:43:02.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:01 smithi162 ceph-mon[44304]: pgmap v1296: 37 pgs: 37 active+clean; 6.6 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 950 op/s 2024-02-09T17:43:02.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:01 smithi162 ceph-mon[44304]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:43:02.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:01 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:43:02.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:01 smithi162 ceph-mon[44304]: osdmap e509: 12 total, 12 up, 11 in 2024-02-09T17:43:02.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:01 smithi099 ceph-mon[39420]: pgmap v1296: 37 pgs: 37 active+clean; 6.6 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 950 op/s 2024-02-09T17:43:02.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:01 smithi099 ceph-mon[39420]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:43:02.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:01 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:43:02.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:01 smithi099 ceph-mon[39420]: osdmap e509: 12 total, 12 up, 11 in 2024-02-09T17:43:02.964 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:43:02.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:02.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:43:03.073 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:43:03.119 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:03.165 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:43:03.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:03.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:43:03.270 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:43:03.334 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:03.653 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T17:43:03.653 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:43:03.695 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:43:03.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:03.699 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:43:03.800 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:43:03.856 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:03.895 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-09T17:43:03.920 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:43:03.923 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:03.923 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:43:03.928 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:03 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:43:03.928 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:03 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:43:03.928 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:03 smithi099 ceph-mon[39420]: pgmap v1298: 37 pgs: 37 active+clean; 6.5 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 917 op/s 2024-02-09T17:43:03.928 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:03 smithi099 ceph-mon[39420]: osdmap e510: 12 total, 12 up, 11 in 2024-02-09T17:43:04.024 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:43:04.079 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:04.086 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:03 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:43:04.086 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:03 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:43:04.086 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:03 smithi179 ceph-mon[43975]: pgmap v1298: 37 pgs: 37 active+clean; 6.5 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 917 op/s 2024-02-09T17:43:04.086 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:03 smithi179 ceph-mon[43975]: osdmap e510: 12 total, 12 up, 11 in 2024-02-09T17:43:04.104 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:43:04.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:04.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:43:04.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:03 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:43:04.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:03 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:43:04.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:03 smithi162 ceph-mon[44304]: pgmap v1298: 37 pgs: 37 active+clean; 6.5 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 917 op/s 2024-02-09T17:43:04.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:03 smithi162 ceph-mon[44304]: osdmap e510: 12 total, 12 up, 11 in 2024-02-09T17:43:04.212 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:43:04.280 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:04.329 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:43:04.340 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:43:04.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:04.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:43:04.347 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:43:04.371 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:43:04.411 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:43:04.432 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:43:04.445 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:43:04.446 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:43:04.474 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:43:04.480 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.10 dump_blocked_ops 2024-02-09T17:43:04.505 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:04.518 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:43:04.529 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:43:04.549 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-02-09T17:43:04.559 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:43:04.591 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:43:04.619 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:43:04.654 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:43:04.699 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:43:04.700 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:43:04.700 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:43:04.700 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:43:04.700 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:43:04.700 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:43:04.700 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:43:04.701 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:43:04.701 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:43:04.701 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:43:04.701 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:43:04.701 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:43:04.701 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:43:04.926 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:43:05.166 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:04 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/151058586' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:43:05.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:04 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/151058586' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:43:05.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:04 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/151058586' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:43:05.470 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:43:05.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:05.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:43:05.579 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:43:05.614 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:05.690 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:43:05.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:05.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:43:05.794 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:43:05.826 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:05.859 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:43:05.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:05.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:43:05.974 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:43:06.019 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:06.064 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:43:06.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:06.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:43:06.169 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:43:06.198 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:05 smithi179 ceph-mon[43975]: 5.d scrub starts 2024-02-09T17:43:06.199 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:05 smithi179 ceph-mon[43975]: 1.10 scrub starts 2024-02-09T17:43:06.199 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:05 smithi179 ceph-mon[43975]: 1.10 scrub ok 2024-02-09T17:43:06.199 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:05 smithi179 ceph-mon[43975]: from='client.25619 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:43:06.199 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:05 smithi179 ceph-mon[43975]: 1.13 scrub starts 2024-02-09T17:43:06.199 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:05 smithi179 ceph-mon[43975]: 1.14 scrub starts 2024-02-09T17:43:06.199 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:05 smithi179 ceph-mon[43975]: 1.14 scrub ok 2024-02-09T17:43:06.199 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:05 smithi179 ceph-mon[43975]: 1.13 scrub ok 2024-02-09T17:43:06.199 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:05 smithi179 ceph-mon[43975]: pgmap v1300: 37 pgs: 37 active+clean; 6.3 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 887 op/s 2024-02-09T17:43:06.201 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:06.240 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:43:06.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:05 smithi099 ceph-mon[39420]: 5.d scrub starts 2024-02-09T17:43:06.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:05 smithi099 ceph-mon[39420]: 1.10 scrub starts 2024-02-09T17:43:06.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:05 smithi099 ceph-mon[39420]: 1.10 scrub ok 2024-02-09T17:43:06.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:05 smithi099 ceph-mon[39420]: from='client.25619 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:43:06.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:05 smithi099 ceph-mon[39420]: 1.13 scrub starts 2024-02-09T17:43:06.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:05 smithi099 ceph-mon[39420]: 1.14 scrub starts 2024-02-09T17:43:06.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:05 smithi099 ceph-mon[39420]: 1.14 scrub ok 2024-02-09T17:43:06.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:05 smithi099 ceph-mon[39420]: 1.13 scrub ok 2024-02-09T17:43:06.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:05 smithi099 ceph-mon[39420]: pgmap v1300: 37 pgs: 37 active+clean; 6.3 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 887 op/s 2024-02-09T17:43:06.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:06.245 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:43:06.346 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:43:06.388 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:05 smithi162 ceph-mon[44304]: 5.d scrub starts 2024-02-09T17:43:06.388 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:05 smithi162 ceph-mon[44304]: 1.10 scrub starts 2024-02-09T17:43:06.388 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:05 smithi162 ceph-mon[44304]: 1.10 scrub ok 2024-02-09T17:43:06.388 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:05 smithi162 ceph-mon[44304]: from='client.25619 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:43:06.388 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:05 smithi162 ceph-mon[44304]: 1.13 scrub starts 2024-02-09T17:43:06.388 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:05 smithi162 ceph-mon[44304]: 1.14 scrub starts 2024-02-09T17:43:06.388 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:05 smithi162 ceph-mon[44304]: 1.14 scrub ok 2024-02-09T17:43:06.389 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:05 smithi162 ceph-mon[44304]: 1.13 scrub ok 2024-02-09T17:43:06.389 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:05 smithi162 ceph-mon[44304]: pgmap v1300: 37 pgs: 37 active+clean; 6.3 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 887 op/s 2024-02-09T17:43:06.393 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:06.425 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:43:06.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:06.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:43:06.533 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:43:06.575 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:06.621 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:43:06.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:06.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:43:06.732 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:43:06.771 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:06.945 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":1299,"stamp":"2024-02-09T17:43:02.907622+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6934037048,"num_objects":105799,"num_object_clones":0,"num_object_copies":317397,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":105799,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152011,"num_write_kb":8249793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3629,"num_bytes_recovered":238499424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154930,"ondisk_log_size":154930,"up":111,"acting":111,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":111,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":25394688,"kb_used_data":20440624,"kb_used_omap":228,"kb_used_meta":4953499,"kb_avail":1005801984,"statfs":{"total":1055945392128,"available":1029941231616,"internally_reserved":0,"allocated":20931198976,"data_stored":20928076133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":233532,"internal_metadata":5072383940},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-471269376,"num_objects":-7191,"num_object_clones":0,"num_object_copies":-21573,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7191,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.087502"},"pg_stats":[{"pgid":"1.14","version":"63'139","reported_seq":977,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-09T17:43:00.907742+0000","last_change":"2024-02-09T17:41:08.077049+0000","last_active":"2024-02-09T17:43:00.907742+0000","last_peered":"2024-02-09T17:43:00.907742+0000","last_clean":"2024-02-09T17:43:00.907742+0000","last_became_active":"2024-02-09T17:39:27.455644+0000","last_became_peered":"2024-02-09T17:39:27.455644+0000","last_unstale":"2024-02-09T17:43:00.907742+0000","last_undegraded":"2024-02-09T17:43:00.907742+0000","last_fullsized":"2024-02-09T17:43:00.907742+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:08.076894+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:38:26.968892+0000","last_clean_scrub_stamp":"2024-02-09T17:41:08.076894+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:04.494671+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"508'9205","reported_seq":9388,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-09T17:42:57.890102+0000","last_change":"2024-02-09T17:42:35.239915+0000","last_active":"2024-02-09T17:42:57.890102+0000","last_peered":"2024-02-09T17:42:57.890102+0000","last_clean":"2024-02-09T17:42:57.890102+0000","last_became_active":"2024-02-09T17:40:48.779925+0000","last_became_peered":"2024-02-09T17:40:48.779925+0000","last_unstale":"2024-02-09T17:42:57.890102+0000","last_undegraded":"2024-02-09T17:42:57.890102+0000","last_fullsized":"2024-02-09T17:42:57.890102+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'7928","last_scrub_stamp":"2024-02-09T17:42:35.239746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:42:35.239746+0000","objects_scrubbed":7892,"log_size":9205,"log_dups_size":0,"ondisk_log_size":9205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:37.174281+0000","scrub_duration":1665,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":434176000,"num_objects":6625,"num_object_clones":0,"num_object_copies":19875,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9205,"num_write_kb":506560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,8],"acting":[10,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"63'139","reported_seq":1329,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-09T17:43:00.907564+0000","last_change":"2024-02-09T17:41:14.521470+0000","last_active":"2024-02-09T17:43:00.907564+0000","last_peered":"2024-02-09T17:43:00.907564+0000","last_clean":"2024-02-09T17:43:00.907564+0000","last_became_active":"2024-02-09T17:41:14.521079+0000","last_became_peered":"2024-02-09T17:41:14.521079+0000","last_unstale":"2024-02-09T17:43:00.907564+0000","last_undegraded":"2024-02-09T17:43:00.907564+0000","last_fullsized":"2024-02-09T17:43:00.907564+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:10.115197+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:41:10.115197+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:04.494706+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"508'9310","reported_seq":9493,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-09T17:42:59.192282+0000","last_change":"2024-02-09T17:42:31.833863+0000","last_active":"2024-02-09T17:42:59.192282+0000","last_peered":"2024-02-09T17:42:59.192282+0000","last_clean":"2024-02-09T17:42:59.192282+0000","last_became_active":"2024-02-09T17:40:48.783218+0000","last_became_peered":"2024-02-09T17:40:48.783218+0000","last_unstale":"2024-02-09T17:42:59.192282+0000","last_undegraded":"2024-02-09T17:42:59.192282+0000","last_fullsized":"2024-02-09T17:42:59.192282+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'7820","last_scrub_stamp":"2024-02-09T17:42:31.833770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:42:31.833770+0000","objects_scrubbed":7802,"log_size":9310,"log_dups_size":0,"ondisk_log_size":9310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:36.690190+0000","scrub_duration":1362,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":433192960,"num_objects":6610,"num_object_clones":0,"num_object_copies":19830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9310,"num_write_kb":509440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"63'139","reported_seq":1067,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-09T17:43:00.907676+0000","last_change":"2024-02-09T17:41:40.216180+0000","last_active":"2024-02-09T17:43:00.907676+0000","last_peered":"2024-02-09T17:43:00.907676+0000","last_clean":"2024-02-09T17:43:00.907676+0000","last_became_active":"2024-02-09T17:40:22.484952+0000","last_became_peered":"2024-02-09T17:40:22.484952+0000","last_unstale":"2024-02-09T17:43:00.907676+0000","last_undegraded":"2024-02-09T17:43:00.907676+0000","last_fullsized":"2024-02-09T17:43:00.907676+0000","mapping_epoch":454,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":455,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:40.216046+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:41:40.216046+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:02.783875+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"508'9318","reported_seq":9503,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-09T17:42:59.173517+0000","last_change":"2024-02-09T17:42:39.441680+0000","last_active":"2024-02-09T17:42:59.173517+0000","last_peered":"2024-02-09T17:42:59.173517+0000","last_clean":"2024-02-09T17:42:59.173517+0000","last_became_active":"2024-02-09T17:40:48.781472+0000","last_became_peered":"2024-02-09T17:40:48.781472+0000","last_unstale":"2024-02-09T17:42:59.173517+0000","last_undegraded":"2024-02-09T17:42:59.173517+0000","last_fullsized":"2024-02-09T17:42:59.173517+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"503'8191","last_scrub_stamp":"2024-02-09T17:42:39.441525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:42:39.441525+0000","objects_scrubbed":7816,"log_size":9318,"log_dups_size":0,"ondisk_log_size":9318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:56.423990+0000","scrub_duration":2957,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":434503680,"num_objects":6630,"num_object_clones":0,"num_object_copies":19890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9318,"num_write_kb":510336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"63'139","reported_seq":517,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-09T17:42:50.146848+0000","last_change":"2024-02-09T17:42:38.047873+0000","last_active":"2024-02-09T17:42:50.146848+0000","last_peered":"2024-02-09T17:42:50.146848+0000","last_clean":"2024-02-09T17:42:50.146848+0000","last_became_active":"2024-02-09T17:40:13.501711+0000","last_became_peered":"2024-02-09T17:40:13.501711+0000","last_unstale":"2024-02-09T17:42:50.146848+0000","last_undegraded":"2024-02-09T17:42:50.146848+0000","last_fullsized":"2024-02-09T17:42:50.146848+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:42:38.047749+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:38:27.045816+0000","last_clean_scrub_stamp":"2024-02-09T17:42:38.047749+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:54.530419+0000","scrub_duration":149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"508'9393","reported_seq":9575,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-09T17:42:58.894450+0000","last_change":"2024-02-09T17:42:27.680013+0000","last_active":"2024-02-09T17:42:58.894450+0000","last_peered":"2024-02-09T17:42:58.894450+0000","last_clean":"2024-02-09T17:42:58.894450+0000","last_became_active":"2024-02-09T17:40:48.780579+0000","last_became_peered":"2024-02-09T17:40:48.780579+0000","last_unstale":"2024-02-09T17:42:58.894450+0000","last_undegraded":"2024-02-09T17:42:58.894450+0000","last_fullsized":"2024-02-09T17:42:58.894450+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'7631","last_scrub_stamp":"2024-02-09T17:42:27.679861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:42:27.679861+0000","objects_scrubbed":7571,"log_size":9393,"log_dups_size":0,"ondisk_log_size":9393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:33.180233+0000","scrub_duration":1709,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":433913856,"num_objects":6621,"num_object_clones":0,"num_object_copies":19863,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9393,"num_write_kb":512448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"63'139","reported_seq":833,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-09T17:42:50.154033+0000","last_change":"2024-02-09T17:42:30.576428+0000","last_active":"2024-02-09T17:42:50.154033+0000","last_peered":"2024-02-09T17:42:50.154033+0000","last_clean":"2024-02-09T17:42:50.154033+0000","last_became_active":"2024-02-09T17:34:01.342607+0000","last_became_peered":"2024-02-09T17:34:01.342607+0000","last_unstale":"2024-02-09T17:42:50.154033+0000","last_undegraded":"2024-02-09T17:42:50.154033+0000","last_fullsized":"2024-02-09T17:42:50.154033+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:42:30.576340+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:42:30.576340+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:33.730137+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.2","version":"509'9731","reported_seq":9921,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-09T17:43:00.912986+0000","last_change":"2024-02-09T17:42:35.385816+0000","last_active":"2024-02-09T17:43:00.912986+0000","last_peered":"2024-02-09T17:43:00.912986+0000","last_clean":"2024-02-09T17:43:00.912986+0000","last_became_active":"2024-02-09T17:40:48.781532+0000","last_became_peered":"2024-02-09T17:40:48.781532+0000","last_unstale":"2024-02-09T17:43:00.912986+0000","last_undegraded":"2024-02-09T17:43:00.912986+0000","last_fullsized":"2024-02-09T17:43:00.912986+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'8226","last_scrub_stamp":"2024-02-09T17:42:35.385605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:42:35.385605+0000","objects_scrubbed":8178,"log_size":9731,"log_dups_size":0,"ondisk_log_size":9731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:43.713428+0000","scrub_duration":1906,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":437059584,"num_objects":6669,"num_object_clones":0,"num_object_copies":20007,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9731,"num_write_kb":524800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"63'139","reported_seq":824,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-09T17:42:50.154167+0000","last_change":"2024-02-09T17:41:44.649428+0000","last_active":"2024-02-09T17:42:50.154167+0000","last_peered":"2024-02-09T17:42:50.154167+0000","last_clean":"2024-02-09T17:42:50.154167+0000","last_became_active":"2024-02-09T17:33:00.284702+0000","last_became_peered":"2024-02-09T17:33:00.284702+0000","last_unstale":"2024-02-09T17:42:50.154167+0000","last_undegraded":"2024-02-09T17:42:50.154167+0000","last_fullsized":"2024-02-09T17:42:50.154167+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:44.649313+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:40:01.416303+0000","last_clean_scrub_stamp":"2024-02-09T17:41:44.649313+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:11.533734+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.3","version":"509'9431","reported_seq":9593,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-09T17:43:00.945995+0000","last_change":"2024-02-09T17:41:55.628782+0000","last_active":"2024-02-09T17:43:00.945995+0000","last_peered":"2024-02-09T17:43:00.945995+0000","last_clean":"2024-02-09T17:43:00.945995+0000","last_became_active":"2024-02-09T17:40:48.783369+0000","last_became_peered":"2024-02-09T17:40:48.783369+0000","last_unstale":"2024-02-09T17:43:00.945995+0000","last_undegraded":"2024-02-09T17:43:00.945995+0000","last_fullsized":"2024-02-09T17:43:00.945995+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"493'5422","last_scrub_stamp":"2024-02-09T17:41:55.628642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:41:55.628642+0000","objects_scrubbed":5380,"log_size":9431,"log_dups_size":0,"ondisk_log_size":9431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:08.609097+0000","scrub_duration":1305,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":426442752,"num_objects":6507,"num_object_clones":0,"num_object_copies":19521,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9431,"num_write_kb":510016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"63'139","reported_seq":498,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-09T17:43:00.907426+0000","last_change":"2024-02-09T17:41:52.484254+0000","last_active":"2024-02-09T17:43:00.907426+0000","last_peered":"2024-02-09T17:43:00.907426+0000","last_clean":"2024-02-09T17:43:00.907426+0000","last_became_active":"2024-02-09T17:40:13.497710+0000","last_became_peered":"2024-02-09T17:40:13.497710+0000","last_unstale":"2024-02-09T17:43:00.907426+0000","last_undegraded":"2024-02-09T17:43:00.907426+0000","last_fullsized":"2024-02-09T17:43:00.907426+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:52.484109+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:41:52.484109+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:17.603694+0000","scrub_duration":125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"508'9541","reported_seq":9727,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-09T17:42:57.766317+0000","last_change":"2024-02-09T17:42:30.704487+0000","last_active":"2024-02-09T17:42:57.766317+0000","last_peered":"2024-02-09T17:42:57.766317+0000","last_clean":"2024-02-09T17:42:57.766317+0000","last_became_active":"2024-02-09T17:40:48.780290+0000","last_became_peered":"2024-02-09T17:40:48.780290+0000","last_unstale":"2024-02-09T17:42:57.766317+0000","last_undegraded":"2024-02-09T17:42:57.766317+0000","last_fullsized":"2024-02-09T17:42:57.766317+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'8033","last_scrub_stamp":"2024-02-09T17:42:30.704387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:42:30.704387+0000","objects_scrubbed":7963,"log_size":9541,"log_dups_size":0,"ondisk_log_size":9541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:30.816560+0000","scrub_duration":2172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":450560000,"num_objects":6875,"num_object_clones":0,"num_object_copies":20625,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9541,"num_write_kb":525312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"63'139","reported_seq":916,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-09T17:42:50.157440+0000","last_change":"2024-02-09T17:42:31.121705+0000","last_active":"2024-02-09T17:42:50.157440+0000","last_peered":"2024-02-09T17:42:50.157440+0000","last_clean":"2024-02-09T17:42:50.157440+0000","last_became_active":"2024-02-09T17:33:00.284924+0000","last_became_peered":"2024-02-09T17:33:00.284924+0000","last_unstale":"2024-02-09T17:42:50.157440+0000","last_undegraded":"2024-02-09T17:42:50.157440+0000","last_fullsized":"2024-02-09T17:42:50.157440+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:42:31.121560+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:42:31.121560+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:53.381458+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.1","version":"508'9436","reported_seq":9620,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-09T17:43:00.761592+0000","last_change":"2024-02-09T17:42:37.561624+0000","last_active":"2024-02-09T17:43:00.761592+0000","last_peered":"2024-02-09T17:43:00.761592+0000","last_clean":"2024-02-09T17:43:00.761592+0000","last_became_active":"2024-02-09T17:40:49.660617+0000","last_became_peered":"2024-02-09T17:40:49.660617+0000","last_unstale":"2024-02-09T17:43:00.761592+0000","last_undegraded":"2024-02-09T17:43:00.761592+0000","last_fullsized":"2024-02-09T17:43:00.761592+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'8068","last_scrub_stamp":"2024-02-09T17:42:37.561384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:42:37.561384+0000","objects_scrubbed":7885,"log_size":9436,"log_dups_size":0,"ondisk_log_size":9436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:41.580029+0000","scrub_duration":2520,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":423362560,"num_objects":6460,"num_object_clones":0,"num_object_copies":19380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9436,"num_write_kb":508672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"63'139","reported_seq":868,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-09T17:42:50.154177+0000","last_change":"2024-02-09T17:42:34.665192+0000","last_active":"2024-02-09T17:42:50.154177+0000","last_peered":"2024-02-09T17:42:50.154177+0000","last_clean":"2024-02-09T17:42:50.154177+0000","last_became_active":"2024-02-09T17:41:14.521098+0000","last_became_peered":"2024-02-09T17:41:14.521098+0000","last_unstale":"2024-02-09T17:42:50.154177+0000","last_undegraded":"2024-02-09T17:42:50.154177+0000","last_fullsized":"2024-02-09T17:42:50.154177+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:42:34.665067+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:42:34.665067+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:58.504994+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11],"acting":[10,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.e","version":"509'9612","reported_seq":9773,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-09T17:43:01.588983+0000","last_change":"2024-02-09T17:41:54.022262+0000","last_active":"2024-02-09T17:43:01.588983+0000","last_peered":"2024-02-09T17:43:01.588983+0000","last_clean":"2024-02-09T17:43:01.588983+0000","last_became_active":"2024-02-09T17:40:48.779366+0000","last_became_peered":"2024-02-09T17:40:48.779366+0000","last_unstale":"2024-02-09T17:43:01.588983+0000","last_undegraded":"2024-02-09T17:43:01.588983+0000","last_fullsized":"2024-02-09T17:43:01.588983+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"493'5450","last_scrub_stamp":"2024-02-09T17:41:54.019230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:41:54.019230+0000","objects_scrubbed":5409,"log_size":9612,"log_dups_size":0,"ondisk_log_size":9612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:17.534919+0000","scrub_duration":1190,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":430047232,"num_objects":6562,"num_object_clones":0,"num_object_copies":19686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9612,"num_write_kb":517568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"63'139","reported_seq":819,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-09T17:43:00.918229+0000","last_change":"2024-02-09T17:41:41.681560+0000","last_active":"2024-02-09T17:43:00.918229+0000","last_peered":"2024-02-09T17:43:00.918229+0000","last_clean":"2024-02-09T17:43:00.918229+0000","last_became_active":"2024-02-09T17:32:35.418200+0000","last_became_peered":"2024-02-09T17:32:35.418200+0000","last_unstale":"2024-02-09T17:43:00.918229+0000","last_undegraded":"2024-02-09T17:43:00.918229+0000","last_fullsized":"2024-02-09T17:43:00.918229+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:41.681434+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:41:41.681434+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:05.846738+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.f","version":"508'9454","reported_seq":12012,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-09T17:42:59.155929+0000","last_change":"2024-02-09T17:42:38.866203+0000","last_active":"2024-02-09T17:42:59.155929+0000","last_peered":"2024-02-09T17:42:59.155929+0000","last_clean":"2024-02-09T17:42:59.155929+0000","last_became_active":"2024-02-09T17:41:20.876180+0000","last_became_peered":"2024-02-09T17:41:20.876180+0000","last_unstale":"2024-02-09T17:42:59.155929+0000","last_undegraded":"2024-02-09T17:42:59.155929+0000","last_fullsized":"2024-02-09T17:42:59.155929+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"503'8226","last_scrub_stamp":"2024-02-09T17:42:38.866022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:42:38.866022+0000","objects_scrubbed":7951,"log_size":9454,"log_dups_size":0,"ondisk_log_size":9454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:54.450060+0000","scrub_duration":3408,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435027968,"num_objects":6638,"num_object_clones":0,"num_object_copies":19914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9454,"num_write_kb":514944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1182,"num_bytes_recovered":77463552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,11],"acting":[5,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"63'139","reported_seq":659,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-09T17:42:50.157358+0000","last_change":"2024-02-09T17:41:46.320889+0000","last_active":"2024-02-09T17:42:50.157358+0000","last_peered":"2024-02-09T17:42:50.157358+0000","last_clean":"2024-02-09T17:42:50.157358+0000","last_became_active":"2024-02-09T17:41:14.522410+0000","last_became_peered":"2024-02-09T17:41:14.522410+0000","last_unstale":"2024-02-09T17:42:50.157358+0000","last_undegraded":"2024-02-09T17:42:50.157358+0000","last_fullsized":"2024-02-09T17:42:50.157358+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:46.320807+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:41:46.320807+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:07.977725+0000","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"508'9464","reported_seq":9629,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-09T17:42:59.702209+0000","last_change":"2024-02-09T17:42:00.784129+0000","last_active":"2024-02-09T17:42:59.702209+0000","last_peered":"2024-02-09T17:42:59.702209+0000","last_clean":"2024-02-09T17:42:59.702209+0000","last_became_active":"2024-02-09T17:40:48.781384+0000","last_became_peered":"2024-02-09T17:40:48.781384+0000","last_unstale":"2024-02-09T17:42:59.702209+0000","last_undegraded":"2024-02-09T17:42:59.702209+0000","last_fullsized":"2024-02-09T17:42:59.702209+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"495'5883","last_scrub_stamp":"2024-02-09T17:42:00.784027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:42:00.784027+0000","objects_scrubbed":5858,"log_size":9464,"log_dups_size":0,"ondisk_log_size":9464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:20.174647+0000","scrub_duration":1061,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":438173696,"num_objects":6686,"num_object_clones":0,"num_object_copies":20058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9464,"num_write_kb":516800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"63'139","reported_seq":874,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-09T17:42:50.156940+0000","last_change":"2024-02-09T17:41:55.371903+0000","last_active":"2024-02-09T17:42:50.156940+0000","last_peered":"2024-02-09T17:42:50.156940+0000","last_clean":"2024-02-09T17:42:50.156940+0000","last_became_active":"2024-02-09T17:39:37.460615+0000","last_became_peered":"2024-02-09T17:39:37.460615+0000","last_unstale":"2024-02-09T17:42:50.156940+0000","last_undegraded":"2024-02-09T17:42:50.156940+0000","last_fullsized":"2024-02-09T17:42:50.156940+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:55.371768+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:41:55.371768+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:56.313752+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"508'9734","reported_seq":12396,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-09T17:43:00.362193+0000","last_change":"2024-02-09T17:41:54.334501+0000","last_active":"2024-02-09T17:43:00.362193+0000","last_peered":"2024-02-09T17:43:00.362193+0000","last_clean":"2024-02-09T17:43:00.362193+0000","last_became_active":"2024-02-09T17:41:21.499187+0000","last_became_peered":"2024-02-09T17:41:21.499187+0000","last_unstale":"2024-02-09T17:43:00.362193+0000","last_undegraded":"2024-02-09T17:43:00.362193+0000","last_fullsized":"2024-02-09T17:43:00.362193+0000","mapping_epoch":481,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"493'5534","last_scrub_stamp":"2024-02-09T17:41:54.334404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:41:54.334404+0000","objects_scrubbed":5491,"log_size":9734,"log_dups_size":0,"ondisk_log_size":9734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:03.660761+0000","scrub_duration":1279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":439681048,"num_objects":6710,"num_object_clones":0,"num_object_copies":20130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6710,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9734,"num_write_kb":526145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1248,"num_bytes_recovered":81788928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"63'139","reported_seq":1322,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-09T17:43:00.907649+0000","last_change":"2024-02-09T17:42:35.605590+0000","last_active":"2024-02-09T17:43:00.907649+0000","last_peered":"2024-02-09T17:43:00.907649+0000","last_clean":"2024-02-09T17:43:00.907649+0000","last_became_active":"2024-02-09T17:39:17.456739+0000","last_became_peered":"2024-02-09T17:39:17.456739+0000","last_unstale":"2024-02-09T17:43:00.907649+0000","last_undegraded":"2024-02-09T17:43:00.907649+0000","last_fullsized":"2024-02-09T17:43:00.907649+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:42:35.605475+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:30.193194+0000","last_clean_scrub_stamp":"2024-02-09T17:42:35.605475+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:49.365305+0000","scrub_duration":222,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"509'9536","reported_seq":9723,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-09T17:43:01.953903+0000","last_change":"2024-02-09T17:42:33.501160+0000","last_active":"2024-02-09T17:43:01.953903+0000","last_peered":"2024-02-09T17:43:01.953903+0000","last_clean":"2024-02-09T17:43:01.953903+0000","last_became_active":"2024-02-09T17:40:48.781591+0000","last_became_peered":"2024-02-09T17:40:48.781591+0000","last_unstale":"2024-02-09T17:43:01.953903+0000","last_undegraded":"2024-02-09T17:43:01.953903+0000","last_fullsized":"2024-02-09T17:43:01.953903+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'7912","last_scrub_stamp":"2024-02-09T17:42:33.501032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:42:33.501032+0000","objects_scrubbed":7852,"log_size":9536,"log_dups_size":0,"ondisk_log_size":9536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:35.792099+0000","scrub_duration":1903,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":421003264,"num_objects":6424,"num_object_clones":0,"num_object_copies":19272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9536,"num_write_kb":510720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,4],"acting":[8,10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.e","version":"63'139","reported_seq":794,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-09T17:42:50.167236+0000","last_change":"2024-02-09T17:42:40.250751+0000","last_active":"2024-02-09T17:42:50.167236+0000","last_peered":"2024-02-09T17:42:50.167236+0000","last_clean":"2024-02-09T17:42:50.167236+0000","last_became_active":"2024-02-09T17:33:05.309824+0000","last_became_peered":"2024-02-09T17:33:05.309824+0000","last_unstale":"2024-02-09T17:42:50.167236+0000","last_undegraded":"2024-02-09T17:42:50.167236+0000","last_fullsized":"2024-02-09T17:42:50.167236+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:42:40.250649+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:42:40.250649+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:53.976568+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"509'9526","reported_seq":9710,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-09T17:43:01.587942+0000","last_change":"2024-02-09T17:42:30.664783+0000","last_active":"2024-02-09T17:43:01.587942+0000","last_peered":"2024-02-09T17:43:01.587942+0000","last_clean":"2024-02-09T17:43:01.587942+0000","last_became_active":"2024-02-09T17:40:48.783235+0000","last_became_peered":"2024-02-09T17:40:48.783235+0000","last_unstale":"2024-02-09T17:43:01.587942+0000","last_undegraded":"2024-02-09T17:43:01.587942+0000","last_fullsized":"2024-02-09T17:43:01.587942+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'7838","last_scrub_stamp":"2024-02-09T17:42:30.664657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:42:30.664657+0000","objects_scrubbed":7804,"log_size":9526,"log_dups_size":0,"ondisk_log_size":9526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:33.872794+0000","scrub_duration":1694,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":429916160,"num_objects":6560,"num_object_clones":0,"num_object_copies":19680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9526,"num_write_kb":514752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.f","version":"63'139","reported_seq":855,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-09T17:43:00.907610+0000","last_change":"2024-02-09T17:42:29.589223+0000","last_active":"2024-02-09T17:43:00.907610+0000","last_peered":"2024-02-09T17:43:00.907610+0000","last_clean":"2024-02-09T17:43:00.907610+0000","last_became_active":"2024-02-09T17:33:05.316100+0000","last_became_peered":"2024-02-09T17:33:05.316100+0000","last_unstale":"2024-02-09T17:43:00.907610+0000","last_undegraded":"2024-02-09T17:43:00.907610+0000","last_fullsized":"2024-02-09T17:43:00.907610+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:42:29.589083+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:36:22.364130+0000","last_clean_scrub_stamp":"2024-02-09T17:42:29.589083+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:39.671033+0000","scrub_duration":136,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"508'9611","reported_seq":12198,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-09T17:43:00.343095+0000","last_change":"2024-02-09T17:42:33.997317+0000","last_active":"2024-02-09T17:43:00.343095+0000","last_peered":"2024-02-09T17:43:00.343095+0000","last_clean":"2024-02-09T17:43:00.343095+0000","last_became_active":"2024-02-09T17:41:20.876434+0000","last_became_peered":"2024-02-09T17:41:20.876434+0000","last_unstale":"2024-02-09T17:43:00.343095+0000","last_undegraded":"2024-02-09T17:43:00.343095+0000","last_fullsized":"2024-02-09T17:43:00.343095+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'8076","last_scrub_stamp":"2024-02-09T17:42:33.997206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:42:33.997206+0000","objects_scrubbed":8030,"log_size":9611,"log_dups_size":0,"ondisk_log_size":9611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:50.309188+0000","scrub_duration":1898,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435224576,"num_objects":6641,"num_object_clones":0,"num_object_copies":19923,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9611,"num_write_kb":520064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1195,"num_bytes_recovered":78315520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"63'139","reported_seq":975,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-09T17:43:00.907124+0000","last_change":"2024-02-09T17:41:43.719325+0000","last_active":"2024-02-09T17:43:00.907124+0000","last_peered":"2024-02-09T17:43:00.907124+0000","last_clean":"2024-02-09T17:43:00.907124+0000","last_became_active":"2024-02-09T17:39:37.471954+0000","last_became_peered":"2024-02-09T17:39:37.471954+0000","last_unstale":"2024-02-09T17:43:00.907124+0000","last_undegraded":"2024-02-09T17:43:00.907124+0000","last_fullsized":"2024-02-09T17:43:00.907124+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:43.719190+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:23.912504+0000","last_clean_scrub_stamp":"2024-02-09T17:41:43.719190+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:10.334978+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"509'9709","reported_seq":9868,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-09T17:43:02.249074+0000","last_change":"2024-02-09T17:41:52.315160+0000","last_active":"2024-02-09T17:43:02.249074+0000","last_peered":"2024-02-09T17:43:02.249074+0000","last_clean":"2024-02-09T17:43:02.249074+0000","last_became_active":"2024-02-09T17:40:49.436285+0000","last_became_peered":"2024-02-09T17:40:49.436285+0000","last_unstale":"2024-02-09T17:43:02.249074+0000","last_undegraded":"2024-02-09T17:43:02.249074+0000","last_fullsized":"2024-02-09T17:43:02.249074+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"492'5271","last_scrub_stamp":"2024-02-09T17:41:52.314972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:41:52.314972+0000","objects_scrubbed":5238,"log_size":9709,"log_dups_size":0,"ondisk_log_size":9709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:02.904584+0000","scrub_duration":1131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":431161344,"num_objects":6579,"num_object_clones":0,"num_object_copies":19737,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9709,"num_write_kb":521216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"63'139","reported_seq":388,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-09T17:43:00.907568+0000","last_change":"2024-02-09T17:42:31.580840+0000","last_active":"2024-02-09T17:43:00.907568+0000","last_peered":"2024-02-09T17:43:00.907568+0000","last_clean":"2024-02-09T17:43:00.907568+0000","last_became_active":"2024-02-09T17:40:22.485257+0000","last_became_peered":"2024-02-09T17:40:22.485257+0000","last_unstale":"2024-02-09T17:43:00.907568+0000","last_undegraded":"2024-02-09T17:43:00.907568+0000","last_fullsized":"2024-02-09T17:43:00.907568+0000","mapping_epoch":454,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":455,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:42:31.580611+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:42:31.580611+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:41.436655+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"63'139","reported_seq":794,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-09T17:42:50.156930+0000","last_change":"2024-02-09T17:41:09.323335+0000","last_active":"2024-02-09T17:42:50.156930+0000","last_peered":"2024-02-09T17:42:50.156930+0000","last_clean":"2024-02-09T17:42:50.156930+0000","last_became_active":"2024-02-09T17:33:05.310104+0000","last_became_peered":"2024-02-09T17:33:05.310104+0000","last_unstale":"2024-02-09T17:42:50.156930+0000","last_undegraded":"2024-02-09T17:42:50.156930+0000","last_fullsized":"2024-02-09T17:42:50.156930+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:09.323236+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:41:09.323236+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:42:50.377513+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"63'139","reported_seq":811,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-09T17:43:00.901183+0000","last_change":"2024-02-09T17:42:33.646446+0000","last_active":"2024-02-09T17:43:00.901183+0000","last_peered":"2024-02-09T17:43:00.901183+0000","last_clean":"2024-02-09T17:43:00.901183+0000","last_became_active":"2024-02-09T17:34:01.353171+0000","last_became_peered":"2024-02-09T17:34:01.353171+0000","last_unstale":"2024-02-09T17:43:00.901183+0000","last_undegraded":"2024-02-09T17:43:00.901183+0000","last_fullsized":"2024-02-09T17:43:00.901183+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:42:33.646309+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:42:33.646309+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:57.573164+0000","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.12","version":"63'139","reported_seq":1321,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-09T17:42:50.157024+0000","last_change":"2024-02-09T17:42:31.533609+0000","last_active":"2024-02-09T17:42:50.157024+0000","last_peered":"2024-02-09T17:42:50.157024+0000","last_clean":"2024-02-09T17:42:50.157024+0000","last_became_active":"2024-02-09T17:40:13.499071+0000","last_became_peered":"2024-02-09T17:40:13.499071+0000","last_unstale":"2024-02-09T17:42:50.157024+0000","last_undegraded":"2024-02-09T17:42:50.157024+0000","last_fullsized":"2024-02-09T17:42:50.157024+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:42:31.533484+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:42:31.533484+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:36.556705+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"63'139","reported_seq":512,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-02-09T17:42:50.146943+0000","last_change":"2024-02-09T17:41:14.521999+0000","last_active":"2024-02-09T17:42:50.146943+0000","last_peered":"2024-02-09T17:42:50.146943+0000","last_clean":"2024-02-09T17:42:50.146943+0000","last_became_active":"2024-02-09T17:41:14.521672+0000","last_became_peered":"2024-02-09T17:41:14.521672+0000","last_unstale":"2024-02-09T17:42:50.146943+0000","last_undegraded":"2024-02-09T17:42:50.146943+0000","last_fullsized":"2024-02-09T17:42:50.146943+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:03.023657+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:41:03.023657+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":6933446680,"num_objects":105797,"num_object_clones":0,"num_object_copies":317391,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":105797,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152011,"num_write_kb":8249793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3625,"num_bytes_recovered":237568000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20819619840,"data_stored":20819607624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152011,"ondisk_log_size":152011,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":931424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2919,"ondisk_log_size":2919,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056888,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2241212,"kb_used_data":1704692,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":91503940,"statfs":{"total":95995035648,"available":93700034560,"internally_reserved":0,"allocated":1745604608,"data_stored":1745307870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":549386692},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:42:28 2024","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":1.5069999999999999,"15min":1.3029999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.216},"max":{"1min":3.2280000000000002,"5min":9.5709999999999997,"15min":14.913},"last":5.6120000000000001},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.321,"15min":1.3089999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.23000000000000001},"max":{"1min":5.367,"5min":9.8219999999999992,"15min":14.869},"last":0.33800000000000002}]},{"osd":1,"last update":"Fri Feb 9 17:42:28 2024","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":1.468,"15min":1.579},"min":{"1min":0.28999999999999998,"5min":0.24299999999999999,"15min":0.22700000000000001},"max":{"1min":5.415,"5min":9.7710000000000008,"15min":32.957999999999998},"last":1.8759999999999999},{"interface":"front","average":{"1min":0.82699999999999996,"5min":1.613,"15min":1.7210000000000001},"min":{"1min":0.31900000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":3.3170000000000002,"5min":9.718,"15min":32.880000000000003},"last":5.7359999999999998}]},{"osd":2,"last update":"Fri Feb 9 17:31:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.415,"15min":1.6859999999999999},"min":{"1min":0.378,"5min":0.247,"15min":0.247},"max":{"1min":4.4189999999999996,"5min":9.4719999999999995,"15min":9.4719999999999995},"last":0.35399999999999998},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.5149999999999999,"15min":1.4399999999999999},"min":{"1min":0.378,"5min":0.252,"15min":0.252},"max":{"1min":4.8579999999999997,"5min":7.96,"15min":7.96},"last":0.60399999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:42:28 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":1.24,"15min":1.552},"min":{"1min":0.29799999999999999,"5min":0.27100000000000002,"15min":0.247},"max":{"1min":5.4539999999999997,"5min":9.5299999999999994,"15min":21.315999999999999},"last":5.423},{"interface":"front","average":{"1min":1.175,"5min":1.734,"15min":1.6879999999999999},"min":{"1min":0.38500000000000001,"5min":0.29799999999999999,"15min":0.26400000000000001},"max":{"1min":5.5499999999999998,"5min":12.706,"15min":18.087},"last":5.4450000000000003}]},{"osd":4,"last update":"Fri Feb 9 17:42:28 2024","interfaces":[{"interface":"back","average":{"1min":2.1110000000000002,"5min":1.45,"15min":1.554},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":6.1790000000000003,"5min":21.030999999999999,"15min":21.030999999999999},"last":0.26900000000000002},{"interface":"front","average":{"1min":1.498,"5min":1.411,"15min":1.3400000000000001},"min":{"1min":0.32600000000000001,"5min":0.309,"15min":0.27700000000000002},"max":{"1min":6.9509999999999996,"5min":20.876999999999999,"15min":20.876999999999999},"last":5.681}]},{"osd":5,"last update":"Fri Feb 9 17:42:28 2024","interfaces":[{"interface":"back","average":{"1min":1.6679999999999999,"5min":1.157,"15min":1.3819999999999999},"min":{"1min":0.36199999999999999,"5min":0.316,"15min":0.28899999999999998},"max":{"1min":5.4800000000000004,"5min":6.4630000000000001,"15min":13.946999999999999},"last":1.843},{"interface":"front","average":{"1min":1.75,"5min":1.3109999999999999,"15min":1.4810000000000001},"min":{"1min":0.317,"5min":0.311,"15min":0.30199999999999999},"max":{"1min":5.4610000000000003,"5min":7.1349999999999998,"15min":13.414},"last":0.37}]},{"osd":6,"last update":"Fri Feb 9 17:42:28 2024","interfaces":[{"interface":"back","average":{"1min":2.0129999999999999,"5min":1.4750000000000001,"15min":1.4810000000000001},"min":{"1min":0.442,"5min":0.39200000000000002,"15min":0.29899999999999999},"max":{"1min":6.8330000000000002,"5min":9.4779999999999998,"15min":12.798999999999999},"last":1.9099999999999999},{"interface":"front","average":{"1min":2.7970000000000002,"5min":1.548,"15min":1.536},"min":{"1min":0.46500000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":6.9249999999999998,"5min":9.2420000000000009,"15min":10.782999999999999},"last":5.6529999999999996}]},{"osd":7,"last update":"Fri Feb 9 17:42:28 2024","interfaces":[{"interface":"back","average":{"1min":1.875,"5min":1.5189999999999999,"15min":1.659},"min":{"1min":0.41599999999999998,"5min":0.38200000000000001,"15min":0.312},"max":{"1min":7.5830000000000002,"5min":20.489999999999998,"15min":20.489999999999998},"last":0.39500000000000002},{"interface":"front","average":{"1min":2.0750000000000002,"5min":1.504,"15min":1.647},"min":{"1min":0.52200000000000002,"5min":0.35299999999999998,"15min":0.34200000000000003},"max":{"1min":10.276999999999999,"5min":20.712,"15min":20.712},"last":1.893}]},{"osd":8,"last update":"Fri Feb 9 17:42:54 2024","interfaces":[{"interface":"back","average":{"1min":1.6339999999999999,"5min":1.0289999999999999,"15min":1.6040000000000001},"min":{"1min":0.38700000000000001,"5min":0.30399999999999999,"15min":0.29499999999999998},"max":{"1min":7.3570000000000002,"5min":7.3570000000000002,"15min":13.199999999999999},"last":5.7000000000000002},{"interface":"front","average":{"1min":1.978,"5min":1.1870000000000001,"15min":1.4510000000000001},"min":{"1min":0.43099999999999999,"5min":0.35399999999999998,"15min":0.318},"max":{"1min":7.6059999999999999,"5min":7.6059999999999999,"15min":9.2940000000000005},"last":5.7519999999999998}]},{"osd":9,"last update":"Fri Feb 9 17:42:16 2024","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.887,"15min":1.4570000000000001},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29399999999999998},"max":{"1min":7.3860000000000001,"5min":62.518000000000001,"15min":62.518000000000001},"last":1.7709999999999999},{"interface":"front","average":{"1min":1.1990000000000001,"5min":1.9379999999999999,"15min":1.371},"min":{"1min":0.44700000000000001,"5min":0.379,"15min":0.34100000000000003},"max":{"1min":5.4089999999999998,"5min":63.241999999999997,"15min":63.241999999999997},"last":0.41899999999999998}]},{"osd":10,"last update":"Fri Feb 9 17:42:28 2024","interfaces":[{"interface":"back","average":{"1min":1.5600000000000001,"5min":1.329,"15min":1.236},"min":{"1min":0.40899999999999997,"5min":0.36799999999999999,"15min":0.33900000000000002},"max":{"1min":9.2460000000000004,"5min":9.3109999999999999,"15min":9.6199999999999992},"last":5.7169999999999996},{"interface":"front","average":{"1min":1.72,"5min":1.417,"15min":1.2789999999999999},"min":{"1min":0.47299999999999998,"5min":0.39100000000000001,"15min":0.33700000000000002},"max":{"1min":7.9329999999999998,"5min":9.593,"15min":12.856},"last":1.8120000000000001}]}]},{"osd":10,"up_from":84,"seq":360777253117,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2931320,"kb_used_data":2593328,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":90813832,"statfs":{"total":95995035648,"available":92993363968,"internally_reserved":0,"allocated":2655567872,"data_stored":2655275206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":346094016},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:42:18 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":1.4019999999999999,"15min":1.6180000000000001},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":3.2149999999999999,"5min":9.343,"15min":9.343},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.67000000000000004,"5min":1.2629999999999999,"15min":1.147},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":3.3159999999999998,"5min":6.3399999999999999,"15min":6.3399999999999999},"last":0.47599999999999998}]},{"osd":1,"last update":"Fri Feb 9 17:42:50 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.66800000000000004,"15min":0.63600000000000001},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":3.726,"5min":3.726,"15min":3.726},"last":0.316},{"interface":"front","average":{"1min":1.252,"5min":0.88,"15min":0.81799999999999995},"min":{"1min":0.33200000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":6.2709999999999999,"5min":6.2709999999999999,"15min":6.2709999999999999},"last":0.36699999999999999}]},{"osd":2,"last update":"Fri Feb 9 17:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":1.774,"15min":1.425},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.222},"max":{"1min":2.4140000000000001,"5min":11.384,"15min":13.351000000000001},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.83699999999999997,"5min":1.425,"15min":1.3859999999999999},"min":{"1min":0.32500000000000001,"5min":0.28799999999999998,"15min":0.26100000000000001},"max":{"1min":3.254,"5min":11.465,"15min":11.465},"last":0.441}]},{"osd":3,"last update":"Fri Feb 9 17:42:43 2024","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":1.534,"15min":1.8759999999999999},"min":{"1min":0.36699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":3.371,"5min":9.1649999999999991,"15min":9.1649999999999991},"last":0.58899999999999997},{"interface":"front","average":{"1min":1.125,"5min":1.6439999999999999,"15min":1.952},"min":{"1min":0.39000000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":3.6890000000000001,"5min":8.9670000000000005,"15min":8.9670000000000005},"last":0.45700000000000002}]},{"osd":4,"last update":"Fri Feb 9 17:42:05 2024","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":0.97199999999999998,"15min":1.0509999999999999},"min":{"1min":0.36899999999999999,"5min":0.29199999999999998,"15min":0.26100000000000001},"max":{"1min":6.4930000000000003,"5min":6.4930000000000003,"15min":7.1360000000000001},"last":4.5380000000000003},{"interface":"front","average":{"1min":1.1080000000000001,"5min":0.92300000000000004,"15min":1.048},"min":{"1min":0.39300000000000002,"5min":0.31,"15min":0.29999999999999999},"max":{"1min":6.5339999999999998,"5min":6.5339999999999998,"15min":6.8369999999999997},"last":4.6609999999999996}]},{"osd":5,"last update":"Fri Feb 9 17:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":1.085,"15min":1.2370000000000001},"min":{"1min":0.35999999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":2.9460000000000002,"5min":7.5179999999999998,"15min":9.2349999999999994},"last":0.48699999999999999},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.0740000000000001,"15min":1.2490000000000001},"min":{"1min":0.47399999999999998,"5min":0.374,"15min":0.32200000000000001},"max":{"1min":6.569,"5min":6.8940000000000001,"15min":8.6159999999999997},"last":0.54200000000000004}]},{"osd":6,"last update":"Fri Feb 9 17:42:00 2024","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.3049999999999999,"15min":1.274},"min":{"1min":0.42399999999999999,"5min":0.308,"15min":0.29099999999999998},"max":{"1min":4.21,"5min":8.6869999999999994,"15min":8.6869999999999994},"last":0.64300000000000002},{"interface":"front","average":{"1min":1.411,"5min":1.1279999999999999,"15min":1.3799999999999999},"min":{"1min":0.497,"5min":0.38600000000000001,"15min":0.35399999999999998},"max":{"1min":6.6040000000000001,"5min":6.6040000000000001,"15min":9.4350000000000005},"last":0.67200000000000004}]},{"osd":7,"last update":"Fri Feb 9 17:39:52 2024","interfaces":[{"interface":"back","average":{"1min":1.6000000000000001,"5min":1.367,"15min":1.415},"min":{"1min":0.42099999999999999,"5min":0.34200000000000003,"15min":0.312},"max":{"1min":4.0430000000000001,"5min":6.7590000000000003,"15min":17.754000000000001},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.498,"5min":1.4359999999999999,"15min":1.5189999999999999},"min":{"1min":0.47399999999999998,"5min":0.38300000000000001,"15min":0.316},"max":{"1min":5.8170000000000002,"5min":11.308999999999999,"15min":11.308999999999999},"last":0.78800000000000003}]},{"osd":8,"last update":"Fri Feb 9 17:42:43 2024","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.141,"15min":1.141},"min":{"1min":0.34200000000000003,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":5.2359999999999998,"5min":7.4359999999999999,"15min":7.4359999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.1579999999999999,"15min":0.98099999999999998},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.32900000000000001},"max":{"1min":3.3359999999999999,"5min":7.7060000000000004,"15min":7.7060000000000004},"last":0.99399999999999999}]},{"osd":9,"last update":"Fri Feb 9 17:42:00 2024","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":1.2729999999999999,"15min":1.0960000000000001},"min":{"1min":0.36799999999999999,"5min":0.36399999999999999,"15min":0.314},"max":{"1min":4.766,"5min":7.4850000000000003,"15min":8.7240000000000002},"last":0.51600000000000001},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.2370000000000001,"15min":1.1950000000000001},"min":{"1min":0.46000000000000002,"5min":0.441,"15min":0.29799999999999999},"max":{"1min":3.8090000000000002,"5min":7.891,"15min":10.1},"last":0.95899999999999996}]},{"osd":11,"last update":"Fri Feb 9 17:42:21 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":0.98899999999999999,"15min":1.0609999999999999},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.36399999999999999},"max":{"1min":4.9630000000000001,"5min":4.9630000000000001,"15min":8.6430000000000007},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.0469999999999999,"15min":1.1140000000000001},"min":{"1min":0.441,"5min":0.441,"15min":0.377},"max":{"1min":4.9960000000000004,"5min":4.9960000000000004,"15min":9.298},"last":0.82199999999999995}]}]},{"osd":9,"up_from":77,"seq":330712482051,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1709624,"kb_used_data":1263344,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":92035528,"statfs":{"total":95995035648,"available":94244380672,"internally_reserved":0,"allocated":1293664256,"data_stored":1293371590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":456980928},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:42:55 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":0.73599999999999999,"15min":0.63400000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":7.2199999999999998,"5min":7.2199999999999998,"15min":7.2199999999999998},"last":4.1689999999999996},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.0509999999999999,"15min":0.98499999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":4.8689999999999998,"5min":4.8689999999999998,"15min":4.8689999999999998},"last":0.374}]},{"osd":1,"last update":"Fri Feb 9 17:42:38 2024","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.7729999999999999,"15min":1.7430000000000001},"min":{"1min":0.23599999999999999,"5min":0.23499999999999999,"15min":0.23100000000000001},"max":{"1min":6.8769999999999998,"5min":17.024999999999999,"15min":89.515000000000001},"last":5.4080000000000004},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.736,"15min":1.714},"min":{"1min":0.33300000000000002,"5min":0.249,"15min":0.222},"max":{"1min":7.3230000000000004,"5min":11.864000000000001,"15min":89.388000000000005},"last":4.2140000000000004}]},{"osd":2,"last update":"Fri Feb 9 17:40:39 2024","interfaces":[{"interface":"back","average":{"1min":1.742,"5min":1.766,"15min":1.514},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":5.6630000000000003,"5min":14.121,"15min":14.121},"last":0.63400000000000001},{"interface":"front","average":{"1min":1.6910000000000001,"5min":1.52,"15min":1.4870000000000001},"min":{"1min":0.34100000000000003,"5min":0.26100000000000001,"15min":0.25600000000000001},"max":{"1min":5.7030000000000003,"5min":8.8040000000000003,"15min":13.69},"last":0.73899999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:42:38 2024","interfaces":[{"interface":"back","average":{"1min":1.5549999999999999,"5min":2.0859999999999999,"15min":1.6160000000000001},"min":{"1min":0.26500000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":6.915,"5min":15.414999999999999,"15min":40.18},"last":5.4480000000000004},{"interface":"front","average":{"1min":1.3109999999999999,"5min":2.0329999999999999,"15min":1.7390000000000001},"min":{"1min":0.38200000000000001,"5min":0.317,"15min":0.27900000000000003},"max":{"1min":7.2519999999999998,"5min":15.372,"15min":40.158000000000001},"last":4.3079999999999998}]},{"osd":4,"last update":"Fri Feb 9 17:42:38 2024","interfaces":[{"interface":"back","average":{"1min":1.893,"5min":1.52,"15min":1.4870000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.28399999999999997},"max":{"1min":7.5570000000000004,"5min":27.381,"15min":27.381},"last":5.5149999999999997},{"interface":"front","average":{"1min":1.754,"5min":1.5289999999999999,"15min":1.3939999999999999},"min":{"1min":0.33000000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":7.3010000000000002,"5min":27.355,"15min":27.355},"last":4.2679999999999998}]},{"osd":5,"last update":"Fri Feb 9 17:42:43 2024","interfaces":[{"interface":"back","average":{"1min":2.1320000000000001,"5min":1.335,"15min":1.4059999999999999},"min":{"1min":0.32700000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":7.367,"5min":8.6110000000000007,"15min":10.689},"last":4.2389999999999999},{"interface":"front","average":{"1min":3.0379999999999998,"5min":1.498,"15min":1.2929999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.311},"max":{"1min":8.8460000000000001,"5min":13.978999999999999,"15min":13.978999999999999},"last":0.40400000000000003}]},{"osd":6,"last update":"Fri Feb 9 17:42:38 2024","interfaces":[{"interface":"back","average":{"1min":2.407,"5min":1.492,"15min":1.4470000000000001},"min":{"1min":0.34499999999999997,"5min":0.316,"15min":0.30599999999999999},"max":{"1min":8.4299999999999997,"5min":14.069000000000001,"15min":14.069000000000001},"last":0.42699999999999999},{"interface":"front","average":{"1min":2.4660000000000002,"5min":1.552,"15min":1.504},"min":{"1min":0.38100000000000001,"5min":0.37,"15min":0.36199999999999999},"max":{"1min":8.1739999999999995,"5min":14.097,"15min":14.097},"last":10.417}]},{"osd":7,"last update":"Fri Feb 9 17:42:38 2024","interfaces":[{"interface":"back","average":{"1min":2.3410000000000002,"5min":1.5309999999999999,"15min":1.464},"min":{"1min":0.39200000000000002,"5min":0.34499999999999997,"15min":0.34300000000000003},"max":{"1min":6.9640000000000004,"5min":9.125,"15min":9.125},"last":5.5430000000000001},{"interface":"front","average":{"1min":2.3820000000000001,"5min":1.577,"15min":1.542},"min":{"1min":0.44500000000000001,"5min":0.39600000000000002,"15min":0.32200000000000001},"max":{"1min":6.9880000000000004,"5min":9.157,"15min":9.157},"last":5.5730000000000004}]},{"osd":8,"last update":"Fri Feb 9 17:42:59 2024","interfaces":[{"interface":"back","average":{"1min":1.478,"5min":1.4199999999999999,"15min":1.284},"min":{"1min":0.37,"5min":0.35399999999999998,"15min":0.34999999999999998},"max":{"1min":7.2759999999999998,"5min":14.045999999999999,"15min":16.209},"last":4.29},{"interface":"front","average":{"1min":1.986,"5min":1.5669999999999999,"15min":1.335},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.34799999999999998},"max":{"1min":8.6950000000000003,"5min":8.6950000000000003,"15min":16.263000000000002},"last":0.44900000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:42:06 2024","interfaces":[{"interface":"back","average":{"1min":1.724,"5min":1.3779999999999999,"15min":1.181},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":9.1869999999999994,"5min":9.1869999999999994,"15min":9.5090000000000003},"last":5.4829999999999997},{"interface":"front","average":{"1min":1.129,"5min":1.3979999999999999,"15min":1.147},"min":{"1min":0.40999999999999998,"5min":0.39500000000000002,"15min":0.38200000000000001},"max":{"1min":4.1779999999999999,"5min":9.7569999999999997,"15min":9.7569999999999997},"last":4.3239999999999998}]},{"osd":11,"last update":"Fri Feb 9 17:42:06 2024","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.5660000000000001,"15min":1.4379999999999999},"min":{"1min":0.434,"5min":0.39000000000000001,"15min":0.34699999999999998},"max":{"1min":6.8070000000000004,"5min":14.02,"15min":14.02},"last":10.464},{"interface":"front","average":{"1min":1.0029999999999999,"5min":1.28,"15min":1.2330000000000001},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.374},"max":{"1min":3.4460000000000002,"5min":7.29,"15min":10.32},"last":5.3879999999999999}]}]},{"osd":8,"up_from":171,"seq":734439407794,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1472468,"kb_used_data":1246604,"kb_used_omap":212,"kb_used_meta":225643,"kb_avail":92272684,"statfs":{"total":95995035648,"available":94487228416,"internally_reserved":0,"allocated":1276522496,"data_stored":1276266063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217579,"internal_metadata":231058965},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:42:38 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":1.0269999999999999,"15min":1.0629999999999999},"min":{"1min":0.22800000000000001,"5min":0.217,"15min":0.217},"max":{"1min":1.7350000000000001,"5min":8.0920000000000005,"15min":8.0920000000000005},"last":1.1200000000000001},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.429,"15min":1.2809999999999999},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":6.4089999999999998,"5min":10.256,"15min":10.256},"last":0.254}]},{"osd":1,"last update":"Fri Feb 9 17:42:38 2024","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.401,"15min":1.294},"min":{"1min":0.28199999999999997,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":6.3570000000000002,"5min":7.867,"15min":7.8719999999999999},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":1.7390000000000001,"15min":1.5529999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.23300000000000001},"max":{"1min":2.3159999999999998,"5min":25.027000000000001,"15min":25.027000000000001},"last":0.34699999999999998}]},{"osd":2,"last update":"Fri Feb 9 17:42:38 2024","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":1.3740000000000001,"15min":1.3919999999999999},"min":{"1min":0.308,"5min":0.28199999999999997,"15min":0.255},"max":{"1min":7.5789999999999997,"5min":8.5649999999999995,"15min":10.423},"last":1.155},{"interface":"front","average":{"1min":1.048,"5min":1.4239999999999999,"15min":1.3999999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.313},"max":{"1min":7.625,"5min":8.5980000000000008,"15min":10.504},"last":1.1839999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:42:38 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":1.542,"15min":1.6379999999999999},"min":{"1min":0.316,"5min":0.29499999999999998,"15min":0.28699999999999998},"max":{"1min":2.347,"5min":15.228999999999999,"15min":24.030999999999999},"last":0.311},{"interface":"front","average":{"1min":0.75600000000000001,"5min":1.8340000000000001,"15min":1.8640000000000001},"min":{"1min":0.36199999999999999,"5min":0.33600000000000002,"15min":0.30299999999999999},"max":{"1min":2.3719999999999999,"5min":14.988,"15min":24.318999999999999},"last":0.36899999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:42:38 2024","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":1.2889999999999999,"15min":1.4419999999999999},"min":{"1min":0.32700000000000001,"5min":0.29899999999999999,"15min":0.27500000000000002},"max":{"1min":5.3479999999999999,"5min":12.936,"15min":53.060000000000002},"last":1.7829999999999999},{"interface":"front","average":{"1min":1.274,"5min":1.333,"15min":1.583},"min":{"1min":0.36399999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":4.718,"5min":7.46,"15min":53.311999999999998},"last":3.863}]},{"osd":6,"last update":"Fri Feb 9 17:42:38 2024","interfaces":[{"interface":"back","average":{"1min":2.0030000000000001,"5min":1.3360000000000001,"15min":1.5600000000000001},"min":{"1min":0.33500000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":9.5,"5min":9.5,"15min":37.356999999999999},"last":2.3290000000000002},{"interface":"front","average":{"1min":1.8759999999999999,"5min":1.3740000000000001,"15min":1.3320000000000001},"min":{"1min":0.436,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":6.4749999999999996,"5min":7.702,"15min":37.25},"last":2.4489999999999998}]},{"osd":7,"last update":"Fri Feb 9 17:42:38 2024","interfaces":[{"interface":"back","average":{"1min":1.5880000000000001,"5min":1.482,"15min":1.5780000000000001},"min":{"1min":0.42899999999999999,"5min":0.39300000000000002,"15min":0.32800000000000001},"max":{"1min":9.0749999999999993,"5min":9.0749999999999993,"15min":37.585000000000001},"last":2.3860000000000001},{"interface":"front","average":{"1min":1.6180000000000001,"5min":1.5229999999999999,"15min":1.6399999999999999},"min":{"1min":0.46200000000000002,"5min":0.38500000000000001,"15min":0.312},"max":{"1min":9.032,"5min":9.032,"15min":37.918999999999997},"last":2.4100000000000001}]},{"osd":9,"last update":"Fri Feb 9 17:42:38 2024","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.0589999999999999,"15min":1.0900000000000001},"min":{"1min":0.315,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":6.444,"5min":6.444,"15min":9.4900000000000002},"last":0.432},{"interface":"front","average":{"1min":0.73199999999999998,"5min":1.1040000000000001,"15min":1.155},"min":{"1min":0.41899999999999998,"5min":0.376,"15min":0.32000000000000001},"max":{"1min":1.653,"5min":7.4189999999999996,"15min":17.434999999999999},"last":1.2330000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:42:38 2024","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.089,"15min":1.075},"min":{"1min":0.34300000000000003,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":7.7270000000000003,"5min":7.7270000000000003,"15min":10.65},"last":1.208},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.3899999999999999,"15min":1.151},"min":{"1min":0.379,"5min":0.34799999999999998,"15min":0.318},"max":{"1min":6.5060000000000002,"5min":11.071,"15min":11.071},"last":0.40000000000000002}]},{"osd":11,"last update":"Fri Feb 9 17:42:38 2024","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":1.264,"15min":1.3260000000000001},"min":{"1min":0.42599999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":7.6959999999999997,"5min":8.5519999999999996,"15min":14.433999999999999},"last":1.2569999999999999},{"interface":"front","average":{"1min":0.99099999999999999,"5min":1.248,"15min":1.2669999999999999},"min":{"1min":0.442,"5min":0.308,"15min":0.308},"max":{"1min":5.452,"5min":7.4359999999999999,"15min":11.622},"last":0.54100000000000004}]}]},{"osd":7,"up_from":57,"seq":244813136140,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2198860,"kb_used_data":1676104,"kb_used_omap":1,"kb_used_meta":522622,"kb_avail":91546292,"statfs":{"total":95995035648,"available":93743403008,"internally_reserved":0,"allocated":1716330496,"data_stored":1716073543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":535165386},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:42:21 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":1.3360000000000001,"15min":2.528},"min":{"1min":0.26000000000000001,"5min":0.252,"15min":0.252},"max":{"1min":2.5899999999999999,"5min":14.348000000000001,"15min":36.753},"last":0.27500000000000002},{"interface":"front","average":{"1min":1.4750000000000001,"5min":1.349,"15min":2.714},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.253},"max":{"1min":6.7960000000000003,"5min":8.2289999999999992,"15min":36.918999999999997},"last":0.35799999999999998}]},{"osd":1,"last update":"Fri Feb 9 17:42:21 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":2.1230000000000002,"15min":2.0019999999999998},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":2.6669999999999998,"5min":21.704999999999998,"15min":21.704999999999998},"last":6.46},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.9590000000000001,"15min":1.8160000000000001},"min":{"1min":0.33300000000000002,"5min":0.26600000000000001,"15min":0.26500000000000001},"max":{"1min":7.2800000000000002,"5min":21.858000000000001,"15min":21.858000000000001},"last":6.5830000000000002}]},{"osd":2,"last update":"Fri Feb 9 17:39:34 2024","interfaces":[{"interface":"back","average":{"1min":1.8520000000000001,"5min":1.7470000000000001,"15min":1.7030000000000001},"min":{"1min":0.315,"5min":0.28499999999999998,"15min":0.26000000000000001},"max":{"1min":6.6470000000000002,"5min":10.622,"15min":11.324},"last":0.76900000000000002},{"interface":"front","average":{"1min":1.8859999999999999,"5min":1.855,"15min":1.7769999999999999},"min":{"1min":0.34699999999999998,"5min":0.32100000000000001,"15min":0.28299999999999997},"max":{"1min":6.6870000000000003,"5min":19.361999999999998,"15min":19.361999999999998},"last":0.82299999999999995}]},{"osd":3,"last update":"Fri Feb 9 17:42:40 2024","interfaces":[{"interface":"back","average":{"1min":1.986,"5min":1.6579999999999999,"15min":1.9199999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.26900000000000002},"max":{"1min":9.2300000000000004,"5min":9.2300000000000004,"15min":27.719000000000001},"last":0.316},{"interface":"front","average":{"1min":1.454,"5min":1.4590000000000001,"15min":1.8380000000000001},"min":{"1min":0.33800000000000002,"5min":0.32000000000000001,"15min":0.26500000000000001},"max":{"1min":5.0250000000000004,"5min":11.945,"15min":24.457000000000001},"last":6.524}]},{"osd":4,"last update":"Fri Feb 9 17:42:21 2024","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":0.84299999999999997,"15min":0.747},"min":{"1min":0.27700000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":4.5789999999999997,"5min":4.5789999999999997,"15min":4.5789999999999997},"last":0.747},{"interface":"front","average":{"1min":2.1509999999999998,"5min":0.91500000000000004,"15min":0.70899999999999996},"min":{"1min":0.312,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":7.9939999999999998,"5min":7.9939999999999998,"15min":7.9939999999999998},"last":1.26}]},{"osd":5,"last update":"Fri Feb 9 17:42:21 2024","interfaces":[{"interface":"back","average":{"1min":2.492,"5min":1.018,"15min":0.76800000000000002},"min":{"1min":0.32100000000000001,"5min":0.249,"15min":0.249},"max":{"1min":8.3689999999999998,"5min":8.3689999999999998,"15min":8.3689999999999998},"last":0.56699999999999995},{"interface":"front","average":{"1min":1.8580000000000001,"5min":0.83299999999999996,"15min":0.61599999999999999},"min":{"1min":0.35999999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":8.0069999999999997,"5min":8.0069999999999997,"15min":8.0069999999999997},"last":6.6799999999999997}]},{"osd":6,"last update":"Fri Feb 9 17:42:40 2024","interfaces":[{"interface":"back","average":{"1min":1.917,"5min":1.073,"15min":1.6020000000000001},"min":{"1min":0.33900000000000002,"5min":0.28100000000000003,"15min":0.26000000000000001},"max":{"1min":8.2089999999999996,"5min":8.2089999999999996,"15min":25.965},"last":6.4950000000000001},{"interface":"front","average":{"1min":2.1779999999999999,"5min":1.202,"15min":1.413},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29799999999999999},"max":{"1min":9.2829999999999995,"5min":9.2829999999999995,"15min":9.4090000000000007},"last":0.60399999999999998}]},{"osd":8,"last update":"Fri Feb 9 17:42:40 2024","interfaces":[{"interface":"back","average":{"1min":2.359,"5min":1.345,"15min":1.4430000000000001},"min":{"1min":0.33100000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":9.2680000000000007,"5min":9.2680000000000007,"15min":9.2680000000000007},"last":0.39200000000000002},{"interface":"front","average":{"1min":1.645,"5min":1.1299999999999999,"15min":1.3260000000000001},"min":{"1min":0.377,"5min":0.36599999999999999,"15min":0.33300000000000002},"max":{"1min":5.0679999999999996,"5min":6.0339999999999998,"15min":11.481},"last":6.6319999999999997}]},{"osd":9,"last update":"Fri Feb 9 17:42:30 2024","interfaces":[{"interface":"back","average":{"1min":1.8220000000000001,"5min":1.242,"15min":1.577},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":6.7380000000000004,"5min":6.7380000000000004,"15min":41.286999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":2.105,"5min":1.373,"15min":1.671},"min":{"1min":0.42099999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":8.0399999999999991,"5min":8.0399999999999991,"15min":41.234999999999999},"last":1.2909999999999999}]},{"osd":10,"last update":"Fri Feb 9 17:42:55 2024","interfaces":[{"interface":"back","average":{"1min":1.5249999999999999,"5min":1.45,"15min":1.4830000000000001},"min":{"1min":0.38900000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":7.3090000000000002,"5min":12.050000000000001,"15min":12.050000000000001},"last":0.63700000000000001},{"interface":"front","average":{"1min":2.028,"5min":1.4510000000000001,"15min":1.5840000000000001},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.35599999999999998},"max":{"1min":9.3879999999999999,"5min":9.3879999999999999,"15min":9.3879999999999999},"last":4.2990000000000004}]},{"osd":11,"last update":"Fri Feb 9 17:42:21 2024","interfaces":[{"interface":"back","average":{"1min":2.0390000000000001,"5min":1.681,"15min":1.77},"min":{"1min":0.42399999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":9.2490000000000006,"5min":21.945,"15min":21.945},"last":0.66800000000000004},{"interface":"front","average":{"1min":1.998,"5min":1.7050000000000001,"15min":1.609},"min":{"1min":0.45500000000000002,"5min":0.40699999999999997,"15min":0.39500000000000002},"max":{"1min":9.3420000000000005,"5min":21.731999999999999,"15min":21.731999999999999},"last":0.78500000000000003}]}]},{"osd":6,"up_from":49,"seq":210453397777,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1683384,"kb_used_data":1277360,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":92061768,"statfs":{"total":95995035648,"available":94271250432,"internally_reserved":0,"allocated":1308016640,"data_stored":1307723974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":415758790},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:42:13 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":1.579,"15min":1.587},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.21299999999999999},"max":{"1min":4.2519999999999998,"5min":9.5739999999999998,"15min":12.544},"last":5.2809999999999997},{"interface":"front","average":{"1min":0.97899999999999998,"5min":1.5189999999999999,"15min":1.4990000000000001},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.217},"max":{"1min":4.9950000000000001,"5min":9.2520000000000007,"15min":9.2520000000000007},"last":5.5300000000000002}]},{"osd":1,"last update":"Fri Feb 9 17:42:13 2024","interfaces":[{"interface":"back","average":{"1min":1.748,"5min":1.9490000000000001,"15min":1.891},"min":{"1min":0.307,"5min":0.28599999999999998,"15min":0.25900000000000001},"max":{"1min":8.3260000000000005,"5min":8.9689999999999994,"15min":17.762},"last":0.40799999999999997},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.869,"15min":1.903},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":4.5270000000000001,"5min":9.0090000000000003,"15min":17.734999999999999},"last":5.3739999999999997}]},{"osd":2,"last update":"Fri Feb 9 17:42:13 2024","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":1.472,"15min":1.5880000000000001},"min":{"1min":0.35699999999999998,"5min":0.29399999999999998,"15min":0.22600000000000001},"max":{"1min":4.2160000000000002,"5min":9.8019999999999996,"15min":10.682},"last":5.5830000000000002},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.5149999999999999,"15min":1.6459999999999999},"min":{"1min":0.372,"5min":0.35599999999999998,"15min":0.28000000000000003},"max":{"1min":4.2850000000000001,"5min":9.8439999999999994,"15min":11.553000000000001},"last":5.5999999999999996}]},{"osd":3,"last update":"Fri Feb 9 17:42:13 2024","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.7969999999999999,"15min":2.0550000000000002},"min":{"1min":0.28399999999999997,"5min":0.26700000000000002,"15min":0.25600000000000001},"max":{"1min":9.6389999999999993,"5min":11.276,"15min":32.545000000000002},"last":0.42499999999999999},{"interface":"front","average":{"1min":1.6419999999999999,"5min":1.8640000000000001,"15min":1.9179999999999999},"min":{"1min":0.36599999999999999,"5min":0.307,"15min":0.26600000000000001},"max":{"1min":8.2520000000000007,"5min":11.122,"15min":32.670999999999999},"last":0.45000000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:42:13 2024","interfaces":[{"interface":"back","average":{"1min":1.796,"5min":1.03,"15min":1.4379999999999999},"min":{"1min":0.252,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":8.2959999999999994,"5min":8.2959999999999994,"15min":19.798999999999999},"last":0.311},{"interface":"front","average":{"1min":1.5660000000000001,"5min":1.147,"15min":1.385},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26000000000000001},"max":{"1min":4.8109999999999999,"5min":5.7270000000000003,"15min":10.702999999999999},"last":5.4160000000000004}]},{"osd":5,"last update":"Fri Feb 9 17:42:13 2024","interfaces":[{"interface":"back","average":{"1min":1.7629999999999999,"5min":1.3180000000000001,"15min":1.4239999999999999},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29199999999999998},"max":{"1min":5.6790000000000003,"5min":6.7679999999999998,"15min":10.968999999999999},"last":5.4480000000000004},{"interface":"front","average":{"1min":1.879,"5min":1.226,"15min":1.4650000000000001},"min":{"1min":0.33700000000000002,"5min":0.316,"15min":0.29299999999999998},"max":{"1min":8.3989999999999991,"5min":8.3989999999999991,"15min":13.91},"last":0.376}]},{"osd":7,"last update":"Fri Feb 9 17:42:55 2024","interfaces":[{"interface":"back","average":{"1min":3.3170000000000002,"5min":1.534,"15min":1.595},"min":{"1min":0.47599999999999998,"5min":0.29399999999999998,"15min":0.28000000000000003},"max":{"1min":11.052,"5min":11.052,"15min":14.022},"last":5.5590000000000002},{"interface":"front","average":{"1min":2.5,"5min":1.5800000000000001,"15min":1.7390000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.309},"max":{"1min":7.7949999999999999,"5min":12.441000000000001,"15min":15.458},"last":0.56599999999999995}]},{"osd":8,"last update":"Fri Feb 9 17:26:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":0.876,"15min":0.79600000000000004},"min":{"1min":0.39900000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":4.0339999999999998,"5min":4.4880000000000004,"15min":4.4880000000000004},"last":4.3840000000000003},{"interface":"front","average":{"1min":1.0469999999999999,"5min":0.88600000000000001,"15min":0.78300000000000003},"min":{"1min":0.375,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":3.6539999999999999,"5min":4.8570000000000002,"15min":4.8570000000000002},"last":4.4210000000000003}]},{"osd":9,"last update":"Fri Feb 9 17:42:19 2024","interfaces":[{"interface":"back","average":{"1min":1.6830000000000001,"5min":1.2889999999999999,"15min":1.3560000000000001},"min":{"1min":0.438,"5min":0.38800000000000001,"15min":0.35899999999999999},"max":{"1min":6.9029999999999996,"5min":10.779999999999999,"15min":10.803000000000001},"last":5.5039999999999996},{"interface":"front","average":{"1min":1.8420000000000001,"5min":1.444,"15min":1.4590000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40300000000000002},"max":{"1min":6.5640000000000001,"5min":6.5640000000000001,"15min":8.1630000000000003},"last":5.4749999999999996}]},{"osd":10,"last update":"Fri Feb 9 17:42:55 2024","interfaces":[{"interface":"back","average":{"1min":2.8380000000000001,"5min":1.4970000000000001,"15min":1.4750000000000001},"min":{"1min":0.34499999999999997,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":11.744999999999999,"5min":11.744999999999999,"15min":11.744999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":2.1659999999999999,"5min":1.3720000000000001,"15min":1.4970000000000001},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":5.5,"5min":5.9509999999999996,"15min":13.836},"last":0.52200000000000002}]},{"osd":11,"last update":"Fri Feb 9 17:42:13 2024","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.2889999999999999,"15min":1.4470000000000001},"min":{"1min":0.46200000000000002,"5min":0.45500000000000002,"15min":0.36099999999999999},"max":{"1min":4.3179999999999996,"5min":7.7450000000000001,"15min":10.273999999999999},"last":5.6189999999999998},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.171,"15min":1.4890000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":4.0620000000000003,"5min":5.2140000000000004,"15min":8.2739999999999991},"last":4.3700000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575147,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2282808,"kb_used_data":1706032,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":91462344,"statfs":{"total":95995035648,"available":93657440256,"internally_reserved":0,"allocated":1746976768,"data_stored":1746684102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":590608837},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:42:28 2024","interfaces":[{"interface":"back","average":{"1min":0.28999999999999998,"5min":1.9850000000000001,"15min":1.758},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.51600000000000001,"5min":19.079000000000001,"15min":19.079000000000001},"last":0.311},{"interface":"front","average":{"1min":0.40999999999999998,"5min":1.881,"15min":1.7390000000000001},"min":{"1min":0.23100000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":1.075,"5min":23.573,"15min":23.573},"last":0.26400000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:42:54 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":1.671,"15min":1.667},"min":{"1min":0.23400000000000001,"5min":0.23200000000000001,"15min":0.22600000000000001},"max":{"1min":1.3029999999999999,"5min":12.52,"15min":26.361999999999998},"last":0.246},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.7030000000000001,"15min":1.919},"min":{"1min":0.27500000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":4.4889999999999999,"5min":9.3810000000000002,"15min":26.087},"last":2.0680000000000001}]},{"osd":3,"last update":"Fri Feb 9 17:40:36 2024","interfaces":[{"interface":"back","average":{"1min":1.8959999999999999,"5min":2.2690000000000001,"15min":1.6060000000000001},"min":{"1min":0.26200000000000001,"5min":0.245,"15min":0.245},"max":{"1min":6.9400000000000004,"5min":10.657,"15min":10.657},"last":0.30299999999999999},{"interface":"front","average":{"1min":2.2349999999999999,"5min":2.508,"15min":1.734},"min":{"1min":0.36599999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":7.1029999999999998,"5min":14.68,"15min":14.68},"last":1.25}]},{"osd":4,"last update":"Fri Feb 9 17:42:22 2024","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.8360000000000001,"15min":1.909},"min":{"1min":0.27700000000000002,"5min":0.25800000000000001,"15min":0.253},"max":{"1min":6.5140000000000002,"5min":9.4649999999999999,"15min":11.766999999999999},"last":5.79},{"interface":"front","average":{"1min":0.86099999999999999,"5min":1.6379999999999999,"15min":1.8320000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":5.0179999999999998,"5min":10.228,"15min":10.512},"last":0.35299999999999998}]},{"osd":5,"last update":"Fri Feb 9 17:42:09 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.9319999999999999,"15min":1.8480000000000001},"min":{"1min":0.29199999999999998,"5min":0.28999999999999998,"15min":0.27000000000000002},"max":{"1min":5.9429999999999996,"5min":9.4299999999999997,"15min":13.544},"last":2.1230000000000002},{"interface":"front","average":{"1min":1.323,"5min":2.0419999999999998,"15min":1.774},"min":{"1min":0.32300000000000001,"5min":0.30499999999999999,"15min":0.26000000000000001},"max":{"1min":5.9569999999999999,"5min":10.531000000000001,"15min":12.117000000000001},"last":0.47999999999999998}]},{"osd":6,"last update":"Fri Feb 9 17:42:13 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.2609999999999999,"15min":1.2689999999999999},"min":{"1min":0.436,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":4.9160000000000004,"5min":5.0119999999999996,"15min":5.0119999999999996},"last":0.44700000000000001},{"interface":"front","average":{"1min":1.238,"5min":1.1870000000000001,"15min":1.179},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":4.952,"5min":4.952,"15min":4.952},"last":0.51600000000000001}]},{"osd":7,"last update":"Fri Feb 9 17:42:13 2024","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":1.147,"15min":1.153},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":3.8700000000000001,"5min":4.0439999999999996,"15min":4.0439999999999996},"last":1.2749999999999999},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.21,"15min":1.228},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":4.0389999999999997,"5min":4.9820000000000002,"15min":4.9820000000000002},"last":1.2969999999999999}]},{"osd":8,"last update":"Fri Feb 9 17:42:22 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.83299999999999996,"15min":0.872},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.296,"5min":2.121,"15min":2.121},"last":0.5},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.81000000000000005,"15min":0.83699999999999997},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.325,"5min":2.1379999999999999,"15min":2.1379999999999999},"last":0.58099999999999996}]},{"osd":9,"last update":"Fri Feb 9 17:42:28 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":1.7250000000000001,"15min":1.6579999999999999},"min":{"1min":0.33900000000000002,"5min":0.33400000000000002,"15min":0.32400000000000001},"max":{"1min":2.516,"5min":9.4960000000000004,"15min":9.5839999999999996},"last":0.46000000000000002},{"interface":"front","average":{"1min":1.331,"5min":1.869,"15min":1.736},"min":{"1min":0.375,"5min":0.32900000000000001,"15min":0.30099999999999999},"max":{"1min":4.3179999999999996,"5min":10.548,"15min":12.117000000000001},"last":2.1429999999999998}]},{"osd":10,"last update":"Fri Feb 9 17:42:09 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":2.0099999999999998,"15min":1.6559999999999999},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.35999999999999999},"max":{"1min":1.1699999999999999,"5min":15.083,"15min":15.083},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.77400000000000002,"5min":2.0529999999999999,"15min":1.871},"min":{"1min":0.51200000000000001,"5min":0.48199999999999998,"15min":0.33400000000000002},"max":{"1min":1.268,"5min":15.121,"15min":15.121},"last":0.55900000000000005}]},{"osd":11,"last update":"Fri Feb 9 17:42:13 2024","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":2.1509999999999998,"5min":2.1509999999999998,"15min":2.1509999999999998},"last":3.5670000000000002},{"interface":"front","average":{"1min":1.226,"5min":1.226,"15min":1.226},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":3.0910000000000002,"5min":3.0910000000000002,"15min":3.0910000000000002},"last":2.5510000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738671,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3575736,"kb_used_data":3025328,"kb_used_omap":1,"kb_used_meta":550398,"kb_avail":90169416,"statfs":{"total":95995035648,"available":92333481984,"internally_reserved":0,"allocated":3097935872,"data_stored":3097643206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":563608005},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 9 17:42:11 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":1.833,"15min":1.6000000000000001},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.20499999999999999},"max":{"1min":3.6600000000000001,"5min":13.842000000000001,"15min":14.166},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.77100000000000002,"5min":2.0859999999999999,"15min":1.833},"min":{"1min":0.32000000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.7050000000000001,"5min":14.122,"15min":20.177},"last":0.36299999999999999}]},{"osd":2,"last update":"Fri Feb 9 17:39:25 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":5.5339999999999998,"5min":5.5339999999999998,"15min":5.5339999999999998},"last":1.744},{"interface":"front","average":{"1min":1.593,"5min":1.593,"15min":1.593},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":13.473000000000001,"5min":13.473000000000001,"15min":13.473000000000001},"last":3.109}]},{"osd":3,"last update":"Fri Feb 9 17:42:45 2024","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":2.1389999999999998,"15min":2.7280000000000002},"min":{"1min":0.29199999999999998,"5min":0.247,"15min":0.247},"max":{"1min":4.3849999999999998,"5min":16.756,"15min":16.756},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.90400000000000003,"5min":1.3580000000000001,"15min":1.6559999999999999},"min":{"1min":0.252,"5min":0.221,"15min":0.221},"max":{"1min":4.165,"5min":6.0570000000000004,"15min":6.0570000000000004},"last":0.42099999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:42:31 2024","interfaces":[{"interface":"back","average":{"1min":1.6830000000000001,"5min":1.3660000000000001,"15min":1.655},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.26400000000000001},"max":{"1min":6.5609999999999999,"5min":7.7560000000000002,"15min":14.419},"last":0.442},{"interface":"front","average":{"1min":1.651,"5min":1.8140000000000001,"15min":1.8879999999999999},"min":{"1min":0.35899999999999999,"5min":0.28000000000000003,"15min":0.27200000000000002},"max":{"1min":6.5919999999999996,"5min":13.308999999999999,"15min":14.465999999999999},"last":0.51200000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:42:51 2024","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.367,"15min":1.6080000000000001},"min":{"1min":0.31,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":4.1479999999999997,"5min":8.4079999999999995,"15min":9.8569999999999993},"last":1.397},{"interface":"front","average":{"1min":1.294,"5min":1.1779999999999999,"15min":1.7050000000000001},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":3.8639999999999999,"5min":5.7839999999999998,"15min":9.9049999999999994},"last":0.54300000000000004}]},{"osd":6,"last update":"Fri Feb 9 17:42:21 2024","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.722,"15min":1.8480000000000001},"min":{"1min":0.55000000000000004,"5min":0.33300000000000002,"15min":0.27800000000000002},"max":{"1min":3.411,"5min":13.407,"15min":13.407},"last":1.3300000000000001},{"interface":"front","average":{"1min":1.538,"5min":1.772,"15min":1.8799999999999999},"min":{"1min":0.57899999999999996,"5min":0.39200000000000002,"15min":0.33000000000000002},"max":{"1min":3.4460000000000002,"5min":13.448,"15min":13.448},"last":1.353}]},{"osd":7,"last update":"Fri Feb 9 17:42:45 2024","interfaces":[{"interface":"back","average":{"1min":1.8520000000000001,"5min":2.0329999999999999,"15min":2.0219999999999998},"min":{"1min":0.312,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":5.5449999999999999,"5min":13.349,"15min":13.349},"last":0.57199999999999995},{"interface":"front","average":{"1min":1.165,"5min":1.851,"15min":1.9730000000000001},"min":{"1min":0.33300000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":3.3719999999999999,"5min":14.451000000000001,"15min":16.385000000000002},"last":1.49}]},{"osd":8,"last update":"Fri Feb 9 17:42:45 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.7689999999999999,"15min":1.538},"min":{"1min":0.33900000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":4.4770000000000003,"5min":14.323,"15min":14.323},"last":0.76400000000000001},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.875,"15min":1.6479999999999999},"min":{"1min":0.376,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":4.5579999999999998,"5min":14.481999999999999,"15min":16.41},"last":0.627}]},{"osd":9,"last update":"Fri Feb 9 17:42:31 2024","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":2.0630000000000002,"15min":1.7709999999999999},"min":{"1min":0.33800000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":4.1139999999999999,"5min":17.952000000000002,"15min":17.952000000000002},"last":0.69399999999999995},{"interface":"front","average":{"1min":1.6379999999999999,"5min":2.0859999999999999,"15min":1.8120000000000001},"min":{"1min":0.40999999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":4.7839999999999998,"5min":17.913,"15min":17.913},"last":0.82099999999999995}]},{"osd":10,"last update":"Fri Feb 9 17:42:51 2024","interfaces":[{"interface":"back","average":{"1min":2.8250000000000002,"5min":2.2029999999999998,"15min":1.8540000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":31.766999999999999,"5min":31.766999999999999,"15min":31.766999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":2.609,"5min":1.333,"15min":1.53},"min":{"1min":0.42199999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":28.048999999999999,"5min":28.048999999999999,"15min":28.048999999999999},"last":0.94099999999999995}]},{"osd":11,"last update":"Fri Feb 9 17:42:21 2024","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.2729999999999999,"15min":1.6739999999999999},"min":{"1min":0.44600000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":4.4550000000000001,"5min":7.8739999999999997,"15min":40.021000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.397,"15min":1.7549999999999999},"min":{"1min":0.48199999999999998,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":3.9630000000000001,"5min":7.899,"15min":40.055999999999997},"last":0.67900000000000005}]}]},{"osd":2,"up_from":0,"seq":77309411621,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":27,"seq":115964117282,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":828496,"kb_used_data":409868,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":92916656,"statfs":{"total":95995035648,"available":95146655744,"internally_reserved":0,"allocated":419704832,"data_stored":419448399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":428538311},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:42:09 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":2.423,"15min":1.8640000000000001},"min":{"1min":0.23300000000000001,"5min":0.186,"15min":0.186},"max":{"1min":15.657999999999999,"5min":38.984999999999999,"15min":38.984999999999999},"last":0.33700000000000002},{"interface":"front","average":{"1min":1.3839999999999999,"5min":2.528,"15min":1.97},"min":{"1min":0.25800000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":16.637,"5min":41.497999999999998,"15min":41.497999999999998},"last":0.36099999999999999}]},{"osd":1,"last update":"Fri Feb 9 17:40:06 2024","interfaces":[{"interface":"back","average":{"1min":4.5670000000000002,"5min":4.5670000000000002,"15min":4.5670000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":25.965,"5min":25.965,"15min":25.965},"last":0.35299999999999998},{"interface":"front","average":{"1min":4.5129999999999999,"5min":4.5129999999999999,"15min":4.5129999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":31.454000000000001,"5min":31.454000000000001,"15min":31.454000000000001},"last":0.375}]},{"osd":2,"last update":"Fri Feb 9 17:42:04 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":2.0600000000000001,"15min":1.802},"min":{"1min":0.32200000000000001,"5min":0.255,"15min":0.22600000000000001},"max":{"1min":4.2409999999999997,"5min":11.663,"15min":14.914999999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.67300000000000004,"5min":1.9239999999999999,"15min":1.8260000000000001},"min":{"1min":0.27600000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":4.258,"5min":12.347,"15min":17.613},"last":0.38500000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:42:11 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":1.4670000000000001,"15min":1.806},"min":{"1min":0.30399999999999999,"5min":0.29099999999999998,"15min":0.26800000000000002},"max":{"1min":2.556,"5min":13.933,"15min":34.898000000000003},"last":0.43099999999999999},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.645,"15min":1.788},"min":{"1min":0.32800000000000001,"5min":0.28000000000000003,"15min":0.27900000000000003},"max":{"1min":4.415,"5min":9.3059999999999992,"15min":14.061999999999999},"last":1.496}]},{"osd":5,"last update":"Fri Feb 9 17:42:49 2024","interfaces":[{"interface":"back","average":{"1min":1.774,"5min":1.698,"15min":1.6919999999999999},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.27800000000000002},"max":{"1min":10.782999999999999,"5min":11.007999999999999,"15min":17.651},"last":0.45000000000000001},{"interface":"front","average":{"1min":1.8149999999999999,"5min":1.762,"15min":1.746},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.34100000000000003},"max":{"1min":10.805999999999999,"5min":11.036,"15min":17.675999999999998},"last":0.625}]},{"osd":6,"last update":"Fri Feb 9 17:42:39 2024","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":2.0459999999999998,"15min":2.512},"min":{"1min":0.36599999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":6.4669999999999996,"5min":12.717000000000001,"15min":12.717000000000001},"last":3.8769999999999998},{"interface":"front","average":{"1min":1.528,"5min":1.321,"15min":1.377},"min":{"1min":0.38400000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":4.6349999999999998,"5min":8.7360000000000007,"15min":8.7360000000000007},"last":0.65700000000000003}]},{"osd":7,"last update":"Fri Feb 9 17:42:20 2024","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":0.96299999999999997,"15min":0.90200000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":3.2320000000000002,"5min":3.3999999999999999,"15min":3.3999999999999999},"last":8.4450000000000003},{"interface":"front","average":{"1min":1.0489999999999999,"5min":0.81299999999999994,"15min":0.77300000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":3.8610000000000002,"5min":3.8610000000000002,"15min":3.8610000000000002},"last":0.53400000000000003}]},{"osd":8,"last update":"Fri Feb 9 17:42:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.653,"15min":1.427},"min":{"1min":0.41299999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":4.3289999999999997,"5min":8.6940000000000008,"15min":8.6940000000000008},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.7569999999999999,"15min":1.222},"min":{"1min":0.42799999999999999,"5min":0.36599999999999999,"15min":0.34499999999999997},"max":{"1min":3.6309999999999998,"5min":13.976000000000001,"15min":14.129},"last":0.68300000000000005}]},{"osd":9,"last update":"Fri Feb 9 17:42:34 2024","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":1.5369999999999999,"15min":1.536},"min":{"1min":0.46000000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":3.698,"5min":9.7929999999999993,"15min":9.7929999999999993},"last":2.8149999999999999},{"interface":"front","average":{"1min":0.97399999999999998,"5min":1.5049999999999999,"15min":1.593},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.32800000000000001},"max":{"1min":3.2650000000000001,"5min":8.7899999999999991,"15min":13.717000000000001},"last":0.54700000000000004}]},{"osd":10,"last update":"Fri Feb 9 17:42:18 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.88500000000000001,"15min":0.86899999999999999},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":3.2120000000000002,"5min":3.2120000000000002,"15min":3.2120000000000002},"last":2.7069999999999999},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.75800000000000001,"15min":0.753},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":2.6179999999999999,"5min":2.6179999999999999,"15min":2.6179999999999999},"last":0.47999999999999998}]},{"osd":11,"last update":"Fri Feb 9 17:42:11 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":1.5680000000000001,"15min":1.609},"min":{"1min":0.42699999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.7410000000000001,"5min":14.300000000000001,"15min":14.300000000000001},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.87,"5min":2.1859999999999999,"15min":1.8779999999999999},"min":{"1min":0.50700000000000001,"5min":0.442,"15min":0.36899999999999999},"max":{"1min":1.917,"5min":17.145,"15min":17.145},"last":0.83299999999999996}]}]},{"osd":4,"up_from":32,"seq":137438953755,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3056760,"kb_used_data":2538288,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":90688392,"statfs":{"total":95995035648,"available":92864913408,"internally_reserved":0,"allocated":2599206912,"data_stored":2598914246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":530905541},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":1.4530000000000001,"15min":1.7130000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":3.1600000000000001,"5min":7.7199999999999998,"15min":50.356999999999999},"last":1.006},{"interface":"front","average":{"1min":0.83299999999999996,"5min":1.6020000000000001,"15min":1.6459999999999999},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.247},"max":{"1min":3.137,"5min":7.8220000000000001,"15min":50.569000000000003},"last":0.97499999999999998}]},{"osd":1,"last update":"Fri Feb 9 17:42:18 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.9850000000000001,"15min":2.089},"min":{"1min":0.27000000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":7.3920000000000003,"5min":8.0609999999999999,"15min":50.473999999999997},"last":0.52800000000000002},{"interface":"front","average":{"1min":1.5660000000000001,"5min":2.0219999999999998,"15min":2.1299999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28100000000000003},"max":{"1min":6.4939999999999998,"5min":8.0990000000000002,"15min":50.499000000000002},"last":1.022}]},{"osd":2,"last update":"Fri Feb 9 17:28:51 2024","interfaces":[{"interface":"back","average":{"1min":2.536,"5min":1.546,"15min":0.91400000000000003},"min":{"1min":0.29999999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":8.8819999999999997,"5min":8.8819999999999997,"15min":8.8819999999999997},"last":0.622},{"interface":"front","average":{"1min":2.214,"5min":1.7330000000000001,"15min":0.97799999999999998},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":9.5039999999999996,"5min":9.5039999999999996,"15min":9.5039999999999996},"last":0.53800000000000003}]},{"osd":3,"last update":"Fri Feb 9 17:42:18 2024","interfaces":[{"interface":"back","average":{"1min":1.591,"5min":1.8049999999999999,"15min":2.1560000000000001},"min":{"1min":0.317,"5min":0.28199999999999997,"15min":0.245},"max":{"1min":6.548,"5min":7.9349999999999996,"15min":50.462000000000003},"last":1.103},{"interface":"front","average":{"1min":0.75,"5min":1.607,"15min":1.9850000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.255},"max":{"1min":2.7799999999999998,"5min":9.1579999999999995,"15min":50.529000000000003},"last":0.88}]},{"osd":5,"last update":"Fri Feb 9 17:42:49 2024","interfaces":[{"interface":"back","average":{"1min":2.9689999999999999,"5min":1.3799999999999999,"15min":1.47},"min":{"1min":0.28399999999999997,"5min":0.22500000000000001,"15min":0.222},"max":{"1min":9.3279999999999994,"5min":9.3279999999999994,"15min":9.5630000000000006},"last":0.55900000000000005},{"interface":"front","average":{"1min":3.2400000000000002,"5min":1.504,"15min":1.7569999999999999},"min":{"1min":0.35699999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":9.3599999999999994,"5min":9.3599999999999994,"15min":50.509999999999998},"last":0.58399999999999996}]},{"osd":6,"last update":"Fri Feb 9 17:42:18 2024","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.3260000000000001,"15min":1.4710000000000001},"min":{"1min":0.35999999999999999,"5min":0.246,"15min":0.246},"max":{"1min":4.2329999999999997,"5min":8.0039999999999996,"15min":18.09},"last":0.84599999999999997},{"interface":"front","average":{"1min":2.415,"5min":1.421,"15min":1.7090000000000001},"min":{"1min":0.312,"5min":0.245,"15min":0.245},"max":{"1min":7.6619999999999999,"5min":7.6619999999999999,"15min":50.402999999999999},"last":4.5570000000000004}]},{"osd":7,"last update":"Fri Feb 9 17:42:54 2024","interfaces":[{"interface":"back","average":{"1min":3.0289999999999999,"5min":1.585,"15min":1.6850000000000001},"min":{"1min":0.42399999999999999,"5min":0.32200000000000001,"15min":0.28399999999999997},"max":{"1min":11.875,"5min":11.875,"15min":14.919},"last":4.3570000000000002},{"interface":"front","average":{"1min":3.4289999999999998,"5min":1.6279999999999999,"15min":1.7290000000000001},"min":{"1min":0.28599999999999998,"5min":0.28199999999999997,"15min":0.28100000000000003},"max":{"1min":12.696,"5min":12.696,"15min":19.047000000000001},"last":1.1359999999999999}]},{"osd":8,"last update":"Fri Feb 9 17:42:49 2024","interfaces":[{"interface":"back","average":{"1min":2.5550000000000002,"5min":1.399,"15min":1.502},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":9.1809999999999992,"5min":9.1809999999999992,"15min":9.1809999999999992},"last":5.359},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.149,"15min":1.5669999999999999},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.308},"max":{"1min":5.2809999999999997,"5min":6,"15min":50.594000000000001},"last":0.94699999999999995}]},{"osd":9,"last update":"Fri Feb 9 17:42:33 2024","interfaces":[{"interface":"back","average":{"1min":2.6589999999999998,"5min":1.4530000000000001,"15min":1.748},"min":{"1min":0.371,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":9.2370000000000001,"5min":9.2370000000000001,"15min":50.442999999999998},"last":5.4290000000000003},{"interface":"front","average":{"1min":1.403,"5min":1.353,"15min":1.4630000000000001},"min":{"1min":0.435,"5min":0.39000000000000001,"15min":0.36199999999999999},"max":{"1min":8.3580000000000005,"5min":8.3580000000000005,"15min":10.555},"last":0.91700000000000004}]},{"osd":10,"last update":"Fri Feb 9 17:42:03 2024","interfaces":[{"interface":"back","average":{"1min":1.4359999999999999,"5min":1.607,"15min":1.891},"min":{"1min":0.52500000000000002,"5min":0.40300000000000002,"15min":0.378},"max":{"1min":3.6160000000000001,"5min":7.0030000000000001,"15min":51.223999999999997},"last":5.3959999999999999},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.4610000000000001,"15min":1.9099999999999999},"min":{"1min":0.42899999999999999,"5min":0.41999999999999998,"15min":0.38100000000000001},"max":{"1min":3.617,"5min":6.3860000000000001,"15min":50.552},"last":1.121}]},{"osd":11,"last update":"Fri Feb 9 17:42:33 2024","interfaces":[{"interface":"back","average":{"1min":2.3959999999999999,"5min":1.718,"15min":2.0150000000000001},"min":{"1min":0.50900000000000001,"5min":0.49099999999999999,"15min":0.40500000000000003},"max":{"1min":9.4060000000000006,"5min":9.4060000000000006,"15min":50.484999999999999},"last":5.4569999999999999},{"interface":"front","average":{"1min":1.135,"5min":1.3740000000000001,"15min":1.6220000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.34899999999999998},"max":{"1min":3.4409999999999998,"5min":7.694,"15min":18.574999999999999},"last":1.6020000000000001}]}]},{"osd":5,"up_from":40,"seq":171798692117,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3414020,"kb_used_data":2999676,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":90331132,"statfs":{"total":95995035648,"available":92499079168,"internally_reserved":0,"allocated":3071668224,"data_stored":3071367934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:42:11 2024","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.046,"15min":1.05},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":7.3730000000000002,"5min":7.3730000000000002,"15min":7.3730000000000002},"last":0.42099999999999999},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.51,"15min":1.712},"min":{"1min":0.28699999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":3.2589999999999999,"5min":6.4939999999999998,"15min":6.4939999999999998},"last":3.4180000000000001}]},{"osd":1,"last update":"Fri Feb 9 17:42:52 2024","interfaces":[{"interface":"back","average":{"1min":1.994,"5min":1.95,"15min":1.9730000000000001},"min":{"1min":0.311,"5min":0.24199999999999999,"15min":0.23200000000000001},"max":{"1min":9.7330000000000005,"5min":48.161999999999999,"15min":48.161999999999999},"last":3.4470000000000001},{"interface":"front","average":{"1min":2.1680000000000001,"5min":2.1890000000000001,"15min":1.96},"min":{"1min":0.30399999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":7.8369999999999997,"5min":48.218000000000004,"15min":48.218000000000004},"last":3.5230000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:40:42 2024","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":1.345,"15min":1.6539999999999999},"min":{"1min":0.32000000000000001,"5min":0.27900000000000003,"15min":0.25600000000000001},"max":{"1min":4.4930000000000003,"5min":11.143000000000001,"15min":11.143000000000001},"last":2.6560000000000001},{"interface":"front","average":{"1min":1.796,"5min":1.218,"15min":1.718},"min":{"1min":0.39100000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":6.1970000000000001,"5min":6.1970000000000001,"15min":14.122},"last":1.5649999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:42:46 2024","interfaces":[{"interface":"back","average":{"1min":2.194,"5min":1.778,"15min":2.105},"min":{"1min":0.33100000000000002,"5min":0.32300000000000001,"15min":0.26500000000000001},"max":{"1min":7.8019999999999996,"5min":16.59,"15min":64.150999999999996},"last":0.47499999999999998},{"interface":"front","average":{"1min":2.0129999999999999,"5min":1.8580000000000001,"15min":2.1549999999999998},"min":{"1min":0.36299999999999999,"5min":0.247,"15min":0.247},"max":{"1min":9.7949999999999999,"5min":12.468,"15min":64.176000000000002},"last":3.472}]},{"osd":4,"last update":"Fri Feb 9 17:42:46 2024","interfaces":[{"interface":"back","average":{"1min":2.6400000000000001,"5min":1.9530000000000001,"15min":1.651},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.23400000000000001},"max":{"1min":8.1980000000000004,"5min":49.313000000000002,"15min":49.313000000000002},"last":0.53700000000000003},{"interface":"front","average":{"1min":2.8149999999999999,"5min":1.871,"15min":1.621},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":9.6869999999999994,"5min":50.505000000000003,"15min":50.505000000000003},"last":3.5379999999999998}]},{"osd":6,"last update":"Fri Feb 9 17:42:22 2024","interfaces":[{"interface":"back","average":{"1min":2.8479999999999999,"5min":1.597,"15min":1.554},"min":{"1min":0.28199999999999997,"5min":0.26400000000000001,"15min":0.253},"max":{"1min":12.425000000000001,"5min":12.425000000000001,"15min":15.375999999999999},"last":3.5070000000000001},{"interface":"front","average":{"1min":3.121,"5min":1.696,"15min":1.677},"min":{"1min":0.374,"5min":0.32800000000000001,"15min":0.28899999999999998},"max":{"1min":11.199,"5min":11.199,"15min":14.250999999999999},"last":0.51000000000000001}]},{"osd":7,"last update":"Fri Feb 9 17:42:46 2024","interfaces":[{"interface":"back","average":{"1min":3.1779999999999999,"5min":1.9450000000000001,"15min":1.643},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.26700000000000002},"max":{"1min":12.475,"5min":50.545000000000002,"15min":50.545000000000002},"last":3.4889999999999999},{"interface":"front","average":{"1min":3.3180000000000001,"5min":3,"15min":2.1869999999999998},"min":{"1min":0.42099999999999999,"5min":0.32100000000000001,"15min":0.28799999999999998},"max":{"1min":10.657,"5min":102.526,"15min":102.526},"last":3.2469999999999999}]},{"osd":8,"last update":"Fri Feb 9 17:42:22 2024","interfaces":[{"interface":"back","average":{"1min":2.2629999999999999,"5min":1.579,"15min":1.6120000000000001},"min":{"1min":0.433,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":9.8520000000000003,"5min":14.542,"15min":14.542},"last":3.5569999999999999},{"interface":"front","average":{"1min":1.9770000000000001,"5min":1.181,"15min":1.7410000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":7.734,"5min":7.734,"15min":9.6240000000000006},"last":3.702}]},{"osd":9,"last update":"Fri Feb 9 17:42:22 2024","interfaces":[{"interface":"back","average":{"1min":2.2930000000000001,"5min":1.4359999999999999,"15min":1.4550000000000001},"min":{"1min":0.40300000000000002,"5min":0.38600000000000001,"15min":0.371},"max":{"1min":7.4180000000000001,"5min":7.4180000000000001,"15min":14.98},"last":2.0670000000000002},{"interface":"front","average":{"1min":1.972,"5min":1.3200000000000001,"15min":1.417},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.312},"max":{"1min":7.7910000000000004,"5min":7.7910000000000004,"15min":9.4239999999999995},"last":2.6829999999999998}]},{"osd":10,"last update":"Fri Feb 9 17:42:46 2024","interfaces":[{"interface":"back","average":{"1min":3.085,"5min":1.5129999999999999,"15min":1.5820000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":9.8829999999999991,"5min":9.8829999999999991,"15min":14.218},"last":3.6179999999999999},{"interface":"front","average":{"1min":2.6619999999999999,"5min":1.5960000000000001,"15min":1.6539999999999999},"min":{"1min":0.46999999999999997,"5min":0.41999999999999998,"15min":0.36099999999999999},"max":{"1min":9.8840000000000003,"5min":10.606,"15min":14.186999999999999},"last":3.5760000000000001}]},{"osd":11,"last update":"Fri Feb 9 17:42:11 2024","interfaces":[{"interface":"back","average":{"1min":1.5369999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":3.597,"5min":3.597,"15min":3.597},"last":4.1420000000000003},{"interface":"front","average":{"1min":1.653,"5min":1.653,"15min":1.653},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":5.593,"5min":5.593,"15min":5.593},"last":3.605}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3074293760,"data_stored":3074293760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1744568320,"data_stored":1744568320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":418185216,"data_stored":418185216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2578579456,"data_stored":2578579456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3048083456,"data_stored":3048079384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1299972096,"data_stored":1299972096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1704660992,"data_stored":1704656920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1269825536,"data_stored":1269825536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1284898816,"data_stored":1284898816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2652700672,"data_stored":2652700672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1743851520,"data_stored":1743847448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-09T17:43:06.981 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-02-09T17:43:06.981 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd pool set .mgr pg_num 31 2024-02-09T17:43:07.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:06 smithi099 ceph-mon[39420]: 1.0 scrub starts 2024-02-09T17:43:07.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:06 smithi099 ceph-mon[39420]: 1.0 scrub ok 2024-02-09T17:43:07.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:06 smithi099 ceph-mon[39420]: 5.d scrub ok 2024-02-09T17:43:07.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:06 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:43:07.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:06 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:43:07.332 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:43:07.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:06 smithi179 ceph-mon[43975]: 1.0 scrub starts 2024-02-09T17:43:07.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:06 smithi179 ceph-mon[43975]: 1.0 scrub ok 2024-02-09T17:43:07.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:06 smithi179 ceph-mon[43975]: 5.d scrub ok 2024-02-09T17:43:07.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:06 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:43:07.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:06 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:43:07.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:07.335 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:43:07.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:06 smithi162 ceph-mon[44304]: 1.0 scrub starts 2024-02-09T17:43:07.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:06 smithi162 ceph-mon[44304]: 1.0 scrub ok 2024-02-09T17:43:07.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:06 smithi162 ceph-mon[44304]: 5.d scrub ok 2024-02-09T17:43:07.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:06 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:43:07.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:06 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:43:07.435 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:43:07.466 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:07.502 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:43:07.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:07.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:43:07.507 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:43:07.607 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:43:07.639 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:07.770 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:43:07.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:07.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:43:07.874 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:43:07.909 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:08.012 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:43:08.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:08.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:43:08.028 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:07 smithi099 ceph-mon[39420]: 1.2 scrub starts 2024-02-09T17:43:08.028 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:07 smithi099 ceph-mon[39420]: 1.2 scrub ok 2024-02-09T17:43:08.028 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:07 smithi099 ceph-mon[39420]: pgmap v1301: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 6.2 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 984 op/s 2024-02-09T17:43:08.028 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:07 smithi099 ceph-mon[39420]: 1.a scrub starts 2024-02-09T17:43:08.028 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:07 smithi099 ceph-mon[39420]: 1.a scrub ok 2024-02-09T17:43:08.028 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:07 smithi099 ceph-mon[39420]: osdmap e511: 12 total, 12 up, 11 in 2024-02-09T17:43:08.120 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:43:08.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:07 smithi179 ceph-mon[43975]: 1.2 scrub starts 2024-02-09T17:43:08.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:07 smithi179 ceph-mon[43975]: 1.2 scrub ok 2024-02-09T17:43:08.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:07 smithi179 ceph-mon[43975]: pgmap v1301: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 6.2 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 984 op/s 2024-02-09T17:43:08.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:07 smithi179 ceph-mon[43975]: 1.a scrub starts 2024-02-09T17:43:08.175 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:07 smithi179 ceph-mon[43975]: 1.a scrub ok 2024-02-09T17:43:08.175 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:07 smithi179 ceph-mon[43975]: osdmap e511: 12 total, 12 up, 11 in 2024-02-09T17:43:08.182 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:08.220 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:43:08.225 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:07 smithi162 ceph-mon[44304]: 1.2 scrub starts 2024-02-09T17:43:08.226 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:07 smithi162 ceph-mon[44304]: 1.2 scrub ok 2024-02-09T17:43:08.226 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:07 smithi162 ceph-mon[44304]: pgmap v1301: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 6.2 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 984 op/s 2024-02-09T17:43:08.226 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:07 smithi162 ceph-mon[44304]: 1.a scrub starts 2024-02-09T17:43:08.226 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:07 smithi162 ceph-mon[44304]: 1.a scrub ok 2024-02-09T17:43:08.226 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:07 smithi162 ceph-mon[44304]: osdmap e511: 12 total, 12 up, 11 in 2024-02-09T17:43:08.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:08.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:43:08.328 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:43:08.332 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:43:08.391 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:08.587 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:43:08.588 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-02-09T17:43:08.588 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-02-09T17:43:08.588 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-02-09T17:43:08.588 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:43:08.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:08.761 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:43:08.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:43:08.862 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:43:08.927 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:08.972 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:43:08.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:08.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:43:08.981 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is unset 2024-02-09T17:43:09.078 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:43:09.135 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:09.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:08 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3550829867' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:43:09.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:08 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:43:09.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:08 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3550829867' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:43:09.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:08 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:43:09.283 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:43:09.284 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:08 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3550829867' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:43:09.284 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:08 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:43:09.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:09.289 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:43:09.327 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.10 dump_historic_ops 2024-02-09T17:43:09.390 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:43:09.425 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:09.631 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:43:09.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:09.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:43:09.735 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:43:09.749 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:43:09.782 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:09.813 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:43:09.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:09.822 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:43:09.923 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:43:09.952 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:10.078 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:09 smithi179 ceph-mon[43975]: 1.b scrub starts 2024-02-09T17:43:10.078 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:09 smithi179 ceph-mon[43975]: 1.b scrub ok 2024-02-09T17:43:10.078 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:09 smithi179 ceph-mon[43975]: 5.9 scrub starts 2024-02-09T17:43:10.078 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:09 smithi179 ceph-mon[43975]: pgmap v1303: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 6.2 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 836 op/s 2024-02-09T17:43:10.079 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:09 smithi179 ceph-mon[43975]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:43:10.079 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:09 smithi179 ceph-mon[43975]: Cluster is now healthy 2024-02-09T17:43:10.079 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:09 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:43:10.079 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:09 smithi179 ceph-mon[43975]: osdmap e512: 12 total, 12 up, 11 in 2024-02-09T17:43:10.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:09 smithi099 ceph-mon[39420]: 1.b scrub starts 2024-02-09T17:43:10.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:09 smithi099 ceph-mon[39420]: 1.b scrub ok 2024-02-09T17:43:10.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:09 smithi099 ceph-mon[39420]: 5.9 scrub starts 2024-02-09T17:43:10.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:09 smithi099 ceph-mon[39420]: pgmap v1303: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 6.2 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 836 op/s 2024-02-09T17:43:10.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:09 smithi099 ceph-mon[39420]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:43:10.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:09 smithi099 ceph-mon[39420]: Cluster is now healthy 2024-02-09T17:43:10.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:09 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:43:10.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:09 smithi099 ceph-mon[39420]: osdmap e512: 12 total, 12 up, 11 in 2024-02-09T17:43:10.406 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:43:10.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:10.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:43:10.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:09 smithi162 ceph-mon[44304]: 1.b scrub starts 2024-02-09T17:43:10.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:09 smithi162 ceph-mon[44304]: 1.b scrub ok 2024-02-09T17:43:10.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:09 smithi162 ceph-mon[44304]: 5.9 scrub starts 2024-02-09T17:43:10.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:09 smithi162 ceph-mon[44304]: pgmap v1303: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 6.2 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 836 op/s 2024-02-09T17:43:10.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:09 smithi162 ceph-mon[44304]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:43:10.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:09 smithi162 ceph-mon[44304]: Cluster is now healthy 2024-02-09T17:43:10.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:09 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:43:10.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:09 smithi162 ceph-mon[44304]: osdmap e512: 12 total, 12 up, 11 in 2024-02-09T17:43:10.512 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:43:10.558 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:10.606 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:43:10.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:10.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:43:10.701 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:43:10.715 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:43:10.784 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:10.810 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:43:10.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:10.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:43:10.917 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:43:10.957 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:11.226 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:43:11.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:11.236 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:43:11.337 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:43:11.392 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:11.538 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:43:11.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:11.545 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:43:11.646 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:43:11.698 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:11.911 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:43:11.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:11.921 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:43:11.973 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:43:11.974 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-02-09T17:43:11.974 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-02-09T17:43:11.974 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-02-09T17:43:11.974 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.974 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:207589 4.7 4:e905c4e5:::benchmark_data_smithi179_103809_object70123:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-02-09T17:43:11.974 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:32.748961+0000", 2024-02-09T17:43:11.974 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 279.21647945500001, 2024-02-09T17:43:11.974 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.216048499, 2024-02-09T17:43:11.974 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:11.974 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:11.974 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:11.975 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:43:11.975 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:43:11.975 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 207589 2024-02-09T17:43:11.975 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.975 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:11.975 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.975 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:11.975 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.748961+0000", 2024-02-09T17:43:11.975 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:11.975 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.975 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.975 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:11.975 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.748961+0000", 2024-02-09T17:43:11.975 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:11.976 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.976 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.976 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:11.976 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.748962+0000", 2024-02-09T17:43:11.976 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.4600000000000003e-07 2024-02-09T17:43:11.976 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.976 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.976 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:11.976 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.748966+0000", 2024-02-09T17:43:11.976 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2130000000000002e-06 2024-02-09T17:43:11.976 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.976 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.976 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:11.976 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.748976+0000", 2024-02-09T17:43:11.977 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5740000000000002e-06 2024-02-09T17:43:11.977 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.977 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.977 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:11.977 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.748983+0000", 2024-02-09T17:43:11.977 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4499999999999998e-06 2024-02-09T17:43:11.977 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.977 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.977 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:11.977 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.753831+0000", 2024-02-09T17:43:11.977 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0048474240000000004 2024-02-09T17:43:11.977 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.977 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.977 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:11.978 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.753976+0000", 2024-02-09T17:43:11.978 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014569600000000001 2024-02-09T17:43:11.978 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.978 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.978 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,1,7", 2024-02-09T17:43:11.978 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.754124+0000", 2024-02-09T17:43:11.978 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000148104 2024-02-09T17:43:11.978 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.978 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.978 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:11.978 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.763802+0000", 2024-02-09T17:43:11.978 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.009677527 2024-02-09T17:43:11.978 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.978 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.978 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:11.979 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.763872+0000", 2024-02-09T17:43:11.979 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9945999999999997e-05 2024-02-09T17:43:11.979 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.979 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.979 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:11.979 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.783491+0000", 2024-02-09T17:43:11.979 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.019619533000000001 2024-02-09T17:43:11.979 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.979 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.979 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:11.979 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.964888+0000", 2024-02-09T17:43:11.980 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.18139617699999999 2024-02-09T17:43:11.980 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.980 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.980 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:11.980 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.964933+0000", 2024-02-09T17:43:11.980 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5852000000000001e-05 2024-02-09T17:43:11.980 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.980 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.980 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:11.981 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.965009+0000", 2024-02-09T17:43:11.981 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6056999999999994e-05 2024-02-09T17:43:11.981 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:11.981 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:11.981 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:11.981 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.981 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.981 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:207592 4.7 4:e615ae58:::benchmark_data_smithi179_103809_object70126:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-02-09T17:43:11.981 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:32.749662+0000", 2024-02-09T17:43:11.982 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 279.215778398, 2024-02-09T17:43:11.982 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.22551052399999999, 2024-02-09T17:43:11.982 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:11.982 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:11.982 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:11.982 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:43:11.982 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:43:11.982 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 207592 2024-02-09T17:43:11.982 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.983 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:11.983 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.983 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:11.983 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.749662+0000", 2024-02-09T17:43:11.983 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:11.983 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.983 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.983 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:11.983 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.749662+0000", 2024-02-09T17:43:11.984 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:11.984 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.984 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.984 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:11.984 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.749663+0000", 2024-02-09T17:43:11.984 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7900000000000007e-07 2024-02-09T17:43:11.984 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.984 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.984 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:11.984 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.749668+0000", 2024-02-09T17:43:11.984 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0159999999999999e-06 2024-02-09T17:43:11.984 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.984 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.984 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:11.985 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.749685+0000", 2024-02-09T17:43:11.985 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6803000000000002e-05 2024-02-09T17:43:11.985 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.985 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.985 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:11.985 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.749702+0000", 2024-02-09T17:43:11.985 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7421000000000001e-05 2024-02-09T17:43:11.985 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.985 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.985 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:11.985 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.758677+0000", 2024-02-09T17:43:11.985 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0089749089999999997 2024-02-09T17:43:11.985 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.986 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.986 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:11.986 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.758809+0000", 2024-02-09T17:43:11.986 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013181900000000001 2024-02-09T17:43:11.986 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.986 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.986 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,1,7", 2024-02-09T17:43:11.986 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.758936+0000", 2024-02-09T17:43:11.986 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012730900000000001 2024-02-09T17:43:11.986 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.986 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.986 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:11.986 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.773609+0000", 2024-02-09T17:43:11.987 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014672810999999999 2024-02-09T17:43:11.987 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.987 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.987 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:11.987 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.773633+0000", 2024-02-09T17:43:11.987 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4340000000000001e-05 2024-02-09T17:43:11.987 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.987 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.987 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:11.987 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.783535+0000", 2024-02-09T17:43:11.987 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0099020099999999993 2024-02-09T17:43:11.987 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.987 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.987 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:11.988 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.975037+0000", 2024-02-09T17:43:11.988 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.19150186399999999 2024-02-09T17:43:11.988 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.988 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.988 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:11.988 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.975080+0000", 2024-02-09T17:43:11.988 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2839000000000001e-05 2024-02-09T17:43:11.988 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.988 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.988 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:11.988 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.975173+0000", 2024-02-09T17:43:11.988 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2404000000000005e-05 2024-02-09T17:43:11.988 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:11.989 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:11.989 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:11.989 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.989 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.989 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:207599 4.7 4:e2960c10:::benchmark_data_smithi179_103809_object70133:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-02-09T17:43:11.989 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:32.754607+0000", 2024-02-09T17:43:11.989 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 279.21083324799997, 2024-02-09T17:43:11.989 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.23920739299999999, 2024-02-09T17:43:11.989 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:11.989 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:11.989 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:11.989 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:43:11.989 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:43:11.990 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 207599 2024-02-09T17:43:11.990 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.990 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:11.990 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.990 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:11.990 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.754607+0000", 2024-02-09T17:43:11.990 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:11.990 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.990 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.990 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:11.990 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.754607+0000", 2024-02-09T17:43:11.990 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:11.991 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.991 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.991 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:11.991 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.754608+0000", 2024-02-09T17:43:11.991 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1119999999999999e-06 2024-02-09T17:43:11.991 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.991 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.991 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:11.991 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.754613+0000", 2024-02-09T17:43:11.991 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3980000000000001e-06 2024-02-09T17:43:11.991 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.991 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.992 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:11.992 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.754624+0000", 2024-02-09T17:43:11.992 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1541e-05 2024-02-09T17:43:11.992 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.992 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.992 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:11.992 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.754632+0000", 2024-02-09T17:43:11.992 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6129999999999998e-06 2024-02-09T17:43:11.992 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.992 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.992 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:11.993 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.763892+0000", 2024-02-09T17:43:11.993 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0092598009999999998 2024-02-09T17:43:11.993 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.993 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.993 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:11.993 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.764014+0000", 2024-02-09T17:43:11.993 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012283300000000001 2024-02-09T17:43:11.993 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.993 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.993 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,1,7", 2024-02-09T17:43:11.993 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.764148+0000", 2024-02-09T17:43:11.993 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013364299999999999 2024-02-09T17:43:11.993 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.993 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.994 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:11.994 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.773862+0000", 2024-02-09T17:43:11.994 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0097142159999999995 2024-02-09T17:43:11.994 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.994 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.994 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:11.994 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.787249+0000", 2024-02-09T17:43:11.994 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013387125 2024-02-09T17:43:11.994 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.994 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.994 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:11.994 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.787619+0000", 2024-02-09T17:43:11.994 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00036931799999999999 2024-02-09T17:43:11.995 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.995 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.995 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:11.995 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.993634+0000", 2024-02-09T17:43:11.995 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.20601524800000001 2024-02-09T17:43:11.995 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.995 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.995 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:11.995 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.993696+0000", 2024-02-09T17:43:11.995 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2101999999999996e-05 2024-02-09T17:43:11.996 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.996 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.996 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:11.996 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.993815+0000", 2024-02-09T17:43:11.996 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000118443 2024-02-09T17:43:11.996 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:11.996 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:11.996 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:11.996 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.996 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.996 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:207601 4.7 4:ea9472b7:::benchmark_data_smithi179_103809_object70135:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-02-09T17:43:11.996 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:32.758215+0000", 2024-02-09T17:43:11.996 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 279.20722570100003, 2024-02-09T17:43:11.996 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.23582645499999999, 2024-02-09T17:43:11.997 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:11.997 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:11.997 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:11.997 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:43:11.997 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:43:11.997 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 207601 2024-02-09T17:43:11.997 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.997 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:11.997 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.997 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:11.997 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.758215+0000", 2024-02-09T17:43:11.997 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:11.998 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.998 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.998 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:11.998 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.758215+0000", 2024-02-09T17:43:11.998 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:11.998 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.998 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.998 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:11.998 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.758216+0000", 2024-02-09T17:43:11.998 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2720000000000001e-06 2024-02-09T17:43:11.998 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.998 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.998 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:11.998 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.758221+0000", 2024-02-09T17:43:11.999 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.921e-06 2024-02-09T17:43:11.999 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.999 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.999 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:11.999 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.758233+0000", 2024-02-09T17:43:11.999 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1987999999999999e-05 2024-02-09T17:43:11.999 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.999 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.999 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:11.999 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.758241+0000", 2024-02-09T17:43:11.999 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7190000000000001e-06 2024-02-09T17:43:11.999 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:11.999 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:11.999 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:12.000 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.768835+0000", 2024-02-09T17:43:12.000 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010594236 2024-02-09T17:43:12.000 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.000 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.000 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:12.000 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.768976+0000", 2024-02-09T17:43:12.000 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000141125 2024-02-09T17:43:12.000 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.000 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.000 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,1,7", 2024-02-09T17:43:12.000 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.769113+0000", 2024-02-09T17:43:12.000 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013665699999999999 2024-02-09T17:43:12.000 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.001 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.001 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:12.001 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.783190+0000", 2024-02-09T17:43:12.001 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014077838000000001 2024-02-09T17:43:12.001 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.001 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.001 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:12.001 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.787290+0000", 2024-02-09T17:43:12.001 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0040999169999999998 2024-02-09T17:43:12.001 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.001 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.001 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:12.001 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.800012+0000", 2024-02-09T17:43:12.002 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.012721193 2024-02-09T17:43:12.002 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.002 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.002 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:12.002 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.993864+0000", 2024-02-09T17:43:12.002 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.19385191299999999 2024-02-09T17:43:12.002 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.002 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.002 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:12.002 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.993924+0000", 2024-02-09T17:43:12.002 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0930000000000001e-05 2024-02-09T17:43:12.002 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.002 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.002 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:12.002 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.994041+0000", 2024-02-09T17:43:12.003 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000116746 2024-02-09T17:43:12.003 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:12.003 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:12.003 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:12.003 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.003 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.003 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:207604 4.7 4:ef4a4ef9:::benchmark_data_smithi179_103809_object70138:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-02-09T17:43:12.003 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:32.764002+0000", 2024-02-09T17:43:12.003 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 279.20143887699999, 2024-02-09T17:43:12.003 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.23024998399999999, 2024-02-09T17:43:12.003 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:12.003 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:12.003 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:12.003 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:43:12.004 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:43:12.004 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 207604 2024-02-09T17:43:12.004 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.004 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:12.004 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.004 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:12.004 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.764002+0000", 2024-02-09T17:43:12.004 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:12.004 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.004 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.004 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:12.004 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.764002+0000", 2024-02-09T17:43:12.004 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:12.005 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.005 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.005 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:12.005 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.764002+0000", 2024-02-09T17:43:12.005 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5899999999999995e-07 2024-02-09T17:43:12.005 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.005 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.005 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:12.005 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.764005+0000", 2024-02-09T17:43:12.005 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4439999999999998e-06 2024-02-09T17:43:12.005 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.005 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.005 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:12.005 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.764017+0000", 2024-02-09T17:43:12.006 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2299e-05 2024-02-09T17:43:12.006 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.006 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.006 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:12.006 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.764022+0000", 2024-02-09T17:43:12.006 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6349999999999997e-06 2024-02-09T17:43:12.006 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.006 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.006 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:12.006 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.773909+0000", 2024-02-09T17:43:12.006 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0098870529999999998 2024-02-09T17:43:12.006 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.006 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.006 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:12.007 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.774054+0000", 2024-02-09T17:43:12.007 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000145317 2024-02-09T17:43:12.007 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.007 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.007 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,1,7", 2024-02-09T17:43:12.007 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.774215+0000", 2024-02-09T17:43:12.007 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016044900000000001 2024-02-09T17:43:12.007 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.007 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.007 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:12.007 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.783200+0000", 2024-02-09T17:43:12.007 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0089857329999999992 2024-02-09T17:43:12.007 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.007 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.007 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:12.008 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.787330+0000", 2024-02-09T17:43:12.008 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0041292100000000003 2024-02-09T17:43:12.008 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.008 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.008 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:12.008 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.806313+0000", 2024-02-09T17:43:12.008 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.018983659999999999 2024-02-09T17:43:12.008 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.008 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.008 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:12.008 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.994082+0000", 2024-02-09T17:43:12.008 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.18776861 2024-02-09T17:43:12.008 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.008 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.009 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:12.009 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.994143+0000", 2024-02-09T17:43:12.009 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0942e-05 2024-02-09T17:43:12.009 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.009 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.009 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:12.009 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.994252+0000", 2024-02-09T17:43:12.009 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000108773 2024-02-09T17:43:12.009 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:12.009 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:12.009 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:12.009 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.009 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.009 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:207614 4.7 4:e8e480d8:::benchmark_data_smithi179_103809_object70148:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-02-09T17:43:12.009 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:32.778084+0000", 2024-02-09T17:43:12.010 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 279.187356636, 2024-02-09T17:43:12.010 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.21637295300000001, 2024-02-09T17:43:12.010 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:12.010 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:12.010 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:12.010 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:43:12.010 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:43:12.010 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 207614 2024-02-09T17:43:12.010 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.010 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:12.010 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.010 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:12.010 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.778084+0000", 2024-02-09T17:43:12.011 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:12.011 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.011 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.011 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:12.011 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.778084+0000", 2024-02-09T17:43:12.011 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:12.011 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.011 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.011 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:12.011 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.778085+0000", 2024-02-09T17:43:12.011 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5230000000000001e-06 2024-02-09T17:43:12.011 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.011 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.011 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:12.012 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.778090+0000", 2024-02-09T17:43:12.012 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8999999999999997e-06 2024-02-09T17:43:12.012 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.012 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.012 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:12.012 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.778102+0000", 2024-02-09T17:43:12.012 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2276000000000001e-05 2024-02-09T17:43:12.012 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.012 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.012 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:12.012 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.778112+0000", 2024-02-09T17:43:12.012 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.1560000000000001e-06 2024-02-09T17:43:12.012 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.012 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.013 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:12.013 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.778551+0000", 2024-02-09T17:43:12.013 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00043917899999999999 2024-02-09T17:43:12.013 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.013 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.013 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:12.013 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.778680+0000", 2024-02-09T17:43:12.013 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012922000000000001 2024-02-09T17:43:12.013 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.013 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.013 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,1,7", 2024-02-09T17:43:12.013 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.778786+0000", 2024-02-09T17:43:12.013 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000106053 2024-02-09T17:43:12.014 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.014 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.014 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:12.014 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.783509+0000", 2024-02-09T17:43:12.014 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0047230220000000003 2024-02-09T17:43:12.014 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.014 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.014 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:12.014 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.787358+0000", 2024-02-09T17:43:12.014 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0038488490000000001 2024-02-09T17:43:12.014 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.014 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.014 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:12.015 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.806355+0000", 2024-02-09T17:43:12.015 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.018997284 2024-02-09T17:43:12.015 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.015 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.015 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:12.015 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.994292+0000", 2024-02-09T17:43:12.015 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.187936361 2024-02-09T17:43:12.015 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.015 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.015 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:12.015 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.994346+0000", 2024-02-09T17:43:12.015 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4518000000000002e-05 2024-02-09T17:43:12.015 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.016 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.016 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:12.016 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.994457+0000", 2024-02-09T17:43:12.016 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000110612 2024-02-09T17:43:12.016 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:12.016 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:12.016 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:12.016 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.016 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.016 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:207627 4.7 4:e1284f13:::benchmark_data_smithi179_103809_object70161:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-02-09T17:43:12.016 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:32.800287+0000", 2024-02-09T17:43:12.016 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 279.16515382099999, 2024-02-09T17:43:12.016 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.194366803, 2024-02-09T17:43:12.017 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:12.017 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:12.017 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:12.017 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:43:12.017 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:43:12.017 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 207627 2024-02-09T17:43:12.017 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.017 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:12.017 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.017 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:12.017 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.800287+0000", 2024-02-09T17:43:12.017 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:12.017 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.017 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.017 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:12.018 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.800287+0000", 2024-02-09T17:43:12.018 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:12.018 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.018 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.018 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:12.018 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.800288+0000", 2024-02-09T17:43:12.018 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.669e-06 2024-02-09T17:43:12.018 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.018 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.018 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:12.018 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.800293+0000", 2024-02-09T17:43:12.018 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1950000000000002e-06 2024-02-09T17:43:12.018 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.019 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.019 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:12.019 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.800304+0000", 2024-02-09T17:43:12.019 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0545e-05 2024-02-09T17:43:12.019 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.019 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.019 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:12.019 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.800312+0000", 2024-02-09T17:43:12.019 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8080000000000005e-06 2024-02-09T17:43:12.019 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.019 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.019 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:12.019 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.800371+0000", 2024-02-09T17:43:12.019 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9151000000000003e-05 2024-02-09T17:43:12.020 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.020 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.020 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:12.020 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.800473+0000", 2024-02-09T17:43:12.020 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000102465 2024-02-09T17:43:12.020 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.020 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.020 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,1,7", 2024-02-09T17:43:12.020 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.800564+0000", 2024-02-09T17:43:12.020 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.0450000000000003e-05 2024-02-09T17:43:12.020 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.020 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.020 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:12.021 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.804199+0000", 2024-02-09T17:43:12.021 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.003635283 2024-02-09T17:43:12.021 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.021 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.021 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:12.021 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.813550+0000", 2024-02-09T17:43:12.021 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0093505749999999999 2024-02-09T17:43:12.021 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.021 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.021 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:12.021 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.817741+0000", 2024-02-09T17:43:12.021 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0041907740000000004 2024-02-09T17:43:12.021 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.021 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.021 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:12.022 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.994498+0000", 2024-02-09T17:43:12.022 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.17675744199999999 2024-02-09T17:43:12.022 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.022 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.022 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:12.022 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.994551+0000", 2024-02-09T17:43:12.022 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3470999999999997e-05 2024-02-09T17:43:12.022 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.022 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.022 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:12.022 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.994653+0000", 2024-02-09T17:43:12.022 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000101975 2024-02-09T17:43:12.022 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:12.022 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:12.022 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:12.023 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.023 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.023 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:207664 4.7 4:e9bc5ca9:::benchmark_data_smithi179_103809_object70198:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-02-09T17:43:12.023 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:32.854556+0000", 2024-02-09T17:43:12.023 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 279.11088398300001, 2024-02-09T17:43:12.023 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.140295527, 2024-02-09T17:43:12.023 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:12.023 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:12.023 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:12.023 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:43:12.023 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:43:12.023 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 207664 2024-02-09T17:43:12.023 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.024 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:12.024 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.024 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:12.024 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.854556+0000", 2024-02-09T17:43:12.024 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:12.024 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.024 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.024 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:12.024 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.854556+0000", 2024-02-09T17:43:12.024 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:12.024 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.024 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.024 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:12.024 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.854558+0000", 2024-02-09T17:43:12.025 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.46e-06 2024-02-09T17:43:12.025 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.025 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.025 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:12.025 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.854564+0000", 2024-02-09T17:43:12.025 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8300000000000001e-06 2024-02-09T17:43:12.025 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.025 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.025 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:12.025 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.854576+0000", 2024-02-09T17:43:12.025 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2381000000000001e-05 2024-02-09T17:43:12.025 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.025 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.025 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:12.026 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.854585+0000", 2024-02-09T17:43:12.026 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7339999999999997e-06 2024-02-09T17:43:12.026 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.026 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.026 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:12.026 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.854653+0000", 2024-02-09T17:43:12.026 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8150000000000003e-05 2024-02-09T17:43:12.026 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.026 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.026 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:12.026 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.854778+0000", 2024-02-09T17:43:12.026 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012533999999999999 2024-02-09T17:43:12.026 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.026 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.026 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,1,7", 2024-02-09T17:43:12.027 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.854872+0000", 2024-02-09T17:43:12.027 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.4075999999999998e-05 2024-02-09T17:43:12.027 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.027 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.027 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:12.027 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.862265+0000", 2024-02-09T17:43:12.027 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0073921489999999998 2024-02-09T17:43:12.027 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.027 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.027 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:12.027 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.865744+0000", 2024-02-09T17:43:12.027 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0034792600000000001 2024-02-09T17:43:12.028 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.028 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.028 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:12.028 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.866870+0000", 2024-02-09T17:43:12.028 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0011265450000000001 2024-02-09T17:43:12.028 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.028 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.028 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:12.028 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.994692+0000", 2024-02-09T17:43:12.029 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.12782187 2024-02-09T17:43:12.029 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.029 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.029 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:12.029 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.994744+0000", 2024-02-09T17:43:12.029 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1499000000000003e-05 2024-02-09T17:43:12.029 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.029 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.029 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:12.030 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.994852+0000", 2024-02-09T17:43:12.030 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010823300000000001 2024-02-09T17:43:12.030 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:12.030 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:12.030 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:12.030 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.030 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.030 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:207665 4.7 4:ea24a272:::benchmark_data_smithi179_103809_object70199:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-02-09T17:43:12.031 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:32.857672+0000", 2024-02-09T17:43:12.031 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 279.107768061, 2024-02-09T17:43:12.031 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.13739986400000001, 2024-02-09T17:43:12.031 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:12.031 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:12.031 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:12.031 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:43:12.031 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:43:12.031 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 207665 2024-02-09T17:43:12.032 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.032 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:12.032 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.032 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:12.032 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.857672+0000", 2024-02-09T17:43:12.032 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:12.032 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.032 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.032 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:12.032 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.857672+0000", 2024-02-09T17:43:12.032 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:12.032 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.032 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.032 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:12.033 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.857674+0000", 2024-02-09T17:43:12.033 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.429e-06 2024-02-09T17:43:12.033 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.033 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.033 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:12.033 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.857680+0000", 2024-02-09T17:43:12.033 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0260000000000004e-06 2024-02-09T17:43:12.033 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.033 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.033 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:12.033 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.857694+0000", 2024-02-09T17:43:12.033 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3935e-05 2024-02-09T17:43:12.033 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.034 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.034 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:12.034 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.857703+0000", 2024-02-09T17:43:12.034 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8349999999999993e-06 2024-02-09T17:43:12.034 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.034 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.034 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:12.034 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.858574+0000", 2024-02-09T17:43:12.034 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00087124800000000003 2024-02-09T17:43:12.034 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.034 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.034 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:12.034 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.858671+0000", 2024-02-09T17:43:12.034 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7087000000000001e-05 2024-02-09T17:43:12.035 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.035 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.035 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,1,7", 2024-02-09T17:43:12.035 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.858750+0000", 2024-02-09T17:43:12.035 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8802999999999998e-05 2024-02-09T17:43:12.035 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.035 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.035 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:12.035 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.865766+0000", 2024-02-09T17:43:12.035 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0070160489999999999 2024-02-09T17:43:12.035 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.035 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.035 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:12.035 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.865790+0000", 2024-02-09T17:43:12.036 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4414999999999999e-05 2024-02-09T17:43:12.036 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.036 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.036 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:12.036 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.866908+0000", 2024-02-09T17:43:12.036 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0011182259999999999 2024-02-09T17:43:12.036 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.036 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.036 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:12.036 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.994902+0000", 2024-02-09T17:43:12.036 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.12799374499999999 2024-02-09T17:43:12.036 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.036 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.036 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:12.036 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.994963+0000", 2024-02-09T17:43:12.037 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0674999999999998e-05 2024-02-09T17:43:12.037 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.037 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.037 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:12.037 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:32.995072+0000", 2024-02-09T17:43:12.037 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000109391 2024-02-09T17:43:12.037 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:12.037 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:12.037 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:12.037 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.037 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.037 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:208399 4.5 4:a2a3b00a:::benchmark_data_smithi179_103809_object70933:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e379)", 2024-02-09T17:43:12.037 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:33.814732+0000", 2024-02-09T17:43:12.037 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 278.150708302, 2024-02-09T17:43:12.037 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2041751440000001, 2024-02-09T17:43:12.038 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:12.038 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-02-09T17:43:12.038 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:12.038 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:43:12.038 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:43:12.038 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 208399 2024-02-09T17:43:12.038 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.038 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:12.038 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.038 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:12.038 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.814732+0000", 2024-02-09T17:43:12.038 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:12.038 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.038 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.038 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:12.039 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.814732+0000", 2024-02-09T17:43:12.039 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:12.039 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.039 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.039 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:12.039 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.814734+0000", 2024-02-09T17:43:12.039 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4720000000000001e-06 2024-02-09T17:43:12.039 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.039 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.039 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:12.039 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.814739+0000", 2024-02-09T17:43:12.039 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3929999999999999e-06 2024-02-09T17:43:12.039 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.039 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.040 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:12.040 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.814755+0000", 2024-02-09T17:43:12.040 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5719e-05 2024-02-09T17:43:12.040 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.040 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.040 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:12.040 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.814764+0000", 2024-02-09T17:43:12.040 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.3060000000000003e-06 2024-02-09T17:43:12.040 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.040 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.040 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:12.040 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.814834+0000", 2024-02-09T17:43:12.040 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9986000000000004e-05 2024-02-09T17:43:12.040 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.041 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.041 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op must wait for map", 2024-02-09T17:43:12.041 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.814841+0000", 2024-02-09T17:43:12.041 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7259999999999996e-06 2024-02-09T17:43:12.041 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.041 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.041 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:12.041 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.819052+0000", 2024-02-09T17:43:12.041 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0042111850000000001 2024-02-09T17:43:12.041 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.041 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.041 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:43:12.041 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.819059+0000", 2024-02-09T17:43:12.041 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.3839999999999999e-06 2024-02-09T17:43:12.042 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.042 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.042 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:12.042 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:34.811056+0000", 2024-02-09T17:43:12.042 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.99199702899999997 2024-02-09T17:43:12.042 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.042 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.042 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:43:12.042 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:34.811139+0000", 2024-02-09T17:43:12.042 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2931999999999998e-05 2024-02-09T17:43:12.042 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.042 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.042 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:12.042 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.015456+0000", 2024-02-09T17:43:12.043 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.204317029 2024-02-09T17:43:12.043 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.043 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.043 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:12.043 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.018907+0000", 2024-02-09T17:43:12.043 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0034509829999999999 2024-02-09T17:43:12.043 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:12.043 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:12.043 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:12.043 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.043 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.043 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:208418 4.5 4:a02ae9eb:::benchmark_data_smithi179_103809_object70952:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-02-09T17:43:12.044 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:33.835944+0000", 2024-02-09T17:43:12.044 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 278.12949638399999, 2024-02-09T17:43:12.044 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.197308585, 2024-02-09T17:43:12.044 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:12.044 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:12.044 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:12.044 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:43:12.044 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:43:12.044 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 208418 2024-02-09T17:43:12.044 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.044 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:12.044 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.044 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:12.045 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.835944+0000", 2024-02-09T17:43:12.045 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:12.045 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.045 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.045 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:12.045 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.835944+0000", 2024-02-09T17:43:12.045 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:12.045 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.045 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.045 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:12.045 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.835945+0000", 2024-02-09T17:43:12.045 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1650000000000001e-06 2024-02-09T17:43:12.045 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.045 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.046 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:12.046 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.835949+0000", 2024-02-09T17:43:12.046 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4410000000000002e-06 2024-02-09T17:43:12.046 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.046 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.046 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:12.046 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.835958+0000", 2024-02-09T17:43:12.046 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2380000000000003e-06 2024-02-09T17:43:12.046 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.046 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.046 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:12.046 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.835963+0000", 2024-02-09T17:43:12.046 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0350000000000001e-06 2024-02-09T17:43:12.047 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.047 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.047 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:12.047 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.836006+0000", 2024-02-09T17:43:12.047 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3275000000000002e-05 2024-02-09T17:43:12.047 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.047 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.047 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:43:12.047 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.836014+0000", 2024-02-09T17:43:12.047 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1580000000000005e-06 2024-02-09T17:43:12.047 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.047 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.048 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:12.048 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:34.811212+0000", 2024-02-09T17:43:12.048 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.97519715799999995 2024-02-09T17:43:12.048 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.048 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.048 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:43:12.048 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:34.811222+0000", 2024-02-09T17:43:12.048 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0030000000000001e-05 2024-02-09T17:43:12.048 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.048 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.048 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:12.048 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.019231+0000", 2024-02-09T17:43:12.049 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2080094990000001 2024-02-09T17:43:12.049 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.049 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.049 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:12.049 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.020160+0000", 2024-02-09T17:43:12.049 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00092906900000000003 2024-02-09T17:43:12.049 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.049 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.049 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,6,7,11", 2024-02-09T17:43:12.049 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.020254+0000", 2024-02-09T17:43:12.049 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.3797999999999999e-05 2024-02-09T17:43:12.049 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.049 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.049 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:12.049 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.024546+0000", 2024-02-09T17:43:12.050 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0042921720000000003 2024-02-09T17:43:12.050 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.050 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.050 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:12.050 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.026292+0000", 2024-02-09T17:43:12.050 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0017459909999999999 2024-02-09T17:43:12.050 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.050 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.050 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:12.050 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.027748+0000", 2024-02-09T17:43:12.050 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001456176 2024-02-09T17:43:12.050 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.050 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.050 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:12.051 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.029364+0000", 2024-02-09T17:43:12.051 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0016155080000000001 2024-02-09T17:43:12.051 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.051 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.051 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:12.051 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.033044+0000", 2024-02-09T17:43:12.051 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0036805330000000002 2024-02-09T17:43:12.051 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.051 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.051 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:12.051 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.033116+0000", 2024-02-09T17:43:12.051 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1848999999999996e-05 2024-02-09T17:43:12.051 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.051 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.051 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:12.052 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.033253+0000", 2024-02-09T17:43:12.052 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013648999999999999 2024-02-09T17:43:12.052 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:12.052 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:12.052 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:12.052 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.052 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.052 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:232581 4.5 4:a36c0d16:::benchmark_data_smithi179_103809_object95115:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-02-09T17:43:12.052 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:39:24.589203+0000", 2024-02-09T17:43:12.052 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 227.376237167, 2024-02-09T17:43:12.052 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0160283800000001, 2024-02-09T17:43:12.052 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:12.052 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:12.052 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:12.052 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:43:12.053 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:43:12.053 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 232581 2024-02-09T17:43:12.053 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.053 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:12.053 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.053 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:12.053 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:24.589203+0000", 2024-02-09T17:43:12.053 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:12.053 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.053 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.053 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:12.053 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:24.589203+0000", 2024-02-09T17:43:12.053 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:12.053 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.054 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.054 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:12.054 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:24.589205+0000", 2024-02-09T17:43:12.054 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.756e-06 2024-02-09T17:43:12.054 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.054 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.054 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:12.055 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:24.589213+0000", 2024-02-09T17:43:12.055 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1939999999999998e-06 2024-02-09T17:43:12.055 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.055 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.055 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:12.055 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:24.589232+0000", 2024-02-09T17:43:12.055 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.912e-05 2024-02-09T17:43:12.056 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.056 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.056 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:12.056 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:24.589242+0000", 2024-02-09T17:43:12.056 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.3700000000000001e-06 2024-02-09T17:43:12.056 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.056 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.057 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:12.057 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:24.589307+0000", 2024-02-09T17:43:12.057 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5356000000000002e-05 2024-02-09T17:43:12.057 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.057 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.057 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:43:12.057 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:24.589316+0000", 2024-02-09T17:43:12.057 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.3260000000000002e-06 2024-02-09T17:43:12.057 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.058 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.058 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:12.058 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:25.599093+0000", 2024-02-09T17:43:12.058 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.009776778 2024-02-09T17:43:12.058 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.058 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.058 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:12.058 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:25.599260+0000", 2024-02-09T17:43:12.059 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016675300000000001 2024-02-09T17:43:12.059 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.059 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.059 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,7", 2024-02-09T17:43:12.059 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:25.599410+0000", 2024-02-09T17:43:12.059 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015003099999999999 2024-02-09T17:43:12.059 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.060 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.060 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:12.060 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:25.605045+0000", 2024-02-09T17:43:12.060 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0056348650000000002 2024-02-09T17:43:12.060 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.060 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.060 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:12.060 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:25.605094+0000", 2024-02-09T17:43:12.061 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9413000000000003e-05 2024-02-09T17:43:12.061 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.061 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.061 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:12.061 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:25.605116+0000", 2024-02-09T17:43:12.061 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2272999999999998e-05 2024-02-09T17:43:12.061 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.061 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.061 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:12.061 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:25.605186+0000", 2024-02-09T17:43:12.061 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9356999999999993e-05 2024-02-09T17:43:12.061 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.061 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.062 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:12.062 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:25.605232+0000", 2024-02-09T17:43:12.062 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5788000000000003e-05 2024-02-09T17:43:12.062 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:12.062 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:12.062 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:12.062 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.062 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.062 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:74597 5.4 5:23ce9f21:::benchmark_data_smithi179_115199_object74596:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e489)", 2024-02-09T17:43:12.062 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:41:45.028674+0000", 2024-02-09T17:43:12.062 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 86.936766821000006, 2024-02-09T17:43:12.062 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.206532944, 2024-02-09T17:43:12.062 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:12.062 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:12.062 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:12.063 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:43:12.063 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:43:12.063 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 74597 2024-02-09T17:43:12.063 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.063 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:12.063 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.063 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:12.063 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.028674+0000", 2024-02-09T17:43:12.063 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:12.063 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.063 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.063 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:12.063 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.028674+0000", 2024-02-09T17:43:12.063 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:12.064 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.064 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.064 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:12.064 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.028675+0000", 2024-02-09T17:43:12.064 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.75e-06 2024-02-09T17:43:12.064 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.064 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.064 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:12.064 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.028694+0000", 2024-02-09T17:43:12.064 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8532000000000001e-05 2024-02-09T17:43:12.064 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.064 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.064 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:12.064 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.028705+0000", 2024-02-09T17:43:12.064 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0898e-05 2024-02-09T17:43:12.065 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.065 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.065 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:12.065 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.028712+0000", 2024-02-09T17:43:12.065 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6760000000000007e-06 2024-02-09T17:43:12.065 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.065 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.065 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:12.065 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.028757+0000", 2024-02-09T17:43:12.065 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4659000000000001e-05 2024-02-09T17:43:12.065 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.065 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.065 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:12.065 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.028877+0000", 2024-02-09T17:43:12.065 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000119491 2024-02-09T17:43:12.066 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.066 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.066 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,8", 2024-02-09T17:43:12.066 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.028976+0000", 2024-02-09T17:43:12.066 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.9679000000000006e-05 2024-02-09T17:43:12.066 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.066 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.066 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:12.066 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.032288+0000", 2024-02-09T17:43:12.066 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.003311839 2024-02-09T17:43:12.066 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.066 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.066 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:12.067 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.033273+0000", 2024-02-09T17:43:12.067 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00098491599999999996 2024-02-09T17:43:12.067 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.067 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.067 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:12.067 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.235125+0000", 2024-02-09T17:43:12.067 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.201852158 2024-02-09T17:43:12.067 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.067 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.067 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:12.067 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.235178+0000", 2024-02-09T17:43:12.067 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2716000000000001e-05 2024-02-09T17:43:12.067 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.067 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.067 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:12.068 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.235207+0000", 2024-02-09T17:43:12.068 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.8629999999999999e-05 2024-02-09T17:43:12.068 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:12.068 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:12.068 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:12.068 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.068 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.068 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:74619 5.4 5:2e16100e:::benchmark_data_smithi179_115199_object74618:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e489)", 2024-02-09T17:43:12.068 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:41:45.041006+0000", 2024-02-09T17:43:12.068 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 86.924434372999997, 2024-02-09T17:43:12.068 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.19672163100000001, 2024-02-09T17:43:12.068 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:12.068 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:12.069 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:12.069 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:43:12.069 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:43:12.069 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 74619 2024-02-09T17:43:12.069 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.069 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:12.070 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.070 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:12.070 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.041006+0000", 2024-02-09T17:43:12.070 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:12.070 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.071 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.071 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:12.071 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.041006+0000", 2024-02-09T17:43:12.071 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:12.071 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.072 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.072 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:12.072 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.041008+0000", 2024-02-09T17:43:12.072 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.604e-06 2024-02-09T17:43:12.073 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.073 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.073 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:12.073 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.041031+0000", 2024-02-09T17:43:12.073 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3536e-05 2024-02-09T17:43:12.073 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.074 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.074 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:12.074 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.041047+0000", 2024-02-09T17:43:12.074 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5597999999999999e-05 2024-02-09T17:43:12.074 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.074 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.075 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:12.075 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.041055+0000", 2024-02-09T17:43:12.075 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1659999999999994e-06 2024-02-09T17:43:12.075 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.075 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.075 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:12.076 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.041103+0000", 2024-02-09T17:43:12.076 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7896999999999999e-05 2024-02-09T17:43:12.076 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.076 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.077 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:12.077 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.041223+0000", 2024-02-09T17:43:12.077 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012044699999999999 2024-02-09T17:43:12.077 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.077 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.077 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,8", 2024-02-09T17:43:12.077 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.041313+0000", 2024-02-09T17:43:12.078 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.9338000000000004e-05 2024-02-09T17:43:12.078 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.078 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.078 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:12.078 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.046007+0000", 2024-02-09T17:43:12.078 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0046941930000000002 2024-02-09T17:43:12.078 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.079 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.079 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:12.080 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.048424+0000", 2024-02-09T17:43:12.080 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0024172460000000001 2024-02-09T17:43:12.081 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.081 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.082 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:12.082 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.237654+0000", 2024-02-09T17:43:12.083 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.189229971 2024-02-09T17:43:12.083 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.083 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.084 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:12.084 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.237700+0000", 2024-02-09T17:43:12.084 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6356999999999997e-05 2024-02-09T17:43:12.084 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.084 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.084 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:12.084 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.237728+0000", 2024-02-09T17:43:12.085 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7277999999999999e-05 2024-02-09T17:43:12.085 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:12.085 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:12.085 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:12.085 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.085 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.086 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:74622 5.4 5:2e293021:::benchmark_data_smithi179_115199_object74621:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e489)", 2024-02-09T17:43:12.086 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:41:45.041876+0000", 2024-02-09T17:43:12.086 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 86.923564820999999, 2024-02-09T17:43:12.086 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.195954448, 2024-02-09T17:43:12.086 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:12.086 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:12.087 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:12.087 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:43:12.087 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:43:12.087 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 74622 2024-02-09T17:43:12.087 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.087 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:12.087 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.087 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:12.087 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.041876+0000", 2024-02-09T17:43:12.087 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:12.088 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.088 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.088 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:12.088 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.041876+0000", 2024-02-09T17:43:12.088 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:12.088 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.088 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.088 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:12.088 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.041877+0000", 2024-02-09T17:43:12.088 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.4200000000000004e-07 2024-02-09T17:43:12.088 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.088 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.088 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:12.088 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.041893+0000", 2024-02-09T17:43:12.089 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6623e-05 2024-02-09T17:43:12.089 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.089 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.089 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:12.089 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.041905+0000", 2024-02-09T17:43:12.089 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1997999999999999e-05 2024-02-09T17:43:12.089 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.089 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.089 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:12.089 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.041913+0000", 2024-02-09T17:43:12.089 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6070000000000002e-06 2024-02-09T17:43:12.089 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.089 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.090 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:12.090 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.043388+0000", 2024-02-09T17:43:12.090 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001474995 2024-02-09T17:43:12.090 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.090 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.090 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:12.090 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.043511+0000", 2024-02-09T17:43:12.090 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000123428 2024-02-09T17:43:12.090 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.090 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.090 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,8", 2024-02-09T17:43:12.090 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.043653+0000", 2024-02-09T17:43:12.090 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014197600000000001 2024-02-09T17:43:12.090 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.090 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.091 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:12.091 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.048092+0000", 2024-02-09T17:43:12.091 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0044388479999999996 2024-02-09T17:43:12.091 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.091 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.091 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:12.091 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.048532+0000", 2024-02-09T17:43:12.091 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00044015299999999999 2024-02-09T17:43:12.091 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.091 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.091 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:12.091 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.237765+0000", 2024-02-09T17:43:12.091 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.189232811 2024-02-09T17:43:12.091 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.092 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.092 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:12.092 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.237807+0000", 2024-02-09T17:43:12.092 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1699999999999997e-05 2024-02-09T17:43:12.092 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.092 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.092 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:12.092 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.237830+0000", 2024-02-09T17:43:12.092 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3366999999999998e-05 2024-02-09T17:43:12.092 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:12.092 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:12.092 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:12.092 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.092 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.093 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:74689 5.4 5:27431827:::benchmark_data_smithi179_115199_object74688:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e489)", 2024-02-09T17:43:12.093 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:41:45.097968+0000", 2024-02-09T17:43:12.093 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 86.867472738000004, 2024-02-09T17:43:12.093 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.13998519500000001, 2024-02-09T17:43:12.093 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:12.093 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:12.093 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:12.093 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:43:12.093 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:43:12.093 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 74689 2024-02-09T17:43:12.093 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.093 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:12.093 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.094 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:12.094 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.097968+0000", 2024-02-09T17:43:12.094 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:12.094 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.094 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.094 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:12.094 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.097968+0000", 2024-02-09T17:43:12.094 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:12.094 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.094 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.094 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:12.094 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.097969+0000", 2024-02-09T17:43:12.094 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.663e-06 2024-02-09T17:43:12.094 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.095 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.095 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:12.095 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.097987+0000", 2024-02-09T17:43:12.095 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7507e-05 2024-02-09T17:43:12.095 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.095 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.095 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:12.095 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.097997+0000", 2024-02-09T17:43:12.095 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0233e-05 2024-02-09T17:43:12.095 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.095 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.095 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:12.095 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.098003+0000", 2024-02-09T17:43:12.095 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6010000000000002e-06 2024-02-09T17:43:12.096 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.096 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.096 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:12.096 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.098057+0000", 2024-02-09T17:43:12.096 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3814999999999999e-05 2024-02-09T17:43:12.096 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.096 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.096 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:12.096 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.098162+0000", 2024-02-09T17:43:12.096 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010594099999999999 2024-02-09T17:43:12.096 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.096 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.096 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,8", 2024-02-09T17:43:12.096 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.098289+0000", 2024-02-09T17:43:12.097 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000126828 2024-02-09T17:43:12.097 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.097 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.097 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:12.097 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.102038+0000", 2024-02-09T17:43:12.097 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0037482309999999999 2024-02-09T17:43:12.097 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.097 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.097 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:12.097 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.102676+0000", 2024-02-09T17:43:12.097 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00063885500000000002 2024-02-09T17:43:12.097 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.097 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.098 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:12.098 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.237878+0000", 2024-02-09T17:43:12.098 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.13520120499999999 2024-02-09T17:43:12.098 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.098 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.098 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:12.098 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.237929+0000", 2024-02-09T17:43:12.098 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1038999999999997e-05 2024-02-09T17:43:12.098 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.098 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.098 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:12.098 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.237953+0000", 2024-02-09T17:43:12.098 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4277000000000002e-05 2024-02-09T17:43:12.098 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:12.099 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:12.099 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:12.099 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.099 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.099 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:74690 5.4 5:21d6c807:::benchmark_data_smithi179_115199_object74689:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e489)", 2024-02-09T17:43:12.099 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:41:45.099590+0000", 2024-02-09T17:43:12.099 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 86.865849949999998, 2024-02-09T17:43:12.099 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.141647513, 2024-02-09T17:43:12.099 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:12.099 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:12.099 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:12.099 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:43:12.099 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:43:12.099 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 74690 2024-02-09T17:43:12.099 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.100 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:12.100 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.100 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:12.100 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.099590+0000", 2024-02-09T17:43:12.100 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:12.100 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.100 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.100 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:12.100 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.099590+0000", 2024-02-09T17:43:12.100 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:12.101 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.101 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.101 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:12.101 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.099592+0000", 2024-02-09T17:43:12.101 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.184e-06 2024-02-09T17:43:12.101 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.101 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.101 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:12.101 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.099613+0000", 2024-02-09T17:43:12.101 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1086000000000001e-05 2024-02-09T17:43:12.101 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.101 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.101 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:12.102 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.099628+0000", 2024-02-09T17:43:12.102 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4813e-05 2024-02-09T17:43:12.102 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.102 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.102 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:12.102 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.099635+0000", 2024-02-09T17:43:12.102 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5769999999999996e-06 2024-02-09T17:43:12.102 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.102 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.102 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:12.102 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.100175+0000", 2024-02-09T17:43:12.102 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00054019699999999996 2024-02-09T17:43:12.102 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.102 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.103 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:12.103 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.100310+0000", 2024-02-09T17:43:12.103 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013460800000000001 2024-02-09T17:43:12.103 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.103 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.103 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,8", 2024-02-09T17:43:12.103 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.100412+0000", 2024-02-09T17:43:12.103 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010189299999999999 2024-02-09T17:43:12.103 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.103 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.104 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:12.104 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.102178+0000", 2024-02-09T17:43:12.104 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0017661630000000001 2024-02-09T17:43:12.104 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.104 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.104 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:12.104 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.102830+0000", 2024-02-09T17:43:12.104 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00065249700000000004 2024-02-09T17:43:12.105 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.105 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.105 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:12.105 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.241143+0000", 2024-02-09T17:43:12.105 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.13831211500000001 2024-02-09T17:43:12.105 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.105 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.105 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:12.105 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.241198+0000", 2024-02-09T17:43:12.106 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5717000000000002e-05 2024-02-09T17:43:12.106 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.106 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.106 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:12.106 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:45.241238+0000", 2024-02-09T17:43:12.106 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9663000000000003e-05 2024-02-09T17:43:12.106 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:12.106 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:12.106 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:12.107 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.107 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.107 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:125163 5.0 5:04848e06:::benchmark_data_smithi179_115199_object125162:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e500)", 2024-02-09T17:43:12.107 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:42:29.994400+0000", 2024-02-09T17:43:12.107 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 41.971040250999998, 2024-02-09T17:43:12.107 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.21890696300000001, 2024-02-09T17:43:12.107 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:12.107 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:12.107 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:12.108 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:43:12.108 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:43:12.108 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 125163 2024-02-09T17:43:12.108 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.108 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:12.108 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.108 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:12.109 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:29.994400+0000", 2024-02-09T17:43:12.109 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:12.109 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.109 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.109 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:12.109 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:29.994400+0000", 2024-02-09T17:43:12.109 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:12.109 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.109 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.109 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:12.109 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:29.994401+0000", 2024-02-09T17:43:12.109 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.159e-06 2024-02-09T17:43:12.109 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.110 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.110 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:12.110 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:29.994436+0000", 2024-02-09T17:43:12.110 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4396000000000001e-05 2024-02-09T17:43:12.110 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.110 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.110 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:12.110 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:29.994458+0000", 2024-02-09T17:43:12.110 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1814000000000002e-05 2024-02-09T17:43:12.110 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.110 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.110 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:12.110 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:29.994483+0000", 2024-02-09T17:43:12.110 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5080999999999999e-05 2024-02-09T17:43:12.111 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.111 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.111 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:12.111 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.004432+0000", 2024-02-09T17:43:12.111 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0099490140000000008 2024-02-09T17:43:12.111 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.111 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.111 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:12.111 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.004537+0000", 2024-02-09T17:43:12.111 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010492699999999999 2024-02-09T17:43:12.111 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.111 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.111 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:43:12.111 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.004634+0000", 2024-02-09T17:43:12.112 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.747e-05 2024-02-09T17:43:12.112 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.112 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.112 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:12.112 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.008771+0000", 2024-02-09T17:43:12.112 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0041364729999999999 2024-02-09T17:43:12.112 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.112 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.112 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:12.112 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.014237+0000", 2024-02-09T17:43:12.112 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0054665019999999998 2024-02-09T17:43:12.112 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.112 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.112 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:12.112 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.213226+0000", 2024-02-09T17:43:12.113 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.19898876600000001 2024-02-09T17:43:12.113 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.113 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.113 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:12.113 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.213265+0000", 2024-02-09T17:43:12.113 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9706999999999998e-05 2024-02-09T17:43:12.113 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.113 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.113 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:12.113 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.213307+0000", 2024-02-09T17:43:12.113 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1653999999999998e-05 2024-02-09T17:43:12.113 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:12.113 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:12.113 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:12.113 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.114 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.114 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:125181 5.0 5:08b0c8c3:::benchmark_data_smithi179_115199_object125180:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e500)", 2024-02-09T17:43:12.114 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:42:30.010351+0000", 2024-02-09T17:43:12.114 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 41.955089563000001, 2024-02-09T17:43:12.114 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.22445062399999999, 2024-02-09T17:43:12.114 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:12.114 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:12.114 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:12.114 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:43:12.114 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:43:12.114 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 125181 2024-02-09T17:43:12.114 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.114 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:12.114 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.115 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:12.115 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.010351+0000", 2024-02-09T17:43:12.115 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:12.115 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.115 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.115 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:12.115 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.010351+0000", 2024-02-09T17:43:12.115 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:12.115 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.115 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.115 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:12.115 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.010353+0000", 2024-02-09T17:43:12.115 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.815e-06 2024-02-09T17:43:12.115 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.115 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.116 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:12.116 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.010400+0000", 2024-02-09T17:43:12.116 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7614000000000002e-05 2024-02-09T17:43:12.116 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.116 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.116 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:12.116 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.010413+0000", 2024-02-09T17:43:12.116 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2792999999999999e-05 2024-02-09T17:43:12.116 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.116 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.116 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:12.116 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.010422+0000", 2024-02-09T17:43:12.116 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4200000000000007e-06 2024-02-09T17:43:12.117 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.117 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.117 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:12.117 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.010475+0000", 2024-02-09T17:43:12.117 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3338000000000003e-05 2024-02-09T17:43:12.117 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.117 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.117 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:12.117 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.010623+0000", 2024-02-09T17:43:12.117 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014819900000000001 2024-02-09T17:43:12.117 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.117 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.117 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:43:12.117 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.010794+0000", 2024-02-09T17:43:12.117 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00017131700000000001 2024-02-09T17:43:12.117 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.118 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.118 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:12.118 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.014254+0000", 2024-02-09T17:43:12.118 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0034598329999999998 2024-02-09T17:43:12.118 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.118 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.118 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:12.118 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.016837+0000", 2024-02-09T17:43:12.118 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0025832559999999999 2024-02-09T17:43:12.118 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.118 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.118 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:12.118 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.234695+0000", 2024-02-09T17:43:12.119 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.21785784599999999 2024-02-09T17:43:12.119 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.119 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.119 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:12.119 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.234738+0000", 2024-02-09T17:43:12.119 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3186999999999998e-05 2024-02-09T17:43:12.119 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.119 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.119 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:12.119 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.234801+0000", 2024-02-09T17:43:12.119 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3006e-05 2024-02-09T17:43:12.119 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:12.119 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:12.119 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:12.119 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.120 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.120 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:125249 5.0 5:07d49e8d:::benchmark_data_smithi179_115199_object125248:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e500)", 2024-02-09T17:43:12.120 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:42:30.085296+0000", 2024-02-09T17:43:12.120 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 41.880144895000001, 2024-02-09T17:43:12.120 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14967023099999999, 2024-02-09T17:43:12.120 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:12.120 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:12.120 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:12.120 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:43:12.120 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:43:12.120 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 125249 2024-02-09T17:43:12.120 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.120 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:12.121 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.121 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:12.121 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.085296+0000", 2024-02-09T17:43:12.121 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:12.121 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.121 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.121 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:12.121 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.085296+0000", 2024-02-09T17:43:12.121 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:12.121 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.121 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.121 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:12.121 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.085296+0000", 2024-02-09T17:43:12.121 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6700000000000002e-07 2024-02-09T17:43:12.121 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.122 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.122 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:12.122 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.085315+0000", 2024-02-09T17:43:12.122 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8185e-05 2024-02-09T17:43:12.122 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.122 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.122 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:12.122 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.085323+0000", 2024-02-09T17:43:12.122 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2600000000000005e-06 2024-02-09T17:43:12.122 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.122 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.122 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:12.122 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.085328+0000", 2024-02-09T17:43:12.122 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.169e-06 2024-02-09T17:43:12.122 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.123 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.123 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:12.123 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.085379+0000", 2024-02-09T17:43:12.123 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0868999999999997e-05 2024-02-09T17:43:12.123 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.123 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.123 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:12.123 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.085506+0000", 2024-02-09T17:43:12.123 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012717100000000001 2024-02-09T17:43:12.123 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.123 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.123 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:43:12.123 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.085641+0000", 2024-02-09T17:43:12.123 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013485099999999999 2024-02-09T17:43:12.124 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.124 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.124 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:12.124 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.092259+0000", 2024-02-09T17:43:12.124 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.006618428 2024-02-09T17:43:12.124 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.124 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.124 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:12.124 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.092280+0000", 2024-02-09T17:43:12.124 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1033999999999999e-05 2024-02-09T17:43:12.124 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.124 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.124 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:12.124 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.234855+0000", 2024-02-09T17:43:12.124 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.142574903 2024-02-09T17:43:12.125 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.125 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.125 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:12.125 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.234915+0000", 2024-02-09T17:43:12.125 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9160000000000003e-05 2024-02-09T17:43:12.125 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:12.125 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:12.125 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:12.125 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.234966+0000", 2024-02-09T17:43:12.125 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1233999999999998e-05 2024-02-09T17:43:12.125 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:12.125 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:12.125 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:12.125 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:12.125 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:12.126 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:43:12.126 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:43:12.127 INFO:teuthology.orchestra.run.smithi099.stderr:set pool 1 pg_num to 31 2024-02-09T17:43:12.147 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set noscrub 2024-02-09T17:43:12.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:11 smithi179 ceph-mon[43975]: 5.9 scrub ok 2024-02-09T17:43:12.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:11 smithi179 ceph-mon[43975]: pgmap v1305: 37 pgs: 37 active+clean; 6.0 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 877 op/s 2024-02-09T17:43:12.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:11 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/17207205' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-02-09T17:43:12.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:11 smithi179 ceph-mon[43975]: 1.6 deep-scrub starts 2024-02-09T17:43:12.185 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:12.225 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:43:12.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:12.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:43:12.332 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:43:12.364 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:11 smithi099 ceph-mon[39420]: 5.9 scrub ok 2024-02-09T17:43:12.368 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:11 smithi099 ceph-mon[39420]: pgmap v1305: 37 pgs: 37 active+clean; 6.0 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 877 op/s 2024-02-09T17:43:12.369 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:11 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/17207205' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-02-09T17:43:12.369 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:11 smithi099 ceph-mon[39420]: 1.6 deep-scrub starts 2024-02-09T17:43:12.370 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:12.408 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:43:12.413 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:12.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:43:12.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:12 smithi162 ceph-mon[44304]: 5.9 scrub ok 2024-02-09T17:43:12.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:12 smithi162 ceph-mon[44304]: pgmap v1305: 37 pgs: 37 active+clean; 6.0 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 877 op/s 2024-02-09T17:43:12.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:12 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/17207205' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-02-09T17:43:12.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:12 smithi162 ceph-mon[44304]: 1.6 deep-scrub starts 2024-02-09T17:43:12.515 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:43:12.546 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:13.202 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:43:13.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:13.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:43:13.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:13 smithi099 ceph-mon[39420]: 1.6 deep-scrub ok 2024-02-09T17:43:13.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:13 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/17207205' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-02-09T17:43:13.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:13 smithi099 ceph-mon[39420]: osdmap e513: 12 total, 12 up, 11 in 2024-02-09T17:43:13.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:13 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-09T17:43:13.307 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:43:13.339 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:13.362 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:43:13.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:13.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:43:13.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:13 smithi179 ceph-mon[43975]: 1.6 deep-scrub ok 2024-02-09T17:43:13.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:13 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/17207205' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-02-09T17:43:13.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:13 smithi179 ceph-mon[43975]: osdmap e513: 12 total, 12 up, 11 in 2024-02-09T17:43:13.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:13 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-09T17:43:13.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:13 smithi162 ceph-mon[44304]: 1.6 deep-scrub ok 2024-02-09T17:43:13.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:13 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/17207205' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-02-09T17:43:13.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:13 smithi162 ceph-mon[44304]: osdmap e513: 12 total, 12 up, 11 in 2024-02-09T17:43:13.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:13 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-09T17:43:13.470 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:43:13.476 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:43:13.507 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:13.523 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:43:13.540 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:43:13.547 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:43:13.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:13.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:43:13.569 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:43:13.591 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:43:13.628 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:43:13.651 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:43:13.661 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:43:13.679 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:13.761 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:43:13.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:13.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:43:13.835 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.11 dump_ops_in_flight 2024-02-09T17:43:13.864 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:43:13.918 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:43:13.923 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:13.947 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:43:13.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:13.953 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:43:13.956 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:43:13.985 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:43:14.013 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:43:14.049 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:43:14.053 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:43:14.085 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:43:14.086 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:43:14.086 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:43:14.086 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:43:14.086 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:43:14.086 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:43:14.086 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:43:14.086 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:43:14.086 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:43:14.087 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:43:14.087 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:43:14.087 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:43:14.087 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:43:14.109 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:14 smithi179 ceph-mon[43975]: pgmap v1307: 37 pgs: 37 active+clean; 5.9 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 762 op/s 2024-02-09T17:43:14.109 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:14 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-02-09T17:43:14.109 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:14 smithi179 ceph-mon[43975]: osdmap e514: 12 total, 12 up, 11 in 2024-02-09T17:43:14.109 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:14 smithi179 ceph-mon[43975]: 5.3 deep-scrub starts 2024-02-09T17:43:14.109 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:14 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1347715390' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:43:14.109 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:14 smithi179 ceph-mon[43975]: osdmap e515: 12 total, 12 up, 11 in 2024-02-09T17:43:14.115 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:14.141 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:43:14.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:14.145 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:43:14.246 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:43:14.263 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:43:14.295 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:14.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:14 smithi162 ceph-mon[44304]: pgmap v1307: 37 pgs: 37 active+clean; 5.9 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 762 op/s 2024-02-09T17:43:14.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:14 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-02-09T17:43:14.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:14 smithi162 ceph-mon[44304]: osdmap e514: 12 total, 12 up, 11 in 2024-02-09T17:43:14.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:14 smithi162 ceph-mon[44304]: 5.3 deep-scrub starts 2024-02-09T17:43:14.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:14 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1347715390' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:43:14.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:14 smithi162 ceph-mon[44304]: osdmap e515: 12 total, 12 up, 11 in 2024-02-09T17:43:14.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:14 smithi099 ceph-mon[39420]: pgmap v1307: 37 pgs: 37 active+clean; 5.9 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 762 op/s 2024-02-09T17:43:14.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:14 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-02-09T17:43:14.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:14 smithi099 ceph-mon[39420]: osdmap e514: 12 total, 12 up, 11 in 2024-02-09T17:43:14.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:14 smithi099 ceph-mon[39420]: 5.3 deep-scrub starts 2024-02-09T17:43:14.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:14 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1347715390' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:43:14.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:14 smithi099 ceph-mon[39420]: osdmap e515: 12 total, 12 up, 11 in 2024-02-09T17:43:14.551 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:43:14.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:14.555 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:43:14.656 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:43:14.708 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:14.745 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:43:14.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:14.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:43:14.860 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:43:14.931 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:14.971 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:43:14.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:14.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:43:15.075 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:43:15.139 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:15.170 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:43:15.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:15.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:43:15.275 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:43:15.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:15 smithi179 ceph-mon[43975]: 1.c deep-scrub starts 2024-02-09T17:43:15.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:15 smithi179 ceph-mon[43975]: 1.c deep-scrub ok 2024-02-09T17:43:15.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:15 smithi179 ceph-mon[43975]: 1.17 scrub starts 2024-02-09T17:43:15.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:15 smithi179 ceph-mon[43975]: 1.17 scrub ok 2024-02-09T17:43:15.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:15 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-02-09T17:43:15.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:15 smithi179 ceph-mon[43975]: 1.6 scrub starts 2024-02-09T17:43:15.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:15 smithi179 ceph-mon[43975]: 1.6 scrub ok 2024-02-09T17:43:15.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:15 smithi162 ceph-mon[44304]: 1.c deep-scrub starts 2024-02-09T17:43:15.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:15 smithi162 ceph-mon[44304]: 1.c deep-scrub ok 2024-02-09T17:43:15.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:15 smithi162 ceph-mon[44304]: 1.17 scrub starts 2024-02-09T17:43:15.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:15 smithi162 ceph-mon[44304]: 1.17 scrub ok 2024-02-09T17:43:15.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:15 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-02-09T17:43:15.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:15 smithi162 ceph-mon[44304]: 1.6 scrub starts 2024-02-09T17:43:15.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:15 smithi162 ceph-mon[44304]: 1.6 scrub ok 2024-02-09T17:43:15.466 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:15.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:15 smithi099 ceph-mon[39420]: 1.c deep-scrub starts 2024-02-09T17:43:15.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:15 smithi099 ceph-mon[39420]: 1.c deep-scrub ok 2024-02-09T17:43:15.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:15 smithi099 ceph-mon[39420]: 1.17 scrub starts 2024-02-09T17:43:15.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:15 smithi099 ceph-mon[39420]: 1.17 scrub ok 2024-02-09T17:43:15.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:15 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-02-09T17:43:15.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:15 smithi099 ceph-mon[39420]: 1.6 scrub starts 2024-02-09T17:43:15.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:15 smithi099 ceph-mon[39420]: 1.6 scrub ok 2024-02-09T17:43:16.044 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:16 smithi179 ceph-mon[43975]: 1.b scrub starts 2024-02-09T17:43:16.044 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:16 smithi179 ceph-mon[43975]: 1.b scrub ok 2024-02-09T17:43:16.045 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:16 smithi179 ceph-mon[43975]: 1.8 scrub starts 2024-02-09T17:43:16.045 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:16 smithi179 ceph-mon[43975]: 1.8 scrub ok 2024-02-09T17:43:16.045 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:16 smithi179 ceph-mon[43975]: 1.d scrub starts 2024-02-09T17:43:16.045 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:16 smithi179 ceph-mon[43975]: 1.d scrub ok 2024-02-09T17:43:16.045 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:16 smithi179 ceph-mon[43975]: pgmap v1310: 47 pgs: 9 unknown, 38 active+clean; 5.8 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 865 op/s 2024-02-09T17:43:16.045 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:16 smithi179 ceph-mon[43975]: 1.a scrub starts 2024-02-09T17:43:16.045 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:16 smithi179 ceph-mon[43975]: 1.a scrub ok 2024-02-09T17:43:16.045 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:16 smithi179 ceph-mon[43975]: 1.e deep-scrub starts 2024-02-09T17:43:16.045 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:16 smithi179 ceph-mon[43975]: 1.e deep-scrub ok 2024-02-09T17:43:16.045 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:16 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-02-09T17:43:16.045 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:16 smithi179 ceph-mon[43975]: osdmap e516: 12 total, 12 up, 11 in 2024-02-09T17:43:16.045 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:16 smithi179 ceph-mon[43975]: 1.7 scrub starts 2024-02-09T17:43:16.045 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:16 smithi179 ceph-mon[43975]: 1.7 scrub ok 2024-02-09T17:43:16.046 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:16 smithi179 ceph-mon[43975]: 1.5 scrub starts 2024-02-09T17:43:16.046 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:16 smithi179 ceph-mon[43975]: 1.5 scrub ok 2024-02-09T17:43:16.223 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:43:16.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:16.227 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:43:16.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:16 smithi099 ceph-mon[39420]: 1.b scrub starts 2024-02-09T17:43:16.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:16 smithi099 ceph-mon[39420]: 1.b scrub ok 2024-02-09T17:43:16.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:16 smithi099 ceph-mon[39420]: 1.8 scrub starts 2024-02-09T17:43:16.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:16 smithi099 ceph-mon[39420]: 1.8 scrub ok 2024-02-09T17:43:16.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:16 smithi099 ceph-mon[39420]: 1.d scrub starts 2024-02-09T17:43:16.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:16 smithi099 ceph-mon[39420]: 1.d scrub ok 2024-02-09T17:43:16.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:16 smithi099 ceph-mon[39420]: pgmap v1310: 47 pgs: 9 unknown, 38 active+clean; 5.8 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 865 op/s 2024-02-09T17:43:16.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:16 smithi099 ceph-mon[39420]: 1.a scrub starts 2024-02-09T17:43:16.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:16 smithi099 ceph-mon[39420]: 1.a scrub ok 2024-02-09T17:43:16.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:16 smithi099 ceph-mon[39420]: 1.e deep-scrub starts 2024-02-09T17:43:16.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:16 smithi099 ceph-mon[39420]: 1.e deep-scrub ok 2024-02-09T17:43:16.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:16 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-02-09T17:43:16.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:16 smithi099 ceph-mon[39420]: osdmap e516: 12 total, 12 up, 11 in 2024-02-09T17:43:16.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:16 smithi099 ceph-mon[39420]: 1.7 scrub starts 2024-02-09T17:43:16.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:16 smithi099 ceph-mon[39420]: 1.7 scrub ok 2024-02-09T17:43:16.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:16 smithi099 ceph-mon[39420]: 1.5 scrub starts 2024-02-09T17:43:16.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:16 smithi099 ceph-mon[39420]: 1.5 scrub ok 2024-02-09T17:43:16.328 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:43:16.363 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:16.391 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:43:16.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:16.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:43:16.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:16 smithi162 ceph-mon[44304]: 1.b scrub starts 2024-02-09T17:43:16.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:16 smithi162 ceph-mon[44304]: 1.b scrub ok 2024-02-09T17:43:16.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:16 smithi162 ceph-mon[44304]: 1.8 scrub starts 2024-02-09T17:43:16.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:16 smithi162 ceph-mon[44304]: 1.8 scrub ok 2024-02-09T17:43:16.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:16 smithi162 ceph-mon[44304]: 1.d scrub starts 2024-02-09T17:43:16.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:16 smithi162 ceph-mon[44304]: 1.d scrub ok 2024-02-09T17:43:16.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:16 smithi162 ceph-mon[44304]: pgmap v1310: 47 pgs: 9 unknown, 38 active+clean; 5.8 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 865 op/s 2024-02-09T17:43:16.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:16 smithi162 ceph-mon[44304]: 1.a scrub starts 2024-02-09T17:43:16.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:16 smithi162 ceph-mon[44304]: 1.a scrub ok 2024-02-09T17:43:16.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:16 smithi162 ceph-mon[44304]: 1.e deep-scrub starts 2024-02-09T17:43:16.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:16 smithi162 ceph-mon[44304]: 1.e deep-scrub ok 2024-02-09T17:43:16.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:16 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-02-09T17:43:16.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:16 smithi162 ceph-mon[44304]: osdmap e516: 12 total, 12 up, 11 in 2024-02-09T17:43:16.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:16 smithi162 ceph-mon[44304]: 1.7 scrub starts 2024-02-09T17:43:16.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:16 smithi162 ceph-mon[44304]: 1.7 scrub ok 2024-02-09T17:43:16.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:16 smithi162 ceph-mon[44304]: 1.5 scrub starts 2024-02-09T17:43:16.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:16 smithi162 ceph-mon[44304]: 1.5 scrub ok 2024-02-09T17:43:16.495 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:43:16.540 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:16.565 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:43:16.568 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:16.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:43:16.670 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:43:16.703 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:16.726 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:43:16.731 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:16.731 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:43:16.832 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:43:16.863 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:16.894 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:43:16.896 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:16.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:43:16.901 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T17:43:16.901 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:43:16.997 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:43:17.048 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:17.060 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is set 2024-02-09T17:43:17.089 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:43:17.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:17.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:43:17.198 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:43:17.235 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:17 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:43:17.235 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:17 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:43:17.235 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:17 smithi179 ceph-mon[43975]: 1.9 deep-scrub starts 2024-02-09T17:43:17.235 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:17 smithi179 ceph-mon[43975]: 1.9 deep-scrub ok 2024-02-09T17:43:17.235 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:17 smithi179 ceph-mon[43975]: osdmap e517: 12 total, 12 up, 11 in 2024-02-09T17:43:17.235 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:17 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1036302674' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:43:17.235 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:17 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:43:17.237 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:17.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:17 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:43:17.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:17 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:43:17.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:17 smithi099 ceph-mon[39420]: 1.9 deep-scrub starts 2024-02-09T17:43:17.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:17 smithi099 ceph-mon[39420]: 1.9 deep-scrub ok 2024-02-09T17:43:17.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:17 smithi099 ceph-mon[39420]: osdmap e517: 12 total, 12 up, 11 in 2024-02-09T17:43:17.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:17 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1036302674' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:43:17.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:17 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:43:17.307 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:43:17.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:17.315 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:43:17.416 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:43:17.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:17 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:43:17.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:17 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:43:17.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:17 smithi162 ceph-mon[44304]: 1.9 deep-scrub starts 2024-02-09T17:43:17.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:17 smithi162 ceph-mon[44304]: 1.9 deep-scrub ok 2024-02-09T17:43:17.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:17 smithi162 ceph-mon[44304]: osdmap e517: 12 total, 12 up, 11 in 2024-02-09T17:43:17.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:17 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1036302674' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:43:17.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:17 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:43:17.446 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:17.660 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.11 dump_blocked_ops 2024-02-09T17:43:17.705 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:43:17.999 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:43:18.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:18.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:43:18.008 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] 2024-02-09T17:43:18.008 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:43:18.008 INFO:tasks.thrashosds.thrasher:Adding osd 2 2024-02-09T17:43:18.008 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd in 2 2024-02-09T17:43:18.049 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:43:18.053 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:43:18.109 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:43:18.174 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:18.206 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:43:18.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:18.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:43:18.314 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:43:18.321 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:18 smithi099 ceph-mon[39420]: pgmap v1313: 47 pgs: 1 active+clean+scrubbing+deep, 9 unknown, 37 active+clean; 5.7 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 856 op/s; 18 KiB/s, 0 objects/s recovering 2024-02-09T17:43:18.322 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:18 smithi099 ceph-mon[39420]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:43:18.322 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:18 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:43:18.322 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:18 smithi099 ceph-mon[39420]: osdmap e518: 12 total, 12 up, 11 in 2024-02-09T17:43:18.322 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:18 smithi099 ceph-mon[39420]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-02-09T17:43:18.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:18 smithi179 ceph-mon[43975]: pgmap v1313: 47 pgs: 1 active+clean+scrubbing+deep, 9 unknown, 37 active+clean; 5.7 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 856 op/s; 18 KiB/s, 0 objects/s recovering 2024-02-09T17:43:18.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:18 smithi179 ceph-mon[43975]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:43:18.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:18 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:43:18.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:18 smithi179 ceph-mon[43975]: osdmap e518: 12 total, 12 up, 11 in 2024-02-09T17:43:18.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:18 smithi179 ceph-mon[43975]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-02-09T17:43:18.353 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:18.379 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:43:18.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:18.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:43:18.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:18 smithi162 ceph-mon[44304]: pgmap v1313: 47 pgs: 1 active+clean+scrubbing+deep, 9 unknown, 37 active+clean; 5.7 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 856 op/s; 18 KiB/s, 0 objects/s recovering 2024-02-09T17:43:18.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:18 smithi162 ceph-mon[44304]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:43:18.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:18 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:43:18.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:18 smithi162 ceph-mon[44304]: osdmap e518: 12 total, 12 up, 11 in 2024-02-09T17:43:18.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:18 smithi162 ceph-mon[44304]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-02-09T17:43:18.484 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:43:18.541 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:18.577 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:43:18.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:18.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:43:18.686 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:43:18.751 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:18.782 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:43:18.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:18.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:43:18.887 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:43:18.918 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:18.958 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:43:18.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:18.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:43:19.066 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:43:19.099 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:19.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:19 smithi099 ceph-mon[39420]: osdmap e519: 12 total, 12 up, 11 in 2024-02-09T17:43:19.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:19 smithi179 ceph-mon[43975]: osdmap e519: 12 total, 12 up, 11 in 2024-02-09T17:43:19.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:19 smithi162 ceph-mon[44304]: osdmap e519: 12 total, 12 up, 11 in 2024-02-09T17:43:19.550 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:43:19.552 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:19.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:43:19.636 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:43:19.653 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:43:19.687 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:19.889 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:43:19.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:19.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:43:19.996 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:43:20.000 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-09T17:43:20.063 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:20.106 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:43:20.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:20.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:43:20.175 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:20 smithi099 ceph-mon[39420]: pgmap v1316: 47 pgs: 1 remapped+peering, 2 active+undersized+remapped, 1 active+clean+scrubbing+deep, 6 unknown, 37 active+clean; 5.7 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 2.7 KiB/s rd, 67 KiB/s wr, 640 op/s 2024-02-09T17:43:20.175 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:20 smithi099 ceph-mon[39420]: osdmap e520: 12 total, 12 up, 11 in 2024-02-09T17:43:20.214 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:43:20.256 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:20.290 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:43:20.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:20.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:43:20.345 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:20 smithi179 ceph-mon[43975]: pgmap v1316: 47 pgs: 1 remapped+peering, 2 active+undersized+remapped, 1 active+clean+scrubbing+deep, 6 unknown, 37 active+clean; 5.7 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 2.7 KiB/s rd, 67 KiB/s wr, 640 op/s 2024-02-09T17:43:20.345 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:20 smithi179 ceph-mon[43975]: osdmap e520: 12 total, 12 up, 11 in 2024-02-09T17:43:20.394 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:43:20.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:20 smithi162 ceph-mon[44304]: pgmap v1316: 47 pgs: 1 remapped+peering, 2 active+undersized+remapped, 1 active+clean+scrubbing+deep, 6 unknown, 37 active+clean; 5.7 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 2.7 KiB/s rd, 67 KiB/s wr, 640 op/s 2024-02-09T17:43:20.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:20 smithi162 ceph-mon[44304]: osdmap e520: 12 total, 12 up, 11 in 2024-02-09T17:43:20.449 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:20.454 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:43:20.483 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:43:20.517 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:43:20.540 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:43:20.565 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:43:20.604 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:43:20.634 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:43:20.657 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:43:20.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:20.664 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:43:20.673 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:43:20.676 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T17:43:20.677 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:43:20.699 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:43:20.723 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:43:20.748 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:43:20.765 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:43:20.771 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:43:20.773 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:43:20.773 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:43:20.773 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:43:20.773 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:43:20.773 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:43:20.773 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:43:20.773 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:43:20.773 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:43:20.774 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:43:20.774 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:43:20.774 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:43:20.774 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:43:20.803 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:20.830 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:43:20.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:20.839 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:43:20.940 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:43:20.972 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:20.997 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:43:21.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:21.000 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:43:21.102 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:43:21.145 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:21.171 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:43:21.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:21.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:43:21.283 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:43:21.318 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:21.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:21 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/860886558' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:43:21.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:21 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/860886558' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:43:21.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:21 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/860886558' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:43:21.679 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:43:21.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:21.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:43:21.785 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:43:21.819 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:21.874 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.11 dump_historic_ops 2024-02-09T17:43:21.923 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:43:21.933 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:21.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:43:22.034 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:43:22.089 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:22.145 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:43:22.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:22.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:43:22.211 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:43:22.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:22 smithi099 ceph-mon[39420]: pgmap v1318: 47 pgs: 1 active+undersized+remapped, 1 active+clean+scrubbing+deep, 4 unknown, 41 active+clean; 5.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 1.9 KiB/s rd, 42 KiB/s wr, 649 op/s; 0 B/s, 0 objects/s recovering 2024-02-09T17:43:22.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:22 smithi099 ceph-mon[39420]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 1 pg peering) 2024-02-09T17:43:22.254 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:43:22.255 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:43:22.328 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:22.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:22 smithi179 ceph-mon[43975]: pgmap v1318: 47 pgs: 1 active+undersized+remapped, 1 active+clean+scrubbing+deep, 4 unknown, 41 active+clean; 5.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 1.9 KiB/s rd, 42 KiB/s wr, 649 op/s; 0 B/s, 0 objects/s recovering 2024-02-09T17:43:22.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:22 smithi179 ceph-mon[43975]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 1 pg peering) 2024-02-09T17:43:22.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:22 smithi162 ceph-mon[44304]: pgmap v1318: 47 pgs: 1 active+undersized+remapped, 1 active+clean+scrubbing+deep, 4 unknown, 41 active+clean; 5.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 1.9 KiB/s rd, 42 KiB/s wr, 649 op/s; 0 B/s, 0 objects/s recovering 2024-02-09T17:43:22.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:22 smithi162 ceph-mon[44304]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 1 pg peering) 2024-02-09T17:43:22.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:22.792 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:43:22.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:43:22.893 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:43:22.926 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:23.715 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:43:23.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:23 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mon-a[39397]: 2024-02-09T17:43:23.444+0000 7feb57590700 -1 mon.a@0(leader).osd e520 definitely_dead 0 2024-02-09T17:43:24.080 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:43:24.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:24.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:43:24.108 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:43:24.166 INFO:teuthology.orchestra.run.smithi099.stderr:marked in osd.2. 2024-02-09T17:43:24.186 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:43:24.215 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:24.329 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:43:24.329 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-02-09T17:43:24.329 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-02-09T17:43:24.329 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-02-09T17:43:24.329 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.329 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:208403 4.8 4:163e1e0f:::benchmark_data_smithi179_103809_object70937:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-02-09T17:43:24.329 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:33.810176+0000", 2024-02-09T17:43:24.329 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 290.51504669399998, 2024-02-09T17:43:24.329 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2455196100000001, 2024-02-09T17:43:24.330 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:24.330 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-02-09T17:43:24.330 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:24.330 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:43:24.330 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:43:24.330 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 208403 2024-02-09T17:43:24.330 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.330 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:24.330 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.330 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:24.330 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.810176+0000", 2024-02-09T17:43:24.330 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:24.331 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.331 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.331 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:24.331 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.810176+0000", 2024-02-09T17:43:24.331 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:24.331 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.331 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.331 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:24.331 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.810177+0000", 2024-02-09T17:43:24.331 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.63e-07 2024-02-09T17:43:24.331 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.331 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.331 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:24.331 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.810179+0000", 2024-02-09T17:43:24.331 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5000000000000002e-06 2024-02-09T17:43:24.332 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.332 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.332 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:24.332 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.810186+0000", 2024-02-09T17:43:24.332 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1650000000000003e-06 2024-02-09T17:43:24.332 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.332 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.332 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:24.332 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.810191+0000", 2024-02-09T17:43:24.332 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0359999999999997e-06 2024-02-09T17:43:24.332 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.332 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.332 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:24.333 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.810238+0000", 2024-02-09T17:43:24.333 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7083000000000003e-05 2024-02-09T17:43:24.333 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.333 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.333 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:24.333 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.810351+0000", 2024-02-09T17:43:24.333 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000112879 2024-02-09T17:43:24.333 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.333 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.333 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 6,7", 2024-02-09T17:43:24.333 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.810458+0000", 2024-02-09T17:43:24.333 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000107941 2024-02-09T17:43:24.333 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.333 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.334 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:24.334 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.816423+0000", 2024-02-09T17:43:24.334 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0059641260000000001 2024-02-09T17:43:24.334 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.334 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.334 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:24.334 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.821967+0000", 2024-02-09T17:43:24.334 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0055448889999999999 2024-02-09T17:43:24.334 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.334 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.334 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:43:24.334 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.821982+0000", 2024-02-09T17:43:24.334 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4341e-05 2024-02-09T17:43:24.335 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.335 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.335 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:24.335 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:34.813273+0000", 2024-02-09T17:43:24.335 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.99129126999999995 2024-02-09T17:43:24.335 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.335 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.335 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:43:24.335 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:34.813286+0000", 2024-02-09T17:43:24.335 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.278e-05 2024-02-09T17:43:24.335 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.335 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.335 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:24.335 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.042595+0000", 2024-02-09T17:43:24.336 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.229308955 2024-02-09T17:43:24.336 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.336 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.336 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for missing object", 2024-02-09T17:43:24.336 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.042737+0000", 2024-02-09T17:43:24.336 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014233599999999999 2024-02-09T17:43:24.336 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.336 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.336 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:24.336 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.055624+0000", 2024-02-09T17:43:24.336 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.012886556 2024-02-09T17:43:24.336 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.336 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.337 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:24.337 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.055696+0000", 2024-02-09T17:43:24.337 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2290000000000001e-05 2024-02-09T17:43:24.337 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:24.337 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:24.337 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:24.337 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.337 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.337 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:208414 4.8 4:1c1fd668:::benchmark_data_smithi179_103809_object70948:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-02-09T17:43:24.337 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:33.829970+0000", 2024-02-09T17:43:24.337 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 290.49525287699998, 2024-02-09T17:43:24.337 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2251981070000002, 2024-02-09T17:43:24.337 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:24.338 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:24.338 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:24.338 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:43:24.338 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:43:24.338 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 208414 2024-02-09T17:43:24.338 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.338 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:24.338 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.338 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:24.338 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.829970+0000", 2024-02-09T17:43:24.338 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:24.338 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.338 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.338 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:24.338 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.829970+0000", 2024-02-09T17:43:24.339 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:24.339 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.339 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.339 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:24.339 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.829972+0000", 2024-02-09T17:43:24.339 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.471e-06 2024-02-09T17:43:24.339 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.339 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.339 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:24.339 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.829978+0000", 2024-02-09T17:43:24.339 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4840000000000001e-06 2024-02-09T17:43:24.339 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.339 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.339 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:24.339 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.829989+0000", 2024-02-09T17:43:24.340 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0446e-05 2024-02-09T17:43:24.340 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.340 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.340 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:24.340 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.829995+0000", 2024-02-09T17:43:24.340 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8549999999999996e-06 2024-02-09T17:43:24.340 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.340 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.340 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:24.340 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.830040+0000", 2024-02-09T17:43:24.340 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4743999999999997e-05 2024-02-09T17:43:24.340 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.340 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.341 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:43:24.341 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.830048+0000", 2024-02-09T17:43:24.341 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7279999999999995e-06 2024-02-09T17:43:24.341 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.341 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.341 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:24.341 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:34.813315+0000", 2024-02-09T17:43:24.341 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.98326734000000005 2024-02-09T17:43:24.341 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.341 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.341 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:43:24.341 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:34.813323+0000", 2024-02-09T17:43:24.341 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7919999999999993e-06 2024-02-09T17:43:24.341 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.342 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.342 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:24.342 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.043262+0000", 2024-02-09T17:43:24.342 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.229938456 2024-02-09T17:43:24.342 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.342 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.342 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:24.342 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.043359+0000", 2024-02-09T17:43:24.342 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7549999999999999e-05 2024-02-09T17:43:24.342 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.342 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.342 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 6,7,9", 2024-02-09T17:43:24.342 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.043666+0000", 2024-02-09T17:43:24.342 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00030711699999999998 2024-02-09T17:43:24.342 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.342 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.343 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.343 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.048474+0000", 2024-02-09T17:43:24.343 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0048078649999999997 2024-02-09T17:43:24.343 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.343 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.343 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:24.343 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.048858+0000", 2024-02-09T17:43:24.343 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00038438799999999999 2024-02-09T17:43:24.343 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.343 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.343 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.343 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.051829+0000", 2024-02-09T17:43:24.343 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0029708260000000002 2024-02-09T17:43:24.344 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.344 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.344 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.344 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.054961+0000", 2024-02-09T17:43:24.344 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0031320850000000002 2024-02-09T17:43:24.344 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.344 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.344 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:24.344 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.055029+0000", 2024-02-09T17:43:24.344 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8108000000000006e-05 2024-02-09T17:43:24.344 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.344 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.344 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:24.344 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.055168+0000", 2024-02-09T17:43:24.345 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013885199999999999 2024-02-09T17:43:24.345 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:24.345 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:24.345 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:24.345 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.345 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.345 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:222266 4.8 4:1579dd15:::benchmark_data_smithi179_103809_object84800:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-02-09T17:43:24.345 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:39:02.600778+0000", 2024-02-09T17:43:24.345 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 261.72444489499998, 2024-02-09T17:43:24.345 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.171074366, 2024-02-09T17:43:24.345 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:24.345 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:24.345 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:24.345 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:43:24.345 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:43:24.346 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 222266 2024-02-09T17:43:24.346 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.346 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:24.346 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.346 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:24.346 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.600778+0000", 2024-02-09T17:43:24.346 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:24.346 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.346 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.346 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:24.346 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.600778+0000", 2024-02-09T17:43:24.346 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:24.346 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.346 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.346 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:24.346 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.600780+0000", 2024-02-09T17:43:24.347 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.764e-06 2024-02-09T17:43:24.347 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.347 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.347 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:24.347 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.600784+0000", 2024-02-09T17:43:24.347 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1389999999999997e-06 2024-02-09T17:43:24.347 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.347 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.347 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:24.347 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.600796+0000", 2024-02-09T17:43:24.347 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1697e-05 2024-02-09T17:43:24.347 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.347 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.347 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:24.348 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.600803+0000", 2024-02-09T17:43:24.348 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2350000000000002e-06 2024-02-09T17:43:24.348 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.348 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.348 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:24.348 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.600850+0000", 2024-02-09T17:43:24.348 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6798999999999997e-05 2024-02-09T17:43:24.348 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.348 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.348 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:43:24.348 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.600859+0000", 2024-02-09T17:43:24.348 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.533e-06 2024-02-09T17:43:24.348 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.348 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.349 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:24.349 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:03.595746+0000", 2024-02-09T17:43:24.349 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.99488617000000001 2024-02-09T17:43:24.349 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.349 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.349 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:43:24.349 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:03.595773+0000", 2024-02-09T17:43:24.349 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7568000000000001e-05 2024-02-09T17:43:24.349 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.349 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.349 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:24.349 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:04.756038+0000", 2024-02-09T17:43:24.349 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.160265001 2024-02-09T17:43:24.349 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.350 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.350 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:24.350 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:04.756175+0000", 2024-02-09T17:43:24.350 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013715 2024-02-09T17:43:24.350 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.350 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.350 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 7,9,10", 2024-02-09T17:43:24.350 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:04.756283+0000", 2024-02-09T17:43:24.350 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010802 2024-02-09T17:43:24.350 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.350 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.350 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.350 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:04.760891+0000", 2024-02-09T17:43:24.350 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0046074280000000002 2024-02-09T17:43:24.350 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.350 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.351 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.351 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:04.760958+0000", 2024-02-09T17:43:24.351 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7621999999999997e-05 2024-02-09T17:43:24.351 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.351 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.351 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:24.351 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:04.760979+0000", 2024-02-09T17:43:24.351 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0386000000000001e-05 2024-02-09T17:43:24.351 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.351 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.351 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.351 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:04.771736+0000", 2024-02-09T17:43:24.351 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010757325999999999 2024-02-09T17:43:24.351 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.351 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.352 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:24.352 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:04.771781+0000", 2024-02-09T17:43:24.352 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4974e-05 2024-02-09T17:43:24.352 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.352 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.352 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:24.352 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:04.771853+0000", 2024-02-09T17:43:24.352 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1554e-05 2024-02-09T17:43:24.352 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:24.352 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:24.352 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:24.352 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.352 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.352 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:222292 4.8 4:10fa3862:::benchmark_data_smithi179_103809_object84826:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e399)", 2024-02-09T17:43:24.352 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:39:03.757917+0000", 2024-02-09T17:43:24.353 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 260.56730634100001, 2024-02-09T17:43:24.353 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0220161219999999, 2024-02-09T17:43:24.353 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:24.353 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:24.353 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:24.353 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:43:24.353 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:43:24.353 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 222292 2024-02-09T17:43:24.353 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.353 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:24.353 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.353 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:24.353 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:03.757917+0000", 2024-02-09T17:43:24.353 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:24.354 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.354 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.354 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:24.354 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:03.757917+0000", 2024-02-09T17:43:24.354 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:24.354 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.354 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.354 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:24.354 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:03.757919+0000", 2024-02-09T17:43:24.354 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2979999999999999e-06 2024-02-09T17:43:24.354 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.354 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.354 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:24.354 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:03.757927+0000", 2024-02-09T17:43:24.354 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8590000000000005e-06 2024-02-09T17:43:24.355 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.355 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.355 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:24.355 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:03.757944+0000", 2024-02-09T17:43:24.355 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.753e-05 2024-02-09T17:43:24.355 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.355 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.355 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:24.355 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:03.757955+0000", 2024-02-09T17:43:24.355 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0563e-05 2024-02-09T17:43:24.355 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.355 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.355 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:24.355 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:03.758024+0000", 2024-02-09T17:43:24.355 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8775999999999993e-05 2024-02-09T17:43:24.356 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.356 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.356 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:43:24.356 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:03.758035+0000", 2024-02-09T17:43:24.356 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0872e-05 2024-02-09T17:43:24.356 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.356 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.356 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:24.356 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:04.760994+0000", 2024-02-09T17:43:24.356 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0029593960000001 2024-02-09T17:43:24.356 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.356 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.356 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:24.357 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:04.761083+0000", 2024-02-09T17:43:24.357 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.9345000000000006e-05 2024-02-09T17:43:24.357 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.357 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.357 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 7,9,10", 2024-02-09T17:43:24.357 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:04.761186+0000", 2024-02-09T17:43:24.357 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000103102 2024-02-09T17:43:24.357 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.357 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.357 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.357 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:04.766313+0000", 2024-02-09T17:43:24.357 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0051261850000000001 2024-02-09T17:43:24.357 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.357 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.357 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.358 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:04.766372+0000", 2024-02-09T17:43:24.358 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9181000000000001e-05 2024-02-09T17:43:24.358 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.358 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.358 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:24.358 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:04.766389+0000", 2024-02-09T17:43:24.358 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.694e-05 2024-02-09T17:43:24.358 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.358 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.358 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.358 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:04.779761+0000", 2024-02-09T17:43:24.358 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013372124000000001 2024-02-09T17:43:24.358 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.358 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.358 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:24.359 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:04.779831+0000", 2024-02-09T17:43:24.359 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9968000000000005e-05 2024-02-09T17:43:24.359 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.359 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.359 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:24.359 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:04.779933+0000", 2024-02-09T17:43:24.359 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000101983 2024-02-09T17:43:24.359 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:24.359 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:24.359 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:24.359 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.359 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.359 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:229698 4.8 4:198ec6a7:::benchmark_data_smithi179_103809_object92232:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e409)", 2024-02-09T17:43:24.359 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:39:17.475832+0000", 2024-02-09T17:43:24.360 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 246.84939148500001, 2024-02-09T17:43:24.360 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.025248959, 2024-02-09T17:43:24.360 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:24.360 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:24.360 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:24.360 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:43:24.360 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:43:24.360 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 229698 2024-02-09T17:43:24.360 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.360 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:24.360 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.360 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:24.360 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:17.475832+0000", 2024-02-09T17:43:24.361 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:24.361 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.361 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.361 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:24.361 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:17.475832+0000", 2024-02-09T17:43:24.361 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:24.361 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.361 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.361 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:24.361 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:17.475833+0000", 2024-02-09T17:43:24.361 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7000000000000003e-07 2024-02-09T17:43:24.361 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.361 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.361 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:24.361 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:17.475837+0000", 2024-02-09T17:43:24.362 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7840000000000003e-06 2024-02-09T17:43:24.362 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.362 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.362 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:24.362 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:17.475856+0000", 2024-02-09T17:43:24.362 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8580000000000002e-05 2024-02-09T17:43:24.362 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.362 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.362 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:24.362 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:17.475862+0000", 2024-02-09T17:43:24.362 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1419999999999998e-06 2024-02-09T17:43:24.362 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.362 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.362 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:24.362 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:17.475917+0000", 2024-02-09T17:43:24.363 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5381999999999999e-05 2024-02-09T17:43:24.363 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.363 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.363 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:43:24.363 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:17.475928+0000", 2024-02-09T17:43:24.363 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0288999999999999e-05 2024-02-09T17:43:24.363 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.363 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.363 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:24.363 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:18.491094+0000", 2024-02-09T17:43:24.363 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0151661599999999 2024-02-09T17:43:24.363 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.363 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.363 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:24.363 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:18.491222+0000", 2024-02-09T17:43:24.364 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012769300000000001 2024-02-09T17:43:24.364 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.364 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.364 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 9,10", 2024-02-09T17:43:24.364 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:18.491373+0000", 2024-02-09T17:43:24.364 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015154099999999999 2024-02-09T17:43:24.364 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.364 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.364 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.364 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:18.496295+0000", 2024-02-09T17:43:24.364 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0049214920000000004 2024-02-09T17:43:24.364 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.364 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.364 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.365 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:18.496363+0000", 2024-02-09T17:43:24.365 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8422000000000003e-05 2024-02-09T17:43:24.365 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.365 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.365 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:24.365 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:18.500979+0000", 2024-02-09T17:43:24.365 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0046163360000000004 2024-02-09T17:43:24.365 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.365 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.365 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:24.365 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:18.501037+0000", 2024-02-09T17:43:24.365 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7222000000000002e-05 2024-02-09T17:43:24.365 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.365 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.365 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:24.366 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:18.501081+0000", 2024-02-09T17:43:24.366 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3946000000000002e-05 2024-02-09T17:43:24.366 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:24.366 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:24.366 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:24.366 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.366 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.366 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:266612 4.8 4:18a47bff:::benchmark_data_smithi179_103809_object129146:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e451)", 2024-02-09T17:43:24.366 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:40:18.489717+0000", 2024-02-09T17:43:24.366 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 185.83550642899999, 2024-02-09T17:43:24.366 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1935931609999999, 2024-02-09T17:43:24.366 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:24.366 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:24.366 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:24.366 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:43:24.367 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:43:24.367 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 266612 2024-02-09T17:43:24.367 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.367 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:24.367 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.367 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:24.367 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.489717+0000", 2024-02-09T17:43:24.367 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:24.367 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.367 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.367 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:24.367 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.489717+0000", 2024-02-09T17:43:24.367 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:24.367 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.367 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.368 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:24.368 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.489718+0000", 2024-02-09T17:43:24.368 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.469e-06 2024-02-09T17:43:24.368 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.368 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.368 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:24.368 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.489723+0000", 2024-02-09T17:43:24.368 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9030000000000003e-06 2024-02-09T17:43:24.368 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.368 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.368 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:24.368 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.489735+0000", 2024-02-09T17:43:24.368 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2111e-05 2024-02-09T17:43:24.368 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.369 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.369 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:24.369 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.489751+0000", 2024-02-09T17:43:24.369 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6099e-05 2024-02-09T17:43:24.369 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.369 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.369 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:24.369 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.489809+0000", 2024-02-09T17:43:24.369 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7327999999999997e-05 2024-02-09T17:43:24.369 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.369 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.369 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:43:24.369 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.489817+0000", 2024-02-09T17:43:24.369 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5639999999999997e-06 2024-02-09T17:43:24.369 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.369 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.370 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:24.370 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.670050+0000", 2024-02-09T17:43:24.370 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1802329739999999 2024-02-09T17:43:24.370 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.370 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.370 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:24.370 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.670154+0000", 2024-02-09T17:43:24.370 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000103429 2024-02-09T17:43:24.370 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.370 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.370 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 6,7,9,10", 2024-02-09T17:43:24.370 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.670246+0000", 2024-02-09T17:43:24.370 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2204e-05 2024-02-09T17:43:24.370 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.370 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.371 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.371 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.674627+0000", 2024-02-09T17:43:24.371 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0043817509999999997 2024-02-09T17:43:24.371 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.371 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.371 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:24.371 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.674639+0000", 2024-02-09T17:43:24.371 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1732999999999999e-05 2024-02-09T17:43:24.371 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.371 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.371 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.371 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.674671+0000", 2024-02-09T17:43:24.371 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2150999999999998e-05 2024-02-09T17:43:24.371 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.372 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.372 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.372 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.683156+0000", 2024-02-09T17:43:24.372 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0084847020000000002 2024-02-09T17:43:24.372 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.372 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.372 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.372 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.683195+0000", 2024-02-09T17:43:24.372 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8481e-05 2024-02-09T17:43:24.372 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.372 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.372 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:24.372 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.683235+0000", 2024-02-09T17:43:24.372 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0068000000000003e-05 2024-02-09T17:43:24.373 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.373 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.373 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:24.373 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.683310+0000", 2024-02-09T17:43:24.373 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5193999999999999e-05 2024-02-09T17:43:24.373 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:24.373 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:24.373 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:24.373 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.373 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.373 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:266621 4.8 4:1943a0f5:::benchmark_data_smithi179_103809_object129155:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e451)", 2024-02-09T17:43:24.373 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:40:18.499697+0000", 2024-02-09T17:43:24.373 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 185.825525769, 2024-02-09T17:43:24.373 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.184130288, 2024-02-09T17:43:24.373 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:24.373 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:24.374 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:24.374 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:43:24.374 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:43:24.374 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 266621 2024-02-09T17:43:24.374 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.374 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:24.374 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.374 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:24.374 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.499697+0000", 2024-02-09T17:43:24.374 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:24.374 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.374 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.374 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:24.374 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.499697+0000", 2024-02-09T17:43:24.375 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:24.375 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.375 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.375 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:24.375 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.499698+0000", 2024-02-09T17:43:24.375 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.009e-06 2024-02-09T17:43:24.375 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.375 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.375 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:24.375 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.499702+0000", 2024-02-09T17:43:24.375 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6600000000000001e-06 2024-02-09T17:43:24.375 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.375 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.375 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:24.375 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.499711+0000", 2024-02-09T17:43:24.375 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7390000000000005e-06 2024-02-09T17:43:24.375 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.376 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.376 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:24.376 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.499718+0000", 2024-02-09T17:43:24.376 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2099999999999996e-06 2024-02-09T17:43:24.376 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.376 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.376 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:24.376 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.499773+0000", 2024-02-09T17:43:24.376 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4901999999999997e-05 2024-02-09T17:43:24.376 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.376 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.376 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:43:24.376 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.499781+0000", 2024-02-09T17:43:24.377 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.0369999999999995e-06 2024-02-09T17:43:24.377 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.377 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.377 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:24.377 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.674807+0000", 2024-02-09T17:43:24.377 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.175026063 2024-02-09T17:43:24.377 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.377 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.377 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:24.377 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.674902+0000", 2024-02-09T17:43:24.377 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5045999999999998e-05 2024-02-09T17:43:24.377 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.377 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.377 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 6,7,9,10", 2024-02-09T17:43:24.377 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.675005+0000", 2024-02-09T17:43:24.378 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000103119 2024-02-09T17:43:24.378 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.378 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.378 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:24.378 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.683168+0000", 2024-02-09T17:43:24.378 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0081628150000000003 2024-02-09T17:43:24.378 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.378 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.378 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.378 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.683361+0000", 2024-02-09T17:43:24.378 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00019343 2024-02-09T17:43:24.378 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.378 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.378 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.378 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.683430+0000", 2024-02-09T17:43:24.379 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8336000000000004e-05 2024-02-09T17:43:24.379 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.379 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.379 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.379 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.683470+0000", 2024-02-09T17:43:24.379 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9857999999999997e-05 2024-02-09T17:43:24.379 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.379 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.379 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.379 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.683693+0000", 2024-02-09T17:43:24.379 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00022318699999999999 2024-02-09T17:43:24.379 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.379 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.379 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:24.379 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.683731+0000", 2024-02-09T17:43:24.380 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8349999999999997e-05 2024-02-09T17:43:24.380 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.380 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.380 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:24.380 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.683828+0000", 2024-02-09T17:43:24.380 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6527000000000006e-05 2024-02-09T17:43:24.380 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:24.380 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:24.380 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:24.380 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.380 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.380 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:266626 4.8 4:11c2026a:::benchmark_data_smithi179_103809_object129160:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e451)", 2024-02-09T17:43:24.380 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:40:18.504323+0000", 2024-02-09T17:43:24.380 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 185.820900026, 2024-02-09T17:43:24.381 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1797135670000001, 2024-02-09T17:43:24.381 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:24.381 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:24.381 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:24.381 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:43:24.381 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:43:24.381 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 266626 2024-02-09T17:43:24.381 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.381 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:24.381 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.381 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:24.381 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.504323+0000", 2024-02-09T17:43:24.381 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:24.381 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.381 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.382 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:24.382 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.504323+0000", 2024-02-09T17:43:24.382 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:24.382 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.382 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.382 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:24.382 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.504324+0000", 2024-02-09T17:43:24.382 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.6499999999999998e-07 2024-02-09T17:43:24.382 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.382 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.382 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:24.382 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.504328+0000", 2024-02-09T17:43:24.382 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8659999999999999e-06 2024-02-09T17:43:24.382 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.383 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.383 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:24.383 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.504339+0000", 2024-02-09T17:43:24.383 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1610000000000001e-05 2024-02-09T17:43:24.383 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.383 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.383 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:24.383 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.504345+0000", 2024-02-09T17:43:24.383 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8590000000000001e-06 2024-02-09T17:43:24.383 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.383 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.383 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:24.383 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.504397+0000", 2024-02-09T17:43:24.383 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.198e-05 2024-02-09T17:43:24.383 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.384 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.384 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:43:24.384 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.504407+0000", 2024-02-09T17:43:24.384 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.3139999999999993e-06 2024-02-09T17:43:24.384 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.384 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.384 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:24.384 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.678978+0000", 2024-02-09T17:43:24.384 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1745714709999999 2024-02-09T17:43:24.384 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.384 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.384 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:24.384 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.679072+0000", 2024-02-09T17:43:24.385 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.4234000000000006e-05 2024-02-09T17:43:24.385 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.385 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.385 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 6,7,9,10", 2024-02-09T17:43:24.385 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.679189+0000", 2024-02-09T17:43:24.385 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000116743 2024-02-09T17:43:24.385 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.385 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.385 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:24.385 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.683485+0000", 2024-02-09T17:43:24.385 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0042956160000000004 2024-02-09T17:43:24.385 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.385 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.385 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.386 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.683522+0000", 2024-02-09T17:43:24.386 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7728999999999997e-05 2024-02-09T17:43:24.386 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.386 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.386 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.386 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.683661+0000", 2024-02-09T17:43:24.386 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013825799999999999 2024-02-09T17:43:24.386 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.386 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.386 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.386 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.683866+0000", 2024-02-09T17:43:24.386 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00020525 2024-02-09T17:43:24.386 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.386 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.386 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.386 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.683907+0000", 2024-02-09T17:43:24.387 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1004999999999998e-05 2024-02-09T17:43:24.387 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.387 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.387 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:24.387 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.683939+0000", 2024-02-09T17:43:24.387 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1986000000000002e-05 2024-02-09T17:43:24.387 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.387 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.387 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:24.387 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.684037+0000", 2024-02-09T17:43:24.387 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7781000000000005e-05 2024-02-09T17:43:24.387 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:24.387 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:24.387 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:24.387 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.388 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.388 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:266690 4.8 4:1a762542:::benchmark_data_smithi179_103809_object129224:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e451)", 2024-02-09T17:43:24.388 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:40:18.574021+0000", 2024-02-09T17:43:24.388 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 185.75120209900001, 2024-02-09T17:43:24.388 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.12055982, 2024-02-09T17:43:24.388 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:24.388 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:24.388 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:24.388 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:43:24.388 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:43:24.388 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 266690 2024-02-09T17:43:24.388 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.388 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:24.389 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.389 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:24.389 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.574021+0000", 2024-02-09T17:43:24.389 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:24.389 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.389 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.389 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:24.389 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.574021+0000", 2024-02-09T17:43:24.389 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:24.389 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.389 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.389 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:24.389 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.574023+0000", 2024-02-09T17:43:24.389 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6020000000000001e-06 2024-02-09T17:43:24.390 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.390 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.390 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:24.390 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.574029+0000", 2024-02-09T17:43:24.390 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2339999999999999e-06 2024-02-09T17:43:24.390 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.390 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.390 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:24.390 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.574045+0000", 2024-02-09T17:43:24.390 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.647e-05 2024-02-09T17:43:24.390 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.390 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.390 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:24.390 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.574053+0000", 2024-02-09T17:43:24.390 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5480000000000004e-06 2024-02-09T17:43:24.391 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.391 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.391 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:24.391 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.574104+0000", 2024-02-09T17:43:24.391 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0966e-05 2024-02-09T17:43:24.391 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.391 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.391 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:43:24.391 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.574112+0000", 2024-02-09T17:43:24.391 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3790000000000006e-06 2024-02-09T17:43:24.391 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.391 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.391 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:24.391 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.684083+0000", 2024-02-09T17:43:24.391 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.109971177 2024-02-09T17:43:24.392 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.392 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.392 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:24.392 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.684227+0000", 2024-02-09T17:43:24.392 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014359600000000001 2024-02-09T17:43:24.392 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.392 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.392 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 6,7,9,10", 2024-02-09T17:43:24.392 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.684373+0000", 2024-02-09T17:43:24.392 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014643499999999999 2024-02-09T17:43:24.392 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.392 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.392 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.392 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.694157+0000", 2024-02-09T17:43:24.393 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0097837280000000002 2024-02-09T17:43:24.393 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.393 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.393 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.393 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.694224+0000", 2024-02-09T17:43:24.393 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6450000000000002e-05 2024-02-09T17:43:24.393 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.393 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.393 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:24.393 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.694239+0000", 2024-02-09T17:43:24.393 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5739999999999998e-05 2024-02-09T17:43:24.393 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.393 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.393 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.394 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.694282+0000", 2024-02-09T17:43:24.394 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2200999999999998e-05 2024-02-09T17:43:24.394 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.394 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.394 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.394 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.694396+0000", 2024-02-09T17:43:24.394 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000114221 2024-02-09T17:43:24.394 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.394 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.394 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:24.394 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.694454+0000", 2024-02-09T17:43:24.394 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8687000000000002e-05 2024-02-09T17:43:24.394 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.394 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.394 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:24.395 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.694581+0000", 2024-02-09T17:43:24.395 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012638600000000001 2024-02-09T17:43:24.395 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:24.395 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:24.395 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:24.395 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.395 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.395 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:266692 4.8 4:1a858ed9:::benchmark_data_smithi179_103809_object129226:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e451)", 2024-02-09T17:43:24.395 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:40:18.576239+0000", 2024-02-09T17:43:24.395 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 185.74898399400001, 2024-02-09T17:43:24.395 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.124045765, 2024-02-09T17:43:24.395 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:24.395 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:24.395 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:24.396 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:43:24.396 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:43:24.396 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 266692 2024-02-09T17:43:24.396 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.396 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:24.396 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.396 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:24.396 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.576239+0000", 2024-02-09T17:43:24.396 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:24.396 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.396 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.396 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:24.396 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.576239+0000", 2024-02-09T17:43:24.396 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:24.396 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.396 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.397 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:24.397 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.576240+0000", 2024-02-09T17:43:24.397 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6599999999999994e-07 2024-02-09T17:43:24.397 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.397 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.397 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:24.397 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.576245+0000", 2024-02-09T17:43:24.397 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4440000000000001e-06 2024-02-09T17:43:24.397 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.397 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.397 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:24.397 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.576256+0000", 2024-02-09T17:43:24.397 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1034e-05 2024-02-09T17:43:24.397 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.397 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.397 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:24.398 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.576266+0000", 2024-02-09T17:43:24.398 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0101999999999999e-05 2024-02-09T17:43:24.398 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.398 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.398 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:24.398 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.576323+0000", 2024-02-09T17:43:24.398 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7278000000000003e-05 2024-02-09T17:43:24.398 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.398 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.398 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:43:24.398 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.576335+0000", 2024-02-09T17:43:24.398 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2546e-05 2024-02-09T17:43:24.398 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.398 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.398 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:24.399 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.689604+0000", 2024-02-09T17:43:24.399 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1132686270000001 2024-02-09T17:43:24.399 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.399 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.399 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:24.399 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.689722+0000", 2024-02-09T17:43:24.399 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000118239 2024-02-09T17:43:24.399 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.399 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.399 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 6,7,9,10", 2024-02-09T17:43:24.399 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.689854+0000", 2024-02-09T17:43:24.399 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013162699999999999 2024-02-09T17:43:24.399 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.399 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.399 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.400 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.694341+0000", 2024-02-09T17:43:24.400 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0044874049999999999 2024-02-09T17:43:24.400 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.400 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.400 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.400 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.699864+0000", 2024-02-09T17:43:24.400 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0055223720000000002 2024-02-09T17:43:24.400 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.400 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.400 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:24.400 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.699884+0000", 2024-02-09T17:43:24.400 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0083999999999999e-05 2024-02-09T17:43:24.400 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.400 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.400 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.401 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.699910+0000", 2024-02-09T17:43:24.401 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5749e-05 2024-02-09T17:43:24.401 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.401 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.401 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.401 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.700059+0000", 2024-02-09T17:43:24.401 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014895899999999999 2024-02-09T17:43:24.401 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.401 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.401 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:24.401 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.700127+0000", 2024-02-09T17:43:24.401 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8801999999999996e-05 2024-02-09T17:43:24.401 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.401 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.401 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:24.402 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.700285+0000", 2024-02-09T17:43:24.402 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015753100000000001 2024-02-09T17:43:24.402 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:24.402 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:24.402 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:24.402 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.402 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.402 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:266705 4.8 4:1346114f:::benchmark_data_smithi179_103809_object129239:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e451)", 2024-02-09T17:43:24.402 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:40:18.604328+0000", 2024-02-09T17:43:24.402 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 185.72089556200001, 2024-02-09T17:43:24.402 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0962643729999999, 2024-02-09T17:43:24.402 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:24.402 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:24.402 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:24.403 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:43:24.403 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:43:24.403 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 266705 2024-02-09T17:43:24.403 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.403 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:24.403 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.403 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:24.403 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.604328+0000", 2024-02-09T17:43:24.403 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:24.403 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.403 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.403 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:24.403 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.604328+0000", 2024-02-09T17:43:24.403 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:24.404 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.404 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.404 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:24.404 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.604329+0000", 2024-02-09T17:43:24.404 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9209999999999999e-06 2024-02-09T17:43:24.404 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.404 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.404 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:24.404 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.604336+0000", 2024-02-09T17:43:24.404 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3570000000000003e-06 2024-02-09T17:43:24.404 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.404 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.404 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:24.404 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.604352+0000", 2024-02-09T17:43:24.405 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6648999999999999e-05 2024-02-09T17:43:24.405 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.405 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.405 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:24.405 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.604360+0000", 2024-02-09T17:43:24.405 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4699999999999996e-06 2024-02-09T17:43:24.405 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.405 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.405 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:24.405 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.604414+0000", 2024-02-09T17:43:24.405 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4215999999999997e-05 2024-02-09T17:43:24.405 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.405 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.405 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:43:24.405 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.604423+0000", 2024-02-09T17:43:24.405 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5459999999999992e-06 2024-02-09T17:43:24.406 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.406 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.406 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:24.406 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.694617+0000", 2024-02-09T17:43:24.406 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.090194154 2024-02-09T17:43:24.406 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.406 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.406 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:24.406 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.694724+0000", 2024-02-09T17:43:24.406 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010733 2024-02-09T17:43:24.406 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.406 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.406 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 6,7,9,10", 2024-02-09T17:43:24.406 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.694808+0000", 2024-02-09T17:43:24.406 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3861000000000002e-05 2024-02-09T17:43:24.407 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.407 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.407 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.407 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.699961+0000", 2024-02-09T17:43:24.407 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0051531679999999996 2024-02-09T17:43:24.407 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.407 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.407 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.407 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.700012+0000", 2024-02-09T17:43:24.407 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0408999999999998e-05 2024-02-09T17:43:24.407 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.407 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.407 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:24.407 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.700299+0000", 2024-02-09T17:43:24.408 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00028777799999999998 2024-02-09T17:43:24.408 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.408 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.408 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.408 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.700352+0000", 2024-02-09T17:43:24.408 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2658000000000003e-05 2024-02-09T17:43:24.408 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.408 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.408 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.408 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.700407+0000", 2024-02-09T17:43:24.408 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4619e-05 2024-02-09T17:43:24.408 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.408 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.409 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:24.409 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.700466+0000", 2024-02-09T17:43:24.409 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9398999999999998e-05 2024-02-09T17:43:24.409 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.409 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.409 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:24.409 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.700592+0000", 2024-02-09T17:43:24.409 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000125838 2024-02-09T17:43:24.409 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:24.409 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:24.409 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:24.409 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.409 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.409 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:266714 4.8 4:162dc382:::benchmark_data_smithi179_103809_object129248:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e451)", 2024-02-09T17:43:24.409 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:40:18.647646+0000", 2024-02-09T17:43:24.410 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 185.677576948, 2024-02-09T17:43:24.410 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0732599810000001, 2024-02-09T17:43:24.410 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:24.410 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:24.410 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:24.410 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:43:24.410 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:43:24.410 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 266714 2024-02-09T17:43:24.410 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.410 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:24.410 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.410 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:24.410 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.647646+0000", 2024-02-09T17:43:24.410 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:24.410 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.411 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.411 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:24.411 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.647646+0000", 2024-02-09T17:43:24.411 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:24.411 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.411 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.411 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:24.411 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.647648+0000", 2024-02-09T17:43:24.411 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.449e-06 2024-02-09T17:43:24.411 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.411 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.411 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:24.411 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.647654+0000", 2024-02-09T17:43:24.411 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.533e-06 2024-02-09T17:43:24.411 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.412 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.412 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:24.412 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.647668+0000", 2024-02-09T17:43:24.412 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3538e-05 2024-02-09T17:43:24.412 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.412 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.412 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:24.412 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.647675+0000", 2024-02-09T17:43:24.412 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7090000000000002e-06 2024-02-09T17:43:24.412 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.412 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.412 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:24.412 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.647724+0000", 2024-02-09T17:43:24.412 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8572000000000002e-05 2024-02-09T17:43:24.413 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.413 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.413 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:43:24.413 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.647732+0000", 2024-02-09T17:43:24.413 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3550000000000005e-06 2024-02-09T17:43:24.413 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.413 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.413 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:24.413 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.700632+0000", 2024-02-09T17:43:24.413 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.052899757 2024-02-09T17:43:24.413 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.413 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.413 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:24.413 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.700749+0000", 2024-02-09T17:43:24.413 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000117259 2024-02-09T17:43:24.414 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.414 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.414 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 6,7,9,10", 2024-02-09T17:43:24.414 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.700835+0000", 2024-02-09T17:43:24.414 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5675999999999998e-05 2024-02-09T17:43:24.414 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.414 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.414 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.414 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.710673+0000", 2024-02-09T17:43:24.414 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.009837742 2024-02-09T17:43:24.414 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.414 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.414 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.414 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.710727+0000", 2024-02-09T17:43:24.414 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4543999999999998e-05 2024-02-09T17:43:24.415 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.415 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.415 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:24.415 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.720616+0000", 2024-02-09T17:43:24.415 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.009888539 2024-02-09T17:43:24.415 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.415 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.415 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.415 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.720699+0000", 2024-02-09T17:43:24.415 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3088000000000001e-05 2024-02-09T17:43:24.415 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.415 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.415 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.415 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.720735+0000", 2024-02-09T17:43:24.415 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6433999999999997e-05 2024-02-09T17:43:24.416 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.416 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.416 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:24.416 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.720777+0000", 2024-02-09T17:43:24.416 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1643000000000001e-05 2024-02-09T17:43:24.416 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.416 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.416 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:24.416 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.720906+0000", 2024-02-09T17:43:24.416 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012914299999999999 2024-02-09T17:43:24.416 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:24.416 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:24.416 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:24.416 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.417 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.417 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:267356 4.8 4:11055ded:::benchmark_data_smithi179_103809_object129890:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e454)", 2024-02-09T17:43:24.417 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:40:21.484386+0000", 2024-02-09T17:43:24.417 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 182.840837276, 2024-02-09T17:43:24.417 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0488926890000001, 2024-02-09T17:43:24.417 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:24.417 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:24.417 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:24.417 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:43:24.417 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:43:24.417 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 267356 2024-02-09T17:43:24.417 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.417 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:24.417 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.417 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:24.418 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.484386+0000", 2024-02-09T17:43:24.418 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:24.418 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.418 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.418 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:24.418 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.484386+0000", 2024-02-09T17:43:24.418 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:24.418 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.418 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.418 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:24.418 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.484387+0000", 2024-02-09T17:43:24.418 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.387e-06 2024-02-09T17:43:24.418 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.418 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.418 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:24.419 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.484392+0000", 2024-02-09T17:43:24.419 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6210000000000003e-06 2024-02-09T17:43:24.419 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.419 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.419 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:24.419 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.484406+0000", 2024-02-09T17:43:24.419 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4039999999999999e-05 2024-02-09T17:43:24.419 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.419 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.419 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:24.419 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.484415+0000", 2024-02-09T17:43:24.419 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.0310000000000004e-06 2024-02-09T17:43:24.419 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.419 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.419 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:24.420 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.484468+0000", 2024-02-09T17:43:24.420 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3177999999999998e-05 2024-02-09T17:43:24.420 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.420 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.420 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:43:24.420 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.484477+0000", 2024-02-09T17:43:24.420 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5080000000000005e-06 2024-02-09T17:43:24.420 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.420 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.420 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:24.420 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.520832+0000", 2024-02-09T17:43:24.420 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.036355763 2024-02-09T17:43:24.420 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.420 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.421 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:24.421 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.520952+0000", 2024-02-09T17:43:24.421 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000119741 2024-02-09T17:43:24.421 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.421 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.421 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 6,7", 2024-02-09T17:43:24.421 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.521079+0000", 2024-02-09T17:43:24.421 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012680800000000001 2024-02-09T17:43:24.421 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.421 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.421 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.421 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.528334+0000", 2024-02-09T17:43:24.421 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0072548159999999999 2024-02-09T17:43:24.421 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.421 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.422 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:24.422 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.530707+0000", 2024-02-09T17:43:24.422 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0023729480000000002 2024-02-09T17:43:24.422 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.422 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.422 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.422 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.533182+0000", 2024-02-09T17:43:24.422 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002475582 2024-02-09T17:43:24.422 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.422 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.422 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:24.422 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.533236+0000", 2024-02-09T17:43:24.422 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3777999999999999e-05 2024-02-09T17:43:24.422 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.422 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.423 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:24.423 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.533279+0000", 2024-02-09T17:43:24.423 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2487999999999997e-05 2024-02-09T17:43:24.423 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:24.423 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:24.423 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:24.423 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.423 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.423 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:267373 4.8 4:12172885:::benchmark_data_smithi179_103809_object129907:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e454)", 2024-02-09T17:43:24.423 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:40:21.504773+0000", 2024-02-09T17:43:24.423 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 182.820449782, 2024-02-09T17:43:24.423 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.036471082, 2024-02-09T17:43:24.423 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:24.423 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:24.424 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:24.424 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:43:24.424 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:43:24.424 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 267373 2024-02-09T17:43:24.424 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.424 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:24.424 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.424 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:24.424 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.504773+0000", 2024-02-09T17:43:24.424 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:24.424 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.424 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.424 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:24.424 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.504773+0000", 2024-02-09T17:43:24.428 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:24.428 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.428 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.428 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:24.428 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.504776+0000", 2024-02-09T17:43:24.428 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3539999999999998e-06 2024-02-09T17:43:24.428 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.428 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.428 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:24.428 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.504784+0000", 2024-02-09T17:43:24.428 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3110000000000006e-06 2024-02-09T17:43:24.428 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.428 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.429 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:24.429 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.504804+0000", 2024-02-09T17:43:24.429 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9947999999999999e-05 2024-02-09T17:43:24.429 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.429 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.429 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:24.429 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.504814+0000", 2024-02-09T17:43:24.429 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7229999999999999e-06 2024-02-09T17:43:24.429 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.429 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.429 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:24.429 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.504878+0000", 2024-02-09T17:43:24.429 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3927e-05 2024-02-09T17:43:24.429 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.429 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.430 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:43:24.430 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.504887+0000", 2024-02-09T17:43:24.430 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5189999999999998e-06 2024-02-09T17:43:24.430 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.430 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.430 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:24.430 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.525454+0000", 2024-02-09T17:43:24.430 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0205668999999999 2024-02-09T17:43:24.430 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.430 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.430 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:24.430 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.525531+0000", 2024-02-09T17:43:24.430 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6773000000000005e-05 2024-02-09T17:43:24.430 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.430 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.431 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 6,7", 2024-02-09T17:43:24.431 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.525606+0000", 2024-02-09T17:43:24.431 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5585000000000002e-05 2024-02-09T17:43:24.431 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.431 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.431 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.431 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.533319+0000", 2024-02-09T17:43:24.431 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0077124380000000003 2024-02-09T17:43:24.431 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.431 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.431 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:24.431 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.537161+0000", 2024-02-09T17:43:24.431 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0038420669999999998 2024-02-09T17:43:24.431 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.431 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.432 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.432 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.541171+0000", 2024-02-09T17:43:24.432 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0040103819999999998 2024-02-09T17:43:24.432 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.432 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.432 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:24.432 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.541208+0000", 2024-02-09T17:43:24.432 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6653999999999998e-05 2024-02-09T17:43:24.432 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.432 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.432 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:24.432 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.541244+0000", 2024-02-09T17:43:24.432 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6501000000000002e-05 2024-02-09T17:43:24.432 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:24.433 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:24.433 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:24.433 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.433 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.433 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:267374 4.8 4:1d92d63e:::benchmark_data_smithi179_103809_object129908:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e454)", 2024-02-09T17:43:24.433 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:40:21.506869+0000", 2024-02-09T17:43:24.433 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 182.81835405000001, 2024-02-09T17:43:24.433 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0345589079999999, 2024-02-09T17:43:24.433 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:24.433 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:24.433 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:24.433 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:43:24.433 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:43:24.433 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 267374 2024-02-09T17:43:24.434 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.434 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:24.434 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.434 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:24.434 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.506869+0000", 2024-02-09T17:43:24.434 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:24.434 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.434 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.434 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:24.434 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.506869+0000", 2024-02-09T17:43:24.434 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:24.434 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.434 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.434 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:24.434 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.506870+0000", 2024-02-09T17:43:24.435 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6599999999999994e-07 2024-02-09T17:43:24.435 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.435 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.435 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:24.435 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.506875+0000", 2024-02-09T17:43:24.435 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8849999999999998e-06 2024-02-09T17:43:24.435 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.435 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.435 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:24.435 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.506885+0000", 2024-02-09T17:43:24.435 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0281e-05 2024-02-09T17:43:24.435 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.435 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.435 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:24.435 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.506892+0000", 2024-02-09T17:43:24.435 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.0169999999999999e-06 2024-02-09T17:43:24.436 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.436 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.436 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:24.436 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.506940+0000", 2024-02-09T17:43:24.436 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8210000000000001e-05 2024-02-09T17:43:24.436 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.436 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.436 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:43:24.436 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.506947+0000", 2024-02-09T17:43:24.436 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5479999999999998e-06 2024-02-09T17:43:24.436 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.436 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.436 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:24.436 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.528366+0000", 2024-02-09T17:43:24.437 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0214189419999999 2024-02-09T17:43:24.437 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.437 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.437 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:24.437 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.528448+0000", 2024-02-09T17:43:24.437 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2445000000000001e-05 2024-02-09T17:43:24.437 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.437 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.437 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 6,7", 2024-02-09T17:43:24.437 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.528521+0000", 2024-02-09T17:43:24.437 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2743999999999993e-05 2024-02-09T17:43:24.437 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.437 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.437 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:24.437 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.537168+0000", 2024-02-09T17:43:24.438 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0086471280000000005 2024-02-09T17:43:24.438 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.438 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.438 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.438 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.541285+0000", 2024-02-09T17:43:24.438 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0041165619999999998 2024-02-09T17:43:24.438 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.438 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.438 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.438 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.541344+0000", 2024-02-09T17:43:24.438 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9629000000000001e-05 2024-02-09T17:43:24.438 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.438 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.438 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:24.438 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.541390+0000", 2024-02-09T17:43:24.439 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5392999999999997e-05 2024-02-09T17:43:24.439 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.439 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.439 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:24.439 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.541428+0000", 2024-02-09T17:43:24.439 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8158000000000003e-05 2024-02-09T17:43:24.439 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:24.439 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:24.439 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:24.439 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.439 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.439 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:267386 4.8 4:1caee092:::benchmark_data_smithi179_103809_object129920:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e454)", 2024-02-09T17:43:24.439 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:40:21.520850+0000", 2024-02-09T17:43:24.439 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 182.80437324499999, 2024-02-09T17:43:24.439 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.020772193, 2024-02-09T17:43:24.440 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:24.440 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:24.440 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:24.440 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:43:24.440 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:43:24.440 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 267386 2024-02-09T17:43:24.440 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.440 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:24.440 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.440 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:24.440 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.520850+0000", 2024-02-09T17:43:24.440 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:24.440 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.440 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.441 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:24.441 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.520850+0000", 2024-02-09T17:43:24.441 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:24.441 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.441 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.441 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:24.441 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.520852+0000", 2024-02-09T17:43:24.441 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9290000000000001e-06 2024-02-09T17:43:24.441 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.441 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.441 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:24.441 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.520859+0000", 2024-02-09T17:43:24.441 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.0940000000000002e-06 2024-02-09T17:43:24.441 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.441 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.441 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:24.442 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.520875+0000", 2024-02-09T17:43:24.442 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5865000000000001e-05 2024-02-09T17:43:24.442 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.442 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.442 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:24.442 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.520884+0000", 2024-02-09T17:43:24.442 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.73e-06 2024-02-09T17:43:24.442 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.442 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.442 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:24.442 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.520948+0000", 2024-02-09T17:43:24.442 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3578999999999996e-05 2024-02-09T17:43:24.442 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.442 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.443 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:43:24.443 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.520960+0000", 2024-02-09T17:43:24.443 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1618e-05 2024-02-09T17:43:24.443 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.443 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.443 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:24.443 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.530725+0000", 2024-02-09T17:43:24.443 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0097657390000001 2024-02-09T17:43:24.443 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.443 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.443 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:24.443 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.530785+0000", 2024-02-09T17:43:24.443 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9923000000000002e-05 2024-02-09T17:43:24.443 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.443 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.443 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 6,7", 2024-02-09T17:43:24.444 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.530848+0000", 2024-02-09T17:43:24.444 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2883000000000001e-05 2024-02-09T17:43:24.444 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.444 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.444 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:24.444 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.541259+0000", 2024-02-09T17:43:24.444 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010411023 2024-02-09T17:43:24.444 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.444 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.444 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.444 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.541494+0000", 2024-02-09T17:43:24.444 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00023479600000000001 2024-02-09T17:43:24.444 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.444 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.445 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.445 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.541541+0000", 2024-02-09T17:43:24.445 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7011000000000001e-05 2024-02-09T17:43:24.445 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.445 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.445 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:24.445 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.541588+0000", 2024-02-09T17:43:24.445 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6910000000000003e-05 2024-02-09T17:43:24.445 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.445 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.445 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:24.445 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.541622+0000", 2024-02-09T17:43:24.445 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4093e-05 2024-02-09T17:43:24.445 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:24.445 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:24.446 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:24.446 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.446 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.446 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:106985 5.8 5:10c6e4f9:::benchmark_data_smithi179_115199_object106984:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-02-09T17:43:24.446 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:42:12.106360+0000", 2024-02-09T17:43:24.446 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 72.218863392000003, 2024-02-09T17:43:24.446 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.21701166199999999, 2024-02-09T17:43:24.446 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:24.446 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:24.446 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:24.446 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:43:24.446 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:43:24.446 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 106985 2024-02-09T17:43:24.446 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.447 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:24.447 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.447 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:24.447 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.106360+0000", 2024-02-09T17:43:24.447 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:24.447 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.447 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.447 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:24.447 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.106360+0000", 2024-02-09T17:43:24.447 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:24.447 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.447 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.447 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:24.447 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.106361+0000", 2024-02-09T17:43:24.447 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3179999999999999e-06 2024-02-09T17:43:24.447 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.448 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.448 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:24.448 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.106388+0000", 2024-02-09T17:43:24.448 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6942999999999999e-05 2024-02-09T17:43:24.448 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.448 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.448 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:24.448 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.106399+0000", 2024-02-09T17:43:24.448 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1046999999999999e-05 2024-02-09T17:43:24.448 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.448 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.448 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:24.448 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.106406+0000", 2024-02-09T17:43:24.448 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4429999999999997e-06 2024-02-09T17:43:24.449 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.449 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.449 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:24.449 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.106485+0000", 2024-02-09T17:43:24.449 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8347000000000001e-05 2024-02-09T17:43:24.449 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.449 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.449 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:24.449 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.106623+0000", 2024-02-09T17:43:24.449 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013791399999999999 2024-02-09T17:43:24.449 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.449 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.449 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,10", 2024-02-09T17:43:24.449 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.106755+0000", 2024-02-09T17:43:24.450 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000132743 2024-02-09T17:43:24.450 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.450 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.450 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:24.450 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.114193+0000", 2024-02-09T17:43:24.450 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0074372270000000002 2024-02-09T17:43:24.450 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.450 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.450 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.450 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.117105+0000", 2024-02-09T17:43:24.450 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002912591 2024-02-09T17:43:24.450 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.450 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.450 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.451 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.323273+0000", 2024-02-09T17:43:24.451 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.20616793999999999 2024-02-09T17:43:24.451 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.451 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.451 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:24.451 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.323329+0000", 2024-02-09T17:43:24.451 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5958999999999997e-05 2024-02-09T17:43:24.451 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.451 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.451 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:24.451 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.323371+0000", 2024-02-09T17:43:24.451 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2190000000000001e-05 2024-02-09T17:43:24.451 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:24.451 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:24.451 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:24.452 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.452 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.452 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:106986 5.8 5:150a5124:::benchmark_data_smithi179_115199_object106985:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-02-09T17:43:24.452 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:42:12.106436+0000", 2024-02-09T17:43:24.452 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 72.218786890999993, 2024-02-09T17:43:24.452 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.217089212, 2024-02-09T17:43:24.452 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:24.452 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:24.452 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:24.452 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:43:24.452 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:43:24.452 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 106986 2024-02-09T17:43:24.452 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.453 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:24.453 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.453 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:24.453 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.106436+0000", 2024-02-09T17:43:24.453 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:24.453 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.453 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.453 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:24.453 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.106436+0000", 2024-02-09T17:43:24.453 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:24.453 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.453 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.453 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:24.453 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.106437+0000", 2024-02-09T17:43:24.453 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4700000000000002e-07 2024-02-09T17:43:24.454 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.454 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.454 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:24.454 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.106455+0000", 2024-02-09T17:43:24.454 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8627999999999999e-05 2024-02-09T17:43:24.454 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.454 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.454 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:24.454 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.106463+0000", 2024-02-09T17:43:24.454 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.909e-06 2024-02-09T17:43:24.454 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.454 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.454 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:24.454 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.106469+0000", 2024-02-09T17:43:24.455 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7150000000000003e-06 2024-02-09T17:43:24.455 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.455 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.455 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:24.455 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.110559+0000", 2024-02-09T17:43:24.455 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0040905079999999996 2024-02-09T17:43:24.455 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.455 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.455 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:24.455 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.110698+0000", 2024-02-09T17:43:24.455 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013882099999999999 2024-02-09T17:43:24.455 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.455 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.455 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,10", 2024-02-09T17:43:24.456 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.110827+0000", 2024-02-09T17:43:24.456 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000129202 2024-02-09T17:43:24.456 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.456 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.456 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:24.456 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.117481+0000", 2024-02-09T17:43:24.456 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.006653155 2024-02-09T17:43:24.456 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.456 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.456 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.456 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.119353+0000", 2024-02-09T17:43:24.456 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0018721499999999999 2024-02-09T17:43:24.456 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.456 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.456 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.457 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.323420+0000", 2024-02-09T17:43:24.457 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.204067464 2024-02-09T17:43:24.457 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.457 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.457 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:24.457 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.323482+0000", 2024-02-09T17:43:24.457 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2100000000000005e-05 2024-02-09T17:43:24.457 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.457 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.457 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:24.457 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.323525+0000", 2024-02-09T17:43:24.457 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3112999999999999e-05 2024-02-09T17:43:24.457 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:24.457 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:24.457 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:24.458 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.458 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.458 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:106996 5.8 5:13d7f260:::benchmark_data_smithi179_115199_object106995:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-02-09T17:43:24.458 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:42:12.111260+0000", 2024-02-09T17:43:24.458 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 72.213962898999995, 2024-02-09T17:43:24.458 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.21567955999999999, 2024-02-09T17:43:24.458 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:24.458 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:24.458 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:24.458 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:43:24.458 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:43:24.458 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 106996 2024-02-09T17:43:24.458 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.458 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:24.458 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.459 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:24.459 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.111260+0000", 2024-02-09T17:43:24.459 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:24.459 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.459 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.459 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:24.459 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.111260+0000", 2024-02-09T17:43:24.459 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:24.459 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.459 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.459 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:24.459 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.111262+0000", 2024-02-09T17:43:24.459 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5290000000000001e-06 2024-02-09T17:43:24.459 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.460 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.460 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:24.460 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.111294+0000", 2024-02-09T17:43:24.460 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1847999999999997e-05 2024-02-09T17:43:24.460 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.460 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.460 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:24.460 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.111307+0000", 2024-02-09T17:43:24.460 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3039e-05 2024-02-09T17:43:24.460 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.460 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.460 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:24.460 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.111315+0000", 2024-02-09T17:43:24.460 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8359999999999998e-06 2024-02-09T17:43:24.460 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.461 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.461 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:24.461 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.114226+0000", 2024-02-09T17:43:24.461 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0029104320000000001 2024-02-09T17:43:24.461 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.461 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.461 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:24.461 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.114327+0000", 2024-02-09T17:43:24.461 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000100912 2024-02-09T17:43:24.461 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.461 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.461 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,10", 2024-02-09T17:43:24.461 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.114412+0000", 2024-02-09T17:43:24.461 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5678000000000002e-05 2024-02-09T17:43:24.461 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.462 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.462 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:24.462 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.117486+0000", 2024-02-09T17:43:24.462 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0030737569999999999 2024-02-09T17:43:24.462 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.462 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.462 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.462 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.119386+0000", 2024-02-09T17:43:24.462 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0018997199999999999 2024-02-09T17:43:24.462 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.462 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.462 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.462 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.326833+0000", 2024-02-09T17:43:24.463 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.207446662 2024-02-09T17:43:24.463 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.463 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.463 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:24.463 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.326899+0000", 2024-02-09T17:43:24.463 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6000000000000005e-05 2024-02-09T17:43:24.463 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.463 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.463 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:24.463 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.326940+0000", 2024-02-09T17:43:24.463 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1146999999999998e-05 2024-02-09T17:43:24.463 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:24.463 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:24.463 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:24.464 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.464 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.464 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:126288 5.d 5:b690dee0:::benchmark_data_smithi179_115199_object126287:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e501)", 2024-02-09T17:43:24.464 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:42:31.632319+0000", 2024-02-09T17:43:24.464 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 52.692904089999999, 2024-02-09T17:43:24.464 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.22240631899999999, 2024-02-09T17:43:24.464 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:24.464 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:24.464 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:24.464 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:43:24.464 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:43:24.464 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 126288 2024-02-09T17:43:24.464 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.464 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:24.464 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.465 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:24.465 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:31.632319+0000", 2024-02-09T17:43:24.465 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:24.465 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.465 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.465 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:24.465 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:31.632319+0000", 2024-02-09T17:43:24.465 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:24.465 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.465 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.465 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:24.465 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:31.632321+0000", 2024-02-09T17:43:24.465 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.731e-06 2024-02-09T17:43:24.465 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.465 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.466 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:24.466 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:31.632368+0000", 2024-02-09T17:43:24.466 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7756999999999997e-05 2024-02-09T17:43:24.466 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.466 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.466 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:24.466 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:31.632385+0000", 2024-02-09T17:43:24.466 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6109000000000001e-05 2024-02-09T17:43:24.466 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.466 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.466 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:24.466 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:31.632393+0000", 2024-02-09T17:43:24.466 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7709999999999995e-06 2024-02-09T17:43:24.466 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.467 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.467 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:24.467 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:31.632459+0000", 2024-02-09T17:43:24.467 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6130999999999994e-05 2024-02-09T17:43:24.467 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.467 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.467 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:24.467 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:31.632587+0000", 2024-02-09T17:43:24.467 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000127995 2024-02-09T17:43:24.467 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.467 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.467 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,7", 2024-02-09T17:43:24.467 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:31.632695+0000", 2024-02-09T17:43:24.467 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000107111 2024-02-09T17:43:24.467 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.468 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.468 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:24.468 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:31.638171+0000", 2024-02-09T17:43:24.468 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0054763989999999999 2024-02-09T17:43:24.468 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.468 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.468 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.468 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:31.639611+0000", 2024-02-09T17:43:24.468 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001439863 2024-02-09T17:43:24.468 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.468 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.468 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:24.468 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:31.854656+0000", 2024-02-09T17:43:24.468 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.21504557699999999 2024-02-09T17:43:24.469 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.469 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.469 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:24.469 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:31.854700+0000", 2024-02-09T17:43:24.469 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3149000000000003e-05 2024-02-09T17:43:24.469 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:24.469 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:24.469 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:24.469 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:31.854725+0000", 2024-02-09T17:43:24.469 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5726000000000001e-05 2024-02-09T17:43:24.469 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:24.469 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:24.469 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:24.469 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:24.469 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:24.470 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:43:24.470 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:24 smithi162 ceph-mon[44304]: pgmap v1319: 47 pgs: 1 active+clean+scrubbing+deep, 1 unknown, 45 active+clean; 5.5 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 1.5 KiB/s rd, 33 KiB/s wr, 655 op/s; 0 B/s, 0 objects/s recovering 2024-02-09T17:43:24.470 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:24 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2242264481' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-02-09T17:43:24.471 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:24 smithi099 ceph-mon[39420]: pgmap v1319: 47 pgs: 1 active+clean+scrubbing+deep, 1 unknown, 45 active+clean; 5.5 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 1.5 KiB/s rd, 33 KiB/s wr, 655 op/s; 0 B/s, 0 objects/s recovering 2024-02-09T17:43:24.471 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:24 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2242264481' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-02-09T17:43:24.471 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:24 smithi179 ceph-mon[43975]: pgmap v1319: 47 pgs: 1 active+clean+scrubbing+deep, 1 unknown, 45 active+clean; 5.5 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 1.5 KiB/s rd, 33 KiB/s wr, 655 op/s; 0 B/s, 0 objects/s recovering 2024-02-09T17:43:24.471 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:24 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2242264481' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-02-09T17:43:24.895 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:43:24.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:24.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:43:24.999 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:43:25.061 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.8 dump_ops_in_flight 2024-02-09T17:43:25.070 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:25.090 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:43:25.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:25.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:43:25.167 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is set 2024-02-09T17:43:25.193 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:43:25.224 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:25.325 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:43:25.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:25 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2242264481' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-02-09T17:43:25.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:25 smithi179 ceph-mon[43975]: osdmap e521: 12 total, 12 up, 12 in 2024-02-09T17:43:25.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:25 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3132648354' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:43:25.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:25 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2242264481' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-02-09T17:43:25.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:25 smithi162 ceph-mon[44304]: osdmap e521: 12 total, 12 up, 12 in 2024-02-09T17:43:25.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:25 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3132648354' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:43:25.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:25 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2242264481' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-02-09T17:43:25.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:25 smithi099 ceph-mon[39420]: osdmap e521: 12 total, 12 up, 12 in 2024-02-09T17:43:25.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:25 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3132648354' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:43:26.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:26 smithi099 ceph-mon[39420]: pgmap v1321: 47 pgs: 3 peering, 1 active+clean+scrubbing+deep, 43 active+clean; 5.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 454 B/s rd, 9.9 KiB/s wr, 630 op/s; 14 KiB/s, 0 objects/s recovering 2024-02-09T17:43:26.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:26 smithi099 ceph-mon[39420]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:43:26.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:26 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3132648354' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:43:26.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:26 smithi099 ceph-mon[39420]: osdmap e522: 12 total, 12 up, 12 in 2024-02-09T17:43:26.379 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:43:26.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:26.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:43:26.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:26 smithi179 ceph-mon[43975]: pgmap v1321: 47 pgs: 3 peering, 1 active+clean+scrubbing+deep, 43 active+clean; 5.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 454 B/s rd, 9.9 KiB/s wr, 630 op/s; 14 KiB/s, 0 objects/s recovering 2024-02-09T17:43:26.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:26 smithi179 ceph-mon[43975]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:43:26.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:26 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3132648354' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:43:26.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:26 smithi179 ceph-mon[43975]: osdmap e522: 12 total, 12 up, 12 in 2024-02-09T17:43:26.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:26 smithi162 ceph-mon[44304]: pgmap v1321: 47 pgs: 3 peering, 1 active+clean+scrubbing+deep, 43 active+clean; 5.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 454 B/s rd, 9.9 KiB/s wr, 630 op/s; 14 KiB/s, 0 objects/s recovering 2024-02-09T17:43:26.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:26 smithi162 ceph-mon[44304]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:43:26.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:26 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3132648354' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:43:26.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:26 smithi162 ceph-mon[44304]: osdmap e522: 12 total, 12 up, 12 in 2024-02-09T17:43:26.484 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:43:26.516 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:26.537 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:43:26.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:26.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:43:26.642 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:43:26.671 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:26.679 INFO:tasks.thrashosds.thrasher:Added osd 2 2024-02-09T17:43:27.120 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:43:27.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:27.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:43:27.225 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:43:27.272 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:27.304 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:43:27.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:27.311 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:43:27.379 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:27 smithi099 ceph-mon[39420]: osdmap e523: 12 total, 12 up, 12 in 2024-02-09T17:43:27.379 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:27 smithi099 ceph-mon[39420]: 5.6 starting backfill to osd.2 from (0'0,0'0] MAX to 520'10507 2024-02-09T17:43:27.379 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:27 smithi099 ceph-mon[39420]: 5.4 starting backfill to osd.2 from (0'0,0'0] MAX to 520'10428 2024-02-09T17:43:27.379 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:27 smithi099 ceph-mon[39420]: 5.a starting backfill to osd.2 from (0'0,0'0] MAX to 520'10584 2024-02-09T17:43:27.379 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:27 smithi099 ceph-mon[39420]: osdmap e524: 12 total, 12 up, 12 in 2024-02-09T17:43:27.412 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:43:27.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:27 smithi179 ceph-mon[43975]: osdmap e523: 12 total, 12 up, 12 in 2024-02-09T17:43:27.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:27 smithi179 ceph-mon[43975]: 5.6 starting backfill to osd.2 from (0'0,0'0] MAX to 520'10507 2024-02-09T17:43:27.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:27 smithi179 ceph-mon[43975]: 5.4 starting backfill to osd.2 from (0'0,0'0] MAX to 520'10428 2024-02-09T17:43:27.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:27 smithi179 ceph-mon[43975]: 5.a starting backfill to osd.2 from (0'0,0'0] MAX to 520'10584 2024-02-09T17:43:27.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:27 smithi179 ceph-mon[43975]: osdmap e524: 12 total, 12 up, 12 in 2024-02-09T17:43:27.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:27 smithi162 ceph-mon[44304]: osdmap e523: 12 total, 12 up, 12 in 2024-02-09T17:43:27.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:27 smithi162 ceph-mon[44304]: 5.6 starting backfill to osd.2 from (0'0,0'0] MAX to 520'10507 2024-02-09T17:43:27.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:27 smithi162 ceph-mon[44304]: 5.4 starting backfill to osd.2 from (0'0,0'0] MAX to 520'10428 2024-02-09T17:43:27.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:27 smithi162 ceph-mon[44304]: 5.a starting backfill to osd.2 from (0'0,0'0] MAX to 520'10584 2024-02-09T17:43:27.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:27 smithi162 ceph-mon[44304]: osdmap e524: 12 total, 12 up, 12 in 2024-02-09T17:43:27.467 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:27.517 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:43:27.529 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:27.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:43:27.630 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:43:27.679 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:27.722 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:43:27.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:27.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:43:27.827 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:43:27.886 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:27.940 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:43:27.972 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:43:27.975 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:43:27.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:27.979 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:43:28.002 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:43:28.034 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:43:28.068 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:43:28.080 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:43:28.096 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:43:28.127 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:43:28.141 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:28.155 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:43:28.155 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-02-09T17:43:28.155 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 0 2024-02-09T17:43:28.155 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:43:28.166 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:43:28.193 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:43:28.194 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:43:28.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:28.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:43:28.219 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:43:28.249 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:43:28.289 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:43:28.293 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:43:28.293 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:43:28.294 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:43:28.294 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:43:28.294 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:43:28.294 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:43:28.294 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:43:28.294 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:43:28.294 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:43:28.294 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:43:28.294 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:43:28.294 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:43:28.303 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:43:28.364 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:28.402 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:43:28.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:28.409 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:28 smithi162 ceph-mon[44304]: pgmap v1325: 47 pgs: 1 active+clean+scrubbing+deep, 3 peering, 43 active+clean; 5.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 557 op/s 2024-02-09T17:43:28.409 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:28 smithi162 ceph-mon[44304]: osdmap e525: 12 total, 12 up, 12 in 2024-02-09T17:43:28.409 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:28 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/119409749' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:43:28.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:43:28.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:28 smithi179 ceph-mon[43975]: pgmap v1325: 47 pgs: 1 active+clean+scrubbing+deep, 3 peering, 43 active+clean; 5.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 557 op/s 2024-02-09T17:43:28.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:28 smithi179 ceph-mon[43975]: osdmap e525: 12 total, 12 up, 12 in 2024-02-09T17:43:28.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:28 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/119409749' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:43:28.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:28 smithi099 ceph-mon[39420]: pgmap v1325: 47 pgs: 1 active+clean+scrubbing+deep, 3 peering, 43 active+clean; 5.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 557 op/s 2024-02-09T17:43:28.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:28 smithi099 ceph-mon[39420]: osdmap e525: 12 total, 12 up, 12 in 2024-02-09T17:43:28.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:28 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/119409749' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:43:28.511 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:43:28.554 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:29.098 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset noscrub 2024-02-09T17:43:29.118 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:43:29.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:29.126 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:43:29.227 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:43:29.281 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:29.319 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:43:29.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:29.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:43:29.422 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:43:29.466 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:29.502 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:43:29.505 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:43:29.509 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:29.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:43:29.610 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:43:29.664 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:29.700 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:43:29.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:29.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:43:29.744 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.8 dump_blocked_ops 2024-02-09T17:43:29.806 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:43:29.841 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:29 smithi099 ceph-mon[39420]: osdmap e526: 12 total, 12 up, 12 in 2024-02-09T17:43:29.841 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:29 smithi099 ceph-mon[39420]: pgmap v1328: 47 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+undersized+remapped, 1 active+clean+scrubbing+deep, 4 peering, 39 active+clean; 5.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 385 op/s; 10525/262071 objects misplaced (4.016%); 3.2 MiB/s, 50 objects/s recovering 2024-02-09T17:43:29.842 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:29.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:29 smithi162 ceph-mon[44304]: osdmap e526: 12 total, 12 up, 12 in 2024-02-09T17:43:29.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:29 smithi162 ceph-mon[44304]: pgmap v1328: 47 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+undersized+remapped, 1 active+clean+scrubbing+deep, 4 peering, 39 active+clean; 5.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 385 op/s; 10525/262071 objects misplaced (4.016%); 3.2 MiB/s, 50 objects/s recovering 2024-02-09T17:43:29.927 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:29 smithi179 ceph-mon[43975]: osdmap e526: 12 total, 12 up, 12 in 2024-02-09T17:43:29.927 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:29 smithi179 ceph-mon[43975]: pgmap v1328: 47 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+undersized+remapped, 1 active+clean+scrubbing+deep, 4 peering, 39 active+clean; 5.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 385 op/s; 10525/262071 objects misplaced (4.016%); 3.2 MiB/s, 50 objects/s recovering 2024-02-09T17:43:30.208 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:43:30.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:30 smithi179 ceph-mon[43975]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:43:30.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:30 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:43:30.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:30 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:43:30.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:30 smithi162 ceph-mon[44304]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:43:30.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:30 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:43:30.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:30 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:43:30.987 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:30 smithi099 ceph-mon[39420]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:43:30.987 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:30 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:43:30.987 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:30 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:43:30.999 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:43:31.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:31.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:43:31.109 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:43:31.162 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:31.238 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:43:31.612 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:31 smithi099 ceph-mon[39420]: osdmap e527: 12 total, 12 up, 12 in 2024-02-09T17:43:31.612 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:31 smithi099 ceph-mon[39420]: pgmap v1330: 47 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+undersized+remapped, 1 active+clean+scrubbing+deep, 1 peering, 41 active+clean; 5.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 715 B/s rd, 21 KiB/s wr, 481 op/s; 15501/257889 objects misplaced (6.011%); 7.3 MiB/s, 116 objects/s recovering 2024-02-09T17:43:31.612 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:31 smithi099 ceph-mon[39420]: osdmap e528: 12 total, 12 up, 12 in 2024-02-09T17:43:31.680 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] 2024-02-09T17:43:31.680 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:43:31.680 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-02-09T17:43:31.680 INFO:tasks.thrashosds.thrasher:Setting osd 7 primary_affinity to 0.598263 2024-02-09T17:43:31.680 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd primary-affinity 7 0.5982631885847566 2024-02-09T17:43:31.730 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:43:31.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:31.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:43:31.839 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:43:31.886 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:31 smithi179 ceph-mon[43975]: osdmap e527: 12 total, 12 up, 12 in 2024-02-09T17:43:31.886 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:31 smithi179 ceph-mon[43975]: pgmap v1330: 47 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+undersized+remapped, 1 active+clean+scrubbing+deep, 1 peering, 41 active+clean; 5.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 715 B/s rd, 21 KiB/s wr, 481 op/s; 15501/257889 objects misplaced (6.011%); 7.3 MiB/s, 116 objects/s recovering 2024-02-09T17:43:31.886 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:31 smithi179 ceph-mon[43975]: osdmap e528: 12 total, 12 up, 12 in 2024-02-09T17:43:31.889 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:31.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:31 smithi162 ceph-mon[44304]: osdmap e527: 12 total, 12 up, 12 in 2024-02-09T17:43:31.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:31 smithi162 ceph-mon[44304]: pgmap v1330: 47 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+undersized+remapped, 1 active+clean+scrubbing+deep, 1 peering, 41 active+clean; 5.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 715 B/s rd, 21 KiB/s wr, 481 op/s; 15501/257889 objects misplaced (6.011%); 7.3 MiB/s, 116 objects/s recovering 2024-02-09T17:43:31.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:31 smithi162 ceph-mon[44304]: osdmap e528: 12 total, 12 up, 12 in 2024-02-09T17:43:32.342 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:43:32.383 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:43:32.857 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:43:32.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:32.863 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:43:32.964 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:43:33.022 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:33.287 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:43:33.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:33.295 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:43:33.396 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:43:33.446 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:33.514 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:43:33.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:33.520 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:43:33.623 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:43:33.660 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:33.696 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T17:43:33.696 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:43:33.909 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:33 smithi099 ceph-mon[39420]: pgmap v1332: 47 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 1 peering, 43 active+clean; 5.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 583 B/s rd, 17 KiB/s wr, 575 op/s; 15274/253692 objects misplaced (6.021%); 6.0 MiB/s, 94 objects/s recovering 2024-02-09T17:43:33.909 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:33 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/261373219' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:43:33.909 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:33 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:43:34.046 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:33 smithi179 ceph-mon[43975]: pgmap v1332: 47 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 1 peering, 43 active+clean; 5.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 583 B/s rd, 17 KiB/s wr, 575 op/s; 15274/253692 objects misplaced (6.021%); 6.0 MiB/s, 94 objects/s recovering 2024-02-09T17:43:34.047 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:33 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/261373219' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:43:34.047 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:33 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:43:34.119 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:43:34.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:34.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:43:34.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:33 smithi162 ceph-mon[44304]: pgmap v1332: 47 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 1 peering, 43 active+clean; 5.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 583 B/s rd, 17 KiB/s wr, 575 op/s; 15274/253692 objects misplaced (6.021%); 6.0 MiB/s, 94 objects/s recovering 2024-02-09T17:43:34.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:33 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/261373219' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:43:34.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:33 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:43:34.223 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:43:34.268 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:34.620 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:43:34.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:34.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:43:34.675 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is unset 2024-02-09T17:43:34.734 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:43:34.811 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:34.914 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:34 smithi099 ceph-mon[39420]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:43:34.914 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:34 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:43:34.914 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:34 smithi099 ceph-mon[39420]: osdmap e529: 12 total, 12 up, 12 in 2024-02-09T17:43:34.914 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:34 smithi162 ceph-mon[44304]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:43:34.914 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:34 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:43:34.914 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:34 smithi162 ceph-mon[44304]: osdmap e529: 12 total, 12 up, 12 in 2024-02-09T17:43:35.149 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:34 smithi179 ceph-mon[43975]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:43:35.150 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:34 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:43:35.150 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:34 smithi179 ceph-mon[43975]: osdmap e529: 12 total, 12 up, 12 in 2024-02-09T17:43:35.170 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:43:35.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:35.176 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:43:35.200 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.8 dump_historic_ops 2024-02-09T17:43:35.277 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:43:35.346 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:35.375 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:43:35.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:35.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:43:35.479 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:43:35.533 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:35.571 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:43:35.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:35.580 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:43:35.672 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:43:35.682 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:43:35.744 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:35.824 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:43:35.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:35.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:43:35.928 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:43:35.973 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:35.996 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:43:35.999 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:43:36.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:36.001 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:43:36.031 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:43:36.048 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:43:36.077 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:43:36.100 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:43:36.102 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:43:36.135 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:43:36.159 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:43:36.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:35 smithi162 ceph-mon[44304]: 1.1b scrub starts 2024-02-09T17:43:36.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:35 smithi162 ceph-mon[44304]: 1.1b scrub ok 2024-02-09T17:43:36.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:35 smithi162 ceph-mon[44304]: pgmap v1334: 47 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 44 active+clean; 5.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 15 KiB/s wr, 760 op/s; 13008/248418 objects misplaced (5.236%); 22 MiB/s, 353 objects/s recovering 2024-02-09T17:43:36.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:35 smithi162 ceph-mon[44304]: 1.1a scrub starts 2024-02-09T17:43:36.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:35 smithi162 ceph-mon[44304]: 1.1a scrub ok 2024-02-09T17:43:36.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:35 smithi162 ceph-mon[44304]: 1.1d scrub starts 2024-02-09T17:43:36.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:35 smithi162 ceph-mon[44304]: 1.1d scrub ok 2024-02-09T17:43:36.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:35 smithi162 ceph-mon[44304]: 5.3 scrub starts 2024-02-09T17:43:36.181 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:36.203 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:43:36.232 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:43:36.233 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:43:36.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:36.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:43:36.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:35 smithi099 ceph-mon[39420]: 1.1b scrub starts 2024-02-09T17:43:36.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:35 smithi099 ceph-mon[39420]: 1.1b scrub ok 2024-02-09T17:43:36.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:35 smithi099 ceph-mon[39420]: pgmap v1334: 47 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 44 active+clean; 5.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 15 KiB/s wr, 760 op/s; 13008/248418 objects misplaced (5.236%); 22 MiB/s, 353 objects/s recovering 2024-02-09T17:43:36.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:35 smithi099 ceph-mon[39420]: 1.1a scrub starts 2024-02-09T17:43:36.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:35 smithi099 ceph-mon[39420]: 1.1a scrub ok 2024-02-09T17:43:36.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:35 smithi099 ceph-mon[39420]: 1.1d scrub starts 2024-02-09T17:43:36.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:35 smithi099 ceph-mon[39420]: 1.1d scrub ok 2024-02-09T17:43:36.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:35 smithi099 ceph-mon[39420]: 5.3 scrub starts 2024-02-09T17:43:36.267 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:43:36.298 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:43:36.316 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:35 smithi179 ceph-mon[43975]: 1.1b scrub starts 2024-02-09T17:43:36.316 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:35 smithi179 ceph-mon[43975]: 1.1b scrub ok 2024-02-09T17:43:36.316 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:35 smithi179 ceph-mon[43975]: pgmap v1334: 47 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 44 active+clean; 5.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 15 KiB/s wr, 760 op/s; 13008/248418 objects misplaced (5.236%); 22 MiB/s, 353 objects/s recovering 2024-02-09T17:43:36.316 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:35 smithi179 ceph-mon[43975]: 1.1a scrub starts 2024-02-09T17:43:36.317 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:35 smithi179 ceph-mon[43975]: 1.1a scrub ok 2024-02-09T17:43:36.317 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:35 smithi179 ceph-mon[43975]: 1.1d scrub starts 2024-02-09T17:43:36.317 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:35 smithi179 ceph-mon[43975]: 1.1d scrub ok 2024-02-09T17:43:36.317 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:35 smithi179 ceph-mon[43975]: 5.3 scrub starts 2024-02-09T17:43:36.331 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:43:36.335 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:43:36.335 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:43:36.335 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:43:36.335 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:43:36.335 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:43:36.335 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:43:36.336 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:43:36.336 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:43:36.336 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:43:36.336 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:43:36.336 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:43:36.336 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:43:36.339 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:43:36.409 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:36.967 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:43:36.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:36.971 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:43:36.998 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:36 smithi099 ceph-mon[39420]: 1.1e scrub starts 2024-02-09T17:43:36.999 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:36 smithi099 ceph-mon[39420]: 1.1e scrub ok 2024-02-09T17:43:36.999 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:36 smithi099 ceph-mon[39420]: 1.1c scrub starts 2024-02-09T17:43:36.999 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:36 smithi099 ceph-mon[39420]: 1.1c scrub ok 2024-02-09T17:43:36.999 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:36 smithi099 ceph-mon[39420]: 1.15 scrub starts 2024-02-09T17:43:36.999 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:36 smithi099 ceph-mon[39420]: 1.15 scrub ok 2024-02-09T17:43:36.999 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:36 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3942916408' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:43:36.999 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:36 smithi099 ceph-mon[39420]: 1.16 scrub starts 2024-02-09T17:43:36.999 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:36 smithi099 ceph-mon[39420]: 5.5 scrub starts 2024-02-09T17:43:36.999 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:36 smithi099 ceph-mon[39420]: 1.16 scrub ok 2024-02-09T17:43:37.072 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:43:37.151 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:37.196 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:43:37.199 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:36 smithi162 ceph-mon[44304]: 1.1e scrub starts 2024-02-09T17:43:37.199 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:36 smithi162 ceph-mon[44304]: 1.1e scrub ok 2024-02-09T17:43:37.199 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:36 smithi162 ceph-mon[44304]: 1.1c scrub starts 2024-02-09T17:43:37.199 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:36 smithi162 ceph-mon[44304]: 1.1c scrub ok 2024-02-09T17:43:37.199 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:36 smithi162 ceph-mon[44304]: 1.15 scrub starts 2024-02-09T17:43:37.199 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:36 smithi162 ceph-mon[44304]: 1.15 scrub ok 2024-02-09T17:43:37.199 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:36 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3942916408' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:43:37.199 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:36 smithi162 ceph-mon[44304]: 1.16 scrub starts 2024-02-09T17:43:37.199 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:36 smithi162 ceph-mon[44304]: 5.5 scrub starts 2024-02-09T17:43:37.199 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:36 smithi162 ceph-mon[44304]: 1.16 scrub ok 2024-02-09T17:43:37.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:37.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:43:37.301 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:43:37.370 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:37.388 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-09T17:43:37.402 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:43:37.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:37.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:43:37.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:36 smithi179 ceph-mon[43975]: 1.1e scrub starts 2024-02-09T17:43:37.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:36 smithi179 ceph-mon[43975]: 1.1e scrub ok 2024-02-09T17:43:37.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:36 smithi179 ceph-mon[43975]: 1.1c scrub starts 2024-02-09T17:43:37.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:36 smithi179 ceph-mon[43975]: 1.1c scrub ok 2024-02-09T17:43:37.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:36 smithi179 ceph-mon[43975]: 1.15 scrub starts 2024-02-09T17:43:37.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:36 smithi179 ceph-mon[43975]: 1.15 scrub ok 2024-02-09T17:43:37.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:36 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3942916408' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:43:37.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:36 smithi179 ceph-mon[43975]: 1.16 scrub starts 2024-02-09T17:43:37.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:36 smithi179 ceph-mon[43975]: 5.5 scrub starts 2024-02-09T17:43:37.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:36 smithi179 ceph-mon[43975]: 1.16 scrub ok 2024-02-09T17:43:37.509 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:43:37.577 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:37.616 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:43:37.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:37.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:43:37.722 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:43:37.756 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:37.837 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:43:37.840 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:37.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:43:37.941 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:43:37.965 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:43:37.990 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:38.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:37 smithi179 ceph-mon[43975]: 1.17 scrub starts 2024-02-09T17:43:38.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:37 smithi179 ceph-mon[43975]: 5.3 scrub ok 2024-02-09T17:43:38.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:37 smithi179 ceph-mon[43975]: 1.17 scrub ok 2024-02-09T17:43:38.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:37 smithi179 ceph-mon[43975]: pgmap v1335: 47 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 43 active+clean; 4.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 812 op/s; 13008/243279 objects misplaced (5.347%); 18 MiB/s, 290 objects/s recovering 2024-02-09T17:43:38.248 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:37 smithi099 ceph-mon[39420]: 1.17 scrub starts 2024-02-09T17:43:38.248 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:37 smithi099 ceph-mon[39420]: 5.3 scrub ok 2024-02-09T17:43:38.248 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:37 smithi099 ceph-mon[39420]: 1.17 scrub ok 2024-02-09T17:43:38.248 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:37 smithi099 ceph-mon[39420]: pgmap v1335: 47 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 43 active+clean; 4.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 812 op/s; 13008/243279 objects misplaced (5.347%); 18 MiB/s, 290 objects/s recovering 2024-02-09T17:43:38.379 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:43:38.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:38.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:43:38.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:37 smithi162 ceph-mon[44304]: 1.17 scrub starts 2024-02-09T17:43:38.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:37 smithi162 ceph-mon[44304]: 5.3 scrub ok 2024-02-09T17:43:38.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:37 smithi162 ceph-mon[44304]: 1.17 scrub ok 2024-02-09T17:43:38.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:37 smithi162 ceph-mon[44304]: pgmap v1335: 47 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 43 active+clean; 4.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 812 op/s; 13008/243279 objects misplaced (5.347%); 18 MiB/s, 290 objects/s recovering 2024-02-09T17:43:38.482 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:43:38.529 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:38.902 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:43:38.906 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:38.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:43:38.976 INFO:teuthology.orchestra.run.smithi099.stderr:set osd.7 primary-affinity to 0.598263 (8392072) 2024-02-09T17:43:39.008 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:43:39.046 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:39.104 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:43:39.106 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:39.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:43:39.207 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:43:39.242 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:38 smithi099 ceph-mon[39420]: 5.5 scrub ok 2024-02-09T17:43:39.242 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:38 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:43:39.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:38 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:43:39.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:38 smithi099 ceph-mon[39420]: 5.e scrub starts 2024-02-09T17:43:39.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:38 smithi099 ceph-mon[39420]: osdmap e530: 12 total, 12 up, 12 in 2024-02-09T17:43:39.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:38 smithi099 ceph-mon[39420]: 5.c scrub starts 2024-02-09T17:43:39.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:38 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4049853432' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.5982631885847566}]: dispatch 2024-02-09T17:43:39.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:38 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.5982631885847566}]: dispatch 2024-02-09T17:43:39.244 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:39.302 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:43:39.356 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:43:39.357 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-02-09T17:43:39.357 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-02-09T17:43:39.357 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-02-09T17:43:39.357 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.357 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:232414 4.0 4:062b8f20:::benchmark_data_smithi179_103809_object94948:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e413)", 2024-02-09T17:43:39.357 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:39:22.735659+0000", 2024-02-09T17:43:39.357 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 256.614971862, 2024-02-09T17:43:39.357 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.85919624400000005, 2024-02-09T17:43:39.357 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:39.357 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-02-09T17:43:39.357 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:39.357 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:43:39.357 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:43:39.357 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 232414 2024-02-09T17:43:39.358 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.358 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:39.358 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.358 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:39.358 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735659+0000", 2024-02-09T17:43:39.358 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:39.358 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.358 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.358 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:39.358 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735659+0000", 2024-02-09T17:43:39.358 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:39.358 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.358 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.358 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:39.358 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735660+0000", 2024-02-09T17:43:39.359 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.279e-06 2024-02-09T17:43:39.359 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.359 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.359 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:39.359 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735664+0000", 2024-02-09T17:43:39.359 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2010000000000001e-06 2024-02-09T17:43:39.359 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.359 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.359 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:39.359 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735674+0000", 2024-02-09T17:43:39.359 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6360000000000006e-06 2024-02-09T17:43:39.359 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.359 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.359 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:39.360 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735682+0000", 2024-02-09T17:43:39.360 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.541e-06 2024-02-09T17:43:39.360 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.360 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.360 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:39.360 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735749+0000", 2024-02-09T17:43:39.360 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6737999999999997e-05 2024-02-09T17:43:39.360 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.360 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.360 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:43:39.360 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735760+0000", 2024-02-09T17:43:39.360 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.096e-05 2024-02-09T17:43:39.360 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.361 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.361 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:39.361 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.594762+0000", 2024-02-09T17:43:39.361 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.85900255199999997 2024-02-09T17:43:39.361 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.361 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.361 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:39.361 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.594855+0000", 2024-02-09T17:43:39.361 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2336999999999994e-05 2024-02-09T17:43:39.361 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:39.361 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:39.361 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:39.361 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.361 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.361 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:232443 4.0 4:06265949:::benchmark_data_smithi179_103809_object94977:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-02-09T17:43:39.362 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:39:22.735704+0000", 2024-02-09T17:43:39.362 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 256.61492607899999, 2024-02-09T17:43:39.362 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.87793835899999995, 2024-02-09T17:43:39.362 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:39.362 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:39.362 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:39.362 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:43:39.362 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:43:39.362 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 232443 2024-02-09T17:43:39.362 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.362 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:39.362 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.362 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:39.362 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735704+0000", 2024-02-09T17:43:39.363 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:39.363 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.363 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.363 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:39.363 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735704+0000", 2024-02-09T17:43:39.363 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:39.363 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.363 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.363 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:39.363 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735705+0000", 2024-02-09T17:43:39.363 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1300000000000001e-07 2024-02-09T17:43:39.363 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.363 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.363 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:39.363 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735707+0000", 2024-02-09T17:43:39.364 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0600000000000002e-06 2024-02-09T17:43:39.364 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.364 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.364 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:39.364 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735713+0000", 2024-02-09T17:43:39.364 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8449999999999999e-06 2024-02-09T17:43:39.364 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.364 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.364 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:39.364 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735719+0000", 2024-02-09T17:43:39.364 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0410000000000003e-06 2024-02-09T17:43:39.364 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.364 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.365 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:39.365 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735792+0000", 2024-02-09T17:43:39.365 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.3719000000000003e-05 2024-02-09T17:43:39.365 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.365 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.365 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:43:39.365 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735798+0000", 2024-02-09T17:43:39.365 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7860000000000002e-06 2024-02-09T17:43:39.365 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.365 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.365 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:39.365 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.595333+0000", 2024-02-09T17:43:39.365 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.85953531500000002 2024-02-09T17:43:39.365 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.365 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.365 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:39.365 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.595449+0000", 2024-02-09T17:43:39.366 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011531600000000001 2024-02-09T17:43:39.366 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.366 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.366 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,9", 2024-02-09T17:43:39.366 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.595565+0000", 2024-02-09T17:43:39.366 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011670100000000001 2024-02-09T17:43:39.366 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.366 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.366 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:39.366 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.603480+0000", 2024-02-09T17:43:39.366 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0079149809999999998 2024-02-09T17:43:39.366 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.366 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.366 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:39.366 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.613425+0000", 2024-02-09T17:43:39.367 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0099445639999999995 2024-02-09T17:43:39.367 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.367 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.367 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:39.367 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.613547+0000", 2024-02-09T17:43:39.367 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012220300000000001 2024-02-09T17:43:39.367 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.367 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.367 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:39.367 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.613607+0000", 2024-02-09T17:43:39.367 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9687e-05 2024-02-09T17:43:39.367 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.367 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.367 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:39.367 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.613643+0000", 2024-02-09T17:43:39.367 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.5728000000000001e-05 2024-02-09T17:43:39.368 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:39.368 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:39.368 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:39.368 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.368 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.368 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:232449 4.0 4:0d4bf3f6:::benchmark_data_smithi179_103809_object94983:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-02-09T17:43:39.368 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:39:22.735743+0000", 2024-02-09T17:43:39.368 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 256.61488778, 2024-02-09T17:43:39.368 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.87819052500000006, 2024-02-09T17:43:39.368 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:39.368 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:39.368 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:39.368 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:43:39.368 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:43:39.368 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 232449 2024-02-09T17:43:39.369 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.369 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:39.369 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.369 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:39.369 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735743+0000", 2024-02-09T17:43:39.369 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:39.369 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.369 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.369 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:39.369 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735743+0000", 2024-02-09T17:43:39.369 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:39.369 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.369 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.369 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:39.369 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735743+0000", 2024-02-09T17:43:39.370 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0500000000000004e-07 2024-02-09T17:43:39.370 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.370 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.370 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:39.370 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735746+0000", 2024-02-09T17:43:39.370 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4710000000000001e-06 2024-02-09T17:43:39.370 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.370 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.370 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:39.370 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735753+0000", 2024-02-09T17:43:39.370 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.097e-06 2024-02-09T17:43:39.370 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.370 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.370 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:39.370 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735761+0000", 2024-02-09T17:43:39.371 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5860000000000005e-06 2024-02-09T17:43:39.371 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.371 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.371 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:39.371 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735823+0000", 2024-02-09T17:43:39.371 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1563000000000007e-05 2024-02-09T17:43:39.371 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.371 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.371 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:43:39.371 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735830+0000", 2024-02-09T17:43:39.371 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5959999999999998e-06 2024-02-09T17:43:39.371 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.371 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.371 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:39.372 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.599236+0000", 2024-02-09T17:43:39.372 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.86340511399999997 2024-02-09T17:43:39.372 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.372 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.372 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:39.372 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.599367+0000", 2024-02-09T17:43:39.372 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013145100000000001 2024-02-09T17:43:39.372 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.372 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.372 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,9", 2024-02-09T17:43:39.372 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.599476+0000", 2024-02-09T17:43:39.372 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000108902 2024-02-09T17:43:39.372 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.373 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.373 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:39.373 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.613658+0000", 2024-02-09T17:43:39.373 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014181808000000001 2024-02-09T17:43:39.373 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.373 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.373 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:39.373 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.613714+0000", 2024-02-09T17:43:39.373 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6283000000000003e-05 2024-02-09T17:43:39.373 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.373 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.373 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:39.373 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.613840+0000", 2024-02-09T17:43:39.373 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000126248 2024-02-09T17:43:39.374 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.374 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.374 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:39.374 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.613895+0000", 2024-02-09T17:43:39.374 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4435000000000003e-05 2024-02-09T17:43:39.374 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.374 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.374 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:39.374 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.613933+0000", 2024-02-09T17:43:39.374 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8466000000000001e-05 2024-02-09T17:43:39.374 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:39.374 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:39.374 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:39.374 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.374 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.375 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:232450 4.0 4:0d2c10a7:::benchmark_data_smithi179_103809_object94984:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-02-09T17:43:39.375 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:39:22.735785+0000", 2024-02-09T17:43:39.375 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 256.614845543, 2024-02-09T17:43:39.375 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.88694171700000002, 2024-02-09T17:43:39.375 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:39.375 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:39.375 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:39.375 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:43:39.375 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:43:39.375 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 232450 2024-02-09T17:43:39.375 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.375 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:39.375 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.375 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:39.376 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735785+0000", 2024-02-09T17:43:39.376 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:39.376 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.376 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.376 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:39.376 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735785+0000", 2024-02-09T17:43:39.376 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:39.376 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.376 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.376 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:39.376 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735785+0000", 2024-02-09T17:43:39.376 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1600000000000001e-07 2024-02-09T17:43:39.376 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.377 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.377 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:39.377 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735788+0000", 2024-02-09T17:43:39.377 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5320000000000001e-06 2024-02-09T17:43:39.377 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.377 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.377 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:39.377 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735795+0000", 2024-02-09T17:43:39.377 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.0770000000000002e-06 2024-02-09T17:43:39.377 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.377 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.377 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:39.377 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735801+0000", 2024-02-09T17:43:39.377 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7930000000000003e-06 2024-02-09T17:43:39.377 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.378 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.378 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:39.378 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735849+0000", 2024-02-09T17:43:39.378 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8727000000000003e-05 2024-02-09T17:43:39.378 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.378 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.378 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:43:39.378 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735853+0000", 2024-02-09T17:43:39.378 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0040000000000001e-06 2024-02-09T17:43:39.378 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.378 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.378 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:39.378 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.603542+0000", 2024-02-09T17:43:39.378 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.86768888 2024-02-09T17:43:39.378 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.378 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.379 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:39.379 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.603668+0000", 2024-02-09T17:43:39.379 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000125714 2024-02-09T17:43:39.379 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.379 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.379 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,9", 2024-02-09T17:43:39.379 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.603811+0000", 2024-02-09T17:43:39.379 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014338799999999999 2024-02-09T17:43:39.379 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.379 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.379 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:39.379 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.613664+0000", 2024-02-09T17:43:39.379 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0098526250000000003 2024-02-09T17:43:39.379 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.379 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.380 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:39.380 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.613770+0000", 2024-02-09T17:43:39.380 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000106165 2024-02-09T17:43:39.380 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.380 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.380 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:39.380 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.622628+0000", 2024-02-09T17:43:39.380 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0088578870000000001 2024-02-09T17:43:39.380 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.380 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.380 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:39.380 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.622686+0000", 2024-02-09T17:43:39.380 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8233999999999998e-05 2024-02-09T17:43:39.381 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.381 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.381 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:39.381 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.622727+0000", 2024-02-09T17:43:39.381 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0175000000000001e-05 2024-02-09T17:43:39.381 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:39.381 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:39.381 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:39.381 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.381 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.381 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:232453 4.0 4:0be208e0:::benchmark_data_smithi179_103809_object94987:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-02-09T17:43:39.381 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:39:22.735817+0000", 2024-02-09T17:43:39.381 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 256.61481318900002, 2024-02-09T17:43:39.381 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.88704940300000001, 2024-02-09T17:43:39.381 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:39.382 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:39.382 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:39.382 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:43:39.382 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:43:39.382 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 232453 2024-02-09T17:43:39.382 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.382 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:39.382 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.382 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:39.382 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735817+0000", 2024-02-09T17:43:39.382 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:39.382 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.382 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.382 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:39.382 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735817+0000", 2024-02-09T17:43:39.382 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:39.383 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.383 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.383 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:39.383 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735818+0000", 2024-02-09T17:43:39.383 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4400000000000001e-07 2024-02-09T17:43:39.383 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.383 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.383 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:39.383 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735820+0000", 2024-02-09T17:43:39.383 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.34e-06 2024-02-09T17:43:39.383 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.383 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.383 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:39.383 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735827+0000", 2024-02-09T17:43:39.384 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8299999999999998e-06 2024-02-09T17:43:39.384 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.384 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.384 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:39.384 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735832+0000", 2024-02-09T17:43:39.384 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6679999999999998e-06 2024-02-09T17:43:39.384 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.384 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.384 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:39.384 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735867+0000", 2024-02-09T17:43:39.384 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4746999999999998e-05 2024-02-09T17:43:39.384 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.384 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.385 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:43:39.385 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735871+0000", 2024-02-09T17:43:39.385 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9210000000000002e-06 2024-02-09T17:43:39.385 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.385 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.385 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:39.385 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.606897+0000", 2024-02-09T17:43:39.385 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.87102594600000005 2024-02-09T17:43:39.385 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.385 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.385 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:39.385 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.607026+0000", 2024-02-09T17:43:39.386 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012845000000000001 2024-02-09T17:43:39.386 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.386 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.386 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,9", 2024-02-09T17:43:39.386 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.607107+0000", 2024-02-09T17:43:39.386 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1458000000000005e-05 2024-02-09T17:43:39.386 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.386 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.386 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:39.386 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.613784+0000", 2024-02-09T17:43:39.386 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0066770040000000003 2024-02-09T17:43:39.386 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.386 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.387 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:39.387 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.621146+0000", 2024-02-09T17:43:39.387 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0073621379999999998 2024-02-09T17:43:39.387 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.387 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.387 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:39.387 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.622785+0000", 2024-02-09T17:43:39.387 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001639199 2024-02-09T17:43:39.387 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.387 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.387 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:39.387 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.622829+0000", 2024-02-09T17:43:39.387 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3457000000000001e-05 2024-02-09T17:43:39.387 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.388 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.388 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:39.388 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.622867+0000", 2024-02-09T17:43:39.388 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7800999999999999e-05 2024-02-09T17:43:39.388 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:39.388 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:39.388 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:39.388 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.388 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.388 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:232474 4.0 4:05e1ef6a:::benchmark_data_smithi179_103809_object95008:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-02-09T17:43:39.388 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:39:22.735849+0000", 2024-02-09T17:43:39.388 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 256.61478137199998, 2024-02-09T17:43:39.388 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.88716656000000005, 2024-02-09T17:43:39.389 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:39.389 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:39.389 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:39.389 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:43:39.389 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:43:39.389 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 232474 2024-02-09T17:43:39.389 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.389 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:39.389 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.389 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:39.389 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735849+0000", 2024-02-09T17:43:39.389 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:39.389 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.389 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.389 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:39.389 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735849+0000", 2024-02-09T17:43:39.390 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:39.390 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.390 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.390 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:39.390 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735849+0000", 2024-02-09T17:43:39.390 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7399999999999999e-07 2024-02-09T17:43:39.390 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.390 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.390 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:39.390 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735852+0000", 2024-02-09T17:43:39.390 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3549999999999999e-06 2024-02-09T17:43:39.390 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.390 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.390 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:39.390 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735858+0000", 2024-02-09T17:43:39.391 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.319e-06 2024-02-09T17:43:39.391 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.391 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.391 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:39.391 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735864+0000", 2024-02-09T17:43:39.391 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1800000000000001e-06 2024-02-09T17:43:39.391 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.391 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.391 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:39.391 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735886+0000", 2024-02-09T17:43:39.391 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1517000000000001e-05 2024-02-09T17:43:39.391 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.391 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.391 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:43:39.392 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735889+0000", 2024-02-09T17:43:39.392 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.41e-06 2024-02-09T17:43:39.392 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.392 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.392 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:39.392 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.609948+0000", 2024-02-09T17:43:39.392 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.87405859299999999 2024-02-09T17:43:39.392 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.392 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.392 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:39.392 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.610078+0000", 2024-02-09T17:43:39.392 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000130204 2024-02-09T17:43:39.392 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.392 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.393 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,9", 2024-02-09T17:43:39.393 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.610219+0000", 2024-02-09T17:43:39.393 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014117000000000001 2024-02-09T17:43:39.393 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.393 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.393 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:39.393 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.613789+0000", 2024-02-09T17:43:39.393 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0035698560000000002 2024-02-09T17:43:39.393 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.393 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.393 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:39.393 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.621196+0000", 2024-02-09T17:43:39.393 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0074071079999999999 2024-02-09T17:43:39.393 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.394 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.394 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:39.394 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.622922+0000", 2024-02-09T17:43:39.394 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001725667 2024-02-09T17:43:39.394 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.394 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.394 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:39.394 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.622980+0000", 2024-02-09T17:43:39.394 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8508999999999998e-05 2024-02-09T17:43:39.394 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.394 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.394 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:39.394 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.623016+0000", 2024-02-09T17:43:39.394 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.5298e-05 2024-02-09T17:43:39.394 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:39.394 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:39.395 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:39.395 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.395 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.395 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:232516 4.0 4:0c99ad98:::benchmark_data_smithi179_103809_object95050:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-02-09T17:43:39.395 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:39:22.735873+0000", 2024-02-09T17:43:39.395 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 256.61475708400002, 2024-02-09T17:43:39.395 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.88729041500000005, 2024-02-09T17:43:39.395 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:39.395 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:39.395 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:39.395 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:43:39.395 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:43:39.395 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 232516 2024-02-09T17:43:39.395 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.395 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:39.396 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.396 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:39.396 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735873+0000", 2024-02-09T17:43:39.396 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:39.396 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.396 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.396 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:39.396 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735873+0000", 2024-02-09T17:43:39.396 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:39.396 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.396 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.396 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:39.397 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735874+0000", 2024-02-09T17:43:39.397 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7800000000000002e-07 2024-02-09T17:43:39.397 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.397 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.397 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:39.397 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735876+0000", 2024-02-09T17:43:39.397 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2069999999999998e-06 2024-02-09T17:43:39.397 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.397 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.397 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:39.397 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735882+0000", 2024-02-09T17:43:39.397 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9220000000000002e-06 2024-02-09T17:43:39.397 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.397 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.397 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:39.398 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735887+0000", 2024-02-09T17:43:39.398 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.31e-06 2024-02-09T17:43:39.398 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.398 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.398 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:39.398 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735909+0000", 2024-02-09T17:43:39.398 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2161e-05 2024-02-09T17:43:39.398 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.398 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.398 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:43:39.398 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.735913+0000", 2024-02-09T17:43:39.398 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.495e-06 2024-02-09T17:43:39.398 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.398 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.399 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:39.399 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.613979+0000", 2024-02-09T17:43:39.399 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.87806641699999999 2024-02-09T17:43:39.399 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.399 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.399 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:39.399 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.614061+0000", 2024-02-09T17:43:39.399 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1575000000000004e-05 2024-02-09T17:43:39.399 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.399 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.399 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,9", 2024-02-09T17:43:39.399 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.614181+0000", 2024-02-09T17:43:39.399 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000120658 2024-02-09T17:43:39.399 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.399 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.400 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:39.400 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.621212+0000", 2024-02-09T17:43:39.400 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0070306910000000004 2024-02-09T17:43:39.400 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.400 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.400 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:39.400 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.621274+0000", 2024-02-09T17:43:39.400 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2323000000000006e-05 2024-02-09T17:43:39.400 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.400 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.400 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:39.400 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.623072+0000", 2024-02-09T17:43:39.400 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0017977850000000001 2024-02-09T17:43:39.400 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.401 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.401 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:39.401 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.623123+0000", 2024-02-09T17:43:39.401 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0671000000000003e-05 2024-02-09T17:43:39.401 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.401 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.401 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:39.401 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.623164+0000", 2024-02-09T17:43:39.401 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0822000000000003e-05 2024-02-09T17:43:39.401 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:39.401 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:39.401 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:39.401 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.401 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.402 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:232537 4.0 4:0c405ed9:::benchmark_data_smithi179_103809_object95071:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-02-09T17:43:39.402 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:39:22.800233+0000", 2024-02-09T17:43:39.402 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 256.55039712600001, 2024-02-09T17:43:39.402 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.82809807000000002, 2024-02-09T17:43:39.402 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:39.402 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:39.402 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:39.402 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:43:39.402 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:43:39.402 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 232537 2024-02-09T17:43:39.402 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.402 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:39.402 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.402 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:39.402 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.800233+0000", 2024-02-09T17:43:39.403 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:39.403 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.403 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.403 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:39.403 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.800233+0000", 2024-02-09T17:43:39.403 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:39.403 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.403 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.403 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:39.403 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.800235+0000", 2024-02-09T17:43:39.403 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.671e-06 2024-02-09T17:43:39.403 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.403 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.403 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:39.404 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.800241+0000", 2024-02-09T17:43:39.404 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2450000000000003e-06 2024-02-09T17:43:39.404 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.404 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.404 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:39.404 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.800255+0000", 2024-02-09T17:43:39.404 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3689999999999999e-05 2024-02-09T17:43:39.404 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.404 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.404 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:39.404 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.800268+0000", 2024-02-09T17:43:39.404 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3546999999999999e-05 2024-02-09T17:43:39.404 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.405 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.405 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:39.405 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.800330+0000", 2024-02-09T17:43:39.405 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1965999999999996e-05 2024-02-09T17:43:39.405 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.405 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.405 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:43:39.405 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:22.800340+0000", 2024-02-09T17:43:39.405 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2040000000000003e-06 2024-02-09T17:43:39.405 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.405 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.405 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:39.405 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.617653+0000", 2024-02-09T17:43:39.405 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.81731293000000005 2024-02-09T17:43:39.405 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.406 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.406 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:39.406 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.617775+0000", 2024-02-09T17:43:39.406 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012293800000000001 2024-02-09T17:43:39.406 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.406 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.406 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,9", 2024-02-09T17:43:39.406 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.617891+0000", 2024-02-09T17:43:39.406 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011540900000000001 2024-02-09T17:43:39.406 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.406 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.406 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:39.406 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.621350+0000", 2024-02-09T17:43:39.406 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0034587469999999999 2024-02-09T17:43:39.407 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.407 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.407 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:39.407 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.623212+0000", 2024-02-09T17:43:39.407 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001862306 2024-02-09T17:43:39.407 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.407 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.407 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:39.407 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.628274+0000", 2024-02-09T17:43:39.407 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0050616360000000004 2024-02-09T17:43:39.407 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.407 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.407 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:39.407 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.628309+0000", 2024-02-09T17:43:39.407 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.5828000000000003e-05 2024-02-09T17:43:39.408 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.408 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.408 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:39.408 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.628331+0000", 2024-02-09T17:43:39.408 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1953000000000001e-05 2024-02-09T17:43:39.408 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:39.408 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:39.408 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:39.408 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.408 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.408 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.25201.0:267603 4.2 e455/392 4:49ab35e5:::benchmark_data_smithi179_103809_object130137:head v 455'16657, mlcod=454'16655)", 2024-02-09T17:43:39.408 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:40:22.574991+0000", 2024-02-09T17:43:39.408 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 196.77563913399999, 2024-02-09T17:43:39.408 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.055128937000000003, 2024-02-09T17:43:39.409 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:39.409 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:39.409 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:39.409 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.409 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:39.409 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.574991+0000", 2024-02-09T17:43:39.409 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:39.409 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.409 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.409 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:39.409 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.574991+0000", 2024-02-09T17:43:39.409 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:39.409 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.409 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.409 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:39.410 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.574993+0000", 2024-02-09T17:43:39.410 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.269e-06 2024-02-09T17:43:39.410 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.410 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.410 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:39.410 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.575002+0000", 2024-02-09T17:43:39.410 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.0620000000000007e-06 2024-02-09T17:43:39.410 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.410 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.410 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:39.410 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.575018+0000", 2024-02-09T17:43:39.410 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6365e-05 2024-02-09T17:43:39.410 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.410 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.410 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:39.411 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.575028+0000", 2024-02-09T17:43:39.411 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0039e-05 2024-02-09T17:43:39.411 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.411 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.411 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:39.411 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.575115+0000", 2024-02-09T17:43:39.411 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7099000000000001e-05 2024-02-09T17:43:39.411 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.411 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.411 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:39.411 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.575141+0000", 2024-02-09T17:43:39.411 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5548999999999999e-05 2024-02-09T17:43:39.411 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.411 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.411 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:39.412 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.630069+0000", 2024-02-09T17:43:39.412 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.054928439000000003 2024-02-09T17:43:39.412 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.412 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.412 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:39.412 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.630120+0000", 2024-02-09T17:43:39.412 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1115000000000001e-05 2024-02-09T17:43:39.412 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:39.412 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:39.412 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:39.412 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.412 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.412 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.25201.0:267607 4.2 e455/392 4:4a859e8e:::benchmark_data_smithi179_103809_object130141:head v 455'16658, mlcod=454'16655)", 2024-02-09T17:43:39.413 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:40:22.581667+0000", 2024-02-09T17:43:39.413 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 196.76896339499999, 2024-02-09T17:43:39.413 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.048716113999999998, 2024-02-09T17:43:39.413 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:39.413 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:39.413 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:39.413 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.413 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:39.413 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.581667+0000", 2024-02-09T17:43:39.413 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:39.413 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.413 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.413 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:39.413 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.581667+0000", 2024-02-09T17:43:39.413 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:39.414 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.414 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.414 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:39.414 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.581668+0000", 2024-02-09T17:43:39.414 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.203e-06 2024-02-09T17:43:39.414 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.414 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.414 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:39.414 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.581679+0000", 2024-02-09T17:43:39.414 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0638999999999999e-05 2024-02-09T17:43:39.414 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.414 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.414 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:39.414 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.581699+0000", 2024-02-09T17:43:39.414 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0247000000000001e-05 2024-02-09T17:43:39.415 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.415 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.415 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:39.415 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.581709+0000", 2024-02-09T17:43:39.415 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6739999999999992e-06 2024-02-09T17:43:39.415 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.415 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.415 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:39.415 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.599182+0000", 2024-02-09T17:43:39.415 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017473299000000001 2024-02-09T17:43:39.415 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.415 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.415 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:39.415 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.599203+0000", 2024-02-09T17:43:39.415 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1106000000000001e-05 2024-02-09T17:43:39.416 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.416 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.416 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:39.416 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.630348+0000", 2024-02-09T17:43:39.416 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.031144727000000001 2024-02-09T17:43:39.416 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.416 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.416 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:39.416 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.630383+0000", 2024-02-09T17:43:39.416 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.5219000000000003e-05 2024-02-09T17:43:39.416 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:39.416 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:39.416 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:39.417 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.417 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.417 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:73304 5.a 5:542d3499:::benchmark_data_smithi179_115199_object73303:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e489)", 2024-02-09T17:43:39.417 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:41:44.143308+0000", 2024-02-09T17:43:39.417 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 115.207322854, 2024-02-09T17:43:39.417 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.049784837999999998, 2024-02-09T17:43:39.417 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:39.417 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:39.417 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:39.417 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:43:39.417 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:43:39.417 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 73304 2024-02-09T17:43:39.417 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.417 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:39.417 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.418 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:39.418 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:44.143308+0000", 2024-02-09T17:43:39.418 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:39.418 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.418 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.418 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:39.418 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:44.143308+0000", 2024-02-09T17:43:39.418 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:39.418 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.418 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.418 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:39.418 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:44.143308+0000", 2024-02-09T17:43:39.418 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9599999999999999e-07 2024-02-09T17:43:39.418 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.418 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.419 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:39.419 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:44.143340+0000", 2024-02-09T17:43:39.419 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2394000000000002e-05 2024-02-09T17:43:39.419 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.419 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.419 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:39.419 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:44.143346+0000", 2024-02-09T17:43:39.419 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5929999999999996e-06 2024-02-09T17:43:39.419 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.419 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.419 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:39.419 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:44.143352+0000", 2024-02-09T17:43:39.419 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.626e-06 2024-02-09T17:43:39.419 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.420 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.420 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:39.420 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:44.176853+0000", 2024-02-09T17:43:39.420 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.033501072 2024-02-09T17:43:39.420 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.420 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.420 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:39.420 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:44.176972+0000", 2024-02-09T17:43:39.420 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000119672 2024-02-09T17:43:39.420 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.420 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.420 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 4,10", 2024-02-09T17:43:39.420 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:44.177107+0000", 2024-02-09T17:43:39.420 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013504599999999999 2024-02-09T17:43:39.421 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.421 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.421 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:39.421 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:44.182275+0000", 2024-02-09T17:43:39.421 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0051680010000000002 2024-02-09T17:43:39.421 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.421 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.421 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:39.421 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:44.182432+0000", 2024-02-09T17:43:39.421 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000156754 2024-02-09T17:43:39.421 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.421 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.421 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:39.421 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:44.192992+0000", 2024-02-09T17:43:39.421 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010559535 2024-02-09T17:43:39.422 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.422 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.422 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:39.422 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:44.193056+0000", 2024-02-09T17:43:39.422 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4342000000000002e-05 2024-02-09T17:43:39.422 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.422 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.422 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:39.422 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:41:44.193092+0000", 2024-02-09T17:43:39.422 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6307000000000003e-05 2024-02-09T17:43:39.422 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:39.422 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:39.422 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:39.422 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.423 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.423 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:160293 5.a 5:52b33aff:::benchmark_data_smithi179_115199_object31387:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e512)", 2024-02-09T17:43:39.423 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:43:09.927749+0000", 2024-02-09T17:43:39.423 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 29.422881559, 2024-02-09T17:43:39.423 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.051307563, 2024-02-09T17:43:39.423 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:39.423 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:39.423 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:39.423 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:43:39.423 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:43:39.423 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 160293 2024-02-09T17:43:39.423 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.423 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:39.423 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.423 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:39.424 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:09.927749+0000", 2024-02-09T17:43:39.424 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:39.424 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.424 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.424 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:39.424 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:09.927749+0000", 2024-02-09T17:43:39.424 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:39.424 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.424 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.424 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:39.424 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:09.927750+0000", 2024-02-09T17:43:39.424 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3149999999999999e-06 2024-02-09T17:43:39.424 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.425 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.425 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:39.425 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:09.927756+0000", 2024-02-09T17:43:39.425 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0530000000000003e-06 2024-02-09T17:43:39.425 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.425 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.425 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:39.425 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:09.927770+0000", 2024-02-09T17:43:39.425 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4241e-05 2024-02-09T17:43:39.425 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.425 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.425 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:39.425 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:09.927780+0000", 2024-02-09T17:43:39.425 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.3880000000000006e-06 2024-02-09T17:43:39.425 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.425 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.426 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:39.426 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:09.936373+0000", 2024-02-09T17:43:39.426 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0085929070000000003 2024-02-09T17:43:39.426 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.426 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.426 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:39.426 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:09.936495+0000", 2024-02-09T17:43:39.426 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012262199999999999 2024-02-09T17:43:39.426 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.426 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.426 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 4,10", 2024-02-09T17:43:39.426 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:09.936610+0000", 2024-02-09T17:43:39.426 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000114914 2024-02-09T17:43:39.426 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.427 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.427 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:39.427 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:09.949811+0000", 2024-02-09T17:43:39.427 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013200943 2024-02-09T17:43:39.427 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.427 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.427 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:39.427 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:09.961404+0000", 2024-02-09T17:43:39.427 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011593233999999999 2024-02-09T17:43:39.427 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.427 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.427 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:39.427 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:09.978952+0000", 2024-02-09T17:43:39.427 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017548033000000001 2024-02-09T17:43:39.428 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.428 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.428 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:39.428 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:09.979015+0000", 2024-02-09T17:43:39.428 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2359000000000004e-05 2024-02-09T17:43:39.428 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.428 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.428 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:39.428 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:09.979056+0000", 2024-02-09T17:43:39.428 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1554000000000002e-05 2024-02-09T17:43:39.428 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:39.428 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:39.428 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:39.428 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.429 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.429 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:170482 5.a 5:5724cd16:::benchmark_data_smithi179_115199_object41576:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e521)", 2024-02-09T17:43:39.429 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:43:24.167032+0000", 2024-02-09T17:43:39.429 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 15.18359845, 2024-02-09T17:43:39.429 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7617426749999998, 2024-02-09T17:43:39.429 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:39.429 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:39.429 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:39.429 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:43:39.429 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:43:39.429 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 170482 2024-02-09T17:43:39.429 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.429 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:39.429 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.429 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:39.430 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.167032+0000", 2024-02-09T17:43:39.430 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:39.430 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.430 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.430 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:39.430 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.167032+0000", 2024-02-09T17:43:39.430 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:39.430 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.430 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.430 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:39.430 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.167033+0000", 2024-02-09T17:43:39.430 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3400000000000001e-06 2024-02-09T17:43:39.430 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.430 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.431 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:39.431 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.167037+0000", 2024-02-09T17:43:39.431 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7110000000000001e-06 2024-02-09T17:43:39.431 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.431 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.431 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:39.431 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.167047+0000", 2024-02-09T17:43:39.431 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0051000000000001e-05 2024-02-09T17:43:39.431 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.431 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.431 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:39.431 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.167058+0000", 2024-02-09T17:43:39.431 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0479000000000001e-05 2024-02-09T17:43:39.431 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.431 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.432 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:39.432 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.167113+0000", 2024-02-09T17:43:39.432 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5609000000000002e-05 2024-02-09T17:43:39.432 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.432 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.432 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op must wait for map", 2024-02-09T17:43:39.432 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.167120+0000", 2024-02-09T17:43:39.432 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.849e-06 2024-02-09T17:43:39.432 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.432 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.432 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:39.432 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.172513+0000", 2024-02-09T17:43:39.432 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0053932110000000002 2024-02-09T17:43:39.432 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.433 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.433 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:43:39.433 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.172524+0000", 2024-02-09T17:43:39.433 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1249e-05 2024-02-09T17:43:39.433 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.433 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.433 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:39.433 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:25.163553+0000", 2024-02-09T17:43:39.433 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.99102885900000004 2024-02-09T17:43:39.433 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.433 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.433 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:43:39.433 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:25.163567+0000", 2024-02-09T17:43:39.433 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4019000000000001e-05 2024-02-09T17:43:39.434 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.434 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.434 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:39.434 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.907286+0000", 2024-02-09T17:43:39.434 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.74371831 2024-02-09T17:43:39.434 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.434 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.434 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:39.434 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.907433+0000", 2024-02-09T17:43:39.434 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000146923 2024-02-09T17:43:39.434 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.434 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.434 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,4,10", 2024-02-09T17:43:39.434 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.907553+0000", 2024-02-09T17:43:39.435 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000120517 2024-02-09T17:43:39.435 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.435 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.435 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:39.435 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.912625+0000", 2024-02-09T17:43:39.435 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.005071819 2024-02-09T17:43:39.435 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.435 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.435 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:39.435 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.912827+0000", 2024-02-09T17:43:39.435 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00020239499999999999 2024-02-09T17:43:39.435 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.435 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.435 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:39.435 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.917813+0000", 2024-02-09T17:43:39.435 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.004986117 2024-02-09T17:43:39.436 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.436 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.436 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:39.436 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.928603+0000", 2024-02-09T17:43:39.436 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010789234 2024-02-09T17:43:39.436 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.436 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.436 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:39.436 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.928665+0000", 2024-02-09T17:43:39.436 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2359000000000004e-05 2024-02-09T17:43:39.436 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.436 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.436 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:39.436 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.928775+0000", 2024-02-09T17:43:39.437 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010962400000000001 2024-02-09T17:43:39.437 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:39.437 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:39.437 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:39.437 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.437 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.437 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:170489 5.a 5:5f4ac418:::benchmark_data_smithi179_115199_object41583:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e521)", 2024-02-09T17:43:39.437 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:43:24.179483+0000", 2024-02-09T17:43:39.437 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 15.171147653, 2024-02-09T17:43:39.437 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7495300939999998, 2024-02-09T17:43:39.437 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:39.437 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:39.437 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:39.437 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:43:39.437 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:43:39.438 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 170489 2024-02-09T17:43:39.438 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.438 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:39.438 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.438 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:39.438 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.179483+0000", 2024-02-09T17:43:39.438 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:39.438 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.438 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.438 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:39.438 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.179483+0000", 2024-02-09T17:43:39.438 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:39.438 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.438 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.438 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:39.438 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.179484+0000", 2024-02-09T17:43:39.439 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.46e-06 2024-02-09T17:43:39.439 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.439 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.439 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:39.439 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.179489+0000", 2024-02-09T17:43:39.439 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1259999999999997e-06 2024-02-09T17:43:39.439 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.439 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.439 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:39.439 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.179501+0000", 2024-02-09T17:43:39.439 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1796e-05 2024-02-09T17:43:39.439 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.439 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.439 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:39.440 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.179509+0000", 2024-02-09T17:43:39.440 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.9270000000000005e-06 2024-02-09T17:43:39.440 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.440 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.440 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:39.440 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.179568+0000", 2024-02-09T17:43:39.440 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8860999999999998e-05 2024-02-09T17:43:39.440 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.440 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.440 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:43:39.440 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.179577+0000", 2024-02-09T17:43:39.440 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.3470000000000005e-06 2024-02-09T17:43:39.440 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.441 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.441 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:39.441 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:25.163622+0000", 2024-02-09T17:43:39.441 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.98404519400000001 2024-02-09T17:43:39.441 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.441 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.441 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:43:39.441 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:25.163631+0000", 2024-02-09T17:43:39.441 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7399999999999993e-06 2024-02-09T17:43:39.441 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.441 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.441 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:39.441 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.912873+0000", 2024-02-09T17:43:39.441 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.749241482 2024-02-09T17:43:39.441 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.442 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.442 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:39.442 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.913065+0000", 2024-02-09T17:43:39.442 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00019188300000000001 2024-02-09T17:43:39.442 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.442 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.442 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,4,10", 2024-02-09T17:43:39.442 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.913234+0000", 2024-02-09T17:43:39.442 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016940999999999999 2024-02-09T17:43:39.442 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.442 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.442 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:39.442 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.918189+0000", 2024-02-09T17:43:39.442 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0049548320000000002 2024-02-09T17:43:39.443 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.443 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.443 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:39.443 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.928556+0000", 2024-02-09T17:43:39.443 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010366820000000001 2024-02-09T17:43:39.443 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.443 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.443 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:39.443 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.928567+0000", 2024-02-09T17:43:39.443 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1113e-05 2024-02-09T17:43:39.443 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.443 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.443 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:39.443 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.928861+0000", 2024-02-09T17:43:39.443 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00029464699999999999 2024-02-09T17:43:39.444 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.444 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.444 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:39.444 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.928914+0000", 2024-02-09T17:43:39.444 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2757999999999999e-05 2024-02-09T17:43:39.444 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.444 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.444 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:39.444 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.929013+0000", 2024-02-09T17:43:39.444 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.8697999999999996e-05 2024-02-09T17:43:39.444 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:39.444 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:39.444 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:39.444 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.445 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.445 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:170494 5.a 5:531d98cd:::benchmark_data_smithi179_115199_object41588:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e521)", 2024-02-09T17:43:39.445 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:43:24.189636+0000", 2024-02-09T17:43:39.445 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 15.160994651999999, 2024-02-09T17:43:39.445 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7396196609999999, 2024-02-09T17:43:39.445 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:39.445 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:39.445 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:39.445 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:43:39.445 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:43:39.445 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 170494 2024-02-09T17:43:39.445 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.445 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:39.445 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.445 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:39.446 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.189636+0000", 2024-02-09T17:43:39.446 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:39.446 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.446 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.446 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:39.446 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.189636+0000", 2024-02-09T17:43:39.446 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:39.446 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.446 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.446 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:39.446 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.189638+0000", 2024-02-09T17:43:39.446 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9230000000000001e-06 2024-02-09T17:43:39.446 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.446 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.447 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:39.447 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.189645+0000", 2024-02-09T17:43:39.447 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2980000000000003e-06 2024-02-09T17:43:39.447 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.447 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.447 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:39.447 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.189664+0000", 2024-02-09T17:43:39.447 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8788999999999999e-05 2024-02-09T17:43:39.447 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.447 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.447 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:39.447 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.189675+0000", 2024-02-09T17:43:39.447 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0852999999999999e-05 2024-02-09T17:43:39.447 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.448 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.448 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:39.448 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.189749+0000", 2024-02-09T17:43:39.448 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4883999999999997e-05 2024-02-09T17:43:39.448 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.448 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.448 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:43:39.448 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.189757+0000", 2024-02-09T17:43:39.448 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8550000000000002e-06 2024-02-09T17:43:39.448 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.448 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.448 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:39.448 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:25.163659+0000", 2024-02-09T17:43:39.448 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.973901825 2024-02-09T17:43:39.449 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.449 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.449 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:43:39.449 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:25.163667+0000", 2024-02-09T17:43:39.449 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2009999999999999e-06 2024-02-09T17:43:39.449 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.449 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.449 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:39.449 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.918248+0000", 2024-02-09T17:43:39.449 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.754580443 2024-02-09T17:43:39.449 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.449 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.449 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:39.449 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.918384+0000", 2024-02-09T17:43:39.450 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013572200000000001 2024-02-09T17:43:39.450 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.450 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.450 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,4,10", 2024-02-09T17:43:39.450 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.918533+0000", 2024-02-09T17:43:39.450 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014900800000000001 2024-02-09T17:43:39.450 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.450 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.450 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:39.450 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.928820+0000", 2024-02-09T17:43:39.450 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010287057 2024-02-09T17:43:39.450 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.450 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.450 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:39.450 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.928829+0000", 2024-02-09T17:43:39.450 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2799999999999992e-06 2024-02-09T17:43:39.451 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.451 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.451 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:39.451 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.929050+0000", 2024-02-09T17:43:39.451 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00022129300000000001 2024-02-09T17:43:39.451 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.451 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.451 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:39.451 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.929128+0000", 2024-02-09T17:43:39.451 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7399999999999998e-05 2024-02-09T17:43:39.451 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.451 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.451 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:39.452 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.929177+0000", 2024-02-09T17:43:39.452 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9469999999999999e-05 2024-02-09T17:43:39.452 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.452 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.452 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:39.452 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.929255+0000", 2024-02-09T17:43:39.452 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8360000000000002e-05 2024-02-09T17:43:39.452 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:39.452 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:39.452 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:39.452 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.452 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.452 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:170511 5.a 5:5e38a700:::benchmark_data_smithi179_115199_object41605:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e521)", 2024-02-09T17:43:39.455 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:43:24.214457+0000", 2024-02-09T17:43:39.455 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 15.136173717, 2024-02-09T17:43:39.455 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7247621959999999, 2024-02-09T17:43:39.456 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:39.456 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:39.456 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:39.456 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:43:39.456 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:43:39.456 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 170511 2024-02-09T17:43:39.456 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.456 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:39.456 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.456 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:39.456 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.214457+0000", 2024-02-09T17:43:39.456 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:39.456 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.456 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.456 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:39.457 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.214457+0000", 2024-02-09T17:43:39.457 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:39.457 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.457 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.457 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:39.457 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.214458+0000", 2024-02-09T17:43:39.457 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3039999999999999e-06 2024-02-09T17:43:39.457 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.457 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.457 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:39.457 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.214463+0000", 2024-02-09T17:43:39.457 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.361e-06 2024-02-09T17:43:39.457 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.457 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.457 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:39.458 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.214475+0000", 2024-02-09T17:43:39.458 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2002e-05 2024-02-09T17:43:39.458 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.458 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.458 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:39.458 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.214482+0000", 2024-02-09T17:43:39.458 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7070000000000003e-06 2024-02-09T17:43:39.458 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.458 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.458 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:39.458 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.214545+0000", 2024-02-09T17:43:39.458 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2873999999999995e-05 2024-02-09T17:43:39.458 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.458 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.459 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:43:39.459 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.214556+0000", 2024-02-09T17:43:39.459 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0861e-05 2024-02-09T17:43:39.459 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.459 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.459 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:39.459 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:25.163694+0000", 2024-02-09T17:43:39.459 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.949138433 2024-02-09T17:43:39.459 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.459 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.459 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:43:39.459 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:25.163702+0000", 2024-02-09T17:43:39.459 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7749999999999993e-06 2024-02-09T17:43:39.459 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.460 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.460 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:39.460 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.923439+0000", 2024-02-09T17:43:39.460 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7597368950000001 2024-02-09T17:43:39.460 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.460 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.460 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:39.460 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.923590+0000", 2024-02-09T17:43:39.460 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000151454 2024-02-09T17:43:39.460 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.460 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.460 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,4,10", 2024-02-09T17:43:39.460 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.923718+0000", 2024-02-09T17:43:39.460 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012752299999999999 2024-02-09T17:43:39.461 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.461 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.461 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:39.461 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.929059+0000", 2024-02-09T17:43:39.461 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.005340951 2024-02-09T17:43:39.461 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.461 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.461 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:39.461 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.929092+0000", 2024-02-09T17:43:39.461 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2855999999999998e-05 2024-02-09T17:43:39.461 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.461 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.461 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:39.461 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.929305+0000", 2024-02-09T17:43:39.462 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00021333499999999999 2024-02-09T17:43:39.462 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.462 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.462 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:39.462 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.939072+0000", 2024-02-09T17:43:39.462 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0097671960000000006 2024-02-09T17:43:39.462 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.462 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.462 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:39.462 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.939118+0000", 2024-02-09T17:43:39.462 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5760000000000002e-05 2024-02-09T17:43:39.462 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.462 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.462 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:39.462 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.939219+0000", 2024-02-09T17:43:39.463 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010090900000000001 2024-02-09T17:43:39.463 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:39.463 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:39.463 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:39.463 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.463 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.463 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:170531 5.a 5:5cd92cb3:::benchmark_data_smithi179_115199_object41625:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e521)", 2024-02-09T17:43:39.463 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:43:24.268530+0000", 2024-02-09T17:43:39.463 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 15.082100273, 2024-02-09T17:43:39.463 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.670929031, 2024-02-09T17:43:39.463 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:39.463 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:39.463 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:39.463 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:43:39.464 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:43:39.464 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 170531 2024-02-09T17:43:39.464 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.464 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:39.464 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.464 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:39.464 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.268530+0000", 2024-02-09T17:43:39.464 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:39.464 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.464 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.464 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:39.464 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.268530+0000", 2024-02-09T17:43:39.464 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:39.464 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.465 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.465 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:39.465 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.268532+0000", 2024-02-09T17:43:39.465 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5850000000000001e-06 2024-02-09T17:43:39.465 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.465 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.465 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:39.466 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:38 smithi162 ceph-mon[44304]: 5.5 scrub ok 2024-02-09T17:43:39.466 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:38 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:43:39.466 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:38 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:43:39.466 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:38 smithi162 ceph-mon[44304]: 5.e scrub starts 2024-02-09T17:43:39.466 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:38 smithi162 ceph-mon[44304]: osdmap e530: 12 total, 12 up, 12 in 2024-02-09T17:43:39.466 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:38 smithi162 ceph-mon[44304]: 5.c scrub starts 2024-02-09T17:43:39.466 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:38 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4049853432' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.5982631885847566}]: dispatch 2024-02-09T17:43:39.466 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:38 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.5982631885847566}]: dispatch 2024-02-09T17:43:39.467 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.268539+0000", 2024-02-09T17:43:39.467 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2479999999999999e-06 2024-02-09T17:43:39.467 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.467 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.467 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:39.467 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.268555+0000", 2024-02-09T17:43:39.467 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6155999999999999e-05 2024-02-09T17:43:39.467 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.467 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.467 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:39.467 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.268563+0000", 2024-02-09T17:43:39.467 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1790000000000008e-06 2024-02-09T17:43:39.468 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.468 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.468 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:39.468 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.268632+0000", 2024-02-09T17:43:39.468 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8942999999999994e-05 2024-02-09T17:43:39.468 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.468 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.468 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:43:39.468 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.268642+0000", 2024-02-09T17:43:39.468 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7380000000000006e-06 2024-02-09T17:43:39.468 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.468 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.468 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:39.468 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:25.163729+0000", 2024-02-09T17:43:39.468 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.89508700100000005 2024-02-09T17:43:39.469 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.469 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.469 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:43:39.469 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:25.163737+0000", 2024-02-09T17:43:39.469 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.0309999999999998e-06 2024-02-09T17:43:39.469 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.469 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.469 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:39.469 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.929339+0000", 2024-02-09T17:43:39.469 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.765602248 2024-02-09T17:43:39.469 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.469 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.469 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:39.469 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.929418+0000", 2024-02-09T17:43:39.470 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8811000000000001e-05 2024-02-09T17:43:39.470 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.470 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.470 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,4,10", 2024-02-09T17:43:39.470 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.929501+0000", 2024-02-09T17:43:39.470 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3084000000000006e-05 2024-02-09T17:43:39.470 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.470 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.470 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:39.470 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.939270+0000", 2024-02-09T17:43:39.470 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0097685900000000006 2024-02-09T17:43:39.470 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.470 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.470 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:39.471 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.939280+0000", 2024-02-09T17:43:39.471 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0013999999999999e-05 2024-02-09T17:43:39.471 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.471 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.471 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:39.471 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.939317+0000", 2024-02-09T17:43:39.471 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7728999999999997e-05 2024-02-09T17:43:39.471 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.471 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.471 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:39.471 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.939344+0000", 2024-02-09T17:43:39.471 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6508000000000001e-05 2024-02-09T17:43:39.471 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.471 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.471 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:39.472 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.939380+0000", 2024-02-09T17:43:39.472 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.5670999999999998e-05 2024-02-09T17:43:39.472 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.472 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.472 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:39.472 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.939459+0000", 2024-02-09T17:43:39.472 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.9494999999999997e-05 2024-02-09T17:43:39.472 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:39.472 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:39.472 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:39.472 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.472 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.472 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:170541 5.a 5:5a433982:::benchmark_data_smithi179_115199_object41635:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e521)", 2024-02-09T17:43:39.472 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:43:24.328851+0000", 2024-02-09T17:43:39.473 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 15.021779782999999, 2024-02-09T17:43:39.473 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6110777179999998, 2024-02-09T17:43:39.473 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:39.473 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:39.473 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:39.473 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:43:39.473 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:43:39.473 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 170541 2024-02-09T17:43:39.473 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.473 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:39.473 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.473 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:39.473 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.328851+0000", 2024-02-09T17:43:39.473 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:39.474 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.474 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.474 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:39.474 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.328851+0000", 2024-02-09T17:43:39.474 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:39.474 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.474 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.474 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:39.474 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.328852+0000", 2024-02-09T17:43:39.474 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.437e-06 2024-02-09T17:43:39.474 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.474 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.474 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:39.474 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.328857+0000", 2024-02-09T17:43:39.474 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3519999999999997e-06 2024-02-09T17:43:39.475 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.475 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.475 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:39.475 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.328873+0000", 2024-02-09T17:43:39.475 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5162e-05 2024-02-09T17:43:39.475 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.475 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.475 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:39.475 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.328883+0000", 2024-02-09T17:43:39.475 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0835999999999999e-05 2024-02-09T17:43:39.475 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.475 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.475 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:39.475 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.328969+0000", 2024-02-09T17:43:39.475 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5359999999999996e-05 2024-02-09T17:43:39.475 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.476 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.476 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:43:39.476 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.328981+0000", 2024-02-09T17:43:39.476 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2507000000000001e-05 2024-02-09T17:43:39.476 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.476 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.476 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:39.476 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:25.163829+0000", 2024-02-09T17:43:39.476 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.83484808200000005 2024-02-09T17:43:39.476 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.476 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.476 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:43:39.476 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:25.163840+0000", 2024-02-09T17:43:39.476 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0665999999999999e-05 2024-02-09T17:43:39.477 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.477 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.477 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:39.477 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.934605+0000", 2024-02-09T17:43:39.477 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7707647199999998 2024-02-09T17:43:39.477 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.477 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.477 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:39.477 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.934723+0000", 2024-02-09T17:43:39.477 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000117971 2024-02-09T17:43:39.477 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.477 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.477 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,4,10", 2024-02-09T17:43:39.477 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.934854+0000", 2024-02-09T17:43:39.477 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013093800000000001 2024-02-09T17:43:39.478 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.478 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.478 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:39.478 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.939468+0000", 2024-02-09T17:43:39.478 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0046148450000000002 2024-02-09T17:43:39.478 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.478 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.478 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:39.478 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.939498+0000", 2024-02-09T17:43:39.478 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0018999999999998e-05 2024-02-09T17:43:39.478 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.478 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.478 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:39.478 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.939525+0000", 2024-02-09T17:43:39.478 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6771000000000002e-05 2024-02-09T17:43:39.478 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.478 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.479 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:39.479 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.939836+0000", 2024-02-09T17:43:39.479 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00031041700000000001 2024-02-09T17:43:39.479 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.479 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.479 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:39.479 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.939870+0000", 2024-02-09T17:43:39.479 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4042999999999999e-05 2024-02-09T17:43:39.479 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.479 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.479 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:39.479 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.939928+0000", 2024-02-09T17:43:39.479 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8591999999999997e-05 2024-02-09T17:43:39.479 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:39.479 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:39.479 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:39.480 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.480 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.480 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:179734 5.a 5:591b0597:::benchmark_data_smithi179_115199_object50828:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e530)", 2024-02-09T17:43:39.480 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:43:38.681441+0000", 2024-02-09T17:43:39.480 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.66918969800000005, 2024-02-09T17:43:39.480 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.047332107999999998, 2024-02-09T17:43:39.480 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:39.480 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:39.480 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:39.480 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:43:39.480 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:43:39.480 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 179734 2024-02-09T17:43:39.480 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.480 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:39.481 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.481 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:39.481 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:38.681441+0000", 2024-02-09T17:43:39.481 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:39.481 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.481 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.481 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:39.481 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:38.681441+0000", 2024-02-09T17:43:39.481 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:39.481 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.481 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.481 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:39.481 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:38.681442+0000", 2024-02-09T17:43:39.481 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2839999999999999e-06 2024-02-09T17:43:39.481 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.481 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.482 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:39.482 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:38.681447+0000", 2024-02-09T17:43:39.482 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.339e-06 2024-02-09T17:43:39.482 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.482 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.482 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:39.482 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:38.681457+0000", 2024-02-09T17:43:39.482 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2590000000000006e-06 2024-02-09T17:43:39.482 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.482 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.482 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:39.482 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:38.681464+0000", 2024-02-09T17:43:39.482 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2960000000000001e-06 2024-02-09T17:43:39.482 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.482 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.483 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:39.483 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:38.681512+0000", 2024-02-09T17:43:39.483 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7942000000000003e-05 2024-02-09T17:43:39.483 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.483 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.483 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:39.483 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:38.681608+0000", 2024-02-09T17:43:39.483 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6475e-05 2024-02-09T17:43:39.483 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.483 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.483 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,4,10", 2024-02-09T17:43:39.483 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:38.681691+0000", 2024-02-09T17:43:39.483 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3143e-05 2024-02-09T17:43:39.483 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.484 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.484 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:39.484 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:38.688840+0000", 2024-02-09T17:43:39.484 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0071488059999999997 2024-02-09T17:43:39.484 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.484 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.484 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:39.484 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:38.688856+0000", 2024-02-09T17:43:39.484 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5667000000000001e-05 2024-02-09T17:43:39.484 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.484 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.484 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:39.484 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:38.691374+0000", 2024-02-09T17:43:39.484 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0025184209999999999 2024-02-09T17:43:39.485 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.485 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.485 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:39.485 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:38.728508+0000", 2024-02-09T17:43:39.485 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.037133748000000001 2024-02-09T17:43:39.485 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.485 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.485 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:39.485 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:38.728642+0000", 2024-02-09T17:43:39.485 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013436399999999999 2024-02-09T17:43:39.485 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.485 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.485 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:39.485 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:38.728773+0000", 2024-02-09T17:43:39.486 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000130364 2024-02-09T17:43:39.486 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:39.486 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:39.486 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:39.486 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.486 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.486 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:179737 5.a 5:5f60bf49:::benchmark_data_smithi179_115199_object50831:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e530)", 2024-02-09T17:43:39.486 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:43:38.682809+0000", 2024-02-09T17:43:39.486 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.66782159100000005, 2024-02-09T17:43:39.486 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.053698149000000001, 2024-02-09T17:43:39.486 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:43:39.486 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:43:39.486 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:43:39.486 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:43:39.487 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:43:39.487 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 179737 2024-02-09T17:43:39.487 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.487 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:43:39.487 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.487 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:43:39.487 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:38.682809+0000", 2024-02-09T17:43:39.487 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:39.487 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.487 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.487 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:43:39.487 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:38.682809+0000", 2024-02-09T17:43:39.487 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:43:39.488 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.488 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.488 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:43:39.488 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:38.682810+0000", 2024-02-09T17:43:39.488 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.0500000000000002e-07 2024-02-09T17:43:39.488 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.488 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.488 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:43:39.488 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:38.682814+0000", 2024-02-09T17:43:39.488 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.69e-06 2024-02-09T17:43:39.488 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.488 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.488 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:43:39.488 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:38.682824+0000", 2024-02-09T17:43:39.488 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5710000000000004e-06 2024-02-09T17:43:39.489 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.489 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.489 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:43:39.489 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:38.682829+0000", 2024-02-09T17:43:39.489 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6400000000000002e-06 2024-02-09T17:43:39.489 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.489 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.489 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:43:39.489 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:38.685427+0000", 2024-02-09T17:43:39.489 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002597041 2024-02-09T17:43:39.489 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.489 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.489 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:43:39.489 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:38.685538+0000", 2024-02-09T17:43:39.490 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000111177 2024-02-09T17:43:39.490 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.490 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.490 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,4,10", 2024-02-09T17:43:39.490 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:38.685618+0000", 2024-02-09T17:43:39.490 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.0465999999999998e-05 2024-02-09T17:43:39.490 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.490 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.490 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:39.490 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:38.688892+0000", 2024-02-09T17:43:39.490 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0032740719999999998 2024-02-09T17:43:39.490 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.490 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.490 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:43:39.490 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:38.688945+0000", 2024-02-09T17:43:39.491 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2500999999999998e-05 2024-02-09T17:43:39.491 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.491 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.491 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:39.491 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:38.695424+0000", 2024-02-09T17:43:39.491 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0064795 2024-02-09T17:43:39.491 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.491 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.491 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:43:39.491 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:38.736361+0000", 2024-02-09T17:43:39.491 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.040937144000000002 2024-02-09T17:43:39.491 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.491 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.491 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:43:39.491 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:38.736406+0000", 2024-02-09T17:43:39.492 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4413999999999999e-05 2024-02-09T17:43:39.492 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:43:39.492 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:43:39.492 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:43:39.492 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:38.736507+0000", 2024-02-09T17:43:39.492 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000101128 2024-02-09T17:43:39.492 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:39.492 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:39.492 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:39.492 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:43:39.492 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:43:39.492 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:43:39.493 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:38 smithi179 ceph-mon[43975]: 5.5 scrub ok 2024-02-09T17:43:39.493 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:38 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:43:39.493 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:38 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:43:39.493 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:38 smithi179 ceph-mon[43975]: 5.e scrub starts 2024-02-09T17:43:39.493 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:38 smithi179 ceph-mon[43975]: osdmap e530: 12 total, 12 up, 12 in 2024-02-09T17:43:39.493 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:38 smithi179 ceph-mon[43975]: 5.c scrub starts 2024-02-09T17:43:39.493 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:38 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4049853432' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.5982631885847566}]: dispatch 2024-02-09T17:43:39.493 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:38 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.5982631885847566}]: dispatch 2024-02-09T17:43:39.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:39 smithi099 ceph-mon[39420]: 1.f scrub starts 2024-02-09T17:43:39.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:39 smithi099 ceph-mon[39420]: 1.f scrub ok 2024-02-09T17:43:39.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:39 smithi099 ceph-mon[39420]: pgmap v1337: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 2 active+remapped+backfilling, 43 active+clean; 4.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 732 op/s; 11616/240519 objects misplaced (4.830%); 24 MiB/s, 380 objects/s recovering 2024-02-09T17:43:39.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:39 smithi099 ceph-mon[39420]: 5.e scrub ok 2024-02-09T17:43:39.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:39 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.5982631885847566}]': finished 2024-02-09T17:43:39.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:39 smithi099 ceph-mon[39420]: osdmap e531: 12 total, 12 up, 12 in 2024-02-09T17:43:39.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:39 smithi099 ceph-mon[39420]: 5.c scrub ok 2024-02-09T17:43:40.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:39 smithi179 ceph-mon[43975]: 1.f scrub starts 2024-02-09T17:43:40.426 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:39 smithi179 ceph-mon[43975]: 1.f scrub ok 2024-02-09T17:43:40.426 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:39 smithi179 ceph-mon[43975]: pgmap v1337: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 2 active+remapped+backfilling, 43 active+clean; 4.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 732 op/s; 11616/240519 objects misplaced (4.830%); 24 MiB/s, 380 objects/s recovering 2024-02-09T17:43:40.426 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:39 smithi179 ceph-mon[43975]: 5.e scrub ok 2024-02-09T17:43:40.426 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:39 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.5982631885847566}]': finished 2024-02-09T17:43:40.426 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:39 smithi179 ceph-mon[43975]: osdmap e531: 12 total, 12 up, 12 in 2024-02-09T17:43:40.426 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:39 smithi179 ceph-mon[43975]: 5.c scrub ok 2024-02-09T17:43:40.434 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:39 smithi162 ceph-mon[44304]: 1.f scrub starts 2024-02-09T17:43:40.434 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:39 smithi162 ceph-mon[44304]: 1.f scrub ok 2024-02-09T17:43:40.434 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:39 smithi162 ceph-mon[44304]: pgmap v1337: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 2 active+remapped+backfilling, 43 active+clean; 4.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 732 op/s; 11616/240519 objects misplaced (4.830%); 24 MiB/s, 380 objects/s recovering 2024-02-09T17:43:40.435 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:39 smithi162 ceph-mon[44304]: 5.e scrub ok 2024-02-09T17:43:40.435 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:39 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.5982631885847566}]': finished 2024-02-09T17:43:40.435 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:39 smithi162 ceph-mon[44304]: osdmap e531: 12 total, 12 up, 12 in 2024-02-09T17:43:40.435 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:39 smithi162 ceph-mon[44304]: 5.c scrub ok 2024-02-09T17:43:40.465 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-09T17:43:40.506 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:43:40.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:40.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:43:40.618 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:43:40.654 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:40.791 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:43:40.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:40.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:43:40.898 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:43:40.899 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:43:40.903 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:43:40.953 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:40.989 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:43:41.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:41.002 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:43:41.103 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:43:41.147 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:40 smithi099 ceph-mon[39420]: osdmap e532: 12 total, 12 up, 12 in 2024-02-09T17:43:41.151 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:41.190 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:43:41.199 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:41.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:43:41.301 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:43:41.333 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:41.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:40 smithi179 ceph-mon[43975]: osdmap e532: 12 total, 12 up, 12 in 2024-02-09T17:43:41.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:40 smithi162 ceph-mon[44304]: osdmap e532: 12 total, 12 up, 12 in 2024-02-09T17:43:41.467 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:43:41.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:41.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:43:41.553 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is unset 2024-02-09T17:43:41.572 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:43:41.632 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:41.664 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:43:41.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:41.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:43:41.770 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:43:41.832 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:41.866 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:43:41.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:41.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:43:41.973 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:43:41.991 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:41 smithi099 ceph-mon[39420]: 5.b scrub starts 2024-02-09T17:43:42.024 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:42.084 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:43:42.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:42.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:43:42.193 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:43:42.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:41 smithi099 ceph-mon[39420]: pgmap v1340: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 2 active+remapped+backfilling, 43 active+clean; 4.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 745 op/s; 10684/234909 objects misplaced (4.548%); 18 MiB/s, 286 objects/s recovering 2024-02-09T17:43:42.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:41 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2509908391' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:43:42.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:41 smithi099 ceph-mon[39420]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:43:42.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:41 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2509908391' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:43:42.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:41 smithi099 ceph-mon[39420]: osdmap e533: 12 total, 12 up, 12 in 2024-02-09T17:43:42.245 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:42.289 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:43:42.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:42.295 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:41 smithi179 ceph-mon[43975]: 5.b scrub starts 2024-02-09T17:43:42.295 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:41 smithi179 ceph-mon[43975]: pgmap v1340: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 2 active+remapped+backfilling, 43 active+clean; 4.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 745 op/s; 10684/234909 objects misplaced (4.548%); 18 MiB/s, 286 objects/s recovering 2024-02-09T17:43:42.295 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:41 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2509908391' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:43:42.295 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:41 smithi179 ceph-mon[43975]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:43:42.295 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:41 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2509908391' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:43:42.295 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:41 smithi179 ceph-mon[43975]: osdmap e533: 12 total, 12 up, 12 in 2024-02-09T17:43:42.296 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:43:42.397 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:43:42.433 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:41 smithi162 ceph-mon[44304]: 5.b scrub starts 2024-02-09T17:43:42.433 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:41 smithi162 ceph-mon[44304]: pgmap v1340: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 2 active+remapped+backfilling, 43 active+clean; 4.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 745 op/s; 10684/234909 objects misplaced (4.548%); 18 MiB/s, 286 objects/s recovering 2024-02-09T17:43:42.433 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:41 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2509908391' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:43:42.433 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:41 smithi162 ceph-mon[44304]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:43:42.433 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:41 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2509908391' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:43:42.433 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:41 smithi162 ceph-mon[44304]: osdmap e533: 12 total, 12 up, 12 in 2024-02-09T17:43:42.435 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:42.462 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:43:42.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:42.469 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:43:42.570 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:43:42.631 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:42.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:42.668 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:43:42.668 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:43:42.769 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:43:42.822 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:43.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:42 smithi179 ceph-mon[43975]: 5.b scrub ok 2024-02-09T17:43:43.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:42 smithi162 ceph-mon[44304]: 5.b scrub ok 2024-02-09T17:43:43.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:42 smithi099 ceph-mon[39420]: 5.b scrub ok 2024-02-09T17:43:43.589 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:43:43.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:43.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:43:43.705 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:43:43.745 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:43.827 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:43:43.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:43.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:43:43.933 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:43:43.983 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:44.127 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:44 smithi099 ceph-mon[39420]: pgmap v1342: 47 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 43 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 761 op/s; 10433/230664 objects misplaced (4.523%); 22 MiB/s, 352 objects/s recovering 2024-02-09T17:43:44.127 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:44 smithi099 ceph-mon[39420]: 5.1 scrub starts 2024-02-09T17:43:44.411 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:43:44.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:44 smithi179 ceph-mon[43975]: pgmap v1342: 47 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 43 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 761 op/s; 10433/230664 objects misplaced (4.523%); 22 MiB/s, 352 objects/s recovering 2024-02-09T17:43:44.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:44 smithi179 ceph-mon[43975]: 5.1 scrub starts 2024-02-09T17:43:44.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:44 smithi162 ceph-mon[44304]: pgmap v1342: 47 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 43 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 761 op/s; 10433/230664 objects misplaced (4.523%); 22 MiB/s, 352 objects/s recovering 2024-02-09T17:43:44.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:44 smithi162 ceph-mon[44304]: 5.1 scrub starts 2024-02-09T17:43:44.445 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:43:44.476 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:43:44.476 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:43:44.479 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:43:44.516 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:43:44.554 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:43:44.599 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:43:44.628 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:43:44.660 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:43:44.692 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:43:44.723 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:43:44.748 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:43:44.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:44.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:43:44.760 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:43:44.791 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:43:44.801 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:43:44.801 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:43:44.802 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:43:44.802 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:43:44.802 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:43:44.802 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:43:44.802 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:43:44.802 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:43:44.802 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:43:44.802 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:43:44.802 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:43:44.802 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:43:44.860 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:43:44.896 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:44.928 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:43:44.933 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:44.934 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:43:45.020 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:45 smithi099 ceph-mon[39420]: 5.1 scrub ok 2024-02-09T17:43:45.020 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:45 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3382397601' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:43:45.035 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:43:45.083 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:45.122 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:43:45.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:45.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:43:45.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:45 smithi179 ceph-mon[43975]: 5.1 scrub ok 2024-02-09T17:43:45.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:45 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3382397601' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:43:45.228 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:43:45.260 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] 2024-02-09T17:43:45.260 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:43:45.261 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-02-09T17:43:45.261 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph pg dump --format=json 2024-02-09T17:43:45.289 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:45.433 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:45 smithi162 ceph-mon[44304]: 5.1 scrub ok 2024-02-09T17:43:45.434 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:45 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3382397601' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:43:45.647 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:43:45.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:45.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:43:45.757 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:43:45.814 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:45.846 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:43:45.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:45.851 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:43:45.952 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:43:45.997 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:46.025 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:43:46.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:46.032 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:43:46.133 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:43:46.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:46 smithi179 ceph-mon[43975]: pgmap v1343: 47 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 43 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 844 op/s; 8567/225279 objects misplaced (3.803%); 25 MiB/s, 394 objects/s recovering 2024-02-09T17:43:46.174 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:46.211 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:43:46.215 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:46.216 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:43:46.317 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:43:46.350 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:46 smithi099 ceph-mon[39420]: pgmap v1343: 47 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 43 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 844 op/s; 8567/225279 objects misplaced (3.803%); 25 MiB/s, 394 objects/s recovering 2024-02-09T17:43:46.353 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:46.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:46 smithi162 ceph-mon[44304]: pgmap v1343: 47 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 43 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 844 op/s; 8567/225279 objects misplaced (3.803%); 25 MiB/s, 394 objects/s recovering 2024-02-09T17:43:46.811 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:43:46.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:46.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:43:46.820 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-09T17:43:46.826 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:43:46.920 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:43:46.958 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:46.995 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:43:47.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:47.003 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:43:47.104 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:43:47.136 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:47.168 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:43:47.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:47.181 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:43:47.282 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:43:47.317 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:47 smithi099 ceph-mon[39420]: 5.2 scrub starts 2024-02-09T17:43:47.318 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:47 smithi099 ceph-mon[39420]: 1.11 scrub starts 2024-02-09T17:43:47.318 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:47 smithi099 ceph-mon[39420]: 1.11 scrub ok 2024-02-09T17:43:47.325 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:47.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:47 smithi179 ceph-mon[43975]: 5.2 scrub starts 2024-02-09T17:43:47.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:47 smithi179 ceph-mon[43975]: 1.11 scrub starts 2024-02-09T17:43:47.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:47 smithi179 ceph-mon[43975]: 1.11 scrub ok 2024-02-09T17:43:47.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:47 smithi162 ceph-mon[44304]: 5.2 scrub starts 2024-02-09T17:43:47.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:47 smithi162 ceph-mon[44304]: 1.11 scrub starts 2024-02-09T17:43:47.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:47 smithi162 ceph-mon[44304]: 1.11 scrub ok 2024-02-09T17:43:47.434 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:43:47.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:47.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:43:47.539 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:43:47.579 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:47.668 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:43:47.839 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:43:47.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:47.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:43:47.871 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:43:47.948 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:43:48.014 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:48.055 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set noscrub 2024-02-09T17:43:48.067 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:43:48.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:48.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:43:48.097 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:48 smithi099 ceph-mon[39420]: pgmap v1344: 47 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 43 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 835 op/s; 8567/220023 objects misplaced (3.894%); 19 MiB/s, 300 objects/s recovering 2024-02-09T17:43:48.097 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:48 smithi099 ceph-mon[39420]: 5.2 scrub ok 2024-02-09T17:43:48.175 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:43:48.218 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:48.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:48 smithi179 ceph-mon[43975]: pgmap v1344: 47 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 43 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 835 op/s; 8567/220023 objects misplaced (3.894%); 19 MiB/s, 300 objects/s recovering 2024-02-09T17:43:48.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:48 smithi179 ceph-mon[43975]: 5.2 scrub ok 2024-02-09T17:43:48.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:48 smithi162 ceph-mon[44304]: pgmap v1344: 47 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 43 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 835 op/s; 8567/220023 objects misplaced (3.894%); 19 MiB/s, 300 objects/s recovering 2024-02-09T17:43:48.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:48 smithi162 ceph-mon[44304]: 5.2 scrub ok 2024-02-09T17:43:49.961 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:43:49.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:49.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:43:50.093 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:43:50.124 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:50 smithi099 ceph-mon[39420]: pgmap v1345: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 2 active+remapped+backfilling, 43 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 744 op/s; 7196/217137 objects misplaced (3.314%); 19 MiB/s, 303 objects/s recovering 2024-02-09T17:43:50.126 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:50.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:50 smithi179 ceph-mon[43975]: pgmap v1345: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 2 active+remapped+backfilling, 43 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 744 op/s; 7196/217137 objects misplaced (3.314%); 19 MiB/s, 303 objects/s recovering 2024-02-09T17:43:50.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:50 smithi162 ceph-mon[44304]: pgmap v1345: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 2 active+remapped+backfilling, 43 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 744 op/s; 7196/217137 objects misplaced (3.314%); 19 MiB/s, 303 objects/s recovering 2024-02-09T17:43:51.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:51 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:43:51.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:51 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:43:51.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:51 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:43:51.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:51 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:43:51.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:51 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:43:51.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:51 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:43:51.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:51 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:43:51.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:51 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:43:51.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:51 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:43:51.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:51 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:43:51.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:51 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:43:51.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:51 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:43:52.021 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:43:52.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:52.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:43:52.125 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:43:52.158 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:52.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:52 smithi099 ceph-mon[39420]: pgmap v1346: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 2 active+remapped+backfilling, 43 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 801 op/s; 6353/210846 objects misplaced (3.013%); 21 MiB/s, 342 objects/s recovering 2024-02-09T17:43:52.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:52 smithi099 ceph-mon[39420]: osdmap e534: 12 total, 12 up, 12 in 2024-02-09T17:43:52.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:52 smithi099 ceph-mon[39420]: osdmap e535: 12 total, 12 up, 12 in 2024-02-09T17:43:52.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:52 smithi099 ceph-mon[39420]: 5.0 scrub starts 2024-02-09T17:43:52.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:52 smithi179 ceph-mon[43975]: pgmap v1346: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 2 active+remapped+backfilling, 43 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 801 op/s; 6353/210846 objects misplaced (3.013%); 21 MiB/s, 342 objects/s recovering 2024-02-09T17:43:52.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:52 smithi179 ceph-mon[43975]: osdmap e534: 12 total, 12 up, 12 in 2024-02-09T17:43:52.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:52 smithi179 ceph-mon[43975]: osdmap e535: 12 total, 12 up, 12 in 2024-02-09T17:43:52.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:52 smithi179 ceph-mon[43975]: 5.0 scrub starts 2024-02-09T17:43:52.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:52 smithi162 ceph-mon[44304]: pgmap v1346: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 2 active+remapped+backfilling, 43 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 801 op/s; 6353/210846 objects misplaced (3.013%); 21 MiB/s, 342 objects/s recovering 2024-02-09T17:43:52.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:52 smithi162 ceph-mon[44304]: osdmap e534: 12 total, 12 up, 12 in 2024-02-09T17:43:52.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:52 smithi162 ceph-mon[44304]: osdmap e535: 12 total, 12 up, 12 in 2024-02-09T17:43:52.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:52 smithi162 ceph-mon[44304]: 5.0 scrub starts 2024-02-09T17:43:52.810 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:43:52.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:52.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:43:52.923 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:43:52.998 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:53.030 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:43:53.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:53.034 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:43:53.134 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:43:53.135 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:43:53.138 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-02-09T17:43:53.166 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:43:53.207 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:53.851 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:53 smithi099 ceph-mon[39420]: 5.0 scrub ok 2024-02-09T17:43:53.851 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:53 smithi099 ceph-mon[39420]: osdmap e536: 12 total, 12 up, 12 in 2024-02-09T17:43:53.851 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:53 smithi099 ceph-mon[39420]: pgmap v1350: 47 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 44 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 760 op/s; 6116/206325 objects misplaced (2.964%); 19 MiB/s, 301 objects/s recovering 2024-02-09T17:43:53.851 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:53 smithi099 ceph-mon[39420]: 1.19 scrub starts 2024-02-09T17:43:53.851 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:53 smithi099 ceph-mon[39420]: 1.3 scrub starts 2024-02-09T17:43:53.851 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:53 smithi099 ceph-mon[39420]: from='client.16008 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:43:53.851 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:53 smithi099 ceph-mon[39420]: 1.3 scrub ok 2024-02-09T17:43:53.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:53 smithi179 ceph-mon[43975]: 5.0 scrub ok 2024-02-09T17:43:53.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:53 smithi179 ceph-mon[43975]: osdmap e536: 12 total, 12 up, 12 in 2024-02-09T17:43:53.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:53 smithi179 ceph-mon[43975]: pgmap v1350: 47 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 44 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 760 op/s; 6116/206325 objects misplaced (2.964%); 19 MiB/s, 301 objects/s recovering 2024-02-09T17:43:53.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:53 smithi179 ceph-mon[43975]: 1.19 scrub starts 2024-02-09T17:43:53.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:53 smithi179 ceph-mon[43975]: 1.3 scrub starts 2024-02-09T17:43:53.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:53 smithi179 ceph-mon[43975]: from='client.16008 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:43:53.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:53 smithi179 ceph-mon[43975]: 1.3 scrub ok 2024-02-09T17:43:53.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:53 smithi162 ceph-mon[44304]: 5.0 scrub ok 2024-02-09T17:43:53.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:53 smithi162 ceph-mon[44304]: osdmap e536: 12 total, 12 up, 12 in 2024-02-09T17:43:53.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:53 smithi162 ceph-mon[44304]: pgmap v1350: 47 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 44 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 760 op/s; 6116/206325 objects misplaced (2.964%); 19 MiB/s, 301 objects/s recovering 2024-02-09T17:43:53.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:53 smithi162 ceph-mon[44304]: 1.19 scrub starts 2024-02-09T17:43:53.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:53 smithi162 ceph-mon[44304]: 1.3 scrub starts 2024-02-09T17:43:53.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:53 smithi162 ceph-mon[44304]: from='client.16008 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:43:53.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:53 smithi162 ceph-mon[44304]: 1.3 scrub ok 2024-02-09T17:43:53.933 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:43:53.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:53.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:43:54.041 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:43:54.075 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:54.099 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:43:54.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:54.115 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:43:54.216 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:43:54.263 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:54.296 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:43:54.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:54.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:43:54.417 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:43:54.455 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:54.488 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:43:54.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:54.492 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:43:54.593 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:43:54.630 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:54.801 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:43:54.836 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:43:54.836 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":1350,"stamp":"2024-02-09T17:43:52.841416+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4507518730,"num_objects":68775,"num_object_clones":0,"num_object_copies":206325,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6116,"num_objects_unfound":0,"num_objects_dirty":68775,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":189041,"num_write_kb":8249882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11089,"num_bytes_recovered":727251069,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":165305,"ondisk_log_size":165305,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":144,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":19086612,"kb_used_data":13768220,"kb_used_omap":229,"kb_used_meta":5317850,"kb_avail":1105855212,"statfs":{"total":1151940427776,"available":1132395737088,"internally_reserved":0,"allocated":14098657280,"data_stored":14094327118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235127,"internal_metadata":5445478793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-299237376,"num_objects":-4566,"num_object_clones":0,"num_object_copies":-13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2451,"num_objects_unfound":0,"num_objects_dirty":-4566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4566,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1812,"num_bytes_recovered":118751232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001444"},"pg_stats":[{"pgid":"1.1e","version":"63'139","reported_seq":295,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-09T17:43:41.555892+0000","last_change":"2024-02-09T17:43:35.170863+0000","last_active":"2024-02-09T17:43:41.555892+0000","last_peered":"2024-02-09T17:43:41.555892+0000","last_clean":"2024-02-09T17:43:41.555892+0000","last_became_active":"2024-02-09T17:43:19.126757+0000","last_became_peered":"2024-02-09T17:43:19.126757+0000","last_unstale":"2024-02-09T17:43:41.555892+0000","last_undegraded":"2024-02-09T17:43:41.555892+0000","last_fullsized":"2024-02-09T17:43:41.555892+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:35.170716+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:43:35.170716+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:35.554568+0000","scrub_duration":73,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"63'139","reported_seq":320,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-09T17:43:41.552148+0000","last_change":"2024-02-09T17:43:35.227956+0000","last_active":"2024-02-09T17:43:41.552148+0000","last_peered":"2024-02-09T17:43:41.552148+0000","last_clean":"2024-02-09T17:43:41.552148+0000","last_became_active":"2024-02-09T17:43:28.594296+0000","last_became_peered":"2024-02-09T17:43:28.594296+0000","last_unstale":"2024-02-09T17:43:41.552148+0000","last_undegraded":"2024-02-09T17:43:41.552148+0000","last_fullsized":"2024-02-09T17:43:41.552148+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:35.227843+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:43:35.227843+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:59.418325+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"63'139","reported_seq":837,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-02-09T17:43:51.567488+0000","last_change":"2024-02-09T17:43:35.698701+0000","last_active":"2024-02-09T17:43:51.567488+0000","last_peered":"2024-02-09T17:43:51.567488+0000","last_clean":"2024-02-09T17:43:51.567488+0000","last_became_active":"2024-02-09T17:43:19.103584+0000","last_became_peered":"2024-02-09T17:43:19.103584+0000","last_unstale":"2024-02-09T17:43:51.567488+0000","last_undegraded":"2024-02-09T17:43:51.567488+0000","last_fullsized":"2024-02-09T17:43:51.567488+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:35.698541+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:23.912504+0000","last_clean_scrub_stamp":"2024-02-09T17:43:35.698541+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:38.487912+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"63'139","reported_seq":635,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-02-09T17:43:51.567441+0000","last_change":"2024-02-09T17:43:34.746841+0000","last_active":"2024-02-09T17:43:51.567441+0000","last_peered":"2024-02-09T17:43:51.567441+0000","last_clean":"2024-02-09T17:43:51.567441+0000","last_became_active":"2024-02-09T17:43:19.102729+0000","last_became_peered":"2024-02-09T17:43:19.102729+0000","last_unstale":"2024-02-09T17:43:51.567441+0000","last_undegraded":"2024-02-09T17:43:51.567441+0000","last_fullsized":"2024-02-09T17:43:51.567441+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:34.746704+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:43:34.746704+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:57.912996+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"63'139","reported_seq":300,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-09T17:43:41.554372+0000","last_change":"2024-02-09T17:43:35.019518+0000","last_active":"2024-02-09T17:43:41.554372+0000","last_peered":"2024-02-09T17:43:41.554372+0000","last_clean":"2024-02-09T17:43:41.554372+0000","last_became_active":"2024-02-09T17:43:19.120814+0000","last_became_peered":"2024-02-09T17:43:19.120814+0000","last_unstale":"2024-02-09T17:43:41.554372+0000","last_undegraded":"2024-02-09T17:43:41.554372+0000","last_fullsized":"2024-02-09T17:43:41.554372+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:35.019416+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:43:35.019416+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:04.082037+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"63'139","reported_seq":1120,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-09T17:43:41.548901+0000","last_change":"2024-02-09T17:43:19.121083+0000","last_active":"2024-02-09T17:43:41.548901+0000","last_peered":"2024-02-09T17:43:41.548901+0000","last_clean":"2024-02-09T17:43:41.548901+0000","last_became_active":"2024-02-09T17:43:19.110535+0000","last_became_peered":"2024-02-09T17:43:19.110535+0000","last_unstale":"2024-02-09T17:43:41.548901+0000","last_undegraded":"2024-02-09T17:43:41.548901+0000","last_fullsized":"2024-02-09T17:43:41.548901+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:42:35.605475+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:30.193194+0000","last_clean_scrub_stamp":"2024-02-09T17:42:35.605475+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:41.848932+0000","scrub_duration":102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"63'139","reported_seq":959,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-09T17:43:41.550152+0000","last_change":"2024-02-09T17:43:28.585942+0000","last_active":"2024-02-09T17:43:41.550152+0000","last_peered":"2024-02-09T17:43:41.550152+0000","last_clean":"2024-02-09T17:43:41.550152+0000","last_became_active":"2024-02-09T17:43:28.585573+0000","last_became_peered":"2024-02-09T17:43:28.585573+0000","last_unstale":"2024-02-09T17:43:41.550152+0000","last_undegraded":"2024-02-09T17:43:41.550152+0000","last_fullsized":"2024-02-09T17:43:41.550152+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:41:55.371768+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:41:55.371768+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:46.245580+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131344,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":704368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"63'139","reported_seq":409,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-02-09T17:43:51.567548+0000","last_change":"2024-02-09T17:43:36.732746+0000","last_active":"2024-02-09T17:43:51.567548+0000","last_peered":"2024-02-09T17:43:51.567548+0000","last_clean":"2024-02-09T17:43:51.567548+0000","last_became_active":"2024-02-09T17:43:19.109864+0000","last_became_peered":"2024-02-09T17:43:19.109864+0000","last_unstale":"2024-02-09T17:43:51.567548+0000","last_undegraded":"2024-02-09T17:43:51.567548+0000","last_fullsized":"2024-02-09T17:43:51.567548+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:36.732627+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:43:36.732627+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:40.299008+0000","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"63'139","reported_seq":295,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-09T17:43:41.551985+0000","last_change":"2024-02-09T17:43:36.150824+0000","last_active":"2024-02-09T17:43:41.551985+0000","last_peered":"2024-02-09T17:43:41.551985+0000","last_clean":"2024-02-09T17:43:41.551985+0000","last_became_active":"2024-02-09T17:43:19.094230+0000","last_became_peered":"2024-02-09T17:43:19.094230+0000","last_unstale":"2024-02-09T17:43:41.551985+0000","last_undegraded":"2024-02-09T17:43:41.551985+0000","last_fullsized":"2024-02-09T17:43:41.551985+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:36.150728+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:11.970338+0000","last_clean_scrub_stamp":"2024-02-09T17:43:36.150728+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:51.342235+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"63'139","reported_seq":753,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-02-09T17:43:51.567157+0000","last_change":"2024-02-09T17:43:35.861836+0000","last_active":"2024-02-09T17:43:51.567157+0000","last_peered":"2024-02-09T17:43:51.567157+0000","last_clean":"2024-02-09T17:43:51.567157+0000","last_became_active":"2024-02-09T17:43:19.138398+0000","last_became_peered":"2024-02-09T17:43:19.138398+0000","last_unstale":"2024-02-09T17:43:51.567157+0000","last_undegraded":"2024-02-09T17:43:51.567157+0000","last_fullsized":"2024-02-09T17:43:51.567157+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:35.861750+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:43:35.861750+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:05.539889+0000","scrub_duration":154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"63'139","reported_seq":789,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-09T17:43:41.557213+0000","last_change":"2024-02-09T17:43:28.589702+0000","last_active":"2024-02-09T17:43:41.557213+0000","last_peered":"2024-02-09T17:43:41.557213+0000","last_clean":"2024-02-09T17:43:41.557213+0000","last_became_active":"2024-02-09T17:43:28.589307+0000","last_became_peered":"2024-02-09T17:43:28.589307+0000","last_unstale":"2024-02-09T17:43:41.557213+0000","last_undegraded":"2024-02-09T17:43:41.557213+0000","last_fullsized":"2024-02-09T17:43:41.557213+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:04.605918+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:38:26.968892+0000","last_clean_scrub_stamp":"2024-02-09T17:43:04.605918+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:45.740101+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17890,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":6,"num_write_kb":89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80141,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.4","version":"533'11451","reported_seq":19845,"reported_epoch":533,"state":"active+remapped+backfilling","last_fresh":"2024-02-09T17:43:47.967918+0000","last_change":"2024-02-09T17:43:26.925690+0000","last_active":"2024-02-09T17:43:47.967918+0000","last_peered":"2024-02-09T17:43:47.967918+0000","last_clean":"2024-02-09T17:43:24.144694+0000","last_became_active":"2024-02-09T17:43:26.883166+0000","last_became_peered":"2024-02-09T17:43:26.883166+0000","last_unstale":"2024-02-09T17:43:47.967918+0000","last_undegraded":"2024-02-09T17:43:47.967918+0000","last_fullsized":"2024-02-09T17:43:47.967918+0000","mapping_epoch":522,"log_start":"478'1400","ondisk_log_start":"478'1400","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'7928","last_scrub_stamp":"2024-02-09T17:42:35.239746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:42:35.239746+0000","objects_scrubbed":7892,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1665,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":286982144,"num_objects":4379,"num_object_clones":0,"num_object_copies":13137,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":700,"num_objects_unfound":0,"num_objects_dirty":4379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11451,"num_write_kb":506560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4077,"num_bytes_recovered":267190272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,5],"acting":[10,5,8],"avail_no_missing":["10","5","8"],"object_location_counts":[{"shards":"5,8,10","objects":4379}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"63'139","reported_seq":1393,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-02-09T17:43:51.567040+0000","last_change":"2024-02-09T17:43:05.628112+0000","last_active":"2024-02-09T17:43:51.567040+0000","last_peered":"2024-02-09T17:43:51.567040+0000","last_clean":"2024-02-09T17:43:51.567040+0000","last_became_active":"2024-02-09T17:41:14.521079+0000","last_became_peered":"2024-02-09T17:41:14.521079+0000","last_unstale":"2024-02-09T17:43:51.567040+0000","last_undegraded":"2024-02-09T17:43:51.567040+0000","last_fullsized":"2024-02-09T17:43:51.567040+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:05.627998+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:43:05.627998+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:29.887705+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"533'11652","reported_seq":18657,"reported_epoch":533,"state":"active+remapped+backfilling","last_fresh":"2024-02-09T17:43:49.284454+0000","last_change":"2024-02-09T17:43:26.921171+0000","last_active":"2024-02-09T17:43:49.284454+0000","last_peered":"2024-02-09T17:43:49.284454+0000","last_clean":"2024-02-09T17:43:24.158353+0000","last_became_active":"2024-02-09T17:43:26.894525+0000","last_became_peered":"2024-02-09T17:43:26.894525+0000","last_unstale":"2024-02-09T17:43:49.284454+0000","last_undegraded":"2024-02-09T17:43:49.284454+0000","last_fullsized":"2024-02-09T17:43:49.284454+0000","mapping_epoch":522,"log_start":"479'1600","ondisk_log_start":"479'1600","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'7820","last_scrub_stamp":"2024-02-09T17:42:31.833770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:42:31.833770+0000","objects_scrubbed":7802,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1362,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279707648,"num_objects":4268,"num_object_clones":0,"num_object_copies":12804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1237,"num_objects_unfound":0,"num_objects_dirty":4268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11652,"num_write_kb":509440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3383,"num_bytes_recovered":221708288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[5,7,10],"avail_no_missing":["5","7","10"],"object_location_counts":[{"shards":"5,7,10","objects":4268}],"blocked_by":[],"up_primary":2,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"63'139","reported_seq":956,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-09T17:43:41.565928+0000","last_change":"2024-02-09T17:43:39.981380+0000","last_active":"2024-02-09T17:43:41.565928+0000","last_peered":"2024-02-09T17:43:41.565928+0000","last_clean":"2024-02-09T17:43:41.565928+0000","last_became_active":"2024-02-09T17:43:39.981054+0000","last_became_peered":"2024-02-09T17:43:39.981054+0000","last_unstale":"2024-02-09T17:43:41.565928+0000","last_undegraded":"2024-02-09T17:43:41.565928+0000","last_fullsized":"2024-02-09T17:43:41.565928+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:06.666752+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:43:06.666752+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:35.304527+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"533'11580","reported_seq":11815,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-09T17:43:49.219413+0000","last_change":"2024-02-09T17:42:39.441680+0000","last_active":"2024-02-09T17:43:49.219413+0000","last_peered":"2024-02-09T17:43:49.219413+0000","last_clean":"2024-02-09T17:43:49.219413+0000","last_became_active":"2024-02-09T17:40:48.781472+0000","last_became_peered":"2024-02-09T17:40:48.781472+0000","last_unstale":"2024-02-09T17:43:49.219413+0000","last_undegraded":"2024-02-09T17:43:49.219413+0000","last_fullsized":"2024-02-09T17:43:49.219413+0000","mapping_epoch":465,"log_start":"478'1500","ondisk_log_start":"478'1500","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"503'8191","last_scrub_stamp":"2024-02-09T17:42:39.441525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:42:39.441525+0000","objects_scrubbed":7816,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:56.423990+0000","scrub_duration":2957,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":286261248,"num_objects":4368,"num_object_clones":0,"num_object_copies":13104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11580,"num_write_kb":510336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"63'139","reported_seq":571,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-09T17:43:41.551794+0000","last_change":"2024-02-09T17:42:38.047873+0000","last_active":"2024-02-09T17:43:41.551794+0000","last_peered":"2024-02-09T17:43:41.551794+0000","last_clean":"2024-02-09T17:43:41.551794+0000","last_became_active":"2024-02-09T17:40:13.501711+0000","last_became_peered":"2024-02-09T17:40:13.501711+0000","last_unstale":"2024-02-09T17:43:41.551794+0000","last_undegraded":"2024-02-09T17:43:41.551794+0000","last_fullsized":"2024-02-09T17:43:41.551794+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:42:38.047749+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:38:27.045816+0000","last_clean_scrub_stamp":"2024-02-09T17:42:38.047749+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:51.361238+0000","scrub_duration":149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"533'11677","reported_seq":11966,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-09T17:43:48.885135+0000","last_change":"2024-02-09T17:43:37.166310+0000","last_active":"2024-02-09T17:43:48.885135+0000","last_peered":"2024-02-09T17:43:48.885135+0000","last_clean":"2024-02-09T17:43:48.885135+0000","last_became_active":"2024-02-09T17:40:48.780579+0000","last_became_peered":"2024-02-09T17:40:48.780579+0000","last_unstale":"2024-02-09T17:43:48.885135+0000","last_undegraded":"2024-02-09T17:43:48.885135+0000","last_fullsized":"2024-02-09T17:43:48.885135+0000","mapping_epoch":465,"log_start":"478'1600","ondisk_log_start":"478'1600","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"529'11051","last_scrub_stamp":"2024-02-09T17:43:37.166082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:43:37.166082+0000","objects_scrubbed":4984,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:53.401985+0000","scrub_duration":1034,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284229632,"num_objects":4337,"num_object_clones":0,"num_object_copies":13011,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11677,"num_write_kb":512448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"63'139","reported_seq":887,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-09T17:43:41.549006+0000","last_change":"2024-02-09T17:42:30.576428+0000","last_active":"2024-02-09T17:43:41.549006+0000","last_peered":"2024-02-09T17:43:41.549006+0000","last_clean":"2024-02-09T17:43:41.549006+0000","last_became_active":"2024-02-09T17:34:01.342607+0000","last_became_peered":"2024-02-09T17:34:01.342607+0000","last_unstale":"2024-02-09T17:43:41.549006+0000","last_undegraded":"2024-02-09T17:43:41.549006+0000","last_fullsized":"2024-02-09T17:43:41.549006+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:42:30.576340+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:42:30.576340+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:42.629091+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.2","version":"533'12092","reported_seq":12385,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-09T17:43:50.999963+0000","last_change":"2024-02-09T17:43:46.878627+0000","last_active":"2024-02-09T17:43:50.999963+0000","last_peered":"2024-02-09T17:43:50.999963+0000","last_clean":"2024-02-09T17:43:50.999963+0000","last_became_active":"2024-02-09T17:40:48.781532+0000","last_became_peered":"2024-02-09T17:40:48.781532+0000","last_unstale":"2024-02-09T17:43:50.999963+0000","last_undegraded":"2024-02-09T17:43:50.999963+0000","last_fullsized":"2024-02-09T17:43:50.999963+0000","mapping_epoch":465,"log_start":"479'2000","ondisk_log_start":"479'2000","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"533'11868","last_scrub_stamp":"2024-02-09T17:43:46.878486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:43:46.878486+0000","objects_scrubbed":4573,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:59.991875+0000","scrub_duration":1380,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282329088,"num_objects":4308,"num_object_clones":0,"num_object_copies":12924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12092,"num_write_kb":524800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"63'139","reported_seq":899,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-09T17:43:41.548875+0000","last_change":"2024-02-09T17:43:14.923680+0000","last_active":"2024-02-09T17:43:41.548875+0000","last_peered":"2024-02-09T17:43:41.548875+0000","last_clean":"2024-02-09T17:43:41.548875+0000","last_became_active":"2024-02-09T17:43:14.024894+0000","last_became_peered":"2024-02-09T17:43:14.024894+0000","last_unstale":"2024-02-09T17:43:41.548875+0000","last_undegraded":"2024-02-09T17:43:41.548875+0000","last_fullsized":"2024-02-09T17:43:41.548875+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:14.923533+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:11.970338+0000","last_clean_scrub_stamp":"2024-02-09T17:43:14.923533+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:26.498125+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.3","version":"533'13227","reported_seq":13563,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-09T17:43:50.950193+0000","last_change":"2024-02-09T17:43:36.701958+0000","last_active":"2024-02-09T17:43:50.950193+0000","last_peered":"2024-02-09T17:43:50.950193+0000","last_clean":"2024-02-09T17:43:50.950193+0000","last_became_active":"2024-02-09T17:40:48.783369+0000","last_became_peered":"2024-02-09T17:40:48.783369+0000","last_unstale":"2024-02-09T17:43:50.950193+0000","last_undegraded":"2024-02-09T17:43:50.950193+0000","last_fullsized":"2024-02-09T17:43:50.950193+0000","mapping_epoch":465,"log_start":"485'3200","ondisk_log_start":"485'3200","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"529'12566","last_scrub_stamp":"2024-02-09T17:43:36.701823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:43:36.701823+0000","objects_scrubbed":4960,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:00.781538+0000","scrub_duration":1352,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279969792,"num_objects":4272,"num_object_clones":0,"num_object_copies":12816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11666,"num_write_kb":510016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"63'139","reported_seq":562,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-09T17:43:41.555424+0000","last_change":"2024-02-09T17:43:15.279635+0000","last_active":"2024-02-09T17:43:41.555424+0000","last_peered":"2024-02-09T17:43:41.555424+0000","last_clean":"2024-02-09T17:43:41.555424+0000","last_became_active":"2024-02-09T17:43:14.038885+0000","last_became_peered":"2024-02-09T17:43:14.038885+0000","last_unstale":"2024-02-09T17:43:41.555424+0000","last_undegraded":"2024-02-09T17:43:41.555424+0000","last_fullsized":"2024-02-09T17:43:41.555424+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:15.279496+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:43:15.279496+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:38.434647+0000","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"533'11957","reported_seq":12193,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-09T17:43:47.934461+0000","last_change":"2024-02-09T17:42:30.704487+0000","last_active":"2024-02-09T17:43:47.934461+0000","last_peered":"2024-02-09T17:43:47.934461+0000","last_clean":"2024-02-09T17:43:47.934461+0000","last_became_active":"2024-02-09T17:40:48.780290+0000","last_became_peered":"2024-02-09T17:40:48.780290+0000","last_unstale":"2024-02-09T17:43:47.934461+0000","last_undegraded":"2024-02-09T17:43:47.934461+0000","last_fullsized":"2024-02-09T17:43:47.934461+0000","mapping_epoch":465,"log_start":"479'1900","ondisk_log_start":"479'1900","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'8033","last_scrub_stamp":"2024-02-09T17:42:30.704387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:42:30.704387+0000","objects_scrubbed":7963,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292225024,"num_objects":4459,"num_object_clones":0,"num_object_copies":13377,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11957,"num_write_kb":525312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"63'139","reported_seq":969,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-09T17:43:41.555791+0000","last_change":"2024-02-09T17:42:31.121705+0000","last_active":"2024-02-09T17:43:41.555791+0000","last_peered":"2024-02-09T17:43:41.555791+0000","last_clean":"2024-02-09T17:43:41.555791+0000","last_became_active":"2024-02-09T17:33:00.284924+0000","last_became_peered":"2024-02-09T17:33:00.284924+0000","last_unstale":"2024-02-09T17:43:41.555791+0000","last_undegraded":"2024-02-09T17:43:41.555791+0000","last_fullsized":"2024-02-09T17:43:41.555791+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:42:31.121560+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:42:31.121560+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:53.668566+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.1","version":"533'11731","reported_seq":12018,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-09T17:43:50.796171+0000","last_change":"2024-02-09T17:43:44.138178+0000","last_active":"2024-02-09T17:43:50.796171+0000","last_peered":"2024-02-09T17:43:50.796171+0000","last_clean":"2024-02-09T17:43:50.796171+0000","last_became_active":"2024-02-09T17:40:49.660617+0000","last_became_peered":"2024-02-09T17:40:49.660617+0000","last_unstale":"2024-02-09T17:43:50.796171+0000","last_undegraded":"2024-02-09T17:43:50.796171+0000","last_fullsized":"2024-02-09T17:43:50.796171+0000","mapping_epoch":465,"log_start":"479'1700","ondisk_log_start":"479'1700","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"533'11388","last_scrub_stamp":"2024-02-09T17:43:44.138002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:43:44.138002+0000","objects_scrubbed":4543,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:01.112942+0000","scrub_duration":1020,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272957440,"num_objects":4165,"num_object_clones":0,"num_object_copies":12495,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11731,"num_write_kb":508672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"63'139","reported_seq":944,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-09T17:43:41.549188+0000","last_change":"2024-02-09T17:43:28.597827+0000","last_active":"2024-02-09T17:43:41.549188+0000","last_peered":"2024-02-09T17:43:41.549188+0000","last_clean":"2024-02-09T17:43:41.549188+0000","last_became_active":"2024-02-09T17:43:28.597612+0000","last_became_peered":"2024-02-09T17:43:28.597612+0000","last_unstale":"2024-02-09T17:43:41.549188+0000","last_undegraded":"2024-02-09T17:43:41.549188+0000","last_fullsized":"2024-02-09T17:43:41.549188+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:15.934522+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:43:15.934522+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:22.255009+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.e","version":"535'11899","reported_seq":12170,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-02-09T17:43:51.595612+0000","last_change":"2024-02-09T17:43:38.843389+0000","last_active":"2024-02-09T17:43:51.595612+0000","last_peered":"2024-02-09T17:43:51.595612+0000","last_clean":"2024-02-09T17:43:51.595612+0000","last_became_active":"2024-02-09T17:40:48.779366+0000","last_became_peered":"2024-02-09T17:40:48.779366+0000","last_unstale":"2024-02-09T17:43:51.595612+0000","last_undegraded":"2024-02-09T17:43:51.595612+0000","last_fullsized":"2024-02-09T17:43:51.595612+0000","mapping_epoch":465,"log_start":"479'1800","ondisk_log_start":"479'1800","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"530'11226","last_scrub_stamp":"2024-02-09T17:43:38.843170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:43:38.843170+0000","objects_scrubbed":4976,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:47.276017+0000","scrub_duration":1147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280166400,"num_objects":4275,"num_object_clones":0,"num_object_copies":12825,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11899,"num_write_kb":517568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"63'139","reported_seq":896,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-02-09T17:43:51.562506+0000","last_change":"2024-02-09T17:43:14.897996+0000","last_active":"2024-02-09T17:43:51.562506+0000","last_peered":"2024-02-09T17:43:51.562506+0000","last_clean":"2024-02-09T17:43:51.562506+0000","last_became_active":"2024-02-09T17:43:14.041291+0000","last_became_peered":"2024-02-09T17:43:14.041291+0000","last_unstale":"2024-02-09T17:43:51.562506+0000","last_undegraded":"2024-02-09T17:43:51.562506+0000","last_fullsized":"2024-02-09T17:43:51.562506+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:14.897897+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:43:14.897897+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:39.424108+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.f","version":"533'11741","reported_seq":14350,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-09T17:43:49.270920+0000","last_change":"2024-02-09T17:42:38.866203+0000","last_active":"2024-02-09T17:43:49.270920+0000","last_peered":"2024-02-09T17:43:49.270920+0000","last_clean":"2024-02-09T17:43:49.270920+0000","last_became_active":"2024-02-09T17:41:20.876180+0000","last_became_peered":"2024-02-09T17:41:20.876180+0000","last_unstale":"2024-02-09T17:43:49.270920+0000","last_undegraded":"2024-02-09T17:43:49.270920+0000","last_fullsized":"2024-02-09T17:43:49.270920+0000","mapping_epoch":480,"log_start":"479'1700","ondisk_log_start":"479'1700","created":465,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"503'8226","last_scrub_stamp":"2024-02-09T17:42:38.866022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:42:38.866022+0000","objects_scrubbed":7951,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:43:54.450060+0000","scrub_duration":3408,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":285147136,"num_objects":4351,"num_object_clones":0,"num_object_copies":13053,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11741,"num_write_kb":514944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1182,"num_bytes_recovered":77463552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,11],"acting":[5,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"63'139","reported_seq":734,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-09T17:43:41.555780+0000","last_change":"2024-02-09T17:43:14.308625+0000","last_active":"2024-02-09T17:43:41.555780+0000","last_peered":"2024-02-09T17:43:41.555780+0000","last_clean":"2024-02-09T17:43:41.555780+0000","last_became_active":"2024-02-09T17:43:14.054943+0000","last_became_peered":"2024-02-09T17:43:14.054943+0000","last_unstale":"2024-02-09T17:43:41.555780+0000","last_undegraded":"2024-02-09T17:43:41.555780+0000","last_fullsized":"2024-02-09T17:43:41.555780+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:14.308481+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:43:14.308481+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:25.700400+0000","scrub_duration":160,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"533'11893","reported_seq":12164,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-09T17:43:49.734118+0000","last_change":"2024-02-09T17:43:39.114851+0000","last_active":"2024-02-09T17:43:49.734118+0000","last_peered":"2024-02-09T17:43:49.734118+0000","last_clean":"2024-02-09T17:43:49.734118+0000","last_became_active":"2024-02-09T17:40:48.781384+0000","last_became_peered":"2024-02-09T17:40:48.781384+0000","last_unstale":"2024-02-09T17:43:49.734118+0000","last_undegraded":"2024-02-09T17:43:49.734118+0000","last_fullsized":"2024-02-09T17:43:49.734118+0000","mapping_epoch":465,"log_start":"479'1800","ondisk_log_start":"479'1800","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"531'11356","last_scrub_stamp":"2024-02-09T17:43:39.114709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:43:39.114709+0000","objects_scrubbed":4812,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:06.435989+0000","scrub_duration":1039,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278986752,"num_objects":4257,"num_object_clones":0,"num_object_copies":12771,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11893,"num_write_kb":516800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"63'139","reported_seq":950,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-09T17:43:41.550260+0000","last_change":"2024-02-09T17:43:28.586677+0000","last_active":"2024-02-09T17:43:41.550260+0000","last_peered":"2024-02-09T17:43:41.550260+0000","last_clean":"2024-02-09T17:43:41.550260+0000","last_became_active":"2024-02-09T17:43:28.586349+0000","last_became_peered":"2024-02-09T17:43:28.586349+0000","last_unstale":"2024-02-09T17:43:41.550260+0000","last_undegraded":"2024-02-09T17:43:41.550260+0000","last_fullsized":"2024-02-09T17:43:41.550260+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:14.332107+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:43:14.332107+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:38.477870+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"533'12065","reported_seq":14851,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-09T17:43:50.403291+0000","last_change":"2024-02-09T17:43:05.986778+0000","last_active":"2024-02-09T17:43:50.403291+0000","last_peered":"2024-02-09T17:43:50.403291+0000","last_clean":"2024-02-09T17:43:50.403291+0000","last_became_active":"2024-02-09T17:41:21.499187+0000","last_became_peered":"2024-02-09T17:41:21.499187+0000","last_unstale":"2024-02-09T17:43:50.403291+0000","last_undegraded":"2024-02-09T17:43:50.403291+0000","last_fullsized":"2024-02-09T17:43:50.403291+0000","mapping_epoch":481,"log_start":"479'2000","ondisk_log_start":"479'2000","created":465,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"510'10057","last_scrub_stamp":"2024-02-09T17:43:05.986640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:43:05.986640+0000","objects_scrubbed":6440,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:28.384349+0000","scrub_duration":1712,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":286916632,"num_objects":4379,"num_object_clones":0,"num_object_copies":13137,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4379,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12065,"num_write_kb":526145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1248,"num_bytes_recovered":81788928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"63'139","reported_seq":1400,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-02-09T17:43:51.567243+0000","last_change":"2024-02-09T17:43:28.591362+0000","last_active":"2024-02-09T17:43:51.567243+0000","last_peered":"2024-02-09T17:43:51.567243+0000","last_clean":"2024-02-09T17:43:51.567243+0000","last_became_active":"2024-02-09T17:43:28.590237+0000","last_became_peered":"2024-02-09T17:43:28.590237+0000","last_unstale":"2024-02-09T17:43:51.567243+0000","last_undegraded":"2024-02-09T17:43:51.567243+0000","last_fullsized":"2024-02-09T17:43:51.567243+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:15.675690+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:15.675690+0000","last_clean_scrub_stamp":"2024-02-09T17:43:15.675690+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:36.929532+0000","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"535'11781","reported_seq":12027,"reported_epoch":535,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-09T17:43:51.795604+0000","last_change":"2024-02-09T17:43:26.917886+0000","last_active":"2024-02-09T17:43:51.795604+0000","last_peered":"2024-02-09T17:43:51.795604+0000","last_clean":"2024-02-09T17:43:24.158212+0000","last_became_active":"2024-02-09T17:43:26.887932+0000","last_became_peered":"2024-02-09T17:43:26.887932+0000","last_unstale":"2024-02-09T17:43:51.795604+0000","last_undegraded":"2024-02-09T17:43:51.795604+0000","last_fullsized":"2024-02-09T17:43:51.795604+0000","mapping_epoch":522,"log_start":"479'1700","ondisk_log_start":"479'1700","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'7912","last_scrub_stamp":"2024-02-09T17:42:33.501032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:42:33.501032+0000","objects_scrubbed":7852,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1903,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":273874944,"num_objects":4179,"num_object_clones":0,"num_object_copies":12537,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4179,"num_objects_unfound":0,"num_objects_dirty":4179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11781,"num_write_kb":510720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,10],"acting":[8,10,4],"avail_no_missing":["8","4","10"],"object_location_counts":[{"shards":"4,8,10","objects":4179}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.e","version":"63'139","reported_seq":860,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-09T17:43:41.554275+0000","last_change":"2024-02-09T17:43:14.981580+0000","last_active":"2024-02-09T17:43:41.554275+0000","last_peered":"2024-02-09T17:43:41.554275+0000","last_clean":"2024-02-09T17:43:41.554275+0000","last_became_active":"2024-02-09T17:43:14.036583+0000","last_became_peered":"2024-02-09T17:43:14.036583+0000","last_unstale":"2024-02-09T17:43:41.554275+0000","last_undegraded":"2024-02-09T17:43:41.554275+0000","last_fullsized":"2024-02-09T17:43:41.554275+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:14.981453+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:14.981453+0000","last_clean_scrub_stamp":"2024-02-09T17:43:14.981453+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:16.190361+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"535'11878","reported_seq":12173,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-02-09T17:43:51.595985+0000","last_change":"2024-02-09T17:43:42.009150+0000","last_active":"2024-02-09T17:43:51.595985+0000","last_peered":"2024-02-09T17:43:51.595985+0000","last_clean":"2024-02-09T17:43:51.595985+0000","last_became_active":"2024-02-09T17:40:48.783235+0000","last_became_peered":"2024-02-09T17:40:48.783235+0000","last_unstale":"2024-02-09T17:43:51.595985+0000","last_undegraded":"2024-02-09T17:43:51.595985+0000","last_fullsized":"2024-02-09T17:43:51.595985+0000","mapping_epoch":465,"log_start":"479'1800","ondisk_log_start":"479'1800","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"533'11389","last_scrub_stamp":"2024-02-09T17:43:42.009039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:43:42.009039+0000","objects_scrubbed":4727,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:05.982895+0000","scrub_duration":1348,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":275775488,"num_objects":4208,"num_object_clones":0,"num_object_copies":12624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11878,"num_write_kb":514752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.f","version":"63'139","reported_seq":914,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-09T17:43:41.555322+0000","last_change":"2024-02-09T17:43:38.588100+0000","last_active":"2024-02-09T17:43:41.555322+0000","last_peered":"2024-02-09T17:43:41.555322+0000","last_clean":"2024-02-09T17:43:41.555322+0000","last_became_active":"2024-02-09T17:33:05.316100+0000","last_became_peered":"2024-02-09T17:33:05.316100+0000","last_unstale":"2024-02-09T17:43:41.555322+0000","last_undegraded":"2024-02-09T17:43:41.555322+0000","last_fullsized":"2024-02-09T17:43:41.555322+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:38.588006+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:36:22.364130+0000","last_clean_scrub_stamp":"2024-02-09T17:43:38.588006+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:55.592335+0000","scrub_duration":165,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"533'11962","reported_seq":14600,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-09T17:43:50.367202+0000","last_change":"2024-02-09T17:42:33.997317+0000","last_active":"2024-02-09T17:43:50.367202+0000","last_peered":"2024-02-09T17:43:50.367202+0000","last_clean":"2024-02-09T17:43:50.367202+0000","last_became_active":"2024-02-09T17:41:20.876434+0000","last_became_peered":"2024-02-09T17:41:20.876434+0000","last_unstale":"2024-02-09T17:43:50.367202+0000","last_undegraded":"2024-02-09T17:43:50.367202+0000","last_fullsized":"2024-02-09T17:43:50.367202+0000","mapping_epoch":480,"log_start":"479'1900","ondisk_log_start":"479'1900","created":465,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'8076","last_scrub_stamp":"2024-02-09T17:42:33.997206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:42:33.997206+0000","objects_scrubbed":8030,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1898,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281149440,"num_objects":4290,"num_object_clones":0,"num_object_copies":12870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11962,"num_write_kb":520064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1195,"num_bytes_recovered":78315520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"63'139","reported_seq":1044,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-02-09T17:43:51.588410+0000","last_change":"2024-02-09T17:43:14.135427+0000","last_active":"2024-02-09T17:43:51.588410+0000","last_peered":"2024-02-09T17:43:51.588410+0000","last_clean":"2024-02-09T17:43:51.588410+0000","last_became_active":"2024-02-09T17:43:14.027715+0000","last_became_peered":"2024-02-09T17:43:14.027715+0000","last_unstale":"2024-02-09T17:43:51.588410+0000","last_undegraded":"2024-02-09T17:43:51.588410+0000","last_fullsized":"2024-02-09T17:43:51.588410+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:14.135321+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:14.135321+0000","last_clean_scrub_stamp":"2024-02-09T17:43:14.135321+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:25.094058+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"535'12010","reported_seq":12293,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-02-09T17:43:51.672901+0000","last_change":"2024-02-09T17:43:10.463364+0000","last_active":"2024-02-09T17:43:51.672901+0000","last_peered":"2024-02-09T17:43:51.672901+0000","last_clean":"2024-02-09T17:43:51.672901+0000","last_became_active":"2024-02-09T17:40:49.436285+0000","last_became_peered":"2024-02-09T17:40:49.436285+0000","last_unstale":"2024-02-09T17:43:51.672901+0000","last_undegraded":"2024-02-09T17:43:51.672901+0000","last_fullsized":"2024-02-09T17:43:51.672901+0000","mapping_epoch":465,"log_start":"479'2000","ondisk_log_start":"479'2000","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"512'10151","last_scrub_stamp":"2024-02-09T17:43:10.463109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:43:10.463109+0000","objects_scrubbed":6190,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:23.731855+0000","scrub_duration":1913,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280363008,"num_objects":4278,"num_object_clones":0,"num_object_copies":12834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12010,"num_write_kb":521216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"63'139","reported_seq":456,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-02-09T17:43:51.567358+0000","last_change":"2024-02-09T17:43:14.652654+0000","last_active":"2024-02-09T17:43:51.567358+0000","last_peered":"2024-02-09T17:43:51.567358+0000","last_clean":"2024-02-09T17:43:51.567358+0000","last_became_active":"2024-02-09T17:43:14.036759+0000","last_became_peered":"2024-02-09T17:43:14.036759+0000","last_unstale":"2024-02-09T17:43:51.567358+0000","last_undegraded":"2024-02-09T17:43:51.567358+0000","last_fullsized":"2024-02-09T17:43:51.567358+0000","mapping_epoch":454,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:14.652552+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:43:14.652552+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:23.595517+0000","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"63'139","reported_seq":855,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-09T17:43:41.550352+0000","last_change":"2024-02-09T17:43:04.345907+0000","last_active":"2024-02-09T17:43:41.550352+0000","last_peered":"2024-02-09T17:43:41.550352+0000","last_clean":"2024-02-09T17:43:41.550352+0000","last_became_active":"2024-02-09T17:33:05.310104+0000","last_became_peered":"2024-02-09T17:33:05.310104+0000","last_unstale":"2024-02-09T17:43:41.550352+0000","last_undegraded":"2024-02-09T17:43:41.550352+0000","last_fullsized":"2024-02-09T17:43:41.550352+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:04.345783+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:43:04.345783+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:06.315071+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"63'139","reported_seq":874,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-02-09T17:43:51.567370+0000","last_change":"2024-02-09T17:43:46.691783+0000","last_active":"2024-02-09T17:43:51.567370+0000","last_peered":"2024-02-09T17:43:51.567370+0000","last_clean":"2024-02-09T17:43:51.567370+0000","last_became_active":"2024-02-09T17:34:01.353171+0000","last_became_peered":"2024-02-09T17:34:01.353171+0000","last_unstale":"2024-02-09T17:43:51.567370+0000","last_undegraded":"2024-02-09T17:43:51.567370+0000","last_fullsized":"2024-02-09T17:43:51.567370+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:46.691649+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:43:46.691649+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:13.524713+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.12","version":"63'139","reported_seq":1374,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-09T17:43:41.550459+0000","last_change":"2024-02-09T17:42:31.533609+0000","last_active":"2024-02-09T17:43:41.550459+0000","last_peered":"2024-02-09T17:43:41.550459+0000","last_clean":"2024-02-09T17:43:41.550459+0000","last_became_active":"2024-02-09T17:40:13.499071+0000","last_became_peered":"2024-02-09T17:40:13.499071+0000","last_unstale":"2024-02-09T17:43:41.550459+0000","last_undegraded":"2024-02-09T17:43:41.550459+0000","last_fullsized":"2024-02-09T17:43:41.550459+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:42:31.533484+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:42:31.533484+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"63'139","reported_seq":573,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-02-09T17:43:41.551913+0000","last_change":"2024-02-09T17:43:04.717874+0000","last_active":"2024-02-09T17:43:41.551913+0000","last_peered":"2024-02-09T17:43:41.551913+0000","last_clean":"2024-02-09T17:43:41.551913+0000","last_became_active":"2024-02-09T17:41:14.521672+0000","last_became_peered":"2024-02-09T17:41:14.521672+0000","last_unstale":"2024-02-09T17:43:41.551913+0000","last_undegraded":"2024-02-09T17:43:41.551913+0000","last_fullsized":"2024-02-09T17:43:41.551913+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:04.717759+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:43:04.717759+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:24.538421+0000","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":4507041816,"num_objects":68773,"num_object_clones":0,"num_object_copies":206319,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6116,"num_objects_unfound":0,"num_objects_dirty":68773,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":189035,"num_write_kb":8249793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11085,"num_bytes_recovered":726466560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14007152640,"data_stored":14007140424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160996,"ondisk_log_size":160996,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":476914,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":6,"num_write_kb":89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":784509,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056898,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1670896,"kb_used_data":1112168,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":92074256,"statfs":{"total":95995035648,"available":94284038144,"internally_reserved":0,"allocated":1138860032,"data_stored":1138499211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":572127684},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:43:29 2024","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.2849999999999999,"15min":1.3160000000000001},"min":{"1min":0.23599999999999999,"5min":0.223,"15min":0.216},"max":{"1min":7.4089999999999998,"5min":9.5709999999999997,"15min":14.913},"last":3.1339999999999999},{"interface":"front","average":{"1min":1.165,"5min":1.258,"15min":1.2849999999999999},"min":{"1min":0.25900000000000001,"5min":0.253,"15min":0.23000000000000001},"max":{"1min":5.1340000000000003,"5min":9.8219999999999992,"15min":14.869},"last":0.873}]},{"osd":1,"last update":"Fri Feb 9 17:43:29 2024","interfaces":[{"interface":"back","average":{"1min":1.754,"5min":1.3839999999999999,"15min":1.4830000000000001},"min":{"1min":0.29599999999999999,"5min":0.27600000000000002,"15min":0.22700000000000001},"max":{"1min":9.7669999999999995,"5min":9.7710000000000008,"15min":17.832999999999998},"last":0.33500000000000002},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.355,"15min":1.583},"min":{"1min":0.29399999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":7.4619999999999997,"5min":9.718,"15min":17.789000000000001},"last":3.165}]},{"osd":2,"last update":"Fri Feb 9 17:31:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.415,"15min":1.6859999999999999},"min":{"1min":0.378,"5min":0.247,"15min":0.247},"max":{"1min":4.4189999999999996,"5min":9.4719999999999995,"15min":9.4719999999999995},"last":0.35399999999999998},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.5149999999999999,"15min":1.4399999999999999},"min":{"1min":0.378,"5min":0.252,"15min":0.252},"max":{"1min":4.8579999999999997,"5min":7.96,"15min":7.96},"last":0.60399999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:43:29 2024","interfaces":[{"interface":"back","average":{"1min":1.958,"5min":1.1719999999999999,"15min":1.484},"min":{"1min":0.27200000000000002,"5min":0.27100000000000002,"15min":0.247},"max":{"1min":9.7289999999999992,"5min":9.7289999999999992,"15min":21.315999999999999},"last":3.8490000000000002},{"interface":"front","average":{"1min":2.0070000000000001,"5min":1.5980000000000001,"15min":1.6819999999999999},"min":{"1min":0.33600000000000002,"5min":0.33000000000000002,"15min":0.26400000000000001},"max":{"1min":9.7479999999999993,"5min":9.7479999999999993,"15min":18.087},"last":3.8809999999999998}]},{"osd":4,"last update":"Fri Feb 9 17:43:29 2024","interfaces":[{"interface":"back","average":{"1min":2.1579999999999999,"5min":1.587,"15min":1.651},"min":{"1min":0.26900000000000002,"5min":0.246,"15min":0.246},"max":{"1min":6.3959999999999999,"5min":21.030999999999999,"15min":21.030999999999999},"last":0.90300000000000002},{"interface":"front","average":{"1min":2.1680000000000001,"5min":1.548,"15min":1.4299999999999999},"min":{"1min":0.318,"5min":0.317,"15min":0.28799999999999998},"max":{"1min":7.5490000000000004,"5min":20.876999999999999,"15min":20.876999999999999},"last":3.2360000000000002}]},{"osd":5,"last update":"Fri Feb 9 17:43:29 2024","interfaces":[{"interface":"back","average":{"1min":2.4380000000000002,"5min":1.3819999999999999,"15min":1.4970000000000001},"min":{"1min":0.47199999999999998,"5min":0.32900000000000001,"15min":0.28899999999999998},"max":{"1min":9.5960000000000001,"5min":9.5960000000000001,"15min":13.946999999999999},"last":9.0690000000000008},{"interface":"front","average":{"1min":2.6360000000000001,"5min":1.486,"15min":1.607},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":7.4580000000000002,"5min":7.4580000000000002,"15min":13.414},"last":3.3889999999999998}]},{"osd":6,"last update":"Fri Feb 9 17:43:29 2024","interfaces":[{"interface":"back","average":{"1min":3.1459999999999999,"5min":1.734,"15min":1.6359999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.29899999999999999},"max":{"1min":12.362,"5min":12.362,"15min":12.798999999999999},"last":9.1679999999999993},{"interface":"front","average":{"1min":2.238,"5min":1.548,"15min":1.631},"min":{"1min":0.34100000000000003,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":7.5800000000000001,"5min":7.5800000000000001,"15min":10.782999999999999},"last":3.2120000000000002}]},{"osd":7,"last update":"Fri Feb 9 17:43:29 2024","interfaces":[{"interface":"back","average":{"1min":2.4849999999999999,"5min":1.6990000000000001,"15min":1.7669999999999999},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.312},"max":{"1min":9.7029999999999994,"5min":20.489999999999998,"15min":20.489999999999998},"last":9.0969999999999995},{"interface":"front","average":{"1min":2.1960000000000002,"5min":1.671,"15min":1.7250000000000001},"min":{"1min":0.376,"5min":0.35299999999999998,"15min":0.34200000000000003},"max":{"1min":7.4210000000000003,"5min":20.712,"15min":20.712},"last":1.0640000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:42:54 2024","interfaces":[{"interface":"back","average":{"1min":1.6339999999999999,"5min":1.0289999999999999,"15min":1.6040000000000001},"min":{"1min":0.38700000000000001,"5min":0.30399999999999999,"15min":0.29499999999999998},"max":{"1min":7.3570000000000002,"5min":7.3570000000000002,"15min":13.199999999999999},"last":7.0049999999999999},{"interface":"front","average":{"1min":1.978,"5min":1.1870000000000001,"15min":1.4510000000000001},"min":{"1min":0.43099999999999999,"5min":0.35399999999999998,"15min":0.318},"max":{"1min":7.6059999999999999,"5min":7.6059999999999999,"15min":9.2940000000000005},"last":3.1899999999999999}]},{"osd":9,"last update":"Fri Feb 9 17:43:22 2024","interfaces":[{"interface":"back","average":{"1min":1.7929999999999999,"5min":2.0070000000000001,"15min":1.5149999999999999},"min":{"1min":0.38200000000000001,"5min":0.34100000000000003,"15min":0.29399999999999998},"max":{"1min":9.8030000000000008,"5min":62.518000000000001,"15min":62.518000000000001},"last":0.48799999999999999},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.988,"15min":1.4219999999999999},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.34100000000000003},"max":{"1min":7.492,"5min":63.241999999999997,"15min":63.241999999999997},"last":1.028}]},{"osd":10,"last update":"Fri Feb 9 17:43:29 2024","interfaces":[{"interface":"back","average":{"1min":2.3900000000000001,"5min":1.3879999999999999,"15min":1.3280000000000001},"min":{"1min":0.38500000000000001,"5min":0.36799999999999999,"15min":0.36199999999999999},"max":{"1min":9.6590000000000007,"5min":9.6590000000000007,"15min":9.6590000000000007},"last":5.3799999999999999},{"interface":"front","average":{"1min":2.5510000000000002,"5min":1.649,"15min":1.405},"min":{"1min":0.47599999999999998,"5min":0.39100000000000001,"15min":0.33900000000000002},"max":{"1min":12.409000000000001,"5min":12.409000000000001,"15min":12.856},"last":0.61399999999999999}]}]},{"osd":10,"up_from":84,"seq":360777253127,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2055468,"kb_used_data":1695268,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":91689684,"statfs":{"total":95995035648,"available":93890236416,"internally_reserved":0,"allocated":1735954432,"data_stored":1735597683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":368835008},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:43:21 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":1.157,"15min":1.5349999999999999},"min":{"1min":0.26400000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":4.5780000000000003,"5min":9.343,"15min":9.343},"last":2.7650000000000001},{"interface":"front","average":{"1min":0.73099999999999998,"5min":1.0229999999999999,"15min":1.1200000000000001},"min":{"1min":0.33100000000000002,"5min":0.253,"15min":0.253},"max":{"1min":4.6980000000000004,"5min":6.3399999999999999,"15min":6.3399999999999999},"last":3.6909999999999998}]},{"osd":1,"last update":"Fri Feb 9 17:42:50 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.66800000000000004,"15min":0.63600000000000001},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":3.726,"5min":3.726,"15min":3.726},"last":2.2909999999999999},{"interface":"front","average":{"1min":1.252,"5min":0.88,"15min":0.81799999999999995},"min":{"1min":0.33200000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":6.2709999999999999,"5min":6.2709999999999999,"15min":6.2709999999999999},"last":0.96399999999999997}]},{"osd":2,"last update":"Fri Feb 9 17:43:02 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":1.4530000000000001,"15min":1.264},"min":{"1min":0.311,"5min":0.27600000000000002,"15min":0.222},"max":{"1min":3.758,"5min":9.0069999999999997,"15min":11.384},"last":5.5279999999999996},{"interface":"front","average":{"1min":1.151,"5min":1.21,"15min":1.2969999999999999},"min":{"1min":0.32500000000000001,"5min":0.28799999999999998,"15min":0.26100000000000001},"max":{"1min":4.6120000000000001,"5min":9.1140000000000008,"15min":11.465},"last":2.7120000000000002}]},{"osd":3,"last update":"Fri Feb 9 17:43:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.3520000000000001,"15min":1.8160000000000001},"min":{"1min":0.36399999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":4.6799999999999997,"5min":9.1649999999999991,"15min":9.1649999999999991},"last":2.6909999999999998},{"interface":"front","average":{"1min":1.635,"5min":1.55,"15min":1.9199999999999999},"min":{"1min":0.38100000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":5.6109999999999998,"5min":8.9670000000000005,"15min":8.9670000000000005},"last":2.3220000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:43:06 2024","interfaces":[{"interface":"back","average":{"1min":1.5700000000000001,"5min":1.097,"15min":1.093},"min":{"1min":0.44500000000000001,"5min":0.29199999999999998,"15min":0.26100000000000001},"max":{"1min":4.5380000000000003,"5min":6.4930000000000003,"15min":7.1360000000000001},"last":4.3860000000000001},{"interface":"front","average":{"1min":1.486,"5min":1.0609999999999999,"15min":1.081},"min":{"1min":0.45400000000000001,"5min":0.32600000000000001,"15min":0.29999999999999999},"max":{"1min":4.6609999999999996,"5min":6.5339999999999998,"15min":6.8369999999999997},"last":2.738}]},{"osd":5,"last update":"Fri Feb 9 17:43:02 2024","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.1259999999999999,"15min":1.248},"min":{"1min":0.29999999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":4.4820000000000002,"5min":7.5179999999999998,"15min":9.2349999999999994},"last":2.262},{"interface":"front","average":{"1min":2.5910000000000002,"5min":1.4319999999999999,"15min":1.3029999999999999},"min":{"1min":0.52000000000000002,"5min":0.38400000000000001,"15min":0.33500000000000002},"max":{"1min":8.7710000000000008,"5min":8.7710000000000008,"15min":8.7710000000000008},"last":2.5760000000000001}]},{"osd":6,"last update":"Fri Feb 9 17:43:02 2024","interfaces":[{"interface":"back","average":{"1min":2.2570000000000001,"5min":1.552,"15min":1.3100000000000001},"min":{"1min":0.40500000000000003,"5min":0.308,"15min":0.29099999999999998},"max":{"1min":10.962,"5min":10.962,"15min":10.962},"last":2.4359999999999999},{"interface":"front","average":{"1min":2.3180000000000001,"5min":1.4470000000000001,"15min":1.4530000000000001},"min":{"1min":0.57999999999999996,"5min":0.38600000000000001,"15min":0.35399999999999998},"max":{"1min":8.8119999999999994,"5min":8.8119999999999994,"15min":9.4350000000000005},"last":2.6640000000000001}]},{"osd":7,"last update":"Fri Feb 9 17:39:52 2024","interfaces":[{"interface":"back","average":{"1min":1.6000000000000001,"5min":1.367,"15min":1.415},"min":{"1min":0.42099999999999999,"5min":0.34200000000000003,"15min":0.312},"max":{"1min":4.0430000000000001,"5min":6.7590000000000003,"15min":17.754000000000001},"last":4.6449999999999996},{"interface":"front","average":{"1min":1.498,"5min":1.4359999999999999,"15min":1.5189999999999999},"min":{"1min":0.47399999999999998,"5min":0.38300000000000001,"15min":0.316},"max":{"1min":5.8170000000000002,"5min":11.308999999999999,"15min":11.308999999999999},"last":0.55000000000000004}]},{"osd":8,"last update":"Fri Feb 9 17:43:45 2024","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.1140000000000001,"15min":1.1519999999999999},"min":{"1min":0.35499999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":6.4269999999999996,"5min":7.4020000000000001,"15min":7.4359999999999999},"last":2.2149999999999999},{"interface":"front","average":{"1min":1.8859999999999999,"5min":1.2669999999999999,"15min":1.04},"min":{"1min":0.40699999999999997,"5min":0.35499999999999998,"15min":0.32900000000000001},"max":{"1min":6.2009999999999996,"5min":7.7060000000000004,"15min":7.7060000000000004},"last":0.995}]},{"osd":9,"last update":"Fri Feb 9 17:43:02 2024","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":1.264,"15min":1.069},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.314},"max":{"1min":3.794,"5min":7.4850000000000003,"15min":8.7240000000000002},"last":2.3490000000000002},{"interface":"front","average":{"1min":0.97399999999999998,"5min":1.228,"15min":1.1339999999999999},"min":{"1min":0.47599999999999998,"5min":0.441,"15min":0.29799999999999999},"max":{"1min":3.819,"5min":7.891,"15min":10.1},"last":2.3900000000000001}]},{"osd":11,"last update":"Fri Feb 9 17:43:23 2024","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.0489999999999999,"15min":1.0840000000000001},"min":{"1min":0.43099999999999999,"5min":0.41699999999999998,"15min":0.379},"max":{"1min":6.226,"5min":6.226,"15min":8.6430000000000007},"last":2.613},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.1100000000000001,"15min":1.1459999999999999},"min":{"1min":0.48999999999999999,"5min":0.441,"15min":0.377},"max":{"1min":6.2590000000000003,"5min":6.2590000000000003,"15min":9.298},"last":2.6389999999999998}]}]},{"osd":9,"up_from":77,"seq":330712482061,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1256588,"kb_used_data":814460,"kb_used_omap":1,"kb_used_meta":442110,"kb_avail":92488564,"statfs":{"total":95995035648,"available":94708289536,"internally_reserved":0,"allocated":834007040,"data_stored":833641489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":452721088},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:42:55 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":0.73599999999999999,"15min":0.63400000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":7.2199999999999998,"5min":7.2199999999999998,"15min":7.2199999999999998},"last":0.30299999999999999},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.0509999999999999,"15min":0.98499999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":4.8689999999999998,"5min":4.8689999999999998,"15min":4.8689999999999998},"last":0.34200000000000003}]},{"osd":1,"last update":"Fri Feb 9 17:43:42 2024","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.78,"15min":1.6699999999999999},"min":{"1min":0.25900000000000001,"5min":0.23599999999999999,"15min":0.23100000000000001},"max":{"1min":5.4080000000000004,"5min":17.024999999999999,"15min":89.515000000000001},"last":0.32200000000000001},{"interface":"front","average":{"1min":1.115,"5min":1.577,"15min":1.631},"min":{"1min":0.29799999999999999,"5min":0.249,"15min":0.222},"max":{"1min":4.5439999999999996,"5min":11.864000000000001,"15min":89.388000000000005},"last":0.375}]},{"osd":2,"last update":"Fri Feb 9 17:40:39 2024","interfaces":[{"interface":"back","average":{"1min":1.742,"5min":1.766,"15min":1.514},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":5.6630000000000003,"5min":14.121,"15min":14.121},"last":0.63400000000000001},{"interface":"front","average":{"1min":1.6910000000000001,"5min":1.52,"15min":1.4870000000000001},"min":{"1min":0.34100000000000003,"5min":0.26100000000000001,"15min":0.25600000000000001},"max":{"1min":5.7030000000000003,"5min":8.8040000000000003,"15min":13.69},"last":0.73899999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:43:42 2024","interfaces":[{"interface":"back","average":{"1min":1.6719999999999999,"5min":1.984,"15min":1.6200000000000001},"min":{"1min":0.35499999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":5.4480000000000004,"5min":15.414999999999999,"15min":40.18},"last":0.38900000000000001},{"interface":"front","average":{"1min":1.131,"5min":1.8080000000000001,"15min":1.6499999999999999},"min":{"1min":0.36699999999999999,"5min":0.32000000000000001,"15min":0.30599999999999999},"max":{"1min":4.6100000000000003,"5min":15.372,"15min":40.158000000000001},"last":0.41699999999999998}]},{"osd":4,"last update":"Fri Feb 9 17:43:42 2024","interfaces":[{"interface":"back","average":{"1min":2.8039999999999998,"5min":1.871,"15min":1.611},"min":{"1min":0.29499999999999998,"5min":0.29299999999999998,"15min":0.28399999999999997},"max":{"1min":8.7189999999999994,"5min":27.381,"15min":27.381},"last":0.85799999999999998},{"interface":"front","average":{"1min":1.7589999999999999,"5min":1.631,"15min":1.462},"min":{"1min":0.34399999999999997,"5min":0.313,"15min":0.26100000000000001},"max":{"1min":4.7859999999999996,"5min":27.355,"15min":27.355},"last":2.5459999999999998}]},{"osd":5,"last update":"Fri Feb 9 17:43:46 2024","interfaces":[{"interface":"back","average":{"1min":2.032,"5min":1.431,"15min":1.4710000000000001},"min":{"1min":0.374,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":6.1849999999999996,"5min":8.6110000000000007,"15min":10.689},"last":1.292},{"interface":"front","average":{"1min":1.5860000000000001,"5min":1.619,"15min":1.3480000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.311},"max":{"1min":6.1559999999999997,"5min":13.978999999999999,"15min":13.978999999999999},"last":1.306}]},{"osd":6,"last update":"Fri Feb 9 17:43:42 2024","interfaces":[{"interface":"back","average":{"1min":2.4569999999999999,"5min":1.734,"15min":1.52},"min":{"1min":0.42699999999999999,"5min":0.34499999999999997,"15min":0.30599999999999999},"max":{"1min":5.7060000000000004,"5min":14.069000000000001,"15min":14.069000000000001},"last":2.4830000000000001},{"interface":"front","average":{"1min":2.98,"5min":1.8879999999999999,"15min":1.613},"min":{"1min":0.38100000000000001,"5min":0.37,"15min":0.37},"max":{"1min":10.417,"5min":14.097,"15min":14.097},"last":1.2090000000000001}]},{"osd":7,"last update":"Fri Feb 9 17:43:42 2024","interfaces":[{"interface":"back","average":{"1min":2.9620000000000002,"5min":1.917,"15min":1.603},"min":{"1min":0.372,"5min":0.372,"15min":0.34300000000000003},"max":{"1min":6.3879999999999999,"5min":9.125,"15min":9.125},"last":0.70499999999999996},{"interface":"front","average":{"1min":2.9980000000000002,"5min":1.958,"15min":1.657},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.32200000000000001},"max":{"1min":6.415,"5min":9.157,"15min":9.157},"last":0.72999999999999998}]},{"osd":8,"last update":"Fri Feb 9 17:42:59 2024","interfaces":[{"interface":"back","average":{"1min":1.478,"5min":1.4199999999999999,"15min":1.284},"min":{"1min":0.37,"5min":0.35399999999999998,"15min":0.34999999999999998},"max":{"1min":7.2759999999999998,"5min":14.045999999999999,"15min":16.209},"last":0.45000000000000001},{"interface":"front","average":{"1min":1.986,"5min":1.5669999999999999,"15min":1.335},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.34799999999999998},"max":{"1min":8.6950000000000003,"5min":8.6950000000000003,"15min":16.263000000000002},"last":0.55800000000000005}]},{"osd":10,"last update":"Fri Feb 9 17:43:06 2024","interfaces":[{"interface":"back","average":{"1min":1.764,"5min":1.4910000000000001,"15min":1.238},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":7.0250000000000004,"5min":9.1869999999999994,"15min":9.5090000000000003},"last":0.56899999999999995},{"interface":"front","average":{"1min":1.7250000000000001,"5min":1.4159999999999999,"15min":1.175},"min":{"1min":0.438,"5min":0.40999999999999998,"15min":0.38200000000000001},"max":{"1min":7.3449999999999998,"5min":9.7569999999999997,"15min":9.7569999999999997},"last":0.54000000000000004}]},{"osd":11,"last update":"Fri Feb 9 17:43:06 2024","interfaces":[{"interface":"back","average":{"1min":2.0699999999999998,"5min":1.712,"15min":1.492},"min":{"1min":0.45700000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":10.464,"5min":14.02,"15min":14.02},"last":0.52100000000000002},{"interface":"front","average":{"1min":2.1520000000000001,"5min":1.4399999999999999,"15min":1.3089999999999999},"min":{"1min":0.44500000000000001,"5min":0.38100000000000001,"15min":0.374},"max":{"1min":10.760999999999999,"5min":10.760999999999999,"15min":10.760999999999999},"last":0.48399999999999999}]}]},{"osd":8,"up_from":171,"seq":734439407804,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1029964,"kb_used_data":808252,"kb_used_omap":212,"kb_used_meta":221483,"kb_avail":92715188,"statfs":{"total":95995035648,"available":94940352512,"internally_reserved":0,"allocated":827650048,"data_stored":827284497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217579,"internal_metadata":226799125},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:43:42 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.94499999999999995,"15min":0.97599999999999998},"min":{"1min":0.224,"5min":0.217,"15min":0.217},"max":{"1min":2.1269999999999998,"5min":8.0920000000000005,"15min":8.0920000000000005},"last":0.52100000000000002},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.474,"15min":1.268},"min":{"1min":0.254,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":4.8449999999999998,"5min":9.6189999999999998,"15min":10.256},"last":0.53300000000000003}]},{"osd":1,"last update":"Fri Feb 9 17:43:42 2024","interfaces":[{"interface":"back","average":{"1min":1.8080000000000001,"5min":1.3680000000000001,"15min":1.321},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.718,"5min":8.718,"15min":8.718},"last":0.61599999999999999},{"interface":"front","average":{"1min":1.897,"5min":1.8200000000000001,"15min":1.5129999999999999},"min":{"1min":0.27600000000000002,"5min":0.25600000000000001,"15min":0.23300000000000001},"max":{"1min":8.8930000000000007,"5min":25.027000000000001,"15min":25.027000000000001},"last":0.308}]},{"osd":2,"last update":"Fri Feb 9 17:42:38 2024","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":1.3740000000000001,"15min":1.3919999999999999},"min":{"1min":0.308,"5min":0.28199999999999997,"15min":0.255},"max":{"1min":7.5789999999999997,"5min":8.5649999999999995,"15min":10.423},"last":0.39400000000000002},{"interface":"front","average":{"1min":1.048,"5min":1.4239999999999999,"15min":1.3999999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.313},"max":{"1min":7.625,"5min":8.5980000000000008,"15min":10.504},"last":0.46100000000000002}]},{"osd":3,"last update":"Fri Feb 9 17:42:38 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":1.542,"15min":1.6379999999999999},"min":{"1min":0.316,"5min":0.29499999999999998,"15min":0.28699999999999998},"max":{"1min":2.347,"5min":15.228999999999999,"15min":24.030999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.75600000000000001,"5min":1.8340000000000001,"15min":1.8640000000000001},"min":{"1min":0.36199999999999999,"5min":0.33600000000000002,"15min":0.30299999999999999},"max":{"1min":2.3719999999999999,"5min":14.988,"15min":24.318999999999999},"last":0.69399999999999995}]},{"osd":4,"last update":"Fri Feb 9 17:43:42 2024","interfaces":[{"interface":"back","average":{"1min":2.7120000000000002,"5min":1.7050000000000001,"15min":1.5760000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.27500000000000002},"max":{"1min":7.9649999999999999,"5min":12.936,"15min":53.060000000000002},"last":0.36099999999999999},{"interface":"front","average":{"1min":2.552,"5min":1.71,"15min":1.681},"min":{"1min":0.32300000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":9.6140000000000008,"5min":9.6140000000000008,"15min":53.311999999999998},"last":0.497}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.7249999999999996}]},{"osd":6,"last update":"Fri Feb 9 17:43:42 2024","interfaces":[{"interface":"back","average":{"1min":1.554,"5min":1.5129999999999999,"15min":1.573},"min":{"1min":0.33000000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":4.9450000000000003,"5min":9.5,"15min":37.356999999999999},"last":0.84599999999999997},{"interface":"front","average":{"1min":2.1150000000000002,"5min":1.6439999999999999,"15min":1.4199999999999999},"min":{"1min":0.36099999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":5.2370000000000001,"5min":7.702,"15min":37.25},"last":0.65800000000000003}]},{"osd":7,"last update":"Fri Feb 9 17:43:42 2024","interfaces":[{"interface":"back","average":{"1min":2.2999999999999998,"5min":1.718,"15min":1.659},"min":{"1min":0.51400000000000001,"5min":0.39600000000000002,"15min":0.32800000000000001},"max":{"1min":4.4989999999999997,"5min":9.0749999999999993,"15min":37.585000000000001},"last":2.274},{"interface":"front","average":{"1min":2.343,"5min":1.7589999999999999,"15min":1.704},"min":{"1min":0.53200000000000003,"5min":0.46200000000000002,"15min":0.312},"max":{"1min":4.5529999999999999,"5min":9.032,"15min":37.918999999999997},"last":2.2970000000000002}]},{"osd":9,"last update":"Fri Feb 9 17:43:42 2024","interfaces":[{"interface":"back","average":{"1min":3.6520000000000001,"5min":1.6200000000000001,"15min":1.2490000000000001},"min":{"1min":0.41199999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":45.616,"5min":45.616,"15min":45.616},"last":0.73699999999999999},{"interface":"front","average":{"1min":3.6190000000000002,"5min":1.6559999999999999,"15min":1.3089999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":45.718000000000004,"5min":45.718000000000004,"15min":45.718000000000004},"last":0.82999999999999996}]},{"osd":10,"last update":"Fri Feb 9 17:43:42 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.143,"15min":1.0620000000000001},"min":{"1min":0.34699999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":5.4589999999999996,"5min":7.7270000000000003,"15min":10.65},"last":0.81200000000000006},{"interface":"front","average":{"1min":1.75,"5min":1.581,"15min":1.1930000000000001},"min":{"1min":0.40000000000000002,"5min":0.34799999999999998,"15min":0.318},"max":{"1min":6.9539999999999997,"5min":11.071,"15min":11.071},"last":0.76900000000000002}]},{"osd":11,"last update":"Fri Feb 9 17:43:42 2024","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.335,"15min":1.3149999999999999},"min":{"1min":0.40000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":4.1029999999999998,"5min":8.5519999999999996,"15min":14.433999999999999},"last":0.79200000000000004},{"interface":"front","average":{"1min":1.798,"5min":1.4219999999999999,"15min":1.2729999999999999},"min":{"1min":0.41399999999999998,"5min":0.308,"15min":0.308},"max":{"1min":4.6180000000000003,"5min":7.4359999999999999,"15min":11.622},"last":0.58399999999999996}]}]},{"osd":7,"up_from":57,"seq":244813136150,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1622152,"kb_used_data":1081468,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":92123000,"statfs":{"total":95995035648,"available":94333952000,"internally_reserved":0,"allocated":1107423232,"data_stored":1107054510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":553646538},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:43:23 2024","interfaces":[{"interface":"back","average":{"1min":1.829,"5min":1.397,"15min":2.407},"min":{"1min":0.27500000000000002,"5min":0.252,"15min":0.252},"max":{"1min":11.74,"5min":14.348000000000001,"15min":36.753},"last":0.434},{"interface":"front","average":{"1min":1.8200000000000001,"5min":1.3799999999999999,"15min":2.5779999999999998},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":11.843999999999999,"5min":11.843999999999999,"15min":36.918999999999997},"last":0.27000000000000002}]},{"osd":1,"last update":"Fri Feb 9 17:43:23 2024","interfaces":[{"interface":"back","average":{"1min":2.4489999999999998,"5min":2.1110000000000002,"15min":2.0369999999999999},"min":{"1min":0.24299999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":6.46,"5min":21.704999999999998,"15min":21.704999999999998},"last":3.5670000000000002},{"interface":"front","average":{"1min":2.649,"5min":2.0379999999999998,"15min":1.8939999999999999},"min":{"1min":0.28499999999999998,"5min":0.27000000000000002,"15min":0.26500000000000001},"max":{"1min":6.5830000000000002,"5min":21.858000000000001,"15min":21.858000000000001},"last":3.5939999999999999}]},{"osd":2,"last update":"Fri Feb 9 17:39:34 2024","interfaces":[{"interface":"back","average":{"1min":1.8520000000000001,"5min":1.7470000000000001,"15min":1.7030000000000001},"min":{"1min":0.315,"5min":0.28499999999999998,"15min":0.26000000000000001},"max":{"1min":6.6470000000000002,"5min":10.622,"15min":11.324},"last":0.49399999999999999},{"interface":"front","average":{"1min":1.8859999999999999,"5min":1.855,"15min":1.7769999999999999},"min":{"1min":0.34699999999999998,"5min":0.32100000000000001,"15min":0.28299999999999997},"max":{"1min":6.6870000000000003,"5min":19.361999999999998,"15min":19.361999999999998},"last":0.32900000000000001}]},{"osd":3,"last update":"Fri Feb 9 17:43:42 2024","interfaces":[{"interface":"back","average":{"1min":1.6930000000000001,"5min":1.6299999999999999,"15min":1.871},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.26900000000000002},"max":{"1min":11.773999999999999,"5min":11.773999999999999,"15min":27.719000000000001},"last":0.51800000000000002},{"interface":"front","average":{"1min":2.7450000000000001,"5min":1.772,"15min":1.8089999999999999},"min":{"1min":0.46300000000000002,"5min":0.32000000000000001,"15min":0.26500000000000001},"max":{"1min":12.393000000000001,"5min":12.393000000000001,"15min":24.457000000000001},"last":3.5}]},{"osd":4,"last update":"Fri Feb 9 17:43:23 2024","interfaces":[{"interface":"back","average":{"1min":1.8680000000000001,"5min":1.077,"15min":0.82499999999999996},"min":{"1min":0.34999999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":5.29,"5min":5.29,"15min":5.29},"last":5.1059999999999999},{"interface":"front","average":{"1min":2.9129999999999998,"5min":1.3759999999999999,"15min":0.86199999999999999},"min":{"1min":0.45900000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":11.967000000000001,"5min":11.967000000000001,"15min":11.967000000000001},"last":0.53400000000000003}]},{"osd":5,"last update":"Fri Feb 9 17:43:23 2024","interfaces":[{"interface":"back","average":{"1min":2.5880000000000001,"5min":1.407,"15min":0.89700000000000002},"min":{"1min":0.29499999999999998,"5min":0.249,"15min":0.249},"max":{"1min":9.6590000000000007,"5min":9.6590000000000007,"15min":9.6590000000000007},"last":0.371},{"interface":"front","average":{"1min":3.879,"5min":1.508,"15min":0.84099999999999997},"min":{"1min":0.37,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":15.228999999999999,"5min":15.228999999999999,"15min":15.228999999999999},"last":3.5409999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:43:42 2024","interfaces":[{"interface":"back","average":{"1min":4.5010000000000003,"5min":1.7849999999999999,"15min":1.839},"min":{"1min":0.49099999999999999,"5min":0.28100000000000003,"15min":0.26000000000000001},"max":{"1min":15.275,"5min":15.275,"15min":25.965},"last":4.6539999999999999},{"interface":"front","average":{"1min":3.2280000000000002,"5min":1.6759999999999999,"15min":1.569},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":18.236999999999998,"5min":18.236999999999998,"15min":18.236999999999998},"last":4.7949999999999999}]},{"osd":8,"last update":"Fri Feb 9 17:43:42 2024","interfaces":[{"interface":"back","average":{"1min":2.4350000000000001,"5min":1.546,"15min":1.508},"min":{"1min":0.39200000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":11.81,"5min":11.81,"15min":11.81},"last":0.55300000000000005},{"interface":"front","average":{"1min":2.7730000000000001,"5min":1.4590000000000001,"15min":1.448},"min":{"1min":0.51800000000000002,"5min":0.36599999999999999,"15min":0.33300000000000002},"max":{"1min":12.058,"5min":12.058,"15min":12.058},"last":3.657}]},{"osd":9,"last update":"Fri Feb 9 17:43:31 2024","interfaces":[{"interface":"back","average":{"1min":2.867,"5min":1.5820000000000001,"15min":1.702},"min":{"1min":0.42299999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":11.941000000000001,"5min":11.941000000000001,"15min":41.286999999999999},"last":4.6139999999999999},{"interface":"front","average":{"1min":2.8690000000000002,"5min":1.6950000000000001,"15min":1.819},"min":{"1min":0.54100000000000004,"5min":0.379,"15min":0.33400000000000002},"max":{"1min":11.906000000000001,"5min":11.906000000000001,"15min":41.234999999999999},"last":4.5880000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:42:55 2024","interfaces":[{"interface":"back","average":{"1min":1.5249999999999999,"5min":1.45,"15min":1.4830000000000001},"min":{"1min":0.38900000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":7.3090000000000002,"5min":12.050000000000001,"15min":12.050000000000001},"last":4.5099999999999998},{"interface":"front","average":{"1min":2.028,"5min":1.4510000000000001,"15min":1.5840000000000001},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.35599999999999998},"max":{"1min":9.3879999999999999,"5min":9.3879999999999999,"15min":9.3879999999999999},"last":0.57299999999999995}]},{"osd":11,"last update":"Fri Feb 9 17:43:23 2024","interfaces":[{"interface":"back","average":{"1min":1.9730000000000001,"5min":1.7490000000000001,"15min":1.794},"min":{"1min":0.42899999999999999,"5min":0.42399999999999999,"15min":0.39500000000000002},"max":{"1min":5.242,"5min":21.945,"15min":21.945},"last":0.48099999999999998},{"interface":"front","average":{"1min":1.8300000000000001,"5min":1.742,"15min":1.659},"min":{"1min":0.501,"5min":0.43099999999999999,"15min":0.39500000000000002},"max":{"1min":4.5199999999999996,"5min":21.731999999999999,"15min":21.731999999999999},"last":3.6200000000000001}]}]},{"osd":6,"up_from":49,"seq":210453397787,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1238572,"kb_used_data":832548,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":92506580,"statfs":{"total":95995035648,"available":94726737920,"internally_reserved":0,"allocated":852529152,"data_stored":852172403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":415758790},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:43:14 2024","interfaces":[{"interface":"back","average":{"1min":2.6749999999999998,"5min":1.742,"15min":1.615},"min":{"1min":0.35999999999999999,"5min":0.23999999999999999,"15min":0.21299999999999999},"max":{"1min":6.4130000000000003,"5min":9.2089999999999996,"15min":12.544},"last":0.44700000000000001},{"interface":"front","average":{"1min":2.661,"5min":1.829,"15min":1.5720000000000001},"min":{"1min":0.42699999999999999,"5min":0.27800000000000002,"15min":0.217},"max":{"1min":6.3460000000000001,"5min":9.2520000000000007,"15min":9.2520000000000007},"last":0.54000000000000004}]},{"osd":1,"last update":"Fri Feb 9 17:43:14 2024","interfaces":[{"interface":"back","average":{"1min":1.8100000000000001,"5min":1.766,"15min":1.8779999999999999},"min":{"1min":0.315,"5min":0.28599999999999998,"15min":0.25900000000000001},"max":{"1min":7.5460000000000003,"5min":8.3260000000000005,"15min":17.762},"last":0.373},{"interface":"front","average":{"1min":2.6160000000000001,"5min":1.841,"15min":1.9410000000000001},"min":{"1min":0.375,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":6.3799999999999999,"5min":6.7300000000000004,"15min":17.734999999999999},"last":0.57199999999999995}]},{"osd":2,"last update":"Fri Feb 9 17:43:14 2024","interfaces":[{"interface":"back","average":{"1min":2.7709999999999999,"5min":1.633,"15min":1.6200000000000001},"min":{"1min":0.48799999999999999,"5min":0.32000000000000001,"15min":0.22600000000000001},"max":{"1min":6.4429999999999996,"5min":9.8019999999999996,"15min":10.682},"last":0.61699999999999999},{"interface":"front","average":{"1min":2.8010000000000002,"5min":1.673,"15min":1.679},"min":{"1min":0.51200000000000001,"5min":0.372,"15min":0.28000000000000003},"max":{"1min":6.5,"5min":9.8439999999999994,"15min":10.66},"last":0.64400000000000002}]},{"osd":3,"last update":"Fri Feb 9 17:43:14 2024","interfaces":[{"interface":"back","average":{"1min":2.0209999999999999,"5min":1.7589999999999999,"15min":1.9710000000000001},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.25600000000000001},"max":{"1min":5.5259999999999998,"5min":9.6389999999999993,"15min":32.545000000000002},"last":0.29599999999999999},{"interface":"front","average":{"1min":2.0979999999999999,"5min":1.889,"15min":1.9330000000000001},"min":{"1min":0.371,"5min":0.307,"15min":0.29599999999999999},"max":{"1min":7.593,"5min":11.122,"15min":32.670999999999999},"last":6.234}]},{"osd":4,"last update":"Fri Feb 9 17:43:14 2024","interfaces":[{"interface":"back","average":{"1min":3.1640000000000001,"5min":1.478,"15min":1.5940000000000001},"min":{"1min":0.311,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":8.6289999999999996,"5min":8.6289999999999996,"15min":19.798999999999999},"last":1.6850000000000001},{"interface":"front","average":{"1min":4.3250000000000002,"5min":1.835,"15min":1.583},"min":{"1min":0.49099999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":11.129,"5min":11.129,"15min":11.129},"last":2.2000000000000002}]},{"osd":5,"last update":"Fri Feb 9 17:43:14 2024","interfaces":[{"interface":"back","average":{"1min":3.3839999999999999,"5min":1.7869999999999999,"15min":1.5840000000000001},"min":{"1min":0.55100000000000005,"5min":0.29799999999999999,"15min":0.29499999999999998},"max":{"1min":9.9849999999999994,"5min":9.9849999999999994,"15min":10.968999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":2.3029999999999999,"5min":1.4530000000000001,"15min":1.5449999999999999},"min":{"1min":0.34699999999999998,"5min":0.316,"15min":0.29299999999999998},"max":{"1min":7.6890000000000001,"5min":8.3989999999999991,"15min":13.91},"last":0.52400000000000002}]},{"osd":7,"last update":"Fri Feb 9 17:42:55 2024","interfaces":[{"interface":"back","average":{"1min":3.3170000000000002,"5min":1.534,"15min":1.595},"min":{"1min":0.47599999999999998,"5min":0.29399999999999998,"15min":0.28000000000000003},"max":{"1min":11.052,"5min":11.052,"15min":14.022},"last":2.1720000000000002},{"interface":"front","average":{"1min":2.5,"5min":1.5800000000000001,"15min":1.7390000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.309},"max":{"1min":7.7949999999999999,"5min":12.441000000000001,"15min":15.458},"last":0.55600000000000005}]},{"osd":8,"last update":"Fri Feb 9 17:26:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":0.876,"15min":0.79600000000000004},"min":{"1min":0.39900000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":4.0339999999999998,"5min":4.4880000000000004,"15min":4.4880000000000004},"last":4.2809999999999997},{"interface":"front","average":{"1min":1.0469999999999999,"5min":0.88600000000000001,"15min":0.78300000000000003},"min":{"1min":0.375,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":3.6539999999999999,"5min":4.8570000000000002,"15min":4.8570000000000002},"last":2.9529999999999998}]},{"osd":9,"last update":"Fri Feb 9 17:43:19 2024","interfaces":[{"interface":"back","average":{"1min":3.1850000000000001,"5min":1.714,"15min":1.482},"min":{"1min":0.61099999999999999,"5min":0.38800000000000001,"15min":0.35899999999999999},"max":{"1min":6.5919999999999996,"5min":10.779999999999999,"15min":10.803000000000001},"last":2.2639999999999998},{"interface":"front","average":{"1min":3.036,"5min":1.8,"15min":1.554},"min":{"1min":0.627,"5min":0.40999999999999998,"15min":0.40400000000000003},"max":{"1min":6.4989999999999997,"5min":6.5640000000000001,"15min":8.1630000000000003},"last":2.2909999999999999}]},{"osd":10,"last update":"Fri Feb 9 17:42:55 2024","interfaces":[{"interface":"back","average":{"1min":2.8380000000000001,"5min":1.4970000000000001,"15min":1.4750000000000001},"min":{"1min":0.34499999999999997,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":11.744999999999999,"5min":11.744999999999999,"15min":11.744999999999999},"last":0.63},{"interface":"front","average":{"1min":2.1659999999999999,"5min":1.3720000000000001,"15min":1.4970000000000001},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":5.5,"5min":5.9509999999999996,"15min":13.836},"last":0.748}]},{"osd":11,"last update":"Fri Feb 9 17:43:14 2024","interfaces":[{"interface":"back","average":{"1min":3.867,"5min":1.8420000000000001,"15min":1.595},"min":{"1min":0.64600000000000002,"5min":0.46200000000000002,"15min":0.36099999999999999},"max":{"1min":11.1,"5min":11.1,"15min":11.1},"last":5.0960000000000001},{"interface":"front","average":{"1min":3.3809999999999998,"5min":1.673,"15min":1.5920000000000001},"min":{"1min":0.40000000000000002,"5min":0.373,"15min":0.373},"max":{"1min":9.3499999999999996,"5min":9.3499999999999996,"15min":9.3499999999999996},"last":3.379}]}]},{"osd":1,"up_from":13,"seq":55834575157,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1714988,"kb_used_data":1116004,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":92030164,"statfs":{"total":95995035648,"available":94238887936,"internally_reserved":0,"allocated":1142788096,"data_stored":1142431347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:43:31 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":1.4930000000000001,"15min":1.6020000000000001},"min":{"1min":0.25600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":2.6629999999999998,"5min":15.991,"15min":19.079000000000001},"last":0.433},{"interface":"front","average":{"1min":0.45100000000000001,"5min":1.294,"15min":1.597},"min":{"1min":0.23499999999999999,"5min":0.218,"15min":0.19400000000000001},"max":{"1min":1.6599999999999999,"5min":12.220000000000001,"15min":23.573},"last":0.371}]},{"osd":2,"last update":"Fri Feb 9 17:42:54 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":1.671,"15min":1.667},"min":{"1min":0.23400000000000001,"5min":0.23200000000000001,"15min":0.22600000000000001},"max":{"1min":1.3029999999999999,"5min":12.52,"15min":26.361999999999998},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.7030000000000001,"15min":1.919},"min":{"1min":0.27500000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":4.4889999999999999,"5min":9.3810000000000002,"15min":26.087},"last":0.38500000000000001}]},{"osd":3,"last update":"Fri Feb 9 17:40:36 2024","interfaces":[{"interface":"back","average":{"1min":1.8959999999999999,"5min":2.2690000000000001,"15min":1.6060000000000001},"min":{"1min":0.26200000000000001,"5min":0.245,"15min":0.245},"max":{"1min":6.9400000000000004,"5min":10.657,"15min":10.657},"last":0.54500000000000004},{"interface":"front","average":{"1min":2.2349999999999999,"5min":2.508,"15min":1.734},"min":{"1min":0.36599999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":7.1029999999999998,"5min":14.68,"15min":14.68},"last":0.60499999999999998}]},{"osd":4,"last update":"Fri Feb 9 17:43:25 2024","interfaces":[{"interface":"back","average":{"1min":1.6899999999999999,"5min":1.776,"15min":1.875},"min":{"1min":0.29099999999999998,"5min":0.25800000000000001,"15min":0.253},"max":{"1min":5.79,"5min":9.4649999999999999,"15min":11.766999999999999},"last":0.624},{"interface":"front","average":{"1min":2.5790000000000002,"5min":1.72,"15min":1.875},"min":{"1min":0.316,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":14.593,"5min":14.593,"15min":14.593},"last":0.59099999999999997}]},{"osd":5,"last update":"Fri Feb 9 17:43:11 2024","interfaces":[{"interface":"back","average":{"1min":2.1219999999999999,"5min":1.895,"15min":1.8280000000000001},"min":{"1min":0.30399999999999999,"5min":0.28999999999999998,"15min":0.27000000000000002},"max":{"1min":6.2060000000000004,"5min":9.4299999999999997,"15min":13.544},"last":0.90500000000000003},{"interface":"front","average":{"1min":1.3580000000000001,"5min":1.8160000000000001,"15min":1.758},"min":{"1min":0.33700000000000002,"5min":0.30499999999999999,"15min":0.26800000000000002},"max":{"1min":4.5659999999999998,"5min":10.263999999999999,"15min":12.117000000000001},"last":0.88500000000000001}]},{"osd":6,"last update":"Fri Feb 9 17:43:17 2024","interfaces":[{"interface":"back","average":{"1min":1.764,"5min":1.359,"15min":1.3009999999999999},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":6.6200000000000001,"5min":6.6200000000000001,"15min":6.6200000000000001},"last":0.67500000000000004},{"interface":"front","average":{"1min":1.806,"5min":1.3129999999999999,"15min":1.2210000000000001},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":6.6449999999999996,"5min":6.6449999999999996,"15min":6.6449999999999996},"last":0.72199999999999998}]},{"osd":7,"last update":"Fri Feb 9 17:43:17 2024","interfaces":[{"interface":"back","average":{"1min":1.6879999999999999,"5min":1.2529999999999999,"15min":1.1890000000000001},"min":{"1min":0.39400000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":4.7320000000000002,"5min":4.7320000000000002,"15min":4.7320000000000002},"last":0.82399999999999995},{"interface":"front","average":{"1min":2.302,"5min":1.423,"15min":1.2989999999999999},"min":{"1min":0.46600000000000003,"5min":0.309,"15min":0.309},"max":{"1min":6.5309999999999997,"5min":6.5309999999999997,"15min":6.5309999999999997},"last":0.84799999999999998}]},{"osd":8,"last update":"Fri Feb 9 17:43:25 2024","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":0.90700000000000003,"15min":0.89700000000000002},"min":{"1min":0.377,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":9.2520000000000007,"5min":9.2520000000000007,"15min":9.2520000000000007},"last":0.75},{"interface":"front","average":{"1min":1.302,"5min":0.90100000000000002,"15min":0.86699999999999999},"min":{"1min":0.499,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":9.2810000000000006,"5min":9.2810000000000006,"15min":9.2810000000000006},"last":0.79900000000000004}]},{"osd":9,"last update":"Fri Feb 9 17:43:31 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":1.4330000000000001,"15min":1.593},"min":{"1min":0.34200000000000003,"5min":0.33400000000000002,"15min":0.32400000000000001},"max":{"1min":2.6240000000000001,"5min":7.0640000000000001,"15min":9.5839999999999996},"last":0.66000000000000003},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.76,"15min":1.6479999999999999},"min":{"1min":0.38800000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":4.4349999999999996,"5min":10.548,"15min":12.117000000000001},"last":3.048}]},{"osd":10,"last update":"Fri Feb 9 17:43:11 2024","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.819,"15min":1.631},"min":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":4.3520000000000003,"5min":15.083,"15min":15.083},"last":3.4630000000000001},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.8620000000000001,"15min":1.8},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.40899999999999997},"max":{"1min":4.3739999999999997,"5min":15.121,"15min":15.121},"last":3.5}]},{"osd":11,"last update":"Fri Feb 9 17:43:17 2024","interfaces":[{"interface":"back","average":{"1min":1.625,"5min":1.085,"15min":0.995},"min":{"1min":0.41599999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":4.4370000000000003,"5min":4.4370000000000003,"15min":4.4370000000000003},"last":5.3099999999999996},{"interface":"front","average":{"1min":1.153,"5min":1.2110000000000001,"15min":1.2210000000000001},"min":{"1min":0.48599999999999999,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":3.657,"5min":3.657,"15min":3.657},"last":0.77500000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738681,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2566956,"kb_used_data":1980452,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":91178196,"statfs":{"total":95995035648,"available":93366472704,"internally_reserved":0,"allocated":2027982848,"data_stored":2027626099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":600570309},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 9 17:43:12 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":1.462,"15min":1.476},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20499999999999999},"max":{"1min":1.923,"5min":13.842000000000001,"15min":13.842000000000001},"last":3.0979999999999999},{"interface":"front","average":{"1min":0.70799999999999996,"5min":1.6080000000000001,"15min":1.6619999999999999},"min":{"1min":0.28899999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.4159999999999999,"5min":14.122,"15min":20.177},"last":3.1299999999999999}]},{"osd":2,"last update":"Fri Feb 9 17:39:25 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":5.5339999999999998,"5min":5.5339999999999998,"15min":5.5339999999999998},"last":1.744},{"interface":"front","average":{"1min":1.593,"5min":1.593,"15min":1.593},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":13.473000000000001,"5min":13.473000000000001,"15min":13.473000000000001},"last":3.109}]},{"osd":3,"last update":"Fri Feb 9 17:42:45 2024","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":2.1389999999999998,"15min":2.7280000000000002},"min":{"1min":0.29199999999999998,"5min":0.247,"15min":0.247},"max":{"1min":4.3849999999999998,"5min":16.756,"15min":16.756},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.90400000000000003,"5min":1.3580000000000001,"15min":1.6559999999999999},"min":{"1min":0.252,"5min":0.221,"15min":0.221},"max":{"1min":4.165,"5min":6.0570000000000004,"15min":6.0570000000000004},"last":0.375}]},{"osd":4,"last update":"Fri Feb 9 17:43:34 2024","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":1.1060000000000001,"15min":1.617},"min":{"1min":0.32900000000000001,"5min":0.28199999999999997,"15min":0.26400000000000001},"max":{"1min":7.7830000000000004,"5min":7.7830000000000004,"15min":14.419},"last":0.27000000000000002},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.609,"15min":1.8480000000000001},"min":{"1min":0.371,"5min":0.29399999999999998,"15min":0.27200000000000002},"max":{"1min":7.8230000000000004,"5min":13.308999999999999,"15min":14.465999999999999},"last":0.32300000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:42:51 2024","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.367,"15min":1.6080000000000001},"min":{"1min":0.31,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":4.1479999999999997,"5min":8.4079999999999995,"15min":9.8569999999999993},"last":4.1790000000000003},{"interface":"front","average":{"1min":1.294,"5min":1.1779999999999999,"15min":1.7050000000000001},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":3.8639999999999999,"5min":5.7839999999999998,"15min":9.9049999999999994},"last":3.1920000000000002}]},{"osd":6,"last update":"Fri Feb 9 17:43:21 2024","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.5229999999999999,"15min":1.7430000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.27800000000000002},"max":{"1min":4.5999999999999996,"5min":13.407,"15min":13.407},"last":3.2679999999999998},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.571,"15min":1.8540000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33000000000000002},"max":{"1min":4.6349999999999998,"5min":13.448,"15min":13.448},"last":3.29}]},{"osd":7,"last update":"Fri Feb 9 17:42:45 2024","interfaces":[{"interface":"back","average":{"1min":1.8520000000000001,"5min":2.0329999999999999,"15min":2.0219999999999998},"min":{"1min":0.312,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":5.5449999999999999,"5min":13.349,"15min":13.349},"last":3.9039999999999999},{"interface":"front","average":{"1min":1.165,"5min":1.851,"15min":1.9730000000000001},"min":{"1min":0.33300000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":3.3719999999999999,"5min":14.451000000000001,"15min":16.385000000000002},"last":3.4670000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:42:45 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.7689999999999999,"15min":1.538},"min":{"1min":0.33900000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":4.4770000000000003,"5min":14.323,"15min":14.323},"last":0.48199999999999998},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.875,"15min":1.6479999999999999},"min":{"1min":0.376,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":4.5579999999999998,"5min":14.481999999999999,"15min":16.41},"last":0.39400000000000002}]},{"osd":9,"last update":"Fri Feb 9 17:43:34 2024","interfaces":[{"interface":"back","average":{"1min":2.9340000000000002,"5min":2.2120000000000002,"15min":1.8260000000000001},"min":{"1min":0.38300000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":9.6010000000000009,"5min":17.952000000000002,"15min":17.952000000000002},"last":0.42299999999999999},{"interface":"front","average":{"1min":2.5289999999999999,"5min":2.1640000000000001,"15min":1.7809999999999999},"min":{"1min":0.436,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":10.858000000000001,"5min":17.913,"15min":17.913},"last":0.45500000000000002}]},{"osd":10,"last update":"Fri Feb 9 17:42:51 2024","interfaces":[{"interface":"back","average":{"1min":2.8250000000000002,"5min":2.2029999999999998,"15min":1.8540000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":31.766999999999999,"5min":31.766999999999999,"15min":31.766999999999999},"last":1.964},{"interface":"front","average":{"1min":2.609,"5min":1.333,"15min":1.53},"min":{"1min":0.42199999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":28.048999999999999,"5min":28.048999999999999,"15min":28.048999999999999},"last":3.1629999999999998}]},{"osd":11,"last update":"Fri Feb 9 17:43:21 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.151,"15min":1.6180000000000001},"min":{"1min":0.45200000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":3.968,"5min":5.8099999999999996,"15min":40.021000000000001},"last":0.442},{"interface":"front","average":{"1min":1.78,"5min":1.3420000000000001,"15min":1.774},"min":{"1min":0.47299999999999998,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":4.6820000000000004,"5min":6.9180000000000001,"15min":40.055999999999997},"last":0.46800000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411631,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":685556,"kb_used_data":451692,"kb_used_omap":1,"kb_used_meta":233854,"kb_avail":93059596,"statfs":{"total":95995035648,"available":95293026304,"internally_reserved":0,"allocated":462532608,"data_stored":462172307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":239466949},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":1.4510000000000001,"15min":1.006},"min":{"1min":0.29999999999999999,"5min":0.23699999999999999,"15min":0.221},"max":{"1min":4.9089999999999998,"5min":17.251000000000001,"15min":17.251000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.74399999999999999,"5min":1.611,"15min":1.113},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.22900000000000001},"max":{"1min":4.7720000000000002,"5min":23.097999999999999,"15min":23.097999999999999},"last":0.875}]},{"osd":1,"last update":"Fri Feb 9 17:43:44 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":1.677,"15min":1.536},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.86199999999999999,"5min":15.43,"15min":22.306000000000001},"last":0.309},{"interface":"front","average":{"1min":0.58199999999999996,"5min":1.627,"15min":1.629},"min":{"1min":0.26900000000000002,"5min":0.252,"15min":0.214},"max":{"1min":1.3400000000000001,"5min":18.181000000000001,"15min":44.838999999999999},"last":0.33400000000000002}]},{"osd":3,"last update":"Fri Feb 9 17:43:16 2024","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.4730000000000001,"15min":1.756},"min":{"1min":0.33600000000000002,"5min":0.249,"15min":0.22700000000000001},"max":{"1min":6.1420000000000003,"5min":17.942,"15min":37.634999999999998},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.49299999999999999,"5min":1.508,"15min":1.7729999999999999},"min":{"1min":0.23999999999999999,"5min":0.217,"15min":0.217},"max":{"1min":2.2320000000000002,"5min":10.912000000000001,"15min":38.924999999999997},"last":0.29699999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:43:16 2024","interfaces":[{"interface":"back","average":{"1min":1.302,"5min":1.5860000000000001,"15min":1.8120000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.26300000000000001},"max":{"1min":7.5469999999999997,"5min":10.048999999999999,"15min":13.526},"last":0.45000000000000001},{"interface":"front","average":{"1min":1.448,"5min":1.4730000000000001,"15min":1.5940000000000001},"min":{"1min":0.30299999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":5.2050000000000001,"5min":8.9920000000000009,"15min":9.9619999999999997},"last":0.372}]},{"osd":5,"last update":"Fri Feb 9 17:43:00 2024","interfaces":[{"interface":"back","average":{"1min":3.3849999999999998,"5min":1.982,"15min":1.8360000000000001},"min":{"1min":0.35599999999999998,"5min":0.33200000000000002,"15min":0.28799999999999998},"max":{"1min":42.664000000000001,"5min":42.664000000000001,"15min":42.664000000000001},"last":0.32200000000000001},{"interface":"front","average":{"1min":3.7250000000000001,"5min":2.0990000000000002,"15min":1.929},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.29899999999999999},"max":{"1min":43.311999999999998,"5min":43.311999999999998,"15min":43.311999999999998},"last":0.68400000000000005}]},{"osd":6,"last update":"Fri Feb 9 17:43:16 2024","interfaces":[{"interface":"back","average":{"1min":1.8520000000000001,"5min":1.766,"15min":1.7769999999999999},"min":{"1min":0.312,"5min":0.312,"15min":0.30099999999999999},"max":{"1min":9.9109999999999996,"5min":10.295,"15min":10.295},"last":1.042},{"interface":"front","average":{"1min":2.085,"5min":2.0030000000000001,"15min":2.0099999999999998},"min":{"1min":0.439,"5min":0.36799999999999999,"15min":0.33800000000000002},"max":{"1min":7.1269999999999998,"5min":10.144,"15min":11.269},"last":0.57699999999999996}]},{"osd":7,"last update":"Fri Feb 9 17:43:44 2024","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.8029999999999999,"15min":2.004},"min":{"1min":0.374,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":4.7679999999999998,"5min":11.082000000000001,"15min":41.802999999999997},"last":0.56100000000000005},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.8440000000000001,"15min":2.056},"min":{"1min":0.35499999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":4.7530000000000001,"5min":11.099,"15min":41.783999999999999},"last":0.61899999999999999}]},{"osd":8,"last update":"Fri Feb 9 17:43:16 2024","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.454,"15min":1.3200000000000001},"min":{"1min":0.40500000000000003,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":4.1219999999999999,"5min":7.9930000000000003,"15min":22.393999999999998},"last":0.59599999999999997},{"interface":"front","average":{"1min":2.0539999999999998,"5min":1.504,"15min":1.498},"min":{"1min":0.438,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":12.58,"5min":12.58,"15min":12.58},"last":0.41999999999999998}]},{"osd":9,"last update":"Fri Feb 9 17:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":1.4650000000000001,"15min":1.651},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.407,"5min":9.0700000000000003,"15min":13.5},"last":12.768000000000001},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.4039999999999999,"15min":1.5820000000000001},"min":{"1min":0.46500000000000002,"5min":0.41599999999999998,"15min":0.36399999999999999},"max":{"1min":4.492,"5min":10.206,"15min":13.589},"last":7.5099999999999998}]},{"osd":10,"last update":"Fri Feb 9 17:43:09 2024","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":1.6519999999999999,"15min":1.49},"min":{"1min":0.432,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":9.1419999999999995,"5min":9.1419999999999995,"15min":22.425999999999998},"last":1.647},{"interface":"front","average":{"1min":1.6299999999999999,"5min":1.6870000000000001,"15min":1.573},"min":{"1min":0.47299999999999998,"5min":0.379,"15min":0.379},"max":{"1min":9.1899999999999995,"5min":9.1899999999999995,"15min":22.454999999999998},"last":1.6839999999999999}]},{"osd":11,"last update":"Fri Feb 9 17:43:16 2024","interfaces":[{"interface":"back","average":{"1min":1.6679999999999999,"5min":3.0920000000000001,"15min":2.0409999999999999},"min":{"1min":0.47799999999999998,"5min":0.437,"15min":0.39100000000000001},"max":{"1min":5.5339999999999998,"5min":137.44300000000001,"15min":137.44300000000001},"last":0.65000000000000002},{"interface":"front","average":{"1min":1.3180000000000001,"5min":2.8399999999999999,"15min":2.198},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.40100000000000002},"max":{"1min":5.6429999999999998,"5min":137.64400000000001,"15min":137.64400000000001},"last":0.53600000000000003}]}]},{"osd":3,"up_from":27,"seq":115964117292,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":676220,"kb_used_data":266032,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":93068932,"statfs":{"total":95995035648,"available":95302586368,"internally_reserved":0,"allocated":272416768,"data_stored":272055670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":420018631},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:43:11 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":1.587,"15min":1.778},"min":{"1min":0.24199999999999999,"5min":0.23300000000000001,"15min":0.186},"max":{"1min":2.2629999999999999,"5min":25.849,"15min":38.984999999999999},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.93300000000000005,"5min":1.6140000000000001,"15min":1.911},"min":{"1min":0.248,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":8.1039999999999992,"5min":30.268999999999998,"15min":41.497999999999998},"last":0.34000000000000002}]},{"osd":1,"last update":"Fri Feb 9 17:40:06 2024","interfaces":[{"interface":"back","average":{"1min":4.5670000000000002,"5min":4.5670000000000002,"15min":4.5670000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":25.965,"5min":25.965,"15min":25.965},"last":0.35299999999999998},{"interface":"front","average":{"1min":4.5129999999999999,"5min":4.5129999999999999,"15min":4.5129999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":31.454000000000001,"5min":31.454000000000001,"15min":31.454000000000001},"last":0.375}]},{"osd":2,"last update":"Fri Feb 9 17:43:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":1.4730000000000001,"15min":1.605},"min":{"1min":0.307,"5min":0.255,"15min":0.22600000000000001},"max":{"1min":2.169,"5min":11,"15min":14.888999999999999},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.61799999999999999,"5min":1.556,"15min":1.653},"min":{"1min":0.28899999999999998,"5min":0.24199999999999999,"15min":0.20200000000000001},"max":{"1min":2.1880000000000002,"5min":12.347,"15min":17.613},"last":0.311}]},{"osd":4,"last update":"Fri Feb 9 17:43:11 2024","interfaces":[{"interface":"back","average":{"1min":1.552,"5min":1.2210000000000001,"15min":1.677},"min":{"1min":0.313,"5min":0.29099999999999998,"15min":0.26800000000000002},"max":{"1min":8.1649999999999991,"5min":10.311999999999999,"15min":34.898000000000003},"last":0.42199999999999999},{"interface":"front","average":{"1min":2.1030000000000002,"5min":1.5069999999999999,"15min":1.76},"min":{"1min":0.29399999999999998,"5min":0.29199999999999998,"15min":0.27900000000000003},"max":{"1min":12.404999999999999,"5min":12.404999999999999,"15min":14.061999999999999},"last":0.39500000000000002}]},{"osd":5,"last update":"Fri Feb 9 17:43:52 2024","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":1.639,"15min":1.663},"min":{"1min":0.313,"5min":0.28000000000000003,"15min":0.27800000000000002},"max":{"1min":4.6980000000000004,"5min":11.007999999999999,"15min":17.651},"last":0.44400000000000001},{"interface":"front","average":{"1min":1.649,"5min":1.7370000000000001,"15min":1.7290000000000001},"min":{"1min":0.42599999999999999,"5min":0.35299999999999998,"15min":0.34100000000000003},"max":{"1min":7.6219999999999999,"5min":11.036,"15min":17.675999999999998},"last":0.436}]},{"osd":6,"last update":"Fri Feb 9 17:43:43 2024","interfaces":[{"interface":"back","average":{"1min":2.1419999999999999,"5min":1.925,"15min":2.4710000000000001},"min":{"1min":0.48999999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":6.4109999999999996,"5min":12.717000000000001,"15min":12.717000000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":2.423,"5min":1.524,"15min":1.4450000000000001},"min":{"1min":0.58599999999999997,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":8.6059999999999999,"5min":8.7360000000000007,"15min":8.7360000000000007},"last":0.40899999999999997}]},{"osd":7,"last update":"Fri Feb 9 17:43:21 2024","interfaces":[{"interface":"back","average":{"1min":2.7029999999999998,"5min":1.329,"15min":1.024},"min":{"1min":0.442,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":8.4450000000000003,"5min":8.4450000000000003,"15min":8.4450000000000003},"last":0.46100000000000002},{"interface":"front","average":{"1min":1.2430000000000001,"5min":0.91000000000000003,"15min":0.80600000000000005},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":3.8719999999999999,"5min":3.8719999999999999,"15min":3.8719999999999999},"last":0.48099999999999998}]},{"osd":8,"last update":"Fri Feb 9 17:43:11 2024","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.4099999999999999,"15min":1.4199999999999999},"min":{"1min":0.41099999999999998,"5min":0.34799999999999998,"15min":0.26800000000000002},"max":{"1min":8.1969999999999992,"5min":8.6940000000000008,"15min":8.6940000000000008},"last":0.49399999999999999},{"interface":"front","average":{"1min":1.21,"5min":1.3680000000000001,"15min":1.2549999999999999},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.34499999999999997},"max":{"1min":4.819,"5min":10.388999999999999,"15min":14.129},"last":0.52700000000000002}]},{"osd":9,"last update":"Fri Feb 9 17:43:36 2024","interfaces":[{"interface":"back","average":{"1min":1.6419999999999999,"5min":1.323,"15min":1.498},"min":{"1min":0.47399999999999998,"5min":0.36199999999999999,"15min":0.32500000000000001},"max":{"1min":8.25,"5min":8.8160000000000007,"15min":9.7929999999999993},"last":0.51600000000000001},{"interface":"front","average":{"1min":1.6120000000000001,"5min":1.468,"15min":1.5029999999999999},"min":{"1min":0.52200000000000002,"5min":0.41199999999999998,"15min":0.32800000000000001},"max":{"1min":8.2260000000000009,"5min":8.7899999999999991,"15min":13.717000000000001},"last":0.54200000000000004}]},{"osd":10,"last update":"Fri Feb 9 17:43:20 2024","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":0.94899999999999995,"15min":0.89000000000000001},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":2.9540000000000002,"5min":3.2120000000000002,"15min":3.2120000000000002},"last":0.503},{"interface":"front","average":{"1min":1.151,"5min":0.83799999999999997,"15min":0.78000000000000003},"min":{"1min":0.41299999999999998,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":3.8109999999999999,"5min":3.8109999999999999,"15min":3.8109999999999999},"last":0.55300000000000005}]},{"osd":11,"last update":"Fri Feb 9 17:43:11 2024","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":1.3109999999999999,"15min":1.514},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":4.1159999999999997,"5min":14.300000000000001,"15min":14.300000000000001},"last":0.58699999999999997},{"interface":"front","average":{"1min":1.006,"5min":1.7330000000000001,"15min":1.7190000000000001},"min":{"1min":0.49299999999999999,"5min":0.442,"15min":0.39300000000000002},"max":{"1min":4.1390000000000002,"5min":17.145,"15min":17.145},"last":0.56999999999999995}]}]},{"osd":4,"up_from":32,"seq":137438953765,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2159788,"kb_used_data":1641252,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":91585364,"statfs":{"total":95995035648,"available":93783412736,"internally_reserved":0,"allocated":1680642048,"data_stored":1680285299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":530905541},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:43:21 2024","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.2509999999999999,"15min":1.7290000000000001},"min":{"1min":0.24399999999999999,"5min":0.222,"15min":0.222},"max":{"1min":6.2699999999999996,"5min":7.7199999999999998,"15min":50.356999999999999},"last":0.374},{"interface":"front","average":{"1min":1.071,"5min":1.296,"15min":1.659},"min":{"1min":0.34200000000000003,"5min":0.26700000000000002,"15min":0.247},"max":{"1min":6.2320000000000002,"5min":7.6390000000000002,"15min":50.569000000000003},"last":0.41699999999999998}]},{"osd":1,"last update":"Fri Feb 9 17:43:21 2024","interfaces":[{"interface":"back","average":{"1min":1.1279999999999999,"5min":1.637,"15min":1.96},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":4.8609999999999998,"5min":7.9569999999999999,"15min":50.473999999999997},"last":0.311},{"interface":"front","average":{"1min":2.762,"5min":1.9930000000000001,"15min":2.1520000000000001},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":11.593,"5min":11.593,"15min":50.499000000000002},"last":0.39600000000000002}]},{"osd":2,"last update":"Fri Feb 9 17:28:51 2024","interfaces":[{"interface":"back","average":{"1min":2.536,"5min":1.546,"15min":0.91400000000000003},"min":{"1min":0.29999999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":8.8819999999999997,"5min":8.8819999999999997,"15min":8.8819999999999997},"last":0.622},{"interface":"front","average":{"1min":2.214,"5min":1.7330000000000001,"15min":0.97799999999999998},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":9.5039999999999996,"5min":9.5039999999999996,"15min":9.5039999999999996},"last":0.53800000000000003}]},{"osd":3,"last update":"Fri Feb 9 17:43:21 2024","interfaces":[{"interface":"back","average":{"1min":2.371,"5min":1.762,"15min":2.1400000000000001},"min":{"1min":0.33600000000000002,"5min":0.28199999999999997,"15min":0.245},"max":{"1min":11.679,"5min":11.679,"15min":50.462000000000003},"last":1.5209999999999999},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.359,"15min":1.8819999999999999},"min":{"1min":0.318,"5min":0.29099999999999998,"15min":0.255},"max":{"1min":6.3040000000000003,"5min":9.1579999999999995,"15min":50.529000000000003},"last":0.44900000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:42:49 2024","interfaces":[{"interface":"back","average":{"1min":2.9689999999999999,"5min":1.3799999999999999,"15min":1.47},"min":{"1min":0.28399999999999997,"5min":0.22500000000000001,"15min":0.222},"max":{"1min":9.3279999999999994,"5min":9.3279999999999994,"15min":9.5630000000000006},"last":10.484999999999999},{"interface":"front","average":{"1min":3.2400000000000002,"5min":1.504,"15min":1.7569999999999999},"min":{"1min":0.35699999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":9.3599999999999994,"5min":9.3599999999999994,"15min":50.509999999999998},"last":10.510999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:43:21 2024","interfaces":[{"interface":"back","average":{"1min":1.9339999999999999,"5min":1.4159999999999999,"15min":1.5029999999999999},"min":{"1min":0.28299999999999997,"5min":0.26200000000000001,"15min":0.246},"max":{"1min":6.516,"5min":8.0039999999999996,"15min":18.09},"last":0.54000000000000004},{"interface":"front","average":{"1min":4.2030000000000003,"5min":1.9650000000000001,"15min":1.911},"min":{"1min":0.48299999999999998,"5min":0.245,"15min":0.245},"max":{"1min":13.208,"5min":13.208,"15min":50.402999999999999},"last":0.58499999999999996}]},{"osd":7,"last update":"Fri Feb 9 17:42:54 2024","interfaces":[{"interface":"back","average":{"1min":3.0289999999999999,"5min":1.585,"15min":1.6850000000000001},"min":{"1min":0.42399999999999999,"5min":0.32200000000000001,"15min":0.28399999999999997},"max":{"1min":11.875,"5min":11.875,"15min":14.919},"last":0.47699999999999998},{"interface":"front","average":{"1min":3.4289999999999998,"5min":1.6279999999999999,"15min":1.7290000000000001},"min":{"1min":0.28599999999999998,"5min":0.28199999999999997,"15min":0.28100000000000003},"max":{"1min":12.696,"5min":12.696,"15min":19.047000000000001},"last":1.2749999999999999}]},{"osd":8,"last update":"Fri Feb 9 17:42:49 2024","interfaces":[{"interface":"back","average":{"1min":2.5550000000000002,"5min":1.399,"15min":1.502},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":9.1809999999999992,"5min":9.1809999999999992,"15min":9.1809999999999992},"last":10.369},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.149,"15min":1.5669999999999999},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.308},"max":{"1min":5.2809999999999997,"5min":6,"15min":50.594000000000001},"last":0.73999999999999999}]},{"osd":9,"last update":"Fri Feb 9 17:43:34 2024","interfaces":[{"interface":"back","average":{"1min":2.2160000000000002,"5min":1.6259999999999999,"15min":1.8169999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":5.6369999999999996,"5min":9.2370000000000001,"15min":50.442999999999998},"last":10.417999999999999},{"interface":"front","average":{"1min":1.6559999999999999,"5min":1.3300000000000001,"15min":1.5029999999999999},"min":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.36199999999999999},"max":{"1min":6.3879999999999999,"5min":8.3580000000000005,"15min":10.555},"last":0.71199999999999997}]},{"osd":10,"last update":"Fri Feb 9 17:43:05 2024","interfaces":[{"interface":"back","average":{"1min":3.2709999999999999,"5min":1.9059999999999999,"15min":2.028},"min":{"1min":0.45600000000000002,"5min":0.40300000000000002,"15min":0.39400000000000002},"max":{"1min":9.2789999999999999,"5min":9.2789999999999999,"15min":51.223999999999997},"last":10.454000000000001},{"interface":"front","average":{"1min":2.0670000000000002,"5min":1.5489999999999999,"15min":1.98},"min":{"1min":0.50800000000000001,"5min":0.41999999999999998,"15min":0.39200000000000002},"max":{"1min":6.5759999999999996,"5min":6.5759999999999996,"15min":50.552},"last":0.55600000000000005}]},{"osd":11,"last update":"Fri Feb 9 17:43:37 2024","interfaces":[{"interface":"back","average":{"1min":1.8720000000000001,"5min":1.7989999999999999,"15min":2.052},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.42099999999999999},"max":{"1min":5.4569999999999999,"5min":9.4060000000000006,"15min":50.484999999999999},"last":4.2039999999999997},{"interface":"front","average":{"1min":1.889,"5min":1.4039999999999999,"15min":1.673},"min":{"1min":0.68999999999999995,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":6.5830000000000002,"5min":7.694,"15min":18.574999999999999},"last":1.3080000000000001}]}]},{"osd":5,"up_from":40,"seq":171798692127,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2409464,"kb_used_data":1968624,"kb_used_omap":1,"kb_used_meta":440702,"kb_avail":91335688,"statfs":{"total":95995035648,"available":93527744512,"internally_reserved":0,"allocated":2015870976,"data_stored":2015506603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":451279301},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:43:12 2024","interfaces":[{"interface":"back","average":{"1min":1.925,"5min":1.22,"15min":1.1080000000000001},"min":{"1min":0.26400000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":6.6829999999999998,"5min":7.3730000000000002,"15min":7.3730000000000002},"last":0.72999999999999998},{"interface":"front","average":{"1min":2.2639999999999998,"5min":1.6000000000000001,"15min":1.742},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":9.7650000000000006,"5min":9.7650000000000006,"15min":9.7650000000000006},"last":2.6640000000000001}]},{"osd":1,"last update":"Fri Feb 9 17:42:52 2024","interfaces":[{"interface":"back","average":{"1min":1.994,"5min":1.95,"15min":1.9730000000000001},"min":{"1min":0.311,"5min":0.24199999999999999,"15min":0.23200000000000001},"max":{"1min":9.7330000000000005,"5min":48.161999999999999,"15min":48.161999999999999},"last":2.7320000000000002},{"interface":"front","average":{"1min":2.1680000000000001,"5min":2.1890000000000001,"15min":1.96},"min":{"1min":0.30399999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":7.8369999999999997,"5min":48.218000000000004,"15min":48.218000000000004},"last":2.7480000000000002}]},{"osd":2,"last update":"Fri Feb 9 17:40:42 2024","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":1.345,"15min":1.6539999999999999},"min":{"1min":0.32000000000000001,"5min":0.27900000000000003,"15min":0.25600000000000001},"max":{"1min":4.4930000000000003,"5min":11.143000000000001,"15min":11.143000000000001},"last":1.5049999999999999},{"interface":"front","average":{"1min":1.796,"5min":1.218,"15min":1.718},"min":{"1min":0.39100000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":6.1970000000000001,"5min":6.1970000000000001,"15min":14.122},"last":3.3039999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:43:49 2024","interfaces":[{"interface":"back","average":{"1min":1.419,"5min":1.472,"15min":2.069},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.26500000000000001},"max":{"1min":8.4239999999999995,"5min":11.081,"15min":64.150999999999996},"last":3.2360000000000002},{"interface":"front","average":{"1min":2.073,"5min":1.7969999999999999,"15min":2.157},"min":{"1min":0.35299999999999998,"5min":0.247,"15min":0.247},"max":{"1min":5.3120000000000003,"5min":12.468,"15min":64.176000000000002},"last":2.7789999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:43:49 2024","interfaces":[{"interface":"back","average":{"1min":2.5640000000000001,"5min":2.1150000000000002,"15min":1.766},"min":{"1min":0.249,"5min":0.249,"15min":0.23400000000000001},"max":{"1min":8.4890000000000008,"5min":49.313000000000002,"15min":49.313000000000002},"last":0.98999999999999999},{"interface":"front","average":{"1min":2.601,"5min":2.2010000000000001,"15min":1.744},"min":{"1min":0.27000000000000002,"5min":0.218,"15min":0.218},"max":{"1min":7.0010000000000003,"5min":50.505000000000003,"15min":50.505000000000003},"last":2.8100000000000001}]},{"osd":6,"last update":"Fri Feb 9 17:43:23 2024","interfaces":[{"interface":"back","average":{"1min":2.6859999999999999,"5min":1.748,"15min":1.675},"min":{"1min":0.36699999999999999,"5min":0.26400000000000001,"15min":0.253},"max":{"1min":7.8940000000000001,"5min":12.425000000000001,"15min":15.375999999999999},"last":2.794},{"interface":"front","average":{"1min":2.0310000000000001,"5min":1.778,"15min":1.7509999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":8.4169999999999998,"5min":11.199,"15min":14.250999999999999},"last":0.76000000000000001}]},{"osd":7,"last update":"Fri Feb 9 17:43:49 2024","interfaces":[{"interface":"back","average":{"1min":2.516,"5min":2.1899999999999999,"15min":1.742},"min":{"1min":0.40200000000000002,"5min":0.27900000000000003,"15min":0.26700000000000002},"max":{"1min":8.5359999999999996,"5min":50.545000000000002,"15min":50.545000000000002},"last":2.7629999999999999},{"interface":"front","average":{"1min":2.847,"5min":2.3130000000000002,"15min":2.3140000000000001},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":16.314,"5min":49.688000000000002,"15min":102.526},"last":1.46}]},{"osd":8,"last update":"Fri Feb 9 17:43:23 2024","interfaces":[{"interface":"back","average":{"1min":1.6499999999999999,"5min":1.639,"15min":1.6020000000000001},"min":{"1min":0.376,"5min":0.35299999999999998,"15min":0.34899999999999998},"max":{"1min":4.8780000000000001,"5min":14.542,"15min":14.542},"last":5.5439999999999996},{"interface":"front","average":{"1min":3.6699999999999999,"5min":1.702,"15min":1.8300000000000001},"min":{"1min":0.37,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":16.359000000000002,"5min":16.359000000000002,"15min":16.359000000000002},"last":0.435}]},{"osd":9,"last update":"Fri Feb 9 17:43:23 2024","interfaces":[{"interface":"back","average":{"1min":2.093,"5min":1.5469999999999999,"15min":1.5},"min":{"1min":0.442,"5min":0.38600000000000001,"15min":0.371},"max":{"1min":16.411000000000001,"5min":16.411000000000001,"15min":16.411000000000001},"last":9.9510000000000005},{"interface":"front","average":{"1min":3.2770000000000001,"5min":1.7150000000000001,"15min":1.585},"min":{"1min":0.48499999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":16.417999999999999,"5min":16.417999999999999,"15min":16.417999999999999},"last":1.581}]},{"osd":10,"last update":"Fri Feb 9 17:43:49 2024","interfaces":[{"interface":"back","average":{"1min":2.782,"5min":1.8660000000000001,"15min":1.7110000000000001},"min":{"1min":0.39800000000000002,"5min":0.317,"15min":0.317},"max":{"1min":16.445,"5min":16.445,"15min":16.445},"last":1.534},{"interface":"front","average":{"1min":2.1749999999999998,"5min":1.6519999999999999,"15min":1.744},"min":{"1min":0.48699999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":5.5999999999999996,"5min":9.8840000000000003,"15min":14.186999999999999},"last":2.6949999999999998}]},{"osd":11,"last update":"Fri Feb 9 17:43:12 2024","interfaces":[{"interface":"back","average":{"1min":4.5,"5min":2.129,"15min":1.734},"min":{"1min":0.51800000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":16.390000000000001,"5min":16.390000000000001,"15min":16.390000000000001},"last":1.556},{"interface":"front","average":{"1min":2.871,"5min":1.8959999999999999,"15min":1.734},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":9.8239999999999998,"5min":9.8239999999999998,"15min":9.8239999999999998},"last":2.714}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2002780160,"data_stored":2002780160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1129709568,"data_stored":1129709568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":471597056,"data_stored":471597056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":269811712,"data_stored":269811712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1665990656,"data_stored":1665990656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1991512064,"data_stored":1991507992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":845545472,"data_stored":845545472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1102450688,"data_stored":1102446616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":826277888,"data_stored":826277888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":833028096,"data_stored":833028096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1733230592,"data_stored":1733230592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1135218688,"data_stored":1135214616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-09T17:43:54.841 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd pool set .mgr pgp_num 31 2024-02-09T17:43:54.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:54.842 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:54 smithi099 ceph-mon[39420]: 1.19 scrub ok 2024-02-09T17:43:54.842 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:54 smithi099 ceph-mon[39420]: 5.8 scrub starts 2024-02-09T17:43:54.843 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:54 smithi099 ceph-mon[39420]: 5.4 deep-scrub starts 2024-02-09T17:43:54.843 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:54 smithi099 ceph-mon[39420]: 5.8 scrub ok 2024-02-09T17:43:54.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:43:54.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:54 smithi179 ceph-mon[43975]: 1.19 scrub ok 2024-02-09T17:43:54.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:54 smithi179 ceph-mon[43975]: 5.8 scrub starts 2024-02-09T17:43:54.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:54 smithi179 ceph-mon[43975]: 5.4 deep-scrub starts 2024-02-09T17:43:54.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:54 smithi179 ceph-mon[43975]: 5.8 scrub ok 2024-02-09T17:43:54.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:54 smithi162 ceph-mon[44304]: 1.19 scrub ok 2024-02-09T17:43:54.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:54 smithi162 ceph-mon[44304]: 5.8 scrub starts 2024-02-09T17:43:54.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:54 smithi162 ceph-mon[44304]: 5.4 deep-scrub starts 2024-02-09T17:43:54.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:54 smithi162 ceph-mon[44304]: 5.8 scrub ok 2024-02-09T17:43:54.944 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:43:54.984 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:55.025 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:43:55.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:55.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:43:55.130 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:43:55.173 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:55.200 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:43:55.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:55.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:43:55.287 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:43:55.287 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:43:55.308 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:43:55.367 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:55.409 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:43:55.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:55.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:43:55.518 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:43:55.554 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:55.586 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:43:55.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:55.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:43:55.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:55 smithi162 ceph-mon[44304]: pgmap v1351: 47 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 44 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 859 op/s; 4242/201657 objects misplaced (2.104%); 26 MiB/s, 423 objects/s recovering 2024-02-09T17:43:55.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:55 smithi162 ceph-mon[44304]: 1.1 scrub starts 2024-02-09T17:43:55.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:55 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:43:55.703 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:43:55.746 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:55.769 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:43:55.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:55.772 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:43:55.872 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:43:55.905 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:55 smithi179 ceph-mon[43975]: pgmap v1351: 47 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 44 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 859 op/s; 4242/201657 objects misplaced (2.104%); 26 MiB/s, 423 objects/s recovering 2024-02-09T17:43:55.906 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:55 smithi179 ceph-mon[43975]: 1.1 scrub starts 2024-02-09T17:43:55.906 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:55 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:43:55.908 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:55.916 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:55 smithi099 ceph-mon[39420]: pgmap v1351: 47 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 44 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 859 op/s; 4242/201657 objects misplaced (2.104%); 26 MiB/s, 423 objects/s recovering 2024-02-09T17:43:55.916 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:55 smithi099 ceph-mon[39420]: 1.1 scrub starts 2024-02-09T17:43:55.917 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:55 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:43:55.933 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:43:55.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:55.936 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:43:55.950 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-09T17:43:56.037 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:43:56.070 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:56.099 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:43:56.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:56.104 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:43:56.205 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:43:56.234 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:56.343 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:43:56.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:56.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:43:56.452 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:43:56.479 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:56.498 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:43:56.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:56.502 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:43:56.603 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:43:56.633 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:56.661 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:43:56.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:56.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:43:56.773 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:43:56.888 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:56 smithi162 ceph-mon[44304]: 1.1 scrub ok 2024-02-09T17:43:56.888 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:56 smithi162 ceph-mon[44304]: 1.12 scrub starts 2024-02-09T17:43:56.888 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:56 smithi162 ceph-mon[44304]: 1.12 scrub ok 2024-02-09T17:43:56.888 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:56 smithi162 ceph-mon[44304]: osdmap e537: 12 total, 12 up, 12 in 2024-02-09T17:43:56.890 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:56.904 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:56 smithi099 ceph-mon[39420]: 1.1 scrub ok 2024-02-09T17:43:56.904 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:56 smithi099 ceph-mon[39420]: 1.12 scrub starts 2024-02-09T17:43:56.904 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:56 smithi099 ceph-mon[39420]: 1.12 scrub ok 2024-02-09T17:43:56.904 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:56 smithi099 ceph-mon[39420]: osdmap e537: 12 total, 12 up, 12 in 2024-02-09T17:43:56.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:56 smithi179 ceph-mon[43975]: 1.1 scrub ok 2024-02-09T17:43:56.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:56 smithi179 ceph-mon[43975]: 1.12 scrub starts 2024-02-09T17:43:56.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:56 smithi179 ceph-mon[43975]: 1.12 scrub ok 2024-02-09T17:43:56.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:56 smithi179 ceph-mon[43975]: osdmap e537: 12 total, 12 up, 12 in 2024-02-09T17:43:56.953 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:43:56.957 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:56.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:43:57.059 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:43:57.110 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:57.170 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:43:57.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:57.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:43:57.285 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:43:57.365 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:57.404 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:43:57.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:43:57.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:43:57.509 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:43:57.564 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:43:57.626 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:57 smithi162 ceph-mon[44304]: 1.4 deep-scrub starts 2024-02-09T17:43:57.626 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:57 smithi162 ceph-mon[44304]: 1.4 deep-scrub ok 2024-02-09T17:43:57.627 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:57 smithi162 ceph-mon[44304]: 5.f scrub starts 2024-02-09T17:43:57.627 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:57 smithi162 ceph-mon[44304]: osdmap e538: 12 total, 12 up, 12 in 2024-02-09T17:43:57.627 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:57 smithi162 ceph-mon[44304]: 5.6 scrub starts 2024-02-09T17:43:57.627 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:57 smithi162 ceph-mon[44304]: pgmap v1354: 47 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 44 active+clean; 4.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 737 op/s; 4242/197625 objects misplaced (2.146%); 21 MiB/s, 342 objects/s recovering 2024-02-09T17:43:57.627 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:57 smithi162 ceph-mon[44304]: 5.f scrub ok 2024-02-09T17:43:57.627 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:57 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:43:57.627 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:57 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:43:57.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:57 smithi179 ceph-mon[43975]: 1.4 deep-scrub starts 2024-02-09T17:43:57.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:57 smithi179 ceph-mon[43975]: 1.4 deep-scrub ok 2024-02-09T17:43:57.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:57 smithi179 ceph-mon[43975]: 5.f scrub starts 2024-02-09T17:43:57.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:57 smithi179 ceph-mon[43975]: osdmap e538: 12 total, 12 up, 12 in 2024-02-09T17:43:57.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:57 smithi179 ceph-mon[43975]: 5.6 scrub starts 2024-02-09T17:43:57.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:57 smithi179 ceph-mon[43975]: pgmap v1354: 47 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 44 active+clean; 4.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 737 op/s; 4242/197625 objects misplaced (2.146%); 21 MiB/s, 342 objects/s recovering 2024-02-09T17:43:57.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:57 smithi179 ceph-mon[43975]: 5.f scrub ok 2024-02-09T17:43:57.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:57 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:43:57.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:57 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:43:57.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:57 smithi099 ceph-mon[39420]: 1.4 deep-scrub starts 2024-02-09T17:43:57.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:57 smithi099 ceph-mon[39420]: 1.4 deep-scrub ok 2024-02-09T17:43:57.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:57 smithi099 ceph-mon[39420]: 5.f scrub starts 2024-02-09T17:43:57.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:57 smithi099 ceph-mon[39420]: osdmap e538: 12 total, 12 up, 12 in 2024-02-09T17:43:57.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:57 smithi099 ceph-mon[39420]: 5.6 scrub starts 2024-02-09T17:43:57.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:57 smithi099 ceph-mon[39420]: pgmap v1354: 47 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 44 active+clean; 4.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 737 op/s; 4242/197625 objects misplaced (2.146%); 21 MiB/s, 342 objects/s recovering 2024-02-09T17:43:57.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:57 smithi099 ceph-mon[39420]: 5.f scrub ok 2024-02-09T17:43:57.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:57 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:43:57.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:57 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:43:58.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:58 smithi179 ceph-mon[43975]: 1.18 scrub starts 2024-02-09T17:43:58.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:58 smithi179 ceph-mon[43975]: 1.18 scrub ok 2024-02-09T17:43:58.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:58 smithi179 ceph-mon[43975]: 5.6 scrub ok 2024-02-09T17:43:58.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:58 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/897147131' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:43:58.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:58 smithi162 ceph-mon[44304]: 1.18 scrub starts 2024-02-09T17:43:58.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:58 smithi162 ceph-mon[44304]: 1.18 scrub ok 2024-02-09T17:43:58.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:58 smithi162 ceph-mon[44304]: 5.6 scrub ok 2024-02-09T17:43:58.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:58 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/897147131' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:43:58.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:58 smithi099 ceph-mon[39420]: 1.18 scrub starts 2024-02-09T17:43:58.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:58 smithi099 ceph-mon[39420]: 1.18 scrub ok 2024-02-09T17:43:58.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:58 smithi099 ceph-mon[39420]: 5.6 scrub ok 2024-02-09T17:43:58.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:58 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/897147131' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:43:59.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:59 smithi179 ceph-mon[43975]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:43:59.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:59 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/897147131' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:43:59.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:59 smithi179 ceph-mon[43975]: osdmap e539: 12 total, 12 up, 12 in 2024-02-09T17:43:59.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:59 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:43:59.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:59 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:43:59.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:59 smithi179 ceph-mon[43975]: pgmap v1356: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing+deep, 1 active+remapped+backfilling, 44 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 610 op/s; 4077/195336 objects misplaced (2.087%); 19 MiB/s, 301 objects/s recovering 2024-02-09T17:43:59.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:59 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:43:59.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:43:59 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:43:59.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:59 smithi162 ceph-mon[44304]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:43:59.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:59 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/897147131' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:43:59.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:59 smithi162 ceph-mon[44304]: osdmap e539: 12 total, 12 up, 12 in 2024-02-09T17:43:59.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:59 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:43:59.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:59 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:43:59.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:59 smithi162 ceph-mon[44304]: pgmap v1356: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing+deep, 1 active+remapped+backfilling, 44 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 610 op/s; 4077/195336 objects misplaced (2.087%); 19 MiB/s, 301 objects/s recovering 2024-02-09T17:43:59.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:59 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:43:59.931 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:43:59 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:43:59.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:59 smithi099 ceph-mon[39420]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:43:59.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:59 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/897147131' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:43:59.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:59 smithi099 ceph-mon[39420]: osdmap e539: 12 total, 12 up, 12 in 2024-02-09T17:43:59.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:59 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:43:59.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:59 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:43:59.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:59 smithi099 ceph-mon[39420]: pgmap v1356: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing+deep, 1 active+remapped+backfilling, 44 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 610 op/s; 4077/195336 objects misplaced (2.087%); 19 MiB/s, 301 objects/s recovering 2024-02-09T17:43:59.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:59 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:43:59.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:43:59 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:44:00.337 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:44:00.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:00.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:44:00.445 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:44:00.491 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:00.518 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:44:00.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:00.527 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:44:00.628 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:44:00.668 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:00.705 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:44:00.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:00.710 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:44:00.811 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:44:00.860 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:00.901 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:44:00.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:00.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:44:00.907 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:00 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:44:00.908 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:00 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:44:00.908 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:00 smithi162 ceph-mon[44304]: osdmap e540: 12 total, 12 up, 12 in 2024-02-09T17:44:00.908 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:00 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:44:00.908 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:00 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:44:00.908 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:00 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:44:00.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:00 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:44:00.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:00 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:44:00.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:00 smithi179 ceph-mon[43975]: osdmap e540: 12 total, 12 up, 12 in 2024-02-09T17:44:00.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:00 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:44:00.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:00 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:44:00.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:00 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:44:00.928 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:00 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:44:00.928 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:00 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:44:00.928 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:00 smithi099 ceph-mon[39420]: osdmap e540: 12 total, 12 up, 12 in 2024-02-09T17:44:00.928 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:00 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:44:00.928 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:00 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:44:00.928 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:00 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:44:01.006 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:44:01.055 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:01.059 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is set 2024-02-09T17:44:01.105 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:44:01.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:01.109 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:44:01.210 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:44:01.243 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:01.265 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:44:01.272 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:44:01.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:01.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:44:01.377 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:44:01.410 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:01.877 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:44:01.917 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:01 smithi099 ceph-mon[39420]: pgmap v1358: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing+deep, 45 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 638 op/s; 4014/190338 objects misplaced (2.109%); 624 KiB/s, 11 objects/s recovering 2024-02-09T17:44:01.917 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:01 smithi099 ceph-mon[39420]: osdmap e541: 12 total, 12 up, 12 in 2024-02-09T17:44:01.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:01 smithi179 ceph-mon[43975]: pgmap v1358: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing+deep, 45 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 638 op/s; 4014/190338 objects misplaced (2.109%); 624 KiB/s, 11 objects/s recovering 2024-02-09T17:44:01.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:01 smithi179 ceph-mon[43975]: osdmap e541: 12 total, 12 up, 12 in 2024-02-09T17:44:01.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:01 smithi162 ceph-mon[44304]: pgmap v1358: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing+deep, 45 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 638 op/s; 4014/190338 objects misplaced (2.109%); 624 KiB/s, 11 objects/s recovering 2024-02-09T17:44:01.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:01 smithi162 ceph-mon[44304]: osdmap e541: 12 total, 12 up, 12 in 2024-02-09T17:44:03.300 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:44:03.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:03.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:44:03.406 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:44:03.458 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:03.485 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:44:03.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:03.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:44:03.589 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:44:03.633 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:44:03.647 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:03.655 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:44:03.682 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:44:03.694 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:44:03.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:03.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:44:03.714 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:44:03.746 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:44:03.786 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:44:03.800 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:44:03.814 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:44:03.840 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:03.858 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:44:03.878 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:44:03.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:03.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:44:03.887 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:44:03.918 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-09T17:44:03.921 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:44:03.949 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:44:03.985 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:44:03.989 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:44:03.990 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:44:03.990 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:44:03.990 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:44:03.991 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:44:03.991 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:44:03.991 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:44:03.991 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:44:03.991 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:44:03.991 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:44:03.991 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:44:03.991 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:44:03.991 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:44:04.014 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:04.036 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:44:04.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:04.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:44:04.140 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:44:04.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:03 smithi179 ceph-mon[43975]: pgmap v1360: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing+deep, 45 active+clean; 3.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 622 op/s; 3809/186411 objects misplaced (2.043%); 546 KiB/s, 9 objects/s recovering 2024-02-09T17:44:04.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:03 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1902128629' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:44:04.181 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:04.189 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:03 smithi162 ceph-mon[44304]: pgmap v1360: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing+deep, 45 active+clean; 3.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 622 op/s; 3809/186411 objects misplaced (2.043%); 546 KiB/s, 9 objects/s recovering 2024-02-09T17:44:04.189 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:03 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1902128629' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:44:04.208 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:44:04.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:04.212 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:44:04.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:03 smithi099 ceph-mon[39420]: pgmap v1360: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing+deep, 45 active+clean; 3.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 622 op/s; 3809/186411 objects misplaced (2.043%); 546 KiB/s, 9 objects/s recovering 2024-02-09T17:44:04.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:03 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1902128629' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:44:04.313 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:44:04.366 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:04.626 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:44:04.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:04.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:44:04.730 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:44:04.763 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:05.618 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:44:05.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:05.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:44:05.724 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:44:05.773 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:05.806 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:44:05.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:05.817 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:44:05.918 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:44:05.957 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:05.991 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:44:05.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:05.995 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:44:06.096 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:44:06.136 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:06.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:05 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:44:06.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:05 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:44:06.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:05 smithi179 ceph-mon[43975]: pgmap v1361: 47 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 45 active+clean; 3.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 699 op/s; 3809/182274 objects misplaced (2.090%); 527 KiB/s, 9 objects/s recovering 2024-02-09T17:44:06.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:05 smithi179 ceph-mon[43975]: osdmap e542: 12 total, 12 up, 12 in 2024-02-09T17:44:06.173 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:44:06.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:06.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:44:06.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:05 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:44:06.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:05 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:44:06.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:05 smithi162 ceph-mon[44304]: pgmap v1361: 47 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 45 active+clean; 3.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 699 op/s; 3809/182274 objects misplaced (2.090%); 527 KiB/s, 9 objects/s recovering 2024-02-09T17:44:06.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:05 smithi162 ceph-mon[44304]: osdmap e542: 12 total, 12 up, 12 in 2024-02-09T17:44:06.181 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:44:06.181 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 20, 2024-02-09T17:44:06.181 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 600, 2024-02-09T17:44:06.181 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [ 2024-02-09T17:44:06.181 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.181 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:142473 4.b 4:d5341d23:::benchmark_data_smithi179_103809_object5007:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-02-09T17:44:06.181 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:37:15.431448+0000", 2024-02-09T17:44:06.181 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 410.38347396299997, 2024-02-09T17:44:06.181 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.22891070199999999, 2024-02-09T17:44:06.182 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:44:06.182 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:44:06.182 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:44:06.182 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:44:06.182 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:44:06.182 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 142473 2024-02-09T17:44:06.182 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.182 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:44:06.182 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.182 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:44:06.182 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.431448+0000", 2024-02-09T17:44:06.182 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:06.182 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.182 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.183 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:44:06.183 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.431448+0000", 2024-02-09T17:44:06.183 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:06.183 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.183 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.183 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:44:06.183 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.431450+0000", 2024-02-09T17:44:06.183 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.79e-06 2024-02-09T17:44:06.183 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.183 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.183 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:44:06.183 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.431458+0000", 2024-02-09T17:44:06.183 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.8269999999999998e-06 2024-02-09T17:44:06.183 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.184 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.184 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:44:06.184 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.431479+0000", 2024-02-09T17:44:06.184 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.1160000000000001e-05 2024-02-09T17:44:06.184 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.184 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.184 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:44:06.184 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.431489+0000", 2024-02-09T17:44:06.184 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0558e-05 2024-02-09T17:44:06.184 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.184 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.184 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:06.184 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.433160+0000", 2024-02-09T17:44:06.185 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0016701369999999999 2024-02-09T17:44:06.185 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.185 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.185 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:44:06.185 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.433300+0000", 2024-02-09T17:44:06.185 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014065700000000001 2024-02-09T17:44:06.185 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.185 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.185 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:44:06.185 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.433419+0000", 2024-02-09T17:44:06.185 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00011865400000000001 2024-02-09T17:44:06.185 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.185 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.185 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:44:06.185 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.447885+0000", 2024-02-09T17:44:06.186 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.014466494 2024-02-09T17:44:06.186 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.186 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.186 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:06.186 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.450722+0000", 2024-02-09T17:44:06.186 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0028368159999999998 2024-02-09T17:44:06.186 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.186 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.186 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:06.186 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.660293+0000", 2024-02-09T17:44:06.186 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20957071599999999 2024-02-09T17:44:06.186 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.186 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.187 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:44:06.187 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.660330+0000", 2024-02-09T17:44:06.187 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.7305000000000003e-05 2024-02-09T17:44:06.187 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.187 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.187 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:44:06.187 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.660359+0000", 2024-02-09T17:44:06.187 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.8588000000000001e-05 2024-02-09T17:44:06.187 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:06.187 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:06.187 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:06.187 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.187 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.187 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:142479 4.b 4:dba015d4:::benchmark_data_smithi179_103809_object5013:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-02-09T17:44:06.188 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:37:15.435623+0000", 2024-02-09T17:44:06.188 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 410.37929899800002, 2024-02-09T17:44:06.188 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.22484269500000001, 2024-02-09T17:44:06.188 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:44:06.188 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:44:06.188 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:44:06.188 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:44:06.188 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:44:06.188 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 142479 2024-02-09T17:44:06.188 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.188 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:44:06.188 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.188 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:44:06.189 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.435623+0000", 2024-02-09T17:44:06.189 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:06.189 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.189 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.189 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:44:06.189 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.435623+0000", 2024-02-09T17:44:06.189 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:06.189 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.189 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.189 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:44:06.189 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.435625+0000", 2024-02-09T17:44:06.189 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.826e-06 2024-02-09T17:44:06.189 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.189 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.189 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:44:06.190 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.435631+0000", 2024-02-09T17:44:06.190 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.998e-06 2024-02-09T17:44:06.190 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.190 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.190 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:44:06.190 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.435649+0000", 2024-02-09T17:44:06.190 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.7958e-05 2024-02-09T17:44:06.190 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.190 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.190 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:44:06.190 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.435659+0000", 2024-02-09T17:44:06.190 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.7100000000000002e-06 2024-02-09T17:44:06.190 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.190 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.190 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:06.191 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.437577+0000", 2024-02-09T17:44:06.191 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0019182590000000001 2024-02-09T17:44:06.191 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.191 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.191 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:44:06.191 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.437717+0000", 2024-02-09T17:44:06.191 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000139844 2024-02-09T17:44:06.191 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.191 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.191 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:44:06.191 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.437832+0000", 2024-02-09T17:44:06.191 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00011568100000000001 2024-02-09T17:44:06.191 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.191 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.191 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:44:06.192 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.447892+0000", 2024-02-09T17:44:06.192 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.010059462 2024-02-09T17:44:06.192 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.192 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.192 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:06.192 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.450758+0000", 2024-02-09T17:44:06.192 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0028664929999999999 2024-02-09T17:44:06.192 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.192 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.192 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:06.192 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.660407+0000", 2024-02-09T17:44:06.192 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.209648365 2024-02-09T17:44:06.192 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.193 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.193 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:44:06.193 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.660442+0000", 2024-02-09T17:44:06.193 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.5188000000000003e-05 2024-02-09T17:44:06.193 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.193 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.193 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:44:06.193 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.660466+0000", 2024-02-09T17:44:06.193 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.3910999999999999e-05 2024-02-09T17:44:06.193 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:06.193 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:06.193 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:06.193 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.193 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.193 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:142480 4.b 4:dadb69b5:::benchmark_data_smithi179_103809_object5014:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-02-09T17:44:06.193 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:37:15.439167+0000", 2024-02-09T17:44:06.194 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 410.37575528299999, 2024-02-09T17:44:06.194 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.22137198999999999, 2024-02-09T17:44:06.194 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:44:06.194 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:44:06.194 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:44:06.194 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:44:06.194 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:44:06.194 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 142480 2024-02-09T17:44:06.194 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.194 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:44:06.194 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.194 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:44:06.194 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.439167+0000", 2024-02-09T17:44:06.194 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:06.195 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.195 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.195 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:44:06.195 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.439167+0000", 2024-02-09T17:44:06.195 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:06.195 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.195 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.195 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:44:06.195 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.439168+0000", 2024-02-09T17:44:06.195 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.403e-06 2024-02-09T17:44:06.195 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.195 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.195 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:44:06.195 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.439173+0000", 2024-02-09T17:44:06.196 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.8389999999999997e-06 2024-02-09T17:44:06.196 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.196 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.196 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:44:06.196 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.439185+0000", 2024-02-09T17:44:06.196 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2191000000000001e-05 2024-02-09T17:44:06.196 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.196 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.196 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:44:06.196 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.439194+0000", 2024-02-09T17:44:06.196 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.4970000000000001e-06 2024-02-09T17:44:06.196 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.196 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.196 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:06.197 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.441558+0000", 2024-02-09T17:44:06.197 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0023641280000000001 2024-02-09T17:44:06.197 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.197 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.197 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:44:06.197 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.441695+0000", 2024-02-09T17:44:06.197 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000136911 2024-02-09T17:44:06.197 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.197 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.197 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:44:06.197 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.441814+0000", 2024-02-09T17:44:06.197 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000119448 2024-02-09T17:44:06.197 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.197 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.198 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:44:06.198 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.449253+0000", 2024-02-09T17:44:06.198 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0074392319999999996 2024-02-09T17:44:06.198 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.198 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.198 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:06.198 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.463169+0000", 2024-02-09T17:44:06.198 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.013915195 2024-02-09T17:44:06.198 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.198 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.198 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:06.198 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.660491+0000", 2024-02-09T17:44:06.198 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.19732286199999999 2024-02-09T17:44:06.198 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.199 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.199 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:44:06.199 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.660519+0000", 2024-02-09T17:44:06.199 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.7861e-05 2024-02-09T17:44:06.199 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.199 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.199 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:44:06.199 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.660539+0000", 2024-02-09T17:44:06.199 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.9423e-05 2024-02-09T17:44:06.199 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:06.199 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:06.199 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:06.199 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.199 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.199 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:142502 4.b 4:d9f571e1:::benchmark_data_smithi179_103809_object5036:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-02-09T17:44:06.200 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:37:15.462075+0000", 2024-02-09T17:44:06.200 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 410.35284715900002, 2024-02-09T17:44:06.200 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.21471936999999999, 2024-02-09T17:44:06.200 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:44:06.200 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:44:06.200 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:44:06.200 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:44:06.200 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:44:06.200 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 142502 2024-02-09T17:44:06.200 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.200 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:44:06.200 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.200 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:44:06.201 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.462075+0000", 2024-02-09T17:44:06.201 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:06.201 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.201 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.201 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:44:06.201 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.462075+0000", 2024-02-09T17:44:06.201 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:06.201 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.201 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.201 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:44:06.201 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.462077+0000", 2024-02-09T17:44:06.201 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.6729999999999999e-06 2024-02-09T17:44:06.201 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.201 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.202 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:44:06.202 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.462082+0000", 2024-02-09T17:44:06.202 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.8749999999999997e-06 2024-02-09T17:44:06.202 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.202 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.202 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:44:06.202 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.462097+0000", 2024-02-09T17:44:06.202 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.437e-05 2024-02-09T17:44:06.202 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.202 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.202 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:44:06.202 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.462106+0000", 2024-02-09T17:44:06.202 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.9800000000000004e-06 2024-02-09T17:44:06.202 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.203 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.203 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:06.203 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.463204+0000", 2024-02-09T17:44:06.203 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0010978069999999999 2024-02-09T17:44:06.203 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.203 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.203 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:44:06.203 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.463352+0000", 2024-02-09T17:44:06.203 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014882099999999999 2024-02-09T17:44:06.203 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.203 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.203 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:44:06.203 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.463477+0000", 2024-02-09T17:44:06.203 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012462500000000001 2024-02-09T17:44:06.204 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.204 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.204 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:44:06.204 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.467898+0000", 2024-02-09T17:44:06.204 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0044207159999999999 2024-02-09T17:44:06.204 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.204 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.204 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:06.204 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.473913+0000", 2024-02-09T17:44:06.204 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0060150890000000004 2024-02-09T17:44:06.205 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.205 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.205 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:06.205 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.676693+0000", 2024-02-09T17:44:06.205 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.202780556 2024-02-09T17:44:06.205 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.205 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.205 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:44:06.205 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.676750+0000", 2024-02-09T17:44:06.205 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7012000000000002e-05 2024-02-09T17:44:06.205 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.205 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.205 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:44:06.205 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.676794+0000", 2024-02-09T17:44:06.206 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.3846e-05 2024-02-09T17:44:06.206 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:06.206 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:06.206 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:06.206 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.206 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.206 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:142521 4.b 4:db739e59:::benchmark_data_smithi179_103809_object5055:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-02-09T17:44:06.206 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:37:15.480646+0000", 2024-02-09T17:44:06.206 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 410.33427618799999, 2024-02-09T17:44:06.206 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.200893504, 2024-02-09T17:44:06.206 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:44:06.206 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:44:06.206 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:44:06.206 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:44:06.207 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:44:06.207 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 142521 2024-02-09T17:44:06.207 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.207 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:44:06.207 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.207 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:44:06.207 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.480646+0000", 2024-02-09T17:44:06.207 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:06.207 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.207 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.207 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:44:06.207 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.480646+0000", 2024-02-09T17:44:06.208 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:06.208 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.208 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.208 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:44:06.208 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.480647+0000", 2024-02-09T17:44:06.208 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.607e-06 2024-02-09T17:44:06.208 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.208 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.208 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:44:06.208 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.480655+0000", 2024-02-09T17:44:06.208 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.0260000000000007e-06 2024-02-09T17:44:06.208 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.208 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.209 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:44:06.209 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.480672+0000", 2024-02-09T17:44:06.209 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.6790999999999999e-05 2024-02-09T17:44:06.209 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.209 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.209 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:44:06.209 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.480683+0000", 2024-02-09T17:44:06.209 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.062e-05 2024-02-09T17:44:06.209 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.209 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.209 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:06.209 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.480744+0000", 2024-02-09T17:44:06.209 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.1344000000000001e-05 2024-02-09T17:44:06.210 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.210 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.210 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:44:06.210 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.480895+0000", 2024-02-09T17:44:06.210 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00015104100000000001 2024-02-09T17:44:06.210 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.210 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.210 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:44:06.210 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.481014+0000", 2024-02-09T17:44:06.210 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00011828999999999999 2024-02-09T17:44:06.210 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.210 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.210 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:44:06.211 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.484952+0000", 2024-02-09T17:44:06.211 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0039388050000000001 2024-02-09T17:44:06.211 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.211 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.211 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:06.211 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.488401+0000", 2024-02-09T17:44:06.211 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0034482369999999998 2024-02-09T17:44:06.211 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.211 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.211 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:06.211 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.681462+0000", 2024-02-09T17:44:06.211 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.19306173500000001 2024-02-09T17:44:06.211 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.211 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.212 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:44:06.212 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.681498+0000", 2024-02-09T17:44:06.212 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.5176999999999999e-05 2024-02-09T17:44:06.212 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.212 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.212 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:44:06.212 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.681539+0000", 2024-02-09T17:44:06.212 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.1831e-05 2024-02-09T17:44:06.212 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:06.212 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:06.212 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:06.212 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.212 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.213 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:142536 4.b 4:d405018c:::benchmark_data_smithi179_103809_object5070:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-02-09T17:44:06.213 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:37:15.494013+0000", 2024-02-09T17:44:06.213 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 410.32090861900002, 2024-02-09T17:44:06.213 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.18767062100000001, 2024-02-09T17:44:06.213 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:44:06.213 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:44:06.213 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:44:06.213 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:44:06.213 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:44:06.213 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 142536 2024-02-09T17:44:06.213 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.213 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:44:06.213 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.213 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:44:06.214 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.494013+0000", 2024-02-09T17:44:06.214 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:06.214 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.214 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.214 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:44:06.214 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.494013+0000", 2024-02-09T17:44:06.214 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:06.214 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.214 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.214 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:44:06.214 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.494015+0000", 2024-02-09T17:44:06.214 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.668e-06 2024-02-09T17:44:06.214 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.214 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.215 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:44:06.215 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.494020+0000", 2024-02-09T17:44:06.215 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1100000000000002e-06 2024-02-09T17:44:06.215 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.215 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.215 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:44:06.215 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.494030+0000", 2024-02-09T17:44:06.215 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.6090000000000007e-06 2024-02-09T17:44:06.215 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.215 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.215 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:44:06.215 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.494036+0000", 2024-02-09T17:44:06.215 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7529999999999998e-06 2024-02-09T17:44:06.215 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.216 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.216 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:06.216 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.494089+0000", 2024-02-09T17:44:06.216 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3791e-05 2024-02-09T17:44:06.216 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.216 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.216 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:44:06.216 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.494201+0000", 2024-02-09T17:44:06.216 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000111827 2024-02-09T17:44:06.216 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.216 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.216 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:44:06.216 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.494286+0000", 2024-02-09T17:44:06.216 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.5203000000000004e-05 2024-02-09T17:44:06.217 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.217 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.217 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:06.217 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.502425+0000", 2024-02-09T17:44:06.217 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0081390779999999992 2024-02-09T17:44:06.217 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.217 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.217 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:44:06.217 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.505478+0000", 2024-02-09T17:44:06.217 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0030527060000000001 2024-02-09T17:44:06.217 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.217 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.217 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:06.217 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.681586+0000", 2024-02-09T17:44:06.218 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.17610826600000001 2024-02-09T17:44:06.218 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.218 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.218 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:44:06.218 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.681643+0000", 2024-02-09T17:44:06.218 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6558999999999998e-05 2024-02-09T17:44:06.218 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.218 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.218 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:44:06.218 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.681684+0000", 2024-02-09T17:44:06.218 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.1050999999999997e-05 2024-02-09T17:44:06.218 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:06.218 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:06.218 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:06.218 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.219 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.219 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:142560 4.b 4:d2b72131:::benchmark_data_smithi179_103809_object5094:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-02-09T17:44:06.219 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:37:15.523291+0000", 2024-02-09T17:44:06.219 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 410.291630785, 2024-02-09T17:44:06.219 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.165307342, 2024-02-09T17:44:06.219 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:44:06.219 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:44:06.219 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:44:06.219 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:44:06.219 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:44:06.219 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 142560 2024-02-09T17:44:06.219 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.219 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:44:06.219 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.219 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:44:06.220 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.523291+0000", 2024-02-09T17:44:06.220 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:06.220 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.220 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.220 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:44:06.220 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.523291+0000", 2024-02-09T17:44:06.220 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:06.220 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.220 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.220 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:44:06.220 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.523293+0000", 2024-02-09T17:44:06.220 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.6139999999999999e-06 2024-02-09T17:44:06.220 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.220 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.221 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:44:06.221 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.523298+0000", 2024-02-09T17:44:06.221 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2780000000000001e-06 2024-02-09T17:44:06.221 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.221 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.221 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:44:06.221 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.523312+0000", 2024-02-09T17:44:06.221 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3987e-05 2024-02-09T17:44:06.221 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.221 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.221 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:44:06.221 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.523321+0000", 2024-02-09T17:44:06.221 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.6449999999999994e-06 2024-02-09T17:44:06.221 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.222 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.222 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:06.222 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.523383+0000", 2024-02-09T17:44:06.222 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.1929999999999998e-05 2024-02-09T17:44:06.222 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.222 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.222 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:44:06.222 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.523527+0000", 2024-02-09T17:44:06.222 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000143924 2024-02-09T17:44:06.222 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.222 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.222 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:44:06.222 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.523649+0000", 2024-02-09T17:44:06.222 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012264599999999999 2024-02-09T17:44:06.223 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.223 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.223 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:44:06.223 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.531462+0000", 2024-02-09T17:44:06.223 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0078131569999999994 2024-02-09T17:44:06.223 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.223 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.223 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:06.223 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.532213+0000", 2024-02-09T17:44:06.223 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00075038000000000004 2024-02-09T17:44:06.223 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.223 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.223 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:06.223 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.688507+0000", 2024-02-09T17:44:06.223 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.156293984 2024-02-09T17:44:06.224 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.224 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.224 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:44:06.224 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.688559+0000", 2024-02-09T17:44:06.224 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2525999999999998e-05 2024-02-09T17:44:06.224 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.224 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.224 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:44:06.224 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.688599+0000", 2024-02-09T17:44:06.224 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9270999999999997e-05 2024-02-09T17:44:06.224 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:06.224 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:06.224 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:06.225 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.225 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.225 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:142574 4.b 4:d276a6a8:::benchmark_data_smithi179_103809_object5108:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-02-09T17:44:06.225 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:37:15.537260+0000", 2024-02-09T17:44:06.225 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 410.27766164000002, 2024-02-09T17:44:06.225 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.15147265600000001, 2024-02-09T17:44:06.225 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:44:06.225 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:44:06.225 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:44:06.225 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:44:06.225 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:44:06.225 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 142574 2024-02-09T17:44:06.225 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.225 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:44:06.226 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.226 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:44:06.226 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.537260+0000", 2024-02-09T17:44:06.226 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:06.226 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.226 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.226 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:44:06.226 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.537260+0000", 2024-02-09T17:44:06.226 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:06.226 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.226 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.226 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:44:06.226 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.537262+0000", 2024-02-09T17:44:06.226 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4479999999999999e-06 2024-02-09T17:44:06.227 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.227 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.227 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:44:06.227 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.537266+0000", 2024-02-09T17:44:06.227 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.5199999999999999e-06 2024-02-09T17:44:06.227 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.227 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.227 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:44:06.227 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.537281+0000", 2024-02-09T17:44:06.227 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4245e-05 2024-02-09T17:44:06.227 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.227 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.227 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:44:06.227 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.537289+0000", 2024-02-09T17:44:06.227 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.1689999999999992e-06 2024-02-09T17:44:06.228 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.228 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.228 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:06.228 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.537347+0000", 2024-02-09T17:44:06.228 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.8223000000000001e-05 2024-02-09T17:44:06.228 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.228 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.228 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:44:06.228 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.537477+0000", 2024-02-09T17:44:06.228 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000130479 2024-02-09T17:44:06.228 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.228 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.228 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:44:06.229 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.537590+0000", 2024-02-09T17:44:06.229 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000112932 2024-02-09T17:44:06.229 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.229 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.229 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:44:06.229 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.547075+0000", 2024-02-09T17:44:06.229 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0094847660000000004 2024-02-09T17:44:06.229 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.229 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.229 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:06.229 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.550800+0000", 2024-02-09T17:44:06.229 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.003724879 2024-02-09T17:44:06.229 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.229 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.229 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:06.230 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.688643+0000", 2024-02-09T17:44:06.230 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.13784332099999999 2024-02-09T17:44:06.230 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.230 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.230 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:44:06.230 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.688696+0000", 2024-02-09T17:44:06.230 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2500000000000002e-05 2024-02-09T17:44:06.230 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.230 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.230 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:44:06.230 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.688733+0000", 2024-02-09T17:44:06.230 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.7174e-05 2024-02-09T17:44:06.230 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:06.230 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:06.230 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:06.231 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.231 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.231 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:142579 4.b 4:da85fca5:::benchmark_data_smithi179_103809_object5113:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-02-09T17:44:06.231 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:37:15.542523+0000", 2024-02-09T17:44:06.231 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 410.27239891599999, 2024-02-09T17:44:06.231 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.15209349699999999, 2024-02-09T17:44:06.231 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:44:06.231 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:44:06.231 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:44:06.231 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:44:06.231 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:44:06.231 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 142579 2024-02-09T17:44:06.231 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.231 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:44:06.232 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.232 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:44:06.232 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.542523+0000", 2024-02-09T17:44:06.232 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:06.232 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.232 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.232 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:44:06.232 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.542523+0000", 2024-02-09T17:44:06.232 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:06.232 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.232 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.232 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:44:06.232 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.542525+0000", 2024-02-09T17:44:06.232 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.643e-06 2024-02-09T17:44:06.233 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.233 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.233 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:44:06.233 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.542529+0000", 2024-02-09T17:44:06.233 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2969999999999997e-06 2024-02-09T17:44:06.233 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.233 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.233 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:44:06.233 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.542544+0000", 2024-02-09T17:44:06.233 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4474999999999999e-05 2024-02-09T17:44:06.233 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.233 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.233 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:44:06.233 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.542552+0000", 2024-02-09T17:44:06.233 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.7320000000000004e-06 2024-02-09T17:44:06.234 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.234 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.234 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:06.234 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.542610+0000", 2024-02-09T17:44:06.234 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7713999999999997e-05 2024-02-09T17:44:06.234 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.234 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.234 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:44:06.234 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.542736+0000", 2024-02-09T17:44:06.234 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012584099999999999 2024-02-09T17:44:06.234 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.234 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.234 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:44:06.234 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.542850+0000", 2024-02-09T17:44:06.235 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000113986 2024-02-09T17:44:06.235 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.235 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.235 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:44:06.235 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.547232+0000", 2024-02-09T17:44:06.235 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0043821950000000002 2024-02-09T17:44:06.235 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.235 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.235 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:06.235 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.550927+0000", 2024-02-09T17:44:06.235 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.003695411 2024-02-09T17:44:06.235 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.235 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.235 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:06.236 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.694523+0000", 2024-02-09T17:44:06.236 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.143595955 2024-02-09T17:44:06.236 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.236 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.236 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:44:06.236 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.694577+0000", 2024-02-09T17:44:06.236 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3545000000000003e-05 2024-02-09T17:44:06.236 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.236 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.236 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:44:06.236 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.694617+0000", 2024-02-09T17:44:06.236 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9703000000000003e-05 2024-02-09T17:44:06.236 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:06.236 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:06.237 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:06.237 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.237 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.237 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:142585 4.b 4:d9b09195:::benchmark_data_smithi179_103809_object5119:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-02-09T17:44:06.237 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:37:15.551256+0000", 2024-02-09T17:44:06.237 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 410.26366605800001, 2024-02-09T17:44:06.237 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.143511992, 2024-02-09T17:44:06.237 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:44:06.237 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:44:06.237 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:44:06.237 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:44:06.237 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:44:06.237 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 142585 2024-02-09T17:44:06.237 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.238 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:44:06.238 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.238 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:44:06.238 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.551256+0000", 2024-02-09T17:44:06.238 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:06.238 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.238 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.238 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:44:06.238 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.551256+0000", 2024-02-09T17:44:06.238 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:06.238 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.238 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.238 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:44:06.238 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.551257+0000", 2024-02-09T17:44:06.238 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.11e-06 2024-02-09T17:44:06.239 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.239 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.239 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:44:06.239 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.551261+0000", 2024-02-09T17:44:06.239 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.4910000000000001e-06 2024-02-09T17:44:06.239 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.239 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.239 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:44:06.239 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.551273+0000", 2024-02-09T17:44:06.239 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2717999999999999e-05 2024-02-09T17:44:06.239 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.239 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.239 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:44:06.239 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.551281+0000", 2024-02-09T17:44:06.240 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.8110000000000003e-06 2024-02-09T17:44:06.240 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.240 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.240 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:06.240 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.551325+0000", 2024-02-09T17:44:06.240 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.4338999999999997e-05 2024-02-09T17:44:06.240 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.240 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.240 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:44:06.240 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.551439+0000", 2024-02-09T17:44:06.240 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00011325299999999999 2024-02-09T17:44:06.240 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.240 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.240 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:44:06.241 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.551536+0000", 2024-02-09T17:44:06.241 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.7312000000000006e-05 2024-02-09T17:44:06.241 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.241 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.241 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:06.241 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.555946+0000", 2024-02-09T17:44:06.241 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0044101080000000003 2024-02-09T17:44:06.241 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.241 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.241 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:44:06.241 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.556017+0000", 2024-02-09T17:44:06.241 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.1048000000000001e-05 2024-02-09T17:44:06.241 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.241 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.242 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:06.242 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.694675+0000", 2024-02-09T17:44:06.242 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.13865815100000001 2024-02-09T17:44:06.242 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.242 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.242 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:44:06.242 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.694730+0000", 2024-02-09T17:44:06.242 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.4379000000000002e-05 2024-02-09T17:44:06.242 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.242 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.242 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:44:06.242 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:37:15.694768+0000", 2024-02-09T17:44:06.242 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.8272000000000002e-05 2024-02-09T17:44:06.242 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:06.242 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:06.242 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:06.243 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.243 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.243 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:229736 4.b 4:d699b73b:::benchmark_data_smithi179_103809_object92270:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e409)", 2024-02-09T17:44:06.243 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:39:17.539047+0000", 2024-02-09T17:44:06.243 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 288.27587468899998, 2024-02-09T17:44:06.243 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.25139220699999998, 2024-02-09T17:44:06.243 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:44:06.243 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:44:06.243 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:44:06.243 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:44:06.243 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:44:06.243 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 229736 2024-02-09T17:44:06.243 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.243 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:44:06.244 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.244 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:44:06.244 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.539047+0000", 2024-02-09T17:44:06.244 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:06.244 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.244 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.244 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:44:06.244 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.539047+0000", 2024-02-09T17:44:06.244 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:06.244 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.244 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.244 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:44:06.244 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.539048+0000", 2024-02-09T17:44:06.244 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.4299999999999998e-07 2024-02-09T17:44:06.244 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.245 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.245 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:44:06.245 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.539051+0000", 2024-02-09T17:44:06.245 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.7499999999999999e-06 2024-02-09T17:44:06.245 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.245 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.245 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:44:06.245 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.539058+0000", 2024-02-09T17:44:06.245 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.4390000000000003e-06 2024-02-09T17:44:06.245 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.245 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.245 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:44:06.245 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.539063+0000", 2024-02-09T17:44:06.245 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.4770000000000002e-06 2024-02-09T17:44:06.245 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.245 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.246 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:06.246 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.554843+0000", 2024-02-09T17:44:06.246 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.015779313999999999 2024-02-09T17:44:06.246 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.246 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.246 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:44:06.246 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.554981+0000", 2024-02-09T17:44:06.246 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013780699999999999 2024-02-09T17:44:06.246 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.246 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.246 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:44:06.246 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.555101+0000", 2024-02-09T17:44:06.246 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012060799999999999 2024-02-09T17:44:06.246 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.246 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.246 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:44:06.247 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.562887+0000", 2024-02-09T17:44:06.247 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0077858459999999999 2024-02-09T17:44:06.247 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.247 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.247 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:06.247 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.565544+0000", 2024-02-09T17:44:06.247 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0026572309999999999 2024-02-09T17:44:06.247 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.247 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.247 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:06.247 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.790321+0000", 2024-02-09T17:44:06.247 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.224776637 2024-02-09T17:44:06.247 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.247 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.247 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:44:06.248 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.790393+0000", 2024-02-09T17:44:06.248 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.2350999999999999e-05 2024-02-09T17:44:06.248 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.248 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.248 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:44:06.248 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.790440+0000", 2024-02-09T17:44:06.248 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.6304000000000003e-05 2024-02-09T17:44:06.248 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:06.248 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:06.248 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:06.248 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.248 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.248 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:229764 4.b 4:dd3428d9:::benchmark_data_smithi179_103809_object92298:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e409)", 2024-02-09T17:44:06.249 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:39:17.585994+0000", 2024-02-09T17:44:06.249 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 288.22892810600001, 2024-02-09T17:44:06.249 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.204602479, 2024-02-09T17:44:06.249 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:44:06.249 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:44:06.249 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:44:06.249 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:44:06.249 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:44:06.249 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 229764 2024-02-09T17:44:06.249 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.249 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:44:06.249 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.249 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:44:06.249 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.585994+0000", 2024-02-09T17:44:06.250 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:06.250 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.250 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.250 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:44:06.250 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.585994+0000", 2024-02-09T17:44:06.250 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:06.250 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.250 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.250 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:44:06.250 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.585996+0000", 2024-02-09T17:44:06.250 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.702e-06 2024-02-09T17:44:06.250 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.250 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.250 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:44:06.250 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.586002+0000", 2024-02-09T17:44:06.251 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.4659999999999996e-06 2024-02-09T17:44:06.251 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.251 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.251 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:44:06.251 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.586017+0000", 2024-02-09T17:44:06.251 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4928e-05 2024-02-09T17:44:06.251 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.251 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.251 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:44:06.251 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.586025+0000", 2024-02-09T17:44:06.251 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.96e-06 2024-02-09T17:44:06.251 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.251 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.251 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:06.252 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.586077+0000", 2024-02-09T17:44:06.252 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1788e-05 2024-02-09T17:44:06.252 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.252 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.252 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:44:06.252 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.586206+0000", 2024-02-09T17:44:06.252 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012875899999999999 2024-02-09T17:44:06.252 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.252 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.252 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:44:06.252 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.586324+0000", 2024-02-09T17:44:06.252 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000118486 2024-02-09T17:44:06.252 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.252 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.253 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:44:06.253 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.591097+0000", 2024-02-09T17:44:06.253 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0047726829999999998 2024-02-09T17:44:06.253 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.253 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.253 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:06.253 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.591679+0000", 2024-02-09T17:44:06.253 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000582747 2024-02-09T17:44:06.253 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.253 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.253 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:06.253 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.790499+0000", 2024-02-09T17:44:06.253 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.19881927199999999 2024-02-09T17:44:06.253 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.253 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.254 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:44:06.254 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.790557+0000", 2024-02-09T17:44:06.254 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.8235000000000001e-05 2024-02-09T17:44:06.254 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.254 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.254 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:44:06.254 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.790596+0000", 2024-02-09T17:44:06.254 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9453000000000003e-05 2024-02-09T17:44:06.254 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:06.254 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:06.254 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:06.254 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.254 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.254 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:229767 4.b 4:d03aef11:::benchmark_data_smithi179_103809_object92301:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e409)", 2024-02-09T17:44:06.255 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:39:17.593089+0000", 2024-02-09T17:44:06.255 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 288.22183266899998, 2024-02-09T17:44:06.255 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.19766324599999999, 2024-02-09T17:44:06.255 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:44:06.255 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:44:06.255 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:44:06.255 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:44:06.255 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:44:06.255 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 229767 2024-02-09T17:44:06.255 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.255 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:44:06.255 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.255 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:44:06.255 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.593089+0000", 2024-02-09T17:44:06.256 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:06.256 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.256 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.256 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:44:06.256 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.593089+0000", 2024-02-09T17:44:06.256 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:06.256 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.256 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.256 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:44:06.256 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.593091+0000", 2024-02-09T17:44:06.256 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.412e-06 2024-02-09T17:44:06.256 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.256 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.257 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:44:06.257 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.593095+0000", 2024-02-09T17:44:06.257 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.386e-06 2024-02-09T17:44:06.257 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.257 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.257 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:44:06.257 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.593107+0000", 2024-02-09T17:44:06.257 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.15e-05 2024-02-09T17:44:06.257 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.257 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.257 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:44:06.257 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.593113+0000", 2024-02-09T17:44:06.257 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.7599999999999997e-06 2024-02-09T17:44:06.257 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.257 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.258 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:06.258 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.593167+0000", 2024-02-09T17:44:06.258 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3773999999999997e-05 2024-02-09T17:44:06.258 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.258 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.258 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:44:06.258 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.593287+0000", 2024-02-09T17:44:06.258 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000119323 2024-02-09T17:44:06.258 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.258 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.258 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:44:06.258 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.593364+0000", 2024-02-09T17:44:06.258 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.7312999999999997e-05 2024-02-09T17:44:06.258 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.258 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.258 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:44:06.258 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.597528+0000", 2024-02-09T17:44:06.259 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00416447 2024-02-09T17:44:06.259 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.259 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.259 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:06.259 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.603409+0000", 2024-02-09T17:44:06.259 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0058809559999999997 2024-02-09T17:44:06.259 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.259 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.259 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:06.259 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.790660+0000", 2024-02-09T17:44:06.259 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.18725065199999999 2024-02-09T17:44:06.259 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.259 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.259 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:44:06.259 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.790714+0000", 2024-02-09T17:44:06.260 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.4014000000000002e-05 2024-02-09T17:44:06.260 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.260 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.260 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:44:06.260 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.790753+0000", 2024-02-09T17:44:06.260 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.8686000000000002e-05 2024-02-09T17:44:06.260 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:06.260 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:06.260 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:06.260 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.260 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.260 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:229801 4.b 4:db62d972:::benchmark_data_smithi179_103809_object92335:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e409)", 2024-02-09T17:44:06.260 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:39:17.651766+0000", 2024-02-09T17:44:06.260 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 288.163155985, 2024-02-09T17:44:06.261 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.13929277200000001, 2024-02-09T17:44:06.261 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:44:06.261 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:44:06.261 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:44:06.261 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:44:06.261 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:44:06.261 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 229801 2024-02-09T17:44:06.261 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.261 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:44:06.261 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.261 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:44:06.261 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.651766+0000", 2024-02-09T17:44:06.261 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:06.261 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.261 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.261 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:44:06.262 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.651766+0000", 2024-02-09T17:44:06.262 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:06.262 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.262 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.262 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:44:06.262 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.651767+0000", 2024-02-09T17:44:06.262 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.4600000000000004e-07 2024-02-09T17:44:06.262 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.262 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.262 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:44:06.262 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.651769+0000", 2024-02-09T17:44:06.262 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.5289999999999998e-06 2024-02-09T17:44:06.262 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.262 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.263 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:44:06.263 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.651775+0000", 2024-02-09T17:44:06.263 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6359999999999999e-06 2024-02-09T17:44:06.263 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.263 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.263 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:44:06.263 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.651778+0000", 2024-02-09T17:44:06.263 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.4649999999999999e-06 2024-02-09T17:44:06.263 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.263 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.263 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:06.263 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.656787+0000", 2024-02-09T17:44:06.263 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0050083769999999996 2024-02-09T17:44:06.263 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.264 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.264 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:44:06.264 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.656862+0000", 2024-02-09T17:44:06.264 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.5763e-05 2024-02-09T17:44:06.264 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.264 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.264 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:44:06.264 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.656973+0000", 2024-02-09T17:44:06.264 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00011006300000000001 2024-02-09T17:44:06.264 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.264 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.264 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:44:06.264 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.661536+0000", 2024-02-09T17:44:06.264 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0045634999999999998 2024-02-09T17:44:06.265 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.265 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.265 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:06.265 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.661879+0000", 2024-02-09T17:44:06.265 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00034313399999999998 2024-02-09T17:44:06.265 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.265 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.265 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:06.265 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.790972+0000", 2024-02-09T17:44:06.265 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.12909321000000001 2024-02-09T17:44:06.265 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.265 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.265 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:44:06.265 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.791026+0000", 2024-02-09T17:44:06.266 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.4128000000000001e-05 2024-02-09T17:44:06.266 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.266 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.266 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:44:06.266 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:17.791059+0000", 2024-02-09T17:44:06.266 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.2320999999999998e-05 2024-02-09T17:44:06.266 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:06.266 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:06.266 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:06.266 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.266 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.266 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:232553 4.7 4:e6318b68:::benchmark_data_smithi179_103809_object95087:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e415)", 2024-02-09T17:44:06.266 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:39:24.529349+0000", 2024-02-09T17:44:06.266 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 281.28557310000002, 2024-02-09T17:44:06.266 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.259789582, 2024-02-09T17:44:06.267 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:44:06.267 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:44:06.267 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:44:06.267 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:44:06.267 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:44:06.267 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 232553 2024-02-09T17:44:06.267 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.267 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:44:06.267 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.267 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:44:06.267 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529349+0000", 2024-02-09T17:44:06.267 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:06.267 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.267 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.268 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:44:06.268 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529349+0000", 2024-02-09T17:44:06.268 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:06.268 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.268 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.268 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:44:06.268 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529351+0000", 2024-02-09T17:44:06.268 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.9400000000000001e-06 2024-02-09T17:44:06.268 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.268 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.268 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:44:06.268 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529360+0000", 2024-02-09T17:44:06.268 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.6640000000000004e-06 2024-02-09T17:44:06.268 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.269 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.269 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:44:06.269 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529381+0000", 2024-02-09T17:44:06.269 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.1449000000000001e-05 2024-02-09T17:44:06.269 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.269 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.269 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:44:06.269 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529404+0000", 2024-02-09T17:44:06.269 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.3328000000000001e-05 2024-02-09T17:44:06.269 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.269 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.269 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:06.269 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529502+0000", 2024-02-09T17:44:06.269 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.7256999999999994e-05 2024-02-09T17:44:06.270 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.270 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.270 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op must wait for map", 2024-02-09T17:44:06.270 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529509+0000", 2024-02-09T17:44:06.270 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.7610000000000007e-06 2024-02-09T17:44:06.270 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.270 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.270 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:06.270 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.531763+0000", 2024-02-09T17:44:06.270 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0022537740000000001 2024-02-09T17:44:06.270 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.270 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.270 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:44:06.270 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.531774+0000", 2024-02-09T17:44:06.270 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0967e-05 2024-02-09T17:44:06.271 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.271 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.271 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:06.271 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.773951+0000", 2024-02-09T17:44:06.271 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2421772010000001 2024-02-09T17:44:06.271 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.271 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.271 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:44:06.271 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.777956+0000", 2024-02-09T17:44:06.271 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0040046150000000004 2024-02-09T17:44:06.271 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.271 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.271 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,7,10", 2024-02-09T17:44:06.271 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.778105+0000", 2024-02-09T17:44:06.272 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014957499999999999 2024-02-09T17:44:06.272 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.272 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.272 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:06.272 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.783408+0000", 2024-02-09T17:44:06.272 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.005302106 2024-02-09T17:44:06.272 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.272 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.272 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:44:06.272 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.783425+0000", 2024-02-09T17:44:06.272 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.7003e-05 2024-02-09T17:44:06.272 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.273 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.273 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:06.273 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.783488+0000", 2024-02-09T17:44:06.273 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.3232999999999996e-05 2024-02-09T17:44:06.273 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.273 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.273 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:06.273 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.789011+0000", 2024-02-09T17:44:06.273 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.005523315 2024-02-09T17:44:06.273 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.273 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.273 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:44:06.273 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.789066+0000", 2024-02-09T17:44:06.273 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.4656e-05 2024-02-09T17:44:06.274 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.274 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.274 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:44:06.274 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.789139+0000", 2024-02-09T17:44:06.274 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.2737999999999994e-05 2024-02-09T17:44:06.274 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:06.274 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:06.274 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:06.274 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.274 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.274 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:232579 4.7 4:eafda101:::benchmark_data_smithi179_103809_object95113:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e415)", 2024-02-09T17:44:06.274 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:39:24.529431+0000", 2024-02-09T17:44:06.274 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 281.285491337, 2024-02-09T17:44:06.274 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2695841749999999, 2024-02-09T17:44:06.274 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:44:06.275 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:44:06.275 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:44:06.275 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:44:06.275 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:44:06.275 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 232579 2024-02-09T17:44:06.275 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.275 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:44:06.275 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.275 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:44:06.275 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529431+0000", 2024-02-09T17:44:06.275 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:06.275 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.275 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.275 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:44:06.276 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529431+0000", 2024-02-09T17:44:06.276 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:06.276 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.276 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.276 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:44:06.276 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529431+0000", 2024-02-09T17:44:06.276 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6400000000000002e-07 2024-02-09T17:44:06.276 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.276 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.276 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:44:06.276 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529435+0000", 2024-02-09T17:44:06.276 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.3739999999999998e-06 2024-02-09T17:44:06.276 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.277 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.277 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:44:06.277 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529443+0000", 2024-02-09T17:44:06.277 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.4279999999999996e-06 2024-02-09T17:44:06.277 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.277 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.277 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:44:06.277 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529450+0000", 2024-02-09T17:44:06.277 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.5400000000000001e-06 2024-02-09T17:44:06.277 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.277 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.277 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:06.277 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529563+0000", 2024-02-09T17:44:06.277 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0001131 2024-02-09T17:44:06.277 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.277 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.278 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting_for_map not empty", 2024-02-09T17:44:06.278 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529568+0000", 2024-02-09T17:44:06.278 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2920000000000003e-06 2024-02-09T17:44:06.278 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.278 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.278 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:06.278 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.532793+0000", 2024-02-09T17:44:06.278 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0032253260000000001 2024-02-09T17:44:06.278 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.278 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.278 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:44:06.278 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.532803+0000", 2024-02-09T17:44:06.278 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.2010000000000005e-06 2024-02-09T17:44:06.278 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.278 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.279 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:06.279 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.783535+0000", 2024-02-09T17:44:06.279 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2507324470000001 2024-02-09T17:44:06.279 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.279 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.279 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:44:06.279 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.783654+0000", 2024-02-09T17:44:06.279 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00011907400000000001 2024-02-09T17:44:06.279 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.279 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.279 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,7,10", 2024-02-09T17:44:06.279 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.783808+0000", 2024-02-09T17:44:06.279 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000153976 2024-02-09T17:44:06.279 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.279 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.280 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:06.280 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.789175+0000", 2024-02-09T17:44:06.280 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0053669319999999996 2024-02-09T17:44:06.280 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.280 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.280 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:44:06.280 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.789183+0000", 2024-02-09T17:44:06.280 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.9359999999999999e-06 2024-02-09T17:44:06.280 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.280 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.280 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:06.280 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.798870+0000", 2024-02-09T17:44:06.280 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0096870970000000004 2024-02-09T17:44:06.281 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.281 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.281 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:06.281 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.798910+0000", 2024-02-09T17:44:06.281 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.0018999999999997e-05 2024-02-09T17:44:06.281 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.281 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.281 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:44:06.281 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.798951+0000", 2024-02-09T17:44:06.281 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.1349e-05 2024-02-09T17:44:06.281 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.281 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.281 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:44:06.281 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.799015+0000", 2024-02-09T17:44:06.281 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.3520000000000002e-05 2024-02-09T17:44:06.282 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:06.282 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:06.282 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:06.282 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.282 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.282 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:232588 4.7 4:e414a3eb:::benchmark_data_smithi179_103809_object95122:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-02-09T17:44:06.282 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:39:24.612490+0000", 2024-02-09T17:44:06.282 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 281.20243153899997, 2024-02-09T17:44:06.282 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.196787614, 2024-02-09T17:44:06.282 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:44:06.282 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:44:06.282 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:44:06.282 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:44:06.282 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:44:06.282 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 232588 2024-02-09T17:44:06.283 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.283 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:44:06.283 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.283 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:44:06.283 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.612490+0000", 2024-02-09T17:44:06.283 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:06.283 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.283 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.283 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:44:06.283 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.612490+0000", 2024-02-09T17:44:06.283 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:06.283 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.283 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.283 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:44:06.283 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.612492+0000", 2024-02-09T17:44:06.284 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.234e-06 2024-02-09T17:44:06.284 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.284 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.284 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:44:06.284 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.612497+0000", 2024-02-09T17:44:06.284 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1959999999999999e-06 2024-02-09T17:44:06.284 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.284 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.284 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:44:06.284 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.612507+0000", 2024-02-09T17:44:06.284 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.04e-05 2024-02-09T17:44:06.284 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.284 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.284 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:44:06.285 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.612513+0000", 2024-02-09T17:44:06.285 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.0340000000000002e-06 2024-02-09T17:44:06.285 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.285 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.285 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:06.285 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.612566+0000", 2024-02-09T17:44:06.285 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2352000000000003e-05 2024-02-09T17:44:06.285 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.285 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.285 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:44:06.285 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.612577+0000", 2024-02-09T17:44:06.285 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0886e-05 2024-02-09T17:44:06.285 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.285 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.285 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:06.286 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.789202+0000", 2024-02-09T17:44:06.286 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1766251320000001 2024-02-09T17:44:06.286 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.286 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.286 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:44:06.286 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.789317+0000", 2024-02-09T17:44:06.286 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000115123 2024-02-09T17:44:06.286 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.286 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.286 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,7,10", 2024-02-09T17:44:06.286 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.789408+0000", 2024-02-09T17:44:06.286 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.1421000000000004e-05 2024-02-09T17:44:06.286 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.286 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.287 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:44:06.287 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.799027+0000", 2024-02-09T17:44:06.287 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0096182330000000003 2024-02-09T17:44:06.287 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.287 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.287 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:06.287 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.799152+0000", 2024-02-09T17:44:06.287 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012501999999999999 2024-02-09T17:44:06.287 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.287 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.287 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:06.287 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.809067+0000", 2024-02-09T17:44:06.287 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.009915119 2024-02-09T17:44:06.287 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.287 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.288 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:06.288 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.809128+0000", 2024-02-09T17:44:06.288 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.1671e-05 2024-02-09T17:44:06.288 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.288 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.288 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:44:06.288 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.809164+0000", 2024-02-09T17:44:06.288 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.5988000000000002e-05 2024-02-09T17:44:06.288 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.288 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.288 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:44:06.288 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.809278+0000", 2024-02-09T17:44:06.288 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000113805 2024-02-09T17:44:06.288 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:06.289 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:06.289 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:06.289 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.289 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.289 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:232589 4.7 4:e37473dd:::benchmark_data_smithi179_103809_object95123:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-02-09T17:44:06.289 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:39:24.619917+0000", 2024-02-09T17:44:06.289 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 281.19500507499998, 2024-02-09T17:44:06.289 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.189582264, 2024-02-09T17:44:06.289 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:44:06.289 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:44:06.289 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:44:06.289 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:44:06.289 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:44:06.289 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 232589 2024-02-09T17:44:06.290 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.290 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:44:06.290 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.290 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:44:06.290 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.619917+0000", 2024-02-09T17:44:06.290 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:06.290 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.290 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.290 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:44:06.290 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.619917+0000", 2024-02-09T17:44:06.290 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:06.290 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.290 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.290 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:44:06.291 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.619919+0000", 2024-02-09T17:44:06.291 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.4099999999999998e-06 2024-02-09T17:44:06.291 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.291 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.291 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:44:06.291 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.619928+0000", 2024-02-09T17:44:06.291 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.3219999999999993e-06 2024-02-09T17:44:06.291 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.291 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.291 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:44:06.291 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.619949+0000", 2024-02-09T17:44:06.291 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.1160000000000001e-05 2024-02-09T17:44:06.291 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.291 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.292 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:44:06.292 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.619960+0000", 2024-02-09T17:44:06.292 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1287000000000001e-05 2024-02-09T17:44:06.292 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.292 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.292 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:06.292 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.620034+0000", 2024-02-09T17:44:06.292 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.4089000000000002e-05 2024-02-09T17:44:06.292 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.292 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.292 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:44:06.292 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.620048+0000", 2024-02-09T17:44:06.292 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3447e-05 2024-02-09T17:44:06.292 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.293 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.293 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:06.293 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.794239+0000", 2024-02-09T17:44:06.293 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.174191516 2024-02-09T17:44:06.293 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.293 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.293 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:44:06.293 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.794361+0000", 2024-02-09T17:44:06.293 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000122256 2024-02-09T17:44:06.293 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.293 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.293 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,7,10", 2024-02-09T17:44:06.293 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.794542+0000", 2024-02-09T17:44:06.293 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00018047899999999999 2024-02-09T17:44:06.294 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.294 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.294 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:44:06.294 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.799034+0000", 2024-02-09T17:44:06.294 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0044917710000000003 2024-02-09T17:44:06.294 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.294 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.294 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:06.294 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.809098+0000", 2024-02-09T17:44:06.294 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.010064619 2024-02-09T17:44:06.294 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.294 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.294 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:06.294 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.809308+0000", 2024-02-09T17:44:06.294 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00020926500000000001 2024-02-09T17:44:06.295 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.295 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.295 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:06.295 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.809355+0000", 2024-02-09T17:44:06.295 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.7741999999999998e-05 2024-02-09T17:44:06.295 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.295 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.295 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:44:06.295 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.809385+0000", 2024-02-09T17:44:06.295 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.9709e-05 2024-02-09T17:44:06.295 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.295 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.295 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:44:06.295 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.809499+0000", 2024-02-09T17:44:06.295 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00011419199999999999 2024-02-09T17:44:06.296 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:06.296 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:06.296 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:06.296 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.296 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.296 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:232593 4.7 4:e255490f:::benchmark_data_smithi179_103809_object95127:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-02-09T17:44:06.296 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:39:25.605375+0000", 2024-02-09T17:44:06.296 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 280.20954752400002, 2024-02-09T17:44:06.296 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.22923427800000001, 2024-02-09T17:44:06.296 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:44:06.296 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:44:06.296 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:44:06.296 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:44:06.296 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:44:06.296 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 232593 2024-02-09T17:44:06.297 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.297 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:44:06.297 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.297 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:44:06.297 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.605375+0000", 2024-02-09T17:44:06.297 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:06.297 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.297 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.297 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:44:06.297 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.605375+0000", 2024-02-09T17:44:06.297 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:06.297 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.297 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.297 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:44:06.298 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.605376+0000", 2024-02-09T17:44:06.298 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.719e-06 2024-02-09T17:44:06.298 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.298 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.298 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:44:06.298 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.605382+0000", 2024-02-09T17:44:06.298 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3149999999999999e-06 2024-02-09T17:44:06.298 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.298 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.298 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:44:06.298 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.605407+0000", 2024-02-09T17:44:06.298 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.4998999999999999e-05 2024-02-09T17:44:06.298 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.298 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.299 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:44:06.299 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.605416+0000", 2024-02-09T17:44:06.299 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.4069999999999999e-06 2024-02-09T17:44:06.299 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.299 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.299 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:06.299 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.606684+0000", 2024-02-09T17:44:06.299 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0012678240000000001 2024-02-09T17:44:06.299 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.299 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.299 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:44:06.299 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.606697+0000", 2024-02-09T17:44:06.299 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2877999999999999e-05 2024-02-09T17:44:06.299 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.299 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.300 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:06.300 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.799193+0000", 2024-02-09T17:44:06.300 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.19249639599999999 2024-02-09T17:44:06.300 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.300 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.300 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:44:06.300 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.799327+0000", 2024-02-09T17:44:06.300 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013401999999999999 2024-02-09T17:44:06.300 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.300 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.300 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,7,10", 2024-02-09T17:44:06.300 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.799498+0000", 2024-02-09T17:44:06.300 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000171137 2024-02-09T17:44:06.300 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.300 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.301 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:44:06.301 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.809075+0000", 2024-02-09T17:44:06.301 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0095770730000000002 2024-02-09T17:44:06.301 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.301 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.301 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:06.301 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.809331+0000", 2024-02-09T17:44:06.301 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00025584099999999998 2024-02-09T17:44:06.301 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.301 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.301 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:06.301 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.820212+0000", 2024-02-09T17:44:06.301 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.010881165 2024-02-09T17:44:06.301 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.301 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.302 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:06.302 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.834245+0000", 2024-02-09T17:44:06.302 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.014032263999999999 2024-02-09T17:44:06.302 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.302 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.302 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:44:06.302 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.834492+0000", 2024-02-09T17:44:06.302 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00024776699999999997 2024-02-09T17:44:06.302 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.302 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.302 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:44:06.302 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.834609+0000", 2024-02-09T17:44:06.302 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00011647299999999999 2024-02-09T17:44:06.302 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:06.303 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:06.303 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:06.303 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.303 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.303 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:232606 4.7 4:eec97097:::benchmark_data_smithi179_103809_object95140:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-02-09T17:44:06.303 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:39:25.642341+0000", 2024-02-09T17:44:06.303 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 280.172580751, 2024-02-09T17:44:06.303 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.19250051600000001, 2024-02-09T17:44:06.303 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:44:06.303 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:44:06.303 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:44:06.303 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:44:06.303 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:44:06.303 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 232606 2024-02-09T17:44:06.304 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.304 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:44:06.304 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.304 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:44:06.304 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.642341+0000", 2024-02-09T17:44:06.304 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:06.304 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.304 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.304 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:44:06.304 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.642341+0000", 2024-02-09T17:44:06.304 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:06.304 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.304 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.304 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:44:06.305 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.642343+0000", 2024-02-09T17:44:06.305 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.863e-06 2024-02-09T17:44:06.305 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.305 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.305 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:44:06.305 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.642350+0000", 2024-02-09T17:44:06.305 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.2860000000000002e-06 2024-02-09T17:44:06.305 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.305 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.305 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:44:06.305 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.642368+0000", 2024-02-09T17:44:06.305 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.7841000000000001e-05 2024-02-09T17:44:06.305 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.305 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.305 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:44:06.306 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.642379+0000", 2024-02-09T17:44:06.306 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1088e-05 2024-02-09T17:44:06.306 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.306 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.306 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:06.306 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.643440+0000", 2024-02-09T17:44:06.306 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.001060213 2024-02-09T17:44:06.306 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.306 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.306 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:44:06.306 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.643451+0000", 2024-02-09T17:44:06.306 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1416999999999999e-05 2024-02-09T17:44:06.306 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.306 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.307 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:06.307 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.804558+0000", 2024-02-09T17:44:06.307 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.16110685799999999 2024-02-09T17:44:06.307 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.307 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.307 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:44:06.307 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.804663+0000", 2024-02-09T17:44:06.307 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000105327 2024-02-09T17:44:06.307 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.307 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.307 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,7,10", 2024-02-09T17:44:06.307 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.804806+0000", 2024-02-09T17:44:06.307 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000143251 2024-02-09T17:44:06.307 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.307 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.308 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:44:06.308 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.820076+0000", 2024-02-09T17:44:06.308 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.015270064999999999 2024-02-09T17:44:06.308 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.308 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.308 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:06.308 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.820143+0000", 2024-02-09T17:44:06.308 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.6710000000000003e-05 2024-02-09T17:44:06.308 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.308 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.308 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:06.308 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.820327+0000", 2024-02-09T17:44:06.308 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00018374099999999999 2024-02-09T17:44:06.308 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.309 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.309 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:06.309 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.834665+0000", 2024-02-09T17:44:06.309 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.014338404000000001 2024-02-09T17:44:06.309 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.309 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.309 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:44:06.309 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.834721+0000", 2024-02-09T17:44:06.309 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.5779999999999998e-05 2024-02-09T17:44:06.309 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:06.309 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:06.309 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:44:06.309 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.834842+0000", 2024-02-09T17:44:06.309 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000120672 2024-02-09T17:44:06.309 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:06.310 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:06.310 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:06.310 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:06.310 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:06.310 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:44:06.310 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:44:06.311 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:05 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:44:06.311 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:05 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:44:06.311 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:05 smithi099 ceph-mon[39420]: pgmap v1361: 47 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 45 active+clean; 3.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 699 op/s; 3809/182274 objects misplaced (2.090%); 527 KiB/s, 9 objects/s recovering 2024-02-09T17:44:06.311 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:05 smithi099 ceph-mon[39420]: osdmap e542: 12 total, 12 up, 12 in 2024-02-09T17:44:06.373 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:06.394 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:44:06.402 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:44:06.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:06.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:44:06.510 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:44:06.572 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:06.617 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:44:06.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:06.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:44:06.726 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:44:06.759 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:06.863 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:44:06.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:06.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:44:06.891 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-09T17:44:06.972 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:44:07.042 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:07.089 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:44:07.131 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:44:07.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:07.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:44:07.236 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:44:07.269 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:07.299 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:44:07.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:07.305 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:44:07.406 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:44:07.454 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:07.507 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:44:07.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:07.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:44:07.613 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:44:07.702 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:07 smithi162 ceph-mon[44304]: osdmap e543: 12 total, 12 up, 12 in 2024-02-09T17:44:07.702 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:07 smithi162 ceph-mon[44304]: pgmap v1364: 47 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 45 active+clean; 3.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 762 op/s; 3809/176607 objects misplaced (2.157%) 2024-02-09T17:44:07.703 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:07.742 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:44:07.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:07 smithi099 ceph-mon[39420]: osdmap e543: 12 total, 12 up, 12 in 2024-02-09T17:44:07.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:07 smithi099 ceph-mon[39420]: pgmap v1364: 47 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 45 active+clean; 3.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 762 op/s; 3809/176607 objects misplaced (2.157%) 2024-02-09T17:44:07.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:07.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:44:07.852 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:44:07.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:07 smithi179 ceph-mon[43975]: osdmap e543: 12 total, 12 up, 12 in 2024-02-09T17:44:07.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:07 smithi179 ceph-mon[43975]: pgmap v1364: 47 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 45 active+clean; 3.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 762 op/s; 3809/176607 objects misplaced (2.157%) 2024-02-09T17:44:07.922 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:07.977 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:44:07.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:07.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:44:08.084 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:44:08.119 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:08.142 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:44:08.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:08.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:44:08.248 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:44:08.286 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:08.314 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:44:08.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:08.319 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:44:08.420 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:44:08.452 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:08.574 INFO:teuthology.orchestra.run.smithi099.stderr:set pool 1 pgp_num to 31 2024-02-09T17:44:08.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:08 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2471822835' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]: dispatch 2024-02-09T17:44:08.931 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:08 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2471822835' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]: dispatch 2024-02-09T17:44:08.976 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:08 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2471822835' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]: dispatch 2024-02-09T17:44:09.498 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:44:09.523 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:44:09.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:09.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:44:09.554 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:44:09.634 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:44:09.678 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:09.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:09 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:44:09.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:09 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:44:09.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:09 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2471822835' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]': finished 2024-02-09T17:44:09.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:09 smithi099 ceph-mon[39420]: osdmap e544: 12 total, 12 up, 12 in 2024-02-09T17:44:09.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:09 smithi099 ceph-mon[39420]: pgmap v1366: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing+deep, 45 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 700 op/s; 3570/173796 objects misplaced (2.054%) 2024-02-09T17:44:09.831 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:44:09.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:09.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:44:09.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:09 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:44:09.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:09 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:44:09.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:09 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2471822835' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]': finished 2024-02-09T17:44:09.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:09 smithi179 ceph-mon[43975]: osdmap e544: 12 total, 12 up, 12 in 2024-02-09T17:44:09.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:09 smithi179 ceph-mon[43975]: pgmap v1366: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing+deep, 45 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 700 op/s; 3570/173796 objects misplaced (2.054%) 2024-02-09T17:44:09.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:09 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:44:09.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:09 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:44:09.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:09 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2471822835' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]': finished 2024-02-09T17:44:09.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:09 smithi162 ceph-mon[44304]: osdmap e544: 12 total, 12 up, 12 in 2024-02-09T17:44:09.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:09 smithi162 ceph-mon[44304]: pgmap v1366: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing+deep, 45 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 700 op/s; 3570/173796 objects misplaced (2.054%) 2024-02-09T17:44:09.935 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:44:09.981 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:10.083 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:44:10.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:10.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:44:10.188 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:44:10.223 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:10.272 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:44:10.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:10.278 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:44:10.379 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:44:10.441 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:10.487 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:44:10.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:10.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:44:10.596 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:44:10.633 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:10.660 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:44:10.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:10.666 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:44:10.767 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:44:10.815 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:10.846 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is set 2024-02-09T17:44:10.855 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:10 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1642255292' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:44:10.855 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:10 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:44:10.856 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:10 smithi162 ceph-mon[44304]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:44:10.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:10 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1642255292' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:44:10.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:10 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:44:10.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:10 smithi179 ceph-mon[43975]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:44:10.974 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:44:10.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:10.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:44:11.084 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:44:11.108 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:10 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1642255292' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:44:11.108 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:10 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:44:11.108 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:10 smithi099 ceph-mon[39420]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:44:11.116 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:11.142 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:44:11.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:11.145 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:44:11.247 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:44:11.308 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:11.343 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:44:11.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:11.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:44:11.448 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:44:11.485 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:11.942 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:44:11.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:11.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:44:12.049 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:44:12.085 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:12.115 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:44:12.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:12.121 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:44:12.167 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:11 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:44:12.167 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:11 smithi099 ceph-mon[39420]: osdmap e545: 12 total, 12 up, 12 in 2024-02-09T17:44:12.167 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:11 smithi099 ceph-mon[39420]: pgmap v1368: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing+deep, 45 active+clean; 3.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 792 op/s; 3570/168171 objects misplaced (2.123%) 2024-02-09T17:44:12.176 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:11 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:44:12.176 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:11 smithi179 ceph-mon[43975]: osdmap e545: 12 total, 12 up, 12 in 2024-02-09T17:44:12.176 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:11 smithi179 ceph-mon[43975]: pgmap v1368: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing+deep, 45 active+clean; 3.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 792 op/s; 3570/168171 objects misplaced (2.123%) 2024-02-09T17:44:12.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:11 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:44:12.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:11 smithi162 ceph-mon[44304]: osdmap e545: 12 total, 12 up, 12 in 2024-02-09T17:44:12.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:11 smithi162 ceph-mon[44304]: pgmap v1368: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing+deep, 45 active+clean; 3.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 792 op/s; 3570/168171 objects misplaced (2.123%) 2024-02-09T17:44:12.222 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:44:12.259 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:13.448 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:44:13.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:13.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:44:13.562 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:44:13.596 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:13.623 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:44:13.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:13.628 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:44:13.729 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:44:13.789 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:13.827 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:44:13.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:13.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:44:13.911 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset noscrub 2024-02-09T17:44:13.933 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:44:13.964 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:13 smithi099 ceph-mon[39420]: pgmap v1369: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing+deep, 45 active+clean; 3.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 750 op/s; 3339/163851 objects misplaced (2.038%) 2024-02-09T17:44:13.967 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:13.993 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:44:14.004 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:14.005 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:44:14.105 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:44:14.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:13 smithi179 ceph-mon[43975]: pgmap v1369: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing+deep, 45 active+clean; 3.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 750 op/s; 3339/163851 objects misplaced (2.038%) 2024-02-09T17:44:14.174 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:14.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:13 smithi162 ceph-mon[44304]: pgmap v1369: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing+deep, 45 active+clean; 3.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 750 op/s; 3339/163851 objects misplaced (2.038%) 2024-02-09T17:44:14.211 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:44:14.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:14.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:44:14.326 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:44:14.361 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:14.384 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:44:14.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:14.391 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:44:14.417 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:44:14.446 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:44:14.472 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:44:14.472 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] 2024-02-09T17:44:14.472 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:44:14.473 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-02-09T17:44:14.473 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-09T17:44:14.491 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:44:14.502 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:44:14.539 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:44:14.539 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [ 2024-02-09T17:44:14.539 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:14.539 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25528.0:206474 5.0 e545/465 5:02ea57d0:::benchmark_data_smithi179_115199_object77568:head v 545'13255, mlcod=545'13252)", 2024-02-09T17:44:14.539 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:44:14.259547+0000", 2024-02-09T17:44:14.539 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.020571044, 2024-02-09T17:44:14.539 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.020601086000000001, 2024-02-09T17:44:14.540 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:44:14.540 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T17:44:14.540 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:44:14.540 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:14.540 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:44:14.540 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:14.259547+0000", 2024-02-09T17:44:14.540 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:14.540 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:14.540 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:14.540 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:44:14.541 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:14.259547+0000", 2024-02-09T17:44:14.541 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:14.541 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:14.541 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:14.541 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:44:14.541 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:14.259548+0000", 2024-02-09T17:44:14.541 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.4999999999999998e-07 2024-02-09T17:44:14.541 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:14.541 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:14.541 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:44:14.541 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:14.259551+0000", 2024-02-09T17:44:14.541 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.3589999999999999e-06 2024-02-09T17:44:14.542 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:14.542 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:14.542 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:44:14.542 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:14.259556+0000", 2024-02-09T17:44:14.542 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.5090000000000004e-06 2024-02-09T17:44:14.542 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:14.542 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:14.542 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:44:14.542 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:14.259559+0000", 2024-02-09T17:44:14.542 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.4809999999999998e-06 2024-02-09T17:44:14.542 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:14.543 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:14.543 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:14.543 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:14.261927+0000", 2024-02-09T17:44:14.543 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0023682669999999999 2024-02-09T17:44:14.543 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:14.543 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:14.543 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:44:14.543 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:14.261950+0000", 2024-02-09T17:44:14.543 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2351999999999999e-05 2024-02-09T17:44:14.543 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:14.543 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:14.543 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:14.543 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:14.544 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:14.544 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25528.0:206480 5.5 e545/465 5:a884bd29:::benchmark_data_smithi179_115199_object77574:head v 545'12827, mlcod=545'12825)", 2024-02-09T17:44:14.544 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:44:14.268849+0000", 2024-02-09T17:44:14.544 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.011269928, 2024-02-09T17:44:14.544 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.011367211, 2024-02-09T17:44:14.544 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:44:14.544 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T17:44:14.544 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:44:14.544 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:14.544 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:44:14.544 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:14.268849+0000", 2024-02-09T17:44:14.544 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:14.544 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:14.544 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:14.545 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:44:14.545 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:14.268849+0000", 2024-02-09T17:44:14.545 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:14.545 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:14.545 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:14.545 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:44:14.545 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:14.268850+0000", 2024-02-09T17:44:14.545 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0529999999999999e-06 2024-02-09T17:44:14.545 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:14.545 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:14.545 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:44:14.545 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:14.268856+0000", 2024-02-09T17:44:14.545 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.6590000000000001e-06 2024-02-09T17:44:14.545 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:14.545 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:14.546 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:44:14.546 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:14.268868+0000", 2024-02-09T17:44:14.546 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1588e-05 2024-02-09T17:44:14.546 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:14.546 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:14.546 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:44:14.546 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:14.268876+0000", 2024-02-09T17:44:14.546 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.5089999999999993e-06 2024-02-09T17:44:14.546 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:14.546 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:14.546 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:14.546 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:14.272449+0000", 2024-02-09T17:44:14.546 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0035728700000000001 2024-02-09T17:44:14.546 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:14.546 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:14.547 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:44:14.547 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:14.272472+0000", 2024-02-09T17:44:14.547 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2892999999999999e-05 2024-02-09T17:44:14.547 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:14.547 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:14.547 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:14.547 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:14.547 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:14.547 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25528.0:206492 5.5 e545/465)", 2024-02-09T17:44:14.547 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:44:14.274542+0000", 2024-02-09T17:44:14.547 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.0055761839999999997, 2024-02-09T17:44:14.547 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0057303049999999998, 2024-02-09T17:44:14.547 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:44:14.547 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "queued for pg", 2024-02-09T17:44:14.547 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:44:14.548 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:14.548 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:44:14.548 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:14.274542+0000", 2024-02-09T17:44:14.548 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:14.548 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:14.548 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:14.548 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:44:14.548 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:14.274542+0000", 2024-02-09T17:44:14.548 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:14.548 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:14.548 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:14.548 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:44:14.548 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:14.274543+0000", 2024-02-09T17:44:14.548 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.1999999999999998e-07 2024-02-09T17:44:14.549 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:14.549 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:14.549 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:44:14.549 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:14.274551+0000", 2024-02-09T17:44:14.549 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.2740000000000007e-06 2024-02-09T17:44:14.549 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:14.549 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:14.549 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:44:14.549 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:14.274564+0000", 2024-02-09T17:44:14.549 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2203e-05 2024-02-09T17:44:14.549 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:14.549 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:14.549 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:44:14.549 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:14.274571+0000", 2024-02-09T17:44:14.549 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.5000000000000002e-06 2024-02-09T17:44:14.549 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:14.550 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:14.550 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:14.550 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:14.550 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:14.550 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25528.0:206495 5.0 e545/465 5:07ebd375:::benchmark_data_smithi179_115199_object77589:head v 545'13256, mlcod=545'13253)", 2024-02-09T17:44:14.550 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:44:14.277503+0000", 2024-02-09T17:44:14.550 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.0026157799999999998, 2024-02-09T17:44:14.550 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.002818152, 2024-02-09T17:44:14.550 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:44:14.550 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T17:44:14.550 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:44:14.550 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:14.550 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:44:14.550 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:14.277503+0000", 2024-02-09T17:44:14.550 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:14.551 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:14.551 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:14.551 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:44:14.551 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:14.277503+0000", 2024-02-09T17:44:14.551 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:14.551 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:14.551 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:14.551 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:44:14.551 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:14.277504+0000", 2024-02-09T17:44:14.551 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.116e-06 2024-02-09T17:44:14.551 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:14.551 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:14.551 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:44:14.552 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:14.277512+0000", 2024-02-09T17:44:14.552 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.9039999999999992e-06 2024-02-09T17:44:14.552 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:14.552 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:14.552 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:44:14.552 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:14.277524+0000", 2024-02-09T17:44:14.552 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1892000000000001e-05 2024-02-09T17:44:14.552 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:14.552 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:14.552 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:44:14.552 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:14.277532+0000", 2024-02-09T17:44:14.552 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.0399999999999993e-06 2024-02-09T17:44:14.552 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:14.552 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:14.553 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:14.553 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:14.277790+0000", 2024-02-09T17:44:14.553 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00025871100000000001 2024-02-09T17:44:14.553 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:14.553 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:14.553 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:44:14.553 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:14.277820+0000", 2024-02-09T17:44:14.553 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.9697e-05 2024-02-09T17:44:14.553 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:14.553 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:14.553 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:14.553 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:14.553 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T17:44:14.553 INFO:teuthology.orchestra.run.smithi099.stdout: "num_ops": 4 2024-02-09T17:44:14.553 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:44:14.554 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:44:14.554 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:14.569 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:44:14.573 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:44:14.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:14.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:44:14.613 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:44:14.659 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:44:14.680 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:44:14.702 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:44:14.751 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:44:14.751 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:14.785 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:44:14.814 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:44:14.819 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:44:14.819 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:44:14.819 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:44:14.819 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:44:14.819 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:44:14.819 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:44:14.819 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:44:14.819 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:44:14.819 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:44:14.819 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:44:14.820 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:44:14.820 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:44:14.893 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:14 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3795857858' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:44:15.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:14 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3795857858' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:44:15.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:14 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3795857858' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:44:16.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:15 smithi179 ceph-mon[43975]: pgmap v1370: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 3.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 721 op/s; 3339/159279 objects misplaced (2.096%) 2024-02-09T17:44:16.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:15 smithi162 ceph-mon[44304]: pgmap v1370: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 3.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 721 op/s; 3339/159279 objects misplaced (2.096%) 2024-02-09T17:44:16.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:15 smithi099 ceph-mon[39420]: pgmap v1370: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 3.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 721 op/s; 3339/159279 objects misplaced (2.096%) 2024-02-09T17:44:17.017 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:44:17.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:17.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:44:17.125 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:44:17.188 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:17.231 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:44:17.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:17.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:44:17.342 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:44:17.379 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:17.424 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:44:17.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:17.432 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:44:17.533 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:44:17.576 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:17.601 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:44:17.605 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:17.605 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:44:17.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:17 smithi179 ceph-mon[43975]: pgmap v1371: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 3.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 849 op/s; 3339/152691 objects misplaced (2.187%) 2024-02-09T17:44:17.698 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-09T17:44:17.706 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:44:17.803 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:17.931 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:17 smithi162 ceph-mon[44304]: pgmap v1371: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 3.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 849 op/s; 3339/152691 objects misplaced (2.187%) 2024-02-09T17:44:17.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:17 smithi099 ceph-mon[39420]: pgmap v1371: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 3.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 849 op/s; 3339/152691 objects misplaced (2.187%) 2024-02-09T17:44:18.123 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:44:18.164 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:44:18.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:18.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:44:18.186 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:44:18.269 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:44:18.320 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:18.346 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:44:18.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:18.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:44:18.451 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:44:18.510 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:18.563 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:44:18.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:18.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:44:18.670 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:44:18.723 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:18.798 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:44:18.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:18.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:44:18.908 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:44:18.955 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:18.981 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:44:18.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:18.986 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:44:19.087 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:44:19.119 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:19.181 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:44:19.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:19.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:44:19.288 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:44:19.317 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:19.689 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:44:19.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:19.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:44:19.794 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:44:19.825 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:19.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:19 smithi099 ceph-mon[39420]: pgmap v1372: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 808 op/s; 3080/149526 objects misplaced (2.060%) 2024-02-09T17:44:20.162 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:44:20.169 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:44:20.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:19 smithi179 ceph-mon[43975]: pgmap v1372: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 808 op/s; 3080/149526 objects misplaced (2.060%) 2024-02-09T17:44:20.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:19 smithi162 ceph-mon[44304]: pgmap v1372: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 808 op/s; 3080/149526 objects misplaced (2.060%) 2024-02-09T17:44:21.513 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:44:21.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:21.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:44:21.622 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:44:21.682 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:21.705 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:44:21.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:21.708 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:44:21.809 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:44:21.861 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:21.892 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:44:21.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:21.898 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:44:21.999 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:44:22.026 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:21 smithi099 ceph-mon[39420]: pgmap v1373: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 835 op/s; 3080/143661 objects misplaced (2.144%) 2024-02-09T17:44:22.073 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:22.131 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:44:22.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:22.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:44:22.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:21 smithi179 ceph-mon[43975]: pgmap v1373: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 835 op/s; 3080/143661 objects misplaced (2.144%) 2024-02-09T17:44:22.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:21 smithi162 ceph-mon[44304]: pgmap v1373: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 835 op/s; 3080/143661 objects misplaced (2.144%) 2024-02-09T17:44:22.241 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:44:22.288 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:22.313 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:44:22.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:22.319 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:44:22.420 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:44:22.497 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:22.920 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is unset 2024-02-09T17:44:23.213 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:44:23.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:23.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:44:23.217 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:22 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3298015755' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:44:23.217 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:22 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:44:23.217 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:22 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/392509994' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-09T17:44:23.317 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:44:23.361 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:23.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:22 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3298015755' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:44:23.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:22 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:44:23.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:22 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/392509994' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-09T17:44:23.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:22 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3298015755' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:44:23.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:22 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:44:23.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:22 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/392509994' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-09T17:44:23.765 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:44:23.766 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:44:23.766 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 545, 2024-02-09T17:44:23.766 INFO:teuthology.orchestra.run.smithi099.stdout: "fsid": "a69c870a-c76e-11ee-95b7-87774f69a715", 2024-02-09T17:44:23.766 INFO:teuthology.orchestra.run.smithi099.stdout: "created": "2024-02-09T17:15:26.113606+0000", 2024-02-09T17:44:23.766 INFO:teuthology.orchestra.run.smithi099.stdout: "modified": "2024-02-09T17:44:10.833433+0000", 2024-02-09T17:44:23.766 INFO:teuthology.orchestra.run.smithi099.stdout: "last_up_change": "2024-02-09T17:28:17.711044+0000", 2024-02-09T17:44:23.766 INFO:teuthology.orchestra.run.smithi099.stdout: "last_in_change": "2024-02-09T17:43:24.138763+0000", 2024-02-09T17:44:23.766 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-09T17:44:23.766 INFO:teuthology.orchestra.run.smithi099.stdout: "flags_num": 5806080, 2024-02-09T17:44:23.766 INFO:teuthology.orchestra.run.smithi099.stdout: "flags_set": [ 2024-02-09T17:44:23.766 INFO:teuthology.orchestra.run.smithi099.stdout: "nodeep-scrub", 2024-02-09T17:44:23.766 INFO:teuthology.orchestra.run.smithi099.stdout: "noscrub", 2024-02-09T17:44:23.766 INFO:teuthology.orchestra.run.smithi099.stdout: "pglog_hardlimit", 2024-02-09T17:44:23.766 INFO:teuthology.orchestra.run.smithi099.stdout: "purged_snapdirs", 2024-02-09T17:44:23.767 INFO:teuthology.orchestra.run.smithi099.stdout: "recovery_deletes", 2024-02-09T17:44:23.767 INFO:teuthology.orchestra.run.smithi099.stdout: "sortbitwise" 2024-02-09T17:44:23.767 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T17:44:23.767 INFO:teuthology.orchestra.run.smithi099.stdout: "crush_version": 27, 2024-02-09T17:44:23.767 INFO:teuthology.orchestra.run.smithi099.stdout: "full_ratio": 0.94999998807907104, 2024-02-09T17:44:23.767 INFO:teuthology.orchestra.run.smithi099.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-09T17:44:23.767 INFO:teuthology.orchestra.run.smithi099.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-09T17:44:23.767 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_snapshot": "", 2024-02-09T17:44:23.767 INFO:teuthology.orchestra.run.smithi099.stdout: "pool_max": 5, 2024-02-09T17:44:23.767 INFO:teuthology.orchestra.run.smithi099.stdout: "max_osd": 12, 2024-02-09T17:44:23.767 INFO:teuthology.orchestra.run.smithi099.stdout: "require_min_compat_client": "luminous", 2024-02-09T17:44:23.767 INFO:teuthology.orchestra.run.smithi099.stdout: "min_compat_client": "jewel", 2024-02-09T17:44:23.767 INFO:teuthology.orchestra.run.smithi099.stdout: "require_osd_release": "squid", 2024-02-09T17:44:23.767 INFO:teuthology.orchestra.run.smithi099.stdout: "allow_crimson": false, 2024-02-09T17:44:23.767 INFO:teuthology.orchestra.run.smithi099.stdout: "pools": [ 2024-02-09T17:44:23.768 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.768 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 1, 2024-02-09T17:44:23.768 INFO:teuthology.orchestra.run.smithi099.stdout: "pool_name": ".mgr", 2024-02-09T17:44:23.768 INFO:teuthology.orchestra.run.smithi099.stdout: "create_time": "2024-02-09T17:18:56.552286+0000", 2024-02-09T17:44:23.768 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 1, 2024-02-09T17:44:23.768 INFO:teuthology.orchestra.run.smithi099.stdout: "flags_names": "hashpspool", 2024-02-09T17:44:23.768 INFO:teuthology.orchestra.run.smithi099.stdout: "type": 1, 2024-02-09T17:44:23.768 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-02-09T17:44:23.768 INFO:teuthology.orchestra.run.smithi099.stdout: "min_size": 2, 2024-02-09T17:44:23.768 INFO:teuthology.orchestra.run.smithi099.stdout: "crush_rule": 0, 2024-02-09T17:44:23.768 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_count": 0, 2024-02-09T17:44:23.768 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_target": 0, 2024-02-09T17:44:23.768 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_barrier": 0, 2024-02-09T17:44:23.768 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-09T17:44:23.769 INFO:teuthology.orchestra.run.smithi099.stdout: "object_hash": 2, 2024-02-09T17:44:23.769 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_autoscale_mode": "off", 2024-02-09T17:44:23.769 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num": 31, 2024-02-09T17:44:23.769 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_placement_num": 31, 2024-02-09T17:44:23.769 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_placement_num_target": 31, 2024-02-09T17:44:23.769 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num_target": 31, 2024-02-09T17:44:23.769 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num_pending": 31, 2024-02-09T17:44:23.769 INFO:teuthology.orchestra.run.smithi099.stdout: "last_pg_merge_meta": { 2024-02-09T17:44:23.769 INFO:teuthology.orchestra.run.smithi099.stdout: "source_pgid": "0.0", 2024-02-09T17:44:23.769 INFO:teuthology.orchestra.run.smithi099.stdout: "ready_epoch": 0, 2024-02-09T17:44:23.769 INFO:teuthology.orchestra.run.smithi099.stdout: "last_epoch_started": 0, 2024-02-09T17:44:23.769 INFO:teuthology.orchestra.run.smithi099.stdout: "last_epoch_clean": 0, 2024-02-09T17:44:23.769 INFO:teuthology.orchestra.run.smithi099.stdout: "source_version": "0'0", 2024-02-09T17:44:23.769 INFO:teuthology.orchestra.run.smithi099.stdout: "target_version": "0'0" 2024-02-09T17:44:23.769 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.770 INFO:teuthology.orchestra.run.smithi099.stdout: "last_change": "544", 2024-02-09T17:44:23.770 INFO:teuthology.orchestra.run.smithi099.stdout: "last_force_op_resend": "0", 2024-02-09T17:44:23.770 INFO:teuthology.orchestra.run.smithi099.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-09T17:44:23.770 INFO:teuthology.orchestra.run.smithi099.stdout: "last_force_op_resend_preluminous": "514", 2024-02-09T17:44:23.770 INFO:teuthology.orchestra.run.smithi099.stdout: "auid": 0, 2024-02-09T17:44:23.770 INFO:teuthology.orchestra.run.smithi099.stdout: "snap_mode": "selfmanaged", 2024-02-09T17:44:23.770 INFO:teuthology.orchestra.run.smithi099.stdout: "snap_seq": 0, 2024-02-09T17:44:23.770 INFO:teuthology.orchestra.run.smithi099.stdout: "snap_epoch": 0, 2024-02-09T17:44:23.770 INFO:teuthology.orchestra.run.smithi099.stdout: "pool_snaps": [], 2024-02-09T17:44:23.770 INFO:teuthology.orchestra.run.smithi099.stdout: "removed_snaps": "[]", 2024-02-09T17:44:23.770 INFO:teuthology.orchestra.run.smithi099.stdout: "quota_max_bytes": 0, 2024-02-09T17:44:23.770 INFO:teuthology.orchestra.run.smithi099.stdout: "quota_max_objects": 0, 2024-02-09T17:44:23.770 INFO:teuthology.orchestra.run.smithi099.stdout: "tiers": [], 2024-02-09T17:44:23.770 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_of": -1, 2024-02-09T17:44:23.770 INFO:teuthology.orchestra.run.smithi099.stdout: "read_tier": -1, 2024-02-09T17:44:23.770 INFO:teuthology.orchestra.run.smithi099.stdout: "write_tier": -1, 2024-02-09T17:44:23.771 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_mode": "none", 2024-02-09T17:44:23.771 INFO:teuthology.orchestra.run.smithi099.stdout: "target_max_bytes": 0, 2024-02-09T17:44:23.771 INFO:teuthology.orchestra.run.smithi099.stdout: "target_max_objects": 0, 2024-02-09T17:44:23.771 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-09T17:44:23.771 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-09T17:44:23.771 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-09T17:44:23.771 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_min_flush_age": 0, 2024-02-09T17:44:23.771 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_min_evict_age": 0, 2024-02-09T17:44:23.771 INFO:teuthology.orchestra.run.smithi099.stdout: "erasure_code_profile": "", 2024-02-09T17:44:23.771 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_params": { 2024-02-09T17:44:23.771 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "none" 2024-02-09T17:44:23.771 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.771 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_period": 0, 2024-02-09T17:44:23.771 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_count": 0, 2024-02-09T17:44:23.771 INFO:teuthology.orchestra.run.smithi099.stdout: "use_gmt_hitset": true, 2024-02-09T17:44:23.771 INFO:teuthology.orchestra.run.smithi099.stdout: "min_read_recency_for_promote": 0, 2024-02-09T17:44:23.772 INFO:teuthology.orchestra.run.smithi099.stdout: "min_write_recency_for_promote": 0, 2024-02-09T17:44:23.772 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_grade_decay_rate": 0, 2024-02-09T17:44:23.772 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_search_last_n": 0, 2024-02-09T17:44:23.772 INFO:teuthology.orchestra.run.smithi099.stdout: "grade_table": [], 2024-02-09T17:44:23.772 INFO:teuthology.orchestra.run.smithi099.stdout: "stripe_width": 0, 2024-02-09T17:44:23.772 INFO:teuthology.orchestra.run.smithi099.stdout: "expected_num_objects": 0, 2024-02-09T17:44:23.772 INFO:teuthology.orchestra.run.smithi099.stdout: "fast_read": false, 2024-02-09T17:44:23.772 INFO:teuthology.orchestra.run.smithi099.stdout: "options": { 2024-02-09T17:44:23.772 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num_max": 32, 2024-02-09T17:44:23.772 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num_min": 1 2024-02-09T17:44:23.772 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.772 INFO:teuthology.orchestra.run.smithi099.stdout: "application_metadata": { 2024-02-09T17:44:23.772 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": {} 2024-02-09T17:44:23.772 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.773 INFO:teuthology.orchestra.run.smithi099.stdout: "read_balance": { 2024-02-09T17:44:23.773 INFO:teuthology.orchestra.run.smithi099.stdout: "score_type": "Fair distribution", 2024-02-09T17:44:23.773 INFO:teuthology.orchestra.run.smithi099.stdout: "score_acting": 1.5499999523162842, 2024-02-09T17:44:23.773 INFO:teuthology.orchestra.run.smithi099.stdout: "score_stable": 1.5499999523162842, 2024-02-09T17:44:23.773 INFO:teuthology.orchestra.run.smithi099.stdout: "optimal_score": 1, 2024-02-09T17:44:23.773 INFO:teuthology.orchestra.run.smithi099.stdout: "raw_score_acting": 1.5499999523162842, 2024-02-09T17:44:23.773 INFO:teuthology.orchestra.run.smithi099.stdout: "raw_score_stable": 1.5499999523162842, 2024-02-09T17:44:23.773 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity_weighted": 0.88067114353179932, 2024-02-09T17:44:23.773 INFO:teuthology.orchestra.run.smithi099.stdout: "average_primary_affinity": 0.87999999523162842, 2024-02-09T17:44:23.773 INFO:teuthology.orchestra.run.smithi099.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-02-09T17:44:23.773 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.773 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.773 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.773 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 5, 2024-02-09T17:44:23.773 INFO:teuthology.orchestra.run.smithi099.stdout: "pool_name": "unique_pool_3", 2024-02-09T17:44:23.773 INFO:teuthology.orchestra.run.smithi099.stdout: "create_time": "2024-02-09T17:40:47.713102+0000", 2024-02-09T17:44:23.774 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 1, 2024-02-09T17:44:23.774 INFO:teuthology.orchestra.run.smithi099.stdout: "flags_names": "hashpspool", 2024-02-09T17:44:23.774 INFO:teuthology.orchestra.run.smithi099.stdout: "type": 1, 2024-02-09T17:44:23.774 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-02-09T17:44:23.774 INFO:teuthology.orchestra.run.smithi099.stdout: "min_size": 2, 2024-02-09T17:44:23.774 INFO:teuthology.orchestra.run.smithi099.stdout: "crush_rule": 0, 2024-02-09T17:44:23.774 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_count": 0, 2024-02-09T17:44:23.774 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_target": 0, 2024-02-09T17:44:23.774 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_barrier": 0, 2024-02-09T17:44:23.774 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-09T17:44:23.774 INFO:teuthology.orchestra.run.smithi099.stdout: "object_hash": 2, 2024-02-09T17:44:23.774 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_autoscale_mode": "off", 2024-02-09T17:44:23.774 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num": 16, 2024-02-09T17:44:23.774 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_placement_num": 16, 2024-02-09T17:44:23.774 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_placement_num_target": 16, 2024-02-09T17:44:23.775 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num_target": 16, 2024-02-09T17:44:23.775 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num_pending": 16, 2024-02-09T17:44:23.775 INFO:teuthology.orchestra.run.smithi099.stdout: "last_pg_merge_meta": { 2024-02-09T17:44:23.775 INFO:teuthology.orchestra.run.smithi099.stdout: "source_pgid": "0.0", 2024-02-09T17:44:23.775 INFO:teuthology.orchestra.run.smithi099.stdout: "ready_epoch": 0, 2024-02-09T17:44:23.775 INFO:teuthology.orchestra.run.smithi099.stdout: "last_epoch_started": 0, 2024-02-09T17:44:23.775 INFO:teuthology.orchestra.run.smithi099.stdout: "last_epoch_clean": 0, 2024-02-09T17:44:23.775 INFO:teuthology.orchestra.run.smithi099.stdout: "source_version": "0'0", 2024-02-09T17:44:23.775 INFO:teuthology.orchestra.run.smithi099.stdout: "target_version": "0'0" 2024-02-09T17:44:23.775 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.775 INFO:teuthology.orchestra.run.smithi099.stdout: "last_change": "507", 2024-02-09T17:44:23.775 INFO:teuthology.orchestra.run.smithi099.stdout: "last_force_op_resend": "0", 2024-02-09T17:44:23.775 INFO:teuthology.orchestra.run.smithi099.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-09T17:44:23.775 INFO:teuthology.orchestra.run.smithi099.stdout: "last_force_op_resend_preluminous": "0", 2024-02-09T17:44:23.775 INFO:teuthology.orchestra.run.smithi099.stdout: "auid": 0, 2024-02-09T17:44:23.775 INFO:teuthology.orchestra.run.smithi099.stdout: "snap_mode": "selfmanaged", 2024-02-09T17:44:23.775 INFO:teuthology.orchestra.run.smithi099.stdout: "snap_seq": 0, 2024-02-09T17:44:23.776 INFO:teuthology.orchestra.run.smithi099.stdout: "snap_epoch": 0, 2024-02-09T17:44:23.776 INFO:teuthology.orchestra.run.smithi099.stdout: "pool_snaps": [], 2024-02-09T17:44:23.776 INFO:teuthology.orchestra.run.smithi099.stdout: "removed_snaps": "[]", 2024-02-09T17:44:23.776 INFO:teuthology.orchestra.run.smithi099.stdout: "quota_max_bytes": 0, 2024-02-09T17:44:23.776 INFO:teuthology.orchestra.run.smithi099.stdout: "quota_max_objects": 0, 2024-02-09T17:44:23.776 INFO:teuthology.orchestra.run.smithi099.stdout: "tiers": [], 2024-02-09T17:44:23.776 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_of": -1, 2024-02-09T17:44:23.776 INFO:teuthology.orchestra.run.smithi099.stdout: "read_tier": -1, 2024-02-09T17:44:23.776 INFO:teuthology.orchestra.run.smithi099.stdout: "write_tier": -1, 2024-02-09T17:44:23.776 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_mode": "none", 2024-02-09T17:44:23.776 INFO:teuthology.orchestra.run.smithi099.stdout: "target_max_bytes": 0, 2024-02-09T17:44:23.776 INFO:teuthology.orchestra.run.smithi099.stdout: "target_max_objects": 0, 2024-02-09T17:44:23.776 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-09T17:44:23.776 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-09T17:44:23.777 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-09T17:44:23.777 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_min_flush_age": 0, 2024-02-09T17:44:23.777 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_min_evict_age": 0, 2024-02-09T17:44:23.777 INFO:teuthology.orchestra.run.smithi099.stdout: "erasure_code_profile": "", 2024-02-09T17:44:23.777 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_params": { 2024-02-09T17:44:23.777 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "none" 2024-02-09T17:44:23.777 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.777 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_period": 0, 2024-02-09T17:44:23.777 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_count": 0, 2024-02-09T17:44:23.777 INFO:teuthology.orchestra.run.smithi099.stdout: "use_gmt_hitset": true, 2024-02-09T17:44:23.777 INFO:teuthology.orchestra.run.smithi099.stdout: "min_read_recency_for_promote": 0, 2024-02-09T17:44:23.777 INFO:teuthology.orchestra.run.smithi099.stdout: "min_write_recency_for_promote": 0, 2024-02-09T17:44:23.777 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_grade_decay_rate": 0, 2024-02-09T17:44:23.777 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_search_last_n": 0, 2024-02-09T17:44:23.777 INFO:teuthology.orchestra.run.smithi099.stdout: "grade_table": [], 2024-02-09T17:44:23.777 INFO:teuthology.orchestra.run.smithi099.stdout: "stripe_width": 0, 2024-02-09T17:44:23.778 INFO:teuthology.orchestra.run.smithi099.stdout: "expected_num_objects": 0, 2024-02-09T17:44:23.778 INFO:teuthology.orchestra.run.smithi099.stdout: "fast_read": false, 2024-02-09T17:44:23.778 INFO:teuthology.orchestra.run.smithi099.stdout: "options": {}, 2024-02-09T17:44:23.778 INFO:teuthology.orchestra.run.smithi099.stdout: "application_metadata": { 2024-02-09T17:44:23.778 INFO:teuthology.orchestra.run.smithi099.stdout: "rados": {} 2024-02-09T17:44:23.778 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.778 INFO:teuthology.orchestra.run.smithi099.stdout: "read_balance": { 2024-02-09T17:44:23.778 INFO:teuthology.orchestra.run.smithi099.stdout: "score_type": "Fair distribution", 2024-02-09T17:44:23.778 INFO:teuthology.orchestra.run.smithi099.stdout: "score_acting": 1.5, 2024-02-09T17:44:23.778 INFO:teuthology.orchestra.run.smithi099.stdout: "score_stable": 1.5, 2024-02-09T17:44:23.778 INFO:teuthology.orchestra.run.smithi099.stdout: "optimal_score": 1, 2024-02-09T17:44:23.778 INFO:teuthology.orchestra.run.smithi099.stdout: "raw_score_acting": 1.5, 2024-02-09T17:44:23.778 INFO:teuthology.orchestra.run.smithi099.stdout: "raw_score_stable": 1.5, 2024-02-09T17:44:23.778 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity_weighted": 0.88067114353179932, 2024-02-09T17:44:23.778 INFO:teuthology.orchestra.run.smithi099.stdout: "average_primary_affinity": 0.87999999523162842, 2024-02-09T17:44:23.779 INFO:teuthology.orchestra.run.smithi099.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-02-09T17:44:23.779 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.779 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.779 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T17:44:23.779 INFO:teuthology.orchestra.run.smithi099.stdout: "osds": [ 2024-02-09T17:44:23.779 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.779 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 0, 2024-02-09T17:44:23.779 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "21739aa7-e278-45da-9a3f-867385f01baf", 2024-02-09T17:44:23.779 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:44:23.779 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:44:23.779 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:44:23.779 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 0.609619140625, 2024-02-09T17:44:23.779 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:44:23.779 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:44:23.779 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 8, 2024-02-09T17:44:23.779 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 531, 2024-02-09T17:44:23.780 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:44:23.780 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:44:23.780 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:44:23.780 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:44:23.780 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.780 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:44:23.780 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6802", 2024-02-09T17:44:23.780 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T17:44:23.780 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.780 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.780 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:44:23.780 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6803", 2024-02-09T17:44:23.781 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T17:44:23.781 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.781 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.781 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.781 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:44:23.781 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:44:23.781 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.781 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:44:23.781 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6804", 2024-02-09T17:44:23.781 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T17:44:23.781 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.781 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.781 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:44:23.782 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6805", 2024-02-09T17:44:23.782 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T17:44:23.782 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.782 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.782 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.782 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:44:23.782 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:44:23.782 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.782 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:44:23.782 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6808", 2024-02-09T17:44:23.782 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T17:44:23.782 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.782 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.782 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:44:23.782 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6809", 2024-02-09T17:44:23.783 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T17:44:23.783 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.783 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.783 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.783 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:44:23.783 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:44:23.783 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.783 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:44:23.783 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6806", 2024-02-09T17:44:23.783 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T17:44:23.783 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.783 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.783 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:44:23.783 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6807", 2024-02-09T17:44:23.783 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T17:44:23.784 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.784 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.784 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.784 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.99:6803/3859868923", 2024-02-09T17:44:23.784 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.99:6805/3859868923", 2024-02-09T17:44:23.784 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.99:6809/3859868923", 2024-02-09T17:44:23.784 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.99:6807/3859868923", 2024-02-09T17:44:23.784 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:44:23.784 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:44:23.784 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:44:23.784 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.784 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.784 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.784 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 1, 2024-02-09T17:44:23.785 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "8eab347c-9065-4de1-93ae-e2eb7e11cab7", 2024-02-09T17:44:23.785 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:44:23.785 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:44:23.785 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:44:23.785 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T17:44:23.785 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:44:23.785 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:44:23.785 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 13, 2024-02-09T17:44:23.785 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 525, 2024-02-09T17:44:23.785 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:44:23.785 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:44:23.785 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:44:23.785 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:44:23.785 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.785 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:44:23.786 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6810", 2024-02-09T17:44:23.786 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T17:44:23.786 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.786 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.786 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:44:23.786 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6811", 2024-02-09T17:44:23.786 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T17:44:23.786 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.786 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.786 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.786 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:44:23.786 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:44:23.786 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.786 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:44:23.786 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6812", 2024-02-09T17:44:23.786 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T17:44:23.787 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.787 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.787 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:44:23.787 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6813", 2024-02-09T17:44:23.787 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T17:44:23.787 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.787 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.787 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.787 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:44:23.787 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:44:23.787 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.787 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:44:23.787 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6816", 2024-02-09T17:44:23.787 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T17:44:23.787 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.788 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.788 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:44:23.788 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6817", 2024-02-09T17:44:23.788 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T17:44:23.788 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.788 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.788 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.788 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:44:23.788 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:44:23.788 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.788 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:44:23.788 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6814", 2024-02-09T17:44:23.788 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T17:44:23.788 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.788 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.789 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:44:23.789 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6815", 2024-02-09T17:44:23.789 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T17:44:23.789 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.789 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.789 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.789 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.99:6811/3868328452", 2024-02-09T17:44:23.789 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.99:6813/3868328452", 2024-02-09T17:44:23.789 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.99:6817/3868328452", 2024-02-09T17:44:23.789 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.99:6815/3868328452", 2024-02-09T17:44:23.789 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:44:23.789 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:44:23.789 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:44:23.789 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.789 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.789 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.789 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 2, 2024-02-09T17:44:23.790 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "575f4257-7e34-485e-963c-2607b69e1220", 2024-02-09T17:44:23.790 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:44:23.790 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:44:23.790 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:44:23.790 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 0.3601837158203125, 2024-02-09T17:44:23.790 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:44:23.790 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:44:23.790 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 18, 2024-02-09T17:44:23.790 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 537, 2024-02-09T17:44:23.790 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:44:23.790 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:44:23.790 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:44:23.790 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:44:23.790 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.790 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:44:23.791 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6818", 2024-02-09T17:44:23.791 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T17:44:23.791 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.791 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.791 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:44:23.791 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6819", 2024-02-09T17:44:23.791 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T17:44:23.791 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.791 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.791 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.791 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:44:23.791 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:44:23.791 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.791 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:44:23.791 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6820", 2024-02-09T17:44:23.791 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T17:44:23.792 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.792 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.792 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:44:23.792 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6821", 2024-02-09T17:44:23.792 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T17:44:23.792 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.792 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.792 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.792 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:44:23.792 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:44:23.792 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.792 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:44:23.792 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6824", 2024-02-09T17:44:23.792 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T17:44:23.793 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.793 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.793 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:44:23.793 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6825", 2024-02-09T17:44:23.793 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T17:44:23.793 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.793 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.793 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.793 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:44:23.793 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:44:23.793 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.793 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:44:23.793 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6822", 2024-02-09T17:44:23.793 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T17:44:23.793 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.794 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.794 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:44:23.794 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6823", 2024-02-09T17:44:23.794 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T17:44:23.794 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.794 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.794 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.794 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.99:6819/537506237", 2024-02-09T17:44:23.794 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.99:6821/537506237", 2024-02-09T17:44:23.794 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.99:6825/537506237", 2024-02-09T17:44:23.794 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.99:6823/537506237", 2024-02-09T17:44:23.794 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:44:23.794 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:44:23.794 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:44:23.794 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.794 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.795 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.795 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 3, 2024-02-09T17:44:23.795 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "1a520bcf-81bb-4f1d-80fb-11e98008e429", 2024-02-09T17:44:23.795 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:44:23.795 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:44:23.795 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:44:23.795 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T17:44:23.795 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:44:23.795 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:44:23.795 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 27, 2024-02-09T17:44:23.795 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 518, 2024-02-09T17:44:23.795 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:44:23.795 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:44:23.795 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:44:23.795 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:44:23.795 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.796 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:44:23.796 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6826", 2024-02-09T17:44:23.796 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 762784793 2024-02-09T17:44:23.796 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.796 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.796 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:44:23.796 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6827", 2024-02-09T17:44:23.796 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 762784793 2024-02-09T17:44:23.796 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.796 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.796 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.796 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:44:23.796 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:44:23.796 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.796 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:44:23.797 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6828", 2024-02-09T17:44:23.797 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 762784793 2024-02-09T17:44:23.797 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.797 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.797 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:44:23.797 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6829", 2024-02-09T17:44:23.797 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 762784793 2024-02-09T17:44:23.797 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.797 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.797 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.797 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:44:23.797 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:44:23.797 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.797 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:44:23.797 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6832", 2024-02-09T17:44:23.798 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 762784793 2024-02-09T17:44:23.798 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.798 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.798 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:44:23.798 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6833", 2024-02-09T17:44:23.798 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 762784793 2024-02-09T17:44:23.798 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.798 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.798 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.798 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:44:23.798 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:44:23.798 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.798 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:44:23.798 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6830", 2024-02-09T17:44:23.798 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 762784793 2024-02-09T17:44:23.799 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.799 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.799 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:44:23.799 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6831", 2024-02-09T17:44:23.799 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 762784793 2024-02-09T17:44:23.799 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.799 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.799 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.799 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.99:6827/762784793", 2024-02-09T17:44:23.799 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.99:6829/762784793", 2024-02-09T17:44:23.799 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.99:6833/762784793", 2024-02-09T17:44:23.799 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.99:6831/762784793", 2024-02-09T17:44:23.799 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:44:23.799 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:44:23.799 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:44:23.799 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.800 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.800 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.800 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 4, 2024-02-09T17:44:23.800 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "4617c2fe-c9d8-40cd-887b-303e3215bdf8", 2024-02-09T17:44:23.800 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:44:23.800 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:44:23.800 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:44:23.800 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T17:44:23.800 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:44:23.800 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:44:23.800 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 32, 2024-02-09T17:44:23.800 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 519, 2024-02-09T17:44:23.800 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:44:23.800 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:44:23.801 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:44:23.801 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:44:23.801 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.801 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:44:23.801 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6800", 2024-02-09T17:44:23.801 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T17:44:23.801 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.801 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.801 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:44:23.801 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6801", 2024-02-09T17:44:23.801 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T17:44:23.801 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.801 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.801 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.801 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:44:23.802 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:44:23.802 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.802 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:44:23.802 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6802", 2024-02-09T17:44:23.802 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T17:44:23.802 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.802 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.802 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:44:23.802 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6803", 2024-02-09T17:44:23.802 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T17:44:23.802 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.802 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.802 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.802 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:44:23.802 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:44:23.802 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.803 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:44:23.803 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6806", 2024-02-09T17:44:23.803 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T17:44:23.803 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.803 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.803 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:44:23.803 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6807", 2024-02-09T17:44:23.803 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T17:44:23.803 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.803 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.803 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.803 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:44:23.803 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:44:23.803 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.803 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:44:23.804 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6804", 2024-02-09T17:44:23.804 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T17:44:23.804 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.804 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.804 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:44:23.804 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6805", 2024-02-09T17:44:23.804 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T17:44:23.804 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.804 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.804 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.804 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.162:6801/4178676504", 2024-02-09T17:44:23.804 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.162:6803/4178676504", 2024-02-09T17:44:23.804 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.162:6807/4178676504", 2024-02-09T17:44:23.804 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.162:6805/4178676504", 2024-02-09T17:44:23.804 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:44:23.805 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:44:23.805 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:44:23.805 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.805 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.805 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.805 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 5, 2024-02-09T17:44:23.805 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "da98e3f3-173e-46fe-be8b-1ccbe8b0dabc", 2024-02-09T17:44:23.805 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:44:23.805 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:44:23.805 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:44:23.805 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T17:44:23.805 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:44:23.805 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:44:23.805 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 40, 2024-02-09T17:44:23.805 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 536, 2024-02-09T17:44:23.806 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:44:23.806 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:44:23.806 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:44:23.806 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:44:23.806 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.806 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:44:23.806 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6808", 2024-02-09T17:44:23.806 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T17:44:23.806 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.806 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.806 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:44:23.806 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6809", 2024-02-09T17:44:23.806 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T17:44:23.806 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.806 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.807 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.807 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:44:23.807 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:44:23.807 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.807 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:44:23.807 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6810", 2024-02-09T17:44:23.807 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T17:44:23.807 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.807 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.807 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:44:23.807 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6811", 2024-02-09T17:44:23.807 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T17:44:23.807 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.807 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.807 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.808 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:44:23.808 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:44:23.808 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.808 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:44:23.808 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6814", 2024-02-09T17:44:23.808 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T17:44:23.808 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.808 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.808 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:44:23.808 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6815", 2024-02-09T17:44:23.808 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T17:44:23.808 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.808 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.808 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.808 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:44:23.808 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:44:23.809 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.809 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:44:23.809 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6812", 2024-02-09T17:44:23.809 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T17:44:23.809 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.809 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.809 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:44:23.809 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6813", 2024-02-09T17:44:23.809 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T17:44:23.809 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.809 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.809 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.809 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.162:6809/1489371339", 2024-02-09T17:44:23.809 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.162:6811/1489371339", 2024-02-09T17:44:23.809 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.162:6815/1489371339", 2024-02-09T17:44:23.809 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.162:6813/1489371339", 2024-02-09T17:44:23.810 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:44:23.810 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:44:23.810 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:44:23.810 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.810 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.810 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.810 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 6, 2024-02-09T17:44:23.810 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "ff50382d-5ddf-4cda-bb0c-ff055feeaa40", 2024-02-09T17:44:23.810 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:44:23.810 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:44:23.810 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:44:23.810 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T17:44:23.810 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:44:23.810 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:44:23.810 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 49, 2024-02-09T17:44:23.810 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 518, 2024-02-09T17:44:23.811 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:44:23.811 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:44:23.811 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:44:23.811 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:44:23.811 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.811 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:44:23.811 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6816", 2024-02-09T17:44:23.811 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T17:44:23.811 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.811 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.811 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:44:23.811 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6817", 2024-02-09T17:44:23.811 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T17:44:23.811 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.811 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.812 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.812 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:44:23.812 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:44:23.812 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.812 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:44:23.812 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6818", 2024-02-09T17:44:23.812 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T17:44:23.812 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.812 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.812 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:44:23.812 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6819", 2024-02-09T17:44:23.812 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T17:44:23.812 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.812 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.812 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.812 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:44:23.813 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:44:23.813 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.813 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:44:23.813 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6822", 2024-02-09T17:44:23.813 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T17:44:23.813 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.813 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.813 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:44:23.813 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6823", 2024-02-09T17:44:23.813 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T17:44:23.813 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.813 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.813 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.813 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:44:23.813 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:44:23.814 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.814 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:44:23.814 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6820", 2024-02-09T17:44:23.814 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T17:44:23.814 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.814 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.814 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:44:23.814 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6821", 2024-02-09T17:44:23.814 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T17:44:23.814 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.814 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.814 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.814 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.162:6817/821169663", 2024-02-09T17:44:23.814 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.162:6819/821169663", 2024-02-09T17:44:23.815 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.162:6823/821169663", 2024-02-09T17:44:23.815 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.162:6821/821169663", 2024-02-09T17:44:23.815 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:44:23.815 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:44:23.815 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:44:23.815 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.815 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.815 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.815 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 7, 2024-02-09T17:44:23.815 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "a82984dc-5e22-424b-a446-ae2ee71eee19", 2024-02-09T17:44:23.815 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:44:23.815 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:44:23.815 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:44:23.815 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 0.5982513427734375, 2024-02-09T17:44:23.815 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:44:23.815 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:44:23.816 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 57, 2024-02-09T17:44:23.816 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 525, 2024-02-09T17:44:23.816 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:44:23.816 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:44:23.816 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:44:23.816 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:44:23.816 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.816 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:44:23.816 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6824", 2024-02-09T17:44:23.816 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T17:44:23.816 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.816 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.816 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:44:23.816 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6825", 2024-02-09T17:44:23.817 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T17:44:23.817 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.817 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.817 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.817 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:44:23.817 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:44:23.817 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.817 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:44:23.817 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6826", 2024-02-09T17:44:23.817 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T17:44:23.817 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.817 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.817 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:44:23.817 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6827", 2024-02-09T17:44:23.817 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T17:44:23.817 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.818 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.818 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.818 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:44:23.818 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:44:23.818 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.818 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:44:23.818 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6830", 2024-02-09T17:44:23.818 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T17:44:23.818 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.818 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.818 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:44:23.818 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6831", 2024-02-09T17:44:23.818 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T17:44:23.818 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.819 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.819 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.819 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:44:23.819 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:44:23.819 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.819 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:44:23.819 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6828", 2024-02-09T17:44:23.819 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T17:44:23.819 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.819 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.819 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:44:23.819 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6829", 2024-02-09T17:44:23.819 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T17:44:23.819 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.819 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.819 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.820 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.162:6825/965264369", 2024-02-09T17:44:23.820 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.162:6827/965264369", 2024-02-09T17:44:23.820 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.162:6831/965264369", 2024-02-09T17:44:23.820 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.162:6829/965264369", 2024-02-09T17:44:23.820 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:44:23.820 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:44:23.820 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:44:23.820 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.820 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.821 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.821 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 8, 2024-02-09T17:44:23.821 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "63f9d79f-0400-44dc-b388-df00fefb5820", 2024-02-09T17:44:23.821 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:44:23.821 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:44:23.821 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:44:23.821 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T17:44:23.821 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 66, 2024-02-09T17:44:23.821 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 145, 2024-02-09T17:44:23.821 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 171, 2024-02-09T17:44:23.821 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 522, 2024-02-09T17:44:23.822 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 146, 2024-02-09T17:44:23.822 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:44:23.822 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:44:23.822 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:44:23.822 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.822 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:44:23.822 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6800", 2024-02-09T17:44:23.822 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T17:44:23.822 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.822 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.822 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:44:23.823 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6801", 2024-02-09T17:44:23.823 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T17:44:23.823 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.823 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.823 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.823 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:44:23.823 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:44:23.823 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.823 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:44:23.823 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6802", 2024-02-09T17:44:23.823 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T17:44:23.824 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.824 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.824 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:44:23.824 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6803", 2024-02-09T17:44:23.824 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T17:44:23.824 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.824 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.824 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.824 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:44:23.824 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:44:23.824 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.825 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:44:23.825 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6806", 2024-02-09T17:44:23.825 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T17:44:23.825 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.825 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.825 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:44:23.825 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6807", 2024-02-09T17:44:23.825 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T17:44:23.825 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.825 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.825 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.825 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:44:23.826 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:44:23.826 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.826 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:44:23.826 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6804", 2024-02-09T17:44:23.826 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T17:44:23.826 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.826 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.826 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:44:23.826 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6805", 2024-02-09T17:44:23.826 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T17:44:23.826 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.826 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.826 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.826 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.179:6801/2558294495", 2024-02-09T17:44:23.826 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.179:6803/2558294495", 2024-02-09T17:44:23.827 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.179:6807/2558294495", 2024-02-09T17:44:23.827 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.179:6805/2558294495", 2024-02-09T17:44:23.827 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:44:23.827 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:44:23.827 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:44:23.827 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.827 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.827 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.827 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 9, 2024-02-09T17:44:23.827 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "7d995854-7328-4dfc-b1a9-0edca8d74f46", 2024-02-09T17:44:23.827 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:44:23.827 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:44:23.827 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:44:23.827 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T17:44:23.827 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:44:23.828 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:44:23.828 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 77, 2024-02-09T17:44:23.828 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 514, 2024-02-09T17:44:23.828 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:44:23.828 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:44:23.828 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:44:23.828 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:44:23.828 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.828 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:44:23.828 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6808", 2024-02-09T17:44:23.828 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T17:44:23.828 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.828 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.828 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:44:23.829 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6809", 2024-02-09T17:44:23.829 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T17:44:23.829 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.829 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.829 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.829 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:44:23.829 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:44:23.829 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.829 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:44:23.829 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6810", 2024-02-09T17:44:23.829 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T17:44:23.829 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.829 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.829 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:44:23.829 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6811", 2024-02-09T17:44:23.830 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T17:44:23.830 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.830 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.830 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.830 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:44:23.830 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:44:23.830 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.830 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:44:23.830 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6814", 2024-02-09T17:44:23.830 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T17:44:23.830 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.830 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.830 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:44:23.830 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6815", 2024-02-09T17:44:23.830 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T17:44:23.831 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.831 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.831 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.831 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:44:23.831 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:44:23.831 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.831 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:44:23.831 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6812", 2024-02-09T17:44:23.831 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T17:44:23.831 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.831 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.831 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:44:23.831 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6813", 2024-02-09T17:44:23.831 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T17:44:23.831 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.832 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.832 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.832 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.179:6809/504427183", 2024-02-09T17:44:23.832 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.179:6811/504427183", 2024-02-09T17:44:23.832 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.179:6815/504427183", 2024-02-09T17:44:23.832 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.179:6813/504427183", 2024-02-09T17:44:23.832 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:44:23.832 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:44:23.832 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:44:23.832 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.833 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.833 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.833 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 10, 2024-02-09T17:44:23.833 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "151fbedd-08e6-46e6-a4e5-a14248399f61", 2024-02-09T17:44:23.833 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:44:23.834 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:44:23.834 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:44:23.834 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T17:44:23.834 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:44:23.834 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:44:23.834 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 84, 2024-02-09T17:44:23.834 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 535, 2024-02-09T17:44:23.834 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:44:23.835 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:44:23.835 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:44:23.835 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:44:23.835 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.836 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:44:23.836 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6816", 2024-02-09T17:44:23.836 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T17:44:23.836 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.836 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.837 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:44:23.837 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6817", 2024-02-09T17:44:23.837 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T17:44:23.837 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.837 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.838 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.838 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:44:23.838 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:44:23.838 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.839 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:44:23.839 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6818", 2024-02-09T17:44:23.839 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T17:44:23.840 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.840 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.840 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:44:23.840 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6819", 2024-02-09T17:44:23.840 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T17:44:23.840 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.840 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.841 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.841 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:44:23.841 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:44:23.841 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.842 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:44:23.842 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6822", 2024-02-09T17:44:23.842 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T17:44:23.842 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.843 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.843 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:44:23.843 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6823", 2024-02-09T17:44:23.843 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T17:44:23.844 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.844 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.844 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.845 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:44:23.845 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:44:23.845 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.845 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:44:23.846 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6820", 2024-02-09T17:44:23.846 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T17:44:23.846 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.846 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.846 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:44:23.846 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6821", 2024-02-09T17:44:23.846 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T17:44:23.846 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.846 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.846 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.847 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.179:6817/2403345164", 2024-02-09T17:44:23.847 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.179:6819/2403345164", 2024-02-09T17:44:23.847 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.179:6823/2403345164", 2024-02-09T17:44:23.847 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.179:6821/2403345164", 2024-02-09T17:44:23.847 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:44:23.847 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:44:23.847 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:44:23.847 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.847 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.847 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.847 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 11, 2024-02-09T17:44:23.847 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "6d3a3ad8-417a-483f-991c-52549dbe7d73", 2024-02-09T17:44:23.848 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:44:23.848 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:44:23.848 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:44:23.848 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T17:44:23.848 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:44:23.848 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:44:23.848 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 90, 2024-02-09T17:44:23.848 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 519, 2024-02-09T17:44:23.848 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:44:23.848 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:44:23.848 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:44:23.848 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:44:23.849 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.849 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:44:23.849 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6824", 2024-02-09T17:44:23.849 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T17:44:23.849 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.849 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.849 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:44:23.849 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6825", 2024-02-09T17:44:23.849 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T17:44:23.849 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.849 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.849 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.850 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:44:23.850 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:44:23.850 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.850 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:44:23.850 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6826", 2024-02-09T17:44:23.850 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T17:44:23.850 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.850 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.850 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:44:23.850 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6827", 2024-02-09T17:44:23.850 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T17:44:23.850 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.851 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.851 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.851 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:44:23.851 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:44:23.851 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.851 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:44:23.851 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6830", 2024-02-09T17:44:23.851 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T17:44:23.851 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.851 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.851 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:44:23.851 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6831", 2024-02-09T17:44:23.851 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T17:44:23.852 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.852 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.852 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.852 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:44:23.852 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:44:23.852 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.852 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:44:23.852 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6828", 2024-02-09T17:44:23.852 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T17:44:23.852 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.852 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.853 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:44:23.853 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6829", 2024-02-09T17:44:23.853 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T17:44:23.853 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.853 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.853 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.853 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.179:6825/2899402373", 2024-02-09T17:44:23.853 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.179:6827/2899402373", 2024-02-09T17:44:23.853 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.179:6831/2899402373", 2024-02-09T17:44:23.853 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.179:6829/2899402373", 2024-02-09T17:44:23.853 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:44:23.853 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:44:23.853 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:44:23.854 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.854 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.854 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T17:44:23.854 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_xinfo": [ 2024-02-09T17:44:23.854 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.854 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 0, 2024-02-09T17:44:23.854 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:44:23.854 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:44:23.854 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:44:23.854 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:44:23.854 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:44:23.854 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:43:58.666762+0000", 2024-02-09T17:44:23.855 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:44:23.855 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.855 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.855 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 1, 2024-02-09T17:44:23.855 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:44:23.855 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:44:23.855 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:44:23.855 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:44:23.855 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:44:23.855 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:43:29.854888+0000", 2024-02-09T17:44:23.855 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:44:23.855 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.855 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.856 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 2, 2024-02-09T17:44:23.856 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:44:23.856 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:44:23.856 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:44:23.856 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:44:23.856 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:44:23.856 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:43:50.747285+0000", 2024-02-09T17:44:23.856 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:44:23.856 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.856 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.856 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 3, 2024-02-09T17:44:23.856 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:44:23.856 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:44:23.857 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:44:23.857 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:44:23.857 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:44:23.857 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:43:06.026036+0000", 2024-02-09T17:44:23.857 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:44:23.857 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.857 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.857 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 4, 2024-02-09T17:44:23.857 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:44:23.857 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:44:23.857 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:44:23.857 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:44:23.857 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:44:23.857 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:44:04.366013+0000", 2024-02-09T17:44:23.858 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:44:23.858 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.858 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.858 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 5, 2024-02-09T17:44:23.858 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:44:23.858 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:44:23.858 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:44:23.858 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:44:23.858 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:44:23.858 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:43:59.000265+0000", 2024-02-09T17:44:23.858 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:44:23.858 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.858 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.858 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 6, 2024-02-09T17:44:23.859 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:44:23.859 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:44:23.859 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:44:23.859 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:44:23.859 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:44:23.859 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:43:37.551816+0000", 2024-02-09T17:44:23.859 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:44:23.859 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.859 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.859 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 7, 2024-02-09T17:44:23.859 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:44:23.859 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:44:23.859 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:44:23.859 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:44:23.860 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:44:23.860 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:43:02.368002+0000", 2024-02-09T17:44:23.860 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:44:23.860 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.860 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.860 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 8, 2024-02-09T17:44:23.860 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:44:23.860 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:44:23.860 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:44:23.860 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:44:23.860 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:44:23.860 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:43:50.456283+0000", 2024-02-09T17:44:23.860 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 145 2024-02-09T17:44:23.860 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.861 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.861 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 9, 2024-02-09T17:44:23.861 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:44:23.861 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:44:23.861 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:44:23.861 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:44:23.861 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:44:23.861 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:43:59.293380+0000", 2024-02-09T17:44:23.861 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:44:23.861 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.861 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.861 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 10, 2024-02-09T17:44:23.862 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:44:23.862 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:44:23.862 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:44:23.862 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:44:23.862 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:44:23.862 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:44:08.158872+0000", 2024-02-09T17:44:23.862 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:44:23.862 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.862 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.862 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 11, 2024-02-09T17:44:23.862 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:44:23.862 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:44:23.862 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:44:23.862 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:44:23.863 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:44:23.863 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:43:15.146250+0000", 2024-02-09T17:44:23.863 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:44:23.863 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.863 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T17:44:23.863 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_upmap": [], 2024-02-09T17:44:23.863 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_upmap_items": [], 2024-02-09T17:44:23.863 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_upmap_primaries": [], 2024-02-09T17:44:23.863 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_temp": [ 2024-02-09T17:44:23.863 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:23.863 INFO:teuthology.orchestra.run.smithi099.stdout: "pgid": "5.a", 2024-02-09T17:44:23.863 INFO:teuthology.orchestra.run.smithi099.stdout: "osds": [ 2024-02-09T17:44:23.863 INFO:teuthology.orchestra.run.smithi099.stdout: 8, 2024-02-09T17:44:23.863 INFO:teuthology.orchestra.run.smithi099.stdout: 10, 2024-02-09T17:44:23.864 INFO:teuthology.orchestra.run.smithi099.stdout: 4 2024-02-09T17:44:23.864 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:23.864 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.864 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T17:44:23.864 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_temp": [], 2024-02-09T17:44:23.864 INFO:teuthology.orchestra.run.smithi099.stdout: "blocklist": { 2024-02-09T17:44:23.864 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:0/3904172782": "2024-02-10T17:16:24.229365+0000", 2024-02-09T17:44:23.864 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:6801/1586034576": "2024-02-10T17:16:24.229365+0000", 2024-02-09T17:44:23.864 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:6800/746248609": "2024-02-10T17:15:54.399201+0000", 2024-02-09T17:44:23.864 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:0/4092784197": "2024-02-10T17:16:24.229365+0000", 2024-02-09T17:44:23.864 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:0/2844598101": "2024-02-10T17:15:54.399201+0000", 2024-02-09T17:44:23.864 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:6801/746248609": "2024-02-10T17:15:54.399201+0000", 2024-02-09T17:44:23.864 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:0/2528750050": "2024-02-10T17:15:54.399201+0000", 2024-02-09T17:44:23.865 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:6800/1586034576": "2024-02-10T17:16:24.229365+0000", 2024-02-09T17:44:23.865 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:0/2214140078": "2024-02-10T17:15:54.399201+0000", 2024-02-09T17:44:23.865 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:0/4217821965": "2024-02-10T17:16:24.229365+0000" 2024-02-09T17:44:23.865 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.865 INFO:teuthology.orchestra.run.smithi099.stdout: "range_blocklist": {}, 2024-02-09T17:44:23.865 INFO:teuthology.orchestra.run.smithi099.stdout: "erasure_code_profiles": { 2024-02-09T17:44:23.865 INFO:teuthology.orchestra.run.smithi099.stdout: "default": { 2024-02-09T17:44:23.865 INFO:teuthology.orchestra.run.smithi099.stdout: "crush-failure-domain": "osd", 2024-02-09T17:44:23.865 INFO:teuthology.orchestra.run.smithi099.stdout: "k": "2", 2024-02-09T17:44:23.865 INFO:teuthology.orchestra.run.smithi099.stdout: "m": "1", 2024-02-09T17:44:23.865 INFO:teuthology.orchestra.run.smithi099.stdout: "plugin": "jerasure", 2024-02-09T17:44:23.865 INFO:teuthology.orchestra.run.smithi099.stdout: "technique": "reed_sol_van" 2024-02-09T17:44:23.866 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.866 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:23.866 INFO:teuthology.orchestra.run.smithi099.stdout: "removed_snaps_queue": [], 2024-02-09T17:44:23.866 INFO:teuthology.orchestra.run.smithi099.stdout: "new_removed_snaps": [], 2024-02-09T17:44:23.866 INFO:teuthology.orchestra.run.smithi099.stdout: "new_purged_snaps": [], 2024-02-09T17:44:23.866 INFO:teuthology.orchestra.run.smithi099.stdout: "crush_node_flags": {}, 2024-02-09T17:44:23.866 INFO:teuthology.orchestra.run.smithi099.stdout: "device_class_flags": {}, 2024-02-09T17:44:23.866 INFO:teuthology.orchestra.run.smithi099.stdout: "stretch_mode": { 2024-02-09T17:44:23.866 INFO:teuthology.orchestra.run.smithi099.stdout: "stretch_mode_enabled": false, 2024-02-09T17:44:23.866 INFO:teuthology.orchestra.run.smithi099.stdout: "stretch_bucket_count": 0, 2024-02-09T17:44:23.866 INFO:teuthology.orchestra.run.smithi099.stdout: "degraded_stretch_mode": 0, 2024-02-09T17:44:23.866 INFO:teuthology.orchestra.run.smithi099.stdout: "recovering_stretch_mode": 0, 2024-02-09T17:44:23.866 INFO:teuthology.orchestra.run.smithi099.stdout: "stretch_mode_bucket": 0 2024-02-09T17:44:23.867 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:23.867 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:44:23.982 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:44:24.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:23 smithi099 ceph-mon[39420]: pgmap v1374: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 2.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 844 op/s; 2807/137769 objects misplaced (2.037%) 2024-02-09T17:44:24.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:23 smithi099 ceph-mon[39420]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:44:24.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:23 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:44:24.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:23 smithi099 ceph-mon[39420]: osdmap e546: 12 total, 12 up, 12 in 2024-02-09T17:44:24.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:23 smithi179 ceph-mon[43975]: pgmap v1374: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 2.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 844 op/s; 2807/137769 objects misplaced (2.037%) 2024-02-09T17:44:24.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:23 smithi179 ceph-mon[43975]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:44:24.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:23 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:44:24.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:23 smithi179 ceph-mon[43975]: osdmap e546: 12 total, 12 up, 12 in 2024-02-09T17:44:24.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:23 smithi162 ceph-mon[44304]: pgmap v1374: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 2.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 844 op/s; 2807/137769 objects misplaced (2.037%) 2024-02-09T17:44:24.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:23 smithi162 ceph-mon[44304]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:44:24.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:23 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:44:24.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:23 smithi162 ceph-mon[44304]: osdmap e546: 12 total, 12 up, 12 in 2024-02-09T17:44:24.492 INFO:tasks.thrashosds.thrasher:j is {'epoch': 545, 'fsid': 'a69c870a-c76e-11ee-95b7-87774f69a715', 'created': '2024-02-09T17:15:26.113606+0000', 'modified': '2024-02-09T17:44:10.833433+0000', 'last_up_change': '2024-02-09T17:28:17.711044+0000', 'last_in_change': '2024-02-09T17:43:24.138763+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 27, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 5, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-09T17:18:56.552286+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '544', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '514', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5499999523162842, 'score_stable': 1.5499999523162842, 'optimal_score': 1, 'raw_score_acting': 1.5499999523162842, 'raw_score_stable': 1.5499999523162842, 'primary_affinity_weighted': 0.8806711435317993, 'average_primary_affinity': 0.8799999952316284, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 5, 'pool_name': 'unique_pool_3', 'create_time': '2024-02-09T17:40:47.713102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '507', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.8806711435317993, 'average_primary_affinity': 0.8799999952316284, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': '21739aa7-e278-45da-9a3f-867385f01baf', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.609619140625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 531, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6802', 'nonce': 3859868923}, {'type': 'v1', 'addr': '172.21.15.99:6803', 'nonce': 3859868923}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6804', 'nonce': 3859868923}, {'type': 'v1', 'addr': '172.21.15.99:6805', 'nonce': 3859868923}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6808', 'nonce': 3859868923}, {'type': 'v1', 'addr': '172.21.15.99:6809', 'nonce': 3859868923}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6806', 'nonce': 3859868923}, {'type': 'v1', 'addr': '172.21.15.99:6807', 'nonce': 3859868923}]}, 'public_addr': '172.21.15.99:6803/3859868923', 'cluster_addr': '172.21.15.99:6805/3859868923', 'heartbeat_back_addr': '172.21.15.99:6809/3859868923', 'heartbeat_front_addr': '172.21.15.99:6807/3859868923', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '8eab347c-9065-4de1-93ae-e2eb7e11cab7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 525, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6810', 'nonce': 3868328452}, {'type': 'v1', 'addr': '172.21.15.99:6811', 'nonce': 3868328452}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6812', 'nonce': 3868328452}, {'type': 'v1', 'addr': '172.21.15.99:6813', 'nonce': 3868328452}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6816', 'nonce': 3868328452}, {'type': 'v1', 'addr': '172.21.15.99:6817', 'nonce': 3868328452}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6814', 'nonce': 3868328452}, {'type': 'v1', 'addr': '172.21.15.99:6815', 'nonce': 3868328452}]}, 'public_addr': '172.21.15.99:6811/3868328452', 'cluster_addr': '172.21.15.99:6813/3868328452', 'heartbeat_back_addr': '172.21.15.99:6817/3868328452', 'heartbeat_front_addr': '172.21.15.99:6815/3868328452', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '575f4257-7e34-485e-963c-2607b69e1220', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.3601837158203125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 537, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6818', 'nonce': 537506237}, {'type': 'v1', 'addr': '172.21.15.99:6819', 'nonce': 537506237}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6820', 'nonce': 537506237}, {'type': 'v1', 'addr': '172.21.15.99:6821', 'nonce': 537506237}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6824', 'nonce': 537506237}, {'type': 'v1', 'addr': '172.21.15.99:6825', 'nonce': 537506237}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6822', 'nonce': 537506237}, {'type': 'v1', 'addr': '172.21.15.99:6823', 'nonce': 537506237}]}, 'public_addr': '172.21.15.99:6819/537506237', 'cluster_addr': '172.21.15.99:6821/537506237', 'heartbeat_back_addr': '172.21.15.99:6825/537506237', 'heartbeat_front_addr': '172.21.15.99:6823/537506237', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '1a520bcf-81bb-4f1d-80fb-11e98008e429', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 518, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6826', 'nonce': 762784793}, {'type': 'v1', 'addr': '172.21.15.99:6827', 'nonce': 762784793}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6828', 'nonce': 762784793}, {'type': 'v1', 'addr': '172.21.15.99:6829', 'nonce': 762784793}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6832', 'nonce': 762784793}, {'type': 'v1', 'addr': '172.21.15.99:6833', 'nonce': 762784793}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6830', 'nonce': 762784793}, {'type': 'v1', 'addr': '172.21.15.99:6831', 'nonce': 762784793}]}, 'public_addr': '172.21.15.99:6827/762784793', 'cluster_addr': '172.21.15.99:6829/762784793', 'heartbeat_back_addr': '172.21.15.99:6833/762784793', 'heartbeat_front_addr': '172.21.15.99:6831/762784793', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '4617c2fe-c9d8-40cd-887b-303e3215bdf8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 519, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6800', 'nonce': 4178676504}, {'type': 'v1', 'addr': '172.21.15.162:6801', 'nonce': 4178676504}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6802', 'nonce': 4178676504}, {'type': 'v1', 'addr': '172.21.15.162:6803', 'nonce': 4178676504}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6806', 'nonce': 4178676504}, {'type': 'v1', 'addr': '172.21.15.162:6807', 'nonce': 4178676504}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6804', 'nonce': 4178676504}, {'type': 'v1', 'addr': '172.21.15.162:6805', 'nonce': 4178676504}]}, 'public_addr': '172.21.15.162:6801/4178676504', 'cluster_addr': '172.21.15.162:6803/4178676504', 'heartbeat_back_addr': '172.21.15.162:6807/4178676504', 'heartbeat_front_addr': '172.21.15.162:6805/4178676504', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'da98e3f3-173e-46fe-be8b-1ccbe8b0dabc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 536, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6808', 'nonce': 1489371339}, {'type': 'v1', 'addr': '172.21.15.162:6809', 'nonce': 1489371339}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6810', 'nonce': 1489371339}, {'type': 'v1', 'addr': '172.21.15.162:6811', 'nonce': 1489371339}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6814', 'nonce': 1489371339}, {'type': 'v1', 'addr': '172.21.15.162:6815', 'nonce': 1489371339}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6812', 'nonce': 1489371339}, {'type': 'v1', 'addr': '172.21.15.162:6813', 'nonce': 1489371339}]}, 'public_addr': '172.21.15.162:6809/1489371339', 'cluster_addr': '172.21.15.162:6811/1489371339', 'heartbeat_back_addr': '172.21.15.162:6815/1489371339', 'heartbeat_front_addr': '172.21.15.162:6813/1489371339', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'ff50382d-5ddf-4cda-bb0c-ff055feeaa40', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 49, 'up_thru': 518, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6816', 'nonce': 821169663}, {'type': 'v1', 'addr': '172.21.15.162:6817', 'nonce': 821169663}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6818', 'nonce': 821169663}, {'type': 'v1', 'addr': '172.21.15.162:6819', 'nonce': 821169663}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6822', 'nonce': 821169663}, {'type': 'v1', 'addr': '172.21.15.162:6823', 'nonce': 821169663}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6820', 'nonce': 821169663}, {'type': 'v1', 'addr': '172.21.15.162:6821', 'nonce': 821169663}]}, 'public_addr': '172.21.15.162:6817/821169663', 'cluster_addr': '172.21.15.162:6819/821169663', 'heartbeat_back_addr': '172.21.15.162:6823/821169663', 'heartbeat_front_addr': '172.21.15.162:6821/821169663', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'a82984dc-5e22-424b-a446-ae2ee71eee19', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.5982513427734375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 57, 'up_thru': 525, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6824', 'nonce': 965264369}, {'type': 'v1', 'addr': '172.21.15.162:6825', 'nonce': 965264369}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6826', 'nonce': 965264369}, {'type': 'v1', 'addr': '172.21.15.162:6827', 'nonce': 965264369}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6830', 'nonce': 965264369}, {'type': 'v1', 'addr': '172.21.15.162:6831', 'nonce': 965264369}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6828', 'nonce': 965264369}, {'type': 'v1', 'addr': '172.21.15.162:6829', 'nonce': 965264369}]}, 'public_addr': '172.21.15.162:6825/965264369', 'cluster_addr': '172.21.15.162:6827/965264369', 'heartbeat_back_addr': '172.21.15.162:6831/965264369', 'heartbeat_front_addr': '172.21.15.162:6829/965264369', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '63f9d79f-0400-44dc-b388-df00fefb5820', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 66, 'last_clean_end': 145, 'up_from': 171, 'up_thru': 522, 'down_at': 146, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6800', 'nonce': 2558294495}, {'type': 'v1', 'addr': '172.21.15.179:6801', 'nonce': 2558294495}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6802', 'nonce': 2558294495}, {'type': 'v1', 'addr': '172.21.15.179:6803', 'nonce': 2558294495}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6806', 'nonce': 2558294495}, {'type': 'v1', 'addr': '172.21.15.179:6807', 'nonce': 2558294495}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6804', 'nonce': 2558294495}, {'type': 'v1', 'addr': '172.21.15.179:6805', 'nonce': 2558294495}]}, 'public_addr': '172.21.15.179:6801/2558294495', 'cluster_addr': '172.21.15.179:6803/2558294495', 'heartbeat_back_addr': '172.21.15.179:6807/2558294495', 'heartbeat_front_addr': '172.21.15.179:6805/2558294495', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '7d995854-7328-4dfc-b1a9-0edca8d74f46', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 77, 'up_thru': 514, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6808', 'nonce': 504427183}, {'type': 'v1', 'addr': '172.21.15.179:6809', 'nonce': 504427183}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6810', 'nonce': 504427183}, {'type': 'v1', 'addr': '172.21.15.179:6811', 'nonce': 504427183}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6814', 'nonce': 504427183}, {'type': 'v1', 'addr': '172.21.15.179:6815', 'nonce': 504427183}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6812', 'nonce': 504427183}, {'type': 'v1', 'addr': '172.21.15.179:6813', 'nonce': 504427183}]}, 'public_addr': '172.21.15.179:6809/504427183', 'cluster_addr': '172.21.15.179:6811/504427183', 'heartbeat_back_addr': '172.21.15.179:6815/504427183', 'heartbeat_front_addr': '172.21.15.179:6813/504427183', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '151fbedd-08e6-46e6-a4e5-a14248399f61', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 84, 'up_thru': 535, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6816', 'nonce': 2403345164}, {'type': 'v1', 'addr': '172.21.15.179:6817', 'nonce': 2403345164}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6818', 'nonce': 2403345164}, {'type': 'v1', 'addr': '172.21.15.179:6819', 'nonce': 2403345164}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6822', 'nonce': 2403345164}, {'type': 'v1', 'addr': '172.21.15.179:6823', 'nonce': 2403345164}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6820', 'nonce': 2403345164}, {'type': 'v1', 'addr': '172.21.15.179:6821', 'nonce': 2403345164}]}, 'public_addr': '172.21.15.179:6817/2403345164', 'cluster_addr': '172.21.15.179:6819/2403345164', 'heartbeat_back_addr': '172.21.15.179:6823/2403345164', 'heartbeat_front_addr': '172.21.15.179:6821/2403345164', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '6d3a3ad8-417a-483f-991c-52549dbe7d73', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 90, 'up_thru': 519, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6824', 'nonce': 2899402373}, {'type': 'v1', 'addr': '172.21.15.179:6825', 'nonce': 2899402373}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6826', 'nonce': 2899402373}, {'type': 'v1', 'addr': '172.21.15.179:6827', 'nonce': 2899402373}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6830', 'nonce': 2899402373}, {'type': 'v1', 'addr': '172.21.15.179:6831', 'nonce': 2899402373}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6828', 'nonce': 2899402373}, {'type': 'v1', 'addr': '172.21.15.179:6829', 'nonce': 2899402373}]}, 'public_addr': '172.21.15.179:6825/2899402373', 'cluster_addr': '172.21.15.179:6827/2899402373', 'heartbeat_back_addr': '172.21.15.179:6831/2899402373', 'heartbeat_front_addr': '172.21.15.179:6829/2899402373', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:43:58.666762+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:43:29.854888+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:43:50.747285+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:43:06.026036+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:44:04.366013+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:43:59.000265+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:43:37.551816+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:43:02.368002+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:43:50.456283+0000', 'dead_epoch': 145}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:43:59.293380+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:44:08.158872+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:43:15.146250+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '5.a', 'osds': [8, 10, 4]}], 'primary_temp': [], 'blocklist': {'172.21.15.99:0/3904172782': '2024-02-10T17:16:24.229365+0000', '172.21.15.99:6801/1586034576': '2024-02-10T17:16:24.229365+0000', '172.21.15.99:6800/746248609': '2024-02-10T17:15:54.399201+0000', '172.21.15.99:0/4092784197': '2024-02-10T17:16:24.229365+0000', '172.21.15.99:0/2844598101': '2024-02-10T17:15:54.399201+0000', '172.21.15.99:6801/746248609': '2024-02-10T17:15:54.399201+0000', '172.21.15.99:0/2528750050': '2024-02-10T17:15:54.399201+0000', '172.21.15.99:6800/1586034576': '2024-02-10T17:16:24.229365+0000', '172.21.15.99:0/2214140078': '2024-02-10T17:15:54.399201+0000', '172.21.15.99:0/4217821965': '2024-02-10T17:16:24.229365+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-09T17:44:24.493 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph pg dump --format=json 2024-02-09T17:44:25.939 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:25 smithi099 ceph-mon[39420]: 5.9 scrub starts 2024-02-09T17:44:25.939 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:25 smithi099 ceph-mon[39420]: pgmap v1376: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 872 op/s; 2807/133092 objects misplaced (2.109%) 2024-02-09T17:44:25.939 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:25 smithi099 ceph-mon[39420]: 1.b scrub starts 2024-02-09T17:44:26.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:25 smithi179 ceph-mon[43975]: 5.9 scrub starts 2024-02-09T17:44:26.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:25 smithi179 ceph-mon[43975]: pgmap v1376: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 872 op/s; 2807/133092 objects misplaced (2.109%) 2024-02-09T17:44:26.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:25 smithi179 ceph-mon[43975]: 1.b scrub starts 2024-02-09T17:44:26.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:25 smithi162 ceph-mon[44304]: 5.9 scrub starts 2024-02-09T17:44:26.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:25 smithi162 ceph-mon[44304]: pgmap v1376: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 872 op/s; 2807/133092 objects misplaced (2.109%) 2024-02-09T17:44:26.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:25 smithi162 ceph-mon[44304]: 1.b scrub starts 2024-02-09T17:44:27.248 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:26 smithi099 ceph-mon[39420]: 1.b scrub ok 2024-02-09T17:44:27.248 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:26 smithi099 ceph-mon[39420]: 1.e scrub starts 2024-02-09T17:44:27.248 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:26 smithi099 ceph-mon[39420]: 1.e scrub ok 2024-02-09T17:44:27.248 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:26 smithi099 ceph-mon[39420]: 1.14 scrub starts 2024-02-09T17:44:27.248 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:26 smithi099 ceph-mon[39420]: 5.9 scrub ok 2024-02-09T17:44:27.248 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:26 smithi099 ceph-mon[39420]: 1.14 scrub ok 2024-02-09T17:44:27.248 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:26 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:44:27.248 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:26 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:44:27.249 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:26 smithi099 ceph-mon[39420]: osdmap e547: 12 total, 12 up, 12 in 2024-02-09T17:44:27.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:26 smithi179 ceph-mon[43975]: 1.b scrub ok 2024-02-09T17:44:27.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:26 smithi179 ceph-mon[43975]: 1.e scrub starts 2024-02-09T17:44:27.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:26 smithi179 ceph-mon[43975]: 1.e scrub ok 2024-02-09T17:44:27.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:26 smithi179 ceph-mon[43975]: 1.14 scrub starts 2024-02-09T17:44:27.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:26 smithi179 ceph-mon[43975]: 5.9 scrub ok 2024-02-09T17:44:27.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:26 smithi179 ceph-mon[43975]: 1.14 scrub ok 2024-02-09T17:44:27.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:26 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:44:27.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:26 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:44:27.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:26 smithi179 ceph-mon[43975]: osdmap e547: 12 total, 12 up, 12 in 2024-02-09T17:44:27.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:26 smithi162 ceph-mon[44304]: 1.b scrub ok 2024-02-09T17:44:27.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:26 smithi162 ceph-mon[44304]: 1.e scrub starts 2024-02-09T17:44:27.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:26 smithi162 ceph-mon[44304]: 1.e scrub ok 2024-02-09T17:44:27.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:26 smithi162 ceph-mon[44304]: 1.14 scrub starts 2024-02-09T17:44:27.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:26 smithi162 ceph-mon[44304]: 5.9 scrub ok 2024-02-09T17:44:27.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:26 smithi162 ceph-mon[44304]: 1.14 scrub ok 2024-02-09T17:44:27.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:26 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:44:27.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:26 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:44:27.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:26 smithi162 ceph-mon[44304]: osdmap e547: 12 total, 12 up, 12 in 2024-02-09T17:44:27.662 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:44:27.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:27.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:44:27.671 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:44:27.671 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:44:27.770 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:44:27.814 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:27.840 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:44:27.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:27.848 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:44:27.949 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:44:27.975 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:27.995 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:44:27.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:27.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:44:28.099 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:44:28.143 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:27 smithi179 ceph-mon[43975]: pgmap v1378: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 949 op/s; 2807/126726 objects misplaced (2.215%) 2024-02-09T17:44:28.143 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:27 smithi179 ceph-mon[43975]: 1.13 scrub starts 2024-02-09T17:44:28.143 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:27 smithi179 ceph-mon[43975]: 1.13 scrub ok 2024-02-09T17:44:28.147 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:28.169 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:44:28.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:28.176 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:44:28.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:27 smithi099 ceph-mon[39420]: pgmap v1378: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 949 op/s; 2807/126726 objects misplaced (2.215%) 2024-02-09T17:44:28.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:27 smithi099 ceph-mon[39420]: 1.13 scrub starts 2024-02-09T17:44:28.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:27 smithi099 ceph-mon[39420]: 1.13 scrub ok 2024-02-09T17:44:28.278 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:44:28.309 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:28.341 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:44:28.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:28.346 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:44:28.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:27 smithi162 ceph-mon[44304]: pgmap v1378: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 949 op/s; 2807/126726 objects misplaced (2.215%) 2024-02-09T17:44:28.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:27 smithi162 ceph-mon[44304]: 1.13 scrub starts 2024-02-09T17:44:28.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:27 smithi162 ceph-mon[44304]: 1.13 scrub ok 2024-02-09T17:44:28.446 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:44:28.471 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:44:28.471 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:44:28.487 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:44:28.490 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:28.498 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:44:28.540 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:44:28.573 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:44:28.603 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:44:28.632 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:44:28.659 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:44:28.698 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:44:28.733 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:44:28.776 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:44:28.816 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:44:28.842 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:44:28.842 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:44:28.842 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:44:28.842 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:44:28.842 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:44:28.842 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:44:28.842 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:44:28.842 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:44:28.842 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:44:28.842 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:44:28.842 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:44:28.842 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:44:28.848 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:44:28.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:28.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:44:28.881 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-09T17:44:28.956 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:44:28.993 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:29.030 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:44:29.036 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:29.036 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:44:29.137 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:44:29.174 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:28 smithi099 ceph-mon[39420]: 1.0 scrub starts 2024-02-09T17:44:29.174 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:28 smithi099 ceph-mon[39420]: 1.0 scrub ok 2024-02-09T17:44:29.174 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:28 smithi099 ceph-mon[39420]: 1.a scrub starts 2024-02-09T17:44:29.174 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:28 smithi099 ceph-mon[39420]: 1.a scrub ok 2024-02-09T17:44:29.175 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:28 smithi099 ceph-mon[39420]: 1.c scrub starts 2024-02-09T17:44:29.175 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:28 smithi099 ceph-mon[39420]: 1.c scrub ok 2024-02-09T17:44:29.175 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:28 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/711718583' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:44:29.180 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:29.310 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:44:29.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:29.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:44:29.384 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-09T17:44:29.415 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:44:29.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:28 smithi179 ceph-mon[43975]: 1.0 scrub starts 2024-02-09T17:44:29.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:28 smithi179 ceph-mon[43975]: 1.0 scrub ok 2024-02-09T17:44:29.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:28 smithi179 ceph-mon[43975]: 1.a scrub starts 2024-02-09T17:44:29.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:28 smithi179 ceph-mon[43975]: 1.a scrub ok 2024-02-09T17:44:29.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:28 smithi179 ceph-mon[43975]: 1.c scrub starts 2024-02-09T17:44:29.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:28 smithi179 ceph-mon[43975]: 1.c scrub ok 2024-02-09T17:44:29.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:28 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/711718583' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:44:29.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:28 smithi162 ceph-mon[44304]: 1.0 scrub starts 2024-02-09T17:44:29.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:28 smithi162 ceph-mon[44304]: 1.0 scrub ok 2024-02-09T17:44:29.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:28 smithi162 ceph-mon[44304]: 1.a scrub starts 2024-02-09T17:44:29.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:28 smithi162 ceph-mon[44304]: 1.a scrub ok 2024-02-09T17:44:29.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:28 smithi162 ceph-mon[44304]: 1.c scrub starts 2024-02-09T17:44:29.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:28 smithi162 ceph-mon[44304]: 1.c scrub ok 2024-02-09T17:44:29.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:28 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/711718583' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:44:29.461 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:29.494 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:44:29.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:29.502 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:44:29.603 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:44:29.652 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:29.675 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:44:29.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:29.679 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:44:29.780 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:44:29.868 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:29.895 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:44:29.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:29.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:44:30.001 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:44:30.060 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:30.108 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:44:30.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:30.116 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:44:30.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:29 smithi162 ceph-mon[44304]: 5.4 scrub starts 2024-02-09T17:44:30.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:29 smithi162 ceph-mon[44304]: 5.4 scrub ok 2024-02-09T17:44:30.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:29 smithi162 ceph-mon[44304]: pgmap v1379: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 845 op/s; 2527/123360 objects misplaced (2.048%); 88 KiB/s, 1 objects/s recovering 2024-02-09T17:44:30.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:29 smithi162 ceph-mon[44304]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-09T17:44:30.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:29 smithi162 ceph-mon[44304]: 5.d scrub starts 2024-02-09T17:44:30.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:29 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:44:30.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:29 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:44:30.217 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:44:30.243 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:30.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:29 smithi099 ceph-mon[39420]: 5.4 scrub starts 2024-02-09T17:44:30.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:30 smithi099 ceph-mon[39420]: 5.4 scrub ok 2024-02-09T17:44:30.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:30 smithi099 ceph-mon[39420]: pgmap v1379: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 845 op/s; 2527/123360 objects misplaced (2.048%); 88 KiB/s, 1 objects/s recovering 2024-02-09T17:44:30.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:30 smithi099 ceph-mon[39420]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-09T17:44:30.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:30 smithi099 ceph-mon[39420]: 5.d scrub starts 2024-02-09T17:44:30.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:30 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:44:30.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:30 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:44:30.264 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:44:30.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:30.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:44:30.370 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:44:30.401 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:30.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:29 smithi179 ceph-mon[43975]: 5.4 scrub starts 2024-02-09T17:44:30.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:29 smithi179 ceph-mon[43975]: 5.4 scrub ok 2024-02-09T17:44:30.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:29 smithi179 ceph-mon[43975]: pgmap v1379: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 845 op/s; 2527/123360 objects misplaced (2.048%); 88 KiB/s, 1 objects/s recovering 2024-02-09T17:44:30.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:29 smithi179 ceph-mon[43975]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-09T17:44:30.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:29 smithi179 ceph-mon[43975]: 5.d scrub starts 2024-02-09T17:44:30.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:29 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:44:30.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:29 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:44:30.905 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:44:30.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:30.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:44:31.013 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:44:31.059 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:31.218 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:44:31.221 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:31 smithi099 ceph-mon[39420]: 5.d scrub ok 2024-02-09T17:44:31.221 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:31 smithi099 ceph-mon[39420]: osdmap e548: 12 total, 12 up, 12 in 2024-02-09T17:44:31.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:31.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:44:31.323 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:44:31.372 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:31.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:31 smithi179 ceph-mon[43975]: 5.d scrub ok 2024-02-09T17:44:31.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:31 smithi179 ceph-mon[43975]: osdmap e548: 12 total, 12 up, 12 in 2024-02-09T17:44:31.431 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:31 smithi162 ceph-mon[44304]: 5.d scrub ok 2024-02-09T17:44:31.431 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:31 smithi162 ceph-mon[44304]: osdmap e548: 12 total, 12 up, 12 in 2024-02-09T17:44:31.924 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:44:31.926 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:44:31.936 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:44:31.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:31.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:44:32.043 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:44:32.093 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:32.122 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:44:32.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:32.129 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:44:32.212 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:32 smithi099 ceph-mon[39420]: 5.7 scrub starts 2024-02-09T17:44:32.212 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:32 smithi099 ceph-mon[39420]: 1.d scrub starts 2024-02-09T17:44:32.212 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:32 smithi099 ceph-mon[39420]: 1.d scrub ok 2024-02-09T17:44:32.212 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:32 smithi099 ceph-mon[39420]: pgmap v1381: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 2.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 834 op/s; 2527/117870 objects misplaced (2.144%); 79 KiB/s, 1 objects/s recovering 2024-02-09T17:44:32.213 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:32 smithi099 ceph-mon[39420]: 5.7 scrub ok 2024-02-09T17:44:32.213 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:32 smithi099 ceph-mon[39420]: osdmap e549: 12 total, 12 up, 12 in 2024-02-09T17:44:32.230 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:44:32.287 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:32.324 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:44:32.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:32.332 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:44:32.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:32 smithi179 ceph-mon[43975]: 5.7 scrub starts 2024-02-09T17:44:32.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:32 smithi179 ceph-mon[43975]: 1.d scrub starts 2024-02-09T17:44:32.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:32 smithi179 ceph-mon[43975]: 1.d scrub ok 2024-02-09T17:44:32.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:32 smithi179 ceph-mon[43975]: pgmap v1381: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 2.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 834 op/s; 2527/117870 objects misplaced (2.144%); 79 KiB/s, 1 objects/s recovering 2024-02-09T17:44:32.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:32 smithi179 ceph-mon[43975]: 5.7 scrub ok 2024-02-09T17:44:32.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:32 smithi179 ceph-mon[43975]: osdmap e549: 12 total, 12 up, 12 in 2024-02-09T17:44:32.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:32 smithi162 ceph-mon[44304]: 5.7 scrub starts 2024-02-09T17:44:32.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:32 smithi162 ceph-mon[44304]: 1.d scrub starts 2024-02-09T17:44:32.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:32 smithi162 ceph-mon[44304]: 1.d scrub ok 2024-02-09T17:44:32.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:32 smithi162 ceph-mon[44304]: pgmap v1381: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 2.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 834 op/s; 2527/117870 objects misplaced (2.144%); 79 KiB/s, 1 objects/s recovering 2024-02-09T17:44:32.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:32 smithi162 ceph-mon[44304]: 5.7 scrub ok 2024-02-09T17:44:32.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:32 smithi162 ceph-mon[44304]: osdmap e549: 12 total, 12 up, 12 in 2024-02-09T17:44:32.433 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:44:32.465 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:32.497 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:44:32.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:32.504 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:44:32.605 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:44:32.657 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:32.678 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:44:32.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:32.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:44:32.784 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:44:32.832 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:32.859 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:44:32.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:32.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:44:32.941 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:44:32.964 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:44:33.026 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:33.065 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:44:33.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:33.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:44:33.171 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:44:33.227 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:33.277 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:44:33.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:33.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:44:33.391 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:44:33.432 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:33.459 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:44:33.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:33.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:44:33.569 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:44:33.614 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:33.641 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:44:33.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:33.645 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:44:33.746 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:44:33.789 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:33.823 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:44:33.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:33.827 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:44:33.928 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:44:33.990 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:34.021 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:44:34.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:34.030 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:44:34.060 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:44:34.063 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-02-09T17:44:34.130 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:44:34.177 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:34.337 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:44:34.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:34.346 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:34 smithi099 ceph-mon[39420]: pgmap v1383: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 2.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 832 op/s; 1300/112524 objects misplaced (1.155%); 10 MiB/s, 162 objects/s recovering 2024-02-09T17:44:34.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:44:34.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:34 smithi179 ceph-mon[43975]: pgmap v1383: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 2.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 832 op/s; 1300/112524 objects misplaced (1.155%); 10 MiB/s, 162 objects/s recovering 2024-02-09T17:44:34.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:34 smithi162 ceph-mon[44304]: pgmap v1383: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 2.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 832 op/s; 1300/112524 objects misplaced (1.155%); 10 MiB/s, 162 objects/s recovering 2024-02-09T17:44:34.447 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:44:34.497 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:34.538 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:44:34.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:34.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:44:34.645 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:44:34.690 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:34.717 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:44:34.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:34.726 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:44:34.827 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:44:34.866 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:34.905 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:44:34.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:34.918 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:44:34.921 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":1383,"stamp":"2024-02-09T17:44:32.856597+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2458518072,"num_objects":37508,"num_object_clones":0,"num_object_copies":112524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1300,"num_objects_unfound":0,"num_objects_dirty":37508,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":220302,"num_write_kb":8249793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13978,"num_bytes_recovered":916062208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":165273,"ondisk_log_size":165273,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":12855744,"kb_used_data":7357000,"kb_used_omap":229,"kb_used_meta":5498330,"kb_avail":1112086080,"statfs":{"total":1151940427776,"available":1138776145920,"internally_reserved":0,"allocated":7533568000,"data_stored":7528486027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235127,"internal_metadata":5630290313},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,7,5,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-344212962,"num_objects":-5252,"num_object_clones":0,"num_object_copies":-15756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1507,"num_objects_unfound":0,"num_objects_dirty":-5252,"num_whiteouts":0,"num_read":-4,"num_read_kb":-3,"num_write":5246,"num_write_kb":-89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1022,"num_bytes_recovered":66897651,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.302009"},"pg_stats":[{"pgid":"1.1e","version":"63'139","reported_seq":326,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-02-09T17:44:30.027977+0000","last_change":"2024-02-09T17:43:35.170863+0000","last_active":"2024-02-09T17:44:30.027977+0000","last_peered":"2024-02-09T17:44:30.027977+0000","last_clean":"2024-02-09T17:44:30.027977+0000","last_became_active":"2024-02-09T17:43:19.126757+0000","last_became_peered":"2024-02-09T17:43:19.126757+0000","last_unstale":"2024-02-09T17:44:30.027977+0000","last_undegraded":"2024-02-09T17:44:30.027977+0000","last_fullsized":"2024-02-09T17:44:30.027977+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:35.170716+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:43:35.170716+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:04.300669+0000","scrub_duration":73,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"63'139","reported_seq":349,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-09T17:44:26.562449+0000","last_change":"2024-02-09T17:43:35.227956+0000","last_active":"2024-02-09T17:44:26.562449+0000","last_peered":"2024-02-09T17:44:26.562449+0000","last_clean":"2024-02-09T17:44:26.562449+0000","last_became_active":"2024-02-09T17:43:28.594296+0000","last_became_peered":"2024-02-09T17:43:28.594296+0000","last_unstale":"2024-02-09T17:44:26.562449+0000","last_undegraded":"2024-02-09T17:44:26.562449+0000","last_fullsized":"2024-02-09T17:44:26.562449+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:35.227843+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:43:35.227843+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:52.776919+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"63'139","reported_seq":866,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-02-09T17:44:31.566636+0000","last_change":"2024-02-09T17:43:35.698701+0000","last_active":"2024-02-09T17:44:31.566636+0000","last_peered":"2024-02-09T17:44:31.566636+0000","last_clean":"2024-02-09T17:44:31.566636+0000","last_became_active":"2024-02-09T17:43:19.103584+0000","last_became_peered":"2024-02-09T17:43:19.103584+0000","last_unstale":"2024-02-09T17:44:31.566636+0000","last_undegraded":"2024-02-09T17:44:31.566636+0000","last_fullsized":"2024-02-09T17:44:31.566636+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:35.698541+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:23.912504+0000","last_clean_scrub_stamp":"2024-02-09T17:43:35.698541+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:47.100243+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"63'139","reported_seq":664,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-02-09T17:44:31.563088+0000","last_change":"2024-02-09T17:43:34.746841+0000","last_active":"2024-02-09T17:44:31.563088+0000","last_peered":"2024-02-09T17:44:31.563088+0000","last_clean":"2024-02-09T17:44:31.563088+0000","last_became_active":"2024-02-09T17:43:19.102729+0000","last_became_peered":"2024-02-09T17:43:19.102729+0000","last_unstale":"2024-02-09T17:44:31.563088+0000","last_undegraded":"2024-02-09T17:44:31.563088+0000","last_fullsized":"2024-02-09T17:44:31.563088+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:34.746704+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:43:34.746704+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:43.557546+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"63'139","reported_seq":331,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-02-09T17:44:30.027132+0000","last_change":"2024-02-09T17:43:35.019518+0000","last_active":"2024-02-09T17:44:30.027132+0000","last_peered":"2024-02-09T17:44:30.027132+0000","last_clean":"2024-02-09T17:44:30.027132+0000","last_became_active":"2024-02-09T17:43:19.120814+0000","last_became_peered":"2024-02-09T17:43:19.120814+0000","last_unstale":"2024-02-09T17:44:30.027132+0000","last_undegraded":"2024-02-09T17:44:30.027132+0000","last_fullsized":"2024-02-09T17:44:30.027132+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:35.019416+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:43:35.019416+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:04.369513+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"63'139","reported_seq":1158,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-09T17:44:26.560888+0000","last_change":"2024-02-09T17:43:53.009587+0000","last_active":"2024-02-09T17:44:26.560888+0000","last_peered":"2024-02-09T17:44:26.560888+0000","last_clean":"2024-02-09T17:44:26.560888+0000","last_became_active":"2024-02-09T17:43:19.110535+0000","last_became_peered":"2024-02-09T17:43:19.110535+0000","last_unstale":"2024-02-09T17:44:26.560888+0000","last_undegraded":"2024-02-09T17:44:26.560888+0000","last_fullsized":"2024-02-09T17:44:26.560888+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:53.009455+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:30.193194+0000","last_clean_scrub_stamp":"2024-02-09T17:43:53.009455+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:21.095174+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"63'139","reported_seq":996,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-09T17:44:26.570860+0000","last_change":"2024-02-09T17:43:57.370069+0000","last_active":"2024-02-09T17:44:26.570860+0000","last_peered":"2024-02-09T17:44:26.570860+0000","last_clean":"2024-02-09T17:44:26.570860+0000","last_became_active":"2024-02-09T17:43:28.585573+0000","last_became_peered":"2024-02-09T17:43:28.585573+0000","last_unstale":"2024-02-09T17:44:26.570860+0000","last_undegraded":"2024-02-09T17:44:26.570860+0000","last_fullsized":"2024-02-09T17:44:26.570860+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:57.369961+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:43:57.369961+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:03.250398+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"63'139","reported_seq":438,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-02-09T17:44:31.563211+0000","last_change":"2024-02-09T17:43:36.732746+0000","last_active":"2024-02-09T17:44:31.563211+0000","last_peered":"2024-02-09T17:44:31.563211+0000","last_clean":"2024-02-09T17:44:31.563211+0000","last_became_active":"2024-02-09T17:43:19.109864+0000","last_became_peered":"2024-02-09T17:43:19.109864+0000","last_unstale":"2024-02-09T17:44:31.563211+0000","last_undegraded":"2024-02-09T17:44:31.563211+0000","last_fullsized":"2024-02-09T17:44:31.563211+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:36.732627+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:43:36.732627+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:49.217779+0000","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"63'139","reported_seq":324,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-09T17:44:26.562295+0000","last_change":"2024-02-09T17:43:36.150824+0000","last_active":"2024-02-09T17:44:26.562295+0000","last_peered":"2024-02-09T17:44:26.562295+0000","last_clean":"2024-02-09T17:44:26.562295+0000","last_became_active":"2024-02-09T17:43:19.094230+0000","last_became_peered":"2024-02-09T17:43:19.094230+0000","last_unstale":"2024-02-09T17:44:26.562295+0000","last_undegraded":"2024-02-09T17:44:26.562295+0000","last_fullsized":"2024-02-09T17:44:26.562295+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:36.150728+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:11.970338+0000","last_clean_scrub_stamp":"2024-02-09T17:43:36.150728+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:42.962981+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"63'139","reported_seq":782,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-02-09T17:44:31.587304+0000","last_change":"2024-02-09T17:43:35.861836+0000","last_active":"2024-02-09T17:44:31.587304+0000","last_peered":"2024-02-09T17:44:31.587304+0000","last_clean":"2024-02-09T17:44:31.587304+0000","last_became_active":"2024-02-09T17:43:19.138398+0000","last_became_peered":"2024-02-09T17:43:19.138398+0000","last_unstale":"2024-02-09T17:44:31.587304+0000","last_undegraded":"2024-02-09T17:44:31.587304+0000","last_fullsized":"2024-02-09T17:44:31.587304+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:35.861750+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:43:35.861750+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:36.200882+0000","scrub_duration":154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"63'139","reported_seq":828,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-02-09T17:44:30.024522+0000","last_change":"2024-02-09T17:44:25.672184+0000","last_active":"2024-02-09T17:44:30.024522+0000","last_peered":"2024-02-09T17:44:30.024522+0000","last_clean":"2024-02-09T17:44:30.024522+0000","last_became_active":"2024-02-09T17:43:28.589307+0000","last_became_peered":"2024-02-09T17:43:28.589307+0000","last_unstale":"2024-02-09T17:44:30.024522+0000","last_undegraded":"2024-02-09T17:44:30.024522+0000","last_fullsized":"2024-02-09T17:44:30.024522+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:25.672084+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:38:26.968892+0000","last_clean_scrub_stamp":"2024-02-09T17:44:25.672084+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:44.241932+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.4","version":"547'15788","reported_seq":25678,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-09T17:44:28.075224+0000","last_change":"2024-02-09T17:44:10.930524+0000","last_active":"2024-02-09T17:44:28.075224+0000","last_peered":"2024-02-09T17:44:28.075224+0000","last_clean":"2024-02-09T17:44:28.075224+0000","last_became_active":"2024-02-09T17:43:52.557816+0000","last_became_peered":"2024-02-09T17:43:52.557816+0000","last_unstale":"2024-02-09T17:44:28.075224+0000","last_undegraded":"2024-02-09T17:44:28.075224+0000","last_fullsized":"2024-02-09T17:44:28.075224+0000","mapping_epoch":535,"log_start":"494'5700","ondisk_log_start":"494'5700","created":465,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'7928","last_scrub_stamp":"2024-02-09T17:42:35.239746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:42:35.239746+0000","objects_scrubbed":2401,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1665,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160104448,"num_objects":2443,"num_object_clones":0,"num_object_copies":7329,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13387,"num_write_kb":506560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4756,"num_bytes_recovered":311689216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,5],"acting":[10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"63'139","reported_seq":1430,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-02-09T17:44:31.587555+0000","last_change":"2024-02-09T17:44:27.625779+0000","last_active":"2024-02-09T17:44:31.587555+0000","last_peered":"2024-02-09T17:44:31.587555+0000","last_clean":"2024-02-09T17:44:31.587555+0000","last_became_active":"2024-02-09T17:41:14.521079+0000","last_became_peered":"2024-02-09T17:41:14.521079+0000","last_unstale":"2024-02-09T17:44:31.587555+0000","last_undegraded":"2024-02-09T17:44:31.587555+0000","last_fullsized":"2024-02-09T17:44:31.587555+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:27.625665+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:44:27.625665+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:44.041378+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"548'13690","reported_seq":23166,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-02-09T17:44:30.336331+0000","last_change":"2024-02-09T17:43:57.644228+0000","last_active":"2024-02-09T17:44:30.336331+0000","last_peered":"2024-02-09T17:44:30.336331+0000","last_clean":"2024-02-09T17:44:30.336331+0000","last_became_active":"2024-02-09T17:43:56.598131+0000","last_became_peered":"2024-02-09T17:43:56.598131+0000","last_unstale":"2024-02-09T17:44:30.336331+0000","last_undegraded":"2024-02-09T17:44:30.336331+0000","last_fullsized":"2024-02-09T17:44:30.336331+0000","mapping_epoch":537,"log_start":"486'3600","ondisk_log_start":"486'3600","created":465,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'11981","last_scrub_stamp":"2024-02-09T17:43:57.644124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:43:57.644124+0000","objects_scrubbed":3961,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:19.331266+0000","scrub_duration":966,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146145280,"num_objects":2230,"num_object_clones":0,"num_object_copies":6690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13690,"num_write_kb":509440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4575,"num_bytes_recovered":299827200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"63'139","reported_seq":985,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-09T17:44:26.560022+0000","last_change":"2024-02-09T17:43:39.981380+0000","last_active":"2024-02-09T17:44:26.560022+0000","last_peered":"2024-02-09T17:44:26.560022+0000","last_clean":"2024-02-09T17:44:26.560022+0000","last_became_active":"2024-02-09T17:43:39.981054+0000","last_became_peered":"2024-02-09T17:43:39.981054+0000","last_unstale":"2024-02-09T17:44:26.560022+0000","last_undegraded":"2024-02-09T17:44:26.560022+0000","last_fullsized":"2024-02-09T17:44:26.560022+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:06.666752+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:43:06.666752+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:36.322472+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"547'13575","reported_seq":13838,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-09T17:44:29.430306+0000","last_change":"2024-02-09T17:42:39.441680+0000","last_active":"2024-02-09T17:44:29.430306+0000","last_peered":"2024-02-09T17:44:29.430306+0000","last_clean":"2024-02-09T17:44:29.430306+0000","last_became_active":"2024-02-09T17:40:48.781472+0000","last_became_peered":"2024-02-09T17:40:48.781472+0000","last_unstale":"2024-02-09T17:44:29.430306+0000","last_undegraded":"2024-02-09T17:44:29.430306+0000","last_fullsized":"2024-02-09T17:44:29.430306+0000","mapping_epoch":465,"log_start":"486'3500","ondisk_log_start":"486'3500","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"503'8191","last_scrub_stamp":"2024-02-09T17:42:39.441525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:42:39.441525+0000","objects_scrubbed":7816,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2957,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155516928,"num_objects":2373,"num_object_clones":0,"num_object_copies":7119,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13575,"num_write_kb":510336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"63'139","reported_seq":608,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-09T17:44:26.562216+0000","last_change":"2024-02-09T17:43:53.218894+0000","last_active":"2024-02-09T17:44:26.562216+0000","last_peered":"2024-02-09T17:44:26.562216+0000","last_clean":"2024-02-09T17:44:26.562216+0000","last_became_active":"2024-02-09T17:40:13.501711+0000","last_became_peered":"2024-02-09T17:40:13.501711+0000","last_unstale":"2024-02-09T17:44:26.562216+0000","last_undegraded":"2024-02-09T17:44:26.562216+0000","last_fullsized":"2024-02-09T17:44:26.562216+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:53.218764+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:38:27.045816+0000","last_clean_scrub_stamp":"2024-02-09T17:43:53.218764+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:20.464157+0000","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"547'13563","reported_seq":13880,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-09T17:44:28.939572+0000","last_change":"2024-02-09T17:43:37.166310+0000","last_active":"2024-02-09T17:44:28.939572+0000","last_peered":"2024-02-09T17:44:28.939572+0000","last_clean":"2024-02-09T17:44:28.939572+0000","last_became_active":"2024-02-09T17:40:48.780579+0000","last_became_peered":"2024-02-09T17:40:48.780579+0000","last_unstale":"2024-02-09T17:44:28.939572+0000","last_undegraded":"2024-02-09T17:44:28.939572+0000","last_fullsized":"2024-02-09T17:44:28.939572+0000","mapping_epoch":465,"log_start":"486'3500","ondisk_log_start":"486'3500","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"529'11051","last_scrub_stamp":"2024-02-09T17:43:37.166082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:43:37.166082+0000","objects_scrubbed":4984,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:53.401985+0000","scrub_duration":1034,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160628736,"num_objects":2451,"num_object_clones":0,"num_object_copies":7353,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13563,"num_write_kb":512448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"63'139","reported_seq":924,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-09T17:44:26.560993+0000","last_change":"2024-02-09T17:43:54.988847+0000","last_active":"2024-02-09T17:44:26.560993+0000","last_peered":"2024-02-09T17:44:26.560993+0000","last_clean":"2024-02-09T17:44:26.560993+0000","last_became_active":"2024-02-09T17:34:01.342607+0000","last_became_peered":"2024-02-09T17:34:01.342607+0000","last_unstale":"2024-02-09T17:44:26.560993+0000","last_undegraded":"2024-02-09T17:44:26.560993+0000","last_fullsized":"2024-02-09T17:44:26.560993+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:54.988700+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:43:54.988700+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:58.197783+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.2","version":"548'13998","reported_seq":14321,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-02-09T17:44:31.014656+0000","last_change":"2024-02-09T17:43:46.878627+0000","last_active":"2024-02-09T17:44:31.014656+0000","last_peered":"2024-02-09T17:44:31.014656+0000","last_clean":"2024-02-09T17:44:31.014656+0000","last_became_active":"2024-02-09T17:40:48.781532+0000","last_became_peered":"2024-02-09T17:40:48.781532+0000","last_unstale":"2024-02-09T17:44:31.014656+0000","last_undegraded":"2024-02-09T17:44:31.014656+0000","last_fullsized":"2024-02-09T17:44:31.014656+0000","mapping_epoch":465,"log_start":"486'3900","ondisk_log_start":"486'3900","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"533'11868","last_scrub_stamp":"2024-02-09T17:43:46.878486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:43:46.878486+0000","objects_scrubbed":4573,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:59.991875+0000","scrub_duration":1380,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157417472,"num_objects":2402,"num_object_clones":0,"num_object_copies":7206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13998,"num_write_kb":524800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"63'139","reported_seq":928,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-09T17:44:26.560961+0000","last_change":"2024-02-09T17:43:14.923680+0000","last_active":"2024-02-09T17:44:26.560961+0000","last_peered":"2024-02-09T17:44:26.560961+0000","last_clean":"2024-02-09T17:44:26.560961+0000","last_became_active":"2024-02-09T17:43:14.024894+0000","last_became_peered":"2024-02-09T17:43:14.024894+0000","last_unstale":"2024-02-09T17:44:26.560961+0000","last_undegraded":"2024-02-09T17:44:26.560961+0000","last_fullsized":"2024-02-09T17:44:26.560961+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:14.923533+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:11.970338+0000","last_clean_scrub_stamp":"2024-02-09T17:43:14.923533+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:37.574983+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.3","version":"548'15178","reported_seq":15544,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-02-09T17:44:30.976997+0000","last_change":"2024-02-09T17:43:36.701958+0000","last_active":"2024-02-09T17:44:30.976997+0000","last_peered":"2024-02-09T17:44:30.976997+0000","last_clean":"2024-02-09T17:44:30.976997+0000","last_became_active":"2024-02-09T17:40:48.783369+0000","last_became_peered":"2024-02-09T17:40:48.783369+0000","last_unstale":"2024-02-09T17:44:30.976997+0000","last_undegraded":"2024-02-09T17:44:30.976997+0000","last_fullsized":"2024-02-09T17:44:30.976997+0000","mapping_epoch":465,"log_start":"492'5100","ondisk_log_start":"492'5100","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"529'12566","last_scrub_stamp":"2024-02-09T17:43:36.701823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:43:36.701823+0000","objects_scrubbed":4960,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:00.781538+0000","scrub_duration":1352,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152109056,"num_objects":2321,"num_object_clones":0,"num_object_copies":6963,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13617,"num_write_kb":510016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"63'139","reported_seq":593,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-02-09T17:44:30.020129+0000","last_change":"2024-02-09T17:43:15.279635+0000","last_active":"2024-02-09T17:44:30.020129+0000","last_peered":"2024-02-09T17:44:30.020129+0000","last_clean":"2024-02-09T17:44:30.020129+0000","last_became_active":"2024-02-09T17:43:14.038885+0000","last_became_peered":"2024-02-09T17:43:14.038885+0000","last_unstale":"2024-02-09T17:44:30.020129+0000","last_undegraded":"2024-02-09T17:44:30.020129+0000","last_fullsized":"2024-02-09T17:44:30.020129+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:15.279496+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:43:15.279496+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:36.260654+0000","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"547'14010","reported_seq":14324,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-09T17:44:28.112972+0000","last_change":"2024-02-09T17:43:52.312478+0000","last_active":"2024-02-09T17:44:28.112972+0000","last_peered":"2024-02-09T17:44:28.112972+0000","last_clean":"2024-02-09T17:44:28.112972+0000","last_became_active":"2024-02-09T17:40:48.780290+0000","last_became_peered":"2024-02-09T17:40:48.780290+0000","last_unstale":"2024-02-09T17:44:28.112972+0000","last_undegraded":"2024-02-09T17:44:28.112972+0000","last_fullsized":"2024-02-09T17:44:28.112972+0000","mapping_epoch":465,"log_start":"486'4000","ondisk_log_start":"486'4000","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"535'12174","last_scrub_stamp":"2024-02-09T17:43:52.312370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:43:52.312370+0000","objects_scrubbed":4242,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:55.425323+0000","scrub_duration":401,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157679616,"num_objects":2406,"num_object_clones":0,"num_object_copies":7218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14010,"num_write_kb":525312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"63'139","reported_seq":1008,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-02-09T17:44:30.027908+0000","last_change":"2024-02-09T17:43:56.105889+0000","last_active":"2024-02-09T17:44:30.027908+0000","last_peered":"2024-02-09T17:44:30.027908+0000","last_clean":"2024-02-09T17:44:30.027908+0000","last_became_active":"2024-02-09T17:33:00.284924+0000","last_became_peered":"2024-02-09T17:33:00.284924+0000","last_unstale":"2024-02-09T17:44:30.027908+0000","last_undegraded":"2024-02-09T17:44:30.027908+0000","last_fullsized":"2024-02-09T17:44:30.027908+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:56.105792+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:56.105792+0000","last_clean_scrub_stamp":"2024-02-09T17:43:56.105792+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:56.755038+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.1","version":"548'13593","reported_seq":13910,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-02-09T17:44:30.861718+0000","last_change":"2024-02-09T17:43:44.138178+0000","last_active":"2024-02-09T17:44:30.861718+0000","last_peered":"2024-02-09T17:44:30.861718+0000","last_clean":"2024-02-09T17:44:30.861718+0000","last_became_active":"2024-02-09T17:40:49.660617+0000","last_became_peered":"2024-02-09T17:40:49.660617+0000","last_unstale":"2024-02-09T17:44:30.861718+0000","last_undegraded":"2024-02-09T17:44:30.861718+0000","last_fullsized":"2024-02-09T17:44:30.861718+0000","mapping_epoch":465,"log_start":"486'3500","ondisk_log_start":"486'3500","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"533'11388","last_scrub_stamp":"2024-02-09T17:43:44.138002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:43:44.138002+0000","objects_scrubbed":4543,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:01.112942+0000","scrub_duration":1020,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150929408,"num_objects":2303,"num_object_clones":0,"num_object_copies":6909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13593,"num_write_kb":508672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"63'139","reported_seq":973,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-09T17:44:26.561051+0000","last_change":"2024-02-09T17:43:28.597827+0000","last_active":"2024-02-09T17:44:26.561051+0000","last_peered":"2024-02-09T17:44:26.561051+0000","last_clean":"2024-02-09T17:44:26.561051+0000","last_became_active":"2024-02-09T17:43:28.597612+0000","last_became_peered":"2024-02-09T17:43:28.597612+0000","last_unstale":"2024-02-09T17:44:26.561051+0000","last_undegraded":"2024-02-09T17:44:26.561051+0000","last_fullsized":"2024-02-09T17:44:26.561051+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:15.934522+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:43:15.934522+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:41.034885+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.e","version":"549'13907","reported_seq":14206,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-02-09T17:44:31.585788+0000","last_change":"2024-02-09T17:43:38.843389+0000","last_active":"2024-02-09T17:44:31.585788+0000","last_peered":"2024-02-09T17:44:31.585788+0000","last_clean":"2024-02-09T17:44:31.585788+0000","last_became_active":"2024-02-09T17:40:48.779366+0000","last_became_peered":"2024-02-09T17:40:48.779366+0000","last_unstale":"2024-02-09T17:44:31.585788+0000","last_undegraded":"2024-02-09T17:44:31.585788+0000","last_fullsized":"2024-02-09T17:44:31.585788+0000","mapping_epoch":465,"log_start":"486'3900","ondisk_log_start":"486'3900","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"530'11226","last_scrub_stamp":"2024-02-09T17:43:38.843170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:43:38.843170+0000","objects_scrubbed":4976,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:47.276017+0000","scrub_duration":1147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148570112,"num_objects":2267,"num_object_clones":0,"num_object_copies":6801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13907,"num_write_kb":517568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"63'139","reported_seq":933,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-02-09T17:44:31.574750+0000","last_change":"2024-02-09T17:44:27.890584+0000","last_active":"2024-02-09T17:44:31.574750+0000","last_peered":"2024-02-09T17:44:31.574750+0000","last_clean":"2024-02-09T17:44:31.574750+0000","last_became_active":"2024-02-09T17:43:14.041291+0000","last_became_peered":"2024-02-09T17:43:14.041291+0000","last_unstale":"2024-02-09T17:44:31.574750+0000","last_undegraded":"2024-02-09T17:44:31.574750+0000","last_fullsized":"2024-02-09T17:44:31.574750+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:27.890510+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:44:27.890510+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:53.802187+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.f","version":"547'13708","reported_seq":16393,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-09T17:44:29.363466+0000","last_change":"2024-02-09T17:43:56.859129+0000","last_active":"2024-02-09T17:44:29.363466+0000","last_peered":"2024-02-09T17:44:29.363466+0000","last_clean":"2024-02-09T17:44:29.363466+0000","last_became_active":"2024-02-09T17:41:20.876180+0000","last_became_peered":"2024-02-09T17:41:20.876180+0000","last_unstale":"2024-02-09T17:44:29.363466+0000","last_undegraded":"2024-02-09T17:44:29.363466+0000","last_fullsized":"2024-02-09T17:44:29.363466+0000","mapping_epoch":480,"log_start":"486'3700","ondisk_log_start":"486'3700","created":465,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'12028","last_scrub_stamp":"2024-02-09T17:43:56.858923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:43:56.858923+0000","objects_scrubbed":4068,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:25.138111+0000","scrub_duration":562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156237824,"num_objects":2384,"num_object_clones":0,"num_object_copies":7152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13708,"num_write_kb":514944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1182,"num_bytes_recovered":77463552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,11],"acting":[5,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"63'139","reported_seq":773,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-02-09T17:44:30.027869+0000","last_change":"2024-02-09T17:44:25.176171+0000","last_active":"2024-02-09T17:44:30.027869+0000","last_peered":"2024-02-09T17:44:30.027869+0000","last_clean":"2024-02-09T17:44:30.027869+0000","last_became_active":"2024-02-09T17:43:14.054943+0000","last_became_peered":"2024-02-09T17:43:14.054943+0000","last_unstale":"2024-02-09T17:44:30.027869+0000","last_undegraded":"2024-02-09T17:44:30.027869+0000","last_fullsized":"2024-02-09T17:44:30.027869+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:25.176022+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:44:25.176022+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:31.031846+0000","scrub_duration":158,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"547'13775","reported_seq":14075,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-09T17:44:29.729645+0000","last_change":"2024-02-09T17:43:39.114851+0000","last_active":"2024-02-09T17:44:29.729645+0000","last_peered":"2024-02-09T17:44:29.729645+0000","last_clean":"2024-02-09T17:44:29.729645+0000","last_became_active":"2024-02-09T17:40:48.781384+0000","last_became_peered":"2024-02-09T17:40:48.781384+0000","last_unstale":"2024-02-09T17:44:29.729645+0000","last_undegraded":"2024-02-09T17:44:29.729645+0000","last_fullsized":"2024-02-09T17:44:29.729645+0000","mapping_epoch":465,"log_start":"486'3700","ondisk_log_start":"486'3700","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"531'11356","last_scrub_stamp":"2024-02-09T17:43:39.114709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:43:39.114709+0000","objects_scrubbed":4812,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:06.435989+0000","scrub_duration":1039,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155648000,"num_objects":2375,"num_object_clones":0,"num_object_copies":7125,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13775,"num_write_kb":516800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"63'139","reported_seq":979,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-09T17:44:26.571109+0000","last_change":"2024-02-09T17:43:28.586677+0000","last_active":"2024-02-09T17:44:26.571109+0000","last_peered":"2024-02-09T17:44:26.571109+0000","last_clean":"2024-02-09T17:44:26.571109+0000","last_became_active":"2024-02-09T17:43:28.586349+0000","last_became_peered":"2024-02-09T17:43:28.586349+0000","last_unstale":"2024-02-09T17:44:26.571109+0000","last_undegraded":"2024-02-09T17:44:26.571109+0000","last_fullsized":"2024-02-09T17:44:26.571109+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:14.332107+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:43:14.332107+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"548'14046","reported_seq":16895,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-02-09T17:44:30.403344+0000","last_change":"2024-02-09T17:44:29.650288+0000","last_active":"2024-02-09T17:44:30.403344+0000","last_peered":"2024-02-09T17:44:30.403344+0000","last_clean":"2024-02-09T17:44:30.403344+0000","last_became_active":"2024-02-09T17:41:21.499187+0000","last_became_peered":"2024-02-09T17:41:21.499187+0000","last_unstale":"2024-02-09T17:44:30.403344+0000","last_undegraded":"2024-02-09T17:44:30.403344+0000","last_fullsized":"2024-02-09T17:44:30.403344+0000","mapping_epoch":481,"log_start":"486'4000","ondisk_log_start":"486'4000","created":465,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"547'14005","last_scrub_stamp":"2024-02-09T17:44:29.650159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:44:29.650159+0000","objects_scrubbed":2458,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:55.225580+0000","scrub_duration":643,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157089816,"num_objects":2398,"num_object_clones":0,"num_object_copies":7194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2398,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14046,"num_write_kb":526145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1248,"num_bytes_recovered":81788928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"63'139","reported_seq":1429,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-02-09T17:44:31.588618+0000","last_change":"2024-02-09T17:43:28.591362+0000","last_active":"2024-02-09T17:44:31.588618+0000","last_peered":"2024-02-09T17:44:31.588618+0000","last_clean":"2024-02-09T17:44:31.588618+0000","last_became_active":"2024-02-09T17:43:28.590237+0000","last_became_peered":"2024-02-09T17:43:28.590237+0000","last_unstale":"2024-02-09T17:44:31.588618+0000","last_undegraded":"2024-02-09T17:44:31.588618+0000","last_fullsized":"2024-02-09T17:44:31.588618+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:15.675690+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:15.675690+0000","last_clean_scrub_stamp":"2024-02-09T17:43:15.675690+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:44.606598+0000","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"549'13691","reported_seq":16016,"reported_epoch":549,"state":"active+remapped+backfilling","last_fresh":"2024-02-09T17:44:32.000937+0000","last_change":"2024-02-09T17:44:26.926646+0000","last_active":"2024-02-09T17:44:32.000937+0000","last_peered":"2024-02-09T17:44:32.000937+0000","last_clean":"2024-02-09T17:43:24.158212+0000","last_became_active":"2024-02-09T17:43:26.887932+0000","last_became_peered":"2024-02-09T17:43:26.887932+0000","last_unstale":"2024-02-09T17:44:32.000937+0000","last_undegraded":"2024-02-09T17:44:32.000937+0000","last_fullsized":"2024-02-09T17:44:32.000937+0000","mapping_epoch":522,"log_start":"486'3600","ondisk_log_start":"486'3600","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'7912","last_scrub_stamp":"2024-02-09T17:42:33.501032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:42:33.501032+0000","objects_scrubbed":7852,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1903,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148701184,"num_objects":2269,"num_object_clones":0,"num_object_copies":6807,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1300,"num_objects_unfound":0,"num_objects_dirty":2269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13691,"num_write_kb":510720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1022,"num_bytes_recovered":66977792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,10],"acting":[8,10,4],"avail_no_missing":["8","4","10"],"object_location_counts":[{"shards":"4,8,10","objects":2269}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.e","version":"63'139","reported_seq":900,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-02-09T17:44:30.025488+0000","last_change":"2024-02-09T17:44:25.525420+0000","last_active":"2024-02-09T17:44:30.025488+0000","last_peered":"2024-02-09T17:44:30.025488+0000","last_clean":"2024-02-09T17:44:30.025488+0000","last_became_active":"2024-02-09T17:43:14.036583+0000","last_became_peered":"2024-02-09T17:43:14.036583+0000","last_unstale":"2024-02-09T17:44:30.025488+0000","last_undegraded":"2024-02-09T17:44:30.025488+0000","last_fullsized":"2024-02-09T17:44:30.025488+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:25.525304+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:14.981453+0000","last_clean_scrub_stamp":"2024-02-09T17:44:25.525304+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:50.128868+0000","scrub_duration":97,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"549'13797","reported_seq":14119,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-02-09T17:44:31.634333+0000","last_change":"2024-02-09T17:43:42.009150+0000","last_active":"2024-02-09T17:44:31.634333+0000","last_peered":"2024-02-09T17:44:31.634333+0000","last_clean":"2024-02-09T17:44:31.634333+0000","last_became_active":"2024-02-09T17:40:48.783235+0000","last_became_peered":"2024-02-09T17:40:48.783235+0000","last_unstale":"2024-02-09T17:44:31.634333+0000","last_undegraded":"2024-02-09T17:44:31.634333+0000","last_fullsized":"2024-02-09T17:44:31.634333+0000","mapping_epoch":465,"log_start":"486'3700","ondisk_log_start":"486'3700","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"533'11389","last_scrub_stamp":"2024-02-09T17:43:42.009039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:43:42.009039+0000","objects_scrubbed":4727,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:05.982895+0000","scrub_duration":1348,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150011904,"num_objects":2289,"num_object_clones":0,"num_object_copies":6867,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13797,"num_write_kb":514752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.f","version":"63'139","reported_seq":945,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-02-09T17:44:30.020038+0000","last_change":"2024-02-09T17:43:38.588100+0000","last_active":"2024-02-09T17:44:30.020038+0000","last_peered":"2024-02-09T17:44:30.020038+0000","last_clean":"2024-02-09T17:44:30.020038+0000","last_became_active":"2024-02-09T17:33:05.316100+0000","last_became_peered":"2024-02-09T17:33:05.316100+0000","last_unstale":"2024-02-09T17:44:30.020038+0000","last_undegraded":"2024-02-09T17:44:30.020038+0000","last_fullsized":"2024-02-09T17:44:30.020038+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:38.588006+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:36:22.364130+0000","last_clean_scrub_stamp":"2024-02-09T17:43:38.588006+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:42.249570+0000","scrub_duration":165,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"548'13937","reported_seq":16655,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-02-09T17:44:30.408224+0000","last_change":"2024-02-09T17:43:54.002412+0000","last_active":"2024-02-09T17:44:30.408224+0000","last_peered":"2024-02-09T17:44:30.408224+0000","last_clean":"2024-02-09T17:44:30.408224+0000","last_became_active":"2024-02-09T17:41:20.876434+0000","last_became_peered":"2024-02-09T17:41:20.876434+0000","last_unstale":"2024-02-09T17:44:30.408224+0000","last_undegraded":"2024-02-09T17:44:30.408224+0000","last_fullsized":"2024-02-09T17:44:30.408224+0000","mapping_epoch":480,"log_start":"486'3900","ondisk_log_start":"486'3900","created":465,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"536'12114","last_scrub_stamp":"2024-02-09T17:43:54.002296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:43:54.002296+0000","objects_scrubbed":4157,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:55.972751+0000","scrub_duration":854,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151715840,"num_objects":2315,"num_object_clones":0,"num_object_copies":6945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13937,"num_write_kb":520064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1195,"num_bytes_recovered":78315520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"63'139","reported_seq":1082,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-02-09T17:44:31.598665+0000","last_change":"2024-02-09T17:44:28.298713+0000","last_active":"2024-02-09T17:44:31.598665+0000","last_peered":"2024-02-09T17:44:31.598665+0000","last_clean":"2024-02-09T17:44:31.598665+0000","last_became_active":"2024-02-09T17:43:14.027715+0000","last_became_peered":"2024-02-09T17:43:14.027715+0000","last_unstale":"2024-02-09T17:44:31.598665+0000","last_undegraded":"2024-02-09T17:44:31.598665+0000","last_fullsized":"2024-02-09T17:44:31.598665+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:28.298578+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:14.135321+0000","last_clean_scrub_stamp":"2024-02-09T17:44:28.298578+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:31.180345+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"549'14008","reported_seq":14355,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-02-09T17:44:32.361412+0000","last_change":"2024-02-09T17:44:25.663805+0000","last_active":"2024-02-09T17:44:32.361412+0000","last_peered":"2024-02-09T17:44:32.361412+0000","last_clean":"2024-02-09T17:44:32.361412+0000","last_became_active":"2024-02-09T17:40:49.436285+0000","last_became_peered":"2024-02-09T17:40:49.436285+0000","last_unstale":"2024-02-09T17:44:32.361412+0000","last_undegraded":"2024-02-09T17:44:32.361412+0000","last_fullsized":"2024-02-09T17:44:32.361412+0000","mapping_epoch":465,"log_start":"486'4000","ondisk_log_start":"486'4000","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"546'13640","last_scrub_stamp":"2024-02-09T17:44:25.663661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:44:25.663661+0000","objects_scrubbed":2677,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:53.939379+0000","scrub_duration":943,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149422080,"num_objects":2280,"num_object_clones":0,"num_object_copies":6840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14008,"num_write_kb":521216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"63'139","reported_seq":493,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-02-09T17:44:31.587407+0000","last_change":"2024-02-09T17:44:30.715902+0000","last_active":"2024-02-09T17:44:31.587407+0000","last_peered":"2024-02-09T17:44:31.587407+0000","last_clean":"2024-02-09T17:44:31.587407+0000","last_became_active":"2024-02-09T17:43:14.036759+0000","last_became_peered":"2024-02-09T17:43:14.036759+0000","last_unstale":"2024-02-09T17:44:31.587407+0000","last_undegraded":"2024-02-09T17:44:31.587407+0000","last_fullsized":"2024-02-09T17:44:31.587407+0000","mapping_epoch":454,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:30.715778+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:44:30.715778+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:57.590268+0000","scrub_duration":144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"63'139","reported_seq":884,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-09T17:44:26.571199+0000","last_change":"2024-02-09T17:43:04.345907+0000","last_active":"2024-02-09T17:44:26.571199+0000","last_peered":"2024-02-09T17:44:26.571199+0000","last_clean":"2024-02-09T17:44:26.571199+0000","last_became_active":"2024-02-09T17:33:05.310104+0000","last_became_peered":"2024-02-09T17:33:05.310104+0000","last_unstale":"2024-02-09T17:44:26.571199+0000","last_undegraded":"2024-02-09T17:44:26.571199+0000","last_fullsized":"2024-02-09T17:44:26.571199+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:04.345783+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:43:04.345783+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:32.154752+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"63'139","reported_seq":903,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-02-09T17:44:31.563035+0000","last_change":"2024-02-09T17:43:46.691783+0000","last_active":"2024-02-09T17:44:31.563035+0000","last_peered":"2024-02-09T17:44:31.563035+0000","last_clean":"2024-02-09T17:44:31.563035+0000","last_became_active":"2024-02-09T17:34:01.353171+0000","last_became_peered":"2024-02-09T17:34:01.353171+0000","last_unstale":"2024-02-09T17:44:31.563035+0000","last_undegraded":"2024-02-09T17:44:31.563035+0000","last_fullsized":"2024-02-09T17:44:31.563035+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:46.691649+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:43:46.691649+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:50.935736+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.12","version":"63'139","reported_seq":1411,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-09T17:44:26.570979+0000","last_change":"2024-02-09T17:43:55.346549+0000","last_active":"2024-02-09T17:44:26.570979+0000","last_peered":"2024-02-09T17:44:26.570979+0000","last_clean":"2024-02-09T17:44:26.570979+0000","last_became_active":"2024-02-09T17:40:13.499071+0000","last_became_peered":"2024-02-09T17:40:13.499071+0000","last_unstale":"2024-02-09T17:44:26.570979+0000","last_undegraded":"2024-02-09T17:44:26.570979+0000","last_fullsized":"2024-02-09T17:44:26.570979+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:55.341130+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:43:55.341130+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:10.488649+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"63'139","reported_seq":611,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-02-09T17:44:27.381410+0000","last_change":"2024-02-09T17:44:27.381453+0000","last_active":"2024-02-09T17:44:27.381410+0000","last_peered":"2024-02-09T17:44:27.381410+0000","last_clean":"2024-02-09T17:44:27.381410+0000","last_became_active":"2024-02-09T17:41:14.521672+0000","last_became_peered":"2024-02-09T17:41:14.521672+0000","last_unstale":"2024-02-09T17:44:27.381410+0000","last_undegraded":"2024-02-09T17:44:27.381410+0000","last_fullsized":"2024-02-09T17:44:27.381410+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:27.381332+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:44:27.381332+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:35.775807+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":2457927704,"num_objects":37506,"num_object_clones":0,"num_object_copies":112518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1300,"num_objects_unfound":0,"num_objects_dirty":37506,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":220302,"num_write_kb":8249793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13978,"num_bytes_recovered":916062208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7437103104,"data_stored":7437090888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160964,"ondisk_log_size":160964,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056906,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1168984,"kb_used_data":610256,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":92576168,"statfs":{"total":95995035648,"available":94797996032,"internally_reserved":0,"allocated":624902144,"data_stored":624479089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":572127684},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:43:29 2024","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.2849999999999999,"15min":1.3160000000000001},"min":{"1min":0.23599999999999999,"5min":0.223,"15min":0.216},"max":{"1min":7.4089999999999998,"5min":9.5709999999999997,"15min":14.913},"last":1.966},{"interface":"front","average":{"1min":1.165,"5min":1.258,"15min":1.2849999999999999},"min":{"1min":0.25900000000000001,"5min":0.253,"15min":0.23000000000000001},"max":{"1min":5.1340000000000003,"5min":9.8219999999999992,"15min":14.869},"last":0.28599999999999998}]},{"osd":1,"last update":"Fri Feb 9 17:43:29 2024","interfaces":[{"interface":"back","average":{"1min":1.754,"5min":1.3839999999999999,"15min":1.4830000000000001},"min":{"1min":0.29599999999999999,"5min":0.27600000000000002,"15min":0.22700000000000001},"max":{"1min":9.7669999999999995,"5min":9.7710000000000008,"15min":17.832999999999998},"last":1.9330000000000001},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.355,"15min":1.583},"min":{"1min":0.29399999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":7.4619999999999997,"5min":9.718,"15min":17.789000000000001},"last":0.36199999999999999}]},{"osd":2,"last update":"Fri Feb 9 17:31:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.415,"15min":1.6859999999999999},"min":{"1min":0.378,"5min":0.247,"15min":0.247},"max":{"1min":4.4189999999999996,"5min":9.4719999999999995,"15min":9.4719999999999995},"last":0.35399999999999998},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.5149999999999999,"15min":1.4399999999999999},"min":{"1min":0.378,"5min":0.252,"15min":0.252},"max":{"1min":4.8579999999999997,"5min":7.96,"15min":7.96},"last":0.60399999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:43:29 2024","interfaces":[{"interface":"back","average":{"1min":1.958,"5min":1.1719999999999999,"15min":1.484},"min":{"1min":0.27200000000000002,"5min":0.27100000000000002,"15min":0.247},"max":{"1min":9.7289999999999992,"5min":9.7289999999999992,"15min":21.315999999999999},"last":0.312},{"interface":"front","average":{"1min":2.0070000000000001,"5min":1.5980000000000001,"15min":1.6819999999999999},"min":{"1min":0.33600000000000002,"5min":0.33000000000000002,"15min":0.26400000000000001},"max":{"1min":9.7479999999999993,"5min":9.7479999999999993,"15min":18.087},"last":0.48999999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:43:29 2024","interfaces":[{"interface":"back","average":{"1min":2.1579999999999999,"5min":1.587,"15min":1.651},"min":{"1min":0.26900000000000002,"5min":0.246,"15min":0.246},"max":{"1min":6.3959999999999999,"5min":21.030999999999999,"15min":21.030999999999999},"last":0.33200000000000002},{"interface":"front","average":{"1min":2.1680000000000001,"5min":1.548,"15min":1.4299999999999999},"min":{"1min":0.318,"5min":0.317,"15min":0.28799999999999998},"max":{"1min":7.5490000000000004,"5min":20.876999999999999,"15min":20.876999999999999},"last":6.1289999999999996}]},{"osd":5,"last update":"Fri Feb 9 17:43:29 2024","interfaces":[{"interface":"back","average":{"1min":2.4380000000000002,"5min":1.3819999999999999,"15min":1.4970000000000001},"min":{"1min":0.47199999999999998,"5min":0.32900000000000001,"15min":0.28899999999999998},"max":{"1min":9.5960000000000001,"5min":9.5960000000000001,"15min":13.946999999999999},"last":0.42499999999999999},{"interface":"front","average":{"1min":2.6360000000000001,"5min":1.486,"15min":1.607},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":7.4580000000000002,"5min":7.4580000000000002,"15min":13.414},"last":0.375}]},{"osd":6,"last update":"Fri Feb 9 17:43:29 2024","interfaces":[{"interface":"back","average":{"1min":3.1459999999999999,"5min":1.734,"15min":1.6359999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.29899999999999999},"max":{"1min":12.362,"5min":12.362,"15min":12.798999999999999},"last":0.79000000000000004},{"interface":"front","average":{"1min":2.238,"5min":1.548,"15min":1.631},"min":{"1min":0.34100000000000003,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":7.5800000000000001,"5min":7.5800000000000001,"15min":10.782999999999999},"last":10.000999999999999}]},{"osd":7,"last update":"Fri Feb 9 17:43:29 2024","interfaces":[{"interface":"back","average":{"1min":2.4849999999999999,"5min":1.6990000000000001,"15min":1.7669999999999999},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.312},"max":{"1min":9.7029999999999994,"5min":20.489999999999998,"15min":20.489999999999998},"last":0.47099999999999997},{"interface":"front","average":{"1min":2.1960000000000002,"5min":1.671,"15min":1.7250000000000001},"min":{"1min":0.376,"5min":0.35299999999999998,"15min":0.34200000000000003},"max":{"1min":7.4210000000000003,"5min":20.712,"15min":20.712},"last":0.53000000000000003}]},{"osd":8,"last update":"Fri Feb 9 17:43:56 2024","interfaces":[{"interface":"back","average":{"1min":2.7570000000000001,"5min":1.345,"15min":1.627},"min":{"1min":0.36299999999999999,"5min":0.30399999999999999,"15min":0.29499999999999998},"max":{"1min":9.5429999999999993,"5min":9.5429999999999993,"15min":13.199999999999999},"last":2.0150000000000001},{"interface":"front","average":{"1min":2.6829999999999998,"5min":1.4610000000000001,"15min":1.5009999999999999},"min":{"1min":0.36199999999999999,"5min":0.35399999999999998,"15min":0.318},"max":{"1min":7.5620000000000003,"5min":7.6059999999999999,"15min":9.2940000000000005},"last":2.0379999999999998}]},{"osd":9,"last update":"Fri Feb 9 17:44:25 2024","interfaces":[{"interface":"back","average":{"1min":1.738,"5min":2.1190000000000002,"15min":1.544},"min":{"1min":0.39000000000000001,"5min":0.34100000000000003,"15min":0.29399999999999998},"max":{"1min":4.4249999999999998,"5min":62.518000000000001,"15min":62.518000000000001},"last":0.38900000000000001},{"interface":"front","average":{"1min":1.45,"5min":2.044,"15min":1.45},"min":{"1min":0.41099999999999998,"5min":0.35399999999999998,"15min":0.35199999999999998},"max":{"1min":4.8330000000000002,"5min":63.241999999999997,"15min":63.241999999999997},"last":0.40500000000000003}]},{"osd":10,"last update":"Fri Feb 9 17:43:29 2024","interfaces":[{"interface":"back","average":{"1min":2.3900000000000001,"5min":1.3879999999999999,"15min":1.3280000000000001},"min":{"1min":0.38500000000000001,"5min":0.36799999999999999,"15min":0.36199999999999999},"max":{"1min":9.6590000000000007,"5min":9.6590000000000007,"15min":9.6590000000000007},"last":1.9910000000000001},{"interface":"front","average":{"1min":2.5510000000000002,"5min":1.649,"15min":1.405},"min":{"1min":0.47599999999999998,"5min":0.39100000000000001,"15min":0.33900000000000002},"max":{"1min":12.409000000000001,"5min":12.409000000000001,"15min":12.856},"last":0.50700000000000001}]}]},{"osd":10,"up_from":84,"seq":360777253135,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1196248,"kb_used_data":799948,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":92548904,"statfs":{"total":95995035648,"available":94770077696,"internally_reserved":0,"allocated":819146752,"data_stored":818727769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":405797312},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:44:24 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.91500000000000004,"15min":1.45},"min":{"1min":0.25,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":2.7650000000000001,"5min":9.343,"15min":9.343},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.92000000000000004,"15min":1.105},"min":{"1min":0.29199999999999998,"5min":0.253,"15min":0.253},"max":{"1min":4.2699999999999996,"5min":6.3399999999999999,"15min":6.3399999999999999},"last":0.28199999999999997}]},{"osd":1,"last update":"Fri Feb 9 17:43:51 2024","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.73699999999999999,"15min":0.65900000000000003},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":3.6960000000000002,"5min":3.726,"15min":3.726},"last":7.0709999999999997},{"interface":"front","average":{"1min":1.6739999999999999,"5min":1.0580000000000001,"15min":0.878},"min":{"1min":0.27300000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":5.5910000000000002,"5min":6.2709999999999999,"15min":6.2709999999999999},"last":1.4179999999999999}]},{"osd":2,"last update":"Fri Feb 9 17:44:02 2024","interfaces":[{"interface":"back","average":{"1min":1.548,"5min":1.2110000000000001,"15min":1.226},"min":{"1min":0.30599999999999999,"5min":0.27600000000000002,"15min":0.222},"max":{"1min":5.5279999999999996,"5min":6.5190000000000001,"15min":11.384},"last":7.0149999999999997},{"interface":"front","average":{"1min":0.96199999999999997,"5min":1.1240000000000001,"15min":1.1890000000000001},"min":{"1min":0.39200000000000002,"5min":0.32500000000000001,"15min":0.26100000000000001},"max":{"1min":3.298,"5min":9.1140000000000008,"15min":11.465},"last":0.45500000000000002}]},{"osd":3,"last update":"Fri Feb 9 17:43:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.3520000000000001,"15min":1.8160000000000001},"min":{"1min":0.36399999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":4.6799999999999997,"5min":9.1649999999999991,"15min":9.1649999999999991},"last":0.48799999999999999},{"interface":"front","average":{"1min":1.635,"5min":1.55,"15min":1.9199999999999999},"min":{"1min":0.38100000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":5.6109999999999998,"5min":8.9670000000000005,"15min":8.9670000000000005},"last":7.0960000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:44:06 2024","interfaces":[{"interface":"back","average":{"1min":1.875,"5min":1.2430000000000001,"15min":1.169},"min":{"1min":0.39200000000000002,"5min":0.30199999999999999,"15min":0.29199999999999998},"max":{"1min":8.5120000000000005,"5min":8.5120000000000005,"15min":8.5120000000000005},"last":1.2070000000000001},{"interface":"front","average":{"1min":2.1459999999999999,"5min":1.3109999999999999,"15min":1.1599999999999999},"min":{"1min":0.41299999999999998,"5min":0.39300000000000002,"15min":0.29999999999999999},"max":{"1min":11.273999999999999,"5min":11.273999999999999,"15min":11.273999999999999},"last":0.623}]},{"osd":5,"last update":"Fri Feb 9 17:44:02 2024","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.0740000000000001,"15min":1.2390000000000001},"min":{"1min":0.34699999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":4.2279999999999998,"5min":4.4820000000000002,"15min":9.2349999999999994},"last":7.1550000000000002},{"interface":"front","average":{"1min":1.522,"5min":1.4970000000000001,"15min":1.3109999999999999},"min":{"1min":0.435,"5min":0.38500000000000001,"15min":0.33500000000000002},"max":{"1min":4.343,"5min":8.7710000000000008,"15min":8.7710000000000008},"last":0.85799999999999998}]},{"osd":6,"last update":"Fri Feb 9 17:44:02 2024","interfaces":[{"interface":"back","average":{"1min":2.5529999999999999,"5min":1.6679999999999999,"15min":1.4139999999999999},"min":{"1min":0.46000000000000002,"5min":0.32400000000000001,"15min":0.29999999999999999},"max":{"1min":6.4210000000000003,"5min":10.962,"15min":10.962},"last":1.4470000000000001},{"interface":"front","average":{"1min":1.292,"5min":1.4430000000000001,"15min":1.448},"min":{"1min":0.495,"5min":0.38600000000000001,"15min":0.35399999999999998},"max":{"1min":6.1280000000000001,"5min":8.8119999999999994,"15min":9.4350000000000005},"last":0.52100000000000002}]},{"osd":7,"last update":"Fri Feb 9 17:39:52 2024","interfaces":[{"interface":"back","average":{"1min":1.6000000000000001,"5min":1.367,"15min":1.415},"min":{"1min":0.42099999999999999,"5min":0.34200000000000003,"15min":0.312},"max":{"1min":4.0430000000000001,"5min":6.7590000000000003,"15min":17.754000000000001},"last":4.6449999999999996},{"interface":"front","average":{"1min":1.498,"5min":1.4359999999999999,"15min":1.5189999999999999},"min":{"1min":0.47399999999999998,"5min":0.38300000000000001,"15min":0.316},"max":{"1min":5.8170000000000002,"5min":11.308999999999999,"15min":11.308999999999999},"last":0.55000000000000004}]},{"osd":8,"last update":"Fri Feb 9 17:43:45 2024","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.1140000000000001,"15min":1.1519999999999999},"min":{"1min":0.35499999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":6.4269999999999996,"5min":7.4020000000000001,"15min":7.4359999999999999},"last":6.9749999999999996},{"interface":"front","average":{"1min":1.8859999999999999,"5min":1.2669999999999999,"15min":1.04},"min":{"1min":0.40699999999999997,"5min":0.35499999999999998,"15min":0.32900000000000001},"max":{"1min":6.2009999999999996,"5min":7.7060000000000004,"15min":7.7060000000000004},"last":1.377}]},{"osd":9,"last update":"Fri Feb 9 17:44:02 2024","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.073,"15min":1.0429999999999999},"min":{"1min":0.433,"5min":0.36599999999999999,"15min":0.34399999999999997},"max":{"1min":3.1429999999999998,"5min":4.766,"15min":7.4850000000000003},"last":7.0419999999999998},{"interface":"front","average":{"1min":2.4950000000000001,"5min":1.377,"15min":1.1819999999999999},"min":{"1min":0.46700000000000003,"5min":0.441,"15min":0.29799999999999999},"max":{"1min":10.622999999999999,"5min":10.622999999999999,"15min":10.622999999999999},"last":7.1299999999999999}]},{"osd":11,"last update":"Fri Feb 9 17:44:24 2024","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.0589999999999999,"15min":1.016},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":3.4500000000000002,"5min":6.226,"15min":7.383},"last":3.1120000000000001},{"interface":"front","average":{"1min":1.159,"5min":1.1200000000000001,"15min":1.0660000000000001},"min":{"1min":0.47399999999999998,"5min":0.441,"15min":0.377},"max":{"1min":3.484,"5min":6.2590000000000003,"15min":7.3019999999999996},"last":3.1480000000000001}]}]},{"osd":9,"up_from":77,"seq":330712482069,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":890340,"kb_used_data":444056,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":92854812,"statfs":{"total":95995035648,"available":95083327488,"internally_reserved":0,"allocated":454713344,"data_stored":454289896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":456980928},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:43:56 2024","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.79000000000000004,"15min":0.65200000000000002},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":4.5830000000000002,"5min":7.2199999999999998,"15min":7.2199999999999998},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.75,"5min":1.0109999999999999,"15min":0.97099999999999997},"min":{"1min":0.28399999999999997,"5min":0.253,"15min":0.253},"max":{"1min":5.6790000000000003,"5min":5.6790000000000003,"15min":5.6790000000000003},"last":3.7229999999999999}]},{"osd":1,"last update":"Fri Feb 9 17:43:42 2024","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.78,"15min":1.6699999999999999},"min":{"1min":0.25900000000000001,"5min":0.23599999999999999,"15min":0.23100000000000001},"max":{"1min":5.4080000000000004,"5min":17.024999999999999,"15min":89.515000000000001},"last":2.5859999999999999},{"interface":"front","average":{"1min":1.115,"5min":1.577,"15min":1.631},"min":{"1min":0.29799999999999999,"5min":0.249,"15min":0.222},"max":{"1min":4.5439999999999996,"5min":11.864000000000001,"15min":89.388000000000005},"last":0.34699999999999998}]},{"osd":2,"last update":"Fri Feb 9 17:40:39 2024","interfaces":[{"interface":"back","average":{"1min":1.742,"5min":1.766,"15min":1.514},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":5.6630000000000003,"5min":14.121,"15min":14.121},"last":0.63400000000000001},{"interface":"front","average":{"1min":1.6910000000000001,"5min":1.52,"15min":1.4870000000000001},"min":{"1min":0.34100000000000003,"5min":0.26100000000000001,"15min":0.25600000000000001},"max":{"1min":5.7030000000000003,"5min":8.8040000000000003,"15min":13.69},"last":0.73899999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:43:42 2024","interfaces":[{"interface":"back","average":{"1min":1.6719999999999999,"5min":1.984,"15min":1.6200000000000001},"min":{"1min":0.35499999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":5.4480000000000004,"5min":15.414999999999999,"15min":40.18},"last":2.6200000000000001},{"interface":"front","average":{"1min":1.131,"5min":1.8080000000000001,"15min":1.6499999999999999},"min":{"1min":0.36699999999999999,"5min":0.32000000000000001,"15min":0.30599999999999999},"max":{"1min":4.6100000000000003,"5min":15.372,"15min":40.158000000000001},"last":2.0609999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:43:42 2024","interfaces":[{"interface":"back","average":{"1min":2.8039999999999998,"5min":1.871,"15min":1.611},"min":{"1min":0.29499999999999998,"5min":0.29299999999999998,"15min":0.28399999999999997},"max":{"1min":8.7189999999999994,"5min":27.381,"15min":27.381},"last":2.5419999999999998},{"interface":"front","average":{"1min":1.7589999999999999,"5min":1.631,"15min":1.462},"min":{"1min":0.34399999999999997,"5min":0.313,"15min":0.26100000000000001},"max":{"1min":4.7859999999999996,"5min":27.355,"15min":27.355},"last":0.39500000000000002}]},{"osd":5,"last update":"Fri Feb 9 17:43:46 2024","interfaces":[{"interface":"back","average":{"1min":2.032,"5min":1.431,"15min":1.4710000000000001},"min":{"1min":0.374,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":6.1849999999999996,"5min":8.6110000000000007,"15min":10.689},"last":0.372},{"interface":"front","average":{"1min":1.5860000000000001,"5min":1.619,"15min":1.3480000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.311},"max":{"1min":6.1559999999999997,"5min":13.978999999999999,"15min":13.978999999999999},"last":3.6589999999999998}]},{"osd":6,"last update":"Fri Feb 9 17:43:42 2024","interfaces":[{"interface":"back","average":{"1min":2.4569999999999999,"5min":1.734,"15min":1.52},"min":{"1min":0.42699999999999999,"5min":0.34499999999999997,"15min":0.30599999999999999},"max":{"1min":5.7060000000000004,"5min":14.069000000000001,"15min":14.069000000000001},"last":3.7080000000000002},{"interface":"front","average":{"1min":2.98,"5min":1.8879999999999999,"15min":1.613},"min":{"1min":0.38100000000000001,"5min":0.37,"15min":0.37},"max":{"1min":10.417,"5min":14.097,"15min":14.097},"last":3.7389999999999999}]},{"osd":7,"last update":"Fri Feb 9 17:43:42 2024","interfaces":[{"interface":"back","average":{"1min":2.9620000000000002,"5min":1.917,"15min":1.603},"min":{"1min":0.372,"5min":0.372,"15min":0.34300000000000003},"max":{"1min":6.3879999999999999,"5min":9.125,"15min":9.125},"last":3.9929999999999999},{"interface":"front","average":{"1min":2.9980000000000002,"5min":1.958,"15min":1.657},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.32200000000000001},"max":{"1min":6.415,"5min":9.157,"15min":9.157},"last":4.0270000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:44:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4610000000000001,"5min":1.3360000000000001,"15min":1.29},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":6.0800000000000001,"5min":7.8369999999999997,"15min":16.209},"last":0.42099999999999999},{"interface":"front","average":{"1min":1.671,"5min":1.524,"15min":1.351},"min":{"1min":0.501,"5min":0.39600000000000002,"15min":0.34799999999999998},"max":{"1min":6.0949999999999998,"5min":8.6950000000000003,"15min":16.263000000000002},"last":3.6890000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:44:08 2024","interfaces":[{"interface":"back","average":{"1min":2.254,"5min":1.639,"15min":1.323},"min":{"1min":0.35599999999999998,"5min":0.312,"15min":0.312},"max":{"1min":10.206,"5min":10.206,"15min":10.206},"last":5.0279999999999996},{"interface":"front","average":{"1min":1.546,"5min":1.351,"15min":1.1830000000000001},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.38200000000000001},"max":{"1min":10.904,"5min":10.904,"15min":10.904},"last":0.44400000000000001}]},{"osd":11,"last update":"Fri Feb 9 17:44:08 2024","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.5289999999999999,"15min":1.474},"min":{"1min":0.439,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":5.7249999999999996,"5min":13.425000000000001,"15min":14.02},"last":3.7639999999999998},{"interface":"front","average":{"1min":2.2850000000000001,"5min":1.5629999999999999,"15min":1.381},"min":{"1min":0.48399999999999999,"5min":0.38100000000000001,"15min":0.374},"max":{"1min":10.863,"5min":10.863,"15min":10.863},"last":0.60299999999999998}]}]},{"osd":8,"up_from":171,"seq":734439407812,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":536676,"kb_used_data":296920,"kb_used_omap":212,"kb_used_meta":239531,"kb_avail":93208476,"statfs":{"total":95995035648,"available":95445479424,"internally_reserved":0,"allocated":304046080,"data_stored":303622632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217579,"internal_metadata":245280277},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:43:42 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.94499999999999995,"15min":0.97599999999999998},"min":{"1min":0.224,"5min":0.217,"15min":0.217},"max":{"1min":2.1269999999999998,"5min":8.0920000000000005,"15min":8.0920000000000005},"last":0.61199999999999999},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.474,"15min":1.268},"min":{"1min":0.254,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":4.8449999999999998,"5min":9.6189999999999998,"15min":10.256},"last":1.385}]},{"osd":1,"last update":"Fri Feb 9 17:43:42 2024","interfaces":[{"interface":"back","average":{"1min":1.8080000000000001,"5min":1.3680000000000001,"15min":1.321},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.718,"5min":8.718,"15min":8.718},"last":1.429},{"interface":"front","average":{"1min":1.897,"5min":1.8200000000000001,"15min":1.5129999999999999},"min":{"1min":0.27600000000000002,"5min":0.25600000000000001,"15min":0.23300000000000001},"max":{"1min":8.8930000000000007,"5min":25.027000000000001,"15min":25.027000000000001},"last":3.7589999999999999}]},{"osd":2,"last update":"Fri Feb 9 17:44:25 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.246,"15min":1.246},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":3.8919999999999999,"5min":3.8919999999999999,"15min":3.8919999999999999},"last":3.6640000000000001},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":3.9470000000000001,"5min":3.9470000000000001,"15min":3.9470000000000001},"last":3.5699999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:42:38 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":1.542,"15min":1.6379999999999999},"min":{"1min":0.316,"5min":0.29499999999999998,"15min":0.28699999999999998},"max":{"1min":2.347,"5min":15.228999999999999,"15min":24.030999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.75600000000000001,"5min":1.8340000000000001,"15min":1.8640000000000001},"min":{"1min":0.36199999999999999,"5min":0.33600000000000002,"15min":0.30299999999999999},"max":{"1min":2.3719999999999999,"5min":14.988,"15min":24.318999999999999},"last":0.69399999999999995}]},{"osd":4,"last update":"Fri Feb 9 17:43:42 2024","interfaces":[{"interface":"back","average":{"1min":2.7120000000000002,"5min":1.7050000000000001,"15min":1.5760000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.27500000000000002},"max":{"1min":7.9649999999999999,"5min":12.936,"15min":53.060000000000002},"last":7.0369999999999999},{"interface":"front","average":{"1min":2.552,"5min":1.71,"15min":1.681},"min":{"1min":0.32300000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":9.6140000000000008,"5min":9.6140000000000008,"15min":53.311999999999998},"last":6.9779999999999998}]},{"osd":5,"last update":"Fri Feb 9 17:44:16 2024","interfaces":[{"interface":"back","average":{"1min":2.4689999999999999,"5min":2.4689999999999999,"15min":2.4689999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":6.4249999999999998,"5min":6.4249999999999998,"15min":6.4249999999999998},"last":3.6019999999999999},{"interface":"front","average":{"1min":2.1779999999999999,"5min":2.1779999999999999,"15min":2.1779999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.383,"5min":6.383,"15min":6.383},"last":0.45000000000000001}]},{"osd":6,"last update":"Fri Feb 9 17:43:42 2024","interfaces":[{"interface":"back","average":{"1min":1.554,"5min":1.5129999999999999,"15min":1.573},"min":{"1min":0.33000000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":4.9450000000000003,"5min":9.5,"15min":37.356999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":2.1150000000000002,"5min":1.6439999999999999,"15min":1.4199999999999999},"min":{"1min":0.36099999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":5.2370000000000001,"5min":7.702,"15min":37.25},"last":4.4349999999999996}]},{"osd":7,"last update":"Fri Feb 9 17:43:42 2024","interfaces":[{"interface":"back","average":{"1min":2.2999999999999998,"5min":1.718,"15min":1.659},"min":{"1min":0.51400000000000001,"5min":0.39600000000000002,"15min":0.32800000000000001},"max":{"1min":4.4989999999999997,"5min":9.0749999999999993,"15min":37.585000000000001},"last":11.278},{"interface":"front","average":{"1min":2.343,"5min":1.7589999999999999,"15min":1.704},"min":{"1min":0.53200000000000003,"5min":0.46200000000000002,"15min":0.312},"max":{"1min":4.5529999999999999,"5min":9.032,"15min":37.918999999999997},"last":11.313000000000001}]},{"osd":9,"last update":"Fri Feb 9 17:43:42 2024","interfaces":[{"interface":"back","average":{"1min":3.6520000000000001,"5min":1.6200000000000001,"15min":1.2490000000000001},"min":{"1min":0.41199999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":45.616,"5min":45.616,"15min":45.616},"last":1.2829999999999999},{"interface":"front","average":{"1min":3.6190000000000002,"5min":1.6559999999999999,"15min":1.3089999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":45.718000000000004,"5min":45.718000000000004,"15min":45.718000000000004},"last":0.374}]},{"osd":10,"last update":"Fri Feb 9 17:43:42 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.143,"15min":1.0620000000000001},"min":{"1min":0.34699999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":5.4589999999999996,"5min":7.7270000000000003,"15min":10.65},"last":0.41299999999999998},{"interface":"front","average":{"1min":1.75,"5min":1.581,"15min":1.1930000000000001},"min":{"1min":0.40000000000000002,"5min":0.34799999999999998,"15min":0.318},"max":{"1min":6.9539999999999997,"5min":11.071,"15min":11.071},"last":1.242}]},{"osd":11,"last update":"Fri Feb 9 17:43:42 2024","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.335,"15min":1.3149999999999999},"min":{"1min":0.40000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":4.1029999999999998,"5min":8.5519999999999996,"15min":14.433999999999999},"last":0.48699999999999999},{"interface":"front","average":{"1min":1.798,"5min":1.4219999999999999,"15min":1.2729999999999999},"min":{"1min":0.41399999999999998,"5min":0.308,"15min":0.308},"max":{"1min":4.6180000000000003,"5min":7.4359999999999999,"15min":11.622},"last":3.6309999999999998}]}]},{"osd":7,"up_from":57,"seq":244813136158,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1121580,"kb_used_data":584996,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":92623572,"statfs":{"total":95995035648,"available":94846537728,"internally_reserved":0,"allocated":599035904,"data_stored":598604832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":549386698},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:44:24 2024","interfaces":[{"interface":"back","average":{"1min":1.804,"5min":1.3029999999999999,"15min":2.2850000000000001},"min":{"1min":0.27400000000000002,"5min":0.26000000000000001,"15min":0.252},"max":{"1min":5.9900000000000002,"5min":11.74,"15min":36.753},"last":0.81200000000000006},{"interface":"front","average":{"1min":1.8620000000000001,"5min":1.5,"15min":2.4449999999999998},"min":{"1min":0.27000000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":5.9429999999999996,"5min":11.843999999999999,"15min":36.918999999999997},"last":0.88500000000000001}]},{"osd":1,"last update":"Fri Feb 9 17:44:24 2024","interfaces":[{"interface":"back","average":{"1min":3.1659999999999999,"5min":2.1070000000000002,"15min":2.1200000000000001},"min":{"1min":0.26200000000000001,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":12.318,"5min":15.509,"15min":21.704999999999998},"last":0.57899999999999996},{"interface":"front","average":{"1min":4.0739999999999998,"5min":2.2919999999999998,"15min":2.0670000000000002},"min":{"1min":0.31,"5min":0.27000000000000002,"15min":0.26500000000000001},"max":{"1min":12.359,"5min":15.276,"15min":21.858000000000001},"last":0.622}]},{"osd":2,"last update":"Fri Feb 9 17:44:24 2024","interfaces":[{"interface":"back","average":{"1min":2.0059999999999998,"5min":2.0059999999999998,"15min":2.0059999999999998},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":6.6340000000000003,"5min":6.6340000000000003,"15min":6.6340000000000003},"last":5.306},{"interface":"front","average":{"1min":2.468,"5min":2.468,"15min":2.468},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":11.968,"5min":11.968,"15min":11.968},"last":0.79700000000000004}]},{"osd":3,"last update":"Fri Feb 9 17:43:42 2024","interfaces":[{"interface":"back","average":{"1min":1.6930000000000001,"5min":1.6299999999999999,"15min":1.871},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.26900000000000002},"max":{"1min":11.773999999999999,"5min":11.773999999999999,"15min":27.719000000000001},"last":0.746},{"interface":"front","average":{"1min":2.7450000000000001,"5min":1.772,"15min":1.8089999999999999},"min":{"1min":0.46300000000000002,"5min":0.32000000000000001,"15min":0.26500000000000001},"max":{"1min":12.393000000000001,"5min":12.393000000000001,"15min":24.457000000000001},"last":0.72799999999999998}]},{"osd":4,"last update":"Fri Feb 9 17:44:24 2024","interfaces":[{"interface":"back","average":{"1min":2.2450000000000001,"5min":1.3859999999999999,"15min":0.92800000000000005},"min":{"1min":0.30399999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":5.2629999999999999,"5min":5.29,"15min":5.29},"last":1.798},{"interface":"front","average":{"1min":1.8540000000000001,"5min":1.6259999999999999,"15min":0.94599999999999995},"min":{"1min":0.27100000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":8.1440000000000001,"5min":11.967000000000001,"15min":11.967000000000001},"last":2.3039999999999998}]},{"osd":5,"last update":"Fri Feb 9 17:44:24 2024","interfaces":[{"interface":"back","average":{"1min":3.1040000000000001,"5min":1.8999999999999999,"15min":1.0620000000000001},"min":{"1min":0.371,"5min":0.249,"15min":0.249},"max":{"1min":9.2170000000000005,"5min":9.6590000000000007,"15min":9.6590000000000007},"last":0.76700000000000002},{"interface":"front","average":{"1min":3.3460000000000001,"5min":2.0750000000000002,"15min":1.03},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":8.9600000000000009,"5min":15.228999999999999,"15min":15.228999999999999},"last":0.54300000000000004}]},{"osd":6,"last update":"Fri Feb 9 17:43:42 2024","interfaces":[{"interface":"back","average":{"1min":4.5010000000000003,"5min":1.7849999999999999,"15min":1.839},"min":{"1min":0.49099999999999999,"5min":0.28100000000000003,"15min":0.26000000000000001},"max":{"1min":15.275,"5min":15.275,"15min":25.965},"last":0.96099999999999997},{"interface":"front","average":{"1min":3.2280000000000002,"5min":1.6759999999999999,"15min":1.569},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":18.236999999999998,"5min":18.236999999999998,"15min":18.236999999999998},"last":0.91800000000000004}]},{"osd":8,"last update":"Fri Feb 9 17:43:42 2024","interfaces":[{"interface":"back","average":{"1min":2.4350000000000001,"5min":1.546,"15min":1.508},"min":{"1min":0.39200000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":11.81,"5min":11.81,"15min":11.81},"last":0.70599999999999996},{"interface":"front","average":{"1min":2.7730000000000001,"5min":1.4590000000000001,"15min":1.448},"min":{"1min":0.51800000000000002,"5min":0.36599999999999999,"15min":0.33300000000000002},"max":{"1min":12.058,"5min":12.058,"15min":12.058},"last":0.68799999999999994}]},{"osd":9,"last update":"Fri Feb 9 17:43:31 2024","interfaces":[{"interface":"back","average":{"1min":2.867,"5min":1.5820000000000001,"15min":1.702},"min":{"1min":0.42299999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":11.941000000000001,"5min":11.941000000000001,"15min":41.286999999999999},"last":0.78500000000000003},{"interface":"front","average":{"1min":2.8690000000000002,"5min":1.6950000000000001,"15min":1.819},"min":{"1min":0.54100000000000004,"5min":0.379,"15min":0.33400000000000002},"max":{"1min":11.906000000000001,"5min":11.906000000000001,"15min":41.234999999999999},"last":0.83799999999999997}]},{"osd":10,"last update":"Fri Feb 9 17:43:58 2024","interfaces":[{"interface":"back","average":{"1min":2.1850000000000001,"5min":1.599,"15min":1.5720000000000001},"min":{"1min":0.38200000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":9.3030000000000008,"5min":12.050000000000001,"15min":12.050000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":2.7469999999999999,"5min":1.6759999999999999,"15min":1.7150000000000001},"min":{"1min":0.48199999999999998,"5min":0.38900000000000001,"15min":0.35599999999999998},"max":{"1min":11.878,"5min":11.878,"15min":11.878},"last":0.873}]},{"osd":11,"last update":"Fri Feb 9 17:44:24 2024","interfaces":[{"interface":"back","average":{"1min":2.5790000000000002,"5min":1.7130000000000001,"15min":1.8779999999999999},"min":{"1min":0.48099999999999998,"5min":0.42399999999999999,"15min":0.39500000000000002},"max":{"1min":5.9089999999999998,"5min":9.2490000000000006,"15min":21.945},"last":0.82399999999999995},{"interface":"front","average":{"1min":3.4300000000000002,"5min":1.8520000000000001,"15min":1.796},"min":{"1min":0.50700000000000001,"5min":0.45500000000000002,"15min":0.39500000000000002},"max":{"1min":12.423999999999999,"5min":12.423999999999999,"15min":21.731999999999999},"last":0.65700000000000003}]}]},{"osd":6,"up_from":49,"seq":210453397795,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":871268,"kb_used_data":461080,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":92873884,"statfs":{"total":95995035648,"available":95102857216,"internally_reserved":0,"allocated":472145920,"data_stored":471722472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":420018630},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:44:15 2024","interfaces":[{"interface":"back","average":{"1min":2.1509999999999998,"5min":1.776,"15min":1.651},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.21299999999999999},"max":{"1min":5.7619999999999996,"5min":9.2089999999999996,"15min":12.544},"last":1.804},{"interface":"front","average":{"1min":2.0289999999999999,"5min":1.825,"15min":1.599},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.217},"max":{"1min":5.8289999999999997,"5min":9.2520000000000007,"15min":9.2520000000000007},"last":1.528}]},{"osd":1,"last update":"Fri Feb 9 17:44:15 2024","interfaces":[{"interface":"back","average":{"1min":2.2610000000000001,"5min":1.7549999999999999,"15min":1.895},"min":{"1min":0.316,"5min":0.28599999999999998,"15min":0.25900000000000001},"max":{"1min":6.1849999999999996,"5min":8.3260000000000005,"15min":17.762},"last":1.548},{"interface":"front","average":{"1min":2.4430000000000001,"5min":1.8580000000000001,"15min":1.9670000000000001},"min":{"1min":0.34799999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":12.475,"5min":12.475,"15min":17.734999999999999},"last":1.883}]},{"osd":2,"last update":"Fri Feb 9 17:44:15 2024","interfaces":[{"interface":"back","average":{"1min":2.097,"5min":1.6899999999999999,"15min":1.579},"min":{"1min":0.40200000000000002,"5min":0.32000000000000001,"15min":0.22600000000000001},"max":{"1min":5.9470000000000001,"5min":9.8019999999999996,"15min":9.8019999999999996},"last":1.845},{"interface":"front","average":{"1min":2.145,"5min":1.7290000000000001,"15min":1.6379999999999999},"min":{"1min":0.433,"5min":0.372,"15min":0.28000000000000003},"max":{"1min":5.9640000000000004,"5min":9.8439999999999994,"15min":10.156000000000001},"last":1.6000000000000001}]},{"osd":3,"last update":"Fri Feb 9 17:44:15 2024","interfaces":[{"interface":"back","average":{"1min":1.9430000000000001,"5min":1.7030000000000001,"15min":1.9930000000000001},"min":{"1min":0.29599999999999999,"5min":0.27800000000000002,"15min":0.25600000000000001},"max":{"1min":13.016,"5min":13.016,"15min":32.545000000000002},"last":0.53800000000000003},{"interface":"front","average":{"1min":2.2250000000000001,"5min":1.734,"15min":1.974},"min":{"1min":0.40000000000000002,"5min":0.34200000000000003,"15min":0.29599999999999999},"max":{"1min":6.234,"5min":9.8219999999999992,"15min":32.670999999999999},"last":2.782}]},{"osd":4,"last update":"Fri Feb 9 17:44:15 2024","interfaces":[{"interface":"back","average":{"1min":3.2989999999999999,"5min":2.0070000000000001,"15min":1.78},"min":{"1min":0.36799999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":15.061999999999999,"5min":15.061999999999999,"15min":19.798999999999999},"last":2.4649999999999999},{"interface":"front","average":{"1min":3.915,"5min":2.4049999999999998,"15min":1.796},"min":{"1min":0.35099999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":15.118,"5min":15.118,"15min":15.118},"last":1.4990000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:44:15 2024","interfaces":[{"interface":"back","average":{"1min":2.9529999999999998,"5min":2.024,"15min":1.7390000000000001},"min":{"1min":0.47199999999999998,"5min":0.29799999999999999,"15min":0.29499999999999998},"max":{"1min":7.8689999999999998,"5min":9.9849999999999994,"15min":10.968999999999999},"last":1.6339999999999999},{"interface":"front","average":{"1min":3.1560000000000001,"5min":1.8400000000000001,"15min":1.7130000000000001},"min":{"1min":0.39400000000000002,"5min":0.316,"15min":0.30199999999999999},"max":{"1min":11.079000000000001,"5min":11.079000000000001,"15min":13.91},"last":2.4409999999999998}]},{"osd":7,"last update":"Fri Feb 9 17:43:55 2024","interfaces":[{"interface":"back","average":{"1min":3.3839999999999999,"5min":2.0649999999999999,"15min":1.76},"min":{"1min":0.45000000000000001,"5min":0.29399999999999998,"15min":0.28000000000000003},"max":{"1min":7.7359999999999998,"5min":11.052,"15min":14.022},"last":0.29699999999999999},{"interface":"front","average":{"1min":2.641,"5min":1.8200000000000001,"15min":1.8300000000000001},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":12.967000000000001,"5min":12.967000000000001,"15min":15.458},"last":1.5629999999999999}]},{"osd":8,"last update":"Fri Feb 9 17:26:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":0.876,"15min":0.79600000000000004},"min":{"1min":0.39900000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":4.0339999999999998,"5min":4.4880000000000004,"15min":4.4880000000000004},"last":4.2809999999999997},{"interface":"front","average":{"1min":1.0469999999999999,"5min":0.88600000000000001,"15min":0.78300000000000003},"min":{"1min":0.375,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":3.6539999999999999,"5min":4.8570000000000002,"15min":4.8570000000000002},"last":2.9529999999999998}]},{"osd":9,"last update":"Fri Feb 9 17:44:20 2024","interfaces":[{"interface":"back","average":{"1min":2.153,"5min":1.9410000000000001,"15min":1.5209999999999999},"min":{"1min":0.58899999999999997,"5min":0.438,"15min":0.35899999999999999},"max":{"1min":5.2599999999999998,"5min":10.779999999999999,"15min":10.779999999999999},"last":1.5820000000000001},{"interface":"front","average":{"1min":2.4580000000000002,"5min":2.024,"15min":1.641},"min":{"1min":0.495,"5min":0.40999999999999998,"15min":0.40400000000000003},"max":{"1min":7.5380000000000003,"5min":7.5380000000000003,"15min":8.1630000000000003},"last":1.615}]},{"osd":10,"last update":"Fri Feb 9 17:43:55 2024","interfaces":[{"interface":"back","average":{"1min":3.6720000000000002,"5min":2.02,"15min":1.643},"min":{"1min":0.50800000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":25.370000000000001,"5min":25.370000000000001,"15min":25.370000000000001},"last":2.4849999999999999},{"interface":"front","average":{"1min":3.552,"5min":1.907,"15min":1.6799999999999999},"min":{"1min":0.374,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":26.027999999999999,"5min":26.027999999999999,"15min":26.027999999999999},"last":0.39800000000000002}]},{"osd":11,"last update":"Fri Feb 9 17:44:15 2024","interfaces":[{"interface":"back","average":{"1min":2.3839999999999999,"5min":1.9650000000000001,"15min":1.661},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.36099999999999999},"max":{"1min":13.109,"5min":13.109,"15min":13.109},"last":0.42699999999999999},{"interface":"front","average":{"1min":2.0779999999999998,"5min":1.8480000000000001,"15min":1.639},"min":{"1min":0.38800000000000001,"5min":0.373,"15min":0.373},"max":{"1min":13.164999999999999,"5min":13.164999999999999,"15min":13.164999999999999},"last":0.45900000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575165,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1208404,"kb_used_data":613580,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":92536748,"statfs":{"total":95995035648,"available":94757629952,"internally_reserved":0,"allocated":628305920,"data_stored":627886937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:43:31 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":1.4930000000000001,"15min":1.6020000000000001},"min":{"1min":0.25600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":2.6629999999999998,"5min":15.991,"15min":19.079000000000001},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.45100000000000001,"5min":1.294,"15min":1.597},"min":{"1min":0.23499999999999999,"5min":0.218,"15min":0.19400000000000001},"max":{"1min":1.6599999999999999,"5min":12.220000000000001,"15min":23.573},"last":0.49099999999999999}]},{"osd":2,"last update":"Fri Feb 9 17:43:57 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":1.423,"15min":1.528},"min":{"1min":0.246,"5min":0.23400000000000001,"15min":0.22600000000000001},"max":{"1min":4.4160000000000004,"5min":12.52,"15min":26.361999999999998},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.89100000000000001,"5min":1.3700000000000001,"15min":1.8180000000000001},"min":{"1min":0.27800000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":3.4870000000000001,"5min":8.2360000000000007,"15min":26.087},"last":0.52500000000000002}]},{"osd":3,"last update":"Fri Feb 9 17:44:18 2024","interfaces":[{"interface":"back","average":{"1min":1.548,"5min":1.548,"15min":1.548},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":6.7649999999999997,"5min":6.7649999999999997,"15min":6.7649999999999997},"last":0.38400000000000001},{"interface":"front","average":{"1min":1.012,"5min":1.012,"15min":1.012},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":4.4790000000000001,"5min":4.4790000000000001,"15min":4.4790000000000001},"last":0.35099999999999998}]},{"osd":4,"last update":"Fri Feb 9 17:44:27 2024","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.476,"15min":1.8140000000000001},"min":{"1min":0.32700000000000001,"5min":0.27700000000000002,"15min":0.253},"max":{"1min":3.907,"5min":6.7679999999999998,"15min":11.766999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":2.8130000000000002,"5min":1.7809999999999999,"15min":1.9339999999999999},"min":{"1min":0.35099999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":13.871,"5min":14.593,"15min":14.593},"last":0.66000000000000003}]},{"osd":5,"last update":"Fri Feb 9 17:44:13 2024","interfaces":[{"interface":"back","average":{"1min":1.5840000000000001,"5min":1.6970000000000001,"15min":1.8069999999999999},"min":{"1min":0.35899999999999999,"5min":0.29199999999999998,"15min":0.27000000000000002},"max":{"1min":6.2850000000000001,"5min":6.8250000000000002,"15min":13.544},"last":0.47799999999999998},{"interface":"front","average":{"1min":1.6739999999999999,"5min":1.4970000000000001,"15min":1.7470000000000001},"min":{"1min":0.40500000000000003,"5min":0.32100000000000001,"15min":0.26800000000000002},"max":{"1min":6.609,"5min":10.263999999999999,"15min":12.117000000000001},"last":0.53900000000000003}]},{"osd":6,"last update":"Fri Feb 9 17:44:18 2024","interfaces":[{"interface":"back","average":{"1min":2.387,"5min":1.5820000000000001,"15min":1.3759999999999999},"min":{"1min":0.377,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":12.57,"5min":12.57,"15min":12.57},"last":0.59599999999999997},{"interface":"front","average":{"1min":2.4369999999999998,"5min":1.5660000000000001,"15min":1.3049999999999999},"min":{"1min":0.51700000000000002,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":12.605,"5min":12.605,"15min":12.605},"last":0.64100000000000001}]},{"osd":7,"last update":"Fri Feb 9 17:44:18 2024","interfaces":[{"interface":"back","average":{"1min":1.8460000000000001,"5min":1.391,"15min":1.2350000000000001},"min":{"1min":0.45200000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":6.5830000000000002,"5min":6.5830000000000002,"15min":6.5830000000000002},"last":0.67600000000000005},{"interface":"front","average":{"1min":1.8859999999999999,"5min":1.552,"15min":1.3420000000000001},"min":{"1min":0.48199999999999998,"5min":0.309,"15min":0.309},"max":{"1min":6.5599999999999996,"5min":6.5599999999999996,"15min":6.5599999999999996},"last":0.69599999999999995}]},{"osd":8,"last update":"Fri Feb 9 17:44:27 2024","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":0.96599999999999997,"15min":0.91600000000000004},"min":{"1min":0.372,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":3.3399999999999999,"5min":9.2520000000000007,"15min":9.2520000000000007},"last":1.004},{"interface":"front","average":{"1min":1.2450000000000001,"5min":0.97899999999999998,"15min":0.89300000000000002},"min":{"1min":0.42899999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":3.367,"5min":9.2810000000000006,"15min":9.2810000000000006},"last":1.034}]},{"osd":9,"last update":"Fri Feb 9 17:43:31 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":1.4330000000000001,"15min":1.593},"min":{"1min":0.34200000000000003,"5min":0.33400000000000002,"15min":0.32400000000000001},"max":{"1min":2.6240000000000001,"5min":7.0640000000000001,"15min":9.5839999999999996},"last":0.92000000000000004},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.76,"15min":1.6479999999999999},"min":{"1min":0.38800000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":4.4349999999999996,"5min":10.548,"15min":12.117000000000001},"last":0.82099999999999995}]},{"osd":10,"last update":"Fri Feb 9 17:44:13 2024","interfaces":[{"interface":"back","average":{"1min":1.482,"5min":1.5429999999999999,"15min":1.6080000000000001},"min":{"1min":0.55800000000000005,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":6.1769999999999996,"5min":15.083,"15min":15.083},"last":1.1200000000000001},{"interface":"front","average":{"1min":1.6160000000000001,"5min":1.605,"15min":1.702},"min":{"1min":0.61499999999999999,"5min":0.46700000000000003,"15min":0.40899999999999997},"max":{"1min":6.2140000000000004,"5min":15.121,"15min":15.121},"last":1.095}]},{"osd":11,"last update":"Fri Feb 9 17:44:18 2024","interfaces":[{"interface":"back","average":{"1min":1.8320000000000001,"5min":1.2609999999999999,"15min":1.0529999999999999},"min":{"1min":0.441,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":6.5449999999999999,"5min":6.5449999999999999,"15min":6.5449999999999999},"last":0.89200000000000002},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.2210000000000001,"15min":1.224},"min":{"1min":0.63200000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":3.2999999999999998,"5min":3.657,"15min":3.657},"last":1.0669999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738689,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1732052,"kb_used_data":1105292,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":92013100,"statfs":{"total":95995035648,"available":94221414400,"internally_reserved":0,"allocated":1131819008,"data_stored":1131400025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 9 17:44:13 2024","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.2889999999999999,"15min":1.3540000000000001},"min":{"1min":0.217,"5min":0.20599999999999999,"15min":0.20499999999999999},"max":{"1min":4.7530000000000001,"5min":13.842000000000001,"15min":13.842000000000001},"last":0.24299999999999999},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.411,"15min":1.5009999999999999},"min":{"1min":0.28299999999999997,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":4.7889999999999997,"5min":14.122,"15min":14.122},"last":0.32700000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:39:25 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":5.5339999999999998,"5min":5.5339999999999998,"15min":5.5339999999999998},"last":1.744},{"interface":"front","average":{"1min":1.593,"5min":1.593,"15min":1.593},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":13.473000000000001,"5min":13.473000000000001,"15min":13.473000000000001},"last":3.109}]},{"osd":3,"last update":"Fri Feb 9 17:43:49 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":1.6850000000000001,"15min":2.577},"min":{"1min":0.32100000000000001,"5min":0.247,"15min":0.247},"max":{"1min":4.5750000000000002,"5min":16.756,"15min":16.756},"last":0.307},{"interface":"front","average":{"1min":1.337,"5min":1.264,"15min":1.625},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":6.5570000000000004,"5min":6.5570000000000004,"15min":6.5570000000000004},"last":1.0129999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:43:34 2024","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":1.1060000000000001,"15min":1.617},"min":{"1min":0.32900000000000001,"5min":0.28199999999999997,"15min":0.26400000000000001},"max":{"1min":7.7830000000000004,"5min":7.7830000000000004,"15min":14.419},"last":2.6539999999999999},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.609,"15min":1.8480000000000001},"min":{"1min":0.371,"5min":0.29399999999999998,"15min":0.27200000000000002},"max":{"1min":7.8230000000000004,"5min":13.308999999999999,"15min":14.465999999999999},"last":2.6720000000000002}]},{"osd":5,"last update":"Fri Feb 9 17:43:52 2024","interfaces":[{"interface":"back","average":{"1min":1.611,"5min":1.371,"15min":1.569},"min":{"1min":0.314,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":7.9690000000000003,"5min":8.4079999999999995,"15min":9.8569999999999993},"last":2.5230000000000001},{"interface":"front","average":{"1min":1.734,"5min":1.206,"15min":1.675},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":7.9950000000000001,"5min":7.9950000000000001,"15min":9.9049999999999994},"last":2.496}]},{"osd":6,"last update":"Fri Feb 9 17:44:24 2024","interfaces":[{"interface":"back","average":{"1min":1.837,"5min":1.466,"15min":1.706},"min":{"1min":0.38500000000000001,"5min":0.28699999999999998,"15min":0.27800000000000002},"max":{"1min":6.4489999999999998,"5min":6.4489999999999998,"15min":13.407},"last":2.1880000000000002},{"interface":"front","average":{"1min":1.8700000000000001,"5min":1.51,"15min":1.772},"min":{"1min":0.42199999999999999,"5min":0.33300000000000002,"15min":0.33000000000000002},"max":{"1min":6.4749999999999996,"5min":6.4749999999999996,"15min":13.448},"last":2.1560000000000001}]},{"osd":7,"last update":"Fri Feb 9 17:43:49 2024","interfaces":[{"interface":"back","average":{"1min":2.161,"5min":1.847,"15min":1.9850000000000001},"min":{"1min":0.312,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":10.023999999999999,"5min":13.349,"15min":13.349},"last":1.8600000000000001},{"interface":"front","average":{"1min":2.5920000000000001,"5min":2.0790000000000002,"15min":1.9870000000000001},"min":{"1min":0.39600000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":9.6980000000000004,"5min":14.451000000000001,"15min":16.385000000000002},"last":0.52200000000000002}]},{"osd":8,"last update":"Fri Feb 9 17:43:49 2024","interfaces":[{"interface":"back","average":{"1min":1.853,"5min":1.8400000000000001,"15min":1.5349999999999999},"min":{"1min":0.38300000000000001,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":9.5359999999999996,"5min":14.323,"15min":14.323},"last":0.42699999999999999},{"interface":"front","average":{"1min":1.823,"5min":1.9390000000000001,"15min":1.655},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":9.5700000000000003,"5min":14.481999999999999,"15min":16.41},"last":0.57099999999999995}]},{"osd":9,"last update":"Fri Feb 9 17:43:34 2024","interfaces":[{"interface":"back","average":{"1min":2.9340000000000002,"5min":2.2120000000000002,"15min":1.8260000000000001},"min":{"1min":0.38300000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":9.6010000000000009,"5min":17.952000000000002,"15min":17.952000000000002},"last":0.496},{"interface":"front","average":{"1min":2.5289999999999999,"5min":2.1640000000000001,"15min":1.7809999999999999},"min":{"1min":0.436,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":10.858000000000001,"5min":17.913,"15min":17.913},"last":0.60599999999999998}]},{"osd":10,"last update":"Fri Feb 9 17:43:52 2024","interfaces":[{"interface":"back","average":{"1min":2.4580000000000002,"5min":2.3119999999999998,"15min":1.833},"min":{"1min":0.40699999999999997,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":9.7270000000000003,"5min":31.766999999999999,"15min":31.766999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.343,"15min":1.47},"min":{"1min":0.41399999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":7.6849999999999996,"5min":28.048999999999999,"15min":28.048999999999999},"last":0.54300000000000004}]},{"osd":11,"last update":"Fri Feb 9 17:44:24 2024","interfaces":[{"interface":"back","average":{"1min":2.0750000000000002,"5min":1.335,"15min":1.583},"min":{"1min":0.442,"5min":0.41899999999999998,"15min":0.40600000000000003},"max":{"1min":13.102,"5min":13.102,"15min":40.021000000000001},"last":0.56200000000000006},{"interface":"front","average":{"1min":1.464,"5min":1.351,"15min":1.615},"min":{"1min":0.46800000000000003,"5min":0.46500000000000002,"15min":0.45200000000000001},"max":{"1min":4.7839999999999998,"5min":5.8390000000000004,"15min":40.055999999999997},"last":0.58599999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411639,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":578332,"kb_used_data":330580,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":93166820,"statfs":{"total":95995035648,"available":95402823680,"internally_reserved":0,"allocated":338513920,"data_stored":338091385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":253688261},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":1.4510000000000001,"15min":1.006},"min":{"1min":0.29999999999999999,"5min":0.23699999999999999,"15min":0.221},"max":{"1min":4.9089999999999998,"5min":17.251000000000001,"15min":17.251000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.74399999999999999,"5min":1.611,"15min":1.113},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.22900000000000001},"max":{"1min":4.7720000000000002,"5min":23.097999999999999,"15min":23.097999999999999},"last":0.875}]},{"osd":1,"last update":"Fri Feb 9 17:43:44 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":1.677,"15min":1.536},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.86199999999999999,"5min":15.43,"15min":22.306000000000001},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.58199999999999996,"5min":1.627,"15min":1.629},"min":{"1min":0.26900000000000002,"5min":0.252,"15min":0.214},"max":{"1min":1.3400000000000001,"5min":18.181000000000001,"15min":44.838999999999999},"last":1.4239999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:44:17 2024","interfaces":[{"interface":"back","average":{"1min":2.0339999999999998,"5min":1.6379999999999999,"15min":1.514},"min":{"1min":0.29599999999999999,"5min":0.249,"15min":0.22700000000000001},"max":{"1min":14.355,"5min":17.942,"15min":24.353000000000002},"last":0.379},{"interface":"front","average":{"1min":2.4089999999999998,"5min":1.3879999999999999,"15min":1.6259999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.6099999999999994,"5min":10.351000000000001,"15min":19.986999999999998},"last":0.32300000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:44:17 2024","interfaces":[{"interface":"back","average":{"1min":1.653,"5min":1.397,"15min":1.7509999999999999},"min":{"1min":0.32100000000000001,"5min":0.29399999999999998,"15min":0.26300000000000001},"max":{"1min":8.7230000000000008,"5min":10.048999999999999,"15min":13.526},"last":0.42699999999999999},{"interface":"front","average":{"1min":1.8839999999999999,"5min":1.49,"15min":1.6080000000000001},"min":{"1min":0.29499999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":6.1660000000000004,"5min":8.9920000000000009,"15min":9.9619999999999997},"last":0.39600000000000002}]},{"osd":5,"last update":"Fri Feb 9 17:44:00 2024","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":1.6910000000000001,"15min":1.746},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":3.5219999999999998,"5min":42.664000000000001,"15min":42.664000000000001},"last":0.44400000000000001},{"interface":"front","average":{"1min":2.0579999999999998,"5min":2.145,"15min":1.964},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":8.6600000000000001,"5min":43.311999999999998,"15min":43.311999999999998},"last":3.2519999999999998}]},{"osd":6,"last update":"Fri Feb 9 17:44:17 2024","interfaces":[{"interface":"back","average":{"1min":2.6760000000000002,"5min":1.7330000000000001,"15min":1.762},"min":{"1min":0.34200000000000003,"5min":0.312,"15min":0.30099999999999999},"max":{"1min":7.8949999999999996,"5min":10.114000000000001,"15min":10.295},"last":0.58299999999999996},{"interface":"front","average":{"1min":2.6080000000000001,"5min":1.8520000000000001,"15min":1.9830000000000001},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":7.726,"5min":10.144,"15min":11.269},"last":0.52000000000000002}]},{"osd":7,"last update":"Fri Feb 9 17:43:44 2024","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.8029999999999999,"15min":2.004},"min":{"1min":0.374,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":4.7679999999999998,"5min":11.082000000000001,"15min":41.802999999999997},"last":1.2869999999999999},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.8440000000000001,"15min":2.056},"min":{"1min":0.35499999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":4.7530000000000001,"5min":11.099,"15min":41.783999999999999},"last":1.268}]},{"osd":8,"last update":"Fri Feb 9 17:44:17 2024","interfaces":[{"interface":"back","average":{"1min":1.137,"5min":1.401,"15min":1.3200000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":8.7729999999999997,"5min":8.7729999999999997,"15min":22.393999999999998},"last":0.59499999999999997},{"interface":"front","average":{"1min":1.5629999999999999,"5min":1.3520000000000001,"15min":1.5049999999999999},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":3.4729999999999999,"5min":12.58,"15min":12.58},"last":0.49299999999999999}]},{"osd":9,"last update":"Fri Feb 9 17:44:28 2024","interfaces":[{"interface":"back","average":{"1min":2.0920000000000001,"5min":1.5129999999999999,"15min":1.6100000000000001},"min":{"1min":0.37,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":12.768000000000001,"5min":12.768000000000001,"15min":13.5},"last":0.56899999999999995},{"interface":"front","average":{"1min":1.929,"5min":1.3660000000000001,"15min":1.5720000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.36399999999999999},"max":{"1min":7.5099999999999998,"5min":9.0060000000000002,"15min":13.589},"last":0.60899999999999999}]},{"osd":10,"last update":"Fri Feb 9 17:44:10 2024","interfaces":[{"interface":"back","average":{"1min":1.877,"5min":1.611,"15min":1.5269999999999999},"min":{"1min":0.42199999999999999,"5min":0.38400000000000001,"15min":0.33600000000000002},"max":{"1min":8.7509999999999994,"5min":9.1419999999999995,"15min":22.425999999999998},"last":0.622},{"interface":"front","average":{"1min":1.905,"5min":1.643,"15min":1.526},"min":{"1min":0.438,"5min":0.438,"15min":0.379},"max":{"1min":8.8109999999999999,"5min":9.1899999999999995,"15min":22.454999999999998},"last":0.64400000000000002}]},{"osd":11,"last update":"Fri Feb 9 17:44:17 2024","interfaces":[{"interface":"back","average":{"1min":1.738,"5min":1.514,"15min":2.04},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.39100000000000001},"max":{"1min":5.1580000000000004,"5min":7.4420000000000002,"15min":137.44300000000001},"last":0.55300000000000005},{"interface":"front","average":{"1min":1.5980000000000001,"5min":1.494,"15min":2.1000000000000001},"min":{"1min":0.45700000000000002,"5min":0.41199999999999998,"15min":0.40100000000000002},"max":{"1min":8.7699999999999996,"5min":9.1029999999999998,"15min":137.64400000000001},"last":0.53500000000000003}]}]},{"osd":3,"up_from":27,"seq":115964117300,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":563504,"kb_used_data":144996,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":93181648,"statfs":{"total":95995035648,"available":95418007552,"internally_reserved":0,"allocated":148475904,"data_stored":148047941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":428538311},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:44:14 2024","interfaces":[{"interface":"back","average":{"1min":1.657,"5min":1.6240000000000001,"15min":1.77},"min":{"1min":0.28499999999999998,"5min":0.23300000000000001,"15min":0.186},"max":{"1min":12.789,"5min":25.849,"15min":38.984999999999999},"last":3.581},{"interface":"front","average":{"1min":1.6100000000000001,"5min":1.718,"15min":1.8959999999999999},"min":{"1min":0.29499999999999998,"5min":0.248,"15min":0.22500000000000001},"max":{"1min":11.932,"5min":30.268999999999998,"15min":41.497999999999998},"last":0.57799999999999996}]},{"osd":1,"last update":"Fri Feb 9 17:40:06 2024","interfaces":[{"interface":"back","average":{"1min":4.5670000000000002,"5min":4.5670000000000002,"15min":4.5670000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":25.965,"5min":25.965,"15min":25.965},"last":0.35299999999999998},{"interface":"front","average":{"1min":4.5129999999999999,"5min":4.5129999999999999,"15min":4.5129999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":31.454000000000001,"5min":31.454000000000001,"15min":31.454000000000001},"last":0.375}]},{"osd":2,"last update":"Fri Feb 9 17:44:10 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.3520000000000001,"15min":1.4419999999999999},"min":{"1min":0.29099999999999998,"5min":0.255,"15min":0.22600000000000001},"max":{"1min":4.6749999999999998,"5min":11,"15min":14.888999999999999},"last":0.92800000000000005},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.456,"15min":1.538},"min":{"1min":0.311,"5min":0.24199999999999999,"15min":0.20200000000000001},"max":{"1min":5.2050000000000001,"5min":12.347,"15min":17.613},"last":0.89200000000000002}]},{"osd":4,"last update":"Fri Feb 9 17:44:14 2024","interfaces":[{"interface":"back","average":{"1min":1.9259999999999999,"5min":1.429,"15min":1.6040000000000001},"min":{"1min":0.36099999999999999,"5min":0.29099999999999998,"15min":0.26800000000000002},"max":{"1min":11.880000000000001,"5min":11.880000000000001,"15min":34.898000000000003},"last":2.0009999999999999},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.4570000000000001,"15min":1.6599999999999999},"min":{"1min":0.29999999999999999,"5min":0.29399999999999998,"15min":0.27900000000000003},"max":{"1min":11.622999999999999,"5min":12.404999999999999,"15min":14.061999999999999},"last":1.887}]},{"osd":5,"last update":"Fri Feb 9 17:43:52 2024","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":1.639,"15min":1.663},"min":{"1min":0.313,"5min":0.28000000000000003,"15min":0.27800000000000002},"max":{"1min":4.6980000000000004,"5min":11.007999999999999,"15min":17.651},"last":1.5760000000000001},{"interface":"front","average":{"1min":1.649,"5min":1.7370000000000001,"15min":1.7290000000000001},"min":{"1min":0.42599999999999999,"5min":0.35299999999999998,"15min":0.34100000000000003},"max":{"1min":7.6219999999999999,"5min":11.036,"15min":17.675999999999998},"last":1.609}]},{"osd":6,"last update":"Fri Feb 9 17:43:43 2024","interfaces":[{"interface":"back","average":{"1min":2.1419999999999999,"5min":1.925,"15min":2.4710000000000001},"min":{"1min":0.48999999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":6.4109999999999996,"5min":12.717000000000001,"15min":12.717000000000001},"last":1.1419999999999999},{"interface":"front","average":{"1min":2.423,"5min":1.524,"15min":1.4450000000000001},"min":{"1min":0.58599999999999997,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":8.6059999999999999,"5min":8.7360000000000007,"15min":8.7360000000000007},"last":1.1200000000000001}]},{"osd":7,"last update":"Fri Feb 9 17:44:25 2024","interfaces":[{"interface":"back","average":{"1min":2.6160000000000001,"5min":1.6779999999999999,"15min":1.1399999999999999},"min":{"1min":0.46100000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":11.769,"5min":11.769,"15min":11.769},"last":1.47},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.1080000000000001,"15min":0.872},"min":{"1min":0.432,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":6.0990000000000002,"5min":6.0990000000000002,"15min":6.0990000000000002},"last":1.1779999999999999}]},{"osd":8,"last update":"Fri Feb 9 17:44:14 2024","interfaces":[{"interface":"back","average":{"1min":2.0910000000000002,"5min":1.5109999999999999,"15min":1.472},"min":{"1min":0.377,"5min":0.377,"15min":0.26800000000000002},"max":{"1min":11.821,"5min":11.821,"15min":11.821},"last":0.69699999999999995},{"interface":"front","average":{"1min":1.7669999999999999,"5min":1.4410000000000001,"15min":1.325},"min":{"1min":0.371,"5min":0.36199999999999999,"15min":0.34499999999999997},"max":{"1min":5.3520000000000003,"5min":10.388999999999999,"15min":14.129},"last":0.82399999999999995}]},{"osd":9,"last update":"Fri Feb 9 17:43:36 2024","interfaces":[{"interface":"back","average":{"1min":1.6419999999999999,"5min":1.323,"15min":1.498},"min":{"1min":0.47399999999999998,"5min":0.36199999999999999,"15min":0.32500000000000001},"max":{"1min":8.25,"5min":8.8160000000000007,"15min":9.7929999999999993},"last":0.62},{"interface":"front","average":{"1min":1.6120000000000001,"5min":1.468,"15min":1.5029999999999999},"min":{"1min":0.52200000000000002,"5min":0.41199999999999998,"15min":0.32800000000000001},"max":{"1min":8.2260000000000009,"5min":8.7899999999999991,"15min":13.717000000000001},"last":0.98699999999999999}]},{"osd":10,"last update":"Fri Feb 9 17:44:25 2024","interfaces":[{"interface":"back","average":{"1min":2.1589999999999998,"5min":1.2090000000000001,"15min":0.97699999999999998},"min":{"1min":0.41099999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":9.7080000000000002,"5min":9.7080000000000002,"15min":9.7080000000000002},"last":1.159},{"interface":"front","average":{"1min":1.6890000000000001,"5min":1.0249999999999999,"15min":0.84199999999999997},"min":{"1min":0.39100000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":5.2469999999999999,"5min":5.2469999999999999,"15min":5.2469999999999999},"last":0.85099999999999998}]},{"osd":11,"last update":"Fri Feb 9 17:44:14 2024","interfaces":[{"interface":"back","average":{"1min":1.5289999999999999,"5min":1.262,"15min":1.4630000000000001},"min":{"1min":0.439,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":5.9329999999999998,"5min":8.8849999999999998,"15min":14.300000000000001},"last":0.67300000000000004},{"interface":"front","average":{"1min":1.738,"5min":1.5449999999999999,"15min":1.6080000000000001},"min":{"1min":0.48299999999999998,"5min":0.442,"15min":0.39300000000000002},"max":{"1min":5.9160000000000004,"5min":9.4390000000000001,"15min":17.145},"last":0.90000000000000002}]}]},{"osd":4,"up_from":32,"seq":137438953773,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1461092,"kb_used_data":898200,"kb_used_omap":1,"kb_used_meta":562878,"kb_avail":92284060,"statfs":{"total":95995035648,"available":94498877440,"internally_reserved":0,"allocated":919756800,"data_stored":919333352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":576387525},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:44:22 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.107,"15min":1.75},"min":{"1min":0.27300000000000002,"5min":0.222,"15min":0.222},"max":{"1min":4.7679999999999998,"5min":7.7199999999999998,"15min":50.356999999999999},"last":0.32400000000000001},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.169,"15min":1.679},"min":{"1min":0.372,"5min":0.26700000000000002,"15min":0.247},"max":{"1min":4.8680000000000003,"5min":7.6390000000000002,"15min":50.569000000000003},"last":0.38900000000000001}]},{"osd":1,"last update":"Fri Feb 9 17:44:22 2024","interfaces":[{"interface":"back","average":{"1min":1.9299999999999999,"5min":1.548,"15min":1.9870000000000001},"min":{"1min":0.253,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":8.0109999999999992,"5min":8.0109999999999992,"15min":50.473999999999997},"last":1.778},{"interface":"front","average":{"1min":2.1469999999999998,"5min":1.9410000000000001,"15min":2.1960000000000002},"min":{"1min":0.32800000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":5.4189999999999996,"5min":11.593,"15min":50.499000000000002},"last":0.255}]},{"osd":2,"last update":"Fri Feb 9 17:28:51 2024","interfaces":[{"interface":"back","average":{"1min":2.536,"5min":1.546,"15min":0.91400000000000003},"min":{"1min":0.29999999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":8.8819999999999997,"5min":8.8819999999999997,"15min":8.8819999999999997},"last":0.622},{"interface":"front","average":{"1min":2.214,"5min":1.7330000000000001,"15min":0.97799999999999998},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":9.5039999999999996,"5min":9.5039999999999996,"15min":9.5039999999999996},"last":0.53800000000000003}]},{"osd":3,"last update":"Fri Feb 9 17:44:22 2024","interfaces":[{"interface":"back","average":{"1min":1.752,"5min":1.661,"15min":2.0880000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.245},"max":{"1min":5.383,"5min":11.679,"15min":50.462000000000003},"last":0.28499999999999998},{"interface":"front","average":{"1min":1.794,"5min":1.3140000000000001,"15min":1.8340000000000001},"min":{"1min":0.39000000000000001,"5min":0.29099999999999998,"15min":0.255},"max":{"1min":6.657,"5min":9.1579999999999995,"15min":50.529000000000003},"last":0.41899999999999998}]},{"osd":5,"last update":"Fri Feb 9 17:43:53 2024","interfaces":[{"interface":"back","average":{"1min":2.7519999999999998,"5min":1.679,"15min":1.5860000000000001},"min":{"1min":0.26300000000000001,"5min":0.22500000000000001,"15min":0.222},"max":{"1min":12.042999999999999,"5min":12.042999999999999,"15min":12.042999999999999},"last":1.8779999999999999},{"interface":"front","average":{"1min":2.8029999999999999,"5min":1.8009999999999999,"15min":1.8720000000000001},"min":{"1min":0.35599999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":12.073,"5min":12.073,"15min":50.509999999999998},"last":1.9039999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:44:22 2024","interfaces":[{"interface":"back","average":{"1min":2.5760000000000001,"5min":1.6080000000000001,"15min":1.6259999999999999},"min":{"1min":0.35999999999999999,"5min":0.28299999999999997,"15min":0.246},"max":{"1min":18.234999999999999,"5min":18.234999999999999,"15min":18.234999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":2.5219999999999998,"5min":2.1949999999999998,"15min":2.016},"min":{"1min":0.316,"5min":0.312,"15min":0.245},"max":{"1min":9.2159999999999993,"5min":13.208,"15min":50.402999999999999},"last":0.626}]},{"osd":7,"last update":"Fri Feb 9 17:43:55 2024","interfaces":[{"interface":"back","average":{"1min":1.6699999999999999,"5min":1.6910000000000001,"15min":1.7330000000000001},"min":{"1min":0.377,"5min":0.32200000000000001,"15min":0.28399999999999997},"max":{"1min":6.415,"5min":11.875,"15min":14.919},"last":0.52100000000000002},{"interface":"front","average":{"1min":2.8090000000000002,"5min":1.893,"15min":1.8520000000000001},"min":{"1min":0.44900000000000001,"5min":0.28199999999999997,"15min":0.28100000000000003},"max":{"1min":11.739000000000001,"5min":12.696,"15min":19.047000000000001},"last":0.71299999999999997}]},{"osd":8,"last update":"Fri Feb 9 17:43:53 2024","interfaces":[{"interface":"back","average":{"1min":2.488,"5min":1.6759999999999999,"15min":1.587},"min":{"1min":0.435,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":10.369,"5min":10.369,"15min":10.369},"last":1.849},{"interface":"front","average":{"1min":1.778,"5min":1.2529999999999999,"15min":1.6180000000000001},"min":{"1min":0.42899999999999999,"5min":0.34699999999999998,"15min":0.308},"max":{"1min":6.3330000000000002,"5min":6.3330000000000002,"15min":50.594000000000001},"last":0.56599999999999995}]},{"osd":9,"last update":"Fri Feb 9 17:43:34 2024","interfaces":[{"interface":"back","average":{"1min":2.2160000000000002,"5min":1.6259999999999999,"15min":1.8169999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":5.6369999999999996,"5min":9.2370000000000001,"15min":50.442999999999998},"last":1.9319999999999999},{"interface":"front","average":{"1min":1.6559999999999999,"5min":1.3300000000000001,"15min":1.5029999999999999},"min":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.36199999999999999},"max":{"1min":6.3879999999999999,"5min":8.3580000000000005,"15min":10.555},"last":0.65100000000000002}]},{"osd":10,"last update":"Fri Feb 9 17:44:08 2024","interfaces":[{"interface":"back","average":{"1min":2.5760000000000001,"5min":2.012,"15min":2.0960000000000001},"min":{"1min":0.41599999999999998,"5min":0.40300000000000002,"15min":0.39400000000000002},"max":{"1min":10.454000000000001,"5min":10.454000000000001,"15min":51.223999999999997},"last":1.8200000000000001},{"interface":"front","average":{"1min":2.6749999999999998,"5min":1.7330000000000001,"15min":2.0739999999999998},"min":{"1min":0.53900000000000003,"5min":0.42899999999999999,"15min":0.39200000000000002},"max":{"1min":11.646000000000001,"5min":11.646000000000001,"15min":50.552},"last":0.47599999999999998}]},{"osd":11,"last update":"Fri Feb 9 17:43:37 2024","interfaces":[{"interface":"back","average":{"1min":1.8720000000000001,"5min":1.7989999999999999,"15min":2.052},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.42099999999999999},"max":{"1min":5.4569999999999999,"5min":9.4060000000000006,"15min":50.484999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":1.889,"5min":1.4039999999999999,"15min":1.673},"min":{"1min":0.68999999999999995,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":6.5830000000000002,"5min":7.694,"15min":18.574999999999999},"last":0.504}]}]},{"osd":5,"up_from":40,"seq":171798692135,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1527264,"kb_used_data":1067096,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":92217888,"statfs":{"total":95995035648,"available":94431117312,"internally_reserved":0,"allocated":1092706304,"data_stored":1092279697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":471202245},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:44:12 2024","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.256,"15min":1.1200000000000001},"min":{"1min":0.24199999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":8.4700000000000006,"5min":8.4700000000000006,"15min":8.4700000000000006},"last":5.3769999999999998},{"interface":"front","average":{"1min":2.1659999999999999,"5min":1.671,"15min":1.766},"min":{"1min":0.32500000000000001,"5min":0.249,"15min":0.249},"max":{"1min":9.0429999999999993,"5min":9.7650000000000006,"15min":9.7650000000000006},"last":3.952}]},{"osd":1,"last update":"Fri Feb 9 17:43:53 2024","interfaces":[{"interface":"back","average":{"1min":2.2789999999999999,"5min":1.5489999999999999,"15min":2.0219999999999998},"min":{"1min":0.28999999999999998,"5min":0.24199999999999999,"15min":0.23899999999999999},"max":{"1min":5.7999999999999998,"5min":9.7330000000000005,"15min":48.161999999999999},"last":7.0800000000000001},{"interface":"front","average":{"1min":2.2599999999999998,"5min":1.8240000000000001,"15min":1.9810000000000001},"min":{"1min":0.32700000000000001,"5min":0.25900000000000001,"15min":0.23599999999999999},"max":{"1min":5.9160000000000004,"5min":11.002000000000001,"15min":48.218000000000004},"last":5.4169999999999998}]},{"osd":2,"last update":"Fri Feb 9 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.526,"15min":1.526},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":4.718,"5min":4.718,"15min":4.718},"last":3.8839999999999999},{"interface":"front","average":{"1min":2.1269999999999998,"5min":2.1269999999999998,"15min":2.1269999999999998},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":8.5079999999999991,"5min":8.5079999999999991,"15min":8.5079999999999991},"last":6.6909999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:43:49 2024","interfaces":[{"interface":"back","average":{"1min":1.419,"5min":1.472,"15min":2.069},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.26500000000000001},"max":{"1min":8.4239999999999995,"5min":11.081,"15min":64.150999999999996},"last":6.7300000000000004},{"interface":"front","average":{"1min":2.073,"5min":1.7969999999999999,"15min":2.157},"min":{"1min":0.35299999999999998,"5min":0.247,"15min":0.247},"max":{"1min":5.3120000000000003,"5min":12.468,"15min":64.176000000000002},"last":4.032}]},{"osd":4,"last update":"Fri Feb 9 17:43:49 2024","interfaces":[{"interface":"back","average":{"1min":2.5640000000000001,"5min":2.1150000000000002,"15min":1.766},"min":{"1min":0.249,"5min":0.249,"15min":0.23400000000000001},"max":{"1min":8.4890000000000008,"5min":49.313000000000002,"15min":49.313000000000002},"last":6.7949999999999999},{"interface":"front","average":{"1min":2.601,"5min":2.2010000000000001,"15min":1.744},"min":{"1min":0.27000000000000002,"5min":0.218,"15min":0.218},"max":{"1min":7.0010000000000003,"5min":50.505000000000003,"15min":50.505000000000003},"last":4.0049999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:44:23 2024","interfaces":[{"interface":"back","average":{"1min":2.214,"5min":1.8979999999999999,"15min":1.758},"min":{"1min":0.29899999999999999,"5min":0.28199999999999997,"15min":0.253},"max":{"1min":8.3390000000000004,"5min":12.425000000000001,"15min":15.375999999999999},"last":3.98},{"interface":"front","average":{"1min":2.1019999999999999,"5min":1.863,"15min":1.8380000000000001},"min":{"1min":0.308,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":8.4489999999999998,"5min":11.199,"15min":14.250999999999999},"last":6.7519999999999998}]},{"osd":7,"last update":"Fri Feb 9 17:43:49 2024","interfaces":[{"interface":"back","average":{"1min":2.516,"5min":2.1899999999999999,"15min":1.742},"min":{"1min":0.40200000000000002,"5min":0.27900000000000003,"15min":0.26700000000000002},"max":{"1min":8.5359999999999996,"5min":50.545000000000002,"15min":50.545000000000002},"last":3.9060000000000001},{"interface":"front","average":{"1min":2.847,"5min":2.3130000000000002,"15min":2.3140000000000001},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":16.314,"5min":49.688000000000002,"15min":102.526},"last":3.9279999999999999}]},{"osd":8,"last update":"Fri Feb 9 17:43:23 2024","interfaces":[{"interface":"back","average":{"1min":1.6499999999999999,"5min":1.639,"15min":1.6020000000000001},"min":{"1min":0.376,"5min":0.35299999999999998,"15min":0.34899999999999998},"max":{"1min":4.8780000000000001,"5min":14.542,"15min":14.542},"last":5.5439999999999996},{"interface":"front","average":{"1min":3.6699999999999999,"5min":1.702,"15min":1.8300000000000001},"min":{"1min":0.37,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":16.359000000000002,"5min":16.359000000000002,"15min":16.359000000000002},"last":0.435}]},{"osd":9,"last update":"Fri Feb 9 17:44:23 2024","interfaces":[{"interface":"back","average":{"1min":1.778,"5min":1.643,"15min":1.5740000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":9.9510000000000005,"5min":16.411000000000001,"15min":16.411000000000001},"last":6.7750000000000004},{"interface":"front","average":{"1min":1.8919999999999999,"5min":1.845,"15min":1.6579999999999999},"min":{"1min":0.42899999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":5.827,"5min":16.417999999999999,"15min":16.417999999999999},"last":4.9530000000000003}]},{"osd":10,"last update":"Fri Feb 9 17:43:49 2024","interfaces":[{"interface":"back","average":{"1min":2.782,"5min":1.8660000000000001,"15min":1.7110000000000001},"min":{"1min":0.39800000000000002,"5min":0.317,"15min":0.317},"max":{"1min":16.445,"5min":16.445,"15min":16.445},"last":3.992},{"interface":"front","average":{"1min":2.1749999999999998,"5min":1.6519999999999999,"15min":1.744},"min":{"1min":0.48699999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":5.5999999999999996,"5min":9.8840000000000003,"15min":14.186999999999999},"last":4.056}]},{"osd":11,"last update":"Fri Feb 9 17:44:12 2024","interfaces":[{"interface":"back","average":{"1min":1.9950000000000001,"5min":2.2210000000000001,"15min":1.7649999999999999},"min":{"1min":0.434,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":5.0700000000000003,"5min":16.390000000000001,"15min":16.390000000000001},"last":3.9630000000000001},{"interface":"front","average":{"1min":2.0190000000000001,"5min":1.9690000000000001,"15min":1.758},"min":{"1min":0.53400000000000003,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":5.9660000000000002,"5min":9.8239999999999998,"15min":9.8239999999999998},"last":4.46}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1113325568,"data_stored":1113325568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":619118592,"data_stored":619118592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":342687744,"data_stored":342687744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":146931712,"data_stored":146931712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":907673600,"data_stored":907673600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1090719744,"data_stored":1090715672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":464781312,"data_stored":464781312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":584585216,"data_stored":584581144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":295960576,"data_stored":295960576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":446496768,"data_stored":446496768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":806289408,"data_stored":806289408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":618532864,"data_stored":618528792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-09T17:44:34.926 INFO:tasks.thrashosds.thrasher:Setting 5.3 to [4, 1, 2] 2024-02-09T17:44:34.926 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '5.3', '4', '1', '2'] 2024-02-09T17:44:34.927 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd pg-upmap 5.3 4 1 2 2024-02-09T17:44:35.018 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:44:35.065 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:35.081 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:35 smithi179 ceph-mon[43975]: from='client.16047 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:44:35.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:35 smithi099 ceph-mon[39420]: from='client.16047 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:44:35.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:35 smithi162 ceph-mon[44304]: from='client.16047 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:44:36.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:36 smithi179 ceph-mon[43975]: 1.8 scrub starts 2024-02-09T17:44:36.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:36 smithi179 ceph-mon[43975]: 1.8 scrub ok 2024-02-09T17:44:36.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:36 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:44:36.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:36 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:44:36.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:36 smithi179 ceph-mon[43975]: pgmap v1384: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 2.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 787 op/s; 1300/107802 objects misplaced (1.206%); 8.0 MiB/s, 127 objects/s recovering 2024-02-09T17:44:36.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:36 smithi179 ceph-mon[43975]: osdmap e550: 12 total, 12 up, 12 in 2024-02-09T17:44:36.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:36 smithi162 ceph-mon[44304]: 1.8 scrub starts 2024-02-09T17:44:36.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:36 smithi162 ceph-mon[44304]: 1.8 scrub ok 2024-02-09T17:44:36.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:36 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:44:36.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:36 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:44:36.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:36 smithi162 ceph-mon[44304]: pgmap v1384: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 2.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 787 op/s; 1300/107802 objects misplaced (1.206%); 8.0 MiB/s, 127 objects/s recovering 2024-02-09T17:44:36.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:36 smithi162 ceph-mon[44304]: osdmap e550: 12 total, 12 up, 12 in 2024-02-09T17:44:36.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:36 smithi099 ceph-mon[39420]: 1.8 scrub starts 2024-02-09T17:44:36.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:36 smithi099 ceph-mon[39420]: 1.8 scrub ok 2024-02-09T17:44:36.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:36 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:44:36.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:36 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:44:36.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:36 smithi099 ceph-mon[39420]: pgmap v1384: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 2.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 787 op/s; 1300/107802 objects misplaced (1.206%); 8.0 MiB/s, 127 objects/s recovering 2024-02-09T17:44:36.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:36 smithi099 ceph-mon[39420]: osdmap e550: 12 total, 12 up, 12 in 2024-02-09T17:44:36.836 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:44:36.840 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:36.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:44:36.941 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:44:36.971 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:36.998 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:44:37.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:37.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:44:37.007 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:44:37.104 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:44:37.165 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:37.192 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:44:37.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:37.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:44:37.296 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:44:37.339 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:37.369 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:44:37.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:37.372 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:44:37.473 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:44:37.542 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:37.587 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:44:37.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:37.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:44:37.626 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:37 smithi099 ceph-mon[39420]: osdmap e551: 12 total, 12 up, 12 in 2024-02-09T17:44:37.626 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:37 smithi099 ceph-mon[39420]: pgmap v1387: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 909 op/s; 1300/101493 objects misplaced (1.281%); 11 MiB/s, 168 objects/s recovering 2024-02-09T17:44:37.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:37 smithi179 ceph-mon[43975]: osdmap e551: 12 total, 12 up, 12 in 2024-02-09T17:44:37.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:37 smithi179 ceph-mon[43975]: pgmap v1387: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 909 op/s; 1300/101493 objects misplaced (1.281%); 11 MiB/s, 168 objects/s recovering 2024-02-09T17:44:37.692 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:44:37.726 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:37.763 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:44:37.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:37.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:44:37.871 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:44:37.871 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:44:37.907 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:37.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:37 smithi162 ceph-mon[44304]: osdmap e551: 12 total, 12 up, 12 in 2024-02-09T17:44:37.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:37 smithi162 ceph-mon[44304]: pgmap v1387: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 909 op/s; 1300/101493 objects misplaced (1.281%); 11 MiB/s, 168 objects/s recovering 2024-02-09T17:44:37.934 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:44:37.935 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:44:37.935 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 20, 2024-02-09T17:44:37.936 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 600, 2024-02-09T17:44:37.936 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [ 2024-02-09T17:44:37.936 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.936 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:208395 4.f 4:fb1bb549:::benchmark_data_smithi179_103809_object70929:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-02-09T17:44:37.936 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:38:33.803932+0000", 2024-02-09T17:44:37.936 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 364.12892540500002, 2024-02-09T17:44:37.936 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2222837719999999, 2024-02-09T17:44:37.936 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:44:37.936 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "reached pg", 2024-02-09T17:44:37.936 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:44:37.936 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:44:37.936 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:44:37.936 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 208395 2024-02-09T17:44:37.937 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.937 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:44:37.937 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.937 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:44:37.937 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.803932+0000", 2024-02-09T17:44:37.937 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:37.937 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.937 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.937 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:44:37.937 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.803932+0000", 2024-02-09T17:44:37.937 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:37.937 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.937 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.938 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:44:37.938 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.803933+0000", 2024-02-09T17:44:37.938 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.2800000000000005e-07 2024-02-09T17:44:37.938 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.938 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.938 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:44:37.938 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.803937+0000", 2024-02-09T17:44:37.938 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.4589999999999998e-06 2024-02-09T17:44:37.938 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.938 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.938 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:44:37.938 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.803946+0000", 2024-02-09T17:44:37.938 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.465e-06 2024-02-09T17:44:37.938 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.939 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.939 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:44:37.939 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.803952+0000", 2024-02-09T17:44:37.939 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.9959999999999999e-06 2024-02-09T17:44:37.939 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.939 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.939 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:37.939 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.803987+0000", 2024-02-09T17:44:37.939 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.4887e-05 2024-02-09T17:44:37.939 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.939 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.939 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:44:37.939 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.804067+0000", 2024-02-09T17:44:37.940 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.9674999999999999e-05 2024-02-09T17:44:37.940 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.940 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.940 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,11", 2024-02-09T17:44:37.940 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.804135+0000", 2024-02-09T17:44:37.940 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.8201000000000007e-05 2024-02-09T17:44:37.940 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.940 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.940 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:44:37.940 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.809243+0000", 2024-02-09T17:44:37.940 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0051077320000000002 2024-02-09T17:44:37.940 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.940 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.940 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:37.941 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.812530+0000", 2024-02-09T17:44:37.941 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0032871210000000001 2024-02-09T17:44:37.941 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.941 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.941 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:37.941 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.831775+0000", 2024-02-09T17:44:37.941 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.019245038999999999 2024-02-09T17:44:37.941 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.941 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.941 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:44:37.941 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.831785+0000", 2024-02-09T17:44:37.942 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.8740000000000007e-06 2024-02-09T17:44:37.942 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.942 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.942 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:37.942 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.810674+0000", 2024-02-09T17:44:37.942 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.97888962099999999 2024-02-09T17:44:37.942 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.942 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.942 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:44:37.942 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.810686+0000", 2024-02-09T17:44:37.942 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1963999999999999e-05 2024-02-09T17:44:37.942 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.942 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.942 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:37.942 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.026131+0000", 2024-02-09T17:44:37.943 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2154448579999999 2024-02-09T17:44:37.943 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.943 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.943 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:44:37.943 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.026216+0000", 2024-02-09T17:44:37.943 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.4951999999999996e-05 2024-02-09T17:44:37.943 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:37.943 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:37.943 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:37.943 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.943 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.943 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:208404 4.a 4:527668e2:::benchmark_data_smithi179_103809_object70938:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e379)", 2024-02-09T17:44:37.943 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:38:33.818190+0000", 2024-02-09T17:44:37.943 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 364.11466732600002, 2024-02-09T17:44:37.943 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2784034790000001, 2024-02-09T17:44:37.944 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:44:37.944 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:44:37.944 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:44:37.944 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:44:37.944 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:44:37.944 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 208404 2024-02-09T17:44:37.944 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.944 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:44:37.944 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.944 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:44:37.944 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.818190+0000", 2024-02-09T17:44:37.944 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:37.944 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.945 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.945 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:44:37.945 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.818190+0000", 2024-02-09T17:44:37.945 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:37.945 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.945 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.945 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:44:37.945 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.818192+0000", 2024-02-09T17:44:37.945 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.358e-06 2024-02-09T17:44:37.945 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.945 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.945 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:44:37.945 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.818195+0000", 2024-02-09T17:44:37.945 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.1020000000000001e-06 2024-02-09T17:44:37.946 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.946 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.946 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:44:37.946 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.818203+0000", 2024-02-09T17:44:37.946 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.585e-06 2024-02-09T17:44:37.946 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.946 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.946 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:44:37.946 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.818210+0000", 2024-02-09T17:44:37.946 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.545e-06 2024-02-09T17:44:37.946 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.946 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.946 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:37.946 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.820340+0000", 2024-02-09T17:44:37.946 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0021302629999999999 2024-02-09T17:44:37.947 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.947 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.947 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:44:37.947 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.820351+0000", 2024-02-09T17:44:37.947 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0514e-05 2024-02-09T17:44:37.947 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.947 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.947 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:37.947 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.810548+0000", 2024-02-09T17:44:37.947 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.99019749999999995 2024-02-09T17:44:37.947 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.947 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.947 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:44:37.947 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.810555+0000", 2024-02-09T17:44:37.947 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.0149999999999998e-06 2024-02-09T17:44:37.948 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.948 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.948 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:37.948 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.063853+0000", 2024-02-09T17:44:37.948 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2532982509999999 2024-02-09T17:44:37.948 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.948 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.948 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:44:37.948 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.067093+0000", 2024-02-09T17:44:37.948 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0032394160000000002 2024-02-09T17:44:37.948 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.948 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.948 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 6,7,10", 2024-02-09T17:44:37.948 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.067178+0000", 2024-02-09T17:44:37.949 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.5220000000000001e-05 2024-02-09T17:44:37.949 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.949 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.949 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:37.949 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.071071+0000", 2024-02-09T17:44:37.949 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0038930480000000001 2024-02-09T17:44:37.949 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.949 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.949 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:44:37.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:37.951 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.074920+0000", 2024-02-09T17:44:37.951 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0038487450000000002 2024-02-09T17:44:37.951 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.951 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.952 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:37.952 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.079243+0000", 2024-02-09T17:44:37.952 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0043232460000000002 2024-02-09T17:44:37.952 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.952 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.952 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:37.952 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.096462+0000", 2024-02-09T17:44:37.952 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.017218444999999999 2024-02-09T17:44:37.952 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.952 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.952 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:44:37.952 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.096526+0000", 2024-02-09T17:44:37.952 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.4646000000000004e-05 2024-02-09T17:44:37.952 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.953 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.953 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:44:37.953 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.096594+0000", 2024-02-09T17:44:37.953 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.758e-05 2024-02-09T17:44:37.953 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:37.953 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:37.953 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:37.953 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.953 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.953 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:208412 4.f 4:f71ab7a9:::benchmark_data_smithi179_103809_object70946:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-02-09T17:44:37.953 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:38:33.828165+0000", 2024-02-09T17:44:37.953 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 364.10469266500002, 2024-02-09T17:44:37.953 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2159022140000002, 2024-02-09T17:44:37.953 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:44:37.954 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:44:37.954 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:44:37.954 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:44:37.954 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:44:37.954 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 208412 2024-02-09T17:44:37.954 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.954 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:44:37.954 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.954 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:44:37.954 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.828165+0000", 2024-02-09T17:44:37.954 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:37.954 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.954 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.954 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:44:37.955 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.828165+0000", 2024-02-09T17:44:37.955 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:37.955 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.955 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.955 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:44:37.955 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.828166+0000", 2024-02-09T17:44:37.955 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0249999999999999e-06 2024-02-09T17:44:37.955 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.955 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.955 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:44:37.956 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.828169+0000", 2024-02-09T17:44:37.956 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.039e-06 2024-02-09T17:44:37.956 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.956 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.956 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:44:37.956 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.828178+0000", 2024-02-09T17:44:37.956 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.1349999999999998e-06 2024-02-09T17:44:37.956 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.956 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.956 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:44:37.956 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.828185+0000", 2024-02-09T17:44:37.956 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.1620000000000002e-06 2024-02-09T17:44:37.956 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.956 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.956 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:37.957 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.831810+0000", 2024-02-09T17:44:37.957 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.003625133 2024-02-09T17:44:37.957 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.957 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.957 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:44:37.957 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.831817+0000", 2024-02-09T17:44:37.957 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.5830000000000003e-06 2024-02-09T17:44:37.957 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.957 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.958 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:37.958 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.810755+0000", 2024-02-09T17:44:37.958 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.97893847300000003 2024-02-09T17:44:37.958 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.958 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.958 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:44:37.958 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.810766+0000", 2024-02-09T17:44:37.958 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0362e-05 2024-02-09T17:44:37.958 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.958 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.958 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:37.958 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.026526+0000", 2024-02-09T17:44:37.958 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2157602970000001 2024-02-09T17:44:37.959 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.959 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.959 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:44:37.959 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.026642+0000", 2024-02-09T17:44:37.959 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00011571899999999999 2024-02-09T17:44:37.959 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.959 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.959 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,9,11", 2024-02-09T17:44:37.959 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.026770+0000", 2024-02-09T17:44:37.959 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012810600000000001 2024-02-09T17:44:37.959 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.959 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.959 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:37.959 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.037750+0000", 2024-02-09T17:44:37.959 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.010980492999999999 2024-02-09T17:44:37.960 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.960 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.960 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:44:37.960 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.038050+0000", 2024-02-09T17:44:37.960 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00029930899999999999 2024-02-09T17:44:37.960 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.960 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.960 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:37.960 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.043224+0000", 2024-02-09T17:44:37.960 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.005174344 2024-02-09T17:44:37.960 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.960 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.960 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:37.960 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.043952+0000", 2024-02-09T17:44:37.961 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00072777900000000001 2024-02-09T17:44:37.961 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.961 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.961 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:44:37.961 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.043996+0000", 2024-02-09T17:44:37.961 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.4234999999999999e-05 2024-02-09T17:44:37.961 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.961 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.961 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:44:37.961 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.044067+0000", 2024-02-09T17:44:37.961 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.1019999999999994e-05 2024-02-09T17:44:37.961 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:37.961 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:37.961 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:37.962 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.962 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.962 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:208420 4.f 4:f8390c68:::benchmark_data_smithi179_103809_object70954:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-02-09T17:44:37.962 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:38:33.837086+0000", 2024-02-09T17:44:37.962 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 364.09577200000001, 2024-02-09T17:44:37.962 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2070928369999998, 2024-02-09T17:44:37.962 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:44:37.962 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:44:37.962 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:44:37.962 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:44:37.962 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:44:37.962 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 208420 2024-02-09T17:44:37.962 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.962 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:44:37.962 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.963 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:44:37.963 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.837086+0000", 2024-02-09T17:44:37.963 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:37.963 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.963 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.963 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:44:37.963 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.837086+0000", 2024-02-09T17:44:37.963 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:37.963 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.963 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.963 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:44:37.963 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.837087+0000", 2024-02-09T17:44:37.963 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.158e-06 2024-02-09T17:44:37.963 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.964 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.964 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:44:37.964 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.837091+0000", 2024-02-09T17:44:37.964 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.0169999999999998e-06 2024-02-09T17:44:37.964 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.964 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.964 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:44:37.964 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.837100+0000", 2024-02-09T17:44:37.964 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.2390000000000008e-06 2024-02-09T17:44:37.964 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.964 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.964 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:44:37.964 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.837107+0000", 2024-02-09T17:44:37.964 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.6959999999999999e-06 2024-02-09T17:44:37.965 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.965 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.965 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:37.965 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.837156+0000", 2024-02-09T17:44:37.965 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.8928999999999999e-05 2024-02-09T17:44:37.965 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.965 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.965 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:44:37.965 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.837163+0000", 2024-02-09T17:44:37.965 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.306e-06 2024-02-09T17:44:37.965 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.965 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.965 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:37.965 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.810843+0000", 2024-02-09T17:44:37.966 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.97367960899999995 2024-02-09T17:44:37.966 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.966 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.966 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:44:37.966 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.810852+0000", 2024-02-09T17:44:37.966 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.944e-06 2024-02-09T17:44:37.966 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.966 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.966 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:37.966 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.032038+0000", 2024-02-09T17:44:37.966 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2211856670000001 2024-02-09T17:44:37.966 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.966 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.966 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:44:37.967 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.032189+0000", 2024-02-09T17:44:37.967 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00015130499999999999 2024-02-09T17:44:37.967 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.967 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.967 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,9,11", 2024-02-09T17:44:37.967 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.032342+0000", 2024-02-09T17:44:37.967 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00015267300000000001 2024-02-09T17:44:37.967 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.967 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.967 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:37.967 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.038032+0000", 2024-02-09T17:44:37.967 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0056899239999999998 2024-02-09T17:44:37.967 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.967 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.967 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:44:37.968 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.038056+0000", 2024-02-09T17:44:37.968 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.3737e-05 2024-02-09T17:44:37.968 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.968 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.968 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:37.968 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.043294+0000", 2024-02-09T17:44:37.968 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0052377689999999998 2024-02-09T17:44:37.968 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.968 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.968 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:37.968 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.044091+0000", 2024-02-09T17:44:37.968 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00079718400000000004 2024-02-09T17:44:37.969 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.969 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.969 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:44:37.969 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.044121+0000", 2024-02-09T17:44:37.969 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.0623000000000001e-05 2024-02-09T17:44:37.969 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.969 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.969 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:44:37.969 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.044178+0000", 2024-02-09T17:44:37.969 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7056999999999999e-05 2024-02-09T17:44:37.969 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:37.969 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:37.969 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:37.969 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.970 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.970 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:208424 4.a 4:57643a18:::benchmark_data_smithi179_103809_object70958:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-02-09T17:44:37.970 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:38:33.852457+0000", 2024-02-09T17:44:37.970 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 364.08040098700002, 2024-02-09T17:44:37.970 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.24435845, 2024-02-09T17:44:37.970 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:44:37.970 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:44:37.970 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:44:37.970 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:44:37.970 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:44:37.970 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 208424 2024-02-09T17:44:37.970 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.971 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:44:37.971 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.971 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:44:37.971 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.852457+0000", 2024-02-09T17:44:37.971 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:37.971 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.971 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.971 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:44:37.971 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.852457+0000", 2024-02-09T17:44:37.971 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:37.971 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.971 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.971 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:44:37.972 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.852458+0000", 2024-02-09T17:44:37.972 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2780000000000001e-06 2024-02-09T17:44:37.972 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.972 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.972 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:44:37.972 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.852463+0000", 2024-02-09T17:44:37.972 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.0189999999999997e-06 2024-02-09T17:44:37.972 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.972 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.972 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:44:37.972 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.852474+0000", 2024-02-09T17:44:37.972 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0637e-05 2024-02-09T17:44:37.972 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.973 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.973 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:44:37.973 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.852480+0000", 2024-02-09T17:44:37.973 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.5069999999999997e-06 2024-02-09T17:44:37.973 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.973 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.973 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:37.973 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.852536+0000", 2024-02-09T17:44:37.973 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6042000000000003e-05 2024-02-09T17:44:37.973 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.973 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.973 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:44:37.974 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:33.852545+0000", 2024-02-09T17:44:37.974 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.8019999999999998e-06 2024-02-09T17:44:37.974 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.974 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.974 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:37.974 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.810592+0000", 2024-02-09T17:44:37.974 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.95804690800000003 2024-02-09T17:44:37.974 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.974 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.974 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:44:37.974 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.810601+0000", 2024-02-09T17:44:37.974 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.132e-06 2024-02-09T17:44:37.974 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.974 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.975 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:37.975 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.071108+0000", 2024-02-09T17:44:37.975 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2605074979999999 2024-02-09T17:44:37.975 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.975 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.975 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:44:37.975 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.071189+0000", 2024-02-09T17:44:37.975 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.0723000000000006e-05 2024-02-09T17:44:37.975 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.975 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.975 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 6,7,10", 2024-02-09T17:44:37.975 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.071301+0000", 2024-02-09T17:44:37.975 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00011174 2024-02-09T17:44:37.975 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.976 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.976 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:37.976 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.079299+0000", 2024-02-09T17:44:37.976 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0079979620000000008 2024-02-09T17:44:37.976 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.976 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.976 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:44:37.976 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.079314+0000", 2024-02-09T17:44:37.976 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4751e-05 2024-02-09T17:44:37.976 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.976 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.976 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:37.976 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.096314+0000", 2024-02-09T17:44:37.977 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.017000767 2024-02-09T17:44:37.977 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.977 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.977 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:37.977 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.096648+0000", 2024-02-09T17:44:37.977 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00033350399999999998 2024-02-09T17:44:37.977 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.977 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.977 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:44:37.977 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.096704+0000", 2024-02-09T17:44:37.977 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6137999999999997e-05 2024-02-09T17:44:37.977 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.978 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.978 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:44:37.978 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.096815+0000", 2024-02-09T17:44:37.978 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000111042 2024-02-09T17:44:37.978 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:37.978 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:37.978 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:37.978 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.978 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.978 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:208410 4.9 4:96199963:::benchmark_data_smithi179_103809_object70944:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e380)", 2024-02-09T17:44:37.978 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:38:34.811330+0000", 2024-02-09T17:44:37.978 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 363.12152733599999, 2024-02-09T17:44:37.978 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.339120402, 2024-02-09T17:44:37.979 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:44:37.979 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:44:37.979 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:44:37.979 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:44:37.979 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:44:37.979 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 208410 2024-02-09T17:44:37.979 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.979 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:44:37.979 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.979 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:44:37.979 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.811330+0000", 2024-02-09T17:44:37.979 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:37.980 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.980 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.980 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:44:37.980 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.811330+0000", 2024-02-09T17:44:37.980 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:37.980 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.980 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.980 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:44:37.980 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.811332+0000", 2024-02-09T17:44:37.980 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2559999999999999e-06 2024-02-09T17:44:37.980 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.980 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.981 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:44:37.981 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.811336+0000", 2024-02-09T17:44:37.981 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.9599999999999999e-06 2024-02-09T17:44:37.981 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.981 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.981 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:44:37.981 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.811346+0000", 2024-02-09T17:44:37.981 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.5340000000000005e-06 2024-02-09T17:44:37.981 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.981 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.981 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:44:37.981 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.811355+0000", 2024-02-09T17:44:37.982 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.2599999999999994e-06 2024-02-09T17:44:37.982 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.982 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.982 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:37.982 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.811423+0000", 2024-02-09T17:44:37.982 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.7872000000000003e-05 2024-02-09T17:44:37.982 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.982 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.982 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:44:37.983 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.811433+0000", 2024-02-09T17:44:37.983 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.8940000000000005e-06 2024-02-09T17:44:37.983 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.983 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.983 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:37.983 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.120834+0000", 2024-02-09T17:44:37.984 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.309400841 2024-02-09T17:44:37.984 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.984 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.984 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:44:37.984 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.124192+0000", 2024-02-09T17:44:37.985 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0033576330000000001 2024-02-09T17:44:37.985 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.985 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.985 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,3,4,6,11", 2024-02-09T17:44:37.985 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.124373+0000", 2024-02-09T17:44:37.985 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000181187 2024-02-09T17:44:37.985 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.985 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.985 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:37.985 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.138957+0000", 2024-02-09T17:44:37.985 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.014584456000000001 2024-02-09T17:44:37.985 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.986 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.986 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:37.986 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.139006+0000", 2024-02-09T17:44:37.986 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.8989999999999997e-05 2024-02-09T17:44:37.986 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.986 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.986 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:37.986 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.139139+0000", 2024-02-09T17:44:37.986 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000132597 2024-02-09T17:44:37.986 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.987 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.987 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:44:37.987 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.144076+0000", 2024-02-09T17:44:37.987 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.004937617 2024-02-09T17:44:37.987 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.987 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.987 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:37.987 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.150149+0000", 2024-02-09T17:44:37.988 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0060727510000000004 2024-02-09T17:44:37.988 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.988 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.988 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:37.988 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.150251+0000", 2024-02-09T17:44:37.988 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000102126 2024-02-09T17:44:37.988 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.988 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.988 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:44:37.988 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.150308+0000", 2024-02-09T17:44:37.989 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7197000000000001e-05 2024-02-09T17:44:37.989 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.989 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.989 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:44:37.989 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.150451+0000", 2024-02-09T17:44:37.989 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014223099999999999 2024-02-09T17:44:37.989 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:37.989 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:37.989 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:37.989 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.989 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.989 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:208415 4.9 4:9dcaa6ee:::benchmark_data_smithi179_103809_object70949:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e380)", 2024-02-09T17:44:37.989 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:38:34.811375+0000", 2024-02-09T17:44:37.990 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 363.12148284, 2024-02-09T17:44:37.990 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.339417547, 2024-02-09T17:44:37.990 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:44:37.990 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:44:37.990 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:44:37.990 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:44:37.990 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:44:37.990 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 208415 2024-02-09T17:44:37.990 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.990 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:44:37.990 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.990 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:44:37.990 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.811375+0000", 2024-02-09T17:44:37.990 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:37.991 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.991 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.991 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:44:37.991 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.811375+0000", 2024-02-09T17:44:37.991 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:37.991 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.991 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.991 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:44:37.991 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.811375+0000", 2024-02-09T17:44:37.991 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2e-07 2024-02-09T17:44:37.991 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.991 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.991 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:44:37.992 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.811378+0000", 2024-02-09T17:44:37.992 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.4420000000000001e-06 2024-02-09T17:44:37.992 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.992 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.992 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:44:37.992 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.811385+0000", 2024-02-09T17:44:37.992 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.9160000000000004e-06 2024-02-09T17:44:37.992 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.992 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.992 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:44:37.993 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.811403+0000", 2024-02-09T17:44:37.993 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.8108999999999999e-05 2024-02-09T17:44:37.993 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.993 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.993 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:37.993 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.811464+0000", 2024-02-09T17:44:37.993 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.1223999999999995e-05 2024-02-09T17:44:37.993 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.993 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.993 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:44:37.994 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.811472+0000", 2024-02-09T17:44:37.994 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.5950000000000001e-06 2024-02-09T17:44:37.994 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.994 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.994 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:37.994 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.128953+0000", 2024-02-09T17:44:37.994 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3174819369999999 2024-02-09T17:44:37.994 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.994 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.994 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:44:37.994 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.129041+0000", 2024-02-09T17:44:37.995 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.7145000000000007e-05 2024-02-09T17:44:37.995 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.995 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.995 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,3,4,6,11", 2024-02-09T17:44:37.995 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.129125+0000", 2024-02-09T17:44:37.995 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.4670000000000001e-05 2024-02-09T17:44:37.995 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.996 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.996 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:37.996 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.139049+0000", 2024-02-09T17:44:37.996 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0099241659999999999 2024-02-09T17:44:37.996 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.996 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.996 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:44:37.996 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.150099+0000", 2024-02-09T17:44:37.996 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.011049188999999999 2024-02-09T17:44:37.996 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.996 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.996 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:37.997 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.150198+0000", 2024-02-09T17:44:37.997 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.946e-05 2024-02-09T17:44:37.997 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.997 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.997 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:37.997 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.150494+0000", 2024-02-09T17:44:37.997 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00029567299999999999 2024-02-09T17:44:37.997 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.997 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.997 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:37.997 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.150587+0000", 2024-02-09T17:44:37.997 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.3043999999999999e-05 2024-02-09T17:44:37.998 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.998 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.998 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:37.998 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.150626+0000", 2024-02-09T17:44:37.998 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9113999999999999e-05 2024-02-09T17:44:37.998 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.998 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.998 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:44:37.998 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.150678+0000", 2024-02-09T17:44:37.998 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2358999999999998e-05 2024-02-09T17:44:37.998 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.998 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.999 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:44:37.999 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.150792+0000", 2024-02-09T17:44:37.999 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000113984 2024-02-09T17:44:37.999 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:37.999 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:37.999 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:37.999 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:37.999 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:37.999 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:208419 4.9 4:94ab8e49:::benchmark_data_smithi179_103809_object70953:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e380)", 2024-02-09T17:44:37.999 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:38:34.811412+0000", 2024-02-09T17:44:37.999 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 363.12144536800002, 2024-02-09T17:44:37.999 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3399166849999999, 2024-02-09T17:44:38.000 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:44:38.000 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:44:38.000 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:44:38.000 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:44:38.000 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:44:38.000 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 208419 2024-02-09T17:44:38.000 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.000 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:44:38.000 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.000 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:44:38.000 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.811412+0000", 2024-02-09T17:44:38.000 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:38.000 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.000 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.001 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:44:38.001 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.811412+0000", 2024-02-09T17:44:38.001 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:38.001 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.001 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.001 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:44:38.001 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.811413+0000", 2024-02-09T17:44:38.001 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.8599999999999999e-07 2024-02-09T17:44:38.001 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.001 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.001 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:44:38.001 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.811415+0000", 2024-02-09T17:44:38.001 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.5009999999999999e-06 2024-02-09T17:44:38.001 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.002 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.002 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:44:38.002 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.811422+0000", 2024-02-09T17:44:38.002 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.5710000000000003e-06 2024-02-09T17:44:38.002 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.002 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.002 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:44:38.002 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.811434+0000", 2024-02-09T17:44:38.002 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2656999999999999e-05 2024-02-09T17:44:38.002 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.002 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.002 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:38.002 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.811496+0000", 2024-02-09T17:44:38.002 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.1896999999999994e-05 2024-02-09T17:44:38.002 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.003 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.003 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:44:38.003 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:34.811503+0000", 2024-02-09T17:44:38.003 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.17e-06 2024-02-09T17:44:38.003 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.003 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.003 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:38.003 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.133540+0000", 2024-02-09T17:44:38.003 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.322036762 2024-02-09T17:44:38.003 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.003 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.003 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:44:38.003 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.133742+0000", 2024-02-09T17:44:38.003 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00020171300000000001 2024-02-09T17:44:38.004 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.004 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.004 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,3,4,6,11", 2024-02-09T17:44:38.004 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.133931+0000", 2024-02-09T17:44:38.004 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00018926500000000001 2024-02-09T17:44:38.004 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.004 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.004 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:38.004 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.139093+0000", 2024-02-09T17:44:38.004 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0051615929999999999 2024-02-09T17:44:38.004 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.004 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.004 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:44:38.004 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.150106+0000", 2024-02-09T17:44:38.005 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.011012815 2024-02-09T17:44:38.005 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.005 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.005 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:38.005 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.150821+0000", 2024-02-09T17:44:38.005 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00071553100000000004 2024-02-09T17:44:38.005 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.005 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.005 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:38.005 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.151065+0000", 2024-02-09T17:44:38.005 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000244007 2024-02-09T17:44:38.005 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.005 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.005 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:38.005 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.151155+0000", 2024-02-09T17:44:38.005 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.0369000000000002e-05 2024-02-09T17:44:38.006 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.006 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.006 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:38.006 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.151198+0000", 2024-02-09T17:44:38.006 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2556999999999999e-05 2024-02-09T17:44:38.006 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.006 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.006 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:44:38.006 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.151232+0000", 2024-02-09T17:44:38.006 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.4082999999999999e-05 2024-02-09T17:44:38.006 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.006 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.006 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:44:38.006 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:38:36.151329+0000", 2024-02-09T17:44:38.006 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.6807999999999999e-05 2024-02-09T17:44:38.007 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:38.007 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:38.007 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:38.007 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.007 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.007 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:222268 4.a 4:5f76f052:::benchmark_data_smithi179_103809_object84802:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-02-09T17:44:38.007 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:39:02.603039+0000", 2024-02-09T17:44:38.007 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 335.32981849399999, 2024-02-09T17:44:38.007 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2423865329999999, 2024-02-09T17:44:38.007 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:44:38.007 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:44:38.007 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:44:38.007 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:44:38.007 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:44:38.007 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 222268 2024-02-09T17:44:38.007 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.008 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:44:38.008 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.008 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:44:38.008 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:02.603039+0000", 2024-02-09T17:44:38.008 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:38.008 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.008 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.009 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:44:38.009 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:02.603039+0000", 2024-02-09T17:44:38.009 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:38.009 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.009 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.009 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:44:38.009 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:02.603041+0000", 2024-02-09T17:44:38.009 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5510000000000001e-06 2024-02-09T17:44:38.010 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.010 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.010 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:44:38.010 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:02.603045+0000", 2024-02-09T17:44:38.010 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2119999999999997e-06 2024-02-09T17:44:38.010 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.010 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.010 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:44:38.010 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:02.603054+0000", 2024-02-09T17:44:38.010 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.2490000000000007e-06 2024-02-09T17:44:38.010 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.010 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.010 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:44:38.010 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:02.603061+0000", 2024-02-09T17:44:38.010 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.3629999999999996e-06 2024-02-09T17:44:38.011 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.011 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.011 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:38.011 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:02.603222+0000", 2024-02-09T17:44:38.011 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00016069599999999999 2024-02-09T17:44:38.011 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.011 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.011 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:44:38.011 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:02.603229+0000", 2024-02-09T17:44:38.011 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.8770000000000004e-06 2024-02-09T17:44:38.011 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.011 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.011 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:38.011 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:03.593353+0000", 2024-02-09T17:44:38.011 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.99012438400000002 2024-02-09T17:44:38.012 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.012 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.012 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:44:38.012 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:03.593372+0000", 2024-02-09T17:44:38.012 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.8258e-05 2024-02-09T17:44:38.012 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.012 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.012 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:38.012 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:04.822274+0000", 2024-02-09T17:44:38.012 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.228901875 2024-02-09T17:44:38.012 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.012 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.013 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:44:38.013 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:04.822386+0000", 2024-02-09T17:44:38.013 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000112428 2024-02-09T17:44:38.013 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.013 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.013 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 5,7,10", 2024-02-09T17:44:38.013 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:04.822515+0000", 2024-02-09T17:44:38.013 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000129338 2024-02-09T17:44:38.013 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.013 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.013 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:44:38.013 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:04.836316+0000", 2024-02-09T17:44:38.013 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.013800442 2024-02-09T17:44:38.013 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.013 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.014 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:38.014 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:04.836359+0000", 2024-02-09T17:44:38.014 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.3720000000000002e-05 2024-02-09T17:44:38.014 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.014 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.014 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:38.014 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:04.845056+0000", 2024-02-09T17:44:38.014 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0086968759999999992 2024-02-09T17:44:38.014 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.014 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.014 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:38.014 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:04.845224+0000", 2024-02-09T17:44:38.014 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000167889 2024-02-09T17:44:38.015 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.015 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.015 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:44:38.015 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:04.845291+0000", 2024-02-09T17:44:38.015 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.7212000000000006e-05 2024-02-09T17:44:38.015 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.015 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.015 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:44:38.015 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:04.845426+0000", 2024-02-09T17:44:38.015 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000134163 2024-02-09T17:44:38.015 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:38.015 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:38.015 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:38.016 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.016 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.016 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:222305 4.a 4:5aeef130:::benchmark_data_smithi179_103809_object84839:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e399)", 2024-02-09T17:44:38.016 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:39:03.789959+0000", 2024-02-09T17:44:38.016 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 334.14289903999997, 2024-02-09T17:44:38.016 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.055702921, 2024-02-09T17:44:38.016 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:44:38.016 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:44:38.016 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:44:38.016 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:44:38.016 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:44:38.016 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 222305 2024-02-09T17:44:38.016 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.017 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:44:38.017 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.017 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:44:38.017 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:03.789959+0000", 2024-02-09T17:44:38.017 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:38.017 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.017 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.017 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:44:38.017 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:03.789959+0000", 2024-02-09T17:44:38.017 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:38.017 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.017 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.017 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:44:38.017 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:03.789960+0000", 2024-02-09T17:44:38.018 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0389999999999999e-06 2024-02-09T17:44:38.018 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.018 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.018 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:44:38.018 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:03.789964+0000", 2024-02-09T17:44:38.018 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2440000000000004e-06 2024-02-09T17:44:38.018 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.018 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.018 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:44:38.018 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:03.789974+0000", 2024-02-09T17:44:38.018 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0047999999999999e-05 2024-02-09T17:44:38.019 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.019 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.019 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:44:38.019 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:03.789980+0000", 2024-02-09T17:44:38.019 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7590000000000003e-06 2024-02-09T17:44:38.019 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.019 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.019 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:38.019 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:03.790042+0000", 2024-02-09T17:44:38.019 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.2090999999999999e-05 2024-02-09T17:44:38.019 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.019 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.019 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:44:38.019 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:03.790055+0000", 2024-02-09T17:44:38.020 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.288e-05 2024-02-09T17:44:38.020 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.020 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.020 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:38.020 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:04.827439+0000", 2024-02-09T17:44:38.020 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0373842870000001 2024-02-09T17:44:38.020 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.020 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.020 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:44:38.020 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:04.827551+0000", 2024-02-09T17:44:38.020 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000112283 2024-02-09T17:44:38.020 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.021 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.021 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 5,7,10", 2024-02-09T17:44:38.021 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:04.827688+0000", 2024-02-09T17:44:38.021 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013701499999999999 2024-02-09T17:44:38.021 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.021 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.021 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:44:38.021 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:04.840580+0000", 2024-02-09T17:44:38.021 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.012891767 2024-02-09T17:44:38.021 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.021 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.021 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:38.021 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:04.845116+0000", 2024-02-09T17:44:38.022 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0045357540000000003 2024-02-09T17:44:38.022 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.022 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.022 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:38.022 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:04.845175+0000", 2024-02-09T17:44:38.022 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.9490999999999997e-05 2024-02-09T17:44:38.022 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.022 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.022 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:38.022 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:04.845474+0000", 2024-02-09T17:44:38.022 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00029928399999999998 2024-02-09T17:44:38.022 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.022 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.022 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:44:38.023 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:04.845541+0000", 2024-02-09T17:44:38.023 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.6828999999999993e-05 2024-02-09T17:44:38.023 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.023 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.023 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:44:38.023 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:04.845661+0000", 2024-02-09T17:44:38.023 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012015000000000001 2024-02-09T17:44:38.023 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:38.023 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:38.023 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:38.023 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.023 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.023 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:232427 4.a 4:58636665:::benchmark_data_smithi179_103809_object94961:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-02-09T17:44:38.023 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:39:22.538697+0000", 2024-02-09T17:44:38.023 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 315.39416034800001, 2024-02-09T17:44:38.024 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.040289537, 2024-02-09T17:44:38.024 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:44:38.024 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:44:38.024 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:44:38.024 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:44:38.024 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:44:38.024 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 232427 2024-02-09T17:44:38.024 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.024 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:44:38.024 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.024 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:44:38.024 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:22.538697+0000", 2024-02-09T17:44:38.024 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:38.025 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.025 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.025 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:44:38.025 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:22.538697+0000", 2024-02-09T17:44:38.025 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:38.025 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.025 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.025 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:44:38.025 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:22.538699+0000", 2024-02-09T17:44:38.025 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.9989999999999998e-06 2024-02-09T17:44:38.025 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.025 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.025 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:44:38.025 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:22.538706+0000", 2024-02-09T17:44:38.026 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.3980000000000004e-06 2024-02-09T17:44:38.026 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.026 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.026 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:44:38.026 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:22.538720+0000", 2024-02-09T17:44:38.026 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4402e-05 2024-02-09T17:44:38.026 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.026 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.026 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:44:38.026 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:22.538728+0000", 2024-02-09T17:44:38.026 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.191e-06 2024-02-09T17:44:38.026 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.026 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.026 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:38.027 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:22.538800+0000", 2024-02-09T17:44:38.027 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.1508999999999996e-05 2024-02-09T17:44:38.027 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.027 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.027 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:44:38.027 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:22.538809+0000", 2024-02-09T17:44:38.027 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.9970000000000004e-06 2024-02-09T17:44:38.027 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.027 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.027 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:38.027 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.568239+0000", 2024-02-09T17:44:38.027 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.029430676 2024-02-09T17:44:38.027 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.027 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.028 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:44:38.028 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.568349+0000", 2024-02-09T17:44:38.028 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00010967 2024-02-09T17:44:38.028 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.028 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.028 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 5,10", 2024-02-09T17:44:38.028 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.568441+0000", 2024-02-09T17:44:38.028 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.2266000000000001e-05 2024-02-09T17:44:38.028 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.028 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.028 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:38.028 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.572628+0000", 2024-02-09T17:44:38.028 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0041869070000000001 2024-02-09T17:44:38.028 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.029 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.029 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:44:38.029 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.575783+0000", 2024-02-09T17:44:38.029 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0031545499999999999 2024-02-09T17:44:38.029 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.029 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.029 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:38.029 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.578929+0000", 2024-02-09T17:44:38.029 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0031464000000000002 2024-02-09T17:44:38.029 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.029 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.029 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:44:38.029 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.578962+0000", 2024-02-09T17:44:38.029 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.2634e-05 2024-02-09T17:44:38.030 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.030 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.030 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:44:38.030 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.578987+0000", 2024-02-09T17:44:38.030 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.4938000000000001e-05 2024-02-09T17:44:38.030 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:38.030 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:38.030 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:38.030 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.030 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.030 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:232455 4.a 4:5c3729de:::benchmark_data_smithi179_103809_object94989:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-02-09T17:44:38.031 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:39:22.579430+0000", 2024-02-09T17:44:38.031 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 315.353427662, 2024-02-09T17:44:38.031 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.011227925, 2024-02-09T17:44:38.031 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:44:38.031 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:44:38.031 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:44:38.031 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:44:38.031 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:44:38.031 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 232455 2024-02-09T17:44:38.031 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.031 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:44:38.031 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.031 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:44:38.032 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:22.579430+0000", 2024-02-09T17:44:38.032 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:38.032 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.032 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.032 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:44:38.032 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:22.579430+0000", 2024-02-09T17:44:38.032 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:38.032 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.032 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.032 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:44:38.032 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:22.579431+0000", 2024-02-09T17:44:38.032 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.22e-06 2024-02-09T17:44:38.032 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.033 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.033 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:44:38.033 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:22.579436+0000", 2024-02-09T17:44:38.033 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2689999999999999e-06 2024-02-09T17:44:38.033 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.033 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.033 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:44:38.033 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:22.579447+0000", 2024-02-09T17:44:38.033 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0338e-05 2024-02-09T17:44:38.033 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.033 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.033 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:44:38.033 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:22.579453+0000", 2024-02-09T17:44:38.033 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.9599999999999997e-06 2024-02-09T17:44:38.034 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.034 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.034 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:38.034 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:22.579511+0000", 2024-02-09T17:44:38.034 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7861e-05 2024-02-09T17:44:38.034 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.034 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.034 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:44:38.034 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:22.579521+0000", 2024-02-09T17:44:38.034 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0033e-05 2024-02-09T17:44:38.034 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.034 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.034 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:38.034 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.572668+0000", 2024-02-09T17:44:38.035 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.99314702600000004 2024-02-09T17:44:38.035 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.035 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.035 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:44:38.035 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.572758+0000", 2024-02-09T17:44:38.035 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.0660000000000003e-05 2024-02-09T17:44:38.035 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.035 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.035 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 5,10", 2024-02-09T17:44:38.035 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.572828+0000", 2024-02-09T17:44:38.035 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.9357999999999996e-05 2024-02-09T17:44:38.035 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.035 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.035 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:38.035 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.578834+0000", 2024-02-09T17:44:38.036 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0060068020000000003 2024-02-09T17:44:38.036 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.036 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.036 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:44:38.036 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.579059+0000", 2024-02-09T17:44:38.036 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00022447299999999999 2024-02-09T17:44:38.036 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.036 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.036 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:38.036 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.590550+0000", 2024-02-09T17:44:38.036 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.011491533 2024-02-09T17:44:38.036 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.036 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.036 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:44:38.036 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.590611+0000", 2024-02-09T17:44:38.037 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.0819999999999997e-05 2024-02-09T17:44:38.037 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.037 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.037 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:44:38.037 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.590658+0000", 2024-02-09T17:44:38.037 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.6572000000000001e-05 2024-02-09T17:44:38.037 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:38.037 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:38.037 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:38.037 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.037 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.037 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:232552 4.9 4:9d8f33fb:::benchmark_data_smithi179_103809_object95086:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e414)", 2024-02-09T17:44:38.037 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:39:23.607410+0000", 2024-02-09T17:44:38.037 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 314.32544797600002, 2024-02-09T17:44:38.037 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.99336764799999999, 2024-02-09T17:44:38.038 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:44:38.038 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:44:38.038 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:44:38.038 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:44:38.038 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:44:38.038 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 232552 2024-02-09T17:44:38.038 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.038 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:44:38.038 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.038 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:44:38.038 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.607410+0000", 2024-02-09T17:44:38.038 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:38.038 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.039 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.039 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:44:38.039 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.607410+0000", 2024-02-09T17:44:38.039 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:38.039 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.039 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.039 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:44:38.039 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.607410+0000", 2024-02-09T17:44:38.039 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2300000000000002e-07 2024-02-09T17:44:38.039 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.039 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.039 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:44:38.039 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.607413+0000", 2024-02-09T17:44:38.039 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.1999999999999999e-06 2024-02-09T17:44:38.040 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.040 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.040 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:44:38.040 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.607425+0000", 2024-02-09T17:44:38.040 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1827e-05 2024-02-09T17:44:38.040 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.040 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.040 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:44:38.040 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.607431+0000", 2024-02-09T17:44:38.040 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.9229999999999999e-06 2024-02-09T17:44:38.040 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.040 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.040 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:38.041 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.607475+0000", 2024-02-09T17:44:38.041 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.4502000000000002e-05 2024-02-09T17:44:38.041 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.041 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.041 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:44:38.041 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.607481+0000", 2024-02-09T17:44:38.041 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6840000000000001e-06 2024-02-09T17:44:38.041 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.041 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.043 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:38.043 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.584768+0000", 2024-02-09T17:44:38.043 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.97728727800000004 2024-02-09T17:44:38.044 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.044 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.044 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:44:38.044 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.584863+0000", 2024-02-09T17:44:38.044 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.4399000000000002e-05 2024-02-09T17:44:38.044 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.044 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.044 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,3,4,11", 2024-02-09T17:44:38.044 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.584978+0000", 2024-02-09T17:44:38.044 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000115016 2024-02-09T17:44:38.044 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.044 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.044 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:44:38.045 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.589764+0000", 2024-02-09T17:44:38.045 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0047862499999999997 2024-02-09T17:44:38.045 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.045 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.045 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:38.045 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.590123+0000", 2024-02-09T17:44:38.045 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00035844000000000002 2024-02-09T17:44:38.045 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.045 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.045 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:38.045 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.590166+0000", 2024-02-09T17:44:38.045 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.3826999999999999e-05 2024-02-09T17:44:38.045 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.045 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.045 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:38.046 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.590208+0000", 2024-02-09T17:44:38.046 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.1360999999999999e-05 2024-02-09T17:44:38.046 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.046 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.046 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:38.046 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.600581+0000", 2024-02-09T17:44:38.046 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.010373746 2024-02-09T17:44:38.046 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.046 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.046 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:44:38.046 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.600650+0000", 2024-02-09T17:44:38.046 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.86e-05 2024-02-09T17:44:38.046 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.047 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.047 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:44:38.047 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.600777+0000", 2024-02-09T17:44:38.047 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012717199999999999 2024-02-09T17:44:38.047 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:38.047 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:38.047 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:38.047 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.047 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.047 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:232568 4.4 4:2cb129ac:::benchmark_data_smithi179_103809_object95102:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e415)", 2024-02-09T17:44:38.047 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:39:24.529753+0000", 2024-02-09T17:44:38.047 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 313.40310418000001, 2024-02-09T17:44:38.048 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.100915619, 2024-02-09T17:44:38.048 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:44:38.048 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:44:38.048 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:44:38.048 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:44:38.048 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:44:38.048 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 232568 2024-02-09T17:44:38.048 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.048 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:44:38.048 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.048 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:44:38.048 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529753+0000", 2024-02-09T17:44:38.048 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:38.048 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.049 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.049 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:44:38.049 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529753+0000", 2024-02-09T17:44:38.049 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:38.049 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.049 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.049 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:44:38.049 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529755+0000", 2024-02-09T17:44:38.049 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.389e-06 2024-02-09T17:44:38.049 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.049 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.049 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:44:38.050 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529760+0000", 2024-02-09T17:44:38.050 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.4840000000000003e-06 2024-02-09T17:44:38.050 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.050 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.050 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:44:38.050 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529771+0000", 2024-02-09T17:44:38.050 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0529e-05 2024-02-09T17:44:38.050 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.050 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.050 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:44:38.050 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529780+0000", 2024-02-09T17:44:38.050 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.1200000000000008e-06 2024-02-09T17:44:38.050 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.050 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.050 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:38.051 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529952+0000", 2024-02-09T17:44:38.051 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000172323 2024-02-09T17:44:38.051 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.051 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.051 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:44:38.051 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529964+0000", 2024-02-09T17:44:38.051 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1834000000000001e-05 2024-02-09T17:44:38.051 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.051 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.051 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:38.051 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.601463+0000", 2024-02-09T17:44:38.051 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0714987389999999 2024-02-09T17:44:38.051 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.052 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.052 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:44:38.052 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.601595+0000", 2024-02-09T17:44:38.052 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013218999999999999 2024-02-09T17:44:38.052 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.052 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.052 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 2,3,7,10", 2024-02-09T17:44:38.052 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.601736+0000", 2024-02-09T17:44:38.052 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014145800000000001 2024-02-09T17:44:38.052 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.052 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.052 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:38.052 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.619078+0000", 2024-02-09T17:44:38.053 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.017341235999999999 2024-02-09T17:44:38.053 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.053 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.053 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:38.053 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.619119+0000", 2024-02-09T17:44:38.053 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.1523999999999997e-05 2024-02-09T17:44:38.053 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.053 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.053 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:38.053 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.619159+0000", 2024-02-09T17:44:38.053 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9576000000000002e-05 2024-02-09T17:44:38.053 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.053 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.053 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:44:38.053 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.625012+0000", 2024-02-09T17:44:38.054 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0058534930000000004 2024-02-09T17:44:38.054 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.054 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.054 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:38.054 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.630457+0000", 2024-02-09T17:44:38.054 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0054445379999999996 2024-02-09T17:44:38.055 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.055 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.055 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:44:38.055 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.630519+0000", 2024-02-09T17:44:38.055 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.2483999999999993e-05 2024-02-09T17:44:38.055 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.055 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.055 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:44:38.055 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.630669+0000", 2024-02-09T17:44:38.055 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014970200000000001 2024-02-09T17:44:38.055 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:38.056 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:38.056 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:38.056 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.056 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.056 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:232574 4.4 4:2e59b7f1:::benchmark_data_smithi179_103809_object95108:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e415)", 2024-02-09T17:44:38.056 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:39:24.529801+0000", 2024-02-09T17:44:38.056 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 313.40305628700003, 2024-02-09T17:44:38.056 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.10112791, 2024-02-09T17:44:38.056 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:44:38.056 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:44:38.056 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:44:38.056 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:44:38.056 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:44:38.056 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 232574 2024-02-09T17:44:38.056 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.057 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:44:38.057 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.057 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:44:38.057 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529801+0000", 2024-02-09T17:44:38.057 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:38.057 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.057 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.057 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:44:38.057 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529801+0000", 2024-02-09T17:44:38.057 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:38.057 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.057 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.057 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:44:38.057 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529802+0000", 2024-02-09T17:44:38.057 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3099999999999998e-07 2024-02-09T17:44:38.058 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.058 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.058 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:44:38.058 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529804+0000", 2024-02-09T17:44:38.058 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.5689999999999999e-06 2024-02-09T17:44:38.058 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.058 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.058 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:44:38.058 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529811+0000", 2024-02-09T17:44:38.058 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.8689999999999998e-06 2024-02-09T17:44:38.058 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.058 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.058 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:44:38.058 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529817+0000", 2024-02-09T17:44:38.058 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.9410000000000004e-06 2024-02-09T17:44:38.058 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.059 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.059 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:38.059 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.529995+0000", 2024-02-09T17:44:38.059 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000177842 2024-02-09T17:44:38.059 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.059 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.059 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:44:38.059 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.530004+0000", 2024-02-09T17:44:38.059 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.0510000000000002e-06 2024-02-09T17:44:38.059 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.059 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.059 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:38.059 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.608017+0000", 2024-02-09T17:44:38.059 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0780130729999999 2024-02-09T17:44:38.059 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.060 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.060 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:44:38.060 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.608153+0000", 2024-02-09T17:44:38.060 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013630000000000001 2024-02-09T17:44:38.060 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.060 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.060 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 2,3,7,10", 2024-02-09T17:44:38.060 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.608294+0000", 2024-02-09T17:44:38.060 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014021699999999999 2024-02-09T17:44:38.060 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.060 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.060 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:38.060 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.619198+0000", 2024-02-09T17:44:38.060 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.010904173 2024-02-09T17:44:38.060 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.060 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.061 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:38.061 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.619240+0000", 2024-02-09T17:44:38.061 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2422999999999997e-05 2024-02-09T17:44:38.061 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.061 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.061 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:44:38.061 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.625019+0000", 2024-02-09T17:44:38.061 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0057783959999999999 2024-02-09T17:44:38.061 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.061 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.061 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:38.061 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.625096+0000", 2024-02-09T17:44:38.061 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.6874999999999998e-05 2024-02-09T17:44:38.061 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.061 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.062 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:38.062 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.630718+0000", 2024-02-09T17:44:38.062 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0056226269999999998 2024-02-09T17:44:38.062 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.062 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.062 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:44:38.062 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.630786+0000", 2024-02-09T17:44:38.062 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.7532000000000003e-05 2024-02-09T17:44:38.062 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.062 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.062 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:44:38.062 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.630929+0000", 2024-02-09T17:44:38.062 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014349100000000001 2024-02-09T17:44:38.062 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:38.062 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:38.063 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:38.063 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.063 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.063 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:232585 4.4 4:26ea6e8c:::benchmark_data_smithi179_103809_object95119:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-02-09T17:44:38.063 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:39:24.601261+0000", 2024-02-09T17:44:38.063 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 313.33159640000002, 2024-02-09T17:44:38.063 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0298792859999999, 2024-02-09T17:44:38.063 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:44:38.063 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:44:38.063 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:44:38.063 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:44:38.063 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:44:38.063 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 232585 2024-02-09T17:44:38.063 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.063 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:44:38.064 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.064 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:44:38.064 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.601261+0000", 2024-02-09T17:44:38.064 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:38.064 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.064 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.064 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:44:38.064 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.601261+0000", 2024-02-09T17:44:38.064 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:38.064 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.064 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.064 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:44:38.064 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.601263+0000", 2024-02-09T17:44:38.064 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3170000000000001e-06 2024-02-09T17:44:38.064 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.065 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.065 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:44:38.065 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.601268+0000", 2024-02-09T17:44:38.065 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.13e-06 2024-02-09T17:44:38.065 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.065 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.065 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:44:38.065 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.601276+0000", 2024-02-09T17:44:38.065 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.7399999999999993e-06 2024-02-09T17:44:38.065 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.065 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.065 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:44:38.065 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.601284+0000", 2024-02-09T17:44:38.065 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.7409999999999992e-06 2024-02-09T17:44:38.065 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.065 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.066 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:38.066 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.601343+0000", 2024-02-09T17:44:38.066 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.9208e-05 2024-02-09T17:44:38.066 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.066 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.066 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:44:38.066 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.601353+0000", 2024-02-09T17:44:38.066 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.2629999999999992e-06 2024-02-09T17:44:38.066 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.066 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.066 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:38.066 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.613145+0000", 2024-02-09T17:44:38.066 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0117919879999999 2024-02-09T17:44:38.066 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.066 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.066 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:44:38.067 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.613289+0000", 2024-02-09T17:44:38.067 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000144883 2024-02-09T17:44:38.067 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.067 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.067 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 2,3,7,10", 2024-02-09T17:44:38.067 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.613455+0000", 2024-02-09T17:44:38.067 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000165854 2024-02-09T17:44:38.067 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.067 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.067 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:38.067 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.624998+0000", 2024-02-09T17:44:38.067 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.011542327 2024-02-09T17:44:38.067 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.067 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.067 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:44:38.067 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.625025+0000", 2024-02-09T17:44:38.068 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.7263999999999998e-05 2024-02-09T17:44:38.068 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.068 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.068 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:38.068 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.625058+0000", 2024-02-09T17:44:38.068 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.3288999999999999e-05 2024-02-09T17:44:38.068 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.068 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.068 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:38.068 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.625193+0000", 2024-02-09T17:44:38.068 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000134442 2024-02-09T17:44:38.068 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.068 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.068 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:38.068 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.630977+0000", 2024-02-09T17:44:38.069 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0057848250000000004 2024-02-09T17:44:38.069 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.069 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.069 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:44:38.069 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.631011+0000", 2024-02-09T17:44:38.069 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.3411000000000002e-05 2024-02-09T17:44:38.069 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.069 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.069 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:44:38.069 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:25.631140+0000", 2024-02-09T17:44:38.069 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012960399999999999 2024-02-09T17:44:38.069 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:38.069 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:38.069 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:38.069 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.070 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.070 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25528.0:205147 5.c 5:3d54b93b:::benchmark_data_smithi179_115199_object76241:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e545)", 2024-02-09T17:44:38.070 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:44:12.647483+0000", 2024-02-09T17:44:38.070 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 25.285374199, 2024-02-09T17:44:38.070 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.21472992199999999, 2024-02-09T17:44:38.070 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:44:38.070 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:44:38.070 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:44:38.070 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25528", 2024-02-09T17:44:38.070 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:44:38.070 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 205147 2024-02-09T17:44:38.070 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.070 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:44:38.070 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.070 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:44:38.071 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.647483+0000", 2024-02-09T17:44:38.071 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:38.071 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.071 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.071 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:44:38.071 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.647483+0000", 2024-02-09T17:44:38.071 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:38.071 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.071 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.071 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:44:38.071 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.647484+0000", 2024-02-09T17:44:38.071 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.3699999999999999e-07 2024-02-09T17:44:38.071 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.071 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.071 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:44:38.072 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.647489+0000", 2024-02-09T17:44:38.072 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.4089999999999996e-06 2024-02-09T17:44:38.072 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.072 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.072 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:44:38.072 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.647499+0000", 2024-02-09T17:44:38.072 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0009e-05 2024-02-09T17:44:38.072 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.072 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.072 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:44:38.072 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.647506+0000", 2024-02-09T17:44:38.072 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.3590000000000002e-06 2024-02-09T17:44:38.072 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.072 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.073 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:38.073 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.654804+0000", 2024-02-09T17:44:38.073 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0072982569999999998 2024-02-09T17:44:38.073 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.073 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.073 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:44:38.073 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.654936+0000", 2024-02-09T17:44:38.073 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013139900000000001 2024-02-09T17:44:38.073 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.073 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.073 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,4", 2024-02-09T17:44:38.073 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.655046+0000", 2024-02-09T17:44:38.073 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000109844 2024-02-09T17:44:38.073 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.073 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.074 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:44:38.074 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.664632+0000", 2024-02-09T17:44:38.074 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0095869839999999998 2024-02-09T17:44:38.074 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.074 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.074 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:38.074 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.674939+0000", 2024-02-09T17:44:38.074 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.010306793 2024-02-09T17:44:38.074 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.074 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.074 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:38.074 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.862106+0000", 2024-02-09T17:44:38.074 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.18716676300000001 2024-02-09T17:44:38.074 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.074 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.075 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:44:38.075 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.862167+0000", 2024-02-09T17:44:38.075 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.1282e-05 2024-02-09T17:44:38.075 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.075 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.075 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:44:38.075 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.862213+0000", 2024-02-09T17:44:38.075 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.5986000000000003e-05 2024-02-09T17:44:38.075 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:38.075 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:38.075 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:38.075 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.075 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.075 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25528.0:205157 5.c 5:3c306125:::benchmark_data_smithi179_115199_object76251:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e545)", 2024-02-09T17:44:38.075 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:44:12.659119+0000", 2024-02-09T17:44:38.076 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 25.273738587, 2024-02-09T17:44:38.076 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.22704324200000001, 2024-02-09T17:44:38.076 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:44:38.076 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:44:38.076 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:44:38.076 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25528", 2024-02-09T17:44:38.076 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:44:38.076 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 205157 2024-02-09T17:44:38.076 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.076 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:44:38.076 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.076 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:44:38.076 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.659119+0000", 2024-02-09T17:44:38.076 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:38.077 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.077 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.077 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:44:38.077 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.659119+0000", 2024-02-09T17:44:38.077 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:38.077 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.077 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.077 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:44:38.077 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.659120+0000", 2024-02-09T17:44:38.077 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.124e-06 2024-02-09T17:44:38.077 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.077 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.077 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:44:38.077 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.659124+0000", 2024-02-09T17:44:38.077 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2100000000000003e-06 2024-02-09T17:44:38.078 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.078 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.078 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:44:38.078 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.659134+0000", 2024-02-09T17:44:38.078 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.4450000000000002e-06 2024-02-09T17:44:38.078 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.078 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.078 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:44:38.078 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.659140+0000", 2024-02-09T17:44:38.078 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.5699999999999998e-06 2024-02-09T17:44:38.078 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.078 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.078 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:38.078 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.660338+0000", 2024-02-09T17:44:38.078 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.001197175 2024-02-09T17:44:38.078 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.079 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.079 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:44:38.079 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.660445+0000", 2024-02-09T17:44:38.079 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000107372 2024-02-09T17:44:38.079 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.079 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.079 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,4", 2024-02-09T17:44:38.079 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.660523+0000", 2024-02-09T17:44:38.079 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.8166999999999999e-05 2024-02-09T17:44:38.079 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.079 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.079 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:44:38.079 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.668656+0000", 2024-02-09T17:44:38.080 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0081332209999999995 2024-02-09T17:44:38.080 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.080 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.080 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:38.080 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.674967+0000", 2024-02-09T17:44:38.080 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0063109539999999997 2024-02-09T17:44:38.080 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.080 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.080 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:38.080 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.886064+0000", 2024-02-09T17:44:38.080 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.21109720600000001 2024-02-09T17:44:38.080 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.080 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.080 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:44:38.080 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.886124+0000", 2024-02-09T17:44:38.081 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.9246000000000002e-05 2024-02-09T17:44:38.081 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.081 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.081 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:44:38.081 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.886162+0000", 2024-02-09T17:44:38.081 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.8552e-05 2024-02-09T17:44:38.081 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:38.081 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:38.081 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:38.081 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.081 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.081 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25528.0:205161 5.c 5:36f4ec77:::benchmark_data_smithi179_115199_object76255:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e545)", 2024-02-09T17:44:38.081 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:44:12.661343+0000", 2024-02-09T17:44:38.081 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 25.271514939999999, 2024-02-09T17:44:38.081 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.23728677200000001, 2024-02-09T17:44:38.081 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:44:38.082 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:44:38.082 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:44:38.082 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25528", 2024-02-09T17:44:38.082 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:44:38.082 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 205161 2024-02-09T17:44:38.082 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.082 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:44:38.082 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.082 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:44:38.082 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.661343+0000", 2024-02-09T17:44:38.082 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:38.082 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.082 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.082 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:44:38.082 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.661343+0000", 2024-02-09T17:44:38.083 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:38.083 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.083 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.083 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:44:38.083 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.661344+0000", 2024-02-09T17:44:38.083 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0109999999999999e-06 2024-02-09T17:44:38.083 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.083 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.083 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:44:38.083 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.661348+0000", 2024-02-09T17:44:38.083 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.6700000000000002e-06 2024-02-09T17:44:38.083 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.083 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.083 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:44:38.084 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.661362+0000", 2024-02-09T17:44:38.084 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3993e-05 2024-02-09T17:44:38.084 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.084 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.084 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:44:38.084 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.661370+0000", 2024-02-09T17:44:38.084 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.6299999999999998e-06 2024-02-09T17:44:38.084 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.084 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.084 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:38.084 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.664672+0000", 2024-02-09T17:44:38.084 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0033018969999999998 2024-02-09T17:44:38.084 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.084 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.084 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:44:38.085 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.664804+0000", 2024-02-09T17:44:38.085 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013193999999999999 2024-02-09T17:44:38.085 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.085 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.085 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,4", 2024-02-09T17:44:38.085 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.664932+0000", 2024-02-09T17:44:38.085 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000127785 2024-02-09T17:44:38.085 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.085 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.085 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:44:38.085 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.668788+0000", 2024-02-09T17:44:38.085 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0038565460000000002 2024-02-09T17:44:38.085 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.085 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.085 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:38.086 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.683656+0000", 2024-02-09T17:44:38.086 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.014868084 2024-02-09T17:44:38.086 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.086 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.086 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:38.086 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.898526+0000", 2024-02-09T17:44:38.086 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.21486998600000001 2024-02-09T17:44:38.086 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.086 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.086 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:44:38.086 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.898588+0000", 2024-02-09T17:44:38.086 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.1705000000000006e-05 2024-02-09T17:44:38.086 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.086 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.086 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:44:38.086 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.898629+0000", 2024-02-09T17:44:38.087 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.1525e-05 2024-02-09T17:44:38.087 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:38.087 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:38.087 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:38.087 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.087 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.087 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25528.0:205169 5.c 5:3fe8ba45:::benchmark_data_smithi179_115199_object76263:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e545)", 2024-02-09T17:44:38.087 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:44:12.676575+0000", 2024-02-09T17:44:38.087 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 25.256282977000001, 2024-02-09T17:44:38.087 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.222204398, 2024-02-09T17:44:38.087 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:44:38.087 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:44:38.087 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:44:38.087 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25528", 2024-02-09T17:44:38.087 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:44:38.087 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 205169 2024-02-09T17:44:38.088 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.088 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:44:38.088 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.088 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:44:38.088 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.676575+0000", 2024-02-09T17:44:38.088 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:38.088 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.088 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.088 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:44:38.088 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.676575+0000", 2024-02-09T17:44:38.088 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:44:38.088 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.088 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.088 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:44:38.088 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.676576+0000", 2024-02-09T17:44:38.089 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.159e-06 2024-02-09T17:44:38.089 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.089 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.089 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:44:38.089 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.676580+0000", 2024-02-09T17:44:38.089 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.5870000000000003e-06 2024-02-09T17:44:38.089 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.089 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.089 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:44:38.089 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.676590+0000", 2024-02-09T17:44:38.089 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.8230000000000006e-06 2024-02-09T17:44:38.089 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.089 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.089 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:44:38.089 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.676596+0000", 2024-02-09T17:44:38.089 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7819999999999999e-06 2024-02-09T17:44:38.090 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.090 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.090 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:44:38.090 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.676635+0000", 2024-02-09T17:44:38.090 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9239999999999997e-05 2024-02-09T17:44:38.090 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.090 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.090 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:44:38.090 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.676719+0000", 2024-02-09T17:44:38.090 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.3477000000000001e-05 2024-02-09T17:44:38.090 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.090 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.090 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,4", 2024-02-09T17:44:38.090 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.676794+0000", 2024-02-09T17:44:38.090 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.5443000000000003e-05 2024-02-09T17:44:38.090 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.091 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.091 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:44:38.091 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.681986+0000", 2024-02-09T17:44:38.091 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0051920159999999998 2024-02-09T17:44:38.091 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.091 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.091 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:38.091 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.685572+0000", 2024-02-09T17:44:38.091 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0035862939999999999 2024-02-09T17:44:38.091 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.091 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.091 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:44:38.091 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.898679+0000", 2024-02-09T17:44:38.091 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.213106183 2024-02-09T17:44:38.091 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.092 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.092 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:44:38.092 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.898737+0000", 2024-02-09T17:44:38.092 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.8146999999999997e-05 2024-02-09T17:44:38.092 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:44:38.092 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:44:38.092 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:44:38.092 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.898779+0000", 2024-02-09T17:44:38.092 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2246999999999997e-05 2024-02-09T17:44:38.092 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:38.092 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:38.092 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:38.092 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:44:38.092 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:44:38.092 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:44:38.093 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:44:38.193 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:44:38.259 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:38.312 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:44:38.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:38.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:44:38.424 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:44:38.461 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:38.486 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:44:38.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:38.491 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:44:38.581 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:38 smithi162 ceph-mon[44304]: 1.7 scrub starts 2024-02-09T17:44:38.581 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:38 smithi162 ceph-mon[44304]: 1.15 scrub starts 2024-02-09T17:44:38.581 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:38 smithi162 ceph-mon[44304]: 1.7 scrub ok 2024-02-09T17:44:38.581 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:38 smithi162 ceph-mon[44304]: 1.15 scrub ok 2024-02-09T17:44:38.581 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:38 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:44:38.581 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:38 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:44:38.592 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:44:38.638 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:38.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:38 smithi099 ceph-mon[39420]: 1.7 scrub starts 2024-02-09T17:44:38.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:38 smithi099 ceph-mon[39420]: 1.15 scrub starts 2024-02-09T17:44:38.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:38 smithi099 ceph-mon[39420]: 1.7 scrub ok 2024-02-09T17:44:38.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:38 smithi099 ceph-mon[39420]: 1.15 scrub ok 2024-02-09T17:44:38.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:38 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:44:38.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:38 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:44:38.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:38 smithi179 ceph-mon[43975]: 1.7 scrub starts 2024-02-09T17:44:38.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:38 smithi179 ceph-mon[43975]: 1.15 scrub starts 2024-02-09T17:44:38.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:38 smithi179 ceph-mon[43975]: 1.7 scrub ok 2024-02-09T17:44:38.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:38 smithi179 ceph-mon[43975]: 1.15 scrub ok 2024-02-09T17:44:38.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:38 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:44:38.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:38 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:44:39.898 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:39 smithi099 ceph-mon[39420]: 1.10 scrub starts 2024-02-09T17:44:39.898 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:39 smithi099 ceph-mon[39420]: 1.10 scrub ok 2024-02-09T17:44:39.898 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:39 smithi099 ceph-mon[39420]: osdmap e552: 12 total, 12 up, 12 in 2024-02-09T17:44:39.898 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:39 smithi099 ceph-mon[39420]: pgmap v1389: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 2.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 791 op/s; 129/98280 objects misplaced (0.131%); 11 MiB/s, 182 objects/s recovering 2024-02-09T17:44:39.910 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:44:39.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:39.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:44:39.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:39 smithi179 ceph-mon[43975]: 1.10 scrub starts 2024-02-09T17:44:39.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:39 smithi179 ceph-mon[43975]: 1.10 scrub ok 2024-02-09T17:44:39.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:39 smithi179 ceph-mon[43975]: osdmap e552: 12 total, 12 up, 12 in 2024-02-09T17:44:39.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:39 smithi179 ceph-mon[43975]: pgmap v1389: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 2.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 791 op/s; 129/98280 objects misplaced (0.131%); 11 MiB/s, 182 objects/s recovering 2024-02-09T17:44:39.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:39 smithi162 ceph-mon[44304]: 1.10 scrub starts 2024-02-09T17:44:39.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:39 smithi162 ceph-mon[44304]: 1.10 scrub ok 2024-02-09T17:44:39.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:39 smithi162 ceph-mon[44304]: osdmap e552: 12 total, 12 up, 12 in 2024-02-09T17:44:39.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:39 smithi162 ceph-mon[44304]: pgmap v1389: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 2.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 791 op/s; 129/98280 objects misplaced (0.131%); 11 MiB/s, 182 objects/s recovering 2024-02-09T17:44:40.015 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:44:40.056 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:40.611 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is unset 2024-02-09T17:44:40.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:40 smithi179 ceph-mon[43975]: osdmap e553: 12 total, 12 up, 12 in 2024-02-09T17:44:40.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:40 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/344466002' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:44:40.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:40 smithi162 ceph-mon[44304]: osdmap e553: 12 total, 12 up, 12 in 2024-02-09T17:44:40.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:40 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/344466002' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:44:40.998 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:40 smithi099 ceph-mon[39420]: osdmap e553: 12 total, 12 up, 12 in 2024-02-09T17:44:40.998 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:40 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/344466002' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:44:41.694 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:44:41.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:41.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:44:41.801 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:44:41.858 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-09T17:44:41.859 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:41 smithi162 ceph-mon[44304]: 1.6 scrub starts 2024-02-09T17:44:41.859 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:41 smithi162 ceph-mon[44304]: 1.6 scrub ok 2024-02-09T17:44:41.859 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:41 smithi162 ceph-mon[44304]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:44:41.859 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:41 smithi162 ceph-mon[44304]: Cluster is now healthy 2024-02-09T17:44:41.859 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:41 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/344466002' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:44:41.859 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:41 smithi162 ceph-mon[44304]: osdmap e554: 12 total, 12 up, 12 in 2024-02-09T17:44:41.859 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:41 smithi162 ceph-mon[44304]: pgmap v1392: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 761 op/s; 129/93216 objects misplaced (0.138%); 16 MiB/s, 255 objects/s recovering 2024-02-09T17:44:41.859 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:41 smithi162 ceph-mon[44304]: osdmap e555: 12 total, 12 up, 12 in 2024-02-09T17:44:41.862 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:41 smithi099 ceph-mon[39420]: 1.6 scrub starts 2024-02-09T17:44:41.862 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:41 smithi099 ceph-mon[39420]: 1.6 scrub ok 2024-02-09T17:44:41.862 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:41 smithi099 ceph-mon[39420]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:44:41.862 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:41 smithi099 ceph-mon[39420]: Cluster is now healthy 2024-02-09T17:44:41.862 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:41 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/344466002' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:44:41.862 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:41 smithi099 ceph-mon[39420]: osdmap e554: 12 total, 12 up, 12 in 2024-02-09T17:44:41.862 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:41 smithi099 ceph-mon[39420]: pgmap v1392: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 761 op/s; 129/93216 objects misplaced (0.138%); 16 MiB/s, 255 objects/s recovering 2024-02-09T17:44:41.862 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:41 smithi099 ceph-mon[39420]: osdmap e555: 12 total, 12 up, 12 in 2024-02-09T17:44:41.867 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:41.910 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:44:41.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:41.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:44:41.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:41 smithi179 ceph-mon[43975]: 1.6 scrub starts 2024-02-09T17:44:41.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:41 smithi179 ceph-mon[43975]: 1.6 scrub ok 2024-02-09T17:44:41.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:41 smithi179 ceph-mon[43975]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:44:41.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:41 smithi179 ceph-mon[43975]: Cluster is now healthy 2024-02-09T17:44:41.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:41 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/344466002' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:44:41.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:41 smithi179 ceph-mon[43975]: osdmap e554: 12 total, 12 up, 12 in 2024-02-09T17:44:41.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:41 smithi179 ceph-mon[43975]: pgmap v1392: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 761 op/s; 129/93216 objects misplaced (0.138%); 16 MiB/s, 255 objects/s recovering 2024-02-09T17:44:41.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:41 smithi179 ceph-mon[43975]: osdmap e555: 12 total, 12 up, 12 in 2024-02-09T17:44:42.016 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:44:42.067 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:42.109 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:44:42.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:42.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:44:42.220 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:44:42.256 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:42.280 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:44:42.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:42.283 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:44:42.384 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:44:42.433 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:42.514 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:44:42.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:42.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:44:42.620 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:44:42.671 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:42.708 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:44:42.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:42.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:44:42.742 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:42 smithi099 ceph-mon[39420]: 1.5 scrub starts 2024-02-09T17:44:42.742 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:42 smithi099 ceph-mon[39420]: 1.5 scrub ok 2024-02-09T17:44:42.815 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:44:42.851 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:42.894 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:44:42.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:42.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:44:42.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:42 smithi179 ceph-mon[43975]: 1.5 scrub starts 2024-02-09T17:44:42.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:42 smithi179 ceph-mon[43975]: 1.5 scrub ok 2024-02-09T17:44:42.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:42 smithi162 ceph-mon[44304]: 1.5 scrub starts 2024-02-09T17:44:42.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:42 smithi162 ceph-mon[44304]: 1.5 scrub ok 2024-02-09T17:44:43.011 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:44:43.085 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:43.125 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:44:43.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:43.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:44:43.234 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:44:43.305 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:43.373 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:44:43.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:43.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:44:43.476 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:44:43.522 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:43.560 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:44:43.600 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:44:43.634 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:44:43.663 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:44:43.714 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:44:43.752 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:44:43.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:43.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:44:43.764 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:44:43.792 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:43 smithi099 ceph-mon[39420]: 1.f scrub starts 2024-02-09T17:44:43.793 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:43 smithi099 ceph-mon[39420]: 1.f scrub ok 2024-02-09T17:44:43.793 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:43 smithi099 ceph-mon[39420]: pgmap v1394: 47 pgs: 47 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 780 op/s; 1.9 MiB/s, 29 objects/s recovering 2024-02-09T17:44:43.793 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:43 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1210995025' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:44:43.810 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:44:43.845 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:44:43.857 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:44:43.877 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:44:43.894 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:43.908 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:44:43.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:43 smithi179 ceph-mon[43975]: 1.f scrub starts 2024-02-09T17:44:43.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:43 smithi179 ceph-mon[43975]: 1.f scrub ok 2024-02-09T17:44:43.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:43 smithi179 ceph-mon[43975]: pgmap v1394: 47 pgs: 47 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 780 op/s; 1.9 MiB/s, 29 objects/s recovering 2024-02-09T17:44:43.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:43 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1210995025' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:44:43.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:43 smithi162 ceph-mon[44304]: 1.f scrub starts 2024-02-09T17:44:43.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:43 smithi162 ceph-mon[44304]: 1.f scrub ok 2024-02-09T17:44:43.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:43 smithi162 ceph-mon[44304]: pgmap v1394: 47 pgs: 47 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 780 op/s; 1.9 MiB/s, 29 objects/s recovering 2024-02-09T17:44:43.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:43 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1210995025' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:44:43.940 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:44:43.972 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:44:43.983 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:44:43.984 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:44:43.984 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:44:43.984 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:44:43.984 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:44:43.984 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:44:43.984 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:44:43.984 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:44:43.984 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:44:43.984 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:44:43.984 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:44:43.984 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:44:44.105 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:44:44.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:44.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:44:44.214 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:44:44.252 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:44.281 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:44:44.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:44.293 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:44:44.394 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:44:44.468 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:44.501 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:44:44.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:44.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:44:44.544 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set noscrub 2024-02-09T17:44:44.610 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:44:44.651 INFO:teuthology.orchestra.run.smithi099.stderr:set 5.3 pg_upmap mapping to [4,1,2] 2024-02-09T17:44:44.676 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:44.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:44 smithi179 ceph-mon[43975]: 5.a scrub starts 2024-02-09T17:44:44.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:44 smithi179 ceph-mon[43975]: 5.a scrub ok 2024-02-09T17:44:44.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:44 smithi179 ceph-mon[43975]: 1.16 scrub starts 2024-02-09T17:44:44.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:44 smithi179 ceph-mon[43975]: 1.16 scrub ok 2024-02-09T17:44:44.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:44 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3271377079' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.3", "id": [4, 1, 2]}]: dispatch 2024-02-09T17:44:44.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:44 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.3", "id": [4, 1, 2]}]: dispatch 2024-02-09T17:44:44.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:44 smithi162 ceph-mon[44304]: 5.a scrub starts 2024-02-09T17:44:44.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:44 smithi162 ceph-mon[44304]: 5.a scrub ok 2024-02-09T17:44:44.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:44 smithi162 ceph-mon[44304]: 1.16 scrub starts 2024-02-09T17:44:44.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:44 smithi162 ceph-mon[44304]: 1.16 scrub ok 2024-02-09T17:44:44.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:44 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3271377079' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.3", "id": [4, 1, 2]}]: dispatch 2024-02-09T17:44:44.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:44 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.3", "id": [4, 1, 2]}]: dispatch 2024-02-09T17:44:44.942 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:44 smithi099 ceph-mon[39420]: 5.a scrub starts 2024-02-09T17:44:44.942 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:44 smithi099 ceph-mon[39420]: 5.a scrub ok 2024-02-09T17:44:44.942 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:44 smithi099 ceph-mon[39420]: 1.16 scrub starts 2024-02-09T17:44:44.942 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:44 smithi099 ceph-mon[39420]: 1.16 scrub ok 2024-02-09T17:44:44.942 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:44 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3271377079' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.3", "id": [4, 1, 2]}]: dispatch 2024-02-09T17:44:44.942 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:44 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.3", "id": [4, 1, 2]}]: dispatch 2024-02-09T17:44:45.177 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:44:45.258 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:44:45.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:45.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:44:45.365 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:44:45.395 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:45.413 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:44:45.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:45.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:44:45.516 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:44:45.546 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:45.567 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:44:45.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:45.571 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:44:45.636 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:44:45.671 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:44:45.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:45 smithi162 ceph-mon[44304]: 1.1b deep-scrub starts 2024-02-09T17:44:45.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:45 smithi162 ceph-mon[44304]: 1.1b deep-scrub ok 2024-02-09T17:44:45.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:45 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.3", "id": [4, 1, 2]}]': finished 2024-02-09T17:44:45.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:45 smithi162 ceph-mon[44304]: osdmap e556: 12 total, 12 up, 12 in 2024-02-09T17:44:45.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:45 smithi162 ceph-mon[44304]: pgmap v1396: 47 pgs: 47 active+clean; 1.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 769 op/s; 1.5 MiB/s, 24 objects/s recovering 2024-02-09T17:44:45.700 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:45.719 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:44:45.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:45.722 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:44:45.823 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:44:45.849 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:45 smithi099 ceph-mon[39420]: 1.1b deep-scrub starts 2024-02-09T17:44:45.850 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:45 smithi099 ceph-mon[39420]: 1.1b deep-scrub ok 2024-02-09T17:44:45.850 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:45 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.3", "id": [4, 1, 2]}]': finished 2024-02-09T17:44:45.850 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:45 smithi099 ceph-mon[39420]: osdmap e556: 12 total, 12 up, 12 in 2024-02-09T17:44:45.850 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:45 smithi099 ceph-mon[39420]: pgmap v1396: 47 pgs: 47 active+clean; 1.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 769 op/s; 1.5 MiB/s, 24 objects/s recovering 2024-02-09T17:44:45.851 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:45.879 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:44:45.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:45.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:44:45.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:45 smithi179 ceph-mon[43975]: 1.1b deep-scrub starts 2024-02-09T17:44:45.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:45 smithi179 ceph-mon[43975]: 1.1b deep-scrub ok 2024-02-09T17:44:45.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:45 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.3", "id": [4, 1, 2]}]': finished 2024-02-09T17:44:45.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:45 smithi179 ceph-mon[43975]: osdmap e556: 12 total, 12 up, 12 in 2024-02-09T17:44:45.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:45 smithi179 ceph-mon[43975]: pgmap v1396: 47 pgs: 47 active+clean; 1.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 769 op/s; 1.5 MiB/s, 24 objects/s recovering 2024-02-09T17:44:45.984 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:44:46.012 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:46.660 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:44:46.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:46.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:44:46.765 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:44:46.797 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:46.818 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:44:46.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:46.822 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:44:46.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:46 smithi179 ceph-mon[43975]: 1.9 scrub starts 2024-02-09T17:44:46.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:46 smithi179 ceph-mon[43975]: 1.9 scrub ok 2024-02-09T17:44:46.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:46 smithi179 ceph-mon[43975]: osdmap e557: 12 total, 12 up, 12 in 2024-02-09T17:44:46.922 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:44:46.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:46 smithi162 ceph-mon[44304]: 1.9 scrub starts 2024-02-09T17:44:46.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:46 smithi162 ceph-mon[44304]: 1.9 scrub ok 2024-02-09T17:44:46.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:46 smithi162 ceph-mon[44304]: osdmap e557: 12 total, 12 up, 12 in 2024-02-09T17:44:46.951 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:46.971 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:44:46.971 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:44:46.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:46.975 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:44:46.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:46 smithi099 ceph-mon[39420]: 1.9 scrub starts 2024-02-09T17:44:46.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:46 smithi099 ceph-mon[39420]: 1.9 scrub ok 2024-02-09T17:44:46.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:46 smithi099 ceph-mon[39420]: osdmap e557: 12 total, 12 up, 12 in 2024-02-09T17:44:47.075 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:44:47.125 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:47.172 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:44:47.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:47.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:44:47.279 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:44:47.315 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:47.365 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:44:47.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:47.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:44:47.470 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:44:47.512 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:47.766 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:44:47.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:47.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:44:47.770 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:47 smithi099 ceph-mon[39420]: osdmap e558: 12 total, 12 up, 12 in 2024-02-09T17:44:47.770 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:47 smithi099 ceph-mon[39420]: 5.3 starting backfill to osd.1 from (0'0,0'0] MAX to 555'15878 2024-02-09T17:44:47.770 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:47 smithi099 ceph-mon[39420]: 5.3 starting backfill to osd.2 from (0'0,0'0] MAX to 555'15878 2024-02-09T17:44:47.771 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:47 smithi099 ceph-mon[39420]: 5.3 starting backfill to osd.4 from (0'0,0'0] MAX to 555'15878 2024-02-09T17:44:47.771 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:47 smithi099 ceph-mon[39420]: pgmap v1399: 47 pgs: 47 active+clean; 1.6 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 747 op/s; 1.5 MiB/s, 24 objects/s recovering 2024-02-09T17:44:47.871 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:44:47.915 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:47.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:47 smithi179 ceph-mon[43975]: osdmap e558: 12 total, 12 up, 12 in 2024-02-09T17:44:47.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:47 smithi179 ceph-mon[43975]: 5.3 starting backfill to osd.1 from (0'0,0'0] MAX to 555'15878 2024-02-09T17:44:47.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:47 smithi179 ceph-mon[43975]: 5.3 starting backfill to osd.2 from (0'0,0'0] MAX to 555'15878 2024-02-09T17:44:47.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:47 smithi179 ceph-mon[43975]: 5.3 starting backfill to osd.4 from (0'0,0'0] MAX to 555'15878 2024-02-09T17:44:47.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:47 smithi179 ceph-mon[43975]: pgmap v1399: 47 pgs: 47 active+clean; 1.6 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 747 op/s; 1.5 MiB/s, 24 objects/s recovering 2024-02-09T17:44:47.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:47 smithi162 ceph-mon[44304]: osdmap e558: 12 total, 12 up, 12 in 2024-02-09T17:44:47.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:47 smithi162 ceph-mon[44304]: 5.3 starting backfill to osd.1 from (0'0,0'0] MAX to 555'15878 2024-02-09T17:44:47.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:47 smithi162 ceph-mon[44304]: 5.3 starting backfill to osd.2 from (0'0,0'0] MAX to 555'15878 2024-02-09T17:44:47.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:47 smithi162 ceph-mon[44304]: 5.3 starting backfill to osd.4 from (0'0,0'0] MAX to 555'15878 2024-02-09T17:44:47.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:47 smithi162 ceph-mon[44304]: pgmap v1399: 47 pgs: 47 active+clean; 1.6 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 747 op/s; 1.5 MiB/s, 24 objects/s recovering 2024-02-09T17:44:48.079 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:44:48.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:48.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:44:48.189 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:44:48.235 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:48.264 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:44:48.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:48.269 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:44:48.370 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:44:48.401 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:48.433 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:44:48.441 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:48.442 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:44:48.543 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:44:48.605 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:48.644 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:44:48.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:48.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:44:48.752 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:44:48.790 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:48.827 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:44:48.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:48.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:44:48.935 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:44:48.964 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:44:48.980 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:49.267 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:44:49.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:49.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:44:49.373 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:44:49.431 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:49.457 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:44:49.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:49.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:44:49.569 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:44:49.613 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:49.826 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:44:49.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:49.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:44:49.915 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:49 smithi162 ceph-mon[44304]: 1.1c scrub starts 2024-02-09T17:44:49.915 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:49 smithi162 ceph-mon[44304]: 1.1c scrub ok 2024-02-09T17:44:49.915 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:49 smithi162 ceph-mon[44304]: pgmap v1400: 47 pgs: 47 active+clean; 1.6 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 596 op/s 2024-02-09T17:44:49.915 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:49 smithi162 ceph-mon[44304]: 5.e scrub starts 2024-02-09T17:44:49.915 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:49 smithi162 ceph-mon[44304]: 5.e scrub ok 2024-02-09T17:44:49.939 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:44:49.979 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:50.023 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:44:50.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:50.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:44:50.127 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:44:50.163 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:49 smithi099 ceph-mon[39420]: 1.1c scrub starts 2024-02-09T17:44:50.163 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:49 smithi099 ceph-mon[39420]: 1.1c scrub ok 2024-02-09T17:44:50.163 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:49 smithi099 ceph-mon[39420]: pgmap v1400: 47 pgs: 47 active+clean; 1.6 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 596 op/s 2024-02-09T17:44:50.163 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:49 smithi099 ceph-mon[39420]: 5.e scrub starts 2024-02-09T17:44:50.163 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:49 smithi099 ceph-mon[39420]: 5.e scrub ok 2024-02-09T17:44:50.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:49 smithi179 ceph-mon[43975]: 1.1c scrub starts 2024-02-09T17:44:50.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:49 smithi179 ceph-mon[43975]: 1.1c scrub ok 2024-02-09T17:44:50.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:49 smithi179 ceph-mon[43975]: pgmap v1400: 47 pgs: 47 active+clean; 1.6 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 596 op/s 2024-02-09T17:44:50.175 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:49 smithi179 ceph-mon[43975]: 5.e scrub starts 2024-02-09T17:44:50.175 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:49 smithi179 ceph-mon[43975]: 5.e scrub ok 2024-02-09T17:44:50.181 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:50.207 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:44:50.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:50.210 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:44:50.311 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:44:50.347 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:50.562 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] 2024-02-09T17:44:50.562 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:44:50.562 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph pg dump --format=json 2024-02-09T17:44:50.610 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:44:50.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:50.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:44:50.719 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:44:50.791 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:50.816 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:44:50.821 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:44:50.821 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:44:50.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:50.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:44:50.927 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:44:50.973 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:51.004 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:44:51.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:51.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:44:51.113 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:44:51.156 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:51.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:50 smithi162 ceph-mon[44304]: 1.2 scrub starts 2024-02-09T17:44:51.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:50 smithi162 ceph-mon[44304]: 1.2 scrub ok 2024-02-09T17:44:51.185 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:44:51.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:51.192 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:44:51.193 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:50 smithi179 ceph-mon[43975]: 1.2 scrub starts 2024-02-09T17:44:51.194 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:50 smithi179 ceph-mon[43975]: 1.2 scrub ok 2024-02-09T17:44:51.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:50 smithi099 ceph-mon[39420]: 1.2 scrub starts 2024-02-09T17:44:51.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:50 smithi099 ceph-mon[39420]: 1.2 scrub ok 2024-02-09T17:44:51.294 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:44:51.330 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:51.362 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:44:51.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:51.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:44:51.466 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:44:51.504 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:51.535 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:44:51.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:51.539 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:44:51.640 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:44:51.694 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:51.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:51 smithi099 ceph-mon[39420]: 1.17 deep-scrub starts 2024-02-09T17:44:51.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:51 smithi099 ceph-mon[39420]: 1.17 deep-scrub ok 2024-02-09T17:44:51.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:51 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:44:51.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:51 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:44:51.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:51 smithi099 ceph-mon[39420]: pgmap v1401: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 611 op/s; 4536/73185 objects misplaced (6.198%) 2024-02-09T17:44:51.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:51 smithi099 ceph-mon[39420]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:44:51.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:51 smithi099 ceph-mon[39420]: osdmap e559: 12 total, 12 up, 12 in 2024-02-09T17:44:51.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:51 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1218863847' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:44:52.049 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-09T17:44:52.362 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:44:52.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:52.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:44:52.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:51 smithi179 ceph-mon[43975]: 1.17 deep-scrub starts 2024-02-09T17:44:52.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:51 smithi179 ceph-mon[43975]: 1.17 deep-scrub ok 2024-02-09T17:44:52.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:51 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:44:52.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:51 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:44:52.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:51 smithi179 ceph-mon[43975]: pgmap v1401: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 611 op/s; 4536/73185 objects misplaced (6.198%) 2024-02-09T17:44:52.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:51 smithi179 ceph-mon[43975]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:44:52.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:51 smithi179 ceph-mon[43975]: osdmap e559: 12 total, 12 up, 12 in 2024-02-09T17:44:52.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:51 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1218863847' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:44:52.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:51 smithi162 ceph-mon[44304]: 1.17 deep-scrub starts 2024-02-09T17:44:52.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:51 smithi162 ceph-mon[44304]: 1.17 deep-scrub ok 2024-02-09T17:44:52.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:51 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:44:52.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:51 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:44:52.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:51 smithi162 ceph-mon[44304]: pgmap v1401: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 611 op/s; 4536/73185 objects misplaced (6.198%) 2024-02-09T17:44:52.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:51 smithi162 ceph-mon[44304]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:44:52.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:51 smithi162 ceph-mon[44304]: osdmap e559: 12 total, 12 up, 12 in 2024-02-09T17:44:52.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:51 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1218863847' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:44:52.475 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:44:52.518 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:52.543 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:44:52.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:52.555 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:44:52.583 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:44:52.584 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:44:52.584 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:44:52.584 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:44:52.584 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:44:52.584 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:44:52.584 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:44:52.584 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:44:52.584 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:44:52.584 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:44:52.584 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:44:52.584 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:44:52.585 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:44:52.585 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:44:52.585 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:44:52.585 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:44:52.585 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:44:52.585 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:44:52.585 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:44:52.585 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:44:52.585 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:44:52.585 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:44:52.585 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:44:52.585 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:44:52.656 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:44:52.689 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:52.711 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:44:52.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:52.716 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:44:52.817 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:44:52.853 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:52.879 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:44:52.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:52.885 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:44:52.986 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:44:53.035 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:53.070 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:44:53.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:53.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:44:53.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:52 smithi179 ceph-mon[43975]: 1.11 deep-scrub starts 2024-02-09T17:44:53.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:52 smithi179 ceph-mon[43975]: 1.11 deep-scrub ok 2024-02-09T17:44:53.176 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:44:53.205 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:53.232 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:44:53.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:53.238 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:44:53.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:52 smithi099 ceph-mon[39420]: 1.11 deep-scrub starts 2024-02-09T17:44:53.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:52 smithi099 ceph-mon[39420]: 1.11 deep-scrub ok 2024-02-09T17:44:53.338 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:44:53.372 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:53.398 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:44:53.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:53.407 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:44:53.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:52 smithi162 ceph-mon[44304]: 1.11 deep-scrub starts 2024-02-09T17:44:53.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:52 smithi162 ceph-mon[44304]: 1.11 deep-scrub ok 2024-02-09T17:44:53.507 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:44:53.554 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:53.690 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:44:53.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:53.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:44:53.799 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:44:53.838 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:53.865 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:44:53.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:53.869 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:44:53.933 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:44:53.969 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:44:53.970 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:53 smithi099 ceph-mon[39420]: pgmap v1403: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 592 op/s; 4536/68817 objects misplaced (6.591%) 2024-02-09T17:44:53.974 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:44:53.998 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:54.024 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:44:54.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:54.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:44:54.133 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:44:54.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:53 smithi179 ceph-mon[43975]: pgmap v1403: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 592 op/s; 4536/68817 objects misplaced (6.591%) 2024-02-09T17:44:54.207 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:54.265 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:44:54.272 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:54.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:44:54.373 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:44:54.431 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:54.431 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:53 smithi162 ceph-mon[44304]: pgmap v1403: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 592 op/s; 4536/68817 objects misplaced (6.591%) 2024-02-09T17:44:54.474 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:44:54.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:54.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:44:54.578 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:44:54.642 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:54.669 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:44:54.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:54.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:44:54.774 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:44:54.835 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:54.994 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is set 2024-02-09T17:44:55.003 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:44:55.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:55.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:44:55.109 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:44:55.143 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:55.155 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:54 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2140346204' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:44:55.155 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:54 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:44:55.177 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:44:55.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:55.184 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:44:55.285 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:44:55.320 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:55.353 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:44:55.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:55.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:44:55.420 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:44:55.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:54 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2140346204' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:44:55.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:54 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:44:55.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:54 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2140346204' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:44:55.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:54 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:44:55.467 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:44:55.524 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:55.556 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:44:55.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:55.561 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:44:55.662 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:44:55.712 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:55.739 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:44:55.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:55.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:44:55.845 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:44:55.901 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:55.988 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:55 smithi162 ceph-mon[44304]: 5.5 scrub starts 2024-02-09T17:44:55.988 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:55 smithi162 ceph-mon[44304]: 5.5 scrub ok 2024-02-09T17:44:55.988 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:55 smithi162 ceph-mon[44304]: pgmap v1404: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 1.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 697 op/s; 3750/63096 objects misplaced (5.943%) 2024-02-09T17:44:55.988 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:55 smithi162 ceph-mon[44304]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:44:55.988 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:55 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:44:55.988 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:55 smithi162 ceph-mon[44304]: osdmap e560: 12 total, 12 up, 12 in 2024-02-09T17:44:56.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:55 smithi099 ceph-mon[39420]: 5.5 scrub starts 2024-02-09T17:44:56.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:55 smithi099 ceph-mon[39420]: 5.5 scrub ok 2024-02-09T17:44:56.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:55 smithi099 ceph-mon[39420]: pgmap v1404: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 1.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 697 op/s; 3750/63096 objects misplaced (5.943%) 2024-02-09T17:44:56.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:55 smithi099 ceph-mon[39420]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:44:56.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:55 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:44:56.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:55 smithi099 ceph-mon[39420]: osdmap e560: 12 total, 12 up, 12 in 2024-02-09T17:44:56.298 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:44:56.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:56.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:44:56.405 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:44:56.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:55 smithi179 ceph-mon[43975]: 5.5 scrub starts 2024-02-09T17:44:56.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:55 smithi179 ceph-mon[43975]: 5.5 scrub ok 2024-02-09T17:44:56.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:55 smithi179 ceph-mon[43975]: pgmap v1404: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 1.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 697 op/s; 3750/63096 objects misplaced (5.943%) 2024-02-09T17:44:56.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:55 smithi179 ceph-mon[43975]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:44:56.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:55 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:44:56.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:55 smithi179 ceph-mon[43975]: osdmap e560: 12 total, 12 up, 12 in 2024-02-09T17:44:56.438 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:56.467 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:44:56.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:56.473 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:44:56.574 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:44:56.633 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:56.661 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:44:56.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:56.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:44:56.770 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:44:56.832 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:56.861 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:44:56.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:56.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:44:56.968 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:44:57.019 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:57.063 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:44:57.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:57.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:44:57.181 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:44:57.245 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:57.286 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:44:57.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:57.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:44:57.395 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:44:57.445 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:57.468 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:44:57.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:57.472 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:44:57.573 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:44:57.574 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:57 smithi162 ceph-mon[44304]: osdmap e561: 12 total, 12 up, 12 in 2024-02-09T17:44:57.574 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:57 smithi162 ceph-mon[44304]: pgmap v1407: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 1.2 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 865 op/s; 3750/57591 objects misplaced (6.511%) 2024-02-09T17:44:57.612 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:57.646 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:57 smithi099 ceph-mon[39420]: osdmap e561: 12 total, 12 up, 12 in 2024-02-09T17:44:57.646 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:57 smithi099 ceph-mon[39420]: pgmap v1407: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 1.2 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 865 op/s; 3750/57591 objects misplaced (6.511%) 2024-02-09T17:44:57.654 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:44:57.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:57.661 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:44:57.762 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:44:57.826 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:57.871 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:44:57.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:57.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:44:57.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:57 smithi179 ceph-mon[43975]: osdmap e561: 12 total, 12 up, 12 in 2024-02-09T17:44:57.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:57 smithi179 ceph-mon[43975]: pgmap v1407: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 1.2 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 865 op/s; 3750/57591 objects misplaced (6.511%) 2024-02-09T17:44:57.978 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:44:58.007 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:58.039 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:44:58.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:58.045 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:44:58.146 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:44:58.180 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:58.210 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:44:58.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:58.218 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:44:58.282 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:44:58.293 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:44:58.303 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-02-09T17:44:58.318 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:44:58.380 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:58.414 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:44:58.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:58.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:44:58.532 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:44:58.596 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:58.597 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:44:58.598 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [], 2024-02-09T17:44:58.598 INFO:teuthology.orchestra.run.smithi099.stdout: "complaint_time": 30, 2024-02-09T17:44:58.598 INFO:teuthology.orchestra.run.smithi099.stdout: "num_blocked_ops": 0 2024-02-09T17:44:58.598 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:44:58.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:58 smithi162 ceph-mon[44304]: from='client.25789 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:44:58.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:58 smithi179 ceph-mon[43975]: from='client.25789 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:44:58.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:58 smithi099 ceph-mon[39420]: from='client.25789 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:44:59.055 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:44:59.256 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:59.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:44:59.357 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:44:59.401 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-09T17:44:59.420 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:59.627 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:44:59.633 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-09T17:44:59.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:59.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:44:59.727 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:44:59 smithi099 ceph-mon[39420]: pgmap v1408: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 1.1 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 763 op/s; 3750/54387 objects misplaced (6.895%) 2024-02-09T17:44:59.734 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:44:59.791 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:44:59.821 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:44:59.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:44:59.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:44:59.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:44:59 smithi179 ceph-mon[43975]: pgmap v1408: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 1.1 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 763 op/s; 3750/54387 objects misplaced (6.895%) 2024-02-09T17:44:59.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:44:59 smithi162 ceph-mon[44304]: pgmap v1408: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 1.1 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 763 op/s; 3750/54387 objects misplaced (6.895%) 2024-02-09T17:44:59.931 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:44:59.997 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:00.607 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":1407,"stamp":"2024-02-09T17:44:56.862780+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1258488376,"num_objects":19197,"num_object_clones":0,"num_object_copies":57591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3750,"num_objects_unfound":0,"num_objects_dirty":19197,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":238613,"num_write_kb":8249793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15203,"num_bytes_recovered":996343808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":164984,"ondisk_log_size":164984,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":145,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":9444764,"kb_used_data":3921956,"kb_used_omap":229,"kb_used_meta":5521946,"kb_avail":1115497060,"statfs":{"total":1151940427776,"available":1142268989440,"internally_reserved":0,"allocated":4016082944,"data_stored":4010367620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235127,"internal_metadata":5654473097},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,9,5,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-340656128,"num_objects":-5198,"num_object_clones":0,"num_object_copies":-15594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-786,"num_objects_unfound":0,"num_objects_dirty":-5198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5198,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002873"},"pg_stats":[{"pgid":"1.1e","version":"63'139","reported_seq":350,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-02-09T17:44:54.989714+0000","last_change":"2024-02-09T17:43:35.170863+0000","last_active":"2024-02-09T17:44:54.989714+0000","last_peered":"2024-02-09T17:44:54.989714+0000","last_clean":"2024-02-09T17:44:54.989714+0000","last_became_active":"2024-02-09T17:43:19.126757+0000","last_became_peered":"2024-02-09T17:43:19.126757+0000","last_unstale":"2024-02-09T17:44:54.989714+0000","last_undegraded":"2024-02-09T17:44:54.989714+0000","last_fullsized":"2024-02-09T17:44:54.989714+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:35.170716+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:43:35.170716+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:04.300669+0000","scrub_duration":73,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"63'139","reported_seq":373,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-02-09T17:44:50.947231+0000","last_change":"2024-02-09T17:43:35.227956+0000","last_active":"2024-02-09T17:44:50.947231+0000","last_peered":"2024-02-09T17:44:50.947231+0000","last_clean":"2024-02-09T17:44:50.947231+0000","last_became_active":"2024-02-09T17:43:28.594296+0000","last_became_peered":"2024-02-09T17:43:28.594296+0000","last_unstale":"2024-02-09T17:44:50.947231+0000","last_undegraded":"2024-02-09T17:44:50.947231+0000","last_fullsized":"2024-02-09T17:44:50.947231+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:35.227843+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:43:35.227843+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:52.776919+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"63'139","reported_seq":894,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-02-09T17:44:50.944673+0000","last_change":"2024-02-09T17:44:48.093391+0000","last_active":"2024-02-09T17:44:50.944673+0000","last_peered":"2024-02-09T17:44:50.944673+0000","last_clean":"2024-02-09T17:44:50.944673+0000","last_became_active":"2024-02-09T17:43:19.103584+0000","last_became_peered":"2024-02-09T17:43:19.103584+0000","last_unstale":"2024-02-09T17:44:50.944673+0000","last_undegraded":"2024-02-09T17:44:50.944673+0000","last_fullsized":"2024-02-09T17:44:50.944673+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:48.093301+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:23.912504+0000","last_clean_scrub_stamp":"2024-02-09T17:44:48.093301+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:59.190873+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"63'139","reported_seq":692,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-02-09T17:44:50.944609+0000","last_change":"2024-02-09T17:44:44.218695+0000","last_active":"2024-02-09T17:44:50.944609+0000","last_peered":"2024-02-09T17:44:50.944609+0000","last_clean":"2024-02-09T17:44:50.944609+0000","last_became_active":"2024-02-09T17:43:19.102729+0000","last_became_peered":"2024-02-09T17:43:19.102729+0000","last_unstale":"2024-02-09T17:44:50.944609+0000","last_undegraded":"2024-02-09T17:44:50.944609+0000","last_fullsized":"2024-02-09T17:44:50.944609+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:44.218550+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:44:44.218550+0000","last_clean_scrub_stamp":"2024-02-09T17:44:44.218550+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:44.873163+0000","scrub_duration":97,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"63'139","reported_seq":355,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-02-09T17:44:54.983217+0000","last_change":"2024-02-09T17:43:35.019518+0000","last_active":"2024-02-09T17:44:54.983217+0000","last_peered":"2024-02-09T17:44:54.983217+0000","last_clean":"2024-02-09T17:44:54.983217+0000","last_became_active":"2024-02-09T17:43:19.120814+0000","last_became_peered":"2024-02-09T17:43:19.120814+0000","last_unstale":"2024-02-09T17:44:54.983217+0000","last_undegraded":"2024-02-09T17:44:54.983217+0000","last_fullsized":"2024-02-09T17:44:54.983217+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:35.019416+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:43:35.019416+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:04.369513+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"63'139","reported_seq":1182,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-02-09T17:44:50.931906+0000","last_change":"2024-02-09T17:43:53.009587+0000","last_active":"2024-02-09T17:44:50.931906+0000","last_peered":"2024-02-09T17:44:50.931906+0000","last_clean":"2024-02-09T17:44:50.931906+0000","last_became_active":"2024-02-09T17:43:19.110535+0000","last_became_peered":"2024-02-09T17:43:19.110535+0000","last_unstale":"2024-02-09T17:44:50.931906+0000","last_undegraded":"2024-02-09T17:44:50.931906+0000","last_fullsized":"2024-02-09T17:44:50.931906+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:53.009455+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:30.193194+0000","last_clean_scrub_stamp":"2024-02-09T17:43:53.009455+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:21.095174+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"63'139","reported_seq":1020,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-02-09T17:44:50.945489+0000","last_change":"2024-02-09T17:43:57.370069+0000","last_active":"2024-02-09T17:44:50.945489+0000","last_peered":"2024-02-09T17:44:50.945489+0000","last_clean":"2024-02-09T17:44:50.945489+0000","last_became_active":"2024-02-09T17:43:28.585573+0000","last_became_peered":"2024-02-09T17:43:28.585573+0000","last_unstale":"2024-02-09T17:44:50.945489+0000","last_undegraded":"2024-02-09T17:44:50.945489+0000","last_fullsized":"2024-02-09T17:44:50.945489+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:57.369961+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:43:57.369961+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:03.250398+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"63'139","reported_seq":466,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-02-09T17:44:50.944914+0000","last_change":"2024-02-09T17:44:50.214834+0000","last_active":"2024-02-09T17:44:50.944914+0000","last_peered":"2024-02-09T17:44:50.944914+0000","last_clean":"2024-02-09T17:44:50.944914+0000","last_became_active":"2024-02-09T17:43:19.109864+0000","last_became_peered":"2024-02-09T17:43:19.109864+0000","last_unstale":"2024-02-09T17:44:50.944914+0000","last_undegraded":"2024-02-09T17:44:50.944914+0000","last_fullsized":"2024-02-09T17:44:50.944914+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:50.214686+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:44:50.214686+0000","last_clean_scrub_stamp":"2024-02-09T17:44:50.214686+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:46:04.216054+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"63'139","reported_seq":356,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-02-09T17:44:50.940793+0000","last_change":"2024-02-09T17:44:44.395036+0000","last_active":"2024-02-09T17:44:50.940793+0000","last_peered":"2024-02-09T17:44:50.940793+0000","last_clean":"2024-02-09T17:44:50.940793+0000","last_became_active":"2024-02-09T17:43:19.094230+0000","last_became_peered":"2024-02-09T17:43:19.094230+0000","last_unstale":"2024-02-09T17:44:50.940793+0000","last_undegraded":"2024-02-09T17:44:50.940793+0000","last_fullsized":"2024-02-09T17:44:50.940793+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:44.394925+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:11.970338+0000","last_clean_scrub_stamp":"2024-02-09T17:44:44.394925+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:46:09.318945+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"63'139","reported_seq":810,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-02-09T17:44:50.952250+0000","last_change":"2024-02-09T17:44:37.706632+0000","last_active":"2024-02-09T17:44:50.952250+0000","last_peered":"2024-02-09T17:44:50.952250+0000","last_clean":"2024-02-09T17:44:50.952250+0000","last_became_active":"2024-02-09T17:43:19.138398+0000","last_became_peered":"2024-02-09T17:43:19.138398+0000","last_unstale":"2024-02-09T17:44:50.952250+0000","last_undegraded":"2024-02-09T17:44:50.952250+0000","last_fullsized":"2024-02-09T17:44:50.952250+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:37.706524+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:44:37.706524+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:58.269158+0000","scrub_duration":186,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"63'139","reported_seq":852,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-02-09T17:44:54.997911+0000","last_change":"2024-02-09T17:44:25.672184+0000","last_active":"2024-02-09T17:44:54.997911+0000","last_peered":"2024-02-09T17:44:54.997911+0000","last_clean":"2024-02-09T17:44:54.997911+0000","last_became_active":"2024-02-09T17:43:28.589307+0000","last_became_peered":"2024-02-09T17:43:28.589307+0000","last_unstale":"2024-02-09T17:44:54.997911+0000","last_undegraded":"2024-02-09T17:44:54.997911+0000","last_fullsized":"2024-02-09T17:44:54.997911+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:25.672084+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:38:26.968892+0000","last_clean_scrub_stamp":"2024-02-09T17:44:25.672084+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:44.241932+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.4","version":"559'17019","reported_seq":26965,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-02-09T17:44:53.097511+0000","last_change":"2024-02-09T17:44:28.850586+0000","last_active":"2024-02-09T17:44:53.097511+0000","last_peered":"2024-02-09T17:44:53.097511+0000","last_clean":"2024-02-09T17:44:53.097511+0000","last_became_active":"2024-02-09T17:43:52.557816+0000","last_became_peered":"2024-02-09T17:43:52.557816+0000","last_unstale":"2024-02-09T17:44:53.097511+0000","last_undegraded":"2024-02-09T17:44:53.097511+0000","last_fullsized":"2024-02-09T17:44:53.097511+0000","mapping_epoch":535,"log_start":"499'7000","ondisk_log_start":"499'7000","created":465,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"547'15824","last_scrub_stamp":"2024-02-09T17:44:28.850416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:44:28.850416+0000","objects_scrubbed":2418,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:49.781140+0000","scrub_duration":552,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79429632,"num_objects":1212,"num_object_clones":0,"num_object_copies":3636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14618,"num_write_kb":506560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4756,"num_bytes_recovered":311689216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,5],"acting":[10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"63'139","reported_seq":1450,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-02-09T17:44:50.951045+0000","last_change":"2024-02-09T17:44:27.625779+0000","last_active":"2024-02-09T17:44:50.951045+0000","last_peered":"2024-02-09T17:44:50.951045+0000","last_clean":"2024-02-09T17:44:50.951045+0000","last_became_active":"2024-02-09T17:41:14.521079+0000","last_became_peered":"2024-02-09T17:41:14.521079+0000","last_unstale":"2024-02-09T17:44:50.951045+0000","last_undegraded":"2024-02-09T17:44:50.951045+0000","last_fullsized":"2024-02-09T17:44:50.951045+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:27.625665+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:44:27.625665+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:44.041378+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"560'14825","reported_seq":24325,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-02-09T17:44:55.446722+0000","last_change":"2024-02-09T17:43:57.644228+0000","last_active":"2024-02-09T17:44:55.446722+0000","last_peered":"2024-02-09T17:44:55.446722+0000","last_clean":"2024-02-09T17:44:55.446722+0000","last_became_active":"2024-02-09T17:43:56.598131+0000","last_became_peered":"2024-02-09T17:43:56.598131+0000","last_unstale":"2024-02-09T17:44:55.446722+0000","last_undegraded":"2024-02-09T17:44:55.446722+0000","last_fullsized":"2024-02-09T17:44:55.446722+0000","mapping_epoch":537,"log_start":"490'4800","ondisk_log_start":"490'4800","created":465,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'11981","last_scrub_stamp":"2024-02-09T17:43:57.644124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:43:57.644124+0000","objects_scrubbed":3961,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:19.331266+0000","scrub_duration":966,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71761920,"num_objects":1095,"num_object_clones":0,"num_object_copies":3285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14825,"num_write_kb":509440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4575,"num_bytes_recovered":299827200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"63'139","reported_seq":1017,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-02-09T17:44:50.945928+0000","last_change":"2024-02-09T17:44:49.486253+0000","last_active":"2024-02-09T17:44:50.945928+0000","last_peered":"2024-02-09T17:44:50.945928+0000","last_clean":"2024-02-09T17:44:50.945928+0000","last_became_active":"2024-02-09T17:43:39.981054+0000","last_became_peered":"2024-02-09T17:43:39.981054+0000","last_unstale":"2024-02-09T17:44:50.945928+0000","last_undegraded":"2024-02-09T17:44:50.945928+0000","last_fullsized":"2024-02-09T17:44:50.945928+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:49.486158+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:44:49.486158+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:46:18.726701+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"559'14758","reported_seq":15076,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-02-09T17:44:54.500790+0000","last_change":"2024-02-09T17:44:31.130917+0000","last_active":"2024-02-09T17:44:54.500790+0000","last_peered":"2024-02-09T17:44:54.500790+0000","last_clean":"2024-02-09T17:44:54.500790+0000","last_became_active":"2024-02-09T17:40:48.781472+0000","last_became_peered":"2024-02-09T17:40:48.781472+0000","last_unstale":"2024-02-09T17:44:54.500790+0000","last_undegraded":"2024-02-09T17:44:54.500790+0000","last_fullsized":"2024-02-09T17:44:54.500790+0000","mapping_epoch":465,"log_start":"490'4700","ondisk_log_start":"490'4700","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"548'13663","last_scrub_stamp":"2024-02-09T17:44:31.130800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:44:31.130800+0000","objects_scrubbed":2301,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:42.320512+0000","scrub_duration":699,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77987840,"num_objects":1190,"num_object_clones":0,"num_object_copies":3570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14758,"num_write_kb":510336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"63'139","reported_seq":632,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-02-09T17:44:50.940521+0000","last_change":"2024-02-09T17:43:53.218894+0000","last_active":"2024-02-09T17:44:50.940521+0000","last_peered":"2024-02-09T17:44:50.940521+0000","last_clean":"2024-02-09T17:44:50.940521+0000","last_became_active":"2024-02-09T17:40:13.501711+0000","last_became_peered":"2024-02-09T17:40:13.501711+0000","last_unstale":"2024-02-09T17:44:50.940521+0000","last_undegraded":"2024-02-09T17:44:50.940521+0000","last_fullsized":"2024-02-09T17:44:50.940521+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:53.218764+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:38:27.045816+0000","last_clean_scrub_stamp":"2024-02-09T17:43:53.218764+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:20.464157+0000","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"559'14740","reported_seq":15079,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-02-09T17:44:53.956703+0000","last_change":"2024-02-09T17:43:37.166310+0000","last_active":"2024-02-09T17:44:53.956703+0000","last_peered":"2024-02-09T17:44:53.956703+0000","last_clean":"2024-02-09T17:44:53.956703+0000","last_became_active":"2024-02-09T17:40:48.780579+0000","last_became_peered":"2024-02-09T17:40:48.780579+0000","last_unstale":"2024-02-09T17:44:53.956703+0000","last_undegraded":"2024-02-09T17:44:53.956703+0000","last_fullsized":"2024-02-09T17:44:53.956703+0000","mapping_epoch":465,"log_start":"489'4700","ondisk_log_start":"489'4700","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"529'11051","last_scrub_stamp":"2024-02-09T17:43:37.166082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:43:37.166082+0000","objects_scrubbed":4984,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1034,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83492864,"num_objects":1274,"num_object_clones":0,"num_object_copies":3822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14740,"num_write_kb":512448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"63'139","reported_seq":948,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-02-09T17:44:50.932108+0000","last_change":"2024-02-09T17:43:54.988847+0000","last_active":"2024-02-09T17:44:50.932108+0000","last_peered":"2024-02-09T17:44:50.932108+0000","last_clean":"2024-02-09T17:44:50.932108+0000","last_became_active":"2024-02-09T17:34:01.342607+0000","last_became_peered":"2024-02-09T17:34:01.342607+0000","last_unstale":"2024-02-09T17:44:50.932108+0000","last_undegraded":"2024-02-09T17:44:50.932108+0000","last_fullsized":"2024-02-09T17:44:50.932108+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:54.988700+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:43:54.988700+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:58.197783+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.2","version":"560'15215","reported_seq":15562,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-02-09T17:44:56.010771+0000","last_change":"2024-02-09T17:43:46.878627+0000","last_active":"2024-02-09T17:44:56.010771+0000","last_peered":"2024-02-09T17:44:56.010771+0000","last_clean":"2024-02-09T17:44:56.010771+0000","last_became_active":"2024-02-09T17:40:48.781532+0000","last_became_peered":"2024-02-09T17:40:48.781532+0000","last_unstale":"2024-02-09T17:44:56.010771+0000","last_undegraded":"2024-02-09T17:44:56.010771+0000","last_fullsized":"2024-02-09T17:44:56.010771+0000","mapping_epoch":465,"log_start":"492'5200","ondisk_log_start":"492'5200","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"533'11868","last_scrub_stamp":"2024-02-09T17:43:46.878486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:43:46.878486+0000","objects_scrubbed":4573,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:59.991875+0000","scrub_duration":1380,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77660160,"num_objects":1185,"num_object_clones":0,"num_object_copies":3555,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15215,"num_write_kb":524800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"63'139","reported_seq":960,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-02-09T17:44:50.931960+0000","last_change":"2024-02-09T17:44:40.373866+0000","last_active":"2024-02-09T17:44:50.931960+0000","last_peered":"2024-02-09T17:44:50.931960+0000","last_clean":"2024-02-09T17:44:50.931960+0000","last_became_active":"2024-02-09T17:43:14.024894+0000","last_became_peered":"2024-02-09T17:43:14.024894+0000","last_unstale":"2024-02-09T17:44:50.931960+0000","last_undegraded":"2024-02-09T17:44:50.931960+0000","last_fullsized":"2024-02-09T17:44:50.931960+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:40.373768+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:11.970338+0000","last_clean_scrub_stamp":"2024-02-09T17:44:40.373768+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:53.854883+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.3","version":"559'16249","reported_seq":16644,"reported_epoch":559,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-09T17:44:53.948295+0000","last_change":"2024-02-09T17:44:47.035729+0000","last_active":"2024-02-09T17:44:53.948295+0000","last_peered":"2024-02-09T17:44:53.948295+0000","last_clean":"2024-02-09T17:44:44.589692+0000","last_became_active":"2024-02-09T17:44:46.971273+0000","last_became_peered":"2024-02-09T17:44:46.971273+0000","last_unstale":"2024-02-09T17:44:53.948295+0000","last_undegraded":"2024-02-09T17:44:53.948295+0000","last_fullsized":"2024-02-09T17:44:53.948295+0000","mapping_epoch":557,"log_start":"498'6200","ondisk_log_start":"498'6200","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"529'12566","last_scrub_stamp":"2024-02-09T17:43:36.701823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:43:36.701823+0000","objects_scrubbed":4960,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1352,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81920000,"num_objects":1250,"num_object_clones":0,"num_object_copies":3750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3750,"num_objects_unfound":0,"num_objects_dirty":1250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14688,"num_write_kb":510016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[0,5,6],"avail_no_missing":["0","5","6"],"object_location_counts":[{"shards":"0,5,6","objects":1250}],"blocked_by":[],"up_primary":4,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"63'139","reported_seq":626,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-02-09T17:44:55.000534+0000","last_change":"2024-02-09T17:44:37.555304+0000","last_active":"2024-02-09T17:44:55.000534+0000","last_peered":"2024-02-09T17:44:55.000534+0000","last_clean":"2024-02-09T17:44:55.000534+0000","last_became_active":"2024-02-09T17:43:14.038885+0000","last_became_peered":"2024-02-09T17:43:14.038885+0000","last_unstale":"2024-02-09T17:44:55.000534+0000","last_undegraded":"2024-02-09T17:44:55.000534+0000","last_fullsized":"2024-02-09T17:44:55.000534+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:37.555166+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:44:37.555166+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:37.616972+0000","scrub_duration":118,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"559'15217","reported_seq":15556,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-02-09T17:44:53.109784+0000","last_change":"2024-02-09T17:43:52.312478+0000","last_active":"2024-02-09T17:44:53.109784+0000","last_peered":"2024-02-09T17:44:53.109784+0000","last_clean":"2024-02-09T17:44:53.109784+0000","last_became_active":"2024-02-09T17:40:48.780290+0000","last_became_peered":"2024-02-09T17:40:48.780290+0000","last_unstale":"2024-02-09T17:44:53.109784+0000","last_undegraded":"2024-02-09T17:44:53.109784+0000","last_fullsized":"2024-02-09T17:44:53.109784+0000","mapping_epoch":465,"log_start":"492'5200","ondisk_log_start":"492'5200","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"535'12174","last_scrub_stamp":"2024-02-09T17:43:52.312370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:43:52.312370+0000","objects_scrubbed":4242,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:55.425323+0000","scrub_duration":401,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78577664,"num_objects":1199,"num_object_clones":0,"num_object_copies":3597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15217,"num_write_kb":525312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"63'139","reported_seq":1032,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-02-09T17:44:54.989403+0000","last_change":"2024-02-09T17:43:56.105889+0000","last_active":"2024-02-09T17:44:54.989403+0000","last_peered":"2024-02-09T17:44:54.989403+0000","last_clean":"2024-02-09T17:44:54.989403+0000","last_became_active":"2024-02-09T17:33:00.284924+0000","last_became_peered":"2024-02-09T17:33:00.284924+0000","last_unstale":"2024-02-09T17:44:54.989403+0000","last_undegraded":"2024-02-09T17:44:54.989403+0000","last_fullsized":"2024-02-09T17:44:54.989403+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:56.105792+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:56.105792+0000","last_clean_scrub_stamp":"2024-02-09T17:43:56.105792+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:56.755038+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.1","version":"560'14779","reported_seq":15121,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-02-09T17:44:55.861365+0000","last_change":"2024-02-09T17:43:44.138178+0000","last_active":"2024-02-09T17:44:55.861365+0000","last_peered":"2024-02-09T17:44:55.861365+0000","last_clean":"2024-02-09T17:44:55.861365+0000","last_became_active":"2024-02-09T17:40:49.660617+0000","last_became_peered":"2024-02-09T17:40:49.660617+0000","last_unstale":"2024-02-09T17:44:55.861365+0000","last_undegraded":"2024-02-09T17:44:55.861365+0000","last_fullsized":"2024-02-09T17:44:55.861365+0000","mapping_epoch":465,"log_start":"490'4700","ondisk_log_start":"490'4700","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"533'11388","last_scrub_stamp":"2024-02-09T17:43:44.138002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:43:44.138002+0000","objects_scrubbed":4543,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:01.112942+0000","scrub_duration":1020,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73203712,"num_objects":1117,"num_object_clones":0,"num_object_copies":3351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14779,"num_write_kb":508672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"63'139","reported_seq":1005,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-02-09T17:44:50.932042+0000","last_change":"2024-02-09T17:44:41.377536+0000","last_active":"2024-02-09T17:44:50.932042+0000","last_peered":"2024-02-09T17:44:50.932042+0000","last_clean":"2024-02-09T17:44:50.932042+0000","last_became_active":"2024-02-09T17:43:28.597612+0000","last_became_peered":"2024-02-09T17:43:28.597612+0000","last_unstale":"2024-02-09T17:44:50.932042+0000","last_undegraded":"2024-02-09T17:44:50.932042+0000","last_fullsized":"2024-02-09T17:44:50.932042+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:41.377454+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:44:41.377454+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:46:09.607508+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.e","version":"561'15108","reported_seq":15454,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-02-09T17:44:56.585988+0000","last_change":"2024-02-09T17:44:49.194328+0000","last_active":"2024-02-09T17:44:56.585988+0000","last_peered":"2024-02-09T17:44:56.585988+0000","last_clean":"2024-02-09T17:44:56.585988+0000","last_became_active":"2024-02-09T17:40:48.779366+0000","last_became_peered":"2024-02-09T17:40:48.779366+0000","last_unstale":"2024-02-09T17:44:56.585988+0000","last_undegraded":"2024-02-09T17:44:56.585988+0000","last_fullsized":"2024-02-09T17:44:56.585988+0000","mapping_epoch":465,"log_start":"492'5100","ondisk_log_start":"492'5100","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"558'14691","last_scrub_stamp":"2024-02-09T17:44:49.194186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:44:49.194186+0000","objects_scrubbed":1492,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:49.458648+0000","scrub_duration":317,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69861376,"num_objects":1066,"num_object_clones":0,"num_object_copies":3198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15108,"num_write_kb":517568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"63'139","reported_seq":957,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-02-09T17:44:56.579556+0000","last_change":"2024-02-09T17:44:27.890584+0000","last_active":"2024-02-09T17:44:56.579556+0000","last_peered":"2024-02-09T17:44:56.579556+0000","last_clean":"2024-02-09T17:44:56.579556+0000","last_became_active":"2024-02-09T17:43:14.041291+0000","last_became_peered":"2024-02-09T17:43:14.041291+0000","last_unstale":"2024-02-09T17:44:56.579556+0000","last_undegraded":"2024-02-09T17:44:56.579556+0000","last_fullsized":"2024-02-09T17:44:56.579556+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:27.890510+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:44:27.890510+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:53.802187+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.f","version":"559'14936","reported_seq":17645,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-02-09T17:44:54.479801+0000","last_change":"2024-02-09T17:43:56.859129+0000","last_active":"2024-02-09T17:44:54.479801+0000","last_peered":"2024-02-09T17:44:54.479801+0000","last_clean":"2024-02-09T17:44:54.479801+0000","last_became_active":"2024-02-09T17:41:20.876180+0000","last_became_peered":"2024-02-09T17:41:20.876180+0000","last_unstale":"2024-02-09T17:44:54.479801+0000","last_undegraded":"2024-02-09T17:44:54.479801+0000","last_fullsized":"2024-02-09T17:44:54.479801+0000","mapping_epoch":480,"log_start":"491'4900","ondisk_log_start":"491'4900","created":465,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'12028","last_scrub_stamp":"2024-02-09T17:43:56.858923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:43:56.858923+0000","objects_scrubbed":4068,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:25.138111+0000","scrub_duration":562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75759616,"num_objects":1156,"num_object_clones":0,"num_object_copies":3468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14936,"num_write_kb":514944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1182,"num_bytes_recovered":77463552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,11],"acting":[5,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"63'139","reported_seq":797,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-02-09T17:44:54.989268+0000","last_change":"2024-02-09T17:44:25.176171+0000","last_active":"2024-02-09T17:44:54.989268+0000","last_peered":"2024-02-09T17:44:54.989268+0000","last_clean":"2024-02-09T17:44:54.989268+0000","last_became_active":"2024-02-09T17:43:14.054943+0000","last_became_peered":"2024-02-09T17:43:14.054943+0000","last_unstale":"2024-02-09T17:44:54.989268+0000","last_undegraded":"2024-02-09T17:44:54.989268+0000","last_fullsized":"2024-02-09T17:44:54.989268+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:25.176022+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:44:25.176022+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:31.031846+0000","scrub_duration":158,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"559'14950","reported_seq":15274,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-02-09T17:44:54.772761+0000","last_change":"2024-02-09T17:43:39.114851+0000","last_active":"2024-02-09T17:44:54.772761+0000","last_peered":"2024-02-09T17:44:54.772761+0000","last_clean":"2024-02-09T17:44:54.772761+0000","last_became_active":"2024-02-09T17:40:48.781384+0000","last_became_peered":"2024-02-09T17:40:48.781384+0000","last_unstale":"2024-02-09T17:44:54.772761+0000","last_undegraded":"2024-02-09T17:44:54.772761+0000","last_fullsized":"2024-02-09T17:44:54.772761+0000","mapping_epoch":465,"log_start":"490'4900","ondisk_log_start":"490'4900","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"531'11356","last_scrub_stamp":"2024-02-09T17:43:39.114709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:43:39.114709+0000","objects_scrubbed":4812,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:06.435989+0000","scrub_duration":1039,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78643200,"num_objects":1200,"num_object_clones":0,"num_object_copies":3600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14950,"num_write_kb":516800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"63'139","reported_seq":1011,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-02-09T17:44:50.945581+0000","last_change":"2024-02-09T17:44:34.388682+0000","last_active":"2024-02-09T17:44:50.945581+0000","last_peered":"2024-02-09T17:44:50.945581+0000","last_clean":"2024-02-09T17:44:50.945581+0000","last_became_active":"2024-02-09T17:43:28.586349+0000","last_became_peered":"2024-02-09T17:43:28.586349+0000","last_unstale":"2024-02-09T17:44:50.945581+0000","last_undegraded":"2024-02-09T17:44:50.945581+0000","last_fullsized":"2024-02-09T17:44:50.945581+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:34.388557+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:44:34.388557+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:46:01.046642+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"560'15261","reported_seq":18134,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-02-09T17:44:55.440117+0000","last_change":"2024-02-09T17:44:29.650288+0000","last_active":"2024-02-09T17:44:55.440117+0000","last_peered":"2024-02-09T17:44:55.440117+0000","last_clean":"2024-02-09T17:44:55.440117+0000","last_became_active":"2024-02-09T17:41:21.499187+0000","last_became_peered":"2024-02-09T17:41:21.499187+0000","last_unstale":"2024-02-09T17:44:55.440117+0000","last_undegraded":"2024-02-09T17:44:55.440117+0000","last_fullsized":"2024-02-09T17:44:55.440117+0000","mapping_epoch":481,"log_start":"492'5200","ondisk_log_start":"492'5200","created":465,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"547'14005","last_scrub_stamp":"2024-02-09T17:44:29.650159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:44:29.650159+0000","objects_scrubbed":2458,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:55.225580+0000","scrub_duration":643,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77463576,"num_objects":1183,"num_object_clones":0,"num_object_copies":3549,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1183,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15261,"num_write_kb":526145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1248,"num_bytes_recovered":81788928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"63'139","reported_seq":1457,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-02-09T17:44:50.951158+0000","last_change":"2024-02-09T17:44:45.563490+0000","last_active":"2024-02-09T17:44:50.951158+0000","last_peered":"2024-02-09T17:44:50.951158+0000","last_clean":"2024-02-09T17:44:50.951158+0000","last_became_active":"2024-02-09T17:43:28.590237+0000","last_became_peered":"2024-02-09T17:43:28.590237+0000","last_unstale":"2024-02-09T17:44:50.951158+0000","last_undegraded":"2024-02-09T17:44:50.951158+0000","last_fullsized":"2024-02-09T17:44:50.951158+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:45.563392+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:15.675690+0000","last_clean_scrub_stamp":"2024-02-09T17:44:45.563392+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:46.329929+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"559'14553","reported_seq":19383,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-02-09T17:44:51.963298+0000","last_change":"2024-02-09T17:44:43.463805+0000","last_active":"2024-02-09T17:44:51.963298+0000","last_peered":"2024-02-09T17:44:51.963298+0000","last_clean":"2024-02-09T17:44:51.963298+0000","last_became_active":"2024-02-09T17:44:39.593392+0000","last_became_peered":"2024-02-09T17:44:39.593392+0000","last_unstale":"2024-02-09T17:44:51.963298+0000","last_undegraded":"2024-02-09T17:44:51.963298+0000","last_fullsized":"2024-02-09T17:44:51.963298+0000","mapping_epoch":552,"log_start":"489'4500","ondisk_log_start":"489'4500","created":465,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'14225","last_scrub_stamp":"2024-02-09T17:44:43.463586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:44:43.463586+0000","objects_scrubbed":1740,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:46:05.557212+0000","scrub_duration":334,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92209152,"num_objects":1407,"num_object_clones":0,"num_object_copies":4221,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14553,"num_write_kb":510720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2247,"num_bytes_recovered":147259392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,10],"acting":[8,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.e","version":"63'139","reported_seq":924,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-02-09T17:44:54.983663+0000","last_change":"2024-02-09T17:44:25.525420+0000","last_active":"2024-02-09T17:44:54.983663+0000","last_peered":"2024-02-09T17:44:54.983663+0000","last_clean":"2024-02-09T17:44:54.983663+0000","last_became_active":"2024-02-09T17:43:14.036583+0000","last_became_peered":"2024-02-09T17:43:14.036583+0000","last_unstale":"2024-02-09T17:44:54.983663+0000","last_undegraded":"2024-02-09T17:44:54.983663+0000","last_fullsized":"2024-02-09T17:44:54.983663+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:25.525304+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:14.981453+0000","last_clean_scrub_stamp":"2024-02-09T17:44:25.525304+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:50.128868+0000","scrub_duration":97,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"561'14989","reported_seq":15336,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-02-09T17:44:56.635376+0000","last_change":"2024-02-09T17:43:42.009150+0000","last_active":"2024-02-09T17:44:56.635376+0000","last_peered":"2024-02-09T17:44:56.635376+0000","last_clean":"2024-02-09T17:44:56.635376+0000","last_became_active":"2024-02-09T17:40:48.783235+0000","last_became_peered":"2024-02-09T17:40:48.783235+0000","last_unstale":"2024-02-09T17:44:56.635376+0000","last_undegraded":"2024-02-09T17:44:56.635376+0000","last_fullsized":"2024-02-09T17:44:56.635376+0000","mapping_epoch":465,"log_start":"490'4900","ondisk_log_start":"490'4900","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"533'11389","last_scrub_stamp":"2024-02-09T17:43:42.009039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:43:42.009039+0000","objects_scrubbed":4727,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:05.982895+0000","scrub_duration":1348,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71892992,"num_objects":1097,"num_object_clones":0,"num_object_copies":3291,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14989,"num_write_kb":514752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.f","version":"63'139","reported_seq":977,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-02-09T17:44:55.000438+0000","last_change":"2024-02-09T17:44:42.643784+0000","last_active":"2024-02-09T17:44:55.000438+0000","last_peered":"2024-02-09T17:44:55.000438+0000","last_clean":"2024-02-09T17:44:55.000438+0000","last_became_active":"2024-02-09T17:33:05.316100+0000","last_became_peered":"2024-02-09T17:33:05.316100+0000","last_unstale":"2024-02-09T17:44:55.000438+0000","last_undegraded":"2024-02-09T17:44:55.000438+0000","last_fullsized":"2024-02-09T17:44:55.000438+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:42.643646+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:36:22.364130+0000","last_clean_scrub_stamp":"2024-02-09T17:44:42.643646+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:43.464413+0000","scrub_duration":145,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"560'15068","reported_seq":17810,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-02-09T17:44:55.451187+0000","last_change":"2024-02-09T17:43:54.002412+0000","last_active":"2024-02-09T17:44:55.451187+0000","last_peered":"2024-02-09T17:44:55.451187+0000","last_clean":"2024-02-09T17:44:55.451187+0000","last_became_active":"2024-02-09T17:41:20.876434+0000","last_became_peered":"2024-02-09T17:41:20.876434+0000","last_unstale":"2024-02-09T17:44:55.451187+0000","last_undegraded":"2024-02-09T17:44:55.451187+0000","last_fullsized":"2024-02-09T17:44:55.451187+0000","mapping_epoch":480,"log_start":"490'5000","ondisk_log_start":"490'5000","created":465,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"536'12114","last_scrub_stamp":"2024-02-09T17:43:54.002296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:43:54.002296+0000","objects_scrubbed":4157,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:44:55.972751+0000","scrub_duration":854,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77594624,"num_objects":1184,"num_object_clones":0,"num_object_copies":3552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15068,"num_write_kb":520064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1195,"num_bytes_recovered":78315520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"63'139","reported_seq":1102,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-02-09T17:44:51.019980+0000","last_change":"2024-02-09T17:44:28.298713+0000","last_active":"2024-02-09T17:44:51.019980+0000","last_peered":"2024-02-09T17:44:51.019980+0000","last_clean":"2024-02-09T17:44:51.019980+0000","last_became_active":"2024-02-09T17:43:14.027715+0000","last_became_peered":"2024-02-09T17:43:14.027715+0000","last_unstale":"2024-02-09T17:44:51.019980+0000","last_undegraded":"2024-02-09T17:44:51.019980+0000","last_fullsized":"2024-02-09T17:44:51.019980+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:28.298578+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:14.135321+0000","last_clean_scrub_stamp":"2024-02-09T17:44:28.298578+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:31.180345+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"559'14908","reported_seq":15275,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-02-09T17:44:52.313982+0000","last_change":"2024-02-09T17:44:25.663805+0000","last_active":"2024-02-09T17:44:52.313982+0000","last_peered":"2024-02-09T17:44:52.313982+0000","last_clean":"2024-02-09T17:44:52.313982+0000","last_became_active":"2024-02-09T17:40:49.436285+0000","last_became_peered":"2024-02-09T17:40:49.436285+0000","last_unstale":"2024-02-09T17:44:52.313982+0000","last_undegraded":"2024-02-09T17:44:52.313982+0000","last_fullsized":"2024-02-09T17:44:52.313982+0000","mapping_epoch":465,"log_start":"490'4900","ondisk_log_start":"490'4900","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"546'13640","last_scrub_stamp":"2024-02-09T17:44:25.663661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:44:25.663661+0000","objects_scrubbed":2677,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:53.939379+0000","scrub_duration":943,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90439680,"num_objects":1380,"num_object_clones":0,"num_object_copies":4140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14908,"num_write_kb":521216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"63'139","reported_seq":514,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-02-09T17:44:50.952400+0000","last_change":"2024-02-09T17:44:30.715902+0000","last_active":"2024-02-09T17:44:50.952400+0000","last_peered":"2024-02-09T17:44:50.952400+0000","last_clean":"2024-02-09T17:44:50.952400+0000","last_became_active":"2024-02-09T17:43:14.036759+0000","last_became_peered":"2024-02-09T17:43:14.036759+0000","last_unstale":"2024-02-09T17:44:50.952400+0000","last_undegraded":"2024-02-09T17:44:50.952400+0000","last_fullsized":"2024-02-09T17:44:50.952400+0000","mapping_epoch":454,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:30.715778+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:44:30.715778+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:57.590268+0000","scrub_duration":144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"63'139","reported_seq":917,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-02-09T17:44:50.945680+0000","last_change":"2024-02-09T17:44:38.302668+0000","last_active":"2024-02-09T17:44:50.945680+0000","last_peered":"2024-02-09T17:44:50.945680+0000","last_clean":"2024-02-09T17:44:50.945680+0000","last_became_active":"2024-02-09T17:33:05.310104+0000","last_became_peered":"2024-02-09T17:33:05.310104+0000","last_unstale":"2024-02-09T17:44:50.945680+0000","last_undegraded":"2024-02-09T17:44:50.945680+0000","last_fullsized":"2024-02-09T17:44:50.945680+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:38.302529+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:44:38.302529+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:57.708195+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"63'139","reported_seq":932,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-02-09T17:44:51.190858+0000","last_change":"2024-02-09T17:44:51.190882+0000","last_active":"2024-02-09T17:44:51.190858+0000","last_peered":"2024-02-09T17:44:51.190858+0000","last_clean":"2024-02-09T17:44:51.190858+0000","last_became_active":"2024-02-09T17:34:01.353171+0000","last_became_peered":"2024-02-09T17:34:01.353171+0000","last_unstale":"2024-02-09T17:44:51.190858+0000","last_undegraded":"2024-02-09T17:44:51.190858+0000","last_fullsized":"2024-02-09T17:44:51.190858+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:51.190789+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:44:51.190789+0000","last_clean_scrub_stamp":"2024-02-09T17:44:51.190789+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:46:12.329895+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.12","version":"63'139","reported_seq":1436,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-02-09T17:44:50.945619+0000","last_change":"2024-02-09T17:43:55.346549+0000","last_active":"2024-02-09T17:44:50.945619+0000","last_peered":"2024-02-09T17:44:50.945619+0000","last_clean":"2024-02-09T17:44:50.945619+0000","last_became_active":"2024-02-09T17:40:13.499071+0000","last_became_peered":"2024-02-09T17:40:13.499071+0000","last_unstale":"2024-02-09T17:44:50.945619+0000","last_undegraded":"2024-02-09T17:44:50.945619+0000","last_fullsized":"2024-02-09T17:44:50.945619+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:55.341130+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:43:55.341130+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:10.488649+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"63'139","reported_seq":635,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-02-09T17:44:50.940603+0000","last_change":"2024-02-09T17:44:27.381453+0000","last_active":"2024-02-09T17:44:50.940603+0000","last_peered":"2024-02-09T17:44:50.940603+0000","last_clean":"2024-02-09T17:44:50.940603+0000","last_became_active":"2024-02-09T17:41:14.521672+0000","last_became_peered":"2024-02-09T17:41:14.521672+0000","last_unstale":"2024-02-09T17:44:50.940603+0000","last_undegraded":"2024-02-09T17:44:50.940603+0000","last_fullsized":"2024-02-09T17:44:50.940603+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:27.381332+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:44:27.381332+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:35.775807+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":1257898008,"num_objects":19195,"num_object_clones":0,"num_object_copies":57585,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3750,"num_objects_unfound":0,"num_objects_dirty":19195,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":238613,"num_write_kb":8249793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15203,"num_bytes_recovered":996343808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3890884608,"data_stored":3890872392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160675,"ondisk_log_size":160675,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056911,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":867640,"kb_used_data":308908,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":92877512,"statfs":{"total":95995035648,"available":95106572288,"internally_reserved":0,"allocated":316321792,"data_stored":315840361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":572127684},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:44:33 2024","interfaces":[{"interface":"back","average":{"1min":2.0990000000000002,"5min":1.2390000000000001,"15min":1.288},"min":{"1min":0.221,"5min":0.221,"15min":0.216},"max":{"1min":5.1989999999999998,"5min":7.4089999999999998,"15min":12.836},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.90600000000000003,"5min":1.0740000000000001,"15min":1.173},"min":{"1min":0.25,"5min":0.25,"15min":0.23000000000000001},"max":{"1min":4.7729999999999997,"5min":9.8219999999999992,"15min":9.8219999999999992},"last":0.27300000000000002}]},{"osd":1,"last update":"Fri Feb 9 17:44:33 2024","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.2889999999999999,"15min":1.331},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.22700000000000001},"max":{"1min":4.2190000000000003,"5min":9.7669999999999995,"15min":12.629},"last":0.624},{"interface":"front","average":{"1min":1.9750000000000001,"5min":1.3560000000000001,"15min":1.4430000000000001},"min":{"1min":0.28599999999999998,"5min":0.26500000000000001,"15min":0.26400000000000001},"max":{"1min":5.0750000000000002,"5min":9.5009999999999994,"15min":12.667},"last":0.39500000000000002}]},{"osd":2,"last update":"Fri Feb 9 17:31:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.415,"15min":1.6859999999999999},"min":{"1min":0.378,"5min":0.247,"15min":0.247},"max":{"1min":4.4189999999999996,"5min":9.4719999999999995,"15min":9.4719999999999995},"last":0.35399999999999998},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.5149999999999999,"15min":1.4399999999999999},"min":{"1min":0.378,"5min":0.252,"15min":0.252},"max":{"1min":4.8579999999999997,"5min":7.96,"15min":7.96},"last":0.60399999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:44:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7110000000000001,"5min":1.1459999999999999,"15min":1.2490000000000001},"min":{"1min":0.29699999999999999,"5min":0.27100000000000002,"15min":0.247},"max":{"1min":4.2830000000000004,"5min":9.7289999999999992,"15min":9.7289999999999992},"last":0.48799999999999999},{"interface":"front","average":{"1min":1.7789999999999999,"5min":1.373,"15min":1.5149999999999999},"min":{"1min":0.38800000000000001,"5min":0.33000000000000002,"15min":0.26400000000000001},"max":{"1min":4.2990000000000004,"5min":9.7479999999999993,"15min":12.706},"last":0.63600000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:44:33 2024","interfaces":[{"interface":"back","average":{"1min":1.488,"5min":1.4630000000000001,"15min":1.681},"min":{"1min":0.30599999999999999,"5min":0.246,"15min":0.246},"max":{"1min":4.806,"5min":6.3959999999999999,"15min":21.030999999999999},"last":8.266},{"interface":"front","average":{"1min":2.8439999999999999,"5min":1.575,"15min":1.5569999999999999},"min":{"1min":0.40899999999999997,"5min":0.317,"15min":0.309},"max":{"1min":6.1289999999999996,"5min":7.5490000000000004,"15min":20.876999999999999},"last":8.2840000000000007}]},{"osd":5,"last update":"Fri Feb 9 17:44:33 2024","interfaces":[{"interface":"back","average":{"1min":2.9340000000000002,"5min":1.704,"15min":1.6140000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.312},"max":{"1min":9.0690000000000008,"5min":9.5960000000000001,"15min":13.946999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":2.1680000000000001,"5min":1.585,"15min":1.6839999999999999},"min":{"1min":0.311,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":9.6259999999999994,"5min":9.6259999999999994,"15min":13.414},"last":3.855}]},{"osd":6,"last update":"Fri Feb 9 17:44:33 2024","interfaces":[{"interface":"back","average":{"1min":2.7360000000000002,"5min":1.909,"15min":1.726},"min":{"1min":0.39200000000000002,"5min":0.38600000000000001,"15min":0.29899999999999999},"max":{"1min":9.1679999999999993,"5min":12.362,"15min":12.798999999999999},"last":1.2989999999999999},{"interface":"front","average":{"1min":3.8090000000000002,"5min":2.0609999999999999,"15min":1.8100000000000001},"min":{"1min":0.442,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":10.000999999999999,"5min":10.000999999999999,"15min":10.782999999999999},"last":4.0449999999999999}]},{"osd":7,"last update":"Fri Feb 9 17:44:33 2024","interfaces":[{"interface":"back","average":{"1min":2.798,"5min":1.728,"15min":1.8500000000000001},"min":{"1min":0.42199999999999999,"5min":0.35199999999999998,"15min":0.312},"max":{"1min":9.0969999999999995,"5min":9.7029999999999994,"15min":20.489999999999998},"last":2.9249999999999998},{"interface":"front","average":{"1min":2.1400000000000001,"5min":1.585,"15min":1.7829999999999999},"min":{"1min":0.40300000000000002,"5min":0.35299999999999998,"15min":0.34200000000000003},"max":{"1min":6.8209999999999997,"5min":10.276999999999999,"15min":20.712},"last":0.60599999999999998}]},{"osd":8,"last update":"Fri Feb 9 17:43:56 2024","interfaces":[{"interface":"back","average":{"1min":2.7570000000000001,"5min":1.345,"15min":1.627},"min":{"1min":0.36299999999999999,"5min":0.30399999999999999,"15min":0.29499999999999998},"max":{"1min":9.5429999999999993,"5min":9.5429999999999993,"15min":13.199999999999999},"last":0.33300000000000002},{"interface":"front","average":{"1min":2.6829999999999998,"5min":1.4610000000000001,"15min":1.5009999999999999},"min":{"1min":0.36199999999999999,"5min":0.35399999999999998,"15min":0.318},"max":{"1min":7.5620000000000003,"5min":7.6059999999999999,"15min":9.2940000000000005},"last":0.42899999999999999}]},{"osd":9,"last update":"Fri Feb 9 17:44:25 2024","interfaces":[{"interface":"back","average":{"1min":1.738,"5min":2.1190000000000002,"15min":1.544},"min":{"1min":0.39000000000000001,"5min":0.34100000000000003,"15min":0.29399999999999998},"max":{"1min":4.4249999999999998,"5min":62.518000000000001,"15min":62.518000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":1.45,"5min":2.044,"15min":1.45},"min":{"1min":0.41099999999999998,"5min":0.35399999999999998,"15min":0.35199999999999998},"max":{"1min":4.8330000000000002,"5min":63.241999999999997,"15min":63.241999999999997},"last":0.64900000000000002}]},{"osd":10,"last update":"Fri Feb 9 17:44:33 2024","interfaces":[{"interface":"back","average":{"1min":2.5459999999999998,"5min":1.6739999999999999,"15min":1.401},"min":{"1min":0.47099999999999997,"5min":0.36799999999999999,"15min":0.36199999999999999},"max":{"1min":5.3799999999999999,"5min":9.6590000000000007,"15min":9.6590000000000007},"last":0.371},{"interface":"front","average":{"1min":1.821,"5min":1.613,"15min":1.4299999999999999},"min":{"1min":0.438,"5min":0.39100000000000001,"15min":0.33900000000000002},"max":{"1min":5.5579999999999998,"5min":12.409000000000001,"15min":12.856},"last":0.56399999999999995}]}]},{"osd":10,"up_from":84,"seq":360777253140,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":825124,"kb_used_data":424668,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":92920028,"statfs":{"total":95995035648,"available":95150108672,"internally_reserved":0,"allocated":434860032,"data_stored":434387155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":410057152},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:44:24 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.91500000000000004,"15min":1.45},"min":{"1min":0.25,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":2.7650000000000001,"5min":9.343,"15min":9.343},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.92000000000000004,"15min":1.105},"min":{"1min":0.29199999999999998,"5min":0.253,"15min":0.253},"max":{"1min":4.2699999999999996,"5min":6.3399999999999999,"15min":6.3399999999999999},"last":1.722}]},{"osd":1,"last update":"Fri Feb 9 17:43:51 2024","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.73699999999999999,"15min":0.65900000000000003},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":3.6960000000000002,"5min":3.726,"15min":3.726},"last":0.32500000000000001},{"interface":"front","average":{"1min":1.6739999999999999,"5min":1.0580000000000001,"15min":0.878},"min":{"1min":0.27300000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":5.5910000000000002,"5min":6.2709999999999999,"15min":6.2709999999999999},"last":1.0069999999999999}]},{"osd":2,"last update":"Fri Feb 9 17:44:02 2024","interfaces":[{"interface":"back","average":{"1min":1.548,"5min":1.2110000000000001,"15min":1.226},"min":{"1min":0.30599999999999999,"5min":0.27600000000000002,"15min":0.222},"max":{"1min":5.5279999999999996,"5min":6.5190000000000001,"15min":11.384},"last":2.25},{"interface":"front","average":{"1min":0.96199999999999997,"5min":1.1240000000000001,"15min":1.1890000000000001},"min":{"1min":0.39200000000000002,"5min":0.32500000000000001,"15min":0.26100000000000001},"max":{"1min":3.298,"5min":9.1140000000000008,"15min":11.465},"last":2.415}]},{"osd":3,"last update":"Fri Feb 9 17:44:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.173,"15min":1.756},"min":{"1min":0.30599999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":3.7559999999999998,"5min":9.1649999999999991,"15min":9.1649999999999991},"last":2.1139999999999999},{"interface":"front","average":{"1min":1.5449999999999999,"5min":1.4379999999999999,"15min":1.883},"min":{"1min":0.39500000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":7.0960000000000001,"5min":8.9670000000000005,"15min":8.9670000000000005},"last":0.66900000000000004}]},{"osd":4,"last update":"Fri Feb 9 17:44:06 2024","interfaces":[{"interface":"back","average":{"1min":1.875,"5min":1.2430000000000001,"15min":1.169},"min":{"1min":0.39200000000000002,"5min":0.30199999999999999,"15min":0.29199999999999998},"max":{"1min":8.5120000000000005,"5min":8.5120000000000005,"15min":8.5120000000000005},"last":9.0990000000000002},{"interface":"front","average":{"1min":2.1459999999999999,"5min":1.3109999999999999,"15min":1.1599999999999999},"min":{"1min":0.41299999999999998,"5min":0.39300000000000002,"15min":0.29999999999999999},"max":{"1min":11.273999999999999,"5min":11.273999999999999,"15min":11.273999999999999},"last":0.90500000000000003}]},{"osd":5,"last update":"Fri Feb 9 17:44:02 2024","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.0740000000000001,"15min":1.2390000000000001},"min":{"1min":0.34699999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":4.2279999999999998,"5min":4.4820000000000002,"15min":9.2349999999999994},"last":3.7930000000000001},{"interface":"front","average":{"1min":1.522,"5min":1.4970000000000001,"15min":1.3109999999999999},"min":{"1min":0.435,"5min":0.38500000000000001,"15min":0.33500000000000002},"max":{"1min":4.343,"5min":8.7710000000000008,"15min":8.7710000000000008},"last":9.0139999999999993}]},{"osd":6,"last update":"Fri Feb 9 17:44:02 2024","interfaces":[{"interface":"back","average":{"1min":2.5529999999999999,"5min":1.6679999999999999,"15min":1.4139999999999999},"min":{"1min":0.46000000000000002,"5min":0.32400000000000001,"15min":0.29999999999999999},"max":{"1min":6.4210000000000003,"5min":10.962,"15min":10.962},"last":1.0389999999999999},{"interface":"front","average":{"1min":1.292,"5min":1.4430000000000001,"15min":1.448},"min":{"1min":0.495,"5min":0.38600000000000001,"15min":0.35399999999999998},"max":{"1min":6.1280000000000001,"5min":8.8119999999999994,"15min":9.4350000000000005},"last":9.0690000000000008}]},{"osd":7,"last update":"Fri Feb 9 17:39:52 2024","interfaces":[{"interface":"back","average":{"1min":1.6000000000000001,"5min":1.367,"15min":1.415},"min":{"1min":0.42099999999999999,"5min":0.34200000000000003,"15min":0.312},"max":{"1min":4.0430000000000001,"5min":6.7590000000000003,"15min":17.754000000000001},"last":4.6449999999999996},{"interface":"front","average":{"1min":1.498,"5min":1.4359999999999999,"15min":1.5189999999999999},"min":{"1min":0.47399999999999998,"5min":0.38300000000000001,"15min":0.316},"max":{"1min":5.8170000000000002,"5min":11.308999999999999,"15min":11.308999999999999},"last":0.55000000000000004}]},{"osd":8,"last update":"Fri Feb 9 17:44:48 2024","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":1.0940000000000001,"15min":1.1359999999999999},"min":{"1min":0.37,"5min":0.34200000000000003,"15min":0.29099999999999998},"max":{"1min":6.9749999999999996,"5min":6.9749999999999996,"15min":7.4359999999999999},"last":1.198},{"interface":"front","average":{"1min":1.371,"5min":1.266,"15min":1.0640000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":5.6870000000000003,"5min":6.2009999999999996,"15min":7.7060000000000004},"last":7.0460000000000003}]},{"osd":9,"last update":"Fri Feb 9 17:44:02 2024","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.073,"15min":1.0429999999999999},"min":{"1min":0.433,"5min":0.36599999999999999,"15min":0.34399999999999997},"max":{"1min":3.1429999999999998,"5min":4.766,"15min":7.4850000000000003},"last":0.70399999999999996},{"interface":"front","average":{"1min":2.4950000000000001,"5min":1.377,"15min":1.1819999999999999},"min":{"1min":0.46700000000000003,"5min":0.441,"15min":0.29799999999999999},"max":{"1min":10.622999999999999,"5min":10.622999999999999,"15min":10.622999999999999},"last":0.73499999999999999}]},{"osd":11,"last update":"Fri Feb 9 17:44:24 2024","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.0589999999999999,"15min":1.016},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":3.4500000000000002,"5min":6.226,"15min":7.383},"last":0.81100000000000005},{"interface":"front","average":{"1min":1.159,"5min":1.1200000000000001,"15min":1.0660000000000001},"min":{"1min":0.47399999999999998,"5min":0.441,"15min":0.377},"max":{"1min":3.484,"5min":6.2590000000000003,"15min":7.3019999999999996},"last":0.84899999999999998}]}]},{"osd":9,"up_from":77,"seq":330712482074,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":662132,"kb_used_data":215720,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":93083020,"statfs":{"total":95995035648,"available":95317012480,"internally_reserved":0,"allocated":220897280,"data_stored":220419921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":456980928},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:43:56 2024","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.79000000000000004,"15min":0.65200000000000002},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":4.5830000000000002,"5min":7.2199999999999998,"15min":7.2199999999999998},"last":3.9239999999999999},{"interface":"front","average":{"1min":0.75,"5min":1.0109999999999999,"15min":0.97099999999999997},"min":{"1min":0.28399999999999997,"5min":0.253,"15min":0.253},"max":{"1min":5.6790000000000003,"5min":5.6790000000000003,"15min":5.6790000000000003},"last":0.26900000000000002}]},{"osd":1,"last update":"Fri Feb 9 17:44:44 2024","interfaces":[{"interface":"back","average":{"1min":1.843,"5min":1.5289999999999999,"15min":1.611},"min":{"1min":0.28299999999999997,"5min":0.23599999999999999,"15min":0.23100000000000001},"max":{"1min":8.1310000000000002,"5min":9.0449999999999999,"15min":89.515000000000001},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.72499999999999998,"5min":1.1870000000000001,"15min":1.5249999999999999},"min":{"1min":0.28100000000000003,"5min":0.249,"15min":0.222},"max":{"1min":1.891,"5min":11.598000000000001,"15min":89.388000000000005},"last":0.28499999999999998}]},{"osd":2,"last update":"Fri Feb 9 17:40:39 2024","interfaces":[{"interface":"back","average":{"1min":1.742,"5min":1.766,"15min":1.514},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":5.6630000000000003,"5min":14.121,"15min":14.121},"last":0.63400000000000001},{"interface":"front","average":{"1min":1.6910000000000001,"5min":1.52,"15min":1.4870000000000001},"min":{"1min":0.34100000000000003,"5min":0.26100000000000001,"15min":0.25600000000000001},"max":{"1min":5.7030000000000003,"5min":8.8040000000000003,"15min":13.69},"last":0.73899999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:44:44 2024","interfaces":[{"interface":"back","average":{"1min":1.6879999999999999,"5min":1.7130000000000001,"15min":1.6060000000000001},"min":{"1min":0.307,"5min":0.26500000000000001,"15min":0.25700000000000001},"max":{"1min":7.9569999999999999,"5min":9.6959999999999997,"15min":40.18},"last":6.3789999999999996},{"interface":"front","average":{"1min":0.999,"5min":1.2829999999999999,"15min":1.5149999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":3.8690000000000002,"5min":9.7140000000000004,"15min":40.158000000000001},"last":3.4079999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:44:44 2024","interfaces":[{"interface":"back","average":{"1min":1.917,"5min":1.6599999999999999,"15min":1.647},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":7.8899999999999997,"5min":9.0920000000000005,"15min":27.381},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.90300000000000002,"5min":1.151,"15min":1.454},"min":{"1min":0.36499999999999999,"5min":0.313,"15min":0.26100000000000001},"max":{"1min":2.5459999999999998,"5min":7.3010000000000002,"15min":27.355},"last":0.35999999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:44:52 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.3109999999999999,"15min":1.4630000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":4.2839999999999998,"5min":7.367,"15min":10.689},"last":3.1760000000000002},{"interface":"front","average":{"1min":1.7589999999999999,"5min":1.6459999999999999,"15min":1.391},"min":{"1min":0.318,"5min":0.318,"15min":0.311},"max":{"1min":7.2640000000000002,"5min":8.8460000000000001,"15min":13.978999999999999},"last":3.1899999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:44:44 2024","interfaces":[{"interface":"back","average":{"1min":1.982,"5min":1.8109999999999999,"15min":1.5760000000000001},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":7.3049999999999997,"5min":8.4299999999999997,"15min":14.069000000000001},"last":0.52100000000000002},{"interface":"front","average":{"1min":2.8490000000000002,"5min":2.1320000000000001,"15min":1.702},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":7.3689999999999998,"5min":10.417,"15min":14.097},"last":2.0619999999999998}]},{"osd":7,"last update":"Fri Feb 9 17:44:44 2024","interfaces":[{"interface":"back","average":{"1min":2.5259999999999998,"5min":2.0219999999999998,"15min":1.673},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":8.4770000000000003,"5min":9.125,"15min":9.125},"last":6.3010000000000002},{"interface":"front","average":{"1min":2.5539999999999998,"5min":2.0579999999999998,"15min":1.7390000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.32200000000000001},"max":{"1min":8.5090000000000003,"5min":9.157,"15min":9.157},"last":6.3380000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:44:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4610000000000001,"5min":1.3360000000000001,"15min":1.29},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":6.0800000000000001,"5min":7.8369999999999997,"15min":16.209},"last":0.53000000000000003},{"interface":"front","average":{"1min":1.671,"5min":1.524,"15min":1.351},"min":{"1min":0.501,"5min":0.39600000000000002,"15min":0.34799999999999998},"max":{"1min":6.0949999999999998,"5min":8.6950000000000003,"15min":16.263000000000002},"last":0.55000000000000004}]},{"osd":10,"last update":"Fri Feb 9 17:44:08 2024","interfaces":[{"interface":"back","average":{"1min":2.254,"5min":1.639,"15min":1.323},"min":{"1min":0.35599999999999998,"5min":0.312,"15min":0.312},"max":{"1min":10.206,"5min":10.206,"15min":10.206},"last":0.92200000000000004},{"interface":"front","average":{"1min":1.546,"5min":1.351,"15min":1.1830000000000001},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.38200000000000001},"max":{"1min":10.904,"5min":10.904,"15min":10.904},"last":0.43099999999999999}]},{"osd":11,"last update":"Fri Feb 9 17:44:08 2024","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.5289999999999999,"15min":1.474},"min":{"1min":0.439,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":5.7249999999999996,"5min":13.425000000000001,"15min":14.02},"last":0.48599999999999999},{"interface":"front","average":{"1min":2.2850000000000001,"5min":1.5629999999999999,"15min":1.381},"min":{"1min":0.48399999999999999,"5min":0.38100000000000001,"15min":0.374},"max":{"1min":10.863,"5min":10.863,"15min":10.863},"last":0.46899999999999997}]}]},{"osd":8,"up_from":171,"seq":734439407816,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":444264,"kb_used_data":182300,"kb_used_omap":212,"kb_used_meta":261739,"kb_avail":93300888,"statfs":{"total":95995035648,"available":95540109312,"internally_reserved":0,"allocated":186675200,"data_stored":186202323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217579,"internal_metadata":268021269},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:44:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.751,"15min":0.89200000000000002},"min":{"1min":0.249,"5min":0.217,"15min":0.217},"max":{"1min":2.0640000000000001,"5min":5.827,"15min":8.0920000000000005},"last":4.1559999999999997},{"interface":"front","average":{"1min":0.83799999999999997,"5min":1.2909999999999999,"15min":1.2150000000000001},"min":{"1min":0.25700000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":4.5860000000000003,"5min":7.2949999999999999,"15min":10.256},"last":2.6000000000000001}]},{"osd":1,"last update":"Fri Feb 9 17:44:45 2024","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":1.2729999999999999,"15min":1.286},"min":{"1min":0.252,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.548,"5min":8.718,"15min":8.718},"last":5.1269999999999998},{"interface":"front","average":{"1min":1.272,"5min":1.506,"15min":1.431},"min":{"1min":0.308,"5min":0.25600000000000001,"15min":0.23300000000000001},"max":{"1min":4.2069999999999999,"5min":25.027000000000001,"15min":25.027000000000001},"last":0.65500000000000003}]},{"osd":2,"last update":"Fri Feb 9 17:44:25 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.246,"15min":1.246},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":3.8919999999999999,"5min":3.8919999999999999,"15min":3.8919999999999999},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":3.9470000000000001,"5min":3.9470000000000001,"15min":3.9470000000000001},"last":0.44400000000000001}]},{"osd":3,"last update":"Fri Feb 9 17:42:38 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":1.542,"15min":1.6379999999999999},"min":{"1min":0.316,"5min":0.29499999999999998,"15min":0.28699999999999998},"max":{"1min":2.347,"5min":15.228999999999999,"15min":24.030999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.75600000000000001,"5min":1.8340000000000001,"15min":1.8640000000000001},"min":{"1min":0.36199999999999999,"5min":0.33600000000000002,"15min":0.30299999999999999},"max":{"1min":2.3719999999999999,"5min":14.988,"15min":24.318999999999999},"last":0.69399999999999995}]},{"osd":4,"last update":"Fri Feb 9 17:44:45 2024","interfaces":[{"interface":"back","average":{"1min":1.8839999999999999,"5min":1.605,"15min":1.655},"min":{"1min":0.35399999999999998,"5min":0.318,"15min":0.27500000000000002},"max":{"1min":7.0369999999999999,"5min":7.9649999999999999,"15min":53.060000000000002},"last":0.49199999999999999},{"interface":"front","average":{"1min":1.667,"5min":1.673,"15min":1.7210000000000001},"min":{"1min":0.31,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":6.9779999999999998,"5min":9.6140000000000008,"15min":53.311999999999998},"last":0.625}]},{"osd":5,"last update":"Fri Feb 9 17:44:16 2024","interfaces":[{"interface":"back","average":{"1min":2.4689999999999999,"5min":2.4689999999999999,"15min":2.4689999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":6.4249999999999998,"5min":6.4249999999999998,"15min":6.4249999999999998},"last":0.55300000000000005},{"interface":"front","average":{"1min":2.1779999999999999,"5min":2.1779999999999999,"15min":2.1779999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.383,"5min":6.383,"15min":6.383},"last":0.432}]},{"osd":6,"last update":"Fri Feb 9 17:44:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.4510000000000001,"15min":1.5609999999999999},"min":{"1min":0.371,"5min":0.33000000000000002,"15min":0.29799999999999999},"max":{"1min":4.875,"5min":9.5,"15min":37.356999999999999},"last":0.39600000000000002},{"interface":"front","average":{"1min":2.0659999999999998,"5min":1.637,"15min":1.504},"min":{"1min":0.32000000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":5.7039999999999997,"5min":6.4749999999999996,"15min":37.25},"last":12.157999999999999}]},{"osd":7,"last update":"Fri Feb 9 17:44:45 2024","interfaces":[{"interface":"back","average":{"1min":2.8620000000000001,"5min":1.921,"15min":1.778},"min":{"1min":0.47599999999999998,"5min":0.39600000000000002,"15min":0.32800000000000001},"max":{"1min":11.278,"5min":11.278,"15min":37.585000000000001},"last":0.52500000000000002},{"interface":"front","average":{"1min":2.895,"5min":1.9590000000000001,"15min":1.804},"min":{"1min":0.502,"5min":0.46200000000000002,"15min":0.312},"max":{"1min":11.313000000000001,"5min":11.313000000000001,"15min":37.918999999999997},"last":0.621}]},{"osd":9,"last update":"Fri Feb 9 17:44:45 2024","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":1.579,"15min":1.2290000000000001},"min":{"1min":0.35899999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":4.617,"5min":45.616,"15min":45.616},"last":1.673},{"interface":"front","average":{"1min":0.91400000000000003,"5min":1.51,"15min":1.2829999999999999},"min":{"1min":0.374,"5min":0.314,"15min":0.314},"max":{"1min":3.4239999999999999,"5min":45.718000000000004,"15min":45.718000000000004},"last":3.9340000000000002}]},{"osd":10,"last update":"Fri Feb 9 17:44:45 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.125,"15min":1.046},"min":{"1min":0.41299999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":7.8360000000000003,"5min":7.8360000000000003,"15min":10.65},"last":0.45500000000000002},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.325,"15min":1.1890000000000001},"min":{"1min":0.433,"5min":0.34799999999999998,"15min":0.318},"max":{"1min":4.9690000000000003,"5min":6.9539999999999997,"15min":11.071},"last":0.51200000000000001}]},{"osd":11,"last update":"Fri Feb 9 17:44:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.1759999999999999,"15min":1.2989999999999999},"min":{"1min":0.46500000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":4.4710000000000001,"5min":7.6959999999999997,"15min":14.433999999999999},"last":4.0010000000000003},{"interface":"front","average":{"1min":1.488,"5min":1.365,"15min":1.258},"min":{"1min":0.54600000000000004,"5min":0.308,"15min":0.308},"max":{"1min":4.444,"5min":7.4359999999999999,"15min":11.622},"last":0.68400000000000005}]}]},{"osd":7,"up_from":57,"seq":244813136162,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":883892,"kb_used_data":351464,"kb_used_omap":1,"kb_used_meta":532350,"kb_avail":92861260,"statfs":{"total":95995035648,"available":95089930240,"internally_reserved":0,"allocated":359899136,"data_stored":359418635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":545126858},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:44:24 2024","interfaces":[{"interface":"back","average":{"1min":1.804,"5min":1.3029999999999999,"15min":2.2850000000000001},"min":{"1min":0.27400000000000002,"5min":0.26000000000000001,"15min":0.252},"max":{"1min":5.9900000000000002,"5min":11.74,"15min":36.753},"last":0.29299999999999998},{"interface":"front","average":{"1min":1.8620000000000001,"5min":1.5,"15min":2.4449999999999998},"min":{"1min":0.27000000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":5.9429999999999996,"5min":11.843999999999999,"15min":36.918999999999997},"last":0.45000000000000001}]},{"osd":1,"last update":"Fri Feb 9 17:44:24 2024","interfaces":[{"interface":"back","average":{"1min":3.1659999999999999,"5min":2.1070000000000002,"15min":2.1200000000000001},"min":{"1min":0.26200000000000001,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":12.318,"5min":15.509,"15min":21.704999999999998},"last":0.435},{"interface":"front","average":{"1min":4.0739999999999998,"5min":2.2919999999999998,"15min":2.0670000000000002},"min":{"1min":0.31,"5min":0.27000000000000002,"15min":0.26500000000000001},"max":{"1min":12.359,"5min":15.276,"15min":21.858000000000001},"last":0.52100000000000002}]},{"osd":2,"last update":"Fri Feb 9 17:44:24 2024","interfaces":[{"interface":"back","average":{"1min":2.0059999999999998,"5min":2.0059999999999998,"15min":2.0059999999999998},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":6.6340000000000003,"5min":6.6340000000000003,"15min":6.6340000000000003},"last":0.46999999999999997},{"interface":"front","average":{"1min":2.468,"5min":2.468,"15min":2.468},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":11.968,"5min":11.968,"15min":11.968},"last":0.41599999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:44:43 2024","interfaces":[{"interface":"back","average":{"1min":1.7230000000000001,"5min":1.5349999999999999,"15min":1.7529999999999999},"min":{"1min":0.34000000000000002,"5min":0.28299999999999997,"15min":0.26900000000000002},"max":{"1min":5.9950000000000001,"5min":11.773999999999999,"15min":20.754000000000001},"last":2.9729999999999999},{"interface":"front","average":{"1min":2.6579999999999999,"5min":1.8240000000000001,"15min":1.8939999999999999},"min":{"1min":0.35499999999999998,"5min":0.32600000000000001,"15min":0.26500000000000001},"max":{"1min":6.7199999999999998,"5min":12.393000000000001,"15min":24.457000000000001},"last":2.9590000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:44:24 2024","interfaces":[{"interface":"back","average":{"1min":2.2450000000000001,"5min":1.3859999999999999,"15min":0.92800000000000005},"min":{"1min":0.30399999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":5.2629999999999999,"5min":5.29,"15min":5.29},"last":0.50800000000000001},{"interface":"front","average":{"1min":1.8540000000000001,"5min":1.6259999999999999,"15min":0.94599999999999995},"min":{"1min":0.27100000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":8.1440000000000001,"5min":11.967000000000001,"15min":11.967000000000001},"last":0.371}]},{"osd":5,"last update":"Fri Feb 9 17:44:24 2024","interfaces":[{"interface":"back","average":{"1min":3.1040000000000001,"5min":1.8999999999999999,"15min":1.0620000000000001},"min":{"1min":0.371,"5min":0.249,"15min":0.249},"max":{"1min":9.2170000000000005,"5min":9.6590000000000007,"15min":9.6590000000000007},"last":0.56999999999999995},{"interface":"front","average":{"1min":3.3460000000000001,"5min":2.0750000000000002,"15min":1.03},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":8.9600000000000009,"5min":15.228999999999999,"15min":15.228999999999999},"last":0.58499999999999996}]},{"osd":6,"last update":"Fri Feb 9 17:44:43 2024","interfaces":[{"interface":"back","average":{"1min":3.2650000000000001,"5min":2.238,"15min":1.9990000000000001},"min":{"1min":0.35199999999999998,"5min":0.28100000000000003,"15min":0.26000000000000001},"max":{"1min":8.9969999999999999,"5min":15.275,"15min":25.965},"last":0.55600000000000005},{"interface":"front","average":{"1min":2.6280000000000001,"5min":1.9099999999999999,"15min":1.7},"min":{"1min":0.44400000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":8.9779999999999998,"5min":18.236999999999998,"15min":18.236999999999998},"last":0.48099999999999998}]},{"osd":8,"last update":"Fri Feb 9 17:44:43 2024","interfaces":[{"interface":"back","average":{"1min":1.7050000000000001,"5min":1.655,"15min":1.524},"min":{"1min":0.48299999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":5.8109999999999999,"5min":11.81,"15min":11.81},"last":1.637},{"interface":"front","average":{"1min":2.5510000000000002,"5min":1.8029999999999999,"15min":1.554},"min":{"1min":0.38700000000000001,"5min":0.36599999999999999,"15min":0.33300000000000002},"max":{"1min":6.7569999999999997,"5min":12.058,"15min":12.058},"last":1.657}]},{"osd":9,"last update":"Fri Feb 9 17:44:34 2024","interfaces":[{"interface":"back","average":{"1min":2.2639999999999998,"5min":1.736,"15min":1.639},"min":{"1min":0.44,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":8.0020000000000007,"5min":11.941000000000001,"15min":11.941000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":2.54,"5min":1.9059999999999999,"15min":1.752},"min":{"1min":0.53700000000000003,"5min":0.379,"15min":0.33400000000000002},"max":{"1min":8.109,"5min":11.906000000000001,"15min":11.906000000000001},"last":4.1749999999999998}]},{"osd":10,"last update":"Fri Feb 9 17:43:58 2024","interfaces":[{"interface":"back","average":{"1min":2.1850000000000001,"5min":1.599,"15min":1.5720000000000001},"min":{"1min":0.38200000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":9.3030000000000008,"5min":12.050000000000001,"15min":12.050000000000001},"last":0.747},{"interface":"front","average":{"1min":2.7469999999999999,"5min":1.6759999999999999,"15min":1.7150000000000001},"min":{"1min":0.48199999999999998,"5min":0.38900000000000001,"15min":0.35599999999999998},"max":{"1min":11.878,"5min":11.878,"15min":11.878},"last":2.7309999999999999}]},{"osd":11,"last update":"Fri Feb 9 17:44:24 2024","interfaces":[{"interface":"back","average":{"1min":2.5790000000000002,"5min":1.7130000000000001,"15min":1.8779999999999999},"min":{"1min":0.48099999999999998,"5min":0.42399999999999999,"15min":0.39500000000000002},"max":{"1min":5.9089999999999998,"5min":9.2490000000000006,"15min":21.945},"last":1.7410000000000001},{"interface":"front","average":{"1min":3.4300000000000002,"5min":1.8520000000000001,"15min":1.796},"min":{"1min":0.50700000000000001,"5min":0.45500000000000002,"15min":0.39500000000000002},"max":{"1min":12.423999999999999,"5min":12.423999999999999,"15min":21.731999999999999},"last":2.6080000000000001}]}]},{"osd":6,"up_from":49,"seq":210453397800,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":640048,"kb_used_data":234024,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":93105104,"statfs":{"total":95995035648,"available":95339626496,"internally_reserved":0,"allocated":239640576,"data_stored":239163217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":415758790},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:44:15 2024","interfaces":[{"interface":"back","average":{"1min":2.1509999999999998,"5min":1.776,"15min":1.651},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.21299999999999999},"max":{"1min":5.7619999999999996,"5min":9.2089999999999996,"15min":12.544},"last":0.38300000000000001},{"interface":"front","average":{"1min":2.0289999999999999,"5min":1.825,"15min":1.599},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.217},"max":{"1min":5.8289999999999997,"5min":9.2520000000000007,"15min":9.2520000000000007},"last":1.0149999999999999}]},{"osd":1,"last update":"Fri Feb 9 17:44:15 2024","interfaces":[{"interface":"back","average":{"1min":2.2610000000000001,"5min":1.7549999999999999,"15min":1.895},"min":{"1min":0.316,"5min":0.28599999999999998,"15min":0.25900000000000001},"max":{"1min":6.1849999999999996,"5min":8.3260000000000005,"15min":17.762},"last":4.5010000000000003},{"interface":"front","average":{"1min":2.4430000000000001,"5min":1.8580000000000001,"15min":1.9670000000000001},"min":{"1min":0.34799999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":12.475,"5min":12.475,"15min":17.734999999999999},"last":4.5339999999999998}]},{"osd":2,"last update":"Fri Feb 9 17:44:15 2024","interfaces":[{"interface":"back","average":{"1min":2.097,"5min":1.6899999999999999,"15min":1.579},"min":{"1min":0.40200000000000002,"5min":0.32000000000000001,"15min":0.22600000000000001},"max":{"1min":5.9470000000000001,"5min":9.8019999999999996,"15min":9.8019999999999996},"last":0.53900000000000003},{"interface":"front","average":{"1min":2.145,"5min":1.7290000000000001,"15min":1.6379999999999999},"min":{"1min":0.433,"5min":0.372,"15min":0.28000000000000003},"max":{"1min":5.9640000000000004,"5min":9.8439999999999994,"15min":10.156000000000001},"last":0.57999999999999996}]},{"osd":3,"last update":"Fri Feb 9 17:44:15 2024","interfaces":[{"interface":"back","average":{"1min":1.9430000000000001,"5min":1.7030000000000001,"15min":1.9930000000000001},"min":{"1min":0.29599999999999999,"5min":0.27800000000000002,"15min":0.25600000000000001},"max":{"1min":13.016,"5min":13.016,"15min":32.545000000000002},"last":1.6120000000000001},{"interface":"front","average":{"1min":2.2250000000000001,"5min":1.734,"15min":1.974},"min":{"1min":0.40000000000000002,"5min":0.34200000000000003,"15min":0.29599999999999999},"max":{"1min":6.234,"5min":9.8219999999999992,"15min":32.670999999999999},"last":0.66700000000000004}]},{"osd":4,"last update":"Fri Feb 9 17:44:15 2024","interfaces":[{"interface":"back","average":{"1min":3.2989999999999999,"5min":2.0070000000000001,"15min":1.78},"min":{"1min":0.36799999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":15.061999999999999,"5min":15.061999999999999,"15min":19.798999999999999},"last":1.8400000000000001},{"interface":"front","average":{"1min":3.915,"5min":2.4049999999999998,"15min":1.796},"min":{"1min":0.35099999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":15.118,"5min":15.118,"15min":15.118},"last":1.8779999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:44:15 2024","interfaces":[{"interface":"back","average":{"1min":2.9529999999999998,"5min":2.024,"15min":1.7390000000000001},"min":{"1min":0.47199999999999998,"5min":0.29799999999999999,"15min":0.29499999999999998},"max":{"1min":7.8689999999999998,"5min":9.9849999999999994,"15min":10.968999999999999},"last":10.254},{"interface":"front","average":{"1min":3.1560000000000001,"5min":1.8400000000000001,"15min":1.7130000000000001},"min":{"1min":0.39400000000000002,"5min":0.316,"15min":0.30199999999999999},"max":{"1min":11.079000000000001,"5min":11.079000000000001,"15min":13.91},"last":9.4199999999999999}]},{"osd":7,"last update":"Fri Feb 9 17:44:55 2024","interfaces":[{"interface":"back","average":{"1min":2.2130000000000001,"5min":2.226,"15min":1.8560000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.28000000000000003},"max":{"1min":7.633,"5min":11.052,"15min":14.022},"last":3.8149999999999999},{"interface":"front","average":{"1min":2.1680000000000001,"5min":1.905,"15min":1.9199999999999999},"min":{"1min":0.38400000000000001,"5min":0.314,"15min":0.314},"max":{"1min":9.0310000000000006,"5min":12.967000000000001,"15min":15.458},"last":0.623}]},{"osd":8,"last update":"Fri Feb 9 17:26:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":0.876,"15min":0.79600000000000004},"min":{"1min":0.39900000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":4.0339999999999998,"5min":4.4880000000000004,"15min":4.4880000000000004},"last":4.2809999999999997},{"interface":"front","average":{"1min":1.0469999999999999,"5min":0.88600000000000001,"15min":0.78300000000000003},"min":{"1min":0.375,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":3.6539999999999999,"5min":4.8570000000000002,"15min":4.8570000000000002},"last":2.9529999999999998}]},{"osd":9,"last update":"Fri Feb 9 17:44:20 2024","interfaces":[{"interface":"back","average":{"1min":2.153,"5min":1.9410000000000001,"15min":1.5209999999999999},"min":{"1min":0.58899999999999997,"5min":0.438,"15min":0.35899999999999999},"max":{"1min":5.2599999999999998,"5min":10.779999999999999,"15min":10.779999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":2.4580000000000002,"5min":2.024,"15min":1.641},"min":{"1min":0.495,"5min":0.40999999999999998,"15min":0.40400000000000003},"max":{"1min":7.5380000000000003,"5min":7.5380000000000003,"15min":8.1630000000000003},"last":1.71}]},{"osd":10,"last update":"Fri Feb 9 17:44:55 2024","interfaces":[{"interface":"back","average":{"1min":1.7809999999999999,"5min":2.1000000000000001,"15min":1.6439999999999999},"min":{"1min":0.36099999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":8.2690000000000001,"5min":25.370000000000001,"15min":25.370000000000001},"last":0.48499999999999999},{"interface":"front","average":{"1min":1.5649999999999999,"5min":1.907,"15min":1.704},"min":{"1min":0.39800000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":7.351,"5min":26.027999999999999,"15min":26.027999999999999},"last":0.83999999999999997}]},{"osd":11,"last update":"Fri Feb 9 17:44:15 2024","interfaces":[{"interface":"back","average":{"1min":2.3839999999999999,"5min":1.9650000000000001,"15min":1.661},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.36099999999999999},"max":{"1min":13.109,"5min":13.109,"15min":13.109},"last":0.55900000000000005},{"interface":"front","average":{"1min":2.0779999999999998,"5min":1.8480000000000001,"15min":1.639},"min":{"1min":0.38800000000000001,"5min":0.373,"15min":0.373},"max":{"1min":13.164999999999999,"5min":13.164999999999999,"15min":13.164999999999999},"last":3.5209999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575170,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":911844,"kb_used_data":312796,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":92833308,"statfs":{"total":95995035648,"available":95061307392,"internally_reserved":0,"allocated":320303104,"data_stored":319830227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:44:33 2024","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.94499999999999995,"15min":1.484},"min":{"1min":0.28000000000000003,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":4.8029999999999999,"5min":7.0220000000000002,"15min":19.079000000000001},"last":2.5609999999999999},{"interface":"front","average":{"1min":1.3280000000000001,"5min":0.96899999999999997,"15min":1.4379999999999999},"min":{"1min":0.29399999999999998,"5min":0.218,"15min":0.19400000000000001},"max":{"1min":5.125,"5min":8.8100000000000005,"15min":23.573},"last":2.3839999999999999}]},{"osd":2,"last update":"Fri Feb 9 17:43:57 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":1.423,"15min":1.528},"min":{"1min":0.246,"5min":0.23400000000000001,"15min":0.22600000000000001},"max":{"1min":4.4160000000000004,"5min":12.52,"15min":26.361999999999998},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.89100000000000001,"5min":1.3700000000000001,"15min":1.8180000000000001},"min":{"1min":0.27800000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":3.4870000000000001,"5min":8.2360000000000007,"15min":26.087},"last":1.8180000000000001}]},{"osd":3,"last update":"Fri Feb 9 17:44:18 2024","interfaces":[{"interface":"back","average":{"1min":1.548,"5min":1.548,"15min":1.548},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":6.7649999999999997,"5min":6.7649999999999997,"15min":6.7649999999999997},"last":1.9790000000000001},{"interface":"front","average":{"1min":1.012,"5min":1.012,"15min":1.012},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":4.4790000000000001,"5min":4.4790000000000001,"15min":4.4790000000000001},"last":2.319}]},{"osd":4,"last update":"Fri Feb 9 17:44:27 2024","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.476,"15min":1.8140000000000001},"min":{"1min":0.32700000000000001,"5min":0.27700000000000002,"15min":0.253},"max":{"1min":3.907,"5min":6.7679999999999998,"15min":11.766999999999999},"last":5.8730000000000002},{"interface":"front","average":{"1min":2.8130000000000002,"5min":1.7809999999999999,"15min":1.9339999999999999},"min":{"1min":0.35099999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":13.871,"5min":14.593,"15min":14.593},"last":0.68100000000000005}]},{"osd":5,"last update":"Fri Feb 9 17:44:13 2024","interfaces":[{"interface":"back","average":{"1min":1.5840000000000001,"5min":1.6970000000000001,"15min":1.8069999999999999},"min":{"1min":0.35899999999999999,"5min":0.29199999999999998,"15min":0.27000000000000002},"max":{"1min":6.2850000000000001,"5min":6.8250000000000002,"15min":13.544},"last":5.1319999999999997},{"interface":"front","average":{"1min":1.6739999999999999,"5min":1.4970000000000001,"15min":1.7470000000000001},"min":{"1min":0.40500000000000003,"5min":0.32100000000000001,"15min":0.26800000000000002},"max":{"1min":6.609,"5min":10.263999999999999,"15min":12.117000000000001},"last":2.2429999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:44:18 2024","interfaces":[{"interface":"back","average":{"1min":2.387,"5min":1.5820000000000001,"15min":1.3759999999999999},"min":{"1min":0.377,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":12.57,"5min":12.57,"15min":12.57},"last":2.3450000000000002},{"interface":"front","average":{"1min":2.4369999999999998,"5min":1.5660000000000001,"15min":1.3049999999999999},"min":{"1min":0.51700000000000002,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":12.605,"5min":12.605,"15min":12.605},"last":2.1539999999999999}]},{"osd":7,"last update":"Fri Feb 9 17:44:18 2024","interfaces":[{"interface":"back","average":{"1min":1.8460000000000001,"5min":1.391,"15min":1.2350000000000001},"min":{"1min":0.45200000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":6.5830000000000002,"5min":6.5830000000000002,"15min":6.5830000000000002},"last":4.665},{"interface":"front","average":{"1min":1.8859999999999999,"5min":1.552,"15min":1.3420000000000001},"min":{"1min":0.48199999999999998,"5min":0.309,"15min":0.309},"max":{"1min":6.5599999999999996,"5min":6.5599999999999996,"15min":6.5599999999999996},"last":4.7039999999999997}]},{"osd":8,"last update":"Fri Feb 9 17:44:27 2024","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":0.96599999999999997,"15min":0.91600000000000004},"min":{"1min":0.372,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":3.3399999999999999,"5min":9.2520000000000007,"15min":9.2520000000000007},"last":2.1880000000000002},{"interface":"front","average":{"1min":1.2450000000000001,"5min":0.97899999999999998,"15min":0.89300000000000002},"min":{"1min":0.42899999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":3.367,"5min":9.2810000000000006,"15min":9.2810000000000006},"last":2.2160000000000002}]},{"osd":9,"last update":"Fri Feb 9 17:44:33 2024","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":1.111,"15min":1.4430000000000001},"min":{"1min":0.39200000000000002,"5min":0.33900000000000002,"15min":0.32400000000000001},"max":{"1min":4.7039999999999997,"5min":6.8819999999999997,"15min":9.5839999999999996},"last":0.64900000000000002},{"interface":"front","average":{"1min":1.395,"5min":1.4810000000000001,"15min":1.5660000000000001},"min":{"1min":0.44400000000000001,"5min":0.375,"15min":0.32900000000000001},"max":{"1min":4.4569999999999999,"5min":8.7780000000000005,"15min":11.366},"last":1.855}]},{"osd":10,"last update":"Fri Feb 9 17:44:13 2024","interfaces":[{"interface":"back","average":{"1min":1.482,"5min":1.5429999999999999,"15min":1.6080000000000001},"min":{"1min":0.55800000000000005,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":6.1769999999999996,"5min":15.083,"15min":15.083},"last":2.266},{"interface":"front","average":{"1min":1.6160000000000001,"5min":1.605,"15min":1.702},"min":{"1min":0.61499999999999999,"5min":0.46700000000000003,"15min":0.40899999999999997},"max":{"1min":6.2140000000000004,"5min":15.121,"15min":15.121},"last":2.29}]},{"osd":11,"last update":"Fri Feb 9 17:44:18 2024","interfaces":[{"interface":"back","average":{"1min":1.8320000000000001,"5min":1.2609999999999999,"15min":1.0529999999999999},"min":{"1min":0.441,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":6.5449999999999999,"5min":6.5449999999999999,"15min":6.5449999999999999},"last":1.881},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.2210000000000001,"15min":1.224},"min":{"1min":0.63200000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":3.2999999999999998,"5min":3.657,"15min":3.657},"last":2.4340000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738694,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1200484,"kb_used_data":577884,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":92544668,"statfs":{"total":95995035648,"available":94765740032,"internally_reserved":0,"allocated":591753216,"data_stored":591280339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 9 17:44:13 2024","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.2889999999999999,"15min":1.3540000000000001},"min":{"1min":0.217,"5min":0.20599999999999999,"15min":0.20499999999999999},"max":{"1min":4.7530000000000001,"5min":13.842000000000001,"15min":13.842000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.411,"15min":1.5009999999999999},"min":{"1min":0.28299999999999997,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":4.7889999999999997,"5min":14.122,"15min":14.122},"last":0.55800000000000005}]},{"osd":2,"last update":"Fri Feb 9 17:39:25 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":5.5339999999999998,"5min":5.5339999999999998,"15min":5.5339999999999998},"last":1.1359999999999999},{"interface":"front","average":{"1min":1.593,"5min":1.593,"15min":1.593},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":13.473000000000001,"5min":13.473000000000001,"15min":13.473000000000001},"last":1.516}]},{"osd":3,"last update":"Fri Feb 9 17:43:49 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":1.6850000000000001,"15min":2.577},"min":{"1min":0.32100000000000001,"5min":0.247,"15min":0.247},"max":{"1min":4.5750000000000002,"5min":16.756,"15min":16.756},"last":0.48899999999999999},{"interface":"front","average":{"1min":1.337,"5min":1.264,"15min":1.625},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":6.5570000000000004,"5min":6.5570000000000004,"15min":6.5570000000000004},"last":0.53700000000000003}]},{"osd":4,"last update":"Fri Feb 9 17:44:37 2024","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.2330000000000001,"15min":1.5209999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.26400000000000001},"max":{"1min":4.5620000000000003,"5min":7.7830000000000004,"15min":14.419},"last":2.242},{"interface":"front","average":{"1min":1.54,"5min":1.5129999999999999,"15min":1.7490000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.27200000000000002},"max":{"1min":4.5330000000000004,"5min":7.8230000000000004,"15min":14.465999999999999},"last":2.2770000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:43:52 2024","interfaces":[{"interface":"back","average":{"1min":1.611,"5min":1.371,"15min":1.569},"min":{"1min":0.314,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":7.9690000000000003,"5min":8.4079999999999995,"15min":9.8569999999999993},"last":0.30299999999999999},{"interface":"front","average":{"1min":1.734,"5min":1.206,"15min":1.675},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":7.9950000000000001,"5min":7.9950000000000001,"15min":9.9049999999999994},"last":0.374}]},{"osd":6,"last update":"Fri Feb 9 17:44:24 2024","interfaces":[{"interface":"back","average":{"1min":1.837,"5min":1.466,"15min":1.706},"min":{"1min":0.38500000000000001,"5min":0.28699999999999998,"15min":0.27800000000000002},"max":{"1min":6.4489999999999998,"5min":6.4489999999999998,"15min":13.407},"last":0.39800000000000002},{"interface":"front","average":{"1min":1.8700000000000001,"5min":1.51,"15min":1.772},"min":{"1min":0.42199999999999999,"5min":0.33300000000000002,"15min":0.33000000000000002},"max":{"1min":6.4749999999999996,"5min":6.4749999999999996,"15min":13.448},"last":0.44}]},{"osd":7,"last update":"Fri Feb 9 17:44:50 2024","interfaces":[{"interface":"back","average":{"1min":2.5710000000000002,"5min":1.8999999999999999,"15min":2.0710000000000002},"min":{"1min":0.34899999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":11.872999999999999,"5min":11.872999999999999,"15min":13.349},"last":0.47599999999999998},{"interface":"front","average":{"1min":2.2360000000000002,"5min":1.869,"15min":1.99},"min":{"1min":0.34999999999999998,"5min":0.33300000000000002,"15min":0.28699999999999998},"max":{"1min":6.3460000000000001,"5min":9.6980000000000004,"15min":14.451000000000001},"last":1.4059999999999999}]},{"osd":8,"last update":"Fri Feb 9 17:44:50 2024","interfaces":[{"interface":"back","average":{"1min":1.494,"5min":1.5029999999999999,"15min":1.508},"min":{"1min":0.39600000000000002,"5min":0.33900000000000002,"15min":0.33100000000000002},"max":{"1min":5.4770000000000003,"5min":9.5359999999999996,"15min":14.323},"last":1.4470000000000001},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.532,"15min":1.633},"min":{"1min":0.36299999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":5.5780000000000003,"5min":9.5700000000000003,"15min":14.481999999999999},"last":1.47}]},{"osd":9,"last update":"Fri Feb 9 17:44:37 2024","interfaces":[{"interface":"back","average":{"1min":2.2189999999999999,"5min":2.1789999999999998,"15min":1.788},"min":{"1min":0.42299999999999999,"5min":0.33800000000000002,"15min":0.32500000000000001},"max":{"1min":14.903,"5min":17.952000000000002,"15min":17.952000000000002},"last":1.494},{"interface":"front","average":{"1min":1.746,"5min":2.1459999999999999,"15min":1.768},"min":{"1min":0.44400000000000001,"5min":0.40999999999999998,"15min":0.34799999999999998},"max":{"1min":9.7170000000000005,"5min":17.913,"15min":17.913},"last":2.782}]},{"osd":10,"last update":"Fri Feb 9 17:43:52 2024","interfaces":[{"interface":"back","average":{"1min":2.4580000000000002,"5min":2.3119999999999998,"15min":1.833},"min":{"1min":0.40699999999999997,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":9.7270000000000003,"5min":31.766999999999999,"15min":31.766999999999999},"last":2.669},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.343,"15min":1.47},"min":{"1min":0.41399999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":7.6849999999999996,"5min":28.048999999999999,"15min":28.048999999999999},"last":0.34200000000000003}]},{"osd":11,"last update":"Fri Feb 9 17:44:24 2024","interfaces":[{"interface":"back","average":{"1min":2.0750000000000002,"5min":1.335,"15min":1.583},"min":{"1min":0.442,"5min":0.41899999999999998,"15min":0.40600000000000003},"max":{"1min":13.102,"5min":13.102,"15min":40.021000000000001},"last":0.41599999999999998},{"interface":"front","average":{"1min":1.464,"5min":1.351,"15min":1.615},"min":{"1min":0.46800000000000003,"5min":0.46500000000000002,"15min":0.45200000000000001},"max":{"1min":4.7839999999999998,"5min":5.8390000000000004,"15min":40.055999999999997},"last":0.45500000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411644,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":488300,"kb_used_data":226660,"kb_used_omap":1,"kb_used_meta":261630,"kb_avail":93256852,"statfs":{"total":95995035648,"available":95495016448,"internally_reserved":0,"allocated":232099840,"data_stored":231623411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":267909573},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":1.4510000000000001,"15min":1.006},"min":{"1min":0.29999999999999999,"5min":0.23699999999999999,"15min":0.221},"max":{"1min":4.9089999999999998,"5min":17.251000000000001,"15min":17.251000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.74399999999999999,"5min":1.611,"15min":1.113},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.22900000000000001},"max":{"1min":4.7720000000000002,"5min":23.097999999999999,"15min":23.097999999999999},"last":0.875}]},{"osd":1,"last update":"Fri Feb 9 17:44:49 2024","interfaces":[{"interface":"back","average":{"1min":1.514,"5min":1.3180000000000001,"15min":1.5189999999999999},"min":{"1min":0.25,"5min":0.216,"15min":0.216},"max":{"1min":9.4380000000000006,"5min":15.186,"15min":22.306000000000001},"last":0.41999999999999998},{"interface":"front","average":{"1min":1.446,"5min":1.3220000000000001,"15min":1.5640000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.214},"max":{"1min":12.936,"5min":18.181000000000001,"15min":44.838999999999999},"last":2.0779999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:44:17 2024","interfaces":[{"interface":"back","average":{"1min":2.0339999999999998,"5min":1.6379999999999999,"15min":1.514},"min":{"1min":0.29599999999999999,"5min":0.249,"15min":0.22700000000000001},"max":{"1min":14.355,"5min":17.942,"15min":24.353000000000002},"last":7.7300000000000004},{"interface":"front","average":{"1min":2.4089999999999998,"5min":1.3879999999999999,"15min":1.6259999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.6099999999999994,"5min":10.351000000000001,"15min":19.986999999999998},"last":2.4039999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:44:17 2024","interfaces":[{"interface":"back","average":{"1min":1.653,"5min":1.397,"15min":1.7509999999999999},"min":{"1min":0.32100000000000001,"5min":0.29399999999999998,"15min":0.26300000000000001},"max":{"1min":8.7230000000000008,"5min":10.048999999999999,"15min":13.526},"last":0.497},{"interface":"front","average":{"1min":1.8839999999999999,"5min":1.49,"15min":1.6080000000000001},"min":{"1min":0.29499999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":6.1660000000000004,"5min":8.9920000000000009,"15min":9.9619999999999997},"last":0.67000000000000004}]},{"osd":5,"last update":"Fri Feb 9 17:44:00 2024","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":1.6910000000000001,"15min":1.746},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":3.5219999999999998,"5min":42.664000000000001,"15min":42.664000000000001},"last":0.70599999999999996},{"interface":"front","average":{"1min":2.0579999999999998,"5min":2.145,"15min":1.964},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":8.6600000000000001,"5min":43.311999999999998,"15min":43.311999999999998},"last":2.306}]},{"osd":6,"last update":"Fri Feb 9 17:44:17 2024","interfaces":[{"interface":"back","average":{"1min":2.6760000000000002,"5min":1.7330000000000001,"15min":1.762},"min":{"1min":0.34200000000000003,"5min":0.312,"15min":0.30099999999999999},"max":{"1min":7.8949999999999996,"5min":10.114000000000001,"15min":10.295},"last":0.83599999999999997},{"interface":"front","average":{"1min":2.6080000000000001,"5min":1.8520000000000001,"15min":1.9830000000000001},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":7.726,"5min":10.144,"15min":11.269},"last":0.73099999999999998}]},{"osd":7,"last update":"Fri Feb 9 17:44:49 2024","interfaces":[{"interface":"back","average":{"1min":2.7839999999999998,"5min":1.718,"15min":2.0619999999999998},"min":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":11.099,"5min":11.099,"15min":41.802999999999997},"last":0.52400000000000002},{"interface":"front","average":{"1min":2.8199999999999998,"5min":2.0009999999999999,"15min":2.0739999999999998},"min":{"1min":0.41599999999999998,"5min":0.34100000000000003,"15min":0.30499999999999999},"max":{"1min":8.4079999999999995,"5min":11.099,"15min":41.783999999999999},"last":0.75600000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:44:17 2024","interfaces":[{"interface":"back","average":{"1min":1.137,"5min":1.401,"15min":1.3200000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":8.7729999999999997,"5min":8.7729999999999997,"15min":22.393999999999998},"last":0.54600000000000004},{"interface":"front","average":{"1min":1.5629999999999999,"5min":1.3520000000000001,"15min":1.5049999999999999},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":3.4729999999999999,"5min":12.58,"15min":12.58},"last":0.78100000000000003}]},{"osd":9,"last update":"Fri Feb 9 17:44:28 2024","interfaces":[{"interface":"back","average":{"1min":2.0920000000000001,"5min":1.5129999999999999,"15min":1.6100000000000001},"min":{"1min":0.37,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":12.768000000000001,"5min":12.768000000000001,"15min":13.5},"last":2.3439999999999999},{"interface":"front","average":{"1min":1.929,"5min":1.3660000000000001,"15min":1.5720000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.36399999999999999},"max":{"1min":7.5099999999999998,"5min":9.0060000000000002,"15min":13.589},"last":0.80300000000000005}]},{"osd":10,"last update":"Fri Feb 9 17:44:10 2024","interfaces":[{"interface":"back","average":{"1min":1.877,"5min":1.611,"15min":1.5269999999999999},"min":{"1min":0.42199999999999999,"5min":0.38400000000000001,"15min":0.33600000000000002},"max":{"1min":8.7509999999999994,"5min":9.1419999999999995,"15min":22.425999999999998},"last":0.82199999999999995},{"interface":"front","average":{"1min":1.905,"5min":1.643,"15min":1.526},"min":{"1min":0.438,"5min":0.438,"15min":0.379},"max":{"1min":8.8109999999999999,"5min":9.1899999999999995,"15min":22.454999999999998},"last":0.84999999999999998}]},{"osd":11,"last update":"Fri Feb 9 17:44:17 2024","interfaces":[{"interface":"back","average":{"1min":1.738,"5min":1.514,"15min":2.04},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.39100000000000001},"max":{"1min":5.1580000000000004,"5min":7.4420000000000002,"15min":137.44300000000001},"last":6.2290000000000001},{"interface":"front","average":{"1min":1.5980000000000001,"5min":1.494,"15min":2.1000000000000001},"min":{"1min":0.45700000000000002,"5min":0.41199999999999998,"15min":0.40100000000000002},"max":{"1min":8.7699999999999996,"5min":9.1029999999999998,"15min":137.64400000000001},"last":2.375}]}]},{"osd":3,"up_from":27,"seq":115964117304,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":494184,"kb_used_data":88156,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":93250968,"statfs":{"total":95995035648,"available":95488991232,"internally_reserved":0,"allocated":90271744,"data_stored":89798867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":415758791},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:44:14 2024","interfaces":[{"interface":"back","average":{"1min":1.657,"5min":1.6240000000000001,"15min":1.77},"min":{"1min":0.28499999999999998,"5min":0.23300000000000001,"15min":0.186},"max":{"1min":12.789,"5min":25.849,"15min":38.984999999999999},"last":1.127},{"interface":"front","average":{"1min":1.6100000000000001,"5min":1.718,"15min":1.8959999999999999},"min":{"1min":0.29499999999999998,"5min":0.248,"15min":0.22500000000000001},"max":{"1min":11.932,"5min":30.268999999999998,"15min":41.497999999999998},"last":1.218}]},{"osd":1,"last update":"Fri Feb 9 17:40:06 2024","interfaces":[{"interface":"back","average":{"1min":4.5670000000000002,"5min":4.5670000000000002,"15min":4.5670000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":25.965,"5min":25.965,"15min":25.965},"last":0.35299999999999998},{"interface":"front","average":{"1min":4.5129999999999999,"5min":4.5129999999999999,"15min":4.5129999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":31.454000000000001,"5min":31.454000000000001,"15min":31.454000000000001},"last":0.375}]},{"osd":2,"last update":"Fri Feb 9 17:44:10 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.3520000000000001,"15min":1.4419999999999999},"min":{"1min":0.29099999999999998,"5min":0.255,"15min":0.22600000000000001},"max":{"1min":4.6749999999999998,"5min":11,"15min":14.888999999999999},"last":0.35099999999999998},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.456,"15min":1.538},"min":{"1min":0.311,"5min":0.24199999999999999,"15min":0.20200000000000001},"max":{"1min":5.2050000000000001,"5min":12.347,"15min":17.613},"last":0.59499999999999997}]},{"osd":4,"last update":"Fri Feb 9 17:44:14 2024","interfaces":[{"interface":"back","average":{"1min":1.9259999999999999,"5min":1.429,"15min":1.6040000000000001},"min":{"1min":0.36099999999999999,"5min":0.29099999999999998,"15min":0.26800000000000002},"max":{"1min":11.880000000000001,"5min":11.880000000000001,"15min":34.898000000000003},"last":0.95699999999999996},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.4570000000000001,"15min":1.6599999999999999},"min":{"1min":0.29999999999999999,"5min":0.29399999999999998,"15min":0.27900000000000003},"max":{"1min":11.622999999999999,"5min":12.404999999999999,"15min":14.061999999999999},"last":0.70599999999999996}]},{"osd":5,"last update":"Fri Feb 9 17:43:52 2024","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":1.639,"15min":1.663},"min":{"1min":0.313,"5min":0.28000000000000003,"15min":0.27800000000000002},"max":{"1min":4.6980000000000004,"5min":11.007999999999999,"15min":17.651},"last":3.5579999999999998},{"interface":"front","average":{"1min":1.649,"5min":1.7370000000000001,"15min":1.7290000000000001},"min":{"1min":0.42599999999999999,"5min":0.35299999999999998,"15min":0.34100000000000003},"max":{"1min":7.6219999999999999,"5min":11.036,"15min":17.675999999999998},"last":3.544}]},{"osd":6,"last update":"Fri Feb 9 17:44:43 2024","interfaces":[{"interface":"back","average":{"1min":1.5309999999999999,"5min":1.6819999999999999,"15min":2.3900000000000001},"min":{"1min":0.33100000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":4.7699999999999996,"5min":12.717000000000001,"15min":12.717000000000001},"last":0.61499999999999999},{"interface":"front","average":{"1min":1.726,"5min":1.5880000000000001,"15min":1.466},"min":{"1min":0.40899999999999997,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":4.9269999999999996,"5min":8.7360000000000007,"15min":8.7360000000000007},"last":1.1799999999999999}]},{"osd":7,"last update":"Fri Feb 9 17:44:25 2024","interfaces":[{"interface":"back","average":{"1min":2.6160000000000001,"5min":1.6779999999999999,"15min":1.1399999999999999},"min":{"1min":0.46100000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":11.769,"5min":11.769,"15min":11.769},"last":1.026},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.1080000000000001,"15min":0.872},"min":{"1min":0.432,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":6.0990000000000002,"5min":6.0990000000000002,"15min":6.0990000000000002},"last":3.9300000000000002}]},{"osd":8,"last update":"Fri Feb 9 17:44:14 2024","interfaces":[{"interface":"back","average":{"1min":2.0910000000000002,"5min":1.5109999999999999,"15min":1.472},"min":{"1min":0.377,"5min":0.377,"15min":0.26800000000000002},"max":{"1min":11.821,"5min":11.821,"15min":11.821},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.7669999999999999,"5min":1.4410000000000001,"15min":1.325},"min":{"1min":0.371,"5min":0.36199999999999999,"15min":0.34499999999999997},"max":{"1min":5.3520000000000003,"5min":10.388999999999999,"15min":14.129},"last":0.65500000000000003}]},{"osd":9,"last update":"Fri Feb 9 17:44:39 2024","interfaces":[{"interface":"back","average":{"1min":2.2410000000000001,"5min":1.452,"15min":1.5169999999999999},"min":{"1min":0.499,"5min":0.36199999999999999,"15min":0.32500000000000001},"max":{"1min":11.851000000000001,"5min":11.851000000000001,"15min":11.851000000000001},"last":1.0900000000000001},{"interface":"front","average":{"1min":2.1829999999999998,"5min":1.5209999999999999,"15min":1.423},"min":{"1min":0.44400000000000001,"5min":0.41199999999999998,"15min":0.32800000000000001},"max":{"1min":11.907,"5min":11.907,"15min":12.815},"last":0.996}]},{"osd":10,"last update":"Fri Feb 9 17:44:25 2024","interfaces":[{"interface":"back","average":{"1min":2.1589999999999998,"5min":1.2090000000000001,"15min":0.97699999999999998},"min":{"1min":0.41099999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":9.7080000000000002,"5min":9.7080000000000002,"15min":9.7080000000000002},"last":0.57899999999999996},{"interface":"front","average":{"1min":1.6890000000000001,"5min":1.0249999999999999,"15min":0.84199999999999997},"min":{"1min":0.39100000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":5.2469999999999999,"5min":5.2469999999999999,"15min":5.2469999999999999},"last":0.67000000000000004}]},{"osd":11,"last update":"Fri Feb 9 17:44:14 2024","interfaces":[{"interface":"back","average":{"1min":1.5289999999999999,"5min":1.262,"15min":1.4630000000000001},"min":{"1min":0.439,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":5.9329999999999998,"5min":8.8849999999999998,"15min":14.300000000000001},"last":0.64200000000000002},{"interface":"front","average":{"1min":1.738,"5min":1.5449999999999999,"15min":1.6080000000000001},"min":{"1min":0.48299999999999998,"5min":0.442,"15min":0.39300000000000002},"max":{"1min":5.9160000000000004,"5min":9.4390000000000001,"15min":17.145},"last":0.68799999999999994}]}]},{"osd":4,"up_from":32,"seq":137438953778,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1016244,"kb_used_data":461416,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":92728908,"statfs":{"total":95995035648,"available":94954401792,"internally_reserved":0,"allocated":472489984,"data_stored":472012625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":567867845},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:44:22 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.107,"15min":1.75},"min":{"1min":0.27300000000000002,"5min":0.222,"15min":0.222},"max":{"1min":4.7679999999999998,"5min":7.7199999999999998,"15min":50.356999999999999},"last":7.5519999999999996},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.169,"15min":1.679},"min":{"1min":0.372,"5min":0.26700000000000002,"15min":0.247},"max":{"1min":4.8680000000000003,"5min":7.6390000000000002,"15min":50.569000000000003},"last":7.577}]},{"osd":1,"last update":"Fri Feb 9 17:44:22 2024","interfaces":[{"interface":"back","average":{"1min":1.9299999999999999,"5min":1.548,"15min":1.9870000000000001},"min":{"1min":0.253,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":8.0109999999999992,"5min":8.0109999999999992,"15min":50.473999999999997},"last":17.297999999999998},{"interface":"front","average":{"1min":2.1469999999999998,"5min":1.9410000000000001,"15min":2.1960000000000002},"min":{"1min":0.32800000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":5.4189999999999996,"5min":11.593,"15min":50.499000000000002},"last":7.758}]},{"osd":2,"last update":"Fri Feb 9 17:28:51 2024","interfaces":[{"interface":"back","average":{"1min":2.536,"5min":1.546,"15min":0.91400000000000003},"min":{"1min":0.29999999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":8.8819999999999997,"5min":8.8819999999999997,"15min":8.8819999999999997},"last":17.494},{"interface":"front","average":{"1min":2.214,"5min":1.7330000000000001,"15min":0.97799999999999998},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":9.5039999999999996,"5min":9.5039999999999996,"15min":9.5039999999999996},"last":7.6669999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:44:22 2024","interfaces":[{"interface":"back","average":{"1min":1.752,"5min":1.661,"15min":2.0880000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.245},"max":{"1min":5.383,"5min":11.679,"15min":50.462000000000003},"last":2.29},{"interface":"front","average":{"1min":1.794,"5min":1.3140000000000001,"15min":1.8340000000000001},"min":{"1min":0.39000000000000001,"5min":0.29099999999999998,"15min":0.255},"max":{"1min":6.657,"5min":9.1579999999999995,"15min":50.529000000000003},"last":7.633}]},{"osd":5,"last update":"Fri Feb 9 17:44:54 2024","interfaces":[{"interface":"back","average":{"1min":3.9689999999999999,"5min":2.2589999999999999,"15min":1.802},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":18.887,"5min":18.887,"15min":18.887},"last":18.887},{"interface":"front","average":{"1min":4.0330000000000004,"5min":2.371,"15min":2.0840000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.29099999999999998},"max":{"1min":18.917000000000002,"5min":18.917000000000002,"15min":50.509999999999998},"last":18.917000000000002}]},{"osd":6,"last update":"Fri Feb 9 17:44:22 2024","interfaces":[{"interface":"back","average":{"1min":2.5760000000000001,"5min":1.6080000000000001,"15min":1.6259999999999999},"min":{"1min":0.35999999999999999,"5min":0.28299999999999997,"15min":0.246},"max":{"1min":18.234999999999999,"5min":18.234999999999999,"15min":18.234999999999999},"last":15.51},{"interface":"front","average":{"1min":2.5219999999999998,"5min":2.1949999999999998,"15min":2.016},"min":{"1min":0.316,"5min":0.312,"15min":0.245},"max":{"1min":9.2159999999999993,"5min":13.208,"15min":50.402999999999999},"last":7.8380000000000001}]},{"osd":7,"last update":"Fri Feb 9 17:43:55 2024","interfaces":[{"interface":"back","average":{"1min":1.6699999999999999,"5min":1.6910000000000001,"15min":1.7330000000000001},"min":{"1min":0.377,"5min":0.32200000000000001,"15min":0.28399999999999997},"max":{"1min":6.415,"5min":11.875,"15min":14.919},"last":2.4380000000000002},{"interface":"front","average":{"1min":2.8090000000000002,"5min":1.893,"15min":1.8520000000000001},"min":{"1min":0.44900000000000001,"5min":0.28199999999999997,"15min":0.28100000000000003},"max":{"1min":11.739000000000001,"5min":12.696,"15min":19.047000000000001},"last":7.6020000000000003}]},{"osd":8,"last update":"Fri Feb 9 17:44:54 2024","interfaces":[{"interface":"back","average":{"1min":3.0880000000000001,"5min":2.0129999999999999,"15min":1.7110000000000001},"min":{"1min":0.35299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":17.335999999999999,"5min":17.335999999999999,"15min":17.335999999999999},"last":17.335999999999999},{"interface":"front","average":{"1min":1.853,"5min":1.367,"15min":1.675},"min":{"1min":0.41299999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":7.5099999999999998,"5min":7.5099999999999998,"15min":50.594000000000001},"last":7.5099999999999998}]},{"osd":9,"last update":"Fri Feb 9 17:44:39 2024","interfaces":[{"interface":"back","average":{"1min":2.5739999999999998,"5min":1.855,"15min":1.881},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":10.778,"5min":10.778,"15min":50.442999999999998},"last":0.73599999999999999},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.331,"15min":1.536},"min":{"1min":0.47699999999999998,"5min":0.42499999999999999,"15min":0.36199999999999999},"max":{"1min":5.165,"5min":8.3580000000000005,"15min":10.555},"last":4.8259999999999996}]},{"osd":10,"last update":"Fri Feb 9 17:44:08 2024","interfaces":[{"interface":"back","average":{"1min":2.5760000000000001,"5min":2.012,"15min":2.0960000000000001},"min":{"1min":0.41599999999999998,"5min":0.40300000000000002,"15min":0.39400000000000002},"max":{"1min":10.454000000000001,"5min":10.454000000000001,"15min":51.223999999999997},"last":17.367000000000001},{"interface":"front","average":{"1min":2.6749999999999998,"5min":1.7330000000000001,"15min":2.0739999999999998},"min":{"1min":0.53900000000000003,"5min":0.42899999999999999,"15min":0.39200000000000002},"max":{"1min":11.646000000000001,"5min":11.646000000000001,"15min":50.552},"last":2.3610000000000002}]},{"osd":11,"last update":"Fri Feb 9 17:44:39 2024","interfaces":[{"interface":"back","average":{"1min":1.4610000000000001,"5min":1.6850000000000001,"15min":2.0179999999999998},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.443},"max":{"1min":5.1399999999999997,"5min":9.4060000000000006,"15min":50.484999999999999},"last":7.46},{"interface":"front","average":{"1min":1.8799999999999999,"5min":1.4319999999999999,"15min":1.71},"min":{"1min":0.504,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":7.9859999999999998,"5min":7.9859999999999998,"15min":18.574999999999999},"last":2.3900000000000001}]}]},{"osd":5,"up_from":40,"seq":171798692140,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1010608,"kb_used_data":537960,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":92734544,"statfs":{"total":95995035648,"available":94960173056,"internally_reserved":0,"allocated":550871040,"data_stored":550390539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:44:12 2024","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.256,"15min":1.1200000000000001},"min":{"1min":0.24199999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":8.4700000000000006,"5min":8.4700000000000006,"15min":8.4700000000000006},"last":5.0819999999999999},{"interface":"front","average":{"1min":2.1659999999999999,"5min":1.671,"15min":1.766},"min":{"1min":0.32500000000000001,"5min":0.249,"15min":0.249},"max":{"1min":9.0429999999999993,"5min":9.7650000000000006,"15min":9.7650000000000006},"last":0.42499999999999999}]},{"osd":1,"last update":"Fri Feb 9 17:44:53 2024","interfaces":[{"interface":"back","average":{"1min":1.635,"5min":1.6120000000000001,"15min":1.954},"min":{"1min":0.34899999999999998,"5min":0.24199999999999999,"15min":0.23899999999999999},"max":{"1min":7.0800000000000001,"5min":9.7330000000000005,"15min":48.161999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":1.74,"5min":1.857,"15min":1.907},"min":{"1min":0.28299999999999997,"5min":0.25900000000000001,"15min":0.23599999999999999},"max":{"1min":5.4169999999999998,"5min":7.8369999999999997,"15min":48.218000000000004},"last":5.2910000000000004}]},{"osd":2,"last update":"Fri Feb 9 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.526,"15min":1.526},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":4.718,"5min":4.718,"15min":4.718},"last":3.4950000000000001},{"interface":"front","average":{"1min":2.1269999999999998,"5min":2.1269999999999998,"15min":2.1269999999999998},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":8.5079999999999991,"5min":8.5079999999999991,"15min":8.5079999999999991},"last":5.2599999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:44:53 2024","interfaces":[{"interface":"back","average":{"1min":2.1560000000000001,"5min":1.702,"15min":2.0840000000000001},"min":{"1min":0.35099999999999998,"5min":0.30499999999999999,"15min":0.26500000000000001},"max":{"1min":7.7750000000000004,"5min":11.081,"15min":64.150999999999996},"last":5.2039999999999997},{"interface":"front","average":{"1min":2.0489999999999999,"5min":1.861,"15min":2.1800000000000002},"min":{"1min":0.36699999999999999,"5min":0.35299999999999998,"15min":0.247},"max":{"1min":6.2729999999999997,"5min":12.468,"15min":64.176000000000002},"last":0.5}]},{"osd":4,"last update":"Fri Feb 9 17:44:53 2024","interfaces":[{"interface":"back","average":{"1min":2.4609999999999999,"5min":1.988,"15min":1.887},"min":{"1min":0.29599999999999999,"5min":0.249,"15min":0.23400000000000001},"max":{"1min":9.8620000000000001,"5min":9.8620000000000001,"15min":49.313000000000002},"last":5.1470000000000002},{"interface":"front","average":{"1min":2.1829999999999998,"5min":1.954,"15min":1.853},"min":{"1min":0.26100000000000001,"5min":0.218,"15min":0.218},"max":{"1min":10.198,"5min":10.198,"15min":50.505000000000003},"last":0.46400000000000002}]},{"osd":6,"last update":"Fri Feb 9 17:44:23 2024","interfaces":[{"interface":"back","average":{"1min":2.214,"5min":1.8979999999999999,"15min":1.758},"min":{"1min":0.29899999999999999,"5min":0.28199999999999997,"15min":0.253},"max":{"1min":8.3390000000000004,"5min":12.425000000000001,"15min":15.375999999999999},"last":5.1760000000000002},{"interface":"front","average":{"1min":2.1019999999999999,"5min":1.863,"15min":1.8380000000000001},"min":{"1min":0.308,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":8.4489999999999998,"5min":11.199,"15min":14.250999999999999},"last":5.1200000000000001}]},{"osd":7,"last update":"Fri Feb 9 17:44:53 2024","interfaces":[{"interface":"back","average":{"1min":2.9340000000000002,"5min":2.0859999999999999,"15min":1.8879999999999999},"min":{"1min":0.434,"5min":0.27900000000000003,"15min":0.26700000000000002},"max":{"1min":10.249000000000001,"5min":12.475,"15min":50.545000000000002},"last":4.9189999999999996},{"interface":"front","average":{"1min":3.1120000000000001,"5min":2.2869999999999999,"15min":2.4729999999999999},"min":{"1min":0.40500000000000003,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":13.869,"5min":16.314,"15min":102.526},"last":4.9340000000000002}]},{"osd":8,"last update":"Fri Feb 9 17:43:23 2024","interfaces":[{"interface":"back","average":{"1min":1.6499999999999999,"5min":1.639,"15min":1.6020000000000001},"min":{"1min":0.376,"5min":0.35299999999999998,"15min":0.34899999999999998},"max":{"1min":4.8780000000000001,"5min":14.542,"15min":14.542},"last":5.5439999999999996},{"interface":"front","average":{"1min":3.6699999999999999,"5min":1.702,"15min":1.8300000000000001},"min":{"1min":0.37,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":16.359000000000002,"5min":16.359000000000002,"15min":16.359000000000002},"last":0.435}]},{"osd":9,"last update":"Fri Feb 9 17:44:23 2024","interfaces":[{"interface":"back","average":{"1min":1.778,"5min":1.643,"15min":1.5740000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":9.9510000000000005,"5min":16.411000000000001,"15min":16.411000000000001},"last":5.234},{"interface":"front","average":{"1min":1.8919999999999999,"5min":1.845,"15min":1.6579999999999999},"min":{"1min":0.42899999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":5.827,"5min":16.417999999999999,"15min":16.417999999999999},"last":4.1040000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:44:53 2024","interfaces":[{"interface":"back","average":{"1min":2.452,"5min":2.129,"15min":1.8120000000000001},"min":{"1min":0.48999999999999999,"5min":0.317,"15min":0.317},"max":{"1min":13.958,"5min":16.445,"15min":16.445},"last":3.5270000000000001},{"interface":"front","average":{"1min":1.702,"5min":1.778,"15min":1.798},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":6.1879999999999997,"5min":9.8840000000000003,"15min":14.186999999999999},"last":0.56200000000000006}]},{"osd":11,"last update":"Fri Feb 9 17:44:12 2024","interfaces":[{"interface":"back","average":{"1min":1.9950000000000001,"5min":2.2210000000000001,"15min":1.7649999999999999},"min":{"1min":0.434,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":5.0700000000000003,"5min":16.390000000000001,"15min":16.390000000000001},"last":3.5659999999999998},{"interface":"front","average":{"1min":2.0190000000000001,"5min":1.9690000000000001,"15min":1.758},"min":{"1min":0.53400000000000003,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":5.9660000000000002,"5min":9.8239999999999998,"15min":9.8239999999999998},"last":0.58499999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":575275008,"data_stored":575275008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":309395456,"data_stored":309395456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":221773824,"data_stored":221773824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":84869120,"data_stored":84869120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":460652544,"data_stored":460652544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":544411648,"data_stored":544407576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":231014400,"data_stored":231014400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":345968640,"data_stored":345964568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":179372032,"data_stored":179372032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":211156992,"data_stored":211156992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":418185216,"data_stored":418185216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":308809728,"data_stored":308805656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-09T17:45:01.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:00 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:45:01.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:00 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:45:01.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:00 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:45:01.509 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:45:01.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:01.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:45:01.568 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:45:01.595 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:45:01.614 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:45:01.648 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:01.695 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:45:01.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:01.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:45:01.802 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:45:01.841 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:01.879 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:45:01.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:01.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:45:01.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:01 smithi162 ceph-mon[44304]: pgmap v1409: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 997 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 874 op/s; 2976/47835 objects misplaced (6.221%) 2024-02-09T17:45:01.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:01 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:45:01.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:01 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:45:01.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:01 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:45:01.988 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:45:02.023 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:02.063 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:45:02.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:02.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:45:02.139 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:01 smithi099 ceph-mon[39420]: pgmap v1409: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 997 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 874 op/s; 2976/47835 objects misplaced (6.221%) 2024-02-09T17:45:02.139 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:01 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:45:02.140 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:01 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:45:02.140 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:01 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:45:02.167 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:45:02.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:01 smithi179 ceph-mon[43975]: pgmap v1409: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 997 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 874 op/s; 2976/47835 objects misplaced (6.221%) 2024-02-09T17:45:02.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:01 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:45:02.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:01 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:45:02.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:01 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:45:02.229 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:02.255 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:45:02.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:02.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:45:02.365 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:45:02.417 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:02.459 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:45:02.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:02.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:45:02.567 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:45:02.611 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:03.385 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:45:03.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:03.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:45:03.493 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:45:03.530 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:04.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:03 smithi179 ceph-mon[43975]: pgmap v1410: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 896 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 837 op/s; 2976/42993 objects misplaced (6.922%) 2024-02-09T17:45:04.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:03 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:45:04.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:03 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:45:04.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:03 smithi162 ceph-mon[44304]: pgmap v1410: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 896 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 837 op/s; 2976/42993 objects misplaced (6.922%) 2024-02-09T17:45:04.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:03 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:45:04.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:03 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:45:04.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:03 smithi099 ceph-mon[39420]: pgmap v1410: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 896 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 837 op/s; 2976/42993 objects misplaced (6.922%) 2024-02-09T17:45:04.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:03 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:45:04.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:03 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:45:05.248 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:04 smithi099 ceph-mon[39420]: osdmap e562: 12 total, 12 up, 12 in 2024-02-09T17:45:05.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:04 smithi179 ceph-mon[43975]: osdmap e562: 12 total, 12 up, 12 in 2024-02-09T17:45:05.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:04 smithi162 ceph-mon[44304]: osdmap e562: 12 total, 12 up, 12 in 2024-02-09T17:45:05.613 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] 2024-02-09T17:45:05.613 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:45:05.613 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_3 2024-02-09T17:45:05.613 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph pg dump --format=json 2024-02-09T17:45:05.940 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:05 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:45:05.940 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:05 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:45:05.940 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:05 smithi099 ceph-mon[39420]: pgmap v1412: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 782 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 872 op/s; 2217/37521 objects misplaced (5.909%) 2024-02-09T17:45:05.941 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:05 smithi099 ceph-mon[39420]: osdmap e563: 12 total, 12 up, 12 in 2024-02-09T17:45:06.090 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:45:06.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:06.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:45:06.197 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:45:06.262 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:05 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:45:06.262 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:05 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:45:06.262 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:05 smithi162 ceph-mon[44304]: pgmap v1412: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 782 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 872 op/s; 2217/37521 objects misplaced (5.909%) 2024-02-09T17:45:06.262 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:05 smithi162 ceph-mon[44304]: osdmap e563: 12 total, 12 up, 12 in 2024-02-09T17:45:06.262 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:06.288 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:45:06.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:06.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:45:06.393 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:45:06.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:05 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:45:06.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:05 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:45:06.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:05 smithi179 ceph-mon[43975]: pgmap v1412: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 782 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 872 op/s; 2217/37521 objects misplaced (5.909%) 2024-02-09T17:45:06.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:05 smithi179 ceph-mon[43975]: osdmap e563: 12 total, 12 up, 12 in 2024-02-09T17:45:06.439 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:06.474 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:45:06.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:06.479 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:45:06.580 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:45:06.635 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:06.653 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:45:06.668 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:45:06.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:06.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:45:06.684 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:45:06.710 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:45:06.748 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:45:06.772 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:45:06.807 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:45:06.811 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:06.840 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:45:06.840 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:45:06.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:06.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:45:06.898 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:45:06.923 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:45:06.948 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:45:06.967 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:45:07.005 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:07.009 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:45:07.046 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:45:07.084 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:45:07.087 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:45:07.087 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:45:07.087 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:45:07.087 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:45:07.087 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:45:07.087 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:45:07.087 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:45:07.088 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:45:07.088 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:45:07.088 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:45:07.088 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:45:07.088 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:45:07.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:07 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/591092966' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:45:07.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:07 smithi179 ceph-mon[43975]: pgmap v1414: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 668 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 930 op/s; 2217/32043 objects misplaced (6.919%) 2024-02-09T17:45:07.934 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:07 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/591092966' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:45:07.934 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:07 smithi162 ceph-mon[44304]: pgmap v1414: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 668 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 930 op/s; 2217/32043 objects misplaced (6.919%) 2024-02-09T17:45:07.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:07 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/591092966' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:45:07.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:07 smithi099 ceph-mon[39420]: pgmap v1414: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 668 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 930 op/s; 2217/32043 objects misplaced (6.919%) 2024-02-09T17:45:08.611 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:45:08.645 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is set 2024-02-09T17:45:08.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:08 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1580574097' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:45:08.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:08 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1580574097' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:45:08.977 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:45:08.980 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:08 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1580574097' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:45:08.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:08.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:45:09.083 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:45:09.146 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:09.355 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:45:09.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:09.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:45:09.461 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:45:09.516 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:09.621 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:09 smithi099 ceph-mon[39420]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:45:09.621 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:09 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1580574097' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:45:09.621 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:09 smithi099 ceph-mon[39420]: osdmap e564: 12 total, 12 up, 12 in 2024-02-09T17:45:09.621 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:09 smithi099 ceph-mon[39420]: pgmap v1416: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 601 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 786 op/s; 2217/28824 objects misplaced (7.692%) 2024-02-09T17:45:09.625 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:45:09.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:09.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:45:09.733 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:45:09.773 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:09.807 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:45:09.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:09.813 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:45:09.913 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:45:09.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:09 smithi179 ceph-mon[43975]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:45:09.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:09 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1580574097' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:45:09.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:09 smithi179 ceph-mon[43975]: osdmap e564: 12 total, 12 up, 12 in 2024-02-09T17:45:09.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:09 smithi179 ceph-mon[43975]: pgmap v1416: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 601 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 786 op/s; 2217/28824 objects misplaced (7.692%) 2024-02-09T17:45:09.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:09 smithi162 ceph-mon[44304]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:45:09.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:09 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1580574097' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:45:09.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:09 smithi162 ceph-mon[44304]: osdmap e564: 12 total, 12 up, 12 in 2024-02-09T17:45:09.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:09 smithi162 ceph-mon[44304]: pgmap v1416: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 601 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 786 op/s; 2217/28824 objects misplaced (7.692%) 2024-02-09T17:45:09.948 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:09.973 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:45:09.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:09.978 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:45:10.079 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:45:10.144 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:10.654 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:45:10.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:10 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:45:10.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:10 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:45:10.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:10 smithi179 ceph-mon[43975]: osdmap e565: 12 total, 12 up, 12 in 2024-02-09T17:45:10.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:10 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:45:10.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:10 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:45:10.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:10 smithi162 ceph-mon[44304]: osdmap e565: 12 total, 12 up, 12 in 2024-02-09T17:45:10.930 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:45:10.933 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:10 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:45:10.933 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:10 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:45:10.933 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:10 smithi099 ceph-mon[39420]: osdmap e565: 12 total, 12 up, 12 in 2024-02-09T17:45:10.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:10.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:45:11.035 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:45:11.073 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:11.101 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:45:11.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:11.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:45:11.207 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:45:11.242 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:45:11.242 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:45:11.263 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:11.298 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:45:11.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:11.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:45:11.402 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:45:11.490 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:11.518 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:45:11.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:11.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:45:11.626 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:45:11.661 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:11.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:11 smithi179 ceph-mon[43975]: pgmap v1418: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 466 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 842 op/s; 1389/22344 objects misplaced (6.216%) 2024-02-09T17:45:11.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:11 smithi162 ceph-mon[44304]: pgmap v1418: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 466 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 842 op/s; 1389/22344 objects misplaced (6.216%) 2024-02-09T17:45:11.998 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:11 smithi099 ceph-mon[39420]: pgmap v1418: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 466 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 842 op/s; 1389/22344 objects misplaced (6.216%) 2024-02-09T17:45:12.101 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-09T17:45:12.243 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:45:12.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:12.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:45:12.351 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:45:12.417 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:12.455 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:45:12.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:12.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:45:12.563 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:45:12.591 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:45:12.657 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:12.663 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:45:12.732 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset noscrub 2024-02-09T17:45:12.908 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:45:12.918 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:12.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:45:12.974 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:45:12.993 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-02-09T17:45:13.020 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:45:13.081 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:14.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:13 smithi099 ceph-mon[39420]: pgmap v1419: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 362 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 846 op/s; 1389/17355 objects misplaced (8.003%) 2024-02-09T17:45:14.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:13 smithi099 ceph-mon[39420]: from='client.16089 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:45:14.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:13 smithi179 ceph-mon[43975]: pgmap v1419: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 362 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 846 op/s; 1389/17355 objects misplaced (8.003%) 2024-02-09T17:45:14.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:13 smithi179 ceph-mon[43975]: from='client.16089 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:45:14.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:13 smithi162 ceph-mon[44304]: pgmap v1419: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 362 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 846 op/s; 1389/17355 objects misplaced (8.003%) 2024-02-09T17:45:14.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:13 smithi162 ceph-mon[44304]: from='client.16089 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:45:15.224 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:14 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:45:15.225 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:14 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:45:15.225 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:14 smithi099 ceph-mon[39420]: osdmap e566: 12 total, 12 up, 12 in 2024-02-09T17:45:15.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:14 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:45:15.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:14 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:45:15.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:14 smithi179 ceph-mon[43975]: osdmap e566: 12 total, 12 up, 12 in 2024-02-09T17:45:15.433 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:14 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:45:15.433 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:14 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:45:15.434 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:14 smithi162 ceph-mon[44304]: osdmap e566: 12 total, 12 up, 12 in 2024-02-09T17:45:15.672 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:45:15.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:15.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:45:15.693 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":1419,"stamp":"2024-02-09T17:45:12.870708+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":379519544,"num_objects":5785,"num_object_clones":0,"num_object_copies":17355,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1389,"num_objects_unfound":0,"num_objects_dirty":5785,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":252025,"num_write_kb":8249793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15203,"num_bytes_recovered":996343808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":165196,"ondisk_log_size":165196,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":145,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":6865836,"kb_used_data":1302372,"kb_used_omap":229,"kb_used_meta":5560794,"kb_avail":1118075988,"statfs":{"total":1151940427776,"available":1144909811712,"internally_reserved":0,"allocated":1333628928,"data_stored":1328859518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235127,"internal_metadata":5694253449},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-440532992,"num_objects":-6722,"num_object_clones":0,"num_object_copies":-20166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-828,"num_objects_unfound":0,"num_objects_dirty":-6722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6722,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.943931"},"pg_stats":[{"pgid":"1.1e","version":"63'139","reported_seq":360,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-09T17:45:09.646943+0000","last_change":"2024-02-09T17:43:35.170863+0000","last_active":"2024-02-09T17:45:09.646943+0000","last_peered":"2024-02-09T17:45:09.646943+0000","last_clean":"2024-02-09T17:45:09.646943+0000","last_became_active":"2024-02-09T17:43:19.126757+0000","last_became_peered":"2024-02-09T17:43:19.126757+0000","last_unstale":"2024-02-09T17:45:09.646943+0000","last_undegraded":"2024-02-09T17:45:09.646943+0000","last_fullsized":"2024-02-09T17:45:09.646943+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:35.170716+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:43:35.170716+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:09.993290+0000","scrub_duration":73,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"63'139","reported_seq":385,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-09T17:45:09.652601+0000","last_change":"2024-02-09T17:43:35.227956+0000","last_active":"2024-02-09T17:45:09.652601+0000","last_peered":"2024-02-09T17:45:09.652601+0000","last_clean":"2024-02-09T17:45:09.652601+0000","last_became_active":"2024-02-09T17:43:28.594296+0000","last_became_peered":"2024-02-09T17:43:28.594296+0000","last_unstale":"2024-02-09T17:45:09.652601+0000","last_undegraded":"2024-02-09T17:45:09.652601+0000","last_fullsized":"2024-02-09T17:45:09.652601+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:35.227843+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:43:35.227843+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:11.195670+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"63'139","reported_seq":906,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-09T17:45:09.649310+0000","last_change":"2024-02-09T17:44:48.093391+0000","last_active":"2024-02-09T17:45:09.649310+0000","last_peered":"2024-02-09T17:45:09.649310+0000","last_clean":"2024-02-09T17:45:09.649310+0000","last_became_active":"2024-02-09T17:43:19.103584+0000","last_became_peered":"2024-02-09T17:43:19.103584+0000","last_unstale":"2024-02-09T17:45:09.649310+0000","last_undegraded":"2024-02-09T17:45:09.649310+0000","last_fullsized":"2024-02-09T17:45:09.649310+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:48.093301+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:23.912504+0000","last_clean_scrub_stamp":"2024-02-09T17:44:48.093301+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:59.190873+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"63'139","reported_seq":704,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-09T17:45:09.652208+0000","last_change":"2024-02-09T17:44:44.218695+0000","last_active":"2024-02-09T17:45:09.652208+0000","last_peered":"2024-02-09T17:45:09.652208+0000","last_clean":"2024-02-09T17:45:09.652208+0000","last_became_active":"2024-02-09T17:43:19.102729+0000","last_became_peered":"2024-02-09T17:43:19.102729+0000","last_unstale":"2024-02-09T17:45:09.652208+0000","last_undegraded":"2024-02-09T17:45:09.652208+0000","last_fullsized":"2024-02-09T17:45:09.652208+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:44.218550+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:44:44.218550+0000","last_clean_scrub_stamp":"2024-02-09T17:44:44.218550+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:44.873163+0000","scrub_duration":97,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"63'139","reported_seq":365,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-09T17:45:09.650713+0000","last_change":"2024-02-09T17:43:35.019518+0000","last_active":"2024-02-09T17:45:09.650713+0000","last_peered":"2024-02-09T17:45:09.650713+0000","last_clean":"2024-02-09T17:45:09.650713+0000","last_became_active":"2024-02-09T17:43:19.120814+0000","last_became_peered":"2024-02-09T17:43:19.120814+0000","last_unstale":"2024-02-09T17:45:09.650713+0000","last_undegraded":"2024-02-09T17:45:09.650713+0000","last_fullsized":"2024-02-09T17:45:09.650713+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:35.019416+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:43:35.019416+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:14.192196+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"63'139","reported_seq":1190,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-02-09T17:45:04.930043+0000","last_change":"2024-02-09T17:43:53.009587+0000","last_active":"2024-02-09T17:45:04.930043+0000","last_peered":"2024-02-09T17:45:04.930043+0000","last_clean":"2024-02-09T17:45:04.930043+0000","last_became_active":"2024-02-09T17:43:19.110535+0000","last_became_peered":"2024-02-09T17:43:19.110535+0000","last_unstale":"2024-02-09T17:45:04.930043+0000","last_undegraded":"2024-02-09T17:45:04.930043+0000","last_fullsized":"2024-02-09T17:45:04.930043+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:53.009455+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:30.193194+0000","last_clean_scrub_stamp":"2024-02-09T17:43:53.009455+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:21.095174+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"63'139","reported_seq":1030,"reported_epoch":564,"state":"active+clean","last_fresh":"2024-02-09T17:45:08.625390+0000","last_change":"2024-02-09T17:43:57.370069+0000","last_active":"2024-02-09T17:45:08.625390+0000","last_peered":"2024-02-09T17:45:08.625390+0000","last_clean":"2024-02-09T17:45:08.625390+0000","last_became_active":"2024-02-09T17:43:28.585573+0000","last_became_peered":"2024-02-09T17:43:28.585573+0000","last_unstale":"2024-02-09T17:45:08.625390+0000","last_undegraded":"2024-02-09T17:45:08.625390+0000","last_fullsized":"2024-02-09T17:45:08.625390+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:57.369961+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:43:57.369961+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:10.096778+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"63'139","reported_seq":478,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-09T17:45:09.649454+0000","last_change":"2024-02-09T17:44:50.214834+0000","last_active":"2024-02-09T17:45:09.649454+0000","last_peered":"2024-02-09T17:45:09.649454+0000","last_clean":"2024-02-09T17:45:09.649454+0000","last_became_active":"2024-02-09T17:43:19.109864+0000","last_became_peered":"2024-02-09T17:43:19.109864+0000","last_unstale":"2024-02-09T17:45:09.649454+0000","last_undegraded":"2024-02-09T17:45:09.649454+0000","last_fullsized":"2024-02-09T17:45:09.649454+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:50.214686+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:44:50.214686+0000","last_clean_scrub_stamp":"2024-02-09T17:44:50.214686+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:46:04.216054+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"63'139","reported_seq":368,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-09T17:45:09.652213+0000","last_change":"2024-02-09T17:44:44.395036+0000","last_active":"2024-02-09T17:45:09.652213+0000","last_peered":"2024-02-09T17:45:09.652213+0000","last_clean":"2024-02-09T17:45:09.652213+0000","last_became_active":"2024-02-09T17:43:19.094230+0000","last_became_peered":"2024-02-09T17:43:19.094230+0000","last_unstale":"2024-02-09T17:45:09.652213+0000","last_undegraded":"2024-02-09T17:45:09.652213+0000","last_fullsized":"2024-02-09T17:45:09.652213+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:44.394925+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:11.970338+0000","last_clean_scrub_stamp":"2024-02-09T17:44:44.394925+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:46:09.318945+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"63'139","reported_seq":822,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-09T17:45:09.647264+0000","last_change":"2024-02-09T17:44:37.706632+0000","last_active":"2024-02-09T17:45:09.647264+0000","last_peered":"2024-02-09T17:45:09.647264+0000","last_clean":"2024-02-09T17:45:09.647264+0000","last_became_active":"2024-02-09T17:43:19.138398+0000","last_became_peered":"2024-02-09T17:43:19.138398+0000","last_unstale":"2024-02-09T17:45:09.647264+0000","last_undegraded":"2024-02-09T17:45:09.647264+0000","last_fullsized":"2024-02-09T17:45:09.647264+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:37.706524+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:44:37.706524+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:58.269158+0000","scrub_duration":186,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"63'139","reported_seq":862,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-09T17:45:09.645039+0000","last_change":"2024-02-09T17:44:25.672184+0000","last_active":"2024-02-09T17:45:09.645039+0000","last_peered":"2024-02-09T17:45:09.645039+0000","last_clean":"2024-02-09T17:45:09.645039+0000","last_became_active":"2024-02-09T17:43:28.589307+0000","last_became_peered":"2024-02-09T17:43:28.589307+0000","last_unstale":"2024-02-09T17:45:09.645039+0000","last_undegraded":"2024-02-09T17:45:09.645039+0000","last_fullsized":"2024-02-09T17:45:09.645039+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:25.672084+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:38:26.968892+0000","last_clean_scrub_stamp":"2024-02-09T17:44:25.672084+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:44.241932+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.4","version":"563'17825","reported_seq":27779,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-02-09T17:45:08.106242+0000","last_change":"2024-02-09T17:44:28.850586+0000","last_active":"2024-02-09T17:45:08.106242+0000","last_peered":"2024-02-09T17:45:08.106242+0000","last_clean":"2024-02-09T17:45:08.106242+0000","last_became_active":"2024-02-09T17:43:52.557816+0000","last_became_peered":"2024-02-09T17:43:52.557816+0000","last_unstale":"2024-02-09T17:45:08.106242+0000","last_undegraded":"2024-02-09T17:45:08.106242+0000","last_fullsized":"2024-02-09T17:45:08.106242+0000","mapping_epoch":535,"log_start":"501'7800","ondisk_log_start":"501'7800","created":465,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"547'15824","last_scrub_stamp":"2024-02-09T17:44:28.850416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:44:28.850416+0000","objects_scrubbed":2418,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:49.781140+0000","scrub_duration":552,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26607616,"num_objects":406,"num_object_clones":0,"num_object_copies":1218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15424,"num_write_kb":506560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4756,"num_bytes_recovered":311689216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,5],"acting":[10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"63'139","reported_seq":1463,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-09T17:45:09.647022+0000","last_change":"2024-02-09T17:44:27.625779+0000","last_active":"2024-02-09T17:45:09.647022+0000","last_peered":"2024-02-09T17:45:09.647022+0000","last_clean":"2024-02-09T17:45:09.647022+0000","last_became_active":"2024-02-09T17:41:14.521079+0000","last_became_peered":"2024-02-09T17:41:14.521079+0000","last_unstale":"2024-02-09T17:45:09.647022+0000","last_undegraded":"2024-02-09T17:45:09.647022+0000","last_fullsized":"2024-02-09T17:45:09.647022+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:27.625665+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:44:27.625665+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:44.041378+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"565'15602","reported_seq":25111,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-09T17:45:10.433184+0000","last_change":"2024-02-09T17:43:57.644228+0000","last_active":"2024-02-09T17:45:10.433184+0000","last_peered":"2024-02-09T17:45:10.433184+0000","last_clean":"2024-02-09T17:45:10.433184+0000","last_became_active":"2024-02-09T17:43:56.598131+0000","last_became_peered":"2024-02-09T17:43:56.598131+0000","last_unstale":"2024-02-09T17:45:10.433184+0000","last_undegraded":"2024-02-09T17:45:10.433184+0000","last_fullsized":"2024-02-09T17:45:10.433184+0000","mapping_epoch":537,"log_start":"493'5600","ondisk_log_start":"493'5600","created":465,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'11981","last_scrub_stamp":"2024-02-09T17:43:57.644124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:43:57.644124+0000","objects_scrubbed":3961,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:19.331266+0000","scrub_duration":966,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20840448,"num_objects":318,"num_object_clones":0,"num_object_copies":954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15602,"num_write_kb":509440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4575,"num_bytes_recovered":299827200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"63'139","reported_seq":1027,"reported_epoch":564,"state":"active+clean","last_fresh":"2024-02-09T17:45:08.639611+0000","last_change":"2024-02-09T17:44:49.486253+0000","last_active":"2024-02-09T17:45:08.639611+0000","last_peered":"2024-02-09T17:45:08.639611+0000","last_clean":"2024-02-09T17:45:08.639611+0000","last_became_active":"2024-02-09T17:43:39.981054+0000","last_became_peered":"2024-02-09T17:43:39.981054+0000","last_unstale":"2024-02-09T17:45:08.639611+0000","last_undegraded":"2024-02-09T17:45:08.639611+0000","last_fullsized":"2024-02-09T17:45:08.639611+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:49.486158+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:44:49.486158+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:46:18.726701+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"564'15542","reported_seq":15871,"reported_epoch":564,"state":"active+clean","last_fresh":"2024-02-09T17:45:09.508387+0000","last_change":"2024-02-09T17:44:31.130917+0000","last_active":"2024-02-09T17:45:09.508387+0000","last_peered":"2024-02-09T17:45:09.508387+0000","last_clean":"2024-02-09T17:45:09.508387+0000","last_became_active":"2024-02-09T17:40:48.781472+0000","last_became_peered":"2024-02-09T17:40:48.781472+0000","last_unstale":"2024-02-09T17:45:09.508387+0000","last_undegraded":"2024-02-09T17:45:09.508387+0000","last_fullsized":"2024-02-09T17:45:09.508387+0000","mapping_epoch":465,"log_start":"493'5500","ondisk_log_start":"493'5500","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"548'13663","last_scrub_stamp":"2024-02-09T17:44:31.130800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:44:31.130800+0000","objects_scrubbed":2301,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:42.320512+0000","scrub_duration":699,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26607616,"num_objects":406,"num_object_clones":0,"num_object_copies":1218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15542,"num_write_kb":510336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"63'139","reported_seq":644,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-09T17:45:09.652205+0000","last_change":"2024-02-09T17:43:53.218894+0000","last_active":"2024-02-09T17:45:09.652205+0000","last_peered":"2024-02-09T17:45:09.652205+0000","last_clean":"2024-02-09T17:45:09.652205+0000","last_became_active":"2024-02-09T17:40:13.501711+0000","last_became_peered":"2024-02-09T17:40:13.501711+0000","last_unstale":"2024-02-09T17:45:09.652205+0000","last_undegraded":"2024-02-09T17:45:09.652205+0000","last_fullsized":"2024-02-09T17:45:09.652205+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:53.218764+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:38:27.045816+0000","last_clean_scrub_stamp":"2024-02-09T17:43:53.218764+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:20.464157+0000","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"564'15564","reported_seq":15936,"reported_epoch":564,"state":"active+clean","last_fresh":"2024-02-09T17:45:08.977483+0000","last_change":"2024-02-09T17:44:54.828592+0000","last_active":"2024-02-09T17:45:08.977483+0000","last_peered":"2024-02-09T17:45:08.977483+0000","last_clean":"2024-02-09T17:45:08.977483+0000","last_became_active":"2024-02-09T17:40:48.780579+0000","last_became_peered":"2024-02-09T17:40:48.780579+0000","last_unstale":"2024-02-09T17:45:08.977483+0000","last_undegraded":"2024-02-09T17:45:08.977483+0000","last_fullsized":"2024-02-09T17:45:08.977483+0000","mapping_epoch":465,"log_start":"493'5500","ondisk_log_start":"493'5500","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"559'14793","last_scrub_stamp":"2024-02-09T17:44:54.828378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:44:54.828378+0000","objects_scrubbed":1230,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:59.650438+0000","scrub_duration":358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29491200,"num_objects":450,"num_object_clones":0,"num_object_copies":1350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15564,"num_write_kb":512448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"63'139","reported_seq":957,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-02-09T17:45:04.930215+0000","last_change":"2024-02-09T17:43:54.988847+0000","last_active":"2024-02-09T17:45:04.930215+0000","last_peered":"2024-02-09T17:45:04.930215+0000","last_clean":"2024-02-09T17:45:04.930215+0000","last_became_active":"2024-02-09T17:34:01.342607+0000","last_became_peered":"2024-02-09T17:34:01.342607+0000","last_unstale":"2024-02-09T17:45:04.930215+0000","last_undegraded":"2024-02-09T17:45:04.930215+0000","last_fullsized":"2024-02-09T17:45:04.930215+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:54.988700+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:43:54.988700+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:08.415319+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.2","version":"565'16087","reported_seq":16444,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-09T17:45:11.033192+0000","last_change":"2024-02-09T17:43:46.878627+0000","last_active":"2024-02-09T17:45:11.033192+0000","last_peered":"2024-02-09T17:45:11.033192+0000","last_clean":"2024-02-09T17:45:11.033192+0000","last_became_active":"2024-02-09T17:40:48.781532+0000","last_became_peered":"2024-02-09T17:40:48.781532+0000","last_unstale":"2024-02-09T17:45:11.033192+0000","last_undegraded":"2024-02-09T17:45:11.033192+0000","last_fullsized":"2024-02-09T17:45:11.033192+0000","mapping_epoch":465,"log_start":"495'6000","ondisk_log_start":"495'6000","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"533'11868","last_scrub_stamp":"2024-02-09T17:43:46.878486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:43:46.878486+0000","objects_scrubbed":4573,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:13.358650+0000","scrub_duration":1380,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20512768,"num_objects":313,"num_object_clones":0,"num_object_copies":939,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16087,"num_write_kb":524800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"63'139","reported_seq":968,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-02-09T17:45:04.930359+0000","last_change":"2024-02-09T17:44:40.373866+0000","last_active":"2024-02-09T17:45:04.930359+0000","last_peered":"2024-02-09T17:45:04.930359+0000","last_clean":"2024-02-09T17:45:04.930359+0000","last_became_active":"2024-02-09T17:43:14.024894+0000","last_became_peered":"2024-02-09T17:43:14.024894+0000","last_unstale":"2024-02-09T17:45:04.930359+0000","last_undegraded":"2024-02-09T17:45:04.930359+0000","last_fullsized":"2024-02-09T17:45:04.930359+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:40.373768+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:11.970338+0000","last_clean_scrub_stamp":"2024-02-09T17:44:40.373768+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:53.854883+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.3","version":"564'17036","reported_seq":17441,"reported_epoch":564,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-09T17:45:08.968799+0000","last_change":"2024-02-09T17:44:47.035729+0000","last_active":"2024-02-09T17:45:08.968799+0000","last_peered":"2024-02-09T17:45:08.968799+0000","last_clean":"2024-02-09T17:44:44.589692+0000","last_became_active":"2024-02-09T17:44:46.971273+0000","last_became_peered":"2024-02-09T17:44:46.971273+0000","last_unstale":"2024-02-09T17:45:08.968799+0000","last_undegraded":"2024-02-09T17:45:08.968799+0000","last_fullsized":"2024-02-09T17:45:08.968799+0000","mapping_epoch":557,"log_start":"499'7000","ondisk_log_start":"499'7000","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"529'12566","last_scrub_stamp":"2024-02-09T17:43:36.701823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:43:36.701823+0000","objects_scrubbed":4960,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1352,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30343168,"num_objects":463,"num_object_clones":0,"num_object_copies":1389,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1389,"num_objects_unfound":0,"num_objects_dirty":463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15475,"num_write_kb":510016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[0,5,6],"avail_no_missing":["0","5","6"],"object_location_counts":[{"shards":"0,5,6","objects":463}],"blocked_by":[],"up_primary":4,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"63'139","reported_seq":636,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-09T17:45:09.654650+0000","last_change":"2024-02-09T17:44:37.555304+0000","last_active":"2024-02-09T17:45:09.654650+0000","last_peered":"2024-02-09T17:45:09.654650+0000","last_clean":"2024-02-09T17:45:09.654650+0000","last_became_active":"2024-02-09T17:43:14.038885+0000","last_became_peered":"2024-02-09T17:43:14.038885+0000","last_unstale":"2024-02-09T17:45:09.654650+0000","last_undegraded":"2024-02-09T17:45:09.654650+0000","last_fullsized":"2024-02-09T17:45:09.654650+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:37.555166+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:44:37.555166+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:37.616972+0000","scrub_duration":118,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"563'15968","reported_seq":16315,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-02-09T17:45:08.130067+0000","last_change":"2024-02-09T17:43:52.312478+0000","last_active":"2024-02-09T17:45:08.130067+0000","last_peered":"2024-02-09T17:45:08.130067+0000","last_clean":"2024-02-09T17:45:08.130067+0000","last_became_active":"2024-02-09T17:40:48.780290+0000","last_became_peered":"2024-02-09T17:40:48.780290+0000","last_unstale":"2024-02-09T17:45:08.130067+0000","last_undegraded":"2024-02-09T17:45:08.130067+0000","last_fullsized":"2024-02-09T17:45:08.130067+0000","mapping_epoch":465,"log_start":"494'5900","ondisk_log_start":"494'5900","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"535'12174","last_scrub_stamp":"2024-02-09T17:43:52.312370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:43:52.312370+0000","objects_scrubbed":4242,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:12.514794+0000","scrub_duration":401,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":448,"num_object_clones":0,"num_object_copies":1344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15968,"num_write_kb":525312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"63'139","reported_seq":1043,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-09T17:45:09.646837+0000","last_change":"2024-02-09T17:43:56.105889+0000","last_active":"2024-02-09T17:45:09.646837+0000","last_peered":"2024-02-09T17:45:09.646837+0000","last_clean":"2024-02-09T17:45:09.646837+0000","last_became_active":"2024-02-09T17:33:00.284924+0000","last_became_peered":"2024-02-09T17:33:00.284924+0000","last_unstale":"2024-02-09T17:45:09.646837+0000","last_undegraded":"2024-02-09T17:45:09.646837+0000","last_fullsized":"2024-02-09T17:45:09.646837+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:56.105792+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:56.105792+0000","last_clean_scrub_stamp":"2024-02-09T17:43:56.105792+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:09.993271+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.1","version":"565'15556","reported_seq":15908,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-09T17:45:10.892366+0000","last_change":"2024-02-09T17:43:44.138178+0000","last_active":"2024-02-09T17:45:10.892366+0000","last_peered":"2024-02-09T17:45:10.892366+0000","last_clean":"2024-02-09T17:45:10.892366+0000","last_became_active":"2024-02-09T17:40:49.660617+0000","last_became_peered":"2024-02-09T17:40:49.660617+0000","last_unstale":"2024-02-09T17:45:10.892366+0000","last_undegraded":"2024-02-09T17:45:10.892366+0000","last_fullsized":"2024-02-09T17:45:10.892366+0000","mapping_epoch":465,"log_start":"493'5500","ondisk_log_start":"493'5500","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"533'11388","last_scrub_stamp":"2024-02-09T17:43:44.138002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:43:44.138002+0000","objects_scrubbed":4543,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:14.192163+0000","scrub_duration":1020,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22282240,"num_objects":340,"num_object_clones":0,"num_object_copies":1020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15556,"num_write_kb":508672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"63'139","reported_seq":1014,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-02-09T17:45:04.930298+0000","last_change":"2024-02-09T17:44:41.377536+0000","last_active":"2024-02-09T17:45:04.930298+0000","last_peered":"2024-02-09T17:45:04.930298+0000","last_clean":"2024-02-09T17:45:04.930298+0000","last_became_active":"2024-02-09T17:43:28.597612+0000","last_became_peered":"2024-02-09T17:43:28.597612+0000","last_unstale":"2024-02-09T17:45:04.930298+0000","last_undegraded":"2024-02-09T17:45:04.930298+0000","last_fullsized":"2024-02-09T17:45:04.930298+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:41.377454+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:44:41.377454+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:46:09.607508+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.e","version":"565'15880","reported_seq":16234,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-09T17:45:11.599888+0000","last_change":"2024-02-09T17:44:49.194328+0000","last_active":"2024-02-09T17:45:11.599888+0000","last_peered":"2024-02-09T17:45:11.599888+0000","last_clean":"2024-02-09T17:45:11.599888+0000","last_became_active":"2024-02-09T17:40:48.779366+0000","last_became_peered":"2024-02-09T17:40:48.779366+0000","last_unstale":"2024-02-09T17:45:11.599888+0000","last_undegraded":"2024-02-09T17:45:11.599888+0000","last_fullsized":"2024-02-09T17:45:11.599888+0000","mapping_epoch":465,"log_start":"494'5800","ondisk_log_start":"494'5800","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"558'14691","last_scrub_stamp":"2024-02-09T17:44:49.194186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:44:49.194186+0000","objects_scrubbed":1492,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:49.458648+0000","scrub_duration":317,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19267584,"num_objects":294,"num_object_clones":0,"num_object_copies":882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15880,"num_write_kb":517568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"63'139","reported_seq":965,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-09T17:45:09.647840+0000","last_change":"2024-02-09T17:44:27.890584+0000","last_active":"2024-02-09T17:45:09.647840+0000","last_peered":"2024-02-09T17:45:09.647840+0000","last_clean":"2024-02-09T17:45:09.647840+0000","last_became_active":"2024-02-09T17:43:14.041291+0000","last_became_peered":"2024-02-09T17:43:14.041291+0000","last_unstale":"2024-02-09T17:45:09.647840+0000","last_undegraded":"2024-02-09T17:45:09.647840+0000","last_fullsized":"2024-02-09T17:45:09.647840+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:27.890510+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:44:27.890510+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:53.802187+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.f","version":"564'15715","reported_seq":18434,"reported_epoch":564,"state":"active+clean","last_fresh":"2024-02-09T17:45:09.511650+0000","last_change":"2024-02-09T17:43:56.859129+0000","last_active":"2024-02-09T17:45:09.511650+0000","last_peered":"2024-02-09T17:45:09.511650+0000","last_clean":"2024-02-09T17:45:09.511650+0000","last_became_active":"2024-02-09T17:41:20.876180+0000","last_became_peered":"2024-02-09T17:41:20.876180+0000","last_unstale":"2024-02-09T17:45:09.511650+0000","last_undegraded":"2024-02-09T17:45:09.511650+0000","last_fullsized":"2024-02-09T17:45:09.511650+0000","mapping_epoch":480,"log_start":"494'5700","ondisk_log_start":"494'5700","created":465,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'12028","last_scrub_stamp":"2024-02-09T17:43:56.858923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:43:56.858923+0000","objects_scrubbed":4068,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:25.138111+0000","scrub_duration":562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24707072,"num_objects":377,"num_object_clones":0,"num_object_copies":1131,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15715,"num_write_kb":514944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1182,"num_bytes_recovered":77463552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,11],"acting":[5,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"63'139","reported_seq":807,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-09T17:45:09.647174+0000","last_change":"2024-02-09T17:44:25.176171+0000","last_active":"2024-02-09T17:45:09.647174+0000","last_peered":"2024-02-09T17:45:09.647174+0000","last_clean":"2024-02-09T17:45:09.647174+0000","last_became_active":"2024-02-09T17:43:14.054943+0000","last_became_peered":"2024-02-09T17:43:14.054943+0000","last_unstale":"2024-02-09T17:45:09.647174+0000","last_undegraded":"2024-02-09T17:45:09.647174+0000","last_fullsized":"2024-02-09T17:45:09.647174+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:25.176022+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:44:25.176022+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:31.031846+0000","scrub_duration":158,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"565'15752","reported_seq":16088,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-09T17:45:09.710240+0000","last_change":"2024-02-09T17:43:39.114851+0000","last_active":"2024-02-09T17:45:09.710240+0000","last_peered":"2024-02-09T17:45:09.710240+0000","last_clean":"2024-02-09T17:45:09.710240+0000","last_became_active":"2024-02-09T17:40:48.781384+0000","last_became_peered":"2024-02-09T17:40:48.781384+0000","last_unstale":"2024-02-09T17:45:09.710240+0000","last_undegraded":"2024-02-09T17:45:09.710240+0000","last_fullsized":"2024-02-09T17:45:09.710240+0000","mapping_epoch":465,"log_start":"494'5700","ondisk_log_start":"494'5700","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"531'11356","last_scrub_stamp":"2024-02-09T17:43:39.114709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:43:39.114709+0000","objects_scrubbed":4812,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:12.187358+0000","scrub_duration":1039,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26083328,"num_objects":398,"num_object_clones":0,"num_object_copies":1194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15752,"num_write_kb":516800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"63'139","reported_seq":1021,"reported_epoch":564,"state":"active+clean","last_fresh":"2024-02-09T17:45:08.625474+0000","last_change":"2024-02-09T17:44:34.388682+0000","last_active":"2024-02-09T17:45:08.625474+0000","last_peered":"2024-02-09T17:45:08.625474+0000","last_clean":"2024-02-09T17:45:08.625474+0000","last_became_active":"2024-02-09T17:43:28.586349+0000","last_became_peered":"2024-02-09T17:43:28.586349+0000","last_unstale":"2024-02-09T17:45:08.625474+0000","last_undegraded":"2024-02-09T17:45:08.625474+0000","last_fullsized":"2024-02-09T17:45:08.625474+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:34.388557+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:44:34.388557+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:46:01.046642+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"565'16076","reported_seq":18959,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-09T17:45:10.447806+0000","last_change":"2024-02-09T17:44:29.650288+0000","last_active":"2024-02-09T17:45:10.447806+0000","last_peered":"2024-02-09T17:45:10.447806+0000","last_clean":"2024-02-09T17:45:10.447806+0000","last_became_active":"2024-02-09T17:41:21.499187+0000","last_became_peered":"2024-02-09T17:41:21.499187+0000","last_unstale":"2024-02-09T17:45:10.447806+0000","last_undegraded":"2024-02-09T17:45:10.447806+0000","last_fullsized":"2024-02-09T17:45:10.447806+0000","mapping_epoch":481,"log_start":"495'6000","ondisk_log_start":"495'6000","created":465,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"547'14005","last_scrub_stamp":"2024-02-09T17:44:29.650159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:44:29.650159+0000","objects_scrubbed":2458,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:55.225580+0000","scrub_duration":643,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24051736,"num_objects":368,"num_object_clones":0,"num_object_copies":1104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":368,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16076,"num_write_kb":526145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1248,"num_bytes_recovered":81788928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"63'139","reported_seq":1469,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-09T17:45:09.647438+0000","last_change":"2024-02-09T17:44:45.563490+0000","last_active":"2024-02-09T17:45:09.647438+0000","last_peered":"2024-02-09T17:45:09.647438+0000","last_clean":"2024-02-09T17:45:09.647438+0000","last_became_active":"2024-02-09T17:43:28.590237+0000","last_became_peered":"2024-02-09T17:43:28.590237+0000","last_unstale":"2024-02-09T17:45:09.647438+0000","last_undegraded":"2024-02-09T17:45:09.647438+0000","last_fullsized":"2024-02-09T17:45:09.647438+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:45.563392+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:15.675690+0000","last_clean_scrub_stamp":"2024-02-09T17:44:45.563392+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:46.329929+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"565'15667","reported_seq":20509,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-09T17:45:11.881530+0000","last_change":"2024-02-09T17:44:43.463805+0000","last_active":"2024-02-09T17:45:11.881530+0000","last_peered":"2024-02-09T17:45:11.881530+0000","last_clean":"2024-02-09T17:45:11.881530+0000","last_became_active":"2024-02-09T17:44:39.593392+0000","last_became_peered":"2024-02-09T17:44:39.593392+0000","last_unstale":"2024-02-09T17:45:11.881530+0000","last_undegraded":"2024-02-09T17:45:11.881530+0000","last_fullsized":"2024-02-09T17:45:11.881530+0000","mapping_epoch":552,"log_start":"494'5600","ondisk_log_start":"494'5600","created":465,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'14225","last_scrub_stamp":"2024-02-09T17:44:43.463586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:44:43.463586+0000","objects_scrubbed":1740,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:46:05.557212+0000","scrub_duration":334,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19202048,"num_objects":293,"num_object_clones":0,"num_object_copies":879,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15667,"num_write_kb":510720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2247,"num_bytes_recovered":147259392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,10],"acting":[8,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.e","version":"63'139","reported_seq":934,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-09T17:45:09.650976+0000","last_change":"2024-02-09T17:44:25.525420+0000","last_active":"2024-02-09T17:45:09.650976+0000","last_peered":"2024-02-09T17:45:09.650976+0000","last_clean":"2024-02-09T17:45:09.650976+0000","last_became_active":"2024-02-09T17:43:14.036583+0000","last_became_peered":"2024-02-09T17:43:14.036583+0000","last_unstale":"2024-02-09T17:45:09.650976+0000","last_undegraded":"2024-02-09T17:45:09.650976+0000","last_fullsized":"2024-02-09T17:45:09.650976+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:25.525304+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:14.981453+0000","last_clean_scrub_stamp":"2024-02-09T17:44:25.525304+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:50.128868+0000","scrub_duration":97,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"565'15799","reported_seq":16154,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-09T17:45:11.604566+0000","last_change":"2024-02-09T17:43:42.009150+0000","last_active":"2024-02-09T17:45:11.604566+0000","last_peered":"2024-02-09T17:45:11.604566+0000","last_clean":"2024-02-09T17:45:11.604566+0000","last_became_active":"2024-02-09T17:40:48.783235+0000","last_became_peered":"2024-02-09T17:40:48.783235+0000","last_unstale":"2024-02-09T17:45:11.604566+0000","last_undegraded":"2024-02-09T17:45:11.604566+0000","last_fullsized":"2024-02-09T17:45:11.604566+0000","mapping_epoch":465,"log_start":"494'5700","ondisk_log_start":"494'5700","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"533'11389","last_scrub_stamp":"2024-02-09T17:43:42.009039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:43:42.009039+0000","objects_scrubbed":4727,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:12.752481+0000","scrub_duration":1348,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18808832,"num_objects":287,"num_object_clones":0,"num_object_copies":861,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15799,"num_write_kb":514752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.f","version":"63'139","reported_seq":987,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-09T17:45:09.654553+0000","last_change":"2024-02-09T17:44:42.643784+0000","last_active":"2024-02-09T17:45:09.654553+0000","last_peered":"2024-02-09T17:45:09.654553+0000","last_clean":"2024-02-09T17:45:09.654553+0000","last_became_active":"2024-02-09T17:33:05.316100+0000","last_became_peered":"2024-02-09T17:33:05.316100+0000","last_unstale":"2024-02-09T17:45:09.654553+0000","last_undegraded":"2024-02-09T17:45:09.654553+0000","last_fullsized":"2024-02-09T17:45:09.654553+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:42.643646+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:36:22.364130+0000","last_clean_scrub_stamp":"2024-02-09T17:44:42.643646+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:43.464413+0000","scrub_duration":145,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"565'15896","reported_seq":18648,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-09T17:45:10.427314+0000","last_change":"2024-02-09T17:43:54.002412+0000","last_active":"2024-02-09T17:45:10.427314+0000","last_peered":"2024-02-09T17:45:10.427314+0000","last_clean":"2024-02-09T17:45:10.427314+0000","last_became_active":"2024-02-09T17:41:20.876434+0000","last_became_peered":"2024-02-09T17:41:20.876434+0000","last_unstale":"2024-02-09T17:45:10.427314+0000","last_undegraded":"2024-02-09T17:45:10.427314+0000","last_fullsized":"2024-02-09T17:45:10.427314+0000","mapping_epoch":480,"log_start":"494'5800","ondisk_log_start":"494'5800","created":465,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"536'12114","last_scrub_stamp":"2024-02-09T17:43:54.002296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:43:54.002296+0000","objects_scrubbed":4157,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:15.102179+0000","scrub_duration":854,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23330816,"num_objects":356,"num_object_clones":0,"num_object_copies":1068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15896,"num_write_kb":520064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1195,"num_bytes_recovered":78315520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"63'139","reported_seq":1114,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-09T17:45:09.679080+0000","last_change":"2024-02-09T17:44:28.298713+0000","last_active":"2024-02-09T17:45:09.679080+0000","last_peered":"2024-02-09T17:45:09.679080+0000","last_clean":"2024-02-09T17:45:09.679080+0000","last_became_active":"2024-02-09T17:43:14.027715+0000","last_became_peered":"2024-02-09T17:43:14.027715+0000","last_unstale":"2024-02-09T17:45:09.679080+0000","last_undegraded":"2024-02-09T17:45:09.679080+0000","last_fullsized":"2024-02-09T17:45:09.679080+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:28.298578+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:14.135321+0000","last_clean_scrub_stamp":"2024-02-09T17:44:28.298578+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:31.180345+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"565'16022","reported_seq":16401,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-09T17:45:12.399863+0000","last_change":"2024-02-09T17:44:25.663805+0000","last_active":"2024-02-09T17:45:12.399863+0000","last_peered":"2024-02-09T17:45:12.399863+0000","last_clean":"2024-02-09T17:45:12.399863+0000","last_became_active":"2024-02-09T17:40:49.436285+0000","last_became_peered":"2024-02-09T17:40:49.436285+0000","last_unstale":"2024-02-09T17:45:12.399863+0000","last_undegraded":"2024-02-09T17:45:12.399863+0000","last_fullsized":"2024-02-09T17:45:12.399863+0000","mapping_epoch":465,"log_start":"496'6000","ondisk_log_start":"496'6000","created":465,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"546'13640","last_scrub_stamp":"2024-02-09T17:44:25.663661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:40:47.763277+0000","last_clean_scrub_stamp":"2024-02-09T17:44:25.663661+0000","objects_scrubbed":2677,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:53.939379+0000","scrub_duration":943,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17432576,"num_objects":266,"num_object_clones":0,"num_object_copies":798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16022,"num_write_kb":521216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"63'139","reported_seq":526,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-09T17:45:09.647349+0000","last_change":"2024-02-09T17:44:30.715902+0000","last_active":"2024-02-09T17:45:09.647349+0000","last_peered":"2024-02-09T17:45:09.647349+0000","last_clean":"2024-02-09T17:45:09.647349+0000","last_became_active":"2024-02-09T17:43:14.036759+0000","last_became_peered":"2024-02-09T17:43:14.036759+0000","last_unstale":"2024-02-09T17:45:09.647349+0000","last_undegraded":"2024-02-09T17:45:09.647349+0000","last_fullsized":"2024-02-09T17:45:09.647349+0000","mapping_epoch":454,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:30.715778+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:44:30.715778+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:57.590268+0000","scrub_duration":144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"63'139","reported_seq":927,"reported_epoch":564,"state":"active+clean","last_fresh":"2024-02-09T17:45:08.625566+0000","last_change":"2024-02-09T17:44:38.302668+0000","last_active":"2024-02-09T17:45:08.625566+0000","last_peered":"2024-02-09T17:45:08.625566+0000","last_clean":"2024-02-09T17:45:08.625566+0000","last_became_active":"2024-02-09T17:33:05.310104+0000","last_became_peered":"2024-02-09T17:33:05.310104+0000","last_unstale":"2024-02-09T17:45:08.625566+0000","last_undegraded":"2024-02-09T17:45:08.625566+0000","last_fullsized":"2024-02-09T17:45:08.625566+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:38.302529+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:44:38.302529+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:57.708195+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"63'139","reported_seq":944,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-09T17:45:09.649238+0000","last_change":"2024-02-09T17:44:51.190882+0000","last_active":"2024-02-09T17:45:09.649238+0000","last_peered":"2024-02-09T17:45:09.649238+0000","last_clean":"2024-02-09T17:45:09.649238+0000","last_became_active":"2024-02-09T17:34:01.353171+0000","last_became_peered":"2024-02-09T17:34:01.353171+0000","last_unstale":"2024-02-09T17:45:09.649238+0000","last_undegraded":"2024-02-09T17:45:09.649238+0000","last_fullsized":"2024-02-09T17:45:09.649238+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:51.190789+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:44:51.190789+0000","last_clean_scrub_stamp":"2024-02-09T17:44:51.190789+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:46:12.329895+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.12","version":"63'139","reported_seq":1446,"reported_epoch":564,"state":"active+clean","last_fresh":"2024-02-09T17:45:08.625473+0000","last_change":"2024-02-09T17:43:55.346549+0000","last_active":"2024-02-09T17:45:08.625473+0000","last_peered":"2024-02-09T17:45:08.625473+0000","last_clean":"2024-02-09T17:45:08.625473+0000","last_became_active":"2024-02-09T17:40:13.499071+0000","last_became_peered":"2024-02-09T17:40:13.499071+0000","last_unstale":"2024-02-09T17:45:08.625473+0000","last_undegraded":"2024-02-09T17:45:08.625473+0000","last_fullsized":"2024-02-09T17:45:08.625473+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:43:55.341130+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:43:55.341130+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:10.488649+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"63'139","reported_seq":647,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-09T17:45:09.652255+0000","last_change":"2024-02-09T17:44:27.381453+0000","last_active":"2024-02-09T17:45:09.652255+0000","last_peered":"2024-02-09T17:45:09.652255+0000","last_clean":"2024-02-09T17:45:09.652255+0000","last_became_active":"2024-02-09T17:41:14.521672+0000","last_became_peered":"2024-02-09T17:41:14.521672+0000","last_unstale":"2024-02-09T17:45:09.652255+0000","last_undegraded":"2024-02-09T17:45:09.652255+0000","last_fullsized":"2024-02-09T17:45:09.652255+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:44:27.381332+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:44:27.381332+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:45:35.775807+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":378929176,"num_objects":5783,"num_object_clones":0,"num_object_copies":17349,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1389,"num_objects_unfound":0,"num_objects_dirty":5783,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":252025,"num_write_kb":8249793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15203,"num_bytes_recovered":996343808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1211445248,"data_stored":1211433032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160887,"ondisk_log_size":160887,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056914,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":652280,"kb_used_data":97344,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":93092872,"statfs":{"total":95995035648,"available":95327100928,"internally_reserved":0,"allocated":99680256,"data_stored":99312093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":567867844},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:44:33 2024","interfaces":[{"interface":"back","average":{"1min":2.0990000000000002,"5min":1.2390000000000001,"15min":1.288},"min":{"1min":0.221,"5min":0.221,"15min":0.216},"max":{"1min":5.1989999999999998,"5min":7.4089999999999998,"15min":12.836},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.90600000000000003,"5min":1.0740000000000001,"15min":1.173},"min":{"1min":0.25,"5min":0.25,"15min":0.23000000000000001},"max":{"1min":4.7729999999999997,"5min":9.8219999999999992,"15min":9.8219999999999992},"last":0.95599999999999996}]},{"osd":1,"last update":"Fri Feb 9 17:44:33 2024","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.2889999999999999,"15min":1.331},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.22700000000000001},"max":{"1min":4.2190000000000003,"5min":9.7669999999999995,"15min":12.629},"last":0.28699999999999998},{"interface":"front","average":{"1min":1.9750000000000001,"5min":1.3560000000000001,"15min":1.4430000000000001},"min":{"1min":0.28599999999999998,"5min":0.26500000000000001,"15min":0.26400000000000001},"max":{"1min":5.0750000000000002,"5min":9.5009999999999994,"15min":12.667},"last":0.33800000000000002}]},{"osd":2,"last update":"Fri Feb 9 17:31:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.415,"15min":1.6859999999999999},"min":{"1min":0.378,"5min":0.247,"15min":0.247},"max":{"1min":4.4189999999999996,"5min":9.4719999999999995,"15min":9.4719999999999995},"last":0.35399999999999998},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.5149999999999999,"15min":1.4399999999999999},"min":{"1min":0.378,"5min":0.252,"15min":0.252},"max":{"1min":4.8579999999999997,"5min":7.96,"15min":7.96},"last":0.60399999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:44:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7110000000000001,"5min":1.1459999999999999,"15min":1.2490000000000001},"min":{"1min":0.29699999999999999,"5min":0.27100000000000002,"15min":0.247},"max":{"1min":4.2830000000000004,"5min":9.7289999999999992,"15min":9.7289999999999992},"last":0.38400000000000001},{"interface":"front","average":{"1min":1.7789999999999999,"5min":1.373,"15min":1.5149999999999999},"min":{"1min":0.38800000000000001,"5min":0.33000000000000002,"15min":0.26400000000000001},"max":{"1min":4.2990000000000004,"5min":9.7479999999999993,"15min":12.706},"last":0.46400000000000002}]},{"osd":4,"last update":"Fri Feb 9 17:44:33 2024","interfaces":[{"interface":"back","average":{"1min":1.488,"5min":1.4630000000000001,"15min":1.681},"min":{"1min":0.30599999999999999,"5min":0.246,"15min":0.246},"max":{"1min":4.806,"5min":6.3959999999999999,"15min":21.030999999999999},"last":0.308},{"interface":"front","average":{"1min":2.8439999999999999,"5min":1.575,"15min":1.5569999999999999},"min":{"1min":0.40899999999999997,"5min":0.317,"15min":0.309},"max":{"1min":6.1289999999999996,"5min":7.5490000000000004,"15min":20.876999999999999},"last":0.40100000000000002}]},{"osd":5,"last update":"Fri Feb 9 17:44:33 2024","interfaces":[{"interface":"back","average":{"1min":2.9340000000000002,"5min":1.704,"15min":1.6140000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.312},"max":{"1min":9.0690000000000008,"5min":9.5960000000000001,"15min":13.946999999999999},"last":1.53},{"interface":"front","average":{"1min":2.1680000000000001,"5min":1.585,"15min":1.6839999999999999},"min":{"1min":0.311,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":9.6259999999999994,"5min":9.6259999999999994,"15min":13.414},"last":0.98799999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:44:33 2024","interfaces":[{"interface":"back","average":{"1min":2.7360000000000002,"5min":1.909,"15min":1.726},"min":{"1min":0.39200000000000002,"5min":0.38600000000000001,"15min":0.29899999999999999},"max":{"1min":9.1679999999999993,"5min":12.362,"15min":12.798999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":3.8090000000000002,"5min":2.0609999999999999,"15min":1.8100000000000001},"min":{"1min":0.442,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":10.000999999999999,"5min":10.000999999999999,"15min":10.782999999999999},"last":0.56000000000000005}]},{"osd":7,"last update":"Fri Feb 9 17:44:33 2024","interfaces":[{"interface":"back","average":{"1min":2.798,"5min":1.728,"15min":1.8500000000000001},"min":{"1min":0.42199999999999999,"5min":0.35199999999999998,"15min":0.312},"max":{"1min":9.0969999999999995,"5min":9.7029999999999994,"15min":20.489999999999998},"last":0.54700000000000004},{"interface":"front","average":{"1min":2.1400000000000001,"5min":1.585,"15min":1.7829999999999999},"min":{"1min":0.40300000000000002,"5min":0.35299999999999998,"15min":0.34200000000000003},"max":{"1min":6.8209999999999997,"5min":10.276999999999999,"15min":20.712},"last":5.4470000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:44:57 2024","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.4990000000000001,"15min":1.5600000000000001},"min":{"1min":0.33300000000000002,"5min":0.316,"15min":0.29499999999999998},"max":{"1min":5.3739999999999997,"5min":9.5429999999999993,"15min":13.199999999999999},"last":0.44800000000000001},{"interface":"front","average":{"1min":1.484,"5min":1.6319999999999999,"15min":1.4710000000000001},"min":{"1min":0.42899999999999999,"5min":0.36199999999999999,"15min":0.318},"max":{"1min":5.4139999999999997,"5min":7.6059999999999999,"15min":9.2940000000000005},"last":0.52800000000000002}]},{"osd":9,"last update":"Fri Feb 9 17:44:25 2024","interfaces":[{"interface":"back","average":{"1min":1.738,"5min":2.1190000000000002,"15min":1.544},"min":{"1min":0.39000000000000001,"5min":0.34100000000000003,"15min":0.29399999999999998},"max":{"1min":4.4249999999999998,"5min":62.518000000000001,"15min":62.518000000000001},"last":0.57699999999999996},{"interface":"front","average":{"1min":1.45,"5min":2.044,"15min":1.45},"min":{"1min":0.41099999999999998,"5min":0.35399999999999998,"15min":0.35199999999999998},"max":{"1min":4.8330000000000002,"5min":63.241999999999997,"15min":63.241999999999997},"last":0.49399999999999999}]},{"osd":10,"last update":"Fri Feb 9 17:44:33 2024","interfaces":[{"interface":"back","average":{"1min":2.5459999999999998,"5min":1.6739999999999999,"15min":1.401},"min":{"1min":0.47099999999999997,"5min":0.36799999999999999,"15min":0.36199999999999999},"max":{"1min":5.3799999999999999,"5min":9.6590000000000007,"15min":9.6590000000000007},"last":0.48299999999999998},{"interface":"front","average":{"1min":1.821,"5min":1.613,"15min":1.4299999999999999},"min":{"1min":0.438,"5min":0.39100000000000001,"15min":0.33900000000000002},"max":{"1min":5.5579999999999998,"5min":12.409000000000001,"15min":12.856},"last":0.59599999999999997}]}]},{"osd":10,"up_from":84,"seq":360777253143,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":562324,"kb_used_data":161868,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":93182828,"statfs":{"total":95995035648,"available":95419215872,"internally_reserved":0,"allocated":165752832,"data_stored":165262127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":410057152},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:44:24 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.91500000000000004,"15min":1.45},"min":{"1min":0.25,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":2.7650000000000001,"5min":9.343,"15min":9.343},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.92000000000000004,"15min":1.105},"min":{"1min":0.29199999999999998,"5min":0.253,"15min":0.253},"max":{"1min":4.2699999999999996,"5min":6.3399999999999999,"15min":6.3399999999999999},"last":1.385}]},{"osd":1,"last update":"Fri Feb 9 17:44:53 2024","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":0.84599999999999997,"15min":0.69499999999999995},"min":{"1min":0.28199999999999997,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":7.0709999999999997,"5min":7.0709999999999997,"15min":7.0709999999999997},"last":0.40600000000000003},{"interface":"front","average":{"1min":1.137,"5min":1.127,"15min":0.90100000000000002},"min":{"1min":0.29399999999999998,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":4.3380000000000001,"5min":6.2709999999999999,"15min":6.2709999999999999},"last":0.27400000000000002}]},{"osd":2,"last update":"Fri Feb 9 17:45:04 2024","interfaces":[{"interface":"back","average":{"1min":1.573,"5min":1.056,"15min":1.284},"min":{"1min":0.30599999999999999,"5min":0.27600000000000002,"15min":0.222},"max":{"1min":7.0149999999999997,"5min":7.0149999999999997,"15min":11.384},"last":0.55000000000000004},{"interface":"front","average":{"1min":1.052,"5min":1.026,"15min":1.1639999999999999},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.26100000000000001},"max":{"1min":4.282,"5min":9.0850000000000009,"15min":11.465},"last":0.42599999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:44:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.173,"15min":1.756},"min":{"1min":0.30599999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":3.7559999999999998,"5min":9.1649999999999991,"15min":9.1649999999999991},"last":0.50600000000000001},{"interface":"front","average":{"1min":1.5449999999999999,"5min":1.4379999999999999,"15min":1.883},"min":{"1min":0.39500000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":7.0960000000000001,"5min":8.9670000000000005,"15min":8.9670000000000005},"last":0.39800000000000002}]},{"osd":4,"last update":"Fri Feb 9 17:45:08 2024","interfaces":[{"interface":"back","average":{"1min":2.0590000000000002,"5min":1.4430000000000001,"15min":1.262},"min":{"1min":0.34200000000000003,"5min":0.314,"15min":0.29199999999999998},"max":{"1min":9.0990000000000002,"5min":9.0990000000000002,"15min":9.0990000000000002},"last":0.503},{"interface":"front","average":{"1min":1.7290000000000001,"5min":1.4319999999999999,"15min":1.24},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.29999999999999999},"max":{"1min":7.0309999999999997,"5min":11.273999999999999,"15min":11.273999999999999},"last":0.47599999999999998}]},{"osd":5,"last update":"Fri Feb 9 17:45:04 2024","interfaces":[{"interface":"back","average":{"1min":2.7919999999999998,"5min":1.3720000000000001,"15min":1.3879999999999999},"min":{"1min":0.33700000000000002,"5min":0.29999999999999999,"15min":0.28799999999999998},"max":{"1min":7.1550000000000002,"5min":7.1550000000000002,"15min":9.2349999999999994},"last":0.54600000000000004},{"interface":"front","average":{"1min":2.8599999999999999,"5min":1.8580000000000001,"15min":1.4370000000000001},"min":{"1min":0.434,"5min":0.434,"15min":0.33500000000000002},"max":{"1min":10.243,"5min":10.243,"15min":10.243},"last":0.39200000000000002}]},{"osd":6,"last update":"Fri Feb 9 17:45:04 2024","interfaces":[{"interface":"back","average":{"1min":1.9410000000000001,"5min":1.714,"15min":1.5},"min":{"1min":0.39000000000000001,"5min":0.32400000000000001,"15min":0.29999999999999999},"max":{"1min":5.9260000000000002,"5min":10.962,"15min":10.962},"last":0.48499999999999999},{"interface":"front","average":{"1min":2.7149999999999999,"5min":1.72,"15min":1.589},"min":{"1min":0.501,"5min":0.41099999999999998,"15min":0.35399999999999998},"max":{"1min":9.0690000000000008,"5min":9.0690000000000008,"15min":9.4350000000000005},"last":0.61899999999999999}]},{"osd":7,"last update":"Fri Feb 9 17:39:52 2024","interfaces":[{"interface":"back","average":{"1min":1.6000000000000001,"5min":1.367,"15min":1.415},"min":{"1min":0.42099999999999999,"5min":0.34200000000000003,"15min":0.312},"max":{"1min":4.0430000000000001,"5min":6.7590000000000003,"15min":17.754000000000001},"last":4.6449999999999996},{"interface":"front","average":{"1min":1.498,"5min":1.4359999999999999,"15min":1.5189999999999999},"min":{"1min":0.47399999999999998,"5min":0.38300000000000001,"15min":0.316},"max":{"1min":5.8170000000000002,"5min":11.308999999999999,"15min":11.308999999999999},"last":0.55000000000000004}]},{"osd":8,"last update":"Fri Feb 9 17:44:48 2024","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":1.0940000000000001,"15min":1.1359999999999999},"min":{"1min":0.37,"5min":0.34200000000000003,"15min":0.29099999999999998},"max":{"1min":6.9749999999999996,"5min":6.9749999999999996,"15min":7.4359999999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":1.371,"5min":1.266,"15min":1.0640000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":5.6870000000000003,"5min":6.2009999999999996,"15min":7.7060000000000004},"last":0.57299999999999995}]},{"osd":9,"last update":"Fri Feb 9 17:45:04 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.089,"15min":1.085},"min":{"1min":0.39600000000000002,"5min":0.36599999999999999,"15min":0.34399999999999997},"max":{"1min":7.0419999999999998,"5min":7.0419999999999998,"15min":7.4850000000000003},"last":0.48999999999999999},{"interface":"front","average":{"1min":1.645,"5min":1.389,"15min":1.1950000000000001},"min":{"1min":0.45500000000000002,"5min":0.441,"15min":0.29799999999999999},"max":{"1min":7.1299999999999999,"5min":10.622999999999999,"15min":10.622999999999999},"last":0.51600000000000001}]},{"osd":11,"last update":"Fri Feb 9 17:44:24 2024","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.0589999999999999,"15min":1.016},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":3.4500000000000002,"5min":6.226,"15min":7.383},"last":0.64300000000000002},{"interface":"front","average":{"1min":1.159,"5min":1.1200000000000001,"15min":1.0660000000000001},"min":{"1min":0.47399999999999998,"5min":0.441,"15min":0.377},"max":{"1min":3.484,"5min":6.2590000000000003,"15min":7.3019999999999996},"last":0.69899999999999995}]}]},{"osd":9,"up_from":77,"seq":330712482077,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":520564,"kb_used_data":65596,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":93224588,"statfs":{"total":95995035648,"available":95461978112,"internally_reserved":0,"allocated":67170304,"data_stored":66806213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":465500608},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:44:57 2024","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":0.94599999999999995,"15min":0.70399999999999996},"min":{"1min":0.25800000000000001,"5min":0.246,"15min":0.246},"max":{"1min":3.9239999999999999,"5min":7.2199999999999998,"15min":7.2199999999999998},"last":0.86399999999999999},{"interface":"front","average":{"1min":2.472,"5min":1.3149999999999999,"15min":1.073},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":11.596,"5min":11.596,"15min":11.596},"last":0.69099999999999995}]},{"osd":1,"last update":"Fri Feb 9 17:44:44 2024","interfaces":[{"interface":"back","average":{"1min":1.843,"5min":1.5289999999999999,"15min":1.611},"min":{"1min":0.28299999999999997,"5min":0.23599999999999999,"15min":0.23100000000000001},"max":{"1min":8.1310000000000002,"5min":9.0449999999999999,"15min":89.515000000000001},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.72499999999999998,"5min":1.1870000000000001,"15min":1.5249999999999999},"min":{"1min":0.28100000000000003,"5min":0.249,"15min":0.222},"max":{"1min":1.891,"5min":11.598000000000001,"15min":89.388000000000005},"last":0.36699999999999999}]},{"osd":2,"last update":"Fri Feb 9 17:40:39 2024","interfaces":[{"interface":"back","average":{"1min":1.742,"5min":1.766,"15min":1.514},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":5.6630000000000003,"5min":14.121,"15min":14.121},"last":0.63400000000000001},{"interface":"front","average":{"1min":1.6910000000000001,"5min":1.52,"15min":1.4870000000000001},"min":{"1min":0.34100000000000003,"5min":0.26100000000000001,"15min":0.25600000000000001},"max":{"1min":5.7030000000000003,"5min":8.8040000000000003,"15min":13.69},"last":0.73899999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:44:44 2024","interfaces":[{"interface":"back","average":{"1min":1.6879999999999999,"5min":1.7130000000000001,"15min":1.6060000000000001},"min":{"1min":0.307,"5min":0.26500000000000001,"15min":0.25700000000000001},"max":{"1min":7.9569999999999999,"5min":9.6959999999999997,"15min":40.18},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.999,"5min":1.2829999999999999,"15min":1.5149999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":3.8690000000000002,"5min":9.7140000000000004,"15min":40.158000000000001},"last":0.81899999999999995}]},{"osd":4,"last update":"Fri Feb 9 17:44:44 2024","interfaces":[{"interface":"back","average":{"1min":1.917,"5min":1.6599999999999999,"15min":1.647},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":7.8899999999999997,"5min":9.0920000000000005,"15min":27.381},"last":1.3799999999999999},{"interface":"front","average":{"1min":0.90300000000000002,"5min":1.151,"15min":1.454},"min":{"1min":0.36499999999999999,"5min":0.313,"15min":0.26100000000000001},"max":{"1min":2.5459999999999998,"5min":7.3010000000000002,"15min":27.355},"last":0.312}]},{"osd":5,"last update":"Fri Feb 9 17:44:52 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.3109999999999999,"15min":1.4630000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":4.2839999999999998,"5min":7.367,"15min":10.689},"last":0.45500000000000002},{"interface":"front","average":{"1min":1.7589999999999999,"5min":1.6459999999999999,"15min":1.391},"min":{"1min":0.318,"5min":0.318,"15min":0.311},"max":{"1min":7.2640000000000002,"5min":8.8460000000000001,"15min":13.978999999999999},"last":0.66900000000000004}]},{"osd":6,"last update":"Fri Feb 9 17:44:44 2024","interfaces":[{"interface":"back","average":{"1min":1.982,"5min":1.8109999999999999,"15min":1.5760000000000001},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":7.3049999999999997,"5min":8.4299999999999997,"15min":14.069000000000001},"last":4.431},{"interface":"front","average":{"1min":2.8490000000000002,"5min":2.1320000000000001,"15min":1.702},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":7.3689999999999998,"5min":10.417,"15min":14.097},"last":0.83699999999999997}]},{"osd":7,"last update":"Fri Feb 9 17:44:44 2024","interfaces":[{"interface":"back","average":{"1min":2.5259999999999998,"5min":2.0219999999999998,"15min":1.673},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":8.4770000000000003,"5min":9.125,"15min":9.125},"last":1.6910000000000001},{"interface":"front","average":{"1min":2.5539999999999998,"5min":2.0579999999999998,"15min":1.7390000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.32200000000000001},"max":{"1min":8.5090000000000003,"5min":9.157,"15min":9.157},"last":1.7150000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:45:02 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":1.1830000000000001,"15min":1.2490000000000001},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.9119999999999999,"5min":7.2759999999999998,"15min":16.209},"last":0.40100000000000002},{"interface":"front","average":{"1min":1.833,"5min":1.605,"15min":1.379},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.34799999999999998},"max":{"1min":12.885999999999999,"5min":12.885999999999999,"15min":16.263000000000002},"last":0.79100000000000004}]},{"osd":10,"last update":"Fri Feb 9 17:44:08 2024","interfaces":[{"interface":"back","average":{"1min":2.254,"5min":1.639,"15min":1.323},"min":{"1min":0.35599999999999998,"5min":0.312,"15min":0.312},"max":{"1min":10.206,"5min":10.206,"15min":10.206},"last":0.34899999999999998},{"interface":"front","average":{"1min":1.546,"5min":1.351,"15min":1.1830000000000001},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.38200000000000001},"max":{"1min":10.904,"5min":10.904,"15min":10.904},"last":0.47799999999999998}]},{"osd":11,"last update":"Fri Feb 9 17:44:08 2024","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.5289999999999999,"15min":1.474},"min":{"1min":0.439,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":5.7249999999999996,"5min":13.425000000000001,"15min":14.02},"last":0.879},{"interface":"front","average":{"1min":2.2850000000000001,"5min":1.5629999999999999,"15min":1.381},"min":{"1min":0.48399999999999999,"5min":0.38100000000000001,"15min":0.374},"max":{"1min":10.863,"5min":10.863,"15min":10.863},"last":0.43099999999999999}]}]},{"osd":8,"up_from":171,"seq":734439407820,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297268,"kb_used_data":43260,"kb_used_omap":212,"kb_used_meta":253419,"kb_avail":93447884,"statfs":{"total":95995035648,"available":95690633216,"internally_reserved":0,"allocated":44298240,"data_stored":43934149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217579,"internal_metadata":259501589},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:44:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.751,"15min":0.89200000000000002},"min":{"1min":0.249,"5min":0.217,"15min":0.217},"max":{"1min":2.0640000000000001,"5min":5.827,"15min":8.0920000000000005},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.83799999999999997,"5min":1.2909999999999999,"15min":1.2150000000000001},"min":{"1min":0.25700000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":4.5860000000000003,"5min":7.2949999999999999,"15min":10.256},"last":1.3360000000000001}]},{"osd":1,"last update":"Fri Feb 9 17:44:45 2024","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":1.2729999999999999,"15min":1.286},"min":{"1min":0.252,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.548,"5min":8.718,"15min":8.718},"last":0.44800000000000001},{"interface":"front","average":{"1min":1.272,"5min":1.506,"15min":1.431},"min":{"1min":0.308,"5min":0.25600000000000001,"15min":0.23300000000000001},"max":{"1min":4.2069999999999999,"5min":25.027000000000001,"15min":25.027000000000001},"last":1.2909999999999999}]},{"osd":2,"last update":"Fri Feb 9 17:44:25 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.246,"15min":1.246},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":3.8919999999999999,"5min":3.8919999999999999,"15min":3.8919999999999999},"last":3.6800000000000002},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":3.9470000000000001,"5min":3.9470000000000001,"15min":3.9470000000000001},"last":3.867}]},{"osd":3,"last update":"Fri Feb 9 17:42:38 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":1.542,"15min":1.6379999999999999},"min":{"1min":0.316,"5min":0.29499999999999998,"15min":0.28699999999999998},"max":{"1min":2.347,"5min":15.228999999999999,"15min":24.030999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.75600000000000001,"5min":1.8340000000000001,"15min":1.8640000000000001},"min":{"1min":0.36199999999999999,"5min":0.33600000000000002,"15min":0.30299999999999999},"max":{"1min":2.3719999999999999,"5min":14.988,"15min":24.318999999999999},"last":0.69399999999999995}]},{"osd":4,"last update":"Fri Feb 9 17:44:45 2024","interfaces":[{"interface":"back","average":{"1min":1.8839999999999999,"5min":1.605,"15min":1.655},"min":{"1min":0.35399999999999998,"5min":0.318,"15min":0.27500000000000002},"max":{"1min":7.0369999999999999,"5min":7.9649999999999999,"15min":53.060000000000002},"last":0.33300000000000002},{"interface":"front","average":{"1min":1.667,"5min":1.673,"15min":1.7210000000000001},"min":{"1min":0.31,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":6.9779999999999998,"5min":9.6140000000000008,"15min":53.311999999999998},"last":1.2589999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:44:16 2024","interfaces":[{"interface":"back","average":{"1min":2.4689999999999999,"5min":2.4689999999999999,"15min":2.4689999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":6.4249999999999998,"5min":6.4249999999999998,"15min":6.4249999999999998},"last":4.0099999999999998},{"interface":"front","average":{"1min":2.1779999999999999,"5min":2.1779999999999999,"15min":2.1779999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.383,"5min":6.383,"15min":6.383},"last":4.0730000000000004}]},{"osd":6,"last update":"Fri Feb 9 17:44:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.4510000000000001,"15min":1.5609999999999999},"min":{"1min":0.371,"5min":0.33000000000000002,"15min":0.29799999999999999},"max":{"1min":4.875,"5min":9.5,"15min":37.356999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":2.0659999999999998,"5min":1.637,"15min":1.504},"min":{"1min":0.32000000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":5.7039999999999997,"5min":6.4749999999999996,"15min":37.25},"last":0.47899999999999998}]},{"osd":7,"last update":"Fri Feb 9 17:44:45 2024","interfaces":[{"interface":"back","average":{"1min":2.8620000000000001,"5min":1.921,"15min":1.778},"min":{"1min":0.47599999999999998,"5min":0.39600000000000002,"15min":0.32800000000000001},"max":{"1min":11.278,"5min":11.278,"15min":37.585000000000001},"last":3.8010000000000002},{"interface":"front","average":{"1min":2.895,"5min":1.9590000000000001,"15min":1.804},"min":{"1min":0.502,"5min":0.46200000000000002,"15min":0.312},"max":{"1min":11.313000000000001,"5min":11.313000000000001,"15min":37.918999999999997},"last":3.8359999999999999}]},{"osd":9,"last update":"Fri Feb 9 17:44:45 2024","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":1.579,"15min":1.2290000000000001},"min":{"1min":0.35899999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":4.617,"5min":45.616,"15min":45.616},"last":2.585},{"interface":"front","average":{"1min":0.91400000000000003,"5min":1.51,"15min":1.2829999999999999},"min":{"1min":0.374,"5min":0.314,"15min":0.314},"max":{"1min":3.4239999999999999,"5min":45.718000000000004,"15min":45.718000000000004},"last":2.605}]},{"osd":10,"last update":"Fri Feb 9 17:44:45 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.125,"15min":1.046},"min":{"1min":0.41299999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":7.8360000000000003,"5min":7.8360000000000003,"15min":10.65},"last":0.56899999999999995},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.325,"15min":1.1890000000000001},"min":{"1min":0.433,"5min":0.34799999999999998,"15min":0.318},"max":{"1min":4.9690000000000003,"5min":6.9539999999999997,"15min":11.071},"last":0.40699999999999997}]},{"osd":11,"last update":"Fri Feb 9 17:44:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.1759999999999999,"15min":1.2989999999999999},"min":{"1min":0.46500000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":4.4710000000000001,"5min":7.6959999999999997,"15min":14.433999999999999},"last":2.4180000000000001},{"interface":"front","average":{"1min":1.488,"5min":1.365,"15min":1.258},"min":{"1min":0.54600000000000004,"5min":0.308,"15min":0.308},"max":{"1min":4.444,"5min":7.4359999999999999,"15min":11.622},"last":2.3999999999999999}]}]},{"osd":7,"up_from":57,"seq":244813136166,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":609360,"kb_used_data":76936,"kb_used_omap":1,"kb_used_meta":532350,"kb_avail":93135792,"statfs":{"total":95995035648,"available":95371051008,"internally_reserved":0,"allocated":78782464,"data_stored":78410749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":545126858},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:44:24 2024","interfaces":[{"interface":"back","average":{"1min":1.804,"5min":1.3029999999999999,"15min":2.2850000000000001},"min":{"1min":0.27400000000000002,"5min":0.26000000000000001,"15min":0.252},"max":{"1min":5.9900000000000002,"5min":11.74,"15min":36.753},"last":0.34899999999999998},{"interface":"front","average":{"1min":1.8620000000000001,"5min":1.5,"15min":2.4449999999999998},"min":{"1min":0.27000000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":5.9429999999999996,"5min":11.843999999999999,"15min":36.918999999999997},"last":2.0019999999999998}]},{"osd":1,"last update":"Fri Feb 9 17:44:24 2024","interfaces":[{"interface":"back","average":{"1min":3.1659999999999999,"5min":2.1070000000000002,"15min":2.1200000000000001},"min":{"1min":0.26200000000000001,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":12.318,"5min":15.509,"15min":21.704999999999998},"last":3.0459999999999998},{"interface":"front","average":{"1min":4.0739999999999998,"5min":2.2919999999999998,"15min":2.0670000000000002},"min":{"1min":0.31,"5min":0.27000000000000002,"15min":0.26500000000000001},"max":{"1min":12.359,"5min":15.276,"15min":21.858000000000001},"last":5.6470000000000002}]},{"osd":2,"last update":"Fri Feb 9 17:44:24 2024","interfaces":[{"interface":"back","average":{"1min":2.0059999999999998,"5min":2.0059999999999998,"15min":2.0059999999999998},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":6.6340000000000003,"5min":6.6340000000000003,"15min":6.6340000000000003},"last":3.0800000000000001},{"interface":"front","average":{"1min":2.468,"5min":2.468,"15min":2.468},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":11.968,"5min":11.968,"15min":11.968},"last":2.0470000000000002}]},{"osd":3,"last update":"Fri Feb 9 17:44:43 2024","interfaces":[{"interface":"back","average":{"1min":1.7230000000000001,"5min":1.5349999999999999,"15min":1.7529999999999999},"min":{"1min":0.34000000000000002,"5min":0.28299999999999997,"15min":0.26900000000000002},"max":{"1min":5.9950000000000001,"5min":11.773999999999999,"15min":20.754000000000001},"last":0.41499999999999998},{"interface":"front","average":{"1min":2.6579999999999999,"5min":1.8240000000000001,"15min":1.8939999999999999},"min":{"1min":0.35499999999999998,"5min":0.32600000000000001,"15min":0.26500000000000001},"max":{"1min":6.7199999999999998,"5min":12.393000000000001,"15min":24.457000000000001},"last":3.109}]},{"osd":4,"last update":"Fri Feb 9 17:44:24 2024","interfaces":[{"interface":"back","average":{"1min":2.2450000000000001,"5min":1.3859999999999999,"15min":0.92800000000000005},"min":{"1min":0.30399999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":5.2629999999999999,"5min":5.29,"15min":5.29},"last":2.5249999999999999},{"interface":"front","average":{"1min":1.8540000000000001,"5min":1.6259999999999999,"15min":0.94599999999999995},"min":{"1min":0.27100000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":8.1440000000000001,"5min":11.967000000000001,"15min":11.967000000000001},"last":2.1739999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:44:24 2024","interfaces":[{"interface":"back","average":{"1min":3.1040000000000001,"5min":1.8999999999999999,"15min":1.0620000000000001},"min":{"1min":0.371,"5min":0.249,"15min":0.249},"max":{"1min":9.2170000000000005,"5min":9.6590000000000007,"15min":9.6590000000000007},"last":2.3599999999999999},{"interface":"front","average":{"1min":3.3460000000000001,"5min":2.0750000000000002,"15min":1.03},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":8.9600000000000009,"5min":15.228999999999999,"15min":15.228999999999999},"last":7.8109999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:44:43 2024","interfaces":[{"interface":"back","average":{"1min":3.2650000000000001,"5min":2.238,"15min":1.9990000000000001},"min":{"1min":0.35199999999999998,"5min":0.28100000000000003,"15min":0.26000000000000001},"max":{"1min":8.9969999999999999,"5min":15.275,"15min":25.965},"last":3.133},{"interface":"front","average":{"1min":2.6280000000000001,"5min":1.9099999999999999,"15min":1.7},"min":{"1min":0.44400000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":8.9779999999999998,"5min":18.236999999999998,"15min":18.236999999999998},"last":2.0779999999999998}]},{"osd":8,"last update":"Fri Feb 9 17:44:43 2024","interfaces":[{"interface":"back","average":{"1min":1.7050000000000001,"5min":1.655,"15min":1.524},"min":{"1min":0.48299999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":5.8109999999999999,"5min":11.81,"15min":11.81},"last":0.46000000000000002},{"interface":"front","average":{"1min":2.5510000000000002,"5min":1.8029999999999999,"15min":1.554},"min":{"1min":0.38700000000000001,"5min":0.36599999999999999,"15min":0.33300000000000002},"max":{"1min":6.7569999999999997,"5min":12.058,"15min":12.058},"last":3.1600000000000001}]},{"osd":9,"last update":"Fri Feb 9 17:44:34 2024","interfaces":[{"interface":"back","average":{"1min":2.2639999999999998,"5min":1.736,"15min":1.639},"min":{"1min":0.44,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":8.0020000000000007,"5min":11.941000000000001,"15min":11.941000000000001},"last":5.0229999999999997},{"interface":"front","average":{"1min":2.54,"5min":1.9059999999999999,"15min":1.752},"min":{"1min":0.53700000000000003,"5min":0.379,"15min":0.33400000000000002},"max":{"1min":8.109,"5min":11.906000000000001,"15min":11.906000000000001},"last":0.48699999999999999}]},{"osd":10,"last update":"Fri Feb 9 17:44:58 2024","interfaces":[{"interface":"back","average":{"1min":2.1560000000000001,"5min":1.7450000000000001,"15min":1.6240000000000001},"min":{"1min":0.51500000000000001,"5min":0.38200000000000001,"15min":0.35099999999999998},"max":{"1min":7.165,"5min":9.3130000000000006,"15min":12.050000000000001},"last":2.2970000000000002},{"interface":"front","average":{"1min":2.286,"5min":1.9330000000000001,"15min":1.7629999999999999},"min":{"1min":0.57299999999999995,"5min":0.38900000000000001,"15min":0.35599999999999998},"max":{"1min":5.2000000000000002,"5min":11.878,"15min":11.878},"last":1.8100000000000001}]},{"osd":11,"last update":"Fri Feb 9 17:44:24 2024","interfaces":[{"interface":"back","average":{"1min":2.5790000000000002,"5min":1.7130000000000001,"15min":1.8779999999999999},"min":{"1min":0.48099999999999998,"5min":0.42399999999999999,"15min":0.39500000000000002},"max":{"1min":5.9089999999999998,"5min":9.2490000000000006,"15min":21.945},"last":7.2119999999999997},{"interface":"front","average":{"1min":3.4300000000000002,"5min":1.8520000000000001,"15min":1.796},"min":{"1min":0.50700000000000001,"5min":0.45500000000000002,"15min":0.39500000000000002},"max":{"1min":12.423999999999999,"5min":12.423999999999999,"15min":21.731999999999999},"last":2.3330000000000002}]}]},{"osd":6,"up_from":49,"seq":210453397803,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":490308,"kb_used_data":71676,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":93254844,"statfs":{"total":95995035648,"available":95492960256,"internally_reserved":0,"allocated":73396224,"data_stored":73032133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":428538310},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:44:15 2024","interfaces":[{"interface":"back","average":{"1min":2.1509999999999998,"5min":1.776,"15min":1.651},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.21299999999999999},"max":{"1min":5.7619999999999996,"5min":9.2089999999999996,"15min":12.544},"last":0.50800000000000001},{"interface":"front","average":{"1min":2.0289999999999999,"5min":1.825,"15min":1.599},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.217},"max":{"1min":5.8289999999999997,"5min":9.2520000000000007,"15min":9.2520000000000007},"last":0.56200000000000006}]},{"osd":1,"last update":"Fri Feb 9 17:44:15 2024","interfaces":[{"interface":"back","average":{"1min":2.2610000000000001,"5min":1.7549999999999999,"15min":1.895},"min":{"1min":0.316,"5min":0.28599999999999998,"15min":0.25900000000000001},"max":{"1min":6.1849999999999996,"5min":8.3260000000000005,"15min":17.762},"last":7.0380000000000003},{"interface":"front","average":{"1min":2.4430000000000001,"5min":1.8580000000000001,"15min":1.9670000000000001},"min":{"1min":0.34799999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":12.475,"5min":12.475,"15min":17.734999999999999},"last":7.0670000000000002}]},{"osd":2,"last update":"Fri Feb 9 17:44:15 2024","interfaces":[{"interface":"back","average":{"1min":2.097,"5min":1.6899999999999999,"15min":1.579},"min":{"1min":0.40200000000000002,"5min":0.32000000000000001,"15min":0.22600000000000001},"max":{"1min":5.9470000000000001,"5min":9.8019999999999996,"15min":9.8019999999999996},"last":6.9669999999999996},{"interface":"front","average":{"1min":2.145,"5min":1.7290000000000001,"15min":1.6379999999999999},"min":{"1min":0.433,"5min":0.372,"15min":0.28000000000000003},"max":{"1min":5.9640000000000004,"5min":9.8439999999999994,"15min":10.156000000000001},"last":7.0039999999999996}]},{"osd":3,"last update":"Fri Feb 9 17:44:15 2024","interfaces":[{"interface":"back","average":{"1min":1.9430000000000001,"5min":1.7030000000000001,"15min":1.9930000000000001},"min":{"1min":0.29599999999999999,"5min":0.27800000000000002,"15min":0.25600000000000001},"max":{"1min":13.016,"5min":13.016,"15min":32.545000000000002},"last":2.2519999999999998},{"interface":"front","average":{"1min":2.2250000000000001,"5min":1.734,"15min":1.974},"min":{"1min":0.40000000000000002,"5min":0.34200000000000003,"15min":0.29599999999999999},"max":{"1min":6.234,"5min":9.8219999999999992,"15min":32.670999999999999},"last":4.6299999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:44:15 2024","interfaces":[{"interface":"back","average":{"1min":3.2989999999999999,"5min":2.0070000000000001,"15min":1.78},"min":{"1min":0.36799999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":15.061999999999999,"5min":15.061999999999999,"15min":19.798999999999999},"last":4.5549999999999997},{"interface":"front","average":{"1min":3.915,"5min":2.4049999999999998,"15min":1.796},"min":{"1min":0.35099999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":15.118,"5min":15.118,"15min":15.118},"last":0.46500000000000002}]},{"osd":5,"last update":"Fri Feb 9 17:44:15 2024","interfaces":[{"interface":"back","average":{"1min":2.9529999999999998,"5min":2.024,"15min":1.7390000000000001},"min":{"1min":0.47199999999999998,"5min":0.29799999999999999,"15min":0.29499999999999998},"max":{"1min":7.8689999999999998,"5min":9.9849999999999994,"15min":10.968999999999999},"last":7.0880000000000001},{"interface":"front","average":{"1min":3.1560000000000001,"5min":1.8400000000000001,"15min":1.7130000000000001},"min":{"1min":0.39400000000000002,"5min":0.316,"15min":0.30199999999999999},"max":{"1min":11.079000000000001,"5min":11.079000000000001,"15min":13.91},"last":6.2069999999999999}]},{"osd":7,"last update":"Fri Feb 9 17:44:55 2024","interfaces":[{"interface":"back","average":{"1min":2.2130000000000001,"5min":2.226,"15min":1.8560000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.28000000000000003},"max":{"1min":7.633,"5min":11.052,"15min":14.022},"last":3.7690000000000001},{"interface":"front","average":{"1min":2.1680000000000001,"5min":1.905,"15min":1.9199999999999999},"min":{"1min":0.38400000000000001,"5min":0.314,"15min":0.314},"max":{"1min":9.0310000000000006,"5min":12.967000000000001,"15min":15.458},"last":0.58899999999999997}]},{"osd":8,"last update":"Fri Feb 9 17:26:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":0.876,"15min":0.79600000000000004},"min":{"1min":0.39900000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":4.0339999999999998,"5min":4.4880000000000004,"15min":4.4880000000000004},"last":4.2809999999999997},{"interface":"front","average":{"1min":1.0469999999999999,"5min":0.88600000000000001,"15min":0.78300000000000003},"min":{"1min":0.375,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":3.6539999999999999,"5min":4.8570000000000002,"15min":4.8570000000000002},"last":2.9529999999999998}]},{"osd":9,"last update":"Fri Feb 9 17:44:20 2024","interfaces":[{"interface":"back","average":{"1min":2.153,"5min":1.9410000000000001,"15min":1.5209999999999999},"min":{"1min":0.58899999999999997,"5min":0.438,"15min":0.35899999999999999},"max":{"1min":5.2599999999999998,"5min":10.779999999999999,"15min":10.779999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":2.4580000000000002,"5min":2.024,"15min":1.641},"min":{"1min":0.495,"5min":0.40999999999999998,"15min":0.40400000000000003},"max":{"1min":7.5380000000000003,"5min":7.5380000000000003,"15min":8.1630000000000003},"last":0.61299999999999999}]},{"osd":10,"last update":"Fri Feb 9 17:44:55 2024","interfaces":[{"interface":"back","average":{"1min":1.7809999999999999,"5min":2.1000000000000001,"15min":1.6439999999999999},"min":{"1min":0.36099999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":8.2690000000000001,"5min":25.370000000000001,"15min":25.370000000000001},"last":2.96},{"interface":"front","average":{"1min":1.5649999999999999,"5min":1.907,"15min":1.704},"min":{"1min":0.39800000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":7.351,"5min":26.027999999999999,"15min":26.027999999999999},"last":4.5979999999999999}]},{"osd":11,"last update":"Fri Feb 9 17:44:15 2024","interfaces":[{"interface":"back","average":{"1min":2.3839999999999999,"5min":1.9650000000000001,"15min":1.661},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.36099999999999999},"max":{"1min":13.109,"5min":13.109,"15min":13.109},"last":2.1680000000000001},{"interface":"front","average":{"1min":2.0779999999999998,"5min":1.8480000000000001,"15min":1.639},"min":{"1min":0.38800000000000001,"5min":0.373,"15min":0.373},"max":{"1min":13.164999999999999,"5min":13.164999999999999,"15min":13.164999999999999},"last":2.2170000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575173,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":704548,"kb_used_data":109476,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":93040604,"statfs":{"total":95995035648,"available":95273578496,"internally_reserved":0,"allocated":112103424,"data_stored":111677224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:44:33 2024","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.94499999999999995,"15min":1.484},"min":{"1min":0.28000000000000003,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":4.8029999999999999,"5min":7.0220000000000002,"15min":19.079000000000001},"last":4.7130000000000001},{"interface":"front","average":{"1min":1.3280000000000001,"5min":0.96899999999999997,"15min":1.4379999999999999},"min":{"1min":0.29399999999999998,"5min":0.218,"15min":0.19400000000000001},"max":{"1min":5.125,"5min":8.8100000000000005,"15min":23.573},"last":4.6360000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:44:57 2024","interfaces":[{"interface":"back","average":{"1min":2.0910000000000002,"5min":1.1799999999999999,"15min":1.524},"min":{"1min":0.32600000000000001,"5min":0.23400000000000001,"15min":0.22600000000000001},"max":{"1min":6.2050000000000001,"5min":9.5609999999999999,"15min":26.361999999999998},"last":6.7069999999999999},{"interface":"front","average":{"1min":1.7789999999999999,"5min":1.238,"15min":1.619},"min":{"1min":0.32500000000000001,"5min":0.27100000000000002,"15min":0.22800000000000001},"max":{"1min":6.2190000000000003,"5min":6.2610000000000001,"15min":26.087},"last":3.4889999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:44:18 2024","interfaces":[{"interface":"back","average":{"1min":1.548,"5min":1.548,"15min":1.548},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":6.7649999999999997,"5min":6.7649999999999997,"15min":6.7649999999999997},"last":0.53600000000000003},{"interface":"front","average":{"1min":1.012,"5min":1.012,"15min":1.012},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":4.4790000000000001,"5min":4.4790000000000001,"15min":4.4790000000000001},"last":4.5670000000000002}]},{"osd":4,"last update":"Fri Feb 9 17:44:27 2024","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.476,"15min":1.8140000000000001},"min":{"1min":0.32700000000000001,"5min":0.27700000000000002,"15min":0.253},"max":{"1min":3.907,"5min":6.7679999999999998,"15min":11.766999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":2.8130000000000002,"5min":1.7809999999999999,"15min":1.9339999999999999},"min":{"1min":0.35099999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":13.871,"5min":14.593,"15min":14.593},"last":15.423}]},{"osd":5,"last update":"Fri Feb 9 17:44:13 2024","interfaces":[{"interface":"back","average":{"1min":1.5840000000000001,"5min":1.6970000000000001,"15min":1.8069999999999999},"min":{"1min":0.35899999999999999,"5min":0.29199999999999998,"15min":0.27000000000000002},"max":{"1min":6.2850000000000001,"5min":6.8250000000000002,"15min":13.544},"last":0.57499999999999996},{"interface":"front","average":{"1min":1.6739999999999999,"5min":1.4970000000000001,"15min":1.7470000000000001},"min":{"1min":0.40500000000000003,"5min":0.32100000000000001,"15min":0.26800000000000002},"max":{"1min":6.609,"5min":10.263999999999999,"15min":12.117000000000001},"last":3.3959999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:44:18 2024","interfaces":[{"interface":"back","average":{"1min":2.387,"5min":1.5820000000000001,"15min":1.3759999999999999},"min":{"1min":0.377,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":12.57,"5min":12.57,"15min":12.57},"last":3.5299999999999998},{"interface":"front","average":{"1min":2.4369999999999998,"5min":1.5660000000000001,"15min":1.3049999999999999},"min":{"1min":0.51700000000000002,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":12.605,"5min":12.605,"15min":12.605},"last":3.448}]},{"osd":7,"last update":"Fri Feb 9 17:44:18 2024","interfaces":[{"interface":"back","average":{"1min":1.8460000000000001,"5min":1.391,"15min":1.2350000000000001},"min":{"1min":0.45200000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":6.5830000000000002,"5min":6.5830000000000002,"15min":6.5830000000000002},"last":3.4780000000000002},{"interface":"front","average":{"1min":1.8859999999999999,"5min":1.552,"15min":1.3420000000000001},"min":{"1min":0.48199999999999998,"5min":0.309,"15min":0.309},"max":{"1min":6.5599999999999996,"5min":6.5599999999999996,"15min":6.5599999999999996},"last":3.5009999999999999}]},{"osd":8,"last update":"Fri Feb 9 17:44:27 2024","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":0.96599999999999997,"15min":0.91600000000000004},"min":{"1min":0.372,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":3.3399999999999999,"5min":9.2520000000000007,"15min":9.2520000000000007},"last":3.6179999999999999},{"interface":"front","average":{"1min":1.2450000000000001,"5min":0.97899999999999998,"15min":0.89300000000000002},"min":{"1min":0.42899999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":3.367,"5min":9.2810000000000006,"15min":9.2810000000000006},"last":3.6469999999999998}]},{"osd":9,"last update":"Fri Feb 9 17:44:33 2024","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":1.111,"15min":1.4430000000000001},"min":{"1min":0.39200000000000002,"5min":0.33900000000000002,"15min":0.32400000000000001},"max":{"1min":4.7039999999999997,"5min":6.8819999999999997,"15min":9.5839999999999996},"last":3.5449999999999999},{"interface":"front","average":{"1min":1.395,"5min":1.4810000000000001,"15min":1.5660000000000001},"min":{"1min":0.44400000000000001,"5min":0.375,"15min":0.32900000000000001},"max":{"1min":4.4569999999999999,"5min":8.7780000000000005,"15min":11.366},"last":0.63400000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:44:13 2024","interfaces":[{"interface":"back","average":{"1min":1.482,"5min":1.5429999999999999,"15min":1.6080000000000001},"min":{"1min":0.55800000000000005,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":6.1769999999999996,"5min":15.083,"15min":15.083},"last":3.6749999999999998},{"interface":"front","average":{"1min":1.6160000000000001,"5min":1.605,"15min":1.702},"min":{"1min":0.61499999999999999,"5min":0.46700000000000003,"15min":0.40899999999999997},"max":{"1min":6.2140000000000004,"5min":15.121,"15min":15.121},"last":3.7010000000000001}]},{"osd":11,"last update":"Fri Feb 9 17:44:18 2024","interfaces":[{"interface":"back","average":{"1min":1.8320000000000001,"5min":1.2609999999999999,"15min":1.0529999999999999},"min":{"1min":0.441,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":6.5449999999999999,"5min":6.5449999999999999,"15min":6.5449999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.2210000000000001,"15min":1.224},"min":{"1min":0.63200000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":3.2999999999999998,"5min":3.657,"15min":3.657},"last":3.5640000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738697,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":841812,"kb_used_data":215052,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":92903340,"statfs":{"total":95995035648,"available":95133020160,"internally_reserved":0,"allocated":220213248,"data_stored":219722543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 9 17:44:13 2024","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.2889999999999999,"15min":1.3540000000000001},"min":{"1min":0.217,"5min":0.20599999999999999,"15min":0.20499999999999999},"max":{"1min":4.7530000000000001,"5min":13.842000000000001,"15min":13.842000000000001},"last":0.35299999999999998},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.411,"15min":1.5009999999999999},"min":{"1min":0.28299999999999997,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":4.7889999999999997,"5min":14.122,"15min":14.122},"last":0.45000000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:39:25 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":5.5339999999999998,"5min":5.5339999999999998,"15min":5.5339999999999998},"last":0.377},{"interface":"front","average":{"1min":1.593,"5min":1.593,"15min":1.593},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":13.473000000000001,"5min":13.473000000000001,"15min":13.473000000000001},"last":1.583}]},{"osd":3,"last update":"Fri Feb 9 17:44:56 2024","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":1.2849999999999999,"15min":2.4430000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":4.3540000000000001,"5min":16.756,"15min":16.756},"last":0.40999999999999998},{"interface":"front","average":{"1min":2.2370000000000001,"5min":1.3500000000000001,"15min":1.6539999999999999},"min":{"1min":0.24099999999999999,"5min":0.217,"15min":0.217},"max":{"1min":7.4500000000000002,"5min":7.4500000000000002,"15min":7.4500000000000002},"last":1.6319999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:44:37 2024","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.2330000000000001,"15min":1.5209999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.26400000000000001},"max":{"1min":4.5620000000000003,"5min":7.7830000000000004,"15min":14.419},"last":1.5720000000000001},{"interface":"front","average":{"1min":1.54,"5min":1.5129999999999999,"15min":1.7490000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.27200000000000002},"max":{"1min":4.5330000000000004,"5min":7.8230000000000004,"15min":14.465999999999999},"last":1.6080000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:44:56 2024","interfaces":[{"interface":"back","average":{"1min":2.492,"5min":1.49,"15min":1.5980000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.29499999999999998},"max":{"1min":6.476,"5min":7.9690000000000003,"15min":9.6329999999999991},"last":1.792},{"interface":"front","average":{"1min":2.5379999999999998,"5min":1.478,"15min":1.6899999999999999},"min":{"1min":0.374,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":6.5110000000000001,"5min":7.9950000000000001,"15min":9.6660000000000004},"last":1.839}]},{"osd":6,"last update":"Fri Feb 9 17:44:24 2024","interfaces":[{"interface":"back","average":{"1min":1.837,"5min":1.466,"15min":1.706},"min":{"1min":0.38500000000000001,"5min":0.28699999999999998,"15min":0.27800000000000002},"max":{"1min":6.4489999999999998,"5min":6.4489999999999998,"15min":13.407},"last":3.0600000000000001},{"interface":"front","average":{"1min":1.8700000000000001,"5min":1.51,"15min":1.772},"min":{"1min":0.42199999999999999,"5min":0.33300000000000002,"15min":0.33000000000000002},"max":{"1min":6.4749999999999996,"5min":6.4749999999999996,"15min":13.448},"last":3.089}]},{"osd":7,"last update":"Fri Feb 9 17:44:50 2024","interfaces":[{"interface":"back","average":{"1min":2.5710000000000002,"5min":1.8999999999999999,"15min":2.0710000000000002},"min":{"1min":0.34899999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":11.872999999999999,"5min":11.872999999999999,"15min":13.349},"last":0.96399999999999997},{"interface":"front","average":{"1min":2.2360000000000002,"5min":1.869,"15min":1.99},"min":{"1min":0.34999999999999998,"5min":0.33300000000000002,"15min":0.28699999999999998},"max":{"1min":6.3460000000000001,"5min":9.6980000000000004,"15min":14.451000000000001},"last":1.4470000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:44:50 2024","interfaces":[{"interface":"back","average":{"1min":1.494,"5min":1.5029999999999999,"15min":1.508},"min":{"1min":0.39600000000000002,"5min":0.33900000000000002,"15min":0.33100000000000002},"max":{"1min":5.4770000000000003,"5min":9.5359999999999996,"15min":14.323},"last":1.496},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.532,"15min":1.633},"min":{"1min":0.36299999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":5.5780000000000003,"5min":9.5700000000000003,"15min":14.481999999999999},"last":1.4219999999999999}]},{"osd":9,"last update":"Fri Feb 9 17:44:37 2024","interfaces":[{"interface":"back","average":{"1min":2.2189999999999999,"5min":2.1789999999999998,"15min":1.788},"min":{"1min":0.42299999999999999,"5min":0.33800000000000002,"15min":0.32500000000000001},"max":{"1min":14.903,"5min":17.952000000000002,"15min":17.952000000000002},"last":1.5269999999999999},{"interface":"front","average":{"1min":1.746,"5min":2.1459999999999999,"15min":1.768},"min":{"1min":0.44400000000000001,"5min":0.40999999999999998,"15min":0.34799999999999998},"max":{"1min":9.7170000000000005,"5min":17.913,"15min":17.913},"last":1.4670000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:44:56 2024","interfaces":[{"interface":"back","average":{"1min":2.02,"5min":1.9970000000000001,"15min":1.8520000000000001},"min":{"1min":0.46800000000000003,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":5.4820000000000002,"5min":31.766999999999999,"15min":31.766999999999999},"last":1.5589999999999999},{"interface":"front","average":{"1min":1.8069999999999999,"5min":1.4630000000000001,"15min":1.431},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":13.148999999999999,"5min":28.048999999999999,"15min":28.048999999999999},"last":0.56999999999999995}]},{"osd":11,"last update":"Fri Feb 9 17:44:24 2024","interfaces":[{"interface":"back","average":{"1min":2.0750000000000002,"5min":1.335,"15min":1.583},"min":{"1min":0.442,"5min":0.41899999999999998,"15min":0.40600000000000003},"max":{"1min":13.102,"5min":13.102,"15min":40.021000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":1.464,"5min":1.351,"15min":1.615},"min":{"1min":0.46800000000000003,"5min":0.46500000000000002,"15min":0.45200000000000001},"max":{"1min":4.7839999999999998,"5min":5.8390000000000004,"15min":40.055999999999997},"last":0.61799999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411647,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":345916,"kb_used_data":71428,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":93399236,"statfs":{"total":95995035648,"available":95640817664,"internally_reserved":0,"allocated":73142272,"data_stored":72774629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":280689093},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":1.4510000000000001,"15min":1.006},"min":{"1min":0.29999999999999999,"5min":0.23699999999999999,"15min":0.221},"max":{"1min":4.9089999999999998,"5min":17.251000000000001,"15min":17.251000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.74399999999999999,"5min":1.611,"15min":1.113},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.22900000000000001},"max":{"1min":4.7720000000000002,"5min":23.097999999999999,"15min":23.097999999999999},"last":0.875}]},{"osd":1,"last update":"Fri Feb 9 17:44:49 2024","interfaces":[{"interface":"back","average":{"1min":1.514,"5min":1.3180000000000001,"15min":1.5189999999999999},"min":{"1min":0.25,"5min":0.216,"15min":0.216},"max":{"1min":9.4380000000000006,"5min":15.186,"15min":22.306000000000001},"last":0.38200000000000001},{"interface":"front","average":{"1min":1.446,"5min":1.3220000000000001,"15min":1.5640000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.214},"max":{"1min":12.936,"5min":18.181000000000001,"15min":44.838999999999999},"last":0.45200000000000001}]},{"osd":3,"last update":"Fri Feb 9 17:44:17 2024","interfaces":[{"interface":"back","average":{"1min":2.0339999999999998,"5min":1.6379999999999999,"15min":1.514},"min":{"1min":0.29599999999999999,"5min":0.249,"15min":0.22700000000000001},"max":{"1min":14.355,"5min":17.942,"15min":24.353000000000002},"last":3.6080000000000001},{"interface":"front","average":{"1min":2.4089999999999998,"5min":1.3879999999999999,"15min":1.6259999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.6099999999999994,"5min":10.351000000000001,"15min":19.986999999999998},"last":0.35599999999999998}]},{"osd":4,"last update":"Fri Feb 9 17:44:17 2024","interfaces":[{"interface":"back","average":{"1min":1.653,"5min":1.397,"15min":1.7509999999999999},"min":{"1min":0.32100000000000001,"5min":0.29399999999999998,"15min":0.26300000000000001},"max":{"1min":8.7230000000000008,"5min":10.048999999999999,"15min":13.526},"last":2.629},{"interface":"front","average":{"1min":1.8839999999999999,"5min":1.49,"15min":1.6080000000000001},"min":{"1min":0.29499999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":6.1660000000000004,"5min":8.9920000000000009,"15min":9.9619999999999997},"last":2.6120000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:45:01 2024","interfaces":[{"interface":"back","average":{"1min":2.6549999999999998,"5min":1.8400000000000001,"15min":1.8220000000000001},"min":{"1min":0.34100000000000003,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":10.48,"5min":42.664000000000001,"15min":42.664000000000001},"last":2.8730000000000002},{"interface":"front","average":{"1min":2.7410000000000001,"5min":2.3239999999999998,"15min":2.0430000000000001},"min":{"1min":0.30599999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":9.1720000000000006,"5min":43.311999999999998,"15min":43.311999999999998},"last":2.8479999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:44:17 2024","interfaces":[{"interface":"back","average":{"1min":2.6760000000000002,"5min":1.7330000000000001,"15min":1.762},"min":{"1min":0.34200000000000003,"5min":0.312,"15min":0.30099999999999999},"max":{"1min":7.8949999999999996,"5min":10.114000000000001,"15min":10.295},"last":0.30199999999999999},{"interface":"front","average":{"1min":2.6080000000000001,"5min":1.8520000000000001,"15min":1.9830000000000001},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":7.726,"5min":10.144,"15min":11.269},"last":2.6459999999999999}]},{"osd":7,"last update":"Fri Feb 9 17:44:49 2024","interfaces":[{"interface":"back","average":{"1min":2.7839999999999998,"5min":1.718,"15min":2.0619999999999998},"min":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":11.099,"5min":11.099,"15min":41.802999999999997},"last":1.9039999999999999},{"interface":"front","average":{"1min":2.8199999999999998,"5min":2.0009999999999999,"15min":2.0739999999999998},"min":{"1min":0.41599999999999998,"5min":0.34100000000000003,"15min":0.30499999999999999},"max":{"1min":8.4079999999999995,"5min":11.099,"15min":41.783999999999999},"last":1.919}]},{"osd":8,"last update":"Fri Feb 9 17:44:17 2024","interfaces":[{"interface":"back","average":{"1min":1.137,"5min":1.401,"15min":1.3200000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":8.7729999999999997,"5min":8.7729999999999997,"15min":22.393999999999998},"last":0.47999999999999998},{"interface":"front","average":{"1min":1.5629999999999999,"5min":1.3520000000000001,"15min":1.5049999999999999},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":3.4729999999999999,"5min":12.58,"15min":12.58},"last":0.41699999999999998}]},{"osd":9,"last update":"Fri Feb 9 17:44:28 2024","interfaces":[{"interface":"back","average":{"1min":2.0920000000000001,"5min":1.5129999999999999,"15min":1.6100000000000001},"min":{"1min":0.37,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":12.768000000000001,"5min":12.768000000000001,"15min":13.5},"last":0.53900000000000003},{"interface":"front","average":{"1min":1.929,"5min":1.3660000000000001,"15min":1.5720000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.36399999999999999},"max":{"1min":7.5099999999999998,"5min":9.0060000000000002,"15min":13.589},"last":0.46500000000000002}]},{"osd":10,"last update":"Fri Feb 9 17:44:10 2024","interfaces":[{"interface":"back","average":{"1min":1.877,"5min":1.611,"15min":1.5269999999999999},"min":{"1min":0.42199999999999999,"5min":0.38400000000000001,"15min":0.33600000000000002},"max":{"1min":8.7509999999999994,"5min":9.1419999999999995,"15min":22.425999999999998},"last":0.622},{"interface":"front","average":{"1min":1.905,"5min":1.643,"15min":1.526},"min":{"1min":0.438,"5min":0.438,"15min":0.379},"max":{"1min":8.8109999999999999,"5min":9.1899999999999995,"15min":22.454999999999998},"last":0.64100000000000001}]},{"osd":11,"last update":"Fri Feb 9 17:44:17 2024","interfaces":[{"interface":"back","average":{"1min":1.738,"5min":1.514,"15min":2.04},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.39100000000000001},"max":{"1min":5.1580000000000004,"5min":7.4420000000000002,"15min":137.44300000000001},"last":0.51500000000000001},{"interface":"front","average":{"1min":1.5980000000000001,"5min":1.494,"15min":2.1000000000000001},"min":{"1min":0.45700000000000002,"5min":0.41199999999999998,"15min":0.40100000000000002},"max":{"1min":8.7699999999999996,"5min":9.1029999999999998,"15min":137.64400000000001},"last":0.57499999999999996}]}]},{"osd":3,"up_from":27,"seq":115964117308,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":431428,"kb_used_data":21244,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":93313724,"statfs":{"total":95995035648,"available":95553253376,"internally_reserved":0,"allocated":21753856,"data_stored":21389765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":420018631},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:44:14 2024","interfaces":[{"interface":"back","average":{"1min":1.657,"5min":1.6240000000000001,"15min":1.77},"min":{"1min":0.28499999999999998,"5min":0.23300000000000001,"15min":0.186},"max":{"1min":12.789,"5min":25.849,"15min":38.984999999999999},"last":0.38900000000000001},{"interface":"front","average":{"1min":1.6100000000000001,"5min":1.718,"15min":1.8959999999999999},"min":{"1min":0.29499999999999998,"5min":0.248,"15min":0.22500000000000001},"max":{"1min":11.932,"5min":30.268999999999998,"15min":41.497999999999998},"last":0.42399999999999999}]},{"osd":1,"last update":"Fri Feb 9 17:40:06 2024","interfaces":[{"interface":"back","average":{"1min":4.5670000000000002,"5min":4.5670000000000002,"15min":4.5670000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":25.965,"5min":25.965,"15min":25.965},"last":0.35299999999999998},{"interface":"front","average":{"1min":4.5129999999999999,"5min":4.5129999999999999,"15min":4.5129999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":31.454000000000001,"5min":31.454000000000001,"15min":31.454000000000001},"last":0.375}]},{"osd":2,"last update":"Fri Feb 9 17:45:12 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.1319999999999999,"15min":1.4319999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.22600000000000001},"max":{"1min":6.4560000000000004,"5min":11,"15min":14.888999999999999},"last":0.442},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.1120000000000001,"15min":1.514},"min":{"1min":0.28399999999999997,"5min":0.27600000000000002,"15min":0.20200000000000001},"max":{"1min":6.9619999999999997,"5min":11.071,"15min":17.613},"last":0.48599999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:44:14 2024","interfaces":[{"interface":"back","average":{"1min":1.9259999999999999,"5min":1.429,"15min":1.6040000000000001},"min":{"1min":0.36099999999999999,"5min":0.29099999999999998,"15min":0.26800000000000002},"max":{"1min":11.880000000000001,"5min":11.880000000000001,"15min":34.898000000000003},"last":0.498},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.4570000000000001,"15min":1.6599999999999999},"min":{"1min":0.29999999999999999,"5min":0.29399999999999998,"15min":0.27900000000000003},"max":{"1min":11.622999999999999,"5min":12.404999999999999,"15min":14.061999999999999},"last":0.40999999999999998}]},{"osd":5,"last update":"Fri Feb 9 17:44:54 2024","interfaces":[{"interface":"back","average":{"1min":2.573,"5min":1.8200000000000001,"15min":1.746},"min":{"1min":0.51900000000000002,"5min":0.28000000000000003,"15min":0.27800000000000002},"max":{"1min":9.9920000000000009,"5min":11.007999999999999,"15min":17.651},"last":0.52300000000000002},{"interface":"front","average":{"1min":2.1160000000000001,"5min":1.8149999999999999,"15min":1.778},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":5.3719999999999999,"5min":11.036,"15min":17.675999999999998},"last":0.53200000000000003}]},{"osd":6,"last update":"Fri Feb 9 17:44:43 2024","interfaces":[{"interface":"back","average":{"1min":1.5309999999999999,"5min":1.6819999999999999,"15min":2.3900000000000001},"min":{"1min":0.33100000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":4.7699999999999996,"5min":12.717000000000001,"15min":12.717000000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":1.726,"5min":1.5880000000000001,"15min":1.466},"min":{"1min":0.40899999999999997,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":4.9269999999999996,"5min":8.7360000000000007,"15min":8.7360000000000007},"last":0.55100000000000005}]},{"osd":7,"last update":"Fri Feb 9 17:44:25 2024","interfaces":[{"interface":"back","average":{"1min":2.6160000000000001,"5min":1.6779999999999999,"15min":1.1399999999999999},"min":{"1min":0.46100000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":11.769,"5min":11.769,"15min":11.769},"last":0.54300000000000004},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.1080000000000001,"15min":0.872},"min":{"1min":0.432,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":6.0990000000000002,"5min":6.0990000000000002,"15min":6.0990000000000002},"last":0.56599999999999995}]},{"osd":8,"last update":"Fri Feb 9 17:44:14 2024","interfaces":[{"interface":"back","average":{"1min":2.0910000000000002,"5min":1.5109999999999999,"15min":1.472},"min":{"1min":0.377,"5min":0.377,"15min":0.26800000000000002},"max":{"1min":11.821,"5min":11.821,"15min":11.821},"last":0.68799999999999994},{"interface":"front","average":{"1min":1.7669999999999999,"5min":1.4410000000000001,"15min":1.325},"min":{"1min":0.371,"5min":0.36199999999999999,"15min":0.34499999999999997},"max":{"1min":5.3520000000000003,"5min":10.388999999999999,"15min":14.129},"last":0.85399999999999998}]},{"osd":9,"last update":"Fri Feb 9 17:44:39 2024","interfaces":[{"interface":"back","average":{"1min":2.2410000000000001,"5min":1.452,"15min":1.5169999999999999},"min":{"1min":0.499,"5min":0.36199999999999999,"15min":0.32500000000000001},"max":{"1min":11.851000000000001,"5min":11.851000000000001,"15min":11.851000000000001},"last":0.76400000000000001},{"interface":"front","average":{"1min":2.1829999999999998,"5min":1.5209999999999999,"15min":1.423},"min":{"1min":0.44400000000000001,"5min":0.41199999999999998,"15min":0.32800000000000001},"max":{"1min":11.907,"5min":11.907,"15min":12.815},"last":0.92500000000000004}]},{"osd":10,"last update":"Fri Feb 9 17:44:25 2024","interfaces":[{"interface":"back","average":{"1min":2.1589999999999998,"5min":1.2090000000000001,"15min":0.97699999999999998},"min":{"1min":0.41099999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":9.7080000000000002,"5min":9.7080000000000002,"15min":9.7080000000000002},"last":0.68100000000000005},{"interface":"front","average":{"1min":1.6890000000000001,"5min":1.0249999999999999,"15min":0.84199999999999997},"min":{"1min":0.39100000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":5.2469999999999999,"5min":5.2469999999999999,"15min":5.2469999999999999},"last":0.94599999999999995}]},{"osd":11,"last update":"Fri Feb 9 17:44:14 2024","interfaces":[{"interface":"back","average":{"1min":1.5289999999999999,"5min":1.262,"15min":1.4630000000000001},"min":{"1min":0.439,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":5.9329999999999998,"5min":8.8849999999999998,"15min":14.300000000000001},"last":0.84499999999999997},{"interface":"front","average":{"1min":1.738,"5min":1.5449999999999999,"15min":1.6080000000000001},"min":{"1min":0.48299999999999998,"5min":0.442,"15min":0.39300000000000002},"max":{"1min":5.9160000000000004,"5min":9.4390000000000001,"15min":17.145},"last":0.874}]}]},{"osd":4,"up_from":32,"seq":137438953781,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":751868,"kb_used_data":183164,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":92993284,"statfs":{"total":95995035648,"available":95225122816,"internally_reserved":0,"allocated":187559936,"data_stored":187195845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":582089157},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:44:22 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.107,"15min":1.75},"min":{"1min":0.27300000000000002,"5min":0.222,"15min":0.222},"max":{"1min":4.7679999999999998,"5min":7.7199999999999998,"15min":50.356999999999999},"last":1.0880000000000001},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.169,"15min":1.679},"min":{"1min":0.372,"5min":0.26700000000000002,"15min":0.247},"max":{"1min":4.8680000000000003,"5min":7.6390000000000002,"15min":50.569000000000003},"last":1.1910000000000001}]},{"osd":1,"last update":"Fri Feb 9 17:44:22 2024","interfaces":[{"interface":"back","average":{"1min":1.9299999999999999,"5min":1.548,"15min":1.9870000000000001},"min":{"1min":0.253,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":8.0109999999999992,"5min":8.0109999999999992,"15min":50.473999999999997},"last":2.6659999999999999},{"interface":"front","average":{"1min":2.1469999999999998,"5min":1.9410000000000001,"15min":2.1960000000000002},"min":{"1min":0.32800000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":5.4189999999999996,"5min":11.593,"15min":50.499000000000002},"last":3.0449999999999999}]},{"osd":2,"last update":"Fri Feb 9 17:28:51 2024","interfaces":[{"interface":"back","average":{"1min":2.536,"5min":1.546,"15min":0.91400000000000003},"min":{"1min":0.29999999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":8.8819999999999997,"5min":8.8819999999999997,"15min":8.8819999999999997},"last":2.6120000000000001},{"interface":"front","average":{"1min":2.214,"5min":1.7330000000000001,"15min":0.97799999999999998},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":9.5039999999999996,"5min":9.5039999999999996,"15min":9.5039999999999996},"last":1.048}]},{"osd":3,"last update":"Fri Feb 9 17:44:22 2024","interfaces":[{"interface":"back","average":{"1min":1.752,"5min":1.661,"15min":2.0880000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.245},"max":{"1min":5.383,"5min":11.679,"15min":50.462000000000003},"last":3.0979999999999999},{"interface":"front","average":{"1min":1.794,"5min":1.3140000000000001,"15min":1.8340000000000001},"min":{"1min":0.39000000000000001,"5min":0.29099999999999998,"15min":0.255},"max":{"1min":6.657,"5min":9.1579999999999995,"15min":50.529000000000003},"last":1.2490000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:44:54 2024","interfaces":[{"interface":"back","average":{"1min":3.9689999999999999,"5min":2.2589999999999999,"15min":1.802},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":18.887,"5min":18.887,"15min":18.887},"last":2.573},{"interface":"front","average":{"1min":4.0330000000000004,"5min":2.371,"15min":2.0840000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.29099999999999998},"max":{"1min":18.917000000000002,"5min":18.917000000000002,"15min":50.509999999999998},"last":2.7069999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:44:22 2024","interfaces":[{"interface":"back","average":{"1min":2.5760000000000001,"5min":1.6080000000000001,"15min":1.6259999999999999},"min":{"1min":0.35999999999999999,"5min":0.28299999999999997,"15min":0.246},"max":{"1min":18.234999999999999,"5min":18.234999999999999,"15min":18.234999999999999},"last":1.22},{"interface":"front","average":{"1min":2.5219999999999998,"5min":2.1949999999999998,"15min":2.016},"min":{"1min":0.316,"5min":0.312,"15min":0.245},"max":{"1min":9.2159999999999993,"5min":13.208,"15min":50.402999999999999},"last":3.4580000000000002}]},{"osd":7,"last update":"Fri Feb 9 17:44:58 2024","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.6699999999999999,"15min":1.7529999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.28399999999999997},"max":{"1min":5.1900000000000004,"5min":11.875,"15min":14.919},"last":2.968},{"interface":"front","average":{"1min":2.1059999999999999,"5min":2.1190000000000002,"15min":1.96},"min":{"1min":0.35499999999999998,"5min":0.28599999999999998,"15min":0.28199999999999997},"max":{"1min":7.6020000000000003,"5min":12.696,"15min":19.047000000000001},"last":1.1140000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:44:54 2024","interfaces":[{"interface":"back","average":{"1min":3.0880000000000001,"5min":2.0129999999999999,"15min":1.7110000000000001},"min":{"1min":0.35299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":17.335999999999999,"5min":17.335999999999999,"15min":17.335999999999999},"last":2.7669999999999999},{"interface":"front","average":{"1min":1.853,"5min":1.367,"15min":1.675},"min":{"1min":0.41299999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":7.5099999999999998,"5min":7.5099999999999998,"15min":50.594000000000001},"last":1.143}]},{"osd":9,"last update":"Fri Feb 9 17:44:39 2024","interfaces":[{"interface":"back","average":{"1min":2.5739999999999998,"5min":1.855,"15min":1.881},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":10.778,"5min":10.778,"15min":50.442999999999998},"last":0.73599999999999999},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.331,"15min":1.536},"min":{"1min":0.47699999999999998,"5min":0.42499999999999999,"15min":0.36199999999999999},"max":{"1min":5.165,"5min":8.3580000000000005,"15min":10.555},"last":4.8259999999999996}]},{"osd":10,"last update":"Fri Feb 9 17:45:08 2024","interfaces":[{"interface":"back","average":{"1min":3.3799999999999999,"5min":2.3519999999999999,"15min":2.2559999999999998},"min":{"1min":0.433,"5min":0.40300000000000002,"15min":0.39400000000000002},"max":{"1min":17.367000000000001,"5min":17.367000000000001,"15min":51.223999999999997},"last":2.7389999999999999},{"interface":"front","average":{"1min":1.7889999999999999,"5min":1.7929999999999999,"15min":2.117},"min":{"1min":0.47599999999999998,"5min":0.42899999999999999,"15min":0.40500000000000003},"max":{"1min":7.9480000000000004,"5min":11.646000000000001,"15min":50.552},"last":3.0720000000000001}]},{"osd":11,"last update":"Fri Feb 9 17:44:39 2024","interfaces":[{"interface":"back","average":{"1min":1.4610000000000001,"5min":1.6850000000000001,"15min":2.0179999999999998},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.443},"max":{"1min":5.1399999999999997,"5min":9.4060000000000006,"15min":50.484999999999999},"last":1.167},{"interface":"front","average":{"1min":1.8799999999999999,"5min":1.4319999999999999,"15min":1.71},"min":{"1min":0.504,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":7.9859999999999998,"5min":7.9859999999999998,"15min":18.574999999999999},"last":3.0049999999999999}]}]},{"osd":5,"up_from":40,"seq":171798692143,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":658160,"kb_used_data":185328,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":93086992,"statfs":{"total":95995035648,"available":95321079808,"internally_reserved":0,"allocated":189775872,"data_stored":189342048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:44:12 2024","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.256,"15min":1.1200000000000001},"min":{"1min":0.24199999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":8.4700000000000006,"5min":8.4700000000000006,"15min":8.4700000000000006},"last":14.711},{"interface":"front","average":{"1min":2.1659999999999999,"5min":1.671,"15min":1.766},"min":{"1min":0.32500000000000001,"5min":0.249,"15min":0.249},"max":{"1min":9.0429999999999993,"5min":9.7650000000000006,"15min":9.7650000000000006},"last":0.40400000000000003}]},{"osd":1,"last update":"Fri Feb 9 17:44:53 2024","interfaces":[{"interface":"back","average":{"1min":1.635,"5min":1.6120000000000001,"15min":1.954},"min":{"1min":0.34899999999999998,"5min":0.24199999999999999,"15min":0.23899999999999999},"max":{"1min":7.0800000000000001,"5min":9.7330000000000005,"15min":48.161999999999999},"last":0.437},{"interface":"front","average":{"1min":1.74,"5min":1.857,"15min":1.907},"min":{"1min":0.28299999999999997,"5min":0.25900000000000001,"15min":0.23599999999999999},"max":{"1min":5.4169999999999998,"5min":7.8369999999999997,"15min":48.218000000000004},"last":16.626999999999999}]},{"osd":2,"last update":"Fri Feb 9 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.526,"15min":1.526},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":4.718,"5min":4.718,"15min":4.718},"last":2.6749999999999998},{"interface":"front","average":{"1min":2.1269999999999998,"5min":2.1269999999999998,"15min":2.1269999999999998},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":8.5079999999999991,"5min":8.5079999999999991,"15min":8.5079999999999991},"last":15.945}]},{"osd":3,"last update":"Fri Feb 9 17:44:53 2024","interfaces":[{"interface":"back","average":{"1min":2.1560000000000001,"5min":1.702,"15min":2.0840000000000001},"min":{"1min":0.35099999999999998,"5min":0.30499999999999999,"15min":0.26500000000000001},"max":{"1min":7.7750000000000004,"5min":11.081,"15min":64.150999999999996},"last":15.819000000000001},{"interface":"front","average":{"1min":2.0489999999999999,"5min":1.861,"15min":2.1800000000000002},"min":{"1min":0.36699999999999999,"5min":0.35299999999999998,"15min":0.247},"max":{"1min":6.2729999999999997,"5min":12.468,"15min":64.176000000000002},"last":0.47399999999999998}]},{"osd":4,"last update":"Fri Feb 9 17:44:53 2024","interfaces":[{"interface":"back","average":{"1min":2.4609999999999999,"5min":1.988,"15min":1.887},"min":{"1min":0.29599999999999999,"5min":0.249,"15min":0.23400000000000001},"max":{"1min":9.8620000000000001,"5min":9.8620000000000001,"15min":49.313000000000002},"last":14.766999999999999},{"interface":"front","average":{"1min":2.1829999999999998,"5min":1.954,"15min":1.853},"min":{"1min":0.26100000000000001,"5min":0.218,"15min":0.218},"max":{"1min":10.198,"5min":10.198,"15min":50.505000000000003},"last":5.1589999999999998}]},{"osd":6,"last update":"Fri Feb 9 17:44:23 2024","interfaces":[{"interface":"back","average":{"1min":2.214,"5min":1.8979999999999999,"15min":1.758},"min":{"1min":0.29899999999999999,"5min":0.28199999999999997,"15min":0.253},"max":{"1min":8.3390000000000004,"5min":12.425000000000001,"15min":15.375999999999999},"last":16.657},{"interface":"front","average":{"1min":2.1019999999999999,"5min":1.863,"15min":1.8380000000000001},"min":{"1min":0.308,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":8.4489999999999998,"5min":11.199,"15min":14.250999999999999},"last":15.903}]},{"osd":7,"last update":"Fri Feb 9 17:44:53 2024","interfaces":[{"interface":"back","average":{"1min":2.9340000000000002,"5min":2.0859999999999999,"15min":1.8879999999999999},"min":{"1min":0.434,"5min":0.27900000000000003,"15min":0.26700000000000002},"max":{"1min":10.249000000000001,"5min":12.475,"15min":50.545000000000002},"last":0.71399999999999997},{"interface":"front","average":{"1min":3.1120000000000001,"5min":2.2869999999999999,"15min":2.4729999999999999},"min":{"1min":0.40500000000000003,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":13.869,"5min":16.314,"15min":102.526},"last":2.6389999999999998}]},{"osd":8,"last update":"Fri Feb 9 17:43:23 2024","interfaces":[{"interface":"back","average":{"1min":1.6499999999999999,"5min":1.639,"15min":1.6020000000000001},"min":{"1min":0.376,"5min":0.35299999999999998,"15min":0.34899999999999998},"max":{"1min":4.8780000000000001,"5min":14.542,"15min":14.542},"last":5.5439999999999996},{"interface":"front","average":{"1min":3.6699999999999999,"5min":1.702,"15min":1.8300000000000001},"min":{"1min":0.37,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":16.359000000000002,"5min":16.359000000000002,"15min":16.359000000000002},"last":0.435}]},{"osd":9,"last update":"Fri Feb 9 17:44:23 2024","interfaces":[{"interface":"back","average":{"1min":1.778,"5min":1.643,"15min":1.5740000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":9.9510000000000005,"5min":16.411000000000001,"15min":16.411000000000001},"last":14.794},{"interface":"front","average":{"1min":1.8919999999999999,"5min":1.845,"15min":1.6579999999999999},"min":{"1min":0.42899999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":5.827,"5min":16.417999999999999,"15min":16.417999999999999},"last":2.698}]},{"osd":10,"last update":"Fri Feb 9 17:44:53 2024","interfaces":[{"interface":"back","average":{"1min":2.452,"5min":2.129,"15min":1.8120000000000001},"min":{"1min":0.48999999999999999,"5min":0.317,"15min":0.317},"max":{"1min":13.958,"5min":16.445,"15min":16.445},"last":2.7450000000000001},{"interface":"front","average":{"1min":1.702,"5min":1.778,"15min":1.798},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":6.1879999999999997,"5min":9.8840000000000003,"15min":14.186999999999999},"last":5.2939999999999996}]},{"osd":11,"last update":"Fri Feb 9 17:44:12 2024","interfaces":[{"interface":"back","average":{"1min":1.9950000000000001,"5min":2.2210000000000001,"15min":1.7649999999999999},"min":{"1min":0.434,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":5.0700000000000003,"5min":16.390000000000001,"15min":16.390000000000001},"last":2.7200000000000002},{"interface":"front","average":{"1min":2.0190000000000001,"5min":1.9690000000000001,"15min":1.758},"min":{"1min":0.53400000000000003,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":5.9660000000000002,"5min":9.8239999999999998,"15min":9.8239999999999998},"last":0.498}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":201523200,"data_stored":201523200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102105088,"data_stored":102105088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65339392,"data_stored":65339392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":17891328,"data_stored":17891328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":172752896,"data_stored":172752896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":178655232,"data_stored":178651160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":64487424,"data_stored":64487424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":67571712,"data_stored":67567640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":37552128,"data_stored":37552128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":56885248,"data_stored":56885248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":153092096,"data_stored":153092096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":93589504,"data_stored":93585432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-09T17:45:15.698 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd pool set unique_pool_3 pgp_num 16 2024-02-09T17:45:15.777 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:45:15.832 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:15.983 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:15 smithi099 ceph-mon[39420]: pgmap v1421: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 248 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 902 op/s; 534/11907 objects misplaced (4.485%) 2024-02-09T17:45:16.405 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:45:16.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:16.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:45:16.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:15 smithi179 ceph-mon[43975]: pgmap v1421: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 248 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 902 op/s; 534/11907 objects misplaced (4.485%) 2024-02-09T17:45:16.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:15 smithi162 ceph-mon[44304]: pgmap v1421: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 248 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 902 op/s; 534/11907 objects misplaced (4.485%) 2024-02-09T17:45:16.520 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:45:16.554 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:16.584 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:45:16.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:16.592 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:45:16.693 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:45:16.729 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:16.757 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:45:16.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:16.763 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:45:16.864 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:45:16.895 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:16.928 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:45:16.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:16.932 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:45:17.033 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:45:17.053 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Removed 128903 objects 2024-02-09T17:45:17.053 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Clean up completed and total clean up time :162.072 2024-02-09T17:45:17.063 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:17.082 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:45:17.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:17.085 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:45:17.186 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:45:17.212 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:17.230 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:45:17.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:17.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:45:17.335 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:45:17.360 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:17.379 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:45:17.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:17.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:45:17.483 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:45:17.509 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:17.528 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:45:17.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:17.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:45:17.632 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:45:17.664 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:17.686 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:45:17.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:17.689 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:45:17.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:17 smithi099 ceph-mon[39420]: pgmap v1422: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 134 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 932 op/s; 534/6414 objects misplaced (8.326%) 2024-02-09T17:45:17.749 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:45:17.749 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:45:17.751 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:45:17.789 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:45:17.820 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:17.842 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:45:17.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:17.845 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:45:17.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:17 smithi179 ceph-mon[43975]: pgmap v1422: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 134 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 932 op/s; 534/6414 objects misplaced (8.326%) 2024-02-09T17:45:17.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:17 smithi162 ceph-mon[44304]: pgmap v1422: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 134 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 932 op/s; 534/6414 objects misplaced (8.326%) 2024-02-09T17:45:17.946 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:45:17.975 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:18.047 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:45:18.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:18.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:45:18.151 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:45:18.184 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:18.205 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:45:18.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:18.208 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:45:18.311 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:45:18.341 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:18.360 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:45:18.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:18.364 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:45:18.465 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:45:18.493 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:18.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:18.516 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:45:18.517 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:45:18.535 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:45:18.554 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:45:18.573 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:45:18.592 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:45:18.614 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:45:18.618 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:45:18.634 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:45:18.645 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:18.654 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:45:18.664 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:45:18.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:18.667 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:45:18.674 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:45:18.694 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:45:18.714 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:45:18.737 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:45:18.756 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:45:18.757 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:45:18.758 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:45:18.758 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:45:18.758 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:45:18.758 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:45:18.758 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:45:18.758 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:45:18.758 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:45:18.758 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:45:18.758 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:45:18.758 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:45:18.758 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:45:18.768 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:45:18.800 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:18.821 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:45:18.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:18.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:45:18.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:18 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:45:18.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:18 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:45:18.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:18 smithi179 ceph-mon[43975]: osdmap e567: 12 total, 12 up, 12 in 2024-02-09T17:45:18.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:18 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2452571538' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:45:18.925 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:45:18.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:18 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:45:18.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:18 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:45:18.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:18 smithi162 ceph-mon[44304]: osdmap e567: 12 total, 12 up, 12 in 2024-02-09T17:45:18.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:18 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2452571538' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:45:18.946 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:18 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:45:18.946 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:18 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:45:18.946 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:18 smithi099 ceph-mon[39420]: osdmap e567: 12 total, 12 up, 12 in 2024-02-09T17:45:18.946 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:18 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2452571538' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:45:18.956 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:18.971 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-09T17:45:18.976 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:45:18.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:18.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:45:19.081 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:45:19.111 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:19.131 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:45:19.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:19.134 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:45:19.235 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:45:19.263 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:19.367 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:45:19.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:19 smithi179 ceph-mon[43975]: pgmap v1424: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 75 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 780 op/s; 534/3591 objects misplaced (14.871%) 2024-02-09T17:45:19.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:19 smithi162 ceph-mon[44304]: pgmap v1424: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 75 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 780 op/s; 534/3591 objects misplaced (14.871%) 2024-02-09T17:45:19.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:19 smithi099 ceph-mon[39420]: pgmap v1424: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 75 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 780 op/s; 534/3591 objects misplaced (14.871%) 2024-02-09T17:45:20.291 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:45:20.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:20.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:45:20.398 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:45:20.427 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:20.552 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:45:20.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:20.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:45:20.657 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:45:20.685 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:21.133 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:45:21.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:21.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:45:21.240 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:45:21.267 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:21.341 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:45:21.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:21.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:45:21.445 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:45:21.477 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:21.498 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:45:21.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:21.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:45:21.603 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:45:21.634 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:21.861 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:45:21.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:21.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:45:21.965 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:45:21.998 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:22.020 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:45:22.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:22.023 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:45:22.124 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:45:22.156 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:22.177 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:45:22.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:22.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:45:22.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:21 smithi099 ceph-mon[39420]: pgmap v1425: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 11 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 700 op/s 2024-02-09T17:45:22.281 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:45:22.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:21 smithi162 ceph-mon[44304]: pgmap v1425: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 11 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 700 op/s 2024-02-09T17:45:22.312 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:22.312 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:45:22.315 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:45:22.332 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:45:22.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:22.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:45:22.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:21 smithi179 ceph-mon[43975]: pgmap v1425: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 11 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 700 op/s 2024-02-09T17:45:22.436 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:45:22.462 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:22.560 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:45:22.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:22.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:45:22.667 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:45:22.699 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:22.720 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:45:22.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:22.724 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:45:22.825 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:45:22.852 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:22.942 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is unset 2024-02-09T17:45:22.975 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:45:22.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:22.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:45:23.079 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:45:23.104 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:22 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1863383' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:45:23.107 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:23.126 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:45:23.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:23.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:45:23.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:22 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1863383' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:45:23.231 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:45:23.261 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:22 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1863383' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:45:23.263 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:23.284 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:45:23.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:23.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:45:23.388 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:45:23.416 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:23.942 INFO:teuthology.orchestra.run.smithi099.stderr:set pool 5 pgp_num to 16 2024-02-09T17:45:24.069 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:45:24.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:24.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:45:24.173 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:45:24.202 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:23 smithi179 ceph-mon[43975]: pgmap v1426: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 594 op/s 2024-02-09T17:45:24.203 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:23 smithi179 ceph-mon[43975]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:45:24.203 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:23 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1863383' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:45:24.203 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:23 smithi179 ceph-mon[43975]: osdmap e568: 12 total, 12 up, 12 in 2024-02-09T17:45:24.203 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:23 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2869472153' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]: dispatch 2024-02-09T17:45:24.205 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:24.226 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:45:24.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:24.229 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:45:24.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:23 smithi099 ceph-mon[39420]: pgmap v1426: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 594 op/s 2024-02-09T17:45:24.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:23 smithi099 ceph-mon[39420]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:45:24.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:23 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1863383' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:45:24.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:23 smithi099 ceph-mon[39420]: osdmap e568: 12 total, 12 up, 12 in 2024-02-09T17:45:24.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:23 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2869472153' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]: dispatch 2024-02-09T17:45:24.331 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:45:24.361 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:24.382 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:45:24.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:24.385 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:45:24.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:23 smithi162 ceph-mon[44304]: pgmap v1426: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 594 op/s 2024-02-09T17:45:24.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:23 smithi162 ceph-mon[44304]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:45:24.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:23 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1863383' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:45:24.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:23 smithi162 ceph-mon[44304]: osdmap e568: 12 total, 12 up, 12 in 2024-02-09T17:45:24.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:23 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2869472153' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]: dispatch 2024-02-09T17:45:24.486 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:45:24.514 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:24.533 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:45:24.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:24.536 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:45:24.637 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:45:24.665 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:24.684 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:45:24.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:24.687 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:45:24.788 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:45:24.817 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:24.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:24 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2869472153' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]': finished 2024-02-09T17:45:24.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:24 smithi099 ceph-mon[39420]: osdmap e569: 12 total, 12 up, 12 in 2024-02-09T17:45:24.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:24 smithi099 ceph-mon[39420]: 5.2 scrub starts 2024-02-09T17:45:24.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:24 smithi099 ceph-mon[39420]: 5.2 scrub ok 2024-02-09T17:45:25.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:24 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2869472153' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]': finished 2024-02-09T17:45:25.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:24 smithi179 ceph-mon[43975]: osdmap e569: 12 total, 12 up, 12 in 2024-02-09T17:45:25.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:24 smithi179 ceph-mon[43975]: 5.2 scrub starts 2024-02-09T17:45:25.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:24 smithi179 ceph-mon[43975]: 5.2 scrub ok 2024-02-09T17:45:25.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:24 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2869472153' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]': finished 2024-02-09T17:45:25.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:24 smithi162 ceph-mon[44304]: osdmap e569: 12 total, 12 up, 12 in 2024-02-09T17:45:25.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:24 smithi162 ceph-mon[44304]: 5.2 scrub starts 2024-02-09T17:45:25.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:24 smithi162 ceph-mon[44304]: 5.2 scrub ok 2024-02-09T17:45:25.667 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:45:25.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:25.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:45:25.772 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:45:25.798 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:25.850 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:45:25.859 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:25.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:45:25.960 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:45:25.991 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:26.013 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:45:26.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:26.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:45:26.117 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:45:26.146 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:26.165 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:45:26.166 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:45:26.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:26.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:45:26.169 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-02-09T17:45:26.169 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-02-09T17:45:26.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:25 smithi099 ceph-mon[39420]: pgmap v1429: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 217 op/s 2024-02-09T17:45:26.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:25 smithi099 ceph-mon[39420]: 1.1a scrub starts 2024-02-09T17:45:26.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:25 smithi099 ceph-mon[39420]: 1.1a scrub ok 2024-02-09T17:45:26.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:25 smithi099 ceph-mon[39420]: 1.1d scrub starts 2024-02-09T17:45:26.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:25 smithi099 ceph-mon[39420]: 1.1d scrub ok 2024-02-09T17:45:26.268 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:45:26.293 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:25 smithi162 ceph-mon[44304]: pgmap v1429: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 217 op/s 2024-02-09T17:45:26.294 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:25 smithi162 ceph-mon[44304]: 1.1a scrub starts 2024-02-09T17:45:26.294 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:25 smithi162 ceph-mon[44304]: 1.1a scrub ok 2024-02-09T17:45:26.294 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:25 smithi162 ceph-mon[44304]: 1.1d scrub starts 2024-02-09T17:45:26.294 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:25 smithi162 ceph-mon[44304]: 1.1d scrub ok 2024-02-09T17:45:26.295 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:26.315 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:45:26.317 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:26.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:45:26.419 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:45:26.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:25 smithi179 ceph-mon[43975]: pgmap v1429: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 217 op/s 2024-02-09T17:45:26.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:25 smithi179 ceph-mon[43975]: 1.1a scrub starts 2024-02-09T17:45:26.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:25 smithi179 ceph-mon[43975]: 1.1a scrub ok 2024-02-09T17:45:26.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:25 smithi179 ceph-mon[43975]: 1.1d scrub starts 2024-02-09T17:45:26.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:25 smithi179 ceph-mon[43975]: 1.1d scrub ok 2024-02-09T17:45:26.451 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:26.472 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:45:26.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:26.475 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:45:26.576 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:45:26.604 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:26.789 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:45:26.789 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [], 2024-02-09T17:45:26.789 INFO:teuthology.orchestra.run.smithi099.stdout: "complaint_time": 30, 2024-02-09T17:45:26.789 INFO:teuthology.orchestra.run.smithi099.stdout: "num_blocked_ops": 0 2024-02-09T17:45:26.789 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:45:27.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:26 smithi099 ceph-mon[39420]: 5.6 scrub starts 2024-02-09T17:45:27.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:26 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:45:27.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:26 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:45:27.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:26 smithi099 ceph-mon[39420]: 5.6 scrub ok 2024-02-09T17:45:27.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:26 smithi099 ceph-mon[39420]: osdmap e570: 12 total, 12 up, 12 in 2024-02-09T17:45:27.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:26 smithi099 ceph-mon[39420]: 1.4 scrub starts 2024-02-09T17:45:27.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:26 smithi099 ceph-mon[39420]: 5.1 scrub starts 2024-02-09T17:45:27.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:26 smithi099 ceph-mon[39420]: 5.1 scrub ok 2024-02-09T17:45:27.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:26 smithi179 ceph-mon[43975]: 5.6 scrub starts 2024-02-09T17:45:27.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:26 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:45:27.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:26 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:45:27.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:26 smithi179 ceph-mon[43975]: 5.6 scrub ok 2024-02-09T17:45:27.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:26 smithi179 ceph-mon[43975]: osdmap e570: 12 total, 12 up, 12 in 2024-02-09T17:45:27.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:26 smithi179 ceph-mon[43975]: 1.4 scrub starts 2024-02-09T17:45:27.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:26 smithi179 ceph-mon[43975]: 5.1 scrub starts 2024-02-09T17:45:27.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:26 smithi179 ceph-mon[43975]: 5.1 scrub ok 2024-02-09T17:45:27.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:26 smithi162 ceph-mon[44304]: 5.6 scrub starts 2024-02-09T17:45:27.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:26 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:45:27.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:26 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:45:27.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:26 smithi162 ceph-mon[44304]: 5.6 scrub ok 2024-02-09T17:45:27.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:26 smithi162 ceph-mon[44304]: osdmap e570: 12 total, 12 up, 12 in 2024-02-09T17:45:27.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:26 smithi162 ceph-mon[44304]: 1.4 scrub starts 2024-02-09T17:45:27.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:26 smithi162 ceph-mon[44304]: 5.1 scrub starts 2024-02-09T17:45:27.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:26 smithi162 ceph-mon[44304]: 5.1 scrub ok 2024-02-09T17:45:27.552 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:45:27.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:27.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:45:27.657 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:45:27.689 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:27.691 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-09T17:45:27.710 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:45:27.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:27.713 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:45:27.814 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:45:27.847 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:27.868 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:45:27.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:27.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:45:27.972 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:45:28.003 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:28.023 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:45:28.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:28.027 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:45:28.092 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:45:28.128 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:45:28.158 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:27 smithi162 ceph-mon[44304]: 1.4 scrub ok 2024-02-09T17:45:28.158 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:27 smithi162 ceph-mon[44304]: 5.0 scrub starts 2024-02-09T17:45:28.158 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:27 smithi162 ceph-mon[44304]: 5.0 scrub ok 2024-02-09T17:45:28.158 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:27 smithi162 ceph-mon[44304]: pgmap v1431: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 28 op/s 2024-02-09T17:45:28.158 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:27 smithi162 ceph-mon[44304]: 1.1e scrub starts 2024-02-09T17:45:28.158 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:27 smithi162 ceph-mon[44304]: 5.c scrub starts 2024-02-09T17:45:28.158 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:27 smithi162 ceph-mon[44304]: 5.c scrub ok 2024-02-09T17:45:28.158 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:27 smithi162 ceph-mon[44304]: 5.b scrub starts 2024-02-09T17:45:28.158 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:27 smithi162 ceph-mon[44304]: 5.b scrub ok 2024-02-09T17:45:28.160 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:28.182 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:45:28.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:28.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:45:28.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:27 smithi099 ceph-mon[39420]: 1.4 scrub ok 2024-02-09T17:45:28.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:27 smithi099 ceph-mon[39420]: 5.0 scrub starts 2024-02-09T17:45:28.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:27 smithi099 ceph-mon[39420]: 5.0 scrub ok 2024-02-09T17:45:28.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:27 smithi099 ceph-mon[39420]: pgmap v1431: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 28 op/s 2024-02-09T17:45:28.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:27 smithi099 ceph-mon[39420]: 1.1e scrub starts 2024-02-09T17:45:28.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:27 smithi099 ceph-mon[39420]: 5.c scrub starts 2024-02-09T17:45:28.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:27 smithi099 ceph-mon[39420]: 5.c scrub ok 2024-02-09T17:45:28.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:27 smithi099 ceph-mon[39420]: 5.b scrub starts 2024-02-09T17:45:28.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:27 smithi099 ceph-mon[39420]: 5.b scrub ok 2024-02-09T17:45:28.285 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:45:28.315 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:27 smithi179 ceph-mon[43975]: 1.4 scrub ok 2024-02-09T17:45:28.315 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:27 smithi179 ceph-mon[43975]: 5.0 scrub starts 2024-02-09T17:45:28.315 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:27 smithi179 ceph-mon[43975]: 5.0 scrub ok 2024-02-09T17:45:28.315 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:27 smithi179 ceph-mon[43975]: pgmap v1431: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 28 op/s 2024-02-09T17:45:28.315 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:27 smithi179 ceph-mon[43975]: 1.1e scrub starts 2024-02-09T17:45:28.315 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:27 smithi179 ceph-mon[43975]: 5.c scrub starts 2024-02-09T17:45:28.315 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:27 smithi179 ceph-mon[43975]: 5.c scrub ok 2024-02-09T17:45:28.315 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:27 smithi179 ceph-mon[43975]: 5.b scrub starts 2024-02-09T17:45:28.315 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:27 smithi179 ceph-mon[43975]: 5.b scrub ok 2024-02-09T17:45:28.320 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:28.338 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:45:28.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:28.342 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:45:28.443 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:45:28.470 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:28.490 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:45:28.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:28.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:45:28.592 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-09T17:45:28.593 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:45:28.620 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:28.640 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:45:28.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:28.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:45:28.744 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:45:28.776 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:28.797 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:45:28.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:28.800 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:45:28.901 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:45:28.929 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:29.098 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:28 smithi099 ceph-mon[39420]: 1.1e scrub ok 2024-02-09T17:45:29.098 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:28 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:45:29.098 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:28 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:45:29.098 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:28 smithi099 ceph-mon[39420]: osdmap e571: 12 total, 12 up, 12 in 2024-02-09T17:45:29.098 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:28 smithi099 ceph-mon[39420]: 1.3 scrub starts 2024-02-09T17:45:29.098 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:28 smithi099 ceph-mon[39420]: 1.3 scrub ok 2024-02-09T17:45:29.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:28 smithi179 ceph-mon[43975]: 1.1e scrub ok 2024-02-09T17:45:29.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:28 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:45:29.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:28 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:45:29.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:28 smithi179 ceph-mon[43975]: osdmap e571: 12 total, 12 up, 12 in 2024-02-09T17:45:29.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:28 smithi179 ceph-mon[43975]: 1.3 scrub starts 2024-02-09T17:45:29.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:28 smithi179 ceph-mon[43975]: 1.3 scrub ok 2024-02-09T17:45:29.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:28 smithi162 ceph-mon[44304]: 1.1e scrub ok 2024-02-09T17:45:29.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:28 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:45:29.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:28 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:45:29.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:28 smithi162 ceph-mon[44304]: osdmap e571: 12 total, 12 up, 12 in 2024-02-09T17:45:29.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:28 smithi162 ceph-mon[44304]: 1.3 scrub starts 2024-02-09T17:45:29.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:28 smithi162 ceph-mon[44304]: 1.3 scrub ok 2024-02-09T17:45:29.443 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:45:29.443 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:45:29.443 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:45:29.444 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:45:29.444 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:45:29.466 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:45:29.487 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:45:29.509 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:45:29.529 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:45:29.549 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:45:29.569 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:45:29.589 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:45:29.590 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:45:29.591 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:45:29.591 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:45:29.591 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:45:29.591 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:45:29.591 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:45:29.591 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:45:29.591 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:45:29.591 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:45:29.591 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:45:29.591 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:45:29.591 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:45:29.778 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:45:29.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:29.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:45:29.882 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:45:29.914 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:29.936 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:45:29.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:29.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:45:30.040 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:45:30.067 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:29 smithi099 ceph-mon[39420]: 1.18 scrub starts 2024-02-09T17:45:30.067 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:29 smithi099 ceph-mon[39420]: 1.18 scrub ok 2024-02-09T17:45:30.067 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:29 smithi099 ceph-mon[39420]: pgmap v1433: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:45:30.067 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:29 smithi099 ceph-mon[39420]: 5.8 scrub starts 2024-02-09T17:45:30.067 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:29 smithi099 ceph-mon[39420]: 5.8 scrub ok 2024-02-09T17:45:30.067 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:29 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3764715490' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:45:30.067 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:29 smithi099 ceph-mon[39420]: from='client.16113 172.21.15.99:0/3764715490' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:45:30.069 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:30.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:29 smithi162 ceph-mon[44304]: 1.18 scrub starts 2024-02-09T17:45:30.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:29 smithi162 ceph-mon[44304]: 1.18 scrub ok 2024-02-09T17:45:30.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:29 smithi162 ceph-mon[44304]: pgmap v1433: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:45:30.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:29 smithi162 ceph-mon[44304]: 5.8 scrub starts 2024-02-09T17:45:30.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:29 smithi162 ceph-mon[44304]: 5.8 scrub ok 2024-02-09T17:45:30.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:29 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3764715490' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:45:30.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:29 smithi162 ceph-mon[44304]: from='client.16113 172.21.15.99:0/3764715490' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:45:30.317 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:45:30.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:30.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:45:30.421 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:45:30.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:29 smithi179 ceph-mon[43975]: 1.18 scrub starts 2024-02-09T17:45:30.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:29 smithi179 ceph-mon[43975]: 1.18 scrub ok 2024-02-09T17:45:30.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:29 smithi179 ceph-mon[43975]: pgmap v1433: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:45:30.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:29 smithi179 ceph-mon[43975]: 5.8 scrub starts 2024-02-09T17:45:30.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:29 smithi179 ceph-mon[43975]: 5.8 scrub ok 2024-02-09T17:45:30.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:29 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3764715490' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:45:30.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:29 smithi179 ceph-mon[43975]: from='client.16113 172.21.15.99:0/3764715490' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:45:30.448 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:30.571 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:45:30.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:30.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:45:30.676 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:45:30.712 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:30.734 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:45:30.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:30.738 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:45:30.839 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:45:30.866 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:31.114 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:45:31.115 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:30 smithi099 ceph-mon[39420]: 1.1 scrub starts 2024-02-09T17:45:31.115 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:30 smithi099 ceph-mon[39420]: 1.1 scrub ok 2024-02-09T17:45:31.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:31.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:45:31.169 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] 2024-02-09T17:45:31.169 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:45:31.170 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-02-09T17:45:31.170 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-09T17:45:31.218 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:45:31.249 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:30 smithi179 ceph-mon[43975]: 1.1 scrub starts 2024-02-09T17:45:31.249 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:30 smithi179 ceph-mon[43975]: 1.1 scrub ok 2024-02-09T17:45:31.251 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:31.272 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:45:31.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:31.276 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:45:31.377 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:45:31.407 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:31.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:30 smithi162 ceph-mon[44304]: 1.1 scrub starts 2024-02-09T17:45:31.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:30 smithi162 ceph-mon[44304]: 1.1 scrub ok 2024-02-09T17:45:31.427 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:45:31.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:31.431 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:45:31.532 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:45:31.561 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:31.580 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:45:31.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:31.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:45:31.641 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:45:31.645 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:45:31.682 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:45:31.683 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:45:31.714 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:32.201 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:31 smithi099 ceph-mon[39420]: 5.f scrub starts 2024-02-09T17:45:32.202 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:31 smithi099 ceph-mon[39420]: 5.f scrub ok 2024-02-09T17:45:32.202 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:31 smithi099 ceph-mon[39420]: pgmap v1434: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:45:32.202 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:31 smithi099 ceph-mon[39420]: 1.c scrub starts 2024-02-09T17:45:32.202 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:31 smithi099 ceph-mon[39420]: 1.c scrub ok 2024-02-09T17:45:32.202 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:31 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3747716881' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-09T17:45:32.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:31 smithi179 ceph-mon[43975]: 5.f scrub starts 2024-02-09T17:45:32.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:31 smithi179 ceph-mon[43975]: 5.f scrub ok 2024-02-09T17:45:32.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:31 smithi179 ceph-mon[43975]: pgmap v1434: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:45:32.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:31 smithi179 ceph-mon[43975]: 1.c scrub starts 2024-02-09T17:45:32.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:31 smithi179 ceph-mon[43975]: 1.c scrub ok 2024-02-09T17:45:32.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:31 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3747716881' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-09T17:45:32.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:31 smithi162 ceph-mon[44304]: 5.f scrub starts 2024-02-09T17:45:32.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:31 smithi162 ceph-mon[44304]: 5.f scrub ok 2024-02-09T17:45:32.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:31 smithi162 ceph-mon[44304]: pgmap v1434: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:45:32.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:31 smithi162 ceph-mon[44304]: 1.c scrub starts 2024-02-09T17:45:32.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:31 smithi162 ceph-mon[44304]: 1.c scrub ok 2024-02-09T17:45:32.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:31 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3747716881' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-09T17:45:32.642 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'unique_pool_3' removed 2024-02-09T17:45:33.118 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:45:33.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:32 smithi099 ceph-mon[39420]: 1.b scrub starts 2024-02-09T17:45:33.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:32 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3747716881' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-02-09T17:45:33.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:32 smithi099 ceph-mon[39420]: osdmap e572: 12 total, 12 up, 12 in 2024-02-09T17:45:33.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:32 smithi099 ceph-mon[39420]: 1.19 scrub starts 2024-02-09T17:45:33.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:32 smithi099 ceph-mon[39420]: 1.19 scrub ok 2024-02-09T17:45:33.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:32 smithi179 ceph-mon[43975]: 1.b scrub starts 2024-02-09T17:45:33.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:32 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3747716881' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-02-09T17:45:33.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:32 smithi179 ceph-mon[43975]: osdmap e572: 12 total, 12 up, 12 in 2024-02-09T17:45:33.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:32 smithi179 ceph-mon[43975]: 1.19 scrub starts 2024-02-09T17:45:33.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:32 smithi179 ceph-mon[43975]: 1.19 scrub ok 2024-02-09T17:45:33.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:32 smithi162 ceph-mon[44304]: 1.b scrub starts 2024-02-09T17:45:33.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:32 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3747716881' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-02-09T17:45:33.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:32 smithi162 ceph-mon[44304]: osdmap e572: 12 total, 12 up, 12 in 2024-02-09T17:45:33.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:32 smithi162 ceph-mon[44304]: 1.19 scrub starts 2024-02-09T17:45:33.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:32 smithi162 ceph-mon[44304]: 1.19 scrub ok 2024-02-09T17:45:33.664 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:45:33.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:33.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:45:33.767 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:45:33.795 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:33.814 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:45:33.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:33.817 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:45:33.918 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:45:33.945 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:33.964 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:45:33.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:33.967 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:45:34.068 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:45:34.097 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:34.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:33 smithi179 ceph-mon[43975]: 1.b scrub ok 2024-02-09T17:45:34.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:33 smithi179 ceph-mon[43975]: pgmap v1436: 31 pgs: 31 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:45:34.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:33 smithi179 ceph-mon[43975]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-09T17:45:34.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:33 smithi099 ceph-mon[39420]: 1.b scrub ok 2024-02-09T17:45:34.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:33 smithi099 ceph-mon[39420]: pgmap v1436: 31 pgs: 31 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:45:34.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:33 smithi099 ceph-mon[39420]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-09T17:45:34.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:33 smithi162 ceph-mon[44304]: 1.b scrub ok 2024-02-09T17:45:34.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:33 smithi162 ceph-mon[44304]: pgmap v1436: 31 pgs: 31 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:45:34.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:33 smithi162 ceph-mon[44304]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-09T17:45:34.689 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-02-09T17:45:34.690 INFO:tasks.radosbench:Beginning radosbench... 2024-02-09T17:45:34.690 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-02-09T17:45:34.691 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd pool create unique_pool_4 16 2024-02-09T17:45:35.046 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:45:35.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:35.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:45:35.150 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:45:35.182 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:45:35.182 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:45:35.182 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:35.203 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:45:35.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:35.207 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:45:35.308 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:45:35.337 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:35.459 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:45:35.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:35.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:45:35.563 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:45:35.595 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:35.616 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:45:35.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:35.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:45:35.721 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:45:35.750 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:35.769 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:45:35.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:35.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:45:35.874 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:45:35.884 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-09T17:45:35.901 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:35.919 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:45:35.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:35.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:45:36.023 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:45:36.050 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:36.150 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:45:36.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:35 smithi162 ceph-mon[44304]: pgmap v1437: 31 pgs: 31 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:45:36.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:35 smithi099 ceph-mon[39420]: pgmap v1437: 31 pgs: 31 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:45:36.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:35 smithi179 ceph-mon[43975]: pgmap v1437: 31 pgs: 31 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:45:36.510 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:45:36.511 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:45:36.517 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:45:36.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:36.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:45:36.621 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:45:36.650 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:36.882 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:45:36.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:36.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:45:36.986 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:45:37.015 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:37.063 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:36 smithi099 ceph-mon[39420]: 1.12 scrub starts 2024-02-09T17:45:37.063 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:36 smithi099 ceph-mon[39420]: 1.12 scrub ok 2024-02-09T17:45:37.063 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:36 smithi099 ceph-mon[39420]: 1.13 scrub starts 2024-02-09T17:45:37.063 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:36 smithi099 ceph-mon[39420]: 1.13 scrub ok 2024-02-09T17:45:37.088 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:45:37.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:37.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:45:37.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:36 smithi162 ceph-mon[44304]: 1.12 scrub starts 2024-02-09T17:45:37.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:36 smithi162 ceph-mon[44304]: 1.12 scrub ok 2024-02-09T17:45:37.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:36 smithi162 ceph-mon[44304]: 1.13 scrub starts 2024-02-09T17:45:37.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:36 smithi162 ceph-mon[44304]: 1.13 scrub ok 2024-02-09T17:45:37.191 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:45:37.220 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:37.368 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:45:37.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:37.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:45:37.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:36 smithi179 ceph-mon[43975]: 1.12 scrub starts 2024-02-09T17:45:37.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:36 smithi179 ceph-mon[43975]: 1.12 scrub ok 2024-02-09T17:45:37.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:36 smithi179 ceph-mon[43975]: 1.13 scrub starts 2024-02-09T17:45:37.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:36 smithi179 ceph-mon[43975]: 1.13 scrub ok 2024-02-09T17:45:37.471 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:45:37.499 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:37.598 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:45:37.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:37.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:45:37.702 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:45:37.734 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:37.755 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:45:37.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:37.758 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:45:37.859 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:45:37.886 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:37.983 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:45:37.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:37.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:45:38.089 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:45:38.121 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:38.144 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:45:38.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:38.145 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:45:38.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:37 smithi162 ceph-mon[44304]: pgmap v1438: 31 pgs: 31 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:45:38.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:37 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4123324067' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:45:38.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:37 smithi099 ceph-mon[39420]: pgmap v1438: 31 pgs: 31 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:45:38.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:37 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4123324067' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:45:38.246 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:45:38.275 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:38.404 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:45:38.404 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:45:38.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:37 smithi179 ceph-mon[43975]: pgmap v1438: 31 pgs: 31 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:45:38.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:37 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4123324067' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:45:38.473 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:45:38.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:38.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:45:38.576 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:45:38.606 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:38.625 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:45:38.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:38.628 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:45:38.729 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:45:38.757 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:39.005 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:45:39.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:39.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:45:39.067 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:39 smithi162 ceph-mon[44304]: 1.7 scrub starts 2024-02-09T17:45:39.068 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:39 smithi162 ceph-mon[44304]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:45:39.068 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:39 smithi162 ceph-mon[44304]: Cluster is now healthy 2024-02-09T17:45:39.068 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:39 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4123324067' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:45:39.068 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:39 smithi162 ceph-mon[44304]: osdmap e573: 12 total, 12 up, 12 in 2024-02-09T17:45:39.092 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-09T17:45:39.109 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:45:39.163 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:39.181 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:45:39.183 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is unset 2024-02-09T17:45:39.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:39.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:45:39.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:38 smithi099 ceph-mon[39420]: 1.7 scrub starts 2024-02-09T17:45:39.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:38 smithi099 ceph-mon[39420]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:45:39.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:38 smithi099 ceph-mon[39420]: Cluster is now healthy 2024-02-09T17:45:39.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:38 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4123324067' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:45:39.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:38 smithi099 ceph-mon[39420]: osdmap e573: 12 total, 12 up, 12 in 2024-02-09T17:45:39.285 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:45:39.313 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:39.333 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:45:39.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:39.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:45:39.361 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:45:39.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:38 smithi179 ceph-mon[43975]: 1.7 scrub starts 2024-02-09T17:45:39.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:38 smithi179 ceph-mon[43975]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:45:39.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:38 smithi179 ceph-mon[43975]: Cluster is now healthy 2024-02-09T17:45:39.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:38 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4123324067' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:45:39.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:38 smithi179 ceph-mon[43975]: osdmap e573: 12 total, 12 up, 12 in 2024-02-09T17:45:39.439 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:45:39.468 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:40.217 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:45:40.219 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:39 smithi099 ceph-mon[39420]: 1.7 scrub ok 2024-02-09T17:45:40.219 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:39 smithi099 ceph-mon[39420]: pgmap v1440: 31 pgs: 31 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:45:40.219 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:39 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:45:40.219 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:39 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:45:40.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:40.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:45:40.276 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:39 smithi162 ceph-mon[44304]: 1.7 scrub ok 2024-02-09T17:45:40.277 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:39 smithi162 ceph-mon[44304]: pgmap v1440: 31 pgs: 31 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:45:40.277 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:39 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:45:40.277 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:39 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:45:40.320 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:45:40.348 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:40.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:39 smithi179 ceph-mon[43975]: 1.7 scrub ok 2024-02-09T17:45:40.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:39 smithi179 ceph-mon[43975]: pgmap v1440: 31 pgs: 31 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:45:40.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:39 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:45:40.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:39 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:45:40.795 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:45:40.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:40.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:45:40.899 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:45:40.934 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:41.010 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:41 smithi099 ceph-mon[39420]: osdmap e574: 12 total, 12 up, 12 in 2024-02-09T17:45:41.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:41 smithi179 ceph-mon[43975]: osdmap e574: 12 total, 12 up, 12 in 2024-02-09T17:45:41.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:41 smithi162 ceph-mon[44304]: osdmap e574: 12 total, 12 up, 12 in 2024-02-09T17:45:41.922 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:45:41.922 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:45:41.923 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 575, 2024-02-09T17:45:41.923 INFO:teuthology.orchestra.run.smithi099.stdout: "fsid": "a69c870a-c76e-11ee-95b7-87774f69a715", 2024-02-09T17:45:41.923 INFO:teuthology.orchestra.run.smithi099.stdout: "created": "2024-02-09T17:15:26.113606+0000", 2024-02-09T17:45:41.923 INFO:teuthology.orchestra.run.smithi099.stdout: "modified": "2024-02-09T17:45:41.004049+0000", 2024-02-09T17:45:41.923 INFO:teuthology.orchestra.run.smithi099.stdout: "last_up_change": "2024-02-09T17:28:17.711044+0000", 2024-02-09T17:45:41.923 INFO:teuthology.orchestra.run.smithi099.stdout: "last_in_change": "2024-02-09T17:43:24.138763+0000", 2024-02-09T17:45:41.923 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-09T17:45:41.923 INFO:teuthology.orchestra.run.smithi099.stdout: "flags_num": 5799936, 2024-02-09T17:45:41.923 INFO:teuthology.orchestra.run.smithi099.stdout: "flags_set": [ 2024-02-09T17:45:41.923 INFO:teuthology.orchestra.run.smithi099.stdout: "pglog_hardlimit", 2024-02-09T17:45:41.923 INFO:teuthology.orchestra.run.smithi099.stdout: "purged_snapdirs", 2024-02-09T17:45:41.923 INFO:teuthology.orchestra.run.smithi099.stdout: "recovery_deletes", 2024-02-09T17:45:41.923 INFO:teuthology.orchestra.run.smithi099.stdout: "sortbitwise" 2024-02-09T17:45:41.923 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T17:45:41.924 INFO:teuthology.orchestra.run.smithi099.stdout: "crush_version": 27, 2024-02-09T17:45:41.924 INFO:teuthology.orchestra.run.smithi099.stdout: "full_ratio": 0.94999998807907104, 2024-02-09T17:45:41.924 INFO:teuthology.orchestra.run.smithi099.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-09T17:45:41.924 INFO:teuthology.orchestra.run.smithi099.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-09T17:45:41.924 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_snapshot": "", 2024-02-09T17:45:41.924 INFO:teuthology.orchestra.run.smithi099.stdout: "pool_max": 5, 2024-02-09T17:45:41.924 INFO:teuthology.orchestra.run.smithi099.stdout: "max_osd": 12, 2024-02-09T17:45:41.924 INFO:teuthology.orchestra.run.smithi099.stdout: "require_min_compat_client": "luminous", 2024-02-09T17:45:41.924 INFO:teuthology.orchestra.run.smithi099.stdout: "min_compat_client": "jewel", 2024-02-09T17:45:41.924 INFO:teuthology.orchestra.run.smithi099.stdout: "require_osd_release": "squid", 2024-02-09T17:45:41.924 INFO:teuthology.orchestra.run.smithi099.stdout: "allow_crimson": false, 2024-02-09T17:45:41.924 INFO:teuthology.orchestra.run.smithi099.stdout: "pools": [ 2024-02-09T17:45:41.924 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.924 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 1, 2024-02-09T17:45:41.925 INFO:teuthology.orchestra.run.smithi099.stdout: "pool_name": ".mgr", 2024-02-09T17:45:41.925 INFO:teuthology.orchestra.run.smithi099.stdout: "create_time": "2024-02-09T17:18:56.552286+0000", 2024-02-09T17:45:41.925 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 1, 2024-02-09T17:45:41.925 INFO:teuthology.orchestra.run.smithi099.stdout: "flags_names": "hashpspool", 2024-02-09T17:45:41.925 INFO:teuthology.orchestra.run.smithi099.stdout: "type": 1, 2024-02-09T17:45:41.925 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-02-09T17:45:41.925 INFO:teuthology.orchestra.run.smithi099.stdout: "min_size": 2, 2024-02-09T17:45:41.925 INFO:teuthology.orchestra.run.smithi099.stdout: "crush_rule": 0, 2024-02-09T17:45:41.925 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_count": 0, 2024-02-09T17:45:41.925 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_target": 0, 2024-02-09T17:45:41.925 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_barrier": 0, 2024-02-09T17:45:41.925 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-09T17:45:41.925 INFO:teuthology.orchestra.run.smithi099.stdout: "object_hash": 2, 2024-02-09T17:45:41.925 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_autoscale_mode": "off", 2024-02-09T17:45:41.925 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num": 31, 2024-02-09T17:45:41.926 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_placement_num": 31, 2024-02-09T17:45:41.926 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_placement_num_target": 31, 2024-02-09T17:45:41.926 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num_target": 31, 2024-02-09T17:45:41.926 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num_pending": 31, 2024-02-09T17:45:41.926 INFO:teuthology.orchestra.run.smithi099.stdout: "last_pg_merge_meta": { 2024-02-09T17:45:41.926 INFO:teuthology.orchestra.run.smithi099.stdout: "source_pgid": "0.0", 2024-02-09T17:45:41.926 INFO:teuthology.orchestra.run.smithi099.stdout: "ready_epoch": 0, 2024-02-09T17:45:41.926 INFO:teuthology.orchestra.run.smithi099.stdout: "last_epoch_started": 0, 2024-02-09T17:45:41.926 INFO:teuthology.orchestra.run.smithi099.stdout: "last_epoch_clean": 0, 2024-02-09T17:45:41.926 INFO:teuthology.orchestra.run.smithi099.stdout: "source_version": "0'0", 2024-02-09T17:45:41.926 INFO:teuthology.orchestra.run.smithi099.stdout: "target_version": "0'0" 2024-02-09T17:45:41.926 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.926 INFO:teuthology.orchestra.run.smithi099.stdout: "last_change": "544", 2024-02-09T17:45:41.927 INFO:teuthology.orchestra.run.smithi099.stdout: "last_force_op_resend": "0", 2024-02-09T17:45:41.927 INFO:teuthology.orchestra.run.smithi099.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-09T17:45:41.927 INFO:teuthology.orchestra.run.smithi099.stdout: "last_force_op_resend_preluminous": "514", 2024-02-09T17:45:41.927 INFO:teuthology.orchestra.run.smithi099.stdout: "auid": 0, 2024-02-09T17:45:41.927 INFO:teuthology.orchestra.run.smithi099.stdout: "snap_mode": "selfmanaged", 2024-02-09T17:45:41.927 INFO:teuthology.orchestra.run.smithi099.stdout: "snap_seq": 0, 2024-02-09T17:45:41.927 INFO:teuthology.orchestra.run.smithi099.stdout: "snap_epoch": 0, 2024-02-09T17:45:41.927 INFO:teuthology.orchestra.run.smithi099.stdout: "pool_snaps": [], 2024-02-09T17:45:41.927 INFO:teuthology.orchestra.run.smithi099.stdout: "removed_snaps": "[]", 2024-02-09T17:45:41.927 INFO:teuthology.orchestra.run.smithi099.stdout: "quota_max_bytes": 0, 2024-02-09T17:45:41.927 INFO:teuthology.orchestra.run.smithi099.stdout: "quota_max_objects": 0, 2024-02-09T17:45:41.927 INFO:teuthology.orchestra.run.smithi099.stdout: "tiers": [], 2024-02-09T17:45:41.927 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_of": -1, 2024-02-09T17:45:41.927 INFO:teuthology.orchestra.run.smithi099.stdout: "read_tier": -1, 2024-02-09T17:45:41.928 INFO:teuthology.orchestra.run.smithi099.stdout: "write_tier": -1, 2024-02-09T17:45:41.928 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_mode": "none", 2024-02-09T17:45:41.928 INFO:teuthology.orchestra.run.smithi099.stdout: "target_max_bytes": 0, 2024-02-09T17:45:41.928 INFO:teuthology.orchestra.run.smithi099.stdout: "target_max_objects": 0, 2024-02-09T17:45:41.928 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-09T17:45:41.928 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-09T17:45:41.928 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-09T17:45:41.928 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_min_flush_age": 0, 2024-02-09T17:45:41.928 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_min_evict_age": 0, 2024-02-09T17:45:41.928 INFO:teuthology.orchestra.run.smithi099.stdout: "erasure_code_profile": "", 2024-02-09T17:45:41.928 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_params": { 2024-02-09T17:45:41.928 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "none" 2024-02-09T17:45:41.928 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.928 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_period": 0, 2024-02-09T17:45:41.929 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_count": 0, 2024-02-09T17:45:41.929 INFO:teuthology.orchestra.run.smithi099.stdout: "use_gmt_hitset": true, 2024-02-09T17:45:41.929 INFO:teuthology.orchestra.run.smithi099.stdout: "min_read_recency_for_promote": 0, 2024-02-09T17:45:41.929 INFO:teuthology.orchestra.run.smithi099.stdout: "min_write_recency_for_promote": 0, 2024-02-09T17:45:41.929 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_grade_decay_rate": 0, 2024-02-09T17:45:41.929 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_search_last_n": 0, 2024-02-09T17:45:41.929 INFO:teuthology.orchestra.run.smithi099.stdout: "grade_table": [], 2024-02-09T17:45:41.929 INFO:teuthology.orchestra.run.smithi099.stdout: "stripe_width": 0, 2024-02-09T17:45:41.929 INFO:teuthology.orchestra.run.smithi099.stdout: "expected_num_objects": 0, 2024-02-09T17:45:41.929 INFO:teuthology.orchestra.run.smithi099.stdout: "fast_read": false, 2024-02-09T17:45:41.929 INFO:teuthology.orchestra.run.smithi099.stdout: "options": { 2024-02-09T17:45:41.929 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num_max": 32, 2024-02-09T17:45:41.929 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num_min": 1 2024-02-09T17:45:41.929 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.929 INFO:teuthology.orchestra.run.smithi099.stdout: "application_metadata": { 2024-02-09T17:45:41.930 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": {} 2024-02-09T17:45:41.930 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.930 INFO:teuthology.orchestra.run.smithi099.stdout: "read_balance": { 2024-02-09T17:45:41.930 INFO:teuthology.orchestra.run.smithi099.stdout: "score_type": "Fair distribution", 2024-02-09T17:45:41.930 INFO:teuthology.orchestra.run.smithi099.stdout: "score_acting": 1.5499999523162842, 2024-02-09T17:45:41.930 INFO:teuthology.orchestra.run.smithi099.stdout: "score_stable": 1.5499999523162842, 2024-02-09T17:45:41.930 INFO:teuthology.orchestra.run.smithi099.stdout: "optimal_score": 1, 2024-02-09T17:45:41.930 INFO:teuthology.orchestra.run.smithi099.stdout: "raw_score_acting": 1.5499999523162842, 2024-02-09T17:45:41.930 INFO:teuthology.orchestra.run.smithi099.stdout: "raw_score_stable": 1.5499999523162842, 2024-02-09T17:45:41.930 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity_weighted": 0.88067114353179932, 2024-02-09T17:45:41.930 INFO:teuthology.orchestra.run.smithi099.stdout: "average_primary_affinity": 0.87999999523162842, 2024-02-09T17:45:41.930 INFO:teuthology.orchestra.run.smithi099.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-02-09T17:45:41.930 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.930 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.930 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T17:45:41.931 INFO:teuthology.orchestra.run.smithi099.stdout: "osds": [ 2024-02-09T17:45:41.931 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.931 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 0, 2024-02-09T17:45:41.931 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "21739aa7-e278-45da-9a3f-867385f01baf", 2024-02-09T17:45:41.931 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:45:41.931 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:45:41.931 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:45:41.931 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 0.609619140625, 2024-02-09T17:45:41.931 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:45:41.931 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:45:41.931 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 8, 2024-02-09T17:45:41.931 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 557, 2024-02-09T17:45:41.931 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:45:41.931 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:45:41.931 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:45:41.932 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:45:41.932 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.932 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:45:41.932 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6802", 2024-02-09T17:45:41.932 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T17:45:41.932 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.932 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.932 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:45:41.932 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6803", 2024-02-09T17:45:41.932 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T17:45:41.932 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.932 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.932 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.932 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:45:41.933 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:45:41.933 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.933 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:45:41.933 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6804", 2024-02-09T17:45:41.933 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T17:45:41.933 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.933 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.933 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:45:41.933 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6805", 2024-02-09T17:45:41.933 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T17:45:41.933 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.933 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.933 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.933 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:45:41.933 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:45:41.934 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.934 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:45:41.934 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6808", 2024-02-09T17:45:41.934 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T17:45:41.934 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.934 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.934 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:45:41.934 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6809", 2024-02-09T17:45:41.934 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T17:45:41.934 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.934 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.934 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.934 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:45:41.934 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:45:41.934 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.934 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:45:41.935 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6806", 2024-02-09T17:45:41.935 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T17:45:41.935 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.935 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.935 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:45:41.935 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6807", 2024-02-09T17:45:41.935 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T17:45:41.935 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.935 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.935 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.935 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.99:6803/3859868923", 2024-02-09T17:45:41.935 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.99:6805/3859868923", 2024-02-09T17:45:41.935 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.99:6809/3859868923", 2024-02-09T17:45:41.935 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.99:6807/3859868923", 2024-02-09T17:45:41.935 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:45:41.935 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:45:41.936 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:45:41.936 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.936 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.936 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.936 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 1, 2024-02-09T17:45:41.936 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "8eab347c-9065-4de1-93ae-e2eb7e11cab7", 2024-02-09T17:45:41.936 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:45:41.936 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:45:41.936 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:45:41.936 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T17:45:41.936 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:45:41.936 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:45:41.936 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 13, 2024-02-09T17:45:41.936 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 525, 2024-02-09T17:45:41.936 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:45:41.936 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:45:41.937 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:45:41.937 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:45:41.937 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.937 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:45:41.937 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6810", 2024-02-09T17:45:41.937 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T17:45:41.937 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.937 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.937 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:45:41.937 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6811", 2024-02-09T17:45:41.937 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T17:45:41.937 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.937 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.937 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.937 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:45:41.937 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:45:41.938 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.938 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:45:41.938 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6812", 2024-02-09T17:45:41.938 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T17:45:41.938 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.938 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.938 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:45:41.938 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6813", 2024-02-09T17:45:41.938 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T17:45:41.938 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.938 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.938 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.938 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:45:41.939 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:45:41.939 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.939 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:45:41.939 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6816", 2024-02-09T17:45:41.939 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T17:45:41.939 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.939 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.939 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:45:41.939 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6817", 2024-02-09T17:45:41.940 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T17:45:41.940 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.940 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.940 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.940 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:45:41.940 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:45:41.940 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.940 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:45:41.940 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6814", 2024-02-09T17:45:41.940 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T17:45:41.940 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.940 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.940 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:45:41.941 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6815", 2024-02-09T17:45:41.941 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T17:45:41.941 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.941 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.941 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.941 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.99:6811/3868328452", 2024-02-09T17:45:41.941 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.99:6813/3868328452", 2024-02-09T17:45:41.941 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.99:6817/3868328452", 2024-02-09T17:45:41.941 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.99:6815/3868328452", 2024-02-09T17:45:41.941 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:45:41.941 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:45:41.941 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:45:41.941 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.941 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.942 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.942 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 2, 2024-02-09T17:45:41.942 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "575f4257-7e34-485e-963c-2607b69e1220", 2024-02-09T17:45:41.942 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:45:41.942 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:45:41.942 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:45:41.942 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 0.3601837158203125, 2024-02-09T17:45:41.942 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:45:41.942 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:45:41.942 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 18, 2024-02-09T17:45:41.942 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 537, 2024-02-09T17:45:41.942 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:45:41.942 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:45:41.942 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:45:41.943 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:45:41.943 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.943 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:45:41.943 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6818", 2024-02-09T17:45:41.943 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T17:45:41.943 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.943 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.943 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:45:41.943 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6819", 2024-02-09T17:45:41.943 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T17:45:41.943 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.943 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.943 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.944 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:45:41.944 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:45:41.944 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.944 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:45:41.944 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6820", 2024-02-09T17:45:41.944 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T17:45:41.944 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.944 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.945 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:45:41.945 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6821", 2024-02-09T17:45:41.945 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T17:45:41.945 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.945 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.945 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.945 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:45:41.945 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:45:41.945 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.945 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:45:41.945 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6824", 2024-02-09T17:45:41.945 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T17:45:41.945 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.945 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.946 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:45:41.946 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6825", 2024-02-09T17:45:41.946 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T17:45:41.946 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.946 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.946 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.946 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:45:41.946 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:45:41.946 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.946 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:45:41.946 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6822", 2024-02-09T17:45:41.946 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T17:45:41.946 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.946 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.947 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:45:41.947 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6823", 2024-02-09T17:45:41.947 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T17:45:41.947 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.947 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.947 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.947 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.99:6819/537506237", 2024-02-09T17:45:41.947 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.99:6821/537506237", 2024-02-09T17:45:41.947 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.99:6825/537506237", 2024-02-09T17:45:41.947 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.99:6823/537506237", 2024-02-09T17:45:41.947 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:45:41.947 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:45:41.947 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:45:41.947 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.948 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.948 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.948 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 3, 2024-02-09T17:45:41.948 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "1a520bcf-81bb-4f1d-80fb-11e98008e429", 2024-02-09T17:45:41.948 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:45:41.948 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:45:41.948 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:45:41.948 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T17:45:41.948 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:45:41.948 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:45:41.948 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 27, 2024-02-09T17:45:41.948 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 518, 2024-02-09T17:45:41.948 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:45:41.949 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:45:41.949 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:45:41.949 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:45:41.949 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.949 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:45:41.949 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6826", 2024-02-09T17:45:41.949 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 762784793 2024-02-09T17:45:41.949 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.949 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.949 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:45:41.949 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6827", 2024-02-09T17:45:41.949 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 762784793 2024-02-09T17:45:41.949 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.949 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.950 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.950 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:45:41.950 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:45:41.950 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.950 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:45:41.950 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6828", 2024-02-09T17:45:41.950 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 762784793 2024-02-09T17:45:41.950 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.950 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.950 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:45:41.950 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6829", 2024-02-09T17:45:41.950 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 762784793 2024-02-09T17:45:41.950 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.950 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.951 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.951 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:45:41.951 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:45:41.951 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.951 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:45:41.951 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6832", 2024-02-09T17:45:41.951 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 762784793 2024-02-09T17:45:41.951 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.951 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.951 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:45:41.951 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6833", 2024-02-09T17:45:41.951 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 762784793 2024-02-09T17:45:41.951 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.951 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.951 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.952 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:45:41.952 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:45:41.952 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.952 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:45:41.952 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6830", 2024-02-09T17:45:41.952 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 762784793 2024-02-09T17:45:41.952 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.952 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.952 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:45:41.952 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6831", 2024-02-09T17:45:41.952 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 762784793 2024-02-09T17:45:41.952 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.952 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.952 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.953 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.99:6827/762784793", 2024-02-09T17:45:41.953 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.99:6829/762784793", 2024-02-09T17:45:41.953 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.99:6833/762784793", 2024-02-09T17:45:41.953 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.99:6831/762784793", 2024-02-09T17:45:41.953 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:45:41.953 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:45:41.953 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:45:41.953 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.953 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.953 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.953 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 4, 2024-02-09T17:45:41.953 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "4617c2fe-c9d8-40cd-887b-303e3215bdf8", 2024-02-09T17:45:41.953 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:45:41.953 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:45:41.954 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:45:41.954 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T17:45:41.954 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:45:41.954 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:45:41.954 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 32, 2024-02-09T17:45:41.954 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 556, 2024-02-09T17:45:41.954 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:45:41.954 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:45:41.954 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:45:41.954 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:45:41.954 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.954 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:45:41.954 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6800", 2024-02-09T17:45:41.954 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T17:45:41.954 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.954 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.955 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:45:41.955 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6801", 2024-02-09T17:45:41.955 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T17:45:41.955 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.955 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.955 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.955 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:45:41.955 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:45:41.955 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.955 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:45:41.955 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6802", 2024-02-09T17:45:41.955 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T17:45:41.955 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.955 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.956 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:45:41.956 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6803", 2024-02-09T17:45:41.956 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T17:45:41.956 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.956 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.956 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.956 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:45:41.956 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:45:41.956 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.956 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:45:41.956 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6806", 2024-02-09T17:45:41.956 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T17:45:41.956 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.957 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.957 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:45:41.957 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6807", 2024-02-09T17:45:41.957 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T17:45:41.957 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.957 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.957 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.957 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:45:41.957 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:45:41.957 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.957 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:45:41.957 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6804", 2024-02-09T17:45:41.957 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T17:45:41.958 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.958 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.958 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:45:41.958 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6805", 2024-02-09T17:45:41.958 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T17:45:41.958 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.958 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.958 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.958 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.162:6801/4178676504", 2024-02-09T17:45:41.958 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.162:6803/4178676504", 2024-02-09T17:45:41.958 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.162:6807/4178676504", 2024-02-09T17:45:41.958 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.162:6805/4178676504", 2024-02-09T17:45:41.958 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:45:41.958 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:45:41.959 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:45:41.959 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.959 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.959 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.959 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 5, 2024-02-09T17:45:41.959 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "da98e3f3-173e-46fe-be8b-1ccbe8b0dabc", 2024-02-09T17:45:41.959 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:45:41.959 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:45:41.959 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:45:41.959 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T17:45:41.959 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:45:41.959 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:45:41.959 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 40, 2024-02-09T17:45:41.959 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 536, 2024-02-09T17:45:41.959 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:45:41.960 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:45:41.960 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:45:41.960 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:45:41.960 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.960 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:45:41.960 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6808", 2024-02-09T17:45:41.960 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T17:45:41.960 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.960 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.960 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:45:41.960 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6809", 2024-02-09T17:45:41.960 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T17:45:41.960 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.961 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.961 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.961 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:45:41.961 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:45:41.961 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.961 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:45:41.961 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6810", 2024-02-09T17:45:41.961 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T17:45:41.961 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.961 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.961 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:45:41.961 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6811", 2024-02-09T17:45:41.961 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T17:45:41.961 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.961 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.962 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.962 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:45:41.962 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:45:41.962 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.962 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:45:41.962 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6814", 2024-02-09T17:45:41.962 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T17:45:41.962 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.962 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.962 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:45:41.962 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6815", 2024-02-09T17:45:41.962 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T17:45:41.962 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.962 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.963 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.963 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:45:41.963 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:45:41.963 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.963 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:45:41.963 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6812", 2024-02-09T17:45:41.963 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T17:45:41.963 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.963 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.963 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:45:41.963 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6813", 2024-02-09T17:45:41.963 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T17:45:41.963 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.963 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.964 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.964 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.162:6809/1489371339", 2024-02-09T17:45:41.964 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.162:6811/1489371339", 2024-02-09T17:45:41.964 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.162:6815/1489371339", 2024-02-09T17:45:41.964 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.162:6813/1489371339", 2024-02-09T17:45:41.964 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:45:41.964 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:45:41.964 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:45:41.964 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.964 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.964 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.964 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 6, 2024-02-09T17:45:41.964 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "ff50382d-5ddf-4cda-bb0c-ff055feeaa40", 2024-02-09T17:45:41.965 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:45:41.965 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:45:41.965 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:45:41.965 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T17:45:41.965 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:45:41.965 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:45:41.965 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 49, 2024-02-09T17:45:41.965 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 518, 2024-02-09T17:45:41.965 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:45:41.965 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:45:41.965 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:45:41.965 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:45:41.965 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.965 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:45:41.965 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6816", 2024-02-09T17:45:41.966 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T17:45:41.966 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.966 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.966 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:45:41.966 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6817", 2024-02-09T17:45:41.966 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T17:45:41.966 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.966 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.966 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.966 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:45:41.966 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:45:41.966 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.966 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:45:41.966 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6818", 2024-02-09T17:45:41.966 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T17:45:41.967 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.967 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.967 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:45:41.967 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6819", 2024-02-09T17:45:41.967 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T17:45:41.967 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.967 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.967 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.967 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:45:41.967 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:45:41.967 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.967 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:45:41.967 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6822", 2024-02-09T17:45:41.968 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T17:45:41.968 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.968 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.968 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:45:41.968 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6823", 2024-02-09T17:45:41.968 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T17:45:41.968 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.968 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.968 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.968 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:45:41.968 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:45:41.968 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.968 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:45:41.968 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6820", 2024-02-09T17:45:41.969 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T17:45:41.969 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.969 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.969 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:45:41.969 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6821", 2024-02-09T17:45:41.969 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T17:45:41.969 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.969 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.969 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.969 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.162:6817/821169663", 2024-02-09T17:45:41.969 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.162:6819/821169663", 2024-02-09T17:45:41.969 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.162:6823/821169663", 2024-02-09T17:45:41.969 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.162:6821/821169663", 2024-02-09T17:45:41.969 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:45:41.969 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:45:41.970 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:45:41.970 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.970 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.970 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.970 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 7, 2024-02-09T17:45:41.970 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "a82984dc-5e22-424b-a446-ae2ee71eee19", 2024-02-09T17:45:41.970 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:45:41.970 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:45:41.970 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:45:41.970 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 0.5982513427734375, 2024-02-09T17:45:41.970 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:45:41.970 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:45:41.970 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 57, 2024-02-09T17:45:41.970 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 525, 2024-02-09T17:45:41.970 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:45:41.970 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:45:41.971 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:45:41.971 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:45:41.971 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.971 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:45:41.971 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6824", 2024-02-09T17:45:41.971 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T17:45:41.971 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.971 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.971 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:45:41.971 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6825", 2024-02-09T17:45:41.971 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T17:45:41.971 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.971 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.971 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.971 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:45:41.972 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:45:41.972 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.972 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:45:41.972 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6826", 2024-02-09T17:45:41.972 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T17:45:41.972 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.972 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.972 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:45:41.972 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6827", 2024-02-09T17:45:41.972 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T17:45:41.972 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.972 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.972 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.973 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:45:41.973 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:45:41.973 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.973 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:45:41.973 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6830", 2024-02-09T17:45:41.973 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T17:45:41.973 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.973 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.973 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:45:41.973 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6831", 2024-02-09T17:45:41.973 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T17:45:41.973 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.973 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.973 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.974 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:45:41.974 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:45:41.974 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.974 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:45:41.974 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6828", 2024-02-09T17:45:41.974 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T17:45:41.974 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.974 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.974 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:45:41.974 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6829", 2024-02-09T17:45:41.974 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T17:45:41.974 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.974 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.974 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.974 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.162:6825/965264369", 2024-02-09T17:45:41.975 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.162:6827/965264369", 2024-02-09T17:45:41.975 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.162:6831/965264369", 2024-02-09T17:45:41.975 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.162:6829/965264369", 2024-02-09T17:45:41.975 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:45:41.975 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:45:41.975 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:45:41.975 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.975 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.975 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.975 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 8, 2024-02-09T17:45:41.975 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "63f9d79f-0400-44dc-b388-df00fefb5820", 2024-02-09T17:45:41.975 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:45:41.975 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:45:41.975 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:45:41.976 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T17:45:41.976 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 66, 2024-02-09T17:45:41.976 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 145, 2024-02-09T17:45:41.976 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 171, 2024-02-09T17:45:41.976 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 552, 2024-02-09T17:45:41.976 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 146, 2024-02-09T17:45:41.976 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:45:41.976 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:45:41.976 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:45:41.976 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.976 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:45:41.976 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6800", 2024-02-09T17:45:41.976 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T17:45:41.977 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.977 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.977 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:45:41.977 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6801", 2024-02-09T17:45:41.977 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T17:45:41.977 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.977 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.977 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.977 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:45:41.977 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:45:41.977 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.977 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:45:41.977 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6802", 2024-02-09T17:45:41.977 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T17:45:41.978 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.978 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.978 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:45:41.978 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6803", 2024-02-09T17:45:41.978 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T17:45:41.978 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.978 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.978 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.978 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:45:41.978 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:45:41.978 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.978 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:45:41.978 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6806", 2024-02-09T17:45:41.978 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T17:45:41.979 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.979 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.979 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:45:41.979 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6807", 2024-02-09T17:45:41.979 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T17:45:41.979 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.979 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.979 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.979 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:45:41.979 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:45:41.979 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.979 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:45:41.979 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6804", 2024-02-09T17:45:41.979 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T17:45:41.979 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.980 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.980 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:45:41.980 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6805", 2024-02-09T17:45:41.980 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T17:45:41.980 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.980 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.980 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.980 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.179:6801/2558294495", 2024-02-09T17:45:41.980 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.179:6803/2558294495", 2024-02-09T17:45:41.980 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.179:6807/2558294495", 2024-02-09T17:45:41.980 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.179:6805/2558294495", 2024-02-09T17:45:41.980 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:45:41.980 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:45:41.980 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:45:41.981 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.981 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.981 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.981 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 9, 2024-02-09T17:45:41.981 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "7d995854-7328-4dfc-b1a9-0edca8d74f46", 2024-02-09T17:45:41.981 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:45:41.981 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:45:41.981 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:45:41.981 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T17:45:41.981 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:45:41.981 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:45:41.981 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 77, 2024-02-09T17:45:41.981 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 514, 2024-02-09T17:45:41.981 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:45:41.982 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:45:41.982 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:45:41.982 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:45:41.982 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.982 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:45:41.982 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6808", 2024-02-09T17:45:41.982 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T17:45:41.982 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.982 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.982 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:45:41.982 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6809", 2024-02-09T17:45:41.982 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T17:45:41.982 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.983 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.983 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.983 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:45:41.983 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:45:41.983 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.983 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:45:41.983 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6810", 2024-02-09T17:45:41.983 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T17:45:41.983 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.983 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.983 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:45:41.983 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6811", 2024-02-09T17:45:41.983 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T17:45:41.983 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.983 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.984 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.984 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:45:41.984 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:45:41.984 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.984 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:45:41.984 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6814", 2024-02-09T17:45:41.984 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T17:45:41.984 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.984 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.984 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:45:41.984 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6815", 2024-02-09T17:45:41.984 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T17:45:41.984 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.984 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.984 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.985 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:45:41.985 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:45:41.985 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.985 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:45:41.985 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6812", 2024-02-09T17:45:41.985 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T17:45:41.985 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.985 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.985 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:45:41.985 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6813", 2024-02-09T17:45:41.985 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T17:45:41.985 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.985 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.985 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.985 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.179:6809/504427183", 2024-02-09T17:45:41.985 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.179:6811/504427183", 2024-02-09T17:45:41.986 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.179:6815/504427183", 2024-02-09T17:45:41.986 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.179:6813/504427183", 2024-02-09T17:45:41.986 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:45:41.986 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:45:41.986 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:45:41.986 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.986 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.986 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.986 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 10, 2024-02-09T17:45:41.986 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "151fbedd-08e6-46e6-a4e5-a14248399f61", 2024-02-09T17:45:41.986 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:45:41.986 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:45:41.986 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:45:41.986 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T17:45:41.986 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:45:41.986 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:45:41.987 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 84, 2024-02-09T17:45:41.987 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 535, 2024-02-09T17:45:41.987 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:45:41.987 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:45:41.987 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:45:41.987 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:45:41.987 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.987 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:45:41.987 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6816", 2024-02-09T17:45:41.987 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T17:45:41.987 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.987 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.987 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:45:41.987 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6817", 2024-02-09T17:45:41.987 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T17:45:41.988 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.988 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.988 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.988 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:45:41.988 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:45:41.988 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.988 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:45:41.988 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6818", 2024-02-09T17:45:41.988 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T17:45:41.988 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.988 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.988 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:45:41.988 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6819", 2024-02-09T17:45:41.988 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T17:45:41.989 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.989 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.989 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.989 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:45:41.989 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:45:41.989 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.989 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:45:41.989 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6822", 2024-02-09T17:45:41.989 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T17:45:41.989 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.989 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.989 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:45:41.989 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6823", 2024-02-09T17:45:41.989 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T17:45:41.989 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.989 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.989 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.990 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:45:41.990 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:45:41.990 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.990 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:45:41.990 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6820", 2024-02-09T17:45:41.990 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T17:45:41.990 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.990 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.990 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:45:41.990 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6821", 2024-02-09T17:45:41.990 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T17:45:41.990 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.990 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.990 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.990 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.179:6817/2403345164", 2024-02-09T17:45:41.990 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.179:6819/2403345164", 2024-02-09T17:45:41.991 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.179:6823/2403345164", 2024-02-09T17:45:41.991 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.179:6821/2403345164", 2024-02-09T17:45:41.991 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:45:41.991 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:45:41.991 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:45:41.991 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.991 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.991 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.991 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 11, 2024-02-09T17:45:41.991 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "6d3a3ad8-417a-483f-991c-52549dbe7d73", 2024-02-09T17:45:41.991 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:45:41.991 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:45:41.991 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:45:41.991 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T17:45:41.991 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:45:41.991 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:45:41.992 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 90, 2024-02-09T17:45:41.992 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 519, 2024-02-09T17:45:41.992 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:45:41.992 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:45:41.992 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:45:41.992 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:45:41.992 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.992 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:45:41.992 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6824", 2024-02-09T17:45:41.992 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T17:45:41.992 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.992 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.992 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:45:41.992 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6825", 2024-02-09T17:45:41.993 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T17:45:41.993 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.993 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.993 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.993 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:45:41.993 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:45:41.993 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.993 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:45:41.993 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6826", 2024-02-09T17:45:41.993 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T17:45:41.993 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.993 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.993 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:45:41.993 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6827", 2024-02-09T17:45:41.993 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T17:45:41.993 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.994 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.994 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.994 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:45:41.994 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:45:41.994 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.994 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:45:41.994 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6830", 2024-02-09T17:45:41.994 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T17:45:41.994 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.994 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.994 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:45:41.994 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6831", 2024-02-09T17:45:41.994 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T17:45:41.994 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.994 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.994 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.994 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:45:41.995 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:45:41.995 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.995 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:45:41.995 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6828", 2024-02-09T17:45:41.995 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T17:45:41.995 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.995 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.995 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:45:41.995 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6829", 2024-02-09T17:45:41.995 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T17:45:41.995 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.995 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.995 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.995 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.179:6825/2899402373", 2024-02-09T17:45:41.995 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.179:6827/2899402373", 2024-02-09T17:45:41.996 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.179:6831/2899402373", 2024-02-09T17:45:41.996 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.179:6829/2899402373", 2024-02-09T17:45:41.996 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:45:41.996 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:45:41.996 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:45:41.996 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:45:41.996 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:41.996 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T17:45:41.996 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_xinfo": [ 2024-02-09T17:45:41.996 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.996 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 0, 2024-02-09T17:45:41.996 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:45:41.996 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:45:41.997 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:45:41.997 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:45:41.997 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:45:41.997 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:45:25.864115+0000", 2024-02-09T17:45:41.997 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:45:41.997 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.997 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.997 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 1, 2024-02-09T17:45:41.997 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:45:41.997 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:45:41.997 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:45:41.997 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:45:41.997 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:45:41.997 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:45:40.613496+0000", 2024-02-09T17:45:41.997 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:45:41.997 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.997 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.998 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 2, 2024-02-09T17:45:41.998 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:45:41.998 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:45:41.998 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:45:41.998 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:45:41.998 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:45:41.998 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:45:13.654372+0000", 2024-02-09T17:45:41.998 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:45:41.998 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.998 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.998 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 3, 2024-02-09T17:45:41.998 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:45:41.998 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:45:41.998 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:45:41.998 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:45:41.999 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:45:41.999 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:44:29.864171+0000", 2024-02-09T17:45:41.999 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:45:41.999 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:41.999 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:41.999 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 4, 2024-02-09T17:45:41.999 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:45:41.999 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:45:41.999 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:45:41.999 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:45:41.999 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:45:41.999 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:45:09.441700+0000", 2024-02-09T17:45:41.999 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:45:41.999 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:42.000 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:42.000 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 5, 2024-02-09T17:45:42.000 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:45:42.000 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:45:42.000 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:45:42.000 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:45:42.000 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:45:42.000 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:45:04.754298+0000", 2024-02-09T17:45:42.000 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:45:42.000 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:42.000 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:42.000 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 6, 2024-02-09T17:45:42.000 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:45:42.000 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:45:42.001 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:45:42.001 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:45:42.001 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:45:42.001 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:44:50.442694+0000", 2024-02-09T17:45:42.001 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:45:42.001 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:42.001 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:42.001 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 7, 2024-02-09T17:45:42.001 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:45:42.001 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:45:42.001 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:45:42.001 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:45:42.001 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:45:42.001 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:44:26.159633+0000", 2024-02-09T17:45:42.001 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:45:42.002 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:42.002 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:42.002 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 8, 2024-02-09T17:45:42.002 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:45:42.002 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:45:42.002 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:45:42.002 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:45:42.002 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:45:42.002 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:45:17.431302+0000", 2024-02-09T17:45:42.002 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 145 2024-02-09T17:45:42.002 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:42.002 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:42.002 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 9, 2024-02-09T17:45:42.002 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:45:42.002 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:45:42.003 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:45:42.003 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:45:42.003 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:45:42.003 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:45:03.406185+0000", 2024-02-09T17:45:42.003 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:45:42.003 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:42.003 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:42.003 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 10, 2024-02-09T17:45:42.003 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:45:42.003 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:45:42.003 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:45:42.003 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:45:42.003 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:45:42.003 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:45:27.199188+0000", 2024-02-09T17:45:42.004 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:45:42.004 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:42.004 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:45:42.004 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 11, 2024-02-09T17:45:42.004 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:45:42.004 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:45:42.004 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:45:42.004 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:45:42.004 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:45:42.004 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:45:39.014845+0000", 2024-02-09T17:45:42.004 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:45:42.004 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:42.004 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T17:45:42.004 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_upmap": [], 2024-02-09T17:45:42.005 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_upmap_items": [], 2024-02-09T17:45:42.005 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_upmap_primaries": [], 2024-02-09T17:45:42.005 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_temp": [], 2024-02-09T17:45:42.005 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_temp": [], 2024-02-09T17:45:42.005 INFO:teuthology.orchestra.run.smithi099.stdout: "blocklist": { 2024-02-09T17:45:42.005 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:0/3904172782": "2024-02-10T17:16:24.229365+0000", 2024-02-09T17:45:42.005 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:6801/1586034576": "2024-02-10T17:16:24.229365+0000", 2024-02-09T17:45:42.005 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:6800/746248609": "2024-02-10T17:15:54.399201+0000", 2024-02-09T17:45:42.005 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:0/4092784197": "2024-02-10T17:16:24.229365+0000", 2024-02-09T17:45:42.005 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:0/2844598101": "2024-02-10T17:15:54.399201+0000", 2024-02-09T17:45:42.005 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:6801/746248609": "2024-02-10T17:15:54.399201+0000", 2024-02-09T17:45:42.005 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:0/2528750050": "2024-02-10T17:15:54.399201+0000", 2024-02-09T17:45:42.005 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:6800/1586034576": "2024-02-10T17:16:24.229365+0000", 2024-02-09T17:45:42.005 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:0/2214140078": "2024-02-10T17:15:54.399201+0000", 2024-02-09T17:45:42.005 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:0/4217821965": "2024-02-10T17:16:24.229365+0000" 2024-02-09T17:45:42.005 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:42.006 INFO:teuthology.orchestra.run.smithi099.stdout: "range_blocklist": {}, 2024-02-09T17:45:42.006 INFO:teuthology.orchestra.run.smithi099.stdout: "erasure_code_profiles": { 2024-02-09T17:45:42.006 INFO:teuthology.orchestra.run.smithi099.stdout: "default": { 2024-02-09T17:45:42.006 INFO:teuthology.orchestra.run.smithi099.stdout: "crush-failure-domain": "osd", 2024-02-09T17:45:42.006 INFO:teuthology.orchestra.run.smithi099.stdout: "k": "2", 2024-02-09T17:45:42.006 INFO:teuthology.orchestra.run.smithi099.stdout: "m": "1", 2024-02-09T17:45:42.006 INFO:teuthology.orchestra.run.smithi099.stdout: "plugin": "jerasure", 2024-02-09T17:45:42.006 INFO:teuthology.orchestra.run.smithi099.stdout: "technique": "reed_sol_van" 2024-02-09T17:45:42.006 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:42.006 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:45:42.006 INFO:teuthology.orchestra.run.smithi099.stdout: "removed_snaps_queue": [], 2024-02-09T17:45:42.006 INFO:teuthology.orchestra.run.smithi099.stdout: "new_removed_snaps": [], 2024-02-09T17:45:42.006 INFO:teuthology.orchestra.run.smithi099.stdout: "new_purged_snaps": [], 2024-02-09T17:45:42.006 INFO:teuthology.orchestra.run.smithi099.stdout: "crush_node_flags": {}, 2024-02-09T17:45:42.006 INFO:teuthology.orchestra.run.smithi099.stdout: "device_class_flags": {}, 2024-02-09T17:45:42.006 INFO:teuthology.orchestra.run.smithi099.stdout: "stretch_mode": { 2024-02-09T17:45:42.006 INFO:teuthology.orchestra.run.smithi099.stdout: "stretch_mode_enabled": false, 2024-02-09T17:45:42.007 INFO:teuthology.orchestra.run.smithi099.stdout: "stretch_bucket_count": 0, 2024-02-09T17:45:42.007 INFO:teuthology.orchestra.run.smithi099.stdout: "degraded_stretch_mode": 0, 2024-02-09T17:45:42.007 INFO:teuthology.orchestra.run.smithi099.stdout: "recovering_stretch_mode": 0, 2024-02-09T17:45:42.007 INFO:teuthology.orchestra.run.smithi099.stdout: "stretch_mode_bucket": 0 2024-02-09T17:45:42.007 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:45:42.007 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:45:42.007 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set noscrub 2024-02-09T17:45:42.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:42.008 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:45:42.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:45:42.109 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:45:42.139 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:42.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:42 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:45:42.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:42 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:45:42.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:42 smithi162 ceph-mon[44304]: pgmap v1442: 31 pgs: 31 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:45:42.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:42 smithi162 ceph-mon[44304]: osdmap e575: 12 total, 12 up, 12 in 2024-02-09T17:45:42.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:42 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/859869092' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-02-09T17:45:42.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:42 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3585019662' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-09T17:45:42.198 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:42 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:45:42.198 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:42 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:45:42.198 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:42 smithi099 ceph-mon[39420]: pgmap v1442: 31 pgs: 31 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:45:42.198 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:42 smithi099 ceph-mon[39420]: osdmap e575: 12 total, 12 up, 12 in 2024-02-09T17:45:42.198 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:42 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/859869092' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-02-09T17:45:42.199 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:42 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3585019662' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-09T17:45:42.213 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:45:42.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:42.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:45:42.318 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:45:42.346 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:42.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:42 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:45:42.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:42 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:45:42.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:42 smithi179 ceph-mon[43975]: pgmap v1442: 31 pgs: 31 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:45:42.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:42 smithi179 ceph-mon[43975]: osdmap e575: 12 total, 12 up, 12 in 2024-02-09T17:45:42.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:42 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/859869092' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-02-09T17:45:42.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:42 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3585019662' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-09T17:45:42.532 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:45:42.532 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:45:42.545 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:45:42.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:42.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:45:42.593 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'unique_pool_4' created 2024-02-09T17:45:42.649 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:45:42.681 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:42.701 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:45:42.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:42.705 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:45:42.806 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:45:42.835 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:42.955 INFO:tasks.thrashosds.thrasher:j is {'epoch': 575, 'fsid': 'a69c870a-c76e-11ee-95b7-87774f69a715', 'created': '2024-02-09T17:15:26.113606+0000', 'modified': '2024-02-09T17:45:41.004049+0000', 'last_up_change': '2024-02-09T17:28:17.711044+0000', 'last_in_change': '2024-02-09T17:43:24.138763+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 27, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 5, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-09T17:18:56.552286+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '544', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '514', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5499999523162842, 'score_stable': 1.5499999523162842, 'optimal_score': 1, 'raw_score_acting': 1.5499999523162842, 'raw_score_stable': 1.5499999523162842, 'primary_affinity_weighted': 0.8806711435317993, 'average_primary_affinity': 0.8799999952316284, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': '21739aa7-e278-45da-9a3f-867385f01baf', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.609619140625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 557, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6802', 'nonce': 3859868923}, {'type': 'v1', 'addr': '172.21.15.99:6803', 'nonce': 3859868923}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6804', 'nonce': 3859868923}, {'type': 'v1', 'addr': '172.21.15.99:6805', 'nonce': 3859868923}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6808', 'nonce': 3859868923}, {'type': 'v1', 'addr': '172.21.15.99:6809', 'nonce': 3859868923}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6806', 'nonce': 3859868923}, {'type': 'v1', 'addr': '172.21.15.99:6807', 'nonce': 3859868923}]}, 'public_addr': '172.21.15.99:6803/3859868923', 'cluster_addr': '172.21.15.99:6805/3859868923', 'heartbeat_back_addr': '172.21.15.99:6809/3859868923', 'heartbeat_front_addr': '172.21.15.99:6807/3859868923', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '8eab347c-9065-4de1-93ae-e2eb7e11cab7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 525, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6810', 'nonce': 3868328452}, {'type': 'v1', 'addr': '172.21.15.99:6811', 'nonce': 3868328452}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6812', 'nonce': 3868328452}, {'type': 'v1', 'addr': '172.21.15.99:6813', 'nonce': 3868328452}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6816', 'nonce': 3868328452}, {'type': 'v1', 'addr': '172.21.15.99:6817', 'nonce': 3868328452}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6814', 'nonce': 3868328452}, {'type': 'v1', 'addr': '172.21.15.99:6815', 'nonce': 3868328452}]}, 'public_addr': '172.21.15.99:6811/3868328452', 'cluster_addr': '172.21.15.99:6813/3868328452', 'heartbeat_back_addr': '172.21.15.99:6817/3868328452', 'heartbeat_front_addr': '172.21.15.99:6815/3868328452', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '575f4257-7e34-485e-963c-2607b69e1220', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.3601837158203125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 537, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6818', 'nonce': 537506237}, {'type': 'v1', 'addr': '172.21.15.99:6819', 'nonce': 537506237}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6820', 'nonce': 537506237}, {'type': 'v1', 'addr': '172.21.15.99:6821', 'nonce': 537506237}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6824', 'nonce': 537506237}, {'type': 'v1', 'addr': '172.21.15.99:6825', 'nonce': 537506237}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6822', 'nonce': 537506237}, {'type': 'v1', 'addr': '172.21.15.99:6823', 'nonce': 537506237}]}, 'public_addr': '172.21.15.99:6819/537506237', 'cluster_addr': '172.21.15.99:6821/537506237', 'heartbeat_back_addr': '172.21.15.99:6825/537506237', 'heartbeat_front_addr': '172.21.15.99:6823/537506237', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '1a520bcf-81bb-4f1d-80fb-11e98008e429', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 518, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6826', 'nonce': 762784793}, {'type': 'v1', 'addr': '172.21.15.99:6827', 'nonce': 762784793}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6828', 'nonce': 762784793}, {'type': 'v1', 'addr': '172.21.15.99:6829', 'nonce': 762784793}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6832', 'nonce': 762784793}, {'type': 'v1', 'addr': '172.21.15.99:6833', 'nonce': 762784793}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6830', 'nonce': 762784793}, {'type': 'v1', 'addr': '172.21.15.99:6831', 'nonce': 762784793}]}, 'public_addr': '172.21.15.99:6827/762784793', 'cluster_addr': '172.21.15.99:6829/762784793', 'heartbeat_back_addr': '172.21.15.99:6833/762784793', 'heartbeat_front_addr': '172.21.15.99:6831/762784793', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '4617c2fe-c9d8-40cd-887b-303e3215bdf8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 556, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6800', 'nonce': 4178676504}, {'type': 'v1', 'addr': '172.21.15.162:6801', 'nonce': 4178676504}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6802', 'nonce': 4178676504}, {'type': 'v1', 'addr': '172.21.15.162:6803', 'nonce': 4178676504}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6806', 'nonce': 4178676504}, {'type': 'v1', 'addr': '172.21.15.162:6807', 'nonce': 4178676504}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6804', 'nonce': 4178676504}, {'type': 'v1', 'addr': '172.21.15.162:6805', 'nonce': 4178676504}]}, 'public_addr': '172.21.15.162:6801/4178676504', 'cluster_addr': '172.21.15.162:6803/4178676504', 'heartbeat_back_addr': '172.21.15.162:6807/4178676504', 'heartbeat_front_addr': '172.21.15.162:6805/4178676504', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'da98e3f3-173e-46fe-be8b-1ccbe8b0dabc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 536, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6808', 'nonce': 1489371339}, {'type': 'v1', 'addr': '172.21.15.162:6809', 'nonce': 1489371339}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6810', 'nonce': 1489371339}, {'type': 'v1', 'addr': '172.21.15.162:6811', 'nonce': 1489371339}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6814', 'nonce': 1489371339}, {'type': 'v1', 'addr': '172.21.15.162:6815', 'nonce': 1489371339}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6812', 'nonce': 1489371339}, {'type': 'v1', 'addr': '172.21.15.162:6813', 'nonce': 1489371339}]}, 'public_addr': '172.21.15.162:6809/1489371339', 'cluster_addr': '172.21.15.162:6811/1489371339', 'heartbeat_back_addr': '172.21.15.162:6815/1489371339', 'heartbeat_front_addr': '172.21.15.162:6813/1489371339', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'ff50382d-5ddf-4cda-bb0c-ff055feeaa40', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 49, 'up_thru': 518, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6816', 'nonce': 821169663}, {'type': 'v1', 'addr': '172.21.15.162:6817', 'nonce': 821169663}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6818', 'nonce': 821169663}, {'type': 'v1', 'addr': '172.21.15.162:6819', 'nonce': 821169663}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6822', 'nonce': 821169663}, {'type': 'v1', 'addr': '172.21.15.162:6823', 'nonce': 821169663}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6820', 'nonce': 821169663}, {'type': 'v1', 'addr': '172.21.15.162:6821', 'nonce': 821169663}]}, 'public_addr': '172.21.15.162:6817/821169663', 'cluster_addr': '172.21.15.162:6819/821169663', 'heartbeat_back_addr': '172.21.15.162:6823/821169663', 'heartbeat_front_addr': '172.21.15.162:6821/821169663', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'a82984dc-5e22-424b-a446-ae2ee71eee19', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.5982513427734375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 57, 'up_thru': 525, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6824', 'nonce': 965264369}, {'type': 'v1', 'addr': '172.21.15.162:6825', 'nonce': 965264369}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6826', 'nonce': 965264369}, {'type': 'v1', 'addr': '172.21.15.162:6827', 'nonce': 965264369}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6830', 'nonce': 965264369}, {'type': 'v1', 'addr': '172.21.15.162:6831', 'nonce': 965264369}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6828', 'nonce': 965264369}, {'type': 'v1', 'addr': '172.21.15.162:6829', 'nonce': 965264369}]}, 'public_addr': '172.21.15.162:6825/965264369', 'cluster_addr': '172.21.15.162:6827/965264369', 'heartbeat_back_addr': '172.21.15.162:6831/965264369', 'heartbeat_front_addr': '172.21.15.162:6829/965264369', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '63f9d79f-0400-44dc-b388-df00fefb5820', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 66, 'last_clean_end': 145, 'up_from': 171, 'up_thru': 552, 'down_at': 146, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6800', 'nonce': 2558294495}, {'type': 'v1', 'addr': '172.21.15.179:6801', 'nonce': 2558294495}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6802', 'nonce': 2558294495}, {'type': 'v1', 'addr': '172.21.15.179:6803', 'nonce': 2558294495}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6806', 'nonce': 2558294495}, {'type': 'v1', 'addr': '172.21.15.179:6807', 'nonce': 2558294495}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6804', 'nonce': 2558294495}, {'type': 'v1', 'addr': '172.21.15.179:6805', 'nonce': 2558294495}]}, 'public_addr': '172.21.15.179:6801/2558294495', 'cluster_addr': '172.21.15.179:6803/2558294495', 'heartbeat_back_addr': '172.21.15.179:6807/2558294495', 'heartbeat_front_addr': '172.21.15.179:6805/2558294495', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '7d995854-7328-4dfc-b1a9-0edca8d74f46', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 77, 'up_thru': 514, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6808', 'nonce': 504427183}, {'type': 'v1', 'addr': '172.21.15.179:6809', 'nonce': 504427183}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6810', 'nonce': 504427183}, {'type': 'v1', 'addr': '172.21.15.179:6811', 'nonce': 504427183}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6814', 'nonce': 504427183}, {'type': 'v1', 'addr': '172.21.15.179:6815', 'nonce': 504427183}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6812', 'nonce': 504427183}, {'type': 'v1', 'addr': '172.21.15.179:6813', 'nonce': 504427183}]}, 'public_addr': '172.21.15.179:6809/504427183', 'cluster_addr': '172.21.15.179:6811/504427183', 'heartbeat_back_addr': '172.21.15.179:6815/504427183', 'heartbeat_front_addr': '172.21.15.179:6813/504427183', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '151fbedd-08e6-46e6-a4e5-a14248399f61', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 84, 'up_thru': 535, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6816', 'nonce': 2403345164}, {'type': 'v1', 'addr': '172.21.15.179:6817', 'nonce': 2403345164}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6818', 'nonce': 2403345164}, {'type': 'v1', 'addr': '172.21.15.179:6819', 'nonce': 2403345164}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6822', 'nonce': 2403345164}, {'type': 'v1', 'addr': '172.21.15.179:6823', 'nonce': 2403345164}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6820', 'nonce': 2403345164}, {'type': 'v1', 'addr': '172.21.15.179:6821', 'nonce': 2403345164}]}, 'public_addr': '172.21.15.179:6817/2403345164', 'cluster_addr': '172.21.15.179:6819/2403345164', 'heartbeat_back_addr': '172.21.15.179:6823/2403345164', 'heartbeat_front_addr': '172.21.15.179:6821/2403345164', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '6d3a3ad8-417a-483f-991c-52549dbe7d73', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 90, 'up_thru': 519, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6824', 'nonce': 2899402373}, {'type': 'v1', 'addr': '172.21.15.179:6825', 'nonce': 2899402373}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6826', 'nonce': 2899402373}, {'type': 'v1', 'addr': '172.21.15.179:6827', 'nonce': 2899402373}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6830', 'nonce': 2899402373}, {'type': 'v1', 'addr': '172.21.15.179:6831', 'nonce': 2899402373}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6828', 'nonce': 2899402373}, {'type': 'v1', 'addr': '172.21.15.179:6829', 'nonce': 2899402373}]}, 'public_addr': '172.21.15.179:6825/2899402373', 'cluster_addr': '172.21.15.179:6827/2899402373', 'heartbeat_back_addr': '172.21.15.179:6831/2899402373', 'heartbeat_front_addr': '172.21.15.179:6829/2899402373', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:45:25.864115+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:45:40.613496+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:45:13.654372+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:44:29.864171+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:45:09.441700+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:45:04.754298+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:44:50.442694+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:44:26.159633+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:45:17.431302+0000', 'dead_epoch': 145}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:45:03.406185+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:45:27.199188+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:45:39.014845+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.99:0/3904172782': '2024-02-10T17:16:24.229365+0000', '172.21.15.99:6801/1586034576': '2024-02-10T17:16:24.229365+0000', '172.21.15.99:6800/746248609': '2024-02-10T17:15:54.399201+0000', '172.21.15.99:0/4092784197': '2024-02-10T17:16:24.229365+0000', '172.21.15.99:0/2844598101': '2024-02-10T17:15:54.399201+0000', '172.21.15.99:6801/746248609': '2024-02-10T17:15:54.399201+0000', '172.21.15.99:0/2528750050': '2024-02-10T17:15:54.399201+0000', '172.21.15.99:6800/1586034576': '2024-02-10T17:16:24.229365+0000', '172.21.15.99:0/2214140078': '2024-02-10T17:15:54.399201+0000', '172.21.15.99:0/4217821965': '2024-02-10T17:16:24.229365+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-09T17:45:42.955 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-02-09T17:45:43.184 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:45:43.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:43.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:45:43.208 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-09T17:45:43.289 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:45:43.320 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:43.418 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:45:43.421 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:43.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:45:43.513 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:45:43.522 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:45:43.550 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:43.573 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:45:43.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:43.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:45:43.676 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:45:43.706 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:43.717 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:45:43.719 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:45:43.726 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:45:43.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:43.729 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:45:43.742 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:45:43.762 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:45:43.781 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:45:43.801 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:45:43.820 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:45:43.830 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:45:43.841 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:45:43.855 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:43 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:45:43.855 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:43 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:45:43.855 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:43 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/859869092' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-02-09T17:45:43.856 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:43 smithi179 ceph-mon[43975]: osdmap e576: 12 total, 12 up, 12 in 2024-02-09T17:45:43.856 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:43 smithi179 ceph-mon[43975]: pgmap v1445: 47 pgs: 16 unknown, 31 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:45:43.856 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:43 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:45:43.856 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:43 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:45:43.857 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:43.860 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:45:43.877 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:45:43.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:43.880 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:45:43.880 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:45:43.900 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:45:43.919 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:45:43.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:43 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:45:43.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:43 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:45:43.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:43 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/859869092' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-02-09T17:45:43.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:43 smithi162 ceph-mon[44304]: osdmap e576: 12 total, 12 up, 12 in 2024-02-09T17:45:43.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:43 smithi162 ceph-mon[44304]: pgmap v1445: 47 pgs: 16 unknown, 31 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:45:43.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:43 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:45:43.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:43 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:45:43.939 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:45:43.941 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:45:43.941 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:45:43.941 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:45:43.941 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:45:43.941 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:45:43.941 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:45:43.941 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:45:43.941 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:45:43.941 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:45:43.941 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:45:43.942 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:45:43.942 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:45:43.960 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:43 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:45:43.960 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:43 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:45:43.960 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:43 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/859869092' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-02-09T17:45:43.960 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:43 smithi099 ceph-mon[39420]: osdmap e576: 12 total, 12 up, 12 in 2024-02-09T17:45:43.960 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:43 smithi099 ceph-mon[39420]: pgmap v1445: 47 pgs: 16 unknown, 31 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:45:43.960 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:43 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:45:43.960 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:43 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:45:43.981 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:45:44.009 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:44.203 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-02-09T17:45:44.791 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:44 smithi162 ceph-mon[44304]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-09T17:45:44.791 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:44 smithi162 ceph-mon[44304]: osdmap e577: 12 total, 12 up, 12 in 2024-02-09T17:45:44.791 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:44 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3617869004' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:45:44.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:44 smithi179 ceph-mon[43975]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-09T17:45:44.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:44 smithi179 ceph-mon[43975]: osdmap e577: 12 total, 12 up, 12 in 2024-02-09T17:45:44.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:44 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3617869004' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:45:44.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:44 smithi099 ceph-mon[39420]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-09T17:45:44.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:44 smithi099 ceph-mon[39420]: osdmap e577: 12 total, 12 up, 12 in 2024-02-09T17:45:44.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:44 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3617869004' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:45:45.037 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:45:45.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:45.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:45:45.141 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:45:45.169 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:45.267 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:45:45.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:45.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:45:45.335 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:45:45.335 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-02-09T17:45:45.335 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-02-09T17:45:45.335 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-02-09T17:45:45.335 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.336 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25201.0:208407 4.e 4:7bdf9d69:::benchmark_data_smithi179_103809_object70941:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-02-09T17:45:45.336 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:38:33.813701+0000", 2024-02-09T17:45:45.336 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 431.51807206000001, 2024-02-09T17:45:45.336 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2211754350000001, 2024-02-09T17:45:45.336 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:45:45.336 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-09T17:45:45.336 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:45:45.336 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25201", 2024-02-09T17:45:45.336 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:45:45.336 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 208407 2024-02-09T17:45:45.336 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.336 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:45:45.336 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.337 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:45:45.337 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:33.813701+0000", 2024-02-09T17:45:45.337 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:45.337 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.337 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.337 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:45:45.337 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:33.813701+0000", 2024-02-09T17:45:45.337 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:45.337 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.337 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.337 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:45:45.337 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:33.813703+0000", 2024-02-09T17:45:45.337 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6479999999999999e-06 2024-02-09T17:45:45.338 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.338 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.338 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:45:45.338 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:33.813709+0000", 2024-02-09T17:45:45.338 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9800000000000003e-06 2024-02-09T17:45:45.338 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.338 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.338 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:45:45.338 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:33.813726+0000", 2024-02-09T17:45:45.338 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6872999999999999e-05 2024-02-09T17:45:45.338 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.338 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.338 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:45:45.338 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:33.813736+0000", 2024-02-09T17:45:45.339 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8600000000000005e-06 2024-02-09T17:45:45.339 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.339 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.339 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:45.339 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:33.814238+0000", 2024-02-09T17:45:45.339 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00050189199999999998 2024-02-09T17:45:45.339 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.339 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.339 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:45:45.339 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:33.814346+0000", 2024-02-09T17:45:45.339 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000108315 2024-02-09T17:45:45.339 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.339 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.339 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,10", 2024-02-09T17:45:45.340 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:33.814458+0000", 2024-02-09T17:45:45.340 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011234 2024-02-09T17:45:45.340 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.340 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.340 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:45:45.340 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:33.820137+0000", 2024-02-09T17:45:45.340 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0056781569999999996 2024-02-09T17:45:45.340 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.340 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.340 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:45.340 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:33.833664+0000", 2024-02-09T17:45:45.340 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013527617 2024-02-09T17:45:45.340 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.341 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.341 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:45:45.341 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:33.833682+0000", 2024-02-09T17:45:45.341 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7796999999999999e-05 2024-02-09T17:45:45.341 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.341 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.341 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:45.341 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:34.815121+0000", 2024-02-09T17:45:45.341 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.98143867200000001 2024-02-09T17:45:45.341 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.341 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.341 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:45:45.341 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:34.815135+0000", 2024-02-09T17:45:45.341 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4551e-05 2024-02-09T17:45:45.342 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.342 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.342 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:45.342 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:36.017611+0000", 2024-02-09T17:45:45.342 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.202475483 2024-02-09T17:45:45.342 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.342 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.342 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for missing object", 2024-02-09T17:45:45.342 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:36.017774+0000", 2024-02-09T17:45:45.342 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016380999999999999 2024-02-09T17:45:45.342 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.342 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.342 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:45.342 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:36.034802+0000", 2024-02-09T17:45:45.343 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.017027706 2024-02-09T17:45:45.343 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.343 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.343 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:45:45.343 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:38:36.034877+0000", 2024-02-09T17:45:45.343 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4734000000000006e-05 2024-02-09T17:45:45.343 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:45.343 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:45:45.343 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:45.343 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.343 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.343 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25201.0:222259 4.e 4:79e2d474:::benchmark_data_smithi179_103809_object84793:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e397)", 2024-02-09T17:45:45.343 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:39:02.584331+0000", 2024-02-09T17:45:45.343 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 402.747442455, 2024-02-09T17:45:45.343 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2140263610000002, 2024-02-09T17:45:45.344 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:45:45.344 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-09T17:45:45.344 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:45:45.344 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25201", 2024-02-09T17:45:45.344 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:45:45.344 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 222259 2024-02-09T17:45:45.344 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.344 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:45:45.344 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.344 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:45:45.344 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.584331+0000", 2024-02-09T17:45:45.344 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:45.344 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.344 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.345 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:45:45.345 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.584331+0000", 2024-02-09T17:45:45.345 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:45.345 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.345 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.345 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:45:45.345 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.584332+0000", 2024-02-09T17:45:45.345 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9900000000000001e-07 2024-02-09T17:45:45.345 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.345 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.345 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:45:45.345 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.584335+0000", 2024-02-09T17:45:45.345 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.151e-06 2024-02-09T17:45:45.345 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.345 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.346 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:45:45.346 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.584343+0000", 2024-02-09T17:45:45.346 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0409999999999998e-06 2024-02-09T17:45:45.346 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.346 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.346 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:45:45.346 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.584349+0000", 2024-02-09T17:45:45.346 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6030000000000001e-06 2024-02-09T17:45:45.346 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.346 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.346 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:45.346 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.584413+0000", 2024-02-09T17:45:45.346 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3984000000000003e-05 2024-02-09T17:45:45.346 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.346 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.347 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:45:45.347 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.584567+0000", 2024-02-09T17:45:45.347 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000153969 2024-02-09T17:45:45.347 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.347 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.347 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,10", 2024-02-09T17:45:45.347 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.584688+0000", 2024-02-09T17:45:45.347 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012019399999999999 2024-02-09T17:45:45.347 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.347 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.347 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:45.347 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.590409+0000", 2024-02-09T17:45:45.347 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0057209849999999996 2024-02-09T17:45:45.347 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.347 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.348 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:45:45.348 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:02.590418+0000", 2024-02-09T17:45:45.348 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7999999999999993e-06 2024-02-09T17:45:45.348 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.348 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.348 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:45.348 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.598122+0000", 2024-02-09T17:45:45.348 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.007703708 2024-02-09T17:45:45.348 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.348 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.348 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:45:45.348 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.598142+0000", 2024-02-09T17:45:45.348 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9797e-05 2024-02-09T17:45:45.348 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.349 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.349 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:45.349 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.798282+0000", 2024-02-09T17:45:45.349 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2001401380000001 2024-02-09T17:45:45.349 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.349 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.349 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:45:45.349 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.798357+0000", 2024-02-09T17:45:45.349 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5492000000000002e-05 2024-02-09T17:45:45.349 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:45.349 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:45:45.349 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:45.349 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.349 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.350 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25201.0:222287 4.e 4:7c7bef54:::benchmark_data_smithi179_103809_object84821:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e399)", 2024-02-09T17:45:45.350 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:39:03.641509+0000", 2024-02-09T17:45:45.350 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 401.69026458899998, 2024-02-09T17:45:45.350 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.168818505, 2024-02-09T17:45:45.350 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:45:45.350 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:45:45.350 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:45:45.350 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25201", 2024-02-09T17:45:45.350 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:45:45.350 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 222287 2024-02-09T17:45:45.350 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.350 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:45:45.350 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.350 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:45:45.350 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.641509+0000", 2024-02-09T17:45:45.351 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:45.351 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.351 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.351 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:45:45.351 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.641509+0000", 2024-02-09T17:45:45.351 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:45.351 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.351 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.351 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:45:45.351 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.641510+0000", 2024-02-09T17:45:45.351 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4890000000000001e-06 2024-02-09T17:45:45.351 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.351 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.351 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:45:45.352 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.641515+0000", 2024-02-09T17:45:45.352 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7809999999999996e-06 2024-02-09T17:45:45.352 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.352 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.352 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:45:45.352 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.641526+0000", 2024-02-09T17:45:45.352 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1060000000000001e-05 2024-02-09T17:45:45.352 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.352 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.352 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:45:45.352 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.641535+0000", 2024-02-09T17:45:45.352 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5180000000000004e-06 2024-02-09T17:45:45.352 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.353 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.353 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:45.353 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.641605+0000", 2024-02-09T17:45:45.353 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0267999999999999e-05 2024-02-09T17:45:45.353 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.353 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.353 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:45:45.353 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.641618+0000", 2024-02-09T17:45:45.353 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2862e-05 2024-02-09T17:45:45.353 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.353 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.353 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:45.353 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.798602+0000", 2024-02-09T17:45:45.353 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.15698368 2024-02-09T17:45:45.354 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.354 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.354 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:45:45.354 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.798717+0000", 2024-02-09T17:45:45.354 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000115429 2024-02-09T17:45:45.354 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.354 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.354 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,5,10", 2024-02-09T17:45:45.354 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.798851+0000", 2024-02-09T17:45:45.354 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013354299999999999 2024-02-09T17:45:45.354 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.354 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.354 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:45.354 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.809916+0000", 2024-02-09T17:45:45.354 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011065042000000001 2024-02-09T17:45:45.355 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.355 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.355 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:45.355 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.809959+0000", 2024-02-09T17:45:45.355 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3517999999999999e-05 2024-02-09T17:45:45.355 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.355 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.355 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:45.355 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.810001+0000", 2024-02-09T17:45:45.355 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1732e-05 2024-02-09T17:45:45.355 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.355 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.355 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:45:45.355 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.810144+0000", 2024-02-09T17:45:45.356 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014349400000000001 2024-02-09T17:45:45.356 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.356 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.356 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:45:45.356 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.810204+0000", 2024-02-09T17:45:45.356 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9463000000000003e-05 2024-02-09T17:45:45.356 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.356 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.356 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:45:45.356 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.810327+0000", 2024-02-09T17:45:45.357 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000123626 2024-02-09T17:45:45.357 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:45.357 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:45:45.357 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:45.357 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.357 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.357 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25201.0:222290 4.e 4:7ebcc15e:::benchmark_data_smithi179_103809_object84824:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e399)", 2024-02-09T17:45:45.357 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:39:03.647256+0000", 2024-02-09T17:45:45.357 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 401.68451733299997, 2024-02-09T17:45:45.357 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.163461198, 2024-02-09T17:45:45.357 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:45:45.357 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:45:45.357 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:45:45.357 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25201", 2024-02-09T17:45:45.358 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:45:45.358 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 222290 2024-02-09T17:45:45.358 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.358 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:45:45.358 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.358 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:45:45.358 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.647256+0000", 2024-02-09T17:45:45.358 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:45.358 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.358 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.358 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:45:45.358 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.647256+0000", 2024-02-09T17:45:45.358 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:45.358 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.359 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.359 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:45:45.359 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.647257+0000", 2024-02-09T17:45:45.359 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0359999999999999e-06 2024-02-09T17:45:45.359 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.359 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.359 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:45:45.359 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.647262+0000", 2024-02-09T17:45:45.359 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8049999999999997e-06 2024-02-09T17:45:45.359 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.359 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.359 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:45:45.359 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.647275+0000", 2024-02-09T17:45:45.359 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2788e-05 2024-02-09T17:45:45.359 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.360 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.360 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:45:45.360 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.647284+0000", 2024-02-09T17:45:45.360 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9819999999999997e-06 2024-02-09T17:45:45.360 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.360 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.360 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:45.360 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.647330+0000", 2024-02-09T17:45:45.360 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5924999999999998e-05 2024-02-09T17:45:45.360 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.360 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.360 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:45:45.360 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:03.647337+0000", 2024-02-09T17:45:45.361 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3209999999999999e-06 2024-02-09T17:45:45.361 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.361 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.361 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:45.361 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.804329+0000", 2024-02-09T17:45:45.361 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.156992231 2024-02-09T17:45:45.361 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.361 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.361 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:45:45.361 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.804477+0000", 2024-02-09T17:45:45.361 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014802899999999999 2024-02-09T17:45:45.361 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.361 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.361 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,5,10", 2024-02-09T17:45:45.361 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.804646+0000", 2024-02-09T17:45:45.361 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000168853 2024-02-09T17:45:45.362 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.362 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.362 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:45.362 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.810042+0000", 2024-02-09T17:45:45.362 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0053958519999999996 2024-02-09T17:45:45.362 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.362 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.362 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:45.362 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.810083+0000", 2024-02-09T17:45:45.362 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0639000000000002e-05 2024-02-09T17:45:45.362 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.362 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.362 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:45.362 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.810126+0000", 2024-02-09T17:45:45.363 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2905999999999999e-05 2024-02-09T17:45:45.363 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.363 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.363 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:45:45.363 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.810565+0000", 2024-02-09T17:45:45.363 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00043988300000000002 2024-02-09T17:45:45.363 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.363 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.363 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:45:45.363 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.810617+0000", 2024-02-09T17:45:45.363 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1560000000000001e-05 2024-02-09T17:45:45.363 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.363 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.363 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:45:45.364 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:04.810717+0000", 2024-02-09T17:45:45.364 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000100388 2024-02-09T17:45:45.364 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:45.364 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:45:45.364 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:45.364 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.364 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.364 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25201.0:230624 4.e 4:7f23c367:::benchmark_data_smithi179_103809_object93158:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e411)", 2024-02-09T17:45:45.364 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:39:19.517268+0000", 2024-02-09T17:45:45.364 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 385.814505483, 2024-02-09T17:45:45.364 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0345313949999999, 2024-02-09T17:45:45.364 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:45:45.364 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:45:45.365 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:45:45.365 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25201", 2024-02-09T17:45:45.365 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:45:45.365 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 230624 2024-02-09T17:45:45.365 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.365 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:45:45.365 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.365 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:45:45.365 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:19.517268+0000", 2024-02-09T17:45:45.365 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:45.365 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.365 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.365 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:45:45.365 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:19.517268+0000", 2024-02-09T17:45:45.366 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:45.366 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.366 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.366 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:45:45.366 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:19.517269+0000", 2024-02-09T17:45:45.366 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2079999999999999e-06 2024-02-09T17:45:45.366 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.366 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.366 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:45:45.366 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:19.517274+0000", 2024-02-09T17:45:45.366 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1000000000000003e-06 2024-02-09T17:45:45.366 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.366 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.366 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:45:45.367 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:19.517285+0000", 2024-02-09T17:45:45.367 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0173000000000001e-05 2024-02-09T17:45:45.367 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.367 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.367 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:45:45.367 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:19.517292+0000", 2024-02-09T17:45:45.367 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5240000000000003e-06 2024-02-09T17:45:45.367 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.367 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.367 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:45.367 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:19.517350+0000", 2024-02-09T17:45:45.367 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7605999999999997e-05 2024-02-09T17:45:45.367 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.367 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.367 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:45:45.368 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:19.517356+0000", 2024-02-09T17:45:45.368 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0290000000000002e-06 2024-02-09T17:45:45.368 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.368 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.368 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:45.368 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:20.536023+0000", 2024-02-09T17:45:45.368 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.01866758 2024-02-09T17:45:45.368 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.368 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.368 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:45:45.368 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:20.536138+0000", 2024-02-09T17:45:45.368 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000114258 2024-02-09T17:45:45.368 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.368 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.369 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5,10", 2024-02-09T17:45:45.369 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:20.536264+0000", 2024-02-09T17:45:45.369 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000126663 2024-02-09T17:45:45.369 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.369 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.369 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:45.369 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:20.551552+0000", 2024-02-09T17:45:45.369 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015288047000000001 2024-02-09T17:45:45.369 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.369 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.369 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:45:45.369 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:20.551567+0000", 2024-02-09T17:45:45.369 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4739999999999999e-05 2024-02-09T17:45:45.369 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.370 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.370 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:45.370 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:20.551689+0000", 2024-02-09T17:45:45.370 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012160899999999999 2024-02-09T17:45:45.370 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.370 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.370 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:45:45.370 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:20.551749+0000", 2024-02-09T17:45:45.370 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0665000000000003e-05 2024-02-09T17:45:45.370 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.370 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.370 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:45:45.370 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:20.551799+0000", 2024-02-09T17:45:45.370 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0192999999999999e-05 2024-02-09T17:45:45.371 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:45.371 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:45:45.371 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:45.371 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.371 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.371 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25201.0:230657 4.e 4:7f594855:::benchmark_data_smithi179_103809_object93191:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e411)", 2024-02-09T17:45:45.371 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:39:19.565283+0000", 2024-02-09T17:45:45.371 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 385.76649018000001, 2024-02-09T17:45:45.371 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99722966499999999, 2024-02-09T17:45:45.371 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:45:45.371 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:45:45.371 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:45:45.371 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25201", 2024-02-09T17:45:45.372 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:45:45.372 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 230657 2024-02-09T17:45:45.372 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.372 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:45:45.372 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.372 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:45:45.372 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:19.565283+0000", 2024-02-09T17:45:45.372 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:45.372 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.372 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.372 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:45:45.372 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:19.565283+0000", 2024-02-09T17:45:45.372 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:45.373 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.373 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.373 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:45:45.373 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:19.565284+0000", 2024-02-09T17:45:45.373 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3200000000000004e-07 2024-02-09T17:45:45.373 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.373 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.373 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:45:45.373 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:19.565289+0000", 2024-02-09T17:45:45.373 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5299999999999998e-06 2024-02-09T17:45:45.373 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.373 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.373 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:45:45.373 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:19.565300+0000", 2024-02-09T17:45:45.373 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0797e-05 2024-02-09T17:45:45.374 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.374 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.374 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:45:45.374 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:19.565307+0000", 2024-02-09T17:45:45.374 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7300000000000005e-06 2024-02-09T17:45:45.374 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.374 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.374 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:45.374 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:19.565421+0000", 2024-02-09T17:45:45.374 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000114135 2024-02-09T17:45:45.374 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.374 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.374 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:45:45.374 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:19.565432+0000", 2024-02-09T17:45:45.375 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0125999999999999e-05 2024-02-09T17:45:45.375 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.375 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.375 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:45.375 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:20.541301+0000", 2024-02-09T17:45:45.375 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.975869498 2024-02-09T17:45:45.375 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.375 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.375 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:45:45.375 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:20.541428+0000", 2024-02-09T17:45:45.375 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012745899999999999 2024-02-09T17:45:45.375 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.375 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.375 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5,10", 2024-02-09T17:45:45.376 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:20.541554+0000", 2024-02-09T17:45:45.376 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000125322 2024-02-09T17:45:45.376 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.376 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.376 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:45:45.376 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:20.551647+0000", 2024-02-09T17:45:45.376 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010093098999999999 2024-02-09T17:45:45.376 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.376 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.376 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:45.376 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:20.551850+0000", 2024-02-09T17:45:45.376 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020347200000000001 2024-02-09T17:45:45.376 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.377 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.377 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:45.377 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:20.562452+0000", 2024-02-09T17:45:45.377 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010601413000000001 2024-02-09T17:45:45.377 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.377 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.377 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:45:45.377 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:20.562487+0000", 2024-02-09T17:45:45.377 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4958e-05 2024-02-09T17:45:45.377 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.377 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.377 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:45:45.377 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:20.562513+0000", 2024-02-09T17:45:45.377 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6293999999999999e-05 2024-02-09T17:45:45.377 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:45.377 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:45:45.378 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:45.378 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.378 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.378 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25201.0:232563 4.e 4:7b844ab9:::benchmark_data_smithi179_103809_object95097:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e414)", 2024-02-09T17:45:45.378 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:39:23.623236+0000", 2024-02-09T17:45:45.378 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 381.70853732500001, 2024-02-09T17:45:45.378 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1374754500000002, 2024-02-09T17:45:45.378 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:45:45.378 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:45:45.378 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:45:45.378 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25201", 2024-02-09T17:45:45.378 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:45:45.378 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 232563 2024-02-09T17:45:45.378 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.379 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:45:45.379 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.379 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:45:45.379 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.623236+0000", 2024-02-09T17:45:45.379 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:45.379 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.379 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.379 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:45:45.379 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.623236+0000", 2024-02-09T17:45:45.379 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:45.379 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.379 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.379 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:45:45.379 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.623237+0000", 2024-02-09T17:45:45.380 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7599999999999996e-07 2024-02-09T17:45:45.380 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.380 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.380 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:45:45.380 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.623244+0000", 2024-02-09T17:45:45.380 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5339999999999996e-06 2024-02-09T17:45:45.380 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.380 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.380 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:45:45.380 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.623253+0000", 2024-02-09T17:45:45.380 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6639999999999993e-06 2024-02-09T17:45:45.380 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.380 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.380 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:45:45.381 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.623260+0000", 2024-02-09T17:45:45.381 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9e-06 2024-02-09T17:45:45.381 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.381 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.381 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:45.381 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.623328+0000", 2024-02-09T17:45:45.381 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8103999999999997e-05 2024-02-09T17:45:45.381 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.381 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.381 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:45:45.381 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.623340+0000", 2024-02-09T17:45:45.381 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1479e-05 2024-02-09T17:45:45.381 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.381 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.382 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:45.382 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:24.532124+0000", 2024-02-09T17:45:45.382 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.90878441499999996 2024-02-09T17:45:45.382 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.382 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.382 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:45:45.382 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:24.532137+0000", 2024-02-09T17:45:45.382 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2692e-05 2024-02-09T17:45:45.382 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.382 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.382 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:45.382 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.756022+0000", 2024-02-09T17:45:45.382 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2238848529999999 2024-02-09T17:45:45.382 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.383 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.383 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:45:45.383 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.756142+0000", 2024-02-09T17:45:45.383 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000120562 2024-02-09T17:45:45.383 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.383 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.383 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,5,10", 2024-02-09T17:45:45.383 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.756238+0000", 2024-02-09T17:45:45.383 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5424e-05 2024-02-09T17:45:45.383 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.383 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.383 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:45.383 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.760515+0000", 2024-02-09T17:45:45.383 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042770830000000001 2024-02-09T17:45:45.383 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.384 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.384 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:45.384 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.760566+0000", 2024-02-09T17:45:45.384 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1172999999999999e-05 2024-02-09T17:45:45.384 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.384 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.384 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:45:45.384 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.760578+0000", 2024-02-09T17:45:45.384 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1693e-05 2024-02-09T17:45:45.384 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.384 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.384 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:45.384 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.760606+0000", 2024-02-09T17:45:45.384 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8138999999999999e-05 2024-02-09T17:45:45.385 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.385 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.385 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:45:45.385 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.760644+0000", 2024-02-09T17:45:45.385 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7966000000000002e-05 2024-02-09T17:45:45.385 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.385 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.385 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:45:45.385 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.760712+0000", 2024-02-09T17:45:45.385 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7992999999999998e-05 2024-02-09T17:45:45.385 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:45.385 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:45:45.385 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:45.385 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.385 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.386 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25201.0:232572 4.e 4:7679dda5:::benchmark_data_smithi179_103809_object95106:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e414)", 2024-02-09T17:45:45.386 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:39:23.637905+0000", 2024-02-09T17:45:45.386 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 381.69386840599998, 2024-02-09T17:45:45.386 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1455606629999999, 2024-02-09T17:45:45.386 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:45:45.386 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:45:45.386 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:45:45.386 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25201", 2024-02-09T17:45:45.386 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:45:45.386 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 232572 2024-02-09T17:45:45.386 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.386 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:45:45.386 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.386 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:45:45.386 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.637905+0000", 2024-02-09T17:45:45.387 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:45.387 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.387 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.387 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:45:45.387 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.637905+0000", 2024-02-09T17:45:45.387 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:45.387 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.387 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.387 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:45:45.387 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.637906+0000", 2024-02-09T17:45:45.387 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8000000000000004e-07 2024-02-09T17:45:45.387 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.387 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.387 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:45:45.387 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.637910+0000", 2024-02-09T17:45:45.388 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2209999999999999e-06 2024-02-09T17:45:45.388 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.388 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.388 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:45:45.388 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.637923+0000", 2024-02-09T17:45:45.388 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2279999999999999e-05 2024-02-09T17:45:45.388 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.388 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.388 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:45:45.388 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.637930+0000", 2024-02-09T17:45:45.388 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5379999999999997e-06 2024-02-09T17:45:45.388 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.388 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.388 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:45.389 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.637990+0000", 2024-02-09T17:45:45.389 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0269999999999997e-05 2024-02-09T17:45:45.389 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.389 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.389 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:45:45.389 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:23.638000+0000", 2024-02-09T17:45:45.389 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6350000000000001e-06 2024-02-09T17:45:45.389 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.389 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.389 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:45.389 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:24.532226+0000", 2024-02-09T17:45:45.389 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.89422645700000003 2024-02-09T17:45:45.389 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.389 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.389 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:45:45.390 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:24.532242+0000", 2024-02-09T17:45:45.390 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5114e-05 2024-02-09T17:45:45.390 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.390 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.390 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:45.390 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.760842+0000", 2024-02-09T17:45:45.390 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2286009120000001 2024-02-09T17:45:45.390 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.390 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.390 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:45:45.390 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.760940+0000", 2024-02-09T17:45:45.390 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7262000000000005e-05 2024-02-09T17:45:45.390 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.390 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.390 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,5,10", 2024-02-09T17:45:45.391 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.761020+0000", 2024-02-09T17:45:45.391 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0433999999999996e-05 2024-02-09T17:45:45.391 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.391 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.391 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:45.391 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.769371+0000", 2024-02-09T17:45:45.391 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0083504789999999992 2024-02-09T17:45:45.391 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.391 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.391 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:45:45.391 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.769381+0000", 2024-02-09T17:45:45.391 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9199999999999999e-06 2024-02-09T17:45:45.391 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.391 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.391 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:45.392 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.774049+0000", 2024-02-09T17:45:45.392 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0046689549999999998 2024-02-09T17:45:45.392 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.392 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.392 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:45.392 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.783345+0000", 2024-02-09T17:45:45.392 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0092952850000000004 2024-02-09T17:45:45.392 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.392 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.392 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:45:45.392 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.783385+0000", 2024-02-09T17:45:45.392 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0299999999999997e-05 2024-02-09T17:45:45.392 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.392 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.393 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:45:45.393 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.783466+0000", 2024-02-09T17:45:45.393 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0721000000000002e-05 2024-02-09T17:45:45.393 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:45.393 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:45:45.393 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:45.393 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.393 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.393 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25201.0:232591 4.e 4:74b9959f:::benchmark_data_smithi179_103809_object95125:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-02-09T17:45:45.393 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:39:24.727541+0000", 2024-02-09T17:45:45.393 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 380.60423257399998, 2024-02-09T17:45:45.393 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0664996879999999, 2024-02-09T17:45:45.393 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:45:45.393 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:45:45.393 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:45:45.394 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25201", 2024-02-09T17:45:45.394 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:45:45.394 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 232591 2024-02-09T17:45:45.394 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.394 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:45:45.394 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.394 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:45:45.394 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:24.727541+0000", 2024-02-09T17:45:45.394 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:45.394 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.394 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.394 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:45:45.394 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:24.727541+0000", 2024-02-09T17:45:45.394 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:45.394 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.395 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.395 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:45:45.395 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:24.727542+0000", 2024-02-09T17:45:45.395 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3510000000000001e-06 2024-02-09T17:45:45.395 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.395 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.395 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:45:45.395 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:24.727548+0000", 2024-02-09T17:45:45.395 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5659999999999997e-06 2024-02-09T17:45:45.395 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.395 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.395 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:45:45.395 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:24.727560+0000", 2024-02-09T17:45:45.395 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1902000000000001e-05 2024-02-09T17:45:45.395 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.396 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.396 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:45:45.396 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:24.727569+0000", 2024-02-09T17:45:45.396 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7790000000000001e-06 2024-02-09T17:45:45.396 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.396 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.396 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:45.396 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:24.727624+0000", 2024-02-09T17:45:45.396 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5831000000000001e-05 2024-02-09T17:45:45.396 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.396 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.396 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:45:45.396 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:24.727632+0000", 2024-02-09T17:45:45.396 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9319999999999996e-06 2024-02-09T17:45:45.396 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.397 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.397 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:45.397 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.764987+0000", 2024-02-09T17:45:45.397 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0373542650000001 2024-02-09T17:45:45.397 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.397 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.397 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:45:45.397 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.765082+0000", 2024-02-09T17:45:45.397 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4952999999999997e-05 2024-02-09T17:45:45.397 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.397 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.397 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,5,10", 2024-02-09T17:45:45.397 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.765165+0000", 2024-02-09T17:45:45.397 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3306000000000005e-05 2024-02-09T17:45:45.397 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.398 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.398 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:45.398 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.769420+0000", 2024-02-09T17:45:45.398 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042556549999999997 2024-02-09T17:45:45.398 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.398 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.398 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:45:45.398 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.774008+0000", 2024-02-09T17:45:45.398 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0045875550000000001 2024-02-09T17:45:45.398 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.398 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.398 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:45.398 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.774092+0000", 2024-02-09T17:45:45.398 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4079000000000006e-05 2024-02-09T17:45:45.398 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.399 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.399 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:45.399 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.793926+0000", 2024-02-09T17:45:45.399 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019833871999999999 2024-02-09T17:45:45.399 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.399 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.399 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:45:45.399 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.793968+0000", 2024-02-09T17:45:45.399 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1757e-05 2024-02-09T17:45:45.399 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.399 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.399 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:45:45.399 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:25.794041+0000", 2024-02-09T17:45:45.399 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2885000000000004e-05 2024-02-09T17:45:45.399 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:45.400 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:45:45.400 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:45.400 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.400 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.400 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25201.0:254074 4.c 4.79812fac (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-02-09T17:45:45.400 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:39:57.530922+0000", 2024-02-09T17:45:45.400 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 347.80085184699999, 2024-02-09T17:45:45.400 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0069641279999999, 2024-02-09T17:45:45.400 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:45:45.400 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-09T17:45:45.400 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:45:45.400 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25201", 2024-02-09T17:45:45.400 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:45:45.400 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 254074 2024-02-09T17:45:45.400 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.401 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:45:45.401 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.401 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:45:45.401 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:57.530922+0000", 2024-02-09T17:45:45.401 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:45.401 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.401 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.401 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:45:45.401 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:57.530922+0000", 2024-02-09T17:45:45.401 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:45.401 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.401 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.401 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:45:45.401 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:57.530923+0000", 2024-02-09T17:45:45.401 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.406e-06 2024-02-09T17:45:45.402 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.402 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.402 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:45:45.402 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:57.530928+0000", 2024-02-09T17:45:45.402 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.164e-06 2024-02-09T17:45:45.402 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.402 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.402 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:45:45.402 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:57.530939+0000", 2024-02-09T17:45:45.402 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1208e-05 2024-02-09T17:45:45.402 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.402 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.402 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:45:45.402 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:57.530946+0000", 2024-02-09T17:45:45.403 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0330000000000003e-06 2024-02-09T17:45:45.403 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.403 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.403 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:45.403 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:57.530997+0000", 2024-02-09T17:45:45.403 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0145999999999997e-05 2024-02-09T17:45:45.403 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.403 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.403 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:45:45.403 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:57.531006+0000", 2024-02-09T17:45:45.403 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0219999999999993e-06 2024-02-09T17:45:45.403 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.403 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.403 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:45.403 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:58.537872+0000", 2024-02-09T17:45:45.404 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.006866217 2024-02-09T17:45:45.404 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.404 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.404 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:45:45.404 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:58.537886+0000", 2024-02-09T17:45:45.404 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3932e-05 2024-02-09T17:45:45.404 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:45.404 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:45:45.404 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:45.404 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.404 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.404 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25201.0:254123 4.c 4.882473c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-02-09T17:45:45.404 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:39:57.580994+0000", 2024-02-09T17:45:45.404 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 347.75077956799998, 2024-02-09T17:45:45.405 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.95720749500000002, 2024-02-09T17:45:45.405 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:45:45.405 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-09T17:45:45.405 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:45:45.405 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25201", 2024-02-09T17:45:45.405 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:45:45.405 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 254123 2024-02-09T17:45:45.405 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.405 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:45:45.405 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.405 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:45:45.405 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:57.580994+0000", 2024-02-09T17:45:45.405 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:45.405 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.405 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.406 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:45:45.406 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:57.580994+0000", 2024-02-09T17:45:45.406 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:45.406 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.406 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.406 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:45:45.406 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:57.580995+0000", 2024-02-09T17:45:45.406 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2900000000000003e-07 2024-02-09T17:45:45.406 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.406 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.406 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:45:45.406 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:57.580997+0000", 2024-02-09T17:45:45.406 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8720000000000002e-06 2024-02-09T17:45:45.406 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.406 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.407 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:45:45.407 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:57.581008+0000", 2024-02-09T17:45:45.407 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0393e-05 2024-02-09T17:45:45.407 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.407 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.407 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:45:45.407 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:57.581013+0000", 2024-02-09T17:45:45.407 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.631e-06 2024-02-09T17:45:45.407 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.407 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.407 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:45.407 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:57.581069+0000", 2024-02-09T17:45:45.407 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5433000000000003e-05 2024-02-09T17:45:45.407 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.407 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.408 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:45:45.408 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:57.581079+0000", 2024-02-09T17:45:45.408 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0287e-05 2024-02-09T17:45:45.408 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.408 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.408 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:45.408 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:58.538193+0000", 2024-02-09T17:45:45.408 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.95711387699999995 2024-02-09T17:45:45.408 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.408 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.408 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:45:45.408 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:39:58.538201+0000", 2024-02-09T17:45:45.408 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3729999999999993e-06 2024-02-09T17:45:45.408 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:45.409 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:45:45.409 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:45.409 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.409 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.409 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:144271 5.1 5:88bc6d29:::benchmark_data_smithi179_115199_object15365:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e508)", 2024-02-09T17:45:45.409 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:42:51.809080+0000", 2024-02-09T17:45:45.409 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 173.52269317899999, 2024-02-09T17:45:45.409 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.061843216999999999, 2024-02-09T17:45:45.409 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:45:45.409 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:45:45.409 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:45:45.409 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:45:45.409 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:45:45.409 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 144271 2024-02-09T17:45:45.409 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.410 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:45:45.410 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.410 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:45:45.410 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.809080+0000", 2024-02-09T17:45:45.410 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:45.410 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.410 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.410 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:45:45.410 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.809080+0000", 2024-02-09T17:45:45.410 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:45.410 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.410 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.410 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:45:45.410 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.809081+0000", 2024-02-09T17:45:45.410 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.96e-07 2024-02-09T17:45:45.411 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.411 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.411 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:45:45.411 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.809082+0000", 2024-02-09T17:45:45.411 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7749999999999999e-06 2024-02-09T17:45:45.411 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.411 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.411 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:45:45.411 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.809087+0000", 2024-02-09T17:45:45.411 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8570000000000003e-06 2024-02-09T17:45:45.411 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.411 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.411 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:45:45.411 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.809092+0000", 2024-02-09T17:45:45.411 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4020000000000004e-06 2024-02-09T17:45:45.412 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.412 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.412 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:45.412 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.839410+0000", 2024-02-09T17:45:45.412 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.030318207 2024-02-09T17:45:45.412 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.412 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.412 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:45:45.412 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.839514+0000", 2024-02-09T17:45:45.412 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010382600000000001 2024-02-09T17:45:45.412 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.412 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.412 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,7", 2024-02-09T17:45:45.412 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.839610+0000", 2024-02-09T17:45:45.412 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5790000000000003e-05 2024-02-09T17:45:45.413 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.413 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.413 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:45:45.413 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.848584+0000", 2024-02-09T17:45:45.413 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0089741690000000006 2024-02-09T17:45:45.413 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.413 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.413 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:45.413 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.857483+0000", 2024-02-09T17:45:45.413 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0088992589999999996 2024-02-09T17:45:45.413 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.413 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.413 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:45.413 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.870877+0000", 2024-02-09T17:45:45.413 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013394015 2024-02-09T17:45:45.414 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.414 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.414 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:45:45.414 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.870904+0000", 2024-02-09T17:45:45.414 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7322999999999999e-05 2024-02-09T17:45:45.414 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.414 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.414 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:45:45.414 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.870924+0000", 2024-02-09T17:45:45.414 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9298000000000001e-05 2024-02-09T17:45:45.414 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:45.414 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:45:45.414 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:45.414 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.414 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.415 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:144278 5.1 5:85e0fe72:::benchmark_data_smithi179_115199_object15372:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e508)", 2024-02-09T17:45:45.415 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:42:51.809097+0000", 2024-02-09T17:45:45.415 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 173.52267700900001, 2024-02-09T17:45:45.415 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.061923522000000002, 2024-02-09T17:45:45.415 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:45:45.415 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:45:45.415 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:45:45.415 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:45:45.415 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:45:45.415 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 144278 2024-02-09T17:45:45.415 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.415 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:45:45.415 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.415 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:45:45.416 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.809097+0000", 2024-02-09T17:45:45.416 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:45.416 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.416 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.416 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:45:45.416 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.809097+0000", 2024-02-09T17:45:45.416 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:45.416 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.416 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.416 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:45:45.416 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.809097+0000", 2024-02-09T17:45:45.416 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6800000000000002e-07 2024-02-09T17:45:45.416 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.416 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.417 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:45:45.417 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.809099+0000", 2024-02-09T17:45:45.417 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.81e-06 2024-02-09T17:45:45.417 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.417 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.417 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:45:45.417 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.809104+0000", 2024-02-09T17:45:45.417 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9830000000000004e-06 2024-02-09T17:45:45.417 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.417 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.417 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:45:45.417 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.809108+0000", 2024-02-09T17:45:45.417 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3980000000000001e-06 2024-02-09T17:45:45.417 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.417 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.418 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:45.418 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.843980+0000", 2024-02-09T17:45:45.418 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.034871740999999998 2024-02-09T17:45:45.418 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.418 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.418 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:45:45.418 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.844106+0000", 2024-02-09T17:45:45.418 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012645399999999999 2024-02-09T17:45:45.418 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.418 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.418 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,7", 2024-02-09T17:45:45.418 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.844207+0000", 2024-02-09T17:45:45.418 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010092199999999999 2024-02-09T17:45:45.418 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.418 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.419 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:45:45.419 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.857500+0000", 2024-02-09T17:45:45.419 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.01329264 2024-02-09T17:45:45.419 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.419 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.419 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:45.419 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.866188+0000", 2024-02-09T17:45:45.419 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0086885169999999998 2024-02-09T17:45:45.419 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.419 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.419 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:45.419 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.870975+0000", 2024-02-09T17:45:45.419 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0047864930000000002 2024-02-09T17:45:45.419 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.419 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.420 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:45:45.420 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.871001+0000", 2024-02-09T17:45:45.420 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6367999999999999e-05 2024-02-09T17:45:45.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.420 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.420 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:45:45.420 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.871020+0000", 2024-02-09T17:45:45.420 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8927999999999999e-05 2024-02-09T17:45:45.420 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:45.420 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:45:45.420 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:45.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.420 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.420 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:144289 5.1 5:81127445:::benchmark_data_smithi179_115199_object15383:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e508)", 2024-02-09T17:45:45.421 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:42:51.809121+0000", 2024-02-09T17:45:45.421 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 173.52265234999999, 2024-02-09T17:45:45.421 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.072278163000000006, 2024-02-09T17:45:45.421 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:45:45.421 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:45:45.421 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:45:45.421 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:45:45.421 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:45:45.421 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 144289 2024-02-09T17:45:45.421 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.421 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:45:45.421 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.421 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:45:45.421 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.809121+0000", 2024-02-09T17:45:45.421 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:45.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.422 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:45:45.422 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.809121+0000", 2024-02-09T17:45:45.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:45.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.422 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:45:45.422 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.809122+0000", 2024-02-09T17:45:45.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5699999999999998e-07 2024-02-09T17:45:45.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.422 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:45:45.422 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.809123+0000", 2024-02-09T17:45:45.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8819999999999999e-06 2024-02-09T17:45:45.423 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.423 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.423 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:45:45.423 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.809129+0000", 2024-02-09T17:45:45.423 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3820000000000003e-06 2024-02-09T17:45:45.423 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.423 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.423 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:45:45.423 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.809133+0000", 2024-02-09T17:45:45.423 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.578e-06 2024-02-09T17:45:45.423 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.423 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.423 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:45.423 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.848646+0000", 2024-02-09T17:45:45.423 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.039512157999999999 2024-02-09T17:45:45.424 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.424 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.424 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:45:45.424 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.848754+0000", 2024-02-09T17:45:45.424 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000108529 2024-02-09T17:45:45.424 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.424 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.424 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,7", 2024-02-09T17:45:45.424 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.848852+0000", 2024-02-09T17:45:45.424 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7768000000000003e-05 2024-02-09T17:45:45.424 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.424 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.424 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:45:45.424 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.866015+0000", 2024-02-09T17:45:45.425 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.017163102999999999 2024-02-09T17:45:45.425 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.425 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.425 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:45.425 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.866225+0000", 2024-02-09T17:45:45.425 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021028 2024-02-09T17:45:45.425 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.425 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.425 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:45.425 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.881290+0000", 2024-02-09T17:45:45.425 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015064971 2024-02-09T17:45:45.425 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.425 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.425 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:45:45.425 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.881353+0000", 2024-02-09T17:45:45.426 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3015000000000006e-05 2024-02-09T17:45:45.426 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.426 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.426 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:45:45.426 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.881399+0000", 2024-02-09T17:45:45.426 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6140000000000002e-05 2024-02-09T17:45:45.426 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:45.426 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:45:45.426 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:45.426 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.426 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.426 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:144291 5.1 5:83ad9ffe:::benchmark_data_smithi179_115199_object15385:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e508)", 2024-02-09T17:45:45.426 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:42:51.809139+0000", 2024-02-09T17:45:45.426 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 173.52263477299999, 2024-02-09T17:45:45.426 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.078352973000000006, 2024-02-09T17:45:45.427 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:45:45.427 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:45:45.427 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:45:45.427 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:45:45.427 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:45:45.427 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 144291 2024-02-09T17:45:45.427 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.427 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:45:45.427 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.427 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:45:45.427 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.809139+0000", 2024-02-09T17:45:45.427 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:45.427 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.427 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.427 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:45:45.428 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.809139+0000", 2024-02-09T17:45:45.428 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:45.428 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.428 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.428 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:45:45.428 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.809139+0000", 2024-02-09T17:45:45.428 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4799999999999999e-07 2024-02-09T17:45:45.428 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.428 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.428 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:45:45.428 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.809141+0000", 2024-02-09T17:45:45.428 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.609e-06 2024-02-09T17:45:45.428 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.429 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.429 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:45:45.429 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.809147+0000", 2024-02-09T17:45:45.429 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.032e-06 2024-02-09T17:45:45.429 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.429 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.429 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:45:45.429 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.809151+0000", 2024-02-09T17:45:45.429 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.741e-06 2024-02-09T17:45:45.429 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.429 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.429 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:45.429 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.853195+0000", 2024-02-09T17:45:45.429 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.044043025 2024-02-09T17:45:45.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.430 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:45:45.430 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.853303+0000", 2024-02-09T17:45:45.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000108929 2024-02-09T17:45:45.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.430 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,7", 2024-02-09T17:45:45.430 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.853369+0000", 2024-02-09T17:45:45.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5172000000000006e-05 2024-02-09T17:45:45.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.430 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:45:45.430 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.866155+0000", 2024-02-09T17:45:45.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012786634 2024-02-09T17:45:45.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.431 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:45.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.870740+0000", 2024-02-09T17:45:45.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.004584734 2024-02-09T17:45:45.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.431 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:45.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.887390+0000", 2024-02-09T17:45:45.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.016649923000000001 2024-02-09T17:45:45.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.431 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:45:45.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.887447+0000", 2024-02-09T17:45:45.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7204000000000003e-05 2024-02-09T17:45:45.432 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.432 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.432 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:45:45.432 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.887492+0000", 2024-02-09T17:45:45.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4622000000000001e-05 2024-02-09T17:45:45.432 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:45.432 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:45:45.432 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:45.432 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.432 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.432 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:144300 5.1 5:8352adf2:::benchmark_data_smithi179_115199_object15394:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e508)", 2024-02-09T17:45:45.432 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:42:51.814738+0000", 2024-02-09T17:45:45.432 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 173.51703597400001, 2024-02-09T17:45:45.433 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.072982151999999995, 2024-02-09T17:45:45.433 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:45:45.433 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:45:45.433 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:45:45.433 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:45:45.433 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:45:45.433 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 144300 2024-02-09T17:45:45.433 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.433 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:45:45.433 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.433 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:45:45.433 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.814738+0000", 2024-02-09T17:45:45.433 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:45.433 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.433 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.434 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:45:45.434 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.814738+0000", 2024-02-09T17:45:45.434 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:45.434 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.434 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.434 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:45:45.434 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.814738+0000", 2024-02-09T17:45:45.434 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4099999999999997e-07 2024-02-09T17:45:45.434 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.434 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.434 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:45:45.434 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.814742+0000", 2024-02-09T17:45:45.434 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.326e-06 2024-02-09T17:45:45.434 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.435 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.435 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:45:45.435 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.814750+0000", 2024-02-09T17:45:45.435 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3129999999999999e-06 2024-02-09T17:45:45.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.435 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.435 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:45:45.435 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.814756+0000", 2024-02-09T17:45:45.435 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9460000000000003e-06 2024-02-09T17:45:45.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.435 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.435 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:45.435 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.857541+0000", 2024-02-09T17:45:45.435 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.042784498999999997 2024-02-09T17:45:45.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.436 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.436 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:45:45.436 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.857659+0000", 2024-02-09T17:45:45.436 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011793 2024-02-09T17:45:45.436 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.436 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.436 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,7", 2024-02-09T17:45:45.436 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.857771+0000", 2024-02-09T17:45:45.436 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000112318 2024-02-09T17:45:45.436 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.436 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.436 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:45.436 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.870858+0000", 2024-02-09T17:45:45.436 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013086726999999999 2024-02-09T17:45:45.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.437 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:45:45.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.871034+0000", 2024-02-09T17:45:45.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000176312 2024-02-09T17:45:45.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.437 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:45.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.887652+0000", 2024-02-09T17:45:45.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.016618601 2024-02-09T17:45:45.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.437 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:45:45.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.887687+0000", 2024-02-09T17:45:45.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4165999999999998e-05 2024-02-09T17:45:45.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:45:45.438 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:51.887720+0000", 2024-02-09T17:45:45.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3073e-05 2024-02-09T17:45:45.438 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:45.438 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:45:45.438 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:45.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.438 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25528.0:179604 5.f e530/480 5:fc3ce55f:::benchmark_data_smithi179_115199_object50698:head v 530'11196, mlcod=530'11194)", 2024-02-09T17:45:45.438 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:43:38.512263+0000", 2024-02-09T17:45:45.438 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 126.81951058200001, 2024-02-09T17:45:45.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.075365690999999999, 2024-02-09T17:45:45.439 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:45:45.439 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:45:45.439 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:45:45.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.439 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:45:45.439 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:38.512263+0000", 2024-02-09T17:45:45.439 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:45.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.439 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:45:45.439 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:38.512263+0000", 2024-02-09T17:45:45.439 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:45.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.439 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:45:45.440 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:38.512264+0000", 2024-02-09T17:45:45.440 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1179999999999999e-06 2024-02-09T17:45:45.440 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.440 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.440 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:45:45.440 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:38.512272+0000", 2024-02-09T17:45:45.440 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9820000000000008e-06 2024-02-09T17:45:45.440 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.440 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.440 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:45:45.440 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:38.512283+0000", 2024-02-09T17:45:45.440 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0514e-05 2024-02-09T17:45:45.440 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.440 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.441 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:45:45.441 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:38.512289+0000", 2024-02-09T17:45:45.441 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6730000000000003e-06 2024-02-09T17:45:45.441 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.441 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.441 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:45.441 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:38.512383+0000", 2024-02-09T17:45:45.441 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4238000000000001e-05 2024-02-09T17:45:45.441 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.441 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.441 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:45:45.441 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:38.512403+0000", 2024-02-09T17:45:45.441 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9338999999999999e-05 2024-02-09T17:45:45.441 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.442 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.442 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:45:45.442 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:38.587592+0000", 2024-02-09T17:45:45.442 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.075189078000000006 2024-02-09T17:45:45.442 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.442 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.442 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:45:45.442 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:38.587629+0000", 2024-02-09T17:45:45.442 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6748999999999997e-05 2024-02-09T17:45:45.442 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:45.442 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:45:45.442 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:45.442 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.442 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.442 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:200968 5.1 5:8668013e:::benchmark_data_smithi179_115199_object72062:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e543)", 2024-02-09T17:45:45.443 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:44:07.427511+0000", 2024-02-09T17:45:45.443 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 97.904262283999998, 2024-02-09T17:45:45.443 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.066971571999999993, 2024-02-09T17:45:45.443 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:45:45.443 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:45:45.443 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:45:45.443 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:45:45.443 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:45:45.443 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 200968 2024-02-09T17:45:45.443 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.443 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:45:45.443 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.443 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:45:45.443 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:44:07.427511+0000", 2024-02-09T17:45:45.444 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:45.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.444 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.444 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:45:45.444 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:44:07.427511+0000", 2024-02-09T17:45:45.444 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:45.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.444 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.444 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:45:45.444 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:44:07.427512+0000", 2024-02-09T17:45:45.444 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7800000000000002e-07 2024-02-09T17:45:45.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.444 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.444 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:45:45.445 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:44:07.427515+0000", 2024-02-09T17:45:45.445 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.219e-06 2024-02-09T17:45:45.445 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.445 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.445 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:45:45.445 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:44:07.427522+0000", 2024-02-09T17:45:45.445 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5629999999999997e-06 2024-02-09T17:45:45.445 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.445 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.445 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:45:45.445 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:44:07.427527+0000", 2024-02-09T17:45:45.445 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9440000000000002e-06 2024-02-09T17:45:45.445 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.445 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.446 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:45.446 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:44:07.432679+0000", 2024-02-09T17:45:45.446 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.005151976 2024-02-09T17:45:45.446 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.446 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.446 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:45:45.446 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:44:07.432802+0000", 2024-02-09T17:45:45.446 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012222200000000001 2024-02-09T17:45:45.446 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.446 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.446 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,7", 2024-02-09T17:45:45.446 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:44:07.432918+0000", 2024-02-09T17:45:45.446 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000116493 2024-02-09T17:45:45.446 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.446 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.447 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:45:45.447 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:44:07.444382+0000", 2024-02-09T17:45:45.447 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011464034 2024-02-09T17:45:45.447 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.447 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.447 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:45.447 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:44:07.450343+0000", 2024-02-09T17:45:45.447 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0059611430000000003 2024-02-09T17:45:45.447 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.447 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.447 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:45.447 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:44:07.494384+0000", 2024-02-09T17:45:45.447 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.044040837999999999 2024-02-09T17:45:45.447 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.447 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.448 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:45:45.448 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:44:07.494442+0000", 2024-02-09T17:45:45.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7920000000000001e-05 2024-02-09T17:45:45.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.448 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:45:45.448 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:44:07.494483+0000", 2024-02-09T17:45:45.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0741999999999997e-05 2024-02-09T17:45:45.448 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:45.448 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:45:45.448 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:45.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.449 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:203909 5.1 5:88268bd8:::benchmark_data_smithi179_115199_object75003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e545)", 2024-02-09T17:45:45.449 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:44:11.267267+0000", 2024-02-09T17:45:45.449 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 94.064506914999996, 2024-02-09T17:45:45.449 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.065714183999999995, 2024-02-09T17:45:45.449 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:45:45.449 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:45:45.449 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:45:45.449 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:45:45.449 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:45:45.449 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 203909 2024-02-09T17:45:45.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.449 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:45:45.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.449 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:45:45.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:44:11.267267+0000", 2024-02-09T17:45:45.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:45.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:45:45.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:44:11.267267+0000", 2024-02-09T17:45:45.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:45.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:45:45.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:44:11.267269+0000", 2024-02-09T17:45:45.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9120000000000001e-06 2024-02-09T17:45:45.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.451 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:45:45.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:44:11.267276+0000", 2024-02-09T17:45:45.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3370000000000002e-06 2024-02-09T17:45:45.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.451 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:45:45.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:44:11.267293+0000", 2024-02-09T17:45:45.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7418000000000002e-05 2024-02-09T17:45:45.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.451 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:45:45.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:44:11.267301+0000", 2024-02-09T17:45:45.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.994e-06 2024-02-09T17:45:45.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.452 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.452 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:45.452 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:44:11.274345+0000", 2024-02-09T17:45:45.452 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0070434169999999997 2024-02-09T17:45:45.452 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.452 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.452 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:45:45.452 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:44:11.274460+0000", 2024-02-09T17:45:45.452 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011576100000000001 2024-02-09T17:45:45.452 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.452 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.452 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,7", 2024-02-09T17:45:45.452 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:44:11.274577+0000", 2024-02-09T17:45:45.452 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000116057 2024-02-09T17:45:45.453 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.453 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.453 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:45:45.453 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:44:11.283337+0000", 2024-02-09T17:45:45.453 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0087606590000000005 2024-02-09T17:45:45.453 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.453 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.453 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:45.453 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:44:11.284642+0000", 2024-02-09T17:45:45.453 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013047779999999999 2024-02-09T17:45:45.453 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.453 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.453 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:45.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:44:11.332886+0000", 2024-02-09T17:45:45.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.048244187000000001 2024-02-09T17:45:45.454 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.454 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:45:45.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:44:11.332943+0000", 2024-02-09T17:45:45.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6486000000000001e-05 2024-02-09T17:45:45.454 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.454 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:45:45.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:44:11.332981+0000", 2024-02-09T17:45:45.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8177999999999999e-05 2024-02-09T17:45:45.454 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:45.454 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:45:45.454 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:45.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.455 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25528.0:227629 5.5 e553/465 5:a7fb1958:::benchmark_data_smithi179_115199_object98723:head v 553'14108, mlcod=553'14106)", 2024-02-09T17:45:45.455 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:44:39.693853+0000", 2024-02-09T17:45:45.455 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 65.637920046000005, 2024-02-09T17:45:45.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.061956444999999999, 2024-02-09T17:45:45.455 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:45:45.455 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:45:45.455 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:45:45.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:45:45.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:44:39.693853+0000", 2024-02-09T17:45:45.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:45.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.456 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.456 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:45:45.456 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:44:39.693853+0000", 2024-02-09T17:45:45.456 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:45.456 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.456 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.456 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:45:45.456 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:44:39.693855+0000", 2024-02-09T17:45:45.456 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.186e-06 2024-02-09T17:45:45.456 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.456 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.456 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:45:45.456 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:44:39.743099+0000", 2024-02-09T17:45:45.457 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.049243923000000002 2024-02-09T17:45:45.457 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.457 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.457 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:45:45.457 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:44:39.743106+0000", 2024-02-09T17:45:45.457 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9419999999999998e-06 2024-02-09T17:45:45.457 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.457 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.457 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:45:45.457 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:44:39.743110+0000", 2024-02-09T17:45:45.457 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8860000000000003e-06 2024-02-09T17:45:45.457 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.457 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.457 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:45.457 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:44:39.743197+0000", 2024-02-09T17:45:45.458 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6172000000000001e-05 2024-02-09T17:45:45.458 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.458 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.458 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:45:45.458 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:44:39.743216+0000", 2024-02-09T17:45:45.458 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9680000000000001e-05 2024-02-09T17:45:45.458 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.458 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.458 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:45:45.458 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:44:39.755767+0000", 2024-02-09T17:45:45.458 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012550863000000001 2024-02-09T17:45:45.458 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:45.458 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:45.458 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:45:45.458 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:44:39.755810+0000", 2024-02-09T17:45:45.459 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2793000000000002e-05 2024-02-09T17:45:45.459 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:45.459 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:45:45.459 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:45.459 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:45.459 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:45:45.459 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:45:45.459 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:45:45.486 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:45.505 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:45:45.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:45.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:45:45.609 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:45:45.640 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:45.660 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:45:45.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:45.663 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:45:45.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:45 smithi162 ceph-mon[44304]: osdmap e578: 12 total, 12 up, 12 in 2024-02-09T17:45:45.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:45 smithi162 ceph-mon[44304]: 1.14 scrub starts 2024-02-09T17:45:45.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:45 smithi162 ceph-mon[44304]: 1.14 scrub ok 2024-02-09T17:45:45.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:45 smithi162 ceph-mon[44304]: pgmap v1448: 47 pgs: 1 creating+peering, 11 unknown, 35 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:45:45.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:45 smithi162 ceph-mon[44304]: 1.1b deep-scrub starts 2024-02-09T17:45:45.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:45 smithi162 ceph-mon[44304]: 1.1b deep-scrub ok 2024-02-09T17:45:45.764 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:45:45.794 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:45.812 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:45:45.813 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:45:45.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:45.817 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:45:45.918 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:45:45.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:45 smithi179 ceph-mon[43975]: osdmap e578: 12 total, 12 up, 12 in 2024-02-09T17:45:45.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:45 smithi179 ceph-mon[43975]: 1.14 scrub starts 2024-02-09T17:45:45.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:45 smithi179 ceph-mon[43975]: 1.14 scrub ok 2024-02-09T17:45:45.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:45 smithi179 ceph-mon[43975]: pgmap v1448: 47 pgs: 1 creating+peering, 11 unknown, 35 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:45:45.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:45 smithi179 ceph-mon[43975]: 1.1b deep-scrub starts 2024-02-09T17:45:45.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:45 smithi179 ceph-mon[43975]: 1.1b deep-scrub ok 2024-02-09T17:45:45.942 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:45 smithi099 ceph-mon[39420]: osdmap e578: 12 total, 12 up, 12 in 2024-02-09T17:45:45.943 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:45 smithi099 ceph-mon[39420]: 1.14 scrub starts 2024-02-09T17:45:45.943 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:45 smithi099 ceph-mon[39420]: 1.14 scrub ok 2024-02-09T17:45:45.943 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:45 smithi099 ceph-mon[39420]: pgmap v1448: 47 pgs: 1 creating+peering, 11 unknown, 35 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:45:45.943 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:45 smithi099 ceph-mon[39420]: 1.1b deep-scrub starts 2024-02-09T17:45:45.943 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:45 smithi099 ceph-mon[39420]: 1.1b deep-scrub ok 2024-02-09T17:45:45.944 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:46.070 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-09T17:45:46.335 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:45:46.594 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:45:46.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:46.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:45:46.698 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:45:46.728 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:46.801 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:45:46.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:46.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:45:46.828 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:46 smithi162 ceph-mon[44304]: 1.f scrub starts 2024-02-09T17:45:46.905 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:45:46.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:46 smithi179 ceph-mon[43975]: 1.f scrub starts 2024-02-09T17:45:46.931 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:46 smithi099 ceph-mon[39420]: 1.f scrub starts 2024-02-09T17:45:46.933 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:47.032 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:45:47.081 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:45:47.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:47.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:45:47.185 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:45:47.213 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:47.235 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:45:47.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:47.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:45:47.347 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:45:47.380 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:47.401 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:45:47.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:47.404 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:45:47.505 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:45:47.537 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:47.559 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:45:47.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:47.562 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:45:47.663 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:45:47.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:47 smithi179 ceph-mon[43975]: 1.f scrub ok 2024-02-09T17:45:47.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:47 smithi179 ceph-mon[43975]: 1.0 scrub starts 2024-02-09T17:45:47.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:47 smithi179 ceph-mon[43975]: 1.0 scrub ok 2024-02-09T17:45:47.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:47 smithi179 ceph-mon[43975]: pgmap v1449: 47 pgs: 1 creating+peering, 7 unknown, 39 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:45:47.691 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:47.711 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:45:47.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:47.714 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:45:47.814 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:45:47.842 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:47.847 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:47 smithi162 ceph-mon[44304]: 1.f scrub ok 2024-02-09T17:45:47.847 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:47 smithi162 ceph-mon[44304]: 1.0 scrub starts 2024-02-09T17:45:47.847 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:47 smithi162 ceph-mon[44304]: 1.0 scrub ok 2024-02-09T17:45:47.847 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:47 smithi162 ceph-mon[44304]: pgmap v1449: 47 pgs: 1 creating+peering, 7 unknown, 39 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:45:47.862 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:45:47.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:47.865 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:45:47.959 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] 2024-02-09T17:45:47.960 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:45:47.960 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-02-09T17:45:47.960 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-09T17:45:47.960 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-09T17:45:47.960 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-02-09T17:45:47.966 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:45:47.992 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:47 smithi099 ceph-mon[39420]: 1.f scrub ok 2024-02-09T17:45:47.992 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:47 smithi099 ceph-mon[39420]: 1.0 scrub starts 2024-02-09T17:45:47.992 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:47 smithi099 ceph-mon[39420]: 1.0 scrub ok 2024-02-09T17:45:47.992 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:47 smithi099 ceph-mon[39420]: pgmap v1449: 47 pgs: 1 creating+peering, 7 unknown, 39 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:45:47.995 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:48.118 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:45:48.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:48.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:45:48.133 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:45:48.140 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:45:48.140 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-09T17:45:48.140 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-02-09T17:45:48.140 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:45:48.223 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:45:48.250 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:48.374 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:45:48.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:48.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:45:48.480 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:45:48.508 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:48.581 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:45:48.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:48.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:45:48.627 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is set 2024-02-09T17:45:48.685 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:45:48.711 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:48.781 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:48 smithi162 ceph-mon[44304]: 1.9 deep-scrub starts 2024-02-09T17:45:48.781 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:48 smithi162 ceph-mon[44304]: 1.9 deep-scrub ok 2024-02-09T17:45:48.782 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:48 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1641003904' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:45:48.806 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-09T17:45:48.828 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:45:48.863 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:48 smithi099 ceph-mon[39420]: 1.9 deep-scrub starts 2024-02-09T17:45:48.863 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:48 smithi099 ceph-mon[39420]: 1.9 deep-scrub ok 2024-02-09T17:45:48.863 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:48 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1641003904' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:45:48.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:48 smithi179 ceph-mon[43975]: 1.9 deep-scrub starts 2024-02-09T17:45:48.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:48 smithi179 ceph-mon[43975]: 1.9 deep-scrub ok 2024-02-09T17:45:48.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:48 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1641003904' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:45:49.637 INFO:teuthology.orchestra.run.smithi099.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-02-09T17:45:49.652 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:49 smithi099 ceph-mon[39420]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:45:49.652 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:49 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1641003904' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:45:49.652 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:49 smithi099 ceph-mon[39420]: osdmap e579: 12 total, 12 up, 12 in 2024-02-09T17:45:49.652 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:49 smithi099 ceph-mon[39420]: pgmap v1451: 47 pgs: 47 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:45:49.653 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:49 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/109089503' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-09T17:45:49.661 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:45:49.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:49.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:45:49.673 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:45:49.765 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:45:49.793 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:49.882 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:49 smithi162 ceph-mon[44304]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:45:49.882 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:49 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1641003904' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:45:49.882 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:49 smithi162 ceph-mon[44304]: osdmap e579: 12 total, 12 up, 12 in 2024-02-09T17:45:49.882 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:49 smithi162 ceph-mon[44304]: pgmap v1451: 47 pgs: 47 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:45:49.882 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:49 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/109089503' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-09T17:45:49.899 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:45:49.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:49.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:45:49.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:49 smithi179 ceph-mon[43975]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:45:49.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:49 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1641003904' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:45:49.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:49 smithi179 ceph-mon[43975]: osdmap e579: 12 total, 12 up, 12 in 2024-02-09T17:45:49.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:49 smithi179 ceph-mon[43975]: pgmap v1451: 47 pgs: 47 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:45:49.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:49 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/109089503' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-09T17:45:50.003 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:45:50.031 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:50.129 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:45:50.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:50.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:45:50.233 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:45:50.262 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:50.810 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:45:50.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:50.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:45:50.914 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:45:50.915 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:50 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/109089503' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-09T17:45:50.915 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:50 smithi099 ceph-mon[39420]: osdmap e580: 12 total, 12 up, 12 in 2024-02-09T17:45:50.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:50 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/109089503' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-09T17:45:50.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:50 smithi179 ceph-mon[43975]: osdmap e580: 12 total, 12 up, 12 in 2024-02-09T17:45:50.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:50 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/109089503' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-09T17:45:50.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:50 smithi162 ceph-mon[44304]: osdmap e580: 12 total, 12 up, 12 in 2024-02-09T17:45:50.945 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:50.966 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:45:50.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:50.969 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:45:51.070 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:45:51.081 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:45:51.081 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:45:51.081 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:45:51.081 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:45:51.082 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:45:51.082 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:45:51.082 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:45:51.082 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:45:51.082 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:45:51.082 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:45:51.082 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:45:51.082 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:45:51.082 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:45:51.082 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:45:51.082 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:45:51.082 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:45:51.082 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:45:51.083 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:45:51.083 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:45:51.083 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:45:51.083 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:45:51.083 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:45:51.083 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:45:51.083 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:45:51.098 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:51.118 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:45:51.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:51.121 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:45:51.224 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:45:51.252 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:51.270 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:45:51.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:51.273 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:45:51.374 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:45:51.403 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:51.422 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:45:51.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:51.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:45:51.534 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:45:51.560 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:51.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:51 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3792315549' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:45:51.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:51 smithi099 ceph-mon[39420]: pgmap v1453: 47 pgs: 47 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:45:51.808 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:45:51.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:51.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:45:51.879 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:45:51.879 INFO:teuthology.orchestra.run.smithi162.stdout: "success": "" 2024-02-09T17:45:51.879 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:45:51.894 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:51 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3792315549' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:45:51.894 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:51 smithi162 ceph-mon[44304]: pgmap v1453: 47 pgs: 47 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:45:51.913 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:45:51.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:51 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3792315549' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:45:51.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:51 smithi179 ceph-mon[43975]: pgmap v1453: 47 pgs: 47 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:45:51.939 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:51.984 DEBUG:teuthology.orchestra.run.smithi179:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-02-09T17:45:51.987 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-02-09T17:45:51.987 INFO:teuthology.orchestra.run:waiting for 3000 2024-02-09T17:45:52.037 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:45:52.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:52.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:45:52.095 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:hints = 1 2024-02-09T17:45:52.095 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-02-09T17:45:52.095 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Object prefix: benchmark_data_smithi179_126206 2024-02-09T17:45:52.096 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-09T17:45:52.096 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 0 0 0 0 0 0 - 0 2024-02-09T17:45:52.152 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:45:52.195 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:52.300 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:45:52.300 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:45:52.370 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:45:52.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:52.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:45:52.478 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:45:52.519 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:52.548 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:45:52.552 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:52.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:45:52.653 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:45:52.703 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:52.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:52 smithi179 ceph-mon[43975]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-09T17:45:52.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:52 smithi162 ceph-mon[44304]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-09T17:45:52.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:52 smithi099 ceph-mon[39420]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-09T17:45:53.096 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 1 16 4633 4617 288.547 288.562 0.00285979 0.00345447 2024-02-09T17:45:53.833 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:45:53.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:53 smithi179 ceph-mon[43975]: pgmap v1454: 47 pgs: 47 active+clean; 21 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s wr, 41 op/s 2024-02-09T17:45:53.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:53 smithi162 ceph-mon[44304]: pgmap v1454: 47 pgs: 47 active+clean; 21 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s wr, 41 op/s 2024-02-09T17:45:53.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:53 smithi099 ceph-mon[39420]: pgmap v1454: 47 pgs: 47 active+clean; 21 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s wr, 41 op/s 2024-02-09T17:45:54.096 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 2 16 7468 7452 232.859 177.188 0.00562778 0.00428788 2024-02-09T17:45:54.228 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:45:54.406 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-09T17:45:54.584 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-09T17:45:54.950 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:45:54.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:54.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:45:55.055 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:45:55.067 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:45:55.085 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:55.096 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 3 16 10017 10001 208.335 159.312 0.00174196 0.00472749 2024-02-09T17:45:55.633 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:45:55.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:55.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:45:55.746 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:45:55.788 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:55.818 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:45:55.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:55.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:45:55.922 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:45:55.942 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:55 smithi099 ceph-mon[39420]: pgmap v1455: 47 pgs: 47 active+clean; 173 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s wr, 343 op/s 2024-02-09T17:45:55.966 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:56.096 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 4 16 12867 12851 200.778 178.125 0.0534149 0.0049669 2024-02-09T17:45:56.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:55 smithi162 ceph-mon[44304]: pgmap v1455: 47 pgs: 47 active+clean; 173 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s wr, 343 op/s 2024-02-09T17:45:56.305 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:45:56.389 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:45:56.421 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:45:56.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:55 smithi179 ceph-mon[43975]: pgmap v1455: 47 pgs: 47 active+clean; 173 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s wr, 343 op/s 2024-02-09T17:45:56.430 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:45:56.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:56.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:45:56.457 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:45:56.491 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:45:56.526 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:45:56.536 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:45:56.561 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:45:56.595 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:45:56.613 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:45:56.648 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:45:56.677 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:45:56.712 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:45:56.753 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:45:56.787 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:45:56.788 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:45:56.789 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:45:56.789 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:45:56.789 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:45:56.789 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:45:56.789 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:45:56.789 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:45:56.789 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:45:56.789 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:45:56.789 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:45:56.789 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:45:56.789 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:45:57.096 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 5 15 16056 16041 200.494 199.375 0.006725 0.00498143 2024-02-09T17:45:57.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:56 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/772841482' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:45:57.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:56 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/772841482' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:45:57.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:56 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/772841482' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:45:58.096 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 6 16 18590 18574 193.46 158.312 0.00170459 0.00503677 2024-02-09T17:45:58.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:57 smithi099 ceph-mon[39420]: pgmap v1456: 47 pgs: 47 active+clean; 371 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 717 op/s 2024-02-09T17:45:58.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:57 smithi179 ceph-mon[43975]: pgmap v1456: 47 pgs: 47 active+clean; 371 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 717 op/s 2024-02-09T17:45:58.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:57 smithi162 ceph-mon[44304]: pgmap v1456: 47 pgs: 47 active+clean; 371 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 717 op/s 2024-02-09T17:45:59.096 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 7 16 20236 20220 180.518 102.875 0.00611459 0.00553412 2024-02-09T17:45:59.467 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:45:59.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:45:59.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:45:59.499 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:45:59.500 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-02-09T17:45:59.500 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-02-09T17:45:59.500 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-02-09T17:45:59.500 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.500 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:33098 5.f 5:fb2113f5:::benchmark_data_smithi179_115199_object33097:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e479)", 2024-02-09T17:45:59.500 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:41:19.860866+0000", 2024-02-09T17:45:59.500 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 279.63402028199999, 2024-02-09T17:45:59.500 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.017314074, 2024-02-09T17:45:59.500 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:45:59.500 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-09T17:45:59.500 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:45:59.500 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:45:59.501 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:45:59.501 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 33098 2024-02-09T17:45:59.501 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.501 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:45:59.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.501 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:45:59.501 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.860866+0000", 2024-02-09T17:45:59.501 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:59.501 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.501 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:45:59.501 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.860866+0000", 2024-02-09T17:45:59.501 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:59.502 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.502 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.502 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:45:59.502 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.860867+0000", 2024-02-09T17:45:59.502 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1969999999999999e-06 2024-02-09T17:45:59.502 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.502 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.502 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:45:59.502 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.862146+0000", 2024-02-09T17:45:59.502 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0012785520000000001 2024-02-09T17:45:59.502 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.502 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.502 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:45:59.502 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.862157+0000", 2024-02-09T17:45:59.503 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1525999999999999e-05 2024-02-09T17:45:59.503 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.503 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.503 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:45:59.503 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.862166+0000", 2024-02-09T17:45:59.503 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.918e-06 2024-02-09T17:45:59.503 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.503 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.503 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:59.503 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.862674+0000", 2024-02-09T17:45:59.503 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00050811300000000003 2024-02-09T17:45:59.503 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.503 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.504 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:45:59.504 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.862804+0000", 2024-02-09T17:45:59.504 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012947699999999999 2024-02-09T17:45:59.504 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.504 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.504 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,11", 2024-02-09T17:45:59.504 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.862947+0000", 2024-02-09T17:45:59.504 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000143378 2024-02-09T17:45:59.504 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.504 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.504 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:45:59.504 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.864397+0000", 2024-02-09T17:45:59.504 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014498410000000001 2024-02-09T17:45:59.505 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.505 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.505 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:59.505 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.867884+0000", 2024-02-09T17:45:59.505 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034867850000000001 2024-02-09T17:45:59.505 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.505 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.505 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:45:59.505 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.867896+0000", 2024-02-09T17:45:59.505 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2411e-05 2024-02-09T17:45:59.505 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.505 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.505 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:59.505 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.878116+0000", 2024-02-09T17:45:59.506 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.010220026 2024-02-09T17:45:59.506 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.506 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.506 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:45:59.506 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.878180+0000", 2024-02-09T17:45:59.506 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3849999999999993e-05 2024-02-09T17:45:59.506 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:59.506 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:45:59.506 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:59.506 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.506 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.506 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:33125 5.f 5:f3e8f580:::benchmark_data_smithi179_115199_object33124:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e480)", 2024-02-09T17:45:59.506 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:41:19.882980+0000", 2024-02-09T17:45:59.506 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 279.61190605100001, 2024-02-09T17:45:59.507 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99991403599999995, 2024-02-09T17:45:59.507 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:45:59.507 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:45:59.507 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:45:59.507 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:45:59.507 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:45:59.507 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 33125 2024-02-09T17:45:59.507 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.507 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:45:59.507 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.507 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:45:59.507 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.882980+0000", 2024-02-09T17:45:59.507 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:59.508 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.508 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.508 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:45:59.508 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.882980+0000", 2024-02-09T17:45:59.508 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:59.508 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.508 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.508 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:45:59.508 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.882982+0000", 2024-02-09T17:45:59.508 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.731e-06 2024-02-09T17:45:59.508 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.508 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.508 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:45:59.508 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.883025+0000", 2024-02-09T17:45:59.509 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2828999999999999e-05 2024-02-09T17:45:59.509 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.509 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.509 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:45:59.509 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.883037+0000", 2024-02-09T17:45:59.509 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2291e-05 2024-02-09T17:45:59.509 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.509 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.509 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:45:59.509 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.883045+0000", 2024-02-09T17:45:59.509 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8499999999999994e-06 2024-02-09T17:45:59.509 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.509 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.509 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:59.509 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.883111+0000", 2024-02-09T17:45:59.510 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.656e-05 2024-02-09T17:45:59.510 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.510 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.510 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:45:59.510 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.883122+0000", 2024-02-09T17:45:59.510 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0838e-05 2024-02-09T17:45:59.510 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.510 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.510 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:59.510 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.878634+0000", 2024-02-09T17:45:59.510 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99551221000000001 2024-02-09T17:45:59.510 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.510 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.510 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:45:59.511 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.878805+0000", 2024-02-09T17:45:59.511 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017077999999999999 2024-02-09T17:45:59.511 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.511 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.511 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,11", 2024-02-09T17:45:59.511 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.878989+0000", 2024-02-09T17:45:59.511 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000183893 2024-02-09T17:45:59.511 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.511 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.511 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:59.511 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.882644+0000", 2024-02-09T17:45:59.511 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0036547139999999999 2024-02-09T17:45:59.511 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.512 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:59.512 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.882718+0000", 2024-02-09T17:45:59.512 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4070000000000001e-05 2024-02-09T17:45:59.512 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.512 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:45:59.512 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.882813+0000", 2024-02-09T17:45:59.512 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5030000000000003e-05 2024-02-09T17:45:59.512 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.512 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:45:59.513 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.882853+0000", 2024-02-09T17:45:59.513 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9771000000000003e-05 2024-02-09T17:45:59.513 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.513 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.513 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:45:59.513 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.882894+0000", 2024-02-09T17:45:59.513 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1468999999999999e-05 2024-02-09T17:45:59.513 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:59.513 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:45:59.513 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:59.513 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.513 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.513 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:33145 5.f 5:f8644e66:::benchmark_data_smithi179_115199_object33144:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e480)", 2024-02-09T17:45:59.514 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:41:19.884388+0000", 2024-02-09T17:45:59.514 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 279.61049772299998, 2024-02-09T17:45:59.514 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99867693599999996, 2024-02-09T17:45:59.514 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:45:59.514 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:45:59.514 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:45:59.514 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:45:59.514 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:45:59.514 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 33145 2024-02-09T17:45:59.514 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.514 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:45:59.514 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.514 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:45:59.514 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.884388+0000", 2024-02-09T17:45:59.515 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:59.515 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.515 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.515 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:45:59.515 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.884388+0000", 2024-02-09T17:45:59.515 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:59.515 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.515 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.515 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:45:59.515 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.884389+0000", 2024-02-09T17:45:59.515 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5400000000000001e-07 2024-02-09T17:45:59.515 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.515 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.515 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:45:59.516 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.884423+0000", 2024-02-09T17:45:59.516 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3884999999999998e-05 2024-02-09T17:45:59.516 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.516 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.516 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:45:59.516 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.884431+0000", 2024-02-09T17:45:59.516 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1880000000000002e-06 2024-02-09T17:45:59.516 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.516 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.516 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:45:59.516 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.884438+0000", 2024-02-09T17:45:59.516 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.533e-06 2024-02-09T17:45:59.516 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.516 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.517 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:59.517 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.884871+0000", 2024-02-09T17:45:59.517 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000433526 2024-02-09T17:45:59.517 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.517 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.517 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:45:59.517 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.884880+0000", 2024-02-09T17:45:59.517 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9709999999999993e-06 2024-02-09T17:45:59.517 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.517 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.517 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:59.517 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.880283+0000", 2024-02-09T17:45:59.517 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99540302000000003 2024-02-09T17:45:59.517 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.517 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.518 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:45:59.518 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.880419+0000", 2024-02-09T17:45:59.518 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013591300000000001 2024-02-09T17:45:59.518 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.518 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.518 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,11", 2024-02-09T17:45:59.518 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.880531+0000", 2024-02-09T17:45:59.518 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011196000000000001 2024-02-09T17:45:59.518 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.518 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.518 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:59.518 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.882950+0000", 2024-02-09T17:45:59.518 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002418548 2024-02-09T17:45:59.518 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.518 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.519 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:59.519 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.882997+0000", 2024-02-09T17:45:59.519 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7614000000000002e-05 2024-02-09T17:45:59.519 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.519 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.519 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:45:59.519 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.883006+0000", 2024-02-09T17:45:59.519 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3289999999999994e-06 2024-02-09T17:45:59.519 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.519 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.519 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:45:59.519 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.883036+0000", 2024-02-09T17:45:59.519 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.025e-05 2024-02-09T17:45:59.519 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.520 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.520 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:45:59.520 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.883065+0000", 2024-02-09T17:45:59.520 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9544999999999999e-05 2024-02-09T17:45:59.520 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:59.520 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:45:59.520 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:59.520 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.520 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.520 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:33147 5.f 5:fc9a5092:::benchmark_data_smithi179_115199_object33146:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e480)", 2024-02-09T17:45:59.520 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:41:19.884563+0000", 2024-02-09T17:45:59.520 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 279.610323526, 2024-02-09T17:45:59.520 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0012039639999999, 2024-02-09T17:45:59.520 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:45:59.521 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:45:59.521 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:45:59.521 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:45:59.521 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:45:59.521 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 33147 2024-02-09T17:45:59.521 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.521 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:45:59.521 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.521 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:45:59.521 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.884563+0000", 2024-02-09T17:45:59.521 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:59.521 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.521 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.521 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:45:59.521 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.884563+0000", 2024-02-09T17:45:59.522 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:59.522 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.522 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.522 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:45:59.522 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.884563+0000", 2024-02-09T17:45:59.522 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0800000000000004e-07 2024-02-09T17:45:59.522 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.522 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.522 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:45:59.522 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.884627+0000", 2024-02-09T17:45:59.522 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3874999999999994e-05 2024-02-09T17:45:59.522 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.522 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.523 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:45:59.523 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.884636+0000", 2024-02-09T17:45:59.523 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1009999999999998e-06 2024-02-09T17:45:59.523 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.523 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.523 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:45:59.523 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.884642+0000", 2024-02-09T17:45:59.523 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4430000000000002e-06 2024-02-09T17:45:59.523 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.523 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.523 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:59.523 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.884902+0000", 2024-02-09T17:45:59.523 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00026010699999999998 2024-02-09T17:45:59.523 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.523 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.524 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:45:59.524 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.884909+0000", 2024-02-09T17:45:59.524 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7100000000000001e-06 2024-02-09T17:45:59.524 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.524 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.524 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:59.524 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.881550+0000", 2024-02-09T17:45:59.524 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.996641369 2024-02-09T17:45:59.524 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.524 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.525 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:45:59.525 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.881685+0000", 2024-02-09T17:45:59.525 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013473599999999999 2024-02-09T17:45:59.525 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.525 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.525 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,11", 2024-02-09T17:45:59.525 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.881845+0000", 2024-02-09T17:45:59.525 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000160531 2024-02-09T17:45:59.525 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.525 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.525 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:45:59.525 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.883072+0000", 2024-02-09T17:45:59.525 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0012270569999999999 2024-02-09T17:45:59.525 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.526 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.526 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:59.526 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.883122+0000", 2024-02-09T17:45:59.526 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9295000000000002e-05 2024-02-09T17:45:59.526 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.526 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.526 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:59.526 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.885662+0000", 2024-02-09T17:45:59.526 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0025401130000000001 2024-02-09T17:45:59.526 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.526 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.526 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:45:59.526 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.885725+0000", 2024-02-09T17:45:59.526 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3044000000000002e-05 2024-02-09T17:45:59.527 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.527 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.527 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:45:59.527 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.885767+0000", 2024-02-09T17:45:59.527 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1774999999999999e-05 2024-02-09T17:45:59.527 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:59.527 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:45:59.527 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:59.527 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.527 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.527 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:33151 5.f 5:fdb17747:::benchmark_data_smithi179_115199_object33150:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e480)", 2024-02-09T17:45:59.527 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:41:19.886978+0000", 2024-02-09T17:45:59.527 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 279.60790793199999, 2024-02-09T17:45:59.527 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99900822099999997, 2024-02-09T17:45:59.528 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:45:59.528 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:45:59.528 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:45:59.528 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:45:59.528 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:45:59.528 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 33151 2024-02-09T17:45:59.528 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.528 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:45:59.528 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.528 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:45:59.528 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.886978+0000", 2024-02-09T17:45:59.528 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:59.528 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:45:59.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.886978+0000", 2024-02-09T17:45:59.529 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:59.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:45:59.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.886979+0000", 2024-02-09T17:45:59.529 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1519999999999999e-06 2024-02-09T17:45:59.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:45:59.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.887089+0000", 2024-02-09T17:45:59.530 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000109262 2024-02-09T17:45:59.530 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.530 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.530 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:45:59.530 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.887104+0000", 2024-02-09T17:45:59.530 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5075999999999999e-05 2024-02-09T17:45:59.530 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.530 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.530 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:45:59.530 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.887111+0000", 2024-02-09T17:45:59.530 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2660000000000004e-06 2024-02-09T17:45:59.530 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.530 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.531 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:59.531 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.887162+0000", 2024-02-09T17:45:59.531 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1078999999999997e-05 2024-02-09T17:45:59.531 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.531 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.531 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:45:59.531 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.887169+0000", 2024-02-09T17:45:59.531 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5309999999999998e-06 2024-02-09T17:45:59.531 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.531 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.531 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:59.531 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.883162+0000", 2024-02-09T17:45:59.531 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99599290200000001 2024-02-09T17:45:59.531 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.532 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.532 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:45:59.532 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.883263+0000", 2024-02-09T17:45:59.532 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010171900000000001 2024-02-09T17:45:59.532 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.532 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.532 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,11", 2024-02-09T17:45:59.532 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.883378+0000", 2024-02-09T17:45:59.532 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011450099999999999 2024-02-09T17:45:59.532 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.532 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.532 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:59.532 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.885834+0000", 2024-02-09T17:45:59.532 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0024559619999999999 2024-02-09T17:45:59.533 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.533 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.533 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:45:59.533 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.885851+0000", 2024-02-09T17:45:59.533 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7444000000000001e-05 2024-02-09T17:45:59.533 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.533 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.533 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:59.533 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.885894+0000", 2024-02-09T17:45:59.533 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2522999999999999e-05 2024-02-09T17:45:59.533 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.533 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.533 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:45:59.533 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.885949+0000", 2024-02-09T17:45:59.534 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5337999999999998e-05 2024-02-09T17:45:59.534 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.534 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.534 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:45:59.534 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.885986+0000", 2024-02-09T17:45:59.534 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7465999999999997e-05 2024-02-09T17:45:59.534 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:59.534 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:45:59.534 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:59.534 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.534 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.534 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:33161 5.f 5:f9e596c5:::benchmark_data_smithi179_115199_object33160:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e480)", 2024-02-09T17:45:59.534 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:41:19.892414+0000", 2024-02-09T17:45:59.535 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 279.60247206299999, 2024-02-09T17:45:59.535 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99654300500000004, 2024-02-09T17:45:59.535 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:45:59.535 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:45:59.535 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:45:59.535 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:45:59.535 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:45:59.535 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 33161 2024-02-09T17:45:59.535 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.535 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:45:59.535 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.535 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:45:59.535 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.892414+0000", 2024-02-09T17:45:59.535 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:59.536 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.536 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.536 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:45:59.536 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.892414+0000", 2024-02-09T17:45:59.536 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:59.536 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.536 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.536 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:45:59.536 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.892415+0000", 2024-02-09T17:45:59.536 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5300000000000004e-07 2024-02-09T17:45:59.536 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.536 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.536 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:45:59.536 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.892465+0000", 2024-02-09T17:45:59.537 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0263000000000003e-05 2024-02-09T17:45:59.537 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.537 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.537 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:45:59.537 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.892477+0000", 2024-02-09T17:45:59.537 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1676e-05 2024-02-09T17:45:59.537 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.537 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.537 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:45:59.537 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.892484+0000", 2024-02-09T17:45:59.537 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1049999999999997e-06 2024-02-09T17:45:59.537 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.537 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.537 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:59.538 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.893715+0000", 2024-02-09T17:45:59.538 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0012314170000000001 2024-02-09T17:45:59.538 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.538 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.538 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:45:59.538 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.893722+0000", 2024-02-09T17:45:59.538 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7399999999999998e-06 2024-02-09T17:45:59.538 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.538 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.538 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:59.538 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.884334+0000", 2024-02-09T17:45:59.538 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99061202599999998 2024-02-09T17:45:59.538 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.539 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.539 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:45:59.539 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.884466+0000", 2024-02-09T17:45:59.539 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013154 2024-02-09T17:45:59.539 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.539 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.539 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,11", 2024-02-09T17:45:59.539 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.884605+0000", 2024-02-09T17:45:59.539 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013961299999999999 2024-02-09T17:45:59.539 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.539 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.539 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:45:59.539 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.888743+0000", 2024-02-09T17:45:59.539 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041376140000000004 2024-02-09T17:45:59.539 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.540 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.540 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:59.540 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.888789+0000", 2024-02-09T17:45:59.540 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6183000000000002e-05 2024-02-09T17:45:59.540 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.540 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.540 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:59.540 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.888858+0000", 2024-02-09T17:45:59.540 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8800000000000005e-05 2024-02-09T17:45:59.540 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.540 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.540 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:45:59.540 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.888920+0000", 2024-02-09T17:45:59.541 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2472999999999996e-05 2024-02-09T17:45:59.541 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.541 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.541 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:45:59.541 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.888957+0000", 2024-02-09T17:45:59.541 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6902e-05 2024-02-09T17:45:59.541 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:59.541 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:45:59.541 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:59.541 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.541 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.541 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:33171 5.f 5:fb959e53:::benchmark_data_smithi179_115199_object33170:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e480)", 2024-02-09T17:45:59.541 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:41:19.897322+0000", 2024-02-09T17:45:59.541 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 279.597564202, 2024-02-09T17:45:59.542 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99605270000000001, 2024-02-09T17:45:59.542 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:45:59.542 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:45:59.542 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:45:59.542 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:45:59.542 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:45:59.542 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 33171 2024-02-09T17:45:59.542 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.542 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:45:59.542 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.542 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:45:59.542 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.897322+0000", 2024-02-09T17:45:59.542 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:59.542 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.542 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.543 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:45:59.543 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.897322+0000", 2024-02-09T17:45:59.543 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:59.543 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.543 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.543 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:45:59.543 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.897323+0000", 2024-02-09T17:45:59.543 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5499999999999997e-07 2024-02-09T17:45:59.543 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.543 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.543 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:45:59.543 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.897380+0000", 2024-02-09T17:45:59.543 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6919999999999997e-05 2024-02-09T17:45:59.543 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.544 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.544 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:45:59.544 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.897390+0000", 2024-02-09T17:45:59.544 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0555e-05 2024-02-09T17:45:59.544 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.544 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.544 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:45:59.544 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.897396+0000", 2024-02-09T17:45:59.544 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.716e-06 2024-02-09T17:45:59.544 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.544 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.544 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:59.544 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.898279+0000", 2024-02-09T17:45:59.545 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000883083 2024-02-09T17:45:59.545 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.545 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.545 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:45:59.545 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.898290+0000", 2024-02-09T17:45:59.545 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0774999999999999e-05 2024-02-09T17:45:59.545 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.545 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.545 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:59.545 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.886029+0000", 2024-02-09T17:45:59.545 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.98773960500000002 2024-02-09T17:45:59.545 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.545 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.545 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:45:59.546 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.886155+0000", 2024-02-09T17:45:59.546 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000125726 2024-02-09T17:45:59.546 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.546 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.546 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,11", 2024-02-09T17:45:59.546 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.886313+0000", 2024-02-09T17:45:59.546 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015792399999999999 2024-02-09T17:45:59.546 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.546 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.546 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:45:59.546 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.888800+0000", 2024-02-09T17:45:59.546 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0024873260000000002 2024-02-09T17:45:59.546 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.547 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.547 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:59.547 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.888823+0000", 2024-02-09T17:45:59.547 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2690000000000001e-05 2024-02-09T17:45:59.547 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.547 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.547 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:59.547 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.893269+0000", 2024-02-09T17:45:59.547 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0044462659999999999 2024-02-09T17:45:59.547 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.547 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.547 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:45:59.547 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.893329+0000", 2024-02-09T17:45:59.548 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9642999999999998e-05 2024-02-09T17:45:59.548 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.548 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.548 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:45:59.548 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.893375+0000", 2024-02-09T17:45:59.548 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5615999999999998e-05 2024-02-09T17:45:59.548 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:59.548 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:45:59.548 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:59.548 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.548 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.548 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:33173 5.f 5:ff65210a:::benchmark_data_smithi179_115199_object33172:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e480)", 2024-02-09T17:45:59.548 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:41:19.897448+0000", 2024-02-09T17:45:59.549 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 279.59743835299997, 2024-02-09T17:45:59.549 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99608639899999996, 2024-02-09T17:45:59.549 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:45:59.549 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:45:59.549 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:45:59.549 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:45:59.549 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:45:59.549 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 33173 2024-02-09T17:45:59.549 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.549 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:45:59.549 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.549 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:45:59.549 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.897448+0000", 2024-02-09T17:45:59.549 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:59.550 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.550 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.550 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:45:59.550 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.897448+0000", 2024-02-09T17:45:59.550 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:59.550 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.550 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.550 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:45:59.550 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.897448+0000", 2024-02-09T17:45:59.550 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2899999999999999e-07 2024-02-09T17:45:59.550 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.550 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.550 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:45:59.550 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.897516+0000", 2024-02-09T17:45:59.550 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8206000000000004e-05 2024-02-09T17:45:59.551 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.551 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.551 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:45:59.551 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.897524+0000", 2024-02-09T17:45:59.551 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9400000000000002e-06 2024-02-09T17:45:59.551 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.551 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.551 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:45:59.551 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.897528+0000", 2024-02-09T17:45:59.551 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9999999999999998e-06 2024-02-09T17:45:59.551 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.551 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.551 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:59.552 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.898384+0000", 2024-02-09T17:45:59.552 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00085551699999999997 2024-02-09T17:45:59.552 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.552 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.552 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:45:59.552 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.898390+0000", 2024-02-09T17:45:59.552 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8379999999999998e-06 2024-02-09T17:45:59.552 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.552 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.552 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:59.552 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.887400+0000", 2024-02-09T17:45:59.552 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.98901084100000003 2024-02-09T17:45:59.552 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.553 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.553 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:45:59.553 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.887534+0000", 2024-02-09T17:45:59.553 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013387100000000001 2024-02-09T17:45:59.553 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.553 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.553 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,11", 2024-02-09T17:45:59.553 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.887678+0000", 2024-02-09T17:45:59.553 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014406799999999999 2024-02-09T17:45:59.553 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.553 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.553 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:45:59.553 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.891737+0000", 2024-02-09T17:45:59.553 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0040586219999999996 2024-02-09T17:45:59.554 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.554 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.554 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:59.554 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.893201+0000", 2024-02-09T17:45:59.554 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014644359999999999 2024-02-09T17:45:59.554 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.554 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.554 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:59.554 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.893443+0000", 2024-02-09T17:45:59.554 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024162399999999999 2024-02-09T17:45:59.554 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.554 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.554 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:45:59.554 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.893494+0000", 2024-02-09T17:45:59.555 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1360999999999998e-05 2024-02-09T17:45:59.555 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.555 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.555 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:45:59.555 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.893534+0000", 2024-02-09T17:45:59.555 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9746000000000002e-05 2024-02-09T17:45:59.555 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:59.555 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:45:59.555 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:59.555 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.555 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.555 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:33179 5.f 5:f9bfb60a:::benchmark_data_smithi179_115199_object33178:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e480)", 2024-02-09T17:45:59.555 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:41:19.900843+0000", 2024-02-09T17:45:59.556 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 279.59404352299998, 2024-02-09T17:45:59.556 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99583007199999996, 2024-02-09T17:45:59.556 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:45:59.556 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:45:59.556 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:45:59.556 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:45:59.556 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:45:59.556 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 33179 2024-02-09T17:45:59.556 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.556 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:45:59.556 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.556 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:45:59.556 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.900843+0000", 2024-02-09T17:45:59.556 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:59.557 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.557 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.557 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:45:59.557 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.900843+0000", 2024-02-09T17:45:59.557 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:59.557 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.557 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.557 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:45:59.557 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.900843+0000", 2024-02-09T17:45:59.557 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2600000000000002e-07 2024-02-09T17:45:59.557 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.557 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.557 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:45:59.558 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.900874+0000", 2024-02-09T17:45:59.558 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0886000000000002e-05 2024-02-09T17:45:59.558 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.558 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.558 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:45:59.558 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.900880+0000", 2024-02-09T17:45:59.558 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.173e-06 2024-02-09T17:45:59.558 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.558 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.558 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:45:59.558 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.900885+0000", 2024-02-09T17:45:59.558 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2660000000000003e-06 2024-02-09T17:45:59.559 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.559 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.559 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:59.559 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.902133+0000", 2024-02-09T17:45:59.559 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001248859 2024-02-09T17:45:59.559 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.559 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.559 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:45:59.559 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.902140+0000", 2024-02-09T17:45:59.559 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4930000000000003e-06 2024-02-09T17:45:59.559 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.559 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.559 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:59.560 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.888995+0000", 2024-02-09T17:45:59.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.98685520100000002 2024-02-09T17:45:59.560 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.560 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.560 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:45:59.560 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.889125+0000", 2024-02-09T17:45:59.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012953800000000001 2024-02-09T17:45:59.560 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.560 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.560 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,11", 2024-02-09T17:45:59.560 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.889285+0000", 2024-02-09T17:45:59.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015997400000000001 2024-02-09T17:45:59.561 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.561 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.561 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:45:59.561 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.893396+0000", 2024-02-09T17:45:59.561 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041114929999999999 2024-02-09T17:45:59.561 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.561 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.561 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:59.561 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.893608+0000", 2024-02-09T17:45:59.561 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000211521 2024-02-09T17:45:59.561 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.561 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.561 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:59.561 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.896477+0000", 2024-02-09T17:45:59.562 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0028690370000000001 2024-02-09T17:45:59.562 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.562 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.562 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:45:59.562 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.896592+0000", 2024-02-09T17:45:59.562 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000115486 2024-02-09T17:45:59.562 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.562 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.562 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:45:59.562 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.896673+0000", 2024-02-09T17:45:59.562 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0519000000000006e-05 2024-02-09T17:45:59.562 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:59.562 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:45:59.562 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:59.563 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.563 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.563 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:33186 5.f 5:fdd239c2:::benchmark_data_smithi179_115199_object33185:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e480)", 2024-02-09T17:45:59.563 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:41:19.904132+0000", 2024-02-09T17:45:59.563 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 279.59075402299999, 2024-02-09T17:45:59.563 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99418198400000002, 2024-02-09T17:45:59.563 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:45:59.563 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:45:59.563 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:45:59.563 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:45:59.563 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:45:59.563 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 33186 2024-02-09T17:45:59.563 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.564 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:45:59.564 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.564 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:45:59.564 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.904132+0000", 2024-02-09T17:45:59.564 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:59.564 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.564 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.564 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:45:59.564 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.904132+0000", 2024-02-09T17:45:59.564 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:59.564 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.564 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.564 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:45:59.564 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.904133+0000", 2024-02-09T17:45:59.564 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2200000000000003e-07 2024-02-09T17:45:59.565 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.565 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.565 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:45:59.565 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.904177+0000", 2024-02-09T17:45:59.565 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3850000000000002e-05 2024-02-09T17:45:59.565 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.565 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.565 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:45:59.565 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.904187+0000", 2024-02-09T17:45:59.565 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9110000000000005e-06 2024-02-09T17:45:59.565 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.565 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.565 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:45:59.565 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.904192+0000", 2024-02-09T17:45:59.566 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1959999999999999e-06 2024-02-09T17:45:59.566 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.566 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.566 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:59.566 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.904246+0000", 2024-02-09T17:45:59.566 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.401e-05 2024-02-09T17:45:59.566 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.566 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.566 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:45:59.566 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.904253+0000", 2024-02-09T17:45:59.566 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0029999999999997e-06 2024-02-09T17:45:59.566 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.566 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.566 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:59.566 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.890398+0000", 2024-02-09T17:45:59.567 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.98614499200000005 2024-02-09T17:45:59.567 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.567 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:45:59.567 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.890527+0000", 2024-02-09T17:45:59.567 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012886500000000001 2024-02-09T17:45:59.567 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.567 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,11", 2024-02-09T17:45:59.567 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.890675+0000", 2024-02-09T17:45:59.567 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000148682 2024-02-09T17:45:59.567 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.567 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:45:59.567 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.893402+0000", 2024-02-09T17:45:59.568 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002726731 2024-02-09T17:45:59.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.568 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.568 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:59.568 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.896720+0000", 2024-02-09T17:45:59.568 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0033173299999999998 2024-02-09T17:45:59.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.568 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.568 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:59.568 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.898206+0000", 2024-02-09T17:45:59.568 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014864990000000001 2024-02-09T17:45:59.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.568 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.569 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:45:59.569 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.898286+0000", 2024-02-09T17:45:59.569 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9513999999999998e-05 2024-02-09T17:45:59.569 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.569 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.569 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:45:59.569 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.898314+0000", 2024-02-09T17:45:59.569 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8578999999999999e-05 2024-02-09T17:45:59.569 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:59.569 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:45:59.569 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:59.569 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.569 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.569 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:33219 5.f 5:ff57a091:::benchmark_data_smithi179_115199_object33218:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e480)", 2024-02-09T17:45:59.570 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:41:19.933494+0000", 2024-02-09T17:45:59.570 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 279.56139187299999, 2024-02-09T17:45:59.570 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.96495839699999997, 2024-02-09T17:45:59.570 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:45:59.570 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:45:59.570 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:45:59.570 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:45:59.570 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:45:59.570 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 33219 2024-02-09T17:45:59.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.570 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:45:59.570 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.570 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:45:59.570 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.933494+0000", 2024-02-09T17:45:59.571 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:59.571 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.571 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.571 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:45:59.571 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.933494+0000", 2024-02-09T17:45:59.571 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:59.571 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.571 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.571 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:45:59.571 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.933495+0000", 2024-02-09T17:45:59.571 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.85e-07 2024-02-09T17:45:59.571 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.571 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.571 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:45:59.572 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.933562+0000", 2024-02-09T17:45:59.572 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7018000000000001e-05 2024-02-09T17:45:59.572 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.572 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.572 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:45:59.572 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.933572+0000", 2024-02-09T17:45:59.572 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3249999999999997e-06 2024-02-09T17:45:59.572 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.572 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.572 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:45:59.572 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.933579+0000", 2024-02-09T17:45:59.572 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.041e-06 2024-02-09T17:45:59.573 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.573 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.573 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:59.573 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.933654+0000", 2024-02-09T17:45:59.573 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5210000000000007e-05 2024-02-09T17:45:59.573 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.573 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.573 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:45:59.573 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.933677+0000", 2024-02-09T17:45:59.573 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2849000000000001e-05 2024-02-09T17:45:59.573 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.573 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.573 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:59.573 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.891776+0000", 2024-02-09T17:45:59.574 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.95809916399999995 2024-02-09T17:45:59.574 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.574 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.574 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:45:59.574 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.891908+0000", 2024-02-09T17:45:59.574 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013222099999999999 2024-02-09T17:45:59.574 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.574 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.574 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,11", 2024-02-09T17:45:59.574 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.892056+0000", 2024-02-09T17:45:59.574 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014792200000000001 2024-02-09T17:45:59.574 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.574 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.575 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:45:59.575 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.893546+0000", 2024-02-09T17:45:59.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014900950000000001 2024-02-09T17:45:59.575 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.575 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.575 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:59.575 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.896771+0000", 2024-02-09T17:45:59.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0032249969999999998 2024-02-09T17:45:59.575 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.575 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.575 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:59.575 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.898381+0000", 2024-02-09T17:45:59.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001610385 2024-02-09T17:45:59.575 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.575 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.576 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:45:59.576 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.898429+0000", 2024-02-09T17:45:59.576 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7216999999999999e-05 2024-02-09T17:45:59.576 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.576 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.576 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:45:59.576 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.898453+0000", 2024-02-09T17:45:59.576 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4068000000000001e-05 2024-02-09T17:45:59.576 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:59.576 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:45:59.576 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:59.576 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.576 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.576 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:33236 5.f 5:fe55e4cc:::benchmark_data_smithi179_115199_object33235:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e480)", 2024-02-09T17:45:59.577 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:41:19.965076+0000", 2024-02-09T17:45:59.577 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 279.52981063700003, 2024-02-09T17:45:59.577 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.93345155099999999, 2024-02-09T17:45:59.577 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:45:59.577 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:45:59.577 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:45:59.577 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:45:59.577 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:45:59.577 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 33236 2024-02-09T17:45:59.577 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.577 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:45:59.577 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.577 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:45:59.577 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.965076+0000", 2024-02-09T17:45:59.578 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:59.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.578 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.578 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:45:59.578 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.965076+0000", 2024-02-09T17:45:59.578 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:59.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.578 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.578 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:45:59.578 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.965076+0000", 2024-02-09T17:45:59.578 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5099999999999998e-07 2024-02-09T17:45:59.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.578 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.578 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:45:59.578 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.965129+0000", 2024-02-09T17:45:59.579 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2562999999999998e-05 2024-02-09T17:45:59.579 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.579 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.579 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:45:59.579 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.965140+0000", 2024-02-09T17:45:59.579 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0628999999999999e-05 2024-02-09T17:45:59.579 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.579 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.579 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:45:59.579 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.965145+0000", 2024-02-09T17:45:59.579 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6180000000000003e-06 2024-02-09T17:45:59.579 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.579 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.579 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:59.579 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.965192+0000", 2024-02-09T17:45:59.580 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6230000000000003e-05 2024-02-09T17:45:59.580 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.580 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.580 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:45:59.580 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.965199+0000", 2024-02-09T17:45:59.580 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5970000000000003e-06 2024-02-09T17:45:59.580 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.580 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.580 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:59.580 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.893642+0000", 2024-02-09T17:45:59.580 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.928443346 2024-02-09T17:45:59.580 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.580 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.580 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:45:59.581 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.893772+0000", 2024-02-09T17:45:59.581 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000129107 2024-02-09T17:45:59.581 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.581 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.581 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,11", 2024-02-09T17:45:59.581 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.893896+0000", 2024-02-09T17:45:59.581 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012469400000000001 2024-02-09T17:45:59.581 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.581 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.581 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:45:59.581 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.896732+0000", 2024-02-09T17:45:59.581 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0028361509999999999 2024-02-09T17:45:59.581 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.581 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.582 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:59.582 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.898345+0000", 2024-02-09T17:45:59.582 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001612229 2024-02-09T17:45:59.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.582 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.582 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:59.582 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.898478+0000", 2024-02-09T17:45:59.582 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000133368 2024-02-09T17:45:59.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.582 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.582 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:45:59.582 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.898507+0000", 2024-02-09T17:45:59.582 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9377999999999999e-05 2024-02-09T17:45:59.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.582 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.583 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:45:59.583 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.898527+0000", 2024-02-09T17:45:59.583 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.969e-05 2024-02-09T17:45:59.583 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:59.583 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:45:59.583 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:59.583 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.583 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.583 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:74616 5.6 5:63c31902:::benchmark_data_smithi179_115199_object74615:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e489)", 2024-02-09T17:45:59.583 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:41:45.041982+0000", 2024-02-09T17:45:59.583 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 254.45290419299999, 2024-02-09T17:45:59.583 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19583514199999999, 2024-02-09T17:45:59.583 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:45:59.583 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:45:59.583 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:45:59.584 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:45:59.584 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:45:59.584 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 74616 2024-02-09T17:45:59.584 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.584 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:45:59.584 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.584 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:45:59.584 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:45.041982+0000", 2024-02-09T17:45:59.584 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:59.584 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.584 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.585 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:45:59.585 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:45.041982+0000", 2024-02-09T17:45:59.585 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:59.585 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.585 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.585 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:45:59.585 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:45.041983+0000", 2024-02-09T17:45:59.585 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2860000000000001e-06 2024-02-09T17:45:59.585 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.585 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.585 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:45:59.585 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:45.042103+0000", 2024-02-09T17:45:59.585 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011937700000000001 2024-02-09T17:45:59.585 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.586 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.586 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:45:59.586 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:45.042113+0000", 2024-02-09T17:45:59.586 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0144e-05 2024-02-09T17:45:59.586 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.586 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.586 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:45:59.586 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:45.042120+0000", 2024-02-09T17:45:59.586 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9539999999999998e-06 2024-02-09T17:45:59.586 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.586 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.586 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:59.586 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:45.042173+0000", 2024-02-09T17:45:59.586 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3300999999999997e-05 2024-02-09T17:45:59.587 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.587 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.587 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:45:59.587 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:45.042315+0000", 2024-02-09T17:45:59.587 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014153200000000001 2024-02-09T17:45:59.587 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.587 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.587 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7,10", 2024-02-09T17:45:59.587 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:45.042428+0000", 2024-02-09T17:45:59.587 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011296800000000001 2024-02-09T17:45:59.587 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.587 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.587 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:45:59.588 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:45.047248+0000", 2024-02-09T17:45:59.588 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0048207079999999999 2024-02-09T17:45:59.588 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.588 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.588 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:59.588 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:45.048755+0000", 2024-02-09T17:45:59.588 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0015067839999999999 2024-02-09T17:45:59.588 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.588 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.588 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:59.588 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:45.237697+0000", 2024-02-09T17:45:59.588 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18894223499999999 2024-02-09T17:45:59.588 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.589 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.589 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:45:59.589 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:45.237757+0000", 2024-02-09T17:45:59.589 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9617000000000002e-05 2024-02-09T17:45:59.589 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.589 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.589 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:45:59.589 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:45.237817+0000", 2024-02-09T17:45:59.589 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0235999999999997e-05 2024-02-09T17:45:59.589 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:59.589 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:45:59.589 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:59.589 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.589 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.590 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:168352 5.6 5:645ef9c0:::benchmark_data_smithi179_115199_object39446:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e520)", 2024-02-09T17:45:59.590 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:43:21.023013+0000", 2024-02-09T17:45:59.590 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 158.47187287599999, 2024-02-09T17:45:59.590 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.234114989, 2024-02-09T17:45:59.590 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:45:59.590 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:45:59.590 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:45:59.590 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:45:59.590 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:45:59.590 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 168352 2024-02-09T17:45:59.590 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.590 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:45:59.590 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.590 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:45:59.591 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.023013+0000", 2024-02-09T17:45:59.591 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:59.591 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.591 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.591 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:45:59.591 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.023013+0000", 2024-02-09T17:45:59.591 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:59.591 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.591 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.591 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:45:59.591 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.023014+0000", 2024-02-09T17:45:59.591 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2299999999999999e-07 2024-02-09T17:45:59.591 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.591 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.592 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:45:59.592 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.023019+0000", 2024-02-09T17:45:59.592 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.118e-06 2024-02-09T17:45:59.592 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.592 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.592 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:45:59.592 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.023030+0000", 2024-02-09T17:45:59.592 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0688e-05 2024-02-09T17:45:59.592 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.592 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.592 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:45:59.592 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.023038+0000", 2024-02-09T17:45:59.592 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8259999999999993e-06 2024-02-09T17:45:59.592 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.593 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.593 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:59.593 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.023094+0000", 2024-02-09T17:45:59.593 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6164e-05 2024-02-09T17:45:59.593 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.593 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.593 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:45:59.593 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.023198+0000", 2024-02-09T17:45:59.593 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000104384 2024-02-09T17:45:59.593 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.593 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.593 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7,10", 2024-02-09T17:45:59.593 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.023297+0000", 2024-02-09T17:45:59.593 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9057999999999999e-05 2024-02-09T17:45:59.594 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.594 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.594 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:45:59.594 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.028576+0000", 2024-02-09T17:45:59.594 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0052786509999999997 2024-02-09T17:45:59.594 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.594 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.594 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:59.594 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.033487+0000", 2024-02-09T17:45:59.594 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0049112899999999996 2024-02-09T17:45:59.594 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.594 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.594 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:59.594 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.257054+0000", 2024-02-09T17:45:59.594 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22356674900000001 2024-02-09T17:45:59.595 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.595 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.595 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:45:59.595 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.257092+0000", 2024-02-09T17:45:59.595 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7500999999999999e-05 2024-02-09T17:45:59.595 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.595 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.595 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:45:59.595 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.257128+0000", 2024-02-09T17:45:59.595 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6637000000000002e-05 2024-02-09T17:45:59.595 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:59.595 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:45:59.595 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:59.595 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.596 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.596 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:168354 5.6 5:62688f71:::benchmark_data_smithi179_115199_object39448:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e520)", 2024-02-09T17:45:59.596 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:43:21.028975+0000", 2024-02-09T17:45:59.596 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 158.46591130600001, 2024-02-09T17:45:59.596 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.228236679, 2024-02-09T17:45:59.596 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:45:59.596 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:45:59.596 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:45:59.596 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:45:59.596 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:45:59.596 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 168354 2024-02-09T17:45:59.596 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.596 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:45:59.597 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.597 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:45:59.597 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.028975+0000", 2024-02-09T17:45:59.597 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:59.597 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.597 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.597 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:45:59.597 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.028975+0000", 2024-02-09T17:45:59.597 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:59.597 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.597 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.597 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:45:59.597 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.028975+0000", 2024-02-09T17:45:59.597 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.32e-07 2024-02-09T17:45:59.597 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.598 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.598 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:45:59.598 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.028977+0000", 2024-02-09T17:45:59.598 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0779999999999998e-06 2024-02-09T17:45:59.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.598 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.598 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:45:59.598 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.028985+0000", 2024-02-09T17:45:59.598 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.515e-06 2024-02-09T17:45:59.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.598 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.598 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:45:59.598 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.029003+0000", 2024-02-09T17:45:59.598 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8247999999999999e-05 2024-02-09T17:45:59.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.599 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.599 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:59.599 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.029055+0000", 2024-02-09T17:45:59.599 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1876000000000003e-05 2024-02-09T17:45:59.599 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.599 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.599 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:45:59.599 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.029175+0000", 2024-02-09T17:45:59.599 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000119752 2024-02-09T17:45:59.599 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.599 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.599 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7,10", 2024-02-09T17:45:59.599 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.029304+0000", 2024-02-09T17:45:59.599 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000129524 2024-02-09T17:45:59.599 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.600 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.600 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:59.600 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.038462+0000", 2024-02-09T17:45:59.600 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0091575519999999994 2024-02-09T17:45:59.600 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.600 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.600 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:45:59.600 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.038479+0000", 2024-02-09T17:45:59.600 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7393000000000001e-05 2024-02-09T17:45:59.600 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.600 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.600 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:59.600 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.257154+0000", 2024-02-09T17:45:59.600 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21867436200000001 2024-02-09T17:45:59.601 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.601 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.601 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:45:59.601 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.257181+0000", 2024-02-09T17:45:59.601 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7747999999999999e-05 2024-02-09T17:45:59.601 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.601 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.601 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:45:59.601 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.257212+0000", 2024-02-09T17:45:59.601 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0199e-05 2024-02-09T17:45:59.601 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:59.601 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:45:59.601 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:59.602 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.602 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.602 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:168372 5.6 5:6fd0bce5:::benchmark_data_smithi179_115199_object39466:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e520)", 2024-02-09T17:45:59.602 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:43:21.059240+0000", 2024-02-09T17:45:59.602 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 158.43564600100001, 2024-02-09T17:45:59.602 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.198070792, 2024-02-09T17:45:59.602 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:45:59.602 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:45:59.602 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:45:59.602 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:45:59.602 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:45:59.602 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 168372 2024-02-09T17:45:59.603 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.603 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:45:59.603 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.603 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:45:59.603 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.059240+0000", 2024-02-09T17:45:59.603 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:59.603 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.603 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.603 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:45:59.603 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.059240+0000", 2024-02-09T17:45:59.603 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:59.603 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.603 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.603 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:45:59.604 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.059242+0000", 2024-02-09T17:45:59.604 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7999999999999999e-06 2024-02-09T17:45:59.604 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.604 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.604 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:45:59.604 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.059246+0000", 2024-02-09T17:45:59.604 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4490000000000001e-06 2024-02-09T17:45:59.604 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.604 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.604 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:45:59.604 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.059254+0000", 2024-02-09T17:45:59.604 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9319999999999996e-06 2024-02-09T17:45:59.604 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.604 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.605 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:45:59.605 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.059259+0000", 2024-02-09T17:45:59.605 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1340000000000003e-06 2024-02-09T17:45:59.605 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.605 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.605 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:59.605 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.059306+0000", 2024-02-09T17:45:59.605 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6492000000000001e-05 2024-02-09T17:45:59.605 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.605 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.605 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:45:59.605 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.059431+0000", 2024-02-09T17:45:59.605 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012514700000000001 2024-02-09T17:45:59.605 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.606 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.606 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7,10", 2024-02-09T17:45:59.606 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.059520+0000", 2024-02-09T17:45:59.606 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8789999999999995e-05 2024-02-09T17:45:59.606 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.606 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.606 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:45:59.606 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.068663+0000", 2024-02-09T17:45:59.606 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0091426000000000007 2024-02-09T17:45:59.606 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.606 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.606 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:59.606 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.070771+0000", 2024-02-09T17:45:59.606 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021080729999999998 2024-02-09T17:45:59.607 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.607 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.607 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:59.607 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.257247+0000", 2024-02-09T17:45:59.607 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.186476065 2024-02-09T17:45:59.607 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.607 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.607 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:45:59.607 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.257281+0000", 2024-02-09T17:45:59.607 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3952000000000003e-05 2024-02-09T17:45:59.607 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.607 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.607 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:45:59.607 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.257311+0000", 2024-02-09T17:45:59.608 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0358e-05 2024-02-09T17:45:59.608 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:59.608 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:45:59.608 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:59.608 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.608 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.608 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:170463 5.6 5:6cc3e09d:::benchmark_data_smithi179_115199_object41557:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e522)", 2024-02-09T17:45:59.608 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:43:25.160777+0000", 2024-02-09T17:45:59.608 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 154.33410956, 2024-02-09T17:45:59.608 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.754693777, 2024-02-09T17:45:59.608 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:45:59.608 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-09T17:45:59.608 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:45:59.609 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:45:59.609 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:45:59.609 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 170463 2024-02-09T17:45:59.609 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.609 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:45:59.609 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.609 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:45:59.609 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:25.160777+0000", 2024-02-09T17:45:59.609 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:59.609 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.609 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.609 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:45:59.609 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:25.160777+0000", 2024-02-09T17:45:59.609 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:59.610 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.610 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.610 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:45:59.610 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:25.160778+0000", 2024-02-09T17:45:59.610 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8589999999999999e-06 2024-02-09T17:45:59.610 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.610 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.610 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:45:59.610 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:25.160782+0000", 2024-02-09T17:45:59.610 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.72e-06 2024-02-09T17:45:59.610 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.610 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.610 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:45:59.610 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:25.160792+0000", 2024-02-09T17:45:59.611 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0268000000000001e-05 2024-02-09T17:45:59.611 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.611 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.611 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:45:59.611 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:25.160799+0000", 2024-02-09T17:45:59.611 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.934e-06 2024-02-09T17:45:59.611 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.611 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.611 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:59.611 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:25.160847+0000", 2024-02-09T17:45:59.611 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7330999999999998e-05 2024-02-09T17:45:59.611 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.611 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.611 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:45:59.612 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:25.160857+0000", 2024-02-09T17:45:59.612 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0726e-05 2024-02-09T17:45:59.612 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.612 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.612 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:59.612 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:26.915388+0000", 2024-02-09T17:45:59.612 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.754530608 2024-02-09T17:45:59.612 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.612 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.612 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:45:59.612 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:26.915470+0000", 2024-02-09T17:45:59.612 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2330999999999995e-05 2024-02-09T17:45:59.612 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:59.612 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:45:59.613 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:59.613 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.613 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.613 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:170464 5.6 5:6e5e1d76:::benchmark_data_smithi179_115199_object41558:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e522)", 2024-02-09T17:45:59.613 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:43:25.160814+0000", 2024-02-09T17:45:59.613 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 154.33407213999999, 2024-02-09T17:45:59.613 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7549528620000001, 2024-02-09T17:45:59.613 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:45:59.613 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-09T17:45:59.613 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:45:59.613 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:45:59.613 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:45:59.613 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 170464 2024-02-09T17:45:59.613 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.614 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:45:59.614 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.614 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:45:59.614 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:25.160814+0000", 2024-02-09T17:45:59.614 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:59.614 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.614 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.614 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:45:59.614 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:25.160814+0000", 2024-02-09T17:45:59.614 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:59.614 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.614 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.614 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:45:59.614 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:25.160815+0000", 2024-02-09T17:45:59.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9100000000000004e-07 2024-02-09T17:45:59.615 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.615 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:45:59.615 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:25.160817+0000", 2024-02-09T17:45:59.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.238e-06 2024-02-09T17:45:59.615 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.615 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:45:59.615 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:25.160825+0000", 2024-02-09T17:45:59.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0260000000000007e-06 2024-02-09T17:45:59.615 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.615 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:45:59.616 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:25.160830+0000", 2024-02-09T17:45:59.616 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0930000000000002e-06 2024-02-09T17:45:59.616 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.616 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.616 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:59.616 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:25.160883+0000", 2024-02-09T17:45:59.616 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3606000000000001e-05 2024-02-09T17:45:59.616 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.616 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.616 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:45:59.616 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:25.160891+0000", 2024-02-09T17:45:59.616 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2060000000000001e-06 2024-02-09T17:45:59.616 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.617 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:59.617 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:26.915714+0000", 2024-02-09T17:45:59.617 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.754823577 2024-02-09T17:45:59.617 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.617 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:45:59.617 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:26.915767+0000", 2024-02-09T17:45:59.617 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2624999999999998e-05 2024-02-09T17:45:59.617 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:59.617 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:45:59.617 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:59.617 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.618 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:170477 5.6 5:6e670660:::benchmark_data_smithi179_115199_object41571:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e522)", 2024-02-09T17:45:59.618 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:43:25.160837+0000", 2024-02-09T17:45:59.618 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 154.33404939900001, 2024-02-09T17:45:59.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7656936920000001, 2024-02-09T17:45:59.618 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:45:59.618 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:45:59.618 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:45:59.618 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:45:59.618 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:45:59.618 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 170477 2024-02-09T17:45:59.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.618 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:45:59.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.619 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:45:59.619 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:25.160837+0000", 2024-02-09T17:45:59.619 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:59.619 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.619 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.619 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:45:59.619 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:25.160837+0000", 2024-02-09T17:45:59.619 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:59.619 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.619 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.619 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:45:59.619 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:25.160837+0000", 2024-02-09T17:45:59.619 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8000000000000001e-07 2024-02-09T17:45:59.620 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.620 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.620 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:45:59.620 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:25.160839+0000", 2024-02-09T17:45:59.620 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.277e-06 2024-02-09T17:45:59.620 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.620 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.620 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:45:59.620 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:25.160846+0000", 2024-02-09T17:45:59.620 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5400000000000001e-06 2024-02-09T17:45:59.620 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.620 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.620 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:45:59.621 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:25.160852+0000", 2024-02-09T17:45:59.621 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7400000000000001e-06 2024-02-09T17:45:59.621 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.621 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.621 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:59.621 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:25.160908+0000", 2024-02-09T17:45:59.621 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6518999999999999e-05 2024-02-09T17:45:59.621 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.621 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.621 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:45:59.621 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:25.160912+0000", 2024-02-09T17:45:59.621 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8770000000000003e-06 2024-02-09T17:45:59.621 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.621 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.622 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:59.622 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:26.915817+0000", 2024-02-09T17:45:59.622 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7549047770000001 2024-02-09T17:45:59.622 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.622 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.622 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:45:59.622 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:26.915912+0000", 2024-02-09T17:45:59.622 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5142000000000005e-05 2024-02-09T17:45:59.622 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.622 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.622 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,7,10", 2024-02-09T17:45:59.622 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:26.916010+0000", 2024-02-09T17:45:59.622 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7956999999999997e-05 2024-02-09T17:45:59.622 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.623 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:59.623 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:26.920814+0000", 2024-02-09T17:45:59.623 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0048041680000000002 2024-02-09T17:45:59.623 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.623 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:45:59.623 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:26.920835+0000", 2024-02-09T17:45:59.623 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1160999999999999e-05 2024-02-09T17:45:59.623 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.623 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:59.623 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:26.921295+0000", 2024-02-09T17:45:59.624 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000459693 2024-02-09T17:45:59.624 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.624 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.624 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:59.624 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:26.926323+0000", 2024-02-09T17:45:59.624 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.005028444 2024-02-09T17:45:59.624 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.624 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.624 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:45:59.624 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:26.926405+0000", 2024-02-09T17:45:59.624 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1656999999999994e-05 2024-02-09T17:45:59.624 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.624 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.625 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:45:59.625 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:26.926530+0000", 2024-02-09T17:45:59.625 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012536000000000001 2024-02-09T17:45:59.625 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:59.625 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:45:59.625 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:59.625 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.625 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.625 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:183800 5.6 5:6f507896:::benchmark_data_smithi179_115199_object54894:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e533)", 2024-02-09T17:45:59.625 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:43:43.785304+0000", 2024-02-09T17:45:59.625 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 135.70958209899999, 2024-02-09T17:45:59.625 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.206321332, 2024-02-09T17:45:59.625 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:45:59.625 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:45:59.625 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:45:59.626 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:45:59.626 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:45:59.626 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 183800 2024-02-09T17:45:59.626 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.626 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:45:59.626 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.626 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:45:59.626 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:43.785304+0000", 2024-02-09T17:45:59.626 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:59.626 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.626 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.626 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:45:59.626 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:43.785304+0000", 2024-02-09T17:45:59.626 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:45:59.627 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.627 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.627 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:45:59.627 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:43.785305+0000", 2024-02-09T17:45:59.627 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5999999999999999e-07 2024-02-09T17:45:59.627 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.627 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.627 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:45:59.627 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:43.785307+0000", 2024-02-09T17:45:59.627 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0219999999999999e-06 2024-02-09T17:45:59.627 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.627 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.627 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:45:59.627 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:43.785312+0000", 2024-02-09T17:45:59.628 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7860000000000002e-06 2024-02-09T17:45:59.628 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.628 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.628 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:45:59.628 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:43.785317+0000", 2024-02-09T17:45:59.628 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.826e-06 2024-02-09T17:45:59.628 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.628 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.628 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:45:59.628 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:43.785369+0000", 2024-02-09T17:45:59.628 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1663999999999999e-05 2024-02-09T17:45:59.628 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.628 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.628 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:45:59.629 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:43.785494+0000", 2024-02-09T17:45:59.629 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012557599999999999 2024-02-09T17:45:59.629 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.629 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.629 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,7,10", 2024-02-09T17:45:59.629 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:43.785618+0000", 2024-02-09T17:45:59.629 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012398800000000001 2024-02-09T17:45:59.629 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.629 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.629 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:59.629 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:43.798410+0000", 2024-02-09T17:45:59.629 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012791234 2024-02-09T17:45:59.629 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.629 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.629 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:45:59.630 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:43.798435+0000", 2024-02-09T17:45:59.630 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5100999999999999e-05 2024-02-09T17:45:59.630 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.630 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.630 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:59.630 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:43.811382+0000", 2024-02-09T17:45:59.630 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012947452 2024-02-09T17:45:59.630 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.630 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.630 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:45:59.630 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:43.991445+0000", 2024-02-09T17:45:59.630 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.180062572 2024-02-09T17:45:59.630 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.631 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:45:59.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:43.991503+0000", 2024-02-09T17:45:59.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7794000000000003e-05 2024-02-09T17:45:59.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:45:59.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:45:59.631 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:45:59.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:43.991625+0000", 2024-02-09T17:45:59.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012285700000000001 2024-02-09T17:45:59.631 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:59.631 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:45:59.631 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:59.631 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:45:59.631 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:45:59.632 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:45:59.632 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:45:59.660 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:00.008 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] 2024-02-09T17:46:00.008 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:46:00.008 INFO:tasks.thrashosds.thrasher:Reweighting osd 10 to 0.375345637646218 2024-02-09T17:46:00.008 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 10 0.375345637646218 2024-02-09T17:46:00.096 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 8 15 22386 22371 174.757 134.438 0.00169288 0.0057173 2024-02-09T17:46:00.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:45:59 smithi162 ceph-mon[44304]: pgmap v1457: 47 pgs: 47 active+clean; 852 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.36k op/s 2024-02-09T17:46:00.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:45:59 smithi099 ceph-mon[39420]: pgmap v1457: 47 pgs: 47 active+clean; 852 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.36k op/s 2024-02-09T17:46:00.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:45:59 smithi179 ceph-mon[43975]: pgmap v1457: 47 pgs: 47 active+clean; 852 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.36k op/s 2024-02-09T17:46:00.636 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-09T17:46:01.075 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:46:01.096 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 9 16 24966 24950 173.248 161.188 0.00524807 0.00576735 2024-02-09T17:46:01.121 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:46:01.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:01.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:46:01.227 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:46:01.265 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:02.096 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 10 16 27676 27660 172.859 169.375 0.00550299 0.00578 2024-02-09T17:46:02.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:01 smithi162 ceph-mon[44304]: pgmap v1458: 47 pgs: 47 active+clean; 1.1 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 98 MiB/s wr, 1.57k op/s 2024-02-09T17:46:02.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:01 smithi099 ceph-mon[39420]: pgmap v1458: 47 pgs: 47 active+clean; 1.1 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 98 MiB/s wr, 1.57k op/s 2024-02-09T17:46:02.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:01 smithi179 ceph-mon[43975]: pgmap v1458: 47 pgs: 47 active+clean; 1.1 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 98 MiB/s wr, 1.57k op/s 2024-02-09T17:46:03.020 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:46:03.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:03.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:46:03.097 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 11 15 29823 29808 169.348 134.25 0.00403238 0.00590009 2024-02-09T17:46:03.124 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:46:03.177 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:03.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:02 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:46:03.253 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:46:03.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:02 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:46:03.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:02 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:46:03.534 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:46:03.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:03.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:46:03.639 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:46:03.684 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:04.097 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 12 16 31956 31940 166.339 133.25 0.00204168 0.0059443 2024-02-09T17:46:04.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:03 smithi099 ceph-mon[39420]: pgmap v1459: 47 pgs: 47 active+clean; 1.5 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 130 MiB/s wr, 2.08k op/s 2024-02-09T17:46:04.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:03 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:46:04.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:03 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:46:04.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:03 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:46:04.405 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:03 smithi162 ceph-mon[44304]: pgmap v1459: 47 pgs: 47 active+clean; 1.5 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 130 MiB/s wr, 2.08k op/s 2024-02-09T17:46:04.405 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:03 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:46:04.405 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:03 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:46:04.405 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:03 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:46:04.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:03 smithi179 ceph-mon[43975]: pgmap v1459: 47 pgs: 47 active+clean; 1.5 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 130 MiB/s wr, 2.08k op/s 2024-02-09T17:46:04.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:03 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:46:04.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:03 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:46:04.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:03 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:46:04.873 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:46:04.873 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-02-09T17:46:04.873 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:04.873 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25837.0:33347 6.3 e580/576 6:c14dd184:::benchmark_data_smithi179_126206_object33346:head v 580'2093, mlcod=580'2091)", 2024-02-09T17:46:04.873 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:46:04.865378+0000", 2024-02-09T17:46:04.873 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0029670349999999998, 2024-02-09T17:46:04.873 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0029983789999999998, 2024-02-09T17:46:04.873 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:46:04.873 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-09T17:46:04.874 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:46:04.874 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:04.874 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:46:04.874 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:04.865378+0000", 2024-02-09T17:46:04.874 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:46:04.874 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:04.874 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:04.874 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:46:04.874 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:04.865378+0000", 2024-02-09T17:46:04.874 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:46:04.874 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:04.874 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:04.874 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:46:04.875 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:04.865379+0000", 2024-02-09T17:46:04.875 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3499999999999996e-07 2024-02-09T17:46:04.875 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:04.875 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:04.875 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:46:04.875 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:04.865411+0000", 2024-02-09T17:46:04.875 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1860999999999999e-05 2024-02-09T17:46:04.875 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:04.875 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:04.875 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:46:04.875 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:04.865421+0000", 2024-02-09T17:46:04.875 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7079999999999992e-06 2024-02-09T17:46:04.875 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:04.875 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:04.876 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:46:04.876 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:04.865430+0000", 2024-02-09T17:46:04.876 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2720000000000003e-06 2024-02-09T17:46:04.876 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:04.876 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:04.876 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:46:04.876 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:04.865503+0000", 2024-02-09T17:46:04.876 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3458e-05 2024-02-09T17:46:04.876 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:04.876 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:04.876 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:46:04.876 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:04.865529+0000", 2024-02-09T17:46:04.877 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5103999999999999e-05 2024-02-09T17:46:04.877 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:46:04.877 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:46:04.877 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:46:04.877 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:04.877 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:04.877 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25837.0:33351 6.c e580/576 6:3f496203:::benchmark_data_smithi179_126206_object33350:head v 580'2141, mlcod=580'2138)", 2024-02-09T17:46:04.877 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:46:04.865930+0000", 2024-02-09T17:46:04.877 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0024154670000000001, 2024-02-09T17:46:04.877 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0025239300000000002, 2024-02-09T17:46:04.877 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:46:04.877 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-09T17:46:04.877 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:46:04.877 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:04.878 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:46:04.878 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:04.865930+0000", 2024-02-09T17:46:04.878 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:46:04.878 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:04.878 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:04.878 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:46:04.878 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:04.865930+0000", 2024-02-09T17:46:04.878 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:46:04.878 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:04.878 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:04.878 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:46:04.878 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:04.865931+0000", 2024-02-09T17:46:04.878 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8100000000000002e-07 2024-02-09T17:46:04.879 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:04.879 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:04.879 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:46:04.879 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:04.866008+0000", 2024-02-09T17:46:04.879 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7568e-05 2024-02-09T17:46:04.879 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:04.879 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:04.879 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:46:04.879 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:04.866018+0000", 2024-02-09T17:46:04.879 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6059999999999992e-06 2024-02-09T17:46:04.879 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:04.879 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:04.879 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:46:04.879 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:04.866023+0000", 2024-02-09T17:46:04.880 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9230000000000001e-06 2024-02-09T17:46:04.880 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:04.880 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:04.880 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:46:04.880 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:04.866072+0000", 2024-02-09T17:46:04.880 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8705000000000002e-05 2024-02-09T17:46:04.880 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:04.880 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:04.880 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:46:04.880 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:04.866086+0000", 2024-02-09T17:46:04.880 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4663e-05 2024-02-09T17:46:04.880 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:46:04.880 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:46:04.880 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:46:04.881 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:04.881 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:04.881 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25837.0:33348 6.a e580/576 6:52e4ef5c:::benchmark_data_smithi179_126206_object33347:head v 580'2114, mlcod=580'2110)", 2024-02-09T17:46:04.881 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:46:04.866789+0000", 2024-02-09T17:46:04.881 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0015561150000000001, 2024-02-09T17:46:04.881 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017338410000000001, 2024-02-09T17:46:04.881 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:46:04.881 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-09T17:46:04.881 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:46:04.881 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:04.881 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:46:04.881 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:04.866789+0000", 2024-02-09T17:46:04.881 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:46:04.882 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:04.882 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:04.882 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:46:04.882 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:04.866789+0000", 2024-02-09T17:46:04.882 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:46:04.882 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:04.882 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:04.882 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:46:04.882 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:04.866790+0000", 2024-02-09T17:46:04.882 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2999999999999999e-07 2024-02-09T17:46:04.882 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:04.882 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:04.882 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:46:04.882 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:04.866813+0000", 2024-02-09T17:46:04.883 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2286999999999999e-05 2024-02-09T17:46:04.883 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:04.883 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:04.883 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:46:04.883 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:04.866826+0000", 2024-02-09T17:46:04.883 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3893999999999999e-05 2024-02-09T17:46:04.883 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:04.883 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:04.883 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:46:04.883 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:04.866834+0000", 2024-02-09T17:46:04.883 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1470000000000004e-06 2024-02-09T17:46:04.883 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:04.883 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:04.884 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:46:04.884 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:04.866889+0000", 2024-02-09T17:46:04.884 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5161000000000003e-05 2024-02-09T17:46:04.884 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:04.884 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:04.884 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:46:04.884 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:04.866907+0000", 2024-02-09T17:46:04.884 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8572000000000001e-05 2024-02-09T17:46:04.884 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:46:04.884 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:46:04.884 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:46:04.885 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:04.885 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:04.885 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25837.0:33340 6.c e580/576 6:394eb63f:::benchmark_data_smithi179_126206_object33339:head v 580'2140, mlcod=580'2137)", 2024-02-09T17:46:04.885 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:46:04.863154+0000", 2024-02-09T17:46:04.885 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0051916189999999997, 2024-02-09T17:46:04.885 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00543978, 2024-02-09T17:46:04.885 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:46:04.885 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-09T17:46:04.885 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:46:04.885 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:04.885 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:46:04.886 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:04.863154+0000", 2024-02-09T17:46:04.886 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:46:04.886 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:04.886 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:04.886 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:46:04.886 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:04.863154+0000", 2024-02-09T17:46:04.886 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:46:04.886 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:04.886 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:04.886 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:46:04.886 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:04.863155+0000", 2024-02-09T17:46:04.887 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.09e-07 2024-02-09T17:46:04.887 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:04.887 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:04.887 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:46:04.887 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:04.863184+0000", 2024-02-09T17:46:04.887 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9519e-05 2024-02-09T17:46:04.887 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:04.887 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:04.887 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:46:04.887 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:04.863193+0000", 2024-02-09T17:46:04.887 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3610000000000001e-06 2024-02-09T17:46:04.887 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:04.888 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:04.888 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:46:04.888 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:04.863198+0000", 2024-02-09T17:46:04.888 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3349999999999997e-06 2024-02-09T17:46:04.888 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:04.888 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:04.888 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:46:04.888 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:04.863244+0000", 2024-02-09T17:46:04.888 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5714999999999999e-05 2024-02-09T17:46:04.888 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:04.888 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:04.889 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:46:04.889 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:04.863262+0000", 2024-02-09T17:46:04.889 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7847999999999999e-05 2024-02-09T17:46:04.889 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:46:04.889 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:46:04.889 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:46:04.889 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:46:04.889 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-09T17:46:04.889 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 4 2024-02-09T17:46:04.889 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:46:05.022 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:46:05.097 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 13 16 33915 33899 162.961 122.438 0.00527864 0.00613206 2024-02-09T17:46:05.554 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:46:05.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:05.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:46:05.659 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:46:05.692 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:05.717 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:46:05.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:05.721 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:46:05.822 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:46:05.843 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:46:05.862 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:06.097 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 14 16 35947 35931 160.392 127 0.00592472 0.00623073 2024-02-09T17:46:06.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:05 smithi162 ceph-mon[44304]: pgmap v1460: 47 pgs: 47 active+clean; 1.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 147 MiB/s wr, 2.35k op/s 2024-02-09T17:46:06.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:05 smithi099 ceph-mon[39420]: pgmap v1460: 47 pgs: 47 active+clean; 1.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 147 MiB/s wr, 2.35k op/s 2024-02-09T17:46:06.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:05 smithi179 ceph-mon[43975]: pgmap v1460: 47 pgs: 47 active+clean; 1.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 147 MiB/s wr, 2.35k op/s 2024-02-09T17:46:06.825 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:46:06.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:06.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:46:06.866 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-09T17:46:06.934 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:46:06.967 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:06.991 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:46:06.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:06.995 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:46:06.995 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is set 2024-02-09T17:46:07.096 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:46:07.097 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 15 16 38269 38253 159.373 145.125 0.00664916 0.00627002 2024-02-09T17:46:07.131 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:06 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2993610604' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:46:07.131 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:06 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:46:07.134 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:07.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:06 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2993610604' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:46:07.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:06 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:46:07.292 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:46:07.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:07.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:46:07.326 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:46:07.401 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:46:07.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:06 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2993610604' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:46:07.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:06 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:46:07.450 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:08.097 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 16 16 40291 40275 157.31 126.375 0.00291832 0.00635157 2024-02-09T17:46:08.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:07 smithi099 ceph-mon[39420]: pgmap v1461: 47 pgs: 47 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 149 MiB/s wr, 2.38k op/s 2024-02-09T17:46:08.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:07 smithi099 ceph-mon[39420]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:46:08.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:07 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:46:08.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:07 smithi099 ceph-mon[39420]: osdmap e581: 12 total, 12 up, 12 in 2024-02-09T17:46:08.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:07 smithi179 ceph-mon[43975]: pgmap v1461: 47 pgs: 47 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 149 MiB/s wr, 2.38k op/s 2024-02-09T17:46:08.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:07 smithi179 ceph-mon[43975]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:46:08.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:07 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:46:08.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:07 smithi179 ceph-mon[43975]: osdmap e581: 12 total, 12 up, 12 in 2024-02-09T17:46:08.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:07 smithi162 ceph-mon[44304]: pgmap v1461: 47 pgs: 47 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 149 MiB/s wr, 2.38k op/s 2024-02-09T17:46:08.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:07 smithi162 ceph-mon[44304]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:46:08.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:07 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:46:08.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:07 smithi162 ceph-mon[44304]: osdmap e581: 12 total, 12 up, 12 in 2024-02-09T17:46:09.097 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 17 16 42298 42282 155.435 125.438 0.0112298 0.00642839 2024-02-09T17:46:09.272 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.10 to 0.375346 (6016) 2024-02-09T17:46:09.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:09 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:46:09.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:09 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:46:09.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:09 smithi179 ceph-mon[43975]: osdmap e582: 12 total, 12 up, 12 in 2024-02-09T17:46:09.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:09 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3057287810' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 0.375345637646218}]: dispatch 2024-02-09T17:46:09.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:09 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 0.375345637646218}]: dispatch 2024-02-09T17:46:09.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:09 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:46:09.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:09 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:46:09.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:09 smithi162 ceph-mon[44304]: osdmap e582: 12 total, 12 up, 12 in 2024-02-09T17:46:09.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:09 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3057287810' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 0.375345637646218}]: dispatch 2024-02-09T17:46:09.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:09 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 0.375345637646218}]: dispatch 2024-02-09T17:46:09.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:09 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:46:09.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:09 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:46:09.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:09 smithi099 ceph-mon[39420]: osdmap e582: 12 total, 12 up, 12 in 2024-02-09T17:46:09.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:09 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3057287810' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 0.375345637646218}]: dispatch 2024-02-09T17:46:09.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:09 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 0.375345637646218}]: dispatch 2024-02-09T17:46:10.097 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 18 16 44049 44033 152.879 109.438 0.00955339 0.00653645 2024-02-09T17:46:10.311 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:10 smithi162 ceph-mon[44304]: pgmap v1464: 47 pgs: 47 active+clean; 2.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 148 MiB/s wr, 2.36k op/s 2024-02-09T17:46:10.311 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:10 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 0.375345637646218}]': finished 2024-02-09T17:46:10.311 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:10 smithi162 ceph-mon[44304]: osdmap e583: 12 total, 12 up, 12 in 2024-02-09T17:46:10.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:10 smithi179 ceph-mon[43975]: pgmap v1464: 47 pgs: 47 active+clean; 2.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 148 MiB/s wr, 2.36k op/s 2024-02-09T17:46:10.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:10 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 0.375345637646218}]': finished 2024-02-09T17:46:10.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:10 smithi179 ceph-mon[43975]: osdmap e583: 12 total, 12 up, 12 in 2024-02-09T17:46:10.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:10 smithi099 ceph-mon[39420]: pgmap v1464: 47 pgs: 47 active+clean; 2.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 148 MiB/s wr, 2.36k op/s 2024-02-09T17:46:10.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:10 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 0.375345637646218}]': finished 2024-02-09T17:46:10.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:10 smithi099 ceph-mon[39420]: osdmap e583: 12 total, 12 up, 12 in 2024-02-09T17:46:10.728 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:46:10.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:10.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:46:10.836 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:46:10.873 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:10.892 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:46:10.892 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-09T17:46:10.892 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-02-09T17:46:10.892 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-02-09T17:46:10.892 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:46:10.897 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:46:10.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:10.902 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:46:11.003 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:46:11.071 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:11.098 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 19 16 46000 45984 151.24 121.938 0.0219074 0.006607 2024-02-09T17:46:11.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:11 smithi179 ceph-mon[43975]: osdmap e584: 12 total, 12 up, 12 in 2024-02-09T17:46:11.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:11 smithi162 ceph-mon[44304]: osdmap e584: 12 total, 12 up, 12 in 2024-02-09T17:46:11.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:11 smithi099 ceph-mon[39420]: osdmap e584: 12 total, 12 up, 12 in 2024-02-09T17:46:11.634 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:46:11.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:11.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:46:11.740 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:46:11.793 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:11.974 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:46:11.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:11.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:46:12.080 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:46:12.098 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:2024-02-09T17:46:12.098198+0000 min lat: 0.00111008 max lat: 0.26167 avg lat: 0.00666669 2024-02-09T17:46:12.098 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-09T17:46:12.098 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 20 16 47951 47935 149.775 121.938 0.00272553 0.00666669 2024-02-09T17:46:12.123 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:12.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:12 smithi099 ceph-mon[39420]: pgmap v1467: 47 pgs: 1 remapped+peering, 46 active+clean; 2.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 134 MiB/s wr, 2.15k op/s 2024-02-09T17:46:12.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:12 smithi099 ceph-mon[39420]: osdmap e585: 12 total, 12 up, 12 in 2024-02-09T17:46:12.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:12 smithi099 ceph-mon[39420]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-09T17:46:12.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:12 smithi099 ceph-mon[39420]: osdmap e586: 12 total, 12 up, 12 in 2024-02-09T17:46:12.387 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:46:12.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:12.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:46:12.393 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:12 smithi162 ceph-mon[44304]: pgmap v1467: 47 pgs: 1 remapped+peering, 46 active+clean; 2.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 134 MiB/s wr, 2.15k op/s 2024-02-09T17:46:12.393 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:12 smithi162 ceph-mon[44304]: osdmap e585: 12 total, 12 up, 12 in 2024-02-09T17:46:12.393 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:12 smithi162 ceph-mon[44304]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-09T17:46:12.393 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:12 smithi162 ceph-mon[44304]: osdmap e586: 12 total, 12 up, 12 in 2024-02-09T17:46:12.415 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:46:12.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:12 smithi179 ceph-mon[43975]: pgmap v1467: 47 pgs: 1 remapped+peering, 46 active+clean; 2.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 134 MiB/s wr, 2.15k op/s 2024-02-09T17:46:12.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:12 smithi179 ceph-mon[43975]: osdmap e585: 12 total, 12 up, 12 in 2024-02-09T17:46:12.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:12 smithi179 ceph-mon[43975]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-09T17:46:12.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:12 smithi179 ceph-mon[43975]: osdmap e586: 12 total, 12 up, 12 in 2024-02-09T17:46:12.449 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:46:12.489 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:46:12.494 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:46:12.527 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:46:12.529 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:12.553 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:46:12.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:12.559 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:46:12.561 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:46:12.595 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:46:12.623 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:46:12.642 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:46:12.660 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:46:12.663 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:46:12.686 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:46:12.690 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:12.707 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:46:12.709 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:46:12.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:12.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:46:12.728 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:46:12.729 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:46:12.729 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:46:12.729 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:46:12.729 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:46:12.730 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:46:12.730 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:46:12.730 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:46:12.730 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:46:12.730 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:46:12.730 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:46:12.730 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:46:12.730 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:46:12.814 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:46:12.889 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:12.919 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:46:12.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:12.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:46:12.927 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-09T17:46:13.028 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:46:13.059 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:13.085 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:46:13.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:13.089 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:46:13.098 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 21 16 49528 49512 147.336 98.5625 0.00879883 0.00678234 2024-02-09T17:46:13.104 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset noscrub 2024-02-09T17:46:13.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:13 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1880161836' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:46:13.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:13 smithi162 ceph-mon[44304]: osdmap e587: 12 total, 12 up, 12 in 2024-02-09T17:46:13.190 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:46:13.263 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:13.344 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:46:13.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:13 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1880161836' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:46:13.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:13 smithi179 ceph-mon[43975]: osdmap e587: 12 total, 12 up, 12 in 2024-02-09T17:46:13.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:13 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1880161836' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:46:13.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:13 smithi099 ceph-mon[39420]: osdmap e587: 12 total, 12 up, 12 in 2024-02-09T17:46:14.099 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 22 16 51243 51227 145.511 107.188 0.0115476 0.00686838 2024-02-09T17:46:14.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:14 smithi179 ceph-mon[43975]: pgmap v1471: 47 pgs: 2 peering, 1 remapped+peering, 44 active+clean; 2.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 151 MiB/s wr, 2.41k op/s; 18 KiB/s, 0 objects/s recovering 2024-02-09T17:46:14.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:14 smithi162 ceph-mon[44304]: pgmap v1471: 47 pgs: 2 peering, 1 remapped+peering, 44 active+clean; 2.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 151 MiB/s wr, 2.41k op/s; 18 KiB/s, 0 objects/s recovering 2024-02-09T17:46:14.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:14 smithi099 ceph-mon[39420]: pgmap v1471: 47 pgs: 2 peering, 1 remapped+peering, 44 active+clean; 2.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 151 MiB/s wr, 2.41k op/s; 18 KiB/s, 0 objects/s recovering 2024-02-09T17:46:14.909 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:46:14.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:14.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:46:14.938 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:46:15.023 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:46:15.062 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:15.087 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:46:15.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:15.090 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:46:15.098 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 23 16 52817 52801 143.461 98.375 0.0117469 0.0069656 2024-02-09T17:46:15.191 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:46:15.230 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:15.673 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] 2024-02-09T17:46:15.674 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:46:15.674 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-02-09T17:46:15.674 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-09T17:46:15.674 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-09T17:46:15.674 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-02-09T17:46:15.701 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:46:15.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:15.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:46:15.816 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:46:15.851 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:16.058 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:46:16.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:16.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:46:16.099 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 24 16 54434 54418 141.695 101.062 0.00577714 0.00705313 2024-02-09T17:46:16.164 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:46:16.218 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:16.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:16 smithi179 ceph-mon[43975]: pgmap v1472: 47 pgs: 2 peering, 1 remapped+peering, 44 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 121 MiB/s wr, 1.94k op/s; 14 KiB/s, 0 objects/s recovering 2024-02-09T17:46:16.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:16 smithi162 ceph-mon[44304]: pgmap v1472: 47 pgs: 2 peering, 1 remapped+peering, 44 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 121 MiB/s wr, 1.94k op/s; 14 KiB/s, 0 objects/s recovering 2024-02-09T17:46:16.489 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:46:16.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:16 smithi099 ceph-mon[39420]: pgmap v1472: 47 pgs: 2 peering, 1 remapped+peering, 44 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 121 MiB/s wr, 1.94k op/s; 14 KiB/s, 0 objects/s recovering 2024-02-09T17:46:16.581 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:46:16.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:16.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:46:16.693 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:46:16.740 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:16.741 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:46:16.741 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:46:16.792 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:46:16.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:16.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:46:16.897 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:46:16.924 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:46:16.943 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:17.099 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 25 16 56096 56080 140.182 103.875 0.00345726 0.00712987 2024-02-09T17:46:17.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:17 smithi179 ceph-mon[43975]: pgmap v1473: 47 pgs: 1 peering, 46 active+clean; 3.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 113 MiB/s wr, 1.82k op/s; 12 KiB/s, 0 objects/s recovering 2024-02-09T17:46:17.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:17 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:46:17.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:17 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:46:17.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:17 smithi162 ceph-mon[44304]: pgmap v1473: 47 pgs: 1 peering, 46 active+clean; 3.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 113 MiB/s wr, 1.82k op/s; 12 KiB/s, 0 objects/s recovering 2024-02-09T17:46:17.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:17 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:46:17.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:17 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:46:17.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:17 smithi099 ceph-mon[39420]: pgmap v1473: 47 pgs: 1 peering, 46 active+clean; 3.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 113 MiB/s wr, 1.82k op/s; 12 KiB/s, 0 objects/s recovering 2024-02-09T17:46:17.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:17 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:46:17.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:17 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:46:18.099 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 26 16 57512 57496 138.194 88.5 0.00806743 0.00723233 2024-02-09T17:46:18.454 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:46:18.614 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:46:18.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:18.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:46:18.646 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is unset 2024-02-09T17:46:18.721 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:46:18.762 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:18.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:18 smithi179 ceph-mon[43975]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-09T17:46:18.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:18 smithi179 ceph-mon[43975]: osdmap e588: 12 total, 12 up, 12 in 2024-02-09T17:46:18.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:18 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/377540274' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:46:18.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:18 smithi162 ceph-mon[44304]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-09T17:46:18.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:18 smithi162 ceph-mon[44304]: osdmap e588: 12 total, 12 up, 12 in 2024-02-09T17:46:18.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:18 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/377540274' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:46:18.932 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:18 smithi099 ceph-mon[39420]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-09T17:46:18.932 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:18 smithi099 ceph-mon[39420]: osdmap e588: 12 total, 12 up, 12 in 2024-02-09T17:46:18.932 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:18 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/377540274' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:46:19.099 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 27 16 59226 59210 137.043 107.125 0.00645838 0.00729209 2024-02-09T17:46:19.333 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-09T17:46:19.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:19 smithi179 ceph-mon[43975]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:46:19.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:19 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/377540274' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:46:19.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:19 smithi179 ceph-mon[43975]: osdmap e589: 12 total, 12 up, 12 in 2024-02-09T17:46:19.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:19 smithi179 ceph-mon[43975]: pgmap v1476: 47 pgs: 47 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.43k op/s 2024-02-09T17:46:19.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:19 smithi162 ceph-mon[44304]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:46:19.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:19 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/377540274' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:46:19.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:19 smithi162 ceph-mon[44304]: osdmap e589: 12 total, 12 up, 12 in 2024-02-09T17:46:19.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:19 smithi162 ceph-mon[44304]: pgmap v1476: 47 pgs: 47 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.43k op/s 2024-02-09T17:46:19.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:19 smithi099 ceph-mon[39420]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:46:19.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:19 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/377540274' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:46:19.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:19 smithi099 ceph-mon[39420]: osdmap e589: 12 total, 12 up, 12 in 2024-02-09T17:46:19.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:19 smithi099 ceph-mon[39420]: pgmap v1476: 47 pgs: 47 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.43k op/s 2024-02-09T17:46:20.099 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 28 16 60847 60831 135.766 101.312 0.00801773 0.00736149 2024-02-09T17:46:20.426 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:46:20.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:20.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:46:20.534 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:46:20.586 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:20.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:20 smithi099 ceph-mon[39420]: 1.1c scrub starts 2024-02-09T17:46:20.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:20 smithi099 ceph-mon[39420]: 1.1c scrub ok 2024-02-09T17:46:20.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:20 smithi179 ceph-mon[43975]: 1.1c scrub starts 2024-02-09T17:46:20.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:20 smithi179 ceph-mon[43975]: 1.1c scrub ok 2024-02-09T17:46:20.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:20 smithi162 ceph-mon[44304]: 1.1c scrub starts 2024-02-09T17:46:20.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:20 smithi162 ceph-mon[44304]: 1.1c scrub ok 2024-02-09T17:46:21.099 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 29 16 62454 62438 134.548 100.438 0.0127151 0.00742841 2024-02-09T17:46:21.358 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:46:21.439 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:46:21.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:21.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:46:21.585 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:46:21.666 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:21.703 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:46:21.704 INFO:teuthology.orchestra.run.smithi162.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-02-09T17:46:21.704 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:46:21.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:21 smithi162 ceph-mon[44304]: 1.2 scrub starts 2024-02-09T17:46:21.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:21 smithi162 ceph-mon[44304]: 1.2 scrub ok 2024-02-09T17:46:21.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:21 smithi162 ceph-mon[44304]: 1.8 scrub starts 2024-02-09T17:46:21.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:21 smithi162 ceph-mon[44304]: 1.8 scrub ok 2024-02-09T17:46:21.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:21 smithi162 ceph-mon[44304]: 1.17 scrub starts 2024-02-09T17:46:21.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:21 smithi162 ceph-mon[44304]: pgmap v1477: 47 pgs: 47 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.44k op/s 2024-02-09T17:46:21.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:21 smithi179 ceph-mon[43975]: 1.2 scrub starts 2024-02-09T17:46:21.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:21 smithi179 ceph-mon[43975]: 1.2 scrub ok 2024-02-09T17:46:21.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:21 smithi179 ceph-mon[43975]: 1.8 scrub starts 2024-02-09T17:46:21.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:21 smithi179 ceph-mon[43975]: 1.8 scrub ok 2024-02-09T17:46:21.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:21 smithi179 ceph-mon[43975]: 1.17 scrub starts 2024-02-09T17:46:21.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:21 smithi179 ceph-mon[43975]: pgmap v1477: 47 pgs: 47 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.44k op/s 2024-02-09T17:46:21.923 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:21 smithi099 ceph-mon[39420]: 1.2 scrub starts 2024-02-09T17:46:21.923 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:21 smithi099 ceph-mon[39420]: 1.2 scrub ok 2024-02-09T17:46:21.923 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:21 smithi099 ceph-mon[39420]: 1.8 scrub starts 2024-02-09T17:46:21.923 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:21 smithi099 ceph-mon[39420]: 1.8 scrub ok 2024-02-09T17:46:21.923 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:21 smithi099 ceph-mon[39420]: 1.17 scrub starts 2024-02-09T17:46:21.923 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:21 smithi099 ceph-mon[39420]: pgmap v1477: 47 pgs: 47 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.44k op/s 2024-02-09T17:46:22.099 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 30 16 64043 64027 133.373 99.3125 0.00553161 0.00749301 2024-02-09T17:46:22.443 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:46:22.476 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:46:22.512 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:46:22.545 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:46:22.580 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:46:22.617 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:46:22.660 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:46:22.697 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:46:22.728 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:46:22.761 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:46:22.790 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:46:22.823 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:46:22.829 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:46:22.829 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:46:22.829 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:46:22.829 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:46:22.829 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:46:22.829 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:46:22.829 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:46:22.829 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:46:22.829 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:46:22.829 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:46:22.829 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:46:22.829 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:46:22.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:22 smithi179 ceph-mon[43975]: 1.17 scrub ok 2024-02-09T17:46:22.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:22 smithi179 ceph-mon[43975]: 1.10 scrub starts 2024-02-09T17:46:22.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:22 smithi179 ceph-mon[43975]: 1.10 scrub ok 2024-02-09T17:46:22.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:22 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1317292368' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:46:22.931 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:22 smithi162 ceph-mon[44304]: 1.17 scrub ok 2024-02-09T17:46:22.931 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:22 smithi162 ceph-mon[44304]: 1.10 scrub starts 2024-02-09T17:46:22.931 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:22 smithi162 ceph-mon[44304]: 1.10 scrub ok 2024-02-09T17:46:22.931 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:22 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1317292368' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:46:22.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:22 smithi099 ceph-mon[39420]: 1.17 scrub ok 2024-02-09T17:46:22.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:22 smithi099 ceph-mon[39420]: 1.10 scrub starts 2024-02-09T17:46:22.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:22 smithi099 ceph-mon[39420]: 1.10 scrub ok 2024-02-09T17:46:22.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:22 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1317292368' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:46:23.099 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 31 16 65596 65580 132.202 97.0625 0.0291856 0.00755998 2024-02-09T17:46:23.593 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-09T17:46:23.762 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:23 smithi162 ceph-mon[44304]: 1.d scrub starts 2024-02-09T17:46:23.762 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:23 smithi162 ceph-mon[44304]: 1.d scrub ok 2024-02-09T17:46:23.763 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:23 smithi162 ceph-mon[44304]: 1.11 scrub starts 2024-02-09T17:46:23.763 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:23 smithi162 ceph-mon[44304]: 1.11 scrub ok 2024-02-09T17:46:23.763 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:23 smithi162 ceph-mon[44304]: pgmap v1478: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 107 MiB/s wr, 1.72k op/s 2024-02-09T17:46:23.763 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:23 smithi162 ceph-mon[44304]: 1.e scrub starts 2024-02-09T17:46:23.763 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:23 smithi162 ceph-mon[44304]: 1.e scrub ok 2024-02-09T17:46:23.843 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:46:23.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:23.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:46:23.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:23 smithi179 ceph-mon[43975]: 1.d scrub starts 2024-02-09T17:46:23.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:23 smithi179 ceph-mon[43975]: 1.d scrub ok 2024-02-09T17:46:23.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:23 smithi179 ceph-mon[43975]: 1.11 scrub starts 2024-02-09T17:46:23.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:23 smithi179 ceph-mon[43975]: 1.11 scrub ok 2024-02-09T17:46:23.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:23 smithi179 ceph-mon[43975]: pgmap v1478: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 107 MiB/s wr, 1.72k op/s 2024-02-09T17:46:23.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:23 smithi179 ceph-mon[43975]: 1.e scrub starts 2024-02-09T17:46:23.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:23 smithi179 ceph-mon[43975]: 1.e scrub ok 2024-02-09T17:46:23.958 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:46:23.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:23 smithi099 ceph-mon[39420]: 1.d scrub starts 2024-02-09T17:46:23.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:23 smithi099 ceph-mon[39420]: 1.d scrub ok 2024-02-09T17:46:23.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:23 smithi099 ceph-mon[39420]: 1.11 scrub starts 2024-02-09T17:46:23.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:23 smithi099 ceph-mon[39420]: 1.11 scrub ok 2024-02-09T17:46:23.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:23 smithi099 ceph-mon[39420]: pgmap v1478: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 107 MiB/s wr, 1.72k op/s 2024-02-09T17:46:23.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:23 smithi099 ceph-mon[39420]: 1.e scrub starts 2024-02-09T17:46:23.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:23 smithi099 ceph-mon[39420]: 1.e scrub ok 2024-02-09T17:46:24.002 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:24.037 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:46:24.042 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:46:24.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:24.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:46:24.099 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 32 16 67062 67046 130.933 91.625 0.0109197 0.0076332 2024-02-09T17:46:24.149 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:46:24.195 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:24.352 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:46:24.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:24.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:46:24.461 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:46:24.516 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:24.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:24 smithi162 ceph-mon[44304]: 1.1 scrub starts 2024-02-09T17:46:24.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:24 smithi162 ceph-mon[44304]: 1.1 scrub ok 2024-02-09T17:46:24.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:24 smithi162 ceph-mon[44304]: 1.6 scrub starts 2024-02-09T17:46:24.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:24 smithi162 ceph-mon[44304]: 1.6 scrub ok 2024-02-09T17:46:24.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:24 smithi162 ceph-mon[44304]: 1.16 scrub starts 2024-02-09T17:46:24.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:24 smithi162 ceph-mon[44304]: 1.16 scrub ok 2024-02-09T17:46:24.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:24 smithi099 ceph-mon[39420]: 1.1 scrub starts 2024-02-09T17:46:24.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:24 smithi099 ceph-mon[39420]: 1.1 scrub ok 2024-02-09T17:46:24.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:24 smithi099 ceph-mon[39420]: 1.6 scrub starts 2024-02-09T17:46:24.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:24 smithi099 ceph-mon[39420]: 1.6 scrub ok 2024-02-09T17:46:24.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:24 smithi099 ceph-mon[39420]: 1.16 scrub starts 2024-02-09T17:46:24.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:24 smithi099 ceph-mon[39420]: 1.16 scrub ok 2024-02-09T17:46:25.027 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:46:25.027 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-02-09T17:46:25.027 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:25.027 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25837.0:68438 6.3 6:cabee8f4:::benchmark_data_smithi179_126206_object68437:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e590)", 2024-02-09T17:46:25.027 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:46:25.008974+0000", 2024-02-09T17:46:25.027 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0051729020000000001, 2024-02-09T17:46:25.027 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0052099529999999998, 2024-02-09T17:46:25.028 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:46:25.028 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-02-09T17:46:25.028 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:46:25.028 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25837", 2024-02-09T17:46:25.028 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:46:25.028 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 68438 2024-02-09T17:46:25.028 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:25.028 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:46:25.028 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:25.028 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:46:25.028 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:25.008974+0000", 2024-02-09T17:46:25.028 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:46:25.028 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:25.029 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:25.029 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:46:25.029 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:25.008974+0000", 2024-02-09T17:46:25.029 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:46:25.029 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:25.029 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:25.029 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:46:25.029 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:25.008975+0000", 2024-02-09T17:46:25.029 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1289999999999999e-06 2024-02-09T17:46:25.029 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:25.029 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:25.029 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:46:25.029 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:25.009047+0000", 2024-02-09T17:46:25.030 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2403000000000005e-05 2024-02-09T17:46:25.030 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:25.030 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:25.030 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:46:25.030 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:25.009059+0000", 2024-02-09T17:46:25.030 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1574e-05 2024-02-09T17:46:25.030 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:25.030 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:25.030 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:46:25.030 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:25.009067+0000", 2024-02-09T17:46:25.030 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7919999999999993e-06 2024-02-09T17:46:25.030 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:25.030 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:25.031 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:46:25.031 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:25.009121+0000", 2024-02-09T17:46:25.031 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4490000000000002e-05 2024-02-09T17:46:25.031 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:25.031 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:25.031 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:46:25.031 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:25.009271+0000", 2024-02-09T17:46:25.031 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015009299999999999 2024-02-09T17:46:25.031 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:25.031 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:25.031 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-02-09T17:46:25.031 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:25.009410+0000", 2024-02-09T17:46:25.031 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013901000000000001 2024-02-09T17:46:25.032 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:46:25.032 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:46:25.032 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:46:25.032 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:25.032 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:25.032 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25837.0:68440 6.5 6:aed2a93e:::benchmark_data_smithi179_126206_object68439:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e590)", 2024-02-09T17:46:25.032 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:46:25.009536+0000", 2024-02-09T17:46:25.032 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0046110860000000004, 2024-02-09T17:46:25.032 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0047410940000000004, 2024-02-09T17:46:25.032 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:46:25.032 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-02-09T17:46:25.032 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:46:25.032 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25837", 2024-02-09T17:46:25.032 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:46:25.033 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 68440 2024-02-09T17:46:25.033 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:25.033 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:46:25.033 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:25.033 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:46:25.033 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:25.009536+0000", 2024-02-09T17:46:25.033 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:46:25.033 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:25.033 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:25.033 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:46:25.033 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:25.009536+0000", 2024-02-09T17:46:25.033 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:46:25.033 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:25.033 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:25.034 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:46:25.034 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:25.009537+0000", 2024-02-09T17:46:25.034 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9000000000000005e-07 2024-02-09T17:46:25.034 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:25.034 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:25.034 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:46:25.034 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:25.009564+0000", 2024-02-09T17:46:25.034 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7973000000000001e-05 2024-02-09T17:46:25.034 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:25.034 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:25.034 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:46:25.034 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:25.009576+0000", 2024-02-09T17:46:25.034 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1677e-05 2024-02-09T17:46:25.034 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:25.035 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:25.035 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:46:25.035 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:25.009584+0000", 2024-02-09T17:46:25.035 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4560000000000003e-06 2024-02-09T17:46:25.035 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:25.035 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:25.035 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:46:25.035 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:25.009642+0000", 2024-02-09T17:46:25.035 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8275e-05 2024-02-09T17:46:25.035 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:25.035 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:25.035 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:46:25.035 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:25.009777+0000", 2024-02-09T17:46:25.035 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013520900000000001 2024-02-09T17:46:25.035 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:25.036 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:25.036 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-02-09T17:46:25.036 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:25.009917+0000", 2024-02-09T17:46:25.036 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014022899999999999 2024-02-09T17:46:25.036 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:25.036 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:25.036 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:46:25.036 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:25.012586+0000", 2024-02-09T17:46:25.036 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002668948 2024-02-09T17:46:25.036 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:46:25.036 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:46:25.036 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:46:25.036 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:25.036 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:25.037 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25837.0:68445 6.b e590/576)", 2024-02-09T17:46:25.037 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:46:25.012379+0000", 2024-02-09T17:46:25.037 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.001767603, 2024-02-09T17:46:25.037 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0019736269999999999, 2024-02-09T17:46:25.037 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:46:25.037 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "queued for pg", 2024-02-09T17:46:25.037 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:46:25.037 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:25.037 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:46:25.037 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:25.012379+0000", 2024-02-09T17:46:25.037 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:46:25.037 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:25.037 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:25.037 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:46:25.038 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:25.012379+0000", 2024-02-09T17:46:25.038 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:46:25.038 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:25.038 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:25.038 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:46:25.038 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:25.012380+0000", 2024-02-09T17:46:25.038 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.114e-06 2024-02-09T17:46:25.038 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:25.038 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:25.038 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:46:25.038 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:25.012413+0000", 2024-02-09T17:46:25.038 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2616000000000001e-05 2024-02-09T17:46:25.038 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:25.038 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:25.038 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:46:25.039 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:25.012423+0000", 2024-02-09T17:46:25.039 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0587e-05 2024-02-09T17:46:25.039 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:25.039 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:25.039 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:46:25.039 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:25.012443+0000", 2024-02-09T17:46:25.039 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9381999999999998e-05 2024-02-09T17:46:25.039 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:46:25.039 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:46:25.039 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:46:25.039 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:25.039 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:25.039 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25837.0:68446 6.6 e590/576 6:6950f29b:::benchmark_data_smithi179_126206_object68445:head v 590'4259, mlcod=590'4257)", 2024-02-09T17:46:25.039 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:46:25.012473+0000", 2024-02-09T17:46:25.039 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0016735680000000001, 2024-02-09T17:46:25.040 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001932153, 2024-02-09T17:46:25.040 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:46:25.040 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-09T17:46:25.040 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:46:25.040 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:25.040 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:46:25.040 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:25.012473+0000", 2024-02-09T17:46:25.040 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:46:25.040 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:25.040 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:25.040 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:46:25.040 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:25.012473+0000", 2024-02-09T17:46:25.040 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:46:25.040 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:25.041 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:25.041 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:46:25.041 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:25.012473+0000", 2024-02-09T17:46:25.041 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9299999999999999e-07 2024-02-09T17:46:25.041 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:25.041 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:25.041 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:46:25.041 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:25.012501+0000", 2024-02-09T17:46:25.041 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7185000000000001e-05 2024-02-09T17:46:25.041 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:25.041 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:25.041 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:46:25.041 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:25.012506+0000", 2024-02-09T17:46:25.041 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7300000000000002e-06 2024-02-09T17:46:25.042 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:25.042 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:25.042 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:46:25.042 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:25.012510+0000", 2024-02-09T17:46:25.042 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7129999999999999e-06 2024-02-09T17:46:25.042 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:25.042 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:25.042 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:46:25.042 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:25.012565+0000", 2024-02-09T17:46:25.042 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4824999999999998e-05 2024-02-09T17:46:25.042 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:25.042 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:25.042 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:46:25.042 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:25.012582+0000", 2024-02-09T17:46:25.042 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7606000000000001e-05 2024-02-09T17:46:25.043 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:46:25.043 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:46:25.043 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:46:25.043 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:25.043 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:25.043 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.25837.0:68440 6.5 e590/576)", 2024-02-09T17:46:25.043 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:46:25.014490+0000", 2024-02-09T17:46:25.043 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 4294967295.9996567, 2024-02-09T17:46:25.043 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4894000000000004e-05, 2024-02-09T17:46:25.043 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:46:25.043 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "queued for pg", 2024-02-09T17:46:25.043 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:46:25.043 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:25.043 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:46:25.043 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:25.014490+0000", 2024-02-09T17:46:25.043 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:46:25.044 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:25.044 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:25.044 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:46:25.044 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:25.014490+0000", 2024-02-09T17:46:25.044 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:46:25.044 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:25.044 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:25.044 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:46:25.044 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:25.014491+0000", 2024-02-09T17:46:25.044 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4000000000000001e-07 2024-02-09T17:46:25.044 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:25.044 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:25.044 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:46:25.044 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:25.014495+0000", 2024-02-09T17:46:25.045 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2409999999999997e-06 2024-02-09T17:46:25.045 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:25.045 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:25.045 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:46:25.045 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:25.014506+0000", 2024-02-09T17:46:25.045 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1041e-05 2024-02-09T17:46:25.045 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:46:25.045 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:46:25.045 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:46:25.045 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:25.014513+0000", 2024-02-09T17:46:25.045 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8889999999999996e-06 2024-02-09T17:46:25.045 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:46:25.045 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:46:25.045 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:46:25.045 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:46:25.046 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-09T17:46:25.046 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 5 2024-02-09T17:46:25.046 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:46:25.099 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 33 16 68578 68562 129.837 94.75 0.00670801 0.00769812 2024-02-09T17:46:25.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:24 smithi179 ceph-mon[43975]: 1.1 scrub starts 2024-02-09T17:46:25.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:24 smithi179 ceph-mon[43975]: 1.1 scrub ok 2024-02-09T17:46:25.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:24 smithi179 ceph-mon[43975]: 1.6 scrub starts 2024-02-09T17:46:25.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:24 smithi179 ceph-mon[43975]: 1.6 scrub ok 2024-02-09T17:46:25.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:24 smithi179 ceph-mon[43975]: 1.16 scrub starts 2024-02-09T17:46:25.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:24 smithi179 ceph-mon[43975]: 1.16 scrub ok 2024-02-09T17:46:25.293 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:46:25.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:25.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:46:25.400 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:46:25.449 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:25.635 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:46:25.799 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:46:25.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:25.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:46:25.838 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-09T17:46:25.906 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:46:25.947 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:25 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:46:25.947 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:25 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:46:25.948 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:25 smithi099 ceph-mon[39420]: osdmap e590: 12 total, 12 up, 12 in 2024-02-09T17:46:25.948 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:25 smithi099 ceph-mon[39420]: pgmap v1480: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.44k op/s 2024-02-09T17:46:25.953 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:26.099 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 34 16 69961 69945 128.56 86.4375 0.0113053 0.00777392 2024-02-09T17:46:26.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:25 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:46:26.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:25 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:46:26.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:25 smithi179 ceph-mon[43975]: osdmap e590: 12 total, 12 up, 12 in 2024-02-09T17:46:26.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:25 smithi179 ceph-mon[43975]: pgmap v1480: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.44k op/s 2024-02-09T17:46:26.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:25 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:46:26.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:25 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:46:26.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:25 smithi162 ceph-mon[44304]: osdmap e590: 12 total, 12 up, 12 in 2024-02-09T17:46:26.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:25 smithi162 ceph-mon[44304]: pgmap v1480: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.44k op/s 2024-02-09T17:46:26.281 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:46:26.407 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:46:26.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:26.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:46:26.515 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:46:26.599 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:26.628 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:46:26.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:26.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:46:26.715 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:26 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:46:26.715 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:26 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:46:26.715 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:26 smithi099 ceph-mon[39420]: osdmap e591: 12 total, 12 up, 12 in 2024-02-09T17:46:26.735 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:46:26.773 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:26.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:26 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:46:26.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:26 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:46:26.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:26 smithi162 ceph-mon[44304]: osdmap e591: 12 total, 12 up, 12 in 2024-02-09T17:46:26.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:26 smithi162 ceph-mon[44304]: osdmap e592: 12 total, 12 up, 12 in 2024-02-09T17:46:26.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:26 smithi162 ceph-mon[44304]: 1.a scrub starts 2024-02-09T17:46:26.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:26 smithi162 ceph-mon[44304]: 1.a scrub ok 2024-02-09T17:46:26.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:26 smithi099 ceph-mon[39420]: osdmap e592: 12 total, 12 up, 12 in 2024-02-09T17:46:26.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:26 smithi099 ceph-mon[39420]: 1.a scrub starts 2024-02-09T17:46:26.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:26 smithi099 ceph-mon[39420]: 1.a scrub ok 2024-02-09T17:46:27.101 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 35 16 71169 71153 127.044 75.5 0.00325569 0.00784719 2024-02-09T17:46:27.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:26 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:46:27.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:26 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:46:27.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:26 smithi179 ceph-mon[43975]: osdmap e591: 12 total, 12 up, 12 in 2024-02-09T17:46:27.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:26 smithi179 ceph-mon[43975]: osdmap e592: 12 total, 12 up, 12 in 2024-02-09T17:46:27.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:26 smithi179 ceph-mon[43975]: 1.a scrub starts 2024-02-09T17:46:27.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:26 smithi179 ceph-mon[43975]: 1.a scrub ok 2024-02-09T17:46:27.435 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:46:27.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:27.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:46:27.484 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:46:27.543 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:46:27.582 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:27.637 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:46:27.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:27.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:46:27.727 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:27 smithi099 ceph-mon[39420]: 1.15 scrub starts 2024-02-09T17:46:27.743 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:46:27.793 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:27.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:27 smithi162 ceph-mon[44304]: 1.15 scrub starts 2024-02-09T17:46:27.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:27 smithi162 ceph-mon[44304]: 1.15 scrub ok 2024-02-09T17:46:27.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:27 smithi162 ceph-mon[44304]: pgmap v1483: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 96 MiB/s wr, 1.54k op/s 2024-02-09T17:46:27.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:27 smithi099 ceph-mon[39420]: 1.15 scrub ok 2024-02-09T17:46:27.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:27 smithi099 ceph-mon[39420]: pgmap v1483: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 96 MiB/s wr, 1.54k op/s 2024-02-09T17:46:28.060 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:46:28.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:28.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:46:28.100 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 36 16 72622 72606 126.038 90.8125 0.00461833 0.00792981 2024-02-09T17:46:28.165 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:46:28.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:27 smithi179 ceph-mon[43975]: 1.15 scrub starts 2024-02-09T17:46:28.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:27 smithi179 ceph-mon[43975]: 1.15 scrub ok 2024-02-09T17:46:28.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:27 smithi179 ceph-mon[43975]: pgmap v1483: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 96 MiB/s wr, 1.54k op/s 2024-02-09T17:46:28.211 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:28.245 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:46:28.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:28.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:46:28.350 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:46:28.379 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:28.415 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:46:28.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:28.419 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:46:28.520 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:46:28.549 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:28.562 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:46:28.563 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-09T17:46:28.563 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-02-09T17:46:28.563 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-02-09T17:46:28.563 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:46:28.576 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:46:28.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:28.585 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:46:28.686 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:46:28.733 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:28.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:28 smithi162 ceph-mon[44304]: 1.5 scrub starts 2024-02-09T17:46:28.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:28 smithi162 ceph-mon[44304]: 1.5 scrub ok 2024-02-09T17:46:28.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:28 smithi099 ceph-mon[39420]: 1.5 scrub starts 2024-02-09T17:46:28.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:28 smithi099 ceph-mon[39420]: 1.5 scrub ok 2024-02-09T17:46:29.100 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 37 16 74089 74073 125.109 91.6875 0.017091 0.00798912 2024-02-09T17:46:29.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:28 smithi179 ceph-mon[43975]: 1.5 scrub starts 2024-02-09T17:46:29.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:28 smithi179 ceph-mon[43975]: 1.5 scrub ok 2024-02-09T17:46:29.197 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:46:29.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:29.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:46:29.303 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:46:29.337 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:29.353 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] 2024-02-09T17:46:29.354 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:46:29.354 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-02-09T17:46:29.354 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-09T17:46:29.354 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-09T17:46:29.354 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.3 config set filestore_inject_stall 3 2024-02-09T17:46:29.364 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:46:29.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:29.369 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:46:29.470 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:46:29.532 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:30.100 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 38 16 75369 75353 123.922 80 0.0110837 0.00806581 2024-02-09T17:46:30.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:29 smithi179 ceph-mon[43975]: pgmap v1484: 47 pgs: 47 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-02-09T17:46:30.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:29 smithi162 ceph-mon[44304]: pgmap v1484: 47 pgs: 47 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-02-09T17:46:30.226 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:46:30.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:30.234 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:29 smithi099 ceph-mon[39420]: pgmap v1484: 47 pgs: 47 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-02-09T17:46:30.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:46:30.335 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:46:30.381 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:30.763 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is unset 2024-02-09T17:46:30.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:30 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/938448155' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:46:30.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:30 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:46:30.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:30 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:46:30.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:30 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:46:31.100 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 39 16 76582 76566 122.688 75.8125 0.00999764 0.00814679 2024-02-09T17:46:31.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:30 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/938448155' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:46:31.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:30 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:46:31.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:30 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:46:31.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:30 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:46:31.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:30 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/938448155' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:46:31.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:30 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:46:31.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:30 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:46:31.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:30 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:46:31.248 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:46:31.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:31.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:46:31.281 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-09T17:46:31.357 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:46:31.388 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:31.409 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:46:31.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:31.415 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:46:31.432 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:46:31.432 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:46:31.432 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:46:31.432 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:46:31.432 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:46:31.432 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:46:31.432 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:46:31.442 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:46:31.470 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:46:31.498 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:46:31.516 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:46:31.525 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:46:31.545 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:31.561 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:46:31.561 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:46:31.561 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:46:31.561 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:46:31.561 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:46:31.562 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:46:31.562 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:46:31.562 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:46:31.562 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:46:31.562 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:46:31.562 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:46:31.562 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:46:31.562 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:46:31.567 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:46:31.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:31.573 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:46:31.674 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:46:31.717 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:31.755 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:46:31.912 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:31 smithi099 ceph-mon[39420]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:46:31.912 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:31 smithi099 ceph-mon[39420]: Cluster is now healthy 2024-02-09T17:46:31.912 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:31 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:46:31.912 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:31 smithi099 ceph-mon[39420]: osdmap e593: 12 total, 12 up, 12 in 2024-02-09T17:46:31.912 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:31 smithi099 ceph-mon[39420]: pgmap v1486: 47 pgs: 47 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-02-09T17:46:31.912 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:31 smithi099 ceph-mon[39420]: 1.1a scrub starts 2024-02-09T17:46:31.912 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:31 smithi099 ceph-mon[39420]: 1.1a scrub ok 2024-02-09T17:46:31.912 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:31 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/207033130' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:46:31.912 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:31 smithi099 ceph-mon[39420]: osdmap e594: 12 total, 12 up, 12 in 2024-02-09T17:46:32.072 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:31 smithi162 ceph-mon[44304]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:46:32.073 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:31 smithi162 ceph-mon[44304]: Cluster is now healthy 2024-02-09T17:46:32.073 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:31 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:46:32.073 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:31 smithi162 ceph-mon[44304]: osdmap e593: 12 total, 12 up, 12 in 2024-02-09T17:46:32.073 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:31 smithi162 ceph-mon[44304]: pgmap v1486: 47 pgs: 47 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-02-09T17:46:32.073 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:31 smithi162 ceph-mon[44304]: 1.1a scrub starts 2024-02-09T17:46:32.073 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:31 smithi162 ceph-mon[44304]: 1.1a scrub ok 2024-02-09T17:46:32.073 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:31 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/207033130' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:46:32.073 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:31 smithi162 ceph-mon[44304]: osdmap e594: 12 total, 12 up, 12 in 2024-02-09T17:46:32.075 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:46:32.076 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:46:32.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:32.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:46:32.100 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:2024-02-09T17:46:32.099765+0000 min lat: 0.00111008 max lat: 0.26167 avg lat: 0.0082144 2024-02-09T17:46:32.100 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-09T17:46:32.100 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 40 16 77903 77887 121.685 82.5625 0.0150915 0.0082144 2024-02-09T17:46:32.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:31 smithi179 ceph-mon[43975]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:46:32.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:31 smithi179 ceph-mon[43975]: Cluster is now healthy 2024-02-09T17:46:32.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:31 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:46:32.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:31 smithi179 ceph-mon[43975]: osdmap e593: 12 total, 12 up, 12 in 2024-02-09T17:46:32.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:31 smithi179 ceph-mon[43975]: pgmap v1486: 47 pgs: 47 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-02-09T17:46:32.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:31 smithi179 ceph-mon[43975]: 1.1a scrub starts 2024-02-09T17:46:32.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:31 smithi179 ceph-mon[43975]: 1.1a scrub ok 2024-02-09T17:46:32.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:31 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/207033130' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:46:32.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:31 smithi179 ceph-mon[43975]: osdmap e594: 12 total, 12 up, 12 in 2024-02-09T17:46:32.184 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:46:32.213 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:32.238 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:46:32.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:32.241 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:46:32.342 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:46:32.387 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:32.417 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:46:32.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:32.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:46:32.526 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:46:32.575 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:33.100 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 41 16 79223 79207 120.729 82.5 0.0134207 0.00827883 2024-02-09T17:46:33.633 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:46:33.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:33.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:46:33.741 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:46:33.786 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:33.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:33 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:46:33.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:33 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:46:33.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:33 smithi162 ceph-mon[44304]: osdmap e595: 12 total, 12 up, 12 in 2024-02-09T17:46:33.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:33 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:46:33.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:33 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:46:33.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:33 smithi162 ceph-mon[44304]: pgmap v1489: 47 pgs: 47 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 103 MiB/s wr, 1.64k op/s 2024-02-09T17:46:33.939 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set noscrub 2024-02-09T17:46:34.100 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 42 16 80565 80549 119.851 83.875 0.0160172 0.00834007 2024-02-09T17:46:34.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:33 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:46:34.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:33 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:46:34.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:33 smithi179 ceph-mon[43975]: osdmap e595: 12 total, 12 up, 12 in 2024-02-09T17:46:34.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:33 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:46:34.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:33 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:46:34.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:33 smithi179 ceph-mon[43975]: pgmap v1489: 47 pgs: 47 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 103 MiB/s wr, 1.64k op/s 2024-02-09T17:46:34.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:33 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:46:34.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:33 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:46:34.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:33 smithi099 ceph-mon[39420]: osdmap e595: 12 total, 12 up, 12 in 2024-02-09T17:46:34.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:33 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:46:34.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:33 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:46:34.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:33 smithi099 ceph-mon[39420]: pgmap v1489: 47 pgs: 47 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 103 MiB/s wr, 1.64k op/s 2024-02-09T17:46:34.696 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:46:34.696 INFO:teuthology.orchestra.run.smithi099.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-02-09T17:46:34.696 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:46:34.703 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:46:34.745 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:46:34.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:34.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:46:34.851 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:46:34.904 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:46:34.923 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:34.993 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:46:34.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:34 smithi099 ceph-mon[39420]: osdmap e596: 12 total, 12 up, 12 in 2024-02-09T17:46:34.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:34.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:46:35.089 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:34 smithi162 ceph-mon[44304]: osdmap e596: 12 total, 12 up, 12 in 2024-02-09T17:46:35.099 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:46:35.100 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 43 16 81899 81883 119.003 83.375 0.00898057 0.00839831 2024-02-09T17:46:35.157 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:35.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:34 smithi179 ceph-mon[43975]: osdmap e596: 12 total, 12 up, 12 in 2024-02-09T17:46:35.294 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:46:35.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:35.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:46:35.401 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:46:35.471 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:35.516 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:46:35.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:35.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:46:35.624 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:46:35.675 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:35.702 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:46:35.703 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:46:35.728 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:46:35.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:35.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:46:35.832 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:46:35.855 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:35 smithi099 ceph-mon[39420]: pgmap v1491: 47 pgs: 47 active+clean; 4.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 94 MiB/s wr, 1.51k op/s 2024-02-09T17:46:35.864 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:35.889 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:46:35.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:35.893 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:46:35.994 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:46:36.051 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:36.100 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 44 16 83217 83201 118.17 82.375 0.0137792 0.00845835 2024-02-09T17:46:36.101 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:46:36.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:36.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:46:36.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:35 smithi179 ceph-mon[43975]: pgmap v1491: 47 pgs: 47 active+clean; 4.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 94 MiB/s wr, 1.51k op/s 2024-02-09T17:46:36.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:35 smithi162 ceph-mon[44304]: pgmap v1491: 47 pgs: 47 active+clean; 4.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 94 MiB/s wr, 1.51k op/s 2024-02-09T17:46:36.210 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:46:36.281 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:36.496 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:46:36.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:36.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:46:36.547 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:46:36.601 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:46:36.631 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:36.655 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:46:36.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:36.659 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:46:36.760 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:46:36.807 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:37.042 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:46:37.056 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:37.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:46:37.093 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.9 dump_ops_in_flight 2024-02-09T17:46:37.100 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 45 16 84573 84557 117.428 84.75 0.0106941 0.00851166 2024-02-09T17:46:37.158 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:46:37.221 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:37.312 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:46:37.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:37.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:46:37.432 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:46:37.455 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:46:37.487 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:37.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:37 smithi099 ceph-mon[39420]: 1.18 scrub starts 2024-02-09T17:46:37.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:37 smithi099 ceph-mon[39420]: 1.18 scrub ok 2024-02-09T17:46:37.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:37 smithi099 ceph-mon[39420]: pgmap v1492: 47 pgs: 47 active+clean; 4.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s 2024-02-09T17:46:37.891 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:37 smithi179 ceph-mon[43975]: 1.18 scrub starts 2024-02-09T17:46:37.891 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:37 smithi179 ceph-mon[43975]: 1.18 scrub ok 2024-02-09T17:46:37.891 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:37 smithi179 ceph-mon[43975]: pgmap v1492: 47 pgs: 47 active+clean; 4.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s 2024-02-09T17:46:37.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:37 smithi162 ceph-mon[44304]: 1.18 scrub starts 2024-02-09T17:46:37.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:37 smithi162 ceph-mon[44304]: 1.18 scrub ok 2024-02-09T17:46:37.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:37 smithi162 ceph-mon[44304]: pgmap v1492: 47 pgs: 47 active+clean; 4.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s 2024-02-09T17:46:38.100 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 46 15 85909 85894 116.691 83.5625 0.0180974 0.00856545 2024-02-09T17:46:39.101 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 47 16 87280 87264 116.03 85.625 0.024532 0.00861449 2024-02-09T17:46:39.224 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:38 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:46:39.224 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:38 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:46:39.224 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:38 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:46:39.224 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:38 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:46:39.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:38 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:46:39.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:38 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:46:39.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:38 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:46:39.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:38 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:46:39.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:38 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:46:39.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:38 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:46:39.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:38 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:46:39.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:38 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:46:40.068 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:46:40.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:40.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:46:40.100 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 48 16 88559 88543 115.278 79.9375 0.00942554 0.00867064 2024-02-09T17:46:40.192 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:46:40.256 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:40.519 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:46:40.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:40.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:46:40.633 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:46:40.640 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] 2024-02-09T17:46:40.640 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:46:40.640 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-02-09T17:46:40.640 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 0.000000 2024-02-09T17:46:40.640 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd primary-affinity 5 0 2024-02-09T17:46:40.688 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:40.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:40 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:46:40.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:40 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:46:40.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:40 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:46:40.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:40 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:46:40.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:40 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:46:40.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:40 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:46:40.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:40 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:46:40.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:40 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:46:40.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:40 smithi162 ceph-mon[44304]: 1.19 scrub starts 2024-02-09T17:46:40.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:40 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:46:40.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:40 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:46:40.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:40 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:46:40.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:40 smithi162 ceph-mon[44304]: 1.19 scrub ok 2024-02-09T17:46:40.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:40 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:46:40.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:40 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:46:40.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:40 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:46:40.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:40 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/704682690' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:46:40.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:40 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:46:40.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:40 smithi162 ceph-mon[44304]: 1.3 deep-scrub starts 2024-02-09T17:46:40.937 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:40 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:46:40.937 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:40 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:46:40.937 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:40 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:46:40.937 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:40 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:46:40.937 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:40 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:46:40.937 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:40 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:46:40.937 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:40 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:46:40.937 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:40 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:46:40.937 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:40 smithi099 ceph-mon[39420]: 1.19 scrub starts 2024-02-09T17:46:40.938 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:40 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:46:40.938 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:40 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:46:40.938 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:40 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:46:40.938 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:40 smithi099 ceph-mon[39420]: 1.19 scrub ok 2024-02-09T17:46:40.938 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:40 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:46:40.938 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:40 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:46:40.938 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:40 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:46:40.938 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:40 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/704682690' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:46:40.938 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:40 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:46:40.938 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:40 smithi099 ceph-mon[39420]: 1.3 deep-scrub starts 2024-02-09T17:46:40.955 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:46:40.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:40.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:46:40.999 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is set 2024-02-09T17:46:41.066 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:46:41.101 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 49 16 89731 89715 114.42 73.25 0.0165317 0.0087353 2024-02-09T17:46:41.115 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:41.149 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:46:41.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:41.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:46:41.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:40 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:46:41.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:40 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:46:41.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:40 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:46:41.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:40 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:46:41.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:40 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:46:41.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:40 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:46:41.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:40 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:46:41.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:40 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:46:41.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:40 smithi179 ceph-mon[43975]: 1.19 scrub starts 2024-02-09T17:46:41.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:40 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:46:41.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:40 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:46:41.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:40 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:46:41.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:40 smithi179 ceph-mon[43975]: 1.19 scrub ok 2024-02-09T17:46:41.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:40 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:46:41.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:40 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:46:41.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:40 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:46:41.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:40 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/704682690' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:46:41.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:40 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:46:41.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:40 smithi179 ceph-mon[43975]: 1.3 deep-scrub starts 2024-02-09T17:46:41.255 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:46:41.309 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:41.361 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:46:41.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:41.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:46:41.425 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:46:41.425 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:46:41.425 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:46:41.458 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:46:41.464 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:46:41.492 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:46:41.497 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:41.506 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:46:41.521 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:46:41.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:41.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:46:41.548 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:46:41.581 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:46:41.620 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:46:41.627 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:46:41.659 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:46:41.662 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:41.698 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:46:41.733 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:46:41.763 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:46:41.770 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:46:41.770 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:46:41.770 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:46:41.770 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:46:41.770 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:46:41.770 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:46:41.770 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:46:41.770 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:46:41.770 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:46:41.770 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:46:41.770 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:46:41.770 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:46:41.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:41 smithi179 ceph-mon[43975]: 1.3 deep-scrub ok 2024-02-09T17:46:41.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:41 smithi179 ceph-mon[43975]: pgmap v1494: 47 pgs: 47 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.11k op/s 2024-02-09T17:46:41.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:41 smithi179 ceph-mon[43975]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:46:41.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:41 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:46:41.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:41 smithi179 ceph-mon[43975]: osdmap e597: 12 total, 12 up, 12 in 2024-02-09T17:46:41.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:41 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1481263527' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:46:41.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:41 smithi179 ceph-mon[43975]: osdmap e598: 12 total, 12 up, 12 in 2024-02-09T17:46:41.932 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:41 smithi099 ceph-mon[39420]: 1.3 deep-scrub ok 2024-02-09T17:46:41.932 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:41 smithi099 ceph-mon[39420]: pgmap v1494: 47 pgs: 47 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.11k op/s 2024-02-09T17:46:41.932 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:41 smithi099 ceph-mon[39420]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:46:41.932 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:41 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:46:41.932 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:41 smithi099 ceph-mon[39420]: osdmap e597: 12 total, 12 up, 12 in 2024-02-09T17:46:41.932 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:41 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1481263527' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:46:41.932 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:41 smithi099 ceph-mon[39420]: osdmap e598: 12 total, 12 up, 12 in 2024-02-09T17:46:42.042 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:46:42.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:42.047 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:46:42.101 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 50 16 90933 90917 113.634 75.125 0.0138577 0.0087961 2024-02-09T17:46:42.148 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:46:42.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:41 smithi162 ceph-mon[44304]: 1.3 deep-scrub ok 2024-02-09T17:46:42.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:41 smithi162 ceph-mon[44304]: pgmap v1494: 47 pgs: 47 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.11k op/s 2024-02-09T17:46:42.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:41 smithi162 ceph-mon[44304]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:46:42.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:41 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:46:42.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:41 smithi162 ceph-mon[44304]: osdmap e597: 12 total, 12 up, 12 in 2024-02-09T17:46:42.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:41 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1481263527' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:46:42.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:41 smithi162 ceph-mon[44304]: osdmap e598: 12 total, 12 up, 12 in 2024-02-09T17:46:42.207 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:42.240 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:46:42.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:42.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:46:42.351 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:46:42.389 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:42.472 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:46:42.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:42.475 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:46:42.577 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:46:42.627 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:46:42.627 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-02-09T17:46:42.627 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.628 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25837.0:91611 6.9 6:93eda3fb:::benchmark_data_smithi179_126206_object91610:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-02-09T17:46:42.628 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:46:42.613619+0000", 2024-02-09T17:46:42.628 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.011988515999999999, 2024-02-09T17:46:42.628 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.012018439000000001, 2024-02-09T17:46:42.628 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:46:42.628 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "waiting for sub ops", 2024-02-09T17:46:42.628 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:46:42.628 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25837", 2024-02-09T17:46:42.628 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:46:42.628 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 91611 2024-02-09T17:46:42.628 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.628 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:46:42.628 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.629 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:46:42.629 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.613619+0000", 2024-02-09T17:46:42.629 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:42.629 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.629 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.629 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:46:42.629 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.613619+0000", 2024-02-09T17:46:42.629 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:42.629 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.629 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.629 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:46:42.629 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.613619+0000", 2024-02-09T17:46:42.629 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.13e-07 2024-02-09T17:46:42.629 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.629 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.629 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:46:42.630 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.613637+0000", 2024-02-09T17:46:42.630 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7379000000000001e-05 2024-02-09T17:46:42.630 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.630 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.630 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:46:42.630 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.613644+0000", 2024-02-09T17:46:42.630 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5190000000000003e-06 2024-02-09T17:46:42.630 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.630 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.630 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:46:42.630 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.613650+0000", 2024-02-09T17:46:42.630 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.879e-06 2024-02-09T17:46:42.630 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.630 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.630 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:46:42.631 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.617179+0000", 2024-02-09T17:46:42.631 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0035289039999999998 2024-02-09T17:46:42.631 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.631 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.631 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:46:42.631 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.617347+0000", 2024-02-09T17:46:42.631 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016768999999999999 2024-02-09T17:46:42.631 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.631 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.631 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,8", 2024-02-09T17:46:42.631 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.617479+0000", 2024-02-09T17:46:42.631 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000132154 2024-02-09T17:46:42.631 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.631 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.632 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:46:42.632 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.619836+0000", 2024-02-09T17:46:42.632 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0023574239999999999 2024-02-09T17:46:42.632 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.632 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.632 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:46:42.632 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.622273+0000", 2024-02-09T17:46:42.632 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0024369510000000001 2024-02-09T17:46:42.632 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:42.632 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:46:42.632 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:42.632 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.632 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.633 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25837.0:91622 6.9 6:9f9c0878:::benchmark_data_smithi179_126206_object91621:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-02-09T17:46:42.633 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:46:42.624452+0000", 2024-02-09T17:46:42.633 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.0011548929999999999, 2024-02-09T17:46:42.633 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0012840410000000001, 2024-02-09T17:46:42.633 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:46:42.633 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "waiting for sub ops", 2024-02-09T17:46:42.633 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:46:42.633 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25837", 2024-02-09T17:46:42.633 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:46:42.633 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 91622 2024-02-09T17:46:42.633 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.633 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:46:42.633 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.633 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:46:42.633 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.624452+0000", 2024-02-09T17:46:42.634 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:42.634 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.634 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.634 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:46:42.634 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.624452+0000", 2024-02-09T17:46:42.634 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:42.634 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.634 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.634 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:46:42.634 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.624454+0000", 2024-02-09T17:46:42.634 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4649999999999999e-06 2024-02-09T17:46:42.634 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.634 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.634 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:46:42.634 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.624471+0000", 2024-02-09T17:46:42.635 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7448999999999998e-05 2024-02-09T17:46:42.635 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.635 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.635 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:46:42.635 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.624482+0000", 2024-02-09T17:46:42.635 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1275e-05 2024-02-09T17:46:42.635 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.635 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.635 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:46:42.635 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.624488+0000", 2024-02-09T17:46:42.635 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6980000000000003e-06 2024-02-09T17:46:42.635 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.635 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.635 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:46:42.635 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.624542+0000", 2024-02-09T17:46:42.636 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4095999999999998e-05 2024-02-09T17:46:42.636 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.636 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.636 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:46:42.636 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.624690+0000", 2024-02-09T17:46:42.636 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014792700000000001 2024-02-09T17:46:42.636 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.636 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.636 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,8", 2024-02-09T17:46:42.636 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.624810+0000", 2024-02-09T17:46:42.636 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000119541 2024-02-09T17:46:42.636 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:42.636 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:46:42.637 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:42.637 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.637 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.637 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25837.0:91623 6.9 6.b36c3f99 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-02-09T17:46:42.637 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:46:42.624611+0000", 2024-02-09T17:46:42.637 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.00099573900000000004, 2024-02-09T17:46:42.637 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0011965439999999999, 2024-02-09T17:46:42.637 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:46:42.637 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "queued for pg", 2024-02-09T17:46:42.637 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:46:42.637 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25837", 2024-02-09T17:46:42.637 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:46:42.637 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 91623 2024-02-09T17:46:42.637 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.637 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:46:42.638 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.638 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:46:42.638 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.624611+0000", 2024-02-09T17:46:42.638 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:42.638 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.638 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.638 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:46:42.638 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.624611+0000", 2024-02-09T17:46:42.638 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:42.638 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.638 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.638 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:46:42.638 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.624612+0000", 2024-02-09T17:46:42.638 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8000000000000005e-07 2024-02-09T17:46:42.638 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.639 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.639 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:46:42.639 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.624628+0000", 2024-02-09T17:46:42.639 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5846e-05 2024-02-09T17:46:42.639 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.639 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.639 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:46:42.639 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.624634+0000", 2024-02-09T17:46:42.639 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8710000000000002e-06 2024-02-09T17:46:42.639 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.639 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.639 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:46:42.639 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.624638+0000", 2024-02-09T17:46:42.639 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5639999999999998e-06 2024-02-09T17:46:42.640 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:42.640 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:46:42.640 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:42.640 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.640 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.640 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop_reply(client.25837.0:91609 6.9 e598/576)", 2024-02-09T17:46:42.640 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:46:42.625694+0000", 2024-02-09T17:46:42.640 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 4294967295.9999132, 2024-02-09T17:46:42.640 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000166, 2024-02-09T17:46:42.640 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:46:42.640 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "queued for pg", 2024-02-09T17:46:42.640 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:46:42.641 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.641 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:46:42.641 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.625694+0000", 2024-02-09T17:46:42.641 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:42.641 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:42.641 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.641 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.642 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:46:42.642 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.625694+0000", 2024-02-09T17:46:42.642 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:42.642 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.642 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.642 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:46:42.642 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.625695+0000", 2024-02-09T17:46:42.642 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8000000000000005e-07 2024-02-09T17:46:42.642 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.642 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.642 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:46:42.642 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.625699+0000", 2024-02-09T17:46:42.642 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.036e-06 2024-02-09T17:46:42.642 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.642 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.643 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:46:42.643 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.625709+0000", 2024-02-09T17:46:42.643 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.9019999999999994e-06 2024-02-09T17:46:42.643 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.643 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.643 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:46:42.643 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.625715+0000", 2024-02-09T17:46:42.643 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4370000000000004e-06 2024-02-09T17:46:42.643 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:42.643 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:46:42.643 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:42.643 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.643 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.643 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop_reply(client.25837.0:91611 6.9 e598/576)", 2024-02-09T17:46:42.643 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:46:42.625855+0000", 2024-02-09T17:46:42.644 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 4294967295.999752, 2024-02-09T17:46:42.644 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0862000000000001e-05, 2024-02-09T17:46:42.644 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:46:42.644 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "queued for pg", 2024-02-09T17:46:42.644 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:46:42.644 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.644 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:46:42.644 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.625855+0000", 2024-02-09T17:46:42.644 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:42.644 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.644 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.644 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:46:42.644 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.625855+0000", 2024-02-09T17:46:42.644 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:42.645 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.645 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.645 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:46:42.645 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.625856+0000", 2024-02-09T17:46:42.645 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6000000000000003e-07 2024-02-09T17:46:42.645 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.645 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.645 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:46:42.645 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.625861+0000", 2024-02-09T17:46:42.645 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8899999999999998e-06 2024-02-09T17:46:42.645 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.645 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.645 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:46:42.645 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.625872+0000", 2024-02-09T17:46:42.645 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1873e-05 2024-02-09T17:46:42.646 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.646 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.646 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:46:42.646 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.625880+0000", 2024-02-09T17:46:42.646 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2400000000000001e-06 2024-02-09T17:46:42.646 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:42.646 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:46:42.646 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:42.646 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.646 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.646 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25837.0:91609 6.9 6:9d6e7e60:::benchmark_data_smithi179_126206_object91608:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-02-09T17:46:42.646 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:46:42.613551+0000", 2024-02-09T17:46:42.646 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.012056025, 2024-02-09T17:46:42.646 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.012418297999999999, 2024-02-09T17:46:42.646 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:46:42.646 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "waiting for sub ops", 2024-02-09T17:46:42.647 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:46:42.647 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25837", 2024-02-09T17:46:42.647 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:46:42.647 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 91609 2024-02-09T17:46:42.647 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.647 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:46:42.647 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.647 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:46:42.647 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.613551+0000", 2024-02-09T17:46:42.647 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:42.647 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.647 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.647 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:46:42.647 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.613551+0000", 2024-02-09T17:46:42.647 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:42.647 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.648 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.648 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:46:42.648 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.613553+0000", 2024-02-09T17:46:42.648 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8899999999999999e-06 2024-02-09T17:46:42.648 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.648 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.648 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:46:42.648 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.613574+0000", 2024-02-09T17:46:42.648 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0746000000000001e-05 2024-02-09T17:46:42.648 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.648 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.648 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:46:42.648 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.613588+0000", 2024-02-09T17:46:42.648 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4133e-05 2024-02-09T17:46:42.649 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.649 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.649 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:46:42.649 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.613596+0000", 2024-02-09T17:46:42.649 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2719999999999997e-06 2024-02-09T17:46:42.649 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.649 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.649 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:46:42.649 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.613657+0000", 2024-02-09T17:46:42.649 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0538000000000002e-05 2024-02-09T17:46:42.649 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.649 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.649 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:46:42.649 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.613793+0000", 2024-02-09T17:46:42.649 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000136489 2024-02-09T17:46:42.649 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.650 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.650 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,8", 2024-02-09T17:46:42.650 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.613905+0000", 2024-02-09T17:46:42.650 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000112318 2024-02-09T17:46:42.650 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.650 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.650 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:46:42.650 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.619708+0000", 2024-02-09T17:46:42.650 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0058026680000000004 2024-02-09T17:46:42.650 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:42.650 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:42.650 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:46:42.650 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.619752+0000", 2024-02-09T17:46:42.650 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3421000000000003e-05 2024-02-09T17:46:42.650 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:42.650 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:46:42.651 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:42.651 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:42.651 INFO:teuthology.orchestra.run.smithi179.stdout: ], 2024-02-09T17:46:42.651 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 6 2024-02-09T17:46:42.651 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:46:42.664 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:46:42.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:42.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:46:42.771 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:46:42.833 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:43.101 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 51 16 92271 92255 113.046 83.625 0.00390366 0.00884213 2024-02-09T17:46:43.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:43 smithi099 ceph-mon[39420]: pgmap v1497: 47 pgs: 47 active+clean; 5.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.38k op/s 2024-02-09T17:46:44.010 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.9 dump_blocked_ops 2024-02-09T17:46:44.100 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:46:44.101 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 52 16 93550 93534 112.409 79.9375 0.0107002 0.00889212 2024-02-09T17:46:44.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:44.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:46:44.206 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:46:44.262 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:44.297 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:46:44.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:44.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:46:44.342 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:46:44.406 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:46:44.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:43 smithi179 ceph-mon[43975]: pgmap v1497: 47 pgs: 47 active+clean; 5.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.38k op/s 2024-02-09T17:46:44.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:43 smithi162 ceph-mon[44304]: pgmap v1497: 47 pgs: 47 active+clean; 5.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.38k op/s 2024-02-09T17:46:44.469 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:44.593 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-09T17:46:44.835 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:46:44.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:44.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:46:44.943 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:46:44.980 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:45.011 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:46:45.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:45.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:46:45.034 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:46:45.101 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 53 16 94685 94669 111.626 70.9375 0.0142085 0.00895418 2024-02-09T17:46:45.120 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:46:45.151 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:46:45.170 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:44 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:46:45.171 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:44 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:46:45.171 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:44 smithi099 ceph-mon[39420]: osdmap e599: 12 total, 12 up, 12 in 2024-02-09T17:46:45.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:44 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:46:45.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:44 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:46:45.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:44 smithi179 ceph-mon[43975]: osdmap e599: 12 total, 12 up, 12 in 2024-02-09T17:46:45.174 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:45.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:44 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:46:45.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:44 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:46:45.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:44 smithi162 ceph-mon[44304]: osdmap e599: 12 total, 12 up, 12 in 2024-02-09T17:46:45.643 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:46:45.647 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:45.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:46:45.749 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:46:45.805 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:45.830 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:46:45.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:45.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:46:45.934 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:46:45.996 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:46.028 INFO:teuthology.orchestra.run.smithi099.stderr:set osd.5 primary-affinity to 0 (802) 2024-02-09T17:46:46.034 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:46:46.036 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:46.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:46:46.101 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 54 16 95824 95808 110.877 71.1875 0.00356574 0.00900275 2024-02-09T17:46:46.138 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:46:46.148 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:46 smithi179 ceph-mon[43975]: pgmap v1499: 47 pgs: 47 active+clean; 5.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s 2024-02-09T17:46:46.148 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:46 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2418723537' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]: dispatch 2024-02-09T17:46:46.148 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:46 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]: dispatch 2024-02-09T17:46:46.167 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:46.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:46 smithi162 ceph-mon[44304]: pgmap v1499: 47 pgs: 47 active+clean; 5.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s 2024-02-09T17:46:46.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:46 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2418723537' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]: dispatch 2024-02-09T17:46:46.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:46 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]: dispatch 2024-02-09T17:46:46.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:46 smithi099 ceph-mon[39420]: pgmap v1499: 47 pgs: 47 active+clean; 5.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s 2024-02-09T17:46:46.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:46 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2418723537' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]: dispatch 2024-02-09T17:46:46.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:46 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]: dispatch 2024-02-09T17:46:46.422 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:46:46.422 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-02-09T17:46:46.422 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-02-09T17:46:46.422 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-02-09T17:46:46.422 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:46:47.017 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:46:47.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:47.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:46:47.101 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 55 16 95912 95896 108.961 5.5 0.00499768 0.00917338 2024-02-09T17:46:47.121 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:46:47.176 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:47.377 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:47 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]': finished 2024-02-09T17:46:47.377 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:47 smithi179 ceph-mon[43975]: osdmap e600: 12 total, 12 up, 12 in 2024-02-09T17:46:47.377 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:47 smithi179 ceph-mon[43975]: osdmap e601: 12 total, 12 up, 12 in 2024-02-09T17:46:47.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:47 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]': finished 2024-02-09T17:46:47.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:47 smithi162 ceph-mon[44304]: osdmap e600: 12 total, 12 up, 12 in 2024-02-09T17:46:47.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:47 smithi162 ceph-mon[44304]: osdmap e601: 12 total, 12 up, 12 in 2024-02-09T17:46:47.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:47 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]': finished 2024-02-09T17:46:47.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:47 smithi099 ceph-mon[39420]: osdmap e600: 12 total, 12 up, 12 in 2024-02-09T17:46:47.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:47 smithi099 ceph-mon[39420]: osdmap e601: 12 total, 12 up, 12 in 2024-02-09T17:46:47.657 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:46:47.657 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:46:47.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:47.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:46:47.712 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.9 dump_historic_ops 2024-02-09T17:46:47.768 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:46:47.836 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:47.860 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:46:47.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:47.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:46:47.966 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:46:48.040 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:48 smithi179 ceph-mon[43975]: pgmap v1501: 47 pgs: 1 peering, 46 active+clean; 5.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.13k op/s 2024-02-09T17:46:48.040 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:48 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:46:48.040 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:48 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:46:48.041 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:48.074 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:46:48.101 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 56 16 97116 97100 108.359 75.25 0.0114822 0.00922473 2024-02-09T17:46:48.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:48 smithi162 ceph-mon[44304]: pgmap v1501: 47 pgs: 1 peering, 46 active+clean; 5.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.13k op/s 2024-02-09T17:46:48.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:48 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:46:48.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:48 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:46:48.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:48 smithi099 ceph-mon[39420]: pgmap v1501: 47 pgs: 1 peering, 46 active+clean; 5.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.13k op/s 2024-02-09T17:46:48.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:48 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:46:48.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:48 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:46:49.060 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is set 2024-02-09T17:46:49.102 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 57 16 98131 98115 107.571 63.4375 0.00823993 0.00929231 2024-02-09T17:46:49.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:49 smithi179 ceph-mon[43975]: osdmap e602: 12 total, 12 up, 12 in 2024-02-09T17:46:49.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:49 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:46:49.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:49 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:46:49.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:49 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3313198918' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:46:49.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:49 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:46:49.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:49 smithi162 ceph-mon[44304]: osdmap e602: 12 total, 12 up, 12 in 2024-02-09T17:46:49.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:49 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:46:49.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:49 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:46:49.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:49 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3313198918' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:46:49.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:49 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:46:49.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:49 smithi099 ceph-mon[39420]: osdmap e602: 12 total, 12 up, 12 in 2024-02-09T17:46:49.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:49 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:46:49.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:49 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:46:49.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:49 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3313198918' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:46:49.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:49 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:46:49.989 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:46:49.990 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-02-09T17:46:49.990 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-02-09T17:46:49.990 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-02-09T17:46:49.990 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:49.990 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:208400 4.2 4:4b2cafe6:::benchmark_data_smithi179_103809_object70934:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-02-09T17:46:49.990 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:33.808091+0000", 2024-02-09T17:46:49.990 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 496.17702940100003, 2024-02-09T17:46:49.990 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3447615150000001, 2024-02-09T17:46:49.990 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:46:49.990 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-02-09T17:46:49.990 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:46:49.990 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:46:49.990 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:46:49.990 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 208400 2024-02-09T17:46:49.991 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:49.991 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:46:49.991 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:49.991 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:46:49.991 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.808091+0000", 2024-02-09T17:46:49.991 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:49.991 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:49.991 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:49.991 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:46:49.991 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.808091+0000", 2024-02-09T17:46:49.991 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:49.991 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:49.991 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:49.991 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:46:49.991 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.808093+0000", 2024-02-09T17:46:49.992 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.48e-06 2024-02-09T17:46:49.992 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:49.992 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:49.992 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:46:49.992 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.808099+0000", 2024-02-09T17:46:49.992 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9329999999999998e-06 2024-02-09T17:46:49.992 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:49.992 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:49.992 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:46:49.992 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.808113+0000", 2024-02-09T17:46:49.992 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4032000000000001e-05 2024-02-09T17:46:49.992 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:49.992 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:49.992 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:46:49.993 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.808120+0000", 2024-02-09T17:46:49.993 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7020000000000001e-06 2024-02-09T17:46:49.993 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:49.993 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:49.993 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:46:49.993 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.808181+0000", 2024-02-09T17:46:49.993 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0825999999999997e-05 2024-02-09T17:46:49.993 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:49.993 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:49.993 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:46:49.993 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.808278+0000", 2024-02-09T17:46:49.993 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6510999999999998e-05 2024-02-09T17:46:49.993 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:49.993 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:49.993 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,11", 2024-02-09T17:46:49.994 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.808354+0000", 2024-02-09T17:46:49.994 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6171e-05 2024-02-09T17:46:49.994 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:49.994 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:49.994 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:46:49.994 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.815614+0000", 2024-02-09T17:46:49.994 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0072599580000000004 2024-02-09T17:46:49.994 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:49.994 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:49.994 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:46:49.994 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.821051+0000", 2024-02-09T17:46:49.994 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.005437023 2024-02-09T17:46:49.994 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:49.994 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:49.994 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:46:49.995 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.821065+0000", 2024-02-09T17:46:49.995 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4011e-05 2024-02-09T17:46:49.995 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:49.995 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:49.995 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:46:49.995 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:34.812708+0000", 2024-02-09T17:46:49.995 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.99164264599999996 2024-02-09T17:46:49.995 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:49.995 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:49.995 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:46:49.995 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:34.812721+0000", 2024-02-09T17:46:49.995 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3475999999999999e-05 2024-02-09T17:46:49.995 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:49.995 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:49.995 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:46:49.996 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.152766+0000", 2024-02-09T17:46:49.996 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3400449459999999 2024-02-09T17:46:49.996 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:49.996 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:49.996 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:46:49.996 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.152853+0000", 2024-02-09T17:46:49.996 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.6799999999999996e-05 2024-02-09T17:46:49.996 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:49.996 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:46:49.996 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:49.996 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:49.996 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:49.996 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:209208 4.2 4:4d9658b8:::benchmark_data_smithi179_103809_object71742:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-02-09T17:46:49.997 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:37.305316+0000", 2024-02-09T17:46:49.997 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 492.679804623, 2024-02-09T17:46:49.997 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.218476264, 2024-02-09T17:46:49.997 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:46:49.997 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:46:49.997 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:46:49.997 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:46:49.997 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:46:49.997 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 209208 2024-02-09T17:46:49.997 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:49.997 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:46:49.997 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:49.997 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:46:49.997 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.305316+0000", 2024-02-09T17:46:49.998 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:49.998 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:49.998 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:49.998 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:46:49.998 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.305316+0000", 2024-02-09T17:46:49.998 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:49.998 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:49.998 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:49.998 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:46:49.998 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.305318+0000", 2024-02-09T17:46:49.998 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7549999999999999e-06 2024-02-09T17:46:49.998 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:49.998 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:49.998 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:46:49.998 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.305324+0000", 2024-02-09T17:46:49.998 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.533e-06 2024-02-09T17:46:49.999 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:49.999 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:49.999 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:46:49.999 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.305338+0000", 2024-02-09T17:46:49.999 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3716999999999999e-05 2024-02-09T17:46:49.999 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:49.999 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:49.999 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:46:49.999 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.305347+0000", 2024-02-09T17:46:49.999 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.9199999999999993e-06 2024-02-09T17:46:49.999 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:49.999 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:49.999 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:46:49.999 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.305412+0000", 2024-02-09T17:46:50.000 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5012999999999996e-05 2024-02-09T17:46:50.000 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.000 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.000 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:46:50.000 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.305574+0000", 2024-02-09T17:46:50.000 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016153599999999999 2024-02-09T17:46:50.000 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.000 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.000 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,8,11", 2024-02-09T17:46:50.000 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.305726+0000", 2024-02-09T17:46:50.000 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015199199999999999 2024-02-09T17:46:50.000 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.000 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.000 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:46:50.000 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.310758+0000", 2024-02-09T17:46:50.001 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0050327130000000003 2024-02-09T17:46:50.001 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.001 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.001 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:46:50.001 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.312487+0000", 2024-02-09T17:46:50.001 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0017286949999999999 2024-02-09T17:46:50.001 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.001 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.001 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:46:50.001 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.313534+0000", 2024-02-09T17:46:50.001 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001047405 2024-02-09T17:46:50.001 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.001 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.001 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:46:50.001 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.523599+0000", 2024-02-09T17:46:50.002 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.21006412599999999 2024-02-09T17:46:50.002 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.002 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.002 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:46:50.002 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.523667+0000", 2024-02-09T17:46:50.002 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8040999999999994e-05 2024-02-09T17:46:50.002 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.002 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.002 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:46:50.002 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.523792+0000", 2024-02-09T17:46:50.002 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012581800000000001 2024-02-09T17:46:50.002 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:50.002 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:46:50.002 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:50.003 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.003 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.003 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:209216 4.2 4:4b5c16ca:::benchmark_data_smithi179_103809_object71750:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-02-09T17:46:50.003 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:37.315200+0000", 2024-02-09T17:46:50.003 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 492.66992024400002, 2024-02-09T17:46:50.003 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.20880422400000001, 2024-02-09T17:46:50.003 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:46:50.003 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:46:50.003 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:46:50.003 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:46:50.003 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:46:50.003 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 209216 2024-02-09T17:46:50.003 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.003 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:46:50.003 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.004 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:46:50.004 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.315200+0000", 2024-02-09T17:46:50.004 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:50.004 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.004 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.004 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:46:50.004 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.315200+0000", 2024-02-09T17:46:50.004 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:50.004 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.004 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.004 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:46:50.004 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.315202+0000", 2024-02-09T17:46:50.004 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5260000000000001e-06 2024-02-09T17:46:50.004 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.005 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.005 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:46:50.005 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.315208+0000", 2024-02-09T17:46:50.005 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7309999999999998e-06 2024-02-09T17:46:50.005 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.005 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.005 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:46:50.005 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.315220+0000", 2024-02-09T17:46:50.005 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2167000000000001e-05 2024-02-09T17:46:50.005 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.005 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.005 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:46:50.005 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.315228+0000", 2024-02-09T17:46:50.005 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6969999999999993e-06 2024-02-09T17:46:50.006 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.006 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.006 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:46:50.006 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.315369+0000", 2024-02-09T17:46:50.006 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014159200000000001 2024-02-09T17:46:50.006 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.006 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.006 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:46:50.006 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.315496+0000", 2024-02-09T17:46:50.006 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000126477 2024-02-09T17:46:50.006 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.006 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.006 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,8,11", 2024-02-09T17:46:50.006 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.315627+0000", 2024-02-09T17:46:50.007 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000131562 2024-02-09T17:46:50.007 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.007 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.007 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:46:50.007 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.322165+0000", 2024-02-09T17:46:50.007 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.006538131 2024-02-09T17:46:50.007 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.007 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.007 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:46:50.007 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.329352+0000", 2024-02-09T17:46:50.007 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.007186947 2024-02-09T17:46:50.007 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.007 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.007 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:46:50.007 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.332320+0000", 2024-02-09T17:46:50.008 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002968142 2024-02-09T17:46:50.008 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.008 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.008 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:46:50.008 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.523840+0000", 2024-02-09T17:46:50.008 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.191519561 2024-02-09T17:46:50.008 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.008 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.008 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:46:50.008 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.523895+0000", 2024-02-09T17:46:50.008 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5000999999999997e-05 2024-02-09T17:46:50.008 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.008 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.009 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:46:50.009 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.524005+0000", 2024-02-09T17:46:50.009 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010969 2024-02-09T17:46:50.009 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:50.009 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:46:50.009 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:50.009 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.009 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.009 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:209219 4.2 4:4d2cfdf8:::benchmark_data_smithi179_103809_object71753:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-02-09T17:46:50.009 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:37.324408+0000", 2024-02-09T17:46:50.009 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 492.66071223400002, 2024-02-09T17:46:50.009 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.19981770700000001, 2024-02-09T17:46:50.009 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:46:50.009 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:46:50.010 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:46:50.010 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:46:50.010 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:46:50.010 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 209219 2024-02-09T17:46:50.010 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.010 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:46:50.010 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.010 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:46:50.010 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.324408+0000", 2024-02-09T17:46:50.010 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:50.010 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.010 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.010 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:46:50.010 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.324408+0000", 2024-02-09T17:46:50.010 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:50.011 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.011 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.011 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:46:50.011 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.324410+0000", 2024-02-09T17:46:50.011 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5510000000000001e-06 2024-02-09T17:46:50.011 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.011 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.011 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:46:50.011 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.324417+0000", 2024-02-09T17:46:50.011 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1219999999999997e-06 2024-02-09T17:46:50.011 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.011 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.011 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:46:50.011 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.324429+0000", 2024-02-09T17:46:50.011 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1741e-05 2024-02-09T17:46:50.012 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.012 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.012 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:46:50.012 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.324438+0000", 2024-02-09T17:46:50.012 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5470000000000002e-06 2024-02-09T17:46:50.012 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.012 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.012 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:46:50.012 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.324502+0000", 2024-02-09T17:46:50.012 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3635999999999999e-05 2024-02-09T17:46:50.012 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.012 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.012 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:46:50.012 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.324668+0000", 2024-02-09T17:46:50.013 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016551700000000001 2024-02-09T17:46:50.013 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.013 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.013 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,8,11", 2024-02-09T17:46:50.014 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.324820+0000", 2024-02-09T17:46:50.014 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015196900000000001 2024-02-09T17:46:50.014 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.014 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.014 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:46:50.014 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.329403+0000", 2024-02-09T17:46:50.014 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0045829399999999998 2024-02-09T17:46:50.014 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.014 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.014 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:46:50.014 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.332252+0000", 2024-02-09T17:46:50.014 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002849037 2024-02-09T17:46:50.014 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.014 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.014 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:46:50.015 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.332378+0000", 2024-02-09T17:46:50.015 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000126301 2024-02-09T17:46:50.015 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.015 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.015 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:46:50.015 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.524048+0000", 2024-02-09T17:46:50.015 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.191670117 2024-02-09T17:46:50.015 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.015 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.015 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:46:50.015 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.524110+0000", 2024-02-09T17:46:50.015 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2341999999999994e-05 2024-02-09T17:46:50.015 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.015 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.015 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:46:50.016 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.524226+0000", 2024-02-09T17:46:50.016 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000115887 2024-02-09T17:46:50.016 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:50.016 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:46:50.016 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:50.016 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.016 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.016 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:209236 4.2 4:4fc96872:::benchmark_data_smithi179_103809_object71770:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-02-09T17:46:50.016 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:37.345768+0000", 2024-02-09T17:46:50.016 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 492.63935271299999, 2024-02-09T17:46:50.016 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.178693559, 2024-02-09T17:46:50.016 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:46:50.016 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:46:50.016 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:46:50.017 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:46:50.017 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:46:50.017 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 209236 2024-02-09T17:46:50.017 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.017 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:46:50.017 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.017 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:46:50.017 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.345768+0000", 2024-02-09T17:46:50.017 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:50.017 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.017 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.017 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:46:50.017 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.345768+0000", 2024-02-09T17:46:50.017 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:50.018 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.018 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.018 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:46:50.018 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.345770+0000", 2024-02-09T17:46:50.018 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7770000000000001e-06 2024-02-09T17:46:50.018 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.018 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.018 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:46:50.018 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.345776+0000", 2024-02-09T17:46:50.018 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0170000000000002e-06 2024-02-09T17:46:50.018 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.018 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.018 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:46:50.018 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.345791+0000", 2024-02-09T17:46:50.018 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5641000000000002e-05 2024-02-09T17:46:50.019 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.019 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.019 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:46:50.019 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.345800+0000", 2024-02-09T17:46:50.019 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1580000000000005e-06 2024-02-09T17:46:50.019 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.019 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.019 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:46:50.019 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.345863+0000", 2024-02-09T17:46:50.019 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3812999999999994e-05 2024-02-09T17:46:50.019 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.019 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.019 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:46:50.019 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.345999+0000", 2024-02-09T17:46:50.020 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013552900000000001 2024-02-09T17:46:50.020 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.020 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.020 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,8,11", 2024-02-09T17:46:50.020 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.346096+0000", 2024-02-09T17:46:50.020 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6912999999999999e-05 2024-02-09T17:46:50.020 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.020 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.020 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:46:50.020 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.351978+0000", 2024-02-09T17:46:50.020 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0058820749999999996 2024-02-09T17:46:50.020 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.020 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.021 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:46:50.021 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.353224+0000", 2024-02-09T17:46:50.021 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001246521 2024-02-09T17:46:50.021 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.021 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.021 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:46:50.021 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.358127+0000", 2024-02-09T17:46:50.021 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0049025830000000003 2024-02-09T17:46:50.021 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.021 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.021 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:46:50.021 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.524271+0000", 2024-02-09T17:46:50.021 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.166144186 2024-02-09T17:46:50.022 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.022 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.022 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:46:50.022 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.524334+0000", 2024-02-09T17:46:50.022 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3032000000000002e-05 2024-02-09T17:46:50.022 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.022 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.022 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:46:50.022 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:37.524462+0000", 2024-02-09T17:46:50.022 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012731400000000001 2024-02-09T17:46:50.022 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:50.022 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:46:50.022 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:50.022 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.022 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.023 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:222253 4.6 4:62bd8438:::benchmark_data_smithi179_103809_object84787:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e397)", 2024-02-09T17:46:50.023 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:39:02.580398+0000", 2024-02-09T17:46:50.023 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 467.404722537, 2024-02-09T17:46:50.023 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0475380700000001, 2024-02-09T17:46:50.023 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:46:50.023 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-02-09T17:46:50.023 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:46:50.023 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:46:50.023 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:46:50.023 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 222253 2024-02-09T17:46:50.023 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.023 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:46:50.023 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.024 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:46:50.024 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.580398+0000", 2024-02-09T17:46:50.024 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:50.024 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.024 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.024 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:46:50.024 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.580398+0000", 2024-02-09T17:46:50.024 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:50.024 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.024 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.024 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:46:50.024 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.580399+0000", 2024-02-09T17:46:50.024 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6800000000000001e-07 2024-02-09T17:46:50.024 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.025 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.025 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:46:50.025 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.580401+0000", 2024-02-09T17:46:50.025 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6010000000000002e-06 2024-02-09T17:46:50.025 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.025 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.025 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:46:50.025 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.580409+0000", 2024-02-09T17:46:50.025 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1119999999999996e-06 2024-02-09T17:46:50.025 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.025 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.025 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:46:50.025 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.580415+0000", 2024-02-09T17:46:50.025 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6540000000000004e-06 2024-02-09T17:46:50.026 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.026 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.026 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:46:50.026 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.580470+0000", 2024-02-09T17:46:50.026 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5068999999999997e-05 2024-02-09T17:46:50.026 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.026 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.026 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:46:50.026 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.580565+0000", 2024-02-09T17:46:50.026 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.4953999999999999e-05 2024-02-09T17:46:50.026 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.026 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.026 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,2,7", 2024-02-09T17:46:50.026 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.580654+0000", 2024-02-09T17:46:50.026 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8467000000000005e-05 2024-02-09T17:46:50.027 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.027 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.027 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:46:50.027 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.590354+0000", 2024-02-09T17:46:50.027 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0097001880000000002 2024-02-09T17:46:50.027 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.027 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.027 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:46:50.027 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.590379+0000", 2024-02-09T17:46:50.027 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5245e-05 2024-02-09T17:46:50.027 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.027 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.027 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:46:50.027 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.594262+0000", 2024-02-09T17:46:50.028 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0038834389999999998 2024-02-09T17:46:50.028 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.028 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.028 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:46:50.028 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.594278+0000", 2024-02-09T17:46:50.028 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5860999999999999e-05 2024-02-09T17:46:50.028 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.028 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.028 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:46:50.028 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:03.627880+0000", 2024-02-09T17:46:50.029 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.033601583 2024-02-09T17:46:50.029 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.029 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.029 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:46:50.029 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:03.627936+0000", 2024-02-09T17:46:50.029 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6428999999999998e-05 2024-02-09T17:46:50.029 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:50.029 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:46:50.029 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:50.029 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.029 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.029 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:222261 4.6 4:64982b2b:::benchmark_data_smithi179_103809_object84795:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e397)", 2024-02-09T17:46:50.029 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:39:02.584432+0000", 2024-02-09T17:46:50.030 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 467.40068832399999, 2024-02-09T17:46:50.030 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.044083275, 2024-02-09T17:46:50.030 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:46:50.030 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-02-09T17:46:50.030 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:46:50.030 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:46:50.030 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:46:50.030 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 222261 2024-02-09T17:46:50.030 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.030 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:46:50.031 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.031 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:46:50.031 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.584432+0000", 2024-02-09T17:46:50.031 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:50.031 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.031 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.031 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:46:50.031 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.584432+0000", 2024-02-09T17:46:50.032 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:50.032 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.032 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.032 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:46:50.032 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.584434+0000", 2024-02-09T17:46:50.032 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.513e-06 2024-02-09T17:46:50.032 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.032 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.032 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:46:50.032 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.584440+0000", 2024-02-09T17:46:50.032 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1210000000000003e-06 2024-02-09T17:46:50.032 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.032 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.032 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:46:50.033 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.584455+0000", 2024-02-09T17:46:50.033 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4980000000000001e-05 2024-02-09T17:46:50.033 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.033 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.033 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:46:50.034 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.584464+0000", 2024-02-09T17:46:50.034 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8009999999999993e-06 2024-02-09T17:46:50.034 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.034 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.034 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:46:50.034 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.584927+0000", 2024-02-09T17:46:50.034 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00046321200000000001 2024-02-09T17:46:50.034 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.034 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.034 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:46:50.034 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.585045+0000", 2024-02-09T17:46:50.034 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000117528 2024-02-09T17:46:50.034 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.035 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.035 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,2,7", 2024-02-09T17:46:50.035 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.585122+0000", 2024-02-09T17:46:50.035 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7447999999999994e-05 2024-02-09T17:46:50.035 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.035 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.035 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:46:50.035 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.594348+0000", 2024-02-09T17:46:50.035 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0092264579999999999 2024-02-09T17:46:50.035 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.035 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.035 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:46:50.035 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.594361+0000", 2024-02-09T17:46:50.036 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2779000000000001e-05 2024-02-09T17:46:50.036 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.036 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.036 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:46:50.036 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:03.628471+0000", 2024-02-09T17:46:50.036 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0341101669999999 2024-02-09T17:46:50.036 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.036 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.036 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:46:50.036 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:03.628516+0000", 2024-02-09T17:46:50.036 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4267999999999997e-05 2024-02-09T17:46:50.036 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:50.036 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:46:50.036 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:50.037 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.037 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.037 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:222276 4.6 4:6cb938f4:::benchmark_data_smithi179_103809_object84810:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-02-09T17:46:50.037 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:39:02.616343+0000", 2024-02-09T17:46:50.037 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 467.36877812400002, 2024-02-09T17:46:50.037 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0185018139999999, 2024-02-09T17:46:50.037 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:46:50.037 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:46:50.037 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:46:50.037 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:46:50.037 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:46:50.037 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 222276 2024-02-09T17:46:50.037 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.037 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:46:50.038 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.038 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:46:50.038 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.616343+0000", 2024-02-09T17:46:50.038 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:50.038 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.038 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.038 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:46:50.038 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.616343+0000", 2024-02-09T17:46:50.038 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:50.038 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.038 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.038 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:46:50.038 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.616344+0000", 2024-02-09T17:46:50.039 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.299e-06 2024-02-09T17:46:50.039 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.039 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.039 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:46:50.039 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.616349+0000", 2024-02-09T17:46:50.039 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7759999999999997e-06 2024-02-09T17:46:50.039 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.039 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.039 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:46:50.039 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.616361+0000", 2024-02-09T17:46:50.039 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2628e-05 2024-02-09T17:46:50.039 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.039 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.039 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:46:50.040 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.616369+0000", 2024-02-09T17:46:50.040 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.9049999999999997e-06 2024-02-09T17:46:50.040 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.040 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.040 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:46:50.040 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.616431+0000", 2024-02-09T17:46:50.040 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2003999999999998e-05 2024-02-09T17:46:50.040 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.040 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.040 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:46:50.040 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.616443+0000", 2024-02-09T17:46:50.040 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1503e-05 2024-02-09T17:46:50.040 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.041 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.041 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:46:50.041 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:03.628548+0000", 2024-02-09T17:46:50.041 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0121055539999999 2024-02-09T17:46:50.041 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.041 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.041 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:46:50.041 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:03.628687+0000", 2024-02-09T17:46:50.041 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000138675 2024-02-09T17:46:50.041 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.041 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.041 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,7", 2024-02-09T17:46:50.041 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:03.628832+0000", 2024-02-09T17:46:50.042 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000144967 2024-02-09T17:46:50.042 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.042 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.042 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:46:50.042 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:03.634518+0000", 2024-02-09T17:46:50.042 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0056860340000000004 2024-02-09T17:46:50.042 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.042 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.042 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:46:50.042 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:03.634577+0000", 2024-02-09T17:46:50.042 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9253999999999999e-05 2024-02-09T17:46:50.042 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.042 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.042 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:46:50.042 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:03.634726+0000", 2024-02-09T17:46:50.043 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014919 2024-02-09T17:46:50.043 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.043 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.043 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:46:50.043 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:03.634790+0000", 2024-02-09T17:46:50.043 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3972000000000003e-05 2024-02-09T17:46:50.043 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.043 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.043 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:46:50.043 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:03.634844+0000", 2024-02-09T17:46:50.043 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4052999999999999e-05 2024-02-09T17:46:50.043 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:50.043 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:46:50.043 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:50.044 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.044 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.044 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:232555 4.6 4:64fb2779:::benchmark_data_smithi179_103809_object95089:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e414)", 2024-02-09T17:46:50.044 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:39:23.613144+0000", 2024-02-09T17:46:50.044 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 446.37197647199997, 2024-02-09T17:46:50.044 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.108887961, 2024-02-09T17:46:50.044 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:46:50.044 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:46:50.044 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:46:50.045 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:46:50.045 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:46:50.045 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 232555 2024-02-09T17:46:50.045 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.045 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:46:50.045 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.045 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:46:50.045 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.613144+0000", 2024-02-09T17:46:50.045 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:50.045 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.046 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.046 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:46:50.046 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.613144+0000", 2024-02-09T17:46:50.046 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:50.046 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.046 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.046 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:46:50.046 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.613146+0000", 2024-02-09T17:46:50.046 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.482e-06 2024-02-09T17:46:50.046 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.046 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.046 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:46:50.046 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.613151+0000", 2024-02-09T17:46:50.047 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5690000000000004e-06 2024-02-09T17:46:50.047 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.047 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.047 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:46:50.047 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.613164+0000", 2024-02-09T17:46:50.047 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2639999999999999e-05 2024-02-09T17:46:50.047 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.047 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.047 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:46:50.047 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.613173+0000", 2024-02-09T17:46:50.047 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7339999999999997e-06 2024-02-09T17:46:50.047 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.047 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.047 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:46:50.048 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.613235+0000", 2024-02-09T17:46:50.048 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1983000000000006e-05 2024-02-09T17:46:50.048 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.048 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.048 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:46:50.048 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.613243+0000", 2024-02-09T17:46:50.048 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5019999999999992e-06 2024-02-09T17:46:50.048 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.048 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.048 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:46:50.048 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:24.714532+0000", 2024-02-09T17:46:50.048 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1012887840000001 2024-02-09T17:46:50.048 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.048 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.049 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:46:50.049 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:24.714633+0000", 2024-02-09T17:46:50.049 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000100679 2024-02-09T17:46:50.049 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.049 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.049 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,2,7", 2024-02-09T17:46:50.049 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:24.714722+0000", 2024-02-09T17:46:50.049 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.9495999999999998e-05 2024-02-09T17:46:50.049 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.049 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.049 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:46:50.049 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:24.718696+0000", 2024-02-09T17:46:50.049 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0039740690000000002 2024-02-09T17:46:50.049 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.049 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.050 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:46:50.050 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:24.718751+0000", 2024-02-09T17:46:50.050 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5046000000000001e-05 2024-02-09T17:46:50.050 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.050 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.050 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:46:50.050 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:24.718762+0000", 2024-02-09T17:46:50.050 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0944000000000001e-05 2024-02-09T17:46:50.050 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.051 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.051 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:46:50.051 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:24.721911+0000", 2024-02-09T17:46:50.051 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0031492439999999998 2024-02-09T17:46:50.051 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.051 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.051 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:46:50.051 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:24.721950+0000", 2024-02-09T17:46:50.051 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8436000000000003e-05 2024-02-09T17:46:50.051 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.051 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.051 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:46:50.051 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:24.722032+0000", 2024-02-09T17:46:50.051 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2353000000000002e-05 2024-02-09T17:46:50.052 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:50.052 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:46:50.052 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:50.052 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.052 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.052 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:232577 4.6 4:62b4537a:::benchmark_data_smithi179_103809_object95111:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e414)", 2024-02-09T17:46:50.052 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:39:23.646998+0000", 2024-02-09T17:46:50.052 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 446.33812279400001, 2024-02-09T17:46:50.052 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0815207609999999, 2024-02-09T17:46:50.052 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:46:50.052 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:46:50.052 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:46:50.052 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:46:50.053 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:46:50.053 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 232577 2024-02-09T17:46:50.053 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.053 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:46:50.053 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.053 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:46:50.053 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.646998+0000", 2024-02-09T17:46:50.053 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:50.053 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.053 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.053 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:46:50.053 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.646998+0000", 2024-02-09T17:46:50.053 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:50.053 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.053 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.054 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:46:50.054 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.647000+0000", 2024-02-09T17:46:50.054 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0880000000000002e-06 2024-02-09T17:46:50.054 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.054 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.054 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:46:50.054 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.647007+0000", 2024-02-09T17:46:50.054 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1069999999999999e-06 2024-02-09T17:46:50.054 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.054 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.054 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:46:50.054 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.647024+0000", 2024-02-09T17:46:50.054 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6980999999999999e-05 2024-02-09T17:46:50.054 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.054 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.055 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:46:50.055 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.647033+0000", 2024-02-09T17:46:50.055 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7029999999999995e-06 2024-02-09T17:46:50.055 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.055 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.055 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:46:50.055 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.647096+0000", 2024-02-09T17:46:50.055 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3133999999999996e-05 2024-02-09T17:46:50.055 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.055 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.055 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:46:50.055 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:23.647104+0000", 2024-02-09T17:46:50.055 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8350000000000004e-06 2024-02-09T17:46:50.055 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.055 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.056 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:46:50.056 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:24.718794+0000", 2024-02-09T17:46:50.056 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0716905809999999 2024-02-09T17:46:50.056 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.056 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.056 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:46:50.056 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:24.718908+0000", 2024-02-09T17:46:50.056 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000113445 2024-02-09T17:46:50.056 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.056 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.056 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,2,7", 2024-02-09T17:46:50.056 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:24.718990+0000", 2024-02-09T17:46:50.056 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2490999999999993e-05 2024-02-09T17:46:50.056 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.057 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.057 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:46:50.057 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:24.722068+0000", 2024-02-09T17:46:50.057 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0030779370000000002 2024-02-09T17:46:50.057 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.057 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.057 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:46:50.057 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:24.722083+0000", 2024-02-09T17:46:50.057 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4841000000000001e-05 2024-02-09T17:46:50.057 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.057 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.057 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:46:50.057 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:24.722481+0000", 2024-02-09T17:46:50.057 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00039761799999999997 2024-02-09T17:46:50.057 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.058 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.058 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:46:50.058 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:24.728406+0000", 2024-02-09T17:46:50.058 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.005924922 2024-02-09T17:46:50.058 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.058 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.058 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:46:50.058 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:24.728445+0000", 2024-02-09T17:46:50.058 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9184999999999998e-05 2024-02-09T17:46:50.058 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.058 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.058 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:46:50.058 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:24.728519+0000", 2024-02-09T17:46:50.058 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.3893000000000006e-05 2024-02-09T17:46:50.058 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:50.059 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:46:50.059 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:50.059 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.059 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.059 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:254694 4.6 4:6527b3da:::benchmark_data_smithi179_103809_object117228:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-02-09T17:46:50.059 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:39:59.167869+0000", 2024-02-09T17:46:50.059 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 410.817252059, 2024-02-09T17:46:50.059 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.22151003899999999, 2024-02-09T17:46:50.059 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:46:50.059 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:46:50.059 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:46:50.059 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:46:50.059 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:46:50.059 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 254694 2024-02-09T17:46:50.060 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.060 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:46:50.060 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.060 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:46:50.060 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:59.167869+0000", 2024-02-09T17:46:50.060 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:50.060 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.060 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.060 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:46:50.060 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:59.167869+0000", 2024-02-09T17:46:50.060 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:50.060 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.060 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.060 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:46:50.061 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:59.167870+0000", 2024-02-09T17:46:50.061 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4499999999999996e-07 2024-02-09T17:46:50.061 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.061 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.061 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:46:50.061 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:59.167874+0000", 2024-02-09T17:46:50.061 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7979999999999996e-06 2024-02-09T17:46:50.061 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.061 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.061 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:46:50.061 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:59.167887+0000", 2024-02-09T17:46:50.061 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3128e-05 2024-02-09T17:46:50.061 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.061 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.062 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:46:50.062 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:59.167893+0000", 2024-02-09T17:46:50.062 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9549999999999997e-06 2024-02-09T17:46:50.062 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.062 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.062 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:46:50.062 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:59.167947+0000", 2024-02-09T17:46:50.062 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3109000000000003e-05 2024-02-09T17:46:50.062 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.062 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.062 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:46:50.062 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:59.168029+0000", 2024-02-09T17:46:50.062 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2936999999999996e-05 2024-02-09T17:46:50.062 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.062 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.063 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,7", 2024-02-09T17:46:50.063 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:59.168102+0000", 2024-02-09T17:46:50.063 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2973999999999996e-05 2024-02-09T17:46:50.063 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.063 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.063 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:46:50.063 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:59.172176+0000", 2024-02-09T17:46:50.063 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0040739620000000004 2024-02-09T17:46:50.063 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.063 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.064 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:46:50.064 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:59.172433+0000", 2024-02-09T17:46:50.064 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00025656499999999999 2024-02-09T17:46:50.064 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.064 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.064 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:46:50.064 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:59.389314+0000", 2024-02-09T17:46:50.064 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.21688068099999999 2024-02-09T17:46:50.064 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.064 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.064 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:46:50.064 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:59.389350+0000", 2024-02-09T17:46:50.064 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6356000000000002e-05 2024-02-09T17:46:50.064 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.065 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.065 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:46:50.065 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:59.389379+0000", 2024-02-09T17:46:50.065 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.8728999999999999e-05 2024-02-09T17:46:50.065 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:50.065 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:46:50.065 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:50.065 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.065 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.065 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:254719 4.6 4:69a51ad5:::benchmark_data_smithi179_103809_object117253:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-02-09T17:46:50.065 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:39:59.206510+0000", 2024-02-09T17:46:50.065 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 410.77861084099999, 2024-02-09T17:46:50.065 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.191680395, 2024-02-09T17:46:50.065 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:46:50.066 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:46:50.066 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:46:50.066 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:46:50.066 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:46:50.066 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 254719 2024-02-09T17:46:50.066 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.066 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:46:50.066 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.066 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:46:50.066 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:59.206510+0000", 2024-02-09T17:46:50.066 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:50.066 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.066 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.066 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:46:50.067 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:59.206510+0000", 2024-02-09T17:46:50.067 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:50.067 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.067 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.067 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:46:50.067 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:59.206512+0000", 2024-02-09T17:46:50.067 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.683e-06 2024-02-09T17:46:50.067 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.067 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.067 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:46:50.067 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:59.206519+0000", 2024-02-09T17:46:50.067 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5399999999999998e-06 2024-02-09T17:46:50.067 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.067 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.068 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:46:50.068 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:59.206534+0000", 2024-02-09T17:46:50.068 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5074e-05 2024-02-09T17:46:50.068 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.068 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.068 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:46:50.068 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:59.206543+0000", 2024-02-09T17:46:50.068 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.217e-06 2024-02-09T17:46:50.068 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.068 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.068 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:46:50.068 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:59.206609+0000", 2024-02-09T17:46:50.068 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6092000000000004e-05 2024-02-09T17:46:50.068 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.069 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.069 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:46:50.069 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:59.206726+0000", 2024-02-09T17:46:50.069 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000116203 2024-02-09T17:46:50.069 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.069 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.069 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,7", 2024-02-09T17:46:50.069 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:59.206832+0000", 2024-02-09T17:46:50.069 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000105824 2024-02-09T17:46:50.069 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.069 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.069 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:46:50.069 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:59.214607+0000", 2024-02-09T17:46:50.069 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.007775288 2024-02-09T17:46:50.070 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.070 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.070 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:46:50.070 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:59.214690+0000", 2024-02-09T17:46:50.070 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2930999999999996e-05 2024-02-09T17:46:50.070 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.070 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.070 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:46:50.070 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:59.398115+0000", 2024-02-09T17:46:50.070 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.183424843 2024-02-09T17:46:50.070 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.070 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.070 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:46:50.070 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:59.398161+0000", 2024-02-09T17:46:50.070 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6857000000000002e-05 2024-02-09T17:46:50.071 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.071 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.071 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:46:50.071 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:59.398190+0000", 2024-02-09T17:46:50.071 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.8843000000000002e-05 2024-02-09T17:46:50.071 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:50.071 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:46:50.071 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:50.071 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.071 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.071 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:254720 4.6 4:6119b182:::benchmark_data_smithi179_103809_object117254:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-02-09T17:46:50.071 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:39:59.206919+0000", 2024-02-09T17:46:50.071 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 410.778202185, 2024-02-09T17:46:50.071 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.19475119900000001, 2024-02-09T17:46:50.072 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:46:50.072 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:46:50.072 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:46:50.072 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:46:50.072 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:46:50.072 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 254720 2024-02-09T17:46:50.072 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.072 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:46:50.072 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.072 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:46:50.072 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:59.206919+0000", 2024-02-09T17:46:50.072 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:50.072 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.072 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.073 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:46:50.073 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:59.206919+0000", 2024-02-09T17:46:50.073 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:50.073 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.073 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.073 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:46:50.073 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:59.206919+0000", 2024-02-09T17:46:50.073 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4600000000000003e-07 2024-02-09T17:46:50.073 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.073 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.073 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:46:50.073 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:59.206924+0000", 2024-02-09T17:46:50.073 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5329999999999996e-06 2024-02-09T17:46:50.073 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.074 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.074 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:46:50.074 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:59.206934+0000", 2024-02-09T17:46:50.074 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0141e-05 2024-02-09T17:46:50.074 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.074 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.074 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:46:50.074 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:59.206948+0000", 2024-02-09T17:46:50.074 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3593e-05 2024-02-09T17:46:50.074 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.074 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.074 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:46:50.074 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:59.210768+0000", 2024-02-09T17:46:50.075 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0038207050000000002 2024-02-09T17:46:50.075 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.075 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.075 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:46:50.075 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:59.210882+0000", 2024-02-09T17:46:50.075 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000113287 2024-02-09T17:46:50.075 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.075 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.075 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,7", 2024-02-09T17:46:50.075 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:59.210952+0000", 2024-02-09T17:46:50.075 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9991000000000001e-05 2024-02-09T17:46:50.075 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.075 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.075 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:46:50.076 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:59.214668+0000", 2024-02-09T17:46:50.076 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.003716352 2024-02-09T17:46:50.076 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.076 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.076 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:46:50.076 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:59.214744+0000", 2024-02-09T17:46:50.076 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6229000000000005e-05 2024-02-09T17:46:50.076 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.076 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.076 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:46:50.076 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:59.401560+0000", 2024-02-09T17:46:50.076 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.18681567099999999 2024-02-09T17:46:50.076 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.077 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.077 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:46:50.077 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:59.401624+0000", 2024-02-09T17:46:50.077 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4399000000000005e-05 2024-02-09T17:46:50.077 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.077 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.077 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:46:50.077 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:59.401670+0000", 2024-02-09T17:46:50.077 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5451999999999998e-05 2024-02-09T17:46:50.077 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:50.077 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:46:50.077 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:50.077 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.077 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.078 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:223741 5.b 5:dd39b1e4:::benchmark_data_smithi179_115199_object94835:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e549)", 2024-02-09T17:46:50.078 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:44:34.397477+0000", 2024-02-09T17:46:50.078 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 135.58764375800001, 2024-02-09T17:46:50.078 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.21936540700000001, 2024-02-09T17:46:50.078 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:46:50.078 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:46:50.078 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:46:50.078 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:46:50.078 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:46:50.078 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 223741 2024-02-09T17:46:50.078 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.078 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:46:50.078 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.079 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:46:50.079 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.397477+0000", 2024-02-09T17:46:50.079 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:50.079 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.079 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.079 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:46:50.079 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.397477+0000", 2024-02-09T17:46:50.079 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:50.079 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.079 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.079 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:46:50.079 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.397479+0000", 2024-02-09T17:46:50.079 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.533e-06 2024-02-09T17:46:50.079 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.079 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.080 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:46:50.080 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.397487+0000", 2024-02-09T17:46:50.080 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7879999999999995e-06 2024-02-09T17:46:50.080 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.080 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.080 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:46:50.080 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.397499+0000", 2024-02-09T17:46:50.080 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.145e-05 2024-02-09T17:46:50.080 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.080 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.080 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:46:50.080 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.397505+0000", 2024-02-09T17:46:50.080 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8069999999999996e-06 2024-02-09T17:46:50.080 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.081 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.081 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:46:50.081 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.397659+0000", 2024-02-09T17:46:50.081 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015451000000000001 2024-02-09T17:46:50.081 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.081 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.081 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:46:50.081 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.397788+0000", 2024-02-09T17:46:50.081 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000128489 2024-02-09T17:46:50.081 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.081 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.081 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,5", 2024-02-09T17:46:50.081 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.397920+0000", 2024-02-09T17:46:50.081 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000132256 2024-02-09T17:46:50.081 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.082 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.082 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:46:50.082 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.402065+0000", 2024-02-09T17:46:50.082 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0041455950000000002 2024-02-09T17:46:50.082 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.082 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.082 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:46:50.082 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.405452+0000", 2024-02-09T17:46:50.082 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0033867390000000002 2024-02-09T17:46:50.082 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.082 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.082 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:46:50.082 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.616745+0000", 2024-02-09T17:46:50.082 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.211293335 2024-02-09T17:46:50.082 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.082 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.083 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:46:50.083 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.616804+0000", 2024-02-09T17:46:50.083 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8258999999999999e-05 2024-02-09T17:46:50.083 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.083 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.083 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:46:50.083 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.616842+0000", 2024-02-09T17:46:50.083 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8646000000000002e-05 2024-02-09T17:46:50.083 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:50.083 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:46:50.083 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:50.083 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.083 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.084 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:223764 5.b 5:df9ccd7d:::benchmark_data_smithi179_115199_object94858:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e549)", 2024-02-09T17:46:50.084 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:44:34.423172+0000", 2024-02-09T17:46:50.084 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 135.56194870600001, 2024-02-09T17:46:50.084 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.19854297700000001, 2024-02-09T17:46:50.084 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:46:50.084 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:46:50.084 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:46:50.084 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:46:50.084 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:46:50.084 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 223764 2024-02-09T17:46:50.084 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.084 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:46:50.084 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.084 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:46:50.085 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.423172+0000", 2024-02-09T17:46:50.085 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:50.085 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.085 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.085 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:46:50.085 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.423172+0000", 2024-02-09T17:46:50.085 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:50.085 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.085 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.085 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:46:50.085 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.423174+0000", 2024-02-09T17:46:50.085 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5710000000000001e-06 2024-02-09T17:46:50.085 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.085 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.086 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:46:50.086 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.423180+0000", 2024-02-09T17:46:50.086 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2249999999999997e-06 2024-02-09T17:46:50.086 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.086 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.086 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:46:50.086 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.423196+0000", 2024-02-09T17:46:50.086 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6317999999999999e-05 2024-02-09T17:46:50.086 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.086 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.086 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:46:50.086 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.423204+0000", 2024-02-09T17:46:50.086 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.0830000000000004e-06 2024-02-09T17:46:50.086 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.086 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.087 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:46:50.087 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.423271+0000", 2024-02-09T17:46:50.087 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7057000000000005e-05 2024-02-09T17:46:50.087 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.087 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.087 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:46:50.087 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.423406+0000", 2024-02-09T17:46:50.087 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013521099999999999 2024-02-09T17:46:50.087 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.087 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.087 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,5", 2024-02-09T17:46:50.087 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.423530+0000", 2024-02-09T17:46:50.087 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012351699999999999 2024-02-09T17:46:50.087 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.088 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.088 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:46:50.088 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.429019+0000", 2024-02-09T17:46:50.088 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0054890659999999999 2024-02-09T17:46:50.088 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.088 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.088 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:46:50.088 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.447367+0000", 2024-02-09T17:46:50.088 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.018348258999999999 2024-02-09T17:46:50.088 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.088 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.088 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:46:50.088 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.621623+0000", 2024-02-09T17:46:50.089 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.17425532399999999 2024-02-09T17:46:50.089 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.089 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.089 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:46:50.089 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.621678+0000", 2024-02-09T17:46:50.089 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5282999999999999e-05 2024-02-09T17:46:50.089 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.089 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.089 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:46:50.089 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.621715+0000", 2024-02-09T17:46:50.089 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7063000000000001e-05 2024-02-09T17:46:50.089 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:50.089 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:46:50.089 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:50.090 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.090 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.090 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:223776 5.b 5:d35b7d02:::benchmark_data_smithi179_115199_object94870:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e549)", 2024-02-09T17:46:50.090 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:44:34.442196+0000", 2024-02-09T17:46:50.090 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 135.542924744, 2024-02-09T17:46:50.090 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.17965571799999999, 2024-02-09T17:46:50.090 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:46:50.090 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:46:50.090 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:46:50.090 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:46:50.090 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:46:50.090 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 223776 2024-02-09T17:46:50.090 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.090 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:46:50.091 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.091 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:46:50.091 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.442196+0000", 2024-02-09T17:46:50.091 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:50.091 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.091 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.091 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:46:50.091 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.442196+0000", 2024-02-09T17:46:50.091 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:50.091 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.091 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.091 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:46:50.091 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.442197+0000", 2024-02-09T17:46:50.091 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.401e-06 2024-02-09T17:46:50.091 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.092 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.092 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:46:50.092 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.442203+0000", 2024-02-09T17:46:50.092 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9460000000000003e-06 2024-02-09T17:46:50.092 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.092 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.092 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:46:50.092 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.442217+0000", 2024-02-09T17:46:50.092 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4126e-05 2024-02-09T17:46:50.092 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.092 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.092 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:46:50.092 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.442226+0000", 2024-02-09T17:46:50.093 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8170000000000005e-06 2024-02-09T17:46:50.093 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.093 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.093 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:46:50.093 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.442442+0000", 2024-02-09T17:46:50.093 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000215725 2024-02-09T17:46:50.093 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.093 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.093 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:46:50.093 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.442575+0000", 2024-02-09T17:46:50.093 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013259 2024-02-09T17:46:50.093 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.093 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.093 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,5", 2024-02-09T17:46:50.094 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.442706+0000", 2024-02-09T17:46:50.094 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000131462 2024-02-09T17:46:50.094 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.094 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.094 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:46:50.094 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.447456+0000", 2024-02-09T17:46:50.094 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.004750074 2024-02-09T17:46:50.094 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.094 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.094 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:46:50.094 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.449641+0000", 2024-02-09T17:46:50.094 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002184666 2024-02-09T17:46:50.094 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.094 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.094 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:46:50.094 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.621763+0000", 2024-02-09T17:46:50.095 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.17212183 2024-02-09T17:46:50.095 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.095 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.095 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:46:50.095 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.621813+0000", 2024-02-09T17:46:50.095 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0871999999999997e-05 2024-02-09T17:46:50.095 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.095 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.095 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:46:50.095 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.621852+0000", 2024-02-09T17:46:50.095 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8209e-05 2024-02-09T17:46:50.095 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:50.095 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:46:50.095 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:50.096 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.096 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.096 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:223783 5.b 5:de86604e:::benchmark_data_smithi179_115199_object94877:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e549)", 2024-02-09T17:46:50.096 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:44:34.449948+0000", 2024-02-09T17:46:50.096 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 135.535172391, 2024-02-09T17:46:50.096 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.180929642, 2024-02-09T17:46:50.096 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:46:50.096 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:46:50.096 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:46:50.096 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:46:50.096 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:46:50.096 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 223783 2024-02-09T17:46:50.096 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.096 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:46:50.096 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.097 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:46:50.097 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.449948+0000", 2024-02-09T17:46:50.097 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:50.097 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.097 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.097 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:46:50.097 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.449948+0000", 2024-02-09T17:46:50.097 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:50.097 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.097 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.097 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:46:50.097 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.449950+0000", 2024-02-09T17:46:50.097 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.283e-06 2024-02-09T17:46:50.097 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.098 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.098 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:46:50.098 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.449955+0000", 2024-02-09T17:46:50.098 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9470000000000002e-06 2024-02-09T17:46:50.098 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.098 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.098 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:46:50.098 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.449968+0000", 2024-02-09T17:46:50.098 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3047000000000001e-05 2024-02-09T17:46:50.098 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.098 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.098 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:46:50.098 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.449976+0000", 2024-02-09T17:46:50.098 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.9929999999999995e-06 2024-02-09T17:46:50.098 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.098 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.099 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:46:50.099 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.450027+0000", 2024-02-09T17:46:50.099 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1328000000000001e-05 2024-02-09T17:46:50.099 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.099 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.099 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:46:50.099 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.450149+0000", 2024-02-09T17:46:50.099 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012234 2024-02-09T17:46:50.099 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.099 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.099 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,5", 2024-02-09T17:46:50.099 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.450278+0000", 2024-02-09T17:46:50.099 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012913200000000001 2024-02-09T17:46:50.099 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.100 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.100 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:46:50.100 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.455485+0000", 2024-02-09T17:46:50.100 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0052070450000000004 2024-02-09T17:46:50.100 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.100 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.100 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:46:50.100 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.465494+0000", 2024-02-09T17:46:50.100 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010008507999999999 2024-02-09T17:46:50.100 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.100 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.100 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:46:50.100 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.630768+0000", 2024-02-09T17:46:50.100 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.165274221 2024-02-09T17:46:50.101 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.101 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.101 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:46:50.101 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.630836+0000", 2024-02-09T17:46:50.101 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8299000000000005e-05 2024-02-09T17:46:50.101 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.101 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.101 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:46:50.101 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:34.630878+0000", 2024-02-09T17:46:50.101 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1498999999999997e-05 2024-02-09T17:46:50.101 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:50.101 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:46:50.101 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:50.101 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.102 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.102 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25837.0:65222 6.9 6:9908b5f0:::benchmark_data_smithi179_126206_object65221:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e589)", 2024-02-09T17:46:50.102 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:46:22.853266+0000", 2024-02-09T17:46:50.102 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 27.131855133999998, 2024-02-09T17:46:50.102 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.040839465999999998, 2024-02-09T17:46:50.102 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:46:50.102 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:46:50.102 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:46:50.102 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25837", 2024-02-09T17:46:50.102 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:46:50.102 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 65222 2024-02-09T17:46:50.102 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.102 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:46:50.102 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.103 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:46:50.103 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:22.853266+0000", 2024-02-09T17:46:50.103 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:50.103 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.103 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.103 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:46:50.103 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:22.853266+0000", 2024-02-09T17:46:50.103 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:50.103 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.103 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.103 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:46:50.103 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:22.853267+0000", 2024-02-09T17:46:50.103 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6220000000000001e-06 2024-02-09T17:46:50.103 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.103 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.104 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:46:50.104 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:22.853289+0000", 2024-02-09T17:46:50.104 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.19e-05 2024-02-09T17:46:50.104 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.104 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.104 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:46:50.104 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:22.853303+0000", 2024-02-09T17:46:50.104 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4178999999999999e-05 2024-02-09T17:46:50.104 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.104 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.104 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:46:50.104 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:22.853311+0000", 2024-02-09T17:46:50.104 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5909999999999998e-06 2024-02-09T17:46:50.105 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.105 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.105 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:46:50.105 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:22.853366+0000", 2024-02-09T17:46:50.105 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4870000000000002e-05 2024-02-09T17:46:50.105 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.105 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.105 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:46:50.105 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:22.853490+0000", 2024-02-09T17:46:50.105 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012436500000000001 2024-02-09T17:46:50.105 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.105 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.105 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,8", 2024-02-09T17:46:50.105 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:22.853623+0000", 2024-02-09T17:46:50.105 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013327 2024-02-09T17:46:50.106 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.106 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.106 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:46:50.106 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:22.869988+0000", 2024-02-09T17:46:50.106 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.016364664000000001 2024-02-09T17:46:50.106 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.106 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.106 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:46:50.106 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:22.888536+0000", 2024-02-09T17:46:50.106 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.018547536 2024-02-09T17:46:50.106 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.106 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.106 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:46:50.107 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:22.894008+0000", 2024-02-09T17:46:50.107 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0054724600000000002 2024-02-09T17:46:50.107 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.107 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.107 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:46:50.107 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:22.894061+0000", 2024-02-09T17:46:50.107 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2809000000000002e-05 2024-02-09T17:46:50.107 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.107 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.107 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:46:50.107 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:22.894105+0000", 2024-02-09T17:46:50.107 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4199999999999997e-05 2024-02-09T17:46:50.107 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:50.107 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:46:50.108 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:50.108 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.108 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.108 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25837.0:91349 6.9 6:90d3d560:::benchmark_data_smithi179_126206_object91348:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-02-09T17:46:50.108 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:46:42.413261+0000", 2024-02-09T17:46:50.108 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 7.571859495, 2024-02-09T17:46:50.108 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.037483649000000001, 2024-02-09T17:46:50.108 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:46:50.108 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:46:50.108 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:46:50.108 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25837", 2024-02-09T17:46:50.108 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:46:50.108 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 91349 2024-02-09T17:46:50.108 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.109 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:46:50.109 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.109 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:46:50.109 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.413261+0000", 2024-02-09T17:46:50.109 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:50.109 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.109 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.109 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:46:50.109 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.413261+0000", 2024-02-09T17:46:50.109 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:50.109 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.109 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.109 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:46:50.109 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.413262+0000", 2024-02-09T17:46:50.110 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8799999999999998e-07 2024-02-09T17:46:50.110 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.110 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.110 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:46:50.110 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.413310+0000", 2024-02-09T17:46:50.110 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8253e-05 2024-02-09T17:46:50.110 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.110 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.110 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:46:50.110 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.413317+0000", 2024-02-09T17:46:50.110 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.3359999999999997e-06 2024-02-09T17:46:50.110 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.110 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.110 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:46:50.111 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.413323+0000", 2024-02-09T17:46:50.111 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9959999999999999e-06 2024-02-09T17:46:50.111 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.111 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.111 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:46:50.111 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.432098+0000", 2024-02-09T17:46:50.111 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.018774644 2024-02-09T17:46:50.111 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.111 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.111 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:46:50.111 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.432247+0000", 2024-02-09T17:46:50.111 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014869500000000001 2024-02-09T17:46:50.111 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.111 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.112 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,8", 2024-02-09T17:46:50.112 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.432390+0000", 2024-02-09T17:46:50.112 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014383399999999999 2024-02-09T17:46:50.112 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.112 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.112 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:46:50.112 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.438626+0000", 2024-02-09T17:46:50.112 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0062356620000000003 2024-02-09T17:46:50.112 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.112 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.112 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:46:50.112 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.438668+0000", 2024-02-09T17:46:50.112 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1467999999999997e-05 2024-02-09T17:46:50.112 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.113 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.113 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:46:50.113 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.450647+0000", 2024-02-09T17:46:50.113 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011979906 2024-02-09T17:46:50.113 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.113 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.113 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:46:50.113 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.450705+0000", 2024-02-09T17:46:50.113 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7606999999999999e-05 2024-02-09T17:46:50.113 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.113 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.113 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:46:50.113 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.450745+0000", 2024-02-09T17:46:50.113 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9860000000000001e-05 2024-02-09T17:46:50.113 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:50.114 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:46:50.114 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:50.114 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.114 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.114 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25837.0:91352 6.9 6:9122948c:::benchmark_data_smithi179_126206_object91351:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-02-09T17:46:50.114 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:46:42.413332+0000", 2024-02-09T17:46:50.114 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 7.5717886290000003, 2024-02-09T17:46:50.114 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.038947330000000002, 2024-02-09T17:46:50.114 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:46:50.115 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:46:50.115 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:46:50.115 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25837", 2024-02-09T17:46:50.115 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:46:50.115 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 91352 2024-02-09T17:46:50.115 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.115 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:46:50.115 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.115 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:46:50.115 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.413332+0000", 2024-02-09T17:46:50.115 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:50.115 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.115 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.115 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:46:50.116 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.413332+0000", 2024-02-09T17:46:50.116 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:50.116 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.116 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.116 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:46:50.116 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.413333+0000", 2024-02-09T17:46:50.116 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5999999999999999e-07 2024-02-09T17:46:50.116 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.116 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.116 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:46:50.116 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.413376+0000", 2024-02-09T17:46:50.116 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3009000000000001e-05 2024-02-09T17:46:50.116 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.116 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.117 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:46:50.117 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.413384+0000", 2024-02-09T17:46:50.117 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2619999999999998e-06 2024-02-09T17:46:50.117 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.117 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.117 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:46:50.117 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.413390+0000", 2024-02-09T17:46:50.117 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2979999999999997e-06 2024-02-09T17:46:50.117 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.117 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.117 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:46:50.117 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.435629+0000", 2024-02-09T17:46:50.117 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.022238759 2024-02-09T17:46:50.117 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.118 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.118 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:46:50.118 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.435755+0000", 2024-02-09T17:46:50.118 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012633800000000001 2024-02-09T17:46:50.118 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.118 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.118 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,8", 2024-02-09T17:46:50.118 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.435888+0000", 2024-02-09T17:46:50.118 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013268199999999999 2024-02-09T17:46:50.118 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.118 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.118 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:46:50.118 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.438936+0000", 2024-02-09T17:46:50.118 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.003048065 2024-02-09T17:46:50.118 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.119 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.119 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:46:50.119 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.439191+0000", 2024-02-09T17:46:50.119 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00025520300000000002 2024-02-09T17:46:50.119 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.119 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.119 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:46:50.119 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.452191+0000", 2024-02-09T17:46:50.119 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013000065 2024-02-09T17:46:50.119 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.119 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.119 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:46:50.119 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.452245+0000", 2024-02-09T17:46:50.119 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3878999999999997e-05 2024-02-09T17:46:50.119 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:50.120 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:50.120 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:46:50.120 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:42.452279+0000", 2024-02-09T17:46:50.120 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4310000000000002e-05 2024-02-09T17:46:50.120 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:50.120 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:46:50.120 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:50.120 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:50.120 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:46:50.120 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:46:50.121 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 58 16 99199 99183 106.867 66.75 0.0150241 0.00935347 2024-02-09T17:46:50.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:50 smithi099 ceph-mon[39420]: pgmap v1504: 47 pgs: 1 peering, 46 active+clean; 5.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 73 MiB/s wr, 1.18k op/s; 64 KiB/s, 0 objects/s recovering 2024-02-09T17:46:50.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:50 smithi099 ceph-mon[39420]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:46:50.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:50 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:46:50.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:50 smithi099 ceph-mon[39420]: osdmap e603: 12 total, 12 up, 12 in 2024-02-09T17:46:50.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:50 smithi179 ceph-mon[43975]: pgmap v1504: 47 pgs: 1 peering, 46 active+clean; 5.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 73 MiB/s wr, 1.18k op/s; 64 KiB/s, 0 objects/s recovering 2024-02-09T17:46:50.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:50 smithi179 ceph-mon[43975]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:46:50.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:50 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:46:50.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:50 smithi179 ceph-mon[43975]: osdmap e603: 12 total, 12 up, 12 in 2024-02-09T17:46:50.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:50 smithi162 ceph-mon[44304]: pgmap v1504: 47 pgs: 1 peering, 46 active+clean; 5.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 73 MiB/s wr, 1.18k op/s; 64 KiB/s, 0 objects/s recovering 2024-02-09T17:46:50.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:50 smithi162 ceph-mon[44304]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:46:50.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:50 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:46:50.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:50 smithi162 ceph-mon[44304]: osdmap e603: 12 total, 12 up, 12 in 2024-02-09T17:46:51.102 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 59 16 100375 100359 106.301 73.5 0.00678552 0.00940353 2024-02-09T17:46:51.324 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.10 dump_ops_in_flight 2024-02-09T17:46:51.622 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:46:51.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:51.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:46:51.714 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:46:51.731 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:46:51.784 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:51.824 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:46:51.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:51.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:46:51.944 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:46:51.989 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:52.102 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:2024-02-09T17:46:52.101605+0000 min lat: 0.00111008 max lat: 1.03613 avg lat: 0.00944687 2024-02-09T17:46:52.102 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-09T17:46:52.102 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 60 16 101608 101592 105.814 77.0625 0.00607424 0.00944687 2024-02-09T17:46:52.228 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:52 smithi179 ceph-mon[43975]: pgmap v1506: 47 pgs: 1 peering, 46 active+clean; 5.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 50 MiB/s wr, 804 op/s; 64 KiB/s, 0 objects/s recovering 2024-02-09T17:46:52.228 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:52 smithi179 ceph-mon[43975]: osdmap e604: 12 total, 12 up, 12 in 2024-02-09T17:46:52.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:52 smithi099 ceph-mon[39420]: pgmap v1506: 47 pgs: 1 peering, 46 active+clean; 5.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 50 MiB/s wr, 804 op/s; 64 KiB/s, 0 objects/s recovering 2024-02-09T17:46:52.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:52 smithi099 ceph-mon[39420]: osdmap e604: 12 total, 12 up, 12 in 2024-02-09T17:46:52.262 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:46:52.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:52.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:46:52.374 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:46:52.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:52 smithi162 ceph-mon[44304]: pgmap v1506: 47 pgs: 1 peering, 46 active+clean; 5.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 50 MiB/s wr, 804 op/s; 64 KiB/s, 0 objects/s recovering 2024-02-09T17:46:52.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:52 smithi162 ceph-mon[44304]: osdmap e604: 12 total, 12 up, 12 in 2024-02-09T17:46:52.495 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:52.697 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] 2024-02-09T17:46:52.697 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 1 1 2024-02-09T17:46:53.102 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 61 16 102790 102774 105.29 73.875 0.00862309 0.00949351 2024-02-09T17:46:53.151 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:46:53.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:53.157 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:46:53.258 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:46:53.289 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:53.324 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:46:53.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:53.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:46:53.430 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:46:53.472 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:53.510 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:46:53.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:53.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:46:53.613 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:46:53.664 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:53.684 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:46:53.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:53.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:46:53.789 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:46:53.860 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:54.102 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 62 16 103845 103829 104.655 65.9375 0.0149543 0.00955076 2024-02-09T17:46:54.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:54 smithi099 ceph-mon[39420]: pgmap v1508: 47 pgs: 47 active+clean; 6.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 868 B/s rd, 80 MiB/s wr, 1.28k op/s; 53 KiB/s, 0 objects/s recovering 2024-02-09T17:46:54.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:54 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:46:54.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:54 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:46:54.294 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:46:54.294 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-02-09T17:46:54.294 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:54.294 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25837.0:104078 6.f 6:f70fe5c8:::benchmark_data_smithi179_126206_object104077:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e605)", 2024-02-09T17:46:54.294 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:46:54.288705+0000", 2024-02-09T17:46:54.294 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.0042683069999999998, 2024-02-09T17:46:54.294 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0042948709999999996, 2024-02-09T17:46:54.294 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:46:54.294 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "waiting for sub ops", 2024-02-09T17:46:54.294 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:46:54.295 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25837", 2024-02-09T17:46:54.295 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:46:54.295 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 104078 2024-02-09T17:46:54.295 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:54.295 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:46:54.295 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:54.295 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:46:54.295 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:54.288705+0000", 2024-02-09T17:46:54.295 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:54.295 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:54.295 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:54.295 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:46:54.295 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:54.288705+0000", 2024-02-09T17:46:54.295 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:54.295 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:54.296 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:54.296 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:46:54.296 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:54.288707+0000", 2024-02-09T17:46:54.296 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7239999999999999e-06 2024-02-09T17:46:54.296 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:54.296 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:54.296 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:46:54.296 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:54.288728+0000", 2024-02-09T17:46:54.296 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1010999999999999e-05 2024-02-09T17:46:54.296 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:54.296 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:54.296 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:46:54.296 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:54.288740+0000", 2024-02-09T17:46:54.296 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2388000000000001e-05 2024-02-09T17:46:54.297 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:54.297 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:54.297 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:46:54.297 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:54.288748+0000", 2024-02-09T17:46:54.297 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5059999999999998e-06 2024-02-09T17:46:54.297 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:54.297 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:54.297 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:46:54.297 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:54.288794+0000", 2024-02-09T17:46:54.297 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6038999999999997e-05 2024-02-09T17:46:54.297 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:54.297 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:54.297 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:46:54.297 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:54.288932+0000", 2024-02-09T17:46:54.297 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013830599999999999 2024-02-09T17:46:54.298 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:54.298 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:54.298 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,9", 2024-02-09T17:46:54.298 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:54.289072+0000", 2024-02-09T17:46:54.298 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014022199999999999 2024-02-09T17:46:54.298 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:54.298 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:54.298 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:46:54.298 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:54.292913+0000", 2024-02-09T17:46:54.298 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.003840829 2024-02-09T17:46:54.298 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:54.298 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:46:54.298 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:54.298 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:54.298 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:54.299 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.25837.0:104070 6.1 e605/576 6:81d0a273:::benchmark_data_smithi179_126206_object104069:head v 605'6507, mlcod=605'6505)", 2024-02-09T17:46:54.299 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:46:54.290618+0000", 2024-02-09T17:46:54.299 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.0023559369999999998, 2024-02-09T17:46:54.299 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0024361180000000001, 2024-02-09T17:46:54.299 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:46:54.299 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-02-09T17:46:54.299 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:46:54.299 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:54.299 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:46:54.299 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:54.290618+0000", 2024-02-09T17:46:54.299 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:54.299 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:54.299 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:54.299 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:46:54.299 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:54.290618+0000", 2024-02-09T17:46:54.299 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:46:54.300 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:54.300 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:54.300 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:46:54.300 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:54.290618+0000", 2024-02-09T17:46:54.300 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6199999999999997e-07 2024-02-09T17:46:54.300 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:54.300 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:54.300 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:46:54.300 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:54.290639+0000", 2024-02-09T17:46:54.300 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0696e-05 2024-02-09T17:46:54.300 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:54.300 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:54.300 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:46:54.300 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:54.290651+0000", 2024-02-09T17:46:54.301 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2109e-05 2024-02-09T17:46:54.301 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:54.301 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:54.301 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:46:54.301 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:54.290658+0000", 2024-02-09T17:46:54.301 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7830000000000001e-06 2024-02-09T17:46:54.301 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:54.301 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:54.301 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:46:54.301 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:54.290728+0000", 2024-02-09T17:46:54.301 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9618999999999999e-05 2024-02-09T17:46:54.301 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:46:54.301 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:46:54.301 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:46:54.301 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:46:54.290746+0000", 2024-02-09T17:46:54.301 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8474e-05 2024-02-09T17:46:54.302 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:54.302 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:46:54.302 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:54.302 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:46:54.302 INFO:teuthology.orchestra.run.smithi179.stdout: ], 2024-02-09T17:46:54.302 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 2 2024-02-09T17:46:54.302 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:46:54.316 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset noscrub 2024-02-09T17:46:54.317 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:54 smithi179 ceph-mon[43975]: pgmap v1508: 47 pgs: 47 active+clean; 6.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 868 B/s rd, 80 MiB/s wr, 1.28k op/s; 53 KiB/s, 0 objects/s recovering 2024-02-09T17:46:54.317 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:54 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:46:54.317 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:54 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:46:54.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:54 smithi162 ceph-mon[44304]: pgmap v1508: 47 pgs: 47 active+clean; 6.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 868 B/s rd, 80 MiB/s wr, 1.28k op/s; 53 KiB/s, 0 objects/s recovering 2024-02-09T17:46:54.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:54 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:46:54.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:54 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:46:54.522 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:46:54.537 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:46:54.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:54.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:46:54.643 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:46:54.674 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:55.102 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 63 16 104996 104980 104.136 71.9375 0.00574731 0.00959817 2024-02-09T17:46:55.326 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:55 smithi099 ceph-mon[39420]: osdmap e605: 12 total, 12 up, 12 in 2024-02-09T17:46:55.371 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:46:55.412 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:46:55.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:55 smithi179 ceph-mon[43975]: osdmap e605: 12 total, 12 up, 12 in 2024-02-09T17:46:55.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:55 smithi162 ceph-mon[44304]: osdmap e605: 12 total, 12 up, 12 in 2024-02-09T17:46:55.451 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:46:55.493 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:46:55.533 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:46:55.569 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:46:55.584 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.10 dump_blocked_ops 2024-02-09T17:46:55.609 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:46:55.630 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:46:55.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:55.635 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:46:55.649 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:46:55.689 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:46:55.725 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:46:55.736 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:46:55.754 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:46:55.771 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:55.797 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:46:55.798 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:46:55.799 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:46:55.799 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:46:55.799 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:46:55.799 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:46:55.799 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:46:55.799 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:46:55.799 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:46:55.799 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:46:55.799 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:46:55.799 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:46:55.799 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:46:55.802 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:46:55.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:55.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:46:55.907 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:46:55.952 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:55.986 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:46:55.994 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:46:56.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:56.001 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:46:56.090 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:56 smithi162 ceph-mon[44304]: pgmap v1510: 47 pgs: 47 active+clean; 6.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 995 op/s 2024-02-09T17:46:56.090 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:56 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1995350387' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:46:56.101 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:46:56.102 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 64 16 106118 106102 103.604 70.125 0.0214179 0.00964793 2024-02-09T17:46:56.153 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:56.187 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:46:56.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:56.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:46:56.296 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:46:56.309 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:56 smithi179 ceph-mon[43975]: pgmap v1510: 47 pgs: 47 active+clean; 6.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 995 op/s 2024-02-09T17:46:56.310 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:56 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1995350387' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:46:56.328 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:56.352 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:46:56.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:56.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:46:56.457 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:46:56.485 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:56 smithi099 ceph-mon[39420]: pgmap v1510: 47 pgs: 47 active+clean; 6.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 995 op/s 2024-02-09T17:46:56.485 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:56 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1995350387' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:46:56.488 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:56.728 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:46:56.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:56.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:46:56.846 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:46:56.898 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:56.931 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:46:56.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:56.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:46:57.038 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:46:57.081 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:57.102 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 65 16 107134 107118 102.987 63.5 0.00859127 0.00970615 2024-02-09T17:46:57.115 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:46:57.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:57.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:46:57.223 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:46:57.277 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:57.387 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:46:57.395 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:46:57.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:57.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:46:57.506 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:46:57.544 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:57.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:57 smithi179 ceph-mon[43975]: osdmap e606: 12 total, 12 up, 12 in 2024-02-09T17:46:57.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:57 smithi179 ceph-mon[43975]: pgmap v1512: 47 pgs: 47 active+clean; 6.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-02-09T17:46:57.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:57 smithi162 ceph-mon[44304]: osdmap e606: 12 total, 12 up, 12 in 2024-02-09T17:46:57.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:57 smithi162 ceph-mon[44304]: pgmap v1512: 47 pgs: 47 active+clean; 6.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-02-09T17:46:57.961 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:57 smithi099 ceph-mon[39420]: osdmap e606: 12 total, 12 up, 12 in 2024-02-09T17:46:57.961 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:57 smithi099 ceph-mon[39420]: pgmap v1512: 47 pgs: 47 active+clean; 6.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-02-09T17:46:58.102 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 66 16 108272 108256 102.505 71.125 0.00856277 0.0097516 2024-02-09T17:46:58.410 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:46:58.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:58.417 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:46:58.518 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:46:58.569 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:58.646 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:46:58.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:58.651 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:46:58.753 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:46:58.781 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:46:58.788 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:58.811 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T17:46:58.811 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:46:58.830 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:46:58.839 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:58.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:46:58.941 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:46:58.949 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.1 to 1 (10000) 2024-02-09T17:46:58.970 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:59.095 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:46:59.098 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:58 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3156528207' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-09T17:46:59.098 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:58 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-09T17:46:59.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:59.099 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:46:59.102 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 67 16 109337 109321 101.968 66.5625 0.00466336 0.00980278 2024-02-09T17:46:59.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:58 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3156528207' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-09T17:46:59.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:58 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-09T17:46:59.200 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:46:59.233 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:59.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:58 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3156528207' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-09T17:46:59.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:58 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-09T17:46:59.592 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:46:59.595 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.10 dump_historic_ops 2024-02-09T17:46:59.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:46:59.598 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:46:59.698 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:46:59.743 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:46:59.936 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:59 smithi099 ceph-mon[39420]: pgmap v1513: 47 pgs: 47 active+clean; 6.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-02-09T17:46:59.936 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:59 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-09T17:46:59.936 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:46:59 smithi099 ceph-mon[39420]: osdmap e607: 12 total, 12 up, 12 in 2024-02-09T17:46:59.980 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:47:00.102 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 68 16 110463 110447 101.503 70.375 0.00620461 0.00984749 2024-02-09T17:47:00.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:59 smithi179 ceph-mon[43975]: pgmap v1513: 47 pgs: 47 active+clean; 6.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-02-09T17:47:00.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:59 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-09T17:47:00.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:46:59 smithi179 ceph-mon[43975]: osdmap e607: 12 total, 12 up, 12 in 2024-02-09T17:47:00.432 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:59 smithi162 ceph-mon[44304]: pgmap v1513: 47 pgs: 47 active+clean; 6.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-02-09T17:47:00.432 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:59 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-09T17:47:00.432 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:46:59 smithi162 ceph-mon[44304]: osdmap e607: 12 total, 12 up, 12 in 2024-02-09T17:47:01.103 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 69 16 111492 111476 100.964 64.3125 0.00825661 0.00989761 2024-02-09T17:47:01.794 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 3 1 2024-02-09T17:47:02.015 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:47:02.047 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:47:02.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:02.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:47:02.059 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:01 smithi099 ceph-mon[39420]: pgmap v1515: 47 pgs: 47 active+clean; 6.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 961 op/s 2024-02-09T17:47:02.059 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:01 smithi099 ceph-mon[39420]: osdmap e608: 12 total, 12 up, 12 in 2024-02-09T17:47:02.103 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 70 16 112558 112542 100.474 66.625 0.00776017 0.00994829 2024-02-09T17:47:02.160 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:47:02.161 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:01 smithi179 ceph-mon[43975]: pgmap v1515: 47 pgs: 47 active+clean; 6.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 961 op/s 2024-02-09T17:47:02.161 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:01 smithi179 ceph-mon[43975]: osdmap e608: 12 total, 12 up, 12 in 2024-02-09T17:47:02.221 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:02.298 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:47:02.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:02.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:47:02.403 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:47:02.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:01 smithi162 ceph-mon[44304]: pgmap v1515: 47 pgs: 47 active+clean; 6.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 961 op/s 2024-02-09T17:47:02.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:01 smithi162 ceph-mon[44304]: osdmap e608: 12 total, 12 up, 12 in 2024-02-09T17:47:02.450 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:02.476 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T17:47:02.476 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:47:02.480 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:47:02.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:02.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:47:02.587 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:47:02.640 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:02.667 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:47:02.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:02.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:47:02.774 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:47:02.844 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:02.982 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:47:02.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:02.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:47:03.093 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:47:03.102 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 71 16 113602 113586 99.9775 65.25 0.0226344 0.00999848 2024-02-09T17:47:03.128 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:03.760 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.11 dump_ops_in_flight 2024-02-09T17:47:03.780 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:47:03.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:03.784 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:47:03.885 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:47:03.951 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:04.103 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 72 16 114608 114592 99.4621 62.875 0.00602228 0.0100491 2024-02-09T17:47:04.123 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:47:04.126 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:47:04.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:04.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:47:04.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:03 smithi179 ceph-mon[43975]: pgmap v1517: 47 pgs: 47 active+clean; 6.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.35k op/s 2024-02-09T17:47:04.230 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:47:04.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:03 smithi099 ceph-mon[39420]: pgmap v1517: 47 pgs: 47 active+clean; 6.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.35k op/s 2024-02-09T17:47:04.260 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:04.283 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:47:04.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:04.291 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:47:04.393 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:47:04.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:03 smithi162 ceph-mon[44304]: pgmap v1517: 47 pgs: 47 active+clean; 6.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.35k op/s 2024-02-09T17:47:04.428 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:04.459 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:47:04.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:04.467 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:47:04.568 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:47:04.637 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:04.778 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:47:04.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:04.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:47:04.782 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:47:04.882 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:47:04.923 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:04.996 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is unset 2024-02-09T17:47:05.103 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 73 16 115653 115637 98.9942 65.3125 0.00503251 0.0100975 2024-02-09T17:47:05.189 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:04 smithi099 ceph-mon[39420]: from='client.16221 172.21.15.99:0/2945839649' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:47:05.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:04 smithi179 ceph-mon[43975]: from='client.16221 172.21.15.99:0/2945839649' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:47:05.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:04 smithi162 ceph-mon[44304]: from='client.16221 172.21.15.99:0/2945839649' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:47:05.514 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:47:05.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:05.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:47:05.628 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:47:05.680 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:05.707 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:47:05.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:05.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:47:05.817 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:47:05.874 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:05.991 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:05 smithi162 ceph-mon[44304]: pgmap v1518: 47 pgs: 47 active+clean; 6.9 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.13k op/s 2024-02-09T17:47:05.991 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:05 smithi162 ceph-mon[44304]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:47:05.991 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:05 smithi162 ceph-mon[44304]: from='client.16221 172.21.15.99:0/2945839649' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:47:05.991 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:05 smithi162 ceph-mon[44304]: osdmap e609: 12 total, 12 up, 12 in 2024-02-09T17:47:06.103 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 74 15 116718 116703 98.5568 66.625 0.0180325 0.0101424 2024-02-09T17:47:06.160 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T17:47:06.160 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:47:06.161 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:05 smithi099 ceph-mon[39420]: pgmap v1518: 47 pgs: 47 active+clean; 6.9 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.13k op/s 2024-02-09T17:47:06.161 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:05 smithi099 ceph-mon[39420]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:47:06.161 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:05 smithi099 ceph-mon[39420]: from='client.16221 172.21.15.99:0/2945839649' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:47:06.161 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:05 smithi099 ceph-mon[39420]: osdmap e609: 12 total, 12 up, 12 in 2024-02-09T17:47:06.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:05 smithi179 ceph-mon[43975]: pgmap v1518: 47 pgs: 47 active+clean; 6.9 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.13k op/s 2024-02-09T17:47:06.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:05 smithi179 ceph-mon[43975]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:47:06.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:05 smithi179 ceph-mon[43975]: from='client.16221 172.21.15.99:0/2945839649' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:47:06.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:05 smithi179 ceph-mon[43975]: osdmap e609: 12 total, 12 up, 12 in 2024-02-09T17:47:07.103 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 75 16 117799 117783 98.1426 67.5 0.0122888 0.0101851 2024-02-09T17:47:07.450 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:47:07.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:07.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:47:07.561 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:47:07.587 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.11 dump_blocked_ops 2024-02-09T17:47:07.621 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:07.627 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:47:07.656 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:47:07.700 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:47:07.738 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:47:07.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:07 smithi099 ceph-mon[39420]: 1.14 scrub starts 2024-02-09T17:47:07.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:07 smithi099 ceph-mon[39420]: 1.14 scrub ok 2024-02-09T17:47:07.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:07 smithi099 ceph-mon[39420]: osdmap e610: 12 total, 12 up, 12 in 2024-02-09T17:47:07.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:07 smithi099 ceph-mon[39420]: pgmap v1521: 47 pgs: 47 active+clean; 7.0 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-02-09T17:47:07.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:07 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1606325008' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:47:07.804 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:47:07.852 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:47:07.881 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:47:07.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:07.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:47:07.892 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:47:07.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:07 smithi179 ceph-mon[43975]: 1.14 scrub starts 2024-02-09T17:47:07.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:07 smithi179 ceph-mon[43975]: 1.14 scrub ok 2024-02-09T17:47:07.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:07 smithi179 ceph-mon[43975]: osdmap e610: 12 total, 12 up, 12 in 2024-02-09T17:47:07.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:07 smithi179 ceph-mon[43975]: pgmap v1521: 47 pgs: 47 active+clean; 7.0 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-02-09T17:47:07.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:07 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1606325008' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:47:07.922 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:47:07.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:07 smithi162 ceph-mon[44304]: 1.14 scrub starts 2024-02-09T17:47:07.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:07 smithi162 ceph-mon[44304]: 1.14 scrub ok 2024-02-09T17:47:07.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:07 smithi162 ceph-mon[44304]: osdmap e610: 12 total, 12 up, 12 in 2024-02-09T17:47:07.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:07 smithi162 ceph-mon[44304]: pgmap v1521: 47 pgs: 47 active+clean; 7.0 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-02-09T17:47:07.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:07 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1606325008' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:47:07.949 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:47:07.990 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:47:07.991 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:47:08.040 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:47:08.041 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:08.070 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:47:08.079 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:47:08.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:08.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:47:08.103 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 76 16 118770 118754 97.6497 60.6875 0.011526 0.0102354 2024-02-09T17:47:08.124 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:47:08.131 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:47:08.131 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:47:08.131 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:47:08.131 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:47:08.131 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:47:08.131 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:47:08.131 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:47:08.132 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:47:08.132 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:47:08.132 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:47:08.132 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:47:08.132 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:47:08.185 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:47:08.236 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:08.512 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:47:08.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:08.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:47:08.619 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:47:08.660 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:08.675 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.3 to 1 (10000) 2024-02-09T17:47:08.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:08 smithi179 ceph-mon[43975]: 6.5 scrub starts 2024-02-09T17:47:08.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:08 smithi179 ceph-mon[43975]: 1.c scrub starts 2024-02-09T17:47:08.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:08 smithi179 ceph-mon[43975]: 1.c scrub ok 2024-02-09T17:47:08.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:08 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/680957465' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-09T17:47:08.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:08 smithi162 ceph-mon[44304]: 6.5 scrub starts 2024-02-09T17:47:08.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:08 smithi162 ceph-mon[44304]: 1.c scrub starts 2024-02-09T17:47:08.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:08 smithi162 ceph-mon[44304]: 1.c scrub ok 2024-02-09T17:47:08.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:08 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/680957465' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-09T17:47:08.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:08 smithi099 ceph-mon[39420]: 6.5 scrub starts 2024-02-09T17:47:08.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:08 smithi099 ceph-mon[39420]: 1.c scrub starts 2024-02-09T17:47:08.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:08 smithi099 ceph-mon[39420]: 1.c scrub ok 2024-02-09T17:47:08.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:08 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/680957465' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-09T17:47:09.103 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 77 16 119747 119731 97.1745 61.0625 0.0228406 0.0102865 2024-02-09T17:47:09.313 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:47:09.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:09.319 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:47:09.420 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:47:09.475 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:09.485 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-09T17:47:09.568 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:47:09.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:09.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:47:09.677 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:47:09.704 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:09.730 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:47:09.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:09.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:47:09.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:09 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/680957465' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-09T17:47:09.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:09 smithi099 ceph-mon[39420]: osdmap e611: 12 total, 12 up, 12 in 2024-02-09T17:47:09.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:09 smithi099 ceph-mon[39420]: 6.6 scrub starts 2024-02-09T17:47:09.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:09 smithi099 ceph-mon[39420]: pgmap v1523: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 7.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 927 op/s 2024-02-09T17:47:09.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:09 smithi099 ceph-mon[39420]: 6.5 scrub ok 2024-02-09T17:47:09.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:09 smithi099 ceph-mon[39420]: 1.1d scrub starts 2024-02-09T17:47:09.836 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:47:09.898 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:09.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:09 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/680957465' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-09T17:47:09.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:09 smithi162 ceph-mon[44304]: osdmap e611: 12 total, 12 up, 12 in 2024-02-09T17:47:09.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:09 smithi162 ceph-mon[44304]: 6.6 scrub starts 2024-02-09T17:47:09.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:09 smithi162 ceph-mon[44304]: pgmap v1523: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 7.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 927 op/s 2024-02-09T17:47:09.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:09 smithi162 ceph-mon[44304]: 6.5 scrub ok 2024-02-09T17:47:09.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:09 smithi162 ceph-mon[44304]: 1.1d scrub starts 2024-02-09T17:47:10.037 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:47:10.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:10.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:47:10.074 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 4 1 2024-02-09T17:47:10.103 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 78 16 120640 120624 96.6441 55.8125 0.00464881 0.0103427 2024-02-09T17:47:10.142 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:47:10.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:09 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/680957465' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-09T17:47:10.170 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:09 smithi179 ceph-mon[43975]: osdmap e611: 12 total, 12 up, 12 in 2024-02-09T17:47:10.170 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:09 smithi179 ceph-mon[43975]: 6.6 scrub starts 2024-02-09T17:47:10.170 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:09 smithi179 ceph-mon[43975]: pgmap v1523: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 7.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 927 op/s 2024-02-09T17:47:10.170 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:09 smithi179 ceph-mon[43975]: 6.5 scrub ok 2024-02-09T17:47:10.170 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:09 smithi179 ceph-mon[43975]: 1.1d scrub starts 2024-02-09T17:47:10.172 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:10.180 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:47:10.274 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:47:10.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:10.281 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:47:10.382 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:47:10.436 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:10.466 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:47:10.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:10.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:47:10.574 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:47:10.613 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:10.641 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:47:10.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:10.646 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:47:10.666 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:47:10.666 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-02-09T17:47:10.666 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-02-09T17:47:10.666 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-02-09T17:47:10.666 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:47:10.747 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:47:10.782 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:10.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:10 smithi179 ceph-mon[43975]: 1.b scrub starts 2024-02-09T17:47:10.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:10 smithi179 ceph-mon[43975]: 1.b scrub ok 2024-02-09T17:47:10.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:10 smithi179 ceph-mon[43975]: 1.1d scrub ok 2024-02-09T17:47:10.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:10 smithi179 ceph-mon[43975]: 1.8 scrub starts 2024-02-09T17:47:10.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:10 smithi179 ceph-mon[43975]: 1.8 scrub ok 2024-02-09T17:47:10.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:10 smithi179 ceph-mon[43975]: 6.9 scrub starts 2024-02-09T17:47:10.960 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:10 smithi099 ceph-mon[39420]: 1.b scrub starts 2024-02-09T17:47:10.960 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:10 smithi099 ceph-mon[39420]: 1.b scrub ok 2024-02-09T17:47:10.960 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:10 smithi099 ceph-mon[39420]: 1.1d scrub ok 2024-02-09T17:47:10.960 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:10 smithi099 ceph-mon[39420]: 1.8 scrub starts 2024-02-09T17:47:10.960 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:10 smithi099 ceph-mon[39420]: 1.8 scrub ok 2024-02-09T17:47:10.960 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:10 smithi099 ceph-mon[39420]: 6.9 scrub starts 2024-02-09T17:47:11.103 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 79 16 121499 121483 96.1003 53.6875 0.0177047 0.0104011 2024-02-09T17:47:11.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:10 smithi162 ceph-mon[44304]: 1.b scrub starts 2024-02-09T17:47:11.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:10 smithi162 ceph-mon[44304]: 1.b scrub ok 2024-02-09T17:47:11.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:10 smithi162 ceph-mon[44304]: 1.1d scrub ok 2024-02-09T17:47:11.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:10 smithi162 ceph-mon[44304]: 1.8 scrub starts 2024-02-09T17:47:11.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:10 smithi162 ceph-mon[44304]: 1.8 scrub ok 2024-02-09T17:47:11.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:10 smithi162 ceph-mon[44304]: 6.9 scrub starts 2024-02-09T17:47:11.205 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:47:11.723 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:11 smithi099 ceph-mon[39420]: 1.10 scrub starts 2024-02-09T17:47:11.723 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:11 smithi099 ceph-mon[39420]: 1.10 scrub ok 2024-02-09T17:47:11.723 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:11 smithi099 ceph-mon[39420]: 6.f scrub starts 2024-02-09T17:47:11.723 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:11 smithi099 ceph-mon[39420]: pgmap v1524: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 7.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 925 op/s 2024-02-09T17:47:11.723 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:11 smithi099 ceph-mon[39420]: 6.2 scrub starts 2024-02-09T17:47:11.724 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:11 smithi099 ceph-mon[39420]: 6.d scrub starts 2024-02-09T17:47:11.724 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:11 smithi099 ceph-mon[39420]: osdmap e612: 12 total, 12 up, 12 in 2024-02-09T17:47:11.739 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:47:11.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:11.747 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:47:11.848 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:47:11.924 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:12.009 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:11 smithi179 ceph-mon[43975]: 1.10 scrub starts 2024-02-09T17:47:12.009 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:11 smithi179 ceph-mon[43975]: 1.10 scrub ok 2024-02-09T17:47:12.009 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:11 smithi179 ceph-mon[43975]: 6.f scrub starts 2024-02-09T17:47:12.009 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:11 smithi179 ceph-mon[43975]: pgmap v1524: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 7.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 925 op/s 2024-02-09T17:47:12.010 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:11 smithi179 ceph-mon[43975]: 6.2 scrub starts 2024-02-09T17:47:12.010 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:11 smithi179 ceph-mon[43975]: 6.d scrub starts 2024-02-09T17:47:12.010 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:11 smithi179 ceph-mon[43975]: osdmap e612: 12 total, 12 up, 12 in 2024-02-09T17:47:12.104 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:2024-02-09T17:47:12.103298+0000 min lat: 0.00111008 max lat: 1.03613 avg lat: 0.0104617 2024-02-09T17:47:12.104 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-09T17:47:12.104 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 80 16 122329 122313 95.5475 51.875 0.0202857 0.0104617 2024-02-09T17:47:12.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:11 smithi162 ceph-mon[44304]: 1.10 scrub starts 2024-02-09T17:47:12.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:11 smithi162 ceph-mon[44304]: 1.10 scrub ok 2024-02-09T17:47:12.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:11 smithi162 ceph-mon[44304]: 6.f scrub starts 2024-02-09T17:47:12.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:11 smithi162 ceph-mon[44304]: pgmap v1524: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 7.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 925 op/s 2024-02-09T17:47:12.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:11 smithi162 ceph-mon[44304]: 6.2 scrub starts 2024-02-09T17:47:12.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:11 smithi162 ceph-mon[44304]: 6.d scrub starts 2024-02-09T17:47:12.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:11 smithi162 ceph-mon[44304]: osdmap e612: 12 total, 12 up, 12 in 2024-02-09T17:47:12.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:12 smithi099 ceph-mon[39420]: 6.6 scrub ok 2024-02-09T17:47:12.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:12 smithi099 ceph-mon[39420]: 1.1e scrub starts 2024-02-09T17:47:12.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:12 smithi099 ceph-mon[39420]: 1.1e scrub ok 2024-02-09T17:47:12.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:12 smithi099 ceph-mon[39420]: 1.0 scrub starts 2024-02-09T17:47:12.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:12 smithi099 ceph-mon[39420]: 1.0 scrub ok 2024-02-09T17:47:12.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:12 smithi099 ceph-mon[39420]: 6.0 scrub starts 2024-02-09T17:47:12.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:12 smithi099 ceph-mon[39420]: 6.9 scrub ok 2024-02-09T17:47:12.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:12 smithi099 ceph-mon[39420]: 6.4 scrub starts 2024-02-09T17:47:13.105 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 81 16 122980 122964 94.8701 40.6875 0.0164456 0.0105367 2024-02-09T17:47:13.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:12 smithi179 ceph-mon[43975]: 6.6 scrub ok 2024-02-09T17:47:13.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:12 smithi179 ceph-mon[43975]: 1.1e scrub starts 2024-02-09T17:47:13.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:12 smithi179 ceph-mon[43975]: 1.1e scrub ok 2024-02-09T17:47:13.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:12 smithi179 ceph-mon[43975]: 1.0 scrub starts 2024-02-09T17:47:13.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:12 smithi179 ceph-mon[43975]: 1.0 scrub ok 2024-02-09T17:47:13.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:12 smithi179 ceph-mon[43975]: 6.0 scrub starts 2024-02-09T17:47:13.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:12 smithi179 ceph-mon[43975]: 6.9 scrub ok 2024-02-09T17:47:13.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:12 smithi179 ceph-mon[43975]: 6.4 scrub starts 2024-02-09T17:47:13.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:12 smithi162 ceph-mon[44304]: 6.6 scrub ok 2024-02-09T17:47:13.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:12 smithi162 ceph-mon[44304]: 1.1e scrub starts 2024-02-09T17:47:13.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:12 smithi162 ceph-mon[44304]: 1.1e scrub ok 2024-02-09T17:47:13.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:12 smithi162 ceph-mon[44304]: 1.0 scrub starts 2024-02-09T17:47:13.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:12 smithi162 ceph-mon[44304]: 1.0 scrub ok 2024-02-09T17:47:13.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:12 smithi162 ceph-mon[44304]: 6.0 scrub starts 2024-02-09T17:47:13.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:12 smithi162 ceph-mon[44304]: 6.9 scrub ok 2024-02-09T17:47:13.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:12 smithi162 ceph-mon[44304]: 6.4 scrub starts 2024-02-09T17:47:13.395 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.11 dump_historic_ops 2024-02-09T17:47:13.718 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:47:14.010 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:13 smithi179 ceph-mon[43975]: 6.3 scrub starts 2024-02-09T17:47:14.010 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:13 smithi179 ceph-mon[43975]: 6.f scrub ok 2024-02-09T17:47:14.010 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:13 smithi179 ceph-mon[43975]: 6.e scrub starts 2024-02-09T17:47:14.011 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:13 smithi179 ceph-mon[43975]: 6.b scrub starts 2024-02-09T17:47:14.011 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:13 smithi179 ceph-mon[43975]: pgmap v1526: 47 pgs: 3 active+clean+scrubbing, 44 active+clean; 7.4 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-02-09T17:47:14.011 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:13 smithi179 ceph-mon[43975]: 6.2 scrub ok 2024-02-09T17:47:14.011 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:13 smithi179 ceph-mon[43975]: 1.12 scrub starts 2024-02-09T17:47:14.011 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:13 smithi179 ceph-mon[43975]: 1.12 scrub ok 2024-02-09T17:47:14.104 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 82 16 123676 123660 94.2436 43.5 0.0322678 0.0106056 2024-02-09T17:47:14.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:13 smithi162 ceph-mon[44304]: 6.3 scrub starts 2024-02-09T17:47:14.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:13 smithi162 ceph-mon[44304]: 6.f scrub ok 2024-02-09T17:47:14.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:13 smithi162 ceph-mon[44304]: 6.e scrub starts 2024-02-09T17:47:14.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:13 smithi162 ceph-mon[44304]: 6.b scrub starts 2024-02-09T17:47:14.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:13 smithi162 ceph-mon[44304]: pgmap v1526: 47 pgs: 3 active+clean+scrubbing, 44 active+clean; 7.4 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-02-09T17:47:14.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:13 smithi162 ceph-mon[44304]: 6.2 scrub ok 2024-02-09T17:47:14.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:13 smithi162 ceph-mon[44304]: 1.12 scrub starts 2024-02-09T17:47:14.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:13 smithi162 ceph-mon[44304]: 1.12 scrub ok 2024-02-09T17:47:14.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:13 smithi099 ceph-mon[39420]: 6.3 scrub starts 2024-02-09T17:47:14.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:13 smithi099 ceph-mon[39420]: 6.f scrub ok 2024-02-09T17:47:14.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:13 smithi099 ceph-mon[39420]: 6.e scrub starts 2024-02-09T17:47:14.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:13 smithi099 ceph-mon[39420]: 6.b scrub starts 2024-02-09T17:47:14.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:13 smithi099 ceph-mon[39420]: pgmap v1526: 47 pgs: 3 active+clean+scrubbing, 44 active+clean; 7.4 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-02-09T17:47:14.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:13 smithi099 ceph-mon[39420]: 6.2 scrub ok 2024-02-09T17:47:14.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:13 smithi099 ceph-mon[39420]: 1.12 scrub starts 2024-02-09T17:47:14.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:13 smithi099 ceph-mon[39420]: 1.12 scrub ok 2024-02-09T17:47:15.105 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 83 16 124336 124320 93.6036 41.25 0.0149292 0.0106794 2024-02-09T17:47:15.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:14 smithi179 ceph-mon[43975]: 1.7 scrub starts 2024-02-09T17:47:15.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:14 smithi179 ceph-mon[43975]: 1.7 scrub ok 2024-02-09T17:47:15.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:14 smithi179 ceph-mon[43975]: 6.d scrub ok 2024-02-09T17:47:15.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:14 smithi179 ceph-mon[43975]: 6.3 scrub ok 2024-02-09T17:47:15.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:14 smithi162 ceph-mon[44304]: 1.7 scrub starts 2024-02-09T17:47:15.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:14 smithi162 ceph-mon[44304]: 1.7 scrub ok 2024-02-09T17:47:15.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:14 smithi162 ceph-mon[44304]: 6.d scrub ok 2024-02-09T17:47:15.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:14 smithi162 ceph-mon[44304]: 6.3 scrub ok 2024-02-09T17:47:15.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:14 smithi099 ceph-mon[39420]: 1.7 scrub starts 2024-02-09T17:47:15.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:14 smithi099 ceph-mon[39420]: 1.7 scrub ok 2024-02-09T17:47:15.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:14 smithi099 ceph-mon[39420]: 6.d scrub ok 2024-02-09T17:47:15.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:14 smithi099 ceph-mon[39420]: 6.3 scrub ok 2024-02-09T17:47:15.762 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:47:15.778 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:47:15.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:15.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:47:15.886 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:47:15.954 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:15.978 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:47:16.003 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:47:16.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:16.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:47:16.055 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:15 smithi179 ceph-mon[43975]: 6.a scrub starts 2024-02-09T17:47:16.055 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:15 smithi179 ceph-mon[43975]: 6.0 scrub ok 2024-02-09T17:47:16.055 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:15 smithi179 ceph-mon[43975]: 6.1 scrub starts 2024-02-09T17:47:16.055 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:15 smithi179 ceph-mon[43975]: pgmap v1527: 47 pgs: 4 active+clean+scrubbing, 43 active+clean; 7.5 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s 2024-02-09T17:47:16.055 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:15 smithi179 ceph-mon[43975]: 6.e scrub ok 2024-02-09T17:47:16.055 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:15 smithi179 ceph-mon[43975]: 6.4 scrub ok 2024-02-09T17:47:16.105 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 84 16 124927 124911 92.929 36.9375 0.0169843 0.0107571 2024-02-09T17:47:16.107 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:47:16.136 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:15 smithi162 ceph-mon[44304]: 6.a scrub starts 2024-02-09T17:47:16.136 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:15 smithi162 ceph-mon[44304]: 6.0 scrub ok 2024-02-09T17:47:16.137 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:15 smithi162 ceph-mon[44304]: 6.1 scrub starts 2024-02-09T17:47:16.137 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:15 smithi162 ceph-mon[44304]: pgmap v1527: 47 pgs: 4 active+clean+scrubbing, 43 active+clean; 7.5 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s 2024-02-09T17:47:16.137 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:15 smithi162 ceph-mon[44304]: 6.e scrub ok 2024-02-09T17:47:16.137 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:15 smithi162 ceph-mon[44304]: 6.4 scrub ok 2024-02-09T17:47:16.138 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:16.171 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:47:16.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:16.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:47:16.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:15 smithi099 ceph-mon[39420]: 6.a scrub starts 2024-02-09T17:47:16.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:15 smithi099 ceph-mon[39420]: 6.0 scrub ok 2024-02-09T17:47:16.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:15 smithi099 ceph-mon[39420]: 6.1 scrub starts 2024-02-09T17:47:16.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:15 smithi099 ceph-mon[39420]: pgmap v1527: 47 pgs: 4 active+clean+scrubbing, 43 active+clean; 7.5 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s 2024-02-09T17:47:16.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:15 smithi099 ceph-mon[39420]: 6.e scrub ok 2024-02-09T17:47:16.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:15 smithi099 ceph-mon[39420]: 6.4 scrub ok 2024-02-09T17:47:16.275 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:47:16.331 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:16.774 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:47:16.774 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-02-09T17:47:16.774 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-02-09T17:47:16.774 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-02-09T17:47:16.774 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.775 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:208403 4.8 4:163e1e0f:::benchmark_data_smithi179_103809_object70937:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-02-09T17:47:16.775 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:33.810176+0000", 2024-02-09T17:47:16.775 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 522.958685694, 2024-02-09T17:47:16.775 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2455196100000001, 2024-02-09T17:47:16.775 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:47:16.775 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-02-09T17:47:16.775 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:47:16.775 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:47:16.775 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:47:16.775 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 208403 2024-02-09T17:47:16.775 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.775 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:47:16.775 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.776 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:47:16.776 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.810176+0000", 2024-02-09T17:47:16.776 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:47:16.776 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.776 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.776 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:47:16.776 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.810176+0000", 2024-02-09T17:47:16.776 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:47:16.776 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.776 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.776 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:47:16.776 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.810177+0000", 2024-02-09T17:47:16.776 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.63e-07 2024-02-09T17:47:16.776 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.777 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.777 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:47:16.777 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.810179+0000", 2024-02-09T17:47:16.777 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5000000000000002e-06 2024-02-09T17:47:16.777 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.777 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.777 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:47:16.777 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.810186+0000", 2024-02-09T17:47:16.777 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1650000000000003e-06 2024-02-09T17:47:16.777 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.777 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.777 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:47:16.777 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.810191+0000", 2024-02-09T17:47:16.777 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0359999999999997e-06 2024-02-09T17:47:16.778 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.778 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.778 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:47:16.778 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.810238+0000", 2024-02-09T17:47:16.778 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7083000000000003e-05 2024-02-09T17:47:16.778 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.778 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.778 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:47:16.778 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.810351+0000", 2024-02-09T17:47:16.778 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000112879 2024-02-09T17:47:16.778 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.778 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.779 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 6,7", 2024-02-09T17:47:16.779 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.810458+0000", 2024-02-09T17:47:16.779 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000107941 2024-02-09T17:47:16.779 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.779 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.779 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:47:16.779 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.816423+0000", 2024-02-09T17:47:16.779 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0059641260000000001 2024-02-09T17:47:16.779 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.779 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.780 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:47:16.780 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.821967+0000", 2024-02-09T17:47:16.780 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0055448889999999999 2024-02-09T17:47:16.780 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.780 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.780 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:47:16.780 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.821982+0000", 2024-02-09T17:47:16.780 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4341e-05 2024-02-09T17:47:16.780 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.780 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.780 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:47:16.780 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:34.813273+0000", 2024-02-09T17:47:16.780 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.99129126999999995 2024-02-09T17:47:16.781 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.781 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.781 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:47:16.781 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:34.813286+0000", 2024-02-09T17:47:16.781 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.278e-05 2024-02-09T17:47:16.781 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.781 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.781 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:47:16.781 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.042595+0000", 2024-02-09T17:47:16.781 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.229308955 2024-02-09T17:47:16.781 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.781 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.781 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for missing object", 2024-02-09T17:47:16.781 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.042737+0000", 2024-02-09T17:47:16.782 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014233599999999999 2024-02-09T17:47:16.782 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.782 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.782 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:47:16.782 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.055624+0000", 2024-02-09T17:47:16.782 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.012886556 2024-02-09T17:47:16.782 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.782 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.782 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:47:16.782 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.055696+0000", 2024-02-09T17:47:16.782 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2290000000000001e-05 2024-02-09T17:47:16.782 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:47:16.782 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:47:16.782 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:47:16.783 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.783 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.783 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:208414 4.8 4:1c1fd668:::benchmark_data_smithi179_103809_object70948:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-02-09T17:47:16.783 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:38:33.829970+0000", 2024-02-09T17:47:16.783 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 522.93889187699995, 2024-02-09T17:47:16.783 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2251981070000002, 2024-02-09T17:47:16.783 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:47:16.783 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:47:16.783 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:47:16.783 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:47:16.783 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:47:16.783 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 208414 2024-02-09T17:47:16.783 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.783 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:47:16.783 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.784 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:47:16.784 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.829970+0000", 2024-02-09T17:47:16.784 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:47:16.784 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.784 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.784 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:47:16.784 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.829970+0000", 2024-02-09T17:47:16.784 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:47:16.784 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.784 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.784 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:47:16.784 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.829972+0000", 2024-02-09T17:47:16.784 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.471e-06 2024-02-09T17:47:16.784 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.785 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.785 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:47:16.785 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.829978+0000", 2024-02-09T17:47:16.785 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4840000000000001e-06 2024-02-09T17:47:16.785 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.785 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.785 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:47:16.785 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.829989+0000", 2024-02-09T17:47:16.785 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0446e-05 2024-02-09T17:47:16.785 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.785 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.785 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:47:16.785 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.829995+0000", 2024-02-09T17:47:16.785 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8549999999999996e-06 2024-02-09T17:47:16.786 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.786 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.786 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:47:16.786 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.830040+0000", 2024-02-09T17:47:16.786 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4743999999999997e-05 2024-02-09T17:47:16.786 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.786 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.786 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:47:16.786 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:33.830048+0000", 2024-02-09T17:47:16.786 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7279999999999995e-06 2024-02-09T17:47:16.786 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.786 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.786 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:47:16.786 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:34.813315+0000", 2024-02-09T17:47:16.786 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.98326734000000005 2024-02-09T17:47:16.786 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.787 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.787 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:47:16.787 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:34.813323+0000", 2024-02-09T17:47:16.787 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7919999999999993e-06 2024-02-09T17:47:16.787 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.787 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.787 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:47:16.787 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.043262+0000", 2024-02-09T17:47:16.787 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.229938456 2024-02-09T17:47:16.787 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.787 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.787 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:47:16.787 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.043359+0000", 2024-02-09T17:47:16.787 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7549999999999999e-05 2024-02-09T17:47:16.787 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.788 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.788 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 6,7,9", 2024-02-09T17:47:16.788 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.043666+0000", 2024-02-09T17:47:16.788 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00030711699999999998 2024-02-09T17:47:16.788 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.788 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.788 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.788 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.048474+0000", 2024-02-09T17:47:16.788 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0048078649999999997 2024-02-09T17:47:16.788 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.788 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.788 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:47:16.788 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.048858+0000", 2024-02-09T17:47:16.789 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00038438799999999999 2024-02-09T17:47:16.789 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.789 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.789 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.789 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.051829+0000", 2024-02-09T17:47:16.789 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0029708260000000002 2024-02-09T17:47:16.789 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.789 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.789 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.789 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.054961+0000", 2024-02-09T17:47:16.789 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0031320850000000002 2024-02-09T17:47:16.789 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.789 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.789 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:47:16.789 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.055029+0000", 2024-02-09T17:47:16.790 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8108000000000006e-05 2024-02-09T17:47:16.790 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.790 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.790 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:47:16.790 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:38:36.055168+0000", 2024-02-09T17:47:16.790 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013885199999999999 2024-02-09T17:47:16.790 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:47:16.790 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:47:16.790 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:47:16.790 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.790 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.790 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:222266 4.8 4:1579dd15:::benchmark_data_smithi179_103809_object84800:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-02-09T17:47:16.790 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:39:02.600778+0000", 2024-02-09T17:47:16.791 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 494.168083895, 2024-02-09T17:47:16.791 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.171074366, 2024-02-09T17:47:16.791 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:47:16.791 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:47:16.791 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:47:16.791 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:47:16.791 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:47:16.791 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 222266 2024-02-09T17:47:16.791 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.791 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:47:16.791 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.791 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:47:16.791 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.600778+0000", 2024-02-09T17:47:16.792 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:47:16.792 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.792 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.792 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:47:16.792 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.600778+0000", 2024-02-09T17:47:16.792 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:47:16.792 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.792 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.792 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:47:16.792 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.600780+0000", 2024-02-09T17:47:16.792 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.764e-06 2024-02-09T17:47:16.792 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.792 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.793 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:47:16.793 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.600784+0000", 2024-02-09T17:47:16.793 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1389999999999997e-06 2024-02-09T17:47:16.793 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.793 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.793 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:47:16.793 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.600796+0000", 2024-02-09T17:47:16.793 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1697e-05 2024-02-09T17:47:16.793 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.793 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.793 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:47:16.793 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.600803+0000", 2024-02-09T17:47:16.793 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2350000000000002e-06 2024-02-09T17:47:16.794 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.794 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.794 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:47:16.794 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.600850+0000", 2024-02-09T17:47:16.794 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6798999999999997e-05 2024-02-09T17:47:16.794 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.794 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.794 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:47:16.794 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:02.600859+0000", 2024-02-09T17:47:16.794 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.533e-06 2024-02-09T17:47:16.794 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.794 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.794 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:47:16.794 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:03.595746+0000", 2024-02-09T17:47:16.794 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.99488617000000001 2024-02-09T17:47:16.795 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.795 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.795 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:47:16.795 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:03.595773+0000", 2024-02-09T17:47:16.795 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7568000000000001e-05 2024-02-09T17:47:16.795 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.795 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.795 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:47:16.795 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:04.756038+0000", 2024-02-09T17:47:16.795 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.160265001 2024-02-09T17:47:16.795 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.795 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.795 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:47:16.795 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:04.756175+0000", 2024-02-09T17:47:16.796 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013715 2024-02-09T17:47:16.796 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.796 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.796 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 7,9,10", 2024-02-09T17:47:16.796 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:04.756283+0000", 2024-02-09T17:47:16.796 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010802 2024-02-09T17:47:16.796 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.796 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.796 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.796 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:04.760891+0000", 2024-02-09T17:47:16.796 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0046074280000000002 2024-02-09T17:47:16.796 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.796 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.797 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.797 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:04.760958+0000", 2024-02-09T17:47:16.797 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7621999999999997e-05 2024-02-09T17:47:16.797 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.797 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.797 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:47:16.797 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:04.760979+0000", 2024-02-09T17:47:16.797 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0386000000000001e-05 2024-02-09T17:47:16.797 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.797 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.797 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.797 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:04.771736+0000", 2024-02-09T17:47:16.797 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010757325999999999 2024-02-09T17:47:16.797 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.797 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.798 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:47:16.798 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:04.771781+0000", 2024-02-09T17:47:16.798 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4974e-05 2024-02-09T17:47:16.798 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.798 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.798 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:47:16.798 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:04.771853+0000", 2024-02-09T17:47:16.798 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1554e-05 2024-02-09T17:47:16.798 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:47:16.798 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:47:16.798 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:47:16.798 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.798 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.798 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:222292 4.8 4:10fa3862:::benchmark_data_smithi179_103809_object84826:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e399)", 2024-02-09T17:47:16.799 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:39:03.757917+0000", 2024-02-09T17:47:16.799 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 493.01094534100002, 2024-02-09T17:47:16.799 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0220161219999999, 2024-02-09T17:47:16.799 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:47:16.799 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:47:16.799 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:47:16.799 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:47:16.799 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:47:16.799 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 222292 2024-02-09T17:47:16.799 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.799 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:47:16.799 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.799 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:47:16.799 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:03.757917+0000", 2024-02-09T17:47:16.800 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:47:16.800 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.800 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.800 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:47:16.800 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:03.757917+0000", 2024-02-09T17:47:16.800 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:47:16.800 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.800 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.800 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:47:16.800 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:03.757919+0000", 2024-02-09T17:47:16.800 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2979999999999999e-06 2024-02-09T17:47:16.800 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.800 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.800 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:47:16.801 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:03.757927+0000", 2024-02-09T17:47:16.801 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8590000000000005e-06 2024-02-09T17:47:16.801 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.801 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.801 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:47:16.801 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:03.757944+0000", 2024-02-09T17:47:16.801 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.753e-05 2024-02-09T17:47:16.801 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.801 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.801 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:47:16.801 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:03.757955+0000", 2024-02-09T17:47:16.801 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0563e-05 2024-02-09T17:47:16.801 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.801 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.802 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:47:16.802 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:03.758024+0000", 2024-02-09T17:47:16.802 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8775999999999993e-05 2024-02-09T17:47:16.802 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.802 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.802 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:47:16.802 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:03.758035+0000", 2024-02-09T17:47:16.802 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0872e-05 2024-02-09T17:47:16.802 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.802 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.802 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:47:16.802 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:04.760994+0000", 2024-02-09T17:47:16.802 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0029593960000001 2024-02-09T17:47:16.803 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.803 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.803 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:47:16.803 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:04.761083+0000", 2024-02-09T17:47:16.803 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.9345000000000006e-05 2024-02-09T17:47:16.803 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.803 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.803 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 7,9,10", 2024-02-09T17:47:16.803 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:04.761186+0000", 2024-02-09T17:47:16.803 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000103102 2024-02-09T17:47:16.803 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.803 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.803 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.803 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:04.766313+0000", 2024-02-09T17:47:16.804 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0051261850000000001 2024-02-09T17:47:16.804 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.804 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.804 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.804 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:04.766372+0000", 2024-02-09T17:47:16.804 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9181000000000001e-05 2024-02-09T17:47:16.804 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.804 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.804 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:47:16.804 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:04.766389+0000", 2024-02-09T17:47:16.804 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.694e-05 2024-02-09T17:47:16.804 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.804 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.804 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.805 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:04.779761+0000", 2024-02-09T17:47:16.805 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013372124000000001 2024-02-09T17:47:16.805 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.805 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.805 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:47:16.805 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:04.779831+0000", 2024-02-09T17:47:16.805 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9968000000000005e-05 2024-02-09T17:47:16.805 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.805 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.805 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:47:16.805 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:04.779933+0000", 2024-02-09T17:47:16.805 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000101983 2024-02-09T17:47:16.805 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:47:16.805 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:47:16.806 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:47:16.806 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.806 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.806 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:229698 4.8 4:198ec6a7:::benchmark_data_smithi179_103809_object92232:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e409)", 2024-02-09T17:47:16.806 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:39:17.475832+0000", 2024-02-09T17:47:16.806 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 479.29303048499997, 2024-02-09T17:47:16.806 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.025248959, 2024-02-09T17:47:16.806 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:47:16.806 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:47:16.806 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:47:16.806 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:47:16.806 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:47:16.806 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 229698 2024-02-09T17:47:16.806 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.806 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:47:16.807 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.807 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:47:16.807 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:17.475832+0000", 2024-02-09T17:47:16.807 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:47:16.807 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.807 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.807 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:47:16.807 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:17.475832+0000", 2024-02-09T17:47:16.807 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:47:16.807 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.807 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.807 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:47:16.807 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:17.475833+0000", 2024-02-09T17:47:16.807 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7000000000000003e-07 2024-02-09T17:47:16.808 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.808 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.808 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:47:16.808 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:17.475837+0000", 2024-02-09T17:47:16.808 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7840000000000003e-06 2024-02-09T17:47:16.808 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.808 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.808 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:47:16.808 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:17.475856+0000", 2024-02-09T17:47:16.808 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8580000000000002e-05 2024-02-09T17:47:16.808 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.808 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.808 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:47:16.808 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:17.475862+0000", 2024-02-09T17:47:16.809 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1419999999999998e-06 2024-02-09T17:47:16.809 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.809 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.809 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:47:16.809 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:17.475917+0000", 2024-02-09T17:47:16.809 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5381999999999999e-05 2024-02-09T17:47:16.809 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.809 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.809 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:47:16.809 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:17.475928+0000", 2024-02-09T17:47:16.809 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0288999999999999e-05 2024-02-09T17:47:16.809 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.809 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.809 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:47:16.810 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:18.491094+0000", 2024-02-09T17:47:16.810 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0151661599999999 2024-02-09T17:47:16.810 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.810 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.810 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:47:16.810 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:18.491222+0000", 2024-02-09T17:47:16.810 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012769300000000001 2024-02-09T17:47:16.810 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.810 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.810 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 9,10", 2024-02-09T17:47:16.810 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:18.491373+0000", 2024-02-09T17:47:16.810 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015154099999999999 2024-02-09T17:47:16.810 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.810 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.810 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.811 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:18.496295+0000", 2024-02-09T17:47:16.811 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0049214920000000004 2024-02-09T17:47:16.811 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.811 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.811 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.811 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:18.496363+0000", 2024-02-09T17:47:16.811 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8422000000000003e-05 2024-02-09T17:47:16.811 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.811 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.811 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:47:16.811 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:18.500979+0000", 2024-02-09T17:47:16.811 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0046163360000000004 2024-02-09T17:47:16.811 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.811 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.812 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:47:16.812 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:18.501037+0000", 2024-02-09T17:47:16.812 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7222000000000002e-05 2024-02-09T17:47:16.812 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.812 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.812 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:47:16.812 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:39:18.501081+0000", 2024-02-09T17:47:16.812 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3946000000000002e-05 2024-02-09T17:47:16.812 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:47:16.812 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:47:16.812 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:47:16.812 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.812 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.812 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:266612 4.8 4:18a47bff:::benchmark_data_smithi179_103809_object129146:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e451)", 2024-02-09T17:47:16.813 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:40:18.489717+0000", 2024-02-09T17:47:16.813 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 418.27914542899998, 2024-02-09T17:47:16.813 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1935931609999999, 2024-02-09T17:47:16.813 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:47:16.813 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:47:16.813 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:47:16.813 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:47:16.813 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:47:16.813 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 266612 2024-02-09T17:47:16.813 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.813 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:47:16.813 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.813 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:47:16.813 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.489717+0000", 2024-02-09T17:47:16.813 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:47:16.814 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.814 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.814 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:47:16.814 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.489717+0000", 2024-02-09T17:47:16.814 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:47:16.814 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.814 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.814 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:47:16.814 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.489718+0000", 2024-02-09T17:47:16.814 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.469e-06 2024-02-09T17:47:16.814 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.814 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.814 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:47:16.814 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.489723+0000", 2024-02-09T17:47:16.815 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9030000000000003e-06 2024-02-09T17:47:16.815 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.815 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.815 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:47:16.815 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.489735+0000", 2024-02-09T17:47:16.815 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2111e-05 2024-02-09T17:47:16.815 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.815 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.815 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:47:16.815 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.489751+0000", 2024-02-09T17:47:16.815 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6099e-05 2024-02-09T17:47:16.815 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.815 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.815 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:47:16.815 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.489809+0000", 2024-02-09T17:47:16.816 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7327999999999997e-05 2024-02-09T17:47:16.816 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.816 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.816 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:47:16.816 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.489817+0000", 2024-02-09T17:47:16.816 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5639999999999997e-06 2024-02-09T17:47:16.816 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.816 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.816 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:47:16.816 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.670050+0000", 2024-02-09T17:47:16.816 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1802329739999999 2024-02-09T17:47:16.816 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.816 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.817 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:47:16.817 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.670154+0000", 2024-02-09T17:47:16.817 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000103429 2024-02-09T17:47:16.817 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.817 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.817 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 6,7,9,10", 2024-02-09T17:47:16.817 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.670246+0000", 2024-02-09T17:47:16.817 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2204e-05 2024-02-09T17:47:16.817 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.817 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.817 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.817 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.674627+0000", 2024-02-09T17:47:16.817 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0043817509999999997 2024-02-09T17:47:16.817 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.818 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.818 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:47:16.818 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.674639+0000", 2024-02-09T17:47:16.818 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1732999999999999e-05 2024-02-09T17:47:16.818 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.818 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.818 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.818 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.674671+0000", 2024-02-09T17:47:16.818 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2150999999999998e-05 2024-02-09T17:47:16.818 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.818 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.818 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.818 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.683156+0000", 2024-02-09T17:47:16.818 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0084847020000000002 2024-02-09T17:47:16.818 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.819 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.819 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.819 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.683195+0000", 2024-02-09T17:47:16.819 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8481e-05 2024-02-09T17:47:16.819 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.819 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.819 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:47:16.819 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.683235+0000", 2024-02-09T17:47:16.819 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0068000000000003e-05 2024-02-09T17:47:16.819 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.819 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.819 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:47:16.819 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.683310+0000", 2024-02-09T17:47:16.819 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5193999999999999e-05 2024-02-09T17:47:16.819 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:47:16.820 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:47:16.820 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:47:16.820 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.820 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.820 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:266621 4.8 4:1943a0f5:::benchmark_data_smithi179_103809_object129155:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e451)", 2024-02-09T17:47:16.820 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:40:18.499697+0000", 2024-02-09T17:47:16.820 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 418.26916476899999, 2024-02-09T17:47:16.820 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.184130288, 2024-02-09T17:47:16.820 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:47:16.820 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:47:16.820 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:47:16.820 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:47:16.820 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:47:16.820 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 266621 2024-02-09T17:47:16.821 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.821 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:47:16.821 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.821 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:47:16.821 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.499697+0000", 2024-02-09T17:47:16.821 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:47:16.821 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.821 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.821 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:47:16.821 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.499697+0000", 2024-02-09T17:47:16.821 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:47:16.821 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.821 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.822 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:47:16.822 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.499698+0000", 2024-02-09T17:47:16.822 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.009e-06 2024-02-09T17:47:16.822 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.822 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.822 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:47:16.822 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.499702+0000", 2024-02-09T17:47:16.822 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6600000000000001e-06 2024-02-09T17:47:16.822 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.822 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.822 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:47:16.822 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.499711+0000", 2024-02-09T17:47:16.822 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7390000000000005e-06 2024-02-09T17:47:16.822 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.823 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.823 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:47:16.823 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.499718+0000", 2024-02-09T17:47:16.823 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2099999999999996e-06 2024-02-09T17:47:16.823 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.823 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.823 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:47:16.823 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.499773+0000", 2024-02-09T17:47:16.823 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4901999999999997e-05 2024-02-09T17:47:16.823 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.823 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.823 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:47:16.823 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.499781+0000", 2024-02-09T17:47:16.823 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.0369999999999995e-06 2024-02-09T17:47:16.824 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.824 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.824 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:47:16.824 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.674807+0000", 2024-02-09T17:47:16.824 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.175026063 2024-02-09T17:47:16.824 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.824 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.824 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:47:16.824 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.674902+0000", 2024-02-09T17:47:16.824 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5045999999999998e-05 2024-02-09T17:47:16.824 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.824 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.824 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 6,7,9,10", 2024-02-09T17:47:16.824 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.675005+0000", 2024-02-09T17:47:16.825 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000103119 2024-02-09T17:47:16.825 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.825 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.825 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:47:16.825 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.683168+0000", 2024-02-09T17:47:16.825 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0081628150000000003 2024-02-09T17:47:16.825 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.825 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.825 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.825 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.683361+0000", 2024-02-09T17:47:16.825 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00019343 2024-02-09T17:47:16.825 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.825 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.825 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.826 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.683430+0000", 2024-02-09T17:47:16.826 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8336000000000004e-05 2024-02-09T17:47:16.826 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.826 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.826 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.826 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.683470+0000", 2024-02-09T17:47:16.826 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9857999999999997e-05 2024-02-09T17:47:16.826 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.826 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.826 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.826 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.683693+0000", 2024-02-09T17:47:16.826 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00022318699999999999 2024-02-09T17:47:16.826 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.826 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.827 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:47:16.827 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.683731+0000", 2024-02-09T17:47:16.827 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8349999999999997e-05 2024-02-09T17:47:16.827 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.827 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.827 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:47:16.827 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.683828+0000", 2024-02-09T17:47:16.827 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6527000000000006e-05 2024-02-09T17:47:16.827 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:47:16.827 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:47:16.827 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:47:16.827 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.827 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.827 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:266626 4.8 4:11c2026a:::benchmark_data_smithi179_103809_object129160:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e451)", 2024-02-09T17:47:16.827 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:40:18.504323+0000", 2024-02-09T17:47:16.828 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 418.26453902600002, 2024-02-09T17:47:16.828 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1797135670000001, 2024-02-09T17:47:16.828 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:47:16.828 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:47:16.828 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:47:16.828 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:47:16.828 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:47:16.828 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 266626 2024-02-09T17:47:16.828 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.828 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:47:16.828 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.828 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:47:16.828 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.504323+0000", 2024-02-09T17:47:16.828 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:47:16.829 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.829 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.829 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:47:16.829 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.504323+0000", 2024-02-09T17:47:16.829 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:47:16.829 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.829 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.829 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:47:16.829 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.504324+0000", 2024-02-09T17:47:16.829 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.6499999999999998e-07 2024-02-09T17:47:16.829 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.829 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.829 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:47:16.829 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.504328+0000", 2024-02-09T17:47:16.829 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8659999999999999e-06 2024-02-09T17:47:16.829 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.830 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.830 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:47:16.830 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.504339+0000", 2024-02-09T17:47:16.830 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1610000000000001e-05 2024-02-09T17:47:16.830 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.830 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.830 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:47:16.830 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.504345+0000", 2024-02-09T17:47:16.830 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8590000000000001e-06 2024-02-09T17:47:16.830 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.830 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.830 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:47:16.830 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.504397+0000", 2024-02-09T17:47:16.830 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.198e-05 2024-02-09T17:47:16.830 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.831 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.831 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:47:16.831 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.504407+0000", 2024-02-09T17:47:16.831 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.3139999999999993e-06 2024-02-09T17:47:16.831 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.831 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.831 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:47:16.831 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.678978+0000", 2024-02-09T17:47:16.831 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1745714709999999 2024-02-09T17:47:16.831 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.831 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.831 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:47:16.831 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.679072+0000", 2024-02-09T17:47:16.831 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.4234000000000006e-05 2024-02-09T17:47:16.831 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.831 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.832 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 6,7,9,10", 2024-02-09T17:47:16.832 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.679189+0000", 2024-02-09T17:47:16.832 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000116743 2024-02-09T17:47:16.832 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.832 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.832 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:47:16.832 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.683485+0000", 2024-02-09T17:47:16.832 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0042956160000000004 2024-02-09T17:47:16.832 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.832 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.832 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.832 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.683522+0000", 2024-02-09T17:47:16.832 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7728999999999997e-05 2024-02-09T17:47:16.833 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.833 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.833 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.833 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.683661+0000", 2024-02-09T17:47:16.833 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013825799999999999 2024-02-09T17:47:16.833 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.833 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.833 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.833 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.683866+0000", 2024-02-09T17:47:16.833 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00020525 2024-02-09T17:47:16.833 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.833 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.833 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.834 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.683907+0000", 2024-02-09T17:47:16.834 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1004999999999998e-05 2024-02-09T17:47:16.834 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.834 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.834 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:47:16.834 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.683939+0000", 2024-02-09T17:47:16.834 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1986000000000002e-05 2024-02-09T17:47:16.834 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.834 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.834 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:47:16.834 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.684037+0000", 2024-02-09T17:47:16.834 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7781000000000005e-05 2024-02-09T17:47:16.834 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:47:16.834 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:47:16.834 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:47:16.835 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.835 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.835 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:266690 4.8 4:1a762542:::benchmark_data_smithi179_103809_object129224:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e451)", 2024-02-09T17:47:16.835 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:40:18.574021+0000", 2024-02-09T17:47:16.835 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 418.19484109899997, 2024-02-09T17:47:16.835 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.12055982, 2024-02-09T17:47:16.835 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:47:16.835 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:47:16.835 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:47:16.835 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:47:16.835 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:47:16.835 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 266690 2024-02-09T17:47:16.835 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.835 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:47:16.836 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.836 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:47:16.836 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.574021+0000", 2024-02-09T17:47:16.836 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:47:16.836 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.836 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.836 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:47:16.836 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.574021+0000", 2024-02-09T17:47:16.836 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:47:16.836 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.836 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.836 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:47:16.836 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.574023+0000", 2024-02-09T17:47:16.836 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6020000000000001e-06 2024-02-09T17:47:16.837 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.837 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.837 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:47:16.837 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.574029+0000", 2024-02-09T17:47:16.837 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2339999999999999e-06 2024-02-09T17:47:16.837 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.837 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.837 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:47:16.837 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.574045+0000", 2024-02-09T17:47:16.837 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.647e-05 2024-02-09T17:47:16.837 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.837 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.837 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:47:16.837 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.574053+0000", 2024-02-09T17:47:16.837 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5480000000000004e-06 2024-02-09T17:47:16.838 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.838 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.838 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:47:16.838 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.574104+0000", 2024-02-09T17:47:16.838 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0966e-05 2024-02-09T17:47:16.838 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.838 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.838 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:47:16.838 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.574112+0000", 2024-02-09T17:47:16.838 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3790000000000006e-06 2024-02-09T17:47:16.838 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.838 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.838 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:47:16.838 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.684083+0000", 2024-02-09T17:47:16.838 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.109971177 2024-02-09T17:47:16.839 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.839 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.839 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:47:16.839 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.684227+0000", 2024-02-09T17:47:16.839 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014359600000000001 2024-02-09T17:47:16.839 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.839 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.839 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 6,7,9,10", 2024-02-09T17:47:16.839 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.684373+0000", 2024-02-09T17:47:16.839 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014643499999999999 2024-02-09T17:47:16.839 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.839 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.839 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.839 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.694157+0000", 2024-02-09T17:47:16.839 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0097837280000000002 2024-02-09T17:47:16.839 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.840 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.840 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.840 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.694224+0000", 2024-02-09T17:47:16.840 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6450000000000002e-05 2024-02-09T17:47:16.840 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.840 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.840 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:47:16.840 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.694239+0000", 2024-02-09T17:47:16.840 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5739999999999998e-05 2024-02-09T17:47:16.840 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.840 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.840 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.840 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.694282+0000", 2024-02-09T17:47:16.841 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2200999999999998e-05 2024-02-09T17:47:16.841 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.841 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.841 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.841 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.694396+0000", 2024-02-09T17:47:16.841 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000114221 2024-02-09T17:47:16.841 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.841 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.841 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:47:16.841 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.694454+0000", 2024-02-09T17:47:16.841 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8687000000000002e-05 2024-02-09T17:47:16.841 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.841 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.841 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:47:16.841 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.694581+0000", 2024-02-09T17:47:16.841 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012638600000000001 2024-02-09T17:47:16.842 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:47:16.842 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:47:16.842 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:47:16.842 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.842 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.842 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:266692 4.8 4:1a858ed9:::benchmark_data_smithi179_103809_object129226:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e451)", 2024-02-09T17:47:16.842 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:40:18.576239+0000", 2024-02-09T17:47:16.842 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 418.19262299399998, 2024-02-09T17:47:16.842 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.124045765, 2024-02-09T17:47:16.842 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:47:16.842 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:47:16.842 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:47:16.842 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:47:16.843 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:47:16.843 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 266692 2024-02-09T17:47:16.843 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.843 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:47:16.843 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.843 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:47:16.843 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.576239+0000", 2024-02-09T17:47:16.843 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:47:16.843 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.843 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.843 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:47:16.843 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.576239+0000", 2024-02-09T17:47:16.843 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:47:16.843 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.844 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.844 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:47:16.844 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.576240+0000", 2024-02-09T17:47:16.844 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6599999999999994e-07 2024-02-09T17:47:16.844 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.844 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.844 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:47:16.844 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.576245+0000", 2024-02-09T17:47:16.844 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4440000000000001e-06 2024-02-09T17:47:16.844 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.844 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.844 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:47:16.844 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.576256+0000", 2024-02-09T17:47:16.845 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1034e-05 2024-02-09T17:47:16.845 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.845 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.845 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:47:16.845 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.576266+0000", 2024-02-09T17:47:16.845 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0101999999999999e-05 2024-02-09T17:47:16.845 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.845 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.845 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:47:16.845 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.576323+0000", 2024-02-09T17:47:16.845 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7278000000000003e-05 2024-02-09T17:47:16.845 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.845 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.845 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:47:16.846 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.576335+0000", 2024-02-09T17:47:16.846 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2546e-05 2024-02-09T17:47:16.846 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.846 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.846 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:47:16.846 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.689604+0000", 2024-02-09T17:47:16.846 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1132686270000001 2024-02-09T17:47:16.846 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.846 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.846 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:47:16.846 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.689722+0000", 2024-02-09T17:47:16.846 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000118239 2024-02-09T17:47:16.846 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.846 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.847 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 6,7,9,10", 2024-02-09T17:47:16.847 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.689854+0000", 2024-02-09T17:47:16.847 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013162699999999999 2024-02-09T17:47:16.847 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.847 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.847 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.847 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.694341+0000", 2024-02-09T17:47:16.847 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0044874049999999999 2024-02-09T17:47:16.847 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.847 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.847 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.847 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.699864+0000", 2024-02-09T17:47:16.847 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0055223720000000002 2024-02-09T17:47:16.847 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.848 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.848 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:47:16.848 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.699884+0000", 2024-02-09T17:47:16.848 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0083999999999999e-05 2024-02-09T17:47:16.848 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.848 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.848 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.848 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.699910+0000", 2024-02-09T17:47:16.848 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5749e-05 2024-02-09T17:47:16.848 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.848 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.848 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.848 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.700059+0000", 2024-02-09T17:47:16.849 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014895899999999999 2024-02-09T17:47:16.849 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.849 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.849 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:47:16.849 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.700127+0000", 2024-02-09T17:47:16.849 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8801999999999996e-05 2024-02-09T17:47:16.849 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.849 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.849 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:47:16.849 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.700285+0000", 2024-02-09T17:47:16.849 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015753100000000001 2024-02-09T17:47:16.849 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:47:16.849 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:47:16.849 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:47:16.850 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.850 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.850 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:266705 4.8 4:1346114f:::benchmark_data_smithi179_103809_object129239:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e451)", 2024-02-09T17:47:16.850 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:40:18.604328+0000", 2024-02-09T17:47:16.850 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 418.16453456199997, 2024-02-09T17:47:16.850 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0962643729999999, 2024-02-09T17:47:16.850 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:47:16.850 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:47:16.850 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:47:16.850 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:47:16.850 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:47:16.850 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 266705 2024-02-09T17:47:16.850 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.850 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:47:16.850 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.850 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:47:16.851 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.604328+0000", 2024-02-09T17:47:16.851 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:47:16.851 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.851 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.851 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:47:16.851 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.604328+0000", 2024-02-09T17:47:16.851 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:47:16.851 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.851 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.851 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:47:16.851 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.604329+0000", 2024-02-09T17:47:16.851 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9209999999999999e-06 2024-02-09T17:47:16.851 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.851 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.852 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:47:16.852 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.604336+0000", 2024-02-09T17:47:16.852 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3570000000000003e-06 2024-02-09T17:47:16.852 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.852 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.852 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:47:16.852 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.604352+0000", 2024-02-09T17:47:16.852 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6648999999999999e-05 2024-02-09T17:47:16.852 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.852 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.852 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:47:16.852 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.604360+0000", 2024-02-09T17:47:16.852 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4699999999999996e-06 2024-02-09T17:47:16.852 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.853 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.853 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:47:16.853 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.604414+0000", 2024-02-09T17:47:16.853 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4215999999999997e-05 2024-02-09T17:47:16.853 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.853 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.853 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:47:16.853 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.604423+0000", 2024-02-09T17:47:16.853 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5459999999999992e-06 2024-02-09T17:47:16.853 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.853 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.853 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:47:16.853 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.694617+0000", 2024-02-09T17:47:16.853 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.090194154 2024-02-09T17:47:16.854 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.854 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.854 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:47:16.854 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.694724+0000", 2024-02-09T17:47:16.854 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010733 2024-02-09T17:47:16.854 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.854 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.854 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 6,7,9,10", 2024-02-09T17:47:16.854 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.694808+0000", 2024-02-09T17:47:16.854 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3861000000000002e-05 2024-02-09T17:47:16.854 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.854 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.854 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.854 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.699961+0000", 2024-02-09T17:47:16.855 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0051531679999999996 2024-02-09T17:47:16.855 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.855 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.855 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.855 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.700012+0000", 2024-02-09T17:47:16.855 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0408999999999998e-05 2024-02-09T17:47:16.855 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.855 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.855 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:47:16.855 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.700299+0000", 2024-02-09T17:47:16.855 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00028777799999999998 2024-02-09T17:47:16.855 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.855 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.855 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.856 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.700352+0000", 2024-02-09T17:47:16.856 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2658000000000003e-05 2024-02-09T17:47:16.856 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.856 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.856 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.856 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.700407+0000", 2024-02-09T17:47:16.856 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4619e-05 2024-02-09T17:47:16.856 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.856 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.856 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:47:16.856 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.700466+0000", 2024-02-09T17:47:16.856 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9398999999999998e-05 2024-02-09T17:47:16.856 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.857 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.857 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:47:16.857 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.700592+0000", 2024-02-09T17:47:16.857 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000125838 2024-02-09T17:47:16.857 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:47:16.857 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:47:16.857 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:47:16.857 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.857 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.857 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:266714 4.8 4:162dc382:::benchmark_data_smithi179_103809_object129248:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e451)", 2024-02-09T17:47:16.857 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:40:18.647646+0000", 2024-02-09T17:47:16.857 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 418.12121594799999, 2024-02-09T17:47:16.857 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0732599810000001, 2024-02-09T17:47:16.857 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:47:16.857 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:47:16.857 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:47:16.858 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:47:16.858 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:47:16.858 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 266714 2024-02-09T17:47:16.858 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.858 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:47:16.858 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.858 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:47:16.858 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.647646+0000", 2024-02-09T17:47:16.858 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:47:16.858 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.858 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.858 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:47:16.858 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.647646+0000", 2024-02-09T17:47:16.858 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:47:16.859 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.859 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.859 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:47:16.859 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.647648+0000", 2024-02-09T17:47:16.859 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.449e-06 2024-02-09T17:47:16.859 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.859 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.859 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:47:16.859 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.647654+0000", 2024-02-09T17:47:16.859 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.533e-06 2024-02-09T17:47:16.859 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.859 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.859 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:47:16.859 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.647668+0000", 2024-02-09T17:47:16.859 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3538e-05 2024-02-09T17:47:16.860 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.860 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.860 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:47:16.860 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.647675+0000", 2024-02-09T17:47:16.860 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7090000000000002e-06 2024-02-09T17:47:16.860 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.860 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.860 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:47:16.860 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.647724+0000", 2024-02-09T17:47:16.860 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8572000000000002e-05 2024-02-09T17:47:16.860 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.860 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.860 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:47:16.860 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:18.647732+0000", 2024-02-09T17:47:16.861 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3550000000000005e-06 2024-02-09T17:47:16.861 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.861 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.861 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:47:16.861 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.700632+0000", 2024-02-09T17:47:16.861 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.052899757 2024-02-09T17:47:16.861 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.861 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.861 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:47:16.861 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.700749+0000", 2024-02-09T17:47:16.861 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000117259 2024-02-09T17:47:16.861 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.861 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.861 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 6,7,9,10", 2024-02-09T17:47:16.862 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.700835+0000", 2024-02-09T17:47:16.862 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5675999999999998e-05 2024-02-09T17:47:16.862 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.862 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.862 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.862 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.710673+0000", 2024-02-09T17:47:16.862 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.009837742 2024-02-09T17:47:16.862 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.862 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.862 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.862 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.710727+0000", 2024-02-09T17:47:16.862 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4543999999999998e-05 2024-02-09T17:47:16.862 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.862 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.862 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:47:16.863 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.720616+0000", 2024-02-09T17:47:16.863 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.009888539 2024-02-09T17:47:16.863 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.863 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.863 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.863 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.720699+0000", 2024-02-09T17:47:16.863 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3088000000000001e-05 2024-02-09T17:47:16.863 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.863 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.863 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.863 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.720735+0000", 2024-02-09T17:47:16.863 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6433999999999997e-05 2024-02-09T17:47:16.863 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.863 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.863 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:47:16.864 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.720777+0000", 2024-02-09T17:47:16.864 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1643000000000001e-05 2024-02-09T17:47:16.864 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.864 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.864 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:47:16.864 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:19.720906+0000", 2024-02-09T17:47:16.864 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012914299999999999 2024-02-09T17:47:16.864 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:47:16.864 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:47:16.864 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:47:16.864 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.864 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.864 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:267356 4.8 4:11055ded:::benchmark_data_smithi179_103809_object129890:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e454)", 2024-02-09T17:47:16.864 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:40:21.484386+0000", 2024-02-09T17:47:16.865 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 415.28447627600002, 2024-02-09T17:47:16.865 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0488926890000001, 2024-02-09T17:47:16.865 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:47:16.865 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:47:16.865 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:47:16.865 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:47:16.865 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:47:16.865 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 267356 2024-02-09T17:47:16.865 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.865 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:47:16.865 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.865 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:47:16.865 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.484386+0000", 2024-02-09T17:47:16.865 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:47:16.866 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.866 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.866 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:47:16.866 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.484386+0000", 2024-02-09T17:47:16.866 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:47:16.866 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.866 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.866 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:47:16.866 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.484387+0000", 2024-02-09T17:47:16.866 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.387e-06 2024-02-09T17:47:16.866 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.866 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.866 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:47:16.866 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.484392+0000", 2024-02-09T17:47:16.866 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6210000000000003e-06 2024-02-09T17:47:16.867 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.867 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.867 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:47:16.867 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.484406+0000", 2024-02-09T17:47:16.867 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4039999999999999e-05 2024-02-09T17:47:16.867 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.867 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.867 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:47:16.867 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.484415+0000", 2024-02-09T17:47:16.867 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.0310000000000004e-06 2024-02-09T17:47:16.867 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.867 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.867 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:47:16.867 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.484468+0000", 2024-02-09T17:47:16.868 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3177999999999998e-05 2024-02-09T17:47:16.868 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.868 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.868 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:47:16.868 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.484477+0000", 2024-02-09T17:47:16.868 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5080000000000005e-06 2024-02-09T17:47:16.868 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.868 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.868 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:47:16.868 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.520832+0000", 2024-02-09T17:47:16.868 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.036355763 2024-02-09T17:47:16.868 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.868 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.868 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:47:16.869 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.520952+0000", 2024-02-09T17:47:16.869 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000119741 2024-02-09T17:47:16.869 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.869 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.869 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 6,7", 2024-02-09T17:47:16.869 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.521079+0000", 2024-02-09T17:47:16.869 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012680800000000001 2024-02-09T17:47:16.869 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.869 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.869 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.869 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.528334+0000", 2024-02-09T17:47:16.869 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0072548159999999999 2024-02-09T17:47:16.869 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.869 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.869 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:47:16.870 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.530707+0000", 2024-02-09T17:47:16.870 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0023729480000000002 2024-02-09T17:47:16.870 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.870 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.870 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.870 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.533182+0000", 2024-02-09T17:47:16.870 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002475582 2024-02-09T17:47:16.870 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.870 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.870 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:47:16.870 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.533236+0000", 2024-02-09T17:47:16.870 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3777999999999999e-05 2024-02-09T17:47:16.870 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.871 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.871 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:47:16.871 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.533279+0000", 2024-02-09T17:47:16.871 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2487999999999997e-05 2024-02-09T17:47:16.871 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:47:16.871 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:47:16.871 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:47:16.871 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.871 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.871 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:267373 4.8 4:12172885:::benchmark_data_smithi179_103809_object129907:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e454)", 2024-02-09T17:47:16.871 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:40:21.504773+0000", 2024-02-09T17:47:16.871 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 415.26408878199999, 2024-02-09T17:47:16.871 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.036471082, 2024-02-09T17:47:16.871 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:47:16.871 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:47:16.872 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:47:16.872 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:47:16.872 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:47:16.872 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 267373 2024-02-09T17:47:16.872 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.872 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:47:16.872 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.872 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:47:16.872 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.504773+0000", 2024-02-09T17:47:16.872 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:47:16.872 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.872 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.872 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:47:16.872 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.504773+0000", 2024-02-09T17:47:16.873 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:47:16.873 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.873 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.873 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:47:16.873 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.504776+0000", 2024-02-09T17:47:16.873 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3539999999999998e-06 2024-02-09T17:47:16.873 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.873 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.873 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:47:16.873 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.504784+0000", 2024-02-09T17:47:16.873 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3110000000000006e-06 2024-02-09T17:47:16.873 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.873 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.873 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:47:16.873 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.504804+0000", 2024-02-09T17:47:16.874 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9947999999999999e-05 2024-02-09T17:47:16.874 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.874 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.874 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:47:16.874 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.504814+0000", 2024-02-09T17:47:16.874 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7229999999999999e-06 2024-02-09T17:47:16.874 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.874 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.874 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:47:16.874 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.504878+0000", 2024-02-09T17:47:16.874 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3927e-05 2024-02-09T17:47:16.874 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.874 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.874 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:47:16.875 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.504887+0000", 2024-02-09T17:47:16.875 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5189999999999998e-06 2024-02-09T17:47:16.875 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.875 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.875 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:47:16.875 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.525454+0000", 2024-02-09T17:47:16.875 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0205668999999999 2024-02-09T17:47:16.875 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.875 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.875 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:47:16.875 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.525531+0000", 2024-02-09T17:47:16.875 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6773000000000005e-05 2024-02-09T17:47:16.875 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.875 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.875 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 6,7", 2024-02-09T17:47:16.876 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.525606+0000", 2024-02-09T17:47:16.876 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5585000000000002e-05 2024-02-09T17:47:16.876 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.876 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.876 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.876 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.533319+0000", 2024-02-09T17:47:16.876 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0077124380000000003 2024-02-09T17:47:16.876 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.876 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.876 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:47:16.876 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.537161+0000", 2024-02-09T17:47:16.876 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0038420669999999998 2024-02-09T17:47:16.876 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.876 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.877 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.877 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.541171+0000", 2024-02-09T17:47:16.877 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0040103819999999998 2024-02-09T17:47:16.877 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.877 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.877 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:47:16.877 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.541208+0000", 2024-02-09T17:47:16.877 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6653999999999998e-05 2024-02-09T17:47:16.877 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.877 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.877 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:47:16.877 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.541244+0000", 2024-02-09T17:47:16.877 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6501000000000002e-05 2024-02-09T17:47:16.877 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:47:16.878 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:47:16.878 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:47:16.878 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.878 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.878 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:267374 4.8 4:1d92d63e:::benchmark_data_smithi179_103809_object129908:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e454)", 2024-02-09T17:47:16.878 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:40:21.506869+0000", 2024-02-09T17:47:16.878 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 415.26199305, 2024-02-09T17:47:16.878 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0345589079999999, 2024-02-09T17:47:16.878 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:47:16.878 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:47:16.878 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:47:16.878 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:47:16.878 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:47:16.878 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 267374 2024-02-09T17:47:16.878 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.879 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:47:16.879 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.879 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:47:16.879 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.506869+0000", 2024-02-09T17:47:16.879 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:47:16.879 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.879 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.879 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:47:16.879 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.506869+0000", 2024-02-09T17:47:16.879 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:47:16.879 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.879 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.879 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:47:16.879 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.506870+0000", 2024-02-09T17:47:16.880 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6599999999999994e-07 2024-02-09T17:47:16.880 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.880 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.880 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:47:16.880 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.506875+0000", 2024-02-09T17:47:16.880 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8849999999999998e-06 2024-02-09T17:47:16.880 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.880 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.880 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:47:16.880 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.506885+0000", 2024-02-09T17:47:16.880 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0281e-05 2024-02-09T17:47:16.880 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.880 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.880 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:47:16.881 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.506892+0000", 2024-02-09T17:47:16.881 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.0169999999999999e-06 2024-02-09T17:47:16.881 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.881 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.881 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:47:16.881 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.506940+0000", 2024-02-09T17:47:16.881 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8210000000000001e-05 2024-02-09T17:47:16.881 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.881 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.881 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:47:16.881 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.506947+0000", 2024-02-09T17:47:16.881 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5479999999999998e-06 2024-02-09T17:47:16.881 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.881 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.881 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:47:16.882 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.528366+0000", 2024-02-09T17:47:16.882 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0214189419999999 2024-02-09T17:47:16.882 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.882 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.882 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:47:16.882 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.528448+0000", 2024-02-09T17:47:16.882 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2445000000000001e-05 2024-02-09T17:47:16.882 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.882 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.882 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 6,7", 2024-02-09T17:47:16.882 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.528521+0000", 2024-02-09T17:47:16.882 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2743999999999993e-05 2024-02-09T17:47:16.882 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.882 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.882 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:47:16.883 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.537168+0000", 2024-02-09T17:47:16.883 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0086471280000000005 2024-02-09T17:47:16.883 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.883 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.883 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.883 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.541285+0000", 2024-02-09T17:47:16.883 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0041165619999999998 2024-02-09T17:47:16.883 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.883 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.883 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.883 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.541344+0000", 2024-02-09T17:47:16.883 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9629000000000001e-05 2024-02-09T17:47:16.883 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.883 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.884 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:47:16.884 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.541390+0000", 2024-02-09T17:47:16.884 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5392999999999997e-05 2024-02-09T17:47:16.884 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.884 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.884 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:47:16.884 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.541428+0000", 2024-02-09T17:47:16.884 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8158000000000003e-05 2024-02-09T17:47:16.884 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:47:16.884 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:47:16.884 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:47:16.884 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.884 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.884 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25201.0:267386 4.8 4:1caee092:::benchmark_data_smithi179_103809_object129920:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e454)", 2024-02-09T17:47:16.884 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:40:21.520850+0000", 2024-02-09T17:47:16.885 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 415.24801224499998, 2024-02-09T17:47:16.885 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.020772193, 2024-02-09T17:47:16.885 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:47:16.885 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:47:16.885 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:47:16.885 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25201", 2024-02-09T17:47:16.885 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:47:16.885 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 267386 2024-02-09T17:47:16.885 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.885 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:47:16.885 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.885 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:47:16.885 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.520850+0000", 2024-02-09T17:47:16.885 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:47:16.886 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.886 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.886 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:47:16.886 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.520850+0000", 2024-02-09T17:47:16.886 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:47:16.886 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.886 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.886 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:47:16.886 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.520852+0000", 2024-02-09T17:47:16.886 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9290000000000001e-06 2024-02-09T17:47:16.886 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.886 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.886 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:47:16.886 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.520859+0000", 2024-02-09T17:47:16.886 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.0940000000000002e-06 2024-02-09T17:47:16.887 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.887 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.887 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:47:16.887 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.520875+0000", 2024-02-09T17:47:16.887 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5865000000000001e-05 2024-02-09T17:47:16.887 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.887 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.887 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:47:16.887 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.520884+0000", 2024-02-09T17:47:16.887 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.73e-06 2024-02-09T17:47:16.887 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.887 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.887 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:47:16.887 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.520948+0000", 2024-02-09T17:47:16.887 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3578999999999996e-05 2024-02-09T17:47:16.888 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.888 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.888 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:47:16.888 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:21.520960+0000", 2024-02-09T17:47:16.888 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1618e-05 2024-02-09T17:47:16.888 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.888 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.888 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:47:16.888 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.530725+0000", 2024-02-09T17:47:16.888 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0097657390000001 2024-02-09T17:47:16.888 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.888 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.888 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:47:16.888 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.530785+0000", 2024-02-09T17:47:16.889 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9923000000000002e-05 2024-02-09T17:47:16.889 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.889 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.889 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 6,7", 2024-02-09T17:47:16.889 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.530848+0000", 2024-02-09T17:47:16.889 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2883000000000001e-05 2024-02-09T17:47:16.889 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.889 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.889 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:47:16.889 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.541259+0000", 2024-02-09T17:47:16.889 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010411023 2024-02-09T17:47:16.889 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.889 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.889 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.890 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.541494+0000", 2024-02-09T17:47:16.890 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00023479600000000001 2024-02-09T17:47:16.890 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.890 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.890 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.890 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.541541+0000", 2024-02-09T17:47:16.890 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7011000000000001e-05 2024-02-09T17:47:16.890 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.890 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.890 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:47:16.890 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.541588+0000", 2024-02-09T17:47:16.890 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6910000000000003e-05 2024-02-09T17:47:16.890 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.890 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.890 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:47:16.891 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:40:22.541622+0000", 2024-02-09T17:47:16.891 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4093e-05 2024-02-09T17:47:16.891 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:47:16.891 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:47:16.891 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:47:16.891 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.891 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.891 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:106985 5.8 5:10c6e4f9:::benchmark_data_smithi179_115199_object106984:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-02-09T17:47:16.891 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:42:12.106360+0000", 2024-02-09T17:47:16.891 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 304.66250239200002, 2024-02-09T17:47:16.891 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.21701166199999999, 2024-02-09T17:47:16.891 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:47:16.891 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:47:16.891 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:47:16.892 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:47:16.892 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:47:16.892 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 106985 2024-02-09T17:47:16.892 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.892 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:47:16.892 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.892 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:47:16.892 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.106360+0000", 2024-02-09T17:47:16.892 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:47:16.892 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.892 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.892 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:47:16.892 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.106360+0000", 2024-02-09T17:47:16.892 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:47:16.892 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.893 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.893 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:47:16.893 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.106361+0000", 2024-02-09T17:47:16.893 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3179999999999999e-06 2024-02-09T17:47:16.893 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.893 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.893 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:47:16.893 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.106388+0000", 2024-02-09T17:47:16.893 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6942999999999999e-05 2024-02-09T17:47:16.893 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.893 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.893 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:47:16.893 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.106399+0000", 2024-02-09T17:47:16.893 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1046999999999999e-05 2024-02-09T17:47:16.894 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.894 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.894 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:47:16.894 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.106406+0000", 2024-02-09T17:47:16.894 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4429999999999997e-06 2024-02-09T17:47:16.894 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.894 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.894 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:47:16.894 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.106485+0000", 2024-02-09T17:47:16.894 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8347000000000001e-05 2024-02-09T17:47:16.894 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.894 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.894 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:47:16.894 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.106623+0000", 2024-02-09T17:47:16.895 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013791399999999999 2024-02-09T17:47:16.895 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.895 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.895 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,10", 2024-02-09T17:47:16.895 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.106755+0000", 2024-02-09T17:47:16.895 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000132743 2024-02-09T17:47:16.895 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.895 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.895 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:47:16.895 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.114193+0000", 2024-02-09T17:47:16.895 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0074372270000000002 2024-02-09T17:47:16.895 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.895 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.895 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.896 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.117105+0000", 2024-02-09T17:47:16.896 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002912591 2024-02-09T17:47:16.896 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.896 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.896 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.896 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.323273+0000", 2024-02-09T17:47:16.896 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.20616793999999999 2024-02-09T17:47:16.896 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.896 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.896 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:47:16.896 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.323329+0000", 2024-02-09T17:47:16.896 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5958999999999997e-05 2024-02-09T17:47:16.896 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.896 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.897 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:47:16.897 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.323371+0000", 2024-02-09T17:47:16.897 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2190000000000001e-05 2024-02-09T17:47:16.897 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:47:16.897 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:47:16.897 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:47:16.897 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.897 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.897 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:106986 5.8 5:150a5124:::benchmark_data_smithi179_115199_object106985:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-02-09T17:47:16.897 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:42:12.106436+0000", 2024-02-09T17:47:16.897 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 304.662425891, 2024-02-09T17:47:16.897 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.217089212, 2024-02-09T17:47:16.897 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:47:16.897 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:47:16.898 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:47:16.898 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:47:16.898 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:47:16.898 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 106986 2024-02-09T17:47:16.898 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.898 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:47:16.898 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.898 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:47:16.898 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.106436+0000", 2024-02-09T17:47:16.898 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:47:16.898 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.898 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.898 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:47:16.898 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.106436+0000", 2024-02-09T17:47:16.898 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:47:16.899 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.899 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.899 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:47:16.899 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.106437+0000", 2024-02-09T17:47:16.899 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4700000000000002e-07 2024-02-09T17:47:16.899 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.899 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.899 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:47:16.899 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.106455+0000", 2024-02-09T17:47:16.899 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8627999999999999e-05 2024-02-09T17:47:16.899 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.899 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.899 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:47:16.899 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.106463+0000", 2024-02-09T17:47:16.899 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.909e-06 2024-02-09T17:47:16.900 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.900 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.900 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:47:16.900 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.106469+0000", 2024-02-09T17:47:16.900 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7150000000000003e-06 2024-02-09T17:47:16.900 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.900 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.900 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:47:16.900 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.110559+0000", 2024-02-09T17:47:16.900 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0040905079999999996 2024-02-09T17:47:16.900 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.900 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.900 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:47:16.901 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.110698+0000", 2024-02-09T17:47:16.901 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013882099999999999 2024-02-09T17:47:16.901 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.901 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.901 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,10", 2024-02-09T17:47:16.901 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.110827+0000", 2024-02-09T17:47:16.901 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000129202 2024-02-09T17:47:16.901 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.901 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.901 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:47:16.901 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.117481+0000", 2024-02-09T17:47:16.901 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.006653155 2024-02-09T17:47:16.901 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.901 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.902 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.902 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.119353+0000", 2024-02-09T17:47:16.902 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0018721499999999999 2024-02-09T17:47:16.902 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.902 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.902 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.902 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.323420+0000", 2024-02-09T17:47:16.902 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.204067464 2024-02-09T17:47:16.902 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.902 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.902 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:47:16.902 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.323482+0000", 2024-02-09T17:47:16.902 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2100000000000005e-05 2024-02-09T17:47:16.902 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.903 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.903 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:47:16.903 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.323525+0000", 2024-02-09T17:47:16.903 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3112999999999999e-05 2024-02-09T17:47:16.903 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:47:16.903 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:47:16.903 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:47:16.903 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.903 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.903 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:106996 5.8 5:13d7f260:::benchmark_data_smithi179_115199_object106995:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-02-09T17:47:16.903 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:42:12.111260+0000", 2024-02-09T17:47:16.903 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 304.65760189899999, 2024-02-09T17:47:16.903 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.21567955999999999, 2024-02-09T17:47:16.903 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:47:16.903 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:47:16.904 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:47:16.904 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:47:16.904 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:47:16.904 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 106996 2024-02-09T17:47:16.904 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.904 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:47:16.904 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.904 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:47:16.904 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.111260+0000", 2024-02-09T17:47:16.904 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:47:16.904 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.904 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.904 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:47:16.905 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.111260+0000", 2024-02-09T17:47:16.905 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:47:16.905 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.905 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.905 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:47:16.905 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.111262+0000", 2024-02-09T17:47:16.905 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5290000000000001e-06 2024-02-09T17:47:16.905 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.905 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.905 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:47:16.905 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.111294+0000", 2024-02-09T17:47:16.905 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1847999999999997e-05 2024-02-09T17:47:16.905 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.905 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.906 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:47:16.906 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.111307+0000", 2024-02-09T17:47:16.906 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3039e-05 2024-02-09T17:47:16.906 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.906 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.906 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:47:16.906 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.111315+0000", 2024-02-09T17:47:16.906 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8359999999999998e-06 2024-02-09T17:47:16.906 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.906 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.906 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:47:16.906 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.114226+0000", 2024-02-09T17:47:16.906 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0029104320000000001 2024-02-09T17:47:16.906 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.906 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.907 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:47:16.907 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.114327+0000", 2024-02-09T17:47:16.907 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000100912 2024-02-09T17:47:16.907 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.907 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.907 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,10", 2024-02-09T17:47:16.907 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.114412+0000", 2024-02-09T17:47:16.907 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5678000000000002e-05 2024-02-09T17:47:16.907 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.907 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.907 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:47:16.907 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.117486+0000", 2024-02-09T17:47:16.907 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0030737569999999999 2024-02-09T17:47:16.907 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.907 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.908 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.908 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.119386+0000", 2024-02-09T17:47:16.908 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0018997199999999999 2024-02-09T17:47:16.908 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.908 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.911 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.911 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.326833+0000", 2024-02-09T17:47:16.911 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.207446662 2024-02-09T17:47:16.911 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.912 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.912 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:47:16.912 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.326899+0000", 2024-02-09T17:47:16.912 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6000000000000005e-05 2024-02-09T17:47:16.912 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.912 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.912 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:47:16.912 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.326940+0000", 2024-02-09T17:47:16.912 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1146999999999998e-05 2024-02-09T17:47:16.912 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:47:16.912 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:47:16.912 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:47:16.912 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.912 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.913 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:126288 5.d 5:b690dee0:::benchmark_data_smithi179_115199_object126287:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e501)", 2024-02-09T17:47:16.913 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:42:31.632319+0000", 2024-02-09T17:47:16.913 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 285.13654308999998, 2024-02-09T17:47:16.913 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.22240631899999999, 2024-02-09T17:47:16.913 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:47:16.913 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:47:16.913 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:47:16.913 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:47:16.913 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:47:16.913 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 126288 2024-02-09T17:47:16.913 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.913 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:47:16.913 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.913 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:47:16.913 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:31.632319+0000", 2024-02-09T17:47:16.914 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:47:16.914 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.914 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.914 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:47:16.914 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:31.632319+0000", 2024-02-09T17:47:16.914 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:47:16.914 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.914 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.914 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:47:16.914 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:31.632321+0000", 2024-02-09T17:47:16.914 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.731e-06 2024-02-09T17:47:16.914 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.914 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.914 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:47:16.914 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:31.632368+0000", 2024-02-09T17:47:16.915 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7756999999999997e-05 2024-02-09T17:47:16.915 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.915 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.915 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:47:16.915 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:31.632385+0000", 2024-02-09T17:47:16.915 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6109000000000001e-05 2024-02-09T17:47:16.915 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.915 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.915 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:47:16.915 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:31.632393+0000", 2024-02-09T17:47:16.915 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7709999999999995e-06 2024-02-09T17:47:16.915 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.915 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.915 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:47:16.915 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:31.632459+0000", 2024-02-09T17:47:16.915 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6130999999999994e-05 2024-02-09T17:47:16.916 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.916 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.916 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:47:16.916 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:31.632587+0000", 2024-02-09T17:47:16.916 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000127995 2024-02-09T17:47:16.916 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.916 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.916 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,7", 2024-02-09T17:47:16.916 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:31.632695+0000", 2024-02-09T17:47:16.916 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000107111 2024-02-09T17:47:16.916 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.916 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.916 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:47:16.917 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:31.638171+0000", 2024-02-09T17:47:16.917 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0054763989999999999 2024-02-09T17:47:16.917 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.917 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.917 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.917 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:31.639611+0000", 2024-02-09T17:47:16.917 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001439863 2024-02-09T17:47:16.917 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.917 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.917 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:47:16.917 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:31.854656+0000", 2024-02-09T17:47:16.917 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.21504557699999999 2024-02-09T17:47:16.917 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.917 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.917 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:47:16.918 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:31.854700+0000", 2024-02-09T17:47:16.918 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3149000000000003e-05 2024-02-09T17:47:16.918 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:47:16.918 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:47:16.918 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:47:16.918 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:31.854725+0000", 2024-02-09T17:47:16.918 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5726000000000001e-05 2024-02-09T17:47:16.918 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:47:16.918 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:47:16.918 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:47:16.918 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:47:16.918 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:47:16.918 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:47:16.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:16.920 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:47:16.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:47:17.021 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:47:17.064 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:16 smithi162 ceph-mon[44304]: 1.4 scrub starts 2024-02-09T17:47:17.064 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:16 smithi162 ceph-mon[44304]: 1.4 scrub ok 2024-02-09T17:47:17.064 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:16 smithi162 ceph-mon[44304]: 6.b scrub ok 2024-02-09T17:47:17.064 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:16 smithi162 ceph-mon[44304]: 6.a scrub ok 2024-02-09T17:47:17.067 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:17.100 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:47:17.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:17.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:47:17.105 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 85 16 125892 125876 92.5452 60.3125 0.00826856 0.0108017 2024-02-09T17:47:17.133 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:16 smithi099 ceph-mon[39420]: 1.4 scrub starts 2024-02-09T17:47:17.133 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:16 smithi099 ceph-mon[39420]: 1.4 scrub ok 2024-02-09T17:47:17.133 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:16 smithi099 ceph-mon[39420]: 6.b scrub ok 2024-02-09T17:47:17.133 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:16 smithi099 ceph-mon[39420]: 6.a scrub ok 2024-02-09T17:47:17.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:16 smithi179 ceph-mon[43975]: 1.4 scrub starts 2024-02-09T17:47:17.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:16 smithi179 ceph-mon[43975]: 1.4 scrub ok 2024-02-09T17:47:17.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:16 smithi179 ceph-mon[43975]: 6.b scrub ok 2024-02-09T17:47:17.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:16 smithi179 ceph-mon[43975]: 6.a scrub ok 2024-02-09T17:47:17.204 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:47:17.235 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:18.105 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 86 16 126875 126859 92.1835 61.4375 0.00744716 0.0108439 2024-02-09T17:47:18.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:17 smithi179 ceph-mon[43975]: 6.1 scrub ok 2024-02-09T17:47:18.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:17 smithi179 ceph-mon[43975]: 1.9 scrub starts 2024-02-09T17:47:18.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:17 smithi179 ceph-mon[43975]: 1.9 scrub ok 2024-02-09T17:47:18.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:17 smithi179 ceph-mon[43975]: 1.1b scrub starts 2024-02-09T17:47:18.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:17 smithi179 ceph-mon[43975]: 1.1b scrub ok 2024-02-09T17:47:18.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:17 smithi179 ceph-mon[43975]: pgmap v1528: 47 pgs: 3 active+clean+scrubbing, 44 active+clean; 7.5 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 746 op/s 2024-02-09T17:47:18.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:17 smithi162 ceph-mon[44304]: 6.1 scrub ok 2024-02-09T17:47:18.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:17 smithi162 ceph-mon[44304]: 1.9 scrub starts 2024-02-09T17:47:18.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:17 smithi162 ceph-mon[44304]: 1.9 scrub ok 2024-02-09T17:47:18.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:17 smithi162 ceph-mon[44304]: 1.1b scrub starts 2024-02-09T17:47:18.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:17 smithi162 ceph-mon[44304]: 1.1b scrub ok 2024-02-09T17:47:18.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:17 smithi162 ceph-mon[44304]: pgmap v1528: 47 pgs: 3 active+clean+scrubbing, 44 active+clean; 7.5 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 746 op/s 2024-02-09T17:47:18.250 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:17 smithi099 ceph-mon[39420]: 6.1 scrub ok 2024-02-09T17:47:18.250 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:17 smithi099 ceph-mon[39420]: 1.9 scrub starts 2024-02-09T17:47:18.250 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:17 smithi099 ceph-mon[39420]: 1.9 scrub ok 2024-02-09T17:47:18.250 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:17 smithi099 ceph-mon[39420]: 1.1b scrub starts 2024-02-09T17:47:18.251 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:17 smithi099 ceph-mon[39420]: 1.1b scrub ok 2024-02-09T17:47:18.251 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:17 smithi099 ceph-mon[39420]: pgmap v1528: 47 pgs: 3 active+clean+scrubbing, 44 active+clean; 7.5 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 746 op/s 2024-02-09T17:47:18.260 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.8 dump_ops_in_flight 2024-02-09T17:47:18.297 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:47:18.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:18.302 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:47:18.403 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:47:18.453 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:18.484 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:47:18.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:18.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:47:18.588 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:47:18.646 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:47:18.661 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:18.827 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:18 smithi162 ceph-mon[44304]: 6.8 scrub starts 2024-02-09T17:47:18.827 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:18 smithi162 ceph-mon[44304]: 6.c scrub starts 2024-02-09T17:47:18.873 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:47:18.888 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:18 smithi099 ceph-mon[39420]: 6.8 scrub starts 2024-02-09T17:47:18.888 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:18 smithi099 ceph-mon[39420]: 6.c scrub starts 2024-02-09T17:47:18.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:18.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:47:18.990 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:47:19.022 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:19.052 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:47:19.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:19.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:47:19.105 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 87 15 127731 127716 91.7395 53.5625 0.0236653 0.0108964 2024-02-09T17:47:19.144 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:18 smithi179 ceph-mon[43975]: 6.8 scrub starts 2024-02-09T17:47:19.145 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:18 smithi179 ceph-mon[43975]: 6.c scrub starts 2024-02-09T17:47:19.159 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:47:19.195 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:19.227 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:47:19.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:19.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:47:19.333 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:47:19.366 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:19.518 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:47:19.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:19.524 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:47:19.625 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:47:19.676 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:19.708 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:47:19.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:19.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:47:19.819 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:47:19.851 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:19 smithi179 ceph-mon[43975]: pgmap v1529: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 7.7 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 823 op/s 2024-02-09T17:47:19.851 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:19 smithi179 ceph-mon[43975]: 1.13 scrub starts 2024-02-09T17:47:19.859 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:19.910 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:47:19.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:19.913 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:47:19.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:19 smithi162 ceph-mon[44304]: pgmap v1529: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 7.7 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 823 op/s 2024-02-09T17:47:19.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:19 smithi162 ceph-mon[44304]: 1.13 scrub starts 2024-02-09T17:47:20.014 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:47:20.051 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:20.106 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 88 15 128661 128646 91.3575 58.125 0.00450239 0.0109416 2024-02-09T17:47:20.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:19 smithi099 ceph-mon[39420]: pgmap v1529: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 7.7 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 823 op/s 2024-02-09T17:47:20.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:19 smithi099 ceph-mon[39420]: 1.13 scrub starts 2024-02-09T17:47:20.508 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:47:20.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:20.511 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:47:20.612 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:47:20.691 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:21.051 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:47:21.051 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-02-09T17:47:21.051 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 0 2024-02-09T17:47:21.051 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:47:21.072 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:20 smithi179 ceph-mon[43975]: 6.8 scrub ok 2024-02-09T17:47:21.072 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:20 smithi179 ceph-mon[43975]: 1.13 scrub ok 2024-02-09T17:47:21.072 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:20 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/196660627' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:47:21.072 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:20 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:47:21.080 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is unset 2024-02-09T17:47:21.106 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 89 16 129451 129435 90.8851 49.3125 0.215848 0.0109896 2024-02-09T17:47:21.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:20 smithi162 ceph-mon[44304]: 6.8 scrub ok 2024-02-09T17:47:21.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:20 smithi162 ceph-mon[44304]: 1.13 scrub ok 2024-02-09T17:47:21.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:20 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/196660627' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:47:21.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:20 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:47:21.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:20 smithi099 ceph-mon[39420]: 6.8 scrub ok 2024-02-09T17:47:21.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:20 smithi099 ceph-mon[39420]: 1.13 scrub ok 2024-02-09T17:47:21.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:20 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/196660627' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:47:21.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:20 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:47:21.769 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.8 dump_blocked_ops 2024-02-09T17:47:22.106 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Total time run: 90.0088 2024-02-09T17:47:22.106 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Total writes made: 130378 2024-02-09T17:47:22.106 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Write size: 65536 2024-02-09T17:47:22.106 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Object size: 65536 2024-02-09T17:47:22.106 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Bandwidth (MB/sec): 90.5314 2024-02-09T17:47:22.106 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Stddev Bandwidth: 40.6847 2024-02-09T17:47:22.106 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Max bandwidth (MB/sec): 288.562 2024-02-09T17:47:22.107 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Min bandwidth (MB/sec): 5.5 2024-02-09T17:47:22.107 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Average IOPS: 1448 2024-02-09T17:47:22.107 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Stddev IOPS: 650.956 2024-02-09T17:47:22.107 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Max IOPS: 4617 2024-02-09T17:47:22.107 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Min IOPS: 88 2024-02-09T17:47:22.107 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Average Latency(s): 0.0110425 2024-02-09T17:47:22.107 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Stddev Latency(s): 0.0168025 2024-02-09T17:47:22.107 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Max latency(s): 1.03613 2024-02-09T17:47:22.107 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Min latency(s): 0.00111008 2024-02-09T17:47:22.112 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Cleaning up (deleting benchmark objects) 2024-02-09T17:47:22.123 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:47:22.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:21 smithi179 ceph-mon[43975]: 6.c scrub ok 2024-02-09T17:47:22.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:21 smithi179 ceph-mon[43975]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:47:22.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:21 smithi179 ceph-mon[43975]: Cluster is now healthy 2024-02-09T17:47:22.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:21 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:47:22.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:21 smithi179 ceph-mon[43975]: osdmap e613: 12 total, 12 up, 12 in 2024-02-09T17:47:22.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:21 smithi179 ceph-mon[43975]: pgmap v1531: 47 pgs: 47 active+clean; 7.7 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 780 op/s 2024-02-09T17:47:22.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:21 smithi179 ceph-mon[43975]: 6.7 scrub starts 2024-02-09T17:47:22.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:21 smithi162 ceph-mon[44304]: 6.c scrub ok 2024-02-09T17:47:22.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:21 smithi162 ceph-mon[44304]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:47:22.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:21 smithi162 ceph-mon[44304]: Cluster is now healthy 2024-02-09T17:47:22.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:21 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:47:22.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:21 smithi162 ceph-mon[44304]: osdmap e613: 12 total, 12 up, 12 in 2024-02-09T17:47:22.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:21 smithi162 ceph-mon[44304]: pgmap v1531: 47 pgs: 47 active+clean; 7.7 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 780 op/s 2024-02-09T17:47:22.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:21 smithi162 ceph-mon[44304]: 6.7 scrub starts 2024-02-09T17:47:22.196 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:21 smithi099 ceph-mon[39420]: 6.c scrub ok 2024-02-09T17:47:22.196 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:21 smithi099 ceph-mon[39420]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:47:22.196 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:21 smithi099 ceph-mon[39420]: Cluster is now healthy 2024-02-09T17:47:22.197 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:21 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:47:22.197 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:21 smithi099 ceph-mon[39420]: osdmap e613: 12 total, 12 up, 12 in 2024-02-09T17:47:22.197 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:21 smithi099 ceph-mon[39420]: pgmap v1531: 47 pgs: 47 active+clean; 7.7 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 780 op/s 2024-02-09T17:47:22.197 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:21 smithi099 ceph-mon[39420]: 6.7 scrub starts 2024-02-09T17:47:22.867 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:22 smithi099 ceph-mon[39420]: 1.2 scrub starts 2024-02-09T17:47:22.867 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:22 smithi099 ceph-mon[39420]: 1.2 scrub ok 2024-02-09T17:47:22.867 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:22 smithi099 ceph-mon[39420]: 1.f scrub starts 2024-02-09T17:47:22.907 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:47:22.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:22 smithi179 ceph-mon[43975]: 1.2 scrub starts 2024-02-09T17:47:22.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:22 smithi179 ceph-mon[43975]: 1.2 scrub ok 2024-02-09T17:47:22.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:22 smithi179 ceph-mon[43975]: 1.f scrub starts 2024-02-09T17:47:22.944 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:47:22.994 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:47:23.020 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:47:23.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:23.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:47:23.040 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:47:23.090 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:47:23.125 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:47:23.142 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:47:23.172 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:47:23.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:22 smithi162 ceph-mon[44304]: 1.2 scrub starts 2024-02-09T17:47:23.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:22 smithi162 ceph-mon[44304]: 1.2 scrub ok 2024-02-09T17:47:23.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:22 smithi162 ceph-mon[44304]: 1.f scrub starts 2024-02-09T17:47:23.216 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:47:23.220 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:23.258 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:47:23.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:23.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:47:23.269 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:47:23.306 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:47:23.349 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:47:23.365 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:47:23.380 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:47:23.403 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:47:23.403 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:47:23.403 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:47:23.403 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:47:23.404 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:47:23.404 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:47:23.404 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:47:23.404 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:47:23.404 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:47:23.404 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:47:23.404 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:47:23.404 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:47:23.404 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:23.433 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:47:23.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:23.438 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:47:23.538 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:47:23.600 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:23.625 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:47:23.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:23.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:47:23.742 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:47:23.815 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:23.875 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:47:23.877 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:23 smithi162 ceph-mon[44304]: 1.f scrub ok 2024-02-09T17:47:23.889 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.4 to 1 (10000) 2024-02-09T17:47:23.896 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:23.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:47:23.986 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set noscrub 2024-02-09T17:47:23.997 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:47:24.034 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:24.080 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:47:24.080 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T17:47:24.080 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:47:24.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:24.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:47:24.102 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:23 smithi179 ceph-mon[43975]: 1.f scrub ok 2024-02-09T17:47:24.102 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:23 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2940065931' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:47:24.102 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:23 smithi179 ceph-mon[43975]: pgmap v1532: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 7.9 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 802 op/s 2024-02-09T17:47:24.102 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:23 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1849934141' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-09T17:47:24.103 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:23 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-09T17:47:24.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:23 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2940065931' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:47:24.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:23 smithi162 ceph-mon[44304]: pgmap v1532: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 7.9 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 802 op/s 2024-02-09T17:47:24.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:23 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1849934141' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-09T17:47:24.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:23 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-09T17:47:24.184 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:47:24.236 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:24.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:23 smithi099 ceph-mon[39420]: 1.f scrub ok 2024-02-09T17:47:24.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:23 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2940065931' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:47:24.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:23 smithi099 ceph-mon[39420]: pgmap v1532: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 7.9 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 802 op/s 2024-02-09T17:47:24.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:23 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1849934141' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-09T17:47:24.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:23 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-09T17:47:24.265 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:47:24.272 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:24.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:47:24.374 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:47:24.438 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:24.611 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:47:24.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:24.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:47:24.716 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:47:24.752 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:25.138 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:24 smithi099 ceph-mon[39420]: 6.7 scrub ok 2024-02-09T17:47:25.138 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:24 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-09T17:47:25.139 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:24 smithi099 ceph-mon[39420]: osdmap e614: 12 total, 12 up, 12 in 2024-02-09T17:47:25.141 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:47:25.167 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 5 1 2024-02-09T17:47:25.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:24 smithi179 ceph-mon[43975]: 6.7 scrub ok 2024-02-09T17:47:25.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:24 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-09T17:47:25.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:24 smithi179 ceph-mon[43975]: osdmap e614: 12 total, 12 up, 12 in 2024-02-09T17:47:25.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:24 smithi162 ceph-mon[44304]: 6.7 scrub ok 2024-02-09T17:47:25.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:24 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-09T17:47:25.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:24 smithi162 ceph-mon[44304]: osdmap e614: 12 total, 12 up, 12 in 2024-02-09T17:47:25.302 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:47:25.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:25.310 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:47:25.411 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:47:25.476 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:25.535 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:47:25.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:25.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:47:25.642 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:47:25.707 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:25.737 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:47:25.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:25.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:47:25.843 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:47:25.884 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:25.900 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.8 dump_historic_ops 2024-02-09T17:47:25.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:25.920 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:47:25.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:47:25.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:25 smithi099 ceph-mon[39420]: pgmap v1534: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 7.9 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 51 MiB/s wr, 881 op/s 2024-02-09T17:47:26.021 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:47:26.098 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:26.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:25 smithi179 ceph-mon[43975]: pgmap v1534: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 7.9 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 51 MiB/s wr, 881 op/s 2024-02-09T17:47:26.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:25 smithi162 ceph-mon[44304]: pgmap v1534: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 7.9 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 51 MiB/s wr, 881 op/s 2024-02-09T17:47:26.225 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:47:26.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:26.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:47:26.251 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:47:26.332 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:47:26.364 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:26.411 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:47:26.567 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:47:26.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:26.573 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:47:26.674 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:47:26.714 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:26.749 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:47:26.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:26.756 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:47:26.857 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:47:26.901 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:27.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:27 smithi099 ceph-mon[39420]: pgmap v1535: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 7.9 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 37 MiB/s wr, 811 op/s 2024-02-09T17:47:28.013 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:27 smithi179 ceph-mon[43975]: pgmap v1535: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 7.9 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 37 MiB/s wr, 811 op/s 2024-02-09T17:47:28.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:27 smithi162 ceph-mon[44304]: pgmap v1535: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 7.9 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 37 MiB/s wr, 811 op/s 2024-02-09T17:47:28.714 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T17:47:28.714 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:47:29.057 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {auth=33,config=2,health=12,kv=53} crc {auth=1450958966,config=2679774077,health=3897446751,kv=3745109799}) 2024-02-09T17:47:29.057 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2056630698}) 2024-02-09T17:47:29.057 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1494878002}) 2024-02-09T17:47:29.057 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3106884640}) 2024-02-09T17:47:29.057 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=816744445}) 2024-02-09T17:47:29.057 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3231824972}) 2024-02-09T17:47:29.057 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1330191448}) 2024-02-09T17:47:29.057 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1192603084}) 2024-02-09T17:47:29.058 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1540686426}) 2024-02-09T17:47:29.058 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3977176247}) 2024-02-09T17:47:29.058 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1882229734}) 2024-02-09T17:47:29.058 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3384756326}) 2024-02-09T17:47:29.058 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2749043887}) 2024-02-09T17:47:29.058 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3894995540}) 2024-02-09T17:47:29.058 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=376524431}) 2024-02-09T17:47:29.058 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1824554594}) 2024-02-09T17:47:29.058 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=29143596}) 2024-02-09T17:47:29.058 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3120815440}) 2024-02-09T17:47:29.058 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2518284121}) 2024-02-09T17:47:29.058 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2241659129}) 2024-02-09T17:47:29.058 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=673005013}) 2024-02-09T17:47:29.058 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1486639706}) 2024-02-09T17:47:29.059 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3715199670}) 2024-02-09T17:47:29.059 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=472543172}) 2024-02-09T17:47:29.059 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=793984776}) 2024-02-09T17:47:29.059 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1933228581}) 2024-02-09T17:47:29.059 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=311503825}) 2024-02-09T17:47:29.059 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4054732757}) 2024-02-09T17:47:29.059 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2363110713}) 2024-02-09T17:47:29.059 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3731821054}) 2024-02-09T17:47:29.059 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=254568500}) 2024-02-09T17:47:29.059 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3165554103}) 2024-02-09T17:47:29.059 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=174592252}) 2024-02-09T17:47:29.059 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3291636279}) 2024-02-09T17:47:29.059 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2727126295}) 2024-02-09T17:47:29.059 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4141718682}) 2024-02-09T17:47:29.059 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3903908258}) 2024-02-09T17:47:29.060 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3087133499}) 2024-02-09T17:47:29.060 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1051785243}) 2024-02-09T17:47:29.060 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3211850045}) 2024-02-09T17:47:29.060 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3607913009}) 2024-02-09T17:47:29.060 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=81270189}) 2024-02-09T17:47:29.060 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2590699091}) 2024-02-09T17:47:29.060 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2091212143}) 2024-02-09T17:47:29.060 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=55796120}) 2024-02-09T17:47:29.060 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3669075404}) 2024-02-09T17:47:29.060 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551701061}) 2024-02-09T17:47:29.060 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=54538955}) 2024-02-09T17:47:29.060 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:28 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3910626320}) 2024-02-09T17:47:29.139 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {auth=33,config=2,health=12,kv=53} crc {auth=1450958966,config=2679774077,health=3897446751,kv=3745109799}) 2024-02-09T17:47:29.139 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2056630698}) 2024-02-09T17:47:29.139 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1494878002}) 2024-02-09T17:47:29.139 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3106884640}) 2024-02-09T17:47:29.139 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=816744445}) 2024-02-09T17:47:29.140 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3231824972}) 2024-02-09T17:47:29.140 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1330191448}) 2024-02-09T17:47:29.140 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1192603084}) 2024-02-09T17:47:29.140 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1540686426}) 2024-02-09T17:47:29.140 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3977176247}) 2024-02-09T17:47:29.140 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1882229734}) 2024-02-09T17:47:29.140 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3384756326}) 2024-02-09T17:47:29.140 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2749043887}) 2024-02-09T17:47:29.140 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3894995540}) 2024-02-09T17:47:29.140 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=376524431}) 2024-02-09T17:47:29.140 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1824554594}) 2024-02-09T17:47:29.140 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=29143596}) 2024-02-09T17:47:29.140 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3120815440}) 2024-02-09T17:47:29.140 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2518284121}) 2024-02-09T17:47:29.141 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2241659129}) 2024-02-09T17:47:29.141 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=673005013}) 2024-02-09T17:47:29.141 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1486639706}) 2024-02-09T17:47:29.141 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3715199670}) 2024-02-09T17:47:29.141 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=472543172}) 2024-02-09T17:47:29.141 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=793984776}) 2024-02-09T17:47:29.141 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1933228581}) 2024-02-09T17:47:29.141 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=311503825}) 2024-02-09T17:47:29.141 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4054732757}) 2024-02-09T17:47:29.141 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2363110713}) 2024-02-09T17:47:29.141 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3731821054}) 2024-02-09T17:47:29.141 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=254568500}) 2024-02-09T17:47:29.141 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3165554103}) 2024-02-09T17:47:29.141 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=174592252}) 2024-02-09T17:47:29.142 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3291636279}) 2024-02-09T17:47:29.142 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2727126295}) 2024-02-09T17:47:29.142 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4141718682}) 2024-02-09T17:47:29.142 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3903908258}) 2024-02-09T17:47:29.142 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3087133499}) 2024-02-09T17:47:29.142 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1051785243}) 2024-02-09T17:47:29.142 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3211850045}) 2024-02-09T17:47:29.142 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3607913009}) 2024-02-09T17:47:29.142 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=81270189}) 2024-02-09T17:47:29.142 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2590699091}) 2024-02-09T17:47:29.142 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2091212143}) 2024-02-09T17:47:29.142 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=55796120}) 2024-02-09T17:47:29.142 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3669075404}) 2024-02-09T17:47:29.142 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551701061}) 2024-02-09T17:47:29.142 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=54538955}) 2024-02-09T17:47:29.142 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:28 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3910626320}) 2024-02-09T17:47:29.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {auth=33,config=2,health=12,kv=53} crc {auth=1450958966,config=2679774077,health=3897446751,kv=3745109799}) 2024-02-09T17:47:29.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2056630698}) 2024-02-09T17:47:29.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1494878002}) 2024-02-09T17:47:29.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3106884640}) 2024-02-09T17:47:29.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=816744445}) 2024-02-09T17:47:29.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3231824972}) 2024-02-09T17:47:29.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1330191448}) 2024-02-09T17:47:29.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1192603084}) 2024-02-09T17:47:29.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1540686426}) 2024-02-09T17:47:29.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3977176247}) 2024-02-09T17:47:29.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1882229734}) 2024-02-09T17:47:29.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3384756326}) 2024-02-09T17:47:29.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2749043887}) 2024-02-09T17:47:29.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3894995540}) 2024-02-09T17:47:29.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=376524431}) 2024-02-09T17:47:29.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1824554594}) 2024-02-09T17:47:29.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=29143596}) 2024-02-09T17:47:29.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3120815440}) 2024-02-09T17:47:29.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2518284121}) 2024-02-09T17:47:29.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2241659129}) 2024-02-09T17:47:29.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=673005013}) 2024-02-09T17:47:29.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1486639706}) 2024-02-09T17:47:29.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3715199670}) 2024-02-09T17:47:29.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=472543172}) 2024-02-09T17:47:29.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=793984776}) 2024-02-09T17:47:29.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1933228581}) 2024-02-09T17:47:29.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=311503825}) 2024-02-09T17:47:29.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4054732757}) 2024-02-09T17:47:29.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2363110713}) 2024-02-09T17:47:29.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3731821054}) 2024-02-09T17:47:29.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=254568500}) 2024-02-09T17:47:29.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3165554103}) 2024-02-09T17:47:29.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=174592252}) 2024-02-09T17:47:29.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3291636279}) 2024-02-09T17:47:29.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2727126295}) 2024-02-09T17:47:29.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4141718682}) 2024-02-09T17:47:29.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3903908258}) 2024-02-09T17:47:29.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3087133499}) 2024-02-09T17:47:29.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1051785243}) 2024-02-09T17:47:29.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3211850045}) 2024-02-09T17:47:29.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3607913009}) 2024-02-09T17:47:29.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=81270189}) 2024-02-09T17:47:29.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2590699091}) 2024-02-09T17:47:29.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2091212143}) 2024-02-09T17:47:29.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=55796120}) 2024-02-09T17:47:29.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3669075404}) 2024-02-09T17:47:29.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551701061}) 2024-02-09T17:47:29.183 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=54538955}) 2024-02-09T17:47:29.183 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:28 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3910626320}) 2024-02-09T17:47:29.184 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:47:29.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:29.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:47:29.293 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:47:29.339 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:29.373 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:47:29.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:29.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:47:29.451 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-09T17:47:29.488 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:47:29.522 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:29.546 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:47:29.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:29.551 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:47:29.652 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:47:29.739 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:29.876 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:47:29.876 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:47:29.880 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:47:29.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:29.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:47:29.989 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:47:29.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4121411338}) 2024-02-09T17:47:29.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1615154294}) 2024-02-09T17:47:29.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1892346308}) 2024-02-09T17:47:29.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1827370008}) 2024-02-09T17:47:29.998 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3756980824}) 2024-02-09T17:47:29.998 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2442508167}) 2024-02-09T17:47:29.998 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3569757417}) 2024-02-09T17:47:29.998 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=520570657}) 2024-02-09T17:47:29.998 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=29,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=9,mon_config_key=38} crc {logm=434314971,mdsmap=3984425268,mgr=1708625883,mgr_command_descs=708342470,mgr_metadata=670763949,mgrstat=2364153727,mon_config_key=1417369807}) 2024-02-09T17:47:29.998 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1025931032}) 2024-02-09T17:47:29.998 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2286369753}) 2024-02-09T17:47:29.998 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=43,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=39} crc {mon_config_key=4088680394,monmap=423865542,osd_metadata=545325961,osd_pg_creating=2002903617,osdmap=1097793631}) 2024-02-09T17:47:29.998 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=70} crc {osdmap=247312547}) 2024-02-09T17:47:29.998 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:29 smithi099 ceph-mon[39420]: pgmap v1536: 47 pgs: 47 active+clean; 7.7 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 126 B/s rd, 31 MiB/s wr, 1.04k op/s 2024-02-09T17:47:30.038 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:30.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4121411338}) 2024-02-09T17:47:30.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1615154294}) 2024-02-09T17:47:30.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1892346308}) 2024-02-09T17:47:30.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1827370008}) 2024-02-09T17:47:30.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3756980824}) 2024-02-09T17:47:30.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2442508167}) 2024-02-09T17:47:30.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3569757417}) 2024-02-09T17:47:30.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=520570657}) 2024-02-09T17:47:30.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=29,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=9,mon_config_key=38} crc {logm=434314971,mdsmap=3984425268,mgr=1708625883,mgr_command_descs=708342470,mgr_metadata=670763949,mgrstat=2364153727,mon_config_key=1417369807}) 2024-02-09T17:47:30.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1025931032}) 2024-02-09T17:47:30.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2286369753}) 2024-02-09T17:47:30.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=43,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=39} crc {mon_config_key=4088680394,monmap=423865542,osd_metadata=545325961,osd_pg_creating=2002903617,osdmap=1097793631}) 2024-02-09T17:47:30.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=70} crc {osdmap=247312547}) 2024-02-09T17:47:30.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:29 smithi179 ceph-mon[43975]: pgmap v1536: 47 pgs: 47 active+clean; 7.7 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 126 B/s rd, 31 MiB/s wr, 1.04k op/s 2024-02-09T17:47:30.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4121411338}) 2024-02-09T17:47:30.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1615154294}) 2024-02-09T17:47:30.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1892346308}) 2024-02-09T17:47:30.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1827370008}) 2024-02-09T17:47:30.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3756980824}) 2024-02-09T17:47:30.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2442508167}) 2024-02-09T17:47:30.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3569757417}) 2024-02-09T17:47:30.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=520570657}) 2024-02-09T17:47:30.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=29,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=9,mon_config_key=38} crc {logm=434314971,mdsmap=3984425268,mgr=1708625883,mgr_command_descs=708342470,mgr_metadata=670763949,mgrstat=2364153727,mon_config_key=1417369807}) 2024-02-09T17:47:30.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1025931032}) 2024-02-09T17:47:30.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2286369753}) 2024-02-09T17:47:30.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=43,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=39} crc {mon_config_key=4088680394,monmap=423865542,osd_metadata=545325961,osd_pg_creating=2002903617,osdmap=1097793631}) 2024-02-09T17:47:30.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=70} crc {osdmap=247312547}) 2024-02-09T17:47:30.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:29 smithi162 ceph-mon[44304]: pgmap v1536: 47 pgs: 47 active+clean; 7.7 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 126 B/s rd, 31 MiB/s wr, 1.04k op/s 2024-02-09T17:47:30.269 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:47:30.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:30.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:47:30.377 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:47:30.417 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:30.441 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:47:30.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:30.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:47:30.548 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:47:30.593 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:30.740 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:47:30.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:30.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:47:30.815 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is set 2024-02-09T17:47:30.845 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:47:30.876 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:30.895 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:47:30.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:30.899 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:47:30.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:30 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1359188440' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:47:30.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:30 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:47:30.999 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:47:31.032 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:31.055 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:47:31.070 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:30 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1359188440' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:47:31.070 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:30 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:47:31.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:31.071 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:47:31.172 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:47:31.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:30 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1359188440' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:47:31.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:30 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:47:31.208 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:31.248 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:47:31.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:31.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:47:31.321 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:47:31.359 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:47:31.413 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:31.451 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:47:31.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:31.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:47:31.555 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:47:31.587 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:31.615 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:47:31.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:31.619 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:47:31.720 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:47:31.764 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:31.792 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:47:31.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:31.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:47:31.897 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:47:31.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:31 smithi179 ceph-mon[43975]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:47:31.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:31 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:47:31.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:31 smithi179 ceph-mon[43975]: osdmap e615: 12 total, 12 up, 12 in 2024-02-09T17:47:31.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:31 smithi179 ceph-mon[43975]: pgmap v1538: 47 pgs: 47 active+clean; 7.6 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 6.2 MiB/s wr, 750 op/s 2024-02-09T17:47:31.972 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:32.004 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:47:32.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:32.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:47:32.110 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:47:32.144 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:31 smithi099 ceph-mon[39420]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:47:32.144 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:31 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:47:32.144 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:31 smithi099 ceph-mon[39420]: osdmap e615: 12 total, 12 up, 12 in 2024-02-09T17:47:32.144 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:31 smithi099 ceph-mon[39420]: pgmap v1538: 47 pgs: 47 active+clean; 7.6 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 6.2 MiB/s wr, 750 op/s 2024-02-09T17:47:32.146 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:32.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:31 smithi162 ceph-mon[44304]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:47:32.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:31 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:47:32.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:31 smithi162 ceph-mon[44304]: osdmap e615: 12 total, 12 up, 12 in 2024-02-09T17:47:32.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:31 smithi162 ceph-mon[44304]: pgmap v1538: 47 pgs: 47 active+clean; 7.6 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 6.2 MiB/s wr, 750 op/s 2024-02-09T17:47:33.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:32 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:47:33.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:32 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:47:33.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:32 smithi179 ceph-mon[43975]: osdmap e616: 12 total, 12 up, 12 in 2024-02-09T17:47:33.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:32 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:47:33.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:32 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:47:33.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:32 smithi162 ceph-mon[44304]: osdmap e616: 12 total, 12 up, 12 in 2024-02-09T17:47:33.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:32 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:47:33.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:32 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:47:33.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:32 smithi099 ceph-mon[39420]: osdmap e616: 12 total, 12 up, 12 in 2024-02-09T17:47:33.318 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:47:33.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:33.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:47:33.424 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:47:33.460 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:33.489 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:47:33.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:33.495 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:47:33.596 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:47:33.713 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:33.736 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:47:33.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:33.740 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:47:33.842 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:47:33.882 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:33.905 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:47:33.909 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:33.909 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:47:34.010 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:47:34.082 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:34.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:34.116 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:33 smithi099 ceph-mon[39420]: pgmap v1540: 47 pgs: 47 active+clean; 7.4 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s wr, 1.09k op/s 2024-02-09T17:47:34.117 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:47:34.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:47:34.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:33 smithi179 ceph-mon[43975]: pgmap v1540: 47 pgs: 47 active+clean; 7.4 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s wr, 1.09k op/s 2024-02-09T17:47:34.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:33 smithi162 ceph-mon[44304]: pgmap v1540: 47 pgs: 47 active+clean; 7.4 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s wr, 1.09k op/s 2024-02-09T17:47:34.218 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:47:34.249 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:34.272 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:47:34.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:34.277 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:47:34.378 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:47:34.414 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:34.435 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:47:34.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:34.438 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:47:34.539 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:47:34.572 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:34.594 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:47:34.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:34.597 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:47:34.698 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:47:34.730 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:34.766 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:47:34.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:34.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:47:34.874 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:47:34.900 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:34.930 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:47:34.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:34.936 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:47:35.037 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:47:35.083 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:35.216 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:47:35.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:35.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:47:35.283 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-09T17:47:35.322 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:47:35.353 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:35.382 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:47:35.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:35.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:47:35.487 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:47:35.576 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:36.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:36 smithi179 ceph-mon[43975]: pgmap v1541: 47 pgs: 47 active+clean; 7.4 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s wr, 1.01k op/s 2024-02-09T17:47:36.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:36 smithi162 ceph-mon[44304]: pgmap v1541: 47 pgs: 47 active+clean; 7.4 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s wr, 1.01k op/s 2024-02-09T17:47:36.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:36 smithi099 ceph-mon[39420]: pgmap v1541: 47 pgs: 47 active+clean; 7.4 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s wr, 1.01k op/s 2024-02-09T17:47:36.699 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:47:36.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:36.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:47:36.805 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:47:36.854 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:36.886 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:47:36.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:36.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:47:36.991 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:47:37.029 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:37.058 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:47:37.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:37.065 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:47:37.165 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:47:37.202 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:37.234 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:47:37.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:37.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:47:37.339 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:47:37.365 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:47:37.368 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:37.393 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:47:37.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:37.395 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:47:37.408 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:47:37.441 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:47:37.483 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:47:37.496 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:47:37.524 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:37.531 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:47:37.544 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:47:37.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:37.547 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:47:37.578 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:47:37.614 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:47:37.614 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [], 2024-02-09T17:47:37.614 INFO:teuthology.orchestra.run.smithi099.stdout: "num_ops": 0 2024-02-09T17:47:37.614 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:47:37.629 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:47:37.647 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:47:37.675 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:47:37.710 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:47:37.754 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:47:37.794 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:47:37.827 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:47:37.842 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:47:37.842 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:47:37.842 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:47:37.842 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:47:37.842 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:47:37.842 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:47:37.843 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:47:37.843 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:47:37.843 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:47:37.843 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:47:37.843 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:47:37.843 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:47:37.843 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:37.873 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:47:37.876 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:37.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:47:37.978 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:47:38.011 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:38.038 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:47:38.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:38.042 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:47:38.143 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:47:38.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:38 smithi162 ceph-mon[44304]: pgmap v1542: 47 pgs: 47 active+clean; 7.3 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 853 op/s 2024-02-09T17:47:38.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:38 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1439245741' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:47:38.199 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:38.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:38 smithi179 ceph-mon[43975]: pgmap v1542: 47 pgs: 47 active+clean; 7.3 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 853 op/s 2024-02-09T17:47:38.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:38 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1439245741' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:47:38.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:38 smithi099 ceph-mon[39420]: pgmap v1542: 47 pgs: 47 active+clean; 7.3 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 853 op/s 2024-02-09T17:47:38.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:38 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1439245741' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:47:39.988 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:47:39.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:39.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:47:40.047 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.5 to 1 (10000) 2024-02-09T17:47:40.098 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:47:40.139 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:40.165 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:47:40.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:40.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:47:40.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:40 smithi099 ceph-mon[39420]: pgmap v1543: 47 pgs: 47 active+clean; 7.1 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-02-09T17:47:40.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:40 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3163857997' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-09T17:47:40.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:40 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-09T17:47:40.280 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:47:40.330 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:40.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:40 smithi179 ceph-mon[43975]: pgmap v1543: 47 pgs: 47 active+clean; 7.1 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-02-09T17:47:40.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:40 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3163857997' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-09T17:47:40.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:40 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-09T17:47:40.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:40 smithi162 ceph-mon[44304]: pgmap v1543: 47 pgs: 47 active+clean; 7.1 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-02-09T17:47:40.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:40 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3163857997' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-09T17:47:40.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:40 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-09T17:47:40.715 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:47:40.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:40.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:47:40.761 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-09T17:47:40.825 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:47:40.921 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:41.181 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:47:41.189 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:47:41.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:41.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:47:41.224 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 6 1 2024-02-09T17:47:41.305 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:47:41.319 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:41 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-09T17:47:41.319 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:41 smithi099 ceph-mon[39420]: osdmap e617: 12 total, 12 up, 12 in 2024-02-09T17:47:41.319 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:41 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:47:41.319 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:41 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:47:41.319 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:41 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:47:41.319 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:41 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:47:41.319 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:41 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:47:41.338 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:41 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-09T17:47:41.338 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:41 smithi162 ceph-mon[44304]: osdmap e617: 12 total, 12 up, 12 in 2024-02-09T17:47:41.338 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:41 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:47:41.339 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:41 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:47:41.339 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:41 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:47:41.339 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:41 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:47:41.339 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:41 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:47:41.339 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:41.375 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:47:41.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:41.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:47:41.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:41 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-09T17:47:41.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:41 smithi179 ceph-mon[43975]: osdmap e617: 12 total, 12 up, 12 in 2024-02-09T17:47:41.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:41 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:47:41.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:41 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:47:41.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:41 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:47:41.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:41 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:47:41.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:41 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:47:41.481 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:47:41.608 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:41.650 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:47:41.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:41.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:47:41.755 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:47:41.831 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:41.868 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:47:41.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:41.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:47:41.942 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:47:41.975 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:47:42.018 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:42.041 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:47:42.044 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:42.045 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:47:42.146 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:47:42.211 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:47:42.220 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:42.246 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:47:42.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:42 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:47:42.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:42 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:47:42.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:42 smithi099 ceph-mon[39420]: pgmap v1545: 47 pgs: 47 active+clean; 7.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 894 op/s 2024-02-09T17:47:42.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:42 smithi099 ceph-mon[39420]: osdmap e618: 12 total, 12 up, 12 in 2024-02-09T17:47:42.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:42 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:47:42.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:42 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:47:42.248 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:42 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:47:42.248 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:42 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:47:42.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:42.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:47:42.350 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:47:42.386 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:42.386 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:42 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:47:42.386 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:42 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:47:42.386 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:42 smithi162 ceph-mon[44304]: pgmap v1545: 47 pgs: 47 active+clean; 7.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 894 op/s 2024-02-09T17:47:42.387 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:42 smithi162 ceph-mon[44304]: osdmap e618: 12 total, 12 up, 12 in 2024-02-09T17:47:42.387 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:42 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:47:42.387 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:42 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:47:42.387 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:42 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:47:42.387 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:42 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:47:42.421 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:47:42.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:42 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:47:42.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:42 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:47:42.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:42 smithi179 ceph-mon[43975]: pgmap v1545: 47 pgs: 47 active+clean; 7.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 894 op/s 2024-02-09T17:47:42.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:42 smithi179 ceph-mon[43975]: osdmap e618: 12 total, 12 up, 12 in 2024-02-09T17:47:42.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:42 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:47:42.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:42 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:47:42.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:42 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:47:42.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:42 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:47:42.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:42.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:47:42.527 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:47:42.581 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:42.854 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:47:42.856 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:47:42.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:42.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:47:42.968 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:47:43.004 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:43.033 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:47:43.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:43.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:47:43.141 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:47:43.177 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:43.207 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:47:43.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:43.211 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:47:43.312 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:47:43.361 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:43.387 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:47:43.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:43.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:47:43.492 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:47:43.519 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:43.539 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:47:43.542 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:43.543 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:47:43.644 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:47:43.672 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:43.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:43 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:47:43.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:43 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:47:43.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:43 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:47:43.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:43 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:47:43.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:43 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:47:43.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:43 smithi162 ceph-mon[44304]: pgmap v1547: 47 pgs: 47 active+clean; 6.9 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s 2024-02-09T17:47:43.703 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:47:43.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:43.709 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:47:43.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:43 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:47:43.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:43 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:47:43.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:43 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:47:43.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:43 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:47:43.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:43 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:47:43.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:43 smithi099 ceph-mon[39420]: pgmap v1547: 47 pgs: 47 active+clean; 6.9 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s 2024-02-09T17:47:43.810 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:47:43.883 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:43.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:43 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:47:43.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:43 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:47:43.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:43 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:47:43.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:43 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:47:43.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:43 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:47:43.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:43 smithi179 ceph-mon[43975]: pgmap v1547: 47 pgs: 47 active+clean; 6.9 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s 2024-02-09T17:47:43.949 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:47:43.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:43.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:47:44.054 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:47:44.095 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:44.132 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:47:44.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:44.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:47:44.237 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:47:44.276 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:44.305 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:47:44.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:44.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:47:44.410 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:47:44.456 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:44.482 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:47:44.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:44.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:47:44.586 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:47:44.615 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:44.639 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:47:44.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:44.642 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:47:44.670 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:47:44.743 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:47:44.772 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:44.794 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:47:44.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:44.797 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:47:44.841 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is set 2024-02-09T17:47:44.898 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:47:44.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:44 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/878423879' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:47:44.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:44 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:47:44.969 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:45.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:44 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/878423879' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:47:45.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:44 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:47:45.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:44 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/878423879' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:47:45.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:44 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:47:45.546 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:47:45.552 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:45.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:47:45.653 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:47:45.709 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:45.795 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:47:45.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:45.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:47:45.904 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:47:45.975 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:45 smithi099 ceph-mon[39420]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:47:45.975 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:45 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:47:45.975 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:45 smithi099 ceph-mon[39420]: osdmap e619: 12 total, 12 up, 12 in 2024-02-09T17:47:45.975 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:45 smithi099 ceph-mon[39420]: pgmap v1549: 47 pgs: 47 active+clean; 6.8 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 809 op/s 2024-02-09T17:47:45.975 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:45 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:47:45.975 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:45 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:47:45.977 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:46.063 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:47:46.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:46.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:47:46.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:45 smithi179 ceph-mon[43975]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:47:46.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:45 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:47:46.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:45 smithi179 ceph-mon[43975]: osdmap e619: 12 total, 12 up, 12 in 2024-02-09T17:47:46.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:45 smithi179 ceph-mon[43975]: pgmap v1549: 47 pgs: 47 active+clean; 6.8 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 809 op/s 2024-02-09T17:47:46.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:45 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:47:46.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:45 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:47:46.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:45 smithi162 ceph-mon[44304]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:47:46.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:45 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:47:46.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:45 smithi162 ceph-mon[44304]: osdmap e619: 12 total, 12 up, 12 in 2024-02-09T17:47:46.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:45 smithi162 ceph-mon[44304]: pgmap v1549: 47 pgs: 47 active+clean; 6.8 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 809 op/s 2024-02-09T17:47:46.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:45 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:47:46.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:45 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:47:46.177 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:47:46.208 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:46.229 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:47:46.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:46.232 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:47:46.333 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:47:46.390 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:46.870 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.6 to 1 (10000) 2024-02-09T17:47:47.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:46 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:47:47.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:46 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:47:47.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:46 smithi179 ceph-mon[43975]: osdmap e620: 12 total, 12 up, 12 in 2024-02-09T17:47:47.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:46 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:47:47.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:46 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:47:47.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:46 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2865553250' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-09T17:47:47.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:46 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-09T17:47:47.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:46 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:47:47.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:46 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:47:47.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:46 smithi162 ceph-mon[44304]: osdmap e620: 12 total, 12 up, 12 in 2024-02-09T17:47:47.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:46 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:47:47.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:46 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:47:47.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:46 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2865553250' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-09T17:47:47.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:46 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-09T17:47:47.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:46 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:47:47.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:46 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:47:47.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:46 smithi099 ceph-mon[39420]: osdmap e620: 12 total, 12 up, 12 in 2024-02-09T17:47:47.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:46 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:47:47.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:46 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:47:47.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:46 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2865553250' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-09T17:47:47.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:46 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-09T17:47:48.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:47 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-09T17:47:48.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:47 smithi179 ceph-mon[43975]: osdmap e621: 12 total, 12 up, 12 in 2024-02-09T17:47:48.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:47 smithi179 ceph-mon[43975]: pgmap v1552: 47 pgs: 47 active+clean; 6.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 818 op/s 2024-02-09T17:47:48.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:47 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-09T17:47:48.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:47 smithi162 ceph-mon[44304]: osdmap e621: 12 total, 12 up, 12 in 2024-02-09T17:47:48.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:47 smithi162 ceph-mon[44304]: pgmap v1552: 47 pgs: 47 active+clean; 6.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 818 op/s 2024-02-09T17:47:48.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:47 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-09T17:47:48.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:47 smithi099 ceph-mon[39420]: osdmap e621: 12 total, 12 up, 12 in 2024-02-09T17:47:48.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:47 smithi099 ceph-mon[39420]: pgmap v1552: 47 pgs: 47 active+clean; 6.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 818 op/s 2024-02-09T17:47:49.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:48 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:47:49.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:48 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:47:49.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:48 smithi179 ceph-mon[43975]: osdmap e622: 12 total, 12 up, 12 in 2024-02-09T17:47:49.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:48 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:47:49.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:48 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:47:49.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:48 smithi162 ceph-mon[44304]: osdmap e622: 12 total, 12 up, 12 in 2024-02-09T17:47:49.200 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:48 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:47:49.200 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:48 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:47:49.200 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:48 smithi099 ceph-mon[39420]: osdmap e622: 12 total, 12 up, 12 in 2024-02-09T17:47:49.722 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:47:49.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:49.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:47:49.829 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:47:49.884 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:49.921 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:47:49.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:49.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:47:49.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:49 smithi099 ceph-mon[39420]: pgmap v1554: 47 pgs: 47 active+clean; 6.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s 2024-02-09T17:47:50.030 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:47:50.058 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:50.085 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:47:50.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:50.092 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:47:50.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:49 smithi179 ceph-mon[43975]: pgmap v1554: 47 pgs: 47 active+clean; 6.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s 2024-02-09T17:47:50.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:49 smithi162 ceph-mon[44304]: pgmap v1554: 47 pgs: 47 active+clean; 6.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s 2024-02-09T17:47:50.193 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:47:50.256 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:50.364 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:47:50.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:50.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:47:50.468 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:47:50.537 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:50.648 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:47:50.680 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:47:50.713 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:47:50.750 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:47:50.781 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:47:50.807 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:47:50.832 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:47:50.858 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:47:50.882 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:47:50.906 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:47:50.928 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:47:50.949 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:47:50.991 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:47:50.991 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:47:50.991 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:47:50.991 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:47:50.991 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:47:50.991 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:47:50.991 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:47:50.991 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:47:50.991 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:47:50.991 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:47:50.991 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:47:50.991 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:47:51.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:50 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1476069543' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:47:51.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:50 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1476069543' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:47:51.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:50 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1476069543' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:47:52.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:51 smithi179 ceph-mon[43975]: pgmap v1555: 47 pgs: 47 active+clean; 6.5 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 792 op/s 2024-02-09T17:47:52.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:51 smithi162 ceph-mon[44304]: pgmap v1555: 47 pgs: 47 active+clean; 6.5 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 792 op/s 2024-02-09T17:47:52.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:51 smithi099 ceph-mon[39420]: pgmap v1555: 47 pgs: 47 active+clean; 6.5 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 792 op/s 2024-02-09T17:47:52.338 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:47:52.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:52.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:47:52.355 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 7 1 2024-02-09T17:47:52.444 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:47:52.492 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:52.516 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:47:52.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:52.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:47:52.621 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:47:52.648 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:52.673 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:47:52.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:52.678 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:47:52.779 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:47:52.821 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:52.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:52.880 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:47:52.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:47:52.981 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:47:53.054 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:53.077 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:47:53.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:53.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:47:53.186 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:47:53.215 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:53.236 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:47:53.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:53.241 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:47:53.342 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:47:53.372 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:53.397 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:47:53.402 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:53.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:47:53.460 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:47:53.460 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:47:53.504 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:47:53.530 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:53.553 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:47:53.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:53.560 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:47:53.661 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:47:53.693 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:53.715 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:47:53.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:53.723 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:47:53.824 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:47:53.853 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:53.878 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:47:53.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:53.882 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:47:53.983 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:47:54.041 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:54.080 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:47:54.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:54.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:47:54.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:54 smithi179 ceph-mon[43975]: pgmap v1556: 47 pgs: 47 active+clean; 6.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s 2024-02-09T17:47:54.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:54 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:47:54.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:54 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:47:54.185 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:47:54.223 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:47:54.242 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:54.321 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:54 smithi099 ceph-mon[39420]: pgmap v1556: 47 pgs: 47 active+clean; 6.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s 2024-02-09T17:47:54.321 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:54 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:47:54.321 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:54 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:47:54.323 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:47:54.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:54.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:47:54.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:54 smithi162 ceph-mon[44304]: pgmap v1556: 47 pgs: 47 active+clean; 6.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s 2024-02-09T17:47:54.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:54 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:47:54.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:54 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:47:54.442 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:47:54.475 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:54.504 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:47:54.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:54.507 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:47:54.608 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:47:54.640 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:54.666 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:47:54.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:54.669 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:47:54.692 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-09T17:47:54.770 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:47:54.797 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:54.822 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:47:54.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:54.829 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:47:54.930 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:47:55.020 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:55.068 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset noscrub 2024-02-09T17:47:55.068 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:47:55.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:55.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:47:55.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:55 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:47:55.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:55 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:47:55.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:55 smithi179 ceph-mon[43975]: osdmap e623: 12 total, 12 up, 12 in 2024-02-09T17:47:55.177 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:47:55.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:55 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:47:55.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:55 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:47:55.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:55 smithi099 ceph-mon[39420]: osdmap e623: 12 total, 12 up, 12 in 2024-02-09T17:47:55.258 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:55.431 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:55 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:47:55.431 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:55 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:47:55.431 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:55 smithi162 ceph-mon[44304]: osdmap e623: 12 total, 12 up, 12 in 2024-02-09T17:47:55.559 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:47:55.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:55.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:47:55.666 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:47:55.718 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:55.748 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:47:55.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:55.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:47:55.859 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:47:55.890 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:55.914 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:47:55.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:55.922 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:47:56.023 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:47:56.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:56 smithi162 ceph-mon[44304]: pgmap v1558: 47 pgs: 47 active+clean; 6.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 897 op/s 2024-02-09T17:47:56.056 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:56.081 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:47:56.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:56.085 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:47:56.160 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:47:56.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:56 smithi179 ceph-mon[43975]: pgmap v1558: 47 pgs: 47 active+clean; 6.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 897 op/s 2024-02-09T17:47:56.174 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:47:56.186 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:47:56.213 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:56.236 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:47:56.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:56.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:47:56.342 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:47:56.371 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:56.393 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:47:56.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:56.398 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:47:56.478 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:56 smithi099 ceph-mon[39420]: pgmap v1558: 47 pgs: 47 active+clean; 6.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 897 op/s 2024-02-09T17:47:56.499 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:47:56.556 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:57.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:57 smithi099 ceph-mon[39420]: pgmap v1559: 47 pgs: 47 active+clean; 6.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 694 op/s 2024-02-09T17:47:57.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:57 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:47:57.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:57 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:47:58.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:57 smithi179 ceph-mon[43975]: pgmap v1559: 47 pgs: 47 active+clean; 6.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 694 op/s 2024-02-09T17:47:58.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:57 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:47:58.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:57 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:47:58.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:57 smithi162 ceph-mon[44304]: pgmap v1559: 47 pgs: 47 active+clean; 6.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 694 op/s 2024-02-09T17:47:58.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:57 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:47:58.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:57 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:47:58.340 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:47:58.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:58.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:47:58.445 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:47:58.502 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:58.542 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:47:58.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:58.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:47:58.646 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:47:58.684 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:58.706 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:58 smithi099 ceph-mon[39420]: osdmap e624: 12 total, 12 up, 12 in 2024-02-09T17:47:58.706 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:58 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1625102737' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-09T17:47:58.707 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:58 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-09T17:47:58.712 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:47:58.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:58.716 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:47:58.755 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.7 to 1 (10000) 2024-02-09T17:47:58.817 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:47:58.864 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:58 smithi162 ceph-mon[44304]: osdmap e624: 12 total, 12 up, 12 in 2024-02-09T17:47:58.864 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:58 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1625102737' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-09T17:47:58.864 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:58 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-09T17:47:58.873 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:58.899 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:47:58.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:58.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:47:59.005 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:47:59.066 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:59.130 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:47:59.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:59.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:47:59.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:58 smithi179 ceph-mon[43975]: osdmap e624: 12 total, 12 up, 12 in 2024-02-09T17:47:59.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:58 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1625102737' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-09T17:47:59.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:58 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-09T17:47:59.234 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:47:59.287 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:59.668 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:47:59.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:59.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:47:59.776 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:47:59.817 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:47:59.848 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:47:59.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:47:59.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:47:59.959 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:47:59.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:59 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-09T17:47:59.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:59 smithi099 ceph-mon[39420]: osdmap e625: 12 total, 12 up, 12 in 2024-02-09T17:47:59.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:59 smithi099 ceph-mon[39420]: pgmap v1562: 47 pgs: 47 active+clean; 6.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 741 op/s 2024-02-09T17:47:59.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:59 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:47:59.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:47:59 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:48:00.031 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:00.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:59 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-09T17:48:00.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:59 smithi179 ceph-mon[43975]: osdmap e625: 12 total, 12 up, 12 in 2024-02-09T17:48:00.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:59 smithi179 ceph-mon[43975]: pgmap v1562: 47 pgs: 47 active+clean; 6.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 741 op/s 2024-02-09T17:48:00.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:59 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:48:00.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:47:59 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:48:00.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:59 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-09T17:48:00.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:59 smithi162 ceph-mon[44304]: osdmap e625: 12 total, 12 up, 12 in 2024-02-09T17:48:00.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:59 smithi162 ceph-mon[44304]: pgmap v1562: 47 pgs: 47 active+clean; 6.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 741 op/s 2024-02-09T17:48:00.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:59 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:48:00.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:47:59 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:48:00.233 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:48:00.234 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:48:00.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:00.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:48:00.269 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 8 1 2024-02-09T17:48:00.338 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:48:00.376 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:00.400 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:48:00.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:00.407 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:48:00.508 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:48:00.539 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:00.578 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:48:00.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:00.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:48:00.685 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:48:00.754 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:00.807 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:48:00.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:00.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:48:00.820 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:00 smithi162 ceph-mon[44304]: osdmap e626: 12 total, 12 up, 12 in 2024-02-09T17:48:00.918 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:48:00.950 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:00.971 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:48:00.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:00.976 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:48:01.077 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:48:01.127 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:01.162 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:48:01.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:00 smithi179 ceph-mon[43975]: osdmap e626: 12 total, 12 up, 12 in 2024-02-09T17:48:01.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:01.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:48:01.248 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:00 smithi099 ceph-mon[39420]: osdmap e626: 12 total, 12 up, 12 in 2024-02-09T17:48:01.274 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:48:01.301 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:01.323 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:48:01.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:01.326 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:48:01.427 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:48:01.487 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:01.544 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:48:01.546 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:48:01.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:01.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:48:01.651 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:48:01.706 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:01.729 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:48:01.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:01.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:48:01.835 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:48:01.894 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:02.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:01 smithi179 ceph-mon[43975]: pgmap v1564: 47 pgs: 47 active+clean; 6.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 744 op/s 2024-02-09T17:48:02.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:01 smithi162 ceph-mon[44304]: pgmap v1564: 47 pgs: 47 active+clean; 6.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 744 op/s 2024-02-09T17:48:02.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:01 smithi099 ceph-mon[39420]: pgmap v1564: 47 pgs: 47 active+clean; 6.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 744 op/s 2024-02-09T17:48:03.361 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:48:03.361 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:48:03.615 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:48:03.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:03.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:48:03.721 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:48:03.762 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:03.788 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:48:03.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:03.802 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:48:03.903 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:48:03.967 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:04.022 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:04 smithi179 ceph-mon[43975]: pgmap v1565: 47 pgs: 47 active+clean; 5.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-02-09T17:48:04.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:04 smithi099 ceph-mon[39420]: pgmap v1565: 47 pgs: 47 active+clean; 5.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-02-09T17:48:04.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:04 smithi162 ceph-mon[44304]: pgmap v1565: 47 pgs: 47 active+clean; 5.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-02-09T17:48:05.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:05 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3504887975' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:48:05.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:05 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3504887975' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:48:05.431 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:05 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3504887975' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:48:05.475 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-09T17:48:05.700 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is unset 2024-02-09T17:48:05.956 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:48:05.970 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:48:05.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:05.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:48:06.085 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:48:06.140 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:06.171 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:48:06.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:06.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:48:06.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:06 smithi099 ceph-mon[39420]: pgmap v1566: 47 pgs: 47 active+clean; 5.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 688 op/s 2024-02-09T17:48:06.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:06 smithi099 ceph-mon[39420]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:48:06.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:06 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3504887975' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:48:06.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:06 smithi099 ceph-mon[39420]: osdmap e627: 12 total, 12 up, 12 in 2024-02-09T17:48:06.278 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:48:06.309 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:06 smithi179 ceph-mon[43975]: pgmap v1566: 47 pgs: 47 active+clean; 5.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 688 op/s 2024-02-09T17:48:06.309 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:06 smithi179 ceph-mon[43975]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:48:06.309 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:06 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3504887975' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:48:06.309 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:06 smithi179 ceph-mon[43975]: osdmap e627: 12 total, 12 up, 12 in 2024-02-09T17:48:06.310 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:06.338 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:48:06.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:06.342 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:48:06.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:06 smithi162 ceph-mon[44304]: pgmap v1566: 47 pgs: 47 active+clean; 5.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 688 op/s 2024-02-09T17:48:06.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:06 smithi162 ceph-mon[44304]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:48:06.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:06 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3504887975' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:48:06.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:06 smithi162 ceph-mon[44304]: osdmap e627: 12 total, 12 up, 12 in 2024-02-09T17:48:06.442 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:48:06.474 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:06.507 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:48:06.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:06.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:48:06.614 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:48:06.665 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:06.697 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:48:06.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:06.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:48:06.807 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:48:06.848 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:06.876 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:48:06.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:06.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:48:06.983 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:48:07.019 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:07.048 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:48:07.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:07.056 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:48:07.157 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:48:07.208 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:07.233 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:48:07.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:07.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:48:07.337 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:48:07.400 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:07.703 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:07 smithi099 ceph-mon[39420]: pgmap v1568: 47 pgs: 47 active+clean; 5.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 753 op/s 2024-02-09T17:48:07.770 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:48:07.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:07.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:48:07.879 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:48:07.931 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:07.961 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:48:07.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:07.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:48:07.965 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:07 smithi162 ceph-mon[44304]: pgmap v1568: 47 pgs: 47 active+clean; 5.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 753 op/s 2024-02-09T17:48:08.066 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:48:08.143 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:08.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:07 smithi179 ceph-mon[43975]: pgmap v1568: 47 pgs: 47 active+clean; 5.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 753 op/s 2024-02-09T17:48:08.204 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:48:08.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:08.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:48:08.313 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:48:08.396 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:08.433 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:48:08.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:08.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:48:08.545 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:48:08.593 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:08.666 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:48:08.694 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:08.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:48:08.795 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:48:08.841 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:08.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:08 smithi099 ceph-mon[39420]: 1.6 scrub starts 2024-02-09T17:48:08.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:08 smithi099 ceph-mon[39420]: 1.6 scrub ok 2024-02-09T17:48:08.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:08 smithi099 ceph-mon[39420]: 1.a scrub starts 2024-02-09T17:48:08.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:08 smithi099 ceph-mon[39420]: 1.a scrub ok 2024-02-09T17:48:09.028 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-09T17:48:09.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:08 smithi179 ceph-mon[43975]: 1.6 scrub starts 2024-02-09T17:48:09.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:08 smithi179 ceph-mon[43975]: 1.6 scrub ok 2024-02-09T17:48:09.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:08 smithi179 ceph-mon[43975]: 1.a scrub starts 2024-02-09T17:48:09.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:08 smithi179 ceph-mon[43975]: 1.a scrub ok 2024-02-09T17:48:09.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:08 smithi162 ceph-mon[44304]: 1.6 scrub starts 2024-02-09T17:48:09.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:08 smithi162 ceph-mon[44304]: 1.6 scrub ok 2024-02-09T17:48:09.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:08 smithi162 ceph-mon[44304]: 1.a scrub starts 2024-02-09T17:48:09.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:08 smithi162 ceph-mon[44304]: 1.a scrub ok 2024-02-09T17:48:09.221 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:48:09.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:09.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:48:09.330 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:48:09.377 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:09.416 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:48:09.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:09.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:48:09.524 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:48:09.575 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:09.611 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:48:09.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:09.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:48:09.719 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:48:09.746 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:09.766 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:48:09.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:09.770 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:48:09.871 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:48:09.931 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:48:09.931 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:48:09.931 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:48:09.931 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:48:09.936 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:09.977 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:48:09.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:09 smithi099 ceph-mon[39420]: 1.19 scrub starts 2024-02-09T17:48:09.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:09 smithi099 ceph-mon[39420]: 1.19 scrub ok 2024-02-09T17:48:09.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:09 smithi099 ceph-mon[39420]: 1.16 scrub starts 2024-02-09T17:48:09.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:09 smithi099 ceph-mon[39420]: 1.16 scrub ok 2024-02-09T17:48:09.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:09 smithi099 ceph-mon[39420]: 1.17 scrub starts 2024-02-09T17:48:09.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:09 smithi099 ceph-mon[39420]: 1.17 scrub ok 2024-02-09T17:48:09.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:09 smithi099 ceph-mon[39420]: pgmap v1569: 47 pgs: 47 active+clean; 5.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 907 op/s 2024-02-09T17:48:09.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:09 smithi099 ceph-mon[39420]: 1.1 scrub starts 2024-02-09T17:48:09.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:09 smithi099 ceph-mon[39420]: 1.1 scrub ok 2024-02-09T17:48:09.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:09 smithi099 ceph-mon[39420]: 1.3 scrub starts 2024-02-09T17:48:10.018 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:48:10.051 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:48:10.095 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:48:10.128 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:48:10.151 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:48:10.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:09 smithi179 ceph-mon[43975]: 1.19 scrub starts 2024-02-09T17:48:10.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:09 smithi179 ceph-mon[43975]: 1.19 scrub ok 2024-02-09T17:48:10.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:09 smithi179 ceph-mon[43975]: 1.16 scrub starts 2024-02-09T17:48:10.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:09 smithi179 ceph-mon[43975]: 1.16 scrub ok 2024-02-09T17:48:10.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:09 smithi179 ceph-mon[43975]: 1.17 scrub starts 2024-02-09T17:48:10.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:09 smithi179 ceph-mon[43975]: 1.17 scrub ok 2024-02-09T17:48:10.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:09 smithi179 ceph-mon[43975]: pgmap v1569: 47 pgs: 47 active+clean; 5.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 907 op/s 2024-02-09T17:48:10.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:09 smithi179 ceph-mon[43975]: 1.1 scrub starts 2024-02-09T17:48:10.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:09 smithi179 ceph-mon[43975]: 1.1 scrub ok 2024-02-09T17:48:10.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:09 smithi179 ceph-mon[43975]: 1.3 scrub starts 2024-02-09T17:48:10.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:09 smithi162 ceph-mon[44304]: 1.19 scrub starts 2024-02-09T17:48:10.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:09 smithi162 ceph-mon[44304]: 1.19 scrub ok 2024-02-09T17:48:10.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:09 smithi162 ceph-mon[44304]: 1.16 scrub starts 2024-02-09T17:48:10.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:09 smithi162 ceph-mon[44304]: 1.16 scrub ok 2024-02-09T17:48:10.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:09 smithi162 ceph-mon[44304]: 1.17 scrub starts 2024-02-09T17:48:10.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:09 smithi162 ceph-mon[44304]: 1.17 scrub ok 2024-02-09T17:48:10.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:09 smithi162 ceph-mon[44304]: pgmap v1569: 47 pgs: 47 active+clean; 5.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 907 op/s 2024-02-09T17:48:10.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:09 smithi162 ceph-mon[44304]: 1.1 scrub starts 2024-02-09T17:48:10.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:09 smithi162 ceph-mon[44304]: 1.1 scrub ok 2024-02-09T17:48:10.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:09 smithi162 ceph-mon[44304]: 1.3 scrub starts 2024-02-09T17:48:10.187 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:48:10.229 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:48:10.241 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:48:10.241 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:48:10.241 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:48:10.241 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:48:10.241 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:48:10.241 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:48:10.241 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:48:10.241 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:48:10.241 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:48:10.241 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:48:10.241 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:48:10.241 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:48:11.175 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:10 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3813782178' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:48:11.176 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:10 smithi179 ceph-mon[43975]: 1.3 scrub ok 2024-02-09T17:48:11.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:10 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3813782178' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:48:11.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:10 smithi162 ceph-mon[44304]: 1.3 scrub ok 2024-02-09T17:48:11.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:10 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3813782178' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:48:11.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:10 smithi099 ceph-mon[39420]: 1.3 scrub ok 2024-02-09T17:48:11.531 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:48:11.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:11.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:48:11.641 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:48:11.669 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:11.696 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:48:11.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:11.700 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:48:11.772 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.8 to 1 (10000) 2024-02-09T17:48:11.801 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:48:11.809 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:11 smithi099 ceph-mon[39420]: 1.18 scrub starts 2024-02-09T17:48:11.809 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:11 smithi099 ceph-mon[39420]: 1.18 scrub ok 2024-02-09T17:48:11.809 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:11 smithi099 ceph-mon[39420]: 1.1c scrub starts 2024-02-09T17:48:11.809 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:11 smithi099 ceph-mon[39420]: 1.1c scrub ok 2024-02-09T17:48:11.809 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:11 smithi099 ceph-mon[39420]: pgmap v1570: 47 pgs: 47 active+clean; 5.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 840 op/s 2024-02-09T17:48:11.809 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:11 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/802360941' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-02-09T17:48:11.835 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:11.867 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:48:11.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:11.871 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:48:11.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:11 smithi179 ceph-mon[43975]: 1.18 scrub starts 2024-02-09T17:48:11.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:11 smithi179 ceph-mon[43975]: 1.18 scrub ok 2024-02-09T17:48:11.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:11 smithi179 ceph-mon[43975]: 1.1c scrub starts 2024-02-09T17:48:11.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:11 smithi179 ceph-mon[43975]: 1.1c scrub ok 2024-02-09T17:48:11.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:11 smithi179 ceph-mon[43975]: pgmap v1570: 47 pgs: 47 active+clean; 5.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 840 op/s 2024-02-09T17:48:11.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:11 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/802360941' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-02-09T17:48:11.972 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:48:12.000 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:12.020 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:48:12.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:12.024 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:48:12.125 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:48:12.167 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:12.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:11 smithi162 ceph-mon[44304]: 1.18 scrub starts 2024-02-09T17:48:12.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:11 smithi162 ceph-mon[44304]: 1.18 scrub ok 2024-02-09T17:48:12.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:11 smithi162 ceph-mon[44304]: 1.1c scrub starts 2024-02-09T17:48:12.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:11 smithi162 ceph-mon[44304]: 1.1c scrub ok 2024-02-09T17:48:12.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:11 smithi162 ceph-mon[44304]: pgmap v1570: 47 pgs: 47 active+clean; 5.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 840 op/s 2024-02-09T17:48:12.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:11 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/802360941' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-02-09T17:48:12.914 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:12 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/802360941' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-02-09T17:48:12.914 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:12 smithi099 ceph-mon[39420]: osdmap e628: 12 total, 12 up, 12 in 2024-02-09T17:48:12.958 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 9 1 2024-02-09T17:48:12.979 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:48:12.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:12.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:48:13.087 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:48:13.123 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:12 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/802360941' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-02-09T17:48:13.123 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:12 smithi179 ceph-mon[43975]: osdmap e628: 12 total, 12 up, 12 in 2024-02-09T17:48:13.123 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:13.141 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:48:13.144 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:48:13.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:13.147 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:48:13.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:12 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/802360941' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-02-09T17:48:13.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:12 smithi162 ceph-mon[44304]: osdmap e628: 12 total, 12 up, 12 in 2024-02-09T17:48:13.215 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:48:13.248 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:48:13.275 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:13.298 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:48:13.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:13.304 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:48:13.405 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:48:13.466 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:13.540 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:48:13.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:13.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:48:13.649 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:48:13.683 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:13.704 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:48:13.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:13.709 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:48:13.811 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:48:13.838 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:13.862 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:48:13.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:13.868 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:48:13.969 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:48:14.010 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:14.055 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:48:14.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:13 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:48:14.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:13 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:48:14.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:13 smithi162 ceph-mon[44304]: osdmap e629: 12 total, 12 up, 12 in 2024-02-09T17:48:14.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:13 smithi162 ceph-mon[44304]: 1.e scrub starts 2024-02-09T17:48:14.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:13 smithi162 ceph-mon[44304]: 1.e scrub ok 2024-02-09T17:48:14.061 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:13 smithi162 ceph-mon[44304]: pgmap v1573: 47 pgs: 47 active+clean; 5.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 962 op/s 2024-02-09T17:48:14.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:14.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:48:14.163 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:48:14.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:13 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:48:14.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:13 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:48:14.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:13 smithi179 ceph-mon[43975]: osdmap e629: 12 total, 12 up, 12 in 2024-02-09T17:48:14.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:13 smithi179 ceph-mon[43975]: 1.e scrub starts 2024-02-09T17:48:14.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:13 smithi179 ceph-mon[43975]: 1.e scrub ok 2024-02-09T17:48:14.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:13 smithi179 ceph-mon[43975]: pgmap v1573: 47 pgs: 47 active+clean; 5.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 962 op/s 2024-02-09T17:48:14.194 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:14.222 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:48:14.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:14.226 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:48:14.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:13 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:48:14.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:13 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:48:14.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:13 smithi099 ceph-mon[39420]: osdmap e629: 12 total, 12 up, 12 in 2024-02-09T17:48:14.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:13 smithi099 ceph-mon[39420]: 1.e scrub starts 2024-02-09T17:48:14.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:13 smithi099 ceph-mon[39420]: 1.e scrub ok 2024-02-09T17:48:14.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:13 smithi099 ceph-mon[39420]: pgmap v1573: 47 pgs: 47 active+clean; 5.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 962 op/s 2024-02-09T17:48:14.326 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:48:14.367 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:14.399 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:48:14.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:14.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:48:14.505 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:48:14.554 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:14.913 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:48:14.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:14.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:48:15.021 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:48:15.074 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:15.194 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:48:16.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:16 smithi179 ceph-mon[43975]: pgmap v1574: 47 pgs: 47 active+clean; 5.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 886 op/s 2024-02-09T17:48:16.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:16 smithi162 ceph-mon[44304]: pgmap v1574: 47 pgs: 47 active+clean; 5.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 886 op/s 2024-02-09T17:48:16.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:16 smithi099 ceph-mon[39420]: pgmap v1574: 47 pgs: 47 active+clean; 5.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 886 op/s 2024-02-09T17:48:16.961 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:48:16.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:16.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:48:17.076 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:48:17.122 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:17.160 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:48:17.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:17.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:48:17.200 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:17 smithi099 ceph-mon[39420]: 1.11 scrub starts 2024-02-09T17:48:17.200 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:17 smithi099 ceph-mon[39420]: 1.11 scrub ok 2024-02-09T17:48:17.265 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:48:17.312 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:17.413 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:48:17.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:17.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:48:17.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:17 smithi179 ceph-mon[43975]: 1.11 scrub starts 2024-02-09T17:48:17.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:17 smithi179 ceph-mon[43975]: 1.11 scrub ok 2024-02-09T17:48:17.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:17 smithi162 ceph-mon[44304]: 1.11 scrub starts 2024-02-09T17:48:17.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:17 smithi162 ceph-mon[44304]: 1.11 scrub ok 2024-02-09T17:48:17.521 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:48:17.584 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:17.724 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:48:17.731 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:17.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:48:17.833 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:48:17.868 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:17.894 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:48:17.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:17.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:48:18.000 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:48:18.030 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:18.059 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:48:18.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:18.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:48:18.164 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:48:18.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:18 smithi162 ceph-mon[44304]: pgmap v1575: 47 pgs: 47 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 767 op/s 2024-02-09T17:48:18.226 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:18.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:18 smithi179 ceph-mon[43975]: pgmap v1575: 47 pgs: 47 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 767 op/s 2024-02-09T17:48:18.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:18 smithi099 ceph-mon[39420]: pgmap v1575: 47 pgs: 47 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 767 op/s 2024-02-09T17:48:18.613 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:48:18.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:18.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:48:18.628 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:48:18.651 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:48:18.720 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:48:18.747 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:48:18.747 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [ 2024-02-09T17:48:18.747 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:18.748 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25837.0:179707 6.4 6:26380f97:::benchmark_data_smithi179_126206_object49326:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e629)", 2024-02-09T17:48:18.748 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:48:18.734469+0000", 2024-02-09T17:48:18.748 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.010635482999999999, 2024-02-09T17:48:18.748 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.010668018, 2024-02-09T17:48:18.748 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:48:18.748 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "waiting for sub ops", 2024-02-09T17:48:18.748 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:48:18.748 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25837", 2024-02-09T17:48:18.748 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:48:18.748 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 179707 2024-02-09T17:48:18.748 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:18.748 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:48:18.748 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:18.749 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:48:18.749 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:18.734469+0000", 2024-02-09T17:48:18.749 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:48:18.749 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:18.749 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:18.749 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:48:18.749 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:18.734469+0000", 2024-02-09T17:48:18.749 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:48:18.749 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:18.749 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:18.749 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:48:18.749 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:18.734470+0000", 2024-02-09T17:48:18.749 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.226e-06 2024-02-09T17:48:18.750 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:18.750 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:18.750 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:48:18.750 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:18.734476+0000", 2024-02-09T17:48:18.750 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.083e-06 2024-02-09T17:48:18.750 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:18.750 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:18.750 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:48:18.750 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:18.734486+0000", 2024-02-09T17:48:18.750 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0135999999999999e-05 2024-02-09T17:48:18.750 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:18.750 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:18.750 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:48:18.750 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:18.734494+0000", 2024-02-09T17:48:18.750 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.9990000000000008e-06 2024-02-09T17:48:18.751 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:18.751 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:18.751 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:48:18.751 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:18.735697+0000", 2024-02-09T17:48:18.751 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.001202524 2024-02-09T17:48:18.751 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:18.751 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:18.751 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:48:18.751 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:18.735819+0000", 2024-02-09T17:48:18.751 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012205399999999999 2024-02-09T17:48:18.751 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:18.751 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:18.751 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,5", 2024-02-09T17:48:18.751 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:18.735946+0000", 2024-02-09T17:48:18.752 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012674100000000001 2024-02-09T17:48:18.752 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:48:18.752 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:48:18.752 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:48:18.752 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:18.752 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:18.752 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25837.0:179706 6.4 6:2d35ed9b:::benchmark_data_smithi179_126206_object49325:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e629)", 2024-02-09T17:48:18.752 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:48:18.724378+0000", 2024-02-09T17:48:18.752 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.020725977999999999, 2024-02-09T17:48:18.752 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.020836251, 2024-02-09T17:48:18.752 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:48:18.752 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:48:18.752 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:48:18.752 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25837", 2024-02-09T17:48:18.753 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:48:18.753 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 179706 2024-02-09T17:48:18.753 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:18.753 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:48:18.753 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:18.753 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:48:18.753 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:18.724378+0000", 2024-02-09T17:48:18.753 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:48:18.753 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:18.753 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:18.753 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:48:18.753 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:18.724378+0000", 2024-02-09T17:48:18.753 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:48:18.753 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:18.754 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:18.754 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:48:18.754 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:18.724381+0000", 2024-02-09T17:48:18.754 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2309999999999999e-06 2024-02-09T17:48:18.754 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:18.754 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:18.754 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:48:18.754 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:18.724400+0000", 2024-02-09T17:48:18.754 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.9273e-05 2024-02-09T17:48:18.754 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:18.754 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:18.754 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:48:18.754 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:18.724417+0000", 2024-02-09T17:48:18.754 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.7034e-05 2024-02-09T17:48:18.755 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:18.755 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:18.755 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:48:18.755 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:18.724428+0000", 2024-02-09T17:48:18.755 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0879999999999999e-05 2024-02-09T17:48:18.755 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:18.755 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:18.755 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:48:18.755 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:18.724499+0000", 2024-02-09T17:48:18.755 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.1284000000000004e-05 2024-02-09T17:48:18.755 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:18.755 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:18.755 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:48:18.755 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:18.724637+0000", 2024-02-09T17:48:18.755 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013813399999999999 2024-02-09T17:48:18.756 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:18.756 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:18.756 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,5", 2024-02-09T17:48:18.756 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:18.724768+0000", 2024-02-09T17:48:18.756 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000130841 2024-02-09T17:48:18.756 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:18.756 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:18.756 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:48:18.756 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:18.734462+0000", 2024-02-09T17:48:18.756 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0096940189999999999 2024-02-09T17:48:18.756 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:18.756 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:18.756 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:48:18.756 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:18.745106+0000", 2024-02-09T17:48:18.757 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.010643671 2024-02-09T17:48:18.757 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:18.757 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:18.757 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:48:18.757 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:18.745124+0000", 2024-02-09T17:48:18.757 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.8067000000000002e-05 2024-02-09T17:48:18.757 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:18.757 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:18.757 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:48:18.757 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:18.745177+0000", 2024-02-09T17:48:18.758 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2673999999999997e-05 2024-02-09T17:48:18.758 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:18.758 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:18.758 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:48:18.758 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:18.745215+0000", 2024-02-09T17:48:18.758 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.8142999999999997e-05 2024-02-09T17:48:18.758 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:48:18.758 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:48:18.758 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:48:18.758 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:18.758 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:18.758 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25837.0:179702 6.b e629/576 6:d5ee3e74:::benchmark_data_smithi179_126206_object49321:head v 629'11153, mlcod=629'11150)", 2024-02-09T17:48:18.758 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:48:18.729521+0000", 2024-02-09T17:48:18.758 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.015583051000000001, 2024-02-09T17:48:18.759 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.015820324, 2024-02-09T17:48:18.759 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:48:18.759 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T17:48:18.759 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:48:18.759 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:18.759 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:48:18.759 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:18.729521+0000", 2024-02-09T17:48:18.759 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:48:18.759 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:18.759 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:18.759 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:48:18.760 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:18.729521+0000", 2024-02-09T17:48:18.760 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:48:18.760 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:18.760 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:18.760 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:48:18.760 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:18.729523+0000", 2024-02-09T17:48:18.760 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1480000000000001e-06 2024-02-09T17:48:18.760 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:18.760 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:18.760 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:48:18.760 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:18.729530+0000", 2024-02-09T17:48:18.760 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.9119999999999998e-06 2024-02-09T17:48:18.760 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:18.760 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:18.760 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:48:18.761 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:18.729543+0000", 2024-02-09T17:48:18.761 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.239e-05 2024-02-09T17:48:18.761 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:18.761 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:18.761 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:48:18.761 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:18.729551+0000", 2024-02-09T17:48:18.761 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.5669999999999995e-06 2024-02-09T17:48:18.761 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:18.761 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:18.761 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:48:18.761 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:18.729699+0000", 2024-02-09T17:48:18.761 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014755399999999999 2024-02-09T17:48:18.761 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:18.761 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:18.762 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:48:18.762 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:18.729724+0000", 2024-02-09T17:48:18.762 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.5418e-05 2024-02-09T17:48:18.762 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:48:18.762 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:48:18.762 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:48:18.762 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:18.762 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:18.762 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25837.0:179705 6.b e629/576 6:d9e8c1e5:::benchmark_data_smithi179_126206_object49324:head v 629'11154, mlcod=629'11151)", 2024-02-09T17:48:18.762 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:48:18.732054+0000", 2024-02-09T17:48:18.762 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.013050156, 2024-02-09T17:48:18.762 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.013358495999999999, 2024-02-09T17:48:18.762 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:48:18.762 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T17:48:18.763 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:48:18.763 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:18.763 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:48:18.763 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:18.732054+0000", 2024-02-09T17:48:18.763 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:48:18.763 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:18.763 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:18.763 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:48:18.763 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:18.732054+0000", 2024-02-09T17:48:18.763 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:48:18.763 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:18.763 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:18.763 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:48:18.764 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:18.732055+0000", 2024-02-09T17:48:18.764 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.2399999999999999e-07 2024-02-09T17:48:18.764 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:18.764 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:18.764 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:48:18.764 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:18.732059+0000", 2024-02-09T17:48:18.764 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.8999999999999999e-06 2024-02-09T17:48:18.764 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:18.764 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:18.764 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:48:18.764 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:18.732063+0000", 2024-02-09T17:48:18.764 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.0409999999999999e-06 2024-02-09T17:48:18.764 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:18.764 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:18.765 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:48:18.765 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:18.732066+0000", 2024-02-09T17:48:18.765 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.9459999999999998e-06 2024-02-09T17:48:18.765 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:18.765 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:18.765 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:48:18.765 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:18.734190+0000", 2024-02-09T17:48:18.765 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0021241089999999999 2024-02-09T17:48:18.765 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:18.765 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:18.765 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:48:18.766 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:18.734203+0000", 2024-02-09T17:48:18.766 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3854e-05 2024-02-09T17:48:18.766 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:48:18.766 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:48:18.766 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:48:18.766 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:48:18.766 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T17:48:18.766 INFO:teuthology.orchestra.run.smithi099.stdout: "num_ops": 4 2024-02-09T17:48:18.766 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:48:18.767 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:18.779 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:48:18.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:18.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:48:18.887 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:48:18.915 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:18.935 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:48:18.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:18.939 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:48:19.040 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:48:19.076 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:19.103 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:48:19.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:19.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:48:19.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:19 smithi179 ceph-mon[43975]: 1.12 scrub starts 2024-02-09T17:48:19.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:19 smithi179 ceph-mon[43975]: 1.12 scrub ok 2024-02-09T17:48:19.182 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is unset 2024-02-09T17:48:19.208 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:48:19.241 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:19.259 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:48:19.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:19.262 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:48:19.363 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:48:19.393 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:19.395 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:19 smithi099 ceph-mon[39420]: 1.12 scrub starts 2024-02-09T17:48:19.395 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:19 smithi099 ceph-mon[39420]: 1.12 scrub ok 2024-02-09T17:48:19.416 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:48:19.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:19.419 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:48:19.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:19 smithi162 ceph-mon[44304]: 1.12 scrub starts 2024-02-09T17:48:19.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:19 smithi162 ceph-mon[44304]: 1.12 scrub ok 2024-02-09T17:48:19.520 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:48:19.559 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:19.584 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:48:19.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:19.598 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:48:19.700 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:48:19.739 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:19.783 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:48:19.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:19.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:48:19.901 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:48:19.928 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:19.948 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:48:19.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:19.953 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:48:20.054 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:48:20.081 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:20.108 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:48:20.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:20.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:48:20.215 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:48:20.270 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:20.306 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:48:20.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:20.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:48:20.413 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:48:20.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:20 smithi179 ceph-mon[43975]: 6.0 scrub starts 2024-02-09T17:48:20.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:20 smithi179 ceph-mon[43975]: 1.1a scrub starts 2024-02-09T17:48:20.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:20 smithi179 ceph-mon[43975]: 1.1a scrub ok 2024-02-09T17:48:20.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:20 smithi179 ceph-mon[43975]: pgmap v1576: 47 pgs: 47 active+clean; 5.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 949 op/s 2024-02-09T17:48:20.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:20 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/711546037' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:48:20.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:20 smithi179 ceph-mon[43975]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:48:20.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:20 smithi179 ceph-mon[43975]: Cluster is now healthy 2024-02-09T17:48:20.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:20 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/711546037' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:48:20.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:20 smithi179 ceph-mon[43975]: osdmap e630: 12 total, 12 up, 12 in 2024-02-09T17:48:20.433 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:20 smithi162 ceph-mon[44304]: 6.0 scrub starts 2024-02-09T17:48:20.433 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:20 smithi162 ceph-mon[44304]: 1.1a scrub starts 2024-02-09T17:48:20.433 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:20 smithi162 ceph-mon[44304]: 1.1a scrub ok 2024-02-09T17:48:20.433 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:20 smithi162 ceph-mon[44304]: pgmap v1576: 47 pgs: 47 active+clean; 5.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 949 op/s 2024-02-09T17:48:20.433 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:20 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/711546037' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:48:20.433 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:20 smithi162 ceph-mon[44304]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:48:20.433 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:20 smithi162 ceph-mon[44304]: Cluster is now healthy 2024-02-09T17:48:20.433 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:20 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/711546037' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:48:20.433 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:20 smithi162 ceph-mon[44304]: osdmap e630: 12 total, 12 up, 12 in 2024-02-09T17:48:20.441 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:20.462 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:48:20.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:20.465 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:48:20.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:20 smithi099 ceph-mon[39420]: 6.0 scrub starts 2024-02-09T17:48:20.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:20 smithi099 ceph-mon[39420]: 1.1a scrub starts 2024-02-09T17:48:20.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:20 smithi099 ceph-mon[39420]: 1.1a scrub ok 2024-02-09T17:48:20.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:20 smithi099 ceph-mon[39420]: pgmap v1576: 47 pgs: 47 active+clean; 5.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 949 op/s 2024-02-09T17:48:20.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:20 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/711546037' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:48:20.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:20 smithi099 ceph-mon[39420]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:48:20.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:20 smithi099 ceph-mon[39420]: Cluster is now healthy 2024-02-09T17:48:20.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:20 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/711546037' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:48:20.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:20 smithi099 ceph-mon[39420]: osdmap e630: 12 total, 12 up, 12 in 2024-02-09T17:48:20.566 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:48:20.621 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:20.793 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-09T17:48:20.996 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:48:21.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:21.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:48:21.107 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:48:21.188 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:21.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:21 smithi179 ceph-mon[43975]: 1.15 scrub starts 2024-02-09T17:48:21.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:21 smithi179 ceph-mon[43975]: 1.15 scrub ok 2024-02-09T17:48:21.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:21 smithi179 ceph-mon[43975]: 6.0 scrub ok 2024-02-09T17:48:21.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:21 smithi179 ceph-mon[43975]: 6.1 scrub starts 2024-02-09T17:48:21.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:21 smithi162 ceph-mon[44304]: 1.15 scrub starts 2024-02-09T17:48:21.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:21 smithi162 ceph-mon[44304]: 1.15 scrub ok 2024-02-09T17:48:21.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:21 smithi162 ceph-mon[44304]: 6.0 scrub ok 2024-02-09T17:48:21.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:21 smithi162 ceph-mon[44304]: 6.1 scrub starts 2024-02-09T17:48:21.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:21 smithi099 ceph-mon[39420]: 1.15 scrub starts 2024-02-09T17:48:21.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:21 smithi099 ceph-mon[39420]: 1.15 scrub ok 2024-02-09T17:48:21.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:21 smithi099 ceph-mon[39420]: 6.0 scrub ok 2024-02-09T17:48:21.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:21 smithi099 ceph-mon[39420]: 6.1 scrub starts 2024-02-09T17:48:21.657 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:48:21.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:21.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:48:21.762 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:48:21.818 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:21.844 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:48:21.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:21.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:48:21.954 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:48:22.004 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:22.030 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:48:22.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:22.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:48:22.134 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:48:22.165 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:22.188 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:48:22.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:22.192 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:48:22.293 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:48:22.322 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:22.347 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:48:22.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:22.350 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:48:22.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:22 smithi179 ceph-mon[43975]: pgmap v1578: 47 pgs: 47 active+clean; 5.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 635 op/s 2024-02-09T17:48:22.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:22 smithi179 ceph-mon[43975]: 6.1 scrub ok 2024-02-09T17:48:22.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:22 smithi179 ceph-mon[43975]: 6.4 scrub starts 2024-02-09T17:48:22.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:22 smithi179 ceph-mon[43975]: 1.b scrub starts 2024-02-09T17:48:22.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:22 smithi162 ceph-mon[44304]: pgmap v1578: 47 pgs: 47 active+clean; 5.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 635 op/s 2024-02-09T17:48:22.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:22 smithi162 ceph-mon[44304]: 6.1 scrub ok 2024-02-09T17:48:22.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:22 smithi162 ceph-mon[44304]: 6.4 scrub starts 2024-02-09T17:48:22.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:22 smithi162 ceph-mon[44304]: 1.b scrub starts 2024-02-09T17:48:22.451 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:48:22.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:22 smithi099 ceph-mon[39420]: pgmap v1578: 47 pgs: 47 active+clean; 5.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 635 op/s 2024-02-09T17:48:22.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:22 smithi099 ceph-mon[39420]: 6.1 scrub ok 2024-02-09T17:48:22.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:22 smithi099 ceph-mon[39420]: 6.4 scrub starts 2024-02-09T17:48:22.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:22 smithi099 ceph-mon[39420]: 1.b scrub starts 2024-02-09T17:48:22.557 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:22.845 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:48:23.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:23 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:48:23.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:23 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:48:23.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:23 smithi099 ceph-mon[39420]: osdmap e631: 12 total, 12 up, 12 in 2024-02-09T17:48:23.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:23 smithi099 ceph-mon[39420]: 1.5 scrub starts 2024-02-09T17:48:23.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:23 smithi099 ceph-mon[39420]: 1.5 scrub ok 2024-02-09T17:48:23.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:23 smithi099 ceph-mon[39420]: 1.b scrub ok 2024-02-09T17:48:23.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:23 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:48:23.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:23 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:48:23.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:23 smithi179 ceph-mon[43975]: osdmap e631: 12 total, 12 up, 12 in 2024-02-09T17:48:23.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:23 smithi179 ceph-mon[43975]: 1.5 scrub starts 2024-02-09T17:48:23.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:23 smithi179 ceph-mon[43975]: 1.5 scrub ok 2024-02-09T17:48:23.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:23 smithi179 ceph-mon[43975]: 1.b scrub ok 2024-02-09T17:48:23.431 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:23 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:48:23.431 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:23 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:48:23.431 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:23 smithi162 ceph-mon[44304]: osdmap e631: 12 total, 12 up, 12 in 2024-02-09T17:48:23.431 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:23 smithi162 ceph-mon[44304]: 1.5 scrub starts 2024-02-09T17:48:23.431 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:23 smithi162 ceph-mon[44304]: 1.5 scrub ok 2024-02-09T17:48:23.431 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:23 smithi162 ceph-mon[44304]: 1.b scrub ok 2024-02-09T17:48:23.442 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:48:23.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:23.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:48:23.548 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:48:23.578 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:23.608 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:48:23.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:23.611 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:48:23.663 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set noscrub 2024-02-09T17:48:23.712 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:48:23.777 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:23.821 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:48:23.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:23.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:48:23.929 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:48:24.012 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:24.063 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:48:24.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:24.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:48:24.168 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:48:24.217 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:24.249 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:48:24.256 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:24.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:48:24.358 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:48:24.420 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:24 smithi162 ceph-mon[44304]: 6.e scrub starts 2024-02-09T17:48:24.420 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:24 smithi162 ceph-mon[44304]: pgmap v1580: 47 pgs: 47 active+clean; 4.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 915 op/s 2024-02-09T17:48:24.420 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:24 smithi162 ceph-mon[44304]: 6.4 scrub ok 2024-02-09T17:48:24.420 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:24 smithi162 ceph-mon[44304]: 6.9 scrub starts 2024-02-09T17:48:24.420 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:24 smithi162 ceph-mon[44304]: 6.e scrub ok 2024-02-09T17:48:24.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:24 smithi179 ceph-mon[43975]: 6.e scrub starts 2024-02-09T17:48:24.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:24 smithi179 ceph-mon[43975]: pgmap v1580: 47 pgs: 47 active+clean; 4.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 915 op/s 2024-02-09T17:48:24.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:24 smithi179 ceph-mon[43975]: 6.4 scrub ok 2024-02-09T17:48:24.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:24 smithi179 ceph-mon[43975]: 6.9 scrub starts 2024-02-09T17:48:24.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:24 smithi179 ceph-mon[43975]: 6.e scrub ok 2024-02-09T17:48:24.423 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:24.446 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:24 smithi099 ceph-mon[39420]: 6.e scrub starts 2024-02-09T17:48:24.446 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:24 smithi099 ceph-mon[39420]: pgmap v1580: 47 pgs: 47 active+clean; 4.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 915 op/s 2024-02-09T17:48:24.446 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:24 smithi099 ceph-mon[39420]: 6.4 scrub ok 2024-02-09T17:48:24.446 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:24 smithi099 ceph-mon[39420]: 6.9 scrub starts 2024-02-09T17:48:24.446 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:24 smithi099 ceph-mon[39420]: 6.e scrub ok 2024-02-09T17:48:24.447 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:48:24.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:24.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:48:24.554 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:48:24.599 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:24.634 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:48:24.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:24.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:48:24.745 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:48:24.781 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:24.808 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:48:24.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:24.810 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:48:24.911 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:48:24.943 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:24.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:24.976 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:48:24.977 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:48:25.078 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:48:25.113 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:25.142 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:48:25.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:25.145 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:48:25.246 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:48:25.297 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:25.322 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:48:25.325 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:25.325 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:48:25.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:25 smithi179 ceph-mon[43975]: 6.8 scrub starts 2024-02-09T17:48:25.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:25 smithi179 ceph-mon[43975]: 6.9 scrub ok 2024-02-09T17:48:25.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:25 smithi179 ceph-mon[43975]: 1.1d deep-scrub starts 2024-02-09T17:48:25.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:25 smithi179 ceph-mon[43975]: 1.1d deep-scrub ok 2024-02-09T17:48:25.426 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:48:25.437 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:25 smithi099 ceph-mon[39420]: 6.8 scrub starts 2024-02-09T17:48:25.437 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:25 smithi099 ceph-mon[39420]: 6.9 scrub ok 2024-02-09T17:48:25.438 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:25 smithi099 ceph-mon[39420]: 1.1d deep-scrub starts 2024-02-09T17:48:25.438 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:25 smithi099 ceph-mon[39420]: 1.1d deep-scrub ok 2024-02-09T17:48:25.517 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:48:25.517 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:25.560 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:48:25.594 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:48:25.631 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:48:25.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:25 smithi162 ceph-mon[44304]: 6.8 scrub starts 2024-02-09T17:48:25.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:25 smithi162 ceph-mon[44304]: 6.9 scrub ok 2024-02-09T17:48:25.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:25 smithi162 ceph-mon[44304]: 1.1d deep-scrub starts 2024-02-09T17:48:25.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:25 smithi162 ceph-mon[44304]: 1.1d deep-scrub ok 2024-02-09T17:48:25.680 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:48:25.724 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:48:25.768 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:48:25.814 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:48:25.831 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:48:25.861 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:48:25.898 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:48:25.931 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:48:25.941 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:48:25.941 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:48:25.941 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:48:25.941 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:48:25.941 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:48:25.942 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:48:25.942 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:48:25.942 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:48:25.942 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:48:25.942 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:48:25.942 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:48:25.942 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:48:26.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:26 smithi099 ceph-mon[39420]: pgmap v1581: 47 pgs: 47 active+clean; 4.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 835 op/s 2024-02-09T17:48:26.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:26 smithi099 ceph-mon[39420]: 6.8 scrub ok 2024-02-09T17:48:26.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:26 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/952241890' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:48:26.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:26 smithi162 ceph-mon[44304]: pgmap v1581: 47 pgs: 47 active+clean; 4.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 835 op/s 2024-02-09T17:48:26.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:26 smithi162 ceph-mon[44304]: 6.8 scrub ok 2024-02-09T17:48:26.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:26 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/952241890' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:48:26.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:26 smithi179 ceph-mon[43975]: pgmap v1581: 47 pgs: 47 active+clean; 4.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 835 op/s 2024-02-09T17:48:26.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:26 smithi179 ceph-mon[43975]: 6.8 scrub ok 2024-02-09T17:48:26.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:26 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/952241890' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:48:27.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:27 smithi099 ceph-mon[39420]: 1.d scrub starts 2024-02-09T17:48:27.498 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:27 smithi099 ceph-mon[39420]: 1.d scrub ok 2024-02-09T17:48:27.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:27 smithi179 ceph-mon[43975]: 1.d scrub starts 2024-02-09T17:48:27.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:27 smithi179 ceph-mon[43975]: 1.d scrub ok 2024-02-09T17:48:27.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:27 smithi162 ceph-mon[44304]: 1.d scrub starts 2024-02-09T17:48:27.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:27 smithi162 ceph-mon[44304]: 1.d scrub ok 2024-02-09T17:48:28.113 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:48:28.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:28.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:48:28.223 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:48:28.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:28 smithi099 ceph-mon[39420]: pgmap v1582: 47 pgs: 47 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 692 op/s 2024-02-09T17:48:28.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:28 smithi099 ceph-mon[39420]: 1.9 scrub starts 2024-02-09T17:48:28.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:28 smithi099 ceph-mon[39420]: 1.14 scrub starts 2024-02-09T17:48:28.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:28 smithi099 ceph-mon[39420]: 1.14 scrub ok 2024-02-09T17:48:28.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:28 smithi099 ceph-mon[39420]: 1.1e scrub starts 2024-02-09T17:48:28.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:28 smithi099 ceph-mon[39420]: 1.9 scrub ok 2024-02-09T17:48:28.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:28 smithi099 ceph-mon[39420]: 1.1e scrub ok 2024-02-09T17:48:28.270 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:28.312 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:48:28.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:28.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:48:28.416 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:48:28.443 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:28 smithi179 ceph-mon[43975]: pgmap v1582: 47 pgs: 47 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 692 op/s 2024-02-09T17:48:28.444 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:28 smithi179 ceph-mon[43975]: 1.9 scrub starts 2024-02-09T17:48:28.444 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:28 smithi179 ceph-mon[43975]: 1.14 scrub starts 2024-02-09T17:48:28.444 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:28 smithi179 ceph-mon[43975]: 1.14 scrub ok 2024-02-09T17:48:28.444 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:28 smithi179 ceph-mon[43975]: 1.1e scrub starts 2024-02-09T17:48:28.444 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:28 smithi179 ceph-mon[43975]: 1.9 scrub ok 2024-02-09T17:48:28.444 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:28 smithi179 ceph-mon[43975]: 1.1e scrub ok 2024-02-09T17:48:28.445 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:28.478 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:48:28.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:28.481 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:48:28.582 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:48:28.610 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:28.630 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:48:28.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:28.637 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:48:28.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:28 smithi162 ceph-mon[44304]: pgmap v1582: 47 pgs: 47 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 692 op/s 2024-02-09T17:48:28.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:28 smithi162 ceph-mon[44304]: 1.9 scrub starts 2024-02-09T17:48:28.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:28 smithi162 ceph-mon[44304]: 1.14 scrub starts 2024-02-09T17:48:28.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:28 smithi162 ceph-mon[44304]: 1.14 scrub ok 2024-02-09T17:48:28.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:28 smithi162 ceph-mon[44304]: 1.1e scrub starts 2024-02-09T17:48:28.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:28 smithi162 ceph-mon[44304]: 1.9 scrub ok 2024-02-09T17:48:28.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:28 smithi162 ceph-mon[44304]: 1.1e scrub ok 2024-02-09T17:48:28.737 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:48:28.767 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:28.800 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:48:28.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:28.803 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:48:28.904 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:48:28.954 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:28.976 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:48:28.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:28.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:48:29.080 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:48:29.136 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:29.212 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:29 smithi162 ceph-mon[44304]: 6.6 scrub starts 2024-02-09T17:48:29.212 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:29 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3825837141' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-02-09T17:48:29.212 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:29 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-02-09T17:48:29.228 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.9 to 1 (10000) 2024-02-09T17:48:29.359 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:48:29.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:29.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:48:29.465 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:48:29.498 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:29 smithi099 ceph-mon[39420]: 6.6 scrub starts 2024-02-09T17:48:29.498 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:29 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3825837141' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-02-09T17:48:29.498 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:29 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-02-09T17:48:29.527 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:29.622 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:48:29.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:29.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:48:29.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:29 smithi179 ceph-mon[43975]: 6.6 scrub starts 2024-02-09T17:48:29.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:29 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3825837141' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-02-09T17:48:29.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:29 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-02-09T17:48:29.736 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:48:29.785 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:29.812 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:48:29.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:29.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:48:29.917 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:48:29.948 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:29.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:29.976 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:48:29.977 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:48:30.078 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:48:30.153 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:30.181 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:48:30.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:30.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:48:30.207 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:30 smithi099 ceph-mon[39420]: pgmap v1583: 47 pgs: 47 active+clean; 4.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 775 op/s 2024-02-09T17:48:30.208 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:30 smithi099 ceph-mon[39420]: 6.3 scrub starts 2024-02-09T17:48:30.208 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:30 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-02-09T17:48:30.208 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:30 smithi099 ceph-mon[39420]: osdmap e632: 12 total, 12 up, 12 in 2024-02-09T17:48:30.208 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:30 smithi099 ceph-mon[39420]: 6.6 scrub ok 2024-02-09T17:48:30.286 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:48:30.365 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:30.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:30 smithi162 ceph-mon[44304]: pgmap v1583: 47 pgs: 47 active+clean; 4.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 775 op/s 2024-02-09T17:48:30.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:30 smithi162 ceph-mon[44304]: 6.3 scrub starts 2024-02-09T17:48:30.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:30 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-02-09T17:48:30.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:30 smithi162 ceph-mon[44304]: osdmap e632: 12 total, 12 up, 12 in 2024-02-09T17:48:30.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:30 smithi162 ceph-mon[44304]: 6.6 scrub ok 2024-02-09T17:48:30.459 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:48:30.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:30.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:48:30.564 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:48:30.598 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:30 smithi179 ceph-mon[43975]: pgmap v1583: 47 pgs: 47 active+clean; 4.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 775 op/s 2024-02-09T17:48:30.598 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:30 smithi179 ceph-mon[43975]: 6.3 scrub starts 2024-02-09T17:48:30.598 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:30 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-02-09T17:48:30.598 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:30 smithi179 ceph-mon[43975]: osdmap e632: 12 total, 12 up, 12 in 2024-02-09T17:48:30.598 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:30 smithi179 ceph-mon[43975]: 6.6 scrub ok 2024-02-09T17:48:30.599 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:30.623 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:48:30.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:30.626 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:48:30.727 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:48:30.750 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:48:30.751 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:48:30.835 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:31.113 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:48:31.113 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:48:31.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:31.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:48:31.155 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 10 1 2024-02-09T17:48:31.225 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:48:31.287 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:31.330 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:48:31.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:31.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:48:31.442 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:48:31.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:31 smithi099 ceph-mon[39420]: 6.7 scrub starts 2024-02-09T17:48:31.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:31 smithi099 ceph-mon[39420]: 6.3 scrub ok 2024-02-09T17:48:31.532 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:31.635 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:48:31.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:31.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:48:31.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:31 smithi179 ceph-mon[43975]: 6.7 scrub starts 2024-02-09T17:48:31.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:31 smithi179 ceph-mon[43975]: 6.3 scrub ok 2024-02-09T17:48:31.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:31 smithi162 ceph-mon[44304]: 6.7 scrub starts 2024-02-09T17:48:31.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:31 smithi162 ceph-mon[44304]: 6.3 scrub ok 2024-02-09T17:48:31.740 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:48:31.768 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:31.791 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:48:31.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:31.794 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:48:31.895 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:48:31.925 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:31.954 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:48:31.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:31.958 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:48:32.060 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:48:32.103 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:32.143 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:48:32.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:32.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:48:32.256 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:48:32.264 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-09T17:48:32.318 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:32.329 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:48:32.370 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:48:32.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:32.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:48:32.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:32 smithi162 ceph-mon[44304]: pgmap v1585: 47 pgs: 47 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 723 op/s 2024-02-09T17:48:32.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:32 smithi162 ceph-mon[44304]: 6.5 scrub starts 2024-02-09T17:48:32.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:32 smithi162 ceph-mon[44304]: 6.b scrub starts 2024-02-09T17:48:32.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:32 smithi162 ceph-mon[44304]: 6.7 scrub ok 2024-02-09T17:48:32.484 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:32 smithi099 ceph-mon[39420]: pgmap v1585: 47 pgs: 47 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 723 op/s 2024-02-09T17:48:32.484 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:32 smithi099 ceph-mon[39420]: 6.5 scrub starts 2024-02-09T17:48:32.484 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:32 smithi099 ceph-mon[39420]: 6.b scrub starts 2024-02-09T17:48:32.484 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:32 smithi099 ceph-mon[39420]: 6.7 scrub ok 2024-02-09T17:48:32.484 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:48:32.512 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:32 smithi179 ceph-mon[43975]: pgmap v1585: 47 pgs: 47 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 723 op/s 2024-02-09T17:48:32.512 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:32 smithi179 ceph-mon[43975]: 6.5 scrub starts 2024-02-09T17:48:32.512 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:32 smithi179 ceph-mon[43975]: 6.b scrub starts 2024-02-09T17:48:32.512 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:32 smithi179 ceph-mon[43975]: 6.7 scrub ok 2024-02-09T17:48:32.514 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:32.539 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:48:32.542 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:32.542 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:48:32.643 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:48:32.675 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:32.702 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:48:32.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:32.705 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:48:32.806 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:48:32.837 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:32.864 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:48:32.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:32.870 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:48:32.971 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:48:33.020 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:33.483 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:48:33.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:33.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:48:33.591 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:48:33.621 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:33 smithi179 ceph-mon[43975]: 1.0 scrub starts 2024-02-09T17:48:33.622 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:33 smithi179 ceph-mon[43975]: 1.0 scrub ok 2024-02-09T17:48:33.622 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:33 smithi179 ceph-mon[43975]: 6.5 scrub ok 2024-02-09T17:48:33.622 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:33 smithi179 ceph-mon[43975]: 6.f scrub starts 2024-02-09T17:48:33.622 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:33 smithi179 ceph-mon[43975]: 6.b scrub ok 2024-02-09T17:48:33.623 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:33.653 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:48:33.659 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:33.660 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:48:33.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:33 smithi162 ceph-mon[44304]: 1.0 scrub starts 2024-02-09T17:48:33.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:33 smithi162 ceph-mon[44304]: 1.0 scrub ok 2024-02-09T17:48:33.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:33 smithi162 ceph-mon[44304]: 6.5 scrub ok 2024-02-09T17:48:33.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:33 smithi162 ceph-mon[44304]: 6.f scrub starts 2024-02-09T17:48:33.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:33 smithi162 ceph-mon[44304]: 6.b scrub ok 2024-02-09T17:48:33.751 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:33 smithi099 ceph-mon[39420]: 1.0 scrub starts 2024-02-09T17:48:33.752 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:33 smithi099 ceph-mon[39420]: 1.0 scrub ok 2024-02-09T17:48:33.752 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:33 smithi099 ceph-mon[39420]: 6.5 scrub ok 2024-02-09T17:48:33.752 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:33 smithi099 ceph-mon[39420]: 6.f scrub starts 2024-02-09T17:48:33.752 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:33 smithi099 ceph-mon[39420]: 6.b scrub ok 2024-02-09T17:48:33.760 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:48:33.801 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:34.561 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:34 smithi099 ceph-mon[39420]: pgmap v1586: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 748 op/s 2024-02-09T17:48:34.561 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:34 smithi099 ceph-mon[39420]: 6.f scrub ok 2024-02-09T17:48:34.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:34 smithi179 ceph-mon[43975]: pgmap v1586: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 748 op/s 2024-02-09T17:48:34.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:34 smithi179 ceph-mon[43975]: 6.f scrub ok 2024-02-09T17:48:34.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:34 smithi162 ceph-mon[44304]: pgmap v1586: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 748 op/s 2024-02-09T17:48:34.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:34 smithi162 ceph-mon[44304]: 6.f scrub ok 2024-02-09T17:48:35.384 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:48:35.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:35.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:48:35.427 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:48:35.493 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:48:35.548 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:35 smithi099 ceph-mon[39420]: 1.f deep-scrub starts 2024-02-09T17:48:35.549 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:35.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:35 smithi179 ceph-mon[43975]: 1.f deep-scrub starts 2024-02-09T17:48:35.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:35 smithi162 ceph-mon[44304]: 1.f deep-scrub starts 2024-02-09T17:48:35.740 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:48:36.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:36 smithi099 ceph-mon[39420]: 1.f deep-scrub ok 2024-02-09T17:48:36.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:36 smithi099 ceph-mon[39420]: pgmap v1587: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 747 op/s 2024-02-09T17:48:36.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:36 smithi099 ceph-mon[39420]: 1.10 deep-scrub starts 2024-02-09T17:48:36.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:36 smithi099 ceph-mon[39420]: 1.10 deep-scrub ok 2024-02-09T17:48:36.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:36 smithi179 ceph-mon[43975]: 1.f deep-scrub ok 2024-02-09T17:48:36.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:36 smithi179 ceph-mon[43975]: pgmap v1587: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 747 op/s 2024-02-09T17:48:36.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:36 smithi179 ceph-mon[43975]: 1.10 deep-scrub starts 2024-02-09T17:48:36.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:36 smithi179 ceph-mon[43975]: 1.10 deep-scrub ok 2024-02-09T17:48:36.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:36 smithi162 ceph-mon[44304]: 1.f deep-scrub ok 2024-02-09T17:48:36.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:36 smithi162 ceph-mon[44304]: pgmap v1587: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 747 op/s 2024-02-09T17:48:36.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:36 smithi162 ceph-mon[44304]: 1.10 deep-scrub starts 2024-02-09T17:48:36.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:36 smithi162 ceph-mon[44304]: 1.10 deep-scrub ok 2024-02-09T17:48:36.952 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:48:36.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:36.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:48:37.065 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:48:37.105 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:37.138 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:48:37.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:37.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:48:37.244 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:48:37.339 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:37.698 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:37 smithi099 ceph-mon[39420]: pgmap v1588: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 765 op/s 2024-02-09T17:48:37.698 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:37 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3541158127' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:48:37.729 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is set 2024-02-09T17:48:38.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:37 smithi179 ceph-mon[43975]: pgmap v1588: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 765 op/s 2024-02-09T17:48:38.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:37 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3541158127' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:48:38.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:37 smithi162 ceph-mon[44304]: pgmap v1588: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 765 op/s 2024-02-09T17:48:38.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:37 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3541158127' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:48:38.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:38 smithi099 ceph-mon[39420]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:48:38.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:38 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3541158127' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:48:38.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:38 smithi099 ceph-mon[39420]: osdmap e633: 12 total, 12 up, 12 in 2024-02-09T17:48:39.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:38 smithi179 ceph-mon[43975]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:48:39.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:38 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3541158127' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:48:39.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:38 smithi179 ceph-mon[43975]: osdmap e633: 12 total, 12 up, 12 in 2024-02-09T17:48:39.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:38 smithi162 ceph-mon[44304]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:48:39.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:38 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3541158127' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:48:39.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:38 smithi162 ceph-mon[44304]: osdmap e633: 12 total, 12 up, 12 in 2024-02-09T17:48:39.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:39 smithi099 ceph-mon[39420]: pgmap v1590: 47 pgs: 47 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 815 op/s 2024-02-09T17:48:40.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:39 smithi179 ceph-mon[43975]: pgmap v1590: 47 pgs: 47 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 815 op/s 2024-02-09T17:48:40.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:39 smithi162 ceph-mon[44304]: pgmap v1590: 47 pgs: 47 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 815 op/s 2024-02-09T17:48:41.363 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:48:41.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:41.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:48:41.376 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:48:41.470 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:48:41.536 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:41.681 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:48:41.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:41.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:48:41.792 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:48:41.823 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:41.855 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:48:41.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:41.861 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:48:41.962 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:48:42.009 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:42.074 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:48:42.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:42.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:48:42.098 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.10 to 1 (10000) 2024-02-09T17:48:42.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:42 smithi179 ceph-mon[43975]: pgmap v1591: 47 pgs: 47 active+clean; 4.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 805 op/s 2024-02-09T17:48:42.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:42 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3683819595' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-02-09T17:48:42.179 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:48:42.233 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:42 smithi099 ceph-mon[39420]: pgmap v1591: 47 pgs: 47 active+clean; 4.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 805 op/s 2024-02-09T17:48:42.233 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:42 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3683819595' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-02-09T17:48:42.247 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:42.277 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:48:42.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:42.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:48:42.310 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:48:42.310 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:48:42.385 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:48:42.419 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:42.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:42 smithi162 ceph-mon[44304]: pgmap v1591: 47 pgs: 47 active+clean; 4.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 805 op/s 2024-02-09T17:48:42.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:42 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3683819595' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-02-09T17:48:42.444 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:48:42.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:42.451 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:48:42.553 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:48:42.592 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:42.619 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:48:42.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:42.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:48:42.731 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:48:42.773 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:42.789 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:48:42.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:42.792 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:48:42.893 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:48:42.930 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:42.965 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:48:42.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:42.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:48:43.074 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:48:43.103 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:43 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3683819595' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-02-09T17:48:43.103 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:43 smithi179 ceph-mon[43975]: osdmap e634: 12 total, 12 up, 12 in 2024-02-09T17:48:43.103 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:43 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:48:43.141 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:43.195 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:48:43.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:43.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:48:43.206 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:43 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3683819595' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-02-09T17:48:43.206 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:43 smithi162 ceph-mon[44304]: osdmap e634: 12 total, 12 up, 12 in 2024-02-09T17:48:43.206 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:43 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:48:43.248 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:43 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3683819595' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-02-09T17:48:43.248 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:43 smithi099 ceph-mon[39420]: osdmap e634: 12 total, 12 up, 12 in 2024-02-09T17:48:43.248 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:43 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:48:43.306 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:48:43.338 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:43.358 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:48:43.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:43.360 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:48:43.435 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-09T17:48:43.461 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:48:43.568 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:44.227 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:48:44.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:44.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:48:44.332 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:48:44.387 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:44 smithi099 ceph-mon[39420]: pgmap v1593: 47 pgs: 47 active+clean; 3.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 979 op/s 2024-02-09T17:48:44.387 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:44 smithi099 ceph-mon[39420]: osdmap e635: 12 total, 12 up, 12 in 2024-02-09T17:48:44.387 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:44 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:48:44.387 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:44 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:48:44.387 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:44 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:48:44.401 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:44.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:44 smithi179 ceph-mon[43975]: pgmap v1593: 47 pgs: 47 active+clean; 3.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 979 op/s 2024-02-09T17:48:44.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:44 smithi179 ceph-mon[43975]: osdmap e635: 12 total, 12 up, 12 in 2024-02-09T17:48:44.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:44 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:48:44.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:44 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:48:44.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:44 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:48:44.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:44 smithi162 ceph-mon[44304]: pgmap v1593: 47 pgs: 47 active+clean; 3.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 979 op/s 2024-02-09T17:48:44.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:44 smithi162 ceph-mon[44304]: osdmap e635: 12 total, 12 up, 12 in 2024-02-09T17:48:44.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:44 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:48:44.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:44 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:48:44.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:44 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:48:44.610 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:48:44.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:44.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:48:44.718 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:48:44.780 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:45.237 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 11 1 2024-02-09T17:48:45.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:45 smithi179 ceph-mon[43975]: osdmap e636: 12 total, 12 up, 12 in 2024-02-09T17:48:45.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:45 smithi162 ceph-mon[44304]: osdmap e636: 12 total, 12 up, 12 in 2024-02-09T17:48:45.439 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:45 smithi099 ceph-mon[39420]: osdmap e636: 12 total, 12 up, 12 in 2024-02-09T17:48:45.460 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:48:45.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:45.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:48:45.574 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:48:45.632 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:45.659 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:48:45.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:45.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:48:45.769 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:48:45.806 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:45.853 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:48:45.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:45.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:48:45.964 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:48:45.992 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:46.017 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:48:46.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:46.022 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:48:46.121 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-09T17:48:46.122 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:48:46.159 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:46.180 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:48:46.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:46.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:48:46.277 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:48:46.277 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:48:46.277 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:48:46.277 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:48:46.284 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:48:46.294 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:48:46.310 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:48:46.345 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:48:46.355 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:46 smithi099 ceph-mon[39420]: pgmap v1596: 47 pgs: 1 remapped+peering, 46 active+clean; 3.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 22 KiB/s wr, 818 op/s; 12 KiB/s, 0 objects/s recovering 2024-02-09T17:48:46.355 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:46 smithi099 ceph-mon[39420]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-09T17:48:46.355 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:46 smithi099 ceph-mon[39420]: osdmap e637: 12 total, 12 up, 12 in 2024-02-09T17:48:46.355 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:46 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2332735771' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:48:46.370 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:48:46.370 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:46.402 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:48:46.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:46 smithi179 ceph-mon[43975]: pgmap v1596: 47 pgs: 1 remapped+peering, 46 active+clean; 3.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 22 KiB/s wr, 818 op/s; 12 KiB/s, 0 objects/s recovering 2024-02-09T17:48:46.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:46 smithi179 ceph-mon[43975]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-09T17:48:46.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:46 smithi179 ceph-mon[43975]: osdmap e637: 12 total, 12 up, 12 in 2024-02-09T17:48:46.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:46 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2332735771' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:48:46.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:46 smithi162 ceph-mon[44304]: pgmap v1596: 47 pgs: 1 remapped+peering, 46 active+clean; 3.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 22 KiB/s wr, 818 op/s; 12 KiB/s, 0 objects/s recovering 2024-02-09T17:48:46.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:46 smithi162 ceph-mon[44304]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-09T17:48:46.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:46 smithi162 ceph-mon[44304]: osdmap e637: 12 total, 12 up, 12 in 2024-02-09T17:48:46.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:46 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2332735771' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:48:46.444 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:48:46.471 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:48:46.508 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:48:46.555 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:48:46.590 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:48:46.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:46.598 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:48:46.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:48:46.616 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:48:46.616 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:48:46.616 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:48:46.616 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:48:46.616 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:48:46.616 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:48:46.616 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:48:46.616 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:48:46.616 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:48:46.616 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:48:46.617 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:48:46.617 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:48:46.699 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:48:46.743 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:46.878 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:48:46.888 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:46.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:48:46.989 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:48:47.017 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:47.041 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:48:47.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:47.048 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:48:47.151 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:48:47.165 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:47 smithi099 ceph-mon[39420]: osdmap e638: 12 total, 12 up, 12 in 2024-02-09T17:48:47.181 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:47.205 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:48:47.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:47.208 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:48:47.309 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:48:47.346 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:47.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:47 smithi179 ceph-mon[43975]: osdmap e638: 12 total, 12 up, 12 in 2024-02-09T17:48:47.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:47 smithi162 ceph-mon[44304]: osdmap e638: 12 total, 12 up, 12 in 2024-02-09T17:48:48.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:48 smithi179 ceph-mon[43975]: pgmap v1599: 47 pgs: 1 remapped+peering, 46 active+clean; 3.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 507 op/s; 18 KiB/s, 0 objects/s recovering 2024-02-09T17:48:48.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:48 smithi162 ceph-mon[44304]: pgmap v1599: 47 pgs: 1 remapped+peering, 46 active+clean; 3.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 507 op/s; 18 KiB/s, 0 objects/s recovering 2024-02-09T17:48:48.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:48 smithi099 ceph-mon[39420]: pgmap v1599: 47 pgs: 1 remapped+peering, 46 active+clean; 3.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 507 op/s; 18 KiB/s, 0 objects/s recovering 2024-02-09T17:48:49.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:49 smithi099 ceph-mon[39420]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-09T17:48:49.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:49 smithi179 ceph-mon[43975]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-09T17:48:49.683 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:49 smithi162 ceph-mon[44304]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-09T17:48:50.446 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:50 smithi099 ceph-mon[39420]: pgmap v1600: 47 pgs: 47 active+clean; 3.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 871 B/s rd, 23 KiB/s wr, 723 op/s; 12 KiB/s, 0 objects/s recovering 2024-02-09T17:48:50.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:50 smithi179 ceph-mon[43975]: pgmap v1600: 47 pgs: 47 active+clean; 3.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 871 B/s rd, 23 KiB/s wr, 723 op/s; 12 KiB/s, 0 objects/s recovering 2024-02-09T17:48:50.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:50 smithi162 ceph-mon[44304]: pgmap v1600: 47 pgs: 47 active+clean; 3.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 871 B/s rd, 23 KiB/s wr, 723 op/s; 12 KiB/s, 0 objects/s recovering 2024-02-09T17:48:51.162 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:48:51.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:51.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:48:51.268 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:48:51.301 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:51.321 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:48:51.336 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:48:51.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:51.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:48:51.448 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:48:51.523 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:51.567 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:48:51.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:51.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:48:51.674 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:48:51.732 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:51.759 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:48:51.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:51.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:48:51.865 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:48:51.903 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:51.930 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:48:51.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:51.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:48:52.033 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:48:52.089 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:52.116 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:48:52.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:52.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:48:52.209 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:48:52.223 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:48:52.255 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:52.280 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:48:52.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:52.281 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:48:52.330 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is set 2024-02-09T17:48:52.381 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:48:52.481 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:52.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:52 smithi179 ceph-mon[43975]: pgmap v1601: 47 pgs: 47 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 692 op/s 2024-02-09T17:48:52.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:52 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/800694053' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:48:52.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:52 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:48:52.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:52 smithi162 ceph-mon[44304]: pgmap v1601: 47 pgs: 47 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 692 op/s 2024-02-09T17:48:52.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:52 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/800694053' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:48:52.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:52 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:48:52.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:52 smithi099 ceph-mon[39420]: pgmap v1601: 47 pgs: 47 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 692 op/s 2024-02-09T17:48:52.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:52 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/800694053' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:48:52.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:52 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:48:53.351 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.11 to 1 (10000) 2024-02-09T17:48:53.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:53 smithi179 ceph-mon[43975]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:48:53.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:53 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:48:53.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:53 smithi179 ceph-mon[43975]: osdmap e639: 12 total, 12 up, 12 in 2024-02-09T17:48:53.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:53 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4154403744' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-02-09T17:48:53.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:53 smithi162 ceph-mon[44304]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:48:53.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:53 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:48:53.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:53 smithi162 ceph-mon[44304]: osdmap e639: 12 total, 12 up, 12 in 2024-02-09T17:48:53.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:53 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4154403744' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-02-09T17:48:53.700 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:48:53.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:53.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:48:53.710 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:53 smithi099 ceph-mon[39420]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:48:53.710 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:53 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:48:53.710 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:53 smithi099 ceph-mon[39420]: osdmap e639: 12 total, 12 up, 12 in 2024-02-09T17:48:53.710 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:53 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4154403744' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-02-09T17:48:53.807 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:48:53.845 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:53.865 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:48:53.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:53.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:48:53.969 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:48:53.995 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:54.014 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:48:54.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:54.017 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:48:54.119 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:48:54.169 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:54.504 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:54 smithi099 ceph-mon[39420]: pgmap v1603: 47 pgs: 47 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 935 op/s 2024-02-09T17:48:54.504 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:54 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:48:54.504 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:54 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:48:54.504 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:54 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4154403744' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-02-09T17:48:54.504 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:54 smithi099 ceph-mon[39420]: osdmap e640: 12 total, 12 up, 12 in 2024-02-09T17:48:54.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:54 smithi179 ceph-mon[43975]: pgmap v1603: 47 pgs: 47 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 935 op/s 2024-02-09T17:48:54.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:54 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:48:54.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:54 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:48:54.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:54 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4154403744' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-02-09T17:48:54.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:54 smithi179 ceph-mon[43975]: osdmap e640: 12 total, 12 up, 12 in 2024-02-09T17:48:54.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:54 smithi162 ceph-mon[44304]: pgmap v1603: 47 pgs: 47 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 935 op/s 2024-02-09T17:48:54.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:54 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:48:54.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:54 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:48:54.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:54 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4154403744' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-02-09T17:48:54.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:54 smithi162 ceph-mon[44304]: osdmap e640: 12 total, 12 up, 12 in 2024-02-09T17:48:54.839 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:48:54.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:54.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:48:54.893 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 0 1 2024-02-09T17:48:54.944 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:48:54.976 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:55.001 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:48:55.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:55.007 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:48:55.108 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:48:55.164 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:55.189 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:48:55.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:55.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:48:55.294 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:48:55.331 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:55.368 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:48:55.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:55.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:48:55.432 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:55 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:48:55.432 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:55 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:48:55.475 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:48:55.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:55 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:48:55.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:55 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:48:55.517 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:55.553 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:48:55.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:55.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:48:55.587 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:48:55.661 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:48:55.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:55 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:48:55.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:55 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:48:55.711 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:55.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:55.863 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:48:55.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:48:55.965 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:48:56.012 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:56.048 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:48:56.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:56.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:48:56.155 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:48:56.216 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:56.342 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:56 smithi162 ceph-mon[44304]: pgmap v1605: 47 pgs: 47 active+clean; 3.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 855 op/s 2024-02-09T17:48:56.343 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:56 smithi162 ceph-mon[44304]: osdmap e641: 12 total, 12 up, 12 in 2024-02-09T17:48:56.502 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:56 smithi099 ceph-mon[39420]: pgmap v1605: 47 pgs: 47 active+clean; 3.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 855 op/s 2024-02-09T17:48:56.502 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:56 smithi099 ceph-mon[39420]: osdmap e641: 12 total, 12 up, 12 in 2024-02-09T17:48:56.582 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:48:56.582 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [ 2024-02-09T17:48:56.583 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.583 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25837.0:210036 6.2 e641/576 6:4415b461:::benchmark_data_smithi179_126206_object79655:head v 641'13512, mlcod=641'13509)", 2024-02-09T17:48:56.583 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:48:56.550971+0000", 2024-02-09T17:48:56.583 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.018277328999999998, 2024-02-09T17:48:56.583 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.018309712999999998, 2024-02-09T17:48:56.583 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:48:56.583 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T17:48:56.583 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:48:56.583 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.583 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:48:56.583 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:56.550971+0000", 2024-02-09T17:48:56.583 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:48:56.583 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:56.583 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.584 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:48:56.584 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:56.550971+0000", 2024-02-09T17:48:56.584 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:48:56.584 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:56.584 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.584 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:48:56.584 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:56.550972+0000", 2024-02-09T17:48:56.584 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.4099999999999997e-07 2024-02-09T17:48:56.584 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:56.584 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.584 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:48:56.584 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:56.550978+0000", 2024-02-09T17:48:56.584 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.1589999999999998e-06 2024-02-09T17:48:56.584 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:56.585 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.585 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:48:56.585 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:56.550992+0000", 2024-02-09T17:48:56.585 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4048e-05 2024-02-09T17:48:56.585 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:56.585 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.585 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:48:56.585 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:56.551000+0000", 2024-02-09T17:48:56.585 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.2690000000000002e-06 2024-02-09T17:48:56.585 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:56.585 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.585 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:48:56.585 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:56.551087+0000", 2024-02-09T17:48:56.585 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.6817999999999994e-05 2024-02-09T17:48:56.585 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:56.586 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.586 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:48:56.586 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:56.551105+0000", 2024-02-09T17:48:56.586 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.8022000000000001e-05 2024-02-09T17:48:56.586 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:48:56.586 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:48:56.586 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:48:56.586 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:56.586 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.586 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25837.0:210040 6.f e641/576 6:fbc49566:::benchmark_data_smithi179_126206_object79659:head v 641'12798, mlcod=641'12796)", 2024-02-09T17:48:56.586 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:48:56.551505+0000", 2024-02-09T17:48:56.586 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.017743564999999999, 2024-02-09T17:48:56.586 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.017842343, 2024-02-09T17:48:56.586 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:48:56.587 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T17:48:56.587 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:48:56.587 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.587 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:48:56.587 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:56.551505+0000", 2024-02-09T17:48:56.587 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:48:56.587 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:56.587 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.587 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:48:56.587 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:56.551505+0000", 2024-02-09T17:48:56.587 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:48:56.587 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:56.587 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.587 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:48:56.587 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:56.551506+0000", 2024-02-09T17:48:56.588 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0839999999999999e-06 2024-02-09T17:48:56.588 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:56.588 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.588 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:48:56.588 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:56.551513+0000", 2024-02-09T17:48:56.588 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.08e-06 2024-02-09T17:48:56.588 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:56.588 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.588 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:48:56.588 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:56.551524+0000", 2024-02-09T17:48:56.588 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0968000000000001e-05 2024-02-09T17:48:56.588 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:56.588 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.589 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:48:56.589 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:56.551532+0000", 2024-02-09T17:48:56.589 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.5480000000000004e-06 2024-02-09T17:48:56.589 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:56.589 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.589 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:48:56.589 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:56.551605+0000", 2024-02-09T17:48:56.589 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.2960999999999995e-05 2024-02-09T17:48:56.589 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:56.589 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.589 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:48:56.589 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:56.551626+0000", 2024-02-09T17:48:56.589 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.1291999999999999e-05 2024-02-09T17:48:56.589 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:48:56.590 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:48:56.590 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:48:56.590 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:56.590 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.590 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25837.0:210039 6.2 e641/576 6:405f2964:::benchmark_data_smithi179_126206_object79658:head v 641'13513, mlcod=641'13509)", 2024-02-09T17:48:56.590 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:48:56.555788+0000", 2024-02-09T17:48:56.590 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.013460765, 2024-02-09T17:48:56.590 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.013626374, 2024-02-09T17:48:56.590 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:48:56.590 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T17:48:56.590 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:48:56.590 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.590 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:48:56.590 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:56.555788+0000", 2024-02-09T17:48:56.590 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:48:56.591 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:56.591 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.591 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:48:56.591 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:56.555788+0000", 2024-02-09T17:48:56.591 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:48:56.591 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:56.591 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.591 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:48:56.591 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:56.555789+0000", 2024-02-09T17:48:56.591 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.5600000000000004e-07 2024-02-09T17:48:56.591 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:56.591 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.591 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:48:56.591 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:56.555797+0000", 2024-02-09T17:48:56.592 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.2689999999999999e-06 2024-02-09T17:48:56.592 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:56.592 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.592 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:48:56.592 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:56.555814+0000", 2024-02-09T17:48:56.592 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.6331e-05 2024-02-09T17:48:56.592 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:56.592 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.592 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:48:56.592 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:56.555822+0000", 2024-02-09T17:48:56.592 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.2879999999999993e-06 2024-02-09T17:48:56.592 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:56.592 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.592 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:48:56.592 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:56.555912+0000", 2024-02-09T17:48:56.593 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.0076999999999998e-05 2024-02-09T17:48:56.593 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:56.593 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.593 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:48:56.593 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:56.555927+0000", 2024-02-09T17:48:56.593 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5058999999999999e-05 2024-02-09T17:48:56.593 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:48:56.593 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:48:56.593 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:48:56.593 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:56.593 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.593 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25837.0:210042 6.2 e641/576 6:48919e8a:::benchmark_data_smithi179_126206_object79661:head v 641'13514, mlcod=641'13509)", 2024-02-09T17:48:56.593 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:48:56.559063+0000", 2024-02-09T17:48:56.593 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.010185563999999999, 2024-02-09T17:48:56.593 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.010409494, 2024-02-09T17:48:56.594 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:48:56.594 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T17:48:56.594 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:48:56.594 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.594 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:48:56.594 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:56.559063+0000", 2024-02-09T17:48:56.594 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:48:56.594 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:56.594 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.594 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:48:56.594 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:56.559063+0000", 2024-02-09T17:48:56.594 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:48:56.594 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:56.594 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.594 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:48:56.595 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:56.559064+0000", 2024-02-09T17:48:56.595 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.0400000000000005e-07 2024-02-09T17:48:56.595 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:56.595 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.595 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:48:56.595 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:56.559071+0000", 2024-02-09T17:48:56.595 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.6839999999999999e-06 2024-02-09T17:48:56.595 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:56.595 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.595 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:48:56.595 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:56.559083+0000", 2024-02-09T17:48:56.595 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.218e-05 2024-02-09T17:48:56.595 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:56.595 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.595 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:48:56.596 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:56.559090+0000", 2024-02-09T17:48:56.596 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.2660000000000004e-06 2024-02-09T17:48:56.596 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:56.596 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.596 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:48:56.596 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:56.559808+0000", 2024-02-09T17:48:56.596 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00071744500000000004 2024-02-09T17:48:56.596 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:56.596 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.596 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:48:56.596 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:56.559827+0000", 2024-02-09T17:48:56.596 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.9828000000000001e-05 2024-02-09T17:48:56.596 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:48:56.596 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:48:56.597 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:48:56.597 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:56.597 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.597 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25837.0:210047 6.f e641/576 6:f6d47e38:::benchmark_data_smithi179_126206_object79666:head v 641'12799, mlcod=641'12796)", 2024-02-09T17:48:56.597 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:48:56.560834+0000", 2024-02-09T17:48:56.597 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.0084142790000000002, 2024-02-09T17:48:56.597 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0086975409999999996, 2024-02-09T17:48:56.597 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:48:56.597 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T17:48:56.597 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:48:56.597 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.597 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:48:56.597 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:56.560834+0000", 2024-02-09T17:48:56.597 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:48:56.597 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:56.598 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.598 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:48:56.598 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:56.560834+0000", 2024-02-09T17:48:56.598 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:48:56.598 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:56.598 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.598 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:48:56.598 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:56.560835+0000", 2024-02-09T17:48:56.598 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.8200000000000008e-07 2024-02-09T17:48:56.598 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:56.598 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.598 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:48:56.598 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:56.560843+0000", 2024-02-09T17:48:56.598 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.7940000000000003e-06 2024-02-09T17:48:56.598 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:56.599 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.599 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:48:56.599 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:56.560857+0000", 2024-02-09T17:48:56.599 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4204e-05 2024-02-09T17:48:56.599 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:56.599 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.599 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:48:56.599 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:56.560866+0000", 2024-02-09T17:48:56.599 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.3180000000000007e-06 2024-02-09T17:48:56.599 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:56.599 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.599 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:48:56.599 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:56.560942+0000", 2024-02-09T17:48:56.599 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.6309000000000004e-05 2024-02-09T17:48:56.599 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:56.600 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.600 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:48:56.600 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:56.560966+0000", 2024-02-09T17:48:56.600 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.4170000000000001e-05 2024-02-09T17:48:56.600 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:48:56.600 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:48:56.600 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:48:56.600 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:56.600 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.600 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25837.0:210043 6.0 e641/576)", 2024-02-09T17:48:56.600 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:48:56.568817+0000", 2024-02-09T17:48:56.600 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.000431977, 2024-02-09T17:48:56.600 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00076885699999999996, 2024-02-09T17:48:56.600 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:48:56.600 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "queued for pg", 2024-02-09T17:48:56.601 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:48:56.601 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.601 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:48:56.601 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:56.568817+0000", 2024-02-09T17:48:56.601 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:48:56.601 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:56.601 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.601 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:48:56.601 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:56.568817+0000", 2024-02-09T17:48:56.601 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:48:56.601 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:56.601 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.601 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:48:56.601 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:56.568818+0000", 2024-02-09T17:48:56.602 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.1900000000000001e-07 2024-02-09T17:48:56.602 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:56.602 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.602 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:48:56.602 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:56.568824+0000", 2024-02-09T17:48:56.602 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.9349999999999999e-06 2024-02-09T17:48:56.602 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:56.602 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.602 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:48:56.602 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:56.568840+0000", 2024-02-09T17:48:56.602 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.6654e-05 2024-02-09T17:48:56.602 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:48:56.603 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:48:56.603 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:48:56.603 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:56.568849+0000", 2024-02-09T17:48:56.603 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.5799999999999992e-06 2024-02-09T17:48:56.603 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:48:56.603 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:48:56.603 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:48:56.603 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:48:56.603 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T17:48:56.603 INFO:teuthology.orchestra.run.smithi099.stdout: "num_ops": 6 2024-02-09T17:48:56.603 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:48:56.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:56 smithi179 ceph-mon[43975]: pgmap v1605: 47 pgs: 47 active+clean; 3.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 855 op/s 2024-02-09T17:48:56.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:56 smithi179 ceph-mon[43975]: osdmap e641: 12 total, 12 up, 12 in 2024-02-09T17:48:57.581 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset noscrub 2024-02-09T17:48:58.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:48:58 smithi179 ceph-mon[43975]: pgmap v1607: 47 pgs: 47 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 847 op/s 2024-02-09T17:48:58.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:48:58 smithi162 ceph-mon[44304]: pgmap v1607: 47 pgs: 47 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 847 op/s 2024-02-09T17:48:58.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:48:58 smithi099 ceph-mon[39420]: pgmap v1607: 47 pgs: 47 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 847 op/s 2024-02-09T17:48:59.264 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-09T17:48:59.271 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:48:59.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:59.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:48:59.290 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:48:59.380 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:48:59.407 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:59.431 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:48:59.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:59.439 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:48:59.540 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:48:59.571 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:59.607 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:48:59.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:59.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:48:59.714 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:48:59.741 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:48:59.765 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:48:59.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:48:59.773 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:48:59.873 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:48:59.925 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:00.086 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:49:00.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:00.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:49:00.197 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:49:00.226 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:00.248 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:49:00.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:00.252 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:49:00.354 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:49:00.416 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:00.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:00 smithi179 ceph-mon[43975]: pgmap v1608: 47 pgs: 47 active+clean; 3.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 916 op/s 2024-02-09T17:49:00.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:00 smithi162 ceph-mon[44304]: pgmap v1608: 47 pgs: 47 active+clean; 3.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 916 op/s 2024-02-09T17:49:00.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:00 smithi099 ceph-mon[39420]: pgmap v1608: 47 pgs: 47 active+clean; 3.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 916 op/s 2024-02-09T17:49:01.193 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:49:01.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:01.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:49:01.307 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:49:01.337 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:49:01.337 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:49:01.337 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:49:01.337 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:49:01.337 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:49:01.338 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:49:01.338 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:49:01.338 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:49:01.339 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:01.352 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:49:01.361 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:49:01.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:01.365 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:49:01.390 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:49:01.426 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:49:01.466 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:49:01.471 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:49:01.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:01 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/366954732' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:49:01.500 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:49:01.500 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:49:01.500 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:49:01.500 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:49:01.500 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:49:01.500 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:49:01.500 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:49:01.500 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:49:01.501 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:49:01.501 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:49:01.501 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:49:01.501 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:49:01.513 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:01.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:01 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/366954732' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:49:01.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:01 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/366954732' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:49:02.197 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:49:02.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:02.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:49:02.209 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:49:02.306 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:49:02.370 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:02.461 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:02 smithi099 ceph-mon[39420]: pgmap v1609: 47 pgs: 47 active+clean; 3.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 636 op/s 2024-02-09T17:49:02.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:02 smithi179 ceph-mon[43975]: pgmap v1609: 47 pgs: 47 active+clean; 3.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 636 op/s 2024-02-09T17:49:02.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:02 smithi162 ceph-mon[44304]: pgmap v1609: 47 pgs: 47 active+clean; 3.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 636 op/s 2024-02-09T17:49:03.133 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:49:03.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:03.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:49:03.237 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:49:03.312 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:03.342 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:49:03.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:03.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:49:03.449 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:49:03.498 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:03.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:03 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:49:03.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:03 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:49:03.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:03 smithi162 ceph-mon[44304]: osdmap e642: 12 total, 12 up, 12 in 2024-02-09T17:49:03.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:03 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:49:03.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:03 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:49:03.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:03 smithi099 ceph-mon[39420]: osdmap e642: 12 total, 12 up, 12 in 2024-02-09T17:49:03.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:03 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:49:03.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:03 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:49:03.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:03 smithi179 ceph-mon[43975]: osdmap e642: 12 total, 12 up, 12 in 2024-02-09T17:49:03.955 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:49:03.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:03.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:49:04.063 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:49:04.094 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:04.120 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:49:04.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:04.123 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:49:04.224 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:49:04.286 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:04.611 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:04 smithi099 ceph-mon[39420]: pgmap v1611: 47 pgs: 47 active+clean; 2.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 934 op/s 2024-02-09T17:49:04.753 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:49:04.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:04.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:49:04.857 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:49:04.890 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:04 smithi162 ceph-mon[44304]: pgmap v1611: 47 pgs: 47 active+clean; 2.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 934 op/s 2024-02-09T17:49:04.899 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:04.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:04 smithi179 ceph-mon[43975]: pgmap v1611: 47 pgs: 47 active+clean; 2.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 934 op/s 2024-02-09T17:49:04.935 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:49:04.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:04.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:49:05.044 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:49:05.088 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:05.225 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:49:05.256 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:05.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:49:05.358 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:49:05.447 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:05.586 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:49:05.591 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:49:05.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:05.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:49:05.696 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:49:05.758 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:05.874 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:49:05.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:05.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:49:05.986 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:49:06.224 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:06.518 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.0 to 1 (10000) 2024-02-09T17:49:06.653 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:49:06.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:06 smithi179 ceph-mon[43975]: pgmap v1612: 47 pgs: 47 active+clean; 2.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 854 op/s 2024-02-09T17:49:06.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:06 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4050957598' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-09T17:49:06.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:06 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:49:06.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:06 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:49:06.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:06 smithi162 ceph-mon[44304]: pgmap v1612: 47 pgs: 47 active+clean; 2.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 854 op/s 2024-02-09T17:49:06.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:06 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4050957598' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-09T17:49:06.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:06 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:49:06.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:06 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:49:06.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:06 smithi099 ceph-mon[39420]: pgmap v1612: 47 pgs: 47 active+clean; 2.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 854 op/s 2024-02-09T17:49:06.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:06 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4050957598' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-09T17:49:06.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:06 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:49:06.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:06 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:49:07.797 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:49:07.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:07.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:49:07.801 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:07 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4050957598' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-09T17:49:07.801 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:07 smithi099 ceph-mon[39420]: osdmap e643: 12 total, 12 up, 12 in 2024-02-09T17:49:07.900 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:49:07.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:07 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4050957598' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-09T17:49:07.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:07 smithi179 ceph-mon[43975]: osdmap e643: 12 total, 12 up, 12 in 2024-02-09T17:49:07.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:07 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4050957598' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-09T17:49:07.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:07 smithi162 ceph-mon[44304]: osdmap e643: 12 total, 12 up, 12 in 2024-02-09T17:49:07.927 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:07.957 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:49:07.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:07.960 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:49:08.061 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:49:08.093 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:08.117 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:49:08.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:08.125 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:49:08.226 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:49:08.267 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:08.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:08 smithi099 ceph-mon[39420]: pgmap v1614: 47 pgs: 47 active+clean; 2.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 754 op/s 2024-02-09T17:49:08.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:08 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/424254877' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:49:08.840 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is unset 2024-02-09T17:49:08.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:08 smithi179 ceph-mon[43975]: pgmap v1614: 47 pgs: 47 active+clean; 2.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 754 op/s 2024-02-09T17:49:08.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:08 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/424254877' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:49:08.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:08 smithi162 ceph-mon[44304]: pgmap v1614: 47 pgs: 47 active+clean; 2.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 754 op/s 2024-02-09T17:49:08.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:08 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/424254877' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:49:09.636 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:09 smithi099 ceph-mon[39420]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:49:09.636 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:09 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/424254877' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:49:09.636 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:09 smithi099 ceph-mon[39420]: osdmap e644: 12 total, 12 up, 12 in 2024-02-09T17:49:09.636 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:09 smithi099 ceph-mon[39420]: 1.1b scrub starts 2024-02-09T17:49:09.636 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:09 smithi099 ceph-mon[39420]: 1.1b scrub ok 2024-02-09T17:49:09.642 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:49:09.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:09.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:49:09.755 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:49:09.794 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:09.799 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:09 smithi162 ceph-mon[44304]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:49:09.799 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:09 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/424254877' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:49:09.799 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:09 smithi162 ceph-mon[44304]: osdmap e644: 12 total, 12 up, 12 in 2024-02-09T17:49:09.799 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:09 smithi162 ceph-mon[44304]: 1.1b scrub starts 2024-02-09T17:49:09.799 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:09 smithi162 ceph-mon[44304]: 1.1b scrub ok 2024-02-09T17:49:09.824 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:49:09.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:09.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:49:09.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:09 smithi179 ceph-mon[43975]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:49:09.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:09 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/424254877' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:49:09.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:09 smithi179 ceph-mon[43975]: osdmap e644: 12 total, 12 up, 12 in 2024-02-09T17:49:09.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:09 smithi179 ceph-mon[43975]: 1.1b scrub starts 2024-02-09T17:49:09.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:09 smithi179 ceph-mon[43975]: 1.1b scrub ok 2024-02-09T17:49:09.930 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:49:09.961 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:09.992 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:49:09.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:09.997 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:49:10.098 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:49:10.135 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:10.176 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:49:10.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:10.179 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:49:10.280 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:49:10.343 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:10.471 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:49:10.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:10.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:49:10.494 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 2 1 2024-02-09T17:49:10.523 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:49:10.578 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:49:10.612 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:10.640 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:49:10.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:10.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:49:10.722 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:10 smithi099 ceph-mon[39420]: pgmap v1616: 47 pgs: 47 active+clean; 2.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 967 op/s 2024-02-09T17:49:10.722 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:10 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:49:10.722 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:10 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:49:10.722 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:10 smithi099 ceph-mon[39420]: osdmap e645: 12 total, 12 up, 12 in 2024-02-09T17:49:10.747 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:49:10.777 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:10.802 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:49:10.806 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:10 smithi179 ceph-mon[43975]: pgmap v1616: 47 pgs: 47 active+clean; 2.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 967 op/s 2024-02-09T17:49:10.806 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:10 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:49:10.806 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:10 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:49:10.806 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:10 smithi179 ceph-mon[43975]: osdmap e645: 12 total, 12 up, 12 in 2024-02-09T17:49:10.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:10.807 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:49:10.908 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:49:10.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:10 smithi162 ceph-mon[44304]: pgmap v1616: 47 pgs: 47 active+clean; 2.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 967 op/s 2024-02-09T17:49:10.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:10 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:49:10.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:10 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:49:10.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:10 smithi162 ceph-mon[44304]: osdmap e645: 12 total, 12 up, 12 in 2024-02-09T17:49:10.953 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:10.991 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:49:10.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:10.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:49:11.098 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:49:11.149 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:11.442 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:49:11.442 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:49:11.475 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:49:11.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:11.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:49:11.585 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:49:11.649 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:11.680 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:49:11.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:11.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:49:11.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:11 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:49:11.748 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:11 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:49:11.748 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:11 smithi099 ceph-mon[39420]: 6.d scrub starts 2024-02-09T17:49:11.748 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:11 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:49:11.748 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:11 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:49:11.748 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:11 smithi099 ceph-mon[39420]: osdmap e646: 12 total, 12 up, 12 in 2024-02-09T17:49:11.748 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:11 smithi099 ceph-mon[39420]: pgmap v1619: 47 pgs: 47 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 775 op/s 2024-02-09T17:49:11.789 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:49:11.816 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:11 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:49:11.816 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:11 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:49:11.816 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:11 smithi179 ceph-mon[43975]: 6.d scrub starts 2024-02-09T17:49:11.816 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:11 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:49:11.817 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:11 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:49:11.817 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:11 smithi179 ceph-mon[43975]: osdmap e646: 12 total, 12 up, 12 in 2024-02-09T17:49:11.817 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:11 smithi179 ceph-mon[43975]: pgmap v1619: 47 pgs: 47 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 775 op/s 2024-02-09T17:49:11.818 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:11.838 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:49:11.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:11.842 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:49:11.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:11 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:49:11.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:11 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:49:11.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:11 smithi162 ceph-mon[44304]: 6.d scrub starts 2024-02-09T17:49:11.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:11 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:49:11.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:11 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:49:11.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:11 smithi162 ceph-mon[44304]: osdmap e646: 12 total, 12 up, 12 in 2024-02-09T17:49:11.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:11 smithi162 ceph-mon[44304]: pgmap v1619: 47 pgs: 47 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 775 op/s 2024-02-09T17:49:11.943 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:49:12.009 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:12.023 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-09T17:49:12.042 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:49:12.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:12.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:49:12.150 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:49:12.180 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:12.204 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:49:12.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:12.208 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:49:12.309 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:49:12.339 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:12.361 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:49:12.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:12.364 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:49:12.465 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:49:12.500 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:12.529 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:49:12.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:12.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:49:12.639 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:49:12.688 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:12.715 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:49:12.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:12.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:49:12.821 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:49:12.874 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:12 smithi099 ceph-mon[39420]: 6.d scrub ok 2024-02-09T17:49:12.874 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:12 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:49:12.874 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:12 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:49:12.874 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:12 smithi099 ceph-mon[39420]: 1.7 scrub starts 2024-02-09T17:49:12.875 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:12 smithi099 ceph-mon[39420]: 1.7 scrub ok 2024-02-09T17:49:12.875 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:12 smithi099 ceph-mon[39420]: 1.1 scrub starts 2024-02-09T17:49:12.875 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:12 smithi099 ceph-mon[39420]: 1.1 scrub ok 2024-02-09T17:49:12.875 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:12 smithi099 ceph-mon[39420]: osdmap e647: 12 total, 12 up, 12 in 2024-02-09T17:49:12.875 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:12 smithi099 ceph-mon[39420]: 1.c scrub starts 2024-02-09T17:49:12.875 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:12 smithi099 ceph-mon[39420]: 1.2 scrub starts 2024-02-09T17:49:12.875 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:12 smithi099 ceph-mon[39420]: 1.2 scrub ok 2024-02-09T17:49:12.875 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:12 smithi099 ceph-mon[39420]: 1.c scrub ok 2024-02-09T17:49:12.875 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:12 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:49:12.875 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:12 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:49:12.875 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:12 smithi099 ceph-mon[39420]: 1.18 scrub starts 2024-02-09T17:49:12.875 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:12 smithi099 ceph-mon[39420]: 1.18 scrub ok 2024-02-09T17:49:12.876 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:12.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:12 smithi179 ceph-mon[43975]: 6.d scrub ok 2024-02-09T17:49:12.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:12 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:49:12.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:12 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:49:12.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:12 smithi179 ceph-mon[43975]: 1.7 scrub starts 2024-02-09T17:49:12.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:12 smithi179 ceph-mon[43975]: 1.7 scrub ok 2024-02-09T17:49:12.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:12 smithi179 ceph-mon[43975]: 1.1 scrub starts 2024-02-09T17:49:12.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:12 smithi179 ceph-mon[43975]: 1.1 scrub ok 2024-02-09T17:49:12.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:12 smithi179 ceph-mon[43975]: osdmap e647: 12 total, 12 up, 12 in 2024-02-09T17:49:12.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:12 smithi179 ceph-mon[43975]: 1.c scrub starts 2024-02-09T17:49:12.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:12 smithi179 ceph-mon[43975]: 1.2 scrub starts 2024-02-09T17:49:12.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:12 smithi179 ceph-mon[43975]: 1.2 scrub ok 2024-02-09T17:49:12.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:12 smithi179 ceph-mon[43975]: 1.c scrub ok 2024-02-09T17:49:12.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:12 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:49:12.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:12 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:49:12.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:12 smithi179 ceph-mon[43975]: 1.18 scrub starts 2024-02-09T17:49:12.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:12 smithi179 ceph-mon[43975]: 1.18 scrub ok 2024-02-09T17:49:12.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:12 smithi162 ceph-mon[44304]: 6.d scrub ok 2024-02-09T17:49:12.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:12 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:49:12.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:12 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:49:12.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:12 smithi162 ceph-mon[44304]: 1.7 scrub starts 2024-02-09T17:49:12.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:12 smithi162 ceph-mon[44304]: 1.7 scrub ok 2024-02-09T17:49:12.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:12 smithi162 ceph-mon[44304]: 1.1 scrub starts 2024-02-09T17:49:12.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:12 smithi162 ceph-mon[44304]: 1.1 scrub ok 2024-02-09T17:49:12.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:12 smithi162 ceph-mon[44304]: osdmap e647: 12 total, 12 up, 12 in 2024-02-09T17:49:12.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:12 smithi162 ceph-mon[44304]: 1.c scrub starts 2024-02-09T17:49:12.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:12 smithi162 ceph-mon[44304]: 1.2 scrub starts 2024-02-09T17:49:12.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:12 smithi162 ceph-mon[44304]: 1.2 scrub ok 2024-02-09T17:49:12.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:12 smithi162 ceph-mon[44304]: 1.c scrub ok 2024-02-09T17:49:12.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:12 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:49:12.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:12 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:49:12.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:12 smithi162 ceph-mon[44304]: 1.18 scrub starts 2024-02-09T17:49:12.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:12 smithi162 ceph-mon[44304]: 1.18 scrub ok 2024-02-09T17:49:14.075 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:49:14.083 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:13 smithi099 ceph-mon[39420]: 1.4 scrub starts 2024-02-09T17:49:14.083 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:13 smithi099 ceph-mon[39420]: 1.4 scrub ok 2024-02-09T17:49:14.083 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:13 smithi099 ceph-mon[39420]: osdmap e648: 12 total, 12 up, 12 in 2024-02-09T17:49:14.083 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:13 smithi099 ceph-mon[39420]: 1.8 scrub starts 2024-02-09T17:49:14.083 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:13 smithi099 ceph-mon[39420]: 1.8 scrub ok 2024-02-09T17:49:14.083 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:13 smithi099 ceph-mon[39420]: pgmap v1622: 47 pgs: 47 active+clean; 2.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 972 op/s 2024-02-09T17:49:14.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:14.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:49:14.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:13 smithi179 ceph-mon[43975]: 1.4 scrub starts 2024-02-09T17:49:14.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:13 smithi179 ceph-mon[43975]: 1.4 scrub ok 2024-02-09T17:49:14.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:13 smithi179 ceph-mon[43975]: osdmap e648: 12 total, 12 up, 12 in 2024-02-09T17:49:14.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:13 smithi179 ceph-mon[43975]: 1.8 scrub starts 2024-02-09T17:49:14.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:13 smithi179 ceph-mon[43975]: 1.8 scrub ok 2024-02-09T17:49:14.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:13 smithi179 ceph-mon[43975]: pgmap v1622: 47 pgs: 47 active+clean; 2.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 972 op/s 2024-02-09T17:49:14.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:13 smithi162 ceph-mon[44304]: 1.4 scrub starts 2024-02-09T17:49:14.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:13 smithi162 ceph-mon[44304]: 1.4 scrub ok 2024-02-09T17:49:14.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:13 smithi162 ceph-mon[44304]: osdmap e648: 12 total, 12 up, 12 in 2024-02-09T17:49:14.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:13 smithi162 ceph-mon[44304]: 1.8 scrub starts 2024-02-09T17:49:14.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:13 smithi162 ceph-mon[44304]: 1.8 scrub ok 2024-02-09T17:49:14.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:13 smithi162 ceph-mon[44304]: pgmap v1622: 47 pgs: 47 active+clean; 2.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 972 op/s 2024-02-09T17:49:14.187 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:49:14.262 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:14.293 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:49:14.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:14.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:49:14.398 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:49:14.435 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:14.464 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:49:14.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:14.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:49:14.569 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:49:14.616 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:14.650 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:49:14.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:14.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:49:14.758 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:49:14.824 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:14.861 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:49:14.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:14.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:49:14.966 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:49:14.985 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-09T17:49:15.058 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:15.143 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:49:15.143 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:49:15.144 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:49:15.144 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:49:15.159 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:49:15.161 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:49:15.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:15.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:49:15.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:14 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3857193179' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:49:15.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:14 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3857193179' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:49:15.273 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:49:15.274 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:49:15.325 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:49:15.366 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:49:15.366 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:15.408 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:49:15.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:14 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3857193179' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:49:15.431 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:49:15.448 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:49:15.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:15.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:49:15.467 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:49:15.519 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:49:15.553 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:49:15.561 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:49:15.571 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:49:15.572 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:49:15.572 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:49:15.572 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:49:15.572 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:49:15.572 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:49:15.572 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:49:15.572 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:49:15.572 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:49:15.572 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:49:15.572 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:49:15.572 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:49:15.604 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:15.644 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:49:15.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:15.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:49:15.754 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:49:15.796 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:15.833 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:49:15.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:15.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:49:15.944 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:49:16.014 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:16.090 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:15 smithi099 ceph-mon[39420]: 6.2 scrub starts 2024-02-09T17:49:16.090 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:15 smithi099 ceph-mon[39420]: pgmap v1623: 47 pgs: 47 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 882 op/s 2024-02-09T17:49:16.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:15 smithi179 ceph-mon[43975]: 6.2 scrub starts 2024-02-09T17:49:16.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:15 smithi179 ceph-mon[43975]: pgmap v1623: 47 pgs: 47 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 882 op/s 2024-02-09T17:49:16.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:15 smithi162 ceph-mon[44304]: 6.2 scrub starts 2024-02-09T17:49:16.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:15 smithi162 ceph-mon[44304]: pgmap v1623: 47 pgs: 47 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 882 op/s 2024-02-09T17:49:17.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:16 smithi099 ceph-mon[39420]: 6.2 scrub ok 2024-02-09T17:49:17.283 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:49:17.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:17.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:49:17.395 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:49:17.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:16 smithi179 ceph-mon[43975]: 6.2 scrub ok 2024-02-09T17:49:17.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:16 smithi162 ceph-mon[44304]: 6.2 scrub ok 2024-02-09T17:49:17.450 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:17.667 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:49:17.669 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:49:17.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:17.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:49:17.776 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:49:17.823 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:17.967 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:49:17.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:17.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:49:17.978 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:49:18.073 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:49:18.110 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:18.132 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:49:18.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:18.135 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:49:18.236 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:49:18.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:17 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:49:18.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:17 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:49:18.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:17 smithi099 ceph-mon[39420]: 1.19 scrub starts 2024-02-09T17:49:18.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:17 smithi099 ceph-mon[39420]: 1.19 scrub ok 2024-02-09T17:49:18.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:17 smithi099 ceph-mon[39420]: osdmap e649: 12 total, 12 up, 12 in 2024-02-09T17:49:18.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:17 smithi099 ceph-mon[39420]: pgmap v1625: 47 pgs: 47 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 836 op/s 2024-02-09T17:49:18.264 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:17 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:49:18.264 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:17 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:49:18.265 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:17 smithi179 ceph-mon[43975]: 1.19 scrub starts 2024-02-09T17:49:18.265 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:17 smithi179 ceph-mon[43975]: 1.19 scrub ok 2024-02-09T17:49:18.265 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:17 smithi179 ceph-mon[43975]: osdmap e649: 12 total, 12 up, 12 in 2024-02-09T17:49:18.265 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:17 smithi179 ceph-mon[43975]: pgmap v1625: 47 pgs: 47 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 836 op/s 2024-02-09T17:49:18.266 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:18.292 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:49:18.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:18.295 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:49:18.396 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:49:18.424 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:18.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:17 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:49:18.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:17 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:49:18.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:17 smithi162 ceph-mon[44304]: 1.19 scrub starts 2024-02-09T17:49:18.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:17 smithi162 ceph-mon[44304]: 1.19 scrub ok 2024-02-09T17:49:18.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:17 smithi162 ceph-mon[44304]: osdmap e649: 12 total, 12 up, 12 in 2024-02-09T17:49:18.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:17 smithi162 ceph-mon[44304]: pgmap v1625: 47 pgs: 47 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 836 op/s 2024-02-09T17:49:18.444 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:49:18.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:18.448 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:49:18.549 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:49:18.613 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:18.780 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:49:18.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:18.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:49:18.885 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:49:18.947 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:18.996 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:49:19.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:19.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:49:19.108 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:49:19.140 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:19.177 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:49:19.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:19.181 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:49:19.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:18 smithi099 ceph-mon[39420]: 1.1c scrub starts 2024-02-09T17:49:19.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:18 smithi099 ceph-mon[39420]: 1.1c scrub ok 2024-02-09T17:49:19.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:18 smithi099 ceph-mon[39420]: 6.a scrub starts 2024-02-09T17:49:19.281 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:49:19.308 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:18 smithi179 ceph-mon[43975]: 1.1c scrub starts 2024-02-09T17:49:19.308 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:18 smithi179 ceph-mon[43975]: 1.1c scrub ok 2024-02-09T17:49:19.308 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:18 smithi179 ceph-mon[43975]: 6.a scrub starts 2024-02-09T17:49:19.310 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:19.315 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:49:19.331 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:49:19.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:19.335 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:49:19.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:19 smithi162 ceph-mon[44304]: 1.1c scrub starts 2024-02-09T17:49:19.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:19 smithi162 ceph-mon[44304]: 1.1c scrub ok 2024-02-09T17:49:19.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:19 smithi162 ceph-mon[44304]: 6.a scrub starts 2024-02-09T17:49:19.435 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:49:19.491 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:20.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:20 smithi179 ceph-mon[43975]: 6.a scrub ok 2024-02-09T17:49:20.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:20 smithi179 ceph-mon[43975]: pgmap v1626: 47 pgs: 47 active+clean; 2.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 880 op/s 2024-02-09T17:49:20.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:20 smithi162 ceph-mon[44304]: 6.a scrub ok 2024-02-09T17:49:20.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:20 smithi162 ceph-mon[44304]: pgmap v1626: 47 pgs: 47 active+clean; 2.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 880 op/s 2024-02-09T17:49:20.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:20 smithi099 ceph-mon[39420]: 6.a scrub ok 2024-02-09T17:49:20.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:20 smithi099 ceph-mon[39420]: pgmap v1626: 47 pgs: 47 active+clean; 2.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 880 op/s 2024-02-09T17:49:21.267 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:21 smithi099 ceph-mon[39420]: 1.13 scrub starts 2024-02-09T17:49:21.267 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:21 smithi099 ceph-mon[39420]: 1.13 scrub ok 2024-02-09T17:49:21.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:21 smithi179 ceph-mon[43975]: 1.13 scrub starts 2024-02-09T17:49:21.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:21 smithi179 ceph-mon[43975]: 1.13 scrub ok 2024-02-09T17:49:21.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:21 smithi162 ceph-mon[44304]: 1.13 scrub starts 2024-02-09T17:49:21.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:21 smithi162 ceph-mon[44304]: 1.13 scrub ok 2024-02-09T17:49:22.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:22 smithi099 ceph-mon[39420]: pgmap v1627: 47 pgs: 47 active+clean; 2.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 612 op/s 2024-02-09T17:49:22.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:22 smithi179 ceph-mon[43975]: pgmap v1627: 47 pgs: 47 active+clean; 2.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 612 op/s 2024-02-09T17:49:22.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:22 smithi162 ceph-mon[44304]: pgmap v1627: 47 pgs: 47 active+clean; 2.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 612 op/s 2024-02-09T17:49:23.944 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:49:23.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:23.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:49:24.050 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:49:24.059 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:49:24.059 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 20, 2024-02-09T17:49:24.059 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 600, 2024-02-09T17:49:24.059 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [ 2024-02-09T17:49:24.060 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.060 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:232568 4.4 4.35948d34 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e414)", 2024-02-09T17:49:24.060 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:39:23.630623+0000", 2024-02-09T17:49:24.060 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 599.58988373299997, 2024-02-09T17:49:24.060 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.89773093800000003, 2024-02-09T17:49:24.060 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:49:24.060 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "reached pg", 2024-02-09T17:49:24.060 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:49:24.060 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:49:24.060 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:49:24.060 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 232568 2024-02-09T17:49:24.060 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.060 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:49:24.061 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.061 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:49:24.061 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.630623+0000", 2024-02-09T17:49:24.061 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:49:24.061 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.061 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.061 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:49:24.061 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.630623+0000", 2024-02-09T17:49:24.061 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:49:24.061 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.061 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.061 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:49:24.061 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.630625+0000", 2024-02-09T17:49:24.061 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.8700000000000001e-06 2024-02-09T17:49:24.061 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.062 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.062 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:49:24.062 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.630632+0000", 2024-02-09T17:49:24.062 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.4270000000000002e-06 2024-02-09T17:49:24.062 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.062 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.062 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:49:24.062 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.630652+0000", 2024-02-09T17:49:24.062 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.9432e-05 2024-02-09T17:49:24.062 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.062 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.062 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:49:24.062 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.630662+0000", 2024-02-09T17:49:24.063 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0651e-05 2024-02-09T17:49:24.063 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.063 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.063 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:49:24.063 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.630736+0000", 2024-02-09T17:49:24.063 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.3244000000000005e-05 2024-02-09T17:49:24.063 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.063 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.063 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:49:24.063 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.630749+0000", 2024-02-09T17:49:24.063 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3168e-05 2024-02-09T17:49:24.063 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.064 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.064 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:49:24.064 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.528332+0000", 2024-02-09T17:49:24.064 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.89758274900000001 2024-02-09T17:49:24.064 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.064 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.064 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:49:24.064 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.528354+0000", 2024-02-09T17:49:24.064 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2396999999999999e-05 2024-02-09T17:49:24.064 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:49:24.064 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:49:24.064 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:49:24.064 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.064 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.065 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:232574 4.4 4.8fed9a74 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e414)", 2024-02-09T17:49:24.065 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:39:23.640912+0000", 2024-02-09T17:49:24.065 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 599.579595122, 2024-02-09T17:49:24.065 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.88749593800000004, 2024-02-09T17:49:24.065 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:49:24.065 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "reached pg", 2024-02-09T17:49:24.065 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:49:24.065 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:49:24.065 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:49:24.065 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 232574 2024-02-09T17:49:24.065 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.065 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:49:24.065 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.066 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:49:24.066 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.640912+0000", 2024-02-09T17:49:24.066 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:49:24.066 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.066 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.066 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:49:24.066 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.640912+0000", 2024-02-09T17:49:24.066 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:49:24.066 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.066 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.066 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:49:24.066 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.640914+0000", 2024-02-09T17:49:24.066 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.086e-06 2024-02-09T17:49:24.067 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.067 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.067 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:49:24.067 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.640921+0000", 2024-02-09T17:49:24.067 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.6190000000000002e-06 2024-02-09T17:49:24.067 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.067 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.067 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:49:24.067 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.640939+0000", 2024-02-09T17:49:24.067 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.8065999999999999e-05 2024-02-09T17:49:24.067 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.067 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.067 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:49:24.068 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.640949+0000", 2024-02-09T17:49:24.068 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.6490000000000003e-06 2024-02-09T17:49:24.068 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.068 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.068 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:49:24.068 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.641011+0000", 2024-02-09T17:49:24.068 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.2223000000000004e-05 2024-02-09T17:49:24.068 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.068 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.069 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:49:24.069 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:23.641027+0000", 2024-02-09T17:49:24.069 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5695000000000001e-05 2024-02-09T17:49:24.069 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.069 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.069 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:49:24.069 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.528383+0000", 2024-02-09T17:49:24.069 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.88735621399999998 2024-02-09T17:49:24.069 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.069 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.069 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:49:24.069 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:39:24.528408+0000", 2024-02-09T17:49:24.069 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.4386e-05 2024-02-09T17:49:24.069 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:49:24.070 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:49:24.070 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:49:24.070 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.070 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.070 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25528.0:170463 5.6 5.b907c336 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e521)", 2024-02-09T17:49:24.070 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:43:24.156515+0000", 2024-02-09T17:49:24.070 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 359.06399213600002, 2024-02-09T17:49:24.070 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0027261409999999, 2024-02-09T17:49:24.070 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:49:24.070 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "reached pg", 2024-02-09T17:49:24.070 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:49:24.070 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25528", 2024-02-09T17:49:24.070 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:49:24.070 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 170463 2024-02-09T17:49:24.071 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.071 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:49:24.071 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.071 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:49:24.071 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:43:24.156515+0000", 2024-02-09T17:49:24.071 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:49:24.071 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.071 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.071 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:49:24.071 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:43:24.156515+0000", 2024-02-09T17:49:24.071 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:49:24.071 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.071 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.071 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:49:24.072 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:43:24.156517+0000", 2024-02-09T17:49:24.072 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.9099999999999999e-06 2024-02-09T17:49:24.072 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.072 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.072 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:49:24.072 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:43:24.156522+0000", 2024-02-09T17:49:24.072 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2340000000000002e-06 2024-02-09T17:49:24.072 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.072 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.072 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:49:24.072 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:43:24.156536+0000", 2024-02-09T17:49:24.072 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3706000000000001e-05 2024-02-09T17:49:24.072 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.073 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.073 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:49:24.073 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:43:24.156546+0000", 2024-02-09T17:49:24.073 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0107e-05 2024-02-09T17:49:24.073 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.073 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.073 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:49:24.073 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:43:24.166021+0000", 2024-02-09T17:49:24.073 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0094749090000000001 2024-02-09T17:49:24.073 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.073 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.073 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:49:24.073 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:43:24.166030+0000", 2024-02-09T17:49:24.074 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.2199999999999998e-06 2024-02-09T17:49:24.074 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.074 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.074 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:49:24.074 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:43:25.159219+0000", 2024-02-09T17:49:24.074 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.99318889899999996 2024-02-09T17:49:24.074 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.074 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.074 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:49:24.075 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:43:25.159241+0000", 2024-02-09T17:49:24.075 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2155999999999999e-05 2024-02-09T17:49:24.075 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:49:24.075 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:49:24.075 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:49:24.075 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.075 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.075 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25528.0:170464 5.6 5.6eb87a76 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e521)", 2024-02-09T17:49:24.075 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:43:24.156575+0000", 2024-02-09T17:49:24.075 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 359.063932067, 2024-02-09T17:49:24.075 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.002724175, 2024-02-09T17:49:24.075 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:49:24.075 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "reached pg", 2024-02-09T17:49:24.076 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:49:24.076 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25528", 2024-02-09T17:49:24.076 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:49:24.076 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 170464 2024-02-09T17:49:24.076 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.076 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:49:24.076 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.076 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:49:24.076 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:43:24.156575+0000", 2024-02-09T17:49:24.076 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:49:24.076 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.076 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.076 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:49:24.076 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:43:24.156575+0000", 2024-02-09T17:49:24.077 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:49:24.077 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.077 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.077 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:49:24.077 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:43:24.156575+0000", 2024-02-09T17:49:24.077 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.6699999999999999e-07 2024-02-09T17:49:24.077 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.077 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.077 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:49:24.077 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:43:24.156578+0000", 2024-02-09T17:49:24.077 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.6639999999999998e-06 2024-02-09T17:49:24.077 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.077 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.077 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:49:24.078 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:43:24.156585+0000", 2024-02-09T17:49:24.078 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.6780000000000003e-06 2024-02-09T17:49:24.078 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.078 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.078 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:49:24.078 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:43:24.156590+0000", 2024-02-09T17:49:24.078 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2329999999999997e-06 2024-02-09T17:49:24.078 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.078 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.078 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:49:24.078 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:43:24.166089+0000", 2024-02-09T17:49:24.078 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0094993260000000006 2024-02-09T17:49:24.078 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.078 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.079 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:49:24.079 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:43:24.166097+0000", 2024-02-09T17:49:24.079 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.4209999999999998e-06 2024-02-09T17:49:24.079 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.079 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.079 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:49:24.079 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:43:25.159287+0000", 2024-02-09T17:49:24.079 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.993190727 2024-02-09T17:49:24.079 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.079 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.079 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:49:24.079 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:43:25.159299+0000", 2024-02-09T17:49:24.079 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1659e-05 2024-02-09T17:49:24.079 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:49:24.080 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:49:24.080 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:49:24.080 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.080 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.080 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25528.0:170477 5.6 5.660e676 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e521)", 2024-02-09T17:49:24.080 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:43:24.162581+0000", 2024-02-09T17:49:24.080 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 359.05792579299998, 2024-02-09T17:49:24.080 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.99676081999999999, 2024-02-09T17:49:24.080 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:49:24.080 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "reached pg", 2024-02-09T17:49:24.080 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:49:24.080 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25528", 2024-02-09T17:49:24.080 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:49:24.081 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 170477 2024-02-09T17:49:24.081 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.081 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:49:24.081 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.081 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:49:24.081 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:43:24.162581+0000", 2024-02-09T17:49:24.081 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:49:24.081 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.081 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.081 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:49:24.081 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:43:24.162581+0000", 2024-02-09T17:49:24.081 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:49:24.081 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.081 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.081 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:49:24.082 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:43:24.162582+0000", 2024-02-09T17:49:24.082 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4440000000000001e-06 2024-02-09T17:49:24.082 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.082 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.082 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:49:24.082 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:43:24.162587+0000", 2024-02-09T17:49:24.082 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.9300000000000002e-06 2024-02-09T17:49:24.082 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.082 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.082 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:49:24.082 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:43:24.162599+0000", 2024-02-09T17:49:24.082 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.152e-05 2024-02-09T17:49:24.082 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.082 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.083 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:49:24.083 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:43:24.162608+0000", 2024-02-09T17:49:24.083 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.6330000000000002e-06 2024-02-09T17:49:24.083 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.083 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.083 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:49:24.083 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:43:24.166144+0000", 2024-02-09T17:49:24.083 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.003536235 2024-02-09T17:49:24.083 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.083 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.083 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:49:24.083 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:43:24.166149+0000", 2024-02-09T17:49:24.083 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3700000000000003e-06 2024-02-09T17:49:24.083 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.083 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.084 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:49:24.084 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:43:25.159330+0000", 2024-02-09T17:49:24.084 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.99318117900000003 2024-02-09T17:49:24.084 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.084 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.084 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:49:24.084 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:43:25.159342+0000", 2024-02-09T17:49:24.084 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1508999999999999e-05 2024-02-09T17:49:24.084 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:49:24.084 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:49:24.084 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:49:24.084 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.084 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.084 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25528.0:220356 5.4 e548/535 5:2ee846eb:::benchmark_data_smithi179_115199_object91450:head v 548'15900, mlcod=548'15898)", 2024-02-09T17:49:24.085 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:44:30.402660+0000", 2024-02-09T17:49:24.085 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 292.81784672999999, 2024-02-09T17:49:24.085 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.085553367000000005, 2024-02-09T17:49:24.085 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:49:24.085 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:49:24.085 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:49:24.085 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.085 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:49:24.085 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:30.402660+0000", 2024-02-09T17:49:24.085 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:49:24.085 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.085 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.085 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:49:24.085 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:30.402660+0000", 2024-02-09T17:49:24.086 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:49:24.086 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.086 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.086 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:49:24.086 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:30.402661+0000", 2024-02-09T17:49:24.086 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2330000000000001e-06 2024-02-09T17:49:24.086 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.086 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.086 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:49:24.086 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:30.402669+0000", 2024-02-09T17:49:24.086 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.9160000000000001e-06 2024-02-09T17:49:24.086 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.086 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.086 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:49:24.087 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:30.473503+0000", 2024-02-09T17:49:24.087 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.070833658999999993 2024-02-09T17:49:24.087 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.087 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.087 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:49:24.087 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:30.473518+0000", 2024-02-09T17:49:24.087 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4946999999999999e-05 2024-02-09T17:49:24.087 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.087 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.087 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:49:24.087 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:30.473575+0000", 2024-02-09T17:49:24.087 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7120000000000002e-05 2024-02-09T17:49:24.087 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.087 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.087 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:49:24.088 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:30.473599+0000", 2024-02-09T17:49:24.088 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.3781000000000002e-05 2024-02-09T17:49:24.088 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.088 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.088 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:49:24.088 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:30.488185+0000", 2024-02-09T17:49:24.088 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.014586323 2024-02-09T17:49:24.088 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.088 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.088 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:49:24.088 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:30.488213+0000", 2024-02-09T17:49:24.088 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.8388e-05 2024-02-09T17:49:24.088 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:49:24.088 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:49:24.088 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:49:24.089 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.089 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.089 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25837.0:97721 6.2 e602/576 6:4259e16e:::benchmark_data_smithi179_126206_object97720:head v 602'6331, mlcod=602'6328)", 2024-02-09T17:49:24.089 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:46:48.615722+0000", 2024-02-09T17:49:24.089 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 154.604784357, 2024-02-09T17:49:24.089 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.215957661, 2024-02-09T17:49:24.089 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:49:24.089 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:49:24.089 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:49:24.089 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.089 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:49:24.089 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.615722+0000", 2024-02-09T17:49:24.089 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:49:24.090 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.090 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.090 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:49:24.090 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.615722+0000", 2024-02-09T17:49:24.090 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:49:24.090 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.090 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.090 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:49:24.090 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.615723+0000", 2024-02-09T17:49:24.090 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.5099999999999999e-07 2024-02-09T17:49:24.090 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.090 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.090 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:49:24.090 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.615752+0000", 2024-02-09T17:49:24.091 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.8892999999999999e-05 2024-02-09T17:49:24.091 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.091 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.091 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:49:24.091 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.615763+0000", 2024-02-09T17:49:24.091 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0886e-05 2024-02-09T17:49:24.091 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.091 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.091 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:49:24.091 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.615771+0000", 2024-02-09T17:49:24.091 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.8849999999999999e-06 2024-02-09T17:49:24.091 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.091 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.091 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:49:24.091 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.616139+0000", 2024-02-09T17:49:24.092 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00036847600000000001 2024-02-09T17:49:24.092 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.092 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.092 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:49:24.092 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.616160+0000", 2024-02-09T17:49:24.092 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0814000000000001e-05 2024-02-09T17:49:24.092 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.092 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.092 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:49:24.092 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.831618+0000", 2024-02-09T17:49:24.092 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.21545820600000001 2024-02-09T17:49:24.092 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.092 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.093 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:49:24.093 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.831680+0000", 2024-02-09T17:49:24.093 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.1749999999999997e-05 2024-02-09T17:49:24.093 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:49:24.093 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:49:24.093 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:49:24.093 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.093 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.093 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25837.0:97717 6.0 e602/576 6:0b4d0d71:::benchmark_data_smithi179_126206_object97716:head v 602'6088, mlcod=602'6086)", 2024-02-09T17:49:24.093 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:46:48.619308+0000", 2024-02-09T17:49:24.093 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 154.60119896, 2024-02-09T17:49:24.093 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.212388257, 2024-02-09T17:49:24.093 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:49:24.093 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:49:24.094 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:49:24.094 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.094 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:49:24.094 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.619308+0000", 2024-02-09T17:49:24.094 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:49:24.094 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.094 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.094 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:49:24.094 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.619308+0000", 2024-02-09T17:49:24.094 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:49:24.094 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.094 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.094 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:49:24.094 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.619309+0000", 2024-02-09T17:49:24.095 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.1800000000000005e-07 2024-02-09T17:49:24.095 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.095 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.095 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:49:24.095 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.619334+0000", 2024-02-09T17:49:24.095 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.5554e-05 2024-02-09T17:49:24.095 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.095 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.095 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:49:24.095 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.619346+0000", 2024-02-09T17:49:24.095 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1324e-05 2024-02-09T17:49:24.095 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.095 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.095 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:49:24.095 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.619353+0000", 2024-02-09T17:49:24.096 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.6839999999999996e-06 2024-02-09T17:49:24.096 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.096 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.096 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:49:24.096 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.619436+0000", 2024-02-09T17:49:24.096 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.2516999999999996e-05 2024-02-09T17:49:24.096 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.096 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.096 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:49:24.096 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.619455+0000", 2024-02-09T17:49:24.096 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.9636999999999999e-05 2024-02-09T17:49:24.096 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.096 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.096 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:49:24.097 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.831620+0000", 2024-02-09T17:49:24.097 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.21216502100000001 2024-02-09T17:49:24.097 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.097 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.097 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:49:24.097 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.831696+0000", 2024-02-09T17:49:24.097 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.5702000000000001e-05 2024-02-09T17:49:24.097 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:49:24.097 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:49:24.097 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:49:24.097 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.097 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.097 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25837.0:97724 6.0 e602/576 6:0fe71232:::benchmark_data_smithi179_126206_object97723:head v 602'6089, mlcod=602'6086)", 2024-02-09T17:49:24.098 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:46:48.622345+0000", 2024-02-09T17:49:24.098 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 154.59816167299999, 2024-02-09T17:49:24.098 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.22331884900000001, 2024-02-09T17:49:24.098 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:49:24.098 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:49:24.098 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:49:24.098 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.098 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:49:24.098 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.622345+0000", 2024-02-09T17:49:24.098 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:49:24.098 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.098 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.098 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:49:24.098 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.622345+0000", 2024-02-09T17:49:24.098 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:49:24.099 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.099 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.099 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:49:24.099 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.622346+0000", 2024-02-09T17:49:24.099 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.4e-07 2024-02-09T17:49:24.099 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.099 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.099 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:49:24.099 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.622371+0000", 2024-02-09T17:49:24.099 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.4890000000000001e-05 2024-02-09T17:49:24.099 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.099 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.099 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:49:24.099 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.831587+0000", 2024-02-09T17:49:24.099 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20921569600000001 2024-02-09T17:49:24.099 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.100 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.100 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:49:24.100 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.831620+0000", 2024-02-09T17:49:24.100 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.3118999999999999e-05 2024-02-09T17:49:24.100 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.100 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.100 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:49:24.100 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.831745+0000", 2024-02-09T17:49:24.100 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012559800000000001 2024-02-09T17:49:24.100 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.100 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.100 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:49:24.100 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.831771+0000", 2024-02-09T17:49:24.100 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.565e-05 2024-02-09T17:49:24.101 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.101 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.101 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:49:24.101 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.845604+0000", 2024-02-09T17:49:24.101 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.013832824000000001 2024-02-09T17:49:24.101 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.101 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.101 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:49:24.101 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.845664+0000", 2024-02-09T17:49:24.101 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.0232000000000002e-05 2024-02-09T17:49:24.101 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:49:24.101 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:49:24.101 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:49:24.102 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.102 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.102 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25837.0:97734 6.e e602/576 6:73d56b68:::benchmark_data_smithi179_126206_object97733:head v 602'5983, mlcod=602'5981)", 2024-02-09T17:49:24.102 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:46:48.628558+0000", 2024-02-09T17:49:24.102 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 154.591948608, 2024-02-09T17:49:24.102 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.21067872100000001, 2024-02-09T17:49:24.102 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:49:24.102 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:49:24.102 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:49:24.102 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.102 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:49:24.102 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.628558+0000", 2024-02-09T17:49:24.102 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:49:24.103 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.103 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.103 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:49:24.103 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.628558+0000", 2024-02-09T17:49:24.103 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:49:24.103 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.103 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.103 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:49:24.103 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.628559+0000", 2024-02-09T17:49:24.103 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1829999999999999e-06 2024-02-09T17:49:24.103 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.103 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.103 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:49:24.103 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.628613+0000", 2024-02-09T17:49:24.103 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3915000000000001e-05 2024-02-09T17:49:24.104 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.104 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.104 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:49:24.104 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.831591+0000", 2024-02-09T17:49:24.104 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20297726999999999 2024-02-09T17:49:24.104 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.104 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.104 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:49:24.104 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.831620+0000", 2024-02-09T17:49:24.104 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.9526000000000002e-05 2024-02-09T17:49:24.104 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.104 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.104 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:49:24.104 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.831691+0000", 2024-02-09T17:49:24.104 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.0726e-05 2024-02-09T17:49:24.105 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.105 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.105 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:49:24.105 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.831720+0000", 2024-02-09T17:49:24.105 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.8809000000000001e-05 2024-02-09T17:49:24.105 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.105 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.105 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:49:24.105 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.839194+0000", 2024-02-09T17:49:24.105 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0074739009999999998 2024-02-09T17:49:24.105 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.105 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.105 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:49:24.105 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.839237+0000", 2024-02-09T17:49:24.106 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.3390999999999998e-05 2024-02-09T17:49:24.106 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:49:24.106 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:49:24.106 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:49:24.106 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.106 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.106 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25837.0:103409 6.0 e604/576 6:0a3a4a2e:::benchmark_data_smithi179_126206_object103408:head v 604'6417, mlcod=604'6415)", 2024-02-09T17:49:24.106 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:46:53.626462+0000", 2024-02-09T17:49:24.106 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 149.59404530500001, 2024-02-09T17:49:24.106 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.16262810999999999, 2024-02-09T17:49:24.106 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:49:24.106 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:49:24.106 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:49:24.106 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.107 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:49:24.107 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.626462+0000", 2024-02-09T17:49:24.107 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:49:24.107 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.107 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.107 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:49:24.107 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.626462+0000", 2024-02-09T17:49:24.107 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:49:24.107 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.107 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.107 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:49:24.107 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.626462+0000", 2024-02-09T17:49:24.107 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.8100000000000002e-07 2024-02-09T17:49:24.107 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.108 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.108 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:49:24.108 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.626498+0000", 2024-02-09T17:49:24.108 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.5880999999999997e-05 2024-02-09T17:49:24.108 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.108 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.108 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:49:24.108 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.626509+0000", 2024-02-09T17:49:24.108 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0509000000000001e-05 2024-02-09T17:49:24.108 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.108 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.108 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:49:24.108 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.626516+0000", 2024-02-09T17:49:24.109 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.6149999999999999e-06 2024-02-09T17:49:24.109 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.109 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.109 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:49:24.109 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.626570+0000", 2024-02-09T17:49:24.109 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3569999999999997e-05 2024-02-09T17:49:24.109 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.109 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.109 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:49:24.109 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.626591+0000", 2024-02-09T17:49:24.109 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0669000000000001e-05 2024-02-09T17:49:24.109 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.109 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.109 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:49:24.109 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.789034+0000", 2024-02-09T17:49:24.110 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.162443646 2024-02-09T17:49:24.110 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.110 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.110 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:49:24.110 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.789090+0000", 2024-02-09T17:49:24.110 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.5439000000000002e-05 2024-02-09T17:49:24.110 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:49:24.110 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:49:24.110 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:49:24.110 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.110 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.110 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25837.0:103410 6.0 e604/576 6:0aa7216a:::benchmark_data_smithi179_126206_object103409:head v 604'6418, mlcod=604'6415)", 2024-02-09T17:49:24.110 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:46:53.626536+0000", 2024-02-09T17:49:24.110 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 149.59397038899999, 2024-02-09T17:49:24.110 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.165807126, 2024-02-09T17:49:24.111 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:49:24.111 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:49:24.111 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:49:24.111 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.111 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:49:24.111 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.626536+0000", 2024-02-09T17:49:24.111 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:49:24.111 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.111 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.111 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:49:24.111 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.626536+0000", 2024-02-09T17:49:24.111 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:49:24.111 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.111 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.111 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:49:24.112 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.626537+0000", 2024-02-09T17:49:24.112 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2300000000000002e-07 2024-02-09T17:49:24.112 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.112 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.112 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:49:24.112 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.626559+0000", 2024-02-09T17:49:24.112 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2011999999999999e-05 2024-02-09T17:49:24.112 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.112 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.112 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:49:24.112 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.626566+0000", 2024-02-09T17:49:24.112 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.2239999999999998e-06 2024-02-09T17:49:24.112 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.112 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.113 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:49:24.113 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.626572+0000", 2024-02-09T17:49:24.113 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.4160000000000003e-06 2024-02-09T17:49:24.113 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.113 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.113 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:49:24.113 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.778496+0000", 2024-02-09T17:49:24.113 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.15192410000000001 2024-02-09T17:49:24.113 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.113 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.113 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:49:24.113 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.778510+0000", 2024-02-09T17:49:24.113 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.397e-05 2024-02-09T17:49:24.113 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.113 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.114 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:49:24.114 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.792303+0000", 2024-02-09T17:49:24.114 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.013793144 2024-02-09T17:49:24.114 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.114 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.114 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:49:24.114 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.792344+0000", 2024-02-09T17:49:24.114 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.0837000000000002e-05 2024-02-09T17:49:24.114 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:49:24.114 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:49:24.114 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:49:24.114 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.114 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.114 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25837.0:103418 6.e e604/576 6:7d5b0857:::benchmark_data_smithi179_126206_object103417:head v 604'6355, mlcod=604'6352)", 2024-02-09T17:49:24.114 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:46:53.630861+0000", 2024-02-09T17:49:24.115 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 149.58964566899999, 2024-02-09T17:49:24.115 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.15047986099999999, 2024-02-09T17:49:24.115 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:49:24.115 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:49:24.115 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:49:24.115 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.115 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:49:24.115 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.630861+0000", 2024-02-09T17:49:24.115 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:49:24.115 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.115 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.115 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:49:24.115 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.630861+0000", 2024-02-09T17:49:24.115 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:49:24.115 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.115 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.116 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:49:24.116 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.630862+0000", 2024-02-09T17:49:24.116 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1209999999999999e-06 2024-02-09T17:49:24.116 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.116 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.116 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:49:24.116 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.630906+0000", 2024-02-09T17:49:24.116 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.3720000000000002e-05 2024-02-09T17:49:24.116 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.116 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.116 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:49:24.116 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.775204+0000", 2024-02-09T17:49:24.116 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.14429837100000001 2024-02-09T17:49:24.117 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.117 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.117 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:49:24.117 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.775231+0000", 2024-02-09T17:49:24.117 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.6798999999999999e-05 2024-02-09T17:49:24.117 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.117 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.117 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:49:24.117 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.775414+0000", 2024-02-09T17:49:24.117 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00018272699999999999 2024-02-09T17:49:24.117 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.117 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.117 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:49:24.117 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.775477+0000", 2024-02-09T17:49:24.117 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.3468999999999998e-05 2024-02-09T17:49:24.118 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.118 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.118 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:49:24.118 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.781290+0000", 2024-02-09T17:49:24.118 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0058122520000000004 2024-02-09T17:49:24.118 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.118 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.118 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:49:24.118 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.781341+0000", 2024-02-09T17:49:24.118 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1402e-05 2024-02-09T17:49:24.118 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:49:24.118 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:49:24.118 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:49:24.118 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.118 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.119 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25837.0:122698 6.f e612/576 6:f5828aff:::benchmark_data_smithi179_126206_object122697:head v 612'7469, mlcod=612'7467)", 2024-02-09T17:49:24.119 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:47:12.576978+0000", 2024-02-09T17:49:24.119 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 130.64352905300001, 2024-02-09T17:49:24.119 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.180264602, 2024-02-09T17:49:24.119 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:49:24.119 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:49:24.119 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:49:24.119 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.119 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:49:24.119 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.576978+0000", 2024-02-09T17:49:24.119 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:49:24.119 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.119 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.119 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:49:24.119 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.576978+0000", 2024-02-09T17:49:24.120 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:49:24.120 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.120 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.120 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:49:24.120 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.576979+0000", 2024-02-09T17:49:24.120 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.4799999999999997e-07 2024-02-09T17:49:24.120 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.120 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.120 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:49:24.120 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.577009+0000", 2024-02-09T17:49:24.120 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.0793999999999997e-05 2024-02-09T17:49:24.120 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.120 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.120 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:49:24.121 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.577021+0000", 2024-02-09T17:49:24.121 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1635e-05 2024-02-09T17:49:24.121 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.121 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.121 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:49:24.121 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.577028+0000", 2024-02-09T17:49:24.121 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.2509999999999997e-06 2024-02-09T17:49:24.121 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.121 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.121 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:49:24.121 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.577080+0000", 2024-02-09T17:49:24.121 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1913000000000003e-05 2024-02-09T17:49:24.121 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.121 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.121 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:49:24.122 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.577107+0000", 2024-02-09T17:49:24.122 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.6746000000000001e-05 2024-02-09T17:49:24.122 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.122 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.122 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:49:24.122 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.757182+0000", 2024-02-09T17:49:24.122 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.180074755 2024-02-09T17:49:24.122 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.122 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.122 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:49:24.122 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.757242+0000", 2024-02-09T17:49:24.122 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.0659999999999999e-05 2024-02-09T17:49:24.122 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:49:24.122 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:49:24.122 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:49:24.123 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.123 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.123 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25837.0:122699 6.e e612/576 6:704d9c8f:::benchmark_data_smithi179_126206_object122698:head v 612'7621, mlcod=612'7619)", 2024-02-09T17:49:24.123 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:47:12.577813+0000", 2024-02-09T17:49:24.123 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 130.64269361800001, 2024-02-09T17:49:24.123 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.19765761600000001, 2024-02-09T17:49:24.123 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:49:24.123 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:49:24.123 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:49:24.123 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.123 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:49:24.123 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.577813+0000", 2024-02-09T17:49:24.123 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:49:24.123 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.123 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.124 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:49:24.124 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.577813+0000", 2024-02-09T17:49:24.124 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:49:24.124 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.124 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.124 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:49:24.124 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.577814+0000", 2024-02-09T17:49:24.124 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.0999999999999998e-07 2024-02-09T17:49:24.124 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.124 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.124 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:49:24.124 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.577940+0000", 2024-02-09T17:49:24.124 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000126425 2024-02-09T17:49:24.124 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.125 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.125 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:49:24.125 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.754078+0000", 2024-02-09T17:49:24.125 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.17613795099999999 2024-02-09T17:49:24.125 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.125 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.125 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:49:24.125 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.754088+0000", 2024-02-09T17:49:24.125 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0127e-05 2024-02-09T17:49:24.125 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.125 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.125 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:49:24.125 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.754219+0000", 2024-02-09T17:49:24.125 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013053000000000001 2024-02-09T17:49:24.125 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.126 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.126 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:49:24.126 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.754239+0000", 2024-02-09T17:49:24.126 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0455999999999999e-05 2024-02-09T17:49:24.126 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.126 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.126 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:49:24.126 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.775427+0000", 2024-02-09T17:49:24.126 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.021187620000000001 2024-02-09T17:49:24.126 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.126 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.126 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:49:24.126 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.775471+0000", 2024-02-09T17:49:24.126 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.3797000000000001e-05 2024-02-09T17:49:24.126 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:49:24.127 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:49:24.127 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:49:24.127 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.127 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.127 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25837.0:122697 6.0 e612/576 6:0380994d:::benchmark_data_smithi179_126206_object122696:head v 612'7592, mlcod=612'7589)", 2024-02-09T17:49:24.127 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:47:12.582479+0000", 2024-02-09T17:49:24.127 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 130.63802752699999, 2024-02-09T17:49:24.127 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.19247419199999999, 2024-02-09T17:49:24.127 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:49:24.127 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:49:24.127 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:49:24.127 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.127 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:49:24.127 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.582479+0000", 2024-02-09T17:49:24.127 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:49:24.127 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.128 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.128 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:49:24.128 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.582479+0000", 2024-02-09T17:49:24.128 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:49:24.128 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.128 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.128 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:49:24.128 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.582481+0000", 2024-02-09T17:49:24.128 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.486e-06 2024-02-09T17:49:24.128 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.128 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.128 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:49:24.128 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.582506+0000", 2024-02-09T17:49:24.128 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.4762000000000001e-05 2024-02-09T17:49:24.129 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.129 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.129 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:49:24.129 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.752968+0000", 2024-02-09T17:49:24.129 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.170462893 2024-02-09T17:49:24.129 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.129 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.129 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:49:24.129 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.752983+0000", 2024-02-09T17:49:24.129 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4499e-05 2024-02-09T17:49:24.129 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.129 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.129 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:49:24.129 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.754864+0000", 2024-02-09T17:49:24.130 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0018807159999999999 2024-02-09T17:49:24.130 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.130 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.130 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:49:24.130 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.754884+0000", 2024-02-09T17:49:24.130 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0460000000000001e-05 2024-02-09T17:49:24.130 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.130 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.130 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:49:24.130 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.774886+0000", 2024-02-09T17:49:24.130 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.020001662 2024-02-09T17:49:24.130 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.130 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.130 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:49:24.131 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.774953+0000", 2024-02-09T17:49:24.131 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.7713999999999996e-05 2024-02-09T17:49:24.131 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:49:24.131 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:49:24.131 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:49:24.131 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.131 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.131 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25837.0:202636 6.2 e638/576 6:4eab2b03:::benchmark_data_smithi179_126206_object72255:head v 638'13061, mlcod=638'13058)", 2024-02-09T17:49:24.131 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:48:47.501898+0000", 2024-02-09T17:49:24.131 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 35.718609289, 2024-02-09T17:49:24.131 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.095471128000000002, 2024-02-09T17:49:24.131 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:49:24.131 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:49:24.131 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:49:24.131 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.132 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:49:24.132 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:47.501898+0000", 2024-02-09T17:49:24.132 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:49:24.132 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.132 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.132 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:49:24.132 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:47.501898+0000", 2024-02-09T17:49:24.132 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:49:24.132 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.132 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.132 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:49:24.132 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:47.501898+0000", 2024-02-09T17:49:24.132 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.7599999999999996e-07 2024-02-09T17:49:24.132 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.133 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.133 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:49:24.133 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:47.501906+0000", 2024-02-09T17:49:24.133 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.3690000000000001e-06 2024-02-09T17:49:24.133 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.133 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.133 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:49:24.133 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:47.501917+0000", 2024-02-09T17:49:24.133 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.11e-05 2024-02-09T17:49:24.133 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.133 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.133 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:49:24.133 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:47.501924+0000", 2024-02-09T17:49:24.133 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.8009999999999998e-06 2024-02-09T17:49:24.134 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.134 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.134 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:49:24.134 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:47.507176+0000", 2024-02-09T17:49:24.134 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0052524 2024-02-09T17:49:24.134 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.134 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.134 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:49:24.134 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:47.507197+0000", 2024-02-09T17:49:24.134 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0828000000000001e-05 2024-02-09T17:49:24.134 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.134 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.134 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:49:24.134 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:47.597328+0000", 2024-02-09T17:49:24.134 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.090130753999999993 2024-02-09T17:49:24.135 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.135 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.135 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:49:24.135 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:47.597369+0000", 2024-02-09T17:49:24.135 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.1100000000000003e-05 2024-02-09T17:49:24.135 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:49:24.135 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:49:24.135 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:49:24.135 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.135 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.135 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25837.0:202640 6.2 e638/576 6:4f333034:::benchmark_data_smithi179_126206_object72259:head v 638'13062, mlcod=638'13058)", 2024-02-09T17:49:24.135 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:48:47.511611+0000", 2024-02-09T17:49:24.135 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 35.708895599999998, 2024-02-09T17:49:24.135 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.085791754999999997, 2024-02-09T17:49:24.135 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:49:24.136 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:49:24.136 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:49:24.136 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.136 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:49:24.136 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:47.511611+0000", 2024-02-09T17:49:24.136 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:49:24.136 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.136 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.136 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:49:24.136 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:47.511611+0000", 2024-02-09T17:49:24.136 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:49:24.136 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.136 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.136 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:49:24.137 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:47.511612+0000", 2024-02-09T17:49:24.137 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.7499999999999998e-07 2024-02-09T17:49:24.137 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.137 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.137 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:49:24.137 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:47.511619+0000", 2024-02-09T17:49:24.137 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.9149999999999999e-06 2024-02-09T17:49:24.137 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.137 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.137 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:49:24.137 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:47.511630+0000", 2024-02-09T17:49:24.137 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.066e-05 2024-02-09T17:49:24.137 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.137 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.137 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:49:24.138 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:47.511637+0000", 2024-02-09T17:49:24.138 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.2649999999999999e-06 2024-02-09T17:49:24.138 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.138 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.138 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:49:24.138 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:47.515915+0000", 2024-02-09T17:49:24.138 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.004278268 2024-02-09T17:49:24.138 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.138 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.138 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:49:24.138 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:47.515936+0000", 2024-02-09T17:49:24.138 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.1183999999999999e-05 2024-02-09T17:49:24.138 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.138 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.138 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:49:24.139 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:47.597374+0000", 2024-02-09T17:49:24.139 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.081437417999999998 2024-02-09T17:49:24.139 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.139 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.139 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:49:24.139 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:47.597403+0000", 2024-02-09T17:49:24.139 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.9070000000000001e-05 2024-02-09T17:49:24.139 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:49:24.139 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:49:24.139 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:49:24.139 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.139 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.139 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25837.0:228026 6.f e649/576 6:faf15fdb:::benchmark_data_smithi179_126206_object97645:head v 649'13897, mlcod=649'13895)", 2024-02-09T17:49:24.139 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:49:18.161195+0000", 2024-02-09T17:49:24.140 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 5.059311568, 2024-02-09T17:49:24.140 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.039139561000000003, 2024-02-09T17:49:24.140 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:49:24.140 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:49:24.140 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:49:24.140 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.140 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:49:24.140 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:18.161195+0000", 2024-02-09T17:49:24.140 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:49:24.140 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.140 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.140 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:49:24.140 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:18.161195+0000", 2024-02-09T17:49:24.140 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:49:24.141 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.141 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.141 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:49:24.141 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:18.161196+0000", 2024-02-09T17:49:24.141 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.6300000000000004e-07 2024-02-09T17:49:24.141 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.141 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.141 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:49:24.141 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:18.161202+0000", 2024-02-09T17:49:24.141 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.8309999999999997e-06 2024-02-09T17:49:24.141 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.141 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.141 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:49:24.141 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:18.161211+0000", 2024-02-09T17:49:24.141 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.4210000000000001e-06 2024-02-09T17:49:24.142 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.142 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.142 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:49:24.142 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:18.161218+0000", 2024-02-09T17:49:24.142 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.0709999999999997e-06 2024-02-09T17:49:24.142 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.142 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.142 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:49:24.142 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:18.161269+0000", 2024-02-09T17:49:24.142 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1091999999999998e-05 2024-02-09T17:49:24.142 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.142 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.142 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:49:24.142 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:18.161293+0000", 2024-02-09T17:49:24.143 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.3419999999999999e-05 2024-02-09T17:49:24.143 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.143 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.143 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:49:24.143 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:18.200291+0000", 2024-02-09T17:49:24.143 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.038998186999999997 2024-02-09T17:49:24.143 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.143 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.143 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:49:24.143 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:18.200335+0000", 2024-02-09T17:49:24.143 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.3776000000000002e-05 2024-02-09T17:49:24.143 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:49:24.143 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:49:24.143 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:49:24.143 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.144 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.144 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25837.0:228032 6.f e649/576 6:f27c6014:::benchmark_data_smithi179_126206_object97651:head v 649'13898, mlcod=649'13895)", 2024-02-09T17:49:24.144 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:49:18.161238+0000", 2024-02-09T17:49:24.144 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 5.0592691250000001, 2024-02-09T17:49:24.144 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.039125462999999999, 2024-02-09T17:49:24.144 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:49:24.144 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:49:24.144 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:49:24.144 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.144 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:49:24.144 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:18.161238+0000", 2024-02-09T17:49:24.144 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:49:24.144 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.144 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.145 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:49:24.145 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:18.161238+0000", 2024-02-09T17:49:24.145 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:49:24.145 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.145 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.145 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:49:24.145 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:18.161238+0000", 2024-02-09T17:49:24.145 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.6100000000000002e-07 2024-02-09T17:49:24.145 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.145 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.145 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:49:24.145 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:18.161242+0000", 2024-02-09T17:49:24.145 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.6789999999999998e-06 2024-02-09T17:49:24.145 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.146 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.146 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:49:24.146 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:18.161246+0000", 2024-02-09T17:49:24.146 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.5600000000000004e-06 2024-02-09T17:49:24.146 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.146 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.146 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:49:24.146 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:18.161252+0000", 2024-02-09T17:49:24.146 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.5589999999999996e-06 2024-02-09T17:49:24.146 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.146 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.146 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:49:24.146 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:18.166151+0000", 2024-02-09T17:49:24.146 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0048990609999999997 2024-02-09T17:49:24.146 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.147 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.147 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:49:24.147 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:18.166171+0000", 2024-02-09T17:49:24.147 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.9678e-05 2024-02-09T17:49:24.147 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.147 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.147 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:49:24.147 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:18.200338+0000", 2024-02-09T17:49:24.147 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.034167142999999997 2024-02-09T17:49:24.147 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:49:24.147 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:49:24.147 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:49:24.147 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:18.200363+0000", 2024-02-09T17:49:24.147 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.5422000000000002e-05 2024-02-09T17:49:24.148 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:49:24.148 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:49:24.148 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:49:24.148 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:49:24.148 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:49:24.148 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:49:24.149 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.2 to 1 (10000) 2024-02-09T17:49:24.149 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is unset 2024-02-09T17:49:24.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:24.150 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:24.150 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:49:24.150 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:49:24.248 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:24 smithi099 ceph-mon[39420]: pgmap v1628: 47 pgs: 47 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 820 op/s 2024-02-09T17:49:24.248 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:24 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2603393749' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:49:24.249 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:24 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:49:24.249 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:24 smithi099 ceph-mon[39420]: 1.16 scrub starts 2024-02-09T17:49:24.249 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:24 smithi099 ceph-mon[39420]: 1.16 scrub ok 2024-02-09T17:49:24.249 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:24 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2983844492' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-09T17:49:24.251 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:49:24.293 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:24 smithi162 ceph-mon[44304]: pgmap v1628: 47 pgs: 47 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 820 op/s 2024-02-09T17:49:24.293 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:24 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2603393749' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:49:24.293 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:24 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:49:24.293 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:24 smithi162 ceph-mon[44304]: 1.16 scrub starts 2024-02-09T17:49:24.293 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:24 smithi162 ceph-mon[44304]: 1.16 scrub ok 2024-02-09T17:49:24.293 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:24 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2983844492' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-09T17:49:24.302 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:24.342 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:49:24.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:24.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:49:24.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:24 smithi179 ceph-mon[43975]: pgmap v1628: 47 pgs: 47 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 820 op/s 2024-02-09T17:49:24.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:24 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2603393749' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:49:24.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:24 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:49:24.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:24 smithi179 ceph-mon[43975]: 1.16 scrub starts 2024-02-09T17:49:24.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:24 smithi179 ceph-mon[43975]: 1.16 scrub ok 2024-02-09T17:49:24.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:24 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2983844492' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-09T17:49:24.453 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:49:24.490 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:24.513 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:49:24.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:24.517 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:49:24.618 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:49:24.679 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:24.721 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:49:24.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:24.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:49:24.831 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:49:24.884 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:24.926 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:49:24.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:24.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:49:25.043 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:49:25.071 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:25.092 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:49:25.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:25.098 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:49:25.109 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-09T17:49:25.199 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:49:25.225 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:25.251 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:49:25.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:25.258 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:49:25.359 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:49:25.359 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:49:25.396 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:25 smithi162 ceph-mon[44304]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:49:25.397 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:25 smithi162 ceph-mon[44304]: Cluster is now healthy 2024-02-09T17:49:25.397 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:25 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:49:25.397 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:25 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2983844492' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-09T17:49:25.397 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:25 smithi162 ceph-mon[44304]: osdmap e650: 12 total, 12 up, 12 in 2024-02-09T17:49:25.397 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:25 smithi162 ceph-mon[44304]: 1.3 scrub starts 2024-02-09T17:49:25.397 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:25 smithi162 ceph-mon[44304]: 1.3 scrub ok 2024-02-09T17:49:25.398 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:25.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:25 smithi179 ceph-mon[43975]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:49:25.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:25 smithi179 ceph-mon[43975]: Cluster is now healthy 2024-02-09T17:49:25.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:25 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:49:25.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:25 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2983844492' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-09T17:49:25.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:25 smithi179 ceph-mon[43975]: osdmap e650: 12 total, 12 up, 12 in 2024-02-09T17:49:25.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:25 smithi179 ceph-mon[43975]: 1.3 scrub starts 2024-02-09T17:49:25.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:25 smithi179 ceph-mon[43975]: 1.3 scrub ok 2024-02-09T17:49:25.439 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:49:25.441 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:25.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:49:25.481 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:25 smithi099 ceph-mon[39420]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:49:25.481 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:25 smithi099 ceph-mon[39420]: Cluster is now healthy 2024-02-09T17:49:25.481 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:25 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:49:25.481 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:25 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2983844492' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-09T17:49:25.481 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:25 smithi099 ceph-mon[39420]: osdmap e650: 12 total, 12 up, 12 in 2024-02-09T17:49:25.481 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:25 smithi099 ceph-mon[39420]: 1.3 scrub starts 2024-02-09T17:49:25.481 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:25 smithi099 ceph-mon[39420]: 1.3 scrub ok 2024-02-09T17:49:25.542 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:49:25.588 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:25.632 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:49:25.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:25.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:49:25.738 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:49:25.790 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:25.870 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:49:25.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:25.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:49:25.908 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-02-09T17:49:25.908 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph pg dump --format=json 2024-02-09T17:49:25.976 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:49:26.036 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:26.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:26 smithi179 ceph-mon[43975]: 1.17 scrub starts 2024-02-09T17:49:26.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:26 smithi179 ceph-mon[43975]: 1.17 scrub ok 2024-02-09T17:49:26.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:26 smithi179 ceph-mon[43975]: pgmap v1630: 47 pgs: 47 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 880 op/s 2024-02-09T17:49:26.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:26 smithi162 ceph-mon[44304]: 1.17 scrub starts 2024-02-09T17:49:26.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:26 smithi162 ceph-mon[44304]: 1.17 scrub ok 2024-02-09T17:49:26.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:26 smithi162 ceph-mon[44304]: pgmap v1630: 47 pgs: 47 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 880 op/s 2024-02-09T17:49:26.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:26 smithi099 ceph-mon[39420]: 1.17 scrub starts 2024-02-09T17:49:26.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:26 smithi099 ceph-mon[39420]: 1.17 scrub ok 2024-02-09T17:49:26.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:26 smithi099 ceph-mon[39420]: pgmap v1630: 47 pgs: 47 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 880 op/s 2024-02-09T17:49:27.061 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:49:27.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:27.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:49:27.171 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:49:27.223 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:27.265 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:49:27.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:27.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:49:27.370 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:49:27.418 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:27 smithi162 ceph-mon[44304]: 6.c scrub starts 2024-02-09T17:49:27.420 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:27.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:27 smithi179 ceph-mon[43975]: 6.c scrub starts 2024-02-09T17:49:27.443 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:49:27.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:27.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:49:27.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:27 smithi099 ceph-mon[39420]: 6.c scrub starts 2024-02-09T17:49:27.547 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:49:27.573 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:27.592 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:49:27.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:27.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:49:27.696 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:49:27.762 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:28.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:28 smithi179 ceph-mon[43975]: 6.c scrub ok 2024-02-09T17:49:28.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:28 smithi179 ceph-mon[43975]: pgmap v1631: 47 pgs: 47 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 839 op/s 2024-02-09T17:49:28.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:28 smithi162 ceph-mon[44304]: 6.c scrub ok 2024-02-09T17:49:28.431 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:28 smithi162 ceph-mon[44304]: pgmap v1631: 47 pgs: 47 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 839 op/s 2024-02-09T17:49:28.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:28 smithi099 ceph-mon[39420]: 6.c scrub ok 2024-02-09T17:49:28.498 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:28 smithi099 ceph-mon[39420]: pgmap v1631: 47 pgs: 47 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 839 op/s 2024-02-09T17:49:29.137 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:49:29.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:29.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:49:29.183 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:49:29.195 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:49:29.246 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:49:29.296 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:29.328 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:49:29.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:29.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:49:29.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:29 smithi179 ceph-mon[43975]: 6.1 deep-scrub starts 2024-02-09T17:49:29.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:29 smithi162 ceph-mon[44304]: 6.1 deep-scrub starts 2024-02-09T17:49:29.437 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:49:29.472 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:29 smithi099 ceph-mon[39420]: 6.1 deep-scrub starts 2024-02-09T17:49:29.485 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:29.515 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:49:29.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:29.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:49:29.622 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:49:29.653 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:29.710 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:49:29.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:29.714 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:49:29.815 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:49:29.850 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:29.892 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:49:29.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:29.901 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:49:30.002 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:49:30.056 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:30.096 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:49:30.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:30.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:49:30.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:30 smithi179 ceph-mon[43975]: pgmap v1632: 47 pgs: 47 active+clean; 1.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 825 op/s 2024-02-09T17:49:30.203 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:49:30.249 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:30.390 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:49:30.395 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:30 smithi099 ceph-mon[39420]: pgmap v1632: 47 pgs: 47 active+clean; 1.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 825 op/s 2024-02-09T17:49:30.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:30.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:49:30.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:30 smithi162 ceph-mon[44304]: pgmap v1632: 47 pgs: 47 active+clean; 1.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 825 op/s 2024-02-09T17:49:30.497 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:49:30.538 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:30.565 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:49:30.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:30.571 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:49:30.593 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set noscrub 2024-02-09T17:49:30.671 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:49:30.705 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:30.737 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:49:30.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:30.741 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:49:30.842 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:49:30.876 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:30.896 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:49:30.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:30.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:49:31.000 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:49:31.038 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:31.061 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:49:31.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:31.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:49:31.139 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:49:31.169 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:49:31.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:31 smithi162 ceph-mon[44304]: 6.0 scrub starts 2024-02-09T17:49:31.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:31 smithi162 ceph-mon[44304]: 1.14 scrub starts 2024-02-09T17:49:31.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:31 smithi162 ceph-mon[44304]: 1.14 scrub ok 2024-02-09T17:49:31.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:31 smithi162 ceph-mon[44304]: 1.11 scrub starts 2024-02-09T17:49:31.179 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:49:31.212 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:31.213 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:49:31.274 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:49:31.308 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:49:31.356 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:49:31.370 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:49:31.405 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:49:31.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:31 smithi179 ceph-mon[43975]: 6.0 scrub starts 2024-02-09T17:49:31.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:31 smithi179 ceph-mon[43975]: 1.14 scrub starts 2024-02-09T17:49:31.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:31 smithi179 ceph-mon[43975]: 1.14 scrub ok 2024-02-09T17:49:31.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:31 smithi179 ceph-mon[43975]: 1.11 scrub starts 2024-02-09T17:49:31.437 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:49:31.468 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:49:31.478 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:31 smithi099 ceph-mon[39420]: 6.0 scrub starts 2024-02-09T17:49:31.478 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:31 smithi099 ceph-mon[39420]: 1.14 scrub starts 2024-02-09T17:49:31.478 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:31 smithi099 ceph-mon[39420]: 1.14 scrub ok 2024-02-09T17:49:31.478 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:31 smithi099 ceph-mon[39420]: 1.11 scrub starts 2024-02-09T17:49:31.494 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:49:31.536 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:49:31.555 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:49:31.555 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:49:31.555 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:49:31.555 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:49:31.555 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:49:31.555 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:49:31.555 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:49:31.555 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:49:31.556 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:49:31.556 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:49:31.556 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:49:31.556 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:49:32.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:32 smithi099 ceph-mon[39420]: 6.0 scrub ok 2024-02-09T17:49:32.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:32 smithi099 ceph-mon[39420]: pgmap v1633: 47 pgs: 47 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 816 op/s 2024-02-09T17:49:32.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:32 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3340345568' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:49:32.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:32 smithi099 ceph-mon[39420]: 1.11 scrub ok 2024-02-09T17:49:32.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:32 smithi179 ceph-mon[43975]: 6.0 scrub ok 2024-02-09T17:49:32.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:32 smithi179 ceph-mon[43975]: pgmap v1633: 47 pgs: 47 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 816 op/s 2024-02-09T17:49:32.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:32 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3340345568' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:49:32.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:32 smithi179 ceph-mon[43975]: 1.11 scrub ok 2024-02-09T17:49:32.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:32 smithi162 ceph-mon[44304]: 6.0 scrub ok 2024-02-09T17:49:32.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:32 smithi162 ceph-mon[44304]: pgmap v1633: 47 pgs: 47 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 816 op/s 2024-02-09T17:49:32.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:32 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3340345568' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:49:32.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:32 smithi162 ceph-mon[44304]: 1.11 scrub ok 2024-02-09T17:49:33.805 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:49:33.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:33.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:49:33.912 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:49:33.951 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:33.987 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:49:33.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:33.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:49:34.092 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:49:34.130 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:34.154 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:49:34.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:34.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:49:34.259 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:49:34.305 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:34 smithi162 ceph-mon[44304]: pgmap v1634: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 763 op/s 2024-02-09T17:49:34.307 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:34.340 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:49:34.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:34.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:49:34.414 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:34 smithi099 ceph-mon[39420]: pgmap v1634: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 763 op/s 2024-02-09T17:49:34.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:34 smithi179 ceph-mon[43975]: pgmap v1634: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 763 op/s 2024-02-09T17:49:34.445 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:49:34.465 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:49:34.534 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:34.595 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:49:34.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:34.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:49:34.700 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:49:34.733 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:34.756 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:49:34.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:34.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:49:34.862 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:49:34.913 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:34.954 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:49:34.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:34.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:49:35.062 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:49:35.110 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:35.137 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:49:35.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:35.146 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:49:35.247 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:49:35.312 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:35.494 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:49:35.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:35.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:49:35.607 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:49:35.660 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:35.668 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:49:35.669 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:49:35.814 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:49:35.926 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:49:35.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:35.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:49:36.012 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:49:36.014 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-02-09T17:49:36.041 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:49:36.082 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:36.124 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:49:36.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:36.128 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:49:36.229 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:49:36.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:36 smithi099 ceph-mon[39420]: pgmap v1635: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 733 op/s 2024-02-09T17:49:36.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:36 smithi099 ceph-mon[39420]: 1.6 scrub starts 2024-02-09T17:49:36.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:36 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:49:36.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:36 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:49:36.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:36 smithi099 ceph-mon[39420]: 1.6 scrub ok 2024-02-09T17:49:36.261 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:36.370 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:49:36.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:36.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:49:36.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:36 smithi179 ceph-mon[43975]: pgmap v1635: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 733 op/s 2024-02-09T17:49:36.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:36 smithi179 ceph-mon[43975]: 1.6 scrub starts 2024-02-09T17:49:36.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:36 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:49:36.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:36 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:49:36.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:36 smithi179 ceph-mon[43975]: 1.6 scrub ok 2024-02-09T17:49:36.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:36 smithi162 ceph-mon[44304]: pgmap v1635: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 733 op/s 2024-02-09T17:49:36.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:36 smithi162 ceph-mon[44304]: 1.6 scrub starts 2024-02-09T17:49:36.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:36 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:49:36.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:36 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:49:36.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:36 smithi162 ceph-mon[44304]: 1.6 scrub ok 2024-02-09T17:49:36.475 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:49:36.505 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:37.058 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-09T17:49:37.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:37 smithi179 ceph-mon[43975]: 6.5 deep-scrub starts 2024-02-09T17:49:37.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:37 smithi179 ceph-mon[43975]: from='client.16365 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:49:37.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:37 smithi179 ceph-mon[43975]: osdmap e651: 12 total, 12 up, 12 in 2024-02-09T17:49:37.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:37 smithi179 ceph-mon[43975]: 1.5 scrub starts 2024-02-09T17:49:37.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:37 smithi179 ceph-mon[43975]: 1.5 scrub ok 2024-02-09T17:49:37.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:37 smithi162 ceph-mon[44304]: 6.5 deep-scrub starts 2024-02-09T17:49:37.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:37 smithi162 ceph-mon[44304]: from='client.16365 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:49:37.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:37 smithi162 ceph-mon[44304]: osdmap e651: 12 total, 12 up, 12 in 2024-02-09T17:49:37.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:37 smithi162 ceph-mon[44304]: 1.5 scrub starts 2024-02-09T17:49:37.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:37 smithi162 ceph-mon[44304]: 1.5 scrub ok 2024-02-09T17:49:37.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:37 smithi099 ceph-mon[39420]: 6.5 deep-scrub starts 2024-02-09T17:49:37.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:37 smithi099 ceph-mon[39420]: from='client.16365 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:49:37.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:37 smithi099 ceph-mon[39420]: osdmap e651: 12 total, 12 up, 12 in 2024-02-09T17:49:37.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:37 smithi099 ceph-mon[39420]: 1.5 scrub starts 2024-02-09T17:49:37.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:37 smithi099 ceph-mon[39420]: 1.5 scrub ok 2024-02-09T17:49:38.064 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:49:38.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:38.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:49:38.101 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:49:38.173 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:49:38.223 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:38.261 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:49:38.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:38.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:49:38.365 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:49:38.393 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:38.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:38 smithi179 ceph-mon[43975]: 1.1a scrub starts 2024-02-09T17:49:38.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:38 smithi179 ceph-mon[43975]: 1.1a scrub ok 2024-02-09T17:49:38.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:38 smithi179 ceph-mon[43975]: pgmap v1637: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 1.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 728 op/s 2024-02-09T17:49:38.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:38 smithi179 ceph-mon[43975]: 6.1 deep-scrub ok 2024-02-09T17:49:38.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:38 smithi179 ceph-mon[43975]: 6.f scrub starts 2024-02-09T17:49:38.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:38 smithi179 ceph-mon[43975]: 6.f scrub ok 2024-02-09T17:49:38.423 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:49:38.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:38 smithi162 ceph-mon[44304]: 1.1a scrub starts 2024-02-09T17:49:38.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:38 smithi162 ceph-mon[44304]: 1.1a scrub ok 2024-02-09T17:49:38.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:38 smithi162 ceph-mon[44304]: pgmap v1637: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 1.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 728 op/s 2024-02-09T17:49:38.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:38 smithi162 ceph-mon[44304]: 6.1 deep-scrub ok 2024-02-09T17:49:38.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:38 smithi162 ceph-mon[44304]: 6.f scrub starts 2024-02-09T17:49:38.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:38 smithi162 ceph-mon[44304]: 6.f scrub ok 2024-02-09T17:49:38.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:38.430 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:49:38.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:38 smithi099 ceph-mon[39420]: 1.1a scrub starts 2024-02-09T17:49:38.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:38 smithi099 ceph-mon[39420]: 1.1a scrub ok 2024-02-09T17:49:38.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:38 smithi099 ceph-mon[39420]: pgmap v1637: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 1.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 728 op/s 2024-02-09T17:49:38.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:38 smithi099 ceph-mon[39420]: 6.1 deep-scrub ok 2024-02-09T17:49:38.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:38 smithi099 ceph-mon[39420]: 6.f scrub starts 2024-02-09T17:49:38.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:38 smithi099 ceph-mon[39420]: 6.f scrub ok 2024-02-09T17:49:38.531 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:49:38.569 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:38.715 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:49:38.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:38.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:49:38.821 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:49:38.858 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:38.887 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:49:38.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:38.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:49:38.994 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:49:39.019 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":1635,"stamp":"2024-02-09T17:49:35.008717+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1415774776,"num_objects":21597,"num_object_clones":0,"num_object_copies":64791,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21597,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":239163,"num_write_kb":8344193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":165033,"ondisk_log_size":165033,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":10940164,"kb_used_data":4218020,"kb_used_omap":229,"kb_used_meta":6721434,"kb_avail":1114001660,"statfs":{"total":1151940427776,"available":1140737699840,"internally_reserved":0,"allocated":4319252480,"data_stored":4315819436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235127,"internal_metadata":6882748809},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,3,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-526647296,"num_objects":-8036,"num_object_clones":0,"num_object_copies":-24108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8036,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.952458"},"pg_stats":[{"pgid":"1.1e","version":"63'139","reported_seq":556,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-02-09T17:49:24.068322+0000","last_change":"2024-02-09T17:48:27.182988+0000","last_active":"2024-02-09T17:49:24.068322+0000","last_peered":"2024-02-09T17:49:24.068322+0000","last_clean":"2024-02-09T17:49:24.068322+0000","last_became_active":"2024-02-09T17:43:19.126757+0000","last_became_peered":"2024-02-09T17:43:19.126757+0000","last_unstale":"2024-02-09T17:49:24.068322+0000","last_undegraded":"2024-02-09T17:49:24.068322+0000","last_fullsized":"2024-02-09T17:49:24.068322+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:48:27.182896+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:48:27.182896+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:49:44.889623+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"63'139","reported_seq":580,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-02-09T17:49:24.067926+0000","last_change":"2024-02-09T17:48:24.634219+0000","last_active":"2024-02-09T17:49:24.067926+0000","last_peered":"2024-02-09T17:49:24.067926+0000","last_clean":"2024-02-09T17:49:24.067926+0000","last_became_active":"2024-02-09T17:43:28.594296+0000","last_became_peered":"2024-02-09T17:43:28.594296+0000","last_unstale":"2024-02-09T17:49:24.067926+0000","last_undegraded":"2024-02-09T17:49:24.067926+0000","last_fullsized":"2024-02-09T17:49:24.067926+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:48:24.634092+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:48:24.634092+0000","last_clean_scrub_stamp":"2024-02-09T17:48:24.634092+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:49:40.623938+0000","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"63'139","reported_seq":1100,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-02-09T17:49:24.098698+0000","last_change":"2024-02-09T17:49:17.868395+0000","last_active":"2024-02-09T17:49:24.098698+0000","last_peered":"2024-02-09T17:49:24.098698+0000","last_clean":"2024-02-09T17:49:24.098698+0000","last_became_active":"2024-02-09T17:43:19.103584+0000","last_became_peered":"2024-02-09T17:43:19.103584+0000","last_unstale":"2024-02-09T17:49:24.098698+0000","last_undegraded":"2024-02-09T17:49:24.098698+0000","last_fullsized":"2024-02-09T17:49:24.098698+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:49:17.868254+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:23.912504+0000","last_clean_scrub_stamp":"2024-02-09T17:49:17.868254+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:28.930122+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"63'139","reported_seq":898,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-02-09T17:49:24.098689+0000","last_change":"2024-02-09T17:49:08.965002+0000","last_active":"2024-02-09T17:49:24.098689+0000","last_peered":"2024-02-09T17:49:24.098689+0000","last_clean":"2024-02-09T17:49:24.098689+0000","last_became_active":"2024-02-09T17:43:19.102729+0000","last_became_peered":"2024-02-09T17:43:19.102729+0000","last_unstale":"2024-02-09T17:49:24.098689+0000","last_undegraded":"2024-02-09T17:49:24.098689+0000","last_fullsized":"2024-02-09T17:49:24.098689+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:49:08.964912+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:45:44.988633+0000","last_clean_scrub_stamp":"2024-02-09T17:49:08.964912+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:32.772979+0000","scrub_duration":126,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"63'139","reported_seq":561,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-02-09T17:49:24.069085+0000","last_change":"2024-02-09T17:48:18.772509+0000","last_active":"2024-02-09T17:49:24.069085+0000","last_peered":"2024-02-09T17:49:24.069085+0000","last_clean":"2024-02-09T17:49:24.069085+0000","last_became_active":"2024-02-09T17:43:19.120814+0000","last_became_peered":"2024-02-09T17:43:19.120814+0000","last_unstale":"2024-02-09T17:49:24.069085+0000","last_undegraded":"2024-02-09T17:49:24.069085+0000","last_fullsized":"2024-02-09T17:49:24.069085+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:48:18.772397+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:48:18.772397+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:49:29.521493+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"63'139","reported_seq":1397,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-02-09T17:49:24.075343+0000","last_change":"2024-02-09T17:49:16.723321+0000","last_active":"2024-02-09T17:49:24.075343+0000","last_peered":"2024-02-09T17:49:24.075343+0000","last_clean":"2024-02-09T17:49:24.075343+0000","last_became_active":"2024-02-09T17:43:19.110535+0000","last_became_peered":"2024-02-09T17:43:19.110535+0000","last_unstale":"2024-02-09T17:49:24.075343+0000","last_undegraded":"2024-02-09T17:49:24.075343+0000","last_fullsized":"2024-02-09T17:49:24.075343+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:49:16.723197+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:30.193194+0000","last_clean_scrub_stamp":"2024-02-09T17:49:16.723197+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:21.466285+0000","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"63'139","reported_seq":1236,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-02-09T17:49:24.087652+0000","last_change":"2024-02-09T17:49:12.248620+0000","last_active":"2024-02-09T17:49:24.087652+0000","last_peered":"2024-02-09T17:49:24.087652+0000","last_clean":"2024-02-09T17:49:24.087652+0000","last_became_active":"2024-02-09T17:43:28.585573+0000","last_became_peered":"2024-02-09T17:43:28.585573+0000","last_unstale":"2024-02-09T17:49:24.087652+0000","last_undegraded":"2024-02-09T17:49:24.087652+0000","last_fullsized":"2024-02-09T17:49:24.087652+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:49:12.248513+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:49:12.248513+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:29.288104+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"63'139","reported_seq":690,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-02-09T17:49:24.923380+0000","last_change":"2024-02-09T17:49:24.923424+0000","last_active":"2024-02-09T17:49:24.923380+0000","last_peered":"2024-02-09T17:49:24.923380+0000","last_clean":"2024-02-09T17:49:24.923380+0000","last_became_active":"2024-02-09T17:48:46.184419+0000","last_became_peered":"2024-02-09T17:48:46.184419+0000","last_unstale":"2024-02-09T17:49:24.923380+0000","last_undegraded":"2024-02-09T17:49:24.923380+0000","last_fullsized":"2024-02-09T17:49:24.923380+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:49:24.923286+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:44:50.214686+0000","last_clean_scrub_stamp":"2024-02-09T17:49:24.923286+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:30.714830+0000","scrub_duration":118,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"63'139","reported_seq":563,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-02-09T17:49:24.068020+0000","last_change":"2024-02-09T17:49:23.348000+0000","last_active":"2024-02-09T17:49:24.068020+0000","last_peered":"2024-02-09T17:49:24.068020+0000","last_clean":"2024-02-09T17:49:24.068020+0000","last_became_active":"2024-02-09T17:43:19.094230+0000","last_became_peered":"2024-02-09T17:43:19.094230+0000","last_unstale":"2024-02-09T17:49:24.068020+0000","last_undegraded":"2024-02-09T17:49:24.068020+0000","last_fullsized":"2024-02-09T17:49:24.068020+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:49:23.347891+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:11.970338+0000","last_clean_scrub_stamp":"2024-02-09T17:49:23.347891+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:29.874655+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"63'139","reported_seq":1010,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-02-09T17:49:24.070864+0000","last_change":"2024-02-09T17:48:20.211497+0000","last_active":"2024-02-09T17:49:24.070864+0000","last_peered":"2024-02-09T17:49:24.070864+0000","last_clean":"2024-02-09T17:49:24.070864+0000","last_became_active":"2024-02-09T17:43:19.138398+0000","last_became_peered":"2024-02-09T17:43:19.138398+0000","last_unstale":"2024-02-09T17:49:24.070864+0000","last_undegraded":"2024-02-09T17:49:24.070864+0000","last_fullsized":"2024-02-09T17:49:24.070864+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:48:20.211388+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:48:20.211388+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:49:40.075285+0000","scrub_duration":170,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"63'139","reported_seq":1056,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-02-09T17:49:24.075787+0000","last_change":"2024-02-09T17:48:27.123018+0000","last_active":"2024-02-09T17:49:24.075787+0000","last_peered":"2024-02-09T17:49:24.075787+0000","last_clean":"2024-02-09T17:49:24.075787+0000","last_became_active":"2024-02-09T17:43:28.589307+0000","last_became_peered":"2024-02-09T17:43:28.589307+0000","last_unstale":"2024-02-09T17:49:24.075787+0000","last_undegraded":"2024-02-09T17:49:24.075787+0000","last_fullsized":"2024-02-09T17:49:24.075787+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:48:27.122932+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:38:26.968892+0000","last_clean_scrub_stamp":"2024-02-09T17:48:27.122932+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:49:29.419381+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"650'14971","reported_seq":15278,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-02-09T17:49:32.670020+0000","last_change":"2024-02-09T17:48:31.957735+0000","last_active":"2024-02-09T17:49:32.670020+0000","last_peered":"2024-02-09T17:49:32.670020+0000","last_clean":"2024-02-09T17:49:32.670020+0000","last_became_active":"2024-02-09T17:46:47.018844+0000","last_became_peered":"2024-02-09T17:46:47.018844+0000","last_unstale":"2024-02-09T17:49:32.670020+0000","last_undegraded":"2024-02-09T17:49:32.670020+0000","last_fullsized":"2024-02-09T17:49:32.670020+0000","mapping_epoch":600,"log_start":"595'4900","ondisk_log_start":"595'4900","created":576,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"632'11810","last_scrub_stamp":"2024-02-09T17:48:31.957610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:45:42.586555+0000","last_clean_scrub_stamp":"2024-02-09T17:48:31.957610+0000","objects_scrubbed":4556,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:49:40.127371+0000","scrub_duration":1643,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88670208,"num_objects":1353,"num_object_clones":0,"num_object_copies":4059,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14971,"num_write_kb":522368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"63'139","reported_seq":1659,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-02-09T17:49:24.065552+0000","last_change":"2024-02-09T17:48:32.090480+0000","last_active":"2024-02-09T17:49:24.065552+0000","last_peered":"2024-02-09T17:49:24.065552+0000","last_clean":"2024-02-09T17:49:24.065552+0000","last_became_active":"2024-02-09T17:41:14.521079+0000","last_became_peered":"2024-02-09T17:41:14.521079+0000","last_unstale":"2024-02-09T17:49:24.065552+0000","last_undegraded":"2024-02-09T17:49:24.065552+0000","last_fullsized":"2024-02-09T17:49:24.065552+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:48:32.090348+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:48:32.090348+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:49:45.676733+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"650'14772","reported_seq":15066,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-02-09T17:49:32.677069+0000","last_change":"2024-02-09T17:48:32.124083+0000","last_active":"2024-02-09T17:49:32.677069+0000","last_peered":"2024-02-09T17:49:32.677069+0000","last_clean":"2024-02-09T17:49:32.677069+0000","last_became_active":"2024-02-09T17:45:43.660463+0000","last_became_peered":"2024-02-09T17:45:43.660463+0000","last_unstale":"2024-02-09T17:49:32.677069+0000","last_undegraded":"2024-02-09T17:49:32.677069+0000","last_fullsized":"2024-02-09T17:49:32.677069+0000","mapping_epoch":576,"log_start":"593'4700","ondisk_log_start":"593'4700","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"632'11692","last_scrub_stamp":"2024-02-09T17:48:32.123901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:45:42.586555+0000","last_clean_scrub_stamp":"2024-02-09T17:48:32.123901+0000","objects_scrubbed":4501,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:49:35.566310+0000","scrub_duration":1080,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91095040,"num_objects":1390,"num_object_clones":0,"num_object_copies":4170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14772,"num_write_kb":517184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"63'139","reported_seq":1224,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-02-09T17:49:24.076176+0000","last_change":"2024-02-09T17:49:11.938719+0000","last_active":"2024-02-09T17:49:24.076176+0000","last_peered":"2024-02-09T17:49:24.076176+0000","last_clean":"2024-02-09T17:49:24.076176+0000","last_became_active":"2024-02-09T17:43:39.981054+0000","last_became_peered":"2024-02-09T17:43:39.981054+0000","last_unstale":"2024-02-09T17:49:24.076176+0000","last_undegraded":"2024-02-09T17:49:24.076176+0000","last_fullsized":"2024-02-09T17:49:24.076176+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:49:11.938609+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:49:11.938609+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:21.174789+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"650'15149","reported_seq":15454,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-02-09T17:49:34.920035+0000","last_change":"2024-02-09T17:48:23.008484+0000","last_active":"2024-02-09T17:49:34.920035+0000","last_peered":"2024-02-09T17:49:34.920035+0000","last_clean":"2024-02-09T17:49:34.920035+0000","last_became_active":"2024-02-09T17:45:43.605145+0000","last_became_peered":"2024-02-09T17:45:43.605145+0000","last_unstale":"2024-02-09T17:49:34.920035+0000","last_undegraded":"2024-02-09T17:49:34.920035+0000","last_fullsized":"2024-02-09T17:49:34.920035+0000","mapping_epoch":576,"log_start":"596'5100","ondisk_log_start":"596'5100","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"631'11553","last_scrub_stamp":"2024-02-09T17:48:23.008362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:45:42.586555+0000","last_clean_scrub_stamp":"2024-02-09T17:48:23.008362+0000","objects_scrubbed":4912,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:49:51.953874+0000","scrub_duration":1446,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84213760,"num_objects":1285,"num_object_clones":0,"num_object_copies":3855,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15149,"num_write_kb":525888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"63'139","reported_seq":847,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-02-09T17:49:24.432871+0000","last_change":"2024-02-09T17:49:24.432905+0000","last_active":"2024-02-09T17:49:24.432871+0000","last_peered":"2024-02-09T17:49:24.432871+0000","last_clean":"2024-02-09T17:49:24.432871+0000","last_became_active":"2024-02-09T17:40:13.501711+0000","last_became_peered":"2024-02-09T17:40:13.501711+0000","last_unstale":"2024-02-09T17:49:24.432871+0000","last_undegraded":"2024-02-09T17:49:24.432871+0000","last_fullsized":"2024-02-09T17:49:24.432871+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:49:24.432810+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:46:40.684762+0000","last_clean_scrub_stamp":"2024-02-09T17:49:24.432810+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:34.977155+0000","scrub_duration":195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"650'14951","reported_seq":15248,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-02-09T17:49:32.643682+0000","last_change":"2024-02-09T17:48:29.818271+0000","last_active":"2024-02-09T17:49:32.643682+0000","last_peered":"2024-02-09T17:49:32.643682+0000","last_clean":"2024-02-09T17:49:32.643682+0000","last_became_active":"2024-02-09T17:45:43.610974+0000","last_became_peered":"2024-02-09T17:45:43.610974+0000","last_unstale":"2024-02-09T17:49:32.643682+0000","last_undegraded":"2024-02-09T17:49:32.643682+0000","last_fullsized":"2024-02-09T17:49:32.643682+0000","mapping_epoch":576,"log_start":"594'4900","ondisk_log_start":"594'4900","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"632'11793","last_scrub_stamp":"2024-02-09T17:48:29.818043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:45:42.586555+0000","last_clean_scrub_stamp":"2024-02-09T17:48:29.818043+0000","objects_scrubbed":4599,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:49:52.446276+0000","scrub_duration":1543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92078080,"num_objects":1405,"num_object_clones":0,"num_object_copies":4215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14951,"num_write_kb":523392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"63'139","reported_seq":374,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-02-09T17:49:24.075355+0000","last_change":"2024-02-09T17:49:11.737157+0000","last_active":"2024-02-09T17:49:24.075355+0000","last_peered":"2024-02-09T17:49:24.075355+0000","last_clean":"2024-02-09T17:49:24.075355+0000","last_became_active":"2024-02-09T17:48:46.207004+0000","last_became_peered":"2024-02-09T17:48:46.207004+0000","last_unstale":"2024-02-09T17:49:24.075355+0000","last_undegraded":"2024-02-09T17:49:24.075355+0000","last_fullsized":"2024-02-09T17:49:24.075355+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:49:11.737062+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:49:11.737062+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:13.922917+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.1","version":"650'15428","reported_seq":15754,"reported_epoch":650,"state":"active+clean+scrubbing+deep","last_fresh":"2024-02-09T17:49:32.089782+0000","last_change":"2024-02-09T17:49:27.764483+0000","last_active":"2024-02-09T17:49:32.089782+0000","last_peered":"2024-02-09T17:49:32.089782+0000","last_clean":"2024-02-09T17:49:32.089782+0000","last_became_active":"2024-02-09T17:45:43.606122+0000","last_became_peered":"2024-02-09T17:45:43.606122+0000","last_unstale":"2024-02-09T17:49:32.089782+0000","last_undegraded":"2024-02-09T17:49:32.089782+0000","last_fullsized":"2024-02-09T17:49:32.089782+0000","mapping_epoch":576,"log_start":"596'5400","ondisk_log_start":"596'5400","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"630'11278","last_scrub_stamp":"2024-02-09T17:48:21.540456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:45:42.586555+0000","last_clean_scrub_stamp":"2024-02-09T17:48:21.540456+0000","objects_scrubbed":561,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"deep scrubbing for 4s","scrub_duration":937,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85917696,"num_objects":1311,"num_object_clones":0,"num_object_copies":3933,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14867,"num_write_kb":517696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"63'139","reported_seq":1159,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-02-09T17:49:24.075515+0000","last_change":"2024-02-09T17:48:07.604811+0000","last_active":"2024-02-09T17:49:24.075515+0000","last_peered":"2024-02-09T17:49:24.075515+0000","last_clean":"2024-02-09T17:49:24.075515+0000","last_became_active":"2024-02-09T17:43:14.024894+0000","last_became_peered":"2024-02-09T17:43:14.024894+0000","last_unstale":"2024-02-09T17:49:24.075515+0000","last_undegraded":"2024-02-09T17:49:24.075515+0000","last_fullsized":"2024-02-09T17:49:24.075515+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:48:07.604596+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:11.970338+0000","last_clean_scrub_stamp":"2024-02-09T17:48:07.604596+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.0","version":"650'14794","reported_seq":15119,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-02-09T17:49:32.681467+0000","last_change":"2024-02-09T17:49:30.219592+0000","last_active":"2024-02-09T17:49:32.681467+0000","last_peered":"2024-02-09T17:49:32.681467+0000","last_clean":"2024-02-09T17:49:32.681467+0000","last_became_active":"2024-02-09T17:45:43.659881+0000","last_became_peered":"2024-02-09T17:45:43.659881+0000","last_unstale":"2024-02-09T17:49:32.681467+0000","last_undegraded":"2024-02-09T17:49:32.681467+0000","last_fullsized":"2024-02-09T17:49:32.681467+0000","mapping_epoch":576,"log_start":"592'4700","ondisk_log_start":"592'4700","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"650'14692","last_scrub_stamp":"2024-02-09T17:49:30.219479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:45:42.586555+0000","last_clean_scrub_stamp":"2024-02-09T17:49:30.219479+0000","objects_scrubbed":1454,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:53.921365+0000","scrub_duration":340,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88342528,"num_objects":1348,"num_object_clones":0,"num_object_copies":4044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14794,"num_write_kb":516544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"63'139","reported_seq":831,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-02-09T17:49:24.074994+0000","last_change":"2024-02-09T17:49:11.483130+0000","last_active":"2024-02-09T17:49:24.074994+0000","last_peered":"2024-02-09T17:49:24.074994+0000","last_clean":"2024-02-09T17:49:24.074994+0000","last_became_active":"2024-02-09T17:43:14.038885+0000","last_became_peered":"2024-02-09T17:43:14.038885+0000","last_unstale":"2024-02-09T17:49:24.074994+0000","last_undegraded":"2024-02-09T17:49:24.074994+0000","last_fullsized":"2024-02-09T17:49:24.074994+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:49:11.483012+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:49:11.483012+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:36.774820+0000","scrub_duration":166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"650'15132","reported_seq":15434,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-02-09T17:49:32.710359+0000","last_change":"2024-02-09T17:48:30.387181+0000","last_active":"2024-02-09T17:49:32.710359+0000","last_peered":"2024-02-09T17:49:32.710359+0000","last_clean":"2024-02-09T17:49:32.710359+0000","last_became_active":"2024-02-09T17:45:43.660407+0000","last_became_peered":"2024-02-09T17:45:43.660407+0000","last_unstale":"2024-02-09T17:49:32.710359+0000","last_undegraded":"2024-02-09T17:49:32.710359+0000","last_fullsized":"2024-02-09T17:49:32.710359+0000","mapping_epoch":576,"log_start":"596'5100","ondisk_log_start":"596'5100","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"632'11815","last_scrub_stamp":"2024-02-09T17:48:30.387098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:45:42.586555+0000","last_clean_scrub_stamp":"2024-02-09T17:48:30.387098+0000","objects_scrubbed":4675,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:49:51.873976+0000","scrub_duration":1299,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86507520,"num_objects":1320,"num_object_clones":0,"num_object_copies":3960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15132,"num_write_kb":526464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"63'139","reported_seq":1238,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-02-09T17:49:24.068197+0000","last_change":"2024-02-09T17:49:12.382010+0000","last_active":"2024-02-09T17:49:24.068197+0000","last_peered":"2024-02-09T17:49:24.068197+0000","last_clean":"2024-02-09T17:49:24.068197+0000","last_became_active":"2024-02-09T17:33:00.284924+0000","last_became_peered":"2024-02-09T17:33:00.284924+0000","last_unstale":"2024-02-09T17:49:24.068197+0000","last_undegraded":"2024-02-09T17:49:24.068197+0000","last_fullsized":"2024-02-09T17:49:24.068197+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:49:12.381894+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:56.105792+0000","last_clean_scrub_stamp":"2024-02-09T17:49:12.381894+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:33.945253+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.2","version":"650'15344","reported_seq":15621,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-02-09T17:49:31.056398+0000","last_change":"2024-02-09T17:49:15.536637+0000","last_active":"2024-02-09T17:49:31.056398+0000","last_peered":"2024-02-09T17:49:31.056398+0000","last_clean":"2024-02-09T17:49:31.056398+0000","last_became_active":"2024-02-09T17:45:43.607871+0000","last_became_peered":"2024-02-09T17:45:43.607871+0000","last_unstale":"2024-02-09T17:49:31.056398+0000","last_undegraded":"2024-02-09T17:49:31.056398+0000","last_fullsized":"2024-02-09T17:49:31.056398+0000","mapping_epoch":576,"log_start":"596'5300","ondisk_log_start":"596'5300","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"648'14550","last_scrub_stamp":"2024-02-09T17:49:15.536338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:45:42.586555+0000","last_clean_scrub_stamp":"2024-02-09T17:49:15.536338+0000","objects_scrubbed":2209,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:36.274708+0000","scrub_duration":778,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91488256,"num_objects":1396,"num_object_clones":0,"num_object_copies":4188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15344,"num_write_kb":535680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"63'139","reported_seq":1079,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-02-09T17:49:24.075419+0000","last_change":"2024-02-09T17:48:46.184610+0000","last_active":"2024-02-09T17:49:24.075419+0000","last_peered":"2024-02-09T17:49:24.075419+0000","last_clean":"2024-02-09T17:49:24.075419+0000","last_became_active":"2024-02-09T17:48:46.184307+0000","last_became_peered":"2024-02-09T17:48:46.184307+0000","last_unstale":"2024-02-09T17:49:24.075419+0000","last_undegraded":"2024-02-09T17:49:24.075419+0000","last_fullsized":"2024-02-09T17:49:24.075419+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:48:22.212557+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:48:22.212557+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:49:31.811843+0000","scrub_duration":73,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.d","version":"650'15130","reported_seq":15412,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-02-09T17:49:34.840330+0000","last_change":"2024-02-09T17:49:10.933973+0000","last_active":"2024-02-09T17:49:34.840330+0000","last_peered":"2024-02-09T17:49:34.840330+0000","last_clean":"2024-02-09T17:49:34.840330+0000","last_became_active":"2024-02-09T17:46:47.013898+0000","last_became_peered":"2024-02-09T17:46:47.013898+0000","last_unstale":"2024-02-09T17:49:34.840330+0000","last_undegraded":"2024-02-09T17:49:34.840330+0000","last_fullsized":"2024-02-09T17:49:34.840330+0000","mapping_epoch":600,"log_start":"596'5100","ondisk_log_start":"596'5100","created":576,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"646'13958","last_scrub_stamp":"2024-02-09T17:49:10.933833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:45:42.586555+0000","last_clean_scrub_stamp":"2024-02-09T17:49:10.933833+0000","objects_scrubbed":2430,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:22.992698+0000","scrub_duration":717,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81592344,"num_objects":1246,"num_object_clones":0,"num_object_copies":3738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15130,"num_write_kb":523969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,10],"acting":[1,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"63'139","reported_seq":1152,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-02-09T17:49:24.057250+0000","last_change":"2024-02-09T17:48:08.415436+0000","last_active":"2024-02-09T17:49:24.057250+0000","last_peered":"2024-02-09T17:49:24.057250+0000","last_clean":"2024-02-09T17:49:24.057250+0000","last_became_active":"2024-02-09T17:43:14.041291+0000","last_became_peered":"2024-02-09T17:43:14.041291+0000","last_unstale":"2024-02-09T17:49:24.057250+0000","last_undegraded":"2024-02-09T17:49:24.057250+0000","last_fullsized":"2024-02-09T17:49:24.057250+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:48:08.415321+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:48:08.415321+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:49:33.667943+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.c","version":"650'15012","reported_seq":15284,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-02-09T17:49:32.647561+0000","last_change":"2024-02-09T17:49:26.520622+0000","last_active":"2024-02-09T17:49:32.647561+0000","last_peered":"2024-02-09T17:49:32.647561+0000","last_clean":"2024-02-09T17:49:32.647561+0000","last_became_active":"2024-02-09T17:45:43.660635+0000","last_became_peered":"2024-02-09T17:45:43.660635+0000","last_unstale":"2024-02-09T17:49:32.647561+0000","last_undegraded":"2024-02-09T17:49:32.647561+0000","last_fullsized":"2024-02-09T17:49:32.647561+0000","mapping_epoch":576,"log_start":"595'5000","ondisk_log_start":"595'5000","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"650'14778","last_scrub_stamp":"2024-02-09T17:49:26.520433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:45:42.586555+0000","last_clean_scrub_stamp":"2024-02-09T17:49:26.520433+0000","objects_scrubbed":1562,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:49.840061+0000","scrub_duration":672,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86114304,"num_objects":1314,"num_object_clones":0,"num_object_copies":3942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15012,"num_write_kb":522432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"63'139","reported_seq":1002,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-02-09T17:49:24.072055+0000","last_change":"2024-02-09T17:48:22.237143+0000","last_active":"2024-02-09T17:49:24.072055+0000","last_peered":"2024-02-09T17:49:24.072055+0000","last_clean":"2024-02-09T17:49:24.072055+0000","last_became_active":"2024-02-09T17:43:14.054943+0000","last_became_peered":"2024-02-09T17:43:14.054943+0000","last_unstale":"2024-02-09T17:49:24.072055+0000","last_undegraded":"2024-02-09T17:49:24.072055+0000","last_fullsized":"2024-02-09T17:49:24.072055+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:48:22.237014+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:48:22.237014+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:49:48.423322+0000","scrub_duration":161,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.f","version":"650'14626","reported_seq":14919,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-02-09T17:49:33.168116+0000","last_change":"2024-02-09T17:48:33.301997+0000","last_active":"2024-02-09T17:49:33.168116+0000","last_peered":"2024-02-09T17:49:33.168116+0000","last_clean":"2024-02-09T17:49:33.168116+0000","last_became_active":"2024-02-09T17:45:43.601656+0000","last_became_peered":"2024-02-09T17:45:43.601656+0000","last_unstale":"2024-02-09T17:49:33.168116+0000","last_undegraded":"2024-02-09T17:49:33.168116+0000","last_fullsized":"2024-02-09T17:49:33.168116+0000","mapping_epoch":576,"log_start":"592'4600","ondisk_log_start":"592'4600","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"632'11634","last_scrub_stamp":"2024-02-09T17:48:33.301784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:45:42.586555+0000","last_clean_scrub_stamp":"2024-02-09T17:48:33.301784+0000","objects_scrubbed":4320,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:49:35.088154+0000","scrub_duration":801,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86245376,"num_objects":1316,"num_object_clones":0,"num_object_copies":3948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14626,"num_write_kb":510144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.8","version":"63'139","reported_seq":794,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-02-09T17:49:24.062239+0000","last_change":"2024-02-09T17:49:12.945703+0000","last_active":"2024-02-09T17:49:24.062239+0000","last_peered":"2024-02-09T17:49:24.062239+0000","last_clean":"2024-02-09T17:49:24.062239+0000","last_became_active":"2024-02-09T17:46:47.013853+0000","last_became_peered":"2024-02-09T17:46:47.013853+0000","last_unstale":"2024-02-09T17:49:24.062239+0000","last_undegraded":"2024-02-09T17:49:24.062239+0000","last_fullsized":"2024-02-09T17:49:24.062239+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:49:12.945588+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:49:12.945588+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:14.177994+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"650'14762","reported_seq":15062,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-02-09T17:49:31.019658+0000","last_change":"2024-02-09T17:48:23.379607+0000","last_active":"2024-02-09T17:49:31.019658+0000","last_peered":"2024-02-09T17:49:31.019658+0000","last_clean":"2024-02-09T17:49:31.019658+0000","last_became_active":"2024-02-09T17:45:43.607953+0000","last_became_peered":"2024-02-09T17:45:43.607953+0000","last_unstale":"2024-02-09T17:49:31.019658+0000","last_undegraded":"2024-02-09T17:49:31.019658+0000","last_fullsized":"2024-02-09T17:49:31.019658+0000","mapping_epoch":576,"log_start":"593'4700","ondisk_log_start":"593'4700","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"631'11314","last_scrub_stamp":"2024-02-09T17:48:23.379391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:45:42.586555+0000","last_clean_scrub_stamp":"2024-02-09T17:48:23.379391+0000","objects_scrubbed":4880,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":658,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93716480,"num_objects":1430,"num_object_clones":0,"num_object_copies":4290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14762,"num_write_kb":518144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"63'139","reported_seq":1665,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-02-09T17:49:24.065342+0000","last_change":"2024-02-09T17:48:27.166794+0000","last_active":"2024-02-09T17:49:24.065342+0000","last_peered":"2024-02-09T17:49:24.065342+0000","last_clean":"2024-02-09T17:49:24.065342+0000","last_became_active":"2024-02-09T17:43:28.590237+0000","last_became_peered":"2024-02-09T17:43:28.590237+0000","last_unstale":"2024-02-09T17:49:24.065342+0000","last_undegraded":"2024-02-09T17:49:24.065342+0000","last_fullsized":"2024-02-09T17:49:24.065342+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:48:27.166671+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:45:47.710252+0000","last_clean_scrub_stamp":"2024-02-09T17:48:27.166671+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:49:39.583573+0000","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"650'14832","reported_seq":15130,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-02-09T17:49:31.705558+0000","last_change":"2024-02-09T17:48:24.314402+0000","last_active":"2024-02-09T17:49:31.705558+0000","last_peered":"2024-02-09T17:49:31.705558+0000","last_clean":"2024-02-09T17:49:31.705558+0000","last_became_active":"2024-02-09T17:45:43.659891+0000","last_became_peered":"2024-02-09T17:45:43.659891+0000","last_unstale":"2024-02-09T17:49:31.705558+0000","last_undegraded":"2024-02-09T17:49:31.705558+0000","last_fullsized":"2024-02-09T17:49:31.705558+0000","mapping_epoch":576,"log_start":"594'4800","ondisk_log_start":"594'4800","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"631'11417","last_scrub_stamp":"2024-02-09T17:48:24.314163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:45:42.586555+0000","last_clean_scrub_stamp":"2024-02-09T17:48:24.314163+0000","objects_scrubbed":4775,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:49:48.910925+0000","scrub_duration":977,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87818240,"num_objects":1340,"num_object_clones":0,"num_object_copies":4020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14832,"num_write_kb":517504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,8],"acting":[9,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.e","version":"63'139","reported_seq":1122,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-02-09T17:49:24.069385+0000","last_change":"2024-02-09T17:48:12.955847+0000","last_active":"2024-02-09T17:49:24.069385+0000","last_peered":"2024-02-09T17:49:24.069385+0000","last_clean":"2024-02-09T17:49:24.069385+0000","last_became_active":"2024-02-09T17:43:14.036583+0000","last_became_peered":"2024-02-09T17:43:14.036583+0000","last_unstale":"2024-02-09T17:49:24.069385+0000","last_undegraded":"2024-02-09T17:49:24.069385+0000","last_fullsized":"2024-02-09T17:49:24.069385+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:48:12.955736+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:14.981453+0000","last_clean_scrub_stamp":"2024-02-09T17:48:12.955736+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:49:37.021474+0000","scrub_duration":141,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"650'14935","reported_seq":15238,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-02-09T17:49:30.534681+0000","last_change":"2024-02-09T17:48:25.328842+0000","last_active":"2024-02-09T17:49:30.534681+0000","last_peered":"2024-02-09T17:49:30.534681+0000","last_clean":"2024-02-09T17:49:30.534681+0000","last_became_active":"2024-02-09T17:45:43.611062+0000","last_became_peered":"2024-02-09T17:45:43.611062+0000","last_unstale":"2024-02-09T17:49:30.534681+0000","last_undegraded":"2024-02-09T17:49:30.534681+0000","last_fullsized":"2024-02-09T17:49:30.534681+0000","mapping_epoch":576,"log_start":"593'4900","ondisk_log_start":"593'4900","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"631'11631","last_scrub_stamp":"2024-02-09T17:48:25.328752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:45:42.586555+0000","last_clean_scrub_stamp":"2024-02-09T17:48:25.328752+0000","objects_scrubbed":4757,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:49:50.239778+0000","scrub_duration":1199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93257728,"num_objects":1423,"num_object_clones":0,"num_object_copies":4269,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14935,"num_write_kb":523456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,3],"acting":[11,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.f","version":"63'139","reported_seq":1182,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-02-09T17:49:24.074903+0000","last_change":"2024-02-09T17:48:34.651789+0000","last_active":"2024-02-09T17:49:24.074903+0000","last_peered":"2024-02-09T17:49:24.074903+0000","last_clean":"2024-02-09T17:49:24.074903+0000","last_became_active":"2024-02-09T17:33:05.316100+0000","last_became_peered":"2024-02-09T17:33:05.316100+0000","last_unstale":"2024-02-09T17:49:24.074903+0000","last_undegraded":"2024-02-09T17:49:24.074903+0000","last_fullsized":"2024-02-09T17:49:24.074903+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:48:34.651647+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:48:34.651647+0000","last_clean_scrub_stamp":"2024-02-09T17:48:34.651647+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:49:36.629744+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"650'14846","reported_seq":15143,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-02-09T17:49:32.669592+0000","last_change":"2024-02-09T17:48:32.876630+0000","last_active":"2024-02-09T17:49:32.669592+0000","last_peered":"2024-02-09T17:49:32.669592+0000","last_clean":"2024-02-09T17:49:32.669592+0000","last_became_active":"2024-02-09T17:45:43.611004+0000","last_became_peered":"2024-02-09T17:45:43.611004+0000","last_unstale":"2024-02-09T17:49:32.669592+0000","last_undegraded":"2024-02-09T17:49:32.669592+0000","last_fullsized":"2024-02-09T17:49:32.669592+0000","mapping_epoch":576,"log_start":"593'4800","ondisk_log_start":"593'4800","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"632'11816","last_scrub_stamp":"2024-02-09T17:48:32.876436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:45:42.586555+0000","last_clean_scrub_stamp":"2024-02-09T17:48:32.876436+0000","objects_scrubbed":4391,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:49:46.247182+0000","scrub_duration":1588,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87293952,"num_objects":1332,"num_object_clones":0,"num_object_copies":3996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14846,"num_write_kb":517696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"63'139","reported_seq":1309,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-02-09T17:49:24.062302+0000","last_change":"2024-02-09T17:49:11.961487+0000","last_active":"2024-02-09T17:49:24.062302+0000","last_peered":"2024-02-09T17:49:24.062302+0000","last_clean":"2024-02-09T17:49:24.062302+0000","last_became_active":"2024-02-09T17:43:14.027715+0000","last_became_peered":"2024-02-09T17:43:14.027715+0000","last_unstale":"2024-02-09T17:49:24.062302+0000","last_undegraded":"2024-02-09T17:49:24.062302+0000","last_fullsized":"2024-02-09T17:49:24.062302+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:49:11.961373+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:14.135321+0000","last_clean_scrub_stamp":"2024-02-09T17:49:11.961373+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:41.624952+0000","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"650'15040","reported_seq":15318,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-02-09T17:49:31.122549+0000","last_change":"2024-02-09T17:49:18.938095+0000","last_active":"2024-02-09T17:49:31.122549+0000","last_peered":"2024-02-09T17:49:31.122549+0000","last_clean":"2024-02-09T17:49:31.122549+0000","last_became_active":"2024-02-09T17:46:47.014763+0000","last_became_peered":"2024-02-09T17:46:47.014763+0000","last_unstale":"2024-02-09T17:49:31.122549+0000","last_undegraded":"2024-02-09T17:49:31.122549+0000","last_fullsized":"2024-02-09T17:49:31.122549+0000","mapping_epoch":600,"log_start":"595'5000","ondisk_log_start":"595'5000","created":576,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"649'14453","last_scrub_stamp":"2024-02-09T17:49:18.937963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:45:42.586555+0000","last_clean_scrub_stamp":"2024-02-09T17:49:18.937963+0000","objects_scrubbed":1986,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:23.347045+0000","scrub_duration":608,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90832896,"num_objects":1386,"num_object_clones":0,"num_object_copies":4158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15040,"num_write_kb":525632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.d","version":"63'139","reported_seq":714,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-02-09T17:49:24.070958+0000","last_change":"2024-02-09T17:48:26.223438+0000","last_active":"2024-02-09T17:49:24.070958+0000","last_peered":"2024-02-09T17:49:24.070958+0000","last_clean":"2024-02-09T17:49:24.070958+0000","last_became_active":"2024-02-09T17:43:14.036759+0000","last_became_peered":"2024-02-09T17:43:14.036759+0000","last_unstale":"2024-02-09T17:49:24.070958+0000","last_undegraded":"2024-02-09T17:49:24.070958+0000","last_fullsized":"2024-02-09T17:49:24.070958+0000","mapping_epoch":454,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:48:26.220964+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:48:26.220964+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:49:41.454418+0000","scrub_duration":185,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"63'139","reported_seq":886,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-02-09T17:49:24.065440+0000","last_change":"2024-02-09T17:48:35.170778+0000","last_active":"2024-02-09T17:49:24.065440+0000","last_peered":"2024-02-09T17:49:24.065440+0000","last_clean":"2024-02-09T17:49:24.065440+0000","last_became_active":"2024-02-09T17:46:47.013920+0000","last_became_peered":"2024-02-09T17:46:47.013920+0000","last_unstale":"2024-02-09T17:49:24.065440+0000","last_undegraded":"2024-02-09T17:49:24.065440+0000","last_fullsized":"2024-02-09T17:49:24.065440+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:48:35.170638+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:48:35.170638+0000","last_clean_scrub_stamp":"2024-02-09T17:48:35.170638+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:49:56.897817+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"63'139","reported_seq":1138,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-02-09T17:49:31.135846+0000","last_change":"2024-02-09T17:49:31.135888+0000","last_active":"2024-02-09T17:49:31.135846+0000","last_peered":"2024-02-09T17:49:31.135846+0000","last_clean":"2024-02-09T17:49:31.135846+0000","last_became_active":"2024-02-09T17:34:01.353171+0000","last_became_peered":"2024-02-09T17:34:01.353171+0000","last_unstale":"2024-02-09T17:49:31.135846+0000","last_undegraded":"2024-02-09T17:49:31.135846+0000","last_fullsized":"2024-02-09T17:49:31.135846+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:49:31.135755+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:44:51.190789+0000","last_clean_scrub_stamp":"2024-02-09T17:49:31.135755+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:58.374348+0000","scrub_duration":343,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.12","version":"63'139","reported_seq":1223,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-02-09T17:49:24.067839+0000","last_change":"2024-02-09T17:48:18.579925+0000","last_active":"2024-02-09T17:49:24.067839+0000","last_peered":"2024-02-09T17:49:24.067839+0000","last_clean":"2024-02-09T17:49:24.067839+0000","last_became_active":"2024-02-09T17:46:47.015622+0000","last_became_peered":"2024-02-09T17:46:47.015622+0000","last_unstale":"2024-02-09T17:49:24.067839+0000","last_undegraded":"2024-02-09T17:49:24.067839+0000","last_fullsized":"2024-02-09T17:49:24.067839+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:48:18.579786+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:48:18.579786+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:49:47.349100+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.13","version":"63'139","reported_seq":842,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-02-09T17:49:24.068126+0000","last_change":"2024-02-09T17:49:20.518555+0000","last_active":"2024-02-09T17:49:24.068126+0000","last_peered":"2024-02-09T17:49:24.068126+0000","last_clean":"2024-02-09T17:49:24.068126+0000","last_became_active":"2024-02-09T17:41:14.521672+0000","last_became_peered":"2024-02-09T17:41:14.521672+0000","last_unstale":"2024-02-09T17:49:24.068126+0000","last_undegraded":"2024-02-09T17:49:24.068126+0000","last_fullsized":"2024-02-09T17:49:24.068126+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:49:20.518438+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:49:20.518438+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:36.344913+0000","scrub_duration":213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":1415184408,"num_objects":21595,"num_object_clones":0,"num_object_copies":64785,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21595,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":239163,"num_write_kb":8344193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4236521472,"data_stored":4236509256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160724,"ondisk_log_size":160724,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056967,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":789976,"kb_used_data":185424,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":92955176,"statfs":{"total":95995035648,"available":95186100224,"internally_reserved":0,"allocated":189874176,"data_stored":189589995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":619051460},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.6120000000000001,"5min":1.21,"15min":1.3089999999999999},"min":{"1min":0.24199999999999999,"5min":0.221,"15min":0.216},"max":{"1min":5.5679999999999996,"5min":5.8719999999999999,"15min":12.836},"last":1.661},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.1080000000000001,"15min":1.2370000000000001},"min":{"1min":0.27500000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":4.5570000000000004,"5min":5.4189999999999996,"15min":9.8219999999999992},"last":1.5309999999999999}]},{"osd":1,"last update":"Fri Feb 9 17:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.7390000000000001,"5min":1.3799999999999999,"15min":1.4019999999999999},"min":{"1min":0.313,"5min":0.26300000000000001,"15min":0.22700000000000001},"max":{"1min":7.6909999999999998,"5min":7.7249999999999996,"15min":9.7710000000000008},"last":2.415},{"interface":"front","average":{"1min":1.728,"5min":1.6100000000000001,"15min":1.5129999999999999},"min":{"1min":0.29999999999999999,"5min":0.27700000000000002,"15min":0.26400000000000001},"max":{"1min":7.673,"5min":7.7110000000000003,"15min":9.718},"last":2.7850000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:31:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.415,"15min":1.6859999999999999},"min":{"1min":0.378,"5min":0.247,"15min":0.247},"max":{"1min":4.4189999999999996,"5min":9.4719999999999995,"15min":9.4719999999999995},"last":0.35399999999999998},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.5149999999999999,"15min":1.4399999999999999},"min":{"1min":0.378,"5min":0.252,"15min":0.252},"max":{"1min":4.8579999999999997,"5min":7.96,"15min":7.96},"last":0.60399999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.7350000000000001,"15min":1.379},"min":{"1min":0.36799999999999999,"5min":0.248,"15min":0.248},"max":{"1min":4.6269999999999998,"5min":21.475000000000001,"15min":21.475000000000001},"last":8.1460000000000008},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.8380000000000001,"15min":1.6699999999999999},"min":{"1min":0.442,"5min":0.35599999999999998,"15min":0.29799999999999999},"max":{"1min":4.6769999999999996,"5min":21.498000000000001,"15min":21.498000000000001},"last":8.1720000000000006}]},{"osd":4,"last update":"Fri Feb 9 17:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":2.044,"15min":1.881},"min":{"1min":0.31,"5min":0.28299999999999997,"15min":0.246},"max":{"1min":4.9089999999999998,"5min":65.109999999999999,"15min":65.109999999999999},"last":1.4690000000000001},{"interface":"front","average":{"1min":1.1990000000000001,"5min":2.254,"15min":1.74},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.309},"max":{"1min":4.899,"5min":65.099000000000004,"15min":65.099000000000004},"last":0.51300000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:48:51 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":1.615,"15min":1.4750000000000001},"min":{"1min":0.30399999999999999,"5min":0.252,"15min":0.252},"max":{"1min":2.9740000000000002,"5min":18.878,"15min":18.878},"last":2.7599999999999998},{"interface":"front","average":{"1min":0.82899999999999996,"5min":1.657,"15min":1.641},"min":{"1min":0.38100000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":2.9529999999999998,"5min":18.895,"15min":18.895},"last":1.554}]},{"osd":6,"last update":"Fri Feb 9 17:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.512,"15min":1.645},"min":{"1min":0.31900000000000001,"5min":0.30399999999999999,"15min":0.29899999999999999},"max":{"1min":5.9710000000000001,"5min":9.1679999999999993,"15min":12.798999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":1.123,"5min":1.6850000000000001,"15min":1.675},"min":{"1min":0.40899999999999997,"5min":0.36699999999999999,"15min":0.32700000000000001},"max":{"1min":5.9619999999999997,"5min":10.000999999999999,"15min":10.000999999999999},"last":2.806}]},{"osd":7,"last update":"Fri Feb 9 17:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.8700000000000001,"15min":1.8340000000000001},"min":{"1min":0.38,"5min":0.32000000000000001,"15min":0.312},"max":{"1min":3.7429999999999999,"5min":18.846,"15min":20.489999999999998},"last":2.7069999999999999},{"interface":"front","average":{"1min":1.8260000000000001,"5min":1.7969999999999999,"15min":1.758},"min":{"1min":0.42099999999999999,"5min":0.36799999999999999,"15min":0.34200000000000003},"max":{"1min":7.165,"5min":19.125,"15min":20.712},"last":1.506}]},{"osd":8,"last update":"Fri Feb 9 17:49:01 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.79800000000000004,"15min":1.0860000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.3069999999999999,"5min":5.3739999999999997,"15min":9.5429999999999993},"last":2.6829999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.86599999999999999,"15min":1.145},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.318},"max":{"1min":1.3879999999999999,"5min":5.4139999999999997,"15min":9.2940000000000005},"last":2.726}]},{"osd":9,"last update":"Fri Feb 9 17:48:32 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":1.129,"15min":1.4399999999999999},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.82399999999999995,"5min":12.27,"15min":62.518000000000001},"last":2.7410000000000001},{"interface":"front","average":{"1min":0.68300000000000005,"5min":1.0669999999999999,"15min":1.3799999999999999},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":2.036,"5min":12.313000000000001,"15min":63.241999999999997},"last":2.8399999999999999}]},{"osd":10,"last update":"Fri Feb 9 17:48:51 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":1.1439999999999999,"15min":1.2749999999999999},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.2410000000000001,"5min":5.3799999999999999,"15min":9.6590000000000007},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.627,"5min":1.157,"15min":1.3500000000000001},"min":{"1min":0.437,"5min":0.377,"15min":0.35799999999999998},"max":{"1min":0.76700000000000002,"5min":5.5579999999999998,"15min":12.856},"last":2.823}]}]},{"osd":10,"up_from":84,"seq":360777253197,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":718236,"kb_used_data":253844,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":93026916,"statfs":{"total":95995035648,"available":95259561984,"internally_reserved":0,"allocated":259936256,"data_stored":259648003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":475462080},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:48:33 2024","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":0.91200000000000003,"15min":1.21},"min":{"1min":0.26000000000000001,"5min":0.24099999999999999,"15min":0.20599999999999999},"max":{"1min":7.4290000000000003,"5min":7.4290000000000003,"15min":9.343},"last":0.30599999999999999},{"interface":"front","average":{"1min":1.5669999999999999,"5min":0.93200000000000005,"15min":1.0549999999999999},"min":{"1min":0.26400000000000001,"5min":0.248,"15min":0.248},"max":{"1min":7.2830000000000004,"5min":7.2830000000000004,"15min":7.2830000000000004},"last":2.3300000000000001}]},{"osd":1,"last update":"Fri Feb 9 17:49:00 2024","interfaces":[{"interface":"back","average":{"1min":1.7070000000000001,"5min":1.274,"15min":0.877},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.23999999999999999},"max":{"1min":9.3640000000000008,"5min":9.3640000000000008,"15min":9.3640000000000008},"last":0.49199999999999999},{"interface":"front","average":{"1min":1.958,"5min":1.357,"15min":1.0669999999999999},"min":{"1min":0.25900000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":8.9789999999999992,"5min":8.9789999999999992,"15min":8.9789999999999992},"last":3.9180000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:49:12 2024","interfaces":[{"interface":"back","average":{"1min":2.3039999999999998,"5min":1.742,"15min":1.5609999999999999},"min":{"1min":0.28699999999999998,"5min":0.28299999999999997,"15min":0.27100000000000002},"max":{"1min":7.1790000000000003,"5min":9.2539999999999996,"15min":11.384},"last":0.39400000000000002},{"interface":"front","average":{"1min":1.625,"5min":1.7849999999999999,"15min":1.427},"min":{"1min":0.308,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":3.6520000000000001,"5min":38.868000000000002,"15min":38.868000000000002},"last":0.52400000000000002}]},{"osd":3,"last update":"Fri Feb 9 17:49:00 2024","interfaces":[{"interface":"back","average":{"1min":2.0710000000000002,"5min":1.873,"15min":1.758},"min":{"1min":0.373,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":7.5880000000000001,"5min":35.113,"15min":35.113},"last":0.46600000000000003},{"interface":"front","average":{"1min":2.2810000000000001,"5min":2.117,"15min":1.9239999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":12.202999999999999,"5min":34.359000000000002,"15min":34.359000000000002},"last":0.64000000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":1.1519999999999999,"15min":1.161},"min":{"1min":0.33700000000000002,"5min":0.32100000000000001,"15min":0.29199999999999998},"max":{"1min":5.3739999999999997,"5min":9.0990000000000002,"15min":9.0990000000000002},"last":2.4220000000000002},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.1060000000000001,"15min":1.1220000000000001},"min":{"1min":0.41299999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":5.2949999999999999,"5min":7.0309999999999997,"15min":11.273999999999999},"last":2.4460000000000002}]},{"osd":5,"last update":"Fri Feb 9 17:49:12 2024","interfaces":[{"interface":"back","average":{"1min":1.871,"5min":1.5329999999999999,"15min":1.2110000000000001},"min":{"1min":0.32800000000000001,"5min":0.29099999999999998,"15min":0.28799999999999998},"max":{"1min":7.2229999999999999,"5min":7.2229999999999999,"15min":7.5179999999999998},"last":0.55600000000000005},{"interface":"front","average":{"1min":1.79,"5min":1.569,"15min":1.4119999999999999},"min":{"1min":0.39700000000000002,"5min":0.38800000000000001,"15min":0.33500000000000002},"max":{"1min":7.2590000000000003,"5min":10.243,"15min":10.243},"last":0.443}]},{"osd":6,"last update":"Fri Feb 9 17:49:12 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":1.4259999999999999,"15min":1.4690000000000001},"min":{"1min":0.34200000000000003,"5min":0.33600000000000002,"15min":0.29999999999999999},"max":{"1min":5.5149999999999997,"5min":9.0820000000000007,"15min":10.962},"last":0.71699999999999997},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.4630000000000001,"15min":1.4299999999999999},"min":{"1min":0.38,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":5.1950000000000003,"5min":9.0690000000000008,"15min":9.0690000000000008},"last":0.41899999999999998}]},{"osd":7,"last update":"Fri Feb 9 17:39:52 2024","interfaces":[{"interface":"back","average":{"1min":1.6000000000000001,"5min":1.367,"15min":1.415},"min":{"1min":0.42099999999999999,"5min":0.34200000000000003,"15min":0.312},"max":{"1min":4.0430000000000001,"5min":6.7590000000000003,"15min":17.754000000000001},"last":4.6449999999999996},{"interface":"front","average":{"1min":1.498,"5min":1.4359999999999999,"15min":1.5189999999999999},"min":{"1min":0.47399999999999998,"5min":0.38300000000000001,"15min":0.316},"max":{"1min":5.8170000000000002,"5min":11.308999999999999,"15min":11.308999999999999},"last":0.55000000000000004}]},{"osd":8,"last update":"Fri Feb 9 17:49:00 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":0.91900000000000004,"15min":1.0409999999999999},"min":{"1min":0.32000000000000001,"5min":0.312,"15min":0.29099999999999998},"max":{"1min":4.6550000000000002,"5min":6.9749999999999996,"15min":7.4359999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":1.105,"5min":1.147,"15min":1.0880000000000001},"min":{"1min":0.36899999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":5.319,"5min":7.0460000000000003,"15min":7.7060000000000004},"last":0.752}]},{"osd":9,"last update":"Fri Feb 9 17:49:12 2024","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":1.0109999999999999,"15min":1.0860000000000001},"min":{"1min":0.35299999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":4.7000000000000002,"5min":7.0419999999999998,"15min":7.4850000000000003},"last":2.29},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.1970000000000001,"15min":1.24},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.35899999999999999},"max":{"1min":6.585,"5min":7.1299999999999999,"15min":10.622999999999999},"last":0.58399999999999996}]},{"osd":11,"last update":"Fri Feb 9 17:48:33 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.89200000000000002,"15min":0.99299999999999999},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.008,"5min":3.5249999999999999,"15min":7.383},"last":2.3660000000000001},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.94299999999999995,"15min":1.0269999999999999},"min":{"1min":0.45600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.117,"5min":3.5960000000000001,"15min":6.2590000000000003},"last":2.3940000000000001}]}]},{"osd":9,"up_from":77,"seq":330712482130,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":891864,"kb_used_data":355344,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":92853288,"statfs":{"total":95995035648,"available":95081766912,"internally_reserved":0,"allocated":363872256,"data_stored":363588075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":549386688},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:49:08 2024","interfaces":[{"interface":"back","average":{"1min":2.113,"5min":1.248,"15min":0.874},"min":{"1min":0.252,"5min":0.215,"15min":0.215},"max":{"1min":7.1289999999999996,"5min":8.4670000000000005,"15min":8.4670000000000005},"last":0.29999999999999999},{"interface":"front","average":{"1min":1.278,"5min":1.5740000000000001,"15min":1.179},"min":{"1min":0.32000000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":3.4569999999999999,"5min":11.596,"15min":11.596},"last":0.316}]},{"osd":1,"last update":"Fri Feb 9 17:48:49 2024","interfaces":[{"interface":"back","average":{"1min":1.863,"5min":1.411,"15min":1.786},"min":{"1min":0.30499999999999999,"5min":0.26500000000000001,"15min":0.23499999999999999},"max":{"1min":5.835,"5min":8.1310000000000002,"15min":89.515000000000001},"last":0.28100000000000003},{"interface":"front","average":{"1min":1.788,"5min":1.1899999999999999,"15min":1.6970000000000001},"min":{"1min":0.31,"5min":0.25,"15min":0.249},"max":{"1min":11.685,"5min":11.685,"15min":89.388000000000005},"last":0.40000000000000002}]},{"osd":2,"last update":"Fri Feb 9 17:49:20 2024","interfaces":[{"interface":"back","average":{"1min":2.2069999999999999,"5min":1.8109999999999999,"15min":1.7410000000000001},"min":{"1min":0.307,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":5.9370000000000003,"5min":5.9370000000000003,"15min":5.9370000000000003},"last":0.88300000000000001},{"interface":"front","average":{"1min":2.4289999999999998,"5min":1.736,"15min":1.6459999999999999},"min":{"1min":0.35399999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":8.4250000000000007,"5min":8.4250000000000007,"15min":8.4250000000000007},"last":1.7749999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:45:45 2024","interfaces":[{"interface":"back","average":{"1min":1.569,"5min":1.516,"15min":1.671},"min":{"1min":0.36899999999999999,"5min":0.26500000000000001,"15min":0.25700000000000001},"max":{"1min":6.3789999999999996,"5min":8.9939999999999998,"15min":40.18},"last":0.31},{"interface":"front","average":{"1min":1.264,"5min":1.079,"15min":1.556},"min":{"1min":0.33300000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":6.048,"5min":7.2519999999999998,"15min":40.158000000000001},"last":0.42299999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:48:49 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":1.194,"15min":1.425},"min":{"1min":0.28199999999999997,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.8260000000000001,"5min":7.8899999999999997,"15min":27.381},"last":0.34799999999999998},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.04,"15min":1.2969999999999999},"min":{"1min":0.34300000000000003,"5min":0.26900000000000002,"15min":0.26100000000000001},"max":{"1min":5.415,"5min":5.415,"15min":27.355},"last":0.36799999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:49:00 2024","interfaces":[{"interface":"back","average":{"1min":1.5389999999999999,"5min":2.1000000000000001,"15min":1.6479999999999999},"min":{"1min":0.33700000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":5.8499999999999996,"5min":77.924000000000007,"15min":77.924000000000007},"last":0.501},{"interface":"front","average":{"1min":1.0629999999999999,"5min":2.1880000000000002,"15min":1.663},"min":{"1min":0.371,"5min":0.318,"15min":0.318},"max":{"1min":3.7149999999999999,"5min":77.899000000000001,"15min":77.899000000000001},"last":0.60899999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:48:49 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.4339999999999999,"15min":1.5069999999999999},"min":{"1min":0.34799999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":4.5620000000000003,"5min":8.8559999999999999,"15min":14.069000000000001},"last":0.67400000000000004},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.8029999999999999,"15min":1.6839999999999999},"min":{"1min":0.38500000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":5.0549999999999997,"5min":9.5250000000000004,"15min":14.097},"last":2.1320000000000001}]},{"osd":7,"last update":"Fri Feb 9 17:48:49 2024","interfaces":[{"interface":"back","average":{"1min":1.6319999999999999,"5min":2.2280000000000002,"15min":1.899},"min":{"1min":0.41499999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":5.4859999999999998,"5min":29.001999999999999,"15min":29.001999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":1.6220000000000001,"5min":2.2040000000000002,"15min":1.907},"min":{"1min":0.434,"5min":0.373,"15min":0.32200000000000001},"max":{"1min":5.4720000000000004,"5min":28.872,"15min":28.872},"last":0.54200000000000004}]},{"osd":8,"last update":"Fri Feb 9 17:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.72899999999999998,"15min":1.1040000000000001},"min":{"1min":0.38200000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":4.2160000000000002,"5min":4.2160000000000002,"15min":14.045999999999999},"last":0.436},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.0189999999999999,"15min":1.2470000000000001},"min":{"1min":0.38100000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":3.5310000000000001,"5min":12.885999999999999,"15min":12.885999999999999},"last":0.57899999999999996}]},{"osd":10,"last update":"Fri Feb 9 17:49:20 2024","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.25,"15min":1.331},"min":{"1min":0.40100000000000002,"5min":0.313,"15min":0.312},"max":{"1min":5.8259999999999996,"5min":11.404,"15min":11.404},"last":0.55600000000000005},{"interface":"front","average":{"1min":1.115,"5min":0.878,"15min":1.159},"min":{"1min":0.40699999999999997,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":3.7040000000000002,"5min":4.5,"15min":10.904},"last":0.46300000000000002}]},{"osd":11,"last update":"Fri Feb 9 17:49:20 2024","interfaces":[{"interface":"back","average":{"1min":1.425,"5min":0.98799999999999999,"15min":1.3200000000000001},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":5.8010000000000002,"5min":7.3330000000000002,"15min":14.02},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.83899999999999997,"15min":1.258},"min":{"1min":0.42099999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":3.9940000000000002,"5min":6.9690000000000003,"15min":10.863},"last":0.63700000000000001}]}]},{"osd":8,"up_from":171,"seq":734439407873,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":470040,"kb_used_data":171984,"kb_used_omap":212,"kb_used_meta":297835,"kb_avail":93275112,"statfs":{"total":95995035648,"available":95513714688,"internally_reserved":0,"allocated":176111616,"data_stored":175827435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217579,"internal_metadata":304983573},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:48:55 2024","interfaces":[{"interface":"back","average":{"1min":1.4810000000000001,"5min":0.93000000000000005,"15min":0.97999999999999998},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.217},"max":{"1min":7.867,"5min":7.867,"15min":8.0920000000000005},"last":0.32100000000000001},{"interface":"front","average":{"1min":1.887,"5min":1.1220000000000001,"15min":1.282},"min":{"1min":0.31900000000000001,"5min":0.23999999999999999,"15min":0.23100000000000001},"max":{"1min":12.045,"5min":12.045,"15min":12.045},"last":0.94899999999999995}]},{"osd":1,"last update":"Fri Feb 9 17:48:55 2024","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.1679999999999999,"15min":1.355},"min":{"1min":0.28699999999999998,"5min":0.252,"15min":0.22700000000000001},"max":{"1min":7.3280000000000003,"5min":7.3280000000000003,"15min":8.718},"last":0.496},{"interface":"front","average":{"1min":1.5660000000000001,"5min":1.212,"15min":1.5229999999999999},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.23300000000000001},"max":{"1min":6.3179999999999996,"5min":6.3179999999999996,"15min":25.027000000000001},"last":0.35099999999999998}]},{"osd":2,"last update":"Fri Feb 9 17:48:32 2024","interfaces":[{"interface":"back","average":{"1min":1.7090000000000001,"5min":1.377,"15min":1.2889999999999999},"min":{"1min":0.30499999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":4.8959999999999999,"5min":9.0060000000000002,"15min":9.0060000000000002},"last":0.442},{"interface":"front","average":{"1min":1.4750000000000001,"5min":1.1859999999999999,"15min":0.93400000000000005},"min":{"1min":0.29699999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":4.2480000000000002,"5min":5.2930000000000001,"15min":5.2930000000000001},"last":0.38900000000000001}]},{"osd":3,"last update":"Fri Feb 9 17:42:38 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":1.542,"15min":1.6379999999999999},"min":{"1min":0.316,"5min":0.29499999999999998,"15min":0.28699999999999998},"max":{"1min":2.347,"5min":15.228999999999999,"15min":24.030999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.75600000000000001,"5min":1.8340000000000001,"15min":1.8640000000000001},"min":{"1min":0.36199999999999999,"5min":0.33600000000000002,"15min":0.30299999999999999},"max":{"1min":2.3719999999999999,"5min":14.988,"15min":24.318999999999999},"last":0.69399999999999995}]},{"osd":4,"last update":"Fri Feb 9 17:48:55 2024","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.1910000000000001,"15min":1.5249999999999999},"min":{"1min":0.35299999999999998,"5min":0.30499999999999999,"15min":0.28199999999999997},"max":{"1min":3.98,"5min":7.0369999999999999,"15min":53.060000000000002},"last":0.41699999999999998},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.5649999999999999,"15min":1.647},"min":{"1min":0.35599999999999998,"5min":0.28299999999999997,"15min":0.27000000000000002},"max":{"1min":3.577,"5min":8.2360000000000007,"15min":53.311999999999998},"last":6.2149999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.133,"15min":2.0230000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":3.165,"5min":5.0819999999999999,"15min":6.4249999999999998},"last":4.0049999999999999},{"interface":"front","average":{"1min":1.534,"5min":1.2470000000000001,"15min":1.867},"min":{"1min":0.42299999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":5.2649999999999997,"5min":5.3940000000000001,"15min":6.383},"last":4.0190000000000001}]},{"osd":6,"last update":"Fri Feb 9 17:48:55 2024","interfaces":[{"interface":"back","average":{"1min":1.919,"5min":1.6180000000000001,"15min":1.4199999999999999},"min":{"1min":0.36199999999999999,"5min":0.29899999999999999,"15min":0.29799999999999999},"max":{"1min":5.8179999999999996,"5min":13.031000000000001,"15min":13.031000000000001},"last":1.0329999999999999},{"interface":"front","average":{"1min":1.9239999999999999,"5min":1.5940000000000001,"15min":1.444},"min":{"1min":0.34200000000000003,"5min":0.32000000000000001,"15min":0.29599999999999999},"max":{"1min":7.016,"5min":12.157999999999999,"15min":12.157999999999999},"last":4.0350000000000001}]},{"osd":7,"last update":"Fri Feb 9 17:48:55 2024","interfaces":[{"interface":"back","average":{"1min":1.865,"5min":1.784,"15min":1.647},"min":{"1min":0.377,"5min":0.34100000000000003,"15min":0.32800000000000001},"max":{"1min":6.766,"5min":11.278,"15min":11.278},"last":0.63200000000000001},{"interface":"front","average":{"1min":2.3839999999999999,"5min":1.944,"15min":1.7250000000000001},"min":{"1min":0.42599999999999999,"5min":0.38700000000000001,"15min":0.38500000000000001},"max":{"1min":7.7709999999999999,"5min":11.313000000000001,"15min":11.313000000000001},"last":8.0449999999999999}]},{"osd":9,"last update":"Fri Feb 9 17:48:55 2024","interfaces":[{"interface":"back","average":{"1min":1.4710000000000001,"5min":0.98899999999999999,"15min":1.2310000000000001},"min":{"1min":0.38800000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":5.6529999999999996,"5min":5.6529999999999996,"15min":45.616},"last":0.98199999999999998},{"interface":"front","average":{"1min":1.579,"5min":1.1259999999999999,"15min":1.363},"min":{"1min":0.504,"5min":0.374,"15min":0.314},"max":{"1min":5.694,"5min":7.5609999999999999,"15min":45.718000000000004},"last":1.0900000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:48:55 2024","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":0.90700000000000003,"15min":0.95299999999999996},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.29699999999999999},"max":{"1min":4.3380000000000001,"5min":7.8360000000000003,"15min":7.8360000000000003},"last":0.53300000000000003},{"interface":"front","average":{"1min":1.2909999999999999,"5min":0.875,"15min":1.1579999999999999},"min":{"1min":0.40600000000000003,"5min":0.34300000000000003,"15min":0.318},"max":{"1min":4.5270000000000001,"5min":4.9690000000000003,"15min":11.071},"last":1.006}]},{"osd":11,"last update":"Fri Feb 9 17:48:55 2024","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.0900000000000001,"15min":1.2010000000000001},"min":{"1min":0.47099999999999997,"5min":0.40899999999999997,"15min":0.34000000000000002},"max":{"1min":3.7149999999999999,"5min":7.6079999999999997,"15min":14.433999999999999},"last":1.0600000000000001},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.1020000000000001,"15min":1.2330000000000001},"min":{"1min":0.39800000000000002,"5min":0.32900000000000001,"15min":0.308},"max":{"1min":6.5720000000000001,"5min":6.5720000000000001,"15min":7.6200000000000001},"last":0.60399999999999998}]}]},{"osd":7,"up_from":57,"seq":244813136218,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1111328,"kb_used_data":435992,"kb_used_omap":1,"kb_used_meta":675326,"kb_avail":92633824,"statfs":{"total":95995035648,"available":94857035776,"internally_reserved":0,"allocated":446455808,"data_stored":446168075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":691534282},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:48:34 2024","interfaces":[{"interface":"back","average":{"1min":2.0579999999999998,"5min":1.7150000000000001,"15min":1.7669999999999999},"min":{"1min":0.32900000000000001,"5min":0.27000000000000002,"15min":0.252},"max":{"1min":9.6470000000000002,"5min":11.047000000000001,"15min":36.753},"last":0.378},{"interface":"front","average":{"1min":2.0449999999999999,"5min":1.7569999999999999,"15min":1.8779999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.24099999999999999},"max":{"1min":9.5960000000000001,"5min":11.095000000000001,"15min":36.918999999999997},"last":0.36299999999999999}]},{"osd":1,"last update":"Fri Feb 9 17:48:34 2024","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":1.8360000000000001,"15min":2.008},"min":{"1min":0.32200000000000001,"5min":0.26200000000000001,"15min":0.24199999999999999},"max":{"1min":6.7249999999999996,"5min":12.318,"15min":21.704999999999998},"last":2.0070000000000001},{"interface":"front","average":{"1min":1.8640000000000001,"5min":2.202,"15min":2.1349999999999998},"min":{"1min":0.26600000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":7.4610000000000003,"5min":12.359,"15min":21.858000000000001},"last":0.31900000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:48:34 2024","interfaces":[{"interface":"back","average":{"1min":1.7949999999999999,"5min":1.732,"15min":1.9139999999999999},"min":{"1min":0.35999999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":6.4850000000000003,"5min":6.6340000000000003,"15min":6.6340000000000003},"last":0.5},{"interface":"front","average":{"1min":2.3879999999999999,"5min":1.8640000000000001,"15min":2.266},"min":{"1min":0.34699999999999998,"5min":0.308,"15min":0.308},"max":{"1min":8.0399999999999991,"5min":11.968,"15min":11.968},"last":2.0339999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:48:55 2024","interfaces":[{"interface":"back","average":{"1min":6.9939999999999998,"5min":2.7989999999999999,"15min":2.161},"min":{"1min":0.39300000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":78.644999999999996,"5min":78.644999999999996,"15min":78.644999999999996},"last":7.0629999999999997},{"interface":"front","average":{"1min":6.1369999999999996,"5min":3.1040000000000001,"15min":2.3450000000000002},"min":{"1min":0.38300000000000001,"5min":0.35499999999999998,"15min":0.308},"max":{"1min":78.414000000000001,"5min":78.414000000000001,"15min":78.414000000000001},"last":13.114000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:48:34 2024","interfaces":[{"interface":"back","average":{"1min":1.6990000000000001,"5min":2.4340000000000002,"15min":1.403},"min":{"1min":0.33300000000000002,"5min":0.25600000000000001,"15min":0.23699999999999999},"max":{"1min":6.157,"5min":86.813000000000002,"15min":86.813000000000002},"last":0.51000000000000001},{"interface":"front","average":{"1min":1.804,"5min":2.3319999999999999,"15min":1.4379999999999999},"min":{"1min":0.36299999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":7.7450000000000001,"5min":86.751000000000005,"15min":86.751000000000005},"last":0.35099999999999998}]},{"osd":5,"last update":"Fri Feb 9 17:48:34 2024","interfaces":[{"interface":"back","average":{"1min":2.2170000000000001,"5min":1.915,"15min":1.3220000000000001},"min":{"1min":0.42099999999999999,"5min":0.33100000000000002,"15min":0.249},"max":{"1min":8.0060000000000002,"5min":9.2170000000000005,"15min":9.6590000000000007},"last":0.30199999999999999},{"interface":"front","average":{"1min":1.383,"5min":2.0030000000000001,"15min":1.339},"min":{"1min":0.38800000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":3.2189999999999999,"5min":8.9600000000000009,"15min":15.228999999999999},"last":0.53400000000000003}]},{"osd":6,"last update":"Fri Feb 9 17:48:55 2024","interfaces":[{"interface":"back","average":{"1min":1.722,"5min":1.9339999999999999,"15min":1.8560000000000001},"min":{"1min":0.47999999999999998,"5min":0.35199999999999998,"15min":0.26000000000000001},"max":{"1min":7.8879999999999999,"5min":8.9969999999999999,"15min":15.275},"last":0.45800000000000002},{"interface":"front","average":{"1min":1.8640000000000001,"5min":1.788,"15min":1.7190000000000001},"min":{"1min":0.51900000000000002,"5min":0.32000000000000001,"15min":0.29199999999999998},"max":{"1min":5.9660000000000002,"5min":8.9779999999999998,"15min":18.236999999999998},"last":0.40200000000000002}]},{"osd":8,"last update":"Fri Feb 9 17:48:55 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":1.395,"15min":1.454},"min":{"1min":0.375,"5min":0.375,"15min":0.28999999999999998},"max":{"1min":6.7080000000000002,"5min":7.8120000000000003,"15min":11.81},"last":0.629},{"interface":"front","average":{"1min":1.131,"5min":1.5640000000000001,"15min":1.4690000000000001},"min":{"1min":0.433,"5min":0.38700000000000001,"15min":0.33300000000000002},"max":{"1min":7.7709999999999999,"5min":7.8949999999999996,"15min":12.058},"last":0.60999999999999999}]},{"osd":9,"last update":"Fri Feb 9 17:48:40 2024","interfaces":[{"interface":"back","average":{"1min":1.7509999999999999,"5min":1.55,"15min":1.534},"min":{"1min":0.375,"5min":0.375,"15min":0.32700000000000001},"max":{"1min":7.9020000000000001,"5min":8.0020000000000007,"15min":11.941000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":1.8400000000000001,"5min":1.6899999999999999,"15min":1.639},"min":{"1min":0.436,"5min":0.436,"15min":0.33400000000000002},"max":{"1min":7.8410000000000002,"5min":8.109,"15min":11.906000000000001},"last":0.66300000000000003}]},{"osd":10,"last update":"Fri Feb 9 17:49:08 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.3520000000000001,"15min":1.4910000000000001},"min":{"1min":0.53200000000000003,"5min":0.38700000000000001,"15min":0.35099999999999998},"max":{"1min":3.8799999999999999,"5min":7.165,"15min":12.050000000000001},"last":0.43099999999999999},{"interface":"front","average":{"1min":1.639,"5min":1.5900000000000001,"15min":1.6379999999999999},"min":{"1min":0.46800000000000003,"5min":0.41999999999999998,"15min":0.38900000000000001},"max":{"1min":6.734,"5min":7.8739999999999997,"15min":11.878},"last":1.625}]},{"osd":11,"last update":"Fri Feb 9 17:48:34 2024","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.7809999999999999,"15min":1.8089999999999999},"min":{"1min":0.48199999999999998,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":5.5620000000000003,"5min":9.984,"15min":21.945},"last":0.48499999999999999},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.778,"15min":1.7529999999999999},"min":{"1min":0.54200000000000004,"5min":0.438,"15min":0.40699999999999997},"max":{"1min":3.1970000000000001,"5min":12.423999999999999,"15min":21.731999999999999},"last":0.56499999999999995}]}]},{"osd":6,"up_from":49,"seq":210453397855,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":937560,"kb_used_data":369104,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":92807592,"statfs":{"total":95995035648,"available":95034974208,"internally_reserved":0,"allocated":377962496,"data_stored":377678315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":582089158},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:49:25 2024","interfaces":[{"interface":"back","average":{"1min":2.2519999999999998,"5min":1.585,"15min":1.7150000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.21299999999999999},"max":{"1min":10.481999999999999,"5min":10.481999999999999,"15min":12.544},"last":0.371},{"interface":"front","average":{"1min":2.1299999999999999,"5min":1.706,"15min":1.7250000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":5.9050000000000002,"5min":7.0860000000000003,"15min":9.2520000000000007},"last":3.976}]},{"osd":1,"last update":"Fri Feb 9 17:49:25 2024","interfaces":[{"interface":"back","average":{"1min":3.3519999999999999,"5min":3.7309999999999999,"15min":2.6320000000000001},"min":{"1min":0.28499999999999998,"5min":0.252,"15min":0.252},"max":{"1min":11.090999999999999,"5min":79.945999999999998,"15min":79.945999999999998},"last":4.1200000000000001},{"interface":"front","average":{"1min":3.383,"5min":3.8149999999999999,"15min":2.7130000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.29899999999999999},"max":{"1min":11.132999999999999,"5min":80.003,"15min":80.003},"last":4.1429999999999998}]},{"osd":2,"last update":"Fri Feb 9 17:49:25 2024","interfaces":[{"interface":"back","average":{"1min":2.2869999999999999,"5min":3.1640000000000001,"15min":2.117},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":5.3070000000000004,"5min":84.912999999999997,"15min":84.912999999999997},"last":0.40500000000000003},{"interface":"front","average":{"1min":2.3220000000000001,"5min":3.1949999999999998,"15min":2.222},"min":{"1min":0.35499999999999998,"5min":0.34000000000000002,"15min":0.315},"max":{"1min":5.3440000000000003,"5min":84.942999999999998,"15min":84.942999999999998},"last":0.436}]},{"osd":3,"last update":"Fri Feb 9 17:49:25 2024","interfaces":[{"interface":"back","average":{"1min":4.7140000000000004,"5min":4.1890000000000001,"15min":2.6949999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":19.555,"5min":80.174999999999997,"15min":80.174999999999997},"last":2.2970000000000002},{"interface":"front","average":{"1min":5.6909999999999998,"5min":4.7679999999999998,"15min":2.887},"min":{"1min":0.41299999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":19.568000000000001,"5min":80.417000000000002,"15min":80.417000000000002},"last":2.3260000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:49:25 2024","interfaces":[{"interface":"back","average":{"1min":3.54,"5min":2.262,"15min":1.909},"min":{"1min":0.36499999999999999,"5min":0.27900000000000003,"15min":0.23599999999999999},"max":{"1min":14.182,"5min":14.182,"15min":19.798999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":3.5899999999999999,"5min":2.1309999999999998,"15min":1.9510000000000001},"min":{"1min":0.39800000000000002,"5min":0.29299999999999998,"15min":0.26400000000000001},"max":{"1min":14.210000000000001,"5min":14.210000000000001,"15min":15.118},"last":0.59299999999999997}]},{"osd":5,"last update":"Fri Feb 9 17:49:25 2024","interfaces":[{"interface":"back","average":{"1min":3.6760000000000002,"5min":2.2749999999999999,"15min":1.9039999999999999},"min":{"1min":0.374,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":14.151999999999999,"5min":14.151999999999999,"15min":14.151999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":2.1219999999999999,"5min":2.351,"15min":1.8580000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":7.6630000000000003,"5min":18.315999999999999,"15min":18.315999999999999},"last":0.58099999999999996}]},{"osd":7,"last update":"Fri Feb 9 17:49:00 2024","interfaces":[{"interface":"back","average":{"1min":1.6259999999999999,"5min":2.0470000000000002,"15min":1.9019999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":5.4139999999999997,"5min":23.265999999999998,"15min":23.265999999999998},"last":0.38500000000000001},{"interface":"front","average":{"1min":1.8169999999999999,"5min":2.0059999999999998,"15min":1.877},"min":{"1min":0.35299999999999998,"5min":0.314,"15min":0.314},"max":{"1min":5.3049999999999997,"5min":10.599,"15min":14.356999999999999},"last":0.626}]},{"osd":8,"last update":"Fri Feb 9 17:26:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":0.876,"15min":0.79600000000000004},"min":{"1min":0.39900000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":4.0339999999999998,"5min":4.4880000000000004,"15min":4.4880000000000004},"last":4.2809999999999997},{"interface":"front","average":{"1min":1.0469999999999999,"5min":0.88600000000000001,"15min":0.78300000000000003},"min":{"1min":0.375,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":3.6539999999999999,"5min":4.8570000000000002,"15min":4.8570000000000002},"last":2.9529999999999998}]},{"osd":9,"last update":"Fri Feb 9 17:49:30 2024","interfaces":[{"interface":"back","average":{"1min":1.5369999999999999,"5min":1.401,"15min":1.524},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":5.2569999999999997,"5min":8.3759999999999994,"15min":10.779999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":1.569,"5min":1.4359999999999999,"15min":1.6000000000000001},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":5.1879999999999997,"5min":8.4130000000000003,"15min":8.4130000000000003},"last":0.67200000000000004}]},{"osd":10,"last update":"Fri Feb 9 17:49:00 2024","interfaces":[{"interface":"back","average":{"1min":1.621,"5min":1.5640000000000001,"15min":1.643},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34300000000000003},"max":{"1min":5.4829999999999997,"5min":8.7289999999999992,"15min":25.370000000000001},"last":0.42099999999999999},{"interface":"front","average":{"1min":2.5310000000000001,"5min":1.917,"15min":1.7569999999999999},"min":{"1min":0.45400000000000001,"5min":0.36399999999999999,"15min":0.35099999999999998},"max":{"1min":10.239000000000001,"5min":11.821999999999999,"15min":26.027999999999999},"last":0.75700000000000001}]},{"osd":11,"last update":"Fri Feb 9 17:49:25 2024","interfaces":[{"interface":"back","average":{"1min":1.903,"5min":1.6040000000000001,"15min":1.619},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":5.5110000000000001,"5min":8.7650000000000006,"15min":13.109},"last":0.46200000000000002},{"interface":"front","average":{"1min":1.978,"5min":1.7010000000000001,"15min":1.609},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.373},"max":{"1min":5.5410000000000004,"5min":8.7889999999999997,"15min":13.164999999999999},"last":0.48499999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575226,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1018012,"kb_used_data":328788,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":92727140,"statfs":{"total":95995035648,"available":94952591360,"internally_reserved":0,"allocated":336678912,"data_stored":336390659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:48:42 2024","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.538,"15min":1.714},"min":{"1min":0.222,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":4.7329999999999997,"5min":9.8529999999999998,"15min":19.079000000000001},"last":0.36899999999999999},{"interface":"front","average":{"1min":2.7330000000000001,"5min":1.827,"15min":1.6930000000000001},"min":{"1min":0.26000000000000001,"5min":0.23699999999999999,"15min":0.19400000000000001},"max":{"1min":11.52,"5min":11.52,"15min":23.573},"last":1.8}]},{"osd":2,"last update":"Fri Feb 9 17:48:59 2024","interfaces":[{"interface":"back","average":{"1min":2.3180000000000001,"5min":2.0390000000000001,"15min":1.7869999999999999},"min":{"1min":0.26100000000000001,"5min":0.24399999999999999,"15min":0.23100000000000001},"max":{"1min":11.301,"5min":11.301,"15min":26.361999999999998},"last":0.59699999999999998},{"interface":"front","average":{"1min":2.3740000000000001,"5min":1.732,"15min":1.8360000000000001},"min":{"1min":0.31,"5min":0.222,"15min":0.222},"max":{"1min":8.9049999999999994,"5min":8.9049999999999994,"15min":26.087},"last":0.61099999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:49:32 2024","interfaces":[{"interface":"back","average":{"1min":4.4530000000000003,"5min":2.5289999999999999,"15min":1.875},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":16.125,"5min":17.959,"15min":17.959},"last":0.35299999999999998},{"interface":"front","average":{"1min":4.6349999999999998,"5min":3.1040000000000001,"15min":1.7090000000000001},"min":{"1min":0.38600000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":19.689,"5min":22.866,"15min":22.866},"last":0.39100000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:48:38 2024","interfaces":[{"interface":"back","average":{"1min":1.6870000000000001,"5min":2.0680000000000001,"15min":1.833},"min":{"1min":0.33800000000000002,"5min":0.30299999999999999,"15min":0.253},"max":{"1min":7.6699999999999999,"5min":45.137999999999998,"15min":45.137999999999998},"last":0.47699999999999998},{"interface":"front","average":{"1min":1.587,"5min":2.298,"15min":1.996},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.28999999999999998},"max":{"1min":5.1059999999999999,"5min":44.950000000000003,"15min":44.950000000000003},"last":0.442}]},{"osd":5,"last update":"Fri Feb 9 17:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":2.1339999999999999,"15min":1.9379999999999999},"min":{"1min":0.35599999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":4.7590000000000003,"5min":45.171999999999997,"15min":45.171999999999997},"last":0.46600000000000003},{"interface":"front","average":{"1min":1.6919999999999999,"5min":2.355,"15min":2.048},"min":{"1min":0.31900000000000001,"5min":0.308,"15min":0.26800000000000002},"max":{"1min":11.763,"5min":45.127000000000002,"15min":45.127000000000002},"last":1.8320000000000001}]},{"osd":6,"last update":"Fri Feb 9 17:49:32 2024","interfaces":[{"interface":"back","average":{"1min":2.379,"5min":2.0739999999999998,"15min":1.643},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":11.808,"5min":11.808,"15min":12.57},"last":1.8660000000000001},{"interface":"front","average":{"1min":2.1829999999999998,"5min":2.2130000000000001,"15min":1.651},"min":{"1min":0.40000000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":11.932,"5min":11.932,"15min":12.605},"last":5.4429999999999996}]},{"osd":7,"last update":"Fri Feb 9 17:49:32 2024","interfaces":[{"interface":"back","average":{"1min":2.093,"5min":2.6139999999999999,"15min":1.7210000000000001},"min":{"1min":0.45900000000000002,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":11.834,"5min":45.740000000000002,"15min":45.740000000000002},"last":1.9419999999999999},{"interface":"front","average":{"1min":2.1400000000000001,"5min":2.6459999999999999,"15min":1.8120000000000001},"min":{"1min":0.48799999999999999,"5min":0.39600000000000002,"15min":0.309},"max":{"1min":11.856999999999999,"5min":45.792000000000002,"15min":45.792000000000002},"last":1.966}]},{"osd":8,"last update":"Fri Feb 9 17:48:38 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.546,"15min":1.115},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":4.6520000000000001,"5min":8.9440000000000008,"15min":9.2520000000000007},"last":0.56499999999999995},{"interface":"front","average":{"1min":1.859,"5min":1.5580000000000001,"15min":1.103},"min":{"1min":0.40300000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":7.7809999999999997,"5min":7.7809999999999997,"15min":9.2810000000000006},"last":0.624}]},{"osd":9,"last update":"Fri Feb 9 17:48:42 2024","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.6259999999999999,"15min":1.5760000000000001},"min":{"1min":0.39500000000000002,"5min":0.39200000000000002,"15min":0.33400000000000002},"max":{"1min":4.6219999999999999,"5min":9.9359999999999999,"15min":9.9359999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.7370000000000001,"15min":1.75},"min":{"1min":0.47099999999999997,"5min":0.34899999999999998,"15min":0.32900000000000001},"max":{"1min":4.4779999999999998,"5min":7.7199999999999998,"15min":11.366},"last":0.498}]},{"osd":10,"last update":"Fri Feb 9 17:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.8620000000000001,"5min":1.885,"15min":1.8089999999999999},"min":{"1min":0.46899999999999997,"5min":0.432,"15min":0.39200000000000002},"max":{"1min":11.880000000000001,"5min":11.880000000000001,"15min":15.083},"last":1.8899999999999999},{"interface":"front","average":{"1min":1.423,"5min":1.7729999999999999,"15min":1.806},"min":{"1min":0.498,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":4.8899999999999997,"5min":10.09,"15min":15.121},"last":0.55400000000000005}]},{"osd":11,"last update":"Fri Feb 9 17:49:32 2024","interfaces":[{"interface":"back","average":{"1min":2.6179999999999999,"5min":1.8340000000000001,"15min":1.3480000000000001},"min":{"1min":0.46800000000000003,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":16.184999999999999,"5min":16.184999999999999,"15min":16.184999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":2.109,"5min":2.0099999999999998,"15min":1.486},"min":{"1min":0.505,"5min":0.45600000000000002,"15min":0.44800000000000001},"max":{"1min":11.904,"5min":11.904,"15min":11.904},"last":1.9139999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738750,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":931608,"kb_used_data":254672,"kb_used_omap":1,"kb_used_meta":676734,"kb_avail":92813544,"statfs":{"total":95995035648,"available":95041069056,"internally_reserved":0,"allocated":260784128,"data_stored":260499947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":692976069},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 9 17:49:28 2024","interfaces":[{"interface":"back","average":{"1min":1.746,"5min":1.8069999999999999,"15min":1.639},"min":{"1min":0.25900000000000001,"5min":0.219,"15min":0.20599999999999999},"max":{"1min":8.7469999999999999,"5min":10.938000000000001,"15min":13.842000000000001},"last":0.32200000000000001},{"interface":"front","average":{"1min":1.8109999999999999,"5min":1.8700000000000001,"15min":1.8049999999999999},"min":{"1min":0.308,"5min":0.29299999999999998,"15min":0.24399999999999999},"max":{"1min":8.7919999999999998,"5min":10.974,"15min":14.122},"last":0.41499999999999998}]},{"osd":2,"last update":"Fri Feb 9 17:39:25 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":5.5339999999999998,"5min":5.5339999999999998,"15min":5.5339999999999998},"last":1.738},{"interface":"front","average":{"1min":1.593,"5min":1.593,"15min":1.593},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":13.473000000000001,"5min":13.473000000000001,"15min":13.473000000000001},"last":0.49299999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:49:08 2024","interfaces":[{"interface":"back","average":{"1min":1.968,"5min":1.7849999999999999,"15min":2.1640000000000001},"min":{"1min":0.313,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":7.593,"5min":36.052,"15min":36.052},"last":1.1850000000000001},{"interface":"front","average":{"1min":1.6399999999999999,"5min":2.3340000000000001,"15min":1.8009999999999999},"min":{"1min":0.28899999999999998,"5min":0.24099999999999999,"15min":0.217},"max":{"1min":6.5590000000000002,"5min":38.383000000000003,"15min":38.383000000000003},"last":3.0990000000000002}]},{"osd":4,"last update":"Fri Feb 9 17:48:54 2024","interfaces":[{"interface":"back","average":{"1min":1.706,"5min":1.5409999999999999,"15min":1.47},"min":{"1min":0.29999999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":8.9420000000000002,"5min":8.9420000000000002,"15min":11.212},"last":0.61399999999999999},{"interface":"front","average":{"1min":1.9199999999999999,"5min":1.944,"15min":1.8169999999999999},"min":{"1min":0.36399999999999999,"5min":0.32300000000000001,"15min":0.27200000000000002},"max":{"1min":8.9580000000000002,"5min":16.916,"15min":16.916},"last":0.76600000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:49:08 2024","interfaces":[{"interface":"back","average":{"1min":1.575,"5min":2.4900000000000002,"15min":1.887},"min":{"1min":0.40899999999999997,"5min":0.30299999999999999,"15min":0.29499999999999998},"max":{"1min":4.7279999999999998,"5min":48.924999999999997,"15min":48.924999999999997},"last":2.923},{"interface":"front","average":{"1min":1.6279999999999999,"5min":2.5339999999999998,"15min":1.9119999999999999},"min":{"1min":0.44700000000000001,"5min":0.374,"15min":0.29599999999999999},"max":{"1min":4.7560000000000002,"5min":48.969999999999999,"15min":48.969999999999999},"last":2.9590000000000001}]},{"osd":6,"last update":"Fri Feb 9 17:48:34 2024","interfaces":[{"interface":"back","average":{"1min":2.0449999999999999,"5min":1.839,"15min":1.7949999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.28699999999999998},"max":{"1min":5.8120000000000003,"5min":11.912000000000001,"15min":13.407},"last":0.69399999999999995},{"interface":"front","average":{"1min":2.1000000000000001,"5min":1.921,"15min":1.8620000000000001},"min":{"1min":0.40000000000000002,"5min":0.375,"15min":0.33300000000000002},"max":{"1min":5.8490000000000002,"5min":11.939,"15min":13.448},"last":0.72699999999999998}]},{"osd":7,"last update":"Fri Feb 9 17:49:00 2024","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":1.8480000000000001,"15min":2.0670000000000002},"min":{"1min":0.35899999999999999,"5min":0.33800000000000002,"15min":0.30299999999999999},"max":{"1min":4.4409999999999998,"5min":15.271000000000001,"15min":15.271000000000001},"last":0.65900000000000003},{"interface":"front","average":{"1min":2.2869999999999999,"5min":2.0550000000000002,"15min":2.004},"min":{"1min":0.45600000000000002,"5min":0.34699999999999998,"15min":0.28699999999999998},"max":{"1min":7.7699999999999996,"5min":11.614000000000001,"15min":14.451000000000001},"last":0.80700000000000005}]},{"osd":8,"last update":"Fri Feb 9 17:49:00 2024","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":1.3819999999999999,"15min":1.583},"min":{"1min":0.41899999999999998,"5min":0.36199999999999999,"15min":0.33100000000000002},"max":{"1min":2.4649999999999999,"5min":11.641999999999999,"15min":14.323},"last":1.8959999999999999},{"interface":"front","average":{"1min":0.879,"5min":1.3520000000000001,"15min":1.671},"min":{"1min":0.48399999999999999,"5min":0.36299999999999999,"15min":0.35099999999999998},"max":{"1min":2.0659999999999998,"5min":11.669,"15min":14.481999999999999},"last":0.84599999999999997}]},{"osd":9,"last update":"Fri Feb 9 17:48:48 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.8009999999999999,"15min":1.966},"min":{"1min":0.53000000000000003,"5min":0.40500000000000003,"15min":0.32500000000000001},"max":{"1min":3.165,"5min":16.936,"15min":17.952000000000002},"last":0.87},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.6559999999999999,"15min":1.891},"min":{"1min":0.56200000000000006,"5min":0.42399999999999999,"15min":0.34799999999999998},"max":{"1min":3.1880000000000002,"5min":16.951000000000001,"15min":17.913},"last":0.92100000000000004}]},{"osd":10,"last update":"Fri Feb 9 17:49:08 2024","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.702,"15min":1.929},"min":{"1min":0.47899999999999998,"5min":0.443,"15min":0.35499999999999998},"max":{"1min":5.9509999999999996,"5min":11.692,"15min":31.766999999999999},"last":0.89600000000000002},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.5089999999999999,"15min":1.4930000000000001},"min":{"1min":0.42299999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":4.2030000000000003,"5min":13.148999999999999,"15min":28.048999999999999},"last":1.643}]},{"osd":11,"last update":"Fri Feb 9 17:48:34 2024","interfaces":[{"interface":"back","average":{"1min":1.74,"5min":1.734,"15min":1.734},"min":{"1min":0.52800000000000002,"5min":0.41599999999999998,"15min":0.40600000000000003},"max":{"1min":8.2650000000000006,"5min":13.102,"15min":40.021000000000001},"last":1.681},{"interface":"front","average":{"1min":1.7490000000000001,"5min":1.8080000000000001,"15min":1.7989999999999999},"min":{"1min":0.496,"5min":0.45500000000000002,"15min":0.45200000000000001},"max":{"1min":8.3420000000000005,"5min":9.5820000000000007,"15min":40.055999999999997},"last":1.708}]}]},{"osd":2,"up_from":18,"seq":77309411699,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":760672,"kb_used_data":382424,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":92984480,"statfs":{"total":95995035648,"available":95216107520,"internally_reserved":0,"allocated":391602176,"data_stored":391314443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":387316165},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":1.4510000000000001,"15min":1.006},"min":{"1min":0.29999999999999999,"5min":0.23699999999999999,"15min":0.221},"max":{"1min":4.9089999999999998,"5min":17.251000000000001,"15min":17.251000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.74399999999999999,"5min":1.611,"15min":1.113},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.22900000000000001},"max":{"1min":4.7720000000000002,"5min":23.097999999999999,"15min":23.097999999999999},"last":0.875}]},{"osd":1,"last update":"Fri Feb 9 17:49:02 2024","interfaces":[{"interface":"back","average":{"1min":3.2589999999999999,"5min":2.3759999999999999,"15min":2.0299999999999998},"min":{"1min":0.27700000000000002,"5min":0.214,"15min":0.214},"max":{"1min":23.044,"5min":52.646000000000001,"15min":52.646000000000001},"last":0.33100000000000002},{"interface":"front","average":{"1min":2.2330000000000001,"5min":2.181,"15min":2.0059999999999998},"min":{"1min":0.32100000000000001,"5min":0.248,"15min":0.214},"max":{"1min":12.368,"5min":52.683999999999997,"15min":52.683999999999997},"last":2.0950000000000002}]},{"osd":3,"last update":"Fri Feb 9 17:49:26 2024","interfaces":[{"interface":"back","average":{"1min":2.8919999999999999,"5min":2.3639999999999999,"15min":1.9530000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.249},"max":{"1min":8.7249999999999996,"5min":22.698,"15min":24.353000000000002},"last":0.44800000000000001},{"interface":"front","average":{"1min":4.1779999999999999,"5min":2.6179999999999999,"15min":2.222},"min":{"1min":0.31,"5min":0.221,"15min":0.20699999999999999},"max":{"1min":15.414,"5min":19.904,"15min":19.986999999999998},"last":0.54700000000000004}]},{"osd":4,"last update":"Fri Feb 9 17:49:26 2024","interfaces":[{"interface":"back","average":{"1min":2.0760000000000001,"5min":1.97,"15min":1.9059999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.26300000000000001},"max":{"1min":8.3580000000000005,"5min":12.398,"15min":13.526},"last":0.53400000000000003},{"interface":"front","average":{"1min":1.431,"5min":1.917,"15min":1.8},"min":{"1min":0.29899999999999999,"5min":0.27900000000000003,"15min":0.27300000000000002},"max":{"1min":6.452,"5min":10.888999999999999,"15min":10.888999999999999},"last":0.69099999999999995}]},{"osd":5,"last update":"Fri Feb 9 17:49:19 2024","interfaces":[{"interface":"back","average":{"1min":2.1219999999999999,"5min":2.702,"15min":2.1339999999999999},"min":{"1min":0.29699999999999999,"5min":0.28499999999999998,"15min":0.28299999999999997},"max":{"1min":6.5460000000000003,"5min":35.436999999999998,"15min":42.664000000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":2.5859999999999999,"5min":2.9849999999999999,"15min":2.4329999999999998},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":9.3390000000000004,"5min":35.484000000000002,"15min":43.311999999999998},"last":0.66900000000000004}]},{"osd":6,"last update":"Fri Feb 9 17:49:26 2024","interfaces":[{"interface":"back","average":{"1min":2.3530000000000002,"5min":2.3140000000000001,"15min":2.0499999999999998},"min":{"1min":0.34100000000000003,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":6.5960000000000001,"5min":11.092000000000001,"15min":11.092000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":1.9259999999999999,"5min":2.1840000000000002,"15min":2.1280000000000001},"min":{"1min":0.39300000000000002,"5min":0.33100000000000002,"15min":0.32800000000000001},"max":{"1min":6.6180000000000003,"5min":11.019,"15min":11.269},"last":0.628}]},{"osd":7,"last update":"Fri Feb 9 17:49:02 2024","interfaces":[{"interface":"back","average":{"1min":2.3119999999999998,"5min":2.331,"15min":2.3010000000000002},"min":{"1min":0.38500000000000001,"5min":0.34699999999999998,"15min":0.32400000000000001},"max":{"1min":12.422000000000001,"5min":16.666,"15min":41.802999999999997},"last":0.504},{"interface":"front","average":{"1min":3.48,"5min":2.9950000000000001,"15min":2.4079999999999999},"min":{"1min":0.439,"5min":0.36599999999999999,"15min":0.30499999999999999},"max":{"1min":10.441000000000001,"5min":16.638999999999999,"15min":41.783999999999999},"last":0.64700000000000002}]},{"osd":8,"last update":"Fri Feb 9 17:49:26 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":1.677,"15min":1.5},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.316},"max":{"1min":8.3840000000000003,"5min":12.307,"15min":22.393999999999998},"last":0.39900000000000002},{"interface":"front","average":{"1min":1.589,"5min":1.744,"15min":1.6000000000000001},"min":{"1min":0.42499999999999999,"5min":0.36899999999999999,"15min":0.34399999999999997},"max":{"1min":6.4889999999999999,"5min":10.964,"15min":12.58},"last":0.56999999999999995}]},{"osd":9,"last update":"Fri Feb 9 17:48:36 2024","interfaces":[{"interface":"back","average":{"1min":2.5230000000000001,"5min":1.966,"15min":1.8109999999999999},"min":{"1min":0.40799999999999997,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":12.404999999999999,"5min":12.768000000000001,"15min":13.5},"last":0.76700000000000002},{"interface":"front","average":{"1min":2.6520000000000001,"5min":2.0470000000000002,"15min":1.845},"min":{"1min":0.371,"5min":0.371,"15min":0.36399999999999999},"max":{"1min":11.058999999999999,"5min":11.058999999999999,"15min":13.589},"last":0.52300000000000002}]},{"osd":10,"last update":"Fri Feb 9 17:49:16 2024","interfaces":[{"interface":"back","average":{"1min":2.6269999999999998,"5min":1.6040000000000001,"15min":1.6459999999999999},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.33600000000000002},"max":{"1min":12.314,"5min":12.314,"15min":22.425999999999998},"last":0.78100000000000003},{"interface":"front","average":{"1min":2.6549999999999998,"5min":1.6539999999999999,"15min":1.6850000000000001},"min":{"1min":0.505,"5min":0.501,"15min":0.379},"max":{"1min":12.342000000000001,"5min":12.342000000000001,"15min":22.454999999999998},"last":0.80300000000000005}]},{"osd":11,"last update":"Fri Feb 9 17:49:26 2024","interfaces":[{"interface":"back","average":{"1min":1.6970000000000001,"5min":2.1030000000000002,"15min":2.2959999999999998},"min":{"1min":0.42099999999999999,"5min":0.41999999999999998,"15min":0.39600000000000002},"max":{"1min":6.6399999999999997,"5min":11.137,"15min":137.44300000000001},"last":0.60899999999999999},{"interface":"front","average":{"1min":2.7650000000000001,"5min":2.0379999999999998,"15min":2.3359999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":9.9830000000000005,"5min":12.609999999999999,"15min":137.64400000000001},"last":0.81399999999999995}]}]},{"osd":3,"up_from":27,"seq":115964117360,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1275928,"kb_used_data":707344,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":92469224,"statfs":{"total":95995035648,"available":94688485376,"internally_reserved":0,"allocated":724320256,"data_stored":724036075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":582089159},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:49:23 2024","interfaces":[{"interface":"back","average":{"1min":3.1709999999999998,"5min":2.0600000000000001,"15min":1.8680000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.186},"max":{"1min":11.385999999999999,"5min":11.385999999999999,"15min":38.984999999999999},"last":4.5570000000000004},{"interface":"front","average":{"1min":3.419,"5min":2.7999999999999998,"15min":2.2210000000000001},"min":{"1min":0.28799999999999998,"5min":0.246,"15min":0.22500000000000001},"max":{"1min":11.079000000000001,"5min":11.506,"15min":41.497999999999998},"last":0.54400000000000004}]},{"osd":1,"last update":"Fri Feb 9 17:48:49 2024","interfaces":[{"interface":"back","average":{"1min":4.2939999999999996,"5min":3.0430000000000001,"15min":2.3929999999999998},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":16.337,"5min":16.337,"15min":16.337},"last":0.34499999999999997},{"interface":"front","average":{"1min":2.048,"5min":1.9099999999999999,"15min":1.4399999999999999},"min":{"1min":0.308,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":8.0630000000000006,"5min":20.138999999999999,"15min":20.138999999999999},"last":3.536}]},{"osd":2,"last update":"Fri Feb 9 17:49:19 2024","interfaces":[{"interface":"back","average":{"1min":2.7970000000000002,"5min":2.0249999999999999,"15min":1.7789999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":11.196999999999999,"5min":11.702999999999999,"15min":14.888999999999999},"last":4.5890000000000004},{"interface":"front","average":{"1min":2.1619999999999999,"5min":2.3100000000000001,"15min":1.909},"min":{"1min":0.34599999999999997,"5min":0.26100000000000001,"15min":0.20200000000000001},"max":{"1min":6.7169999999999996,"5min":19.879000000000001,"15min":19.879000000000001},"last":3.9140000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:49:23 2024","interfaces":[{"interface":"back","average":{"1min":3.5409999999999999,"5min":2.3439999999999999,"15min":1.8939999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.26800000000000002},"max":{"1min":11.117000000000001,"5min":11.117000000000001,"15min":34.898000000000003},"last":0.309},{"interface":"front","average":{"1min":1.706,"5min":2.0569999999999999,"15min":1.758},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.28000000000000003},"max":{"1min":7.9459999999999997,"5min":19.713999999999999,"15min":19.713999999999999},"last":3.3940000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:49:02 2024","interfaces":[{"interface":"back","average":{"1min":4.2590000000000003,"5min":2.4590000000000001,"15min":1.9570000000000001},"min":{"1min":0.46400000000000002,"5min":0.41099999999999998,"15min":0.27800000000000002},"max":{"1min":9.6010000000000009,"5min":13.470000000000001,"15min":17.651},"last":4.4550000000000001},{"interface":"front","average":{"1min":4.3120000000000003,"5min":2.2269999999999999,"15min":1.9299999999999999},"min":{"1min":0.48699999999999999,"5min":0.316,"15min":0.316},"max":{"1min":9.6389999999999993,"5min":9.6389999999999993,"15min":17.675999999999998},"last":4.4909999999999997}]},{"osd":6,"last update":"Fri Feb 9 17:48:49 2024","interfaces":[{"interface":"back","average":{"1min":3.964,"5min":1.964,"15min":2.2109999999999999},"min":{"1min":0.436,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":15.061999999999999,"5min":15.061999999999999,"15min":15.061999999999999},"last":4.5149999999999997},{"interface":"front","average":{"1min":3.3170000000000002,"5min":2.0169999999999999,"15min":1.649},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":9.032,"5min":11.813000000000001,"15min":11.813000000000001},"last":4.6120000000000001}]},{"osd":7,"last update":"Fri Feb 9 17:48:32 2024","interfaces":[{"interface":"back","average":{"1min":2.899,"5min":2.2509999999999999,"15min":1.484},"min":{"1min":0.40999999999999998,"5min":0.36499999999999999,"15min":0.34200000000000003},"max":{"1min":7.7649999999999997,"5min":19.788,"15min":19.788},"last":3.4369999999999998},{"interface":"front","average":{"1min":3.9350000000000001,"5min":2.379,"15min":1.347},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.34899999999999998},"max":{"1min":10.949999999999999,"5min":11.858000000000001,"15min":11.858000000000001},"last":4.6310000000000002}]},{"osd":8,"last update":"Fri Feb 9 17:45:16 2024","interfaces":[{"interface":"back","average":{"1min":1.714,"5min":1.4450000000000001,"15min":1.498},"min":{"1min":0.377,"5min":0.377,"15min":0.26800000000000002},"max":{"1min":8.6020000000000003,"5min":11.821,"15min":11.821},"last":0.67700000000000005},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.1719999999999999,"15min":1.351},"min":{"1min":0.46500000000000002,"5min":0.36199999999999999,"15min":0.34499999999999997},"max":{"1min":6.8390000000000004,"5min":6.8390000000000004,"15min":14.129},"last":0.78200000000000003}]},{"osd":9,"last update":"Fri Feb 9 17:48:44 2024","interfaces":[{"interface":"back","average":{"1min":1.9339999999999999,"5min":1.879,"15min":1.7010000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.32500000000000001},"max":{"1min":10.103,"5min":11.851000000000001,"15min":11.851000000000001},"last":0.42899999999999999},{"interface":"front","average":{"1min":2.1909999999999998,"5min":1.8879999999999999,"15min":1.6739999999999999},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":10.132999999999999,"5min":11.907,"15min":12.815},"last":0.46300000000000002}]},{"osd":10,"last update":"Fri Feb 9 17:48:32 2024","interfaces":[{"interface":"back","average":{"1min":2.3079999999999998,"5min":1.548,"15min":1.119},"min":{"1min":0.439,"5min":0.41099999999999998,"15min":0.39100000000000001},"max":{"1min":6.0949999999999998,"5min":9.7080000000000002,"15min":9.7080000000000002},"last":4.5300000000000002},{"interface":"front","average":{"1min":1.9410000000000001,"5min":1.8959999999999999,"15min":1.161},"min":{"1min":0.435,"5min":0.39100000000000001,"15min":0.38400000000000001},"max":{"1min":7.0430000000000001,"5min":19.859000000000002,"15min":19.859000000000002},"last":3.4609999999999999}]},{"osd":11,"last update":"Fri Feb 9 17:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.806,"5min":1.9119999999999999,"15min":1.6499999999999999},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.40300000000000002},"max":{"1min":5.3310000000000004,"5min":19.824000000000002,"15min":19.824000000000002},"last":3.488},{"interface":"front","average":{"1min":1.9330000000000001,"5min":1.474,"15min":1.7110000000000001},"min":{"1min":0.5,"5min":0.47499999999999998,"15min":0.442},"max":{"1min":5.9859999999999998,"5min":6.1219999999999999,"15min":17.145},"last":3.512}]}]},{"osd":4,"up_from":32,"seq":137438953834,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1022552,"kb_used_data":369424,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":92722600,"statfs":{"total":95995035648,"available":94947942400,"internally_reserved":0,"allocated":378290176,"data_stored":378005995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":668793285},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:48:31 2024","interfaces":[{"interface":"back","average":{"1min":3.1619999999999999,"5min":1.7769999999999999,"15min":1.8300000000000001},"min":{"1min":0.27900000000000003,"5min":0.22800000000000001,"15min":0.222},"max":{"1min":9.0139999999999993,"5min":10.301,"15min":50.356999999999999},"last":0.28299999999999997},{"interface":"front","average":{"1min":3.2149999999999999,"5min":1.8440000000000001,"15min":1.8879999999999999},"min":{"1min":0.37,"5min":0.30299999999999999,"15min":0.247},"max":{"1min":9.0510000000000002,"5min":10.407999999999999,"15min":50.569000000000003},"last":0.39400000000000002}]},{"osd":1,"last update":"Fri Feb 9 17:48:31 2024","interfaces":[{"interface":"back","average":{"1min":2.746,"5min":2.4209999999999998,"15min":2.2999999999999998},"min":{"1min":0.26900000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":8.3119999999999994,"5min":17.297999999999998,"15min":50.473999999999997},"last":0.376},{"interface":"front","average":{"1min":3.621,"5min":2.4990000000000001,"15min":2.4940000000000002},"min":{"1min":0.309,"5min":0.255,"15min":0.255},"max":{"1min":12.279999999999999,"5min":12.279999999999999,"15min":50.499000000000002},"last":0.72099999999999997}]},{"osd":2,"last update":"Fri Feb 9 17:48:50 2024","interfaces":[{"interface":"back","average":{"1min":2.198,"5min":2.4500000000000002,"15min":2.8780000000000001},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":8.5139999999999993,"5min":17.494,"15min":17.494},"last":0.35899999999999999},{"interface":"front","average":{"1min":2.8479999999999999,"5min":2.508,"15min":2.8260000000000001},"min":{"1min":0.36899999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":17.283999999999999,"5min":17.283999999999999,"15min":17.283999999999999},"last":0.49399999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:48:31 2024","interfaces":[{"interface":"back","average":{"1min":2.1509999999999998,"5min":1.881,"15min":1.9970000000000001},"min":{"1min":0.28499999999999998,"5min":0.28199999999999997,"15min":0.245},"max":{"1min":7.04,"5min":14.448,"15min":50.462000000000003},"last":0.66400000000000003},{"interface":"front","average":{"1min":2.3479999999999999,"5min":2.0680000000000001,"15min":1.917},"min":{"1min":0.33700000000000002,"5min":0.28199999999999997,"15min":0.255},"max":{"1min":4.835,"5min":10.382,"15min":50.529000000000003},"last":0.434}]},{"osd":5,"last update":"Fri Feb 9 17:48:59 2024","interfaces":[{"interface":"back","average":{"1min":2.0019999999999998,"5min":2.7450000000000001,"15min":1.98},"min":{"1min":0.255,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":6.9340000000000002,"5min":18.887,"15min":18.887},"last":0.68500000000000005},{"interface":"front","average":{"1min":2.0579999999999998,"5min":2.8100000000000001,"15min":2.2480000000000002},"min":{"1min":0.31,"5min":0.31,"15min":0.29099999999999998},"max":{"1min":7.0119999999999996,"5min":18.917000000000002,"15min":50.509999999999998},"last":0.753}]},{"osd":6,"last update":"Fri Feb 9 17:48:31 2024","interfaces":[{"interface":"back","average":{"1min":1.9219999999999999,"5min":2.0800000000000001,"15min":1.6739999999999999},"min":{"1min":0.29499999999999998,"5min":0.26300000000000001,"15min":0.246},"max":{"1min":11.257,"5min":18.234999999999999,"15min":18.234999999999999},"last":0.46899999999999997},{"interface":"front","average":{"1min":1.7010000000000001,"5min":1.97,"15min":2.0129999999999999},"min":{"1min":0.33800000000000002,"5min":0.31,"15min":0.245},"max":{"1min":5.7590000000000003,"5min":10.215,"15min":50.402999999999999},"last":1.5449999999999999}]},{"osd":7,"last update":"Fri Feb 9 17:49:09 2024","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.7270000000000001,"15min":1.635},"min":{"1min":0.35499999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":5.0529999999999999,"5min":10.185,"15min":11.875},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.88700000000000001,"5min":1.798,"15min":1.7789999999999999},"min":{"1min":0.39800000000000002,"5min":0.29999999999999999,"15min":0.28199999999999997},"max":{"1min":4.1050000000000004,"5min":10.547000000000001,"15min":12.696},"last":0.76800000000000002}]},{"osd":8,"last update":"Fri Feb 9 17:48:59 2024","interfaces":[{"interface":"back","average":{"1min":1.996,"5min":2.097,"15min":1.6919999999999999},"min":{"1min":0.42099999999999999,"5min":0.35299999999999998,"15min":0.33200000000000002},"max":{"1min":6.9790000000000001,"5min":17.335999999999999,"15min":17.335999999999999},"last":1.8560000000000001},{"interface":"front","average":{"1min":0.95999999999999996,"5min":1.278,"15min":1.5},"min":{"1min":0.44800000000000001,"5min":0.36699999999999999,"15min":0.34699999999999998},"max":{"1min":5.859,"5min":10.433,"15min":50.594000000000001},"last":0.73799999999999999}]},{"osd":9,"last update":"Fri Feb 9 17:44:39 2024","interfaces":[{"interface":"back","average":{"1min":2.5739999999999998,"5min":1.855,"15min":1.881},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":10.778,"5min":10.778,"15min":50.442999999999998},"last":0.73599999999999999},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.331,"15min":1.536},"min":{"1min":0.47699999999999998,"5min":0.42499999999999999,"15min":0.36199999999999999},"max":{"1min":5.165,"5min":8.3580000000000005,"15min":10.555},"last":4.8259999999999996}]},{"osd":10,"last update":"Fri Feb 9 17:49:14 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":1.524,"15min":1.98},"min":{"1min":0.42699999999999999,"5min":0.40500000000000003,"15min":0.39400000000000002},"max":{"1min":0.79700000000000004,"5min":17.367000000000001,"15min":51.223999999999997},"last":0.70899999999999996},{"interface":"front","average":{"1min":1.331,"5min":1.675,"15min":1.9730000000000001},"min":{"1min":0.443,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":5.1550000000000002,"5min":12.888,"15min":50.552},"last":0.88200000000000001}]},{"osd":11,"last update":"Fri Feb 9 17:48:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.381,"15min":1.8480000000000001},"min":{"1min":0.51500000000000001,"5min":0.44600000000000001,"15min":0.443},"max":{"1min":5.7960000000000003,"5min":10.521000000000001,"15min":50.484999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.8240000000000001,"15min":1.6060000000000001},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.40999999999999998},"max":{"1min":5.1310000000000002,"5min":13.154999999999999,"15min":13.154999999999999},"last":0.86399999999999999}]}]},{"osd":5,"up_from":40,"seq":171798692197,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1012388,"kb_used_data":403676,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":92732764,"statfs":{"total":95995035648,"available":94958350336,"internally_reserved":0,"allocated":413364224,"data_stored":413072419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":623311301},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:49:15 2024","interfaces":[{"interface":"back","average":{"1min":7.3319999999999999,"5min":2.8290000000000002,"15min":1.712},"min":{"1min":0.27600000000000002,"5min":0.24299999999999999,"15min":0.23599999999999999},"max":{"1min":116.77800000000001,"5min":116.77800000000001,"15min":116.77800000000001},"last":0.39800000000000002},{"interface":"front","average":{"1min":8.3539999999999992,"5min":3.0720000000000001,"15min":2.1850000000000001},"min":{"1min":0.29299999999999998,"5min":0.252,"15min":0.249},"max":{"1min":116.959,"5min":116.959,"15min":116.959},"last":0.29899999999999999}]},{"osd":1,"last update":"Fri Feb 9 17:49:04 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":1.9830000000000001,"15min":1.9690000000000001},"min":{"1min":0.26700000000000002,"5min":0.254,"15min":0.24199999999999999},"max":{"1min":8.4809999999999999,"5min":13.494999999999999,"15min":48.161999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":2.6269999999999998,"5min":2.052,"15min":2.077},"min":{"1min":0.35199999999999998,"5min":0.248,"15min":0.23599999999999999},"max":{"1min":15.097,"5min":16.626999999999999,"15min":48.218000000000004},"last":0.42199999999999999}]},{"osd":2,"last update":"Fri Feb 9 17:48:37 2024","interfaces":[{"interface":"back","average":{"1min":3.3330000000000002,"5min":2.4020000000000001,"15min":1.8180000000000001},"min":{"1min":0.35399999999999998,"5min":0.249,"15min":0.249},"max":{"1min":7.915,"5min":13.929,"15min":13.929},"last":0.318},{"interface":"front","average":{"1min":2.665,"5min":2.3010000000000002,"15min":2.1850000000000001},"min":{"1min":0.39700000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":12.028,"5min":15.945,"15min":15.945},"last":0.441}]},{"osd":3,"last update":"Fri Feb 9 17:49:04 2024","interfaces":[{"interface":"back","average":{"1min":3.456,"5min":2.5489999999999999,"15min":2.2850000000000001},"min":{"1min":0.40500000000000003,"5min":0.34699999999999998,"15min":0.26800000000000002},"max":{"1min":15.154999999999999,"5min":15.819000000000001,"15min":64.150999999999996},"last":0.502},{"interface":"front","average":{"1min":3.024,"5min":2.629,"15min":2.3650000000000002},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.247},"max":{"1min":8.7650000000000006,"5min":11.747,"15min":64.176000000000002},"last":0.45300000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:49:04 2024","interfaces":[{"interface":"back","average":{"1min":1.873,"5min":1.9690000000000001,"15min":1.8859999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.249},"max":{"1min":28.481999999999999,"5min":28.481999999999999,"15min":49.313000000000002},"last":0.34000000000000002},{"interface":"front","average":{"1min":2.3180000000000001,"5min":2.0619999999999998,"15min":1.8979999999999999},"min":{"1min":0.28499999999999998,"5min":0.26100000000000001,"15min":0.218},"max":{"1min":15.843,"5min":15.843,"15min":50.505000000000003},"last":0.55500000000000005}]},{"osd":6,"last update":"Fri Feb 9 17:49:33 2024","interfaces":[{"interface":"back","average":{"1min":1.954,"5min":1.911,"15min":1.7270000000000001},"min":{"1min":0.33000000000000002,"5min":0.27900000000000003,"15min":0.253},"max":{"1min":15.026,"5min":16.657,"15min":16.657},"last":0.38300000000000001},{"interface":"front","average":{"1min":2.3690000000000002,"5min":2.0939999999999999,"15min":1.8149999999999999},"min":{"1min":0.36499999999999999,"5min":0.32700000000000001,"15min":0.26200000000000001},"max":{"1min":20.192,"5min":20.192,"15min":20.192},"last":0.57299999999999995}]},{"osd":7,"last update":"Fri Feb 9 17:49:04 2024","interfaces":[{"interface":"back","average":{"1min":1.7989999999999999,"5min":2.0510000000000002,"15min":1.9119999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.26700000000000002},"max":{"1min":15.891999999999999,"5min":15.891999999999999,"15min":50.545000000000002},"last":0.47699999999999998},{"interface":"front","average":{"1min":3.5169999999999999,"5min":2.4569999999999999,"15min":2.4449999999999998},"min":{"1min":0.35899999999999999,"5min":0.31,"15min":0.31},"max":{"1min":12.326000000000001,"5min":13.869,"15min":102.526},"last":0.48299999999999998}]},{"osd":8,"last update":"Fri Feb 9 17:43:23 2024","interfaces":[{"interface":"back","average":{"1min":1.6499999999999999,"5min":1.639,"15min":1.6020000000000001},"min":{"1min":0.376,"5min":0.35299999999999998,"15min":0.34899999999999998},"max":{"1min":4.8780000000000001,"5min":14.542,"15min":14.542},"last":5.5439999999999996},{"interface":"front","average":{"1min":3.6699999999999999,"5min":1.702,"15min":1.8300000000000001},"min":{"1min":0.37,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":16.359000000000002,"5min":16.359000000000002,"15min":16.359000000000002},"last":0.435}]},{"osd":9,"last update":"Fri Feb 9 17:49:33 2024","interfaces":[{"interface":"back","average":{"1min":1.778,"5min":1.577,"15min":1.498},"min":{"1min":0.42599999999999999,"5min":0.36599999999999999,"15min":0.35299999999999998},"max":{"1min":15.124000000000001,"5min":15.124000000000001,"15min":16.411000000000001},"last":0.54500000000000004},{"interface":"front","average":{"1min":2.0110000000000001,"5min":1.9139999999999999,"15min":1.6659999999999999},"min":{"1min":0.379,"5min":0.35999999999999999,"15min":0.35299999999999998},"max":{"1min":5.7300000000000004,"5min":14.013,"15min":16.417999999999999},"last":0.53000000000000003}]},{"osd":10,"last update":"Fri Feb 9 17:49:04 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":2.5129999999999999,"15min":1.9079999999999999},"min":{"1min":0.40899999999999997,"5min":0.39400000000000002,"15min":0.317},"max":{"1min":5.069,"5min":80.042000000000002,"15min":80.042000000000002},"last":0.80400000000000005},{"interface":"front","average":{"1min":1.4390000000000001,"5min":2.4180000000000001,"15min":1.851},"min":{"1min":0.42199999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":8.7309999999999999,"5min":79.807000000000002,"15min":79.807000000000002},"last":0.72599999999999998}]},{"osd":11,"last update":"Fri Feb 9 17:49:15 2024","interfaces":[{"interface":"back","average":{"1min":2.149,"5min":2.194,"15min":1.984},"min":{"1min":0.48299999999999998,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":5.7679999999999998,"5min":13.987,"15min":16.390000000000001},"last":0.56200000000000006},{"interface":"front","average":{"1min":1.635,"5min":1.5820000000000001,"15min":1.7350000000000001},"min":{"1min":0.45600000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":5.3940000000000001,"5min":7.1470000000000002,"15min":9.8239999999999998},"last":0.82699999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":253165568,"data_stored":253165568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":326963200,"data_stored":326959128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":381419520,"data_stored":381419520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":707657728,"data_stored":707657728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":375062528,"data_stored":375062528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":407769088,"data_stored":407765016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":367919104,"data_stored":367919104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":443088896,"data_stored":443088896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":172818432,"data_stored":172818432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":358744064,"data_stored":358744064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":255070208,"data_stored":255066136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":186843136,"data_stored":186843136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-09T17:49:39.025 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph pg dump --format=json 2024-02-09T17:49:39.044 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:39.081 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:49:39.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:39.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:49:39.196 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:49:39.225 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:39.248 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:49:39.253 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:39.254 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:49:39.305 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:49:39.355 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:49:39.395 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:39 smithi099 ceph-mon[39420]: 6.e scrub starts 2024-02-09T17:49:39.396 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:39 smithi099 ceph-mon[39420]: 6.e scrub ok 2024-02-09T17:49:39.406 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:39.444 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:49:39.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:39.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:49:39.554 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:49:39.616 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:39.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:39 smithi179 ceph-mon[43975]: 6.e scrub starts 2024-02-09T17:49:39.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:39 smithi179 ceph-mon[43975]: 6.e scrub ok 2024-02-09T17:49:39.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:39 smithi162 ceph-mon[44304]: 6.e scrub starts 2024-02-09T17:49:39.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:39 smithi162 ceph-mon[44304]: 6.e scrub ok 2024-02-09T17:49:40.205 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is set 2024-02-09T17:49:40.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:40 smithi099 ceph-mon[39420]: pgmap v1638: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 1.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 701 op/s 2024-02-09T17:49:40.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:40 smithi099 ceph-mon[39420]: 1.a deep-scrub starts 2024-02-09T17:49:40.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:40 smithi099 ceph-mon[39420]: 1.a deep-scrub ok 2024-02-09T17:49:40.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:40 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3619535595' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:49:40.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:40 smithi179 ceph-mon[43975]: pgmap v1638: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 1.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 701 op/s 2024-02-09T17:49:40.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:40 smithi179 ceph-mon[43975]: 1.a deep-scrub starts 2024-02-09T17:49:40.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:40 smithi179 ceph-mon[43975]: 1.a deep-scrub ok 2024-02-09T17:49:40.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:40 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3619535595' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:49:40.681 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:40 smithi162 ceph-mon[44304]: pgmap v1638: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 1.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 701 op/s 2024-02-09T17:49:40.681 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:40 smithi162 ceph-mon[44304]: 1.a deep-scrub starts 2024-02-09T17:49:40.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:40 smithi162 ceph-mon[44304]: 1.a deep-scrub ok 2024-02-09T17:49:40.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:40 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3619535595' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:49:41.082 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:49:41.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:41.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:49:41.186 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:49:41.229 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:41.409 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:41 smithi099 ceph-mon[39420]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:49:41.409 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:41 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3619535595' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:49:41.409 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:41 smithi099 ceph-mon[39420]: osdmap e652: 12 total, 12 up, 12 in 2024-02-09T17:49:41.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:41 smithi179 ceph-mon[43975]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:49:41.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:41 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3619535595' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:49:41.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:41 smithi179 ceph-mon[43975]: osdmap e652: 12 total, 12 up, 12 in 2024-02-09T17:49:41.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:41 smithi162 ceph-mon[44304]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:49:41.681 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:41 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3619535595' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:49:41.681 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:41 smithi162 ceph-mon[44304]: osdmap e652: 12 total, 12 up, 12 in 2024-02-09T17:49:42.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:42 smithi099 ceph-mon[39420]: pgmap v1640: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 1.1 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 538 op/s 2024-02-09T17:49:42.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:42 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:49:42.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:42 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:49:42.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:42 smithi179 ceph-mon[43975]: pgmap v1640: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 1.1 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 538 op/s 2024-02-09T17:49:42.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:42 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:49:42.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:42 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:49:42.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:42 smithi162 ceph-mon[44304]: pgmap v1640: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 1.1 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 538 op/s 2024-02-09T17:49:42.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:42 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:49:42.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:42 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:49:43.037 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:49:43.039 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:43.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:49:43.141 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:49:43.199 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:43.458 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:43 smithi099 ceph-mon[39420]: osdmap e653: 12 total, 12 up, 12 in 2024-02-09T17:49:43.464 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:49:43.483 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:43.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:49:43.584 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:49:43.656 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:43 smithi162 ceph-mon[44304]: osdmap e653: 12 total, 12 up, 12 in 2024-02-09T17:49:43.658 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:43.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:43 smithi179 ceph-mon[43975]: osdmap e653: 12 total, 12 up, 12 in 2024-02-09T17:49:43.727 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:49:43.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:43.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:49:43.837 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:49:43.910 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:44.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:44 smithi099 ceph-mon[39420]: pgmap v1642: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 956 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 816 op/s 2024-02-09T17:49:44.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:44 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:49:44.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:44 smithi179 ceph-mon[43975]: pgmap v1642: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 956 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 816 op/s 2024-02-09T17:49:44.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:44 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:49:44.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:44 smithi162 ceph-mon[44304]: pgmap v1642: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 956 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 816 op/s 2024-02-09T17:49:44.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:44 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:49:45.273 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:49:45.293 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:49:45.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:45.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:49:45.315 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:45 smithi099 ceph-mon[39420]: 6.5 deep-scrub ok 2024-02-09T17:49:45.400 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:49:45.451 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:45.476 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:49:45.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:45.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:49:45.583 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:49:45.609 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:45 smithi162 ceph-mon[44304]: 6.5 deep-scrub ok 2024-02-09T17:49:45.611 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:45.636 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:49:45.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:45.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:49:45.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:45 smithi179 ceph-mon[43975]: 6.5 deep-scrub ok 2024-02-09T17:49:45.746 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:49:45.776 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:45.803 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:49:45.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:45.807 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:49:45.908 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:49:45.982 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:46.104 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:49:46.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:46.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:49:46.210 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:49:46.241 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:46.264 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:49:46.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:46.269 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:49:46.370 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:49:46.411 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:46.437 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:49:46.441 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:46.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:49:46.543 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:49:46.595 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:49:46.595 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:49:46.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:46 smithi162 ceph-mon[44304]: pgmap v1643: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 912 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 735 op/s 2024-02-09T17:49:46.596 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:46.612 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:49:46.613 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:46 smithi099 ceph-mon[39420]: pgmap v1643: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 912 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 735 op/s 2024-02-09T17:49:46.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:46.629 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:49:46.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:49:46.664 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:49:46.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:46 smithi179 ceph-mon[43975]: pgmap v1643: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 912 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 735 op/s 2024-02-09T17:49:46.716 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:49:46.730 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:49:46.771 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:46.796 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:49:46.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:46.799 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:49:46.900 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:49:46.930 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:46.959 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:49:46.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:46.962 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:49:46.982 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:49:47.020 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:49:47.055 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:49:47.062 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:49:47.100 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-09T17:49:47.115 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:47.119 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:49:47.138 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:49:47.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:47.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:49:47.185 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:49:47.224 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:49:47.245 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:49:47.265 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:49:47.286 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:47.298 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:49:47.313 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:49:47.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:47.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:49:47.360 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:47 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/529317562' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:49:47.378 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:49:47.378 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:49:47.378 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:49:47.378 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:49:47.378 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:49:47.378 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:49:47.378 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:49:47.378 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:49:47.378 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:49:47.379 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:49:47.379 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:49:47.379 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:49:47.379 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:49:47.417 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:49:47.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:47 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/529317562' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:49:47.465 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:47.494 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:49:47.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:47.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:49:47.602 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:49:47.633 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:47.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:47 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/529317562' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:49:48.192 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:49:48.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:48.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:49:48.299 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:49:48.329 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:48.353 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:49:48.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:48.357 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:49:48.452 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:48 smithi099 ceph-mon[39420]: pgmap v1644: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 840 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 640 op/s 2024-02-09T17:49:48.458 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:49:48.501 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:48.546 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:49:48.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:48.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:49:48.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:48 smithi162 ceph-mon[44304]: pgmap v1644: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 840 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 640 op/s 2024-02-09T17:49:48.655 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:49:48.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:48 smithi179 ceph-mon[43975]: pgmap v1644: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 840 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 640 op/s 2024-02-09T17:49:48.686 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:48.705 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:49:48.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:48.710 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:49:48.811 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:49:48.842 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:48.869 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:49:48.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:48.878 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:49:48.979 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:49:48.994 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-09T17:49:49.011 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:49.038 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:49:49.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:49.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:49:49.147 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:49:49.227 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:49.587 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:49:49.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:49.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:49:49.696 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:49:49.765 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:50.138 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:49:50.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:50.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:49:50.244 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:49:50.287 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:50.427 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:50 smithi099 ceph-mon[39420]: pgmap v1645: 47 pgs: 47 active+clean; 699 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 791 op/s 2024-02-09T17:49:50.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:50 smithi179 ceph-mon[43975]: pgmap v1645: 47 pgs: 47 active+clean; 699 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 791 op/s 2024-02-09T17:49:50.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:50 smithi162 ceph-mon[44304]: pgmap v1645: 47 pgs: 47 active+clean; 699 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 791 op/s 2024-02-09T17:49:51.581 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:49:51.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:51.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:49:51.690 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:49:51.743 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:51.778 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:49:51.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:51.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:49:51.816 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:49:51.839 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-02-09T17:49:51.887 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:49:51.916 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:51.942 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:49:51.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:51.946 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:49:52.047 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:49:52.111 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:52.156 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:49:52.164 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:49:52.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:52.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:49:52.277 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:49:52.305 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:52.331 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:49:52.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:52.335 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:49:52.436 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:49:52.466 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:52.494 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:49:52.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:52 smithi099 ceph-mon[39420]: pgmap v1646: 47 pgs: 47 active+clean; 651 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 752 op/s 2024-02-09T17:49:52.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:52.500 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:49:52.601 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:49:52.640 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:52 smithi179 ceph-mon[43975]: pgmap v1646: 47 pgs: 47 active+clean; 651 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 752 op/s 2024-02-09T17:49:52.642 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:52.668 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:49:52.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:52.673 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:49:52.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:52 smithi162 ceph-mon[44304]: pgmap v1646: 47 pgs: 47 active+clean; 651 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 752 op/s 2024-02-09T17:49:52.773 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:49:52.801 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:52.835 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:49:52.839 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:52.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:49:52.940 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:49:52.986 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:53.391 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:49:53.391 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:49:53.636 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:53 smithi099 ceph-mon[39420]: from='client.26062 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:49:53.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:53 smithi179 ceph-mon[43975]: from='client.26062 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:49:53.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:53 smithi162 ceph-mon[44304]: from='client.26062 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:49:54.111 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:49:54.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:54.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:49:54.219 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:49:54.272 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:54.303 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:49:54.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:54.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:49:54.411 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:49:54.440 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:54.464 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:49:54.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:54.471 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:49:54.572 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:49:54.605 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:54.637 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:49:54.642 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:54 smithi179 ceph-mon[43975]: pgmap v1647: 47 pgs: 47 active+clean; 459 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 881 op/s 2024-02-09T17:49:54.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:54.643 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:49:54.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:54 smithi162 ceph-mon[44304]: pgmap v1647: 47 pgs: 47 active+clean; 459 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 881 op/s 2024-02-09T17:49:54.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:54 smithi099 ceph-mon[39420]: pgmap v1647: 47 pgs: 47 active+clean; 459 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 881 op/s 2024-02-09T17:49:54.744 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:49:54.794 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:56.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:56 smithi179 ceph-mon[43975]: pgmap v1648: 47 pgs: 47 active+clean; 407 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 731 op/s 2024-02-09T17:49:56.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:56 smithi162 ceph-mon[44304]: pgmap v1648: 47 pgs: 47 active+clean; 407 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 731 op/s 2024-02-09T17:49:56.692 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:56 smithi099 ceph-mon[39420]: pgmap v1648: 47 pgs: 47 active+clean; 407 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 731 op/s 2024-02-09T17:49:57.657 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:57 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:49:57.657 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:57 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:49:57.657 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:57 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:49:57.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:57 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:49:57.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:57 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:49:57.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:57 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:49:57.676 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:49:57.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:57 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:49:57.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:57 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:49:57.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:57 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:49:57.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:57.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:49:57.686 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":1646,"stamp":"2024-02-09T17:49:51.016979+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":682623544,"num_objects":10410,"num_object_clones":0,"num_object_copies":31230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10410,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":250350,"num_write_kb":8344193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":165124,"ondisk_log_size":165124,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":8827124,"kb_used_data":2034196,"kb_used_omap":229,"kb_used_meta":6792218,"kb_avail":1116114700,"statfs":{"total":1151940427776,"available":1142901452800,"internally_reserved":0,"allocated":2083016704,"data_stored":2079421132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235127,"internal_metadata":6955231625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,8,0,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-493355008,"num_objects":-7528,"num_object_clones":0,"num_object_copies":-22584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7528,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.006087"},"pg_stats":[{"pgid":"1.1e","version":"63'139","reported_seq":562,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-09T17:49:42.256900+0000","last_change":"2024-02-09T17:48:27.182988+0000","last_active":"2024-02-09T17:49:42.256900+0000","last_peered":"2024-02-09T17:49:42.256900+0000","last_clean":"2024-02-09T17:49:42.256900+0000","last_became_active":"2024-02-09T17:43:19.126757+0000","last_became_peered":"2024-02-09T17:43:19.126757+0000","last_unstale":"2024-02-09T17:49:42.256900+0000","last_undegraded":"2024-02-09T17:49:42.256900+0000","last_fullsized":"2024-02-09T17:49:42.256900+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:48:27.182896+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:48:27.182896+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:49:44.889623+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"63'139","reported_seq":586,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-09T17:49:42.225593+0000","last_change":"2024-02-09T17:48:24.634219+0000","last_active":"2024-02-09T17:49:42.225593+0000","last_peered":"2024-02-09T17:49:42.225593+0000","last_clean":"2024-02-09T17:49:42.225593+0000","last_became_active":"2024-02-09T17:43:28.594296+0000","last_became_peered":"2024-02-09T17:43:28.594296+0000","last_unstale":"2024-02-09T17:49:42.225593+0000","last_undegraded":"2024-02-09T17:49:42.225593+0000","last_fullsized":"2024-02-09T17:49:42.225593+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:48:24.634092+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:48:24.634092+0000","last_clean_scrub_stamp":"2024-02-09T17:48:24.634092+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"63'139","reported_seq":1106,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-09T17:49:42.227023+0000","last_change":"2024-02-09T17:49:17.868395+0000","last_active":"2024-02-09T17:49:42.227023+0000","last_peered":"2024-02-09T17:49:42.227023+0000","last_clean":"2024-02-09T17:49:42.227023+0000","last_became_active":"2024-02-09T17:43:19.103584+0000","last_became_peered":"2024-02-09T17:43:19.103584+0000","last_unstale":"2024-02-09T17:49:42.227023+0000","last_undegraded":"2024-02-09T17:49:42.227023+0000","last_fullsized":"2024-02-09T17:49:42.227023+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:49:17.868254+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:23.912504+0000","last_clean_scrub_stamp":"2024-02-09T17:49:17.868254+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:28.930122+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"63'139","reported_seq":904,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-09T17:49:42.227014+0000","last_change":"2024-02-09T17:49:08.965002+0000","last_active":"2024-02-09T17:49:42.227014+0000","last_peered":"2024-02-09T17:49:42.227014+0000","last_clean":"2024-02-09T17:49:42.227014+0000","last_became_active":"2024-02-09T17:43:19.102729+0000","last_became_peered":"2024-02-09T17:43:19.102729+0000","last_unstale":"2024-02-09T17:49:42.227014+0000","last_undegraded":"2024-02-09T17:49:42.227014+0000","last_fullsized":"2024-02-09T17:49:42.227014+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:49:08.964912+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:45:44.988633+0000","last_clean_scrub_stamp":"2024-02-09T17:49:08.964912+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:32.772979+0000","scrub_duration":126,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"63'139","reported_seq":575,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-09T17:49:42.227184+0000","last_change":"2024-02-09T17:49:36.820953+0000","last_active":"2024-02-09T17:49:42.227184+0000","last_peered":"2024-02-09T17:49:42.227184+0000","last_clean":"2024-02-09T17:49:42.227184+0000","last_became_active":"2024-02-09T17:43:19.120814+0000","last_became_peered":"2024-02-09T17:43:19.120814+0000","last_unstale":"2024-02-09T17:49:42.227184+0000","last_undegraded":"2024-02-09T17:49:42.227184+0000","last_fullsized":"2024-02-09T17:49:42.227184+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:49:36.820839+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:49:36.820839+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:56.220878+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"63'139","reported_seq":1403,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-09T17:49:42.226541+0000","last_change":"2024-02-09T17:49:16.723321+0000","last_active":"2024-02-09T17:49:42.226541+0000","last_peered":"2024-02-09T17:49:42.226541+0000","last_clean":"2024-02-09T17:49:42.226541+0000","last_became_active":"2024-02-09T17:43:19.110535+0000","last_became_peered":"2024-02-09T17:43:19.110535+0000","last_unstale":"2024-02-09T17:49:42.226541+0000","last_undegraded":"2024-02-09T17:49:42.226541+0000","last_fullsized":"2024-02-09T17:49:42.226541+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:49:16.723197+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:30.193194+0000","last_clean_scrub_stamp":"2024-02-09T17:49:16.723197+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:21.466285+0000","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"63'139","reported_seq":1242,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-09T17:49:42.233299+0000","last_change":"2024-02-09T17:49:12.248620+0000","last_active":"2024-02-09T17:49:42.233299+0000","last_peered":"2024-02-09T17:49:42.233299+0000","last_clean":"2024-02-09T17:49:42.233299+0000","last_became_active":"2024-02-09T17:43:28.585573+0000","last_became_peered":"2024-02-09T17:43:28.585573+0000","last_unstale":"2024-02-09T17:49:42.233299+0000","last_undegraded":"2024-02-09T17:49:42.233299+0000","last_fullsized":"2024-02-09T17:49:42.233299+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:49:12.248513+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:49:12.248513+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:29.288104+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"63'139","reported_seq":696,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-09T17:49:42.227330+0000","last_change":"2024-02-09T17:49:24.923424+0000","last_active":"2024-02-09T17:49:42.227330+0000","last_peered":"2024-02-09T17:49:42.227330+0000","last_clean":"2024-02-09T17:49:42.227330+0000","last_became_active":"2024-02-09T17:48:46.184419+0000","last_became_peered":"2024-02-09T17:48:46.184419+0000","last_unstale":"2024-02-09T17:49:42.227330+0000","last_undegraded":"2024-02-09T17:49:42.227330+0000","last_fullsized":"2024-02-09T17:49:42.227330+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:49:24.923286+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:44:50.214686+0000","last_clean_scrub_stamp":"2024-02-09T17:49:24.923286+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:30.714830+0000","scrub_duration":118,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"63'139","reported_seq":569,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-09T17:49:42.225979+0000","last_change":"2024-02-09T17:49:23.348000+0000","last_active":"2024-02-09T17:49:42.225979+0000","last_peered":"2024-02-09T17:49:42.225979+0000","last_clean":"2024-02-09T17:49:42.225979+0000","last_became_active":"2024-02-09T17:43:19.094230+0000","last_became_peered":"2024-02-09T17:43:19.094230+0000","last_unstale":"2024-02-09T17:49:42.225979+0000","last_undegraded":"2024-02-09T17:49:42.225979+0000","last_fullsized":"2024-02-09T17:49:42.225979+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:49:23.347891+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:11.970338+0000","last_clean_scrub_stamp":"2024-02-09T17:49:23.347891+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:29.874655+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"63'139","reported_seq":1016,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-09T17:49:42.253443+0000","last_change":"2024-02-09T17:48:20.211497+0000","last_active":"2024-02-09T17:49:42.253443+0000","last_peered":"2024-02-09T17:49:42.253443+0000","last_clean":"2024-02-09T17:49:42.253443+0000","last_became_active":"2024-02-09T17:43:19.138398+0000","last_became_peered":"2024-02-09T17:43:19.138398+0000","last_unstale":"2024-02-09T17:49:42.253443+0000","last_undegraded":"2024-02-09T17:49:42.253443+0000","last_fullsized":"2024-02-09T17:49:42.253443+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:48:20.211388+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:48:20.211388+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":170,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"63'139","reported_seq":1070,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-09T17:49:42.225281+0000","last_change":"2024-02-09T17:49:30.296419+0000","last_active":"2024-02-09T17:49:42.225281+0000","last_peered":"2024-02-09T17:49:42.225281+0000","last_clean":"2024-02-09T17:49:42.225281+0000","last_became_active":"2024-02-09T17:43:28.589307+0000","last_became_peered":"2024-02-09T17:43:28.589307+0000","last_unstale":"2024-02-09T17:49:42.225281+0000","last_undegraded":"2024-02-09T17:49:42.225281+0000","last_fullsized":"2024-02-09T17:49:42.225281+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:49:30.296287+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:38:26.968892+0000","last_clean_scrub_stamp":"2024-02-09T17:49:30.296287+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:48.296433+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"653'15652","reported_seq":15967,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-09T17:49:47.640701+0000","last_change":"2024-02-09T17:48:31.957735+0000","last_active":"2024-02-09T17:49:47.640701+0000","last_peered":"2024-02-09T17:49:47.640701+0000","last_clean":"2024-02-09T17:49:47.640701+0000","last_became_active":"2024-02-09T17:46:47.018844+0000","last_became_peered":"2024-02-09T17:46:47.018844+0000","last_unstale":"2024-02-09T17:49:47.640701+0000","last_undegraded":"2024-02-09T17:49:47.640701+0000","last_fullsized":"2024-02-09T17:49:47.640701+0000","mapping_epoch":600,"log_start":"598'5600","ondisk_log_start":"598'5600","created":576,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"632'11810","last_scrub_stamp":"2024-02-09T17:48:31.957610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:45:42.586555+0000","last_clean_scrub_stamp":"2024-02-09T17:48:31.957610+0000","objects_scrubbed":4556,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1643,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44040192,"num_objects":672,"num_object_clones":0,"num_object_copies":2016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15652,"num_write_kb":522368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"63'139","reported_seq":1665,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-09T17:49:42.237305+0000","last_change":"2024-02-09T17:48:32.090480+0000","last_active":"2024-02-09T17:49:42.237305+0000","last_peered":"2024-02-09T17:49:42.237305+0000","last_clean":"2024-02-09T17:49:42.237305+0000","last_became_active":"2024-02-09T17:41:14.521079+0000","last_became_peered":"2024-02-09T17:41:14.521079+0000","last_unstale":"2024-02-09T17:49:42.237305+0000","last_undegraded":"2024-02-09T17:49:42.237305+0000","last_fullsized":"2024-02-09T17:49:42.237305+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:48:32.090348+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:48:32.090348+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:49:45.676733+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"653'16552","reported_seq":16905,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-09T17:49:47.848411+0000","last_change":"2024-02-09T17:49:43.834243+0000","last_active":"2024-02-09T17:49:47.848411+0000","last_peered":"2024-02-09T17:49:47.848411+0000","last_clean":"2024-02-09T17:49:47.848411+0000","last_became_active":"2024-02-09T17:45:43.660463+0000","last_became_peered":"2024-02-09T17:45:43.660463+0000","last_unstale":"2024-02-09T17:49:47.848411+0000","last_undegraded":"2024-02-09T17:49:47.848411+0000","last_fullsized":"2024-02-09T17:49:47.848411+0000","mapping_epoch":576,"log_start":"605'6500","ondisk_log_start":"605'6500","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"653'16342","last_scrub_stamp":"2024-02-09T17:49:43.834050+0000","last_deep_scrub":"653'16342","last_deep_scrub_stamp":"2024-02-09T17:49:43.834050+0000","last_clean_scrub_stamp":"2024-02-09T17:49:43.834050+0000","objects_scrubbed":1080,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":9,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:51:03.100199+0000","scrub_duration":8263,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45219840,"num_objects":690,"num_object_clones":0,"num_object_copies":2070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15472,"num_write_kb":517184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"63'139","reported_seq":1230,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-09T17:49:42.247112+0000","last_change":"2024-02-09T17:49:11.938719+0000","last_active":"2024-02-09T17:49:42.247112+0000","last_peered":"2024-02-09T17:49:42.247112+0000","last_clean":"2024-02-09T17:49:42.247112+0000","last_became_active":"2024-02-09T17:43:39.981054+0000","last_became_peered":"2024-02-09T17:43:39.981054+0000","last_unstale":"2024-02-09T17:49:42.247112+0000","last_undegraded":"2024-02-09T17:49:42.247112+0000","last_fullsized":"2024-02-09T17:49:42.247112+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:49:11.938609+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:49:11.938609+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:21.174789+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"653'15857","reported_seq":16168,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-09T17:49:49.911513+0000","last_change":"2024-02-09T17:48:23.008484+0000","last_active":"2024-02-09T17:49:49.911513+0000","last_peered":"2024-02-09T17:49:49.911513+0000","last_clean":"2024-02-09T17:49:49.911513+0000","last_became_active":"2024-02-09T17:45:43.605145+0000","last_became_peered":"2024-02-09T17:45:43.605145+0000","last_unstale":"2024-02-09T17:49:49.911513+0000","last_undegraded":"2024-02-09T17:49:49.911513+0000","last_fullsized":"2024-02-09T17:49:49.911513+0000","mapping_epoch":576,"log_start":"598'5800","ondisk_log_start":"598'5800","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"631'11553","last_scrub_stamp":"2024-02-09T17:48:23.008362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:45:42.586555+0000","last_clean_scrub_stamp":"2024-02-09T17:48:23.008362+0000","objects_scrubbed":4912,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:49:51.953874+0000","scrub_duration":1446,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37814272,"num_objects":577,"num_object_clones":0,"num_object_copies":1731,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15857,"num_write_kb":525888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"63'139","reported_seq":853,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-09T17:49:42.227470+0000","last_change":"2024-02-09T17:49:24.432905+0000","last_active":"2024-02-09T17:49:42.227470+0000","last_peered":"2024-02-09T17:49:42.227470+0000","last_clean":"2024-02-09T17:49:42.227470+0000","last_became_active":"2024-02-09T17:40:13.501711+0000","last_became_peered":"2024-02-09T17:40:13.501711+0000","last_unstale":"2024-02-09T17:49:42.227470+0000","last_undegraded":"2024-02-09T17:49:42.227470+0000","last_fullsized":"2024-02-09T17:49:42.227470+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:49:24.432810+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:46:40.684762+0000","last_clean_scrub_stamp":"2024-02-09T17:49:24.432810+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:34.977155+0000","scrub_duration":195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"653'15679","reported_seq":15982,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-09T17:49:47.636545+0000","last_change":"2024-02-09T17:48:29.818271+0000","last_active":"2024-02-09T17:49:47.636545+0000","last_peered":"2024-02-09T17:49:47.636545+0000","last_clean":"2024-02-09T17:49:47.636545+0000","last_became_active":"2024-02-09T17:45:43.610974+0000","last_became_peered":"2024-02-09T17:45:43.610974+0000","last_unstale":"2024-02-09T17:49:47.636545+0000","last_undegraded":"2024-02-09T17:49:47.636545+0000","last_fullsized":"2024-02-09T17:49:47.636545+0000","mapping_epoch":576,"log_start":"597'5600","ondisk_log_start":"597'5600","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"632'11793","last_scrub_stamp":"2024-02-09T17:48:29.818043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:45:42.586555+0000","last_clean_scrub_stamp":"2024-02-09T17:48:29.818043+0000","objects_scrubbed":4599,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:49:52.446276+0000","scrub_duration":1543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44367872,"num_objects":677,"num_object_clones":0,"num_object_copies":2031,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15679,"num_write_kb":523392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"63'139","reported_seq":380,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-09T17:49:42.226553+0000","last_change":"2024-02-09T17:49:11.737157+0000","last_active":"2024-02-09T17:49:42.226553+0000","last_peered":"2024-02-09T17:49:42.226553+0000","last_clean":"2024-02-09T17:49:42.226553+0000","last_became_active":"2024-02-09T17:48:46.207004+0000","last_became_peered":"2024-02-09T17:48:46.207004+0000","last_unstale":"2024-02-09T17:49:42.226553+0000","last_undegraded":"2024-02-09T17:49:42.226553+0000","last_fullsized":"2024-02-09T17:49:42.226553+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:49:11.737062+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:49:11.737062+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:13.922917+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.1","version":"653'16776","reported_seq":17144,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-09T17:49:47.054792+0000","last_change":"2024-02-09T17:49:37.533631+0000","last_active":"2024-02-09T17:49:47.054792+0000","last_peered":"2024-02-09T17:49:47.054792+0000","last_clean":"2024-02-09T17:49:47.054792+0000","last_became_active":"2024-02-09T17:45:43.606122+0000","last_became_peered":"2024-02-09T17:45:43.606122+0000","last_unstale":"2024-02-09T17:49:47.054792+0000","last_undegraded":"2024-02-09T17:49:47.054792+0000","last_fullsized":"2024-02-09T17:49:47.054792+0000","mapping_epoch":576,"log_start":"606'6701","ondisk_log_start":"606'6701","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"651'16365","last_scrub_stamp":"2024-02-09T17:49:37.533468+0000","last_deep_scrub":"651'16365","last_deep_scrub_stamp":"2024-02-09T17:49:37.533468+0000","last_clean_scrub_stamp":"2024-02-09T17:49:37.533468+0000","objects_scrubbed":1286,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":10,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:58.633057+0000","scrub_duration":9768,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45088768,"num_objects":688,"num_object_clones":0,"num_object_copies":2064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15490,"num_write_kb":517696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"63'139","reported_seq":1173,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-09T17:49:42.226640+0000","last_change":"2024-02-09T17:49:35.724086+0000","last_active":"2024-02-09T17:49:42.226640+0000","last_peered":"2024-02-09T17:49:42.226640+0000","last_clean":"2024-02-09T17:49:42.226640+0000","last_became_active":"2024-02-09T17:43:14.024894+0000","last_became_peered":"2024-02-09T17:43:14.024894+0000","last_unstale":"2024-02-09T17:49:42.226640+0000","last_undegraded":"2024-02-09T17:49:42.226640+0000","last_fullsized":"2024-02-09T17:49:42.226640+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:49:35.723959+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:11.970338+0000","last_clean_scrub_stamp":"2024-02-09T17:49:35.723959+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:44.719263+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.0","version":"653'15497","reported_seq":15828,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-09T17:49:47.684922+0000","last_change":"2024-02-09T17:49:30.219592+0000","last_active":"2024-02-09T17:49:47.684922+0000","last_peered":"2024-02-09T17:49:47.684922+0000","last_clean":"2024-02-09T17:49:47.684922+0000","last_became_active":"2024-02-09T17:45:43.659881+0000","last_became_peered":"2024-02-09T17:45:43.659881+0000","last_unstale":"2024-02-09T17:49:47.684922+0000","last_undegraded":"2024-02-09T17:49:47.684922+0000","last_fullsized":"2024-02-09T17:49:47.684922+0000","mapping_epoch":576,"log_start":"596'5400","ondisk_log_start":"596'5400","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"650'14692","last_scrub_stamp":"2024-02-09T17:49:30.219479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:45:42.586555+0000","last_clean_scrub_stamp":"2024-02-09T17:49:30.219479+0000","objects_scrubbed":1454,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:53.921365+0000","scrub_duration":340,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42270720,"num_objects":645,"num_object_clones":0,"num_object_copies":1935,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15497,"num_write_kb":516544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"63'139","reported_seq":837,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-09T17:49:42.225859+0000","last_change":"2024-02-09T17:49:11.483130+0000","last_active":"2024-02-09T17:49:42.225859+0000","last_peered":"2024-02-09T17:49:42.225859+0000","last_clean":"2024-02-09T17:49:42.225859+0000","last_became_active":"2024-02-09T17:43:14.038885+0000","last_became_peered":"2024-02-09T17:43:14.038885+0000","last_unstale":"2024-02-09T17:49:42.225859+0000","last_undegraded":"2024-02-09T17:49:42.225859+0000","last_fullsized":"2024-02-09T17:49:42.225859+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:49:11.483012+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:49:11.483012+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:36.774820+0000","scrub_duration":166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"653'15813","reported_seq":16121,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-09T17:49:47.848825+0000","last_change":"2024-02-09T17:48:30.387181+0000","last_active":"2024-02-09T17:49:47.848825+0000","last_peered":"2024-02-09T17:49:47.848825+0000","last_clean":"2024-02-09T17:49:47.848825+0000","last_became_active":"2024-02-09T17:45:43.660407+0000","last_became_peered":"2024-02-09T17:45:43.660407+0000","last_unstale":"2024-02-09T17:49:47.848825+0000","last_undegraded":"2024-02-09T17:49:47.848825+0000","last_fullsized":"2024-02-09T17:49:47.848825+0000","mapping_epoch":576,"log_start":"598'5800","ondisk_log_start":"598'5800","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"632'11815","last_scrub_stamp":"2024-02-09T17:48:30.387098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:45:42.586555+0000","last_clean_scrub_stamp":"2024-02-09T17:48:30.387098+0000","objects_scrubbed":4675,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:49:51.873976+0000","scrub_duration":1299,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41877504,"num_objects":639,"num_object_clones":0,"num_object_copies":1917,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15813,"num_write_kb":526464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"63'139","reported_seq":1244,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-09T17:49:42.256913+0000","last_change":"2024-02-09T17:49:12.382010+0000","last_active":"2024-02-09T17:49:42.256913+0000","last_peered":"2024-02-09T17:49:42.256913+0000","last_clean":"2024-02-09T17:49:42.256913+0000","last_became_active":"2024-02-09T17:33:00.284924+0000","last_became_peered":"2024-02-09T17:33:00.284924+0000","last_unstale":"2024-02-09T17:49:42.256913+0000","last_undegraded":"2024-02-09T17:49:42.256913+0000","last_fullsized":"2024-02-09T17:49:42.256913+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:49:12.381894+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:56.105792+0000","last_clean_scrub_stamp":"2024-02-09T17:49:12.381894+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:33.945253+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.2","version":"653'16015","reported_seq":16298,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-09T17:49:46.064339+0000","last_change":"2024-02-09T17:49:15.536637+0000","last_active":"2024-02-09T17:49:46.064339+0000","last_peered":"2024-02-09T17:49:46.064339+0000","last_clean":"2024-02-09T17:49:46.064339+0000","last_became_active":"2024-02-09T17:45:43.607871+0000","last_became_peered":"2024-02-09T17:45:43.607871+0000","last_unstale":"2024-02-09T17:49:46.064339+0000","last_undegraded":"2024-02-09T17:49:46.064339+0000","last_fullsized":"2024-02-09T17:49:46.064339+0000","mapping_epoch":576,"log_start":"598'6000","ondisk_log_start":"598'6000","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"648'14550","last_scrub_stamp":"2024-02-09T17:49:15.536338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:45:42.586555+0000","last_clean_scrub_stamp":"2024-02-09T17:49:15.536338+0000","objects_scrubbed":2209,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:36.274708+0000","scrub_duration":778,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47513600,"num_objects":725,"num_object_clones":0,"num_object_copies":2175,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16015,"num_write_kb":535680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"63'139","reported_seq":1093,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-09T17:49:42.226712+0000","last_change":"2024-02-09T17:49:36.733644+0000","last_active":"2024-02-09T17:49:42.226712+0000","last_peered":"2024-02-09T17:49:42.226712+0000","last_clean":"2024-02-09T17:49:42.226712+0000","last_became_active":"2024-02-09T17:48:46.184307+0000","last_became_peered":"2024-02-09T17:48:46.184307+0000","last_unstale":"2024-02-09T17:49:42.226712+0000","last_undegraded":"2024-02-09T17:49:42.226712+0000","last_fullsized":"2024-02-09T17:49:42.226712+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:49:36.733515+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:49:36.733515+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:42.342994+0000","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.d","version":"653'15806","reported_seq":16094,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-09T17:49:49.917494+0000","last_change":"2024-02-09T17:49:10.933973+0000","last_active":"2024-02-09T17:49:49.917494+0000","last_peered":"2024-02-09T17:49:49.917494+0000","last_clean":"2024-02-09T17:49:49.917494+0000","last_became_active":"2024-02-09T17:46:47.013898+0000","last_became_peered":"2024-02-09T17:46:47.013898+0000","last_unstale":"2024-02-09T17:49:49.917494+0000","last_undegraded":"2024-02-09T17:49:49.917494+0000","last_fullsized":"2024-02-09T17:49:49.917494+0000","mapping_epoch":600,"log_start":"598'5800","ondisk_log_start":"598'5800","created":576,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"646'13958","last_scrub_stamp":"2024-02-09T17:49:10.933833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:45:42.586555+0000","last_clean_scrub_stamp":"2024-02-09T17:49:10.933833+0000","objects_scrubbed":2430,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:22.992698+0000","scrub_duration":717,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37290008,"num_objects":570,"num_object_clones":0,"num_object_copies":1710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":570,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15806,"num_write_kb":523969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,10],"acting":[1,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"63'139","reported_seq":1166,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-09T17:49:42.232032+0000","last_change":"2024-02-09T17:49:39.575691+0000","last_active":"2024-02-09T17:49:42.232032+0000","last_peered":"2024-02-09T17:49:42.232032+0000","last_clean":"2024-02-09T17:49:42.232032+0000","last_became_active":"2024-02-09T17:43:14.041291+0000","last_became_peered":"2024-02-09T17:43:14.041291+0000","last_unstale":"2024-02-09T17:49:42.232032+0000","last_undegraded":"2024-02-09T17:49:42.232032+0000","last_fullsized":"2024-02-09T17:49:42.232032+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:49:39.575571+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:49:39.575571+0000","last_clean_scrub_stamp":"2024-02-09T17:49:39.575571+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:45.375863+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.c","version":"653'15670","reported_seq":15949,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-09T17:49:47.572579+0000","last_change":"2024-02-09T17:49:26.520622+0000","last_active":"2024-02-09T17:49:47.572579+0000","last_peered":"2024-02-09T17:49:47.572579+0000","last_clean":"2024-02-09T17:49:47.572579+0000","last_became_active":"2024-02-09T17:45:43.660635+0000","last_became_peered":"2024-02-09T17:45:43.660635+0000","last_unstale":"2024-02-09T17:49:47.572579+0000","last_undegraded":"2024-02-09T17:49:47.572579+0000","last_fullsized":"2024-02-09T17:49:47.572579+0000","mapping_epoch":576,"log_start":"596'5600","ondisk_log_start":"596'5600","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"650'14778","last_scrub_stamp":"2024-02-09T17:49:26.520433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:45:42.586555+0000","last_clean_scrub_stamp":"2024-02-09T17:49:26.520433+0000","objects_scrubbed":1562,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:49.840061+0000","scrub_duration":672,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42991616,"num_objects":656,"num_object_clones":0,"num_object_copies":1968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15670,"num_write_kb":522432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"63'139","reported_seq":1008,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-09T17:49:42.260284+0000","last_change":"2024-02-09T17:48:22.237143+0000","last_active":"2024-02-09T17:49:42.260284+0000","last_peered":"2024-02-09T17:49:42.260284+0000","last_clean":"2024-02-09T17:49:42.260284+0000","last_became_active":"2024-02-09T17:43:14.054943+0000","last_became_peered":"2024-02-09T17:43:14.054943+0000","last_unstale":"2024-02-09T17:49:42.260284+0000","last_undegraded":"2024-02-09T17:49:42.260284+0000","last_fullsized":"2024-02-09T17:49:42.260284+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:48:22.237014+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:48:22.237014+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:49:48.423322+0000","scrub_duration":161,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.f","version":"653'15277","reported_seq":15595,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-09T17:49:48.190514+0000","last_change":"2024-02-09T17:49:37.862550+0000","last_active":"2024-02-09T17:49:48.190514+0000","last_peered":"2024-02-09T17:49:48.190514+0000","last_clean":"2024-02-09T17:49:48.190514+0000","last_became_active":"2024-02-09T17:45:43.601656+0000","last_became_peered":"2024-02-09T17:45:43.601656+0000","last_unstale":"2024-02-09T17:49:48.190514+0000","last_undegraded":"2024-02-09T17:49:48.190514+0000","last_fullsized":"2024-02-09T17:49:48.190514+0000","mapping_epoch":576,"log_start":"596'5200","ondisk_log_start":"596'5200","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"651'14795","last_scrub_stamp":"2024-02-09T17:49:37.862349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:45:42.586555+0000","last_clean_scrub_stamp":"2024-02-09T17:49:37.862349+0000","objects_scrubbed":1149,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:57.689853+0000","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43581440,"num_objects":665,"num_object_clones":0,"num_object_copies":1995,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15277,"num_write_kb":510144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.8","version":"63'139","reported_seq":800,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-09T17:49:42.238532+0000","last_change":"2024-02-09T17:49:12.945703+0000","last_active":"2024-02-09T17:49:42.238532+0000","last_peered":"2024-02-09T17:49:42.238532+0000","last_clean":"2024-02-09T17:49:42.238532+0000","last_became_active":"2024-02-09T17:46:47.013853+0000","last_became_peered":"2024-02-09T17:46:47.013853+0000","last_unstale":"2024-02-09T17:49:42.238532+0000","last_undegraded":"2024-02-09T17:49:42.238532+0000","last_fullsized":"2024-02-09T17:49:42.238532+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:49:12.945588+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:49:12.945588+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:14.177994+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"653'15461","reported_seq":15786,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-09T17:49:45.897436+0000","last_change":"2024-02-09T17:49:38.222916+0000","last_active":"2024-02-09T17:49:45.897436+0000","last_peered":"2024-02-09T17:49:45.897436+0000","last_clean":"2024-02-09T17:49:45.897436+0000","last_became_active":"2024-02-09T17:45:43.607953+0000","last_became_peered":"2024-02-09T17:45:43.607953+0000","last_unstale":"2024-02-09T17:49:45.897436+0000","last_undegraded":"2024-02-09T17:49:45.897436+0000","last_fullsized":"2024-02-09T17:49:45.897436+0000","mapping_epoch":576,"log_start":"596'5400","ondisk_log_start":"596'5400","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"651'15080","last_scrub_stamp":"2024-02-09T17:49:38.222626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:45:42.586555+0000","last_clean_scrub_stamp":"2024-02-09T17:49:38.222626+0000","objects_scrubbed":1116,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:44.702938+0000","scrub_duration":381,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47906816,"num_objects":731,"num_object_clones":0,"num_object_copies":2193,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15461,"num_write_kb":518144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"63'139","reported_seq":1671,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-09T17:49:42.237066+0000","last_change":"2024-02-09T17:48:27.166794+0000","last_active":"2024-02-09T17:49:42.237066+0000","last_peered":"2024-02-09T17:49:42.237066+0000","last_clean":"2024-02-09T17:49:42.237066+0000","last_became_active":"2024-02-09T17:43:28.590237+0000","last_became_peered":"2024-02-09T17:43:28.590237+0000","last_unstale":"2024-02-09T17:49:42.237066+0000","last_undegraded":"2024-02-09T17:49:42.237066+0000","last_fullsized":"2024-02-09T17:49:42.237066+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:48:27.166671+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:45:47.710252+0000","last_clean_scrub_stamp":"2024-02-09T17:48:27.166671+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"653'15523","reported_seq":15828,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-09T17:49:46.750726+0000","last_change":"2024-02-09T17:48:24.314402+0000","last_active":"2024-02-09T17:49:46.750726+0000","last_peered":"2024-02-09T17:49:46.750726+0000","last_clean":"2024-02-09T17:49:46.750726+0000","last_became_active":"2024-02-09T17:45:43.659891+0000","last_became_peered":"2024-02-09T17:45:43.659891+0000","last_unstale":"2024-02-09T17:49:46.750726+0000","last_undegraded":"2024-02-09T17:49:46.750726+0000","last_fullsized":"2024-02-09T17:49:46.750726+0000","mapping_epoch":576,"log_start":"596'5500","ondisk_log_start":"596'5500","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"631'11417","last_scrub_stamp":"2024-02-09T17:48:24.314163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:45:42.586555+0000","last_clean_scrub_stamp":"2024-02-09T17:48:24.314163+0000","objects_scrubbed":4775,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:49:48.910925+0000","scrub_duration":977,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42532864,"num_objects":649,"num_object_clones":0,"num_object_copies":1947,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15523,"num_write_kb":517504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,8],"acting":[9,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.e","version":"63'139","reported_seq":1128,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-09T17:49:42.231246+0000","last_change":"2024-02-09T17:48:12.955847+0000","last_active":"2024-02-09T17:49:42.231246+0000","last_peered":"2024-02-09T17:49:42.231246+0000","last_clean":"2024-02-09T17:49:42.231246+0000","last_became_active":"2024-02-09T17:43:14.036583+0000","last_became_peered":"2024-02-09T17:43:14.036583+0000","last_unstale":"2024-02-09T17:49:42.231246+0000","last_undegraded":"2024-02-09T17:49:42.231246+0000","last_fullsized":"2024-02-09T17:49:42.231246+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:48:12.955736+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:14.981453+0000","last_clean_scrub_stamp":"2024-02-09T17:48:12.955736+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":141,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"653'15867","reported_seq":16176,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-09T17:49:50.562387+0000","last_change":"2024-02-09T17:48:25.328842+0000","last_active":"2024-02-09T17:49:50.562387+0000","last_peered":"2024-02-09T17:49:50.562387+0000","last_clean":"2024-02-09T17:49:50.562387+0000","last_became_active":"2024-02-09T17:45:43.611062+0000","last_became_peered":"2024-02-09T17:45:43.611062+0000","last_unstale":"2024-02-09T17:49:50.562387+0000","last_undegraded":"2024-02-09T17:49:50.562387+0000","last_fullsized":"2024-02-09T17:49:50.562387+0000","mapping_epoch":576,"log_start":"598'5800","ondisk_log_start":"598'5800","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"631'11631","last_scrub_stamp":"2024-02-09T17:48:25.328752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:45:42.586555+0000","last_clean_scrub_stamp":"2024-02-09T17:48:25.328752+0000","objects_scrubbed":4757,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32178176,"num_objects":491,"num_object_clones":0,"num_object_copies":1473,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15867,"num_write_kb":523456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,3],"acting":[11,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.f","version":"63'139","reported_seq":1189,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-09T17:49:42.225804+0000","last_change":"2024-02-09T17:48:34.651789+0000","last_active":"2024-02-09T17:49:42.225804+0000","last_peered":"2024-02-09T17:49:42.225804+0000","last_clean":"2024-02-09T17:49:42.225804+0000","last_became_active":"2024-02-09T17:33:05.316100+0000","last_became_peered":"2024-02-09T17:33:05.316100+0000","last_unstale":"2024-02-09T17:49:42.225804+0000","last_undegraded":"2024-02-09T17:49:42.225804+0000","last_fullsized":"2024-02-09T17:49:42.225804+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:48:34.651647+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:48:34.651647+0000","last_clean_scrub_stamp":"2024-02-09T17:48:34.651647+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"653'15567","reported_seq":15866,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-09T17:49:47.696095+0000","last_change":"2024-02-09T17:48:32.876630+0000","last_active":"2024-02-09T17:49:47.696095+0000","last_peered":"2024-02-09T17:49:47.696095+0000","last_clean":"2024-02-09T17:49:47.696095+0000","last_became_active":"2024-02-09T17:45:43.611004+0000","last_became_peered":"2024-02-09T17:45:43.611004+0000","last_unstale":"2024-02-09T17:49:47.696095+0000","last_undegraded":"2024-02-09T17:49:47.696095+0000","last_fullsized":"2024-02-09T17:49:47.696095+0000","mapping_epoch":576,"log_start":"596'5500","ondisk_log_start":"596'5500","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"632'11816","last_scrub_stamp":"2024-02-09T17:48:32.876436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:45:42.586555+0000","last_clean_scrub_stamp":"2024-02-09T17:48:32.876436+0000","objects_scrubbed":4391,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1588,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40042496,"num_objects":611,"num_object_clones":0,"num_object_copies":1833,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15567,"num_write_kb":517696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"63'139","reported_seq":1315,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-09T17:49:42.238678+0000","last_change":"2024-02-09T17:49:11.961487+0000","last_active":"2024-02-09T17:49:42.238678+0000","last_peered":"2024-02-09T17:49:42.238678+0000","last_clean":"2024-02-09T17:49:42.238678+0000","last_became_active":"2024-02-09T17:43:14.027715+0000","last_became_peered":"2024-02-09T17:43:14.027715+0000","last_unstale":"2024-02-09T17:49:42.238678+0000","last_undegraded":"2024-02-09T17:49:42.238678+0000","last_fullsized":"2024-02-09T17:49:42.238678+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:49:11.961373+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:14.135321+0000","last_clean_scrub_stamp":"2024-02-09T17:49:11.961373+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:41.624952+0000","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"653'15704","reported_seq":15988,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-09T17:49:46.219214+0000","last_change":"2024-02-09T17:49:18.938095+0000","last_active":"2024-02-09T17:49:46.219214+0000","last_peered":"2024-02-09T17:49:46.219214+0000","last_clean":"2024-02-09T17:49:46.219214+0000","last_became_active":"2024-02-09T17:46:47.014763+0000","last_became_peered":"2024-02-09T17:46:47.014763+0000","last_unstale":"2024-02-09T17:49:46.219214+0000","last_undegraded":"2024-02-09T17:49:46.219214+0000","last_fullsized":"2024-02-09T17:49:46.219214+0000","mapping_epoch":600,"log_start":"596'5700","ondisk_log_start":"596'5700","created":576,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"649'14453","last_scrub_stamp":"2024-02-09T17:49:18.937963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:45:42.586555+0000","last_clean_scrub_stamp":"2024-02-09T17:49:18.937963+0000","objects_scrubbed":1986,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:23.347045+0000","scrub_duration":608,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47316992,"num_objects":722,"num_object_clones":0,"num_object_copies":2166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15704,"num_write_kb":525632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.d","version":"63'139","reported_seq":720,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-09T17:49:42.253546+0000","last_change":"2024-02-09T17:48:26.223438+0000","last_active":"2024-02-09T17:49:42.253546+0000","last_peered":"2024-02-09T17:49:42.253546+0000","last_clean":"2024-02-09T17:49:42.253546+0000","last_became_active":"2024-02-09T17:43:14.036759+0000","last_became_peered":"2024-02-09T17:43:14.036759+0000","last_unstale":"2024-02-09T17:49:42.253546+0000","last_undegraded":"2024-02-09T17:49:42.253546+0000","last_fullsized":"2024-02-09T17:49:42.253546+0000","mapping_epoch":454,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:48:26.220964+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:48:26.220964+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":185,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"63'139","reported_seq":892,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-09T17:49:42.237197+0000","last_change":"2024-02-09T17:48:35.170778+0000","last_active":"2024-02-09T17:49:42.237197+0000","last_peered":"2024-02-09T17:49:42.237197+0000","last_clean":"2024-02-09T17:49:42.237197+0000","last_became_active":"2024-02-09T17:46:47.013920+0000","last_became_peered":"2024-02-09T17:46:47.013920+0000","last_unstale":"2024-02-09T17:49:42.237197+0000","last_undegraded":"2024-02-09T17:49:42.237197+0000","last_fullsized":"2024-02-09T17:49:42.237197+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:48:35.170638+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:48:35.170638+0000","last_clean_scrub_stamp":"2024-02-09T17:48:35.170638+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:49:56.897817+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"63'139","reported_seq":1144,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-09T17:49:42.227039+0000","last_change":"2024-02-09T17:49:31.135888+0000","last_active":"2024-02-09T17:49:42.227039+0000","last_peered":"2024-02-09T17:49:42.227039+0000","last_clean":"2024-02-09T17:49:42.227039+0000","last_became_active":"2024-02-09T17:34:01.353171+0000","last_became_peered":"2024-02-09T17:34:01.353171+0000","last_unstale":"2024-02-09T17:49:42.227039+0000","last_undegraded":"2024-02-09T17:49:42.227039+0000","last_fullsized":"2024-02-09T17:49:42.227039+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:49:31.135755+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:44:51.190789+0000","last_clean_scrub_stamp":"2024-02-09T17:49:31.135755+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:58.374348+0000","scrub_duration":343,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.12","version":"63'139","reported_seq":1229,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-09T17:49:42.225340+0000","last_change":"2024-02-09T17:48:18.579925+0000","last_active":"2024-02-09T17:49:42.225340+0000","last_peered":"2024-02-09T17:49:42.225340+0000","last_clean":"2024-02-09T17:49:42.225340+0000","last_became_active":"2024-02-09T17:46:47.015622+0000","last_became_peered":"2024-02-09T17:46:47.015622+0000","last_unstale":"2024-02-09T17:49:42.225340+0000","last_undegraded":"2024-02-09T17:49:42.225340+0000","last_fullsized":"2024-02-09T17:49:42.225340+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:48:18.579786+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:48:18.579786+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:49:47.349100+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.13","version":"63'139","reported_seq":848,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-09T17:49:42.227563+0000","last_change":"2024-02-09T17:49:20.518555+0000","last_active":"2024-02-09T17:49:42.227563+0000","last_peered":"2024-02-09T17:49:42.227563+0000","last_clean":"2024-02-09T17:49:42.227563+0000","last_became_active":"2024-02-09T17:41:14.521672+0000","last_became_peered":"2024-02-09T17:41:14.521672+0000","last_unstale":"2024-02-09T17:49:42.227563+0000","last_undegraded":"2024-02-09T17:49:42.227563+0000","last_fullsized":"2024-02-09T17:49:42.227563+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:49:20.518438+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:49:20.518438+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:36.344913+0000","scrub_duration":213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":682033176,"num_objects":10408,"num_object_clones":0,"num_object_copies":31224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10408,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":250350,"num_write_kb":8344193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1992306688,"data_stored":1992294472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160815,"ondisk_log_size":160815,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056971,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":680572,"kb_used_data":67700,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":93064580,"statfs":{"total":95995035648,"available":95298129920,"internally_reserved":0,"allocated":69324800,"data_stored":69027075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":627571140},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.6120000000000001,"5min":1.21,"15min":1.3089999999999999},"min":{"1min":0.24199999999999999,"5min":0.221,"15min":0.216},"max":{"1min":5.5679999999999996,"5min":5.8719999999999999,"15min":12.836},"last":0.67100000000000004},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.1080000000000001,"15min":1.2370000000000001},"min":{"1min":0.27500000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":4.5570000000000004,"5min":5.4189999999999996,"15min":9.8219999999999992},"last":2.5510000000000002}]},{"osd":1,"last update":"Fri Feb 9 17:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.7390000000000001,"5min":1.3799999999999999,"15min":1.4019999999999999},"min":{"1min":0.313,"5min":0.26300000000000001,"15min":0.22700000000000001},"max":{"1min":7.6909999999999998,"5min":7.7249999999999996,"15min":9.7710000000000008},"last":0.313},{"interface":"front","average":{"1min":1.728,"5min":1.6100000000000001,"15min":1.5129999999999999},"min":{"1min":0.29999999999999999,"5min":0.27700000000000002,"15min":0.26400000000000001},"max":{"1min":7.673,"5min":7.7110000000000003,"15min":9.718},"last":0.40600000000000003}]},{"osd":2,"last update":"Fri Feb 9 17:31:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.415,"15min":1.6859999999999999},"min":{"1min":0.378,"5min":0.247,"15min":0.247},"max":{"1min":4.4189999999999996,"5min":9.4719999999999995,"15min":9.4719999999999995},"last":0.35399999999999998},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.5149999999999999,"15min":1.4399999999999999},"min":{"1min":0.378,"5min":0.252,"15min":0.252},"max":{"1min":4.8579999999999997,"5min":7.96,"15min":7.96},"last":0.60399999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.7350000000000001,"15min":1.379},"min":{"1min":0.36799999999999999,"5min":0.248,"15min":0.248},"max":{"1min":4.6269999999999998,"5min":21.475000000000001,"15min":21.475000000000001},"last":0.32700000000000001},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.8380000000000001,"15min":1.6699999999999999},"min":{"1min":0.442,"5min":0.35599999999999998,"15min":0.29799999999999999},"max":{"1min":4.6769999999999996,"5min":21.498000000000001,"15min":21.498000000000001},"last":0.441}]},{"osd":4,"last update":"Fri Feb 9 17:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":2.044,"15min":1.881},"min":{"1min":0.31,"5min":0.28299999999999997,"15min":0.246},"max":{"1min":4.9089999999999998,"5min":65.109999999999999,"15min":65.109999999999999},"last":0.34499999999999997},{"interface":"front","average":{"1min":1.1990000000000001,"5min":2.254,"15min":1.74},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.309},"max":{"1min":4.899,"5min":65.099000000000004,"15min":65.099000000000004},"last":0.52000000000000002}]},{"osd":5,"last update":"Fri Feb 9 17:48:51 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":1.615,"15min":1.4750000000000001},"min":{"1min":0.30399999999999999,"5min":0.252,"15min":0.252},"max":{"1min":2.9740000000000002,"5min":18.878,"15min":18.878},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.82899999999999996,"5min":1.657,"15min":1.641},"min":{"1min":0.38100000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":2.9529999999999998,"5min":18.895,"15min":18.895},"last":0.39200000000000002}]},{"osd":6,"last update":"Fri Feb 9 17:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.512,"15min":1.645},"min":{"1min":0.31900000000000001,"5min":0.30399999999999999,"15min":0.29899999999999999},"max":{"1min":5.9710000000000001,"5min":9.1679999999999993,"15min":12.798999999999999},"last":2.4249999999999998},{"interface":"front","average":{"1min":1.123,"5min":1.6850000000000001,"15min":1.675},"min":{"1min":0.40899999999999997,"5min":0.36699999999999999,"15min":0.32700000000000001},"max":{"1min":5.9619999999999997,"5min":10.000999999999999,"15min":10.000999999999999},"last":2.4369999999999998}]},{"osd":7,"last update":"Fri Feb 9 17:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.8700000000000001,"15min":1.8340000000000001},"min":{"1min":0.38,"5min":0.32000000000000001,"15min":0.312},"max":{"1min":3.7429999999999999,"5min":18.846,"15min":20.489999999999998},"last":0.502},{"interface":"front","average":{"1min":1.8260000000000001,"5min":1.7969999999999999,"15min":1.758},"min":{"1min":0.42099999999999999,"5min":0.36799999999999999,"15min":0.34200000000000003},"max":{"1min":7.165,"5min":19.125,"15min":20.712},"last":0.57699999999999996}]},{"osd":8,"last update":"Fri Feb 9 17:49:01 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.79800000000000004,"15min":1.0860000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.3069999999999999,"5min":5.3739999999999997,"15min":9.5429999999999993},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.86599999999999999,"15min":1.145},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.318},"max":{"1min":1.3879999999999999,"5min":5.4139999999999997,"15min":9.2940000000000005},"last":0.63500000000000001}]},{"osd":9,"last update":"Fri Feb 9 17:49:32 2024","interfaces":[{"interface":"back","average":{"1min":1.6000000000000001,"5min":1.101,"15min":1.3759999999999999},"min":{"1min":0.40600000000000003,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":7.2720000000000002,"5min":12.27,"15min":62.518000000000001},"last":0.55000000000000004},{"interface":"front","average":{"1min":1.6679999999999999,"5min":1.1100000000000001,"15min":1.363},"min":{"1min":0.48599999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":7.3449999999999998,"5min":12.313000000000001,"15min":63.241999999999997},"last":0.65300000000000002}]},{"osd":10,"last update":"Fri Feb 9 17:48:51 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":1.1439999999999999,"15min":1.2749999999999999},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.2410000000000001,"5min":5.3799999999999999,"15min":9.6590000000000007},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.627,"5min":1.157,"15min":1.3500000000000001},"min":{"1min":0.437,"5min":0.377,"15min":0.35799999999999998},"max":{"1min":0.76700000000000002,"5min":5.5579999999999998,"15min":12.856},"last":0.68700000000000006}]}]},{"osd":10,"up_from":84,"seq":360777253200,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":607808,"kb_used_data":131000,"kb_used_omap":1,"kb_used_meta":476798,"kb_avail":93137344,"statfs":{"total":95995035648,"available":95372640256,"internally_reserved":0,"allocated":134144000,"data_stored":133842203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":488241600},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.0569999999999999,"15min":1.173},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.20599999999999999},"max":{"1min":4.7430000000000003,"5min":7.4290000000000003,"15min":9.343},"last":8.1669999999999998},{"interface":"front","average":{"1min":1.581,"5min":1.0680000000000001,"15min":1.085},"min":{"1min":0.28000000000000003,"5min":0.248,"15min":0.248},"max":{"1min":5.4169999999999998,"5min":7.2830000000000004,"15min":7.2830000000000004},"last":2.073}]},{"osd":1,"last update":"Fri Feb 9 17:49:00 2024","interfaces":[{"interface":"back","average":{"1min":1.7070000000000001,"5min":1.274,"15min":0.877},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.23999999999999999},"max":{"1min":9.3640000000000008,"5min":9.3640000000000008,"15min":9.3640000000000008},"last":4.3559999999999999},{"interface":"front","average":{"1min":1.958,"5min":1.357,"15min":1.0669999999999999},"min":{"1min":0.25900000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":8.9789999999999992,"5min":8.9789999999999992,"15min":8.9789999999999992},"last":1.857}]},{"osd":2,"last update":"Fri Feb 9 17:49:12 2024","interfaces":[{"interface":"back","average":{"1min":2.3039999999999998,"5min":1.742,"15min":1.5609999999999999},"min":{"1min":0.28699999999999998,"5min":0.28299999999999997,"15min":0.27100000000000002},"max":{"1min":7.1790000000000003,"5min":9.2539999999999996,"15min":11.384},"last":1.927},{"interface":"front","average":{"1min":1.625,"5min":1.7849999999999999,"15min":1.427},"min":{"1min":0.308,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":3.6520000000000001,"5min":38.868000000000002,"15min":38.868000000000002},"last":3.8969999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:49:00 2024","interfaces":[{"interface":"back","average":{"1min":2.0710000000000002,"5min":1.873,"15min":1.758},"min":{"1min":0.373,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":7.5880000000000001,"5min":35.113,"15min":35.113},"last":4.0750000000000002},{"interface":"front","average":{"1min":2.2810000000000001,"5min":2.117,"15min":1.9239999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":12.202999999999999,"5min":34.359000000000002,"15min":34.359000000000002},"last":4.125}]},{"osd":4,"last update":"Fri Feb 9 17:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":1.1519999999999999,"15min":1.161},"min":{"1min":0.33700000000000002,"5min":0.32100000000000001,"15min":0.29199999999999998},"max":{"1min":5.3739999999999997,"5min":9.0990000000000002,"15min":9.0990000000000002},"last":0.33800000000000002},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.1060000000000001,"15min":1.1220000000000001},"min":{"1min":0.41299999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":5.2949999999999999,"5min":7.0309999999999997,"15min":11.273999999999999},"last":0.48899999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:49:12 2024","interfaces":[{"interface":"back","average":{"1min":1.871,"5min":1.5329999999999999,"15min":1.2110000000000001},"min":{"1min":0.32800000000000001,"5min":0.29099999999999998,"15min":0.28799999999999998},"max":{"1min":7.2229999999999999,"5min":7.2229999999999999,"15min":7.5179999999999998},"last":0.505},{"interface":"front","average":{"1min":1.79,"5min":1.569,"15min":1.4119999999999999},"min":{"1min":0.39700000000000002,"5min":0.38800000000000001,"15min":0.33500000000000002},"max":{"1min":7.2590000000000003,"5min":10.243,"15min":10.243},"last":0.45700000000000002}]},{"osd":6,"last update":"Fri Feb 9 17:49:12 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":1.4259999999999999,"15min":1.4690000000000001},"min":{"1min":0.34200000000000003,"5min":0.33600000000000002,"15min":0.29999999999999999},"max":{"1min":5.5149999999999997,"5min":9.0820000000000007,"15min":10.962},"last":4.3140000000000001},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.4630000000000001,"15min":1.4299999999999999},"min":{"1min":0.38,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":5.1950000000000003,"5min":9.0690000000000008,"15min":9.0690000000000008},"last":3.544}]},{"osd":7,"last update":"Fri Feb 9 17:39:52 2024","interfaces":[{"interface":"back","average":{"1min":1.6000000000000001,"5min":1.367,"15min":1.415},"min":{"1min":0.42099999999999999,"5min":0.34200000000000003,"15min":0.312},"max":{"1min":4.0430000000000001,"5min":6.7590000000000003,"15min":17.754000000000001},"last":4.6449999999999996},{"interface":"front","average":{"1min":1.498,"5min":1.4359999999999999,"15min":1.5189999999999999},"min":{"1min":0.47399999999999998,"5min":0.38300000000000001,"15min":0.316},"max":{"1min":5.8170000000000002,"5min":11.308999999999999,"15min":11.308999999999999},"last":0.55000000000000004}]},{"osd":8,"last update":"Fri Feb 9 17:49:00 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":0.91900000000000004,"15min":1.0409999999999999},"min":{"1min":0.32000000000000001,"5min":0.312,"15min":0.29099999999999998},"max":{"1min":4.6550000000000002,"5min":6.9749999999999996,"15min":7.4359999999999999},"last":0.42399999999999999},{"interface":"front","average":{"1min":1.105,"5min":1.147,"15min":1.0880000000000001},"min":{"1min":0.36899999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":5.319,"5min":7.0460000000000003,"15min":7.7060000000000004},"last":0.53300000000000003}]},{"osd":9,"last update":"Fri Feb 9 17:49:12 2024","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":1.0109999999999999,"15min":1.0860000000000001},"min":{"1min":0.35299999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":4.7000000000000002,"5min":7.0419999999999998,"15min":7.4850000000000003},"last":0.44},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.1970000000000001,"15min":1.24},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.35899999999999999},"max":{"1min":6.585,"5min":7.1299999999999999,"15min":10.622999999999999},"last":0.47799999999999998}]},{"osd":11,"last update":"Fri Feb 9 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":0.873,"15min":0.96299999999999997},"min":{"1min":0.40500000000000003,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":3.5990000000000002,"5min":3.5990000000000002,"15min":6.226},"last":0.40899999999999997},{"interface":"front","average":{"1min":1.0669999999999999,"5min":0.92500000000000004,"15min":1.0089999999999999},"min":{"1min":0.48299999999999998,"5min":0.438,"15min":0.438},"max":{"1min":3.625,"5min":3.625,"15min":6.2590000000000003},"last":0.52100000000000002}]}]},{"osd":9,"up_from":77,"seq":330712482133,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":755772,"kb_used_data":183092,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":92989380,"statfs":{"total":95995035648,"available":95221125120,"internally_reserved":0,"allocated":187486208,"data_stored":187188483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":586348992},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:49:08 2024","interfaces":[{"interface":"back","average":{"1min":2.113,"5min":1.248,"15min":0.874},"min":{"1min":0.252,"5min":0.215,"15min":0.215},"max":{"1min":7.1289999999999996,"5min":8.4670000000000005,"15min":8.4670000000000005},"last":3.2250000000000001},{"interface":"front","average":{"1min":1.278,"5min":1.5740000000000001,"15min":1.179},"min":{"1min":0.32000000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":3.4569999999999999,"5min":11.596,"15min":11.596},"last":0.30399999999999999}]},{"osd":1,"last update":"Fri Feb 9 17:48:49 2024","interfaces":[{"interface":"back","average":{"1min":1.863,"5min":1.411,"15min":1.786},"min":{"1min":0.30499999999999999,"5min":0.26500000000000001,"15min":0.23499999999999999},"max":{"1min":5.835,"5min":8.1310000000000002,"15min":89.515000000000001},"last":0.28599999999999998},{"interface":"front","average":{"1min":1.788,"5min":1.1899999999999999,"15min":1.6970000000000001},"min":{"1min":0.31,"5min":0.25,"15min":0.249},"max":{"1min":11.685,"5min":11.685,"15min":89.388000000000005},"last":0.40200000000000002}]},{"osd":2,"last update":"Fri Feb 9 17:49:20 2024","interfaces":[{"interface":"back","average":{"1min":2.2069999999999999,"5min":1.8109999999999999,"15min":1.7410000000000001},"min":{"1min":0.307,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":5.9370000000000003,"5min":5.9370000000000003,"15min":5.9370000000000003},"last":0.26900000000000002},{"interface":"front","average":{"1min":2.4289999999999998,"5min":1.736,"15min":1.6459999999999999},"min":{"1min":0.35399999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":8.4250000000000007,"5min":8.4250000000000007,"15min":8.4250000000000007},"last":4.3869999999999996}]},{"osd":3,"last update":"Fri Feb 9 17:45:45 2024","interfaces":[{"interface":"back","average":{"1min":1.569,"5min":1.516,"15min":1.671},"min":{"1min":0.36899999999999999,"5min":0.26500000000000001,"15min":0.25700000000000001},"max":{"1min":6.3789999999999996,"5min":8.9939999999999998,"15min":40.18},"last":0.31},{"interface":"front","average":{"1min":1.264,"5min":1.079,"15min":1.556},"min":{"1min":0.33300000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":6.048,"5min":7.2519999999999998,"15min":40.158000000000001},"last":0.42299999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:48:49 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":1.194,"15min":1.425},"min":{"1min":0.28199999999999997,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.8260000000000001,"5min":7.8899999999999997,"15min":27.381},"last":1.115},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.04,"15min":1.2969999999999999},"min":{"1min":0.34300000000000003,"5min":0.26900000000000002,"15min":0.26100000000000001},"max":{"1min":5.415,"5min":5.415,"15min":27.355},"last":0.35099999999999998}]},{"osd":5,"last update":"Fri Feb 9 17:49:00 2024","interfaces":[{"interface":"back","average":{"1min":1.5389999999999999,"5min":2.1000000000000001,"15min":1.6479999999999999},"min":{"1min":0.33700000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":5.8499999999999996,"5min":77.924000000000007,"15min":77.924000000000007},"last":0.33000000000000002},{"interface":"front","average":{"1min":1.0629999999999999,"5min":2.1880000000000002,"15min":1.663},"min":{"1min":0.371,"5min":0.318,"15min":0.318},"max":{"1min":3.7149999999999999,"5min":77.899000000000001,"15min":77.899000000000001},"last":0.38700000000000001}]},{"osd":6,"last update":"Fri Feb 9 17:48:49 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.4339999999999999,"15min":1.5069999999999999},"min":{"1min":0.34799999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":4.5620000000000003,"5min":8.8559999999999999,"15min":14.069000000000001},"last":5.0640000000000001},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.8029999999999999,"15min":1.6839999999999999},"min":{"1min":0.38500000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":5.0549999999999997,"5min":9.5250000000000004,"15min":14.097},"last":8.0860000000000003}]},{"osd":7,"last update":"Fri Feb 9 17:48:49 2024","interfaces":[{"interface":"back","average":{"1min":1.6319999999999999,"5min":2.2280000000000002,"15min":1.899},"min":{"1min":0.41499999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":5.4859999999999998,"5min":29.001999999999999,"15min":29.001999999999999},"last":0.83699999999999997},{"interface":"front","average":{"1min":1.6220000000000001,"5min":2.2040000000000002,"15min":1.907},"min":{"1min":0.434,"5min":0.373,"15min":0.32200000000000001},"max":{"1min":5.4720000000000004,"5min":28.872,"15min":28.872},"last":0.84899999999999998}]},{"osd":8,"last update":"Fri Feb 9 17:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.72899999999999998,"15min":1.1040000000000001},"min":{"1min":0.38200000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":4.2160000000000002,"5min":4.2160000000000002,"15min":14.045999999999999},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.0189999999999999,"15min":1.2470000000000001},"min":{"1min":0.38100000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":3.5310000000000001,"5min":12.885999999999999,"15min":12.885999999999999},"last":0.42599999999999999}]},{"osd":10,"last update":"Fri Feb 9 17:49:20 2024","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.25,"15min":1.331},"min":{"1min":0.40100000000000002,"5min":0.313,"15min":0.312},"max":{"1min":5.8259999999999996,"5min":11.404,"15min":11.404},"last":0.435},{"interface":"front","average":{"1min":1.115,"5min":0.878,"15min":1.159},"min":{"1min":0.40699999999999997,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":3.7040000000000002,"5min":4.5,"15min":10.904},"last":0.44900000000000001}]},{"osd":11,"last update":"Fri Feb 9 17:49:20 2024","interfaces":[{"interface":"back","average":{"1min":1.425,"5min":0.98799999999999999,"15min":1.3200000000000001},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":5.8010000000000002,"5min":7.3330000000000002,"15min":14.02},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.83899999999999997,"15min":1.258},"min":{"1min":0.42099999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":3.9940000000000002,"5min":6.9690000000000003,"15min":10.863},"last":0.55000000000000004}]}]},{"osd":8,"up_from":171,"seq":734439407876,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":376700,"kb_used_data":86964,"kb_used_omap":212,"kb_used_meta":289515,"kb_avail":93368452,"statfs":{"total":95995035648,"available":95609294848,"internally_reserved":0,"allocated":89051136,"data_stored":88753411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217579,"internal_metadata":296463893},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:48:55 2024","interfaces":[{"interface":"back","average":{"1min":1.4810000000000001,"5min":0.93000000000000005,"15min":0.97999999999999998},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.217},"max":{"1min":7.867,"5min":7.867,"15min":8.0920000000000005},"last":0.42299999999999999},{"interface":"front","average":{"1min":1.887,"5min":1.1220000000000001,"15min":1.282},"min":{"1min":0.31900000000000001,"5min":0.23999999999999999,"15min":0.23100000000000001},"max":{"1min":12.045,"5min":12.045,"15min":12.045},"last":0.89000000000000001}]},{"osd":1,"last update":"Fri Feb 9 17:48:55 2024","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.1679999999999999,"15min":1.355},"min":{"1min":0.28699999999999998,"5min":0.252,"15min":0.22700000000000001},"max":{"1min":7.3280000000000003,"5min":7.3280000000000003,"15min":8.718},"last":2.242},{"interface":"front","average":{"1min":1.5660000000000001,"5min":1.212,"15min":1.5229999999999999},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.23300000000000001},"max":{"1min":6.3179999999999996,"5min":6.3179999999999996,"15min":25.027000000000001},"last":2.032}]},{"osd":2,"last update":"Fri Feb 9 17:49:34 2024","interfaces":[{"interface":"back","average":{"1min":1.796,"5min":1.4870000000000001,"15min":1.3260000000000001},"min":{"1min":0.29199999999999998,"5min":0.28199999999999997,"15min":0.28000000000000003},"max":{"1min":4.4619999999999997,"5min":9.0060000000000002,"15min":9.0060000000000002},"last":0.46999999999999997},{"interface":"front","average":{"1min":1.9650000000000001,"5min":1.417,"15min":1.0109999999999999},"min":{"1min":0.35499999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":4.5170000000000003,"5min":5.2930000000000001,"15min":5.2930000000000001},"last":2.3769999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:42:38 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":1.542,"15min":1.6379999999999999},"min":{"1min":0.316,"5min":0.29499999999999998,"15min":0.28699999999999998},"max":{"1min":2.347,"5min":15.228999999999999,"15min":24.030999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.75600000000000001,"5min":1.8340000000000001,"15min":1.8640000000000001},"min":{"1min":0.36199999999999999,"5min":0.33600000000000002,"15min":0.30299999999999999},"max":{"1min":2.3719999999999999,"5min":14.988,"15min":24.318999999999999},"last":0.69399999999999995}]},{"osd":4,"last update":"Fri Feb 9 17:48:55 2024","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.1910000000000001,"15min":1.5249999999999999},"min":{"1min":0.35299999999999998,"5min":0.30499999999999999,"15min":0.28199999999999997},"max":{"1min":3.98,"5min":7.0369999999999999,"15min":53.060000000000002},"last":0.51500000000000001},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.5649999999999999,"15min":1.647},"min":{"1min":0.35599999999999998,"5min":0.28299999999999997,"15min":0.27000000000000002},"max":{"1min":3.577,"5min":8.2360000000000007,"15min":53.311999999999998},"last":1.8620000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.133,"15min":2.0230000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":3.165,"5min":5.0819999999999999,"15min":6.4249999999999998},"last":0.54300000000000004},{"interface":"front","average":{"1min":1.534,"5min":1.2470000000000001,"15min":1.867},"min":{"1min":0.42299999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":5.2649999999999997,"5min":5.3940000000000001,"15min":6.383},"last":0.59099999999999997}]},{"osd":6,"last update":"Fri Feb 9 17:48:55 2024","interfaces":[{"interface":"back","average":{"1min":1.919,"5min":1.6180000000000001,"15min":1.4199999999999999},"min":{"1min":0.36199999999999999,"5min":0.29899999999999999,"15min":0.29799999999999999},"max":{"1min":5.8179999999999996,"5min":13.031000000000001,"15min":13.031000000000001},"last":1.603},{"interface":"front","average":{"1min":1.9239999999999999,"5min":1.5940000000000001,"15min":1.444},"min":{"1min":0.34200000000000003,"5min":0.32000000000000001,"15min":0.29599999999999999},"max":{"1min":7.016,"5min":12.157999999999999,"15min":12.157999999999999},"last":0.501}]},{"osd":7,"last update":"Fri Feb 9 17:48:55 2024","interfaces":[{"interface":"back","average":{"1min":1.865,"5min":1.784,"15min":1.647},"min":{"1min":0.377,"5min":0.34100000000000003,"15min":0.32800000000000001},"max":{"1min":6.766,"5min":11.278,"15min":11.278},"last":0.56000000000000005},{"interface":"front","average":{"1min":2.3839999999999999,"5min":1.944,"15min":1.7250000000000001},"min":{"1min":0.42599999999999999,"5min":0.38700000000000001,"15min":0.38500000000000001},"max":{"1min":7.7709999999999999,"5min":11.313000000000001,"15min":11.313000000000001},"last":1.9139999999999999}]},{"osd":9,"last update":"Fri Feb 9 17:48:55 2024","interfaces":[{"interface":"back","average":{"1min":1.4710000000000001,"5min":0.98899999999999999,"15min":1.2310000000000001},"min":{"1min":0.38800000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":5.6529999999999996,"5min":5.6529999999999996,"15min":45.616},"last":0.60999999999999999},{"interface":"front","average":{"1min":1.579,"5min":1.1259999999999999,"15min":1.363},"min":{"1min":0.504,"5min":0.374,"15min":0.314},"max":{"1min":5.694,"5min":7.5609999999999999,"15min":45.718000000000004},"last":0.63800000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:48:55 2024","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":0.90700000000000003,"15min":0.95299999999999996},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.29699999999999999},"max":{"1min":4.3380000000000001,"5min":7.8360000000000003,"15min":7.8360000000000003},"last":0.72499999999999998},{"interface":"front","average":{"1min":1.2909999999999999,"5min":0.875,"15min":1.1579999999999999},"min":{"1min":0.40600000000000003,"5min":0.34300000000000003,"15min":0.318},"max":{"1min":4.5270000000000001,"5min":4.9690000000000003,"15min":11.071},"last":0.70199999999999996}]},{"osd":11,"last update":"Fri Feb 9 17:48:55 2024","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.0900000000000001,"15min":1.2010000000000001},"min":{"1min":0.47099999999999997,"5min":0.40899999999999997,"15min":0.34000000000000002},"max":{"1min":3.7149999999999999,"5min":7.6079999999999997,"15min":14.433999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.1020000000000001,"15min":1.2330000000000001},"min":{"1min":0.39800000000000002,"5min":0.32900000000000001,"15min":0.308},"max":{"1min":6.5720000000000001,"5min":6.5720000000000001,"15min":7.6200000000000001},"last":0.622}]}]},{"osd":7,"up_from":57,"seq":244813136221,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":879428,"kb_used_data":208252,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":92865724,"statfs":{"total":95995035648,"available":95094501376,"internally_reserved":0,"allocated":213250048,"data_stored":212948771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":687274442},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:49:37 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.6850000000000001,"15min":1.635},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":8.6289999999999996,"5min":11.047000000000001,"15min":36.753},"last":0.68200000000000005},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.651,"15min":1.71},"min":{"1min":0.27400000000000002,"5min":0.26800000000000002,"15min":0.24099999999999999},"max":{"1min":4.1340000000000003,"5min":11.095000000000001,"15min":36.918999999999997},"last":0.64300000000000002}]},{"osd":1,"last update":"Fri Feb 9 17:49:37 2024","interfaces":[{"interface":"back","average":{"1min":1.597,"5min":1.522,"15min":1.9870000000000001},"min":{"1min":0.30599999999999999,"5min":0.29199999999999998,"15min":0.24199999999999999},"max":{"1min":7.827,"5min":7.827,"15min":21.704999999999998},"last":3.9990000000000001},{"interface":"front","average":{"1min":1.7849999999999999,"5min":1.744,"15min":2.1549999999999998},"min":{"1min":0.31900000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":8.6579999999999995,"5min":8.6579999999999995,"15min":21.858000000000001},"last":0.439}]},{"osd":2,"last update":"Fri Feb 9 17:49:37 2024","interfaces":[{"interface":"back","average":{"1min":1.96,"5min":1.7230000000000001,"15min":1.911},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":5.8659999999999997,"5min":6.4850000000000003,"15min":6.6340000000000003},"last":0.745},{"interface":"front","average":{"1min":2.0129999999999999,"5min":1.7729999999999999,"15min":2.2360000000000002},"min":{"1min":0.33500000000000002,"5min":0.308,"15min":0.308},"max":{"1min":10.159000000000001,"5min":10.159000000000001,"15min":11.968},"last":4.5510000000000002}]},{"osd":3,"last update":"Fri Feb 9 17:48:55 2024","interfaces":[{"interface":"back","average":{"1min":6.9939999999999998,"5min":2.7989999999999999,"15min":2.161},"min":{"1min":0.39300000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":78.644999999999996,"5min":78.644999999999996,"15min":78.644999999999996},"last":5.9420000000000002},{"interface":"front","average":{"1min":6.1369999999999996,"5min":3.1040000000000001,"15min":2.3450000000000002},"min":{"1min":0.38300000000000001,"5min":0.35499999999999998,"15min":0.308},"max":{"1min":78.414000000000001,"5min":78.414000000000001,"15min":78.414000000000001},"last":5.6840000000000002}]},{"osd":4,"last update":"Fri Feb 9 17:49:37 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":2.1349999999999998,"15min":1.407},"min":{"1min":0.309,"5min":0.25600000000000001,"15min":0.23699999999999999},"max":{"1min":1.875,"5min":86.813000000000002,"15min":86.813000000000002},"last":0.46700000000000003},{"interface":"front","average":{"1min":2.0419999999999998,"5min":2.3700000000000001,"15min":1.534},"min":{"1min":0.30399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":14.087999999999999,"5min":86.751000000000005,"15min":86.751000000000005},"last":0.51900000000000002}]},{"osd":5,"last update":"Fri Feb 9 17:49:37 2024","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.5569999999999999,"15min":1.3660000000000001},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.249},"max":{"1min":6.4000000000000004,"5min":8.0060000000000002,"15min":9.6590000000000007},"last":0.55100000000000005},{"interface":"front","average":{"1min":1.7769999999999999,"5min":1.6890000000000001,"15min":1.4239999999999999},"min":{"1min":0.34699999999999998,"5min":0.32100000000000001,"15min":0.31900000000000001},"max":{"1min":7.867,"5min":8.8610000000000007,"15min":15.228999999999999},"last":0.48899999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:48:55 2024","interfaces":[{"interface":"back","average":{"1min":1.722,"5min":1.9339999999999999,"15min":1.8560000000000001},"min":{"1min":0.47999999999999998,"5min":0.35199999999999998,"15min":0.26000000000000001},"max":{"1min":7.8879999999999999,"5min":8.9969999999999999,"15min":15.275},"last":0.45500000000000002},{"interface":"front","average":{"1min":1.8640000000000001,"5min":1.788,"15min":1.7190000000000001},"min":{"1min":0.51900000000000002,"5min":0.32000000000000001,"15min":0.29199999999999998},"max":{"1min":5.9660000000000002,"5min":8.9779999999999998,"15min":18.236999999999998},"last":0.58699999999999997}]},{"osd":8,"last update":"Fri Feb 9 17:48:55 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":1.395,"15min":1.454},"min":{"1min":0.375,"5min":0.375,"15min":0.28999999999999998},"max":{"1min":6.7080000000000002,"5min":7.8120000000000003,"15min":11.81},"last":0.627},{"interface":"front","average":{"1min":1.131,"5min":1.5640000000000001,"15min":1.4690000000000001},"min":{"1min":0.433,"5min":0.38700000000000001,"15min":0.33300000000000002},"max":{"1min":7.7709999999999999,"5min":7.8949999999999996,"15min":12.058},"last":0.69199999999999995}]},{"osd":9,"last update":"Fri Feb 9 17:49:43 2024","interfaces":[{"interface":"back","average":{"1min":2.0249999999999999,"5min":1.5029999999999999,"15min":1.5289999999999999},"min":{"1min":0.42999999999999999,"5min":0.375,"15min":0.32700000000000001},"max":{"1min":11.542999999999999,"5min":11.542999999999999,"15min":11.941000000000001},"last":0.71999999999999997},{"interface":"front","average":{"1min":2.2450000000000001,"5min":1.631,"15min":1.659},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.33400000000000002},"max":{"1min":11.635999999999999,"5min":11.635999999999999,"15min":11.906000000000001},"last":0.65500000000000003}]},{"osd":10,"last update":"Fri Feb 9 17:49:08 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.3520000000000001,"15min":1.4910000000000001},"min":{"1min":0.53200000000000003,"5min":0.38700000000000001,"15min":0.35099999999999998},"max":{"1min":3.8799999999999999,"5min":7.165,"15min":12.050000000000001},"last":0.67100000000000004},{"interface":"front","average":{"1min":1.639,"5min":1.5900000000000001,"15min":1.6379999999999999},"min":{"1min":0.46800000000000003,"5min":0.41999999999999998,"15min":0.38900000000000001},"max":{"1min":6.734,"5min":7.8739999999999997,"15min":11.878},"last":0.754}]},{"osd":11,"last update":"Fri Feb 9 17:49:37 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":1.4279999999999999,"15min":1.6339999999999999},"min":{"1min":0.48499999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.9710000000000001,"5min":9.984,"15min":21.945},"last":4.4480000000000004},{"interface":"front","average":{"1min":1.468,"5min":1.385,"15min":1.7330000000000001},"min":{"1min":0.46400000000000002,"5min":0.438,"15min":0.40699999999999997},"max":{"1min":7.8019999999999996,"5min":7.8019999999999996,"15min":21.731999999999999},"last":4.3280000000000003}]}]},{"osd":6,"up_from":49,"seq":210453397858,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":781372,"kb_used_data":200372,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":92963780,"statfs":{"total":95995035648,"available":95194910720,"internally_reserved":0,"allocated":205180928,"data_stored":204883203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":594868678},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:49:25 2024","interfaces":[{"interface":"back","average":{"1min":2.2519999999999998,"5min":1.585,"15min":1.7150000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.21299999999999999},"max":{"1min":10.481999999999999,"5min":10.481999999999999,"15min":12.544},"last":1.0109999999999999},{"interface":"front","average":{"1min":2.1299999999999999,"5min":1.706,"15min":1.7250000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":5.9050000000000002,"5min":7.0860000000000003,"15min":9.2520000000000007},"last":4.9640000000000004}]},{"osd":1,"last update":"Fri Feb 9 17:49:25 2024","interfaces":[{"interface":"back","average":{"1min":3.3519999999999999,"5min":3.7309999999999999,"15min":2.6320000000000001},"min":{"1min":0.28499999999999998,"5min":0.252,"15min":0.252},"max":{"1min":11.090999999999999,"5min":79.945999999999998,"15min":79.945999999999998},"last":3.3479999999999999},{"interface":"front","average":{"1min":3.383,"5min":3.8149999999999999,"15min":2.7130000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.29899999999999999},"max":{"1min":11.132999999999999,"5min":80.003,"15min":80.003},"last":3.3700000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:49:25 2024","interfaces":[{"interface":"back","average":{"1min":2.2869999999999999,"5min":3.1640000000000001,"15min":2.117},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":5.3070000000000004,"5min":84.912999999999997,"15min":84.912999999999997},"last":1.8919999999999999},{"interface":"front","average":{"1min":2.3220000000000001,"5min":3.1949999999999998,"15min":2.222},"min":{"1min":0.35499999999999998,"5min":0.34000000000000002,"15min":0.315},"max":{"1min":5.3440000000000003,"5min":84.942999999999998,"15min":84.942999999999998},"last":1.9139999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:49:25 2024","interfaces":[{"interface":"back","average":{"1min":4.7140000000000004,"5min":4.1890000000000001,"15min":2.6949999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":19.555,"5min":80.174999999999997,"15min":80.174999999999997},"last":2.9159999999999999},{"interface":"front","average":{"1min":5.6909999999999998,"5min":4.7679999999999998,"15min":2.887},"min":{"1min":0.41299999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":19.568000000000001,"5min":80.417000000000002,"15min":80.417000000000002},"last":2.9289999999999998}]},{"osd":4,"last update":"Fri Feb 9 17:49:25 2024","interfaces":[{"interface":"back","average":{"1min":3.54,"5min":2.262,"15min":1.909},"min":{"1min":0.36499999999999999,"5min":0.27900000000000003,"15min":0.23599999999999999},"max":{"1min":14.182,"5min":14.182,"15min":19.798999999999999},"last":0.27900000000000003},{"interface":"front","average":{"1min":3.5899999999999999,"5min":2.1309999999999998,"15min":1.9510000000000001},"min":{"1min":0.39800000000000002,"5min":0.29299999999999998,"15min":0.26400000000000001},"max":{"1min":14.210000000000001,"5min":14.210000000000001,"15min":15.118},"last":0.36699999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:49:25 2024","interfaces":[{"interface":"back","average":{"1min":3.6760000000000002,"5min":2.2749999999999999,"15min":1.9039999999999999},"min":{"1min":0.374,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":14.151999999999999,"5min":14.151999999999999,"15min":14.151999999999999},"last":0.33000000000000002},{"interface":"front","average":{"1min":2.1219999999999999,"5min":2.351,"15min":1.8580000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":7.6630000000000003,"5min":18.315999999999999,"15min":18.315999999999999},"last":0.34599999999999997}]},{"osd":7,"last update":"Fri Feb 9 17:49:00 2024","interfaces":[{"interface":"back","average":{"1min":1.6259999999999999,"5min":2.0470000000000002,"15min":1.9019999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":5.4139999999999997,"5min":23.265999999999998,"15min":23.265999999999998},"last":0.40000000000000002},{"interface":"front","average":{"1min":1.8169999999999999,"5min":2.0059999999999998,"15min":1.877},"min":{"1min":0.35299999999999998,"5min":0.314,"15min":0.314},"max":{"1min":5.3049999999999997,"5min":10.599,"15min":14.356999999999999},"last":0.46000000000000002}]},{"osd":8,"last update":"Fri Feb 9 17:26:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":0.876,"15min":0.79600000000000004},"min":{"1min":0.39900000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":4.0339999999999998,"5min":4.4880000000000004,"15min":4.4880000000000004},"last":4.2809999999999997},{"interface":"front","average":{"1min":1.0469999999999999,"5min":0.88600000000000001,"15min":0.78300000000000003},"min":{"1min":0.375,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":3.6539999999999999,"5min":4.8570000000000002,"15min":4.8570000000000002},"last":2.9529999999999998}]},{"osd":9,"last update":"Fri Feb 9 17:49:30 2024","interfaces":[{"interface":"back","average":{"1min":1.5369999999999999,"5min":1.401,"15min":1.524},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":5.2569999999999997,"5min":8.3759999999999994,"15min":10.779999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":1.569,"5min":1.4359999999999999,"15min":1.6000000000000001},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":5.1879999999999997,"5min":8.4130000000000003,"15min":8.4130000000000003},"last":0.42999999999999999}]},{"osd":10,"last update":"Fri Feb 9 17:49:00 2024","interfaces":[{"interface":"back","average":{"1min":1.621,"5min":1.5640000000000001,"15min":1.643},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34300000000000003},"max":{"1min":5.4829999999999997,"5min":8.7289999999999992,"15min":25.370000000000001},"last":0.49099999999999999},{"interface":"front","average":{"1min":2.5310000000000001,"5min":1.917,"15min":1.7569999999999999},"min":{"1min":0.45400000000000001,"5min":0.36399999999999999,"15min":0.35099999999999998},"max":{"1min":10.239000000000001,"5min":11.821999999999999,"15min":26.027999999999999},"last":0.41499999999999998}]},{"osd":11,"last update":"Fri Feb 9 17:49:25 2024","interfaces":[{"interface":"back","average":{"1min":1.903,"5min":1.6040000000000001,"15min":1.619},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":5.5110000000000001,"5min":8.7650000000000006,"15min":13.109},"last":0.44400000000000001},{"interface":"front","average":{"1min":1.978,"5min":1.7010000000000001,"15min":1.609},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.373},"max":{"1min":5.5410000000000004,"5min":8.7889999999999997,"15min":13.164999999999999},"last":0.47099999999999997}]}]},{"osd":1,"up_from":13,"seq":55834575229,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":831232,"kb_used_data":146168,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":92913920,"statfs":{"total":95995035648,"available":95143854080,"internally_reserved":0,"allocated":149676032,"data_stored":149374235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:49:43 2024","interfaces":[{"interface":"back","average":{"1min":2.077,"5min":1.7669999999999999,"15min":1.8140000000000001},"min":{"1min":0.23499999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.9420000000000002,"5min":9.8529999999999998,"15min":19.079000000000001},"last":4.4199999999999999},{"interface":"front","average":{"1min":2.085,"5min":1.978,"15min":1.782},"min":{"1min":0.214,"5min":0.214,"15min":0.19400000000000001},"max":{"1min":11.956,"5min":11.956,"15min":23.573},"last":1.8680000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:48:59 2024","interfaces":[{"interface":"back","average":{"1min":2.3180000000000001,"5min":2.0390000000000001,"15min":1.7869999999999999},"min":{"1min":0.26100000000000001,"5min":0.24399999999999999,"15min":0.23100000000000001},"max":{"1min":11.301,"5min":11.301,"15min":26.361999999999998},"last":0.44700000000000001},{"interface":"front","average":{"1min":2.3740000000000001,"5min":1.732,"15min":1.8360000000000001},"min":{"1min":0.31,"5min":0.222,"15min":0.222},"max":{"1min":8.9049999999999994,"5min":8.9049999999999994,"15min":26.087},"last":1.4159999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:49:32 2024","interfaces":[{"interface":"back","average":{"1min":4.4530000000000003,"5min":2.5289999999999999,"15min":1.875},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":16.125,"5min":17.959,"15min":17.959},"last":2.0449999999999999},{"interface":"front","average":{"1min":4.6349999999999998,"5min":3.1040000000000001,"15min":1.7090000000000001},"min":{"1min":0.38600000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":19.689,"5min":22.866,"15min":22.866},"last":1.4730000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:49:41 2024","interfaces":[{"interface":"back","average":{"1min":1.804,"5min":2.1739999999999999,"15min":1.855},"min":{"1min":0.312,"5min":0.30299999999999999,"15min":0.253},"max":{"1min":7.8490000000000002,"5min":45.137999999999998,"15min":45.137999999999998},"last":3.347},{"interface":"front","average":{"1min":1.641,"5min":2.0640000000000001,"15min":2.0150000000000001},"min":{"1min":0.39900000000000002,"5min":0.29399999999999998,"15min":0.28999999999999998},"max":{"1min":5.7430000000000003,"5min":44.950000000000003,"15min":44.950000000000003},"last":2.5310000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":2.1339999999999999,"15min":1.9379999999999999},"min":{"1min":0.35599999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":4.7590000000000003,"5min":45.171999999999997,"15min":45.171999999999997},"last":1.6899999999999999},{"interface":"front","average":{"1min":1.6919999999999999,"5min":2.355,"15min":2.048},"min":{"1min":0.31900000000000001,"5min":0.308,"15min":0.26800000000000002},"max":{"1min":11.763,"5min":45.127000000000002,"15min":45.127000000000002},"last":1.7050000000000001}]},{"osd":6,"last update":"Fri Feb 9 17:49:32 2024","interfaces":[{"interface":"back","average":{"1min":2.379,"5min":2.0739999999999998,"15min":1.643},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":11.808,"5min":11.808,"15min":12.57},"last":3.6230000000000002},{"interface":"front","average":{"1min":2.1829999999999998,"5min":2.2130000000000001,"15min":1.651},"min":{"1min":0.40000000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":11.932,"5min":11.932,"15min":12.605},"last":1.7490000000000001}]},{"osd":7,"last update":"Fri Feb 9 17:49:32 2024","interfaces":[{"interface":"back","average":{"1min":2.093,"5min":2.6139999999999999,"15min":1.7210000000000001},"min":{"1min":0.45900000000000002,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":11.834,"5min":45.740000000000002,"15min":45.740000000000002},"last":3.7240000000000002},{"interface":"front","average":{"1min":2.1400000000000001,"5min":2.6459999999999999,"15min":1.8120000000000001},"min":{"1min":0.48799999999999999,"5min":0.39600000000000002,"15min":0.309},"max":{"1min":11.856999999999999,"5min":45.792000000000002,"15min":45.792000000000002},"last":3.7490000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:49:41 2024","interfaces":[{"interface":"back","average":{"1min":1.5880000000000001,"5min":1.6259999999999999,"15min":1.161},"min":{"1min":0.51900000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":5.7839999999999998,"5min":8.9440000000000008,"15min":9.2520000000000007},"last":0.68400000000000005},{"interface":"front","average":{"1min":1.919,"5min":1.6930000000000001,"15min":1.1739999999999999},"min":{"1min":0.39000000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":12.69,"5min":12.69,"15min":12.69},"last":0.47999999999999998}]},{"osd":9,"last update":"Fri Feb 9 17:49:43 2024","interfaces":[{"interface":"back","average":{"1min":1.7370000000000001,"5min":1.778,"15min":1.595},"min":{"1min":0.45100000000000001,"5min":0.39300000000000002,"15min":0.33400000000000002},"max":{"1min":5.8140000000000001,"5min":9.9359999999999999,"15min":9.9359999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":1.522,"5min":1.762,"15min":1.772},"min":{"1min":0.41399999999999998,"5min":0.34899999999999998,"15min":0.32900000000000001},"max":{"1min":5.2140000000000004,"5min":7.7199999999999998,"15min":11.366},"last":1.4970000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:49:24 2024","interfaces":[{"interface":"back","average":{"1min":1.8620000000000001,"5min":1.885,"15min":1.8089999999999999},"min":{"1min":0.46899999999999997,"5min":0.432,"15min":0.39200000000000002},"max":{"1min":11.880000000000001,"5min":11.880000000000001,"15min":15.083},"last":5.9660000000000002},{"interface":"front","average":{"1min":1.423,"5min":1.7729999999999999,"15min":1.806},"min":{"1min":0.498,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":4.8899999999999997,"5min":10.09,"15min":15.121},"last":2.5699999999999998}]},{"osd":11,"last update":"Fri Feb 9 17:49:32 2024","interfaces":[{"interface":"back","average":{"1min":2.6179999999999999,"5min":1.8340000000000001,"15min":1.3480000000000001},"min":{"1min":0.46800000000000003,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":16.184999999999999,"5min":16.184999999999999,"15min":16.184999999999999},"last":6.0570000000000004},{"interface":"front","average":{"1min":2.109,"5min":2.0099999999999998,"15min":1.486},"min":{"1min":0.505,"5min":0.45600000000000002,"15min":0.44800000000000001},"max":{"1min":11.904,"5min":11.904,"15min":11.904},"last":1.837}]}]},{"osd":0,"up_from":8,"seq":34359738753,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":808124,"kb_used_data":123060,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":92937028,"statfs":{"total":95995035648,"available":95167516672,"internally_reserved":0,"allocated":126013440,"data_stored":125715715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 9 17:49:28 2024","interfaces":[{"interface":"back","average":{"1min":1.746,"5min":1.8069999999999999,"15min":1.639},"min":{"1min":0.25900000000000001,"5min":0.219,"15min":0.20599999999999999},"max":{"1min":8.7469999999999999,"5min":10.938000000000001,"15min":13.842000000000001},"last":1.2869999999999999},{"interface":"front","average":{"1min":1.8109999999999999,"5min":1.8700000000000001,"15min":1.8049999999999999},"min":{"1min":0.308,"5min":0.29299999999999998,"15min":0.24399999999999999},"max":{"1min":8.7919999999999998,"5min":10.974,"15min":14.122},"last":1.3540000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:49:44 2024","interfaces":[{"interface":"back","average":{"1min":2.052,"5min":2.052,"15min":2.052},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":8.859,"5min":8.859,"15min":8.859},"last":4.2560000000000002},{"interface":"front","average":{"1min":1.52,"5min":1.52,"15min":1.52},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":4.3940000000000001,"5min":4.3940000000000001,"15min":4.3940000000000001},"last":0.40300000000000002}]},{"osd":3,"last update":"Fri Feb 9 17:49:08 2024","interfaces":[{"interface":"back","average":{"1min":1.968,"5min":1.7849999999999999,"15min":2.1640000000000001},"min":{"1min":0.313,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":7.593,"5min":36.052,"15min":36.052},"last":4.093},{"interface":"front","average":{"1min":1.6399999999999999,"5min":2.3340000000000001,"15min":1.8009999999999999},"min":{"1min":0.28899999999999998,"5min":0.24099999999999999,"15min":0.217},"max":{"1min":6.5590000000000002,"5min":38.383000000000003,"15min":38.383000000000003},"last":5.3440000000000003}]},{"osd":4,"last update":"Fri Feb 9 17:48:54 2024","interfaces":[{"interface":"back","average":{"1min":1.706,"5min":1.5409999999999999,"15min":1.47},"min":{"1min":0.29999999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":8.9420000000000002,"5min":8.9420000000000002,"15min":11.212},"last":0.495},{"interface":"front","average":{"1min":1.9199999999999999,"5min":1.944,"15min":1.8169999999999999},"min":{"1min":0.36399999999999999,"5min":0.32300000000000001,"15min":0.27200000000000002},"max":{"1min":8.9580000000000002,"5min":16.916,"15min":16.916},"last":5.8099999999999996}]},{"osd":5,"last update":"Fri Feb 9 17:49:08 2024","interfaces":[{"interface":"back","average":{"1min":1.575,"5min":2.4900000000000002,"15min":1.887},"min":{"1min":0.40899999999999997,"5min":0.30299999999999999,"15min":0.29499999999999998},"max":{"1min":4.7279999999999998,"5min":48.924999999999997,"15min":48.924999999999997},"last":1.393},{"interface":"front","average":{"1min":1.6279999999999999,"5min":2.5339999999999998,"15min":1.9119999999999999},"min":{"1min":0.44700000000000001,"5min":0.374,"15min":0.29599999999999999},"max":{"1min":4.7560000000000002,"5min":48.969999999999999,"15min":48.969999999999999},"last":1.4450000000000001}]},{"osd":6,"last update":"Fri Feb 9 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.8400000000000001,"5min":1.839,"15min":1.7549999999999999},"min":{"1min":0.35099999999999998,"5min":0.307,"15min":0.28699999999999998},"max":{"1min":7.9429999999999996,"5min":11.912000000000001,"15min":13.407},"last":0.46200000000000002},{"interface":"front","average":{"1min":2.3610000000000002,"5min":2.0190000000000001,"15min":1.8540000000000001},"min":{"1min":0.38,"5min":0.375,"15min":0.33300000000000002},"max":{"1min":7.9740000000000002,"5min":11.939,"15min":13.448},"last":0.53300000000000003}]},{"osd":7,"last update":"Fri Feb 9 17:49:00 2024","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":1.8480000000000001,"15min":2.0670000000000002},"min":{"1min":0.35899999999999999,"5min":0.33800000000000002,"15min":0.30299999999999999},"max":{"1min":4.4409999999999998,"5min":15.271000000000001,"15min":15.271000000000001},"last":2.1139999999999999},{"interface":"front","average":{"1min":2.2869999999999999,"5min":2.0550000000000002,"15min":2.004},"min":{"1min":0.45600000000000002,"5min":0.34699999999999998,"15min":0.28699999999999998},"max":{"1min":7.7699999999999996,"5min":11.614000000000001,"15min":14.451000000000001},"last":5.8490000000000002}]},{"osd":8,"last update":"Fri Feb 9 17:49:00 2024","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":1.3819999999999999,"15min":1.583},"min":{"1min":0.41899999999999998,"5min":0.36199999999999999,"15min":0.33100000000000002},"max":{"1min":2.4649999999999999,"5min":11.641999999999999,"15min":14.323},"last":5.8769999999999998},{"interface":"front","average":{"1min":0.879,"5min":1.3520000000000001,"15min":1.671},"min":{"1min":0.48399999999999999,"5min":0.36299999999999999,"15min":0.35099999999999998},"max":{"1min":2.0659999999999998,"5min":11.669,"15min":14.481999999999999},"last":5.9039999999999999}]},{"osd":9,"last update":"Fri Feb 9 17:48:48 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.8009999999999999,"15min":1.966},"min":{"1min":0.53000000000000003,"5min":0.40500000000000003,"15min":0.32500000000000001},"max":{"1min":3.165,"5min":16.936,"15min":17.952000000000002},"last":5.9299999999999997},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.6559999999999999,"15min":1.891},"min":{"1min":0.56200000000000006,"5min":0.42399999999999999,"15min":0.34799999999999998},"max":{"1min":3.1880000000000002,"5min":16.951000000000001,"15min":17.913},"last":5.9539999999999997}]},{"osd":10,"last update":"Fri Feb 9 17:49:08 2024","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.702,"15min":1.929},"min":{"1min":0.47899999999999998,"5min":0.443,"15min":0.35499999999999998},"max":{"1min":5.9509999999999996,"5min":11.692,"15min":31.766999999999999},"last":1.4710000000000001},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.5089999999999999,"15min":1.4930000000000001},"min":{"1min":0.42299999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":4.2030000000000003,"5min":13.148999999999999,"15min":28.048999999999999},"last":1.419}]},{"osd":11,"last update":"Fri Feb 9 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.601,"15min":1.659},"min":{"1min":0.499,"5min":0.41599999999999998,"15min":0.40600000000000003},"max":{"1min":6.274,"5min":10.282,"15min":40.021000000000001},"last":1.6120000000000001},{"interface":"front","average":{"1min":1.671,"5min":1.849,"15min":1.7849999999999999},"min":{"1min":0.53300000000000003,"5min":0.45500000000000002,"15min":0.45200000000000001},"max":{"1min":8.1820000000000004,"5min":9.5820000000000007,"15min":40.055999999999997},"last":1.556}]}]},{"osd":2,"up_from":18,"seq":77309411703,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":545348,"kb_used_data":148860,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":93199804,"statfs":{"total":95995035648,"available":95436599296,"internally_reserved":0,"allocated":152432640,"data_stored":152131363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":1.4510000000000001,"15min":1.006},"min":{"1min":0.29999999999999999,"5min":0.23699999999999999,"15min":0.221},"max":{"1min":4.9089999999999998,"5min":17.251000000000001,"15min":17.251000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.74399999999999999,"5min":1.611,"15min":1.113},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.22900000000000001},"max":{"1min":4.7720000000000002,"5min":23.097999999999999,"15min":23.097999999999999},"last":0.875}]},{"osd":1,"last update":"Fri Feb 9 17:49:02 2024","interfaces":[{"interface":"back","average":{"1min":3.2589999999999999,"5min":2.3759999999999999,"15min":2.0299999999999998},"min":{"1min":0.27700000000000002,"5min":0.214,"15min":0.214},"max":{"1min":23.044,"5min":52.646000000000001,"15min":52.646000000000001},"last":15.352},{"interface":"front","average":{"1min":2.2330000000000001,"5min":2.181,"15min":2.0059999999999998},"min":{"1min":0.32100000000000001,"5min":0.248,"15min":0.214},"max":{"1min":12.368,"5min":52.683999999999997,"15min":52.683999999999997},"last":15.31}]},{"osd":3,"last update":"Fri Feb 9 17:49:26 2024","interfaces":[{"interface":"back","average":{"1min":2.8919999999999999,"5min":2.3639999999999999,"15min":1.9530000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.249},"max":{"1min":8.7249999999999996,"5min":22.698,"15min":24.353000000000002},"last":4.5880000000000001},{"interface":"front","average":{"1min":4.1779999999999999,"5min":2.6179999999999999,"15min":2.222},"min":{"1min":0.31,"5min":0.221,"15min":0.20699999999999999},"max":{"1min":15.414,"5min":19.904,"15min":19.986999999999998},"last":8.3539999999999992}]},{"osd":4,"last update":"Fri Feb 9 17:49:26 2024","interfaces":[{"interface":"back","average":{"1min":2.0760000000000001,"5min":1.97,"15min":1.9059999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.26300000000000001},"max":{"1min":8.3580000000000005,"5min":12.398,"15min":13.526},"last":4.0700000000000003},{"interface":"front","average":{"1min":1.431,"5min":1.917,"15min":1.8},"min":{"1min":0.29899999999999999,"5min":0.27900000000000003,"15min":0.27300000000000002},"max":{"1min":6.452,"5min":10.888999999999999,"15min":10.888999999999999},"last":0.56699999999999995}]},{"osd":5,"last update":"Fri Feb 9 17:49:19 2024","interfaces":[{"interface":"back","average":{"1min":2.1219999999999999,"5min":2.702,"15min":2.1339999999999999},"min":{"1min":0.29699999999999999,"5min":0.28499999999999998,"15min":0.28299999999999997},"max":{"1min":6.5460000000000003,"5min":35.436999999999998,"15min":42.664000000000001},"last":4.4539999999999997},{"interface":"front","average":{"1min":2.5859999999999999,"5min":2.9849999999999999,"15min":2.4329999999999998},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":9.3390000000000004,"5min":35.484000000000002,"15min":43.311999999999998},"last":0.53300000000000003}]},{"osd":6,"last update":"Fri Feb 9 17:49:26 2024","interfaces":[{"interface":"back","average":{"1min":2.3530000000000002,"5min":2.3140000000000001,"15min":2.0499999999999998},"min":{"1min":0.34100000000000003,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":6.5960000000000001,"5min":11.092000000000001,"15min":11.092000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":1.9259999999999999,"5min":2.1840000000000002,"15min":2.1280000000000001},"min":{"1min":0.39300000000000002,"5min":0.33100000000000002,"15min":0.32800000000000001},"max":{"1min":6.6180000000000003,"5min":11.019,"15min":11.269},"last":0.68100000000000005}]},{"osd":7,"last update":"Fri Feb 9 17:49:02 2024","interfaces":[{"interface":"back","average":{"1min":2.3119999999999998,"5min":2.331,"15min":2.3010000000000002},"min":{"1min":0.38500000000000001,"5min":0.34699999999999998,"15min":0.32400000000000001},"max":{"1min":12.422000000000001,"5min":16.666,"15min":41.802999999999997},"last":4.1369999999999996},{"interface":"front","average":{"1min":3.48,"5min":2.9950000000000001,"15min":2.4079999999999999},"min":{"1min":0.439,"5min":0.36599999999999999,"15min":0.30499999999999999},"max":{"1min":10.441000000000001,"5min":16.638999999999999,"15min":41.783999999999999},"last":1.1950000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:49:26 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":1.677,"15min":1.5},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.316},"max":{"1min":8.3840000000000003,"5min":12.307,"15min":22.393999999999998},"last":4.1180000000000003},{"interface":"front","average":{"1min":1.589,"5min":1.744,"15min":1.6000000000000001},"min":{"1min":0.42499999999999999,"5min":0.36899999999999999,"15min":0.34399999999999997},"max":{"1min":6.4889999999999999,"5min":10.964,"15min":12.58},"last":0.61399999999999999}]},{"osd":9,"last update":"Fri Feb 9 17:49:40 2024","interfaces":[{"interface":"back","average":{"1min":2.7519999999999998,"5min":2.0979999999999999,"15min":1.895},"min":{"1min":0.33200000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":9.9399999999999995,"5min":12.404999999999999,"15min":13.5},"last":4.4880000000000004},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.9139999999999999,"15min":1.8029999999999999},"min":{"1min":0.45900000000000002,"5min":0.371,"15min":0.36399999999999999},"max":{"1min":3.4020000000000001,"5min":11.058999999999999,"15min":13.589},"last":0.63700000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:49:16 2024","interfaces":[{"interface":"back","average":{"1min":2.6269999999999998,"5min":1.6040000000000001,"15min":1.6459999999999999},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.33600000000000002},"max":{"1min":12.314,"5min":12.314,"15min":22.425999999999998},"last":4.1020000000000003},{"interface":"front","average":{"1min":2.6549999999999998,"5min":1.6539999999999999,"15min":1.6850000000000001},"min":{"1min":0.505,"5min":0.501,"15min":0.379},"max":{"1min":12.342000000000001,"5min":12.342000000000001,"15min":22.454999999999998},"last":4.1559999999999997}]},{"osd":11,"last update":"Fri Feb 9 17:49:26 2024","interfaces":[{"interface":"back","average":{"1min":1.6970000000000001,"5min":2.1030000000000002,"15min":2.2959999999999998},"min":{"1min":0.42099999999999999,"5min":0.41999999999999998,"15min":0.39600000000000002},"max":{"1min":6.6399999999999997,"5min":11.137,"15min":137.44300000000001},"last":0.65900000000000003},{"interface":"front","average":{"1min":2.7650000000000001,"5min":2.0379999999999998,"15min":2.3359999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":9.9830000000000005,"5min":12.609999999999999,"15min":137.64400000000001},"last":4.524}]}]},{"osd":3,"up_from":27,"seq":115964117363,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":929020,"kb_used_data":356404,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":92816132,"statfs":{"total":95995035648,"available":95043719168,"internally_reserved":0,"allocated":364957696,"data_stored":364659971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":586348999},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,0,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:49:23 2024","interfaces":[{"interface":"back","average":{"1min":3.1709999999999998,"5min":2.0600000000000001,"15min":1.8680000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.186},"max":{"1min":11.385999999999999,"5min":11.385999999999999,"15min":38.984999999999999},"last":1.575},{"interface":"front","average":{"1min":3.419,"5min":2.7999999999999998,"15min":2.2210000000000001},"min":{"1min":0.28799999999999998,"5min":0.246,"15min":0.22500000000000001},"max":{"1min":11.079000000000001,"5min":11.506,"15min":41.497999999999998},"last":13.113}]},{"osd":1,"last update":"Fri Feb 9 17:48:49 2024","interfaces":[{"interface":"back","average":{"1min":4.2939999999999996,"5min":3.0430000000000001,"15min":2.3929999999999998},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":16.337,"5min":16.337,"15min":16.337},"last":5.1619999999999999},{"interface":"front","average":{"1min":2.048,"5min":1.9099999999999999,"15min":1.4399999999999999},"min":{"1min":0.308,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":8.0630000000000006,"5min":20.138999999999999,"15min":20.138999999999999},"last":6.8120000000000003}]},{"osd":2,"last update":"Fri Feb 9 17:49:19 2024","interfaces":[{"interface":"back","average":{"1min":2.7970000000000002,"5min":2.0249999999999999,"15min":1.7789999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":11.196999999999999,"5min":11.702999999999999,"15min":14.888999999999999},"last":8.2300000000000004},{"interface":"front","average":{"1min":2.1619999999999999,"5min":2.3100000000000001,"15min":1.909},"min":{"1min":0.34599999999999997,"5min":0.26100000000000001,"15min":0.20200000000000001},"max":{"1min":6.7169999999999996,"5min":19.879000000000001,"15min":19.879000000000001},"last":8.1240000000000006}]},{"osd":4,"last update":"Fri Feb 9 17:49:23 2024","interfaces":[{"interface":"back","average":{"1min":3.5409999999999999,"5min":2.3439999999999999,"15min":1.8939999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.26800000000000002},"max":{"1min":11.117000000000001,"5min":11.117000000000001,"15min":34.898000000000003},"last":3.3660000000000001},{"interface":"front","average":{"1min":1.706,"5min":2.0569999999999999,"15min":1.758},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.28000000000000003},"max":{"1min":7.9459999999999997,"5min":19.713999999999999,"15min":19.713999999999999},"last":4.3109999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:49:02 2024","interfaces":[{"interface":"back","average":{"1min":4.2590000000000003,"5min":2.4590000000000001,"15min":1.9570000000000001},"min":{"1min":0.46400000000000002,"5min":0.41099999999999998,"15min":0.27800000000000002},"max":{"1min":9.6010000000000009,"5min":13.470000000000001,"15min":17.651},"last":6.6829999999999998},{"interface":"front","average":{"1min":4.3120000000000003,"5min":2.2269999999999999,"15min":1.9299999999999999},"min":{"1min":0.48699999999999999,"5min":0.316,"15min":0.316},"max":{"1min":9.6389999999999993,"5min":9.6389999999999993,"15min":17.675999999999998},"last":6.7069999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:48:49 2024","interfaces":[{"interface":"back","average":{"1min":3.964,"5min":1.964,"15min":2.2109999999999999},"min":{"1min":0.436,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":15.061999999999999,"5min":15.061999999999999,"15min":15.061999999999999},"last":6.6559999999999997},{"interface":"front","average":{"1min":3.3170000000000002,"5min":2.0169999999999999,"15min":1.649},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":9.032,"5min":11.813000000000001,"15min":11.813000000000001},"last":6.7300000000000004}]},{"osd":7,"last update":"Fri Feb 9 17:49:32 2024","interfaces":[{"interface":"back","average":{"1min":1.8720000000000001,"5min":2.1030000000000002,"15min":1.5509999999999999},"min":{"1min":0.45800000000000002,"5min":0.36499999999999999,"15min":0.34200000000000003},"max":{"1min":5.2800000000000002,"5min":19.788,"15min":19.788},"last":4.3470000000000004},{"interface":"front","average":{"1min":2.8010000000000002,"5min":2.5910000000000002,"15min":1.484},"min":{"1min":0.48499999999999999,"5min":0.40100000000000002,"15min":0.34899999999999998},"max":{"1min":7.0510000000000002,"5min":11.858000000000001,"15min":11.858000000000001},"last":6.6310000000000002}]},{"osd":8,"last update":"Fri Feb 9 17:45:16 2024","interfaces":[{"interface":"back","average":{"1min":1.714,"5min":1.4450000000000001,"15min":1.498},"min":{"1min":0.377,"5min":0.377,"15min":0.26800000000000002},"max":{"1min":8.6020000000000003,"5min":11.821,"15min":11.821},"last":0.67700000000000005},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.1719999999999999,"15min":1.351},"min":{"1min":0.46500000000000002,"5min":0.36199999999999999,"15min":0.34499999999999997},"max":{"1min":6.8390000000000004,"5min":6.8390000000000004,"15min":14.129},"last":0.78200000000000003}]},{"osd":9,"last update":"Fri Feb 9 17:49:47 2024","interfaces":[{"interface":"back","average":{"1min":3.153,"5min":2.0619999999999998,"15min":1.7689999999999999},"min":{"1min":0.374,"5min":0.34599999999999997,"15min":0.32500000000000001},"max":{"1min":11.144,"5min":11.144,"15min":11.851000000000001},"last":3.4460000000000002},{"interface":"front","average":{"1min":3.1680000000000001,"5min":2.085,"15min":1.784},"min":{"1min":0.40400000000000003,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":11.297000000000001,"5min":11.297000000000001,"15min":12.815},"last":3.419}]},{"osd":10,"last update":"Fri Feb 9 17:49:32 2024","interfaces":[{"interface":"back","average":{"1min":2.306,"5min":1.5780000000000001,"15min":1.216},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.39100000000000001},"max":{"1min":5.7699999999999996,"5min":6.0949999999999998,"15min":9.7080000000000002},"last":6.5940000000000003},{"interface":"front","average":{"1min":1.919,"5min":1.9419999999999999,"15min":1.2390000000000001},"min":{"1min":0.625,"5min":0.435,"15min":0.38400000000000001},"max":{"1min":5.3540000000000001,"5min":19.859000000000002,"15min":19.859000000000002},"last":4.375}]},{"osd":11,"last update":"Fri Feb 9 17:49:23 2024","interfaces":[{"interface":"back","average":{"1min":1.806,"5min":1.9119999999999999,"15min":1.6499999999999999},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.40300000000000002},"max":{"1min":5.3310000000000004,"5min":19.824000000000002,"15min":19.824000000000002},"last":4.4039999999999999},{"interface":"front","average":{"1min":1.9330000000000001,"5min":1.474,"15min":1.7110000000000001},"min":{"1min":0.5,"5min":0.47499999999999998,"15min":0.442},"max":{"1min":5.9859999999999998,"5min":6.1219999999999999,"15min":17.145},"last":4.431}]}]},{"osd":4,"up_from":32,"seq":137438953837,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":839420,"kb_used_data":194612,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":92905732,"statfs":{"total":95995035648,"available":95135469568,"internally_reserved":0,"allocated":199282688,"data_stored":198984963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":660273605},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:49:34 2024","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":1.7450000000000001,"15min":1.7829999999999999},"min":{"1min":0.28299999999999997,"5min":0.22800000000000001,"15min":0.222},"max":{"1min":3.4950000000000001,"5min":10.301,"15min":50.356999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.98499999999999999,"5min":1.8089999999999999,"15min":1.861},"min":{"1min":0.372,"5min":0.30299999999999999,"15min":0.247},"max":{"1min":3.5379999999999998,"5min":10.407999999999999,"15min":50.569000000000003},"last":0.58499999999999996}]},{"osd":1,"last update":"Fri Feb 9 17:49:34 2024","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":2.3029999999999999,"15min":2.3029999999999999},"min":{"1min":0.28299999999999997,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":4.9770000000000003,"5min":17.297999999999998,"15min":50.473999999999997},"last":0.26600000000000001},{"interface":"front","average":{"1min":2.5289999999999999,"5min":2.5760000000000001,"15min":2.5419999999999998},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":9.9109999999999996,"5min":12.279999999999999,"15min":50.499000000000002},"last":0.52500000000000002}]},{"osd":2,"last update":"Fri Feb 9 17:48:50 2024","interfaces":[{"interface":"back","average":{"1min":2.198,"5min":2.4500000000000002,"15min":2.8780000000000001},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":8.5139999999999993,"5min":17.494,"15min":17.494},"last":1.599},{"interface":"front","average":{"1min":2.8479999999999999,"5min":2.508,"15min":2.8260000000000001},"min":{"1min":0.36899999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":17.283999999999999,"5min":17.283999999999999,"15min":17.283999999999999},"last":0.23300000000000001}]},{"osd":3,"last update":"Fri Feb 9 17:49:34 2024","interfaces":[{"interface":"back","average":{"1min":1.784,"5min":1.887,"15min":2.0099999999999998},"min":{"1min":0.35099999999999998,"5min":0.28499999999999998,"15min":0.245},"max":{"1min":5.3970000000000002,"5min":14.448,"15min":50.462000000000003},"last":6.7370000000000001},{"interface":"front","average":{"1min":2.5459999999999998,"5min":2.218,"15min":1.988},"min":{"1min":0.42399999999999999,"5min":0.28199999999999997,"15min":0.25800000000000001},"max":{"1min":9.9339999999999993,"5min":10.382,"15min":50.529000000000003},"last":0.89200000000000002}]},{"osd":5,"last update":"Fri Feb 9 17:48:59 2024","interfaces":[{"interface":"back","average":{"1min":2.0019999999999998,"5min":2.7450000000000001,"15min":1.98},"min":{"1min":0.255,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":6.9340000000000002,"5min":18.887,"15min":18.887},"last":6.7889999999999997},{"interface":"front","average":{"1min":2.0579999999999998,"5min":2.8100000000000001,"15min":2.2480000000000002},"min":{"1min":0.31,"5min":0.31,"15min":0.29099999999999998},"max":{"1min":7.0119999999999996,"5min":18.917000000000002,"15min":50.509999999999998},"last":6.8090000000000002}]},{"osd":6,"last update":"Fri Feb 9 17:49:34 2024","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.8,"15min":1.647},"min":{"1min":0.29999999999999999,"5min":0.26300000000000001,"15min":0.246},"max":{"1min":5.0129999999999999,"5min":15.51,"15min":18.234999999999999},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.77800000000000002,"5min":1.6220000000000001,"15min":1.9610000000000001},"min":{"1min":0.34000000000000002,"5min":0.31,"15min":0.245},"max":{"1min":2.3740000000000001,"5min":10.215,"15min":50.402999999999999},"last":0.65100000000000002}]},{"osd":7,"last update":"Fri Feb 9 17:49:09 2024","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.7270000000000001,"15min":1.635},"min":{"1min":0.35499999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":5.0529999999999999,"5min":10.185,"15min":11.875},"last":0.379},{"interface":"front","average":{"1min":0.88700000000000001,"5min":1.798,"15min":1.7789999999999999},"min":{"1min":0.39800000000000002,"5min":0.29999999999999999,"15min":0.28199999999999997},"max":{"1min":4.1050000000000004,"5min":10.547000000000001,"15min":12.696},"last":0.44700000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:48:59 2024","interfaces":[{"interface":"back","average":{"1min":1.996,"5min":2.097,"15min":1.6919999999999999},"min":{"1min":0.42099999999999999,"5min":0.35299999999999998,"15min":0.33200000000000002},"max":{"1min":6.9790000000000001,"5min":17.335999999999999,"15min":17.335999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.95999999999999996,"5min":1.278,"15min":1.5},"min":{"1min":0.44800000000000001,"5min":0.36699999999999999,"15min":0.34699999999999998},"max":{"1min":5.859,"5min":10.433,"15min":50.594000000000001},"last":6.7039999999999997}]},{"osd":9,"last update":"Fri Feb 9 17:44:39 2024","interfaces":[{"interface":"back","average":{"1min":2.5739999999999998,"5min":1.855,"15min":1.881},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":10.778,"5min":10.778,"15min":50.442999999999998},"last":0.73599999999999999},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.331,"15min":1.536},"min":{"1min":0.47699999999999998,"5min":0.42499999999999999,"15min":0.36199999999999999},"max":{"1min":5.165,"5min":8.3580000000000005,"15min":10.555},"last":4.8259999999999996}]},{"osd":10,"last update":"Fri Feb 9 17:49:14 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":1.524,"15min":1.98},"min":{"1min":0.42699999999999999,"5min":0.40500000000000003,"15min":0.39400000000000002},"max":{"1min":0.79700000000000004,"5min":17.367000000000001,"15min":51.223999999999997},"last":0.61899999999999999},{"interface":"front","average":{"1min":1.331,"5min":1.675,"15min":1.9730000000000001},"min":{"1min":0.443,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":5.1550000000000002,"5min":12.888,"15min":50.552},"last":0.49399999999999999}]},{"osd":11,"last update":"Fri Feb 9 17:48:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.381,"15min":1.8480000000000001},"min":{"1min":0.51500000000000001,"5min":0.44600000000000001,"15min":0.443},"max":{"1min":5.7960000000000003,"5min":10.521000000000001,"15min":50.484999999999999},"last":6.7560000000000002},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.8240000000000001,"15min":1.6060000000000001},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.40999999999999998},"max":{"1min":5.1310000000000002,"5min":13.154999999999999,"15min":13.154999999999999},"last":4.2649999999999997}]}]},{"osd":5,"up_from":40,"seq":171798692200,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":792328,"kb_used_data":187712,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":92952824,"statfs":{"total":95995035648,"available":95183691776,"internally_reserved":0,"allocated":192217088,"data_stored":191911739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":619051461},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:49:15 2024","interfaces":[{"interface":"back","average":{"1min":7.3319999999999999,"5min":2.8290000000000002,"15min":1.712},"min":{"1min":0.27600000000000002,"5min":0.24299999999999999,"15min":0.23599999999999999},"max":{"1min":116.77800000000001,"5min":116.77800000000001,"15min":116.77800000000001},"last":0.30299999999999999},{"interface":"front","average":{"1min":8.3539999999999992,"5min":3.0720000000000001,"15min":2.1850000000000001},"min":{"1min":0.29299999999999998,"5min":0.252,"15min":0.249},"max":{"1min":116.959,"5min":116.959,"15min":116.959},"last":0.374}]},{"osd":1,"last update":"Fri Feb 9 17:49:04 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":1.9830000000000001,"15min":1.9690000000000001},"min":{"1min":0.26700000000000002,"5min":0.254,"15min":0.24199999999999999},"max":{"1min":8.4809999999999999,"5min":13.494999999999999,"15min":48.161999999999999},"last":5.3140000000000001},{"interface":"front","average":{"1min":2.6269999999999998,"5min":2.052,"15min":2.077},"min":{"1min":0.35199999999999998,"5min":0.248,"15min":0.23599999999999999},"max":{"1min":15.097,"5min":16.626999999999999,"15min":48.218000000000004},"last":5.3300000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:49:41 2024","interfaces":[{"interface":"back","average":{"1min":2.6200000000000001,"5min":2.621,"15min":1.891},"min":{"1min":0.317,"5min":0.249,"15min":0.249},"max":{"1min":7.6600000000000001,"5min":13.929,"15min":13.929},"last":0.27900000000000003},{"interface":"front","average":{"1min":3.073,"5min":2.4900000000000002,"15min":2.2480000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":14.978,"5min":15.945,"15min":15.945},"last":0.32400000000000001}]},{"osd":3,"last update":"Fri Feb 9 17:49:04 2024","interfaces":[{"interface":"back","average":{"1min":3.456,"5min":2.5489999999999999,"15min":2.2850000000000001},"min":{"1min":0.40500000000000003,"5min":0.34699999999999998,"15min":0.26800000000000002},"max":{"1min":15.154999999999999,"5min":15.819000000000001,"15min":64.150999999999996},"last":5.0830000000000002},{"interface":"front","average":{"1min":3.024,"5min":2.629,"15min":2.3650000000000002},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.247},"max":{"1min":8.7650000000000006,"5min":11.747,"15min":64.176000000000002},"last":0.35099999999999998}]},{"osd":4,"last update":"Fri Feb 9 17:49:04 2024","interfaces":[{"interface":"back","average":{"1min":1.873,"5min":1.9690000000000001,"15min":1.8859999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.249},"max":{"1min":28.481999999999999,"5min":28.481999999999999,"15min":49.313000000000002},"last":0.45900000000000002},{"interface":"front","average":{"1min":2.3180000000000001,"5min":2.0619999999999998,"15min":1.8979999999999999},"min":{"1min":0.28499999999999998,"5min":0.26100000000000001,"15min":0.218},"max":{"1min":15.843,"5min":15.843,"15min":50.505000000000003},"last":0.42499999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:49:33 2024","interfaces":[{"interface":"back","average":{"1min":1.954,"5min":1.911,"15min":1.7270000000000001},"min":{"1min":0.33000000000000002,"5min":0.27900000000000003,"15min":0.253},"max":{"1min":15.026,"5min":16.657,"15min":16.657},"last":0.40100000000000002},{"interface":"front","average":{"1min":2.3690000000000002,"5min":2.0939999999999999,"15min":1.8149999999999999},"min":{"1min":0.36499999999999999,"5min":0.32700000000000001,"15min":0.26200000000000001},"max":{"1min":20.192,"5min":20.192,"15min":20.192},"last":0.501}]},{"osd":7,"last update":"Fri Feb 9 17:49:04 2024","interfaces":[{"interface":"back","average":{"1min":1.7989999999999999,"5min":2.0510000000000002,"15min":1.9119999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.26700000000000002},"max":{"1min":15.891999999999999,"5min":15.891999999999999,"15min":50.545000000000002},"last":0.46700000000000003},{"interface":"front","average":{"1min":3.5169999999999999,"5min":2.4569999999999999,"15min":2.4449999999999998},"min":{"1min":0.35899999999999999,"5min":0.31,"15min":0.31},"max":{"1min":12.326000000000001,"5min":13.869,"15min":102.526},"last":0.44800000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:43:23 2024","interfaces":[{"interface":"back","average":{"1min":1.6499999999999999,"5min":1.639,"15min":1.6020000000000001},"min":{"1min":0.376,"5min":0.35299999999999998,"15min":0.34899999999999998},"max":{"1min":4.8780000000000001,"5min":14.542,"15min":14.542},"last":5.5439999999999996},{"interface":"front","average":{"1min":3.6699999999999999,"5min":1.702,"15min":1.8300000000000001},"min":{"1min":0.37,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":16.359000000000002,"5min":16.359000000000002,"15min":16.359000000000002},"last":0.435}]},{"osd":9,"last update":"Fri Feb 9 17:49:33 2024","interfaces":[{"interface":"back","average":{"1min":1.778,"5min":1.577,"15min":1.498},"min":{"1min":0.42599999999999999,"5min":0.36599999999999999,"15min":0.35299999999999998},"max":{"1min":15.124000000000001,"5min":15.124000000000001,"15min":16.411000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":2.0110000000000001,"5min":1.9139999999999999,"15min":1.6659999999999999},"min":{"1min":0.379,"5min":0.35999999999999999,"15min":0.35299999999999998},"max":{"1min":5.7300000000000004,"5min":14.013,"15min":16.417999999999999},"last":0.38700000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:49:04 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":2.5129999999999999,"15min":1.9079999999999999},"min":{"1min":0.40899999999999997,"5min":0.39400000000000002,"15min":0.317},"max":{"1min":5.069,"5min":80.042000000000002,"15min":80.042000000000002},"last":0.49399999999999999},{"interface":"front","average":{"1min":1.4390000000000001,"5min":2.4180000000000001,"15min":1.851},"min":{"1min":0.42199999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":8.7309999999999999,"5min":79.807000000000002,"15min":79.807000000000002},"last":0.51600000000000001}]},{"osd":11,"last update":"Fri Feb 9 17:49:15 2024","interfaces":[{"interface":"back","average":{"1min":2.149,"5min":2.194,"15min":1.984},"min":{"1min":0.48299999999999998,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":5.7679999999999998,"5min":13.987,"15min":16.390000000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":1.635,"5min":1.5820000000000001,"15min":1.7350000000000001},"min":{"1min":0.45600000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":5.3940000000000001,"5min":7.1470000000000002,"15min":9.8239999999999998},"last":0.54700000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":118161408,"data_stored":118161408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":142217216,"data_stored":142213144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":141230080,"data_stored":141230080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":345178112,"data_stored":345178112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":193265664,"data_stored":193265664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":185602048,"data_stored":185597976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":191692800,"data_stored":191692800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":212074496,"data_stored":212074496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":86507520,"data_stored":86507520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":184614912,"data_stored":184614912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":127471616,"data_stored":127467544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":64290816,"data_stored":64290816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-09T17:49:57.691 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-02-09T17:49:57.691 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:49:57.691 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-02-09T17:49:57.692 INFO:tasks.thrashosds.thrasher:Setting osd 3 primary_affinity to 0.801503 2024-02-09T17:49:57.692 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd primary-affinity 3 0.8015031811988723 2024-02-09T17:49:57.786 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:49:57.816 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:57.839 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:49:57.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:57.845 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:49:57.946 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:49:58.038 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:58.591 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:49:58 smithi099 ceph-mon[39420]: pgmap v1649: 47 pgs: 47 active+clean; 330 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 774 op/s 2024-02-09T17:49:58.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:49:58 smithi179 ceph-mon[43975]: pgmap v1649: 47 pgs: 47 active+clean; 330 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 774 op/s 2024-02-09T17:49:58.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:49:58 smithi162 ceph-mon[44304]: pgmap v1649: 47 pgs: 47 active+clean; 330 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 774 op/s 2024-02-09T17:49:58.807 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:49:58.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:58.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:49:58.915 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:49:58.941 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:58.963 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:49:58.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:58.970 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:49:59.071 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:49:59.118 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:59.145 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:49:59.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:59.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:49:59.253 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:49:59.287 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:59.332 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:49:59.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:59.339 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:49:59.440 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:49:59.468 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:59.493 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:49:59.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:59.503 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:49:59.604 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:49:59.650 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:59.676 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:49:59.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:59.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:49:59.784 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:49:59.825 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:49:59.861 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:49:59.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:49:59.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:49:59.966 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:50:00.024 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:00.249 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:00 smithi099 ceph-mon[39420]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:50:00.249 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:00 smithi099 ceph-mon[39420]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:50:00.249 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:00 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:50:00.249 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:00 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:50:00.249 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:00 smithi099 ceph-mon[39420]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:50:00.250 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:00 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:50:00.250 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:00 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:50:00.250 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:00 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:50:00.250 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:00 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:50:00.250 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:00 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:50:00.250 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:00 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:50:00.250 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:00 smithi099 ceph-mon[39420]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:50:00.250 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:00 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:50:00.251 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:00 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:50:00.251 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:00 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:50:00.251 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:00 smithi099 ceph-mon[39420]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:50:00.251 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:00 smithi099 ceph-mon[39420]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-02-09T17:50:00.251 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:00 smithi099 ceph-mon[39420]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-02-09T17:50:00.310 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:50:00.324 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:50:00.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:00 smithi179 ceph-mon[43975]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:50:00.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:00 smithi179 ceph-mon[43975]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:50:00.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:00 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:50:00.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:00 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:50:00.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:00 smithi179 ceph-mon[43975]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:50:00.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:00 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:50:00.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:00 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:50:00.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:00 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:50:00.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:00 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:50:00.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:00 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:50:00.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:00 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:50:00.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:00 smithi179 ceph-mon[43975]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:50:00.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:00 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:50:00.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:00 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:50:00.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:00 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:50:00.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:00 smithi179 ceph-mon[43975]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:50:00.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:00 smithi179 ceph-mon[43975]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-02-09T17:50:00.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:00 smithi179 ceph-mon[43975]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-02-09T17:50:00.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:00 smithi162 ceph-mon[44304]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:50:00.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:00 smithi162 ceph-mon[44304]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:50:00.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:00 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:50:00.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:00 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:50:00.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:00 smithi162 ceph-mon[44304]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:50:00.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:00 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:50:00.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:00 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:50:00.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:00 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:50:00.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:00 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:50:00.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:00 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:50:00.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:00 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:50:00.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:00 smithi162 ceph-mon[44304]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:50:00.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:00 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:50:00.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:00 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:50:00.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:00 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:50:00.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:00 smithi162 ceph-mon[44304]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:50:00.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:00 smithi162 ceph-mon[44304]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-02-09T17:50:00.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:00 smithi162 ceph-mon[44304]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-02-09T17:50:00.572 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:50:00.572 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 20, 2024-02-09T17:50:00.572 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 600, 2024-02-09T17:50:00.572 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [ 2024-02-09T17:50:00.572 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.573 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:266604 4.0 4:0b2906d5:::benchmark_data_smithi179_103809_object129138:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e450)", 2024-02-09T17:50:00.573 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:40:18.476844+0000", 2024-02-09T17:50:00.573 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 582.08233041200003, 2024-02-09T17:50:00.573 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.3778037960000002, 2024-02-09T17:50:00.573 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:50:00.573 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "reached pg", 2024-02-09T17:50:00.573 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:50:00.573 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:50:00.573 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:50:00.573 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 266604 2024-02-09T17:50:00.573 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.573 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:50:00.573 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.573 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:50:00.574 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.476844+0000", 2024-02-09T17:50:00.574 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:50:00.574 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.574 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.574 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:50:00.574 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.476844+0000", 2024-02-09T17:50:00.574 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:50:00.574 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.574 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.574 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:50:00.574 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.476846+0000", 2024-02-09T17:50:00.574 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1230000000000001e-06 2024-02-09T17:50:00.574 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.574 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.574 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:50:00.575 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.476849+0000", 2024-02-09T17:50:00.575 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.878e-06 2024-02-09T17:50:00.575 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.575 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.575 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:50:00.575 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.476860+0000", 2024-02-09T17:50:00.575 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0628999999999999e-05 2024-02-09T17:50:00.575 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.575 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.575 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:50:00.575 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.476868+0000", 2024-02-09T17:50:00.575 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.7549999999999994e-06 2024-02-09T17:50:00.575 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.575 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.576 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:50:00.576 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.481266+0000", 2024-02-09T17:50:00.576 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0043979129999999998 2024-02-09T17:50:00.576 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.576 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.576 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:50:00.576 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.481427+0000", 2024-02-09T17:50:00.576 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00016158500000000001 2024-02-09T17:50:00.576 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.576 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.576 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,8", 2024-02-09T17:50:00.576 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.481557+0000", 2024-02-09T17:50:00.576 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000129827 2024-02-09T17:50:00.577 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.577 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.577 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:50:00.577 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.485332+0000", 2024-02-09T17:50:00.577 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.003775203 2024-02-09T17:50:00.577 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.577 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.577 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:50:00.577 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.488518+0000", 2024-02-09T17:50:00.577 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0031852690000000001 2024-02-09T17:50:00.577 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.577 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.577 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:50:00.578 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.488530+0000", 2024-02-09T17:50:00.578 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.26e-05 2024-02-09T17:50:00.578 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.578 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.578 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:50:00.578 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.483159+0000", 2024-02-09T17:50:00.578 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.99462922899999995 2024-02-09T17:50:00.578 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.578 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.578 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:50:00.578 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.483180+0000", 2024-02-09T17:50:00.578 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0540999999999999e-05 2024-02-09T17:50:00.578 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.578 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.579 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:50:00.579 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.854593+0000", 2024-02-09T17:50:00.579 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.371412949 2024-02-09T17:50:00.579 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.579 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.579 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:50:00.579 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.854648+0000", 2024-02-09T17:50:00.579 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.5294999999999998e-05 2024-02-09T17:50:00.579 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:00.579 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:00.579 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:00.579 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.579 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.579 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:266623 4.0 4:073f302d:::benchmark_data_smithi179_103809_object129157:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e451)", 2024-02-09T17:50:00.579 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:40:18.501942+0000", 2024-02-09T17:50:00.580 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 582.05723313800002, 2024-02-09T17:50:00.580 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.3660463040000002, 2024-02-09T17:50:00.580 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:50:00.580 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:00.580 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:50:00.580 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:50:00.580 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:50:00.580 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 266623 2024-02-09T17:50:00.580 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.580 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:50:00.580 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.580 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:50:00.580 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.501942+0000", 2024-02-09T17:50:00.580 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:50:00.581 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.581 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.581 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:50:00.581 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.501942+0000", 2024-02-09T17:50:00.581 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:50:00.581 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.581 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.581 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:50:00.581 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.501944+0000", 2024-02-09T17:50:00.581 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0020000000000001e-06 2024-02-09T17:50:00.581 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.581 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.581 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:50:00.581 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.501951+0000", 2024-02-09T17:50:00.581 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.4749999999999996e-06 2024-02-09T17:50:00.581 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.582 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.582 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:50:00.582 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.501967+0000", 2024-02-09T17:50:00.582 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5773e-05 2024-02-09T17:50:00.582 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.582 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.582 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:50:00.582 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.501975+0000", 2024-02-09T17:50:00.582 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.2050000000000002e-06 2024-02-09T17:50:00.582 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.582 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.582 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:50:00.582 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.502044+0000", 2024-02-09T17:50:00.583 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.8769000000000005e-05 2024-02-09T17:50:00.583 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.583 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.583 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:50:00.583 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.502052+0000", 2024-02-09T17:50:00.583 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.9009999999999994e-06 2024-02-09T17:50:00.583 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.583 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.583 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:50:00.583 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.483232+0000", 2024-02-09T17:50:00.583 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.9811801 2024-02-09T17:50:00.583 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.583 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.583 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:50:00.583 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.483243+0000", 2024-02-09T17:50:00.583 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1342000000000001e-05 2024-02-09T17:50:00.584 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.584 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.584 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:50:00.584 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.854856+0000", 2024-02-09T17:50:00.584 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3716124439999999 2024-02-09T17:50:00.584 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.584 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.584 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:50:00.584 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.854941+0000", 2024-02-09T17:50:00.584 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.4888000000000004e-05 2024-02-09T17:50:00.584 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.584 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.584 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,7,8", 2024-02-09T17:50:00.584 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.855027+0000", 2024-02-09T17:50:00.585 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.6080000000000003e-05 2024-02-09T17:50:00.585 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.585 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.585 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:50:00.585 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.867473+0000", 2024-02-09T17:50:00.585 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.012446726 2024-02-09T17:50:00.585 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.585 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.585 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.585 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.867528+0000", 2024-02-09T17:50:00.585 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.4917999999999998e-05 2024-02-09T17:50:00.585 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.585 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.585 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.586 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.867662+0000", 2024-02-09T17:50:00.586 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000133738 2024-02-09T17:50:00.586 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.586 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.586 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.586 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.867788+0000", 2024-02-09T17:50:00.586 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012569 2024-02-09T17:50:00.586 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.586 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.586 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:50:00.586 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.867856+0000", 2024-02-09T17:50:00.586 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.8036999999999999e-05 2024-02-09T17:50:00.586 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.586 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.586 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:50:00.587 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.867988+0000", 2024-02-09T17:50:00.587 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000132216 2024-02-09T17:50:00.587 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:00.587 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:00.587 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:00.587 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.587 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.587 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:266638 4.0 4:0dbae452:::benchmark_data_smithi179_103809_object129172:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e451)", 2024-02-09T17:50:00.587 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:40:18.517152+0000", 2024-02-09T17:50:00.587 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 582.04202320000002, 2024-02-09T17:50:00.587 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.3513199650000001, 2024-02-09T17:50:00.587 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:50:00.587 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:00.587 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:50:00.587 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:50:00.587 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:50:00.588 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 266638 2024-02-09T17:50:00.588 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.588 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:50:00.588 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.588 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:50:00.588 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.517152+0000", 2024-02-09T17:50:00.588 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:50:00.588 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.588 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.588 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:50:00.588 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.517152+0000", 2024-02-09T17:50:00.588 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:50:00.588 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.588 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.589 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:50:00.589 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.517153+0000", 2024-02-09T17:50:00.589 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2580000000000001e-06 2024-02-09T17:50:00.589 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.589 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.589 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:50:00.589 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.517157+0000", 2024-02-09T17:50:00.589 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.3000000000000003e-06 2024-02-09T17:50:00.589 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.589 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.589 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:50:00.589 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.517168+0000", 2024-02-09T17:50:00.589 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1049e-05 2024-02-09T17:50:00.589 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.589 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.589 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:50:00.590 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.517174+0000", 2024-02-09T17:50:00.590 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7219999999999996e-06 2024-02-09T17:50:00.590 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.590 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.590 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:50:00.590 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.517229+0000", 2024-02-09T17:50:00.590 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.4761e-05 2024-02-09T17:50:00.590 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.590 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.590 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:50:00.590 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.517236+0000", 2024-02-09T17:50:00.590 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.1099999999999997e-06 2024-02-09T17:50:00.590 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.590 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.590 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:50:00.591 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.483283+0000", 2024-02-09T17:50:00.591 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.96604708800000005 2024-02-09T17:50:00.591 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.591 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.591 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:50:00.591 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.483290+0000", 2024-02-09T17:50:00.591 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.9779999999999999e-06 2024-02-09T17:50:00.591 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.591 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.591 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:50:00.591 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.858015+0000", 2024-02-09T17:50:00.591 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3747255599999999 2024-02-09T17:50:00.591 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.591 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.591 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:50:00.591 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.858147+0000", 2024-02-09T17:50:00.592 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013195399999999999 2024-02-09T17:50:00.592 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.592 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.592 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,7,8", 2024-02-09T17:50:00.592 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.858296+0000", 2024-02-09T17:50:00.592 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014873100000000001 2024-02-09T17:50:00.592 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.592 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.592 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:50:00.592 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.867483+0000", 2024-02-09T17:50:00.592 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0091869550000000001 2024-02-09T17:50:00.592 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.592 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.592 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.592 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.867588+0000", 2024-02-09T17:50:00.593 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000104653 2024-02-09T17:50:00.593 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.593 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.593 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.593 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.867729+0000", 2024-02-09T17:50:00.593 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014101399999999999 2024-02-09T17:50:00.593 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.593 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.593 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.593 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.868283+0000", 2024-02-09T17:50:00.593 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00055400800000000004 2024-02-09T17:50:00.593 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.593 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.593 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:50:00.593 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.868343+0000", 2024-02-09T17:50:00.594 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.0504999999999998e-05 2024-02-09T17:50:00.594 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.594 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.594 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:50:00.594 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.868472+0000", 2024-02-09T17:50:00.594 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000128319 2024-02-09T17:50:00.594 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:00.594 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:00.594 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:00.594 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.594 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.594 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:266641 4.0 4:04e380f9:::benchmark_data_smithi179_103809_object129175:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e451)", 2024-02-09T17:50:00.594 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:40:18.521970+0000", 2024-02-09T17:50:00.594 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 582.03720497100005, 2024-02-09T17:50:00.594 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.346639878, 2024-02-09T17:50:00.595 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:50:00.595 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:00.595 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:50:00.595 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:50:00.595 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:50:00.595 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 266641 2024-02-09T17:50:00.595 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.595 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:50:00.595 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.595 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:50:00.595 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.521970+0000", 2024-02-09T17:50:00.595 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:50:00.595 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.595 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.595 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:50:00.596 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.521970+0000", 2024-02-09T17:50:00.596 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:50:00.596 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.596 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.596 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:50:00.596 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.521971+0000", 2024-02-09T17:50:00.596 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4929999999999999e-06 2024-02-09T17:50:00.596 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.596 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.596 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:50:00.596 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.521975+0000", 2024-02-09T17:50:00.596 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.54e-06 2024-02-09T17:50:00.596 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.596 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.596 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:50:00.597 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.521986+0000", 2024-02-09T17:50:00.597 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0917000000000001e-05 2024-02-09T17:50:00.597 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.597 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.597 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:50:00.597 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.521992+0000", 2024-02-09T17:50:00.597 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.1569999999999997e-06 2024-02-09T17:50:00.597 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.597 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.597 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:50:00.597 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.522046+0000", 2024-02-09T17:50:00.597 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3733000000000002e-05 2024-02-09T17:50:00.597 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.597 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.597 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:50:00.598 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.522057+0000", 2024-02-09T17:50:00.598 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0991e-05 2024-02-09T17:50:00.598 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.598 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.598 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:50:00.598 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.483324+0000", 2024-02-09T17:50:00.598 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.961266858 2024-02-09T17:50:00.598 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.598 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.598 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:50:00.598 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.483334+0000", 2024-02-09T17:50:00.598 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0054999999999999e-05 2024-02-09T17:50:00.598 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.598 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.598 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:50:00.599 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.862180+0000", 2024-02-09T17:50:00.599 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.378846161 2024-02-09T17:50:00.599 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.599 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.599 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:50:00.599 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.862335+0000", 2024-02-09T17:50:00.599 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000155162 2024-02-09T17:50:00.599 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.599 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.599 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,7,8", 2024-02-09T17:50:00.599 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.862512+0000", 2024-02-09T17:50:00.599 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000176589 2024-02-09T17:50:00.599 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.599 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.599 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:50:00.599 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.867607+0000", 2024-02-09T17:50:00.600 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0050958469999999997 2024-02-09T17:50:00.600 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.600 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.600 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.600 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.868029+0000", 2024-02-09T17:50:00.600 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000421156 2024-02-09T17:50:00.600 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.600 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.600 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.600 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.868079+0000", 2024-02-09T17:50:00.600 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.0906999999999999e-05 2024-02-09T17:50:00.600 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.600 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.600 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.600 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.868512+0000", 2024-02-09T17:50:00.601 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00043259100000000002 2024-02-09T17:50:00.601 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.601 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.601 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:50:00.601 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.868548+0000", 2024-02-09T17:50:00.601 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.6307000000000003e-05 2024-02-09T17:50:00.601 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.601 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.601 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:50:00.601 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.868610+0000", 2024-02-09T17:50:00.601 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.1414000000000005e-05 2024-02-09T17:50:00.601 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:00.601 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:00.601 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:00.601 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.602 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.602 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:266661 4.0 4:074c4478:::benchmark_data_smithi179_103809_object129195:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e451)", 2024-02-09T17:50:00.602 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:40:18.539555+0000", 2024-02-09T17:50:00.602 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 582.01961979299995, 2024-02-09T17:50:00.602 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.359289032, 2024-02-09T17:50:00.602 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:50:00.602 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:00.602 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:50:00.602 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:50:00.602 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:50:00.602 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 266661 2024-02-09T17:50:00.602 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.602 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:50:00.602 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.602 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:50:00.602 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.539555+0000", 2024-02-09T17:50:00.603 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:50:00.603 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.603 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.603 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:50:00.603 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.539555+0000", 2024-02-09T17:50:00.603 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:50:00.603 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.603 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.603 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:50:00.603 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.539557+0000", 2024-02-09T17:50:00.603 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.463e-06 2024-02-09T17:50:00.603 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.603 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.603 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:50:00.604 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.539562+0000", 2024-02-09T17:50:00.604 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.5110000000000003e-06 2024-02-09T17:50:00.604 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.604 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.604 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:50:00.604 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.539579+0000", 2024-02-09T17:50:00.604 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.6719999999999999e-05 2024-02-09T17:50:00.604 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.604 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.604 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:50:00.604 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.539587+0000", 2024-02-09T17:50:00.604 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.7229999999999993e-06 2024-02-09T17:50:00.604 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.604 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.604 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:50:00.605 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.539655+0000", 2024-02-09T17:50:00.605 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.7496000000000006e-05 2024-02-09T17:50:00.605 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.605 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.605 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:50:00.605 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.539665+0000", 2024-02-09T17:50:00.605 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0074999999999999e-05 2024-02-09T17:50:00.605 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.605 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.605 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:50:00.605 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.483371+0000", 2024-02-09T17:50:00.605 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.94370588600000005 2024-02-09T17:50:00.605 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.605 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.605 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:50:00.605 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.483380+0000", 2024-02-09T17:50:00.606 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.3230000000000004e-06 2024-02-09T17:50:00.606 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.606 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.606 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:50:00.606 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.868647+0000", 2024-02-09T17:50:00.606 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3852665559999999 2024-02-09T17:50:00.606 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.606 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.606 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:50:00.606 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.868796+0000", 2024-02-09T17:50:00.606 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000148873 2024-02-09T17:50:00.606 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.606 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.606 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,7,8", 2024-02-09T17:50:00.606 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.868957+0000", 2024-02-09T17:50:00.607 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00016103100000000001 2024-02-09T17:50:00.607 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.607 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.607 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.607 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.882839+0000", 2024-02-09T17:50:00.607 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.013881968 2024-02-09T17:50:00.607 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.607 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.607 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:50:00.607 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.882857+0000", 2024-02-09T17:50:00.607 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.8167000000000001e-05 2024-02-09T17:50:00.607 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.607 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.607 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.607 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.882908+0000", 2024-02-09T17:50:00.607 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1359000000000001e-05 2024-02-09T17:50:00.608 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.608 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.608 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.608 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.898655+0000", 2024-02-09T17:50:00.608 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.015746577000000001 2024-02-09T17:50:00.608 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.608 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.608 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:50:00.608 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.898721+0000", 2024-02-09T17:50:00.608 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.6357000000000002e-05 2024-02-09T17:50:00.608 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.608 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.608 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:50:00.608 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.898844+0000", 2024-02-09T17:50:00.608 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012294699999999999 2024-02-09T17:50:00.609 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:00.609 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:00.609 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:00.609 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.609 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.609 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:266678 4.0 4:028686c1:::benchmark_data_smithi179_103809_object129212:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e451)", 2024-02-09T17:50:00.609 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:40:18.561032+0000", 2024-02-09T17:50:00.609 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 581.99814269700005, 2024-02-09T17:50:00.609 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.3696253609999998, 2024-02-09T17:50:00.609 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:50:00.609 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:00.609 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:50:00.609 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:50:00.609 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:50:00.609 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 266678 2024-02-09T17:50:00.610 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.610 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:50:00.610 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.610 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:50:00.610 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.561032+0000", 2024-02-09T17:50:00.610 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:50:00.610 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.610 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.610 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:50:00.610 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.561032+0000", 2024-02-09T17:50:00.610 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:50:00.610 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.610 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.610 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:50:00.610 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.561034+0000", 2024-02-09T17:50:00.611 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.745e-06 2024-02-09T17:50:00.611 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.611 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.611 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:50:00.611 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.561042+0000", 2024-02-09T17:50:00.611 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.6669999999999996e-06 2024-02-09T17:50:00.611 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.611 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.611 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:50:00.611 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.561057+0000", 2024-02-09T17:50:00.611 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5900999999999999e-05 2024-02-09T17:50:00.611 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.611 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.611 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:50:00.611 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.561066+0000", 2024-02-09T17:50:00.611 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.9989999999999997e-06 2024-02-09T17:50:00.612 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.612 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.612 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:50:00.612 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.561141+0000", 2024-02-09T17:50:00.612 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.4782000000000003e-05 2024-02-09T17:50:00.612 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.612 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.612 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:50:00.612 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.561155+0000", 2024-02-09T17:50:00.612 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3713000000000001e-05 2024-02-09T17:50:00.612 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.612 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.612 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:50:00.612 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.483417+0000", 2024-02-09T17:50:00.612 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.92226239399999999 2024-02-09T17:50:00.613 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.613 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.613 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:50:00.613 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.483427+0000", 2024-02-09T17:50:00.613 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.6390000000000004e-06 2024-02-09T17:50:00.613 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.613 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.613 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:50:00.613 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.872509+0000", 2024-02-09T17:50:00.613 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3890821090000001 2024-02-09T17:50:00.613 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.613 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.613 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:50:00.613 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.872638+0000", 2024-02-09T17:50:00.613 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012891399999999999 2024-02-09T17:50:00.614 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.614 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.614 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,7,8", 2024-02-09T17:50:00.614 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.872748+0000", 2024-02-09T17:50:00.614 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000109566 2024-02-09T17:50:00.614 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.614 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.614 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:50:00.614 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.882864+0000", 2024-02-09T17:50:00.614 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.010116066 2024-02-09T17:50:00.614 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.614 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.614 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.614 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.891269+0000", 2024-02-09T17:50:00.614 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0084049529999999997 2024-02-09T17:50:00.614 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.615 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.615 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.615 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.898893+0000", 2024-02-09T17:50:00.615 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0076241540000000002 2024-02-09T17:50:00.615 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.615 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.615 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.615 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.930449+0000", 2024-02-09T17:50:00.615 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.031556188999999998 2024-02-09T17:50:00.615 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.615 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.615 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:50:00.615 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.930520+0000", 2024-02-09T17:50:00.615 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.1021999999999999e-05 2024-02-09T17:50:00.615 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.616 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.616 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:50:00.616 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.930658+0000", 2024-02-09T17:50:00.616 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013754800000000001 2024-02-09T17:50:00.616 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:00.616 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:00.616 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:00.616 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.616 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.616 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:266686 4.0 4:0303d92f:::benchmark_data_smithi179_103809_object129220:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e451)", 2024-02-09T17:50:00.616 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:40:18.569926+0000", 2024-02-09T17:50:00.616 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 581.98924855500002, 2024-02-09T17:50:00.616 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.3609762810000001, 2024-02-09T17:50:00.616 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:50:00.616 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:00.617 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:50:00.617 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:50:00.617 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:50:00.617 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 266686 2024-02-09T17:50:00.617 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.617 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:50:00.617 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.617 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:50:00.617 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.569926+0000", 2024-02-09T17:50:00.617 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:50:00.617 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.617 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.617 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:50:00.617 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.569926+0000", 2024-02-09T17:50:00.617 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:50:00.617 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.618 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.618 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:50:00.618 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.569928+0000", 2024-02-09T17:50:00.618 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.739e-06 2024-02-09T17:50:00.618 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.618 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.618 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:50:00.618 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.569935+0000", 2024-02-09T17:50:00.618 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.9990000000000002e-06 2024-02-09T17:50:00.618 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.618 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.618 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:50:00.618 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.569951+0000", 2024-02-09T17:50:00.618 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.6235999999999998e-05 2024-02-09T17:50:00.618 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.619 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.619 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:50:00.619 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.569961+0000", 2024-02-09T17:50:00.619 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.4080000000000004e-06 2024-02-09T17:50:00.619 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.619 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.619 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:50:00.619 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.570034+0000", 2024-02-09T17:50:00.619 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.3468999999999997e-05 2024-02-09T17:50:00.619 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.619 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.619 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:50:00.619 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.570045+0000", 2024-02-09T17:50:00.619 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0720000000000001e-05 2024-02-09T17:50:00.619 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.620 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.620 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:50:00.620 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.483456+0000", 2024-02-09T17:50:00.620 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.91341088699999995 2024-02-09T17:50:00.620 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.620 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.620 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:50:00.620 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.483461+0000", 2024-02-09T17:50:00.620 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.5659999999999997e-06 2024-02-09T17:50:00.620 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.620 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.620 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:50:00.620 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.876132+0000", 2024-02-09T17:50:00.620 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.392670587 2024-02-09T17:50:00.621 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.621 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.621 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:50:00.621 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.876277+0000", 2024-02-09T17:50:00.621 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000145196 2024-02-09T17:50:00.621 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.621 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.621 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,7,8", 2024-02-09T17:50:00.621 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.876416+0000", 2024-02-09T17:50:00.621 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013853300000000001 2024-02-09T17:50:00.621 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.621 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.621 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:50:00.621 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.891214+0000", 2024-02-09T17:50:00.621 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.014798830000000001 2024-02-09T17:50:00.621 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.622 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.622 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.622 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.891310+0000", 2024-02-09T17:50:00.622 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.5465999999999997e-05 2024-02-09T17:50:00.622 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.622 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.622 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.622 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.898935+0000", 2024-02-09T17:50:00.622 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0076249250000000003 2024-02-09T17:50:00.622 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.622 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.622 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.622 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.930706+0000", 2024-02-09T17:50:00.622 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.031771630000000002 2024-02-09T17:50:00.622 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.623 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.623 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:50:00.623 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.930776+0000", 2024-02-09T17:50:00.623 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.9413999999999996e-05 2024-02-09T17:50:00.623 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.623 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.623 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:50:00.623 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.930903+0000", 2024-02-09T17:50:00.623 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012667599999999999 2024-02-09T17:50:00.623 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:00.623 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:00.623 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:00.623 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.623 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.624 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:266691 4.0 4:09e5f71f:::benchmark_data_smithi179_103809_object129225:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e451)", 2024-02-09T17:50:00.624 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:40:18.575773+0000", 2024-02-09T17:50:00.624 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 581.98340218800001, 2024-02-09T17:50:00.624 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.3553655469999999, 2024-02-09T17:50:00.624 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:50:00.624 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:00.624 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:50:00.624 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:50:00.624 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:50:00.624 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 266691 2024-02-09T17:50:00.624 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.624 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:50:00.624 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.624 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:50:00.625 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.575773+0000", 2024-02-09T17:50:00.625 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:50:00.625 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.625 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.625 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:50:00.625 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.575773+0000", 2024-02-09T17:50:00.625 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:50:00.625 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.625 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.625 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:50:00.625 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.575774+0000", 2024-02-09T17:50:00.625 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5719999999999999e-06 2024-02-09T17:50:00.625 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.625 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.625 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:50:00.626 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.575780+0000", 2024-02-09T17:50:00.626 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6389999999999997e-06 2024-02-09T17:50:00.626 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.626 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.626 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:50:00.626 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.575790+0000", 2024-02-09T17:50:00.626 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0131e-05 2024-02-09T17:50:00.626 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.626 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.626 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:50:00.626 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.575796+0000", 2024-02-09T17:50:00.626 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.1090000000000003e-06 2024-02-09T17:50:00.626 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.626 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.627 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:50:00.627 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.575853+0000", 2024-02-09T17:50:00.627 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7098000000000001e-05 2024-02-09T17:50:00.627 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.627 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.627 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:50:00.627 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.575866+0000", 2024-02-09T17:50:00.627 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2649999999999999e-05 2024-02-09T17:50:00.627 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.627 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.627 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:50:00.627 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.483490+0000", 2024-02-09T17:50:00.627 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.90762436999999996 2024-02-09T17:50:00.627 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.627 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.628 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:50:00.628 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.483499+0000", 2024-02-09T17:50:00.628 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.4710000000000007e-06 2024-02-09T17:50:00.628 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.628 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.628 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:50:00.628 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.879384+0000", 2024-02-09T17:50:00.628 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3958855050000001 2024-02-09T17:50:00.628 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.628 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.628 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:50:00.628 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.879513+0000", 2024-02-09T17:50:00.628 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012911300000000001 2024-02-09T17:50:00.628 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.629 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.629 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,7,8", 2024-02-09T17:50:00.629 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.879633+0000", 2024-02-09T17:50:00.629 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00011935 2024-02-09T17:50:00.629 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.629 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.629 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:50:00.629 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.891222+0000", 2024-02-09T17:50:00.629 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.011589153 2024-02-09T17:50:00.629 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.629 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.629 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.629 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.891355+0000", 2024-02-09T17:50:00.629 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000133363 2024-02-09T17:50:00.629 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.629 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.630 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.630 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.898976+0000", 2024-02-09T17:50:00.630 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0076204810000000001 2024-02-09T17:50:00.630 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.630 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.630 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.630 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.930950+0000", 2024-02-09T17:50:00.630 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.031974834000000001 2024-02-09T17:50:00.630 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.630 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.630 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:50:00.630 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.931012+0000", 2024-02-09T17:50:00.630 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.1252000000000002e-05 2024-02-09T17:50:00.630 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.631 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.631 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:50:00.631 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.931138+0000", 2024-02-09T17:50:00.631 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000126456 2024-02-09T17:50:00.631 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:00.631 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:00.631 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:00.631 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.631 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.631 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:266698 4.0 4:013e4c1f:::benchmark_data_smithi179_103809_object129232:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e451)", 2024-02-09T17:50:00.631 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:40:18.585928+0000", 2024-02-09T17:50:00.631 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 581.97324701699995, 2024-02-09T17:50:00.631 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.3480866659999999, 2024-02-09T17:50:00.631 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:50:00.631 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:00.632 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:50:00.632 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:50:00.632 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:50:00.632 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 266698 2024-02-09T17:50:00.632 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.632 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:50:00.632 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.632 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:50:00.632 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.585928+0000", 2024-02-09T17:50:00.632 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:50:00.632 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.632 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.633 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:50:00.633 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.585928+0000", 2024-02-09T17:50:00.633 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:50:00.633 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.633 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.633 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:50:00.635 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.585929+0000", 2024-02-09T17:50:00.635 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4640000000000001e-06 2024-02-09T17:50:00.635 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.636 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.636 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:50:00.636 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.585936+0000", 2024-02-09T17:50:00.636 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.0639999999999996e-06 2024-02-09T17:50:00.636 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.636 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.636 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:50:00.636 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.585950+0000", 2024-02-09T17:50:00.636 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.384e-05 2024-02-09T17:50:00.636 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.636 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.636 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:50:00.636 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.585959+0000", 2024-02-09T17:50:00.636 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.7229999999999993e-06 2024-02-09T17:50:00.636 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.637 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.637 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:50:00.637 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.586016+0000", 2024-02-09T17:50:00.637 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7076e-05 2024-02-09T17:50:00.637 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.637 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.637 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:50:00.637 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:18.586025+0000", 2024-02-09T17:50:00.637 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.8310000000000007e-06 2024-02-09T17:50:00.637 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.637 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.637 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:50:00.637 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.483514+0000", 2024-02-09T17:50:00.637 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.89748914899999999 2024-02-09T17:50:00.637 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.638 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.638 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:50:00.638 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.483518+0000", 2024-02-09T17:50:00.638 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.4240000000000003e-06 2024-02-09T17:50:00.638 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.638 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.638 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:50:00.638 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.882954+0000", 2024-02-09T17:50:00.638 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.399435593 2024-02-09T17:50:00.638 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.638 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.638 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:50:00.638 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.883080+0000", 2024-02-09T17:50:00.638 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012615799999999999 2024-02-09T17:50:00.638 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.639 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.639 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,7,8", 2024-02-09T17:50:00.639 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.883168+0000", 2024-02-09T17:50:00.639 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.8119000000000001e-05 2024-02-09T17:50:00.639 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.639 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.639 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.639 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.899022+0000", 2024-02-09T17:50:00.639 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.015853882999999999 2024-02-09T17:50:00.639 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.639 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.639 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.639 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.899067+0000", 2024-02-09T17:50:00.639 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.4490000000000003e-05 2024-02-09T17:50:00.639 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.640 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.640 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:50:00.640 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.914844+0000", 2024-02-09T17:50:00.640 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.015777831999999999 2024-02-09T17:50:00.640 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.640 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.640 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.640 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.933838+0000", 2024-02-09T17:50:00.640 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.018993385000000002 2024-02-09T17:50:00.640 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.640 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.640 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:50:00.640 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.933897+0000", 2024-02-09T17:50:00.640 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.9117999999999999e-05 2024-02-09T17:50:00.640 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.641 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.641 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:50:00.641 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.934014+0000", 2024-02-09T17:50:00.641 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00011751700000000001 2024-02-09T17:50:00.641 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:00.641 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:00.641 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:00.641 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.641 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.641 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:266724 4.0 4:034dec6d:::benchmark_data_smithi179_103809_object129258:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e452)", 2024-02-09T17:50:00.641 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:40:19.700381+0000", 2024-02-09T17:50:00.641 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 580.85879369600002, 2024-02-09T17:50:00.641 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2338328670000001, 2024-02-09T17:50:00.641 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:50:00.641 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:00.642 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:50:00.642 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:50:00.642 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:50:00.642 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 266724 2024-02-09T17:50:00.642 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.642 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:50:00.642 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.642 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:50:00.642 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.700381+0000", 2024-02-09T17:50:00.642 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:50:00.642 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.642 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.642 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:50:00.642 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.700381+0000", 2024-02-09T17:50:00.642 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:50:00.643 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.643 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.643 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:50:00.643 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.700383+0000", 2024-02-09T17:50:00.643 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.657e-06 2024-02-09T17:50:00.643 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.643 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.643 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:50:00.643 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.700393+0000", 2024-02-09T17:50:00.643 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0210000000000001e-05 2024-02-09T17:50:00.643 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.643 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.643 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:50:00.643 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.700407+0000", 2024-02-09T17:50:00.643 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4229999999999999e-05 2024-02-09T17:50:00.644 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.644 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.644 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:50:00.644 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.700432+0000", 2024-02-09T17:50:00.644 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.4698999999999998e-05 2024-02-09T17:50:00.644 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.644 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.644 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:50:00.644 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.700497+0000", 2024-02-09T17:50:00.644 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.5208000000000003e-05 2024-02-09T17:50:00.644 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.644 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.644 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:50:00.644 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.700507+0000", 2024-02-09T17:50:00.645 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.764e-06 2024-02-09T17:50:00.645 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.645 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.645 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:50:00.645 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.886708+0000", 2024-02-09T17:50:00.645 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.186200994 2024-02-09T17:50:00.645 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.645 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.645 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:50:00.645 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.886821+0000", 2024-02-09T17:50:00.645 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000112748 2024-02-09T17:50:00.645 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.645 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.645 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,7,8", 2024-02-09T17:50:00.646 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.886944+0000", 2024-02-09T17:50:00.646 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000123511 2024-02-09T17:50:00.646 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.646 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.646 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.646 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.899110+0000", 2024-02-09T17:50:00.646 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.012165904999999999 2024-02-09T17:50:00.646 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.646 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.646 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.646 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.906494+0000", 2024-02-09T17:50:00.646 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0073838860000000001 2024-02-09T17:50:00.646 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.646 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.647 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:50:00.647 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.914852+0000", 2024-02-09T17:50:00.647 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0083584909999999991 2024-02-09T17:50:00.647 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.647 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.647 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.647 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.934059+0000", 2024-02-09T17:50:00.647 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.019206961000000002 2024-02-09T17:50:00.647 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.647 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.647 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:50:00.647 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.934115+0000", 2024-02-09T17:50:00.647 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.5525000000000001e-05 2024-02-09T17:50:00.647 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.647 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.648 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:50:00.648 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.934214+0000", 2024-02-09T17:50:00.648 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.9078000000000002e-05 2024-02-09T17:50:00.648 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:00.648 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:00.648 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:00.648 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.648 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.648 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:266737 4.0 4:0a3fc49d:::benchmark_data_smithi179_103809_object129271:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e452)", 2024-02-09T17:50:00.648 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:40:19.720932+0000", 2024-02-09T17:50:00.648 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 580.83824285900005, 2024-02-09T17:50:00.648 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2134803000000001, 2024-02-09T17:50:00.648 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:50:00.648 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:00.649 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:50:00.649 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:50:00.649 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:50:00.649 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 266737 2024-02-09T17:50:00.649 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.649 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:50:00.649 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.649 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:50:00.649 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.720932+0000", 2024-02-09T17:50:00.649 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:50:00.649 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.649 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.649 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:50:00.649 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.720932+0000", 2024-02-09T17:50:00.649 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:50:00.650 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.650 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.650 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:50:00.650 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.720933+0000", 2024-02-09T17:50:00.650 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.435e-06 2024-02-09T17:50:00.650 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.650 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.650 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:50:00.650 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.720938+0000", 2024-02-09T17:50:00.650 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.9570000000000001e-06 2024-02-09T17:50:00.650 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.650 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.650 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:50:00.650 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.720949+0000", 2024-02-09T17:50:00.650 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0434e-05 2024-02-09T17:50:00.651 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.651 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.651 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:50:00.651 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.720955+0000", 2024-02-09T17:50:00.651 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.8549999999999998e-06 2024-02-09T17:50:00.651 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.651 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.651 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:50:00.651 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.721020+0000", 2024-02-09T17:50:00.651 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.5612999999999997e-05 2024-02-09T17:50:00.651 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.651 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.651 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:50:00.651 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.721032+0000", 2024-02-09T17:50:00.651 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.202e-05 2024-02-09T17:50:00.651 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.652 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.652 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:50:00.652 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.891395+0000", 2024-02-09T17:50:00.652 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.170362312 2024-02-09T17:50:00.652 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.652 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.652 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:50:00.652 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.891537+0000", 2024-02-09T17:50:00.652 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000141927 2024-02-09T17:50:00.652 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.652 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.652 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,7,8", 2024-02-09T17:50:00.652 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.891665+0000", 2024-02-09T17:50:00.652 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012851400000000001 2024-02-09T17:50:00.653 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.653 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.653 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.653 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.906526+0000", 2024-02-09T17:50:00.653 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.014861048 2024-02-09T17:50:00.653 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.653 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.653 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.653 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.914898+0000", 2024-02-09T17:50:00.653 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0083720670000000004 2024-02-09T17:50:00.653 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.653 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.653 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:50:00.653 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.918333+0000", 2024-02-09T17:50:00.653 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.003434772 2024-02-09T17:50:00.654 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.654 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.654 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.654 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.934252+0000", 2024-02-09T17:50:00.654 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.015919295999999999 2024-02-09T17:50:00.654 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.654 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.654 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:50:00.654 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.934300+0000", 2024-02-09T17:50:00.654 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.7465e-05 2024-02-09T17:50:00.654 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.654 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.654 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:50:00.654 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.934412+0000", 2024-02-09T17:50:00.654 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00011258499999999999 2024-02-09T17:50:00.655 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:00.655 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:00.655 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:00.655 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.655 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.655 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:266746 4.0 4:0131b814:::benchmark_data_smithi179_103809_object129280:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e452)", 2024-02-09T17:50:00.655 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:40:19.735426+0000", 2024-02-09T17:50:00.655 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 580.82374896700003, 2024-02-09T17:50:00.655 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.199168945, 2024-02-09T17:50:00.655 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:50:00.655 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:00.655 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:50:00.655 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:50:00.655 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:50:00.655 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 266746 2024-02-09T17:50:00.655 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.656 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:50:00.656 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.656 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:50:00.656 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.735426+0000", 2024-02-09T17:50:00.656 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:50:00.656 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.656 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.656 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:50:00.656 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.735426+0000", 2024-02-09T17:50:00.656 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:50:00.656 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.656 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.656 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:50:00.656 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.735427+0000", 2024-02-09T17:50:00.656 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.325e-06 2024-02-09T17:50:00.657 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.657 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.657 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:50:00.657 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.735432+0000", 2024-02-09T17:50:00.657 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.006e-06 2024-02-09T17:50:00.657 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.657 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.657 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:50:00.657 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.735443+0000", 2024-02-09T17:50:00.657 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0974e-05 2024-02-09T17:50:00.657 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.657 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.657 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:50:00.657 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.735450+0000", 2024-02-09T17:50:00.657 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.0190000000000001e-06 2024-02-09T17:50:00.658 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.658 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.658 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:50:00.658 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.735510+0000", 2024-02-09T17:50:00.658 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.9342999999999997e-05 2024-02-09T17:50:00.658 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.658 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.658 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:50:00.658 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.735519+0000", 2024-02-09T17:50:00.658 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.2410000000000001e-06 2024-02-09T17:50:00.658 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.658 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.658 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:50:00.658 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.894839+0000", 2024-02-09T17:50:00.658 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.159320006 2024-02-09T17:50:00.659 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.659 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.659 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:50:00.659 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.894939+0000", 2024-02-09T17:50:00.659 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.9728000000000005e-05 2024-02-09T17:50:00.659 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.659 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.659 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,7,8", 2024-02-09T17:50:00.659 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.895056+0000", 2024-02-09T17:50:00.659 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000117343 2024-02-09T17:50:00.659 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.659 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.659 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.659 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.906557+0000", 2024-02-09T17:50:00.659 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.011500707000000001 2024-02-09T17:50:00.659 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.660 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.660 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:50:00.660 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.918340+0000", 2024-02-09T17:50:00.660 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.011783745999999999 2024-02-09T17:50:00.660 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.660 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.660 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.660 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.918525+0000", 2024-02-09T17:50:00.660 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00018445000000000001 2024-02-09T17:50:00.660 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.660 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.660 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.660 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.934450+0000", 2024-02-09T17:50:00.660 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.015925290000000002 2024-02-09T17:50:00.660 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.661 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.661 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:50:00.661 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.934498+0000", 2024-02-09T17:50:00.661 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.7579e-05 2024-02-09T17:50:00.661 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.661 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.661 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:50:00.661 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.934595+0000", 2024-02-09T17:50:00.661 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.7188000000000005e-05 2024-02-09T17:50:00.661 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:00.661 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:00.661 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:00.661 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.661 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.661 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:266759 4.0 4:0f82b435:::benchmark_data_smithi179_103809_object129293:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e452)", 2024-02-09T17:50:00.661 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:40:19.759481+0000", 2024-02-09T17:50:00.662 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 580.79969337299997, 2024-02-09T17:50:00.662 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1752948759999999, 2024-02-09T17:50:00.662 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:50:00.662 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:00.662 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:50:00.662 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:50:00.662 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:50:00.662 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 266759 2024-02-09T17:50:00.662 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.662 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:50:00.662 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.662 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:50:00.662 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.759481+0000", 2024-02-09T17:50:00.662 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:50:00.662 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.663 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.663 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:50:00.663 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.759481+0000", 2024-02-09T17:50:00.663 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:50:00.663 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.663 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.663 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:50:00.663 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.759483+0000", 2024-02-09T17:50:00.663 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4100000000000001e-06 2024-02-09T17:50:00.663 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.663 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.663 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:50:00.663 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.759487+0000", 2024-02-09T17:50:00.663 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.7969999999999998e-06 2024-02-09T17:50:00.663 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.663 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.664 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:50:00.664 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.759498+0000", 2024-02-09T17:50:00.664 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1554e-05 2024-02-09T17:50:00.664 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.664 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.664 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:50:00.664 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.759505+0000", 2024-02-09T17:50:00.664 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.4370000000000004e-06 2024-02-09T17:50:00.664 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.664 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.664 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:50:00.664 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.759568+0000", 2024-02-09T17:50:00.664 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.3644000000000003e-05 2024-02-09T17:50:00.664 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.665 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.665 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:50:00.665 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.759579+0000", 2024-02-09T17:50:00.665 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0844e-05 2024-02-09T17:50:00.665 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.665 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.665 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:50:00.665 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.899144+0000", 2024-02-09T17:50:00.665 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.139565269 2024-02-09T17:50:00.665 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.665 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.665 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:50:00.665 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.899266+0000", 2024-02-09T17:50:00.665 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012115700000000001 2024-02-09T17:50:00.665 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.665 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.666 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,7,8", 2024-02-09T17:50:00.666 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.899409+0000", 2024-02-09T17:50:00.666 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014385999999999999 2024-02-09T17:50:00.666 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.666 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.666 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.666 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.914832+0000", 2024-02-09T17:50:00.666 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.015422531999999999 2024-02-09T17:50:00.666 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.666 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.666 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:50:00.666 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.918348+0000", 2024-02-09T17:50:00.666 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.003515822 2024-02-09T17:50:00.666 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.666 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.667 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.667 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.921402+0000", 2024-02-09T17:50:00.667 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0030539170000000002 2024-02-09T17:50:00.667 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.667 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.667 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.667 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.934632+0000", 2024-02-09T17:50:00.667 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.013230508 2024-02-09T17:50:00.667 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.667 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.667 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:50:00.667 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.934680+0000", 2024-02-09T17:50:00.667 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.8058e-05 2024-02-09T17:50:00.667 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.667 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.667 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:50:00.668 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.934776+0000", 2024-02-09T17:50:00.668 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.6067e-05 2024-02-09T17:50:00.668 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:00.668 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:00.668 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:00.668 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.668 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.668 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:266774 4.0 4:01d3b903:::benchmark_data_smithi179_103809_object129308:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e452)", 2024-02-09T17:50:00.668 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:40:19.784783+0000", 2024-02-09T17:50:00.668 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 580.77439232999996, 2024-02-09T17:50:00.668 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.150173686, 2024-02-09T17:50:00.668 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:50:00.668 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:00.668 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:50:00.669 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:50:00.669 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:50:00.669 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 266774 2024-02-09T17:50:00.669 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.669 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:50:00.669 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.669 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:50:00.669 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.784783+0000", 2024-02-09T17:50:00.669 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:50:00.669 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.669 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.669 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:50:00.669 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.784783+0000", 2024-02-09T17:50:00.669 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:50:00.669 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.670 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.670 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:50:00.670 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.784784+0000", 2024-02-09T17:50:00.670 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1480000000000001e-06 2024-02-09T17:50:00.670 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.670 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.670 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:50:00.670 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.784789+0000", 2024-02-09T17:50:00.670 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1139999999999997e-06 2024-02-09T17:50:00.670 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.670 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.670 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:50:00.670 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.784799+0000", 2024-02-09T17:50:00.670 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0662999999999999e-05 2024-02-09T17:50:00.670 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.671 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.671 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:50:00.671 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.784806+0000", 2024-02-09T17:50:00.671 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.5509999999999996e-06 2024-02-09T17:50:00.671 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.671 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.671 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:50:00.671 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.784865+0000", 2024-02-09T17:50:00.671 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.9222999999999999e-05 2024-02-09T17:50:00.671 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.671 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.671 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:50:00.671 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.784873+0000", 2024-02-09T17:50:00.671 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.1380000000000007e-06 2024-02-09T17:50:00.671 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.672 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.672 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:50:00.672 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.903402+0000", 2024-02-09T17:50:00.672 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1185289940000001 2024-02-09T17:50:00.672 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.672 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.672 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:50:00.672 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.903512+0000", 2024-02-09T17:50:00.672 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000109701 2024-02-09T17:50:00.672 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.672 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.672 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,7,8", 2024-02-09T17:50:00.673 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.903602+0000", 2024-02-09T17:50:00.673 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.9748999999999997e-05 2024-02-09T17:50:00.673 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.673 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.673 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:50:00.673 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.918353+0000", 2024-02-09T17:50:00.673 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.014751151000000001 2024-02-09T17:50:00.673 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.673 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.673 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.673 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.918427+0000", 2024-02-09T17:50:00.673 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.4368000000000004e-05 2024-02-09T17:50:00.673 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.673 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.673 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.673 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.921461+0000", 2024-02-09T17:50:00.674 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.003033745 2024-02-09T17:50:00.674 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.674 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.674 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.674 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.934813+0000", 2024-02-09T17:50:00.674 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.013352039 2024-02-09T17:50:00.674 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.674 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.674 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:50:00.674 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.934861+0000", 2024-02-09T17:50:00.674 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.7649999999999999e-05 2024-02-09T17:50:00.674 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.674 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.675 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:50:00.675 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.934956+0000", 2024-02-09T17:50:00.675 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.5451999999999993e-05 2024-02-09T17:50:00.675 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:00.675 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:00.675 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:00.675 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.675 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.675 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25201.0:266827 4.0 4:04e5bf2f:::benchmark_data_smithi179_103809_object129361:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e452)", 2024-02-09T17:50:00.675 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:40:19.928083+0000", 2024-02-09T17:50:00.675 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 580.63109135000002, 2024-02-09T17:50:00.675 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.007421744, 2024-02-09T17:50:00.675 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:50:00.675 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:00.675 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:50:00.675 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25201", 2024-02-09T17:50:00.676 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3746359631", 2024-02-09T17:50:00.676 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 266827 2024-02-09T17:50:00.676 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.676 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:50:00.676 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.676 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:50:00.676 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.928083+0000", 2024-02-09T17:50:00.676 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:50:00.676 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.676 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.676 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:50:00.676 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.928083+0000", 2024-02-09T17:50:00.676 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:50:00.676 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.677 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.677 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:50:00.677 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.928085+0000", 2024-02-09T17:50:00.677 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.6750000000000001e-06 2024-02-09T17:50:00.677 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.677 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.677 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:50:00.677 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.928091+0000", 2024-02-09T17:50:00.677 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.0170000000000002e-06 2024-02-09T17:50:00.677 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.677 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.677 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:50:00.678 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.928104+0000", 2024-02-09T17:50:00.678 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2798e-05 2024-02-09T17:50:00.678 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.678 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.678 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:50:00.678 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.928111+0000", 2024-02-09T17:50:00.678 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.2599999999999999e-06 2024-02-09T17:50:00.678 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.678 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.678 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:50:00.678 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.928173+0000", 2024-02-09T17:50:00.678 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.1481000000000003e-05 2024-02-09T17:50:00.678 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.679 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.679 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:50:00.679 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:19.928183+0000", 2024-02-09T17:50:00.679 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.006e-05 2024-02-09T17:50:00.679 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.679 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.679 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:50:00.679 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.906587+0000", 2024-02-09T17:50:00.679 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.97840395999999996 2024-02-09T17:50:00.679 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.679 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.679 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:50:00.679 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.906685+0000", 2024-02-09T17:50:00.679 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.8684000000000006e-05 2024-02-09T17:50:00.680 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.680 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.680 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,7,8", 2024-02-09T17:50:00.680 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.906772+0000", 2024-02-09T17:50:00.680 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.6808e-05 2024-02-09T17:50:00.680 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.680 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.680 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:50:00.680 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.918357+0000", 2024-02-09T17:50:00.680 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.011585201999999999 2024-02-09T17:50:00.680 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.680 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.680 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.680 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.918486+0000", 2024-02-09T17:50:00.680 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012893599999999999 2024-02-09T17:50:00.681 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.681 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.681 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.681 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.925167+0000", 2024-02-09T17:50:00.681 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.006680704 2024-02-09T17:50:00.681 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.681 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.681 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.681 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.935339+0000", 2024-02-09T17:50:00.681 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.010172015 2024-02-09T17:50:00.681 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.681 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.681 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:50:00.681 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.935403+0000", 2024-02-09T17:50:00.681 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.4136999999999999e-05 2024-02-09T17:50:00.681 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.682 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.682 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:50:00.682 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:40:20.935505+0000", 2024-02-09T17:50:00.682 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000102007 2024-02-09T17:50:00.682 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:00.682 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:00.682 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:00.682 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.682 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.682 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25837.0:95789 6.7 6:e710fbc8:::benchmark_data_smithi179_126206_object95788:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e600)", 2024-02-09T17:50:00.682 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:46:46.008692+0000", 2024-02-09T17:50:00.682 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 194.550483305, 2024-02-09T17:50:00.682 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.030868195, 2024-02-09T17:50:00.682 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:50:00.682 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:00.683 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:50:00.683 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25837", 2024-02-09T17:50:00.683 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:50:00.683 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 95789 2024-02-09T17:50:00.683 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.683 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:50:00.683 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.683 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:50:00.683 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.008692+0000", 2024-02-09T17:50:00.683 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:50:00.683 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.683 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.683 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:50:00.683 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.008692+0000", 2024-02-09T17:50:00.683 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:50:00.684 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.684 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.684 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:50:00.684 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.008693+0000", 2024-02-09T17:50:00.684 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.7489999999999999e-06 2024-02-09T17:50:00.684 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.684 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.684 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:50:00.684 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.008767+0000", 2024-02-09T17:50:00.684 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.4172999999999996e-05 2024-02-09T17:50:00.684 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.684 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.684 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:50:00.684 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.008781+0000", 2024-02-09T17:50:00.684 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3900000000000001e-05 2024-02-09T17:50:00.685 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.685 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.685 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:50:00.685 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.008792+0000", 2024-02-09T17:50:00.685 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0606e-05 2024-02-09T17:50:00.685 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.685 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.685 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:50:00.685 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.008857+0000", 2024-02-09T17:50:00.685 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.5377999999999996e-05 2024-02-09T17:50:00.685 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.685 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.685 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op must wait for map", 2024-02-09T17:50:00.685 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.008864+0000", 2024-02-09T17:50:00.685 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.6939999999999998e-06 2024-02-09T17:50:00.685 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.686 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.686 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:50:00.686 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.018780+0000", 2024-02-09T17:50:00.686 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0099160010000000007 2024-02-09T17:50:00.686 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.686 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.686 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:50:00.686 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.018789+0000", 2024-02-09T17:50:00.686 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.1370000000000008e-06 2024-02-09T17:50:00.686 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.686 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.686 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:50:00.686 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.023915+0000", 2024-02-09T17:50:00.686 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.005126108 2024-02-09T17:50:00.686 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.687 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.687 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:50:00.687 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.026409+0000", 2024-02-09T17:50:00.687 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.002493725 2024-02-09T17:50:00.687 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.687 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.687 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 4,5", 2024-02-09T17:50:00.687 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.026569+0000", 2024-02-09T17:50:00.687 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000159992 2024-02-09T17:50:00.687 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.687 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.687 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:50:00.687 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.039289+0000", 2024-02-09T17:50:00.687 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.012720189999999999 2024-02-09T17:50:00.687 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.688 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.688 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.688 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.039395+0000", 2024-02-09T17:50:00.688 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000106006 2024-02-09T17:50:00.688 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.688 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.688 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.688 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.039488+0000", 2024-02-09T17:50:00.688 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.2325999999999997e-05 2024-02-09T17:50:00.688 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.688 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.688 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:50:00.688 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.039524+0000", 2024-02-09T17:50:00.688 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.6658e-05 2024-02-09T17:50:00.688 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.689 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.689 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:50:00.689 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.039560+0000", 2024-02-09T17:50:00.689 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.5552000000000001e-05 2024-02-09T17:50:00.689 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:00.689 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:00.689 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:00.689 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.689 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.689 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25837.0:95797 6.7 6:e0774e6e:::benchmark_data_smithi179_126206_object95796:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e600)", 2024-02-09T17:50:00.689 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:46:46.019695+0000", 2024-02-09T17:50:00.689 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 194.53948028600001, 2024-02-09T17:50:00.689 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.020019789, 2024-02-09T17:50:00.689 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:50:00.689 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:00.690 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:50:00.690 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25837", 2024-02-09T17:50:00.690 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:50:00.690 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 95797 2024-02-09T17:50:00.690 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.690 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:50:00.690 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.690 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:50:00.690 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.019695+0000", 2024-02-09T17:50:00.690 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:50:00.690 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.690 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.690 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:50:00.690 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.019695+0000", 2024-02-09T17:50:00.690 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:50:00.691 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.691 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.691 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:50:00.691 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.019696+0000", 2024-02-09T17:50:00.691 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4419999999999999e-06 2024-02-09T17:50:00.691 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.691 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.691 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:50:00.691 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.019734+0000", 2024-02-09T17:50:00.691 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.8302e-05 2024-02-09T17:50:00.691 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.691 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.691 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:50:00.691 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.019748+0000", 2024-02-09T17:50:00.691 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3487e-05 2024-02-09T17:50:00.692 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.692 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.692 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:50:00.692 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.019756+0000", 2024-02-09T17:50:00.692 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.2109999999999998e-06 2024-02-09T17:50:00.692 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.692 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.692 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:50:00.692 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.019915+0000", 2024-02-09T17:50:00.692 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000159197 2024-02-09T17:50:00.692 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.692 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.692 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:50:00.692 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.019927+0000", 2024-02-09T17:50:00.692 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1566000000000001e-05 2024-02-09T17:50:00.693 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.693 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.693 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:50:00.693 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.030359+0000", 2024-02-09T17:50:00.693 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0104322539999999 2024-02-09T17:50:00.693 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.693 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.693 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:50:00.693 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.030559+0000", 2024-02-09T17:50:00.693 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00019977100000000001 2024-02-09T17:50:00.693 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.693 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.693 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 4,5", 2024-02-09T17:50:00.693 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.030698+0000", 2024-02-09T17:50:00.693 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013915299999999999 2024-02-09T17:50:00.694 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.694 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.694 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.694 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.039445+0000", 2024-02-09T17:50:00.694 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0087474330000000006 2024-02-09T17:50:00.694 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.694 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.694 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.694 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.039617+0000", 2024-02-09T17:50:00.694 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00017129499999999999 2024-02-09T17:50:00.694 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.694 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.694 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:50:00.694 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.039630+0000", 2024-02-09T17:50:00.694 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3573e-05 2024-02-09T17:50:00.694 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.695 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.695 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:50:00.695 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.039679+0000", 2024-02-09T17:50:00.695 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.8398e-05 2024-02-09T17:50:00.695 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.695 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.695 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:50:00.695 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.039714+0000", 2024-02-09T17:50:00.695 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.5707000000000002e-05 2024-02-09T17:50:00.695 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:00.695 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:00.695 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:00.695 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.695 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.695 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25837.0:95801 6.7 6:e618dc55:::benchmark_data_smithi179_126206_object95800:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e600)", 2024-02-09T17:50:00.696 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:46:46.027443+0000", 2024-02-09T17:50:00.696 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 194.53173210099999, 2024-02-09T17:50:00.696 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0151571669999999, 2024-02-09T17:50:00.696 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:50:00.696 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:00.696 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:50:00.696 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25837", 2024-02-09T17:50:00.696 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:50:00.696 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 95801 2024-02-09T17:50:00.696 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.696 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:50:00.696 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.696 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:50:00.696 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.027443+0000", 2024-02-09T17:50:00.696 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:50:00.697 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.697 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.697 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:50:00.697 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.027443+0000", 2024-02-09T17:50:00.697 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:50:00.697 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.697 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.697 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:50:00.697 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.027444+0000", 2024-02-09T17:50:00.697 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.53e-06 2024-02-09T17:50:00.697 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.697 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.697 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:50:00.697 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.027497+0000", 2024-02-09T17:50:00.697 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3223999999999997e-05 2024-02-09T17:50:00.698 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.698 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.698 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:50:00.698 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.027510+0000", 2024-02-09T17:50:00.698 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2767e-05 2024-02-09T17:50:00.698 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.698 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.698 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:50:00.698 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.027519+0000", 2024-02-09T17:50:00.698 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.2280000000000004e-06 2024-02-09T17:50:00.698 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.698 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.698 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:50:00.698 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.027586+0000", 2024-02-09T17:50:00.698 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.6222000000000004e-05 2024-02-09T17:50:00.698 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.699 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.699 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:50:00.699 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.027597+0000", 2024-02-09T17:50:00.699 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1466e-05 2024-02-09T17:50:00.699 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.699 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.699 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:50:00.699 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.034153+0000", 2024-02-09T17:50:00.699 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.00655592 2024-02-09T17:50:00.699 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.699 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.699 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:50:00.699 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.034261+0000", 2024-02-09T17:50:00.699 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000108043 2024-02-09T17:50:00.699 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.700 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.700 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 4,5", 2024-02-09T17:50:00.700 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.034359+0000", 2024-02-09T17:50:00.700 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.8159000000000007e-05 2024-02-09T17:50:00.700 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.700 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.700 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:50:00.700 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.039724+0000", 2024-02-09T17:50:00.700 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0053644640000000002 2024-02-09T17:50:00.700 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.700 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.700 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.700 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.042479+0000", 2024-02-09T17:50:00.700 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0027548569999999999 2024-02-09T17:50:00.700 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.701 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.701 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.701 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.042526+0000", 2024-02-09T17:50:00.701 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.7376999999999997e-05 2024-02-09T17:50:00.701 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.701 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.701 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:50:00.701 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.042562+0000", 2024-02-09T17:50:00.701 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.5589000000000001e-05 2024-02-09T17:50:00.701 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.701 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.701 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:50:00.701 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.042600+0000", 2024-02-09T17:50:00.701 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.8321000000000001e-05 2024-02-09T17:50:00.701 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:00.702 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:00.702 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:00.702 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.702 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.702 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25837.0:250068 6.b 6:d0990057:::benchmark_data_smithi179_126206_object119687:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:50:00.702 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:49:47.478512+0000", 2024-02-09T17:50:00.702 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 13.080662535, 2024-02-09T17:50:00.702 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.21750245900000001, 2024-02-09T17:50:00.702 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:50:00.702 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:00.702 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:50:00.702 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25837", 2024-02-09T17:50:00.702 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:50:00.702 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 250068 2024-02-09T17:50:00.702 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.703 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:50:00.703 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.703 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:50:00.703 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.478512+0000", 2024-02-09T17:50:00.703 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:50:00.703 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.703 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.703 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:50:00.703 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.478512+0000", 2024-02-09T17:50:00.703 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:50:00.703 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.703 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.703 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:50:00.703 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.478514+0000", 2024-02-09T17:50:00.704 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.663e-06 2024-02-09T17:50:00.704 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.704 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.704 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:50:00.704 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.478522+0000", 2024-02-09T17:50:00.704 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.7700000000000001e-06 2024-02-09T17:50:00.704 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.704 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.704 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:50:00.704 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.478536+0000", 2024-02-09T17:50:00.704 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4489e-05 2024-02-09T17:50:00.704 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.704 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.704 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:50:00.705 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.478547+0000", 2024-02-09T17:50:00.705 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0882e-05 2024-02-09T17:50:00.705 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.705 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.705 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:50:00.705 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.479985+0000", 2024-02-09T17:50:00.705 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0014383379999999999 2024-02-09T17:50:00.705 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.705 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.705 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:50:00.705 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.480092+0000", 2024-02-09T17:50:00.705 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00010629 2024-02-09T17:50:00.705 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.705 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.705 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,7", 2024-02-09T17:50:00.705 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.480199+0000", 2024-02-09T17:50:00.706 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000107755 2024-02-09T17:50:00.706 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.706 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.706 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:50:00.706 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.490461+0000", 2024-02-09T17:50:00.706 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.01026174 2024-02-09T17:50:00.706 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.706 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.706 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.706 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.499549+0000", 2024-02-09T17:50:00.706 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0090881069999999998 2024-02-09T17:50:00.706 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.706 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.706 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.707 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.695912+0000", 2024-02-09T17:50:00.707 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.196362389 2024-02-09T17:50:00.707 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.707 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.707 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:50:00.707 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.695961+0000", 2024-02-09T17:50:00.707 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.9200999999999999e-05 2024-02-09T17:50:00.707 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.707 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.707 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:50:00.707 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.696015+0000", 2024-02-09T17:50:00.707 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3835000000000002e-05 2024-02-09T17:50:00.707 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:00.707 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:00.707 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:00.708 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.708 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.708 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25837.0:250084 6.b 6:d1697981:::benchmark_data_smithi179_126206_object119703:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:50:00.708 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:49:47.494317+0000", 2024-02-09T17:50:00.708 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 13.064858101, 2024-02-09T17:50:00.708 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.214235552, 2024-02-09T17:50:00.708 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:50:00.708 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:00.708 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:50:00.708 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25837", 2024-02-09T17:50:00.708 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:50:00.708 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 250084 2024-02-09T17:50:00.708 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.708 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:50:00.708 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.709 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:50:00.709 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.494317+0000", 2024-02-09T17:50:00.709 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:50:00.709 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.709 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.709 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:50:00.709 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.494317+0000", 2024-02-09T17:50:00.709 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:50:00.709 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.709 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.709 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:50:00.709 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.494318+0000", 2024-02-09T17:50:00.709 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.02e-07 2024-02-09T17:50:00.709 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.709 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.710 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:50:00.710 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.494321+0000", 2024-02-09T17:50:00.710 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.4369999999999999e-06 2024-02-09T17:50:00.710 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.710 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.710 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:50:00.710 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.494329+0000", 2024-02-09T17:50:00.710 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.7489999999999998e-06 2024-02-09T17:50:00.710 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.710 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.710 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:50:00.710 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.494334+0000", 2024-02-09T17:50:00.710 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.7450000000000003e-06 2024-02-09T17:50:00.710 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.710 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.711 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:50:00.711 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.494371+0000", 2024-02-09T17:50:00.711 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.7307e-05 2024-02-09T17:50:00.711 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.711 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.711 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:50:00.711 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.494473+0000", 2024-02-09T17:50:00.711 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000101636 2024-02-09T17:50:00.711 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.711 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.711 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,7", 2024-02-09T17:50:00.711 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.494558+0000", 2024-02-09T17:50:00.711 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.5359999999999996e-05 2024-02-09T17:50:00.711 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.711 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.711 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:50:00.712 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.499609+0000", 2024-02-09T17:50:00.712 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00505085 2024-02-09T17:50:00.712 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.712 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.712 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.712 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.511769+0000", 2024-02-09T17:50:00.712 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.012160064999999999 2024-02-09T17:50:00.712 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.712 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.712 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:00.712 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.708448+0000", 2024-02-09T17:50:00.712 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.19667905599999999 2024-02-09T17:50:00.712 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.712 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.712 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:50:00.713 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.708498+0000", 2024-02-09T17:50:00.713 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.0552e-05 2024-02-09T17:50:00.713 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:00.713 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:00.713 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:50:00.713 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.708552+0000", 2024-02-09T17:50:00.713 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3893000000000001e-05 2024-02-09T17:50:00.713 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:00.713 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:00.713 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:00.713 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:00.713 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:00.713 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:50:00.715 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Removed 130378 objects 2024-02-09T17:50:00.715 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Clean up completed and total clean up time :158.524 2024-02-09T17:50:00.811 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:50:00.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:00.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:50:00.914 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:50:00.944 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:01.046 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is set 2024-02-09T17:50:01.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:01 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/806017597' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:50:01.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:01 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/806017597' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:50:01.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:01 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/806017597' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:50:02.139 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-02-09T17:50:02.140 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-02-09T17:50:02.294 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:50:02.296 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:02 smithi099 ceph-mon[39420]: pgmap v1651: 47 pgs: 47 active+clean; 141 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 743 op/s 2024-02-09T17:50:02.296 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:02 smithi099 ceph-mon[39420]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:50:02.296 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:02 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/806017597' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:50:02.296 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:02 smithi099 ceph-mon[39420]: osdmap e654: 12 total, 12 up, 12 in 2024-02-09T17:50:02.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:02.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:50:02.398 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:50:02.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:02 smithi179 ceph-mon[43975]: pgmap v1651: 47 pgs: 47 active+clean; 141 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 743 op/s 2024-02-09T17:50:02.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:02 smithi179 ceph-mon[43975]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:50:02.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:02 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/806017597' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:50:02.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:02 smithi179 ceph-mon[43975]: osdmap e654: 12 total, 12 up, 12 in 2024-02-09T17:50:02.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:02 smithi162 ceph-mon[44304]: pgmap v1651: 47 pgs: 47 active+clean; 141 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 743 op/s 2024-02-09T17:50:02.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:02 smithi162 ceph-mon[44304]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:50:02.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:02 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/806017597' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:50:02.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:02 smithi162 ceph-mon[44304]: osdmap e654: 12 total, 12 up, 12 in 2024-02-09T17:50:02.431 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:02.452 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:50:02.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:02.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:50:02.556 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:50:02.584 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:02.611 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:50:02.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:02.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:50:02.715 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:50:02.747 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:02.768 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:50:02.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:02.772 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:50:02.873 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:50:02.903 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:02.923 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:50:02.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:02.927 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:50:03.028 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:50:03.057 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:03.076 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:50:03.079 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:03.080 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:50:03.181 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:50:03.207 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:03.907 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-09T17:50:04.184 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:50:04.216 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:04 smithi099 ceph-mon[39420]: pgmap v1653: 47 pgs: 47 active+clean; 13 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 712 op/s 2024-02-09T17:50:04.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:04 smithi179 ceph-mon[43975]: pgmap v1653: 47 pgs: 47 active+clean; 13 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 712 op/s 2024-02-09T17:50:04.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:04 smithi162 ceph-mon[44304]: pgmap v1653: 47 pgs: 47 active+clean; 13 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 712 op/s 2024-02-09T17:50:05.258 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:50:05.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:05.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:50:05.361 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:50:05.392 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:05.741 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:50:05.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:05.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:50:05.800 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:50:05.845 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:50:05.874 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:06.022 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:50:06.025 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:06.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:50:06.054 INFO:teuthology.orchestra.run.smithi099.stderr:set osd.3 primary-affinity to 0.801503 (8525272) 2024-02-09T17:50:06.126 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:50:06.151 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:06 smithi099 ceph-mon[39420]: pgmap v1654: 47 pgs: 47 active+clean; 961 KiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 649 op/s 2024-02-09T17:50:06.152 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:06 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2115361428' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.8015031811988723}]: dispatch 2024-02-09T17:50:06.154 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:06.240 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset noscrub 2024-02-09T17:50:06.277 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:50:06.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:06.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:50:06.383 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:50:06.413 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:06 smithi179 ceph-mon[43975]: pgmap v1654: 47 pgs: 47 active+clean; 961 KiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 649 op/s 2024-02-09T17:50:06.413 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:06 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2115361428' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.8015031811988723}]: dispatch 2024-02-09T17:50:06.415 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:06.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:06 smithi162 ceph-mon[44304]: pgmap v1654: 47 pgs: 47 active+clean; 961 KiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 649 op/s 2024-02-09T17:50:06.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:06 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2115361428' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.8015031811988723}]: dispatch 2024-02-09T17:50:06.436 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:50:06.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:06.440 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:50:06.541 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:50:06.571 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:06.579 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:50:06.579 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-09T17:50:06.579 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-02-09T17:50:06.580 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:50:06.591 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:50:06.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:06.595 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:50:06.696 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:50:06.723 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:06.797 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:50:06.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:06.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:50:06.901 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:50:06.930 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:07.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:07 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2115361428' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.8015031811988723}]': finished 2024-02-09T17:50:07.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:07 smithi099 ceph-mon[39420]: osdmap e655: 12 total, 12 up, 12 in 2024-02-09T17:50:07.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:07 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/757546446' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:50:07.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:07 smithi099 ceph-mon[39420]: osdmap e656: 12 total, 12 up, 12 in 2024-02-09T17:50:07.255 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:07 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2115361428' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.8015031811988723}]': finished 2024-02-09T17:50:07.255 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:07 smithi162 ceph-mon[44304]: osdmap e655: 12 total, 12 up, 12 in 2024-02-09T17:50:07.255 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:07 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/757546446' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:50:07.255 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:07 smithi162 ceph-mon[44304]: osdmap e656: 12 total, 12 up, 12 in 2024-02-09T17:50:07.280 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-09T17:50:07.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:07 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2115361428' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.8015031811988723}]': finished 2024-02-09T17:50:07.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:07 smithi179 ceph-mon[43975]: osdmap e655: 12 total, 12 up, 12 in 2024-02-09T17:50:07.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:07 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/757546446' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:50:07.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:07 smithi179 ceph-mon[43975]: osdmap e656: 12 total, 12 up, 12 in 2024-02-09T17:50:07.540 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:50:08.025 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:50:08.025 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:50:08.025 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:50:08.025 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:50:08.025 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:50:08.025 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:50:08.026 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:50:08.026 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:50:08.026 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:50:08.026 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:50:08.026 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:50:08.026 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:50:08.026 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:50:08.026 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:50:08.026 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:50:08.026 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:50:08.026 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:50:08.026 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:50:08.027 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:50:08.027 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:50:08.027 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:50:08.027 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:50:08.027 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:50:08.027 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:50:08.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:08 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:50:08.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:08 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:50:08.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:08 smithi099 ceph-mon[39420]: pgmap v1656: 47 pgs: 2 peering, 45 active+clean; 577 KiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 375 op/s 2024-02-09T17:50:08.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:08 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:50:08.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:08 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:50:08.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:08 smithi179 ceph-mon[43975]: pgmap v1656: 47 pgs: 2 peering, 45 active+clean; 577 KiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 375 op/s 2024-02-09T17:50:08.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:08 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:50:08.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:08 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:50:08.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:08 smithi162 ceph-mon[44304]: pgmap v1656: 47 pgs: 2 peering, 45 active+clean; 577 KiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 375 op/s 2024-02-09T17:50:09.210 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:50:09.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:09.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:50:09.224 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:50:09.286 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:50:09.286 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:50:09.314 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:50:09.343 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:09.466 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:50:09.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:09.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:50:09.570 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:50:09.603 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:09.624 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:50:09.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:09.627 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:50:09.728 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:50:09.756 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:10.004 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:50:10.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:10.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:50:10.008 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-09T17:50:10.077 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'unique_pool_4' removed 2024-02-09T17:50:10.107 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:50:10.134 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:10.153 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:50:10.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:10.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:50:10.257 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:50:10.283 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:50:10.283 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:10 smithi099 ceph-mon[39420]: pgmap v1658: 47 pgs: 2 peering, 45 active+clean; 577 KiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 281 op/s 2024-02-09T17:50:10.283 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:10 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/196810663' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-09T17:50:10.285 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:10.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:10 smithi179 ceph-mon[43975]: pgmap v1658: 47 pgs: 2 peering, 45 active+clean; 577 KiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 281 op/s 2024-02-09T17:50:10.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:10 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/196810663' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-09T17:50:10.431 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:10 smithi162 ceph-mon[44304]: pgmap v1658: 47 pgs: 2 peering, 45 active+clean; 577 KiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 281 op/s 2024-02-09T17:50:10.432 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:10 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/196810663' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-09T17:50:10.432 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:50:10.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:10.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:50:10.537 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:50:10.563 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:10.583 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:50:10.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:10.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:50:10.662 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:50:10.689 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:50:10.715 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:10.768 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:50:10.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:10.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:50:10.872 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:50:10.899 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:11.148 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:50:11.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:11.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:50:11.252 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:50:11.284 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:11.305 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:50:11.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:11.308 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:50:11.409 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:50:11.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:11 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/196810663' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-02-09T17:50:11.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:11 smithi179 ceph-mon[43975]: osdmap e657: 12 total, 12 up, 12 in 2024-02-09T17:50:11.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:11 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/196810663' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-02-09T17:50:11.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:11 smithi162 ceph-mon[44304]: osdmap e657: 12 total, 12 up, 12 in 2024-02-09T17:50:11.436 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:11.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:11 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/196810663' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-02-09T17:50:11.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:11 smithi099 ceph-mon[39420]: osdmap e657: 12 total, 12 up, 12 in 2024-02-09T17:50:11.635 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:50:11.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:11.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:50:11.691 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-02-09T17:50:11.692 INFO:tasks.radosbench:Beginning radosbench... 2024-02-09T17:50:11.692 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_5 2024-02-09T17:50:11.692 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd pool create unique_pool_5 16 2024-02-09T17:50:11.739 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:50:11.765 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:11.966 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:50:11.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:11.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:50:12.071 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:50:12.100 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:12.120 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:50:12.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:12.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:50:12.147 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:50:12.147 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:50:12.169 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:12 smithi162 ceph-mon[44304]: pgmap v1660: 31 pgs: 31 active+clean; 577 KiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:50:12.219 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:50:12.225 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:50:12.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:12 smithi099 ceph-mon[39420]: pgmap v1660: 31 pgs: 31 active+clean; 577 KiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:50:12.253 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:50:12.257 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:12.278 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:50:12.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:12.281 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:50:12.382 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:50:12.409 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:12.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:12 smithi179 ceph-mon[43975]: pgmap v1660: 31 pgs: 31 active+clean; 577 KiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:50:12.438 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:50:12.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:12.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:50:12.542 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:50:12.571 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:12.937 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] 2024-02-09T17:50:12.937 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:50:12.937 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-02-09T17:50:12.937 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-09T17:50:12.937 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-09T17:50:12.937 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-02-09T17:50:12.944 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-09T17:50:13.077 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:13 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3396836799' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:50:13.078 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is unset 2024-02-09T17:50:13.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:13 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3396836799' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:50:13.215 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:50:13.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:13 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3396836799' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:50:14.220 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:50:14.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:14.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:50:14.331 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:50:14.357 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:14 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:50:14.357 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:14 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:50:14.357 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:14 smithi162 ceph-mon[44304]: pgmap v1661: 31 pgs: 31 active+clean; 577 KiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:50:14.357 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:14 smithi162 ceph-mon[44304]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:50:14.358 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:14 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3396836799' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:50:14.358 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:14 smithi162 ceph-mon[44304]: osdmap e658: 12 total, 12 up, 12 in 2024-02-09T17:50:14.358 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:14 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:50:14.358 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:14 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:50:14.358 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:14 smithi099 ceph-mon[39420]: pgmap v1661: 31 pgs: 31 active+clean; 577 KiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:50:14.358 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:14 smithi099 ceph-mon[39420]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:50:14.358 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:14 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3396836799' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:50:14.358 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:14 smithi099 ceph-mon[39420]: osdmap e658: 12 total, 12 up, 12 in 2024-02-09T17:50:14.360 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:14.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:14 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:50:14.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:14 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:50:14.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:14 smithi179 ceph-mon[43975]: pgmap v1661: 31 pgs: 31 active+clean; 577 KiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:50:14.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:14 smithi179 ceph-mon[43975]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:50:14.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:14 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3396836799' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:50:14.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:14 smithi179 ceph-mon[43975]: osdmap e658: 12 total, 12 up, 12 in 2024-02-09T17:50:14.963 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:50:14.964 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:50:15.008 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:50:15.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:15.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:50:15.113 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:50:15.141 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:15.161 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:50:15.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:15.164 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:50:15.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:15 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:50:15.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:15 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:50:15.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:15 smithi099 ceph-mon[39420]: osdmap e659: 12 total, 12 up, 12 in 2024-02-09T17:50:15.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:15 smithi099 ceph-mon[39420]: 1.1 scrub starts 2024-02-09T17:50:15.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:15 smithi099 ceph-mon[39420]: 1.1 scrub ok 2024-02-09T17:50:15.259 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:50:15.265 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:50:15.277 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:50:15.293 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:15 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:50:15.293 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:15 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:50:15.293 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:15 smithi162 ceph-mon[44304]: osdmap e659: 12 total, 12 up, 12 in 2024-02-09T17:50:15.293 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:15 smithi162 ceph-mon[44304]: 1.1 scrub starts 2024-02-09T17:50:15.293 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:15 smithi162 ceph-mon[44304]: 1.1 scrub ok 2024-02-09T17:50:15.296 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:15.296 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:50:15.316 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:50:15.335 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:50:15.355 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:50:15.375 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:50:15.395 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:50:15.415 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:50:15.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:15 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:50:15.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:15 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:50:15.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:15 smithi179 ceph-mon[43975]: osdmap e659: 12 total, 12 up, 12 in 2024-02-09T17:50:15.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:15 smithi179 ceph-mon[43975]: 1.1 scrub starts 2024-02-09T17:50:15.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:15 smithi179 ceph-mon[43975]: 1.1 scrub ok 2024-02-09T17:50:15.434 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:50:15.454 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:50:15.473 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:50:15.474 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:50:15.474 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:50:15.474 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:50:15.475 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:50:15.475 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:50:15.475 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:50:15.475 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:50:15.475 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:50:15.475 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:50:15.475 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:50:15.475 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:50:15.475 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:50:15.715 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-09T17:50:15.746 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:50:15.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:15.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:50:15.814 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-09T17:50:15.850 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:50:15.877 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:15.895 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:50:15.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:15.898 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:50:15.999 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:50:16.003 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:50:16.028 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:16.092 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:16 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:50:16.092 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:16 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:50:16.134 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:50:16.151 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:50:16.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:16.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:50:16.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:16 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:50:16.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:16 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:50:16.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:16 smithi179 ceph-mon[43975]: 1.9 scrub starts 2024-02-09T17:50:16.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:16 smithi179 ceph-mon[43975]: pgmap v1664: 31 pgs: 31 active+clean; 577 KiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:50:16.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:16 smithi179 ceph-mon[43975]: 1.9 scrub ok 2024-02-09T17:50:16.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:16 smithi179 ceph-mon[43975]: osdmap e660: 12 total, 12 up, 12 in 2024-02-09T17:50:16.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:16 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4191082419' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:50:16.256 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:50:16.283 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:16.302 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:50:16.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:16.305 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:50:16.350 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:16 smithi099 ceph-mon[39420]: 1.9 scrub starts 2024-02-09T17:50:16.350 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:16 smithi099 ceph-mon[39420]: pgmap v1664: 31 pgs: 31 active+clean; 577 KiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:50:16.350 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:16 smithi099 ceph-mon[39420]: 1.9 scrub ok 2024-02-09T17:50:16.350 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:16 smithi099 ceph-mon[39420]: osdmap e660: 12 total, 12 up, 12 in 2024-02-09T17:50:16.350 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:16 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4191082419' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:50:16.405 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:50:16.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:16 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:50:16.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:16 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:50:16.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:16 smithi162 ceph-mon[44304]: 1.9 scrub starts 2024-02-09T17:50:16.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:16 smithi162 ceph-mon[44304]: pgmap v1664: 31 pgs: 31 active+clean; 577 KiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:50:16.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:16 smithi162 ceph-mon[44304]: 1.9 scrub ok 2024-02-09T17:50:16.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:16 smithi162 ceph-mon[44304]: osdmap e660: 12 total, 12 up, 12 in 2024-02-09T17:50:16.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:16 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4191082419' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:50:16.434 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:16.453 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:50:16.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:16.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:50:16.558 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:50:16.586 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:16.605 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:50:16.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:16.608 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:50:16.709 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:50:16.738 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:16.863 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:50:16.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:16.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:50:16.967 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:50:17.000 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:17.020 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:50:17.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:17.023 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:50:17.124 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:50:17.152 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:17.173 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:50:17.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:17.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:50:17.277 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:50:17.311 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:17.331 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:50:17.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:17.335 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:50:17.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:17 smithi162 ceph-mon[44304]: 1.1e scrub starts 2024-02-09T17:50:17.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:17 smithi162 ceph-mon[44304]: 1.1e scrub ok 2024-02-09T17:50:17.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:17 smithi162 ceph-mon[44304]: 1.f scrub starts 2024-02-09T17:50:17.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:17 smithi162 ceph-mon[44304]: 1.8 scrub starts 2024-02-09T17:50:17.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:17 smithi162 ceph-mon[44304]: 1.8 scrub ok 2024-02-09T17:50:17.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:17 smithi162 ceph-mon[44304]: 1.f scrub ok 2024-02-09T17:50:17.435 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:50:17.466 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:17.485 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:50:17.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:17.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:50:17.589 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:50:17.618 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:17.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:17 smithi179 ceph-mon[43975]: 1.1e scrub starts 2024-02-09T17:50:17.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:17 smithi179 ceph-mon[43975]: 1.1e scrub ok 2024-02-09T17:50:17.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:17 smithi179 ceph-mon[43975]: 1.f scrub starts 2024-02-09T17:50:17.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:17 smithi179 ceph-mon[43975]: 1.8 scrub starts 2024-02-09T17:50:17.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:17 smithi179 ceph-mon[43975]: 1.8 scrub ok 2024-02-09T17:50:17.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:17 smithi179 ceph-mon[43975]: 1.f scrub ok 2024-02-09T17:50:17.720 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:50:17.721 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:50:17.721 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-09T17:50:17.721 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-02-09T17:50:17.721 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-02-09T17:50:17.721 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:50:17.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:17.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:50:17.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:17 smithi099 ceph-mon[39420]: 1.1e scrub starts 2024-02-09T17:50:17.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:17 smithi099 ceph-mon[39420]: 1.1e scrub ok 2024-02-09T17:50:17.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:17 smithi099 ceph-mon[39420]: 1.f scrub starts 2024-02-09T17:50:17.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:17 smithi099 ceph-mon[39420]: 1.8 scrub starts 2024-02-09T17:50:17.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:17 smithi099 ceph-mon[39420]: 1.8 scrub ok 2024-02-09T17:50:17.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:17 smithi099 ceph-mon[39420]: 1.f scrub ok 2024-02-09T17:50:17.744 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'unique_pool_5' created 2024-02-09T17:50:17.824 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:50:17.852 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:18.501 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:50:18.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:18.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:50:18.606 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:50:18.635 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:18 smithi099 ceph-mon[39420]: 1.e scrub starts 2024-02-09T17:50:18.636 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:18 smithi099 ceph-mon[39420]: 1.e scrub ok 2024-02-09T17:50:18.636 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:18 smithi099 ceph-mon[39420]: 1.15 scrub starts 2024-02-09T17:50:18.636 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:18 smithi099 ceph-mon[39420]: pgmap v1666: 31 pgs: 31 active+clean; 577 KiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:50:18.636 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:18 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2873802956' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]: dispatch 2024-02-09T17:50:18.636 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:18 smithi099 ceph-mon[39420]: 1.15 scrub ok 2024-02-09T17:50:18.636 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:18 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:50:18.636 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:18 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:50:18.636 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:18 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2873802956' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-02-09T17:50:18.636 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:18 smithi099 ceph-mon[39420]: osdmap e661: 12 total, 12 up, 12 in 2024-02-09T17:50:18.638 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:18.657 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:50:18.657 INFO:teuthology.orchestra.run.smithi099.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-02-09T17:50:18.657 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:50:18.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:18 smithi179 ceph-mon[43975]: 1.e scrub starts 2024-02-09T17:50:18.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:18 smithi179 ceph-mon[43975]: 1.e scrub ok 2024-02-09T17:50:18.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:18 smithi179 ceph-mon[43975]: 1.15 scrub starts 2024-02-09T17:50:18.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:18 smithi179 ceph-mon[43975]: pgmap v1666: 31 pgs: 31 active+clean; 577 KiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:50:18.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:18 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2873802956' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]: dispatch 2024-02-09T17:50:18.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:18 smithi179 ceph-mon[43975]: 1.15 scrub ok 2024-02-09T17:50:18.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:18 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:50:18.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:18 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:50:18.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:18 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2873802956' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-02-09T17:50:18.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:18 smithi179 ceph-mon[43975]: osdmap e661: 12 total, 12 up, 12 in 2024-02-09T17:50:18.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:18 smithi162 ceph-mon[44304]: 1.e scrub starts 2024-02-09T17:50:18.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:18 smithi162 ceph-mon[44304]: 1.e scrub ok 2024-02-09T17:50:18.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:18 smithi162 ceph-mon[44304]: 1.15 scrub starts 2024-02-09T17:50:18.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:18 smithi162 ceph-mon[44304]: pgmap v1666: 31 pgs: 31 active+clean; 577 KiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:50:18.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:18 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2873802956' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]: dispatch 2024-02-09T17:50:18.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:18 smithi162 ceph-mon[44304]: 1.15 scrub ok 2024-02-09T17:50:18.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:18 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:50:18.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:18 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:50:18.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:18 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2873802956' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-02-09T17:50:18.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:18 smithi162 ceph-mon[44304]: osdmap e661: 12 total, 12 up, 12 in 2024-02-09T17:50:19.189 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:50:19.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:19.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:50:19.294 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:50:19.322 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:19.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:19 smithi099 ceph-mon[39420]: 1.d scrub starts 2024-02-09T17:50:19.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:19 smithi099 ceph-mon[39420]: 1.d scrub ok 2024-02-09T17:50:19.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:19 smithi099 ceph-mon[39420]: osdmap e662: 12 total, 12 up, 12 in 2024-02-09T17:50:19.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:19 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:50:19.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:19 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:50:19.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:19 smithi179 ceph-mon[43975]: 1.d scrub starts 2024-02-09T17:50:19.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:19 smithi179 ceph-mon[43975]: 1.d scrub ok 2024-02-09T17:50:19.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:19 smithi179 ceph-mon[43975]: osdmap e662: 12 total, 12 up, 12 in 2024-02-09T17:50:19.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:19 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:50:19.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:19 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:50:19.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:19 smithi162 ceph-mon[44304]: 1.d scrub starts 2024-02-09T17:50:19.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:19 smithi162 ceph-mon[44304]: 1.d scrub ok 2024-02-09T17:50:19.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:19 smithi162 ceph-mon[44304]: osdmap e662: 12 total, 12 up, 12 in 2024-02-09T17:50:19.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:19 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:50:19.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:19 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:50:19.772 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:50:19.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:19.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:50:19.876 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:50:19.904 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:20.119 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:50:20.150 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd pool application enable unique_pool_5 rados --yes-i-really-mean-it || true 2024-02-09T17:50:20.151 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:50:20.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:20.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:50:20.255 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:50:20.281 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:20.319 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:50:20.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:20.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:50:20.380 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:20 smithi099 ceph-mon[39420]: pgmap v1669: 47 pgs: 4 creating+peering, 12 unknown, 31 active+clean; 577 KiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:50:20.380 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:20 smithi099 ceph-mon[39420]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-09T17:50:20.380 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:20 smithi099 ceph-mon[39420]: osdmap e663: 12 total, 12 up, 12 in 2024-02-09T17:50:20.423 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:50:20.451 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:20.469 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-09T17:50:20.549 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:50:20.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:20.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:50:20.653 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:50:20.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:20 smithi179 ceph-mon[43975]: pgmap v1669: 47 pgs: 4 creating+peering, 12 unknown, 31 active+clean; 577 KiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:50:20.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:20 smithi179 ceph-mon[43975]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-09T17:50:20.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:20 smithi179 ceph-mon[43975]: osdmap e663: 12 total, 12 up, 12 in 2024-02-09T17:50:20.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:20 smithi162 ceph-mon[44304]: pgmap v1669: 47 pgs: 4 creating+peering, 12 unknown, 31 active+clean; 577 KiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:50:20.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:20 smithi162 ceph-mon[44304]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-09T17:50:20.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:20 smithi162 ceph-mon[44304]: osdmap e663: 12 total, 12 up, 12 in 2024-02-09T17:50:20.680 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:20.737 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:50:20.874 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:50:20.876 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:50:20.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:20.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:50:20.951 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:50:20.981 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:50:21.010 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:21.032 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:50:21.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:21.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:50:21.136 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:50:21.164 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:21.217 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:50:21.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:21.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:50:21.323 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:50:21.353 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:21.639 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:21 smithi162 ceph-mon[44304]: 1.0 scrub starts 2024-02-09T17:50:21.639 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:21 smithi162 ceph-mon[44304]: 1.0 scrub ok 2024-02-09T17:50:21.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:21 smithi179 ceph-mon[43975]: 1.0 scrub starts 2024-02-09T17:50:21.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:21 smithi179 ceph-mon[43975]: 1.0 scrub ok 2024-02-09T17:50:21.685 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:21 smithi099 ceph-mon[39420]: 1.0 scrub starts 2024-02-09T17:50:21.685 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:21 smithi099 ceph-mon[39420]: 1.0 scrub ok 2024-02-09T17:50:21.701 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:50:21.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:21.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:50:21.808 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:50:21.835 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:21.854 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:50:21.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:21.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:50:21.958 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:50:21.991 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:22.012 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:50:22.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:22.015 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:50:22.117 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:50:22.146 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:22.166 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:50:22.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:22.169 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:50:22.270 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:50:22.298 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:22.525 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:50:22.525 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-02-09T17:50:22.525 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-02-09T17:50:22.526 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-02-09T17:50:22.526 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.526 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:33098 5.f 5:fb2113f5:::benchmark_data_smithi179_115199_object33097:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e479)", 2024-02-09T17:50:22.526 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:41:19.860866+0000", 2024-02-09T17:50:22.526 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 542.65986948900002, 2024-02-09T17:50:22.526 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.017314074, 2024-02-09T17:50:22.526 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:50:22.526 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-09T17:50:22.526 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:50:22.526 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:50:22.526 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:50:22.527 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 33098 2024-02-09T17:50:22.527 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.527 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:50:22.527 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.527 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:50:22.527 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.860866+0000", 2024-02-09T17:50:22.527 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:22.527 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.527 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.527 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:50:22.527 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.860866+0000", 2024-02-09T17:50:22.527 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:22.528 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.528 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.528 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:50:22.528 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.860867+0000", 2024-02-09T17:50:22.528 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1969999999999999e-06 2024-02-09T17:50:22.528 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.528 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.528 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:50:22.528 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.862146+0000", 2024-02-09T17:50:22.528 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0012785520000000001 2024-02-09T17:50:22.528 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:50:22.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.862157+0000", 2024-02-09T17:50:22.529 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1525999999999999e-05 2024-02-09T17:50:22.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:50:22.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.862166+0000", 2024-02-09T17:50:22.529 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.918e-06 2024-02-09T17:50:22.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:22.530 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.862674+0000", 2024-02-09T17:50:22.530 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00050811300000000003 2024-02-09T17:50:22.530 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.530 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.530 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:50:22.530 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.862804+0000", 2024-02-09T17:50:22.530 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012947699999999999 2024-02-09T17:50:22.530 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.530 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.530 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,11", 2024-02-09T17:50:22.530 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.862947+0000", 2024-02-09T17:50:22.530 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000143378 2024-02-09T17:50:22.531 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.531 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.531 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:50:22.531 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.864397+0000", 2024-02-09T17:50:22.531 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014498410000000001 2024-02-09T17:50:22.531 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.531 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.531 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:22.531 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.867884+0000", 2024-02-09T17:50:22.531 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034867850000000001 2024-02-09T17:50:22.531 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.531 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.531 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:50:22.532 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.867896+0000", 2024-02-09T17:50:22.532 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2411e-05 2024-02-09T17:50:22.532 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.532 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.532 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:22.532 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.878116+0000", 2024-02-09T17:50:22.532 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.010220026 2024-02-09T17:50:22.532 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.532 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.532 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:50:22.532 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.878180+0000", 2024-02-09T17:50:22.533 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3849999999999993e-05 2024-02-09T17:50:22.533 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:22.533 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:50:22.533 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:22.533 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.533 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.533 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:33125 5.f 5:f3e8f580:::benchmark_data_smithi179_115199_object33124:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e480)", 2024-02-09T17:50:22.533 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:41:19.882980+0000", 2024-02-09T17:50:22.533 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 542.63775525799997, 2024-02-09T17:50:22.533 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99991403599999995, 2024-02-09T17:50:22.533 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:50:22.533 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:22.534 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:50:22.534 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:50:22.534 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:50:22.534 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 33125 2024-02-09T17:50:22.534 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.534 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:50:22.534 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.534 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:50:22.534 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.882980+0000", 2024-02-09T17:50:22.534 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:22.534 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.534 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.535 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:50:22.535 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.882980+0000", 2024-02-09T17:50:22.535 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:22.535 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.535 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.535 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:50:22.535 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.882982+0000", 2024-02-09T17:50:22.535 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.731e-06 2024-02-09T17:50:22.535 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.535 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.535 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:50:22.535 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.883025+0000", 2024-02-09T17:50:22.536 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2828999999999999e-05 2024-02-09T17:50:22.536 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.536 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.536 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:50:22.536 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.883037+0000", 2024-02-09T17:50:22.536 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2291e-05 2024-02-09T17:50:22.536 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.536 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.536 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:50:22.536 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.883045+0000", 2024-02-09T17:50:22.536 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8499999999999994e-06 2024-02-09T17:50:22.537 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.537 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.537 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:22.537 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.883111+0000", 2024-02-09T17:50:22.537 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.656e-05 2024-02-09T17:50:22.537 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.537 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.537 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:50:22.537 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.883122+0000", 2024-02-09T17:50:22.537 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0838e-05 2024-02-09T17:50:22.537 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.537 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.537 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:22.538 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.878634+0000", 2024-02-09T17:50:22.538 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99551221000000001 2024-02-09T17:50:22.538 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.538 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.538 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:50:22.538 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.878805+0000", 2024-02-09T17:50:22.538 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017077999999999999 2024-02-09T17:50:22.538 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.538 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.538 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,11", 2024-02-09T17:50:22.538 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.878989+0000", 2024-02-09T17:50:22.538 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000183893 2024-02-09T17:50:22.538 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.539 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.539 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:22.539 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.882644+0000", 2024-02-09T17:50:22.539 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0036547139999999999 2024-02-09T17:50:22.539 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.539 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.539 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:22.539 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.882718+0000", 2024-02-09T17:50:22.539 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4070000000000001e-05 2024-02-09T17:50:22.539 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.539 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.539 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:50:22.540 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.882813+0000", 2024-02-09T17:50:22.540 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5030000000000003e-05 2024-02-09T17:50:22.540 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.540 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.540 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:50:22.540 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.882853+0000", 2024-02-09T17:50:22.540 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9771000000000003e-05 2024-02-09T17:50:22.540 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.540 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.540 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:50:22.540 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.882894+0000", 2024-02-09T17:50:22.540 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1468999999999999e-05 2024-02-09T17:50:22.541 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:22.541 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:50:22.541 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:22.541 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.541 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.541 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:33145 5.f 5:f8644e66:::benchmark_data_smithi179_115199_object33144:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e480)", 2024-02-09T17:50:22.541 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:41:19.884388+0000", 2024-02-09T17:50:22.541 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 542.63634692999995, 2024-02-09T17:50:22.541 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99867693599999996, 2024-02-09T17:50:22.541 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:50:22.541 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:22.542 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:50:22.542 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:50:22.542 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:50:22.542 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 33145 2024-02-09T17:50:22.542 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.542 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:50:22.542 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.542 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:50:22.542 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.884388+0000", 2024-02-09T17:50:22.542 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:22.542 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.542 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.543 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:50:22.543 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.884388+0000", 2024-02-09T17:50:22.543 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:22.543 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.543 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.543 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:50:22.543 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.884389+0000", 2024-02-09T17:50:22.543 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5400000000000001e-07 2024-02-09T17:50:22.543 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.543 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.543 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:50:22.543 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.884423+0000", 2024-02-09T17:50:22.543 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3884999999999998e-05 2024-02-09T17:50:22.544 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.544 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.544 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:50:22.544 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.884431+0000", 2024-02-09T17:50:22.544 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1880000000000002e-06 2024-02-09T17:50:22.544 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.544 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.544 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:50:22.544 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.884438+0000", 2024-02-09T17:50:22.544 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.533e-06 2024-02-09T17:50:22.544 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.544 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.545 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:22.545 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.884871+0000", 2024-02-09T17:50:22.545 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000433526 2024-02-09T17:50:22.545 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.545 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.545 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:50:22.545 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.884880+0000", 2024-02-09T17:50:22.545 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9709999999999993e-06 2024-02-09T17:50:22.545 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.545 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.545 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:22.545 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.880283+0000", 2024-02-09T17:50:22.545 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99540302000000003 2024-02-09T17:50:22.546 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.546 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.546 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:50:22.546 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.880419+0000", 2024-02-09T17:50:22.546 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013591300000000001 2024-02-09T17:50:22.546 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.546 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.546 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,11", 2024-02-09T17:50:22.546 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.880531+0000", 2024-02-09T17:50:22.546 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011196000000000001 2024-02-09T17:50:22.546 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.546 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.547 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:22.547 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.882950+0000", 2024-02-09T17:50:22.547 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002418548 2024-02-09T17:50:22.547 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.547 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.547 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:22.547 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.882997+0000", 2024-02-09T17:50:22.547 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7614000000000002e-05 2024-02-09T17:50:22.547 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.547 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.547 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:50:22.547 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.883006+0000", 2024-02-09T17:50:22.548 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3289999999999994e-06 2024-02-09T17:50:22.548 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.548 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.548 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:50:22.548 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.883036+0000", 2024-02-09T17:50:22.548 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.025e-05 2024-02-09T17:50:22.548 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.548 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.548 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:50:22.548 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.883065+0000", 2024-02-09T17:50:22.548 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9544999999999999e-05 2024-02-09T17:50:22.548 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:22.549 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:50:22.549 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:22.549 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.549 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.549 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:33147 5.f 5:fc9a5092:::benchmark_data_smithi179_115199_object33146:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e480)", 2024-02-09T17:50:22.549 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:41:19.884563+0000", 2024-02-09T17:50:22.549 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 542.63617273299997, 2024-02-09T17:50:22.549 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0012039639999999, 2024-02-09T17:50:22.549 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:50:22.549 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:22.549 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:50:22.549 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:50:22.550 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:50:22.550 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 33147 2024-02-09T17:50:22.550 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.550 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:50:22.550 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.550 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:50:22.550 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.884563+0000", 2024-02-09T17:50:22.550 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:22.550 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.550 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.550 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:50:22.550 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.884563+0000", 2024-02-09T17:50:22.550 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:22.551 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.551 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.551 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:50:22.551 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.884563+0000", 2024-02-09T17:50:22.551 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0800000000000004e-07 2024-02-09T17:50:22.551 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.551 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.551 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:50:22.551 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.884627+0000", 2024-02-09T17:50:22.551 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3874999999999994e-05 2024-02-09T17:50:22.551 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.551 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.552 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:50:22.552 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.884636+0000", 2024-02-09T17:50:22.552 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1009999999999998e-06 2024-02-09T17:50:22.552 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.552 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.552 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:50:22.552 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.884642+0000", 2024-02-09T17:50:22.552 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4430000000000002e-06 2024-02-09T17:50:22.552 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.552 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.552 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:22.552 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.884902+0000", 2024-02-09T17:50:22.552 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00026010699999999998 2024-02-09T17:50:22.553 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.553 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.553 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:50:22.553 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.884909+0000", 2024-02-09T17:50:22.553 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7100000000000001e-06 2024-02-09T17:50:22.553 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.553 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.553 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:22.553 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.881550+0000", 2024-02-09T17:50:22.553 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.996641369 2024-02-09T17:50:22.553 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.553 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.554 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:50:22.554 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.881685+0000", 2024-02-09T17:50:22.554 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013473599999999999 2024-02-09T17:50:22.554 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.554 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.554 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,11", 2024-02-09T17:50:22.554 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.881845+0000", 2024-02-09T17:50:22.554 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000160531 2024-02-09T17:50:22.554 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.554 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.554 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:50:22.554 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.883072+0000", 2024-02-09T17:50:22.555 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0012270569999999999 2024-02-09T17:50:22.555 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.555 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.555 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:22.555 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.883122+0000", 2024-02-09T17:50:22.555 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9295000000000002e-05 2024-02-09T17:50:22.555 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.555 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.555 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:22.555 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.885662+0000", 2024-02-09T17:50:22.555 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0025401130000000001 2024-02-09T17:50:22.555 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.555 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.556 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:50:22.556 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.885725+0000", 2024-02-09T17:50:22.556 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3044000000000002e-05 2024-02-09T17:50:22.556 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.556 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.556 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:50:22.556 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.885767+0000", 2024-02-09T17:50:22.556 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1774999999999999e-05 2024-02-09T17:50:22.556 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:22.556 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:50:22.556 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:22.556 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.557 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.557 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:33151 5.f 5:fdb17747:::benchmark_data_smithi179_115199_object33150:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e480)", 2024-02-09T17:50:22.557 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:41:19.886978+0000", 2024-02-09T17:50:22.557 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 542.63375713899995, 2024-02-09T17:50:22.557 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99900822099999997, 2024-02-09T17:50:22.557 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:50:22.557 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:22.557 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:50:22.557 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:50:22.557 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:50:22.557 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 33151 2024-02-09T17:50:22.557 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.558 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:50:22.558 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.558 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:50:22.558 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.886978+0000", 2024-02-09T17:50:22.558 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:22.558 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.558 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.558 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:50:22.558 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.886978+0000", 2024-02-09T17:50:22.558 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:22.558 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.558 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.558 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:50:22.559 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.886979+0000", 2024-02-09T17:50:22.559 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1519999999999999e-06 2024-02-09T17:50:22.559 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.559 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.559 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:50:22.559 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.887089+0000", 2024-02-09T17:50:22.559 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000109262 2024-02-09T17:50:22.559 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.559 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.559 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:50:22.559 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.887104+0000", 2024-02-09T17:50:22.559 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5075999999999999e-05 2024-02-09T17:50:22.559 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.560 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.560 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:50:22.560 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.887111+0000", 2024-02-09T17:50:22.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2660000000000004e-06 2024-02-09T17:50:22.560 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.560 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.560 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:22.560 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.887162+0000", 2024-02-09T17:50:22.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1078999999999997e-05 2024-02-09T17:50:22.560 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.560 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.561 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:50:22.561 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.887169+0000", 2024-02-09T17:50:22.561 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5309999999999998e-06 2024-02-09T17:50:22.561 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.561 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.561 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:22.561 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.883162+0000", 2024-02-09T17:50:22.561 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99599290200000001 2024-02-09T17:50:22.561 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.561 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.561 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:50:22.561 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.883263+0000", 2024-02-09T17:50:22.562 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010171900000000001 2024-02-09T17:50:22.562 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.562 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.562 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,11", 2024-02-09T17:50:22.562 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.883378+0000", 2024-02-09T17:50:22.562 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011450099999999999 2024-02-09T17:50:22.562 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.562 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.562 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:22.562 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.885834+0000", 2024-02-09T17:50:22.562 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0024559619999999999 2024-02-09T17:50:22.562 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.563 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.563 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:50:22.563 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.885851+0000", 2024-02-09T17:50:22.563 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7444000000000001e-05 2024-02-09T17:50:22.563 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.563 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.563 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:22.563 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.885894+0000", 2024-02-09T17:50:22.563 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2522999999999999e-05 2024-02-09T17:50:22.563 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.563 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.563 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:50:22.563 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.885949+0000", 2024-02-09T17:50:22.564 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5337999999999998e-05 2024-02-09T17:50:22.564 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.564 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.564 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:50:22.564 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.885986+0000", 2024-02-09T17:50:22.564 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7465999999999997e-05 2024-02-09T17:50:22.564 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:22.564 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:50:22.564 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:22.564 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.564 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.564 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:33161 5.f 5:f9e596c5:::benchmark_data_smithi179_115199_object33160:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e480)", 2024-02-09T17:50:22.565 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:41:19.892414+0000", 2024-02-09T17:50:22.565 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 542.62832127000001, 2024-02-09T17:50:22.565 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99654300500000004, 2024-02-09T17:50:22.565 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:50:22.565 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:22.565 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:50:22.565 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:50:22.565 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:50:22.565 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 33161 2024-02-09T17:50:22.565 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.565 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:50:22.565 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.565 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:50:22.566 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.892414+0000", 2024-02-09T17:50:22.566 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:22.566 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.566 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.566 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:50:22.566 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.892414+0000", 2024-02-09T17:50:22.566 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:22.566 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.566 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.566 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:50:22.566 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.892415+0000", 2024-02-09T17:50:22.566 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5300000000000004e-07 2024-02-09T17:50:22.567 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.567 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:50:22.567 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.892465+0000", 2024-02-09T17:50:22.567 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0263000000000003e-05 2024-02-09T17:50:22.567 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.567 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:50:22.567 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.892477+0000", 2024-02-09T17:50:22.567 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1676e-05 2024-02-09T17:50:22.567 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.568 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:50:22.568 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.892484+0000", 2024-02-09T17:50:22.568 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1049999999999997e-06 2024-02-09T17:50:22.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.568 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.568 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:22.568 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.893715+0000", 2024-02-09T17:50:22.569 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0012314170000000001 2024-02-09T17:50:22.569 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.569 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.569 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:50:22.569 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.893722+0000", 2024-02-09T17:50:22.569 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7399999999999998e-06 2024-02-09T17:50:22.569 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.569 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.570 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:22.570 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.884334+0000", 2024-02-09T17:50:22.570 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99061202599999998 2024-02-09T17:50:22.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.570 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.570 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:50:22.570 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.884466+0000", 2024-02-09T17:50:22.570 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013154 2024-02-09T17:50:22.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.570 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.570 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,11", 2024-02-09T17:50:22.570 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.884605+0000", 2024-02-09T17:50:22.571 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013961299999999999 2024-02-09T17:50:22.571 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.571 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.571 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:50:22.571 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.888743+0000", 2024-02-09T17:50:22.571 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041376140000000004 2024-02-09T17:50:22.571 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.571 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.571 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:22.571 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.888789+0000", 2024-02-09T17:50:22.571 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6183000000000002e-05 2024-02-09T17:50:22.571 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.572 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.572 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:22.572 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.888858+0000", 2024-02-09T17:50:22.572 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8800000000000005e-05 2024-02-09T17:50:22.572 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.572 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.572 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:50:22.572 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.888920+0000", 2024-02-09T17:50:22.572 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2472999999999996e-05 2024-02-09T17:50:22.572 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.572 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.572 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:50:22.573 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.888957+0000", 2024-02-09T17:50:22.573 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6902e-05 2024-02-09T17:50:22.573 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:22.573 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:50:22.573 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:22.573 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.573 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.573 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:33171 5.f 5:fb959e53:::benchmark_data_smithi179_115199_object33170:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e480)", 2024-02-09T17:50:22.573 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:41:19.897322+0000", 2024-02-09T17:50:22.573 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 542.62341340900002, 2024-02-09T17:50:22.573 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99605270000000001, 2024-02-09T17:50:22.573 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:50:22.574 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:22.574 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:50:22.574 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:50:22.574 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:50:22.574 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 33171 2024-02-09T17:50:22.574 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.574 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:50:22.574 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.574 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:50:22.574 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.897322+0000", 2024-02-09T17:50:22.574 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:22.574 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.574 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.575 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:50:22.575 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.897322+0000", 2024-02-09T17:50:22.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:22.575 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.575 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.575 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:50:22.575 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.897323+0000", 2024-02-09T17:50:22.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5499999999999997e-07 2024-02-09T17:50:22.575 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.575 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.575 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:50:22.575 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.897380+0000", 2024-02-09T17:50:22.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6919999999999997e-05 2024-02-09T17:50:22.576 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.576 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.576 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:50:22.576 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.897390+0000", 2024-02-09T17:50:22.576 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0555e-05 2024-02-09T17:50:22.576 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.576 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.576 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:50:22.576 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.897396+0000", 2024-02-09T17:50:22.576 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.716e-06 2024-02-09T17:50:22.576 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.576 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.577 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:22.577 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.898279+0000", 2024-02-09T17:50:22.577 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000883083 2024-02-09T17:50:22.577 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.577 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.577 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:50:22.577 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.898290+0000", 2024-02-09T17:50:22.577 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0774999999999999e-05 2024-02-09T17:50:22.577 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.577 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.577 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:22.577 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.886029+0000", 2024-02-09T17:50:22.577 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.98773960500000002 2024-02-09T17:50:22.577 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.578 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.578 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:50:22.578 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.886155+0000", 2024-02-09T17:50:22.578 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000125726 2024-02-09T17:50:22.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.578 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.578 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,11", 2024-02-09T17:50:22.578 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.886313+0000", 2024-02-09T17:50:22.578 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015792399999999999 2024-02-09T17:50:22.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.578 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.578 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:50:22.578 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.888800+0000", 2024-02-09T17:50:22.579 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0024873260000000002 2024-02-09T17:50:22.579 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.579 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.579 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:22.579 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.888823+0000", 2024-02-09T17:50:22.579 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2690000000000001e-05 2024-02-09T17:50:22.579 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.579 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.579 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:22.579 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.893269+0000", 2024-02-09T17:50:22.579 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0044462659999999999 2024-02-09T17:50:22.579 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.579 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.580 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:50:22.580 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.893329+0000", 2024-02-09T17:50:22.580 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9642999999999998e-05 2024-02-09T17:50:22.580 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.580 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.580 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:50:22.580 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.893375+0000", 2024-02-09T17:50:22.580 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5615999999999998e-05 2024-02-09T17:50:22.580 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:22.580 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:50:22.580 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:22.580 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.581 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.581 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:33173 5.f 5:ff65210a:::benchmark_data_smithi179_115199_object33172:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e480)", 2024-02-09T17:50:22.581 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:41:19.897448+0000", 2024-02-09T17:50:22.581 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 542.62328755999999, 2024-02-09T17:50:22.581 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99608639899999996, 2024-02-09T17:50:22.581 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:50:22.581 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:22.581 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:50:22.581 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:50:22.581 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:50:22.581 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 33173 2024-02-09T17:50:22.581 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.582 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:50:22.582 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.582 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:50:22.582 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.897448+0000", 2024-02-09T17:50:22.582 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:22.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.582 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.582 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:50:22.582 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.897448+0000", 2024-02-09T17:50:22.582 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:22.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.582 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.582 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:50:22.583 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.897448+0000", 2024-02-09T17:50:22.583 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2899999999999999e-07 2024-02-09T17:50:22.583 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.583 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.583 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:50:22.583 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.897516+0000", 2024-02-09T17:50:22.583 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8206000000000004e-05 2024-02-09T17:50:22.583 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.583 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.583 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:50:22.583 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.897524+0000", 2024-02-09T17:50:22.583 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9400000000000002e-06 2024-02-09T17:50:22.583 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.584 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.584 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:50:22.584 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.897528+0000", 2024-02-09T17:50:22.584 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9999999999999998e-06 2024-02-09T17:50:22.584 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.584 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.584 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:22.584 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.898384+0000", 2024-02-09T17:50:22.584 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00085551699999999997 2024-02-09T17:50:22.584 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.584 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.584 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:50:22.585 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.898390+0000", 2024-02-09T17:50:22.585 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8379999999999998e-06 2024-02-09T17:50:22.585 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.585 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.585 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:22.585 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.887400+0000", 2024-02-09T17:50:22.585 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.98901084100000003 2024-02-09T17:50:22.585 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.585 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.585 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:50:22.585 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.887534+0000", 2024-02-09T17:50:22.585 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013387100000000001 2024-02-09T17:50:22.585 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.586 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.586 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,11", 2024-02-09T17:50:22.586 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.887678+0000", 2024-02-09T17:50:22.586 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014406799999999999 2024-02-09T17:50:22.586 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.586 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.586 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:50:22.586 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.891737+0000", 2024-02-09T17:50:22.586 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0040586219999999996 2024-02-09T17:50:22.586 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.586 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.586 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:22.586 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.893201+0000", 2024-02-09T17:50:22.587 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014644359999999999 2024-02-09T17:50:22.587 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.587 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.587 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:22.587 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.893443+0000", 2024-02-09T17:50:22.587 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024162399999999999 2024-02-09T17:50:22.587 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.587 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.587 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:50:22.587 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.893494+0000", 2024-02-09T17:50:22.587 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1360999999999998e-05 2024-02-09T17:50:22.587 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.588 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.588 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:50:22.588 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.893534+0000", 2024-02-09T17:50:22.588 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9746000000000002e-05 2024-02-09T17:50:22.588 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:22.588 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:50:22.588 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:22.588 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.588 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.588 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:33179 5.f 5:f9bfb60a:::benchmark_data_smithi179_115199_object33178:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e480)", 2024-02-09T17:50:22.588 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:41:19.900843+0000", 2024-02-09T17:50:22.588 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 542.61989272999995, 2024-02-09T17:50:22.589 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99583007199999996, 2024-02-09T17:50:22.589 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:50:22.589 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:22.589 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:50:22.589 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:50:22.589 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:50:22.589 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 33179 2024-02-09T17:50:22.590 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.590 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:50:22.590 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.590 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:50:22.590 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.900843+0000", 2024-02-09T17:50:22.590 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:22.590 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.590 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.591 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:50:22.591 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.900843+0000", 2024-02-09T17:50:22.591 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:22.591 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.591 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.591 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:50:22.591 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.900843+0000", 2024-02-09T17:50:22.591 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2600000000000002e-07 2024-02-09T17:50:22.591 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.591 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.591 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:50:22.591 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.900874+0000", 2024-02-09T17:50:22.592 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0886000000000002e-05 2024-02-09T17:50:22.592 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.592 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.592 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:50:22.592 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.900880+0000", 2024-02-09T17:50:22.592 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.173e-06 2024-02-09T17:50:22.592 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.592 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.592 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:50:22.592 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.900885+0000", 2024-02-09T17:50:22.592 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2660000000000003e-06 2024-02-09T17:50:22.592 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.593 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.593 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:22.593 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.902133+0000", 2024-02-09T17:50:22.593 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001248859 2024-02-09T17:50:22.593 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.593 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.593 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:50:22.593 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.902140+0000", 2024-02-09T17:50:22.593 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4930000000000003e-06 2024-02-09T17:50:22.593 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.593 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.593 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:22.593 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.888995+0000", 2024-02-09T17:50:22.593 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.98685520100000002 2024-02-09T17:50:22.594 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.594 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.594 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:50:22.594 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.889125+0000", 2024-02-09T17:50:22.594 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012953800000000001 2024-02-09T17:50:22.594 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.594 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.594 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,11", 2024-02-09T17:50:22.594 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.889285+0000", 2024-02-09T17:50:22.594 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015997400000000001 2024-02-09T17:50:22.594 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.594 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.595 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:50:22.595 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.893396+0000", 2024-02-09T17:50:22.595 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041114929999999999 2024-02-09T17:50:22.595 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.595 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.595 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:22.595 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.893608+0000", 2024-02-09T17:50:22.595 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000211521 2024-02-09T17:50:22.595 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.595 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.596 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:22.596 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.896477+0000", 2024-02-09T17:50:22.596 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0028690370000000001 2024-02-09T17:50:22.596 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.596 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.596 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:50:22.596 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.896592+0000", 2024-02-09T17:50:22.596 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000115486 2024-02-09T17:50:22.596 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.596 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.596 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:50:22.597 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.896673+0000", 2024-02-09T17:50:22.597 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0519000000000006e-05 2024-02-09T17:50:22.597 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:22.597 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:50:22.597 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:22.597 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.597 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.597 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:33186 5.f 5:fdd239c2:::benchmark_data_smithi179_115199_object33185:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e480)", 2024-02-09T17:50:22.597 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:41:19.904132+0000", 2024-02-09T17:50:22.597 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 542.61660323000001, 2024-02-09T17:50:22.597 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99418198400000002, 2024-02-09T17:50:22.597 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:50:22.598 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:22.598 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:50:22.598 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:50:22.598 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:50:22.598 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 33186 2024-02-09T17:50:22.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.598 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:50:22.598 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.598 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:50:22.598 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.904132+0000", 2024-02-09T17:50:22.598 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:22.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.598 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.599 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:50:22.599 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.904132+0000", 2024-02-09T17:50:22.599 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:22.599 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.599 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.599 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:50:22.599 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.904133+0000", 2024-02-09T17:50:22.599 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2200000000000003e-07 2024-02-09T17:50:22.599 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.599 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.599 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:50:22.599 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.904177+0000", 2024-02-09T17:50:22.599 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3850000000000002e-05 2024-02-09T17:50:22.600 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.600 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.600 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:50:22.600 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.904187+0000", 2024-02-09T17:50:22.600 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9110000000000005e-06 2024-02-09T17:50:22.600 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.600 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.600 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:50:22.600 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.904192+0000", 2024-02-09T17:50:22.600 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1959999999999999e-06 2024-02-09T17:50:22.600 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.600 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.600 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:22.601 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.904246+0000", 2024-02-09T17:50:22.601 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.401e-05 2024-02-09T17:50:22.601 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.601 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.601 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:50:22.601 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.904253+0000", 2024-02-09T17:50:22.601 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0029999999999997e-06 2024-02-09T17:50:22.601 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.601 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.601 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:22.601 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.890398+0000", 2024-02-09T17:50:22.601 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.98614499200000005 2024-02-09T17:50:22.601 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.602 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.602 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:50:22.602 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.890527+0000", 2024-02-09T17:50:22.602 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012886500000000001 2024-02-09T17:50:22.602 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.602 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.602 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,11", 2024-02-09T17:50:22.602 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.890675+0000", 2024-02-09T17:50:22.602 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000148682 2024-02-09T17:50:22.602 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.602 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.602 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:50:22.603 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.893402+0000", 2024-02-09T17:50:22.603 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002726731 2024-02-09T17:50:22.603 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.603 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.603 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:22.603 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.896720+0000", 2024-02-09T17:50:22.603 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0033173299999999998 2024-02-09T17:50:22.603 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.603 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.603 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:22.603 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.898206+0000", 2024-02-09T17:50:22.603 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014864990000000001 2024-02-09T17:50:22.604 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.604 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.604 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:50:22.604 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.898286+0000", 2024-02-09T17:50:22.604 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9513999999999998e-05 2024-02-09T17:50:22.604 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.604 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.604 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:50:22.604 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.898314+0000", 2024-02-09T17:50:22.604 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8578999999999999e-05 2024-02-09T17:50:22.604 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:22.604 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:50:22.604 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:22.605 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.605 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.605 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:33219 5.f 5:ff57a091:::benchmark_data_smithi179_115199_object33218:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e480)", 2024-02-09T17:50:22.605 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:41:19.933494+0000", 2024-02-09T17:50:22.605 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 542.58724108000001, 2024-02-09T17:50:22.605 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.96495839699999997, 2024-02-09T17:50:22.605 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:50:22.605 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:22.605 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:50:22.605 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:50:22.605 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:50:22.605 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 33219 2024-02-09T17:50:22.605 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.606 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:50:22.606 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.606 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:50:22.606 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.933494+0000", 2024-02-09T17:50:22.607 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:22.607 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.607 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.607 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:50:22.607 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.933494+0000", 2024-02-09T17:50:22.607 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:22.607 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.607 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.607 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:50:22.607 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.933495+0000", 2024-02-09T17:50:22.607 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.85e-07 2024-02-09T17:50:22.607 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.608 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.608 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:50:22.608 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.933562+0000", 2024-02-09T17:50:22.608 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7018000000000001e-05 2024-02-09T17:50:22.608 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.608 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.608 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:50:22.608 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.933572+0000", 2024-02-09T17:50:22.608 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3249999999999997e-06 2024-02-09T17:50:22.608 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.608 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.608 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:50:22.608 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.933579+0000", 2024-02-09T17:50:22.608 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.041e-06 2024-02-09T17:50:22.609 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.609 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.609 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:22.609 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.933654+0000", 2024-02-09T17:50:22.609 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5210000000000007e-05 2024-02-09T17:50:22.609 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.609 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.609 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:50:22.609 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.933677+0000", 2024-02-09T17:50:22.609 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2849000000000001e-05 2024-02-09T17:50:22.609 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.609 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.609 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:22.609 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.891776+0000", 2024-02-09T17:50:22.609 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.95809916399999995 2024-02-09T17:50:22.610 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.610 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.610 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:50:22.610 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.891908+0000", 2024-02-09T17:50:22.610 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013222099999999999 2024-02-09T17:50:22.610 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.610 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.610 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,11", 2024-02-09T17:50:22.610 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.892056+0000", 2024-02-09T17:50:22.610 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014792200000000001 2024-02-09T17:50:22.610 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.610 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.610 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:50:22.610 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.893546+0000", 2024-02-09T17:50:22.610 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014900950000000001 2024-02-09T17:50:22.611 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.611 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.611 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:22.611 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.896771+0000", 2024-02-09T17:50:22.611 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0032249969999999998 2024-02-09T17:50:22.611 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.611 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.611 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:22.611 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.898381+0000", 2024-02-09T17:50:22.611 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001610385 2024-02-09T17:50:22.611 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.611 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.611 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:50:22.611 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.898429+0000", 2024-02-09T17:50:22.611 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7216999999999999e-05 2024-02-09T17:50:22.612 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.612 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.612 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:50:22.612 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.898453+0000", 2024-02-09T17:50:22.612 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4068000000000001e-05 2024-02-09T17:50:22.612 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:22.612 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:50:22.612 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:22.612 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.612 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.612 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:33236 5.f 5:fe55e4cc:::benchmark_data_smithi179_115199_object33235:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e480)", 2024-02-09T17:50:22.612 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:41:19.965076+0000", 2024-02-09T17:50:22.612 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 542.55565984400005, 2024-02-09T17:50:22.612 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.93345155099999999, 2024-02-09T17:50:22.613 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:50:22.613 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:22.613 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:50:22.613 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:50:22.613 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:50:22.613 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 33236 2024-02-09T17:50:22.613 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.613 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:50:22.613 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.613 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:50:22.613 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.965076+0000", 2024-02-09T17:50:22.613 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:22.613 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.613 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.614 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:50:22.614 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.965076+0000", 2024-02-09T17:50:22.614 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:22.614 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.614 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.614 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:50:22.614 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.965076+0000", 2024-02-09T17:50:22.614 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5099999999999998e-07 2024-02-09T17:50:22.614 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.614 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.614 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:50:22.614 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.965129+0000", 2024-02-09T17:50:22.614 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2562999999999998e-05 2024-02-09T17:50:22.614 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.615 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:50:22.615 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.965140+0000", 2024-02-09T17:50:22.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0628999999999999e-05 2024-02-09T17:50:22.615 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.615 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:50:22.615 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.965145+0000", 2024-02-09T17:50:22.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6180000000000003e-06 2024-02-09T17:50:22.615 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.615 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:22.615 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.965192+0000", 2024-02-09T17:50:22.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6230000000000003e-05 2024-02-09T17:50:22.615 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.616 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:50:22.616 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:19.965199+0000", 2024-02-09T17:50:22.616 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5970000000000003e-06 2024-02-09T17:50:22.616 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.616 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.616 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:22.616 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.893642+0000", 2024-02-09T17:50:22.616 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.928443346 2024-02-09T17:50:22.616 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.616 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.616 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:50:22.616 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.893772+0000", 2024-02-09T17:50:22.616 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000129107 2024-02-09T17:50:22.617 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.617 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,11", 2024-02-09T17:50:22.617 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.893896+0000", 2024-02-09T17:50:22.617 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012469400000000001 2024-02-09T17:50:22.617 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.617 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:50:22.617 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.896732+0000", 2024-02-09T17:50:22.617 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0028361509999999999 2024-02-09T17:50:22.617 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.617 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:22.617 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.898345+0000", 2024-02-09T17:50:22.617 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001612229 2024-02-09T17:50:22.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.618 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:22.618 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.898478+0000", 2024-02-09T17:50:22.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000133368 2024-02-09T17:50:22.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.618 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:50:22.618 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.898507+0000", 2024-02-09T17:50:22.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9377999999999999e-05 2024-02-09T17:50:22.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.618 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:50:22.618 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:20.898527+0000", 2024-02-09T17:50:22.619 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.969e-05 2024-02-09T17:50:22.619 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:22.619 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:50:22.619 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:22.619 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.619 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.619 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:74616 5.6 5:63c31902:::benchmark_data_smithi179_115199_object74615:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e489)", 2024-02-09T17:50:22.619 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:41:45.041982+0000", 2024-02-09T17:50:22.619 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 517.47875339999996, 2024-02-09T17:50:22.619 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19583514199999999, 2024-02-09T17:50:22.619 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:50:22.619 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:22.619 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:50:22.619 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:50:22.619 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:50:22.620 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 74616 2024-02-09T17:50:22.620 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.620 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:50:22.620 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.620 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:50:22.620 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:45.041982+0000", 2024-02-09T17:50:22.620 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:22.620 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.620 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.620 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:50:22.620 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:45.041982+0000", 2024-02-09T17:50:22.620 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:22.620 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.620 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.621 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:50:22.621 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:45.041983+0000", 2024-02-09T17:50:22.621 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2860000000000001e-06 2024-02-09T17:50:22.621 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.621 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.621 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:50:22.621 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:45.042103+0000", 2024-02-09T17:50:22.621 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011937700000000001 2024-02-09T17:50:22.621 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.621 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.621 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:50:22.621 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:45.042113+0000", 2024-02-09T17:50:22.621 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0144e-05 2024-02-09T17:50:22.621 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.622 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.622 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:50:22.622 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:45.042120+0000", 2024-02-09T17:50:22.622 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9539999999999998e-06 2024-02-09T17:50:22.622 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.622 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.622 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:22.622 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:45.042173+0000", 2024-02-09T17:50:22.622 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3300999999999997e-05 2024-02-09T17:50:22.622 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.622 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.622 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:50:22.622 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:45.042315+0000", 2024-02-09T17:50:22.622 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014153200000000001 2024-02-09T17:50:22.622 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.623 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7,10", 2024-02-09T17:50:22.623 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:45.042428+0000", 2024-02-09T17:50:22.623 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011296800000000001 2024-02-09T17:50:22.623 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.623 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:50:22.623 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:45.047248+0000", 2024-02-09T17:50:22.623 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0048207079999999999 2024-02-09T17:50:22.623 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.623 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:22.624 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:45.048755+0000", 2024-02-09T17:50:22.624 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0015067839999999999 2024-02-09T17:50:22.624 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.624 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.624 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:22.624 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:45.237697+0000", 2024-02-09T17:50:22.624 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18894223499999999 2024-02-09T17:50:22.624 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.624 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.624 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:50:22.624 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:45.237757+0000", 2024-02-09T17:50:22.625 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9617000000000002e-05 2024-02-09T17:50:22.625 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.625 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.625 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:50:22.625 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:41:45.237817+0000", 2024-02-09T17:50:22.625 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0235999999999997e-05 2024-02-09T17:50:22.625 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:22.625 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:50:22.625 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:22.625 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.625 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.625 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:168352 5.6 5:645ef9c0:::benchmark_data_smithi179_115199_object39446:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e520)", 2024-02-09T17:50:22.625 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:43:21.023013+0000", 2024-02-09T17:50:22.626 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 421.49772208299999, 2024-02-09T17:50:22.626 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.234114989, 2024-02-09T17:50:22.626 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:50:22.626 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:22.626 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:50:22.626 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:50:22.626 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:50:22.626 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 168352 2024-02-09T17:50:22.626 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.626 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:50:22.626 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.626 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:50:22.627 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.023013+0000", 2024-02-09T17:50:22.627 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:22.627 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.627 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.627 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:50:22.627 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.023013+0000", 2024-02-09T17:50:22.627 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:22.627 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.627 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.627 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:50:22.627 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.023014+0000", 2024-02-09T17:50:22.627 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2299999999999999e-07 2024-02-09T17:50:22.627 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.627 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.627 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:50:22.628 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.023019+0000", 2024-02-09T17:50:22.628 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.118e-06 2024-02-09T17:50:22.628 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.629 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.629 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:50:22.629 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.023030+0000", 2024-02-09T17:50:22.629 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0688e-05 2024-02-09T17:50:22.629 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.629 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.629 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:50:22.629 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.023038+0000", 2024-02-09T17:50:22.629 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8259999999999993e-06 2024-02-09T17:50:22.629 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.629 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.630 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:22.630 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.023094+0000", 2024-02-09T17:50:22.630 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6164e-05 2024-02-09T17:50:22.630 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.630 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.630 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:50:22.630 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.023198+0000", 2024-02-09T17:50:22.630 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000104384 2024-02-09T17:50:22.630 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.630 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.630 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7,10", 2024-02-09T17:50:22.630 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.023297+0000", 2024-02-09T17:50:22.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9057999999999999e-05 2024-02-09T17:50:22.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.631 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:50:22.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.028576+0000", 2024-02-09T17:50:22.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0052786509999999997 2024-02-09T17:50:22.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.631 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:22.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.033487+0000", 2024-02-09T17:50:22.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0049112899999999996 2024-02-09T17:50:22.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.631 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:22.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.257054+0000", 2024-02-09T17:50:22.632 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22356674900000001 2024-02-09T17:50:22.632 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.632 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.632 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:50:22.632 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.257092+0000", 2024-02-09T17:50:22.632 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7500999999999999e-05 2024-02-09T17:50:22.632 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.632 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.632 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:50:22.632 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.257128+0000", 2024-02-09T17:50:22.632 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6637000000000002e-05 2024-02-09T17:50:22.632 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:22.632 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:50:22.633 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:22.633 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.633 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.633 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:168354 5.6 5:62688f71:::benchmark_data_smithi179_115199_object39448:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e520)", 2024-02-09T17:50:22.633 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:43:21.028975+0000", 2024-02-09T17:50:22.633 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 421.49176051299997, 2024-02-09T17:50:22.633 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.228236679, 2024-02-09T17:50:22.633 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:50:22.633 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:22.633 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:50:22.633 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:50:22.633 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:50:22.633 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 168354 2024-02-09T17:50:22.633 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.634 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:50:22.634 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.634 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:50:22.634 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.028975+0000", 2024-02-09T17:50:22.634 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:22.634 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.634 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.634 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:50:22.634 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.028975+0000", 2024-02-09T17:50:22.634 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:22.634 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.634 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.634 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:50:22.634 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.028975+0000", 2024-02-09T17:50:22.635 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.32e-07 2024-02-09T17:50:22.635 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.635 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.635 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:50:22.635 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.028977+0000", 2024-02-09T17:50:22.635 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0779999999999998e-06 2024-02-09T17:50:22.635 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.635 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.635 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:50:22.635 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.028985+0000", 2024-02-09T17:50:22.635 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.515e-06 2024-02-09T17:50:22.635 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.635 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.635 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:50:22.635 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.029003+0000", 2024-02-09T17:50:22.636 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8247999999999999e-05 2024-02-09T17:50:22.636 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.636 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.636 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:22.636 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.029055+0000", 2024-02-09T17:50:22.636 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1876000000000003e-05 2024-02-09T17:50:22.636 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.636 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.636 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:50:22.636 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.029175+0000", 2024-02-09T17:50:22.636 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000119752 2024-02-09T17:50:22.636 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.636 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.636 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7,10", 2024-02-09T17:50:22.636 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.029304+0000", 2024-02-09T17:50:22.637 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000129524 2024-02-09T17:50:22.637 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.637 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.637 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:22.637 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.038462+0000", 2024-02-09T17:50:22.637 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0091575519999999994 2024-02-09T17:50:22.637 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.637 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.637 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:50:22.637 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.038479+0000", 2024-02-09T17:50:22.637 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7393000000000001e-05 2024-02-09T17:50:22.637 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.637 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.637 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:22.637 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.257154+0000", 2024-02-09T17:50:22.638 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21867436200000001 2024-02-09T17:50:22.638 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.638 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.638 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:50:22.638 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.257181+0000", 2024-02-09T17:50:22.638 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7747999999999999e-05 2024-02-09T17:50:22.638 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.638 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.638 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:50:22.638 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.257212+0000", 2024-02-09T17:50:22.638 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0199e-05 2024-02-09T17:50:22.638 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:22.638 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:50:22.638 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:22.638 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.638 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.639 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:168372 5.6 5:6fd0bce5:::benchmark_data_smithi179_115199_object39466:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e520)", 2024-02-09T17:50:22.639 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:43:21.059240+0000", 2024-02-09T17:50:22.639 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 421.46149520799997, 2024-02-09T17:50:22.639 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.198070792, 2024-02-09T17:50:22.639 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:50:22.639 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:22.639 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:50:22.639 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:50:22.639 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:50:22.639 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 168372 2024-02-09T17:50:22.640 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.640 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:50:22.640 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.640 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:50:22.640 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.059240+0000", 2024-02-09T17:50:22.640 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:22.640 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.640 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.640 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:50:22.640 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.059240+0000", 2024-02-09T17:50:22.640 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:22.640 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.640 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.640 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:50:22.641 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.059242+0000", 2024-02-09T17:50:22.641 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7999999999999999e-06 2024-02-09T17:50:22.641 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.641 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.641 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:50:22.641 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.059246+0000", 2024-02-09T17:50:22.641 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4490000000000001e-06 2024-02-09T17:50:22.641 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.641 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.641 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:50:22.641 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.059254+0000", 2024-02-09T17:50:22.641 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9319999999999996e-06 2024-02-09T17:50:22.641 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.641 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.641 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:50:22.641 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.059259+0000", 2024-02-09T17:50:22.642 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1340000000000003e-06 2024-02-09T17:50:22.642 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.642 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.642 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:22.642 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.059306+0000", 2024-02-09T17:50:22.642 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6492000000000001e-05 2024-02-09T17:50:22.642 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.642 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.642 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:50:22.642 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.059431+0000", 2024-02-09T17:50:22.642 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012514700000000001 2024-02-09T17:50:22.642 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.642 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.642 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7,10", 2024-02-09T17:50:22.643 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.059520+0000", 2024-02-09T17:50:22.644 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8789999999999995e-05 2024-02-09T17:50:22.644 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.644 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.644 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:50:22.644 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.068663+0000", 2024-02-09T17:50:22.644 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0091426000000000007 2024-02-09T17:50:22.644 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.644 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.644 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:22.644 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.070771+0000", 2024-02-09T17:50:22.645 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021080729999999998 2024-02-09T17:50:22.645 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.645 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.645 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:22.645 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.257247+0000", 2024-02-09T17:50:22.645 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.186476065 2024-02-09T17:50:22.645 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.645 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.645 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:50:22.645 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.257281+0000", 2024-02-09T17:50:22.645 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3952000000000003e-05 2024-02-09T17:50:22.645 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.645 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.645 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:50:22.645 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:21.257311+0000", 2024-02-09T17:50:22.646 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0358e-05 2024-02-09T17:50:22.646 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:22.646 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:50:22.646 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:22.646 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.646 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.647 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:170463 5.6 5:6cc3e09d:::benchmark_data_smithi179_115199_object41557:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e522)", 2024-02-09T17:50:22.647 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:43:25.160777+0000", 2024-02-09T17:50:22.647 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 417.35995876700002, 2024-02-09T17:50:22.647 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.754693777, 2024-02-09T17:50:22.647 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:50:22.647 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-09T17:50:22.647 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:50:22.647 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:50:22.647 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:50:22.647 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 170463 2024-02-09T17:50:22.647 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.647 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:50:22.647 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.648 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:50:22.648 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:25.160777+0000", 2024-02-09T17:50:22.648 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:22.648 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.648 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.648 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:50:22.648 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:25.160777+0000", 2024-02-09T17:50:22.648 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:22.648 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.648 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.648 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:50:22.648 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:25.160778+0000", 2024-02-09T17:50:22.649 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8589999999999999e-06 2024-02-09T17:50:22.649 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.649 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.649 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:50:22.649 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:25.160782+0000", 2024-02-09T17:50:22.649 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.72e-06 2024-02-09T17:50:22.649 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.649 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.649 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:50:22.649 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:25.160792+0000", 2024-02-09T17:50:22.649 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0268000000000001e-05 2024-02-09T17:50:22.649 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.649 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.649 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:50:22.650 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:25.160799+0000", 2024-02-09T17:50:22.650 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.934e-06 2024-02-09T17:50:22.650 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.650 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.650 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:22.650 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:25.160847+0000", 2024-02-09T17:50:22.650 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7330999999999998e-05 2024-02-09T17:50:22.650 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.650 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.650 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:50:22.650 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:25.160857+0000", 2024-02-09T17:50:22.650 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0726e-05 2024-02-09T17:50:22.650 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.651 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.651 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:22.651 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:26.915388+0000", 2024-02-09T17:50:22.651 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.754530608 2024-02-09T17:50:22.651 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.651 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.651 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:50:22.651 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:26.915470+0000", 2024-02-09T17:50:22.651 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2330999999999995e-05 2024-02-09T17:50:22.651 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:22.651 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:50:22.651 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:22.651 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.652 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.652 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:170464 5.6 5:6e5e1d76:::benchmark_data_smithi179_115199_object41558:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e522)", 2024-02-09T17:50:22.652 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:43:25.160814+0000", 2024-02-09T17:50:22.652 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 417.35992134700001, 2024-02-09T17:50:22.652 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7549528620000001, 2024-02-09T17:50:22.652 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:50:22.652 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-09T17:50:22.652 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:50:22.652 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:50:22.652 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:50:22.652 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 170464 2024-02-09T17:50:22.652 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.652 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:50:22.652 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.652 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:50:22.653 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:25.160814+0000", 2024-02-09T17:50:22.653 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:22.653 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.653 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:50:22.653 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:25.160814+0000", 2024-02-09T17:50:22.653 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:22.655 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.655 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.655 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:50:22.655 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:25.160815+0000", 2024-02-09T17:50:22.655 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9100000000000004e-07 2024-02-09T17:50:22.655 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.656 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:50:22.656 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:25.160817+0000", 2024-02-09T17:50:22.656 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.238e-06 2024-02-09T17:50:22.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.656 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:50:22.656 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:25.160825+0000", 2024-02-09T17:50:22.656 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0260000000000007e-06 2024-02-09T17:50:22.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.656 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:50:22.656 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:25.160830+0000", 2024-02-09T17:50:22.656 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0930000000000002e-06 2024-02-09T17:50:22.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.657 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.657 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:22.657 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:25.160883+0000", 2024-02-09T17:50:22.657 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3606000000000001e-05 2024-02-09T17:50:22.657 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.657 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.657 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:50:22.657 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:25.160891+0000", 2024-02-09T17:50:22.657 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2060000000000001e-06 2024-02-09T17:50:22.657 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.657 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.657 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:22.657 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:26.915714+0000", 2024-02-09T17:50:22.657 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.754823577 2024-02-09T17:50:22.657 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.658 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.658 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:50:22.658 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:26.915767+0000", 2024-02-09T17:50:22.658 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2624999999999998e-05 2024-02-09T17:50:22.658 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:22.658 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:50:22.658 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:22.658 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.658 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.658 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:170477 5.6 5:6e670660:::benchmark_data_smithi179_115199_object41571:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e522)", 2024-02-09T17:50:22.658 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:43:25.160837+0000", 2024-02-09T17:50:22.658 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 417.359898606, 2024-02-09T17:50:22.658 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7656936920000001, 2024-02-09T17:50:22.658 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:50:22.658 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:22.659 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:50:22.659 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:50:22.659 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:50:22.659 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 170477 2024-02-09T17:50:22.659 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.659 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:50:22.659 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.659 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:50:22.659 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:25.160837+0000", 2024-02-09T17:50:22.659 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:22.659 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.659 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.659 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:50:22.659 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:25.160837+0000", 2024-02-09T17:50:22.659 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:22.660 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.660 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.660 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:50:22.660 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:25.160837+0000", 2024-02-09T17:50:22.660 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8000000000000001e-07 2024-02-09T17:50:22.660 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.660 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.660 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:50:22.660 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:25.160839+0000", 2024-02-09T17:50:22.660 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.277e-06 2024-02-09T17:50:22.660 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.660 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.660 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:50:22.660 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:25.160846+0000", 2024-02-09T17:50:22.660 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5400000000000001e-06 2024-02-09T17:50:22.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.661 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:50:22.661 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:25.160852+0000", 2024-02-09T17:50:22.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7400000000000001e-06 2024-02-09T17:50:22.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.661 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:22.661 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:25.160908+0000", 2024-02-09T17:50:22.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6518999999999999e-05 2024-02-09T17:50:22.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.661 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T17:50:22.661 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:25.160912+0000", 2024-02-09T17:50:22.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8770000000000003e-06 2024-02-09T17:50:22.662 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.662 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.662 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:22.662 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:26.915817+0000", 2024-02-09T17:50:22.663 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7549047770000001 2024-02-09T17:50:22.663 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.663 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.663 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:50:22.663 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:26.915912+0000", 2024-02-09T17:50:22.664 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5142000000000005e-05 2024-02-09T17:50:22.664 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.664 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.664 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,7,10", 2024-02-09T17:50:22.664 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:26.916010+0000", 2024-02-09T17:50:22.665 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7956999999999997e-05 2024-02-09T17:50:22.665 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.665 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.665 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:22.665 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:26.920814+0000", 2024-02-09T17:50:22.665 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0048041680000000002 2024-02-09T17:50:22.665 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.665 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.665 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:50:22.665 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:26.920835+0000", 2024-02-09T17:50:22.665 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1160999999999999e-05 2024-02-09T17:50:22.665 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.665 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.666 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:22.666 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:26.921295+0000", 2024-02-09T17:50:22.666 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000459693 2024-02-09T17:50:22.666 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.666 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.666 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:22.666 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:26.926323+0000", 2024-02-09T17:50:22.666 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.005028444 2024-02-09T17:50:22.666 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.666 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.666 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:50:22.667 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:26.926405+0000", 2024-02-09T17:50:22.667 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1656999999999994e-05 2024-02-09T17:50:22.667 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.667 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.667 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:50:22.667 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:26.926530+0000", 2024-02-09T17:50:22.667 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012536000000000001 2024-02-09T17:50:22.667 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:22.667 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:50:22.667 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:22.667 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.667 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.668 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:183800 5.6 5:6f507896:::benchmark_data_smithi179_115199_object54894:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e533)", 2024-02-09T17:50:22.668 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:43:43.785304+0000", 2024-02-09T17:50:22.668 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 398.73543130600001, 2024-02-09T17:50:22.668 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.206321332, 2024-02-09T17:50:22.668 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:50:22.668 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:22.668 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:50:22.668 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:50:22.668 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:50:22.668 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 183800 2024-02-09T17:50:22.668 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.668 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:50:22.668 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.668 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:50:22.669 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:43.785304+0000", 2024-02-09T17:50:22.669 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:22.669 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.669 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.669 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:50:22.669 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:43.785304+0000", 2024-02-09T17:50:22.669 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:22.669 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.669 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.669 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:50:22.669 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:43.785305+0000", 2024-02-09T17:50:22.669 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5999999999999999e-07 2024-02-09T17:50:22.669 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.669 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.670 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:50:22.670 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:43.785307+0000", 2024-02-09T17:50:22.670 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0219999999999999e-06 2024-02-09T17:50:22.670 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.670 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.670 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:50:22.670 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:43.785312+0000", 2024-02-09T17:50:22.670 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7860000000000002e-06 2024-02-09T17:50:22.670 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.670 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.670 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:50:22.670 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:43.785317+0000", 2024-02-09T17:50:22.670 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.826e-06 2024-02-09T17:50:22.670 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.671 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.671 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:22.671 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:43.785369+0000", 2024-02-09T17:50:22.671 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1663999999999999e-05 2024-02-09T17:50:22.671 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.671 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.671 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:50:22.671 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:43.785494+0000", 2024-02-09T17:50:22.671 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012557599999999999 2024-02-09T17:50:22.671 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.671 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.672 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,7,10", 2024-02-09T17:50:22.672 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:43.785618+0000", 2024-02-09T17:50:22.672 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012398800000000001 2024-02-09T17:50:22.672 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.672 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.672 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:22.672 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:43.798410+0000", 2024-02-09T17:50:22.672 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012791234 2024-02-09T17:50:22.672 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.672 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.672 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:50:22.672 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:43.798435+0000", 2024-02-09T17:50:22.672 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5100999999999999e-05 2024-02-09T17:50:22.672 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.673 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.673 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:22.673 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:43.811382+0000", 2024-02-09T17:50:22.673 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012947452 2024-02-09T17:50:22.673 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.673 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.673 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:22.673 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:43.991445+0000", 2024-02-09T17:50:22.673 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.180062572 2024-02-09T17:50:22.673 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.673 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.673 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:50:22.673 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:43.991503+0000", 2024-02-09T17:50:22.673 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7794000000000003e-05 2024-02-09T17:50:22.674 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:22.674 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:22.674 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:50:22.674 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:43:43.991625+0000", 2024-02-09T17:50:22.674 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012285700000000001 2024-02-09T17:50:22.674 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:22.674 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:50:22.674 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:22.674 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:22.674 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:50:22.674 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:50:22.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:22 smithi179 ceph-mon[43975]: 1.10 scrub starts 2024-02-09T17:50:22.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:22 smithi179 ceph-mon[43975]: pgmap v1671: 47 pgs: 4 creating+peering, 9 unknown, 34 active+clean; 577 KiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:50:22.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:22 smithi179 ceph-mon[43975]: 1.10 scrub ok 2024-02-09T17:50:22.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:22 smithi179 ceph-mon[43975]: 1.1d scrub starts 2024-02-09T17:50:22.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:22 smithi179 ceph-mon[43975]: 1.1d scrub ok 2024-02-09T17:50:22.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:22 smithi162 ceph-mon[44304]: 1.10 scrub starts 2024-02-09T17:50:22.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:22 smithi162 ceph-mon[44304]: pgmap v1671: 47 pgs: 4 creating+peering, 9 unknown, 34 active+clean; 577 KiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:50:22.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:22 smithi162 ceph-mon[44304]: 1.10 scrub ok 2024-02-09T17:50:22.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:22 smithi162 ceph-mon[44304]: 1.1d scrub starts 2024-02-09T17:50:22.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:22 smithi162 ceph-mon[44304]: 1.1d scrub ok 2024-02-09T17:50:22.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:22 smithi099 ceph-mon[39420]: 1.10 scrub starts 2024-02-09T17:50:22.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:22 smithi099 ceph-mon[39420]: pgmap v1671: 47 pgs: 4 creating+peering, 9 unknown, 34 active+clean; 577 KiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:50:22.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:22 smithi099 ceph-mon[39420]: 1.10 scrub ok 2024-02-09T17:50:22.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:22 smithi099 ceph-mon[39420]: 1.1d scrub starts 2024-02-09T17:50:22.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:22 smithi099 ceph-mon[39420]: 1.1d scrub ok 2024-02-09T17:50:22.848 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:50:22.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:22.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:50:22.951 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:50:22.982 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:23.001 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:50:23.004 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:23.004 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:50:23.070 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:50:23.106 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:50:23.134 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:23.212 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-09T17:50:23.332 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:50:23.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:23.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:50:23.438 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:50:23.469 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:23.479 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:50:23.489 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:50:23.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:23.492 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:50:23.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:23 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:50:23.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:23 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:50:23.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:23 smithi099 ceph-mon[39420]: 1.2 scrub starts 2024-02-09T17:50:23.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:23 smithi099 ceph-mon[39420]: 1.2 scrub ok 2024-02-09T17:50:23.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:23 smithi099 ceph-mon[39420]: 1.b scrub starts 2024-02-09T17:50:23.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:23 smithi099 ceph-mon[39420]: osdmap e664: 12 total, 12 up, 12 in 2024-02-09T17:50:23.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:23 smithi099 ceph-mon[39420]: 1.b scrub ok 2024-02-09T17:50:23.593 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:50:23.618 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:23 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:50:23.618 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:23 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:50:23.618 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:23 smithi162 ceph-mon[44304]: 1.2 scrub starts 2024-02-09T17:50:23.618 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:23 smithi162 ceph-mon[44304]: 1.2 scrub ok 2024-02-09T17:50:23.618 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:23 smithi162 ceph-mon[44304]: 1.b scrub starts 2024-02-09T17:50:23.618 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:23 smithi162 ceph-mon[44304]: osdmap e664: 12 total, 12 up, 12 in 2024-02-09T17:50:23.618 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:23 smithi162 ceph-mon[44304]: 1.b scrub ok 2024-02-09T17:50:23.620 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:23.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:23 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:50:23.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:23 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:50:23.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:23 smithi179 ceph-mon[43975]: 1.2 scrub starts 2024-02-09T17:50:23.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:23 smithi179 ceph-mon[43975]: 1.2 scrub ok 2024-02-09T17:50:23.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:23 smithi179 ceph-mon[43975]: 1.b scrub starts 2024-02-09T17:50:23.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:23 smithi179 ceph-mon[43975]: osdmap e664: 12 total, 12 up, 12 in 2024-02-09T17:50:23.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:23 smithi179 ceph-mon[43975]: 1.b scrub ok 2024-02-09T17:50:23.742 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:50:23.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:23.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:50:23.846 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:50:23.873 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:24.322 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:50:24.323 INFO:teuthology.orchestra.run.smithi099.stderr:enabled application 'rados' on pool 'unique_pool_5' 2024-02-09T17:50:24.323 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is unset 2024-02-09T17:50:24.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:24.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:50:24.427 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:50:24.456 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:24.478 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:50:24.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:24.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:50:24.591 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:50:24.617 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:24 smithi099 ceph-mon[39420]: pgmap v1673: 47 pgs: 3 creating+peering, 44 active+clean; 577 KiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:50:24.617 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:24 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1346879268' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:50:24.617 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:24 smithi099 ceph-mon[39420]: 1.12 scrub starts 2024-02-09T17:50:24.617 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:24 smithi099 ceph-mon[39420]: 1.12 scrub ok 2024-02-09T17:50:24.618 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:24 smithi099 ceph-mon[39420]: 1.19 scrub starts 2024-02-09T17:50:24.618 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:24 smithi099 ceph-mon[39420]: 1.19 scrub ok 2024-02-09T17:50:24.618 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:24 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/430148357' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-09T17:50:24.621 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:24.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:24 smithi179 ceph-mon[43975]: pgmap v1673: 47 pgs: 3 creating+peering, 44 active+clean; 577 KiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:50:24.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:24 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1346879268' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:50:24.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:24 smithi179 ceph-mon[43975]: 1.12 scrub starts 2024-02-09T17:50:24.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:24 smithi179 ceph-mon[43975]: 1.12 scrub ok 2024-02-09T17:50:24.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:24 smithi179 ceph-mon[43975]: 1.19 scrub starts 2024-02-09T17:50:24.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:24 smithi179 ceph-mon[43975]: 1.19 scrub ok 2024-02-09T17:50:24.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:24 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/430148357' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-09T17:50:24.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:24 smithi162 ceph-mon[44304]: pgmap v1673: 47 pgs: 3 creating+peering, 44 active+clean; 577 KiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:50:24.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:24 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1346879268' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:50:24.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:24 smithi162 ceph-mon[44304]: 1.12 scrub starts 2024-02-09T17:50:24.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:24 smithi162 ceph-mon[44304]: 1.12 scrub ok 2024-02-09T17:50:24.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:24 smithi162 ceph-mon[44304]: 1.19 scrub starts 2024-02-09T17:50:24.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:24 smithi162 ceph-mon[44304]: 1.19 scrub ok 2024-02-09T17:50:24.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:24 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/430148357' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-09T17:50:24.682 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] 2024-02-09T17:50:24.682 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:50:24.682 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-02-09T17:50:24.683 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-09T17:50:24.744 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:50:24.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:24.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:50:24.848 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:50:24.874 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:24.973 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:50:24.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:24.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:50:25.076 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:50:25.109 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:25.129 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:50:25.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:25.133 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:50:25.233 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:50:25.264 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:25.284 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:50:25.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:25.287 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:50:25.345 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:50:25.345 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-09T17:50:25.345 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-02-09T17:50:25.345 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:50:25.387 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:50:25.416 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:25.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:25 smithi179 ceph-mon[43975]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:50:25.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:25 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1346879268' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:50:25.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:25 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/430148357' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-09T17:50:25.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:25 smithi179 ceph-mon[43975]: osdmap e665: 12 total, 12 up, 12 in 2024-02-09T17:50:25.441 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:50:25.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:25.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:50:25.545 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:50:25.576 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:25.596 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:50:25.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:25.599 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:50:25.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:25 smithi162 ceph-mon[44304]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:50:25.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:25 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1346879268' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:50:25.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:25 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/430148357' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-09T17:50:25.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:25 smithi162 ceph-mon[44304]: osdmap e665: 12 total, 12 up, 12 in 2024-02-09T17:50:25.699 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:50:25.728 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:25.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:25 smithi099 ceph-mon[39420]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:50:25.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:25 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1346879268' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:50:25.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:25 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/430148357' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-09T17:50:25.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:25 smithi099 ceph-mon[39420]: osdmap e665: 12 total, 12 up, 12 in 2024-02-09T17:50:26.058 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-09T17:50:26.078 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:50:26.081 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:26.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:50:26.111 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:50:26.182 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:50:26.210 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:26.236 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:50:26.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:26.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:50:26.316 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:26 smithi162 ceph-mon[44304]: pgmap v1675: 47 pgs: 47 active+clean; 577 KiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:50:26.316 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:26 smithi162 ceph-mon[44304]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-09T17:50:26.316 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:26 smithi162 ceph-mon[44304]: Cluster is now healthy 2024-02-09T17:50:26.340 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:50:26.350 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:50:26.358 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:26 smithi099 ceph-mon[39420]: pgmap v1675: 47 pgs: 47 active+clean; 577 KiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:50:26.359 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:26 smithi099 ceph-mon[39420]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-09T17:50:26.359 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:26 smithi099 ceph-mon[39420]: Cluster is now healthy 2024-02-09T17:50:26.372 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:26.441 DEBUG:teuthology.orchestra.run.smithi179:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_5 bench 90 write' 2024-02-09T17:50:26.444 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-02-09T17:50:26.444 INFO:teuthology.orchestra.run:waiting for 3000 2024-02-09T17:50:26.500 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:hints = 1 2024-02-09T17:50:26.501 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-02-09T17:50:26.501 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Object prefix: benchmark_data_smithi179_135804 2024-02-09T17:50:26.501 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-09T17:50:26.501 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 0 0 0 0 0 0 - 0 2024-02-09T17:50:26.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:26 smithi179 ceph-mon[43975]: pgmap v1675: 47 pgs: 47 active+clean; 577 KiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:50:26.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:26 smithi179 ceph-mon[43975]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-09T17:50:26.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:26 smithi179 ceph-mon[43975]: Cluster is now healthy 2024-02-09T17:50:27.159 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:50:27.164 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:27.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:50:27.256 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:50:27.256 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:50:27.256 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:50:27.256 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:50:27.256 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:50:27.257 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:50:27.265 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:50:27.291 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:50:27.302 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:27.329 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:50:27.369 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:50:27.408 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:50:27.446 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:50:27.483 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:50:27.487 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:50:27.488 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:50:27.488 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:50:27.488 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:50:27.488 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:50:27.488 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:50:27.488 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:50:27.488 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:50:27.488 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:50:27.488 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:50:27.488 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:50:27.488 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:50:27.501 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 1 16 4104 4088 255.477 255.5 0.00303015 0.00390148 2024-02-09T17:50:27.864 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:27 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3490391700' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:50:27.864 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:27 smithi099 ceph-mon[39420]: pgmap v1676: 47 pgs: 47 active+clean; 11 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s wr, 23 op/s 2024-02-09T17:50:28.111 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set noscrub 2024-02-09T17:50:28.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:27 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3490391700' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:50:28.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:27 smithi179 ceph-mon[43975]: pgmap v1676: 47 pgs: 47 active+clean; 11 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s wr, 23 op/s 2024-02-09T17:50:28.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:27 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3490391700' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:50:28.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:27 smithi162 ceph-mon[44304]: pgmap v1676: 47 pgs: 47 active+clean; 11 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s wr, 23 op/s 2024-02-09T17:50:28.501 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 2 16 6647 6631 207.2 158.938 0.00219257 0.00481748 2024-02-09T17:50:28.660 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:50:28.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:28.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:50:28.771 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:50:28.830 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:28.857 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:50:28.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:28.861 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:50:28.963 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:50:29.003 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:29.034 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:50:29.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:29.038 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:50:29.139 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:50:29.176 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:29.203 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:50:29.203 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:50:29.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:29.207 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:50:29.308 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:50:29.352 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:29.383 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:50:29.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:29.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:50:29.488 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:50:29.502 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 3 16 10039 10023 208.72 212 0.00486735 0.0047845 2024-02-09T17:50:29.505 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:50:29.505 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:50:29.526 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:29.556 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:50:29.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:29.560 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:50:29.581 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:50:29.581 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:50:29.581 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 665, 2024-02-09T17:50:29.581 INFO:teuthology.orchestra.run.smithi099.stdout: "fsid": "a69c870a-c76e-11ee-95b7-87774f69a715", 2024-02-09T17:50:29.581 INFO:teuthology.orchestra.run.smithi099.stdout: "created": "2024-02-09T17:15:26.113606+0000", 2024-02-09T17:50:29.581 INFO:teuthology.orchestra.run.smithi099.stdout: "modified": "2024-02-09T17:50:24.310359+0000", 2024-02-09T17:50:29.581 INFO:teuthology.orchestra.run.smithi099.stdout: "last_up_change": "2024-02-09T17:28:17.711044+0000", 2024-02-09T17:50:29.581 INFO:teuthology.orchestra.run.smithi099.stdout: "last_in_change": "2024-02-09T17:43:24.138763+0000", 2024-02-09T17:50:29.581 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-09T17:50:29.582 INFO:teuthology.orchestra.run.smithi099.stdout: "flags_num": 5799936, 2024-02-09T17:50:29.582 INFO:teuthology.orchestra.run.smithi099.stdout: "flags_set": [ 2024-02-09T17:50:29.582 INFO:teuthology.orchestra.run.smithi099.stdout: "pglog_hardlimit", 2024-02-09T17:50:29.582 INFO:teuthology.orchestra.run.smithi099.stdout: "purged_snapdirs", 2024-02-09T17:50:29.582 INFO:teuthology.orchestra.run.smithi099.stdout: "recovery_deletes", 2024-02-09T17:50:29.582 INFO:teuthology.orchestra.run.smithi099.stdout: "sortbitwise" 2024-02-09T17:50:29.582 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T17:50:29.582 INFO:teuthology.orchestra.run.smithi099.stdout: "crush_version": 27, 2024-02-09T17:50:29.582 INFO:teuthology.orchestra.run.smithi099.stdout: "full_ratio": 0.94999998807907104, 2024-02-09T17:50:29.582 INFO:teuthology.orchestra.run.smithi099.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-09T17:50:29.583 INFO:teuthology.orchestra.run.smithi099.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-09T17:50:29.583 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_snapshot": "", 2024-02-09T17:50:29.583 INFO:teuthology.orchestra.run.smithi099.stdout: "pool_max": 7, 2024-02-09T17:50:29.583 INFO:teuthology.orchestra.run.smithi099.stdout: "max_osd": 12, 2024-02-09T17:50:29.583 INFO:teuthology.orchestra.run.smithi099.stdout: "require_min_compat_client": "luminous", 2024-02-09T17:50:29.583 INFO:teuthology.orchestra.run.smithi099.stdout: "min_compat_client": "jewel", 2024-02-09T17:50:29.584 INFO:teuthology.orchestra.run.smithi099.stdout: "require_osd_release": "squid", 2024-02-09T17:50:29.584 INFO:teuthology.orchestra.run.smithi099.stdout: "allow_crimson": false, 2024-02-09T17:50:29.584 INFO:teuthology.orchestra.run.smithi099.stdout: "pools": [ 2024-02-09T17:50:29.584 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.584 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 1, 2024-02-09T17:50:29.584 INFO:teuthology.orchestra.run.smithi099.stdout: "pool_name": ".mgr", 2024-02-09T17:50:29.584 INFO:teuthology.orchestra.run.smithi099.stdout: "create_time": "2024-02-09T17:18:56.552286+0000", 2024-02-09T17:50:29.584 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 1, 2024-02-09T17:50:29.584 INFO:teuthology.orchestra.run.smithi099.stdout: "flags_names": "hashpspool", 2024-02-09T17:50:29.584 INFO:teuthology.orchestra.run.smithi099.stdout: "type": 1, 2024-02-09T17:50:29.584 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-02-09T17:50:29.584 INFO:teuthology.orchestra.run.smithi099.stdout: "min_size": 2, 2024-02-09T17:50:29.584 INFO:teuthology.orchestra.run.smithi099.stdout: "crush_rule": 0, 2024-02-09T17:50:29.584 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_count": 0, 2024-02-09T17:50:29.585 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_target": 0, 2024-02-09T17:50:29.585 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_barrier": 0, 2024-02-09T17:50:29.585 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-09T17:50:29.585 INFO:teuthology.orchestra.run.smithi099.stdout: "object_hash": 2, 2024-02-09T17:50:29.585 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_autoscale_mode": "off", 2024-02-09T17:50:29.585 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num": 31, 2024-02-09T17:50:29.585 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_placement_num": 31, 2024-02-09T17:50:29.585 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_placement_num_target": 31, 2024-02-09T17:50:29.585 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num_target": 31, 2024-02-09T17:50:29.585 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num_pending": 31, 2024-02-09T17:50:29.585 INFO:teuthology.orchestra.run.smithi099.stdout: "last_pg_merge_meta": { 2024-02-09T17:50:29.585 INFO:teuthology.orchestra.run.smithi099.stdout: "source_pgid": "0.0", 2024-02-09T17:50:29.585 INFO:teuthology.orchestra.run.smithi099.stdout: "ready_epoch": 0, 2024-02-09T17:50:29.585 INFO:teuthology.orchestra.run.smithi099.stdout: "last_epoch_started": 0, 2024-02-09T17:50:29.586 INFO:teuthology.orchestra.run.smithi099.stdout: "last_epoch_clean": 0, 2024-02-09T17:50:29.586 INFO:teuthology.orchestra.run.smithi099.stdout: "source_version": "0'0", 2024-02-09T17:50:29.586 INFO:teuthology.orchestra.run.smithi099.stdout: "target_version": "0'0" 2024-02-09T17:50:29.586 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.586 INFO:teuthology.orchestra.run.smithi099.stdout: "last_change": "544", 2024-02-09T17:50:29.586 INFO:teuthology.orchestra.run.smithi099.stdout: "last_force_op_resend": "0", 2024-02-09T17:50:29.586 INFO:teuthology.orchestra.run.smithi099.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-09T17:50:29.586 INFO:teuthology.orchestra.run.smithi099.stdout: "last_force_op_resend_preluminous": "514", 2024-02-09T17:50:29.586 INFO:teuthology.orchestra.run.smithi099.stdout: "auid": 0, 2024-02-09T17:50:29.586 INFO:teuthology.orchestra.run.smithi099.stdout: "snap_mode": "selfmanaged", 2024-02-09T17:50:29.586 INFO:teuthology.orchestra.run.smithi099.stdout: "snap_seq": 0, 2024-02-09T17:50:29.586 INFO:teuthology.orchestra.run.smithi099.stdout: "snap_epoch": 0, 2024-02-09T17:50:29.586 INFO:teuthology.orchestra.run.smithi099.stdout: "pool_snaps": [], 2024-02-09T17:50:29.587 INFO:teuthology.orchestra.run.smithi099.stdout: "removed_snaps": "[]", 2024-02-09T17:50:29.587 INFO:teuthology.orchestra.run.smithi099.stdout: "quota_max_bytes": 0, 2024-02-09T17:50:29.587 INFO:teuthology.orchestra.run.smithi099.stdout: "quota_max_objects": 0, 2024-02-09T17:50:29.587 INFO:teuthology.orchestra.run.smithi099.stdout: "tiers": [], 2024-02-09T17:50:29.587 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_of": -1, 2024-02-09T17:50:29.587 INFO:teuthology.orchestra.run.smithi099.stdout: "read_tier": -1, 2024-02-09T17:50:29.587 INFO:teuthology.orchestra.run.smithi099.stdout: "write_tier": -1, 2024-02-09T17:50:29.587 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_mode": "none", 2024-02-09T17:50:29.587 INFO:teuthology.orchestra.run.smithi099.stdout: "target_max_bytes": 0, 2024-02-09T17:50:29.587 INFO:teuthology.orchestra.run.smithi099.stdout: "target_max_objects": 0, 2024-02-09T17:50:29.587 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-09T17:50:29.587 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-09T17:50:29.587 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-09T17:50:29.587 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_min_flush_age": 0, 2024-02-09T17:50:29.588 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_min_evict_age": 0, 2024-02-09T17:50:29.588 INFO:teuthology.orchestra.run.smithi099.stdout: "erasure_code_profile": "", 2024-02-09T17:50:29.588 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_params": { 2024-02-09T17:50:29.588 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "none" 2024-02-09T17:50:29.588 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.588 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_period": 0, 2024-02-09T17:50:29.588 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_count": 0, 2024-02-09T17:50:29.588 INFO:teuthology.orchestra.run.smithi099.stdout: "use_gmt_hitset": true, 2024-02-09T17:50:29.588 INFO:teuthology.orchestra.run.smithi099.stdout: "min_read_recency_for_promote": 0, 2024-02-09T17:50:29.588 INFO:teuthology.orchestra.run.smithi099.stdout: "min_write_recency_for_promote": 0, 2024-02-09T17:50:29.588 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_grade_decay_rate": 0, 2024-02-09T17:50:29.588 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_search_last_n": 0, 2024-02-09T17:50:29.588 INFO:teuthology.orchestra.run.smithi099.stdout: "grade_table": [], 2024-02-09T17:50:29.589 INFO:teuthology.orchestra.run.smithi099.stdout: "stripe_width": 0, 2024-02-09T17:50:29.589 INFO:teuthology.orchestra.run.smithi099.stdout: "expected_num_objects": 0, 2024-02-09T17:50:29.589 INFO:teuthology.orchestra.run.smithi099.stdout: "fast_read": false, 2024-02-09T17:50:29.589 INFO:teuthology.orchestra.run.smithi099.stdout: "options": { 2024-02-09T17:50:29.589 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num_max": 32, 2024-02-09T17:50:29.589 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num_min": 1 2024-02-09T17:50:29.589 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.589 INFO:teuthology.orchestra.run.smithi099.stdout: "application_metadata": { 2024-02-09T17:50:29.589 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": {} 2024-02-09T17:50:29.589 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.589 INFO:teuthology.orchestra.run.smithi099.stdout: "read_balance": { 2024-02-09T17:50:29.589 INFO:teuthology.orchestra.run.smithi099.stdout: "score_type": "Fair distribution", 2024-02-09T17:50:29.589 INFO:teuthology.orchestra.run.smithi099.stdout: "score_acting": 5.9899997711181641, 2024-02-09T17:50:29.589 INFO:teuthology.orchestra.run.smithi099.stdout: "score_stable": 1.7799999713897705, 2024-02-09T17:50:29.590 INFO:teuthology.orchestra.run.smithi099.stdout: "optimal_score": 1.0900000333786011, 2024-02-09T17:50:29.590 INFO:teuthology.orchestra.run.smithi099.stdout: "raw_score_acting": 6.5300002098083496, 2024-02-09T17:50:29.590 INFO:teuthology.orchestra.run.smithi099.stdout: "raw_score_stable": 1.940000057220459, 2024-02-09T17:50:29.590 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity_weighted": 0.78079599142074585, 2024-02-09T17:50:29.590 INFO:teuthology.orchestra.run.smithi099.stdout: "average_primary_affinity": 0.85000002384185791, 2024-02-09T17:50:29.590 INFO:teuthology.orchestra.run.smithi099.stdout: "average_primary_affinity_weighted": 0.77999997138977051, 2024-02-09T17:50:29.590 INFO:teuthology.orchestra.run.smithi099.stdout: "error_message": "pool 1 has acting primaries on OSD(s) with primary affinity 0, read balance score is not accurate" 2024-02-09T17:50:29.590 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.590 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.590 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.590 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 7, 2024-02-09T17:50:29.590 INFO:teuthology.orchestra.run.smithi099.stdout: "pool_name": "unique_pool_5", 2024-02-09T17:50:29.590 INFO:teuthology.orchestra.run.smithi099.stdout: "create_time": "2024-02-09T17:50:17.290198+0000", 2024-02-09T17:50:29.591 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 1, 2024-02-09T17:50:29.591 INFO:teuthology.orchestra.run.smithi099.stdout: "flags_names": "hashpspool", 2024-02-09T17:50:29.591 INFO:teuthology.orchestra.run.smithi099.stdout: "type": 1, 2024-02-09T17:50:29.591 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-02-09T17:50:29.591 INFO:teuthology.orchestra.run.smithi099.stdout: "min_size": 2, 2024-02-09T17:50:29.591 INFO:teuthology.orchestra.run.smithi099.stdout: "crush_rule": 0, 2024-02-09T17:50:29.591 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_count": 0, 2024-02-09T17:50:29.591 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_target": 0, 2024-02-09T17:50:29.591 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_barrier": 0, 2024-02-09T17:50:29.591 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-09T17:50:29.591 INFO:teuthology.orchestra.run.smithi099.stdout: "object_hash": 2, 2024-02-09T17:50:29.591 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_autoscale_mode": "off", 2024-02-09T17:50:29.591 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num": 16, 2024-02-09T17:50:29.591 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_placement_num": 16, 2024-02-09T17:50:29.592 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_placement_num_target": 16, 2024-02-09T17:50:29.592 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num_target": 16, 2024-02-09T17:50:29.592 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num_pending": 16, 2024-02-09T17:50:29.592 INFO:teuthology.orchestra.run.smithi099.stdout: "last_pg_merge_meta": { 2024-02-09T17:50:29.592 INFO:teuthology.orchestra.run.smithi099.stdout: "source_pgid": "0.0", 2024-02-09T17:50:29.592 INFO:teuthology.orchestra.run.smithi099.stdout: "ready_epoch": 0, 2024-02-09T17:50:29.592 INFO:teuthology.orchestra.run.smithi099.stdout: "last_epoch_started": 0, 2024-02-09T17:50:29.592 INFO:teuthology.orchestra.run.smithi099.stdout: "last_epoch_clean": 0, 2024-02-09T17:50:29.592 INFO:teuthology.orchestra.run.smithi099.stdout: "source_version": "0'0", 2024-02-09T17:50:29.592 INFO:teuthology.orchestra.run.smithi099.stdout: "target_version": "0'0" 2024-02-09T17:50:29.592 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.592 INFO:teuthology.orchestra.run.smithi099.stdout: "last_change": "665", 2024-02-09T17:50:29.593 INFO:teuthology.orchestra.run.smithi099.stdout: "last_force_op_resend": "0", 2024-02-09T17:50:29.593 INFO:teuthology.orchestra.run.smithi099.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-09T17:50:29.593 INFO:teuthology.orchestra.run.smithi099.stdout: "last_force_op_resend_preluminous": "0", 2024-02-09T17:50:29.593 INFO:teuthology.orchestra.run.smithi099.stdout: "auid": 0, 2024-02-09T17:50:29.593 INFO:teuthology.orchestra.run.smithi099.stdout: "snap_mode": "selfmanaged", 2024-02-09T17:50:29.593 INFO:teuthology.orchestra.run.smithi099.stdout: "snap_seq": 0, 2024-02-09T17:50:29.593 INFO:teuthology.orchestra.run.smithi099.stdout: "snap_epoch": 0, 2024-02-09T17:50:29.593 INFO:teuthology.orchestra.run.smithi099.stdout: "pool_snaps": [], 2024-02-09T17:50:29.593 INFO:teuthology.orchestra.run.smithi099.stdout: "removed_snaps": "[]", 2024-02-09T17:50:29.593 INFO:teuthology.orchestra.run.smithi099.stdout: "quota_max_bytes": 0, 2024-02-09T17:50:29.593 INFO:teuthology.orchestra.run.smithi099.stdout: "quota_max_objects": 0, 2024-02-09T17:50:29.593 INFO:teuthology.orchestra.run.smithi099.stdout: "tiers": [], 2024-02-09T17:50:29.593 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_of": -1, 2024-02-09T17:50:29.593 INFO:teuthology.orchestra.run.smithi099.stdout: "read_tier": -1, 2024-02-09T17:50:29.594 INFO:teuthology.orchestra.run.smithi099.stdout: "write_tier": -1, 2024-02-09T17:50:29.594 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_mode": "none", 2024-02-09T17:50:29.594 INFO:teuthology.orchestra.run.smithi099.stdout: "target_max_bytes": 0, 2024-02-09T17:50:29.594 INFO:teuthology.orchestra.run.smithi099.stdout: "target_max_objects": 0, 2024-02-09T17:50:29.594 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-09T17:50:29.594 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-09T17:50:29.594 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-09T17:50:29.594 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_min_flush_age": 0, 2024-02-09T17:50:29.594 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_min_evict_age": 0, 2024-02-09T17:50:29.594 INFO:teuthology.orchestra.run.smithi099.stdout: "erasure_code_profile": "", 2024-02-09T17:50:29.594 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_params": { 2024-02-09T17:50:29.594 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "none" 2024-02-09T17:50:29.594 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.595 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_period": 0, 2024-02-09T17:50:29.595 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_count": 0, 2024-02-09T17:50:29.595 INFO:teuthology.orchestra.run.smithi099.stdout: "use_gmt_hitset": true, 2024-02-09T17:50:29.595 INFO:teuthology.orchestra.run.smithi099.stdout: "min_read_recency_for_promote": 0, 2024-02-09T17:50:29.595 INFO:teuthology.orchestra.run.smithi099.stdout: "min_write_recency_for_promote": 0, 2024-02-09T17:50:29.595 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_grade_decay_rate": 0, 2024-02-09T17:50:29.595 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_search_last_n": 0, 2024-02-09T17:50:29.595 INFO:teuthology.orchestra.run.smithi099.stdout: "grade_table": [], 2024-02-09T17:50:29.595 INFO:teuthology.orchestra.run.smithi099.stdout: "stripe_width": 0, 2024-02-09T17:50:29.595 INFO:teuthology.orchestra.run.smithi099.stdout: "expected_num_objects": 0, 2024-02-09T17:50:29.595 INFO:teuthology.orchestra.run.smithi099.stdout: "fast_read": false, 2024-02-09T17:50:29.595 INFO:teuthology.orchestra.run.smithi099.stdout: "options": {}, 2024-02-09T17:50:29.595 INFO:teuthology.orchestra.run.smithi099.stdout: "application_metadata": { 2024-02-09T17:50:29.595 INFO:teuthology.orchestra.run.smithi099.stdout: "rados": {} 2024-02-09T17:50:29.596 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.596 INFO:teuthology.orchestra.run.smithi099.stdout: "read_balance": { 2024-02-09T17:50:29.596 INFO:teuthology.orchestra.run.smithi099.stdout: "score_type": "Fair distribution", 2024-02-09T17:50:29.596 INFO:teuthology.orchestra.run.smithi099.stdout: "score_acting": 2.059999942779541, 2024-02-09T17:50:29.596 INFO:teuthology.orchestra.run.smithi099.stdout: "score_stable": 2.059999942779541, 2024-02-09T17:50:29.596 INFO:teuthology.orchestra.run.smithi099.stdout: "optimal_score": 1.0900000333786011, 2024-02-09T17:50:29.596 INFO:teuthology.orchestra.run.smithi099.stdout: "raw_score_acting": 2.25, 2024-02-09T17:50:29.596 INFO:teuthology.orchestra.run.smithi099.stdout: "raw_score_stable": 2.25, 2024-02-09T17:50:29.596 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity_weighted": 0.78079599142074585, 2024-02-09T17:50:29.596 INFO:teuthology.orchestra.run.smithi099.stdout: "average_primary_affinity": 0.85000002384185791, 2024-02-09T17:50:29.596 INFO:teuthology.orchestra.run.smithi099.stdout: "average_primary_affinity_weighted": 0.77999997138977051 2024-02-09T17:50:29.596 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.596 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.596 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T17:50:29.597 INFO:teuthology.orchestra.run.smithi099.stdout: "osds": [ 2024-02-09T17:50:29.597 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.597 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 0, 2024-02-09T17:50:29.597 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "21739aa7-e278-45da-9a3f-867385f01baf", 2024-02-09T17:50:29.597 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:50:29.597 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:50:29.597 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:50:29.597 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 0.609619140625, 2024-02-09T17:50:29.597 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:50:29.597 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:50:29.597 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 8, 2024-02-09T17:50:29.597 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 661, 2024-02-09T17:50:29.597 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:50:29.597 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:50:29.598 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:50:29.598 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:50:29.598 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.598 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:50:29.598 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6802", 2024-02-09T17:50:29.598 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T17:50:29.598 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.598 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.598 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:50:29.598 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6803", 2024-02-09T17:50:29.598 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T17:50:29.598 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.598 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.598 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.599 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:50:29.599 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:50:29.599 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.599 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:50:29.599 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6804", 2024-02-09T17:50:29.599 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T17:50:29.599 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.599 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.599 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:50:29.599 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6805", 2024-02-09T17:50:29.599 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T17:50:29.599 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.600 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.600 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.600 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:50:29.600 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:50:29.600 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.600 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:50:29.600 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6808", 2024-02-09T17:50:29.600 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T17:50:29.600 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.600 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.600 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:50:29.600 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6809", 2024-02-09T17:50:29.601 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T17:50:29.601 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.601 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.601 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.601 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:50:29.601 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:50:29.601 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.601 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:50:29.601 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6806", 2024-02-09T17:50:29.601 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T17:50:29.601 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.601 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.601 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:50:29.601 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6807", 2024-02-09T17:50:29.601 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T17:50:29.602 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.602 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.602 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.602 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.99:6803/3859868923", 2024-02-09T17:50:29.602 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.99:6805/3859868923", 2024-02-09T17:50:29.602 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.99:6809/3859868923", 2024-02-09T17:50:29.602 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.99:6807/3859868923", 2024-02-09T17:50:29.602 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:50:29.602 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:50:29.602 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:50:29.602 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.602 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.602 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.603 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 1, 2024-02-09T17:50:29.603 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "8eab347c-9065-4de1-93ae-e2eb7e11cab7", 2024-02-09T17:50:29.603 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:50:29.603 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:50:29.603 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:50:29.603 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T17:50:29.603 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:50:29.603 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:50:29.603 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 13, 2024-02-09T17:50:29.604 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 661, 2024-02-09T17:50:29.604 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:50:29.604 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:50:29.604 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:50:29.604 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:50:29.604 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.604 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:50:29.604 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6810", 2024-02-09T17:50:29.604 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T17:50:29.604 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.604 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.604 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:50:29.605 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6811", 2024-02-09T17:50:29.605 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T17:50:29.605 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.605 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.605 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.605 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:50:29.605 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:50:29.605 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.605 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:50:29.605 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6812", 2024-02-09T17:50:29.605 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T17:50:29.605 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.605 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.605 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:50:29.606 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6813", 2024-02-09T17:50:29.606 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T17:50:29.606 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.606 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.606 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.606 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:50:29.606 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:50:29.606 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.606 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:50:29.606 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6816", 2024-02-09T17:50:29.606 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T17:50:29.606 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.606 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.607 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:50:29.607 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6817", 2024-02-09T17:50:29.607 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T17:50:29.607 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.607 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.607 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.607 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:50:29.607 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:50:29.607 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.607 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:50:29.607 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6814", 2024-02-09T17:50:29.607 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T17:50:29.607 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.607 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.607 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:50:29.608 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6815", 2024-02-09T17:50:29.608 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T17:50:29.608 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.608 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.608 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.608 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.99:6811/3868328452", 2024-02-09T17:50:29.608 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.99:6813/3868328452", 2024-02-09T17:50:29.608 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.99:6817/3868328452", 2024-02-09T17:50:29.608 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.99:6815/3868328452", 2024-02-09T17:50:29.608 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:50:29.608 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:50:29.608 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:50:29.608 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.609 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.609 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.609 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 2, 2024-02-09T17:50:29.609 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "575f4257-7e34-485e-963c-2607b69e1220", 2024-02-09T17:50:29.609 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:50:29.609 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:50:29.609 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:50:29.609 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 0.3601837158203125, 2024-02-09T17:50:29.609 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:50:29.609 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:50:29.609 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 18, 2024-02-09T17:50:29.609 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 661, 2024-02-09T17:50:29.609 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:50:29.609 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:50:29.609 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:50:29.610 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:50:29.610 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.610 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:50:29.610 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6818", 2024-02-09T17:50:29.610 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T17:50:29.610 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.610 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.610 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:50:29.610 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6819", 2024-02-09T17:50:29.610 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T17:50:29.610 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.610 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.610 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.610 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:50:29.610 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:50:29.611 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.611 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:50:29.611 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6820", 2024-02-09T17:50:29.611 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T17:50:29.611 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.611 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.611 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:50:29.611 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6821", 2024-02-09T17:50:29.611 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T17:50:29.611 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.611 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.611 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.611 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:50:29.611 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:50:29.612 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.612 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:50:29.612 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6824", 2024-02-09T17:50:29.612 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T17:50:29.612 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.612 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.612 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:50:29.612 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6825", 2024-02-09T17:50:29.612 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T17:50:29.612 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.612 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.612 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.612 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:50:29.612 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:50:29.613 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.613 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:50:29.613 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6822", 2024-02-09T17:50:29.613 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T17:50:29.613 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.613 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.613 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:50:29.613 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6823", 2024-02-09T17:50:29.613 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T17:50:29.613 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.613 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.613 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.613 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.99:6819/537506237", 2024-02-09T17:50:29.613 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.99:6821/537506237", 2024-02-09T17:50:29.613 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.99:6825/537506237", 2024-02-09T17:50:29.614 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.99:6823/537506237", 2024-02-09T17:50:29.614 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:50:29.614 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:50:29.614 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:50:29.614 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.614 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.614 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.614 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 3, 2024-02-09T17:50:29.614 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "1a520bcf-81bb-4f1d-80fb-11e98008e429", 2024-02-09T17:50:29.614 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:50:29.614 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:50:29.614 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:50:29.614 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 0.8014984130859375, 2024-02-09T17:50:29.614 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:50:29.615 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:50:29.615 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 27, 2024-02-09T17:50:29.615 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 661, 2024-02-09T17:50:29.615 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:50:29.615 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:50:29.615 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:50:29.615 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:50:29.615 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.615 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:50:29.615 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6826", 2024-02-09T17:50:29.615 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 762784793 2024-02-09T17:50:29.615 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.615 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.615 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:50:29.615 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6827", 2024-02-09T17:50:29.616 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 762784793 2024-02-09T17:50:29.616 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.616 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.616 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.616 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:50:29.616 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:50:29.616 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.616 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:50:29.616 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6828", 2024-02-09T17:50:29.616 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 762784793 2024-02-09T17:50:29.616 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.616 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.616 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:50:29.616 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6829", 2024-02-09T17:50:29.617 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 762784793 2024-02-09T17:50:29.617 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.617 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.617 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.617 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:50:29.617 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:50:29.617 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.617 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:50:29.617 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6832", 2024-02-09T17:50:29.617 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 762784793 2024-02-09T17:50:29.617 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.617 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.617 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:50:29.617 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6833", 2024-02-09T17:50:29.618 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 762784793 2024-02-09T17:50:29.618 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.618 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.618 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.618 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:50:29.618 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:50:29.618 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.618 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:50:29.618 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6830", 2024-02-09T17:50:29.618 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 762784793 2024-02-09T17:50:29.618 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.618 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.618 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:50:29.619 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6831", 2024-02-09T17:50:29.619 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 762784793 2024-02-09T17:50:29.619 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.619 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.619 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.619 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.99:6827/762784793", 2024-02-09T17:50:29.619 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.99:6829/762784793", 2024-02-09T17:50:29.619 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.99:6833/762784793", 2024-02-09T17:50:29.619 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.99:6831/762784793", 2024-02-09T17:50:29.619 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:50:29.619 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:50:29.619 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:50:29.619 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.619 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.619 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.620 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 4, 2024-02-09T17:50:29.620 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "4617c2fe-c9d8-40cd-887b-303e3215bdf8", 2024-02-09T17:50:29.620 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:50:29.620 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:50:29.620 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:50:29.620 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T17:50:29.620 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:50:29.620 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:50:29.620 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 32, 2024-02-09T17:50:29.620 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 661, 2024-02-09T17:50:29.620 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:50:29.620 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:50:29.620 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:50:29.620 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:50:29.621 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.621 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:50:29.621 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6800", 2024-02-09T17:50:29.621 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T17:50:29.621 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.621 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.621 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:50:29.621 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6801", 2024-02-09T17:50:29.621 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T17:50:29.621 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.621 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.621 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.621 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:50:29.621 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:50:29.621 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.622 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:50:29.622 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6802", 2024-02-09T17:50:29.622 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T17:50:29.622 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.622 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.622 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:50:29.622 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6803", 2024-02-09T17:50:29.622 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T17:50:29.622 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.622 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.622 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.622 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:50:29.622 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:50:29.622 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.622 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:50:29.623 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6806", 2024-02-09T17:50:29.623 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T17:50:29.623 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.623 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.623 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:50:29.623 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6807", 2024-02-09T17:50:29.623 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T17:50:29.623 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.623 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.623 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.623 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:50:29.623 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:50:29.623 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.623 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:50:29.623 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6804", 2024-02-09T17:50:29.624 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T17:50:29.624 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.624 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.624 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:50:29.624 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6805", 2024-02-09T17:50:29.624 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T17:50:29.624 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.624 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.624 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.624 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.162:6801/4178676504", 2024-02-09T17:50:29.624 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.162:6803/4178676504", 2024-02-09T17:50:29.624 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.162:6807/4178676504", 2024-02-09T17:50:29.624 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.162:6805/4178676504", 2024-02-09T17:50:29.624 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:50:29.625 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:50:29.625 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:50:29.625 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.625 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.625 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.625 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 5, 2024-02-09T17:50:29.625 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "da98e3f3-173e-46fe-be8b-1ccbe8b0dabc", 2024-02-09T17:50:29.625 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:50:29.625 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:50:29.625 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:50:29.625 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 0, 2024-02-09T17:50:29.626 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:50:29.626 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:50:29.626 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 40, 2024-02-09T17:50:29.626 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 576, 2024-02-09T17:50:29.626 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:50:29.626 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:50:29.626 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:50:29.626 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:50:29.626 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.626 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:50:29.626 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6808", 2024-02-09T17:50:29.626 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T17:50:29.626 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.627 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.627 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:50:29.627 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6809", 2024-02-09T17:50:29.627 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T17:50:29.627 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.627 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.627 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.627 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:50:29.627 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:50:29.627 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.627 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:50:29.627 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6810", 2024-02-09T17:50:29.627 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T17:50:29.628 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.628 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.628 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:50:29.628 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6811", 2024-02-09T17:50:29.628 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T17:50:29.628 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.628 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.628 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.628 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:50:29.628 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:50:29.628 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.628 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:50:29.628 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6814", 2024-02-09T17:50:29.629 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T17:50:29.629 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.629 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.629 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:50:29.629 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6815", 2024-02-09T17:50:29.629 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T17:50:29.629 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.629 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.629 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.629 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:50:29.629 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:50:29.629 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.629 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:50:29.629 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6812", 2024-02-09T17:50:29.630 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T17:50:29.630 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.630 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.630 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:50:29.630 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6813", 2024-02-09T17:50:29.630 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T17:50:29.630 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.630 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.630 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.630 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.162:6809/1489371339", 2024-02-09T17:50:29.630 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.162:6811/1489371339", 2024-02-09T17:50:29.630 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.162:6815/1489371339", 2024-02-09T17:50:29.630 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.162:6813/1489371339", 2024-02-09T17:50:29.631 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:50:29.631 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:50:29.631 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:50:29.631 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.631 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.631 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.631 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 6, 2024-02-09T17:50:29.631 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "ff50382d-5ddf-4cda-bb0c-ff055feeaa40", 2024-02-09T17:50:29.631 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:50:29.631 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:50:29.631 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:50:29.631 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T17:50:29.631 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:50:29.631 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:50:29.631 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 49, 2024-02-09T17:50:29.632 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 661, 2024-02-09T17:50:29.632 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:50:29.632 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:50:29.632 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:50:29.632 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:50:29.632 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.632 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:50:29.632 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6816", 2024-02-09T17:50:29.632 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T17:50:29.632 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.632 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.632 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:50:29.632 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6817", 2024-02-09T17:50:29.633 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T17:50:29.633 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.633 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.633 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.633 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:50:29.633 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:50:29.633 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.633 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:50:29.633 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6818", 2024-02-09T17:50:29.633 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T17:50:29.633 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.633 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.633 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:50:29.633 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6819", 2024-02-09T17:50:29.633 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T17:50:29.634 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.634 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.634 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.634 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:50:29.634 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:50:29.634 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.634 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:50:29.634 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6822", 2024-02-09T17:50:29.634 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T17:50:29.634 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.634 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.634 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:50:29.634 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6823", 2024-02-09T17:50:29.634 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T17:50:29.635 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.635 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.635 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.635 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:50:29.635 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:50:29.635 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.635 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:50:29.635 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6820", 2024-02-09T17:50:29.635 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T17:50:29.635 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.635 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.635 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:50:29.635 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6821", 2024-02-09T17:50:29.635 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T17:50:29.635 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.636 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.636 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.636 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.162:6817/821169663", 2024-02-09T17:50:29.636 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.162:6819/821169663", 2024-02-09T17:50:29.636 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.162:6823/821169663", 2024-02-09T17:50:29.636 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.162:6821/821169663", 2024-02-09T17:50:29.636 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:50:29.636 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:50:29.636 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:50:29.636 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.636 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.636 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.636 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 7, 2024-02-09T17:50:29.636 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "a82984dc-5e22-424b-a446-ae2ee71eee19", 2024-02-09T17:50:29.637 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:50:29.637 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:50:29.637 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:50:29.637 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 0.5982513427734375, 2024-02-09T17:50:29.637 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:50:29.637 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:50:29.637 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 57, 2024-02-09T17:50:29.637 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 661, 2024-02-09T17:50:29.637 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:50:29.637 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:50:29.637 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:50:29.637 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:50:29.637 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.637 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:50:29.637 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6824", 2024-02-09T17:50:29.638 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T17:50:29.638 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.638 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.638 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:50:29.638 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6825", 2024-02-09T17:50:29.638 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T17:50:29.638 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.638 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.638 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.638 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:50:29.638 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:50:29.638 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.638 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:50:29.638 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6826", 2024-02-09T17:50:29.639 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T17:50:29.639 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.639 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.639 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:50:29.639 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6827", 2024-02-09T17:50:29.639 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T17:50:29.639 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.639 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.639 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.639 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:50:29.639 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:50:29.639 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.639 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:50:29.639 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6830", 2024-02-09T17:50:29.639 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T17:50:29.640 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.640 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.640 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:50:29.640 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6831", 2024-02-09T17:50:29.640 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T17:50:29.640 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.640 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.640 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.640 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:50:29.640 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:50:29.640 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.640 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:50:29.640 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6828", 2024-02-09T17:50:29.640 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T17:50:29.641 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.641 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.641 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:50:29.641 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6829", 2024-02-09T17:50:29.641 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T17:50:29.641 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.641 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.641 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.641 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.162:6825/965264369", 2024-02-09T17:50:29.641 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.162:6827/965264369", 2024-02-09T17:50:29.641 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.162:6831/965264369", 2024-02-09T17:50:29.641 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.162:6829/965264369", 2024-02-09T17:50:29.641 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:50:29.641 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:50:29.641 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:50:29.642 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.642 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.642 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.642 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 8, 2024-02-09T17:50:29.642 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "63f9d79f-0400-44dc-b388-df00fefb5820", 2024-02-09T17:50:29.642 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:50:29.642 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:50:29.642 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:50:29.642 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T17:50:29.642 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 66, 2024-02-09T17:50:29.642 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 145, 2024-02-09T17:50:29.642 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 171, 2024-02-09T17:50:29.642 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 661, 2024-02-09T17:50:29.642 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 146, 2024-02-09T17:50:29.642 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:50:29.643 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:50:29.643 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:50:29.643 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.643 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:50:29.643 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6800", 2024-02-09T17:50:29.643 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T17:50:29.643 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.643 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.643 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:50:29.643 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6801", 2024-02-09T17:50:29.643 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T17:50:29.644 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.644 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.644 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.644 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:50:29.644 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:50:29.644 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.644 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:50:29.644 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6802", 2024-02-09T17:50:29.644 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T17:50:29.644 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.644 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.644 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:50:29.644 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6803", 2024-02-09T17:50:29.644 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T17:50:29.645 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.645 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.645 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.645 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:50:29.645 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:50:29.645 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.645 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:50:29.645 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6806", 2024-02-09T17:50:29.645 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T17:50:29.645 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.645 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.645 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:50:29.645 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6807", 2024-02-09T17:50:29.646 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T17:50:29.646 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.646 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.646 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.646 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:50:29.646 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:50:29.646 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.646 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:50:29.646 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6804", 2024-02-09T17:50:29.646 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T17:50:29.646 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.646 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.646 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:50:29.646 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6805", 2024-02-09T17:50:29.646 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T17:50:29.647 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.647 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.647 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.647 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.179:6801/2558294495", 2024-02-09T17:50:29.647 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.179:6803/2558294495", 2024-02-09T17:50:29.647 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.179:6807/2558294495", 2024-02-09T17:50:29.647 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.179:6805/2558294495", 2024-02-09T17:50:29.647 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:50:29.647 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:50:29.647 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:50:29.647 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.647 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.647 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.648 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 9, 2024-02-09T17:50:29.648 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "7d995854-7328-4dfc-b1a9-0edca8d74f46", 2024-02-09T17:50:29.648 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:50:29.648 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:50:29.648 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:50:29.648 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T17:50:29.648 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:50:29.648 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:50:29.648 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 77, 2024-02-09T17:50:29.648 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 661, 2024-02-09T17:50:29.648 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:50:29.648 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:50:29.648 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:50:29.649 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:50:29.649 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.649 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:50:29.649 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6808", 2024-02-09T17:50:29.649 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T17:50:29.649 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.649 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.649 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:50:29.649 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6809", 2024-02-09T17:50:29.649 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T17:50:29.649 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.649 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.649 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.649 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:50:29.649 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:50:29.650 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.650 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:50:29.650 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6810", 2024-02-09T17:50:29.650 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T17:50:29.650 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.650 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.650 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:50:29.650 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6811", 2024-02-09T17:50:29.650 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T17:50:29.650 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.650 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.650 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.650 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:50:29.650 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:50:29.651 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.651 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:50:29.651 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6814", 2024-02-09T17:50:29.651 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T17:50:29.651 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.651 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.651 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:50:29.651 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6815", 2024-02-09T17:50:29.651 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T17:50:29.651 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.651 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.651 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.652 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:50:29.652 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:50:29.652 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.652 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:50:29.652 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6812", 2024-02-09T17:50:29.652 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T17:50:29.652 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.652 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.652 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:50:29.652 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6813", 2024-02-09T17:50:29.652 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T17:50:29.652 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.652 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.653 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.653 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.179:6809/504427183", 2024-02-09T17:50:29.653 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.179:6811/504427183", 2024-02-09T17:50:29.653 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.179:6815/504427183", 2024-02-09T17:50:29.653 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.179:6813/504427183", 2024-02-09T17:50:29.653 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:50:29.653 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:50:29.653 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:50:29.653 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.653 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.653 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.653 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 10, 2024-02-09T17:50:29.654 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "151fbedd-08e6-46e6-a4e5-a14248399f61", 2024-02-09T17:50:29.654 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:50:29.654 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:50:29.654 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:50:29.654 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T17:50:29.654 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:50:29.654 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:50:29.654 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 84, 2024-02-09T17:50:29.654 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 637, 2024-02-09T17:50:29.654 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:50:29.654 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:50:29.654 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:50:29.655 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:50:29.655 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.655 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:50:29.655 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6816", 2024-02-09T17:50:29.655 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T17:50:29.655 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.655 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.655 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:50:29.655 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6817", 2024-02-09T17:50:29.655 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T17:50:29.655 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.655 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.656 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.656 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:50:29.656 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:50:29.656 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.656 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:50:29.656 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6818", 2024-02-09T17:50:29.656 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T17:50:29.656 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.656 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.656 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:50:29.656 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6819", 2024-02-09T17:50:29.656 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T17:50:29.656 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.657 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.657 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.657 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:50:29.657 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:50:29.657 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.657 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:50:29.657 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6822", 2024-02-09T17:50:29.657 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T17:50:29.657 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.657 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.657 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:50:29.657 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6823", 2024-02-09T17:50:29.657 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T17:50:29.657 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.658 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.658 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.658 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:50:29.658 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:50:29.658 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.658 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:50:29.658 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6820", 2024-02-09T17:50:29.658 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T17:50:29.658 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.658 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.658 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:50:29.658 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6821", 2024-02-09T17:50:29.658 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T17:50:29.659 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.659 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.659 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.659 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.179:6817/2403345164", 2024-02-09T17:50:29.659 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.179:6819/2403345164", 2024-02-09T17:50:29.659 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.179:6823/2403345164", 2024-02-09T17:50:29.659 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.179:6821/2403345164", 2024-02-09T17:50:29.659 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:50:29.659 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:50:29.660 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:50:29.660 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.660 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.660 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.660 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 11, 2024-02-09T17:50:29.660 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "6d3a3ad8-417a-483f-991c-52549dbe7d73", 2024-02-09T17:50:29.660 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:50:29.660 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:50:29.660 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:50:29.660 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T17:50:29.660 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:50:29.660 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:50:29.661 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 90, 2024-02-09T17:50:29.661 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 661, 2024-02-09T17:50:29.661 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:50:29.661 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:50:29.661 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:50:29.661 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:50:29.661 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.661 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:50:29.661 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6824", 2024-02-09T17:50:29.661 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T17:50:29.661 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.661 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.661 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:50:29.661 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6825", 2024-02-09T17:50:29.662 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T17:50:29.662 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.662 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.662 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.662 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:50:29.662 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:50:29.662 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.662 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:50:29.662 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6826", 2024-02-09T17:50:29.662 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T17:50:29.662 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.662 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.662 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:50:29.662 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6827", 2024-02-09T17:50:29.663 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T17:50:29.663 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.663 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.663 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.663 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:50:29.663 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:50:29.663 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.663 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:50:29.663 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6830", 2024-02-09T17:50:29.663 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T17:50:29.663 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.663 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.663 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:50:29.663 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6831", 2024-02-09T17:50:29.664 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T17:50:29.664 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.664 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.664 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.664 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:50:29.664 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:50:29.664 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.664 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:50:29.664 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6828", 2024-02-09T17:50:29.664 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T17:50:29.664 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.664 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.664 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:50:29.664 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6829", 2024-02-09T17:50:29.665 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T17:50:29.665 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.665 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.665 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.665 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.179:6825/2899402373", 2024-02-09T17:50:29.665 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.179:6827/2899402373", 2024-02-09T17:50:29.665 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.179:6831/2899402373", 2024-02-09T17:50:29.665 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.179:6829/2899402373", 2024-02-09T17:50:29.666 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:50:29.666 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:50:29.666 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:50:29.666 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:50:29.666 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.666 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T17:50:29.666 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_xinfo": [ 2024-02-09T17:50:29.666 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.666 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 0, 2024-02-09T17:50:29.666 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:50:29.666 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:50:29.666 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:50:29.666 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:50:29.667 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:50:29.667 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:50:21.602808+0000", 2024-02-09T17:50:29.667 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:50:29.667 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.667 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.667 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 1, 2024-02-09T17:50:29.667 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:50:29.667 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:50:29.667 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:50:29.667 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:50:29.667 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:50:29.667 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:49:35.722437+0000", 2024-02-09T17:50:29.667 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:50:29.667 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.668 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.668 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 2, 2024-02-09T17:50:29.668 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:50:29.668 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:50:29.668 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:50:29.668 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:50:29.668 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:50:29.668 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:50:12.823330+0000", 2024-02-09T17:50:29.668 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:50:29.668 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.668 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.668 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 3, 2024-02-09T17:50:29.668 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:50:29.668 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:50:29.669 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:50:29.669 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:50:29.669 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:50:29.669 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:50:18.812445+0000", 2024-02-09T17:50:29.669 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:50:29.669 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.669 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.669 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 4, 2024-02-09T17:50:29.669 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:50:29.669 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:50:29.669 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:50:29.669 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:50:29.669 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:50:29.669 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:49:09.429320+0000", 2024-02-09T17:50:29.670 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:50:29.670 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.670 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.670 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 5, 2024-02-09T17:50:29.670 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:50:29.670 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:50:29.670 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:50:29.670 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:50:29.670 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:50:29.670 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:50:06.998100+0000", 2024-02-09T17:50:29.670 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:50:29.670 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.670 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.670 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 6, 2024-02-09T17:50:29.671 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:50:29.671 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:50:29.671 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:50:29.671 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:50:29.671 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:50:29.671 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:50:14.967122+0000", 2024-02-09T17:50:29.671 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:50:29.671 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.671 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.671 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 7, 2024-02-09T17:50:29.671 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:50:29.671 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:50:29.671 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:50:29.671 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:50:29.672 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:50:29.672 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:49:12.022681+0000", 2024-02-09T17:50:29.672 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:50:29.672 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.672 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.672 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 8, 2024-02-09T17:50:29.672 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:50:29.672 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:50:29.672 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:50:29.672 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:50:29.672 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:50:29.672 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:50:13.769816+0000", 2024-02-09T17:50:29.672 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 145 2024-02-09T17:50:29.673 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.673 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.673 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 9, 2024-02-09T17:50:29.673 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:50:29.673 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:50:29.673 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:50:29.673 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:50:29.673 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:50:29.673 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:49:10.159693+0000", 2024-02-09T17:50:29.673 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:50:29.673 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.673 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.673 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 10, 2024-02-09T17:50:29.673 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:50:29.674 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:50:29.674 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:50:29.674 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:50:29.674 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:50:29.674 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:50:17.580622+0000", 2024-02-09T17:50:29.674 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:50:29.674 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.674 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:50:29.674 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 11, 2024-02-09T17:50:29.674 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:50:29.674 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:50:29.674 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:50:29.674 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:50:29.674 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:50:29.674 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:49:41.769879+0000", 2024-02-09T17:50:29.675 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:50:29.675 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.675 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T17:50:29.675 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_upmap": [], 2024-02-09T17:50:29.675 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_upmap_items": [], 2024-02-09T17:50:29.675 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_upmap_primaries": [], 2024-02-09T17:50:29.675 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_temp": [], 2024-02-09T17:50:29.675 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_temp": [], 2024-02-09T17:50:29.675 INFO:teuthology.orchestra.run.smithi099.stdout: "blocklist": { 2024-02-09T17:50:29.675 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:0/3904172782": "2024-02-10T17:16:24.229365+0000", 2024-02-09T17:50:29.675 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:6801/1586034576": "2024-02-10T17:16:24.229365+0000", 2024-02-09T17:50:29.675 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:6800/746248609": "2024-02-10T17:15:54.399201+0000", 2024-02-09T17:50:29.675 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:0/4092784197": "2024-02-10T17:16:24.229365+0000", 2024-02-09T17:50:29.675 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:0/2844598101": "2024-02-10T17:15:54.399201+0000", 2024-02-09T17:50:29.675 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:6801/746248609": "2024-02-10T17:15:54.399201+0000", 2024-02-09T17:50:29.676 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:0/2528750050": "2024-02-10T17:15:54.399201+0000", 2024-02-09T17:50:29.676 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:6800/1586034576": "2024-02-10T17:16:24.229365+0000", 2024-02-09T17:50:29.676 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:0/2214140078": "2024-02-10T17:15:54.399201+0000", 2024-02-09T17:50:29.676 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:0/4217821965": "2024-02-10T17:16:24.229365+0000" 2024-02-09T17:50:29.676 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.676 INFO:teuthology.orchestra.run.smithi099.stdout: "range_blocklist": {}, 2024-02-09T17:50:29.676 INFO:teuthology.orchestra.run.smithi099.stdout: "erasure_code_profiles": { 2024-02-09T17:50:29.676 INFO:teuthology.orchestra.run.smithi099.stdout: "default": { 2024-02-09T17:50:29.676 INFO:teuthology.orchestra.run.smithi099.stdout: "crush-failure-domain": "osd", 2024-02-09T17:50:29.676 INFO:teuthology.orchestra.run.smithi099.stdout: "k": "2", 2024-02-09T17:50:29.676 INFO:teuthology.orchestra.run.smithi099.stdout: "m": "1", 2024-02-09T17:50:29.676 INFO:teuthology.orchestra.run.smithi099.stdout: "plugin": "jerasure", 2024-02-09T17:50:29.677 INFO:teuthology.orchestra.run.smithi099.stdout: "technique": "reed_sol_van" 2024-02-09T17:50:29.677 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.677 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:50:29.677 INFO:teuthology.orchestra.run.smithi099.stdout: "removed_snaps_queue": [], 2024-02-09T17:50:29.677 INFO:teuthology.orchestra.run.smithi099.stdout: "new_removed_snaps": [], 2024-02-09T17:50:29.677 INFO:teuthology.orchestra.run.smithi099.stdout: "new_purged_snaps": [], 2024-02-09T17:50:29.677 INFO:teuthology.orchestra.run.smithi099.stdout: "crush_node_flags": {}, 2024-02-09T17:50:29.677 INFO:teuthology.orchestra.run.smithi099.stdout: "device_class_flags": {}, 2024-02-09T17:50:29.677 INFO:teuthology.orchestra.run.smithi099.stdout: "stretch_mode": { 2024-02-09T17:50:29.677 INFO:teuthology.orchestra.run.smithi099.stdout: "stretch_mode_enabled": false, 2024-02-09T17:50:29.677 INFO:teuthology.orchestra.run.smithi099.stdout: "stretch_bucket_count": 0, 2024-02-09T17:50:29.677 INFO:teuthology.orchestra.run.smithi099.stdout: "degraded_stretch_mode": 0, 2024-02-09T17:50:29.677 INFO:teuthology.orchestra.run.smithi099.stdout: "recovering_stretch_mode": 0, 2024-02-09T17:50:29.677 INFO:teuthology.orchestra.run.smithi099.stdout: "stretch_mode_bucket": 0 2024-02-09T17:50:29.677 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:50:29.678 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:50:29.678 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:50:29.729 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:30.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:30 smithi162 ceph-mon[44304]: pgmap v1677: 47 pgs: 47 active+clean; 155 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s wr, 308 op/s 2024-02-09T17:50:30.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:30 smithi162 ceph-mon[44304]: 1.18 scrub starts 2024-02-09T17:50:30.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:30 smithi162 ceph-mon[44304]: 1.18 scrub ok 2024-02-09T17:50:30.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:30 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/813834116' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-09T17:50:30.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:30 smithi179 ceph-mon[43975]: pgmap v1677: 47 pgs: 47 active+clean; 155 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s wr, 308 op/s 2024-02-09T17:50:30.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:30 smithi179 ceph-mon[43975]: 1.18 scrub starts 2024-02-09T17:50:30.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:30 smithi179 ceph-mon[43975]: 1.18 scrub ok 2024-02-09T17:50:30.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:30 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/813834116' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-09T17:50:30.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:30 smithi099 ceph-mon[39420]: pgmap v1677: 47 pgs: 47 active+clean; 155 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s wr, 308 op/s 2024-02-09T17:50:30.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:30 smithi099 ceph-mon[39420]: 1.18 scrub starts 2024-02-09T17:50:30.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:30 smithi099 ceph-mon[39420]: 1.18 scrub ok 2024-02-09T17:50:30.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:30 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/813834116' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-09T17:50:30.503 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 4 16 13382 13366 208.717 208.938 0.00283162 0.00478117 2024-02-09T17:50:30.604 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:50:31.297 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:50:31.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:31.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:50:31.363 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:31 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:50:31.363 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:31 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:50:31.363 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:31 smithi162 ceph-mon[44304]: osdmap e666: 12 total, 12 up, 12 in 2024-02-09T17:50:31.364 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-09T17:50:31.402 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:50:31.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:31 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:50:31.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:31 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:50:31.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:31 smithi179 ceph-mon[43975]: osdmap e666: 12 total, 12 up, 12 in 2024-02-09T17:50:31.445 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:31 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:50:31.445 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:31 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:50:31.445 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:31 smithi099 ceph-mon[39420]: osdmap e666: 12 total, 12 up, 12 in 2024-02-09T17:50:31.453 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:31.503 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 5 15 16211 16196 202.348 176.875 0.00431803 0.00493452 2024-02-09T17:50:31.709 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:50:31.927 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:50:31.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:31.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:50:32.029 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:50:32.067 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:32.095 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:50:32.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:32.100 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:50:32.201 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:50:32.235 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:32 smithi162 ceph-mon[44304]: pgmap v1679: 47 pgs: 47 active+clean; 298 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 594 op/s 2024-02-09T17:50:32.235 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:32 smithi162 ceph-mon[44304]: 1.16 scrub starts 2024-02-09T17:50:32.235 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:32 smithi162 ceph-mon[44304]: 1.1c scrub starts 2024-02-09T17:50:32.235 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:32 smithi162 ceph-mon[44304]: 1.1c scrub ok 2024-02-09T17:50:32.236 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:32 smithi162 ceph-mon[44304]: 1.16 scrub ok 2024-02-09T17:50:32.238 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:32.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:32 smithi099 ceph-mon[39420]: pgmap v1679: 47 pgs: 47 active+clean; 298 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 594 op/s 2024-02-09T17:50:32.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:32 smithi099 ceph-mon[39420]: 1.16 scrub starts 2024-02-09T17:50:32.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:32 smithi099 ceph-mon[39420]: 1.1c scrub starts 2024-02-09T17:50:32.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:32 smithi099 ceph-mon[39420]: 1.1c scrub ok 2024-02-09T17:50:32.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:32 smithi099 ceph-mon[39420]: 1.16 scrub ok 2024-02-09T17:50:32.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:32 smithi179 ceph-mon[43975]: pgmap v1679: 47 pgs: 47 active+clean; 298 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 594 op/s 2024-02-09T17:50:32.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:32 smithi179 ceph-mon[43975]: 1.16 scrub starts 2024-02-09T17:50:32.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:32 smithi179 ceph-mon[43975]: 1.1c scrub starts 2024-02-09T17:50:32.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:32 smithi179 ceph-mon[43975]: 1.1c scrub ok 2024-02-09T17:50:32.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:32 smithi179 ceph-mon[43975]: 1.16 scrub ok 2024-02-09T17:50:32.503 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 6 16 19262 19246 200.394 190.625 0.00197127 0.00496432 2024-02-09T17:50:33.107 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:33 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:50:33.108 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:33 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:50:33.108 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:33 smithi162 ceph-mon[44304]: 1.17 scrub starts 2024-02-09T17:50:33.108 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:33 smithi162 ceph-mon[44304]: 1.17 scrub ok 2024-02-09T17:50:33.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:33 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:50:33.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:33 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:50:33.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:33 smithi179 ceph-mon[43975]: 1.17 scrub starts 2024-02-09T17:50:33.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:33 smithi179 ceph-mon[43975]: 1.17 scrub ok 2024-02-09T17:50:33.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:33 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:50:33.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:33 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:50:33.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:33 smithi099 ceph-mon[39420]: 1.17 scrub starts 2024-02-09T17:50:33.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:33 smithi099 ceph-mon[39420]: 1.17 scrub ok 2024-02-09T17:50:33.503 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 7 16 21453 21437 191.327 136.938 0.00547645 0.00522021 2024-02-09T17:50:33.796 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:50:33.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:33.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:50:33.901 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:50:33.940 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:33.968 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:50:33.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:33.971 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:50:34.050 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:50:34.050 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:50:34.072 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:50:34.113 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:34 smithi099 ceph-mon[39420]: pgmap v1680: 47 pgs: 47 active+clean; 902 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 104 MiB/s wr, 1.66k op/s 2024-02-09T17:50:34.114 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:34 smithi099 ceph-mon[39420]: osdmap e667: 12 total, 12 up, 12 in 2024-02-09T17:50:34.114 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:34 smithi099 ceph-mon[39420]: 1.1b scrub starts 2024-02-09T17:50:34.114 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:34 smithi099 ceph-mon[39420]: 1.1b scrub ok 2024-02-09T17:50:34.129 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:34.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:34 smithi179 ceph-mon[43975]: pgmap v1680: 47 pgs: 47 active+clean; 902 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 104 MiB/s wr, 1.66k op/s 2024-02-09T17:50:34.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:34 smithi179 ceph-mon[43975]: osdmap e667: 12 total, 12 up, 12 in 2024-02-09T17:50:34.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:34 smithi179 ceph-mon[43975]: 1.1b scrub starts 2024-02-09T17:50:34.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:34 smithi179 ceph-mon[43975]: 1.1b scrub ok 2024-02-09T17:50:34.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:34 smithi162 ceph-mon[44304]: pgmap v1680: 47 pgs: 47 active+clean; 902 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 104 MiB/s wr, 1.66k op/s 2024-02-09T17:50:34.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:34 smithi162 ceph-mon[44304]: osdmap e667: 12 total, 12 up, 12 in 2024-02-09T17:50:34.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:34 smithi162 ceph-mon[44304]: 1.1b scrub starts 2024-02-09T17:50:34.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:34 smithi162 ceph-mon[44304]: 1.1b scrub ok 2024-02-09T17:50:34.504 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 8 16 24143 24127 188.424 168.125 0.00796597 0.00530174 2024-02-09T17:50:35.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:35 smithi179 ceph-mon[43975]: 1.4 scrub starts 2024-02-09T17:50:35.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:35 smithi179 ceph-mon[43975]: 1.4 scrub ok 2024-02-09T17:50:35.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:35 smithi162 ceph-mon[44304]: 1.4 scrub starts 2024-02-09T17:50:35.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:35 smithi162 ceph-mon[44304]: 1.4 scrub ok 2024-02-09T17:50:35.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:35 smithi099 ceph-mon[39420]: 1.4 scrub starts 2024-02-09T17:50:35.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:35 smithi099 ceph-mon[39420]: 1.4 scrub ok 2024-02-09T17:50:35.504 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 9 16 26553 26537 184.222 150.625 0.0017543 0.00535661 2024-02-09T17:50:35.785 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:50:35.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:35.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:50:35.889 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:50:35.935 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:36.001 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:50:36.004 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:36.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:50:36.106 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:50:36.127 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:36 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:50:36.127 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:36 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:50:36.127 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:36 smithi162 ceph-mon[44304]: pgmap v1682: 47 pgs: 47 active+clean; 1.1 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 140 MiB/s wr, 2.24k op/s 2024-02-09T17:50:36.127 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:36 smithi162 ceph-mon[44304]: osdmap e668: 12 total, 12 up, 12 in 2024-02-09T17:50:36.127 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:36 smithi162 ceph-mon[44304]: 1.3 scrub starts 2024-02-09T17:50:36.127 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:36 smithi162 ceph-mon[44304]: 1.3 scrub ok 2024-02-09T17:50:36.168 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:36.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:36 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:50:36.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:36 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:50:36.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:36 smithi099 ceph-mon[39420]: pgmap v1682: 47 pgs: 47 active+clean; 1.1 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 140 MiB/s wr, 2.24k op/s 2024-02-09T17:50:36.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:36 smithi099 ceph-mon[39420]: osdmap e668: 12 total, 12 up, 12 in 2024-02-09T17:50:36.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:36 smithi099 ceph-mon[39420]: 1.3 scrub starts 2024-02-09T17:50:36.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:36 smithi099 ceph-mon[39420]: 1.3 scrub ok 2024-02-09T17:50:36.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:36 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:50:36.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:36 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:50:36.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:36 smithi179 ceph-mon[43975]: pgmap v1682: 47 pgs: 47 active+clean; 1.1 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 140 MiB/s wr, 2.24k op/s 2024-02-09T17:50:36.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:36 smithi179 ceph-mon[43975]: osdmap e668: 12 total, 12 up, 12 in 2024-02-09T17:50:36.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:36 smithi179 ceph-mon[43975]: 1.3 scrub starts 2024-02-09T17:50:36.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:36 smithi179 ceph-mon[43975]: 1.3 scrub ok 2024-02-09T17:50:36.437 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:50:36.447 INFO:tasks.thrashosds.thrasher:j is {'epoch': 665, 'fsid': 'a69c870a-c76e-11ee-95b7-87774f69a715', 'created': '2024-02-09T17:15:26.113606+0000', 'modified': '2024-02-09T17:50:24.310359+0000', 'last_up_change': '2024-02-09T17:28:17.711044+0000', 'last_in_change': '2024-02-09T17:43:24.138763+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 27, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-09T17:18:56.552286+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '544', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '514', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 5.989999771118164, 'score_stable': 1.7799999713897705, 'optimal_score': 1.090000033378601, 'raw_score_acting': 6.53000020980835, 'raw_score_stable': 1.940000057220459, 'primary_affinity_weighted': 0.7807959914207458, 'average_primary_affinity': 0.8500000238418579, 'average_primary_affinity_weighted': 0.7799999713897705, 'error_message': 'pool 1 has acting primaries on OSD(s) with primary affinity 0, read balance score is not accurate'}}, {'pool': 7, 'pool_name': 'unique_pool_5', 'create_time': '2024-02-09T17:50:17.290198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '665', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.059999942779541, 'score_stable': 2.059999942779541, 'optimal_score': 1.090000033378601, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.7807959914207458, 'average_primary_affinity': 0.8500000238418579, 'average_primary_affinity_weighted': 0.7799999713897705}}], 'osds': [{'osd': 0, 'uuid': '21739aa7-e278-45da-9a3f-867385f01baf', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.609619140625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 661, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6802', 'nonce': 3859868923}, {'type': 'v1', 'addr': '172.21.15.99:6803', 'nonce': 3859868923}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6804', 'nonce': 3859868923}, {'type': 'v1', 'addr': '172.21.15.99:6805', 'nonce': 3859868923}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6808', 'nonce': 3859868923}, {'type': 'v1', 'addr': '172.21.15.99:6809', 'nonce': 3859868923}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6806', 'nonce': 3859868923}, {'type': 'v1', 'addr': '172.21.15.99:6807', 'nonce': 3859868923}]}, 'public_addr': '172.21.15.99:6803/3859868923', 'cluster_addr': '172.21.15.99:6805/3859868923', 'heartbeat_back_addr': '172.21.15.99:6809/3859868923', 'heartbeat_front_addr': '172.21.15.99:6807/3859868923', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '8eab347c-9065-4de1-93ae-e2eb7e11cab7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 661, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6810', 'nonce': 3868328452}, {'type': 'v1', 'addr': '172.21.15.99:6811', 'nonce': 3868328452}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6812', 'nonce': 3868328452}, {'type': 'v1', 'addr': '172.21.15.99:6813', 'nonce': 3868328452}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6816', 'nonce': 3868328452}, {'type': 'v1', 'addr': '172.21.15.99:6817', 'nonce': 3868328452}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6814', 'nonce': 3868328452}, {'type': 'v1', 'addr': '172.21.15.99:6815', 'nonce': 3868328452}]}, 'public_addr': '172.21.15.99:6811/3868328452', 'cluster_addr': '172.21.15.99:6813/3868328452', 'heartbeat_back_addr': '172.21.15.99:6817/3868328452', 'heartbeat_front_addr': '172.21.15.99:6815/3868328452', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '575f4257-7e34-485e-963c-2607b69e1220', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.3601837158203125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 661, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6818', 'nonce': 537506237}, {'type': 'v1', 'addr': '172.21.15.99:6819', 'nonce': 537506237}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6820', 'nonce': 537506237}, {'type': 'v1', 'addr': '172.21.15.99:6821', 'nonce': 537506237}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6824', 'nonce': 537506237}, {'type': 'v1', 'addr': '172.21.15.99:6825', 'nonce': 537506237}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6822', 'nonce': 537506237}, {'type': 'v1', 'addr': '172.21.15.99:6823', 'nonce': 537506237}]}, 'public_addr': '172.21.15.99:6819/537506237', 'cluster_addr': '172.21.15.99:6821/537506237', 'heartbeat_back_addr': '172.21.15.99:6825/537506237', 'heartbeat_front_addr': '172.21.15.99:6823/537506237', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '1a520bcf-81bb-4f1d-80fb-11e98008e429', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.8014984130859375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 661, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6826', 'nonce': 762784793}, {'type': 'v1', 'addr': '172.21.15.99:6827', 'nonce': 762784793}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6828', 'nonce': 762784793}, {'type': 'v1', 'addr': '172.21.15.99:6829', 'nonce': 762784793}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6832', 'nonce': 762784793}, {'type': 'v1', 'addr': '172.21.15.99:6833', 'nonce': 762784793}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6830', 'nonce': 762784793}, {'type': 'v1', 'addr': '172.21.15.99:6831', 'nonce': 762784793}]}, 'public_addr': '172.21.15.99:6827/762784793', 'cluster_addr': '172.21.15.99:6829/762784793', 'heartbeat_back_addr': '172.21.15.99:6833/762784793', 'heartbeat_front_addr': '172.21.15.99:6831/762784793', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '4617c2fe-c9d8-40cd-887b-303e3215bdf8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 661, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6800', 'nonce': 4178676504}, {'type': 'v1', 'addr': '172.21.15.162:6801', 'nonce': 4178676504}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6802', 'nonce': 4178676504}, {'type': 'v1', 'addr': '172.21.15.162:6803', 'nonce': 4178676504}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6806', 'nonce': 4178676504}, {'type': 'v1', 'addr': '172.21.15.162:6807', 'nonce': 4178676504}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6804', 'nonce': 4178676504}, {'type': 'v1', 'addr': '172.21.15.162:6805', 'nonce': 4178676504}]}, 'public_addr': '172.21.15.162:6801/4178676504', 'cluster_addr': '172.21.15.162:6803/4178676504', 'heartbeat_back_addr': '172.21.15.162:6807/4178676504', 'heartbeat_front_addr': '172.21.15.162:6805/4178676504', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'da98e3f3-173e-46fe-be8b-1ccbe8b0dabc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 576, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6808', 'nonce': 1489371339}, {'type': 'v1', 'addr': '172.21.15.162:6809', 'nonce': 1489371339}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6810', 'nonce': 1489371339}, {'type': 'v1', 'addr': '172.21.15.162:6811', 'nonce': 1489371339}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6814', 'nonce': 1489371339}, {'type': 'v1', 'addr': '172.21.15.162:6815', 'nonce': 1489371339}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6812', 'nonce': 1489371339}, {'type': 'v1', 'addr': '172.21.15.162:6813', 'nonce': 1489371339}]}, 'public_addr': '172.21.15.162:6809/1489371339', 'cluster_addr': '172.21.15.162:6811/1489371339', 'heartbeat_back_addr': '172.21.15.162:6815/1489371339', 'heartbeat_front_addr': '172.21.15.162:6813/1489371339', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'ff50382d-5ddf-4cda-bb0c-ff055feeaa40', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 49, 'up_thru': 661, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6816', 'nonce': 821169663}, {'type': 'v1', 'addr': '172.21.15.162:6817', 'nonce': 821169663}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6818', 'nonce': 821169663}, {'type': 'v1', 'addr': '172.21.15.162:6819', 'nonce': 821169663}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6822', 'nonce': 821169663}, {'type': 'v1', 'addr': '172.21.15.162:6823', 'nonce': 821169663}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6820', 'nonce': 821169663}, {'type': 'v1', 'addr': '172.21.15.162:6821', 'nonce': 821169663}]}, 'public_addr': '172.21.15.162:6817/821169663', 'cluster_addr': '172.21.15.162:6819/821169663', 'heartbeat_back_addr': '172.21.15.162:6823/821169663', 'heartbeat_front_addr': '172.21.15.162:6821/821169663', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'a82984dc-5e22-424b-a446-ae2ee71eee19', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.5982513427734375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 57, 'up_thru': 661, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6824', 'nonce': 965264369}, {'type': 'v1', 'addr': '172.21.15.162:6825', 'nonce': 965264369}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6826', 'nonce': 965264369}, {'type': 'v1', 'addr': '172.21.15.162:6827', 'nonce': 965264369}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6830', 'nonce': 965264369}, {'type': 'v1', 'addr': '172.21.15.162:6831', 'nonce': 965264369}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6828', 'nonce': 965264369}, {'type': 'v1', 'addr': '172.21.15.162:6829', 'nonce': 965264369}]}, 'public_addr': '172.21.15.162:6825/965264369', 'cluster_addr': '172.21.15.162:6827/965264369', 'heartbeat_back_addr': '172.21.15.162:6831/965264369', 'heartbeat_front_addr': '172.21.15.162:6829/965264369', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '63f9d79f-0400-44dc-b388-df00fefb5820', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 66, 'last_clean_end': 145, 'up_from': 171, 'up_thru': 661, 'down_at': 146, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6800', 'nonce': 2558294495}, {'type': 'v1', 'addr': '172.21.15.179:6801', 'nonce': 2558294495}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6802', 'nonce': 2558294495}, {'type': 'v1', 'addr': '172.21.15.179:6803', 'nonce': 2558294495}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6806', 'nonce': 2558294495}, {'type': 'v1', 'addr': '172.21.15.179:6807', 'nonce': 2558294495}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6804', 'nonce': 2558294495}, {'type': 'v1', 'addr': '172.21.15.179:6805', 'nonce': 2558294495}]}, 'public_addr': '172.21.15.179:6801/2558294495', 'cluster_addr': '172.21.15.179:6803/2558294495', 'heartbeat_back_addr': '172.21.15.179:6807/2558294495', 'heartbeat_front_addr': '172.21.15.179:6805/2558294495', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '7d995854-7328-4dfc-b1a9-0edca8d74f46', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 77, 'up_thru': 661, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6808', 'nonce': 504427183}, {'type': 'v1', 'addr': '172.21.15.179:6809', 'nonce': 504427183}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6810', 'nonce': 504427183}, {'type': 'v1', 'addr': '172.21.15.179:6811', 'nonce': 504427183}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6814', 'nonce': 504427183}, {'type': 'v1', 'addr': '172.21.15.179:6815', 'nonce': 504427183}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6812', 'nonce': 504427183}, {'type': 'v1', 'addr': '172.21.15.179:6813', 'nonce': 504427183}]}, 'public_addr': '172.21.15.179:6809/504427183', 'cluster_addr': '172.21.15.179:6811/504427183', 'heartbeat_back_addr': '172.21.15.179:6815/504427183', 'heartbeat_front_addr': '172.21.15.179:6813/504427183', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '151fbedd-08e6-46e6-a4e5-a14248399f61', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 84, 'up_thru': 637, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6816', 'nonce': 2403345164}, {'type': 'v1', 'addr': '172.21.15.179:6817', 'nonce': 2403345164}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6818', 'nonce': 2403345164}, {'type': 'v1', 'addr': '172.21.15.179:6819', 'nonce': 2403345164}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6822', 'nonce': 2403345164}, {'type': 'v1', 'addr': '172.21.15.179:6823', 'nonce': 2403345164}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6820', 'nonce': 2403345164}, {'type': 'v1', 'addr': '172.21.15.179:6821', 'nonce': 2403345164}]}, 'public_addr': '172.21.15.179:6817/2403345164', 'cluster_addr': '172.21.15.179:6819/2403345164', 'heartbeat_back_addr': '172.21.15.179:6823/2403345164', 'heartbeat_front_addr': '172.21.15.179:6821/2403345164', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '6d3a3ad8-417a-483f-991c-52549dbe7d73', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 90, 'up_thru': 661, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6824', 'nonce': 2899402373}, {'type': 'v1', 'addr': '172.21.15.179:6825', 'nonce': 2899402373}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6826', 'nonce': 2899402373}, {'type': 'v1', 'addr': '172.21.15.179:6827', 'nonce': 2899402373}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6830', 'nonce': 2899402373}, {'type': 'v1', 'addr': '172.21.15.179:6831', 'nonce': 2899402373}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6828', 'nonce': 2899402373}, {'type': 'v1', 'addr': '172.21.15.179:6829', 'nonce': 2899402373}]}, 'public_addr': '172.21.15.179:6825/2899402373', 'cluster_addr': '172.21.15.179:6827/2899402373', 'heartbeat_back_addr': '172.21.15.179:6831/2899402373', 'heartbeat_front_addr': '172.21.15.179:6829/2899402373', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:50:21.602808+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:49:35.722437+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:50:12.823330+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:50:18.812445+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:49:09.429320+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:50:06.998100+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:50:14.967122+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:49:12.022681+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:50:13.769816+0000', 'dead_epoch': 145}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:49:10.159693+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:50:17.580622+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:49:41.769879+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.99:0/3904172782': '2024-02-10T17:16:24.229365+0000', '172.21.15.99:6801/1586034576': '2024-02-10T17:16:24.229365+0000', '172.21.15.99:6800/746248609': '2024-02-10T17:15:54.399201+0000', '172.21.15.99:0/4092784197': '2024-02-10T17:16:24.229365+0000', '172.21.15.99:0/2844598101': '2024-02-10T17:15:54.399201+0000', '172.21.15.99:6801/746248609': '2024-02-10T17:15:54.399201+0000', '172.21.15.99:0/2528750050': '2024-02-10T17:15:54.399201+0000', '172.21.15.99:6800/1586034576': '2024-02-10T17:16:24.229365+0000', '172.21.15.99:0/2214140078': '2024-02-10T17:15:54.399201+0000', '172.21.15.99:0/4217821965': '2024-02-10T17:16:24.229365+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-09T17:50:36.448 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph pg dump --format=json 2024-02-09T17:50:36.504 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 10 16 28871 28855 180.287 144.875 0.00549143 0.00554086 2024-02-09T17:50:36.505 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:50:36.509 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:36.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:50:36.536 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-09T17:50:36.611 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:50:36.672 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:36.737 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:50:36.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:36.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:50:36.840 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:50:36.882 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:36.910 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:50:36.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:36.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:50:36.922 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:50:37.014 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:50:37.055 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:37.086 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:50:37.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:37.089 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:50:37.190 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:50:37.238 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:37.504 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 11 15 31114 31099 176.647 140.25 0.0047083 0.00565561 2024-02-09T17:50:37.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:37 smithi162 ceph-mon[44304]: osdmap e669: 12 total, 12 up, 12 in 2024-02-09T17:50:37.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:37 smithi162 ceph-mon[44304]: pgmap v1685: 47 pgs: 47 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 202 MiB/s wr, 3.23k op/s 2024-02-09T17:50:38.026 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:37 smithi099 ceph-mon[39420]: osdmap e669: 12 total, 12 up, 12 in 2024-02-09T17:50:38.026 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:37 smithi099 ceph-mon[39420]: pgmap v1685: 47 pgs: 47 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 202 MiB/s wr, 3.23k op/s 2024-02-09T17:50:38.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:37 smithi179 ceph-mon[43975]: osdmap e669: 12 total, 12 up, 12 in 2024-02-09T17:50:38.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:37 smithi179 ceph-mon[43975]: pgmap v1685: 47 pgs: 47 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 202 MiB/s wr, 3.23k op/s 2024-02-09T17:50:38.504 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 12 16 32930 32914 171.38 113.438 0.00805896 0.00582504 2024-02-09T17:50:38.758 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:38 smithi162 ceph-mon[44304]: 1.7 scrub starts 2024-02-09T17:50:38.798 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:38 smithi162 ceph-mon[44304]: 1.7 scrub ok 2024-02-09T17:50:38.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:38 smithi099 ceph-mon[39420]: 1.7 scrub starts 2024-02-09T17:50:38.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:38 smithi099 ceph-mon[39420]: 1.7 scrub ok 2024-02-09T17:50:39.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:38 smithi179 ceph-mon[43975]: 1.7 scrub starts 2024-02-09T17:50:39.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:38 smithi179 ceph-mon[43975]: 1.7 scrub ok 2024-02-09T17:50:39.338 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:50:39.338 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:50:39.505 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 13 16 35181 35165 169.003 140.688 0.0048142 0.00591189 2024-02-09T17:50:40.105 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:50:40.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:40.111 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:39 smithi099 ceph-mon[39420]: 1.13 scrub starts 2024-02-09T17:50:40.111 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:39 smithi099 ceph-mon[39420]: 1.13 scrub ok 2024-02-09T17:50:40.111 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:39 smithi099 ceph-mon[39420]: pgmap v1686: 47 pgs: 47 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 154 MiB/s wr, 2.46k op/s 2024-02-09T17:50:40.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:50:40.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:39 smithi179 ceph-mon[43975]: 1.13 scrub starts 2024-02-09T17:50:40.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:39 smithi179 ceph-mon[43975]: 1.13 scrub ok 2024-02-09T17:50:40.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:39 smithi179 ceph-mon[43975]: pgmap v1686: 47 pgs: 47 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 154 MiB/s wr, 2.46k op/s 2024-02-09T17:50:40.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:39 smithi162 ceph-mon[44304]: 1.13 scrub starts 2024-02-09T17:50:40.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:39 smithi162 ceph-mon[44304]: 1.13 scrub ok 2024-02-09T17:50:40.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:39 smithi162 ceph-mon[44304]: pgmap v1686: 47 pgs: 47 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 154 MiB/s wr, 2.46k op/s 2024-02-09T17:50:40.213 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:50:40.257 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:40.286 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:50:40.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:40.290 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:50:40.391 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:50:40.431 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:40.505 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 14 16 37255 37239 166.191 129.625 0.00411053 0.00601139 2024-02-09T17:50:40.608 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:50:40.803 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is set 2024-02-09T17:50:40.935 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:40 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1624886057' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:50:41.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:40 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1624886057' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:50:41.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:40 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1624886057' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:50:41.506 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 15 16 38871 38855 161.845 101 0.00461605 0.00617353 2024-02-09T17:50:41.689 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:50:41.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:41.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:50:41.698 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-09T17:50:41.793 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:50:41.848 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:41.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:41 smithi162 ceph-mon[44304]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:50:41.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:41 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1624886057' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:50:41.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:41 smithi162 ceph-mon[44304]: osdmap e670: 12 total, 12 up, 12 in 2024-02-09T17:50:41.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:41 smithi162 ceph-mon[44304]: pgmap v1688: 47 pgs: 47 active+clean; 2.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 157 MiB/s wr, 2.52k op/s 2024-02-09T17:50:41.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:41 smithi162 ceph-mon[44304]: osdmap e671: 12 total, 12 up, 12 in 2024-02-09T17:50:42.080 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:50:42.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:41 smithi179 ceph-mon[43975]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:50:42.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:41 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1624886057' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:50:42.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:41 smithi179 ceph-mon[43975]: osdmap e670: 12 total, 12 up, 12 in 2024-02-09T17:50:42.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:41 smithi179 ceph-mon[43975]: pgmap v1688: 47 pgs: 47 active+clean; 2.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 157 MiB/s wr, 2.52k op/s 2024-02-09T17:50:42.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:41 smithi179 ceph-mon[43975]: osdmap e671: 12 total, 12 up, 12 in 2024-02-09T17:50:42.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:41 smithi099 ceph-mon[39420]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:50:42.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:41 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1624886057' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:50:42.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:41 smithi099 ceph-mon[39420]: osdmap e670: 12 total, 12 up, 12 in 2024-02-09T17:50:42.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:41 smithi099 ceph-mon[39420]: pgmap v1688: 47 pgs: 47 active+clean; 2.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 157 MiB/s wr, 2.52k op/s 2024-02-09T17:50:42.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:41 smithi099 ceph-mon[39420]: osdmap e671: 12 total, 12 up, 12 in 2024-02-09T17:50:42.506 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 16 16 40818 40802 159.335 121.688 0.00633123 0.0062713 2024-02-09T17:50:43.313 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:50:43.317 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:43.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:50:43.418 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:50:43.473 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:43.499 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:50:43.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:43.503 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:50:43.506 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 17 16 42840 42824 157.396 126.375 0.0121465 0.00634901 2024-02-09T17:50:43.604 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:50:43.641 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:43.665 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:50:43.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:43.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:50:43.772 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:50:43.803 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:43.901 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:50:43.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:43.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:50:44.005 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:50:44.048 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:44.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:44 smithi162 ceph-mon[44304]: pgmap v1690: 47 pgs: 47 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 148 MiB/s wr, 2.37k op/s 2024-02-09T17:50:44.197 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:50:44.199 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:44.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:50:44.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:44 smithi099 ceph-mon[39420]: pgmap v1690: 47 pgs: 47 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 148 MiB/s wr, 2.37k op/s 2024-02-09T17:50:44.301 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:50:44.342 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:44 smithi179 ceph-mon[43975]: pgmap v1690: 47 pgs: 47 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 148 MiB/s wr, 2.37k op/s 2024-02-09T17:50:44.343 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:44.367 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:50:44.367 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:50:44.372 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:50:44.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:44.376 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:50:44.477 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:50:44.506 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 18 16 44390 44374 154.034 96.875 0.00267023 0.00647993 2024-02-09T17:50:44.507 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:44.526 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:50:44.529 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:44.530 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:50:44.533 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:50:44.533 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:50:44.566 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:50:44.603 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:50:44.630 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:50:44.631 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:50:44.696 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:44.696 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:50:44.704 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:50:44.740 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:50:44.775 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:50:44.813 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:50:44.852 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:50:44.896 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:50:44.912 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:50:44.912 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:50:44.912 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:50:44.912 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:50:44.912 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:50:44.912 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:50:44.912 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:50:44.912 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:50:44.912 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:50:44.912 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:50:44.912 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:50:44.912 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:50:45.265 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:50:45.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:45.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:50:45.384 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:50:45.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:45 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3151914186' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:50:45.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:45 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3151914186' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:50:45.433 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:45 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3151914186' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:50:45.437 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:45.506 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 19 16 46109 46093 151.581 107.438 0.0137742 0.00659191 2024-02-09T17:50:45.913 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:50:45.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:45.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:50:46.008 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:50:46.021 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:50:46.033 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-02-09T17:50:46.052 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:46.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:46 smithi099 ceph-mon[39420]: from='client.26015 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:50:46.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:46 smithi099 ceph-mon[39420]: pgmap v1691: 47 pgs: 47 active+clean; 2.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 123 MiB/s wr, 1.97k op/s 2024-02-09T17:50:46.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:46 smithi179 ceph-mon[43975]: from='client.26015 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:50:46.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:46 smithi179 ceph-mon[43975]: pgmap v1691: 47 pgs: 47 active+clean; 2.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 123 MiB/s wr, 1.97k op/s 2024-02-09T17:50:46.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:46 smithi162 ceph-mon[44304]: from='client.26015 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:50:46.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:46 smithi162 ceph-mon[44304]: pgmap v1691: 47 pgs: 47 active+clean; 2.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 123 MiB/s wr, 1.97k op/s 2024-02-09T17:50:46.508 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:2024-02-09T17:50:46.507271+0000 min lat: 0.0011365 max lat: 0.211587 avg lat: 0.00666048 2024-02-09T17:50:46.508 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-09T17:50:46.508 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 20 15 48043 48028 150.037 120.938 0.0123411 0.00666048 2024-02-09T17:50:46.509 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:50:46.515 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-09T17:50:46.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:46.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:50:46.617 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:50:46.651 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:46.682 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:50:46.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:46.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:50:46.787 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:50:46.835 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:46.909 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:50:46.933 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:50:46.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:46.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:50:47.041 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:50:47.075 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:47.207 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:50:47.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:47.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:50:47.310 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:50:47.339 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:47.464 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:50:47.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:47.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:50:47.483 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":1690,"stamp":"2024-02-09T17:50:43.032320+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2462777888,"num_objects":37572,"num_object_clones":0,"num_object_copies":112716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37570,"num_write_kb":2404480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41879,"ondisk_log_size":41879,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":13962292,"kb_used_data":7001644,"kb_used_omap":229,"kb_used_meta":6960218,"kb_avail":1110979532,"statfs":{"total":1151940427776,"available":1137643040768,"internally_reserved":0,"allocated":7169683456,"data_stored":7165693624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235127,"internal_metadata":7127263625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":977928192,"num_objects":14922,"num_object_clones":0,"num_object_copies":44766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14922,"num_write_kb":955008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.283371"},"pg_stats":[{"pgid":"1.1e","version":"63'139","reported_seq":602,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-09T17:50:36.752356+0000","last_change":"2024-02-09T17:50:15.259059+0000","last_active":"2024-02-09T17:50:36.752356+0000","last_peered":"2024-02-09T17:50:36.752356+0000","last_clean":"2024-02-09T17:50:36.752356+0000","last_became_active":"2024-02-09T17:43:19.126757+0000","last_became_peered":"2024-02-09T17:43:19.126757+0000","last_unstale":"2024-02-09T17:50:36.752356+0000","last_undegraded":"2024-02-09T17:50:36.752356+0000","last_fullsized":"2024-02-09T17:50:36.752356+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:50:15.258944+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:50:15.258944+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:51:25.540541+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"63'139","reported_seq":626,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-09T17:50:36.754082+0000","last_change":"2024-02-09T17:50:21.514936+0000","last_active":"2024-02-09T17:50:36.754082+0000","last_peered":"2024-02-09T17:50:36.754082+0000","last_clean":"2024-02-09T17:50:36.754082+0000","last_became_active":"2024-02-09T17:43:28.594296+0000","last_became_peered":"2024-02-09T17:43:28.594296+0000","last_unstale":"2024-02-09T17:50:36.754082+0000","last_undegraded":"2024-02-09T17:50:36.754082+0000","last_fullsized":"2024-02-09T17:50:36.754082+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:50:21.514819+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:48:24.634092+0000","last_clean_scrub_stamp":"2024-02-09T17:50:21.514819+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:51:36.274392+0000","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"63'139","reported_seq":1150,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-02-09T17:50:41.758881+0000","last_change":"2024-02-09T17:50:31.454680+0000","last_active":"2024-02-09T17:50:41.758881+0000","last_peered":"2024-02-09T17:50:41.758881+0000","last_clean":"2024-02-09T17:50:41.758881+0000","last_became_active":"2024-02-09T17:43:19.103584+0000","last_became_peered":"2024-02-09T17:43:19.103584+0000","last_unstale":"2024-02-09T17:50:41.758881+0000","last_undegraded":"2024-02-09T17:50:41.758881+0000","last_fullsized":"2024-02-09T17:50:41.758881+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:50:31.454595+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:23.912504+0000","last_clean_scrub_stamp":"2024-02-09T17:50:31.454595+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:51:51.110326+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"63'139","reported_seq":948,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-02-09T17:50:41.758877+0000","last_change":"2024-02-09T17:50:33.641959+0000","last_active":"2024-02-09T17:50:41.758877+0000","last_peered":"2024-02-09T17:50:41.758877+0000","last_clean":"2024-02-09T17:50:41.758877+0000","last_became_active":"2024-02-09T17:43:19.102729+0000","last_became_peered":"2024-02-09T17:43:19.102729+0000","last_unstale":"2024-02-09T17:50:41.758877+0000","last_undegraded":"2024-02-09T17:50:41.758877+0000","last_fullsized":"2024-02-09T17:50:41.758877+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:50:33.641822+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:45:44.988633+0000","last_clean_scrub_stamp":"2024-02-09T17:50:33.641822+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:51:46.299796+0000","scrub_duration":164,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"63'139","reported_seq":609,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-09T17:50:40.795296+0000","last_change":"2024-02-09T17:49:36.820953+0000","last_active":"2024-02-09T17:50:40.795296+0000","last_peered":"2024-02-09T17:50:40.795296+0000","last_clean":"2024-02-09T17:50:40.795296+0000","last_became_active":"2024-02-09T17:43:19.120814+0000","last_became_peered":"2024-02-09T17:43:19.120814+0000","last_unstale":"2024-02-09T17:50:40.795296+0000","last_undegraded":"2024-02-09T17:50:40.795296+0000","last_fullsized":"2024-02-09T17:50:40.795296+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:49:36.820839+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:49:36.820839+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:56.220878+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"63'139","reported_seq":1443,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-09T17:50:36.755934+0000","last_change":"2024-02-09T17:50:23.652894+0000","last_active":"2024-02-09T17:50:36.755934+0000","last_peered":"2024-02-09T17:50:36.755934+0000","last_clean":"2024-02-09T17:50:36.755934+0000","last_became_active":"2024-02-09T17:43:19.110535+0000","last_became_peered":"2024-02-09T17:43:19.110535+0000","last_unstale":"2024-02-09T17:50:36.755934+0000","last_undegraded":"2024-02-09T17:50:36.755934+0000","last_fullsized":"2024-02-09T17:50:36.755934+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:50:23.652765+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:30.193194+0000","last_clean_scrub_stamp":"2024-02-09T17:50:23.652765+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:51:23.794048+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"63'139","reported_seq":1283,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-09T17:50:36.758406+0000","last_change":"2024-02-09T17:50:29.488810+0000","last_active":"2024-02-09T17:50:36.758406+0000","last_peered":"2024-02-09T17:50:36.758406+0000","last_clean":"2024-02-09T17:50:36.758406+0000","last_became_active":"2024-02-09T17:43:28.585573+0000","last_became_peered":"2024-02-09T17:43:28.585573+0000","last_unstale":"2024-02-09T17:50:36.758406+0000","last_undegraded":"2024-02-09T17:50:36.758406+0000","last_fullsized":"2024-02-09T17:50:36.758406+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:50:29.488679+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:50:29.488679+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:51:35.216499+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"63'139","reported_seq":741,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-02-09T17:50:41.759741+0000","last_change":"2024-02-09T17:50:32.563676+0000","last_active":"2024-02-09T17:50:41.759741+0000","last_peered":"2024-02-09T17:50:41.759741+0000","last_clean":"2024-02-09T17:50:41.759741+0000","last_became_active":"2024-02-09T17:48:46.184419+0000","last_became_peered":"2024-02-09T17:48:46.184419+0000","last_unstale":"2024-02-09T17:50:41.759741+0000","last_undegraded":"2024-02-09T17:50:41.759741+0000","last_fullsized":"2024-02-09T17:50:41.759741+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:50:32.563555+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:44:50.214686+0000","last_clean_scrub_stamp":"2024-02-09T17:50:32.563555+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:51:34.877103+0000","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"63'139","reported_seq":610,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-09T17:50:36.754020+0000","last_change":"2024-02-09T17:50:31.509946+0000","last_active":"2024-02-09T17:50:36.754020+0000","last_peered":"2024-02-09T17:50:36.754020+0000","last_clean":"2024-02-09T17:50:36.754020+0000","last_became_active":"2024-02-09T17:43:19.094230+0000","last_became_peered":"2024-02-09T17:43:19.094230+0000","last_unstale":"2024-02-09T17:50:36.754020+0000","last_undegraded":"2024-02-09T17:50:36.754020+0000","last_fullsized":"2024-02-09T17:50:36.754020+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:50:31.509861+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:11.970338+0000","last_clean_scrub_stamp":"2024-02-09T17:50:31.509861+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:51:53.596808+0000","scrub_duration":153,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"63'139","reported_seq":1061,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-02-09T17:50:41.760637+0000","last_change":"2024-02-09T17:50:17.104141+0000","last_active":"2024-02-09T17:50:41.760637+0000","last_peered":"2024-02-09T17:50:41.760637+0000","last_clean":"2024-02-09T17:50:41.760637+0000","last_became_active":"2024-02-09T17:43:19.138398+0000","last_became_peered":"2024-02-09T17:43:19.138398+0000","last_unstale":"2024-02-09T17:50:41.760637+0000","last_undegraded":"2024-02-09T17:50:41.760637+0000","last_fullsized":"2024-02-09T17:50:41.760637+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:50:17.104041+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:50:17.104041+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:51:40.633366+0000","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"63'139","reported_seq":1103,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-09T17:50:36.774550+0000","last_change":"2024-02-09T17:49:30.296419+0000","last_active":"2024-02-09T17:50:36.774550+0000","last_peered":"2024-02-09T17:50:36.774550+0000","last_clean":"2024-02-09T17:50:36.774550+0000","last_became_active":"2024-02-09T17:43:28.589307+0000","last_became_peered":"2024-02-09T17:43:28.589307+0000","last_unstale":"2024-02-09T17:50:36.774550+0000","last_undegraded":"2024-02-09T17:50:36.774550+0000","last_fullsized":"2024-02-09T17:50:36.774550+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:49:30.296287+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:38:26.968892+0000","last_clean_scrub_stamp":"2024-02-09T17:49:30.296287+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:48.296433+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"669'2020","reported_seq":2046,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-09T17:50:38.314626+0000","last_change":"2024-02-09T17:50:18.752622+0000","last_active":"2024-02-09T17:50:38.314626+0000","last_peered":"2024-02-09T17:50:38.314626+0000","last_clean":"2024-02-09T17:50:38.314626+0000","last_became_active":"2024-02-09T17:50:18.752333+0000","last_became_peered":"2024-02-09T17:50:18.752333+0000","last_unstale":"2024-02-09T17:50:38.314626+0000","last_undegraded":"2024-02-09T17:50:38.314626+0000","last_fullsized":"2024-02-09T17:50:38.314626+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_clean_scrub_stamp":"2024-02-09T17:50:17.733761+0000","objects_scrubbed":0,"log_size":2020,"log_dups_size":0,"ondisk_log_size":2020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:51:33.842098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132382720,"num_objects":2020,"num_object_clones":0,"num_object_copies":6060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2020,"num_write_kb":129280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"63'139","reported_seq":1709,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-02-09T17:50:41.760400+0000","last_change":"2024-02-09T17:50:19.968754+0000","last_active":"2024-02-09T17:50:41.760400+0000","last_peered":"2024-02-09T17:50:41.760400+0000","last_clean":"2024-02-09T17:50:41.760400+0000","last_became_active":"2024-02-09T17:41:14.521079+0000","last_became_peered":"2024-02-09T17:41:14.521079+0000","last_unstale":"2024-02-09T17:50:41.760400+0000","last_undegraded":"2024-02-09T17:50:41.760400+0000","last_fullsized":"2024-02-09T17:50:41.760400+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:50:19.968667+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:50:19.968667+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:51:37.352149+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"669'2032","reported_seq":2058,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-09T17:50:38.310515+0000","last_change":"2024-02-09T17:50:18.752003+0000","last_active":"2024-02-09T17:50:38.310515+0000","last_peered":"2024-02-09T17:50:38.310515+0000","last_clean":"2024-02-09T17:50:38.310515+0000","last_became_active":"2024-02-09T17:50:18.751714+0000","last_became_peered":"2024-02-09T17:50:18.751714+0000","last_unstale":"2024-02-09T17:50:38.310515+0000","last_undegraded":"2024-02-09T17:50:38.310515+0000","last_fullsized":"2024-02-09T17:50:38.310515+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_clean_scrub_stamp":"2024-02-09T17:50:17.733761+0000","objects_scrubbed":0,"log_size":2032,"log_dups_size":0,"ondisk_log_size":2032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:51:39.748934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133169152,"num_objects":2032,"num_object_clones":0,"num_object_copies":6096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2032,"num_write_kb":130048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"63'139","reported_seq":1271,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-09T17:50:36.751418+0000","last_change":"2024-02-09T17:50:21.778738+0000","last_active":"2024-02-09T17:50:36.751418+0000","last_peered":"2024-02-09T17:50:36.751418+0000","last_clean":"2024-02-09T17:50:36.751418+0000","last_became_active":"2024-02-09T17:43:39.981054+0000","last_became_peered":"2024-02-09T17:43:39.981054+0000","last_unstale":"2024-02-09T17:50:36.751418+0000","last_undegraded":"2024-02-09T17:50:36.751418+0000","last_fullsized":"2024-02-09T17:50:36.751418+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:50:21.778647+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:50:21.778647+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:51:47.621719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.5","version":"669'2387","reported_seq":2412,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-09T17:50:40.558787+0000","last_change":"2024-02-09T17:50:18.752557+0000","last_active":"2024-02-09T17:50:40.558787+0000","last_peered":"2024-02-09T17:50:40.558787+0000","last_clean":"2024-02-09T17:50:40.558787+0000","last_became_active":"2024-02-09T17:50:18.752186+0000","last_became_peered":"2024-02-09T17:50:18.752186+0000","last_unstale":"2024-02-09T17:50:40.558787+0000","last_undegraded":"2024-02-09T17:50:40.558787+0000","last_fullsized":"2024-02-09T17:50:40.558787+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_clean_scrub_stamp":"2024-02-09T17:50:17.733761+0000","objects_scrubbed":0,"log_size":2387,"log_dups_size":0,"ondisk_log_size":2387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:51:32.582151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156434432,"num_objects":2387,"num_object_clones":0,"num_object_copies":7161,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2387,"num_write_kb":152768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,0],"acting":[11,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.3","version":"63'139","reported_seq":894,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-09T17:50:36.754196+0000","last_change":"2024-02-09T17:50:35.557106+0000","last_active":"2024-02-09T17:50:36.754196+0000","last_peered":"2024-02-09T17:50:36.754196+0000","last_clean":"2024-02-09T17:50:36.754196+0000","last_became_active":"2024-02-09T17:40:13.501711+0000","last_became_peered":"2024-02-09T17:40:13.501711+0000","last_unstale":"2024-02-09T17:50:36.754196+0000","last_undegraded":"2024-02-09T17:50:36.754196+0000","last_fullsized":"2024-02-09T17:50:36.754196+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:50:35.556968+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:46:40.684762+0000","last_clean_scrub_stamp":"2024-02-09T17:50:35.556968+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:51:55.700214+0000","scrub_duration":188,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"669'2316","reported_seq":2342,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-09T17:50:40.335119+0000","last_change":"2024-02-09T17:50:18.752835+0000","last_active":"2024-02-09T17:50:40.335119+0000","last_peered":"2024-02-09T17:50:40.335119+0000","last_clean":"2024-02-09T17:50:40.335119+0000","last_became_active":"2024-02-09T17:50:18.752519+0000","last_became_peered":"2024-02-09T17:50:18.752519+0000","last_unstale":"2024-02-09T17:50:40.335119+0000","last_undegraded":"2024-02-09T17:50:40.335119+0000","last_fullsized":"2024-02-09T17:50:40.335119+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_clean_scrub_stamp":"2024-02-09T17:50:17.733761+0000","objects_scrubbed":0,"log_size":2316,"log_dups_size":0,"ondisk_log_size":2316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:51:34.407844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151781376,"num_objects":2316,"num_object_clones":0,"num_object_copies":6948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2316,"num_write_kb":148224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,3],"acting":[6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"63'139","reported_seq":420,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-09T17:50:36.755988+0000","last_change":"2024-02-09T17:50:14.694902+0000","last_active":"2024-02-09T17:50:36.755988+0000","last_peered":"2024-02-09T17:50:36.755988+0000","last_clean":"2024-02-09T17:50:36.755988+0000","last_became_active":"2024-02-09T17:48:46.207004+0000","last_became_peered":"2024-02-09T17:48:46.207004+0000","last_unstale":"2024-02-09T17:50:36.755988+0000","last_undegraded":"2024-02-09T17:50:36.755988+0000","last_fullsized":"2024-02-09T17:50:36.755988+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:50:14.694755+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:50:14.694755+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:51:35.214075+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.0","version":"671'2483","reported_seq":2513,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-02-09T17:50:41.799064+0000","last_change":"2024-02-09T17:50:19.468462+0000","last_active":"2024-02-09T17:50:41.799064+0000","last_peered":"2024-02-09T17:50:41.799064+0000","last_clean":"2024-02-09T17:50:41.799064+0000","last_became_active":"2024-02-09T17:50:19.467979+0000","last_became_peered":"2024-02-09T17:50:19.467979+0000","last_unstale":"2024-02-09T17:50:41.799064+0000","last_undegraded":"2024-02-09T17:50:41.799064+0000","last_fullsized":"2024-02-09T17:50:41.799064+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_clean_scrub_stamp":"2024-02-09T17:50:17.733761+0000","objects_scrubbed":0,"log_size":2483,"log_dups_size":0,"ondisk_log_size":2483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:51:20.165965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162725888,"num_objects":2483,"num_object_clones":0,"num_object_copies":7449,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2483,"num_write_kb":158912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,3],"acting":[9,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.6","version":"63'139","reported_seq":1206,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-09T17:50:36.755687+0000","last_change":"2024-02-09T17:49:35.724086+0000","last_active":"2024-02-09T17:50:36.755687+0000","last_peered":"2024-02-09T17:50:36.755687+0000","last_clean":"2024-02-09T17:50:36.755687+0000","last_became_active":"2024-02-09T17:43:14.024894+0000","last_became_peered":"2024-02-09T17:43:14.024894+0000","last_unstale":"2024-02-09T17:50:36.755687+0000","last_undegraded":"2024-02-09T17:50:36.755687+0000","last_fullsized":"2024-02-09T17:50:36.755687+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:49:35.723959+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:11.970338+0000","last_clean_scrub_stamp":"2024-02-09T17:49:35.723959+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:44.719263+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.1","version":"670'2452","reported_seq":2480,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-09T17:50:41.104012+0000","last_change":"2024-02-09T17:50:19.468610+0000","last_active":"2024-02-09T17:50:41.104012+0000","last_peered":"2024-02-09T17:50:41.104012+0000","last_clean":"2024-02-09T17:50:41.104012+0000","last_became_active":"2024-02-09T17:50:19.468190+0000","last_became_peered":"2024-02-09T17:50:19.468190+0000","last_unstale":"2024-02-09T17:50:41.104012+0000","last_undegraded":"2024-02-09T17:50:41.104012+0000","last_fullsized":"2024-02-09T17:50:41.104012+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_clean_scrub_stamp":"2024-02-09T17:50:17.733761+0000","objects_scrubbed":0,"log_size":2452,"log_dups_size":0,"ondisk_log_size":2452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:51:42.341902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160694272,"num_objects":2452,"num_object_clones":0,"num_object_copies":7356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2452,"num_write_kb":156928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,3],"acting":[4,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"63'139","reported_seq":877,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-09T17:50:38.171923+0000","last_change":"2024-02-09T17:50:38.171964+0000","last_active":"2024-02-09T17:50:38.171923+0000","last_peered":"2024-02-09T17:50:38.171923+0000","last_clean":"2024-02-09T17:50:38.171923+0000","last_became_active":"2024-02-09T17:43:14.038885+0000","last_became_peered":"2024-02-09T17:43:14.038885+0000","last_unstale":"2024-02-09T17:50:38.171923+0000","last_undegraded":"2024-02-09T17:50:38.171923+0000","last_fullsized":"2024-02-09T17:50:38.171923+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:50:38.171846+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:50:38.171846+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:52:06.414099+0000","scrub_duration":182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2","version":"671'2548","reported_seq":2578,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-02-09T17:50:42.745882+0000","last_change":"2024-02-09T17:50:19.468360+0000","last_active":"2024-02-09T17:50:42.745882+0000","last_peered":"2024-02-09T17:50:42.745882+0000","last_clean":"2024-02-09T17:50:42.745882+0000","last_became_active":"2024-02-09T17:50:19.468029+0000","last_became_peered":"2024-02-09T17:50:19.468029+0000","last_unstale":"2024-02-09T17:50:42.745882+0000","last_undegraded":"2024-02-09T17:50:42.745882+0000","last_fullsized":"2024-02-09T17:50:42.745882+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_clean_scrub_stamp":"2024-02-09T17:50:17.733761+0000","objects_scrubbed":0,"log_size":2548,"log_dups_size":0,"ondisk_log_size":2548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:51:45.148706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166985728,"num_objects":2548,"num_object_clones":0,"num_object_copies":7644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2548,"num_write_kb":163072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,2],"acting":[3,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"63'139","reported_seq":1284,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-09T17:50:36.752079+0000","last_change":"2024-02-09T17:50:34.344695+0000","last_active":"2024-02-09T17:50:36.752079+0000","last_peered":"2024-02-09T17:50:36.752079+0000","last_clean":"2024-02-09T17:50:36.752079+0000","last_became_active":"2024-02-09T17:33:00.284924+0000","last_became_peered":"2024-02-09T17:33:00.284924+0000","last_unstale":"2024-02-09T17:50:36.752079+0000","last_undegraded":"2024-02-09T17:50:36.752079+0000","last_fullsized":"2024-02-09T17:50:36.752079+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:50:34.344581+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:56.105792+0000","last_clean_scrub_stamp":"2024-02-09T17:50:34.344581+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:52:00.438291+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.3","version":"671'2549","reported_seq":2579,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-02-09T17:50:42.741875+0000","last_change":"2024-02-09T17:50:18.752162+0000","last_active":"2024-02-09T17:50:42.741875+0000","last_peered":"2024-02-09T17:50:42.741875+0000","last_clean":"2024-02-09T17:50:42.741875+0000","last_became_active":"2024-02-09T17:50:18.751775+0000","last_became_peered":"2024-02-09T17:50:18.751775+0000","last_unstale":"2024-02-09T17:50:42.741875+0000","last_undegraded":"2024-02-09T17:50:42.741875+0000","last_fullsized":"2024-02-09T17:50:42.741875+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_clean_scrub_stamp":"2024-02-09T17:50:17.733761+0000","objects_scrubbed":0,"log_size":2549,"log_dups_size":0,"ondisk_log_size":2549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:51:45.698773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167051264,"num_objects":2549,"num_object_clones":0,"num_object_copies":7647,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2549,"num_write_kb":163136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"63'139","reported_seq":1125,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-09T17:50:36.755504+0000","last_change":"2024-02-09T17:49:36.733644+0000","last_active":"2024-02-09T17:50:36.755504+0000","last_peered":"2024-02-09T17:50:36.755504+0000","last_clean":"2024-02-09T17:50:36.755504+0000","last_became_active":"2024-02-09T17:48:46.184307+0000","last_became_peered":"2024-02-09T17:48:46.184307+0000","last_unstale":"2024-02-09T17:50:36.755504+0000","last_undegraded":"2024-02-09T17:50:36.755504+0000","last_fullsized":"2024-02-09T17:50:36.755504+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:49:36.733515+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:49:36.733515+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:42.342994+0000","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.c","version":"669'2331","reported_seq":2356,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-09T17:50:40.368517+0000","last_change":"2024-02-09T17:50:18.753679+0000","last_active":"2024-02-09T17:50:40.368517+0000","last_peered":"2024-02-09T17:50:40.368517+0000","last_clean":"2024-02-09T17:50:40.368517+0000","last_became_active":"2024-02-09T17:50:18.753348+0000","last_became_peered":"2024-02-09T17:50:18.753348+0000","last_unstale":"2024-02-09T17:50:40.368517+0000","last_undegraded":"2024-02-09T17:50:40.368517+0000","last_fullsized":"2024-02-09T17:50:40.368517+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_clean_scrub_stamp":"2024-02-09T17:50:17.733761+0000","objects_scrubbed":0,"log_size":2331,"log_dups_size":0,"ondisk_log_size":2331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:51:38.632080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152764416,"num_objects":2331,"num_object_clones":0,"num_object_copies":6993,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2331,"num_write_kb":149184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,11],"acting":[6,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"63'139","reported_seq":1203,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-02-09T17:50:41.763103+0000","last_change":"2024-02-09T17:49:39.575691+0000","last_active":"2024-02-09T17:50:41.763103+0000","last_peered":"2024-02-09T17:50:41.763103+0000","last_clean":"2024-02-09T17:50:41.763103+0000","last_became_active":"2024-02-09T17:43:14.041291+0000","last_became_peered":"2024-02-09T17:43:14.041291+0000","last_unstale":"2024-02-09T17:50:41.763103+0000","last_undegraded":"2024-02-09T17:50:41.763103+0000","last_fullsized":"2024-02-09T17:50:41.763103+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:49:39.575571+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:49:39.575571+0000","last_clean_scrub_stamp":"2024-02-09T17:49:39.575571+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:45.375863+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.d","version":"669'2342","reported_seq":2368,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-09T17:50:40.362392+0000","last_change":"2024-02-09T17:50:18.752741+0000","last_active":"2024-02-09T17:50:40.362392+0000","last_peered":"2024-02-09T17:50:40.362392+0000","last_clean":"2024-02-09T17:50:40.362392+0000","last_became_active":"2024-02-09T17:50:18.752364+0000","last_became_peered":"2024-02-09T17:50:18.752364+0000","last_unstale":"2024-02-09T17:50:40.362392+0000","last_undegraded":"2024-02-09T17:50:40.362392+0000","last_fullsized":"2024-02-09T17:50:40.362392+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_clean_scrub_stamp":"2024-02-09T17:50:17.733761+0000","objects_scrubbed":0,"log_size":2342,"log_dups_size":0,"ondisk_log_size":2342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:51:28.313257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153485312,"num_objects":2342,"num_object_clones":0,"num_object_copies":7026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2342,"num_write_kb":149888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"63'139","reported_seq":1049,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-09T17:50:36.752172+0000","last_change":"2024-02-09T17:50:22.355807+0000","last_active":"2024-02-09T17:50:36.752172+0000","last_peered":"2024-02-09T17:50:36.752172+0000","last_clean":"2024-02-09T17:50:36.752172+0000","last_became_active":"2024-02-09T17:43:14.054943+0000","last_became_peered":"2024-02-09T17:43:14.054943+0000","last_unstale":"2024-02-09T17:50:36.752172+0000","last_undegraded":"2024-02-09T17:50:36.752172+0000","last_fullsized":"2024-02-09T17:50:36.752172+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:50:22.355668+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:50:22.355668+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:51:37.874130+0000","scrub_duration":145,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.e","version":"669'2261","reported_seq":2287,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-09T17:50:40.134818+0000","last_change":"2024-02-09T17:50:19.468403+0000","last_active":"2024-02-09T17:50:40.134818+0000","last_peered":"2024-02-09T17:50:40.134818+0000","last_clean":"2024-02-09T17:50:40.134818+0000","last_became_active":"2024-02-09T17:50:19.468067+0000","last_became_peered":"2024-02-09T17:50:19.468067+0000","last_unstale":"2024-02-09T17:50:40.134818+0000","last_undegraded":"2024-02-09T17:50:40.134818+0000","last_fullsized":"2024-02-09T17:50:40.134818+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_clean_scrub_stamp":"2024-02-09T17:50:17.733761+0000","objects_scrubbed":0,"log_size":2261,"log_dups_size":0,"ondisk_log_size":2261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:51:19.544919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148176896,"num_objects":2261,"num_object_clones":0,"num_object_copies":6783,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2261,"num_write_kb":144704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"63'139","reported_seq":844,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-02-09T17:50:41.762397+0000","last_change":"2024-02-09T17:50:16.061037+0000","last_active":"2024-02-09T17:50:41.762397+0000","last_peered":"2024-02-09T17:50:41.762397+0000","last_clean":"2024-02-09T17:50:41.762397+0000","last_became_active":"2024-02-09T17:46:47.013853+0000","last_became_peered":"2024-02-09T17:46:47.013853+0000","last_unstale":"2024-02-09T17:50:41.762397+0000","last_undegraded":"2024-02-09T17:50:41.762397+0000","last_fullsized":"2024-02-09T17:50:41.762397+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:50:16.060909+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:50:16.060909+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:51:17.790190+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"671'2491","reported_seq":2521,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-02-09T17:50:42.045644+0000","last_change":"2024-02-09T17:50:18.750226+0000","last_active":"2024-02-09T17:50:42.045644+0000","last_peered":"2024-02-09T17:50:42.045644+0000","last_clean":"2024-02-09T17:50:42.045644+0000","last_became_active":"2024-02-09T17:50:18.749879+0000","last_became_peered":"2024-02-09T17:50:18.749879+0000","last_unstale":"2024-02-09T17:50:42.045644+0000","last_undegraded":"2024-02-09T17:50:42.045644+0000","last_fullsized":"2024-02-09T17:50:42.045644+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_clean_scrub_stamp":"2024-02-09T17:50:17.733761+0000","objects_scrubbed":0,"log_size":2491,"log_dups_size":0,"ondisk_log_size":2491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:51:39.973378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163250176,"num_objects":2491,"num_object_clones":0,"num_object_copies":7473,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2491,"num_write_kb":159424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.9","version":"63'139","reported_seq":1716,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-02-09T17:50:41.760350+0000","last_change":"2024-02-09T17:50:15.083453+0000","last_active":"2024-02-09T17:50:41.760350+0000","last_peered":"2024-02-09T17:50:41.760350+0000","last_clean":"2024-02-09T17:50:41.760350+0000","last_became_active":"2024-02-09T17:43:28.590237+0000","last_became_peered":"2024-02-09T17:43:28.590237+0000","last_unstale":"2024-02-09T17:50:41.760350+0000","last_undegraded":"2024-02-09T17:50:41.760350+0000","last_fullsized":"2024-02-09T17:50:41.760350+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:50:15.083356+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:45:47.710252+0000","last_clean_scrub_stamp":"2024-02-09T17:50:15.083356+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:51:34.398905+0000","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"671'2577","reported_seq":2607,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-02-09T17:50:42.782841+0000","last_change":"2024-02-09T17:50:18.754173+0000","last_active":"2024-02-09T17:50:42.782841+0000","last_peered":"2024-02-09T17:50:42.782841+0000","last_clean":"2024-02-09T17:50:42.782841+0000","last_became_active":"2024-02-09T17:50:18.753808+0000","last_became_peered":"2024-02-09T17:50:18.753808+0000","last_unstale":"2024-02-09T17:50:42.782841+0000","last_undegraded":"2024-02-09T17:50:42.782841+0000","last_fullsized":"2024-02-09T17:50:42.782841+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_clean_scrub_stamp":"2024-02-09T17:50:17.733761+0000","objects_scrubbed":0,"log_size":2577,"log_dups_size":0,"ondisk_log_size":2577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:51:41.579971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168886272,"num_objects":2577,"num_object_clones":0,"num_object_copies":7731,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2577,"num_write_kb":164928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"63'139","reported_seq":1170,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-02-09T17:50:40.795390+0000","last_change":"2024-02-09T17:50:16.807981+0000","last_active":"2024-02-09T17:50:40.795390+0000","last_peered":"2024-02-09T17:50:40.795390+0000","last_clean":"2024-02-09T17:50:40.795390+0000","last_became_active":"2024-02-09T17:43:14.036583+0000","last_became_peered":"2024-02-09T17:43:14.036583+0000","last_unstale":"2024-02-09T17:50:40.795390+0000","last_undegraded":"2024-02-09T17:50:40.795390+0000","last_fullsized":"2024-02-09T17:50:40.795390+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:50:16.807851+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:14.981453+0000","last_clean_scrub_stamp":"2024-02-09T17:50:16.807851+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:51:35.192992+0000","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"669'2046","reported_seq":2072,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-09T17:50:38.316922+0000","last_change":"2024-02-09T17:50:18.751019+0000","last_active":"2024-02-09T17:50:38.316922+0000","last_peered":"2024-02-09T17:50:38.316922+0000","last_clean":"2024-02-09T17:50:38.316922+0000","last_became_active":"2024-02-09T17:50:18.750735+0000","last_became_peered":"2024-02-09T17:50:18.750735+0000","last_unstale":"2024-02-09T17:50:38.316922+0000","last_undegraded":"2024-02-09T17:50:38.316922+0000","last_fullsized":"2024-02-09T17:50:38.316922+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_clean_scrub_stamp":"2024-02-09T17:50:17.733761+0000","objects_scrubbed":0,"log_size":2046,"log_dups_size":0,"ondisk_log_size":2046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:51:37.628810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134086656,"num_objects":2046,"num_object_clones":0,"num_object_copies":6138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2046,"num_write_kb":130944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,8],"acting":[0,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"63'139","reported_seq":1229,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-09T17:50:36.753900+0000","last_change":"2024-02-09T17:50:16.087650+0000","last_active":"2024-02-09T17:50:36.753900+0000","last_peered":"2024-02-09T17:50:36.753900+0000","last_clean":"2024-02-09T17:50:36.753900+0000","last_became_active":"2024-02-09T17:33:05.316100+0000","last_became_peered":"2024-02-09T17:33:05.316100+0000","last_unstale":"2024-02-09T17:50:36.753900+0000","last_undegraded":"2024-02-09T17:50:36.753900+0000","last_fullsized":"2024-02-09T17:50:36.753900+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:50:16.087552+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:48:34.651647+0000","last_clean_scrub_stamp":"2024-02-09T17:50:16.087552+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:51:22.060680+0000","scrub_duration":141,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"671'2504","reported_seq":2534,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-02-09T17:50:41.810096+0000","last_change":"2024-02-09T17:50:18.752322+0000","last_active":"2024-02-09T17:50:41.810096+0000","last_peered":"2024-02-09T17:50:41.810096+0000","last_clean":"2024-02-09T17:50:41.810096+0000","last_became_active":"2024-02-09T17:50:18.752003+0000","last_became_peered":"2024-02-09T17:50:18.752003+0000","last_unstale":"2024-02-09T17:50:41.810096+0000","last_undegraded":"2024-02-09T17:50:41.810096+0000","last_fullsized":"2024-02-09T17:50:41.810096+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_clean_scrub_stamp":"2024-02-09T17:50:17.733761+0000","objects_scrubbed":0,"log_size":2504,"log_dups_size":0,"ondisk_log_size":2504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:51:44.385573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164102144,"num_objects":2504,"num_object_clones":0,"num_object_copies":7512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2504,"num_write_kb":160256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.c","version":"63'139","reported_seq":1351,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-02-09T17:50:41.762686+0000","last_change":"2024-02-09T17:49:11.961487+0000","last_active":"2024-02-09T17:50:41.762686+0000","last_peered":"2024-02-09T17:50:41.762686+0000","last_clean":"2024-02-09T17:50:41.762686+0000","last_became_active":"2024-02-09T17:43:14.027715+0000","last_became_peered":"2024-02-09T17:43:14.027715+0000","last_unstale":"2024-02-09T17:50:41.762686+0000","last_undegraded":"2024-02-09T17:50:41.762686+0000","last_fullsized":"2024-02-09T17:50:41.762686+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:49:11.961373+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:14.135321+0000","last_clean_scrub_stamp":"2024-02-09T17:49:11.961373+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"669'2231","reported_seq":2256,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-09T17:50:39.960230+0000","last_change":"2024-02-09T17:50:19.468334+0000","last_active":"2024-02-09T17:50:39.960230+0000","last_peered":"2024-02-09T17:50:39.960230+0000","last_clean":"2024-02-09T17:50:39.960230+0000","last_became_active":"2024-02-09T17:50:19.467991+0000","last_became_peered":"2024-02-09T17:50:19.467991+0000","last_unstale":"2024-02-09T17:50:39.960230+0000","last_undegraded":"2024-02-09T17:50:39.960230+0000","last_fullsized":"2024-02-09T17:50:39.960230+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_clean_scrub_stamp":"2024-02-09T17:50:17.733761+0000","objects_scrubbed":0,"log_size":2231,"log_dups_size":0,"ondisk_log_size":2231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:51:21.190376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146210816,"num_objects":2231,"num_object_clones":0,"num_object_copies":6693,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2231,"num_write_kb":142784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"63'139","reported_seq":764,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-02-09T17:50:41.760698+0000","last_change":"2024-02-09T17:50:18.057735+0000","last_active":"2024-02-09T17:50:41.760698+0000","last_peered":"2024-02-09T17:50:41.760698+0000","last_clean":"2024-02-09T17:50:41.760698+0000","last_became_active":"2024-02-09T17:43:14.036759+0000","last_became_peered":"2024-02-09T17:43:14.036759+0000","last_unstale":"2024-02-09T17:50:41.760698+0000","last_undegraded":"2024-02-09T17:50:41.760698+0000","last_fullsized":"2024-02-09T17:50:41.760698+0000","mapping_epoch":454,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:50:18.057634+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:50:18.057634+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:51:19.829150+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"63'139","reported_seq":936,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-02-09T17:50:41.760300+0000","last_change":"2024-02-09T17:50:21.030323+0000","last_active":"2024-02-09T17:50:41.760300+0000","last_peered":"2024-02-09T17:50:41.760300+0000","last_clean":"2024-02-09T17:50:41.760300+0000","last_became_active":"2024-02-09T17:46:47.013920+0000","last_became_peered":"2024-02-09T17:46:47.013920+0000","last_unstale":"2024-02-09T17:50:41.760300+0000","last_undegraded":"2024-02-09T17:50:41.760300+0000","last_fullsized":"2024-02-09T17:50:41.760300+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:50:21.030206+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:48:35.170638+0000","last_clean_scrub_stamp":"2024-02-09T17:50:21.030206+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:51:29.692519+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"63'139","reported_seq":1180,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-02-09T17:50:41.759567+0000","last_change":"2024-02-09T17:49:31.135888+0000","last_active":"2024-02-09T17:50:41.759567+0000","last_peered":"2024-02-09T17:50:41.759567+0000","last_clean":"2024-02-09T17:50:41.759567+0000","last_became_active":"2024-02-09T17:34:01.353171+0000","last_became_peered":"2024-02-09T17:34:01.353171+0000","last_unstale":"2024-02-09T17:50:41.759567+0000","last_undegraded":"2024-02-09T17:50:41.759567+0000","last_fullsized":"2024-02-09T17:50:41.759567+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:49:31.135755+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:44:51.190789+0000","last_clean_scrub_stamp":"2024-02-09T17:49:31.135755+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:50:58.374348+0000","scrub_duration":343,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.12","version":"63'139","reported_seq":1269,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-09T17:50:36.753912+0000","last_change":"2024-02-09T17:50:23.374531+0000","last_active":"2024-02-09T17:50:36.753912+0000","last_peered":"2024-02-09T17:50:36.753912+0000","last_clean":"2024-02-09T17:50:36.753912+0000","last_became_active":"2024-02-09T17:46:47.015622+0000","last_became_peered":"2024-02-09T17:46:47.015622+0000","last_unstale":"2024-02-09T17:50:36.753912+0000","last_undegraded":"2024-02-09T17:50:36.753912+0000","last_fullsized":"2024-02-09T17:50:36.753912+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:50:23.374413+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:50:23.374413+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:51:25.784181+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.13","version":"63'139","reported_seq":888,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-02-09T17:50:38.611098+0000","last_change":"2024-02-09T17:50:38.611124+0000","last_active":"2024-02-09T17:50:38.611098+0000","last_peered":"2024-02-09T17:50:38.611098+0000","last_clean":"2024-02-09T17:50:38.611098+0000","last_became_active":"2024-02-09T17:41:14.521672+0000","last_became_peered":"2024-02-09T17:41:14.521672+0000","last_unstale":"2024-02-09T17:50:38.611098+0000","last_undegraded":"2024-02-09T17:50:38.611098+0000","last_fullsized":"2024-02-09T17:50:38.611098+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:50:38.611046+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:50:38.611046+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:51:49.743583+0000","scrub_duration":202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":2462187520,"num_objects":37570,"num_object_clones":0,"num_object_copies":112710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37570,"num_write_kb":2404480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7369981952,"data_stored":7369981952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37570,"ondisk_log_size":37570,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056981,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1368072,"kb_used_data":737152,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":92377080,"statfs":{"total":95995035648,"available":94594129920,"internally_reserved":0,"allocated":754843648,"data_stored":754514703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":646052292},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.5900000000000001,"5min":1.1080000000000001,"15min":1.3100000000000001},"min":{"1min":0.25800000000000001,"5min":0.24199999999999999,"15min":0.216},"max":{"1min":16.050999999999998,"5min":16.050999999999998,"15min":16.050999999999998},"last":1.49},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.1459999999999999,"15min":1.163},"min":{"1min":0.25700000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":4.0439999999999996,"5min":5.4189999999999996,"15min":9.8219999999999992},"last":2.21}]},{"osd":1,"last update":"Fri Feb 9 17:49:54 2024","interfaces":[{"interface":"back","average":{"1min":2.1800000000000002,"5min":1.5509999999999999,"15min":1.4950000000000001},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.22700000000000001},"max":{"1min":11.576000000000001,"5min":11.576000000000001,"15min":11.576000000000001},"last":1.5029999999999999},{"interface":"front","average":{"1min":3.1030000000000002,"5min":1.835,"15min":1.661},"min":{"1min":0.31900000000000001,"5min":0.27700000000000002,"15min":0.26400000000000001},"max":{"1min":11.595000000000001,"5min":11.595000000000001,"15min":11.595000000000001},"last":0.91900000000000004}]},{"osd":2,"last update":"Fri Feb 9 17:31:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.415,"15min":1.6859999999999999},"min":{"1min":0.378,"5min":0.247,"15min":0.247},"max":{"1min":4.4189999999999996,"5min":9.4719999999999995,"15min":9.4719999999999995},"last":0.35399999999999998},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.5149999999999999,"15min":1.4399999999999999},"min":{"1min":0.378,"5min":0.252,"15min":0.252},"max":{"1min":4.8579999999999997,"5min":7.96,"15min":7.96},"last":0.60399999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:49:54 2024","interfaces":[{"interface":"back","average":{"1min":2.7000000000000002,"5min":1.9330000000000001,"15min":1.4890000000000001},"min":{"1min":0.32700000000000001,"5min":0.248,"15min":0.248},"max":{"1min":8.5950000000000006,"5min":21.475000000000001,"15min":21.475000000000001},"last":0.315},{"interface":"front","average":{"1min":2.7240000000000002,"5min":2.0270000000000001,"15min":1.7829999999999999},"min":{"1min":0.441,"5min":0.35599999999999998,"15min":0.29799999999999999},"max":{"1min":8.5549999999999997,"5min":21.498000000000001,"15min":21.498000000000001},"last":0.41499999999999998}]},{"osd":4,"last update":"Fri Feb 9 17:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":2.0779999999999998,"15min":1.6539999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.246},"max":{"1min":4.7750000000000004,"5min":65.109999999999999,"15min":65.109999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":1.883,"5min":2.0619999999999998,"15min":1.734},"min":{"1min":0.36799999999999999,"5min":0.35299999999999998,"15min":0.309},"max":{"1min":6.5039999999999996,"5min":65.099000000000004,"15min":65.099000000000004},"last":1.4750000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:49:54 2024","interfaces":[{"interface":"back","average":{"1min":2.1739999999999999,"5min":1.4630000000000001,"15min":1.4790000000000001},"min":{"1min":0.432,"5min":0.252,"15min":0.252},"max":{"1min":7.3689999999999998,"5min":18.878,"15min":18.878},"last":0.44900000000000001},{"interface":"front","average":{"1min":1.286,"5min":1.48,"15min":1.46},"min":{"1min":0.33300000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":4.1180000000000003,"5min":18.895,"15min":18.895},"last":0.57999999999999996}]},{"osd":6,"last update":"Fri Feb 9 17:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.792,"5min":1.323,"15min":1.595},"min":{"1min":0.40000000000000002,"5min":0.30399999999999999,"15min":0.29899999999999999},"max":{"1min":5.0540000000000003,"5min":6.4020000000000001,"15min":12.362},"last":1.526},{"interface":"front","average":{"1min":2.3420000000000001,"5min":1.3919999999999999,"15min":1.6180000000000001},"min":{"1min":0.48099999999999998,"5min":0.36699999999999999,"15min":0.32700000000000001},"max":{"1min":7.3090000000000002,"5min":7.3090000000000002,"15min":10.000999999999999},"last":0.47299999999999998}]},{"osd":7,"last update":"Fri Feb 9 17:49:54 2024","interfaces":[{"interface":"back","average":{"1min":2.0880000000000001,"5min":1.728,"15min":1.667},"min":{"1min":0.34699999999999998,"5min":0.32000000000000001,"15min":0.312},"max":{"1min":9.0500000000000007,"5min":18.846,"15min":20.489999999999998},"last":0.49399999999999999},{"interface":"front","average":{"1min":2.1219999999999999,"5min":1.794,"15min":1.613},"min":{"1min":0.40899999999999997,"5min":0.36799999999999999,"15min":0.34200000000000003},"max":{"1min":9.0350000000000001,"5min":19.125,"15min":20.712},"last":0.60399999999999998}]},{"osd":8,"last update":"Fri Feb 9 17:50:03 2024","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":0.69799999999999995,"15min":1.0800000000000001},"min":{"1min":0.28299999999999997,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":5.1600000000000001,"5min":5.1600000000000001,"15min":9.5429999999999993},"last":1.4430000000000001},{"interface":"front","average":{"1min":0.97899999999999998,"5min":0.76500000000000001,"15min":1.1439999999999999},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":5.1760000000000002,"5min":5.1760000000000002,"15min":9.2940000000000005},"last":1.571}]},{"osd":9,"last update":"Fri Feb 9 17:50:36 2024","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":0.96599999999999997,"15min":1.3979999999999999},"min":{"1min":0.313,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":5.492,"5min":7.2720000000000002,"15min":62.518000000000001},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.96099999999999997,"15min":1.3640000000000001},"min":{"1min":0.35299999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":2.8340000000000001,"5min":7.3449999999999998,"15min":63.241999999999997},"last":0.80300000000000005}]},{"osd":10,"last update":"Fri Feb 9 17:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":0.89300000000000002,"15min":1.252},"min":{"1min":0.442,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":5.556,"5min":5.556,"15min":9.6590000000000007},"last":1.548},{"interface":"front","average":{"1min":1.5680000000000001,"5min":1.1060000000000001,"15min":1.296},"min":{"1min":0.503,"5min":0.377,"15min":0.35799999999999998},"max":{"1min":7.391,"5min":7.391,"15min":12.409000000000001},"last":0.53900000000000003}]}]},{"osd":10,"up_from":84,"seq":360777253211,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":847696,"kb_used_data":375040,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":92897456,"statfs":{"total":95995035648,"available":95126994944,"internally_reserved":0,"allocated":384040960,"data_stored":383712015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":483981760},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.0569999999999999,"15min":1.173},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.20599999999999999},"max":{"1min":4.7430000000000003,"5min":7.4290000000000003,"15min":9.343},"last":0.60499999999999998},{"interface":"front","average":{"1min":1.581,"5min":1.0680000000000001,"15min":1.085},"min":{"1min":0.28000000000000003,"5min":0.248,"15min":0.248},"max":{"1min":5.4169999999999998,"5min":7.2830000000000004,"15min":7.2830000000000004},"last":0.53600000000000003}]},{"osd":1,"last update":"Fri Feb 9 17:50:01 2024","interfaces":[{"interface":"back","average":{"1min":1.9450000000000001,"5min":1.4299999999999999,"15min":0.96499999999999997},"min":{"1min":0.246,"5min":0.246,"15min":0.23999999999999999},"max":{"1min":9.3819999999999997,"5min":9.3819999999999997,"15min":9.3819999999999997},"last":0.54500000000000004},{"interface":"front","average":{"1min":1.6639999999999999,"5min":1.462,"15min":1.1259999999999999},"min":{"1min":0.26900000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":5.1559999999999997,"5min":8.9789999999999992,"15min":8.9789999999999992},"last":0.63500000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:50:15 2024","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.6699999999999999,"15min":1.544},"min":{"1min":0.371,"5min":0.28299999999999997,"15min":0.27600000000000002},"max":{"1min":5.4080000000000004,"5min":9.2539999999999996,"15min":11.384},"last":0.57099999999999995},{"interface":"front","average":{"1min":1.796,"5min":1.9330000000000001,"15min":1.466},"min":{"1min":0.32300000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":8.4459999999999997,"5min":38.868000000000002,"15min":38.868000000000002},"last":0.65000000000000002}]},{"osd":3,"last update":"Fri Feb 9 17:50:01 2024","interfaces":[{"interface":"back","average":{"1min":2.081,"5min":2.0590000000000002,"15min":1.76},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":7.3440000000000003,"5min":35.113,"15min":35.113},"last":0.55400000000000005},{"interface":"front","average":{"1min":2.653,"5min":2.339,"15min":1.9610000000000001},"min":{"1min":0.29999999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":14.276999999999999,"5min":34.359000000000002,"15min":34.359000000000002},"last":0.61899999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:50:26 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":0.97699999999999998,"15min":1.1180000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29199999999999998},"max":{"1min":6.851,"5min":6.851,"15min":9.0990000000000002},"last":0.316},{"interface":"front","average":{"1min":1.5269999999999999,"5min":1.0649999999999999,"15min":1.167},"min":{"1min":0.36299999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":8.4610000000000003,"5min":8.4610000000000003,"15min":11.273999999999999},"last":0.376}]},{"osd":5,"last update":"Fri Feb 9 17:50:13 2024","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.2130000000000001,"15min":1.202},"min":{"1min":0.35299999999999998,"5min":0.29099999999999998,"15min":0.28799999999999998},"max":{"1min":7.9900000000000002,"5min":7.9900000000000002,"15min":7.9900000000000002},"last":0.499},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.25,"15min":1.3620000000000001},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":8.0060000000000002,"5min":8.0060000000000002,"15min":10.243},"last":0.44}]},{"osd":6,"last update":"Fri Feb 9 17:50:13 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.294,"15min":1.4510000000000001},"min":{"1min":0.38700000000000001,"5min":0.33600000000000002,"15min":0.308},"max":{"1min":5.218,"5min":9.0820000000000007,"15min":10.962},"last":0.45500000000000002},{"interface":"front","average":{"1min":1.601,"5min":1.24,"15min":1.367},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":5.7130000000000001,"5min":5.7130000000000001,"15min":9.0690000000000008},"last":0.51200000000000001}]},{"osd":7,"last update":"Fri Feb 9 17:39:52 2024","interfaces":[{"interface":"back","average":{"1min":1.6000000000000001,"5min":1.367,"15min":1.415},"min":{"1min":0.42099999999999999,"5min":0.34200000000000003,"15min":0.312},"max":{"1min":4.0430000000000001,"5min":6.7590000000000003,"15min":17.754000000000001},"last":4.6449999999999996},{"interface":"front","average":{"1min":1.498,"5min":1.4359999999999999,"15min":1.5189999999999999},"min":{"1min":0.47399999999999998,"5min":0.38300000000000001,"15min":0.316},"max":{"1min":5.8170000000000002,"5min":11.308999999999999,"15min":11.308999999999999},"last":0.55000000000000004}]},{"osd":8,"last update":"Fri Feb 9 17:50:01 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.82599999999999996,"15min":0.98199999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.29099999999999998},"max":{"1min":1.4850000000000001,"5min":4.6550000000000002,"15min":7.4359999999999999},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.66400000000000003,"5min":1.006,"15min":1.0780000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":2.2970000000000002,"5min":7.0460000000000003,"15min":7.7060000000000004},"last":0.52200000000000002}]},{"osd":9,"last update":"Fri Feb 9 17:50:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.84699999999999998,"15min":1.0449999999999999},"min":{"1min":0.35299999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":2.29,"5min":4.7000000000000002,"15min":7.4850000000000003},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.68200000000000005,"5min":1.004,"15min":1.1990000000000001},"min":{"1min":0.41199999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.667,"5min":6.585,"15min":10.622999999999999},"last":0.42399999999999999}]},{"osd":11,"last update":"Fri Feb 9 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":0.873,"15min":0.96299999999999997},"min":{"1min":0.40500000000000003,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":3.5990000000000002,"5min":3.5990000000000002,"15min":6.226},"last":0.48399999999999999},{"interface":"front","average":{"1min":1.0669999999999999,"5min":0.92500000000000004,"15min":1.0089999999999999},"min":{"1min":0.48299999999999998,"5min":0.438,"15min":0.438},"max":{"1min":3.625,"5min":3.625,"15min":6.2590000000000003},"last":0.58699999999999997}]}]},{"osd":9,"up_from":77,"seq":330712482144,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1364248,"kb_used_data":769356,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":92380904,"statfs":{"total":95995035648,"available":94598045696,"internally_reserved":0,"allocated":787820544,"data_stored":787487051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":609089984},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:50:10 2024","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":1.165,"15min":0.89800000000000002},"min":{"1min":0.25700000000000001,"5min":0.215,"15min":0.215},"max":{"1min":3.2250000000000001,"5min":8.4670000000000005,"15min":8.4670000000000005},"last":0.58599999999999997},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.3100000000000001,"15min":1.1919999999999999},"min":{"1min":0.255,"5min":0.253,"15min":0.24299999999999999},"max":{"1min":4.7729999999999997,"5min":9.5530000000000008,"15min":11.596},"last":0.45000000000000001}]},{"osd":1,"last update":"Fri Feb 9 17:49:53 2024","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.331,"15min":1.8260000000000001},"min":{"1min":0.26900000000000002,"5min":0.26500000000000001,"15min":0.23499999999999999},"max":{"1min":4.298,"5min":7.3490000000000002,"15min":89.515000000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":2.0270000000000001,"5min":1.45,"15min":1.772},"min":{"1min":0.30399999999999999,"5min":0.25,"15min":0.249},"max":{"1min":6.3410000000000002,"5min":11.685,"15min":89.388000000000005},"last":0.48399999999999999}]},{"osd":2,"last update":"Fri Feb 9 17:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.7250000000000001,"5min":1.8149999999999999,"15min":1.742},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":6.718,"5min":6.718,"15min":6.718},"last":0.436},{"interface":"front","average":{"1min":1.5600000000000001,"5min":1.728,"15min":1.643},"min":{"1min":0.376,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":4.5670000000000002,"5min":8.4250000000000007,"15min":8.4250000000000007},"last":0.52700000000000002}]},{"osd":3,"last update":"Fri Feb 9 17:45:45 2024","interfaces":[{"interface":"back","average":{"1min":1.569,"5min":1.516,"15min":1.671},"min":{"1min":0.36899999999999999,"5min":0.26500000000000001,"15min":0.25700000000000001},"max":{"1min":6.3789999999999996,"5min":8.9939999999999998,"15min":40.18},"last":0.57599999999999996},{"interface":"front","average":{"1min":1.264,"5min":1.079,"15min":1.556},"min":{"1min":0.33300000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":6.048,"5min":7.2519999999999998,"15min":40.158000000000001},"last":0.51600000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:49:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.0820000000000001,"15min":1.425},"min":{"1min":0.311,"5min":0.27300000000000002,"15min":0.26800000000000002},"max":{"1min":5.867,"5min":7.335,"15min":27.381},"last":0.55300000000000005},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.0840000000000001,"15min":1.228},"min":{"1min":0.34300000000000003,"5min":0.26900000000000002,"15min":0.26100000000000001},"max":{"1min":4.2510000000000003,"5min":5.415,"15min":27.355},"last":0.54000000000000004}]},{"osd":5,"last update":"Fri Feb 9 17:50:04 2024","interfaces":[{"interface":"back","average":{"1min":1.599,"5min":2.2240000000000002,"15min":1.6439999999999999},"min":{"1min":0.32100000000000001,"5min":0.30299999999999999,"15min":0.26900000000000002},"max":{"1min":8.9429999999999996,"5min":77.924000000000007,"15min":77.924000000000007},"last":0.61399999999999999},{"interface":"front","average":{"1min":1.2350000000000001,"5min":2.0830000000000002,"15min":1.647},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":8.9529999999999994,"5min":77.899000000000001,"15min":77.899000000000001},"last":0.56799999999999995}]},{"osd":6,"last update":"Fri Feb 9 17:49:53 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.2849999999999999,"15min":1.47},"min":{"1min":0.33800000000000002,"5min":0.314,"15min":0.30199999999999999},"max":{"1min":5.0640000000000001,"5min":8.8559999999999999,"15min":14.069000000000001},"last":0.47299999999999998},{"interface":"front","average":{"1min":1.331,"5min":1.4990000000000001,"15min":1.6519999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":8.0860000000000003,"5min":9.5250000000000004,"15min":14.097},"last":0.496}]},{"osd":7,"last update":"Fri Feb 9 17:49:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.9670000000000001,"15min":1.8400000000000001},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.33400000000000002},"max":{"1min":3.7909999999999999,"5min":29.001999999999999,"15min":29.001999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":1.127,"5min":1.919,"15min":1.841},"min":{"1min":0.39400000000000002,"5min":0.373,"15min":0.32200000000000001},"max":{"1min":3.7749999999999999,"5min":28.872,"15min":28.872},"last":0.47399999999999998}]},{"osd":8,"last update":"Fri Feb 9 17:50:10 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.70599999999999996,"15min":1.056},"min":{"1min":0.32800000000000001,"5min":0.32600000000000001,"15min":0.30099999999999999},"max":{"1min":2.4660000000000002,"5min":4.2160000000000002,"15min":14.045999999999999},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.79200000000000004,"15min":1.2230000000000001},"min":{"1min":0.35499999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":2.2109999999999999,"5min":9.4849999999999994,"15min":12.885999999999999},"last":0.35499999999999998}]},{"osd":10,"last update":"Fri Feb 9 17:50:24 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.88800000000000001,"15min":1.3029999999999999},"min":{"1min":0.38900000000000001,"5min":0.313,"15min":0.312},"max":{"1min":2.423,"5min":6.758,"15min":11.404},"last":0.372},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.878,"15min":1.1539999999999999},"min":{"1min":0.36399999999999999,"5min":0.35799999999999998,"15min":0.34300000000000003},"max":{"1min":5.5309999999999997,"5min":5.5309999999999997,"15min":10.904},"last":0.40999999999999998}]},{"osd":11,"last update":"Fri Feb 9 17:50:24 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.84399999999999997,"15min":1.29},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":2.6030000000000002,"5min":5.8010000000000002,"15min":14.02},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.74299999999999999,"15min":1.202},"min":{"1min":0.44700000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.016,"5min":3.9940000000000002,"15min":10.863},"last":0.39000000000000001}]}]},{"osd":8,"up_from":171,"seq":734439407887,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":762456,"kb_used_data":468556,"kb_used_omap":212,"kb_used_meta":293675,"kb_avail":92982696,"statfs":{"total":95995035648,"available":95214280704,"internally_reserved":0,"allocated":479801344,"data_stored":479467851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217579,"internal_metadata":300723733},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:49:57 2024","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.085,"15min":0.97299999999999998},"min":{"1min":0.22900000000000001,"5min":0.22600000000000001,"15min":0.217},"max":{"1min":5.7199999999999998,"5min":7.867,"15min":8.0920000000000005},"last":0.64700000000000002},{"interface":"front","average":{"1min":1.9570000000000001,"5min":1.3460000000000001,"15min":1.3180000000000001},"min":{"1min":0.29999999999999999,"5min":0.23999999999999999,"15min":0.23100000000000001},"max":{"1min":9.7110000000000003,"5min":12.045,"15min":12.045},"last":0.66200000000000003}]},{"osd":1,"last update":"Fri Feb 9 17:49:57 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.262,"15min":1.345},"min":{"1min":0.27300000000000002,"5min":0.25900000000000001,"15min":0.22700000000000001},"max":{"1min":4.2430000000000003,"5min":7.3280000000000003,"15min":8.718},"last":0.27500000000000002},{"interface":"front","average":{"1min":2.3769999999999998,"5min":1.4330000000000001,"15min":1.573},"min":{"1min":0.28499999999999998,"5min":0.24099999999999999,"15min":0.23300000000000001},"max":{"1min":7.2359999999999998,"5min":7.2359999999999998,"15min":25.027000000000001},"last":0.50600000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:50:34 2024","interfaces":[{"interface":"back","average":{"1min":2.222,"5min":1.722,"15min":1.391},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28000000000000003},"max":{"1min":12.227,"5min":12.227,"15min":12.227},"last":0.56899999999999995},{"interface":"front","average":{"1min":2.5049999999999999,"5min":1.665,"15min":1.1240000000000001},"min":{"1min":0.42199999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":12.747999999999999,"5min":12.747999999999999,"15min":12.747999999999999},"last":1.5680000000000001}]},{"osd":3,"last update":"Fri Feb 9 17:42:38 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":1.542,"15min":1.6379999999999999},"min":{"1min":0.316,"5min":0.29499999999999998,"15min":0.28699999999999998},"max":{"1min":2.347,"5min":15.228999999999999,"15min":24.030999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.75600000000000001,"5min":1.8340000000000001,"15min":1.8640000000000001},"min":{"1min":0.36199999999999999,"5min":0.33600000000000002,"15min":0.30299999999999999},"max":{"1min":2.3719999999999999,"5min":14.988,"15min":24.318999999999999},"last":0.69399999999999995}]},{"osd":4,"last update":"Fri Feb 9 17:49:57 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.97899999999999998,"15min":1.454},"min":{"1min":0.41099999999999998,"5min":0.30499999999999999,"15min":0.28199999999999997},"max":{"1min":4.3940000000000001,"5min":6.2800000000000002,"15min":53.060000000000002},"last":0.29499999999999998},{"interface":"front","average":{"1min":1.694,"5min":1.5700000000000001,"15min":1.629},"min":{"1min":0.34899999999999998,"5min":0.28299999999999997,"15min":0.27000000000000002},"max":{"1min":6.2149999999999999,"5min":8.2360000000000007,"15min":53.311999999999998},"last":0.35799999999999998}]},{"osd":5,"last update":"Fri Feb 9 17:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.171,"15min":1.9530000000000001},"min":{"1min":0.46100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":6.4329999999999998,"5min":6.4329999999999998,"15min":6.4329999999999998},"last":0.311},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.25,"15min":1.79},"min":{"1min":0.35199999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":4.0190000000000001,"5min":5.3940000000000001,"15min":6.383},"last":0.40400000000000003}]},{"osd":6,"last update":"Fri Feb 9 17:49:57 2024","interfaces":[{"interface":"back","average":{"1min":1.8600000000000001,"5min":1.7529999999999999,"15min":1.419},"min":{"1min":0.34599999999999997,"5min":0.29899999999999999,"15min":0.29799999999999999},"max":{"1min":9.5899999999999999,"5min":13.031000000000001,"15min":13.031000000000001},"last":0.34499999999999997},{"interface":"front","average":{"1min":1.472,"5min":1.4750000000000001,"15min":1.4419999999999999},"min":{"1min":0.35699999999999998,"5min":0.34200000000000003,"15min":0.29599999999999999},"max":{"1min":9.4239999999999995,"5min":12.157999999999999,"15min":12.157999999999999},"last":0.42099999999999999}]},{"osd":7,"last update":"Fri Feb 9 17:49:57 2024","interfaces":[{"interface":"back","average":{"1min":2.0270000000000001,"5min":1.617,"15min":1.579},"min":{"1min":0.48499999999999999,"5min":0.34100000000000003,"15min":0.32800000000000001},"max":{"1min":6.5640000000000001,"5min":6.766,"15min":11.278},"last":0.433},{"interface":"front","average":{"1min":2.1419999999999999,"5min":1.7929999999999999,"15min":1.6619999999999999},"min":{"1min":0.50900000000000001,"5min":0.38700000000000001,"15min":0.38500000000000001},"max":{"1min":8.0449999999999999,"5min":8.0449999999999999,"15min":11.313000000000001},"last":0.47099999999999997}]},{"osd":9,"last update":"Fri Feb 9 17:49:57 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.97499999999999998,"15min":1.1859999999999999},"min":{"1min":0.35699999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":4.8979999999999997,"5min":5.6529999999999996,"15min":45.616},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.1419999999999999,"15min":1.2529999999999999},"min":{"1min":0.44900000000000001,"5min":0.38100000000000001,"15min":0.314},"max":{"1min":5.0049999999999999,"5min":7.5609999999999999,"15min":45.718000000000004},"last":0.44900000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:49:57 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.81799999999999995,"15min":0.93899999999999995},"min":{"1min":0.33300000000000002,"5min":0.32300000000000001,"15min":0.29699999999999999},"max":{"1min":1.796,"5min":4.3380000000000001,"15min":7.8360000000000003},"last":2.0019999999999998},{"interface":"front","average":{"1min":1.202,"5min":0.90400000000000003,"15min":1.131},"min":{"1min":0.46300000000000002,"5min":0.34300000000000003,"15min":0.318},"max":{"1min":4.9569999999999999,"5min":4.9569999999999999,"15min":11.071},"last":1.966}]},{"osd":11,"last update":"Fri Feb 9 17:49:57 2024","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":1.0620000000000001,"15min":1.141},"min":{"1min":0.42499999999999999,"5min":0.40899999999999997,"15min":0.34000000000000002},"max":{"1min":4.9809999999999999,"5min":7.6079999999999997,"15min":8.5519999999999996},"last":0.49399999999999999},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.0089999999999999,"15min":1.181},"min":{"1min":0.43099999999999999,"5min":0.32900000000000001,"15min":0.308},"max":{"1min":4.29,"5min":6.5720000000000001,"15min":7.4359999999999999},"last":0.53800000000000003}]}]},{"osd":7,"up_from":57,"seq":244813136232,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1009584,"kb_used_data":320352,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":92735568,"statfs":{"total":95995035648,"available":94961221632,"internally_reserved":0,"allocated":328040448,"data_stored":327698855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":705755594},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:50:37 2024","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.5449999999999999,"15min":1.4830000000000001},"min":{"1min":0.315,"5min":0.249,"15min":0.249},"max":{"1min":11.478999999999999,"5min":11.478999999999999,"15min":14.348000000000001},"last":1.1850000000000001},{"interface":"front","average":{"1min":0.58799999999999997,"5min":1.335,"15min":1.492},"min":{"1min":0.33400000000000002,"5min":0.26800000000000002,"15min":0.24099999999999999},"max":{"1min":1.1930000000000001,"5min":9.5960000000000001,"15min":11.843999999999999},"last":1.1930000000000001}]},{"osd":1,"last update":"Fri Feb 9 17:50:37 2024","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.456,"15min":1.831},"min":{"1min":0.34999999999999998,"5min":0.29199999999999998,"15min":0.24199999999999999},"max":{"1min":4.8949999999999996,"5min":7.827,"15min":21.704999999999998},"last":0.36199999999999999},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.55,"15min":2.004},"min":{"1min":0.27000000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":11.673,"5min":11.673,"15min":21.858000000000001},"last":0.48699999999999999}]},{"osd":2,"last update":"Fri Feb 9 17:50:37 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.5149999999999999,"15min":1.849},"min":{"1min":0.42399999999999999,"5min":0.25,"15min":0.25},"max":{"1min":5.0540000000000003,"5min":6.4850000000000003,"15min":6.6340000000000003},"last":1.4910000000000001},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.6499999999999999,"15min":2.1589999999999998},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":4.5529999999999999,"5min":10.159000000000001,"15min":11.968},"last":1.9019999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:49:56 2024","interfaces":[{"interface":"back","average":{"1min":5.0510000000000002,"5min":3.464,"15min":2.3079999999999998},"min":{"1min":0.46899999999999997,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":28.408000000000001,"5min":78.644999999999996,"15min":78.644999999999996},"last":0.61399999999999999},{"interface":"front","average":{"1min":4.5910000000000002,"5min":3.4910000000000001,"15min":2.4550000000000001},"min":{"1min":0.42199999999999999,"5min":0.36799999999999999,"15min":0.308},"max":{"1min":28.253,"5min":78.414000000000001,"15min":78.414000000000001},"last":0.56499999999999995}]},{"osd":4,"last update":"Fri Feb 9 17:50:37 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":1.867,"15min":1.399},"min":{"1min":0.25900000000000001,"5min":0.25600000000000001,"15min":0.23699999999999999},"max":{"1min":2.9969999999999999,"5min":86.813000000000002,"15min":86.813000000000002},"last":0.38400000000000001},{"interface":"front","average":{"1min":1.304,"5min":2.2930000000000001,"15min":1.5800000000000001},"min":{"1min":0.29299999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":11.433,"5min":86.751000000000005,"15min":86.751000000000005},"last":0.40300000000000002}]},{"osd":5,"last update":"Fri Feb 9 17:50:37 2024","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":1.3879999999999999,"15min":1.381},"min":{"1min":0.34799999999999998,"5min":0.30199999999999999,"15min":0.249},"max":{"1min":3.5779999999999998,"5min":8.0060000000000002,"15min":9.6590000000000007},"last":0.443},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.411,"15min":1.462},"min":{"1min":0.35099999999999998,"5min":0.32100000000000001,"15min":0.31900000000000001},"max":{"1min":4.9800000000000004,"5min":7.867,"15min":15.228999999999999},"last":0.52000000000000002}]},{"osd":6,"last update":"Fri Feb 9 17:49:56 2024","interfaces":[{"interface":"back","average":{"1min":1.597,"5min":1.601,"15min":1.7609999999999999},"min":{"1min":0.45500000000000002,"5min":0.35699999999999998,"15min":0.26000000000000001},"max":{"1min":7.1479999999999997,"5min":7.8879999999999999,"15min":15.275},"last":0.47199999999999998},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.518,"15min":1.601},"min":{"1min":0.38200000000000001,"5min":0.32000000000000001,"15min":0.29199999999999998},"max":{"1min":7.1669999999999998,"5min":7.1669999999999998,"15min":18.236999999999998},"last":0.498}]},{"osd":8,"last update":"Fri Feb 9 17:49:56 2024","interfaces":[{"interface":"back","average":{"1min":1.96,"5min":1.446,"15min":1.423},"min":{"1min":0.45000000000000001,"5min":0.375,"15min":0.28999999999999998},"max":{"1min":11.513,"5min":11.513,"15min":11.81},"last":0.45800000000000002},{"interface":"front","average":{"1min":1.526,"5min":1.359,"15min":1.429},"min":{"1min":0.38,"5min":0.38,"15min":0.33300000000000002},"max":{"1min":5.6399999999999997,"5min":7.8949999999999996,"15min":12.058},"last":0.59999999999999998}]},{"osd":9,"last update":"Fri Feb 9 17:49:43 2024","interfaces":[{"interface":"back","average":{"1min":2.0249999999999999,"5min":1.5029999999999999,"15min":1.5289999999999999},"min":{"1min":0.42999999999999999,"5min":0.375,"15min":0.32700000000000001},"max":{"1min":11.542999999999999,"5min":11.542999999999999,"15min":11.941000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":2.2450000000000001,"5min":1.631,"15min":1.659},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.33400000000000002},"max":{"1min":11.635999999999999,"5min":11.635999999999999,"15min":11.906000000000001},"last":0.50800000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:50:13 2024","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":1.113,"15min":1.387},"min":{"1min":0.43099999999999999,"5min":0.38700000000000001,"15min":0.35099999999999998},"max":{"1min":3.5089999999999999,"5min":5.5099999999999998,"15min":12.050000000000001},"last":0.55400000000000005},{"interface":"front","average":{"1min":2.1850000000000001,"5min":1.5700000000000001,"15min":1.613},"min":{"1min":0.51900000000000002,"5min":0.41999999999999998,"15min":0.38900000000000001},"max":{"1min":11.571999999999999,"5min":11.571999999999999,"15min":11.878},"last":0.53700000000000003}]},{"osd":11,"last update":"Fri Feb 9 17:50:37 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.052,"15min":1.5580000000000001},"min":{"1min":0.46300000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":4.4480000000000004,"5min":5.5620000000000003,"15min":21.945},"last":0.57699999999999996},{"interface":"front","average":{"1min":1.337,"5min":1.2130000000000001,"15min":1.647},"min":{"1min":0.44,"5min":0.438,"15min":0.40699999999999997},"max":{"1min":4.952,"5min":7.8019999999999996,"15min":21.731999999999999},"last":0.68000000000000005}]}]},{"osd":6,"up_from":49,"seq":210453397869,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1005704,"kb_used_data":437248,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":92739448,"statfs":{"total":95995035648,"available":94965194752,"internally_reserved":0,"allocated":447741952,"data_stored":447413007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":582089158},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:50:28 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":1.446,"15min":1.5940000000000001},"min":{"1min":0.26600000000000001,"5min":0.24099999999999999,"15min":0.22800000000000001},"max":{"1min":2.0670000000000002,"5min":10.481999999999999,"15min":12.544},"last":0.60599999999999998},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.595,"15min":1.6299999999999999},"min":{"1min":0.29899999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":4.9640000000000004,"5min":7.0860000000000003,"15min":9.2520000000000007},"last":0.54600000000000004}]},{"osd":1,"last update":"Fri Feb 9 17:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":3.0870000000000002,"15min":2.4409999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.252},"max":{"1min":4.1200000000000001,"5min":79.945999999999998,"15min":79.945999999999998},"last":0.39500000000000002},{"interface":"front","average":{"1min":1.2490000000000001,"5min":3.1339999999999999,"15min":2.5169999999999999},"min":{"1min":0.41499999999999998,"5min":0.32600000000000001,"15min":0.29899999999999999},"max":{"1min":4.1429999999999998,"5min":80.003,"15min":80.003},"last":0.58099999999999996}]},{"osd":2,"last update":"Fri Feb 9 17:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":2.952,"15min":2.0760000000000001},"min":{"1min":0.34799999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":7.4139999999999997,"5min":84.912999999999997,"15min":84.912999999999997},"last":0.65100000000000002},{"interface":"front","average":{"1min":1.2569999999999999,"5min":3.0059999999999998,"15min":2.1549999999999998},"min":{"1min":0.42899999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":7.3860000000000001,"5min":84.942999999999998,"15min":84.942999999999998},"last":0.68700000000000006}]},{"osd":3,"last update":"Fri Feb 9 17:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":4.0720000000000001,"15min":2.5419999999999998},"min":{"1min":0.30199999999999999,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":10.147,"5min":80.174999999999997,"15min":80.174999999999997},"last":0.52800000000000002},{"interface":"front","average":{"1min":1.2889999999999999,"5min":4.4880000000000004,"15min":2.6920000000000002},"min":{"1min":0.39400000000000002,"5min":0.32700000000000001,"15min":0.28000000000000003},"max":{"1min":2.9289999999999998,"5min":80.417000000000002,"15min":80.417000000000002},"last":0.56299999999999994}]},{"osd":4,"last update":"Fri Feb 9 17:50:28 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":1.8440000000000001,"15min":1.7070000000000001},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.23599999999999999},"max":{"1min":2.863,"5min":14.182,"15min":15.061999999999999},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.877,"5min":1.909,"15min":1.851},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.26400000000000001},"max":{"1min":2.9430000000000001,"5min":14.210000000000001,"15min":15.118},"last":0.59599999999999997}]},{"osd":5,"last update":"Fri Feb 9 17:50:28 2024","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":1.883,"15min":1.819},"min":{"1min":0.33000000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":4.7389999999999999,"5min":14.151999999999999,"15min":14.151999999999999},"last":0.625},{"interface":"front","average":{"1min":0.88,"5min":1.7470000000000001,"15min":1.7230000000000001},"min":{"1min":0.34599999999999997,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":4.7249999999999996,"5min":15.509,"15min":18.315999999999999},"last":0.73899999999999999}]},{"osd":7,"last update":"Fri Feb 9 17:50:01 2024","interfaces":[{"interface":"back","average":{"1min":2.6589999999999998,"5min":2.1360000000000001,"15min":1.8260000000000001},"min":{"1min":0.32200000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":10.98,"5min":23.265999999999998,"15min":23.265999999999998},"last":0.42399999999999999},{"interface":"front","average":{"1min":1.5529999999999999,"5min":1.883,"15min":1.7090000000000001},"min":{"1min":0.42099999999999999,"5min":0.314,"15min":0.314},"max":{"1min":5.0759999999999996,"5min":10.599,"15min":12.967000000000001},"last":0.71399999999999997}]},{"osd":8,"last update":"Fri Feb 9 17:26:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":0.876,"15min":0.79600000000000004},"min":{"1min":0.39900000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":4.0339999999999998,"5min":4.4880000000000004,"15min":4.4880000000000004},"last":0.66400000000000003},{"interface":"front","average":{"1min":1.0469999999999999,"5min":0.88600000000000001,"15min":0.78300000000000003},"min":{"1min":0.375,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":3.6539999999999999,"5min":4.8570000000000002,"15min":4.8570000000000002},"last":0.75600000000000001}]},{"osd":9,"last update":"Fri Feb 9 17:50:32 2024","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":1.248,"15min":1.4239999999999999},"min":{"1min":0.38700000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":2.1659999999999999,"5min":6.0359999999999996,"15min":10.779999999999999},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.88100000000000001,"5min":1.2450000000000001,"15min":1.4970000000000001},"min":{"1min":0.42999999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":2.1440000000000001,"5min":6.157,"15min":8.4130000000000003},"last":0.80600000000000005}]},{"osd":10,"last update":"Fri Feb 9 17:50:01 2024","interfaces":[{"interface":"back","average":{"1min":1.7769999999999999,"5min":1.5629999999999999,"15min":1.5860000000000001},"min":{"1min":0.41699999999999998,"5min":0.34399999999999997,"15min":0.34300000000000003},"max":{"1min":10.097,"5min":10.097,"15min":25.370000000000001},"last":0.72499999999999998},{"interface":"front","average":{"1min":2.113,"5min":2.0259999999999998,"15min":1.7},"min":{"1min":0.41499999999999998,"5min":0.36399999999999999,"15min":0.35099999999999998},"max":{"1min":14.119999999999999,"5min":14.119999999999999,"15min":26.027999999999999},"last":0.79400000000000004}]},{"osd":11,"last update":"Fri Feb 9 17:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.5860000000000001,"5min":1.5509999999999999,"15min":1.635},"min":{"1min":0.41999999999999998,"5min":0.372,"15min":0.372},"max":{"1min":10.176,"5min":10.176,"15min":13.109},"last":0.67600000000000005},{"interface":"front","average":{"1min":1.474,"5min":1.5700000000000001,"15min":1.569},"min":{"1min":0.45700000000000002,"5min":0.39700000000000002,"15min":0.373},"max":{"1min":10.202,"5min":10.202,"15min":13.164999999999999},"last":0.70099999999999996}]}]},{"osd":1,"up_from":13,"seq":55834575239,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1104200,"kb_used_data":419136,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":92640952,"statfs":{"total":95995035648,"available":94864334848,"internally_reserved":0,"allocated":429195264,"data_stored":428866319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:49:43 2024","interfaces":[{"interface":"back","average":{"1min":2.077,"5min":1.7669999999999999,"15min":1.8140000000000001},"min":{"1min":0.23499999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.9420000000000002,"5min":9.8529999999999998,"15min":19.079000000000001},"last":1.1659999999999999},{"interface":"front","average":{"1min":2.085,"5min":1.978,"15min":1.782},"min":{"1min":0.214,"5min":0.214,"15min":0.19400000000000001},"max":{"1min":11.956,"5min":11.956,"15min":23.573},"last":1.151}]},{"osd":2,"last update":"Fri Feb 9 17:50:09 2024","interfaces":[{"interface":"back","average":{"1min":3.923,"5min":2.4060000000000001,"15min":1.9830000000000001},"min":{"1min":0.30299999999999999,"5min":0.24399999999999999,"15min":0.23200000000000001},"max":{"1min":14.83,"5min":14.83,"15min":26.361999999999998},"last":1.2849999999999999},{"interface":"front","average":{"1min":3.0680000000000001,"5min":1.99,"15min":1.976},"min":{"1min":0.28499999999999998,"5min":0.222,"15min":0.222},"max":{"1min":9.3870000000000005,"5min":9.3870000000000005,"15min":26.087},"last":1.319}]},{"osd":3,"last update":"Fri Feb 9 17:50:36 2024","interfaces":[{"interface":"back","average":{"1min":2.786,"5min":2.831,"15min":1.9570000000000001},"min":{"1min":0.29999999999999999,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":17.878,"5min":17.959,"15min":17.959},"last":0.61499999999999999},{"interface":"front","average":{"1min":2.1760000000000002,"5min":3.1869999999999998,"15min":1.7869999999999999},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.26300000000000001},"max":{"1min":11.526,"5min":22.866,"15min":22.866},"last":0.89800000000000002}]},{"osd":4,"last update":"Fri Feb 9 17:49:41 2024","interfaces":[{"interface":"back","average":{"1min":1.804,"5min":2.1739999999999999,"15min":1.855},"min":{"1min":0.312,"5min":0.30299999999999999,"15min":0.253},"max":{"1min":7.8490000000000002,"5min":45.137999999999998,"15min":45.137999999999998},"last":0.64900000000000002},{"interface":"front","average":{"1min":1.641,"5min":2.0640000000000001,"15min":2.0150000000000001},"min":{"1min":0.39900000000000002,"5min":0.29399999999999998,"15min":0.28999999999999998},"max":{"1min":5.7430000000000003,"5min":44.950000000000003,"15min":44.950000000000003},"last":0.32200000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.8089999999999999,"5min":2.1269999999999998,"15min":1.9399999999999999},"min":{"1min":0.37,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":9.2409999999999997,"5min":45.171999999999997,"15min":45.171999999999997},"last":0.40200000000000002},{"interface":"front","average":{"1min":3.859,"5min":2.8239999999999998,"15min":2.1930000000000001},"min":{"1min":0.438,"5min":0.31900000000000001,"15min":0.30499999999999999},"max":{"1min":33.247,"5min":45.127000000000002,"15min":45.127000000000002},"last":1.1850000000000001}]},{"osd":6,"last update":"Fri Feb 9 17:50:36 2024","interfaces":[{"interface":"back","average":{"1min":3.8620000000000001,"5min":2.399,"15min":1.8149999999999999},"min":{"1min":0.46300000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":33.338000000000001,"5min":33.338000000000001,"15min":33.338000000000001},"last":1.238},{"interface":"front","average":{"1min":3.8109999999999999,"5min":2.492,"15min":1.827},"min":{"1min":0.51600000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":33.286999999999999,"5min":33.286999999999999,"15min":33.286999999999999},"last":1.45}]},{"osd":7,"last update":"Fri Feb 9 17:50:36 2024","interfaces":[{"interface":"back","average":{"1min":3.3959999999999999,"5min":2.863,"15min":1.8700000000000001},"min":{"1min":0.53500000000000003,"5min":0.44800000000000001,"15min":0.34399999999999997},"max":{"1min":33.314999999999998,"5min":45.740000000000002,"15min":45.740000000000002},"last":1.333},{"interface":"front","average":{"1min":3.4249999999999998,"5min":2.895,"15min":1.9570000000000001},"min":{"1min":0.56999999999999995,"5min":0.40200000000000002,"15min":0.309},"max":{"1min":33.363999999999997,"5min":45.792000000000002,"15min":45.792000000000002},"last":1.298}]},{"osd":8,"last update":"Fri Feb 9 17:49:41 2024","interfaces":[{"interface":"back","average":{"1min":1.5880000000000001,"5min":1.6259999999999999,"15min":1.161},"min":{"1min":0.51900000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":5.7839999999999998,"5min":8.9440000000000008,"15min":9.2520000000000007},"last":0.69499999999999995},{"interface":"front","average":{"1min":1.919,"5min":1.6930000000000001,"15min":1.1739999999999999},"min":{"1min":0.39000000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":12.69,"5min":12.69,"15min":12.69},"last":0.72899999999999998}]},{"osd":9,"last update":"Fri Feb 9 17:49:43 2024","interfaces":[{"interface":"back","average":{"1min":1.7370000000000001,"5min":1.778,"15min":1.595},"min":{"1min":0.45100000000000001,"5min":0.39300000000000002,"15min":0.33400000000000002},"max":{"1min":5.8140000000000001,"5min":9.9359999999999999,"15min":9.9359999999999999},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.522,"5min":1.762,"15min":1.772},"min":{"1min":0.41399999999999998,"5min":0.34899999999999998,"15min":0.32900000000000001},"max":{"1min":5.2140000000000004,"5min":7.7199999999999998,"15min":11.366},"last":0.438}]},{"osd":10,"last update":"Fri Feb 9 17:50:24 2024","interfaces":[{"interface":"back","average":{"1min":3.706,"5min":2.242,"15min":1.9570000000000001},"min":{"1min":0.501,"5min":0.432,"15min":0.39200000000000002},"max":{"1min":33.213999999999999,"5min":33.213999999999999,"15min":33.213999999999999},"last":1.214},{"interface":"front","average":{"1min":1.8680000000000001,"5min":1.7370000000000001,"15min":1.829},"min":{"1min":0.51300000000000001,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":9.2919999999999998,"5min":10.09,"15min":15.121},"last":0.56100000000000005}]},{"osd":11,"last update":"Fri Feb 9 17:50:36 2024","interfaces":[{"interface":"back","average":{"1min":1.891,"5min":1.8799999999999999,"15min":1.411},"min":{"1min":0.47399999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":9.3140000000000001,"5min":16.184999999999999,"15min":16.184999999999999},"last":0.59099999999999997},{"interface":"front","average":{"1min":3.5110000000000001,"5min":2.2839999999999998,"15min":1.6379999999999999},"min":{"1min":0.55100000000000005,"5min":0.48299999999999998,"15min":0.44800000000000001},"max":{"1min":33.161000000000001,"5min":33.161000000000001,"15min":33.161000000000001},"last":1.264}]}]},{"osd":0,"up_from":8,"seq":34359738763,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1479756,"kb_used_data":766912,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":92265396,"statfs":{"total":95995035648,"available":94479765504,"internally_reserved":0,"allocated":785317888,"data_stored":784988943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 9 17:50:30 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":1.653,"15min":1.5640000000000001},"min":{"1min":0.26300000000000001,"5min":0.22900000000000001,"15min":0.20599999999999999},"max":{"1min":3.4089999999999998,"5min":10.938000000000001,"15min":13.842000000000001},"last":1.1200000000000001},{"interface":"front","average":{"1min":0.95099999999999996,"5min":1.718,"15min":1.7430000000000001},"min":{"1min":0.372,"5min":0.29299999999999998,"15min":0.24399999999999999},"max":{"1min":3.4449999999999998,"5min":10.974,"15min":14.122},"last":1.143}]},{"osd":2,"last update":"Fri Feb 9 17:49:44 2024","interfaces":[{"interface":"back","average":{"1min":2.052,"5min":2.052,"15min":2.052},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":8.859,"5min":8.859,"15min":8.859},"last":0.27700000000000002},{"interface":"front","average":{"1min":1.52,"5min":1.52,"15min":1.52},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":4.3940000000000001,"5min":4.3940000000000001,"15min":4.3940000000000001},"last":0.95499999999999996}]},{"osd":3,"last update":"Fri Feb 9 17:50:08 2024","interfaces":[{"interface":"back","average":{"1min":2.5259999999999998,"5min":2.0859999999999999,"15min":2.1309999999999998},"min":{"1min":0.245,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":16.757999999999999,"5min":36.052,"15min":36.052},"last":1.365},{"interface":"front","average":{"1min":3.0699999999999998,"5min":2.5,"15min":1.885},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.217},"max":{"1min":12.064,"5min":38.383000000000003,"15min":38.383000000000003},"last":1.3500000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:49:58 2024","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.494,"15min":1.3600000000000001},"min":{"1min":0.35599999999999998,"5min":0.28000000000000003,"15min":0.27000000000000002},"max":{"1min":7.5499999999999998,"5min":8.9420000000000002,"15min":8.9420000000000002},"last":0.29599999999999999},{"interface":"front","average":{"1min":3.2959999999999998,"5min":2.2949999999999999,"15min":1.8560000000000001},"min":{"1min":0.33500000000000002,"5min":0.32700000000000001,"15min":0.27200000000000002},"max":{"1min":11.882999999999999,"5min":16.916,"15min":16.916},"last":0.34699999999999998}]},{"osd":5,"last update":"Fri Feb 9 17:50:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4039999999999999,"5min":2.4729999999999999,"15min":1.873},"min":{"1min":0.32400000000000001,"5min":0.311,"15min":0.29499999999999998},"max":{"1min":12.949999999999999,"5min":48.924999999999997,"15min":48.924999999999997},"last":0.46600000000000003},{"interface":"front","average":{"1min":2.456,"5min":2.5179999999999998,"15min":1.8859999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.29599999999999999},"max":{"1min":12.981999999999999,"5min":48.969999999999999,"15min":48.969999999999999},"last":0.56299999999999994}]},{"osd":6,"last update":"Fri Feb 9 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.8400000000000001,"5min":1.839,"15min":1.7549999999999999},"min":{"1min":0.35099999999999998,"5min":0.307,"15min":0.28699999999999998},"max":{"1min":7.9429999999999996,"5min":11.912000000000001,"15min":13.407},"last":0.33100000000000002},{"interface":"front","average":{"1min":2.3610000000000002,"5min":2.0190000000000001,"15min":1.8540000000000001},"min":{"1min":0.38,"5min":0.375,"15min":0.33300000000000002},"max":{"1min":7.9740000000000002,"5min":11.939,"15min":13.448},"last":0.40000000000000002}]},{"osd":7,"last update":"Fri Feb 9 17:50:01 2024","interfaces":[{"interface":"back","average":{"1min":2.2050000000000001,"5min":1.7749999999999999,"15min":1.9810000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":7.9160000000000004,"5min":15.271000000000001,"15min":15.271000000000001},"last":0.36799999999999999},{"interface":"front","average":{"1min":3.4319999999999999,"5min":2.294,"15min":2.105},"min":{"1min":0.42899999999999999,"5min":0.34699999999999998,"15min":0.28699999999999998},"max":{"1min":9.3330000000000002,"5min":11.614000000000001,"15min":14.451000000000001},"last":0.41899999999999998}]},{"osd":8,"last update":"Fri Feb 9 17:50:01 2024","interfaces":[{"interface":"back","average":{"1min":3.1499999999999999,"5min":1.7130000000000001,"15min":1.714},"min":{"1min":0.54200000000000004,"5min":0.36199999999999999,"15min":0.33100000000000002},"max":{"1min":8.3599999999999994,"5min":11.641999999999999,"15min":14.323},"last":0.38800000000000001},{"interface":"front","average":{"1min":2.9769999999999999,"5min":1.6719999999999999,"15min":1.778},"min":{"1min":0.495,"5min":0.39200000000000002,"15min":0.35099999999999998},"max":{"1min":8.4320000000000004,"5min":11.669,"15min":14.481999999999999},"last":0.76300000000000001}]},{"osd":9,"last update":"Fri Feb 9 17:49:49 2024","interfaces":[{"interface":"back","average":{"1min":2.6789999999999998,"5min":1.893,"15min":1.962},"min":{"1min":0.50700000000000001,"5min":0.40500000000000003,"15min":0.32500000000000001},"max":{"1min":8.9160000000000004,"5min":16.936,"15min":17.952000000000002},"last":0.93500000000000005},{"interface":"front","average":{"1min":2.7240000000000002,"5min":1.8520000000000001,"15min":1.986},"min":{"1min":0.55200000000000005,"5min":0.42399999999999999,"15min":0.34799999999999998},"max":{"1min":8.9489999999999998,"5min":16.951000000000001,"15min":17.913},"last":0.97499999999999998}]},{"osd":10,"last update":"Fri Feb 9 17:50:08 2024","interfaces":[{"interface":"back","average":{"1min":2.1869999999999998,"5min":1.7350000000000001,"15min":1.968},"min":{"1min":0.46400000000000002,"5min":0.443,"15min":0.35499999999999998},"max":{"1min":8.4090000000000007,"5min":11.692,"15min":31.766999999999999},"last":0.91700000000000004},{"interface":"front","average":{"1min":1.6839999999999999,"5min":1.484,"15min":1.4810000000000001},"min":{"1min":0.46400000000000002,"5min":0.41899999999999998,"15min":0.34200000000000003},"max":{"1min":6.2480000000000002,"5min":8.2210000000000001,"15min":28.048999999999999},"last":0.54000000000000004}]},{"osd":11,"last update":"Fri Feb 9 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.601,"15min":1.659},"min":{"1min":0.499,"5min":0.41599999999999998,"15min":0.40600000000000003},"max":{"1min":6.274,"5min":10.282,"15min":40.021000000000001},"last":0.51600000000000001},{"interface":"front","average":{"1min":1.671,"5min":1.849,"15min":1.7849999999999999},"min":{"1min":0.53300000000000003,"5min":0.45500000000000002,"15min":0.45200000000000001},"max":{"1min":8.1820000000000004,"5min":9.5820000000000007,"15min":40.055999999999997},"last":0.80900000000000005}]}]},{"osd":2,"up_from":18,"seq":77309411713,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1076176,"kb_used_data":688200,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":92668976,"statfs":{"total":95995035648,"available":94893031424,"internally_reserved":0,"allocated":704716800,"data_stored":704384303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":397277637},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":1.4510000000000001,"15min":1.006},"min":{"1min":0.29999999999999999,"5min":0.23699999999999999,"15min":0.221},"max":{"1min":4.9089999999999998,"5min":17.251000000000001,"15min":17.251000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.74399999999999999,"5min":1.611,"15min":1.113},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.22900000000000001},"max":{"1min":4.7720000000000002,"5min":23.097999999999999,"15min":23.097999999999999},"last":0.875}]},{"osd":1,"last update":"Fri Feb 9 17:50:04 2024","interfaces":[{"interface":"back","average":{"1min":5.4139999999999997,"5min":3.1560000000000001,"15min":2.2469999999999999},"min":{"1min":0.23000000000000001,"5min":0.214,"15min":0.214},"max":{"1min":46.698999999999998,"5min":52.646000000000001,"15min":52.646000000000001},"last":0.88800000000000001},{"interface":"front","average":{"1min":5.6669999999999998,"5min":3.0249999999999999,"15min":2.3159999999999998},"min":{"1min":0.25800000000000001,"5min":0.248,"15min":0.214},"max":{"1min":46.728999999999999,"5min":52.683999999999997,"15min":52.683999999999997},"last":1.0149999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:50:30 2024","interfaces":[{"interface":"back","average":{"1min":1.976,"5min":2.3639999999999999,"15min":1.8879999999999999},"min":{"1min":0.33600000000000002,"5min":0.27700000000000002,"15min":0.249},"max":{"1min":8.0009999999999994,"5min":22.698,"15min":22.698},"last":1.327},{"interface":"front","average":{"1min":2.145,"5min":2.7469999999999999,"15min":2.1989999999999998},"min":{"1min":0.309,"5min":0.221,"15min":0.20699999999999999},"max":{"1min":8.3539999999999992,"5min":19.904,"15min":19.986999999999998},"last":2.5009999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:50:30 2024","interfaces":[{"interface":"back","average":{"1min":1.9219999999999999,"5min":1.861,"15min":1.8460000000000001},"min":{"1min":0.34799999999999998,"5min":0.315,"15min":0.26300000000000001},"max":{"1min":10.419,"5min":12.398,"15min":13.526},"last":0.95099999999999996},{"interface":"front","average":{"1min":0.82699999999999996,"5min":1.704,"15min":1.621},"min":{"1min":0.36099999999999999,"5min":0.28000000000000003,"15min":0.27300000000000002},"max":{"1min":3.8119999999999998,"5min":10.888999999999999,"15min":10.888999999999999},"last":0.35999999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:50:20 2024","interfaces":[{"interface":"back","average":{"1min":2.5510000000000002,"5min":2.681,"15min":2.1560000000000001},"min":{"1min":0.31900000000000001,"5min":0.28499999999999998,"15min":0.28299999999999997},"max":{"1min":15.455,"5min":35.436999999999998,"15min":42.664000000000001},"last":2.4359999999999999},{"interface":"front","average":{"1min":1.7729999999999999,"5min":2.7909999999999999,"15min":2.387},"min":{"1min":0.29699999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":15.487,"5min":35.484000000000002,"15min":43.311999999999998},"last":0.44700000000000001}]},{"osd":6,"last update":"Fri Feb 9 17:50:30 2024","interfaces":[{"interface":"back","average":{"1min":1.8240000000000001,"5min":2.1789999999999998,"15min":1.9750000000000001},"min":{"1min":0.32400000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":18.518999999999998,"5min":18.518999999999998,"15min":18.518999999999998},"last":0.41399999999999998},{"interface":"front","average":{"1min":2.036,"5min":2.0899999999999999,"15min":2.044},"min":{"1min":0.33700000000000002,"5min":0.33100000000000002,"15min":0.32800000000000001},"max":{"1min":18.481999999999999,"5min":18.481999999999999,"15min":18.481999999999999},"last":0.48199999999999998}]},{"osd":7,"last update":"Fri Feb 9 17:50:04 2024","interfaces":[{"interface":"back","average":{"1min":2.9300000000000002,"5min":2.3599999999999999,"15min":2.367},"min":{"1min":0.39000000000000001,"5min":0.36799999999999999,"15min":0.32400000000000001},"max":{"1min":10.348000000000001,"5min":16.666,"15min":41.802999999999997},"last":0.92200000000000004},{"interface":"front","average":{"1min":1.3420000000000001,"5min":2.6989999999999998,"15min":2.2559999999999998},"min":{"1min":0.41899999999999998,"5min":0.36599999999999999,"15min":0.30499999999999999},"max":{"1min":3.8769999999999998,"5min":16.638999999999999,"15min":41.783999999999999},"last":1.028}]},{"osd":8,"last update":"Fri Feb 9 17:50:30 2024","interfaces":[{"interface":"back","average":{"1min":1.667,"5min":1.6930000000000001,"15min":1.4590000000000001},"min":{"1min":0.39900000000000002,"5min":0.35499999999999998,"15min":0.316},"max":{"1min":7.8760000000000003,"5min":12.307,"15min":12.307},"last":0.98299999999999998},{"interface":"front","average":{"1min":1.161,"5min":1.6459999999999999,"15min":1.5720000000000001},"min":{"1min":0.39200000000000002,"5min":0.36899999999999999,"15min":0.34399999999999997},"max":{"1min":6.8019999999999996,"5min":10.964,"15min":12.58},"last":0.53200000000000003}]},{"osd":9,"last update":"Fri Feb 9 17:49:40 2024","interfaces":[{"interface":"back","average":{"1min":2.7519999999999998,"5min":2.0979999999999999,"15min":1.895},"min":{"1min":0.33200000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":9.9399999999999995,"5min":12.404999999999999,"15min":13.5},"last":2.4740000000000002},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.9139999999999999,"15min":1.8029999999999999},"min":{"1min":0.45900000000000002,"5min":0.371,"15min":0.36399999999999999},"max":{"1min":3.4020000000000001,"5min":11.058999999999999,"15min":13.589},"last":1.0940000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:50:16 2024","interfaces":[{"interface":"back","average":{"1min":2.1349999999999998,"5min":1.6559999999999999,"15min":1.627},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.33600000000000002},"max":{"1min":7.9269999999999996,"5min":12.314,"15min":12.314},"last":1.1299999999999999},{"interface":"front","average":{"1min":2.1680000000000001,"5min":1.6899999999999999,"15min":1.6659999999999999},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.379},"max":{"1min":7.9509999999999996,"5min":12.342000000000001,"15min":12.342000000000001},"last":1.163}]},{"osd":11,"last update":"Fri Feb 9 17:50:30 2024","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.8380000000000001,"15min":2.2509999999999999},"min":{"1min":0.44,"5min":0.41999999999999998,"15min":0.39600000000000002},"max":{"1min":6.7359999999999998,"5min":11.137,"15min":137.44300000000001},"last":0.999},{"interface":"front","average":{"1min":1.4750000000000001,"5min":1.998,"15min":2.2999999999999998},"min":{"1min":0.504,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":4.524,"5min":12.609999999999999,"15min":137.64400000000001},"last":3.1499999999999999}]}]},{"osd":3,"up_from":27,"seq":115964117374,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1742952,"kb_used_data":1134232,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":92002200,"statfs":{"total":95995035648,"available":94210252800,"internally_reserved":0,"allocated":1161453568,"data_stored":1161115527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":623311303},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:50:24 2024","interfaces":[{"interface":"back","average":{"1min":2.093,"5min":2.2200000000000002,"15min":1.889},"min":{"1min":0.22800000000000001,"5min":0.221,"15min":0.186},"max":{"1min":14.06,"5min":14.06,"15min":38.984999999999999},"last":6.2050000000000001},{"interface":"front","average":{"1min":2.298,"5min":2.9380000000000002,"15min":2.2530000000000001},"min":{"1min":0.247,"5min":0.246,"15min":0.22500000000000001},"max":{"1min":13.113,"5min":13.113,"15min":41.497999999999998},"last":0.81599999999999995}]},{"osd":1,"last update":"Fri Feb 9 17:49:50 2024","interfaces":[{"interface":"back","average":{"1min":5.7220000000000004,"5min":3.7730000000000001,"15min":2.637},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":53.152999999999999,"5min":53.152999999999999,"15min":53.152999999999999},"last":0.67000000000000004},{"interface":"front","average":{"1min":5.3460000000000001,"5min":2.738,"15min":1.716},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":53.118000000000002,"5min":53.118000000000002,"15min":53.118000000000002},"last":0.23200000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:50:23 2024","interfaces":[{"interface":"back","average":{"1min":3.504,"5min":2.4590000000000001,"15min":1.8979999999999999},"min":{"1min":0.29699999999999999,"5min":0.248,"15min":0.248},"max":{"1min":15.734999999999999,"5min":15.734999999999999,"15min":15.734999999999999},"last":6.2720000000000002},{"interface":"front","average":{"1min":2.6120000000000001,"5min":2.5299999999999998,"15min":1.9359999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.20200000000000001},"max":{"1min":8.1240000000000006,"5min":19.879000000000001,"15min":19.879000000000001},"last":0.25800000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:50:24 2024","interfaces":[{"interface":"back","average":{"1min":2.032,"5min":2.3730000000000002,"15min":1.766},"min":{"1min":0.309,"5min":0.28699999999999998,"15min":0.26800000000000002},"max":{"1min":11.374000000000001,"5min":11.374000000000001,"15min":13.933},"last":0.71099999999999997},{"interface":"front","average":{"1min":1.667,"5min":2.1110000000000002,"15min":1.7190000000000001},"min":{"1min":0.309,"5min":0.309,"15min":0.28000000000000003},"max":{"1min":5.6150000000000002,"5min":19.713999999999999,"15min":19.713999999999999},"last":0.35099999999999998}]},{"osd":5,"last update":"Fri Feb 9 17:50:03 2024","interfaces":[{"interface":"back","average":{"1min":2.8460000000000001,"5min":2.5139999999999998,"15min":1.9970000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.27800000000000002},"max":{"1min":14.106,"5min":14.106,"15min":17.651},"last":5.5609999999999999},{"interface":"front","average":{"1min":2.9009999999999998,"5min":2.3839999999999999,"15min":1.9710000000000001},"min":{"1min":0.45900000000000002,"5min":0.42399999999999999,"15min":0.316},"max":{"1min":14.138999999999999,"5min":14.138999999999999,"15min":17.675999999999998},"last":5.5389999999999997}]},{"osd":6,"last update":"Fri Feb 9 17:49:50 2024","interfaces":[{"interface":"back","average":{"1min":3.2909999999999999,"5min":2.3159999999999998,"15min":2.2469999999999999},"min":{"1min":0.434,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":14.178000000000001,"5min":15.061999999999999,"15min":15.061999999999999},"last":5.5049999999999999},{"interface":"front","average":{"1min":2.641,"5min":2.2000000000000002,"15min":1.7310000000000001},"min":{"1min":0.38500000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":6.7300000000000004,"5min":11.813000000000001,"15min":11.813000000000001},"last":5.4749999999999996}]},{"osd":7,"last update":"Fri Feb 9 17:50:34 2024","interfaces":[{"interface":"back","average":{"1min":1.833,"5min":2.2010000000000001,"15min":1.615},"min":{"1min":0.436,"5min":0.36499999999999999,"15min":0.34200000000000003},"max":{"1min":5.6420000000000003,"5min":19.788,"15min":19.788},"last":0.42899999999999999},{"interface":"front","average":{"1min":1.635,"5min":2.5659999999999998,"15min":1.5429999999999999},"min":{"1min":0.52200000000000002,"5min":0.40100000000000002,"15min":0.34899999999999998},"max":{"1min":6.6310000000000002,"5min":11.858000000000001,"15min":11.858000000000001},"last":6.2439999999999998}]},{"osd":8,"last update":"Fri Feb 9 17:45:16 2024","interfaces":[{"interface":"back","average":{"1min":1.714,"5min":1.4450000000000001,"15min":1.498},"min":{"1min":0.377,"5min":0.377,"15min":0.26800000000000002},"max":{"1min":8.6020000000000003,"5min":11.821,"15min":11.821},"last":0.67700000000000005},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.1719999999999999,"15min":1.351},"min":{"1min":0.46500000000000002,"5min":0.36199999999999999,"15min":0.34499999999999997},"max":{"1min":6.8390000000000004,"5min":6.8390000000000004,"15min":14.129},"last":0.78200000000000003}]},{"osd":9,"last update":"Fri Feb 9 17:49:47 2024","interfaces":[{"interface":"back","average":{"1min":3.153,"5min":2.0619999999999998,"15min":1.7689999999999999},"min":{"1min":0.374,"5min":0.34599999999999997,"15min":0.32500000000000001},"max":{"1min":11.144,"5min":11.144,"15min":11.851000000000001},"last":0.79100000000000004},{"interface":"front","average":{"1min":3.1680000000000001,"5min":2.085,"15min":1.784},"min":{"1min":0.40400000000000003,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":11.297000000000001,"5min":11.297000000000001,"15min":12.815},"last":0.83999999999999997}]},{"osd":10,"last update":"Fri Feb 9 17:50:34 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.6830000000000001,"15min":1.2509999999999999},"min":{"1min":0.436,"5min":0.40899999999999997,"15min":0.39100000000000001},"max":{"1min":6.5940000000000003,"5min":6.5940000000000003,"15min":9.7080000000000002},"last":5.5899999999999999},{"interface":"front","average":{"1min":1.603,"5min":1.9450000000000001,"15min":1.296},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":5.7169999999999996,"5min":19.859000000000002,"15min":19.859000000000002},"last":0.46800000000000003}]},{"osd":11,"last update":"Fri Feb 9 17:50:24 2024","interfaces":[{"interface":"back","average":{"1min":1.794,"5min":1.9950000000000001,"15min":1.6160000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.40300000000000002},"max":{"1min":5.6680000000000001,"5min":19.824000000000002,"15min":19.824000000000002},"last":1.3280000000000001},{"interface":"front","average":{"1min":1.8260000000000001,"5min":1.661,"15min":1.7030000000000001},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.442},"max":{"1min":5.742,"5min":6.1219999999999999,"15min":17.145},"last":1.3540000000000001}]}]},{"osd":4,"up_from":32,"seq":137438953849,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1288280,"kb_used_data":607372,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":92456872,"statfs":{"total":95995035648,"available":94675836928,"internally_reserved":0,"allocated":621948928,"data_stored":621615435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":697235909},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:50:38 2024","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.52,"15min":1.478},"min":{"1min":0.23899999999999999,"5min":0.22800000000000001,"15min":0.222},"max":{"1min":9.4949999999999992,"5min":10.301,"15min":10.301},"last":0.33000000000000002},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.5820000000000001,"15min":1.641},"min":{"1min":0.33500000000000002,"5min":0.30299999999999999,"15min":0.247},"max":{"1min":9.5250000000000004,"5min":10.407999999999999,"15min":15.731999999999999},"last":0.35499999999999998}]},{"osd":1,"last update":"Fri Feb 9 17:50:38 2024","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.806,"15min":2.0630000000000002},"min":{"1min":0.26600000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":4.6609999999999996,"5min":8.6120000000000001,"15min":28.143000000000001},"last":1.073},{"interface":"front","average":{"1min":0.98299999999999998,"5min":2.2440000000000002,"15min":2.286},"min":{"1min":0.312,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":3.4060000000000001,"5min":12.279999999999999,"15min":28.103000000000002},"last":0.251}]},{"osd":2,"last update":"Fri Feb 9 17:49:53 2024","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":2.048,"15min":2.7440000000000002},"min":{"1min":0.316,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":4.585,"5min":17.494,"15min":17.494},"last":2.7109999999999999},{"interface":"front","average":{"1min":1.401,"5min":2.1920000000000002,"15min":2.7200000000000002},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":6.0519999999999996,"5min":17.283999999999999,"15min":17.283999999999999},"last":1.3859999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:50:38 2024","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":1.6719999999999999,"15min":1.762},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.245},"max":{"1min":6.7370000000000001,"5min":7.7699999999999996,"15min":14.448},"last":0.92700000000000005},{"interface":"front","average":{"1min":1.7130000000000001,"5min":2.1269999999999998,"15min":1.7829999999999999},"min":{"1min":0.30299999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":9.9949999999999992,"5min":10.382,"15min":10.382},"last":1.0489999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:49:59 2024","interfaces":[{"interface":"back","average":{"1min":1.877,"5min":2.327,"15min":1.9630000000000001},"min":{"1min":0.314,"5min":0.255,"15min":0.22500000000000001},"max":{"1min":8.3480000000000008,"5min":14.779999999999999,"15min":18.887},"last":0.29599999999999999},{"interface":"front","average":{"1min":1.9470000000000001,"5min":2.3929999999999998,"15min":2.0470000000000002},"min":{"1min":0.35399999999999998,"5min":0.31,"15min":0.29099999999999998},"max":{"1min":8.3800000000000008,"5min":14.855,"15min":18.917000000000002},"last":0.433}]},{"osd":6,"last update":"Fri Feb 9 17:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":1.423,"15min":1.5489999999999999},"min":{"1min":0.33200000000000002,"5min":0.29499999999999998,"15min":0.246},"max":{"1min":4.5970000000000004,"5min":11.257,"15min":18.234999999999999},"last":1.0149999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":1.167,"15min":1.633},"min":{"1min":0.316,"5min":0.31,"15min":0.245},"max":{"1min":0.79500000000000004,"5min":10.215,"15min":13.208},"last":0.70299999999999996}]},{"osd":7,"last update":"Fri Feb 9 17:50:12 2024","interfaces":[{"interface":"back","average":{"1min":2.1179999999999999,"5min":1.9239999999999999,"15min":1.6140000000000001},"min":{"1min":0.379,"5min":0.35499999999999998,"15min":0.29899999999999999},"max":{"1min":7.3890000000000002,"5min":10.185,"15min":11.875},"last":1.127},{"interface":"front","average":{"1min":2.657,"5min":1.9079999999999999,"15min":1.784},"min":{"1min":0.41599999999999998,"5min":0.29999999999999999,"15min":0.28199999999999997},"max":{"1min":9.5519999999999996,"5min":10.547000000000001,"15min":12.696},"last":0.84399999999999997}]},{"osd":8,"last update":"Fri Feb 9 17:49:59 2024","interfaces":[{"interface":"back","average":{"1min":1.53,"5min":1.786,"15min":1.661},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33200000000000002},"max":{"1min":5.9550000000000001,"5min":10.641,"15min":17.335999999999999},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.90900000000000003,"5min":1.089,"15min":1.2390000000000001},"min":{"1min":0.378,"5min":0.36699999999999999,"15min":0.34699999999999998},"max":{"1min":6.7039999999999997,"5min":10.433,"15min":10.433},"last":0.73299999999999998}]},{"osd":9,"last update":"Fri Feb 9 17:44:39 2024","interfaces":[{"interface":"back","average":{"1min":2.5739999999999998,"5min":1.855,"15min":1.881},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":10.778,"5min":10.778,"15min":50.442999999999998},"last":0.73599999999999999},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.331,"15min":1.536},"min":{"1min":0.47699999999999998,"5min":0.42499999999999999,"15min":0.36199999999999999},"max":{"1min":5.165,"5min":8.3580000000000005,"15min":10.555},"last":4.8259999999999996}]},{"osd":10,"last update":"Fri Feb 9 17:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.98299999999999998,"15min":1.609},"min":{"1min":0.42499999999999999,"5min":0.40500000000000003,"15min":0.39400000000000002},"max":{"1min":1.27,"5min":10.666,"15min":17.367000000000001},"last":0.40200000000000002},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.601,"15min":1.629},"min":{"1min":0.433,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":4.702,"5min":12.888,"15min":12.888},"last":1.1659999999999999}]},{"osd":11,"last update":"Fri Feb 9 17:49:46 2024","interfaces":[{"interface":"back","average":{"1min":1.137,"5min":1.3169999999999999,"15min":1.583},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":6.7560000000000002,"5min":10.521000000000001,"15min":10.521000000000001},"last":1.099},{"interface":"front","average":{"1min":1.448,"5min":1.738,"15min":1.5740000000000001},"min":{"1min":0.47799999999999998,"5min":0.45100000000000001,"15min":0.40999999999999998},"max":{"1min":4.2649999999999997,"5min":13.154999999999999,"15min":13.154999999999999},"last":0.67000000000000004}]}]},{"osd":5,"up_from":40,"seq":171798692210,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":913168,"kb_used_data":278088,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":92831984,"statfs":{"total":95995035648,"available":95059951616,"internally_reserved":0,"allocated":284762112,"data_stored":284429615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:50:18 2024","interfaces":[{"interface":"back","average":{"1min":1.716,"5min":2.6030000000000002,"15min":1.7569999999999999},"min":{"1min":0.26600000000000001,"5min":0.24299999999999999,"15min":0.23599999999999999},"max":{"1min":8.1430000000000007,"5min":116.77800000000001,"15min":116.77800000000001},"last":1.0980000000000001},{"interface":"front","average":{"1min":1.1839999999999999,"5min":2.9969999999999999,"15min":2.1429999999999998},"min":{"1min":0.27200000000000002,"5min":0.252,"15min":0.249},"max":{"1min":8.4339999999999993,"5min":116.959,"15min":116.959},"last":2.0299999999999998}]},{"osd":1,"last update":"Fri Feb 9 17:50:05 2024","interfaces":[{"interface":"back","average":{"1min":3.4140000000000001,"5min":2.3380000000000001,"15min":2.0710000000000002},"min":{"1min":0.30199999999999999,"5min":0.254,"15min":0.24199999999999999},"max":{"1min":19.079999999999998,"5min":19.079999999999998,"15min":48.161999999999999},"last":2.5049999999999999},{"interface":"front","average":{"1min":2.7290000000000001,"5min":2.25,"15min":2.0950000000000002},"min":{"1min":0.32100000000000001,"5min":0.248,"15min":0.23599999999999999},"max":{"1min":19.065999999999999,"5min":19.065999999999999,"15min":48.218000000000004},"last":0.25800000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:49:41 2024","interfaces":[{"interface":"back","average":{"1min":2.6200000000000001,"5min":2.621,"15min":1.891},"min":{"1min":0.317,"5min":0.249,"15min":0.249},"max":{"1min":7.6600000000000001,"5min":13.929,"15min":13.929},"last":0.89300000000000002},{"interface":"front","average":{"1min":3.073,"5min":2.4900000000000002,"15min":2.2480000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":14.978,"5min":15.945,"15min":15.945},"last":2.3220000000000001}]},{"osd":3,"last update":"Fri Feb 9 17:50:05 2024","interfaces":[{"interface":"back","average":{"1min":3.2879999999999998,"5min":2.7759999999999998,"15min":2.3769999999999998},"min":{"1min":0.377,"5min":0.34699999999999998,"15min":0.26800000000000002},"max":{"1min":11.914999999999999,"5min":15.819000000000001,"15min":64.150999999999996},"last":2.1949999999999998},{"interface":"front","average":{"1min":2.597,"5min":2.738,"15min":2.4020000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.247},"max":{"1min":8.7210000000000001,"5min":11.747,"15min":64.176000000000002},"last":0.63600000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:50:05 2024","interfaces":[{"interface":"back","average":{"1min":1.6890000000000001,"5min":1.8149999999999999,"15min":1.8460000000000001},"min":{"1min":0.32900000000000001,"5min":0.28499999999999998,"15min":0.249},"max":{"1min":8.3480000000000008,"5min":28.481999999999999,"15min":49.313000000000002},"last":1.984},{"interface":"front","average":{"1min":2.121,"5min":2.0499999999999998,"15min":1.877},"min":{"1min":0.34100000000000003,"5min":0.28499999999999998,"15min":0.218},"max":{"1min":8.6609999999999996,"5min":15.843,"15min":50.505000000000003},"last":0.316}]},{"osd":6,"last update":"Fri Feb 9 17:50:38 2024","interfaces":[{"interface":"back","average":{"1min":1.45,"5min":1.452,"15min":1.6879999999999999},"min":{"1min":0.30199999999999999,"5min":0.27900000000000003,"15min":0.26400000000000001},"max":{"1min":8.3829999999999991,"5min":15.026,"15min":16.657},"last":2.0129999999999999},{"interface":"front","average":{"1min":1.601,"5min":1.653,"15min":1.774},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.26200000000000001},"max":{"1min":8.4120000000000008,"5min":20.192,"15min":20.192},"last":2.048}]},{"osd":7,"last update":"Fri Feb 9 17:50:05 2024","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.768,"15min":1.857},"min":{"1min":0.38600000000000001,"5min":0.313,"15min":0.26700000000000002},"max":{"1min":7.0839999999999996,"5min":15.891999999999999,"15min":50.545000000000002},"last":0.60199999999999998},{"interface":"front","average":{"1min":1.7729999999999999,"5min":2.1890000000000001,"15min":2.4390000000000001},"min":{"1min":0.39600000000000002,"5min":0.31,"15min":0.31},"max":{"1min":5.3650000000000002,"5min":12.326000000000001,"15min":102.526},"last":0.78600000000000003}]},{"osd":8,"last update":"Fri Feb 9 17:43:23 2024","interfaces":[{"interface":"back","average":{"1min":1.6499999999999999,"5min":1.639,"15min":1.6020000000000001},"min":{"1min":0.376,"5min":0.35299999999999998,"15min":0.34899999999999998},"max":{"1min":4.8780000000000001,"5min":14.542,"15min":14.542},"last":5.5439999999999996},{"interface":"front","average":{"1min":3.6699999999999999,"5min":1.702,"15min":1.8300000000000001},"min":{"1min":0.37,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":16.359000000000002,"5min":16.359000000000002,"15min":16.359000000000002},"last":0.435}]},{"osd":9,"last update":"Fri Feb 9 17:50:38 2024","interfaces":[{"interface":"back","average":{"1min":1.5189999999999999,"5min":1.26,"15min":1.5},"min":{"1min":0.42899999999999999,"5min":0.36599999999999999,"15min":0.35299999999999998},"max":{"1min":8.3179999999999996,"5min":15.124000000000001,"15min":16.411000000000001},"last":2.0649999999999999},{"interface":"front","average":{"1min":1.508,"5min":1.6739999999999999,"15min":1.665},"min":{"1min":0.38700000000000001,"5min":0.379,"15min":0.35299999999999998},"max":{"1min":5.2969999999999997,"5min":7.7560000000000002,"15min":16.417999999999999},"last":0.83099999999999996}]},{"osd":10,"last update":"Fri Feb 9 17:50:05 2024","interfaces":[{"interface":"back","average":{"1min":1.468,"5min":2.3159999999999998,"15min":1.861},"min":{"1min":0.41599999999999998,"5min":0.39400000000000002,"15min":0.317},"max":{"1min":4.4000000000000004,"5min":80.042000000000002,"15min":80.042000000000002},"last":0.373},{"interface":"front","average":{"1min":1.754,"5min":2.4279999999999999,"15min":1.843},"min":{"1min":0.44500000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":5.3419999999999996,"5min":79.807000000000002,"15min":79.807000000000002},"last":1.399}]},{"osd":11,"last update":"Fri Feb 9 17:50:18 2024","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.8280000000000001,"15min":1.9750000000000001},"min":{"1min":0.47199999999999998,"5min":0.39400000000000002,"15min":0.38400000000000001},"max":{"1min":5.335,"5min":7.798,"15min":16.390000000000001},"last":0.85999999999999999},{"interface":"front","average":{"1min":1.7569999999999999,"5min":1.597,"15min":1.742},"min":{"1min":0.47599999999999998,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":5.2910000000000004,"5min":7.1470000000000002,"15min":9.8239999999999998},"last":0.44}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":806813696,"data_stored":806813696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":442236928,"data_stored":442236928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":741736448,"data_stored":741736448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1189478400,"data_stored":1189478400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":626917376,"data_stored":626917376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":292093952,"data_stored":292093952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":458031104,"data_stored":458031104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":339607552,"data_stored":339607552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":489881600,"data_stored":489881600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":815267840,"data_stored":815267840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":397410304,"data_stored":397410304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":770506752,"data_stored":770506752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-09T17:50:47.488 INFO:tasks.thrashosds.thrasher:Setting 1.1e to [6, 1, 9, 10, 8, 3] 2024-02-09T17:50:47.488 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '1.1e', '6', '1', '9', '10', '8', '3'] 2024-02-09T17:50:47.488 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd pg-upmap-items 1.1e 6 1 9 10 8 3 2024-02-09T17:50:47.507 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 21 16 49638 49622 147.637 99.625 0.0125767 0.00676909 2024-02-09T17:50:47.575 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:50:47.607 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:47.636 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:50:47.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:47.642 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:50:47.743 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:50:47.779 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:47.809 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:50:47.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:47.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:50:47.813 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:47 smithi162 ceph-mon[44304]: osdmap e672: 12 total, 12 up, 12 in 2024-02-09T17:50:47.813 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:47 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:50:47.814 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:47 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:50:47.814 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:47 smithi162 ceph-mon[44304]: pgmap v1693: 47 pgs: 47 active+clean; 2.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 112 MiB/s wr, 1.79k op/s 2024-02-09T17:50:47.914 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:50:47.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:47 smithi179 ceph-mon[43975]: osdmap e672: 12 total, 12 up, 12 in 2024-02-09T17:50:47.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:47 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:50:47.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:47 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:50:47.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:47 smithi179 ceph-mon[43975]: pgmap v1693: 47 pgs: 47 active+clean; 2.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 112 MiB/s wr, 1.79k op/s 2024-02-09T17:50:47.948 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:47.955 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:50:47.958 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-09T17:50:48.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:47 smithi099 ceph-mon[39420]: osdmap e672: 12 total, 12 up, 12 in 2024-02-09T17:50:48.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:47 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:50:48.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:47 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:50:48.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:47 smithi099 ceph-mon[39420]: pgmap v1693: 47 pgs: 47 active+clean; 2.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 112 MiB/s wr, 1.79k op/s 2024-02-09T17:50:48.508 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 22 16 51287 51271 145.611 103.062 0.00798089 0.0068632 2024-02-09T17:50:48.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:48 smithi162 ceph-mon[44304]: osdmap e673: 12 total, 12 up, 12 in 2024-02-09T17:50:48.975 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:50:48.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:48 smithi099 ceph-mon[39420]: osdmap e673: 12 total, 12 up, 12 in 2024-02-09T17:50:49.003 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:50:49.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:49.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:50:49.109 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:50:49.138 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:49.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:48 smithi179 ceph-mon[43975]: osdmap e673: 12 total, 12 up, 12 in 2024-02-09T17:50:49.335 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:50:49.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:49.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:50:49.441 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:50:49.470 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:49.508 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 23 16 52229 52213 141.84 58.875 0.00235365 0.00688326 2024-02-09T17:50:49.857 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:50:49.857 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-02-09T17:50:49.857 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-02-09T17:50:49.858 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-02-09T17:50:49.858 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.858 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:126294 5.9 5:96d777d6:::benchmark_data_smithi179_115199_object126293:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e501)", 2024-02-09T17:50:49.858 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:42:31.641173+0000", 2024-02-09T17:50:49.858 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 498.21149403999999, 2024-02-09T17:50:49.858 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.226279324, 2024-02-09T17:50:49.858 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:50:49.858 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:49.858 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:50:49.858 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:50:49.858 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:50:49.858 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 126294 2024-02-09T17:50:49.858 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.858 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:50:49.859 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.859 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:50:49.859 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.641173+0000", 2024-02-09T17:50:49.859 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:49.859 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.859 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.859 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:50:49.859 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.641173+0000", 2024-02-09T17:50:49.859 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:49.859 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.859 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.859 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:50:49.860 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.641174+0000", 2024-02-09T17:50:49.860 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.438e-06 2024-02-09T17:50:49.860 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.860 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.860 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:50:49.860 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.641245+0000", 2024-02-09T17:50:49.860 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1007000000000006e-05 2024-02-09T17:50:49.860 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.860 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.860 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:50:49.860 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.641263+0000", 2024-02-09T17:50:49.860 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7717e-05 2024-02-09T17:50:49.860 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.860 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.861 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:50:49.861 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.641273+0000", 2024-02-09T17:50:49.861 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9410000000000002e-06 2024-02-09T17:50:49.861 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.861 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.861 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:49.861 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.641392+0000", 2024-02-09T17:50:49.861 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011917300000000001 2024-02-09T17:50:49.861 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.861 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.861 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:50:49.861 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.641540+0000", 2024-02-09T17:50:49.861 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000147415 2024-02-09T17:50:49.861 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.862 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.862 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-02-09T17:50:49.862 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.641654+0000", 2024-02-09T17:50:49.862 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000113904 2024-02-09T17:50:49.862 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.862 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.862 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:50:49.862 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.649067+0000", 2024-02-09T17:50:49.862 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0074131860000000004 2024-02-09T17:50:49.862 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.862 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.862 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:49.862 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.654523+0000", 2024-02-09T17:50:49.862 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0054562480000000003 2024-02-09T17:50:49.862 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.863 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.863 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:49.863 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.867341+0000", 2024-02-09T17:50:49.863 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21281793500000001 2024-02-09T17:50:49.863 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.863 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.863 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:50:49.863 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.867407+0000", 2024-02-09T17:50:49.863 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5939999999999995e-05 2024-02-09T17:50:49.863 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.863 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.863 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:50:49.863 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.867452+0000", 2024-02-09T17:50:49.863 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5420000000000002e-05 2024-02-09T17:50:49.864 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:49.864 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:50:49.864 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:49.864 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.864 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.864 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25528.0:126295 5.9 5:99c08477:::benchmark_data_smithi179_115199_object126294:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e501)", 2024-02-09T17:50:49.864 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:42:31.641296+0000", 2024-02-09T17:50:49.864 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 498.211371482, 2024-02-09T17:50:49.864 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22630713299999999, 2024-02-09T17:50:49.864 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:50:49.864 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:49.864 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:50:49.864 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25528", 2024-02-09T17:50:49.864 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:50:49.865 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 126295 2024-02-09T17:50:49.865 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.865 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:50:49.865 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.865 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:50:49.865 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.641296+0000", 2024-02-09T17:50:49.865 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:49.865 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.865 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.865 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:50:49.865 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.641296+0000", 2024-02-09T17:50:49.865 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:49.865 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.865 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.865 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:50:49.866 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.641296+0000", 2024-02-09T17:50:49.866 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1900000000000001e-07 2024-02-09T17:50:49.866 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.866 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.866 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:50:49.866 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.641330+0000", 2024-02-09T17:50:49.866 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3914000000000001e-05 2024-02-09T17:50:49.866 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.866 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.866 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:50:49.866 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.641339+0000", 2024-02-09T17:50:49.866 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.456e-06 2024-02-09T17:50:49.866 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.866 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.866 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:50:49.867 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.641345+0000", 2024-02-09T17:50:49.867 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5830000000000003e-06 2024-02-09T17:50:49.867 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.867 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.867 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:49.867 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.645645+0000", 2024-02-09T17:50:49.867 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0043000649999999996 2024-02-09T17:50:49.867 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.867 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.867 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:50:49.867 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.645792+0000", 2024-02-09T17:50:49.867 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014633699999999999 2024-02-09T17:50:49.867 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.867 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.867 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-02-09T17:50:49.867 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.645886+0000", 2024-02-09T17:50:49.868 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4126e-05 2024-02-09T17:50:49.868 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.868 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.868 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:50:49.868 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.649221+0000", 2024-02-09T17:50:49.868 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003335438 2024-02-09T17:50:49.868 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.868 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.868 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:49.868 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.669407+0000", 2024-02-09T17:50:49.868 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020185446999999999 2024-02-09T17:50:49.868 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.868 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.868 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:49.869 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.867504+0000", 2024-02-09T17:50:49.869 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19809727499999999 2024-02-09T17:50:49.869 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.869 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.869 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:50:49.869 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.867565+0000", 2024-02-09T17:50:49.869 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0692000000000001e-05 2024-02-09T17:50:49.869 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.869 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.869 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:50:49.869 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:42:31.867603+0000", 2024-02-09T17:50:49.869 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7981000000000001e-05 2024-02-09T17:50:49.869 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:49.870 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:50:49.870 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:49.870 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.870 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.870 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25837.0:31846 6.3 6:c78d3534:::benchmark_data_smithi179_126206_object31845:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e580)", 2024-02-09T17:50:49.870 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:46:03.943859+0000", 2024-02-09T17:50:49.870 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 285.90880766999999, 2024-02-09T17:50:49.870 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20644096200000001, 2024-02-09T17:50:49.870 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:50:49.870 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:49.870 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:50:49.870 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25837", 2024-02-09T17:50:49.870 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:50:49.870 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 31846 2024-02-09T17:50:49.870 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.871 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:50:49.871 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.871 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:50:49.871 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:03.943859+0000", 2024-02-09T17:50:49.871 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:49.871 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.871 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.871 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:50:49.871 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:03.943859+0000", 2024-02-09T17:50:49.871 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:49.871 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.871 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.871 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:50:49.871 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:03.943860+0000", 2024-02-09T17:50:49.872 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.08e-06 2024-02-09T17:50:49.872 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.872 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.872 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:50:49.872 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:03.943888+0000", 2024-02-09T17:50:49.872 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7860000000000001e-05 2024-02-09T17:50:49.872 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.872 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.872 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:50:49.872 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:03.943897+0000", 2024-02-09T17:50:49.872 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7050000000000005e-06 2024-02-09T17:50:49.872 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.872 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.872 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:50:49.872 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:03.943904+0000", 2024-02-09T17:50:49.873 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8050000000000001e-06 2024-02-09T17:50:49.873 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.873 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.873 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:49.873 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:03.943965+0000", 2024-02-09T17:50:49.873 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0979e-05 2024-02-09T17:50:49.873 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.873 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.873 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:50:49.873 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:03.944116+0000", 2024-02-09T17:50:49.873 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015082600000000001 2024-02-09T17:50:49.873 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.873 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.873 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-02-09T17:50:49.874 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:03.944264+0000", 2024-02-09T17:50:49.874 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014844500000000001 2024-02-09T17:50:49.874 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.874 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.874 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:50:49.874 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:03.945608+0000", 2024-02-09T17:50:49.874 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013436660000000001 2024-02-09T17:50:49.874 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.874 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.874 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:49.874 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:03.946051+0000", 2024-02-09T17:50:49.874 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00044373500000000001 2024-02-09T17:50:49.874 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.874 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.875 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:49.875 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:04.150179+0000", 2024-02-09T17:50:49.875 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20412766399999999 2024-02-09T17:50:49.875 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.875 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.875 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:50:49.875 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:04.150255+0000", 2024-02-09T17:50:49.875 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5561000000000004e-05 2024-02-09T17:50:49.875 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.875 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.875 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:50:49.875 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:04.150300+0000", 2024-02-09T17:50:49.875 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5636000000000001e-05 2024-02-09T17:50:49.875 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:49.875 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:50:49.876 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:49.876 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.876 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.876 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25837.0:31857 6.3 6:c8202ba4:::benchmark_data_smithi179_126206_object31856:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e580)", 2024-02-09T17:50:49.876 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:46:03.946335+0000", 2024-02-09T17:50:49.876 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 285.90633254800002, 2024-02-09T17:50:49.876 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20583396500000001, 2024-02-09T17:50:49.876 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:50:49.876 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:49.876 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:50:49.876 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25837", 2024-02-09T17:50:49.876 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:50:49.876 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 31857 2024-02-09T17:50:49.876 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.877 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:50:49.877 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.877 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:50:49.877 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:03.946335+0000", 2024-02-09T17:50:49.877 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:49.877 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.877 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.877 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:50:49.877 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:03.946335+0000", 2024-02-09T17:50:49.877 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:49.877 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.877 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.877 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:50:49.877 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:03.946336+0000", 2024-02-09T17:50:49.877 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3769999999999999e-06 2024-02-09T17:50:49.878 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.878 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.878 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:50:49.878 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:03.946404+0000", 2024-02-09T17:50:49.878 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7681000000000005e-05 2024-02-09T17:50:49.878 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.878 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.878 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:50:49.878 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:03.946414+0000", 2024-02-09T17:50:49.878 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0232e-05 2024-02-09T17:50:49.878 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.878 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.878 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:50:49.878 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:03.946421+0000", 2024-02-09T17:50:49.878 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0280000000000003e-06 2024-02-09T17:50:49.879 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.879 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.879 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:49.879 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:03.946477+0000", 2024-02-09T17:50:49.879 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6447999999999999e-05 2024-02-09T17:50:49.879 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.879 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.879 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:50:49.879 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:03.946597+0000", 2024-02-09T17:50:49.879 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000119992 2024-02-09T17:50:49.879 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.879 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.879 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-02-09T17:50:49.879 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:03.946728+0000", 2024-02-09T17:50:49.879 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013100300000000001 2024-02-09T17:50:49.880 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.880 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.880 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:50:49.880 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:03.949543+0000", 2024-02-09T17:50:49.880 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0028148980000000001 2024-02-09T17:50:49.880 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.880 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.880 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:49.880 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:03.950881+0000", 2024-02-09T17:50:49.880 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013373980000000001 2024-02-09T17:50:49.880 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.880 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.880 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:49.880 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:04.152092+0000", 2024-02-09T17:50:49.881 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.201211682 2024-02-09T17:50:49.881 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.881 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.881 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:50:49.881 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:04.152128+0000", 2024-02-09T17:50:49.881 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5299999999999997e-05 2024-02-09T17:50:49.881 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.881 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.881 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:50:49.881 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:04.152168+0000", 2024-02-09T17:50:49.881 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0926000000000001e-05 2024-02-09T17:50:49.881 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:49.881 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:50:49.881 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:49.881 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.882 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.882 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25837.0:229857 6.3 6:c0fa7fc4:::benchmark_data_smithi179_126206_object99476:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e649)", 2024-02-09T17:50:49.882 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:49:20.410330+0000", 2024-02-09T17:50:49.882 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 89.442337359999996, 2024-02-09T17:50:49.882 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.091095494999999999, 2024-02-09T17:50:49.882 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:50:49.882 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:49.882 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:50:49.882 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25837", 2024-02-09T17:50:49.882 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:50:49.882 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 229857 2024-02-09T17:50:49.882 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.882 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:50:49.882 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.882 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:50:49.883 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:20.410330+0000", 2024-02-09T17:50:49.883 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:49.883 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.883 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.883 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:50:49.883 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:20.410330+0000", 2024-02-09T17:50:49.883 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:49.883 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.883 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.883 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:50:49.883 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:20.410332+0000", 2024-02-09T17:50:49.883 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8449999999999999e-06 2024-02-09T17:50:49.883 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.883 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.884 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:50:49.884 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:20.410336+0000", 2024-02-09T17:50:49.884 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7960000000000003e-06 2024-02-09T17:50:49.884 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.884 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.884 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:50:49.884 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:20.410350+0000", 2024-02-09T17:50:49.884 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3182e-05 2024-02-09T17:50:49.884 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.884 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.884 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:50:49.884 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:20.410358+0000", 2024-02-09T17:50:49.884 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0120000000000005e-06 2024-02-09T17:50:49.885 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.885 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.885 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:49.885 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:20.410417+0000", 2024-02-09T17:50:49.885 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9354999999999997e-05 2024-02-09T17:50:49.885 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.885 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.885 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:50:49.885 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:20.410541+0000", 2024-02-09T17:50:49.885 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012428299999999999 2024-02-09T17:50:49.885 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.885 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.885 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-02-09T17:50:49.885 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:20.410661+0000", 2024-02-09T17:50:49.885 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000119806 2024-02-09T17:50:49.886 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.886 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.886 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:50:49.886 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:20.414955+0000", 2024-02-09T17:50:49.886 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042935300000000003 2024-02-09T17:50:49.886 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.886 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.886 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:49.886 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:20.420304+0000", 2024-02-09T17:50:49.886 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0053495720000000004 2024-02-09T17:50:49.886 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.886 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.886 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:49.886 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:20.501344+0000", 2024-02-09T17:50:49.887 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.081040323999999997 2024-02-09T17:50:49.887 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.887 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.887 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:50:49.887 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:20.501382+0000", 2024-02-09T17:50:49.887 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7913000000000001e-05 2024-02-09T17:50:49.887 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.887 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.887 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:50:49.887 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:20.501425+0000", 2024-02-09T17:50:49.887 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2877000000000003e-05 2024-02-09T17:50:49.887 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:49.887 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:50:49.887 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:49.887 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.888 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.888 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25837.0:254297 6.3 6:cd45665f:::benchmark_data_smithi179_126206_object123916:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:50:49.888 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:49:52.578440+0000", 2024-02-09T17:50:49.888 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 57.274227031000002, 2024-02-09T17:50:49.888 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.218960764, 2024-02-09T17:50:49.888 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:50:49.888 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:49.888 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:50:49.888 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25837", 2024-02-09T17:50:49.888 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:50:49.888 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 254297 2024-02-09T17:50:49.888 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.888 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:50:49.888 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.888 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:50:49.889 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.578440+0000", 2024-02-09T17:50:49.889 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:49.889 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.889 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.889 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:50:49.889 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.578440+0000", 2024-02-09T17:50:49.889 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:49.889 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.889 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.889 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:50:49.889 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.578441+0000", 2024-02-09T17:50:49.889 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.42e-06 2024-02-09T17:50:49.889 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.889 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.890 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:50:49.890 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.578445+0000", 2024-02-09T17:50:49.890 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6119999999999999e-06 2024-02-09T17:50:49.890 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.890 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.890 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:50:49.890 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.578454+0000", 2024-02-09T17:50:49.890 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9279999999999999e-06 2024-02-09T17:50:49.890 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.890 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.890 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:50:49.890 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.578460+0000", 2024-02-09T17:50:49.890 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.472e-06 2024-02-09T17:50:49.890 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.890 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.891 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:49.891 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.578519+0000", 2024-02-09T17:50:49.891 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8715999999999998e-05 2024-02-09T17:50:49.891 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.891 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.891 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:50:49.891 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.578651+0000", 2024-02-09T17:50:49.891 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013221500000000001 2024-02-09T17:50:49.891 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.891 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.891 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-02-09T17:50:49.891 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.578764+0000", 2024-02-09T17:50:49.891 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000112608 2024-02-09T17:50:49.891 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.891 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.892 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:50:49.892 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.588646+0000", 2024-02-09T17:50:49.892 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0098819219999999996 2024-02-09T17:50:49.892 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.892 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.892 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:49.892 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.593408+0000", 2024-02-09T17:50:49.892 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0047621299999999998 2024-02-09T17:50:49.892 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.892 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.892 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:49.892 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.797292+0000", 2024-02-09T17:50:49.892 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20388404099999999 2024-02-09T17:50:49.892 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.893 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.893 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:50:49.893 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.797354+0000", 2024-02-09T17:50:49.893 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2098000000000001e-05 2024-02-09T17:50:49.893 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.893 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.893 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:50:49.893 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.797401+0000", 2024-02-09T17:50:49.893 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6601999999999999e-05 2024-02-09T17:50:49.893 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:49.893 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:50:49.893 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:49.893 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.893 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.893 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25837.0:254298 6.3 6:c9777fa5:::benchmark_data_smithi179_126206_object123917:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:50:49.894 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:49:52.578475+0000", 2024-02-09T17:50:49.894 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 57.274192223, 2024-02-09T17:50:49.894 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24596101500000001, 2024-02-09T17:50:49.894 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:50:49.894 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:49.894 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:50:49.894 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25837", 2024-02-09T17:50:49.894 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:50:49.894 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 254298 2024-02-09T17:50:49.894 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.894 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:50:49.894 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.894 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:50:49.894 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.578475+0000", 2024-02-09T17:50:49.894 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:49.894 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.895 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.895 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:50:49.895 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.578475+0000", 2024-02-09T17:50:49.895 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:49.895 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.895 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.895 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:50:49.895 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.578475+0000", 2024-02-09T17:50:49.895 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1399999999999997e-07 2024-02-09T17:50:49.895 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.895 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.895 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:50:49.895 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.578478+0000", 2024-02-09T17:50:49.895 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2759999999999999e-06 2024-02-09T17:50:49.895 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.896 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.896 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:50:49.896 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583629+0000", 2024-02-09T17:50:49.896 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0051514940000000004 2024-02-09T17:50:49.896 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.896 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.896 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:50:49.896 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583640+0000", 2024-02-09T17:50:49.896 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0791999999999999e-05 2024-02-09T17:50:49.896 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.896 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.896 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:49.896 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.584392+0000", 2024-02-09T17:50:49.897 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00075225900000000002 2024-02-09T17:50:49.897 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.897 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.897 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:50:49.897 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.584488+0000", 2024-02-09T17:50:49.897 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5838e-05 2024-02-09T17:50:49.897 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.897 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.897 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-02-09T17:50:49.897 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.584555+0000", 2024-02-09T17:50:49.898 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7478999999999995e-05 2024-02-09T17:50:49.898 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.898 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.898 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:50:49.898 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.593428+0000", 2024-02-09T17:50:49.898 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0088728939999999992 2024-02-09T17:50:49.898 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.898 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.898 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:49.898 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.596904+0000", 2024-02-09T17:50:49.898 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034754249999999999 2024-02-09T17:50:49.898 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.898 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.898 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:49.899 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.824355+0000", 2024-02-09T17:50:49.899 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.227451715 2024-02-09T17:50:49.899 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.899 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.899 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:50:49.899 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.824395+0000", 2024-02-09T17:50:49.899 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9900000000000001e-05 2024-02-09T17:50:49.899 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.899 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.899 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:50:49.900 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.824436+0000", 2024-02-09T17:50:49.900 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0528999999999998e-05 2024-02-09T17:50:49.900 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:49.900 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:50:49.900 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:49.900 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.900 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.900 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25837.0:254300 6.5 6:aac8f1b4:::benchmark_data_smithi179_126206_object123919:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:50:49.900 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:49:52.583663+0000", 2024-02-09T17:50:49.900 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 57.269003566999999, 2024-02-09T17:50:49.900 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.212262326, 2024-02-09T17:50:49.900 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:50:49.901 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:49.901 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:50:49.901 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25837", 2024-02-09T17:50:49.901 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:50:49.901 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 254300 2024-02-09T17:50:49.901 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.901 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:50:49.901 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.901 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:50:49.901 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583663+0000", 2024-02-09T17:50:49.901 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:49.901 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.901 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.901 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:50:49.901 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583663+0000", 2024-02-09T17:50:49.902 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:49.902 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.902 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.902 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:50:49.902 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583665+0000", 2024-02-09T17:50:49.902 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.217e-06 2024-02-09T17:50:49.902 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.902 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.902 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:50:49.902 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583669+0000", 2024-02-09T17:50:49.902 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4739999999999999e-06 2024-02-09T17:50:49.902 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.902 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.903 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:50:49.903 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583679+0000", 2024-02-09T17:50:49.903 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7559999999999994e-06 2024-02-09T17:50:49.903 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.903 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.903 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:50:49.903 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583685+0000", 2024-02-09T17:50:49.903 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6130000000000003e-06 2024-02-09T17:50:49.903 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.903 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.903 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:49.903 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583729+0000", 2024-02-09T17:50:49.903 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4237999999999999e-05 2024-02-09T17:50:49.903 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.903 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.904 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:50:49.904 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583847+0000", 2024-02-09T17:50:49.904 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000118322 2024-02-09T17:50:49.904 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.904 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.904 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-02-09T17:50:49.904 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583952+0000", 2024-02-09T17:50:49.904 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000105154 2024-02-09T17:50:49.904 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.904 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.904 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:49.904 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.593158+0000", 2024-02-09T17:50:49.905 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0092052690000000003 2024-02-09T17:50:49.905 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.905 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.905 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:50:49.905 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.593588+0000", 2024-02-09T17:50:49.905 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000430176 2024-02-09T17:50:49.905 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.905 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.905 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:49.905 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.795858+0000", 2024-02-09T17:50:49.905 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20227031700000001 2024-02-09T17:50:49.905 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.905 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.905 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:50:49.906 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.795896+0000", 2024-02-09T17:50:49.906 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8470999999999998e-05 2024-02-09T17:50:49.906 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.906 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.906 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:50:49.906 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.795926+0000", 2024-02-09T17:50:49.906 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9319000000000002e-05 2024-02-09T17:50:49.906 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:49.906 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:50:49.906 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:49.906 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.906 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.906 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25837.0:254302 6.5 6:ac558352:::benchmark_data_smithi179_126206_object123921:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:50:49.906 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:49:52.583698+0000", 2024-02-09T17:50:49.907 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 57.268969159999997, 2024-02-09T17:50:49.907 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22801817599999999, 2024-02-09T17:50:49.907 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:50:49.907 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:49.907 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:50:49.907 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25837", 2024-02-09T17:50:49.907 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:50:49.907 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 254302 2024-02-09T17:50:49.907 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.907 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:50:49.907 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.907 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:50:49.907 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583698+0000", 2024-02-09T17:50:49.907 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:49.908 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.908 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.908 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:50:49.908 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583698+0000", 2024-02-09T17:50:49.908 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:49.908 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.908 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.908 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:50:49.908 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583698+0000", 2024-02-09T17:50:49.908 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9499999999999998e-07 2024-02-09T17:50:49.908 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.908 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.908 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:50:49.908 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583701+0000", 2024-02-09T17:50:49.909 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3760000000000002e-06 2024-02-09T17:50:49.909 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.909 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.909 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:50:49.909 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583707+0000", 2024-02-09T17:50:49.909 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.241e-06 2024-02-09T17:50:49.909 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.909 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.909 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:50:49.909 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583712+0000", 2024-02-09T17:50:49.909 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1800000000000004e-06 2024-02-09T17:50:49.909 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.909 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.910 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:49.910 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.589123+0000", 2024-02-09T17:50:49.910 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0054113909999999998 2024-02-09T17:50:49.910 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.910 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.910 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:50:49.910 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.589215+0000", 2024-02-09T17:50:49.910 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1038999999999994e-05 2024-02-09T17:50:49.910 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.910 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.910 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-02-09T17:50:49.910 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.589298+0000", 2024-02-09T17:50:49.910 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3084999999999995e-05 2024-02-09T17:50:49.910 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.911 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.911 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:50:49.911 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.593594+0000", 2024-02-09T17:50:49.911 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042961290000000001 2024-02-09T17:50:49.911 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.911 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.911 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:49.911 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.594360+0000", 2024-02-09T17:50:49.911 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00076610199999999995 2024-02-09T17:50:49.911 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.911 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.911 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:49.911 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.811612+0000", 2024-02-09T17:50:49.912 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21725206 2024-02-09T17:50:49.912 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.912 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.912 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:50:49.912 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.811672+0000", 2024-02-09T17:50:49.912 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9901000000000001e-05 2024-02-09T17:50:49.912 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.912 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.912 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:50:49.912 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.811716+0000", 2024-02-09T17:50:49.912 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4277000000000003e-05 2024-02-09T17:50:49.912 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:49.912 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:50:49.913 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:49.913 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.913 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.913 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25837.0:254319 6.3 6:c6f40c4d:::benchmark_data_smithi179_126206_object123938:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:50:49.913 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:49:52.603086+0000", 2024-02-09T17:50:49.913 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 57.249581216000003, 2024-02-09T17:50:49.913 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22144201199999999, 2024-02-09T17:50:49.913 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:50:49.913 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:49.913 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:50:49.913 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25837", 2024-02-09T17:50:49.913 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:50:49.913 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 254319 2024-02-09T17:50:49.913 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.913 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:50:49.914 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.914 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:50:49.914 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.603086+0000", 2024-02-09T17:50:49.914 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:49.914 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.914 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.914 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:50:49.914 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.603086+0000", 2024-02-09T17:50:49.914 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:49.914 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.914 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.914 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:50:49.914 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.603087+0000", 2024-02-09T17:50:49.915 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5409999999999999e-06 2024-02-09T17:50:49.915 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.915 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.915 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:50:49.915 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.603093+0000", 2024-02-09T17:50:49.915 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9599999999999997e-06 2024-02-09T17:50:49.915 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.915 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.915 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:50:49.915 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.603106+0000", 2024-02-09T17:50:49.915 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2849e-05 2024-02-09T17:50:49.915 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.915 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.915 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:50:49.916 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.603114+0000", 2024-02-09T17:50:49.916 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9689999999999994e-06 2024-02-09T17:50:49.916 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.916 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.916 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:49.916 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.603179+0000", 2024-02-09T17:50:49.916 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4348000000000001e-05 2024-02-09T17:50:49.916 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.916 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.916 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:50:49.916 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.603326+0000", 2024-02-09T17:50:49.916 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014761700000000001 2024-02-09T17:50:49.916 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.916 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.917 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-02-09T17:50:49.917 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.603460+0000", 2024-02-09T17:50:49.917 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013390799999999999 2024-02-09T17:50:49.917 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.917 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.917 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:49.917 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.609555+0000", 2024-02-09T17:50:49.917 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0060945360000000002 2024-02-09T17:50:49.917 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.917 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.917 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:50:49.917 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.610250+0000", 2024-02-09T17:50:49.917 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00069547599999999999 2024-02-09T17:50:49.918 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.918 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.918 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:49.918 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.824478+0000", 2024-02-09T17:50:49.918 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.214227852 2024-02-09T17:50:49.918 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.918 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.918 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:50:49.918 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.824507+0000", 2024-02-09T17:50:49.918 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9278e-05 2024-02-09T17:50:49.918 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.918 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.918 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:50:49.919 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.824528+0000", 2024-02-09T17:50:49.919 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0678000000000001e-05 2024-02-09T17:50:49.919 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:49.919 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:50:49.919 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:49.919 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.919 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.919 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25837.0:254323 6.5 6:aaac2b69:::benchmark_data_smithi179_126206_object123942:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:50:49.919 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:49:52.604437+0000", 2024-02-09T17:50:49.919 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 57.248230241000002, 2024-02-09T17:50:49.919 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.217027939, 2024-02-09T17:50:49.919 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:50:49.919 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:49.919 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:50:49.920 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25837", 2024-02-09T17:50:49.920 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:50:49.920 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 254323 2024-02-09T17:50:49.920 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.920 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:50:49.920 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.920 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:50:49.920 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.604437+0000", 2024-02-09T17:50:49.920 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:49.920 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.920 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.920 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:50:49.920 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.604437+0000", 2024-02-09T17:50:49.920 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:49.921 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.921 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.921 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:50:49.921 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.604438+0000", 2024-02-09T17:50:49.921 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.051e-06 2024-02-09T17:50:49.921 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.921 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.921 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:50:49.921 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.604443+0000", 2024-02-09T17:50:49.921 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.147e-06 2024-02-09T17:50:49.921 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.921 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.921 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:50:49.921 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.604456+0000", 2024-02-09T17:50:49.922 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3414000000000001e-05 2024-02-09T17:50:49.922 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.922 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.922 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:50:49.922 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.604465+0000", 2024-02-09T17:50:49.922 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5359999999999992e-06 2024-02-09T17:50:49.922 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.922 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.922 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:49.922 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.604517+0000", 2024-02-09T17:50:49.922 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2478999999999997e-05 2024-02-09T17:50:49.922 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.922 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.922 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:50:49.923 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.604630+0000", 2024-02-09T17:50:49.923 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011217099999999999 2024-02-09T17:50:49.923 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.923 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.923 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-02-09T17:50:49.923 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.604706+0000", 2024-02-09T17:50:49.923 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6616e-05 2024-02-09T17:50:49.923 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.923 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.923 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:49.923 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.609972+0000", 2024-02-09T17:50:49.923 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0052659459999999996 2024-02-09T17:50:49.923 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.923 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.924 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:50:49.924 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.610237+0000", 2024-02-09T17:50:49.924 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00026531299999999999 2024-02-09T17:50:49.924 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.924 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.924 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:49.924 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.821367+0000", 2024-02-09T17:50:49.924 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21112936800000001 2024-02-09T17:50:49.924 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.924 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.924 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:50:49.924 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.821423+0000", 2024-02-09T17:50:49.924 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6109000000000001e-05 2024-02-09T17:50:49.924 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.924 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.925 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:50:49.925 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.821465+0000", 2024-02-09T17:50:49.925 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1789000000000003e-05 2024-02-09T17:50:49.925 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:49.925 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:50:49.925 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:49.925 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.925 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.925 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25837.0:254329 6.5 6:abdecdae:::benchmark_data_smithi179_126206_object123948:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:50:49.925 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:49:52.613907+0000", 2024-02-09T17:50:49.925 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 57.238759870000003, 2024-02-09T17:50:49.925 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20766915699999999, 2024-02-09T17:50:49.925 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:50:49.925 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:49.925 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:50:49.926 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25837", 2024-02-09T17:50:49.926 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:50:49.926 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 254329 2024-02-09T17:50:49.926 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.926 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:50:49.926 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.926 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:50:49.926 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.613907+0000", 2024-02-09T17:50:49.926 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:49.926 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.926 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.926 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:50:49.926 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.613907+0000", 2024-02-09T17:50:49.926 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:49.927 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.927 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.927 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:50:49.927 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.613909+0000", 2024-02-09T17:50:49.927 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6929999999999999e-06 2024-02-09T17:50:49.927 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.927 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.927 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:50:49.927 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.613915+0000", 2024-02-09T17:50:49.927 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4230000000000002e-06 2024-02-09T17:50:49.927 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.927 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.927 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:50:49.927 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.613930+0000", 2024-02-09T17:50:49.928 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5007999999999999e-05 2024-02-09T17:50:49.928 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.928 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.928 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:50:49.928 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.613940+0000", 2024-02-09T17:50:49.928 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5000000000000005e-06 2024-02-09T17:50:49.928 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.928 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.928 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:49.928 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.614001+0000", 2024-02-09T17:50:49.928 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0888999999999999e-05 2024-02-09T17:50:49.928 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.928 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.928 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:50:49.928 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.614152+0000", 2024-02-09T17:50:49.929 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015174200000000001 2024-02-09T17:50:49.929 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.929 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.929 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-02-09T17:50:49.929 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.614283+0000", 2024-02-09T17:50:49.929 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013045099999999999 2024-02-09T17:50:49.929 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.929 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.929 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:50:49.929 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.618788+0000", 2024-02-09T17:50:49.929 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0045055859999999998 2024-02-09T17:50:49.929 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.929 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.929 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:49.930 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.620381+0000", 2024-02-09T17:50:49.930 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0015923070000000001 2024-02-09T17:50:49.930 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.930 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.930 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:49.930 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.821500+0000", 2024-02-09T17:50:49.930 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20111922300000001 2024-02-09T17:50:49.930 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.930 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.930 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:50:49.930 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.821545+0000", 2024-02-09T17:50:49.930 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4802000000000002e-05 2024-02-09T17:50:49.930 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.931 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.931 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:50:49.931 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.821576+0000", 2024-02-09T17:50:49.931 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1532999999999998e-05 2024-02-09T17:50:49.931 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:49.931 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:50:49.931 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:49.931 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.931 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.931 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25837.0:254336 6.3 6:c045a3b9:::benchmark_data_smithi179_126206_object123955:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:50:49.931 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:49:52.620572+0000", 2024-02-09T17:50:49.931 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 57.232094815000004, 2024-02-09T17:50:49.931 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.204024654, 2024-02-09T17:50:49.931 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:50:49.931 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:49.932 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:50:49.932 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25837", 2024-02-09T17:50:49.932 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:50:49.932 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 254336 2024-02-09T17:50:49.932 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.932 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:50:49.932 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.932 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:50:49.932 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.620572+0000", 2024-02-09T17:50:49.932 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:49.932 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.932 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.932 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:50:49.932 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.620572+0000", 2024-02-09T17:50:49.933 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:49.933 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.933 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.933 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:50:49.933 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.620574+0000", 2024-02-09T17:50:49.933 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.826e-06 2024-02-09T17:50:49.933 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.933 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.933 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:50:49.933 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.620579+0000", 2024-02-09T17:50:49.933 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2039999999999996e-06 2024-02-09T17:50:49.933 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.933 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.933 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:50:49.934 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.620592+0000", 2024-02-09T17:50:49.934 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3052e-05 2024-02-09T17:50:49.934 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.934 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.934 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:50:49.934 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.620600+0000", 2024-02-09T17:50:49.934 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5610000000000001e-06 2024-02-09T17:50:49.934 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.934 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.934 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:49.934 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.620656+0000", 2024-02-09T17:50:49.934 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5738000000000001e-05 2024-02-09T17:50:49.934 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.934 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.934 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:50:49.935 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.620773+0000", 2024-02-09T17:50:49.935 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000117471 2024-02-09T17:50:49.935 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.935 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.935 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-02-09T17:50:49.935 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.620885+0000", 2024-02-09T17:50:49.935 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000111544 2024-02-09T17:50:49.935 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.935 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.935 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:49.935 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.626153+0000", 2024-02-09T17:50:49.935 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0052686039999999996 2024-02-09T17:50:49.935 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.935 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.936 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:50:49.936 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.628281+0000", 2024-02-09T17:50:49.936 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021282380000000002 2024-02-09T17:50:49.936 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.936 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.936 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:49.936 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.824552+0000", 2024-02-09T17:50:49.936 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19627041200000001 2024-02-09T17:50:49.936 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.936 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.936 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:50:49.936 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.824579+0000", 2024-02-09T17:50:49.936 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6631e-05 2024-02-09T17:50:49.936 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.936 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.937 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:50:49.937 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.824597+0000", 2024-02-09T17:50:49.937 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8372999999999998e-05 2024-02-09T17:50:49.937 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:49.937 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:50:49.937 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:49.937 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.937 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.937 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25837.0:254361 6.5 6:addcacb0:::benchmark_data_smithi179_126206_object123980:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:50:49.937 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:49:52.655698+0000", 2024-02-09T17:50:49.937 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 57.196969041999999, 2024-02-09T17:50:49.937 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16599267300000001, 2024-02-09T17:50:49.937 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:50:49.937 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:49.938 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:50:49.938 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25837", 2024-02-09T17:50:49.938 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:50:49.938 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 254361 2024-02-09T17:50:49.938 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.938 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:50:49.938 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.938 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:50:49.938 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.655698+0000", 2024-02-09T17:50:49.938 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:49.938 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.938 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.938 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:50:49.938 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.655698+0000", 2024-02-09T17:50:49.939 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:49.939 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.939 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.939 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:50:49.939 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.655699+0000", 2024-02-09T17:50:49.939 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.283e-06 2024-02-09T17:50:49.939 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.939 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.939 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:50:49.939 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.655705+0000", 2024-02-09T17:50:49.939 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2789999999999998e-06 2024-02-09T17:50:49.939 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.939 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.939 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:50:49.939 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.655716+0000", 2024-02-09T17:50:49.940 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1877e-05 2024-02-09T17:50:49.940 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.940 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.940 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:50:49.940 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.655725+0000", 2024-02-09T17:50:49.940 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4419999999999998e-06 2024-02-09T17:50:49.940 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.940 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.940 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:49.940 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.655768+0000", 2024-02-09T17:50:49.940 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3458000000000003e-05 2024-02-09T17:50:49.940 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.940 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.940 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:50:49.941 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.655849+0000", 2024-02-09T17:50:49.941 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0966999999999999e-05 2024-02-09T17:50:49.941 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.941 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.941 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-02-09T17:50:49.941 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.655949+0000", 2024-02-09T17:50:49.941 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9517000000000003e-05 2024-02-09T17:50:49.941 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.941 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.941 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:50:49.941 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.659984+0000", 2024-02-09T17:50:49.941 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0040348299999999997 2024-02-09T17:50:49.941 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.941 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.941 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:49.942 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.668934+0000", 2024-02-09T17:50:49.942 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0089507319999999994 2024-02-09T17:50:49.942 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.942 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.942 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:49.942 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.821610+0000", 2024-02-09T17:50:49.942 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.15267540900000001 2024-02-09T17:50:49.942 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.942 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.942 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:50:49.942 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.821658+0000", 2024-02-09T17:50:49.942 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8593e-05 2024-02-09T17:50:49.942 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.943 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.943 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:50:49.943 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.821691+0000", 2024-02-09T17:50:49.943 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2286000000000002e-05 2024-02-09T17:50:49.943 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:49.943 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:50:49.943 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:49.943 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.943 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.943 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25837.0:254364 6.5 6:a9cce28d:::benchmark_data_smithi179_126206_object123983:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:50:49.943 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:49:52.661673+0000", 2024-02-09T17:50:49.943 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 57.190994314999998, 2024-02-09T17:50:49.943 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17981395999999999, 2024-02-09T17:50:49.943 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:50:49.944 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:49.944 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:50:49.944 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25837", 2024-02-09T17:50:49.944 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:50:49.944 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 254364 2024-02-09T17:50:49.944 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.944 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:50:49.944 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.944 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:50:49.944 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.661673+0000", 2024-02-09T17:50:49.944 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:49.944 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.944 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.944 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:50:49.945 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.661673+0000", 2024-02-09T17:50:49.945 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:49.945 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.945 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.945 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:50:49.945 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.661674+0000", 2024-02-09T17:50:49.945 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.063e-06 2024-02-09T17:50:49.945 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.947 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.947 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:50:49.947 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.661679+0000", 2024-02-09T17:50:49.947 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1800000000000004e-06 2024-02-09T17:50:49.947 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.947 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.947 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:50:49.948 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.661690+0000", 2024-02-09T17:50:49.948 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1096e-05 2024-02-09T17:50:49.948 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.948 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.948 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:50:49.948 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.661697+0000", 2024-02-09T17:50:49.948 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.596e-06 2024-02-09T17:50:49.948 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.948 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.948 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:49.948 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.661740+0000", 2024-02-09T17:50:49.948 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3582999999999999e-05 2024-02-09T17:50:49.948 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.949 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.949 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:50:49.949 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.661835+0000", 2024-02-09T17:50:49.949 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4518999999999994e-05 2024-02-09T17:50:49.949 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.949 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.949 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-02-09T17:50:49.949 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.661916+0000", 2024-02-09T17:50:49.949 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1212999999999996e-05 2024-02-09T17:50:49.949 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.949 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.949 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:50:49.949 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.666085+0000", 2024-02-09T17:50:49.950 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041686609999999997 2024-02-09T17:50:49.950 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.950 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.950 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:49.950 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.669054+0000", 2024-02-09T17:50:49.950 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0029690010000000002 2024-02-09T17:50:49.950 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.950 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.950 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:49.950 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.841347+0000", 2024-02-09T17:50:49.950 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17229327999999999 2024-02-09T17:50:49.950 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.950 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.951 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:50:49.951 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.841404+0000", 2024-02-09T17:50:49.951 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6719999999999999e-05 2024-02-09T17:50:49.951 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.951 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.951 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:50:49.951 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.841487+0000", 2024-02-09T17:50:49.951 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3047999999999995e-05 2024-02-09T17:50:49.951 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:49.951 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:50:49.951 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:49.951 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.951 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.951 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25837.0:254374 6.3 6:c071e061:::benchmark_data_smithi179_126206_object123993:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:50:49.951 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:49:52.675875+0000", 2024-02-09T17:50:49.952 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 57.176792206999998, 2024-02-09T17:50:49.952 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.151919742, 2024-02-09T17:50:49.952 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:50:49.952 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:49.952 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:50:49.952 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25837", 2024-02-09T17:50:49.952 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:50:49.952 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 254374 2024-02-09T17:50:49.952 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.952 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:50:49.952 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.952 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:50:49.952 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.675875+0000", 2024-02-09T17:50:49.952 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:49.953 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.953 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.953 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:50:49.953 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.675875+0000", 2024-02-09T17:50:49.953 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:49.953 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.953 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.953 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:50:49.953 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.675877+0000", 2024-02-09T17:50:49.953 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8780000000000001e-06 2024-02-09T17:50:49.953 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.953 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.953 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:50:49.953 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.675883+0000", 2024-02-09T17:50:49.953 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0560000000000001e-06 2024-02-09T17:50:49.954 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.954 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.954 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:50:49.954 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.675898+0000", 2024-02-09T17:50:49.954 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4976e-05 2024-02-09T17:50:49.954 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.954 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.954 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:50:49.954 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.675906+0000", 2024-02-09T17:50:49.954 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1210000000000007e-06 2024-02-09T17:50:49.954 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.954 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.954 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:49.954 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.675972+0000", 2024-02-09T17:50:49.954 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5710999999999995e-05 2024-02-09T17:50:49.955 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.955 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.955 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:50:49.955 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.676127+0000", 2024-02-09T17:50:49.955 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015538300000000001 2024-02-09T17:50:49.955 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.955 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.955 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-02-09T17:50:49.955 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.676274+0000", 2024-02-09T17:50:49.955 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000147078 2024-02-09T17:50:49.955 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.955 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.955 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:50:49.955 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.680551+0000", 2024-02-09T17:50:49.956 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042773500000000001 2024-02-09T17:50:49.956 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.956 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.956 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:49.956 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.680811+0000", 2024-02-09T17:50:49.956 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025923599999999998 2024-02-09T17:50:49.956 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.956 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.956 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:49.956 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.827702+0000", 2024-02-09T17:50:49.956 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.14689158099999999 2024-02-09T17:50:49.956 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.956 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.956 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:50:49.957 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.827755+0000", 2024-02-09T17:50:49.957 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2315000000000003e-05 2024-02-09T17:50:49.957 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.957 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.957 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:50:49.957 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.827795+0000", 2024-02-09T17:50:49.957 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0057e-05 2024-02-09T17:50:49.957 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:49.957 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:50:49.957 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:49.957 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.957 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.957 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25837.0:254405 6.3 6:ce8470de:::benchmark_data_smithi179_126206_object124024:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:50:49.957 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:49:52.728404+0000", 2024-02-09T17:50:49.957 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 57.124263145999997, 2024-02-09T17:50:49.958 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.099531968999999998, 2024-02-09T17:50:49.958 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:50:49.958 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:49.958 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:50:49.958 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25837", 2024-02-09T17:50:49.958 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:50:49.958 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 254405 2024-02-09T17:50:49.958 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.958 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:50:49.958 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.958 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:50:49.958 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.728404+0000", 2024-02-09T17:50:49.958 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:49.958 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.958 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.959 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:50:49.959 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.728404+0000", 2024-02-09T17:50:49.959 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:49.959 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.959 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.959 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:50:49.959 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.728406+0000", 2024-02-09T17:50:49.959 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.725e-06 2024-02-09T17:50:49.959 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.959 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.959 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:50:49.959 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.728410+0000", 2024-02-09T17:50:49.959 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3100000000000002e-06 2024-02-09T17:50:49.959 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.960 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.960 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:50:49.960 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.728420+0000", 2024-02-09T17:50:49.960 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0324e-05 2024-02-09T17:50:49.960 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.960 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.960 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:50:49.960 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.728427+0000", 2024-02-09T17:50:49.960 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7240000000000003e-06 2024-02-09T17:50:49.960 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.960 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.960 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:49.960 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.728477+0000", 2024-02-09T17:50:49.960 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9712000000000001e-05 2024-02-09T17:50:49.961 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.961 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.961 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:50:49.961 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.728576+0000", 2024-02-09T17:50:49.961 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9467000000000002e-05 2024-02-09T17:50:49.961 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.961 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.961 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-02-09T17:50:49.961 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.728660+0000", 2024-02-09T17:50:49.961 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3489e-05 2024-02-09T17:50:49.961 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.961 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.961 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:49.961 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.737975+0000", 2024-02-09T17:50:49.961 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0093150009999999998 2024-02-09T17:50:49.962 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.962 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.962 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:50:49.962 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.737985+0000", 2024-02-09T17:50:49.962 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0118999999999999e-05 2024-02-09T17:50:49.962 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.962 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.962 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:49.962 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.827840+0000", 2024-02-09T17:50:49.962 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.089855669999999999 2024-02-09T17:50:49.962 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.962 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.962 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:50:49.962 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.827898+0000", 2024-02-09T17:50:49.962 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7701000000000002e-05 2024-02-09T17:50:49.963 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.963 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.963 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:50:49.963 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.827936+0000", 2024-02-09T17:50:49.963 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7727e-05 2024-02-09T17:50:49.963 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:49.963 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:50:49.963 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:49.963 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.963 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.963 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25837.0:254406 6.3 6:c56758bf:::benchmark_data_smithi179_126206_object124025:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:50:49.963 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:49:52.728445+0000", 2024-02-09T17:50:49.963 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 57.124222457000002, 2024-02-09T17:50:49.963 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.10753573299999999, 2024-02-09T17:50:49.963 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:50:49.964 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:49.964 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:50:49.964 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25837", 2024-02-09T17:50:49.964 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:50:49.964 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 254406 2024-02-09T17:50:49.964 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.964 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:50:49.964 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.964 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:50:49.964 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.728445+0000", 2024-02-09T17:50:49.964 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:49.964 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.964 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.964 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:50:49.965 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.728445+0000", 2024-02-09T17:50:49.965 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:49.965 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.965 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.965 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:50:49.965 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.728445+0000", 2024-02-09T17:50:49.965 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6899999999999998e-07 2024-02-09T17:50:49.965 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.965 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.965 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:50:49.965 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.728447+0000", 2024-02-09T17:50:49.965 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9549999999999999e-06 2024-02-09T17:50:49.965 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.965 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.966 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:50:49.966 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.728452+0000", 2024-02-09T17:50:49.966 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.412e-06 2024-02-09T17:50:49.966 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.966 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.966 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:50:49.966 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.728456+0000", 2024-02-09T17:50:49.966 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8689999999999997e-06 2024-02-09T17:50:49.966 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.966 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.966 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:49.966 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.732702+0000", 2024-02-09T17:50:49.966 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042453489999999998 2024-02-09T17:50:49.966 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.967 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.967 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:50:49.967 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.732837+0000", 2024-02-09T17:50:49.967 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013582899999999999 2024-02-09T17:50:49.967 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.967 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.967 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-02-09T17:50:49.967 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.732964+0000", 2024-02-09T17:50:49.967 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012613300000000001 2024-02-09T17:50:49.967 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.967 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.967 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:49.968 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.738024+0000", 2024-02-09T17:50:49.968 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0050601520000000001 2024-02-09T17:50:49.968 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.968 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.968 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:50:49.968 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.738069+0000", 2024-02-09T17:50:49.968 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5358000000000002e-05 2024-02-09T17:50:49.968 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.968 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.968 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:49.968 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.835884+0000", 2024-02-09T17:50:49.968 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.097814605999999998 2024-02-09T17:50:49.968 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.969 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.969 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:50:49.969 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.835939+0000", 2024-02-09T17:50:49.969 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5676e-05 2024-02-09T17:50:49.969 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.969 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.969 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:50:49.969 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.835980+0000", 2024-02-09T17:50:49.969 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1025000000000001e-05 2024-02-09T17:50:49.969 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:49.969 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:50:49.969 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:49.969 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.969 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.969 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25837.0:254410 6.5 6:a137bf3f:::benchmark_data_smithi179_126206_object124029:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:50:49.970 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:49:52.733427+0000", 2024-02-09T17:50:49.970 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 57.119239870000001, 2024-02-09T17:50:49.970 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.108200144, 2024-02-09T17:50:49.970 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:50:49.970 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:49.970 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:50:49.970 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25837", 2024-02-09T17:50:49.970 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:50:49.970 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 254410 2024-02-09T17:50:49.970 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.970 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:50:49.970 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.970 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:50:49.970 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.733427+0000", 2024-02-09T17:50:49.971 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:49.971 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.971 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.971 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:50:49.971 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.733427+0000", 2024-02-09T17:50:49.971 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:49.971 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.971 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.971 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:50:49.971 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.733428+0000", 2024-02-09T17:50:49.971 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.22e-06 2024-02-09T17:50:49.971 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.971 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.971 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:50:49.971 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.733432+0000", 2024-02-09T17:50:49.972 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3459999999999998e-06 2024-02-09T17:50:49.972 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.972 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.972 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:50:49.972 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.733442+0000", 2024-02-09T17:50:49.972 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0719e-05 2024-02-09T17:50:49.972 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.972 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.972 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:50:49.972 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.733451+0000", 2024-02-09T17:50:49.972 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2609999999999993e-06 2024-02-09T17:50:49.972 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.972 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.972 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:49.973 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.733496+0000", 2024-02-09T17:50:49.973 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5549e-05 2024-02-09T17:50:49.973 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.973 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.973 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:50:49.973 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.733579+0000", 2024-02-09T17:50:49.973 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3036999999999998e-05 2024-02-09T17:50:49.973 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.973 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.973 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-02-09T17:50:49.973 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.733651+0000", 2024-02-09T17:50:49.973 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1594999999999995e-05 2024-02-09T17:50:49.973 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.973 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.973 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:50:49.974 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.737521+0000", 2024-02-09T17:50:49.974 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0038698840000000001 2024-02-09T17:50:49.974 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.974 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.974 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:49.974 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.737808+0000", 2024-02-09T17:50:49.974 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00028684499999999998 2024-02-09T17:50:49.974 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.974 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.974 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:49.974 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.841538+0000", 2024-02-09T17:50:49.974 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.103730278 2024-02-09T17:50:49.974 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.974 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.974 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:50:49.975 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.841591+0000", 2024-02-09T17:50:49.975 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2840999999999998e-05 2024-02-09T17:50:49.975 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.975 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.975 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:50:49.975 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.841627+0000", 2024-02-09T17:50:49.975 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6569000000000002e-05 2024-02-09T17:50:49.975 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:49.975 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:50:49.975 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:49.975 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.975 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.975 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25837.0:254414 6.3 6:cef9b0c0:::benchmark_data_smithi179_126206_object124033:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:50:49.975 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:49:52.744610+0000", 2024-02-09T17:50:49.975 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 57.108056929999996, 2024-02-09T17:50:49.976 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.091493557000000003, 2024-02-09T17:50:49.976 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:50:49.976 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:50:49.976 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:50:49.976 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25837", 2024-02-09T17:50:49.976 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:50:49.976 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 254414 2024-02-09T17:50:49.976 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.976 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:50:49.976 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.976 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:50:49.976 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.744610+0000", 2024-02-09T17:50:49.976 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:49.976 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.976 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.977 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:50:49.977 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.744610+0000", 2024-02-09T17:50:49.977 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:50:49.977 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.977 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.977 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:50:49.977 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.744611+0000", 2024-02-09T17:50:49.977 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.19e-06 2024-02-09T17:50:49.977 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.977 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.977 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:50:49.977 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.744616+0000", 2024-02-09T17:50:49.977 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2119999999999997e-06 2024-02-09T17:50:49.977 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.977 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.978 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:50:49.978 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.744627+0000", 2024-02-09T17:50:49.978 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1141999999999999e-05 2024-02-09T17:50:49.978 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.978 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.978 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:50:49.978 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.744634+0000", 2024-02-09T17:50:49.978 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8150000000000006e-06 2024-02-09T17:50:49.978 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.978 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.978 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:50:49.978 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.744690+0000", 2024-02-09T17:50:49.978 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5417999999999997e-05 2024-02-09T17:50:49.978 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.978 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.979 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:50:49.979 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.744812+0000", 2024-02-09T17:50:49.979 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012238799999999999 2024-02-09T17:50:49.979 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.979 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.979 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-02-09T17:50:49.979 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.744935+0000", 2024-02-09T17:50:49.979 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012295000000000001 2024-02-09T17:50:49.979 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.979 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.979 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:49.979 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.749939+0000", 2024-02-09T17:50:49.979 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.005003904 2024-02-09T17:50:49.979 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.979 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.979 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:50:49.980 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.750032+0000", 2024-02-09T17:50:49.980 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3051000000000001e-05 2024-02-09T17:50:49.980 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.980 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.980 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:50:49.980 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.836021+0000", 2024-02-09T17:50:49.980 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.085988808 2024-02-09T17:50:49.980 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.980 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.980 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:50:49.980 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.836070+0000", 2024-02-09T17:50:49.980 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9177e-05 2024-02-09T17:50:49.980 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:50:49.980 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:50:49.981 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:50:49.981 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.836104+0000", 2024-02-09T17:50:49.981 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3501999999999999e-05 2024-02-09T17:50:49.981 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:49.981 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:50:49.981 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:49.981 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:50:49.981 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:50:49.981 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:50:50.108 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:50:50.120 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:50:50.127 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:49 smithi099 ceph-mon[39420]: pgmap v1695: 47 pgs: 47 active+clean; 2.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 123 MiB/s wr, 1.97k op/s 2024-02-09T17:50:50.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:50.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:50:50.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:49 smithi179 ceph-mon[43975]: pgmap v1695: 47 pgs: 47 active+clean; 2.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 123 MiB/s wr, 1.97k op/s 2024-02-09T17:50:50.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:49 smithi162 ceph-mon[44304]: pgmap v1695: 47 pgs: 47 active+clean; 2.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 123 MiB/s wr, 1.97k op/s 2024-02-09T17:50:50.229 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:50:50.264 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:50.294 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:50:50.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:50.299 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:50:50.400 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:50:50.437 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:50.466 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:50:50.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:50.471 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:50:50.508 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 24 16 53939 53923 140.383 106.875 0.0127335 0.00711848 2024-02-09T17:50:50.572 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:50:50.634 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:50.671 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:50:50.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:50.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:50:50.682 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.9 dump_ops_in_flight 2024-02-09T17:50:50.776 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:50:50.807 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:51.085 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:50:51.457 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:50:51.509 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 25 16 55477 55461 138.613 96.125 0.0192889 0.0072094 2024-02-09T17:50:51.660 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:50:51.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:51.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:50:51.771 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:50:51.808 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:52.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:52 smithi099 ceph-mon[39420]: pgmap v1696: 47 pgs: 47 active+clean; 3.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 106 MiB/s wr, 1.70k op/s 2024-02-09T17:50:52.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:52 smithi099 ceph-mon[39420]: osdmap e674: 12 total, 12 up, 12 in 2024-02-09T17:50:52.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:52 smithi179 ceph-mon[43975]: pgmap v1696: 47 pgs: 47 active+clean; 3.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 106 MiB/s wr, 1.70k op/s 2024-02-09T17:50:52.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:52 smithi179 ceph-mon[43975]: osdmap e674: 12 total, 12 up, 12 in 2024-02-09T17:50:52.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:52 smithi162 ceph-mon[44304]: pgmap v1696: 47 pgs: 47 active+clean; 3.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 106 MiB/s wr, 1.70k op/s 2024-02-09T17:50:52.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:52 smithi162 ceph-mon[44304]: osdmap e674: 12 total, 12 up, 12 in 2024-02-09T17:50:52.508 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 26 16 57120 57104 137.231 102.688 0.009858 0.0072823 2024-02-09T17:50:53.508 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 27 16 58711 58695 135.831 99.4375 0.00489948 0.00735679 2024-02-09T17:50:54.180 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:50:54.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:54.184 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:50:54.285 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:50:54.330 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:54.371 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:50:54.374 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:54 smithi162 ceph-mon[44304]: pgmap v1698: 47 pgs: 47 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 111 MiB/s wr, 1.78k op/s 2024-02-09T17:50:54.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:54.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:50:54.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:54 smithi179 ceph-mon[43975]: pgmap v1698: 47 pgs: 47 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 111 MiB/s wr, 1.78k op/s 2024-02-09T17:50:54.477 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:50:54.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:54 smithi099 ceph-mon[39420]: pgmap v1698: 47 pgs: 47 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 111 MiB/s wr, 1.78k op/s 2024-02-09T17:50:54.508 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 28 16 60229 60213 134.369 94.875 0.0120179 0.00743755 2024-02-09T17:50:54.524 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:54.551 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:50:54.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:54.554 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:50:54.655 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:50:54.700 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:54.728 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:50:54.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:54.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:50:54.834 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:50:54.884 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:55.118 INFO:teuthology.orchestra.run.smithi099.stderr:set 1.1e pg_upmap_items mapping to [6->1,9->10,8->3] 2024-02-09T17:50:55.137 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:55 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/853781360' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.1e", "id": [6, 1, 9, 10, 8, 3]}]: dispatch 2024-02-09T17:50:55.137 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:55 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.1e", "id": [6, 1, 9, 10, 8, 3]}]: dispatch 2024-02-09T17:50:55.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:55 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/853781360' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.1e", "id": [6, 1, 9, 10, 8, 3]}]: dispatch 2024-02-09T17:50:55.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:55 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.1e", "id": [6, 1, 9, 10, 8, 3]}]: dispatch 2024-02-09T17:50:55.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:55 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/853781360' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.1e", "id": [6, 1, 9, 10, 8, 3]}]: dispatch 2024-02-09T17:50:55.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:55 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.1e", "id": [6, 1, 9, 10, 8, 3]}]: dispatch 2024-02-09T17:50:55.508 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 29 16 61535 61519 132.55 81.625 0.00799118 0.00753942 2024-02-09T17:50:55.646 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:50:55.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:55.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:50:55.752 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:50:55.807 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:55.939 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:50:55.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:55.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:50:56.047 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:50:56.090 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:56.212 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:56 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:50:56.212 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:56 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:50:56.213 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:56 smithi099 ceph-mon[39420]: pgmap v1699: 47 pgs: 47 active+clean; 3.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.49k op/s 2024-02-09T17:50:56.213 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:56 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.1e", "id": [6, 1, 9, 10, 8, 3]}]': finished 2024-02-09T17:50:56.213 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:56 smithi099 ceph-mon[39420]: osdmap e675: 12 total, 12 up, 12 in 2024-02-09T17:50:56.306 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:50:56.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:56.309 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:50:56.410 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:50:56.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:56 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:50:56.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:56 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:50:56.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:56 smithi179 ceph-mon[43975]: pgmap v1699: 47 pgs: 47 active+clean; 3.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.49k op/s 2024-02-09T17:50:56.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:56 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.1e", "id": [6, 1, 9, 10, 8, 3]}]': finished 2024-02-09T17:50:56.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:56 smithi179 ceph-mon[43975]: osdmap e675: 12 total, 12 up, 12 in 2024-02-09T17:50:56.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:56 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:50:56.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:56 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:50:56.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:56 smithi162 ceph-mon[44304]: pgmap v1699: 47 pgs: 47 active+clean; 3.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.49k op/s 2024-02-09T17:50:56.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:56 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.1e", "id": [6, 1, 9, 10, 8, 3]}]': finished 2024-02-09T17:50:56.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:56 smithi162 ceph-mon[44304]: osdmap e675: 12 total, 12 up, 12 in 2024-02-09T17:50:56.442 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:56.465 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:50:56.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:56.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:50:56.509 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 30 16 63048 63032 131.284 94.5625 0.00418649 0.00761182 2024-02-09T17:50:56.582 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:50:56.611 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:56.636 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:50:56.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:56.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:50:56.741 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:50:56.777 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:56.796 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:50:56.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:56.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:50:56.900 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:50:56.933 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:56.956 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:50:56.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:56.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:50:57.061 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:50:57.109 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:57.139 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:57 smithi162 ceph-mon[44304]: osdmap e676: 12 total, 12 up, 12 in 2024-02-09T17:50:57.139 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:57 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1915422790' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:50:57.139 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:57 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:50:57.142 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:57 smithi099 ceph-mon[39420]: osdmap e676: 12 total, 12 up, 12 in 2024-02-09T17:50:57.142 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:57 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1915422790' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:50:57.142 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:57 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:50:57.296 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:57 smithi179 ceph-mon[43975]: osdmap e676: 12 total, 12 up, 12 in 2024-02-09T17:50:57.299 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:57 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1915422790' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:50:57.299 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:57 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:50:57.324 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:50:57.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:57.333 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:50:57.434 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:50:57.495 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:57.509 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 31 16 64484 64468 129.944 89.75 0.0112551 0.00769049 2024-02-09T17:50:57.552 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is set 2024-02-09T17:50:57.647 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:50:57.647 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-02-09T17:50:57.647 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:50:57.647 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.16440.0:64677 7.f e677/661 7:f7fc5c9b:::benchmark_data_smithi179_135804_object64676:head v 677'4078, mlcod=677'4076)", 2024-02-09T17:50:57.647 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:50:57.643367+0000", 2024-02-09T17:50:57.647 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.001630776, 2024-02-09T17:50:57.647 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001660466, 2024-02-09T17:50:57.647 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:50:57.647 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-02-09T17:50:57.647 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:50:57.647 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:50:57.648 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:50:57.648 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:57.643367+0000", 2024-02-09T17:50:57.648 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:50:57.648 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:50:57.648 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:50:57.648 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:50:57.648 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:57.643367+0000", 2024-02-09T17:50:57.648 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:50:57.648 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:50:57.648 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:50:57.648 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:50:57.648 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:57.643368+0000", 2024-02-09T17:50:57.648 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0109999999999999e-06 2024-02-09T17:50:57.648 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:50:57.649 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:50:57.649 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:50:57.649 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:57.643392+0000", 2024-02-09T17:50:57.649 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4134e-05 2024-02-09T17:50:57.649 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:50:57.649 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:50:57.649 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:50:57.649 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:57.643407+0000", 2024-02-09T17:50:57.649 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5596000000000001e-05 2024-02-09T17:50:57.649 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:50:57.649 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:50:57.649 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:50:57.649 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:57.643415+0000", 2024-02-09T17:50:57.649 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6199999999999999e-06 2024-02-09T17:50:57.649 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:50:57.650 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:50:57.650 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:50:57.650 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:57.643475+0000", 2024-02-09T17:50:57.650 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9549000000000002e-05 2024-02-09T17:50:57.650 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:50:57.650 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:50:57.650 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:50:57.650 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:57.643495+0000", 2024-02-09T17:50:57.650 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0248999999999999e-05 2024-02-09T17:50:57.650 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:50:57.650 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:50:57.650 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:50:57.650 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:50:57.650 INFO:teuthology.orchestra.run.smithi179.stdout: ], 2024-02-09T17:50:57.651 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 1 2024-02-09T17:50:57.651 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:50:57.757 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:50:57.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:57.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:50:57.864 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:50:57.906 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:58.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:58 smithi179 ceph-mon[43975]: pgmap v1702: 47 pgs: 1 peering, 46 active+clean; 3.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.49k op/s 2024-02-09T17:50:58.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:58 smithi179 ceph-mon[43975]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:50:58.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:58 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:50:58.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:58 smithi179 ceph-mon[43975]: osdmap e677: 12 total, 12 up, 12 in 2024-02-09T17:50:58.378 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.9 dump_blocked_ops 2024-02-09T17:50:58.400 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:50:58.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:58.406 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:58 smithi099 ceph-mon[39420]: pgmap v1702: 47 pgs: 1 peering, 46 active+clean; 3.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.49k op/s 2024-02-09T17:50:58.406 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:58 smithi099 ceph-mon[39420]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:50:58.406 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:58 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:50:58.406 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:58 smithi099 ceph-mon[39420]: osdmap e677: 12 total, 12 up, 12 in 2024-02-09T17:50:58.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:50:58.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:58 smithi162 ceph-mon[44304]: pgmap v1702: 47 pgs: 1 peering, 46 active+clean; 3.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.49k op/s 2024-02-09T17:50:58.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:58 smithi162 ceph-mon[44304]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:50:58.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:58 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:50:58.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:58 smithi162 ceph-mon[44304]: osdmap e677: 12 total, 12 up, 12 in 2024-02-09T17:50:58.508 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:50:58.509 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 32 16 65958 65942 128.762 92.125 0.00870314 0.00776181 2024-02-09T17:50:58.562 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:58.759 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:50:58.931 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:50:58.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:58.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:50:59.037 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:50:59.094 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:59.173 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:50:59 smithi099 ceph-mon[39420]: osdmap e678: 12 total, 12 up, 12 in 2024-02-09T17:50:59.319 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:50:59.325 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:59.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:50:59.426 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:50:59.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:50:59 smithi162 ceph-mon[44304]: osdmap e678: 12 total, 12 up, 12 in 2024-02-09T17:50:59.462 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:59.464 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:50:59 smithi179 ceph-mon[43975]: osdmap e678: 12 total, 12 up, 12 in 2024-02-09T17:50:59.485 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:50:59.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:59.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:50:59.508 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 33 16 67409 67393 127.608 90.6875 0.0136317 0.00783213 2024-02-09T17:50:59.589 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:50:59.624 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:59.647 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:50:59.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:50:59.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:50:59.752 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:50:59.784 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:50:59.797 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:50:59.810 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:50:59.854 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:50:59.900 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:50:59.939 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:50:59.977 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:51:00.018 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:51:00.051 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:51:00.088 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:51:00.120 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:51:00.171 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:51:00.217 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:51:00.230 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:51:00.230 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:51:00.230 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:51:00.230 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:51:00.230 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:51:00.230 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:51:00.230 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:51:00.230 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:51:00.230 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:51:00.230 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:51:00.230 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:51:00.230 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:51:00.444 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset noscrub 2024-02-09T17:51:00.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:00 smithi099 ceph-mon[39420]: pgmap v1705: 47 pgs: 1 peering, 46 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 109 MiB/s wr, 1.75k op/s 2024-02-09T17:51:00.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:00 smithi099 ceph-mon[39420]: osdmap e679: 12 total, 12 up, 12 in 2024-02-09T17:51:00.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:00 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2029635003' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:51:00.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:00 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:51:00.509 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 34 16 68738 68722 126.298 83.0625 0.0133514 0.00791258 2024-02-09T17:51:00.655 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:51:00.658 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:00 smithi179 ceph-mon[43975]: pgmap v1705: 47 pgs: 1 peering, 46 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 109 MiB/s wr, 1.75k op/s 2024-02-09T17:51:00.659 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:00 smithi179 ceph-mon[43975]: osdmap e679: 12 total, 12 up, 12 in 2024-02-09T17:51:00.659 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:00 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2029635003' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:51:00.659 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:00 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:51:00.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:00.662 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:51:00.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:00 smithi162 ceph-mon[44304]: pgmap v1705: 47 pgs: 1 peering, 46 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 109 MiB/s wr, 1.75k op/s 2024-02-09T17:51:00.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:00 smithi162 ceph-mon[44304]: osdmap e679: 12 total, 12 up, 12 in 2024-02-09T17:51:00.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:00 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2029635003' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:51:00.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:00 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:51:00.762 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:51:00.799 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:01.509 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 35 16 70099 70083 125.12 85.0625 0.00540417 0.00798833 2024-02-09T17:51:01.831 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] 2024-02-09T17:51:01.831 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:51:01.831 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-02-09T17:51:01.832 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight-by-utilization 110 1.0 2024-02-09T17:51:01.890 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T17:51:01.890 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:51:02.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:02 smithi099 ceph-mon[39420]: pgmap v1707: 47 pgs: 47 active+clean; 4.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 94 MiB/s wr, 1.50k op/s 2024-02-09T17:51:02.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:02 smithi099 ceph-mon[39420]: osdmap e680: 12 total, 12 up, 12 in 2024-02-09T17:51:02.509 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 36 16 71310 71294 123.747 75.6875 0.00478121 0.00807705 2024-02-09T17:51:02.615 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:51:02.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:02.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:51:02.633 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:02 smithi179 ceph-mon[43975]: pgmap v1707: 47 pgs: 47 active+clean; 4.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 94 MiB/s wr, 1.50k op/s 2024-02-09T17:51:02.633 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:02 smithi179 ceph-mon[43975]: osdmap e680: 12 total, 12 up, 12 in 2024-02-09T17:51:02.663 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.9 dump_historic_ops 2024-02-09T17:51:02.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:02 smithi162 ceph-mon[44304]: pgmap v1707: 47 pgs: 47 active+clean; 4.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 94 MiB/s wr, 1.50k op/s 2024-02-09T17:51:02.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:02 smithi162 ceph-mon[44304]: osdmap e680: 12 total, 12 up, 12 in 2024-02-09T17:51:02.701 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:51:02.721 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:51:02.748 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:51:02.750 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:02.771 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:51:02.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:02.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:51:02.876 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:51:02.920 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:03.018 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:51:03.509 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 37 16 72588 72572 122.561 79.875 0.0170699 0.00815421 2024-02-09T17:51:04.300 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:51:04.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:04.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:51:04.330 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:04 smithi179 ceph-mon[43975]: pgmap v1709: 47 pgs: 47 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.41k op/s 2024-02-09T17:51:04.406 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:51:04.435 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:04.457 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:51:04.459 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:04 smithi162 ceph-mon[44304]: pgmap v1709: 47 pgs: 47 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.41k op/s 2024-02-09T17:51:04.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:04.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:51:04.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:04 smithi099 ceph-mon[39420]: pgmap v1709: 47 pgs: 47 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.41k op/s 2024-02-09T17:51:04.509 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 38 16 73227 73211 120.387 39.9375 0.00770454 0.00830246 2024-02-09T17:51:04.565 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:51:04.630 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:04.659 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:51:04.900 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:51:04.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:04.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:51:05.013 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:51:05.052 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:05.082 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:51:05.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:05.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:51:05.191 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:51:05.229 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:05.509 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 39 16 74532 74516 119.391 81.5625 0.0101262 0.00837191 2024-02-09T17:51:05.671 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T17:51:05.671 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:51:06.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:06 smithi099 ceph-mon[39420]: pgmap v1710: 47 pgs: 47 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.24k op/s 2024-02-09T17:51:06.512 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:2024-02-09T17:51:06.508988+0000 min lat: 0.0011365 max lat: 0.621457 avg lat: 0.00843575 2024-02-09T17:51:06.512 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-09T17:51:06.512 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 40 16 75863 75847 118.486 83.1875 0.0115662 0.00843575 2024-02-09T17:51:06.598 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:51:06.605 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:06.606 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:51:06.606 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:06 smithi179 ceph-mon[43975]: pgmap v1710: 47 pgs: 47 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.24k op/s 2024-02-09T17:51:06.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:06 smithi162 ceph-mon[44304]: pgmap v1710: 47 pgs: 47 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.24k op/s 2024-02-09T17:51:06.706 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:51:06.748 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:06.782 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:51:06.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:06.787 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:51:06.888 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:51:06.942 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:07.243 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.10 dump_ops_in_flight 2024-02-09T17:51:07.511 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 41 16 77188 77172 117.612 82.8125 0.0053724 0.00849786 2024-02-09T17:51:07.631 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:51:08.040 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:07 smithi099 ceph-mon[39420]: pgmap v1711: 47 pgs: 47 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s 2024-02-09T17:51:08.042 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:51:08.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:08.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:51:08.148 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:51:08.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:07 smithi179 ceph-mon[43975]: pgmap v1711: 47 pgs: 47 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s 2024-02-09T17:51:08.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:07 smithi162 ceph-mon[44304]: pgmap v1711: 47 pgs: 47 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s 2024-02-09T17:51:08.194 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:08.511 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 42 16 78420 78404 116.645 77 0.0118239 0.00856883 2024-02-09T17:51:08.714 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:51:08.796 INFO:teuthology.orchestra.run.smithi099.stdout:moved 15 / 141 (10.6383%) 2024-02-09T17:51:08.796 INFO:teuthology.orchestra.run.smithi099.stdout:avg 11.75 2024-02-09T17:51:08.796 INFO:teuthology.orchestra.run.smithi099.stdout:stddev 1.36168 -> 2.24072 (expected baseline 3.28189) 2024-02-09T17:51:08.796 INFO:teuthology.orchestra.run.smithi099.stdout:min osd.5 with 10 -> 11 pgs (0.851064 -> 0.93617 * mean) 2024-02-09T17:51:08.796 INFO:teuthology.orchestra.run.smithi099.stdout:max osd.0 with 14 -> 8 pgs (1.19149 -> 0.680851 * mean) 2024-02-09T17:51:08.797 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:51:08.797 INFO:teuthology.orchestra.run.smithi099.stdout:oload 110 2024-02-09T17:51:08.797 INFO:teuthology.orchestra.run.smithi099.stdout:max_change 1 2024-02-09T17:51:08.797 INFO:teuthology.orchestra.run.smithi099.stdout:max_change_osds 4 2024-02-09T17:51:08.797 INFO:teuthology.orchestra.run.smithi099.stdout:average_utilization 0.0190 2024-02-09T17:51:08.797 INFO:teuthology.orchestra.run.smithi099.stdout:overload_utilization 0.0209 2024-02-09T17:51:08.797 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3 weight 1.0000 -> 0.6593 2024-02-09T17:51:08.797 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0 weight 1.0000 -> 0.7243 2024-02-09T17:51:08.797 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9 weight 1.0000 -> 0.8394 2024-02-09T17:51:08.797 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11 weight 1.0000 -> 0.8403 2024-02-09T17:51:08.797 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is unset 2024-02-09T17:51:09.152 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:51:09.154 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:08 smithi179 ceph-mon[43975]: from='client.26149 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:51:09.154 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:08 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'47471','3':'43210','9':'55009','11':'55069'}"}]: dispatch 2024-02-09T17:51:09.154 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:08 smithi179 ceph-mon[43975]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'47471','3':'43210','9':'55009','11':'55069'}"}]: dispatch 2024-02-09T17:51:09.154 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:08 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/354055781' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:51:09.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:09.155 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:51:09.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:08 smithi162 ceph-mon[44304]: from='client.26149 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:51:09.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:08 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'47471','3':'43210','9':'55009','11':'55069'}"}]: dispatch 2024-02-09T17:51:09.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:08 smithi162 ceph-mon[44304]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'47471','3':'43210','9':'55009','11':'55069'}"}]: dispatch 2024-02-09T17:51:09.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:08 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/354055781' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:51:09.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:08 smithi099 ceph-mon[39420]: from='client.26149 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:51:09.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:08 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'47471','3':'43210','9':'55009','11':'55069'}"}]: dispatch 2024-02-09T17:51:09.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:08 smithi099 ceph-mon[39420]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'47471','3':'43210','9':'55009','11':'55069'}"}]: dispatch 2024-02-09T17:51:09.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:08 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/354055781' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:51:09.256 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:51:09.285 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:09.304 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:51:09.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:09.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:51:09.408 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:51:09.438 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:09.458 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:51:09.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:09.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:51:09.511 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 43 16 78616 78600 114.218 12.25 0.00326267 0.00860546 2024-02-09T17:51:09.570 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:51:09.610 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:09.709 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:51:09.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:09.751 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:51:09.852 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:51:09.885 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:09.886 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-02-09T17:51:09.886 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight-by-utilization 1000 0.05 2024-02-09T17:51:09.903 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:51:09.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:09.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:51:09.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:09 smithi179 ceph-mon[43975]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:51:09.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:09 smithi179 ceph-mon[43975]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'47471','3':'43210','9':'55009','11':'55069'}"}]': finished 2024-02-09T17:51:09.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:09 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/354055781' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:51:09.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:09 smithi179 ceph-mon[43975]: osdmap e681: 12 total, 12 up, 12 in 2024-02-09T17:51:09.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:09 smithi179 ceph-mon[43975]: pgmap v1713: 47 pgs: 47 active+clean; 4.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-02-09T17:51:09.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:09 smithi179 ceph-mon[43975]: osdmap e682: 12 total, 12 up, 12 in 2024-02-09T17:51:10.009 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:51:10.041 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:10.157 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:09 smithi099 ceph-mon[39420]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:51:10.157 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:09 smithi099 ceph-mon[39420]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'47471','3':'43210','9':'55009','11':'55069'}"}]': finished 2024-02-09T17:51:10.158 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:09 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/354055781' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:51:10.158 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:09 smithi099 ceph-mon[39420]: osdmap e681: 12 total, 12 up, 12 in 2024-02-09T17:51:10.158 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:09 smithi099 ceph-mon[39420]: pgmap v1713: 47 pgs: 47 active+clean; 4.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-02-09T17:51:10.158 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:09 smithi099 ceph-mon[39420]: osdmap e682: 12 total, 12 up, 12 in 2024-02-09T17:51:10.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:09 smithi162 ceph-mon[44304]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:51:10.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:09 smithi162 ceph-mon[44304]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'47471','3':'43210','9':'55009','11':'55069'}"}]': finished 2024-02-09T17:51:10.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:09 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/354055781' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:51:10.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:09 smithi162 ceph-mon[44304]: osdmap e681: 12 total, 12 up, 12 in 2024-02-09T17:51:10.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:09 smithi162 ceph-mon[44304]: pgmap v1713: 47 pgs: 47 active+clean; 4.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-02-09T17:51:10.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:09 smithi162 ceph-mon[44304]: osdmap e682: 12 total, 12 up, 12 in 2024-02-09T17:51:10.239 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:51:10.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:10.241 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:51:10.263 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T17:51:10.263 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:51:10.342 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:51:10.375 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:10.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:10.402 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:51:10.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:51:10.503 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:51:10.511 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 44 16 78616 78600 111.622 0 - 0.00860546 2024-02-09T17:51:10.533 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:10.553 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:51:10.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:10.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:51:10.657 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:51:10.685 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:10.950 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:51:10.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:10.956 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:51:10.974 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.10 dump_blocked_ops 2024-02-09T17:51:10.991 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:51:10.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:10 smithi099 ceph-mon[39420]: 1.c scrub starts 2024-02-09T17:51:10.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:10 smithi099 ceph-mon[39420]: 1.c scrub ok 2024-02-09T17:51:10.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:10 smithi099 ceph-mon[39420]: 1.14 scrub starts 2024-02-09T17:51:10.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:10 smithi099 ceph-mon[39420]: 1.14 scrub ok 2024-02-09T17:51:10.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:10 smithi099 ceph-mon[39420]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:51:10.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:10 smithi099 ceph-mon[39420]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:51:10.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:10 smithi099 ceph-mon[39420]: osdmap e683: 12 total, 12 up, 12 in 2024-02-09T17:51:11.057 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:51:11.086 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:10 smithi162 ceph-mon[44304]: 1.c scrub starts 2024-02-09T17:51:11.086 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:10 smithi162 ceph-mon[44304]: 1.c scrub ok 2024-02-09T17:51:11.086 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:10 smithi162 ceph-mon[44304]: 1.14 scrub starts 2024-02-09T17:51:11.086 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:10 smithi162 ceph-mon[44304]: 1.14 scrub ok 2024-02-09T17:51:11.087 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:10 smithi162 ceph-mon[44304]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:51:11.087 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:10 smithi162 ceph-mon[44304]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:51:11.087 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:10 smithi162 ceph-mon[44304]: osdmap e683: 12 total, 12 up, 12 in 2024-02-09T17:51:11.088 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:11.110 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:51:11.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:11.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:51:11.175 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:10 smithi179 ceph-mon[43975]: 1.c scrub starts 2024-02-09T17:51:11.176 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:10 smithi179 ceph-mon[43975]: 1.c scrub ok 2024-02-09T17:51:11.176 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:10 smithi179 ceph-mon[43975]: 1.14 scrub starts 2024-02-09T17:51:11.176 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:10 smithi179 ceph-mon[43975]: 1.14 scrub ok 2024-02-09T17:51:11.176 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:10 smithi179 ceph-mon[43975]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:51:11.176 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:10 smithi179 ceph-mon[43975]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:51:11.176 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:10 smithi179 ceph-mon[43975]: osdmap e683: 12 total, 12 up, 12 in 2024-02-09T17:51:11.214 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:51:11.252 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:11.283 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:51:11.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:11.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:51:11.389 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:51:11.438 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:11.462 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:51:11.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:11.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:51:11.511 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 45 16 78881 78865 109.51 8.28125 0.00827041 0.00912574 2024-02-09T17:51:11.526 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:51:11.570 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:51:11.607 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:11.828 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:11 smithi162 ceph-mon[44304]: pgmap v1716: 47 pgs: 6 remapped+peering, 41 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-02-09T17:51:11.828 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:11 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:51:11.828 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:11 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:51:11.828 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:11 smithi162 ceph-mon[44304]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:51:11.828 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:11 smithi162 ceph-mon[44304]: osdmap e684: 12 total, 12 up, 12 in 2024-02-09T17:51:11.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:11 smithi099 ceph-mon[39420]: pgmap v1716: 47 pgs: 6 remapped+peering, 41 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-02-09T17:51:11.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:11 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:51:11.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:11 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:51:11.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:11 smithi099 ceph-mon[39420]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:51:11.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:11 smithi099 ceph-mon[39420]: osdmap e684: 12 total, 12 up, 12 in 2024-02-09T17:51:12.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:11 smithi179 ceph-mon[43975]: pgmap v1716: 47 pgs: 6 remapped+peering, 41 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-02-09T17:51:12.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:11 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:51:12.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:11 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:51:12.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:11 smithi179 ceph-mon[43975]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:51:12.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:11 smithi179 ceph-mon[43975]: osdmap e684: 12 total, 12 up, 12 in 2024-02-09T17:51:12.184 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-09T17:51:12.299 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:51:12.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:12.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:51:12.406 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:51:12.463 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:12.511 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 46 16 79501 79485 107.972 38.75 0.00944745 0.00925596 2024-02-09T17:51:12.839 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:51:12.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:12.848 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:51:12.949 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:51:12.984 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:13.511 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 47 16 80055 80039 106.411 34.625 0.0224712 0.00939256 2024-02-09T17:51:13.837 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:51:13.839 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:13.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:51:13.941 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:51:13.985 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:14.019 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:51:14.025 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:14.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:51:14.077 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:13 smithi099 ceph-mon[39420]: osdmap e685: 12 total, 12 up, 12 in 2024-02-09T17:51:14.077 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:13 smithi099 ceph-mon[39420]: pgmap v1719: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 2 peering, 2 active+recovering+undersized+remapped, 6 remapped+peering, 36 active+clean; 4.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 708 op/s; 1/236499 objects degraded (0.000%); 14715/236499 objects misplaced (6.222%); 656 KiB/s, 10 objects/s recovering 2024-02-09T17:51:14.126 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:51:14.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:13 smithi179 ceph-mon[43975]: osdmap e685: 12 total, 12 up, 12 in 2024-02-09T17:51:14.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:13 smithi179 ceph-mon[43975]: pgmap v1719: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 2 peering, 2 active+recovering+undersized+remapped, 6 remapped+peering, 36 active+clean; 4.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 708 op/s; 1/236499 objects degraded (0.000%); 14715/236499 objects misplaced (6.222%); 656 KiB/s, 10 objects/s recovering 2024-02-09T17:51:14.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:13 smithi162 ceph-mon[44304]: osdmap e685: 12 total, 12 up, 12 in 2024-02-09T17:51:14.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:13 smithi162 ceph-mon[44304]: pgmap v1719: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 2 peering, 2 active+recovering+undersized+remapped, 6 remapped+peering, 36 active+clean; 4.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 708 op/s; 1/236499 objects degraded (0.000%); 14715/236499 objects misplaced (6.222%); 656 KiB/s, 10 objects/s recovering 2024-02-09T17:51:14.188 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:14.511 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 48 16 80727 80711 105.07 42 0.0152365 0.00951171 2024-02-09T17:51:14.791 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:14 smithi179 ceph-mon[43975]: Health check failed: Degraded data redundancy: 1/236499 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:51:15.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:14 smithi162 ceph-mon[44304]: Health check failed: Degraded data redundancy: 1/236499 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:51:15.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:14 smithi099 ceph-mon[39420]: Health check failed: Degraded data redundancy: 1/236499 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:51:15.511 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 49 16 81444 81428 103.84 44.8125 0.0101528 0.00962393 2024-02-09T17:51:15.856 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:51:15.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:15.863 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:51:15.965 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:51:15.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:15 smithi099 ceph-mon[39420]: pgmap v1720: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 2 peering, 3 active+recovering+undersized+remapped, 5 remapped+peering, 36 active+clean; 4.8 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 591 op/s; 1/237309 objects degraded (0.000%); 19531/237309 objects misplaced (8.230%); 1.5 MiB/s, 23 objects/s recovering 2024-02-09T17:51:16.044 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:16.080 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:51:16.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:16.090 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:51:16.090 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:15 smithi179 ceph-mon[43975]: pgmap v1720: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 2 peering, 3 active+recovering+undersized+remapped, 5 remapped+peering, 36 active+clean; 4.8 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 591 op/s; 1/237309 objects degraded (0.000%); 19531/237309 objects misplaced (8.230%); 1.5 MiB/s, 23 objects/s recovering 2024-02-09T17:51:16.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:15 smithi162 ceph-mon[44304]: pgmap v1720: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 2 peering, 3 active+recovering+undersized+remapped, 5 remapped+peering, 36 active+clean; 4.8 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 591 op/s; 1/237309 objects degraded (0.000%); 19531/237309 objects misplaced (8.230%); 1.5 MiB/s, 23 objects/s recovering 2024-02-09T17:51:16.191 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:51:16.228 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:16.487 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:51:16.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:16.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:51:16.511 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 50 15 82087 82072 102.568 40.25 0.00696141 0.00974348 2024-02-09T17:51:16.594 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:51:16.652 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:16.925 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:51:16.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:16.938 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:51:16.952 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:51:16.952 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:51:16.973 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:51:17.008 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:51:17.012 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:51:17.035 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:51:17.039 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:51:17.071 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:51:17.088 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:17.107 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:51:17.154 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:51:17.189 INFO:teuthology.orchestra.run.smithi099.stdout:moved 0 / 141 (0%) 2024-02-09T17:51:17.189 INFO:teuthology.orchestra.run.smithi099.stdout:avg 11.75 2024-02-09T17:51:17.189 INFO:teuthology.orchestra.run.smithi099.stdout:stddev 2.24072 -> 2.24072 (expected baseline 3.28189) 2024-02-09T17:51:17.189 INFO:teuthology.orchestra.run.smithi099.stdout:min osd.0 with 8 -> 8 pgs (0.680851 -> 0.680851 * mean) 2024-02-09T17:51:17.189 INFO:teuthology.orchestra.run.smithi099.stdout:max osd.7 with 15 -> 15 pgs (1.2766 -> 1.2766 * mean) 2024-02-09T17:51:17.189 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:51:17.189 INFO:teuthology.orchestra.run.smithi099.stdout:oload 1000 2024-02-09T17:51:17.189 INFO:teuthology.orchestra.run.smithi099.stdout:max_change 0.05 2024-02-09T17:51:17.189 INFO:teuthology.orchestra.run.smithi099.stdout:max_change_osds 4 2024-02-09T17:51:17.189 INFO:teuthology.orchestra.run.smithi099.stdout:average_utilization 0.0204 2024-02-09T17:51:17.190 INFO:teuthology.orchestra.run.smithi099.stdout:overload_utilization 0.2042 2024-02-09T17:51:17.190 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:51:17.208 INFO:teuthology.orchestra.run.smithi099.stderr:no change 2024-02-09T17:51:17.232 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:51:17.257 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:51:17.299 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:51:17.304 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:51:17.304 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:51:17.304 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:51:17.304 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:51:17.304 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:51:17.304 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:51:17.304 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:51:17.304 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:51:17.304 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:51:17.304 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:51:17.304 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:51:17.304 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:51:17.512 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 51 16 82716 82700 101.327 39.25 0.0117506 0.00986248 2024-02-09T17:51:17.785 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:17 smithi179 ceph-mon[43975]: osdmap e686: 12 total, 12 up, 12 in 2024-02-09T17:51:17.785 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:17 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1260444802' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:51:17.785 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:17 smithi179 ceph-mon[43975]: pgmap v1722: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 2 peering, 7 active+recovering+undersized+remapped, 37 active+clean; 5.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 579 op/s; 1/243327 objects degraded (0.000%); 38110/243327 objects misplaced (15.662%); 12 MiB/s, 187 objects/s recovering 2024-02-09T17:51:17.785 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:17 smithi179 ceph-mon[43975]: from='client.16476 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:51:17.877 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:51:17.878 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-02-09T17:51:17.878 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-02-09T17:51:17.878 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-02-09T17:51:17.878 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:51:18.152 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:51:18.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:18.158 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:51:18.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:17 smithi162 ceph-mon[44304]: osdmap e686: 12 total, 12 up, 12 in 2024-02-09T17:51:18.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:17 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1260444802' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:51:18.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:17 smithi162 ceph-mon[44304]: pgmap v1722: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 2 peering, 7 active+recovering+undersized+remapped, 37 active+clean; 5.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 579 op/s; 1/243327 objects degraded (0.000%); 38110/243327 objects misplaced (15.662%); 12 MiB/s, 187 objects/s recovering 2024-02-09T17:51:18.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:17 smithi162 ceph-mon[44304]: from='client.16476 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:51:18.237 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:17 smithi099 ceph-mon[39420]: osdmap e686: 12 total, 12 up, 12 in 2024-02-09T17:51:18.238 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:17 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1260444802' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:51:18.238 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:17 smithi099 ceph-mon[39420]: pgmap v1722: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 2 peering, 7 active+recovering+undersized+remapped, 37 active+clean; 5.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 579 op/s; 1/243327 objects degraded (0.000%); 38110/243327 objects misplaced (15.662%); 12 MiB/s, 187 objects/s recovering 2024-02-09T17:51:18.238 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:17 smithi099 ceph-mon[39420]: from='client.16476 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:51:18.258 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:51:18.300 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:18.512 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 52 16 83387 83371 100.185 41.9375 0.0312799 0.00997458 2024-02-09T17:51:19.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:19 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 1/245787 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:51:19.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:19 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 1/245787 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:51:19.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:19 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 1/245787 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:51:19.513 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 53 16 84044 84028 99.066 41.0625 0.0389375 0.0100878 2024-02-09T17:51:19.559 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:51:19.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:19.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:51:19.664 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:51:19.708 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:20.127 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.10 dump_historic_ops 2024-02-09T17:51:20.428 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:20 smithi179 ceph-mon[43975]: pgmap v1723: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s wr, 535 op/s; 1/245787 objects degraded (0.000%); 37811/245787 objects misplaced (15.384%); 12 MiB/s, 193 objects/s recovering 2024-02-09T17:51:20.428 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:20 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:51:20.428 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:20 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:51:20.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:20 smithi162 ceph-mon[44304]: pgmap v1723: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s wr, 535 op/s; 1/245787 objects degraded (0.000%); 37811/245787 objects misplaced (15.384%); 12 MiB/s, 193 objects/s recovering 2024-02-09T17:51:20.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:20 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:51:20.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:20 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:51:20.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:20 smithi099 ceph-mon[39420]: pgmap v1723: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s wr, 535 op/s; 1/245787 objects degraded (0.000%); 37811/245787 objects misplaced (15.384%); 12 MiB/s, 193 objects/s recovering 2024-02-09T17:51:20.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:20 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:51:20.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:20 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:51:20.514 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 54 16 84709 84693 98.0012 41.5625 0.0370304 0.0101976 2024-02-09T17:51:20.609 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:51:20.756 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-02-09T17:51:20.756 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd test-reweight-by-utilization 1000 1.0 2024-02-09T17:51:20.963 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:51:20.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:20.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:51:21.071 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:51:21.116 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:21.141 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:51:21.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:21.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:51:21.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:21 smithi099 ceph-mon[39420]: osdmap e687: 12 total, 12 up, 12 in 2024-02-09T17:51:21.248 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:51:21.283 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:21.419 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:51:21.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:21 smithi179 ceph-mon[43975]: osdmap e687: 12 total, 12 up, 12 in 2024-02-09T17:51:21.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:21.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:51:21.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:21 smithi162 ceph-mon[44304]: osdmap e687: 12 total, 12 up, 12 in 2024-02-09T17:51:21.514 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 55 16 85266 85250 96.8524 34.8125 0.0508447 0.0103191 2024-02-09T17:51:21.525 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:51:21.569 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:21.708 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:51:21.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:21.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:51:21.812 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:51:21.882 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:22.127 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:51:22.463 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:22 smithi099 ceph-mon[39420]: pgmap v1725: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 568 op/s; 1/250152 objects degraded (0.000%); 37127/250152 objects misplaced (14.842%); 16 MiB/s, 257 objects/s recovering 2024-02-09T17:51:22.463 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:22 smithi099 ceph-mon[39420]: osdmap e688: 12 total, 12 up, 12 in 2024-02-09T17:51:22.514 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 56 16 85906 85890 95.8372 40 0.0430181 0.0104269 2024-02-09T17:51:22.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:22 smithi179 ceph-mon[43975]: pgmap v1725: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 568 op/s; 1/250152 objects degraded (0.000%); 37127/250152 objects misplaced (14.842%); 16 MiB/s, 257 objects/s recovering 2024-02-09T17:51:22.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:22 smithi179 ceph-mon[43975]: osdmap e688: 12 total, 12 up, 12 in 2024-02-09T17:51:22.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:22 smithi162 ceph-mon[44304]: pgmap v1725: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 568 op/s; 1/250152 objects degraded (0.000%); 37127/250152 objects misplaced (14.842%); 16 MiB/s, 257 objects/s recovering 2024-02-09T17:51:22.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:22 smithi162 ceph-mon[44304]: osdmap e688: 12 total, 12 up, 12 in 2024-02-09T17:51:23.227 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is unset 2024-02-09T17:51:23.328 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:51:23.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:23.337 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:51:23.438 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:51:23.460 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:51:23.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:23 smithi099 ceph-mon[39420]: 1.8 scrub starts 2024-02-09T17:51:23.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:23 smithi099 ceph-mon[39420]: 1.8 scrub ok 2024-02-09T17:51:23.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:23 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1097618595' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:51:23.496 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:23 smithi179 ceph-mon[43975]: 1.8 scrub starts 2024-02-09T17:51:23.496 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:23 smithi179 ceph-mon[43975]: 1.8 scrub ok 2024-02-09T17:51:23.496 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:23 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1097618595' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:51:23.504 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:23.514 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 57 16 86530 86514 94.8402 39 0.0118582 0.010538 2024-02-09T17:51:23.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:23 smithi162 ceph-mon[44304]: 1.8 scrub starts 2024-02-09T17:51:23.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:23 smithi162 ceph-mon[44304]: 1.8 scrub ok 2024-02-09T17:51:23.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:23 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1097618595' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:51:23.681 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:51:23.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:23.687 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:51:23.788 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:51:23.841 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:23.892 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:51:23.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:23.899 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:51:24.000 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:51:24.039 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:24.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:24 smithi099 ceph-mon[39420]: pgmap v1727: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 43 MiB/s wr, 680 op/s; 1/254967 objects degraded (0.000%); 36587/254967 objects misplaced (14.350%); 15 MiB/s, 243 objects/s recovering 2024-02-09T17:51:24.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:24 smithi099 ceph-mon[39420]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:51:24.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:24 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1097618595' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:51:24.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:24 smithi099 ceph-mon[39420]: osdmap e689: 12 total, 12 up, 12 in 2024-02-09T17:51:24.517 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 58 15 87182 87167 93.9045 40.8125 0.039613 0.0106432 2024-02-09T17:51:24.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:24 smithi179 ceph-mon[43975]: pgmap v1727: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 43 MiB/s wr, 680 op/s; 1/254967 objects degraded (0.000%); 36587/254967 objects misplaced (14.350%); 15 MiB/s, 243 objects/s recovering 2024-02-09T17:51:24.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:24 smithi179 ceph-mon[43975]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:51:24.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:24 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1097618595' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:51:24.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:24 smithi179 ceph-mon[43975]: osdmap e689: 12 total, 12 up, 12 in 2024-02-09T17:51:24.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:24 smithi162 ceph-mon[44304]: pgmap v1727: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 43 MiB/s wr, 680 op/s; 1/254967 objects degraded (0.000%); 36587/254967 objects misplaced (14.350%); 15 MiB/s, 243 objects/s recovering 2024-02-09T17:51:24.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:24 smithi162 ceph-mon[44304]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:51:24.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:24 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1097618595' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:51:24.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:24 smithi162 ceph-mon[44304]: osdmap e689: 12 total, 12 up, 12 in 2024-02-09T17:51:25.091 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:51:25.091 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-02-09T17:51:25.091 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-02-09T17:51:25.091 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-02-09T17:51:25.091 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.091 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:125181 5.0 5:08b0c8c3:::benchmark_data_smithi179_115199_object125180:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e500)", 2024-02-09T17:51:25.091 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:42:30.010351+0000", 2024-02-09T17:51:25.091 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 535.07389311700001, 2024-02-09T17:51:25.092 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.22445062399999999, 2024-02-09T17:51:25.092 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:51:25.092 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:51:25.092 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:51:25.092 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:51:25.092 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:51:25.092 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 125181 2024-02-09T17:51:25.092 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.092 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:51:25.092 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.092 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:51:25.092 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.010351+0000", 2024-02-09T17:51:25.092 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:25.093 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.093 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.093 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:51:25.093 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.010351+0000", 2024-02-09T17:51:25.093 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:25.093 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.093 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.093 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:51:25.093 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.010353+0000", 2024-02-09T17:51:25.093 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.815e-06 2024-02-09T17:51:25.093 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.093 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.093 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:51:25.094 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.010400+0000", 2024-02-09T17:51:25.094 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7614000000000002e-05 2024-02-09T17:51:25.094 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.094 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.094 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:51:25.094 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.010413+0000", 2024-02-09T17:51:25.094 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2792999999999999e-05 2024-02-09T17:51:25.094 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.094 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.094 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:51:25.094 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.010422+0000", 2024-02-09T17:51:25.094 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4200000000000007e-06 2024-02-09T17:51:25.094 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.095 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.095 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:25.095 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.010475+0000", 2024-02-09T17:51:25.095 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3338000000000003e-05 2024-02-09T17:51:25.095 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.095 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.095 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:51:25.095 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.010623+0000", 2024-02-09T17:51:25.095 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014819900000000001 2024-02-09T17:51:25.095 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.095 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.095 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:51:25.095 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.010794+0000", 2024-02-09T17:51:25.095 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00017131700000000001 2024-02-09T17:51:25.095 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.096 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.096 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:51:25.096 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.014254+0000", 2024-02-09T17:51:25.096 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0034598329999999998 2024-02-09T17:51:25.096 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.096 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.096 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:25.096 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.016837+0000", 2024-02-09T17:51:25.096 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0025832559999999999 2024-02-09T17:51:25.096 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.096 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.096 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:25.097 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.234695+0000", 2024-02-09T17:51:25.097 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.21785784599999999 2024-02-09T17:51:25.097 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.097 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.097 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:51:25.097 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.234738+0000", 2024-02-09T17:51:25.097 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3186999999999998e-05 2024-02-09T17:51:25.097 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.097 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.097 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:51:25.097 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:30.234801+0000", 2024-02-09T17:51:25.097 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3006e-05 2024-02-09T17:51:25.097 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:25.098 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:51:25.098 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:25.098 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.098 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.098 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:168345 5.4 5:2c8dde30:::benchmark_data_smithi179_115199_object39439:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e520)", 2024-02-09T17:51:25.098 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:43:21.022404+0000", 2024-02-09T17:51:25.098 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 484.061840448, 2024-02-09T17:51:25.098 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.234700621, 2024-02-09T17:51:25.098 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:51:25.098 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:51:25.098 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:51:25.098 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:51:25.098 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:51:25.098 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 168345 2024-02-09T17:51:25.099 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.099 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:51:25.099 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.099 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:51:25.099 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:21.022404+0000", 2024-02-09T17:51:25.099 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:25.099 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.099 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.099 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:51:25.099 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:21.022404+0000", 2024-02-09T17:51:25.099 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:25.099 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.099 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.099 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:51:25.100 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:21.022405+0000", 2024-02-09T17:51:25.100 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.807e-06 2024-02-09T17:51:25.100 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.100 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.100 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:51:25.100 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:21.022411+0000", 2024-02-09T17:51:25.100 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4500000000000003e-06 2024-02-09T17:51:25.100 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.100 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.100 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:51:25.100 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:21.022426+0000", 2024-02-09T17:51:25.100 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4994000000000001e-05 2024-02-09T17:51:25.100 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.100 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.101 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:51:25.101 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:21.022434+0000", 2024-02-09T17:51:25.101 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3610000000000001e-06 2024-02-09T17:51:25.101 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.101 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.101 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:25.101 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:21.022494+0000", 2024-02-09T17:51:25.101 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0318000000000001e-05 2024-02-09T17:51:25.101 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.101 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.101 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:51:25.101 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:21.022618+0000", 2024-02-09T17:51:25.101 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012379399999999999 2024-02-09T17:51:25.101 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.102 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.102 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,8", 2024-02-09T17:51:25.102 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:21.022742+0000", 2024-02-09T17:51:25.102 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000123666 2024-02-09T17:51:25.102 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.102 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.102 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:51:25.102 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:21.026963+0000", 2024-02-09T17:51:25.102 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0042212309999999998 2024-02-09T17:51:25.102 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.102 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.102 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:25.102 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:21.028498+0000", 2024-02-09T17:51:25.102 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001534948 2024-02-09T17:51:25.103 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.103 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.103 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:25.103 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:21.257025+0000", 2024-02-09T17:51:25.103 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.22852671499999999 2024-02-09T17:51:25.103 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.103 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.103 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:51:25.103 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:21.257060+0000", 2024-02-09T17:51:25.103 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.5352000000000003e-05 2024-02-09T17:51:25.103 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.103 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.103 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:51:25.104 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:21.257104+0000", 2024-02-09T17:51:25.104 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3985e-05 2024-02-09T17:51:25.104 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:25.104 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:51:25.104 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:25.104 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.104 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.104 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:170476 5.4 5:2c84d8c7:::benchmark_data_smithi179_115199_object41570:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e521)", 2024-02-09T17:51:25.104 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:43:24.162418+0000", 2024-02-09T17:51:25.104 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 480.92182615500002, 2024-02-09T17:51:25.104 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7630616310000002, 2024-02-09T17:51:25.104 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:51:25.104 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:51:25.104 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:51:25.105 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:51:25.105 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:51:25.105 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 170476 2024-02-09T17:51:25.105 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.105 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:51:25.105 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.105 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:51:25.105 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.162418+0000", 2024-02-09T17:51:25.105 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:25.105 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.105 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.105 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:51:25.105 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.162418+0000", 2024-02-09T17:51:25.106 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:25.106 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.106 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.106 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:51:25.106 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.162420+0000", 2024-02-09T17:51:25.106 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7749999999999999e-06 2024-02-09T17:51:25.106 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.106 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.107 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:51:25.107 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.162426+0000", 2024-02-09T17:51:25.107 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2129999999999996e-06 2024-02-09T17:51:25.107 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.107 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.107 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:51:25.107 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.162438+0000", 2024-02-09T17:51:25.107 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2174999999999999e-05 2024-02-09T17:51:25.107 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.108 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.108 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:51:25.108 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.162447+0000", 2024-02-09T17:51:25.108 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7679999999999997e-06 2024-02-09T17:51:25.108 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.108 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.108 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:25.108 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.165709+0000", 2024-02-09T17:51:25.108 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00326202 2024-02-09T17:51:25.108 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.108 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.108 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:51:25.108 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.165721+0000", 2024-02-09T17:51:25.108 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1915e-05 2024-02-09T17:51:25.109 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.109 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.109 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:25.109 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:25.160715+0000", 2024-02-09T17:51:25.109 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.99499405299999999 2024-02-09T17:51:25.109 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.109 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.109 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:51:25.109 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:25.160727+0000", 2024-02-09T17:51:25.109 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2225999999999999e-05 2024-02-09T17:51:25.109 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.109 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.109 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:25.110 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.909159+0000", 2024-02-09T17:51:25.110 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7484321459999999 2024-02-09T17:51:25.110 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.110 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.110 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:51:25.110 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.909302+0000", 2024-02-09T17:51:25.111 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000142402 2024-02-09T17:51:25.111 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.111 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.111 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,5,8", 2024-02-09T17:51:25.111 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.909397+0000", 2024-02-09T17:51:25.111 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5115e-05 2024-02-09T17:51:25.111 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.111 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.111 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:25.111 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.915292+0000", 2024-02-09T17:51:25.111 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0058958639999999998 2024-02-09T17:51:25.111 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.111 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.112 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:51:25.112 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.920288+0000", 2024-02-09T17:51:25.112 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0049951830000000003 2024-02-09T17:51:25.112 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.112 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.112 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:25.112 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.925296+0000", 2024-02-09T17:51:25.112 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0050083530000000001 2024-02-09T17:51:25.112 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.112 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.112 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:25.113 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.925364+0000", 2024-02-09T17:51:25.113 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8219000000000005e-05 2024-02-09T17:51:25.113 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.113 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.113 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:51:25.113 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.925409+0000", 2024-02-09T17:51:25.114 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4731000000000003e-05 2024-02-09T17:51:25.114 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.114 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.114 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:51:25.114 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.925479+0000", 2024-02-09T17:51:25.114 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.0473000000000001e-05 2024-02-09T17:51:25.115 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:25.115 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:51:25.115 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:25.115 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.115 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.115 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:170500 5.4 5:217860fd:::benchmark_data_smithi179_115199_object41594:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e521)", 2024-02-09T17:51:25.116 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:43:24.199462+0000", 2024-02-09T17:51:25.116 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 480.88478190799998, 2024-02-09T17:51:25.116 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7363271839999999, 2024-02-09T17:51:25.116 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:51:25.116 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:51:25.117 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:51:25.117 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:51:25.117 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:51:25.117 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 170500 2024-02-09T17:51:25.117 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.118 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:51:25.118 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.118 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:51:25.118 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.199462+0000", 2024-02-09T17:51:25.118 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:25.118 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.119 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.119 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:51:25.119 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.199462+0000", 2024-02-09T17:51:25.119 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:25.119 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.119 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.120 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:51:25.120 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.199464+0000", 2024-02-09T17:51:25.120 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9190000000000002e-06 2024-02-09T17:51:25.120 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.120 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.120 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:51:25.121 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.199472+0000", 2024-02-09T17:51:25.121 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2460000000000003e-06 2024-02-09T17:51:25.121 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.121 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.121 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:51:25.121 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.199491+0000", 2024-02-09T17:51:25.121 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8654000000000001e-05 2024-02-09T17:51:25.121 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.121 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.122 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:51:25.122 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.199502+0000", 2024-02-09T17:51:25.122 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0801e-05 2024-02-09T17:51:25.122 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.122 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.122 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:25.122 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.199584+0000", 2024-02-09T17:51:25.122 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2402000000000002e-05 2024-02-09T17:51:25.122 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.122 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.122 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:51:25.123 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.199596+0000", 2024-02-09T17:51:25.123 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1429e-05 2024-02-09T17:51:25.123 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.123 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.123 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:25.123 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:25.160752+0000", 2024-02-09T17:51:25.123 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.96115640800000002 2024-02-09T17:51:25.123 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.124 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.124 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:51:25.124 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:25.160760+0000", 2024-02-09T17:51:25.124 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5939999999999996e-06 2024-02-09T17:51:25.124 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.124 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.125 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:25.125 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.915360+0000", 2024-02-09T17:51:25.125 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.754599998 2024-02-09T17:51:25.125 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.125 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.125 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:51:25.126 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.915514+0000", 2024-02-09T17:51:25.126 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015449800000000001 2024-02-09T17:51:25.126 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.126 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.126 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,5,8", 2024-02-09T17:51:25.126 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.915676+0000", 2024-02-09T17:51:25.126 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016166600000000001 2024-02-09T17:51:25.127 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.127 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.127 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:25.127 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.925536+0000", 2024-02-09T17:51:25.127 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0098606619999999992 2024-02-09T17:51:25.127 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.128 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.128 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:51:25.128 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.925548+0000", 2024-02-09T17:51:25.128 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2038000000000001e-05 2024-02-09T17:51:25.128 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.128 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.128 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:25.129 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.925636+0000", 2024-02-09T17:51:25.129 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7386999999999995e-05 2024-02-09T17:51:25.129 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.129 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.129 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:25.129 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.935621+0000", 2024-02-09T17:51:25.130 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0099851450000000008 2024-02-09T17:51:25.130 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.130 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.130 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:51:25.130 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.935679+0000", 2024-02-09T17:51:25.130 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7735000000000002e-05 2024-02-09T17:51:25.130 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.130 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.130 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:51:25.130 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.935789+0000", 2024-02-09T17:51:25.130 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000110602 2024-02-09T17:51:25.130 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:25.131 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:51:25.131 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:25.131 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.131 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.131 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:170510 5.4 5:2bc37255:::benchmark_data_smithi179_115199_object41604:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e521)", 2024-02-09T17:51:25.131 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:43:24.214260+0000", 2024-02-09T17:51:25.131 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 480.86998382899998, 2024-02-09T17:51:25.131 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.72188294, 2024-02-09T17:51:25.131 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:51:25.131 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:51:25.131 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:51:25.131 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:51:25.131 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:51:25.131 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 170510 2024-02-09T17:51:25.132 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.132 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:51:25.132 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.132 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:51:25.132 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.214260+0000", 2024-02-09T17:51:25.132 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:25.132 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.132 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.132 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:51:25.132 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.214260+0000", 2024-02-09T17:51:25.132 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:25.132 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.132 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.132 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:51:25.133 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.214262+0000", 2024-02-09T17:51:25.133 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.435e-06 2024-02-09T17:51:25.133 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.133 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.133 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:51:25.133 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.214269+0000", 2024-02-09T17:51:25.133 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1300000000000003e-06 2024-02-09T17:51:25.133 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.133 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.133 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:51:25.133 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.214284+0000", 2024-02-09T17:51:25.133 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4912000000000001e-05 2024-02-09T17:51:25.133 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.134 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.134 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:51:25.134 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.214293+0000", 2024-02-09T17:51:25.134 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.3530000000000001e-06 2024-02-09T17:51:25.134 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.134 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.134 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:25.134 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.214389+0000", 2024-02-09T17:51:25.134 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5817999999999996e-05 2024-02-09T17:51:25.134 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.134 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.134 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:51:25.134 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.214402+0000", 2024-02-09T17:51:25.134 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3046e-05 2024-02-09T17:51:25.135 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.135 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.135 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:25.135 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:25.160781+0000", 2024-02-09T17:51:25.135 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.94637950900000001 2024-02-09T17:51:25.135 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.135 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.135 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:51:25.135 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:25.160789+0000", 2024-02-09T17:51:25.135 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.9699999999999999e-06 2024-02-09T17:51:25.135 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.135 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.135 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:25.135 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.920342+0000", 2024-02-09T17:51:25.136 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7595531690000001 2024-02-09T17:51:25.136 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.136 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.136 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:51:25.136 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.920496+0000", 2024-02-09T17:51:25.136 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000153356 2024-02-09T17:51:25.136 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.136 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.136 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,5,8", 2024-02-09T17:51:25.136 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.920654+0000", 2024-02-09T17:51:25.136 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015795499999999999 2024-02-09T17:51:25.136 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.136 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.136 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:51:25.137 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.925556+0000", 2024-02-09T17:51:25.137 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0049018250000000003 2024-02-09T17:51:25.137 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.137 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.137 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:25.137 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.925591+0000", 2024-02-09T17:51:25.137 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.5104999999999997e-05 2024-02-09T17:51:25.137 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.137 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.137 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:25.137 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.935892+0000", 2024-02-09T17:51:25.137 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010301161 2024-02-09T17:51:25.137 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.137 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.137 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:25.137 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.935982+0000", 2024-02-09T17:51:25.138 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.9733000000000003e-05 2024-02-09T17:51:25.138 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.138 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.138 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:51:25.138 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.936034+0000", 2024-02-09T17:51:25.138 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2454000000000003e-05 2024-02-09T17:51:25.138 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.138 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.138 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:51:25.138 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.936143+0000", 2024-02-09T17:51:25.138 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000109009 2024-02-09T17:51:25.138 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:25.138 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:51:25.138 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:25.138 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.139 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.139 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:170515 5.4 5:29030551:::benchmark_data_smithi179_115199_object41609:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e521)", 2024-02-09T17:51:25.139 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:43:24.221521+0000", 2024-02-09T17:51:25.139 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 480.86272321199999, 2024-02-09T17:51:25.139 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7230485099999999, 2024-02-09T17:51:25.139 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:51:25.139 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:51:25.139 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:51:25.139 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:51:25.139 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:51:25.140 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 170515 2024-02-09T17:51:25.140 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.140 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:51:25.140 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.140 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:51:25.140 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.221521+0000", 2024-02-09T17:51:25.140 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:25.140 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.140 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.140 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:51:25.140 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.221521+0000", 2024-02-09T17:51:25.140 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:25.140 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.141 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.141 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:51:25.141 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.221522+0000", 2024-02-09T17:51:25.141 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5090000000000001e-06 2024-02-09T17:51:25.141 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.141 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.141 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:51:25.141 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.221528+0000", 2024-02-09T17:51:25.141 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1009999999999997e-06 2024-02-09T17:51:25.141 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.141 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.141 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:51:25.141 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.221542+0000", 2024-02-09T17:51:25.141 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3129000000000001e-05 2024-02-09T17:51:25.142 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.142 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.142 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:51:25.142 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.221550+0000", 2024-02-09T17:51:25.142 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3820000000000004e-06 2024-02-09T17:51:25.142 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.142 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.142 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:25.142 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.221609+0000", 2024-02-09T17:51:25.142 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9543000000000002e-05 2024-02-09T17:51:25.142 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.142 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.142 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:51:25.142 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.221618+0000", 2024-02-09T17:51:25.143 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.6119999999999999e-06 2024-02-09T17:51:25.143 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.143 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.143 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:25.143 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:25.160812+0000", 2024-02-09T17:51:25.143 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.93919381700000004 2024-02-09T17:51:25.143 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.143 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.143 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:51:25.143 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:25.160820+0000", 2024-02-09T17:51:25.143 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8329999999999994e-06 2024-02-09T17:51:25.143 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.143 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.143 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:25.143 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.925769+0000", 2024-02-09T17:51:25.144 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.764948945 2024-02-09T17:51:25.144 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.144 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.144 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:51:25.144 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.925887+0000", 2024-02-09T17:51:25.144 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000118733 2024-02-09T17:51:25.144 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.144 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.144 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,5,8", 2024-02-09T17:51:25.144 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.925986+0000", 2024-02-09T17:51:25.144 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.8902999999999998e-05 2024-02-09T17:51:25.144 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.144 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.144 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:25.144 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.935841+0000", 2024-02-09T17:51:25.145 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0098549109999999992 2024-02-09T17:51:25.145 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.145 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.145 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:51:25.145 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.935939+0000", 2024-02-09T17:51:25.145 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7590000000000006e-05 2024-02-09T17:51:25.145 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.145 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.145 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:25.145 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.936183+0000", 2024-02-09T17:51:25.145 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00024420900000000001 2024-02-09T17:51:25.145 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.145 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.145 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:25.145 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.944452+0000", 2024-02-09T17:51:25.145 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0082693750000000007 2024-02-09T17:51:25.146 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.146 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.146 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:51:25.146 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.944489+0000", 2024-02-09T17:51:25.146 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6965000000000003e-05 2024-02-09T17:51:25.146 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.146 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.146 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:51:25.146 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.944569+0000", 2024-02-09T17:51:25.146 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.9952999999999998e-05 2024-02-09T17:51:25.146 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:25.146 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:51:25.147 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:25.147 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.147 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.147 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:170516 5.4 5:29803107:::benchmark_data_smithi179_115199_object41610:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e521)", 2024-02-09T17:51:25.147 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:43:24.225937+0000", 2024-02-09T17:51:25.147 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 480.85830719299997, 2024-02-09T17:51:25.147 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.736096597, 2024-02-09T17:51:25.147 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:51:25.147 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:51:25.147 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:51:25.147 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:51:25.147 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:51:25.147 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 170516 2024-02-09T17:51:25.147 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.148 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:51:25.148 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.148 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:51:25.148 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.225937+0000", 2024-02-09T17:51:25.148 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:25.148 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.148 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.148 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:51:25.148 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.225937+0000", 2024-02-09T17:51:25.148 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:25.148 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.148 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.148 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:51:25.149 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.225938+0000", 2024-02-09T17:51:25.149 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.091e-06 2024-02-09T17:51:25.149 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.149 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.149 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:51:25.149 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.225944+0000", 2024-02-09T17:51:25.149 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.728e-06 2024-02-09T17:51:25.149 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.149 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.149 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:51:25.149 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.225958+0000", 2024-02-09T17:51:25.149 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4803e-05 2024-02-09T17:51:25.149 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.149 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.150 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:51:25.150 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.225967+0000", 2024-02-09T17:51:25.150 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8119999999999997e-06 2024-02-09T17:51:25.150 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.150 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.150 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:25.150 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.226032+0000", 2024-02-09T17:51:25.150 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4403e-05 2024-02-09T17:51:25.150 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.150 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.150 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:51:25.150 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.226044+0000", 2024-02-09T17:51:25.150 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.255e-05 2024-02-09T17:51:25.151 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.151 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.151 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:25.151 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:25.160844+0000", 2024-02-09T17:51:25.151 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.93479992499999998 2024-02-09T17:51:25.151 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.151 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.151 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:51:25.151 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:25.160849+0000", 2024-02-09T17:51:25.151 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0719999999999999e-06 2024-02-09T17:51:25.151 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.151 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.151 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:25.152 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.930109+0000", 2024-02-09T17:51:25.152 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7692595610000001 2024-02-09T17:51:25.152 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.152 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.152 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:51:25.152 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.930238+0000", 2024-02-09T17:51:25.152 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012917899999999999 2024-02-09T17:51:25.152 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.152 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.152 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,5,8", 2024-02-09T17:51:25.152 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.930365+0000", 2024-02-09T17:51:25.152 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012663 2024-02-09T17:51:25.153 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.153 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.153 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:25.153 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.935930+0000", 2024-02-09T17:51:25.153 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0055652369999999998 2024-02-09T17:51:25.153 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.153 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.153 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:51:25.153 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.935943+0000", 2024-02-09T17:51:25.153 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3335e-05 2024-02-09T17:51:25.153 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.153 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.154 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:25.154 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.936233+0000", 2024-02-09T17:51:25.154 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00028995900000000001 2024-02-09T17:51:25.154 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.154 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.154 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:25.154 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.961920+0000", 2024-02-09T17:51:25.154 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.025686499000000002 2024-02-09T17:51:25.154 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.154 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.154 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:51:25.154 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.961960+0000", 2024-02-09T17:51:25.154 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0123999999999997e-05 2024-02-09T17:51:25.155 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.155 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.155 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:51:25.155 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.962033+0000", 2024-02-09T17:51:25.155 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.3689000000000005e-05 2024-02-09T17:51:25.155 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:25.155 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:51:25.155 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:25.155 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.155 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.155 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:170544 5.4 5:2e189078:::benchmark_data_smithi179_115199_object41638:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e521)", 2024-02-09T17:51:25.155 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:43:24.335450+0000", 2024-02-09T17:51:25.156 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 480.74879401599998, 2024-02-09T17:51:25.156 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6268050789999999, 2024-02-09T17:51:25.156 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:51:25.156 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:51:25.156 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:51:25.156 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:51:25.156 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:51:25.156 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 170544 2024-02-09T17:51:25.156 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.156 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:51:25.156 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.156 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:51:25.157 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.335450+0000", 2024-02-09T17:51:25.157 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:25.157 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.157 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.157 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:51:25.157 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.335450+0000", 2024-02-09T17:51:25.157 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:25.157 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.157 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.157 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:51:25.157 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.335452+0000", 2024-02-09T17:51:25.157 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2689999999999998e-06 2024-02-09T17:51:25.157 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.158 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.158 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:51:25.158 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.335459+0000", 2024-02-09T17:51:25.158 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6139999999999997e-06 2024-02-09T17:51:25.158 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.158 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.158 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:51:25.158 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.335474+0000", 2024-02-09T17:51:25.158 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5586e-05 2024-02-09T17:51:25.158 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.158 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.158 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:51:25.159 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.335485+0000", 2024-02-09T17:51:25.159 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0403e-05 2024-02-09T17:51:25.159 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.159 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.159 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:25.159 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.335550+0000", 2024-02-09T17:51:25.159 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5500000000000006e-05 2024-02-09T17:51:25.159 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.159 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.159 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:51:25.159 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.335560+0000", 2024-02-09T17:51:25.160 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.91e-06 2024-02-09T17:51:25.160 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.160 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.160 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:25.160 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:25.160862+0000", 2024-02-09T17:51:25.160 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.82530139599999996 2024-02-09T17:51:25.160 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.160 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.160 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:51:25.160 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:25.160866+0000", 2024-02-09T17:51:25.160 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.318e-06 2024-02-09T17:51:25.160 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.161 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.161 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:25.161 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.936265+0000", 2024-02-09T17:51:25.161 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7753993299999999 2024-02-09T17:51:25.161 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.161 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.161 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:51:25.161 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.936346+0000", 2024-02-09T17:51:25.161 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.0321000000000006e-05 2024-02-09T17:51:25.161 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.161 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.161 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,5,8", 2024-02-09T17:51:25.161 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.936430+0000", 2024-02-09T17:51:25.162 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4352999999999997e-05 2024-02-09T17:51:25.162 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.162 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.162 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:25.162 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.944614+0000", 2024-02-09T17:51:25.162 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0081839860000000007 2024-02-09T17:51:25.162 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.162 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.162 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:25.162 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.944684+0000", 2024-02-09T17:51:25.162 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.0196000000000004e-05 2024-02-09T17:51:25.162 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.162 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.163 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:51:25.163 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.944693+0000", 2024-02-09T17:51:25.163 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5890000000000003e-06 2024-02-09T17:51:25.163 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.163 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.163 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:25.163 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.962081+0000", 2024-02-09T17:51:25.163 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017388763000000002 2024-02-09T17:51:25.163 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.163 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.163 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:51:25.163 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.962146+0000", 2024-02-09T17:51:25.163 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4245999999999994e-05 2024-02-09T17:51:25.164 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.164 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.164 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:51:25.164 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.962255+0000", 2024-02-09T17:51:25.164 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000109299 2024-02-09T17:51:25.164 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:25.164 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:51:25.164 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:25.164 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.164 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.164 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:170545 5.4 5:2d50f37a:::benchmark_data_smithi179_115199_object41639:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e521)", 2024-02-09T17:51:25.164 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:43:24.340050+0000", 2024-02-09T17:51:25.164 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 480.74419356599998, 2024-02-09T17:51:25.165 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.622335708, 2024-02-09T17:51:25.165 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:51:25.165 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:51:25.165 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:51:25.165 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:51:25.165 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:51:25.165 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 170545 2024-02-09T17:51:25.165 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.165 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:51:25.165 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.165 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:51:25.165 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.340050+0000", 2024-02-09T17:51:25.165 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:25.166 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.166 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.166 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:51:25.166 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.340050+0000", 2024-02-09T17:51:25.166 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:25.166 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.166 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.166 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:51:25.166 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.340052+0000", 2024-02-09T17:51:25.166 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4750000000000001e-06 2024-02-09T17:51:25.166 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.166 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.166 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:51:25.166 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.340058+0000", 2024-02-09T17:51:25.167 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.852e-06 2024-02-09T17:51:25.167 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.167 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.167 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:51:25.167 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.340071+0000", 2024-02-09T17:51:25.167 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3512999999999999e-05 2024-02-09T17:51:25.167 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.167 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.167 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:51:25.167 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.340080+0000", 2024-02-09T17:51:25.167 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2269999999999999e-06 2024-02-09T17:51:25.167 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.167 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.167 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:25.168 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.340146+0000", 2024-02-09T17:51:25.168 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5246000000000005e-05 2024-02-09T17:51:25.168 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.168 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.168 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:51:25.168 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.340156+0000", 2024-02-09T17:51:25.168 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0287e-05 2024-02-09T17:51:25.168 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.168 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.168 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:25.168 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:25.160878+0000", 2024-02-09T17:51:25.168 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.82072214799999998 2024-02-09T17:51:25.169 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.169 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.169 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:51:25.169 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:25.160882+0000", 2024-02-09T17:51:25.169 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2379999999999999e-06 2024-02-09T17:51:25.169 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.169 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.169 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:25.169 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.940430+0000", 2024-02-09T17:51:25.169 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.779547695 2024-02-09T17:51:25.169 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.169 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.169 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:51:25.170 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.940515+0000", 2024-02-09T17:51:25.170 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5368e-05 2024-02-09T17:51:25.170 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.170 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.170 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,5,8", 2024-02-09T17:51:25.170 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.940596+0000", 2024-02-09T17:51:25.170 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.0252000000000003e-05 2024-02-09T17:51:25.170 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.170 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.170 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:25.170 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.944650+0000", 2024-02-09T17:51:25.170 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0040546660000000002 2024-02-09T17:51:25.170 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.170 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.171 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:51:25.171 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.944697+0000", 2024-02-09T17:51:25.171 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6694999999999999e-05 2024-02-09T17:51:25.171 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.171 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.171 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:25.171 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.961866+0000", 2024-02-09T17:51:25.171 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017169336 2024-02-09T17:51:25.171 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.171 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.171 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:25.171 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.962299+0000", 2024-02-09T17:51:25.171 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00043296799999999999 2024-02-09T17:51:25.171 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.172 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.172 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:51:25.172 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.962329+0000", 2024-02-09T17:51:25.172 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.9382000000000001e-05 2024-02-09T17:51:25.172 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.172 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.172 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:51:25.172 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.962386+0000", 2024-02-09T17:51:25.172 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.736e-05 2024-02-09T17:51:25.172 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:25.172 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:51:25.172 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:25.172 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.172 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.173 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:199899 5.4 5:2e91c8e1:::benchmark_data_smithi179_115199_object70993:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e542)", 2024-02-09T17:51:25.173 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:44:05.980287+0000", 2024-02-09T17:51:25.173 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 439.10395694900001, 2024-02-09T17:51:25.173 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.248743991, 2024-02-09T17:51:25.173 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:51:25.173 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:51:25.173 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:51:25.173 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:51:25.173 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:51:25.173 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 199899 2024-02-09T17:51:25.173 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.173 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:51:25.173 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.173 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:51:25.174 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:05.980287+0000", 2024-02-09T17:51:25.174 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:25.174 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.174 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.174 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:51:25.174 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:05.980287+0000", 2024-02-09T17:51:25.174 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:25.174 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.174 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.174 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:51:25.174 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:05.980289+0000", 2024-02-09T17:51:25.174 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4810000000000001e-06 2024-02-09T17:51:25.174 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.174 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.174 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:51:25.175 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:05.980294+0000", 2024-02-09T17:51:25.175 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6529999999999999e-06 2024-02-09T17:51:25.175 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.175 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.175 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:51:25.175 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:05.980310+0000", 2024-02-09T17:51:25.175 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5648999999999999e-05 2024-02-09T17:51:25.175 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.175 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.175 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:51:25.175 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:05.980318+0000", 2024-02-09T17:51:25.175 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8550000000000002e-06 2024-02-09T17:51:25.175 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.175 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.176 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:25.176 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:06.050760+0000", 2024-02-09T17:51:25.176 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.070441846000000002 2024-02-09T17:51:25.176 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.176 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.176 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:51:25.176 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:06.050885+0000", 2024-02-09T17:51:25.176 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000125477 2024-02-09T17:51:25.176 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.176 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.176 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,5", 2024-02-09T17:51:25.176 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:06.051012+0000", 2024-02-09T17:51:25.176 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012747799999999999 2024-02-09T17:51:25.177 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.177 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.177 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:51:25.177 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:06.055662+0000", 2024-02-09T17:51:25.177 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0046493239999999998 2024-02-09T17:51:25.177 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.177 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.177 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:25.177 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:06.069944+0000", 2024-02-09T17:51:25.177 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014282398999999999 2024-02-09T17:51:25.177 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.177 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.178 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:25.178 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:06.228930+0000", 2024-02-09T17:51:25.178 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15898531399999999 2024-02-09T17:51:25.178 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.178 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.178 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:51:25.178 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:06.228982+0000", 2024-02-09T17:51:25.178 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2299000000000002e-05 2024-02-09T17:51:25.178 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.178 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.178 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:51:25.178 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:06.229031+0000", 2024-02-09T17:51:25.178 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9215999999999998e-05 2024-02-09T17:51:25.178 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:25.179 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:51:25.179 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:25.179 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.179 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.179 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:199919 5.4 5:28d692dd:::benchmark_data_smithi179_115199_object71013:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e542)", 2024-02-09T17:51:25.179 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:44:06.007830+0000", 2024-02-09T17:51:25.179 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 439.07641395799999, 2024-02-09T17:51:25.179 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.22133669, 2024-02-09T17:51:25.179 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:51:25.179 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:51:25.179 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:51:25.179 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:51:25.179 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:51:25.179 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 199919 2024-02-09T17:51:25.180 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.180 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:51:25.180 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.180 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:51:25.180 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:06.007830+0000", 2024-02-09T17:51:25.180 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:25.180 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.180 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.180 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:51:25.180 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:06.007830+0000", 2024-02-09T17:51:25.180 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:25.180 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.180 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.180 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:51:25.180 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:06.007831+0000", 2024-02-09T17:51:25.181 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.421e-06 2024-02-09T17:51:25.181 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.181 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.181 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:51:25.181 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:06.007837+0000", 2024-02-09T17:51:25.181 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0810000000000001e-06 2024-02-09T17:51:25.181 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.181 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.181 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:51:25.181 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:06.007851+0000", 2024-02-09T17:51:25.181 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4141999999999999e-05 2024-02-09T17:51:25.181 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.181 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.181 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:51:25.182 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:06.007859+0000", 2024-02-09T17:51:25.182 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8380000000000002e-06 2024-02-09T17:51:25.182 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.182 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.182 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:25.182 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:06.056088+0000", 2024-02-09T17:51:25.182 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.048229279999999999 2024-02-09T17:51:25.182 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.182 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.182 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:51:25.182 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:06.056176+0000", 2024-02-09T17:51:25.182 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8040000000000004e-05 2024-02-09T17:51:25.182 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.182 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.182 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,5", 2024-02-09T17:51:25.183 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:06.056247+0000", 2024-02-09T17:51:25.183 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1163999999999998e-05 2024-02-09T17:51:25.183 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.183 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.183 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:51:25.183 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:06.065741+0000", 2024-02-09T17:51:25.183 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0094945010000000007 2024-02-09T17:51:25.183 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.183 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.183 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:25.183 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:06.069982+0000", 2024-02-09T17:51:25.183 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0042408530000000002 2024-02-09T17:51:25.183 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.183 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.183 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:25.184 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:06.229068+0000", 2024-02-09T17:51:25.184 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15908560499999999 2024-02-09T17:51:25.184 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.184 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.184 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:51:25.184 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:06.229114+0000", 2024-02-09T17:51:25.184 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5649000000000003e-05 2024-02-09T17:51:25.184 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.184 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.184 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:51:25.184 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:06.229167+0000", 2024-02-09T17:51:25.184 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3115999999999998e-05 2024-02-09T17:51:25.184 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:25.184 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:51:25.184 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:25.185 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.185 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.185 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:202669 5.4 5:26ebc188:::benchmark_data_smithi179_115199_object73763:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e544)", 2024-02-09T17:51:25.185 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:44:09.667589+0000", 2024-02-09T17:51:25.185 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 435.41665527800001, 2024-02-09T17:51:25.185 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.24079398199999999, 2024-02-09T17:51:25.185 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:51:25.185 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:51:25.185 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:51:25.185 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:51:25.185 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:51:25.185 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 202669 2024-02-09T17:51:25.185 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.185 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:51:25.186 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.186 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:51:25.186 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:09.667589+0000", 2024-02-09T17:51:25.186 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:25.186 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.186 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.186 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:51:25.186 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:09.667589+0000", 2024-02-09T17:51:25.186 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:25.186 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.186 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.186 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:51:25.186 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:09.667590+0000", 2024-02-09T17:51:25.186 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7090000000000001e-06 2024-02-09T17:51:25.187 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.187 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.187 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:51:25.187 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:09.667597+0000", 2024-02-09T17:51:25.187 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2299999999999996e-06 2024-02-09T17:51:25.187 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.187 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.187 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:51:25.187 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:09.667611+0000", 2024-02-09T17:51:25.187 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4448e-05 2024-02-09T17:51:25.187 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.187 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.187 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:51:25.187 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:09.667620+0000", 2024-02-09T17:51:25.187 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.9500000000000007e-06 2024-02-09T17:51:25.188 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.188 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.188 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:25.188 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:09.736485+0000", 2024-02-09T17:51:25.188 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.068865169000000004 2024-02-09T17:51:25.188 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.188 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.188 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:51:25.188 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:09.736594+0000", 2024-02-09T17:51:25.188 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000108873 2024-02-09T17:51:25.188 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.188 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.188 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,5", 2024-02-09T17:51:25.188 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:09.736698+0000", 2024-02-09T17:51:25.188 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010390700000000001 2024-02-09T17:51:25.189 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.189 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.189 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:25.189 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:09.741985+0000", 2024-02-09T17:51:25.189 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0052871100000000002 2024-02-09T17:51:25.189 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.189 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.189 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:51:25.189 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:09.748677+0000", 2024-02-09T17:51:25.189 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0066922689999999998 2024-02-09T17:51:25.189 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.189 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.189 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:25.190 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:09.908262+0000", 2024-02-09T17:51:25.190 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15958415200000001 2024-02-09T17:51:25.190 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.190 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.190 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:51:25.190 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:09.908324+0000", 2024-02-09T17:51:25.190 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2130000000000003e-05 2024-02-09T17:51:25.190 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.190 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.190 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:51:25.190 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:09.908383+0000", 2024-02-09T17:51:25.190 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9035e-05 2024-02-09T17:51:25.190 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:25.190 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:51:25.190 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:25.191 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.191 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.191 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:202677 5.4 5:29dbf231:::benchmark_data_smithi179_115199_object73771:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e544)", 2024-02-09T17:51:25.191 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:44:09.677962+0000", 2024-02-09T17:51:25.191 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 435.406282412, 2024-02-09T17:51:25.191 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.230588665, 2024-02-09T17:51:25.191 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:51:25.191 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:51:25.191 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:51:25.191 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:51:25.191 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:51:25.191 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 202677 2024-02-09T17:51:25.191 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.191 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:51:25.192 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.192 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:51:25.192 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:09.677962+0000", 2024-02-09T17:51:25.192 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:25.192 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.192 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.192 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:51:25.192 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:09.677962+0000", 2024-02-09T17:51:25.192 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:25.192 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.192 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.192 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:51:25.192 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:09.677963+0000", 2024-02-09T17:51:25.193 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6589999999999999e-06 2024-02-09T17:51:25.193 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.193 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.193 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:51:25.193 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:09.677969+0000", 2024-02-09T17:51:25.193 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.122e-06 2024-02-09T17:51:25.193 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.193 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.193 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:51:25.193 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:09.677983+0000", 2024-02-09T17:51:25.193 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3658999999999999e-05 2024-02-09T17:51:25.193 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.193 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.193 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:51:25.194 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:09.677991+0000", 2024-02-09T17:51:25.194 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4479999999999994e-06 2024-02-09T17:51:25.194 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.194 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.194 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:25.194 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:09.744180+0000", 2024-02-09T17:51:25.194 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.066188962000000004 2024-02-09T17:51:25.194 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.194 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.194 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:51:25.194 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:09.744304+0000", 2024-02-09T17:51:25.194 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012361499999999999 2024-02-09T17:51:25.194 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.195 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.195 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,5", 2024-02-09T17:51:25.195 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:09.744430+0000", 2024-02-09T17:51:25.195 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000126074 2024-02-09T17:51:25.195 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.195 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.195 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:51:25.195 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:09.748942+0000", 2024-02-09T17:51:25.195 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0045120949999999998 2024-02-09T17:51:25.195 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.195 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.195 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:25.195 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:09.749316+0000", 2024-02-09T17:51:25.195 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000373987 2024-02-09T17:51:25.196 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.196 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.196 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:25.196 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:09.908442+0000", 2024-02-09T17:51:25.196 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15912567999999999 2024-02-09T17:51:25.196 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.196 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.196 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:51:25.196 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:09.908497+0000", 2024-02-09T17:51:25.196 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5225000000000001e-05 2024-02-09T17:51:25.196 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.196 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.196 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:51:25.197 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:09.908550+0000", 2024-02-09T17:51:25.197 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3139000000000001e-05 2024-02-09T17:51:25.197 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:25.197 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:51:25.197 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:25.197 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.197 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.197 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25837.0:202623 6.f 6:fc8a4001:::benchmark_data_smithi179_126206_object72242:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e638)", 2024-02-09T17:51:25.197 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:48:47.488364+0000", 2024-02-09T17:51:25.197 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 157.595879701, 2024-02-09T17:51:25.197 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.092512484000000006, 2024-02-09T17:51:25.197 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:51:25.197 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:51:25.197 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:51:25.197 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25837", 2024-02-09T17:51:25.197 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:51:25.198 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 202623 2024-02-09T17:51:25.198 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.198 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:51:25.198 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.198 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:51:25.198 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.488364+0000", 2024-02-09T17:51:25.198 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:25.198 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.198 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.198 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:51:25.198 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.488364+0000", 2024-02-09T17:51:25.198 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:25.198 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.199 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.199 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:51:25.199 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.488365+0000", 2024-02-09T17:51:25.199 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.47e-07 2024-02-09T17:51:25.199 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.199 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.199 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:51:25.199 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.488370+0000", 2024-02-09T17:51:25.199 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4120000000000003e-06 2024-02-09T17:51:25.199 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.199 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.199 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:51:25.200 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.488388+0000", 2024-02-09T17:51:25.200 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8479e-05 2024-02-09T17:51:25.200 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.200 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.200 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:51:25.200 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.488396+0000", 2024-02-09T17:51:25.200 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.661e-06 2024-02-09T17:51:25.200 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.200 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.200 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:25.200 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.497037+0000", 2024-02-09T17:51:25.200 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0086414300000000003 2024-02-09T17:51:25.200 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.200 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.201 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:51:25.201 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.497173+0000", 2024-02-09T17:51:25.201 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013531099999999999 2024-02-09T17:51:25.201 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.201 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.201 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,9", 2024-02-09T17:51:25.201 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.497293+0000", 2024-02-09T17:51:25.201 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000120549 2024-02-09T17:51:25.201 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.201 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.201 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:25.201 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.501982+0000", 2024-02-09T17:51:25.202 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0046894010000000002 2024-02-09T17:51:25.202 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.202 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.202 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:51:25.202 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.506566+0000", 2024-02-09T17:51:25.202 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00458334 2024-02-09T17:51:25.202 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.202 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.202 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:25.202 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.580816+0000", 2024-02-09T17:51:25.202 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.074250110999999994 2024-02-09T17:51:25.202 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.203 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.203 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:51:25.203 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.580852+0000", 2024-02-09T17:51:25.203 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.5929000000000001e-05 2024-02-09T17:51:25.203 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.203 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.203 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:51:25.203 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.580877+0000", 2024-02-09T17:51:25.203 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4913999999999999e-05 2024-02-09T17:51:25.203 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:25.203 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:51:25.203 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:25.204 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.204 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.204 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25837.0:202637 6.f 6:f258b00e:::benchmark_data_smithi179_126206_object72256:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e638)", 2024-02-09T17:51:25.204 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:48:47.501920+0000", 2024-02-09T17:51:25.204 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 157.582323532, 2024-02-09T17:51:25.204 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.079088620999999998, 2024-02-09T17:51:25.204 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:51:25.204 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:51:25.204 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:51:25.204 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25837", 2024-02-09T17:51:25.204 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:51:25.204 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 202637 2024-02-09T17:51:25.205 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.205 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:51:25.205 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.205 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:51:25.205 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.501920+0000", 2024-02-09T17:51:25.205 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:25.205 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.205 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.205 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:51:25.205 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.501920+0000", 2024-02-09T17:51:25.205 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:25.205 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.205 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.206 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:51:25.206 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.501922+0000", 2024-02-09T17:51:25.206 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0190000000000001e-06 2024-02-09T17:51:25.206 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.206 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.206 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:51:25.206 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.501927+0000", 2024-02-09T17:51:25.206 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0010000000000001e-06 2024-02-09T17:51:25.206 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.206 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.206 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:51:25.206 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.501939+0000", 2024-02-09T17:51:25.206 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.146e-05 2024-02-09T17:51:25.206 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.207 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.207 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:51:25.207 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.501946+0000", 2024-02-09T17:51:25.207 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4320000000000002e-06 2024-02-09T17:51:25.207 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.207 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.207 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:25.207 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.502019+0000", 2024-02-09T17:51:25.207 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2898000000000006e-05 2024-02-09T17:51:25.207 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.207 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.207 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:51:25.207 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.502156+0000", 2024-02-09T17:51:25.208 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013662400000000001 2024-02-09T17:51:25.208 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.208 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.208 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,9", 2024-02-09T17:51:25.208 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.502280+0000", 2024-02-09T17:51:25.208 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012392800000000001 2024-02-09T17:51:25.208 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.208 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.208 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:25.208 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.506713+0000", 2024-02-09T17:51:25.208 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.004433607 2024-02-09T17:51:25.208 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.209 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.209 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:51:25.209 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.515170+0000", 2024-02-09T17:51:25.209 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0084569550000000004 2024-02-09T17:51:25.209 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.209 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.209 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:25.209 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.580924+0000", 2024-02-09T17:51:25.209 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.065753354 2024-02-09T17:51:25.209 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.209 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.209 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:51:25.209 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.580972+0000", 2024-02-09T17:51:25.210 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8059000000000002e-05 2024-02-09T17:51:25.210 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.210 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.210 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:51:25.210 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.581009+0000", 2024-02-09T17:51:25.210 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7283999999999997e-05 2024-02-09T17:51:25.210 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:25.210 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:51:25.210 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:25.210 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.210 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.211 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25837.0:202638 6.f 6:fc3bb8d9:::benchmark_data_smithi179_126206_object72257:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e638)", 2024-02-09T17:51:25.211 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:48:47.504642+0000", 2024-02-09T17:51:25.211 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 157.57960181199999, 2024-02-09T17:51:25.211 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.076473263999999999, 2024-02-09T17:51:25.211 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:51:25.211 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:51:25.211 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:51:25.211 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25837", 2024-02-09T17:51:25.211 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:51:25.211 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 202638 2024-02-09T17:51:25.211 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.212 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:51:25.212 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.212 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:51:25.212 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.504642+0000", 2024-02-09T17:51:25.212 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:25.212 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.212 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.212 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:51:25.212 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.504642+0000", 2024-02-09T17:51:25.212 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:25.212 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.212 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.213 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:51:25.213 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.504643+0000", 2024-02-09T17:51:25.213 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.879999999999999e-07 2024-02-09T17:51:25.213 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.213 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.213 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:51:25.213 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.504648+0000", 2024-02-09T17:51:25.213 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3810000000000001e-06 2024-02-09T17:51:25.213 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.213 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.213 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:51:25.213 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.504658+0000", 2024-02-09T17:51:25.213 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0549e-05 2024-02-09T17:51:25.214 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.214 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.214 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:51:25.214 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.504664+0000", 2024-02-09T17:51:25.214 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1689999999999997e-06 2024-02-09T17:51:25.214 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.214 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.214 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:25.214 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.506739+0000", 2024-02-09T17:51:25.214 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00207435 2024-02-09T17:51:25.214 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.214 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.214 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:51:25.215 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.506833+0000", 2024-02-09T17:51:25.215 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.4765999999999994e-05 2024-02-09T17:51:25.215 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.215 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.215 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,9", 2024-02-09T17:51:25.215 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.506907+0000", 2024-02-09T17:51:25.215 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.3178999999999998e-05 2024-02-09T17:51:25.215 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.215 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.215 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:25.215 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.511798+0000", 2024-02-09T17:51:25.215 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0048913070000000001 2024-02-09T17:51:25.216 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.216 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.216 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:51:25.216 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.515176+0000", 2024-02-09T17:51:25.216 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0033782669999999999 2024-02-09T17:51:25.216 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.216 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.216 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:25.216 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.581037+0000", 2024-02-09T17:51:25.216 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.065861123999999993 2024-02-09T17:51:25.216 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.216 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.216 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:51:25.216 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.581081+0000", 2024-02-09T17:51:25.217 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3374999999999997e-05 2024-02-09T17:51:25.217 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.217 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.217 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:51:25.217 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.581115+0000", 2024-02-09T17:51:25.217 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4808999999999998e-05 2024-02-09T17:51:25.217 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:25.217 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:51:25.217 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:25.217 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.217 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.217 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25837.0:228026 6.f 6:faf15fdb:::benchmark_data_smithi179_126206_object97645:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e649)", 2024-02-09T17:51:25.217 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:49:18.154652+0000", 2024-02-09T17:51:25.217 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 126.929591594, 2024-02-09T17:51:25.217 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.048060255000000003, 2024-02-09T17:51:25.218 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:51:25.218 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:51:25.218 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:51:25.218 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25837", 2024-02-09T17:51:25.218 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:51:25.218 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 228026 2024-02-09T17:51:25.218 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.218 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:51:25.218 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.218 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:51:25.218 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:18.154652+0000", 2024-02-09T17:51:25.218 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:25.218 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.218 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.218 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:51:25.219 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:18.154652+0000", 2024-02-09T17:51:25.219 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:25.219 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.219 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.219 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:51:25.219 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:18.154654+0000", 2024-02-09T17:51:25.219 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5e-06 2024-02-09T17:51:25.219 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.219 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.219 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:51:25.219 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:18.154660+0000", 2024-02-09T17:51:25.219 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8309999999999997e-06 2024-02-09T17:51:25.219 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.219 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.220 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:51:25.220 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:18.154673+0000", 2024-02-09T17:51:25.220 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3385e-05 2024-02-09T17:51:25.220 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.220 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.220 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:51:25.220 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:18.154681+0000", 2024-02-09T17:51:25.220 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6040000000000004e-06 2024-02-09T17:51:25.220 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.220 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.220 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:25.220 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:18.154956+0000", 2024-02-09T17:51:25.220 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00027560900000000001 2024-02-09T17:51:25.220 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.221 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.221 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:51:25.221 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:18.155074+0000", 2024-02-09T17:51:25.221 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000118183 2024-02-09T17:51:25.221 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.221 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.221 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,9", 2024-02-09T17:51:25.221 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:18.155194+0000", 2024-02-09T17:51:25.221 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000119033 2024-02-09T17:51:25.221 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.221 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.221 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:51:25.221 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:18.164007+0000", 2024-02-09T17:51:25.222 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0088137819999999992 2024-02-09T17:51:25.222 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.222 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.222 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:25.222 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:18.169208+0000", 2024-02-09T17:51:25.222 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0052007779999999997 2024-02-09T17:51:25.222 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.222 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.222 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:25.222 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:18.202634+0000", 2024-02-09T17:51:25.222 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.033425671999999997 2024-02-09T17:51:25.222 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.222 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.222 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:51:25.222 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:18.202672+0000", 2024-02-09T17:51:25.222 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7843999999999999e-05 2024-02-09T17:51:25.223 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.223 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.223 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:51:25.223 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:18.202713+0000", 2024-02-09T17:51:25.223 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1034e-05 2024-02-09T17:51:25.223 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:25.223 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:51:25.223 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:25.223 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.223 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.223 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25837.0:228032 6.f 6:f27c6014:::benchmark_data_smithi179_126206_object97651:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e649)", 2024-02-09T17:51:25.223 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:49:18.154863+0000", 2024-02-09T17:51:25.223 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 126.929380997, 2024-02-09T17:51:25.223 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.047990536, 2024-02-09T17:51:25.224 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:51:25.224 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:51:25.224 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:51:25.224 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25837", 2024-02-09T17:51:25.224 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:51:25.224 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 228032 2024-02-09T17:51:25.224 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.224 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:51:25.224 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.224 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:51:25.224 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:18.154863+0000", 2024-02-09T17:51:25.224 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:25.225 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.225 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.225 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:51:25.225 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:18.154863+0000", 2024-02-09T17:51:25.225 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:25.225 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.225 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.225 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:51:25.225 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:18.154864+0000", 2024-02-09T17:51:25.225 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8999999999999996e-07 2024-02-09T17:51:25.225 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.225 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.225 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:51:25.225 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:18.154867+0000", 2024-02-09T17:51:25.225 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1159999999999999e-06 2024-02-09T17:51:25.226 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.226 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.226 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:51:25.226 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:18.154874+0000", 2024-02-09T17:51:25.226 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4109999999999999e-06 2024-02-09T17:51:25.226 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.226 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.226 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:51:25.226 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:18.154881+0000", 2024-02-09T17:51:25.226 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8889999999999996e-06 2024-02-09T17:51:25.226 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.226 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.227 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:25.227 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:18.159158+0000", 2024-02-09T17:51:25.227 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0042767810000000003 2024-02-09T17:51:25.227 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.227 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.227 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:51:25.227 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:18.159244+0000", 2024-02-09T17:51:25.227 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5869000000000001e-05 2024-02-09T17:51:25.227 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.227 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.227 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,9", 2024-02-09T17:51:25.227 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:18.159342+0000", 2024-02-09T17:51:25.227 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.8361000000000002e-05 2024-02-09T17:51:25.228 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.228 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.228 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:51:25.228 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:18.169169+0000", 2024-02-09T17:51:25.228 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0098272919999999996 2024-02-09T17:51:25.228 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.228 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.228 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:25.228 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:18.170298+0000", 2024-02-09T17:51:25.228 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001129038 2024-02-09T17:51:25.228 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.228 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.228 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:25.228 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:18.202766+0000", 2024-02-09T17:51:25.229 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.032468021 2024-02-09T17:51:25.229 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.229 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.229 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:51:25.229 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:18.202819+0000", 2024-02-09T17:51:25.229 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2587999999999999e-05 2024-02-09T17:51:25.229 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.229 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.229 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:51:25.229 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:18.202854+0000", 2024-02-09T17:51:25.229 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4579999999999998e-05 2024-02-09T17:51:25.229 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:25.229 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:51:25.229 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:25.230 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.230 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.230 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25837.0:259884 6.f 6:f3d07a77:::benchmark_data_smithi179_126206_object129503:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:51:25.230 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:49:59.505931+0000", 2024-02-09T17:51:25.230 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 85.578312741000005, 2024-02-09T17:51:25.230 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.050978498999999997, 2024-02-09T17:51:25.230 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:51:25.230 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:51:25.230 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:51:25.230 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25837", 2024-02-09T17:51:25.230 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:51:25.230 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 259884 2024-02-09T17:51:25.230 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.230 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:51:25.231 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.231 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:51:25.231 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:59.505931+0000", 2024-02-09T17:51:25.231 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:25.231 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.231 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.231 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:51:25.231 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:59.505931+0000", 2024-02-09T17:51:25.231 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:25.231 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.231 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.231 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:51:25.231 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:59.505932+0000", 2024-02-09T17:51:25.231 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8999999999999995e-07 2024-02-09T17:51:25.232 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.232 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.232 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:51:25.232 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:59.505936+0000", 2024-02-09T17:51:25.232 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3159999999999998e-06 2024-02-09T17:51:25.232 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.232 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.232 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:51:25.232 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:59.505948+0000", 2024-02-09T17:51:25.232 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1899000000000001e-05 2024-02-09T17:51:25.232 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.232 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.233 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:51:25.233 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:59.505956+0000", 2024-02-09T17:51:25.233 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2479999999999999e-06 2024-02-09T17:51:25.233 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.233 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.233 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:25.233 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:59.509313+0000", 2024-02-09T17:51:25.233 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0033570269999999998 2024-02-09T17:51:25.233 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.233 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.233 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:51:25.233 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:59.509406+0000", 2024-02-09T17:51:25.233 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.3148999999999999e-05 2024-02-09T17:51:25.233 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.233 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.234 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,9", 2024-02-09T17:51:25.234 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:59.509482+0000", 2024-02-09T17:51:25.234 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6129000000000002e-05 2024-02-09T17:51:25.234 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.234 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.234 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:51:25.234 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:59.513188+0000", 2024-02-09T17:51:25.234 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0037064950000000002 2024-02-09T17:51:25.234 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.234 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.234 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:25.234 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:59.513281+0000", 2024-02-09T17:51:25.234 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.3002000000000002e-05 2024-02-09T17:51:25.235 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.235 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.235 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:25.235 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:59.556806+0000", 2024-02-09T17:51:25.235 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.043524188999999998 2024-02-09T17:51:25.235 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.235 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.235 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:51:25.235 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:59.556868+0000", 2024-02-09T17:51:25.235 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2453000000000007e-05 2024-02-09T17:51:25.235 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.235 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.235 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:51:25.235 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:59.556910+0000", 2024-02-09T17:51:25.236 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1702000000000002e-05 2024-02-09T17:51:25.236 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:25.236 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:51:25.236 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:25.236 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.236 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.236 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.16440.0:87189 7.6 e689/682 7:6b309ca4:::benchmark_data_smithi179_135804_object87188:head v 689'5431, mlcod=665'460)", 2024-02-09T17:51:25.236 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:51:24.538505+0000", 2024-02-09T17:51:25.236 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.545739164, 2024-02-09T17:51:25.236 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.047833563000000003, 2024-02-09T17:51:25.236 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:51:25.236 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:51:25.236 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:51:25.236 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.237 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:51:25.237 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.538505+0000", 2024-02-09T17:51:25.237 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:25.237 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.237 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.237 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:51:25.237 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.538505+0000", 2024-02-09T17:51:25.237 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:25.237 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.237 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.237 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:51:25.237 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.538506+0000", 2024-02-09T17:51:25.237 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7399999999999991e-07 2024-02-09T17:51:25.237 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.238 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.238 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:51:25.238 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.538543+0000", 2024-02-09T17:51:25.238 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7416000000000002e-05 2024-02-09T17:51:25.238 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.238 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.238 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:51:25.238 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.538556+0000", 2024-02-09T17:51:25.238 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2768999999999999e-05 2024-02-09T17:51:25.238 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.238 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.238 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:51:25.238 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.538564+0000", 2024-02-09T17:51:25.238 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3189999999999995e-06 2024-02-09T17:51:25.239 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.239 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.239 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:25.239 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.542729+0000", 2024-02-09T17:51:25.239 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0041643790000000002 2024-02-09T17:51:25.239 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.239 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.239 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:51:25.239 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.542757+0000", 2024-02-09T17:51:25.239 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.8809000000000001e-05 2024-02-09T17:51:25.239 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.239 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.239 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:51:25.239 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.586276+0000", 2024-02-09T17:51:25.239 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.043518669000000003 2024-02-09T17:51:25.239 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:25.240 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:25.240 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:51:25.240 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.586338+0000", 2024-02-09T17:51:25.240 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2228000000000001e-05 2024-02-09T17:51:25.240 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:25.240 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:51:25.240 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:25.240 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:25.240 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:51:25.240 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:51:25.429 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:25 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 1/256164 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:51:25.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:25 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 1/256164 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:51:25.517 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 59 16 87850 87834 93.0195 41.6875 0.0149363 0.0107449 2024-02-09T17:51:25.652 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:51:25.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:25 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 1/256164 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:51:25.698 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:51:25.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:25.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:51:25.804 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:51:25.859 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:26.358 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:51:26.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:26.370 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:51:26.471 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:51:26.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:26 smithi099 ceph-mon[39420]: pgmap v1729: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 576 op/s; 1/256164 objects degraded (0.000%); 36409/256164 objects misplaced (14.213%); 15 MiB/s, 233 objects/s recovering 2024-02-09T17:51:26.510 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:26.519 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:2024-02-09T17:51:26.519012+0000 min lat: 0.0011365 max lat: 2.41216 avg lat: 0.0108561 2024-02-09T17:51:26.519 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-09T17:51:26.519 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 60 16 88425 88409 92.0644 35.9375 0.0307727 0.0108561 2024-02-09T17:51:26.534 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:51:26.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:26.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:51:26.566 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:26 smithi179 ceph-mon[43975]: pgmap v1729: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 576 op/s; 1/256164 objects degraded (0.000%); 36409/256164 objects misplaced (14.213%); 15 MiB/s, 233 objects/s recovering 2024-02-09T17:51:26.641 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:51:26.679 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:26.681 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:26 smithi162 ceph-mon[44304]: pgmap v1729: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 576 op/s; 1/256164 objects degraded (0.000%); 36409/256164 objects misplaced (14.213%); 15 MiB/s, 233 objects/s recovering 2024-02-09T17:51:26.711 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:51:26.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:26.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:51:26.818 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:51:26.864 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:27.242 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set noscrub 2024-02-09T17:51:27.522 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 61 16 89023 89007 91.1635 37.375 0.0322804 0.0109611 2024-02-09T17:51:27.600 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.11 dump_ops_in_flight 2024-02-09T17:51:27.669 INFO:teuthology.orchestra.run.smithi099.stdout:moved 0 / 141 (0%) 2024-02-09T17:51:27.669 INFO:teuthology.orchestra.run.smithi099.stdout:avg 11.75 2024-02-09T17:51:27.669 INFO:teuthology.orchestra.run.smithi099.stdout:stddev 2.24072 -> 2.24072 (expected baseline 3.28189) 2024-02-09T17:51:27.669 INFO:teuthology.orchestra.run.smithi099.stdout:min osd.0 with 8 -> 8 pgs (0.680851 -> 0.680851 * mean) 2024-02-09T17:51:27.669 INFO:teuthology.orchestra.run.smithi099.stdout:max osd.7 with 15 -> 15 pgs (1.2766 -> 1.2766 * mean) 2024-02-09T17:51:27.669 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:51:27.669 INFO:teuthology.orchestra.run.smithi099.stdout:oload 1000 2024-02-09T17:51:27.670 INFO:teuthology.orchestra.run.smithi099.stdout:max_change 1 2024-02-09T17:51:27.670 INFO:teuthology.orchestra.run.smithi099.stdout:max_change_osds 4 2024-02-09T17:51:27.670 INFO:teuthology.orchestra.run.smithi099.stdout:average_utilization 0.0217 2024-02-09T17:51:27.670 INFO:teuthology.orchestra.run.smithi099.stdout:overload_utilization 0.2168 2024-02-09T17:51:27.670 INFO:teuthology.orchestra.run.smithi099.stderr:no change 2024-02-09T17:51:27.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:27 smithi179 ceph-mon[43975]: 1.1e scrub starts 2024-02-09T17:51:27.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:27 smithi179 ceph-mon[43975]: osdmap e690: 12 total, 12 up, 12 in 2024-02-09T17:51:27.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:27 smithi179 ceph-mon[43975]: 1.1e scrub ok 2024-02-09T17:51:27.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:27 smithi179 ceph-mon[43975]: pgmap v1731: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 43 MiB/s wr, 686 op/s; 1/262512 objects degraded (0.000%); 35271/262512 objects misplaced (13.436%); 19 MiB/s, 309 objects/s recovering 2024-02-09T17:51:27.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:27 smithi099 ceph-mon[39420]: 1.1e scrub starts 2024-02-09T17:51:27.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:27 smithi099 ceph-mon[39420]: osdmap e690: 12 total, 12 up, 12 in 2024-02-09T17:51:27.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:27 smithi099 ceph-mon[39420]: 1.1e scrub ok 2024-02-09T17:51:27.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:27 smithi099 ceph-mon[39420]: pgmap v1731: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 43 MiB/s wr, 686 op/s; 1/262512 objects degraded (0.000%); 35271/262512 objects misplaced (13.436%); 19 MiB/s, 309 objects/s recovering 2024-02-09T17:51:28.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:27 smithi162 ceph-mon[44304]: 1.1e scrub starts 2024-02-09T17:51:28.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:27 smithi162 ceph-mon[44304]: osdmap e690: 12 total, 12 up, 12 in 2024-02-09T17:51:28.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:27 smithi162 ceph-mon[44304]: 1.1e scrub ok 2024-02-09T17:51:28.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:27 smithi162 ceph-mon[44304]: pgmap v1731: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 43 MiB/s wr, 686 op/s; 1/262512 objects degraded (0.000%); 35271/262512 objects misplaced (13.436%); 19 MiB/s, 309 objects/s recovering 2024-02-09T17:51:28.307 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:51:28.523 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 62 16 89662 89646 90.3374 39.9375 0.0116287 0.0110648 2024-02-09T17:51:28.927 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:28 smithi179 ceph-mon[43975]: from='client.26048 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:51:29.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:28 smithi162 ceph-mon[44304]: from='client.26048 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:51:29.220 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:28 smithi099 ceph-mon[39420]: from='client.26048 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:51:29.523 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 63 16 90336 90320 89.5722 42.125 0.0207701 0.0111583 2024-02-09T17:51:29.725 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-02-09T17:51:29.725 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight-by-utilization 1000 1.0 2024-02-09T17:51:29.832 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:29 smithi099 ceph-mon[39420]: pgmap v1732: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 593 op/s; 1/264885 objects degraded (0.000%); 34971/264885 objects misplaced (13.202%); 15 MiB/s, 240 objects/s recovering 2024-02-09T17:51:29.838 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:29 smithi179 ceph-mon[43975]: pgmap v1732: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 593 op/s; 1/264885 objects degraded (0.000%); 34971/264885 objects misplaced (13.202%); 15 MiB/s, 240 objects/s recovering 2024-02-09T17:51:29.984 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:51:29.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:29.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:51:30.092 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:51:30.148 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:29 smithi162 ceph-mon[44304]: pgmap v1732: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 593 op/s; 1/264885 objects degraded (0.000%); 34971/264885 objects misplaced (13.202%); 15 MiB/s, 240 objects/s recovering 2024-02-09T17:51:30.148 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:30.167 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:51:30.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:30.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:51:30.271 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:51:30.311 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:51:30.363 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:30.369 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:51:30.419 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:51:30.445 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:51:30.473 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:51:30.502 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:51:30.524 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 64 16 90999 90983 88.8198 41.4375 0.0133883 0.0112539 2024-02-09T17:51:30.531 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:51:30.534 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:51:30.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:30.537 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:51:30.552 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:51:30.579 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:51:30.609 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:51:30.638 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:51:30.673 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:51:30.719 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:51:30.721 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:30.736 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:51:30.736 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-02-09T17:51:30.736 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:30.736 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.16440.0:91103 7.9 7:9bc337ba:::benchmark_data_smithi179_135804_object91102:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e690)", 2024-02-09T17:51:30.737 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:51:30.722402+0000", 2024-02-09T17:51:30.737 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.012778161999999999, 2024-02-09T17:51:30.737 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.012826821, 2024-02-09T17:51:30.737 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:51:30.737 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "waiting for sub ops", 2024-02-09T17:51:30.737 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:51:30.737 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.16440", 2024-02-09T17:51:30.737 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:51:30.737 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 91103 2024-02-09T17:51:30.737 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:30.737 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:51:30.737 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:30.737 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:51:30.737 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:30.722402+0000", 2024-02-09T17:51:30.738 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:30.738 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:30.738 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:30.738 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:51:30.738 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:30.722402+0000", 2024-02-09T17:51:30.738 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:30.738 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:30.738 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:30.738 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:51:30.738 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:30.722404+0000", 2024-02-09T17:51:30.738 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0679999999999999e-06 2024-02-09T17:51:30.738 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:30.738 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:30.738 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:51:30.738 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:30.722430+0000", 2024-02-09T17:51:30.739 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5607000000000001e-05 2024-02-09T17:51:30.739 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:30.739 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:30.739 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:51:30.739 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:30.723050+0000", 2024-02-09T17:51:30.739 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00062009700000000001 2024-02-09T17:51:30.739 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:30.739 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:30.739 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:51:30.739 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:30.723061+0000", 2024-02-09T17:51:30.739 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1284000000000001e-05 2024-02-09T17:51:30.739 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:30.739 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:30.739 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:30.739 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:30.729872+0000", 2024-02-09T17:51:30.740 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0068109039999999996 2024-02-09T17:51:30.740 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:30.740 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:30.740 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:51:30.740 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:30.730066+0000", 2024-02-09T17:51:30.740 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00019426600000000001 2024-02-09T17:51:30.740 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:30.740 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:30.740 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 8,10", 2024-02-09T17:51:30.740 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:30.730275+0000", 2024-02-09T17:51:30.740 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00020839400000000001 2024-02-09T17:51:30.740 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:30.740 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:51:30.740 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:30.740 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:30.741 INFO:teuthology.orchestra.run.smithi179.stdout: ], 2024-02-09T17:51:30.741 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 1 2024-02-09T17:51:30.741 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:51:30.758 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:51:30.761 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:51:30.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:30.767 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:51:30.791 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:51:30.792 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:51:30.793 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:51:30.793 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:51:30.793 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:51:30.793 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:51:30.793 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:51:30.793 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:51:30.793 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:51:30.793 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:51:30.793 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:51:30.793 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:51:30.793 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:51:30.840 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:30 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:51:30.840 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:30 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:51:30.840 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:30 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1632339479' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:51:30.867 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:51:30.906 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:30 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:51:30.906 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:30 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:51:30.907 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:30 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1632339479' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:51:30.929 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:31.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:30 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:51:31.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:30 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:51:31.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:30 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1632339479' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:51:31.526 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 65 16 91611 91595 88.0375 38.25 0.0222959 0.0113518 2024-02-09T17:51:32.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:31 smithi179 ceph-mon[43975]: osdmap e691: 12 total, 12 up, 12 in 2024-02-09T17:51:32.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:31 smithi179 ceph-mon[43975]: pgmap v1734: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 599 op/s; 1/269049 objects degraded (0.000%); 34234/269049 objects misplaced (12.724%); 19 MiB/s, 300 objects/s recovering 2024-02-09T17:51:32.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:31 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 1/264885 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:51:32.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:31 smithi179 ceph-mon[43975]: osdmap e692: 12 total, 12 up, 12 in 2024-02-09T17:51:32.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:31 smithi162 ceph-mon[44304]: osdmap e691: 12 total, 12 up, 12 in 2024-02-09T17:51:32.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:31 smithi162 ceph-mon[44304]: pgmap v1734: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 599 op/s; 1/269049 objects degraded (0.000%); 34234/269049 objects misplaced (12.724%); 19 MiB/s, 300 objects/s recovering 2024-02-09T17:51:32.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:31 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 1/264885 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:51:32.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:31 smithi162 ceph-mon[44304]: osdmap e692: 12 total, 12 up, 12 in 2024-02-09T17:51:32.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:31 smithi099 ceph-mon[39420]: osdmap e691: 12 total, 12 up, 12 in 2024-02-09T17:51:32.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:31 smithi099 ceph-mon[39420]: pgmap v1734: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 599 op/s; 1/269049 objects degraded (0.000%); 34234/269049 objects misplaced (12.724%); 19 MiB/s, 300 objects/s recovering 2024-02-09T17:51:32.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:31 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 1/264885 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:51:32.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:31 smithi099 ceph-mon[39420]: osdmap e692: 12 total, 12 up, 12 in 2024-02-09T17:51:32.528 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 66 16 92296 92280 87.3508 42.8125 0.0407589 0.0114429 2024-02-09T17:51:32.589 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:51:32.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:32.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:51:32.693 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:51:32.728 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:32.761 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:51:32.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:32.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:51:32.869 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:51:32.927 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:33.529 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 67 16 92987 92971 86.6918 43.1875 0.0263489 0.0115295 2024-02-09T17:51:34.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:34 smithi179 ceph-mon[43975]: pgmap v1736: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.6 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 41 MiB/s wr, 661 op/s; 1/273867 objects degraded (0.000%); 33685/273867 objects misplaced (12.300%); 16 MiB/s, 252 objects/s recovering 2024-02-09T17:51:34.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:34 smithi162 ceph-mon[44304]: pgmap v1736: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.6 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 41 MiB/s wr, 661 op/s; 1/273867 objects degraded (0.000%); 33685/273867 objects misplaced (12.300%); 16 MiB/s, 252 objects/s recovering 2024-02-09T17:51:34.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:34 smithi099 ceph-mon[39420]: pgmap v1736: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.6 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 41 MiB/s wr, 661 op/s; 1/273867 objects degraded (0.000%); 33685/273867 objects misplaced (12.300%); 16 MiB/s, 252 objects/s recovering 2024-02-09T17:51:34.529 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 68 16 93660 93644 86.035 42.0625 0.0301856 0.0116181 2024-02-09T17:51:34.758 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.11 dump_blocked_ops 2024-02-09T17:51:34.780 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:51:34.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:34.790 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:51:34.892 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:51:34.963 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:35.019 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:51:35.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:35.032 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:51:35.134 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:51:35.187 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:35.334 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:51:35.395 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:51:35.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:35.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:51:35.499 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:51:35.533 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 69 16 94316 94300 85.3781 41 0.0162071 0.0117076 2024-02-09T17:51:35.560 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:35.597 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:51:35.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:35.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:51:35.709 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:51:35.783 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:36.143 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:36 smithi179 ceph-mon[43975]: pgmap v1737: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.6 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s wr, 523 op/s; 1/275076 objects degraded (0.000%); 33517/275076 objects misplaced (12.185%); 14 MiB/s, 219 objects/s recovering 2024-02-09T17:51:36.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:36 smithi099 ceph-mon[39420]: pgmap v1737: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.6 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s wr, 523 op/s; 1/275076 objects degraded (0.000%); 33517/275076 objects misplaced (12.185%); 14 MiB/s, 219 objects/s recovering 2024-02-09T17:51:36.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:36 smithi162 ceph-mon[44304]: pgmap v1737: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.6 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s wr, 523 op/s; 1/275076 objects degraded (0.000%); 33517/275076 objects misplaced (12.185%); 14 MiB/s, 219 objects/s recovering 2024-02-09T17:51:36.532 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 70 16 94983 94967 84.7541 41.6875 0.0377926 0.0117929 2024-02-09T17:51:36.980 INFO:teuthology.orchestra.run.smithi099.stdout:moved 0 / 141 (0%) 2024-02-09T17:51:36.980 INFO:teuthology.orchestra.run.smithi099.stdout:avg 11.75 2024-02-09T17:51:36.980 INFO:teuthology.orchestra.run.smithi099.stdout:stddev 2.24072 -> 2.24072 (expected baseline 3.28189) 2024-02-09T17:51:36.981 INFO:teuthology.orchestra.run.smithi099.stdout:min osd.0 with 8 -> 8 pgs (0.680851 -> 0.680851 * mean) 2024-02-09T17:51:36.981 INFO:teuthology.orchestra.run.smithi099.stdout:max osd.7 with 15 -> 15 pgs (1.2766 -> 1.2766 * mean) 2024-02-09T17:51:36.981 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:51:36.981 INFO:teuthology.orchestra.run.smithi099.stdout:oload 1000 2024-02-09T17:51:36.981 INFO:teuthology.orchestra.run.smithi099.stdout:max_change 1 2024-02-09T17:51:36.981 INFO:teuthology.orchestra.run.smithi099.stdout:max_change_osds 4 2024-02-09T17:51:36.982 INFO:teuthology.orchestra.run.smithi099.stdout:average_utilization 0.0229 2024-02-09T17:51:36.982 INFO:teuthology.orchestra.run.smithi099.stdout:overload_utilization 0.2290 2024-02-09T17:51:36.989 INFO:teuthology.orchestra.run.smithi099.stderr:no change 2024-02-09T17:51:37.045 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:51:37.051 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:37.052 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:51:37.153 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:51:37.172 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:51:37.213 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:37.379 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:51:37.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:37.384 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:51:37.384 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:37 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:51:37.384 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:37 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:51:37.384 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:37 smithi179 ceph-mon[43975]: osdmap e693: 12 total, 12 up, 12 in 2024-02-09T17:51:37.384 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:37 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 1/275076 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:51:37.384 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:37 smithi179 ceph-mon[43975]: osdmap e694: 12 total, 12 up, 12 in 2024-02-09T17:51:37.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:37 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:51:37.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:37 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:51:37.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:37 smithi162 ceph-mon[44304]: osdmap e693: 12 total, 12 up, 12 in 2024-02-09T17:51:37.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:37 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 1/275076 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:51:37.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:37 smithi162 ceph-mon[44304]: osdmap e694: 12 total, 12 up, 12 in 2024-02-09T17:51:37.485 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:51:37.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:37 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:51:37.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:37 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:51:37.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:37 smithi099 ceph-mon[39420]: osdmap e693: 12 total, 12 up, 12 in 2024-02-09T17:51:37.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:37 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 1/275076 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:51:37.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:37 smithi099 ceph-mon[39420]: osdmap e694: 12 total, 12 up, 12 in 2024-02-09T17:51:37.530 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:37.532 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 71 16 95615 95599 84.1169 39.5 0.010265 0.0118818 2024-02-09T17:51:37.564 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:51:37.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:37.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:51:37.668 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:51:37.741 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:37.783 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:51:37.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:37.791 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:51:37.815 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is set 2024-02-09T17:51:37.892 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:51:37.943 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:37.968 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:51:37.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:37.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:51:38.080 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:51:38.139 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:38.161 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:38 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:51:38.161 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:38 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:51:38.161 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:38 smithi099 ceph-mon[39420]: from='client.16503 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:51:38.161 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:38 smithi099 ceph-mon[39420]: pgmap v1740: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.7 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 726 op/s; 1/282123 objects degraded (0.000%); 32358/282123 objects misplaced (11.469%); 20 MiB/s, 312 objects/s recovering 2024-02-09T17:51:38.161 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:38 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3768889893' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:51:38.161 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:38 smithi099 ceph-mon[39420]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:51:38.161 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:38 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3768889893' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:51:38.161 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:38 smithi099 ceph-mon[39420]: osdmap e695: 12 total, 12 up, 12 in 2024-02-09T17:51:38.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:38 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:51:38.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:38 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:51:38.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:38 smithi179 ceph-mon[43975]: from='client.16503 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:51:38.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:38 smithi179 ceph-mon[43975]: pgmap v1740: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.7 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 726 op/s; 1/282123 objects degraded (0.000%); 32358/282123 objects misplaced (11.469%); 20 MiB/s, 312 objects/s recovering 2024-02-09T17:51:38.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:38 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3768889893' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:51:38.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:38 smithi179 ceph-mon[43975]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:51:38.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:38 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3768889893' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:51:38.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:38 smithi179 ceph-mon[43975]: osdmap e695: 12 total, 12 up, 12 in 2024-02-09T17:51:38.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:38 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:51:38.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:38 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:51:38.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:38 smithi162 ceph-mon[44304]: from='client.16503 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:51:38.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:38 smithi162 ceph-mon[44304]: pgmap v1740: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.7 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 726 op/s; 1/282123 objects degraded (0.000%); 32358/282123 objects misplaced (11.469%); 20 MiB/s, 312 objects/s recovering 2024-02-09T17:51:38.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:38 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3768889893' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:51:38.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:38 smithi162 ceph-mon[44304]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:51:38.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:38 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3768889893' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:51:38.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:38 smithi162 ceph-mon[44304]: osdmap e695: 12 total, 12 up, 12 in 2024-02-09T17:51:38.532 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 72 16 96298 96282 83.5416 42.6875 0.0207856 0.0119646 2024-02-09T17:51:39.534 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 73 16 96882 96866 82.896 36.5 0.0356587 0.0120569 2024-02-09T17:51:39.806 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:51:39.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:39.811 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:51:39.912 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:51:39.949 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:39.962 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-02-09T17:51:39.962 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd test-reweight-by-utilization 110 3.0 2024-02-09T17:51:39.993 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:51:39.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:39.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:51:40.008 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:51:40.098 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:51:40.170 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:40.233 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:51:40.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:40.241 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:51:40.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:40 smithi099 ceph-mon[39420]: pgmap v1742: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.8 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 592 op/s; 1/284526 objects degraded (0.000%); 32046/284526 objects misplaced (11.263%); 17 MiB/s, 273 objects/s recovering 2024-02-09T17:51:40.342 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:51:40.384 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:40.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:40 smithi179 ceph-mon[43975]: pgmap v1742: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.8 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 592 op/s; 1/284526 objects degraded (0.000%); 32046/284526 objects misplaced (11.263%); 17 MiB/s, 273 objects/s recovering 2024-02-09T17:51:40.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:40 smithi162 ceph-mon[44304]: pgmap v1742: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.8 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 592 op/s; 1/284526 objects degraded (0.000%); 32046/284526 objects misplaced (11.263%); 17 MiB/s, 273 objects/s recovering 2024-02-09T17:51:40.494 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:51:40.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:40.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:51:40.548 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 74 16 97536 97520 82.3229 40.875 0.0469289 0.0121422 2024-02-09T17:51:40.598 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:51:40.637 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:40.857 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T17:51:40.857 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:51:40.894 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:51:40.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:40.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:51:41.002 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:51:41.041 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:41.071 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:51:41.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:41.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:51:41.177 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:51:41.226 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:41.250 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:51:41.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:41.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:51:41.355 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:51:41.397 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:41.422 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:51:41.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:41.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:51:41.445 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:51:41.495 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-09T17:51:41.530 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:51:41.540 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 75 16 98157 98141 81.7411 38.8125 0.0406503 0.0122287 2024-02-09T17:51:41.589 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:41.958 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:51:41.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:41.961 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:51:42.062 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:51:42.100 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:42.128 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:51:42.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:42.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:51:42.237 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:51:42.272 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:42 smithi179 ceph-mon[43975]: pgmap v1743: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.9 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 769 op/s; 1/288930 objects degraded (0.000%); 31275/288930 objects misplaced (10.824%); 23 MiB/s, 373 objects/s recovering 2024-02-09T17:51:42.272 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:42 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:51:42.272 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:42 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:51:42.272 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:42 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 1/288930 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:51:42.272 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:42 smithi179 ceph-mon[43975]: osdmap e696: 12 total, 12 up, 12 in 2024-02-09T17:51:42.296 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:42.331 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:51:42.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:42.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:51:42.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:42 smithi162 ceph-mon[44304]: pgmap v1743: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.9 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 769 op/s; 1/288930 objects degraded (0.000%); 31275/288930 objects misplaced (10.824%); 23 MiB/s, 373 objects/s recovering 2024-02-09T17:51:42.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:42 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:51:42.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:42 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:51:42.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:42 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 1/288930 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:51:42.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:42 smithi162 ceph-mon[44304]: osdmap e696: 12 total, 12 up, 12 in 2024-02-09T17:51:42.436 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:51:42.493 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:42.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:42 smithi099 ceph-mon[39420]: pgmap v1743: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.9 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 769 op/s; 1/288930 objects degraded (0.000%); 31275/288930 objects misplaced (10.824%); 23 MiB/s, 373 objects/s recovering 2024-02-09T17:51:42.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:42 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:51:42.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:42 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:51:42.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:42 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 1/288930 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:51:42.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:42 smithi099 ceph-mon[39420]: osdmap e696: 12 total, 12 up, 12 in 2024-02-09T17:51:42.541 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 76 16 98814 98798 81.2059 41.0625 0.0189125 0.012309 2024-02-09T17:51:43.486 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:51:43.517 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:51:43.541 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 77 16 99471 99455 80.6847 41.0625 0.0177412 0.0123897 2024-02-09T17:51:43.739 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:51:43.769 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:51:43.810 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:51:43.846 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:51:43.884 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:51:43.913 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:51:43.944 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:51:43.975 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:51:44.003 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:51:44.041 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:51:44.043 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:51:44.043 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:51:44.044 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:51:44.044 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:51:44.045 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:51:44.045 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:51:44.045 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:51:44.045 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:51:44.045 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:51:44.045 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:51:44.045 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:51:44.045 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:51:44.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:44 smithi099 ceph-mon[39420]: pgmap v1745: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 42 MiB/s wr, 678 op/s; 1/293613 objects degraded (0.000%); 30689/293613 objects misplaced (10.452%); 17 MiB/s, 266 objects/s recovering 2024-02-09T17:51:44.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:44 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1941705183' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:51:44.541 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 78 16 100121 100105 80.1713 40.625 0.0200547 0.0124665 2024-02-09T17:51:44.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:44 smithi179 ceph-mon[43975]: pgmap v1745: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 42 MiB/s wr, 678 op/s; 1/293613 objects degraded (0.000%); 30689/293613 objects misplaced (10.452%); 17 MiB/s, 266 objects/s recovering 2024-02-09T17:51:44.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:44 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1941705183' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:51:44.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:44 smithi162 ceph-mon[44304]: pgmap v1745: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 42 MiB/s wr, 678 op/s; 1/293613 objects degraded (0.000%); 30689/293613 objects misplaced (10.452%); 17 MiB/s, 266 objects/s recovering 2024-02-09T17:51:44.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:44 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1941705183' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:51:44.907 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:51:45.000 INFO:teuthology.orchestra.run.smithi099.stdout:moved 9 / 141 (6.38298%) 2024-02-09T17:51:45.000 INFO:teuthology.orchestra.run.smithi099.stdout:avg 11.75 2024-02-09T17:51:45.000 INFO:teuthology.orchestra.run.smithi099.stdout:stddev 2.24072 -> 3.34477 (expected baseline 3.28189) 2024-02-09T17:51:45.000 INFO:teuthology.orchestra.run.smithi099.stdout:min osd.0 with 8 -> 9 pgs (0.680851 -> 0.765957 * mean) 2024-02-09T17:51:45.000 INFO:teuthology.orchestra.run.smithi099.stdout:max osd.7 with 15 -> 15 pgs (1.2766 -> 1.2766 * mean) 2024-02-09T17:51:45.000 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:51:45.000 INFO:teuthology.orchestra.run.smithi099.stdout:oload 110 2024-02-09T17:51:45.001 INFO:teuthology.orchestra.run.smithi099.stdout:max_change 3 2024-02-09T17:51:45.001 INFO:teuthology.orchestra.run.smithi099.stdout:max_change_osds 4 2024-02-09T17:51:45.001 INFO:teuthology.orchestra.run.smithi099.stdout:average_utilization 0.0237 2024-02-09T17:51:45.001 INFO:teuthology.orchestra.run.smithi099.stdout:overload_utilization 0.0261 2024-02-09T17:51:45.001 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3 weight 0.6593 -> 0.4685 2024-02-09T17:51:45.001 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0 weight 0.7243 -> 0.5845 2024-02-09T17:51:45.001 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11 weight 0.8403 -> 0.6821 2024-02-09T17:51:45.001 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9 weight 0.8394 -> 0.6824 2024-02-09T17:51:45.012 INFO:teuthology.orchestra.run.smithi099.stderr:no change 2024-02-09T17:51:45.012 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:51:45.025 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:45.025 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:51:45.058 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.11 dump_historic_ops 2024-02-09T17:51:45.126 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:51:45.170 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:45.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:45 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:51:45.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:45 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:51:45.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:45 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:51:45.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:45 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:51:45.494 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 17:51:44 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y[39672]: 2024-02-09T17:51:44.995+0000 7f5169e95700 -1 mgr.server reply reply (4) Interrupted system call no change 2024-02-09T17:51:45.524 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:51:45.541 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 79 16 100738 100722 79.6448 38.5625 0.0162843 0.0125513 2024-02-09T17:51:45.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:45 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:51:45.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:45 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:51:45.768 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:51:45.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:45.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:51:45.876 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:51:45.929 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:45.959 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:51:45.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:45.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:51:46.068 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:51:46.111 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:46.219 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:51:46.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:46.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:51:46.328 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:51:46.364 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:46 smithi099 ceph-mon[39420]: from='client.16515 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:51:46.364 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:46 smithi099 ceph-mon[39420]: pgmap v1746: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.0 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s wr, 530 op/s; 1/294864 objects degraded (0.000%); 30502/294864 objects misplaced (10.344%); 14 MiB/s, 231 objects/s recovering 2024-02-09T17:51:46.364 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:46 smithi099 ceph-mon[39420]: osdmap e697: 12 total, 12 up, 12 in 2024-02-09T17:51:46.367 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:46.541 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:2024-02-09T17:51:46.540724+0000 min lat: 0.0011365 max lat: 2.41216 avg lat: 0.0126281 2024-02-09T17:51:46.541 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-09T17:51:46.541 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 80 16 101387 101371 79.1564 40.5625 0.034401 0.0126281 2024-02-09T17:51:46.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:46 smithi179 ceph-mon[43975]: from='client.16515 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:51:46.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:46 smithi179 ceph-mon[43975]: pgmap v1746: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.0 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s wr, 530 op/s; 1/294864 objects degraded (0.000%); 30502/294864 objects misplaced (10.344%); 14 MiB/s, 231 objects/s recovering 2024-02-09T17:51:46.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:46 smithi179 ceph-mon[43975]: osdmap e697: 12 total, 12 up, 12 in 2024-02-09T17:51:46.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:46 smithi162 ceph-mon[44304]: from='client.16515 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:51:46.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:46 smithi162 ceph-mon[44304]: pgmap v1746: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.0 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s wr, 530 op/s; 1/294864 objects degraded (0.000%); 30502/294864 objects misplaced (10.344%); 14 MiB/s, 231 objects/s recovering 2024-02-09T17:51:46.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:46 smithi162 ceph-mon[44304]: osdmap e697: 12 total, 12 up, 12 in 2024-02-09T17:51:46.726 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:51:46.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:46.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:51:46.836 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:51:46.908 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:47.146 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:51:47.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:47.158 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:51:47.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:47 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 1/294864 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:51:47.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:47 smithi099 ceph-mon[39420]: osdmap e698: 12 total, 12 up, 12 in 2024-02-09T17:51:47.259 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:51:47.311 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:47.347 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:51:47.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:47.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:51:47.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:47 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 1/294864 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:51:47.426 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:47 smithi179 ceph-mon[43975]: osdmap e698: 12 total, 12 up, 12 in 2024-02-09T17:51:47.452 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:51:47.535 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:51:47.541 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 81 16 102008 101992 78.6585 38.8125 0.0285651 0.0127063 2024-02-09T17:51:47.547 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:47.619 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:51:47.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:47.627 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:51:47.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:47 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 1/294864 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:51:47.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:47 smithi162 ceph-mon[44304]: osdmap e698: 12 total, 12 up, 12 in 2024-02-09T17:51:47.727 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:51:47.810 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:48.026 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:51:48.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:48.033 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:51:48.134 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:51:48.206 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:48.269 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:51:48.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:48.277 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:51:48.378 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:51:48.420 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:48 smithi099 ceph-mon[39420]: pgmap v1749: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 699 op/s; 1/301527 objects degraded (0.000%); 29265/301527 objects misplaced (9.706%); 21 MiB/s, 334 objects/s recovering 2024-02-09T17:51:48.427 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:48.523 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:51:48.539 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:51:48.541 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 82 16 102624 102608 78.1689 38.5 0.0286417 0.012788 2024-02-09T17:51:48.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:48.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:51:48.645 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:51:48.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:48 smithi179 ceph-mon[43975]: pgmap v1749: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 699 op/s; 1/301527 objects degraded (0.000%); 29265/301527 objects misplaced (9.706%); 21 MiB/s, 334 objects/s recovering 2024-02-09T17:51:48.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:48 smithi162 ceph-mon[44304]: pgmap v1749: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 699 op/s; 1/301527 objects degraded (0.000%); 29265/301527 objects misplaced (9.706%); 21 MiB/s, 334 objects/s recovering 2024-02-09T17:51:48.693 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:48.890 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:51:48.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:48.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:51:49.003 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:51:49.076 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:49.113 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:51:49.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:49.117 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:51:49.219 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:51:49.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:49 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:51:49.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:49 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:51:49.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:49 smithi099 ceph-mon[39420]: osdmap e699: 12 total, 12 up, 12 in 2024-02-09T17:51:49.270 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:49.309 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:51:49.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:49.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:51:49.417 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:51:49.429 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:49 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:51:49.429 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:49 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:51:49.429 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:49 smithi179 ceph-mon[43975]: osdmap e699: 12 total, 12 up, 12 in 2024-02-09T17:51:49.461 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:49.492 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:51:49.495 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:49 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:51:49.495 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:49 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:51:49.496 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:49 smithi162 ceph-mon[44304]: osdmap e699: 12 total, 12 up, 12 in 2024-02-09T17:51:49.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:49.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:51:49.541 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 83 16 103237 103221 77.6888 38.3125 0.00476939 0.0128656 2024-02-09T17:51:49.597 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:51:49.633 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:49.664 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:51:49.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:49.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:51:49.772 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:51:49.811 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:49.850 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:51:49.857 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:51:49.857 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-02-09T17:51:49.857 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-02-09T17:51:49.857 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-02-09T17:51:49.858 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.858 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:106985 5.8 5:10c6e4f9:::benchmark_data_smithi179_115199_object106984:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-02-09T17:51:49.858 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:42:12.106360+0000", 2024-02-09T17:51:49.858 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 577.73787481199997, 2024-02-09T17:51:49.858 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.21701166199999999, 2024-02-09T17:51:49.858 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:51:49.859 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:51:49.859 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:51:49.859 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:51:49.859 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:51:49.859 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 106985 2024-02-09T17:51:49.859 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.860 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:51:49.860 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.860 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:51:49.860 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.106360+0000", 2024-02-09T17:51:49.860 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:49.861 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.861 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.861 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:51:49.861 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.106360+0000", 2024-02-09T17:51:49.861 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:49.861 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.862 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.862 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:51:49.862 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.106361+0000", 2024-02-09T17:51:49.862 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3179999999999999e-06 2024-02-09T17:51:49.862 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.862 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.863 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:51:49.863 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.106388+0000", 2024-02-09T17:51:49.863 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6942999999999999e-05 2024-02-09T17:51:49.863 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.863 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.863 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:51:49.864 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.106399+0000", 2024-02-09T17:51:49.864 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1046999999999999e-05 2024-02-09T17:51:49.864 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.864 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.864 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:51:49.865 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.106406+0000", 2024-02-09T17:51:49.865 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4429999999999997e-06 2024-02-09T17:51:49.865 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.865 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.865 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:49.865 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.106485+0000", 2024-02-09T17:51:49.866 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8347000000000001e-05 2024-02-09T17:51:49.866 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.866 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.866 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:51:49.866 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.106623+0000", 2024-02-09T17:51:49.866 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013791399999999999 2024-02-09T17:51:49.867 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.867 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.867 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,10", 2024-02-09T17:51:49.867 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.106755+0000", 2024-02-09T17:51:49.867 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000132743 2024-02-09T17:51:49.867 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.868 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.868 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:51:49.868 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.114193+0000", 2024-02-09T17:51:49.868 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0074372270000000002 2024-02-09T17:51:49.868 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.869 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.869 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:49.869 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.117105+0000", 2024-02-09T17:51:49.869 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002912591 2024-02-09T17:51:49.869 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.869 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.870 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:49.870 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.323273+0000", 2024-02-09T17:51:49.870 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.20616793999999999 2024-02-09T17:51:49.870 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.870 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.870 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:51:49.871 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.323329+0000", 2024-02-09T17:51:49.871 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5958999999999997e-05 2024-02-09T17:51:49.871 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.871 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.871 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:51:49.871 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.323371+0000", 2024-02-09T17:51:49.872 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2190000000000001e-05 2024-02-09T17:51:49.872 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:49.872 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:51:49.872 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:49.872 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.873 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.873 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:106986 5.8 5:150a5124:::benchmark_data_smithi179_115199_object106985:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-02-09T17:51:49.873 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:42:12.106436+0000", 2024-02-09T17:51:49.873 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 577.73779831100001, 2024-02-09T17:51:49.873 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.217089212, 2024-02-09T17:51:49.874 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:51:49.874 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:51:49.874 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:51:49.874 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:51:49.874 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:51:49.874 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 106986 2024-02-09T17:51:49.875 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.875 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:51:49.875 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.875 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:51:49.875 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.106436+0000", 2024-02-09T17:51:49.876 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:49.876 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.876 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.876 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:51:49.876 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.106436+0000", 2024-02-09T17:51:49.876 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:49.877 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.877 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.877 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:51:49.877 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.106437+0000", 2024-02-09T17:51:49.877 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4700000000000002e-07 2024-02-09T17:51:49.877 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.878 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.878 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:51:49.878 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.106455+0000", 2024-02-09T17:51:49.878 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8627999999999999e-05 2024-02-09T17:51:49.878 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.879 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.879 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:51:49.879 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.106463+0000", 2024-02-09T17:51:49.879 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.909e-06 2024-02-09T17:51:49.879 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.879 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.880 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:51:49.880 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.106469+0000", 2024-02-09T17:51:49.880 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7150000000000003e-06 2024-02-09T17:51:49.880 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.880 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.880 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:49.881 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.110559+0000", 2024-02-09T17:51:49.881 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0040905079999999996 2024-02-09T17:51:49.881 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.881 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.881 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:51:49.881 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.110698+0000", 2024-02-09T17:51:49.882 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013882099999999999 2024-02-09T17:51:49.882 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.882 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.882 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,10", 2024-02-09T17:51:49.882 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.110827+0000", 2024-02-09T17:51:49.882 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000129202 2024-02-09T17:51:49.883 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.883 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.884 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:51:49.884 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.117481+0000", 2024-02-09T17:51:49.884 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.006653155 2024-02-09T17:51:49.884 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.884 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.884 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:49.884 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.119353+0000", 2024-02-09T17:51:49.884 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0018721499999999999 2024-02-09T17:51:49.884 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.884 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.885 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:49.885 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.323420+0000", 2024-02-09T17:51:49.885 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.204067464 2024-02-09T17:51:49.885 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.885 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.885 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:51:49.885 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.323482+0000", 2024-02-09T17:51:49.885 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2100000000000005e-05 2024-02-09T17:51:49.885 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.885 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.885 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:51:49.885 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.323525+0000", 2024-02-09T17:51:49.886 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3112999999999999e-05 2024-02-09T17:51:49.886 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:49.886 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:51:49.886 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:49.886 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.886 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.886 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:106996 5.8 5:13d7f260:::benchmark_data_smithi179_115199_object106995:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-02-09T17:51:49.886 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:42:12.111260+0000", 2024-02-09T17:51:49.886 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 577.73297431900005, 2024-02-09T17:51:49.886 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.21567955999999999, 2024-02-09T17:51:49.886 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:51:49.886 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:51:49.886 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:51:49.886 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:51:49.887 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:51:49.887 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 106996 2024-02-09T17:51:49.887 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.887 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:51:49.887 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.887 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:51:49.887 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.111260+0000", 2024-02-09T17:51:49.887 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:49.887 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.887 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.887 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:51:49.887 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.111260+0000", 2024-02-09T17:51:49.887 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:49.888 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.888 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.888 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:51:49.888 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.111262+0000", 2024-02-09T17:51:49.888 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5290000000000001e-06 2024-02-09T17:51:49.888 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.888 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.888 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:51:49.888 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.111294+0000", 2024-02-09T17:51:49.888 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1847999999999997e-05 2024-02-09T17:51:49.888 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.888 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.888 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:51:49.888 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.111307+0000", 2024-02-09T17:51:49.889 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3039e-05 2024-02-09T17:51:49.889 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.889 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.889 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:51:49.889 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.111315+0000", 2024-02-09T17:51:49.889 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8359999999999998e-06 2024-02-09T17:51:49.889 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.889 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.889 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:49.889 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.114226+0000", 2024-02-09T17:51:49.889 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0029104320000000001 2024-02-09T17:51:49.889 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.890 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.890 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:51:49.890 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.114327+0000", 2024-02-09T17:51:49.890 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000100912 2024-02-09T17:51:49.890 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.890 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.890 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,10", 2024-02-09T17:51:49.890 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.114412+0000", 2024-02-09T17:51:49.890 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5678000000000002e-05 2024-02-09T17:51:49.890 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.890 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.890 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:51:49.890 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.117486+0000", 2024-02-09T17:51:49.891 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0030737569999999999 2024-02-09T17:51:49.891 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.891 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.891 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:49.891 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.119386+0000", 2024-02-09T17:51:49.891 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0018997199999999999 2024-02-09T17:51:49.891 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.891 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.891 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:49.891 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.326833+0000", 2024-02-09T17:51:49.891 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.207446662 2024-02-09T17:51:49.891 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.891 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.892 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:51:49.892 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.326899+0000", 2024-02-09T17:51:49.892 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6000000000000005e-05 2024-02-09T17:51:49.892 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.892 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.892 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:51:49.892 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:12.326940+0000", 2024-02-09T17:51:49.892 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1146999999999998e-05 2024-02-09T17:51:49.892 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:49.892 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:51:49.892 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:49.892 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.892 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.892 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:126288 5.d 5:b690dee0:::benchmark_data_smithi179_115199_object126287:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e501)", 2024-02-09T17:51:49.893 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:42:31.632319+0000", 2024-02-09T17:51:49.893 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 558.21191551000004, 2024-02-09T17:51:49.893 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.22240631899999999, 2024-02-09T17:51:49.893 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:51:49.893 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:51:49.893 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:51:49.893 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:51:49.893 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:51:49.893 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 126288 2024-02-09T17:51:49.893 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.893 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:51:49.893 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.893 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:51:49.894 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:31.632319+0000", 2024-02-09T17:51:49.894 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:49.894 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.894 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.894 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:51:49.894 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:31.632319+0000", 2024-02-09T17:51:49.894 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:49.894 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.894 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.894 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:51:49.894 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:31.632321+0000", 2024-02-09T17:51:49.894 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.731e-06 2024-02-09T17:51:49.894 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.894 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.895 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:51:49.895 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:31.632368+0000", 2024-02-09T17:51:49.895 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7756999999999997e-05 2024-02-09T17:51:49.895 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.895 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.895 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:51:49.895 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:31.632385+0000", 2024-02-09T17:51:49.895 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6109000000000001e-05 2024-02-09T17:51:49.895 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.895 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.895 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:51:49.895 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:31.632393+0000", 2024-02-09T17:51:49.895 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7709999999999995e-06 2024-02-09T17:51:49.895 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.895 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.896 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:49.896 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:31.632459+0000", 2024-02-09T17:51:49.896 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6130999999999994e-05 2024-02-09T17:51:49.896 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.896 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.896 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:51:49.896 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:31.632587+0000", 2024-02-09T17:51:49.896 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000127995 2024-02-09T17:51:49.896 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.896 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.896 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,7", 2024-02-09T17:51:49.896 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:31.632695+0000", 2024-02-09T17:51:49.896 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000107111 2024-02-09T17:51:49.896 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.897 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.897 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:51:49.897 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:31.638171+0000", 2024-02-09T17:51:49.897 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0054763989999999999 2024-02-09T17:51:49.897 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.897 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.897 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:49.897 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:31.639611+0000", 2024-02-09T17:51:49.897 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001439863 2024-02-09T17:51:49.897 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.897 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.897 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:49.897 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:31.854656+0000", 2024-02-09T17:51:49.898 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.21504557699999999 2024-02-09T17:51:49.898 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.898 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.898 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:51:49.898 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:31.854700+0000", 2024-02-09T17:51:49.898 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3149000000000003e-05 2024-02-09T17:51:49.898 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.898 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.898 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:51:49.898 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:42:31.854725+0000", 2024-02-09T17:51:49.898 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5726000000000001e-05 2024-02-09T17:51:49.898 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:49.898 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:51:49.899 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:49.899 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.899 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.899 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25837.0:212109 6.8 6:12d751a2:::benchmark_data_smithi179_126206_object81728:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e641)", 2024-02-09T17:51:49.899 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:48:58.903495+0000", 2024-02-09T17:51:49.899 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 170.94073974599999, 2024-02-09T17:51:49.899 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14948397099999999, 2024-02-09T17:51:49.899 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:51:49.899 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:51:49.899 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:51:49.899 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25837", 2024-02-09T17:51:49.899 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:51:49.899 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 212109 2024-02-09T17:51:49.900 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.900 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:51:49.900 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.900 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:51:49.900 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:58.903495+0000", 2024-02-09T17:51:49.900 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:49.900 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.900 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.900 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:51:49.900 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:58.903495+0000", 2024-02-09T17:51:49.900 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:49.900 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.901 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.901 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:51:49.901 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:58.903497+0000", 2024-02-09T17:51:49.901 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5900000000000002e-06 2024-02-09T17:51:49.901 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.901 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.901 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:51:49.901 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:58.903509+0000", 2024-02-09T17:51:49.901 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1603000000000001e-05 2024-02-09T17:51:49.901 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.902 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.902 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:51:49.902 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:58.903540+0000", 2024-02-09T17:51:49.902 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0956e-05 2024-02-09T17:51:49.902 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.902 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.902 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:51:49.902 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:58.903558+0000", 2024-02-09T17:51:49.902 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8257000000000002e-05 2024-02-09T17:51:49.902 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.902 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.902 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:49.903 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:58.903671+0000", 2024-02-09T17:51:49.903 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011299 2024-02-09T17:51:49.903 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.903 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.903 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:51:49.903 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:58.903788+0000", 2024-02-09T17:51:49.903 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000116396 2024-02-09T17:51:49.903 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.903 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.903 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,7", 2024-02-09T17:51:49.903 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:58.903865+0000", 2024-02-09T17:51:49.904 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7911000000000007e-05 2024-02-09T17:51:49.904 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.904 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.904 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:51:49.904 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:58.907956+0000", 2024-02-09T17:51:49.904 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0040901039999999998 2024-02-09T17:51:49.904 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.904 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.904 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:49.904 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:58.908920+0000", 2024-02-09T17:51:49.905 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000964233 2024-02-09T17:51:49.905 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.905 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.905 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:49.905 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:59.052875+0000", 2024-02-09T17:51:49.905 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14395539099999999 2024-02-09T17:51:49.905 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.905 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.905 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:51:49.905 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:59.052938+0000", 2024-02-09T17:51:49.905 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3196999999999998e-05 2024-02-09T17:51:49.906 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.906 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.906 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:51:49.906 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:59.052979+0000", 2024-02-09T17:51:49.906 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0343000000000003e-05 2024-02-09T17:51:49.906 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:49.906 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:51:49.906 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:49.906 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.906 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.906 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.25837.0:229857 6.3 e649/576 6:c0fa7fc4:::benchmark_data_smithi179_126206_object99476:head v 649'14509, mlcod=649'14507)", 2024-02-09T17:51:49.907 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:49:20.410753+0000", 2024-02-09T17:51:49.907 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 149.43348139299999, 2024-02-09T17:51:49.907 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.088566826000000001, 2024-02-09T17:51:49.907 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:51:49.907 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:51:49.907 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:51:49.907 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.907 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:51:49.907 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:20.410753+0000", 2024-02-09T17:51:49.907 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:49.907 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.907 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.908 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:51:49.908 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:20.410753+0000", 2024-02-09T17:51:49.908 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:49.908 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.908 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.908 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:51:49.908 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:20.410754+0000", 2024-02-09T17:51:49.908 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7899999999999998e-07 2024-02-09T17:51:49.908 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.908 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.908 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:51:49.908 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:20.410759+0000", 2024-02-09T17:51:49.909 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4140000000000002e-06 2024-02-09T17:51:49.909 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.909 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.909 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:51:49.909 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:20.410769+0000", 2024-02-09T17:51:49.909 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0249e-05 2024-02-09T17:51:49.909 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.909 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.909 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:51:49.909 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:20.410777+0000", 2024-02-09T17:51:49.909 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.3849999999999996e-06 2024-02-09T17:51:49.909 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.909 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.909 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:49.910 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:20.410842+0000", 2024-02-09T17:51:49.910 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5413000000000005e-05 2024-02-09T17:51:49.910 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.910 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.910 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:51:49.910 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:20.410865+0000", 2024-02-09T17:51:49.910 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3244e-05 2024-02-09T17:51:49.910 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.910 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.910 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:51:49.910 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:20.499271+0000", 2024-02-09T17:51:49.910 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.088405200000000003 2024-02-09T17:51:49.910 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.910 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.910 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:51:49.910 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:20.499320+0000", 2024-02-09T17:51:49.911 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9242000000000001e-05 2024-02-09T17:51:49.911 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:49.911 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:51:49.911 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:49.911 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.911 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.911 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25837.0:256618 6.8 6:10514f6e:::benchmark_data_smithi179_126206_object126237:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:51:49.911 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:49:55.307984+0000", 2024-02-09T17:51:49.911 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 114.536250577, 2024-02-09T17:51:49.911 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.16505310500000001, 2024-02-09T17:51:49.911 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:51:49.911 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:51:49.911 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:51:49.911 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25837", 2024-02-09T17:51:49.912 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:51:49.912 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 256618 2024-02-09T17:51:49.912 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.912 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:51:49.912 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.912 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:51:49.912 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.307984+0000", 2024-02-09T17:51:49.912 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:49.912 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.912 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.912 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:51:49.912 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.307984+0000", 2024-02-09T17:51:49.912 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:49.912 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.913 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.913 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:51:49.913 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.307985+0000", 2024-02-09T17:51:49.913 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1319999999999999e-06 2024-02-09T17:51:49.913 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.913 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.913 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:51:49.913 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.307990+0000", 2024-02-09T17:51:49.913 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5870000000000003e-06 2024-02-09T17:51:49.913 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.913 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.913 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:51:49.913 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.308003+0000", 2024-02-09T17:51:49.914 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3020000000000001e-05 2024-02-09T17:51:49.914 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.914 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.914 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:51:49.914 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.308010+0000", 2024-02-09T17:51:49.914 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6429999999999995e-06 2024-02-09T17:51:49.914 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.914 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.914 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:49.914 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.308054+0000", 2024-02-09T17:51:49.914 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3430000000000003e-05 2024-02-09T17:51:49.914 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.914 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.914 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:51:49.914 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.308162+0000", 2024-02-09T17:51:49.915 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000107821 2024-02-09T17:51:49.915 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.915 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.915 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,7", 2024-02-09T17:51:49.915 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.308229+0000", 2024-02-09T17:51:49.915 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7150999999999995e-05 2024-02-09T17:51:49.915 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.915 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.915 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:51:49.915 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.312154+0000", 2024-02-09T17:51:49.915 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0039250910000000003 2024-02-09T17:51:49.915 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.915 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.915 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:49.916 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.317879+0000", 2024-02-09T17:51:49.916 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0057249930000000003 2024-02-09T17:51:49.916 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.916 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.916 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:49.916 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.472974+0000", 2024-02-09T17:51:49.916 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.155095488 2024-02-09T17:51:49.916 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.916 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.916 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:51:49.916 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.473011+0000", 2024-02-09T17:51:49.916 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6575000000000001e-05 2024-02-09T17:51:49.917 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.917 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.917 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:51:49.917 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.473037+0000", 2024-02-09T17:51:49.917 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6174e-05 2024-02-09T17:51:49.917 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:49.917 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:51:49.917 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:49.917 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.917 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.917 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25837.0:256647 6.8 6:100d7332:::benchmark_data_smithi179_126206_object126266:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:51:49.917 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:49:55.338986+0000", 2024-02-09T17:51:49.917 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 114.505248339, 2024-02-09T17:51:49.917 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.140859136, 2024-02-09T17:51:49.917 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:51:49.918 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:51:49.918 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:51:49.918 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25837", 2024-02-09T17:51:49.918 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:51:49.918 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 256647 2024-02-09T17:51:49.918 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.918 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:51:49.918 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.918 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:51:49.918 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.338986+0000", 2024-02-09T17:51:49.918 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:49.918 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.918 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.919 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:51:49.919 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.338986+0000", 2024-02-09T17:51:49.919 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:49.919 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.919 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.919 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:51:49.919 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.338988+0000", 2024-02-09T17:51:49.919 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6440000000000001e-06 2024-02-09T17:51:49.919 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.919 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.919 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:51:49.919 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.338995+0000", 2024-02-09T17:51:49.919 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7909999999999999e-06 2024-02-09T17:51:49.919 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.919 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.920 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:51:49.920 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.339009+0000", 2024-02-09T17:51:49.920 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4881e-05 2024-02-09T17:51:49.920 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.920 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.920 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:51:49.920 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.339018+0000", 2024-02-09T17:51:49.920 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3049999999999993e-06 2024-02-09T17:51:49.920 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.920 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.920 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:49.920 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.339100+0000", 2024-02-09T17:51:49.920 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2579999999999999e-05 2024-02-09T17:51:49.921 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.921 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.921 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:51:49.921 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.339204+0000", 2024-02-09T17:51:49.921 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010393800000000001 2024-02-09T17:51:49.921 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.921 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.921 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,7", 2024-02-09T17:51:49.921 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.339281+0000", 2024-02-09T17:51:49.921 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.716e-05 2024-02-09T17:51:49.921 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.921 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.921 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:51:49.921 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.343222+0000", 2024-02-09T17:51:49.922 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0039405159999999998 2024-02-09T17:51:49.922 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.922 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.922 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:49.922 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.343582+0000", 2024-02-09T17:51:49.922 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000359599 2024-02-09T17:51:49.922 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.922 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.922 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:49.922 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.479787+0000", 2024-02-09T17:51:49.922 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.13620548399999999 2024-02-09T17:51:49.922 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.922 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.922 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:51:49.923 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.479822+0000", 2024-02-09T17:51:49.923 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4641999999999998e-05 2024-02-09T17:51:49.923 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.923 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.923 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:51:49.923 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.479845+0000", 2024-02-09T17:51:49.923 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3595999999999999e-05 2024-02-09T17:51:49.923 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:49.923 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:51:49.923 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:49.923 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.923 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.923 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.16440.0:54346 7.5 7:aeab48b6:::benchmark_data_smithi179_135804_object54345:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e673)", 2024-02-09T17:51:49.924 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:50:50.770786+0000", 2024-02-09T17:51:49.924 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 59.073448761000002, 2024-02-09T17:51:49.924 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.089464228000000007, 2024-02-09T17:51:49.924 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:51:49.924 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:51:49.924 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:51:49.924 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.16440", 2024-02-09T17:51:49.924 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:51:49.924 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 54346 2024-02-09T17:51:49.924 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.924 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:51:49.924 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.925 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:51:49.925 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:50.770786+0000", 2024-02-09T17:51:49.925 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:49.925 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.925 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.925 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:51:49.925 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:50.770786+0000", 2024-02-09T17:51:49.925 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:49.925 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.925 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.925 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:51:49.925 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:50.770787+0000", 2024-02-09T17:51:49.925 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.114e-06 2024-02-09T17:51:49.926 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.926 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.926 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:51:49.926 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:50.770812+0000", 2024-02-09T17:51:49.926 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5171e-05 2024-02-09T17:51:49.926 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.926 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.926 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:51:49.926 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:50.770823+0000", 2024-02-09T17:51:49.926 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0749e-05 2024-02-09T17:51:49.926 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.926 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.926 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:51:49.926 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:50.770829+0000", 2024-02-09T17:51:49.927 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3339999999999998e-06 2024-02-09T17:51:49.927 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.927 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.927 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:49.927 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:50.770893+0000", 2024-02-09T17:51:49.927 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3555999999999999e-05 2024-02-09T17:51:49.927 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.927 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.927 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:51:49.927 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:50.771074+0000", 2024-02-09T17:51:49.927 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00018171800000000001 2024-02-09T17:51:49.927 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.927 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.928 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,9", 2024-02-09T17:51:49.928 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:50.771230+0000", 2024-02-09T17:51:49.928 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015531599999999999 2024-02-09T17:51:49.928 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.928 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.928 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:51:49.928 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:50.773318+0000", 2024-02-09T17:51:49.928 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0020887060000000001 2024-02-09T17:51:49.928 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.928 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.928 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:49.928 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:50.773998+0000", 2024-02-09T17:51:49.928 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00067934699999999996 2024-02-09T17:51:49.929 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.929 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.929 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:49.929 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:50.860145+0000", 2024-02-09T17:51:49.929 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.086147267999999999 2024-02-09T17:51:49.929 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.929 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.929 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:51:49.929 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:50.860206+0000", 2024-02-09T17:51:49.929 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0680000000000002e-05 2024-02-09T17:51:49.929 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.929 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.929 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:51:49.929 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:50.860250+0000", 2024-02-09T17:51:49.929 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4268999999999999e-05 2024-02-09T17:51:49.930 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:49.930 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:51:49.930 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:49.930 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.930 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.930 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.16440.0:54367 7.5 7:aeeb03a1:::benchmark_data_smithi179_135804_object54366:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e673)", 2024-02-09T17:51:49.930 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:50:50.780457+0000", 2024-02-09T17:51:49.930 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 59.06377732, 2024-02-09T17:51:49.930 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.085569135000000004, 2024-02-09T17:51:49.930 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:51:49.930 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:51:49.930 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:51:49.931 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.16440", 2024-02-09T17:51:49.931 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:51:49.931 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 54367 2024-02-09T17:51:49.931 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.931 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:51:49.931 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.931 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:51:49.931 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:50.780457+0000", 2024-02-09T17:51:49.931 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:49.931 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.931 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.931 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:51:49.932 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:50.780457+0000", 2024-02-09T17:51:49.932 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:49.932 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.932 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.932 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:51:49.932 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:50.780459+0000", 2024-02-09T17:51:49.932 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7740000000000001e-06 2024-02-09T17:51:49.932 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.932 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.932 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:51:49.932 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:50.780482+0000", 2024-02-09T17:51:49.932 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3227e-05 2024-02-09T17:51:49.933 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.933 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.933 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:51:49.933 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:50.780496+0000", 2024-02-09T17:51:49.933 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3750999999999999e-05 2024-02-09T17:51:49.933 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.933 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.933 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:51:49.933 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:50.780504+0000", 2024-02-09T17:51:49.933 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.9310000000000008e-06 2024-02-09T17:51:49.933 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.933 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.934 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:49.934 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:50.780576+0000", 2024-02-09T17:51:49.934 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2484999999999995e-05 2024-02-09T17:51:49.934 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.934 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.934 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:51:49.934 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:50.780703+0000", 2024-02-09T17:51:49.934 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000126317 2024-02-09T17:51:49.934 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.934 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.934 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,9", 2024-02-09T17:51:49.935 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:50.780826+0000", 2024-02-09T17:51:49.935 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012303699999999999 2024-02-09T17:51:49.935 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.935 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.935 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:51:49.935 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:50.783002+0000", 2024-02-09T17:51:49.935 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0021762550000000002 2024-02-09T17:51:49.935 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.935 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.935 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:49.935 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:50.783854+0000", 2024-02-09T17:51:49.935 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00085168199999999998 2024-02-09T17:51:49.935 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.936 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.936 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:49.936 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:50.865950+0000", 2024-02-09T17:51:49.936 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.082096789000000003 2024-02-09T17:51:49.936 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.936 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.936 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:51:49.936 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:50.865988+0000", 2024-02-09T17:51:49.936 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7372000000000001e-05 2024-02-09T17:51:49.936 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.936 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.937 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:51:49.937 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:50.866026+0000", 2024-02-09T17:51:49.937 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8515e-05 2024-02-09T17:51:49.937 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:49.937 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:51:49.937 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:49.937 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.937 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.937 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.16440.0:61006 7.5 7:ad455a63:::benchmark_data_smithi179_135804_object61005:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e674)", 2024-02-09T17:51:49.937 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:50:55.020008+0000", 2024-02-09T17:51:49.938 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 54.824226424000003, 2024-02-09T17:51:49.938 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.196220321, 2024-02-09T17:51:49.938 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:51:49.938 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:51:49.938 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:51:49.938 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.16440", 2024-02-09T17:51:49.938 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:51:49.938 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 61006 2024-02-09T17:51:49.938 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.938 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:51:49.938 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.938 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:51:49.938 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.020008+0000", 2024-02-09T17:51:49.939 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:49.939 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.939 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.939 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:51:49.939 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.020008+0000", 2024-02-09T17:51:49.939 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:49.939 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.939 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.939 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:51:49.939 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.020010+0000", 2024-02-09T17:51:49.939 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.885e-06 2024-02-09T17:51:49.939 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.940 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.940 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:51:49.940 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.020036+0000", 2024-02-09T17:51:49.940 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.561e-05 2024-02-09T17:51:49.940 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.940 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.940 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:51:49.940 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.020050+0000", 2024-02-09T17:51:49.940 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4465e-05 2024-02-09T17:51:49.940 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.940 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.940 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:51:49.941 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.020058+0000", 2024-02-09T17:51:49.941 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4270000000000008e-06 2024-02-09T17:51:49.941 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.941 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.941 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:49.941 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.020116+0000", 2024-02-09T17:51:49.941 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7695999999999998e-05 2024-02-09T17:51:49.941 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.941 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.941 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:51:49.941 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.020257+0000", 2024-02-09T17:51:49.941 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014065400000000001 2024-02-09T17:51:49.942 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.942 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.942 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,9", 2024-02-09T17:51:49.942 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.020354+0000", 2024-02-09T17:51:49.942 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7031e-05 2024-02-09T17:51:49.942 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.942 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.942 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:49.942 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.022998+0000", 2024-02-09T17:51:49.942 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0026436900000000002 2024-02-09T17:51:49.942 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.943 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.943 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:51:49.943 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.023073+0000", 2024-02-09T17:51:49.943 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5365999999999996e-05 2024-02-09T17:51:49.943 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.943 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.943 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:49.943 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.216125+0000", 2024-02-09T17:51:49.943 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.193051797 2024-02-09T17:51:49.943 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.943 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.943 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:51:49.944 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.216189+0000", 2024-02-09T17:51:49.944 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3959000000000002e-05 2024-02-09T17:51:49.944 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.944 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.944 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:51:49.944 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.216228+0000", 2024-02-09T17:51:49.944 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9740999999999998e-05 2024-02-09T17:51:49.944 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:49.944 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:51:49.944 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:49.944 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.945 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.945 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.16440.0:61016 7.5 7:a0e287c6:::benchmark_data_smithi179_135804_object61015:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e674)", 2024-02-09T17:51:49.945 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:50:55.027713+0000", 2024-02-09T17:51:49.945 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 54.816521059999999, 2024-02-09T17:51:49.945 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.188635202, 2024-02-09T17:51:49.945 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:51:49.945 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:51:49.945 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:51:49.945 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.16440", 2024-02-09T17:51:49.945 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:51:49.945 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 61016 2024-02-09T17:51:49.946 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.946 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:51:49.946 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.946 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:51:49.946 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.027713+0000", 2024-02-09T17:51:49.946 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:49.946 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.946 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.946 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:51:49.946 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.027713+0000", 2024-02-09T17:51:49.946 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:49.947 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.947 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.947 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:51:49.947 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.027715+0000", 2024-02-09T17:51:49.947 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.668e-06 2024-02-09T17:51:49.947 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.947 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.947 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:51:49.947 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.027736+0000", 2024-02-09T17:51:49.947 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0661000000000001e-05 2024-02-09T17:51:49.947 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.947 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.948 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:51:49.948 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.027747+0000", 2024-02-09T17:51:49.948 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1085e-05 2024-02-09T17:51:49.948 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.948 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.948 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:51:49.948 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.027754+0000", 2024-02-09T17:51:49.948 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7920000000000004e-06 2024-02-09T17:51:49.948 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.948 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.948 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:49.949 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.027798+0000", 2024-02-09T17:51:49.949 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3971000000000003e-05 2024-02-09T17:51:49.949 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.949 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.949 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:51:49.949 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.027895+0000", 2024-02-09T17:51:49.949 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7120999999999994e-05 2024-02-09T17:51:49.949 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.949 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.949 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,9", 2024-02-09T17:51:49.949 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.028002+0000", 2024-02-09T17:51:49.949 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000106792 2024-02-09T17:51:49.950 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.950 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.950 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:51:49.950 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.030226+0000", 2024-02-09T17:51:49.950 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0022244550000000002 2024-02-09T17:51:49.950 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.950 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.950 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:49.950 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.030810+0000", 2024-02-09T17:51:49.950 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000583608 2024-02-09T17:51:49.951 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.951 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.951 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:49.951 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.216269+0000", 2024-02-09T17:51:49.951 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.18545963100000001 2024-02-09T17:51:49.951 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.951 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.951 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:51:49.951 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.216316+0000", 2024-02-09T17:51:49.951 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6542000000000003e-05 2024-02-09T17:51:49.951 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.952 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.952 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:51:49.952 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.216349+0000", 2024-02-09T17:51:49.952 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2876000000000002e-05 2024-02-09T17:51:49.952 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:49.952 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:51:49.952 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:49.952 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.952 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.952 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.16440.0:61018 7.5 7:aabe8c5f:::benchmark_data_smithi179_135804_object61017:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e674)", 2024-02-09T17:51:49.952 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:50:55.030684+0000", 2024-02-09T17:51:49.953 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 54.813550640000003, 2024-02-09T17:51:49.953 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.185782171, 2024-02-09T17:51:49.953 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:51:49.953 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:51:49.953 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:51:49.953 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.16440", 2024-02-09T17:51:49.953 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:51:49.953 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 61018 2024-02-09T17:51:49.953 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.953 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:51:49.953 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.953 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:51:49.953 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.030684+0000", 2024-02-09T17:51:49.954 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:49.954 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.954 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.954 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:51:49.954 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.030684+0000", 2024-02-09T17:51:49.954 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:49.954 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.954 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.954 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:51:49.954 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.030685+0000", 2024-02-09T17:51:49.954 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5349999999999999e-06 2024-02-09T17:51:49.955 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.955 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.955 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:51:49.955 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.030712+0000", 2024-02-09T17:51:49.955 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7016e-05 2024-02-09T17:51:49.955 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.955 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.955 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:51:49.955 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.030725+0000", 2024-02-09T17:51:49.955 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2259000000000001e-05 2024-02-09T17:51:49.955 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.956 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.956 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:51:49.956 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.030732+0000", 2024-02-09T17:51:49.956 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7940000000000003e-06 2024-02-09T17:51:49.956 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.956 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.956 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:49.956 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.030827+0000", 2024-02-09T17:51:49.956 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.4452999999999998e-05 2024-02-09T17:51:49.956 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.956 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.957 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:51:49.957 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.030927+0000", 2024-02-09T17:51:49.957 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000100083 2024-02-09T17:51:49.957 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.957 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.957 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,9", 2024-02-09T17:51:49.957 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.031017+0000", 2024-02-09T17:51:49.957 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.9994999999999995e-05 2024-02-09T17:51:49.957 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.957 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.957 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:51:49.957 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.037230+0000", 2024-02-09T17:51:49.957 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0062132890000000003 2024-02-09T17:51:49.957 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.958 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.958 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:49.958 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.037574+0000", 2024-02-09T17:51:49.958 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000343228 2024-02-09T17:51:49.958 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.958 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.958 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:49.958 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.216386+0000", 2024-02-09T17:51:49.958 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.17881296099999999 2024-02-09T17:51:49.958 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.958 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.958 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:51:49.958 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.216437+0000", 2024-02-09T17:51:49.958 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0027000000000001e-05 2024-02-09T17:51:49.959 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.959 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.959 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:51:49.959 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.216466+0000", 2024-02-09T17:51:49.959 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.9530999999999999e-05 2024-02-09T17:51:49.959 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:49.959 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:51:49.959 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:49.959 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.959 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.959 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.16440.0:61020 7.5 7:af0a572b:::benchmark_data_smithi179_135804_object61019:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e674)", 2024-02-09T17:51:49.959 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:50:55.030745+0000", 2024-02-09T17:51:49.959 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 54.813489746999998, 2024-02-09T17:51:49.959 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.18584134399999999, 2024-02-09T17:51:49.959 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:51:49.960 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:51:49.960 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:51:49.960 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.16440", 2024-02-09T17:51:49.960 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:51:49.960 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 61020 2024-02-09T17:51:49.960 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.960 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:51:49.960 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.960 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:51:49.960 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.030745+0000", 2024-02-09T17:51:49.961 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:49.961 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.961 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.961 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:51:49.961 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.030745+0000", 2024-02-09T17:51:49.961 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:49.961 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.961 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.961 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:51:49.961 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.030745+0000", 2024-02-09T17:51:49.961 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7599999999999997e-07 2024-02-09T17:51:49.961 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.961 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.962 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:51:49.962 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.030762+0000", 2024-02-09T17:51:49.962 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6869000000000001e-05 2024-02-09T17:51:49.962 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.962 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.962 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:51:49.962 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.030767+0000", 2024-02-09T17:51:49.962 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9169999999999997e-06 2024-02-09T17:51:49.962 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.962 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.962 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:51:49.963 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.030771+0000", 2024-02-09T17:51:49.963 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2890000000000002e-06 2024-02-09T17:51:49.963 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.963 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.963 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:49.963 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.033207+0000", 2024-02-09T17:51:49.963 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0024362950000000002 2024-02-09T17:51:49.963 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.963 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.963 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:51:49.963 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.033351+0000", 2024-02-09T17:51:49.963 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014458299999999999 2024-02-09T17:51:49.964 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.964 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.964 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,9", 2024-02-09T17:51:49.964 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.033507+0000", 2024-02-09T17:51:49.964 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015522800000000001 2024-02-09T17:51:49.964 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.964 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.964 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:51:49.964 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.037237+0000", 2024-02-09T17:51:49.964 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0037301830000000002 2024-02-09T17:51:49.964 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.964 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.965 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:49.965 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.037639+0000", 2024-02-09T17:51:49.965 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00040180700000000001 2024-02-09T17:51:49.965 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.965 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.965 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:49.965 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.216500+0000", 2024-02-09T17:51:49.965 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.178861613 2024-02-09T17:51:49.965 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.965 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.966 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:51:49.966 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.216547+0000", 2024-02-09T17:51:49.966 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6845999999999998e-05 2024-02-09T17:51:49.966 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.966 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.966 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:51:49.966 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.216586+0000", 2024-02-09T17:51:49.966 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9038000000000001e-05 2024-02-09T17:51:49.966 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:49.966 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:51:49.966 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:49.967 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.967 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.967 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.16440.0:61023 7.5 7:a4a9218b:::benchmark_data_smithi179_135804_object61022:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e674)", 2024-02-09T17:51:49.967 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:50:55.035155+0000", 2024-02-09T17:51:49.967 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 54.809079103999998, 2024-02-09T17:51:49.967 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.18159081199999999, 2024-02-09T17:51:49.967 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:51:49.967 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:51:49.967 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:51:49.967 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.16440", 2024-02-09T17:51:49.967 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:51:49.967 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 61023 2024-02-09T17:51:49.968 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.968 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:51:49.968 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.968 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:51:49.968 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.035155+0000", 2024-02-09T17:51:49.968 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:49.968 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.968 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.968 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:51:49.968 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.035155+0000", 2024-02-09T17:51:49.969 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:49.969 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.969 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.969 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:51:49.969 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.035157+0000", 2024-02-09T17:51:49.969 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.378e-06 2024-02-09T17:51:49.969 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.969 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.969 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:51:49.969 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.035182+0000", 2024-02-09T17:51:49.969 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5622e-05 2024-02-09T17:51:49.969 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.970 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.970 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:51:49.970 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.035198+0000", 2024-02-09T17:51:49.970 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.52e-05 2024-02-09T17:51:49.970 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.970 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.970 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:51:49.970 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.035206+0000", 2024-02-09T17:51:49.970 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3510000000000002e-06 2024-02-09T17:51:49.970 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.970 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.971 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:49.971 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.035372+0000", 2024-02-09T17:51:49.971 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016588599999999999 2024-02-09T17:51:49.971 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.971 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.971 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:51:49.971 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.035480+0000", 2024-02-09T17:51:49.971 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000107996 2024-02-09T17:51:49.971 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.971 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.971 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,9", 2024-02-09T17:51:49.971 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.035577+0000", 2024-02-09T17:51:49.972 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6902000000000002e-05 2024-02-09T17:51:49.972 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.972 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.972 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:51:49.972 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.037304+0000", 2024-02-09T17:51:49.972 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0017267739999999999 2024-02-09T17:51:49.972 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.972 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.972 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:49.972 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.037800+0000", 2024-02-09T17:51:49.972 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000496866 2024-02-09T17:51:49.973 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.973 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.973 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:49.973 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.216654+0000", 2024-02-09T17:51:49.973 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.17885345999999999 2024-02-09T17:51:49.973 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.973 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.973 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:51:49.973 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.216710+0000", 2024-02-09T17:51:49.973 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6313000000000001e-05 2024-02-09T17:51:49.973 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.974 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.974 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:51:49.974 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.216746+0000", 2024-02-09T17:51:49.974 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6063999999999999e-05 2024-02-09T17:51:49.974 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:49.974 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:51:49.974 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:49.974 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.974 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.974 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.16440.0:61029 7.5 7:af86aa1e:::benchmark_data_smithi179_135804_object61028:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e674)", 2024-02-09T17:51:49.974 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:50:55.040695+0000", 2024-02-09T17:51:49.974 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 54.803539868000001, 2024-02-09T17:51:49.975 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.17622264900000001, 2024-02-09T17:51:49.975 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:51:49.975 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:51:49.975 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:51:49.975 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.16440", 2024-02-09T17:51:49.975 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:51:49.975 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 61029 2024-02-09T17:51:49.975 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.975 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:51:49.975 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.975 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:51:49.975 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.040695+0000", 2024-02-09T17:51:49.975 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:49.975 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.976 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.976 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:51:49.976 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.040695+0000", 2024-02-09T17:51:49.976 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:49.976 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.976 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.976 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:51:49.976 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.040696+0000", 2024-02-09T17:51:49.976 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2929999999999999e-06 2024-02-09T17:51:49.976 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.976 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.976 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:51:49.976 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.040717+0000", 2024-02-09T17:51:49.976 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1535999999999998e-05 2024-02-09T17:51:49.977 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.977 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.977 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:51:49.977 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.040731+0000", 2024-02-09T17:51:49.977 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3207000000000001e-05 2024-02-09T17:51:49.977 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.977 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.977 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:51:49.977 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.040738+0000", 2024-02-09T17:51:49.977 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4850000000000003e-06 2024-02-09T17:51:49.977 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.977 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.977 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:49.977 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.040779+0000", 2024-02-09T17:51:49.978 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0958000000000003e-05 2024-02-09T17:51:49.978 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.978 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.978 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:51:49.978 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.040876+0000", 2024-02-09T17:51:49.978 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6836999999999995e-05 2024-02-09T17:51:49.978 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.978 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.978 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,9", 2024-02-09T17:51:49.978 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.040972+0000", 2024-02-09T17:51:49.978 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5735000000000004e-05 2024-02-09T17:51:49.978 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.978 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.978 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:51:49.979 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.042563+0000", 2024-02-09T17:51:49.979 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001591271 2024-02-09T17:51:49.979 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.979 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.979 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:49.979 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.042649+0000", 2024-02-09T17:51:49.979 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5702e-05 2024-02-09T17:51:49.979 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.979 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.979 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:49.979 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.216796+0000", 2024-02-09T17:51:49.979 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.17414739100000001 2024-02-09T17:51:49.979 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.979 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.979 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:51:49.980 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.216864+0000", 2024-02-09T17:51:49.980 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7535999999999998e-05 2024-02-09T17:51:49.980 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.980 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.980 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:51:49.980 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.216917+0000", 2024-02-09T17:51:49.980 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3698e-05 2024-02-09T17:51:49.980 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:49.980 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:51:49.980 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:49.980 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.980 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.980 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.16440.0:87042 7.9 7:94885936:::benchmark_data_smithi179_135804_object87041:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e689)", 2024-02-09T17:51:49.980 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:51:24.297849+0000", 2024-02-09T17:51:49.981 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 25.546385818000001, 2024-02-09T17:51:49.981 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.092246924999999994, 2024-02-09T17:51:49.981 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:51:49.981 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:51:49.981 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:51:49.981 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.16440", 2024-02-09T17:51:49.981 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:51:49.981 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 87042 2024-02-09T17:51:49.981 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.981 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:51:49.981 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.981 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:51:49.981 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.297849+0000", 2024-02-09T17:51:49.981 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:49.981 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.982 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.982 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:51:49.982 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.297849+0000", 2024-02-09T17:51:49.982 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:49.982 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.982 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.982 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:51:49.982 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.297850+0000", 2024-02-09T17:51:49.982 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2279999999999999e-06 2024-02-09T17:51:49.982 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.982 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.982 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:51:49.982 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.297870+0000", 2024-02-09T17:51:49.982 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9919999999999999e-05 2024-02-09T17:51:49.983 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.983 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.983 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:51:49.983 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.297880+0000", 2024-02-09T17:51:49.983 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0316000000000001e-05 2024-02-09T17:51:49.983 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.983 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.983 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:51:49.983 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.297887+0000", 2024-02-09T17:51:49.983 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7700000000000004e-06 2024-02-09T17:51:49.983 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.983 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.983 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:49.983 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.305648+0000", 2024-02-09T17:51:49.983 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0077607489999999999 2024-02-09T17:51:49.984 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.984 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.984 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:51:49.984 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.305760+0000", 2024-02-09T17:51:49.984 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000112188 2024-02-09T17:51:49.984 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.984 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.984 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 8,10", 2024-02-09T17:51:49.984 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.305870+0000", 2024-02-09T17:51:49.984 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010965500000000001 2024-02-09T17:51:49.984 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.984 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.984 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:51:49.984 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.315651+0000", 2024-02-09T17:51:49.985 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0097812980000000008 2024-02-09T17:51:49.985 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.985 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.985 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:49.985 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.386344+0000", 2024-02-09T17:51:49.985 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.070693164000000003 2024-02-09T17:51:49.985 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.985 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.985 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:49.985 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.386394+0000", 2024-02-09T17:51:49.985 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9526e-05 2024-02-09T17:51:49.985 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.985 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.985 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:51:49.986 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.386465+0000", 2024-02-09T17:51:49.986 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1798000000000006e-05 2024-02-09T17:51:49.986 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.986 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.986 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:51:49.986 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.390096+0000", 2024-02-09T17:51:49.986 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.003630313 2024-02-09T17:51:49.986 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:49.986 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:51:49.986 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:49.986 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.986 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.986 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.16440.0:87045 7.5 7:a59d9409:::benchmark_data_smithi179_135804_object87044:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e689)", 2024-02-09T17:51:49.986 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:51:24.306329+0000", 2024-02-09T17:51:49.987 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 25.537905590000001, 2024-02-09T17:51:49.987 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.085360673999999997, 2024-02-09T17:51:49.987 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:51:49.987 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:51:49.987 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:51:49.987 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.16440", 2024-02-09T17:51:49.987 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:51:49.987 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 87045 2024-02-09T17:51:49.987 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.987 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:51:49.987 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.987 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:51:49.988 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.306329+0000", 2024-02-09T17:51:49.988 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:49.988 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.988 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.988 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:51:49.988 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.306329+0000", 2024-02-09T17:51:49.988 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:49.988 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.988 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.988 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:51:49.988 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.306331+0000", 2024-02-09T17:51:49.988 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.72e-06 2024-02-09T17:51:49.989 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.989 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.989 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:51:49.989 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.306368+0000", 2024-02-09T17:51:49.989 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7416999999999998e-05 2024-02-09T17:51:49.989 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.989 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.989 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:51:49.989 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.306382+0000", 2024-02-09T17:51:49.989 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3702e-05 2024-02-09T17:51:49.989 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.989 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.989 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:51:49.990 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.306392+0000", 2024-02-09T17:51:49.990 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0288e-05 2024-02-09T17:51:49.990 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.990 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.990 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:49.990 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.306464+0000", 2024-02-09T17:51:49.990 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2242000000000004e-05 2024-02-09T17:51:49.990 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.990 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.990 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:51:49.990 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.306620+0000", 2024-02-09T17:51:49.990 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015574299999999999 2024-02-09T17:51:49.990 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.990 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.991 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,9", 2024-02-09T17:51:49.991 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.306783+0000", 2024-02-09T17:51:49.991 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016311 2024-02-09T17:51:49.991 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.991 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.991 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:49.991 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.319130+0000", 2024-02-09T17:51:49.991 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.012346500999999999 2024-02-09T17:51:49.991 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.991 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.991 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:49.991 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.319181+0000", 2024-02-09T17:51:49.991 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1564000000000003e-05 2024-02-09T17:51:49.991 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.992 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.992 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:51:49.992 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.388250+0000", 2024-02-09T17:51:49.992 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.069068835999999995 2024-02-09T17:51:49.992 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.992 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.992 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:51:49.992 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.388307+0000", 2024-02-09T17:51:49.992 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6656000000000001e-05 2024-02-09T17:51:49.992 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.992 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.993 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:51:49.993 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.391690+0000", 2024-02-09T17:51:49.993 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0033828949999999999 2024-02-09T17:51:49.993 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:49.993 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:51:49.993 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:49.993 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.993 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.993 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.16440.0:87047 7.9 7:9e518648:::benchmark_data_smithi179_135804_object87046:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e689)", 2024-02-09T17:51:49.993 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:51:24.306412+0000", 2024-02-09T17:51:49.993 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 25.537822691999999, 2024-02-09T17:51:49.993 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.087686572000000004, 2024-02-09T17:51:49.994 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:51:49.994 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:51:49.994 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:51:49.994 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.16440", 2024-02-09T17:51:49.994 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:51:49.994 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 87047 2024-02-09T17:51:49.994 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.994 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:51:49.994 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.994 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:51:49.994 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.306412+0000", 2024-02-09T17:51:49.995 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:49.995 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.995 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.995 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:51:49.995 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.306412+0000", 2024-02-09T17:51:49.995 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:49.995 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.995 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.995 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:51:49.995 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.306412+0000", 2024-02-09T17:51:49.996 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7100000000000002e-07 2024-02-09T17:51:49.996 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.996 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.996 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:51:49.996 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.306433+0000", 2024-02-09T17:51:49.996 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1007999999999999e-05 2024-02-09T17:51:49.996 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.996 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.996 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:51:49.996 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.306442+0000", 2024-02-09T17:51:49.996 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2009999999999999e-06 2024-02-09T17:51:49.997 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.997 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.997 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:51:49.997 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.306448+0000", 2024-02-09T17:51:49.997 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9490000000000001e-06 2024-02-09T17:51:49.997 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.997 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.997 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:49.997 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.309175+0000", 2024-02-09T17:51:49.997 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0027274600000000001 2024-02-09T17:51:49.997 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.998 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.998 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:51:49.998 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.309329+0000", 2024-02-09T17:51:49.998 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015419000000000001 2024-02-09T17:51:49.998 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.998 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.998 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 8,10", 2024-02-09T17:51:49.998 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.309499+0000", 2024-02-09T17:51:49.998 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00017029 2024-02-09T17:51:49.998 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.998 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.998 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:51:49.999 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.315837+0000", 2024-02-09T17:51:49.999 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0063371499999999997 2024-02-09T17:51:49.999 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.999 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.999 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:49.999 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.390167+0000", 2024-02-09T17:51:49.999 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.074330014999999999 2024-02-09T17:51:49.999 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:49.999 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:49.999 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:49.999 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.390217+0000", 2024-02-09T17:51:49.999 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9997999999999998e-05 2024-02-09T17:51:50.000 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:50.000 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:50.000 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:51:50.000 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.390296+0000", 2024-02-09T17:51:50.000 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8906000000000007e-05 2024-02-09T17:51:50.000 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:50.000 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:50.000 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:51:50.000 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.394098+0000", 2024-02-09T17:51:50.000 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0038028340000000002 2024-02-09T17:51:50.000 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:50.001 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:51:50.001 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:50.001 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:50.001 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:50.001 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.16440.0:87055 7.9 7:9ea8d80f:::benchmark_data_smithi179_135804_object87054:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e689)", 2024-02-09T17:51:50.001 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:51:24.316409+0000", 2024-02-09T17:51:50.001 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 25.527825966000002, 2024-02-09T17:51:50.001 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.085232191999999998, 2024-02-09T17:51:50.001 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:51:50.001 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:51:50.002 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:51:50.002 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.16440", 2024-02-09T17:51:50.002 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:51:50.002 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 87055 2024-02-09T17:51:50.002 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:50.002 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:51:50.002 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:50.002 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:51:50.002 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.316409+0000", 2024-02-09T17:51:50.002 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:50.002 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:50.003 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:50.003 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:51:50.003 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.316409+0000", 2024-02-09T17:51:50.003 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:51:50.003 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:50.003 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:50.003 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:51:50.003 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.316409+0000", 2024-02-09T17:51:50.003 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.54e-07 2024-02-09T17:51:50.003 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:50.003 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:50.003 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:51:50.004 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.316432+0000", 2024-02-09T17:51:50.004 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2405999999999999e-05 2024-02-09T17:51:50.004 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:50.004 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:50.004 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:51:50.004 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.316441+0000", 2024-02-09T17:51:50.004 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.3230000000000004e-06 2024-02-09T17:51:50.004 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:50.004 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:50.004 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:51:50.004 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.316449+0000", 2024-02-09T17:51:50.005 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6620000000000005e-06 2024-02-09T17:51:50.005 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:50.005 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:50.005 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:51:50.005 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.319408+0000", 2024-02-09T17:51:50.005 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0029587139999999999 2024-02-09T17:51:50.005 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:50.005 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:50.005 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:51:50.005 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.383130+0000", 2024-02-09T17:51:50.005 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.063722013999999993 2024-02-09T17:51:50.005 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:50.006 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:50.006 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 8,10", 2024-02-09T17:51:50.006 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.383328+0000", 2024-02-09T17:51:50.006 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00019874199999999999 2024-02-09T17:51:50.006 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:50.006 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:50.006 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:51:50.006 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.397594+0000", 2024-02-09T17:51:50.006 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014265464 2024-02-09T17:51:50.006 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:50.006 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:50.006 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:50.006 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.397642+0000", 2024-02-09T17:51:50.007 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8296e-05 2024-02-09T17:51:50.007 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:50.007 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:50.007 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:51:50.007 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.397692+0000", 2024-02-09T17:51:50.007 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0179999999999997e-05 2024-02-09T17:51:50.007 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:50.007 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:50.007 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:51:50.007 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.397768+0000", 2024-02-09T17:51:50.007 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5768999999999999e-05 2024-02-09T17:51:50.007 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:51:50.007 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:51:50.008 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:51:50.008 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:51:24.401641+0000", 2024-02-09T17:51:50.008 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.003872768 2024-02-09T17:51:50.008 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:50.008 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:51:50.008 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:50.008 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:51:50.008 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:51:50.008 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:51:50.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:50.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:51:50.110 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:51:50.150 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:50.186 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:51:50.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:50.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:51:50.263 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is set 2024-02-09T17:51:50.298 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:51:50.376 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:50.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:50 smithi162 ceph-mon[44304]: pgmap v1751: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.2 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 35 MiB/s wr, 564 op/s; 1/303780 objects degraded (0.000%); 28953/303780 objects misplaced (9.531%); 18 MiB/s, 289 objects/s recovering 2024-02-09T17:51:50.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:50 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1007396160' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:51:50.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:50 smithi099 ceph-mon[39420]: pgmap v1751: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.2 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 35 MiB/s wr, 564 op/s; 1/303780 objects degraded (0.000%); 28953/303780 objects misplaced (9.531%); 18 MiB/s, 289 objects/s recovering 2024-02-09T17:51:50.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:50 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1007396160' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:51:50.546 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 84 16 103877 103861 77.2358 40 0.0240924 0.0129411 2024-02-09T17:51:50.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:50 smithi179 ceph-mon[43975]: pgmap v1751: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.2 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 35 MiB/s wr, 564 op/s; 1/303780 objects degraded (0.000%); 28953/303780 objects misplaced (9.531%); 18 MiB/s, 289 objects/s recovering 2024-02-09T17:51:50.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:50 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1007396160' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:51:51.249 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:51 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:51:51.249 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:51 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:51:51.249 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:51 smithi099 ceph-mon[39420]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:51:51.249 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:51 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1007396160' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:51:51.250 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:51 smithi099 ceph-mon[39420]: osdmap e700: 12 total, 12 up, 12 in 2024-02-09T17:51:51.250 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:51 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:51:51.250 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:51 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:51:51.257 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:51:51.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:51.263 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:51:51.364 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:51:51.418 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:51.546 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 85 16 104544 104528 76.8177 41.6875 0.0188366 0.0130128 2024-02-09T17:51:51.625 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:51:51.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:51.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:51:51.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:51 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:51:51.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:51 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:51:51.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:51 smithi179 ceph-mon[43975]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:51:51.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:51 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1007396160' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:51:51.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:51 smithi179 ceph-mon[43975]: osdmap e700: 12 total, 12 up, 12 in 2024-02-09T17:51:51.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:51 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:51:51.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:51 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:51:51.681 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:51 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:51:51.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:51 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:51:51.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:51 smithi162 ceph-mon[44304]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:51:51.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:51 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1007396160' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:51:51.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:51 smithi162 ceph-mon[44304]: osdmap e700: 12 total, 12 up, 12 in 2024-02-09T17:51:51.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:51 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:51:51.683 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:51 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:51:51.732 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:51:51.754 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] 2024-02-09T17:51:51.755 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:51:51.755 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-02-09T17:51:51.755 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-09T17:51:51.811 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:52.066 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:51:52.108 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:51:52.160 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:51:52.192 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:51:52.231 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:51:52.246 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:51:52.266 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:51:52.282 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.8 dump_ops_in_flight 2024-02-09T17:51:52.290 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:51:52.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:52.299 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:51:52.299 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:51:52.324 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:51:52.370 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:51:52.400 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:51:52.414 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:51:52.431 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:52.471 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:51:52.516 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:51:52.522 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:51:52.522 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:51:52.522 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:51:52.522 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:51:52.522 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:51:52.522 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:51:52.523 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:51:52.523 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:51:52.523 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:51:52.523 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:51:52.523 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:51:52.523 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:51:52.547 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:52 smithi099 ceph-mon[39420]: pgmap v1753: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 704 op/s; 1/307851 objects degraded (0.000%); 28120/307851 objects misplaced (9.134%); 25 MiB/s, 407 objects/s recovering 2024-02-09T17:51:52.547 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:52 smithi099 ceph-mon[39420]: osdmap e701: 12 total, 12 up, 12 in 2024-02-09T17:51:52.547 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:52 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 1/307851 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:51:52.547 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:52 smithi099 ceph-mon[39420]: osdmap e702: 12 total, 12 up, 12 in 2024-02-09T17:51:52.547 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:52 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/295830132' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:51:52.549 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 86 16 105157 105141 76.3702 38.3125 0.00757616 0.0130882 2024-02-09T17:51:52.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:52 smithi179 ceph-mon[43975]: pgmap v1753: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 704 op/s; 1/307851 objects degraded (0.000%); 28120/307851 objects misplaced (9.134%); 25 MiB/s, 407 objects/s recovering 2024-02-09T17:51:52.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:52 smithi179 ceph-mon[43975]: osdmap e701: 12 total, 12 up, 12 in 2024-02-09T17:51:52.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:52 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 1/307851 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:51:52.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:52 smithi179 ceph-mon[43975]: osdmap e702: 12 total, 12 up, 12 in 2024-02-09T17:51:52.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:52 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/295830132' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:51:52.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:52 smithi162 ceph-mon[44304]: pgmap v1753: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 704 op/s; 1/307851 objects degraded (0.000%); 28120/307851 objects misplaced (9.134%); 25 MiB/s, 407 objects/s recovering 2024-02-09T17:51:52.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:52 smithi162 ceph-mon[44304]: osdmap e701: 12 total, 12 up, 12 in 2024-02-09T17:51:52.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:52 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 1/307851 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:51:52.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:52 smithi162 ceph-mon[44304]: osdmap e702: 12 total, 12 up, 12 in 2024-02-09T17:51:52.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:52 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/295830132' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:51:52.760 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:51:53.390 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset noscrub 2024-02-09T17:51:53.549 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 87 15 105793 105778 75.9491 39.8125 0.0293705 0.0131608 2024-02-09T17:51:54.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:53 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:51:54.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:53 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:51:54.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:53 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:51:54.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:53 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:51:54.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:53 smithi179 ceph-mon[43975]: osdmap e703: 12 total, 12 up, 12 in 2024-02-09T17:51:54.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:53 smithi179 ceph-mon[43975]: pgmap v1757: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 747 op/s; 1/312747 objects degraded (0.000%); 27505/312747 objects misplaced (8.795%); 23 MiB/s, 361 objects/s recovering 2024-02-09T17:51:54.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:53 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:51:54.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:53 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:51:54.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:53 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:51:54.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:53 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:51:54.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:53 smithi162 ceph-mon[44304]: osdmap e703: 12 total, 12 up, 12 in 2024-02-09T17:51:54.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:53 smithi162 ceph-mon[44304]: pgmap v1757: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 747 op/s; 1/312747 objects degraded (0.000%); 27505/312747 objects misplaced (8.795%); 23 MiB/s, 361 objects/s recovering 2024-02-09T17:51:54.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:53 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:51:54.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:53 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:51:54.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:53 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:51:54.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:53 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:51:54.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:53 smithi099 ceph-mon[39420]: osdmap e703: 12 total, 12 up, 12 in 2024-02-09T17:51:54.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:53 smithi099 ceph-mon[39420]: pgmap v1757: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 747 op/s; 1/312747 objects degraded (0.000%); 27505/312747 objects misplaced (8.795%); 23 MiB/s, 361 objects/s recovering 2024-02-09T17:51:54.438 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:51:54.441 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:54.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:51:54.543 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:51:54.547 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 88 16 106462 106446 75.5606 41.75 0.0204996 0.0132296 2024-02-09T17:51:54.579 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:55.044 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:51:55.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:55.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:51:55.152 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:51:55.219 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:55.548 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 89 16 107148 107132 75.193 42.875 0.0369403 0.0132937 2024-02-09T17:51:56.395 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:56 smithi179 ceph-mon[43975]: pgmap v1758: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 39 MiB/s wr, 623 op/s; 1/313932 objects degraded (0.000%); 27312/313932 objects misplaced (8.700%); 21 MiB/s, 341 objects/s recovering 2024-02-09T17:51:56.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:56 smithi162 ceph-mon[44304]: pgmap v1758: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 39 MiB/s wr, 623 op/s; 1/313932 objects degraded (0.000%); 27312/313932 objects misplaced (8.700%); 21 MiB/s, 341 objects/s recovering 2024-02-09T17:51:56.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:56 smithi099 ceph-mon[39420]: pgmap v1758: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 39 MiB/s wr, 623 op/s; 1/313932 objects degraded (0.000%); 27312/313932 objects misplaced (8.700%); 21 MiB/s, 341 objects/s recovering 2024-02-09T17:51:56.549 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Total time run: 90.0186 2024-02-09T17:51:56.549 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Total writes made: 107788 2024-02-09T17:51:56.550 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Write size: 65536 2024-02-09T17:51:56.550 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Object size: 65536 2024-02-09T17:51:56.550 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Bandwidth (MB/sec): 74.8373 2024-02-09T17:51:56.550 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Stddev Bandwidth: 50.0756 2024-02-09T17:51:56.550 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Max bandwidth (MB/sec): 255.5 2024-02-09T17:51:56.551 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Min bandwidth (MB/sec): 0 2024-02-09T17:51:56.551 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Average IOPS: 1197 2024-02-09T17:51:56.551 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Stddev IOPS: 801.217 2024-02-09T17:51:56.551 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Max IOPS: 4088 2024-02-09T17:51:56.551 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Min IOPS: 0 2024-02-09T17:51:56.551 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Average Latency(s): 0.013358 2024-02-09T17:51:56.551 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Stddev Latency(s): 0.0330286 2024-02-09T17:51:56.551 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Max latency(s): 2.41216 2024-02-09T17:51:56.552 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Min latency(s): 0.0011365 2024-02-09T17:51:56.554 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Cleaning up (deleting benchmark objects) 2024-02-09T17:51:57.381 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:57 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 1/313932 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:51:57.381 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:57 smithi099 ceph-mon[39420]: osdmap e704: 12 total, 12 up, 12 in 2024-02-09T17:51:57.387 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:51:57.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:57.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:51:57.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:57 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 1/313932 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:51:57.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:57 smithi162 ceph-mon[44304]: osdmap e704: 12 total, 12 up, 12 in 2024-02-09T17:51:57.428 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:57 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 1/313932 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:51:57.428 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:57 smithi179 ceph-mon[43975]: osdmap e704: 12 total, 12 up, 12 in 2024-02-09T17:51:57.498 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:51:57.578 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T17:51:57.578 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:51:57.602 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:57.639 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:51:57.647 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:57.648 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:51:57.749 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:51:57.796 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:57.816 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:51:58.122 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:51:58.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:58.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:51:58.228 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:51:58.262 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:58.358 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:51:58.362 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:51:58.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:51:58.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:51:58 smithi179 ceph-mon[43975]: pgmap v1760: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.5 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 46 MiB/s wr, 738 op/s; 1/320481 objects degraded (0.000%); 25942/320481 objects misplaced (8.095%); 24 MiB/s, 377 objects/s recovering 2024-02-09T17:51:58.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:51:58 smithi162 ceph-mon[44304]: pgmap v1760: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.5 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 46 MiB/s wr, 738 op/s; 1/320481 objects degraded (0.000%); 25942/320481 objects misplaced (8.095%); 24 MiB/s, 377 objects/s recovering 2024-02-09T17:51:58.464 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:51:58.478 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:51:58 smithi099 ceph-mon[39420]: pgmap v1760: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.5 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 46 MiB/s wr, 738 op/s; 1/320481 objects degraded (0.000%); 25942/320481 objects misplaced (8.095%); 24 MiB/s, 377 objects/s recovering 2024-02-09T17:51:58.504 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:51:59.415 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:51:59.600 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.8 dump_blocked_ops 2024-02-09T17:52:00.168 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:52:00.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:00.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:52:00.222 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:52:00.272 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:52:00.322 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:00.355 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:00 smithi099 ceph-mon[39420]: pgmap v1761: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.5 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 619 op/s; 1/321654 objects degraded (0.000%); 25607/321654 objects misplaced (7.961%); 17 MiB/s, 279 objects/s recovering 2024-02-09T17:52:00.396 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:52:00.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:00.404 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:52:00.427 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:00 smithi179 ceph-mon[43975]: pgmap v1761: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.5 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 619 op/s; 1/321654 objects degraded (0.000%); 25607/321654 objects misplaced (7.961%); 17 MiB/s, 279 objects/s recovering 2024-02-09T17:52:00.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:00 smithi162 ceph-mon[44304]: pgmap v1761: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.5 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 619 op/s; 1/321654 objects degraded (0.000%); 25607/321654 objects misplaced (7.961%); 17 MiB/s, 279 objects/s recovering 2024-02-09T17:52:00.504 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:52:00.541 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:00.617 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:52:00.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:00.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:52:00.721 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:52:00.791 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:01.455 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:52:01.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:01.460 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:52:01.477 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:52:01.561 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:52:01.591 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:01.624 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:52:01.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:01.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:52:01.646 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:52:01.646 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:52:01.647 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 704, 2024-02-09T17:52:01.647 INFO:teuthology.orchestra.run.smithi099.stdout: "fsid": "a69c870a-c76e-11ee-95b7-87774f69a715", 2024-02-09T17:52:01.647 INFO:teuthology.orchestra.run.smithi099.stdout: "created": "2024-02-09T17:15:26.113606+0000", 2024-02-09T17:52:01.647 INFO:teuthology.orchestra.run.smithi099.stdout: "modified": "2024-02-09T17:51:56.787414+0000", 2024-02-09T17:52:01.647 INFO:teuthology.orchestra.run.smithi099.stdout: "last_up_change": "2024-02-09T17:28:17.711044+0000", 2024-02-09T17:52:01.647 INFO:teuthology.orchestra.run.smithi099.stdout: "last_in_change": "2024-02-09T17:43:24.138763+0000", 2024-02-09T17:52:01.647 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-09T17:52:01.647 INFO:teuthology.orchestra.run.smithi099.stdout: "flags_num": 5806080, 2024-02-09T17:52:01.647 INFO:teuthology.orchestra.run.smithi099.stdout: "flags_set": [ 2024-02-09T17:52:01.647 INFO:teuthology.orchestra.run.smithi099.stdout: "nodeep-scrub", 2024-02-09T17:52:01.647 INFO:teuthology.orchestra.run.smithi099.stdout: "noscrub", 2024-02-09T17:52:01.647 INFO:teuthology.orchestra.run.smithi099.stdout: "pglog_hardlimit", 2024-02-09T17:52:01.647 INFO:teuthology.orchestra.run.smithi099.stdout: "purged_snapdirs", 2024-02-09T17:52:01.648 INFO:teuthology.orchestra.run.smithi099.stdout: "recovery_deletes", 2024-02-09T17:52:01.648 INFO:teuthology.orchestra.run.smithi099.stdout: "sortbitwise" 2024-02-09T17:52:01.648 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T17:52:01.648 INFO:teuthology.orchestra.run.smithi099.stdout: "crush_version": 27, 2024-02-09T17:52:01.648 INFO:teuthology.orchestra.run.smithi099.stdout: "full_ratio": 0.94999998807907104, 2024-02-09T17:52:01.648 INFO:teuthology.orchestra.run.smithi099.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-09T17:52:01.648 INFO:teuthology.orchestra.run.smithi099.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-09T17:52:01.648 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_snapshot": "", 2024-02-09T17:52:01.648 INFO:teuthology.orchestra.run.smithi099.stdout: "pool_max": 7, 2024-02-09T17:52:01.648 INFO:teuthology.orchestra.run.smithi099.stdout: "max_osd": 12, 2024-02-09T17:52:01.648 INFO:teuthology.orchestra.run.smithi099.stdout: "require_min_compat_client": "luminous", 2024-02-09T17:52:01.648 INFO:teuthology.orchestra.run.smithi099.stdout: "min_compat_client": "luminous", 2024-02-09T17:52:01.649 INFO:teuthology.orchestra.run.smithi099.stdout: "require_osd_release": "squid", 2024-02-09T17:52:01.649 INFO:teuthology.orchestra.run.smithi099.stdout: "allow_crimson": false, 2024-02-09T17:52:01.649 INFO:teuthology.orchestra.run.smithi099.stdout: "pools": [ 2024-02-09T17:52:01.649 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.649 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 1, 2024-02-09T17:52:01.649 INFO:teuthology.orchestra.run.smithi099.stdout: "pool_name": ".mgr", 2024-02-09T17:52:01.649 INFO:teuthology.orchestra.run.smithi099.stdout: "create_time": "2024-02-09T17:18:56.552286+0000", 2024-02-09T17:52:01.649 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 1, 2024-02-09T17:52:01.649 INFO:teuthology.orchestra.run.smithi099.stdout: "flags_names": "hashpspool", 2024-02-09T17:52:01.649 INFO:teuthology.orchestra.run.smithi099.stdout: "type": 1, 2024-02-09T17:52:01.649 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-02-09T17:52:01.649 INFO:teuthology.orchestra.run.smithi099.stdout: "min_size": 2, 2024-02-09T17:52:01.649 INFO:teuthology.orchestra.run.smithi099.stdout: "crush_rule": 0, 2024-02-09T17:52:01.650 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_count": 0, 2024-02-09T17:52:01.650 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_target": 0, 2024-02-09T17:52:01.650 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_barrier": 0, 2024-02-09T17:52:01.650 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-09T17:52:01.650 INFO:teuthology.orchestra.run.smithi099.stdout: "object_hash": 2, 2024-02-09T17:52:01.650 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_autoscale_mode": "off", 2024-02-09T17:52:01.650 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num": 31, 2024-02-09T17:52:01.650 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_placement_num": 31, 2024-02-09T17:52:01.650 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_placement_num_target": 31, 2024-02-09T17:52:01.650 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num_target": 31, 2024-02-09T17:52:01.650 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num_pending": 31, 2024-02-09T17:52:01.650 INFO:teuthology.orchestra.run.smithi099.stdout: "last_pg_merge_meta": { 2024-02-09T17:52:01.650 INFO:teuthology.orchestra.run.smithi099.stdout: "source_pgid": "0.0", 2024-02-09T17:52:01.650 INFO:teuthology.orchestra.run.smithi099.stdout: "ready_epoch": 0, 2024-02-09T17:52:01.651 INFO:teuthology.orchestra.run.smithi099.stdout: "last_epoch_started": 0, 2024-02-09T17:52:01.651 INFO:teuthology.orchestra.run.smithi099.stdout: "last_epoch_clean": 0, 2024-02-09T17:52:01.651 INFO:teuthology.orchestra.run.smithi099.stdout: "source_version": "0'0", 2024-02-09T17:52:01.651 INFO:teuthology.orchestra.run.smithi099.stdout: "target_version": "0'0" 2024-02-09T17:52:01.651 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.651 INFO:teuthology.orchestra.run.smithi099.stdout: "last_change": "544", 2024-02-09T17:52:01.651 INFO:teuthology.orchestra.run.smithi099.stdout: "last_force_op_resend": "0", 2024-02-09T17:52:01.651 INFO:teuthology.orchestra.run.smithi099.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-09T17:52:01.651 INFO:teuthology.orchestra.run.smithi099.stdout: "last_force_op_resend_preluminous": "514", 2024-02-09T17:52:01.651 INFO:teuthology.orchestra.run.smithi099.stdout: "auid": 0, 2024-02-09T17:52:01.651 INFO:teuthology.orchestra.run.smithi099.stdout: "snap_mode": "selfmanaged", 2024-02-09T17:52:01.651 INFO:teuthology.orchestra.run.smithi099.stdout: "snap_seq": 0, 2024-02-09T17:52:01.651 INFO:teuthology.orchestra.run.smithi099.stdout: "snap_epoch": 0, 2024-02-09T17:52:01.652 INFO:teuthology.orchestra.run.smithi099.stdout: "pool_snaps": [], 2024-02-09T17:52:01.652 INFO:teuthology.orchestra.run.smithi099.stdout: "removed_snaps": "[]", 2024-02-09T17:52:01.652 INFO:teuthology.orchestra.run.smithi099.stdout: "quota_max_bytes": 0, 2024-02-09T17:52:01.652 INFO:teuthology.orchestra.run.smithi099.stdout: "quota_max_objects": 0, 2024-02-09T17:52:01.652 INFO:teuthology.orchestra.run.smithi099.stdout: "tiers": [], 2024-02-09T17:52:01.652 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_of": -1, 2024-02-09T17:52:01.652 INFO:teuthology.orchestra.run.smithi099.stdout: "read_tier": -1, 2024-02-09T17:52:01.652 INFO:teuthology.orchestra.run.smithi099.stdout: "write_tier": -1, 2024-02-09T17:52:01.652 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_mode": "none", 2024-02-09T17:52:01.652 INFO:teuthology.orchestra.run.smithi099.stdout: "target_max_bytes": 0, 2024-02-09T17:52:01.652 INFO:teuthology.orchestra.run.smithi099.stdout: "target_max_objects": 0, 2024-02-09T17:52:01.652 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-09T17:52:01.652 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-09T17:52:01.653 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-09T17:52:01.653 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_min_flush_age": 0, 2024-02-09T17:52:01.653 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_min_evict_age": 0, 2024-02-09T17:52:01.653 INFO:teuthology.orchestra.run.smithi099.stdout: "erasure_code_profile": "", 2024-02-09T17:52:01.653 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_params": { 2024-02-09T17:52:01.653 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "none" 2024-02-09T17:52:01.653 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.653 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_period": 0, 2024-02-09T17:52:01.653 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_count": 0, 2024-02-09T17:52:01.653 INFO:teuthology.orchestra.run.smithi099.stdout: "use_gmt_hitset": true, 2024-02-09T17:52:01.653 INFO:teuthology.orchestra.run.smithi099.stdout: "min_read_recency_for_promote": 0, 2024-02-09T17:52:01.653 INFO:teuthology.orchestra.run.smithi099.stdout: "min_write_recency_for_promote": 0, 2024-02-09T17:52:01.653 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_grade_decay_rate": 0, 2024-02-09T17:52:01.654 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_search_last_n": 0, 2024-02-09T17:52:01.654 INFO:teuthology.orchestra.run.smithi099.stdout: "grade_table": [], 2024-02-09T17:52:01.654 INFO:teuthology.orchestra.run.smithi099.stdout: "stripe_width": 0, 2024-02-09T17:52:01.654 INFO:teuthology.orchestra.run.smithi099.stdout: "expected_num_objects": 0, 2024-02-09T17:52:01.654 INFO:teuthology.orchestra.run.smithi099.stdout: "fast_read": false, 2024-02-09T17:52:01.654 INFO:teuthology.orchestra.run.smithi099.stdout: "options": { 2024-02-09T17:52:01.654 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num_max": 32, 2024-02-09T17:52:01.654 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num_min": 1 2024-02-09T17:52:01.654 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.654 INFO:teuthology.orchestra.run.smithi099.stdout: "application_metadata": { 2024-02-09T17:52:01.654 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": {} 2024-02-09T17:52:01.654 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.654 INFO:teuthology.orchestra.run.smithi099.stdout: "read_balance": { 2024-02-09T17:52:01.655 INFO:teuthology.orchestra.run.smithi099.stdout: "score_type": "Fair distribution", 2024-02-09T17:52:01.655 INFO:teuthology.orchestra.run.smithi099.stdout: "score_acting": 5.9899997711181641, 2024-02-09T17:52:01.655 INFO:teuthology.orchestra.run.smithi099.stdout: "score_stable": 1.7799999713897705, 2024-02-09T17:52:01.655 INFO:teuthology.orchestra.run.smithi099.stdout: "optimal_score": 1.0900000333786011, 2024-02-09T17:52:01.655 INFO:teuthology.orchestra.run.smithi099.stdout: "raw_score_acting": 6.5300002098083496, 2024-02-09T17:52:01.655 INFO:teuthology.orchestra.run.smithi099.stdout: "raw_score_stable": 1.940000057220459, 2024-02-09T17:52:01.655 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity_weighted": 0.78079599142074585, 2024-02-09T17:52:01.655 INFO:teuthology.orchestra.run.smithi099.stdout: "average_primary_affinity": 0.85000002384185791, 2024-02-09T17:52:01.655 INFO:teuthology.orchestra.run.smithi099.stdout: "average_primary_affinity_weighted": 0.77999997138977051, 2024-02-09T17:52:01.655 INFO:teuthology.orchestra.run.smithi099.stdout: "error_message": "pool 1 has acting primaries on OSD(s) with primary affinity 0, read balance score is not accurate" 2024-02-09T17:52:01.655 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.655 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.655 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.656 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 7, 2024-02-09T17:52:01.656 INFO:teuthology.orchestra.run.smithi099.stdout: "pool_name": "unique_pool_5", 2024-02-09T17:52:01.656 INFO:teuthology.orchestra.run.smithi099.stdout: "create_time": "2024-02-09T17:50:17.290198+0000", 2024-02-09T17:52:01.656 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 1, 2024-02-09T17:52:01.656 INFO:teuthology.orchestra.run.smithi099.stdout: "flags_names": "hashpspool", 2024-02-09T17:52:01.656 INFO:teuthology.orchestra.run.smithi099.stdout: "type": 1, 2024-02-09T17:52:01.656 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-02-09T17:52:01.656 INFO:teuthology.orchestra.run.smithi099.stdout: "min_size": 2, 2024-02-09T17:52:01.656 INFO:teuthology.orchestra.run.smithi099.stdout: "crush_rule": 0, 2024-02-09T17:52:01.656 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_count": 0, 2024-02-09T17:52:01.656 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_target": 0, 2024-02-09T17:52:01.656 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_barrier": 0, 2024-02-09T17:52:01.656 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-09T17:52:01.656 INFO:teuthology.orchestra.run.smithi099.stdout: "object_hash": 2, 2024-02-09T17:52:01.657 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_autoscale_mode": "off", 2024-02-09T17:52:01.657 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num": 16, 2024-02-09T17:52:01.657 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_placement_num": 16, 2024-02-09T17:52:01.657 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_placement_num_target": 16, 2024-02-09T17:52:01.657 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num_target": 16, 2024-02-09T17:52:01.657 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num_pending": 16, 2024-02-09T17:52:01.657 INFO:teuthology.orchestra.run.smithi099.stdout: "last_pg_merge_meta": { 2024-02-09T17:52:01.657 INFO:teuthology.orchestra.run.smithi099.stdout: "source_pgid": "0.0", 2024-02-09T17:52:01.657 INFO:teuthology.orchestra.run.smithi099.stdout: "ready_epoch": 0, 2024-02-09T17:52:01.657 INFO:teuthology.orchestra.run.smithi099.stdout: "last_epoch_started": 0, 2024-02-09T17:52:01.657 INFO:teuthology.orchestra.run.smithi099.stdout: "last_epoch_clean": 0, 2024-02-09T17:52:01.657 INFO:teuthology.orchestra.run.smithi099.stdout: "source_version": "0'0", 2024-02-09T17:52:01.657 INFO:teuthology.orchestra.run.smithi099.stdout: "target_version": "0'0" 2024-02-09T17:52:01.657 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.658 INFO:teuthology.orchestra.run.smithi099.stdout: "last_change": "665", 2024-02-09T17:52:01.658 INFO:teuthology.orchestra.run.smithi099.stdout: "last_force_op_resend": "0", 2024-02-09T17:52:01.658 INFO:teuthology.orchestra.run.smithi099.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-09T17:52:01.658 INFO:teuthology.orchestra.run.smithi099.stdout: "last_force_op_resend_preluminous": "0", 2024-02-09T17:52:01.658 INFO:teuthology.orchestra.run.smithi099.stdout: "auid": 0, 2024-02-09T17:52:01.658 INFO:teuthology.orchestra.run.smithi099.stdout: "snap_mode": "selfmanaged", 2024-02-09T17:52:01.658 INFO:teuthology.orchestra.run.smithi099.stdout: "snap_seq": 0, 2024-02-09T17:52:01.658 INFO:teuthology.orchestra.run.smithi099.stdout: "snap_epoch": 0, 2024-02-09T17:52:01.658 INFO:teuthology.orchestra.run.smithi099.stdout: "pool_snaps": [], 2024-02-09T17:52:01.658 INFO:teuthology.orchestra.run.smithi099.stdout: "removed_snaps": "[]", 2024-02-09T17:52:01.658 INFO:teuthology.orchestra.run.smithi099.stdout: "quota_max_bytes": 0, 2024-02-09T17:52:01.658 INFO:teuthology.orchestra.run.smithi099.stdout: "quota_max_objects": 0, 2024-02-09T17:52:01.658 INFO:teuthology.orchestra.run.smithi099.stdout: "tiers": [], 2024-02-09T17:52:01.658 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_of": -1, 2024-02-09T17:52:01.659 INFO:teuthology.orchestra.run.smithi099.stdout: "read_tier": -1, 2024-02-09T17:52:01.659 INFO:teuthology.orchestra.run.smithi099.stdout: "write_tier": -1, 2024-02-09T17:52:01.659 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_mode": "none", 2024-02-09T17:52:01.659 INFO:teuthology.orchestra.run.smithi099.stdout: "target_max_bytes": 0, 2024-02-09T17:52:01.659 INFO:teuthology.orchestra.run.smithi099.stdout: "target_max_objects": 0, 2024-02-09T17:52:01.659 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-09T17:52:01.659 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-09T17:52:01.659 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-09T17:52:01.659 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_min_flush_age": 0, 2024-02-09T17:52:01.659 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_min_evict_age": 0, 2024-02-09T17:52:01.659 INFO:teuthology.orchestra.run.smithi099.stdout: "erasure_code_profile": "", 2024-02-09T17:52:01.659 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_params": { 2024-02-09T17:52:01.659 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "none" 2024-02-09T17:52:01.659 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.660 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_period": 0, 2024-02-09T17:52:01.660 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_count": 0, 2024-02-09T17:52:01.660 INFO:teuthology.orchestra.run.smithi099.stdout: "use_gmt_hitset": true, 2024-02-09T17:52:01.660 INFO:teuthology.orchestra.run.smithi099.stdout: "min_read_recency_for_promote": 0, 2024-02-09T17:52:01.660 INFO:teuthology.orchestra.run.smithi099.stdout: "min_write_recency_for_promote": 0, 2024-02-09T17:52:01.660 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_grade_decay_rate": 0, 2024-02-09T17:52:01.660 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_search_last_n": 0, 2024-02-09T17:52:01.660 INFO:teuthology.orchestra.run.smithi099.stdout: "grade_table": [], 2024-02-09T17:52:01.660 INFO:teuthology.orchestra.run.smithi099.stdout: "stripe_width": 0, 2024-02-09T17:52:01.660 INFO:teuthology.orchestra.run.smithi099.stdout: "expected_num_objects": 0, 2024-02-09T17:52:01.660 INFO:teuthology.orchestra.run.smithi099.stdout: "fast_read": false, 2024-02-09T17:52:01.660 INFO:teuthology.orchestra.run.smithi099.stdout: "options": {}, 2024-02-09T17:52:01.660 INFO:teuthology.orchestra.run.smithi099.stdout: "application_metadata": { 2024-02-09T17:52:01.660 INFO:teuthology.orchestra.run.smithi099.stdout: "rados": {} 2024-02-09T17:52:01.661 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.661 INFO:teuthology.orchestra.run.smithi099.stdout: "read_balance": { 2024-02-09T17:52:01.661 INFO:teuthology.orchestra.run.smithi099.stdout: "score_type": "Fair distribution", 2024-02-09T17:52:01.661 INFO:teuthology.orchestra.run.smithi099.stdout: "score_acting": 2.059999942779541, 2024-02-09T17:52:01.661 INFO:teuthology.orchestra.run.smithi099.stdout: "score_stable": 2.059999942779541, 2024-02-09T17:52:01.661 INFO:teuthology.orchestra.run.smithi099.stdout: "optimal_score": 1.0900000333786011, 2024-02-09T17:52:01.661 INFO:teuthology.orchestra.run.smithi099.stdout: "raw_score_acting": 2.25, 2024-02-09T17:52:01.661 INFO:teuthology.orchestra.run.smithi099.stdout: "raw_score_stable": 2.25, 2024-02-09T17:52:01.661 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity_weighted": 0.78079599142074585, 2024-02-09T17:52:01.661 INFO:teuthology.orchestra.run.smithi099.stdout: "average_primary_affinity": 0.85000002384185791, 2024-02-09T17:52:01.661 INFO:teuthology.orchestra.run.smithi099.stdout: "average_primary_affinity_weighted": 0.77999997138977051 2024-02-09T17:52:01.661 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.661 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.661 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T17:52:01.662 INFO:teuthology.orchestra.run.smithi099.stdout: "osds": [ 2024-02-09T17:52:01.662 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.662 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 0, 2024-02-09T17:52:01.662 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "21739aa7-e278-45da-9a3f-867385f01baf", 2024-02-09T17:52:01.662 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:52:01.662 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:52:01.662 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 0.7243499755859375, 2024-02-09T17:52:01.662 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 0.609619140625, 2024-02-09T17:52:01.662 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:52:01.662 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:52:01.662 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 8, 2024-02-09T17:52:01.662 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 684, 2024-02-09T17:52:01.662 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:52:01.662 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:52:01.662 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:52:01.663 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:52:01.663 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.663 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:52:01.663 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6802", 2024-02-09T17:52:01.663 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T17:52:01.663 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.663 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.663 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:52:01.663 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6803", 2024-02-09T17:52:01.663 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T17:52:01.663 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.663 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.663 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.664 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:52:01.664 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:52:01.664 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.664 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:52:01.664 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6804", 2024-02-09T17:52:01.664 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T17:52:01.664 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.664 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.664 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:52:01.664 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6805", 2024-02-09T17:52:01.664 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T17:52:01.664 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.664 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.664 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.665 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:52:01.665 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:52:01.665 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.665 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:52:01.665 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6808", 2024-02-09T17:52:01.665 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T17:52:01.665 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.665 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.665 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:52:01.665 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6809", 2024-02-09T17:52:01.665 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T17:52:01.665 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.665 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.665 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.666 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:52:01.666 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:52:01.666 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.666 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:52:01.666 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6806", 2024-02-09T17:52:01.666 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T17:52:01.666 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.666 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.666 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:52:01.666 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6807", 2024-02-09T17:52:01.666 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T17:52:01.666 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.666 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.666 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.667 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.99:6803/3859868923", 2024-02-09T17:52:01.667 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.99:6805/3859868923", 2024-02-09T17:52:01.667 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.99:6809/3859868923", 2024-02-09T17:52:01.667 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.99:6807/3859868923", 2024-02-09T17:52:01.667 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:52:01.667 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:52:01.667 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:52:01.667 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.667 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.667 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.667 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 1, 2024-02-09T17:52:01.667 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "8eab347c-9065-4de1-93ae-e2eb7e11cab7", 2024-02-09T17:52:01.667 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:52:01.667 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:52:01.668 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:52:01.668 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T17:52:01.668 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:52:01.668 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:52:01.668 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 13, 2024-02-09T17:52:01.668 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 684, 2024-02-09T17:52:01.668 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:52:01.668 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:52:01.668 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:52:01.668 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:52:01.668 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.668 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:52:01.668 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6810", 2024-02-09T17:52:01.668 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T17:52:01.668 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.669 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.669 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:52:01.669 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6811", 2024-02-09T17:52:01.669 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T17:52:01.669 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.669 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.669 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.669 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:52:01.669 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:52:01.669 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.669 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:52:01.669 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6812", 2024-02-09T17:52:01.669 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T17:52:01.670 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.670 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.670 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:52:01.670 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6813", 2024-02-09T17:52:01.670 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T17:52:01.670 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.670 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.670 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.670 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:52:01.670 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:52:01.670 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.670 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:52:01.670 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6816", 2024-02-09T17:52:01.670 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T17:52:01.670 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.671 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.671 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:52:01.671 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6817", 2024-02-09T17:52:01.671 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T17:52:01.671 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.671 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.671 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.671 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:52:01.671 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:52:01.671 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.671 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:52:01.671 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6814", 2024-02-09T17:52:01.671 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T17:52:01.671 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.671 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.672 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:52:01.672 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6815", 2024-02-09T17:52:01.672 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T17:52:01.672 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.672 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.672 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.672 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.99:6811/3868328452", 2024-02-09T17:52:01.672 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.99:6813/3868328452", 2024-02-09T17:52:01.672 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.99:6817/3868328452", 2024-02-09T17:52:01.672 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.99:6815/3868328452", 2024-02-09T17:52:01.672 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:52:01.672 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:52:01.672 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:52:01.673 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.673 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.673 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.673 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 2, 2024-02-09T17:52:01.673 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "575f4257-7e34-485e-963c-2607b69e1220", 2024-02-09T17:52:01.673 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:52:01.673 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:52:01.673 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:52:01.673 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 0.3601837158203125, 2024-02-09T17:52:01.673 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:52:01.673 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:52:01.673 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 18, 2024-02-09T17:52:01.673 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 661, 2024-02-09T17:52:01.673 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:52:01.673 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:52:01.674 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:52:01.674 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:52:01.674 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.674 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:52:01.674 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6818", 2024-02-09T17:52:01.674 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T17:52:01.674 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.674 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.674 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:52:01.674 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6819", 2024-02-09T17:52:01.674 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T17:52:01.674 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.674 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.674 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.674 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:52:01.675 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:52:01.675 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.675 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:52:01.675 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6820", 2024-02-09T17:52:01.675 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T17:52:01.675 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.675 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.675 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:52:01.675 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6821", 2024-02-09T17:52:01.675 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T17:52:01.675 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.675 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.675 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.675 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:52:01.675 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:52:01.676 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.676 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:52:01.676 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6824", 2024-02-09T17:52:01.676 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T17:52:01.676 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.676 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.676 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:52:01.676 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6825", 2024-02-09T17:52:01.676 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T17:52:01.676 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.676 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.676 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.676 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:52:01.676 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:52:01.676 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.677 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:52:01.677 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6822", 2024-02-09T17:52:01.677 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T17:52:01.677 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.677 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.677 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:52:01.677 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6823", 2024-02-09T17:52:01.677 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T17:52:01.677 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.677 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.677 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.677 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.99:6819/537506237", 2024-02-09T17:52:01.677 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.99:6821/537506237", 2024-02-09T17:52:01.677 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.99:6825/537506237", 2024-02-09T17:52:01.678 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.99:6823/537506237", 2024-02-09T17:52:01.678 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:52:01.678 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:52:01.678 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:52:01.678 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.678 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.678 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.678 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 3, 2024-02-09T17:52:01.678 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "1a520bcf-81bb-4f1d-80fb-11e98008e429", 2024-02-09T17:52:01.678 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:52:01.678 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:52:01.678 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 0.659332275390625, 2024-02-09T17:52:01.678 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 0.8014984130859375, 2024-02-09T17:52:01.679 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:52:01.679 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:52:01.679 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 27, 2024-02-09T17:52:01.679 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 684, 2024-02-09T17:52:01.679 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:52:01.679 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:52:01.679 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:52:01.679 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:52:01.679 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.679 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:52:01.679 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6826", 2024-02-09T17:52:01.679 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 762784793 2024-02-09T17:52:01.679 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.679 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.679 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:52:01.680 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6827", 2024-02-09T17:52:01.680 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 762784793 2024-02-09T17:52:01.680 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.680 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.680 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.680 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:52:01.680 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:52:01.680 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.680 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:52:01.680 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6828", 2024-02-09T17:52:01.680 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 762784793 2024-02-09T17:52:01.680 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.680 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.680 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:52:01.680 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6829", 2024-02-09T17:52:01.681 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 762784793 2024-02-09T17:52:01.681 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.681 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.681 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.681 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:52:01.681 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:52:01.681 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.681 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:52:01.681 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6832", 2024-02-09T17:52:01.681 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 762784793 2024-02-09T17:52:01.681 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.681 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.681 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:52:01.682 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6833", 2024-02-09T17:52:01.682 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 762784793 2024-02-09T17:52:01.682 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.682 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.682 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.682 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:52:01.682 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:52:01.682 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.682 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:52:01.682 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6830", 2024-02-09T17:52:01.682 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 762784793 2024-02-09T17:52:01.682 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.682 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.683 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:52:01.683 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6831", 2024-02-09T17:52:01.683 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 762784793 2024-02-09T17:52:01.683 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.683 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.683 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.683 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.99:6827/762784793", 2024-02-09T17:52:01.683 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.99:6829/762784793", 2024-02-09T17:52:01.683 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.99:6833/762784793", 2024-02-09T17:52:01.683 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.99:6831/762784793", 2024-02-09T17:52:01.683 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:52:01.683 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:52:01.683 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:52:01.683 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.684 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.684 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.684 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 4, 2024-02-09T17:52:01.684 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "4617c2fe-c9d8-40cd-887b-303e3215bdf8", 2024-02-09T17:52:01.684 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:52:01.684 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:52:01.684 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:52:01.684 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T17:52:01.684 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:52:01.684 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:52:01.684 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 32, 2024-02-09T17:52:01.684 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 682, 2024-02-09T17:52:01.684 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:52:01.685 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:52:01.685 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:52:01.685 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:52:01.685 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.685 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:52:01.685 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6800", 2024-02-09T17:52:01.685 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T17:52:01.685 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.685 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.685 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:52:01.685 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6801", 2024-02-09T17:52:01.685 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T17:52:01.685 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.685 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.686 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.686 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:52:01.686 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:52:01.686 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.686 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:52:01.686 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6802", 2024-02-09T17:52:01.686 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T17:52:01.686 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.686 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.686 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:52:01.686 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6803", 2024-02-09T17:52:01.686 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T17:52:01.686 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.686 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.687 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.687 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:52:01.687 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:52:01.687 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.687 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:52:01.687 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6806", 2024-02-09T17:52:01.687 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T17:52:01.687 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.687 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.687 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:52:01.687 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6807", 2024-02-09T17:52:01.687 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T17:52:01.687 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.688 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.688 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.688 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:52:01.688 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:52:01.688 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.688 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:52:01.688 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6804", 2024-02-09T17:52:01.688 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T17:52:01.688 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.688 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.688 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:52:01.688 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6805", 2024-02-09T17:52:01.688 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T17:52:01.689 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.689 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.689 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.689 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.162:6801/4178676504", 2024-02-09T17:52:01.689 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.162:6803/4178676504", 2024-02-09T17:52:01.689 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.162:6807/4178676504", 2024-02-09T17:52:01.689 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.162:6805/4178676504", 2024-02-09T17:52:01.689 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:52:01.689 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:52:01.689 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:52:01.689 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.689 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.689 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.689 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 5, 2024-02-09T17:52:01.690 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "da98e3f3-173e-46fe-be8b-1ccbe8b0dabc", 2024-02-09T17:52:01.690 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:52:01.690 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:52:01.690 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:52:01.690 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 0, 2024-02-09T17:52:01.690 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:52:01.690 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:52:01.690 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 40, 2024-02-09T17:52:01.690 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 576, 2024-02-09T17:52:01.690 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:52:01.690 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:52:01.690 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:52:01.690 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:52:01.690 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.691 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:52:01.691 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6808", 2024-02-09T17:52:01.691 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T17:52:01.691 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.691 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.691 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:52:01.691 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6809", 2024-02-09T17:52:01.691 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T17:52:01.691 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.691 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.691 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.691 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:52:01.691 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:52:01.692 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.692 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:52:01.692 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6810", 2024-02-09T17:52:01.692 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T17:52:01.692 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.692 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.692 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:52:01.692 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6811", 2024-02-09T17:52:01.692 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T17:52:01.692 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.692 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.692 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.692 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:52:01.692 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:52:01.693 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.693 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:52:01.693 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6814", 2024-02-09T17:52:01.693 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T17:52:01.693 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.693 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.693 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:52:01.693 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6815", 2024-02-09T17:52:01.693 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T17:52:01.693 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.693 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.693 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.693 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:52:01.693 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:52:01.693 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.694 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:52:01.694 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6812", 2024-02-09T17:52:01.694 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T17:52:01.694 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.694 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.694 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:52:01.694 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6813", 2024-02-09T17:52:01.694 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T17:52:01.694 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.694 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.694 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.694 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.162:6809/1489371339", 2024-02-09T17:52:01.694 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.162:6811/1489371339", 2024-02-09T17:52:01.694 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.162:6815/1489371339", 2024-02-09T17:52:01.695 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.162:6813/1489371339", 2024-02-09T17:52:01.695 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:52:01.695 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:52:01.695 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:52:01.695 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.695 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.695 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.695 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 6, 2024-02-09T17:52:01.695 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "ff50382d-5ddf-4cda-bb0c-ff055feeaa40", 2024-02-09T17:52:01.695 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:52:01.695 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:52:01.695 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:52:01.695 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T17:52:01.695 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:52:01.696 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:52:01.696 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 49, 2024-02-09T17:52:01.696 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 684, 2024-02-09T17:52:01.696 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:52:01.696 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:52:01.696 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:52:01.696 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:52:01.696 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.696 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:52:01.696 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6816", 2024-02-09T17:52:01.696 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T17:52:01.696 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.696 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.696 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:52:01.697 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6817", 2024-02-09T17:52:01.697 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T17:52:01.697 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.697 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.697 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.697 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:52:01.697 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:52:01.697 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.697 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:52:01.697 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6818", 2024-02-09T17:52:01.697 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T17:52:01.697 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.697 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.698 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:52:01.698 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6819", 2024-02-09T17:52:01.698 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T17:52:01.698 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.698 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.698 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.698 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:52:01.698 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:52:01.698 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.698 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:52:01.698 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6822", 2024-02-09T17:52:01.698 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T17:52:01.698 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.698 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.698 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:52:01.699 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6823", 2024-02-09T17:52:01.699 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T17:52:01.699 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.699 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.699 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.699 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:52:01.699 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:52:01.699 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.699 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:52:01.699 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6820", 2024-02-09T17:52:01.699 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T17:52:01.699 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.699 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.699 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:52:01.699 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6821", 2024-02-09T17:52:01.700 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T17:52:01.700 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.700 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.700 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.700 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.162:6817/821169663", 2024-02-09T17:52:01.700 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.162:6819/821169663", 2024-02-09T17:52:01.700 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.162:6823/821169663", 2024-02-09T17:52:01.700 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.162:6821/821169663", 2024-02-09T17:52:01.700 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:52:01.700 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:52:01.700 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:52:01.700 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.700 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.700 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.701 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 7, 2024-02-09T17:52:01.701 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "a82984dc-5e22-424b-a446-ae2ee71eee19", 2024-02-09T17:52:01.701 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:52:01.701 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:52:01.701 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:52:01.701 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 0.5982513427734375, 2024-02-09T17:52:01.701 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:52:01.701 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:52:01.701 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 57, 2024-02-09T17:52:01.701 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 661, 2024-02-09T17:52:01.701 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:52:01.701 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:52:01.701 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:52:01.701 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:52:01.701 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.702 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:52:01.702 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6824", 2024-02-09T17:52:01.702 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T17:52:01.702 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.702 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.702 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:52:01.702 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6825", 2024-02-09T17:52:01.702 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T17:52:01.702 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.702 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.702 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.702 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:52:01.702 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:52:01.702 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.702 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:52:01.703 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6826", 2024-02-09T17:52:01.703 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T17:52:01.703 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.703 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.703 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:52:01.703 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6827", 2024-02-09T17:52:01.703 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T17:52:01.703 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.703 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.703 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.703 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:52:01.703 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:52:01.703 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.703 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:52:01.703 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6830", 2024-02-09T17:52:01.704 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T17:52:01.704 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.704 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.704 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:52:01.704 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6831", 2024-02-09T17:52:01.704 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T17:52:01.704 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.704 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.704 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.704 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:52:01.704 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:52:01.704 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.704 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:52:01.704 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6828", 2024-02-09T17:52:01.705 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T17:52:01.705 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.705 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.705 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:52:01.705 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6829", 2024-02-09T17:52:01.705 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T17:52:01.705 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.705 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.705 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.705 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.162:6825/965264369", 2024-02-09T17:52:01.705 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.162:6827/965264369", 2024-02-09T17:52:01.705 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.162:6831/965264369", 2024-02-09T17:52:01.705 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.162:6829/965264369", 2024-02-09T17:52:01.705 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:52:01.705 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:52:01.706 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:52:01.706 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.706 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.706 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.706 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 8, 2024-02-09T17:52:01.706 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "63f9d79f-0400-44dc-b388-df00fefb5820", 2024-02-09T17:52:01.706 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:52:01.706 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:52:01.706 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:52:01.706 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T17:52:01.706 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 66, 2024-02-09T17:52:01.706 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 145, 2024-02-09T17:52:01.706 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 171, 2024-02-09T17:52:01.706 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 684, 2024-02-09T17:52:01.707 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 146, 2024-02-09T17:52:01.707 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:52:01.707 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:52:01.707 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:52:01.707 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.707 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:52:01.707 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6800", 2024-02-09T17:52:01.707 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T17:52:01.707 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.707 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.707 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:52:01.707 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6801", 2024-02-09T17:52:01.707 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T17:52:01.707 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.708 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.708 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.708 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:52:01.708 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:52:01.708 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.708 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:52:01.708 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6802", 2024-02-09T17:52:01.708 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T17:52:01.708 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.708 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.708 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:52:01.708 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6803", 2024-02-09T17:52:01.708 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T17:52:01.708 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.709 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.709 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.709 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:52:01.709 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:52:01.709 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.709 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:52:01.709 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6806", 2024-02-09T17:52:01.709 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T17:52:01.709 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.709 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.709 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:52:01.709 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6807", 2024-02-09T17:52:01.709 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T17:52:01.709 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.710 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.710 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.710 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:52:01.710 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:52:01.710 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.710 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:52:01.710 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6804", 2024-02-09T17:52:01.710 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T17:52:01.710 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.710 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.710 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:52:01.710 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6805", 2024-02-09T17:52:01.710 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T17:52:01.710 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.711 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.711 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.711 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.179:6801/2558294495", 2024-02-09T17:52:01.711 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.179:6803/2558294495", 2024-02-09T17:52:01.711 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.179:6807/2558294495", 2024-02-09T17:52:01.711 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.179:6805/2558294495", 2024-02-09T17:52:01.711 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:52:01.711 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:52:01.711 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:52:01.711 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.711 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.711 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.711 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 9, 2024-02-09T17:52:01.711 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "7d995854-7328-4dfc-b1a9-0edca8d74f46", 2024-02-09T17:52:01.712 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:52:01.712 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:52:01.712 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 0.8393707275390625, 2024-02-09T17:52:01.712 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T17:52:01.712 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:52:01.712 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:52:01.712 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 77, 2024-02-09T17:52:01.712 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 661, 2024-02-09T17:52:01.712 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:52:01.712 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:52:01.712 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:52:01.712 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:52:01.712 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.713 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:52:01.713 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6808", 2024-02-09T17:52:01.713 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T17:52:01.713 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.713 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.713 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:52:01.713 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6809", 2024-02-09T17:52:01.713 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T17:52:01.713 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.713 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.713 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.713 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:52:01.713 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:52:01.713 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.714 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:52:01.714 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6810", 2024-02-09T17:52:01.714 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T17:52:01.714 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.714 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.714 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:52:01.714 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6811", 2024-02-09T17:52:01.714 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T17:52:01.714 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.714 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.714 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.714 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:52:01.714 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:52:01.715 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.715 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:52:01.715 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6814", 2024-02-09T17:52:01.715 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T17:52:01.715 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.715 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.715 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:52:01.715 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6815", 2024-02-09T17:52:01.715 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T17:52:01.715 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.715 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.715 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.715 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:52:01.715 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:52:01.716 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.716 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:52:01.716 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6812", 2024-02-09T17:52:01.716 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T17:52:01.716 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.716 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.716 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:52:01.716 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6813", 2024-02-09T17:52:01.716 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T17:52:01.716 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.716 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.716 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.716 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.179:6809/504427183", 2024-02-09T17:52:01.716 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.179:6811/504427183", 2024-02-09T17:52:01.716 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.179:6815/504427183", 2024-02-09T17:52:01.717 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.179:6813/504427183", 2024-02-09T17:52:01.717 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:52:01.717 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:52:01.717 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:52:01.717 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.717 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.717 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.717 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 10, 2024-02-09T17:52:01.717 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "151fbedd-08e6-46e6-a4e5-a14248399f61", 2024-02-09T17:52:01.717 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:52:01.717 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:52:01.717 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:52:01.717 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T17:52:01.718 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:52:01.718 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:52:01.718 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 84, 2024-02-09T17:52:01.718 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 684, 2024-02-09T17:52:01.718 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:52:01.718 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:52:01.718 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:52:01.718 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:52:01.718 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.718 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:52:01.718 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6816", 2024-02-09T17:52:01.718 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T17:52:01.718 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.718 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.719 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:52:01.719 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6817", 2024-02-09T17:52:01.719 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T17:52:01.719 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.719 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.719 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.719 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:52:01.719 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:52:01.719 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.719 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:52:01.719 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6818", 2024-02-09T17:52:01.719 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T17:52:01.719 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.719 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.720 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:52:01.720 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6819", 2024-02-09T17:52:01.720 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T17:52:01.720 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.720 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.720 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.720 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:52:01.720 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:52:01.720 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.720 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:52:01.720 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6822", 2024-02-09T17:52:01.720 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T17:52:01.720 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.720 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.721 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:52:01.721 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6823", 2024-02-09T17:52:01.721 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T17:52:01.721 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.721 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.721 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.721 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:52:01.721 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:52:01.721 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.721 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:52:01.721 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6820", 2024-02-09T17:52:01.721 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T17:52:01.721 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.721 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.721 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:52:01.722 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6821", 2024-02-09T17:52:01.722 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T17:52:01.722 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.722 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.722 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.722 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.179:6817/2403345164", 2024-02-09T17:52:01.722 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.179:6819/2403345164", 2024-02-09T17:52:01.722 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.179:6823/2403345164", 2024-02-09T17:52:01.722 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.179:6821/2403345164", 2024-02-09T17:52:01.722 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:52:01.722 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:52:01.722 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:52:01.722 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.723 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.723 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.723 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 11, 2024-02-09T17:52:01.723 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "6d3a3ad8-417a-483f-991c-52549dbe7d73", 2024-02-09T17:52:01.723 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:52:01.723 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:52:01.723 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 0.8402862548828125, 2024-02-09T17:52:01.723 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T17:52:01.723 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:52:01.723 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:52:01.723 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 90, 2024-02-09T17:52:01.723 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 682, 2024-02-09T17:52:01.723 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:52:01.724 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:52:01.724 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:52:01.724 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:52:01.724 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.724 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:52:01.724 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6824", 2024-02-09T17:52:01.724 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T17:52:01.724 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.724 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.724 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:52:01.724 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6825", 2024-02-09T17:52:01.724 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T17:52:01.724 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.724 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.725 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.725 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:52:01.725 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:52:01.725 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.725 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:52:01.725 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6826", 2024-02-09T17:52:01.725 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T17:52:01.725 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.725 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.725 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:52:01.725 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6827", 2024-02-09T17:52:01.725 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T17:52:01.725 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.725 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.725 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.726 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:52:01.726 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:52:01.726 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.726 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:52:01.726 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6830", 2024-02-09T17:52:01.726 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T17:52:01.726 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.726 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.726 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:52:01.726 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6831", 2024-02-09T17:52:01.726 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T17:52:01.726 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.726 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.726 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.726 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:52:01.727 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:52:01.727 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.727 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:52:01.727 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6828", 2024-02-09T17:52:01.727 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T17:52:01.727 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.727 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.727 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:52:01.727 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6829", 2024-02-09T17:52:01.727 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T17:52:01.727 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.727 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.727 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.727 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.179:6825/2899402373", 2024-02-09T17:52:01.727 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.179:6827/2899402373", 2024-02-09T17:52:01.727 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.179:6831/2899402373", 2024-02-09T17:52:01.728 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.179:6829/2899402373", 2024-02-09T17:52:01.728 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:52:01.728 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:52:01.728 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:52:01.728 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.728 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.728 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T17:52:01.728 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_xinfo": [ 2024-02-09T17:52:01.728 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.728 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 0, 2024-02-09T17:52:01.728 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:52:01.728 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:52:01.728 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:52:01.729 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:52:01.729 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:52:01.729 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:51:49.767472+0000", 2024-02-09T17:52:01.729 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:52:01.729 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.729 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.729 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 1, 2024-02-09T17:52:01.729 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:52:01.729 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:52:01.729 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:52:01.729 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:52:01.729 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:52:01.729 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:51:51.916964+0000", 2024-02-09T17:52:01.729 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:52:01.730 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.730 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.730 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 2, 2024-02-09T17:52:01.730 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:52:01.730 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:52:01.730 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:52:01.730 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:52:01.730 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:52:01.730 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:51:29.925627+0000", 2024-02-09T17:52:01.730 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:52:01.730 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.730 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.730 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 3, 2024-02-09T17:52:01.730 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:52:01.731 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:52:01.731 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:52:01.731 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:52:01.731 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:52:01.731 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:51:47.591375+0000", 2024-02-09T17:52:01.731 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:52:01.731 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.731 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.731 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 4, 2024-02-09T17:52:01.731 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:52:01.731 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:52:01.731 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:52:01.731 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:52:01.732 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:52:01.732 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:51:36.773344+0000", 2024-02-09T17:52:01.732 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:52:01.732 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.732 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.732 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 5, 2024-02-09T17:52:01.732 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:52:01.732 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:52:01.735 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:52:01.735 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:52:01.735 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:52:01.735 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:51:19.216055+0000", 2024-02-09T17:52:01.735 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:52:01.735 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.736 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.736 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 6, 2024-02-09T17:52:01.736 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:52:01.736 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:52:01.736 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:52:01.736 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:52:01.736 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:52:01.736 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:51:36.061452+0000", 2024-02-09T17:52:01.736 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:52:01.736 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.736 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.736 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 7, 2024-02-09T17:52:01.736 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:52:01.736 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:52:01.737 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:52:01.737 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:52:01.737 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:52:01.737 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:51:52.560157+0000", 2024-02-09T17:52:01.737 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:52:01.737 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.737 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.737 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 8, 2024-02-09T17:52:01.737 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:52:01.737 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:52:01.737 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:52:01.737 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:52:01.737 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:52:01.737 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:51:41.744510+0000", 2024-02-09T17:52:01.738 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 145 2024-02-09T17:52:01.738 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.738 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.738 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 9, 2024-02-09T17:52:01.738 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:52:01.738 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:52:01.738 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:52:01.738 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:52:01.738 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:52:01.738 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:51:50.447405+0000", 2024-02-09T17:52:01.738 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:52:01.738 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.738 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.738 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 10, 2024-02-09T17:52:01.739 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:52:01.739 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:52:01.739 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:52:01.739 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:52:01.739 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:52:01.739 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:51:44.252870+0000", 2024-02-09T17:52:01.739 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:52:01.739 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.739 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.739 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 11, 2024-02-09T17:52:01.739 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:52:01.739 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:52:01.739 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:52:01.739 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:52:01.739 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:52:01.740 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:50:54.440289+0000", 2024-02-09T17:52:01.740 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:52:01.740 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.740 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T17:52:01.740 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_upmap": [], 2024-02-09T17:52:01.740 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_upmap_items": [ 2024-02-09T17:52:01.740 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.740 INFO:teuthology.orchestra.run.smithi099.stdout: "pgid": "1.1e", 2024-02-09T17:52:01.740 INFO:teuthology.orchestra.run.smithi099.stdout: "mappings": [ 2024-02-09T17:52:01.740 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.740 INFO:teuthology.orchestra.run.smithi099.stdout: "from": 8, 2024-02-09T17:52:01.740 INFO:teuthology.orchestra.run.smithi099.stdout: "to": 3 2024-02-09T17:52:01.741 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.741 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.741 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.741 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T17:52:01.741 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_upmap_primaries": [], 2024-02-09T17:52:01.741 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_temp": [ 2024-02-09T17:52:01.741 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.741 INFO:teuthology.orchestra.run.smithi099.stdout: "pgid": "7.1", 2024-02-09T17:52:01.741 INFO:teuthology.orchestra.run.smithi099.stdout: "osds": [ 2024-02-09T17:52:01.741 INFO:teuthology.orchestra.run.smithi099.stdout: 4, 2024-02-09T17:52:01.741 INFO:teuthology.orchestra.run.smithi099.stdout: 10 2024-02-09T17:52:01.741 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.741 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.741 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.742 INFO:teuthology.orchestra.run.smithi099.stdout: "pgid": "7.4", 2024-02-09T17:52:01.742 INFO:teuthology.orchestra.run.smithi099.stdout: "osds": [ 2024-02-09T17:52:01.742 INFO:teuthology.orchestra.run.smithi099.stdout: 4, 2024-02-09T17:52:01.742 INFO:teuthology.orchestra.run.smithi099.stdout: 1 2024-02-09T17:52:01.742 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.742 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.742 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.742 INFO:teuthology.orchestra.run.smithi099.stdout: "pgid": "7.5", 2024-02-09T17:52:01.742 INFO:teuthology.orchestra.run.smithi099.stdout: "osds": [ 2024-02-09T17:52:01.742 INFO:teuthology.orchestra.run.smithi099.stdout: 11, 2024-02-09T17:52:01.742 INFO:teuthology.orchestra.run.smithi099.stdout: 9 2024-02-09T17:52:01.742 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.742 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.742 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.743 INFO:teuthology.orchestra.run.smithi099.stdout: "pgid": "7.6", 2024-02-09T17:52:01.743 INFO:teuthology.orchestra.run.smithi099.stdout: "osds": [ 2024-02-09T17:52:01.743 INFO:teuthology.orchestra.run.smithi099.stdout: 0, 2024-02-09T17:52:01.743 INFO:teuthology.orchestra.run.smithi099.stdout: 7 2024-02-09T17:52:01.743 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.743 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.743 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.743 INFO:teuthology.orchestra.run.smithi099.stdout: "pgid": "7.7", 2024-02-09T17:52:01.743 INFO:teuthology.orchestra.run.smithi099.stdout: "osds": [ 2024-02-09T17:52:01.743 INFO:teuthology.orchestra.run.smithi099.stdout: 6, 2024-02-09T17:52:01.743 INFO:teuthology.orchestra.run.smithi099.stdout: 8 2024-02-09T17:52:01.743 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.743 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.743 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.743 INFO:teuthology.orchestra.run.smithi099.stdout: "pgid": "7.9", 2024-02-09T17:52:01.744 INFO:teuthology.orchestra.run.smithi099.stdout: "osds": [ 2024-02-09T17:52:01.744 INFO:teuthology.orchestra.run.smithi099.stdout: 11, 2024-02-09T17:52:01.744 INFO:teuthology.orchestra.run.smithi099.stdout: 8 2024-02-09T17:52:01.744 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.744 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.744 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.744 INFO:teuthology.orchestra.run.smithi099.stdout: "pgid": "7.c", 2024-02-09T17:52:01.744 INFO:teuthology.orchestra.run.smithi099.stdout: "osds": [ 2024-02-09T17:52:01.744 INFO:teuthology.orchestra.run.smithi099.stdout: 6, 2024-02-09T17:52:01.744 INFO:teuthology.orchestra.run.smithi099.stdout: 11 2024-02-09T17:52:01.744 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.744 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.744 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:01.744 INFO:teuthology.orchestra.run.smithi099.stdout: "pgid": "7.f", 2024-02-09T17:52:01.745 INFO:teuthology.orchestra.run.smithi099.stdout: "osds": [ 2024-02-09T17:52:01.745 INFO:teuthology.orchestra.run.smithi099.stdout: 8, 2024-02-09T17:52:01.745 INFO:teuthology.orchestra.run.smithi099.stdout: 9 2024-02-09T17:52:01.745 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:01.745 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.745 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T17:52:01.745 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_temp": [], 2024-02-09T17:52:01.745 INFO:teuthology.orchestra.run.smithi099.stdout: "blocklist": { 2024-02-09T17:52:01.745 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:0/3904172782": "2024-02-10T17:16:24.229365+0000", 2024-02-09T17:52:01.745 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:6801/1586034576": "2024-02-10T17:16:24.229365+0000", 2024-02-09T17:52:01.745 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:6800/746248609": "2024-02-10T17:15:54.399201+0000", 2024-02-09T17:52:01.745 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:0/4092784197": "2024-02-10T17:16:24.229365+0000", 2024-02-09T17:52:01.745 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:0/2844598101": "2024-02-10T17:15:54.399201+0000", 2024-02-09T17:52:01.745 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:6801/746248609": "2024-02-10T17:15:54.399201+0000", 2024-02-09T17:52:01.745 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:0/2528750050": "2024-02-10T17:15:54.399201+0000", 2024-02-09T17:52:01.746 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:6800/1586034576": "2024-02-10T17:16:24.229365+0000", 2024-02-09T17:52:01.746 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:0/2214140078": "2024-02-10T17:15:54.399201+0000", 2024-02-09T17:52:01.746 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:0/4217821965": "2024-02-10T17:16:24.229365+0000" 2024-02-09T17:52:01.746 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.746 INFO:teuthology.orchestra.run.smithi099.stdout: "range_blocklist": {}, 2024-02-09T17:52:01.746 INFO:teuthology.orchestra.run.smithi099.stdout: "erasure_code_profiles": { 2024-02-09T17:52:01.746 INFO:teuthology.orchestra.run.smithi099.stdout: "default": { 2024-02-09T17:52:01.746 INFO:teuthology.orchestra.run.smithi099.stdout: "crush-failure-domain": "osd", 2024-02-09T17:52:01.746 INFO:teuthology.orchestra.run.smithi099.stdout: "k": "2", 2024-02-09T17:52:01.746 INFO:teuthology.orchestra.run.smithi099.stdout: "m": "1", 2024-02-09T17:52:01.746 INFO:teuthology.orchestra.run.smithi099.stdout: "plugin": "jerasure", 2024-02-09T17:52:01.746 INFO:teuthology.orchestra.run.smithi099.stdout: "technique": "reed_sol_van" 2024-02-09T17:52:01.746 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.746 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:01.747 INFO:teuthology.orchestra.run.smithi099.stdout: "removed_snaps_queue": [], 2024-02-09T17:52:01.747 INFO:teuthology.orchestra.run.smithi099.stdout: "new_removed_snaps": [], 2024-02-09T17:52:01.747 INFO:teuthology.orchestra.run.smithi099.stdout: "new_purged_snaps": [], 2024-02-09T17:52:01.747 INFO:teuthology.orchestra.run.smithi099.stdout: "crush_node_flags": {}, 2024-02-09T17:52:01.747 INFO:teuthology.orchestra.run.smithi099.stdout: "device_class_flags": {}, 2024-02-09T17:52:01.747 INFO:teuthology.orchestra.run.smithi099.stdout: "stretch_mode": { 2024-02-09T17:52:01.747 INFO:teuthology.orchestra.run.smithi099.stdout: "stretch_mode_enabled": false, 2024-02-09T17:52:01.747 INFO:teuthology.orchestra.run.smithi099.stdout: "stretch_bucket_count": 0, 2024-02-09T17:52:01.747 INFO:teuthology.orchestra.run.smithi099.stdout: "degraded_stretch_mode": 0, 2024-02-09T17:52:01.747 INFO:teuthology.orchestra.run.smithi099.stdout: "recovering_stretch_mode": 0, 2024-02-09T17:52:01.747 INFO:teuthology.orchestra.run.smithi099.stdout: "stretch_mode_bucket": 0 2024-02-09T17:52:01.747 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:01.747 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:52:01.748 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:52:01.792 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:02.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:02 smithi162 ceph-mon[44304]: pgmap v1762: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.5 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 123 B/s rd, 27 MiB/s wr, 591 op/s; 1/319239 objects degraded (0.000%); 24692/319239 objects misplaced (7.735%); 21 MiB/s, 340 objects/s recovering 2024-02-09T17:52:02.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:02 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1218087492' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-09T17:52:02.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:02 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 1/319239 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:52:02.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:02 smithi099 ceph-mon[39420]: pgmap v1762: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.5 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 123 B/s rd, 27 MiB/s wr, 591 op/s; 1/319239 objects degraded (0.000%); 24692/319239 objects misplaced (7.735%); 21 MiB/s, 340 objects/s recovering 2024-02-09T17:52:02.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:02 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1218087492' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-09T17:52:02.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:02 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 1/319239 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:52:02.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:02 smithi179 ceph-mon[43975]: pgmap v1762: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.5 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 123 B/s rd, 27 MiB/s wr, 591 op/s; 1/319239 objects degraded (0.000%); 24692/319239 objects misplaced (7.735%); 21 MiB/s, 340 objects/s recovering 2024-02-09T17:52:02.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:02 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1218087492' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-09T17:52:02.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:02 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 1/319239 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:52:03.838 INFO:tasks.thrashosds.thrasher:j is {'epoch': 704, 'fsid': 'a69c870a-c76e-11ee-95b7-87774f69a715', 'created': '2024-02-09T17:15:26.113606+0000', 'modified': '2024-02-09T17:51:56.787414+0000', 'last_up_change': '2024-02-09T17:28:17.711044+0000', 'last_in_change': '2024-02-09T17:43:24.138763+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 27, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-09T17:18:56.552286+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '544', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '514', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 5.989999771118164, 'score_stable': 1.7799999713897705, 'optimal_score': 1.090000033378601, 'raw_score_acting': 6.53000020980835, 'raw_score_stable': 1.940000057220459, 'primary_affinity_weighted': 0.7807959914207458, 'average_primary_affinity': 0.8500000238418579, 'average_primary_affinity_weighted': 0.7799999713897705, 'error_message': 'pool 1 has acting primaries on OSD(s) with primary affinity 0, read balance score is not accurate'}}, {'pool': 7, 'pool_name': 'unique_pool_5', 'create_time': '2024-02-09T17:50:17.290198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '665', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.059999942779541, 'score_stable': 2.059999942779541, 'optimal_score': 1.090000033378601, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.7807959914207458, 'average_primary_affinity': 0.8500000238418579, 'average_primary_affinity_weighted': 0.7799999713897705}}], 'osds': [{'osd': 0, 'uuid': '21739aa7-e278-45da-9a3f-867385f01baf', 'up': 1, 'in': 1, 'weight': 0.7243499755859375, 'primary_affinity': 0.609619140625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 684, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6802', 'nonce': 3859868923}, {'type': 'v1', 'addr': '172.21.15.99:6803', 'nonce': 3859868923}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6804', 'nonce': 3859868923}, {'type': 'v1', 'addr': '172.21.15.99:6805', 'nonce': 3859868923}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6808', 'nonce': 3859868923}, {'type': 'v1', 'addr': '172.21.15.99:6809', 'nonce': 3859868923}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6806', 'nonce': 3859868923}, {'type': 'v1', 'addr': '172.21.15.99:6807', 'nonce': 3859868923}]}, 'public_addr': '172.21.15.99:6803/3859868923', 'cluster_addr': '172.21.15.99:6805/3859868923', 'heartbeat_back_addr': '172.21.15.99:6809/3859868923', 'heartbeat_front_addr': '172.21.15.99:6807/3859868923', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '8eab347c-9065-4de1-93ae-e2eb7e11cab7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 684, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6810', 'nonce': 3868328452}, {'type': 'v1', 'addr': '172.21.15.99:6811', 'nonce': 3868328452}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6812', 'nonce': 3868328452}, {'type': 'v1', 'addr': '172.21.15.99:6813', 'nonce': 3868328452}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6816', 'nonce': 3868328452}, {'type': 'v1', 'addr': '172.21.15.99:6817', 'nonce': 3868328452}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6814', 'nonce': 3868328452}, {'type': 'v1', 'addr': '172.21.15.99:6815', 'nonce': 3868328452}]}, 'public_addr': '172.21.15.99:6811/3868328452', 'cluster_addr': '172.21.15.99:6813/3868328452', 'heartbeat_back_addr': '172.21.15.99:6817/3868328452', 'heartbeat_front_addr': '172.21.15.99:6815/3868328452', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '575f4257-7e34-485e-963c-2607b69e1220', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.3601837158203125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 661, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6818', 'nonce': 537506237}, {'type': 'v1', 'addr': '172.21.15.99:6819', 'nonce': 537506237}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6820', 'nonce': 537506237}, {'type': 'v1', 'addr': '172.21.15.99:6821', 'nonce': 537506237}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6824', 'nonce': 537506237}, {'type': 'v1', 'addr': '172.21.15.99:6825', 'nonce': 537506237}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6822', 'nonce': 537506237}, {'type': 'v1', 'addr': '172.21.15.99:6823', 'nonce': 537506237}]}, 'public_addr': '172.21.15.99:6819/537506237', 'cluster_addr': '172.21.15.99:6821/537506237', 'heartbeat_back_addr': '172.21.15.99:6825/537506237', 'heartbeat_front_addr': '172.21.15.99:6823/537506237', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '1a520bcf-81bb-4f1d-80fb-11e98008e429', 'up': 1, 'in': 1, 'weight': 0.659332275390625, 'primary_affinity': 0.8014984130859375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 684, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6826', 'nonce': 762784793}, {'type': 'v1', 'addr': '172.21.15.99:6827', 'nonce': 762784793}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6828', 'nonce': 762784793}, {'type': 'v1', 'addr': '172.21.15.99:6829', 'nonce': 762784793}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6832', 'nonce': 762784793}, {'type': 'v1', 'addr': '172.21.15.99:6833', 'nonce': 762784793}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6830', 'nonce': 762784793}, {'type': 'v1', 'addr': '172.21.15.99:6831', 'nonce': 762784793}]}, 'public_addr': '172.21.15.99:6827/762784793', 'cluster_addr': '172.21.15.99:6829/762784793', 'heartbeat_back_addr': '172.21.15.99:6833/762784793', 'heartbeat_front_addr': '172.21.15.99:6831/762784793', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '4617c2fe-c9d8-40cd-887b-303e3215bdf8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 682, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6800', 'nonce': 4178676504}, {'type': 'v1', 'addr': '172.21.15.162:6801', 'nonce': 4178676504}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6802', 'nonce': 4178676504}, {'type': 'v1', 'addr': '172.21.15.162:6803', 'nonce': 4178676504}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6806', 'nonce': 4178676504}, {'type': 'v1', 'addr': '172.21.15.162:6807', 'nonce': 4178676504}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6804', 'nonce': 4178676504}, {'type': 'v1', 'addr': '172.21.15.162:6805', 'nonce': 4178676504}]}, 'public_addr': '172.21.15.162:6801/4178676504', 'cluster_addr': '172.21.15.162:6803/4178676504', 'heartbeat_back_addr': '172.21.15.162:6807/4178676504', 'heartbeat_front_addr': '172.21.15.162:6805/4178676504', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'da98e3f3-173e-46fe-be8b-1ccbe8b0dabc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 576, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6808', 'nonce': 1489371339}, {'type': 'v1', 'addr': '172.21.15.162:6809', 'nonce': 1489371339}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6810', 'nonce': 1489371339}, {'type': 'v1', 'addr': '172.21.15.162:6811', 'nonce': 1489371339}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6814', 'nonce': 1489371339}, {'type': 'v1', 'addr': '172.21.15.162:6815', 'nonce': 1489371339}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6812', 'nonce': 1489371339}, {'type': 'v1', 'addr': '172.21.15.162:6813', 'nonce': 1489371339}]}, 'public_addr': '172.21.15.162:6809/1489371339', 'cluster_addr': '172.21.15.162:6811/1489371339', 'heartbeat_back_addr': '172.21.15.162:6815/1489371339', 'heartbeat_front_addr': '172.21.15.162:6813/1489371339', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'ff50382d-5ddf-4cda-bb0c-ff055feeaa40', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 49, 'up_thru': 684, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6816', 'nonce': 821169663}, {'type': 'v1', 'addr': '172.21.15.162:6817', 'nonce': 821169663}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6818', 'nonce': 821169663}, {'type': 'v1', 'addr': '172.21.15.162:6819', 'nonce': 821169663}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6822', 'nonce': 821169663}, {'type': 'v1', 'addr': '172.21.15.162:6823', 'nonce': 821169663}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6820', 'nonce': 821169663}, {'type': 'v1', 'addr': '172.21.15.162:6821', 'nonce': 821169663}]}, 'public_addr': '172.21.15.162:6817/821169663', 'cluster_addr': '172.21.15.162:6819/821169663', 'heartbeat_back_addr': '172.21.15.162:6823/821169663', 'heartbeat_front_addr': '172.21.15.162:6821/821169663', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'a82984dc-5e22-424b-a446-ae2ee71eee19', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.5982513427734375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 57, 'up_thru': 661, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6824', 'nonce': 965264369}, {'type': 'v1', 'addr': '172.21.15.162:6825', 'nonce': 965264369}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6826', 'nonce': 965264369}, {'type': 'v1', 'addr': '172.21.15.162:6827', 'nonce': 965264369}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6830', 'nonce': 965264369}, {'type': 'v1', 'addr': '172.21.15.162:6831', 'nonce': 965264369}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6828', 'nonce': 965264369}, {'type': 'v1', 'addr': '172.21.15.162:6829', 'nonce': 965264369}]}, 'public_addr': '172.21.15.162:6825/965264369', 'cluster_addr': '172.21.15.162:6827/965264369', 'heartbeat_back_addr': '172.21.15.162:6831/965264369', 'heartbeat_front_addr': '172.21.15.162:6829/965264369', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '63f9d79f-0400-44dc-b388-df00fefb5820', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 66, 'last_clean_end': 145, 'up_from': 171, 'up_thru': 684, 'down_at': 146, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6800', 'nonce': 2558294495}, {'type': 'v1', 'addr': '172.21.15.179:6801', 'nonce': 2558294495}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6802', 'nonce': 2558294495}, {'type': 'v1', 'addr': '172.21.15.179:6803', 'nonce': 2558294495}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6806', 'nonce': 2558294495}, {'type': 'v1', 'addr': '172.21.15.179:6807', 'nonce': 2558294495}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6804', 'nonce': 2558294495}, {'type': 'v1', 'addr': '172.21.15.179:6805', 'nonce': 2558294495}]}, 'public_addr': '172.21.15.179:6801/2558294495', 'cluster_addr': '172.21.15.179:6803/2558294495', 'heartbeat_back_addr': '172.21.15.179:6807/2558294495', 'heartbeat_front_addr': '172.21.15.179:6805/2558294495', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '7d995854-7328-4dfc-b1a9-0edca8d74f46', 'up': 1, 'in': 1, 'weight': 0.8393707275390625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 77, 'up_thru': 661, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6808', 'nonce': 504427183}, {'type': 'v1', 'addr': '172.21.15.179:6809', 'nonce': 504427183}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6810', 'nonce': 504427183}, {'type': 'v1', 'addr': '172.21.15.179:6811', 'nonce': 504427183}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6814', 'nonce': 504427183}, {'type': 'v1', 'addr': '172.21.15.179:6815', 'nonce': 504427183}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6812', 'nonce': 504427183}, {'type': 'v1', 'addr': '172.21.15.179:6813', 'nonce': 504427183}]}, 'public_addr': '172.21.15.179:6809/504427183', 'cluster_addr': '172.21.15.179:6811/504427183', 'heartbeat_back_addr': '172.21.15.179:6815/504427183', 'heartbeat_front_addr': '172.21.15.179:6813/504427183', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '151fbedd-08e6-46e6-a4e5-a14248399f61', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 84, 'up_thru': 684, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6816', 'nonce': 2403345164}, {'type': 'v1', 'addr': '172.21.15.179:6817', 'nonce': 2403345164}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6818', 'nonce': 2403345164}, {'type': 'v1', 'addr': '172.21.15.179:6819', 'nonce': 2403345164}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6822', 'nonce': 2403345164}, {'type': 'v1', 'addr': '172.21.15.179:6823', 'nonce': 2403345164}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6820', 'nonce': 2403345164}, {'type': 'v1', 'addr': '172.21.15.179:6821', 'nonce': 2403345164}]}, 'public_addr': '172.21.15.179:6817/2403345164', 'cluster_addr': '172.21.15.179:6819/2403345164', 'heartbeat_back_addr': '172.21.15.179:6823/2403345164', 'heartbeat_front_addr': '172.21.15.179:6821/2403345164', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '6d3a3ad8-417a-483f-991c-52549dbe7d73', 'up': 1, 'in': 1, 'weight': 0.8402862548828125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 90, 'up_thru': 682, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6824', 'nonce': 2899402373}, {'type': 'v1', 'addr': '172.21.15.179:6825', 'nonce': 2899402373}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6826', 'nonce': 2899402373}, {'type': 'v1', 'addr': '172.21.15.179:6827', 'nonce': 2899402373}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6830', 'nonce': 2899402373}, {'type': 'v1', 'addr': '172.21.15.179:6831', 'nonce': 2899402373}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6828', 'nonce': 2899402373}, {'type': 'v1', 'addr': '172.21.15.179:6829', 'nonce': 2899402373}]}, 'public_addr': '172.21.15.179:6825/2899402373', 'cluster_addr': '172.21.15.179:6827/2899402373', 'heartbeat_back_addr': '172.21.15.179:6831/2899402373', 'heartbeat_front_addr': '172.21.15.179:6829/2899402373', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:51:49.767472+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:51:51.916964+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:51:29.925627+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:51:47.591375+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:51:36.773344+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:51:19.216055+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:51:36.061452+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:51:52.560157+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:51:41.744510+0000', 'dead_epoch': 145}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:51:50.447405+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:51:44.252870+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:50:54.440289+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '1.1e', 'mappings': [{'from': 8, 'to': 3}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '7.1', 'osds': [4, 10]}, {'pgid': '7.4', 'osds': [4, 1]}, {'pgid': '7.5', 'osds': [11, 9]}, {'pgid': '7.6', 'osds': [0, 7]}, {'pgid': '7.7', 'osds': [6, 8]}, {'pgid': '7.9', 'osds': [11, 8]}, {'pgid': '7.c', 'osds': [6, 11]}, {'pgid': '7.f', 'osds': [8, 9]}], 'primary_temp': [], 'blocklist': {'172.21.15.99:0/3904172782': '2024-02-10T17:16:24.229365+0000', '172.21.15.99:6801/1586034576': '2024-02-10T17:16:24.229365+0000', '172.21.15.99:6800/746248609': '2024-02-10T17:15:54.399201+0000', '172.21.15.99:0/4092784197': '2024-02-10T17:16:24.229365+0000', '172.21.15.99:0/2844598101': '2024-02-10T17:15:54.399201+0000', '172.21.15.99:6801/746248609': '2024-02-10T17:15:54.399201+0000', '172.21.15.99:0/2528750050': '2024-02-10T17:15:54.399201+0000', '172.21.15.99:6800/1586034576': '2024-02-10T17:16:24.229365+0000', '172.21.15.99:0/2214140078': '2024-02-10T17:15:54.399201+0000', '172.21.15.99:0/4217821965': '2024-02-10T17:16:24.229365+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-09T17:52:03.839 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph pg dump --format=json 2024-02-09T17:52:04.171 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:04 smithi099 ceph-mon[39420]: pgmap v1763: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 22 MiB/s wr, 650 op/s; 1/314466 objects degraded (0.000%); 23998/314466 objects misplaced (7.631%); 22 MiB/s, 350 objects/s recovering 2024-02-09T17:52:04.456 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:04 smithi179 ceph-mon[43975]: pgmap v1763: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 22 MiB/s wr, 650 op/s; 1/314466 objects degraded (0.000%); 23998/314466 objects misplaced (7.631%); 22 MiB/s, 350 objects/s recovering 2024-02-09T17:52:04.502 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:04 smithi162 ceph-mon[44304]: pgmap v1763: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 22 MiB/s wr, 650 op/s; 1/314466 objects degraded (0.000%); 23998/314466 objects misplaced (7.631%); 22 MiB/s, 350 objects/s recovering 2024-02-09T17:52:04.527 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:52:04.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:04.536 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:52:04.637 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:52:04.676 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:04.710 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:52:04.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:04.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:52:04.803 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:52:04.815 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:52:04.849 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:04.878 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:52:04.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:04.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:52:04.987 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:52:05.040 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:05.199 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is unset 2024-02-09T17:52:05.249 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:52:05.276 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:52:05.301 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:52:05.325 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:52:05.352 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:52:05.385 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:52:05.396 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:05 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3899605329' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:52:05.403 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:52:05.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:05.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:52:05.422 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:52:05.450 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:52:05.511 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:52:05.564 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:05 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3899605329' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:52:05.572 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:05.635 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:52:05.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:05.640 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:52:05.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:05 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3899605329' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:52:05.702 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:52:05.735 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:52:05.741 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:52:05.772 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:52:05.808 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:05.831 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:52:05.841 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:52:05.843 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:52:05.844 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:52:05.844 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:52:05.844 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:52:05.844 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:52:05.844 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:52:05.844 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:52:05.844 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:52:05.844 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:52:05.844 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:52:05.844 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:52:05.844 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:52:05.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:05.852 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:52:05.953 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:52:05.983 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:52:05.983 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-02-09T17:52:05.983 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-02-09T17:52:05.983 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-02-09T17:52:05.983 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:52:05.991 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:06.084 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:52:06.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:06.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:52:06.195 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:52:06.267 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:06.298 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:52:06.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:06.303 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:52:06.405 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:52:06.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:06 smithi162 ceph-mon[44304]: pgmap v1764: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 20 MiB/s wr, 649 op/s; 1/313335 objects degraded (0.000%); 23796/313335 objects misplaced (7.594%); 22 MiB/s, 351 objects/s recovering 2024-02-09T17:52:06.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:06 smithi162 ceph-mon[44304]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:52:06.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:06 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3899605329' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:52:06.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:06 smithi162 ceph-mon[44304]: osdmap e705: 12 total, 12 up, 12 in 2024-02-09T17:52:06.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:06 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/73028' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:52:06.443 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:06.480 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:52:06.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:06.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:52:06.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:06 smithi099 ceph-mon[39420]: pgmap v1764: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 20 MiB/s wr, 649 op/s; 1/313335 objects degraded (0.000%); 23796/313335 objects misplaced (7.594%); 22 MiB/s, 351 objects/s recovering 2024-02-09T17:52:06.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:06 smithi099 ceph-mon[39420]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:52:06.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:06 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3899605329' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:52:06.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:06 smithi099 ceph-mon[39420]: osdmap e705: 12 total, 12 up, 12 in 2024-02-09T17:52:06.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:06 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/73028' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:52:06.584 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:52:06.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:06 smithi179 ceph-mon[43975]: pgmap v1764: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 20 MiB/s wr, 649 op/s; 1/313335 objects degraded (0.000%); 23796/313335 objects misplaced (7.594%); 22 MiB/s, 351 objects/s recovering 2024-02-09T17:52:06.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:06 smithi179 ceph-mon[43975]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:52:06.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:06 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3899605329' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:52:06.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:06 smithi179 ceph-mon[43975]: osdmap e705: 12 total, 12 up, 12 in 2024-02-09T17:52:06.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:06 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/73028' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:52:06.672 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:07.139 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:52:07.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:07.148 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:52:07.249 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:52:07.319 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:07.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:07 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 1/313335 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:52:07.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:07 smithi179 ceph-mon[43975]: osdmap e706: 12 total, 12 up, 12 in 2024-02-09T17:52:07.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:07 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 1/313335 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:52:07.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:07 smithi099 ceph-mon[39420]: osdmap e706: 12 total, 12 up, 12 in 2024-02-09T17:52:07.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:07 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 1/313335 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:52:07.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:07 smithi162 ceph-mon[44304]: osdmap e706: 12 total, 12 up, 12 in 2024-02-09T17:52:08.018 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:52:08.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:08.031 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:52:08.042 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.8 dump_historic_ops 2024-02-09T17:52:08.131 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:52:08.190 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:08.238 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:52:08.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:08.246 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:08 smithi179 ceph-mon[43975]: pgmap v1767: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 2.5 MiB/s wr, 711 op/s; 1/306522 objects degraded (0.000%); 22223/306522 objects misplaced (7.250%); 26 MiB/s, 422 objects/s recovering 2024-02-09T17:52:08.247 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:52:08.348 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:52:08.455 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:08.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:08 smithi099 ceph-mon[39420]: pgmap v1767: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 2.5 MiB/s wr, 711 op/s; 1/306522 objects degraded (0.000%); 22223/306522 objects misplaced (7.250%); 26 MiB/s, 422 objects/s recovering 2024-02-09T17:52:08.496 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:52:08.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:08.502 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:52:08.601 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:52:08.602 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:52:08.648 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:08 smithi162 ceph-mon[44304]: pgmap v1767: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 2.5 MiB/s wr, 711 op/s; 1/306522 objects degraded (0.000%); 22223/306522 objects misplaced (7.250%); 26 MiB/s, 422 objects/s recovering 2024-02-09T17:52:08.650 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:08.680 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:52:08.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:08.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:52:08.785 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:52:08.829 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:08.857 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:52:08.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:08.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:52:08.966 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:52:09.054 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:09.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:09 smithi179 ceph-mon[43975]: 1.18 scrub starts 2024-02-09T17:52:09.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:09 smithi179 ceph-mon[43975]: 1.18 scrub ok 2024-02-09T17:52:09.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:09 smithi099 ceph-mon[39420]: 1.18 scrub starts 2024-02-09T17:52:09.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:09 smithi099 ceph-mon[39420]: 1.18 scrub ok 2024-02-09T17:52:09.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:09 smithi162 ceph-mon[44304]: 1.18 scrub starts 2024-02-09T17:52:09.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:09 smithi162 ceph-mon[44304]: 1.18 scrub ok 2024-02-09T17:52:09.825 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:52:09.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:09.834 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:52:09.935 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:52:10.008 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:10.055 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:52:10.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:10.058 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:52:10.159 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:52:10.218 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:10.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:10 smithi099 ceph-mon[39420]: pgmap v1768: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s wr, 637 op/s; 1/304041 objects degraded (0.000%); 21885/304041 objects misplaced (7.198%); 22 MiB/s, 350 objects/s recovering 2024-02-09T17:52:10.678 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:10 smithi179 ceph-mon[43975]: pgmap v1768: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s wr, 637 op/s; 1/304041 objects degraded (0.000%); 21885/304041 objects misplaced (7.198%); 22 MiB/s, 350 objects/s recovering 2024-02-09T17:52:10.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:10 smithi162 ceph-mon[44304]: pgmap v1768: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s wr, 637 op/s; 1/304041 objects degraded (0.000%); 21885/304041 objects misplaced (7.198%); 22 MiB/s, 350 objects/s recovering 2024-02-09T17:52:11.288 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:52:11.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:11.293 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:52:11.393 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:52:11.467 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:11.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:11 smithi099 ceph-mon[39420]: 7.2 scrub starts 2024-02-09T17:52:11.618 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:52:11.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:11.625 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:11 smithi179 ceph-mon[43975]: 7.2 scrub starts 2024-02-09T17:52:11.626 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:52:11.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:11 smithi162 ceph-mon[44304]: 7.2 scrub starts 2024-02-09T17:52:11.726 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:52:11.783 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:12.438 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:12 smithi099 ceph-mon[39420]: pgmap v1769: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 607 op/s; 1/299871 objects degraded (0.000%); 20835/299871 objects misplaced (6.948%); 25 MiB/s, 395 objects/s recovering 2024-02-09T17:52:12.438 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:12 smithi099 ceph-mon[39420]: 7.2 scrub ok 2024-02-09T17:52:12.438 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:12 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:52:12.438 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:12 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 1/299871 objects degraded (0.000%), 1 pg degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:52:12.446 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:52:12.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:12.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:52:12.550 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:52:12.599 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:12.641 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:52:12.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:12.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:52:12.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:12 smithi179 ceph-mon[43975]: pgmap v1769: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 607 op/s; 1/299871 objects degraded (0.000%); 20835/299871 objects misplaced (6.948%); 25 MiB/s, 395 objects/s recovering 2024-02-09T17:52:12.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:12 smithi179 ceph-mon[43975]: 7.2 scrub ok 2024-02-09T17:52:12.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:12 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:52:12.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:12 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 1/299871 objects degraded (0.000%), 1 pg degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:52:12.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:12 smithi162 ceph-mon[44304]: pgmap v1769: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 607 op/s; 1/299871 objects degraded (0.000%); 20835/299871 objects misplaced (6.948%); 25 MiB/s, 395 objects/s recovering 2024-02-09T17:52:12.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:12 smithi162 ceph-mon[44304]: 7.2 scrub ok 2024-02-09T17:52:12.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:12 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:52:12.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:12 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 1/299871 objects degraded (0.000%), 1 pg degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:52:12.751 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:52:12.787 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:13.377 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:52:13.381 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-02-09T17:52:13.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:13 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:52:13.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:13 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:52:13.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:13 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:52:13.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:13 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:52:13.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:13 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:52:13.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:13 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:52:14.144 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-09T17:52:14.245 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:52:14.313 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":1771,"stamp":"2024-02-09T17:52:13.246347+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6443303480,"num_objects":98311,"num_object_clones":0,"num_object_copies":294933,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":20041,"num_objects_unfound":0,"num_objects_dirty":98311,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117269,"num_write_kb":6898433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19192,"num_bytes_recovered":1257766912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121578,"ondisk_log_size":121578,"up":141,"acting":133,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":149,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":27816032,"kb_used_data":20205428,"kb_used_omap":229,"kb_used_meta":7609946,"kb_avail":1097125792,"statfs":{"total":1151940427776,"available":1123456811008,"internally_reserved":0,"allocated":20690358272,"data_stored":20684478020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235127,"internal_metadata":7792585097},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,3,5,3,5,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-401997824,"num_objects":-6134,"num_object_clones":0,"num_object_copies":-18402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-3755,"num_objects_unfound":0,"num_objects_dirty":-6134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6134,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3755,"num_bytes_recovered":246087680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.069803"},"pg_stats":[{"pgid":"1.1e","version":"63'139","reported_seq":685,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-09T17:52:06.835771+0000","last_change":"2024-02-09T17:51:26.903371+0000","last_active":"2024-02-09T17:52:06.835771+0000","last_peered":"2024-02-09T17:52:06.835771+0000","last_clean":"2024-02-09T17:52:06.835771+0000","last_became_active":"2024-02-09T17:51:12.786195+0000","last_became_peered":"2024-02-09T17:51:12.786195+0000","last_unstale":"2024-02-09T17:52:06.835771+0000","last_undegraded":"2024-02-09T17:52:06.835771+0000","last_fullsized":"2024-02-09T17:52:06.835771+0000","mapping_epoch":684,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":685,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:51:26.903291+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:51:26.903291+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:52:35.066230+0000","scrub_duration":213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,10],"acting":[3,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1d","version":"63'139","reported_seq":701,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-09T17:52:06.834240+0000","last_change":"2024-02-09T17:50:21.514936+0000","last_active":"2024-02-09T17:52:06.834240+0000","last_peered":"2024-02-09T17:52:06.834240+0000","last_clean":"2024-02-09T17:52:06.834240+0000","last_became_active":"2024-02-09T17:43:28.594296+0000","last_became_peered":"2024-02-09T17:43:28.594296+0000","last_unstale":"2024-02-09T17:52:06.834240+0000","last_undegraded":"2024-02-09T17:52:06.834240+0000","last_fullsized":"2024-02-09T17:52:06.834240+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:50:21.514819+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:48:24.634092+0000","last_clean_scrub_stamp":"2024-02-09T17:50:21.514819+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"63'139","reported_seq":1221,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-09T17:52:06.808933+0000","last_change":"2024-02-09T17:50:31.454680+0000","last_active":"2024-02-09T17:52:06.808933+0000","last_peered":"2024-02-09T17:52:06.808933+0000","last_clean":"2024-02-09T17:52:06.808933+0000","last_became_active":"2024-02-09T17:43:19.103584+0000","last_became_peered":"2024-02-09T17:43:19.103584+0000","last_unstale":"2024-02-09T17:52:06.808933+0000","last_undegraded":"2024-02-09T17:52:06.808933+0000","last_fullsized":"2024-02-09T17:52:06.808933+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:50:31.454595+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:23.912504+0000","last_clean_scrub_stamp":"2024-02-09T17:50:31.454595+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"63'139","reported_seq":1019,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-09T17:52:06.808886+0000","last_change":"2024-02-09T17:50:33.641959+0000","last_active":"2024-02-09T17:52:06.808886+0000","last_peered":"2024-02-09T17:52:06.808886+0000","last_clean":"2024-02-09T17:52:06.808886+0000","last_became_active":"2024-02-09T17:43:19.102729+0000","last_became_peered":"2024-02-09T17:43:19.102729+0000","last_unstale":"2024-02-09T17:52:06.808886+0000","last_undegraded":"2024-02-09T17:52:06.808886+0000","last_fullsized":"2024-02-09T17:52:06.808886+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:50:33.641822+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:45:44.988633+0000","last_clean_scrub_stamp":"2024-02-09T17:50:33.641822+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":164,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"63'139","reported_seq":682,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-09T17:52:06.819344+0000","last_change":"2024-02-09T17:49:36.820953+0000","last_active":"2024-02-09T17:52:06.819344+0000","last_peered":"2024-02-09T17:52:06.819344+0000","last_clean":"2024-02-09T17:52:06.819344+0000","last_became_active":"2024-02-09T17:43:19.120814+0000","last_became_peered":"2024-02-09T17:43:19.120814+0000","last_unstale":"2024-02-09T17:52:06.819344+0000","last_undegraded":"2024-02-09T17:52:06.819344+0000","last_fullsized":"2024-02-09T17:52:06.819344+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:49:36.820839+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:49:36.820839+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"63'139","reported_seq":1527,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-09T17:52:06.833111+0000","last_change":"2024-02-09T17:51:12.795820+0000","last_active":"2024-02-09T17:52:06.833111+0000","last_peered":"2024-02-09T17:52:06.833111+0000","last_clean":"2024-02-09T17:52:06.833111+0000","last_became_active":"2024-02-09T17:51:12.795469+0000","last_became_peered":"2024-02-09T17:51:12.795469+0000","last_unstale":"2024-02-09T17:52:06.833111+0000","last_undegraded":"2024-02-09T17:52:06.833111+0000","last_fullsized":"2024-02-09T17:52:06.833111+0000","mapping_epoch":684,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":685,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:50:23.652765+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:30.193194+0000","last_clean_scrub_stamp":"2024-02-09T17:50:23.652765+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,7],"acting":[10,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"63'139","reported_seq":1365,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-09T17:52:07.890845+0000","last_change":"2024-02-09T17:52:07.890893+0000","last_active":"2024-02-09T17:52:07.890845+0000","last_peered":"2024-02-09T17:52:07.890845+0000","last_clean":"2024-02-09T17:52:07.890845+0000","last_became_active":"2024-02-09T17:43:28.585573+0000","last_became_peered":"2024-02-09T17:43:28.585573+0000","last_unstale":"2024-02-09T17:52:07.890845+0000","last_undegraded":"2024-02-09T17:52:07.890845+0000","last_fullsized":"2024-02-09T17:52:07.890845+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:52:07.890754+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:52:07.890754+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:53:32.342120+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"63'139","reported_seq":811,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-09T17:52:06.809183+0000","last_change":"2024-02-09T17:50:32.563676+0000","last_active":"2024-02-09T17:52:06.809183+0000","last_peered":"2024-02-09T17:52:06.809183+0000","last_clean":"2024-02-09T17:52:06.809183+0000","last_became_active":"2024-02-09T17:48:46.184419+0000","last_became_peered":"2024-02-09T17:48:46.184419+0000","last_unstale":"2024-02-09T17:52:06.809183+0000","last_undegraded":"2024-02-09T17:52:06.809183+0000","last_fullsized":"2024-02-09T17:52:06.809183+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:50:32.563555+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:44:50.214686+0000","last_clean_scrub_stamp":"2024-02-09T17:50:32.563555+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"63'139","reported_seq":693,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-09T17:52:06.834144+0000","last_change":"2024-02-09T17:51:12.798017+0000","last_active":"2024-02-09T17:52:06.834144+0000","last_peered":"2024-02-09T17:52:06.834144+0000","last_clean":"2024-02-09T17:52:06.834144+0000","last_became_active":"2024-02-09T17:51:12.797605+0000","last_became_peered":"2024-02-09T17:51:12.797605+0000","last_unstale":"2024-02-09T17:52:06.834144+0000","last_undegraded":"2024-02-09T17:52:06.834144+0000","last_fullsized":"2024-02-09T17:52:06.834144+0000","mapping_epoch":684,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":685,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:50:31.509861+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:11.970338+0000","last_clean_scrub_stamp":"2024-02-09T17:50:31.509861+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":153,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,11],"acting":[1,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"63'139","reported_seq":1131,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-09T17:52:06.825589+0000","last_change":"2024-02-09T17:50:17.104141+0000","last_active":"2024-02-09T17:52:06.825589+0000","last_peered":"2024-02-09T17:52:06.825589+0000","last_clean":"2024-02-09T17:52:06.825589+0000","last_became_active":"2024-02-09T17:43:19.138398+0000","last_became_peered":"2024-02-09T17:43:19.138398+0000","last_unstale":"2024-02-09T17:52:06.825589+0000","last_undegraded":"2024-02-09T17:52:06.825589+0000","last_fullsized":"2024-02-09T17:52:06.825589+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:50:17.104041+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:50:17.104041+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"63'139","reported_seq":1185,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-09T17:52:06.811332+0000","last_change":"2024-02-09T17:51:10.365739+0000","last_active":"2024-02-09T17:52:06.811332+0000","last_peered":"2024-02-09T17:52:06.811332+0000","last_clean":"2024-02-09T17:52:06.811332+0000","last_became_active":"2024-02-09T17:43:28.589307+0000","last_became_peered":"2024-02-09T17:43:28.589307+0000","last_unstale":"2024-02-09T17:52:06.811332+0000","last_undegraded":"2024-02-09T17:52:06.811332+0000","last_fullsized":"2024-02-09T17:52:06.811332+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:51:10.365637+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:38:26.968892+0000","last_clean_scrub_stamp":"2024-02-09T17:51:10.365637+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:52:26.611343+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"706'7183","reported_seq":11459,"reported_epoch":706,"state":"active+recovering+undersized+remapped","last_fresh":"2024-02-09T17:52:08.462785+0000","last_change":"2024-02-09T17:51:11.034479+0000","last_active":"2024-02-09T17:52:08.462785+0000","last_peered":"2024-02-09T17:52:08.462785+0000","last_clean":"2024-02-09T17:51:08.770486+0000","last_became_active":"2024-02-09T17:51:11.002287+0000","last_became_peered":"2024-02-09T17:51:11.002287+0000","last_unstale":"2024-02-09T17:52:08.462785+0000","last_undegraded":"2024-02-09T17:52:08.462785+0000","last_fullsized":"2024-02-09T17:51:10.792147+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_clean_scrub_stamp":"2024-02-09T17:50:17.733761+0000","objects_scrubbed":0,"log_size":7183,"log_dups_size":0,"ondisk_log_size":7183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":407044096,"num_objects":6211,"num_object_clones":0,"num_object_copies":18633,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2815,"num_objects_unfound":0,"num_objects_dirty":6211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7183,"num_write_kb":428608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":136708096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,10],"acting":[0,7],"avail_no_missing":["0","3","7"],"object_location_counts":[{"shards":"0,3,7","objects":2815},{"shards":"0,7","objects":3396}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"63'139","reported_seq":1780,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-09T17:52:06.825741+0000","last_change":"2024-02-09T17:50:19.968754+0000","last_active":"2024-02-09T17:52:06.825741+0000","last_peered":"2024-02-09T17:52:06.825741+0000","last_clean":"2024-02-09T17:52:06.825741+0000","last_became_active":"2024-02-09T17:41:14.521079+0000","last_became_peered":"2024-02-09T17:41:14.521079+0000","last_unstale":"2024-02-09T17:52:06.825741+0000","last_undegraded":"2024-02-09T17:52:06.825741+0000","last_fullsized":"2024-02-09T17:52:06.825741+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:50:19.968667+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:50:19.968667+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"706'7212","reported_seq":12019,"reported_epoch":706,"state":"active+recovering+undersized+remapped","last_fresh":"2024-02-09T17:52:11.325518+0000","last_change":"2024-02-09T17:51:11.046403+0000","last_active":"2024-02-09T17:52:11.325518+0000","last_peered":"2024-02-09T17:52:11.325518+0000","last_clean":"2024-02-09T17:51:08.601796+0000","last_became_active":"2024-02-09T17:51:11.006433+0000","last_became_peered":"2024-02-09T17:51:11.006433+0000","last_unstale":"2024-02-09T17:52:11.325518+0000","last_undegraded":"2024-02-09T17:52:11.325518+0000","last_fullsized":"2024-02-09T17:51:10.791412+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_clean_scrub_stamp":"2024-02-09T17:50:17.733761+0000","objects_scrubbed":0,"log_size":7212,"log_dups_size":0,"ondisk_log_size":7212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393871360,"num_objects":6010,"num_object_clones":0,"num_object_copies":18030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2479,"num_objects_unfound":0,"num_objects_dirty":6010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7212,"num_write_kb":423104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2351,"num_bytes_recovered":154075136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,9],"acting":[4,1],"avail_no_missing":["4","0","1"],"object_location_counts":[{"shards":"0,1,4","objects":2479},{"shards":"1,4","objects":3531}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"63'139","reported_seq":1346,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-09T17:52:06.811443+0000","last_change":"2024-02-09T17:50:21.778738+0000","last_active":"2024-02-09T17:52:06.811443+0000","last_peered":"2024-02-09T17:52:06.811443+0000","last_clean":"2024-02-09T17:52:06.811443+0000","last_became_active":"2024-02-09T17:43:39.981054+0000","last_became_peered":"2024-02-09T17:43:39.981054+0000","last_unstale":"2024-02-09T17:52:06.811443+0000","last_undegraded":"2024-02-09T17:52:06.811443+0000","last_fullsized":"2024-02-09T17:52:06.811443+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:50:21.778647+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:50:21.778647+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.5","version":"706'7230","reported_seq":11558,"reported_epoch":706,"state":"active+recovering+undersized+remapped","last_fresh":"2024-02-09T17:52:11.017171+0000","last_change":"2024-02-09T17:51:11.001478+0000","last_active":"2024-02-09T17:52:11.017171+0000","last_peered":"2024-02-09T17:52:11.017171+0000","last_clean":"2024-02-09T17:51:08.786164+0000","last_became_active":"2024-02-09T17:51:10.969838+0000","last_became_peered":"2024-02-09T17:51:10.969838+0000","last_unstale":"2024-02-09T17:52:11.017171+0000","last_undegraded":"2024-02-09T17:52:11.017171+0000","last_fullsized":"2024-02-09T17:51:10.794112+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_clean_scrub_stamp":"2024-02-09T17:50:17.733761+0000","objects_scrubbed":0,"log_size":7230,"log_dups_size":0,"ondisk_log_size":7230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395575296,"num_objects":6036,"num_object_clones":0,"num_object_copies":18108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2719,"num_objects_unfound":0,"num_objects_dirty":6036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7230,"num_write_kb":424512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2112,"num_bytes_recovered":138412032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,1],"acting":[11,9],"avail_no_missing":["11","0","9"],"object_location_counts":[{"shards":"0,9,11","objects":2719},{"shards":"9,11","objects":3317}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.3","version":"63'139","reported_seq":969,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-09T17:52:06.834056+0000","last_change":"2024-02-09T17:50:35.557106+0000","last_active":"2024-02-09T17:52:06.834056+0000","last_peered":"2024-02-09T17:52:06.834056+0000","last_clean":"2024-02-09T17:52:06.834056+0000","last_became_active":"2024-02-09T17:40:13.501711+0000","last_became_peered":"2024-02-09T17:40:13.501711+0000","last_unstale":"2024-02-09T17:52:06.834056+0000","last_undegraded":"2024-02-09T17:52:06.834056+0000","last_fullsized":"2024-02-09T17:52:06.834056+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:50:35.556968+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:46:40.684762+0000","last_clean_scrub_stamp":"2024-02-09T17:50:35.556968+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":188,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"706'7362","reported_seq":13286,"reported_epoch":706,"state":"active+recovering+undersized+remapped","last_fresh":"2024-02-09T17:52:10.565147+0000","last_change":"2024-02-09T17:51:10.993811+0000","last_active":"2024-02-09T17:52:10.565147+0000","last_peered":"2024-02-09T17:52:10.565147+0000","last_clean":"2024-02-09T17:51:08.789437+0000","last_became_active":"2024-02-09T17:51:10.968660+0000","last_became_peered":"2024-02-09T17:51:10.968660+0000","last_unstale":"2024-02-09T17:52:10.565147+0000","last_undegraded":"2024-02-09T17:52:10.565147+0000","last_fullsized":"2024-02-09T17:51:10.793133+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_clean_scrub_stamp":"2024-02-09T17:50:17.733761+0000","objects_scrubbed":0,"log_size":7362,"log_dups_size":0,"ondisk_log_size":7362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":408420352,"num_objects":6232,"num_object_clones":0,"num_object_copies":18696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2022,"num_objects_unfound":0,"num_objects_dirty":6232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7362,"num_write_kb":435008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2910,"num_bytes_recovered":190709760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,1],"acting":[6,8],"avail_no_missing":["6","3","8"],"object_location_counts":[{"shards":"3,6,8","objects":2022},{"shards":"6,8","objects":4210}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"63'139","reported_seq":504,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-09T17:52:06.837917+0000","last_change":"2024-02-09T17:51:12.783707+0000","last_active":"2024-02-09T17:52:06.837917+0000","last_peered":"2024-02-09T17:52:06.837917+0000","last_clean":"2024-02-09T17:52:06.837917+0000","last_became_active":"2024-02-09T17:51:12.782912+0000","last_became_peered":"2024-02-09T17:51:12.782912+0000","last_unstale":"2024-02-09T17:52:06.837917+0000","last_undegraded":"2024-02-09T17:52:06.837917+0000","last_fullsized":"2024-02-09T17:52:06.837917+0000","mapping_epoch":684,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":685,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:50:14.694755+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:50:14.694755+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,2],"acting":[10,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.0","version":"706'7465","reported_seq":7565,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-09T17:52:11.802130+0000","last_change":"2024-02-09T17:50:19.468462+0000","last_active":"2024-02-09T17:52:11.802130+0000","last_peered":"2024-02-09T17:52:11.802130+0000","last_clean":"2024-02-09T17:52:11.802130+0000","last_became_active":"2024-02-09T17:50:19.467979+0000","last_became_peered":"2024-02-09T17:50:19.467979+0000","last_unstale":"2024-02-09T17:52:11.802130+0000","last_undegraded":"2024-02-09T17:52:11.802130+0000","last_fullsized":"2024-02-09T17:52:11.802130+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_clean_scrub_stamp":"2024-02-09T17:50:17.733761+0000","objects_scrubbed":0,"log_size":7465,"log_dups_size":0,"ondisk_log_size":7465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404029440,"num_objects":6165,"num_object_clones":0,"num_object_copies":18495,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7465,"num_write_kb":436160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,3],"acting":[9,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.6","version":"63'139","reported_seq":1281,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-09T17:52:06.827588+0000","last_change":"2024-02-09T17:49:35.724086+0000","last_active":"2024-02-09T17:52:06.827588+0000","last_peered":"2024-02-09T17:52:06.827588+0000","last_clean":"2024-02-09T17:52:06.827588+0000","last_became_active":"2024-02-09T17:43:14.024894+0000","last_became_peered":"2024-02-09T17:43:14.024894+0000","last_unstale":"2024-02-09T17:52:06.827588+0000","last_undegraded":"2024-02-09T17:52:06.827588+0000","last_fullsized":"2024-02-09T17:52:06.827588+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:49:35.723959+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:11.970338+0000","last_clean_scrub_stamp":"2024-02-09T17:49:35.723959+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.1","version":"706'7423","reported_seq":14377,"reported_epoch":706,"state":"active+recovering+undersized+remapped","last_fresh":"2024-02-09T17:52:11.331274+0000","last_change":"2024-02-09T17:51:11.052388+0000","last_active":"2024-02-09T17:52:11.331274+0000","last_peered":"2024-02-09T17:52:11.331274+0000","last_clean":"2024-02-09T17:51:08.773364+0000","last_became_active":"2024-02-09T17:51:11.020599+0000","last_became_peered":"2024-02-09T17:51:11.020599+0000","last_unstale":"2024-02-09T17:52:11.331274+0000","last_undegraded":"2024-02-09T17:52:11.331274+0000","last_fullsized":"2024-02-09T17:51:10.790868+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_clean_scrub_stamp":"2024-02-09T17:50:17.733761+0000","objects_scrubbed":0,"log_size":7423,"log_dups_size":0,"ondisk_log_size":7423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405995520,"num_objects":6195,"num_object_clones":0,"num_object_copies":18585,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1544,"num_objects_unfound":0,"num_objects_dirty":6195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7423,"num_write_kb":435776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3426,"num_bytes_recovered":224526336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,6],"acting":[4,10],"avail_no_missing":["4","3","10"],"object_location_counts":[{"shards":"3,4,10","objects":1544},{"shards":"4,10","objects":4651}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"63'139","reported_seq":961,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-09T17:52:06.820977+0000","last_change":"2024-02-09T17:51:12.793131+0000","last_active":"2024-02-09T17:52:06.820977+0000","last_peered":"2024-02-09T17:52:06.820977+0000","last_clean":"2024-02-09T17:52:06.820977+0000","last_became_active":"2024-02-09T17:51:12.792798+0000","last_became_peered":"2024-02-09T17:51:12.792798+0000","last_unstale":"2024-02-09T17:52:06.820977+0000","last_undegraded":"2024-02-09T17:52:06.820977+0000","last_fullsized":"2024-02-09T17:52:06.820977+0000","mapping_epoch":684,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":685,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:50:38.171846+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:50:38.171846+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2","version":"706'7324","reported_seq":7493,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-09T17:52:12.747823+0000","last_change":"2024-02-09T17:52:11.244321+0000","last_active":"2024-02-09T17:52:12.747823+0000","last_peered":"2024-02-09T17:52:12.747823+0000","last_clean":"2024-02-09T17:52:12.747823+0000","last_became_active":"2024-02-09T17:50:19.468029+0000","last_became_peered":"2024-02-09T17:50:19.468029+0000","last_unstale":"2024-02-09T17:52:12.747823+0000","last_undegraded":"2024-02-09T17:52:12.747823+0000","last_fullsized":"2024-02-09T17:52:12.747823+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"706'7259","last_scrub_stamp":"2024-02-09T17:52:11.244135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_clean_scrub_stamp":"2024-02-09T17:52:11.244135+0000","objects_scrubbed":6119,"log_size":7324,"log_dups_size":0,"ondisk_log_size":7324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:53:31.276211+0000","scrub_duration":1352,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395706368,"num_objects":6038,"num_object_clones":0,"num_object_copies":18114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7324,"num_write_kb":427584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,2],"acting":[3,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"63'139","reported_seq":1359,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-09T17:52:06.820148+0000","last_change":"2024-02-09T17:50:34.344695+0000","last_active":"2024-02-09T17:52:06.820148+0000","last_peered":"2024-02-09T17:52:06.820148+0000","last_clean":"2024-02-09T17:52:06.820148+0000","last_became_active":"2024-02-09T17:33:00.284924+0000","last_became_peered":"2024-02-09T17:33:00.284924+0000","last_unstale":"2024-02-09T17:52:06.820148+0000","last_undegraded":"2024-02-09T17:52:06.820148+0000","last_fullsized":"2024-02-09T17:52:06.820148+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:50:34.344581+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:56.105792+0000","last_clean_scrub_stamp":"2024-02-09T17:50:34.344581+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.3","version":"706'7460","reported_seq":7560,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-09T17:52:12.729332+0000","last_change":"2024-02-09T17:50:18.752162+0000","last_active":"2024-02-09T17:52:12.729332+0000","last_peered":"2024-02-09T17:52:12.729332+0000","last_clean":"2024-02-09T17:52:12.729332+0000","last_became_active":"2024-02-09T17:50:18.751775+0000","last_became_peered":"2024-02-09T17:50:18.751775+0000","last_unstale":"2024-02-09T17:52:12.729332+0000","last_undegraded":"2024-02-09T17:52:12.729332+0000","last_fullsized":"2024-02-09T17:52:12.729332+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_clean_scrub_stamp":"2024-02-09T17:50:17.733761+0000","objects_scrubbed":0,"log_size":7460,"log_dups_size":0,"ondisk_log_size":7460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400031744,"num_objects":6104,"num_object_clones":0,"num_object_copies":18312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7460,"num_write_kb":434048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"63'139","reported_seq":1200,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-09T17:52:06.825278+0000","last_change":"2024-02-09T17:49:36.733644+0000","last_active":"2024-02-09T17:52:06.825278+0000","last_peered":"2024-02-09T17:52:06.825278+0000","last_clean":"2024-02-09T17:52:06.825278+0000","last_became_active":"2024-02-09T17:48:46.184307+0000","last_became_peered":"2024-02-09T17:48:46.184307+0000","last_unstale":"2024-02-09T17:52:06.825278+0000","last_undegraded":"2024-02-09T17:52:06.825278+0000","last_fullsized":"2024-02-09T17:52:06.825278+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:49:36.733515+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:49:36.733515+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.c","version":"706'7372","reported_seq":13410,"reported_epoch":706,"state":"active+recovering+undersized+remapped","last_fresh":"2024-02-09T17:52:10.564548+0000","last_change":"2024-02-09T17:51:11.212222+0000","last_active":"2024-02-09T17:52:10.564548+0000","last_peered":"2024-02-09T17:52:10.564548+0000","last_clean":"2024-02-09T17:51:08.767034+0000","last_became_active":"2024-02-09T17:51:11.168956+0000","last_became_peered":"2024-02-09T17:51:11.168956+0000","last_unstale":"2024-02-09T17:52:10.564548+0000","last_undegraded":"2024-02-09T17:52:10.564548+0000","last_fullsized":"2024-02-09T17:51:10.792405+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_clean_scrub_stamp":"2024-02-09T17:50:17.733761+0000","objects_scrubbed":0,"log_size":7372,"log_dups_size":0,"ondisk_log_size":7372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":409731072,"num_objects":6252,"num_object_clones":0,"num_object_copies":18756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2002,"num_objects_unfound":0,"num_objects_dirty":6252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7372,"num_write_kb":435968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2969,"num_bytes_recovered":194576384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,2],"acting":[6,11],"avail_no_missing":["6","3","11"],"object_location_counts":[{"shards":"3,6,11","objects":2002},{"shards":"6,11","objects":4250}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"63'139","reported_seq":1274,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-09T17:52:06.819784+0000","last_change":"2024-02-09T17:49:39.575691+0000","last_active":"2024-02-09T17:52:06.819784+0000","last_peered":"2024-02-09T17:52:06.819784+0000","last_clean":"2024-02-09T17:52:06.819784+0000","last_became_active":"2024-02-09T17:43:14.041291+0000","last_became_peered":"2024-02-09T17:43:14.041291+0000","last_unstale":"2024-02-09T17:52:06.819784+0000","last_undegraded":"2024-02-09T17:52:06.819784+0000","last_fullsized":"2024-02-09T17:52:06.819784+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:49:39.575571+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:49:39.575571+0000","last_clean_scrub_stamp":"2024-02-09T17:49:39.575571+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.d","version":"706'7315","reported_seq":7416,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-09T17:52:10.357038+0000","last_change":"2024-02-09T17:50:18.752741+0000","last_active":"2024-02-09T17:52:10.357038+0000","last_peered":"2024-02-09T17:52:10.357038+0000","last_clean":"2024-02-09T17:52:10.357038+0000","last_became_active":"2024-02-09T17:50:18.752364+0000","last_became_peered":"2024-02-09T17:50:18.752364+0000","last_unstale":"2024-02-09T17:52:10.357038+0000","last_undegraded":"2024-02-09T17:52:10.357038+0000","last_fullsized":"2024-02-09T17:52:10.357038+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_clean_scrub_stamp":"2024-02-09T17:50:17.733761+0000","objects_scrubbed":0,"log_size":7315,"log_dups_size":0,"ondisk_log_size":7315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405012504,"num_objects":6181,"num_object_clones":0,"num_object_copies":18543,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6181,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7315,"num_write_kb":431809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"63'139","reported_seq":646,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-09T17:52:06.811624+0000","last_change":"2024-02-09T17:51:12.786031+0000","last_active":"2024-02-09T17:52:06.811624+0000","last_peered":"2024-02-09T17:52:06.811624+0000","last_clean":"2024-02-09T17:52:06.811624+0000","last_became_active":"2024-02-09T17:51:12.785679+0000","last_became_peered":"2024-02-09T17:51:12.785679+0000","last_unstale":"2024-02-09T17:52:06.811624+0000","last_undegraded":"2024-02-09T17:52:06.811624+0000","last_fullsized":"2024-02-09T17:52:06.811624+0000","mapping_epoch":684,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":685,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:50:22.355668+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:50:22.355668+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:52:11.388037+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"706'7355","reported_seq":7455,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-09T17:52:10.168146+0000","last_change":"2024-02-09T17:50:19.468403+0000","last_active":"2024-02-09T17:52:10.168146+0000","last_peered":"2024-02-09T17:52:10.168146+0000","last_clean":"2024-02-09T17:52:10.168146+0000","last_became_active":"2024-02-09T17:50:19.468067+0000","last_became_peered":"2024-02-09T17:50:19.468067+0000","last_unstale":"2024-02-09T17:52:10.168146+0000","last_undegraded":"2024-02-09T17:52:10.168146+0000","last_fullsized":"2024-02-09T17:52:10.168146+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_clean_scrub_stamp":"2024-02-09T17:50:17.733761+0000","objects_scrubbed":0,"log_size":7355,"log_dups_size":0,"ondisk_log_size":7355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410189824,"num_objects":6259,"num_object_clones":0,"num_object_copies":18777,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7355,"num_write_kb":435648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"63'139","reported_seq":923,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-09T17:52:06.835594+0000","last_change":"2024-02-09T17:51:21.668217+0000","last_active":"2024-02-09T17:52:06.835594+0000","last_peered":"2024-02-09T17:52:06.835594+0000","last_clean":"2024-02-09T17:52:06.835594+0000","last_became_active":"2024-02-09T17:46:47.013853+0000","last_became_peered":"2024-02-09T17:46:47.013853+0000","last_unstale":"2024-02-09T17:52:06.835594+0000","last_undegraded":"2024-02-09T17:52:06.835594+0000","last_fullsized":"2024-02-09T17:52:06.835594+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:51:21.668120+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:51:21.668120+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:52:37.602252+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"706'7413","reported_seq":10839,"reported_epoch":706,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-02-09T17:52:12.212609+0000","last_change":"2024-02-09T17:51:11.117975+0000","last_active":"2024-02-09T17:52:12.212609+0000","last_peered":"2024-02-09T17:52:12.212609+0000","last_clean":"2024-02-09T17:51:08.767639+0000","last_became_active":"2024-02-09T17:51:11.049958+0000","last_became_peered":"2024-02-09T17:51:11.049958+0000","last_unstale":"2024-02-09T17:52:12.212609+0000","last_undegraded":"2024-02-09T17:51:10.956956+0000","last_fullsized":"2024-02-09T17:51:10.793151+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_clean_scrub_stamp":"2024-02-09T17:50:17.733761+0000","objects_scrubbed":0,"log_size":7413,"log_dups_size":0,"ondisk_log_size":7413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402849792,"num_objects":6147,"num_object_clones":0,"num_object_copies":18441,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":3295,"num_objects_unfound":0,"num_objects_dirty":6147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7413,"num_write_kb":433920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1661,"num_bytes_recovered":108855296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,11],"acting":[8,9],"avail_no_missing":["8","0","9"],"object_location_counts":[{"shards":"0,8,9","objects":3295},{"shards":"8,9","objects":2851}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.9","version":"63'139","reported_seq":1787,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-09T17:52:06.826059+0000","last_change":"2024-02-09T17:50:15.083453+0000","last_active":"2024-02-09T17:52:06.826059+0000","last_peered":"2024-02-09T17:52:06.826059+0000","last_clean":"2024-02-09T17:52:06.826059+0000","last_became_active":"2024-02-09T17:43:28.590237+0000","last_became_peered":"2024-02-09T17:43:28.590237+0000","last_unstale":"2024-02-09T17:52:06.826059+0000","last_undegraded":"2024-02-09T17:52:06.826059+0000","last_fullsized":"2024-02-09T17:52:06.826059+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:50:15.083356+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:45:47.710252+0000","last_clean_scrub_stamp":"2024-02-09T17:50:15.083356+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"706'7327","reported_seq":7427,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-09T17:52:11.334665+0000","last_change":"2024-02-09T17:50:18.754173+0000","last_active":"2024-02-09T17:52:11.334665+0000","last_peered":"2024-02-09T17:52:11.334665+0000","last_clean":"2024-02-09T17:52:11.334665+0000","last_became_active":"2024-02-09T17:50:18.753808+0000","last_became_peered":"2024-02-09T17:50:18.753808+0000","last_unstale":"2024-02-09T17:52:11.334665+0000","last_undegraded":"2024-02-09T17:52:11.334665+0000","last_fullsized":"2024-02-09T17:52:11.334665+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_clean_scrub_stamp":"2024-02-09T17:50:17.733761+0000","objects_scrubbed":0,"log_size":7327,"log_dups_size":0,"ondisk_log_size":7327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":401408000,"num_objects":6125,"num_object_clones":0,"num_object_copies":18375,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7327,"num_write_kb":430464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"63'139","reported_seq":1243,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-09T17:52:06.819468+0000","last_change":"2024-02-09T17:50:16.807981+0000","last_active":"2024-02-09T17:52:06.819468+0000","last_peered":"2024-02-09T17:52:06.819468+0000","last_clean":"2024-02-09T17:52:06.819468+0000","last_became_active":"2024-02-09T17:43:14.036583+0000","last_became_peered":"2024-02-09T17:43:14.036583+0000","last_unstale":"2024-02-09T17:52:06.819468+0000","last_undegraded":"2024-02-09T17:52:06.819468+0000","last_fullsized":"2024-02-09T17:52:06.819468+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:50:16.807851+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:14.981453+0000","last_clean_scrub_stamp":"2024-02-09T17:50:16.807851+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"706'7244","reported_seq":10701,"reported_epoch":706,"state":"active+recovering+undersized+remapped","last_fresh":"2024-02-09T17:52:11.012625+0000","last_change":"2024-02-09T17:51:11.054316+0000","last_active":"2024-02-09T17:52:11.012625+0000","last_peered":"2024-02-09T17:52:11.012625+0000","last_clean":"2024-02-09T17:51:08.632960+0000","last_became_active":"2024-02-09T17:51:11.022701+0000","last_became_peered":"2024-02-09T17:51:11.022701+0000","last_unstale":"2024-02-09T17:52:11.012625+0000","last_undegraded":"2024-02-09T17:52:11.012625+0000","last_fullsized":"2024-02-09T17:51:10.793802+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_clean_scrub_stamp":"2024-02-09T17:50:17.733761+0000","objects_scrubbed":0,"log_size":7244,"log_dups_size":0,"ondisk_log_size":7244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395051008,"num_objects":6028,"num_object_clones":0,"num_object_copies":18084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3165,"num_objects_unfound":0,"num_objects_dirty":6028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7244,"num_write_kb":424704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1677,"num_bytes_recovered":109903872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,10],"acting":[11,8],"avail_no_missing":["11","0","8"],"object_location_counts":[{"shards":"0,8,11","objects":3165},{"shards":"8,11","objects":2863}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.f","version":"63'139","reported_seq":1304,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-09T17:52:06.820879+0000","last_change":"2024-02-09T17:50:16.087650+0000","last_active":"2024-02-09T17:52:06.820879+0000","last_peered":"2024-02-09T17:52:06.820879+0000","last_clean":"2024-02-09T17:52:06.820879+0000","last_became_active":"2024-02-09T17:33:05.316100+0000","last_became_peered":"2024-02-09T17:33:05.316100+0000","last_unstale":"2024-02-09T17:52:06.820879+0000","last_undegraded":"2024-02-09T17:52:06.820879+0000","last_fullsized":"2024-02-09T17:52:06.820879+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:50:16.087552+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:48:34.651647+0000","last_clean_scrub_stamp":"2024-02-09T17:50:16.087552+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":141,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"706'7380","reported_seq":7480,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-09T17:52:11.829489+0000","last_change":"2024-02-09T17:50:18.752322+0000","last_active":"2024-02-09T17:52:11.829489+0000","last_peered":"2024-02-09T17:52:11.829489+0000","last_clean":"2024-02-09T17:52:11.829489+0000","last_became_active":"2024-02-09T17:50:18.752003+0000","last_became_peered":"2024-02-09T17:50:18.752003+0000","last_unstale":"2024-02-09T17:52:11.829489+0000","last_undegraded":"2024-02-09T17:52:11.829489+0000","last_fullsized":"2024-02-09T17:52:11.829489+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_clean_scrub_stamp":"2024-02-09T17:50:17.733761+0000","objects_scrubbed":0,"log_size":7380,"log_dups_size":0,"ondisk_log_size":7380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":401735680,"num_objects":6130,"num_object_clones":0,"num_object_copies":18390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7380,"num_write_kb":432320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.c","version":"63'139","reported_seq":1430,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-09T17:52:06.835704+0000","last_change":"2024-02-09T17:51:09.611200+0000","last_active":"2024-02-09T17:52:06.835704+0000","last_peered":"2024-02-09T17:52:06.835704+0000","last_clean":"2024-02-09T17:52:06.835704+0000","last_became_active":"2024-02-09T17:43:14.027715+0000","last_became_peered":"2024-02-09T17:43:14.027715+0000","last_unstale":"2024-02-09T17:52:06.835704+0000","last_undegraded":"2024-02-09T17:52:06.835704+0000","last_fullsized":"2024-02-09T17:52:06.835704+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:51:09.611122+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:14.135321+0000","last_clean_scrub_stamp":"2024-02-09T17:51:09.611122+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:52:37.968398+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"706'7204","reported_seq":7304,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-09T17:52:09.969963+0000","last_change":"2024-02-09T17:50:19.468334+0000","last_active":"2024-02-09T17:52:09.969963+0000","last_peered":"2024-02-09T17:52:09.969963+0000","last_clean":"2024-02-09T17:52:09.969963+0000","last_became_active":"2024-02-09T17:50:19.467991+0000","last_became_peered":"2024-02-09T17:50:19.467991+0000","last_unstale":"2024-02-09T17:52:09.969963+0000","last_undegraded":"2024-02-09T17:52:09.969963+0000","last_fullsized":"2024-02-09T17:52:09.969963+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:50:17.733761+0000","last_clean_scrub_stamp":"2024-02-09T17:50:17.733761+0000","objects_scrubbed":0,"log_size":7204,"log_dups_size":0,"ondisk_log_size":7204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406061056,"num_objects":6196,"num_object_clones":0,"num_object_copies":18588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7204,"num_write_kb":428800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"63'139","reported_seq":835,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-09T17:52:06.825673+0000","last_change":"2024-02-09T17:50:18.057735+0000","last_active":"2024-02-09T17:52:06.825673+0000","last_peered":"2024-02-09T17:52:06.825673+0000","last_clean":"2024-02-09T17:52:06.825673+0000","last_became_active":"2024-02-09T17:43:14.036759+0000","last_became_peered":"2024-02-09T17:43:14.036759+0000","last_unstale":"2024-02-09T17:52:06.825673+0000","last_undegraded":"2024-02-09T17:52:06.825673+0000","last_fullsized":"2024-02-09T17:52:06.825673+0000","mapping_epoch":454,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:50:18.057634+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:50:18.057634+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"63'139","reported_seq":1007,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-09T17:52:06.825540+0000","last_change":"2024-02-09T17:50:21.030323+0000","last_active":"2024-02-09T17:52:06.825540+0000","last_peered":"2024-02-09T17:52:06.825540+0000","last_clean":"2024-02-09T17:52:06.825540+0000","last_became_active":"2024-02-09T17:46:47.013920+0000","last_became_peered":"2024-02-09T17:46:47.013920+0000","last_unstale":"2024-02-09T17:52:06.825540+0000","last_undegraded":"2024-02-09T17:52:06.825540+0000","last_fullsized":"2024-02-09T17:52:06.825540+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:50:21.030206+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:48:35.170638+0000","last_clean_scrub_stamp":"2024-02-09T17:50:21.030206+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"63'139","reported_seq":1260,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-09T17:52:06.808965+0000","last_change":"2024-02-09T17:51:12.798969+0000","last_active":"2024-02-09T17:52:06.808965+0000","last_peered":"2024-02-09T17:52:06.808965+0000","last_clean":"2024-02-09T17:52:06.808965+0000","last_became_active":"2024-02-09T17:51:12.798632+0000","last_became_peered":"2024-02-09T17:51:12.798632+0000","last_unstale":"2024-02-09T17:52:06.808965+0000","last_undegraded":"2024-02-09T17:52:06.808965+0000","last_fullsized":"2024-02-09T17:52:06.808965+0000","mapping_epoch":684,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":685,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:49:31.135755+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:44:51.190789+0000","last_clean_scrub_stamp":"2024-02-09T17:49:31.135755+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":343,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,3],"acting":[8,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.12","version":"63'139","reported_seq":1344,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-09T17:52:06.834046+0000","last_change":"2024-02-09T17:50:23.374531+0000","last_active":"2024-02-09T17:52:06.834046+0000","last_peered":"2024-02-09T17:52:06.834046+0000","last_clean":"2024-02-09T17:52:06.834046+0000","last_became_active":"2024-02-09T17:46:47.015622+0000","last_became_peered":"2024-02-09T17:46:47.015622+0000","last_unstale":"2024-02-09T17:52:06.834046+0000","last_undegraded":"2024-02-09T17:52:06.834046+0000","last_fullsized":"2024-02-09T17:52:06.834046+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:50:23.374413+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:50:23.374413+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.13","version":"63'139","reported_seq":963,"reported_epoch":706,"state":"active+clean","last_fresh":"2024-02-09T17:52:06.834156+0000","last_change":"2024-02-09T17:50:38.611124+0000","last_active":"2024-02-09T17:52:06.834156+0000","last_peered":"2024-02-09T17:52:06.834156+0000","last_clean":"2024-02-09T17:52:06.834156+0000","last_became_active":"2024-02-09T17:41:14.521672+0000","last_became_peered":"2024-02-09T17:41:14.521672+0000","last_unstale":"2024-02-09T17:52:06.834156+0000","last_undegraded":"2024-02-09T17:52:06.834156+0000","last_fullsized":"2024-02-09T17:52:06.834156+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:50:38.611046+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:50:38.611046+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":6442713112,"num_objects":98309,"num_object_clones":0,"num_object_copies":294927,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":20041,"num_objects_unfound":0,"num_objects_dirty":98309,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117269,"num_write_kb":6898433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19192,"num_bytes_recovered":1257766912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20624060416,"data_stored":20624048200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117269,"ondisk_log_size":117269,"up":48,"acting":40,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547056999,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2832516,"kb_used_data":2151612,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":90912636,"statfs":{"total":95995035648,"available":93094539264,"internally_reserved":0,"allocated":2203250688,"data_stored":2202762573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":697235908},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:51:56 2024","interfaces":[{"interface":"back","average":{"1min":3.4529999999999998,"5min":1.764,"15min":1.4870000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":20.003,"5min":20.003,"15min":20.003},"last":5.8019999999999996},{"interface":"front","average":{"1min":1.349,"5min":1.2909999999999999,"15min":1.2110000000000001},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.23100000000000001},"max":{"1min":6.7439999999999998,"5min":6.7439999999999998,"15min":9.8219999999999992},"last":1.028}]},{"osd":1,"last update":"Fri Feb 9 17:51:56 2024","interfaces":[{"interface":"back","average":{"1min":1.48,"5min":1.7270000000000001,"15min":1.506},"min":{"1min":0.247,"5min":0.247,"15min":0.24299999999999999},"max":{"1min":6.7050000000000001,"5min":11.576000000000001,"15min":11.576000000000001},"last":1.929},{"interface":"front","average":{"1min":2.0379999999999998,"5min":2.0640000000000001,"15min":1.6970000000000001},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":14.287000000000001,"5min":14.287000000000001,"15min":14.287000000000001},"last":4.8150000000000004}]},{"osd":2,"last update":"Fri Feb 9 17:31:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.415,"15min":1.6859999999999999},"min":{"1min":0.378,"5min":0.247,"15min":0.247},"max":{"1min":4.4189999999999996,"5min":9.4719999999999995,"15min":9.4719999999999995},"last":0.35399999999999998},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.5149999999999999,"15min":1.4399999999999999},"min":{"1min":0.378,"5min":0.252,"15min":0.252},"max":{"1min":4.8579999999999997,"5min":7.96,"15min":7.96},"last":0.60399999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:51:56 2024","interfaces":[{"interface":"back","average":{"1min":1.819,"5min":1.9179999999999999,"15min":1.573},"min":{"1min":0.32200000000000001,"5min":0.27900000000000003,"15min":0.248},"max":{"1min":9.4130000000000003,"5min":9.4130000000000003,"15min":21.475000000000001},"last":4.9870000000000001},{"interface":"front","average":{"1min":1.8600000000000001,"5min":2.0059999999999998,"15min":1.8100000000000001},"min":{"1min":0.375,"5min":0.375,"15min":0.29799999999999999},"max":{"1min":9.5419999999999998,"5min":9.5419999999999998,"15min":21.498000000000001},"last":5.0099999999999998}]},{"osd":4,"last update":"Fri Feb 9 17:51:56 2024","interfaces":[{"interface":"back","average":{"1min":3.6760000000000002,"5min":1.6859999999999999,"15min":1.764},"min":{"1min":0.29199999999999998,"5min":0.26300000000000001,"15min":0.246},"max":{"1min":19.956,"5min":19.956,"15min":65.109999999999999},"last":5.8710000000000004},{"interface":"front","average":{"1min":1.6990000000000001,"5min":1.355,"15min":1.71},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":6.7869999999999999,"5min":6.7869999999999999,"15min":65.099000000000004},"last":1.044}]},{"osd":5,"last update":"Fri Feb 9 17:51:56 2024","interfaces":[{"interface":"back","average":{"1min":1.6799999999999999,"5min":1.3580000000000001,"15min":1.4770000000000001},"min":{"1min":0.34899999999999998,"5min":0.252,"15min":0.252},"max":{"1min":9.5169999999999995,"5min":9.5169999999999995,"15min":18.878},"last":5.8419999999999996},{"interface":"front","average":{"1min":3.5699999999999998,"5min":1.601,"15min":1.6120000000000001},"min":{"1min":0.313,"5min":0.29199999999999998,"15min":0.28999999999999998},"max":{"1min":20.038,"5min":20.038,"15min":20.038},"last":4.8540000000000001}]},{"osd":6,"last update":"Fri Feb 9 17:51:56 2024","interfaces":[{"interface":"back","average":{"1min":1.887,"5min":1.3380000000000001,"15min":1.5900000000000001},"min":{"1min":0.34300000000000003,"5min":0.308,"15min":0.29899999999999999},"max":{"1min":6.8170000000000002,"5min":6.8170000000000002,"15min":12.362},"last":2.7549999999999999},{"interface":"front","average":{"1min":2.2349999999999999,"5min":1.534,"15min":1.6579999999999999},"min":{"1min":0.39600000000000002,"5min":0.38400000000000001,"15min":0.32700000000000001},"max":{"1min":9.4789999999999992,"5min":9.4789999999999992,"15min":10.000999999999999},"last":4.9610000000000003}]},{"osd":7,"last update":"Fri Feb 9 17:51:56 2024","interfaces":[{"interface":"back","average":{"1min":2.3300000000000001,"5min":1.571,"15min":1.7170000000000001},"min":{"1min":0.38100000000000001,"5min":0.317,"15min":0.317},"max":{"1min":9.5660000000000007,"5min":9.5660000000000007,"15min":20.489999999999998},"last":4.9329999999999998},{"interface":"front","average":{"1min":4.3579999999999997,"5min":2.105,"15min":1.8},"min":{"1min":0.35699999999999998,"5min":0.318,"15min":0.318},"max":{"1min":24.373999999999999,"5min":24.373999999999999,"15min":24.373999999999999},"last":5.8959999999999999}]},{"osd":8,"last update":"Fri Feb 9 17:52:08 2024","interfaces":[{"interface":"back","average":{"1min":3.1520000000000001,"5min":1.2230000000000001,"15min":1.244},"min":{"1min":0.55700000000000005,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":13.574,"5min":13.574,"15min":13.574},"last":1.0029999999999999},{"interface":"front","average":{"1min":3.1850000000000001,"5min":1.2809999999999999,"15min":1.3200000000000001},"min":{"1min":0.57999999999999996,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":13.606,"5min":13.606,"15min":13.606},"last":1.0760000000000001}]},{"osd":9,"last update":"Fri Feb 9 17:51:37 2024","interfaces":[{"interface":"back","average":{"1min":2.2650000000000001,"5min":1.2529999999999999,"15min":1.508},"min":{"1min":0.42999999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":14.337,"5min":14.337,"15min":62.518000000000001},"last":4.9059999999999997},{"interface":"front","average":{"1min":1.708,"5min":1.157,"15min":1.4319999999999999},"min":{"1min":0.48099999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":9.4489999999999998,"5min":9.4489999999999998,"15min":63.241999999999997},"last":4.8810000000000002}]},{"osd":10,"last update":"Fri Feb 9 17:51:56 2024","interfaces":[{"interface":"back","average":{"1min":2.9900000000000002,"5min":1.3160000000000001,"15min":1.385},"min":{"1min":0.50800000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":12.359,"5min":12.359,"15min":12.359},"last":1.0589999999999999},{"interface":"front","average":{"1min":2.944,"5min":1.4159999999999999,"15min":1.4570000000000001},"min":{"1min":0.58999999999999997,"5min":0.39400000000000002,"15min":0.377},"max":{"1min":9.5909999999999993,"5min":9.5909999999999993,"15min":12.409000000000001},"last":5.0339999999999998}]}]},{"osd":10,"up_from":84,"seq":360777253230,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2196360,"kb_used_data":1615228,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":91548792,"statfs":{"total":95995035648,"available":93745963008,"internally_reserved":0,"allocated":1653993472,"data_stored":1653505357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":594868672},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:51:41 2024","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":1.2490000000000001,"15min":1.1639999999999999},"min":{"1min":0.27100000000000002,"5min":0.23999999999999999,"15min":0.20599999999999999},"max":{"1min":6.1710000000000003,"5min":8.1669999999999998,"15min":9.343},"last":0.39300000000000002},{"interface":"front","average":{"1min":1.365,"5min":1.206,"15min":1.113},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.248},"max":{"1min":6.2450000000000001,"5min":7.2830000000000004,"15min":7.2830000000000004},"last":0.84599999999999997}]},{"osd":1,"last update":"Fri Feb 9 17:51:04 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":1.4430000000000001,"15min":0.97699999999999998},"min":{"1min":0.307,"5min":0.246,"15min":0.23999999999999999},"max":{"1min":3.464,"5min":9.3819999999999997,"15min":9.3819999999999997},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.81599999999999995,"5min":1.5249999999999999,"15min":1.1279999999999999},"min":{"1min":0.29899999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":4.3049999999999997,"5min":8.9789999999999992,"15min":8.9789999999999992},"last":0.42499999999999999}]},{"osd":2,"last update":"Fri Feb 9 17:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":1.6459999999999999,"15min":1.5329999999999999},"min":{"1min":0.32200000000000001,"5min":0.28299999999999997,"15min":0.27600000000000002},"max":{"1min":3.4060000000000001,"5min":9.2539999999999996,"15min":11.384},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.81699999999999995,"5min":1.9610000000000001,"15min":1.4570000000000001},"min":{"1min":0.47599999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":2.9489999999999998,"5min":38.868000000000002,"15min":38.868000000000002},"last":0.54700000000000004}]},{"osd":3,"last update":"Fri Feb 9 17:52:06 2024","interfaces":[{"interface":"back","average":{"1min":2.452,"5min":1.9330000000000001,"15min":1.7090000000000001},"min":{"1min":0.27900000000000003,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":12.465,"5min":13.861000000000001,"15min":35.113},"last":0.42899999999999999},{"interface":"front","average":{"1min":2.956,"5min":2.2549999999999999,"15min":1.9339999999999999},"min":{"1min":0.33900000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":9.3160000000000007,"5min":14.276999999999999,"15min":34.359000000000002},"last":9.3160000000000007}]},{"osd":4,"last update":"Fri Feb 9 17:51:30 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.0389999999999999,"15min":1.1339999999999999},"min":{"1min":0.316,"5min":0.29799999999999999,"15min":0.29199999999999998},"max":{"1min":6.2800000000000002,"5min":6.851,"15min":9.0990000000000002},"last":0.46999999999999997},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.147,"15min":1.149},"min":{"1min":0.34300000000000003,"5min":0.33600000000000002,"15min":0.27000000000000002},"max":{"1min":6.3109999999999999,"5min":8.4610000000000003,"15min":11.273999999999999},"last":2.6899999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":1.234,"15min":1.224},"min":{"1min":0.36799999999999999,"5min":0.32800000000000001,"15min":0.28799999999999998},"max":{"1min":5.9029999999999996,"5min":7.9900000000000002,"15min":7.9900000000000002},"last":3.4940000000000002},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.2849999999999999,"15min":1.3799999999999999},"min":{"1min":0.42399999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":10.401999999999999,"5min":10.401999999999999,"15min":10.401999999999999},"last":0.57299999999999995}]},{"osd":6,"last update":"Fri Feb 9 17:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":1.2569999999999999,"15min":1.4279999999999999},"min":{"1min":0.34300000000000003,"5min":0.33600000000000002,"15min":0.308},"max":{"1min":3.3849999999999998,"5min":9.0820000000000007,"15min":10.962},"last":3.3879999999999999},{"interface":"front","average":{"1min":1.228,"5min":1.349,"15min":1.355},"min":{"1min":0.42499999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":10.346,"5min":10.346,"15min":10.346},"last":0.49199999999999999}]},{"osd":7,"last update":"Fri Feb 9 17:39:52 2024","interfaces":[{"interface":"back","average":{"1min":1.6000000000000001,"5min":1.367,"15min":1.415},"min":{"1min":0.42099999999999999,"5min":0.34200000000000003,"15min":0.312},"max":{"1min":4.0430000000000001,"5min":6.7590000000000003,"15min":17.754000000000001},"last":9.0760000000000005},{"interface":"front","average":{"1min":1.498,"5min":1.4359999999999999,"15min":1.5189999999999999},"min":{"1min":0.47399999999999998,"5min":0.38300000000000001,"15min":0.316},"max":{"1min":5.8170000000000002,"5min":11.308999999999999,"15min":11.308999999999999},"last":3.464}]},{"osd":8,"last update":"Fri Feb 9 17:52:06 2024","interfaces":[{"interface":"back","average":{"1min":2.0720000000000001,"5min":1.079,"15min":1.0640000000000001},"min":{"1min":0.33700000000000002,"5min":0.31,"15min":0.29099999999999998},"max":{"1min":7.5990000000000002,"5min":7.5990000000000002,"15min":7.5990000000000002},"last":3.4279999999999999},{"interface":"front","average":{"1min":3.8279999999999998,"5min":1.516,"15min":1.2909999999999999},"min":{"1min":0.51900000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":12.249000000000001,"5min":12.249000000000001,"15min":12.249000000000001},"last":9.2870000000000008}]},{"osd":9,"last update":"Fri Feb 9 17:51:15 2024","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":0.98399999999999999,"15min":1.1040000000000001},"min":{"1min":0.378,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":11.602,"5min":11.602,"15min":11.602},"last":9.1950000000000003},{"interface":"front","average":{"1min":1.2529999999999999,"5min":1.095,"15min":1.238},"min":{"1min":0.42399999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":6.7149999999999999,"5min":6.7149999999999999,"15min":10.622999999999999},"last":9.2590000000000003}]},{"osd":11,"last update":"Fri Feb 9 17:51:41 2024","interfaces":[{"interface":"back","average":{"1min":2.6179999999999999,"5min":1.1519999999999999,"15min":1.052},"min":{"1min":0.45600000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":23.579000000000001,"5min":23.579000000000001,"15min":23.579000000000001},"last":2.7330000000000001},{"interface":"front","average":{"1min":2.6579999999999999,"5min":1.2070000000000001,"15min":1.1080000000000001},"min":{"1min":0.52300000000000002,"5min":0.45600000000000002,"15min":0.438},"max":{"1min":23.619,"5min":23.619,"15min":23.619},"last":2.7639999999999998}]}]},{"osd":9,"up_from":77,"seq":330712482162,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2845700,"kb_used_data":2196732,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":90899452,"statfs":{"total":95995035648,"available":93081038848,"internally_reserved":0,"allocated":2249453568,"data_stored":2248965453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":664533440},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:51:12 2024","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":1.1359999999999999,"15min":0.91600000000000004},"min":{"1min":0.29399999999999998,"5min":0.215,"15min":0.215},"max":{"1min":4.508,"5min":7.1289999999999996,"15min":8.4670000000000005},"last":1.294},{"interface":"front","average":{"1min":0.87,"5min":1.1910000000000001,"15min":1.1870000000000001},"min":{"1min":0.26800000000000002,"5min":0.253,"15min":0.24299999999999999},"max":{"1min":3.3079999999999998,"5min":5.8719999999999999,"15min":11.596},"last":1.218}]},{"osd":1,"last update":"Fri Feb 9 17:52:01 2024","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.3740000000000001,"15min":1.8200000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.23499999999999999},"max":{"1min":5.2089999999999996,"5min":7.3490000000000002,"15min":89.515000000000001},"last":15.196},{"interface":"front","average":{"1min":2.5150000000000001,"5min":1.794,"15min":1.8680000000000001},"min":{"1min":0.28699999999999998,"5min":0.27100000000000002,"15min":0.249},"max":{"1min":10.821999999999999,"5min":11.685,"15min":89.388000000000005},"last":17.741}]},{"osd":2,"last update":"Fri Feb 9 17:51:25 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.774,"15min":1.728},"min":{"1min":0.29099999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":10.657999999999999,"5min":10.657999999999999,"15min":10.657999999999999},"last":1.3300000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":1.5169999999999999,"15min":1.573},"min":{"1min":0.30599999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.2010000000000001,"5min":8.4250000000000007,"15min":8.4250000000000007},"last":0.64200000000000002}]},{"osd":3,"last update":"Fri Feb 9 17:51:21 2024","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.179,"15min":1.179},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":4.3860000000000001,"5min":4.3860000000000001,"15min":4.3860000000000001},"last":1.4279999999999999},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":1.4510000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:52:01 2024","interfaces":[{"interface":"back","average":{"1min":2.7229999999999999,"5min":1.349,"15min":1.472},"min":{"1min":0.33900000000000002,"5min":0.28199999999999997,"15min":0.26800000000000002},"max":{"1min":10.763999999999999,"5min":10.763999999999999,"15min":27.381},"last":1.3560000000000001},{"interface":"front","average":{"1min":1.8859999999999999,"5min":1.337,"15min":1.3240000000000001},"min":{"1min":0.30299999999999999,"5min":0.26900000000000002,"15min":0.26100000000000001},"max":{"1min":5.2510000000000003,"5min":5.5300000000000002,"15min":27.355},"last":0.30399999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:52:05 2024","interfaces":[{"interface":"back","average":{"1min":1.99,"5min":1.5820000000000001,"15min":1.6830000000000001},"min":{"1min":0.35399999999999998,"5min":0.30299999999999999,"15min":0.26900000000000002},"max":{"1min":7.6189999999999998,"5min":8.9429999999999996,"15min":77.924000000000007},"last":1.3799999999999999},{"interface":"front","average":{"1min":2.2189999999999999,"5min":1.4850000000000001,"15min":1.726},"min":{"1min":0.41599999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":10.643000000000001,"5min":10.643000000000001,"15min":77.899000000000001},"last":0.748}]},{"osd":6,"last update":"Fri Feb 9 17:52:01 2024","interfaces":[{"interface":"back","average":{"1min":1.6799999999999999,"5min":1.29,"15min":1.498},"min":{"1min":0.37,"5min":0.314,"15min":0.30199999999999999},"max":{"1min":10.67,"5min":10.67,"15min":14.069000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":2.0430000000000001,"5min":1.5409999999999999,"15min":1.706},"min":{"1min":0.44600000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":10.721,"5min":10.721,"15min":14.097},"last":0.70899999999999996}]},{"osd":7,"last update":"Fri Feb 9 17:49:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.9670000000000001,"15min":1.8400000000000001},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.33400000000000002},"max":{"1min":3.7909999999999999,"5min":29.001999999999999,"15min":29.001999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":1.127,"5min":1.919,"15min":1.841},"min":{"1min":0.39400000000000002,"5min":0.373,"15min":0.32200000000000001},"max":{"1min":3.7749999999999999,"5min":28.872,"15min":28.872},"last":0.47399999999999998}]},{"osd":8,"last update":"Fri Feb 9 17:51:12 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.72599999999999998,"15min":1.0660000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":2.1110000000000002,"5min":4.2160000000000002,"15min":14.045999999999999},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.72099999999999997,"15min":1.234},"min":{"1min":0.35499999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":2.145,"5min":3.5310000000000001,"15min":12.885999999999999},"last":0.79300000000000004}]},{"osd":10,"last update":"Fri Feb 9 17:51:25 2024","interfaces":[{"interface":"back","average":{"1min":1.8320000000000001,"5min":1.05,"15min":1.3720000000000001},"min":{"1min":0.372,"5min":0.313,"15min":0.312},"max":{"1min":10.792999999999999,"5min":10.792999999999999,"15min":11.404},"last":17.814},{"interface":"front","average":{"1min":1.387,"5min":0.98299999999999998,"15min":1.2},"min":{"1min":0.40999999999999998,"5min":0.35799999999999998,"15min":0.34300000000000003},"max":{"1min":5.3079999999999998,"5min":5.5309999999999997,"15min":10.904},"last":15.323}]},{"osd":11,"last update":"Fri Feb 9 17:51:25 2024","interfaces":[{"interface":"back","average":{"1min":1.974,"5min":1.087,"15min":1.3560000000000001},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":10.731999999999999,"5min":10.731999999999999,"15min":14.02},"last":1.4039999999999999},{"interface":"front","average":{"1min":1.0309999999999999,"5min":0.77700000000000002,"15min":1.2070000000000001},"min":{"1min":0.39000000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":4.2809999999999997,"5min":4.2809999999999997,"15min":10.863},"last":0.66600000000000004}]}]},{"osd":8,"up_from":171,"seq":734439407905,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1546116,"kb_used_data":1175740,"kb_used_omap":212,"kb_used_meta":370027,"kb_avail":92199036,"statfs":{"total":95995035648,"available":94411812864,"internally_reserved":0,"allocated":1203957760,"data_stored":1203469645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217579,"internal_metadata":378908181},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:52:02 2024","interfaces":[{"interface":"back","average":{"1min":2.085,"5min":1.5329999999999999,"15min":1.073},"min":{"1min":0.309,"5min":0.22600000000000001,"15min":0.217},"max":{"1min":7.8369999999999997,"5min":12.712999999999999,"15min":12.712999999999999},"last":0.371},{"interface":"front","average":{"1min":1.181,"5min":1.6140000000000001,"15min":1.369},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23100000000000001},"max":{"1min":5.5129999999999999,"5min":12.725,"15min":12.725},"last":8.2690000000000001}]},{"osd":1,"last update":"Fri Feb 9 17:52:02 2024","interfaces":[{"interface":"back","average":{"1min":1.766,"5min":1.3700000000000001,"15min":1.3380000000000001},"min":{"1min":0.29399999999999998,"5min":0.27300000000000002,"15min":0.22700000000000001},"max":{"1min":7.9260000000000002,"5min":7.9260000000000002,"15min":8.718},"last":0.34000000000000002},{"interface":"front","average":{"1min":1.498,"5min":1.534,"15min":1.556},"min":{"1min":0.27800000000000002,"5min":0.26000000000000001,"15min":0.24099999999999999},"max":{"1min":5.6749999999999998,"5min":7.2359999999999998,"15min":25.027000000000001},"last":8.3170000000000002}]},{"osd":2,"last update":"Fri Feb 9 17:50:34 2024","interfaces":[{"interface":"back","average":{"1min":2.222,"5min":1.722,"15min":1.391},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28000000000000003},"max":{"1min":12.227,"5min":12.227,"15min":12.227},"last":0.44400000000000001},{"interface":"front","average":{"1min":2.5049999999999999,"5min":1.665,"15min":1.1240000000000001},"min":{"1min":0.42199999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":12.747999999999999,"5min":12.747999999999999,"15min":12.747999999999999},"last":0.36199999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:42:38 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":1.542,"15min":1.6379999999999999},"min":{"1min":0.316,"5min":0.29499999999999998,"15min":0.28699999999999998},"max":{"1min":2.347,"5min":15.228999999999999,"15min":24.030999999999999},"last":3.0110000000000001},{"interface":"front","average":{"1min":0.75600000000000001,"5min":1.8340000000000001,"15min":1.8640000000000001},"min":{"1min":0.36199999999999999,"5min":0.33600000000000002,"15min":0.30299999999999999},"max":{"1min":2.3719999999999999,"5min":14.988,"15min":24.318999999999999},"last":0.39600000000000002}]},{"osd":4,"last update":"Fri Feb 9 17:52:02 2024","interfaces":[{"interface":"back","average":{"1min":1.6910000000000001,"5min":1.1279999999999999,"15min":1.278},"min":{"1min":0.34999999999999998,"5min":0.29499999999999998,"15min":0.29099999999999998},"max":{"1min":7.4039999999999999,"5min":7.4039999999999999,"15min":12.936},"last":8.7959999999999994},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.4279999999999999,"15min":1.427},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.27000000000000002},"max":{"1min":5.5640000000000001,"5min":8.2360000000000007,"15min":9.6140000000000008},"last":8.8780000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:51:24 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.206,"15min":1.8580000000000001},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":7.7439999999999998,"5min":7.7439999999999998,"15min":7.7439999999999998},"last":3.0870000000000002},{"interface":"front","average":{"1min":0.77400000000000002,"5min":1.2529999999999999,"15min":1.696},"min":{"1min":0.40400000000000003,"5min":0.35199999999999998,"15min":0.29899999999999999},"max":{"1min":1.835,"5min":5.3940000000000001,"15min":6.383},"last":8.8260000000000005}]},{"osd":6,"last update":"Fri Feb 9 17:52:02 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.8109999999999999,"15min":1.508},"min":{"1min":0.313,"5min":0.29899999999999999,"15min":0.29799999999999999},"max":{"1min":9.6289999999999996,"5min":9.6289999999999996,"15min":13.031000000000001},"last":3.1629999999999998},{"interface":"front","average":{"1min":2.1749999999999998,"5min":1.5,"15min":1.464},"min":{"1min":0.34499999999999997,"5min":0.34000000000000002,"15min":0.29599999999999999},"max":{"1min":7.9669999999999996,"5min":9.4239999999999995,"15min":12.157999999999999},"last":0.64900000000000002}]},{"osd":7,"last update":"Fri Feb 9 17:52:02 2024","interfaces":[{"interface":"back","average":{"1min":2.823,"5min":1.8280000000000001,"15min":1.6619999999999999},"min":{"1min":0.55800000000000005,"5min":0.377,"15min":0.34100000000000003},"max":{"1min":7.4560000000000004,"5min":7.4560000000000004,"15min":11.278},"last":8.8539999999999992},{"interface":"front","average":{"1min":1.327,"5min":1.6910000000000001,"15min":1.641},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":5.6639999999999997,"5min":8.0449999999999999,"15min":11.313000000000001},"last":8.9090000000000007}]},{"osd":9,"last update":"Fri Feb 9 17:52:02 2024","interfaces":[{"interface":"back","average":{"1min":3.21,"5min":1.472,"15min":1.3160000000000001},"min":{"1min":0.34399999999999997,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":9.6010000000000009,"5min":9.6010000000000009,"15min":45.616},"last":3.0619999999999998},{"interface":"front","average":{"1min":3.2530000000000001,"5min":1.556,"15min":1.415},"min":{"1min":0.376,"5min":0.376,"15min":0.314},"max":{"1min":9.6829999999999998,"5min":9.6829999999999998,"15min":45.718000000000004},"last":3.1219999999999999}]},{"osd":10,"last update":"Fri Feb 9 17:52:02 2024","interfaces":[{"interface":"back","average":{"1min":2.4089999999999998,"5min":1.175,"15min":1.054},"min":{"1min":0.36499999999999999,"5min":0.32300000000000001,"15min":0.29699999999999999},"max":{"1min":7.7910000000000004,"5min":7.7910000000000004,"15min":7.8360000000000003},"last":0.67700000000000005},{"interface":"front","average":{"1min":3.5459999999999998,"5min":1.5369999999999999,"15min":1.302},"min":{"1min":0.39400000000000002,"5min":0.38700000000000001,"15min":0.34300000000000003},"max":{"1min":13.096,"5min":13.096,"15min":13.096},"last":3.1040000000000001}]},{"osd":11,"last update":"Fri Feb 9 17:52:02 2024","interfaces":[{"interface":"back","average":{"1min":4.1310000000000002,"5min":1.589,"15min":1.339},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.34000000000000002},"max":{"1min":15.893000000000001,"5min":15.893000000000001,"15min":15.893000000000001},"last":5.9710000000000001},{"interface":"front","average":{"1min":1.6579999999999999,"5min":1.145,"15min":1.2090000000000001},"min":{"1min":0.33900000000000002,"5min":0.32900000000000001,"15min":0.308},"max":{"1min":5.5810000000000004,"5min":6.5720000000000001,"15min":7.4359999999999999},"last":20.831}]}]},{"osd":7,"up_from":57,"seq":244813136250,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1487436,"kb_used_data":788484,"kb_used_omap":1,"kb_used_meta":698942,"kb_avail":92257716,"statfs":{"total":95995035648,"available":94471901184,"internally_reserved":0,"allocated":807407616,"data_stored":806915949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":715717066},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:51:40 2024","interfaces":[{"interface":"back","average":{"1min":5.6950000000000003,"5min":2.5049999999999999,"15min":1.7949999999999999},"min":{"1min":0.32900000000000001,"5min":0.249,"15min":0.249},"max":{"1min":101.167,"5min":101.167,"15min":101.167},"last":0.69799999999999995},{"interface":"front","average":{"1min":5.7569999999999997,"5min":2.3180000000000001,"15min":1.796},"min":{"1min":0.33300000000000002,"5min":0.26800000000000002,"15min":0.24099999999999999},"max":{"1min":101.07599999999999,"5min":101.07599999999999,"15min":101.07599999999999},"last":0.54700000000000004}]},{"osd":1,"last update":"Fri Feb 9 17:51:40 2024","interfaces":[{"interface":"back","average":{"1min":1.639,"5min":1.6140000000000001,"15min":1.877},"min":{"1min":0.313,"5min":0.30599999999999999,"15min":0.24199999999999999},"max":{"1min":13.212,"5min":13.212,"15min":21.704999999999998},"last":0.64000000000000001},{"interface":"front","average":{"1min":1.484,"5min":1.6479999999999999,"15min":2.0489999999999999},"min":{"1min":0.30499999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":8.9960000000000004,"5min":11.673,"15min":21.858000000000001},"last":0.627}]},{"osd":2,"last update":"Fri Feb 9 17:51:40 2024","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.6020000000000001,"15min":1.7909999999999999},"min":{"1min":0.35999999999999999,"5min":0.25,"15min":0.25},"max":{"1min":3.4950000000000001,"5min":6.4850000000000003,"15min":6.6340000000000003},"last":0.46300000000000002},{"interface":"front","average":{"1min":1.198,"5min":1.6850000000000001,"15min":2.0739999999999998},"min":{"1min":0.38800000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":8.3559999999999999,"5min":10.159000000000001,"15min":11.968},"last":0.78300000000000003}]},{"osd":3,"last update":"Fri Feb 9 17:52:01 2024","interfaces":[{"interface":"back","average":{"1min":1.74,"5min":3.4529999999999998,"15min":2.2269999999999999},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.25600000000000001},"max":{"1min":8.4380000000000006,"5min":78.644999999999996,"15min":78.644999999999996},"last":0.75800000000000001},{"interface":"front","average":{"1min":1.573,"5min":3.379,"15min":2.3370000000000002},"min":{"1min":0.41299999999999998,"5min":0.38300000000000001,"15min":0.308},"max":{"1min":6.4260000000000002,"5min":78.414000000000001,"15min":78.414000000000001},"last":0.68899999999999995}]},{"osd":4,"last update":"Fri Feb 9 17:51:40 2024","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.161,"15min":1.4350000000000001},"min":{"1min":0.28100000000000003,"5min":0.25600000000000001,"15min":0.23699999999999999},"max":{"1min":4.7270000000000003,"5min":7.2999999999999998,"15min":86.813000000000002},"last":0.58299999999999996},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.5489999999999999,"15min":1.6259999999999999},"min":{"1min":0.31,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":8.2880000000000003,"5min":14.087999999999999,"15min":86.751000000000005},"last":0.72899999999999998}]},{"osd":5,"last update":"Fri Feb 9 17:51:40 2024","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.4970000000000001,"15min":1.429},"min":{"1min":0.33800000000000002,"5min":0.30199999999999999,"15min":0.249},"max":{"1min":4.6539999999999999,"5min":8.0060000000000002,"15min":9.6590000000000007},"last":2.1040000000000001},{"interface":"front","average":{"1min":1.147,"5min":1.4450000000000001,"15min":1.504},"min":{"1min":0.35199999999999998,"5min":0.32100000000000001,"15min":0.31900000000000001},"max":{"1min":3.9929999999999999,"5min":7.867,"15min":15.228999999999999},"last":2.1179999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:52:01 2024","interfaces":[{"interface":"back","average":{"1min":1.7170000000000001,"5min":1.514,"15min":1.708},"min":{"1min":0.377,"5min":0.35699999999999998,"15min":0.28100000000000003},"max":{"1min":6.3520000000000003,"5min":7.8879999999999999,"15min":15.275},"last":0.80900000000000005},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.385,"15min":1.554},"min":{"1min":0.46100000000000002,"5min":0.35199999999999998,"15min":0.29199999999999998},"max":{"1min":4.6980000000000004,"5min":7.1669999999999998,"15min":18.236999999999998},"last":0.61299999999999999}]},{"osd":8,"last update":"Fri Feb 9 17:52:01 2024","interfaces":[{"interface":"back","average":{"1min":1.774,"5min":1.506,"15min":1.4550000000000001},"min":{"1min":0.53300000000000003,"5min":0.375,"15min":0.28999999999999998},"max":{"1min":8.6829999999999998,"5min":11.513,"15min":11.81},"last":0.82499999999999996},{"interface":"front","average":{"1min":1.9510000000000001,"5min":1.3700000000000001,"15min":1.4470000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.36599999999999999},"max":{"1min":8.3949999999999996,"5min":8.3949999999999996,"15min":12.058},"last":0.877}]},{"osd":9,"last update":"Fri Feb 9 17:51:47 2024","interfaces":[{"interface":"back","average":{"1min":2.3260000000000001,"5min":1.641,"15min":1.5780000000000001},"min":{"1min":0.57899999999999996,"5min":0.375,"15min":0.32700000000000001},"max":{"1min":8.8119999999999994,"5min":11.542999999999999,"15min":11.941000000000001},"last":4.7679999999999998},{"interface":"front","average":{"1min":2.1259999999999999,"5min":1.706,"15min":1.667},"min":{"1min":0.61699999999999999,"5min":0.40000000000000002,"15min":0.33400000000000002},"max":{"1min":15.852,"5min":15.852,"15min":15.852},"last":4.4720000000000004}]},{"osd":10,"last update":"Fri Feb 9 17:51:14 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.0660000000000001,"15min":1.351},"min":{"1min":0.51900000000000002,"5min":0.38700000000000001,"15min":0.35099999999999998},"max":{"1min":3.5779999999999998,"5min":5.5099999999999998,"15min":12.050000000000001},"last":2.9790000000000001},{"interface":"front","average":{"1min":0.94999999999999996,"5min":1.5549999999999999,"15min":1.5700000000000001},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.38900000000000001},"max":{"1min":3.5649999999999999,"5min":11.571999999999999,"15min":11.878},"last":3.181}]},{"osd":11,"last update":"Fri Feb 9 17:51:40 2024","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.121,"15min":1.589},"min":{"1min":0.46200000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":3.5950000000000002,"5min":5.5620000000000003,"15min":21.945},"last":3.7349999999999999},{"interface":"front","average":{"1min":1.774,"5min":1.3859999999999999,"15min":1.6839999999999999},"min":{"1min":0.48199999999999998,"5min":0.438,"15min":0.40699999999999997},"max":{"1min":5.8440000000000003,"5min":7.8019999999999996,"15min":21.731999999999999},"last":0.95999999999999996}]}]},{"osd":6,"up_from":49,"seq":210453397888,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2114632,"kb_used_data":1492032,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":91630520,"statfs":{"total":95995035648,"available":93829652480,"internally_reserved":0,"allocated":1527840768,"data_stored":1527348581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":637532614},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:51:31 2024","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":1.3879999999999999,"15min":1.5669999999999999},"min":{"1min":0.26100000000000001,"5min":0.24099999999999999,"15min":0.22800000000000001},"max":{"1min":5.3529999999999998,"5min":10.481999999999999,"15min":10.481999999999999},"last":0.372},{"interface":"front","average":{"1min":1.7030000000000001,"5min":1.663,"15min":1.665},"min":{"1min":0.27400000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":7.6100000000000003,"5min":7.6100000000000003,"15min":9.2520000000000007},"last":0.64100000000000001}]},{"osd":1,"last update":"Fri Feb 9 17:51:31 2024","interfaces":[{"interface":"back","average":{"1min":1.645,"5min":3.105,"15min":2.4670000000000001},"min":{"1min":0.28799999999999998,"5min":0.26800000000000002,"15min":0.252},"max":{"1min":9.593,"5min":79.945999999999998,"15min":79.945999999999998},"last":0.53400000000000003},{"interface":"front","average":{"1min":1.6879999999999999,"5min":3.1440000000000001,"15min":2.5419999999999998},"min":{"1min":0.40000000000000002,"5min":0.32600000000000001,"15min":0.29899999999999999},"max":{"1min":9.6329999999999991,"5min":80.003,"15min":80.003},"last":0.57199999999999995}]},{"osd":2,"last update":"Fri Feb 9 17:51:31 2024","interfaces":[{"interface":"back","average":{"1min":1.5660000000000001,"5min":2.9630000000000001,"15min":2.1320000000000001},"min":{"1min":0.29999999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":5.4269999999999996,"5min":84.912999999999997,"15min":84.912999999999997},"last":0.47199999999999998},{"interface":"front","average":{"1min":1.6180000000000001,"5min":3.016,"15min":2.1760000000000002},"min":{"1min":0.34599999999999997,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":5.4649999999999999,"5min":84.942999999999998,"15min":84.942999999999998},"last":0.60599999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:51:31 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":3.327,"15min":2.5659999999999998},"min":{"1min":0.32800000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":4.9020000000000001,"5min":80.174999999999997,"15min":80.174999999999997},"last":0.50700000000000001},{"interface":"front","average":{"1min":1.3320000000000001,"5min":3.786,"15min":2.7280000000000002},"min":{"1min":0.42799999999999999,"5min":0.33900000000000002,"15min":0.28000000000000003},"max":{"1min":4.9320000000000004,"5min":80.417000000000002,"15min":80.417000000000002},"last":0.49199999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:51:31 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":1.7509999999999999,"15min":1.6910000000000001},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.23599999999999999},"max":{"1min":6.0529999999999999,"5min":14.182,"15min":15.061999999999999},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.92200000000000004,"5min":1.8129999999999999,"15min":1.823},"min":{"1min":0.314,"5min":0.314,"15min":0.26400000000000001},"max":{"1min":6.0910000000000002,"5min":14.210000000000001,"15min":15.118},"last":0.44500000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:51:31 2024","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":1.825,"15min":1.8200000000000001},"min":{"1min":0.34899999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":7.4930000000000003,"5min":14.151999999999999,"15min":14.151999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":1.387,"5min":1.7549999999999999,"15min":1.756},"min":{"1min":0.32800000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":7.5899999999999999,"5min":15.509,"15min":18.315999999999999},"last":0.67400000000000004}]},{"osd":7,"last update":"Fri Feb 9 17:52:08 2024","interfaces":[{"interface":"back","average":{"1min":2.0459999999999998,"5min":1.9330000000000001,"15min":1.8420000000000001},"min":{"1min":0.318,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.343,"5min":23.265999999999998,"15min":23.265999999999998},"last":0.58999999999999997},{"interface":"front","average":{"1min":1.45,"5min":1.47,"15min":1.6759999999999999},"min":{"1min":0.34699999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":4.7439999999999998,"5min":10.599,"15min":12.967000000000001},"last":0.55300000000000005}]},{"osd":8,"last update":"Fri Feb 9 17:51:18 2024","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":3.3220000000000001,"5min":3.3220000000000001,"15min":3.3220000000000001},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":2.3610000000000002,"5min":2.3610000000000002,"15min":2.3610000000000002},"last":4.5250000000000004}]},{"osd":9,"last update":"Fri Feb 9 17:51:33 2024","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.1819999999999999,"15min":1.4379999999999999},"min":{"1min":0.434,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":3.8140000000000001,"5min":5.2569999999999997,"15min":10.779999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.208,"15min":1.516},"min":{"1min":0.52200000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":3.5910000000000002,"5min":5.2510000000000003,"15min":8.4130000000000003},"last":0.747}]},{"osd":10,"last update":"Fri Feb 9 17:52:08 2024","interfaces":[{"interface":"back","average":{"1min":3.3500000000000001,"5min":1.8049999999999999,"15min":1.73},"min":{"1min":0.46000000000000002,"5min":0.34399999999999997,"15min":0.34300000000000003},"max":{"1min":15.701000000000001,"5min":15.701000000000001,"15min":25.370000000000001},"last":1.103},{"interface":"front","average":{"1min":2.96,"5min":2.198,"15min":1.7809999999999999},"min":{"1min":0.46899999999999997,"5min":0.40200000000000002,"15min":0.35099999999999998},"max":{"1min":6.0190000000000001,"5min":14.119999999999999,"15min":26.027999999999999},"last":1.1200000000000001}]},{"osd":11,"last update":"Fri Feb 9 17:51:31 2024","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":1.47,"15min":1.6319999999999999},"min":{"1min":0.40200000000000002,"5min":0.372,"15min":0.372},"max":{"1min":4.0140000000000002,"5min":10.176,"15min":13.109},"last":1.8},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.4950000000000001,"15min":1.54},"min":{"1min":0.51200000000000001,"5min":0.39700000000000002,"15min":0.373},"max":{"1min":4.867,"5min":10.202,"15min":13.164999999999999},"last":4.3869999999999996}]}]},{"osd":1,"up_from":13,"seq":55834575257,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2450380,"kb_used_data":1665344,"kb_used_omap":1,"kb_used_meta":785022,"kb_avail":91294772,"statfs":{"total":95995035648,"available":93485846528,"internally_reserved":0,"allocated":1705312256,"data_stored":1704820069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":803862981},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:51:46 2024","interfaces":[{"interface":"back","average":{"1min":1.371,"5min":1.855,"15min":1.796},"min":{"1min":0.27000000000000002,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":7.5919999999999996,"5min":9.468,"15min":19.079000000000001},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":2.3079999999999998,"15min":1.819},"min":{"1min":0.22,"5min":0.214,"15min":0.19400000000000001},"max":{"1min":2.3210000000000002,"5min":33.389000000000003,"15min":33.389000000000003},"last":0.91900000000000004}]},{"osd":2,"last update":"Fri Feb 9 17:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":2.2400000000000002,"15min":1.857},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":6.726,"5min":14.83,"15min":14.83},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.627,"5min":1.8180000000000001,"15min":1.8260000000000001},"min":{"1min":0.29299999999999998,"5min":0.222,"15min":0.222},"max":{"1min":1.847,"5min":9.3870000000000005,"15min":11.346},"last":1.0049999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:50:36 2024","interfaces":[{"interface":"back","average":{"1min":2.786,"5min":2.831,"15min":1.9570000000000001},"min":{"1min":0.29999999999999999,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":17.878,"5min":17.959,"15min":17.959},"last":0.40899999999999997},{"interface":"front","average":{"1min":2.1760000000000002,"5min":3.1869999999999998,"15min":1.7869999999999999},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.26300000000000001},"max":{"1min":11.526,"5min":22.866,"15min":22.866},"last":0.32000000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:51:46 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":1.569,"15min":1.8069999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.25800000000000001},"max":{"1min":4.077,"5min":7.8490000000000002,"15min":45.137999999999998},"last":0.80500000000000005},{"interface":"front","average":{"1min":1.054,"5min":1.51,"15min":1.905},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":3.625,"5min":9.2050000000000001,"15min":44.950000000000003},"last":0.68600000000000005}]},{"osd":5,"last update":"Fri Feb 9 17:51:25 2024","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":1.653,"15min":1.8720000000000001},"min":{"1min":0.37,"5min":0.313,"15min":0.27900000000000003},"max":{"1min":3.5819999999999999,"5min":9.2409999999999997,"15min":45.171999999999997},"last":0.66700000000000004},{"interface":"front","average":{"1min":1.1759999999999999,"5min":2.444,"15min":2.137},"min":{"1min":0.377,"5min":0.31900000000000001,"15min":0.30499999999999999},"max":{"1min":3.6789999999999998,"5min":33.247,"15min":45.127000000000002},"last":0.74199999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:51:38 2024","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":2.4430000000000001,"15min":1.806},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":3.593,"5min":33.338000000000001,"15min":33.338000000000001},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.89300000000000002,"5min":2.4729999999999999,"15min":1.8080000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":3.1760000000000002,"5min":33.286999999999999,"15min":33.286999999999999},"last":3.1520000000000001}]},{"osd":7,"last update":"Fri Feb 9 17:51:38 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":2.4990000000000001,"15min":1.873},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.34399999999999997},"max":{"1min":3.8479999999999999,"5min":33.314999999999998,"15min":45.740000000000002},"last":0.76100000000000001},{"interface":"front","average":{"1min":1.244,"5min":2.5299999999999998,"15min":1.958},"min":{"1min":0.53800000000000003,"5min":0.40200000000000002,"15min":0.309},"max":{"1min":3.8839999999999999,"5min":33.363999999999997,"15min":45.792000000000002},"last":0.72899999999999998}]},{"osd":8,"last update":"Fri Feb 9 17:51:46 2024","interfaces":[{"interface":"back","average":{"1min":1.548,"5min":1.681,"15min":1.2270000000000001},"min":{"1min":0.46400000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":4.9269999999999996,"5min":8.9440000000000008,"15min":9.2520000000000007},"last":8.4390000000000001},{"interface":"front","average":{"1min":2.149,"5min":1.849,"15min":1.2769999999999999},"min":{"1min":0.53600000000000003,"5min":0.39000000000000001,"15min":0.35599999999999998},"max":{"1min":7.1459999999999999,"5min":12.69,"15min":12.69},"last":10.568}]},{"osd":9,"last update":"Fri Feb 9 17:51:46 2024","interfaces":[{"interface":"back","average":{"1min":1.5980000000000001,"5min":1.7390000000000001,"15min":1.5960000000000001},"min":{"1min":0.497,"5min":0.39500000000000002,"15min":0.33400000000000002},"max":{"1min":5.7009999999999996,"5min":9.0329999999999995,"15min":9.9359999999999999},"last":1.8700000000000001},{"interface":"front","average":{"1min":2.2109999999999999,"5min":1.913,"15min":1.8240000000000001},"min":{"1min":0.39400000000000002,"5min":0.37,"15min":0.32900000000000001},"max":{"1min":6.9980000000000002,"5min":9.2650000000000006,"15min":11.366},"last":0.71699999999999997}]},{"osd":10,"last update":"Fri Feb 9 17:51:25 2024","interfaces":[{"interface":"back","average":{"1min":1.9470000000000001,"5min":2.448,"15min":1.9850000000000001},"min":{"1min":0.55000000000000004,"5min":0.432,"15min":0.39200000000000002},"max":{"1min":7.0949999999999998,"5min":33.213999999999999,"15min":33.213999999999999},"last":12.471},{"interface":"front","average":{"1min":2.1880000000000002,"5min":1.9850000000000001,"15min":1.871},"min":{"1min":0.47399999999999998,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":7.0599999999999996,"5min":10.09,"15min":15.121},"last":10.897}]},{"osd":11,"last update":"Fri Feb 9 17:51:38 2024","interfaces":[{"interface":"back","average":{"1min":3.2690000000000001,"5min":2.3260000000000001,"15min":1.5660000000000001},"min":{"1min":0.498,"5min":0.46800000000000003,"15min":0.39100000000000001},"max":{"1min":19.648,"5min":19.648,"15min":19.648},"last":7.4779999999999998},{"interface":"front","average":{"1min":2.1440000000000001,"5min":2.5190000000000001,"15min":1.6990000000000001},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":19.077000000000002,"5min":33.161000000000001,"15min":33.161000000000001},"last":0.69999999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2783308,"kb_used_data":2044096,"kb_used_omap":1,"kb_used_meta":739198,"kb_avail":90961844,"statfs":{"total":95995035648,"available":93144928256,"internally_reserved":0,"allocated":2093154304,"data_stored":2092662117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":756939205},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 9 17:51:31 2024","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.7430000000000001,"15min":1.5289999999999999},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.20599999999999999},"max":{"1min":6.8239999999999998,"5min":10.938000000000001,"15min":13.842000000000001},"last":0.41799999999999998},{"interface":"front","average":{"1min":1.131,"5min":1.806,"15min":1.7569999999999999},"min":{"1min":0.32100000000000001,"5min":0.308,"15min":0.24399999999999999},"max":{"1min":6.8419999999999996,"5min":10.974,"15min":14.122},"last":0.79700000000000004}]},{"osd":2,"last update":"Fri Feb 9 17:51:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":1.6220000000000001,"15min":1.9079999999999999},"min":{"1min":0.29499999999999998,"5min":0.245,"15min":0.245},"max":{"1min":5.1079999999999997,"5min":8.859,"15min":8.859},"last":1.625},{"interface":"front","average":{"1min":1.53,"5min":1.5269999999999999,"15min":1.522},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":8.4019999999999992,"5min":8.4019999999999992,"15min":8.4019999999999992},"last":0.44400000000000001}]},{"osd":3,"last update":"Fri Feb 9 17:51:08 2024","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":2.177,"15min":2.0249999999999999},"min":{"1min":0.26100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":6.8040000000000003,"5min":36.052,"15min":36.052},"last":0.747},{"interface":"front","average":{"1min":1.976,"5min":2.7450000000000001,"15min":1.897},"min":{"1min":0.35099999999999998,"5min":0.27500000000000002,"15min":0.217},"max":{"1min":7.4699999999999998,"5min":38.383000000000003,"15min":38.383000000000003},"last":0.58499999999999996}]},{"osd":4,"last update":"Fri Feb 9 17:51:59 2024","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.504,"15min":1.3759999999999999},"min":{"1min":0.35499999999999998,"5min":0.29399999999999998,"15min":0.27000000000000002},"max":{"1min":5.452,"5min":8.9420000000000002,"15min":8.9420000000000002},"last":1.117},{"interface":"front","average":{"1min":1.516,"5min":2.214,"15min":1.764},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":5.46,"5min":16.916,"15min":16.916},"last":1.1020000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:51:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":2.4209999999999998,"15min":1.821},"min":{"1min":0.35199999999999998,"5min":0.311,"15min":0.29499999999999998},"max":{"1min":3.7759999999999998,"5min":48.924999999999997,"15min":48.924999999999997},"last":0.85299999999999998},{"interface":"front","average":{"1min":1.0289999999999999,"5min":2.4710000000000001,"15min":1.833},"min":{"1min":0.39400000000000002,"5min":0.35299999999999998,"15min":0.29599999999999999},"max":{"1min":3.871,"5min":48.969999999999999,"15min":48.969999999999999},"last":0.88700000000000001}]},{"osd":6,"last update":"Fri Feb 9 17:51:41 2024","interfaces":[{"interface":"back","average":{"1min":1.5649999999999999,"5min":1.698,"15min":1.681},"min":{"1min":0.377,"5min":0.307,"15min":0.28699999999999998},"max":{"1min":5.6100000000000003,"5min":7.9429999999999996,"15min":13.407},"last":0.83699999999999997},{"interface":"front","average":{"1min":1.6850000000000001,"5min":1.8500000000000001,"15min":1.7749999999999999},"min":{"1min":0.47199999999999998,"5min":0.375,"15min":0.33300000000000002},"max":{"1min":5.6630000000000003,"5min":7.9740000000000002,"15min":13.448},"last":0.86599999999999999}]},{"osd":7,"last update":"Fri Feb 9 17:51:06 2024","interfaces":[{"interface":"back","average":{"1min":1.5640000000000001,"5min":1.8169999999999999,"15min":1.8700000000000001},"min":{"1min":0.36799999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":5.6399999999999997,"5min":15.271000000000001,"15min":15.271000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.97899999999999998,"5min":2.2280000000000002,"15min":2.0099999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":2.9169999999999998,"5min":11.614000000000001,"15min":14.451000000000001},"last":1.1819999999999999}]},{"osd":8,"last update":"Fri Feb 9 17:51:06 2024","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.7090000000000001,"15min":1.669},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":3.0499999999999998,"5min":11.641999999999999,"15min":14.323},"last":0.50900000000000001},{"interface":"front","average":{"1min":1.294,"5min":1.716,"15min":1.716},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.35099999999999998},"max":{"1min":3.6139999999999999,"5min":11.669,"15min":14.481999999999999},"last":1.3759999999999999}]},{"osd":9,"last update":"Fri Feb 9 17:51:53 2024","interfaces":[{"interface":"back","average":{"1min":3.6179999999999999,"5min":2.3820000000000001,"15min":2.0950000000000002},"min":{"1min":0.45000000000000001,"5min":0.41999999999999998,"15min":0.32500000000000001},"max":{"1min":24.757000000000001,"5min":24.757000000000001,"15min":24.757000000000001},"last":0.64600000000000002},{"interface":"front","average":{"1min":3.5680000000000001,"5min":2.4020000000000001,"15min":2.0979999999999999},"min":{"1min":0.45900000000000002,"5min":0.44400000000000001,"15min":0.34799999999999998},"max":{"1min":24.681999999999999,"5min":24.681999999999999,"15min":24.681999999999999},"last":1.0329999999999999}]},{"osd":10,"last update":"Fri Feb 9 17:51:08 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.6890000000000001,"15min":1.9410000000000001},"min":{"1min":0.54900000000000004,"5min":0.443,"15min":0.35499999999999998},"max":{"1min":3.9169999999999998,"5min":11.692,"15min":31.766999999999999},"last":2.2370000000000001},{"interface":"front","average":{"1min":1.597,"5min":1.5860000000000001,"15min":1.496},"min":{"1min":0.54000000000000004,"5min":0.41899999999999998,"15min":0.34200000000000003},"max":{"1min":6.2990000000000004,"5min":8.2210000000000001,"15min":28.048999999999999},"last":3.6789999999999998}]},{"osd":11,"last update":"Fri Feb 9 17:51:41 2024","interfaces":[{"interface":"back","average":{"1min":2.931,"5min":2.0539999999999998,"15min":1.585},"min":{"1min":0.45300000000000001,"5min":0.437,"15min":0.40600000000000003},"max":{"1min":25.994,"5min":25.994,"15min":25.994},"last":1},{"interface":"front","average":{"1min":3.1880000000000002,"5min":2.2330000000000001,"15min":1.7130000000000001},"min":{"1min":0.51000000000000001,"5min":0.496,"15min":0.45200000000000001},"max":{"1min":25.515999999999998,"5min":25.515999999999998,"15min":25.515999999999998},"last":1.081}]}]},{"osd":2,"up_from":18,"seq":77309411731,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2723980,"kb_used_data":2255492,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":91021172,"statfs":{"total":95995035648,"available":93205680128,"internally_reserved":0,"allocated":2309623808,"data_stored":2309132141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":1.4510000000000001,"15min":1.006},"min":{"1min":0.29999999999999999,"5min":0.23699999999999999,"15min":0.221},"max":{"1min":4.9089999999999998,"5min":17.251000000000001,"15min":17.251000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.74399999999999999,"5min":1.611,"15min":1.113},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.22900000000000001},"max":{"1min":4.7720000000000002,"5min":23.097999999999999,"15min":23.097999999999999},"last":0.875}]},{"osd":1,"last update":"Fri Feb 9 17:52:06 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":2.5710000000000002,"15min":2.117},"min":{"1min":0.22800000000000001,"5min":0.214,"15min":0.214},"max":{"1min":1.1910000000000001,"5min":46.698999999999998,"15min":52.646000000000001},"last":1.014},{"interface":"front","average":{"1min":1.3999999999999999,"5min":2.4790000000000001,"15min":2.1019999999999999},"min":{"1min":0.29099999999999998,"5min":0.248,"15min":0.214},"max":{"1min":10.906000000000001,"5min":46.728999999999999,"15min":52.683999999999997},"last":1.1100000000000001}]},{"osd":3,"last update":"Fri Feb 9 17:51:31 2024","interfaces":[{"interface":"back","average":{"1min":1.25,"5min":2.4079999999999999,"15min":1.901},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":7.718,"5min":22.698,"15min":22.698},"last":2.758},{"interface":"front","average":{"1min":1.3069999999999999,"5min":2.8820000000000001,"15min":2.2410000000000001},"min":{"1min":0.29199999999999998,"5min":0.221,"15min":0.20699999999999999},"max":{"1min":3.6200000000000001,"5min":19.904,"15min":19.986999999999998},"last":2.77}]},{"osd":4,"last update":"Fri Feb 9 17:51:31 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":1.857,"15min":1.8580000000000001},"min":{"1min":0.32000000000000001,"5min":0.315,"15min":0.26300000000000001},"max":{"1min":2.3239999999999998,"5min":12.398,"15min":13.526},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.149,"5min":1.7869999999999999,"15min":1.6599999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":4.391,"5min":10.888999999999999,"15min":10.888999999999999},"last":3.0299999999999998}]},{"osd":5,"last update":"Fri Feb 9 17:51:22 2024","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":2.6080000000000001,"15min":2.133},"min":{"1min":0.29299999999999998,"5min":0.28499999999999998,"15min":0.28299999999999997},"max":{"1min":3.681,"5min":35.436999999999998,"15min":42.664000000000001},"last":2.7069999999999999},{"interface":"front","average":{"1min":1.407,"5min":2.6429999999999998,"15min":2.395},"min":{"1min":0.33200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":5.6669999999999998,"5min":35.484000000000002,"15min":43.311999999999998},"last":3.1339999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:51:31 2024","interfaces":[{"interface":"back","average":{"1min":1.736,"5min":2.4049999999999998,"15min":2.0489999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":8.2899999999999991,"5min":18.518999999999998,"15min":18.518999999999998},"last":3.073},{"interface":"front","average":{"1min":1.8300000000000001,"5min":2.302,"15min":2.1179999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":8.9450000000000003,"5min":18.481999999999999,"15min":18.481999999999999},"last":3.1850000000000001}]},{"osd":7,"last update":"Fri Feb 9 17:52:06 2024","interfaces":[{"interface":"back","average":{"1min":2.0259999999999998,"5min":2.2010000000000001,"15min":2.2669999999999999},"min":{"1min":0.45400000000000001,"5min":0.38500000000000001,"15min":0.32400000000000001},"max":{"1min":10.933,"5min":12.904999999999999,"15min":41.802999999999997},"last":1.1830000000000001},{"interface":"front","average":{"1min":1.6200000000000001,"5min":2.3620000000000001,"15min":2.2370000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.30499999999999999},"max":{"1min":4.7300000000000004,"5min":11.167,"15min":41.783999999999999},"last":3.0979999999999999}]},{"osd":8,"last update":"Fri Feb 9 17:51:31 2024","interfaces":[{"interface":"back","average":{"1min":1.411,"5min":1.8120000000000001,"15min":1.4810000000000001},"min":{"1min":0.42499999999999999,"5min":0.35499999999999998,"15min":0.316},"max":{"1min":7.6719999999999997,"5min":12.307,"15min":12.307},"last":1.218},{"interface":"front","average":{"1min":4.1050000000000004,"5min":2.323,"15min":1.7949999999999999},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.34399999999999997},"max":{"1min":54.759999999999998,"5min":54.759999999999998,"15min":54.759999999999998},"last":4.9089999999999998}]},{"osd":9,"last update":"Fri Feb 9 17:51:43 2024","interfaces":[{"interface":"back","average":{"1min":4.0629999999999997,"5min":2.5680000000000001,"15min":2.1040000000000001},"min":{"1min":0.36099999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":53.935000000000002,"5min":53.935000000000002,"15min":53.935000000000002},"last":2.823},{"interface":"front","average":{"1min":4.2069999999999999,"5min":2.3450000000000002,"15min":1.992},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":54.670999999999999,"5min":54.670999999999999,"15min":54.670999999999999},"last":3.1600000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:51:17 2024","interfaces":[{"interface":"back","average":{"1min":3.5249999999999999,"5min":2.1899999999999999,"15min":1.8160000000000001},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":54.039999999999999,"5min":54.039999999999999,"15min":54.039999999999999},"last":1.244},{"interface":"front","average":{"1min":3.581,"5min":2.226,"15min":1.857},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.379},"max":{"1min":54.118000000000002,"5min":54.118000000000002,"15min":54.118000000000002},"last":1.264}]},{"osd":11,"last update":"Fri Feb 9 17:51:31 2024","interfaces":[{"interface":"back","average":{"1min":5.3490000000000002,"5min":2.7549999999999999,"15min":2.54},"min":{"1min":0.47599999999999998,"5min":0.41999999999999998,"15min":0.39600000000000002},"max":{"1min":54.799999999999997,"5min":54.799999999999997,"15min":137.44300000000001},"last":3.214},{"interface":"front","average":{"1min":4.6769999999999996,"5min":2.762,"15min":2.5470000000000002},"min":{"1min":0.44800000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":54.103000000000002,"5min":54.103000000000002,"15min":137.64400000000001},"last":2.7410000000000001}]}]},{"osd":3,"up_from":27,"seq":115964117392,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3120260,"kb_used_data":2443516,"kb_used_omap":1,"kb_used_meta":676734,"kb_avail":90624892,"statfs":{"total":95995035648,"available":92799889408,"internally_reserved":0,"allocated":2502160384,"data_stored":2501672269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":692976071},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:51:24 2024","interfaces":[{"interface":"back","average":{"1min":1.976,"5min":2.4849999999999999,"15min":1.9690000000000001},"min":{"1min":0.26300000000000001,"5min":0.221,"15min":0.186},"max":{"1min":6.2050000000000001,"5min":14.06,"15min":38.984999999999999},"last":0.436},{"interface":"front","average":{"1min":1.0880000000000001,"5min":3.028,"15min":2.2719999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.22500000000000001},"max":{"1min":5.3289999999999997,"5min":13.113,"15min":41.497999999999998},"last":0.30399999999999999}]},{"osd":1,"last update":"Fri Feb 9 17:51:54 2024","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":3.5190000000000001,"15min":2.552},"min":{"1min":0.309,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":5.4260000000000002,"5min":53.152999999999999,"15min":53.152999999999999},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.995,"5min":2.8919999999999999,"15min":1.768},"min":{"1min":0.249,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":4.3899999999999997,"5min":53.118000000000002,"15min":53.118000000000002},"last":0.33900000000000002}]},{"osd":2,"last update":"Fri Feb 9 17:51:24 2024","interfaces":[{"interface":"back","average":{"1min":1.806,"5min":2.5760000000000001,"15min":1.8979999999999999},"min":{"1min":0.40600000000000003,"5min":0.248,"15min":0.248},"max":{"1min":6.2720000000000002,"5min":15.734999999999999,"15min":15.734999999999999},"last":0.58299999999999996},{"interface":"front","average":{"1min":1.3859999999999999,"5min":2.6339999999999999,"15min":1.889},"min":{"1min":0.25800000000000001,"5min":0.23499999999999999,"15min":0.20200000000000001},"max":{"1min":4.5549999999999997,"5min":19.879000000000001,"15min":19.879000000000001},"last":0.73799999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:51:24 2024","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":2.379,"15min":1.7909999999999999},"min":{"1min":0.34699999999999998,"5min":0.28699999999999998,"15min":0.26800000000000002},"max":{"1min":5.375,"5min":11.374000000000001,"15min":13.933},"last":2.2229999999999999},{"interface":"front","average":{"1min":1.151,"5min":2.1349999999999998,"15min":1.7549999999999999},"min":{"1min":0.35099999999999998,"5min":0.309,"15min":0.28000000000000003},"max":{"1min":4.452,"5min":19.713999999999999,"15min":19.713999999999999},"last":0.39000000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:52:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":2.4710000000000001,"15min":1.978},"min":{"1min":0.36599999999999999,"5min":0.33000000000000002,"15min":0.28000000000000003},"max":{"1min":6.6959999999999997,"5min":14.106,"15min":14.106},"last":0.64000000000000001},{"interface":"front","average":{"1min":1.212,"5min":2.573,"15min":1.966},"min":{"1min":0.52700000000000002,"5min":0.45900000000000002,"15min":0.316},"max":{"1min":6.7480000000000002,"5min":14.138999999999999,"15min":14.138999999999999},"last":0.68899999999999995}]},{"osd":6,"last update":"Fri Feb 9 17:51:54 2024","interfaces":[{"interface":"back","average":{"1min":1.6739999999999999,"5min":2.5390000000000001,"15min":2.0910000000000002},"min":{"1min":0.54300000000000004,"5min":0.36499999999999999,"15min":0.28899999999999998},"max":{"1min":6.6600000000000001,"5min":15.061999999999999,"15min":15.061999999999999},"last":0.621},{"interface":"front","average":{"1min":1.6879999999999999,"5min":2.3119999999999998,"15min":1.7589999999999999},"min":{"1min":0.50900000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":6.7889999999999997,"5min":9.032,"15min":11.813000000000001},"last":1.0409999999999999}]},{"osd":7,"last update":"Fri Feb 9 17:51:37 2024","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":2.2650000000000001,"15min":1.6419999999999999},"min":{"1min":0.42899999999999999,"5min":0.36899999999999999,"15min":0.34200000000000003},"max":{"1min":4.423,"5min":19.788,"15min":19.788},"last":0.81899999999999995},{"interface":"front","average":{"1min":2.3799999999999999,"5min":2.7080000000000002,"15min":1.651},"min":{"1min":0.54300000000000004,"5min":0.40100000000000002,"15min":0.34899999999999998},"max":{"1min":6.835,"5min":11.858000000000001,"15min":11.858000000000001},"last":0.84999999999999998}]},{"osd":8,"last update":"Fri Feb 9 17:45:16 2024","interfaces":[{"interface":"back","average":{"1min":1.714,"5min":1.4450000000000001,"15min":1.498},"min":{"1min":0.377,"5min":0.377,"15min":0.26800000000000002},"max":{"1min":8.6020000000000003,"5min":11.821,"15min":11.821},"last":0.67700000000000005},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.1719999999999999,"15min":1.351},"min":{"1min":0.46500000000000002,"5min":0.36199999999999999,"15min":0.34499999999999997},"max":{"1min":6.8390000000000004,"5min":6.8390000000000004,"15min":14.129},"last":0.78200000000000003}]},{"osd":9,"last update":"Fri Feb 9 17:51:54 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.278,"15min":1.833},"min":{"1min":0.46800000000000003,"5min":0.34599999999999997,"15min":0.32500000000000001},"max":{"1min":9.2159999999999993,"5min":11.417,"15min":11.851000000000001},"last":0.67600000000000005},{"interface":"front","average":{"1min":2.3439999999999999,"5min":2.3170000000000002,"15min":1.8380000000000001},"min":{"1min":0.52300000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":10.462999999999999,"5min":11.492000000000001,"15min":12.815},"last":0.70699999999999996}]},{"osd":10,"last update":"Fri Feb 9 17:51:37 2024","interfaces":[{"interface":"back","average":{"1min":2.5899999999999999,"5min":2.0209999999999999,"15min":1.3660000000000001},"min":{"1min":0.60899999999999999,"5min":0.40899999999999997,"15min":0.39100000000000001},"max":{"1min":5.5899999999999999,"5min":6.5940000000000003,"15min":9.7080000000000002},"last":0.79100000000000004},{"interface":"front","average":{"1min":2.1840000000000002,"5min":2.1829999999999998,"15min":1.3919999999999999},"min":{"1min":0.46800000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":8.2240000000000002,"5min":19.859000000000002,"15min":19.859000000000002},"last":0.66500000000000004}]},{"osd":11,"last update":"Fri Feb 9 17:51:24 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":2.1080000000000001,"15min":1.665},"min":{"1min":0.64400000000000002,"5min":0.442,"15min":0.40300000000000002},"max":{"1min":4.9909999999999997,"5min":19.824000000000002,"15min":19.824000000000002},"last":0.94799999999999995},{"interface":"front","average":{"1min":1.766,"5min":1.835,"15min":1.7549999999999999},"min":{"1min":0.61699999999999999,"5min":0.45900000000000002,"15min":0.442},"max":{"1min":5.024,"5min":6.1219999999999999,"15min":17.145},"last":0.97499999999999998}]}]},{"osd":4,"up_from":32,"seq":137438953867,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2284868,"kb_used_data":1573436,"kb_used_omap":1,"kb_used_meta":711422,"kb_avail":91460284,"statfs":{"total":95995035648,"available":93655330816,"internally_reserved":0,"allocated":1611198464,"data_stored":1610710349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":728496581},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:51:43 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":1.395,"15min":1.4670000000000001},"min":{"1min":0.252,"5min":0.23899999999999999,"15min":0.222},"max":{"1min":2.7999999999999998,"5min":9.4949999999999992,"15min":10.301},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.70299999999999996,"5min":1.46,"15min":1.5640000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.247},"max":{"1min":2.8300000000000001,"5min":9.5250000000000004,"15min":10.407999999999999},"last":0.39800000000000002}]},{"osd":1,"last update":"Fri Feb 9 17:51:43 2024","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":1.788,"15min":2},"min":{"1min":0.26800000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":3.54,"5min":8.6120000000000001,"15min":28.143000000000001},"last":0.503},{"interface":"front","average":{"1min":1.077,"5min":2.2040000000000002,"15min":2.2210000000000001},"min":{"1min":0.23699999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":3.9870000000000001,"5min":12.279999999999999,"15min":28.103000000000002},"last":0.54300000000000004}]},{"osd":2,"last update":"Fri Feb 9 17:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.663,"15min":2.6150000000000002},"min":{"1min":0.38700000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":4.2220000000000004,"5min":8.5139999999999993,"15min":17.494},"last":0.46000000000000002},{"interface":"front","average":{"1min":1.351,"5min":1.865,"15min":2.6110000000000002},"min":{"1min":0.38,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.6609999999999996,"5min":17.283999999999999,"15min":17.283999999999999},"last":0.44800000000000001}]},{"osd":3,"last update":"Fri Feb 9 17:51:43 2024","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.6870000000000001,"15min":1.7709999999999999},"min":{"1min":0.30099999999999999,"5min":0.28499999999999998,"15min":0.245},"max":{"1min":3.0670000000000002,"5min":7.7699999999999996,"15min":14.448},"last":2.1859999999999999},{"interface":"front","average":{"1min":1.506,"5min":2.125,"15min":1.819},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":3.609,"5min":9.9949999999999992,"15min":10.382},"last":0.38100000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:52:01 2024","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.8,"15min":1.8879999999999999},"min":{"1min":0.30399999999999999,"5min":0.255,"15min":0.22500000000000001},"max":{"1min":3.8940000000000001,"5min":14.779999999999999,"15min":18.887},"last":0.48899999999999999},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.8640000000000001,"15min":1.972},"min":{"1min":0.39400000000000002,"5min":0.31,"15min":0.29099999999999998},"max":{"1min":3.9359999999999999,"5min":14.855,"15min":18.917000000000002},"last":0.51800000000000002}]},{"osd":6,"last update":"Fri Feb 9 17:51:43 2024","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.373,"15min":1.5620000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":3.5840000000000001,"5min":11.257,"15min":18.234999999999999},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.94999999999999996,"5min":1.0129999999999999,"15min":1.6279999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.245},"max":{"1min":3.7509999999999999,"5min":5.7590000000000003,"15min":13.208},"last":0.33600000000000002}]},{"osd":7,"last update":"Fri Feb 9 17:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":1.7290000000000001,"15min":1.516},"min":{"1min":0.371,"5min":0.35499999999999998,"15min":0.29899999999999999},"max":{"1min":1.778,"5min":8.673,"15min":11.875},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.59499999999999997,"5min":1.6080000000000001,"15min":1.708},"min":{"1min":0.379,"5min":0.29999999999999999,"15min":0.28199999999999997},"max":{"1min":1.4950000000000001,"5min":9.5519999999999996,"15min":12.696},"last":0.57599999999999996}]},{"osd":8,"last update":"Fri Feb 9 17:52:01 2024","interfaces":[{"interface":"back","average":{"1min":2.4289999999999998,"5min":1.8180000000000001,"15min":1.7},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33200000000000002},"max":{"1min":8.548,"5min":8.548,"15min":17.335999999999999},"last":3.0030000000000001},{"interface":"front","average":{"1min":1.8680000000000001,"5min":1.0900000000000001,"15min":1.218},"min":{"1min":0.44800000000000001,"5min":0.378,"15min":0.34699999999999998},"max":{"1min":5.0049999999999999,"5min":6.7039999999999997,"15min":10.433},"last":3.9929999999999999}]},{"osd":9,"last update":"Fri Feb 9 17:52:10 2024","interfaces":[{"interface":"back","average":{"1min":2.7770000000000001,"5min":2.7770000000000001,"15min":2.7770000000000001},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":7.4900000000000002,"5min":7.4900000000000002,"15min":7.4900000000000002},"last":2.4830000000000001},{"interface":"front","average":{"1min":1.7290000000000001,"5min":1.7290000000000001,"15min":1.7290000000000001},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":7.234,"5min":7.234,"15min":7.234},"last":0.60699999999999998}]},{"osd":10,"last update":"Fri Feb 9 17:51:20 2024","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.91700000000000004,"15min":1.597},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.39400000000000002},"max":{"1min":3.9540000000000002,"5min":5.8300000000000001,"15min":17.367000000000001},"last":0.63300000000000001},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.655,"15min":1.625},"min":{"1min":0.53000000000000003,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":3.8700000000000001,"5min":12.888,"15min":12.888},"last":0.53000000000000003}]},{"osd":11,"last update":"Fri Feb 9 17:51:49 2024","interfaces":[{"interface":"back","average":{"1min":3.133,"5min":1.4970000000000001,"15min":1.5760000000000001},"min":{"1min":0.56499999999999995,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":7.8410000000000002,"5min":7.8410000000000002,"15min":10.521000000000001},"last":2.2410000000000001},{"interface":"front","average":{"1min":2.2999999999999998,"5min":1.7170000000000001,"15min":1.573},"min":{"1min":0.55700000000000005,"5min":0.45100000000000001,"15min":0.40999999999999998},"max":{"1min":7.859,"5min":7.859,"15min":13.154999999999999},"last":2.23}]}]},{"osd":5,"up_from":40,"seq":171798692228,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1430476,"kb_used_data":803716,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":92314676,"statfs":{"total":95995035648,"available":94530228224,"internally_reserved":0,"allocated":823005184,"data_stored":822513517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:51:20 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":2.5859999999999999,"15min":1.736},"min":{"1min":0.28000000000000003,"5min":0.26600000000000001,"15min":0.23599999999999999},"max":{"1min":2.9540000000000002,"5min":116.77800000000001,"15min":116.77800000000001},"last":0.48999999999999999},{"interface":"front","average":{"1min":1.026,"5min":3.0739999999999998,"15min":2.0910000000000002},"min":{"1min":0.27000000000000002,"5min":0.252,"15min":0.249},"max":{"1min":4.4850000000000003,"5min":116.959,"15min":116.959},"last":0.70999999999999996}]},{"osd":1,"last update":"Fri Feb 9 17:51:08 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":2.1099999999999999,"15min":1.931},"min":{"1min":0.308,"5min":0.254,"15min":0.24199999999999999},"max":{"1min":2.5049999999999999,"5min":19.079999999999998,"15min":48.161999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.60099999999999998,"5min":1.819,"15min":1.9770000000000001},"min":{"1min":0.25800000000000001,"5min":0.248,"15min":0.23599999999999999},"max":{"1min":1.3260000000000001,"5min":19.065999999999999,"15min":48.218000000000004},"last":0.59099999999999997}]},{"osd":2,"last update":"Fri Feb 9 17:51:41 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":2.2749999999999999,"15min":1.8420000000000001},"min":{"1min":0.30199999999999999,"5min":0.27900000000000003,"15min":0.249},"max":{"1min":3.5600000000000001,"5min":9.673,"15min":13.929},"last":0.52000000000000002},{"interface":"front","average":{"1min":1.0129999999999999,"5min":2.3239999999999998,"15min":2.165},"min":{"1min":0.35099999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":4.468,"5min":14.978,"15min":15.945},"last":0.629}]},{"osd":3,"last update":"Fri Feb 9 17:51:08 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":2.5710000000000002,"15min":2.2120000000000002},"min":{"1min":0.39000000000000001,"5min":0.34699999999999998,"15min":0.29799999999999999},"max":{"1min":3.282,"5min":15.154999999999999,"15min":64.150999999999996},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.84899999999999998,"5min":2.7000000000000002,"15min":2.2250000000000001},"min":{"1min":0.379,"5min":0.313,"15min":0.247},"max":{"1min":4.5890000000000004,"5min":11.747,"15min":64.176000000000002},"last":0.69699999999999995}]},{"osd":4,"last update":"Fri Feb 9 17:51:08 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":1.456,"15min":1.756},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.249},"max":{"1min":2.2290000000000001,"5min":28.481999999999999,"15min":49.313000000000002},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.58999999999999997,"5min":1.8360000000000001,"15min":1.788},"min":{"1min":0.316,"5min":0.28499999999999998,"15min":0.218},"max":{"1min":1.6919999999999999,"5min":15.843,"15min":50.505000000000003},"last":0.66100000000000003}]},{"osd":6,"last update":"Fri Feb 9 17:51:39 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":1.4530000000000001,"15min":1.7070000000000001},"min":{"1min":0.33200000000000002,"5min":0.27900000000000003,"15min":0.26400000000000001},"max":{"1min":4.4290000000000003,"5min":15.026,"15min":16.657},"last":1.742},{"interface":"front","average":{"1min":1.536,"5min":1.7809999999999999,"15min":1.821},"min":{"1min":0.374,"5min":0.33300000000000002,"15min":0.26200000000000001},"max":{"1min":9.7200000000000006,"5min":20.192,"15min":20.192},"last":0.89500000000000002}]},{"osd":7,"last update":"Fri Feb 9 17:51:08 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.849,"15min":1.839},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.26700000000000002},"max":{"1min":18.134,"5min":18.134,"15min":50.545000000000002},"last":0.72999999999999998},{"interface":"front","average":{"1min":1.6359999999999999,"5min":2.2679999999999998,"15min":2.3799999999999999},"min":{"1min":0.36099999999999999,"5min":0.31,"15min":0.31},"max":{"1min":19.300000000000001,"5min":19.300000000000001,"15min":102.526},"last":0.46999999999999997}]},{"osd":8,"last update":"Fri Feb 9 17:43:23 2024","interfaces":[{"interface":"back","average":{"1min":1.6499999999999999,"5min":1.639,"15min":1.6020000000000001},"min":{"1min":0.376,"5min":0.35299999999999998,"15min":0.34899999999999998},"max":{"1min":4.8780000000000001,"5min":14.542,"15min":14.542},"last":5.5439999999999996},{"interface":"front","average":{"1min":3.6699999999999999,"5min":1.702,"15min":1.8300000000000001},"min":{"1min":0.37,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":16.359000000000002,"5min":16.359000000000002,"15min":16.359000000000002},"last":0.435}]},{"osd":9,"last update":"Fri Feb 9 17:51:39 2024","interfaces":[{"interface":"back","average":{"1min":2.1280000000000001,"5min":1.512,"15min":1.5900000000000001},"min":{"1min":0.432,"5min":0.36599999999999999,"15min":0.35299999999999998},"max":{"1min":9.6579999999999995,"5min":15.124000000000001,"15min":16.411000000000001},"last":1.1240000000000001},{"interface":"front","average":{"1min":1.657,"5min":1.8049999999999999,"15min":1.7230000000000001},"min":{"1min":0.42799999999999999,"5min":0.379,"15min":0.35299999999999998},"max":{"1min":9.7050000000000001,"5min":9.7050000000000001,"15min":16.417999999999999},"last":0.55200000000000005}]},{"osd":10,"last update":"Fri Feb 9 17:51:08 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":2.246,"15min":1.7869999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.317},"max":{"1min":1.613,"5min":80.042000000000002,"15min":80.042000000000002},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.88800000000000001,"5min":2.3319999999999999,"15min":1.804},"min":{"1min":0.45400000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":2.746,"5min":79.807000000000002,"15min":79.807000000000002},"last":1.476}]},{"osd":11,"last update":"Fri Feb 9 17:51:20 2024","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.921,"15min":1.964},"min":{"1min":0.46700000000000003,"5min":0.39400000000000002,"15min":0.38400000000000001},"max":{"1min":9.7680000000000007,"5min":9.7680000000000007,"15min":16.390000000000001},"last":1.8540000000000001},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.6619999999999999,"15min":1.71},"min":{"1min":0.44,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":4.4649999999999999,"5min":7.1470000000000002,"15min":9.8239999999999998},"last":10.324999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2091388928,"data_stored":2091384856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1701318656,"data_stored":1701314584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2306080768,"data_stored":2306080768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2493448192,"data_stored":2493448192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1605042176,"data_stored":1605042176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":816250880,"data_stored":816250880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1525288960,"data_stored":1525284888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":800587776,"data_stored":800587776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1198915584,"data_stored":1198915584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2241921024,"data_stored":2241921024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1648427008,"data_stored":1648427008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2195390464,"data_stored":2195390464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-09T17:52:14.319 INFO:tasks.thrashosds.thrasher:Setting 1.7 to [1, 8, 6] 2024-02-09T17:52:14.319 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '1.7', '1', '8', '6'] 2024-02-09T17:52:14.319 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd pg-upmap 1.7 1 8 6 2024-02-09T17:52:14.334 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:52:14.334 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-02-09T17:52:14.334 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-02-09T17:52:14.334 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-02-09T17:52:14.334 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.334 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:170482 5.a 5:5724cd16:::benchmark_data_smithi179_115199_object41576:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e521)", 2024-02-09T17:52:14.334 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:43:24.167032+0000", 2024-02-09T17:52:14.334 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 530.16029867899999, 2024-02-09T17:52:14.334 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7617426749999998, 2024-02-09T17:52:14.334 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:52:14.334 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:52:14.335 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:52:14.335 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:52:14.335 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:52:14.335 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 170482 2024-02-09T17:52:14.335 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.335 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:52:14.335 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.335 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:52:14.335 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.167032+0000", 2024-02-09T17:52:14.335 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:52:14.335 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.335 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.335 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:52:14.335 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.167032+0000", 2024-02-09T17:52:14.336 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:52:14.336 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.336 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.336 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:52:14.336 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.167033+0000", 2024-02-09T17:52:14.336 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3400000000000001e-06 2024-02-09T17:52:14.336 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.336 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.336 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:52:14.336 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.167037+0000", 2024-02-09T17:52:14.336 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7110000000000001e-06 2024-02-09T17:52:14.336 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.336 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.336 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:52:14.337 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.167047+0000", 2024-02-09T17:52:14.337 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0051000000000001e-05 2024-02-09T17:52:14.337 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.337 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.337 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:52:14.337 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.167058+0000", 2024-02-09T17:52:14.337 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0479000000000001e-05 2024-02-09T17:52:14.337 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.337 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.337 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:52:14.337 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.167113+0000", 2024-02-09T17:52:14.337 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5609000000000002e-05 2024-02-09T17:52:14.337 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.337 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.338 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op must wait for map", 2024-02-09T17:52:14.338 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.167120+0000", 2024-02-09T17:52:14.338 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.849e-06 2024-02-09T17:52:14.338 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.338 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.338 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:52:14.338 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.172513+0000", 2024-02-09T17:52:14.338 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0053932110000000002 2024-02-09T17:52:14.338 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.338 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.338 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:52:14.338 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.172524+0000", 2024-02-09T17:52:14.338 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1249e-05 2024-02-09T17:52:14.338 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.338 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.339 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:52:14.339 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:25.163553+0000", 2024-02-09T17:52:14.339 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.99102885900000004 2024-02-09T17:52:14.339 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.339 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.339 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:52:14.339 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:25.163567+0000", 2024-02-09T17:52:14.339 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4019000000000001e-05 2024-02-09T17:52:14.339 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.339 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.339 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:52:14.339 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.907286+0000", 2024-02-09T17:52:14.339 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.74371831 2024-02-09T17:52:14.340 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.340 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.340 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:52:14.340 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.907433+0000", 2024-02-09T17:52:14.340 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000146923 2024-02-09T17:52:14.340 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.340 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.340 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,4,10", 2024-02-09T17:52:14.340 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.907553+0000", 2024-02-09T17:52:14.340 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000120517 2024-02-09T17:52:14.340 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.340 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.340 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:14.340 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.912625+0000", 2024-02-09T17:52:14.340 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.005071819 2024-02-09T17:52:14.341 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.341 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.341 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:52:14.341 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.912827+0000", 2024-02-09T17:52:14.341 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00020239499999999999 2024-02-09T17:52:14.341 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.341 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.341 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:14.341 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.917813+0000", 2024-02-09T17:52:14.341 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.004986117 2024-02-09T17:52:14.341 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.341 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.341 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:14.341 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.928603+0000", 2024-02-09T17:52:14.341 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010789234 2024-02-09T17:52:14.342 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.342 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.342 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:52:14.342 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.928665+0000", 2024-02-09T17:52:14.342 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2359000000000004e-05 2024-02-09T17:52:14.342 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.342 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.342 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:52:14.342 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.928775+0000", 2024-02-09T17:52:14.342 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010962400000000001 2024-02-09T17:52:14.342 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:52:14.342 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:52:14.342 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:52:14.342 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.343 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.343 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:170489 5.a 5:5f4ac418:::benchmark_data_smithi179_115199_object41583:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e521)", 2024-02-09T17:52:14.343 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:43:24.179483+0000", 2024-02-09T17:52:14.343 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 530.14784788199995, 2024-02-09T17:52:14.343 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7495300939999998, 2024-02-09T17:52:14.343 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:52:14.343 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:52:14.343 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:52:14.343 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:52:14.343 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:52:14.343 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 170489 2024-02-09T17:52:14.343 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.343 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:52:14.343 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.344 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:52:14.344 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.179483+0000", 2024-02-09T17:52:14.344 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:52:14.344 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.344 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.344 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:52:14.344 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.179483+0000", 2024-02-09T17:52:14.344 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:52:14.344 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.344 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.344 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:52:14.344 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.179484+0000", 2024-02-09T17:52:14.344 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.46e-06 2024-02-09T17:52:14.345 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.345 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.345 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:52:14.345 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.179489+0000", 2024-02-09T17:52:14.345 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1259999999999997e-06 2024-02-09T17:52:14.345 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.345 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.345 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:52:14.345 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.179501+0000", 2024-02-09T17:52:14.345 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1796e-05 2024-02-09T17:52:14.345 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.345 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.345 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:52:14.346 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.179509+0000", 2024-02-09T17:52:14.346 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.9270000000000005e-06 2024-02-09T17:52:14.346 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.346 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.346 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:52:14.346 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.179568+0000", 2024-02-09T17:52:14.346 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8860999999999998e-05 2024-02-09T17:52:14.346 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.346 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.346 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:52:14.346 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.179577+0000", 2024-02-09T17:52:14.346 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.3470000000000005e-06 2024-02-09T17:52:14.346 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.347 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.347 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:52:14.347 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:25.163622+0000", 2024-02-09T17:52:14.347 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.98404519400000001 2024-02-09T17:52:14.347 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.347 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.347 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:52:14.347 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:25.163631+0000", 2024-02-09T17:52:14.347 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7399999999999993e-06 2024-02-09T17:52:14.347 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.347 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.348 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:52:14.348 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.912873+0000", 2024-02-09T17:52:14.348 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.749241482 2024-02-09T17:52:14.348 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.348 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.348 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:52:14.348 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.913065+0000", 2024-02-09T17:52:14.348 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00019188300000000001 2024-02-09T17:52:14.348 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.348 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.348 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,4,10", 2024-02-09T17:52:14.349 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.913234+0000", 2024-02-09T17:52:14.349 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016940999999999999 2024-02-09T17:52:14.349 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.349 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.349 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:14.349 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.918189+0000", 2024-02-09T17:52:14.349 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0049548320000000002 2024-02-09T17:52:14.349 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.349 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.349 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:14.349 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.928556+0000", 2024-02-09T17:52:14.349 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010366820000000001 2024-02-09T17:52:14.349 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.350 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.350 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:52:14.350 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.928567+0000", 2024-02-09T17:52:14.350 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1113e-05 2024-02-09T17:52:14.350 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.350 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.350 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:14.350 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.928861+0000", 2024-02-09T17:52:14.350 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00029464699999999999 2024-02-09T17:52:14.350 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.350 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.350 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:52:14.351 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.928914+0000", 2024-02-09T17:52:14.351 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2757999999999999e-05 2024-02-09T17:52:14.351 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.351 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.351 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:52:14.351 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.929013+0000", 2024-02-09T17:52:14.351 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.8697999999999996e-05 2024-02-09T17:52:14.351 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:52:14.351 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:52:14.351 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:52:14.351 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.351 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.351 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:170494 5.a 5:531d98cd:::benchmark_data_smithi179_115199_object41588:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e521)", 2024-02-09T17:52:14.352 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:43:24.189636+0000", 2024-02-09T17:52:14.352 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 530.13769488100002, 2024-02-09T17:52:14.352 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7396196609999999, 2024-02-09T17:52:14.352 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:52:14.352 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:52:14.352 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:52:14.352 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:52:14.352 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:52:14.352 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 170494 2024-02-09T17:52:14.352 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.352 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:52:14.352 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.353 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:52:14.353 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.189636+0000", 2024-02-09T17:52:14.353 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:52:14.353 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.353 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.353 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:52:14.353 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.189636+0000", 2024-02-09T17:52:14.353 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:52:14.353 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.353 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.353 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:52:14.353 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.189638+0000", 2024-02-09T17:52:14.354 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9230000000000001e-06 2024-02-09T17:52:14.354 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.354 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.354 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:52:14.354 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.189645+0000", 2024-02-09T17:52:14.354 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2980000000000003e-06 2024-02-09T17:52:14.354 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.354 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.354 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:52:14.354 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.189664+0000", 2024-02-09T17:52:14.354 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8788999999999999e-05 2024-02-09T17:52:14.354 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.354 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.354 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:52:14.355 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.189675+0000", 2024-02-09T17:52:14.355 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0852999999999999e-05 2024-02-09T17:52:14.355 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.355 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.355 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:52:14.355 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.189749+0000", 2024-02-09T17:52:14.355 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4883999999999997e-05 2024-02-09T17:52:14.355 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.355 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.355 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:52:14.355 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.189757+0000", 2024-02-09T17:52:14.355 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8550000000000002e-06 2024-02-09T17:52:14.355 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.356 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.356 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:52:14.356 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:25.163659+0000", 2024-02-09T17:52:14.356 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.973901825 2024-02-09T17:52:14.356 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.356 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.356 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:52:14.356 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:25.163667+0000", 2024-02-09T17:52:14.356 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2009999999999999e-06 2024-02-09T17:52:14.356 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.356 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.356 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:52:14.357 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.918248+0000", 2024-02-09T17:52:14.357 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.754580443 2024-02-09T17:52:14.357 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.357 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.357 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:52:14.357 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.918384+0000", 2024-02-09T17:52:14.357 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013572200000000001 2024-02-09T17:52:14.357 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.357 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.357 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,4,10", 2024-02-09T17:52:14.357 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.918533+0000", 2024-02-09T17:52:14.357 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014900800000000001 2024-02-09T17:52:14.357 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.357 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.358 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:14.358 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.928820+0000", 2024-02-09T17:52:14.358 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010287057 2024-02-09T17:52:14.358 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.358 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.358 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:52:14.358 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.928829+0000", 2024-02-09T17:52:14.358 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2799999999999992e-06 2024-02-09T17:52:14.358 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.358 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.358 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:14.358 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.929050+0000", 2024-02-09T17:52:14.358 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00022129300000000001 2024-02-09T17:52:14.359 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.359 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.359 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:14.359 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.929128+0000", 2024-02-09T17:52:14.359 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7399999999999998e-05 2024-02-09T17:52:14.359 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.359 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.359 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:52:14.359 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.929177+0000", 2024-02-09T17:52:14.359 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9469999999999999e-05 2024-02-09T17:52:14.359 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.359 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.359 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:52:14.359 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.929255+0000", 2024-02-09T17:52:14.360 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8360000000000002e-05 2024-02-09T17:52:14.360 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:52:14.360 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:52:14.360 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:52:14.360 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.360 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.360 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:170511 5.a 5:5e38a700:::benchmark_data_smithi179_115199_object41605:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e521)", 2024-02-09T17:52:14.360 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:43:24.214457+0000", 2024-02-09T17:52:14.360 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 530.11287394600004, 2024-02-09T17:52:14.360 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7247621959999999, 2024-02-09T17:52:14.360 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:52:14.360 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:52:14.360 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:52:14.360 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:52:14.361 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:52:14.361 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 170511 2024-02-09T17:52:14.361 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.361 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:52:14.361 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.361 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:52:14.361 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.214457+0000", 2024-02-09T17:52:14.361 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:52:14.361 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.361 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.361 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:52:14.361 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.214457+0000", 2024-02-09T17:52:14.361 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:52:14.361 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.361 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.362 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:52:14.362 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.214458+0000", 2024-02-09T17:52:14.362 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3039999999999999e-06 2024-02-09T17:52:14.362 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.362 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.362 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:52:14.362 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.214463+0000", 2024-02-09T17:52:14.362 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.361e-06 2024-02-09T17:52:14.362 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.362 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.362 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:52:14.362 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.214475+0000", 2024-02-09T17:52:14.362 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2002e-05 2024-02-09T17:52:14.362 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.363 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.363 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:52:14.363 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.214482+0000", 2024-02-09T17:52:14.363 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7070000000000003e-06 2024-02-09T17:52:14.363 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.363 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.363 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:52:14.363 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.214545+0000", 2024-02-09T17:52:14.363 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2873999999999995e-05 2024-02-09T17:52:14.363 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.363 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.363 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:52:14.363 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.214556+0000", 2024-02-09T17:52:14.363 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0861e-05 2024-02-09T17:52:14.364 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.364 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.364 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:52:14.364 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:25.163694+0000", 2024-02-09T17:52:14.364 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.949138433 2024-02-09T17:52:14.364 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.364 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.364 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:52:14.364 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:25.163702+0000", 2024-02-09T17:52:14.364 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7749999999999993e-06 2024-02-09T17:52:14.364 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.364 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.364 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:52:14.364 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.923439+0000", 2024-02-09T17:52:14.365 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7597368950000001 2024-02-09T17:52:14.365 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.365 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.365 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:52:14.365 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.923590+0000", 2024-02-09T17:52:14.365 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000151454 2024-02-09T17:52:14.365 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.365 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.365 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,4,10", 2024-02-09T17:52:14.365 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.923718+0000", 2024-02-09T17:52:14.365 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012752299999999999 2024-02-09T17:52:14.365 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.365 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.365 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:52:14.366 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.929059+0000", 2024-02-09T17:52:14.366 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.005340951 2024-02-09T17:52:14.366 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.366 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.366 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:14.366 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.929092+0000", 2024-02-09T17:52:14.366 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2855999999999998e-05 2024-02-09T17:52:14.366 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.366 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.366 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:14.366 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.929305+0000", 2024-02-09T17:52:14.366 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00021333499999999999 2024-02-09T17:52:14.366 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.366 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.366 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:14.367 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.939072+0000", 2024-02-09T17:52:14.367 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0097671960000000006 2024-02-09T17:52:14.367 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.367 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.367 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:52:14.367 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.939118+0000", 2024-02-09T17:52:14.367 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5760000000000002e-05 2024-02-09T17:52:14.367 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.367 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.367 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:52:14.367 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.939219+0000", 2024-02-09T17:52:14.367 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010090900000000001 2024-02-09T17:52:14.367 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:52:14.367 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:52:14.367 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:52:14.368 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.368 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.368 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:170531 5.a 5:5cd92cb3:::benchmark_data_smithi179_115199_object41625:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e521)", 2024-02-09T17:52:14.368 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:43:24.268530+0000", 2024-02-09T17:52:14.368 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 530.05880050200005, 2024-02-09T17:52:14.368 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.670929031, 2024-02-09T17:52:14.368 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:52:14.368 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:52:14.368 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:52:14.368 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:52:14.368 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:52:14.368 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 170531 2024-02-09T17:52:14.368 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.368 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:52:14.369 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.369 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:52:14.369 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.268530+0000", 2024-02-09T17:52:14.369 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:52:14.369 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.369 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.369 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:52:14.369 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.268530+0000", 2024-02-09T17:52:14.369 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:52:14.369 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.369 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.369 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:52:14.369 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.268532+0000", 2024-02-09T17:52:14.369 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5850000000000001e-06 2024-02-09T17:52:14.369 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.369 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.370 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:52:14.370 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.268539+0000", 2024-02-09T17:52:14.370 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2479999999999999e-06 2024-02-09T17:52:14.370 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.370 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.370 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:52:14.370 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.268555+0000", 2024-02-09T17:52:14.370 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6155999999999999e-05 2024-02-09T17:52:14.370 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.370 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.370 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:52:14.370 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.268563+0000", 2024-02-09T17:52:14.370 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1790000000000008e-06 2024-02-09T17:52:14.370 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.370 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.371 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:52:14.371 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.268632+0000", 2024-02-09T17:52:14.371 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8942999999999994e-05 2024-02-09T17:52:14.371 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.371 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.371 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:52:14.371 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.268642+0000", 2024-02-09T17:52:14.371 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7380000000000006e-06 2024-02-09T17:52:14.371 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.371 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.371 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:52:14.371 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:25.163729+0000", 2024-02-09T17:52:14.371 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.89508700100000005 2024-02-09T17:52:14.371 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.371 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.372 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:52:14.372 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:25.163737+0000", 2024-02-09T17:52:14.372 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.0309999999999998e-06 2024-02-09T17:52:14.372 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.372 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.372 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:52:14.372 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.929339+0000", 2024-02-09T17:52:14.372 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.765602248 2024-02-09T17:52:14.372 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.372 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.372 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:52:14.372 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.929418+0000", 2024-02-09T17:52:14.372 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8811000000000001e-05 2024-02-09T17:52:14.372 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.373 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.373 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,4,10", 2024-02-09T17:52:14.373 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.929501+0000", 2024-02-09T17:52:14.373 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3084000000000006e-05 2024-02-09T17:52:14.373 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.373 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.373 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:14.373 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.939270+0000", 2024-02-09T17:52:14.373 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0097685900000000006 2024-02-09T17:52:14.373 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.373 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.373 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:52:14.373 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.939280+0000", 2024-02-09T17:52:14.373 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0013999999999999e-05 2024-02-09T17:52:14.373 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.374 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.374 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:14.374 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.939317+0000", 2024-02-09T17:52:14.374 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7728999999999997e-05 2024-02-09T17:52:14.374 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.374 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.374 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:14.374 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.939344+0000", 2024-02-09T17:52:14.374 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6508000000000001e-05 2024-02-09T17:52:14.374 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.374 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.374 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:52:14.374 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.939380+0000", 2024-02-09T17:52:14.374 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.5670999999999998e-05 2024-02-09T17:52:14.374 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.375 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.375 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:52:14.375 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.939459+0000", 2024-02-09T17:52:14.375 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.9494999999999997e-05 2024-02-09T17:52:14.375 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:52:14.375 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:52:14.375 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:52:14.375 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.375 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.375 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:170541 5.a 5:5a433982:::benchmark_data_smithi179_115199_object41635:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e521)", 2024-02-09T17:52:14.375 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:43:24.328851+0000", 2024-02-09T17:52:14.375 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 529.99848001199996, 2024-02-09T17:52:14.375 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6110777179999998, 2024-02-09T17:52:14.375 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:52:14.375 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:52:14.376 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:52:14.376 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:52:14.376 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:52:14.376 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 170541 2024-02-09T17:52:14.376 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.376 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:52:14.376 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.376 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:52:14.376 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.328851+0000", 2024-02-09T17:52:14.376 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:52:14.376 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.376 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.376 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:52:14.377 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.328851+0000", 2024-02-09T17:52:14.377 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:52:14.377 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.377 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.377 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:52:14.377 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.328852+0000", 2024-02-09T17:52:14.377 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.437e-06 2024-02-09T17:52:14.377 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.377 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.377 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:52:14.377 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.328857+0000", 2024-02-09T17:52:14.377 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3519999999999997e-06 2024-02-09T17:52:14.377 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.377 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.377 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:52:14.378 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.328873+0000", 2024-02-09T17:52:14.378 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5162e-05 2024-02-09T17:52:14.378 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.378 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.378 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:52:14.378 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.328883+0000", 2024-02-09T17:52:14.378 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0835999999999999e-05 2024-02-09T17:52:14.378 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.378 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.378 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:52:14.378 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.328969+0000", 2024-02-09T17:52:14.378 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5359999999999996e-05 2024-02-09T17:52:14.378 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.379 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.379 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:52:14.379 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:24.328981+0000", 2024-02-09T17:52:14.379 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2507000000000001e-05 2024-02-09T17:52:14.379 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.379 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.379 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:52:14.379 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:25.163829+0000", 2024-02-09T17:52:14.379 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.83484808200000005 2024-02-09T17:52:14.379 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.379 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.379 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:52:14.379 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:25.163840+0000", 2024-02-09T17:52:14.379 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0665999999999999e-05 2024-02-09T17:52:14.380 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.380 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.380 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:52:14.380 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.934605+0000", 2024-02-09T17:52:14.380 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7707647199999998 2024-02-09T17:52:14.380 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.380 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.380 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:52:14.380 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.934723+0000", 2024-02-09T17:52:14.380 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000117971 2024-02-09T17:52:14.380 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.380 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.380 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,4,10", 2024-02-09T17:52:14.380 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.934854+0000", 2024-02-09T17:52:14.381 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013093800000000001 2024-02-09T17:52:14.381 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.381 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.381 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:52:14.381 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.939468+0000", 2024-02-09T17:52:14.381 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0046148450000000002 2024-02-09T17:52:14.381 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.381 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.381 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:14.381 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.939498+0000", 2024-02-09T17:52:14.381 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0018999999999998e-05 2024-02-09T17:52:14.381 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.381 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.381 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:14.382 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.939525+0000", 2024-02-09T17:52:14.382 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6771000000000002e-05 2024-02-09T17:52:14.382 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.382 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.382 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:14.382 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.939836+0000", 2024-02-09T17:52:14.382 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00031041700000000001 2024-02-09T17:52:14.382 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.382 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.382 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:52:14.382 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.939870+0000", 2024-02-09T17:52:14.382 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4042999999999999e-05 2024-02-09T17:52:14.382 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.382 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.383 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:52:14.383 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:43:26.939928+0000", 2024-02-09T17:52:14.383 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8591999999999997e-05 2024-02-09T17:52:14.383 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:52:14.383 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:52:14.383 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:52:14.383 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.383 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.383 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:227314 5.a 5:5de15c45:::benchmark_data_smithi179_115199_object98408:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e552)", 2024-02-09T17:52:14.383 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:44:38.585097+0000", 2024-02-09T17:52:14.383 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 455.74223400099999, 2024-02-09T17:52:14.383 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.053613887, 2024-02-09T17:52:14.383 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:52:14.383 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:52:14.384 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:52:14.384 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:52:14.384 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:52:14.384 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 227314 2024-02-09T17:52:14.384 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.384 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:52:14.384 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.384 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:52:14.384 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.585097+0000", 2024-02-09T17:52:14.384 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:52:14.384 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.384 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.384 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:52:14.384 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.585097+0000", 2024-02-09T17:52:14.385 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:52:14.385 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.385 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.385 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:52:14.385 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.585098+0000", 2024-02-09T17:52:14.385 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6619999999999999e-06 2024-02-09T17:52:14.385 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.385 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.385 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:52:14.385 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.585105+0000", 2024-02-09T17:52:14.385 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1160000000000001e-06 2024-02-09T17:52:14.385 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.385 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.385 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:52:14.386 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.585126+0000", 2024-02-09T17:52:14.386 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0305e-05 2024-02-09T17:52:14.386 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.386 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.386 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:52:14.386 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.585144+0000", 2024-02-09T17:52:14.386 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8218000000000001e-05 2024-02-09T17:52:14.386 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.386 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.386 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:52:14.386 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.585212+0000", 2024-02-09T17:52:14.386 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7749999999999993e-05 2024-02-09T17:52:14.386 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.386 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.387 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op must wait for map", 2024-02-09T17:52:14.387 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.585218+0000", 2024-02-09T17:52:14.387 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6710000000000002e-06 2024-02-09T17:52:14.387 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.387 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.387 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:52:14.387 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.589152+0000", 2024-02-09T17:52:14.387 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0039341050000000002 2024-02-09T17:52:14.387 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.387 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.387 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:52:14.387 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.589166+0000", 2024-02-09T17:52:14.387 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3801999999999999e-05 2024-02-09T17:52:14.387 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.388 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.388 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:52:14.388 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.625759+0000", 2024-02-09T17:52:14.388 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.03659273 2024-02-09T17:52:14.388 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.388 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.388 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:52:14.388 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.625917+0000", 2024-02-09T17:52:14.388 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015817699999999999 2024-02-09T17:52:14.388 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.388 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.388 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,10", 2024-02-09T17:52:14.388 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.626064+0000", 2024-02-09T17:52:14.389 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014713700000000001 2024-02-09T17:52:14.389 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.389 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.389 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:14.389 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.630798+0000", 2024-02-09T17:52:14.389 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0047334630000000003 2024-02-09T17:52:14.389 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.389 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.389 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:14.389 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.638592+0000", 2024-02-09T17:52:14.389 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0077939050000000003 2024-02-09T17:52:14.389 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.389 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.390 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:52:14.390 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.638609+0000", 2024-02-09T17:52:14.390 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7655e-05 2024-02-09T17:52:14.390 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.390 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.390 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:52:14.390 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.638665+0000", 2024-02-09T17:52:14.390 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5943000000000003e-05 2024-02-09T17:52:14.391 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.391 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.391 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:52:14.391 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.638710+0000", 2024-02-09T17:52:14.391 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5247999999999998e-05 2024-02-09T17:52:14.391 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:52:14.391 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:52:14.391 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:52:14.391 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.391 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.391 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:227358 5.a 5:5dae4617:::benchmark_data_smithi179_115199_object98452:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e552)", 2024-02-09T17:52:14.391 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:44:38.643271+0000", 2024-02-09T17:52:14.391 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 455.68405976999998, 2024-02-09T17:52:14.391 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.99969172100000003, 2024-02-09T17:52:14.392 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:52:14.392 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:52:14.392 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:52:14.392 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:52:14.392 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:52:14.392 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 227358 2024-02-09T17:52:14.392 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.392 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:52:14.392 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.392 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:52:14.392 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.643271+0000", 2024-02-09T17:52:14.392 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:52:14.392 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.393 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.393 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:52:14.393 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.643271+0000", 2024-02-09T17:52:14.393 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:52:14.393 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.393 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.393 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:52:14.393 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.643273+0000", 2024-02-09T17:52:14.393 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0030000000000001e-06 2024-02-09T17:52:14.393 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.393 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.393 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:52:14.393 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.643280+0000", 2024-02-09T17:52:14.393 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4080000000000001e-06 2024-02-09T17:52:14.394 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.394 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.394 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:52:14.394 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.643299+0000", 2024-02-09T17:52:14.394 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9049999999999999e-05 2024-02-09T17:52:14.394 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.394 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.394 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:52:14.394 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.643309+0000", 2024-02-09T17:52:14.394 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6779999999999995e-06 2024-02-09T17:52:14.394 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.394 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.394 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:52:14.394 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.643380+0000", 2024-02-09T17:52:14.395 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1025999999999994e-05 2024-02-09T17:52:14.395 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.395 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.395 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:52:14.395 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.643392+0000", 2024-02-09T17:52:14.395 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1533e-05 2024-02-09T17:52:14.395 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.395 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.395 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:52:14.395 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.630849+0000", 2024-02-09T17:52:14.395 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.98745765399999996 2024-02-09T17:52:14.395 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.395 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.395 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:52:14.395 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.630964+0000", 2024-02-09T17:52:14.396 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011497699999999999 2024-02-09T17:52:14.396 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.396 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.396 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,10", 2024-02-09T17:52:14.396 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.631068+0000", 2024-02-09T17:52:14.396 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000103704 2024-02-09T17:52:14.396 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.396 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.396 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:14.396 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.638743+0000", 2024-02-09T17:52:14.396 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0076754600000000003 2024-02-09T17:52:14.396 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.396 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.396 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:52:14.397 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.638890+0000", 2024-02-09T17:52:14.397 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000146766 2024-02-09T17:52:14.397 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.397 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.397 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:14.397 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.642886+0000", 2024-02-09T17:52:14.397 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0039955829999999996 2024-02-09T17:52:14.397 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.397 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.397 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:52:14.397 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.642923+0000", 2024-02-09T17:52:14.397 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6918000000000001e-05 2024-02-09T17:52:14.397 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.397 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.397 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:52:14.398 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.642963+0000", 2024-02-09T17:52:14.398 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9960999999999999e-05 2024-02-09T17:52:14.398 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:52:14.398 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:52:14.398 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:52:14.398 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.398 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.398 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:227361 5.a 5:5e7824db:::benchmark_data_smithi179_115199_object98455:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e552)", 2024-02-09T17:52:14.398 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:44:38.643695+0000", 2024-02-09T17:52:14.398 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 455.68363543100003, 2024-02-09T17:52:14.398 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.99941951299999998, 2024-02-09T17:52:14.398 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:52:14.398 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:52:14.398 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:52:14.399 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:52:14.399 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:52:14.399 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 227361 2024-02-09T17:52:14.399 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.399 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:52:14.399 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.399 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:52:14.399 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.643695+0000", 2024-02-09T17:52:14.399 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:52:14.399 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.399 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.399 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:52:14.399 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.643695+0000", 2024-02-09T17:52:14.399 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:52:14.400 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.400 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.400 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:52:14.400 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.643696+0000", 2024-02-09T17:52:14.400 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8199999999999999e-07 2024-02-09T17:52:14.400 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.400 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.400 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:52:14.400 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.643700+0000", 2024-02-09T17:52:14.400 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0339999999999998e-06 2024-02-09T17:52:14.400 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.400 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.400 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:52:14.400 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.643711+0000", 2024-02-09T17:52:14.401 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0618000000000001e-05 2024-02-09T17:52:14.401 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.401 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.401 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:52:14.401 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.643717+0000", 2024-02-09T17:52:14.401 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.905e-06 2024-02-09T17:52:14.401 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.401 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.401 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:52:14.401 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.643766+0000", 2024-02-09T17:52:14.401 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8510999999999997e-05 2024-02-09T17:52:14.401 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.401 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.402 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:52:14.402 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.643774+0000", 2024-02-09T17:52:14.402 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6839999999999996e-06 2024-02-09T17:52:14.402 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.402 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.402 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:52:14.402 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.634822+0000", 2024-02-09T17:52:14.402 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.99104824899999999 2024-02-09T17:52:14.402 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.402 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.402 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:52:14.402 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.634897+0000", 2024-02-09T17:52:14.402 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5352999999999995e-05 2024-02-09T17:52:14.402 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.403 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.403 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,10", 2024-02-09T17:52:14.403 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.634974+0000", 2024-02-09T17:52:14.403 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6576999999999995e-05 2024-02-09T17:52:14.403 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.403 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.403 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:14.403 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.638798+0000", 2024-02-09T17:52:14.403 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0038244239999999999 2024-02-09T17:52:14.403 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.403 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.403 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:52:14.403 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.642853+0000", 2024-02-09T17:52:14.404 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0040544370000000001 2024-02-09T17:52:14.404 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.404 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.404 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:14.404 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.643020+0000", 2024-02-09T17:52:14.404 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016751799999999999 2024-02-09T17:52:14.404 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.404 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.404 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:52:14.404 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.643076+0000", 2024-02-09T17:52:14.404 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5467999999999998e-05 2024-02-09T17:52:14.404 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.404 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.404 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:52:14.405 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.643115+0000", 2024-02-09T17:52:14.405 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9053e-05 2024-02-09T17:52:14.405 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:52:14.405 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:52:14.405 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:52:14.405 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.405 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.405 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:227405 5.a 5:5076b4ce:::benchmark_data_smithi179_115199_object98499:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e552)", 2024-02-09T17:52:14.405 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:44:38.699279+0000", 2024-02-09T17:52:14.405 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 455.62805147699999, 2024-02-09T17:52:14.405 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.95481171300000001, 2024-02-09T17:52:14.405 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:52:14.405 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:52:14.406 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:52:14.406 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:52:14.406 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:52:14.406 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 227405 2024-02-09T17:52:14.406 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.406 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:52:14.406 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.406 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:52:14.406 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.699279+0000", 2024-02-09T17:52:14.406 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:52:14.406 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.406 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.406 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:52:14.406 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.699279+0000", 2024-02-09T17:52:14.407 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:52:14.407 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.407 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.407 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:52:14.407 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.699281+0000", 2024-02-09T17:52:14.407 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3680000000000001e-06 2024-02-09T17:52:14.407 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.407 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.407 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:52:14.407 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.699286+0000", 2024-02-09T17:52:14.407 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7209999999999999e-06 2024-02-09T17:52:14.407 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.407 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.408 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:52:14.408 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.699300+0000", 2024-02-09T17:52:14.408 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3953e-05 2024-02-09T17:52:14.408 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.408 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.408 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:52:14.408 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.699309+0000", 2024-02-09T17:52:14.408 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.9449999999999999e-06 2024-02-09T17:52:14.408 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.408 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.408 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:52:14.408 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.699374+0000", 2024-02-09T17:52:14.408 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4641999999999995e-05 2024-02-09T17:52:14.408 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.409 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.409 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:52:14.409 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.699389+0000", 2024-02-09T17:52:14.409 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5273000000000001e-05 2024-02-09T17:52:14.409 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.409 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.409 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:52:14.409 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.638918+0000", 2024-02-09T17:52:14.409 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.93952892600000004 2024-02-09T17:52:14.409 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.409 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.409 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:52:14.409 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.639025+0000", 2024-02-09T17:52:14.409 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000107234 2024-02-09T17:52:14.410 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.410 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.410 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,10", 2024-02-09T17:52:14.410 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.639129+0000", 2024-02-09T17:52:14.410 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000103595 2024-02-09T17:52:14.410 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.410 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.410 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:14.410 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.643203+0000", 2024-02-09T17:52:14.410 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0040737209999999998 2024-02-09T17:52:14.410 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.410 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.410 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:52:14.410 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.648477+0000", 2024-02-09T17:52:14.410 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0052741250000000002 2024-02-09T17:52:14.411 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.411 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.411 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:14.411 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.653979+0000", 2024-02-09T17:52:14.411 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0055023600000000004 2024-02-09T17:52:14.411 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.411 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.411 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:52:14.411 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.654045+0000", 2024-02-09T17:52:14.411 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5970999999999996e-05 2024-02-09T17:52:14.411 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.411 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.411 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:52:14.411 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.654091+0000", 2024-02-09T17:52:14.412 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5878999999999999e-05 2024-02-09T17:52:14.412 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:52:14.412 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:52:14.412 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:52:14.412 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.412 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.412 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:227427 5.a 5:5b32f294:::benchmark_data_smithi179_115199_object98521:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e552)", 2024-02-09T17:52:14.412 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:44:38.725715+0000", 2024-02-09T17:52:14.412 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 455.60161602099998, 2024-02-09T17:52:14.412 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.92858523299999995, 2024-02-09T17:52:14.412 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:52:14.412 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:52:14.412 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:52:14.413 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:52:14.413 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:52:14.413 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 227427 2024-02-09T17:52:14.413 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.413 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:52:14.413 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.413 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:52:14.413 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.725715+0000", 2024-02-09T17:52:14.413 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:52:14.413 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.413 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.413 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:52:14.413 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.725715+0000", 2024-02-09T17:52:14.413 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:52:14.414 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.414 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.414 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:52:14.414 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.725716+0000", 2024-02-09T17:52:14.414 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4270000000000001e-06 2024-02-09T17:52:14.414 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.414 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.414 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:52:14.414 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.725721+0000", 2024-02-09T17:52:14.414 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.882e-06 2024-02-09T17:52:14.414 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.414 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.414 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:52:14.414 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.725733+0000", 2024-02-09T17:52:14.414 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1629e-05 2024-02-09T17:52:14.415 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.415 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.415 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:52:14.415 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.725741+0000", 2024-02-09T17:52:14.415 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.9700000000000005e-06 2024-02-09T17:52:14.415 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.415 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.415 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:52:14.415 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.725800+0000", 2024-02-09T17:52:14.415 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8749000000000003e-05 2024-02-09T17:52:14.415 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.415 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.415 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:52:14.416 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.725808+0000", 2024-02-09T17:52:14.416 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4730000000000003e-06 2024-02-09T17:52:14.416 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.416 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.416 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:52:14.416 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.643243+0000", 2024-02-09T17:52:14.416 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.91743481299999996 2024-02-09T17:52:14.416 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.416 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.416 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:52:14.416 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.643360+0000", 2024-02-09T17:52:14.416 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011779500000000001 2024-02-09T17:52:14.417 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.417 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.417 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,10", 2024-02-09T17:52:14.417 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.643487+0000", 2024-02-09T17:52:14.417 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012694099999999999 2024-02-09T17:52:14.417 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.417 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.417 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:14.417 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.654166+0000", 2024-02-09T17:52:14.417 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010678254999999999 2024-02-09T17:52:14.417 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.417 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.417 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:14.418 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.654217+0000", 2024-02-09T17:52:14.418 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1285000000000001e-05 2024-02-09T17:52:14.418 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.418 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.418 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:52:14.418 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.654229+0000", 2024-02-09T17:52:14.418 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2381999999999999e-05 2024-02-09T17:52:14.418 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.418 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.418 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:52:14.418 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.654268+0000", 2024-02-09T17:52:14.419 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9255000000000003e-05 2024-02-09T17:52:14.419 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.419 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.419 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:52:14.419 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.654300+0000", 2024-02-09T17:52:14.419 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1377000000000001e-05 2024-02-09T17:52:14.419 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:52:14.419 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:52:14.419 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:52:14.419 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.419 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.419 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:227428 5.a 5:5c9193ea:::benchmark_data_smithi179_115199_object98522:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e552)", 2024-02-09T17:52:14.419 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:44:38.726172+0000", 2024-02-09T17:52:14.419 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 455.60115813300001, 2024-02-09T17:52:14.420 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.93274266800000005, 2024-02-09T17:52:14.420 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:52:14.420 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:52:14.420 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:52:14.420 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:52:14.420 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:52:14.420 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 227428 2024-02-09T17:52:14.420 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.420 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:52:14.420 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.420 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:52:14.420 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.726172+0000", 2024-02-09T17:52:14.420 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:52:14.420 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.421 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.421 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:52:14.421 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.726172+0000", 2024-02-09T17:52:14.421 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:52:14.421 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.421 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.421 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:52:14.421 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.726173+0000", 2024-02-09T17:52:14.421 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8500000000000001e-07 2024-02-09T17:52:14.421 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.421 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.421 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:52:14.421 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.726177+0000", 2024-02-09T17:52:14.421 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.433e-06 2024-02-09T17:52:14.422 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.422 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.422 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:52:14.422 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.726185+0000", 2024-02-09T17:52:14.422 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2900000000000002e-06 2024-02-09T17:52:14.422 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.422 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.422 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:52:14.422 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.726192+0000", 2024-02-09T17:52:14.422 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1420000000000004e-06 2024-02-09T17:52:14.422 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.422 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.422 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:52:14.422 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.726241+0000", 2024-02-09T17:52:14.422 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8707e-05 2024-02-09T17:52:14.423 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.423 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.423 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:52:14.423 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.726246+0000", 2024-02-09T17:52:14.423 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2549999999999997e-06 2024-02-09T17:52:14.423 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.423 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.423 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:52:14.423 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.648524+0000", 2024-02-09T17:52:14.423 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.92227784599999996 2024-02-09T17:52:14.423 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.423 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.423 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:52:14.423 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.648636+0000", 2024-02-09T17:52:14.424 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000111668 2024-02-09T17:52:14.424 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.424 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.424 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,10", 2024-02-09T17:52:14.424 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.648710+0000", 2024-02-09T17:52:14.424 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4856000000000003e-05 2024-02-09T17:52:14.424 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.424 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.424 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:52:14.424 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.654309+0000", 2024-02-09T17:52:14.424 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0055982599999999999 2024-02-09T17:52:14.424 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.424 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.424 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:14.424 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.654345+0000", 2024-02-09T17:52:14.425 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6531e-05 2024-02-09T17:52:14.425 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.425 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.425 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:14.425 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.658812+0000", 2024-02-09T17:52:14.425 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0044670029999999998 2024-02-09T17:52:14.425 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.425 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.425 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:52:14.425 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.658872+0000", 2024-02-09T17:52:14.425 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9851e-05 2024-02-09T17:52:14.425 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.425 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.425 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:52:14.426 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.658915+0000", 2024-02-09T17:52:14.426 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3140999999999999e-05 2024-02-09T17:52:14.426 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:52:14.426 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:52:14.426 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:52:14.426 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.426 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.426 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:227463 5.a 5:5452728d:::benchmark_data_smithi179_115199_object98557:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e552)", 2024-02-09T17:52:14.426 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:44:38.764325+0000", 2024-02-09T17:52:14.426 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 455.56300597500001, 2024-02-09T17:52:14.426 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.90314712500000005, 2024-02-09T17:52:14.426 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:52:14.426 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:52:14.426 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:52:14.426 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:52:14.427 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:52:14.427 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 227463 2024-02-09T17:52:14.427 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.427 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:52:14.427 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.427 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:52:14.427 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.764325+0000", 2024-02-09T17:52:14.427 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:52:14.427 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.427 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.427 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:52:14.427 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.764325+0000", 2024-02-09T17:52:14.427 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:52:14.427 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.427 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.428 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:52:14.428 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.764326+0000", 2024-02-09T17:52:14.428 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.75e-06 2024-02-09T17:52:14.428 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.428 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.428 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:52:14.428 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.764335+0000", 2024-02-09T17:52:14.428 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1179999999999992e-06 2024-02-09T17:52:14.428 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.428 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.428 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:52:14.428 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.764353+0000", 2024-02-09T17:52:14.428 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8559999999999998e-05 2024-02-09T17:52:14.429 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.429 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.429 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:52:14.429 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.764364+0000", 2024-02-09T17:52:14.429 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0702e-05 2024-02-09T17:52:14.429 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.429 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.429 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:52:14.429 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.764440+0000", 2024-02-09T17:52:14.429 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5867999999999999e-05 2024-02-09T17:52:14.429 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.429 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.429 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:52:14.429 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.764450+0000", 2024-02-09T17:52:14.430 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0176e-05 2024-02-09T17:52:14.430 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.430 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.430 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:52:14.430 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.654374+0000", 2024-02-09T17:52:14.430 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.88992394900000005 2024-02-09T17:52:14.430 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.430 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.430 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:52:14.430 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.654490+0000", 2024-02-09T17:52:14.430 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000115813 2024-02-09T17:52:14.430 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.430 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.430 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,10", 2024-02-09T17:52:14.431 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.654586+0000", 2024-02-09T17:52:14.431 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6508999999999994e-05 2024-02-09T17:52:14.431 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.431 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.431 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:14.431 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.659014+0000", 2024-02-09T17:52:14.431 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0044276460000000004 2024-02-09T17:52:14.431 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.431 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.431 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:14.431 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.667368+0000", 2024-02-09T17:52:14.431 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0083542600000000005 2024-02-09T17:52:14.431 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.431 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.432 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:52:14.432 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.667417+0000", 2024-02-09T17:52:14.432 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8535999999999997e-05 2024-02-09T17:52:14.432 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.432 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.432 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:52:14.432 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.667447+0000", 2024-02-09T17:52:14.432 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0127000000000001e-05 2024-02-09T17:52:14.432 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.432 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.432 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:52:14.432 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.667472+0000", 2024-02-09T17:52:14.432 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5111000000000001e-05 2024-02-09T17:52:14.432 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:52:14.433 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:52:14.433 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:52:14.433 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.433 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.433 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25528.0:227465 5.a 5:5c1d5b7b:::benchmark_data_smithi179_115199_object98559:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e552)", 2024-02-09T17:52:14.433 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:44:38.766743+0000", 2024-02-09T17:52:14.433 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 455.56058800099999, 2024-02-09T17:52:14.433 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.91118358799999999, 2024-02-09T17:52:14.433 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:52:14.433 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:52:14.433 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:52:14.433 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25528", 2024-02-09T17:52:14.433 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:52:14.434 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 227465 2024-02-09T17:52:14.434 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.434 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:52:14.434 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.434 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:52:14.434 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.766743+0000", 2024-02-09T17:52:14.434 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:52:14.434 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.434 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.434 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:52:14.434 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.766743+0000", 2024-02-09T17:52:14.434 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:52:14.434 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.434 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.435 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:52:14.435 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.766744+0000", 2024-02-09T17:52:14.435 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.094e-06 2024-02-09T17:52:14.435 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.435 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.435 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:52:14.435 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.766750+0000", 2024-02-09T17:52:14.435 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9540000000000001e-06 2024-02-09T17:52:14.435 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.435 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.435 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:52:14.435 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.766763+0000", 2024-02-09T17:52:14.435 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3291e-05 2024-02-09T17:52:14.435 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.435 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.436 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:52:14.436 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.766771+0000", 2024-02-09T17:52:14.436 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1499999999999999e-06 2024-02-09T17:52:14.436 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.436 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.436 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:52:14.436 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.766831+0000", 2024-02-09T17:52:14.436 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9991999999999998e-05 2024-02-09T17:52:14.436 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.436 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.436 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T17:52:14.436 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:38.766838+0000", 2024-02-09T17:52:14.436 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4409999999999998e-06 2024-02-09T17:52:14.436 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.437 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.437 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:52:14.437 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.659047+0000", 2024-02-09T17:52:14.437 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.89220939799999999 2024-02-09T17:52:14.437 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.437 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.437 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:52:14.437 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.659140+0000", 2024-02-09T17:52:14.437 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2998999999999995e-05 2024-02-09T17:52:14.437 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.437 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.437 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,10", 2024-02-09T17:52:14.437 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.659229+0000", 2024-02-09T17:52:14.437 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8743000000000001e-05 2024-02-09T17:52:14.438 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.438 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.438 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:14.438 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.667408+0000", 2024-02-09T17:52:14.438 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0081791120000000005 2024-02-09T17:52:14.438 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.438 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.438 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:52:14.438 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.667478+0000", 2024-02-09T17:52:14.438 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9764999999999994e-05 2024-02-09T17:52:14.438 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.438 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.438 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:14.438 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.677833+0000", 2024-02-09T17:52:14.438 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.01035582 2024-02-09T17:52:14.439 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.439 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.439 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:52:14.439 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.677886+0000", 2024-02-09T17:52:14.439 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2661000000000003e-05 2024-02-09T17:52:14.439 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.439 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.439 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:52:14.439 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:44:39.677926+0000", 2024-02-09T17:52:14.439 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0167999999999999e-05 2024-02-09T17:52:14.439 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:52:14.439 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:52:14.439 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:52:14.439 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.439 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.439 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25837.0:235942 6.1 6:8e014300:::benchmark_data_smithi179_126206_object105561:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e650)", 2024-02-09T17:52:14.440 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:49:27.780742+0000", 2024-02-09T17:52:14.440 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 166.546588566, 2024-02-09T17:52:14.440 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.154863529, 2024-02-09T17:52:14.440 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:52:14.440 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:52:14.440 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:52:14.440 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25837", 2024-02-09T17:52:14.440 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:52:14.440 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 235942 2024-02-09T17:52:14.440 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.440 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:52:14.440 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.440 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:52:14.440 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:27.780742+0000", 2024-02-09T17:52:14.441 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:52:14.441 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.441 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.441 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:52:14.441 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:27.780742+0000", 2024-02-09T17:52:14.441 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:52:14.441 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.441 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.441 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:52:14.441 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:27.780744+0000", 2024-02-09T17:52:14.441 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5239999999999999e-06 2024-02-09T17:52:14.441 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.441 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.441 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:52:14.442 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:27.780749+0000", 2024-02-09T17:52:14.442 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3589999999999999e-06 2024-02-09T17:52:14.442 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.442 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.442 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:52:14.442 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:27.780765+0000", 2024-02-09T17:52:14.442 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6409000000000002e-05 2024-02-09T17:52:14.442 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.442 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.442 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:52:14.442 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:27.780777+0000", 2024-02-09T17:52:14.442 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1877e-05 2024-02-09T17:52:14.442 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.442 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.443 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:52:14.443 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:27.920118+0000", 2024-02-09T17:52:14.443 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.13934075000000001 2024-02-09T17:52:14.443 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.443 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.443 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:52:14.443 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:27.920252+0000", 2024-02-09T17:52:14.443 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013443400000000001 2024-02-09T17:52:14.443 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.443 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.443 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 9,10", 2024-02-09T17:52:14.443 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:27.920351+0000", 2024-02-09T17:52:14.443 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.8820999999999995e-05 2024-02-09T17:52:14.443 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.444 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.444 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:14.444 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:27.935349+0000", 2024-02-09T17:52:14.444 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014997474 2024-02-09T17:52:14.444 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.444 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.444 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:14.444 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:27.935419+0000", 2024-02-09T17:52:14.444 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9973000000000002e-05 2024-02-09T17:52:14.444 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.444 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.444 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:52:14.445 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:27.935487+0000", 2024-02-09T17:52:14.445 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8534000000000005e-05 2024-02-09T17:52:14.445 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.445 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.445 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:52:14.445 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:27.935545+0000", 2024-02-09T17:52:14.445 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8298000000000003e-05 2024-02-09T17:52:14.445 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.445 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.445 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:52:14.445 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:27.935606+0000", 2024-02-09T17:52:14.445 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0075999999999999e-05 2024-02-09T17:52:14.445 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:52:14.446 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:52:14.446 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:52:14.446 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.446 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.446 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25837.0:236814 6.1 6:8eb96a14:::benchmark_data_smithi179_126206_object106433:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e650)", 2024-02-09T17:52:14.446 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:49:29.065928+0000", 2024-02-09T17:52:14.446 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 165.26140277799999, 2024-02-09T17:52:14.446 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15266506499999999, 2024-02-09T17:52:14.446 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:52:14.446 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:52:14.446 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:52:14.446 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25837", 2024-02-09T17:52:14.446 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:52:14.446 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 236814 2024-02-09T17:52:14.447 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.447 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:52:14.447 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.447 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:52:14.447 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.065928+0000", 2024-02-09T17:52:14.447 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:52:14.447 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.447 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.447 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:52:14.447 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.065928+0000", 2024-02-09T17:52:14.447 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:52:14.447 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.447 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.447 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:52:14.447 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.065929+0000", 2024-02-09T17:52:14.448 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3880000000000001e-06 2024-02-09T17:52:14.448 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.448 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.448 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:52:14.448 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.065935+0000", 2024-02-09T17:52:14.448 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6760000000000004e-06 2024-02-09T17:52:14.448 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.448 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.448 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:52:14.448 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.065950+0000", 2024-02-09T17:52:14.448 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5362e-05 2024-02-09T17:52:14.448 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.448 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.448 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:52:14.449 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.065959+0000", 2024-02-09T17:52:14.449 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4290000000000001e-06 2024-02-09T17:52:14.449 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.449 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.449 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:52:14.449 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.184807+0000", 2024-02-09T17:52:14.449 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.118848074 2024-02-09T17:52:14.449 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.449 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.449 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:52:14.449 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.184919+0000", 2024-02-09T17:52:14.449 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011227599999999999 2024-02-09T17:52:14.449 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.450 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.450 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 9,10", 2024-02-09T17:52:14.450 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.185038+0000", 2024-02-09T17:52:14.450 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000119134 2024-02-09T17:52:14.450 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.450 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.450 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:52:14.450 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.198772+0000", 2024-02-09T17:52:14.450 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013733624 2024-02-09T17:52:14.450 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.450 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.450 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:14.450 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.203165+0000", 2024-02-09T17:52:14.451 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0043933399999999999 2024-02-09T17:52:14.451 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.451 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.451 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:14.451 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.218443+0000", 2024-02-09T17:52:14.451 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.015278214 2024-02-09T17:52:14.451 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.451 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.451 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:52:14.451 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.218514+0000", 2024-02-09T17:52:14.451 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.0494000000000006e-05 2024-02-09T17:52:14.451 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.451 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.451 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:52:14.452 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.218593+0000", 2024-02-09T17:52:14.452 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.9054000000000006e-05 2024-02-09T17:52:14.452 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:52:14.452 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:52:14.452 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:52:14.452 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.452 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.452 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25837.0:237156 6.1 6:8bdaac68:::benchmark_data_smithi179_126206_object106775:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e650)", 2024-02-09T17:52:14.452 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:49:29.581251+0000", 2024-02-09T17:52:14.452 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 164.746079769, 2024-02-09T17:52:14.452 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.154685823, 2024-02-09T17:52:14.452 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:52:14.453 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:52:14.453 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:52:14.453 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25837", 2024-02-09T17:52:14.453 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:52:14.453 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 237156 2024-02-09T17:52:14.453 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.453 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:52:14.453 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.453 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:52:14.453 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.581251+0000", 2024-02-09T17:52:14.453 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:52:14.453 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.453 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.453 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:52:14.453 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.581251+0000", 2024-02-09T17:52:14.454 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:52:14.454 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.454 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.454 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:52:14.454 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.581252+0000", 2024-02-09T17:52:14.454 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.305e-06 2024-02-09T17:52:14.454 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.454 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.454 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:52:14.454 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.581258+0000", 2024-02-09T17:52:14.454 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.733e-06 2024-02-09T17:52:14.454 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.454 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.455 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:52:14.455 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.581270+0000", 2024-02-09T17:52:14.455 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1657999999999999e-05 2024-02-09T17:52:14.455 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.455 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.455 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:52:14.455 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.581277+0000", 2024-02-09T17:52:14.455 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1380000000000001e-06 2024-02-09T17:52:14.455 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.455 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.455 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:52:14.455 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.597925+0000", 2024-02-09T17:52:14.455 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.016648323 2024-02-09T17:52:14.456 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.456 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.456 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:52:14.456 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.598067+0000", 2024-02-09T17:52:14.456 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014238900000000001 2024-02-09T17:52:14.456 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.456 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.456 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 9,10", 2024-02-09T17:52:14.456 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.598214+0000", 2024-02-09T17:52:14.456 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000146928 2024-02-09T17:52:14.459 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.459 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.459 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:52:14.459 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.602475+0000", 2024-02-09T17:52:14.459 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.004261 2024-02-09T17:52:14.460 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.460 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.460 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:14.460 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.613072+0000", 2024-02-09T17:52:14.460 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010596639999999999 2024-02-09T17:52:14.460 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.460 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.460 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:14.460 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.735814+0000", 2024-02-09T17:52:14.460 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.122742486 2024-02-09T17:52:14.460 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.460 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.460 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:52:14.461 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.735879+0000", 2024-02-09T17:52:14.461 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4683000000000004e-05 2024-02-09T17:52:14.461 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.461 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.461 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:52:14.461 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.735937+0000", 2024-02-09T17:52:14.461 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7540000000000001e-05 2024-02-09T17:52:14.461 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:52:14.461 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:52:14.461 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:52:14.461 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.461 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.461 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25837.0:237191 6.1 6:849b175c:::benchmark_data_smithi179_126206_object106810:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e650)", 2024-02-09T17:52:14.462 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:49:29.628563+0000", 2024-02-09T17:52:14.462 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 164.69876768399999, 2024-02-09T17:52:14.462 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15515859200000001, 2024-02-09T17:52:14.462 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:52:14.462 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:52:14.462 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:52:14.462 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25837", 2024-02-09T17:52:14.462 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:52:14.462 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 237191 2024-02-09T17:52:14.462 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.462 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:52:14.462 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.462 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:52:14.462 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.628563+0000", 2024-02-09T17:52:14.463 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:52:14.463 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.463 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.463 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:52:14.463 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.628563+0000", 2024-02-09T17:52:14.463 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:52:14.463 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.463 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.463 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:52:14.463 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.628565+0000", 2024-02-09T17:52:14.463 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7540000000000001e-06 2024-02-09T17:52:14.463 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.463 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.464 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:52:14.464 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.628572+0000", 2024-02-09T17:52:14.464 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2810000000000003e-06 2024-02-09T17:52:14.464 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.464 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.464 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:52:14.464 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.628591+0000", 2024-02-09T17:52:14.464 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8553999999999999e-05 2024-02-09T17:52:14.464 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.464 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.464 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:52:14.464 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.628600+0000", 2024-02-09T17:52:14.464 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.4550000000000001e-06 2024-02-09T17:52:14.464 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.465 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.465 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:52:14.465 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.762772+0000", 2024-02-09T17:52:14.465 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.134172286 2024-02-09T17:52:14.465 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.465 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.465 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:52:14.465 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.762911+0000", 2024-02-09T17:52:14.465 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000138844 2024-02-09T17:52:14.465 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.465 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.465 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 9,10", 2024-02-09T17:52:14.465 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.763035+0000", 2024-02-09T17:52:14.465 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012403300000000001 2024-02-09T17:52:14.466 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.466 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.466 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:14.466 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.771870+0000", 2024-02-09T17:52:14.466 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0088346220000000003 2024-02-09T17:52:14.466 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.466 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.466 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:52:14.466 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.783598+0000", 2024-02-09T17:52:14.466 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011728456999999999 2024-02-09T17:52:14.466 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.466 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.466 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:14.466 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.783642+0000", 2024-02-09T17:52:14.467 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4039999999999998e-05 2024-02-09T17:52:14.467 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.467 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.467 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:52:14.467 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.783678+0000", 2024-02-09T17:52:14.467 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6202999999999999e-05 2024-02-09T17:52:14.467 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.467 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.467 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:52:14.467 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.783722+0000", 2024-02-09T17:52:14.467 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3062999999999997e-05 2024-02-09T17:52:14.467 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:52:14.467 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:52:14.467 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:52:14.468 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.468 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.468 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25837.0:237768 6.1 6:8a49837c:::benchmark_data_smithi179_126206_object107387:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e650)", 2024-02-09T17:52:14.468 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:49:30.433317+0000", 2024-02-09T17:52:14.468 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 163.89401391300001, 2024-02-09T17:52:14.468 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.16119182700000001, 2024-02-09T17:52:14.468 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:52:14.468 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:52:14.468 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:52:14.468 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25837", 2024-02-09T17:52:14.468 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:52:14.468 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 237768 2024-02-09T17:52:14.468 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.468 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:52:14.469 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.469 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:52:14.469 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:30.433317+0000", 2024-02-09T17:52:14.469 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:52:14.469 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.469 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.469 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:52:14.469 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:30.433317+0000", 2024-02-09T17:52:14.469 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:52:14.469 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.469 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.469 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:52:14.469 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:30.433318+0000", 2024-02-09T17:52:14.469 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4759999999999999e-06 2024-02-09T17:52:14.469 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.470 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.470 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:52:14.470 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:30.433324+0000", 2024-02-09T17:52:14.470 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0460000000000002e-06 2024-02-09T17:52:14.470 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.470 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.470 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:52:14.470 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:30.433338+0000", 2024-02-09T17:52:14.470 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3521e-05 2024-02-09T17:52:14.470 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.470 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.470 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:52:14.470 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:30.433346+0000", 2024-02-09T17:52:14.470 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2840000000000006e-06 2024-02-09T17:52:14.471 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.471 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.471 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:52:14.471 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:30.571264+0000", 2024-02-09T17:52:14.471 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.13791782499999999 2024-02-09T17:52:14.471 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.471 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.471 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:52:14.471 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:30.571360+0000", 2024-02-09T17:52:14.471 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5884000000000006e-05 2024-02-09T17:52:14.471 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.471 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.471 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 9,10", 2024-02-09T17:52:14.471 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:30.571448+0000", 2024-02-09T17:52:14.471 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8554000000000006e-05 2024-02-09T17:52:14.472 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.472 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.472 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:14.472 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:30.584991+0000", 2024-02-09T17:52:14.472 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013543084 2024-02-09T17:52:14.472 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.472 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.472 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:14.472 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:30.585031+0000", 2024-02-09T17:52:14.472 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9777000000000002e-05 2024-02-09T17:52:14.472 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.472 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.472 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:52:14.472 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:30.594412+0000", 2024-02-09T17:52:14.473 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0093810730000000002 2024-02-09T17:52:14.473 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.473 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.473 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:52:14.473 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:30.594455+0000", 2024-02-09T17:52:14.473 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3066999999999999e-05 2024-02-09T17:52:14.473 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.473 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.473 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:52:14.473 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:30.594509+0000", 2024-02-09T17:52:14.473 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3236000000000003e-05 2024-02-09T17:52:14.473 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:52:14.473 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:52:14.473 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:52:14.473 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.474 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.474 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.16440.0:61011 7.f 7:f12e60e5:::benchmark_data_smithi179_135804_object61010:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e674)", 2024-02-09T17:52:14.474 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:50:55.026216+0000", 2024-02-09T17:52:14.474 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 79.301114612999996, 2024-02-09T17:52:14.474 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.17895054299999999, 2024-02-09T17:52:14.474 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:52:14.474 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:52:14.474 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:52:14.474 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.16440", 2024-02-09T17:52:14.474 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:52:14.474 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 61011 2024-02-09T17:52:14.474 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.474 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:52:14.475 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.475 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:52:14.475 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.026216+0000", 2024-02-09T17:52:14.475 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:52:14.475 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.475 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.475 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:52:14.475 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.026216+0000", 2024-02-09T17:52:14.475 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:52:14.475 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.475 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.475 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:52:14.475 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.026218+0000", 2024-02-09T17:52:14.476 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8920000000000001e-06 2024-02-09T17:52:14.476 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.476 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.476 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:52:14.476 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.026242+0000", 2024-02-09T17:52:14.476 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4587999999999999e-05 2024-02-09T17:52:14.476 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.476 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.476 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:52:14.476 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.026257+0000", 2024-02-09T17:52:14.476 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4266e-05 2024-02-09T17:52:14.476 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.476 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.477 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:52:14.477 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.026265+0000", 2024-02-09T17:52:14.477 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1790000000000008e-06 2024-02-09T17:52:14.477 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.477 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.477 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:52:14.477 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.026335+0000", 2024-02-09T17:52:14.477 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9949000000000004e-05 2024-02-09T17:52:14.477 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.477 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.477 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:52:14.477 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.026483+0000", 2024-02-09T17:52:14.477 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014850599999999999 2024-02-09T17:52:14.478 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.478 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.478 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,9", 2024-02-09T17:52:14.478 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.026642+0000", 2024-02-09T17:52:14.478 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000158493 2024-02-09T17:52:14.478 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.478 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.478 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:52:14.478 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.030530+0000", 2024-02-09T17:52:14.478 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0038884900000000001 2024-02-09T17:52:14.478 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.478 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.478 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:14.479 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.030740+0000", 2024-02-09T17:52:14.479 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000209593 2024-02-09T17:52:14.479 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.479 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.479 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:14.479 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.205087+0000", 2024-02-09T17:52:14.479 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.17434675499999999 2024-02-09T17:52:14.479 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.479 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.479 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:52:14.479 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.205129+0000", 2024-02-09T17:52:14.479 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2512000000000002e-05 2024-02-09T17:52:14.479 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:52:14.479 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:52:14.480 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:52:14.480 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.205167+0000", 2024-02-09T17:52:14.480 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7320000000000002e-05 2024-02-09T17:52:14.480 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:52:14.480 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:52:14.480 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:52:14.480 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:52:14.480 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:52:14.480 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:52:14.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:14 smithi099 ceph-mon[39420]: pgmap v1770: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.0 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 766 op/s; 1/294933 objects degraded (0.000%); 20041/294933 objects misplaced (6.795%); 29 MiB/s, 469 objects/s recovering 2024-02-09T17:52:14.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:14 smithi099 ceph-mon[39420]: osdmap e707: 12 total, 12 up, 12 in 2024-02-09T17:52:14.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:14 smithi099 ceph-mon[39420]: from='client.16545 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:52:14.552 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:52:14.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:14 smithi162 ceph-mon[44304]: pgmap v1770: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.0 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 766 op/s; 1/294933 objects degraded (0.000%); 20041/294933 objects misplaced (6.795%); 29 MiB/s, 469 objects/s recovering 2024-02-09T17:52:14.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:14 smithi162 ceph-mon[44304]: osdmap e707: 12 total, 12 up, 12 in 2024-02-09T17:52:14.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:14 smithi162 ceph-mon[44304]: from='client.16545 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:52:14.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:14.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:52:14.618 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:52:14.664 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:52:14.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:14 smithi179 ceph-mon[43975]: pgmap v1770: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.0 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 766 op/s; 1/294933 objects degraded (0.000%); 20041/294933 objects misplaced (6.795%); 29 MiB/s, 469 objects/s recovering 2024-02-09T17:52:14.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:14 smithi179 ceph-mon[43975]: osdmap e707: 12 total, 12 up, 12 in 2024-02-09T17:52:14.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:14 smithi179 ceph-mon[43975]: from='client.16545 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:52:14.701 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:14.727 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:52:14.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:14.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:52:14.834 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:52:14.870 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:15.232 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:52:15.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:15.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:52:15.339 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:52:15.378 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:15.407 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:52:15.410 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:15.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:52:15.511 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:52:15.548 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:15.573 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-09T17:52:16.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:16 smithi179 ceph-mon[43975]: pgmap v1772: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.0 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 567 op/s; 1/293721 objects degraded (0.000%); 19848/293721 objects misplaced (6.757%); 18 MiB/s, 287 objects/s recovering 2024-02-09T17:52:16.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:16 smithi162 ceph-mon[44304]: pgmap v1772: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.0 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 567 op/s; 1/293721 objects degraded (0.000%); 19848/293721 objects misplaced (6.757%); 18 MiB/s, 287 objects/s recovering 2024-02-09T17:52:16.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:16 smithi099 ceph-mon[39420]: pgmap v1772: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 6.0 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 567 op/s; 1/293721 objects degraded (0.000%); 19848/293721 objects misplaced (6.757%); 18 MiB/s, 287 objects/s recovering 2024-02-09T17:52:16.804 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:52:16.813 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:52:16.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:16.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:52:16.920 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:52:16.992 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:17.040 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:52:17.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:17.053 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:52:17.155 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:52:17.202 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:17.512 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:52:17.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:17 smithi179 ceph-mon[43975]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:52:17.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:17 smithi179 ceph-mon[43975]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:52:17.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:17 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 1/293721 objects degraded (0.000%), 1 pg degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:52:17.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:17 smithi179 ceph-mon[43975]: osdmap e708: 12 total, 12 up, 12 in 2024-02-09T17:52:17.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:17 smithi179 ceph-mon[43975]: pgmap v1774: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 698 op/s; 1/287265 objects degraded (0.000%); 18048/287265 objects misplaced (6.283%); 30 MiB/s, 479 objects/s recovering 2024-02-09T17:52:17.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:17 smithi162 ceph-mon[44304]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:52:17.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:17 smithi162 ceph-mon[44304]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:52:17.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:17 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 1/293721 objects degraded (0.000%), 1 pg degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:52:17.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:17 smithi162 ceph-mon[44304]: osdmap e708: 12 total, 12 up, 12 in 2024-02-09T17:52:17.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:17 smithi162 ceph-mon[44304]: pgmap v1774: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 698 op/s; 1/287265 objects degraded (0.000%); 18048/287265 objects misplaced (6.283%); 30 MiB/s, 479 objects/s recovering 2024-02-09T17:52:17.950 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:17 smithi099 ceph-mon[39420]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:52:17.950 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:17 smithi099 ceph-mon[39420]: from='mgr.14148 ' entity='mgr.y' 2024-02-09T17:52:17.950 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:17 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 1/293721 objects degraded (0.000%), 1 pg degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:52:17.950 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:17 smithi099 ceph-mon[39420]: osdmap e708: 12 total, 12 up, 12 in 2024-02-09T17:52:17.950 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:17 smithi099 ceph-mon[39420]: pgmap v1774: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 698 op/s; 1/287265 objects degraded (0.000%); 18048/287265 objects misplaced (6.283%); 30 MiB/s, 479 objects/s recovering 2024-02-09T17:52:20.275 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:52:20.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:20.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:52:20.383 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:52:20.383 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is unset 2024-02-09T17:52:20.424 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:20 smithi099 ceph-mon[39420]: 7.3 scrub starts 2024-02-09T17:52:20.424 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:20 smithi099 ceph-mon[39420]: pgmap v1775: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 625 op/s; 1/284865 objects degraded (0.000%); 17681/284865 objects misplaced (6.207%); 25 MiB/s, 394 objects/s recovering 2024-02-09T17:52:20.424 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:20 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2679353239' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:52:20.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:20 smithi179 ceph-mon[43975]: 7.3 scrub starts 2024-02-09T17:52:20.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:20 smithi179 ceph-mon[43975]: pgmap v1775: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 625 op/s; 1/284865 objects degraded (0.000%); 17681/284865 objects misplaced (6.207%); 25 MiB/s, 394 objects/s recovering 2024-02-09T17:52:20.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:20 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2679353239' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:52:20.425 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:20.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:20 smithi162 ceph-mon[44304]: 7.3 scrub starts 2024-02-09T17:52:20.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:20 smithi162 ceph-mon[44304]: pgmap v1775: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 625 op/s; 1/284865 objects degraded (0.000%); 17681/284865 objects misplaced (6.207%); 25 MiB/s, 394 objects/s recovering 2024-02-09T17:52:20.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:20 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2679353239' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:52:20.633 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:52:20.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:20.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:52:20.739 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:52:20.793 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:20.829 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:52:20.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:20.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:52:20.935 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:52:21.020 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:21.049 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:52:21.056 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:21.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:52:21.157 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:52:21.218 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:21.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:21 smithi179 ceph-mon[43975]: 7.3 scrub ok 2024-02-09T17:52:21.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:21 smithi179 ceph-mon[43975]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:52:21.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:21 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2679353239' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:52:21.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:21 smithi179 ceph-mon[43975]: osdmap e709: 12 total, 12 up, 12 in 2024-02-09T17:52:21.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:21 smithi162 ceph-mon[44304]: 7.3 scrub ok 2024-02-09T17:52:21.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:21 smithi162 ceph-mon[44304]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:52:21.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:21 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2679353239' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:52:21.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:21 smithi162 ceph-mon[44304]: osdmap e709: 12 total, 12 up, 12 in 2024-02-09T17:52:21.457 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:21 smithi099 ceph-mon[39420]: 7.3 scrub ok 2024-02-09T17:52:21.457 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:21 smithi099 ceph-mon[39420]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:52:21.457 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:21 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2679353239' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:52:21.457 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:21 smithi099 ceph-mon[39420]: osdmap e709: 12 total, 12 up, 12 in 2024-02-09T17:52:22.037 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:52:22.180 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:52:22.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:22.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:52:22.286 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:52:22.300 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:22 smithi099 ceph-mon[39420]: pgmap v1777: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 564 op/s; 1/281688 objects degraded (0.000%); 16908/281688 objects misplaced (6.002%); 25 MiB/s, 400 objects/s recovering 2024-02-09T17:52:22.300 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:22 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 1/281688 objects degraded (0.000%), 1 pg degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:52:22.300 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:22 smithi099 ceph-mon[39420]: osdmap e710: 12 total, 12 up, 12 in 2024-02-09T17:52:22.346 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:22.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:22 smithi162 ceph-mon[44304]: pgmap v1777: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 564 op/s; 1/281688 objects degraded (0.000%); 16908/281688 objects misplaced (6.002%); 25 MiB/s, 400 objects/s recovering 2024-02-09T17:52:22.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:22 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 1/281688 objects degraded (0.000%), 1 pg degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:52:22.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:22 smithi162 ceph-mon[44304]: osdmap e710: 12 total, 12 up, 12 in 2024-02-09T17:52:22.428 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:22 smithi179 ceph-mon[43975]: pgmap v1777: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 564 op/s; 1/281688 objects degraded (0.000%); 16908/281688 objects misplaced (6.002%); 25 MiB/s, 400 objects/s recovering 2024-02-09T17:52:22.428 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:22 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 1/281688 objects degraded (0.000%), 1 pg degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:52:22.428 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:22 smithi179 ceph-mon[43975]: osdmap e710: 12 total, 12 up, 12 in 2024-02-09T17:52:22.707 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:52:22.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:22.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:52:22.815 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:52:22.846 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:24.084 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set noscrub 2024-02-09T17:52:24.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:24 smithi179 ceph-mon[43975]: pgmap v1779: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 699 op/s; 1/275313 objects degraded (0.000%); 15550/275313 objects misplaced (5.648%); 25 MiB/s, 399 objects/s recovering 2024-02-09T17:52:24.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:24 smithi162 ceph-mon[44304]: pgmap v1779: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 699 op/s; 1/275313 objects degraded (0.000%); 15550/275313 objects misplaced (5.648%); 25 MiB/s, 399 objects/s recovering 2024-02-09T17:52:24.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:24 smithi099 ceph-mon[39420]: pgmap v1779: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 699 op/s; 1/275313 objects degraded (0.000%); 15550/275313 objects misplaced (5.648%); 25 MiB/s, 399 objects/s recovering 2024-02-09T17:52:26.228 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:26 smithi099 ceph-mon[39420]: pgmap v1780: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 549 op/s; 1/274068 objects degraded (0.000%); 15334/274068 objects misplaced (5.595%); 21 MiB/s, 339 objects/s recovering 2024-02-09T17:52:26.229 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:26 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:52:26.229 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:26 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:52:26.229 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:26 smithi099 ceph-mon[39420]: osdmap e711: 12 total, 12 up, 12 in 2024-02-09T17:52:26.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:26 smithi179 ceph-mon[43975]: pgmap v1780: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 549 op/s; 1/274068 objects degraded (0.000%); 15334/274068 objects misplaced (5.595%); 21 MiB/s, 339 objects/s recovering 2024-02-09T17:52:26.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:26 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:52:26.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:26 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:52:26.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:26 smithi179 ceph-mon[43975]: osdmap e711: 12 total, 12 up, 12 in 2024-02-09T17:52:26.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:26 smithi162 ceph-mon[44304]: pgmap v1780: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 39 active+clean; 5.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 549 op/s; 1/274068 objects degraded (0.000%); 15334/274068 objects misplaced (5.595%); 21 MiB/s, 339 objects/s recovering 2024-02-09T17:52:26.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:26 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:52:26.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:26 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:52:26.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:26 smithi162 ceph-mon[44304]: osdmap e711: 12 total, 12 up, 12 in 2024-02-09T17:52:27.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:27 smithi099 ceph-mon[39420]: osdmap e712: 12 total, 12 up, 12 in 2024-02-09T17:52:27.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:27 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 1/274068 objects degraded (0.000%), 1 pg degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:52:27.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:27 smithi179 ceph-mon[43975]: osdmap e712: 12 total, 12 up, 12 in 2024-02-09T17:52:27.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:27 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 1/274068 objects degraded (0.000%), 1 pg degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:52:27.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:27 smithi162 ceph-mon[44304]: osdmap e712: 12 total, 12 up, 12 in 2024-02-09T17:52:27.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:27 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 1/274068 objects degraded (0.000%), 1 pg degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:52:27.921 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:52:27.921 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [ 2024-02-09T17:52:27.922 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:27.922 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.16440.0:127469 7.6 7.3c0a59f6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e712)", 2024-02-09T17:52:27.922 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:52:27.082427+0000", 2024-02-09T17:52:27.922 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.0041440289999999996, 2024-02-09T17:52:27.922 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0041672050000000002, 2024-02-09T17:52:27.922 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:52:27.922 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "queued for pg", 2024-02-09T17:52:27.922 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:52:27.922 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.16440", 2024-02-09T17:52:27.922 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:52:27.922 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 127469 2024-02-09T17:52:27.922 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:27.923 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:52:27.923 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:27.923 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:52:27.923 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:27.082427+0000", 2024-02-09T17:52:27.923 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:52:27.923 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:27.923 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:27.923 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:52:27.923 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:27.082427+0000", 2024-02-09T17:52:27.923 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:52:27.923 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:27.923 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:27.924 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:52:27.924 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:27.082429+0000", 2024-02-09T17:52:27.924 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.77e-06 2024-02-09T17:52:27.924 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:27.924 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:27.924 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:52:27.924 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:27.082437+0000", 2024-02-09T17:52:27.924 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.8550000000000002e-06 2024-02-09T17:52:27.924 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:27.924 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:27.924 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:52:27.925 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:27.082454+0000", 2024-02-09T17:52:27.925 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.6874000000000001e-05 2024-02-09T17:52:27.925 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:27.925 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:27.925 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:52:27.925 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:27.082464+0000", 2024-02-09T17:52:27.925 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0128e-05 2024-02-09T17:52:27.925 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:27.925 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:27.925 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:27.925 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:27.925 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:27.925 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.16440.0:127472 7.6 7.5a92c316 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e712)", 2024-02-09T17:52:27.926 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:52:27.082481+0000", 2024-02-09T17:52:27.926 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.0040902930000000001, 2024-02-09T17:52:27.926 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0041530489999999998, 2024-02-09T17:52:27.926 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:52:27.926 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "queued for pg", 2024-02-09T17:52:27.926 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:52:27.926 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.16440", 2024-02-09T17:52:27.926 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:52:27.926 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 127472 2024-02-09T17:52:27.926 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:27.926 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:52:27.926 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:27.926 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:52:27.926 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:27.082481+0000", 2024-02-09T17:52:27.927 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:52:27.927 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:27.927 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:27.927 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:52:27.927 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:27.082481+0000", 2024-02-09T17:52:27.927 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:52:27.927 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:27.927 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:27.927 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:52:27.927 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:27.082482+0000", 2024-02-09T17:52:27.927 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.03e-07 2024-02-09T17:52:27.927 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:27.927 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:27.928 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:52:27.928 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:27.082484+0000", 2024-02-09T17:52:27.928 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.9179999999999998e-06 2024-02-09T17:52:27.928 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:27.928 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:27.928 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:52:27.928 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:27.082491+0000", 2024-02-09T17:52:27.928 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.3090000000000001e-06 2024-02-09T17:52:27.928 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:27.928 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:27.928 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:52:27.928 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:27.082496+0000", 2024-02-09T17:52:27.928 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1100000000000002e-06 2024-02-09T17:52:27.928 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:27.929 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:27.929 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:27.929 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:27.929 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:27.929 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.16440.0:127478 7.6 7.d5ab3896 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e712)", 2024-02-09T17:52:27.929 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:52:27.082502+0000", 2024-02-09T17:52:27.929 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.0040694360000000001, 2024-02-09T17:52:27.929 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0041661429999999998, 2024-02-09T17:52:27.929 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:52:27.929 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "queued for pg", 2024-02-09T17:52:27.929 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:52:27.929 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.16440", 2024-02-09T17:52:27.929 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:52:27.929 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 127478 2024-02-09T17:52:27.929 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:27.930 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:52:27.930 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:27.930 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:52:27.930 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:27.082502+0000", 2024-02-09T17:52:27.930 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:52:27.930 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:27.930 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:27.930 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:52:27.930 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:27.082502+0000", 2024-02-09T17:52:27.930 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:52:27.930 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:27.930 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:27.930 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:52:27.931 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:27.082502+0000", 2024-02-09T17:52:27.931 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.34e-07 2024-02-09T17:52:27.931 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:27.931 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:27.931 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:52:27.931 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:27.082504+0000", 2024-02-09T17:52:27.931 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.9709999999999998e-06 2024-02-09T17:52:27.931 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:27.931 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:27.931 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:52:27.931 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:27.082510+0000", 2024-02-09T17:52:27.931 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3469999999999998e-06 2024-02-09T17:52:27.931 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:27.932 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:27.932 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:52:27.932 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:27.082514+0000", 2024-02-09T17:52:27.932 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.8019999999999999e-06 2024-02-09T17:52:27.932 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:27.932 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:27.932 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:27.932 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:27.932 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T17:52:27.932 INFO:teuthology.orchestra.run.smithi099.stdout: "num_ops": 3 2024-02-09T17:52:27.932 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:52:28.196 INFO:teuthology.orchestra.run.smithi099.stderr:set 1.7 pg_upmap mapping to [1,8,6] 2024-02-09T17:52:28.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:28 smithi099 ceph-mon[39420]: pgmap v1783: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 40 active+clean; 5.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 783 op/s; 1/267576 objects degraded (0.000%); 13215/267576 objects misplaced (4.939%); 38 MiB/s, 615 objects/s recovering 2024-02-09T17:52:28.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:28 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4069215504' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.7", "id": [1, 8, 6]}]: dispatch 2024-02-09T17:52:28.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:28 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.7", "id": [1, 8, 6]}]: dispatch 2024-02-09T17:52:28.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:28 smithi179 ceph-mon[43975]: pgmap v1783: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 40 active+clean; 5.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 783 op/s; 1/267576 objects degraded (0.000%); 13215/267576 objects misplaced (4.939%); 38 MiB/s, 615 objects/s recovering 2024-02-09T17:52:28.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:28 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4069215504' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.7", "id": [1, 8, 6]}]: dispatch 2024-02-09T17:52:28.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:28 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.7", "id": [1, 8, 6]}]: dispatch 2024-02-09T17:52:28.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:28 smithi162 ceph-mon[44304]: pgmap v1783: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 40 active+clean; 5.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 783 op/s; 1/267576 objects degraded (0.000%); 13215/267576 objects misplaced (4.939%); 38 MiB/s, 615 objects/s recovering 2024-02-09T17:52:28.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:28 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4069215504' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.7", "id": [1, 8, 6]}]: dispatch 2024-02-09T17:52:28.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:28 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.7", "id": [1, 8, 6]}]: dispatch 2024-02-09T17:52:29.492 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.7", "id": [1, 8, 6]}]': finished 2024-02-09T17:52:29.492 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: osdmap e713: 12 total, 12 up, 12 in 2024-02-09T17:52:29.492 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:52:29.492 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:52:29.492 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:52:29.492 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {auth=36,config=2,health=12,kv=50} crc {auth=3726034028,config=2679774077,health=2358433560,kv=2479472241}) 2024-02-09T17:52:29.492 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=2581019592,logm=693138112}) 2024-02-09T17:52:29.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2641530603}) 2024-02-09T17:52:29.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3369378131}) 2024-02-09T17:52:29.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=987794322}) 2024-02-09T17:52:29.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3543500629}) 2024-02-09T17:52:29.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1794125110}) 2024-02-09T17:52:29.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2685904833}) 2024-02-09T17:52:29.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1815408220}) 2024-02-09T17:52:29.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1049761385}) 2024-02-09T17:52:29.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3692899555}) 2024-02-09T17:52:29.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3725711444}) 2024-02-09T17:52:29.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1329952475}) 2024-02-09T17:52:29.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1840565149}) 2024-02-09T17:52:29.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4013453060}) 2024-02-09T17:52:29.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3474624020}) 2024-02-09T17:52:29.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=875156298}) 2024-02-09T17:52:29.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1936445995}) 2024-02-09T17:52:29.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3888672456}) 2024-02-09T17:52:29.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4193149348}) 2024-02-09T17:52:29.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=784059894}) 2024-02-09T17:52:29.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1925472296}) 2024-02-09T17:52:29.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=616085356}) 2024-02-09T17:52:29.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2352593797}) 2024-02-09T17:52:29.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=75294404}) 2024-02-09T17:52:29.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=747995734}) 2024-02-09T17:52:29.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3039983195}) 2024-02-09T17:52:29.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3376180212}) 2024-02-09T17:52:29.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1435771368}) 2024-02-09T17:52:29.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1911286590}) 2024-02-09T17:52:29.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2765284494}) 2024-02-09T17:52:29.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4193510056}) 2024-02-09T17:52:29.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2642213996}) 2024-02-09T17:52:29.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=305334610}) 2024-02-09T17:52:29.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2148257381}) 2024-02-09T17:52:29.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=716823422}) 2024-02-09T17:52:29.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3973661381}) 2024-02-09T17:52:29.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=223018748}) 2024-02-09T17:52:29.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3921419015}) 2024-02-09T17:52:29.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1912924333}) 2024-02-09T17:52:29.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1065880929}) 2024-02-09T17:52:29.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3930132954}) 2024-02-09T17:52:29.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2433867159}) 2024-02-09T17:52:29.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=394111936}) 2024-02-09T17:52:29.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1128309119}) 2024-02-09T17:52:29.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1060357053}) 2024-02-09T17:52:29.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2950546449}) 2024-02-09T17:52:29.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=612507347}) 2024-02-09T17:52:29.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3195810193}) 2024-02-09T17:52:29.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2651683485}) 2024-02-09T17:52:29.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3631291717}) 2024-02-09T17:52:29.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=521124260}) 2024-02-09T17:52:29.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3962259200}) 2024-02-09T17:52:29.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2293813173}) 2024-02-09T17:52:29.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=42977872}) 2024-02-09T17:52:29.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1416517130}) 2024-02-09T17:52:29.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=135830465}) 2024-02-09T17:52:29.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=895560970}) 2024-02-09T17:52:29.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1275861334}) 2024-02-09T17:52:29.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=370396730}) 2024-02-09T17:52:29.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3091117485}) 2024-02-09T17:52:29.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1053172150}) 2024-02-09T17:52:29.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3307235001}) 2024-02-09T17:52:29.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1035134272}) 2024-02-09T17:52:29.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=87,mdsmap=3,mgr=10} crc {logm=3352315522,mdsmap=3984425268,mgr=1984106602}) 2024-02-09T17:52:29.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {mgr=8,mgr_command_descs=1,mgr_metadata=2,mgrstat=9,mon_config_key=80} crc {mgr=847209252,mgr_command_descs=708342470,mgr_metadata=670763949,mgrstat=3675038452,mon_config_key=2748818933}) 2024-02-09T17:52:29.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4081470227}) 2024-02-09T17:52:29.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1965080380}) 2024-02-09T17:52:29.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=1,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=81} crc {mon_config_key=2902139199,monmap=423865542,osd_metadata=545325961,osd_pg_creating=1980598198,osdmap=944141559}) 2024-02-09T17:52:29.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=26} crc {osdmap=3142275117}) 2024-02-09T17:52:29.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.7", "id": [1, 8, 6]}]': finished 2024-02-09T17:52:29.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: osdmap e713: 12 total, 12 up, 12 in 2024-02-09T17:52:29.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:52:29.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:52:29.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:52:29.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {auth=36,config=2,health=12,kv=50} crc {auth=3726034028,config=2679774077,health=2358433560,kv=2479472241}) 2024-02-09T17:52:29.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=2581019592,logm=693138112}) 2024-02-09T17:52:29.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2641530603}) 2024-02-09T17:52:29.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3369378131}) 2024-02-09T17:52:29.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=987794322}) 2024-02-09T17:52:29.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3543500629}) 2024-02-09T17:52:29.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1794125110}) 2024-02-09T17:52:29.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2685904833}) 2024-02-09T17:52:29.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1815408220}) 2024-02-09T17:52:29.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1049761385}) 2024-02-09T17:52:29.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3692899555}) 2024-02-09T17:52:29.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3725711444}) 2024-02-09T17:52:29.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1329952475}) 2024-02-09T17:52:29.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1840565149}) 2024-02-09T17:52:29.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4013453060}) 2024-02-09T17:52:29.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3474624020}) 2024-02-09T17:52:29.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=875156298}) 2024-02-09T17:52:29.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1936445995}) 2024-02-09T17:52:29.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3888672456}) 2024-02-09T17:52:29.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4193149348}) 2024-02-09T17:52:29.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=784059894}) 2024-02-09T17:52:29.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1925472296}) 2024-02-09T17:52:29.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=616085356}) 2024-02-09T17:52:29.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2352593797}) 2024-02-09T17:52:29.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=75294404}) 2024-02-09T17:52:29.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=747995734}) 2024-02-09T17:52:29.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3039983195}) 2024-02-09T17:52:29.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3376180212}) 2024-02-09T17:52:29.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1435771368}) 2024-02-09T17:52:29.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1911286590}) 2024-02-09T17:52:29.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2765284494}) 2024-02-09T17:52:29.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4193510056}) 2024-02-09T17:52:29.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2642213996}) 2024-02-09T17:52:29.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=305334610}) 2024-02-09T17:52:29.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2148257381}) 2024-02-09T17:52:29.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=716823422}) 2024-02-09T17:52:29.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3973661381}) 2024-02-09T17:52:29.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=223018748}) 2024-02-09T17:52:29.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3921419015}) 2024-02-09T17:52:29.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1912924333}) 2024-02-09T17:52:29.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1065880929}) 2024-02-09T17:52:29.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3930132954}) 2024-02-09T17:52:29.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2433867159}) 2024-02-09T17:52:29.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=394111936}) 2024-02-09T17:52:29.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1128309119}) 2024-02-09T17:52:29.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1060357053}) 2024-02-09T17:52:29.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2950546449}) 2024-02-09T17:52:29.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=612507347}) 2024-02-09T17:52:29.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3195810193}) 2024-02-09T17:52:29.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2651683485}) 2024-02-09T17:52:29.677 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3631291717}) 2024-02-09T17:52:29.677 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=521124260}) 2024-02-09T17:52:29.677 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3962259200}) 2024-02-09T17:52:29.677 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2293813173}) 2024-02-09T17:52:29.677 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=42977872}) 2024-02-09T17:52:29.677 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1416517130}) 2024-02-09T17:52:29.677 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=135830465}) 2024-02-09T17:52:29.677 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=895560970}) 2024-02-09T17:52:29.677 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1275861334}) 2024-02-09T17:52:29.677 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=370396730}) 2024-02-09T17:52:29.677 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3091117485}) 2024-02-09T17:52:29.677 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1053172150}) 2024-02-09T17:52:29.677 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3307235001}) 2024-02-09T17:52:29.677 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1035134272}) 2024-02-09T17:52:29.677 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=87,mdsmap=3,mgr=10} crc {logm=3352315522,mdsmap=3984425268,mgr=1984106602}) 2024-02-09T17:52:29.678 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {mgr=8,mgr_command_descs=1,mgr_metadata=2,mgrstat=9,mon_config_key=80} crc {mgr=847209252,mgr_command_descs=708342470,mgr_metadata=670763949,mgrstat=3675038452,mon_config_key=2748818933}) 2024-02-09T17:52:29.678 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4081470227}) 2024-02-09T17:52:29.678 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1965080380}) 2024-02-09T17:52:29.678 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=1,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=81} crc {mon_config_key=2902139199,monmap=423865542,osd_metadata=545325961,osd_pg_creating=1980598198,osdmap=944141559}) 2024-02-09T17:52:29.678 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=26} crc {osdmap=3142275117}) 2024-02-09T17:52:29.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.7", "id": [1, 8, 6]}]': finished 2024-02-09T17:52:29.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: osdmap e713: 12 total, 12 up, 12 in 2024-02-09T17:52:29.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:52:29.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:52:29.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:52:29.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {auth=36,config=2,health=12,kv=50} crc {auth=3726034028,config=2679774077,health=2358433560,kv=2479472241}) 2024-02-09T17:52:29.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=2581019592,logm=693138112}) 2024-02-09T17:52:29.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2641530603}) 2024-02-09T17:52:29.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3369378131}) 2024-02-09T17:52:29.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=987794322}) 2024-02-09T17:52:29.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3543500629}) 2024-02-09T17:52:29.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1794125110}) 2024-02-09T17:52:29.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2685904833}) 2024-02-09T17:52:29.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1815408220}) 2024-02-09T17:52:29.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1049761385}) 2024-02-09T17:52:29.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3692899555}) 2024-02-09T17:52:29.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3725711444}) 2024-02-09T17:52:29.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1329952475}) 2024-02-09T17:52:29.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1840565149}) 2024-02-09T17:52:29.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4013453060}) 2024-02-09T17:52:29.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3474624020}) 2024-02-09T17:52:29.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=875156298}) 2024-02-09T17:52:29.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1936445995}) 2024-02-09T17:52:29.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3888672456}) 2024-02-09T17:52:29.681 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4193149348}) 2024-02-09T17:52:29.681 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=784059894}) 2024-02-09T17:52:29.681 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1925472296}) 2024-02-09T17:52:29.681 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=616085356}) 2024-02-09T17:52:29.681 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2352593797}) 2024-02-09T17:52:29.681 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=75294404}) 2024-02-09T17:52:29.681 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=747995734}) 2024-02-09T17:52:29.681 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3039983195}) 2024-02-09T17:52:29.681 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3376180212}) 2024-02-09T17:52:29.681 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1435771368}) 2024-02-09T17:52:29.681 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1911286590}) 2024-02-09T17:52:29.681 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2765284494}) 2024-02-09T17:52:29.681 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4193510056}) 2024-02-09T17:52:29.681 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2642213996}) 2024-02-09T17:52:29.681 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=305334610}) 2024-02-09T17:52:29.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2148257381}) 2024-02-09T17:52:29.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=716823422}) 2024-02-09T17:52:29.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3973661381}) 2024-02-09T17:52:29.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=223018748}) 2024-02-09T17:52:29.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3921419015}) 2024-02-09T17:52:29.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1912924333}) 2024-02-09T17:52:29.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1065880929}) 2024-02-09T17:52:29.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3930132954}) 2024-02-09T17:52:29.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2433867159}) 2024-02-09T17:52:29.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=394111936}) 2024-02-09T17:52:29.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1128309119}) 2024-02-09T17:52:29.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1060357053}) 2024-02-09T17:52:29.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2950546449}) 2024-02-09T17:52:29.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=612507347}) 2024-02-09T17:52:29.683 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3195810193}) 2024-02-09T17:52:29.683 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2651683485}) 2024-02-09T17:52:29.683 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3631291717}) 2024-02-09T17:52:29.683 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=521124260}) 2024-02-09T17:52:29.683 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3962259200}) 2024-02-09T17:52:29.683 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2293813173}) 2024-02-09T17:52:29.683 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=42977872}) 2024-02-09T17:52:29.683 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1416517130}) 2024-02-09T17:52:29.683 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=135830465}) 2024-02-09T17:52:29.683 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=895560970}) 2024-02-09T17:52:29.683 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1275861334}) 2024-02-09T17:52:29.683 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=370396730}) 2024-02-09T17:52:29.683 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3091117485}) 2024-02-09T17:52:29.683 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1053172150}) 2024-02-09T17:52:29.683 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3307235001}) 2024-02-09T17:52:29.684 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1035134272}) 2024-02-09T17:52:29.684 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=87,mdsmap=3,mgr=10} crc {logm=3352315522,mdsmap=3984425268,mgr=1984106602}) 2024-02-09T17:52:29.684 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {mgr=8,mgr_command_descs=1,mgr_metadata=2,mgrstat=9,mon_config_key=80} crc {mgr=847209252,mgr_command_descs=708342470,mgr_metadata=670763949,mgrstat=3675038452,mon_config_key=2748818933}) 2024-02-09T17:52:29.684 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4081470227}) 2024-02-09T17:52:29.684 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1965080380}) 2024-02-09T17:52:29.684 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=1,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=81} crc {mon_config_key=2902139199,monmap=423865542,osd_metadata=545325961,osd_pg_creating=1980598198,osdmap=944141559}) 2024-02-09T17:52:29.684 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=26} crc {osdmap=3142275117}) 2024-02-09T17:52:29.894 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:52:29.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:29.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:52:29.999 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:52:30.050 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:30.127 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:52:30.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:30.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:52:30.290 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:52:30.325 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:30.350 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:52:30.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:30.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:52:30.456 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:52:30.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:30 smithi099 ceph-mon[39420]: pgmap v1785: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 40 active+clean; 5.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 556 op/s; 1/265293 objects degraded (0.000%); 12760/265293 objects misplaced (4.810%); 29 MiB/s, 464 objects/s recovering 2024-02-09T17:52:30.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:30 smithi099 ceph-mon[39420]: osdmap e714: 12 total, 12 up, 12 in 2024-02-09T17:52:30.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:30 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:52:30.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:30 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:52:30.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:30 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:52:30.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:30 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:52:30.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:30 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:52:30.503 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:30.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:30 smithi162 ceph-mon[44304]: pgmap v1785: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 40 active+clean; 5.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 556 op/s; 1/265293 objects degraded (0.000%); 12760/265293 objects misplaced (4.810%); 29 MiB/s, 464 objects/s recovering 2024-02-09T17:52:30.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:30 smithi162 ceph-mon[44304]: osdmap e714: 12 total, 12 up, 12 in 2024-02-09T17:52:30.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:30 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:52:30.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:30 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:52:30.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:30 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:52:30.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:30 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:52:30.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:30 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:52:30.678 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:30 smithi179 ceph-mon[43975]: pgmap v1785: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 40 active+clean; 5.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 556 op/s; 1/265293 objects degraded (0.000%); 12760/265293 objects misplaced (4.810%); 29 MiB/s, 464 objects/s recovering 2024-02-09T17:52:30.679 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:30 smithi179 ceph-mon[43975]: osdmap e714: 12 total, 12 up, 12 in 2024-02-09T17:52:30.679 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:30 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:52:30.679 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:30 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:52:30.679 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:30 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:52:30.679 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:30 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:52:30.679 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:30 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:52:31.073 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:52:31.104 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:52:31.138 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:52:31.166 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:52:31.195 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:52:31.223 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:52:31.262 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:52:31.293 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:52:31.339 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:52:31.376 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:52:31.383 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:52:31.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:31.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:52:31.419 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-09T17:52:31.424 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:52:31.469 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:52:31.472 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:52:31.472 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:52:31.472 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:52:31.472 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:52:31.472 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:52:31.472 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:52:31.472 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:52:31.472 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:52:31.472 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:52:31.472 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:52:31.472 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:52:31.473 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:52:31.495 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:52:31.499 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:31 smithi099 ceph-mon[39420]: osdmap e715: 12 total, 12 up, 12 in 2024-02-09T17:52:31.500 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:31 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:52:31.500 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:31 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:52:31.500 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:31 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:52:31.500 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:31 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:52:31.500 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:31 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1379549892' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:52:31.526 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:31 smithi162 ceph-mon[44304]: osdmap e715: 12 total, 12 up, 12 in 2024-02-09T17:52:31.526 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:31 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:52:31.526 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:31 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:52:31.526 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:31 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:52:31.526 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:31 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:52:31.526 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:31 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1379549892' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:52:31.527 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:31.547 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:52:31.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:31.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:52:31.651 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:52:31.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:31 smithi179 ceph-mon[43975]: osdmap e715: 12 total, 12 up, 12 in 2024-02-09T17:52:31.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:31 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:52:31.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:31 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:52:31.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:31 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:52:31.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:31 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:52:31.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:31 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1379549892' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:52:31.679 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:31.700 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:52:31.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:31.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:52:31.804 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:52:31.831 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:31.851 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:52:31.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:31.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:52:31.957 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:52:31.990 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:32.010 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:52:32.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:32.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:52:32.096 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:52:32.114 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:52:32.152 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:32.501 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:32 smithi099 ceph-mon[39420]: pgmap v1788: 47 pgs: 1 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 39 active+clean; 5.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 542 op/s; 1/262419 objects degraded (0.000%); 11586/262419 objects misplaced (4.415%); 30 MiB/s, 485 objects/s recovering 2024-02-09T17:52:32.501 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:32 smithi099 ceph-mon[39420]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-09T17:52:32.501 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:32 smithi099 ceph-mon[39420]: osdmap e716: 12 total, 12 up, 12 in 2024-02-09T17:52:32.501 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:32 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 1/262419 objects degraded (0.000%), 1 pg degraded, 7 pgs undersized (PG_DEGRADED) 2024-02-09T17:52:32.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:32 smithi179 ceph-mon[43975]: pgmap v1788: 47 pgs: 1 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 39 active+clean; 5.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 542 op/s; 1/262419 objects degraded (0.000%); 11586/262419 objects misplaced (4.415%); 30 MiB/s, 485 objects/s recovering 2024-02-09T17:52:32.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:32 smithi179 ceph-mon[43975]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-09T17:52:32.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:32 smithi179 ceph-mon[43975]: osdmap e716: 12 total, 12 up, 12 in 2024-02-09T17:52:32.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:32 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 1/262419 objects degraded (0.000%), 1 pg degraded, 7 pgs undersized (PG_DEGRADED) 2024-02-09T17:52:32.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:32 smithi162 ceph-mon[44304]: pgmap v1788: 47 pgs: 1 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 39 active+clean; 5.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 542 op/s; 1/262419 objects degraded (0.000%); 11586/262419 objects misplaced (4.415%); 30 MiB/s, 485 objects/s recovering 2024-02-09T17:52:32.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:32 smithi162 ceph-mon[44304]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-09T17:52:32.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:32 smithi162 ceph-mon[44304]: osdmap e716: 12 total, 12 up, 12 in 2024-02-09T17:52:32.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:32 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 1/262419 objects degraded (0.000%), 1 pg degraded, 7 pgs undersized (PG_DEGRADED) 2024-02-09T17:52:32.856 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:52:33.407 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:52:33.411 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:33 smithi099 ceph-mon[39420]: osdmap e717: 12 total, 12 up, 12 in 2024-02-09T17:52:33.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:33.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:52:33.513 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:52:33.545 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:33.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:33 smithi179 ceph-mon[43975]: osdmap e717: 12 total, 12 up, 12 in 2024-02-09T17:52:33.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:33 smithi162 ceph-mon[44304]: osdmap e717: 12 total, 12 up, 12 in 2024-02-09T17:52:33.748 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:52:33.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:33.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:52:33.852 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:52:33.891 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:33.916 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:52:33.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:33.921 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:52:34.022 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:52:34.056 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:34.083 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:52:34.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:34.088 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:52:34.190 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:52:34.244 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:34.280 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:52:34.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:34.289 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:52:34.390 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:52:34.417 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:34.438 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:52:34.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:34.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:52:34.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:34 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:52:34.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:34 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:52:34.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:34 smithi099 ceph-mon[39420]: pgmap v1791: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 remapped+peering, 6 active+recovering+undersized+remapped, 39 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 739 op/s; 1/256413 objects degraded (0.000%); 10663/256413 objects misplaced (4.159%); 33 MiB/s, 523 objects/s recovering 2024-02-09T17:52:34.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:34 smithi099 ceph-mon[39420]: osdmap e718: 12 total, 12 up, 12 in 2024-02-09T17:52:34.541 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:52:34.574 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:34 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:52:34.574 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:34 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:52:34.574 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:34 smithi162 ceph-mon[44304]: pgmap v1791: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 remapped+peering, 6 active+recovering+undersized+remapped, 39 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 739 op/s; 1/256413 objects degraded (0.000%); 10663/256413 objects misplaced (4.159%); 33 MiB/s, 523 objects/s recovering 2024-02-09T17:52:34.574 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:34 smithi162 ceph-mon[44304]: osdmap e718: 12 total, 12 up, 12 in 2024-02-09T17:52:34.576 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:34.597 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:52:34.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:34.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:52:34.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:34 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:52:34.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:34 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:52:34.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:34 smithi179 ceph-mon[43975]: pgmap v1791: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 remapped+peering, 6 active+recovering+undersized+remapped, 39 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 739 op/s; 1/256413 objects degraded (0.000%); 10663/256413 objects misplaced (4.159%); 33 MiB/s, 523 objects/s recovering 2024-02-09T17:52:34.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:34 smithi179 ceph-mon[43975]: osdmap e718: 12 total, 12 up, 12 in 2024-02-09T17:52:34.701 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:52:34.768 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:34.824 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:52:34.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:34.829 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:52:34.898 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:52:34.930 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:52:34.997 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:35.052 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:52:35.056 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:35.057 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:52:35.158 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:52:35.233 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:35.270 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:52:35.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:35.275 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:52:35.326 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:35 smithi179 ceph-mon[43975]: 7.e scrub starts 2024-02-09T17:52:35.326 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:35 smithi179 ceph-mon[43975]: osdmap e719: 12 total, 12 up, 12 in 2024-02-09T17:52:35.377 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:52:35.409 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:35.430 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:52:35.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:35.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:52:35.535 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:52:35.566 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:35.587 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:52:35.589 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:35 smithi162 ceph-mon[44304]: 7.e scrub starts 2024-02-09T17:52:35.589 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:35 smithi162 ceph-mon[44304]: osdmap e719: 12 total, 12 up, 12 in 2024-02-09T17:52:35.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:35.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:52:35.691 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:52:35.739 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:35.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:35 smithi099 ceph-mon[39420]: 7.e scrub starts 2024-02-09T17:52:35.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:35 smithi099 ceph-mon[39420]: osdmap e719: 12 total, 12 up, 12 in 2024-02-09T17:52:35.785 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:52:35.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:35.791 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:52:35.892 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:52:35.924 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:36.057 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:52:36.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:36.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:52:36.165 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:52:36.235 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:36.294 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:52:36.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:36.303 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:52:36.404 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:52:36.433 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:36.454 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:52:36.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:36.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:52:36.562 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:52:36.589 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:36 smithi162 ceph-mon[44304]: 7.e scrub ok 2024-02-09T17:52:36.589 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:36 smithi162 ceph-mon[44304]: pgmap v1794: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 remapped+peering, 6 active+recovering+undersized+remapped, 39 active+clean; 5.2 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 553 op/s; 1/255780 objects degraded (0.000%); 10413/255780 objects misplaced (4.071%); 18 MiB/s, 293 objects/s recovering 2024-02-09T17:52:36.591 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:36.612 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:52:36.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:36.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:52:36.664 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:36 smithi099 ceph-mon[39420]: 7.e scrub ok 2024-02-09T17:52:36.664 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:36 smithi099 ceph-mon[39420]: pgmap v1794: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 remapped+peering, 6 active+recovering+undersized+remapped, 39 active+clean; 5.2 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 553 op/s; 1/255780 objects degraded (0.000%); 10413/255780 objects misplaced (4.071%); 18 MiB/s, 293 objects/s recovering 2024-02-09T17:52:36.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:36 smithi179 ceph-mon[43975]: 7.e scrub ok 2024-02-09T17:52:36.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:36 smithi179 ceph-mon[43975]: pgmap v1794: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 remapped+peering, 6 active+recovering+undersized+remapped, 39 active+clean; 5.2 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 553 op/s; 1/255780 objects degraded (0.000%); 10413/255780 objects misplaced (4.071%); 18 MiB/s, 293 objects/s recovering 2024-02-09T17:52:36.716 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:52:36.757 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:36.800 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:52:36.803 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:36.804 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:52:36.849 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] 2024-02-09T17:52:36.849 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 1 1 2024-02-09T17:52:36.905 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:52:36.934 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:36.955 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:52:36.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:36.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:52:37.060 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:52:37.089 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:37.122 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:52:37.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:37.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:52:37.227 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:52:37.260 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:37.288 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:52:37.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:37.294 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:52:37.394 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:52:37.426 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:37.459 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:52:37.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:37.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:52:37.464 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:37 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 1/255780 objects degraded (0.000%), 1 pg degraded, 7 pgs undersized (PG_DEGRADED) 2024-02-09T17:52:37.465 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:37 smithi099 ceph-mon[39420]: 1.1e scrub starts 2024-02-09T17:52:37.563 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:52:37.593 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:37 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 1/255780 objects degraded (0.000%), 1 pg degraded, 7 pgs undersized (PG_DEGRADED) 2024-02-09T17:52:37.593 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:37 smithi162 ceph-mon[44304]: 1.1e scrub starts 2024-02-09T17:52:37.594 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:37.617 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:52:37.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:37.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:52:37.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:37 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 1/255780 objects degraded (0.000%), 1 pg degraded, 7 pgs undersized (PG_DEGRADED) 2024-02-09T17:52:37.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:37 smithi179 ceph-mon[43975]: 1.1e scrub starts 2024-02-09T17:52:37.723 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:52:37.751 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:37.776 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:52:37.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:37.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:52:37.884 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:52:37.920 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:38.047 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:52:38.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:38.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:52:38.162 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:52:38.192 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:38.330 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:38 smithi099 ceph-mon[39420]: 1.14 scrub starts 2024-02-09T17:52:38.330 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:38 smithi099 ceph-mon[39420]: 1.14 scrub ok 2024-02-09T17:52:38.330 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:38 smithi099 ceph-mon[39420]: 7.7 deep-scrub starts 2024-02-09T17:52:38.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:38 smithi179 ceph-mon[43975]: 1.14 scrub starts 2024-02-09T17:52:38.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:38 smithi179 ceph-mon[43975]: 1.14 scrub ok 2024-02-09T17:52:38.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:38 smithi179 ceph-mon[43975]: 7.7 deep-scrub starts 2024-02-09T17:52:38.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:38 smithi179 ceph-mon[43975]: pgmap v1795: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 643 op/s; 1/250011 objects degraded (0.000%); 8968/250011 objects misplaced (3.587%); 23 MiB/s, 375 objects/s recovering 2024-02-09T17:52:38.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:38 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:52:38.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:38 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:52:38.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:38 smithi179 ceph-mon[43975]: 1.1e scrub ok 2024-02-09T17:52:38.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:38 smithi179 ceph-mon[43975]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-09T17:52:38.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:38 smithi179 ceph-mon[43975]: osdmap e720: 12 total, 12 up, 12 in 2024-02-09T17:52:38.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:38 smithi162 ceph-mon[44304]: 1.14 scrub starts 2024-02-09T17:52:38.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:38 smithi162 ceph-mon[44304]: 1.14 scrub ok 2024-02-09T17:52:38.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:38 smithi162 ceph-mon[44304]: 7.7 deep-scrub starts 2024-02-09T17:52:38.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:38 smithi162 ceph-mon[44304]: pgmap v1795: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 643 op/s; 1/250011 objects degraded (0.000%); 8968/250011 objects misplaced (3.587%); 23 MiB/s, 375 objects/s recovering 2024-02-09T17:52:38.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:38 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:52:38.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:38 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:52:38.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:38 smithi162 ceph-mon[44304]: 1.1e scrub ok 2024-02-09T17:52:38.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:38 smithi162 ceph-mon[44304]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-09T17:52:38.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:38 smithi162 ceph-mon[44304]: osdmap e720: 12 total, 12 up, 12 in 2024-02-09T17:52:38.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:38 smithi099 ceph-mon[39420]: pgmap v1795: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 643 op/s; 1/250011 objects degraded (0.000%); 8968/250011 objects misplaced (3.587%); 23 MiB/s, 375 objects/s recovering 2024-02-09T17:52:38.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:38 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:52:38.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:38 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:52:38.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:38 smithi099 ceph-mon[39420]: 1.1e scrub ok 2024-02-09T17:52:38.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:38 smithi099 ceph-mon[39420]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-09T17:52:38.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:38 smithi099 ceph-mon[39420]: osdmap e720: 12 total, 12 up, 12 in 2024-02-09T17:52:38.747 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:52:38.761 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:52:38.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:38.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:52:38.822 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:52:38.822 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:52:38.868 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:52:38.899 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:38.925 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:52:38.931 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:38.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:52:39.033 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:52:39.065 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:39.084 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:52:39.357 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is set 2024-02-09T17:52:39.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:39 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1695082893' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:52:39.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:39 smithi179 ceph-mon[43975]: 1.8 scrub starts 2024-02-09T17:52:39.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:39 smithi179 ceph-mon[43975]: 1.8 scrub ok 2024-02-09T17:52:39.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:39 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1695082893' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:52:39.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:39 smithi162 ceph-mon[44304]: 1.8 scrub starts 2024-02-09T17:52:39.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:39 smithi162 ceph-mon[44304]: 1.8 scrub ok 2024-02-09T17:52:39.685 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:39 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1695082893' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:52:39.685 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:39 smithi099 ceph-mon[39420]: 1.8 scrub starts 2024-02-09T17:52:39.685 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:39 smithi099 ceph-mon[39420]: 1.8 scrub ok 2024-02-09T17:52:40.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:40 smithi179 ceph-mon[43975]: pgmap v1797: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 469 op/s; 1/247962 objects degraded (0.000%); 8464/247962 objects misplaced (3.413%); 23 MiB/s, 366 objects/s recovering 2024-02-09T17:52:40.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:40 smithi179 ceph-mon[43975]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:52:40.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:40 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1695082893' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:52:40.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:40 smithi179 ceph-mon[43975]: osdmap e721: 12 total, 12 up, 12 in 2024-02-09T17:52:40.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:40 smithi162 ceph-mon[44304]: pgmap v1797: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 469 op/s; 1/247962 objects degraded (0.000%); 8464/247962 objects misplaced (3.413%); 23 MiB/s, 366 objects/s recovering 2024-02-09T17:52:40.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:40 smithi162 ceph-mon[44304]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:52:40.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:40 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1695082893' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:52:40.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:40 smithi162 ceph-mon[44304]: osdmap e721: 12 total, 12 up, 12 in 2024-02-09T17:52:40.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:40 smithi099 ceph-mon[39420]: pgmap v1797: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 42 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 469 op/s; 1/247962 objects degraded (0.000%); 8464/247962 objects misplaced (3.413%); 23 MiB/s, 366 objects/s recovering 2024-02-09T17:52:40.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:40 smithi099 ceph-mon[39420]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:52:40.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:40 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1695082893' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:52:40.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:40 smithi099 ceph-mon[39420]: osdmap e721: 12 total, 12 up, 12 in 2024-02-09T17:52:41.048 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:52:41.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:41.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:52:41.161 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:52:41.189 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:41.214 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:52:41.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:41.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:52:41.321 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:52:41.353 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:41.540 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:52:41.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:41.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:52:41.581 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-09T17:52:41.645 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:52:41.673 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:41.694 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:52:41.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:41.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:52:41.798 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:52:41.827 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:41.851 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:52:41.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:41.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:52:41.958 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:52:41.991 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:42.017 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:52:42.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:42.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:52:42.121 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:52:42.181 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:42.227 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:52:42.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:42.231 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:52:42.332 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:52:42.379 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:42.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:42 smithi099 ceph-mon[39420]: pgmap v1799: 47 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 41 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 526 op/s; 1/245073 objects degraded (0.000%); 8464/245073 objects misplaced (3.454%); 18 MiB/s, 287 objects/s recovering 2024-02-09T17:52:42.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:42 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 1/245073 objects degraded (0.000%), 1 pg degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-09T17:52:42.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:42 smithi179 ceph-mon[43975]: pgmap v1799: 47 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 41 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 526 op/s; 1/245073 objects degraded (0.000%); 8464/245073 objects misplaced (3.454%); 18 MiB/s, 287 objects/s recovering 2024-02-09T17:52:42.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:42 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 1/245073 objects degraded (0.000%), 1 pg degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-09T17:52:42.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:42 smithi162 ceph-mon[44304]: pgmap v1799: 47 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 41 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 526 op/s; 1/245073 objects degraded (0.000%); 8464/245073 objects misplaced (3.454%); 18 MiB/s, 287 objects/s recovering 2024-02-09T17:52:42.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:42 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 1/245073 objects degraded (0.000%), 1 pg degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-09T17:52:42.947 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:52:42.977 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:52:43.009 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:52:43.042 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:52:43.064 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:52:43.100 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:52:43.123 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:52:43.159 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:52:43.192 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:52:43.231 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:52:43.263 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:52:43.297 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:52:43.299 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:52:43.299 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:52:43.299 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:52:43.299 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:52:43.299 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:52:43.299 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:52:43.299 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:52:43.299 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:52:43.299 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:52:43.299 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:52:43.299 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:52:43.299 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:52:43.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:43 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3582333148' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:52:43.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:43 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3582333148' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:52:43.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:43 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3582333148' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:52:44.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:44 smithi179 ceph-mon[43975]: pgmap v1800: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 4 active+recovering+undersized+remapped, 41 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 683 op/s; 1/239379 objects degraded (0.000%); 7506/239379 objects misplaced (3.136%); 23 MiB/s, 363 objects/s recovering 2024-02-09T17:52:44.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:44 smithi162 ceph-mon[44304]: pgmap v1800: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 4 active+recovering+undersized+remapped, 41 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 683 op/s; 1/239379 objects degraded (0.000%); 7506/239379 objects misplaced (3.136%); 23 MiB/s, 363 objects/s recovering 2024-02-09T17:52:44.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:44 smithi099 ceph-mon[39420]: pgmap v1800: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 4 active+recovering+undersized+remapped, 41 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 683 op/s; 1/239379 objects degraded (0.000%); 7506/239379 objects misplaced (3.136%); 23 MiB/s, 363 objects/s recovering 2024-02-09T17:52:44.923 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:52:44.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:44.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:52:45.033 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:52:45.062 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:45.090 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:52:45.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:45.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:52:45.194 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:52:45.233 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:45.295 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:52:45.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:45.302 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:52:45.403 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:52:45.450 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:46.398 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:52:46.411 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:52:46.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:46.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:52:46.520 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:52:46.551 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:46.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:46 smithi179 ceph-mon[43975]: pgmap v1801: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 4 active+recovering+undersized+remapped, 41 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 463 op/s; 1/238881 objects degraded (0.000%); 7199/238881 objects misplaced (3.014%); 14 MiB/s, 221 objects/s recovering 2024-02-09T17:52:46.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:46 smithi162 ceph-mon[44304]: pgmap v1801: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 4 active+recovering+undersized+remapped, 41 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 463 op/s; 1/238881 objects degraded (0.000%); 7199/238881 objects misplaced (3.014%); 14 MiB/s, 221 objects/s recovering 2024-02-09T17:52:46.688 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:52:46.692 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:46 smithi099 ceph-mon[39420]: pgmap v1801: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 4 active+recovering+undersized+remapped, 41 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 463 op/s; 1/238881 objects degraded (0.000%); 7199/238881 objects misplaced (3.014%); 14 MiB/s, 221 objects/s recovering 2024-02-09T17:52:46.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:46.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:52:46.795 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:52:46.840 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:46.866 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:52:46.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:46.871 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:52:46.957 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-09T17:52:46.972 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:52:47.057 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:47.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:47 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 1/238881 objects degraded (0.000%), 1 pg degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-09T17:52:47.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:47 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 1/238881 objects degraded (0.000%), 1 pg degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-09T17:52:47.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:47 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 1/238881 objects degraded (0.000%), 1 pg degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-09T17:52:48.390 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.1 to 1 (10000) 2024-02-09T17:52:48.442 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:52:48.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:48 smithi179 ceph-mon[43975]: pgmap v1802: 47 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 41 active+clean; 4.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 578 op/s; 1/232359 objects degraded (0.000%); 6170/232359 objects misplaced (2.655%); 18 MiB/s, 280 objects/s recovering 2024-02-09T17:52:48.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:48 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/450600735' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-09T17:52:48.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:48 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-09T17:52:48.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:48 smithi162 ceph-mon[44304]: pgmap v1802: 47 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 41 active+clean; 4.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 578 op/s; 1/232359 objects degraded (0.000%); 6170/232359 objects misplaced (2.655%); 18 MiB/s, 280 objects/s recovering 2024-02-09T17:52:48.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:48 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/450600735' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-09T17:52:48.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:48 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-09T17:52:48.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:48 smithi099 ceph-mon[39420]: pgmap v1802: 47 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 41 active+clean; 4.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 578 op/s; 1/232359 objects degraded (0.000%); 6170/232359 objects misplaced (2.655%); 18 MiB/s, 280 objects/s recovering 2024-02-09T17:52:48.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:48 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/450600735' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-09T17:52:48.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:48 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-09T17:52:49.539 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:52:49.543 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:49 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-09T17:52:49.543 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:49 smithi099 ceph-mon[39420]: osdmap e722: 12 total, 12 up, 12 in 2024-02-09T17:52:49.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:49.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:52:49.645 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:52:49.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:49 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-09T17:52:49.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:49 smithi179 ceph-mon[43975]: osdmap e722: 12 total, 12 up, 12 in 2024-02-09T17:52:49.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:49 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-09T17:52:49.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:49 smithi162 ceph-mon[44304]: osdmap e722: 12 total, 12 up, 12 in 2024-02-09T17:52:49.703 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:49.732 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:52:49.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:49.735 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:52:49.836 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:52:49.865 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:49.892 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:52:49.895 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:49.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:52:49.997 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:52:50.039 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:50.066 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:52:50.073 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:50.073 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:52:50.174 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:52:50.203 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:50.223 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:52:50.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:50.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:52:50.327 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:52:50.403 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:50.456 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:52:50.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:50.462 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:52:50.563 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:52:50.589 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:50 smithi162 ceph-mon[44304]: pgmap v1804: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 4 active+recovering+undersized+remapped, 41 active+clean; 4.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 605 op/s; 1/230295 objects degraded (0.000%); 5813/230295 objects misplaced (2.524%); 17 MiB/s, 272 objects/s recovering 2024-02-09T17:52:50.591 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:50.614 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:52:50.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:50.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:52:50.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:50 smithi179 ceph-mon[43975]: pgmap v1804: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 4 active+recovering+undersized+remapped, 41 active+clean; 4.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 605 op/s; 1/230295 objects degraded (0.000%); 5813/230295 objects misplaced (2.524%); 17 MiB/s, 272 objects/s recovering 2024-02-09T17:52:50.717 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:52:50.748 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:50 smithi099 ceph-mon[39420]: pgmap v1804: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 4 active+recovering+undersized+remapped, 41 active+clean; 4.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 605 op/s; 1/230295 objects degraded (0.000%); 5813/230295 objects misplaced (2.524%); 17 MiB/s, 272 objects/s recovering 2024-02-09T17:52:50.757 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:50.797 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 3 1 2024-02-09T17:52:50.798 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:52:50.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:50.801 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:52:50.851 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:52:50.851 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 20, 2024-02-09T17:52:50.851 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 600, 2024-02-09T17:52:50.851 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [ 2024-02-09T17:52:50.851 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.851 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25528.0:205155 5.5 5:a6c98e0c:::benchmark_data_smithi179_115199_object76249:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e545)", 2024-02-09T17:52:50.851 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:44:12.654941+0000", 2024-02-09T17:52:50.852 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 517.95455974200001, 2024-02-09T17:52:50.852 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.23019305900000001, 2024-02-09T17:52:50.852 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:52:50.852 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:52:50.852 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:52:50.852 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25528", 2024-02-09T17:52:50.852 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:52:50.852 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 205155 2024-02-09T17:52:50.852 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.852 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:52:50.852 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.852 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:52:50.852 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.654941+0000", 2024-02-09T17:52:50.853 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:52:50.853 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.853 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.853 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:52:50.853 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.654941+0000", 2024-02-09T17:52:50.853 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:52:50.853 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.853 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.853 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:52:50.853 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.654943+0000", 2024-02-09T17:52:50.853 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2510000000000001e-06 2024-02-09T17:52:50.853 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.853 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.853 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:52:50.853 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.654951+0000", 2024-02-09T17:52:50.854 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.2389999999999996e-06 2024-02-09T17:52:50.854 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.854 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.854 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:52:50.854 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.654964+0000", 2024-02-09T17:52:50.854 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3354e-05 2024-02-09T17:52:50.854 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.854 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.854 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:52:50.854 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.654975+0000", 2024-02-09T17:52:50.854 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.101e-05 2024-02-09T17:52:50.854 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.854 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.854 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:52:50.854 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.655021+0000", 2024-02-09T17:52:50.855 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.6204e-05 2024-02-09T17:52:50.855 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.855 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.855 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:52:50.855 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.655144+0000", 2024-02-09T17:52:50.855 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012307400000000001 2024-02-09T17:52:50.855 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.855 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.855 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:52:50.855 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.655244+0000", 2024-02-09T17:52:50.855 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000100178 2024-02-09T17:52:50.855 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.855 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.855 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:52:50.855 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.660324+0000", 2024-02-09T17:52:50.856 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.005079259 2024-02-09T17:52:50.856 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.856 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.856 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:50.856 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.676421+0000", 2024-02-09T17:52:50.856 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.016096946000000001 2024-02-09T17:52:50.856 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.856 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.856 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:50.856 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.885044+0000", 2024-02-09T17:52:50.856 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20862339599999999 2024-02-09T17:52:50.856 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.856 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.856 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:52:50.856 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.885088+0000", 2024-02-09T17:52:50.857 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.4288e-05 2024-02-09T17:52:50.857 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.857 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.857 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:52:50.857 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.885134+0000", 2024-02-09T17:52:50.857 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.5859999999999998e-05 2024-02-09T17:52:50.857 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:50.857 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:50.857 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:50.857 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.857 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.857 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25528.0:205160 5.5 5:a68b4e92:::benchmark_data_smithi179_115199_object76254:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e545)", 2024-02-09T17:52:50.857 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:44:12.660914+0000", 2024-02-09T17:52:50.857 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 517.94858738999994, 2024-02-09T17:52:50.857 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.23356561100000001, 2024-02-09T17:52:50.858 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:52:50.858 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:52:50.858 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:52:50.858 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25528", 2024-02-09T17:52:50.858 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:52:50.858 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 205160 2024-02-09T17:52:50.858 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.858 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:52:50.858 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.858 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:52:50.858 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.660914+0000", 2024-02-09T17:52:50.858 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:52:50.858 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.858 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.858 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:52:50.859 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.660914+0000", 2024-02-09T17:52:50.859 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:52:50.859 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.859 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.859 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:52:50.859 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.660915+0000", 2024-02-09T17:52:50.859 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5629999999999999e-06 2024-02-09T17:52:50.859 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.859 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.859 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:52:50.859 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.660920+0000", 2024-02-09T17:52:50.859 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.6990000000000002e-06 2024-02-09T17:52:50.859 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.859 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.859 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:52:50.860 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.660931+0000", 2024-02-09T17:52:50.860 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1392999999999999e-05 2024-02-09T17:52:50.860 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.860 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.860 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:52:50.860 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.660940+0000", 2024-02-09T17:52:50.860 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.3060000000000003e-06 2024-02-09T17:52:50.860 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.860 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.860 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:52:50.860 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.661001+0000", 2024-02-09T17:52:50.860 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.0114000000000001e-05 2024-02-09T17:52:50.860 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.860 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.860 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:52:50.861 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.661118+0000", 2024-02-09T17:52:50.861 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000117557 2024-02-09T17:52:50.861 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.861 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.861 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:52:50.861 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.661238+0000", 2024-02-09T17:52:50.861 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000119533 2024-02-09T17:52:50.861 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.861 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.861 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:52:50.861 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.665353+0000", 2024-02-09T17:52:50.861 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0041152209999999996 2024-02-09T17:52:50.861 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.861 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.861 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:50.861 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.676453+0000", 2024-02-09T17:52:50.862 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.011100303000000001 2024-02-09T17:52:50.862 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.862 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.862 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:50.862 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.894370+0000", 2024-02-09T17:52:50.862 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.21791716899999999 2024-02-09T17:52:50.862 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.862 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.862 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:52:50.862 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.894433+0000", 2024-02-09T17:52:50.862 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.2411999999999998e-05 2024-02-09T17:52:50.862 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.862 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.862 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:52:50.862 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.894479+0000", 2024-02-09T17:52:50.863 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.6341000000000002e-05 2024-02-09T17:52:50.863 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:50.863 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:50.863 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:50.863 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.863 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.863 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25528.0:205183 5.5 5:a26b1f5f:::benchmark_data_smithi179_115199_object76277:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e545)", 2024-02-09T17:52:50.863 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:44:12.695666+0000", 2024-02-09T17:52:50.863 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 517.91383507, 2024-02-09T17:52:50.863 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.198935274, 2024-02-09T17:52:50.863 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:52:50.863 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:52:50.863 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:52:50.863 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25528", 2024-02-09T17:52:50.863 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:52:50.864 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 205183 2024-02-09T17:52:50.864 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.864 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:52:50.864 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.864 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:52:50.864 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.695666+0000", 2024-02-09T17:52:50.864 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:52:50.864 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.864 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.864 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:52:50.864 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.695666+0000", 2024-02-09T17:52:50.864 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:52:50.864 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.865 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.865 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:52:50.865 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.695668+0000", 2024-02-09T17:52:50.865 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2670000000000001e-06 2024-02-09T17:52:50.865 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.865 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.865 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:52:50.865 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.695675+0000", 2024-02-09T17:52:50.865 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.0450000000000003e-06 2024-02-09T17:52:50.865 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.865 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.865 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:52:50.865 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.695688+0000", 2024-02-09T17:52:50.865 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2724999999999999e-05 2024-02-09T17:52:50.866 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.866 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.866 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:52:50.866 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.695696+0000", 2024-02-09T17:52:50.866 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.4929999999999998e-06 2024-02-09T17:52:50.866 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.866 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.866 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:52:50.866 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.695745+0000", 2024-02-09T17:52:50.866 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.8887000000000001e-05 2024-02-09T17:52:50.866 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.866 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.866 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:52:50.866 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.695844+0000", 2024-02-09T17:52:50.866 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.8979000000000002e-05 2024-02-09T17:52:50.867 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.867 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.867 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:52:50.867 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.695929+0000", 2024-02-09T17:52:50.867 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.4920999999999996e-05 2024-02-09T17:52:50.867 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.867 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.867 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:52:50.867 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.700078+0000", 2024-02-09T17:52:50.867 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0041486120000000003 2024-02-09T17:52:50.867 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.867 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.867 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:50.867 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.702002+0000", 2024-02-09T17:52:50.867 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.001924523 2024-02-09T17:52:50.868 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.868 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.868 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:50.868 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.894525+0000", 2024-02-09T17:52:50.868 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.192522793 2024-02-09T17:52:50.868 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.868 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.868 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:52:50.868 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.894569+0000", 2024-02-09T17:52:50.868 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.4197999999999999e-05 2024-02-09T17:52:50.868 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.868 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.868 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:52:50.868 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.894601+0000", 2024-02-09T17:52:50.868 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.1831000000000001e-05 2024-02-09T17:52:50.869 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:50.869 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:50.869 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:50.869 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.869 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.869 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25528.0:205191 5.5 5:ae0e8ee0:::benchmark_data_smithi179_115199_object76285:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e545)", 2024-02-09T17:52:50.869 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:44:12.702285+0000", 2024-02-09T17:52:50.869 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 517.90721548900001, 2024-02-09T17:52:50.869 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.19246898100000001, 2024-02-09T17:52:50.869 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:52:50.869 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:52:50.869 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:52:50.869 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25528", 2024-02-09T17:52:50.869 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:52:50.869 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 205191 2024-02-09T17:52:50.870 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.870 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:52:50.870 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.870 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:52:50.870 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.702285+0000", 2024-02-09T17:52:50.870 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:52:50.870 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.870 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.870 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:52:50.870 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.702285+0000", 2024-02-09T17:52:50.870 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:52:50.870 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.870 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.870 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:52:50.870 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.702287+0000", 2024-02-09T17:52:50.871 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.325e-06 2024-02-09T17:52:50.871 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.871 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.871 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:52:50.871 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.702292+0000", 2024-02-09T17:52:50.871 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.643e-06 2024-02-09T17:52:50.871 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.871 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.871 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:52:50.871 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.702306+0000", 2024-02-09T17:52:50.871 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3702e-05 2024-02-09T17:52:50.871 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.871 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.871 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:52:50.872 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.702314+0000", 2024-02-09T17:52:50.872 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.8599999999999993e-06 2024-02-09T17:52:50.872 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.872 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.872 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:52:50.872 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.702359+0000", 2024-02-09T17:52:50.872 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.5389999999999997e-05 2024-02-09T17:52:50.872 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.872 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.872 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:52:50.872 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.702461+0000", 2024-02-09T17:52:50.872 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000101847 2024-02-09T17:52:50.872 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.872 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.872 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:52:50.873 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.702544+0000", 2024-02-09T17:52:50.873 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.2647999999999999e-05 2024-02-09T17:52:50.873 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.873 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.873 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:52:50.873 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.706431+0000", 2024-02-09T17:52:50.873 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0038872870000000001 2024-02-09T17:52:50.873 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.873 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.873 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:50.873 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.712657+0000", 2024-02-09T17:52:50.873 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0062254249999999997 2024-02-09T17:52:50.873 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.873 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.873 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:50.874 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.894650+0000", 2024-02-09T17:52:50.874 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.181993129 2024-02-09T17:52:50.874 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.874 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.874 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:52:50.874 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.894717+0000", 2024-02-09T17:52:50.874 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.7626999999999995e-05 2024-02-09T17:52:50.874 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.874 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.874 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:52:50.874 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.894754+0000", 2024-02-09T17:52:50.874 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.7098000000000003e-05 2024-02-09T17:52:50.874 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:50.874 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:50.874 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:50.874 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.875 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.875 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25528.0:205222 5.5 5:a39a8093:::benchmark_data_smithi179_115199_object76316:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e545)", 2024-02-09T17:52:50.875 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:44:12.742493+0000", 2024-02-09T17:52:50.875 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 517.86700748500004, 2024-02-09T17:52:50.875 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.17220997699999999, 2024-02-09T17:52:50.875 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:52:50.875 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:52:50.875 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:52:50.875 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25528", 2024-02-09T17:52:50.875 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2603788052", 2024-02-09T17:52:50.875 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 205222 2024-02-09T17:52:50.875 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.875 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:52:50.875 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.875 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:52:50.875 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.742493+0000", 2024-02-09T17:52:50.876 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:52:50.876 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.876 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.876 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:52:50.876 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.742493+0000", 2024-02-09T17:52:50.876 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:52:50.876 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.876 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.876 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:52:50.876 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.742494+0000", 2024-02-09T17:52:50.876 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.850000000000001e-07 2024-02-09T17:52:50.876 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.876 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.876 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:52:50.877 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.742501+0000", 2024-02-09T17:52:50.877 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.6810000000000001e-06 2024-02-09T17:52:50.877 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.877 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.877 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:52:50.877 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.742510+0000", 2024-02-09T17:52:50.877 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.1859999999999998e-06 2024-02-09T17:52:50.877 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.877 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.877 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:52:50.877 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.742516+0000", 2024-02-09T17:52:50.877 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.1840000000000004e-06 2024-02-09T17:52:50.877 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.877 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.878 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:52:50.878 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.745363+0000", 2024-02-09T17:52:50.878 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0028462990000000001 2024-02-09T17:52:50.878 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.878 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.878 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:52:50.878 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.745444+0000", 2024-02-09T17:52:50.878 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.1693000000000005e-05 2024-02-09T17:52:50.878 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.878 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.878 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:52:50.878 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.745517+0000", 2024-02-09T17:52:50.878 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.2238999999999997e-05 2024-02-09T17:52:50.878 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.878 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.878 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:52:50.879 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.748385+0000", 2024-02-09T17:52:50.879 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0028679740000000001 2024-02-09T17:52:50.879 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.879 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.879 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:50.879 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.748847+0000", 2024-02-09T17:52:50.879 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000462557 2024-02-09T17:52:50.879 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.879 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.879 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:50.879 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.914607+0000", 2024-02-09T17:52:50.879 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.165760034 2024-02-09T17:52:50.879 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.879 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.879 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:52:50.879 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.914662+0000", 2024-02-09T17:52:50.880 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.4709000000000001e-05 2024-02-09T17:52:50.880 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.880 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.880 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:52:50.880 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:44:12.914703+0000", 2024-02-09T17:52:50.880 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.1436000000000001e-05 2024-02-09T17:52:50.880 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:50.880 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:50.880 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:50.880 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.880 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.880 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25837.0:180603 6.a e630/600 6:57888cf7:::benchmark_data_smithi179_126206_object50222:head v 630'11416, mlcod=630'11414)", 2024-02-09T17:52:50.880 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:48:19.898551+0000", 2024-02-09T17:52:50.880 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 270.71094963299998, 2024-02-09T17:52:50.881 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.169202886, 2024-02-09T17:52:50.881 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:52:50.881 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:52:50.881 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:52:50.881 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.881 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:52:50.881 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:19.898551+0000", 2024-02-09T17:52:50.881 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:52:50.881 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.881 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.881 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:52:50.881 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:19.898551+0000", 2024-02-09T17:52:50.881 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:52:50.881 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.881 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.882 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:52:50.882 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:19.898552+0000", 2024-02-09T17:52:50.882 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.9899999999999999e-07 2024-02-09T17:52:50.882 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.882 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.882 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:52:50.882 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:19.898560+0000", 2024-02-09T17:52:50.882 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.3490000000000003e-06 2024-02-09T17:52:50.882 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.882 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.882 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:52:50.882 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:19.898574+0000", 2024-02-09T17:52:50.882 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4508000000000001e-05 2024-02-09T17:52:50.882 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.883 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.883 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:52:50.883 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:19.898582+0000", 2024-02-09T17:52:50.883 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.2519999999999999e-06 2024-02-09T17:52:50.883 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.883 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.883 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:52:50.883 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:20.055533+0000", 2024-02-09T17:52:50.883 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.15695081299999999 2024-02-09T17:52:50.883 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.883 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.883 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:52:50.883 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:20.055562+0000", 2024-02-09T17:52:50.883 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.9283000000000001e-05 2024-02-09T17:52:50.883 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.883 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.884 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:52:50.884 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:20.067711+0000", 2024-02-09T17:52:50.884 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.012148871 2024-02-09T17:52:50.884 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.884 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.884 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:52:50.884 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:20.067754+0000", 2024-02-09T17:52:50.884 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2911000000000003e-05 2024-02-09T17:52:50.884 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:50.884 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:50.884 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:50.884 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.884 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.884 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25837.0:180607 6.a e630/600 6:5c83d64d:::benchmark_data_smithi179_126206_object50226:head v 630'11417, mlcod=630'11414)", 2024-02-09T17:52:50.885 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:48:19.907823+0000", 2024-02-09T17:52:50.885 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 270.70167742400002, 2024-02-09T17:52:50.885 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.172944088, 2024-02-09T17:52:50.885 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:52:50.885 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:52:50.885 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:52:50.885 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.885 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:52:50.885 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:19.907823+0000", 2024-02-09T17:52:50.885 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:52:50.885 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.885 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.885 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:52:50.885 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:19.907823+0000", 2024-02-09T17:52:50.885 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:52:50.885 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.886 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.886 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:52:50.886 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:19.907825+0000", 2024-02-09T17:52:50.886 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.404e-06 2024-02-09T17:52:50.886 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.886 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.886 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:52:50.886 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:19.907837+0000", 2024-02-09T17:52:50.886 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1936000000000001e-05 2024-02-09T17:52:50.886 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.886 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.886 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:52:50.886 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:20.055528+0000", 2024-02-09T17:52:50.886 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.14769067999999999 2024-02-09T17:52:50.887 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.887 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.887 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:52:50.887 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:20.055538+0000", 2024-02-09T17:52:50.887 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0548e-05 2024-02-09T17:52:50.887 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.887 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.887 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:52:50.887 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:20.060269+0000", 2024-02-09T17:52:50.887 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.004731257 2024-02-09T17:52:50.887 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.887 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.887 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:52:50.887 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:20.060284+0000", 2024-02-09T17:52:50.887 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4518999999999999e-05 2024-02-09T17:52:50.888 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.888 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.888 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:52:50.888 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:20.080736+0000", 2024-02-09T17:52:50.888 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.020452436000000001 2024-02-09T17:52:50.888 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.888 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.888 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:52:50.888 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:20.080768+0000", 2024-02-09T17:52:50.888 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.1307999999999999e-05 2024-02-09T17:52:50.888 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:50.888 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:50.888 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:50.888 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.888 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.889 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25837.0:180601 6.0 e630/576 6:03594e9d:::benchmark_data_smithi179_126206_object50220:head v 630'11216, mlcod=630'11214)", 2024-02-09T17:52:50.889 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:48:19.913305+0000", 2024-02-09T17:52:50.889 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 270.69619623, 2024-02-09T17:52:50.889 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.16802792599999999, 2024-02-09T17:52:50.889 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:52:50.889 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:52:50.889 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:52:50.889 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.889 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:52:50.889 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:19.913305+0000", 2024-02-09T17:52:50.889 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:52:50.889 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.889 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.889 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:52:50.889 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:19.913305+0000", 2024-02-09T17:52:50.890 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:52:50.890 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.890 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.890 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:52:50.891 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:19.913306+0000", 2024-02-09T17:52:50.891 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.164e-06 2024-02-09T17:52:50.891 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.891 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.891 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:52:50.892 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:19.913316+0000", 2024-02-09T17:52:50.892 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.9049999999999992e-06 2024-02-09T17:52:50.892 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.892 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.892 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:52:50.892 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:20.055524+0000", 2024-02-09T17:52:50.892 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.14220781499999999 2024-02-09T17:52:50.892 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.892 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.892 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:52:50.892 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:20.055538+0000", 2024-02-09T17:52:50.892 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4231e-05 2024-02-09T17:52:50.892 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.892 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.892 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:52:50.893 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:20.055593+0000", 2024-02-09T17:52:50.893 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.4960000000000002e-05 2024-02-09T17:52:50.893 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.893 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.893 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:52:50.893 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:20.055616+0000", 2024-02-09T17:52:50.893 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.281e-05 2024-02-09T17:52:50.893 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.893 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.893 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:52:50.893 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:20.081299+0000", 2024-02-09T17:52:50.893 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.025683113 2024-02-09T17:52:50.893 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.893 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.893 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:52:50.894 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:20.081333+0000", 2024-02-09T17:52:50.894 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.3927999999999998e-05 2024-02-09T17:52:50.894 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:50.894 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:50.894 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:50.894 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.894 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.894 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.16440.0:60983 7.f e674/661 7:ff502c2e:::benchmark_data_smithi179_135804_object60982:head v 674'3853, mlcod=674'3850)", 2024-02-09T17:52:50.894 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:50:55.011600+0000", 2024-02-09T17:52:50.894 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 115.597901294, 2024-02-09T17:52:50.894 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.19042357100000001, 2024-02-09T17:52:50.894 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:52:50.894 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:52:50.894 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:52:50.894 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.894 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:52:50.895 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.011600+0000", 2024-02-09T17:52:50.895 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:52:50.895 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.895 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.895 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:52:50.895 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.011600+0000", 2024-02-09T17:52:50.895 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:52:50.895 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.895 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.895 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:52:50.895 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.011600+0000", 2024-02-09T17:52:50.895 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.13e-07 2024-02-09T17:52:50.895 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.895 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.896 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:52:50.896 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.011635+0000", 2024-02-09T17:52:50.896 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.4453999999999999e-05 2024-02-09T17:52:50.896 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.896 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.896 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:52:50.896 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.011650+0000", 2024-02-09T17:52:50.896 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.488e-05 2024-02-09T17:52:50.896 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.896 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.896 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:52:50.896 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.011656+0000", 2024-02-09T17:52:50.896 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.5649999999999998e-06 2024-02-09T17:52:50.896 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.896 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.897 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:52:50.897 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.011764+0000", 2024-02-09T17:52:50.897 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000107988 2024-02-09T17:52:50.897 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.897 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.897 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:52:50.897 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.011781+0000", 2024-02-09T17:52:50.897 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.6806000000000001e-05 2024-02-09T17:52:50.897 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.897 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.897 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:52:50.897 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.013991+0000", 2024-02-09T17:52:50.897 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0022102889999999998 2024-02-09T17:52:50.897 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.897 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.898 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:52:50.898 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.202023+0000", 2024-02-09T17:52:50.898 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.18803197599999999 2024-02-09T17:52:50.898 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:50.898 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:50.898 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:50.898 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.898 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.898 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.16440.0:60992 7.4 7:25bdd397:::benchmark_data_smithi179_135804_object60991:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e674)", 2024-02-09T17:52:50.898 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:50:55.012006+0000", 2024-02-09T17:52:50.898 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 115.597495387, 2024-02-09T17:52:50.898 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.19498057899999999, 2024-02-09T17:52:50.898 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:52:50.898 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:52:50.898 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:52:50.899 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.16440", 2024-02-09T17:52:50.899 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:52:50.899 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 60992 2024-02-09T17:52:50.899 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.899 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:52:50.899 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.899 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:52:50.899 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.012006+0000", 2024-02-09T17:52:50.899 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:52:50.899 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.899 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.899 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:52:50.899 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.012006+0000", 2024-02-09T17:52:50.899 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:52:50.900 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.900 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.900 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:52:50.900 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.012007+0000", 2024-02-09T17:52:50.900 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.319e-06 2024-02-09T17:52:50.900 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.900 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.900 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:52:50.900 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.012071+0000", 2024-02-09T17:52:50.900 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.4219000000000003e-05 2024-02-09T17:52:50.900 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.900 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.900 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:52:50.900 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.012081+0000", 2024-02-09T17:52:50.900 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.9019999999999994e-06 2024-02-09T17:52:50.901 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.901 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.901 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:52:50.901 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.012088+0000", 2024-02-09T17:52:50.901 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.1749999999999999e-06 2024-02-09T17:52:50.901 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.901 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.901 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:52:50.901 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.012139+0000", 2024-02-09T17:52:50.901 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.0408999999999998e-05 2024-02-09T17:52:50.901 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.901 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.901 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:52:50.901 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.012262+0000", 2024-02-09T17:52:50.901 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012394000000000001 2024-02-09T17:52:50.901 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.902 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.902 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:52:50.902 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.012402+0000", 2024-02-09T17:52:50.902 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013949500000000001 2024-02-09T17:52:50.902 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.902 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.902 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:52:50.902 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.204409+0000", 2024-02-09T17:52:50.902 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.192007386 2024-02-09T17:52:50.902 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.902 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.902 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:50.902 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.206652+0000", 2024-02-09T17:52:50.902 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0022421519999999999 2024-02-09T17:52:50.902 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.903 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.903 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:50.903 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.206898+0000", 2024-02-09T17:52:50.903 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00024642899999999999 2024-02-09T17:52:50.903 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.903 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.903 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:52:50.903 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.206950+0000", 2024-02-09T17:52:50.903 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2176000000000003e-05 2024-02-09T17:52:50.903 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.903 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.903 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:52:50.903 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.206986+0000", 2024-02-09T17:52:50.903 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.5976999999999998e-05 2024-02-09T17:52:50.903 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:50.904 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:50.904 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:50.904 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.904 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.904 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.16440.0:60993 7.4 7:2a843e6d:::benchmark_data_smithi179_135804_object60992:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e674)", 2024-02-09T17:52:50.904 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:50:55.012103+0000", 2024-02-09T17:52:50.904 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 115.597398356, 2024-02-09T17:52:50.904 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20198994000000001, 2024-02-09T17:52:50.904 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:52:50.904 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:52:50.904 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:52:50.904 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.16440", 2024-02-09T17:52:50.904 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:52:50.905 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 60993 2024-02-09T17:52:50.905 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.905 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:52:50.905 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.905 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:52:50.905 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.012103+0000", 2024-02-09T17:52:50.905 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:52:50.905 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.905 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.905 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:52:50.905 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.012103+0000", 2024-02-09T17:52:50.905 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:52:50.905 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.905 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.905 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:52:50.906 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.012103+0000", 2024-02-09T17:52:50.906 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1900000000000003e-07 2024-02-09T17:52:50.906 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.906 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.906 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:52:50.906 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.012138+0000", 2024-02-09T17:52:50.906 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.5407999999999997e-05 2024-02-09T17:52:50.906 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.906 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.906 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:52:50.906 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.012146+0000", 2024-02-09T17:52:50.906 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.4189999999999996e-06 2024-02-09T17:52:50.906 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.906 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.906 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:52:50.907 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.012151+0000", 2024-02-09T17:52:50.907 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3970000000000002e-06 2024-02-09T17:52:50.907 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.907 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.907 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:52:50.907 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.202126+0000", 2024-02-09T17:52:50.907 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.18997508199999999 2024-02-09T17:52:50.907 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.907 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.907 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:52:50.907 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.202292+0000", 2024-02-09T17:52:50.907 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000166012 2024-02-09T17:52:50.907 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.907 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.908 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:52:50.908 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.202428+0000", 2024-02-09T17:52:50.908 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000135711 2024-02-09T17:52:50.908 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.908 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.908 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:52:50.908 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.206666+0000", 2024-02-09T17:52:50.908 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.004237879 2024-02-09T17:52:50.908 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.908 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.908 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:50.908 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.207053+0000", 2024-02-09T17:52:50.908 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00038739900000000002 2024-02-09T17:52:50.908 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.909 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.909 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:50.909 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.214006+0000", 2024-02-09T17:52:50.909 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0069521499999999998 2024-02-09T17:52:50.909 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.909 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.909 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:52:50.909 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.214057+0000", 2024-02-09T17:52:50.909 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1211999999999997e-05 2024-02-09T17:52:50.909 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.909 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.909 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:52:50.909 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.214092+0000", 2024-02-09T17:52:50.909 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.5751999999999999e-05 2024-02-09T17:52:50.909 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:50.909 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:50.910 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:50.910 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.910 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.910 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.16440.0:60994 7.4 7:28e19133:::benchmark_data_smithi179_135804_object60993:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e674)", 2024-02-09T17:52:50.910 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:50:55.012158+0000", 2024-02-09T17:52:50.910 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 115.59734255799999, 2024-02-09T17:52:50.910 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20203644600000001, 2024-02-09T17:52:50.910 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:52:50.910 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:52:50.910 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:52:50.910 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.16440", 2024-02-09T17:52:50.910 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:52:50.910 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 60994 2024-02-09T17:52:50.911 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.911 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:52:50.911 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.911 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:52:50.911 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.012158+0000", 2024-02-09T17:52:50.911 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:52:50.911 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.911 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.911 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:52:50.911 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.012158+0000", 2024-02-09T17:52:50.911 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:52:50.911 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.911 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.911 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:52:50.911 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.012159+0000", 2024-02-09T17:52:50.912 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2600000000000002e-07 2024-02-09T17:52:50.912 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.912 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.912 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:52:50.912 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.012191+0000", 2024-02-09T17:52:50.912 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.1888999999999999e-05 2024-02-09T17:52:50.912 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.912 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.912 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:52:50.912 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.012197+0000", 2024-02-09T17:52:50.912 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.4989999999999999e-06 2024-02-09T17:52:50.912 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.913 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.913 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:52:50.913 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.012202+0000", 2024-02-09T17:52:50.913 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.6630000000000001e-06 2024-02-09T17:52:50.913 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.913 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.913 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:52:50.913 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.204440+0000", 2024-02-09T17:52:50.913 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.19223841899999999 2024-02-09T17:52:50.913 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.913 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.913 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:52:50.913 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.204583+0000", 2024-02-09T17:52:50.913 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000142483 2024-02-09T17:52:50.914 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.914 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.914 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:52:50.914 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.204727+0000", 2024-02-09T17:52:50.914 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000144413 2024-02-09T17:52:50.914 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.914 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.914 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:50.914 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.207102+0000", 2024-02-09T17:52:50.914 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0023749190000000001 2024-02-09T17:52:50.914 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.914 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.914 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:52:50.914 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.211602+0000", 2024-02-09T17:52:50.914 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0044995649999999996 2024-02-09T17:52:50.915 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.915 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.915 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:50.915 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.214129+0000", 2024-02-09T17:52:50.915 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0025270620000000001 2024-02-09T17:52:50.915 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.915 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.915 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:52:50.915 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.214160+0000", 2024-02-09T17:52:50.915 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.1038000000000003e-05 2024-02-09T17:52:50.915 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.915 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.915 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:52:50.915 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.214195+0000", 2024-02-09T17:52:50.915 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.4969999999999999e-05 2024-02-09T17:52:50.916 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:50.916 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:50.916 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:50.916 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.916 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.916 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.16440.0:60995 7.d e674/661 7:be52cce4:::benchmark_data_smithi179_135804_object60994:head v 674'3776, mlcod=674'3774)", 2024-02-09T17:52:50.916 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:50:55.016422+0000", 2024-02-09T17:52:50.916 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 115.593079241, 2024-02-09T17:52:50.916 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.196075797, 2024-02-09T17:52:50.916 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:52:50.916 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:52:50.917 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:52:50.917 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.917 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:52:50.917 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.016422+0000", 2024-02-09T17:52:50.917 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:52:50.917 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.917 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.917 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:52:50.917 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.016422+0000", 2024-02-09T17:52:50.917 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:52:50.917 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.917 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.917 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:52:50.917 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.016422+0000", 2024-02-09T17:52:50.917 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.0400000000000005e-07 2024-02-09T17:52:50.918 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.918 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.918 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:52:50.918 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.016557+0000", 2024-02-09T17:52:50.918 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000135017 2024-02-09T17:52:50.918 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.918 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.918 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:52:50.918 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.202022+0000", 2024-02-09T17:52:50.918 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.185464823 2024-02-09T17:52:50.918 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.919 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.919 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:52:50.919 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.202037+0000", 2024-02-09T17:52:50.919 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4538e-05 2024-02-09T17:52:50.919 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.919 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.919 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:52:50.920 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.202113+0000", 2024-02-09T17:52:50.920 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.6124000000000005e-05 2024-02-09T17:52:50.920 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.920 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.920 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:52:50.920 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.202138+0000", 2024-02-09T17:52:50.920 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.5415e-05 2024-02-09T17:52:50.920 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.920 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.920 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:52:50.920 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.208662+0000", 2024-02-09T17:52:50.920 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0065240009999999998 2024-02-09T17:52:50.920 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.920 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.921 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:52:50.921 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.212497+0000", 2024-02-09T17:52:50.921 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0038350749999999999 2024-02-09T17:52:50.921 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:50.921 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:50.921 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:50.921 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.921 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.921 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.16440.0:61011 7.f e674/661 7:f12e60e5:::benchmark_data_smithi179_135804_object61010:head v 674'3854, mlcod=674'3852)", 2024-02-09T17:52:50.921 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:50:55.026760+0000", 2024-02-09T17:52:50.921 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 115.582741313, 2024-02-09T17:52:50.921 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.177944042, 2024-02-09T17:52:50.921 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:52:50.921 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:52:50.921 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:52:50.922 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.922 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:52:50.922 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.026760+0000", 2024-02-09T17:52:50.922 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:52:50.922 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.922 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.922 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:52:50.922 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.026760+0000", 2024-02-09T17:52:50.922 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:52:50.922 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.922 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.922 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:52:50.922 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.026761+0000", 2024-02-09T17:52:50.922 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.128e-06 2024-02-09T17:52:50.922 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.923 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.923 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:52:50.923 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.026821+0000", 2024-02-09T17:52:50.923 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.0739999999999998e-05 2024-02-09T17:52:50.923 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.923 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.923 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:52:50.923 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.202043+0000", 2024-02-09T17:52:50.923 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.17522120699999999 2024-02-09T17:52:50.923 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.923 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.923 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:52:50.923 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.202053+0000", 2024-02-09T17:52:50.923 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.8649999999999996e-06 2024-02-09T17:52:50.923 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.924 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.924 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:52:50.924 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.202183+0000", 2024-02-09T17:52:50.924 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013011000000000001 2024-02-09T17:52:50.924 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.924 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.924 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:52:50.924 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.202201+0000", 2024-02-09T17:52:50.924 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.8499999999999999e-05 2024-02-09T17:52:50.924 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.924 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.924 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:52:50.925 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.204620+0000", 2024-02-09T17:52:50.925 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00241881 2024-02-09T17:52:50.925 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.925 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.925 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:52:50.925 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.204704+0000", 2024-02-09T17:52:50.925 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.3682000000000003e-05 2024-02-09T17:52:50.926 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:50.926 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:50.926 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:50.926 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.926 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.926 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.16440.0:64896 7.4 7:22763377:::benchmark_data_smithi179_135804_object64895:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e677)", 2024-02-09T17:52:50.926 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:50:57.781322+0000", 2024-02-09T17:52:50.926 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 112.82817901, 2024-02-09T17:52:50.926 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.21076742700000001, 2024-02-09T17:52:50.927 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:52:50.927 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:52:50.927 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:52:50.927 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.16440", 2024-02-09T17:52:50.927 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:52:50.927 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 64896 2024-02-09T17:52:50.927 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.927 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:52:50.927 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.928 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:52:50.928 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.781322+0000", 2024-02-09T17:52:50.928 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:52:50.928 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.928 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.928 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:52:50.928 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.781322+0000", 2024-02-09T17:52:50.928 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:52:50.928 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.928 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.928 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:52:50.928 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.781322+0000", 2024-02-09T17:52:50.928 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.5699999999999998e-07 2024-02-09T17:52:50.929 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.929 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.929 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:52:50.929 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.781365+0000", 2024-02-09T17:52:50.929 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2533000000000001e-05 2024-02-09T17:52:50.929 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.929 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.929 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:52:50.929 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.781372+0000", 2024-02-09T17:52:50.929 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.391e-06 2024-02-09T17:52:50.929 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.929 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.929 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:52:50.929 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.781378+0000", 2024-02-09T17:52:50.929 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.5350000000000004e-06 2024-02-09T17:52:50.930 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.930 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.930 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:52:50.930 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.781431+0000", 2024-02-09T17:52:50.930 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3520000000000003e-05 2024-02-09T17:52:50.930 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.930 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.930 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:52:50.930 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.781545+0000", 2024-02-09T17:52:50.930 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000113267 2024-02-09T17:52:50.930 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.930 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.930 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:52:50.930 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.781663+0000", 2024-02-09T17:52:50.930 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000118776 2024-02-09T17:52:50.931 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.931 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.931 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:52:50.931 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.785902+0000", 2024-02-09T17:52:50.931 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0042387529999999996 2024-02-09T17:52:50.931 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.931 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.931 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:50.931 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.792579+0000", 2024-02-09T17:52:50.931 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0066770900000000001 2024-02-09T17:52:50.931 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.931 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.931 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:50.931 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.992022+0000", 2024-02-09T17:52:50.931 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.19944258600000001 2024-02-09T17:52:50.932 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.932 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.932 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:52:50.932 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.992062+0000", 2024-02-09T17:52:50.932 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9737999999999998e-05 2024-02-09T17:52:50.932 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.932 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.932 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:52:50.932 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.992089+0000", 2024-02-09T17:52:50.932 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.7781e-05 2024-02-09T17:52:50.932 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:50.932 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:50.932 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:50.932 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.932 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.933 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.16440.0:64918 7.4 7:21f41282:::benchmark_data_smithi179_135804_object64917:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e677)", 2024-02-09T17:52:50.933 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:50:57.796365+0000", 2024-02-09T17:52:50.933 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 112.81313633000001, 2024-02-09T17:52:50.933 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.198345465, 2024-02-09T17:52:50.933 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:52:50.933 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:52:50.933 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:52:50.933 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.16440", 2024-02-09T17:52:50.933 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:52:50.933 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 64918 2024-02-09T17:52:50.933 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.933 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:52:50.933 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.933 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:52:50.934 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.796365+0000", 2024-02-09T17:52:50.934 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:52:50.934 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.934 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.934 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:52:50.934 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.796365+0000", 2024-02-09T17:52:50.934 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:52:50.934 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.934 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.934 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:52:50.934 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.796365+0000", 2024-02-09T17:52:50.934 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.0599999999999999e-07 2024-02-09T17:52:50.934 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.934 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.934 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:52:50.935 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.796386+0000", 2024-02-09T17:52:50.935 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0540999999999999e-05 2024-02-09T17:52:50.935 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.935 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.935 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:52:50.935 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.796401+0000", 2024-02-09T17:52:50.935 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5362e-05 2024-02-09T17:52:50.935 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.935 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.935 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:52:50.935 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.796408+0000", 2024-02-09T17:52:50.935 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.861e-06 2024-02-09T17:52:50.935 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.935 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.936 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:52:50.936 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.796452+0000", 2024-02-09T17:52:50.936 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.4135999999999999e-05 2024-02-09T17:52:50.936 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.936 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.936 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:52:50.936 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.796609+0000", 2024-02-09T17:52:50.936 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00015707899999999999 2024-02-09T17:52:50.936 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.936 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.936 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:52:50.936 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.796764+0000", 2024-02-09T17:52:50.936 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00015433499999999999 2024-02-09T17:52:50.937 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.937 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.937 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:52:50.937 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.798793+0000", 2024-02-09T17:52:50.937 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0020289610000000001 2024-02-09T17:52:50.937 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.937 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.937 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:50.937 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.801981+0000", 2024-02-09T17:52:50.937 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.003188115 2024-02-09T17:52:50.937 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.937 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.937 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:50.937 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.994653+0000", 2024-02-09T17:52:50.937 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.192671912 2024-02-09T17:52:50.938 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.938 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.938 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:52:50.938 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.994687+0000", 2024-02-09T17:52:50.938 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.4570000000000003e-05 2024-02-09T17:52:50.938 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.938 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.938 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:52:50.938 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.994710+0000", 2024-02-09T17:52:50.938 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2687000000000001e-05 2024-02-09T17:52:50.938 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:50.938 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:50.938 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:50.938 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.938 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.939 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.16440.0:64934 7.4 7:2d16648d:::benchmark_data_smithi179_135804_object64933:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e677)", 2024-02-09T17:52:50.939 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:50:57.805129+0000", 2024-02-09T17:52:50.939 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 112.804371901, 2024-02-09T17:52:50.939 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.18967294900000001, 2024-02-09T17:52:50.939 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:52:50.939 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:52:50.939 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:52:50.939 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.16440", 2024-02-09T17:52:50.939 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:52:50.939 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 64934 2024-02-09T17:52:50.939 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.939 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:52:50.939 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.939 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:52:50.939 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.805129+0000", 2024-02-09T17:52:50.940 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:52:50.940 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.940 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.940 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:52:50.940 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.805129+0000", 2024-02-09T17:52:50.940 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:52:50.940 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.940 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.940 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:52:50.940 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.805131+0000", 2024-02-09T17:52:50.940 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.745e-06 2024-02-09T17:52:50.940 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.940 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.941 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:52:50.941 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.805159+0000", 2024-02-09T17:52:50.941 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.8586e-05 2024-02-09T17:52:50.941 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.941 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.941 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:52:50.941 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.805172+0000", 2024-02-09T17:52:50.941 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2452999999999999e-05 2024-02-09T17:52:50.941 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.941 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.941 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:52:50.941 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.805180+0000", 2024-02-09T17:52:50.941 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.4610000000000008e-06 2024-02-09T17:52:50.941 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.942 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.942 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:52:50.942 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.805237+0000", 2024-02-09T17:52:50.942 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6345999999999999e-05 2024-02-09T17:52:50.942 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.942 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.942 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:52:50.942 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.805374+0000", 2024-02-09T17:52:50.942 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000137844 2024-02-09T17:52:50.942 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.942 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.942 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:52:50.942 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.805513+0000", 2024-02-09T17:52:50.942 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013850099999999999 2024-02-09T17:52:50.942 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.943 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.943 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:52:50.943 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.810399+0000", 2024-02-09T17:52:50.943 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0048859159999999997 2024-02-09T17:52:50.943 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.943 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.943 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:50.943 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.811445+0000", 2024-02-09T17:52:50.943 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0010456949999999999 2024-02-09T17:52:50.943 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.943 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.943 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:50.943 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.994744+0000", 2024-02-09T17:52:50.943 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.18329941699999999 2024-02-09T17:52:50.943 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.944 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.944 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:52:50.944 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.994778+0000", 2024-02-09T17:52:50.944 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.3612000000000003e-05 2024-02-09T17:52:50.944 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.944 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.944 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:52:50.944 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.994802+0000", 2024-02-09T17:52:50.944 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.4372999999999999e-05 2024-02-09T17:52:50.944 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:50.944 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:50.944 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:50.944 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.944 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.944 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.16440.0:64958 7.4 7:26eaf9c4:::benchmark_data_smithi179_135804_object64957:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e677)", 2024-02-09T17:52:50.945 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:50:57.820974+0000", 2024-02-09T17:52:50.945 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 112.78852690399999, 2024-02-09T17:52:50.945 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.17518472700000001, 2024-02-09T17:52:50.945 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:52:50.945 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:52:50.945 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:52:50.945 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.16440", 2024-02-09T17:52:50.945 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:52:50.945 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 64958 2024-02-09T17:52:50.945 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.945 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:52:50.945 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.946 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:52:50.946 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.820974+0000", 2024-02-09T17:52:50.946 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:52:50.946 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.946 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.946 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:52:50.946 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.820974+0000", 2024-02-09T17:52:50.946 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:52:50.946 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.946 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.946 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:52:50.946 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.820975+0000", 2024-02-09T17:52:50.946 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.328e-06 2024-02-09T17:52:50.946 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.947 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.947 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:52:50.947 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.821048+0000", 2024-02-09T17:52:50.947 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.2988e-05 2024-02-09T17:52:50.947 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.947 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.947 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:52:50.947 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.821060+0000", 2024-02-09T17:52:50.947 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1438e-05 2024-02-09T17:52:50.947 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.947 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.947 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:52:50.947 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.821068+0000", 2024-02-09T17:52:50.947 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.3310000000000004e-06 2024-02-09T17:52:50.947 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.948 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.948 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:52:50.948 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.821127+0000", 2024-02-09T17:52:50.948 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.8465999999999999e-05 2024-02-09T17:52:50.948 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.948 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.948 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:52:50.948 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.821259+0000", 2024-02-09T17:52:50.948 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013286400000000001 2024-02-09T17:52:50.948 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.948 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.948 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:52:50.948 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.821421+0000", 2024-02-09T17:52:50.948 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00016158399999999999 2024-02-09T17:52:50.949 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.949 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.949 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:52:50.949 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.826247+0000", 2024-02-09T17:52:50.949 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0048264249999999996 2024-02-09T17:52:50.949 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.949 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.949 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:50.949 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.831024+0000", 2024-02-09T17:52:50.949 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.004776443 2024-02-09T17:52:50.949 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.949 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.949 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:50.949 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.996071+0000", 2024-02-09T17:52:50.949 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.16504697900000001 2024-02-09T17:52:50.950 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.950 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.950 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:52:50.950 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.996124+0000", 2024-02-09T17:52:50.950 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3575999999999997e-05 2024-02-09T17:52:50.950 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.950 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.950 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:52:50.950 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.996159+0000", 2024-02-09T17:52:50.950 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.4304999999999998e-05 2024-02-09T17:52:50.950 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:50.950 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:50.951 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:50.951 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.951 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.951 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.16440.0:64974 7.4 7:2cbf5b37:::benchmark_data_smithi179_135804_object64973:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e677)", 2024-02-09T17:52:50.951 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:50:57.832735+0000", 2024-02-09T17:52:50.951 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 112.776765735, 2024-02-09T17:52:50.951 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.16353572899999999, 2024-02-09T17:52:50.951 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:52:50.951 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:52:50.952 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:52:50.952 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.16440", 2024-02-09T17:52:50.952 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:52:50.952 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 64974 2024-02-09T17:52:50.952 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.952 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:52:50.952 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.952 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:52:50.952 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.832735+0000", 2024-02-09T17:52:50.952 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:52:50.952 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.952 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.952 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:52:50.952 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.832735+0000", 2024-02-09T17:52:50.952 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:52:50.953 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.953 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.953 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:52:50.953 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.832736+0000", 2024-02-09T17:52:50.953 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0780000000000001e-06 2024-02-09T17:52:50.953 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.953 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.953 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:52:50.953 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.832846+0000", 2024-02-09T17:52:50.953 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000110194 2024-02-09T17:52:50.953 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.953 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.953 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:52:50.953 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.832858+0000", 2024-02-09T17:52:50.953 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.114e-05 2024-02-09T17:52:50.954 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.954 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.954 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:52:50.954 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.832866+0000", 2024-02-09T17:52:50.954 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.0819999999999999e-06 2024-02-09T17:52:50.954 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.954 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.954 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:52:50.954 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.832925+0000", 2024-02-09T17:52:50.954 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.8956999999999998e-05 2024-02-09T17:52:50.954 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.954 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.954 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:52:50.954 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.833080+0000", 2024-02-09T17:52:50.954 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000155771 2024-02-09T17:52:50.955 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.955 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.955 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:52:50.955 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.833239+0000", 2024-02-09T17:52:50.955 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00015893600000000001 2024-02-09T17:52:50.955 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.955 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.955 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:50.955 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.838117+0000", 2024-02-09T17:52:50.955 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0048777650000000001 2024-02-09T17:52:50.955 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.955 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.955 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:52:50.955 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.840606+0000", 2024-02-09T17:52:50.955 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0024888250000000001 2024-02-09T17:52:50.955 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.956 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.956 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:50.956 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.996193+0000", 2024-02-09T17:52:50.956 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.155587322 2024-02-09T17:52:50.956 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.956 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.956 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:52:50.956 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.996240+0000", 2024-02-09T17:52:50.956 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.6706000000000003e-05 2024-02-09T17:52:50.956 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.956 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.956 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:52:50.956 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.996271+0000", 2024-02-09T17:52:50.956 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.0953e-05 2024-02-09T17:52:50.957 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:50.957 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:50.957 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:50.957 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.957 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.957 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.16440.0:64980 7.4 7:23886a11:::benchmark_data_smithi179_135804_object64979:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e677)", 2024-02-09T17:52:50.957 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:50:57.836493+0000", 2024-02-09T17:52:50.957 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 112.773007795, 2024-02-09T17:52:50.957 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.16304988000000001, 2024-02-09T17:52:50.957 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:52:50.957 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:52:50.957 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:52:50.957 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.16440", 2024-02-09T17:52:50.957 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:52:50.957 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 64980 2024-02-09T17:52:50.958 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.958 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:52:50.958 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.958 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:52:50.958 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.836493+0000", 2024-02-09T17:52:50.958 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:52:50.958 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.958 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.958 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:52:50.958 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.836493+0000", 2024-02-09T17:52:50.958 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:52:50.958 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.958 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.958 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:52:50.958 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.836494+0000", 2024-02-09T17:52:50.959 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.8899999999999998e-07 2024-02-09T17:52:50.959 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.959 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.959 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:52:50.959 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.838056+0000", 2024-02-09T17:52:50.959 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0015616429999999999 2024-02-09T17:52:50.959 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.959 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.959 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:52:50.959 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.838066+0000", 2024-02-09T17:52:50.959 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0723000000000001e-05 2024-02-09T17:52:50.959 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.959 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.959 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:52:50.959 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.838074+0000", 2024-02-09T17:52:50.960 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.6029999999999999e-06 2024-02-09T17:52:50.960 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.960 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.960 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:52:50.960 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.838152+0000", 2024-02-09T17:52:50.960 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.8015000000000004e-05 2024-02-09T17:52:50.960 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.960 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.960 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:52:50.960 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.838294+0000", 2024-02-09T17:52:50.960 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014194500000000001 2024-02-09T17:52:50.960 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.960 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.961 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:52:50.961 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.838449+0000", 2024-02-09T17:52:50.961 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00015503500000000001 2024-02-09T17:52:50.961 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.961 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.961 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:52:50.961 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.840725+0000", 2024-02-09T17:52:50.961 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0022764249999999999 2024-02-09T17:52:50.961 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.961 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.961 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:50.961 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.844384+0000", 2024-02-09T17:52:50.961 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0036583280000000002 2024-02-09T17:52:50.961 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.961 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.962 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:52:50.962 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.999442+0000", 2024-02-09T17:52:50.962 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.155058533 2024-02-09T17:52:50.962 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.962 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.962 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:52:50.962 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.999501+0000", 2024-02-09T17:52:50.962 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.9018999999999999e-05 2024-02-09T17:52:50.962 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:52:50.962 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:52:50.962 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:52:50.962 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.999543+0000", 2024-02-09T17:52:50.962 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.1622000000000002e-05 2024-02-09T17:52:50.962 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:50.962 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:50.962 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:50.963 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:52:50.963 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:52:50.963 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:52:50.963 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:52:50.997 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:51.105 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:52:51.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:51.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:52:51.212 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:52:51.241 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:51.595 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-09T17:52:51.596 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:52:51.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:51.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:52:51.609 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:52:51.633 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:52:51.649 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:52:51.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:51 smithi162 ceph-mon[44304]: osdmap e723: 12 total, 12 up, 12 in 2024-02-09T17:52:51.704 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:52:51.733 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:51.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:51 smithi099 ceph-mon[39420]: osdmap e723: 12 total, 12 up, 12 in 2024-02-09T17:52:51.755 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:52:51.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:51.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:52:51.861 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:52:51.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:51 smithi179 ceph-mon[43975]: osdmap e723: 12 total, 12 up, 12 in 2024-02-09T17:52:51.936 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:52.710 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:52:52.716 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:52 smithi099 ceph-mon[39420]: pgmap v1806: 47 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 41 active+clean; 4.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 489 op/s; 1/227634 objects degraded (0.000%); 5813/227634 objects misplaced (2.554%); 13 MiB/s, 211 objects/s recovering 2024-02-09T17:52:52.716 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:52 smithi099 ceph-mon[39420]: osdmap e724: 12 total, 12 up, 12 in 2024-02-09T17:52:52.716 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:52 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 1/227634 objects degraded (0.000%), 1 pg degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-09T17:52:52.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:52.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:52:52.818 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:52:52.853 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:52.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:52 smithi179 ceph-mon[43975]: pgmap v1806: 47 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 41 active+clean; 4.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 489 op/s; 1/227634 objects degraded (0.000%); 5813/227634 objects misplaced (2.554%); 13 MiB/s, 211 objects/s recovering 2024-02-09T17:52:52.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:52 smithi179 ceph-mon[43975]: osdmap e724: 12 total, 12 up, 12 in 2024-02-09T17:52:52.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:52 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 1/227634 objects degraded (0.000%), 1 pg degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-09T17:52:52.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:52 smithi162 ceph-mon[44304]: pgmap v1806: 47 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 41 active+clean; 4.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 489 op/s; 1/227634 objects degraded (0.000%); 5813/227634 objects misplaced (2.554%); 13 MiB/s, 211 objects/s recovering 2024-02-09T17:52:52.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:52 smithi162 ceph-mon[44304]: osdmap e724: 12 total, 12 up, 12 in 2024-02-09T17:52:52.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:52 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 1/227634 objects degraded (0.000%), 1 pg degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-09T17:52:53.451 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:53 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:52:53.452 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:53 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:52:53.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:53 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:52:53.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:53 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:52:53.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:53 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:52:53.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:53 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:52:54.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:54 smithi099 ceph-mon[39420]: pgmap v1808: 47 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 41 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 527 op/s; 1/222861 objects degraded (0.000%); 4684/222861 objects misplaced (2.102%); 15 MiB/s, 247 objects/s recovering 2024-02-09T17:52:54.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:54 smithi099 ceph-mon[39420]: osdmap e725: 12 total, 12 up, 12 in 2024-02-09T17:52:54.880 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:52:54.900 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:52:54.921 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:52:54.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:54 smithi179 ceph-mon[43975]: pgmap v1808: 47 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 41 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 527 op/s; 1/222861 objects degraded (0.000%); 4684/222861 objects misplaced (2.102%); 15 MiB/s, 247 objects/s recovering 2024-02-09T17:52:54.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:54 smithi179 ceph-mon[43975]: osdmap e725: 12 total, 12 up, 12 in 2024-02-09T17:52:54.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:54 smithi162 ceph-mon[44304]: pgmap v1808: 47 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 41 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 527 op/s; 1/222861 objects degraded (0.000%); 4684/222861 objects misplaced (2.102%); 15 MiB/s, 247 objects/s recovering 2024-02-09T17:52:54.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:54 smithi162 ceph-mon[44304]: osdmap e725: 12 total, 12 up, 12 in 2024-02-09T17:52:54.948 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:52:54.968 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:52:54.989 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:52:55.019 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:52:55.040 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:52:55.061 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:52:55.081 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:52:55.103 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:52:55.126 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:52:55.127 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:52:55.130 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:52:55.130 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:52:55.131 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:52:55.131 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:52:55.131 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:52:55.131 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:52:55.131 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:52:55.131 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:52:55.131 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:52:55.131 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:52:55.131 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:52:55.131 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:52:55.131 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:52:55.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:55.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:52:55.233 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:52:55.264 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:55.286 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:52:55.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:55.288 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:52:55.389 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:52:55.419 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:55.458 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:52:55.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:55.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:52:55.564 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:52:55.595 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:55.620 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:52:55.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:55.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:52:55.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:55 smithi179 ceph-mon[43975]: osdmap e726: 12 total, 12 up, 12 in 2024-02-09T17:52:55.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:55 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/519563253' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:52:55.729 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:52:55.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:55 smithi099 ceph-mon[39420]: osdmap e726: 12 total, 12 up, 12 in 2024-02-09T17:52:55.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:55 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/519563253' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:52:55.753 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:55 smithi162 ceph-mon[44304]: osdmap e726: 12 total, 12 up, 12 in 2024-02-09T17:52:55.753 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:55 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/519563253' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:52:55.755 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:55.776 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:52:55.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:55.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:52:55.881 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:52:55.912 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:55.934 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:52:55.936 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:55.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:52:56.038 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:52:56.068 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:56.093 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:52:56.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:56.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:52:56.201 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:52:56.237 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:56.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:56 smithi099 ceph-mon[39420]: pgmap v1811: 47 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 41 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 496 op/s; 1/222240 objects degraded (0.000%); 4283/222240 objects misplaced (1.927%); 21 MiB/s, 328 objects/s recovering 2024-02-09T17:52:56.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:56 smithi099 ceph-mon[39420]: osdmap e727: 12 total, 12 up, 12 in 2024-02-09T17:52:56.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:56 smithi162 ceph-mon[44304]: pgmap v1811: 47 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 41 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 496 op/s; 1/222240 objects degraded (0.000%); 4283/222240 objects misplaced (1.927%); 21 MiB/s, 328 objects/s recovering 2024-02-09T17:52:56.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:56 smithi162 ceph-mon[44304]: osdmap e727: 12 total, 12 up, 12 in 2024-02-09T17:52:56.927 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:56 smithi179 ceph-mon[43975]: pgmap v1811: 47 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+remapped, 41 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 496 op/s; 1/222240 objects degraded (0.000%); 4283/222240 objects misplaced (1.927%); 21 MiB/s, 328 objects/s recovering 2024-02-09T17:52:56.927 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:56 smithi179 ceph-mon[43975]: osdmap e727: 12 total, 12 up, 12 in 2024-02-09T17:52:57.394 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:52:57.399 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:57.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:52:57.500 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:52:57.570 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:57.608 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:52:57.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:57.612 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:52:57.713 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:52:57.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:57 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:52:57.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:57 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:52:57.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:57 smithi099 ceph-mon[39420]: osdmap e728: 12 total, 12 up, 12 in 2024-02-09T17:52:57.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:57 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 1/222240 objects degraded (0.000%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:52:57.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:57 smithi099 ceph-mon[39420]: osdmap e729: 12 total, 12 up, 12 in 2024-02-09T17:52:57.746 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:57.773 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:52:57.780 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:57 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:52:57.781 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:57 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:52:57.781 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:57 smithi162 ceph-mon[44304]: osdmap e728: 12 total, 12 up, 12 in 2024-02-09T17:52:57.781 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:57 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 1/222240 objects degraded (0.000%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:52:57.781 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:57 smithi162 ceph-mon[44304]: osdmap e729: 12 total, 12 up, 12 in 2024-02-09T17:52:57.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:57.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:52:57.883 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:52:57.919 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:57.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:57 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:52:57.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:57 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:52:57.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:57 smithi179 ceph-mon[43975]: osdmap e728: 12 total, 12 up, 12 in 2024-02-09T17:52:57.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:57 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 1/222240 objects degraded (0.000%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-09T17:52:57.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:57 smithi179 ceph-mon[43975]: osdmap e729: 12 total, 12 up, 12 in 2024-02-09T17:52:58.021 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:52:58.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:58.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:52:58.128 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:52:58.163 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:58.193 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:52:58.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:58.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:52:58.298 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:52:58.344 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:58.377 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:52:58.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:58.396 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:52:58.497 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:52:58.551 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:58.577 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:52:58.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:58.585 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:52:58.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:58 smithi179 ceph-mon[43975]: pgmap v1815: 47 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 43 active+clean; 4.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 581 op/s; 1/216540 objects degraded (0.000%); 3622/216540 objects misplaced (1.673%); 18 MiB/s, 292 objects/s recovering 2024-02-09T17:52:58.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:58 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/598031227' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-09T17:52:58.685 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:52:58.718 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:58.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:58 smithi099 ceph-mon[39420]: pgmap v1815: 47 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 43 active+clean; 4.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 581 op/s; 1/216540 objects degraded (0.000%); 3622/216540 objects misplaced (1.673%); 18 MiB/s, 292 objects/s recovering 2024-02-09T17:52:58.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:58 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/598031227' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-09T17:52:58.747 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:52:58.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:58.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:52:58.755 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:58 smithi162 ceph-mon[44304]: pgmap v1815: 47 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 43 active+clean; 4.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 581 op/s; 1/216540 objects degraded (0.000%); 3622/216540 objects misplaced (1.673%); 18 MiB/s, 292 objects/s recovering 2024-02-09T17:52:58.755 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:58 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/598031227' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-09T17:52:58.836 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.3 to 1 (10000) 2024-02-09T17:52:58.855 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:52:58.890 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:52:59.300 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:52:59.300 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:52:59.900 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:52:59.938 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-09T17:52:59.939 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:52:59.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:52:59.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:53:00.043 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:53:00.069 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:00.170 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:53:00.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:59 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/598031227' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-09T17:53:00.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:59 smithi179 ceph-mon[43975]: osdmap e730: 12 total, 12 up, 12 in 2024-02-09T17:53:00.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:52:59 smithi179 ceph-mon[43975]: pgmap v1817: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+remapped, 44 active+clean; 4.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 628 op/s; 1/214692 objects degraded (0.000%); 3241/214692 objects misplaced (1.510%); 16 MiB/s, 260 objects/s recovering 2024-02-09T17:53:00.173 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:59 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/598031227' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-09T17:53:00.173 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:59 smithi099 ceph-mon[39420]: osdmap e730: 12 total, 12 up, 12 in 2024-02-09T17:53:00.173 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:52:59 smithi099 ceph-mon[39420]: pgmap v1817: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+remapped, 44 active+clean; 4.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 628 op/s; 1/214692 objects degraded (0.000%); 3241/214692 objects misplaced (1.510%); 16 MiB/s, 260 objects/s recovering 2024-02-09T17:53:00.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:00.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:53:00.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:59 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/598031227' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-09T17:53:00.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:59 smithi162 ceph-mon[44304]: osdmap e730: 12 total, 12 up, 12 in 2024-02-09T17:53:00.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:52:59 smithi162 ceph-mon[44304]: pgmap v1817: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+remapped, 44 active+clean; 4.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 628 op/s; 1/214692 objects degraded (0.000%); 3241/214692 objects misplaced (1.510%); 16 MiB/s, 260 objects/s recovering 2024-02-09T17:53:00.275 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:53:00.304 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:00.651 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:53:00.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:00.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:53:00.657 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 4 1 2024-02-09T17:53:00.755 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:53:00.783 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:00.804 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:53:00.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:00.809 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:53:00.910 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:53:00.916 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:53:00.918 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:53:00.919 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:00 smithi099 ceph-mon[39420]: osdmap e731: 12 total, 12 up, 12 in 2024-02-09T17:53:00.939 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:00.963 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:53:00.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:00.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:53:01.067 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:53:01.098 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:01.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:00 smithi179 ceph-mon[43975]: osdmap e731: 12 total, 12 up, 12 in 2024-02-09T17:53:01.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:00 smithi162 ceph-mon[44304]: osdmap e731: 12 total, 12 up, 12 in 2024-02-09T17:53:02.153 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:01 smithi099 ceph-mon[39420]: osdmap e732: 12 total, 12 up, 12 in 2024-02-09T17:53:02.153 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:01 smithi099 ceph-mon[39420]: 7.7 starting backfill to osd.3 from (0'0,0'0] MAX to 729'11755 2024-02-09T17:53:02.153 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:01 smithi099 ceph-mon[39420]: pgmap v1820: 47 pgs: 3 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 42 active+clean; 4.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 378 op/s; 1/212718 objects degraded (0.000%); 3241/212718 objects misplaced (1.524%); 5.6 MiB/s, 89 objects/s recovering 2024-02-09T17:53:02.153 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:01 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 1/214692 objects degraded (0.000%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-09T17:53:02.153 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:01 smithi099 ceph-mon[39420]: osdmap e733: 12 total, 12 up, 12 in 2024-02-09T17:53:02.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:01 smithi179 ceph-mon[43975]: osdmap e732: 12 total, 12 up, 12 in 2024-02-09T17:53:02.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:01 smithi179 ceph-mon[43975]: 7.7 starting backfill to osd.3 from (0'0,0'0] MAX to 729'11755 2024-02-09T17:53:02.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:01 smithi179 ceph-mon[43975]: pgmap v1820: 47 pgs: 3 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 42 active+clean; 4.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 378 op/s; 1/212718 objects degraded (0.000%); 3241/212718 objects misplaced (1.524%); 5.6 MiB/s, 89 objects/s recovering 2024-02-09T17:53:02.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:01 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 1/214692 objects degraded (0.000%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-09T17:53:02.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:01 smithi179 ceph-mon[43975]: osdmap e733: 12 total, 12 up, 12 in 2024-02-09T17:53:02.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:01 smithi162 ceph-mon[44304]: osdmap e732: 12 total, 12 up, 12 in 2024-02-09T17:53:02.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:01 smithi162 ceph-mon[44304]: 7.7 starting backfill to osd.3 from (0'0,0'0] MAX to 729'11755 2024-02-09T17:53:02.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:01 smithi162 ceph-mon[44304]: pgmap v1820: 47 pgs: 3 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 42 active+clean; 4.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 378 op/s; 1/212718 objects degraded (0.000%); 3241/212718 objects misplaced (1.524%); 5.6 MiB/s, 89 objects/s recovering 2024-02-09T17:53:02.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:01 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 1/214692 objects degraded (0.000%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-09T17:53:02.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:01 smithi162 ceph-mon[44304]: osdmap e733: 12 total, 12 up, 12 in 2024-02-09T17:53:03.662 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:53:03.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:03.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:53:03.771 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:53:03.845 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:03.886 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is set 2024-02-09T17:53:03.895 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:53:03.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:03.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:53:03.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:03 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:53:03.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:03 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:53:03.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:03 smithi099 ceph-mon[39420]: osdmap e734: 12 total, 12 up, 12 in 2024-02-09T17:53:03.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:03 smithi099 ceph-mon[39420]: pgmap v1823: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 3 remapped+peering, 41 active+clean; 4.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 416 op/s; 6810/209694 objects degraded (3.248%); 2284/209694 objects misplaced (1.089%); 15 MiB/s, 239 objects/s recovering 2024-02-09T17:53:03.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:03 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1809054859' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:53:04.016 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:53:04.048 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:04.078 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:53:04.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:04.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:53:04.175 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:03 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:53:04.175 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:03 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:53:04.175 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:03 smithi179 ceph-mon[43975]: osdmap e734: 12 total, 12 up, 12 in 2024-02-09T17:53:04.175 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:03 smithi179 ceph-mon[43975]: pgmap v1823: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 3 remapped+peering, 41 active+clean; 4.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 416 op/s; 6810/209694 objects degraded (3.248%); 2284/209694 objects misplaced (1.089%); 15 MiB/s, 239 objects/s recovering 2024-02-09T17:53:04.175 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:03 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1809054859' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:53:04.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:03 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:53:04.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:03 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:53:04.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:03 smithi162 ceph-mon[44304]: osdmap e734: 12 total, 12 up, 12 in 2024-02-09T17:53:04.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:03 smithi162 ceph-mon[44304]: pgmap v1823: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 3 remapped+peering, 41 active+clean; 4.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 416 op/s; 6810/209694 objects degraded (3.248%); 2284/209694 objects misplaced (1.089%); 15 MiB/s, 239 objects/s recovering 2024-02-09T17:53:04.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:03 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1809054859' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:53:04.187 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:53:04.240 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:04.271 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:53:04.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:04.277 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:53:04.378 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:53:04.448 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:04.931 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:53:04.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:04.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:53:04.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:04 smithi099 ceph-mon[39420]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:53:04.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:04 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1809054859' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:53:04.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:04 smithi099 ceph-mon[39420]: osdmap e735: 12 total, 12 up, 12 in 2024-02-09T17:53:04.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:04 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:53:04.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:04 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:53:05.040 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:53:05.076 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:05.103 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:53:05.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:05.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:53:05.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:04 smithi179 ceph-mon[43975]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:53:05.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:04 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1809054859' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:53:05.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:04 smithi179 ceph-mon[43975]: osdmap e735: 12 total, 12 up, 12 in 2024-02-09T17:53:05.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:04 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:53:05.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:04 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:53:05.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:04 smithi162 ceph-mon[44304]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:53:05.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:04 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1809054859' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:53:05.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:04 smithi162 ceph-mon[44304]: osdmap e735: 12 total, 12 up, 12 in 2024-02-09T17:53:05.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:04 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:53:05.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:04 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:53:05.211 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:53:05.238 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:05.260 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:53:05.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:05.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:53:05.368 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:53:05.400 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:05.419 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:53:05.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:05.422 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:53:05.506 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:53:05.506 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:53:05.522 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:53:05.550 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:05.571 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:53:05.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:05.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:53:05.676 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:53:05.703 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:05.723 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:53:05.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:05.726 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:53:05.827 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:53:05.855 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:05.876 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:53:05.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:05.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:53:05.984 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:53:06.039 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:05 smithi179 ceph-mon[43975]: osdmap e736: 12 total, 12 up, 12 in 2024-02-09T17:53:06.039 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:05 smithi179 ceph-mon[43975]: pgmap v1826: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 3 remapped+peering, 40 active+clean; 4.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 278 op/s; 6862/209376 objects degraded (3.277%); 6513/209376 objects misplaced (3.111%); 16 MiB/s, 249 objects/s recovering 2024-02-09T17:53:06.040 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:06.072 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:53:06.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:06.077 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:53:06.178 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:53:06.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:05 smithi162 ceph-mon[44304]: osdmap e736: 12 total, 12 up, 12 in 2024-02-09T17:53:06.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:05 smithi162 ceph-mon[44304]: pgmap v1826: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 3 remapped+peering, 40 active+clean; 4.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 278 op/s; 6862/209376 objects degraded (3.277%); 6513/209376 objects misplaced (3.111%); 16 MiB/s, 249 objects/s recovering 2024-02-09T17:53:06.232 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:06.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:05 smithi099 ceph-mon[39420]: osdmap e736: 12 total, 12 up, 12 in 2024-02-09T17:53:06.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:05 smithi099 ceph-mon[39420]: pgmap v1826: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 3 remapped+peering, 40 active+clean; 4.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 278 op/s; 6862/209376 objects degraded (3.277%); 6513/209376 objects misplaced (3.111%); 16 MiB/s, 249 objects/s recovering 2024-02-09T17:53:06.258 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:53:06.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:06.264 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:53:06.364 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:53:06.398 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:06.411 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-09T17:53:06.438 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:53:06.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:06.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:53:06.546 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:53:06.639 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:06.877 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:53:06.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:06.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:53:06.889 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:53:06.988 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:53:07.025 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:07.056 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:53:07.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:07.063 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:53:07.144 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:06 smithi099 ceph-mon[39420]: osdmap e737: 12 total, 12 up, 12 in 2024-02-09T17:53:07.144 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:06 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 6862/209376 objects degraded (3.277%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-09T17:53:07.164 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:53:07.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:06 smithi179 ceph-mon[43975]: osdmap e737: 12 total, 12 up, 12 in 2024-02-09T17:53:07.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:06 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 6862/209376 objects degraded (3.277%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-09T17:53:07.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:06 smithi162 ceph-mon[44304]: osdmap e737: 12 total, 12 up, 12 in 2024-02-09T17:53:07.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:06 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 6862/209376 objects degraded (3.277%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-09T17:53:07.224 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:07.302 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:53:07.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:07.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:53:07.409 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:53:07.455 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:07.498 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:53:07.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:07.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:53:07.607 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:53:07.651 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:07.682 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:53:07.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:07.686 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:53:07.707 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:53:07.737 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:53:07.776 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:53:07.787 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:53:07.820 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:07.824 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:53:07.852 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:53:07.860 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:07.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:53:07.870 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:53:07.920 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:07 smithi179 ceph-mon[43975]: pgmap v1828: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 4.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 483 op/s; 166/203565 objects degraded (0.082%); 18843/203565 objects misplaced (9.257%); 11 MiB/s, 170 objects/s recovering 2024-02-09T17:53:07.920 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:07 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3095071557' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:53:07.926 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:53:07.961 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:53:07.962 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:53:07.997 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:07.998 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:53:08.027 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:53:08.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:08.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:53:08.035 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:53:08.096 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:53:08.131 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:53:08.139 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:53:08.169 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:08.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:07 smithi162 ceph-mon[44304]: pgmap v1828: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 4.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 483 op/s; 166/203565 objects degraded (0.082%); 18843/203565 objects misplaced (9.257%); 11 MiB/s, 170 objects/s recovering 2024-02-09T17:53:08.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:07 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3095071557' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:53:08.210 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:53:08.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:08.218 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:53:08.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:07 smithi099 ceph-mon[39420]: pgmap v1828: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 4.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 483 op/s; 166/203565 objects degraded (0.082%); 18843/203565 objects misplaced (9.257%); 11 MiB/s, 170 objects/s recovering 2024-02-09T17:53:08.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:07 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3095071557' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:53:08.319 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:53:08.364 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:08.385 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:53:08.391 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:53:08.391 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:53:08.391 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:53:08.391 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:53:08.391 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:53:08.391 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:53:08.391 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:53:08.391 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:53:08.391 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:53:08.391 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:53:08.392 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:53:08.392 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:53:08.397 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:53:08.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:08.405 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:53:08.506 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:53:08.557 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:08.588 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:53:08.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:08.593 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:53:08.694 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:53:08.725 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:08.749 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:53:08.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:08.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:53:08.859 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:53:08.920 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:08.929 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset noscrub 2024-02-09T17:53:09.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:08 smithi162 ceph-mon[44304]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:53:09.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:08 smithi099 ceph-mon[39420]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:53:09.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:08 smithi179 ceph-mon[43975]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:53:10.248 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:09 smithi099 ceph-mon[39420]: pgmap v1829: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 4.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 457 op/s; 194/201447 objects degraded (0.096%); 18289/201447 objects misplaced (9.079%); 13 MiB/s, 207 objects/s recovering 2024-02-09T17:53:10.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:09 smithi162 ceph-mon[44304]: pgmap v1829: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 4.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 457 op/s; 194/201447 objects degraded (0.096%); 18289/201447 objects misplaced (9.079%); 13 MiB/s, 207 objects/s recovering 2024-02-09T17:53:10.427 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:09 smithi179 ceph-mon[43975]: pgmap v1829: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 4.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 457 op/s; 194/201447 objects degraded (0.096%); 18289/201447 objects misplaced (9.079%); 13 MiB/s, 207 objects/s recovering 2024-02-09T17:53:11.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:10 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:53:11.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:10 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:53:11.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:10 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:53:11.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:10 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:53:11.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:10 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:53:11.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:10 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:53:11.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:10 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:53:11.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:10 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:53:11.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:10 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:53:11.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:10 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:53:11.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:10 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:53:11.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:10 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:53:12.149 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:53:12.206 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:53:12.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:12.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:53:12.315 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:53:12.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:12 smithi179 ceph-mon[43975]: osdmap e738: 12 total, 12 up, 12 in 2024-02-09T17:53:12.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:12 smithi179 ceph-mon[43975]: pgmap v1831: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 4.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 581 op/s; 256/198564 objects degraded (0.129%); 17904/198564 objects misplaced (9.017%); 13 MiB/s, 214 objects/s recovering 2024-02-09T17:53:12.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:12 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 194/201447 objects degraded (0.096%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-02-09T17:53:12.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:12 smithi179 ceph-mon[43975]: osdmap e739: 12 total, 12 up, 12 in 2024-02-09T17:53:12.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:12 smithi162 ceph-mon[44304]: osdmap e738: 12 total, 12 up, 12 in 2024-02-09T17:53:12.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:12 smithi162 ceph-mon[44304]: pgmap v1831: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 4.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 581 op/s; 256/198564 objects degraded (0.129%); 17904/198564 objects misplaced (9.017%); 13 MiB/s, 214 objects/s recovering 2024-02-09T17:53:12.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:12 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 194/201447 objects degraded (0.096%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-02-09T17:53:12.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:12 smithi162 ceph-mon[44304]: osdmap e739: 12 total, 12 up, 12 in 2024-02-09T17:53:12.431 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:12.455 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:53:12.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:12.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:53:12.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:12 smithi099 ceph-mon[39420]: osdmap e738: 12 total, 12 up, 12 in 2024-02-09T17:53:12.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:12 smithi099 ceph-mon[39420]: pgmap v1831: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 4.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 581 op/s; 256/198564 objects degraded (0.129%); 17904/198564 objects misplaced (9.017%); 13 MiB/s, 214 objects/s recovering 2024-02-09T17:53:12.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:12 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 194/201447 objects degraded (0.096%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-02-09T17:53:12.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:12 smithi099 ceph-mon[39420]: osdmap e739: 12 total, 12 up, 12 in 2024-02-09T17:53:12.559 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:53:12.619 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:12.662 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:53:12.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:12.671 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:53:12.772 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:53:12.918 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:12.942 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:53:12.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:12.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:53:13.053 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:53:13.115 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:14.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:14 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:53:14.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:14 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:53:14.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:14 smithi179 ceph-mon[43975]: pgmap v1833: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 689 op/s; 347/192147 objects degraded (0.181%); 17324/192147 objects misplaced (9.016%); 15 MiB/s, 235 objects/s recovering 2024-02-09T17:53:14.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:14 smithi179 ceph-mon[43975]: osdmap e740: 12 total, 12 up, 12 in 2024-02-09T17:53:14.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:14 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:53:14.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:14 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:53:14.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:14 smithi162 ceph-mon[44304]: pgmap v1833: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 689 op/s; 347/192147 objects degraded (0.181%); 17324/192147 objects misplaced (9.016%); 15 MiB/s, 235 objects/s recovering 2024-02-09T17:53:14.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:14 smithi162 ceph-mon[44304]: osdmap e740: 12 total, 12 up, 12 in 2024-02-09T17:53:14.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:14 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:53:14.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:14 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:53:14.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:14 smithi099 ceph-mon[39420]: pgmap v1833: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 689 op/s; 347/192147 objects degraded (0.181%); 17324/192147 objects misplaced (9.016%); 15 MiB/s, 235 objects/s recovering 2024-02-09T17:53:14.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:14 smithi099 ceph-mon[39420]: osdmap e740: 12 total, 12 up, 12 in 2024-02-09T17:53:14.994 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:53:15.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:15.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:53:15.102 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:53:15.133 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:15.206 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:53:15.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:15.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:53:15.318 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:53:15.361 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:15.385 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:53:15.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:15.389 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:53:15.490 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:53:15.519 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:15.551 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:53:15.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:15.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:53:15.656 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:53:15.736 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:16.417 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:16 smithi099 ceph-mon[39420]: pgmap v1835: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 550 op/s; 439/191517 objects degraded (0.229%); 17114/191517 objects misplaced (8.936%); 7.6 MiB/s, 122 objects/s recovering 2024-02-09T17:53:16.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:16 smithi179 ceph-mon[43975]: pgmap v1835: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 550 op/s; 439/191517 objects degraded (0.229%); 17114/191517 objects misplaced (8.936%); 7.6 MiB/s, 122 objects/s recovering 2024-02-09T17:53:16.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:16 smithi162 ceph-mon[44304]: pgmap v1835: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 550 op/s; 439/191517 objects degraded (0.229%); 17114/191517 objects misplaced (8.936%); 7.6 MiB/s, 122 objects/s recovering 2024-02-09T17:53:16.427 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:53:16.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:16.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:53:16.535 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:53:16.568 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:16.607 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:53:16.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:16.614 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:53:16.715 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:53:16.760 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:16.793 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:53:16.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:16.796 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:53:16.855 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.4 to 1 (10000) 2024-02-09T17:53:16.897 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:53:16.982 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:17.067 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:53:17.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:17.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:53:17.173 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:53:17.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:17 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/742083729' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-09T17:53:17.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:17 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-09T17:53:17.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:17 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 439/191517 objects degraded (0.229%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-02-09T17:53:17.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:17 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-09T17:53:17.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:17 smithi099 ceph-mon[39420]: osdmap e741: 12 total, 12 up, 12 in 2024-02-09T17:53:17.258 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:17.305 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:53:17.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:17.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:53:17.416 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:53:17.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:17 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/742083729' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-09T17:53:17.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:17 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-09T17:53:17.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:17 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 439/191517 objects degraded (0.229%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-02-09T17:53:17.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:17 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-09T17:53:17.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:17 smithi179 ceph-mon[43975]: osdmap e741: 12 total, 12 up, 12 in 2024-02-09T17:53:17.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:17 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/742083729' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-09T17:53:17.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:17 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-09T17:53:17.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:17 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 439/191517 objects degraded (0.229%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-02-09T17:53:17.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:17 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-09T17:53:17.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:17 smithi162 ceph-mon[44304]: osdmap e741: 12 total, 12 up, 12 in 2024-02-09T17:53:17.480 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:17.815 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:53:17.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:17.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:53:17.848 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 5 1 2024-02-09T17:53:17.867 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:53:17.919 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:53:17.962 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:17.993 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:53:18.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:18.000 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:53:18.101 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:53:18.204 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:18.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:18 smithi179 ceph-mon[43975]: pgmap v1837: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 811 op/s; 599/183930 objects degraded (0.326%); 16461/183930 objects misplaced (8.950%); 9.1 MiB/s, 146 objects/s recovering 2024-02-09T17:53:18.425 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:18 smithi099 ceph-mon[39420]: pgmap v1837: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 811 op/s; 599/183930 objects degraded (0.326%); 16461/183930 objects misplaced (8.950%); 9.1 MiB/s, 146 objects/s recovering 2024-02-09T17:53:18.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:18 smithi162 ceph-mon[44304]: pgmap v1837: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 811 op/s; 599/183930 objects degraded (0.326%); 16461/183930 objects misplaced (8.950%); 9.1 MiB/s, 146 objects/s recovering 2024-02-09T17:53:18.438 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:53:18.439 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:53:18.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:18.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:53:18.549 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:53:18.636 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:19.229 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:19 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:53:19.229 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:19 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:53:19.229 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:19 smithi099 ceph-mon[39420]: osdmap e742: 12 total, 12 up, 12 in 2024-02-09T17:53:19.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:19 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:53:19.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:19 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:53:19.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:19 smithi179 ceph-mon[43975]: osdmap e742: 12 total, 12 up, 12 in 2024-02-09T17:53:19.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:19 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:53:19.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:19 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:53:19.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:19 smithi162 ceph-mon[44304]: osdmap e742: 12 total, 12 up, 12 in 2024-02-09T17:53:20.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:20 smithi179 ceph-mon[43975]: pgmap v1839: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 597 op/s; 717/181374 objects degraded (0.395%); 15808/181374 objects misplaced (8.716%); 9.6 MiB/s, 154 objects/s recovering 2024-02-09T17:53:20.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:20 smithi162 ceph-mon[44304]: pgmap v1839: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 597 op/s; 717/181374 objects degraded (0.395%); 15808/181374 objects misplaced (8.716%); 9.6 MiB/s, 154 objects/s recovering 2024-02-09T17:53:20.498 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:20 smithi099 ceph-mon[39420]: pgmap v1839: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 597 op/s; 717/181374 objects degraded (0.395%); 15808/181374 objects misplaced (8.716%); 9.6 MiB/s, 154 objects/s recovering 2024-02-09T17:53:20.705 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:53:20.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:20.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:53:20.814 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:53:20.837 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:53:20.838 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 20, 2024-02-09T17:53:20.838 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 600, 2024-02-09T17:53:20.838 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [ 2024-02-09T17:53:20.838 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.838 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25837.0:95790 6.d 6:b0c9024c:::benchmark_data_smithi179_126206_object95789:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e600)", 2024-02-09T17:53:20.839 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:46:46.008749+0000", 2024-02-09T17:53:20.839 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 392.83264939200001, 2024-02-09T17:53:20.839 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.02436664, 2024-02-09T17:53:20.839 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:53:20.839 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:53:20.839 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:53:20.840 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25837", 2024-02-09T17:53:20.840 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:53:20.840 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 95790 2024-02-09T17:53:20.840 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.840 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:53:20.841 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.841 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:53:20.841 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.008749+0000", 2024-02-09T17:53:20.841 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:53:20.841 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.841 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.842 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:53:20.842 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.008749+0000", 2024-02-09T17:53:20.842 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:53:20.842 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.842 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.842 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:53:20.843 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.008751+0000", 2024-02-09T17:53:20.843 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.66e-06 2024-02-09T17:53:20.843 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.843 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.843 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:53:20.844 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.008819+0000", 2024-02-09T17:53:20.844 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.8002000000000004e-05 2024-02-09T17:53:20.844 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.844 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.844 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:53:20.845 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.008836+0000", 2024-02-09T17:53:20.845 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.7635e-05 2024-02-09T17:53:20.845 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.845 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.845 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:53:20.845 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.008846+0000", 2024-02-09T17:53:20.846 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.5910000000000002e-06 2024-02-09T17:53:20.846 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.846 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.846 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:53:20.846 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.008928+0000", 2024-02-09T17:53:20.846 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.1951000000000002e-05 2024-02-09T17:53:20.847 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.847 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.847 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op must wait for map", 2024-02-09T17:53:20.847 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.008935+0000", 2024-02-09T17:53:20.847 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.3209999999999999e-06 2024-02-09T17:53:20.847 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.848 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.848 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:53:20.848 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.014882+0000", 2024-02-09T17:53:20.848 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0059469120000000004 2024-02-09T17:53:20.848 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.849 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.849 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:53:20.849 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.014893+0000", 2024-02-09T17:53:20.849 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0665999999999999e-05 2024-02-09T17:53:20.849 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.849 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.850 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:53:20.850 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.019586+0000", 2024-02-09T17:53:20.850 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.004693697 2024-02-09T17:53:20.850 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.850 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.850 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:53:20.851 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.022300+0000", 2024-02-09T17:53:20.851 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0027133790000000001 2024-02-09T17:53:20.851 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.851 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.851 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 5,10", 2024-02-09T17:53:20.851 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.022474+0000", 2024-02-09T17:53:20.852 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00017432999999999999 2024-02-09T17:53:20.852 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.852 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.852 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:53:20.852 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.032943+0000", 2024-02-09T17:53:20.853 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.010468682999999999 2024-02-09T17:53:20.853 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.853 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.853 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:53:20.853 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.032957+0000", 2024-02-09T17:53:20.853 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4732000000000001e-05 2024-02-09T17:53:20.853 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.854 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.854 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:53:20.854 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.033003+0000", 2024-02-09T17:53:20.854 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.5176999999999998e-05 2024-02-09T17:53:20.854 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.854 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.855 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:53:20.855 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.033079+0000", 2024-02-09T17:53:20.855 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.5944000000000003e-05 2024-02-09T17:53:20.855 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.855 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.855 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:53:20.856 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.033116+0000", 2024-02-09T17:53:20.856 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.6959999999999998e-05 2024-02-09T17:53:20.856 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:53:20.856 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:53:20.856 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:53:20.857 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.857 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.857 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25837.0:95791 6.d 6:bcc813c7:::benchmark_data_smithi179_126206_object95790:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e600)", 2024-02-09T17:53:20.857 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:46:46.008877+0000", 2024-02-09T17:53:20.857 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 392.83252124500001, 2024-02-09T17:53:20.857 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0295319190000001, 2024-02-09T17:53:20.858 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:53:20.858 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:53:20.858 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:53:20.858 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25837", 2024-02-09T17:53:20.858 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:53:20.858 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 95791 2024-02-09T17:53:20.859 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.859 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:53:20.859 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.859 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:53:20.859 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.008877+0000", 2024-02-09T17:53:20.859 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:53:20.860 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.860 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.860 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:53:20.860 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.008877+0000", 2024-02-09T17:53:20.860 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:53:20.860 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.861 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.861 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:53:20.861 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.008878+0000", 2024-02-09T17:53:20.861 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.4199999999999995e-07 2024-02-09T17:53:20.861 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.861 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.862 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:53:20.862 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.008910+0000", 2024-02-09T17:53:20.862 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.2728000000000002e-05 2024-02-09T17:53:20.862 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.862 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.862 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:53:20.863 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.008926+0000", 2024-02-09T17:53:20.863 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5146e-05 2024-02-09T17:53:20.863 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.863 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.863 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:53:20.863 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.008934+0000", 2024-02-09T17:53:20.864 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.3639999999999999e-06 2024-02-09T17:53:20.864 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.864 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.864 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:53:20.864 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.009005+0000", 2024-02-09T17:53:20.865 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.0797999999999995e-05 2024-02-09T17:53:20.865 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.865 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.865 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting_for_map not empty", 2024-02-09T17:53:20.865 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.009012+0000", 2024-02-09T17:53:20.865 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.883e-06 2024-02-09T17:53:20.866 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.866 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.866 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:53:20.866 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.014990+0000", 2024-02-09T17:53:20.866 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.005978751 2024-02-09T17:53:20.866 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.867 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.867 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:53:20.867 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.014999+0000", 2024-02-09T17:53:20.867 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.6980000000000004e-06 2024-02-09T17:53:20.867 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.867 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.868 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:53:20.868 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.026059+0000", 2024-02-09T17:53:20.868 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.011059572 2024-02-09T17:53:20.868 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.868 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.868 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:53:20.869 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.026213+0000", 2024-02-09T17:53:20.869 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00015468799999999999 2024-02-09T17:53:20.869 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.869 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.869 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 5,10", 2024-02-09T17:53:20.869 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.026378+0000", 2024-02-09T17:53:20.870 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00016482000000000001 2024-02-09T17:53:20.870 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.870 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.870 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:53:20.870 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.033200+0000", 2024-02-09T17:53:20.870 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0068217970000000001 2024-02-09T17:53:20.871 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.871 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.871 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:53:20.871 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.035887+0000", 2024-02-09T17:53:20.871 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0026866239999999999 2024-02-09T17:53:20.872 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.872 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.872 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:53:20.872 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.038337+0000", 2024-02-09T17:53:20.872 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00245036 2024-02-09T17:53:20.872 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.873 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.873 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:53:20.873 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.038374+0000", 2024-02-09T17:53:20.873 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.7583000000000003e-05 2024-02-09T17:53:20.873 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.873 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.874 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:53:20.874 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.038409+0000", 2024-02-09T17:53:20.874 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.4465000000000003e-05 2024-02-09T17:53:20.874 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:53:20.874 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:53:20.875 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:53:20.875 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.875 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.875 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25837.0:95794 6.d 6:b9be935f:::benchmark_data_smithi179_126206_object95793:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e600)", 2024-02-09T17:53:20.875 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:46:46.013038+0000", 2024-02-09T17:53:20.875 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 392.82836057999998, 2024-02-09T17:53:20.876 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0359706259999999, 2024-02-09T17:53:20.876 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:53:20.876 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:53:20.876 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:53:20.876 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25837", 2024-02-09T17:53:20.876 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:53:20.877 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 95794 2024-02-09T17:53:20.877 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.877 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:53:20.877 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.877 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:53:20.877 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.013038+0000", 2024-02-09T17:53:20.878 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:53:20.878 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.878 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.878 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:53:20.878 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.013038+0000", 2024-02-09T17:53:20.878 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:53:20.879 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.879 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.879 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:53:20.879 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.013039+0000", 2024-02-09T17:53:20.879 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.697e-06 2024-02-09T17:53:20.879 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.880 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.880 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:53:20.880 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.013107+0000", 2024-02-09T17:53:20.880 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.7761000000000004e-05 2024-02-09T17:53:20.880 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.881 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.881 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:53:20.881 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.013122+0000", 2024-02-09T17:53:20.881 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4938e-05 2024-02-09T17:53:20.881 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.881 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.882 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:53:20.882 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.013131+0000", 2024-02-09T17:53:20.882 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.38e-06 2024-02-09T17:53:20.882 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.882 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.882 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:53:20.883 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.015124+0000", 2024-02-09T17:53:20.883 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0019929679999999999 2024-02-09T17:53:20.883 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.883 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.883 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:53:20.883 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.015134+0000", 2024-02-09T17:53:20.884 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.9150000000000008e-06 2024-02-09T17:53:20.884 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.884 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.884 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:53:20.884 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.029797+0000", 2024-02-09T17:53:20.884 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0146626889999999 2024-02-09T17:53:20.885 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.885 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.885 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:53:20.885 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.029951+0000", 2024-02-09T17:53:20.885 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00015371699999999999 2024-02-09T17:53:20.885 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.886 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.886 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 5,10", 2024-02-09T17:53:20.886 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.030110+0000", 2024-02-09T17:53:20.886 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00015944399999999999 2024-02-09T17:53:20.886 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.886 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.887 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:53:20.887 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.035893+0000", 2024-02-09T17:53:20.887 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0057831139999999998 2024-02-09T17:53:20.887 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.887 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.887 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:53:20.888 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.038453+0000", 2024-02-09T17:53:20.888 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0025593090000000001 2024-02-09T17:53:20.888 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.888 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.888 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:53:20.888 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.048925+0000", 2024-02-09T17:53:20.889 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.010472162 2024-02-09T17:53:20.889 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.889 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.889 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:53:20.889 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.048970+0000", 2024-02-09T17:53:20.889 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.4829000000000001e-05 2024-02-09T17:53:20.890 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.890 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.890 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:53:20.890 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.049008+0000", 2024-02-09T17:53:20.890 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.8702999999999999e-05 2024-02-09T17:53:20.891 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:53:20.891 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:53:20.891 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:53:20.891 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.891 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.891 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25837.0:95810 6.d 6:b4935c8d:::benchmark_data_smithi179_126206_object95809:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e600)", 2024-02-09T17:53:20.892 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:46:46.037705+0000", 2024-02-09T17:53:20.892 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 392.80369310200001, 2024-02-09T17:53:20.892 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0114070319999999, 2024-02-09T17:53:20.892 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:53:20.892 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:53:20.892 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:53:20.893 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25837", 2024-02-09T17:53:20.893 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:53:20.893 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 95810 2024-02-09T17:53:20.893 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.893 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:53:20.893 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.894 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:53:20.894 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.037705+0000", 2024-02-09T17:53:20.894 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:53:20.894 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.894 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.894 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:53:20.895 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.037705+0000", 2024-02-09T17:53:20.895 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:53:20.895 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.895 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.895 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:53:20.895 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.037706+0000", 2024-02-09T17:53:20.896 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3399999999999999e-07 2024-02-09T17:53:20.896 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.896 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.896 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:53:20.896 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.037760+0000", 2024-02-09T17:53:20.896 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.4540999999999998e-05 2024-02-09T17:53:20.897 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.897 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.897 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:53:20.897 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.037780+0000", 2024-02-09T17:53:20.897 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.9847000000000002e-05 2024-02-09T17:53:20.897 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.898 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.898 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:53:20.898 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.037787+0000", 2024-02-09T17:53:20.898 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.2810000000000003e-06 2024-02-09T17:53:20.898 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.898 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.899 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:53:20.899 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.037838+0000", 2024-02-09T17:53:20.899 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.0141e-05 2024-02-09T17:53:20.899 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.899 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.899 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:53:20.900 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.037845+0000", 2024-02-09T17:53:20.900 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.7360000000000001e-06 2024-02-09T17:53:20.900 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.900 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.900 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:53:20.900 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.033235+0000", 2024-02-09T17:53:20.901 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.99538948500000002 2024-02-09T17:53:20.901 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.901 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.901 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:53:20.901 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.033338+0000", 2024-02-09T17:53:20.901 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00010342500000000001 2024-02-09T17:53:20.901 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.901 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.901 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 5,10", 2024-02-09T17:53:20.901 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.033443+0000", 2024-02-09T17:53:20.901 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000104813 2024-02-09T17:53:20.901 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.901 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.902 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:53:20.902 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.041791+0000", 2024-02-09T17:53:20.902 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0083478760000000006 2024-02-09T17:53:20.902 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.902 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.902 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:53:20.902 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.045338+0000", 2024-02-09T17:53:20.902 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.003547185 2024-02-09T17:53:20.902 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.902 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.902 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:53:20.902 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.049049+0000", 2024-02-09T17:53:20.902 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0037113010000000002 2024-02-09T17:53:20.902 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.902 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.903 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:53:20.903 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.049080+0000", 2024-02-09T17:53:20.903 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.1050000000000003e-05 2024-02-09T17:53:20.903 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.903 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.903 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:53:20.903 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.049112+0000", 2024-02-09T17:53:20.903 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.1816999999999997e-05 2024-02-09T17:53:20.903 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:53:20.903 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:53:20.903 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:53:20.903 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.903 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.903 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25837.0:95811 6.d 6:be94929c:::benchmark_data_smithi179_126206_object95810:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e600)", 2024-02-09T17:53:20.903 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:46:46.037906+0000", 2024-02-09T17:53:20.904 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 392.80349193199999, 2024-02-09T17:53:20.904 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.011378764, 2024-02-09T17:53:20.904 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:53:20.904 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:53:20.904 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:53:20.904 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25837", 2024-02-09T17:53:20.904 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:53:20.904 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 95811 2024-02-09T17:53:20.904 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.904 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:53:20.904 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.904 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:53:20.904 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.037906+0000", 2024-02-09T17:53:20.904 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:53:20.904 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.905 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.905 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:53:20.905 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.037906+0000", 2024-02-09T17:53:20.905 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:53:20.905 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.905 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.905 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:53:20.905 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.037916+0000", 2024-02-09T17:53:20.905 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.5459999999999997e-06 2024-02-09T17:53:20.905 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.905 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.905 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:53:20.905 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.037977+0000", 2024-02-09T17:53:20.905 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.1408999999999994e-05 2024-02-09T17:53:20.905 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.906 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.906 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:53:20.906 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.037986+0000", 2024-02-09T17:53:20.906 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.9050000000000003e-06 2024-02-09T17:53:20.906 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.906 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.906 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:53:20.906 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.037993+0000", 2024-02-09T17:53:20.906 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.7530000000000004e-06 2024-02-09T17:53:20.906 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.906 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.906 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:53:20.906 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.038033+0000", 2024-02-09T17:53:20.906 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9876000000000002e-05 2024-02-09T17:53:20.906 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.907 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.907 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:53:20.907 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.038037+0000", 2024-02-09T17:53:20.907 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.566e-06 2024-02-09T17:53:20.907 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.907 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.907 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:53:20.907 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.035925+0000", 2024-02-09T17:53:20.907 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.99788725099999998 2024-02-09T17:53:20.907 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.907 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.907 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:53:20.907 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.036014+0000", 2024-02-09T17:53:20.907 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.8861000000000002e-05 2024-02-09T17:53:20.907 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.908 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.908 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 5,10", 2024-02-09T17:53:20.908 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.036106+0000", 2024-02-09T17:53:20.908 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.2325999999999997e-05 2024-02-09T17:53:20.908 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.908 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.908 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:53:20.908 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.045251+0000", 2024-02-09T17:53:20.908 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0091448120000000004 2024-02-09T17:53:20.908 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.908 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.908 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:53:20.908 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.049154+0000", 2024-02-09T17:53:20.908 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0039037519999999999 2024-02-09T17:53:20.909 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.909 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.909 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:53:20.909 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.049187+0000", 2024-02-09T17:53:20.909 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.2295000000000002e-05 2024-02-09T17:53:20.909 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.909 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.909 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:53:20.909 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.049249+0000", 2024-02-09T17:53:20.909 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.2639999999999997e-05 2024-02-09T17:53:20.909 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.909 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.909 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:53:20.909 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.049285+0000", 2024-02-09T17:53:20.909 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.5772000000000002e-05 2024-02-09T17:53:20.909 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:53:20.910 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:53:20.910 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:53:20.910 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.910 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.910 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25837.0:95812 6.d 6:b4eddf87:::benchmark_data_smithi179_126206_object95811:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e600)", 2024-02-09T17:53:20.910 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:46:46.038029+0000", 2024-02-09T17:53:20.910 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 392.80336911799998, 2024-02-09T17:53:20.910 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0149304299999999, 2024-02-09T17:53:20.910 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:53:20.910 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:53:20.910 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:53:20.910 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25837", 2024-02-09T17:53:20.910 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:53:20.910 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 95812 2024-02-09T17:53:20.910 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.911 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:53:20.911 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.911 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:53:20.911 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.038029+0000", 2024-02-09T17:53:20.911 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:53:20.911 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.911 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.911 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:53:20.911 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.038029+0000", 2024-02-09T17:53:20.911 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:53:20.911 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.911 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.911 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:53:20.911 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.038030+0000", 2024-02-09T17:53:20.911 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.4300000000000003e-07 2024-02-09T17:53:20.912 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.912 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.912 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:53:20.912 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.038100+0000", 2024-02-09T17:53:20.912 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.0368000000000001e-05 2024-02-09T17:53:20.912 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.912 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.912 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:53:20.912 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.038108+0000", 2024-02-09T17:53:20.912 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.3000000000000002e-06 2024-02-09T17:53:20.912 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.912 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.912 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:53:20.912 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.038114+0000", 2024-02-09T17:53:20.912 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.9769999999999997e-06 2024-02-09T17:53:20.913 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.913 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.913 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:53:20.913 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.038175+0000", 2024-02-09T17:53:20.913 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.1043999999999994e-05 2024-02-09T17:53:20.913 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.913 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.913 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:53:20.913 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.038187+0000", 2024-02-09T17:53:20.913 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1015999999999999e-05 2024-02-09T17:53:20.913 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.913 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.913 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:53:20.913 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.038505+0000", 2024-02-09T17:53:20.914 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0003183099999999 2024-02-09T17:53:20.914 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.914 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.914 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:53:20.914 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.038631+0000", 2024-02-09T17:53:20.914 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012646599999999999 2024-02-09T17:53:20.914 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.914 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.914 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 5,10", 2024-02-09T17:53:20.914 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.038777+0000", 2024-02-09T17:53:20.914 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014590100000000001 2024-02-09T17:53:20.914 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.914 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.914 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:53:20.914 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.048883+0000", 2024-02-09T17:53:20.915 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.010105563999999999 2024-02-09T17:53:20.915 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.915 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.915 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:53:20.915 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.049641+0000", 2024-02-09T17:53:20.915 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00075840100000000002 2024-02-09T17:53:20.915 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.915 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.915 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:53:20.915 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.052863+0000", 2024-02-09T17:53:20.915 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0032220249999999999 2024-02-09T17:53:20.916 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.916 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.916 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:53:20.916 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.052916+0000", 2024-02-09T17:53:20.916 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3186000000000002e-05 2024-02-09T17:53:20.916 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.916 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.916 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:53:20.916 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.052960+0000", 2024-02-09T17:53:20.916 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.3229000000000002e-05 2024-02-09T17:53:20.916 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:53:20.916 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:53:20.917 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:53:20.917 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.917 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.917 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25837.0:203763 6.4 6:2535f01d:::benchmark_data_smithi179_126206_object73382:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e638)", 2024-02-09T17:53:20.917 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:48:48.769263+0000", 2024-02-09T17:53:20.917 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 270.07213568100002, 2024-02-09T17:53:20.917 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.221425328, 2024-02-09T17:53:20.917 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:53:20.917 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:53:20.917 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:53:20.917 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25837", 2024-02-09T17:53:20.917 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:53:20.917 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 203763 2024-02-09T17:53:20.917 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.917 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:53:20.917 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.918 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:53:20.918 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.769263+0000", 2024-02-09T17:53:20.918 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:53:20.918 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.918 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.918 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:53:20.918 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.769263+0000", 2024-02-09T17:53:20.918 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:53:20.918 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.918 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.918 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:53:20.918 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.769264+0000", 2024-02-09T17:53:20.918 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.751e-06 2024-02-09T17:53:20.918 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.918 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.919 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:53:20.919 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.769270+0000", 2024-02-09T17:53:20.919 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.5099999999999998e-06 2024-02-09T17:53:20.919 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.919 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.919 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:53:20.919 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.769280+0000", 2024-02-09T17:53:20.919 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.7790000000000007e-06 2024-02-09T17:53:20.919 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.919 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.919 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:53:20.919 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.769286+0000", 2024-02-09T17:53:20.919 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.2740000000000004e-06 2024-02-09T17:53:20.919 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.919 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.919 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:53:20.920 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.769333+0000", 2024-02-09T17:53:20.920 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.7534999999999998e-05 2024-02-09T17:53:20.920 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.920 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.920 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:53:20.920 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.769453+0000", 2024-02-09T17:53:20.920 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000119601 2024-02-09T17:53:20.920 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.920 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.920 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,5", 2024-02-09T17:53:20.920 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.769565+0000", 2024-02-09T17:53:20.920 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000111505 2024-02-09T17:53:20.920 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.920 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.920 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:53:20.921 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.790278+0000", 2024-02-09T17:53:20.921 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.020713809999999999 2024-02-09T17:53:20.921 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.921 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.921 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:53:20.921 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.811239+0000", 2024-02-09T17:53:20.921 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.020960138999999999 2024-02-09T17:53:20.921 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.921 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.921 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:53:20.921 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.990621+0000", 2024-02-09T17:53:20.921 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.17938268199999999 2024-02-09T17:53:20.921 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.921 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.921 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:53:20.922 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.990660+0000", 2024-02-09T17:53:20.922 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9014999999999998e-05 2024-02-09T17:53:20.922 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.922 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.922 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:53:20.922 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.990688+0000", 2024-02-09T17:53:20.922 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.7727000000000001e-05 2024-02-09T17:53:20.922 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:53:20.922 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:53:20.922 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:53:20.922 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.922 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.922 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25837.0:203764 6.4 6:282276a4:::benchmark_data_smithi179_126206_object73383:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e638)", 2024-02-09T17:53:20.922 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:48:48.769301+0000", 2024-02-09T17:53:20.922 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 270.07209689199999, 2024-02-09T17:53:20.923 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.22147773000000001, 2024-02-09T17:53:20.923 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:53:20.923 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:53:20.923 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:53:20.923 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25837", 2024-02-09T17:53:20.923 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:53:20.923 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 203764 2024-02-09T17:53:20.923 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.923 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:53:20.923 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.923 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:53:20.923 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.769301+0000", 2024-02-09T17:53:20.923 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:53:20.923 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.923 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.923 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:53:20.924 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.769301+0000", 2024-02-09T17:53:20.924 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:53:20.924 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.924 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.924 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:53:20.924 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.769302+0000", 2024-02-09T17:53:20.924 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.5900000000000002e-07 2024-02-09T17:53:20.924 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.924 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.924 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:53:20.924 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.769304+0000", 2024-02-09T17:53:20.924 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.5639999999999999e-06 2024-02-09T17:53:20.924 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.924 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.924 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:53:20.925 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.769311+0000", 2024-02-09T17:53:20.925 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.6950000000000003e-06 2024-02-09T17:53:20.925 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.925 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.925 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:53:20.925 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.769317+0000", 2024-02-09T17:53:20.925 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.4219999999999999e-06 2024-02-09T17:53:20.925 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.925 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.925 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:53:20.925 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.774400+0000", 2024-02-09T17:53:20.925 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0050838890000000003 2024-02-09T17:53:20.925 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.925 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.925 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:53:20.926 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.774528+0000", 2024-02-09T17:53:20.926 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012776199999999999 2024-02-09T17:53:20.926 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.926 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.926 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,5", 2024-02-09T17:53:20.926 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.774657+0000", 2024-02-09T17:53:20.926 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000129123 2024-02-09T17:53:20.926 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.926 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.926 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:53:20.926 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.790498+0000", 2024-02-09T17:53:20.926 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.015840559000000001 2024-02-09T17:53:20.926 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.926 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.926 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:53:20.926 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.811292+0000", 2024-02-09T17:53:20.927 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.020793736 2024-02-09T17:53:20.927 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.927 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.927 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:53:20.927 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.990722+0000", 2024-02-09T17:53:20.927 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.179430535 2024-02-09T17:53:20.927 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.927 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.927 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:53:20.927 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.990755+0000", 2024-02-09T17:53:20.927 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.2574999999999999e-05 2024-02-09T17:53:20.927 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.927 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.927 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:53:20.927 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.990779+0000", 2024-02-09T17:53:20.928 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.4411000000000001e-05 2024-02-09T17:53:20.928 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:53:20.928 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:53:20.928 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:53:20.928 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.928 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.928 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25837.0:203767 6.4 6:2ef8b14c:::benchmark_data_smithi179_126206_object73386:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e638)", 2024-02-09T17:53:20.928 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:48:48.771761+0000", 2024-02-09T17:53:20.928 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 270.06963712200002, 2024-02-09T17:53:20.928 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.229900145, 2024-02-09T17:53:20.928 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:53:20.928 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:53:20.928 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:53:20.928 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25837", 2024-02-09T17:53:20.929 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:53:20.929 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 203767 2024-02-09T17:53:20.929 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.929 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:53:20.929 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.929 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:53:20.929 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.771761+0000", 2024-02-09T17:53:20.929 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:53:20.929 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.929 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.929 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:53:20.929 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.771761+0000", 2024-02-09T17:53:20.929 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:53:20.929 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.929 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.929 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:53:20.930 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.771762+0000", 2024-02-09T17:53:20.930 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1960000000000001e-06 2024-02-09T17:53:20.930 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.930 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.930 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:53:20.930 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.771767+0000", 2024-02-09T17:53:20.930 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.7090000000000001e-06 2024-02-09T17:53:20.930 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.930 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.930 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:53:20.930 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.771780+0000", 2024-02-09T17:53:20.930 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2500999999999999e-05 2024-02-09T17:53:20.930 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.930 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.931 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:53:20.931 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.771788+0000", 2024-02-09T17:53:20.931 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.2560000000000002e-06 2024-02-09T17:53:20.931 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.931 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.931 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:53:20.931 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.784186+0000", 2024-02-09T17:53:20.931 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.012397874 2024-02-09T17:53:20.931 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.931 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.931 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:53:20.931 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.784322+0000", 2024-02-09T17:53:20.931 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000136038 2024-02-09T17:53:20.931 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.931 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.931 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,5", 2024-02-09T17:53:20.932 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.784437+0000", 2024-02-09T17:53:20.932 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00011572400000000001 2024-02-09T17:53:20.932 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.932 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.932 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:53:20.932 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.801411+0000", 2024-02-09T17:53:20.932 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.016973722 2024-02-09T17:53:20.932 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.932 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.932 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:53:20.932 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.811405+0000", 2024-02-09T17:53:20.932 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0099935340000000001 2024-02-09T17:53:20.932 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.932 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.932 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:53:20.933 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:49.001602+0000", 2024-02-09T17:53:20.933 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.19019723099999999 2024-02-09T17:53:20.933 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.933 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.933 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:53:20.933 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:49.001637+0000", 2024-02-09T17:53:20.933 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.5113000000000001e-05 2024-02-09T17:53:20.933 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.933 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.933 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:53:20.933 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:49.001661+0000", 2024-02-09T17:53:20.933 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.4247e-05 2024-02-09T17:53:20.933 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:53:20.933 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:53:20.933 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:53:20.934 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.934 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.934 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25837.0:203782 6.4 6:29d9fa0d:::benchmark_data_smithi179_126206_object73401:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e638)", 2024-02-09T17:53:20.934 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:48:48.791451+0000", 2024-02-09T17:53:20.934 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 270.04994766599998, 2024-02-09T17:53:20.934 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.23181099899999999, 2024-02-09T17:53:20.934 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:53:20.934 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:53:20.934 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:53:20.934 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25837", 2024-02-09T17:53:20.934 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:53:20.934 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 203782 2024-02-09T17:53:20.934 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.934 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:53:20.935 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.935 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:53:20.935 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.791451+0000", 2024-02-09T17:53:20.935 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:53:20.935 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.935 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.935 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:53:20.935 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.791451+0000", 2024-02-09T17:53:20.935 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:53:20.935 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.935 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.935 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:53:20.936 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.791452+0000", 2024-02-09T17:53:20.936 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5200000000000001e-06 2024-02-09T17:53:20.936 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.936 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.936 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:53:20.936 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.791457+0000", 2024-02-09T17:53:20.936 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.9049999999999996e-06 2024-02-09T17:53:20.936 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.936 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.936 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:53:20.936 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.791470+0000", 2024-02-09T17:53:20.936 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2860999999999999e-05 2024-02-09T17:53:20.936 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.936 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.936 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:53:20.937 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.791479+0000", 2024-02-09T17:53:20.937 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.8589999999999994e-06 2024-02-09T17:53:20.937 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.937 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.937 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:53:20.937 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.791534+0000", 2024-02-09T17:53:20.937 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.5389000000000001e-05 2024-02-09T17:53:20.937 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.937 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.937 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:53:20.937 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.791658+0000", 2024-02-09T17:53:20.937 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000123638 2024-02-09T17:53:20.937 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.937 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.937 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,5", 2024-02-09T17:53:20.937 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.791780+0000", 2024-02-09T17:53:20.938 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000121746 2024-02-09T17:53:20.938 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.938 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.938 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:53:20.938 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.801416+0000", 2024-02-09T17:53:20.938 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0096366400000000001 2024-02-09T17:53:20.938 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.938 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.938 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:53:20.938 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.811443+0000", 2024-02-09T17:53:20.938 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.010027230999999999 2024-02-09T17:53:20.938 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.938 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.938 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:53:20.938 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:49.023163+0000", 2024-02-09T17:53:20.938 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.211719871 2024-02-09T17:53:20.939 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.939 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.939 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:53:20.939 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:49.023220+0000", 2024-02-09T17:53:20.939 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6338999999999997e-05 2024-02-09T17:53:20.939 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.939 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.939 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:53:20.939 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:49.023262+0000", 2024-02-09T17:53:20.939 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.1999999999999998e-05 2024-02-09T17:53:20.939 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:53:20.939 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:53:20.939 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:53:20.939 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.939 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.940 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25837.0:203783 6.4 6:2118d7a2:::benchmark_data_smithi179_126206_object73402:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e638)", 2024-02-09T17:53:20.940 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:48:48.791775+0000", 2024-02-09T17:53:20.940 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 270.04962335099998, 2024-02-09T17:53:20.940 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.237107436, 2024-02-09T17:53:20.940 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:53:20.940 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:53:20.940 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:53:20.940 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25837", 2024-02-09T17:53:20.940 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:53:20.940 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 203783 2024-02-09T17:53:20.940 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.940 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:53:20.940 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.940 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:53:20.940 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.791775+0000", 2024-02-09T17:53:20.941 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:53:20.941 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.941 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.941 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:53:20.941 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.791775+0000", 2024-02-09T17:53:20.941 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:53:20.941 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.941 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.941 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:53:20.941 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.791776+0000", 2024-02-09T17:53:20.941 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.8500000000000001e-07 2024-02-09T17:53:20.941 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.941 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.941 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:53:20.941 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.791779+0000", 2024-02-09T17:53:20.942 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.2440000000000002e-06 2024-02-09T17:53:20.942 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.942 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.942 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:53:20.942 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.791794+0000", 2024-02-09T17:53:20.942 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4722999999999999e-05 2024-02-09T17:53:20.942 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.942 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.942 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:53:20.942 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.791800+0000", 2024-02-09T17:53:20.942 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.8519999999999998e-06 2024-02-09T17:53:20.942 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.942 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.942 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:53:20.942 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.797008+0000", 2024-02-09T17:53:20.943 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0052077060000000003 2024-02-09T17:53:20.943 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.943 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.943 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:53:20.943 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.797134+0000", 2024-02-09T17:53:20.943 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012541700000000001 2024-02-09T17:53:20.943 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.943 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.943 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,5", 2024-02-09T17:53:20.943 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.797259+0000", 2024-02-09T17:53:20.943 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012533999999999999 2024-02-09T17:53:20.943 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.943 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.943 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:53:20.943 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.802889+0000", 2024-02-09T17:53:20.943 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0056301090000000003 2024-02-09T17:53:20.944 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.944 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.944 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:53:20.944 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.811505+0000", 2024-02-09T17:53:20.944 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0086157820000000006 2024-02-09T17:53:20.944 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.944 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.944 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:53:20.944 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:49.028781+0000", 2024-02-09T17:53:20.944 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.21727664999999999 2024-02-09T17:53:20.944 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.944 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.944 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:53:20.944 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:49.028839+0000", 2024-02-09T17:53:20.944 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7933999999999998e-05 2024-02-09T17:53:20.945 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.945 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.945 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:53:20.945 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:49.028882+0000", 2024-02-09T17:53:20.945 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2994000000000002e-05 2024-02-09T17:53:20.945 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:53:20.945 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:53:20.945 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:53:20.945 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.945 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.945 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25837.0:203801 6.4 6:217ce34f:::benchmark_data_smithi179_126206_object73420:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e638)", 2024-02-09T17:53:20.945 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:48:48.813837+0000", 2024-02-09T17:53:20.945 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 270.027561279, 2024-02-09T17:53:20.945 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.215159932, 2024-02-09T17:53:20.945 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:53:20.946 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:53:20.946 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:53:20.946 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25837", 2024-02-09T17:53:20.946 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:53:20.946 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 203801 2024-02-09T17:53:20.946 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.946 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:53:20.946 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.946 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:53:20.946 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.813837+0000", 2024-02-09T17:53:20.946 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:53:20.946 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.946 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.946 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:53:20.946 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.813837+0000", 2024-02-09T17:53:20.946 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:53:20.947 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.947 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.947 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:53:20.947 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.813838+0000", 2024-02-09T17:53:20.947 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.077e-06 2024-02-09T17:53:20.947 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.947 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.947 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:53:20.947 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.813842+0000", 2024-02-09T17:53:20.947 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2259999999999999e-06 2024-02-09T17:53:20.947 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.947 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.947 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:53:20.947 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.813853+0000", 2024-02-09T17:53:20.948 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0304e-05 2024-02-09T17:53:20.948 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.948 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.948 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:53:20.948 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.813863+0000", 2024-02-09T17:53:20.948 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.049e-05 2024-02-09T17:53:20.948 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.948 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.948 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:53:20.948 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.813917+0000", 2024-02-09T17:53:20.948 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3625000000000002e-05 2024-02-09T17:53:20.948 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.948 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.948 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:53:20.948 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.814067+0000", 2024-02-09T17:53:20.949 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00015053700000000001 2024-02-09T17:53:20.949 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.949 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.949 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,5", 2024-02-09T17:53:20.949 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.814191+0000", 2024-02-09T17:53:20.949 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012389500000000001 2024-02-09T17:53:20.949 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.949 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.949 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:53:20.949 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.819437+0000", 2024-02-09T17:53:20.949 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0052463240000000001 2024-02-09T17:53:20.949 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.949 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.949 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:53:20.949 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.819577+0000", 2024-02-09T17:53:20.950 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013955700000000001 2024-02-09T17:53:20.950 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.950 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.950 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:53:20.950 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:49.028918+0000", 2024-02-09T17:53:20.950 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20934101599999999 2024-02-09T17:53:20.950 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.950 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.950 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:53:20.950 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:49.028964+0000", 2024-02-09T17:53:20.950 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.6236999999999998e-05 2024-02-09T17:53:20.950 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.950 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.950 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:53:20.950 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:49.028997+0000", 2024-02-09T17:53:20.950 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.2644000000000001e-05 2024-02-09T17:53:20.951 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:53:20.951 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:53:20.951 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:53:20.951 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.951 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.951 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25837.0:203816 6.4 6:22e11e42:::benchmark_data_smithi179_126206_object73435:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e638)", 2024-02-09T17:53:20.951 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:48:48.836412+0000", 2024-02-09T17:53:20.951 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 270.004986583, 2024-02-09T17:53:20.951 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.19269656800000001, 2024-02-09T17:53:20.951 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:53:20.951 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:53:20.951 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:53:20.951 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25837", 2024-02-09T17:53:20.951 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:53:20.951 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 203816 2024-02-09T17:53:20.951 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.952 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:53:20.952 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.952 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:53:20.952 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.836412+0000", 2024-02-09T17:53:20.952 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:53:20.952 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.952 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.952 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:53:20.952 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.836412+0000", 2024-02-09T17:53:20.952 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:53:20.952 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.953 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.953 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:53:20.953 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.836412+0000", 2024-02-09T17:53:20.953 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.0200000000000002e-07 2024-02-09T17:53:20.953 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.953 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.953 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:53:20.953 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.836415+0000", 2024-02-09T17:53:20.953 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.3700000000000002e-06 2024-02-09T17:53:20.953 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.953 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.953 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:53:20.954 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.836420+0000", 2024-02-09T17:53:20.954 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3900000000000001e-06 2024-02-09T17:53:20.954 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.954 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.954 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:53:20.954 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.836424+0000", 2024-02-09T17:53:20.954 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.805e-06 2024-02-09T17:53:20.954 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.954 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.954 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:53:20.954 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.836475+0000", 2024-02-09T17:53:20.954 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1493000000000003e-05 2024-02-09T17:53:20.954 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.954 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.955 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:53:20.955 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.836585+0000", 2024-02-09T17:53:20.955 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00010932199999999999 2024-02-09T17:53:20.955 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.955 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.955 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,5", 2024-02-09T17:53:20.955 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.836696+0000", 2024-02-09T17:53:20.955 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000111456 2024-02-09T17:53:20.955 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.955 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.955 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:53:20.955 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.844807+0000", 2024-02-09T17:53:20.955 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0081107969999999995 2024-02-09T17:53:20.955 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.955 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.956 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:53:20.956 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.853708+0000", 2024-02-09T17:53:20.956 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0089015420000000001 2024-02-09T17:53:20.956 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.956 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.956 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:53:20.956 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:49.029030+0000", 2024-02-09T17:53:20.956 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.17532159899999999 2024-02-09T17:53:20.956 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.956 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.956 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:53:20.956 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:49.029074+0000", 2024-02-09T17:53:20.956 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.4195e-05 2024-02-09T17:53:20.956 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.957 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.957 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:53:20.957 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:49.029108+0000", 2024-02-09T17:53:20.957 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.4097000000000002e-05 2024-02-09T17:53:20.957 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:53:20.957 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:53:20.957 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:53:20.957 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.957 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.957 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25837.0:236091 6.4 6:2309183f:::benchmark_data_smithi179_126206_object105710:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e650)", 2024-02-09T17:53:20.957 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:49:27.986674+0000", 2024-02-09T17:53:20.957 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 230.854724105, 2024-02-09T17:53:20.957 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.177845955, 2024-02-09T17:53:20.957 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:53:20.957 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:53:20.957 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:53:20.958 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25837", 2024-02-09T17:53:20.958 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:53:20.958 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 236091 2024-02-09T17:53:20.958 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.958 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:53:20.958 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.958 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:53:20.958 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:27.986674+0000", 2024-02-09T17:53:20.958 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:53:20.958 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.958 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.958 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:53:20.958 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:27.986674+0000", 2024-02-09T17:53:20.958 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:53:20.958 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.959 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.959 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:53:20.959 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:27.986675+0000", 2024-02-09T17:53:20.959 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.2699999999999998e-07 2024-02-09T17:53:20.959 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.959 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.959 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:53:20.959 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:27.986680+0000", 2024-02-09T17:53:20.959 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.7269999999999998e-06 2024-02-09T17:53:20.959 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.959 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.959 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:53:20.959 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:27.986690+0000", 2024-02-09T17:53:20.959 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0538e-05 2024-02-09T17:53:20.959 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.959 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.960 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:53:20.960 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:27.986698+0000", 2024-02-09T17:53:20.960 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.5610000000000001e-06 2024-02-09T17:53:20.960 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.960 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.960 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:53:20.960 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:27.991656+0000", 2024-02-09T17:53:20.960 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0049586140000000001 2024-02-09T17:53:20.960 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.960 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.960 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:53:20.960 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:27.991783+0000", 2024-02-09T17:53:20.960 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012684400000000001 2024-02-09T17:53:20.960 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.960 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.961 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,5", 2024-02-09T17:53:20.961 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:27.991926+0000", 2024-02-09T17:53:20.961 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000142502 2024-02-09T17:53:20.961 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.961 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.961 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:53:20.961 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:28.007772+0000", 2024-02-09T17:53:20.961 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.015846352000000001 2024-02-09T17:53:20.961 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.961 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.961 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:53:20.961 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:28.007816+0000", 2024-02-09T17:53:20.961 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.422e-05 2024-02-09T17:53:20.961 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.961 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.962 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:53:20.962 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:28.164425+0000", 2024-02-09T17:53:20.962 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.156608475 2024-02-09T17:53:20.962 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.962 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.962 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:53:20.962 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:28.164482+0000", 2024-02-09T17:53:20.962 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7247999999999998e-05 2024-02-09T17:53:20.962 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.962 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.962 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:53:20.962 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:28.164520+0000", 2024-02-09T17:53:20.962 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.8046999999999997e-05 2024-02-09T17:53:20.962 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:53:20.962 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:53:20.962 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:53:20.963 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.963 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.963 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25837.0:236094 6.4 6:247164a1:::benchmark_data_smithi179_126206_object105713:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e650)", 2024-02-09T17:53:20.963 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:49:27.987229+0000", 2024-02-09T17:53:20.963 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 230.85416965600001, 2024-02-09T17:53:20.963 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.17744489499999999, 2024-02-09T17:53:20.963 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:53:20.963 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:53:20.963 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:53:20.963 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25837", 2024-02-09T17:53:20.963 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:53:20.963 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 236094 2024-02-09T17:53:20.963 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.963 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:53:20.963 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.964 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:53:20.964 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:27.987229+0000", 2024-02-09T17:53:20.964 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:53:20.964 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.964 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.964 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:53:20.964 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:27.987229+0000", 2024-02-09T17:53:20.964 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:53:20.964 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.964 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.964 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:53:20.964 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:27.987229+0000", 2024-02-09T17:53:20.964 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.7400000000000003e-07 2024-02-09T17:53:20.964 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.964 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.964 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:53:20.965 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:27.987233+0000", 2024-02-09T17:53:20.965 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.8249999999999998e-06 2024-02-09T17:53:20.965 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.965 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.965 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:53:20.965 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:27.987246+0000", 2024-02-09T17:53:20.965 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2503999999999999e-05 2024-02-09T17:53:20.965 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.965 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.965 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:53:20.965 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:27.987254+0000", 2024-02-09T17:53:20.965 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.0160000000000008e-06 2024-02-09T17:53:20.965 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.965 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.966 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:53:20.966 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:27.996880+0000", 2024-02-09T17:53:20.966 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0096268259999999998 2024-02-09T17:53:20.966 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.966 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.966 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:53:20.966 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:27.996978+0000", 2024-02-09T17:53:20.966 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.7718000000000002e-05 2024-02-09T17:53:20.966 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.966 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.966 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,5", 2024-02-09T17:53:20.966 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:27.997056+0000", 2024-02-09T17:53:20.966 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.7391999999999994e-05 2024-02-09T17:53:20.966 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.966 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.966 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:53:20.967 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:28.007779+0000", 2024-02-09T17:53:20.967 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.010723099 2024-02-09T17:53:20.967 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.967 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.967 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:53:20.967 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:28.007852+0000", 2024-02-09T17:53:20.967 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.3286000000000003e-05 2024-02-09T17:53:20.967 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.967 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.967 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:53:20.967 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:28.164567+0000", 2024-02-09T17:53:20.967 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.15671502100000001 2024-02-09T17:53:20.967 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.967 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.967 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:53:20.968 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:28.164634+0000", 2024-02-09T17:53:20.968 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.6625000000000007e-05 2024-02-09T17:53:20.968 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.968 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.968 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:53:20.968 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:28.164674+0000", 2024-02-09T17:53:20.968 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9909e-05 2024-02-09T17:53:20.968 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:53:20.968 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:53:20.968 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:53:20.968 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.968 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.968 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25837.0:236095 6.4 6:2e2979da:::benchmark_data_smithi179_126206_object105714:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e650)", 2024-02-09T17:53:20.968 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:49:27.989799+0000", 2024-02-09T17:53:20.969 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 230.85159940099999, 2024-02-09T17:53:20.969 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.181794135, 2024-02-09T17:53:20.969 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:53:20.969 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:53:20.969 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:53:20.969 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25837", 2024-02-09T17:53:20.969 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:53:20.969 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 236095 2024-02-09T17:53:20.969 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.969 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:53:20.969 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.969 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:53:20.969 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:27.989799+0000", 2024-02-09T17:53:20.969 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:53:20.969 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.970 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.970 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:53:20.970 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:27.989799+0000", 2024-02-09T17:53:20.970 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:53:20.970 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.970 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.970 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:53:20.970 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:27.989800+0000", 2024-02-09T17:53:20.970 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.13e-06 2024-02-09T17:53:20.970 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.970 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.970 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:53:20.970 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:27.989806+0000", 2024-02-09T17:53:20.970 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.0909999999999998e-06 2024-02-09T17:53:20.970 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.971 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.971 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:53:20.971 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:27.989822+0000", 2024-02-09T17:53:20.971 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.6090999999999999e-05 2024-02-09T17:53:20.971 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.971 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.971 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:53:20.971 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:27.989831+0000", 2024-02-09T17:53:20.971 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.7520000000000002e-06 2024-02-09T17:53:20.971 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.971 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.971 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:53:20.971 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:28.000942+0000", 2024-02-09T17:53:20.971 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.011110762999999999 2024-02-09T17:53:20.971 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.972 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.972 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:53:20.972 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:28.001025+0000", 2024-02-09T17:53:20.972 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.3324000000000004e-05 2024-02-09T17:53:20.972 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.972 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.972 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,5", 2024-02-09T17:53:20.972 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:28.001099+0000", 2024-02-09T17:53:20.972 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.3559999999999994e-05 2024-02-09T17:53:20.972 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.972 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.973 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:53:20.973 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:28.007863+0000", 2024-02-09T17:53:20.973 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0067640299999999999 2024-02-09T17:53:20.973 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.973 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.973 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:53:20.973 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:28.007888+0000", 2024-02-09T17:53:20.973 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.4973999999999998e-05 2024-02-09T17:53:20.973 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.973 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.973 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:53:20.973 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:28.171493+0000", 2024-02-09T17:53:20.973 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.16360524900000001 2024-02-09T17:53:20.974 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.974 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.974 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:53:20.974 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:28.171552+0000", 2024-02-09T17:53:20.974 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.9103e-05 2024-02-09T17:53:20.974 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.974 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.974 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:53:20.974 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:28.171593+0000", 2024-02-09T17:53:20.974 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.1068000000000001e-05 2024-02-09T17:53:20.974 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:53:20.974 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:53:20.974 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:53:20.974 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.974 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.975 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25837.0:250108 6.4 6:2d609273:::benchmark_data_smithi179_126206_object119727:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:53:20.975 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:49:47.518298+0000", 2024-02-09T17:53:20.975 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 211.323099824, 2024-02-09T17:53:20.975 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.242809575, 2024-02-09T17:53:20.975 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:53:20.975 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:53:20.975 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:53:20.975 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25837", 2024-02-09T17:53:20.975 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:53:20.975 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 250108 2024-02-09T17:53:20.975 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.975 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:53:20.975 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.975 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:53:20.975 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.518298+0000", 2024-02-09T17:53:20.975 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:53:20.976 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.976 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.976 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:53:20.976 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.518298+0000", 2024-02-09T17:53:20.976 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:53:20.976 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.976 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.976 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:53:20.976 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.518300+0000", 2024-02-09T17:53:20.976 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5260000000000001e-06 2024-02-09T17:53:20.976 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.976 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.976 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:53:20.976 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.518305+0000", 2024-02-09T17:53:20.976 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.5279999999999997e-06 2024-02-09T17:53:20.977 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.977 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.977 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:53:20.977 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.518316+0000", 2024-02-09T17:53:20.977 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1148000000000001e-05 2024-02-09T17:53:20.977 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.977 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.977 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:53:20.977 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.518323+0000", 2024-02-09T17:53:20.978 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.5140000000000004e-06 2024-02-09T17:53:20.978 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.978 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.978 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:53:20.978 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.518367+0000", 2024-02-09T17:53:20.978 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.3683000000000002e-05 2024-02-09T17:53:20.978 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.978 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.978 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:53:20.978 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.518467+0000", 2024-02-09T17:53:20.978 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.9852999999999994e-05 2024-02-09T17:53:20.978 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.978 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.978 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,5", 2024-02-09T17:53:20.978 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.518542+0000", 2024-02-09T17:53:20.979 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.5304999999999998e-05 2024-02-09T17:53:20.979 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.979 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.979 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:53:20.979 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.522701+0000", 2024-02-09T17:53:20.979 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0041589640000000002 2024-02-09T17:53:20.979 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.979 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.979 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:53:20.979 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.533905+0000", 2024-02-09T17:53:20.979 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.011204492 2024-02-09T17:53:20.979 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.979 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.979 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:53:20.979 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.760994+0000", 2024-02-09T17:53:20.980 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.22708893099999999 2024-02-09T17:53:20.980 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.980 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.980 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:53:20.980 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.761061+0000", 2024-02-09T17:53:20.980 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.6198000000000006e-05 2024-02-09T17:53:20.980 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.980 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.980 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:53:20.980 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.761108+0000", 2024-02-09T17:53:20.980 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.7432999999999998e-05 2024-02-09T17:53:20.980 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:53:20.980 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:53:20.980 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:53:20.980 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.981 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.981 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25837.0:250121 6.4 6:264967d5:::benchmark_data_smithi179_126206_object119740:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:53:20.981 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:49:47.538505+0000", 2024-02-09T17:53:20.981 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 211.30289341299999, 2024-02-09T17:53:20.981 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.222748849, 2024-02-09T17:53:20.981 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:53:20.981 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:53:20.981 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:53:20.981 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25837", 2024-02-09T17:53:20.981 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:53:20.981 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 250121 2024-02-09T17:53:20.981 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.981 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:53:20.981 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.981 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:53:20.982 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.538505+0000", 2024-02-09T17:53:20.982 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:53:20.982 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.982 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.982 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:53:20.982 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.538505+0000", 2024-02-09T17:53:20.982 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:53:20.982 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.982 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.982 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:53:20.982 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.538507+0000", 2024-02-09T17:53:20.982 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.7400000000000001e-06 2024-02-09T17:53:20.982 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.982 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.982 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:53:20.982 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.538515+0000", 2024-02-09T17:53:20.983 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.0190000000000006e-06 2024-02-09T17:53:20.983 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.983 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.983 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:53:20.983 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.538530+0000", 2024-02-09T17:53:20.983 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5335000000000001e-05 2024-02-09T17:53:20.983 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.983 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.983 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:53:20.983 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.538539+0000", 2024-02-09T17:53:20.983 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.3489999999999998e-06 2024-02-09T17:53:20.983 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.983 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.983 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:53:20.983 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.538609+0000", 2024-02-09T17:53:20.984 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.957e-05 2024-02-09T17:53:20.984 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.984 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.984 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:53:20.984 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.538757+0000", 2024-02-09T17:53:20.984 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014793200000000001 2024-02-09T17:53:20.984 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.984 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.984 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,5", 2024-02-09T17:53:20.984 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.538883+0000", 2024-02-09T17:53:20.984 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000125711 2024-02-09T17:53:20.984 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.984 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.984 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:53:20.985 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.543299+0000", 2024-02-09T17:53:20.985 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.004416906 2024-02-09T17:53:20.985 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.985 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.985 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:53:20.985 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.554527+0000", 2024-02-09T17:53:20.985 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.011227391 2024-02-09T17:53:20.985 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.985 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.985 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:53:20.985 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.761161+0000", 2024-02-09T17:53:20.985 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.206634028 2024-02-09T17:53:20.985 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.985 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.985 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:53:20.986 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.761217+0000", 2024-02-09T17:53:20.986 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6282000000000001e-05 2024-02-09T17:53:20.986 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.986 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.986 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:53:20.986 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.761254+0000", 2024-02-09T17:53:20.986 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.6585999999999998e-05 2024-02-09T17:53:20.986 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:53:20.986 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:53:20.986 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:53:20.986 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.986 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.986 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25837.0:250138 6.4 6:263dfb23:::benchmark_data_smithi179_126206_object119757:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:53:20.986 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:49:47.556378+0000", 2024-02-09T17:53:20.986 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 211.28502038100001, 2024-02-09T17:53:20.986 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20504225100000001, 2024-02-09T17:53:20.987 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:53:20.987 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:53:20.987 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:53:20.987 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25837", 2024-02-09T17:53:20.987 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:53:20.987 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 250138 2024-02-09T17:53:20.987 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.987 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:53:20.987 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.987 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:53:20.987 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.556378+0000", 2024-02-09T17:53:20.987 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:53:20.987 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.987 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.987 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:53:20.988 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.556378+0000", 2024-02-09T17:53:20.988 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:53:20.988 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.988 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.988 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:53:20.988 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.556379+0000", 2024-02-09T17:53:20.988 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.217e-06 2024-02-09T17:53:20.988 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.988 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.988 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:53:20.988 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.556384+0000", 2024-02-09T17:53:20.988 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.4039999999999997e-06 2024-02-09T17:53:20.988 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.988 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.988 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:53:20.989 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.556399+0000", 2024-02-09T17:53:20.989 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5355000000000001e-05 2024-02-09T17:53:20.989 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.989 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.989 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:53:20.989 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.556407+0000", 2024-02-09T17:53:20.989 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.0409999999999998e-06 2024-02-09T17:53:20.989 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.989 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.989 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:53:20.989 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.556461+0000", 2024-02-09T17:53:20.989 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.4159000000000001e-05 2024-02-09T17:53:20.989 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.989 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.989 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:53:20.989 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.556584+0000", 2024-02-09T17:53:20.990 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012253899999999999 2024-02-09T17:53:20.990 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.990 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.990 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,5", 2024-02-09T17:53:20.990 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.556680+0000", 2024-02-09T17:53:20.990 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.6075000000000004e-05 2024-02-09T17:53:20.990 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.990 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.990 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:53:20.990 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.561115+0000", 2024-02-09T17:53:20.991 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0044352219999999999 2024-02-09T17:53:20.991 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.991 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.991 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:53:20.991 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.570136+0000", 2024-02-09T17:53:20.991 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0090209369999999997 2024-02-09T17:53:20.991 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.991 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.991 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:53:20.991 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.761321+0000", 2024-02-09T17:53:20.991 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.191185144 2024-02-09T17:53:20.991 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.991 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.991 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:53:20.992 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.761371+0000", 2024-02-09T17:53:20.992 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.9793999999999998e-05 2024-02-09T17:53:20.992 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.992 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.992 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:53:20.992 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.761420+0000", 2024-02-09T17:53:20.992 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.9363999999999997e-05 2024-02-09T17:53:20.992 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:53:20.992 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:53:20.992 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:53:20.992 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.992 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.992 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25837.0:250144 6.4 6:2d888af1:::benchmark_data_smithi179_126206_object119763:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:53:20.992 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:49:47.562632+0000", 2024-02-09T17:53:20.992 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 211.27876599199999, 2024-02-09T17:53:20.993 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20241316000000001, 2024-02-09T17:53:20.993 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:53:20.993 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:53:20.993 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:53:20.993 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25837", 2024-02-09T17:53:20.993 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:53:20.993 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 250144 2024-02-09T17:53:20.993 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.993 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:53:20.993 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.993 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:53:20.993 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.562632+0000", 2024-02-09T17:53:20.993 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:53:20.993 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.993 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.994 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:53:20.994 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.562632+0000", 2024-02-09T17:53:20.994 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:53:20.994 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.994 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.994 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:53:20.994 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.562634+0000", 2024-02-09T17:53:20.994 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2929999999999999e-06 2024-02-09T17:53:20.994 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.994 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.994 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:53:20.994 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.562637+0000", 2024-02-09T17:53:20.994 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.8410000000000002e-06 2024-02-09T17:53:20.994 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.994 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.994 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:53:20.995 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.562644+0000", 2024-02-09T17:53:20.995 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.4330000000000001e-06 2024-02-09T17:53:20.995 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.995 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.995 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:53:20.995 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.562648+0000", 2024-02-09T17:53:20.995 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2679999999999996e-06 2024-02-09T17:53:20.995 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.995 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.995 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:53:20.995 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.562708+0000", 2024-02-09T17:53:20.995 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.0024e-05 2024-02-09T17:53:20.995 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.995 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.995 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:53:20.995 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.562842+0000", 2024-02-09T17:53:20.996 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013413699999999999 2024-02-09T17:53:20.996 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.996 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.996 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,5", 2024-02-09T17:53:20.996 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.562971+0000", 2024-02-09T17:53:20.996 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012912000000000001 2024-02-09T17:53:20.996 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.996 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.996 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:53:20.996 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.567206+0000", 2024-02-09T17:53:20.996 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0042345400000000002 2024-02-09T17:53:20.996 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.996 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.996 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:53:20.996 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.571339+0000", 2024-02-09T17:53:20.997 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0041335349999999998 2024-02-09T17:53:20.997 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.997 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.997 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:53:20.997 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.764960+0000", 2024-02-09T17:53:20.997 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.19362069500000001 2024-02-09T17:53:20.997 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.997 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.997 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:53:20.997 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.765013+0000", 2024-02-09T17:53:20.997 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2379000000000001e-05 2024-02-09T17:53:20.997 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:53:20.997 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:53:20.997 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:53:20.997 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.765045+0000", 2024-02-09T17:53:20.998 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.2895000000000003e-05 2024-02-09T17:53:20.998 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:53:20.998 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:53:20.998 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:53:20.998 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:53:20.998 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:53:20.998 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:53:20.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:20.999 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:20.999 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:53:21.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:53:21.100 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:53:21.179 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:21.233 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:53:21.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:21.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:53:21.340 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:53:21.378 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:21.402 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:53:21.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:21.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:53:21.509 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:53:21.559 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:21.604 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:53:21.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:21.610 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:53:21.711 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:53:21.748 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-09T17:53:21.749 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:21.776 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:53:21.782 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:21.783 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:53:21.884 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:53:21.962 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:22.232 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:53:22.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:22.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:53:22.285 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:53:22.313 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:53:22.339 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:53:22.354 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:53:22.369 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:22.394 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:53:22.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:22.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:53:22.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:22 smithi099 ceph-mon[39420]: pgmap v1840: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 576 op/s; 784/178368 objects degraded (0.440%); 15401/178368 objects misplaced (8.634%); 8.3 MiB/s, 133 objects/s recovering 2024-02-09T17:53:22.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:22 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 784/178368 objects degraded (0.440%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-02-09T17:53:22.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:22 smithi099 ceph-mon[39420]: osdmap e743: 12 total, 12 up, 12 in 2024-02-09T17:53:22.500 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:53:22.559 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:22 smithi179 ceph-mon[43975]: pgmap v1840: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 576 op/s; 784/178368 objects degraded (0.440%); 15401/178368 objects misplaced (8.634%); 8.3 MiB/s, 133 objects/s recovering 2024-02-09T17:53:22.560 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:22 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 784/178368 objects degraded (0.440%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-02-09T17:53:22.560 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:22 smithi179 ceph-mon[43975]: osdmap e743: 12 total, 12 up, 12 in 2024-02-09T17:53:22.562 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:22.599 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:53:22.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:22.607 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:53:22.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:22 smithi162 ceph-mon[44304]: pgmap v1840: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 576 op/s; 784/178368 objects degraded (0.440%); 15401/178368 objects misplaced (8.634%); 8.3 MiB/s, 133 objects/s recovering 2024-02-09T17:53:22.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:22 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 784/178368 objects degraded (0.440%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-02-09T17:53:22.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:22 smithi162 ceph-mon[44304]: osdmap e743: 12 total, 12 up, 12 in 2024-02-09T17:53:22.707 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:53:22.794 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:23.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:23 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:53:23.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:23 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:53:23.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:23 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:53:23.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:23 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:53:23.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:23 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:53:23.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:23 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:53:23.705 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:53:23.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:23.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:53:23.821 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:53:23.888 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:24.346 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:24 smithi099 ceph-mon[39420]: pgmap v1842: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 689 op/s; 853/172218 objects degraded (0.495%); 14690/172218 objects misplaced (8.530%); 13 MiB/s, 209 objects/s recovering 2024-02-09T17:53:24.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:24 smithi179 ceph-mon[43975]: pgmap v1842: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 689 op/s; 853/172218 objects degraded (0.495%); 14690/172218 objects misplaced (8.530%); 13 MiB/s, 209 objects/s recovering 2024-02-09T17:53:24.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:24 smithi162 ceph-mon[44304]: pgmap v1842: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 689 op/s; 853/172218 objects degraded (0.495%); 14690/172218 objects misplaced (8.530%); 13 MiB/s, 209 objects/s recovering 2024-02-09T17:53:25.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:25 smithi179 ceph-mon[43975]: osdmap e744: 12 total, 12 up, 12 in 2024-02-09T17:53:25.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:25 smithi162 ceph-mon[44304]: osdmap e744: 12 total, 12 up, 12 in 2024-02-09T17:53:25.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:25 smithi099 ceph-mon[39420]: osdmap e744: 12 total, 12 up, 12 in 2024-02-09T17:53:26.381 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:53:26.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:26.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:53:26.488 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:53:26.541 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:26 smithi179 ceph-mon[43975]: pgmap v1844: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 521 op/s; 945/171600 objects degraded (0.551%); 14483/171600 objects misplaced (8.440%); 8.9 MiB/s, 142 objects/s recovering 2024-02-09T17:53:26.541 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:26 smithi179 ceph-mon[43975]: osdmap e745: 12 total, 12 up, 12 in 2024-02-09T17:53:26.543 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:26.566 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:53:26.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:26.570 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:53:26.671 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:53:26.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:26 smithi162 ceph-mon[44304]: pgmap v1844: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 521 op/s; 945/171600 objects degraded (0.551%); 14483/171600 objects misplaced (8.440%); 8.9 MiB/s, 142 objects/s recovering 2024-02-09T17:53:26.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:26 smithi162 ceph-mon[44304]: osdmap e745: 12 total, 12 up, 12 in 2024-02-09T17:53:26.724 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:26 smithi099 ceph-mon[39420]: pgmap v1844: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 521 op/s; 945/171600 objects degraded (0.551%); 14483/171600 objects misplaced (8.440%); 8.9 MiB/s, 142 objects/s recovering 2024-02-09T17:53:26.724 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:26 smithi099 ceph-mon[39420]: osdmap e745: 12 total, 12 up, 12 in 2024-02-09T17:53:26.729 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:26.961 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:53:26.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:26.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:53:27.071 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:53:27.122 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:27.309 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is unset 2024-02-09T17:53:27.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:27 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1135172927' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:53:27.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:27 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 945/171600 objects degraded (0.551%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-02-09T17:53:27.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:27 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1135172927' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:53:27.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:27 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 945/171600 objects degraded (0.551%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-02-09T17:53:27.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:27 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1135172927' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:53:27.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:27 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 945/171600 objects degraded (0.551%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-02-09T17:53:28.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:28 smithi179 ceph-mon[43975]: pgmap v1846: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 730 op/s; 1026/165213 objects degraded (0.621%); 13895/165213 objects misplaced (8.410%); 11 MiB/s, 179 objects/s recovering 2024-02-09T17:53:28.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:28 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:53:28.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:28 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:53:28.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:28 smithi179 ceph-mon[43975]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:53:28.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:28 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1135172927' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:53:28.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:28 smithi179 ceph-mon[43975]: osdmap e746: 12 total, 12 up, 12 in 2024-02-09T17:53:28.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:28 smithi162 ceph-mon[44304]: pgmap v1846: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 730 op/s; 1026/165213 objects degraded (0.621%); 13895/165213 objects misplaced (8.410%); 11 MiB/s, 179 objects/s recovering 2024-02-09T17:53:28.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:28 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:53:28.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:28 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:53:28.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:28 smithi162 ceph-mon[44304]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:53:28.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:28 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1135172927' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:53:28.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:28 smithi162 ceph-mon[44304]: osdmap e746: 12 total, 12 up, 12 in 2024-02-09T17:53:28.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:28 smithi099 ceph-mon[39420]: pgmap v1846: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 730 op/s; 1026/165213 objects degraded (0.621%); 13895/165213 objects misplaced (8.410%); 11 MiB/s, 179 objects/s recovering 2024-02-09T17:53:28.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:28 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:53:28.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:28 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:53:28.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:28 smithi099 ceph-mon[39420]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:53:28.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:28 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1135172927' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:53:28.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:28 smithi099 ceph-mon[39420]: osdmap e746: 12 total, 12 up, 12 in 2024-02-09T17:53:30.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:30 smithi099 ceph-mon[39420]: pgmap v1848: 47 pgs: 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 515 op/s; 1065/162933 objects degraded (0.654%); 13538/162933 objects misplaced (8.309%); 7.8 MiB/s, 125 objects/s recovering 2024-02-09T17:53:30.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:30 smithi179 ceph-mon[43975]: pgmap v1848: 47 pgs: 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 515 op/s; 1065/162933 objects degraded (0.654%); 13538/162933 objects misplaced (8.309%); 7.8 MiB/s, 125 objects/s recovering 2024-02-09T17:53:30.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:30 smithi162 ceph-mon[44304]: pgmap v1848: 47 pgs: 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 515 op/s; 1065/162933 objects degraded (0.654%); 13538/162933 objects misplaced (8.309%); 7.8 MiB/s, 125 objects/s recovering 2024-02-09T17:53:30.726 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:53:30.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:30.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:53:30.832 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:53:30.867 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:30.893 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:53:30.896 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:30.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:53:30.997 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:53:31.048 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:31.072 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:53:31.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:31.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:53:31.178 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:53:31.253 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:31.315 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.5 to 1 (10000) 2024-02-09T17:53:31.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:31 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:53:31.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:31 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3476912086' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-09T17:53:31.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:31 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:53:31.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:31 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3476912086' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-09T17:53:31.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:31 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:53:31.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:31 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3476912086' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-09T17:53:32.073 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:53:32.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:32.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:53:32.178 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:53:32.251 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:32.343 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:32 smithi099 ceph-mon[39420]: 1.11 scrub starts 2024-02-09T17:53:32.343 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:32 smithi099 ceph-mon[39420]: 1.11 scrub ok 2024-02-09T17:53:32.344 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:32 smithi099 ceph-mon[39420]: pgmap v1849: 47 pgs: 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 572 op/s; 1104/159789 objects degraded (0.691%); 13167/159789 objects misplaced (8.240%); 7.0 MiB/s, 112 objects/s recovering 2024-02-09T17:53:32.344 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:32 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3476912086' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-09T17:53:32.344 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:32 smithi099 ceph-mon[39420]: osdmap e747: 12 total, 12 up, 12 in 2024-02-09T17:53:32.344 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:32 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 1104/159789 objects degraded (0.691%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:53:32.631 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:53:32.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:32.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:53:32.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:32 smithi179 ceph-mon[43975]: 1.11 scrub starts 2024-02-09T17:53:32.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:32 smithi179 ceph-mon[43975]: 1.11 scrub ok 2024-02-09T17:53:32.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:32 smithi179 ceph-mon[43975]: pgmap v1849: 47 pgs: 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 572 op/s; 1104/159789 objects degraded (0.691%); 13167/159789 objects misplaced (8.240%); 7.0 MiB/s, 112 objects/s recovering 2024-02-09T17:53:32.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:32 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3476912086' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-09T17:53:32.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:32 smithi179 ceph-mon[43975]: osdmap e747: 12 total, 12 up, 12 in 2024-02-09T17:53:32.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:32 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 1104/159789 objects degraded (0.691%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:53:32.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:32 smithi162 ceph-mon[44304]: 1.11 scrub starts 2024-02-09T17:53:32.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:32 smithi162 ceph-mon[44304]: 1.11 scrub ok 2024-02-09T17:53:32.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:32 smithi162 ceph-mon[44304]: pgmap v1849: 47 pgs: 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 572 op/s; 1104/159789 objects degraded (0.691%); 13167/159789 objects misplaced (8.240%); 7.0 MiB/s, 112 objects/s recovering 2024-02-09T17:53:32.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:32 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3476912086' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-09T17:53:32.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:32 smithi162 ceph-mon[44304]: osdmap e747: 12 total, 12 up, 12 in 2024-02-09T17:53:32.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:32 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 1104/159789 objects degraded (0.691%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:53:32.745 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:53:32.778 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:32.824 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:53:32.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:32.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:53:32.932 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:53:32.975 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:33.344 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:53:33.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:33.348 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:53:33.375 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 6 1 2024-02-09T17:53:33.448 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:53:33.491 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:33.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:33 smithi179 ceph-mon[43975]: 7.b scrub starts 2024-02-09T17:53:33.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:33 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1272600255' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:53:33.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:33 smithi162 ceph-mon[44304]: 7.b scrub starts 2024-02-09T17:53:33.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:33 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1272600255' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:53:33.683 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:33 smithi099 ceph-mon[39420]: 7.b scrub starts 2024-02-09T17:53:33.683 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:33 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1272600255' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:53:33.696 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:53:33.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:33.704 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:53:33.805 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:53:33.847 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:53:33.847 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:53:33.847 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:53:33.848 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:53:33.848 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:53:33.848 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:53:33.848 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:53:33.848 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:53:33.848 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:53:33.848 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:53:33.848 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:53:33.873 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:53:33.882 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:33.883 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-09T17:53:33.891 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:53:33.892 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:53:33.892 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:53:33.892 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:53:33.892 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:53:33.892 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:53:33.893 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:53:33.893 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:53:33.894 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:53:33.894 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:53:33.894 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:53:33.894 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:53:34.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:34 smithi179 ceph-mon[43975]: 7.b scrub ok 2024-02-09T17:53:34.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:34 smithi179 ceph-mon[43975]: pgmap v1851: 47 pgs: 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 718 op/s; 1175/153534 objects degraded (0.765%); 12961/153534 objects misplaced (8.442%); 7.2 MiB/s, 115 objects/s recovering 2024-02-09T17:53:34.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:34 smithi179 ceph-mon[43975]: 7.f scrub starts 2024-02-09T17:53:34.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:34 smithi179 ceph-mon[43975]: 1.a scrub starts 2024-02-09T17:53:34.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:34 smithi179 ceph-mon[43975]: 1.a scrub ok 2024-02-09T17:53:34.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:34 smithi162 ceph-mon[44304]: 7.b scrub ok 2024-02-09T17:53:34.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:34 smithi162 ceph-mon[44304]: pgmap v1851: 47 pgs: 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 718 op/s; 1175/153534 objects degraded (0.765%); 12961/153534 objects misplaced (8.442%); 7.2 MiB/s, 115 objects/s recovering 2024-02-09T17:53:34.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:34 smithi162 ceph-mon[44304]: 7.f scrub starts 2024-02-09T17:53:34.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:34 smithi162 ceph-mon[44304]: 1.a scrub starts 2024-02-09T17:53:34.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:34 smithi162 ceph-mon[44304]: 1.a scrub ok 2024-02-09T17:53:34.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:34 smithi099 ceph-mon[39420]: 7.b scrub ok 2024-02-09T17:53:34.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:34 smithi099 ceph-mon[39420]: pgmap v1851: 47 pgs: 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 718 op/s; 1175/153534 objects degraded (0.765%); 12961/153534 objects misplaced (8.442%); 7.2 MiB/s, 115 objects/s recovering 2024-02-09T17:53:34.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:34 smithi099 ceph-mon[39420]: 7.f scrub starts 2024-02-09T17:53:34.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:34 smithi099 ceph-mon[39420]: 1.a scrub starts 2024-02-09T17:53:34.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:34 smithi099 ceph-mon[39420]: 1.a scrub ok 2024-02-09T17:53:35.508 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:53:35.508 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:53:35.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:35 smithi179 ceph-mon[43975]: 7.f scrub ok 2024-02-09T17:53:35.681 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:35 smithi162 ceph-mon[44304]: 7.f scrub ok 2024-02-09T17:53:35.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:35 smithi099 ceph-mon[39420]: 7.f scrub ok 2024-02-09T17:53:36.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:36 smithi179 ceph-mon[43975]: 1.12 scrub starts 2024-02-09T17:53:36.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:36 smithi179 ceph-mon[43975]: 1.12 scrub ok 2024-02-09T17:53:36.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:36 smithi179 ceph-mon[43975]: pgmap v1852: 47 pgs: 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 512 op/s; 1283/152907 objects degraded (0.839%); 12752/152907 objects misplaced (8.340%); 5.3 MiB/s, 85 objects/s recovering 2024-02-09T17:53:36.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:36 smithi162 ceph-mon[44304]: 1.12 scrub starts 2024-02-09T17:53:36.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:36 smithi162 ceph-mon[44304]: 1.12 scrub ok 2024-02-09T17:53:36.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:36 smithi162 ceph-mon[44304]: pgmap v1852: 47 pgs: 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 512 op/s; 1283/152907 objects degraded (0.839%); 12752/152907 objects misplaced (8.340%); 5.3 MiB/s, 85 objects/s recovering 2024-02-09T17:53:36.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:36 smithi099 ceph-mon[39420]: 1.12 scrub starts 2024-02-09T17:53:36.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:36 smithi099 ceph-mon[39420]: 1.12 scrub ok 2024-02-09T17:53:36.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:36 smithi099 ceph-mon[39420]: pgmap v1852: 47 pgs: 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 512 op/s; 1283/152907 objects degraded (0.839%); 12752/152907 objects misplaced (8.340%); 5.3 MiB/s, 85 objects/s recovering 2024-02-09T17:53:37.399 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:53:37.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:37.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:53:37.510 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:53:37.584 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:37.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:37 smithi162 ceph-mon[44304]: 1.1d scrub starts 2024-02-09T17:53:37.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:37 smithi162 ceph-mon[44304]: 1.1d scrub ok 2024-02-09T17:53:37.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:37 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 1283/152907 objects degraded (0.839%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:53:37.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:37 smithi099 ceph-mon[39420]: 1.1d scrub starts 2024-02-09T17:53:37.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:37 smithi099 ceph-mon[39420]: 1.1d scrub ok 2024-02-09T17:53:37.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:37 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 1283/152907 objects degraded (0.839%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:53:37.898 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:37 smithi179 ceph-mon[43975]: 1.1d scrub starts 2024-02-09T17:53:37.899 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:37 smithi179 ceph-mon[43975]: 1.1d scrub ok 2024-02-09T17:53:37.899 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:37 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 1283/152907 objects degraded (0.839%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:53:38.088 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:53:38.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:38.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:53:38.132 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-09T17:53:38.198 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:53:38.236 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:38.278 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:53:38.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:38.284 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:53:38.385 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:53:38.497 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:38.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:38 smithi179 ceph-mon[43975]: pgmap v1853: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 666 op/s; 1431/145545 objects degraded (0.983%); 12141/145545 objects misplaced (8.342%); 6.0 MiB/s, 96 objects/s recovering 2024-02-09T17:53:38.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:38 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:53:38.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:38 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:53:38.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:38 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:53:38.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:38 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:53:38.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:38 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:53:38.709 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:38 smithi099 ceph-mon[39420]: pgmap v1853: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 666 op/s; 1431/145545 objects degraded (0.983%); 12141/145545 objects misplaced (8.342%); 6.0 MiB/s, 96 objects/s recovering 2024-02-09T17:53:38.710 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:38 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:53:38.710 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:38 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:53:38.710 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:38 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:53:38.710 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:38 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:53:38.710 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:38 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:53:38.714 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:53:38.715 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:53:38.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:38.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:53:38.775 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:53:38.821 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:53:38.891 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:38.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:38 smithi162 ceph-mon[44304]: pgmap v1853: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 666 op/s; 1431/145545 objects degraded (0.983%); 12141/145545 objects misplaced (8.342%); 6.0 MiB/s, 96 objects/s recovering 2024-02-09T17:53:38.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:38 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:53:38.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:38 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:53:38.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:38 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:53:38.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:38 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:53:38.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:38 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:53:39.012 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:53:39.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:39.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:53:39.122 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:53:39.152 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:39.174 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:53:39.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:39.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:53:39.282 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:53:39.317 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:39.342 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:53:39.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:39.346 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:53:39.447 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:53:39.479 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:39.503 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:53:39.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:39.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:53:39.607 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:53:39.640 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:39.668 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:53:39.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:39.676 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:53:39.777 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:53:39.832 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:39.869 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:53:39.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:39.875 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:53:39.976 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:53:40.043 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:40.072 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:53:40.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:40.078 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:53:40.179 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:53:40.224 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:40.251 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:53:40.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:40.257 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:53:40.358 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:53:40.415 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:40.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:40 smithi099 ceph-mon[39420]: 1.18 scrub starts 2024-02-09T17:53:40.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:40 smithi099 ceph-mon[39420]: 1.18 scrub ok 2024-02-09T17:53:40.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:40 smithi099 ceph-mon[39420]: 1.13 scrub starts 2024-02-09T17:53:40.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:40 smithi099 ceph-mon[39420]: pgmap v1854: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 667 op/s; 1560/142884 objects degraded (1.092%); 11906/142884 objects misplaced (8.333%); 4.6 MiB/s, 73 objects/s recovering 2024-02-09T17:53:40.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:40 smithi099 ceph-mon[39420]: 1.13 scrub ok 2024-02-09T17:53:40.578 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:53:40.593 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:53:40.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:40.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:53:40.708 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:53:40.742 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:40 smithi162 ceph-mon[44304]: 1.18 scrub starts 2024-02-09T17:53:40.742 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:40 smithi162 ceph-mon[44304]: 1.18 scrub ok 2024-02-09T17:53:40.742 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:40 smithi162 ceph-mon[44304]: 1.13 scrub starts 2024-02-09T17:53:40.742 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:40 smithi162 ceph-mon[44304]: pgmap v1854: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 667 op/s; 1560/142884 objects degraded (1.092%); 11906/142884 objects misplaced (8.333%); 4.6 MiB/s, 73 objects/s recovering 2024-02-09T17:53:40.743 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:40 smithi162 ceph-mon[44304]: 1.13 scrub ok 2024-02-09T17:53:40.743 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:40.753 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:53:40.770 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:53:40.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:40.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:53:40.879 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:53:40.908 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:40.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:40 smithi179 ceph-mon[43975]: 1.18 scrub starts 2024-02-09T17:53:40.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:40 smithi179 ceph-mon[43975]: 1.18 scrub ok 2024-02-09T17:53:40.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:40 smithi179 ceph-mon[43975]: 1.13 scrub starts 2024-02-09T17:53:40.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:40 smithi179 ceph-mon[43975]: pgmap v1854: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 667 op/s; 1560/142884 objects degraded (1.092%); 11906/142884 objects misplaced (8.333%); 4.6 MiB/s, 73 objects/s recovering 2024-02-09T17:53:40.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:40 smithi179 ceph-mon[43975]: 1.13 scrub ok 2024-02-09T17:53:40.935 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:53:40.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:40.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:53:41.040 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:53:41.069 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:41.096 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:53:41.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:41.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:53:41.202 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:53:41.242 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:41.262 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:53:41.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:41.270 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:53:41.371 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:53:41.405 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:41.429 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:53:41.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:41.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:53:41.484 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:41 smithi099 ceph-mon[39420]: 7.0 scrub starts 2024-02-09T17:53:41.533 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:53:41.589 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:41.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:41 smithi162 ceph-mon[44304]: 7.0 scrub starts 2024-02-09T17:53:41.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:41 smithi162 ceph-mon[44304]: 1.3 scrub starts 2024-02-09T17:53:41.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:41 smithi162 ceph-mon[44304]: 1.3 scrub ok 2024-02-09T17:53:41.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:41 smithi162 ceph-mon[44304]: 7.0 scrub ok 2024-02-09T17:53:41.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:41 smithi099 ceph-mon[39420]: 1.3 scrub starts 2024-02-09T17:53:41.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:41 smithi099 ceph-mon[39420]: 1.3 scrub ok 2024-02-09T17:53:41.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:41 smithi099 ceph-mon[39420]: 7.0 scrub ok 2024-02-09T17:53:41.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:41 smithi179 ceph-mon[43975]: 7.0 scrub starts 2024-02-09T17:53:41.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:41 smithi179 ceph-mon[43975]: 1.3 scrub starts 2024-02-09T17:53:41.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:41 smithi179 ceph-mon[43975]: 1.3 scrub ok 2024-02-09T17:53:41.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:41 smithi179 ceph-mon[43975]: 7.0 scrub ok 2024-02-09T17:53:42.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:42 smithi099 ceph-mon[39420]: 1.16 scrub starts 2024-02-09T17:53:42.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:42 smithi099 ceph-mon[39420]: 1.16 scrub ok 2024-02-09T17:53:42.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:42 smithi099 ceph-mon[39420]: pgmap v1855: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+clean+scrubbing, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 671 op/s; 1636/139629 objects degraded (1.172%); 11472/139629 objects misplaced (8.216%); 4.6 MiB/s, 73 objects/s recovering 2024-02-09T17:53:42.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:42 smithi099 ceph-mon[39420]: 1.17 scrub starts 2024-02-09T17:53:42.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:42 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 1636/139629 objects degraded (1.172%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:53:42.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:42 smithi099 ceph-mon[39420]: 1.17 scrub ok 2024-02-09T17:53:42.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:42 smithi179 ceph-mon[43975]: 1.16 scrub starts 2024-02-09T17:53:42.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:42 smithi179 ceph-mon[43975]: 1.16 scrub ok 2024-02-09T17:53:42.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:42 smithi179 ceph-mon[43975]: pgmap v1855: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+clean+scrubbing, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 671 op/s; 1636/139629 objects degraded (1.172%); 11472/139629 objects misplaced (8.216%); 4.6 MiB/s, 73 objects/s recovering 2024-02-09T17:53:42.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:42 smithi179 ceph-mon[43975]: 1.17 scrub starts 2024-02-09T17:53:42.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:42 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 1636/139629 objects degraded (1.172%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:53:42.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:42 smithi179 ceph-mon[43975]: 1.17 scrub ok 2024-02-09T17:53:42.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:42 smithi162 ceph-mon[44304]: 1.16 scrub starts 2024-02-09T17:53:42.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:42 smithi162 ceph-mon[44304]: 1.16 scrub ok 2024-02-09T17:53:42.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:42 smithi162 ceph-mon[44304]: pgmap v1855: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+clean+scrubbing, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 671 op/s; 1636/139629 objects degraded (1.172%); 11472/139629 objects misplaced (8.216%); 4.6 MiB/s, 73 objects/s recovering 2024-02-09T17:53:42.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:42 smithi162 ceph-mon[44304]: 1.17 scrub starts 2024-02-09T17:53:42.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:42 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 1636/139629 objects degraded (1.172%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:53:42.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:42 smithi162 ceph-mon[44304]: 1.17 scrub ok 2024-02-09T17:53:43.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:43 smithi099 ceph-mon[39420]: 1.14 scrub starts 2024-02-09T17:53:43.787 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:43 smithi099 ceph-mon[39420]: 1.14 scrub ok 2024-02-09T17:53:43.788 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:43 smithi099 ceph-mon[39420]: 1.1b scrub starts 2024-02-09T17:53:43.788 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:43 smithi099 ceph-mon[39420]: 1.1b scrub ok 2024-02-09T17:53:43.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:43 smithi179 ceph-mon[43975]: 1.14 scrub starts 2024-02-09T17:53:43.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:43 smithi179 ceph-mon[43975]: 1.14 scrub ok 2024-02-09T17:53:43.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:43 smithi179 ceph-mon[43975]: 1.1b scrub starts 2024-02-09T17:53:43.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:43 smithi179 ceph-mon[43975]: 1.1b scrub ok 2024-02-09T17:53:43.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:43 smithi162 ceph-mon[44304]: 1.14 scrub starts 2024-02-09T17:53:43.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:43 smithi162 ceph-mon[44304]: 1.14 scrub ok 2024-02-09T17:53:43.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:43 smithi162 ceph-mon[44304]: 1.1b scrub starts 2024-02-09T17:53:43.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:43 smithi162 ceph-mon[44304]: 1.1b scrub ok 2024-02-09T17:53:44.780 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:44 smithi099 ceph-mon[39420]: 7.a scrub starts 2024-02-09T17:53:44.780 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:44 smithi099 ceph-mon[39420]: 1.7 scrub starts 2024-02-09T17:53:44.780 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:44 smithi099 ceph-mon[39420]: pgmap v1856: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+clean+scrubbing, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 713 op/s; 1729/133347 objects degraded (1.297%); 11252/133347 objects misplaced (8.438%); 4.5 MiB/s, 72 objects/s recovering 2024-02-09T17:53:44.780 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:44 smithi099 ceph-mon[39420]: 1.7 scrub ok 2024-02-09T17:53:44.780 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:44 smithi099 ceph-mon[39420]: 7.a scrub ok 2024-02-09T17:53:44.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:44 smithi179 ceph-mon[43975]: 7.a scrub starts 2024-02-09T17:53:44.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:44 smithi179 ceph-mon[43975]: 1.7 scrub starts 2024-02-09T17:53:44.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:44 smithi179 ceph-mon[43975]: pgmap v1856: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+clean+scrubbing, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 713 op/s; 1729/133347 objects degraded (1.297%); 11252/133347 objects misplaced (8.438%); 4.5 MiB/s, 72 objects/s recovering 2024-02-09T17:53:44.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:44 smithi179 ceph-mon[43975]: 1.7 scrub ok 2024-02-09T17:53:44.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:44 smithi179 ceph-mon[43975]: 7.a scrub ok 2024-02-09T17:53:44.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:44 smithi162 ceph-mon[44304]: 7.a scrub starts 2024-02-09T17:53:44.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:44 smithi162 ceph-mon[44304]: 1.7 scrub starts 2024-02-09T17:53:44.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:44 smithi162 ceph-mon[44304]: pgmap v1856: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+clean+scrubbing, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 713 op/s; 1729/133347 objects degraded (1.297%); 11252/133347 objects misplaced (8.438%); 4.5 MiB/s, 72 objects/s recovering 2024-02-09T17:53:44.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:44 smithi162 ceph-mon[44304]: 1.7 scrub ok 2024-02-09T17:53:44.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:44 smithi162 ceph-mon[44304]: 7.a scrub ok 2024-02-09T17:53:46.313 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:53:46.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:46.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:53:46.427 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:53:46.497 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:46.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:46 smithi099 ceph-mon[39420]: pgmap v1857: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+clean+scrubbing, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 576 op/s; 1810/132747 objects degraded (1.363%); 11051/132747 objects misplaced (8.325%); 4.4 MiB/s, 70 objects/s recovering 2024-02-09T17:53:46.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:46 smithi179 ceph-mon[43975]: pgmap v1857: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+clean+scrubbing, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 576 op/s; 1810/132747 objects degraded (1.363%); 11051/132747 objects misplaced (8.325%); 4.4 MiB/s, 70 objects/s recovering 2024-02-09T17:53:46.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:46 smithi162 ceph-mon[44304]: pgmap v1857: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+clean+scrubbing, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 576 op/s; 1810/132747 objects degraded (1.363%); 11051/132747 objects misplaced (8.325%); 4.4 MiB/s, 70 objects/s recovering 2024-02-09T17:53:47.683 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:47 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 1810/132747 objects degraded (1.363%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:53:47.683 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:47 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/897759694' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-09T17:53:47.683 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:47 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-09T17:53:47.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:47 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 1810/132747 objects degraded (1.363%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:53:47.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:47 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/897759694' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-09T17:53:47.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:47 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-09T17:53:47.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:47 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 1810/132747 objects degraded (1.363%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:53:47.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:47 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/897759694' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-09T17:53:47.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:47 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-09T17:53:47.934 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.6 to 1 (10000) 2024-02-09T17:53:48.382 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:53:48.382 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:53:48.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:48 smithi179 ceph-mon[43975]: pgmap v1858: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 773 op/s; 1947/125028 objects degraded (1.557%); 10405/125028 objects misplaced (8.322%); 5.4 MiB/s, 86 objects/s recovering 2024-02-09T17:53:48.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:48 smithi179 ceph-mon[43975]: 1.1c scrub starts 2024-02-09T17:53:48.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:48 smithi179 ceph-mon[43975]: 1.1c scrub ok 2024-02-09T17:53:48.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:48 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-09T17:53:48.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:48 smithi179 ceph-mon[43975]: osdmap e748: 12 total, 12 up, 12 in 2024-02-09T17:53:48.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:48 smithi162 ceph-mon[44304]: pgmap v1858: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 773 op/s; 1947/125028 objects degraded (1.557%); 10405/125028 objects misplaced (8.322%); 5.4 MiB/s, 86 objects/s recovering 2024-02-09T17:53:48.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:48 smithi162 ceph-mon[44304]: 1.1c scrub starts 2024-02-09T17:53:48.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:48 smithi162 ceph-mon[44304]: 1.1c scrub ok 2024-02-09T17:53:48.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:48 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-09T17:53:48.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:48 smithi162 ceph-mon[44304]: osdmap e748: 12 total, 12 up, 12 in 2024-02-09T17:53:48.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:48 smithi099 ceph-mon[39420]: pgmap v1858: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 773 op/s; 1947/125028 objects degraded (1.557%); 10405/125028 objects misplaced (8.322%); 5.4 MiB/s, 86 objects/s recovering 2024-02-09T17:53:48.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:48 smithi099 ceph-mon[39420]: 1.1c scrub starts 2024-02-09T17:53:48.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:48 smithi099 ceph-mon[39420]: 1.1c scrub ok 2024-02-09T17:53:48.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:48 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-09T17:53:48.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:48 smithi099 ceph-mon[39420]: osdmap e748: 12 total, 12 up, 12 in 2024-02-09T17:53:49.799 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:53:49.803 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:49.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:53:49.824 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:53:49.905 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:53:49.939 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:49.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-02-09T17:53:49.955+0000 7f4565d28700 0 --2- 172.21.15.99:0/3823670910 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4560113c40 0x7f456006f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-02-09T17:53:49.963 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:53:49.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:49.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:53:50.071 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:53:50.099 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:50.125 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:53:50.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:50.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:53:50.231 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:53:50.264 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:50.288 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:53:50.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:50.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:53:50.398 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:53:50.462 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:50.540 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:53:50.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:50.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:53:50.582 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:50 smithi162 ceph-mon[44304]: pgmap v1860: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 678 op/s; 2019/122520 objects degraded (1.648%); 10192/122520 objects misplaced (8.319%); 5.0 MiB/s, 81 objects/s recovering 2024-02-09T17:53:50.582 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:50 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3823670910' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:53:50.583 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:50 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:53:50.583 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is unset 2024-02-09T17:53:50.584 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-09T17:53:50.652 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:53:50.727 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:50 smithi099 ceph-mon[39420]: pgmap v1860: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 678 op/s; 2019/122520 objects degraded (1.648%); 10192/122520 objects misplaced (8.319%); 5.0 MiB/s, 81 objects/s recovering 2024-02-09T17:53:50.727 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:50 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3823670910' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:53:50.727 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:50 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:53:50.734 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:50.912 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:53:50.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:50 smithi179 ceph-mon[43975]: pgmap v1860: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 678 op/s; 2019/122520 objects degraded (1.648%); 10192/122520 objects misplaced (8.319%); 5.0 MiB/s, 81 objects/s recovering 2024-02-09T17:53:50.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:50 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3823670910' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:53:50.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:50 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:53:50.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:50.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:53:50.962 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 7 1 2024-02-09T17:53:51.024 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:53:51.054 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:51.084 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:53:51.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:51.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:53:51.190 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:53:51.260 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:51.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:51 smithi179 ceph-mon[43975]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:53:51.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:51 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:53:51.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:51 smithi179 ceph-mon[43975]: osdmap e749: 12 total, 12 up, 12 in 2024-02-09T17:53:51.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:51 smithi162 ceph-mon[44304]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:53:51.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:51 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:53:51.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:51 smithi162 ceph-mon[44304]: osdmap e749: 12 total, 12 up, 12 in 2024-02-09T17:53:51.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:51 smithi099 ceph-mon[39420]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:53:51.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:51 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:53:51.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:51 smithi099 ceph-mon[39420]: osdmap e749: 12 total, 12 up, 12 in 2024-02-09T17:53:52.483 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:53:52.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:52.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:53:52.590 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:53:52.664 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:52.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:52 smithi099 ceph-mon[39420]: pgmap v1862: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 581 op/s; 2083/119379 objects degraded (1.745%); 9784/119379 objects misplaced (8.196%); 5.4 MiB/s, 87 objects/s recovering 2024-02-09T17:53:52.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:52 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 2083/119379 objects degraded (1.745%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:53:52.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:52 smithi179 ceph-mon[43975]: pgmap v1862: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 581 op/s; 2083/119379 objects degraded (1.745%); 9784/119379 objects misplaced (8.196%); 5.4 MiB/s, 87 objects/s recovering 2024-02-09T17:53:52.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:52 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 2083/119379 objects degraded (1.745%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:53:52.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:52 smithi162 ceph-mon[44304]: pgmap v1862: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 581 op/s; 2083/119379 objects degraded (1.745%); 9784/119379 objects misplaced (8.196%); 5.4 MiB/s, 87 objects/s recovering 2024-02-09T17:53:52.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:52 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 2083/119379 objects degraded (1.745%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:53:53.142 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:53:53.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:53.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:53:53.260 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:53:53.303 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:53.483 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:53:53.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:53.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:53:53.603 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:53:53.640 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:53.698 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:53:53.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:53.704 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:53:53.805 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:53:53.858 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:53.880 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:53:53.888 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:53.888 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:53:53.943 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:53:53.950 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:53:53.989 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:53:54.039 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:54.194 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:53:54.238 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:53:54.289 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:53:54.303 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:53:54.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:54.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:53:54.349 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:53:54.395 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:53:54.422 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:53:54.473 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:54.496 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:53:54.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:54.504 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:53:54.605 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:53:54.635 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:54.658 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:53:54.660 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:54 smithi099 ceph-mon[39420]: pgmap v1863: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 812 op/s; 2132/113250 objects degraded (1.883%); 9587/113250 objects misplaced (8.465%); 5.6 MiB/s, 91 objects/s recovering 2024-02-09T17:53:54.660 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:54 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4128459553' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:53:54.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:54.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:53:54.697 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:53:54.705 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:53:54.738 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:53:54.764 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:53:54.775 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:53:54.794 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:54.805 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:53:54.815 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:53:54.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:54.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:53:54.841 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:53:54.866 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:53:54.869 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:53:54.870 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:53:54.870 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:53:54.870 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:53:54.870 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:53:54.870 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:53:54.870 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:53:54.870 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:53:54.870 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:53:54.870 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:53:54.870 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:53:54.919 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:53:54.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:54 smithi179 ceph-mon[43975]: pgmap v1863: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 812 op/s; 2132/113250 objects degraded (1.883%); 9587/113250 objects misplaced (8.465%); 5.6 MiB/s, 91 objects/s recovering 2024-02-09T17:53:54.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:54 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4128459553' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:53:54.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:54 smithi162 ceph-mon[44304]: pgmap v1863: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 812 op/s; 2132/113250 objects degraded (1.883%); 9587/113250 objects misplaced (8.465%); 5.6 MiB/s, 91 objects/s recovering 2024-02-09T17:53:54.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:54 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4128459553' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:53:54.945 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:54.966 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:53:54.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:54.973 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:53:55.074 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:53:55.115 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:55.137 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:53:55.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:55.145 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:53:55.196 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set noscrub 2024-02-09T17:53:55.246 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:53:55.341 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:55.459 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:53:55.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:55.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:53:55.565 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:53:55.638 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:55.715 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:53:55.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:55.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:53:55.818 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:53:55.890 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:56.926 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:56 smithi179 ceph-mon[43975]: pgmap v1864: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 512 op/s; 2178/112716 objects degraded (1.932%); 9407/112716 objects misplaced (8.346%); 4.4 MiB/s, 71 objects/s recovering 2024-02-09T17:53:56.926 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:56 smithi179 ceph-mon[43975]: 7.e scrub starts 2024-02-09T17:53:56.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:56 smithi162 ceph-mon[44304]: pgmap v1864: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 512 op/s; 2178/112716 objects degraded (1.932%); 9407/112716 objects misplaced (8.346%); 4.4 MiB/s, 71 objects/s recovering 2024-02-09T17:53:56.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:56 smithi162 ceph-mon[44304]: 7.e scrub starts 2024-02-09T17:53:57.005 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:56 smithi099 ceph-mon[39420]: pgmap v1864: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 512 op/s; 2178/112716 objects degraded (1.932%); 9407/112716 objects misplaced (8.346%); 4.4 MiB/s, 71 objects/s recovering 2024-02-09T17:53:57.005 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:56 smithi099 ceph-mon[39420]: 7.e scrub starts 2024-02-09T17:53:57.766 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:53:57.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:57.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:53:57.873 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:53:57.917 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:57.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:57 smithi162 ceph-mon[44304]: 7.e scrub ok 2024-02-09T17:53:57.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:57 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 2178/112716 objects degraded (1.932%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:53:57.943 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:53:57.945 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:57 smithi179 ceph-mon[43975]: 7.e scrub ok 2024-02-09T17:53:57.945 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:57 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 2178/112716 objects degraded (1.932%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:53:57.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:57.946 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:53:57.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:57 smithi099 ceph-mon[39420]: 7.e scrub ok 2024-02-09T17:53:57.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:57 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 2178/112716 objects degraded (1.932%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:53:58.047 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:53:58.081 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:58.104 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:53:58.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:58.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:53:58.211 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:53:58.249 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:58.270 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:53:58.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:58.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:53:58.375 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:53:58.402 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:58.420 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:53:58.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:53:58.423 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:53:58.524 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:53:58.573 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:53:58.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:53:58 smithi179 ceph-mon[43975]: pgmap v1865: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 674 op/s; 2275/105714 objects degraded (2.152%); 8841/105714 objects misplaced (8.363%); 5.9 MiB/s, 94 objects/s recovering 2024-02-09T17:53:58.886 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:53:58 smithi099 ceph-mon[39420]: pgmap v1865: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 674 op/s; 2275/105714 objects degraded (2.152%); 8841/105714 objects misplaced (8.363%); 5.9 MiB/s, 94 objects/s recovering 2024-02-09T17:53:58.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:53:58 smithi162 ceph-mon[44304]: pgmap v1865: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 674 op/s; 2275/105714 objects degraded (2.152%); 8841/105714 objects misplaced (8.363%); 5.9 MiB/s, 94 objects/s recovering 2024-02-09T17:54:00.189 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:54:00.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:00.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:54:00.300 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:54:00.329 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:00.361 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:54:00.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:00.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:54:00.466 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:54:00.497 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:00.526 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:54:00.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:00.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:54:00.633 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:54:00.743 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:00.894 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:54:00.901 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:00 smithi099 ceph-mon[39420]: pgmap v1866: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 643 op/s; 2321/103221 objects degraded (2.249%); 8648/103221 objects misplaced (8.378%); 5.5 MiB/s, 88 objects/s recovering 2024-02-09T17:54:00.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:00.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:54:01.003 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:54:01.030 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:00 smithi179 ceph-mon[43975]: pgmap v1866: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 643 op/s; 2321/103221 objects degraded (2.249%); 8648/103221 objects misplaced (8.378%); 5.5 MiB/s, 88 objects/s recovering 2024-02-09T17:54:01.032 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:01.052 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:54:01.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:01.059 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:54:01.160 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:54:01.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:00 smithi162 ceph-mon[44304]: pgmap v1866: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 643 op/s; 2321/103221 objects degraded (2.249%); 8648/103221 objects misplaced (8.378%); 5.5 MiB/s, 88 objects/s recovering 2024-02-09T17:54:01.246 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:01.391 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:54:01.749 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:54:01.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:01.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:54:01.863 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:54:01.906 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:01.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:01.946 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:54:01.946 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:54:01.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:01 smithi099 ceph-mon[39420]: pgmap v1867: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 661 op/s; 2343/100326 objects degraded (2.335%); 8231/100326 objects misplaced (8.204%); 6.2 MiB/s, 100 objects/s recovering 2024-02-09T17:54:02.047 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:54:02.135 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:02.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:01 smithi179 ceph-mon[43975]: pgmap v1867: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 661 op/s; 2343/100326 objects degraded (2.335%); 8231/100326 objects misplaced (8.204%); 6.2 MiB/s, 100 objects/s recovering 2024-02-09T17:54:02.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:01 smithi162 ceph-mon[44304]: pgmap v1867: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 661 op/s; 2343/100326 objects degraded (2.335%); 8231/100326 objects misplaced (8.204%); 6.2 MiB/s, 100 objects/s recovering 2024-02-09T17:54:02.773 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:54:02.807 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:54:02.807 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 20, 2024-02-09T17:54:02.807 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 600, 2024-02-09T17:54:02.808 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [ 2024-02-09T17:54:02.808 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.808 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25837.0:97721 6.2 e602/576 6:4259e16e:::benchmark_data_smithi179_126206_object97720:head v 602'6331, mlcod=602'6328)", 2024-02-09T17:54:02.808 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:46:48.615722+0000", 2024-02-09T17:54:02.808 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 434.18229558600001, 2024-02-09T17:54:02.808 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.215957661, 2024-02-09T17:54:02.808 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:54:02.808 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:54:02.808 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:54:02.808 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.808 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:54:02.808 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.615722+0000", 2024-02-09T17:54:02.808 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:54:02.809 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.809 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.809 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:54:02.809 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.615722+0000", 2024-02-09T17:54:02.809 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:54:02.809 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.809 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.809 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:54:02.809 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.615723+0000", 2024-02-09T17:54:02.809 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.5099999999999999e-07 2024-02-09T17:54:02.809 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.809 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.810 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:54:02.810 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.615752+0000", 2024-02-09T17:54:02.810 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.8892999999999999e-05 2024-02-09T17:54:02.810 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.810 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.810 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:54:02.810 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.615763+0000", 2024-02-09T17:54:02.810 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0886e-05 2024-02-09T17:54:02.810 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.810 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.810 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:54:02.810 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.615771+0000", 2024-02-09T17:54:02.810 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.8849999999999999e-06 2024-02-09T17:54:02.810 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.810 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.810 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:54:02.811 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.616139+0000", 2024-02-09T17:54:02.811 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00036847600000000001 2024-02-09T17:54:02.811 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.811 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.811 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:54:02.811 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.616160+0000", 2024-02-09T17:54:02.811 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0814000000000001e-05 2024-02-09T17:54:02.811 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.811 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.811 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:54:02.811 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.831618+0000", 2024-02-09T17:54:02.811 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.21545820600000001 2024-02-09T17:54:02.811 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.811 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.812 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:54:02.812 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.831680+0000", 2024-02-09T17:54:02.812 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.1749999999999997e-05 2024-02-09T17:54:02.812 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:54:02.812 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:54:02.812 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:54:02.812 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.812 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.812 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25837.0:97717 6.0 e602/576 6:0b4d0d71:::benchmark_data_smithi179_126206_object97716:head v 602'6088, mlcod=602'6086)", 2024-02-09T17:54:02.812 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:46:48.619308+0000", 2024-02-09T17:54:02.812 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 434.17871018900001, 2024-02-09T17:54:02.812 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.212388257, 2024-02-09T17:54:02.812 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:54:02.812 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:54:02.813 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:54:02.813 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.813 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:54:02.813 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.619308+0000", 2024-02-09T17:54:02.813 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:54:02.813 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.813 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.813 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:54:02.813 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.619308+0000", 2024-02-09T17:54:02.813 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:54:02.813 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.813 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.813 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:54:02.813 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.619309+0000", 2024-02-09T17:54:02.814 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.1800000000000005e-07 2024-02-09T17:54:02.814 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.814 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.814 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:54:02.814 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.619334+0000", 2024-02-09T17:54:02.814 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.5554e-05 2024-02-09T17:54:02.814 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.814 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.814 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:54:02.814 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.619346+0000", 2024-02-09T17:54:02.814 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1324e-05 2024-02-09T17:54:02.814 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.814 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.814 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:54:02.814 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.619353+0000", 2024-02-09T17:54:02.815 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.6839999999999996e-06 2024-02-09T17:54:02.815 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.815 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.815 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:54:02.815 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.619436+0000", 2024-02-09T17:54:02.815 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.2516999999999996e-05 2024-02-09T17:54:02.815 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.815 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.815 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:54:02.815 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.619455+0000", 2024-02-09T17:54:02.815 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.9636999999999999e-05 2024-02-09T17:54:02.815 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.815 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.815 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:54:02.815 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.831620+0000", 2024-02-09T17:54:02.815 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.21216502100000001 2024-02-09T17:54:02.816 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.816 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.816 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:54:02.816 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.831696+0000", 2024-02-09T17:54:02.816 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.5702000000000001e-05 2024-02-09T17:54:02.816 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:54:02.816 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:54:02.816 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:54:02.816 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.816 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.816 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25837.0:97724 6.0 e602/576 6:0fe71232:::benchmark_data_smithi179_126206_object97723:head v 602'6089, mlcod=602'6086)", 2024-02-09T17:54:02.816 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:46:48.622345+0000", 2024-02-09T17:54:02.816 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 434.17567290199997, 2024-02-09T17:54:02.816 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.22331884900000001, 2024-02-09T17:54:02.817 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:54:02.817 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:54:02.817 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:54:02.817 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.817 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:54:02.817 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.622345+0000", 2024-02-09T17:54:02.817 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:54:02.817 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.817 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.817 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:54:02.817 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.622345+0000", 2024-02-09T17:54:02.817 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:54:02.817 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.817 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.817 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:54:02.818 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.622346+0000", 2024-02-09T17:54:02.818 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.4e-07 2024-02-09T17:54:02.818 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.818 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.818 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:54:02.818 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.622371+0000", 2024-02-09T17:54:02.818 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.4890000000000001e-05 2024-02-09T17:54:02.818 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.818 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.818 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:54:02.818 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.831587+0000", 2024-02-09T17:54:02.818 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20921569600000001 2024-02-09T17:54:02.818 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.818 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.818 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:54:02.818 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.831620+0000", 2024-02-09T17:54:02.819 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.3118999999999999e-05 2024-02-09T17:54:02.819 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.819 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.819 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:54:02.819 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.831745+0000", 2024-02-09T17:54:02.819 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012559800000000001 2024-02-09T17:54:02.819 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.819 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.819 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:54:02.819 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.831771+0000", 2024-02-09T17:54:02.819 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.565e-05 2024-02-09T17:54:02.819 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.819 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.820 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:54:02.820 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.845604+0000", 2024-02-09T17:54:02.820 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.013832824000000001 2024-02-09T17:54:02.820 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.820 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.820 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:54:02.820 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.845664+0000", 2024-02-09T17:54:02.820 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.0232000000000002e-05 2024-02-09T17:54:02.820 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:54:02.820 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:54:02.820 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:54:02.820 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.820 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.820 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25837.0:97734 6.e e602/576 6:73d56b68:::benchmark_data_smithi179_126206_object97733:head v 602'5983, mlcod=602'5981)", 2024-02-09T17:54:02.820 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:46:48.628558+0000", 2024-02-09T17:54:02.821 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 434.16945983699998, 2024-02-09T17:54:02.821 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.21067872100000001, 2024-02-09T17:54:02.821 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:54:02.822 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:54:02.822 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:54:02.822 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.822 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:54:02.822 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.628558+0000", 2024-02-09T17:54:02.822 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:54:02.822 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.822 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.822 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:54:02.822 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.628558+0000", 2024-02-09T17:54:02.822 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:54:02.823 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.823 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.823 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:54:02.823 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.628559+0000", 2024-02-09T17:54:02.823 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1829999999999999e-06 2024-02-09T17:54:02.823 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.823 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.823 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:54:02.823 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.628613+0000", 2024-02-09T17:54:02.823 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3915000000000001e-05 2024-02-09T17:54:02.823 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.823 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.823 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:54:02.823 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.831591+0000", 2024-02-09T17:54:02.823 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20297726999999999 2024-02-09T17:54:02.824 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.824 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.824 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:54:02.824 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.831620+0000", 2024-02-09T17:54:02.824 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.9526000000000002e-05 2024-02-09T17:54:02.824 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.824 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.824 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:54:02.824 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.831691+0000", 2024-02-09T17:54:02.824 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.0726e-05 2024-02-09T17:54:02.824 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.824 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.824 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:54:02.824 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.831720+0000", 2024-02-09T17:54:02.825 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.8809000000000001e-05 2024-02-09T17:54:02.825 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.825 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.825 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:54:02.825 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.839194+0000", 2024-02-09T17:54:02.825 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0074739009999999998 2024-02-09T17:54:02.825 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.825 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.825 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:54:02.825 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:48.839237+0000", 2024-02-09T17:54:02.825 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.3390999999999998e-05 2024-02-09T17:54:02.825 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:54:02.825 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:54:02.825 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:54:02.825 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.826 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.826 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25837.0:103409 6.0 e604/576 6:0a3a4a2e:::benchmark_data_smithi179_126206_object103408:head v 604'6417, mlcod=604'6415)", 2024-02-09T17:54:02.826 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:46:53.626462+0000", 2024-02-09T17:54:02.826 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 429.17155653399999, 2024-02-09T17:54:02.826 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.16262810999999999, 2024-02-09T17:54:02.826 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:54:02.826 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:54:02.826 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:54:02.826 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.826 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:54:02.826 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.626462+0000", 2024-02-09T17:54:02.826 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:54:02.826 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.826 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.826 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:54:02.827 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.626462+0000", 2024-02-09T17:54:02.827 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:54:02.827 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.827 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.827 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:54:02.827 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.626462+0000", 2024-02-09T17:54:02.827 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.8100000000000002e-07 2024-02-09T17:54:02.827 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.827 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.827 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:54:02.827 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.626498+0000", 2024-02-09T17:54:02.827 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.5880999999999997e-05 2024-02-09T17:54:02.827 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.827 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.827 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:54:02.828 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.626509+0000", 2024-02-09T17:54:02.828 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0509000000000001e-05 2024-02-09T17:54:02.828 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.828 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.828 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:54:02.828 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.626516+0000", 2024-02-09T17:54:02.828 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.6149999999999999e-06 2024-02-09T17:54:02.828 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.828 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.828 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:54:02.828 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.626570+0000", 2024-02-09T17:54:02.828 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3569999999999997e-05 2024-02-09T17:54:02.828 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.828 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.828 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:54:02.829 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.626591+0000", 2024-02-09T17:54:02.829 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0669000000000001e-05 2024-02-09T17:54:02.829 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.829 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.829 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:54:02.829 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.789034+0000", 2024-02-09T17:54:02.829 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.162443646 2024-02-09T17:54:02.829 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.829 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.829 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:54:02.829 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.789090+0000", 2024-02-09T17:54:02.829 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.5439000000000002e-05 2024-02-09T17:54:02.829 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:54:02.829 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:54:02.829 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:54:02.830 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.830 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.830 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25837.0:103410 6.0 e604/576 6:0aa7216a:::benchmark_data_smithi179_126206_object103409:head v 604'6418, mlcod=604'6415)", 2024-02-09T17:54:02.830 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:46:53.626536+0000", 2024-02-09T17:54:02.830 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 429.17148161799997, 2024-02-09T17:54:02.830 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.165807126, 2024-02-09T17:54:02.830 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:54:02.830 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:54:02.830 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:54:02.830 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.830 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:54:02.830 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.626536+0000", 2024-02-09T17:54:02.830 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:54:02.830 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.830 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.831 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:54:02.831 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.626536+0000", 2024-02-09T17:54:02.831 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:54:02.831 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.831 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.831 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:54:02.831 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.626537+0000", 2024-02-09T17:54:02.831 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2300000000000002e-07 2024-02-09T17:54:02.831 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.831 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.831 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:54:02.831 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.626559+0000", 2024-02-09T17:54:02.831 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2011999999999999e-05 2024-02-09T17:54:02.831 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.831 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.832 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:54:02.832 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.626566+0000", 2024-02-09T17:54:02.832 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.2239999999999998e-06 2024-02-09T17:54:02.832 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.832 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.832 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:54:02.832 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.626572+0000", 2024-02-09T17:54:02.832 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.4160000000000003e-06 2024-02-09T17:54:02.832 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.832 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.832 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:54:02.832 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.778496+0000", 2024-02-09T17:54:02.832 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.15192410000000001 2024-02-09T17:54:02.832 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.832 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.833 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:54:02.833 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.778510+0000", 2024-02-09T17:54:02.833 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.397e-05 2024-02-09T17:54:02.833 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.833 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.833 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:54:02.833 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.792303+0000", 2024-02-09T17:54:02.833 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.013793144 2024-02-09T17:54:02.833 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.833 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.833 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:54:02.833 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.792344+0000", 2024-02-09T17:54:02.833 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.0837000000000002e-05 2024-02-09T17:54:02.833 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:54:02.833 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:54:02.834 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:54:02.834 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.834 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.834 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25837.0:103418 6.e e604/576 6:7d5b0857:::benchmark_data_smithi179_126206_object103417:head v 604'6355, mlcod=604'6352)", 2024-02-09T17:54:02.834 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:46:53.630861+0000", 2024-02-09T17:54:02.834 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 429.16715689799997, 2024-02-09T17:54:02.834 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.15047986099999999, 2024-02-09T17:54:02.834 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:54:02.834 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:54:02.834 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:54:02.834 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.834 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:54:02.834 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.630861+0000", 2024-02-09T17:54:02.834 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:54:02.835 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.835 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.835 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:54:02.835 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.630861+0000", 2024-02-09T17:54:02.835 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:54:02.835 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.835 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.835 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:54:02.835 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.630862+0000", 2024-02-09T17:54:02.835 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1209999999999999e-06 2024-02-09T17:54:02.835 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.835 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.835 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:54:02.835 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.630906+0000", 2024-02-09T17:54:02.835 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.3720000000000002e-05 2024-02-09T17:54:02.836 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.836 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.836 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:54:02.836 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.775204+0000", 2024-02-09T17:54:02.836 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.14429837100000001 2024-02-09T17:54:02.836 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.836 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.836 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:54:02.836 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.775231+0000", 2024-02-09T17:54:02.836 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.6798999999999999e-05 2024-02-09T17:54:02.836 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.836 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.836 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:54:02.836 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.775414+0000", 2024-02-09T17:54:02.837 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00018272699999999999 2024-02-09T17:54:02.837 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.837 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.837 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:54:02.837 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.775477+0000", 2024-02-09T17:54:02.837 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.3468999999999998e-05 2024-02-09T17:54:02.837 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.837 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.837 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:54:02.837 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.781290+0000", 2024-02-09T17:54:02.837 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0058122520000000004 2024-02-09T17:54:02.837 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.837 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.837 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:54:02.837 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:53.781341+0000", 2024-02-09T17:54:02.838 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1402e-05 2024-02-09T17:54:02.838 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:54:02.838 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:54:02.838 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:54:02.838 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.838 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.838 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25837.0:122698 6.f e612/576 6:f5828aff:::benchmark_data_smithi179_126206_object122697:head v 612'7469, mlcod=612'7467)", 2024-02-09T17:54:02.838 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:47:12.576978+0000", 2024-02-09T17:54:02.838 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 410.22104028199999, 2024-02-09T17:54:02.838 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.180264602, 2024-02-09T17:54:02.838 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:54:02.838 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:54:02.838 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:54:02.838 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.838 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:54:02.839 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.576978+0000", 2024-02-09T17:54:02.839 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:54:02.839 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.839 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.839 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:54:02.839 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.576978+0000", 2024-02-09T17:54:02.839 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:54:02.839 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.839 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.839 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:54:02.839 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.576979+0000", 2024-02-09T17:54:02.839 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.4799999999999997e-07 2024-02-09T17:54:02.839 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.839 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.839 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:54:02.839 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.577009+0000", 2024-02-09T17:54:02.840 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.0793999999999997e-05 2024-02-09T17:54:02.840 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.840 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.840 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:54:02.840 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.577021+0000", 2024-02-09T17:54:02.840 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1635e-05 2024-02-09T17:54:02.840 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.840 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.840 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:54:02.840 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.577028+0000", 2024-02-09T17:54:02.840 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.2509999999999997e-06 2024-02-09T17:54:02.840 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.840 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.841 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:54:02.841 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.577080+0000", 2024-02-09T17:54:02.841 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1913000000000003e-05 2024-02-09T17:54:02.841 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.841 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.841 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:54:02.841 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.577107+0000", 2024-02-09T17:54:02.841 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.6746000000000001e-05 2024-02-09T17:54:02.841 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.841 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.841 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:54:02.841 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.757182+0000", 2024-02-09T17:54:02.841 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.180074755 2024-02-09T17:54:02.841 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.841 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.841 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:54:02.842 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.757242+0000", 2024-02-09T17:54:02.842 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.0659999999999999e-05 2024-02-09T17:54:02.842 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:54:02.842 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:54:02.842 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:54:02.842 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.842 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.842 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25837.0:122699 6.e e612/576 6:704d9c8f:::benchmark_data_smithi179_126206_object122698:head v 612'7621, mlcod=612'7619)", 2024-02-09T17:54:02.842 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:47:12.577813+0000", 2024-02-09T17:54:02.842 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 410.22020484699999, 2024-02-09T17:54:02.842 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.19765761600000001, 2024-02-09T17:54:02.842 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:54:02.842 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:54:02.842 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:54:02.843 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.843 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:54:02.843 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.577813+0000", 2024-02-09T17:54:02.843 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:54:02.843 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.843 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.843 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:54:02.843 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.577813+0000", 2024-02-09T17:54:02.843 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:54:02.843 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.843 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.843 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:54:02.843 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.577814+0000", 2024-02-09T17:54:02.843 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.0999999999999998e-07 2024-02-09T17:54:02.844 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.844 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.844 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:54:02.844 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.577940+0000", 2024-02-09T17:54:02.844 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000126425 2024-02-09T17:54:02.844 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.844 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.844 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:54:02.844 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.754078+0000", 2024-02-09T17:54:02.844 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.17613795099999999 2024-02-09T17:54:02.844 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.844 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.844 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:54:02.844 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.754088+0000", 2024-02-09T17:54:02.844 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0127e-05 2024-02-09T17:54:02.845 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.845 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.845 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:54:02.845 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.754219+0000", 2024-02-09T17:54:02.845 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013053000000000001 2024-02-09T17:54:02.845 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.845 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.845 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:54:02.845 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.754239+0000", 2024-02-09T17:54:02.845 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0455999999999999e-05 2024-02-09T17:54:02.845 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.845 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.845 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:54:02.845 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.775427+0000", 2024-02-09T17:54:02.845 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.021187620000000001 2024-02-09T17:54:02.845 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.846 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.846 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:54:02.846 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.775471+0000", 2024-02-09T17:54:02.846 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.3797000000000001e-05 2024-02-09T17:54:02.846 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:54:02.846 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:54:02.846 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:54:02.846 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.846 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.846 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25837.0:122697 6.0 e612/576 6:0380994d:::benchmark_data_smithi179_126206_object122696:head v 612'7592, mlcod=612'7589)", 2024-02-09T17:54:02.846 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:47:12.582479+0000", 2024-02-09T17:54:02.846 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 410.215538756, 2024-02-09T17:54:02.846 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.19247419199999999, 2024-02-09T17:54:02.846 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:54:02.846 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:54:02.847 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:54:02.847 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.847 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:54:02.847 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.582479+0000", 2024-02-09T17:54:02.847 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:54:02.847 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.847 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.847 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:54:02.847 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.582479+0000", 2024-02-09T17:54:02.847 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:54:02.847 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.847 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.847 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:54:02.847 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.582481+0000", 2024-02-09T17:54:02.847 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.486e-06 2024-02-09T17:54:02.847 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.848 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.848 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:54:02.848 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.582506+0000", 2024-02-09T17:54:02.848 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.4762000000000001e-05 2024-02-09T17:54:02.848 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.848 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.848 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:54:02.848 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.752968+0000", 2024-02-09T17:54:02.848 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.170462893 2024-02-09T17:54:02.848 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.848 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.848 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:54:02.848 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.752983+0000", 2024-02-09T17:54:02.848 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4499e-05 2024-02-09T17:54:02.849 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.849 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.849 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:54:02.849 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.754864+0000", 2024-02-09T17:54:02.849 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0018807159999999999 2024-02-09T17:54:02.849 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.849 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.849 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:54:02.849 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.754884+0000", 2024-02-09T17:54:02.849 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0460000000000001e-05 2024-02-09T17:54:02.849 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.849 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.849 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:54:02.849 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.774886+0000", 2024-02-09T17:54:02.849 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.020001662 2024-02-09T17:54:02.850 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.850 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.850 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:54:02.850 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:47:12.774953+0000", 2024-02-09T17:54:02.850 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.7713999999999996e-05 2024-02-09T17:54:02.850 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:54:02.850 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:54:02.850 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:54:02.850 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.850 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.850 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.16440.0:83375 7.3 e686/661 7:c9c8e507:::benchmark_data_smithi179_135804_object83374:head v 686'5229, mlcod=686'5226)", 2024-02-09T17:54:02.850 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:51:18.489551+0000", 2024-02-09T17:54:02.850 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 164.30846677599999, 2024-02-09T17:54:02.850 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.099479135999999996, 2024-02-09T17:54:02.850 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:54:02.851 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:54:02.851 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:54:02.851 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.851 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:54:02.851 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.489551+0000", 2024-02-09T17:54:02.851 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:54:02.851 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.851 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.851 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:54:02.851 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.489551+0000", 2024-02-09T17:54:02.851 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:54:02.851 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.851 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.851 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:54:02.851 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.489552+0000", 2024-02-09T17:54:02.852 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0279999999999999e-06 2024-02-09T17:54:02.852 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.852 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.852 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:54:02.852 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.489574+0000", 2024-02-09T17:54:02.852 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2113999999999999e-05 2024-02-09T17:54:02.852 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.852 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.852 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:54:02.852 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.489584+0000", 2024-02-09T17:54:02.852 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.4269999999999997e-06 2024-02-09T17:54:02.852 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.852 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.852 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:54:02.852 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.489590+0000", 2024-02-09T17:54:02.853 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.3579999999999999e-06 2024-02-09T17:54:02.853 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.853 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.853 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:54:02.853 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.489644+0000", 2024-02-09T17:54:02.853 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3961000000000001e-05 2024-02-09T17:54:02.853 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.853 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.853 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:54:02.853 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.489665+0000", 2024-02-09T17:54:02.853 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0577e-05 2024-02-09T17:54:02.853 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.853 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.853 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:54:02.853 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.588987+0000", 2024-02-09T17:54:02.853 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.099321999999999994 2024-02-09T17:54:02.854 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.854 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.854 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:54:02.854 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.589030+0000", 2024-02-09T17:54:02.854 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.3671000000000003e-05 2024-02-09T17:54:02.854 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:54:02.854 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:54:02.854 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:54:02.854 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.854 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.854 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.16440.0:83376 7.8 e686/661 7:14268f94:::benchmark_data_smithi179_135804_object83375:head v 686'5198, mlcod=686'5194)", 2024-02-09T17:54:02.854 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:51:18.490376+0000", 2024-02-09T17:54:02.854 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 164.30764176599999, 2024-02-09T17:54:02.854 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.109104801, 2024-02-09T17:54:02.854 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:54:02.855 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:54:02.855 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:54:02.855 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.855 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:54:02.855 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.490376+0000", 2024-02-09T17:54:02.855 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:54:02.855 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.855 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.855 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:54:02.855 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.490376+0000", 2024-02-09T17:54:02.855 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:54:02.855 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.855 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.855 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:54:02.855 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.490377+0000", 2024-02-09T17:54:02.856 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.5499999999999997e-07 2024-02-09T17:54:02.856 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.856 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.856 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:54:02.856 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.490445+0000", 2024-02-09T17:54:02.856 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.8361999999999994e-05 2024-02-09T17:54:02.856 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.856 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.856 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:54:02.856 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.490455+0000", 2024-02-09T17:54:02.856 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.7999999999999993e-06 2024-02-09T17:54:02.856 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.856 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.856 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:54:02.857 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.490461+0000", 2024-02-09T17:54:02.857 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.28e-06 2024-02-09T17:54:02.857 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.857 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.857 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:54:02.857 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.589005+0000", 2024-02-09T17:54:02.857 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.098543447000000006 2024-02-09T17:54:02.857 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.857 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.857 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:54:02.857 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.589030+0000", 2024-02-09T17:54:02.857 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.4782000000000001e-05 2024-02-09T17:54:02.857 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.857 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.857 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:54:02.858 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.599426+0000", 2024-02-09T17:54:02.858 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.010396720999999999 2024-02-09T17:54:02.858 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.858 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.858 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:54:02.858 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.599481+0000", 2024-02-09T17:54:02.858 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.4654000000000002e-05 2024-02-09T17:54:02.858 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:54:02.858 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:54:02.858 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:54:02.858 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.858 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.858 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.16440.0:83381 7.8 e686/661 7:1d33db6a:::benchmark_data_smithi179_135804_object83380:head v 686'5199, mlcod=686'5196)", 2024-02-09T17:54:02.858 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:51:18.495720+0000", 2024-02-09T17:54:02.858 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 164.30229801499999, 2024-02-09T17:54:02.858 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.10379113299999999, 2024-02-09T17:54:02.859 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:54:02.859 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:54:02.859 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:54:02.859 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.859 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:54:02.859 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.495720+0000", 2024-02-09T17:54:02.859 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:54:02.859 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.859 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.859 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:54:02.859 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.495720+0000", 2024-02-09T17:54:02.859 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:54:02.859 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.859 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.859 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:54:02.860 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.495722+0000", 2024-02-09T17:54:02.860 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4890000000000001e-06 2024-02-09T17:54:02.860 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.860 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.860 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:54:02.860 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.495769+0000", 2024-02-09T17:54:02.860 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.7274000000000002e-05 2024-02-09T17:54:02.860 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.860 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.860 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:54:02.860 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.588978+0000", 2024-02-09T17:54:02.860 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.093209146000000007 2024-02-09T17:54:02.860 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.860 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.860 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:54:02.861 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.588992+0000", 2024-02-09T17:54:02.861 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4178e-05 2024-02-09T17:54:02.861 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.861 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.861 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:54:02.861 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.591707+0000", 2024-02-09T17:54:02.861 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0027146589999999999 2024-02-09T17:54:02.861 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.861 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.861 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:54:02.861 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.591728+0000", 2024-02-09T17:54:02.861 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0798999999999999e-05 2024-02-09T17:54:02.861 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.861 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.862 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:54:02.862 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.599487+0000", 2024-02-09T17:54:02.862 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0077592609999999999 2024-02-09T17:54:02.862 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.862 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.862 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:54:02.862 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.599511+0000", 2024-02-09T17:54:02.862 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.4326999999999999e-05 2024-02-09T17:54:02.862 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:54:02.862 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:54:02.862 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:54:02.862 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.862 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.862 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "MOSDPGPush(7.c 706/682 [PushOp(7:32f3405a:::benchmark_data_smithi179_135804_object46814:head, version: 671'2962, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:32f3405a:::benchmark_data_smithi179_135804_object46814:head@671'2962, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-02-09T17:54:02.862 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:52:10.321219+0000", 2024-02-09T17:54:02.863 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 112.47679863099999, 2024-02-09T17:54:02.863 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.151726903, 2024-02-09T17:54:02.863 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:54:02.863 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T17:54:02.863 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:54:02.863 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.863 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:54:02.863 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.321219+0000", 2024-02-09T17:54:02.863 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:54:02.863 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.863 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.863 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:54:02.863 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.321219+0000", 2024-02-09T17:54:02.863 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:54:02.863 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.864 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.864 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:54:02.864 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.321220+0000", 2024-02-09T17:54:02.864 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.9800000000000003e-07 2024-02-09T17:54:02.864 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.864 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.864 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:54:02.864 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.321276+0000", 2024-02-09T17:54:02.864 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.5442000000000002e-05 2024-02-09T17:54:02.864 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.864 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.864 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:54:02.864 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.321291+0000", 2024-02-09T17:54:02.864 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4966e-05 2024-02-09T17:54:02.865 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.865 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.865 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:54:02.865 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.321302+0000", 2024-02-09T17:54:02.865 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1117e-05 2024-02-09T17:54:02.865 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.865 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.865 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:54:02.865 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.321365+0000", 2024-02-09T17:54:02.865 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.3261000000000003e-05 2024-02-09T17:54:02.865 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.865 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.865 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:54:02.865 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.321377+0000", 2024-02-09T17:54:02.865 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1558e-05 2024-02-09T17:54:02.866 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.866 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.866 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:54:02.866 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.472946+0000", 2024-02-09T17:54:02.866 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.151569761 2024-02-09T17:54:02.866 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:54:02.866 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:54:02.866 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:54:02.866 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.866 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.866 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.16440.0:116779 7.3 e706/661 7:cf34c6cd:::benchmark_data_smithi179_135804_object8988:head v 706'7373, mlcod=706'7371)", 2024-02-09T17:54:02.866 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:52:10.322168+0000", 2024-02-09T17:54:02.866 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 112.475849622, 2024-02-09T17:54:02.866 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.16296088, 2024-02-09T17:54:02.866 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:54:02.866 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:54:02.867 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:54:02.867 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.867 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:54:02.867 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.322168+0000", 2024-02-09T17:54:02.867 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:54:02.867 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.867 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.867 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:54:02.867 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.322168+0000", 2024-02-09T17:54:02.867 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:54:02.867 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.867 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.867 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:54:02.867 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.322169+0000", 2024-02-09T17:54:02.867 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.0800000000000005e-07 2024-02-09T17:54:02.868 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.868 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.868 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:54:02.868 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.322174+0000", 2024-02-09T17:54:02.868 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.9570000000000001e-06 2024-02-09T17:54:02.868 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.868 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.868 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:54:02.868 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.322179+0000", 2024-02-09T17:54:02.868 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.0259999999999998e-06 2024-02-09T17:54:02.868 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.868 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.868 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:54:02.868 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.322183+0000", 2024-02-09T17:54:02.869 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.4549999999999997e-06 2024-02-09T17:54:02.869 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.869 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.869 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:54:02.869 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.472977+0000", 2024-02-09T17:54:02.869 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.15079350799999999 2024-02-09T17:54:02.869 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.869 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.869 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:54:02.869 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.473001+0000", 2024-02-09T17:54:02.869 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.4403999999999999e-05 2024-02-09T17:54:02.869 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.869 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.869 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:54:02.869 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.485098+0000", 2024-02-09T17:54:02.870 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.012096361999999999 2024-02-09T17:54:02.870 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.870 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.870 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:54:02.870 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.485129+0000", 2024-02-09T17:54:02.870 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.1659999999999998e-05 2024-02-09T17:54:02.870 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:54:02.870 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:54:02.870 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:54:02.870 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.870 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.870 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.16440.0:116799 7.a e706/661 7:5a8c05a2:::benchmark_data_smithi179_135804_object9008:head v 706'7334, mlcod=706'7332)", 2024-02-09T17:54:02.870 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:52:10.349258+0000", 2024-02-09T17:54:02.870 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 112.448760264, 2024-02-09T17:54:02.870 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.139991741, 2024-02-09T17:54:02.871 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:54:02.871 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:54:02.871 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:54:02.871 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.871 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:54:02.871 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.349258+0000", 2024-02-09T17:54:02.871 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:54:02.871 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.871 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.871 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:54:02.871 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.349258+0000", 2024-02-09T17:54:02.871 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:54:02.871 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.871 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.872 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:54:02.872 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.349259+0000", 2024-02-09T17:54:02.872 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2500000000000001e-06 2024-02-09T17:54:02.872 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.872 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.872 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:54:02.872 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.349269+0000", 2024-02-09T17:54:02.872 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.9119999999999993e-06 2024-02-09T17:54:02.872 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.872 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.872 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:54:02.872 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.472952+0000", 2024-02-09T17:54:02.872 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.123683038 2024-02-09T17:54:02.872 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.872 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.873 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:54:02.873 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.472962+0000", 2024-02-09T17:54:02.873 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.8830000000000001e-06 2024-02-09T17:54:02.873 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.873 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.873 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:54:02.873 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.473014+0000", 2024-02-09T17:54:02.873 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1987999999999998e-05 2024-02-09T17:54:02.873 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.873 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.873 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:54:02.873 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.473035+0000", 2024-02-09T17:54:02.873 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0667000000000001e-05 2024-02-09T17:54:02.873 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.873 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.874 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:54:02.874 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.489212+0000", 2024-02-09T17:54:02.874 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.016177252 2024-02-09T17:54:02.874 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.874 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.874 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:54:02.874 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.489250+0000", 2024-02-09T17:54:02.874 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.7750999999999998e-05 2024-02-09T17:54:02.874 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:54:02.874 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:54:02.874 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:54:02.874 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.874 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.874 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.16440.0:116805 7.8 e706/661 7:1a85ab55:::benchmark_data_smithi179_135804_object9014:head v 706'7283, mlcod=706'7281)", 2024-02-09T17:54:02.874 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:52:10.358900+0000", 2024-02-09T17:54:02.875 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 112.439118405, 2024-02-09T17:54:02.875 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.121220444, 2024-02-09T17:54:02.875 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:54:02.875 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:54:02.875 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:54:02.875 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.875 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:54:02.875 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.358900+0000", 2024-02-09T17:54:02.875 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:54:02.875 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.875 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.875 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:54:02.875 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.358900+0000", 2024-02-09T17:54:02.875 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:54:02.875 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.876 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.876 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:54:02.876 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.358901+0000", 2024-02-09T17:54:02.876 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.7370000000000001e-06 2024-02-09T17:54:02.876 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.876 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.876 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:54:02.876 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.358910+0000", 2024-02-09T17:54:02.876 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.4630000000000001e-06 2024-02-09T17:54:02.876 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.876 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.876 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:54:02.876 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.472964+0000", 2024-02-09T17:54:02.876 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.11405451799999999 2024-02-09T17:54:02.877 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.877 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.877 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:54:02.877 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.472974+0000", 2024-02-09T17:54:02.877 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.4129999999999995e-06 2024-02-09T17:54:02.877 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.877 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.877 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:54:02.877 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.473037+0000", 2024-02-09T17:54:02.877 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.3367999999999994e-05 2024-02-09T17:54:02.877 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.877 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.877 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:54:02.877 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.473059+0000", 2024-02-09T17:54:02.877 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2010000000000001e-05 2024-02-09T17:54:02.877 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.879 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.879 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:54:02.879 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.480071+0000", 2024-02-09T17:54:02.879 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0070118530000000002 2024-02-09T17:54:02.879 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.879 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.879 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:54:02.879 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.480120+0000", 2024-02-09T17:54:02.879 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.9082000000000002e-05 2024-02-09T17:54:02.880 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:54:02.880 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:54:02.880 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:54:02.880 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.880 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.880 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.16440.0:165021 7.a e747/661 7:56d9fdc9:::benchmark_data_smithi179_135804_object57230:head v 747'10337, mlcod=747'10335)", 2024-02-09T17:54:02.880 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:53:32.191017+0000", 2024-02-09T17:54:02.880 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 30.607000871, 2024-02-09T17:54:02.880 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.051436793000000001, 2024-02-09T17:54:02.880 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:54:02.880 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:54:02.880 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:54:02.880 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.880 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:54:02.880 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:32.191017+0000", 2024-02-09T17:54:02.881 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:54:02.881 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.881 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.881 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:54:02.881 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:32.191017+0000", 2024-02-09T17:54:02.881 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:54:02.881 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.881 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.881 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:54:02.881 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:32.191018+0000", 2024-02-09T17:54:02.881 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.082e-06 2024-02-09T17:54:02.881 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.881 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.881 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:54:02.881 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:32.191027+0000", 2024-02-09T17:54:02.882 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.7080000000000003e-06 2024-02-09T17:54:02.882 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.882 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.882 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:54:02.882 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:32.191041+0000", 2024-02-09T17:54:02.882 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3903000000000001e-05 2024-02-09T17:54:02.882 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.882 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.882 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:54:02.882 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:32.191049+0000", 2024-02-09T17:54:02.882 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.2700000000000004e-06 2024-02-09T17:54:02.882 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.882 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.882 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:54:02.882 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:32.191149+0000", 2024-02-09T17:54:02.883 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.9596e-05 2024-02-09T17:54:02.883 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.883 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.883 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:54:02.883 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:32.191170+0000", 2024-02-09T17:54:02.883 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0896000000000001e-05 2024-02-09T17:54:02.883 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.883 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.883 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:54:02.883 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:32.242416+0000", 2024-02-09T17:54:02.883 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.051246660999999999 2024-02-09T17:54:02.883 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.883 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.883 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:54:02.883 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:32.242454+0000", 2024-02-09T17:54:02.884 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.7676999999999999e-05 2024-02-09T17:54:02.884 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:54:02.884 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:54:02.884 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:54:02.884 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.884 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.884 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.16440.0:180990 7.e 7:7ea734bc:::benchmark_data_smithi179_135804_object73199:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e749)", 2024-02-09T17:54:02.884 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:53:56.373358+0000", 2024-02-09T17:54:02.884 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 6.4246602230000001, 2024-02-09T17:54:02.884 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.055614563999999998, 2024-02-09T17:54:02.884 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:54:02.884 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:54:02.884 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:54:02.884 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.16440", 2024-02-09T17:54:02.885 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:54:02.885 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 180990 2024-02-09T17:54:02.885 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.885 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:54:02.885 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.885 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:54:02.885 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:56.373358+0000", 2024-02-09T17:54:02.885 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:54:02.885 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.885 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.885 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:54:02.885 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:56.373358+0000", 2024-02-09T17:54:02.885 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:54:02.885 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.885 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.885 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:54:02.886 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:56.373359+0000", 2024-02-09T17:54:02.886 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.48e-06 2024-02-09T17:54:02.886 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.886 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.886 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:54:02.886 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:56.373364+0000", 2024-02-09T17:54:02.886 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1850000000000003e-06 2024-02-09T17:54:02.886 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.886 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.886 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:54:02.886 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:56.373378+0000", 2024-02-09T17:54:02.886 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3176999999999999e-05 2024-02-09T17:54:02.886 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.886 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.886 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:54:02.886 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:56.373386+0000", 2024-02-09T17:54:02.887 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.1599999999999998e-06 2024-02-09T17:54:02.887 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.887 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.887 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:54:02.887 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:56.373594+0000", 2024-02-09T17:54:02.887 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000207918 2024-02-09T17:54:02.887 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.887 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.887 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:54:02.887 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:56.373709+0000", 2024-02-09T17:54:02.887 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000115402 2024-02-09T17:54:02.887 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.887 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.887 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 9,10", 2024-02-09T17:54:02.887 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:56.373791+0000", 2024-02-09T17:54:02.888 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.1882e-05 2024-02-09T17:54:02.888 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.888 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.888 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:54:02.888 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:56.423533+0000", 2024-02-09T17:54:02.888 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.049742197000000002 2024-02-09T17:54:02.888 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.888 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.888 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:54:02.888 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:56.423604+0000", 2024-02-09T17:54:02.888 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.0786999999999998e-05 2024-02-09T17:54:02.888 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.888 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.888 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:54:02.888 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:56.428868+0000", 2024-02-09T17:54:02.889 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0052643789999999996 2024-02-09T17:54:02.889 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.889 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.889 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:54:02.889 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:56.428919+0000", 2024-02-09T17:54:02.889 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.0359999999999999e-05 2024-02-09T17:54:02.889 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.889 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.889 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:54:02.889 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:56.428972+0000", 2024-02-09T17:54:02.889 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3637000000000002e-05 2024-02-09T17:54:02.889 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:54:02.889 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:54:02.889 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:54:02.889 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.890 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.890 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.16440.0:180992 7.e 7:798e33c2:::benchmark_data_smithi179_135804_object73201:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e749)", 2024-02-09T17:54:02.890 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:53:56.373719+0000", 2024-02-09T17:54:02.890 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 6.4242987100000004, 2024-02-09T17:54:02.890 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.065513932999999996, 2024-02-09T17:54:02.890 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:54:02.890 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:54:02.890 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:54:02.890 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.16440", 2024-02-09T17:54:02.890 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:54:02.890 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 180992 2024-02-09T17:54:02.890 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.890 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:54:02.890 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.890 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:54:02.891 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:56.373719+0000", 2024-02-09T17:54:02.891 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:54:02.891 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.891 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.891 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:54:02.891 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:56.373719+0000", 2024-02-09T17:54:02.891 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:54:02.891 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.891 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.891 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:54:02.891 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:56.373720+0000", 2024-02-09T17:54:02.891 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.6499999999999999e-07 2024-02-09T17:54:02.891 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.891 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.891 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:54:02.892 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:56.373723+0000", 2024-02-09T17:54:02.892 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.4709999999999999e-06 2024-02-09T17:54:02.892 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.892 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.892 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:54:02.892 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:56.373732+0000", 2024-02-09T17:54:02.892 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.2830000000000001e-06 2024-02-09T17:54:02.892 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.892 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.892 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:54:02.892 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:56.373737+0000", 2024-02-09T17:54:02.892 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1320000000000002e-06 2024-02-09T17:54:02.892 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.892 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.892 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:54:02.893 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:56.423647+0000", 2024-02-09T17:54:02.893 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.049910383000000003 2024-02-09T17:54:02.893 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.893 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.893 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:54:02.893 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:56.423785+0000", 2024-02-09T17:54:02.893 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013814200000000001 2024-02-09T17:54:02.893 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.893 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.893 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 9,10", 2024-02-09T17:54:02.893 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:56.423923+0000", 2024-02-09T17:54:02.893 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013731999999999999 2024-02-09T17:54:02.893 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.893 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.894 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:54:02.894 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:56.438991+0000", 2024-02-09T17:54:02.894 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.015068332 2024-02-09T17:54:02.894 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.894 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.894 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:54:02.894 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:56.439089+0000", 2024-02-09T17:54:02.894 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.8023999999999995e-05 2024-02-09T17:54:02.894 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.894 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.894 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:54:02.894 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:56.439130+0000", 2024-02-09T17:54:02.894 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.0754999999999999e-05 2024-02-09T17:54:02.894 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.894 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.894 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:54:02.895 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:56.439181+0000", 2024-02-09T17:54:02.895 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1419999999999999e-05 2024-02-09T17:54:02.895 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:54:02.895 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:54:02.895 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:54:02.895 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:56.439233+0000", 2024-02-09T17:54:02.895 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2006000000000003e-05 2024-02-09T17:54:02.895 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:54:02.895 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:54:02.895 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:54:02.895 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:54:02.895 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:54:02.895 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:54:02.896 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:02.896 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:54:02.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:54:02.997 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:54:03.026 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:02 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 2343/100326 objects degraded (2.335%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-09T17:54:03.026 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:02 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:54:03.026 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:02 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:54:03.028 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:03.052 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:54:03.056 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:03.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:54:03.158 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:54:03.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:02 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 2343/100326 objects degraded (2.335%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-09T17:54:03.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:02 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:54:03.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:02 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:54:03.187 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:03.215 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:54:03.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:03.220 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:54:03.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:02 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 2343/100326 objects degraded (2.335%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-09T17:54:03.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:02 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:54:03.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:02 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:54:03.322 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:54:03.375 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:03.480 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:54:03.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:03.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:54:03.593 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:54:03.625 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:03.661 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:54:03.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:03.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:54:03.766 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:54:03.817 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:03.819 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.7 to 1 (10000) 2024-02-09T17:54:03.842 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:54:03.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:03.845 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:54:03.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:03 smithi162 ceph-mon[44304]: osdmap e750: 12 total, 12 up, 12 in 2024-02-09T17:54:03.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:03 smithi162 ceph-mon[44304]: pgmap v1869: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 636 op/s; 2368/94140 objects degraded (2.515%); 8037/94140 objects misplaced (8.537%); 5.8 MiB/s, 93 objects/s recovering 2024-02-09T17:54:03.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:03 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3441520215' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-09T17:54:03.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:03 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-09T17:54:03.945 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:54:04.029 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:04.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:03 smithi179 ceph-mon[43975]: osdmap e750: 12 total, 12 up, 12 in 2024-02-09T17:54:04.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:03 smithi179 ceph-mon[43975]: pgmap v1869: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 636 op/s; 2368/94140 objects degraded (2.515%); 8037/94140 objects misplaced (8.537%); 5.8 MiB/s, 93 objects/s recovering 2024-02-09T17:54:04.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:03 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3441520215' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-09T17:54:04.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:03 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-09T17:54:04.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:03 smithi099 ceph-mon[39420]: osdmap e750: 12 total, 12 up, 12 in 2024-02-09T17:54:04.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:03 smithi099 ceph-mon[39420]: pgmap v1869: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 636 op/s; 2368/94140 objects degraded (2.515%); 8037/94140 objects misplaced (8.537%); 5.8 MiB/s, 93 objects/s recovering 2024-02-09T17:54:04.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:03 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3441520215' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-09T17:54:04.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:03 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-09T17:54:05.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:04 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-09T17:54:05.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:04 smithi179 ceph-mon[43975]: osdmap e751: 12 total, 12 up, 12 in 2024-02-09T17:54:05.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:04 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-09T17:54:05.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:04 smithi162 ceph-mon[44304]: osdmap e751: 12 total, 12 up, 12 in 2024-02-09T17:54:05.234 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:04 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-09T17:54:05.234 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:04 smithi099 ceph-mon[39420]: osdmap e751: 12 total, 12 up, 12 in 2024-02-09T17:54:05.267 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-09T17:54:05.712 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:54:05.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:05.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:54:05.733 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 8 1 2024-02-09T17:54:05.820 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:54:05.904 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:05.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:05 smithi099 ceph-mon[39420]: pgmap v1871: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 507 op/s; 2419/93537 objects degraded (2.586%); 7844/93537 objects misplaced (8.386%); 4.9 MiB/s, 79 objects/s recovering 2024-02-09T17:54:06.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:05 smithi179 ceph-mon[43975]: pgmap v1871: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 507 op/s; 2419/93537 objects degraded (2.586%); 7844/93537 objects misplaced (8.386%); 4.9 MiB/s, 79 objects/s recovering 2024-02-09T17:54:06.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:05 smithi162 ceph-mon[44304]: pgmap v1871: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 507 op/s; 2419/93537 objects degraded (2.586%); 7844/93537 objects misplaced (8.386%); 4.9 MiB/s, 79 objects/s recovering 2024-02-09T17:54:06.179 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:54:06.179 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:54:06.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:06.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:54:06.285 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:54:06.315 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:06.344 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:54:06.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:06.350 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:54:06.451 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:54:06.488 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:06.522 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:54:06.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:06.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:54:06.626 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:54:06.716 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:06.861 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:06 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:54:06.861 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:06 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:54:06.948 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:54:06.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:06.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:54:07.061 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:54:07.138 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:06 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:54:07.138 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:06 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:54:07.146 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:07.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:06 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:54:07.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:06 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:54:07.844 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:54:07.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:07.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:54:07.956 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:54:07.996 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:08.017 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:54:08.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:08.020 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:54:08.121 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:54:08.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:07 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 2419/93537 objects degraded (2.586%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-09T17:54:08.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:07 smithi179 ceph-mon[43975]: osdmap e752: 12 total, 12 up, 12 in 2024-02-09T17:54:08.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:07 smithi179 ceph-mon[43975]: pgmap v1873: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 800 op/s; 2517/85914 objects degraded (2.930%); 7207/85914 objects misplaced (8.389%); 6.3 MiB/s, 102 objects/s recovering 2024-02-09T17:54:08.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:07 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 2419/93537 objects degraded (2.586%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-09T17:54:08.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:07 smithi162 ceph-mon[44304]: osdmap e752: 12 total, 12 up, 12 in 2024-02-09T17:54:08.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:07 smithi162 ceph-mon[44304]: pgmap v1873: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 800 op/s; 2517/85914 objects degraded (2.930%); 7207/85914 objects misplaced (8.389%); 6.3 MiB/s, 102 objects/s recovering 2024-02-09T17:54:08.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:07 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 2419/93537 objects degraded (2.586%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-09T17:54:08.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:07 smithi099 ceph-mon[39420]: osdmap e752: 12 total, 12 up, 12 in 2024-02-09T17:54:08.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:07 smithi099 ceph-mon[39420]: pgmap v1873: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 800 op/s; 2517/85914 objects degraded (2.930%); 7207/85914 objects misplaced (8.389%); 6.3 MiB/s, 102 objects/s recovering 2024-02-09T17:54:08.260 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:08.985 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is set 2024-02-09T17:54:09.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:09 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2684717481' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:54:09.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:09 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2684717481' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:54:09.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:08 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2684717481' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:54:09.861 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:54:09.881 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:54:09.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:09.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:54:09.991 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:54:10.029 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:10.064 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:54:10.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:10.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:54:10.170 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:54:10.209 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:10.237 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:54:10.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:10.244 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:54:10.248 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:10 smithi099 ceph-mon[39420]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:54:10.248 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:10 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2684717481' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:54:10.248 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:10 smithi099 ceph-mon[39420]: osdmap e753: 12 total, 12 up, 12 in 2024-02-09T17:54:10.248 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:10 smithi099 ceph-mon[39420]: pgmap v1875: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 590 op/s; 2583/83508 objects degraded (3.093%); 7012/83508 objects misplaced (8.397%); 5.9 MiB/s, 95 objects/s recovering 2024-02-09T17:54:10.344 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:54:10.400 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:10 smithi179 ceph-mon[43975]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:54:10.400 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:10 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2684717481' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:54:10.400 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:10 smithi179 ceph-mon[43975]: osdmap e753: 12 total, 12 up, 12 in 2024-02-09T17:54:10.400 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:10 smithi179 ceph-mon[43975]: pgmap v1875: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 590 op/s; 2583/83508 objects degraded (3.093%); 7012/83508 objects misplaced (8.397%); 5.9 MiB/s, 95 objects/s recovering 2024-02-09T17:54:10.402 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:10.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:10 smithi162 ceph-mon[44304]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:54:10.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:10 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2684717481' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:54:10.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:10 smithi162 ceph-mon[44304]: osdmap e753: 12 total, 12 up, 12 in 2024-02-09T17:54:10.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:10 smithi162 ceph-mon[44304]: pgmap v1875: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 590 op/s; 2583/83508 objects degraded (3.093%); 7012/83508 objects misplaced (8.397%); 5.9 MiB/s, 95 objects/s recovering 2024-02-09T17:54:10.427 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:54:10.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:10.431 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:54:10.531 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:54:10.564 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:10.587 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:54:10.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:10.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:54:10.691 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:54:10.722 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:10.756 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:54:10.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:10.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:54:10.860 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:54:10.895 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:10.923 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:54:10.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:10.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:54:11.029 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:54:11.064 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:11.091 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:54:11.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:11.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:54:11.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:11 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:54:11.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:11 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:54:11.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:11 smithi162 ceph-mon[44304]: osdmap e754: 12 total, 12 up, 12 in 2024-02-09T17:54:11.197 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:54:11.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:11 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:54:11.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:11 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:54:11.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:11 smithi099 ceph-mon[39420]: osdmap e754: 12 total, 12 up, 12 in 2024-02-09T17:54:11.312 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:11.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:11 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:54:11.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:11 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:54:11.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:11 smithi179 ceph-mon[43975]: osdmap e754: 12 total, 12 up, 12 in 2024-02-09T17:54:11.600 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:54:11.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:11.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:54:11.709 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:54:11.740 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:11.774 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:54:11.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:11.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:54:11.880 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:54:11.909 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:11.932 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:54:11.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:11.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:54:12.023 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:12 smithi162 ceph-mon[44304]: pgmap v1877: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 736 op/s; 2616/80259 objects degraded (3.259%); 6604/80259 objects misplaced (8.228%); 6.1 MiB/s, 98 objects/s recovering 2024-02-09T17:54:12.023 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:12 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 2583/83508 objects degraded (3.093%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-09T17:54:12.024 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:12 smithi162 ceph-mon[44304]: osdmap e755: 12 total, 12 up, 12 in 2024-02-09T17:54:12.036 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:54:12.074 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:12.096 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:54:12.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:12.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:54:12.202 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:54:12.234 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:12.262 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:54:12.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:12.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:54:12.279 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:12 smithi099 ceph-mon[39420]: pgmap v1877: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 736 op/s; 2616/80259 objects degraded (3.259%); 6604/80259 objects misplaced (8.228%); 6.1 MiB/s, 98 objects/s recovering 2024-02-09T17:54:12.279 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:12 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 2583/83508 objects degraded (3.093%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-09T17:54:12.279 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:12 smithi099 ceph-mon[39420]: osdmap e755: 12 total, 12 up, 12 in 2024-02-09T17:54:12.365 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:54:12.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:12 smithi179 ceph-mon[43975]: pgmap v1877: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 736 op/s; 2616/80259 objects degraded (3.259%); 6604/80259 objects misplaced (8.228%); 6.1 MiB/s, 98 objects/s recovering 2024-02-09T17:54:12.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:12 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 2583/83508 objects degraded (3.093%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-09T17:54:12.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:12 smithi179 ceph-mon[43975]: osdmap e755: 12 total, 12 up, 12 in 2024-02-09T17:54:12.443 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:13.316 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:54:13.316 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:54:13.317 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:13 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:54:13.317 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:13 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:54:13.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:13.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:54:13.330 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-09T17:54:13.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:13 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:54:13.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:13 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:54:13.426 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:54:13.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:13 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:54:13.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:13 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:54:13.454 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:13.481 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:54:13.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:13.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:54:13.590 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:54:13.653 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:14.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:14 smithi179 ceph-mon[43975]: pgmap v1879: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 672 op/s; 2662/73779 objects degraded (3.608%); 6402/73779 objects misplaced (8.677%); 4.6 MiB/s, 74 objects/s recovering 2024-02-09T17:54:14.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:14 smithi162 ceph-mon[44304]: pgmap v1879: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 672 op/s; 2662/73779 objects degraded (3.608%); 6402/73779 objects misplaced (8.677%); 4.6 MiB/s, 74 objects/s recovering 2024-02-09T17:54:14.440 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:14 smithi099 ceph-mon[39420]: pgmap v1879: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 672 op/s; 2662/73779 objects degraded (3.608%); 6402/73779 objects misplaced (8.677%); 4.6 MiB/s, 74 objects/s recovering 2024-02-09T17:54:16.078 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.8 to 1 (10000) 2024-02-09T17:54:16.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:16 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:54:16.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:16 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:54:16.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:16 smithi179 ceph-mon[43975]: osdmap e756: 12 total, 12 up, 12 in 2024-02-09T17:54:16.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:16 smithi179 ceph-mon[43975]: pgmap v1881: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 576 op/s; 2734/73125 objects degraded (3.739%); 6185/73125 objects misplaced (8.458%); 4.8 MiB/s, 77 objects/s recovering 2024-02-09T17:54:16.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:16 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/831347615' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-02-09T17:54:16.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:16 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-02-09T17:54:16.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:16 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:54:16.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:16 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:54:16.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:16 smithi162 ceph-mon[44304]: osdmap e756: 12 total, 12 up, 12 in 2024-02-09T17:54:16.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:16 smithi162 ceph-mon[44304]: pgmap v1881: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 576 op/s; 2734/73125 objects degraded (3.739%); 6185/73125 objects misplaced (8.458%); 4.8 MiB/s, 77 objects/s recovering 2024-02-09T17:54:16.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:16 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/831347615' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-02-09T17:54:16.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:16 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-02-09T17:54:16.506 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:16 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:54:16.506 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:16 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:54:16.506 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:16 smithi099 ceph-mon[39420]: osdmap e756: 12 total, 12 up, 12 in 2024-02-09T17:54:16.506 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:16 smithi099 ceph-mon[39420]: pgmap v1881: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 576 op/s; 2734/73125 objects degraded (3.739%); 6185/73125 objects misplaced (8.458%); 4.8 MiB/s, 77 objects/s recovering 2024-02-09T17:54:16.506 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:16 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/831347615' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-02-09T17:54:16.506 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:16 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-02-09T17:54:16.669 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:54:16.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:16.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:54:16.775 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:54:16.803 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:16.826 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:54:16.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:16.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:54:16.933 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:54:16.965 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:16.992 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:54:16.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:16.998 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:54:17.076 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:17 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-02-09T17:54:17.076 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:17 smithi162 ceph-mon[44304]: osdmap e757: 12 total, 12 up, 12 in 2024-02-09T17:54:17.076 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:17 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:54:17.076 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:17 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:54:17.076 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:17 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 2734/73125 objects degraded (3.739%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-09T17:54:17.100 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:54:17.134 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:17.161 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:54:17.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:17.170 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:54:17.271 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:54:17.296 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:17 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-02-09T17:54:17.296 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:17 smithi179 ceph-mon[43975]: osdmap e757: 12 total, 12 up, 12 in 2024-02-09T17:54:17.296 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:17 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:54:17.296 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:17 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:54:17.296 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:17 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 2734/73125 objects degraded (3.739%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-09T17:54:17.298 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:17.330 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:54:17.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:17.338 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:54:17.349 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:17 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-02-09T17:54:17.349 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:17 smithi099 ceph-mon[39420]: osdmap e757: 12 total, 12 up, 12 in 2024-02-09T17:54:17.349 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:17 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:54:17.350 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:17 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:54:17.350 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:17 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 2734/73125 objects degraded (3.739%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-09T17:54:17.438 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:54:17.472 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:17.502 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:54:17.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:17.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:54:17.610 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:54:17.712 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:18.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:18 smithi179 ceph-mon[43975]: osdmap e758: 12 total, 12 up, 12 in 2024-02-09T17:54:18.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:18 smithi179 ceph-mon[43975]: pgmap v1884: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 750 op/s; 2877/65235 objects degraded (4.410%); 5514/65235 objects misplaced (8.453%); 5.5 MiB/s, 89 objects/s recovering 2024-02-09T17:54:18.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:18 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1133891310' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:54:18.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:18 smithi162 ceph-mon[44304]: osdmap e758: 12 total, 12 up, 12 in 2024-02-09T17:54:18.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:18 smithi162 ceph-mon[44304]: pgmap v1884: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 750 op/s; 2877/65235 objects degraded (4.410%); 5514/65235 objects misplaced (8.453%); 5.5 MiB/s, 89 objects/s recovering 2024-02-09T17:54:18.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:18 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1133891310' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:54:18.470 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:18 smithi099 ceph-mon[39420]: osdmap e758: 12 total, 12 up, 12 in 2024-02-09T17:54:18.471 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:18 smithi099 ceph-mon[39420]: pgmap v1884: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 750 op/s; 2877/65235 objects degraded (4.410%); 5514/65235 objects misplaced (8.453%); 5.5 MiB/s, 89 objects/s recovering 2024-02-09T17:54:18.471 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:18 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1133891310' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:54:18.538 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 9 1 2024-02-09T17:54:18.853 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:54:18.854 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:54:18.854 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:54:18.854 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:54:18.854 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:54:18.854 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:54:18.854 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:54:18.854 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:54:18.854 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:54:18.854 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:54:18.854 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:54:18.854 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:54:18.855 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:54:18.855 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:54:18.855 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:54:18.855 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:54:18.855 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:54:18.855 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:54:18.855 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:54:18.855 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:54:18.855 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:54:18.855 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:54:18.855 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:54:18.855 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:54:19.080 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:54:19.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:19.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:54:19.193 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:54:19.263 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:19.836 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:54:19.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:19.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:54:19.960 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:54:19.995 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:20.020 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:54:20.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:20.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:54:20.125 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:54:20.160 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:20.182 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:54:20.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:20.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:54:20.240 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:54:20.289 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:54:20.327 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:54:20.327 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:54:20.368 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:20.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:20 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:54:20.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:20 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:54:20.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:20 smithi179 ceph-mon[43975]: osdmap e759: 12 total, 12 up, 12 in 2024-02-09T17:54:20.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:20 smithi179 ceph-mon[43975]: pgmap v1886: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 856 op/s; 2967/62622 objects degraded (4.738%); 5293/62622 objects misplaced (8.452%); 6.9 MiB/s, 111 objects/s recovering 2024-02-09T17:54:20.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:20 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:54:20.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:20 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:54:20.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:20 smithi162 ceph-mon[44304]: osdmap e759: 12 total, 12 up, 12 in 2024-02-09T17:54:20.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:20 smithi162 ceph-mon[44304]: pgmap v1886: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 856 op/s; 2967/62622 objects degraded (4.738%); 5293/62622 objects misplaced (8.452%); 6.9 MiB/s, 111 objects/s recovering 2024-02-09T17:54:20.498 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:20 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:54:20.499 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:20 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:54:20.499 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:20 smithi099 ceph-mon[39420]: osdmap e759: 12 total, 12 up, 12 in 2024-02-09T17:54:20.499 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:20 smithi099 ceph-mon[39420]: pgmap v1886: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 856 op/s; 2967/62622 objects degraded (4.738%); 5293/62622 objects misplaced (8.452%); 6.9 MiB/s, 111 objects/s recovering 2024-02-09T17:54:21.387 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:54:21.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:21.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:54:21.401 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:54:21.496 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:54:21.534 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:21.562 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:54:21.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:21.566 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:54:21.667 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:54:21.697 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:21.733 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:54:21.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:21.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:54:21.840 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:54:21.870 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:21.889 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:54:21.896 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:54:21.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:21.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:54:22.002 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:54:22.033 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:22.047 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-09T17:54:22.055 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:54:22.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:22.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:54:22.162 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:54:22.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:22 smithi099 ceph-mon[39420]: pgmap v1887: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 761 op/s; 3003/59400 objects degraded (5.056%); 4861/59400 objects misplaced (8.184%); 6.3 MiB/s, 102 objects/s recovering 2024-02-09T17:54:22.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:22 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 3003/59400 objects degraded (5.056%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-09T17:54:22.307 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:22.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:22 smithi179 ceph-mon[43975]: pgmap v1887: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 761 op/s; 3003/59400 objects degraded (5.056%); 4861/59400 objects misplaced (8.184%); 6.3 MiB/s, 102 objects/s recovering 2024-02-09T17:54:22.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:22 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 3003/59400 objects degraded (5.056%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-09T17:54:22.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:22 smithi162 ceph-mon[44304]: pgmap v1887: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 761 op/s; 3003/59400 objects degraded (5.056%); 4861/59400 objects misplaced (8.184%); 6.3 MiB/s, 102 objects/s recovering 2024-02-09T17:54:22.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:22 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 3003/59400 objects degraded (5.056%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-09T17:54:23.186 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:23 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:54:23.186 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:23 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:54:23.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:23 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:54:23.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:23 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:54:23.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:23 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:54:23.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:23 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:54:23.942 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:54:23.945 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:54:24.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:24 smithi099 ceph-mon[39420]: pgmap v1888: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 800 op/s; 3060/52833 objects degraded (5.792%); 4646/52833 objects misplaced (8.794%); 5.3 MiB/s, 85 objects/s recovering 2024-02-09T17:54:24.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:24 smithi099 ceph-mon[39420]: osdmap e760: 12 total, 12 up, 12 in 2024-02-09T17:54:24.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:24 smithi179 ceph-mon[43975]: pgmap v1888: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 800 op/s; 3060/52833 objects degraded (5.792%); 4646/52833 objects misplaced (8.794%); 5.3 MiB/s, 85 objects/s recovering 2024-02-09T17:54:24.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:24 smithi179 ceph-mon[43975]: osdmap e760: 12 total, 12 up, 12 in 2024-02-09T17:54:24.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:24 smithi162 ceph-mon[44304]: pgmap v1888: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 800 op/s; 3060/52833 objects degraded (5.792%); 4646/52833 objects misplaced (8.794%); 5.3 MiB/s, 85 objects/s recovering 2024-02-09T17:54:24.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:24 smithi162 ceph-mon[44304]: osdmap e760: 12 total, 12 up, 12 in 2024-02-09T17:54:25.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:25 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:54:25.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:25 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:54:25.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:25 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:54:25.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:25 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:54:25.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:25 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:54:25.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:25 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:54:26.275 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is set 2024-02-09T17:54:26.348 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:54:26.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:26.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:54:26.458 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:54:26.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:26 smithi099 ceph-mon[39420]: pgmap v1890: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 542 op/s; 3149/52188 objects degraded (6.034%); 4432/52188 objects misplaced (8.492%); 4.5 MiB/s, 71 objects/s recovering 2024-02-09T17:54:26.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:26 smithi099 ceph-mon[39420]: osdmap e761: 12 total, 12 up, 12 in 2024-02-09T17:54:26.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:26 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3395237718' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:54:26.583 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:26.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:26 smithi179 ceph-mon[43975]: pgmap v1890: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 542 op/s; 3149/52188 objects degraded (6.034%); 4432/52188 objects misplaced (8.492%); 4.5 MiB/s, 71 objects/s recovering 2024-02-09T17:54:26.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:26 smithi179 ceph-mon[43975]: osdmap e761: 12 total, 12 up, 12 in 2024-02-09T17:54:26.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:26 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3395237718' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:54:26.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:26 smithi162 ceph-mon[44304]: pgmap v1890: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 542 op/s; 3149/52188 objects degraded (6.034%); 4432/52188 objects misplaced (8.492%); 4.5 MiB/s, 71 objects/s recovering 2024-02-09T17:54:26.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:26 smithi162 ceph-mon[44304]: osdmap e761: 12 total, 12 up, 12 in 2024-02-09T17:54:26.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:26 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3395237718' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:54:27.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:27 smithi179 ceph-mon[43975]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:54:27.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:27 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3395237718' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:54:27.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:27 smithi179 ceph-mon[43975]: osdmap e762: 12 total, 12 up, 12 in 2024-02-09T17:54:27.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:27 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 3149/52188 objects degraded (6.034%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-09T17:54:27.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:27 smithi162 ceph-mon[44304]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:54:27.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:27 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3395237718' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:54:27.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:27 smithi162 ceph-mon[44304]: osdmap e762: 12 total, 12 up, 12 in 2024-02-09T17:54:27.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:27 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 3149/52188 objects degraded (6.034%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-09T17:54:27.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:27 smithi099 ceph-mon[39420]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:54:27.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:27 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3395237718' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:54:27.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:27 smithi099 ceph-mon[39420]: osdmap e762: 12 total, 12 up, 12 in 2024-02-09T17:54:27.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:27 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 3149/52188 objects degraded (6.034%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-09T17:54:28.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:28 smithi179 ceph-mon[43975]: pgmap v1893: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 924 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 836 op/s; 3283/44319 objects degraded (7.408%); 3742/44319 objects misplaced (8.443%); 6.4 MiB/s, 102 objects/s recovering 2024-02-09T17:54:28.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:28 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3769908800' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-02-09T17:54:28.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:28 smithi162 ceph-mon[44304]: pgmap v1893: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 924 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 836 op/s; 3283/44319 objects degraded (7.408%); 3742/44319 objects misplaced (8.443%); 6.4 MiB/s, 102 objects/s recovering 2024-02-09T17:54:28.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:28 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3769908800' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-02-09T17:54:28.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:28 smithi099 ceph-mon[39420]: pgmap v1893: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 924 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 836 op/s; 3283/44319 objects degraded (7.408%); 3742/44319 objects misplaced (8.443%); 6.4 MiB/s, 102 objects/s recovering 2024-02-09T17:54:28.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:28 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3769908800' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-02-09T17:54:29.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:29 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3769908800' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-02-09T17:54:29.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:29 smithi179 ceph-mon[43975]: osdmap e763: 12 total, 12 up, 12 in 2024-02-09T17:54:29.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:29 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3769908800' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-02-09T17:54:29.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:29 smithi162 ceph-mon[44304]: osdmap e763: 12 total, 12 up, 12 in 2024-02-09T17:54:29.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:29 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3769908800' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-02-09T17:54:29.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:29 smithi099 ceph-mon[39420]: osdmap e763: 12 total, 12 up, 12 in 2024-02-09T17:54:30.301 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:54:30.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:30.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:54:30.409 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:54:30.493 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:30.560 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:30 smithi099 ceph-mon[39420]: pgmap v1895: 47 pgs: 1 peering, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 871 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 617 op/s; 3361/41772 objects degraded (8.046%); 3518/41772 objects misplaced (8.422%); 6.3 MiB/s, 101 objects/s recovering 2024-02-09T17:54:30.560 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:30 smithi099 ceph-mon[39420]: osdmap e764: 12 total, 12 up, 12 in 2024-02-09T17:54:30.597 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.9 to 1 (10000) 2024-02-09T17:54:30.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:30 smithi179 ceph-mon[43975]: pgmap v1895: 47 pgs: 1 peering, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 871 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 617 op/s; 3361/41772 objects degraded (8.046%); 3518/41772 objects misplaced (8.422%); 6.3 MiB/s, 101 objects/s recovering 2024-02-09T17:54:30.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:30 smithi179 ceph-mon[43975]: osdmap e764: 12 total, 12 up, 12 in 2024-02-09T17:54:30.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:30 smithi162 ceph-mon[44304]: pgmap v1895: 47 pgs: 1 peering, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 871 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 617 op/s; 3361/41772 objects degraded (8.046%); 3518/41772 objects misplaced (8.422%); 6.3 MiB/s, 101 objects/s recovering 2024-02-09T17:54:30.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:30 smithi162 ceph-mon[44304]: osdmap e764: 12 total, 12 up, 12 in 2024-02-09T17:54:30.855 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:54:30.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:30.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:54:30.963 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:54:31.010 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:31.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:31 smithi162 ceph-mon[44304]: osdmap e765: 12 total, 12 up, 12 in 2024-02-09T17:54:31.677 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:31 smithi179 ceph-mon[43975]: osdmap e765: 12 total, 12 up, 12 in 2024-02-09T17:54:31.750 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:31 smithi099 ceph-mon[39420]: osdmap e765: 12 total, 12 up, 12 in 2024-02-09T17:54:32.587 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 10 1 2024-02-09T17:54:32.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:32 smithi179 ceph-mon[43975]: pgmap v1898: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 peering, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 799 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 705 op/s; 3400/38355 objects degraded (8.865%); 3078/38355 objects misplaced (8.025%); 6.5 MiB/s, 105 objects/s recovering 2024-02-09T17:54:32.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:32 smithi179 ceph-mon[43975]: osdmap e766: 12 total, 12 up, 12 in 2024-02-09T17:54:32.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:32 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 3400/38355 objects degraded (8.865%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-09T17:54:32.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:32 smithi162 ceph-mon[44304]: pgmap v1898: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 peering, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 799 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 705 op/s; 3400/38355 objects degraded (8.865%); 3078/38355 objects misplaced (8.025%); 6.5 MiB/s, 105 objects/s recovering 2024-02-09T17:54:32.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:32 smithi162 ceph-mon[44304]: osdmap e766: 12 total, 12 up, 12 in 2024-02-09T17:54:32.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:32 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 3400/38355 objects degraded (8.865%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-09T17:54:32.679 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:32 smithi099 ceph-mon[39420]: pgmap v1898: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 peering, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 799 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 705 op/s; 3400/38355 objects degraded (8.865%); 3078/38355 objects misplaced (8.025%); 6.5 MiB/s, 105 objects/s recovering 2024-02-09T17:54:32.679 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:32 smithi099 ceph-mon[39420]: osdmap e766: 12 total, 12 up, 12 in 2024-02-09T17:54:32.679 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:32 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 3400/38355 objects degraded (8.865%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-09T17:54:32.907 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:54:32.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:32.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:54:33.022 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:54:33.113 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:33.377 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:54:33.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:33.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:54:33.481 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:54:33.573 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:33.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:33 smithi179 ceph-mon[43975]: osdmap e767: 12 total, 12 up, 12 in 2024-02-09T17:54:33.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:33 smithi162 ceph-mon[44304]: osdmap e767: 12 total, 12 up, 12 in 2024-02-09T17:54:33.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:33 smithi099 ceph-mon[39420]: osdmap e767: 12 total, 12 up, 12 in 2024-02-09T17:54:33.820 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:54:33.861 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:54:33.862 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:54:33.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:33.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:54:33.926 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:54:33.968 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:54:33.980 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:54:34.004 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:34.016 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:54:34.031 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:54:34.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:34.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:54:34.076 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:54:34.116 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:54:34.141 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:54:34.159 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:54:34.175 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:34.206 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:54:34.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:34.209 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:54:34.212 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:54:34.266 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:54:34.310 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:54:34.318 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:54:34.345 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:34.359 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:54:34.369 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:54:34.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:34.372 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:54:34.391 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:54:34.391 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:54:34.391 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:54:34.392 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:54:34.392 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:54:34.392 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:54:34.392 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:54:34.392 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:54:34.392 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:54:34.392 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:54:34.392 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:54:34.392 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:54:34.410 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:34 smithi162 ceph-mon[44304]: pgmap v1901: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 peering, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 670 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 802 op/s; 3409/32127 objects degraded (10.611%); 2878/32127 objects misplaced (8.958%); 5.9 MiB/s, 96 objects/s recovering 2024-02-09T17:54:34.410 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:34 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:54:34.410 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:34 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:54:34.410 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:34 smithi162 ceph-mon[44304]: osdmap e768: 12 total, 12 up, 12 in 2024-02-09T17:54:34.410 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:34 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1018273293' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:54:34.416 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:54:34.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:34 smithi179 ceph-mon[43975]: pgmap v1901: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 peering, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 670 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 802 op/s; 3409/32127 objects degraded (10.611%); 2878/32127 objects misplaced (8.958%); 5.9 MiB/s, 96 objects/s recovering 2024-02-09T17:54:34.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:34 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:54:34.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:34 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:54:34.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:34 smithi179 ceph-mon[43975]: osdmap e768: 12 total, 12 up, 12 in 2024-02-09T17:54:34.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:34 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1018273293' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:54:34.473 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:54:34.541 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:34.687 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:54:34.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:34.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:54:34.698 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:34 smithi099 ceph-mon[39420]: pgmap v1901: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 peering, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 670 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 802 op/s; 3409/32127 objects degraded (10.611%); 2878/32127 objects misplaced (8.958%); 5.9 MiB/s, 96 objects/s recovering 2024-02-09T17:54:34.698 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:34 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:54:34.698 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:34 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:54:34.698 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:34 smithi099 ceph-mon[39420]: osdmap e768: 12 total, 12 up, 12 in 2024-02-09T17:54:34.699 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:34 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1018273293' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:54:34.798 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:54:34.828 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:34.858 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:54:34.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:34.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:54:34.963 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:54:35.040 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:35.137 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:54:35.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:35.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:54:35.244 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:54:35.274 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:35.303 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:54:35.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:35.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:54:35.396 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset noscrub 2024-02-09T17:54:35.408 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:54:35.439 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:35.471 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:54:35.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:35.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:54:35.578 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:54:35.606 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:35.630 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:54:35.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:35.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:54:35.741 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:54:35.772 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:35.808 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:54:35.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:35.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:54:35.915 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:54:35.961 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:35.988 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:54:35.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:35.992 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:54:36.093 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:54:36.127 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:36.168 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:54:36.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:36.179 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:54:36.280 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:54:36.323 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:36.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:36 smithi179 ceph-mon[43975]: pgmap v1903: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 658 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 608 op/s; 3416/31566 objects degraded (10.822%); 2692/31566 objects misplaced (8.528%); 7.2 MiB/s, 117 objects/s recovering 2024-02-09T17:54:36.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:36 smithi162 ceph-mon[44304]: pgmap v1903: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 658 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 608 op/s; 3416/31566 objects degraded (10.822%); 2692/31566 objects misplaced (8.528%); 7.2 MiB/s, 117 objects/s recovering 2024-02-09T17:54:36.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:36 smithi099 ceph-mon[39420]: pgmap v1903: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 658 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 608 op/s; 3416/31566 objects degraded (10.822%); 2692/31566 objects misplaced (8.528%); 7.2 MiB/s, 117 objects/s recovering 2024-02-09T17:54:37.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:37 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 3416/31566 objects degraded (10.822%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-09T17:54:37.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:37 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 3416/31566 objects degraded (10.822%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-09T17:54:37.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:37 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 3416/31566 objects degraded (10.822%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-09T17:54:38.719 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:38 smithi099 ceph-mon[39420]: pgmap v1904: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 500 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 796 op/s; 3479/23994 objects degraded (14.499%); 2007/23994 objects misplaced (8.365%); 7.6 MiB/s, 122 objects/s recovering 2024-02-09T17:54:38.719 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:38 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:54:38.719 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:38 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:54:38.719 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:38 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:54:38.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:38 smithi179 ceph-mon[43975]: pgmap v1904: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 500 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 796 op/s; 3479/23994 objects degraded (14.499%); 2007/23994 objects misplaced (8.365%); 7.6 MiB/s, 122 objects/s recovering 2024-02-09T17:54:38.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:38 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:54:38.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:38 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:54:38.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:38 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:54:38.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:38 smithi162 ceph-mon[44304]: pgmap v1904: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 500 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 796 op/s; 3479/23994 objects degraded (14.499%); 2007/23994 objects misplaced (8.365%); 7.6 MiB/s, 122 objects/s recovering 2024-02-09T17:54:38.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:38 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:54:38.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:38 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:54:38.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:38 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:54:39.295 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:54:39.295 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [], 2024-02-09T17:54:39.295 INFO:teuthology.orchestra.run.smithi099.stdout: "complaint_time": 30, 2024-02-09T17:54:39.295 INFO:teuthology.orchestra.run.smithi099.stdout: "num_blocked_ops": 0 2024-02-09T17:54:39.295 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:54:39.467 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:39 smithi099 ceph-mon[39420]: osdmap e769: 12 total, 12 up, 12 in 2024-02-09T17:54:39.851 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:54:39.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:39.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:54:39.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:39 smithi179 ceph-mon[43975]: osdmap e769: 12 total, 12 up, 12 in 2024-02-09T17:54:39.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:39 smithi162 ceph-mon[44304]: osdmap e769: 12 total, 12 up, 12 in 2024-02-09T17:54:39.960 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:54:39.993 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:40.019 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:54:40.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:40.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:54:40.124 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:54:40.172 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:40.481 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:40 smithi099 ceph-mon[39420]: pgmap v1906: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 447 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 616 op/s; 3568/21444 objects degraded (16.639%); 1779/21444 objects misplaced (8.296%); 6.3 MiB/s, 101 objects/s recovering 2024-02-09T17:54:40.781 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-09T17:54:40.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:40 smithi179 ceph-mon[43975]: pgmap v1906: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 447 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 616 op/s; 3568/21444 objects degraded (16.639%); 1779/21444 objects misplaced (8.296%); 6.3 MiB/s, 101 objects/s recovering 2024-02-09T17:54:40.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:40 smithi162 ceph-mon[44304]: pgmap v1906: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 447 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 616 op/s; 3568/21444 objects degraded (16.639%); 1779/21444 objects misplaced (8.296%); 6.3 MiB/s, 101 objects/s recovering 2024-02-09T17:54:41.157 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:54:41.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:41.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:54:41.273 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:54:41.364 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:54:41.424 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:41.524 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.10 to 1 (10000) 2024-02-09T17:54:41.708 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:41 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3746976009' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-02-09T17:54:41.708 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:41 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:54:41.708 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:41 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:54:41.926 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:41 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3746976009' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-02-09T17:54:41.926 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:41 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:54:41.926 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:41 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:54:41.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:41 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3746976009' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-02-09T17:54:41.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:41 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:54:41.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:41 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:54:42.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:42 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3746976009' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-02-09T17:54:42.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:42 smithi179 ceph-mon[43975]: osdmap e770: 12 total, 12 up, 12 in 2024-02-09T17:54:42.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:42 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:54:42.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:42 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:54:42.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:42 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:54:42.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:42 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 3607/18141 objects degraded (19.883%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-09T17:54:42.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:42 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:54:42.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:42 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:54:42.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:42 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:54:42.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:42 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:54:42.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:42 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:54:42.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:42 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:54:42.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:42 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:54:42.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:42 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3746976009' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-02-09T17:54:42.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:42 smithi162 ceph-mon[44304]: osdmap e770: 12 total, 12 up, 12 in 2024-02-09T17:54:42.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:42 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:54:42.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:42 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:54:42.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:42 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:54:42.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:42 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 3607/18141 objects degraded (19.883%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-09T17:54:42.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:42 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:54:42.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:42 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:54:42.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:42 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:54:42.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:42 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:54:42.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:42 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:54:42.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:42 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:54:42.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:42 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:54:43.000 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:42 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3746976009' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-02-09T17:54:43.000 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:42 smithi099 ceph-mon[39420]: osdmap e770: 12 total, 12 up, 12 in 2024-02-09T17:54:43.000 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:42 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:54:43.000 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:42 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:54:43.000 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:42 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:54:43.000 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:42 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 3607/18141 objects degraded (19.883%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-09T17:54:43.001 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:42 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:54:43.001 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:42 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:54:43.001 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:42 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:54:43.001 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:42 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:54:43.001 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:42 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:54:43.001 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:42 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:54:43.001 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:42 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:54:43.051 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:54:43.177 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:54:44.322 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:54:44.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:44.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:54:44.437 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:54:44.499 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:44.693 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:44 smithi099 ceph-mon[39420]: pgmap v1909: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 242 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 831 op/s; 3657/11595 objects degraded (31.539%); 1099/11595 objects misplaced (9.478%); 6.8 MiB/s, 109 objects/s recovering 2024-02-09T17:54:44.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:44 smithi179 ceph-mon[43975]: pgmap v1909: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 242 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 831 op/s; 3657/11595 objects degraded (31.539%); 1099/11595 objects misplaced (9.478%); 6.8 MiB/s, 109 objects/s recovering 2024-02-09T17:54:44.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:44 smithi162 ceph-mon[44304]: pgmap v1909: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 242 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 831 op/s; 3657/11595 objects degraded (31.539%); 1099/11595 objects misplaced (9.478%); 6.8 MiB/s, 109 objects/s recovering 2024-02-09T17:54:45.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:45 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:54:45.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:45 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:54:45.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:45 smithi179 ceph-mon[43975]: osdmap e771: 12 total, 12 up, 12 in 2024-02-09T17:54:45.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:45 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:54:45.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:45 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:54:45.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:45 smithi162 ceph-mon[44304]: osdmap e771: 12 total, 12 up, 12 in 2024-02-09T17:54:45.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:45 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:54:45.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:45 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:54:45.998 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:45 smithi099 ceph-mon[39420]: osdmap e771: 12 total, 12 up, 12 in 2024-02-09T17:54:46.720 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Removed 107788 objects 2024-02-09T17:54:46.720 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Clean up completed and total clean up time :170.166 2024-02-09T17:54:46.901 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:46 smithi099 ceph-mon[39420]: pgmap v1911: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 229 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 554 op/s; 3707/10950 objects degraded (33.854%); 892/10950 objects misplaced (8.146%); 6.2 MiB/s, 98 objects/s recovering 2024-02-09T17:54:46.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:46 smithi179 ceph-mon[43975]: pgmap v1911: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 229 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 554 op/s; 3707/10950 objects degraded (33.854%); 892/10950 objects misplaced (8.146%); 6.2 MiB/s, 98 objects/s recovering 2024-02-09T17:54:46.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:46 smithi162 ceph-mon[44304]: pgmap v1911: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 229 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 554 op/s; 3707/10950 objects degraded (33.854%); 892/10950 objects misplaced (8.146%); 6.2 MiB/s, 98 objects/s recovering 2024-02-09T17:54:47.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:47 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 3707/10950 objects degraded (33.854%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-09T17:54:47.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:47 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 3707/10950 objects degraded (33.854%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-09T17:54:47.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:47 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 3707/10950 objects degraded (33.854%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-09T17:54:48.807 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_5 2024-02-09T17:54:48.807 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd pool rm unique_pool_5 unique_pool_5 --yes-i-really-really-mean-it 2024-02-09T17:54:48.887 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:48 smithi099 ceph-mon[39420]: pgmap v1912: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 63 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 767 op/s; 3783/3027 objects degraded (124.975%); 216/3027 objects misplaced (7.136%); 7.1 MiB/s, 113 objects/s recovering 2024-02-09T17:54:48.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:48 smithi179 ceph-mon[43975]: pgmap v1912: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 63 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 767 op/s; 3783/3027 objects degraded (124.975%); 216/3027 objects misplaced (7.136%); 7.1 MiB/s, 113 objects/s recovering 2024-02-09T17:54:48.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:48 smithi162 ceph-mon[44304]: pgmap v1912: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 63 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 767 op/s; 3783/3027 objects degraded (124.975%); 216/3027 objects misplaced (7.136%); 7.1 MiB/s, 113 objects/s recovering 2024-02-09T17:54:49.569 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is unset 2024-02-09T17:54:49.733 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:54:49.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:49.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:54:49.838 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:54:49.865 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:49 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3620981617' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:54:49.869 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:49 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3620981617' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:54:49.872 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:49.895 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:54:49.896 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 11 1 2024-02-09T17:54:49.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:49.898 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:54:49.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:49 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3620981617' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:54:49.999 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:54:50.031 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:50.091 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:54:50.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:50.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:54:50.197 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:54:50.230 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:50.252 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:54:50.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:50.255 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:54:50.356 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:54:50.385 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:50.405 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:54:50.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:50.408 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:54:50.509 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:54:50.537 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:50.556 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:54:50.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:50.559 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:54:50.567 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:50 smithi099 ceph-mon[39420]: pgmap v1913: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 22 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 713 op/s; 3829/1017 objects degraded (376.500%); 81/1017 objects misplaced (7.965%); 6.3 MiB/s, 128 objects/s recovering 2024-02-09T17:54:50.567 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:50 smithi099 ceph-mon[39420]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:54:50.567 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:50 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3620981617' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:54:50.567 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:50 smithi099 ceph-mon[39420]: osdmap e772: 12 total, 12 up, 12 in 2024-02-09T17:54:50.567 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:50 smithi099 ceph-mon[39420]: 1.18 scrub starts 2024-02-09T17:54:50.567 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:50 smithi099 ceph-mon[39420]: 1.18 scrub ok 2024-02-09T17:54:50.659 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:54:50.687 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:50.708 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:54:50.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:50.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:54:50.812 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:54:50.855 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:50 smithi162 ceph-mon[44304]: pgmap v1913: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 22 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 713 op/s; 3829/1017 objects degraded (376.500%); 81/1017 objects misplaced (7.965%); 6.3 MiB/s, 128 objects/s recovering 2024-02-09T17:54:50.855 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:50 smithi162 ceph-mon[44304]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:54:50.855 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:50 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3620981617' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:54:50.855 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:50 smithi162 ceph-mon[44304]: osdmap e772: 12 total, 12 up, 12 in 2024-02-09T17:54:50.856 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:50 smithi162 ceph-mon[44304]: 1.18 scrub starts 2024-02-09T17:54:50.856 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:50 smithi162 ceph-mon[44304]: 1.18 scrub ok 2024-02-09T17:54:50.857 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:50.879 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:54:50.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:50.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:54:50.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:50 smithi179 ceph-mon[43975]: pgmap v1913: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 22 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 713 op/s; 3829/1017 objects degraded (376.500%); 81/1017 objects misplaced (7.965%); 6.3 MiB/s, 128 objects/s recovering 2024-02-09T17:54:50.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:50 smithi179 ceph-mon[43975]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:54:50.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:50 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3620981617' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:54:50.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:50 smithi179 ceph-mon[43975]: osdmap e772: 12 total, 12 up, 12 in 2024-02-09T17:54:50.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:50 smithi179 ceph-mon[43975]: 1.18 scrub starts 2024-02-09T17:54:50.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:50 smithi179 ceph-mon[43975]: 1.18 scrub ok 2024-02-09T17:54:50.984 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:54:51.012 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:51.034 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:54:51.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:51.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:54:51.142 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:54:51.174 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:51.195 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:54:51.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:51.199 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:54:51.257 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:54:51.263 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:54:51.266 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:54:51.300 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:54:51.327 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:51.349 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:54:51.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:51.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:54:51.453 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:54:51.482 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:51.502 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:54:51.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:51.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:54:51.606 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:54:51.640 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:51.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:51 smithi162 ceph-mon[44304]: 7.b scrub starts 2024-02-09T17:54:51.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:51 smithi162 ceph-mon[44304]: 7.b scrub ok 2024-02-09T17:54:51.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:51 smithi179 ceph-mon[43975]: 7.b scrub starts 2024-02-09T17:54:51.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:51 smithi179 ceph-mon[43975]: 7.b scrub ok 2024-02-09T17:54:51.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:51 smithi099 ceph-mon[39420]: 7.b scrub starts 2024-02-09T17:54:51.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:51 smithi099 ceph-mon[39420]: 7.b scrub ok 2024-02-09T17:54:52.647 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-09T17:54:52.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:52 smithi179 ceph-mon[43975]: 1.3 scrub starts 2024-02-09T17:54:52.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:52 smithi179 ceph-mon[43975]: pgmap v1915: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 5.5 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 472 op/s; 3842/243 objects degraded (1581.070%); 7/243 objects misplaced (2.881%); 5.1 MiB/s, 168 objects/s recovering 2024-02-09T17:54:52.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:52 smithi179 ceph-mon[43975]: 1.3 scrub ok 2024-02-09T17:54:52.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:52 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 3842/243 objects degraded (1581.070%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-09T17:54:52.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:52 smithi179 ceph-mon[43975]: osdmap e773: 12 total, 12 up, 12 in 2024-02-09T17:54:52.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:52 smithi162 ceph-mon[44304]: 1.3 scrub starts 2024-02-09T17:54:52.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:52 smithi162 ceph-mon[44304]: pgmap v1915: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 5.5 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 472 op/s; 3842/243 objects degraded (1581.070%); 7/243 objects misplaced (2.881%); 5.1 MiB/s, 168 objects/s recovering 2024-02-09T17:54:52.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:52 smithi162 ceph-mon[44304]: 1.3 scrub ok 2024-02-09T17:54:52.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:52 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 3842/243 objects degraded (1581.070%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-09T17:54:52.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:52 smithi162 ceph-mon[44304]: osdmap e773: 12 total, 12 up, 12 in 2024-02-09T17:54:52.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:52 smithi099 ceph-mon[39420]: 1.3 scrub starts 2024-02-09T17:54:52.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:52 smithi099 ceph-mon[39420]: pgmap v1915: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 5.5 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 472 op/s; 3842/243 objects degraded (1581.070%); 7/243 objects misplaced (2.881%); 5.1 MiB/s, 168 objects/s recovering 2024-02-09T17:54:52.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:52 smithi099 ceph-mon[39420]: 1.3 scrub ok 2024-02-09T17:54:52.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:52 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 3842/243 objects degraded (1581.070%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-09T17:54:52.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:52 smithi099 ceph-mon[39420]: osdmap e773: 12 total, 12 up, 12 in 2024-02-09T17:54:53.595 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:54:53.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:53.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:54:53.699 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:54:53.727 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:53.751 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:54:53.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:53.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:54:53.856 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:54:53.886 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:53 smithi179 ceph-mon[43975]: 7.f scrub starts 2024-02-09T17:54:53.886 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:53 smithi179 ceph-mon[43975]: 7.f scrub ok 2024-02-09T17:54:53.888 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:53.908 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:54:53.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:53.911 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:54:53.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:53 smithi162 ceph-mon[44304]: 7.f scrub starts 2024-02-09T17:54:53.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:53 smithi162 ceph-mon[44304]: 7.f scrub ok 2024-02-09T17:54:53.943 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:53 smithi099 ceph-mon[39420]: 7.f scrub starts 2024-02-09T17:54:53.943 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:53 smithi099 ceph-mon[39420]: 7.f scrub ok 2024-02-09T17:54:54.012 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:54:54.040 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:54.062 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:54:54.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:54.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:54:54.167 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:54:54.195 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:54.216 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:54:54.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:54.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:54:54.321 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:54:54.350 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:54.554 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:54:54.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:54.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:54:54.658 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:54:54.686 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:54.706 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:54:54.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:54.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:54:54.812 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:54:54.838 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:54 smithi099 ceph-mon[39420]: pgmap v1917: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 455 op/s; 3842/6 objects degraded (64033.333%); 4.0 MiB/s, 233 objects/s recovering 2024-02-09T17:54:54.842 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:54.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:54 smithi179 ceph-mon[43975]: pgmap v1917: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 455 op/s; 3842/6 objects degraded (64033.333%); 4.0 MiB/s, 233 objects/s recovering 2024-02-09T17:54:54.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:54 smithi162 ceph-mon[44304]: pgmap v1917: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 455 op/s; 3842/6 objects degraded (64033.333%); 4.0 MiB/s, 233 objects/s recovering 2024-02-09T17:54:54.969 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:54:54.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:54.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:54:55.073 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:54:55.103 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:55.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:55 smithi179 ceph-mon[43975]: 1.13 scrub starts 2024-02-09T17:54:55.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:55 smithi179 ceph-mon[43975]: 1.13 scrub ok 2024-02-09T17:54:55.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:55 smithi162 ceph-mon[44304]: 1.13 scrub starts 2024-02-09T17:54:55.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:55 smithi162 ceph-mon[44304]: 1.13 scrub ok 2024-02-09T17:54:55.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:55 smithi099 ceph-mon[39420]: 1.13 scrub starts 2024-02-09T17:54:55.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:55 smithi099 ceph-mon[39420]: 1.13 scrub ok 2024-02-09T17:54:56.863 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:56 smithi099 ceph-mon[39420]: pgmap v1918: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 125 op/s; 3842/6 objects degraded (64033.333%); 1000 KiB/s, 277 objects/s recovering 2024-02-09T17:54:56.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:56 smithi179 ceph-mon[43975]: pgmap v1918: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 125 op/s; 3842/6 objects degraded (64033.333%); 1000 KiB/s, 277 objects/s recovering 2024-02-09T17:54:56.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:56 smithi162 ceph-mon[44304]: pgmap v1918: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 125 op/s; 3842/6 objects degraded (64033.333%); 1000 KiB/s, 277 objects/s recovering 2024-02-09T17:54:57.655 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:54:57.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:57.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:54:57.765 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:54:57.794 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:57.818 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:54:57.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:54:57.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:54:57.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:57 smithi179 ceph-mon[43975]: 1.16 scrub starts 2024-02-09T17:54:57.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:57 smithi179 ceph-mon[43975]: 1.16 scrub ok 2024-02-09T17:54:57.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:57 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 3842/6 objects degraded (64033.333%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-09T17:54:57.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:57 smithi179 ceph-mon[43975]: osdmap e774: 12 total, 12 up, 12 in 2024-02-09T17:54:57.922 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:54:57.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:57 smithi162 ceph-mon[44304]: 1.16 scrub starts 2024-02-09T17:54:57.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:57 smithi162 ceph-mon[44304]: 1.16 scrub ok 2024-02-09T17:54:57.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:57 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 3842/6 objects degraded (64033.333%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-09T17:54:57.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:57 smithi162 ceph-mon[44304]: osdmap e774: 12 total, 12 up, 12 in 2024-02-09T17:54:57.951 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:57 smithi099 ceph-mon[39420]: 1.16 scrub starts 2024-02-09T17:54:57.951 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:57 smithi099 ceph-mon[39420]: 1.16 scrub ok 2024-02-09T17:54:57.951 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:57 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 3842/6 objects degraded (64033.333%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-09T17:54:57.952 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:57 smithi099 ceph-mon[39420]: osdmap e774: 12 total, 12 up, 12 in 2024-02-09T17:54:57.954 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:54:58.588 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:58 smithi099 ceph-mon[39420]: 1.12 scrub starts 2024-02-09T17:54:58.589 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:58 smithi099 ceph-mon[39420]: 1.12 scrub ok 2024-02-09T17:54:58.589 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:54:58 smithi099 ceph-mon[39420]: pgmap v1920: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 44 op/s; 3842/6 objects degraded (64033.333%); 303 KiB/s, 509 objects/s recovering 2024-02-09T17:54:58.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:58 smithi179 ceph-mon[43975]: 1.12 scrub starts 2024-02-09T17:54:58.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:58 smithi179 ceph-mon[43975]: 1.12 scrub ok 2024-02-09T17:54:58.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:54:58 smithi179 ceph-mon[43975]: pgmap v1920: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 44 op/s; 3842/6 objects degraded (64033.333%); 303 KiB/s, 509 objects/s recovering 2024-02-09T17:54:58.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:58 smithi162 ceph-mon[44304]: 1.12 scrub starts 2024-02-09T17:54:58.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:58 smithi162 ceph-mon[44304]: 1.12 scrub ok 2024-02-09T17:54:58.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:54:58 smithi162 ceph-mon[44304]: pgmap v1920: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 44 op/s; 3842/6 objects degraded (64033.333%); 303 KiB/s, 509 objects/s recovering 2024-02-09T17:55:00.021 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:55:00.021 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 20, 2024-02-09T17:55:00.021 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 600, 2024-02-09T17:55:00.022 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [ 2024-02-09T17:55:00.022 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.022 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25837.0:95789 6.7 6:e710fbc8:::benchmark_data_smithi179_126206_object95788:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e600)", 2024-02-09T17:55:00.022 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:46:46.008692+0000", 2024-02-09T17:55:00.022 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 493.29956680499998, 2024-02-09T17:55:00.022 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.030868195, 2024-02-09T17:55:00.022 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:55:00.022 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:00.022 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:55:00.022 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25837", 2024-02-09T17:55:00.022 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:55:00.022 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 95789 2024-02-09T17:55:00.022 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.023 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:55:00.023 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.023 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:55:00.023 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.008692+0000", 2024-02-09T17:55:00.023 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:55:00.023 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.023 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.023 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:55:00.023 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.008692+0000", 2024-02-09T17:55:00.023 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:55:00.023 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.023 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.024 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:55:00.024 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.008693+0000", 2024-02-09T17:55:00.024 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.7489999999999999e-06 2024-02-09T17:55:00.024 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.024 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.024 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:55:00.024 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.008767+0000", 2024-02-09T17:55:00.024 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.4172999999999996e-05 2024-02-09T17:55:00.024 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.024 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.024 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:55:00.024 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.008781+0000", 2024-02-09T17:55:00.024 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3900000000000001e-05 2024-02-09T17:55:00.025 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.025 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.025 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:55:00.025 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.008792+0000", 2024-02-09T17:55:00.025 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0606e-05 2024-02-09T17:55:00.025 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.025 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.025 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:55:00.025 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.008857+0000", 2024-02-09T17:55:00.025 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.5377999999999996e-05 2024-02-09T17:55:00.025 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.025 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.025 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op must wait for map", 2024-02-09T17:55:00.026 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.008864+0000", 2024-02-09T17:55:00.026 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.6939999999999998e-06 2024-02-09T17:55:00.026 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.026 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.026 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:55:00.026 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.018780+0000", 2024-02-09T17:55:00.026 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0099160010000000007 2024-02-09T17:55:00.026 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.026 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.026 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:55:00.026 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.018789+0000", 2024-02-09T17:55:00.026 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.1370000000000008e-06 2024-02-09T17:55:00.027 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.027 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.027 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:55:00.027 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.023915+0000", 2024-02-09T17:55:00.027 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.005126108 2024-02-09T17:55:00.027 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.027 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.027 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:55:00.027 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.026409+0000", 2024-02-09T17:55:00.027 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.002493725 2024-02-09T17:55:00.027 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.027 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.027 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 4,5", 2024-02-09T17:55:00.027 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.026569+0000", 2024-02-09T17:55:00.028 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000159992 2024-02-09T17:55:00.028 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.028 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.028 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:55:00.028 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.039289+0000", 2024-02-09T17:55:00.028 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.012720189999999999 2024-02-09T17:55:00.028 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.028 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.028 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:00.028 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.039395+0000", 2024-02-09T17:55:00.028 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000106006 2024-02-09T17:55:00.028 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.028 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.029 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:00.029 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.039488+0000", 2024-02-09T17:55:00.029 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.2325999999999997e-05 2024-02-09T17:55:00.029 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.029 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.029 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:55:00.029 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.039524+0000", 2024-02-09T17:55:00.029 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.6658e-05 2024-02-09T17:55:00.029 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.029 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.029 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:55:00.029 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.039560+0000", 2024-02-09T17:55:00.029 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.5552000000000001e-05 2024-02-09T17:55:00.029 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:55:00.030 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:55:00.030 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:55:00.030 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.030 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.030 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25837.0:95797 6.7 6:e0774e6e:::benchmark_data_smithi179_126206_object95796:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e600)", 2024-02-09T17:55:00.030 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:46:46.019695+0000", 2024-02-09T17:55:00.030 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 493.288563786, 2024-02-09T17:55:00.030 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.020019789, 2024-02-09T17:55:00.030 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:55:00.030 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:00.030 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:55:00.030 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25837", 2024-02-09T17:55:00.030 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:55:00.030 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 95797 2024-02-09T17:55:00.030 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.031 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:55:00.031 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.031 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:55:00.031 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.019695+0000", 2024-02-09T17:55:00.031 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:55:00.031 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.031 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.031 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:55:00.031 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.019695+0000", 2024-02-09T17:55:00.031 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:55:00.031 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.031 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.031 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:55:00.031 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.019696+0000", 2024-02-09T17:55:00.032 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4419999999999999e-06 2024-02-09T17:55:00.032 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.032 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.032 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:55:00.032 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.019734+0000", 2024-02-09T17:55:00.032 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.8302e-05 2024-02-09T17:55:00.032 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.032 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.032 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:55:00.032 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.019748+0000", 2024-02-09T17:55:00.032 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3487e-05 2024-02-09T17:55:00.032 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.032 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.032 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:55:00.032 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.019756+0000", 2024-02-09T17:55:00.033 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.2109999999999998e-06 2024-02-09T17:55:00.033 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.033 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.033 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:55:00.033 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.019915+0000", 2024-02-09T17:55:00.033 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000159197 2024-02-09T17:55:00.033 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.033 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.033 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:55:00.033 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.019927+0000", 2024-02-09T17:55:00.033 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1566000000000001e-05 2024-02-09T17:55:00.033 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.033 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.033 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:55:00.033 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.030359+0000", 2024-02-09T17:55:00.034 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0104322539999999 2024-02-09T17:55:00.034 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.034 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.034 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:55:00.034 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.030559+0000", 2024-02-09T17:55:00.034 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00019977100000000001 2024-02-09T17:55:00.034 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.034 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.034 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 4,5", 2024-02-09T17:55:00.034 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.030698+0000", 2024-02-09T17:55:00.034 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013915299999999999 2024-02-09T17:55:00.034 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.034 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.034 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:00.034 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.039445+0000", 2024-02-09T17:55:00.035 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0087474330000000006 2024-02-09T17:55:00.035 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.035 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.035 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:00.035 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.039617+0000", 2024-02-09T17:55:00.035 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00017129499999999999 2024-02-09T17:55:00.035 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.035 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.035 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:55:00.035 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.039630+0000", 2024-02-09T17:55:00.035 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3573e-05 2024-02-09T17:55:00.035 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.035 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.035 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:55:00.036 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.039679+0000", 2024-02-09T17:55:00.036 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.8398e-05 2024-02-09T17:55:00.036 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.036 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.036 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:55:00.036 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.039714+0000", 2024-02-09T17:55:00.036 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.5707000000000002e-05 2024-02-09T17:55:00.036 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:55:00.036 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:55:00.036 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:55:00.036 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.036 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.036 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25837.0:95801 6.7 6:e618dc55:::benchmark_data_smithi179_126206_object95800:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e600)", 2024-02-09T17:55:00.036 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:46:46.027443+0000", 2024-02-09T17:55:00.037 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 493.28081560099997, 2024-02-09T17:55:00.037 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0151571669999999, 2024-02-09T17:55:00.037 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:55:00.037 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:00.037 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:55:00.037 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25837", 2024-02-09T17:55:00.037 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:55:00.037 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 95801 2024-02-09T17:55:00.037 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.037 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:55:00.037 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.037 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:55:00.037 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.027443+0000", 2024-02-09T17:55:00.037 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:55:00.037 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.038 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.038 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:55:00.038 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.027443+0000", 2024-02-09T17:55:00.038 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:55:00.038 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.038 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.038 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:55:00.038 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.027444+0000", 2024-02-09T17:55:00.038 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.53e-06 2024-02-09T17:55:00.038 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.038 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.038 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:55:00.038 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.027497+0000", 2024-02-09T17:55:00.038 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3223999999999997e-05 2024-02-09T17:55:00.038 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.039 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.039 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:55:00.039 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.027510+0000", 2024-02-09T17:55:00.039 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2767e-05 2024-02-09T17:55:00.039 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.039 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.039 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:55:00.039 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.027519+0000", 2024-02-09T17:55:00.039 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.2280000000000004e-06 2024-02-09T17:55:00.039 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.039 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.039 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:55:00.039 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.027586+0000", 2024-02-09T17:55:00.039 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.6222000000000004e-05 2024-02-09T17:55:00.040 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.040 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.040 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T17:55:00.040 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:46.027597+0000", 2024-02-09T17:55:00.040 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1466e-05 2024-02-09T17:55:00.040 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.040 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.040 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:55:00.040 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.034153+0000", 2024-02-09T17:55:00.040 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.00655592 2024-02-09T17:55:00.040 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.040 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.040 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:55:00.040 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.034261+0000", 2024-02-09T17:55:00.041 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000108043 2024-02-09T17:55:00.041 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.041 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.041 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 4,5", 2024-02-09T17:55:00.041 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.034359+0000", 2024-02-09T17:55:00.041 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.8159000000000007e-05 2024-02-09T17:55:00.041 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.041 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.041 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:55:00.041 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.039724+0000", 2024-02-09T17:55:00.041 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0053644640000000002 2024-02-09T17:55:00.041 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.041 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.041 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:00.041 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.042479+0000", 2024-02-09T17:55:00.042 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0027548569999999999 2024-02-09T17:55:00.042 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.042 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.042 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:00.042 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.042526+0000", 2024-02-09T17:55:00.042 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.7376999999999997e-05 2024-02-09T17:55:00.042 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.042 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.042 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:55:00.042 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.042562+0000", 2024-02-09T17:55:00.042 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.5589000000000001e-05 2024-02-09T17:55:00.042 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.042 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.042 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:55:00.042 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:46:47.042600+0000", 2024-02-09T17:55:00.043 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.8321000000000001e-05 2024-02-09T17:55:00.043 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:55:00.043 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:55:00.043 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:55:00.043 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.043 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.043 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25837.0:250068 6.b 6:d0990057:::benchmark_data_smithi179_126206_object119687:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:55:00.043 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:49:47.478512+0000", 2024-02-09T17:55:00.043 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 311.82974603500003, 2024-02-09T17:55:00.043 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.21750245900000001, 2024-02-09T17:55:00.043 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:55:00.043 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:00.043 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:55:00.043 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25837", 2024-02-09T17:55:00.044 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:55:00.044 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 250068 2024-02-09T17:55:00.044 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.044 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:55:00.044 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.044 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:55:00.044 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.478512+0000", 2024-02-09T17:55:00.044 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:55:00.044 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.044 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.044 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:55:00.044 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.478512+0000", 2024-02-09T17:55:00.044 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:55:00.044 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.045 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.045 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:55:00.045 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.478514+0000", 2024-02-09T17:55:00.045 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.663e-06 2024-02-09T17:55:00.045 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.045 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.045 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:55:00.045 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.478522+0000", 2024-02-09T17:55:00.045 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.7700000000000001e-06 2024-02-09T17:55:00.045 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.045 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.045 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:55:00.045 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.478536+0000", 2024-02-09T17:55:00.045 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4489e-05 2024-02-09T17:55:00.045 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.046 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.046 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:55:00.046 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.478547+0000", 2024-02-09T17:55:00.046 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0882e-05 2024-02-09T17:55:00.046 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.046 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.046 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:55:00.046 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.479985+0000", 2024-02-09T17:55:00.046 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0014383379999999999 2024-02-09T17:55:00.046 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.046 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.046 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:55:00.046 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.480092+0000", 2024-02-09T17:55:00.046 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00010629 2024-02-09T17:55:00.046 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.047 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.047 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,7", 2024-02-09T17:55:00.047 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.480199+0000", 2024-02-09T17:55:00.047 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000107755 2024-02-09T17:55:00.047 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.047 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.047 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:55:00.047 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.490461+0000", 2024-02-09T17:55:00.047 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.01026174 2024-02-09T17:55:00.047 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.047 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.047 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:00.047 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.499549+0000", 2024-02-09T17:55:00.047 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0090881069999999998 2024-02-09T17:55:00.048 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.048 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.048 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:00.048 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.695912+0000", 2024-02-09T17:55:00.048 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.196362389 2024-02-09T17:55:00.048 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.048 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.048 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:55:00.048 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.695961+0000", 2024-02-09T17:55:00.048 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.9200999999999999e-05 2024-02-09T17:55:00.049 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.049 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.049 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:55:00.049 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.696015+0000", 2024-02-09T17:55:00.049 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3835000000000002e-05 2024-02-09T17:55:00.049 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:55:00.049 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:55:00.049 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:55:00.049 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.049 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.049 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25837.0:250084 6.b 6:d1697981:::benchmark_data_smithi179_126206_object119703:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:55:00.049 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:49:47.494317+0000", 2024-02-09T17:55:00.049 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 311.81394160100001, 2024-02-09T17:55:00.049 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.214235552, 2024-02-09T17:55:00.050 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:55:00.050 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:00.050 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:55:00.050 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25837", 2024-02-09T17:55:00.050 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:55:00.050 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 250084 2024-02-09T17:55:00.050 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.050 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:55:00.050 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.050 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:55:00.050 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.494317+0000", 2024-02-09T17:55:00.050 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:55:00.050 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.050 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.051 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:55:00.051 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.494317+0000", 2024-02-09T17:55:00.051 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:55:00.051 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.051 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.051 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:55:00.051 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.494318+0000", 2024-02-09T17:55:00.051 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.02e-07 2024-02-09T17:55:00.051 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.051 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.051 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:55:00.051 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.494321+0000", 2024-02-09T17:55:00.051 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.4369999999999999e-06 2024-02-09T17:55:00.051 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.052 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.052 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:55:00.052 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.494329+0000", 2024-02-09T17:55:00.052 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.7489999999999998e-06 2024-02-09T17:55:00.052 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.052 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.052 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:55:00.052 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.494334+0000", 2024-02-09T17:55:00.052 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.7450000000000003e-06 2024-02-09T17:55:00.052 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.052 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.053 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:55:00.053 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.494371+0000", 2024-02-09T17:55:00.053 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.7307e-05 2024-02-09T17:55:00.053 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.053 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.053 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:55:00.053 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.494473+0000", 2024-02-09T17:55:00.053 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000101636 2024-02-09T17:55:00.053 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.053 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.053 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,7", 2024-02-09T17:55:00.053 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.494558+0000", 2024-02-09T17:55:00.053 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.5359999999999996e-05 2024-02-09T17:55:00.054 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.054 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.054 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:55:00.054 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.499609+0000", 2024-02-09T17:55:00.054 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00505085 2024-02-09T17:55:00.054 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.054 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.054 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:00.054 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.511769+0000", 2024-02-09T17:55:00.054 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.012160064999999999 2024-02-09T17:55:00.054 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.054 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.054 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:00.054 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.708448+0000", 2024-02-09T17:55:00.054 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.19667905599999999 2024-02-09T17:55:00.055 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.055 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.055 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:55:00.055 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.708498+0000", 2024-02-09T17:55:00.055 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.0552e-05 2024-02-09T17:55:00.055 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.055 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.055 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:55:00.055 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.708552+0000", 2024-02-09T17:55:00.055 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3893000000000001e-05 2024-02-09T17:55:00.055 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:55:00.055 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:55:00.055 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:55:00.055 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.056 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.056 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.16440.0:49421 7.3 7:c73e2a1a:::benchmark_data_smithi179_135804_object49420:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e672)", 2024-02-09T17:55:00.056 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:50:47.244406+0000", 2024-02-09T17:55:00.056 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 252.06385259999999, 2024-02-09T17:55:00.056 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.21023229900000001, 2024-02-09T17:55:00.056 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:55:00.056 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:00.056 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:55:00.056 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.16440", 2024-02-09T17:55:00.056 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:55:00.056 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 49421 2024-02-09T17:55:00.056 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.056 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:55:00.057 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.057 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:55:00.057 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.244406+0000", 2024-02-09T17:55:00.057 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:55:00.057 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.057 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.057 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:55:00.057 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.244406+0000", 2024-02-09T17:55:00.057 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:55:00.057 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.057 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.057 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:55:00.057 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.244407+0000", 2024-02-09T17:55:00.057 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5209999999999999e-06 2024-02-09T17:55:00.057 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.058 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.058 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:55:00.058 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.245868+0000", 2024-02-09T17:55:00.058 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0014611400000000001 2024-02-09T17:55:00.058 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.058 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.058 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:55:00.058 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.245895+0000", 2024-02-09T17:55:00.058 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.6381e-05 2024-02-09T17:55:00.058 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.058 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.058 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:55:00.058 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.245906+0000", 2024-02-09T17:55:00.058 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1496e-05 2024-02-09T17:55:00.058 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.059 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.059 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:55:00.059 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.246250+0000", 2024-02-09T17:55:00.059 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00034349000000000001 2024-02-09T17:55:00.059 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.059 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.059 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:55:00.059 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.246423+0000", 2024-02-09T17:55:00.059 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00017337 2024-02-09T17:55:00.059 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.059 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.059 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 2,4", 2024-02-09T17:55:00.059 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.246584+0000", 2024-02-09T17:55:00.059 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000160383 2024-02-09T17:55:00.060 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.060 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.060 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:55:00.060 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.248463+0000", 2024-02-09T17:55:00.060 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.001879672 2024-02-09T17:55:00.060 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.060 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.060 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:00.060 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.250728+0000", 2024-02-09T17:55:00.060 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0022651609999999999 2024-02-09T17:55:00.060 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.060 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.060 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:00.060 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.454565+0000", 2024-02-09T17:55:00.061 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20383691600000001 2024-02-09T17:55:00.061 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.061 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.061 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:55:00.061 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.454607+0000", 2024-02-09T17:55:00.061 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2074999999999999e-05 2024-02-09T17:55:00.061 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.061 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.061 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:55:00.061 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.454638+0000", 2024-02-09T17:55:00.061 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.0694000000000001e-05 2024-02-09T17:55:00.061 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:55:00.061 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:55:00.061 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:55:00.061 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.062 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.062 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.16440.0:49423 7.3 7:cb66cc0b:::benchmark_data_smithi179_135804_object49422:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e672)", 2024-02-09T17:55:00.062 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:50:47.248490+0000", 2024-02-09T17:55:00.062 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 252.05976872299999, 2024-02-09T17:55:00.062 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.21032324299999999, 2024-02-09T17:55:00.062 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:55:00.062 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:00.062 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:55:00.062 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.16440", 2024-02-09T17:55:00.062 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:55:00.062 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 49423 2024-02-09T17:55:00.062 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.062 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:55:00.062 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.063 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:55:00.063 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.248490+0000", 2024-02-09T17:55:00.063 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:55:00.063 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.063 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.063 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:55:00.063 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.248490+0000", 2024-02-09T17:55:00.063 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:55:00.063 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.063 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.063 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:55:00.063 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.248491+0000", 2024-02-09T17:55:00.063 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2669999999999999e-06 2024-02-09T17:55:00.063 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.063 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.064 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:55:00.064 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.248529+0000", 2024-02-09T17:55:00.064 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.7877999999999999e-05 2024-02-09T17:55:00.064 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.064 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.064 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:55:00.064 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.248545+0000", 2024-02-09T17:55:00.064 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.6378000000000001e-05 2024-02-09T17:55:00.064 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.064 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.064 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:55:00.064 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.248555+0000", 2024-02-09T17:55:00.064 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.7550000000000006e-06 2024-02-09T17:55:00.064 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.065 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.065 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:55:00.065 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.248610+0000", 2024-02-09T17:55:00.065 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.4716000000000003e-05 2024-02-09T17:55:00.065 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.065 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.065 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:55:00.065 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.248755+0000", 2024-02-09T17:55:00.065 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014516299999999999 2024-02-09T17:55:00.065 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.065 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.065 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 2,4", 2024-02-09T17:55:00.065 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.248901+0000", 2024-02-09T17:55:00.065 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000146241 2024-02-09T17:55:00.066 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.066 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.066 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:55:00.066 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.251344+0000", 2024-02-09T17:55:00.066 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0024433660000000002 2024-02-09T17:55:00.066 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.066 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.066 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:00.066 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.255513+0000", 2024-02-09T17:55:00.066 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0041685059999999998 2024-02-09T17:55:00.066 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.066 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.066 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:00.066 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.458641+0000", 2024-02-09T17:55:00.066 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20312849399999999 2024-02-09T17:55:00.067 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.067 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.067 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:55:00.067 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.458742+0000", 2024-02-09T17:55:00.067 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000100407 2024-02-09T17:55:00.067 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.067 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.067 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:55:00.067 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.458813+0000", 2024-02-09T17:55:00.067 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.1072e-05 2024-02-09T17:55:00.067 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:55:00.067 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:55:00.067 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:55:00.067 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.068 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.068 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.16440.0:49436 7.3 7:c0da5b98:::benchmark_data_smithi179_135804_object49435:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e672)", 2024-02-09T17:55:00.068 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:50:47.253224+0000", 2024-02-09T17:55:00.068 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 252.05503463900001, 2024-02-09T17:55:00.068 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20575738800000001, 2024-02-09T17:55:00.068 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:55:00.068 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:00.068 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:55:00.068 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.16440", 2024-02-09T17:55:00.068 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:55:00.068 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 49436 2024-02-09T17:55:00.068 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.068 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:55:00.069 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.069 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:55:00.069 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.253224+0000", 2024-02-09T17:55:00.069 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:55:00.069 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.069 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.069 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:55:00.069 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.253224+0000", 2024-02-09T17:55:00.069 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:55:00.069 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.069 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.069 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:55:00.069 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.253225+0000", 2024-02-09T17:55:00.069 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3200000000000001e-06 2024-02-09T17:55:00.070 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.070 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.070 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:55:00.070 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.253252+0000", 2024-02-09T17:55:00.070 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.7212999999999998e-05 2024-02-09T17:55:00.070 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.070 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.070 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:55:00.070 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.253263+0000", 2024-02-09T17:55:00.070 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1046000000000001e-05 2024-02-09T17:55:00.070 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.070 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.070 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:55:00.070 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.253271+0000", 2024-02-09T17:55:00.071 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.7619999999999995e-06 2024-02-09T17:55:00.071 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.071 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.071 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:55:00.071 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.253342+0000", 2024-02-09T17:55:00.071 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.0600999999999997e-05 2024-02-09T17:55:00.071 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.071 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.071 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:55:00.071 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.253480+0000", 2024-02-09T17:55:00.071 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013812500000000001 2024-02-09T17:55:00.071 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.071 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.071 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 2,4", 2024-02-09T17:55:00.072 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.253643+0000", 2024-02-09T17:55:00.072 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00016274400000000001 2024-02-09T17:55:00.072 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.072 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.072 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:55:00.072 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.255583+0000", 2024-02-09T17:55:00.072 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0019404520000000001 2024-02-09T17:55:00.072 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.072 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.072 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:00.072 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.263470+0000", 2024-02-09T17:55:00.072 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0078871510000000002 2024-02-09T17:55:00.072 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.073 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.073 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:00.073 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.458892+0000", 2024-02-09T17:55:00.073 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.19542220099999999 2024-02-09T17:55:00.073 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.073 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.073 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:55:00.073 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.458946+0000", 2024-02-09T17:55:00.073 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3776000000000001e-05 2024-02-09T17:55:00.073 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.073 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.073 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:55:00.073 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.458981+0000", 2024-02-09T17:55:00.073 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.4996999999999997e-05 2024-02-09T17:55:00.074 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:55:00.074 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:55:00.074 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:55:00.074 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.074 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.074 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.16440.0:49484 7.3 7:cc61907a:::benchmark_data_smithi179_135804_object49483:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e672)", 2024-02-09T17:55:00.074 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:50:47.279415+0000", 2024-02-09T17:55:00.074 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 252.02884323800001, 2024-02-09T17:55:00.074 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.17966059500000001, 2024-02-09T17:55:00.074 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:55:00.074 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:00.074 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:55:00.074 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.16440", 2024-02-09T17:55:00.074 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:55:00.075 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 49484 2024-02-09T17:55:00.075 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.075 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:55:00.075 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.075 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:55:00.075 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.279415+0000", 2024-02-09T17:55:00.075 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:55:00.075 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.075 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.075 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:55:00.075 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.279415+0000", 2024-02-09T17:55:00.075 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:55:00.075 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.075 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.076 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:55:00.076 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.279417+0000", 2024-02-09T17:55:00.076 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5650000000000001e-06 2024-02-09T17:55:00.076 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.076 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.076 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:55:00.076 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.279452+0000", 2024-02-09T17:55:00.076 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.5553999999999999e-05 2024-02-09T17:55:00.076 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.076 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.076 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:55:00.076 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.279466+0000", 2024-02-09T17:55:00.076 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.345e-05 2024-02-09T17:55:00.077 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.077 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.077 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:55:00.077 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.279474+0000", 2024-02-09T17:55:00.077 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.3340000000000002e-06 2024-02-09T17:55:00.077 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.077 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.077 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:55:00.077 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.279530+0000", 2024-02-09T17:55:00.077 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.5855e-05 2024-02-09T17:55:00.077 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.077 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.077 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:55:00.077 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.279643+0000", 2024-02-09T17:55:00.077 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000113581 2024-02-09T17:55:00.078 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.078 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.078 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 2,4", 2024-02-09T17:55:00.078 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.279740+0000", 2024-02-09T17:55:00.078 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.7003000000000006e-05 2024-02-09T17:55:00.078 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.078 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.078 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:55:00.078 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.281711+0000", 2024-02-09T17:55:00.078 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0019709810000000001 2024-02-09T17:55:00.078 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.078 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.078 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:00.078 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.281849+0000", 2024-02-09T17:55:00.078 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000137933 2024-02-09T17:55:00.079 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.079 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.079 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:00.079 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.459017+0000", 2024-02-09T17:55:00.079 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.177167457 2024-02-09T17:55:00.079 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.079 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.079 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:55:00.079 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.459050+0000", 2024-02-09T17:55:00.079 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.3293999999999997e-05 2024-02-09T17:55:00.079 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.079 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.079 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:55:00.079 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.459076+0000", 2024-02-09T17:55:00.079 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.5588e-05 2024-02-09T17:55:00.080 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:55:00.080 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:55:00.080 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:55:00.080 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.080 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.080 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.16440.0:49514 7.3 7:c295d3fb:::benchmark_data_smithi179_135804_object49513:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e672)", 2024-02-09T17:55:00.080 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:50:47.302589+0000", 2024-02-09T17:55:00.080 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 252.00566930100001, 2024-02-09T17:55:00.080 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.158984036, 2024-02-09T17:55:00.080 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:55:00.080 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:00.080 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:55:00.080 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.16440", 2024-02-09T17:55:00.080 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:55:00.081 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 49514 2024-02-09T17:55:00.081 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.081 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:55:00.081 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.081 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:55:00.081 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.302589+0000", 2024-02-09T17:55:00.081 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:55:00.081 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.081 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.081 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:55:00.081 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.302589+0000", 2024-02-09T17:55:00.081 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:55:00.081 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.081 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.082 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:55:00.082 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.302590+0000", 2024-02-09T17:55:00.082 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.139e-06 2024-02-09T17:55:00.082 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.082 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.082 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:55:00.082 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.302650+0000", 2024-02-09T17:55:00.082 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.9970999999999999e-05 2024-02-09T17:55:00.082 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.082 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.082 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:55:00.082 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.302662+0000", 2024-02-09T17:55:00.082 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2126999999999999e-05 2024-02-09T17:55:00.082 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.083 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.083 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:55:00.083 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.302670+0000", 2024-02-09T17:55:00.083 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.1089999999999998e-06 2024-02-09T17:55:00.083 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.083 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.083 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:55:00.083 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.302720+0000", 2024-02-09T17:55:00.083 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.9830000000000002e-05 2024-02-09T17:55:00.083 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.083 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.083 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:55:00.083 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.302837+0000", 2024-02-09T17:55:00.083 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000116525 2024-02-09T17:55:00.084 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.084 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.084 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 2,4", 2024-02-09T17:55:00.084 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.302943+0000", 2024-02-09T17:55:00.084 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0001058 2024-02-09T17:55:00.084 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.084 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.084 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:55:00.084 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.304480+0000", 2024-02-09T17:55:00.084 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0015376980000000001 2024-02-09T17:55:00.084 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.084 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.084 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:00.085 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.304992+0000", 2024-02-09T17:55:00.085 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00051207599999999998 2024-02-09T17:55:00.085 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.085 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.085 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:00.085 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.461470+0000", 2024-02-09T17:55:00.085 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.156477323 2024-02-09T17:55:00.085 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.085 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.085 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:55:00.085 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.461529+0000", 2024-02-09T17:55:00.085 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.9083999999999999e-05 2024-02-09T17:55:00.085 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.085 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.086 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:55:00.086 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:47.461573+0000", 2024-02-09T17:55:00.086 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.4354000000000003e-05 2024-02-09T17:55:00.086 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:55:00.086 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:55:00.086 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:55:00.086 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.086 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.086 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.16440.0:116779 7.3 7:cf34c6cd:::benchmark_data_smithi179_135804_object8988:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e706)", 2024-02-09T17:55:00.086 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:52:10.321326+0000", 2024-02-09T17:55:00.086 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 168.98693270300001, 2024-02-09T17:55:00.086 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.16409868499999999, 2024-02-09T17:55:00.086 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:55:00.086 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:00.086 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:55:00.087 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.16440", 2024-02-09T17:55:00.087 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:55:00.087 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 116779 2024-02-09T17:55:00.087 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.087 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:55:00.087 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.087 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:55:00.087 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.321326+0000", 2024-02-09T17:55:00.087 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:55:00.087 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.087 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.087 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:55:00.087 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.321326+0000", 2024-02-09T17:55:00.087 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:55:00.088 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.088 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.088 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:55:00.088 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.321327+0000", 2024-02-09T17:55:00.088 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4950000000000001e-06 2024-02-09T17:55:00.088 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.088 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.088 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:55:00.088 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.321333+0000", 2024-02-09T17:55:00.088 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7169999999999996e-06 2024-02-09T17:55:00.088 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.088 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.088 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:55:00.088 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.321347+0000", 2024-02-09T17:55:00.088 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3664e-05 2024-02-09T17:55:00.089 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.089 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.089 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:55:00.089 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.321353+0000", 2024-02-09T17:55:00.089 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.7209999999999997e-06 2024-02-09T17:55:00.089 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.089 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.089 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:55:00.089 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.321422+0000", 2024-02-09T17:55:00.089 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.8411999999999995e-05 2024-02-09T17:55:00.089 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.089 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.089 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:55:00.089 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.321554+0000", 2024-02-09T17:55:00.090 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000132624 2024-02-09T17:55:00.090 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.090 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.090 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 2,4", 2024-02-09T17:55:00.090 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.321667+0000", 2024-02-09T17:55:00.090 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00011291899999999999 2024-02-09T17:55:00.090 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.090 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.090 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:00.090 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.329634+0000", 2024-02-09T17:55:00.090 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0079667620000000005 2024-02-09T17:55:00.090 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.090 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.090 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:55:00.091 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.329693+0000", 2024-02-09T17:55:00.091 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.9244999999999999e-05 2024-02-09T17:55:00.091 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.091 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.091 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:00.091 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.485311+0000", 2024-02-09T17:55:00.091 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.155617854 2024-02-09T17:55:00.091 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.091 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.091 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:55:00.091 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.485373+0000", 2024-02-09T17:55:00.091 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.1891999999999996e-05 2024-02-09T17:55:00.091 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.091 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.091 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:55:00.092 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.485424+0000", 2024-02-09T17:55:00.092 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1379999999999999e-05 2024-02-09T17:55:00.092 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:55:00.092 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:55:00.092 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:55:00.092 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.092 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.092 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.16440.0:116781 7.3 7:c11f805d:::benchmark_data_smithi179_135804_object8990:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e706)", 2024-02-09T17:55:00.092 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:52:10.321368+0000", 2024-02-09T17:55:00.092 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 168.98689040299999, 2024-02-09T17:55:00.092 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.16419692699999999, 2024-02-09T17:55:00.092 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:55:00.092 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:00.092 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:55:00.093 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.16440", 2024-02-09T17:55:00.093 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:55:00.093 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 116781 2024-02-09T17:55:00.093 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.093 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:55:00.093 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.093 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:55:00.093 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.321368+0000", 2024-02-09T17:55:00.093 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:55:00.093 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.093 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.093 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:55:00.093 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.321368+0000", 2024-02-09T17:55:00.093 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:55:00.094 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.094 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.094 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:55:00.094 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.321368+0000", 2024-02-09T17:55:00.094 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.1100000000000001e-07 2024-02-09T17:55:00.094 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.094 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.094 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:55:00.094 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.321370+0000", 2024-02-09T17:55:00.094 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0470000000000001e-06 2024-02-09T17:55:00.094 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.094 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.094 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:55:00.094 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.321375+0000", 2024-02-09T17:55:00.094 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.8579999999999999e-06 2024-02-09T17:55:00.095 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.095 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.095 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:55:00.095 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.321379+0000", 2024-02-09T17:55:00.095 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.4599999999999999e-06 2024-02-09T17:55:00.095 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.095 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.095 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:55:00.095 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.325607+0000", 2024-02-09T17:55:00.095 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0042286390000000002 2024-02-09T17:55:00.095 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.095 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.095 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:55:00.095 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.325720+0000", 2024-02-09T17:55:00.095 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000113127 2024-02-09T17:55:00.096 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.096 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.096 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 2,4", 2024-02-09T17:55:00.096 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.325798+0000", 2024-02-09T17:55:00.096 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.7226999999999998e-05 2024-02-09T17:55:00.096 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.096 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.096 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:00.096 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.329679+0000", 2024-02-09T17:55:00.096 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0038811639999999999 2024-02-09T17:55:00.096 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.096 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.096 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:55:00.096 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.329730+0000", 2024-02-09T17:55:00.097 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1507e-05 2024-02-09T17:55:00.097 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.097 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.097 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:00.097 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.485481+0000", 2024-02-09T17:55:00.097 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.15575064799999999 2024-02-09T17:55:00.097 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.097 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.097 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:55:00.097 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.485530+0000", 2024-02-09T17:55:00.097 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.8894000000000003e-05 2024-02-09T17:55:00.097 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.097 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.097 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:55:00.098 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.485565+0000", 2024-02-09T17:55:00.098 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.4944999999999998e-05 2024-02-09T17:55:00.098 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:55:00.098 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:55:00.098 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:55:00.098 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.098 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.098 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.16440.0:116787 7.2 7:4d7c487d:::benchmark_data_smithi179_135804_object8996:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e706)", 2024-02-09T17:55:00.098 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:52:10.326017+0000", 2024-02-09T17:55:00.098 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 168.982240985, 2024-02-09T17:55:00.098 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.170713584, 2024-02-09T17:55:00.098 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:55:00.098 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:00.098 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:55:00.098 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.16440", 2024-02-09T17:55:00.099 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:55:00.099 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 116787 2024-02-09T17:55:00.099 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.099 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:55:00.099 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.099 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:55:00.099 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.326017+0000", 2024-02-09T17:55:00.099 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:55:00.099 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.099 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.099 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:55:00.099 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.326017+0000", 2024-02-09T17:55:00.099 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:55:00.099 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.099 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.099 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:55:00.100 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.326019+0000", 2024-02-09T17:55:00.100 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1909999999999999e-06 2024-02-09T17:55:00.100 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.100 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.100 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:55:00.100 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.326023+0000", 2024-02-09T17:55:00.100 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9709999999999998e-06 2024-02-09T17:55:00.100 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.100 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.100 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:55:00.100 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.326033+0000", 2024-02-09T17:55:00.100 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0925e-05 2024-02-09T17:55:00.100 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.101 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.101 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:55:00.101 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.326040+0000", 2024-02-09T17:55:00.101 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.8329999999999996e-06 2024-02-09T17:55:00.101 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.101 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.101 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:55:00.101 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.326099+0000", 2024-02-09T17:55:00.101 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.893e-05 2024-02-09T17:55:00.101 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.101 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.101 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:55:00.101 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.326213+0000", 2024-02-09T17:55:00.101 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000114017 2024-02-09T17:55:00.101 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.102 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.102 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 2,10", 2024-02-09T17:55:00.102 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.326311+0000", 2024-02-09T17:55:00.102 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.7757999999999995e-05 2024-02-09T17:55:00.102 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.102 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.102 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:55:00.102 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.329994+0000", 2024-02-09T17:55:00.102 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0036834979999999999 2024-02-09T17:55:00.102 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.102 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.102 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:00.102 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.342503+0000", 2024-02-09T17:55:00.102 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.012508797 2024-02-09T17:55:00.103 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.103 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.103 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:00.103 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.496612+0000", 2024-02-09T17:55:00.103 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.15410845300000001 2024-02-09T17:55:00.103 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.103 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.103 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:55:00.103 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.496671+0000", 2024-02-09T17:55:00.103 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.8881000000000001e-05 2024-02-09T17:55:00.103 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.103 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.103 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:55:00.103 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.496731+0000", 2024-02-09T17:55:00.104 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.033e-05 2024-02-09T17:55:00.104 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:55:00.104 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:55:00.104 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:55:00.104 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.104 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.104 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.16440.0:116794 7.2 7:4704a320:::benchmark_data_smithi179_135804_object9003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e706)", 2024-02-09T17:55:00.104 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:52:10.342122+0000", 2024-02-09T17:55:00.104 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 168.966136632, 2024-02-09T17:55:00.104 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.157635846, 2024-02-09T17:55:00.104 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:55:00.104 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:00.104 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:55:00.104 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.16440", 2024-02-09T17:55:00.104 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:55:00.105 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 116794 2024-02-09T17:55:00.105 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.105 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:55:00.105 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.105 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:55:00.105 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.342122+0000", 2024-02-09T17:55:00.105 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:55:00.105 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.105 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.105 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:55:00.105 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.342122+0000", 2024-02-09T17:55:00.105 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:55:00.105 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.105 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.105 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:55:00.106 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.342124+0000", 2024-02-09T17:55:00.106 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.9539999999999998e-06 2024-02-09T17:55:00.106 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.106 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.106 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:55:00.106 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.342131+0000", 2024-02-09T17:55:00.106 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.6009999999999997e-06 2024-02-09T17:55:00.106 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.106 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.106 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:55:00.106 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.342146+0000", 2024-02-09T17:55:00.106 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4819e-05 2024-02-09T17:55:00.106 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.106 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.107 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:55:00.107 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.342156+0000", 2024-02-09T17:55:00.107 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0275000000000001e-05 2024-02-09T17:55:00.107 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.107 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.107 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:55:00.107 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.342546+0000", 2024-02-09T17:55:00.107 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00038933199999999999 2024-02-09T17:55:00.107 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.107 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.107 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:55:00.107 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.342679+0000", 2024-02-09T17:55:00.107 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013281999999999999 2024-02-09T17:55:00.108 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.108 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.108 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 2,10", 2024-02-09T17:55:00.108 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.342816+0000", 2024-02-09T17:55:00.108 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013794 2024-02-09T17:55:00.108 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.108 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.108 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:55:00.108 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.346784+0000", 2024-02-09T17:55:00.108 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0039672960000000004 2024-02-09T17:55:00.108 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.108 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.108 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:00.108 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.357297+0000", 2024-02-09T17:55:00.109 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.010513351000000001 2024-02-09T17:55:00.109 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.109 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.109 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:00.109 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.499635+0000", 2024-02-09T17:55:00.109 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.142337402 2024-02-09T17:55:00.109 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.109 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.109 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:55:00.109 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.499693+0000", 2024-02-09T17:55:00.109 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.8816000000000001e-05 2024-02-09T17:55:00.109 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.109 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.109 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:55:00.110 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.499758+0000", 2024-02-09T17:55:00.110 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.4239999999999995e-05 2024-02-09T17:55:00.110 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:55:00.110 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:55:00.110 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:55:00.110 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.110 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.110 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.16440.0:116800 7.2 7:4bdb7679:::benchmark_data_smithi179_135804_object9009:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e706)", 2024-02-09T17:55:00.110 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:52:10.350228+0000", 2024-02-09T17:55:00.110 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 168.95802997000001, 2024-02-09T17:55:00.110 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.14967960899999999, 2024-02-09T17:55:00.110 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:55:00.110 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:00.111 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:55:00.111 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.16440", 2024-02-09T17:55:00.111 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:55:00.111 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 116800 2024-02-09T17:55:00.111 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.111 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:55:00.111 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.111 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:55:00.111 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.350228+0000", 2024-02-09T17:55:00.111 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:55:00.111 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.111 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.111 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:55:00.111 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.350228+0000", 2024-02-09T17:55:00.111 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:55:00.112 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.112 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.112 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:55:00.112 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.350230+0000", 2024-02-09T17:55:00.112 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.238e-06 2024-02-09T17:55:00.112 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.112 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.112 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:55:00.112 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.350234+0000", 2024-02-09T17:55:00.112 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9709999999999998e-06 2024-02-09T17:55:00.112 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.112 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.113 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:55:00.113 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.350245+0000", 2024-02-09T17:55:00.113 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1059e-05 2024-02-09T17:55:00.113 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.113 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.113 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:55:00.113 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.350253+0000", 2024-02-09T17:55:00.113 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.0059999999999992e-06 2024-02-09T17:55:00.113 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.113 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.113 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:55:00.113 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.350301+0000", 2024-02-09T17:55:00.113 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.8458999999999998e-05 2024-02-09T17:55:00.113 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.114 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.114 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:55:00.114 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.350428+0000", 2024-02-09T17:55:00.114 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012703399999999999 2024-02-09T17:55:00.114 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.114 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.114 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 2,10", 2024-02-09T17:55:00.114 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.350545+0000", 2024-02-09T17:55:00.114 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000116748 2024-02-09T17:55:00.114 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.114 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.114 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:55:00.114 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.353590+0000", 2024-02-09T17:55:00.114 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0030454129999999999 2024-02-09T17:55:00.115 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.115 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.115 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:00.115 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.357528+0000", 2024-02-09T17:55:00.115 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0039377359999999998 2024-02-09T17:55:00.115 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.115 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.115 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:00.115 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.499807+0000", 2024-02-09T17:55:00.115 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.14227865100000001 2024-02-09T17:55:00.115 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.115 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.115 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:55:00.115 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.499860+0000", 2024-02-09T17:55:00.116 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3779000000000001e-05 2024-02-09T17:55:00.116 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.116 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.116 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:55:00.116 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.499908+0000", 2024-02-09T17:55:00.116 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.7515000000000002e-05 2024-02-09T17:55:00.116 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:55:00.116 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:55:00.116 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:55:00.116 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.116 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.116 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.16440.0:116804 7.2 7:4f780da5:::benchmark_data_smithi179_135804_object9013:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e706)", 2024-02-09T17:55:00.116 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:52:10.358875+0000", 2024-02-09T17:55:00.116 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 168.94938360699999, 2024-02-09T17:55:00.117 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.141168294, 2024-02-09T17:55:00.117 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:55:00.117 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:00.117 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:55:00.117 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.16440", 2024-02-09T17:55:00.117 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:55:00.117 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 116804 2024-02-09T17:55:00.117 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.117 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:55:00.117 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.117 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:55:00.117 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.358875+0000", 2024-02-09T17:55:00.117 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:55:00.117 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.118 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.118 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:55:00.118 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.358875+0000", 2024-02-09T17:55:00.118 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:55:00.118 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.118 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.118 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:55:00.118 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.358876+0000", 2024-02-09T17:55:00.118 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4699999999999999e-06 2024-02-09T17:55:00.118 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.118 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.118 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:55:00.118 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.358881+0000", 2024-02-09T17:55:00.118 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.7119999999999999e-06 2024-02-09T17:55:00.118 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.119 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.119 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:55:00.119 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.358894+0000", 2024-02-09T17:55:00.119 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2971e-05 2024-02-09T17:55:00.119 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.119 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.119 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:55:00.119 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.358901+0000", 2024-02-09T17:55:00.119 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.0129999999999996e-06 2024-02-09T17:55:00.119 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.120 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.120 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:55:00.120 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.358951+0000", 2024-02-09T17:55:00.120 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.011e-05 2024-02-09T17:55:00.120 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.120 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.120 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:55:00.120 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.359105+0000", 2024-02-09T17:55:00.120 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000154246 2024-02-09T17:55:00.120 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.120 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.120 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 2,10", 2024-02-09T17:55:00.120 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.359216+0000", 2024-02-09T17:55:00.121 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00011109 2024-02-09T17:55:00.121 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.121 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.121 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:55:00.121 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.365275+0000", 2024-02-09T17:55:00.121 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0060589099999999998 2024-02-09T17:55:00.121 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.121 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.121 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:00.121 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.368357+0000", 2024-02-09T17:55:00.121 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0030816020000000001 2024-02-09T17:55:00.121 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.121 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.121 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:00.122 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.499947+0000", 2024-02-09T17:55:00.122 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.131590336 2024-02-09T17:55:00.122 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.122 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.122 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:55:00.122 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.499998+0000", 2024-02-09T17:55:00.122 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1294000000000001e-05 2024-02-09T17:55:00.122 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.122 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.122 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:55:00.122 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.500043+0000", 2024-02-09T17:55:00.122 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.4539999999999997e-05 2024-02-09T17:55:00.122 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:55:00.122 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:55:00.123 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:55:00.123 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.123 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.123 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.16440.0:116806 7.3 7:c69db5e3:::benchmark_data_smithi179_135804_object9015:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e706)", 2024-02-09T17:55:00.123 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:52:10.358918+0000", 2024-02-09T17:55:00.123 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 168.94933986500001, 2024-02-09T17:55:00.123 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.126767458, 2024-02-09T17:55:00.123 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:55:00.123 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:00.123 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:55:00.123 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.16440", 2024-02-09T17:55:00.123 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:55:00.123 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 116806 2024-02-09T17:55:00.123 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.124 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:55:00.124 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.124 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:55:00.124 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.358918+0000", 2024-02-09T17:55:00.124 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:55:00.124 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.124 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.124 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:55:00.124 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.358918+0000", 2024-02-09T17:55:00.124 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:55:00.124 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.124 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.124 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:55:00.124 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.358919+0000", 2024-02-09T17:55:00.124 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.5400000000000002e-07 2024-02-09T17:55:00.125 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.125 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.125 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:55:00.125 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.358922+0000", 2024-02-09T17:55:00.125 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.7659999999999999e-06 2024-02-09T17:55:00.125 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.125 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.125 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:55:00.125 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.358929+0000", 2024-02-09T17:55:00.125 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.729e-06 2024-02-09T17:55:00.125 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.125 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.125 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:55:00.125 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.358934+0000", 2024-02-09T17:55:00.125 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.0590000000000002e-06 2024-02-09T17:55:00.126 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.126 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.126 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:55:00.126 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.358980+0000", 2024-02-09T17:55:00.126 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.5775000000000001e-05 2024-02-09T17:55:00.126 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.126 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.127 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:55:00.127 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.359101+0000", 2024-02-09T17:55:00.128 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012025 2024-02-09T17:55:00.128 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.128 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.128 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 2,4", 2024-02-09T17:55:00.128 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.359179+0000", 2024-02-09T17:55:00.128 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.8566000000000006e-05 2024-02-09T17:55:00.128 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.128 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.128 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:55:00.128 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.365791+0000", 2024-02-09T17:55:00.128 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0066116889999999996 2024-02-09T17:55:00.128 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.128 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.128 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:00.129 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.365825+0000", 2024-02-09T17:55:00.129 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.4063999999999998e-05 2024-02-09T17:55:00.129 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.129 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.129 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:00.129 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.485604+0000", 2024-02-09T17:55:00.129 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.11977892599999999 2024-02-09T17:55:00.129 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.129 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.129 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:55:00.129 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.485654+0000", 2024-02-09T17:55:00.129 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.9789000000000001e-05 2024-02-09T17:55:00.129 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.129 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.129 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:55:00.130 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.485686+0000", 2024-02-09T17:55:00.130 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.2391000000000002e-05 2024-02-09T17:55:00.130 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:55:00.130 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:55:00.130 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:55:00.130 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.130 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.130 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.16440.0:116808 7.2 7:4d48ea24:::benchmark_data_smithi179_135804_object9017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e706)", 2024-02-09T17:55:00.130 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:52:10.358952+0000", 2024-02-09T17:55:00.130 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 168.94930590499999, 2024-02-09T17:55:00.130 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.141225988, 2024-02-09T17:55:00.130 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:55:00.130 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:00.130 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:55:00.131 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.16440", 2024-02-09T17:55:00.131 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:55:00.131 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 116808 2024-02-09T17:55:00.131 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.131 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:55:00.131 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.131 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:55:00.131 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.358952+0000", 2024-02-09T17:55:00.131 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:55:00.131 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.131 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.131 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:55:00.131 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.358952+0000", 2024-02-09T17:55:00.131 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:55:00.132 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.132 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.132 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:55:00.132 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.358953+0000", 2024-02-09T17:55:00.132 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.34e-07 2024-02-09T17:55:00.132 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.132 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.132 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:55:00.132 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.358955+0000", 2024-02-09T17:55:00.132 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.4940000000000002e-06 2024-02-09T17:55:00.132 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.132 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.132 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:55:00.132 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.358962+0000", 2024-02-09T17:55:00.133 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.2879999999999998e-06 2024-02-09T17:55:00.133 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.133 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.133 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:55:00.133 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.358966+0000", 2024-02-09T17:55:00.133 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.7430000000000002e-06 2024-02-09T17:55:00.133 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.133 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.133 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:55:00.133 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.362213+0000", 2024-02-09T17:55:00.133 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0032467440000000002 2024-02-09T17:55:00.133 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.133 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.133 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:55:00.133 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.362350+0000", 2024-02-09T17:55:00.134 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013652400000000001 2024-02-09T17:55:00.134 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.134 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.134 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 2,10", 2024-02-09T17:55:00.134 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.362488+0000", 2024-02-09T17:55:00.134 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013874100000000001 2024-02-09T17:55:00.134 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.134 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.134 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:55:00.134 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.365452+0000", 2024-02-09T17:55:00.134 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0029636060000000001 2024-02-09T17:55:00.134 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.134 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.134 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:00.135 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.380136+0000", 2024-02-09T17:55:00.135 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.014683926999999999 2024-02-09T17:55:00.135 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.135 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.135 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:00.135 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.500083+0000", 2024-02-09T17:55:00.135 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.11994705899999999 2024-02-09T17:55:00.135 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.135 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.135 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:55:00.135 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.500135+0000", 2024-02-09T17:55:00.135 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1649e-05 2024-02-09T17:55:00.135 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.135 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.135 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:55:00.135 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.500178+0000", 2024-02-09T17:55:00.136 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.3779000000000002e-05 2024-02-09T17:55:00.136 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:55:00.136 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:55:00.136 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:55:00.136 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.136 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.136 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.16440.0:116809 7.3 7:c783cc3a:::benchmark_data_smithi179_135804_object9018:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e706)", 2024-02-09T17:55:00.136 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:52:10.361334+0000", 2024-02-09T17:55:00.136 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 168.94692474799999, 2024-02-09T17:55:00.136 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.12704122600000001, 2024-02-09T17:55:00.136 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:55:00.136 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:00.136 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:55:00.136 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.16440", 2024-02-09T17:55:00.137 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:55:00.137 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 116809 2024-02-09T17:55:00.137 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.137 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:55:00.137 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.137 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:55:00.137 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.361334+0000", 2024-02-09T17:55:00.137 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:55:00.137 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.137 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.137 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:55:00.137 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.361334+0000", 2024-02-09T17:55:00.137 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:55:00.137 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.137 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.137 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:55:00.138 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.361336+0000", 2024-02-09T17:55:00.138 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.9300000000000002e-06 2024-02-09T17:55:00.138 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.138 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.138 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:55:00.138 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.361340+0000", 2024-02-09T17:55:00.138 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.8720000000000001e-06 2024-02-09T17:55:00.138 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.138 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.138 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:55:00.138 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.361354+0000", 2024-02-09T17:55:00.138 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3569e-05 2024-02-09T17:55:00.138 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.138 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.138 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:55:00.139 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.361363+0000", 2024-02-09T17:55:00.139 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.5860000000000005e-06 2024-02-09T17:55:00.139 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.139 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.139 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:55:00.139 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.362764+0000", 2024-02-09T17:55:00.139 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.001401085 2024-02-09T17:55:00.139 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.139 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.139 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:55:00.139 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.362866+0000", 2024-02-09T17:55:00.139 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00010217699999999999 2024-02-09T17:55:00.139 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.139 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.139 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 2,4", 2024-02-09T17:55:00.140 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.362929+0000", 2024-02-09T17:55:00.140 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.3402e-05 2024-02-09T17:55:00.140 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.140 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.140 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:55:00.140 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.365908+0000", 2024-02-09T17:55:00.140 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0029785369999999999 2024-02-09T17:55:00.140 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.140 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.140 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:00.140 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.366951+0000", 2024-02-09T17:55:00.140 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.001042837 2024-02-09T17:55:00.140 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.140 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.140 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:00.141 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.488287+0000", 2024-02-09T17:55:00.141 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.121336234 2024-02-09T17:55:00.141 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.141 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.141 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:55:00.141 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.488340+0000", 2024-02-09T17:55:00.141 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2836e-05 2024-02-09T17:55:00.141 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.141 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.141 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:55:00.141 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.488375+0000", 2024-02-09T17:55:00.141 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.5160999999999998e-05 2024-02-09T17:55:00.141 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:55:00.141 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:55:00.141 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:55:00.141 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.142 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.142 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.16440.0:164156 7.1 e746/731 7:8903898e:::benchmark_data_smithi179_135804_object56365:head v 746'10437, mlcod=674'3627)", 2024-02-09T17:55:00.142 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:53:30.844019+0000", 2024-02-09T17:55:00.142 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 88.464239610999996, 2024-02-09T17:55:00.142 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.14627306800000001, 2024-02-09T17:55:00.142 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:55:00.142 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:00.142 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:55:00.142 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.142 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:55:00.142 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:30.844019+0000", 2024-02-09T17:55:00.142 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:55:00.142 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.142 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.142 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:55:00.143 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:30.844019+0000", 2024-02-09T17:55:00.143 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:55:00.143 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.143 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.143 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:55:00.143 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:30.844020+0000", 2024-02-09T17:55:00.143 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.2900000000000002e-07 2024-02-09T17:55:00.143 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.143 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.143 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:55:00.143 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:30.844029+0000", 2024-02-09T17:55:00.143 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.0119999999999994e-06 2024-02-09T17:55:00.143 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.143 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.143 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:55:00.144 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:30.844041+0000", 2024-02-09T17:55:00.144 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1847e-05 2024-02-09T17:55:00.144 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.144 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.144 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:55:00.144 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:30.844048+0000", 2024-02-09T17:55:00.144 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.2559999999999996e-06 2024-02-09T17:55:00.144 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.144 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.144 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:55:00.144 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:30.844999+0000", 2024-02-09T17:55:00.144 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00095115499999999997 2024-02-09T17:55:00.144 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.144 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.144 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:55:00.145 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:30.845028+0000", 2024-02-09T17:55:00.145 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.9053000000000001e-05 2024-02-09T17:55:00.145 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.145 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.145 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:55:00.145 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:30.990241+0000", 2024-02-09T17:55:00.145 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.145212799 2024-02-09T17:55:00.145 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:55:00.145 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:55:00.145 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:55:00.145 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:30.990292+0000", 2024-02-09T17:55:00.145 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1017000000000003e-05 2024-02-09T17:55:00.145 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:55:00.146 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:55:00.146 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:55:00.146 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:55:00.146 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:55:00.146 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:55:00.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:00 smithi179 ceph-mon[43975]: 1.a scrub starts 2024-02-09T17:55:00.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:00 smithi179 ceph-mon[43975]: 1.a scrub ok 2024-02-09T17:55:00.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:00 smithi179 ceph-mon[43975]: 1.1d scrub starts 2024-02-09T17:55:00.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:00 smithi179 ceph-mon[43975]: 1.1d scrub ok 2024-02-09T17:55:00.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:00 smithi179 ceph-mon[43975]: pgmap v1921: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 9 op/s; 3842/6 objects degraded (64033.333%); 56 KiB/s, 496 objects/s recovering 2024-02-09T17:55:00.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:00 smithi162 ceph-mon[44304]: 1.a scrub starts 2024-02-09T17:55:00.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:00 smithi162 ceph-mon[44304]: 1.a scrub ok 2024-02-09T17:55:00.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:00 smithi162 ceph-mon[44304]: 1.1d scrub starts 2024-02-09T17:55:00.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:00 smithi162 ceph-mon[44304]: 1.1d scrub ok 2024-02-09T17:55:00.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:00 smithi162 ceph-mon[44304]: pgmap v1921: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 9 op/s; 3842/6 objects degraded (64033.333%); 56 KiB/s, 496 objects/s recovering 2024-02-09T17:55:00.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:00 smithi099 ceph-mon[39420]: 1.a scrub starts 2024-02-09T17:55:00.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:00 smithi099 ceph-mon[39420]: 1.a scrub ok 2024-02-09T17:55:00.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:00 smithi099 ceph-mon[39420]: 1.1d scrub starts 2024-02-09T17:55:00.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:00 smithi099 ceph-mon[39420]: 1.1d scrub ok 2024-02-09T17:55:00.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:00 smithi099 ceph-mon[39420]: pgmap v1921: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 9 op/s; 3842/6 objects degraded (64033.333%); 56 KiB/s, 496 objects/s recovering 2024-02-09T17:55:02.309 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:55:02.310 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:55:02.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:02.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:55:02.416 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:55:02.449 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:02.470 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:55:02.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:02.473 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:55:02.574 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:55:02.595 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:02 smithi099 ceph-mon[39420]: 7.0 scrub starts 2024-02-09T17:55:02.595 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:02 smithi099 ceph-mon[39420]: 7.0 scrub ok 2024-02-09T17:55:02.595 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:02 smithi099 ceph-mon[39420]: pgmap v1922: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3842/6 objects degraded (64033.333%); 0 B/s, 449 objects/s recovering 2024-02-09T17:55:02.595 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:02 smithi099 ceph-mon[39420]: 1.11 scrub starts 2024-02-09T17:55:02.595 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:02 smithi099 ceph-mon[39420]: 1.11 scrub ok 2024-02-09T17:55:02.595 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:02 smithi099 ceph-mon[39420]: osdmap e775: 12 total, 12 up, 12 in 2024-02-09T17:55:02.611 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:02.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:02 smithi179 ceph-mon[43975]: 7.0 scrub starts 2024-02-09T17:55:02.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:02 smithi179 ceph-mon[43975]: 7.0 scrub ok 2024-02-09T17:55:02.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:02 smithi179 ceph-mon[43975]: pgmap v1922: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3842/6 objects degraded (64033.333%); 0 B/s, 449 objects/s recovering 2024-02-09T17:55:02.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:02 smithi179 ceph-mon[43975]: 1.11 scrub starts 2024-02-09T17:55:02.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:02 smithi179 ceph-mon[43975]: 1.11 scrub ok 2024-02-09T17:55:02.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:02 smithi179 ceph-mon[43975]: osdmap e775: 12 total, 12 up, 12 in 2024-02-09T17:55:02.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:02 smithi162 ceph-mon[44304]: 7.0 scrub starts 2024-02-09T17:55:02.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:02 smithi162 ceph-mon[44304]: 7.0 scrub ok 2024-02-09T17:55:02.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:02 smithi162 ceph-mon[44304]: pgmap v1922: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3842/6 objects degraded (64033.333%); 0 B/s, 449 objects/s recovering 2024-02-09T17:55:02.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:02 smithi162 ceph-mon[44304]: 1.11 scrub starts 2024-02-09T17:55:02.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:02 smithi162 ceph-mon[44304]: 1.11 scrub ok 2024-02-09T17:55:02.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:02 smithi162 ceph-mon[44304]: osdmap e775: 12 total, 12 up, 12 in 2024-02-09T17:55:03.903 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:55:03.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:03.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:55:04.016 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:55:04.048 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:04.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:03 smithi179 ceph-mon[43975]: 7.a scrub starts 2024-02-09T17:55:04.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:03 smithi179 ceph-mon[43975]: 7.a scrub ok 2024-02-09T17:55:04.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:03 smithi179 ceph-mon[43975]: osdmap e776: 12 total, 12 up, 12 in 2024-02-09T17:55:04.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:03 smithi179 ceph-mon[43975]: pgmap v1925: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2820/6 objects degraded (47000.000%); 0 B/s, 459 objects/s recovering 2024-02-09T17:55:04.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:03 smithi162 ceph-mon[44304]: 7.a scrub starts 2024-02-09T17:55:04.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:03 smithi162 ceph-mon[44304]: 7.a scrub ok 2024-02-09T17:55:04.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:03 smithi162 ceph-mon[44304]: osdmap e776: 12 total, 12 up, 12 in 2024-02-09T17:55:04.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:03 smithi162 ceph-mon[44304]: pgmap v1925: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2820/6 objects degraded (47000.000%); 0 B/s, 459 objects/s recovering 2024-02-09T17:55:04.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:03 smithi099 ceph-mon[39420]: 7.a scrub starts 2024-02-09T17:55:04.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:03 smithi099 ceph-mon[39420]: 7.a scrub ok 2024-02-09T17:55:04.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:03 smithi099 ceph-mon[39420]: osdmap e776: 12 total, 12 up, 12 in 2024-02-09T17:55:04.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:03 smithi099 ceph-mon[39420]: pgmap v1925: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2820/6 objects degraded (47000.000%); 0 B/s, 459 objects/s recovering 2024-02-09T17:55:05.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:04 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 2820/6 objects degraded (47000.000%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-09T17:55:05.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:04 smithi179 ceph-mon[43975]: osdmap e777: 12 total, 12 up, 12 in 2024-02-09T17:55:05.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:04 smithi179 ceph-mon[43975]: 1.1a scrub starts 2024-02-09T17:55:05.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:04 smithi179 ceph-mon[43975]: 1.1a scrub ok 2024-02-09T17:55:05.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:04 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/764024902' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-02-09T17:55:05.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:04 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 2820/6 objects degraded (47000.000%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-09T17:55:05.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:04 smithi162 ceph-mon[44304]: osdmap e777: 12 total, 12 up, 12 in 2024-02-09T17:55:05.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:04 smithi162 ceph-mon[44304]: 1.1a scrub starts 2024-02-09T17:55:05.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:04 smithi162 ceph-mon[44304]: 1.1a scrub ok 2024-02-09T17:55:05.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:04 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/764024902' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-02-09T17:55:05.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:04 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 2820/6 objects degraded (47000.000%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-09T17:55:05.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:04 smithi099 ceph-mon[39420]: osdmap e777: 12 total, 12 up, 12 in 2024-02-09T17:55:05.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:04 smithi099 ceph-mon[39420]: 1.1a scrub starts 2024-02-09T17:55:05.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:04 smithi099 ceph-mon[39420]: 1.1a scrub ok 2024-02-09T17:55:05.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:04 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/764024902' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-02-09T17:55:06.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:06 smithi099 ceph-mon[39420]: 1.17 scrub starts 2024-02-09T17:55:06.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:06 smithi099 ceph-mon[39420]: 1.17 scrub ok 2024-02-09T17:55:06.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:06 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/764024902' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-02-09T17:55:06.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:06 smithi099 ceph-mon[39420]: osdmap e778: 12 total, 12 up, 12 in 2024-02-09T17:55:06.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:06 smithi099 ceph-mon[39420]: pgmap v1928: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2820/6 objects degraded (47000.000%); 0 B/s, 549 objects/s recovering 2024-02-09T17:55:06.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:06 smithi099 ceph-mon[39420]: 1.7 scrub starts 2024-02-09T17:55:06.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:06 smithi099 ceph-mon[39420]: 1.7 scrub ok 2024-02-09T17:55:06.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:06 smithi179 ceph-mon[43975]: 1.17 scrub starts 2024-02-09T17:55:06.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:06 smithi179 ceph-mon[43975]: 1.17 scrub ok 2024-02-09T17:55:06.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:06 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/764024902' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-02-09T17:55:06.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:06 smithi179 ceph-mon[43975]: osdmap e778: 12 total, 12 up, 12 in 2024-02-09T17:55:06.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:06 smithi179 ceph-mon[43975]: pgmap v1928: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2820/6 objects degraded (47000.000%); 0 B/s, 549 objects/s recovering 2024-02-09T17:55:06.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:06 smithi179 ceph-mon[43975]: 1.7 scrub starts 2024-02-09T17:55:06.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:06 smithi179 ceph-mon[43975]: 1.7 scrub ok 2024-02-09T17:55:06.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:06 smithi162 ceph-mon[44304]: 1.17 scrub starts 2024-02-09T17:55:06.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:06 smithi162 ceph-mon[44304]: 1.17 scrub ok 2024-02-09T17:55:06.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:06 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/764024902' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-02-09T17:55:06.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:06 smithi162 ceph-mon[44304]: osdmap e778: 12 total, 12 up, 12 in 2024-02-09T17:55:06.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:06 smithi162 ceph-mon[44304]: pgmap v1928: 47 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2820/6 objects degraded (47000.000%); 0 B/s, 549 objects/s recovering 2024-02-09T17:55:06.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:06 smithi162 ceph-mon[44304]: 1.7 scrub starts 2024-02-09T17:55:06.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:06 smithi162 ceph-mon[44304]: 1.7 scrub ok 2024-02-09T17:55:06.940 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'unique_pool_5' removed 2024-02-09T17:55:07.085 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.11 to 1 (10000) 2024-02-09T17:55:07.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:07 smithi099 ceph-mon[39420]: 1.5 scrub starts 2024-02-09T17:55:07.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:07 smithi099 ceph-mon[39420]: osdmap e779: 12 total, 12 up, 12 in 2024-02-09T17:55:07.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:07 smithi099 ceph-mon[39420]: 1.5 scrub ok 2024-02-09T17:55:07.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:07 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1789511673' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:55:07.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:07 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3178486368' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-09T17:55:07.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:07 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3178486368' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-02-09T17:55:07.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:07 smithi099 ceph-mon[39420]: osdmap e780: 12 total, 12 up, 12 in 2024-02-09T17:55:07.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:07 smithi179 ceph-mon[43975]: 1.5 scrub starts 2024-02-09T17:55:07.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:07 smithi179 ceph-mon[43975]: osdmap e779: 12 total, 12 up, 12 in 2024-02-09T17:55:07.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:07 smithi179 ceph-mon[43975]: 1.5 scrub ok 2024-02-09T17:55:07.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:07 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1789511673' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:55:07.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:07 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3178486368' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-09T17:55:07.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:07 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3178486368' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-02-09T17:55:07.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:07 smithi179 ceph-mon[43975]: osdmap e780: 12 total, 12 up, 12 in 2024-02-09T17:55:07.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:07 smithi162 ceph-mon[44304]: 1.5 scrub starts 2024-02-09T17:55:07.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:07 smithi162 ceph-mon[44304]: osdmap e779: 12 total, 12 up, 12 in 2024-02-09T17:55:07.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:07 smithi162 ceph-mon[44304]: 1.5 scrub ok 2024-02-09T17:55:07.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:07 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1789511673' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:55:07.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:07 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3178486368' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-09T17:55:07.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:07 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3178486368' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-02-09T17:55:07.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:07 smithi162 ceph-mon[44304]: osdmap e780: 12 total, 12 up, 12 in 2024-02-09T17:55:08.238 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 0 1 2024-02-09T17:55:08.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:08 smithi179 ceph-mon[43975]: pgmap v1931: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:55:08.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:08 smithi179 ceph-mon[43975]: 1.c scrub starts 2024-02-09T17:55:08.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:08 smithi179 ceph-mon[43975]: 1.c scrub ok 2024-02-09T17:55:08.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:08 smithi179 ceph-mon[43975]: osdmap e781: 12 total, 12 up, 12 in 2024-02-09T17:55:08.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:08 smithi162 ceph-mon[44304]: pgmap v1931: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:55:08.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:08 smithi162 ceph-mon[44304]: 1.c scrub starts 2024-02-09T17:55:08.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:08 smithi162 ceph-mon[44304]: 1.c scrub ok 2024-02-09T17:55:08.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:08 smithi162 ceph-mon[44304]: osdmap e781: 12 total, 12 up, 12 in 2024-02-09T17:55:08.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:08 smithi099 ceph-mon[39420]: pgmap v1931: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:55:08.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:08 smithi099 ceph-mon[39420]: 1.c scrub starts 2024-02-09T17:55:08.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:08 smithi099 ceph-mon[39420]: 1.c scrub ok 2024-02-09T17:55:08.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:08 smithi099 ceph-mon[39420]: osdmap e781: 12 total, 12 up, 12 in 2024-02-09T17:55:08.840 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-09T17:55:08.912 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:55:08.913 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:55:08.913 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:55:08.913 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:55:08.913 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:55:08.913 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:55:08.913 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:55:08.913 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:55:08.913 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:55:08.913 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:55:08.913 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:55:08.913 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:55:08.913 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:55:08.914 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:55:08.914 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:55:08.914 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:55:08.914 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:55:08.914 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:55:08.914 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:55:08.914 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:55:08.914 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:55:08.914 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:55:08.914 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:55:08.914 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:55:09.106 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:55:09.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:09 smithi162 ceph-mon[44304]: 1.e scrub starts 2024-02-09T17:55:09.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:09 smithi162 ceph-mon[44304]: 1.e scrub ok 2024-02-09T17:55:09.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:09 smithi162 ceph-mon[44304]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-09T17:55:09.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:09 smithi162 ceph-mon[44304]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2820/6 objects degraded (47000.000%), 3 pgs degraded, 3 pgs undersized) 2024-02-09T17:55:09.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:09 smithi162 ceph-mon[44304]: 1.f scrub starts 2024-02-09T17:55:09.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:09 smithi162 ceph-mon[44304]: 1.f scrub ok 2024-02-09T17:55:09.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:09 smithi162 ceph-mon[44304]: osdmap e782: 12 total, 12 up, 12 in 2024-02-09T17:55:09.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:09 smithi179 ceph-mon[43975]: 1.e scrub starts 2024-02-09T17:55:09.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:09 smithi179 ceph-mon[43975]: 1.e scrub ok 2024-02-09T17:55:09.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:09 smithi179 ceph-mon[43975]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-09T17:55:09.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:09 smithi179 ceph-mon[43975]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2820/6 objects degraded (47000.000%), 3 pgs degraded, 3 pgs undersized) 2024-02-09T17:55:09.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:09 smithi179 ceph-mon[43975]: 1.f scrub starts 2024-02-09T17:55:09.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:09 smithi179 ceph-mon[43975]: 1.f scrub ok 2024-02-09T17:55:09.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:09 smithi179 ceph-mon[43975]: osdmap e782: 12 total, 12 up, 12 in 2024-02-09T17:55:09.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:09 smithi099 ceph-mon[39420]: 1.e scrub starts 2024-02-09T17:55:09.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:09 smithi099 ceph-mon[39420]: 1.e scrub ok 2024-02-09T17:55:09.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:09 smithi099 ceph-mon[39420]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-09T17:55:09.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:09 smithi099 ceph-mon[39420]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2820/6 objects degraded (47000.000%), 3 pgs degraded, 3 pgs undersized) 2024-02-09T17:55:09.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:09 smithi099 ceph-mon[39420]: 1.f scrub starts 2024-02-09T17:55:09.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:09 smithi099 ceph-mon[39420]: 1.f scrub ok 2024-02-09T17:55:09.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:09 smithi099 ceph-mon[39420]: osdmap e782: 12 total, 12 up, 12 in 2024-02-09T17:55:10.152 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-02-09T17:55:10.153 INFO:tasks.radosbench:Beginning radosbench... 2024-02-09T17:55:10.153 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_6 2024-02-09T17:55:10.153 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd pool create unique_pool_6 16 2024-02-09T17:55:10.359 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:10 smithi099 ceph-mon[39420]: 1.d scrub starts 2024-02-09T17:55:10.359 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:10 smithi099 ceph-mon[39420]: 1.d scrub ok 2024-02-09T17:55:10.359 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:10 smithi099 ceph-mon[39420]: 1.14 scrub starts 2024-02-09T17:55:10.359 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:10 smithi099 ceph-mon[39420]: 1.14 scrub ok 2024-02-09T17:55:10.359 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:10 smithi099 ceph-mon[39420]: pgmap v1934: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:55:10.359 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:10 smithi099 ceph-mon[39420]: 1.2 scrub starts 2024-02-09T17:55:10.359 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:10 smithi099 ceph-mon[39420]: 1.2 scrub ok 2024-02-09T17:55:10.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:10 smithi179 ceph-mon[43975]: 1.d scrub starts 2024-02-09T17:55:10.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:10 smithi179 ceph-mon[43975]: 1.d scrub ok 2024-02-09T17:55:10.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:10 smithi179 ceph-mon[43975]: 1.14 scrub starts 2024-02-09T17:55:10.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:10 smithi179 ceph-mon[43975]: 1.14 scrub ok 2024-02-09T17:55:10.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:10 smithi179 ceph-mon[43975]: pgmap v1934: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:55:10.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:10 smithi179 ceph-mon[43975]: 1.2 scrub starts 2024-02-09T17:55:10.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:10 smithi179 ceph-mon[43975]: 1.2 scrub ok 2024-02-09T17:55:10.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:10 smithi162 ceph-mon[44304]: 1.d scrub starts 2024-02-09T17:55:10.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:10 smithi162 ceph-mon[44304]: 1.d scrub ok 2024-02-09T17:55:10.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:10 smithi162 ceph-mon[44304]: 1.14 scrub starts 2024-02-09T17:55:10.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:10 smithi162 ceph-mon[44304]: 1.14 scrub ok 2024-02-09T17:55:10.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:10 smithi162 ceph-mon[44304]: pgmap v1934: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:55:10.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:10 smithi162 ceph-mon[44304]: 1.2 scrub starts 2024-02-09T17:55:10.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:10 smithi162 ceph-mon[44304]: 1.2 scrub ok 2024-02-09T17:55:10.807 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:55:10.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:10.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:55:10.856 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:55:10.857 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:55:10.913 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:55:10.947 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:10.969 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:55:10.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:10.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:55:11.073 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:55:11.080 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is unset 2024-02-09T17:55:11.105 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:11.127 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:55:11.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:11.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:55:11.231 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:55:11.264 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:11.285 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:55:11.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:11.289 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:55:11.390 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:55:11.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:11 smithi179 ceph-mon[43975]: 1.b scrub starts 2024-02-09T17:55:11.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:11 smithi179 ceph-mon[43975]: 1.b scrub ok 2024-02-09T17:55:11.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:11 smithi179 ceph-mon[43975]: 1.8 scrub starts 2024-02-09T17:55:11.418 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:11 smithi179 ceph-mon[43975]: 1.8 scrub ok 2024-02-09T17:55:11.418 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:11 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3128585170' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:55:11.420 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:11.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:11 smithi162 ceph-mon[44304]: 1.b scrub starts 2024-02-09T17:55:11.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:11 smithi162 ceph-mon[44304]: 1.b scrub ok 2024-02-09T17:55:11.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:11 smithi162 ceph-mon[44304]: 1.8 scrub starts 2024-02-09T17:55:11.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:11 smithi162 ceph-mon[44304]: 1.8 scrub ok 2024-02-09T17:55:11.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:11 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3128585170' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:55:11.439 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:55:11.441 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:11.441 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:55:11.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:11 smithi099 ceph-mon[39420]: 1.b scrub starts 2024-02-09T17:55:11.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:11 smithi099 ceph-mon[39420]: 1.b scrub ok 2024-02-09T17:55:11.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:11 smithi099 ceph-mon[39420]: 1.8 scrub starts 2024-02-09T17:55:11.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:11 smithi099 ceph-mon[39420]: 1.8 scrub ok 2024-02-09T17:55:11.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:11 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3128585170' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:55:11.542 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:55:11.572 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:11.921 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:55:11.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:11.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:55:12.026 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:55:12.053 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:12.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:12 smithi162 ceph-mon[44304]: 1.10 scrub starts 2024-02-09T17:55:12.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:12 smithi162 ceph-mon[44304]: 1.10 scrub ok 2024-02-09T17:55:12.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:12 smithi162 ceph-mon[44304]: 1.6 scrub starts 2024-02-09T17:55:12.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:12 smithi162 ceph-mon[44304]: 1.6 scrub ok 2024-02-09T17:55:12.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:12 smithi162 ceph-mon[44304]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:55:12.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:12 smithi162 ceph-mon[44304]: Cluster is now healthy 2024-02-09T17:55:12.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:12 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3128585170' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:55:12.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:12 smithi162 ceph-mon[44304]: osdmap e783: 12 total, 12 up, 12 in 2024-02-09T17:55:12.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:12 smithi162 ceph-mon[44304]: pgmap v1936: 31 pgs: 31 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:55:12.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:12 smithi162 ceph-mon[44304]: osdmap e784: 12 total, 12 up, 12 in 2024-02-09T17:55:12.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:12 smithi099 ceph-mon[39420]: 1.10 scrub starts 2024-02-09T17:55:12.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:12 smithi099 ceph-mon[39420]: 1.10 scrub ok 2024-02-09T17:55:12.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:12 smithi099 ceph-mon[39420]: 1.6 scrub starts 2024-02-09T17:55:12.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:12 smithi099 ceph-mon[39420]: 1.6 scrub ok 2024-02-09T17:55:12.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:12 smithi099 ceph-mon[39420]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:55:12.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:12 smithi099 ceph-mon[39420]: Cluster is now healthy 2024-02-09T17:55:12.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:12 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3128585170' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:55:12.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:12 smithi099 ceph-mon[39420]: osdmap e783: 12 total, 12 up, 12 in 2024-02-09T17:55:12.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:12 smithi099 ceph-mon[39420]: pgmap v1936: 31 pgs: 31 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:55:12.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:12 smithi099 ceph-mon[39420]: osdmap e784: 12 total, 12 up, 12 in 2024-02-09T17:55:12.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:12 smithi179 ceph-mon[43975]: 1.10 scrub starts 2024-02-09T17:55:12.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:12 smithi179 ceph-mon[43975]: 1.10 scrub ok 2024-02-09T17:55:12.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:12 smithi179 ceph-mon[43975]: 1.6 scrub starts 2024-02-09T17:55:12.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:12 smithi179 ceph-mon[43975]: 1.6 scrub ok 2024-02-09T17:55:12.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:12 smithi179 ceph-mon[43975]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:55:12.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:12 smithi179 ceph-mon[43975]: Cluster is now healthy 2024-02-09T17:55:12.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:12 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3128585170' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:55:12.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:12 smithi179 ceph-mon[43975]: osdmap e783: 12 total, 12 up, 12 in 2024-02-09T17:55:12.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:12 smithi179 ceph-mon[43975]: pgmap v1936: 31 pgs: 31 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:55:12.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:12 smithi179 ceph-mon[43975]: osdmap e784: 12 total, 12 up, 12 in 2024-02-09T17:55:12.536 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:55:12.919 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:55:12.919 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:55:13.258 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:13 smithi162 ceph-mon[44304]: 1.1b scrub starts 2024-02-09T17:55:13.258 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:13 smithi162 ceph-mon[44304]: 1.1b scrub ok 2024-02-09T17:55:13.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:13 smithi179 ceph-mon[43975]: 1.1b scrub starts 2024-02-09T17:55:13.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:13 smithi179 ceph-mon[43975]: 1.1b scrub ok 2024-02-09T17:55:13.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:13 smithi099 ceph-mon[39420]: 1.1b scrub starts 2024-02-09T17:55:13.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:13 smithi099 ceph-mon[39420]: 1.1b scrub ok 2024-02-09T17:55:13.600 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-09T17:55:13.871 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:55:14.103 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:55:14.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:14.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:55:14.138 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:55:14.209 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:55:14.239 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:14.258 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:55:14.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:14.261 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:55:14.349 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:14 smithi162 ceph-mon[44304]: 1.1 scrub starts 2024-02-09T17:55:14.349 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:14 smithi162 ceph-mon[44304]: 1.1 scrub ok 2024-02-09T17:55:14.349 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:14 smithi162 ceph-mon[44304]: pgmap v1938: 31 pgs: 31 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:55:14.362 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:55:14.388 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:14 smithi099 ceph-mon[39420]: 1.1 scrub starts 2024-02-09T17:55:14.388 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:14 smithi099 ceph-mon[39420]: 1.1 scrub ok 2024-02-09T17:55:14.388 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:14 smithi099 ceph-mon[39420]: pgmap v1938: 31 pgs: 31 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:55:14.390 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:14.412 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:55:14.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:14.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:55:14.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:14 smithi179 ceph-mon[43975]: 1.1 scrub starts 2024-02-09T17:55:14.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:14 smithi179 ceph-mon[43975]: 1.1 scrub ok 2024-02-09T17:55:14.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:14 smithi179 ceph-mon[43975]: pgmap v1938: 31 pgs: 31 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:55:14.481 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set noscrub 2024-02-09T17:55:14.516 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:55:14.545 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:14.643 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:55:14.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:14.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:55:14.748 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:55:14.771 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:55:14.781 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:14.802 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:55:14.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:14.806 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:55:14.907 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:55:14.937 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:14.958 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:55:14.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:14.961 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:55:15.062 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:55:15.089 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:15.108 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:55:15.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:15.110 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:55:15.195 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'unique_pool_6' created 2024-02-09T17:55:15.211 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:55:15.239 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:15.420 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:15 smithi162 ceph-mon[44304]: 1.9 scrub starts 2024-02-09T17:55:15.420 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:15 smithi162 ceph-mon[44304]: 1.9 scrub ok 2024-02-09T17:55:15.420 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:15 smithi162 ceph-mon[44304]: 1.4 scrub starts 2024-02-09T17:55:15.421 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:15 smithi162 ceph-mon[44304]: 1.4 scrub ok 2024-02-09T17:55:15.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:15 smithi179 ceph-mon[43975]: 1.9 scrub starts 2024-02-09T17:55:15.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:15 smithi179 ceph-mon[43975]: 1.9 scrub ok 2024-02-09T17:55:15.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:15 smithi179 ceph-mon[43975]: 1.4 scrub starts 2024-02-09T17:55:15.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:15 smithi179 ceph-mon[43975]: 1.4 scrub ok 2024-02-09T17:55:15.438 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:55:15.441 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:15.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:55:15.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:15 smithi099 ceph-mon[39420]: 1.9 scrub starts 2024-02-09T17:55:15.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:15 smithi099 ceph-mon[39420]: 1.9 scrub ok 2024-02-09T17:55:15.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:15 smithi099 ceph-mon[39420]: 1.4 scrub starts 2024-02-09T17:55:15.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:15 smithi099 ceph-mon[39420]: 1.4 scrub ok 2024-02-09T17:55:15.542 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:55:15.571 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:15.592 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:55:15.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:15.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:55:15.696 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:55:15.714 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:55:15.715 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:55:15.726 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:16.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:16 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:55:16.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:16 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:55:16.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:16 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/500951602' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]: dispatch 2024-02-09T17:55:16.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:16 smithi162 ceph-mon[44304]: 1.19 scrub starts 2024-02-09T17:55:16.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:16 smithi162 ceph-mon[44304]: 1.19 scrub ok 2024-02-09T17:55:16.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:16 smithi162 ceph-mon[44304]: pgmap v1939: 31 pgs: 31 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:55:16.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:16 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/500951602' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-02-09T17:55:16.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:16 smithi162 ceph-mon[44304]: osdmap e785: 12 total, 12 up, 12 in 2024-02-09T17:55:16.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:16 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:55:16.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:16 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:55:16.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:16 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/500951602' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]: dispatch 2024-02-09T17:55:16.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:16 smithi099 ceph-mon[39420]: 1.19 scrub starts 2024-02-09T17:55:16.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:16 smithi099 ceph-mon[39420]: 1.19 scrub ok 2024-02-09T17:55:16.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:16 smithi099 ceph-mon[39420]: pgmap v1939: 31 pgs: 31 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:55:16.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:16 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/500951602' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-02-09T17:55:16.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:16 smithi099 ceph-mon[39420]: osdmap e785: 12 total, 12 up, 12 in 2024-02-09T17:55:16.530 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd pool application enable unique_pool_6 rados --yes-i-really-mean-it || true 2024-02-09T17:55:16.574 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-09T17:55:16.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:16 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:55:16.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:16 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:55:16.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:16 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/500951602' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]: dispatch 2024-02-09T17:55:16.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:16 smithi179 ceph-mon[43975]: 1.19 scrub starts 2024-02-09T17:55:16.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:16 smithi179 ceph-mon[43975]: 1.19 scrub ok 2024-02-09T17:55:16.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:16 smithi179 ceph-mon[43975]: pgmap v1939: 31 pgs: 31 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:55:16.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:16 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/500951602' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-02-09T17:55:16.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:16 smithi179 ceph-mon[43975]: osdmap e785: 12 total, 12 up, 12 in 2024-02-09T17:55:16.837 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:55:17.207 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.0 to 1 (10000) 2024-02-09T17:55:17.335 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:17 smithi162 ceph-mon[44304]: osdmap e786: 12 total, 12 up, 12 in 2024-02-09T17:55:17.335 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:17 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/655489049' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-09T17:55:17.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:17 smithi099 ceph-mon[39420]: osdmap e786: 12 total, 12 up, 12 in 2024-02-09T17:55:17.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:17 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/655489049' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-09T17:55:17.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:17 smithi179 ceph-mon[43975]: osdmap e786: 12 total, 12 up, 12 in 2024-02-09T17:55:17.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:17 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/655489049' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-09T17:55:17.953 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 2 1 2024-02-09T17:55:17.976 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:55:17.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:17.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:55:18.080 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:55:18.111 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:18.132 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:55:18.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:18.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:55:18.237 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:55:18.264 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:18.421 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:55:18.425 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:18 smithi162 ceph-mon[44304]: 1.1e scrub starts 2024-02-09T17:55:18.425 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:18 smithi162 ceph-mon[44304]: 1.1e scrub ok 2024-02-09T17:55:18.425 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:18 smithi162 ceph-mon[44304]: pgmap v1942: 47 pgs: 3 creating+activating, 2 creating+peering, 8 unknown, 34 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:55:18.425 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:18 smithi162 ceph-mon[44304]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-09T17:55:18.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:18 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/655489049' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-09T17:55:18.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:18 smithi162 ceph-mon[44304]: osdmap e787: 12 total, 12 up, 12 in 2024-02-09T17:55:18.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:18.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:55:18.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:18 smithi099 ceph-mon[39420]: 1.1e scrub starts 2024-02-09T17:55:18.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:18 smithi099 ceph-mon[39420]: 1.1e scrub ok 2024-02-09T17:55:18.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:18 smithi099 ceph-mon[39420]: pgmap v1942: 47 pgs: 3 creating+activating, 2 creating+peering, 8 unknown, 34 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:55:18.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:18 smithi099 ceph-mon[39420]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-09T17:55:18.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:18 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/655489049' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-09T17:55:18.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:18 smithi099 ceph-mon[39420]: osdmap e787: 12 total, 12 up, 12 in 2024-02-09T17:55:18.527 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:55:18.557 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:18.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:18 smithi179 ceph-mon[43975]: 1.1e scrub starts 2024-02-09T17:55:18.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:18 smithi179 ceph-mon[43975]: 1.1e scrub ok 2024-02-09T17:55:18.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:18 smithi179 ceph-mon[43975]: pgmap v1942: 47 pgs: 3 creating+activating, 2 creating+peering, 8 unknown, 34 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:55:18.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:18 smithi179 ceph-mon[43975]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-09T17:55:18.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:18 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/655489049' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-09T17:55:18.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:18 smithi179 ceph-mon[43975]: osdmap e787: 12 total, 12 up, 12 in 2024-02-09T17:55:18.713 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:55:18.713 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:55:19.188 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:55:19.188 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:55:19.188 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:55:19.188 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:55:19.188 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:55:19.189 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:55:19.189 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:55:19.189 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:55:19.189 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:55:19.189 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:55:19.189 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:55:19.189 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:55:19.189 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:55:19.189 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:55:19.189 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:55:19.190 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:55:19.190 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:55:19.190 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:55:19.190 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:55:19.190 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:55:19.190 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:55:19.190 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:55:19.190 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:55:19.190 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:55:19.326 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:19 smithi162 ceph-mon[44304]: osdmap e788: 12 total, 12 up, 12 in 2024-02-09T17:55:19.326 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:19 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/913731019' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:55:19.327 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-09T17:55:19.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:19 smithi099 ceph-mon[39420]: osdmap e788: 12 total, 12 up, 12 in 2024-02-09T17:55:19.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:19 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/913731019' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:55:19.597 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:55:19.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:19 smithi179 ceph-mon[43975]: osdmap e788: 12 total, 12 up, 12 in 2024-02-09T17:55:19.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:19 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/913731019' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:55:19.907 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:55:19.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:19.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:55:20.012 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:55:20.040 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:20.060 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:55:20.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:20.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:55:20.164 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:55:20.196 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:20.217 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:55:20.220 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:20 smithi099 ceph-mon[39420]: pgmap v1945: 47 pgs: 1 remapped+peering, 1 peering, 3 creating+activating, 2 creating+peering, 40 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:55:20.220 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:20 smithi099 ceph-mon[39420]: osdmap e789: 12 total, 12 up, 12 in 2024-02-09T17:55:20.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:20.221 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:55:20.322 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:55:20.327 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:55:20.353 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:20.373 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:55:20.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:20.376 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:55:20.477 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:55:20.503 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:20 smithi162 ceph-mon[44304]: pgmap v1945: 47 pgs: 1 remapped+peering, 1 peering, 3 creating+activating, 2 creating+peering, 40 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:55:20.504 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:20 smithi162 ceph-mon[44304]: osdmap e789: 12 total, 12 up, 12 in 2024-02-09T17:55:20.506 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:20.526 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:55:20.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:20.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:55:20.630 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:55:20.658 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:20.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:20 smithi179 ceph-mon[43975]: pgmap v1945: 47 pgs: 1 remapped+peering, 1 peering, 3 creating+activating, 2 creating+peering, 40 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:55:20.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:20 smithi179 ceph-mon[43975]: osdmap e789: 12 total, 12 up, 12 in 2024-02-09T17:55:21.086 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:55:21.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:21.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:55:21.191 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:55:21.221 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:21.422 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:55:21.424 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:21 smithi162 ceph-mon[44304]: 1.0 scrub starts 2024-02-09T17:55:21.424 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:21 smithi162 ceph-mon[44304]: 1.0 scrub ok 2024-02-09T17:55:21.424 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:21 smithi162 ceph-mon[44304]: osdmap e790: 12 total, 12 up, 12 in 2024-02-09T17:55:21.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:21.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:55:21.427 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:55:21.427 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-09T17:55:21.427 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-02-09T17:55:21.427 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:55:21.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:21 smithi099 ceph-mon[39420]: 1.0 scrub starts 2024-02-09T17:55:21.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:21 smithi099 ceph-mon[39420]: 1.0 scrub ok 2024-02-09T17:55:21.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:21 smithi099 ceph-mon[39420]: osdmap e790: 12 total, 12 up, 12 in 2024-02-09T17:55:21.527 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:55:21.560 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:21.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:21 smithi179 ceph-mon[43975]: 1.0 scrub starts 2024-02-09T17:55:21.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:21 smithi179 ceph-mon[43975]: 1.0 scrub ok 2024-02-09T17:55:21.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:21 smithi179 ceph-mon[43975]: osdmap e790: 12 total, 12 up, 12 in 2024-02-09T17:55:21.908 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is set 2024-02-09T17:55:21.908 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:55:21.909 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:55:21.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:21.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:55:21.935 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:55:22.017 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:55:22.046 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:22.144 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:55:22.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:22.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:55:22.159 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-09T17:55:22.249 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:55:22.282 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:22.303 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:55:22.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:22.306 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:55:22.407 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:55:22.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:22 smithi162 ceph-mon[44304]: pgmap v1948: 47 pgs: 1 remapped+peering, 1 peering, 3 creating+activating, 42 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:55:22.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:22 smithi162 ceph-mon[44304]: osdmap e791: 12 total, 12 up, 12 in 2024-02-09T17:55:22.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:22 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2646913065' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:55:22.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:22 smithi162 ceph-mon[44304]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:55:22.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:22 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2646913065' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:55:22.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:22 smithi162 ceph-mon[44304]: osdmap e792: 12 total, 12 up, 12 in 2024-02-09T17:55:22.435 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:22.464 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:55:22.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:22.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:55:22.470 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:55:22.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:22 smithi099 ceph-mon[39420]: pgmap v1948: 47 pgs: 1 remapped+peering, 1 peering, 3 creating+activating, 42 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:55:22.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:22 smithi099 ceph-mon[39420]: osdmap e791: 12 total, 12 up, 12 in 2024-02-09T17:55:22.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:22 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2646913065' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:55:22.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:22 smithi099 ceph-mon[39420]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:55:22.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:22 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2646913065' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:55:22.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:22 smithi099 ceph-mon[39420]: osdmap e792: 12 total, 12 up, 12 in 2024-02-09T17:55:22.569 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:55:22.595 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:22.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:22 smithi179 ceph-mon[43975]: pgmap v1948: 47 pgs: 1 remapped+peering, 1 peering, 3 creating+activating, 42 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:55:22.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:22 smithi179 ceph-mon[43975]: osdmap e791: 12 total, 12 up, 12 in 2024-02-09T17:55:22.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:22 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2646913065' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:55:22.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:22 smithi179 ceph-mon[43975]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:55:22.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:22 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2646913065' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:55:22.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:22 smithi179 ceph-mon[43975]: osdmap e792: 12 total, 12 up, 12 in 2024-02-09T17:55:22.719 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:55:22.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:22.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:55:22.824 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:55:22.850 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:22.948 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:55:22.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:22.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:55:23.053 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:55:23.080 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:23.235 INFO:teuthology.orchestra.run.smithi099.stderr:enabled application 'rados' on pool 'unique_pool_6' 2024-02-09T17:55:23.342 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:23 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/354613616' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-09T17:55:23.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:23 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/354613616' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-09T17:55:23.629 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:55:23.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:23.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:55:23.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:23 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/354613616' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-09T17:55:23.734 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:55:23.762 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:23.910 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:55:23.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:23.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:55:24.015 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:55:24.048 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:24.069 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:55:24.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:24.072 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:55:24.173 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:55:24.188 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:55:24.188 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-09T17:55:24.188 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-02-09T17:55:24.188 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-02-09T17:55:24.189 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:55:24.201 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:24.221 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:55:24.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:24.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:55:24.325 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:55:24.352 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:24.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:24 smithi179 ceph-mon[43975]: pgmap v1951: 47 pgs: 1 remapped+peering, 46 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:55:24.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:24 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/354613616' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-09T17:55:24.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:24 smithi179 ceph-mon[43975]: osdmap e793: 12 total, 12 up, 12 in 2024-02-09T17:55:24.451 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:55:24.452 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:24 smithi162 ceph-mon[44304]: pgmap v1951: 47 pgs: 1 remapped+peering, 46 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:55:24.453 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:24 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/354613616' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-09T17:55:24.453 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:24 smithi162 ceph-mon[44304]: osdmap e793: 12 total, 12 up, 12 in 2024-02-09T17:55:24.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:24.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:55:24.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:24 smithi099 ceph-mon[39420]: pgmap v1951: 47 pgs: 1 remapped+peering, 46 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:55:24.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:24 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/354613616' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-09T17:55:24.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:24 smithi099 ceph-mon[39420]: osdmap e793: 12 total, 12 up, 12 in 2024-02-09T17:55:24.555 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:55:24.587 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:24.608 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:55:24.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:24.611 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:55:24.712 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:55:24.744 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:24.764 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:55:24.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:24.768 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:55:24.869 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:55:24.880 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:55:24.902 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:24.950 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-09T17:55:25.214 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:55:25.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:25 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:55:25.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:25 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:55:25.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:25 smithi099 ceph-mon[39420]: osdmap e794: 12 total, 12 up, 12 in 2024-02-09T17:55:25.500 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:25 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:55:25.500 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:25 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:55:25.500 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:25 smithi162 ceph-mon[44304]: osdmap e794: 12 total, 12 up, 12 in 2024-02-09T17:55:25.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:25 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:55:25.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:25 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:55:25.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:25 smithi179 ceph-mon[43975]: osdmap e794: 12 total, 12 up, 12 in 2024-02-09T17:55:26.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:26 smithi162 ceph-mon[44304]: pgmap v1954: 47 pgs: 47 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:55:26.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:26 smithi162 ceph-mon[44304]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-09T17:55:26.529 DEBUG:teuthology.orchestra.run.smithi179:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_6 bench 90 write' 2024-02-09T17:55:26.532 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-02-09T17:55:26.532 INFO:teuthology.orchestra.run:waiting for 3000 2024-02-09T17:55:26.536 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:26 smithi099 ceph-mon[39420]: pgmap v1954: 47 pgs: 47 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:55:26.536 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:26 smithi099 ceph-mon[39420]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-09T17:55:26.553 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:55:26.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:26.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:55:26.586 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:hints = 1 2024-02-09T17:55:26.586 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-02-09T17:55:26.586 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Object prefix: benchmark_data_smithi179_148205 2024-02-09T17:55:26.586 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-09T17:55:26.586 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 0 0 0 0 0 0 - 0 2024-02-09T17:55:26.661 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:55:26.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:26 smithi179 ceph-mon[43975]: pgmap v1954: 47 pgs: 47 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T17:55:26.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:26 smithi179 ceph-mon[43975]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-09T17:55:26.679 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-09T17:55:26.704 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:26.728 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:55:26.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:26.730 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:55:26.831 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:55:26.861 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:26.935 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:55:26.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:26.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:55:27.040 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:55:27.088 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:27.092 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:55:27.092 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-02-09T17:55:27.093 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-02-09T17:55:27.093 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-02-09T17:55:27.093 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.093 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "rep_scrubmap(1.18 e525 from shard 2)", 2024-02-09T17:55:27.093 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:52:07.856326+0000", 2024-02-09T17:55:27.093 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 199.23146736300001, 2024-02-09T17:55:27.093 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.029983534999999999, 2024-02-09T17:55:27.093 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:55:27.093 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-09T17:55:27.094 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:55:27.094 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.094 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:55:27.094 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:07.856326+0000", 2024-02-09T17:55:27.094 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:27.094 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.094 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.094 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:55:27.094 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:07.856326+0000", 2024-02-09T17:55:27.094 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:27.094 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.095 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.095 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:55:27.095 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:07.856327+0000", 2024-02-09T17:55:27.095 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.04e-06 2024-02-09T17:55:27.095 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.095 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.095 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:55:27.095 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:07.856333+0000", 2024-02-09T17:55:27.095 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4929999999999998e-06 2024-02-09T17:55:27.095 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.096 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.096 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:55:27.096 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:07.856344+0000", 2024-02-09T17:55:27.096 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1697e-05 2024-02-09T17:55:27.096 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.096 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.096 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:55:27.096 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:07.856356+0000", 2024-02-09T17:55:27.096 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1656e-05 2024-02-09T17:55:27.097 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.097 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.097 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:55:27.097 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:07.886262+0000", 2024-02-09T17:55:27.097 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.029905902000000002 2024-02-09T17:55:27.097 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.097 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.097 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:55:27.097 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:07.886274+0000", 2024-02-09T17:55:27.097 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1735e-05 2024-02-09T17:55:27.097 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.098 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.098 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:55:27.098 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:07.886310+0000", 2024-02-09T17:55:27.098 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6012e-05 2024-02-09T17:55:27.098 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:27.098 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:55:27.098 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:27.098 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.098 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.098 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "rep_scrubmap(1.18 e525 from shard 7)", 2024-02-09T17:55:27.099 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:52:07.857823+0000", 2024-02-09T17:55:27.099 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 199.22997081, 2024-02-09T17:55:27.099 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.032244243999999998, 2024-02-09T17:55:27.099 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:55:27.099 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-09T17:55:27.099 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:55:27.099 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.099 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:55:27.099 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:07.857823+0000", 2024-02-09T17:55:27.099 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:27.100 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.100 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.100 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:55:27.100 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:07.857823+0000", 2024-02-09T17:55:27.100 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:27.100 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.100 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.100 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:55:27.100 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:07.857824+0000", 2024-02-09T17:55:27.101 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5199999999999995e-07 2024-02-09T17:55:27.101 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.101 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.102 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:55:27.102 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:07.857829+0000", 2024-02-09T17:55:27.102 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7420000000000003e-06 2024-02-09T17:55:27.102 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.102 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.102 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:55:27.102 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:07.857840+0000", 2024-02-09T17:55:27.102 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0733999999999999e-05 2024-02-09T17:55:27.102 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.102 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.102 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:55:27.102 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:07.857848+0000", 2024-02-09T17:55:27.103 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1249999999999993e-06 2024-02-09T17:55:27.103 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.103 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.103 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:55:27.103 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:07.890031+0000", 2024-02-09T17:55:27.103 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.032182708999999997 2024-02-09T17:55:27.103 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.103 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.103 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:55:27.103 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:07.890044+0000", 2024-02-09T17:55:27.104 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3044000000000001e-05 2024-02-09T17:55:27.104 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.104 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.104 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:55:27.104 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:07.890067+0000", 2024-02-09T17:55:27.104 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3037999999999999e-05 2024-02-09T17:55:27.104 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:27.104 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:55:27.104 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:27.104 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.104 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.104 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.16440.0:183052 7.b e749/661 7:d60a7b48:::benchmark_data_smithi179_135804_object75261:head v 749'11340, mlcod=749'11337)", 2024-02-09T17:55:27.105 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:53:59.682510+0000", 2024-02-09T17:55:27.105 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 87.405283840999999, 2024-02-09T17:55:27.105 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018230869, 2024-02-09T17:55:27.105 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:55:27.105 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:27.105 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:55:27.105 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.105 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:55:27.105 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:59.682510+0000", 2024-02-09T17:55:27.105 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:27.105 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.106 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.106 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:55:27.106 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:59.682510+0000", 2024-02-09T17:55:27.106 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:27.106 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.106 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.106 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:55:27.106 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:59.682511+0000", 2024-02-09T17:55:27.106 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.232e-06 2024-02-09T17:55:27.106 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.106 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.107 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:55:27.107 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:59.682520+0000", 2024-02-09T17:55:27.107 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8089999999999999e-06 2024-02-09T17:55:27.107 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.107 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.107 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:55:27.107 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:59.682533+0000", 2024-02-09T17:55:27.107 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3040000000000001e-05 2024-02-09T17:55:27.107 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.108 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.108 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:55:27.108 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:59.682541+0000", 2024-02-09T17:55:27.108 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1659999999999994e-06 2024-02-09T17:55:27.108 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.108 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.108 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:55:27.108 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:59.682606+0000", 2024-02-09T17:55:27.108 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5380000000000001e-05 2024-02-09T17:55:27.108 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.108 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.108 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:55:27.109 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:59.682632+0000", 2024-02-09T17:55:27.109 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5694000000000002e-05 2024-02-09T17:55:27.109 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.109 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.109 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:55:27.109 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:59.700705+0000", 2024-02-09T17:55:27.109 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018072988000000002 2024-02-09T17:55:27.109 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.110 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.110 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:55:27.110 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:59.700741+0000", 2024-02-09T17:55:27.110 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5559999999999998e-05 2024-02-09T17:55:27.110 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:27.110 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:55:27.111 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:27.111 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.111 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.111 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.16440.0:185444 7.0 e750/661 7:0d715ece:::benchmark_data_smithi179_135804_object77653:head v 750'11748, mlcod=750'11744)", 2024-02-09T17:55:27.111 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:54:03.399909+0000", 2024-02-09T17:55:27.112 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 83.687884780999994, 2024-02-09T17:55:27.112 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018655689, 2024-02-09T17:55:27.112 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:55:27.112 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:27.112 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:55:27.112 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.112 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:55:27.112 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.399909+0000", 2024-02-09T17:55:27.112 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:27.112 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.112 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.113 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:55:27.113 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.399909+0000", 2024-02-09T17:55:27.113 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:27.113 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.113 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.113 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:55:27.113 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.399909+0000", 2024-02-09T17:55:27.113 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.53e-07 2024-02-09T17:55:27.113 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.113 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.113 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:55:27.114 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.399913+0000", 2024-02-09T17:55:27.114 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.267e-06 2024-02-09T17:55:27.114 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.114 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.114 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:55:27.114 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.399920+0000", 2024-02-09T17:55:27.114 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3239999999999999e-06 2024-02-09T17:55:27.114 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.114 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.114 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:55:27.114 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.399924+0000", 2024-02-09T17:55:27.115 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7310000000000001e-06 2024-02-09T17:55:27.115 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.115 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.115 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:55:27.115 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.404062+0000", 2024-02-09T17:55:27.115 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041377410000000003 2024-02-09T17:55:27.115 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.115 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.115 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:55:27.115 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.404087+0000", 2024-02-09T17:55:27.115 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4600000000000002e-05 2024-02-09T17:55:27.116 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.116 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.116 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:55:27.116 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.418519+0000", 2024-02-09T17:55:27.116 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014432141000000001 2024-02-09T17:55:27.116 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.116 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.116 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:55:27.116 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.418565+0000", 2024-02-09T17:55:27.116 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5531999999999997e-05 2024-02-09T17:55:27.116 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:27.117 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:55:27.117 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:27.117 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.117 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.117 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.16440.0:185497 7.0 e750/661 7:038256ed:::benchmark_data_smithi179_135804_object77706:head v 750'11749, mlcod=750'11745)", 2024-02-09T17:55:27.117 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:54:03.399949+0000", 2024-02-09T17:55:27.117 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 83.687844960999996, 2024-02-09T17:55:27.117 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.032876163999999999, 2024-02-09T17:55:27.117 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:55:27.117 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:27.117 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:55:27.118 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.118 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:55:27.118 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.399949+0000", 2024-02-09T17:55:27.118 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:27.118 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.118 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.118 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:55:27.118 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.399949+0000", 2024-02-09T17:55:27.118 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:27.118 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.119 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.119 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:55:27.119 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.399949+0000", 2024-02-09T17:55:27.119 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3900000000000001e-07 2024-02-09T17:55:27.119 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.119 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.119 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:55:27.119 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.399956+0000", 2024-02-09T17:55:27.119 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9229999999999996e-06 2024-02-09T17:55:27.119 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.119 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.120 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:55:27.120 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.399962+0000", 2024-02-09T17:55:27.120 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7089999999999999e-06 2024-02-09T17:55:27.120 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.120 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.120 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:55:27.120 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.399966+0000", 2024-02-09T17:55:27.120 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3270000000000002e-06 2024-02-09T17:55:27.120 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.120 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.120 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:55:27.121 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.407776+0000", 2024-02-09T17:55:27.121 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0078096779999999996 2024-02-09T17:55:27.121 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.121 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.121 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:55:27.121 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.407800+0000", 2024-02-09T17:55:27.121 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4501000000000001e-05 2024-02-09T17:55:27.122 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.122 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.122 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:55:27.122 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.432778+0000", 2024-02-09T17:55:27.122 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.024977961999999999 2024-02-09T17:55:27.122 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.123 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.123 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:55:27.123 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.432825+0000", 2024-02-09T17:55:27.123 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6825e-05 2024-02-09T17:55:27.123 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:27.123 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:55:27.123 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:27.123 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.123 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.123 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.16440.0:185537 7.0 e750/661 7:0149913e:::benchmark_data_smithi179_135804_object77746:head v 750'11750, mlcod=750'11745)", 2024-02-09T17:55:27.123 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:54:03.399972+0000", 2024-02-09T17:55:27.124 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 83.687821498999995, 2024-02-09T17:55:27.124 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.032888150999999997, 2024-02-09T17:55:27.124 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:55:27.124 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:27.124 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:55:27.124 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.124 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:55:27.124 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.399972+0000", 2024-02-09T17:55:27.124 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:27.125 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.125 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.125 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:55:27.125 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.399972+0000", 2024-02-09T17:55:27.125 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:27.125 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.125 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.125 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:55:27.125 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.399972+0000", 2024-02-09T17:55:27.125 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7800000000000001e-07 2024-02-09T17:55:27.125 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.125 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.126 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:55:27.126 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.399975+0000", 2024-02-09T17:55:27.126 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.853e-06 2024-02-09T17:55:27.126 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.126 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.126 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:55:27.126 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.399981+0000", 2024-02-09T17:55:27.126 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.891e-06 2024-02-09T17:55:27.126 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.126 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.126 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:55:27.127 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.399985+0000", 2024-02-09T17:55:27.127 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0450000000000002e-06 2024-02-09T17:55:27.127 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.127 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.127 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:55:27.127 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.410966+0000", 2024-02-09T17:55:27.127 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010981156000000001 2024-02-09T17:55:27.127 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.127 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.127 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:55:27.127 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.410982+0000", 2024-02-09T17:55:27.128 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6124e-05 2024-02-09T17:55:27.128 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.128 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.128 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:55:27.128 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.432828+0000", 2024-02-09T17:55:27.128 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.021845975 2024-02-09T17:55:27.128 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.128 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.128 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:55:27.128 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.432860+0000", 2024-02-09T17:55:27.128 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1928999999999999e-05 2024-02-09T17:55:27.129 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:27.129 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:55:27.129 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:27.129 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.129 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.129 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.16440.0:185544 7.0 e750/661 7:00813485:::benchmark_data_smithi179_135804_object77753:head v 750'11751, mlcod=750'11745)", 2024-02-09T17:55:27.129 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:54:03.399991+0000", 2024-02-09T17:55:27.129 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 83.687802547000004, 2024-02-09T17:55:27.129 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.032906051999999998, 2024-02-09T17:55:27.129 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:55:27.129 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:27.129 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:55:27.130 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.130 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:55:27.130 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.399991+0000", 2024-02-09T17:55:27.130 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:27.130 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.130 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.130 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:55:27.130 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.399991+0000", 2024-02-09T17:55:27.130 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:27.130 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.131 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.131 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:55:27.131 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.399991+0000", 2024-02-09T17:55:27.131 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9500000000000001e-07 2024-02-09T17:55:27.131 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.131 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.131 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:55:27.131 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.399994+0000", 2024-02-09T17:55:27.131 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0750000000000002e-06 2024-02-09T17:55:27.131 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.131 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.131 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:55:27.131 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.399999+0000", 2024-02-09T17:55:27.132 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6990000000000002e-06 2024-02-09T17:55:27.132 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.132 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.132 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:55:27.132 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.400003+0000", 2024-02-09T17:55:27.132 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.749e-06 2024-02-09T17:55:27.132 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.132 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.132 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:55:27.132 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.414393+0000", 2024-02-09T17:55:27.132 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014390719999999999 2024-02-09T17:55:27.133 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.133 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.133 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:55:27.133 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.414410+0000", 2024-02-09T17:55:27.133 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6269999999999998e-05 2024-02-09T17:55:27.133 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.133 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.133 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:55:27.133 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.432865+0000", 2024-02-09T17:55:27.133 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018454958 2024-02-09T17:55:27.133 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.134 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.134 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:55:27.134 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.432897+0000", 2024-02-09T17:55:27.134 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2385999999999998e-05 2024-02-09T17:55:27.134 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:27.134 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:55:27.134 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:27.134 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.134 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.134 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.16440.0:185547 7.0 e750/661 7:0f2c1c56:::benchmark_data_smithi179_135804_object77756:head v 750'11752, mlcod=750'11745)", 2024-02-09T17:55:27.134 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:54:03.400008+0000", 2024-02-09T17:55:27.134 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 83.687785281999993, 2024-02-09T17:55:27.135 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.032913027999999997, 2024-02-09T17:55:27.135 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:55:27.135 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:27.135 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:55:27.135 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.135 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:55:27.135 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.400008+0000", 2024-02-09T17:55:27.135 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:27.135 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.135 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.135 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:55:27.136 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.400008+0000", 2024-02-09T17:55:27.136 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:27.136 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.136 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.136 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:55:27.136 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.400008+0000", 2024-02-09T17:55:27.136 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8e-07 2024-02-09T17:55:27.136 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.136 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.136 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:55:27.136 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.400016+0000", 2024-02-09T17:55:27.137 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5399999999999998e-06 2024-02-09T17:55:27.137 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.137 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.137 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:55:27.137 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.400022+0000", 2024-02-09T17:55:27.137 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0449999999999997e-06 2024-02-09T17:55:27.137 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.137 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.137 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:55:27.137 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.400026+0000", 2024-02-09T17:55:27.137 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0849999999999999e-06 2024-02-09T17:55:27.138 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.138 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.138 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:55:27.138 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.418606+0000", 2024-02-09T17:55:27.138 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018579759000000001 2024-02-09T17:55:27.138 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.138 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.138 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:55:27.139 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.418624+0000", 2024-02-09T17:55:27.139 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7875000000000001e-05 2024-02-09T17:55:27.139 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.139 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.139 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:55:27.139 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.432899+0000", 2024-02-09T17:55:27.139 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014275183 2024-02-09T17:55:27.139 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.139 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.139 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:55:27.139 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.432921+0000", 2024-02-09T17:55:27.139 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2361000000000002e-05 2024-02-09T17:55:27.140 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:27.140 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:55:27.140 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:27.140 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.140 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.140 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.16440.0:185552 7.0 e750/661 7:058536e0:::benchmark_data_smithi179_135804_object77761:head v 750'11753, mlcod=750'11745)", 2024-02-09T17:55:27.140 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:54:03.400033+0000", 2024-02-09T17:55:27.140 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 83.687760967000003, 2024-02-09T17:55:27.140 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.032909176999999998, 2024-02-09T17:55:27.140 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:55:27.140 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:27.141 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:55:27.141 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.141 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:55:27.141 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.400033+0000", 2024-02-09T17:55:27.141 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:27.141 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.141 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.141 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:55:27.141 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.400033+0000", 2024-02-09T17:55:27.141 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:27.141 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.141 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.142 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:55:27.142 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.400033+0000", 2024-02-09T17:55:27.142 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1299999999999999e-07 2024-02-09T17:55:27.142 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.142 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.142 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:55:27.142 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.400036+0000", 2024-02-09T17:55:27.142 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1599999999999998e-06 2024-02-09T17:55:27.142 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.142 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.142 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:55:27.142 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.400041+0000", 2024-02-09T17:55:27.143 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2660000000000001e-06 2024-02-09T17:55:27.143 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.143 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.143 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:55:27.143 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.400045+0000", 2024-02-09T17:55:27.143 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2089999999999999e-06 2024-02-09T17:55:27.143 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.143 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.143 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:55:27.144 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.423525+0000", 2024-02-09T17:55:27.144 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.023479159999999999 2024-02-09T17:55:27.144 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.144 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.144 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:55:27.144 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.423545+0000", 2024-02-09T17:55:27.144 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0047999999999998e-05 2024-02-09T17:55:27.144 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.144 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.144 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:55:27.145 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.432926+0000", 2024-02-09T17:55:27.145 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0093810079999999997 2024-02-09T17:55:27.145 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.145 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.145 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:55:27.145 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.432942+0000", 2024-02-09T17:55:27.145 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6113e-05 2024-02-09T17:55:27.145 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:27.145 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:55:27.145 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:27.145 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.146 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.146 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.16440.0:185568 7.0 e750/661 7:008a058d:::benchmark_data_smithi179_135804_object77777:head v 750'11754, mlcod=750'11745)", 2024-02-09T17:55:27.146 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:54:03.400051+0000", 2024-02-09T17:55:27.146 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 83.687742193999995, 2024-02-09T17:55:27.146 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.032910209000000003, 2024-02-09T17:55:27.146 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:55:27.146 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:27.146 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:55:27.146 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.146 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:55:27.146 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.400051+0000", 2024-02-09T17:55:27.147 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:27.147 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.147 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.147 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:55:27.147 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.400051+0000", 2024-02-09T17:55:27.147 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:27.147 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.147 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.147 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:55:27.147 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.400052+0000", 2024-02-09T17:55:27.148 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9600000000000001e-07 2024-02-09T17:55:27.148 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.148 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.148 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:55:27.148 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.400055+0000", 2024-02-09T17:55:27.148 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3340000000000002e-06 2024-02-09T17:55:27.148 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.148 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.148 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:55:27.148 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.400061+0000", 2024-02-09T17:55:27.148 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6949999999999997e-06 2024-02-09T17:55:27.148 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.149 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.149 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:55:27.149 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.400065+0000", 2024-02-09T17:55:27.149 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0509999999999998e-06 2024-02-09T17:55:27.149 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.149 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.149 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:55:27.149 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.428819+0000", 2024-02-09T17:55:27.149 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.028754122 2024-02-09T17:55:27.149 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.150 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.150 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:55:27.150 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.428841+0000", 2024-02-09T17:55:27.150 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2481999999999999e-05 2024-02-09T17:55:27.150 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.150 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.150 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:55:27.150 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.432944+0000", 2024-02-09T17:55:27.150 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.004102451 2024-02-09T17:55:27.150 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.150 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.150 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:55:27.151 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:03.432962+0000", 2024-02-09T17:55:27.151 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7878000000000001e-05 2024-02-09T17:55:27.151 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:27.151 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:55:27.151 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:27.151 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.151 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.151 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.16440.0:187622 7.0 e751/661 7:06d278b8:::benchmark_data_smithi179_135804_object79831:head v 751'11902, mlcod=751'11898)", 2024-02-09T17:55:27.151 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:54:06.453668+0000", 2024-02-09T17:55:27.151 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 80.634125342999994, 2024-02-09T17:55:27.151 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019485499, 2024-02-09T17:55:27.152 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:55:27.152 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:27.152 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:55:27.152 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.152 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:55:27.152 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:06.453668+0000", 2024-02-09T17:55:27.152 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:27.152 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.152 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.152 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:55:27.153 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:06.453668+0000", 2024-02-09T17:55:27.153 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:27.153 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.153 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.153 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:55:27.153 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:06.453669+0000", 2024-02-09T17:55:27.153 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.04e-06 2024-02-09T17:55:27.153 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.153 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.153 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:55:27.153 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:06.453678+0000", 2024-02-09T17:55:27.154 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8959999999999992e-06 2024-02-09T17:55:27.154 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.154 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.154 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:55:27.154 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:06.453694+0000", 2024-02-09T17:55:27.154 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5752000000000001e-05 2024-02-09T17:55:27.154 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.154 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.154 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:55:27.154 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:06.453703+0000", 2024-02-09T17:55:27.154 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8890000000000008e-06 2024-02-09T17:55:27.155 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.155 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.155 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:55:27.155 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:06.457837+0000", 2024-02-09T17:55:27.155 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041342269999999999 2024-02-09T17:55:27.155 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.155 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.155 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:55:27.155 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:06.457865+0000", 2024-02-09T17:55:27.155 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7809000000000001e-05 2024-02-09T17:55:27.155 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.155 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.156 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:55:27.156 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:06.473106+0000", 2024-02-09T17:55:27.156 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015241562 2024-02-09T17:55:27.156 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.156 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.156 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:55:27.156 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:06.473154+0000", 2024-02-09T17:55:27.156 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7324000000000003e-05 2024-02-09T17:55:27.156 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:27.157 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:55:27.157 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:27.157 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.157 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.157 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.16440.0:191393 7.b e754/661 7:dcf28c3e:::benchmark_data_smithi179_135804_object83602:head v 754'11889, mlcod=754'11886)", 2024-02-09T17:55:27.157 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:54:12.035191+0000", 2024-02-09T17:55:27.157 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 75.052602747999998, 2024-02-09T17:55:27.157 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019891230999999999, 2024-02-09T17:55:27.157 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:55:27.157 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:27.157 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:55:27.158 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.158 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:55:27.158 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.035191+0000", 2024-02-09T17:55:27.158 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:27.158 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.158 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.158 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:55:27.158 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.035191+0000", 2024-02-09T17:55:27.158 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:27.158 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.158 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.158 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:55:27.159 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.035192+0000", 2024-02-09T17:55:27.159 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6899999999999996e-07 2024-02-09T17:55:27.159 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.159 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.159 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:55:27.159 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.035207+0000", 2024-02-09T17:55:27.159 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5126000000000001e-05 2024-02-09T17:55:27.159 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.159 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.159 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:55:27.159 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.035216+0000", 2024-02-09T17:55:27.160 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3719999999999994e-06 2024-02-09T17:55:27.160 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.160 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.160 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:55:27.160 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.035223+0000", 2024-02-09T17:55:27.160 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7490000000000001e-06 2024-02-09T17:55:27.160 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.160 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.160 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:55:27.160 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.035473+0000", 2024-02-09T17:55:27.161 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000250595 2024-02-09T17:55:27.161 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.161 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.161 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:55:27.161 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.035497+0000", 2024-02-09T17:55:27.161 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3417e-05 2024-02-09T17:55:27.161 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.161 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.161 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:55:27.161 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.055022+0000", 2024-02-09T17:55:27.161 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019524989 2024-02-09T17:55:27.162 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.162 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.162 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:55:27.162 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.055082+0000", 2024-02-09T17:55:27.162 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0214000000000003e-05 2024-02-09T17:55:27.162 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:27.162 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:55:27.162 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:27.162 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.162 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.162 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.16440.0:191395 7.b e754/661 7:dcd175db:::benchmark_data_smithi179_135804_object83604:head v 754'11890, mlcod=754'11886)", 2024-02-09T17:55:27.162 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:54:12.035254+0000", 2024-02-09T17:55:27.162 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 75.052539679999995, 2024-02-09T17:55:27.163 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019862050999999999, 2024-02-09T17:55:27.163 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:55:27.163 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:27.163 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:55:27.163 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.163 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:55:27.163 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.035254+0000", 2024-02-09T17:55:27.163 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:27.163 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.163 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.164 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:55:27.164 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.035254+0000", 2024-02-09T17:55:27.164 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:27.164 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.164 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.164 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:55:27.164 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.035254+0000", 2024-02-09T17:55:27.164 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.58e-07 2024-02-09T17:55:27.164 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.164 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.164 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:55:27.164 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.035262+0000", 2024-02-09T17:55:27.165 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0979999999999994e-06 2024-02-09T17:55:27.165 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.165 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.165 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:55:27.165 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.035270+0000", 2024-02-09T17:55:27.165 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.255e-06 2024-02-09T17:55:27.165 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.165 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.165 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:55:27.165 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.035274+0000", 2024-02-09T17:55:27.166 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7690000000000004e-06 2024-02-09T17:55:27.166 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.166 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.166 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:55:27.166 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.040545+0000", 2024-02-09T17:55:27.166 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0052701359999999999 2024-02-09T17:55:27.166 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.166 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.166 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:55:27.166 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.040571+0000", 2024-02-09T17:55:27.166 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.614e-05 2024-02-09T17:55:27.167 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.167 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.168 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:55:27.168 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.055090+0000", 2024-02-09T17:55:27.168 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014519155000000001 2024-02-09T17:55:27.168 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.168 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.168 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:55:27.168 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.055116+0000", 2024-02-09T17:55:27.168 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.614e-05 2024-02-09T17:55:27.169 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:27.169 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:55:27.169 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:27.169 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.169 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.169 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.16440.0:191414 7.b e754/661 7:d3e59cd7:::benchmark_data_smithi179_135804_object83623:head v 754'11891, mlcod=754'11886)", 2024-02-09T17:55:27.169 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:54:12.035282+0000", 2024-02-09T17:55:27.169 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 75.052512093999994, 2024-02-09T17:55:27.169 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.024151573999999999, 2024-02-09T17:55:27.169 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:55:27.170 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:27.170 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:55:27.170 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.170 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:55:27.170 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.035282+0000", 2024-02-09T17:55:27.170 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:27.170 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.170 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.170 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:55:27.170 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.035282+0000", 2024-02-09T17:55:27.170 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:27.171 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.171 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.171 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:55:27.171 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.035282+0000", 2024-02-09T17:55:27.171 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.11e-07 2024-02-09T17:55:27.171 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.171 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.171 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:55:27.171 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.035285+0000", 2024-02-09T17:55:27.171 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0900000000000001e-06 2024-02-09T17:55:27.171 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.172 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.172 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:55:27.172 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.035290+0000", 2024-02-09T17:55:27.172 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.677e-06 2024-02-09T17:55:27.172 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.172 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.172 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:55:27.172 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.035294+0000", 2024-02-09T17:55:27.173 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0090000000000001e-06 2024-02-09T17:55:27.173 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.173 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.173 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:55:27.173 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.045573+0000", 2024-02-09T17:55:27.173 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010278785 2024-02-09T17:55:27.175 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.175 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.175 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:55:27.175 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.045595+0000", 2024-02-09T17:55:27.175 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2167999999999998e-05 2024-02-09T17:55:27.175 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.175 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.175 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:55:27.175 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.059382+0000", 2024-02-09T17:55:27.175 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013786402999999999 2024-02-09T17:55:27.176 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.176 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.176 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:55:27.176 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.059433+0000", 2024-02-09T17:55:27.176 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1230999999999998e-05 2024-02-09T17:55:27.176 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:27.176 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:55:27.176 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:27.176 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.176 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.176 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.16440.0:191416 7.b e755/661 7:dc14956f:::benchmark_data_smithi179_135804_object83625:head v 755'11892, mlcod=754'11886)", 2024-02-09T17:55:27.177 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:54:12.035308+0000", 2024-02-09T17:55:27.177 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 75.052485301999994, 2024-02-09T17:55:27.177 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.032752260999999998, 2024-02-09T17:55:27.177 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:55:27.177 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:27.177 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:55:27.177 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.177 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:55:27.177 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.035308+0000", 2024-02-09T17:55:27.177 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:27.177 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.177 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.177 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:55:27.178 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.035308+0000", 2024-02-09T17:55:27.178 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:27.178 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.178 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.178 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:55:27.178 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.035308+0000", 2024-02-09T17:55:27.178 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8799999999999999e-07 2024-02-09T17:55:27.178 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.178 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.178 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:55:27.178 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.035311+0000", 2024-02-09T17:55:27.178 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0130000000000002e-06 2024-02-09T17:55:27.179 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.179 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.179 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:55:27.179 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.035319+0000", 2024-02-09T17:55:27.179 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7479999999999993e-06 2024-02-09T17:55:27.179 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.179 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.179 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:55:27.179 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.035325+0000", 2024-02-09T17:55:27.179 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4269999999999999e-06 2024-02-09T17:55:27.179 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.179 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.179 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:55:27.180 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.050532+0000", 2024-02-09T17:55:27.180 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015207669 2024-02-09T17:55:27.180 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.180 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.180 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:55:27.180 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.050554+0000", 2024-02-09T17:55:27.180 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2150999999999998e-05 2024-02-09T17:55:27.180 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.180 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.180 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:55:27.180 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.068010+0000", 2024-02-09T17:55:27.180 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.017455392 2024-02-09T17:55:27.181 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.181 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.181 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:55:27.181 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.068061+0000", 2024-02-09T17:55:27.181 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0673000000000001e-05 2024-02-09T17:55:27.181 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:27.181 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:55:27.181 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:27.181 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.181 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.181 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.16440.0:191420 7.b e755/661 7:dbba14c3:::benchmark_data_smithi179_135804_object83629:head v 755'11893, mlcod=754'11886)", 2024-02-09T17:55:27.181 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:54:12.035330+0000", 2024-02-09T17:55:27.181 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 75.052463373999998, 2024-02-09T17:55:27.182 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.032762160999999998, 2024-02-09T17:55:27.182 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:55:27.182 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:27.182 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:55:27.182 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.182 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:55:27.182 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.035330+0000", 2024-02-09T17:55:27.182 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:27.182 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.182 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.182 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:55:27.182 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.035330+0000", 2024-02-09T17:55:27.182 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:27.183 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.183 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.183 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:55:27.183 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.035330+0000", 2024-02-09T17:55:27.183 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.22e-07 2024-02-09T17:55:27.183 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.183 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.183 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:55:27.183 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.035337+0000", 2024-02-09T17:55:27.183 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8050000000000001e-06 2024-02-09T17:55:27.183 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.183 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.183 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:55:27.184 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.035342+0000", 2024-02-09T17:55:27.184 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2109999999999997e-06 2024-02-09T17:55:27.184 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.184 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.184 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:55:27.184 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.035346+0000", 2024-02-09T17:55:27.184 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8850000000000001e-06 2024-02-09T17:55:27.184 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.184 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.184 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:55:27.184 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.055156+0000", 2024-02-09T17:55:27.184 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019809567 2024-02-09T17:55:27.185 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.185 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.185 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:55:27.185 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.055178+0000", 2024-02-09T17:55:27.185 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1724000000000001e-05 2024-02-09T17:55:27.185 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.185 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.185 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:55:27.185 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.068066+0000", 2024-02-09T17:55:27.185 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012887899 2024-02-09T17:55:27.185 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.186 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.186 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:55:27.186 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.068092+0000", 2024-02-09T17:55:27.186 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6848000000000001e-05 2024-02-09T17:55:27.186 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:27.186 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:55:27.186 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:27.186 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.186 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.186 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.16440.0:191438 7.b e755/661 7:dceb10b9:::benchmark_data_smithi179_135804_object83647:head v 755'11894, mlcod=754'11886)", 2024-02-09T17:55:27.186 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:54:12.035352+0000", 2024-02-09T17:55:27.187 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 75.052441646999995, 2024-02-09T17:55:27.188 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.032765435000000002, 2024-02-09T17:55:27.188 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:55:27.188 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:27.188 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:55:27.188 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.188 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:55:27.188 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.035352+0000", 2024-02-09T17:55:27.188 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:27.188 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.188 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.188 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:55:27.188 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.035352+0000", 2024-02-09T17:55:27.189 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:27.189 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.189 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.189 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:55:27.189 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.035352+0000", 2024-02-09T17:55:27.189 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8900000000000001e-07 2024-02-09T17:55:27.189 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.189 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.189 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:55:27.189 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.035355+0000", 2024-02-09T17:55:27.189 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7549999999999999e-06 2024-02-09T17:55:27.189 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.189 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.190 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:55:27.190 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.035359+0000", 2024-02-09T17:55:27.190 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5210000000000004e-06 2024-02-09T17:55:27.190 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.190 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.190 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:55:27.190 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.035363+0000", 2024-02-09T17:55:27.190 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5980000000000001e-06 2024-02-09T17:55:27.190 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.190 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.190 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:55:27.190 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.059485+0000", 2024-02-09T17:55:27.190 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.024122029 2024-02-09T17:55:27.191 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.191 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.191 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:55:27.191 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.059509+0000", 2024-02-09T17:55:27.191 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3952e-05 2024-02-09T17:55:27.191 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.191 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.191 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:55:27.191 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.068095+0000", 2024-02-09T17:55:27.191 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0085856309999999998 2024-02-09T17:55:27.191 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.191 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.192 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:55:27.192 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.068117+0000", 2024-02-09T17:55:27.192 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2759999999999999e-05 2024-02-09T17:55:27.192 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:27.192 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:55:27.192 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:27.192 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.192 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.192 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.16440.0:191465 7.b e755/661 7:d12f454d:::benchmark_data_smithi179_135804_object83674:head v 755'11895, mlcod=754'11886)", 2024-02-09T17:55:27.192 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:54:12.035368+0000", 2024-02-09T17:55:27.192 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 75.052426021000002, 2024-02-09T17:55:27.192 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.032824177000000003, 2024-02-09T17:55:27.193 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:55:27.193 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:27.193 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:55:27.193 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.193 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:55:27.193 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.035368+0000", 2024-02-09T17:55:27.193 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:27.193 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.193 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.193 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:55:27.193 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.035368+0000", 2024-02-09T17:55:27.193 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:27.193 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.194 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.194 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:55:27.194 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.035368+0000", 2024-02-09T17:55:27.194 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7700000000000001e-07 2024-02-09T17:55:27.194 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.194 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.194 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:55:27.194 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.035372+0000", 2024-02-09T17:55:27.194 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7319999999999998e-06 2024-02-09T17:55:27.194 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.194 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.194 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:55:27.194 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.035377+0000", 2024-02-09T17:55:27.195 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6870000000000002e-06 2024-02-09T17:55:27.195 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.195 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.195 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:55:27.195 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.035381+0000", 2024-02-09T17:55:27.195 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7469999999999999e-06 2024-02-09T17:55:27.195 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.195 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.195 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:55:27.195 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.063826+0000", 2024-02-09T17:55:27.195 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.028444911999999999 2024-02-09T17:55:27.195 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.196 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.196 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:55:27.196 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.063849+0000", 2024-02-09T17:55:27.196 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2883000000000001e-05 2024-02-09T17:55:27.196 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.196 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.196 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:55:27.196 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.068145+0000", 2024-02-09T17:55:27.196 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042965720000000002 2024-02-09T17:55:27.196 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.196 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.196 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:55:27.196 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.068192+0000", 2024-02-09T17:55:27.197 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6467000000000001e-05 2024-02-09T17:55:27.197 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:27.197 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:55:27.197 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:27.197 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.197 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.197 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "rep_scrubmap(1.18 e525 from shard 7)", 2024-02-09T17:55:27.197 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:54:50.032277+0000", 2024-02-09T17:55:27.197 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 37.055517080000001, 2024-02-09T17:55:27.197 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020736905, 2024-02-09T17:55:27.197 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:55:27.197 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-09T17:55:27.198 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:55:27.198 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.198 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:55:27.198 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:50.032277+0000", 2024-02-09T17:55:27.198 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:27.198 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.198 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.198 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:55:27.198 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:50.032277+0000", 2024-02-09T17:55:27.198 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:27.198 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.198 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.199 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:55:27.199 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:50.032277+0000", 2024-02-09T17:55:27.199 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9700000000000005e-07 2024-02-09T17:55:27.199 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.199 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.199 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:55:27.199 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:50.032283+0000", 2024-02-09T17:55:27.199 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0190000000000003e-06 2024-02-09T17:55:27.199 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.199 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.199 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:55:27.199 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:50.032296+0000", 2024-02-09T17:55:27.199 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2619999999999999e-05 2024-02-09T17:55:27.199 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.200 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.200 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:55:27.200 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:50.032305+0000", 2024-02-09T17:55:27.200 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7590000000000003e-06 2024-02-09T17:55:27.200 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.200 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.200 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:55:27.200 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:50.052988+0000", 2024-02-09T17:55:27.200 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.02068294 2024-02-09T17:55:27.200 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.200 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.200 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:55:27.201 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:50.052996+0000", 2024-02-09T17:55:27.201 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4780000000000008e-06 2024-02-09T17:55:27.201 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.201 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.201 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:55:27.201 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:50.053013+0000", 2024-02-09T17:55:27.201 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7192000000000001e-05 2024-02-09T17:55:27.201 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:27.201 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:55:27.201 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:27.201 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.201 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.201 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "rep_scrubmap(1.18 e525 from shard 2)", 2024-02-09T17:55:27.202 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:54:50.032920+0000", 2024-02-09T17:55:27.202 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 37.054873895, 2024-02-09T17:55:27.202 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020135506000000001, 2024-02-09T17:55:27.202 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:55:27.202 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-09T17:55:27.202 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:55:27.202 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.202 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:55:27.202 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:50.032920+0000", 2024-02-09T17:55:27.202 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:27.202 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.202 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.202 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:55:27.203 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:50.032920+0000", 2024-02-09T17:55:27.203 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:27.203 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.203 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.203 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:55:27.203 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:50.032920+0000", 2024-02-09T17:55:27.203 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1600000000000001e-07 2024-02-09T17:55:27.203 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.203 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.203 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:55:27.203 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:50.032925+0000", 2024-02-09T17:55:27.203 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6029999999999998e-06 2024-02-09T17:55:27.204 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.204 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.204 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:55:27.204 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:50.032935+0000", 2024-02-09T17:55:27.204 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0108999999999999e-05 2024-02-09T17:55:27.204 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.204 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.204 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:55:27.204 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:50.032942+0000", 2024-02-09T17:55:27.204 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5470000000000002e-06 2024-02-09T17:55:27.204 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.204 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.204 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:55:27.205 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:50.053038+0000", 2024-02-09T17:55:27.205 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020096342999999999 2024-02-09T17:55:27.205 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.205 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.205 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:55:27.205 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:50.053044+0000", 2024-02-09T17:55:27.205 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6389999999999997e-06 2024-02-09T17:55:27.205 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:27.205 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:27.205 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:55:27.205 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:50.053055+0000", 2024-02-09T17:55:27.206 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1549000000000001e-05 2024-02-09T17:55:27.206 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:27.206 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:55:27.206 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:27.206 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:27.207 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:55:27.207 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:55:27.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:27.207 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:55:27.208 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:55:27.308 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:55:27.309 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.2 to 1 (10000) 2024-02-09T17:55:27.335 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:27.353 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:55:27.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:27.356 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:55:27.457 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:55:27.501 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:27.532 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:55:27.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:27.536 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:55:27.586 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 1 16 2250 2234 139.618 139.625 0.00345782 0.00714139 2024-02-09T17:55:27.638 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:55:27.670 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:27.822 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:55:27.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:27.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:55:27.840 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-09T17:55:27.927 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:55:27.964 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:27.987 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:55:27.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:27.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:55:28.092 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:55:28.121 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:27 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:55:28.121 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:27 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:55:28.121 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:27 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1870894321' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-09T17:55:28.121 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:27 smithi162 ceph-mon[44304]: pgmap v1955: 47 pgs: 47 active+clean; 6.3 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 979 KiB/s wr, 15 op/s 2024-02-09T17:55:28.121 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:27 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1870894321' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-09T17:55:28.121 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:27 smithi162 ceph-mon[44304]: osdmap e795: 12 total, 12 up, 12 in 2024-02-09T17:55:28.124 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:28.133 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-02-09T17:55:28.133 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph pg dump --format=json 2024-02-09T17:55:28.148 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:55:28.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:28.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:55:28.168 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:27 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:55:28.168 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:27 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:55:28.168 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:27 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1870894321' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-09T17:55:28.168 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:27 smithi099 ceph-mon[39420]: pgmap v1955: 47 pgs: 47 active+clean; 6.3 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 979 KiB/s wr, 15 op/s 2024-02-09T17:55:28.169 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:27 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1870894321' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-09T17:55:28.169 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:27 smithi099 ceph-mon[39420]: osdmap e795: 12 total, 12 up, 12 in 2024-02-09T17:55:28.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:27 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:55:28.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:27 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:55:28.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:27 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1870894321' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-09T17:55:28.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:27 smithi179 ceph-mon[43975]: pgmap v1955: 47 pgs: 47 active+clean; 6.3 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 979 KiB/s wr, 15 op/s 2024-02-09T17:55:28.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:27 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1870894321' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-09T17:55:28.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:27 smithi179 ceph-mon[43975]: osdmap e795: 12 total, 12 up, 12 in 2024-02-09T17:55:28.219 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:55:28.253 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:55:28.295 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:28.326 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:55:28.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:28.331 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:55:28.432 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:55:28.483 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:28.515 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:55:28.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:28.520 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:55:28.587 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 2 16 5684 5668 177.112 214.625 0.00393805 0.0056365 2024-02-09T17:55:28.621 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:55:28.623 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:55:28.653 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:28.708 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:55:28.709 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:55:28.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:28.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:55:28.737 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:55:28.766 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:55:28.797 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:55:28.813 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:55:28.828 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:55:28.853 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:28.855 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:55:28.881 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:55:28.881 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:55:28.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:28.885 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:55:28.912 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:55:28.945 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:55:28.969 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:55:28.986 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:55:28.991 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:55:29.015 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:55:29.016 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:55:29.016 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:55:29.016 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:55:29.016 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:55:29.016 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:55:29.017 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:55:29.017 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:55:29.017 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:55:29.017 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:55:29.017 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:55:29.017 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:55:29.017 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:55:29.017 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:29.138 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:28 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:55:29.138 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:28 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:55:29.138 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:28 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:55:29.138 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:28 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:55:29.138 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:28 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1602763090' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:55:29.169 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:28 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:55:29.169 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:28 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:55:29.169 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:28 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:55:29.169 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:28 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:55:29.169 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:28 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1602763090' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:55:29.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:28 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:55:29.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:28 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:55:29.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:28 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:55:29.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:28 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:55:29.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:28 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1602763090' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:55:29.467 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:55:29.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:29.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:55:29.573 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:55:29.586 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 3 16 8650 8634 179.861 185.375 0.00478556 0.00555036 2024-02-09T17:55:29.606 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:29.631 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:55:29.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:29.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:55:29.735 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:55:29.768 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:29.924 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:55:29.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:29.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:55:30.030 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:55:30.068 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:30.180 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:55:30.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:30.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:55:30.184 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:29 smithi099 ceph-mon[39420]: osdmap e796: 12 total, 12 up, 12 in 2024-02-09T17:55:30.185 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:29 smithi099 ceph-mon[39420]: pgmap v1958: 47 pgs: 47 active+clean; 141 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s wr, 378 op/s 2024-02-09T17:55:30.244 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:55:30.244 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:55:30.244 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:55:30.268 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:29 smithi162 ceph-mon[44304]: osdmap e796: 12 total, 12 up, 12 in 2024-02-09T17:55:30.268 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:29 smithi162 ceph-mon[44304]: pgmap v1958: 47 pgs: 47 active+clean; 141 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s wr, 378 op/s 2024-02-09T17:55:30.285 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:55:30.323 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:30.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:29 smithi179 ceph-mon[43975]: osdmap e796: 12 total, 12 up, 12 in 2024-02-09T17:55:30.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:29 smithi179 ceph-mon[43975]: pgmap v1958: 47 pgs: 47 active+clean; 141 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s wr, 378 op/s 2024-02-09T17:55:30.525 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:55:30.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:30.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:55:30.587 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 4 16 11858 11842 185.017 200.5 0.00223023 0.00539728 2024-02-09T17:55:30.629 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:55:30.661 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:30.973 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-09T17:55:31.014 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:55:31.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:31.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:55:31.118 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:55:31.166 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:31.195 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:55:31.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:31.198 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:55:31.300 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:55:31.342 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:55:31.347 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:31.473 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:55:31.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:31.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:55:31.578 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:55:31.587 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 5 16 14778 14762 184.51 182.5 0.00913357 0.00541286 2024-02-09T17:55:31.616 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:31.638 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:55:31.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:31.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:55:31.742 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:55:31.769 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:31.787 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:55:31.789 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:31.789 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:55:31.890 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:55:31.916 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is set 2024-02-09T17:55:31.936 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:32.225 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:55:32.398 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:55:32.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:32.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:55:32.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:32 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:55:32.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:32 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:55:32.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:32 smithi162 ceph-mon[44304]: pgmap v1959: 47 pgs: 47 active+clean; 228 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s wr, 523 op/s 2024-02-09T17:55:32.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:32 smithi162 ceph-mon[44304]: osdmap e797: 12 total, 12 up, 12 in 2024-02-09T17:55:32.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:32 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1764887665' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:55:32.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:32 smithi162 ceph-mon[44304]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:55:32.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:32 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1764887665' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:55:32.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:32 smithi162 ceph-mon[44304]: osdmap e798: 12 total, 12 up, 12 in 2024-02-09T17:55:32.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:32 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:55:32.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:32 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:55:32.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:32 smithi099 ceph-mon[39420]: pgmap v1959: 47 pgs: 47 active+clean; 228 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s wr, 523 op/s 2024-02-09T17:55:32.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:32 smithi099 ceph-mon[39420]: osdmap e797: 12 total, 12 up, 12 in 2024-02-09T17:55:32.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:32 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1764887665' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:55:32.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:32 smithi099 ceph-mon[39420]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:55:32.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:32 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1764887665' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:55:32.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:32 smithi099 ceph-mon[39420]: osdmap e798: 12 total, 12 up, 12 in 2024-02-09T17:55:32.502 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:55:32.548 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:32.587 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 6 16 17120 17104 178.152 146.375 0.00655909 0.00560714 2024-02-09T17:55:32.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:32 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:55:32.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:32 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:55:32.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:32 smithi179 ceph-mon[43975]: pgmap v1959: 47 pgs: 47 active+clean; 228 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s wr, 523 op/s 2024-02-09T17:55:32.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:32 smithi179 ceph-mon[43975]: osdmap e797: 12 total, 12 up, 12 in 2024-02-09T17:55:32.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:32 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1764887665' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:55:32.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:32 smithi179 ceph-mon[43975]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:55:32.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:32 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1764887665' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:55:32.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:32 smithi179 ceph-mon[43975]: osdmap e798: 12 total, 12 up, 12 in 2024-02-09T17:55:33.587 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 7 16 19883 19867 177.37 172.688 0.00214482 0.00563094 2024-02-09T17:55:33.597 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:55:33.597 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:55:34.006 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:55:34.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:34.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:55:34.114 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:55:34.154 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:34.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:34 smithi099 ceph-mon[39420]: pgmap v1962: 47 pgs: 47 active+clean; 790 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 132 MiB/s wr, 2.12k op/s 2024-02-09T17:55:34.408 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:55:34.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:34.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:55:34.420 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:34 smithi162 ceph-mon[44304]: pgmap v1962: 47 pgs: 47 active+clean; 790 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 132 MiB/s wr, 2.12k op/s 2024-02-09T17:55:34.446 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-09T17:55:34.512 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:55:34.560 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:34 smithi179 ceph-mon[43975]: pgmap v1962: 47 pgs: 47 active+clean; 790 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 132 MiB/s wr, 2.12k op/s 2024-02-09T17:55:34.564 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:34.587 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 8 16 22024 22008 171.924 133.812 0.00672258 0.00581225 2024-02-09T17:55:34.597 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:55:34.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:34.602 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:55:34.679 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:55:34.703 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:55:34.752 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:34.765 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:55:34.785 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:55:34.789 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:34.789 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:55:34.890 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:55:34.924 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:35.028 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:55:35.031 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:35.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:55:35.109 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:55:35.111 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-02-09T17:55:35.131 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:55:35.165 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:35.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:35 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:55:35.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:35 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:55:35.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:35 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:55:35.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:35 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:55:35.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:35 smithi099 ceph-mon[39420]: osdmap e799: 12 total, 12 up, 12 in 2024-02-09T17:55:35.388 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset noscrub 2024-02-09T17:55:35.587 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 9 16 24487 24471 169.923 153.938 0.00631366 0.00587984 2024-02-09T17:55:35.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:35 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:55:35.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:35 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:55:35.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:35 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:55:35.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:35 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:55:35.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:35 smithi179 ceph-mon[43975]: osdmap e799: 12 total, 12 up, 12 in 2024-02-09T17:55:35.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:35 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:55:35.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:35 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:55:35.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:35 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:55:35.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:35 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:55:35.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:35 smithi162 ceph-mon[44304]: osdmap e799: 12 total, 12 up, 12 in 2024-02-09T17:55:36.019 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:55:36.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:36.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:55:36.125 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:55:36.155 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:36.331 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:36 smithi162 ceph-mon[44304]: from='client.26285 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:55:36.331 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:36 smithi162 ceph-mon[44304]: pgmap v1964: 47 pgs: 47 active+clean; 1016 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 146 MiB/s wr, 2.33k op/s 2024-02-09T17:55:36.357 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:55:36.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:36.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:55:36.462 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:55:36.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:36 smithi099 ceph-mon[39420]: from='client.26285 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:55:36.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:36 smithi099 ceph-mon[39420]: pgmap v1964: 47 pgs: 47 active+clean; 1016 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 146 MiB/s wr, 2.33k op/s 2024-02-09T17:55:36.502 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:36 smithi179 ceph-mon[43975]: from='client.26285 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:55:36.502 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:36 smithi179 ceph-mon[43975]: pgmap v1964: 47 pgs: 47 active+clean; 1016 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 146 MiB/s wr, 2.33k op/s 2024-02-09T17:55:36.505 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:36.537 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:55:36.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:36.542 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:55:36.587 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 10 16 26891 26875 167.955 150.25 0.0060012 0.00594892 2024-02-09T17:55:36.643 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:55:36.682 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:36.685 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:55:36.685 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:55:37.587 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 11 16 29217 29201 165.901 145.375 0.00690283 0.00602299 2024-02-09T17:55:38.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:37 smithi179 ceph-mon[43975]: osdmap e800: 12 total, 12 up, 12 in 2024-02-09T17:55:38.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:37 smithi179 ceph-mon[43975]: pgmap v1966: 47 pgs: 47 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 195 MiB/s wr, 3.13k op/s 2024-02-09T17:55:38.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:37 smithi162 ceph-mon[44304]: osdmap e800: 12 total, 12 up, 12 in 2024-02-09T17:55:38.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:37 smithi162 ceph-mon[44304]: pgmap v1966: 47 pgs: 47 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 195 MiB/s wr, 3.13k op/s 2024-02-09T17:55:38.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:37 smithi099 ceph-mon[39420]: osdmap e800: 12 total, 12 up, 12 in 2024-02-09T17:55:38.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:37 smithi099 ceph-mon[39420]: pgmap v1966: 47 pgs: 47 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 195 MiB/s wr, 3.13k op/s 2024-02-09T17:55:38.587 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 12 16 31292 31276 162.882 129.688 0.00207127 0.00607697 2024-02-09T17:55:39.587 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 13 16 33243 33227 159.731 121.938 0.00415734 0.00625564 2024-02-09T17:55:40.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:40 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:55:40.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:40 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:55:40.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:40 smithi179 ceph-mon[43975]: osdmap e801: 12 total, 12 up, 12 in 2024-02-09T17:55:40.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:40 smithi179 ceph-mon[43975]: pgmap v1968: 47 pgs: 47 active+clean; 1.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 163 MiB/s wr, 2.61k op/s 2024-02-09T17:55:40.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:40 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:55:40.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:40 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:55:40.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:40 smithi162 ceph-mon[44304]: osdmap e801: 12 total, 12 up, 12 in 2024-02-09T17:55:40.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:40 smithi162 ceph-mon[44304]: pgmap v1968: 47 pgs: 47 active+clean; 1.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 163 MiB/s wr, 2.61k op/s 2024-02-09T17:55:40.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:40 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:55:40.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:40 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:55:40.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:40 smithi099 ceph-mon[39420]: osdmap e801: 12 total, 12 up, 12 in 2024-02-09T17:55:40.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:40 smithi099 ceph-mon[39420]: pgmap v1968: 47 pgs: 47 active+clean; 1.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 163 MiB/s wr, 2.61k op/s 2024-02-09T17:55:40.588 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 14 16 35243 35227 157.249 125 0.00384964 0.00635399 2024-02-09T17:55:41.061 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-09T17:55:41.390 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:55:41.481 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":1963,"stamp":"2024-02-09T17:55:34.203999+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1064829472,"num_objects":16241,"num_object_clones":0,"num_object_copies":48723,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16239,"num_write_kb":1039296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20548,"ondisk_log_size":20548,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":145,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":11069948,"kb_used_data":2721060,"kb_used_omap":229,"kb_used_meta":8348506,"kb_avail":1113871876,"statfs":{"total":1151940427776,"available":1140604801024,"internally_reserved":0,"allocated":2786365440,"data_stored":2781257392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235127,"internal_metadata":8548870537},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":916717568,"num_objects":13988,"num_object_clones":0,"num_object_copies":41964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13988,"num_write_kb":895232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.280226"},"pg_stats":[{"pgid":"1.1e","version":"63'139","reported_seq":888,"reported_epoch":797,"state":"active+clean","last_fresh":"2024-02-09T17:55:31.190730+0000","last_change":"2024-02-09T17:55:17.114116+0000","last_active":"2024-02-09T17:55:31.190730+0000","last_peered":"2024-02-09T17:55:31.190730+0000","last_clean":"2024-02-09T17:55:31.190730+0000","last_became_active":"2024-02-09T17:55:08.938117+0000","last_became_peered":"2024-02-09T17:55:08.938117+0000","last_unstale":"2024-02-09T17:55:31.190730+0000","last_undegraded":"2024-02-09T17:55:31.190730+0000","last_fullsized":"2024-02-09T17:55:31.190730+0000","mapping_epoch":781,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":782,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:17.113960+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:55:17.113960+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:31.972827+0000","scrub_duration":125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,7],"acting":[11,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"63'139","reported_seq":898,"reported_epoch":796,"state":"active+clean","last_fresh":"2024-02-09T17:55:28.926244+0000","last_change":"2024-02-09T17:54:59.163874+0000","last_active":"2024-02-09T17:55:28.926244+0000","last_peered":"2024-02-09T17:55:28.926244+0000","last_clean":"2024-02-09T17:55:28.926244+0000","last_became_active":"2024-02-09T17:43:28.594296+0000","last_became_peered":"2024-02-09T17:43:28.594296+0000","last_unstale":"2024-02-09T17:55:28.926244+0000","last_undegraded":"2024-02-09T17:55:28.926244+0000","last_fullsized":"2024-02-09T17:55:28.926244+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:54:59.163739+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:48:24.634092+0000","last_clean_scrub_stamp":"2024-02-09T17:54:59.163739+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:26.933451+0000","scrub_duration":123,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"63'139","reported_seq":1414,"reported_epoch":798,"state":"active+clean","last_fresh":"2024-02-09T17:55:31.916099+0000","last_change":"2024-02-09T17:53:47.835143+0000","last_active":"2024-02-09T17:55:31.916099+0000","last_peered":"2024-02-09T17:55:31.916099+0000","last_clean":"2024-02-09T17:55:31.916099+0000","last_became_active":"2024-02-09T17:43:19.103584+0000","last_became_peered":"2024-02-09T17:43:19.103584+0000","last_unstale":"2024-02-09T17:55:31.916099+0000","last_undegraded":"2024-02-09T17:55:31.916099+0000","last_fullsized":"2024-02-09T17:55:31.916099+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:53:47.835005+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:23.912504+0000","last_clean_scrub_stamp":"2024-02-09T17:53:47.835005+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:55:35.298325+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"63'139","reported_seq":1220,"reported_epoch":798,"state":"active+clean","last_fresh":"2024-02-09T17:55:31.915628+0000","last_change":"2024-02-09T17:55:11.475048+0000","last_active":"2024-02-09T17:55:31.915628+0000","last_peered":"2024-02-09T17:55:31.915628+0000","last_clean":"2024-02-09T17:55:31.915628+0000","last_became_active":"2024-02-09T17:43:19.102729+0000","last_became_peered":"2024-02-09T17:43:19.102729+0000","last_unstale":"2024-02-09T17:55:31.915628+0000","last_undegraded":"2024-02-09T17:55:31.915628+0000","last_fullsized":"2024-02-09T17:55:31.915628+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:11.474947+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:45:44.988633+0000","last_clean_scrub_stamp":"2024-02-09T17:55:11.474947+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:38.756171+0000","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"63'139","reported_seq":873,"reported_epoch":797,"state":"active+clean","last_fresh":"2024-02-09T17:55:31.190785+0000","last_change":"2024-02-09T17:55:03.984081+0000","last_active":"2024-02-09T17:55:31.190785+0000","last_peered":"2024-02-09T17:55:31.190785+0000","last_clean":"2024-02-09T17:55:31.190785+0000","last_became_active":"2024-02-09T17:43:19.120814+0000","last_became_peered":"2024-02-09T17:43:19.120814+0000","last_unstale":"2024-02-09T17:55:31.190785+0000","last_undegraded":"2024-02-09T17:55:31.190785+0000","last_fullsized":"2024-02-09T17:55:31.190785+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:03.983998+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:55:03.983998+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:29.536408+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"63'139","reported_seq":1729,"reported_epoch":798,"state":"active+clean","last_fresh":"2024-02-09T17:55:31.919653+0000","last_change":"2024-02-09T17:55:21.231646+0000","last_active":"2024-02-09T17:55:31.919653+0000","last_peered":"2024-02-09T17:55:31.919653+0000","last_clean":"2024-02-09T17:55:31.919653+0000","last_became_active":"2024-02-09T17:55:21.231291+0000","last_became_peered":"2024-02-09T17:55:21.231291+0000","last_unstale":"2024-02-09T17:55:31.919653+0000","last_undegraded":"2024-02-09T17:55:31.919653+0000","last_fullsized":"2024-02-09T17:55:31.919653+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":791,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:15.085677+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:30.193194+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.085677+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:27.587337+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"63'139","reported_seq":1563,"reported_epoch":796,"state":"active+clean","last_fresh":"2024-02-09T17:55:28.926483+0000","last_change":"2024-02-09T17:54:50.053517+0000","last_active":"2024-02-09T17:55:28.926483+0000","last_peered":"2024-02-09T17:55:28.926483+0000","last_clean":"2024-02-09T17:55:28.926483+0000","last_became_active":"2024-02-09T17:43:28.585573+0000","last_became_peered":"2024-02-09T17:43:28.585573+0000","last_unstale":"2024-02-09T17:55:28.926483+0000","last_undegraded":"2024-02-09T17:55:28.926483+0000","last_fullsized":"2024-02-09T17:55:28.926483+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:54:50.053414+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:54:50.053414+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:13.442075+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"63'139","reported_seq":1013,"reported_epoch":798,"state":"active+clean","last_fresh":"2024-02-09T17:55:31.917248+0000","last_change":"2024-02-09T17:55:04.470767+0000","last_active":"2024-02-09T17:55:31.917248+0000","last_peered":"2024-02-09T17:55:31.917248+0000","last_clean":"2024-02-09T17:55:31.917248+0000","last_became_active":"2024-02-09T17:48:46.184419+0000","last_became_peered":"2024-02-09T17:48:46.184419+0000","last_unstale":"2024-02-09T17:55:31.917248+0000","last_undegraded":"2024-02-09T17:55:31.917248+0000","last_fullsized":"2024-02-09T17:55:31.917248+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:04.470661+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:44:50.214686+0000","last_clean_scrub_stamp":"2024-02-09T17:55:04.470661+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:29.156638+0000","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"63'139","reported_seq":900,"reported_epoch":796,"state":"active+clean","last_fresh":"2024-02-09T17:55:28.926102+0000","last_change":"2024-02-09T17:54:56.134959+0000","last_active":"2024-02-09T17:55:28.926102+0000","last_peered":"2024-02-09T17:55:28.926102+0000","last_clean":"2024-02-09T17:55:28.926102+0000","last_became_active":"2024-02-09T17:53:02.869767+0000","last_became_peered":"2024-02-09T17:53:02.869767+0000","last_unstale":"2024-02-09T17:55:28.926102+0000","last_undegraded":"2024-02-09T17:55:28.926102+0000","last_fullsized":"2024-02-09T17:55:28.926102+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:54:56.134826+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:11.970338+0000","last_clean_scrub_stamp":"2024-02-09T17:54:56.134826+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:19.204901+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"63'139","reported_seq":1315,"reported_epoch":797,"state":"active+clean","last_fresh":"2024-02-09T17:55:31.186926+0000","last_change":"2024-02-09T17:50:17.104141+0000","last_active":"2024-02-09T17:55:31.186926+0000","last_peered":"2024-02-09T17:55:31.186926+0000","last_clean":"2024-02-09T17:55:31.186926+0000","last_became_active":"2024-02-09T17:43:19.138398+0000","last_became_peered":"2024-02-09T17:43:19.138398+0000","last_unstale":"2024-02-09T17:55:31.186926+0000","last_undegraded":"2024-02-09T17:55:31.186926+0000","last_fullsized":"2024-02-09T17:55:31.186926+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:50:17.104041+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:50:17.104041+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"63'139","reported_seq":1391,"reported_epoch":796,"state":"active+clean","last_fresh":"2024-02-09T17:55:28.924329+0000","last_change":"2024-02-09T17:55:09.116245+0000","last_active":"2024-02-09T17:55:28.924329+0000","last_peered":"2024-02-09T17:55:28.924329+0000","last_clean":"2024-02-09T17:55:28.924329+0000","last_became_active":"2024-02-09T17:43:28.589307+0000","last_became_peered":"2024-02-09T17:43:28.589307+0000","last_unstale":"2024-02-09T17:55:28.924329+0000","last_undegraded":"2024-02-09T17:55:28.924329+0000","last_fullsized":"2024-02-09T17:55:28.924329+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:09.116128+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:38:26.968892+0000","last_clean_scrub_stamp":"2024-02-09T17:55:09.116128+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:31.309149+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"798'1251","reported_seq":1287,"reported_epoch":798,"state":"active+clean","last_fresh":"2024-02-09T17:55:33.447125+0000","last_change":"2024-02-09T17:55:16.444774+0000","last_active":"2024-02-09T17:55:33.447125+0000","last_peered":"2024-02-09T17:55:33.447125+0000","last_clean":"2024-02-09T17:55:33.447125+0000","last_became_active":"2024-02-09T17:55:16.444412+0000","last_became_peered":"2024-02-09T17:55:16.444412+0000","last_unstale":"2024-02-09T17:55:33.447125+0000","last_undegraded":"2024-02-09T17:55:33.447125+0000","last_fullsized":"2024-02-09T17:55:33.447125+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":1251,"log_dups_size":0,"ondisk_log_size":1251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:39.138631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81985536,"num_objects":1251,"num_object_clones":0,"num_object_copies":3753,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1251,"num_write_kb":80064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,0],"acting":[10,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"63'139","reported_seq":1972,"reported_epoch":797,"state":"active+clean","last_fresh":"2024-02-09T17:55:31.186927+0000","last_change":"2024-02-09T17:55:19.636501+0000","last_active":"2024-02-09T17:55:31.186927+0000","last_peered":"2024-02-09T17:55:31.186927+0000","last_clean":"2024-02-09T17:55:31.186927+0000","last_became_active":"2024-02-09T17:41:14.521079+0000","last_became_peered":"2024-02-09T17:41:14.521079+0000","last_unstale":"2024-02-09T17:55:31.186927+0000","last_undegraded":"2024-02-09T17:55:31.186927+0000","last_fullsized":"2024-02-09T17:55:31.186927+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:19.636423+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:55:19.636423+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:27.811356+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"798'1010","reported_seq":1046,"reported_epoch":798,"state":"active+clean","last_fresh":"2024-02-09T17:55:32.423743+0000","last_change":"2024-02-09T17:55:16.913627+0000","last_active":"2024-02-09T17:55:32.423743+0000","last_peered":"2024-02-09T17:55:32.423743+0000","last_clean":"2024-02-09T17:55:32.423743+0000","last_became_active":"2024-02-09T17:55:16.913155+0000","last_became_peered":"2024-02-09T17:55:16.913155+0000","last_unstale":"2024-02-09T17:55:32.423743+0000","last_undegraded":"2024-02-09T17:55:32.423743+0000","last_fullsized":"2024-02-09T17:55:32.423743+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":1010,"log_dups_size":0,"ondisk_log_size":1010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:36.474670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66191360,"num_objects":1010,"num_object_clones":0,"num_object_copies":3030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1010,"num_write_kb":64640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,1],"acting":[8,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.2","version":"63'139","reported_seq":1539,"reported_epoch":798,"state":"active+clean","last_fresh":"2024-02-09T17:55:31.923696+0000","last_change":"2024-02-09T17:55:09.292488+0000","last_active":"2024-02-09T17:55:31.923696+0000","last_peered":"2024-02-09T17:55:31.923696+0000","last_clean":"2024-02-09T17:55:31.923696+0000","last_became_active":"2024-02-09T17:43:39.981054+0000","last_became_peered":"2024-02-09T17:43:39.981054+0000","last_unstale":"2024-02-09T17:55:31.923696+0000","last_undegraded":"2024-02-09T17:55:31.923696+0000","last_fullsized":"2024-02-09T17:55:31.923696+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:09.292403+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:55:09.292403+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:12.597930+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"798'1049","reported_seq":1085,"reported_epoch":798,"state":"active+clean","last_fresh":"2024-02-09T17:55:32.398494+0000","last_change":"2024-02-09T17:55:16.204365+0000","last_active":"2024-02-09T17:55:32.398494+0000","last_peered":"2024-02-09T17:55:32.398494+0000","last_clean":"2024-02-09T17:55:32.398494+0000","last_became_active":"2024-02-09T17:55:16.204001+0000","last_became_peered":"2024-02-09T17:55:16.204001+0000","last_unstale":"2024-02-09T17:55:32.398494+0000","last_undegraded":"2024-02-09T17:55:32.398494+0000","last_fullsized":"2024-02-09T17:55:32.398494+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":1049,"log_dups_size":0,"ondisk_log_size":1049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:19.806947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68747264,"num_objects":1049,"num_object_clones":0,"num_object_copies":3147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1049,"num_write_kb":67136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,9],"acting":[8,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"63'139","reported_seq":1166,"reported_epoch":796,"state":"active+clean","last_fresh":"2024-02-09T17:55:28.926219+0000","last_change":"2024-02-09T17:54:51.201051+0000","last_active":"2024-02-09T17:55:28.926219+0000","last_peered":"2024-02-09T17:55:28.926219+0000","last_clean":"2024-02-09T17:55:28.926219+0000","last_became_active":"2024-02-09T17:40:13.501711+0000","last_became_peered":"2024-02-09T17:40:13.501711+0000","last_unstale":"2024-02-09T17:55:28.926219+0000","last_undegraded":"2024-02-09T17:55:28.926219+0000","last_fullsized":"2024-02-09T17:55:28.926219+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:54:51.200917+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:46:40.684762+0000","last_clean_scrub_stamp":"2024-02-09T17:54:51.200917+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:10.063418+0000","scrub_duration":164,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"798'1183","reported_seq":1219,"reported_epoch":798,"state":"active+clean","last_fresh":"2024-02-09T17:55:33.124056+0000","last_change":"2024-02-09T17:55:16.445066+0000","last_active":"2024-02-09T17:55:33.124056+0000","last_peered":"2024-02-09T17:55:33.124056+0000","last_clean":"2024-02-09T17:55:33.124056+0000","last_became_active":"2024-02-09T17:55:16.444743+0000","last_became_peered":"2024-02-09T17:55:16.444743+0000","last_unstale":"2024-02-09T17:55:33.124056+0000","last_undegraded":"2024-02-09T17:55:33.124056+0000","last_fullsized":"2024-02-09T17:55:33.124056+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":1183,"log_dups_size":0,"ondisk_log_size":1183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:27.459931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77529088,"num_objects":1183,"num_object_clones":0,"num_object_copies":3549,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1183,"num_write_kb":75712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,9],"acting":[3,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"63'139","reported_seq":707,"reported_epoch":798,"state":"active+clean","last_fresh":"2024-02-09T17:55:31.919775+0000","last_change":"2024-02-09T17:55:13.048648+0000","last_active":"2024-02-09T17:55:31.919775+0000","last_peered":"2024-02-09T17:55:31.919775+0000","last_clean":"2024-02-09T17:55:31.919775+0000","last_became_active":"2024-02-09T17:54:32.415032+0000","last_became_peered":"2024-02-09T17:54:32.415032+0000","last_unstale":"2024-02-09T17:55:31.919775+0000","last_undegraded":"2024-02-09T17:55:31.919775+0000","last_fullsized":"2024-02-09T17:55:31.919775+0000","mapping_epoch":766,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":767,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:13.048497+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:55:13.048497+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:35.168553+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.f","version":"797'896","reported_seq":930,"reported_epoch":797,"state":"active+clean","last_fresh":"2024-02-09T17:55:31.336796+0000","last_change":"2024-02-09T17:55:16.205753+0000","last_active":"2024-02-09T17:55:31.336796+0000","last_peered":"2024-02-09T17:55:31.336796+0000","last_clean":"2024-02-09T17:55:31.336796+0000","last_became_active":"2024-02-09T17:55:16.205394+0000","last_became_peered":"2024-02-09T17:55:16.205394+0000","last_unstale":"2024-02-09T17:55:31.336796+0000","last_undegraded":"2024-02-09T17:55:31.336796+0000","last_fullsized":"2024-02-09T17:55:31.336796+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":896,"log_dups_size":0,"ondisk_log_size":896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:24.386617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":896,"num_object_clones":0,"num_object_copies":2688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":896,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,1],"acting":[11,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"63'139","reported_seq":1474,"reported_epoch":798,"state":"active+clean","last_fresh":"2024-02-09T17:55:31.919632+0000","last_change":"2024-02-09T17:55:11.015552+0000","last_active":"2024-02-09T17:55:31.919632+0000","last_peered":"2024-02-09T17:55:31.919632+0000","last_clean":"2024-02-09T17:55:31.919632+0000","last_became_active":"2024-02-09T17:43:14.024894+0000","last_became_peered":"2024-02-09T17:43:14.024894+0000","last_unstale":"2024-02-09T17:55:31.919632+0000","last_undegraded":"2024-02-09T17:55:31.919632+0000","last_fullsized":"2024-02-09T17:55:31.919632+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:11.015439+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:11.970338+0000","last_clean_scrub_stamp":"2024-02-09T17:55:11.015439+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:11.829031+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.e","version":"798'1261","reported_seq":1297,"reported_epoch":798,"state":"active+clean","last_fresh":"2024-02-09T17:55:33.792792+0000","last_change":"2024-02-09T17:55:16.913062+0000","last_active":"2024-02-09T17:55:33.792792+0000","last_peered":"2024-02-09T17:55:33.792792+0000","last_clean":"2024-02-09T17:55:33.792792+0000","last_became_active":"2024-02-09T17:55:16.912769+0000","last_became_peered":"2024-02-09T17:55:16.912769+0000","last_unstale":"2024-02-09T17:55:33.792792+0000","last_undegraded":"2024-02-09T17:55:33.792792+0000","last_fullsized":"2024-02-09T17:55:33.792792+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":1261,"log_dups_size":0,"ondisk_log_size":1261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:27.866364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82640896,"num_objects":1261,"num_object_clones":0,"num_object_copies":3783,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1261,"num_write_kb":80704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"63'139","reported_seq":1094,"reported_epoch":796,"state":"active+clean","last_fresh":"2024-02-09T17:55:28.926159+0000","last_change":"2024-02-09T17:55:05.394372+0000","last_active":"2024-02-09T17:55:28.926159+0000","last_peered":"2024-02-09T17:55:28.926159+0000","last_clean":"2024-02-09T17:55:28.926159+0000","last_became_active":"2024-02-09T17:52:32.292519+0000","last_became_peered":"2024-02-09T17:52:32.292519+0000","last_unstale":"2024-02-09T17:55:28.926159+0000","last_undegraded":"2024-02-09T17:55:28.926159+0000","last_fullsized":"2024-02-09T17:55:28.926159+0000","mapping_epoch":716,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":717,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:05.394288+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:55:05.394288+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:30.532392+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.d","version":"797'873","reported_seq":907,"reported_epoch":797,"state":"active+clean","last_fresh":"2024-02-09T17:55:31.323329+0000","last_change":"2024-02-09T17:55:16.445197+0000","last_active":"2024-02-09T17:55:31.323329+0000","last_peered":"2024-02-09T17:55:31.323329+0000","last_clean":"2024-02-09T17:55:31.323329+0000","last_became_active":"2024-02-09T17:55:16.444812+0000","last_became_peered":"2024-02-09T17:55:16.444812+0000","last_unstale":"2024-02-09T17:55:31.323329+0000","last_undegraded":"2024-02-09T17:55:31.323329+0000","last_fullsized":"2024-02-09T17:55:31.323329+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":873,"log_dups_size":0,"ondisk_log_size":873,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:43.526067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57212928,"num_objects":873,"num_object_clones":0,"num_object_copies":2619,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":873,"num_write_kb":55872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,2],"acting":[11,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"63'139","reported_seq":1551,"reported_epoch":797,"state":"active+clean","last_fresh":"2024-02-09T17:55:31.183747+0000","last_change":"2024-02-09T17:55:13.976245+0000","last_active":"2024-02-09T17:55:31.183747+0000","last_peered":"2024-02-09T17:55:31.183747+0000","last_clean":"2024-02-09T17:55:31.183747+0000","last_became_active":"2024-02-09T17:33:00.284924+0000","last_became_peered":"2024-02-09T17:33:00.284924+0000","last_unstale":"2024-02-09T17:55:31.183747+0000","last_undegraded":"2024-02-09T17:55:31.183747+0000","last_fullsized":"2024-02-09T17:55:31.183747+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:13.976095+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:56.105792+0000","last_clean_scrub_stamp":"2024-02-09T17:55:13.976095+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:25.434647+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.c","version":"798'953","reported_seq":989,"reported_epoch":798,"state":"active+clean","last_fresh":"2024-02-09T17:55:32.003942+0000","last_change":"2024-02-09T17:55:16.207610+0000","last_active":"2024-02-09T17:55:32.003942+0000","last_peered":"2024-02-09T17:55:32.003942+0000","last_clean":"2024-02-09T17:55:32.003942+0000","last_became_active":"2024-02-09T17:55:16.207033+0000","last_became_peered":"2024-02-09T17:55:16.207033+0000","last_unstale":"2024-02-09T17:55:32.003942+0000","last_undegraded":"2024-02-09T17:55:32.003942+0000","last_fullsized":"2024-02-09T17:55:32.003942+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":953,"log_dups_size":0,"ondisk_log_size":953,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:18.430656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62455808,"num_objects":953,"num_object_clones":0,"num_object_copies":2859,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":953,"num_write_kb":60992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.5","version":"63'139","reported_seq":1394,"reported_epoch":798,"state":"active+clean","last_fresh":"2024-02-09T17:55:31.920002+0000","last_change":"2024-02-09T17:55:06.072812+0000","last_active":"2024-02-09T17:55:31.920002+0000","last_peered":"2024-02-09T17:55:31.920002+0000","last_clean":"2024-02-09T17:55:31.920002+0000","last_became_active":"2024-02-09T17:48:46.184307+0000","last_became_peered":"2024-02-09T17:48:46.184307+0000","last_unstale":"2024-02-09T17:55:31.920002+0000","last_undegraded":"2024-02-09T17:55:31.920002+0000","last_fullsized":"2024-02-09T17:55:31.920002+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:06.072671+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:55:06.072671+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:13.050346+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.3","version":"797'879","reported_seq":913,"reported_epoch":797,"state":"active+clean","last_fresh":"2024-02-09T17:55:31.337069+0000","last_change":"2024-02-09T17:55:16.445395+0000","last_active":"2024-02-09T17:55:31.337069+0000","last_peered":"2024-02-09T17:55:31.337069+0000","last_clean":"2024-02-09T17:55:31.337069+0000","last_became_active":"2024-02-09T17:55:16.445048+0000","last_became_peered":"2024-02-09T17:55:16.445048+0000","last_unstale":"2024-02-09T17:55:31.337069+0000","last_undegraded":"2024-02-09T17:55:31.337069+0000","last_fullsized":"2024-02-09T17:55:31.337069+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":879,"log_dups_size":0,"ondisk_log_size":879,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:31.073410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57606144,"num_objects":879,"num_object_clones":0,"num_object_copies":2637,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":879,"num_write_kb":56256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"63'139","reported_seq":1475,"reported_epoch":798,"state":"active+clean","last_fresh":"2024-02-09T17:55:31.912208+0000","last_change":"2024-02-09T17:54:58.972744+0000","last_active":"2024-02-09T17:55:31.912208+0000","last_peered":"2024-02-09T17:55:31.912208+0000","last_clean":"2024-02-09T17:55:31.912208+0000","last_became_active":"2024-02-09T17:43:14.041291+0000","last_became_peered":"2024-02-09T17:43:14.041291+0000","last_unstale":"2024-02-09T17:55:31.912208+0000","last_undegraded":"2024-02-09T17:55:31.912208+0000","last_fullsized":"2024-02-09T17:55:31.912208+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:54:58.972617+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:49:39.575571+0000","last_clean_scrub_stamp":"2024-02-09T17:54:58.972617+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:09.245500+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.2","version":"798'1240","reported_seq":1276,"reported_epoch":798,"state":"active+clean","last_fresh":"2024-02-09T17:55:33.448926+0000","last_change":"2024-02-09T17:55:16.914203+0000","last_active":"2024-02-09T17:55:33.448926+0000","last_peered":"2024-02-09T17:55:33.448926+0000","last_clean":"2024-02-09T17:55:33.448926+0000","last_became_active":"2024-02-09T17:55:16.913786+0000","last_became_peered":"2024-02-09T17:55:16.913786+0000","last_unstale":"2024-02-09T17:55:33.448926+0000","last_undegraded":"2024-02-09T17:55:33.448926+0000","last_fullsized":"2024-02-09T17:55:33.448926+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":1240,"log_dups_size":0,"ondisk_log_size":1240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:17.666318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81264640,"num_objects":1240,"num_object_clones":0,"num_object_copies":3720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1240,"num_write_kb":79360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,4],"acting":[10,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.b","version":"63'139","reported_seq":841,"reported_epoch":797,"state":"active+clean","last_fresh":"2024-02-09T17:55:31.183760+0000","last_change":"2024-02-09T17:55:10.144538+0000","last_active":"2024-02-09T17:55:31.183760+0000","last_peered":"2024-02-09T17:55:31.183760+0000","last_clean":"2024-02-09T17:55:31.183760+0000","last_became_active":"2024-02-09T17:55:08.938203+0000","last_became_peered":"2024-02-09T17:55:08.938203+0000","last_unstale":"2024-02-09T17:55:31.183760+0000","last_undegraded":"2024-02-09T17:55:31.183760+0000","last_fullsized":"2024-02-09T17:55:31.183760+0000","mapping_epoch":781,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":782,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:10.144404+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:55:10.144404+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:25.671268+0000","scrub_duration":181,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.1","version":"796'699","reported_seq":731,"reported_epoch":796,"state":"active+clean","last_fresh":"2024-02-09T17:55:30.391597+0000","last_change":"2024-02-09T17:55:16.205903+0000","last_active":"2024-02-09T17:55:30.391597+0000","last_peered":"2024-02-09T17:55:30.391597+0000","last_clean":"2024-02-09T17:55:30.391597+0000","last_became_active":"2024-02-09T17:55:16.205574+0000","last_became_peered":"2024-02-09T17:55:16.205574+0000","last_unstale":"2024-02-09T17:55:30.391597+0000","last_undegraded":"2024-02-09T17:55:30.391597+0000","last_fullsized":"2024-02-09T17:55:30.391597+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":699,"log_dups_size":0,"ondisk_log_size":699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:24.914406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45809664,"num_objects":699,"num_object_clones":0,"num_object_copies":2097,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":699,"num_write_kb":44736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,2],"acting":[1,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"63'139","reported_seq":1125,"reported_epoch":798,"state":"active+clean","last_fresh":"2024-02-09T17:55:31.909479+0000","last_change":"2024-02-09T17:55:10.313343+0000","last_active":"2024-02-09T17:55:31.909479+0000","last_peered":"2024-02-09T17:55:31.909479+0000","last_clean":"2024-02-09T17:55:31.909479+0000","last_became_active":"2024-02-09T17:46:47.013853+0000","last_became_peered":"2024-02-09T17:46:47.013853+0000","last_unstale":"2024-02-09T17:55:31.909479+0000","last_undegraded":"2024-02-09T17:55:31.909479+0000","last_fullsized":"2024-02-09T17:55:31.909479+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:10.313268+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:55:10.313268+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:24.198160+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"798'1188","reported_seq":1224,"reported_epoch":798,"state":"active+clean","last_fresh":"2024-02-09T17:55:33.119226+0000","last_change":"2024-02-09T17:55:16.444927+0000","last_active":"2024-02-09T17:55:33.119226+0000","last_peered":"2024-02-09T17:55:33.119226+0000","last_clean":"2024-02-09T17:55:33.119226+0000","last_became_active":"2024-02-09T17:55:16.444683+0000","last_became_peered":"2024-02-09T17:55:16.444683+0000","last_unstale":"2024-02-09T17:55:33.119226+0000","last_undegraded":"2024-02-09T17:55:33.119226+0000","last_fullsized":"2024-02-09T17:55:33.119226+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":1188,"log_dups_size":0,"ondisk_log_size":1188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:37.733967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77856768,"num_objects":1188,"num_object_clones":0,"num_object_copies":3564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1188,"num_write_kb":76032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,9],"acting":[3,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"63'139","reported_seq":1978,"reported_epoch":797,"state":"active+clean","last_fresh":"2024-02-09T17:55:31.186897+0000","last_change":"2024-02-09T17:55:13.823711+0000","last_active":"2024-02-09T17:55:31.186897+0000","last_peered":"2024-02-09T17:55:31.186897+0000","last_clean":"2024-02-09T17:55:31.186897+0000","last_became_active":"2024-02-09T17:43:28.590237+0000","last_became_peered":"2024-02-09T17:43:28.590237+0000","last_unstale":"2024-02-09T17:55:31.186897+0000","last_undegraded":"2024-02-09T17:55:31.186897+0000","last_fullsized":"2024-02-09T17:55:31.186897+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:13.823598+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:45:47.710252+0000","last_clean_scrub_stamp":"2024-02-09T17:55:13.823598+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:41.832421+0000","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.7","version":"796'682","reported_seq":713,"reported_epoch":796,"state":"active+clean","last_fresh":"2024-02-09T17:55:30.380177+0000","last_change":"2024-02-09T17:55:16.206804+0000","last_active":"2024-02-09T17:55:30.380177+0000","last_peered":"2024-02-09T17:55:30.380177+0000","last_clean":"2024-02-09T17:55:30.380177+0000","last_became_active":"2024-02-09T17:55:16.206561+0000","last_became_peered":"2024-02-09T17:55:16.206561+0000","last_unstale":"2024-02-09T17:55:30.380177+0000","last_undegraded":"2024-02-09T17:55:30.380177+0000","last_fullsized":"2024-02-09T17:55:30.380177+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":682,"log_dups_size":0,"ondisk_log_size":682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:41.254300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44695552,"num_objects":682,"num_object_clones":0,"num_object_copies":2046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":682,"num_write_kb":43648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.e","version":"63'139","reported_seq":1434,"reported_epoch":797,"state":"active+clean","last_fresh":"2024-02-09T17:55:31.190997+0000","last_change":"2024-02-09T17:55:08.044579+0000","last_active":"2024-02-09T17:55:31.190997+0000","last_peered":"2024-02-09T17:55:31.190997+0000","last_clean":"2024-02-09T17:55:31.190997+0000","last_became_active":"2024-02-09T17:43:14.036583+0000","last_became_peered":"2024-02-09T17:43:14.036583+0000","last_unstale":"2024-02-09T17:55:31.190997+0000","last_undegraded":"2024-02-09T17:55:31.190997+0000","last_fullsized":"2024-02-09T17:55:31.190997+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:08.044477+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:14.981453+0000","last_clean_scrub_stamp":"2024-02-09T17:55:08.044477+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:28.800003+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"797'899","reported_seq":933,"reported_epoch":797,"state":"active+clean","last_fresh":"2024-02-09T17:55:31.323335+0000","last_change":"2024-02-09T17:55:16.446161+0000","last_active":"2024-02-09T17:55:31.323335+0000","last_peered":"2024-02-09T17:55:31.323335+0000","last_clean":"2024-02-09T17:55:31.323335+0000","last_became_active":"2024-02-09T17:55:16.445702+0000","last_became_peered":"2024-02-09T17:55:16.445702+0000","last_unstale":"2024-02-09T17:55:31.323335+0000","last_undegraded":"2024-02-09T17:55:31.323335+0000","last_fullsized":"2024-02-09T17:55:31.323335+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":899,"log_dups_size":0,"ondisk_log_size":899,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:18.722073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58916864,"num_objects":899,"num_object_clones":0,"num_object_copies":2697,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":899,"num_write_kb":57536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,7],"acting":[11,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.f","version":"63'139","reported_seq":1493,"reported_epoch":796,"state":"active+clean","last_fresh":"2024-02-09T17:55:28.922088+0000","last_change":"2024-02-09T17:55:08.355359+0000","last_active":"2024-02-09T17:55:28.922088+0000","last_peered":"2024-02-09T17:55:28.922088+0000","last_clean":"2024-02-09T17:55:28.922088+0000","last_became_active":"2024-02-09T17:33:05.316100+0000","last_became_peered":"2024-02-09T17:33:05.316100+0000","last_unstale":"2024-02-09T17:55:28.922088+0000","last_undegraded":"2024-02-09T17:55:28.922088+0000","last_fullsized":"2024-02-09T17:55:28.922088+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:08.355221+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:48:34.651647+0000","last_clean_scrub_stamp":"2024-02-09T17:55:08.355221+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:11.673721+0000","scrub_duration":144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.5","version":"798'1219","reported_seq":1254,"reported_epoch":798,"state":"active+clean","last_fresh":"2024-02-09T17:55:33.442625+0000","last_change":"2024-02-09T17:55:16.913593+0000","last_active":"2024-02-09T17:55:33.442625+0000","last_peered":"2024-02-09T17:55:33.442625+0000","last_clean":"2024-02-09T17:55:33.442625+0000","last_became_active":"2024-02-09T17:55:16.913201+0000","last_became_peered":"2024-02-09T17:55:16.913201+0000","last_unstale":"2024-02-09T17:55:33.442625+0000","last_undegraded":"2024-02-09T17:55:33.442625+0000","last_fullsized":"2024-02-09T17:55:33.442625+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":1219,"log_dups_size":0,"ondisk_log_size":1219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:26.249705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79888384,"num_objects":1219,"num_object_clones":0,"num_object_copies":3657,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1219,"num_write_kb":78016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,1],"acting":[10,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.c","version":"63'139","reported_seq":1624,"reported_epoch":798,"state":"active+clean","last_fresh":"2024-02-09T17:55:31.909444+0000","last_change":"2024-02-09T17:55:07.304327+0000","last_active":"2024-02-09T17:55:31.909444+0000","last_peered":"2024-02-09T17:55:31.909444+0000","last_clean":"2024-02-09T17:55:31.909444+0000","last_became_active":"2024-02-09T17:43:14.027715+0000","last_became_peered":"2024-02-09T17:43:14.027715+0000","last_unstale":"2024-02-09T17:55:31.909444+0000","last_undegraded":"2024-02-09T17:55:31.909444+0000","last_fullsized":"2024-02-09T17:55:31.909444+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:07.304236+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:14.135321+0000","last_clean_scrub_stamp":"2024-02-09T17:55:07.304236+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:16.017635+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"798'957","reported_seq":993,"reported_epoch":798,"state":"active+clean","last_fresh":"2024-02-09T17:55:32.010738+0000","last_change":"2024-02-09T17:55:16.445202+0000","last_active":"2024-02-09T17:55:32.010738+0000","last_peered":"2024-02-09T17:55:32.010738+0000","last_clean":"2024-02-09T17:55:32.010738+0000","last_became_active":"2024-02-09T17:55:16.444827+0000","last_became_peered":"2024-02-09T17:55:16.444827+0000","last_unstale":"2024-02-09T17:55:32.010738+0000","last_undegraded":"2024-02-09T17:55:32.010738+0000","last_fullsized":"2024-02-09T17:55:32.010738+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":957,"log_dups_size":0,"ondisk_log_size":957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:20.836494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62717952,"num_objects":957,"num_object_clones":0,"num_object_copies":2871,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":957,"num_write_kb":61248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.d","version":"63'139","reported_seq":1027,"reported_epoch":797,"state":"active+clean","last_fresh":"2024-02-09T17:55:31.187395+0000","last_change":"2024-02-09T17:55:08.898009+0000","last_active":"2024-02-09T17:55:31.187395+0000","last_peered":"2024-02-09T17:55:31.187395+0000","last_clean":"2024-02-09T17:55:31.187395+0000","last_became_active":"2024-02-09T17:43:14.036759+0000","last_became_peered":"2024-02-09T17:43:14.036759+0000","last_unstale":"2024-02-09T17:55:31.187395+0000","last_undegraded":"2024-02-09T17:55:31.187395+0000","last_fullsized":"2024-02-09T17:55:31.187395+0000","mapping_epoch":454,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:08.897882+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:55:08.897882+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:15.119150+0000","scrub_duration":145,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"63'139","reported_seq":1198,"reported_epoch":797,"state":"active+clean","last_fresh":"2024-02-09T17:55:31.186846+0000","last_change":"2024-02-09T17:55:10.810070+0000","last_active":"2024-02-09T17:55:31.186846+0000","last_peered":"2024-02-09T17:55:31.186846+0000","last_clean":"2024-02-09T17:55:31.186846+0000","last_became_active":"2024-02-09T17:46:47.013920+0000","last_became_peered":"2024-02-09T17:46:47.013920+0000","last_unstale":"2024-02-09T17:55:31.186846+0000","last_undegraded":"2024-02-09T17:55:31.186846+0000","last_fullsized":"2024-02-09T17:55:31.186846+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:10.809935+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:48:35.170638+0000","last_clean_scrub_stamp":"2024-02-09T17:55:10.809935+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:31.681608+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"63'139","reported_seq":1470,"reported_epoch":798,"state":"active+clean","last_fresh":"2024-02-09T17:55:31.915498+0000","last_change":"2024-02-09T17:55:21.232444+0000","last_active":"2024-02-09T17:55:31.915498+0000","last_peered":"2024-02-09T17:55:31.915498+0000","last_clean":"2024-02-09T17:55:31.915498+0000","last_became_active":"2024-02-09T17:55:21.231902+0000","last_became_peered":"2024-02-09T17:55:21.231902+0000","last_unstale":"2024-02-09T17:55:31.915498+0000","last_undegraded":"2024-02-09T17:55:31.915498+0000","last_fullsized":"2024-02-09T17:55:31.915498+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":791,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:01.456654+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:44:51.190789+0000","last_clean_scrub_stamp":"2024-02-09T17:55:01.456654+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:22.186440+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.12","version":"63'139","reported_seq":1541,"reported_epoch":796,"state":"active+clean","last_fresh":"2024-02-09T17:55:28.925993+0000","last_change":"2024-02-09T17:54:57.092672+0000","last_active":"2024-02-09T17:55:28.925993+0000","last_peered":"2024-02-09T17:55:28.925993+0000","last_clean":"2024-02-09T17:55:28.925993+0000","last_became_active":"2024-02-09T17:46:47.015622+0000","last_became_peered":"2024-02-09T17:46:47.015622+0000","last_unstale":"2024-02-09T17:55:28.925993+0000","last_undegraded":"2024-02-09T17:55:28.925993+0000","last_fullsized":"2024-02-09T17:55:28.925993+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:54:57.092525+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:54:57.092525+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:55:58.889416+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.13","version":"63'139","reported_seq":1160,"reported_epoch":796,"state":"active+clean","last_fresh":"2024-02-09T17:55:28.926318+0000","last_change":"2024-02-09T17:54:54.282606+0000","last_active":"2024-02-09T17:55:28.926318+0000","last_peered":"2024-02-09T17:55:28.926318+0000","last_clean":"2024-02-09T17:55:28.926318+0000","last_became_active":"2024-02-09T17:41:14.521672+0000","last_became_peered":"2024-02-09T17:41:14.521672+0000","last_unstale":"2024-02-09T17:55:28.926318+0000","last_undegraded":"2024-02-09T17:55:28.926318+0000","last_fullsized":"2024-02-09T17:55:28.926318+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:54:54.282472+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:54:54.282472+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:19.415044+0000","scrub_duration":214,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":1064239104,"num_objects":16239,"num_object_clones":0,"num_object_copies":48717,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16239,"num_write_kb":1039296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2888564736,"data_stored":2888564736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16239,"ondisk_log_size":16239,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547057039,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1043044,"kb_used_data":258008,"kb_used_omap":1,"kb_used_meta":785022,"kb_avail":92702108,"statfs":{"total":95995035648,"available":94926958592,"internally_reserved":0,"allocated":264200192,"data_stored":263778753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":803862980},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:54:59 2024","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":2.206,"15min":1.5169999999999999},"min":{"1min":0.27100000000000002,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":4.492,"5min":20.003,"15min":20.003},"last":0.28999999999999998},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.8500000000000001,"15min":1.357},"min":{"1min":0.248,"5min":0.248,"15min":0.23100000000000001},"max":{"1min":4.8109999999999999,"5min":9.6259999999999994,"15min":9.8219999999999992},"last":0.48799999999999999}]},{"osd":1,"last update":"Fri Feb 9 17:54:59 2024","interfaces":[{"interface":"back","average":{"1min":1.5720000000000001,"5min":1.8180000000000001,"15min":1.5529999999999999},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":8.1790000000000003,"5min":9.532,"15min":11.576000000000001},"last":0.37},{"interface":"front","average":{"1min":1.4550000000000001,"5min":1.7609999999999999,"15min":1.651},"min":{"1min":0.314,"5min":0.25,"15min":0.25},"max":{"1min":4.9100000000000001,"5min":14.287000000000001,"15min":14.287000000000001},"last":5.0410000000000004}]},{"osd":2,"last update":"Fri Feb 9 17:31:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.415,"15min":1.6859999999999999},"min":{"1min":0.378,"5min":0.247,"15min":0.247},"max":{"1min":4.4189999999999996,"5min":9.4719999999999995,"15min":9.4719999999999995},"last":4.9989999999999997},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.5149999999999999,"15min":1.4399999999999999},"min":{"1min":0.378,"5min":0.252,"15min":0.252},"max":{"1min":4.8579999999999997,"5min":7.96,"15min":7.96},"last":0.45600000000000002}]},{"osd":3,"last update":"Fri Feb 9 17:54:59 2024","interfaces":[{"interface":"back","average":{"1min":3.669,"5min":2.3580000000000001,"15min":1.8120000000000001},"min":{"1min":0.34899999999999998,"5min":0.27900000000000003,"15min":0.248},"max":{"1min":43.344000000000001,"5min":43.344000000000001,"15min":43.344000000000001},"last":5.1509999999999998},{"interface":"front","average":{"1min":3.7029999999999998,"5min":2.399,"15min":1.9330000000000001},"min":{"1min":0.44500000000000001,"5min":0.375,"15min":0.33000000000000002},"max":{"1min":43.378,"5min":43.378,"15min":43.378},"last":5.1769999999999996}]},{"osd":4,"last update":"Fri Feb 9 17:54:59 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":1.746,"15min":1.762},"min":{"1min":0.254,"5min":0.254,"15min":0.246},"max":{"1min":4.7460000000000004,"5min":19.956,"15min":65.109999999999999},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.89000000000000001,"5min":1.496,"15min":1.7110000000000001},"min":{"1min":0.309,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":3.3490000000000002,"5min":9.5690000000000008,"15min":65.099000000000004},"last":0.50700000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:54:59 2024","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.5049999999999999,"15min":1.5580000000000001},"min":{"1min":0.30199999999999999,"5min":0.27500000000000002,"15min":0.252},"max":{"1min":4.782,"5min":11.119999999999999,"15min":18.878},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.86799999999999999,"5min":1.617,"15min":1.5609999999999999},"min":{"1min":0.38900000000000001,"5min":0.29199999999999998,"15min":0.28999999999999998},"max":{"1min":4.7930000000000001,"5min":20.038,"15min":20.038},"last":5.069}]},{"osd":6,"last update":"Fri Feb 9 17:54:59 2024","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.6419999999999999,"15min":1.625},"min":{"1min":0.33500000000000002,"5min":0.315,"15min":0.30399999999999999},"max":{"1min":3.484,"5min":9.5980000000000008,"15min":12.362},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.91000000000000003,"5min":1.4990000000000001,"15min":1.651},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":3.2109999999999999,"5min":15.409000000000001,"15min":15.409000000000001},"last":5.125}]},{"osd":7,"last update":"Fri Feb 9 17:54:59 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":1.4670000000000001,"15min":1.641},"min":{"1min":0.39300000000000002,"5min":0.317,"15min":0.317},"max":{"1min":3.3519999999999999,"5min":15.51,"15min":18.846},"last":5.0979999999999999},{"interface":"front","average":{"1min":0.88500000000000001,"5min":2.1179999999999999,"15min":1.8320000000000001},"min":{"1min":0.372,"5min":0.318,"15min":0.318},"max":{"1min":4.8390000000000004,"5min":24.373999999999999,"15min":24.373999999999999},"last":0.76200000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:55:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":2.1019999999999999,"15min":1.4330000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.27900000000000003},"max":{"1min":4.0279999999999996,"5min":13.574,"15min":13.574},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.97099999999999997,"5min":2.1309999999999998,"15min":1.51},"min":{"1min":0.46999999999999997,"5min":0.40999999999999998,"15min":0.32700000000000001},"max":{"1min":3.383,"5min":13.606,"15min":13.606},"last":1.6140000000000001}]},{"osd":9,"last update":"Fri Feb 9 17:54:42 2024","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":2.4470000000000001,"15min":1.889},"min":{"1min":0.32900000000000001,"5min":0.313,"15min":0.29399999999999998},"max":{"1min":5.4290000000000003,"5min":22.613,"15min":62.518000000000001},"last":5.2960000000000003},{"interface":"front","average":{"1min":1.3260000000000001,"5min":1.8480000000000001,"15min":1.667},"min":{"1min":0.38300000000000001,"5min":0.35299999999999998,"15min":0.34399999999999997},"max":{"1min":4.8689999999999998,"5min":12.289,"15min":63.241999999999997},"last":0.79100000000000004}]},{"osd":10,"last update":"Fri Feb 9 17:54:59 2024","interfaces":[{"interface":"back","average":{"1min":1.8460000000000001,"5min":2.4670000000000001,"15min":1.6779999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.32700000000000001},"max":{"1min":7.8150000000000004,"5min":17.373000000000001,"15min":17.373000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.96899999999999997,"5min":1.806,"15min":1.508},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":5.2699999999999996,"5min":20.233000000000001,"15min":20.233000000000001},"last":5.2370000000000001}]}]},{"osd":10,"up_from":84,"seq":360777253271,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1103288,"kb_used_data":454320,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":92641864,"statfs":{"total":95995035648,"available":94865268736,"internally_reserved":0,"allocated":465223680,"data_stored":464793319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":664533440},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:54:44 2024","interfaces":[{"interface":"back","average":{"1min":3.3199999999999998,"5min":1.6870000000000001,"15min":1.22},"min":{"1min":0.25700000000000001,"5min":0.245,"15min":0.20599999999999999},"max":{"1min":7.6349999999999998,"5min":10.532,"15min":10.532},"last":0.29699999999999999},{"interface":"front","average":{"1min":3.0259999999999998,"5min":1.6619999999999999,"15min":1.216},"min":{"1min":0.31,"5min":0.26400000000000001,"15min":0.248},"max":{"1min":9.5779999999999994,"5min":10.573,"15min":10.573},"last":0.41699999999999998}]},{"osd":1,"last update":"Fri Feb 9 17:51:04 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":1.4430000000000001,"15min":0.97699999999999998},"min":{"1min":0.307,"5min":0.246,"15min":0.23999999999999999},"max":{"1min":3.464,"5min":9.3819999999999997,"15min":9.3819999999999997},"last":1.6180000000000001},{"interface":"front","average":{"1min":0.81599999999999995,"5min":1.5249999999999999,"15min":1.1279999999999999},"min":{"1min":0.29899999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":4.3049999999999997,"5min":8.9789999999999992,"15min":8.9789999999999992},"last":0.60099999999999998}]},{"osd":2,"last update":"Fri Feb 9 17:55:20 2024","interfaces":[{"interface":"back","average":{"1min":1.8400000000000001,"5min":1.96,"15min":1.5620000000000001},"min":{"1min":0.27800000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":7.399,"5min":13.215999999999999,"15min":13.215999999999999},"last":0.377},{"interface":"front","average":{"1min":1.661,"5min":2.101,"15min":1.6870000000000001},"min":{"1min":0.46500000000000002,"5min":0.371,"15min":0.25600000000000001},"max":{"1min":5.8239999999999998,"5min":13.259,"15min":38.868000000000002},"last":1.2649999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:55:09 2024","interfaces":[{"interface":"back","average":{"1min":2.282,"5min":2.1549999999999998,"15min":1.796},"min":{"1min":0.29999999999999999,"5min":0.27700000000000002,"15min":0.26100000000000001},"max":{"1min":6.9909999999999997,"5min":14.689,"15min":35.113},"last":0.65700000000000003},{"interface":"front","average":{"1min":1.296,"5min":2.4550000000000001,"15min":2.077},"min":{"1min":0.27800000000000002,"5min":0.249,"15min":0.249},"max":{"1min":3.5449999999999999,"5min":11.209,"15min":34.359000000000002},"last":0.54400000000000004}]},{"osd":4,"last update":"Fri Feb 9 17:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":1.1719999999999999,"15min":1.1890000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":3.9060000000000001,"5min":11.705,"15min":11.705},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.66400000000000003,"5min":1.3700000000000001,"15min":1.262},"min":{"1min":0.34799999999999998,"5min":0.33900000000000002,"15min":0.27000000000000002},"max":{"1min":3.8799999999999999,"5min":11.744,"15min":11.744},"last":0.55400000000000005}]},{"osd":5,"last update":"Fri Feb 9 17:55:20 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":1.292,"15min":1.2929999999999999},"min":{"1min":0.33200000000000002,"5min":0.30099999999999999,"15min":0.29099999999999998},"max":{"1min":0.82899999999999996,"5min":10.759,"15min":10.759},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.97499999999999998,"5min":1.4039999999999999,"15min":1.504},"min":{"1min":0.46700000000000003,"5min":0.373,"15min":0.33500000000000002},"max":{"1min":5.5389999999999997,"5min":11.779,"15min":11.779},"last":0.58699999999999997}]},{"osd":6,"last update":"Fri Feb 9 17:55:24 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":1.272,"15min":1.4259999999999999},"min":{"1min":0.34599999999999997,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.77000000000000002,"5min":10.776999999999999,"15min":10.962},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.5780000000000001,"15min":1.5129999999999999},"min":{"1min":0.434,"5min":0.35099999999999998,"15min":0.32600000000000001},"max":{"1min":5.5910000000000002,"5min":12.414,"15min":12.414},"last":0.61899999999999999}]},{"osd":7,"last update":"Fri Feb 9 17:55:15 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":2.0680000000000001,"15min":3.0099999999999998},"min":{"1min":0.40799999999999997,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.976,"5min":11.288,"15min":11.288},"last":0.5},{"interface":"front","average":{"1min":0.877,"5min":2.0920000000000001,"15min":3.032},"min":{"1min":0.438,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":3.242,"5min":10.736000000000001,"15min":10.736000000000001},"last":0.63600000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:55:09 2024","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":1.6519999999999999,"15min":1.1899999999999999},"min":{"1min":0.34200000000000003,"5min":0.32400000000000001,"15min":0.31},"max":{"1min":5.0099999999999998,"5min":10.792999999999999,"15min":10.792999999999999},"last":0.436},{"interface":"front","average":{"1min":0.66500000000000004,"5min":1.776,"15min":1.349},"min":{"1min":0.41399999999999998,"5min":0.35299999999999998,"15min":0.30599999999999999},"max":{"1min":1.4950000000000001,"5min":12.249000000000001,"15min":12.249000000000001},"last":0.56699999999999995}]},{"osd":9,"last update":"Fri Feb 9 17:55:20 2024","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":2.3220000000000001,"15min":1.419},"min":{"1min":0.373,"5min":0.34300000000000003,"15min":0.33300000000000002},"max":{"1min":4.2350000000000003,"5min":13.298,"15min":13.298},"last":1.5680000000000001},{"interface":"front","average":{"1min":1.1140000000000001,"5min":2.206,"15min":1.5329999999999999},"min":{"1min":0.44,"5min":0.39600000000000002,"15min":0.36099999999999999},"max":{"1min":4.7080000000000002,"5min":10.473000000000001,"15min":10.622999999999999},"last":2.0619999999999998}]},{"osd":11,"last update":"Fri Feb 9 17:54:44 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.8080000000000001,"15min":1.2470000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.35399999999999998},"max":{"1min":5.4249999999999998,"5min":23.579000000000001,"15min":23.579000000000001},"last":0.84399999999999997},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.857,"15min":1.3009999999999999},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.438},"max":{"1min":5.5650000000000004,"5min":23.619,"15min":23.619},"last":0.92100000000000004}]}]},{"osd":9,"up_from":77,"seq":330712482202,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1064624,"kb_used_data":361508,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":92680528,"statfs":{"total":95995035648,"available":94904860672,"internally_reserved":0,"allocated":370184192,"data_stored":369758317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":719976896},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:55:17 2024","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.8919999999999999,"15min":1.3340000000000001},"min":{"1min":0.27900000000000003,"5min":0.23899999999999999,"15min":0.215},"max":{"1min":9.718,"5min":11.539999999999999,"15min":11.539999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.91200000000000003,"5min":1.454,"15min":1.359},"min":{"1min":0.28999999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.5640000000000001,"5min":10.566000000000001,"15min":11.596},"last":0.40600000000000003}]},{"osd":1,"last update":"Fri Feb 9 17:55:04 2024","interfaces":[{"interface":"back","average":{"1min":1.5309999999999999,"5min":2.278,"15min":1.7130000000000001},"min":{"1min":0.252,"5min":0.24099999999999999,"15min":0.23599999999999999},"max":{"1min":9.6170000000000009,"5min":61.031999999999996,"15min":61.031999999999996},"last":0.45300000000000001},{"interface":"front","average":{"1min":1.514,"5min":2.7530000000000001,"15min":1.7969999999999999},"min":{"1min":0.33400000000000002,"5min":0.28699999999999998,"15min":0.249},"max":{"1min":9.9039999999999999,"5min":61.970999999999997,"15min":61.970999999999997},"last":0.621}]},{"osd":2,"last update":"Fri Feb 9 17:54:29 2024","interfaces":[{"interface":"back","average":{"1min":2.2290000000000001,"5min":2.044,"15min":1.853},"min":{"1min":0.33800000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":7.1420000000000003,"5min":10.657999999999999,"15min":10.657999999999999},"last":0.93899999999999995},{"interface":"front","average":{"1min":3.2850000000000001,"5min":1.7929999999999999,"15min":1.71},"min":{"1min":0.32800000000000001,"5min":0.248,"15min":0.248},"max":{"1min":11.35,"5min":11.35,"15min":11.35},"last":0.54800000000000004}]},{"osd":3,"last update":"Fri Feb 9 17:55:31 2024","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":2.1320000000000001,"15min":1.496},"min":{"1min":0.40899999999999997,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":10.064,"5min":10.537000000000001,"15min":10.537000000000001},"last":0.65700000000000003},{"interface":"front","average":{"1min":1.581,"5min":2.2090000000000001,"15min":1.149},"min":{"1min":0.377,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":10.093,"5min":10.388,"15min":10.388},"last":0.49399999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:55:04 2024","interfaces":[{"interface":"back","average":{"1min":2.258,"5min":1.871,"15min":1.538},"min":{"1min":0.33600000000000002,"5min":0.32900000000000001,"15min":0.26800000000000002},"max":{"1min":19.414000000000001,"5min":19.414000000000001,"15min":19.414000000000001},"last":0.84399999999999997},{"interface":"front","average":{"1min":2.3180000000000001,"5min":1.623,"15min":1.286},"min":{"1min":0.32400000000000001,"5min":0.30299999999999999,"15min":0.26900000000000002},"max":{"1min":19.224,"5min":19.224,"15min":19.224},"last":0.64700000000000002}]},{"osd":5,"last update":"Fri Feb 9 17:55:14 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.506,"15min":1.6799999999999999},"min":{"1min":0.439,"5min":0.35399999999999998,"15min":0.26900000000000002},"max":{"1min":9.7729999999999997,"5min":9.7729999999999997,"15min":77.924000000000007},"last":0.88500000000000001},{"interface":"front","average":{"1min":1.034,"5min":1.2909999999999999,"15min":1.673},"min":{"1min":0.40500000000000003,"5min":0.35299999999999998,"15min":0.28000000000000003},"max":{"1min":4.4480000000000004,"5min":10.643000000000001,"15min":77.899000000000001},"last":0.86799999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:55:04 2024","interfaces":[{"interface":"back","average":{"1min":2.0990000000000002,"5min":1.542,"15min":1.546},"min":{"1min":0.39500000000000002,"5min":0.33400000000000002,"15min":0.30199999999999999},"max":{"1min":19.375,"5min":19.375,"15min":19.375},"last":0.83099999999999996},{"interface":"front","average":{"1min":2.194,"5min":1.5620000000000001,"15min":1.7310000000000001},"min":{"1min":0.53700000000000003,"5min":0.34999999999999998,"15min":0.33500000000000002},"max":{"1min":19.178000000000001,"5min":19.178000000000001,"15min":19.178000000000001},"last":0.69899999999999995}]},{"osd":7,"last update":"Fri Feb 9 17:49:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.9670000000000001,"15min":1.8400000000000001},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.33400000000000002},"max":{"1min":3.7909999999999999,"5min":29.001999999999999,"15min":29.001999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":1.127,"5min":1.919,"15min":1.841},"min":{"1min":0.39400000000000002,"5min":0.373,"15min":0.32200000000000001},"max":{"1min":3.7749999999999999,"5min":28.872,"15min":28.872},"last":0.47399999999999998}]},{"osd":8,"last update":"Fri Feb 9 17:55:17 2024","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.677,"15min":1.1879999999999999},"min":{"1min":0.38600000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":9.4930000000000003,"5min":10.744,"15min":10.744},"last":0.63600000000000001},{"interface":"front","average":{"1min":1.46,"5min":1.7430000000000001,"15min":1.3799999999999999},"min":{"1min":0.52500000000000002,"5min":0.35499999999999998,"15min":0.32200000000000001},"max":{"1min":9.5289999999999999,"5min":10.768000000000001,"15min":12.885999999999999},"last":0.80800000000000005}]},{"osd":10,"last update":"Fri Feb 9 17:55:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":2.3279999999999998,"15min":1.675},"min":{"1min":0.50800000000000001,"5min":0.372,"15min":0.312},"max":{"1min":9.8109999999999999,"5min":20.773,"15min":20.773},"last":0.73999999999999999},{"interface":"front","average":{"1min":1.284,"5min":1.78,"15min":1.2889999999999999},"min":{"1min":0.39000000000000001,"5min":0.34699999999999998,"15min":0.34300000000000003},"max":{"1min":9.5879999999999992,"5min":20.637,"15min":20.637},"last":0.52400000000000002}]},{"osd":11,"last update":"Fri Feb 9 17:55:31 2024","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":2.1190000000000002,"15min":1.4870000000000001},"min":{"1min":0.47799999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":9.9789999999999992,"5min":11.324,"15min":11.324},"last":0.68600000000000005},{"interface":"front","average":{"1min":1.524,"5min":1.7829999999999999,"15min":1.331},"min":{"1min":0.58799999999999997,"5min":0.39000000000000001,"15min":0.36499999999999999},"max":{"1min":6.101,"5min":11.388,"15min":11.388},"last":0.67100000000000004}]}]},{"osd":8,"up_from":171,"seq":734439407945,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":590848,"kb_used_data":192816,"kb_used_omap":212,"kb_used_meta":397803,"kb_avail":93154304,"statfs":{"total":95995035648,"available":95390007296,"internally_reserved":0,"allocated":197443584,"data_stored":197013223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217579,"internal_metadata":407350805},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:55:09 2024","interfaces":[{"interface":"back","average":{"1min":2.2050000000000001,"5min":2.0430000000000001,"15min":1.2929999999999999},"min":{"1min":0.254,"5min":0.249,"15min":0.217},"max":{"1min":10.832000000000001,"5min":12.712999999999999,"15min":12.712999999999999},"last":0.437},{"interface":"front","average":{"1min":1.212,"5min":1.643,"15min":1.427},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23100000000000001},"max":{"1min":5.0579999999999998,"5min":12.725,"15min":12.725},"last":0.90200000000000002}]},{"osd":1,"last update":"Fri Feb 9 17:55:09 2024","interfaces":[{"interface":"back","average":{"1min":2.1139999999999999,"5min":1.792,"15min":1.4430000000000001},"min":{"1min":0.25800000000000001,"5min":0.23799999999999999,"15min":0.22700000000000001},"max":{"1min":7.6989999999999998,"5min":8.2550000000000008,"15min":8.718},"last":0.41799999999999998},{"interface":"front","average":{"1min":2.6219999999999999,"5min":2.0369999999999999,"15min":1.6579999999999999},"min":{"1min":0.311,"5min":0.27100000000000002,"15min":0.24099999999999999},"max":{"1min":21.399000000000001,"5min":21.399000000000001,"15min":25.027000000000001},"last":0.94399999999999995}]},{"osd":2,"last update":"Fri Feb 9 17:50:34 2024","interfaces":[{"interface":"back","average":{"1min":2.222,"5min":1.722,"15min":1.391},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28000000000000003},"max":{"1min":12.227,"5min":12.227,"15min":12.227},"last":0.45300000000000001},{"interface":"front","average":{"1min":2.5049999999999999,"5min":1.665,"15min":1.1240000000000001},"min":{"1min":0.42199999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":12.747999999999999,"5min":12.747999999999999,"15min":12.747999999999999},"last":1.0429999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:55:19 2024","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":2.8889999999999998,"15min":2.9159999999999999},"min":{"1min":0.32400000000000001,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":4.5490000000000004,"5min":15.654,"15min":15.654},"last":0.246},{"interface":"front","average":{"1min":2.0790000000000002,"5min":2.427,"15min":2.0529999999999999},"min":{"1min":0.378,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":14.35,"5min":15.609999999999999,"15min":15.609999999999999},"last":0.34300000000000003}]},{"osd":4,"last update":"Fri Feb 9 17:55:09 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":1.1419999999999999,"15min":1.242},"min":{"1min":0.36499999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":4.2729999999999997,"5min":8.7959999999999994,"15min":8.7959999999999994},"last":1.016},{"interface":"front","average":{"1min":0.81599999999999995,"5min":1.042,"15min":1.4279999999999999},"min":{"1min":0.36099999999999999,"5min":0.28000000000000003,"15min":0.27000000000000002},"max":{"1min":4.3079999999999998,"5min":8.8780000000000001,"15min":9.6140000000000008},"last":1.105}]},{"osd":5,"last update":"Fri Feb 9 17:54:32 2024","interfaces":[{"interface":"back","average":{"1min":1.591,"5min":1.7949999999999999,"15min":1.7989999999999999},"min":{"1min":0.30399999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":9.3550000000000004,"5min":9.657,"15min":9.657},"last":0.28399999999999997},{"interface":"front","average":{"1min":1.929,"5min":1.278,"15min":1.5669999999999999},"min":{"1min":0.36299999999999999,"5min":0.30299999999999999,"15min":0.29899999999999999},"max":{"1min":9.375,"5min":9.375,"15min":9.375},"last":0.98499999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:55:09 2024","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.71,"15min":1.6379999999999999},"min":{"1min":0.34499999999999997,"5min":0.313,"15min":0.29899999999999999},"max":{"1min":9.7539999999999996,"5min":9.7539999999999996,"15min":13.031000000000001},"last":0.36599999999999999},{"interface":"front","average":{"1min":1.004,"5min":1.4470000000000001,"15min":1.52},"min":{"1min":0.311,"5min":0.311,"15min":0.29599999999999999},"max":{"1min":4.0970000000000004,"5min":10.178000000000001,"15min":12.157999999999999},"last":0.46899999999999997}]},{"osd":7,"last update":"Fri Feb 9 17:55:09 2024","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.6259999999999999,"15min":1.7210000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":9.3130000000000006,"5min":9.3130000000000006,"15min":11.278},"last":1.075},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.3580000000000001,"15min":1.7030000000000001},"min":{"1min":0.45400000000000001,"5min":0.371,"15min":0.371},"max":{"1min":9.7110000000000003,"5min":9.7110000000000003,"15min":11.313000000000001},"last":0.309}]},{"osd":9,"last update":"Fri Feb 9 17:55:09 2024","interfaces":[{"interface":"back","average":{"1min":1.5680000000000001,"5min":2.1909999999999998,"15min":1.581},"min":{"1min":0.373,"5min":0.33800000000000002,"15min":0.26800000000000002},"max":{"1min":5.548,"5min":16.358000000000001,"15min":45.616},"last":0.32700000000000001},{"interface":"front","average":{"1min":1.6799999999999999,"5min":2.2679999999999998,"15min":1.6399999999999999},"min":{"1min":0.48399999999999999,"5min":0.376,"15min":0.314},"max":{"1min":5.577,"5min":16.422999999999998,"15min":45.718000000000004},"last":0.40600000000000003}]},{"osd":10,"last update":"Fri Feb 9 17:55:09 2024","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.645,"15min":1.196},"min":{"1min":0.34799999999999998,"5min":0.34599999999999997,"15min":0.29699999999999999},"max":{"1min":4.1340000000000003,"5min":9.157,"15min":9.157},"last":0.50700000000000001},{"interface":"front","average":{"1min":1.135,"5min":2.1720000000000002,"15min":1.4670000000000001},"min":{"1min":0.51900000000000002,"5min":0.38700000000000001,"15min":0.34300000000000003},"max":{"1min":3.4089999999999998,"5min":13.096,"15min":13.096},"last":2.4860000000000002}]},{"osd":11,"last update":"Fri Feb 9 17:55:09 2024","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":2.262,"15min":1.5},"min":{"1min":0.46000000000000002,"5min":0.36099999999999999,"15min":0.34000000000000002},"max":{"1min":5.3029999999999999,"5min":16.466999999999999,"15min":16.466999999999999},"last":0.68799999999999994},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.696,"15min":1.357},"min":{"1min":0.39700000000000002,"5min":0.33900000000000002,"15min":0.308},"max":{"1min":5.2850000000000001,"5min":20.831,"15min":20.831},"last":0.437}]}]},{"osd":7,"up_from":57,"seq":244813136290,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":815720,"kb_used_data":94560,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":92929432,"statfs":{"total":95995035648,"available":95159738368,"internally_reserved":0,"allocated":96829440,"data_stored":96404449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":738458058},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:54:44 2024","interfaces":[{"interface":"back","average":{"1min":2.2029999999999998,"5min":2.181,"15min":1.7230000000000001},"min":{"1min":0.29999999999999999,"5min":0.28000000000000003,"15min":0.249},"max":{"1min":7.032,"5min":101.167,"15min":101.167},"last":0.86499999999999999},{"interface":"front","average":{"1min":1.998,"5min":1.9990000000000001,"15min":1.7170000000000001},"min":{"1min":0.30299999999999999,"5min":0.29399999999999998,"15min":0.24099999999999999},"max":{"1min":7.0170000000000003,"5min":101.07599999999999,"15min":101.07599999999999},"last":0.376}]},{"osd":1,"last update":"Fri Feb 9 17:54:44 2024","interfaces":[{"interface":"back","average":{"1min":3.75,"5min":1.774,"15min":1.8009999999999999},"min":{"1min":0.29599999999999999,"5min":0.29199999999999998,"15min":0.24199999999999999},"max":{"1min":17.314,"5min":17.314,"15min":17.314},"last":0.35399999999999998},{"interface":"front","average":{"1min":2.331,"5min":1.448,"15min":1.8280000000000001},"min":{"1min":0.38400000000000001,"5min":0.27000000000000002,"15min":0.26300000000000001},"max":{"1min":7.5949999999999998,"5min":11.673,"15min":15.276},"last":0.51500000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:54:44 2024","interfaces":[{"interface":"back","average":{"1min":2.3889999999999998,"5min":1.413,"15min":1.714},"min":{"1min":0.34399999999999997,"5min":0.30599999999999999,"15min":0.25},"max":{"1min":9.8279999999999994,"5min":9.8279999999999994,"15min":9.8279999999999994},"last":0.41399999999999998},{"interface":"front","average":{"1min":2.464,"5min":1.627,"15min":1.956},"min":{"1min":0.432,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":5.968,"5min":8.3889999999999993,"15min":11.968},"last":0.83599999999999997}]},{"osd":3,"last update":"Fri Feb 9 17:55:05 2024","interfaces":[{"interface":"back","average":{"1min":2.093,"5min":1.7270000000000001,"15min":2.242},"min":{"1min":0.39900000000000002,"5min":0.32800000000000001,"15min":0.25600000000000001},"max":{"1min":9.1430000000000007,"5min":11.112,"15min":78.644999999999996},"last":1.2350000000000001},{"interface":"front","average":{"1min":3.1299999999999999,"5min":1.7829999999999999,"15min":2.3660000000000001},"min":{"1min":0.52300000000000002,"5min":0.34899999999999998,"15min":0.32600000000000001},"max":{"1min":14.592000000000001,"5min":14.592000000000001,"15min":78.414000000000001},"last":0.67000000000000004}]},{"osd":4,"last update":"Fri Feb 9 17:54:44 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.85499999999999998,"15min":1.4590000000000001},"min":{"1min":0.30499999999999999,"5min":0.25900000000000001,"15min":0.23699999999999999},"max":{"1min":4.274,"5min":4.7270000000000003,"15min":86.813000000000002},"last":1.042},{"interface":"front","average":{"1min":0.746,"5min":1.0129999999999999,"15min":1.669},"min":{"1min":0.29499999999999998,"5min":0.26300000000000001,"15min":0.23899999999999999},"max":{"1min":2.028,"5min":11.433,"15min":86.751000000000005},"last":0.44400000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:54:44 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.0229999999999999,"15min":1.4930000000000001},"min":{"1min":0.40100000000000002,"5min":0.33100000000000002,"15min":0.249},"max":{"1min":4.0469999999999997,"5min":4.6539999999999999,"15min":9.6590000000000007},"last":0.53300000000000003},{"interface":"front","average":{"1min":1.244,"5min":1.129,"15min":1.631},"min":{"1min":0.41999999999999998,"5min":0.35099999999999998,"15min":0.31900000000000001},"max":{"1min":4.0679999999999996,"5min":6.9379999999999997,"15min":15.228999999999999},"last":0.63800000000000001}]},{"osd":6,"last update":"Fri Feb 9 17:55:10 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":1.032,"15min":1.623},"min":{"1min":0.42199999999999999,"5min":0.33200000000000002,"15min":0.28100000000000003},"max":{"1min":2.081,"5min":6.3520000000000003,"15min":15.275},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.94999999999999996,"5min":0.92500000000000004,"15min":1.4510000000000001},"min":{"1min":0.433,"5min":0.36599999999999999,"15min":0.29199999999999998},"max":{"1min":2.5569999999999999,"5min":4.6980000000000004,"15min":18.236999999999998},"last":0.59099999999999997}]},{"osd":8,"last update":"Fri Feb 9 17:55:05 2024","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":1.228,"15min":1.4430000000000001},"min":{"1min":0.45300000000000001,"5min":0.39600000000000002,"15min":0.28999999999999998},"max":{"1min":4.3250000000000002,"5min":9.2249999999999996,"15min":11.81},"last":0.88500000000000001},{"interface":"front","average":{"1min":1.04,"5min":1.3089999999999999,"15min":1.49},"min":{"1min":0.375,"5min":0.37,"15min":0.36599999999999999},"max":{"1min":4.3369999999999997,"5min":8.3949999999999996,"15min":12.058},"last":0.74199999999999999}]},{"osd":9,"last update":"Fri Feb 9 17:54:53 2024","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.7470000000000001,"15min":1.6619999999999999},"min":{"1min":0.53300000000000003,"5min":0.38900000000000001,"15min":0.32700000000000001},"max":{"1min":4.4029999999999996,"5min":8.8119999999999994,"15min":11.941000000000001},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.95399999999999996,"5min":1.659,"15min":1.732},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.379},"max":{"1min":4.4459999999999997,"5min":15.852,"15min":15.852},"last":1.4099999999999999}]},{"osd":10,"last update":"Fri Feb 9 17:55:25 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":1.516,"15min":1.458},"min":{"1min":0.498,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":3.0139999999999998,"5min":9.5350000000000001,"15min":9.5350000000000001},"last":0.93600000000000005},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.889,"15min":1.7969999999999999},"min":{"1min":0.59999999999999998,"5min":0.40400000000000003,"15min":0.38900000000000001},"max":{"1min":4.4240000000000004,"5min":10.263999999999999,"15min":11.878},"last":1.4430000000000001}]},{"osd":11,"last update":"Fri Feb 9 17:54:44 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":1.6040000000000001,"15min":1.5820000000000001},"min":{"1min":0.375,"5min":0.375,"15min":0.35999999999999999},"max":{"1min":4.1459999999999999,"5min":11.961,"15min":11.961},"last":0.65400000000000003},{"interface":"front","average":{"1min":2.4249999999999998,"5min":1.895,"15min":1.7110000000000001},"min":{"1min":0.49099999999999999,"5min":0.44,"15min":0.438},"max":{"1min":12.243,"5min":12.243,"15min":12.423999999999999},"last":0.71299999999999997}]}]},{"osd":6,"up_from":49,"seq":210453397928,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":678184,"kb_used_data":1432,"kb_used_omap":1,"kb_used_meta":676734,"kb_avail":93066968,"statfs":{"total":95995035648,"available":95300575232,"internally_reserved":0,"allocated":1466368,"data_stored":1044929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":692976070},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:54:34 2024","interfaces":[{"interface":"back","average":{"1min":2.633,"5min":1.395,"15min":1.585},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.22800000000000001},"max":{"1min":10.356,"5min":10.356,"15min":10.481999999999999},"last":0.48399999999999999},{"interface":"front","average":{"1min":1.4550000000000001,"5min":1.5009999999999999,"15min":1.677},"min":{"1min":0.28000000000000003,"5min":0.27400000000000002,"15min":0.24399999999999999},"max":{"1min":4.298,"5min":10.436,"15min":10.436},"last":0.53600000000000003}]},{"osd":1,"last update":"Fri Feb 9 17:54:34 2024","interfaces":[{"interface":"back","average":{"1min":1.6140000000000001,"5min":1.494,"15min":2.327},"min":{"1min":0.29099999999999998,"5min":0.25,"15min":0.25},"max":{"1min":6.9800000000000004,"5min":9.593,"15min":79.945999999999998},"last":0.68400000000000005},{"interface":"front","average":{"1min":1.675,"5min":1.5429999999999999,"15min":2.4049999999999998},"min":{"1min":0.42699999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":7.0170000000000003,"5min":9.6329999999999991,"15min":80.003},"last":0.70499999999999996}]},{"osd":2,"last update":"Fri Feb 9 17:54:32 2024","interfaces":[{"interface":"back","average":{"1min":1.9670000000000001,"5min":1.5660000000000001,"15min":2.1400000000000001},"min":{"1min":0.31900000000000001,"5min":0.29999999999999999,"15min":0.28100000000000003},"max":{"1min":5.657,"5min":9.1080000000000005,"15min":84.912999999999997},"last":1.1020000000000001},{"interface":"front","average":{"1min":2.0249999999999999,"5min":1.6240000000000001,"15min":2.1829999999999998},"min":{"1min":0.34999999999999998,"5min":0.34599999999999997,"15min":0.34000000000000002},"max":{"1min":5.6849999999999996,"5min":9.0850000000000009,"15min":84.942999999999998},"last":1.1399999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:55:07 2024","interfaces":[{"interface":"back","average":{"1min":1.8240000000000001,"5min":1.5389999999999999,"15min":1.4910000000000001},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":9.0129999999999999,"5min":9.0129999999999999,"15min":9.0129999999999999},"last":0.54500000000000004},{"interface":"front","average":{"1min":1.841,"5min":1.8169999999999999,"15min":1.8129999999999999},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":6.6769999999999996,"5min":8.9410000000000007,"15min":8.9410000000000007},"last":0.48399999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:54:32 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.0509999999999999,"15min":1.7729999999999999},"min":{"1min":0.28399999999999997,"5min":0.26200000000000001,"15min":0.23599999999999999},"max":{"1min":8.1349999999999998,"5min":8.1349999999999998,"15min":15.061999999999999},"last":0.33200000000000002},{"interface":"front","average":{"1min":1.242,"5min":1.1140000000000001,"15min":1.883},"min":{"1min":0.33600000000000002,"5min":0.314,"15min":0.26400000000000001},"max":{"1min":8.1809999999999992,"5min":8.1809999999999992,"15min":15.118},"last":0.443}]},{"osd":5,"last update":"Fri Feb 9 17:54:32 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.2509999999999999,"15min":1.8500000000000001},"min":{"1min":0.29599999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":6.5800000000000001,"5min":7.4930000000000003,"15min":14.151999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.78100000000000003,"5min":1.1910000000000001,"15min":1.794},"min":{"1min":0.35799999999999998,"5min":0.32800000000000001,"15min":0.26100000000000001},"max":{"1min":3.8450000000000002,"5min":8.9629999999999992,"15min":18.315999999999999},"last":0.79800000000000004}]},{"osd":7,"last update":"Fri Feb 9 17:55:10 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":1.0489999999999999,"15min":1.8029999999999999},"min":{"1min":0.33000000000000002,"5min":0.29499999999999998,"15min":0.28699999999999998},"max":{"1min":4.2119999999999997,"5min":6.343,"15min":23.265999999999998},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.91000000000000003,"15min":1.5660000000000001},"min":{"1min":0.38,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":4.2779999999999996,"5min":5.0670000000000002,"15min":12.967000000000001},"last":0.378}]},{"osd":8,"last update":"Fri Feb 9 17:55:24 2024","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":1.258,"15min":1.048},"min":{"1min":0.438,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":4.6280000000000001,"5min":9.9480000000000004,"15min":9.9480000000000004},"last":1.1180000000000001},{"interface":"front","average":{"1min":0.80000000000000004,"5min":1.7789999999999999,"15min":1.0629999999999999},"min":{"1min":0.54200000000000004,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.3,"5min":11.316000000000001,"15min":11.316000000000001},"last":1.173}]},{"osd":9,"last update":"Fri Feb 9 17:54:38 2024","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.6830000000000001,"15min":1.675},"min":{"1min":0.41799999999999998,"5min":0.38700000000000001,"15min":0.32600000000000001},"max":{"1min":8.4629999999999992,"5min":10.58,"15min":10.779999999999999},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.80000000000000004,"5min":2.0230000000000001,"15min":1.827},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":3.173,"5min":10.609,"15min":10.609},"last":1.0389999999999999}]},{"osd":10,"last update":"Fri Feb 9 17:55:10 2024","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.6659999999999999,"15min":1.776},"min":{"1min":0.41599999999999998,"5min":0.36099999999999999,"15min":0.34300000000000003},"max":{"1min":4.6100000000000003,"5min":15.701000000000001,"15min":25.370000000000001},"last":0.64900000000000002},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.5820000000000001,"15min":1.8380000000000001},"min":{"1min":0.45300000000000001,"5min":0.40100000000000002,"15min":0.35099999999999998},"max":{"1min":4.2480000000000002,"5min":6.0190000000000001,"15min":26.027999999999999},"last":0.69299999999999995}]},{"osd":11,"last update":"Fri Feb 9 17:54:32 2024","interfaces":[{"interface":"back","average":{"1min":1.9039999999999999,"5min":1.7609999999999999,"15min":1.7769999999999999},"min":{"1min":0.44400000000000001,"5min":0.40200000000000002,"15min":0.372},"max":{"1min":4.9820000000000002,"5min":10.176,"15min":13.109},"last":1.022},{"interface":"front","average":{"1min":1.9419999999999999,"5min":1.863,"15min":1.804},"min":{"1min":0.48699999999999999,"5min":0.44500000000000001,"15min":0.373},"max":{"1min":5.0410000000000004,"5min":10.788,"15min":13.164999999999999},"last":0.98699999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575297,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1120036,"kb_used_data":258648,"kb_used_omap":1,"kb_used_meta":861374,"kb_avail":92625116,"statfs":{"total":95995035648,"available":94848118784,"internally_reserved":0,"allocated":264855552,"data_stored":264434113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":882047429},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:54:51 2024","interfaces":[{"interface":"back","average":{"1min":3.9260000000000002,"5min":2.0379999999999998,"15min":1.583},"min":{"1min":0.23300000000000001,"5min":0.19,"15min":0.19},"max":{"1min":14.885999999999999,"5min":14.885999999999999,"15min":14.885999999999999},"last":0.502},{"interface":"front","average":{"1min":4.0119999999999996,"5min":2.415,"15min":1.7869999999999999},"min":{"1min":0.26100000000000001,"5min":0.22,"15min":0.214},"max":{"1min":15.411,"5min":33.389000000000003,"15min":33.389000000000003},"last":0.47599999999999998}]},{"osd":2,"last update":"Fri Feb 9 17:55:15 2024","interfaces":[{"interface":"back","average":{"1min":2.1709999999999998,"5min":1.617,"15min":1.734},"min":{"1min":0.33400000000000002,"5min":0.222,"15min":0.222},"max":{"1min":9.2240000000000002,"5min":13.173,"15min":14.83},"last":0.63800000000000001},{"interface":"front","average":{"1min":2.2890000000000001,"5min":1.5109999999999999,"15min":1.5800000000000001},"min":{"1min":0.378,"5min":0.29299999999999998,"15min":0.222},"max":{"1min":10.4,"5min":11.103,"15min":11.103},"last":0.54700000000000004}]},{"osd":3,"last update":"Fri Feb 9 17:55:04 2024","interfaces":[{"interface":"back","average":{"1min":3.085,"5min":3.657,"15min":3.7519999999999998},"min":{"1min":0.37,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":16.835000000000001,"5min":16.835000000000001,"15min":16.835000000000001},"last":0.81000000000000005},{"interface":"front","average":{"1min":3.4260000000000002,"5min":4.1980000000000004,"15min":4.3259999999999996},"min":{"1min":0.23999999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":15.337999999999999,"5min":15.337999999999999,"15min":15.337999999999999},"last":0.57599999999999996}]},{"osd":4,"last update":"Fri Feb 9 17:54:51 2024","interfaces":[{"interface":"back","average":{"1min":1.8140000000000001,"5min":1.2450000000000001,"15min":1.6319999999999999},"min":{"1min":0.36799999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":9.032,"5min":9.032,"15min":45.137999999999998},"last":1.2},{"interface":"front","average":{"1min":3.1699999999999999,"5min":1.9039999999999999,"15min":1.9159999999999999},"min":{"1min":0.28199999999999997,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":15.331,"5min":15.331,"15min":44.950000000000003},"last":0.73299999999999998}]},{"osd":5,"last update":"Fri Feb 9 17:55:29 2024","interfaces":[{"interface":"back","average":{"1min":1.669,"5min":1.5740000000000001,"15min":1.786},"min":{"1min":0.41999999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":15.106999999999999,"5min":15.106999999999999,"15min":45.171999999999997},"last":0.87},{"interface":"front","average":{"1min":1.742,"5min":1.8819999999999999,"15min":2.036},"min":{"1min":0.34999999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":15.355,"5min":15.355,"15min":45.127000000000002},"last":0.84999999999999998}]},{"osd":6,"last update":"Fri Feb 9 17:54:42 2024","interfaces":[{"interface":"back","average":{"1min":2.238,"5min":2.1699999999999999,"15min":1.9419999999999999},"min":{"1min":0.39800000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":15.154999999999999,"5min":33.338000000000001,"15min":33.338000000000001},"last":0.90700000000000003},{"interface":"front","average":{"1min":2.4870000000000001,"5min":2.2170000000000001,"15min":1.998},"min":{"1min":0.38100000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":15.253,"5min":33.286999999999999,"15min":33.286999999999999},"last":1.0029999999999999}]},{"osd":7,"last update":"Fri Feb 9 17:54:42 2024","interfaces":[{"interface":"back","average":{"1min":2.476,"5min":2.1859999999999999,"15min":2.0640000000000001},"min":{"1min":0.42599999999999999,"5min":0.38900000000000001,"15min":0.34399999999999997},"max":{"1min":15.287000000000001,"5min":33.314999999999998,"15min":45.740000000000002},"last":1.0580000000000001},{"interface":"front","average":{"1min":2.5190000000000001,"5min":2.2250000000000001,"15min":2.141},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.309},"max":{"1min":15.311,"5min":33.363999999999997,"15min":45.792000000000002},"last":1.091}]},{"osd":8,"last update":"Fri Feb 9 17:54:51 2024","interfaces":[{"interface":"back","average":{"1min":2.3519999999999999,"5min":1.7749999999999999,"15min":1.456},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.35599999999999998},"max":{"1min":9.1999999999999993,"5min":9.1999999999999993,"15min":9.2520000000000007},"last":1.3140000000000001},{"interface":"front","average":{"1min":2.3690000000000002,"5min":2.21,"15min":1.627},"min":{"1min":0.45500000000000002,"5min":0.43099999999999999,"15min":0.35599999999999998},"max":{"1min":9.0909999999999993,"5min":12.464,"15min":12.69},"last":1.2629999999999999}]},{"osd":9,"last update":"Fri Feb 9 17:54:51 2024","interfaces":[{"interface":"back","average":{"1min":3.601,"5min":2.089,"15min":1.659},"min":{"1min":0.69699999999999995,"5min":0.42799999999999999,"15min":0.33900000000000002},"max":{"1min":20.789000000000001,"5min":20.789000000000001,"15min":20.789000000000001},"last":1.4710000000000001},{"interface":"front","average":{"1min":2.7639999999999998,"5min":2.0830000000000002,"15min":1.7749999999999999},"min":{"1min":0.438,"5min":0.39400000000000002,"15min":0.34899999999999998},"max":{"1min":9.1210000000000004,"5min":9.5239999999999991,"15min":9.5239999999999991},"last":1.417}]},{"osd":10,"last update":"Fri Feb 9 17:55:29 2024","interfaces":[{"interface":"back","average":{"1min":1.7090000000000001,"5min":2.0910000000000002,"15min":1.879},"min":{"1min":0.55800000000000005,"5min":0.47099999999999997,"15min":0.39200000000000002},"max":{"1min":15.211,"5min":15.211,"15min":33.213999999999999},"last":0.93799999999999994},{"interface":"front","average":{"1min":2.04,"5min":2.4009999999999998,"15min":1.841},"min":{"1min":0.55000000000000004,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":15.271000000000001,"5min":15.271000000000001,"15min":15.271000000000001},"last":0.94899999999999995}]},{"osd":11,"last update":"Fri Feb 9 17:54:42 2024","interfaces":[{"interface":"back","average":{"1min":3.5699999999999998,"5min":3.0600000000000001,"15min":2.052},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.39100000000000001},"max":{"1min":15.301,"5min":19.648,"15min":19.648},"last":1.1140000000000001},{"interface":"front","average":{"1min":3.2869999999999999,"5min":3.004,"15min":2.0779999999999998},"min":{"1min":0.51900000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":15.185,"5min":33.161000000000001,"15min":33.161000000000001},"last":1.294}]}]},{"osd":0,"up_from":8,"seq":34359738823,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1151680,"kb_used_data":344432,"kb_used_omap":1,"kb_used_meta":807230,"kb_avail":92593472,"statfs":{"total":95995035648,"available":94815715328,"internally_reserved":0,"allocated":352698368,"data_stored":352268007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":826603973},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 9 17:54:36 2024","interfaces":[{"interface":"back","average":{"1min":3.633,"5min":1.752,"15min":1.6160000000000001},"min":{"1min":0.26600000000000001,"5min":0.23699999999999999,"15min":0.20599999999999999},"max":{"1min":18.698,"5min":18.698,"15min":18.698},"last":0.28199999999999997},{"interface":"front","average":{"1min":3.714,"5min":1.8260000000000001,"15min":1.702},"min":{"1min":0.36799999999999999,"5min":0.32100000000000001,"15min":0.24399999999999999},"max":{"1min":18.742999999999999,"5min":18.742999999999999,"15min":18.742999999999999},"last":0.44900000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:54:58 2024","interfaces":[{"interface":"back","average":{"1min":4.5289999999999999,"5min":1.966,"15min":2.0230000000000001},"min":{"1min":0.30099999999999999,"5min":0.27700000000000002,"15min":0.245},"max":{"1min":20.013999999999999,"5min":20.013999999999999,"15min":20.013999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":4.2279999999999998,"5min":2.165,"15min":1.7350000000000001},"min":{"1min":0.28100000000000003,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":20.103999999999999,"5min":20.103999999999999,"15min":20.103999999999999},"last":0.307}]},{"osd":3,"last update":"Fri Feb 9 17:55:22 2024","interfaces":[{"interface":"back","average":{"1min":2.1400000000000001,"5min":2.4700000000000002,"15min":1.9470000000000001},"min":{"1min":0.442,"5min":0.26100000000000001,"15min":0.23100000000000001},"max":{"1min":8.827,"5min":20.469999999999999,"15min":36.052},"last":0.38100000000000001},{"interface":"front","average":{"1min":2.2669999999999999,"5min":2.6429999999999998,"15min":2.1640000000000001},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.217},"max":{"1min":10.414999999999999,"5min":21.920000000000002,"15min":38.383000000000003},"last":0.36599999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:55:06 2024","interfaces":[{"interface":"back","average":{"1min":2.4350000000000001,"5min":1.766,"15min":1.498},"min":{"1min":0.35299999999999998,"5min":0.28100000000000003,"15min":0.27000000000000002},"max":{"1min":12.757,"5min":12.757,"15min":12.757},"last":0.48699999999999999},{"interface":"front","average":{"1min":2.9820000000000002,"5min":1.8100000000000001,"15min":1.873},"min":{"1min":0.35899999999999999,"5min":0.249,"15min":0.249},"max":{"1min":12.82,"5min":17.091999999999999,"15min":17.091999999999999},"last":0.32900000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:55:22 2024","interfaces":[{"interface":"back","average":{"1min":1.746,"5min":1.407,"15min":1.79},"min":{"1min":0.45700000000000002,"5min":0.34300000000000003,"15min":0.30299999999999999},"max":{"1min":8.7710000000000008,"5min":9.7929999999999993,"15min":48.924999999999997},"last":0.52000000000000002},{"interface":"front","average":{"1min":1.782,"5min":1.46,"15min":1.8180000000000001},"min":{"1min":0.52400000000000002,"5min":0.38200000000000001,"15min":0.29599999999999999},"max":{"1min":8.798,"5min":9.8230000000000004,"15min":48.969999999999999},"last":0.55200000000000005}]},{"osd":6,"last update":"Fri Feb 9 17:54:47 2024","interfaces":[{"interface":"back","average":{"1min":3.3250000000000002,"5min":2.101,"15min":1.802},"min":{"1min":0.307,"5min":0.307,"15min":0.28699999999999998},"max":{"1min":15.667,"5min":15.667,"15min":15.667},"last":0.41999999999999998},{"interface":"front","average":{"1min":3.3969999999999998,"5min":2.1680000000000001,"15min":1.899},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.33300000000000002},"max":{"1min":15.702999999999999,"5min":15.702999999999999,"15min":15.702999999999999},"last":0.5}]},{"osd":7,"last update":"Fri Feb 9 17:55:13 2024","interfaces":[{"interface":"back","average":{"1min":1.8400000000000001,"5min":1.847,"15min":1.841},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.29199999999999998},"max":{"1min":6.1829999999999998,"5min":11.436,"15min":15.271000000000001},"last":0.46400000000000002},{"interface":"front","average":{"1min":1.704,"5min":1.625,"15min":1.929},"min":{"1min":0.40400000000000003,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":8.2509999999999994,"5min":17.152999999999999,"15min":17.152999999999999},"last":0.53500000000000003}]},{"osd":8,"last update":"Fri Feb 9 17:55:13 2024","interfaces":[{"interface":"back","average":{"1min":2.04,"5min":2.3359999999999999,"15min":1.8500000000000001},"min":{"1min":0.46800000000000003,"5min":0.313,"15min":0.313},"max":{"1min":8.3030000000000008,"5min":17.189,"15min":17.189},"last":0.434},{"interface":"front","average":{"1min":1.7430000000000001,"5min":1.9570000000000001,"15min":1.72},"min":{"1min":0.439,"5min":0.36899999999999999,"15min":0.35099999999999998},"max":{"1min":8.2810000000000006,"5min":17.27,"15min":17.27},"last":1.0600000000000001}]},{"osd":9,"last update":"Fri Feb 9 17:54:58 2024","interfaces":[{"interface":"back","average":{"1min":3.407,"5min":2.6549999999999998,"15min":2.242},"min":{"1min":0.42399999999999999,"5min":0.39900000000000002,"15min":0.33800000000000002},"max":{"1min":11.134,"5min":24.757000000000001,"15min":24.757000000000001},"last":0.47499999999999998},{"interface":"front","average":{"1min":3.5649999999999999,"5min":2.5129999999999999,"15min":2.1699999999999999},"min":{"1min":0.42599999999999999,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":19.986999999999998,"5min":24.681999999999999,"15min":24.681999999999999},"last":0.57599999999999996}]},{"osd":10,"last update":"Fri Feb 9 17:55:22 2024","interfaces":[{"interface":"back","average":{"1min":2.9929999999999999,"5min":2.5590000000000002,"15min":2.097},"min":{"1min":0.42499999999999999,"5min":0.41999999999999998,"15min":0.35499999999999998},"max":{"1min":15.614000000000001,"5min":15.614000000000001,"15min":31.766999999999999},"last":1.373},{"interface":"front","average":{"1min":2.1379999999999999,"5min":2.7679999999999998,"15min":1.905},"min":{"1min":0.51800000000000002,"5min":0.40400000000000003,"15min":0.34200000000000003},"max":{"1min":11.096,"5min":17.242000000000001,"15min":28.048999999999999},"last":0.68200000000000005}]},{"osd":11,"last update":"Fri Feb 9 17:54:47 2024","interfaces":[{"interface":"back","average":{"1min":3.7599999999999998,"5min":2.6469999999999998,"15min":1.861},"min":{"1min":0.61499999999999999,"5min":0.45300000000000001,"15min":0.41599999999999998},"max":{"1min":19.951000000000001,"5min":25.994,"15min":25.994},"last":1.109},{"interface":"front","average":{"1min":4.1870000000000003,"5min":2.6150000000000002,"15min":1.9379999999999999},"min":{"1min":0.57399999999999995,"5min":0.441,"15min":0.441},"max":{"1min":19.916,"5min":25.515999999999998,"15min":25.515999999999998},"last":3.6549999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411771,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":727144,"kb_used_data":214240,"kb_used_omap":1,"kb_used_meta":512894,"kb_avail":93018008,"statfs":{"total":95995035648,"available":95250440192,"internally_reserved":0,"allocated":219381760,"data_stored":218956769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":525203909},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":1.4510000000000001,"15min":1.006},"min":{"1min":0.29999999999999999,"5min":0.23699999999999999,"15min":0.221},"max":{"1min":4.9089999999999998,"5min":17.251000000000001,"15min":17.251000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.74399999999999999,"5min":1.611,"15min":1.113},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.22900000000000001},"max":{"1min":4.7720000000000002,"5min":23.097999999999999,"15min":23.097999999999999},"last":0.875}]},{"osd":1,"last update":"Fri Feb 9 17:55:12 2024","interfaces":[{"interface":"back","average":{"1min":2.238,"5min":1.7929999999999999,"15min":2.089},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":15.378,"5min":15.378,"15min":52.646000000000001},"last":0.38100000000000001},{"interface":"front","average":{"1min":2.198,"5min":1.8919999999999999,"15min":2.0800000000000001},"min":{"1min":0.29799999999999999,"5min":0.248,"15min":0.248},"max":{"1min":8.6440000000000001,"5min":15.092000000000001,"15min":52.683999999999997},"last":0.505}]},{"osd":3,"last update":"Fri Feb 9 17:54:36 2024","interfaces":[{"interface":"back","average":{"1min":4.2619999999999996,"5min":2.9609999999999999,"15min":2.3210000000000002},"min":{"1min":0.254,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":15.862,"5min":46.927999999999997,"15min":46.927999999999997},"last":0.50900000000000001},{"interface":"front","average":{"1min":4.0910000000000002,"5min":2.8559999999999999,"15min":2.2869999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":15.904,"5min":45.798999999999999,"15min":45.798999999999999},"last":0.28999999999999998}]},{"osd":4,"last update":"Fri Feb 9 17:54:36 2024","interfaces":[{"interface":"back","average":{"1min":1.722,"5min":1.3879999999999999,"15min":1.585},"min":{"1min":0.32100000000000001,"5min":0.245,"15min":0.245},"max":{"1min":5.9550000000000001,"5min":10.419,"15min":12.398},"last":0.55800000000000005},{"interface":"front","average":{"1min":1.956,"5min":1.4350000000000001,"15min":1.6140000000000001},"min":{"1min":0.29399999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":10.529,"5min":10.529,"15min":10.888999999999999},"last":0.36499999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:54:31 2024","interfaces":[{"interface":"back","average":{"1min":2.214,"5min":1.8799999999999999,"15min":2.0910000000000002},"min":{"1min":0.28799999999999998,"5min":0.245,"15min":0.245},"max":{"1min":8.8529999999999998,"5min":15.455,"15min":42.664000000000001},"last":0.40799999999999997},{"interface":"front","average":{"1min":1.96,"5min":1.6299999999999999,"15min":2.2530000000000001},"min":{"1min":0.30499999999999999,"5min":0.29699999999999999,"15min":0.27200000000000002},"max":{"1min":10.603,"5min":15.487,"15min":43.311999999999998},"last":0.48899999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:54:36 2024","interfaces":[{"interface":"back","average":{"1min":1.9399999999999999,"5min":1.7390000000000001,"15min":1.929},"min":{"1min":0.29699999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":10.573,"5min":18.518999999999998,"15min":18.518999999999998},"last":0.437},{"interface":"front","average":{"1min":2.1139999999999999,"5min":1.8100000000000001,"15min":1.9490000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":10.731,"5min":18.481999999999999,"15min":18.481999999999999},"last":0.54700000000000004}]},{"osd":7,"last update":"Fri Feb 9 17:55:12 2024","interfaces":[{"interface":"back","average":{"1min":1.4490000000000001,"5min":1.5880000000000001,"15min":1.889},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.32400000000000001},"max":{"1min":8.5700000000000003,"5min":10.933,"15min":16.666},"last":0.82699999999999996},{"interface":"front","average":{"1min":1.167,"5min":1.413,"15min":2.0379999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":4.9630000000000001,"5min":10.316000000000001,"15min":16.638999999999999},"last":0.52800000000000002}]},{"osd":8,"last update":"Fri Feb 9 17:54:36 2024","interfaces":[{"interface":"back","average":{"1min":2.3690000000000002,"5min":2.4199999999999999,"15min":1.833},"min":{"1min":0.432,"5min":0.39900000000000002,"15min":0.316},"max":{"1min":10.755000000000001,"5min":12.500999999999999,"15min":12.500999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":2.0840000000000001,"5min":2.6120000000000001,"15min":1.903},"min":{"1min":0.40999999999999998,"5min":0.36399999999999999,"15min":0.34399999999999997},"max":{"1min":10.629,"5min":54.759999999999998,"15min":54.759999999999998},"last":0.877}]},{"osd":9,"last update":"Fri Feb 9 17:54:48 2024","interfaces":[{"interface":"back","average":{"1min":2.5779999999999998,"5min":2.9630000000000001,"15min":2.1909999999999998},"min":{"1min":0.41299999999999998,"5min":0.318,"15min":0.318},"max":{"1min":8.8109999999999999,"5min":53.935000000000002,"15min":53.935000000000002},"last":0.76700000000000002},{"interface":"front","average":{"1min":2.343,"5min":2.7709999999999999,"15min":2.0169999999999999},"min":{"1min":0.36199999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":10.657,"5min":54.670999999999999,"15min":54.670999999999999},"last":0.60899999999999999}]},{"osd":10,"last update":"Fri Feb 9 17:55:26 2024","interfaces":[{"interface":"back","average":{"1min":1.1279999999999999,"5min":2.4670000000000001,"15min":1.9319999999999999},"min":{"1min":0.46200000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":8.6720000000000006,"5min":54.039999999999999,"15min":54.039999999999999},"last":0.96799999999999997},{"interface":"front","average":{"1min":1.165,"5min":2.5049999999999999,"15min":1.972},"min":{"1min":0.505,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":8.6959999999999997,"5min":54.118000000000002,"15min":54.118000000000002},"last":1.004}]},{"osd":11,"last update":"Fri Feb 9 17:54:36 2024","interfaces":[{"interface":"back","average":{"1min":2.3919999999999999,"5min":2.8159999999999998,"15min":2.1440000000000001},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.40600000000000003},"max":{"1min":10.702,"5min":54.799999999999997,"15min":54.799999999999997},"last":0.72199999999999998},{"interface":"front","average":{"1min":2.5859999999999999,"5min":2.9289999999999998,"15min":2.153},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":8.8800000000000008,"5min":54.103000000000002,"15min":54.103000000000002},"last":0.80800000000000005}]}]},{"osd":3,"up_from":27,"seq":115964117433,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1015104,"kb_used_data":195376,"kb_used_omap":1,"kb_used_meta":819710,"kb_avail":92730048,"statfs":{"total":95995035648,"available":94955569152,"internally_reserved":0,"allocated":200065024,"data_stored":199634663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":839383495},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:55:33 2024","interfaces":[{"interface":"back","average":{"1min":1.8620000000000001,"5min":2.4260000000000002,"15min":1.8999999999999999},"min":{"1min":0.26500000000000001,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":12.391,"5min":17.006,"15min":17.006},"last":0.26500000000000001},{"interface":"front","average":{"1min":1.3580000000000001,"5min":1.694,"15min":1.9430000000000001},"min":{"1min":0.24099999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":6.343,"5min":12.288,"15min":16.637},"last":0.52600000000000002}]},{"osd":1,"last update":"Fri Feb 9 17:55:01 2024","interfaces":[{"interface":"back","average":{"1min":2.306,"5min":1.9710000000000001,"15min":2.6040000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":9.2400000000000002,"5min":12.609,"15min":53.152999999999999},"last":1.069},{"interface":"front","average":{"1min":2.4380000000000002,"5min":1.74,"15min":1.8939999999999999},"min":{"1min":0.24299999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":12.237,"5min":12.237,"15min":53.118000000000002},"last":0.87}]},{"osd":2,"last update":"Fri Feb 9 17:55:33 2024","interfaces":[{"interface":"back","average":{"1min":1.903,"5min":2.2080000000000002,"15min":1.9330000000000001},"min":{"1min":0.27600000000000002,"5min":0.26700000000000002,"15min":0.248},"max":{"1min":12.349,"5min":17.477,"15min":17.477},"last":0.30099999999999999},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.9730000000000001,"15min":1.8720000000000001},"min":{"1min":0.26600000000000001,"5min":0.23699999999999999,"15min":0.23499999999999999},"max":{"1min":7.4059999999999997,"5min":21.103000000000002,"15min":21.103000000000002},"last":1.139}]},{"osd":4,"last update":"Fri Feb 9 17:55:33 2024","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":1.4430000000000001,"15min":1.724},"min":{"1min":0.29299999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":3.7149999999999999,"5min":12.718,"15min":12.718},"last":1.181},{"interface":"front","average":{"1min":0.77200000000000002,"5min":1.2210000000000001,"15min":1.591},"min":{"1min":0.28699999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":5.6280000000000001,"5min":5.8849999999999998,"15min":19.713999999999999},"last":0.47999999999999998}]},{"osd":5,"last update":"Fri Feb 9 17:55:16 2024","interfaces":[{"interface":"back","average":{"1min":1.619,"5min":1.8300000000000001,"15min":2.0550000000000002},"min":{"1min":0.40799999999999997,"5min":0.33000000000000002,"15min":0.28000000000000003},"max":{"1min":5.3559999999999999,"5min":13.321,"15min":14.106},"last":0.55500000000000005},{"interface":"front","average":{"1min":1.6890000000000001,"5min":1.9279999999999999,"15min":2.0430000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.316},"max":{"1min":5.391,"5min":13.347,"15min":14.138999999999999},"last":0.60899999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:55:01 2024","interfaces":[{"interface":"back","average":{"1min":2.9159999999999999,"5min":1.998,"15min":1.9990000000000001},"min":{"1min":0.39600000000000002,"5min":0.35799999999999998,"15min":0.28899999999999998},"max":{"1min":13.207000000000001,"5min":13.207000000000001,"15min":15.061999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":3.0470000000000002,"5min":2.1589999999999998,"15min":1.9830000000000001},"min":{"1min":0.375,"5min":0.375,"15min":0.36399999999999999},"max":{"1min":13.298,"5min":13.298,"15min":13.298},"last":0.66300000000000003}]},{"osd":7,"last update":"Fri Feb 9 17:54:48 2024","interfaces":[{"interface":"back","average":{"1min":2.585,"5min":1.8859999999999999,"15min":1.889},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":18.587,"5min":18.587,"15min":19.788},"last":1.1140000000000001},{"interface":"front","average":{"1min":3.2869999999999999,"5min":2.081,"15min":1.9259999999999999},"min":{"1min":0.45200000000000001,"5min":0.437,"15min":0.34899999999999998},"max":{"1min":13.239000000000001,"5min":13.239000000000001,"15min":13.239000000000001},"last":0.63700000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:45:16 2024","interfaces":[{"interface":"back","average":{"1min":1.714,"5min":1.4450000000000001,"15min":1.498},"min":{"1min":0.377,"5min":0.377,"15min":0.26800000000000002},"max":{"1min":8.6020000000000003,"5min":11.821,"15min":11.821},"last":0.67700000000000005},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.1719999999999999,"15min":1.351},"min":{"1min":0.46500000000000002,"5min":0.36199999999999999,"15min":0.34499999999999997},"max":{"1min":6.8390000000000004,"5min":6.8390000000000004,"15min":14.129},"last":0.78200000000000003}]},{"osd":9,"last update":"Fri Feb 9 17:55:01 2024","interfaces":[{"interface":"back","average":{"1min":2.1680000000000001,"5min":1.9750000000000001,"15min":1.829},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":11.039,"5min":11.417,"15min":11.851000000000001},"last":2.2639999999999998},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.9430000000000001,"15min":1.8500000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":5.6900000000000004,"5min":11.492000000000001,"15min":11.907},"last":0.56499999999999995}]},{"osd":10,"last update":"Fri Feb 9 17:54:48 2024","interfaces":[{"interface":"back","average":{"1min":3.0899999999999999,"5min":2.4790000000000001,"15min":1.7549999999999999},"min":{"1min":0.40999999999999998,"5min":0.371,"15min":0.371},"max":{"1min":18.632999999999999,"5min":18.632999999999999,"15min":18.632999999999999},"last":1.1559999999999999},{"interface":"front","average":{"1min":3.5,"5min":2.411,"15min":1.7929999999999999},"min":{"1min":0.5,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":13.272,"5min":13.272,"15min":19.859000000000002},"last":0.68999999999999995}]},{"osd":11,"last update":"Fri Feb 9 17:55:33 2024","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":2.6070000000000002,"15min":1.899},"min":{"1min":0.44,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":7.2679999999999998,"5min":18.66,"15min":19.824000000000002},"last":1.5569999999999999},{"interface":"front","average":{"1min":1.4019999999999999,"5min":2.6589999999999998,"15min":1.7889999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":7.3739999999999997,"5min":18.687999999999999,"15min":18.687999999999999},"last":1.5309999999999999}]}]},{"osd":4,"up_from":32,"seq":137438953907,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":947236,"kb_used_data":203864,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":92797916,"statfs":{"total":95995035648,"available":95025065984,"internally_reserved":0,"allocated":208756736,"data_stored":208335297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":761199045},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:54:51 2024","interfaces":[{"interface":"back","average":{"1min":1.702,"5min":1.4430000000000001,"15min":1.4319999999999999},"min":{"1min":0.255,"5min":0.23899999999999999,"15min":0.222},"max":{"1min":7.7359999999999998,"5min":10.005000000000001,"15min":10.301},"last":0.36799999999999999},{"interface":"front","average":{"1min":1.7709999999999999,"5min":1.516,"15min":1.498},"min":{"1min":0.34699999999999998,"5min":0.28599999999999998,"15min":0.26700000000000002},"max":{"1min":7.7729999999999997,"5min":10.036,"15min":10.407999999999999},"last":0.46400000000000002}]},{"osd":1,"last update":"Fri Feb 9 17:54:51 2024","interfaces":[{"interface":"back","average":{"1min":1.589,"5min":1.6040000000000001,"15min":1.8180000000000001},"min":{"1min":0.32800000000000001,"5min":0.26100000000000001,"15min":0.23699999999999999},"max":{"1min":5.6390000000000002,"5min":12.429,"15min":17.297999999999998},"last":0.41499999999999998},{"interface":"front","average":{"1min":2.452,"5min":1.8740000000000001,"15min":2.1299999999999999},"min":{"1min":0.28199999999999997,"5min":0.23699999999999999,"15min":0.23100000000000001},"max":{"1min":9.1560000000000006,"5min":12.413,"15min":12.413},"last":0.97299999999999998}]},{"osd":2,"last update":"Fri Feb 9 17:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.663,"15min":2.6150000000000002},"min":{"1min":0.38700000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":4.2220000000000004,"5min":8.5139999999999993,"15min":17.494},"last":0.46000000000000002},{"interface":"front","average":{"1min":1.351,"5min":1.865,"15min":2.6110000000000002},"min":{"1min":0.38,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.6609999999999996,"5min":17.283999999999999,"15min":17.283999999999999},"last":0.44800000000000001}]},{"osd":3,"last update":"Fri Feb 9 17:54:51 2024","interfaces":[{"interface":"back","average":{"1min":1.7609999999999999,"5min":1.4179999999999999,"15min":1.655},"min":{"1min":0.29999999999999999,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":10.484999999999999,"5min":10.484999999999999,"15min":14.448},"last":0.502},{"interface":"front","average":{"1min":1.681,"5min":1.6759999999999999,"15min":1.736},"min":{"1min":0.315,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":10.455,"5min":10.455,"15min":10.455},"last":0.67400000000000004}]},{"osd":5,"last update":"Fri Feb 9 17:55:08 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.79800000000000004,"15min":1.7949999999999999},"min":{"1min":0.26700000000000002,"5min":0.253,"15min":0.22600000000000001},"max":{"1min":3.3250000000000002,"5min":4.8019999999999996,"15min":18.887},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.86799999999999999,"15min":1.877},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31},"max":{"1min":3.3599999999999999,"5min":4.8639999999999999,"15min":18.917000000000002},"last":0.48399999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":1.133,"15min":1.514},"min":{"1min":0.28499999999999998,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.6839999999999999,"5min":23.343,"15min":23.343},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.65900000000000003,"5min":1.0249999999999999,"15min":1.6140000000000001},"min":{"1min":0.33700000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":2.6579999999999999,"5min":21.861000000000001,"15min":21.861000000000001},"last":0.55100000000000005}]},{"osd":7,"last update":"Fri Feb 9 17:55:28 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":1.0129999999999999,"15min":1.536},"min":{"1min":0.41599999999999998,"5min":0.32900000000000001,"15min":0.29899999999999999},"max":{"1min":0.77200000000000002,"5min":6.9550000000000001,"15min":11.875},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.51100000000000001,"5min":1.0389999999999999,"15min":1.6890000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.28599999999999998},"max":{"1min":0.748,"5min":8.6950000000000003,"15min":12.696},"last":0.53500000000000003}]},{"osd":8,"last update":"Fri Feb 9 17:55:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.575,"15min":1.7909999999999999},"min":{"1min":0.35899999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":6.6550000000000002,"5min":8.548,"15min":17.335999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.3600000000000001,"15min":1.272},"min":{"1min":0.39800000000000002,"5min":0.373,"15min":0.34699999999999998},"max":{"1min":4.8040000000000003,"5min":5.6470000000000002,"15min":10.433},"last":0.68899999999999995}]},{"osd":9,"last update":"Fri Feb 9 17:55:12 2024","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":2.1349999999999998,"15min":2.5630000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":5.3639999999999999,"5min":7.665,"15min":7.665},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.92700000000000005,"5min":1.5249999999999999,"15min":1.661},"min":{"1min":0.498,"5min":0.44,"15min":0.44},"max":{"1min":4.2240000000000002,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.63100000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:55:28 2024","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":1.264,"15min":1.5329999999999999},"min":{"1min":0.48899999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":6.9509999999999996,"5min":6.9509999999999996,"15min":17.367000000000001},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.83699999999999997,"5min":1.3680000000000001,"15min":1.587},"min":{"1min":0.47199999999999998,"5min":0.38,"15min":0.38},"max":{"1min":3.028,"5min":5.7439999999999998,"15min":12.888},"last":0.74399999999999999}]},{"osd":11,"last update":"Fri Feb 9 17:54:58 2024","interfaces":[{"interface":"back","average":{"1min":1.486,"5min":1.7729999999999999,"15min":1.5920000000000001},"min":{"1min":0.56499999999999995,"5min":0.46500000000000002,"15min":0.39900000000000002},"max":{"1min":5.9779999999999998,"5min":7.8410000000000002,"15min":10.521000000000001},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.95199999999999996,"5min":1.7070000000000001,"15min":1.625},"min":{"1min":0.502,"5min":0.46000000000000002,"15min":0.40999999999999998},"max":{"1min":3.4809999999999999,"5min":8.4420000000000002,"15min":13.154999999999999},"last":1.0349999999999999}]}]},{"osd":5,"up_from":40,"seq":171798692268,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":813040,"kb_used_data":141856,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":92932112,"statfs":{"total":95995035648,"available":95162482688,"internally_reserved":0,"allocated":145260544,"data_stored":144835553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":687274437},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:52:23 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":2.4900000000000002,"15min":1.71},"min":{"1min":0.27600000000000002,"5min":0.26600000000000001,"15min":0.23599999999999999},"max":{"1min":2.613,"5min":116.77800000000001,"15min":116.77800000000001},"last":0.748},{"interface":"front","average":{"1min":1.246,"5min":2.867,"15min":2.0529999999999999},"min":{"1min":0.36299999999999999,"5min":0.252,"15min":0.249},"max":{"1min":3.2959999999999998,"5min":116.959,"15min":116.959},"last":0.36599999999999999}]},{"osd":1,"last update":"Fri Feb 9 17:55:15 2024","interfaces":[{"interface":"back","average":{"1min":1.552,"5min":1.0820000000000001,"15min":1.6779999999999999},"min":{"1min":0.30099999999999999,"5min":0.27900000000000003,"15min":0.24199999999999999},"max":{"1min":9.532,"5min":9.532,"15min":19.079999999999998},"last":0.78100000000000003},{"interface":"front","average":{"1min":1.1379999999999999,"5min":0.89600000000000002,"15min":1.6679999999999999},"min":{"1min":0.28299999999999997,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":6.2190000000000003,"5min":6.2190000000000003,"15min":19.065999999999999},"last":0.83799999999999997}]},{"osd":2,"last update":"Fri Feb 9 17:54:48 2024","interfaces":[{"interface":"back","average":{"1min":1.55,"5min":1.22,"15min":1.7889999999999999},"min":{"1min":0.312,"5min":0.26100000000000001,"15min":0.249},"max":{"1min":7.5670000000000002,"5min":7.5670000000000002,"15min":13.929},"last":0.81499999999999995},{"interface":"front","average":{"1min":1.5629999999999999,"5min":1.54,"15min":2.052},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":6.3230000000000004,"5min":8.2870000000000008,"15min":15.945},"last":0.79800000000000004}]},{"osd":3,"last update":"Fri Feb 9 17:55:15 2024","interfaces":[{"interface":"back","average":{"1min":2.0219999999999998,"5min":1.8779999999999999,"15min":2.1179999999999999},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":20.690000000000001,"5min":20.690000000000001,"15min":20.690000000000001},"last":0.86199999999999999},{"interface":"front","average":{"1min":1.591,"5min":1.5069999999999999,"15min":2.036},"min":{"1min":0.35499999999999998,"5min":0.316,"15min":0.313},"max":{"1min":6.9269999999999996,"5min":17.209,"15min":17.209},"last":0.77300000000000002}]},{"osd":4,"last update":"Fri Feb 9 17:55:15 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.88,"15min":1.5609999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.249},"max":{"1min":3.8719999999999999,"5min":4.3849999999999998,"15min":28.481999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.89700000000000002,"15min":1.6339999999999999},"min":{"1min":0.32900000000000001,"5min":0.22500000000000001,"15min":0.218},"max":{"1min":1.3129999999999999,"5min":4.9189999999999996,"15min":15.843},"last":0.60199999999999998}]},{"osd":6,"last update":"Fri Feb 9 17:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":1.103,"15min":1.637},"min":{"1min":0.33000000000000002,"5min":0.30199999999999999,"15min":0.27900000000000003},"max":{"1min":2.3079999999999998,"5min":8.3829999999999991,"15min":16.657},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.80200000000000005,"5min":1.3340000000000001,"15min":1.764},"min":{"1min":0.33900000000000002,"5min":0.33300000000000002,"15min":0.26200000000000001},"max":{"1min":2.4609999999999999,"5min":9.7200000000000006,"15min":20.192},"last":0.68100000000000005}]},{"osd":7,"last update":"Fri Feb 9 17:55:19 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":1.2589999999999999,"15min":1.704},"min":{"1min":0.39000000000000001,"5min":0.29499999999999998,"15min":0.27900000000000003},"max":{"1min":3.6880000000000002,"5min":18.134,"15min":18.134},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.59199999999999997,"5min":1.1830000000000001,"15min":1.8859999999999999},"min":{"1min":0.39800000000000002,"5min":0.32600000000000001,"15min":0.31},"max":{"1min":1.0760000000000001,"5min":19.300000000000001,"15min":19.300000000000001},"last":0.65400000000000003}]},{"osd":8,"last update":"Fri Feb 9 17:54:36 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":1.665,"15min":1.8089999999999999},"min":{"1min":0.40400000000000003,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":3.0449999999999999,"5min":9.6110000000000007,"15min":9.6110000000000007},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.877,"5min":1.734,"15min":1.877},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":4.3719999999999999,"5min":8.1829999999999998,"15min":8.1829999999999998},"last":0.875}]},{"osd":9,"last update":"Fri Feb 9 17:54:48 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.956,"15min":1.7250000000000001},"min":{"1min":0.42099999999999999,"5min":0.41499999999999998,"15min":0.35299999999999998},"max":{"1min":4.3369999999999997,"5min":10.987,"15min":16.411000000000001},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.879,"5min":1.484,"15min":1.748},"min":{"1min":0.438,"5min":0.373,"15min":0.35299999999999998},"max":{"1min":2.387,"5min":9.7050000000000001,"15min":16.417999999999999},"last":0.96499999999999997}]},{"osd":10,"last update":"Fri Feb 9 17:55:15 2024","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":1.5549999999999999,"15min":2},"min":{"1min":0.52100000000000002,"5min":0.373,"15min":0.317},"max":{"1min":4.0140000000000002,"5min":16.292000000000002,"15min":80.042000000000002},"last":0.95599999999999996},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.502,"15min":1.9019999999999999},"min":{"1min":0.54800000000000004,"5min":0.41199999999999998,"15min":0.38200000000000001},"max":{"1min":4.4960000000000004,"5min":12.426,"15min":79.807000000000002},"last":0.88900000000000001}]},{"osd":11,"last update":"Fri Feb 9 17:54:26 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.5860000000000001,"15min":2},"min":{"1min":0.44800000000000001,"5min":0.44700000000000001,"15min":0.38400000000000001},"max":{"1min":4.657,"5min":9.7680000000000007,"15min":16.390000000000001},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.94199999999999995,"5min":2.1269999999999998,"15min":1.893},"min":{"1min":0.442,"5min":0.44,"15min":0.42199999999999999},"max":{"1min":3.1200000000000001,"5min":20.981000000000002,"15min":20.981000000000002},"last":0.91100000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":254279680,"data_stored":254279680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":274530304,"data_stored":274530304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":223608832,"data_stored":223608832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":230293504,"data_stored":230293504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":235470848,"data_stored":235470848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":195624960,"data_stored":195624960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":119603200,"data_stored":119603200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":202768384,"data_stored":202768384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":382926848,"data_stored":382926848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":481034240,"data_stored":481034240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":288423936,"data_stored":288423936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-09T17:55:41.490 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph pg dump --format=json 2024-02-09T17:55:41.545 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:55:41.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:41.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:55:41.588 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 15 16 37174 37158 154.811 120.688 0.00668716 0.00645549 2024-02-09T17:55:41.662 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:55:41.693 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:41.726 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:55:41.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:41.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:55:41.831 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:55:41.870 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:42.043 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:55:42.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:42.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:55:42.127 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:55:42.148 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:55:42.153 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:55:42.176 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:42.252 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:55:42.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:42.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:55:42.356 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:55:42.391 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:42.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:42 smithi162 ceph-mon[44304]: pgmap v1969: 47 pgs: 47 active+clean; 1.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 120 MiB/s wr, 1.92k op/s 2024-02-09T17:55:42.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:42 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:55:42.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:42 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:55:42.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:42 smithi162 ceph-mon[44304]: osdmap e802: 12 total, 12 up, 12 in 2024-02-09T17:55:42.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:42 smithi099 ceph-mon[39420]: pgmap v1969: 47 pgs: 47 active+clean; 1.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 120 MiB/s wr, 1.92k op/s 2024-02-09T17:55:42.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:42 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:55:42.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:42 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:55:42.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:42 smithi099 ceph-mon[39420]: osdmap e802: 12 total, 12 up, 12 in 2024-02-09T17:55:42.588 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 16 16 39113 39097 152.705 121.188 0.00515045 0.00654462 2024-02-09T17:55:42.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:42 smithi179 ceph-mon[43975]: pgmap v1969: 47 pgs: 47 active+clean; 1.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 120 MiB/s wr, 1.92k op/s 2024-02-09T17:55:42.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:42 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:55:42.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:42 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:55:42.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:42 smithi179 ceph-mon[43975]: osdmap e802: 12 total, 12 up, 12 in 2024-02-09T17:55:43.245 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:55:43.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:43.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:55:43.349 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:55:43.390 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:43.431 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:55:43.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:43.436 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:55:43.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:43 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:55:43.510 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:43 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:55:43.539 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:55:43.574 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:43.589 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 17 16 40542 40526 148.967 89.3125 0.0116002 0.00670821 2024-02-09T17:55:43.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:43 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:55:43.826 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:55:43.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:43.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:55:43.884 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:55:43.884 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:55:43.931 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:55:43.963 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:44.119 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:55:44.120 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:55:44.130 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:55:44.160 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:55:44.164 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:55:44.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:44.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:55:44.179 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:55:44.204 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:55:44.227 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:55:44.247 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:55:44.268 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:55:44.269 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:55:44.288 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:55:44.294 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:44.310 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:55:44.313 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:55:44.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:44.316 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:55:44.329 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:55:44.330 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:55:44.331 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:55:44.331 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:55:44.331 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:55:44.331 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:55:44.331 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:55:44.331 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:55:44.331 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:55:44.331 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:55:44.331 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:55:44.331 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:55:44.331 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:55:44.417 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:55:44.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:44 smithi162 ceph-mon[44304]: pgmap v1971: 47 pgs: 47 active+clean; 2.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 151 MiB/s wr, 2.41k op/s 2024-02-09T17:55:44.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:44 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2724878821' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:55:44.458 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:44.589 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 18 16 42144 42128 146.253 100.125 0.00273246 0.00683255 2024-02-09T17:55:44.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:44 smithi179 ceph-mon[43975]: pgmap v1971: 47 pgs: 47 active+clean; 2.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 151 MiB/s wr, 2.41k op/s 2024-02-09T17:55:44.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:44 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2724878821' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:55:44.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:44 smithi099 ceph-mon[39420]: pgmap v1971: 47 pgs: 47 active+clean; 2.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 151 MiB/s wr, 2.41k op/s 2024-02-09T17:55:44.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:44 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2724878821' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:55:44.761 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-09T17:55:45.087 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:55:45.119 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:55:45.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:45.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:55:45.225 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:55:45.283 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:45.319 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:55:45.328 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:45.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:55:45.429 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:55:45.479 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:45.591 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 19 15 43998 43983 144.641 115.938 0.0148186 0.00690821 2024-02-09T17:55:46.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:46 smithi162 ceph-mon[44304]: pgmap v1972: 47 pgs: 47 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 124 MiB/s wr, 1.98k op/s 2024-02-09T17:55:46.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:46 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:55:46.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:46 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:55:46.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:46 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:55:46.441 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:46 smithi099 ceph-mon[39420]: pgmap v1972: 47 pgs: 47 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 124 MiB/s wr, 1.98k op/s 2024-02-09T17:55:46.441 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:46 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:55:46.441 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:46 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:55:46.441 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:46 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:55:46.593 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:2024-02-09T17:55:46.592821+0000 min lat: 0.00109523 max lat: 0.219811 avg lat: 0.00699613 2024-02-09T17:55:46.593 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-09T17:55:46.593 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 20 16 45740 45724 142.839 108.812 0.00594203 0.00699613 2024-02-09T17:55:46.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:46 smithi179 ceph-mon[43975]: pgmap v1972: 47 pgs: 47 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 124 MiB/s wr, 1.98k op/s 2024-02-09T17:55:46.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:46 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:55:46.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:46 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:55:46.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:46 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:55:46.847 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:55:46.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:46.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:55:46.955 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:55:46.981 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:47.000 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:55:47.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:47.002 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:55:47.046 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:55:47.047 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-09T17:55:47.047 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-02-09T17:55:47.047 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-02-09T17:55:47.047 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:55:47.103 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:55:47.146 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:47.595 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 21 16 47288 47272 140.643 96.75 0.00685684 0.00710573 2024-02-09T17:55:47.987 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:47 smithi099 ceph-mon[39420]: pgmap v1973: 47 pgs: 47 active+clean; 2.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 109 MiB/s wr, 1.75k op/s 2024-02-09T17:55:48.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:47 smithi179 ceph-mon[43975]: pgmap v1973: 47 pgs: 47 active+clean; 2.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 109 MiB/s wr, 1.75k op/s 2024-02-09T17:55:48.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:47 smithi162 ceph-mon[44304]: pgmap v1973: 47 pgs: 47 active+clean; 2.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 109 MiB/s wr, 1.75k op/s 2024-02-09T17:55:48.593 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 22 16 49036 49020 139.216 109.25 0.00861939 0.00717836 2024-02-09T17:55:48.700 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:55:48.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:48.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:55:48.724 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-09T17:55:48.804 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:55:48.849 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:48.877 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:55:48.879 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:55:48.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:48.884 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:55:48.985 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:55:49.052 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:49.071 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:55:49.099 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:55:49.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:49.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:55:49.210 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:55:49.239 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:55:49.246 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is unset 2024-02-09T17:55:49.257 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:49.260 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-02-09T17:55:49.284 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:55:49.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:49.288 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:55:49.334 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:55:49.389 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:55:49.420 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:49.445 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:55:49.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:49.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:55:49.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:49 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2587558376' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:55:49.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:49 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:55:49.550 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:55:49.595 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:49 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2587558376' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:55:49.595 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:49 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:55:49.595 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 23 16 50552 50536 137.278 94.75 0.0100136 0.00727987 2024-02-09T17:55:49.597 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:49.608 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:49 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2587558376' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:55:49.608 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:49 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:55:49.626 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:55:49.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:49.631 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:55:49.732 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:55:49.760 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:49.912 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:55:49.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:49.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:55:50.016 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:55:50.044 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:50.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:50 smithi099 ceph-mon[39420]: from='client.26326 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:55:50.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:50 smithi099 ceph-mon[39420]: pgmap v1974: 47 pgs: 47 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 115 MiB/s wr, 1.84k op/s 2024-02-09T17:55:50.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:50 smithi099 ceph-mon[39420]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:55:50.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:50 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:55:50.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:50 smithi099 ceph-mon[39420]: osdmap e803: 12 total, 12 up, 12 in 2024-02-09T17:55:50.541 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:50 smithi162 ceph-mon[44304]: from='client.26326 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:55:50.541 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:50 smithi162 ceph-mon[44304]: pgmap v1974: 47 pgs: 47 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 115 MiB/s wr, 1.84k op/s 2024-02-09T17:55:50.541 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:50 smithi162 ceph-mon[44304]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:55:50.541 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:50 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:55:50.541 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:50 smithi162 ceph-mon[44304]: osdmap e803: 12 total, 12 up, 12 in 2024-02-09T17:55:50.595 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 24 16 51968 51952 135.245 88.5 0.00495914 0.00738975 2024-02-09T17:55:50.600 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:55:50.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:50.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:55:50.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:50 smithi179 ceph-mon[43975]: from='client.26326 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:55:50.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:50 smithi179 ceph-mon[43975]: pgmap v1974: 47 pgs: 47 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 115 MiB/s wr, 1.84k op/s 2024-02-09T17:55:50.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:50 smithi179 ceph-mon[43975]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:55:50.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:50 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:55:50.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:50 smithi179 ceph-mon[43975]: osdmap e803: 12 total, 12 up, 12 in 2024-02-09T17:55:50.705 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:55:50.749 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:50.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:50.781 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:55:50.781 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:55:50.882 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:55:50.916 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:51.170 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:55:51.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:51.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:55:51.213 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:55:51.213 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-02-09T17:55:51.213 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-02-09T17:55:51.213 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-02-09T17:55:51.214 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.214 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25837.0:31846 6.3 6:c78d3534:::benchmark_data_smithi179_126206_object31845:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e580)", 2024-02-09T17:55:51.214 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:46:03.943859+0000", 2024-02-09T17:55:51.214 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 587.26605594399996, 2024-02-09T17:55:51.214 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20644096200000001, 2024-02-09T17:55:51.214 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:55:51.214 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:51.214 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:55:51.214 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25837", 2024-02-09T17:55:51.214 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:55:51.214 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 31846 2024-02-09T17:55:51.214 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.214 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:55:51.214 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.215 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:55:51.215 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:03.943859+0000", 2024-02-09T17:55:51.215 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:51.215 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.215 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.215 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:55:51.215 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:03.943859+0000", 2024-02-09T17:55:51.215 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:51.215 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.215 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.215 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:55:51.215 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:03.943860+0000", 2024-02-09T17:55:51.215 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.08e-06 2024-02-09T17:55:51.215 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.216 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.216 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:55:51.216 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:03.943888+0000", 2024-02-09T17:55:51.216 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7860000000000001e-05 2024-02-09T17:55:51.216 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.216 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.216 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:55:51.216 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:03.943897+0000", 2024-02-09T17:55:51.216 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7050000000000005e-06 2024-02-09T17:55:51.216 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.216 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.216 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:55:51.216 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:03.943904+0000", 2024-02-09T17:55:51.216 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8050000000000001e-06 2024-02-09T17:55:51.217 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.217 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.217 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:55:51.217 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:03.943965+0000", 2024-02-09T17:55:51.217 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0979e-05 2024-02-09T17:55:51.217 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.217 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.217 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:55:51.217 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:03.944116+0000", 2024-02-09T17:55:51.217 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015082600000000001 2024-02-09T17:55:51.217 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.217 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.217 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-02-09T17:55:51.217 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:03.944264+0000", 2024-02-09T17:55:51.218 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014844500000000001 2024-02-09T17:55:51.218 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.218 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.218 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:55:51.218 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:03.945608+0000", 2024-02-09T17:55:51.218 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013436660000000001 2024-02-09T17:55:51.218 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.218 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.218 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:51.218 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:03.946051+0000", 2024-02-09T17:55:51.218 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00044373500000000001 2024-02-09T17:55:51.218 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.218 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.218 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:51.218 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:04.150179+0000", 2024-02-09T17:55:51.219 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20412766399999999 2024-02-09T17:55:51.219 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.219 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.219 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:55:51.219 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:04.150255+0000", 2024-02-09T17:55:51.219 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5561000000000004e-05 2024-02-09T17:55:51.219 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.219 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.219 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:55:51.219 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:04.150300+0000", 2024-02-09T17:55:51.219 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5636000000000001e-05 2024-02-09T17:55:51.219 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:51.219 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:55:51.220 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:51.220 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.220 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.220 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25837.0:31857 6.3 6:c8202ba4:::benchmark_data_smithi179_126206_object31856:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e580)", 2024-02-09T17:55:51.220 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:46:03.946335+0000", 2024-02-09T17:55:51.220 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 587.26358082199999, 2024-02-09T17:55:51.220 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20583396500000001, 2024-02-09T17:55:51.220 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:55:51.220 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:51.220 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:55:51.220 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25837", 2024-02-09T17:55:51.220 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:55:51.220 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 31857 2024-02-09T17:55:51.220 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.220 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:55:51.221 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.221 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:55:51.221 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:03.946335+0000", 2024-02-09T17:55:51.221 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:51.221 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.221 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.221 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:55:51.221 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:03.946335+0000", 2024-02-09T17:55:51.221 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:51.221 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.221 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.221 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:55:51.221 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:03.946336+0000", 2024-02-09T17:55:51.221 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3769999999999999e-06 2024-02-09T17:55:51.222 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.222 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.222 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:55:51.222 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:03.946404+0000", 2024-02-09T17:55:51.222 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7681000000000005e-05 2024-02-09T17:55:51.222 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.222 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.222 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:55:51.222 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:03.946414+0000", 2024-02-09T17:55:51.222 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0232e-05 2024-02-09T17:55:51.222 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.222 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.222 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:55:51.222 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:03.946421+0000", 2024-02-09T17:55:51.223 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0280000000000003e-06 2024-02-09T17:55:51.223 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.223 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.223 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:55:51.223 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:03.946477+0000", 2024-02-09T17:55:51.223 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6447999999999999e-05 2024-02-09T17:55:51.223 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.223 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.223 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:55:51.223 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:03.946597+0000", 2024-02-09T17:55:51.223 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000119992 2024-02-09T17:55:51.223 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.223 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.223 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-02-09T17:55:51.223 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:03.946728+0000", 2024-02-09T17:55:51.223 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013100300000000001 2024-02-09T17:55:51.224 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.224 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.224 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:55:51.224 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:03.949543+0000", 2024-02-09T17:55:51.224 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0028148980000000001 2024-02-09T17:55:51.224 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.224 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.224 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:51.224 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:03.950881+0000", 2024-02-09T17:55:51.224 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013373980000000001 2024-02-09T17:55:51.224 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.224 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.224 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:51.224 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:04.152092+0000", 2024-02-09T17:55:51.225 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.201211682 2024-02-09T17:55:51.225 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.225 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.225 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:55:51.225 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:04.152128+0000", 2024-02-09T17:55:51.225 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5299999999999997e-05 2024-02-09T17:55:51.225 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.225 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.225 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:55:51.225 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:46:04.152168+0000", 2024-02-09T17:55:51.225 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0926000000000001e-05 2024-02-09T17:55:51.225 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:51.225 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:55:51.225 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:51.225 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.225 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.226 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25837.0:254297 6.3 6:cd45665f:::benchmark_data_smithi179_126206_object123916:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:55:51.226 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:49:52.578440+0000", 2024-02-09T17:55:51.226 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 358.63147530499998, 2024-02-09T17:55:51.226 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.218960764, 2024-02-09T17:55:51.226 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:55:51.226 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:51.226 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:55:51.226 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25837", 2024-02-09T17:55:51.226 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:55:51.226 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 254297 2024-02-09T17:55:51.226 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.226 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:55:51.226 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.226 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:55:51.227 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.578440+0000", 2024-02-09T17:55:51.227 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:51.227 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.227 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.227 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:55:51.227 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.578440+0000", 2024-02-09T17:55:51.227 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:51.227 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.227 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.227 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:55:51.227 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.578441+0000", 2024-02-09T17:55:51.227 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.42e-06 2024-02-09T17:55:51.227 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.227 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.227 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:55:51.228 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.578445+0000", 2024-02-09T17:55:51.228 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6119999999999999e-06 2024-02-09T17:55:51.228 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.228 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.228 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:55:51.228 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.578454+0000", 2024-02-09T17:55:51.228 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9279999999999999e-06 2024-02-09T17:55:51.228 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.228 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.228 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:55:51.228 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.578460+0000", 2024-02-09T17:55:51.228 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.472e-06 2024-02-09T17:55:51.228 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.228 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.229 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:55:51.229 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.578519+0000", 2024-02-09T17:55:51.229 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8715999999999998e-05 2024-02-09T17:55:51.229 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.229 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.229 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:55:51.229 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.578651+0000", 2024-02-09T17:55:51.229 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013221500000000001 2024-02-09T17:55:51.229 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.229 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.229 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-02-09T17:55:51.229 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.578764+0000", 2024-02-09T17:55:51.229 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000112608 2024-02-09T17:55:51.229 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.230 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.230 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:55:51.230 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.588646+0000", 2024-02-09T17:55:51.230 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0098819219999999996 2024-02-09T17:55:51.230 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.230 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.230 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:51.230 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.593408+0000", 2024-02-09T17:55:51.230 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0047621299999999998 2024-02-09T17:55:51.230 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.230 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.230 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:51.230 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.797292+0000", 2024-02-09T17:55:51.230 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20388404099999999 2024-02-09T17:55:51.230 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.231 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.231 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:55:51.231 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.797354+0000", 2024-02-09T17:55:51.231 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2098000000000001e-05 2024-02-09T17:55:51.231 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.231 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.231 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:55:51.231 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.797401+0000", 2024-02-09T17:55:51.231 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6601999999999999e-05 2024-02-09T17:55:51.231 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:51.231 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:55:51.231 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:51.231 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.231 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.231 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25837.0:254298 6.3 6:c9777fa5:::benchmark_data_smithi179_126206_object123917:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:55:51.232 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:49:52.578475+0000", 2024-02-09T17:55:51.232 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 358.63144049700003, 2024-02-09T17:55:51.232 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24596101500000001, 2024-02-09T17:55:51.232 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:55:51.232 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:51.232 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:55:51.232 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25837", 2024-02-09T17:55:51.232 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:55:51.232 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 254298 2024-02-09T17:55:51.232 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.232 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:55:51.232 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.232 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:55:51.232 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.578475+0000", 2024-02-09T17:55:51.233 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:51.233 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.233 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.233 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:55:51.233 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.578475+0000", 2024-02-09T17:55:51.233 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:51.233 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.233 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.233 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:55:51.233 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.578475+0000", 2024-02-09T17:55:51.233 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1399999999999997e-07 2024-02-09T17:55:51.233 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.233 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.233 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:55:51.234 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.578478+0000", 2024-02-09T17:55:51.234 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2759999999999999e-06 2024-02-09T17:55:51.234 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.234 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.234 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:55:51.234 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583629+0000", 2024-02-09T17:55:51.234 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0051514940000000004 2024-02-09T17:55:51.234 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.234 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.234 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:55:51.234 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583640+0000", 2024-02-09T17:55:51.234 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0791999999999999e-05 2024-02-09T17:55:51.234 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.234 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.234 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:55:51.235 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.584392+0000", 2024-02-09T17:55:51.235 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00075225900000000002 2024-02-09T17:55:51.235 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.235 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.235 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:55:51.235 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.584488+0000", 2024-02-09T17:55:51.235 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5838e-05 2024-02-09T17:55:51.235 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.235 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.235 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-02-09T17:55:51.235 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.584555+0000", 2024-02-09T17:55:51.235 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7478999999999995e-05 2024-02-09T17:55:51.235 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.235 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.236 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:55:51.236 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.593428+0000", 2024-02-09T17:55:51.236 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0088728939999999992 2024-02-09T17:55:51.236 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.236 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.236 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:51.236 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.596904+0000", 2024-02-09T17:55:51.236 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034754249999999999 2024-02-09T17:55:51.236 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.236 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.236 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:51.236 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.824355+0000", 2024-02-09T17:55:51.236 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.227451715 2024-02-09T17:55:51.236 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.237 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.237 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:55:51.237 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.824395+0000", 2024-02-09T17:55:51.237 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9900000000000001e-05 2024-02-09T17:55:51.237 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.237 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.237 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:55:51.237 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.824436+0000", 2024-02-09T17:55:51.237 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0528999999999998e-05 2024-02-09T17:55:51.237 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:51.237 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:55:51.237 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:51.237 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.237 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.238 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25837.0:254300 6.5 6:aac8f1b4:::benchmark_data_smithi179_126206_object123919:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:55:51.238 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:49:52.583663+0000", 2024-02-09T17:55:51.238 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 358.626251841, 2024-02-09T17:55:51.238 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.212262326, 2024-02-09T17:55:51.238 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:55:51.238 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:51.238 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:55:51.238 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25837", 2024-02-09T17:55:51.238 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:55:51.238 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 254300 2024-02-09T17:55:51.238 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.238 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:55:51.238 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.238 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:55:51.238 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583663+0000", 2024-02-09T17:55:51.239 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:51.239 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.239 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.239 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:55:51.239 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583663+0000", 2024-02-09T17:55:51.239 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:51.239 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.239 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.239 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:55:51.239 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583665+0000", 2024-02-09T17:55:51.239 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.217e-06 2024-02-09T17:55:51.239 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.239 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.239 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:55:51.240 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583669+0000", 2024-02-09T17:55:51.240 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4739999999999999e-06 2024-02-09T17:55:51.240 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.240 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.240 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:55:51.240 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583679+0000", 2024-02-09T17:55:51.240 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7559999999999994e-06 2024-02-09T17:55:51.240 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.240 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.240 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:55:51.240 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583685+0000", 2024-02-09T17:55:51.240 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6130000000000003e-06 2024-02-09T17:55:51.240 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.240 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.240 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:55:51.241 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583729+0000", 2024-02-09T17:55:51.241 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4237999999999999e-05 2024-02-09T17:55:51.241 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.241 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.241 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:55:51.241 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583847+0000", 2024-02-09T17:55:51.241 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000118322 2024-02-09T17:55:51.241 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.241 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.241 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-02-09T17:55:51.241 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583952+0000", 2024-02-09T17:55:51.241 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000105154 2024-02-09T17:55:51.241 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.241 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.242 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:51.242 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.593158+0000", 2024-02-09T17:55:51.242 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0092052690000000003 2024-02-09T17:55:51.242 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.242 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.242 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:55:51.242 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.593588+0000", 2024-02-09T17:55:51.242 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000430176 2024-02-09T17:55:51.242 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.242 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.242 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:51.242 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.795858+0000", 2024-02-09T17:55:51.242 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20227031700000001 2024-02-09T17:55:51.242 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.243 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.243 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:55:51.243 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.795896+0000", 2024-02-09T17:55:51.243 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8470999999999998e-05 2024-02-09T17:55:51.243 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.243 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.243 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:55:51.243 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.795926+0000", 2024-02-09T17:55:51.243 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9319000000000002e-05 2024-02-09T17:55:51.243 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:51.243 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:55:51.243 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:51.243 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.243 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.244 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25837.0:254302 6.5 6:ac558352:::benchmark_data_smithi179_126206_object123921:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:55:51.244 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:49:52.583698+0000", 2024-02-09T17:55:51.244 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 358.62621743400001, 2024-02-09T17:55:51.244 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22801817599999999, 2024-02-09T17:55:51.244 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:55:51.244 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:51.244 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:55:51.244 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25837", 2024-02-09T17:55:51.244 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:55:51.244 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 254302 2024-02-09T17:55:51.244 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.244 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:55:51.244 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.244 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:55:51.245 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583698+0000", 2024-02-09T17:55:51.245 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:51.245 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.245 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.245 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:55:51.245 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583698+0000", 2024-02-09T17:55:51.245 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:51.245 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.245 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.245 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:55:51.245 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583698+0000", 2024-02-09T17:55:51.245 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9499999999999998e-07 2024-02-09T17:55:51.245 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.245 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.245 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:55:51.246 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583701+0000", 2024-02-09T17:55:51.246 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3760000000000002e-06 2024-02-09T17:55:51.246 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.246 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.246 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:55:51.246 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583707+0000", 2024-02-09T17:55:51.246 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.241e-06 2024-02-09T17:55:51.246 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.246 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.246 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:55:51.246 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583712+0000", 2024-02-09T17:55:51.246 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1800000000000004e-06 2024-02-09T17:55:51.246 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.246 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.247 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:55:51.247 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.589123+0000", 2024-02-09T17:55:51.247 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0054113909999999998 2024-02-09T17:55:51.247 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.247 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.247 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:55:51.247 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.589215+0000", 2024-02-09T17:55:51.247 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1038999999999994e-05 2024-02-09T17:55:51.247 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.247 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.247 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-02-09T17:55:51.247 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.589298+0000", 2024-02-09T17:55:51.247 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3084999999999995e-05 2024-02-09T17:55:51.247 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.247 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.248 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:55:51.248 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.593594+0000", 2024-02-09T17:55:51.248 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042961290000000001 2024-02-09T17:55:51.248 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.248 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.248 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:51.248 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.594360+0000", 2024-02-09T17:55:51.248 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00076610199999999995 2024-02-09T17:55:51.248 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.248 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.248 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:51.248 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.811612+0000", 2024-02-09T17:55:51.248 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21725206 2024-02-09T17:55:51.249 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.249 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.249 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:55:51.249 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.811672+0000", 2024-02-09T17:55:51.249 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9901000000000001e-05 2024-02-09T17:55:51.249 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.249 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.249 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:55:51.249 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.811716+0000", 2024-02-09T17:55:51.249 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4277000000000003e-05 2024-02-09T17:55:51.249 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:51.249 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:55:51.249 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:51.249 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.250 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.250 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25837.0:254319 6.3 6:c6f40c4d:::benchmark_data_smithi179_126206_object123938:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:55:51.250 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:49:52.603086+0000", 2024-02-09T17:55:51.250 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 358.60682949, 2024-02-09T17:55:51.250 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22144201199999999, 2024-02-09T17:55:51.250 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:55:51.250 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:51.250 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:55:51.250 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25837", 2024-02-09T17:55:51.250 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:55:51.250 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 254319 2024-02-09T17:55:51.250 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.250 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:55:51.250 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.251 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:55:51.251 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.603086+0000", 2024-02-09T17:55:51.251 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:51.251 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.251 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.251 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:55:51.251 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.603086+0000", 2024-02-09T17:55:51.251 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:51.251 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.251 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.251 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:55:51.251 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.603087+0000", 2024-02-09T17:55:51.251 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5409999999999999e-06 2024-02-09T17:55:51.251 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.251 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.252 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:55:51.252 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.603093+0000", 2024-02-09T17:55:51.252 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9599999999999997e-06 2024-02-09T17:55:51.252 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.252 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.252 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:55:51.252 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.603106+0000", 2024-02-09T17:55:51.252 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2849e-05 2024-02-09T17:55:51.252 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.252 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.252 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:55:51.252 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.603114+0000", 2024-02-09T17:55:51.252 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9689999999999994e-06 2024-02-09T17:55:51.252 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.252 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.253 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:55:51.253 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.603179+0000", 2024-02-09T17:55:51.253 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4348000000000001e-05 2024-02-09T17:55:51.253 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.253 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.253 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:55:51.253 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.603326+0000", 2024-02-09T17:55:51.253 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014761700000000001 2024-02-09T17:55:51.253 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.253 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.253 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-02-09T17:55:51.253 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.603460+0000", 2024-02-09T17:55:51.253 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013390799999999999 2024-02-09T17:55:51.253 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.253 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.254 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:51.254 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.609555+0000", 2024-02-09T17:55:51.254 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0060945360000000002 2024-02-09T17:55:51.254 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.254 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.254 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:55:51.254 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.610250+0000", 2024-02-09T17:55:51.254 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00069547599999999999 2024-02-09T17:55:51.254 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.254 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.254 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:51.254 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.824478+0000", 2024-02-09T17:55:51.254 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.214227852 2024-02-09T17:55:51.254 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.254 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.255 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:55:51.255 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.824507+0000", 2024-02-09T17:55:51.255 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9278e-05 2024-02-09T17:55:51.255 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.255 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.255 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:55:51.255 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.824528+0000", 2024-02-09T17:55:51.255 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0678000000000001e-05 2024-02-09T17:55:51.255 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:51.255 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:55:51.255 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:51.255 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.255 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.255 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25837.0:254323 6.5 6:aaac2b69:::benchmark_data_smithi179_126206_object123942:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:55:51.256 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:49:52.604437+0000", 2024-02-09T17:55:51.256 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 358.60547851500002, 2024-02-09T17:55:51.256 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.217027939, 2024-02-09T17:55:51.256 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:55:51.256 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:51.256 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:55:51.256 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25837", 2024-02-09T17:55:51.256 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:55:51.256 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 254323 2024-02-09T17:55:51.256 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.256 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:55:51.256 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.256 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:55:51.256 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.604437+0000", 2024-02-09T17:55:51.257 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:51.257 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.257 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.257 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:55:51.257 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.604437+0000", 2024-02-09T17:55:51.257 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:51.257 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.257 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.257 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:55:51.257 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.604438+0000", 2024-02-09T17:55:51.257 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.051e-06 2024-02-09T17:55:51.257 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.257 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.257 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:55:51.257 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.604443+0000", 2024-02-09T17:55:51.258 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.147e-06 2024-02-09T17:55:51.258 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.258 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.258 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:55:51.258 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.604456+0000", 2024-02-09T17:55:51.258 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3414000000000001e-05 2024-02-09T17:55:51.258 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.258 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.258 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:55:51.258 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.604465+0000", 2024-02-09T17:55:51.258 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5359999999999992e-06 2024-02-09T17:55:51.258 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.258 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.258 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:55:51.259 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.604517+0000", 2024-02-09T17:55:51.259 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2478999999999997e-05 2024-02-09T17:55:51.259 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.259 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.259 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:55:51.259 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.604630+0000", 2024-02-09T17:55:51.259 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011217099999999999 2024-02-09T17:55:51.259 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.259 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.259 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-02-09T17:55:51.259 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.604706+0000", 2024-02-09T17:55:51.259 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6616e-05 2024-02-09T17:55:51.259 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.259 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.259 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:51.260 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.609972+0000", 2024-02-09T17:55:51.260 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0052659459999999996 2024-02-09T17:55:51.260 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.260 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.260 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:55:51.260 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.610237+0000", 2024-02-09T17:55:51.260 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00026531299999999999 2024-02-09T17:55:51.260 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.260 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.260 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:51.263 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.821367+0000", 2024-02-09T17:55:51.263 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21112936800000001 2024-02-09T17:55:51.263 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.263 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.263 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:55:51.264 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.821423+0000", 2024-02-09T17:55:51.264 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6109000000000001e-05 2024-02-09T17:55:51.264 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.264 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.264 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:55:51.264 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.821465+0000", 2024-02-09T17:55:51.264 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1789000000000003e-05 2024-02-09T17:55:51.264 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:51.264 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:55:51.264 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:51.264 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.264 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.264 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25837.0:254329 6.5 6:abdecdae:::benchmark_data_smithi179_126206_object123948:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:55:51.264 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:49:52.613907+0000", 2024-02-09T17:55:51.265 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 358.596008144, 2024-02-09T17:55:51.265 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20766915699999999, 2024-02-09T17:55:51.265 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:55:51.265 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:51.265 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:55:51.265 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25837", 2024-02-09T17:55:51.265 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:55:51.265 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 254329 2024-02-09T17:55:51.265 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.265 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:55:51.265 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.265 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:55:51.265 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.613907+0000", 2024-02-09T17:55:51.265 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:51.266 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.266 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.266 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:55:51.266 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.613907+0000", 2024-02-09T17:55:51.266 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:51.266 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.266 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.266 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:55:51.266 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.613909+0000", 2024-02-09T17:55:51.266 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6929999999999999e-06 2024-02-09T17:55:51.266 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.266 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.267 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:55:51.267 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.613915+0000", 2024-02-09T17:55:51.267 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4230000000000002e-06 2024-02-09T17:55:51.267 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.267 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.267 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:55:51.267 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.613930+0000", 2024-02-09T17:55:51.267 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5007999999999999e-05 2024-02-09T17:55:51.267 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.267 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.267 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:55:51.267 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.613940+0000", 2024-02-09T17:55:51.267 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5000000000000005e-06 2024-02-09T17:55:51.267 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.267 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.268 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:55:51.268 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.614001+0000", 2024-02-09T17:55:51.268 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0888999999999999e-05 2024-02-09T17:55:51.268 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.268 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.268 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:55:51.268 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.614152+0000", 2024-02-09T17:55:51.268 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015174200000000001 2024-02-09T17:55:51.268 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.268 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.268 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-02-09T17:55:51.268 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.614283+0000", 2024-02-09T17:55:51.268 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013045099999999999 2024-02-09T17:55:51.268 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.268 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.269 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:55:51.269 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.618788+0000", 2024-02-09T17:55:51.269 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0045055859999999998 2024-02-09T17:55:51.269 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.269 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.269 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:51.269 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.620381+0000", 2024-02-09T17:55:51.269 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0015923070000000001 2024-02-09T17:55:51.269 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.269 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.269 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:51.269 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.821500+0000", 2024-02-09T17:55:51.269 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20111922300000001 2024-02-09T17:55:51.269 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.270 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.270 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:55:51.270 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.821545+0000", 2024-02-09T17:55:51.270 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4802000000000002e-05 2024-02-09T17:55:51.270 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.270 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.270 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:55:51.270 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.821576+0000", 2024-02-09T17:55:51.270 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1532999999999998e-05 2024-02-09T17:55:51.270 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:51.270 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:55:51.270 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:51.270 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.270 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.271 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25837.0:254336 6.3 6:c045a3b9:::benchmark_data_smithi179_126206_object123955:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:55:51.271 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:49:52.620572+0000", 2024-02-09T17:55:51.271 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 358.58934308900001, 2024-02-09T17:55:51.271 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.204024654, 2024-02-09T17:55:51.271 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:55:51.271 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:51.271 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:55:51.271 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25837", 2024-02-09T17:55:51.271 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:55:51.271 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 254336 2024-02-09T17:55:51.271 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.271 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:55:51.271 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.271 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:55:51.271 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.620572+0000", 2024-02-09T17:55:51.272 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:51.272 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.272 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.272 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:55:51.272 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.620572+0000", 2024-02-09T17:55:51.272 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:51.272 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.272 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.272 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:55:51.272 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.620574+0000", 2024-02-09T17:55:51.272 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.826e-06 2024-02-09T17:55:51.272 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.272 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.272 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:55:51.273 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.620579+0000", 2024-02-09T17:55:51.273 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2039999999999996e-06 2024-02-09T17:55:51.273 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.273 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.273 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:55:51.273 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.620592+0000", 2024-02-09T17:55:51.273 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3052e-05 2024-02-09T17:55:51.273 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.273 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.273 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:55:51.273 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.620600+0000", 2024-02-09T17:55:51.273 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5610000000000001e-06 2024-02-09T17:55:51.273 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.273 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.273 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:55:51.274 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.620656+0000", 2024-02-09T17:55:51.274 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5738000000000001e-05 2024-02-09T17:55:51.274 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.274 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.274 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:55:51.274 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.620773+0000", 2024-02-09T17:55:51.274 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000117471 2024-02-09T17:55:51.274 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.274 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.274 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-02-09T17:55:51.274 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.620885+0000", 2024-02-09T17:55:51.274 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000111544 2024-02-09T17:55:51.274 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.274 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.274 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:51.275 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.626153+0000", 2024-02-09T17:55:51.275 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0052686039999999996 2024-02-09T17:55:51.275 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.275 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.275 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:55:51.275 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.628281+0000", 2024-02-09T17:55:51.275 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021282380000000002 2024-02-09T17:55:51.275 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.275 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.275 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:51.275 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.824552+0000", 2024-02-09T17:55:51.275 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19627041200000001 2024-02-09T17:55:51.275 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.275 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.275 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:55:51.276 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.824579+0000", 2024-02-09T17:55:51.276 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6631e-05 2024-02-09T17:55:51.276 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.276 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.276 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:55:51.276 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.824597+0000", 2024-02-09T17:55:51.276 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8372999999999998e-05 2024-02-09T17:55:51.276 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:51.276 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:55:51.276 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:51.276 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.276 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.276 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25837.0:254361 6.5 6:addcacb0:::benchmark_data_smithi179_126206_object123980:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:55:51.276 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:49:52.655698+0000", 2024-02-09T17:55:51.277 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 358.55421731600001, 2024-02-09T17:55:51.277 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16599267300000001, 2024-02-09T17:55:51.277 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:55:51.277 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:51.277 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:55:51.277 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25837", 2024-02-09T17:55:51.277 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:55:51.277 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 254361 2024-02-09T17:55:51.277 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.277 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:55:51.277 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.277 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:55:51.277 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.655698+0000", 2024-02-09T17:55:51.277 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:51.277 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.278 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.278 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:55:51.278 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.655698+0000", 2024-02-09T17:55:51.278 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:51.278 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.278 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.278 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:55:51.278 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.655699+0000", 2024-02-09T17:55:51.278 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.283e-06 2024-02-09T17:55:51.278 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.278 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.278 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:55:51.278 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.655705+0000", 2024-02-09T17:55:51.278 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2789999999999998e-06 2024-02-09T17:55:51.278 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.278 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.279 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:55:51.279 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.655716+0000", 2024-02-09T17:55:51.279 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1877e-05 2024-02-09T17:55:51.279 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.279 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.279 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:55:51.279 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.655725+0000", 2024-02-09T17:55:51.279 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4419999999999998e-06 2024-02-09T17:55:51.279 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.279 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.279 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:55:51.279 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.655768+0000", 2024-02-09T17:55:51.279 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3458000000000003e-05 2024-02-09T17:55:51.279 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.279 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.280 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:55:51.280 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.655849+0000", 2024-02-09T17:55:51.280 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0966999999999999e-05 2024-02-09T17:55:51.280 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.280 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.280 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-02-09T17:55:51.280 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.655949+0000", 2024-02-09T17:55:51.280 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9517000000000003e-05 2024-02-09T17:55:51.280 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.280 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.280 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:55:51.280 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.659984+0000", 2024-02-09T17:55:51.280 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0040348299999999997 2024-02-09T17:55:51.280 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.280 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.281 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:51.281 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.668934+0000", 2024-02-09T17:55:51.281 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0089507319999999994 2024-02-09T17:55:51.281 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.281 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.281 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:51.281 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.821610+0000", 2024-02-09T17:55:51.281 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.15267540900000001 2024-02-09T17:55:51.281 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.281 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.281 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:55:51.281 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.821658+0000", 2024-02-09T17:55:51.281 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8593e-05 2024-02-09T17:55:51.281 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.282 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.282 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:55:51.282 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.821691+0000", 2024-02-09T17:55:51.282 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2286000000000002e-05 2024-02-09T17:55:51.282 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:51.282 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:55:51.282 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:51.282 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.282 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.282 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25837.0:254364 6.5 6:a9cce28d:::benchmark_data_smithi179_126206_object123983:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:55:51.282 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:49:52.661673+0000", 2024-02-09T17:55:51.282 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 358.54824258899998, 2024-02-09T17:55:51.282 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17981395999999999, 2024-02-09T17:55:51.282 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:55:51.282 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:51.283 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:55:51.283 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25837", 2024-02-09T17:55:51.283 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:55:51.283 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 254364 2024-02-09T17:55:51.283 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.283 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:55:51.283 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.283 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:55:51.283 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.661673+0000", 2024-02-09T17:55:51.283 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:51.283 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.283 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.283 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:55:51.283 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.661673+0000", 2024-02-09T17:55:51.283 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:51.284 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.284 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.284 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:55:51.284 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.661674+0000", 2024-02-09T17:55:51.284 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.063e-06 2024-02-09T17:55:51.284 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.284 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.284 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:55:51.284 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.661679+0000", 2024-02-09T17:55:51.284 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1800000000000004e-06 2024-02-09T17:55:51.284 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.284 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.284 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:55:51.284 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.661690+0000", 2024-02-09T17:55:51.285 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1096e-05 2024-02-09T17:55:51.285 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.285 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.285 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:55:51.285 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.661697+0000", 2024-02-09T17:55:51.285 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.596e-06 2024-02-09T17:55:51.285 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.285 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.285 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:55:51.285 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.661740+0000", 2024-02-09T17:55:51.285 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3582999999999999e-05 2024-02-09T17:55:51.285 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.285 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.285 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:55:51.286 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.661835+0000", 2024-02-09T17:55:51.286 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4518999999999994e-05 2024-02-09T17:55:51.286 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.286 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.286 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-02-09T17:55:51.286 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.661916+0000", 2024-02-09T17:55:51.286 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1212999999999996e-05 2024-02-09T17:55:51.286 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.286 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.286 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:55:51.286 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.666085+0000", 2024-02-09T17:55:51.286 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041686609999999997 2024-02-09T17:55:51.286 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.286 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.286 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:51.287 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.669054+0000", 2024-02-09T17:55:51.287 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0029690010000000002 2024-02-09T17:55:51.287 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.287 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.287 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:51.287 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.841347+0000", 2024-02-09T17:55:51.287 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17229327999999999 2024-02-09T17:55:51.287 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.287 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.287 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:55:51.287 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.841404+0000", 2024-02-09T17:55:51.287 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6719999999999999e-05 2024-02-09T17:55:51.287 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.287 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.287 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:55:51.287 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.841487+0000", 2024-02-09T17:55:51.288 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3047999999999995e-05 2024-02-09T17:55:51.288 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:51.288 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:55:51.288 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:51.288 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.288 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.288 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25837.0:254374 6.3 6:c071e061:::benchmark_data_smithi179_126206_object123993:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:55:51.288 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:49:52.675875+0000", 2024-02-09T17:55:51.288 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 358.53404048099998, 2024-02-09T17:55:51.288 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.151919742, 2024-02-09T17:55:51.288 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:55:51.288 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:51.288 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:55:51.289 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25837", 2024-02-09T17:55:51.289 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:55:51.289 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 254374 2024-02-09T17:55:51.289 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.289 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:55:51.289 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.289 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:55:51.289 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.675875+0000", 2024-02-09T17:55:51.289 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:51.289 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.289 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.289 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:55:51.289 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.675875+0000", 2024-02-09T17:55:51.289 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:51.289 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.290 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.290 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:55:51.290 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.675877+0000", 2024-02-09T17:55:51.290 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8780000000000001e-06 2024-02-09T17:55:51.290 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.290 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.290 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:55:51.290 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.675883+0000", 2024-02-09T17:55:51.290 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0560000000000001e-06 2024-02-09T17:55:51.290 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.290 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.290 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:55:51.290 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.675898+0000", 2024-02-09T17:55:51.290 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4976e-05 2024-02-09T17:55:51.291 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.291 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.291 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:55:51.291 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.675906+0000", 2024-02-09T17:55:51.291 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1210000000000007e-06 2024-02-09T17:55:51.291 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.291 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.291 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:55:51.291 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.675972+0000", 2024-02-09T17:55:51.291 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5710999999999995e-05 2024-02-09T17:55:51.291 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.291 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.291 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:55:51.291 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.676127+0000", 2024-02-09T17:55:51.291 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015538300000000001 2024-02-09T17:55:51.292 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.292 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.292 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-02-09T17:55:51.292 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.676274+0000", 2024-02-09T17:55:51.292 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000147078 2024-02-09T17:55:51.292 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.292 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.292 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:55:51.292 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.680551+0000", 2024-02-09T17:55:51.292 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042773500000000001 2024-02-09T17:55:51.292 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.292 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.292 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:51.292 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.680811+0000", 2024-02-09T17:55:51.293 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025923599999999998 2024-02-09T17:55:51.293 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.293 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.293 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:51.293 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.827702+0000", 2024-02-09T17:55:51.293 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.14689158099999999 2024-02-09T17:55:51.293 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.293 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.293 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:55:51.293 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.827755+0000", 2024-02-09T17:55:51.293 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2315000000000003e-05 2024-02-09T17:55:51.293 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.293 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.293 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:55:51.293 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.827795+0000", 2024-02-09T17:55:51.294 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0057e-05 2024-02-09T17:55:51.294 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:51.294 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:55:51.294 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:51.294 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.294 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.294 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25837.0:254406 6.3 6:c56758bf:::benchmark_data_smithi179_126206_object124025:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:55:51.294 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:49:52.728445+0000", 2024-02-09T17:55:51.294 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 358.481470731, 2024-02-09T17:55:51.294 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.10753573299999999, 2024-02-09T17:55:51.294 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:55:51.294 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:51.294 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:55:51.294 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25837", 2024-02-09T17:55:51.294 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:55:51.295 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 254406 2024-02-09T17:55:51.295 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.295 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:55:51.295 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.295 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:55:51.295 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.728445+0000", 2024-02-09T17:55:51.295 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:51.295 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.295 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.295 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:55:51.295 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.728445+0000", 2024-02-09T17:55:51.295 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:51.295 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.295 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.295 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:55:51.295 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.728445+0000", 2024-02-09T17:55:51.296 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6899999999999998e-07 2024-02-09T17:55:51.296 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.296 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.296 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:55:51.296 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.728447+0000", 2024-02-09T17:55:51.296 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9549999999999999e-06 2024-02-09T17:55:51.296 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.296 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.296 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:55:51.296 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.728452+0000", 2024-02-09T17:55:51.296 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.412e-06 2024-02-09T17:55:51.296 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.296 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.296 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:55:51.297 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.728456+0000", 2024-02-09T17:55:51.297 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8689999999999997e-06 2024-02-09T17:55:51.297 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.297 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.297 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:55:51.297 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.732702+0000", 2024-02-09T17:55:51.297 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042453489999999998 2024-02-09T17:55:51.297 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.297 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.297 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:55:51.297 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.732837+0000", 2024-02-09T17:55:51.297 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013582899999999999 2024-02-09T17:55:51.297 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.297 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.297 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-02-09T17:55:51.297 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.732964+0000", 2024-02-09T17:55:51.298 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012613300000000001 2024-02-09T17:55:51.298 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.298 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.298 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:51.298 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.738024+0000", 2024-02-09T17:55:51.298 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0050601520000000001 2024-02-09T17:55:51.298 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.298 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.298 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:55:51.298 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.738069+0000", 2024-02-09T17:55:51.298 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5358000000000002e-05 2024-02-09T17:55:51.298 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.298 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.298 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:51.298 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.835884+0000", 2024-02-09T17:55:51.299 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.097814605999999998 2024-02-09T17:55:51.299 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.299 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.299 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:55:51.299 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.835939+0000", 2024-02-09T17:55:51.299 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5676e-05 2024-02-09T17:55:51.299 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.299 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.299 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:55:51.299 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.835980+0000", 2024-02-09T17:55:51.299 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1025000000000001e-05 2024-02-09T17:55:51.299 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:51.299 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:55:51.299 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:51.299 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.300 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.300 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25837.0:254410 6.5 6:a137bf3f:::benchmark_data_smithi179_126206_object124029:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:55:51.300 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:49:52.733427+0000", 2024-02-09T17:55:51.300 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 358.47648814399997, 2024-02-09T17:55:51.300 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.108200144, 2024-02-09T17:55:51.300 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:55:51.300 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:51.300 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:55:51.300 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25837", 2024-02-09T17:55:51.300 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:55:51.300 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 254410 2024-02-09T17:55:51.300 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.300 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:55:51.300 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.301 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:55:51.301 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.733427+0000", 2024-02-09T17:55:51.301 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:51.301 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.301 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.301 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:55:51.301 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.733427+0000", 2024-02-09T17:55:51.301 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:51.301 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.301 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.301 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:55:51.301 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.733428+0000", 2024-02-09T17:55:51.301 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.22e-06 2024-02-09T17:55:51.301 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.301 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.302 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:55:51.302 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.733432+0000", 2024-02-09T17:55:51.302 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3459999999999998e-06 2024-02-09T17:55:51.302 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.302 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.302 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:55:51.302 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.733442+0000", 2024-02-09T17:55:51.302 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0719e-05 2024-02-09T17:55:51.302 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.302 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.302 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:55:51.302 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.733451+0000", 2024-02-09T17:55:51.302 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2609999999999993e-06 2024-02-09T17:55:51.302 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.302 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.303 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:55:51.303 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.733496+0000", 2024-02-09T17:55:51.303 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5549e-05 2024-02-09T17:55:51.303 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.303 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.303 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:55:51.303 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.733579+0000", 2024-02-09T17:55:51.303 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3036999999999998e-05 2024-02-09T17:55:51.303 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.303 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.303 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-02-09T17:55:51.303 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.733651+0000", 2024-02-09T17:55:51.303 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1594999999999995e-05 2024-02-09T17:55:51.303 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.304 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.304 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:55:51.304 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.737521+0000", 2024-02-09T17:55:51.304 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0038698840000000001 2024-02-09T17:55:51.304 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.304 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.304 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:51.304 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.737808+0000", 2024-02-09T17:55:51.304 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00028684499999999998 2024-02-09T17:55:51.304 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.304 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.304 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:51.304 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.841538+0000", 2024-02-09T17:55:51.304 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.103730278 2024-02-09T17:55:51.304 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.305 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.305 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:55:51.305 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.841591+0000", 2024-02-09T17:55:51.305 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2840999999999998e-05 2024-02-09T17:55:51.305 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.305 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.305 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:55:51.305 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.841627+0000", 2024-02-09T17:55:51.305 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6569000000000002e-05 2024-02-09T17:55:51.305 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:51.305 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:55:51.305 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:51.305 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.305 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.305 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.16440.0:83376 7.8 7:14268f94:::benchmark_data_smithi179_135804_object83375:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e686)", 2024-02-09T17:55:51.305 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:51:18.486683+0000", 2024-02-09T17:55:51.306 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 272.72323183600002, 2024-02-09T17:55:51.306 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.113171218, 2024-02-09T17:55:51.306 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:55:51.306 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:51.306 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:55:51.306 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.16440", 2024-02-09T17:55:51.306 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:55:51.306 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 83376 2024-02-09T17:55:51.306 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.306 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:55:51.306 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.306 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:55:51.306 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.486683+0000", 2024-02-09T17:55:51.306 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:51.306 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.306 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.307 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:55:51.307 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.486683+0000", 2024-02-09T17:55:51.307 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:51.307 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.307 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.307 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:55:51.307 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.486685+0000", 2024-02-09T17:55:51.307 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.271e-06 2024-02-09T17:55:51.307 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.307 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.307 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:55:51.307 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.486746+0000", 2024-02-09T17:55:51.307 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1264000000000002e-05 2024-02-09T17:55:51.307 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.307 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.307 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:55:51.308 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.486760+0000", 2024-02-09T17:55:51.308 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.362e-05 2024-02-09T17:55:51.308 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.308 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.308 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:55:51.308 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.486767+0000", 2024-02-09T17:55:51.308 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6159999999999996e-06 2024-02-09T17:55:51.308 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.308 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.308 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:55:51.308 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.486821+0000", 2024-02-09T17:55:51.308 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3344999999999998e-05 2024-02-09T17:55:51.308 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.308 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.309 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:55:51.309 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.486956+0000", 2024-02-09T17:55:51.309 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013512800000000001 2024-02-09T17:55:51.309 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.309 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.309 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,4", 2024-02-09T17:55:51.309 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.487087+0000", 2024-02-09T17:55:51.309 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000131371 2024-02-09T17:55:51.309 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.309 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.309 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:55:51.309 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.490391+0000", 2024-02-09T17:55:51.309 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003304319 2024-02-09T17:55:51.309 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.309 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.310 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:51.310 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.494399+0000", 2024-02-09T17:55:51.310 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0040079709999999999 2024-02-09T17:55:51.310 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.310 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.310 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:51.310 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.599744+0000", 2024-02-09T17:55:51.310 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.10534436799999999 2024-02-09T17:55:51.310 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.310 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.310 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:55:51.310 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.599810+0000", 2024-02-09T17:55:51.310 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6365999999999994e-05 2024-02-09T17:55:51.310 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.310 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.311 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:55:51.311 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.599855+0000", 2024-02-09T17:55:51.311 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4579000000000001e-05 2024-02-09T17:55:51.311 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:51.311 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:55:51.311 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:51.311 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.311 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.311 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.16440.0:83381 7.8 7:1d33db6a:::benchmark_data_smithi179_135804_object83380:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e686)", 2024-02-09T17:55:51.311 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:51:18.495214+0000", 2024-02-09T17:55:51.311 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 272.71470083299999, 2024-02-09T17:55:51.311 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.10478098600000001, 2024-02-09T17:55:51.311 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:55:51.311 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:51.311 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:55:51.312 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.16440", 2024-02-09T17:55:51.312 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:55:51.312 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 83381 2024-02-09T17:55:51.312 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.312 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:55:51.312 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.312 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:55:51.312 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.495214+0000", 2024-02-09T17:55:51.312 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:51.312 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.312 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.312 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:55:51.312 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.495214+0000", 2024-02-09T17:55:51.312 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:51.312 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.313 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.313 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:55:51.313 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.495216+0000", 2024-02-09T17:55:51.313 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.418e-06 2024-02-09T17:55:51.313 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.313 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.313 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:55:51.313 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.495267+0000", 2024-02-09T17:55:51.313 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1190999999999998e-05 2024-02-09T17:55:51.313 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.313 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.313 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:55:51.313 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.495282+0000", 2024-02-09T17:55:51.313 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5197999999999999e-05 2024-02-09T17:55:51.313 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.314 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.314 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:55:51.314 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.495290+0000", 2024-02-09T17:55:51.314 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9030000000000004e-06 2024-02-09T17:55:51.314 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.314 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.314 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:55:51.314 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.495343+0000", 2024-02-09T17:55:51.314 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2788999999999999e-05 2024-02-09T17:55:51.314 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.314 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.314 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:55:51.314 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.495486+0000", 2024-02-09T17:55:51.314 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014301599999999999 2024-02-09T17:55:51.314 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.315 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.315 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,4", 2024-02-09T17:55:51.315 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.495624+0000", 2024-02-09T17:55:51.315 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013796500000000001 2024-02-09T17:55:51.315 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.315 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.315 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:51.315 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.501850+0000", 2024-02-09T17:55:51.315 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.006226077 2024-02-09T17:55:51.315 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.315 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.315 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:55:51.315 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.501876+0000", 2024-02-09T17:55:51.315 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5675000000000001e-05 2024-02-09T17:55:51.315 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.316 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.316 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:51.316 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.599910+0000", 2024-02-09T17:55:51.316 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.098033911000000001 2024-02-09T17:55:51.316 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.316 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.316 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:55:51.316 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.599961+0000", 2024-02-09T17:55:51.316 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1347999999999997e-05 2024-02-09T17:55:51.316 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.316 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.316 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:55:51.316 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.599995+0000", 2024-02-09T17:55:51.316 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4495000000000001e-05 2024-02-09T17:55:51.317 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:51.317 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:55:51.317 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:51.317 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.317 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.317 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.16440.0:116805 7.8 7:1a85ab55:::benchmark_data_smithi179_135804_object9014:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e706)", 2024-02-09T17:55:51.317 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:52:10.358547+0000", 2024-02-09T17:55:51.317 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 220.85136813, 2024-02-09T17:55:51.317 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.12194231799999999, 2024-02-09T17:55:51.317 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:55:51.317 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:51.317 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:55:51.317 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.16440", 2024-02-09T17:55:51.317 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:55:51.317 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 116805 2024-02-09T17:55:51.318 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.318 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:55:51.318 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.318 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:55:51.318 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:10.358547+0000", 2024-02-09T17:55:51.318 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:51.318 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.318 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.318 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:55:51.318 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:10.358547+0000", 2024-02-09T17:55:51.318 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:51.318 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.318 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.318 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:55:51.318 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:10.358549+0000", 2024-02-09T17:55:51.318 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6199999999999999e-06 2024-02-09T17:55:51.319 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.319 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.319 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:55:51.319 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:10.358556+0000", 2024-02-09T17:55:51.319 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3490000000000003e-06 2024-02-09T17:55:51.319 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.319 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.319 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:55:51.319 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:10.358574+0000", 2024-02-09T17:55:51.319 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7382000000000001e-05 2024-02-09T17:55:51.319 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.319 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.319 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:55:51.319 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:10.358582+0000", 2024-02-09T17:55:51.319 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4200000000000007e-06 2024-02-09T17:55:51.320 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.320 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.320 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:55:51.320 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:10.358634+0000", 2024-02-09T17:55:51.320 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.164e-05 2024-02-09T17:55:51.320 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.320 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.320 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:55:51.320 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:10.358730+0000", 2024-02-09T17:55:51.320 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6274999999999996e-05 2024-02-09T17:55:51.320 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.320 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.320 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,4", 2024-02-09T17:55:51.320 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:10.358804+0000", 2024-02-09T17:55:51.321 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4564e-05 2024-02-09T17:55:51.321 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.321 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.321 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:55:51.321 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:10.362801+0000", 2024-02-09T17:55:51.321 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003996131 2024-02-09T17:55:51.321 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.321 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.321 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:51.321 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:10.362991+0000", 2024-02-09T17:55:51.321 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019082999999999999 2024-02-09T17:55:51.321 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.321 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.321 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:51.321 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:10.480375+0000", 2024-02-09T17:55:51.322 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.117383336 2024-02-09T17:55:51.322 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.322 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.322 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:55:51.322 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:10.480438+0000", 2024-02-09T17:55:51.322 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3260000000000001e-05 2024-02-09T17:55:51.322 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.322 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.322 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:55:51.322 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:10.480490+0000", 2024-02-09T17:55:51.322 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1511000000000002e-05 2024-02-09T17:55:51.322 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:51.322 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:55:51.322 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:51.322 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.323 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.323 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.16440.0:155045 7.8 7:1e01ae0d:::benchmark_data_smithi179_135804_object47254:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e740)", 2024-02-09T17:55:51.323 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:53:16.377293+0000", 2024-02-09T17:55:51.323 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 154.83262276599999, 2024-02-09T17:55:51.323 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.067280977000000006, 2024-02-09T17:55:51.323 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:55:51.323 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:51.323 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:55:51.323 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.16440", 2024-02-09T17:55:51.323 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:55:51.323 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 155045 2024-02-09T17:55:51.323 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.323 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:55:51.323 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.324 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:55:51.324 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:16.377293+0000", 2024-02-09T17:55:51.324 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:51.324 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.324 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.324 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:55:51.324 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:16.377293+0000", 2024-02-09T17:55:51.324 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:51.324 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.324 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.324 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:55:51.324 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:16.377295+0000", 2024-02-09T17:55:51.324 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8229999999999999e-06 2024-02-09T17:55:51.324 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.325 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.325 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:55:51.325 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:16.377306+0000", 2024-02-09T17:55:51.325 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.076e-05 2024-02-09T17:55:51.325 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.325 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.325 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:55:51.325 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:16.377335+0000", 2024-02-09T17:55:51.325 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9298999999999998e-05 2024-02-09T17:55:51.325 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.325 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.325 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:55:51.325 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:16.377352+0000", 2024-02-09T17:55:51.325 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6626999999999999e-05 2024-02-09T17:55:51.325 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.325 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.326 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:55:51.326 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:16.377422+0000", 2024-02-09T17:55:51.326 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0190000000000004e-05 2024-02-09T17:55:51.326 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.326 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.326 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:55:51.326 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:16.377509+0000", 2024-02-09T17:55:51.326 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6955999999999999e-05 2024-02-09T17:55:51.326 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.326 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.326 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,4", 2024-02-09T17:55:51.326 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:16.377598+0000", 2024-02-09T17:55:51.326 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9052000000000001e-05 2024-02-09T17:55:51.326 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.326 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.327 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:51.327 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:16.382019+0000", 2024-02-09T17:55:51.327 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0044204659999999996 2024-02-09T17:55:51.327 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.327 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.327 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:55:51.327 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:16.382068+0000", 2024-02-09T17:55:51.327 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9410000000000003e-05 2024-02-09T17:55:51.327 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.327 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.327 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:51.327 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:16.444507+0000", 2024-02-09T17:55:51.327 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.062439348999999998 2024-02-09T17:55:51.327 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.327 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.328 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:55:51.328 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:16.444548+0000", 2024-02-09T17:55:51.328 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0086999999999998e-05 2024-02-09T17:55:51.328 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.328 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.328 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:55:51.328 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:16.444574+0000", 2024-02-09T17:55:51.328 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5958000000000001e-05 2024-02-09T17:55:51.328 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:51.328 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:55:51.328 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:51.328 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.328 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.329 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.16440.0:208650 7.8 7:1322050c:::benchmark_data_smithi179_135804_object100859:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e768)", 2024-02-09T17:55:51.329 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:54:36.886251+0000", 2024-02-09T17:55:51.329 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 74.323664237000003, 2024-02-09T17:55:51.329 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.069564574000000004, 2024-02-09T17:55:51.329 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:55:51.329 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:55:51.329 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:55:51.329 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.16440", 2024-02-09T17:55:51.329 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:55:51.329 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 208650 2024-02-09T17:55:51.329 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.329 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:55:51.329 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.329 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:55:51.329 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:36.886251+0000", 2024-02-09T17:55:51.330 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:51.330 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.330 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.330 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:55:51.330 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:36.886251+0000", 2024-02-09T17:55:51.330 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:55:51.330 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.330 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.330 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:55:51.330 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:36.886253+0000", 2024-02-09T17:55:51.330 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1380000000000001e-06 2024-02-09T17:55:51.330 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.330 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.330 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:55:51.330 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:36.886259+0000", 2024-02-09T17:55:51.331 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0719999999999996e-06 2024-02-09T17:55:51.331 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.331 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.331 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:55:51.331 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:36.886276+0000", 2024-02-09T17:55:51.331 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6860000000000001e-05 2024-02-09T17:55:51.331 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.331 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.331 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:55:51.331 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:36.886287+0000", 2024-02-09T17:55:51.331 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0757000000000001e-05 2024-02-09T17:55:51.331 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.331 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.331 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:55:51.331 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:36.886356+0000", 2024-02-09T17:55:51.332 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9047000000000005e-05 2024-02-09T17:55:51.332 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.332 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.332 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:55:51.332 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:36.886498+0000", 2024-02-09T17:55:51.332 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014212200000000001 2024-02-09T17:55:51.332 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.332 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.332 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,4", 2024-02-09T17:55:51.332 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:36.886638+0000", 2024-02-09T17:55:51.332 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000139808 2024-02-09T17:55:51.332 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.332 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.332 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:55:51.333 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:36.890893+0000", 2024-02-09T17:55:51.333 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042549900000000002 2024-02-09T17:55:51.333 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.333 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.333 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:51.333 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:36.899446+0000", 2024-02-09T17:55:51.333 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0085535909999999993 2024-02-09T17:55:51.333 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.333 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.333 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:55:51.333 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:36.955701+0000", 2024-02-09T17:55:51.333 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.056254353999999999 2024-02-09T17:55:51.333 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.333 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.333 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:55:51.334 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:36.955767+0000", 2024-02-09T17:55:51.334 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5747999999999995e-05 2024-02-09T17:55:51.334 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:55:51.334 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:55:51.334 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:55:51.334 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:36.955816+0000", 2024-02-09T17:55:51.334 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9086999999999999e-05 2024-02-09T17:55:51.334 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:51.334 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:55:51.334 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:51.334 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:55:51.334 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:55:51.334 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:55:51.335 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:55:51.392 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:51.419 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:55:51.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:51.424 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:55:51.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:51 smithi162 ceph-mon[44304]: 1.1c scrub starts 2024-02-09T17:55:51.435 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:51 smithi099 ceph-mon[39420]: 1.1c scrub starts 2024-02-09T17:55:51.524 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:55:51.580 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:51 smithi179 ceph-mon[43975]: 1.1c scrub starts 2024-02-09T17:55:51.581 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:51.596 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 25 16 53603 53587 133.916 102.188 0.00891211 0.00746131 2024-02-09T17:55:51.608 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:55:51.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:51.615 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:55:51.716 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:55:51.753 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:51.909 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:55:51.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:51.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:55:51.940 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.9 dump_ops_in_flight 2024-02-09T17:55:52.014 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:55:52.061 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:52.138 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:55:52.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:52.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:55:52.182 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":1973,"stamp":"2024-02-09T17:55:47.179697+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2781479456,"num_objects":42435,"num_object_clones":0,"num_object_copies":127305,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":42433,"num_write_kb":2715712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46742,"ondisk_log_size":46742,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":143,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":16726156,"kb_used_data":8199576,"kb_used_omap":229,"kb_used_meta":8526234,"kb_avail":1108215668,"statfs":{"total":1151940427776,"available":1134812844032,"internally_reserved":0,"allocated":8396365824,"data_stored":8390977908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235127,"internal_metadata":8730864009},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":927203328,"num_objects":14148,"num_object_clones":0,"num_object_copies":42444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14148,"num_write_kb":905472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.078227"},"pg_stats":[{"pgid":"1.1e","version":"63'139","reported_seq":898,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-02-09T17:55:41.915866+0000","last_change":"2024-02-09T17:55:17.114116+0000","last_active":"2024-02-09T17:55:41.915866+0000","last_peered":"2024-02-09T17:55:41.915866+0000","last_clean":"2024-02-09T17:55:41.915866+0000","last_became_active":"2024-02-09T17:55:08.938117+0000","last_became_peered":"2024-02-09T17:55:08.938117+0000","last_unstale":"2024-02-09T17:55:41.915866+0000","last_undegraded":"2024-02-09T17:55:41.915866+0000","last_fullsized":"2024-02-09T17:55:41.915866+0000","mapping_epoch":781,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":782,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:17.113960+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:55:17.113960+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:31.972827+0000","scrub_duration":125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,7],"acting":[11,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"63'139","reported_seq":910,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-02-09T17:55:41.916859+0000","last_change":"2024-02-09T17:54:59.163874+0000","last_active":"2024-02-09T17:55:41.916859+0000","last_peered":"2024-02-09T17:55:41.916859+0000","last_clean":"2024-02-09T17:55:41.916859+0000","last_became_active":"2024-02-09T17:43:28.594296+0000","last_became_peered":"2024-02-09T17:43:28.594296+0000","last_unstale":"2024-02-09T17:55:41.916859+0000","last_undegraded":"2024-02-09T17:55:41.916859+0000","last_fullsized":"2024-02-09T17:55:41.916859+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:54:59.163739+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:48:24.634092+0000","last_clean_scrub_stamp":"2024-02-09T17:54:59.163739+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:26.933451+0000","scrub_duration":123,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"63'139","reported_seq":1422,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-02-09T17:55:41.912246+0000","last_change":"2024-02-09T17:53:47.835143+0000","last_active":"2024-02-09T17:55:41.912246+0000","last_peered":"2024-02-09T17:55:41.912246+0000","last_clean":"2024-02-09T17:55:41.912246+0000","last_became_active":"2024-02-09T17:43:19.103584+0000","last_became_peered":"2024-02-09T17:43:19.103584+0000","last_unstale":"2024-02-09T17:55:41.912246+0000","last_undegraded":"2024-02-09T17:55:41.912246+0000","last_fullsized":"2024-02-09T17:55:41.912246+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:53:47.835005+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:23.912504+0000","last_clean_scrub_stamp":"2024-02-09T17:53:47.835005+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:55:42.240999+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"63'139","reported_seq":1228,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-02-09T17:55:41.912486+0000","last_change":"2024-02-09T17:55:11.475048+0000","last_active":"2024-02-09T17:55:41.912486+0000","last_peered":"2024-02-09T17:55:41.912486+0000","last_clean":"2024-02-09T17:55:41.912486+0000","last_became_active":"2024-02-09T17:43:19.102729+0000","last_became_peered":"2024-02-09T17:43:19.102729+0000","last_unstale":"2024-02-09T17:55:41.912486+0000","last_undegraded":"2024-02-09T17:55:41.912486+0000","last_fullsized":"2024-02-09T17:55:41.912486+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:11.474947+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:45:44.988633+0000","last_clean_scrub_stamp":"2024-02-09T17:55:11.474947+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:38.756171+0000","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"63'139","reported_seq":884,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-02-09T17:55:41.917070+0000","last_change":"2024-02-09T17:55:03.984081+0000","last_active":"2024-02-09T17:55:41.917070+0000","last_peered":"2024-02-09T17:55:41.917070+0000","last_clean":"2024-02-09T17:55:41.917070+0000","last_became_active":"2024-02-09T17:43:19.120814+0000","last_became_peered":"2024-02-09T17:43:19.120814+0000","last_unstale":"2024-02-09T17:55:41.917070+0000","last_undegraded":"2024-02-09T17:55:41.917070+0000","last_fullsized":"2024-02-09T17:55:41.917070+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:03.983998+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:55:03.983998+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:29.536408+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"63'139","reported_seq":1737,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-02-09T17:55:41.915553+0000","last_change":"2024-02-09T17:55:21.231646+0000","last_active":"2024-02-09T17:55:41.915553+0000","last_peered":"2024-02-09T17:55:41.915553+0000","last_clean":"2024-02-09T17:55:41.915553+0000","last_became_active":"2024-02-09T17:55:21.231291+0000","last_became_peered":"2024-02-09T17:55:21.231291+0000","last_unstale":"2024-02-09T17:55:41.915553+0000","last_undegraded":"2024-02-09T17:55:41.915553+0000","last_fullsized":"2024-02-09T17:55:41.915553+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":791,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:15.085677+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:30.193194+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.085677+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:27.587337+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"63'139","reported_seq":1575,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-02-09T17:55:41.919001+0000","last_change":"2024-02-09T17:54:50.053517+0000","last_active":"2024-02-09T17:55:41.919001+0000","last_peered":"2024-02-09T17:55:41.919001+0000","last_clean":"2024-02-09T17:55:41.919001+0000","last_became_active":"2024-02-09T17:43:28.585573+0000","last_became_peered":"2024-02-09T17:43:28.585573+0000","last_unstale":"2024-02-09T17:55:41.919001+0000","last_undegraded":"2024-02-09T17:55:41.919001+0000","last_fullsized":"2024-02-09T17:55:41.919001+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:54:50.053414+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:54:50.053414+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:13.442075+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"63'139","reported_seq":1021,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-02-09T17:55:41.912150+0000","last_change":"2024-02-09T17:55:04.470767+0000","last_active":"2024-02-09T17:55:41.912150+0000","last_peered":"2024-02-09T17:55:41.912150+0000","last_clean":"2024-02-09T17:55:41.912150+0000","last_became_active":"2024-02-09T17:48:46.184419+0000","last_became_peered":"2024-02-09T17:48:46.184419+0000","last_unstale":"2024-02-09T17:55:41.912150+0000","last_undegraded":"2024-02-09T17:55:41.912150+0000","last_fullsized":"2024-02-09T17:55:41.912150+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:04.470661+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:44:50.214686+0000","last_clean_scrub_stamp":"2024-02-09T17:55:04.470661+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:29.156638+0000","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"63'139","reported_seq":912,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-02-09T17:55:41.916201+0000","last_change":"2024-02-09T17:54:56.134959+0000","last_active":"2024-02-09T17:55:41.916201+0000","last_peered":"2024-02-09T17:55:41.916201+0000","last_clean":"2024-02-09T17:55:41.916201+0000","last_became_active":"2024-02-09T17:53:02.869767+0000","last_became_peered":"2024-02-09T17:53:02.869767+0000","last_unstale":"2024-02-09T17:55:41.916201+0000","last_undegraded":"2024-02-09T17:55:41.916201+0000","last_fullsized":"2024-02-09T17:55:41.916201+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:54:56.134826+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:11.970338+0000","last_clean_scrub_stamp":"2024-02-09T17:54:56.134826+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:19.204901+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"63'139","reported_seq":1325,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-02-09T17:55:41.920912+0000","last_change":"2024-02-09T17:50:17.104141+0000","last_active":"2024-02-09T17:55:41.920912+0000","last_peered":"2024-02-09T17:55:41.920912+0000","last_clean":"2024-02-09T17:55:41.920912+0000","last_became_active":"2024-02-09T17:43:19.138398+0000","last_became_peered":"2024-02-09T17:43:19.138398+0000","last_unstale":"2024-02-09T17:55:41.920912+0000","last_undegraded":"2024-02-09T17:55:41.920912+0000","last_fullsized":"2024-02-09T17:55:41.920912+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:50:17.104041+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:50:17.104041+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:55:45.474062+0000","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"63'139","reported_seq":1403,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-02-09T17:55:41.914775+0000","last_change":"2024-02-09T17:55:09.116245+0000","last_active":"2024-02-09T17:55:41.914775+0000","last_peered":"2024-02-09T17:55:41.914775+0000","last_clean":"2024-02-09T17:55:41.914775+0000","last_became_active":"2024-02-09T17:43:28.589307+0000","last_became_peered":"2024-02-09T17:43:28.589307+0000","last_unstale":"2024-02-09T17:55:41.914775+0000","last_undegraded":"2024-02-09T17:55:41.914775+0000","last_fullsized":"2024-02-09T17:55:41.914775+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:09.116128+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:38:26.968892+0000","last_clean_scrub_stamp":"2024-02-09T17:55:09.116128+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:31.309149+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"802'2535","reported_seq":2579,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-02-09T17:55:43.437282+0000","last_change":"2024-02-09T17:55:16.444774+0000","last_active":"2024-02-09T17:55:43.437282+0000","last_peered":"2024-02-09T17:55:43.437282+0000","last_clean":"2024-02-09T17:55:43.437282+0000","last_became_active":"2024-02-09T17:55:16.444412+0000","last_became_peered":"2024-02-09T17:55:16.444412+0000","last_unstale":"2024-02-09T17:55:43.437282+0000","last_undegraded":"2024-02-09T17:55:43.437282+0000","last_fullsized":"2024-02-09T17:55:43.437282+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":2535,"log_dups_size":0,"ondisk_log_size":2535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:39.138631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166133760,"num_objects":2535,"num_object_clones":0,"num_object_copies":7605,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2535,"num_write_kb":162240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,0],"acting":[10,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"63'139","reported_seq":1982,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-02-09T17:55:41.920724+0000","last_change":"2024-02-09T17:55:19.636501+0000","last_active":"2024-02-09T17:55:41.920724+0000","last_peered":"2024-02-09T17:55:41.920724+0000","last_clean":"2024-02-09T17:55:41.920724+0000","last_became_active":"2024-02-09T17:41:14.521079+0000","last_became_peered":"2024-02-09T17:41:14.521079+0000","last_unstale":"2024-02-09T17:55:41.920724+0000","last_undegraded":"2024-02-09T17:55:41.920724+0000","last_fullsized":"2024-02-09T17:55:41.920724+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:19.636423+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:55:19.636423+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:27.811356+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"802'2355","reported_seq":2399,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-02-09T17:55:42.419340+0000","last_change":"2024-02-09T17:55:16.913627+0000","last_active":"2024-02-09T17:55:42.419340+0000","last_peered":"2024-02-09T17:55:42.419340+0000","last_clean":"2024-02-09T17:55:42.419340+0000","last_became_active":"2024-02-09T17:55:16.913155+0000","last_became_peered":"2024-02-09T17:55:16.913155+0000","last_unstale":"2024-02-09T17:55:42.419340+0000","last_undegraded":"2024-02-09T17:55:42.419340+0000","last_fullsized":"2024-02-09T17:55:42.419340+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":2355,"log_dups_size":0,"ondisk_log_size":2355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:36.474670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154337280,"num_objects":2355,"num_object_clones":0,"num_object_copies":7065,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2355,"num_write_kb":150720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,1],"acting":[8,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.2","version":"63'139","reported_seq":1547,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-02-09T17:55:41.917979+0000","last_change":"2024-02-09T17:55:09.292488+0000","last_active":"2024-02-09T17:55:41.917979+0000","last_peered":"2024-02-09T17:55:41.917979+0000","last_clean":"2024-02-09T17:55:41.917979+0000","last_became_active":"2024-02-09T17:43:39.981054+0000","last_became_peered":"2024-02-09T17:43:39.981054+0000","last_unstale":"2024-02-09T17:55:41.917979+0000","last_undegraded":"2024-02-09T17:55:41.917979+0000","last_fullsized":"2024-02-09T17:55:41.917979+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:09.292403+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:55:09.292403+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:12.597930+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"802'2414","reported_seq":2458,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-02-09T17:55:42.420901+0000","last_change":"2024-02-09T17:55:16.204365+0000","last_active":"2024-02-09T17:55:42.420901+0000","last_peered":"2024-02-09T17:55:42.420901+0000","last_clean":"2024-02-09T17:55:42.420901+0000","last_became_active":"2024-02-09T17:55:16.204001+0000","last_became_peered":"2024-02-09T17:55:16.204001+0000","last_unstale":"2024-02-09T17:55:42.420901+0000","last_undegraded":"2024-02-09T17:55:42.420901+0000","last_fullsized":"2024-02-09T17:55:42.420901+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":2414,"log_dups_size":0,"ondisk_log_size":2414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:19.806947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158203904,"num_objects":2414,"num_object_clones":0,"num_object_copies":7242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2414,"num_write_kb":154496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,9],"acting":[8,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"63'139","reported_seq":1179,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-02-09T17:55:41.916253+0000","last_change":"2024-02-09T17:54:51.201051+0000","last_active":"2024-02-09T17:55:41.916253+0000","last_peered":"2024-02-09T17:55:41.916253+0000","last_clean":"2024-02-09T17:55:41.916253+0000","last_became_active":"2024-02-09T17:40:13.501711+0000","last_became_peered":"2024-02-09T17:40:13.501711+0000","last_unstale":"2024-02-09T17:55:41.916253+0000","last_undegraded":"2024-02-09T17:55:41.916253+0000","last_fullsized":"2024-02-09T17:55:41.916253+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:54:51.200917+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:46:40.684762+0000","last_clean_scrub_stamp":"2024-02-09T17:54:51.200917+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:10.063418+0000","scrub_duration":164,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"802'2559","reported_seq":2602,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-02-09T17:55:43.089278+0000","last_change":"2024-02-09T17:55:16.445066+0000","last_active":"2024-02-09T17:55:43.089278+0000","last_peered":"2024-02-09T17:55:43.089278+0000","last_clean":"2024-02-09T17:55:43.089278+0000","last_became_active":"2024-02-09T17:55:16.444743+0000","last_became_peered":"2024-02-09T17:55:16.444743+0000","last_unstale":"2024-02-09T17:55:43.089278+0000","last_undegraded":"2024-02-09T17:55:43.089278+0000","last_fullsized":"2024-02-09T17:55:43.089278+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":2559,"log_dups_size":0,"ondisk_log_size":2559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:27.459931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167706624,"num_objects":2559,"num_object_clones":0,"num_object_copies":7677,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2559,"num_write_kb":163776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,9],"acting":[3,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"63'139","reported_seq":715,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-02-09T17:55:41.915651+0000","last_change":"2024-02-09T17:55:13.048648+0000","last_active":"2024-02-09T17:55:41.915651+0000","last_peered":"2024-02-09T17:55:41.915651+0000","last_clean":"2024-02-09T17:55:41.915651+0000","last_became_active":"2024-02-09T17:54:32.415032+0000","last_became_peered":"2024-02-09T17:54:32.415032+0000","last_unstale":"2024-02-09T17:55:41.915651+0000","last_undegraded":"2024-02-09T17:55:41.915651+0000","last_fullsized":"2024-02-09T17:55:41.915651+0000","mapping_epoch":766,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":767,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:13.048497+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:55:13.048497+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:35.168553+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.f","version":"802'2830","reported_seq":2872,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-02-09T17:55:46.340492+0000","last_change":"2024-02-09T17:55:16.205753+0000","last_active":"2024-02-09T17:55:46.340492+0000","last_peered":"2024-02-09T17:55:46.340492+0000","last_clean":"2024-02-09T17:55:46.340492+0000","last_became_active":"2024-02-09T17:55:16.205394+0000","last_became_peered":"2024-02-09T17:55:16.205394+0000","last_unstale":"2024-02-09T17:55:46.340492+0000","last_undegraded":"2024-02-09T17:55:46.340492+0000","last_fullsized":"2024-02-09T17:55:46.340492+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":2830,"log_dups_size":0,"ondisk_log_size":2830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:24.386617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185466880,"num_objects":2830,"num_object_clones":0,"num_object_copies":8490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2830,"num_write_kb":181120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,1],"acting":[11,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"63'139","reported_seq":1482,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-02-09T17:55:41.915786+0000","last_change":"2024-02-09T17:55:11.015552+0000","last_active":"2024-02-09T17:55:41.915786+0000","last_peered":"2024-02-09T17:55:41.915786+0000","last_clean":"2024-02-09T17:55:41.915786+0000","last_became_active":"2024-02-09T17:43:14.024894+0000","last_became_peered":"2024-02-09T17:43:14.024894+0000","last_unstale":"2024-02-09T17:55:41.915786+0000","last_undegraded":"2024-02-09T17:55:41.915786+0000","last_fullsized":"2024-02-09T17:55:41.915786+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:11.015439+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:11.970338+0000","last_clean_scrub_stamp":"2024-02-09T17:55:11.015439+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:11.829031+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.e","version":"802'2550","reported_seq":2594,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-02-09T17:55:43.780802+0000","last_change":"2024-02-09T17:55:16.913062+0000","last_active":"2024-02-09T17:55:43.780802+0000","last_peered":"2024-02-09T17:55:43.780802+0000","last_clean":"2024-02-09T17:55:43.780802+0000","last_became_active":"2024-02-09T17:55:16.912769+0000","last_became_peered":"2024-02-09T17:55:16.912769+0000","last_unstale":"2024-02-09T17:55:43.780802+0000","last_undegraded":"2024-02-09T17:55:43.780802+0000","last_fullsized":"2024-02-09T17:55:43.780802+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":2550,"log_dups_size":0,"ondisk_log_size":2550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:27.866364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167116800,"num_objects":2550,"num_object_clones":0,"num_object_copies":7650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2550,"num_write_kb":163200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"63'139","reported_seq":1106,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-02-09T17:55:41.916312+0000","last_change":"2024-02-09T17:55:05.394372+0000","last_active":"2024-02-09T17:55:41.916312+0000","last_peered":"2024-02-09T17:55:41.916312+0000","last_clean":"2024-02-09T17:55:41.916312+0000","last_became_active":"2024-02-09T17:52:32.292519+0000","last_became_peered":"2024-02-09T17:52:32.292519+0000","last_unstale":"2024-02-09T17:55:41.916312+0000","last_undegraded":"2024-02-09T17:55:41.916312+0000","last_fullsized":"2024-02-09T17:55:41.916312+0000","mapping_epoch":716,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":717,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:05.394288+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:55:05.394288+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:30.532392+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.d","version":"802'2757","reported_seq":2801,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-02-09T17:55:46.336345+0000","last_change":"2024-02-09T17:55:16.445197+0000","last_active":"2024-02-09T17:55:46.336345+0000","last_peered":"2024-02-09T17:55:46.336345+0000","last_clean":"2024-02-09T17:55:46.336345+0000","last_became_active":"2024-02-09T17:55:16.444812+0000","last_became_peered":"2024-02-09T17:55:16.444812+0000","last_unstale":"2024-02-09T17:55:46.336345+0000","last_undegraded":"2024-02-09T17:55:46.336345+0000","last_fullsized":"2024-02-09T17:55:46.336345+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":2757,"log_dups_size":0,"ondisk_log_size":2757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:43.526067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180682752,"num_objects":2757,"num_object_clones":0,"num_object_copies":8271,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2757,"num_write_kb":176448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,2],"acting":[11,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"63'139","reported_seq":1561,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-02-09T17:55:41.915799+0000","last_change":"2024-02-09T17:55:13.976245+0000","last_active":"2024-02-09T17:55:41.915799+0000","last_peered":"2024-02-09T17:55:41.915799+0000","last_clean":"2024-02-09T17:55:41.915799+0000","last_became_active":"2024-02-09T17:33:00.284924+0000","last_became_peered":"2024-02-09T17:33:00.284924+0000","last_unstale":"2024-02-09T17:55:41.915799+0000","last_undegraded":"2024-02-09T17:55:41.915799+0000","last_fullsized":"2024-02-09T17:55:41.915799+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:13.976095+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:56.105792+0000","last_clean_scrub_stamp":"2024-02-09T17:55:13.976095+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:25.434647+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.c","version":"802'2893","reported_seq":2937,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-02-09T17:55:46.912473+0000","last_change":"2024-02-09T17:55:16.207610+0000","last_active":"2024-02-09T17:55:46.912473+0000","last_peered":"2024-02-09T17:55:46.912473+0000","last_clean":"2024-02-09T17:55:46.912473+0000","last_became_active":"2024-02-09T17:55:16.207033+0000","last_became_peered":"2024-02-09T17:55:16.207033+0000","last_unstale":"2024-02-09T17:55:46.912473+0000","last_undegraded":"2024-02-09T17:55:46.912473+0000","last_fullsized":"2024-02-09T17:55:46.912473+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":2893,"log_dups_size":0,"ondisk_log_size":2893,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:18.430656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189595648,"num_objects":2893,"num_object_clones":0,"num_object_copies":8679,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2893,"num_write_kb":185152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.5","version":"63'139","reported_seq":1402,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-02-09T17:55:41.915631+0000","last_change":"2024-02-09T17:55:06.072812+0000","last_active":"2024-02-09T17:55:41.915631+0000","last_peered":"2024-02-09T17:55:41.915631+0000","last_clean":"2024-02-09T17:55:41.915631+0000","last_became_active":"2024-02-09T17:48:46.184307+0000","last_became_peered":"2024-02-09T17:48:46.184307+0000","last_unstale":"2024-02-09T17:55:41.915631+0000","last_undegraded":"2024-02-09T17:55:41.915631+0000","last_fullsized":"2024-02-09T17:55:41.915631+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:06.072671+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:55:06.072671+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:13.050346+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.3","version":"802'2792","reported_seq":2836,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-02-09T17:55:46.326977+0000","last_change":"2024-02-09T17:55:16.445395+0000","last_active":"2024-02-09T17:55:46.326977+0000","last_peered":"2024-02-09T17:55:46.326977+0000","last_clean":"2024-02-09T17:55:46.326977+0000","last_became_active":"2024-02-09T17:55:16.445048+0000","last_became_peered":"2024-02-09T17:55:16.445048+0000","last_unstale":"2024-02-09T17:55:46.326977+0000","last_undegraded":"2024-02-09T17:55:46.326977+0000","last_fullsized":"2024-02-09T17:55:46.326977+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":2792,"log_dups_size":0,"ondisk_log_size":2792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:31.073410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182976512,"num_objects":2792,"num_object_clones":0,"num_object_copies":8376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2792,"num_write_kb":178688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"63'139","reported_seq":1483,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-02-09T17:55:41.916728+0000","last_change":"2024-02-09T17:54:58.972744+0000","last_active":"2024-02-09T17:55:41.916728+0000","last_peered":"2024-02-09T17:55:41.916728+0000","last_clean":"2024-02-09T17:55:41.916728+0000","last_became_active":"2024-02-09T17:43:14.041291+0000","last_became_peered":"2024-02-09T17:43:14.041291+0000","last_unstale":"2024-02-09T17:55:41.916728+0000","last_undegraded":"2024-02-09T17:55:41.916728+0000","last_fullsized":"2024-02-09T17:55:41.916728+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:54:58.972617+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:49:39.575571+0000","last_clean_scrub_stamp":"2024-02-09T17:54:58.972617+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:09.245500+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.2","version":"802'2515","reported_seq":2559,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-02-09T17:55:43.443105+0000","last_change":"2024-02-09T17:55:16.914203+0000","last_active":"2024-02-09T17:55:43.443105+0000","last_peered":"2024-02-09T17:55:43.443105+0000","last_clean":"2024-02-09T17:55:43.443105+0000","last_became_active":"2024-02-09T17:55:16.913786+0000","last_became_peered":"2024-02-09T17:55:16.913786+0000","last_unstale":"2024-02-09T17:55:43.443105+0000","last_undegraded":"2024-02-09T17:55:43.443105+0000","last_fullsized":"2024-02-09T17:55:43.443105+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":2515,"log_dups_size":0,"ondisk_log_size":2515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:17.666318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164823040,"num_objects":2515,"num_object_clones":0,"num_object_copies":7545,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2515,"num_write_kb":160960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,4],"acting":[10,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.b","version":"63'139","reported_seq":851,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-02-09T17:55:41.919734+0000","last_change":"2024-02-09T17:55:10.144538+0000","last_active":"2024-02-09T17:55:41.919734+0000","last_peered":"2024-02-09T17:55:41.919734+0000","last_clean":"2024-02-09T17:55:41.919734+0000","last_became_active":"2024-02-09T17:55:08.938203+0000","last_became_peered":"2024-02-09T17:55:08.938203+0000","last_unstale":"2024-02-09T17:55:41.919734+0000","last_undegraded":"2024-02-09T17:55:41.919734+0000","last_fullsized":"2024-02-09T17:55:41.919734+0000","mapping_epoch":781,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":782,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:10.144404+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:55:10.144404+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:25.671268+0000","scrub_duration":181,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.1","version":"802'2688","reported_seq":2732,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-02-09T17:55:45.413574+0000","last_change":"2024-02-09T17:55:16.205903+0000","last_active":"2024-02-09T17:55:45.413574+0000","last_peered":"2024-02-09T17:55:45.413574+0000","last_clean":"2024-02-09T17:55:45.413574+0000","last_became_active":"2024-02-09T17:55:16.205574+0000","last_became_peered":"2024-02-09T17:55:16.205574+0000","last_unstale":"2024-02-09T17:55:45.413574+0000","last_undegraded":"2024-02-09T17:55:45.413574+0000","last_fullsized":"2024-02-09T17:55:45.413574+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":2688,"log_dups_size":0,"ondisk_log_size":2688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:24.914406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176160768,"num_objects":2688,"num_object_clones":0,"num_object_copies":8064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2688,"num_write_kb":172032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,2],"acting":[1,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"63'139","reported_seq":1133,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-02-09T17:55:41.914713+0000","last_change":"2024-02-09T17:55:10.313343+0000","last_active":"2024-02-09T17:55:41.914713+0000","last_peered":"2024-02-09T17:55:41.914713+0000","last_clean":"2024-02-09T17:55:41.914713+0000","last_became_active":"2024-02-09T17:46:47.013853+0000","last_became_peered":"2024-02-09T17:46:47.013853+0000","last_unstale":"2024-02-09T17:55:41.914713+0000","last_undegraded":"2024-02-09T17:55:41.914713+0000","last_fullsized":"2024-02-09T17:55:41.914713+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:10.313268+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:55:10.313268+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:24.198160+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"802'2579","reported_seq":2623,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-02-09T17:55:43.123823+0000","last_change":"2024-02-09T17:55:16.444927+0000","last_active":"2024-02-09T17:55:43.123823+0000","last_peered":"2024-02-09T17:55:43.123823+0000","last_clean":"2024-02-09T17:55:43.123823+0000","last_became_active":"2024-02-09T17:55:16.444683+0000","last_became_peered":"2024-02-09T17:55:16.444683+0000","last_unstale":"2024-02-09T17:55:43.123823+0000","last_undegraded":"2024-02-09T17:55:43.123823+0000","last_fullsized":"2024-02-09T17:55:43.123823+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":2579,"log_dups_size":0,"ondisk_log_size":2579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:37.733967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169017344,"num_objects":2579,"num_object_clones":0,"num_object_copies":7737,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2579,"num_write_kb":165056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,9],"acting":[3,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"63'139","reported_seq":1988,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-02-09T17:55:41.920738+0000","last_change":"2024-02-09T17:55:13.823711+0000","last_active":"2024-02-09T17:55:41.920738+0000","last_peered":"2024-02-09T17:55:41.920738+0000","last_clean":"2024-02-09T17:55:41.920738+0000","last_became_active":"2024-02-09T17:43:28.590237+0000","last_became_peered":"2024-02-09T17:43:28.590237+0000","last_unstale":"2024-02-09T17:55:41.920738+0000","last_undegraded":"2024-02-09T17:55:41.920738+0000","last_fullsized":"2024-02-09T17:55:41.920738+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:13.823598+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:45:47.710252+0000","last_clean_scrub_stamp":"2024-02-09T17:55:13.823598+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:41.832421+0000","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.7","version":"802'2689","reported_seq":2733,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-02-09T17:55:45.403450+0000","last_change":"2024-02-09T17:55:16.206804+0000","last_active":"2024-02-09T17:55:45.403450+0000","last_peered":"2024-02-09T17:55:45.403450+0000","last_clean":"2024-02-09T17:55:45.403450+0000","last_became_active":"2024-02-09T17:55:16.206561+0000","last_became_peered":"2024-02-09T17:55:16.206561+0000","last_unstale":"2024-02-09T17:55:45.403450+0000","last_undegraded":"2024-02-09T17:55:45.403450+0000","last_fullsized":"2024-02-09T17:55:45.403450+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":2689,"log_dups_size":0,"ondisk_log_size":2689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:41.254300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176226304,"num_objects":2689,"num_object_clones":0,"num_object_copies":8067,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2689,"num_write_kb":172096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.e","version":"63'139","reported_seq":1445,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-02-09T17:55:41.917184+0000","last_change":"2024-02-09T17:55:08.044579+0000","last_active":"2024-02-09T17:55:41.917184+0000","last_peered":"2024-02-09T17:55:41.917184+0000","last_clean":"2024-02-09T17:55:41.917184+0000","last_became_active":"2024-02-09T17:43:14.036583+0000","last_became_peered":"2024-02-09T17:43:14.036583+0000","last_unstale":"2024-02-09T17:55:41.917184+0000","last_undegraded":"2024-02-09T17:55:41.917184+0000","last_fullsized":"2024-02-09T17:55:41.917184+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:08.044477+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:14.981453+0000","last_clean_scrub_stamp":"2024-02-09T17:55:08.044477+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:28.800003+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"802'2888","reported_seq":2932,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-02-09T17:55:46.297207+0000","last_change":"2024-02-09T17:55:16.446161+0000","last_active":"2024-02-09T17:55:46.297207+0000","last_peered":"2024-02-09T17:55:46.297207+0000","last_clean":"2024-02-09T17:55:46.297207+0000","last_became_active":"2024-02-09T17:55:16.445702+0000","last_became_peered":"2024-02-09T17:55:16.445702+0000","last_unstale":"2024-02-09T17:55:46.297207+0000","last_undegraded":"2024-02-09T17:55:46.297207+0000","last_fullsized":"2024-02-09T17:55:46.297207+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":2888,"log_dups_size":0,"ondisk_log_size":2888,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:18.722073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189267968,"num_objects":2888,"num_object_clones":0,"num_object_copies":8664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2888,"num_write_kb":184832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,7],"acting":[11,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.f","version":"63'139","reported_seq":1505,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-02-09T17:55:41.925235+0000","last_change":"2024-02-09T17:55:08.355359+0000","last_active":"2024-02-09T17:55:41.925235+0000","last_peered":"2024-02-09T17:55:41.925235+0000","last_clean":"2024-02-09T17:55:41.925235+0000","last_became_active":"2024-02-09T17:33:05.316100+0000","last_became_peered":"2024-02-09T17:33:05.316100+0000","last_unstale":"2024-02-09T17:55:41.925235+0000","last_undegraded":"2024-02-09T17:55:41.925235+0000","last_fullsized":"2024-02-09T17:55:41.925235+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:08.355221+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:48:34.651647+0000","last_clean_scrub_stamp":"2024-02-09T17:55:08.355221+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:11.673721+0000","scrub_duration":144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.5","version":"802'2523","reported_seq":2567,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-02-09T17:55:43.441446+0000","last_change":"2024-02-09T17:55:16.913593+0000","last_active":"2024-02-09T17:55:43.441446+0000","last_peered":"2024-02-09T17:55:43.441446+0000","last_clean":"2024-02-09T17:55:43.441446+0000","last_became_active":"2024-02-09T17:55:16.913201+0000","last_became_peered":"2024-02-09T17:55:16.913201+0000","last_unstale":"2024-02-09T17:55:43.441446+0000","last_undegraded":"2024-02-09T17:55:43.441446+0000","last_fullsized":"2024-02-09T17:55:43.441446+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":2523,"log_dups_size":0,"ondisk_log_size":2523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:26.249705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165347328,"num_objects":2523,"num_object_clones":0,"num_object_copies":7569,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2523,"num_write_kb":161472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,1],"acting":[10,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.c","version":"63'139","reported_seq":1632,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-02-09T17:55:41.914883+0000","last_change":"2024-02-09T17:55:07.304327+0000","last_active":"2024-02-09T17:55:41.914883+0000","last_peered":"2024-02-09T17:55:41.914883+0000","last_clean":"2024-02-09T17:55:41.914883+0000","last_became_active":"2024-02-09T17:43:14.027715+0000","last_became_peered":"2024-02-09T17:43:14.027715+0000","last_unstale":"2024-02-09T17:55:41.914883+0000","last_undegraded":"2024-02-09T17:55:41.914883+0000","last_fullsized":"2024-02-09T17:55:41.914883+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:07.304236+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:14.135321+0000","last_clean_scrub_stamp":"2024-02-09T17:55:07.304236+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:16.017635+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"802'2866","reported_seq":2910,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-02-09T17:55:46.900327+0000","last_change":"2024-02-09T17:55:16.445202+0000","last_active":"2024-02-09T17:55:46.900327+0000","last_peered":"2024-02-09T17:55:46.900327+0000","last_clean":"2024-02-09T17:55:46.900327+0000","last_became_active":"2024-02-09T17:55:16.444827+0000","last_became_peered":"2024-02-09T17:55:16.444827+0000","last_unstale":"2024-02-09T17:55:46.900327+0000","last_undegraded":"2024-02-09T17:55:46.900327+0000","last_fullsized":"2024-02-09T17:55:46.900327+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":2866,"log_dups_size":0,"ondisk_log_size":2866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:20.836494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187826176,"num_objects":2866,"num_object_clones":0,"num_object_copies":8598,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2866,"num_write_kb":183424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.d","version":"63'139","reported_seq":1037,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-02-09T17:55:41.921346+0000","last_change":"2024-02-09T17:55:08.898009+0000","last_active":"2024-02-09T17:55:41.921346+0000","last_peered":"2024-02-09T17:55:41.921346+0000","last_clean":"2024-02-09T17:55:41.921346+0000","last_became_active":"2024-02-09T17:43:14.036759+0000","last_became_peered":"2024-02-09T17:43:14.036759+0000","last_unstale":"2024-02-09T17:55:41.921346+0000","last_undegraded":"2024-02-09T17:55:41.921346+0000","last_fullsized":"2024-02-09T17:55:41.921346+0000","mapping_epoch":454,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:08.897882+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:55:08.897882+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:15.119150+0000","scrub_duration":145,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"63'139","reported_seq":1208,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-02-09T17:55:41.920818+0000","last_change":"2024-02-09T17:55:10.810070+0000","last_active":"2024-02-09T17:55:41.920818+0000","last_peered":"2024-02-09T17:55:41.920818+0000","last_clean":"2024-02-09T17:55:41.920818+0000","last_became_active":"2024-02-09T17:46:47.013920+0000","last_became_peered":"2024-02-09T17:46:47.013920+0000","last_unstale":"2024-02-09T17:55:41.920818+0000","last_undegraded":"2024-02-09T17:55:41.920818+0000","last_fullsized":"2024-02-09T17:55:41.920818+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:10.809935+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:48:35.170638+0000","last_clean_scrub_stamp":"2024-02-09T17:55:10.809935+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:31.681608+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"63'139","reported_seq":1478,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-02-09T17:55:41.912583+0000","last_change":"2024-02-09T17:55:21.232444+0000","last_active":"2024-02-09T17:55:41.912583+0000","last_peered":"2024-02-09T17:55:41.912583+0000","last_clean":"2024-02-09T17:55:41.912583+0000","last_became_active":"2024-02-09T17:55:21.231902+0000","last_became_peered":"2024-02-09T17:55:21.231902+0000","last_unstale":"2024-02-09T17:55:41.912583+0000","last_undegraded":"2024-02-09T17:55:41.912583+0000","last_fullsized":"2024-02-09T17:55:41.912583+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":791,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:01.456654+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:44:51.190789+0000","last_clean_scrub_stamp":"2024-02-09T17:55:01.456654+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:22.186440+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.12","version":"63'139","reported_seq":1553,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-02-09T17:55:41.916108+0000","last_change":"2024-02-09T17:54:57.092672+0000","last_active":"2024-02-09T17:55:41.916108+0000","last_peered":"2024-02-09T17:55:41.916108+0000","last_clean":"2024-02-09T17:55:41.916108+0000","last_became_active":"2024-02-09T17:46:47.015622+0000","last_became_peered":"2024-02-09T17:46:47.015622+0000","last_unstale":"2024-02-09T17:55:41.916108+0000","last_undegraded":"2024-02-09T17:55:41.916108+0000","last_fullsized":"2024-02-09T17:55:41.916108+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:54:57.092525+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:54:57.092525+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:55:58.889416+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.13","version":"63'139","reported_seq":1172,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-02-09T17:55:41.916370+0000","last_change":"2024-02-09T17:54:54.282606+0000","last_active":"2024-02-09T17:55:41.916370+0000","last_peered":"2024-02-09T17:55:41.916370+0000","last_clean":"2024-02-09T17:55:41.916370+0000","last_became_active":"2024-02-09T17:41:14.521672+0000","last_became_peered":"2024-02-09T17:41:14.521672+0000","last_unstale":"2024-02-09T17:55:41.916370+0000","last_undegraded":"2024-02-09T17:55:41.916370+0000","last_fullsized":"2024-02-09T17:55:41.916370+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:54:54.282472+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:54:54.282472+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:19.415044+0000","scrub_duration":214,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":2780889088,"num_objects":42433,"num_object_clones":0,"num_object_copies":127299,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":42433,"num_write_kb":2715712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8478195712,"data_stored":8478195712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42433,"ondisk_log_size":42433,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547057042,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1677224,"kb_used_data":892192,"kb_used_omap":1,"kb_used_meta":785022,"kb_avail":92067928,"statfs":{"total":95995035648,"available":94277558272,"internally_reserved":0,"allocated":913604608,"data_stored":913156503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":803862980},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:54:59 2024","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":2.206,"15min":1.5169999999999999},"min":{"1min":0.27100000000000002,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":4.492,"5min":20.003,"15min":20.003},"last":1.3380000000000001},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.8500000000000001,"15min":1.357},"min":{"1min":0.248,"5min":0.248,"15min":0.23100000000000001},"max":{"1min":4.8109999999999999,"5min":9.6259999999999994,"15min":9.8219999999999992},"last":1.4239999999999999}]},{"osd":1,"last update":"Fri Feb 9 17:54:59 2024","interfaces":[{"interface":"back","average":{"1min":1.5720000000000001,"5min":1.8180000000000001,"15min":1.5529999999999999},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":8.1790000000000003,"5min":9.532,"15min":11.576000000000001},"last":0.65400000000000003},{"interface":"front","average":{"1min":1.4550000000000001,"5min":1.7609999999999999,"15min":1.651},"min":{"1min":0.314,"5min":0.25,"15min":0.25},"max":{"1min":4.9100000000000001,"5min":14.287000000000001,"15min":14.287000000000001},"last":2.4329999999999998}]},{"osd":2,"last update":"Fri Feb 9 17:31:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.415,"15min":1.6859999999999999},"min":{"1min":0.378,"5min":0.247,"15min":0.247},"max":{"1min":4.4189999999999996,"5min":9.4719999999999995,"15min":9.4719999999999995},"last":2.5350000000000001},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.5149999999999999,"15min":1.4399999999999999},"min":{"1min":0.378,"5min":0.252,"15min":0.252},"max":{"1min":4.8579999999999997,"5min":7.96,"15min":7.96},"last":1.2549999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:54:59 2024","interfaces":[{"interface":"back","average":{"1min":3.669,"5min":2.3580000000000001,"15min":1.8120000000000001},"min":{"1min":0.34899999999999998,"5min":0.27900000000000003,"15min":0.248},"max":{"1min":43.344000000000001,"5min":43.344000000000001,"15min":43.344000000000001},"last":2.569},{"interface":"front","average":{"1min":3.7029999999999998,"5min":2.399,"15min":1.9330000000000001},"min":{"1min":0.44500000000000001,"5min":0.375,"15min":0.33000000000000002},"max":{"1min":43.378,"5min":43.378,"15min":43.378},"last":2.661}]},{"osd":4,"last update":"Fri Feb 9 17:54:59 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":1.746,"15min":1.762},"min":{"1min":0.254,"5min":0.254,"15min":0.246},"max":{"1min":4.7460000000000004,"5min":19.956,"15min":65.109999999999999},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.89000000000000001,"5min":1.496,"15min":1.7110000000000001},"min":{"1min":0.309,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":3.3490000000000002,"5min":9.5690000000000008,"15min":65.099000000000004},"last":0.50700000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:54:59 2024","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.5049999999999999,"15min":1.5580000000000001},"min":{"1min":0.30199999999999999,"5min":0.27500000000000002,"15min":0.252},"max":{"1min":4.782,"5min":11.119999999999999,"15min":18.878},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.86799999999999999,"5min":1.617,"15min":1.5609999999999999},"min":{"1min":0.38900000000000001,"5min":0.29199999999999998,"15min":0.28999999999999998},"max":{"1min":4.7930000000000001,"5min":20.038,"15min":20.038},"last":2.6019999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:54:59 2024","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.6419999999999999,"15min":1.625},"min":{"1min":0.33500000000000002,"5min":0.315,"15min":0.30399999999999999},"max":{"1min":3.484,"5min":9.5980000000000008,"15min":12.362},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.91000000000000003,"5min":1.4990000000000001,"15min":1.651},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":3.2109999999999999,"5min":15.409000000000001,"15min":15.409000000000001},"last":2.6880000000000002}]},{"osd":7,"last update":"Fri Feb 9 17:54:59 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":1.4670000000000001,"15min":1.641},"min":{"1min":0.39300000000000002,"5min":0.317,"15min":0.317},"max":{"1min":3.3519999999999999,"5min":15.51,"15min":18.846},"last":2.633},{"interface":"front","average":{"1min":0.88500000000000001,"5min":2.1179999999999999,"15min":1.8320000000000001},"min":{"1min":0.372,"5min":0.318,"15min":0.318},"max":{"1min":4.8390000000000004,"5min":24.373999999999999,"15min":24.373999999999999},"last":0.44}]},{"osd":8,"last update":"Fri Feb 9 17:55:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":2.1019999999999999,"15min":1.4330000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.27900000000000003},"max":{"1min":4.0279999999999996,"5min":13.574,"15min":13.574},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.97099999999999997,"5min":2.1309999999999998,"15min":1.51},"min":{"1min":0.46999999999999997,"5min":0.40999999999999998,"15min":0.32700000000000001},"max":{"1min":3.383,"5min":13.606,"15min":13.606},"last":3.1749999999999998}]},{"osd":9,"last update":"Fri Feb 9 17:55:45 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":2.4510000000000001,"15min":1.6259999999999999},"min":{"1min":0.47099999999999997,"5min":0.32900000000000001,"15min":0.29399999999999998},"max":{"1min":5.2960000000000003,"5min":22.613,"15min":22.613},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.76900000000000002,"5min":1.845,"15min":1.3740000000000001},"min":{"1min":0.49099999999999999,"5min":0.38300000000000001,"15min":0.34399999999999997},"max":{"1min":1.488,"5min":12.289,"15min":12.313000000000001},"last":0.63700000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:54:59 2024","interfaces":[{"interface":"back","average":{"1min":1.8460000000000001,"5min":2.4670000000000001,"15min":1.6779999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.32700000000000001},"max":{"1min":7.8150000000000004,"5min":17.373000000000001,"15min":17.373000000000001},"last":3.141},{"interface":"front","average":{"1min":0.96899999999999997,"5min":1.806,"15min":1.508},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":5.2699999999999996,"5min":20.233000000000001,"15min":20.233000000000001},"last":4.7069999999999999}]}]},{"osd":10,"up_from":84,"seq":360777253273,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1625260,"kb_used_data":966560,"kb_used_omap":1,"kb_used_meta":658686,"kb_avail":92119892,"statfs":{"total":95995035648,"available":94330769408,"internally_reserved":0,"allocated":989757440,"data_stored":989309335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":674494912},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:54:44 2024","interfaces":[{"interface":"back","average":{"1min":3.3199999999999998,"5min":1.6870000000000001,"15min":1.22},"min":{"1min":0.25700000000000001,"5min":0.245,"15min":0.20599999999999999},"max":{"1min":7.6349999999999998,"5min":10.532,"15min":10.532},"last":0.27200000000000002},{"interface":"front","average":{"1min":3.0259999999999998,"5min":1.6619999999999999,"15min":1.216},"min":{"1min":0.31,"5min":0.26400000000000001,"15min":0.248},"max":{"1min":9.5779999999999994,"5min":10.573,"15min":10.573},"last":0.35699999999999998}]},{"osd":1,"last update":"Fri Feb 9 17:51:04 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":1.4430000000000001,"15min":0.97699999999999998},"min":{"1min":0.307,"5min":0.246,"15min":0.23999999999999999},"max":{"1min":3.464,"5min":9.3819999999999997,"15min":9.3819999999999997},"last":1.861},{"interface":"front","average":{"1min":0.81599999999999995,"5min":1.5249999999999999,"15min":1.1279999999999999},"min":{"1min":0.29899999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":4.3049999999999997,"5min":8.9789999999999992,"15min":8.9789999999999992},"last":1.8939999999999999}]},{"osd":2,"last update":"Fri Feb 9 17:55:20 2024","interfaces":[{"interface":"back","average":{"1min":1.8400000000000001,"5min":1.96,"15min":1.5620000000000001},"min":{"1min":0.27800000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":7.399,"5min":13.215999999999999,"15min":13.215999999999999},"last":0.39700000000000002},{"interface":"front","average":{"1min":1.661,"5min":2.101,"15min":1.6870000000000001},"min":{"1min":0.46500000000000002,"5min":0.371,"15min":0.25600000000000001},"max":{"1min":5.8239999999999998,"5min":13.259,"15min":38.868000000000002},"last":0.43099999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:55:09 2024","interfaces":[{"interface":"back","average":{"1min":2.282,"5min":2.1549999999999998,"15min":1.796},"min":{"1min":0.29999999999999999,"5min":0.27700000000000002,"15min":0.26100000000000001},"max":{"1min":6.9909999999999997,"5min":14.689,"15min":35.113},"last":0.65700000000000003},{"interface":"front","average":{"1min":1.296,"5min":2.4550000000000001,"15min":2.077},"min":{"1min":0.27800000000000002,"5min":0.249,"15min":0.249},"max":{"1min":3.5449999999999999,"5min":11.209,"15min":34.359000000000002},"last":0.54400000000000004}]},{"osd":4,"last update":"Fri Feb 9 17:55:42 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":1.1220000000000001,"15min":1.181},"min":{"1min":0.43099999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":5.4509999999999996,"5min":11.705,"15min":11.705},"last":0.48699999999999999},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.2669999999999999,"15min":1.2549999999999999},"min":{"1min":0.36499999999999999,"5min":0.33900000000000002,"15min":0.27000000000000002},"max":{"1min":5.5039999999999996,"5min":11.744,"15min":11.744},"last":0.45800000000000002}]},{"osd":5,"last update":"Fri Feb 9 17:55:20 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":1.292,"15min":1.2929999999999999},"min":{"1min":0.33200000000000002,"5min":0.30099999999999999,"15min":0.29099999999999998},"max":{"1min":0.82899999999999996,"5min":10.759,"15min":10.759},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.97499999999999998,"5min":1.4039999999999999,"15min":1.504},"min":{"1min":0.46700000000000003,"5min":0.373,"15min":0.33500000000000002},"max":{"1min":5.5389999999999997,"5min":11.779,"15min":11.779},"last":0.51900000000000002}]},{"osd":6,"last update":"Fri Feb 9 17:55:24 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":1.272,"15min":1.4259999999999999},"min":{"1min":0.34599999999999997,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.77000000000000002,"5min":10.776999999999999,"15min":10.962},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.5780000000000001,"15min":1.5129999999999999},"min":{"1min":0.434,"5min":0.35099999999999998,"15min":0.32600000000000001},"max":{"1min":5.5910000000000002,"5min":12.414,"15min":12.414},"last":0.54900000000000004}]},{"osd":7,"last update":"Fri Feb 9 17:55:15 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":2.0680000000000001,"15min":3.0099999999999998},"min":{"1min":0.40799999999999997,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.976,"5min":11.288,"15min":11.288},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.877,"5min":2.0920000000000001,"15min":3.032},"min":{"1min":0.438,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":3.242,"5min":10.736000000000001,"15min":10.736000000000001},"last":0.70499999999999996}]},{"osd":8,"last update":"Fri Feb 9 17:55:09 2024","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":1.6519999999999999,"15min":1.1899999999999999},"min":{"1min":0.34200000000000003,"5min":0.32400000000000001,"15min":0.31},"max":{"1min":5.0099999999999998,"5min":10.792999999999999,"15min":10.792999999999999},"last":2.258},{"interface":"front","average":{"1min":0.66500000000000004,"5min":1.776,"15min":1.349},"min":{"1min":0.41399999999999998,"5min":0.35299999999999998,"15min":0.30599999999999999},"max":{"1min":1.4950000000000001,"5min":12.249000000000001,"15min":12.249000000000001},"last":0.60099999999999998}]},{"osd":9,"last update":"Fri Feb 9 17:55:20 2024","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":2.3220000000000001,"15min":1.419},"min":{"1min":0.373,"5min":0.34300000000000003,"15min":0.33300000000000002},"max":{"1min":4.2350000000000003,"5min":13.298,"15min":13.298},"last":2.234},{"interface":"front","average":{"1min":1.1140000000000001,"5min":2.206,"15min":1.5329999999999999},"min":{"1min":0.44,"5min":0.39600000000000002,"15min":0.36099999999999999},"max":{"1min":4.7080000000000002,"5min":10.473000000000001,"15min":10.622999999999999},"last":2.2799999999999998}]},{"osd":11,"last update":"Fri Feb 9 17:54:44 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.8080000000000001,"15min":1.2470000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.35399999999999998},"max":{"1min":5.4249999999999998,"5min":23.579000000000001,"15min":23.579000000000001},"last":1.077},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.857,"15min":1.3009999999999999},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.438},"max":{"1min":5.5650000000000004,"5min":23.619,"15min":23.619},"last":0.83899999999999997}]}]},{"osd":9,"up_from":77,"seq":330712482205,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1832872,"kb_used_data":1107552,"kb_used_omap":1,"kb_used_meta":725310,"kb_avail":91912280,"statfs":{"total":95995035648,"available":94118174720,"internally_reserved":0,"allocated":1134133248,"data_stored":1133685143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":742717888},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:55:17 2024","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.8919999999999999,"15min":1.3340000000000001},"min":{"1min":0.27900000000000003,"5min":0.23899999999999999,"15min":0.215},"max":{"1min":9.718,"5min":11.539999999999999,"15min":11.539999999999999},"last":2.444},{"interface":"front","average":{"1min":0.91200000000000003,"5min":1.454,"15min":1.359},"min":{"1min":0.28999999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.5640000000000001,"5min":10.566000000000001,"15min":11.596},"last":0.26000000000000001}]},{"osd":1,"last update":"Fri Feb 9 17:55:04 2024","interfaces":[{"interface":"back","average":{"1min":1.5309999999999999,"5min":2.278,"15min":1.7130000000000001},"min":{"1min":0.252,"5min":0.24099999999999999,"15min":0.23599999999999999},"max":{"1min":9.6170000000000009,"5min":61.031999999999996,"15min":61.031999999999996},"last":0.52500000000000002},{"interface":"front","average":{"1min":1.514,"5min":2.7530000000000001,"15min":1.7969999999999999},"min":{"1min":0.33400000000000002,"5min":0.28699999999999998,"15min":0.249},"max":{"1min":9.9039999999999999,"5min":61.970999999999997,"15min":61.970999999999997},"last":2.278}]},{"osd":2,"last update":"Fri Feb 9 17:55:35 2024","interfaces":[{"interface":"back","average":{"1min":1.4730000000000001,"5min":1.994,"15min":1.8380000000000001},"min":{"1min":0.32600000000000001,"5min":0.27600000000000002,"15min":0.26900000000000002},"max":{"1min":9.8689999999999998,"5min":10.657999999999999,"15min":10.657999999999999},"last":2.246},{"interface":"front","average":{"1min":1.292,"5min":1.74,"15min":1.6890000000000001},"min":{"1min":0.32100000000000001,"5min":0.248,"15min":0.248},"max":{"1min":8.0389999999999997,"5min":11.35,"15min":11.35},"last":0.318}]},{"osd":3,"last update":"Fri Feb 9 17:55:31 2024","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":2.1320000000000001,"15min":1.496},"min":{"1min":0.40899999999999997,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":10.064,"5min":10.537000000000001,"15min":10.537000000000001},"last":2.996},{"interface":"front","average":{"1min":1.581,"5min":2.2090000000000001,"15min":1.149},"min":{"1min":0.377,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":10.093,"5min":10.388,"15min":10.388},"last":2.891}]},{"osd":4,"last update":"Fri Feb 9 17:55:04 2024","interfaces":[{"interface":"back","average":{"1min":2.258,"5min":1.871,"15min":1.538},"min":{"1min":0.33600000000000002,"5min":0.32900000000000001,"15min":0.26800000000000002},"max":{"1min":19.414000000000001,"5min":19.414000000000001,"15min":19.414000000000001},"last":2.1890000000000001},{"interface":"front","average":{"1min":2.3180000000000001,"5min":1.623,"15min":1.286},"min":{"1min":0.32400000000000001,"5min":0.30299999999999999,"15min":0.26900000000000002},"max":{"1min":19.224,"5min":19.224,"15min":19.224},"last":1.5780000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:55:14 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.506,"15min":1.6799999999999999},"min":{"1min":0.439,"5min":0.35399999999999998,"15min":0.26900000000000002},"max":{"1min":9.7729999999999997,"5min":9.7729999999999997,"15min":77.924000000000007},"last":2.2189999999999999},{"interface":"front","average":{"1min":1.034,"5min":1.2909999999999999,"15min":1.673},"min":{"1min":0.40500000000000003,"5min":0.35299999999999998,"15min":0.28000000000000003},"max":{"1min":4.4480000000000004,"5min":10.643000000000001,"15min":77.899000000000001},"last":0.34699999999999998}]},{"osd":6,"last update":"Fri Feb 9 17:55:04 2024","interfaces":[{"interface":"back","average":{"1min":2.0990000000000002,"5min":1.542,"15min":1.546},"min":{"1min":0.39500000000000002,"5min":0.33400000000000002,"15min":0.30199999999999999},"max":{"1min":19.375,"5min":19.375,"15min":19.375},"last":0.36899999999999999},{"interface":"front","average":{"1min":2.194,"5min":1.5620000000000001,"15min":1.7310000000000001},"min":{"1min":0.53700000000000003,"5min":0.34999999999999998,"15min":0.33500000000000002},"max":{"1min":19.178000000000001,"5min":19.178000000000001,"15min":19.178000000000001},"last":1.266}]},{"osd":7,"last update":"Fri Feb 9 17:49:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.9670000000000001,"15min":1.8400000000000001},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.33400000000000002},"max":{"1min":3.7909999999999999,"5min":29.001999999999999,"15min":29.001999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":1.127,"5min":1.919,"15min":1.841},"min":{"1min":0.39400000000000002,"5min":0.373,"15min":0.32200000000000001},"max":{"1min":3.7749999999999999,"5min":28.872,"15min":28.872},"last":0.47399999999999998}]},{"osd":8,"last update":"Fri Feb 9 17:55:17 2024","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.677,"15min":1.1879999999999999},"min":{"1min":0.38600000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":9.4930000000000003,"5min":10.744,"15min":10.744},"last":0.63600000000000001},{"interface":"front","average":{"1min":1.46,"5min":1.7430000000000001,"15min":1.3799999999999999},"min":{"1min":0.52500000000000002,"5min":0.35499999999999998,"15min":0.32200000000000001},"max":{"1min":9.5289999999999999,"5min":10.768000000000001,"15min":12.885999999999999},"last":0.65700000000000003}]},{"osd":10,"last update":"Fri Feb 9 17:55:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":2.3279999999999998,"15min":1.675},"min":{"1min":0.50800000000000001,"5min":0.372,"15min":0.312},"max":{"1min":9.8109999999999999,"5min":20.773,"15min":20.773},"last":2.3090000000000002},{"interface":"front","average":{"1min":1.284,"5min":1.78,"15min":1.2889999999999999},"min":{"1min":0.39000000000000001,"5min":0.34699999999999998,"15min":0.34300000000000003},"max":{"1min":9.5879999999999992,"5min":20.637,"15min":20.637},"last":1.502}]},{"osd":11,"last update":"Fri Feb 9 17:55:31 2024","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":2.1190000000000002,"15min":1.4870000000000001},"min":{"1min":0.47799999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":9.9789999999999992,"5min":11.324,"15min":11.324},"last":3.121},{"interface":"front","average":{"1min":1.524,"5min":1.7829999999999999,"15min":1.331},"min":{"1min":0.58799999999999997,"5min":0.39000000000000001,"15min":0.36499999999999999},"max":{"1min":6.101,"5min":11.388,"15min":11.388},"last":0.60699999999999998}]}]},{"osd":8,"up_from":171,"seq":734439407947,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":853356,"kb_used_data":455328,"kb_used_omap":212,"kb_used_meta":397803,"kb_avail":92891796,"statfs":{"total":95995035648,"available":95121199104,"internally_reserved":0,"allocated":466255872,"data_stored":465807767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217579,"internal_metadata":407350805},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:55:09 2024","interfaces":[{"interface":"back","average":{"1min":2.2050000000000001,"5min":2.0430000000000001,"15min":1.2929999999999999},"min":{"1min":0.254,"5min":0.249,"15min":0.217},"max":{"1min":10.832000000000001,"5min":12.712999999999999,"15min":12.712999999999999},"last":0.377},{"interface":"front","average":{"1min":1.212,"5min":1.643,"15min":1.427},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23100000000000001},"max":{"1min":5.0579999999999998,"5min":12.725,"15min":12.725},"last":0.40400000000000003}]},{"osd":1,"last update":"Fri Feb 9 17:55:09 2024","interfaces":[{"interface":"back","average":{"1min":2.1139999999999999,"5min":1.792,"15min":1.4430000000000001},"min":{"1min":0.25800000000000001,"5min":0.23799999999999999,"15min":0.22700000000000001},"max":{"1min":7.6989999999999998,"5min":8.2550000000000008,"15min":8.718},"last":3.9399999999999999},{"interface":"front","average":{"1min":2.6219999999999999,"5min":2.0369999999999999,"15min":1.6579999999999999},"min":{"1min":0.311,"5min":0.27100000000000002,"15min":0.24099999999999999},"max":{"1min":21.399000000000001,"5min":21.399000000000001,"15min":25.027000000000001},"last":0.626}]},{"osd":2,"last update":"Fri Feb 9 17:50:34 2024","interfaces":[{"interface":"back","average":{"1min":2.222,"5min":1.722,"15min":1.391},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28000000000000003},"max":{"1min":12.227,"5min":12.227,"15min":12.227},"last":3.3730000000000002},{"interface":"front","average":{"1min":2.5049999999999999,"5min":1.665,"15min":1.1240000000000001},"min":{"1min":0.42199999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":12.747999999999999,"5min":12.747999999999999,"15min":12.747999999999999},"last":3.3029999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:55:19 2024","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":2.8889999999999998,"15min":2.9159999999999999},"min":{"1min":0.32400000000000001,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":4.5490000000000004,"5min":15.654,"15min":15.654},"last":0.80500000000000005},{"interface":"front","average":{"1min":2.0790000000000002,"5min":2.427,"15min":2.0529999999999999},"min":{"1min":0.378,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":14.35,"5min":15.609999999999999,"15min":15.609999999999999},"last":0.84099999999999997}]},{"osd":4,"last update":"Fri Feb 9 17:55:09 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":1.1419999999999999,"15min":1.242},"min":{"1min":0.36499999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":4.2729999999999997,"5min":8.7959999999999994,"15min":8.7959999999999994},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.81599999999999995,"5min":1.042,"15min":1.4279999999999999},"min":{"1min":0.36099999999999999,"5min":0.28000000000000003,"15min":0.27000000000000002},"max":{"1min":4.3079999999999998,"5min":8.8780000000000001,"15min":9.6140000000000008},"last":0.504}]},{"osd":5,"last update":"Fri Feb 9 17:55:36 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":1.6739999999999999,"15min":1.6879999999999999},"min":{"1min":0.28399999999999997,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":2.625,"5min":9.657,"15min":9.657},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.71999999999999997,"5min":1.2190000000000001,"15min":1.47},"min":{"1min":0.40100000000000002,"5min":0.30299999999999999,"15min":0.29899999999999999},"max":{"1min":1.887,"5min":9.375,"15min":9.375},"last":0.56100000000000005}]},{"osd":6,"last update":"Fri Feb 9 17:55:09 2024","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.71,"15min":1.6379999999999999},"min":{"1min":0.34499999999999997,"5min":0.313,"15min":0.29899999999999999},"max":{"1min":9.7539999999999996,"5min":9.7539999999999996,"15min":13.031000000000001},"last":0.59099999999999997},{"interface":"front","average":{"1min":1.004,"5min":1.4470000000000001,"15min":1.52},"min":{"1min":0.311,"5min":0.311,"15min":0.29599999999999999},"max":{"1min":4.0970000000000004,"5min":10.178000000000001,"15min":12.157999999999999},"last":0.78600000000000003}]},{"osd":7,"last update":"Fri Feb 9 17:55:09 2024","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.6259999999999999,"15min":1.7210000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":9.3130000000000006,"5min":9.3130000000000006,"15min":11.278},"last":0.66100000000000003},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.3580000000000001,"15min":1.7030000000000001},"min":{"1min":0.45400000000000001,"5min":0.371,"15min":0.371},"max":{"1min":9.7110000000000003,"5min":9.7110000000000003,"15min":11.313000000000001},"last":0.54400000000000004}]},{"osd":9,"last update":"Fri Feb 9 17:55:09 2024","interfaces":[{"interface":"back","average":{"1min":1.5680000000000001,"5min":2.1909999999999998,"15min":1.581},"min":{"1min":0.373,"5min":0.33800000000000002,"15min":0.26800000000000002},"max":{"1min":5.548,"5min":16.358000000000001,"15min":45.616},"last":3.0920000000000001},{"interface":"front","average":{"1min":1.6799999999999999,"5min":2.2679999999999998,"15min":1.6399999999999999},"min":{"1min":0.48399999999999999,"5min":0.376,"15min":0.314},"max":{"1min":5.577,"5min":16.422999999999998,"15min":45.718000000000004},"last":3.1280000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:55:09 2024","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.645,"15min":1.196},"min":{"1min":0.34799999999999998,"5min":0.34599999999999997,"15min":0.29699999999999999},"max":{"1min":4.1340000000000003,"5min":9.157,"15min":9.157},"last":0.70699999999999996},{"interface":"front","average":{"1min":1.135,"5min":2.1720000000000002,"15min":1.4670000000000001},"min":{"1min":0.51900000000000002,"5min":0.38700000000000001,"15min":0.34300000000000003},"max":{"1min":3.4089999999999998,"5min":13.096,"15min":13.096},"last":0.68899999999999995}]},{"osd":11,"last update":"Fri Feb 9 17:55:09 2024","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":2.262,"15min":1.5},"min":{"1min":0.46000000000000002,"5min":0.36099999999999999,"15min":0.34000000000000002},"max":{"1min":5.3029999999999999,"5min":16.466999999999999,"15min":16.466999999999999},"last":0.68799999999999994},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.696,"15min":1.357},"min":{"1min":0.39700000000000002,"5min":0.33900000000000002,"15min":0.308},"max":{"1min":5.2850000000000001,"5min":20.831,"15min":20.831},"last":0.437}]}]},{"osd":7,"up_from":57,"seq":244813136293,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1084528,"kb_used_data":363368,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":92660624,"statfs":{"total":95995035648,"available":94884478976,"internally_reserved":0,"allocated":372088832,"data_stored":371637175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":738458058},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:55:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":2.0499999999999998,"15min":1.6799999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.249},"max":{"1min":3.3650000000000002,"5min":101.167,"15min":101.167},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.71199999999999997,"5min":2.024,"15min":1.6539999999999999},"min":{"1min":0.376,"5min":0.29399999999999998,"15min":0.24099999999999999},"max":{"1min":3.3759999999999999,"5min":101.07599999999999,"15min":101.07599999999999},"last":0.64800000000000002}]},{"osd":1,"last update":"Fri Feb 9 17:55:45 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":1.641,"15min":1.6739999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.24199999999999999},"max":{"1min":3.6320000000000001,"5min":17.314,"15min":17.314},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.70599999999999996,"5min":1.3100000000000001,"15min":1.7470000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.1180000000000003,"5min":8.9960000000000004,"15min":15.276},"last":0.316}]},{"osd":2,"last update":"Fri Feb 9 17:55:45 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":1.319,"15min":1.6200000000000001},"min":{"1min":0.34899999999999998,"5min":0.30599999999999999,"15min":0.25},"max":{"1min":1.1200000000000001,"5min":9.8279999999999994,"15min":9.8279999999999994},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.90500000000000003,"5min":1.546,"15min":1.851},"min":{"1min":0.371,"5min":0.33800000000000002,"15min":0.29499999999999998},"max":{"1min":5.5670000000000002,"5min":8.3889999999999993,"15min":11.968},"last":0.628}]},{"osd":3,"last update":"Fri Feb 9 17:55:05 2024","interfaces":[{"interface":"back","average":{"1min":2.093,"5min":1.7270000000000001,"15min":2.242},"min":{"1min":0.39900000000000002,"5min":0.32800000000000001,"15min":0.25600000000000001},"max":{"1min":9.1430000000000007,"5min":11.112,"15min":78.644999999999996},"last":0.74099999999999999},{"interface":"front","average":{"1min":3.1299999999999999,"5min":1.7829999999999999,"15min":2.3660000000000001},"min":{"1min":0.52300000000000002,"5min":0.34899999999999998,"15min":0.32600000000000001},"max":{"1min":14.592000000000001,"5min":14.592000000000001,"15min":78.414000000000001},"last":0.34899999999999998}]},{"osd":4,"last update":"Fri Feb 9 17:55:45 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.83799999999999997,"15min":1.4450000000000001},"min":{"1min":0.29499999999999998,"5min":0.28100000000000003,"15min":0.23699999999999999},"max":{"1min":1.1819999999999999,"5min":4.7270000000000003,"15min":86.813000000000002},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.85699999999999998,"15min":1.6639999999999999},"min":{"1min":0.315,"5min":0.26300000000000001,"15min":0.23899999999999999},"max":{"1min":1.085,"5min":8.2880000000000003,"15min":86.751000000000005},"last":0.72099999999999997}]},{"osd":5,"last update":"Fri Feb 9 17:55:45 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.93500000000000005,"15min":1.478},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.65200000000000002,"5min":4.6539999999999999,"15min":9.6590000000000007},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":1.022,"15min":1.633},"min":{"1min":0.40899999999999997,"5min":0.35199999999999998,"15min":0.31900000000000001},"max":{"1min":0.74199999999999999,"5min":6.9379999999999997,"15min":15.228999999999999},"last":0.59999999999999998}]},{"osd":6,"last update":"Fri Feb 9 17:55:10 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":1.032,"15min":1.623},"min":{"1min":0.42199999999999999,"5min":0.33200000000000002,"15min":0.28100000000000003},"max":{"1min":2.081,"5min":6.3520000000000003,"15min":15.275},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.94999999999999996,"5min":0.92500000000000004,"15min":1.4510000000000001},"min":{"1min":0.433,"5min":0.36599999999999999,"15min":0.29199999999999998},"max":{"1min":2.5569999999999999,"5min":4.6980000000000004,"15min":18.236999999999998},"last":0.47599999999999998}]},{"osd":8,"last update":"Fri Feb 9 17:55:05 2024","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":1.228,"15min":1.4430000000000001},"min":{"1min":0.45300000000000001,"5min":0.39600000000000002,"15min":0.28999999999999998},"max":{"1min":4.3250000000000002,"5min":9.2249999999999996,"15min":11.81},"last":0.70099999999999996},{"interface":"front","average":{"1min":1.04,"5min":1.3089999999999999,"15min":1.49},"min":{"1min":0.375,"5min":0.37,"15min":0.36599999999999999},"max":{"1min":4.3369999999999997,"5min":8.3949999999999996,"15min":12.058},"last":0.432}]},{"osd":9,"last update":"Fri Feb 9 17:54:53 2024","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.7470000000000001,"15min":1.6619999999999999},"min":{"1min":0.53300000000000003,"5min":0.38900000000000001,"15min":0.32700000000000001},"max":{"1min":4.4029999999999996,"5min":8.8119999999999994,"15min":11.941000000000001},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.95399999999999996,"5min":1.659,"15min":1.732},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.379},"max":{"1min":4.4459999999999997,"5min":15.852,"15min":15.852},"last":1.3}]},{"osd":10,"last update":"Fri Feb 9 17:55:25 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":1.516,"15min":1.458},"min":{"1min":0.498,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":3.0139999999999998,"5min":9.5350000000000001,"15min":9.5350000000000001},"last":1.698},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.889,"15min":1.7969999999999999},"min":{"1min":0.59999999999999998,"5min":0.40400000000000003,"15min":0.38900000000000001},"max":{"1min":4.4240000000000004,"5min":10.263999999999999,"15min":11.878},"last":0.78800000000000003}]},{"osd":11,"last update":"Fri Feb 9 17:55:45 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":1.5469999999999999,"15min":1.5669999999999999},"min":{"1min":0.56000000000000005,"5min":0.375,"15min":0.35999999999999999},"max":{"1min":1.7270000000000001,"5min":11.961,"15min":11.961},"last":1.236},{"interface":"front","average":{"1min":0.79300000000000004,"5min":1.786,"15min":1.6950000000000001},"min":{"1min":0.52500000000000002,"5min":0.48199999999999998,"15min":0.438},"max":{"1min":1.5640000000000001,"5min":12.243,"15min":12.423999999999999},"last":2.081}]}]},{"osd":6,"up_from":49,"seq":210453397932,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":678252,"kb_used_data":1504,"kb_used_omap":1,"kb_used_meta":676734,"kb_avail":93066900,"statfs":{"total":95995035648,"available":95300505600,"internally_reserved":0,"allocated":1540096,"data_stored":1091991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":692976070},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:55:36 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":1.3899999999999999,"15min":1.4850000000000001},"min":{"1min":0.29699999999999999,"5min":0.23999999999999999,"15min":0.22800000000000001},"max":{"1min":6.048,"5min":10.356,"15min":10.481999999999999},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.84599999999999997,"5min":1.4370000000000001,"15min":1.583},"min":{"1min":0.36599999999999999,"5min":0.27400000000000002,"15min":0.24399999999999999},"max":{"1min":6.8440000000000003,"5min":10.436,"15min":10.436},"last":0.45000000000000001}]},{"osd":1,"last update":"Fri Feb 9 17:55:36 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":1.377,"15min":2.218},"min":{"1min":0.37,"5min":0.25,"15min":0.25},"max":{"1min":0.94499999999999995,"5min":9.593,"15min":79.945999999999998},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.67800000000000005,"5min":1.429,"15min":2.298},"min":{"1min":0.46300000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.97399999999999998,"5min":9.6329999999999991,"15min":80.003},"last":0.51500000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:55:34 2024","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":1.4950000000000001,"15min":2.0899999999999999},"min":{"1min":0.436,"5min":0.29999999999999999,"15min":0.28100000000000003},"max":{"1min":3.569,"5min":9.1080000000000005,"15min":84.912999999999997},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.81899999999999995,"5min":1.536,"15min":2.129},"min":{"1min":0.35099999999999998,"5min":0.34599999999999997,"15min":0.34000000000000002},"max":{"1min":3.6019999999999999,"5min":9.0850000000000009,"15min":84.942999999999998},"last":0.54700000000000004}]},{"osd":3,"last update":"Fri Feb 9 17:55:07 2024","interfaces":[{"interface":"back","average":{"1min":1.8240000000000001,"5min":1.5389999999999999,"15min":1.4910000000000001},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":9.0129999999999999,"5min":9.0129999999999999,"15min":9.0129999999999999},"last":0.54500000000000004},{"interface":"front","average":{"1min":1.841,"5min":1.8169999999999999,"15min":1.8129999999999999},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":6.6769999999999996,"5min":8.9410000000000007,"15min":8.9410000000000007},"last":0.48399999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:55:34 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.97799999999999998,"15min":1.7190000000000001},"min":{"1min":0.33200000000000002,"5min":0.26200000000000001,"15min":0.252},"max":{"1min":0.68200000000000005,"5min":8.1349999999999998,"15min":15.061999999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.51200000000000001,"5min":1.0409999999999999,"15min":1.8300000000000001},"min":{"1min":0.39300000000000002,"5min":0.314,"15min":0.26400000000000001},"max":{"1min":0.70499999999999996,"5min":8.1809999999999992,"15min":15.118},"last":0.41499999999999998}]},{"osd":5,"last update":"Fri Feb 9 17:55:34 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":1.1579999999999999,"15min":1.8129999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":1.294,"5min":7.4930000000000003,"15min":14.151999999999999},"last":0.504},{"interface":"front","average":{"1min":0.61799999999999999,"5min":1.139,"15min":1.76},"min":{"1min":0.35999999999999999,"5min":0.32800000000000001,"15min":0.26100000000000001},"max":{"1min":0.84699999999999998,"5min":8.9629999999999992,"15min":18.315999999999999},"last":0.56899999999999995}]},{"osd":7,"last update":"Fri Feb 9 17:55:10 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":1.0489999999999999,"15min":1.8029999999999999},"min":{"1min":0.33000000000000002,"5min":0.29499999999999998,"15min":0.28699999999999998},"max":{"1min":4.2119999999999997,"5min":6.343,"15min":23.265999999999998},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.91000000000000003,"15min":1.5660000000000001},"min":{"1min":0.38,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":4.2779999999999996,"5min":5.0670000000000002,"15min":12.967000000000001},"last":0.53200000000000003}]},{"osd":8,"last update":"Fri Feb 9 17:55:24 2024","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":1.258,"15min":1.048},"min":{"1min":0.438,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":4.6280000000000001,"5min":9.9480000000000004,"15min":9.9480000000000004},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.80000000000000004,"5min":1.7789999999999999,"15min":1.0629999999999999},"min":{"1min":0.54200000000000004,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.3,"5min":11.316000000000001,"15min":11.316000000000001},"last":0.64500000000000002}]},{"osd":9,"last update":"Fri Feb 9 17:55:41 2024","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":1.679,"15min":1.629},"min":{"1min":0.46600000000000003,"5min":0.41799999999999998,"15min":0.32600000000000001},"max":{"1min":2.867,"5min":10.58,"15min":10.58},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.83499999999999996,"5min":2.0139999999999998,"15min":1.774},"min":{"1min":0.499,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.792,"5min":10.609,"15min":10.609},"last":0.62}]},{"osd":10,"last update":"Fri Feb 9 17:55:10 2024","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.6659999999999999,"15min":1.776},"min":{"1min":0.41599999999999998,"5min":0.36099999999999999,"15min":0.34300000000000003},"max":{"1min":4.6100000000000003,"5min":15.701000000000001,"15min":25.370000000000001},"last":21.364999999999998},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.5820000000000001,"15min":1.8380000000000001},"min":{"1min":0.45300000000000001,"5min":0.40100000000000002,"15min":0.35099999999999998},"max":{"1min":4.2480000000000002,"5min":6.0190000000000001,"15min":26.027999999999999},"last":21.408999999999999}]},{"osd":11,"last update":"Fri Feb 9 17:55:34 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":1.6060000000000001,"15min":1.7629999999999999},"min":{"1min":0.51800000000000002,"5min":0.40200000000000002,"15min":0.372},"max":{"1min":1.359,"5min":8.4969999999999999,"15min":13.109},"last":1.0660000000000001},{"interface":"front","average":{"1min":0.82799999999999996,"5min":1.734,"15min":1.7629999999999999},"min":{"1min":0.53900000000000003,"5min":0.44500000000000001,"15min":0.373},"max":{"1min":1.383,"5min":10.788,"15min":13.164999999999999},"last":1.0940000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575300,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1943720,"kb_used_data":1037920,"kb_used_omap":1,"kb_used_meta":905790,"kb_avail":91801432,"statfs":{"total":95995035648,"available":94004666368,"internally_reserved":0,"allocated":1062830080,"data_stored":1062381975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":927529413},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:54:51 2024","interfaces":[{"interface":"back","average":{"1min":3.9260000000000002,"5min":2.0379999999999998,"15min":1.583},"min":{"1min":0.23300000000000001,"5min":0.19,"15min":0.19},"max":{"1min":14.885999999999999,"5min":14.885999999999999,"15min":14.885999999999999},"last":0.40799999999999997},{"interface":"front","average":{"1min":4.0119999999999996,"5min":2.415,"15min":1.7869999999999999},"min":{"1min":0.26100000000000001,"5min":0.22,"15min":0.214},"max":{"1min":15.411,"5min":33.389000000000003,"15min":33.389000000000003},"last":0.372}]},{"osd":2,"last update":"Fri Feb 9 17:55:15 2024","interfaces":[{"interface":"back","average":{"1min":2.1709999999999998,"5min":1.617,"15min":1.734},"min":{"1min":0.33400000000000002,"5min":0.222,"15min":0.222},"max":{"1min":9.2240000000000002,"5min":13.173,"15min":14.83},"last":0.32700000000000001},{"interface":"front","average":{"1min":2.2890000000000001,"5min":1.5109999999999999,"15min":1.5800000000000001},"min":{"1min":0.378,"5min":0.29299999999999998,"15min":0.222},"max":{"1min":10.4,"5min":11.103,"15min":11.103},"last":0.442}]},{"osd":3,"last update":"Fri Feb 9 17:55:04 2024","interfaces":[{"interface":"back","average":{"1min":3.085,"5min":3.657,"15min":3.7519999999999998},"min":{"1min":0.37,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":16.835000000000001,"5min":16.835000000000001,"15min":16.835000000000001},"last":0.34599999999999997},{"interface":"front","average":{"1min":3.4260000000000002,"5min":4.1980000000000004,"15min":4.3259999999999996},"min":{"1min":0.23999999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":15.337999999999999,"5min":15.337999999999999,"15min":15.337999999999999},"last":0.53300000000000003}]},{"osd":4,"last update":"Fri Feb 9 17:54:51 2024","interfaces":[{"interface":"back","average":{"1min":1.8140000000000001,"5min":1.2450000000000001,"15min":1.6319999999999999},"min":{"1min":0.36799999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":9.032,"5min":9.032,"15min":45.137999999999998},"last":0.47599999999999998},{"interface":"front","average":{"1min":3.1699999999999999,"5min":1.9039999999999999,"15min":1.9159999999999999},"min":{"1min":0.28199999999999997,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":15.331,"5min":15.331,"15min":44.950000000000003},"last":0.57599999999999996}]},{"osd":5,"last update":"Fri Feb 9 17:55:29 2024","interfaces":[{"interface":"back","average":{"1min":1.669,"5min":1.5740000000000001,"15min":1.786},"min":{"1min":0.41999999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":15.106999999999999,"5min":15.106999999999999,"15min":45.171999999999997},"last":0.56299999999999994},{"interface":"front","average":{"1min":1.742,"5min":1.8819999999999999,"15min":2.036},"min":{"1min":0.34999999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":15.355,"5min":15.355,"15min":45.127000000000002},"last":0.42699999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:55:42 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":1.5669999999999999,"15min":1.913},"min":{"1min":0.313,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.6749999999999998,"5min":15.154999999999999,"15min":33.338000000000001},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.85099999999999998,"5min":1.625,"15min":1.9770000000000001},"min":{"1min":0.371,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":5.7279999999999998,"5min":15.253,"15min":33.286999999999999},"last":0.68200000000000005}]},{"osd":7,"last update":"Fri Feb 9 17:55:42 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":1.681,"15min":2.0449999999999999},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.34399999999999997},"max":{"1min":5.7000000000000002,"5min":15.287000000000001,"15min":45.740000000000002},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.7250000000000001,"15min":2.1200000000000001},"min":{"1min":0.48799999999999999,"5min":0.46899999999999997,"15min":0.309},"max":{"1min":5.782,"5min":15.311,"15min":45.792000000000002},"last":0.74199999999999999}]},{"osd":8,"last update":"Fri Feb 9 17:54:51 2024","interfaces":[{"interface":"back","average":{"1min":2.3519999999999999,"5min":1.7749999999999999,"15min":1.456},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.35599999999999998},"max":{"1min":9.1999999999999993,"5min":9.1999999999999993,"15min":9.2520000000000007},"last":2.2210000000000001},{"interface":"front","average":{"1min":2.3690000000000002,"5min":2.21,"15min":1.627},"min":{"1min":0.45500000000000002,"5min":0.43099999999999999,"15min":0.35599999999999998},"max":{"1min":9.0909999999999993,"5min":12.464,"15min":12.69},"last":0.623}]},{"osd":9,"last update":"Fri Feb 9 17:54:51 2024","interfaces":[{"interface":"back","average":{"1min":3.601,"5min":2.089,"15min":1.659},"min":{"1min":0.69699999999999995,"5min":0.42799999999999999,"15min":0.33900000000000002},"max":{"1min":20.789000000000001,"5min":20.789000000000001,"15min":20.789000000000001},"last":2.2599999999999998},{"interface":"front","average":{"1min":2.7639999999999998,"5min":2.0830000000000002,"15min":1.7749999999999999},"min":{"1min":0.438,"5min":0.39400000000000002,"15min":0.34899999999999998},"max":{"1min":9.1210000000000004,"5min":9.5239999999999991,"15min":9.5239999999999991},"last":0.72099999999999997}]},{"osd":10,"last update":"Fri Feb 9 17:55:29 2024","interfaces":[{"interface":"back","average":{"1min":1.7090000000000001,"5min":2.0910000000000002,"15min":1.879},"min":{"1min":0.55800000000000005,"5min":0.47099999999999997,"15min":0.39200000000000002},"max":{"1min":15.211,"5min":15.211,"15min":33.213999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":2.04,"5min":2.4009999999999998,"15min":1.841},"min":{"1min":0.55000000000000004,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":15.271000000000001,"5min":15.271000000000001,"15min":15.271000000000001},"last":0.63600000000000001}]},{"osd":11,"last update":"Fri Feb 9 17:55:42 2024","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":2.9529999999999998,"15min":2.0790000000000002},"min":{"1min":0.57099999999999995,"5min":0.42899999999999999,"15min":0.39100000000000001},"max":{"1min":7.234,"5min":19.648,"15min":19.648},"last":0.73299999999999998},{"interface":"front","average":{"1min":1.099,"5min":2.5209999999999999,"15min":2.0699999999999998},"min":{"1min":0.58299999999999996,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":5.7549999999999999,"5min":19.077000000000002,"15min":33.161000000000001},"last":2.73}]}]},{"osd":0,"up_from":8,"seq":34359738825,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1350696,"kb_used_data":543456,"kb_used_omap":1,"kb_used_meta":807230,"kb_avail":92394456,"statfs":{"total":95995035648,"available":94611922944,"internally_reserved":0,"allocated":556498944,"data_stored":556050839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":826603973},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 9 17:55:38 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":1.6599999999999999,"15min":1.4550000000000001},"min":{"1min":0.28199999999999997,"5min":0.23699999999999999,"15min":0.20599999999999999},"max":{"1min":0.73999999999999999,"5min":18.698,"15min":18.698},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.503,"5min":1.736,"15min":1.5469999999999999},"min":{"1min":0.38600000000000001,"5min":0.32100000000000001,"15min":0.24399999999999999},"max":{"1min":0.77800000000000002,"5min":18.742999999999999,"15min":18.742999999999999},"last":0.623}]},{"osd":2,"last update":"Fri Feb 9 17:54:58 2024","interfaces":[{"interface":"back","average":{"1min":4.5289999999999999,"5min":1.966,"15min":2.0230000000000001},"min":{"1min":0.30099999999999999,"5min":0.27700000000000002,"15min":0.245},"max":{"1min":20.013999999999999,"5min":20.013999999999999,"15min":20.013999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":4.2279999999999998,"5min":2.165,"15min":1.7350000000000001},"min":{"1min":0.28100000000000003,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":20.103999999999999,"5min":20.103999999999999,"15min":20.103999999999999},"last":0.43099999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:55:22 2024","interfaces":[{"interface":"back","average":{"1min":2.1400000000000001,"5min":2.4700000000000002,"15min":1.9470000000000001},"min":{"1min":0.442,"5min":0.26100000000000001,"15min":0.23100000000000001},"max":{"1min":8.827,"5min":20.469999999999999,"15min":36.052},"last":0.502},{"interface":"front","average":{"1min":2.2669999999999999,"5min":2.6429999999999998,"15min":2.1640000000000001},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.217},"max":{"1min":10.414999999999999,"5min":21.920000000000002,"15min":38.383000000000003},"last":0.38900000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:55:06 2024","interfaces":[{"interface":"back","average":{"1min":2.4350000000000001,"5min":1.766,"15min":1.498},"min":{"1min":0.35299999999999998,"5min":0.28100000000000003,"15min":0.27000000000000002},"max":{"1min":12.757,"5min":12.757,"15min":12.757},"last":0.69199999999999995},{"interface":"front","average":{"1min":2.9820000000000002,"5min":1.8100000000000001,"15min":1.873},"min":{"1min":0.35899999999999999,"5min":0.249,"15min":0.249},"max":{"1min":12.82,"5min":17.091999999999999,"15min":17.091999999999999},"last":0.47599999999999998}]},{"osd":5,"last update":"Fri Feb 9 17:55:22 2024","interfaces":[{"interface":"back","average":{"1min":1.746,"5min":1.407,"15min":1.79},"min":{"1min":0.45700000000000002,"5min":0.34300000000000003,"15min":0.30299999999999999},"max":{"1min":8.7710000000000008,"5min":9.7929999999999993,"15min":48.924999999999997},"last":0.66700000000000004},{"interface":"front","average":{"1min":1.782,"5min":1.46,"15min":1.8180000000000001},"min":{"1min":0.52400000000000002,"5min":0.38200000000000001,"15min":0.29599999999999999},"max":{"1min":8.798,"5min":9.8230000000000004,"15min":48.969999999999999},"last":0.71799999999999997}]},{"osd":6,"last update":"Fri Feb 9 17:54:47 2024","interfaces":[{"interface":"back","average":{"1min":3.3250000000000002,"5min":2.101,"15min":1.802},"min":{"1min":0.307,"5min":0.307,"15min":0.28699999999999998},"max":{"1min":15.667,"5min":15.667,"15min":15.667},"last":0.55700000000000005},{"interface":"front","average":{"1min":3.3969999999999998,"5min":2.1680000000000001,"15min":1.899},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.33300000000000002},"max":{"1min":15.702999999999999,"5min":15.702999999999999,"15min":15.702999999999999},"last":0.57799999999999996}]},{"osd":7,"last update":"Fri Feb 9 17:55:13 2024","interfaces":[{"interface":"back","average":{"1min":1.8400000000000001,"5min":1.847,"15min":1.841},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.29199999999999998},"max":{"1min":6.1829999999999998,"5min":11.436,"15min":15.271000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":1.704,"5min":1.625,"15min":1.929},"min":{"1min":0.40400000000000003,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":8.2509999999999994,"5min":17.152999999999999,"15min":17.152999999999999},"last":0.60099999999999998}]},{"osd":8,"last update":"Fri Feb 9 17:55:13 2024","interfaces":[{"interface":"back","average":{"1min":2.04,"5min":2.3359999999999999,"15min":1.8500000000000001},"min":{"1min":0.46800000000000003,"5min":0.313,"15min":0.313},"max":{"1min":8.3030000000000008,"5min":17.189,"15min":17.189},"last":0.70199999999999996},{"interface":"front","average":{"1min":1.7430000000000001,"5min":1.9570000000000001,"15min":1.72},"min":{"1min":0.439,"5min":0.36899999999999999,"15min":0.35099999999999998},"max":{"1min":8.2810000000000006,"5min":17.27,"15min":17.27},"last":0.68200000000000005}]},{"osd":9,"last update":"Fri Feb 9 17:54:58 2024","interfaces":[{"interface":"back","average":{"1min":3.407,"5min":2.6549999999999998,"15min":2.242},"min":{"1min":0.42399999999999999,"5min":0.39900000000000002,"15min":0.33800000000000002},"max":{"1min":11.134,"5min":24.757000000000001,"15min":24.757000000000001},"last":1.4890000000000001},{"interface":"front","average":{"1min":3.5649999999999999,"5min":2.5129999999999999,"15min":2.1699999999999999},"min":{"1min":0.42599999999999999,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":19.986999999999998,"5min":24.681999999999999,"15min":24.681999999999999},"last":1.502}]},{"osd":10,"last update":"Fri Feb 9 17:55:22 2024","interfaces":[{"interface":"back","average":{"1min":2.9929999999999999,"5min":2.5590000000000002,"15min":2.097},"min":{"1min":0.42499999999999999,"5min":0.41999999999999998,"15min":0.35499999999999998},"max":{"1min":15.614000000000001,"5min":15.614000000000001,"15min":31.766999999999999},"last":0.77300000000000002},{"interface":"front","average":{"1min":2.1379999999999999,"5min":2.7679999999999998,"15min":1.905},"min":{"1min":0.51800000000000002,"5min":0.40400000000000003,"15min":0.34200000000000003},"max":{"1min":11.096,"5min":17.242000000000001,"15min":28.048999999999999},"last":0.64800000000000002}]},{"osd":11,"last update":"Fri Feb 9 17:54:47 2024","interfaces":[{"interface":"back","average":{"1min":3.7599999999999998,"5min":2.6469999999999998,"15min":1.861},"min":{"1min":0.61499999999999999,"5min":0.45300000000000001,"15min":0.41599999999999998},"max":{"1min":19.951000000000001,"5min":25.994,"15min":25.994},"last":0.75900000000000001},{"interface":"front","average":{"1min":4.1870000000000003,"5min":2.6150000000000002,"15min":1.9379999999999999},"min":{"1min":0.57399999999999995,"5min":0.441,"15min":0.441},"max":{"1min":19.916,"5min":25.515999999999998,"15min":25.515999999999998},"last":0.73699999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411774,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1413488,"kb_used_data":826984,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":92331664,"statfs":{"total":95995035648,"available":94547623936,"internally_reserved":0,"allocated":846831616,"data_stored":846379959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":600570309},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":1.4510000000000001,"15min":1.006},"min":{"1min":0.29999999999999999,"5min":0.23699999999999999,"15min":0.221},"max":{"1min":4.9089999999999998,"5min":17.251000000000001,"15min":17.251000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.74399999999999999,"5min":1.611,"15min":1.113},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.22900000000000001},"max":{"1min":4.7720000000000002,"5min":23.097999999999999,"15min":23.097999999999999},"last":0.875}]},{"osd":1,"last update":"Fri Feb 9 17:55:12 2024","interfaces":[{"interface":"back","average":{"1min":2.238,"5min":1.7929999999999999,"15min":2.089},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":15.378,"5min":15.378,"15min":52.646000000000001},"last":0.32700000000000001},{"interface":"front","average":{"1min":2.198,"5min":1.8919999999999999,"15min":2.0800000000000001},"min":{"1min":0.29799999999999999,"5min":0.248,"15min":0.248},"max":{"1min":8.6440000000000001,"5min":15.092000000000001,"15min":52.683999999999997},"last":0.47899999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:55:40 2024","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":2.8069999999999999,"15min":2.1920000000000002},"min":{"1min":0.29799999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":8.7200000000000006,"5min":46.927999999999997,"15min":46.927999999999997},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.92000000000000004,"5min":2.6110000000000002,"15min":2.1970000000000001},"min":{"1min":0.28000000000000003,"5min":0.192,"15min":0.192},"max":{"1min":5.8140000000000001,"5min":45.798999999999999,"15min":45.798999999999999},"last":0.36699999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:55:40 2024","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":1.198,"15min":1.5569999999999999},"min":{"1min":0.36599999999999999,"5min":0.245,"15min":0.245},"max":{"1min":8.6099999999999994,"5min":10.365,"15min":12.398},"last":0.497},{"interface":"front","average":{"1min":0.69099999999999995,"5min":1.4079999999999999,"15min":1.536},"min":{"1min":0.26600000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":3.8149999999999999,"5min":10.529,"15min":10.888999999999999},"last":0.46400000000000002}]},{"osd":5,"last update":"Fri Feb 9 17:55:35 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":1.542,"15min":2.0209999999999999},"min":{"1min":0.34200000000000003,"5min":0.245,"15min":0.245},"max":{"1min":5.742,"5min":12.305999999999999,"15min":42.664000000000001},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.68000000000000005,"5min":1.4119999999999999,"15min":2.1760000000000002},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":3.718,"5min":10.603,"15min":43.311999999999998},"last":0.41599999999999998}]},{"osd":6,"last update":"Fri Feb 9 17:55:40 2024","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":1.5629999999999999,"15min":1.891},"min":{"1min":0.312,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":5.8879999999999999,"5min":10.573,"15min":18.518999999999998},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.97099999999999997,"5min":1.597,"15min":1.905},"min":{"1min":0.34799999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":5.915,"5min":10.731,"15min":18.481999999999999},"last":0.48599999999999999}]},{"osd":7,"last update":"Fri Feb 9 17:55:12 2024","interfaces":[{"interface":"back","average":{"1min":1.4490000000000001,"5min":1.5880000000000001,"15min":1.889},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.32400000000000001},"max":{"1min":8.5700000000000003,"5min":10.933,"15min":16.666},"last":0.53500000000000003},{"interface":"front","average":{"1min":1.167,"5min":1.413,"15min":2.0379999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":4.9630000000000001,"5min":10.316000000000001,"15min":16.638999999999999},"last":0.50700000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:55:40 2024","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":2.2589999999999999,"15min":1.754},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.316},"max":{"1min":3.7839999999999998,"5min":12.500999999999999,"15min":12.500999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.82399999999999995,"5min":2.5449999999999999,"15min":1.857},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.34399999999999997},"max":{"1min":3.8370000000000002,"5min":54.759999999999998,"15min":54.759999999999998},"last":0.52800000000000002}]},{"osd":9,"last update":"Fri Feb 9 17:54:48 2024","interfaces":[{"interface":"back","average":{"1min":2.5779999999999998,"5min":2.9630000000000001,"15min":2.1909999999999998},"min":{"1min":0.41299999999999998,"5min":0.318,"15min":0.318},"max":{"1min":8.8109999999999999,"5min":53.935000000000002,"15min":53.935000000000002},"last":0.61699999999999999},{"interface":"front","average":{"1min":2.343,"5min":2.7709999999999999,"15min":2.0169999999999999},"min":{"1min":0.36199999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":10.657,"5min":54.670999999999999,"15min":54.670999999999999},"last":0.64000000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:55:26 2024","interfaces":[{"interface":"back","average":{"1min":1.1279999999999999,"5min":2.4670000000000001,"15min":1.9319999999999999},"min":{"1min":0.46200000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":8.6720000000000006,"5min":54.039999999999999,"15min":54.039999999999999},"last":0.56699999999999995},{"interface":"front","average":{"1min":1.165,"5min":2.5049999999999999,"15min":1.972},"min":{"1min":0.505,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":8.6959999999999997,"5min":54.118000000000002,"15min":54.118000000000002},"last":0.63}]},{"osd":11,"last update":"Fri Feb 9 17:55:40 2024","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":2.7519999999999998,"15min":2.0600000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.40600000000000003},"max":{"1min":3.8660000000000001,"5min":54.799999999999997,"15min":54.799999999999997},"last":0.60299999999999998},{"interface":"front","average":{"1min":1.0780000000000001,"5min":2.8490000000000002,"15min":2.0609999999999999},"min":{"1min":0.51300000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":5.7869999999999999,"5min":54.103000000000002,"15min":54.103000000000002},"last":0.54900000000000004}]}]},{"osd":3,"up_from":27,"seq":115964117435,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1285740,"kb_used_data":466016,"kb_used_omap":1,"kb_used_meta":819710,"kb_avail":92459412,"statfs":{"total":95995035648,"available":94678437888,"internally_reserved":0,"allocated":477200384,"data_stored":476752279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":839383495},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:55:33 2024","interfaces":[{"interface":"back","average":{"1min":1.8620000000000001,"5min":2.4260000000000002,"15min":1.8999999999999999},"min":{"1min":0.26500000000000001,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":12.391,"5min":17.006,"15min":17.006},"last":2.9860000000000002},{"interface":"front","average":{"1min":1.3580000000000001,"5min":1.694,"15min":1.9430000000000001},"min":{"1min":0.24099999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":6.343,"5min":12.288,"15min":16.637},"last":0.69199999999999995}]},{"osd":1,"last update":"Fri Feb 9 17:55:01 2024","interfaces":[{"interface":"back","average":{"1min":2.306,"5min":1.9710000000000001,"15min":2.6040000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":9.2400000000000002,"5min":12.609,"15min":53.152999999999999},"last":0.34300000000000003},{"interface":"front","average":{"1min":2.4380000000000002,"5min":1.74,"15min":1.8939999999999999},"min":{"1min":0.24299999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":12.237,"5min":12.237,"15min":53.118000000000002},"last":0.39700000000000002}]},{"osd":2,"last update":"Fri Feb 9 17:55:33 2024","interfaces":[{"interface":"back","average":{"1min":1.903,"5min":2.2080000000000002,"15min":1.9330000000000001},"min":{"1min":0.27600000000000002,"5min":0.26700000000000002,"15min":0.248},"max":{"1min":12.349,"5min":17.477,"15min":17.477},"last":0.94599999999999995},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.9730000000000001,"15min":1.8720000000000001},"min":{"1min":0.26600000000000001,"5min":0.23699999999999999,"15min":0.23499999999999999},"max":{"1min":7.4059999999999997,"5min":21.103000000000002,"15min":21.103000000000002},"last":0.435}]},{"osd":4,"last update":"Fri Feb 9 17:55:33 2024","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":1.4430000000000001,"15min":1.724},"min":{"1min":0.29299999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":3.7149999999999999,"5min":12.718,"15min":12.718},"last":0.376},{"interface":"front","average":{"1min":0.77200000000000002,"5min":1.2210000000000001,"15min":1.591},"min":{"1min":0.28699999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":5.6280000000000001,"5min":5.8849999999999998,"15min":19.713999999999999},"last":0.45400000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:55:16 2024","interfaces":[{"interface":"back","average":{"1min":1.619,"5min":1.8300000000000001,"15min":2.0550000000000002},"min":{"1min":0.40799999999999997,"5min":0.33000000000000002,"15min":0.28000000000000003},"max":{"1min":5.3559999999999999,"5min":13.321,"15min":14.106},"last":0.82999999999999996},{"interface":"front","average":{"1min":1.6890000000000001,"5min":1.9279999999999999,"15min":2.0430000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.316},"max":{"1min":5.391,"5min":13.347,"15min":14.138999999999999},"last":0.80800000000000005}]},{"osd":6,"last update":"Fri Feb 9 17:55:01 2024","interfaces":[{"interface":"back","average":{"1min":2.9159999999999999,"5min":1.998,"15min":1.9990000000000001},"min":{"1min":0.39600000000000002,"5min":0.35799999999999998,"15min":0.28899999999999998},"max":{"1min":13.207000000000001,"5min":13.207000000000001,"15min":15.061999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":3.0470000000000002,"5min":2.1589999999999998,"15min":1.9830000000000001},"min":{"1min":0.375,"5min":0.375,"15min":0.36399999999999999},"max":{"1min":13.298,"5min":13.298,"15min":13.298},"last":0.78300000000000003}]},{"osd":7,"last update":"Fri Feb 9 17:54:48 2024","interfaces":[{"interface":"back","average":{"1min":2.585,"5min":1.8859999999999999,"15min":1.889},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":18.587,"5min":18.587,"15min":19.788},"last":0.61799999999999999},{"interface":"front","average":{"1min":3.2869999999999999,"5min":2.081,"15min":1.9259999999999999},"min":{"1min":0.45200000000000001,"5min":0.437,"15min":0.34899999999999998},"max":{"1min":13.239000000000001,"5min":13.239000000000001,"15min":13.239000000000001},"last":0.75700000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:45:16 2024","interfaces":[{"interface":"back","average":{"1min":1.714,"5min":1.4450000000000001,"15min":1.498},"min":{"1min":0.377,"5min":0.377,"15min":0.26800000000000002},"max":{"1min":8.6020000000000003,"5min":11.821,"15min":11.821},"last":0.67700000000000005},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.1719999999999999,"15min":1.351},"min":{"1min":0.46500000000000002,"5min":0.36199999999999999,"15min":0.34499999999999997},"max":{"1min":6.8390000000000004,"5min":6.8390000000000004,"15min":14.129},"last":0.78200000000000003}]},{"osd":9,"last update":"Fri Feb 9 17:55:01 2024","interfaces":[{"interface":"back","average":{"1min":2.1680000000000001,"5min":1.9750000000000001,"15min":1.829},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":11.039,"5min":11.417,"15min":11.851000000000001},"last":0.497},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.9430000000000001,"15min":1.8500000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":5.6900000000000004,"5min":11.492000000000001,"15min":11.907},"last":0.52700000000000002}]},{"osd":10,"last update":"Fri Feb 9 17:54:48 2024","interfaces":[{"interface":"back","average":{"1min":3.0899999999999999,"5min":2.4790000000000001,"15min":1.7549999999999999},"min":{"1min":0.40999999999999998,"5min":0.371,"15min":0.371},"max":{"1min":18.632999999999999,"5min":18.632999999999999,"15min":18.632999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":3.5,"5min":2.411,"15min":1.7929999999999999},"min":{"1min":0.5,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":13.272,"5min":13.272,"15min":19.859000000000002},"last":0.878}]},{"osd":11,"last update":"Fri Feb 9 17:55:33 2024","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":2.6070000000000002,"15min":1.899},"min":{"1min":0.44,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":7.2679999999999998,"5min":18.66,"15min":19.824000000000002},"last":1.992},{"interface":"front","average":{"1min":1.4019999999999999,"5min":2.6589999999999998,"15min":1.7889999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":7.3739999999999997,"5min":18.687999999999999,"15min":18.687999999999999},"last":1.954}]}]},{"osd":4,"up_from":32,"seq":137438953910,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1481256,"kb_used_data":705952,"kb_used_omap":1,"kb_used_meta":775294,"kb_avail":92263896,"statfs":{"total":95995035648,"available":94478229504,"internally_reserved":0,"allocated":722894848,"data_stored":722446743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":793901509},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:54:51 2024","interfaces":[{"interface":"back","average":{"1min":1.702,"5min":1.4430000000000001,"15min":1.4319999999999999},"min":{"1min":0.255,"5min":0.23899999999999999,"15min":0.222},"max":{"1min":7.7359999999999998,"5min":10.005000000000001,"15min":10.301},"last":0.38500000000000001},{"interface":"front","average":{"1min":1.7709999999999999,"5min":1.516,"15min":1.498},"min":{"1min":0.34699999999999998,"5min":0.28599999999999998,"15min":0.26700000000000002},"max":{"1min":7.7729999999999997,"5min":10.036,"15min":10.407999999999999},"last":0.44700000000000001}]},{"osd":1,"last update":"Fri Feb 9 17:54:51 2024","interfaces":[{"interface":"back","average":{"1min":1.589,"5min":1.6040000000000001,"15min":1.8180000000000001},"min":{"1min":0.32800000000000001,"5min":0.26100000000000001,"15min":0.23699999999999999},"max":{"1min":5.6390000000000002,"5min":12.429,"15min":17.297999999999998},"last":50.255000000000003},{"interface":"front","average":{"1min":2.452,"5min":1.8740000000000001,"15min":2.1299999999999999},"min":{"1min":0.28199999999999997,"5min":0.23699999999999999,"15min":0.23100000000000001},"max":{"1min":9.1560000000000006,"5min":12.413,"15min":12.413},"last":49.991999999999997}]},{"osd":2,"last update":"Fri Feb 9 17:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.663,"15min":2.6150000000000002},"min":{"1min":0.38700000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":4.2220000000000004,"5min":8.5139999999999993,"15min":17.494},"last":0.46000000000000002},{"interface":"front","average":{"1min":1.351,"5min":1.865,"15min":2.6110000000000002},"min":{"1min":0.38,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.6609999999999996,"5min":17.283999999999999,"15min":17.283999999999999},"last":0.44800000000000001}]},{"osd":3,"last update":"Fri Feb 9 17:54:51 2024","interfaces":[{"interface":"back","average":{"1min":1.7609999999999999,"5min":1.4179999999999999,"15min":1.655},"min":{"1min":0.29999999999999999,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":10.484999999999999,"5min":10.484999999999999,"15min":14.448},"last":0.42199999999999999},{"interface":"front","average":{"1min":1.681,"5min":1.6759999999999999,"15min":1.736},"min":{"1min":0.315,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":10.455,"5min":10.455,"15min":10.455},"last":0.315}]},{"osd":5,"last update":"Fri Feb 9 17:55:08 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.79800000000000004,"15min":1.7949999999999999},"min":{"1min":0.26700000000000002,"5min":0.253,"15min":0.22600000000000001},"max":{"1min":3.3250000000000002,"5min":4.8019999999999996,"15min":18.887},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.86799999999999999,"15min":1.877},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31},"max":{"1min":3.3599999999999999,"5min":4.8639999999999999,"15min":18.917000000000002},"last":0.56599999999999995}]},{"osd":6,"last update":"Fri Feb 9 17:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":1.133,"15min":1.514},"min":{"1min":0.28499999999999998,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.6839999999999999,"5min":23.343,"15min":23.343},"last":0.375},{"interface":"front","average":{"1min":0.65900000000000003,"5min":1.0249999999999999,"15min":1.6140000000000001},"min":{"1min":0.33700000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":2.6579999999999999,"5min":21.861000000000001,"15min":21.861000000000001},"last":0.435}]},{"osd":7,"last update":"Fri Feb 9 17:55:28 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":1.0129999999999999,"15min":1.536},"min":{"1min":0.41599999999999998,"5min":0.32900000000000001,"15min":0.29899999999999999},"max":{"1min":0.77200000000000002,"5min":6.9550000000000001,"15min":11.875},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.51100000000000001,"5min":1.0389999999999999,"15min":1.6890000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.28599999999999998},"max":{"1min":0.748,"5min":8.6950000000000003,"15min":12.696},"last":0.57799999999999996}]},{"osd":8,"last update":"Fri Feb 9 17:55:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.575,"15min":1.7909999999999999},"min":{"1min":0.35899999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":6.6550000000000002,"5min":8.548,"15min":17.335999999999999},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.3600000000000001,"15min":1.272},"min":{"1min":0.39800000000000002,"5min":0.373,"15min":0.34699999999999998},"max":{"1min":4.8040000000000003,"5min":5.6470000000000002,"15min":10.433},"last":0.65100000000000002}]},{"osd":9,"last update":"Fri Feb 9 17:55:12 2024","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":2.1349999999999998,"15min":2.5630000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":5.3639999999999999,"5min":7.665,"15min":7.665},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.92700000000000005,"5min":1.5249999999999999,"15min":1.661},"min":{"1min":0.498,"5min":0.44,"15min":0.44},"max":{"1min":4.2240000000000002,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.751}]},{"osd":10,"last update":"Fri Feb 9 17:55:28 2024","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":1.264,"15min":1.5329999999999999},"min":{"1min":0.48899999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":6.9509999999999996,"5min":6.9509999999999996,"15min":17.367000000000001},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.83699999999999997,"5min":1.3680000000000001,"15min":1.587},"min":{"1min":0.47199999999999998,"5min":0.38,"15min":0.38},"max":{"1min":3.028,"5min":5.7439999999999998,"15min":12.888},"last":0.50800000000000001}]},{"osd":11,"last update":"Fri Feb 9 17:54:58 2024","interfaces":[{"interface":"back","average":{"1min":1.486,"5min":1.7729999999999999,"15min":1.5920000000000001},"min":{"1min":0.56499999999999995,"5min":0.46500000000000002,"15min":0.39900000000000002},"max":{"1min":5.9779999999999998,"5min":7.8410000000000002,"15min":10.521000000000001},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.95199999999999996,"5min":1.7070000000000001,"15min":1.625},"min":{"1min":0.502,"5min":0.46000000000000002,"15min":0.40999999999999998},"max":{"1min":3.4809999999999999,"5min":8.4420000000000002,"15min":13.154999999999999},"last":0.48499999999999999}]}]},{"osd":5,"up_from":40,"seq":171798692271,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1499764,"kb_used_data":832744,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":92245388,"statfs":{"total":95995035648,"available":94459277312,"internally_reserved":0,"allocated":852729856,"data_stored":852278199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":683014597},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:52:23 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":2.4900000000000002,"15min":1.71},"min":{"1min":0.27600000000000002,"5min":0.26600000000000001,"15min":0.23599999999999999},"max":{"1min":2.613,"5min":116.77800000000001,"15min":116.77800000000001},"last":0.748},{"interface":"front","average":{"1min":1.246,"5min":2.867,"15min":2.0529999999999999},"min":{"1min":0.36299999999999999,"5min":0.252,"15min":0.249},"max":{"1min":3.2959999999999998,"5min":116.959,"15min":116.959},"last":0.36599999999999999}]},{"osd":1,"last update":"Fri Feb 9 17:55:15 2024","interfaces":[{"interface":"back","average":{"1min":1.552,"5min":1.0820000000000001,"15min":1.6779999999999999},"min":{"1min":0.30099999999999999,"5min":0.27900000000000003,"15min":0.24199999999999999},"max":{"1min":9.532,"5min":9.532,"15min":19.079999999999998},"last":3.04},{"interface":"front","average":{"1min":1.1379999999999999,"5min":0.89600000000000002,"15min":1.6679999999999999},"min":{"1min":0.28299999999999997,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":6.2190000000000003,"5min":6.2190000000000003,"15min":19.065999999999999},"last":0.40699999999999997}]},{"osd":2,"last update":"Fri Feb 9 17:54:48 2024","interfaces":[{"interface":"back","average":{"1min":1.55,"5min":1.22,"15min":1.7889999999999999},"min":{"1min":0.312,"5min":0.26100000000000001,"15min":0.249},"max":{"1min":7.5670000000000002,"5min":7.5670000000000002,"15min":13.929},"last":0.34200000000000003},{"interface":"front","average":{"1min":1.5629999999999999,"5min":1.54,"15min":2.052},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":6.3230000000000004,"5min":8.2870000000000008,"15min":15.945},"last":0.46600000000000003}]},{"osd":3,"last update":"Fri Feb 9 17:55:15 2024","interfaces":[{"interface":"back","average":{"1min":2.0219999999999998,"5min":1.8779999999999999,"15min":2.1179999999999999},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":20.690000000000001,"5min":20.690000000000001,"15min":20.690000000000001},"last":0.42999999999999999},{"interface":"front","average":{"1min":1.591,"5min":1.5069999999999999,"15min":2.036},"min":{"1min":0.35499999999999998,"5min":0.316,"15min":0.313},"max":{"1min":6.9269999999999996,"5min":17.209,"15min":17.209},"last":0.63900000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:55:15 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.88,"15min":1.5609999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.249},"max":{"1min":3.8719999999999999,"5min":4.3849999999999998,"15min":28.481999999999999},"last":1.321},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.89700000000000002,"15min":1.6339999999999999},"min":{"1min":0.32900000000000001,"5min":0.22500000000000001,"15min":0.218},"max":{"1min":1.3129999999999999,"5min":4.9189999999999996,"15min":15.843},"last":0.38900000000000001}]},{"osd":6,"last update":"Fri Feb 9 17:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":1.103,"15min":1.637},"min":{"1min":0.33000000000000002,"5min":0.30199999999999999,"15min":0.27900000000000003},"max":{"1min":2.3079999999999998,"5min":8.3829999999999991,"15min":16.657},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.80200000000000005,"5min":1.3340000000000001,"15min":1.764},"min":{"1min":0.33900000000000002,"5min":0.33300000000000002,"15min":0.26200000000000001},"max":{"1min":2.4609999999999999,"5min":9.7200000000000006,"15min":20.192},"last":0.60199999999999998}]},{"osd":7,"last update":"Fri Feb 9 17:55:19 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":1.2589999999999999,"15min":1.704},"min":{"1min":0.39000000000000001,"5min":0.29499999999999998,"15min":0.27900000000000003},"max":{"1min":3.6880000000000002,"5min":18.134,"15min":18.134},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.59199999999999997,"5min":1.1830000000000001,"15min":1.8859999999999999},"min":{"1min":0.39800000000000002,"5min":0.32600000000000001,"15min":0.31},"max":{"1min":1.0760000000000001,"5min":19.300000000000001,"15min":19.300000000000001},"last":0.53900000000000003}]},{"osd":8,"last update":"Fri Feb 9 17:55:38 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":1.4410000000000001,"15min":1.734},"min":{"1min":0.56799999999999995,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.577,"5min":9.6110000000000007,"15min":9.6110000000000007},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.90600000000000003,"5min":1.526,"15min":1.8080000000000001},"min":{"1min":0.39600000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":4.1849999999999996,"5min":8.1829999999999998,"15min":8.1829999999999998},"last":0.57199999999999995}]},{"osd":9,"last update":"Fri Feb 9 17:54:48 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.956,"15min":1.7250000000000001},"min":{"1min":0.42099999999999999,"5min":0.41499999999999998,"15min":0.35299999999999998},"max":{"1min":4.3369999999999997,"5min":10.987,"15min":16.411000000000001},"last":1.347},{"interface":"front","average":{"1min":0.879,"5min":1.484,"15min":1.748},"min":{"1min":0.438,"5min":0.373,"15min":0.35299999999999998},"max":{"1min":2.387,"5min":9.7050000000000001,"15min":16.417999999999999},"last":1.212}]},{"osd":10,"last update":"Fri Feb 9 17:55:15 2024","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":1.5549999999999999,"15min":2},"min":{"1min":0.52100000000000002,"5min":0.373,"15min":0.317},"max":{"1min":4.0140000000000002,"5min":16.292000000000002,"15min":80.042000000000002},"last":0.65300000000000002},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.502,"15min":1.9019999999999999},"min":{"1min":0.54800000000000004,"5min":0.41199999999999998,"15min":0.38200000000000001},"max":{"1min":4.4960000000000004,"5min":12.426,"15min":79.807000000000002},"last":0.69099999999999995}]},{"osd":11,"last update":"Fri Feb 9 17:55:32 2024","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.548,"15min":1.9790000000000001},"min":{"1min":0.51500000000000001,"5min":0.44700000000000001,"15min":0.38400000000000001},"max":{"1min":7.5270000000000001,"5min":9.7680000000000007,"15min":16.390000000000001},"last":3.2759999999999998},{"interface":"front","average":{"1min":0.78600000000000003,"5min":1.9330000000000001,"15min":1.835},"min":{"1min":0.434,"5min":0.434,"15min":0.42199999999999999},"max":{"1min":2.101,"5min":20.981000000000002,"15min":20.981000000000002},"last":2.1030000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":504037376,"data_stored":504037376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1064632320,"data_stored":1064632320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":884211712,"data_stored":884211712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":499974144,"data_stored":499974144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":745734144,"data_stored":745734144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":880541696,"data_stored":880541696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":371916800,"data_stored":371916800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":469958656,"data_stored":469958656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1142816768,"data_stored":1142816768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":993460224,"data_stored":993460224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":920911872,"data_stored":920911872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-09T17:55:52.186 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-02-09T17:55:52.187 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:55:52.187 INFO:tasks.thrashosds.thrasher:inject_pause on osd.10 2024-02-09T17:55:52.187 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-09T17:55:52.187 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-09T17:55:52.187 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.10 config set filestore_inject_stall 3 2024-02-09T17:55:52.243 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:55:52.280 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:52.367 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:52 smithi099 ceph-mon[39420]: 1.1c scrub ok 2024-02-09T17:55:52.367 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:52 smithi099 ceph-mon[39420]: pgmap v1976: 47 pgs: 47 active+clean; 2.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 104 MiB/s wr, 1.67k op/s 2024-02-09T17:55:52.368 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:52 smithi099 ceph-mon[39420]: osdmap e804: 12 total, 12 up, 12 in 2024-02-09T17:55:52.396 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:55:52.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:52 smithi179 ceph-mon[43975]: 1.1c scrub ok 2024-02-09T17:55:52.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:52 smithi179 ceph-mon[43975]: pgmap v1976: 47 pgs: 47 active+clean; 2.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 104 MiB/s wr, 1.67k op/s 2024-02-09T17:55:52.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:52 smithi179 ceph-mon[43975]: osdmap e804: 12 total, 12 up, 12 in 2024-02-09T17:55:52.596 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 26 16 55034 55018 132.205 89.4375 0.00437758 0.00755896 2024-02-09T17:55:52.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:52 smithi162 ceph-mon[44304]: 1.1c scrub ok 2024-02-09T17:55:52.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:52 smithi162 ceph-mon[44304]: pgmap v1976: 47 pgs: 47 active+clean; 2.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 104 MiB/s wr, 1.67k op/s 2024-02-09T17:55:52.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:52 smithi162 ceph-mon[44304]: osdmap e804: 12 total, 12 up, 12 in 2024-02-09T17:55:53.234 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:55:53.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:53.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:55:53.342 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:55:53.388 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:53 smithi179 ceph-mon[43975]: 1.15 scrub starts 2024-02-09T17:55:53.388 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:53 smithi179 ceph-mon[43975]: 1.15 scrub ok 2024-02-09T17:55:53.388 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:53 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:55:53.388 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:53 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:55:53.392 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:53.474 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-09T17:55:53.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:53 smithi099 ceph-mon[39420]: 1.15 scrub starts 2024-02-09T17:55:53.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:53 smithi099 ceph-mon[39420]: 1.15 scrub ok 2024-02-09T17:55:53.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:53 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:55:53.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:53 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:55:53.596 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 27 16 56521 56505 130.751 92.9375 0.0104607 0.00764213 2024-02-09T17:55:53.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:53 smithi162 ceph-mon[44304]: 1.15 scrub starts 2024-02-09T17:55:53.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:53 smithi162 ceph-mon[44304]: 1.15 scrub ok 2024-02-09T17:55:53.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:53 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:55:53.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:53 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:55:53.853 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:55:53.859 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:53.859 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:55:53.885 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:55:53.909 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:55:53.921 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:55:53.940 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:55:53.960 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:55:53.983 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:55:54.009 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:54.021 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:55:54.056 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:55:54.088 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:55:54.121 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:55:54.159 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:55:54.400 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:55:54.438 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:55:54.446 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:54 smithi179 ceph-mon[43975]: pgmap v1978: 47 pgs: 47 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 112 MiB/s wr, 1.79k op/s 2024-02-09T17:55:54.446 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:54 smithi179 ceph-mon[43975]: osdmap e805: 12 total, 12 up, 12 in 2024-02-09T17:55:54.446 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:54 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2099696804' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:55:54.478 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:55:54.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:54 smithi099 ceph-mon[39420]: pgmap v1978: 47 pgs: 47 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 112 MiB/s wr, 1.79k op/s 2024-02-09T17:55:54.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:54 smithi099 ceph-mon[39420]: osdmap e805: 12 total, 12 up, 12 in 2024-02-09T17:55:54.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:54 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2099696804' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:55:54.507 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:55:54.509 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:55:54.509 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:55:54.509 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:55:54.509 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:55:54.509 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:55:54.509 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:55:54.509 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:55:54.509 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:55:54.509 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:55:54.509 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:55:54.510 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:55:54.596 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 28 16 57948 57932 129.267 89.1875 0.0133846 0.00773108 2024-02-09T17:55:54.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:54 smithi162 ceph-mon[44304]: pgmap v1978: 47 pgs: 47 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 112 MiB/s wr, 1.79k op/s 2024-02-09T17:55:54.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:54 smithi162 ceph-mon[44304]: osdmap e805: 12 total, 12 up, 12 in 2024-02-09T17:55:54.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:54 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2099696804' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:55:54.785 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T17:55:54.785 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:55:55.596 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 29 16 59469 59453 128.087 95.0625 0.00529531 0.00780111 2024-02-09T17:55:55.868 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:55:55.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:55.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:55:55.974 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:55:56.020 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:56.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:56 smithi099 ceph-mon[39420]: pgmap v1980: 47 pgs: 47 active+clean; 3.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-02-09T17:55:56.596 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 30 16 60945 60929 126.893 92.25 0.00764144 0.00787595 2024-02-09T17:55:56.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:56 smithi179 ceph-mon[43975]: pgmap v1980: 47 pgs: 47 active+clean; 3.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-02-09T17:55:56.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:56 smithi162 ceph-mon[44304]: pgmap v1980: 47 pgs: 47 active+clean; 3.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-02-09T17:55:57.413 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:55:57.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:57.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:55:57.518 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:55:57.552 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:57.597 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 31 16 62439 62423 125.812 93.375 0.00279096 0.00794169 2024-02-09T17:55:57.817 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:55:57.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:57.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:55:57.921 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:55:57.953 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:57.994 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:55:57.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:57.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:55:58.098 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:55:58.147 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:57 smithi099 ceph-mon[39420]: osdmap e806: 12 total, 12 up, 12 in 2024-02-09T17:55:58.148 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:57 smithi099 ceph-mon[39420]: pgmap v1982: 47 pgs: 47 active+clean; 3.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 108 MiB/s wr, 1.74k op/s 2024-02-09T17:55:58.148 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:58.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:57 smithi179 ceph-mon[43975]: osdmap e806: 12 total, 12 up, 12 in 2024-02-09T17:55:58.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:57 smithi179 ceph-mon[43975]: pgmap v1982: 47 pgs: 47 active+clean; 3.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 108 MiB/s wr, 1.74k op/s 2024-02-09T17:55:58.172 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:55:58.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:58.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:55:58.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:57 smithi162 ceph-mon[44304]: osdmap e806: 12 total, 12 up, 12 in 2024-02-09T17:55:58.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:57 smithi162 ceph-mon[44304]: pgmap v1982: 47 pgs: 47 active+clean; 3.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 108 MiB/s wr, 1.74k op/s 2024-02-09T17:55:58.277 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:55:58.344 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:58.381 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:55:58.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:55:58.387 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:55:58.488 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:55:58.534 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:55:58.597 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 32 16 63857 63841 124.65 88.625 0.00829822 0.00801808 2024-02-09T17:55:58.830 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:55:58.960 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is unset 2024-02-09T17:55:59.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:58 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:55:59.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:58 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:55:59.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:58 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4252642986' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:55:59.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:58 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:55:59.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:58 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:55:59.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:58 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:55:59.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:58 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4252642986' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:55:59.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:58 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:55:59.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:58 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:55:59.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:58 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:55:59.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:58 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4252642986' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:55:59.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:58 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:55:59.597 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 33 16 65232 65216 123.476 85.9375 0.0083941 0.00809472 2024-02-09T17:56:00.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:59 smithi099 ceph-mon[39420]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:56:00.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:59 smithi099 ceph-mon[39420]: Cluster is now healthy 2024-02-09T17:56:00.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:59 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:56:00.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:59 smithi099 ceph-mon[39420]: osdmap e807: 12 total, 12 up, 12 in 2024-02-09T17:56:00.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:55:59 smithi099 ceph-mon[39420]: pgmap v1984: 47 pgs: 47 active+clean; 3.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.55k op/s 2024-02-09T17:56:00.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:59 smithi179 ceph-mon[43975]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:56:00.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:59 smithi179 ceph-mon[43975]: Cluster is now healthy 2024-02-09T17:56:00.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:59 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:56:00.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:59 smithi179 ceph-mon[43975]: osdmap e807: 12 total, 12 up, 12 in 2024-02-09T17:56:00.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:55:59 smithi179 ceph-mon[43975]: pgmap v1984: 47 pgs: 47 active+clean; 3.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.55k op/s 2024-02-09T17:56:00.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:59 smithi162 ceph-mon[44304]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:56:00.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:59 smithi162 ceph-mon[44304]: Cluster is now healthy 2024-02-09T17:56:00.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:59 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:56:00.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:59 smithi162 ceph-mon[44304]: osdmap e807: 12 total, 12 up, 12 in 2024-02-09T17:56:00.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:55:59 smithi162 ceph-mon[44304]: pgmap v1984: 47 pgs: 47 active+clean; 3.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.55k op/s 2024-02-09T17:56:00.597 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 34 16 66527 66511 122.225 80.9375 0.00612448 0.00817706 2024-02-09T17:56:01.064 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:56:01.597 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 35 16 67879 67863 121.147 84.5 0.0110855 0.00824878 2024-02-09T17:56:02.098 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.9 dump_blocked_ops 2024-02-09T17:56:02.107 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:56:02.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:02.114 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:56:02.215 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:56:02.277 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:02.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:02 smithi179 ceph-mon[43975]: pgmap v1985: 47 pgs: 47 active+clean; 3.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-02-09T17:56:02.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:02 smithi179 ceph-mon[43975]: osdmap e808: 12 total, 12 up, 12 in 2024-02-09T17:56:02.490 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:02 smithi099 ceph-mon[39420]: pgmap v1985: 47 pgs: 47 active+clean; 3.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-02-09T17:56:02.490 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:02 smithi099 ceph-mon[39420]: osdmap e808: 12 total, 12 up, 12 in 2024-02-09T17:56:02.597 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 36 16 69158 69142 120.002 79.9375 0.00574126 0.00832889 2024-02-09T17:56:02.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:02 smithi162 ceph-mon[44304]: pgmap v1985: 47 pgs: 47 active+clean; 3.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-02-09T17:56:02.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:02 smithi162 ceph-mon[44304]: osdmap e808: 12 total, 12 up, 12 in 2024-02-09T17:56:03.079 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set noscrub 2024-02-09T17:56:03.149 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:56:03.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:03.156 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:56:03.257 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:56:03.312 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:03.336 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:56:03.378 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:56:03.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:03 smithi179 ceph-mon[43975]: 1.12 scrub starts 2024-02-09T17:56:03.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:03 smithi179 ceph-mon[43975]: 1.12 scrub ok 2024-02-09T17:56:03.428 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:56:03.428 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:56:03.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:03.433 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:56:03.465 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:56:03.492 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:56:03.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:03 smithi099 ceph-mon[39420]: 1.12 scrub starts 2024-02-09T17:56:03.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:03 smithi099 ceph-mon[39420]: 1.12 scrub ok 2024-02-09T17:56:03.507 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:56:03.521 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:56:03.534 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:56:03.557 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:56:03.588 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:03.593 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:56:03.599 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 37 16 70469 70453 118.973 81.9375 0.0228734 0.00840034 2024-02-09T17:56:03.613 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:56:03.631 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:56:03.658 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:56:03.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:03 smithi162 ceph-mon[44304]: 1.12 scrub starts 2024-02-09T17:56:03.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:03 smithi162 ceph-mon[44304]: 1.12 scrub ok 2024-02-09T17:56:03.689 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:56:03.706 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:56:03.706 INFO:teuthology.orchestra.run.smithi179.stdout: "success": "" 2024-02-09T17:56:03.706 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:56:03.723 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:56:03.724 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:56:03.725 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:56:03.725 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:56:03.725 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:56:03.725 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:56:03.725 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:56:03.725 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:56:03.725 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:56:03.725 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:56:03.725 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:56:03.725 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:56:03.725 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:56:04.253 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:04 smithi179 ceph-mon[43975]: pgmap v1987: 47 pgs: 47 active+clean; 4.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.55k op/s 2024-02-09T17:56:04.253 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:04 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1859873182' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:56:04.450 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:56:04.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:04.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:56:04.461 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:04 smithi099 ceph-mon[39420]: pgmap v1987: 47 pgs: 47 active+clean; 4.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.55k op/s 2024-02-09T17:56:04.461 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:04 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1859873182' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:56:04.560 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:56:04.589 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:04 smithi162 ceph-mon[44304]: pgmap v1987: 47 pgs: 47 active+clean; 4.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.55k op/s 2024-02-09T17:56:04.589 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:04 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1859873182' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:56:04.591 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:04.598 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 38 16 71744 71728 117.935 79.6875 0.0061628 0.0084745 2024-02-09T17:56:04.615 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:56:04.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:04.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:56:04.730 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:56:04.766 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:05.325 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:56:05.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:05.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:56:05.436 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:56:05.466 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:05.489 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:56:05.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:05.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:56:05.594 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:56:05.599 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 39 16 73130 73114 117.133 86.625 0.0122431 0.00853314 2024-02-09T17:56:05.622 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:05.646 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:56:05.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:05.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:56:05.750 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:56:05.780 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:05.803 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:56:05.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:05.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:56:05.907 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:56:05.959 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:06.124 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:56:06.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:06.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:56:06.229 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:56:06.287 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:06.329 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:56:06.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:06.337 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:56:06.438 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:56:06.467 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:06.489 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:56:06.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:06 smithi099 ceph-mon[39420]: pgmap v1988: 47 pgs: 47 active+clean; 4.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s 2024-02-09T17:56:06.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:06.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:56:06.596 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:56:06.599 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:2024-02-09T17:56:06.598544+0000 min lat: 0.00109523 max lat: 0.219811 avg lat: 0.00860065 2024-02-09T17:56:06.599 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-09T17:56:06.599 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 40 16 74415 74399 116.212 80.3125 0.010228 0.00860065 2024-02-09T17:56:06.635 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:06 smithi179 ceph-mon[43975]: pgmap v1988: 47 pgs: 47 active+clean; 4.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s 2024-02-09T17:56:06.641 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:06.670 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:56:06.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:06.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:56:06.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:06 smithi162 ceph-mon[44304]: pgmap v1988: 47 pgs: 47 active+clean; 4.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s 2024-02-09T17:56:06.774 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:56:06.817 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:06.847 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:56:06.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:06.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:56:06.951 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:56:06.983 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:07.008 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:56:07.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:07.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:56:07.112 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:56:07.146 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:07.174 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:56:07.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:07.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:56:07.275 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is set 2024-02-09T17:56:07.279 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:56:07.288 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:56:07.288 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-02-09T17:56:07.288 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-02-09T17:56:07.289 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-02-09T17:56:07.289 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:56:07.310 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:07.336 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:56:07.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:07.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:56:07.441 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:56:07.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:07 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/659856495' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:56:07.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:07 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:56:07.500 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:07.533 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:56:07.536 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:56:07.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:07.538 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:56:07.538 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:07 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/659856495' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:56:07.539 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:07 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:56:07.599 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 41 16 75684 75668 115.312 79.3125 0.0158544 0.00866816 2024-02-09T17:56:07.640 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:56:07.669 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:07 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/659856495' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:56:07.669 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:07 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:56:07.671 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:07.695 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:56:07.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:07.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:56:07.800 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:56:07.829 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:07.849 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:56:07.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:07.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:56:07.954 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:56:07.981 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:08.004 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:56:08.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:08.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:56:08.110 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:56:08.139 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:08.161 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:56:08.164 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:08.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:56:08.261 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:08 smithi162 ceph-mon[44304]: pgmap v1989: 47 pgs: 47 active+clean; 4.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.23k op/s 2024-02-09T17:56:08.261 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:08 smithi162 ceph-mon[44304]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:56:08.261 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:08 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:56:08.261 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:08 smithi162 ceph-mon[44304]: osdmap e809: 12 total, 12 up, 12 in 2024-02-09T17:56:08.265 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:56:08.296 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:08.315 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:56:08.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:08.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:56:08.422 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:56:08.459 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:08.599 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 42 16 76848 76832 114.299 72.75 0.00665668 0.00874451 2024-02-09T17:56:08.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:08 smithi179 ceph-mon[43975]: pgmap v1989: 47 pgs: 47 active+clean; 4.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.23k op/s 2024-02-09T17:56:08.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:08 smithi179 ceph-mon[43975]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:56:08.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:08 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:56:08.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:08 smithi179 ceph-mon[43975]: osdmap e809: 12 total, 12 up, 12 in 2024-02-09T17:56:08.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:08 smithi099 ceph-mon[39420]: pgmap v1989: 47 pgs: 47 active+clean; 4.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.23k op/s 2024-02-09T17:56:08.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:08 smithi099 ceph-mon[39420]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:56:08.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:08 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:56:08.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:08 smithi099 ceph-mon[39420]: osdmap e809: 12 total, 12 up, 12 in 2024-02-09T17:56:09.599 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 43 16 78119 78103 113.488 79.4375 0.01014 0.0088068 2024-02-09T17:56:09.885 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] 2024-02-09T17:56:09.886 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:56:09.886 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-02-09T17:56:09.886 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-09T17:56:10.230 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:56:10.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:10.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:56:10.340 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:56:10.344 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:56:10.348 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.9 dump_historic_ops 2024-02-09T17:56:10.361 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:56:10.397 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:10.428 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:56:10.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:10.435 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:56:10.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:10 smithi099 ceph-mon[39420]: pgmap v1991: 47 pgs: 47 active+clean; 4.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.55k op/s 2024-02-09T17:56:10.536 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:56:10.563 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:10 smithi162 ceph-mon[44304]: pgmap v1991: 47 pgs: 47 active+clean; 4.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.55k op/s 2024-02-09T17:56:10.565 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:10.588 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:56:10.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:10.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:56:10.601 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 44 16 79206 79190 112.447 67.9375 0.0226707 0.00888822 2024-02-09T17:56:10.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:10 smithi179 ceph-mon[43975]: pgmap v1991: 47 pgs: 47 active+clean; 4.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.55k op/s 2024-02-09T17:56:10.693 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:56:10.742 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:10.786 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:56:10.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:10.791 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:56:10.841 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:56:10.892 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:56:10.942 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:10.980 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:56:10.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:10.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:56:11.090 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:56:11.141 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:11.409 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:56:11.413 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:11.413 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:56:11.514 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:56:11.577 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:11.602 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 45 16 80381 80365 111.58 73.4375 0.0150062 0.00895775 2024-02-09T17:56:11.690 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:56:11.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:11.697 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:56:11.797 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:56:11.868 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:12.079 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:56:12.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:12.085 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:56:12.187 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:56:12.232 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:12.269 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-09T17:56:12.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:12 smithi179 ceph-mon[43975]: pgmap v1992: 47 pgs: 47 active+clean; 4.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.23k op/s 2024-02-09T17:56:12.525 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:12 smithi099 ceph-mon[39420]: pgmap v1992: 47 pgs: 47 active+clean; 4.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.23k op/s 2024-02-09T17:56:12.602 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 46 16 81589 81573 110.796 75.5 0.010431 0.00901964 2024-02-09T17:56:12.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:12 smithi162 ceph-mon[44304]: pgmap v1992: 47 pgs: 47 active+clean; 4.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.23k op/s 2024-02-09T17:56:13.394 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:56:13.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:13.405 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:56:13.506 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:56:13.536 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:13.558 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:56:13.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:13.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:56:13.602 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 47 16 82664 82648 109.868 67.1875 0.0122595 0.00909769 2024-02-09T17:56:13.662 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:56:13.699 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:13.910 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:56:13.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:13.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:56:14.013 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:56:14.068 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:14.200 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:56:14.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:14.207 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:56:14.273 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:56:14.274 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-02-09T17:56:14.274 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-02-09T17:56:14.274 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-02-09T17:56:14.274 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.274 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25837.0:204166 6.9 6:971f5589:::benchmark_data_smithi179_126206_object73785:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e638)", 2024-02-09T17:56:14.274 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:48:49.292646+0000", 2024-02-09T17:56:14.274 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 444.97542640099999, 2024-02-09T17:56:14.274 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.247854925, 2024-02-09T17:56:14.274 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:14.274 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:14.274 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:14.274 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25837", 2024-02-09T17:56:14.275 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:56:14.275 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 204166 2024-02-09T17:56:14.275 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.275 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:14.275 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.275 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:14.275 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.292646+0000", 2024-02-09T17:56:14.275 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:14.275 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.275 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.275 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:14.275 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.292646+0000", 2024-02-09T17:56:14.275 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:14.275 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.276 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.276 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:14.276 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.292648+0000", 2024-02-09T17:56:14.276 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9910000000000001e-06 2024-02-09T17:56:14.276 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.276 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.276 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:14.276 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.292653+0000", 2024-02-09T17:56:14.276 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.147e-06 2024-02-09T17:56:14.276 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.276 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.276 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:14.276 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.292666+0000", 2024-02-09T17:56:14.277 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3293e-05 2024-02-09T17:56:14.277 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.277 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.277 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:14.277 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.292676+0000", 2024-02-09T17:56:14.277 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2909999999999996e-06 2024-02-09T17:56:14.277 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.277 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.277 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:14.277 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.292735+0000", 2024-02-09T17:56:14.277 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9601000000000001e-05 2024-02-09T17:56:14.277 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.277 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.277 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:14.277 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.292854+0000", 2024-02-09T17:56:14.278 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000118671 2024-02-09T17:56:14.278 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.278 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.278 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,8", 2024-02-09T17:56:14.278 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.292959+0000", 2024-02-09T17:56:14.278 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000105234 2024-02-09T17:56:14.278 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.278 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.278 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:14.278 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.297818+0000", 2024-02-09T17:56:14.278 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0048587250000000004 2024-02-09T17:56:14.278 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.278 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.278 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:14.278 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.298042+0000", 2024-02-09T17:56:14.279 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000224281 2024-02-09T17:56:14.279 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.279 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.279 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:14.279 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.540399+0000", 2024-02-09T17:56:14.279 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.242357292 2024-02-09T17:56:14.279 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.279 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.279 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:14.279 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.540460+0000", 2024-02-09T17:56:14.279 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0921000000000002e-05 2024-02-09T17:56:14.279 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.279 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.280 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:14.280 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.540501+0000", 2024-02-09T17:56:14.280 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0478000000000001e-05 2024-02-09T17:56:14.280 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:14.280 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:14.280 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:14.280 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.280 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.280 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25837.0:204176 6.9 6:9114561b:::benchmark_data_smithi179_126206_object73795:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e638)", 2024-02-09T17:56:14.280 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:48:49.308130+0000", 2024-02-09T17:56:14.280 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 444.95994217200001, 2024-02-09T17:56:14.280 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.232528922, 2024-02-09T17:56:14.280 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:14.280 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:14.281 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:14.281 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25837", 2024-02-09T17:56:14.281 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:56:14.281 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 204176 2024-02-09T17:56:14.281 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.281 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:14.281 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.281 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:14.281 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.308130+0000", 2024-02-09T17:56:14.281 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:14.281 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.281 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.281 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:14.281 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.308130+0000", 2024-02-09T17:56:14.282 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:14.282 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.282 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.282 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:14.282 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.308139+0000", 2024-02-09T17:56:14.282 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4749999999999993e-06 2024-02-09T17:56:14.282 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.282 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.282 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:14.282 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.308144+0000", 2024-02-09T17:56:14.282 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4110000000000004e-06 2024-02-09T17:56:14.282 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.282 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.282 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:14.282 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.308156+0000", 2024-02-09T17:56:14.283 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1992e-05 2024-02-09T17:56:14.283 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.283 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.283 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:14.283 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.308165+0000", 2024-02-09T17:56:14.283 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8480000000000007e-06 2024-02-09T17:56:14.283 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.283 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.283 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:14.283 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.308225+0000", 2024-02-09T17:56:14.283 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0013999999999998e-05 2024-02-09T17:56:14.283 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.283 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.283 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:14.284 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.308317+0000", 2024-02-09T17:56:14.284 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.1995000000000003e-05 2024-02-09T17:56:14.284 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.284 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.284 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,8", 2024-02-09T17:56:14.284 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.308392+0000", 2024-02-09T17:56:14.284 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5588999999999997e-05 2024-02-09T17:56:14.284 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.284 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.284 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:14.284 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.316040+0000", 2024-02-09T17:56:14.284 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0076479620000000003 2024-02-09T17:56:14.284 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.284 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.284 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:14.285 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.316087+0000", 2024-02-09T17:56:14.285 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6789999999999998e-05 2024-02-09T17:56:14.285 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.285 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.285 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:14.285 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.540564+0000", 2024-02-09T17:56:14.285 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.22447653200000001 2024-02-09T17:56:14.285 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.285 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.285 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:14.285 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.540621+0000", 2024-02-09T17:56:14.285 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7058999999999997e-05 2024-02-09T17:56:14.285 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.285 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.286 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:14.286 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.540659+0000", 2024-02-09T17:56:14.286 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8254999999999999e-05 2024-02-09T17:56:14.286 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:14.286 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:14.286 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:14.286 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.286 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.286 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25837.0:204177 6.9 6:9a54deb9:::benchmark_data_smithi179_126206_object73796:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e638)", 2024-02-09T17:56:14.286 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:48:49.308230+0000", 2024-02-09T17:56:14.286 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 444.959842133, 2024-02-09T17:56:14.286 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.23923287800000001, 2024-02-09T17:56:14.286 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:14.286 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:14.286 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:14.287 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25837", 2024-02-09T17:56:14.287 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:56:14.287 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 204177 2024-02-09T17:56:14.287 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.287 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:14.287 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.287 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:14.287 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.308230+0000", 2024-02-09T17:56:14.287 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:14.287 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.287 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.287 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:14.287 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.308230+0000", 2024-02-09T17:56:14.287 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:14.287 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.288 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.288 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:14.288 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.308231+0000", 2024-02-09T17:56:14.288 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8500000000000001e-07 2024-02-09T17:56:14.288 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.288 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.288 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:14.288 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.308234+0000", 2024-02-09T17:56:14.288 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.49e-06 2024-02-09T17:56:14.288 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.288 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.288 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:14.288 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.308242+0000", 2024-02-09T17:56:14.288 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.0730000000000005e-06 2024-02-09T17:56:14.289 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.289 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.289 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:14.289 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.308249+0000", 2024-02-09T17:56:14.289 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3570000000000003e-06 2024-02-09T17:56:14.289 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.289 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.289 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:14.289 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.312312+0000", 2024-02-09T17:56:14.289 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0040633559999999997 2024-02-09T17:56:14.289 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.289 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.289 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:14.289 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.312398+0000", 2024-02-09T17:56:14.289 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5815000000000004e-05 2024-02-09T17:56:14.290 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.290 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.290 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,8", 2024-02-09T17:56:14.290 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.312509+0000", 2024-02-09T17:56:14.290 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000110825 2024-02-09T17:56:14.290 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.290 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.290 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:14.290 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.316189+0000", 2024-02-09T17:56:14.290 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0036803059999999999 2024-02-09T17:56:14.290 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.290 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.290 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:14.290 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.316427+0000", 2024-02-09T17:56:14.291 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000237753 2024-02-09T17:56:14.291 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.291 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.291 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:14.291 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.547356+0000", 2024-02-09T17:56:14.291 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.23092874499999999 2024-02-09T17:56:14.291 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.291 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.291 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:14.291 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.547417+0000", 2024-02-09T17:56:14.291 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1882000000000001e-05 2024-02-09T17:56:14.291 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.291 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.291 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:14.291 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.547463+0000", 2024-02-09T17:56:14.292 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5590999999999998e-05 2024-02-09T17:56:14.292 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:14.292 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:14.292 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:14.292 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.292 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.292 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25837.0:204190 6.9 6:972c490f:::benchmark_data_smithi179_126206_object73809:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e638)", 2024-02-09T17:56:14.292 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:48:49.324946+0000", 2024-02-09T17:56:14.292 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 444.94312643799998, 2024-02-09T17:56:14.292 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.22261397799999999, 2024-02-09T17:56:14.292 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:14.292 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:14.292 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:14.292 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25837", 2024-02-09T17:56:14.293 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:56:14.293 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 204190 2024-02-09T17:56:14.293 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.293 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:14.293 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.293 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:14.293 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.324946+0000", 2024-02-09T17:56:14.293 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:14.293 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.293 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.293 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:14.293 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.324946+0000", 2024-02-09T17:56:14.293 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:14.293 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.294 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.294 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:14.294 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.324947+0000", 2024-02-09T17:56:14.294 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.356e-06 2024-02-09T17:56:14.294 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.294 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.294 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:14.294 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.324952+0000", 2024-02-09T17:56:14.294 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3749999999999996e-06 2024-02-09T17:56:14.294 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.294 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.294 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:14.294 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.324962+0000", 2024-02-09T17:56:14.294 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0755e-05 2024-02-09T17:56:14.294 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.295 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.295 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:14.295 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.324968+0000", 2024-02-09T17:56:14.295 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8839999999999999e-06 2024-02-09T17:56:14.295 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.295 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.295 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:14.295 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.325043+0000", 2024-02-09T17:56:14.295 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4284999999999998e-05 2024-02-09T17:56:14.295 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.295 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.295 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:14.295 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.325127+0000", 2024-02-09T17:56:14.295 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4357000000000006e-05 2024-02-09T17:56:14.295 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.296 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.296 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,8", 2024-02-09T17:56:14.296 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.325207+0000", 2024-02-09T17:56:14.296 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.0438000000000005e-05 2024-02-09T17:56:14.296 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.296 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.296 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:14.296 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.329079+0000", 2024-02-09T17:56:14.296 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0038717309999999998 2024-02-09T17:56:14.296 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.296 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.296 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:14.296 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.329380+0000", 2024-02-09T17:56:14.296 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00030110899999999998 2024-02-09T17:56:14.296 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.297 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.297 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:14.297 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.547506+0000", 2024-02-09T17:56:14.297 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.21812578299999999 2024-02-09T17:56:14.297 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.297 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.297 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:14.297 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.547535+0000", 2024-02-09T17:56:14.297 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.9405000000000001e-05 2024-02-09T17:56:14.297 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.297 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.297 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:14.297 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.547560+0000", 2024-02-09T17:56:14.297 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4499999999999999e-05 2024-02-09T17:56:14.297 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:14.298 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:14.298 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:14.298 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.298 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.298 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25837.0:204216 6.9 6:99e3e93f:::benchmark_data_smithi179_126206_object73835:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e638)", 2024-02-09T17:56:14.298 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:48:49.362151+0000", 2024-02-09T17:56:14.298 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 444.905920907, 2024-02-09T17:56:14.298 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.18551853900000001, 2024-02-09T17:56:14.298 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:14.298 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:14.298 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:14.298 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25837", 2024-02-09T17:56:14.298 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:56:14.298 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 204216 2024-02-09T17:56:14.298 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.299 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:14.299 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.299 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:14.299 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.362151+0000", 2024-02-09T17:56:14.299 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:14.299 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.299 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.299 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:14.299 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.362151+0000", 2024-02-09T17:56:14.299 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:14.299 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.299 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.299 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:14.299 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.362153+0000", 2024-02-09T17:56:14.299 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.144e-06 2024-02-09T17:56:14.300 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.300 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.300 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:14.300 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.362158+0000", 2024-02-09T17:56:14.300 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9699999999999998e-06 2024-02-09T17:56:14.300 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.300 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.300 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:14.300 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.362169+0000", 2024-02-09T17:56:14.300 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1776e-05 2024-02-09T17:56:14.300 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.300 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.300 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:14.300 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.362178+0000", 2024-02-09T17:56:14.300 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.6039999999999993e-06 2024-02-09T17:56:14.301 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.301 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.301 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:14.301 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.362231+0000", 2024-02-09T17:56:14.301 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3573999999999999e-05 2024-02-09T17:56:14.301 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.301 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.301 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:14.301 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.362329+0000", 2024-02-09T17:56:14.301 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7305000000000004e-05 2024-02-09T17:56:14.301 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.301 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.301 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,8", 2024-02-09T17:56:14.301 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.362433+0000", 2024-02-09T17:56:14.301 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000104077 2024-02-09T17:56:14.302 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.302 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.302 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:14.302 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.370291+0000", 2024-02-09T17:56:14.302 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0078582310000000002 2024-02-09T17:56:14.302 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.302 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.302 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:14.302 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.370340+0000", 2024-02-09T17:56:14.302 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8513999999999997e-05 2024-02-09T17:56:14.302 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.302 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.302 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:14.302 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.547598+0000", 2024-02-09T17:56:14.302 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.17725817299999999 2024-02-09T17:56:14.303 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.303 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.303 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:14.303 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.547634+0000", 2024-02-09T17:56:14.303 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6294999999999997e-05 2024-02-09T17:56:14.303 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.303 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.303 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:14.303 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.547670+0000", 2024-02-09T17:56:14.303 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.5876e-05 2024-02-09T17:56:14.303 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:14.303 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:14.303 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:14.303 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.304 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.304 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25837.0:204221 6.9 6:939189ff:::benchmark_data_smithi179_126206_object73840:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e638)", 2024-02-09T17:56:14.304 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:48:49.366243+0000", 2024-02-09T17:56:14.304 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 444.90182916399999, 2024-02-09T17:56:14.304 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.181567432, 2024-02-09T17:56:14.304 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:14.304 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:14.304 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:14.304 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25837", 2024-02-09T17:56:14.304 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:56:14.304 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 204221 2024-02-09T17:56:14.304 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.304 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:14.304 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.304 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:14.305 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.366243+0000", 2024-02-09T17:56:14.305 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:14.305 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.305 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.305 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:14.305 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.366243+0000", 2024-02-09T17:56:14.305 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:14.305 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.305 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.305 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:14.305 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.366245+0000", 2024-02-09T17:56:14.305 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4950000000000001e-06 2024-02-09T17:56:14.305 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.305 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.305 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:14.306 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.366251+0000", 2024-02-09T17:56:14.306 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4949999999999996e-06 2024-02-09T17:56:14.306 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.306 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.306 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:14.306 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.366268+0000", 2024-02-09T17:56:14.306 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6572e-05 2024-02-09T17:56:14.306 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.306 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.306 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:14.306 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.366276+0000", 2024-02-09T17:56:14.306 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7900000000000005e-06 2024-02-09T17:56:14.306 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.306 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.307 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:14.307 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.366383+0000", 2024-02-09T17:56:14.307 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010607699999999999 2024-02-09T17:56:14.307 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.307 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.307 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:14.307 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.366467+0000", 2024-02-09T17:56:14.307 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4641000000000005e-05 2024-02-09T17:56:14.307 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.307 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.308 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,8", 2024-02-09T17:56:14.308 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.366587+0000", 2024-02-09T17:56:14.308 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011932199999999999 2024-02-09T17:56:14.308 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.308 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.308 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:14.308 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.370435+0000", 2024-02-09T17:56:14.308 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0038482870000000001 2024-02-09T17:56:14.308 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.308 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.308 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:14.308 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.370663+0000", 2024-02-09T17:56:14.308 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00022793100000000001 2024-02-09T17:56:14.308 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.308 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.309 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:14.309 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.547716+0000", 2024-02-09T17:56:14.309 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.177053609 2024-02-09T17:56:14.309 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.309 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.309 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:14.309 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.547773+0000", 2024-02-09T17:56:14.309 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6743000000000002e-05 2024-02-09T17:56:14.309 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.309 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.309 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:14.309 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.547811+0000", 2024-02-09T17:56:14.309 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7469999999999999e-05 2024-02-09T17:56:14.309 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:14.309 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:14.310 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:14.310 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.310 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.310 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25837.0:204229 6.9 6:998a194a:::benchmark_data_smithi179_126206_object73848:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e638)", 2024-02-09T17:56:14.310 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:48:49.372914+0000", 2024-02-09T17:56:14.310 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 444.89515810699999, 2024-02-09T17:56:14.310 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.17502336900000001, 2024-02-09T17:56:14.310 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:14.310 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:14.310 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:14.310 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25837", 2024-02-09T17:56:14.310 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:56:14.310 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 204229 2024-02-09T17:56:14.310 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.310 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:14.311 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.311 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:14.311 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.372914+0000", 2024-02-09T17:56:14.311 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:14.311 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.311 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.311 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:14.311 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.372914+0000", 2024-02-09T17:56:14.311 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:14.311 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.311 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.311 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:14.311 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.372915+0000", 2024-02-09T17:56:14.311 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.099e-06 2024-02-09T17:56:14.311 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.312 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.312 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:14.312 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.372920+0000", 2024-02-09T17:56:14.312 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7330000000000003e-06 2024-02-09T17:56:14.312 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.312 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.312 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:14.312 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.372933+0000", 2024-02-09T17:56:14.312 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3234999999999999e-05 2024-02-09T17:56:14.312 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.312 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.312 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:14.312 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.372942+0000", 2024-02-09T17:56:14.312 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.6440000000000006e-06 2024-02-09T17:56:14.312 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.313 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.313 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:14.313 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.372982+0000", 2024-02-09T17:56:14.313 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9857000000000001e-05 2024-02-09T17:56:14.313 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.313 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.313 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:14.313 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.373061+0000", 2024-02-09T17:56:14.313 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.9048000000000006e-05 2024-02-09T17:56:14.313 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.313 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.313 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,8", 2024-02-09T17:56:14.313 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.373128+0000", 2024-02-09T17:56:14.313 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6728000000000002e-05 2024-02-09T17:56:14.314 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.314 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.314 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:14.314 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.376915+0000", 2024-02-09T17:56:14.314 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.003786995 2024-02-09T17:56:14.314 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.314 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.314 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:14.314 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.377261+0000", 2024-02-09T17:56:14.314 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00034642100000000002 2024-02-09T17:56:14.314 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.314 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.314 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:14.314 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.547856+0000", 2024-02-09T17:56:14.315 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.17059540000000001 2024-02-09T17:56:14.315 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.315 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.315 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:14.315 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.547903+0000", 2024-02-09T17:56:14.315 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6733999999999997e-05 2024-02-09T17:56:14.315 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.315 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.315 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:14.315 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:49.547938+0000", 2024-02-09T17:56:14.315 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4474999999999998e-05 2024-02-09T17:56:14.315 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:14.315 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:14.315 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:14.316 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.316 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.316 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25837.0:237182 6.9 6:984d9f93:::benchmark_data_smithi179_126206_object106801:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e650)", 2024-02-09T17:56:14.316 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:49:29.617181+0000", 2024-02-09T17:56:14.316 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 404.65089159500002, 2024-02-09T17:56:14.316 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.130885117, 2024-02-09T17:56:14.316 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:14.316 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:14.316 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:14.316 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25837", 2024-02-09T17:56:14.316 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:56:14.316 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 237182 2024-02-09T17:56:14.316 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.316 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:14.317 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.317 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:14.317 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.617181+0000", 2024-02-09T17:56:14.317 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:14.317 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.317 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.317 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:14.317 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.617181+0000", 2024-02-09T17:56:14.317 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:14.317 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.317 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.317 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:14.317 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.617182+0000", 2024-02-09T17:56:14.317 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.432e-06 2024-02-09T17:56:14.317 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.318 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.318 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:14.318 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.617188+0000", 2024-02-09T17:56:14.318 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4960000000000004e-06 2024-02-09T17:56:14.318 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.318 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.318 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:14.318 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.617201+0000", 2024-02-09T17:56:14.318 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.379e-05 2024-02-09T17:56:14.318 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.318 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.318 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:14.318 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.617210+0000", 2024-02-09T17:56:14.318 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4100000000000008e-06 2024-02-09T17:56:14.318 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.319 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.319 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:14.319 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.619755+0000", 2024-02-09T17:56:14.319 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0025451240000000002 2024-02-09T17:56:14.319 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.319 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.319 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:14.319 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.619853+0000", 2024-02-09T17:56:14.319 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.8238000000000004e-05 2024-02-09T17:56:14.319 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.319 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.319 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,8", 2024-02-09T17:56:14.319 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.619966+0000", 2024-02-09T17:56:14.319 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000112551 2024-02-09T17:56:14.319 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.320 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.320 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:14.320 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.624240+0000", 2024-02-09T17:56:14.320 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0042745049999999996 2024-02-09T17:56:14.320 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.320 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.320 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:14.320 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.624952+0000", 2024-02-09T17:56:14.320 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00071185300000000003 2024-02-09T17:56:14.320 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.320 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.320 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:14.320 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.747992+0000", 2024-02-09T17:56:14.320 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.12303966 2024-02-09T17:56:14.320 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.321 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.321 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:14.321 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.748028+0000", 2024-02-09T17:56:14.321 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6647000000000003e-05 2024-02-09T17:56:14.321 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.321 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.321 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:14.321 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:29.748066+0000", 2024-02-09T17:56:14.321 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7410999999999998e-05 2024-02-09T17:56:14.321 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:14.321 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:14.321 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:14.321 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.321 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.322 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25837.0:237478 6.9 6:9dfda08e:::benchmark_data_smithi179_126206_object107097:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e650)", 2024-02-09T17:56:14.322 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:49:30.004763+0000", 2024-02-09T17:56:14.322 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 404.263309779, 2024-02-09T17:56:14.322 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.13184420199999999, 2024-02-09T17:56:14.322 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:14.322 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:14.322 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:14.322 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25837", 2024-02-09T17:56:14.322 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:56:14.322 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 237478 2024-02-09T17:56:14.322 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.322 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:14.322 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.322 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:14.322 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:30.004763+0000", 2024-02-09T17:56:14.323 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:14.323 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.323 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.323 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:14.323 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:30.004763+0000", 2024-02-09T17:56:14.323 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:14.323 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.323 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.323 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:14.323 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:30.004764+0000", 2024-02-09T17:56:14.323 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3909999999999999e-06 2024-02-09T17:56:14.323 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.323 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.323 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:14.323 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:30.004769+0000", 2024-02-09T17:56:14.324 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2549999999999997e-06 2024-02-09T17:56:14.324 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.324 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.324 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:14.324 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:30.004781+0000", 2024-02-09T17:56:14.324 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1518000000000001e-05 2024-02-09T17:56:14.324 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.324 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.324 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:14.324 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:30.004787+0000", 2024-02-09T17:56:14.324 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4749999999999998e-06 2024-02-09T17:56:14.324 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.324 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.324 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:14.324 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:30.007015+0000", 2024-02-09T17:56:14.325 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0022282410000000002 2024-02-09T17:56:14.325 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.325 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.325 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:14.325 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:30.007144+0000", 2024-02-09T17:56:14.325 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012876399999999999 2024-02-09T17:56:14.325 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.325 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.325 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,8", 2024-02-09T17:56:14.325 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:30.007264+0000", 2024-02-09T17:56:14.325 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000119632 2024-02-09T17:56:14.325 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.325 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.325 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:14.325 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:30.012262+0000", 2024-02-09T17:56:14.326 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.004997792 2024-02-09T17:56:14.326 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.326 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.326 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:14.326 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:30.012466+0000", 2024-02-09T17:56:14.326 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00020460699999999999 2024-02-09T17:56:14.326 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.326 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.326 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:14.326 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:30.136502+0000", 2024-02-09T17:56:14.326 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.12403586799999999 2024-02-09T17:56:14.326 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.326 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.326 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:14.327 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:30.136562+0000", 2024-02-09T17:56:14.327 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0205999999999999e-05 2024-02-09T17:56:14.327 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.327 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.327 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:14.327 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:30.136607+0000", 2024-02-09T17:56:14.327 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4453000000000003e-05 2024-02-09T17:56:14.327 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:14.327 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:14.327 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:14.327 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.327 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.327 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25837.0:237894 6.9 6:9c3dd516:::benchmark_data_smithi179_126206_object107513:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e650)", 2024-02-09T17:56:14.327 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:49:30.615234+0000", 2024-02-09T17:56:14.327 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 403.65283809699997, 2024-02-09T17:56:14.328 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.13106290000000001, 2024-02-09T17:56:14.328 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:14.328 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:14.328 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:14.328 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25837", 2024-02-09T17:56:14.328 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:56:14.328 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 237894 2024-02-09T17:56:14.328 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.328 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:14.328 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.328 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:14.328 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:30.615234+0000", 2024-02-09T17:56:14.328 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:14.328 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.329 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.329 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:14.329 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:30.615234+0000", 2024-02-09T17:56:14.329 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:14.329 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.329 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.329 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:14.329 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:30.615235+0000", 2024-02-09T17:56:14.329 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2800000000000002e-07 2024-02-09T17:56:14.329 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.329 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.329 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:14.329 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:30.615237+0000", 2024-02-09T17:56:14.329 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6819999999999999e-06 2024-02-09T17:56:14.329 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.330 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.330 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:14.330 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:30.615244+0000", 2024-02-09T17:56:14.330 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.0929999999999997e-06 2024-02-09T17:56:14.330 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.330 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.330 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:14.330 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:30.615249+0000", 2024-02-09T17:56:14.330 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0259999999999998e-06 2024-02-09T17:56:14.330 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.330 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.330 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:14.330 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:30.622575+0000", 2024-02-09T17:56:14.330 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0073259029999999999 2024-02-09T17:56:14.331 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.331 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.331 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:14.331 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:30.622695+0000", 2024-02-09T17:56:14.331 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000120047 2024-02-09T17:56:14.331 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.331 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.331 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,8", 2024-02-09T17:56:14.331 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:30.622811+0000", 2024-02-09T17:56:14.331 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000115277 2024-02-09T17:56:14.331 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.331 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.331 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:14.331 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:30.639674+0000", 2024-02-09T17:56:14.331 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.016863586 2024-02-09T17:56:14.332 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.332 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.332 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:14.332 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:30.652238+0000", 2024-02-09T17:56:14.332 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.012564139 2024-02-09T17:56:14.332 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.332 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.332 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:14.332 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:30.746196+0000", 2024-02-09T17:56:14.332 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.093958102000000002 2024-02-09T17:56:14.332 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.332 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.332 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:14.332 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:30.746256+0000", 2024-02-09T17:56:14.332 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9483e-05 2024-02-09T17:56:14.333 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.333 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.333 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:14.333 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:30.746297+0000", 2024-02-09T17:56:14.333 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1134000000000003e-05 2024-02-09T17:56:14.333 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:14.333 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:14.333 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:14.333 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.333 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.333 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25837.0:238253 6.9 6:9bd20849:::benchmark_data_smithi179_126206_object107872:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e650)", 2024-02-09T17:56:14.333 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:49:31.164587+0000", 2024-02-09T17:56:14.333 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 403.10348578899999, 2024-02-09T17:56:14.333 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.13307923199999999, 2024-02-09T17:56:14.333 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:14.334 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:14.334 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:14.334 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25837", 2024-02-09T17:56:14.334 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:56:14.334 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 238253 2024-02-09T17:56:14.334 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.334 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:14.334 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.334 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:14.334 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:31.164587+0000", 2024-02-09T17:56:14.334 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:14.334 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.334 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.334 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:14.335 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:31.164587+0000", 2024-02-09T17:56:14.335 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:14.335 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.335 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.335 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:14.335 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:31.164587+0000", 2024-02-09T17:56:14.335 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3300000000000002e-07 2024-02-09T17:56:14.335 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.335 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.335 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:14.335 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:31.164589+0000", 2024-02-09T17:56:14.335 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4389999999999999e-06 2024-02-09T17:56:14.335 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.335 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.335 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:14.336 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:31.164597+0000", 2024-02-09T17:56:14.336 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4649999999999997e-06 2024-02-09T17:56:14.336 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.336 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.336 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:14.336 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:31.164602+0000", 2024-02-09T17:56:14.336 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6809999999999997e-06 2024-02-09T17:56:14.336 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.336 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.336 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:14.336 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:31.176534+0000", 2024-02-09T17:56:14.336 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011932681000000001 2024-02-09T17:56:14.336 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.336 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.336 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:14.337 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:31.176615+0000", 2024-02-09T17:56:14.337 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1124000000000005e-05 2024-02-09T17:56:14.337 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.337 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.337 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,8", 2024-02-09T17:56:14.337 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:31.176692+0000", 2024-02-09T17:56:14.337 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6340000000000004e-05 2024-02-09T17:56:14.337 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.337 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.337 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:14.337 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:31.182005+0000", 2024-02-09T17:56:14.337 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0053130139999999996 2024-02-09T17:56:14.337 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.337 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.337 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:14.338 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:31.183562+0000", 2024-02-09T17:56:14.338 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0015572959999999999 2024-02-09T17:56:14.338 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.338 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.338 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:14.338 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:31.297563+0000", 2024-02-09T17:56:14.338 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.114001093 2024-02-09T17:56:14.338 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.338 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.338 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:14.338 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:31.297622+0000", 2024-02-09T17:56:14.338 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8879000000000003e-05 2024-02-09T17:56:14.338 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.339 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.339 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:14.339 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:31.297666+0000", 2024-02-09T17:56:14.339 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3686999999999997e-05 2024-02-09T17:56:14.339 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:14.339 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:14.339 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:14.339 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.339 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.339 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.16440.0:116799 7.a 7:5a8c05a2:::benchmark_data_smithi179_135804_object9008:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e706)", 2024-02-09T17:56:14.339 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:52:10.344899+0000", 2024-02-09T17:56:14.339 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 243.923173031, 2024-02-09T17:56:14.339 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.144607598, 2024-02-09T17:56:14.339 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:14.340 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:14.340 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:14.340 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.16440", 2024-02-09T17:56:14.340 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:56:14.340 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 116799 2024-02-09T17:56:14.340 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.340 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:14.340 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.340 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:14.340 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:52:10.344899+0000", 2024-02-09T17:56:14.340 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:14.340 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.340 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.340 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:14.340 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:52:10.344899+0000", 2024-02-09T17:56:14.341 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:14.341 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.341 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.341 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:14.341 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:52:10.344901+0000", 2024-02-09T17:56:14.341 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3939999999999999e-06 2024-02-09T17:56:14.341 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.341 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.341 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:14.341 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:52:10.344907+0000", 2024-02-09T17:56:14.341 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9359999999999996e-06 2024-02-09T17:56:14.341 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.341 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.341 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:14.342 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:52:10.344920+0000", 2024-02-09T17:56:14.342 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3614999999999999e-05 2024-02-09T17:56:14.342 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.342 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.342 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:14.342 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:52:10.344928+0000", 2024-02-09T17:56:14.342 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.3000000000000004e-06 2024-02-09T17:56:14.342 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.342 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.342 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:14.342 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:52:10.345000+0000", 2024-02-09T17:56:14.342 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2356999999999999e-05 2024-02-09T17:56:14.342 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.342 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.342 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:14.343 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:52:10.345117+0000", 2024-02-09T17:56:14.343 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000117272 2024-02-09T17:56:14.343 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.343 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.343 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,11", 2024-02-09T17:56:14.343 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:52:10.345227+0000", 2024-02-09T17:56:14.343 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010989000000000001 2024-02-09T17:56:14.343 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.343 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.343 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:14.343 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:52:10.350381+0000", 2024-02-09T17:56:14.343 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0051542439999999997 2024-02-09T17:56:14.343 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.343 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.343 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:14.344 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:52:10.354666+0000", 2024-02-09T17:56:14.344 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00428496 2024-02-09T17:56:14.344 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.344 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.344 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:14.344 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:52:10.489394+0000", 2024-02-09T17:56:14.344 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.13472764800000001 2024-02-09T17:56:14.344 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.344 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.344 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:14.344 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:52:10.489461+0000", 2024-02-09T17:56:14.344 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6891999999999996e-05 2024-02-09T17:56:14.344 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.344 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.345 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:14.345 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:52:10.489507+0000", 2024-02-09T17:56:14.345 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6090000000000001e-05 2024-02-09T17:56:14.345 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:14.345 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:14.345 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:14.345 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.345 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.345 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.16440.0:116807 7.a 7:541dc07f:::benchmark_data_smithi179_135804_object9016:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e706)", 2024-02-09T17:56:14.345 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:52:10.359166+0000", 2024-02-09T17:56:14.345 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 243.90890586500001, 2024-02-09T17:56:14.345 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14020807399999999, 2024-02-09T17:56:14.345 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:14.345 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:14.345 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:14.346 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.16440", 2024-02-09T17:56:14.346 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:56:14.346 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 116807 2024-02-09T17:56:14.346 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.346 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:14.346 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.346 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:14.346 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:52:10.359166+0000", 2024-02-09T17:56:14.346 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:14.346 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.346 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.346 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:14.346 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:52:10.359166+0000", 2024-02-09T17:56:14.346 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:14.346 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.346 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.347 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:14.347 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:52:10.359168+0000", 2024-02-09T17:56:14.347 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9800000000000001e-06 2024-02-09T17:56:14.347 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.347 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.347 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:14.347 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:52:10.359177+0000", 2024-02-09T17:56:14.347 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2619999999999998e-06 2024-02-09T17:56:14.347 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.347 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.347 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:14.347 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:52:10.359198+0000", 2024-02-09T17:56:14.347 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0999e-05 2024-02-09T17:56:14.347 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.347 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.348 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:14.348 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:52:10.359208+0000", 2024-02-09T17:56:14.348 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0142999999999999e-05 2024-02-09T17:56:14.348 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.348 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.348 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:14.348 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:52:10.359275+0000", 2024-02-09T17:56:14.348 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6828999999999993e-05 2024-02-09T17:56:14.348 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.348 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.348 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:14.348 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:52:10.359419+0000", 2024-02-09T17:56:14.348 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000144804 2024-02-09T17:56:14.348 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.349 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.349 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,11", 2024-02-09T17:56:14.349 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:52:10.359538+0000", 2024-02-09T17:56:14.349 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000118226 2024-02-09T17:56:14.349 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.349 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.349 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:14.349 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:52:10.365534+0000", 2024-02-09T17:56:14.349 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.005996187 2024-02-09T17:56:14.349 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.349 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.349 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:14.349 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:52:10.375192+0000", 2024-02-09T17:56:14.349 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0096585690000000005 2024-02-09T17:56:14.349 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.350 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.350 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:14.350 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:52:10.499275+0000", 2024-02-09T17:56:14.350 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.124082891 2024-02-09T17:56:14.350 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.350 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.350 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:14.350 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:52:10.499332+0000", 2024-02-09T17:56:14.350 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6894000000000001e-05 2024-02-09T17:56:14.350 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.350 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.350 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:14.350 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:52:10.499375+0000", 2024-02-09T17:56:14.350 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2290000000000003e-05 2024-02-09T17:56:14.350 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:14.350 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:14.351 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:14.351 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.351 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.351 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.16440.0:185443 7.0 7:03c708bb:::benchmark_data_smithi179_135804_object77652:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e750)", 2024-02-09T17:56:14.351 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:54:03.197202+0000", 2024-02-09T17:56:14.351 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 131.07087068499999, 2024-02-09T17:56:14.351 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.21396905599999999, 2024-02-09T17:56:14.351 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:14.351 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:14.351 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:14.351 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.16440", 2024-02-09T17:56:14.351 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:56:14.351 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 185443 2024-02-09T17:56:14.351 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.351 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:14.352 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.352 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:14.352 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.197202+0000", 2024-02-09T17:56:14.352 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:14.352 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.352 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.352 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:14.352 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.197202+0000", 2024-02-09T17:56:14.352 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:14.352 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.352 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.352 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:14.352 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.197203+0000", 2024-02-09T17:56:14.352 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.672e-06 2024-02-09T17:56:14.352 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.353 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.353 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:14.353 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.197208+0000", 2024-02-09T17:56:14.353 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0139999999999998e-06 2024-02-09T17:56:14.353 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.353 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.353 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:14.353 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.197220+0000", 2024-02-09T17:56:14.353 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1234e-05 2024-02-09T17:56:14.353 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.353 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.353 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:14.353 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.197226+0000", 2024-02-09T17:56:14.353 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6120000000000004e-06 2024-02-09T17:56:14.353 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.354 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.354 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:14.354 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.197286+0000", 2024-02-09T17:56:14.354 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0189000000000003e-05 2024-02-09T17:56:14.354 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.354 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.354 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:14.354 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.197430+0000", 2024-02-09T17:56:14.354 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000143953 2024-02-09T17:56:14.354 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.354 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.354 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,5", 2024-02-09T17:56:14.354 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.197567+0000", 2024-02-09T17:56:14.354 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013683400000000001 2024-02-09T17:56:14.354 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.355 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.355 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:14.355 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.207675+0000", 2024-02-09T17:56:14.355 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010107649999999999 2024-02-09T17:56:14.355 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.355 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.355 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:14.355 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.219491+0000", 2024-02-09T17:56:14.355 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011816455999999999 2024-02-09T17:56:14.355 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.355 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.355 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:14.355 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.411090+0000", 2024-02-09T17:56:14.355 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.19159906800000001 2024-02-09T17:56:14.355 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.356 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.356 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:14.356 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.411139+0000", 2024-02-09T17:56:14.356 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8937000000000003e-05 2024-02-09T17:56:14.356 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.356 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.356 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:14.356 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.411171+0000", 2024-02-09T17:56:14.356 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1436999999999997e-05 2024-02-09T17:56:14.356 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:14.356 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:14.356 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:14.356 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.356 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.357 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.16440.0:185444 7.0 7:0d715ece:::benchmark_data_smithi179_135804_object77653:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e750)", 2024-02-09T17:56:14.357 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:54:03.198499+0000", 2024-02-09T17:56:14.357 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 131.069573561, 2024-02-09T17:56:14.357 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.22028868099999999, 2024-02-09T17:56:14.357 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:14.357 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:14.357 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:14.357 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.16440", 2024-02-09T17:56:14.357 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:56:14.357 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 185444 2024-02-09T17:56:14.357 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.357 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:14.357 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.357 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:14.357 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.198499+0000", 2024-02-09T17:56:14.358 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:14.358 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.358 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.358 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:14.358 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.198499+0000", 2024-02-09T17:56:14.358 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:14.358 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.358 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.358 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:14.358 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.198500+0000", 2024-02-09T17:56:14.358 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.9699999999999994e-07 2024-02-09T17:56:14.358 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.358 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.358 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:14.358 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.198504+0000", 2024-02-09T17:56:14.359 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4950000000000002e-06 2024-02-09T17:56:14.359 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.359 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.359 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:14.359 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.198514+0000", 2024-02-09T17:56:14.359 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0247e-05 2024-02-09T17:56:14.359 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.359 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.359 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:14.359 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.198521+0000", 2024-02-09T17:56:14.359 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.888e-06 2024-02-09T17:56:14.359 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.359 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.359 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:14.359 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.202421+0000", 2024-02-09T17:56:14.360 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0038995330000000002 2024-02-09T17:56:14.360 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.360 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.360 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:14.360 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.202583+0000", 2024-02-09T17:56:14.360 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016227 2024-02-09T17:56:14.360 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.360 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.360 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,5", 2024-02-09T17:56:14.360 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.202753+0000", 2024-02-09T17:56:14.360 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016971900000000001 2024-02-09T17:56:14.360 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.360 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.360 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:14.361 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.208757+0000", 2024-02-09T17:56:14.361 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0060043859999999996 2024-02-09T17:56:14.361 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.361 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.361 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:14.361 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.219534+0000", 2024-02-09T17:56:14.361 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010776737 2024-02-09T17:56:14.361 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.361 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.361 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:14.361 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.418698+0000", 2024-02-09T17:56:14.361 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.19916429699999999 2024-02-09T17:56:14.361 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.361 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.361 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:14.362 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.418749+0000", 2024-02-09T17:56:14.362 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0470999999999998e-05 2024-02-09T17:56:14.362 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.362 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.362 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:14.362 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.418787+0000", 2024-02-09T17:56:14.362 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8640999999999998e-05 2024-02-09T17:56:14.362 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:14.362 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:14.362 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:14.362 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.362 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.362 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.16440.0:185497 7.0 7:038256ed:::benchmark_data_smithi179_135804_object77706:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e750)", 2024-02-09T17:56:14.362 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:54:03.277731+0000", 2024-02-09T17:56:14.362 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 130.99034168899999, 2024-02-09T17:56:14.363 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.157956928, 2024-02-09T17:56:14.363 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:14.363 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:14.363 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:14.363 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.16440", 2024-02-09T17:56:14.363 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:56:14.363 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 185497 2024-02-09T17:56:14.363 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.363 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:14.363 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.363 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:14.363 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.277731+0000", 2024-02-09T17:56:14.363 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:14.363 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.363 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.364 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:14.364 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.277731+0000", 2024-02-09T17:56:14.364 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:14.364 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.364 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.364 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:14.364 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.277732+0000", 2024-02-09T17:56:14.364 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4890000000000001e-06 2024-02-09T17:56:14.364 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.364 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.364 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:14.364 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.277738+0000", 2024-02-09T17:56:14.364 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.032e-06 2024-02-09T17:56:14.364 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.365 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.365 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:14.365 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.277754+0000", 2024-02-09T17:56:14.365 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5364999999999999e-05 2024-02-09T17:56:14.365 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.365 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.365 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:14.365 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.277763+0000", 2024-02-09T17:56:14.365 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.4229999999999994e-06 2024-02-09T17:56:14.365 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.365 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.365 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:14.365 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.277826+0000", 2024-02-09T17:56:14.365 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2810000000000003e-05 2024-02-09T17:56:14.365 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.366 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.366 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:14.366 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.277952+0000", 2024-02-09T17:56:14.366 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000126308 2024-02-09T17:56:14.366 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.366 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.366 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,5", 2024-02-09T17:56:14.366 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.278087+0000", 2024-02-09T17:56:14.366 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013537799999999999 2024-02-09T17:56:14.366 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.366 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.366 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:14.366 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.282395+0000", 2024-02-09T17:56:14.366 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0043079420000000004 2024-02-09T17:56:14.366 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.367 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.367 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:14.367 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.289860+0000", 2024-02-09T17:56:14.367 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0074650389999999997 2024-02-09T17:56:14.367 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.367 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.367 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:14.367 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.435584+0000", 2024-02-09T17:56:14.367 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.145723824 2024-02-09T17:56:14.367 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.367 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.367 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:14.367 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.435646+0000", 2024-02-09T17:56:14.367 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2255999999999995e-05 2024-02-09T17:56:14.367 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.368 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.368 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:14.368 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.435688+0000", 2024-02-09T17:56:14.368 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1062000000000001e-05 2024-02-09T17:56:14.368 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:14.368 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:14.368 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:14.368 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.368 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.368 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.16440.0:208983 7.0 7:002d3dd2:::benchmark_data_smithi179_135804_object101192:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e768)", 2024-02-09T17:56:14.368 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:54:37.365113+0000", 2024-02-09T17:56:14.368 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 96.902959674000002, 2024-02-09T17:56:14.368 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.101179614, 2024-02-09T17:56:14.368 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:14.369 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:14.369 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:14.369 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.16440", 2024-02-09T17:56:14.369 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:56:14.369 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 208983 2024-02-09T17:56:14.369 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.369 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:14.369 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.369 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:14.369 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:37.365113+0000", 2024-02-09T17:56:14.369 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:14.369 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.369 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.369 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:14.369 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:37.365113+0000", 2024-02-09T17:56:14.369 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:14.370 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.370 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.370 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:14.370 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:37.365114+0000", 2024-02-09T17:56:14.370 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8300000000000001e-06 2024-02-09T17:56:14.370 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.370 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.370 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:14.370 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:37.365119+0000", 2024-02-09T17:56:14.370 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9060000000000001e-06 2024-02-09T17:56:14.370 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.370 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.370 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:14.370 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:37.365135+0000", 2024-02-09T17:56:14.370 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5947000000000002e-05 2024-02-09T17:56:14.371 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.371 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.371 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:14.371 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:37.365143+0000", 2024-02-09T17:56:14.371 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.0190000000000006e-06 2024-02-09T17:56:14.371 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.371 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.371 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:14.371 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:37.365200+0000", 2024-02-09T17:56:14.371 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6227000000000002e-05 2024-02-09T17:56:14.371 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.371 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.371 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:14.371 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:37.365316+0000", 2024-02-09T17:56:14.371 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011674400000000001 2024-02-09T17:56:14.372 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.372 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.372 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,5", 2024-02-09T17:56:14.372 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:37.365428+0000", 2024-02-09T17:56:14.372 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000111554 2024-02-09T17:56:14.372 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.372 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.372 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:14.372 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:37.370393+0000", 2024-02-09T17:56:14.372 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0049647190000000002 2024-02-09T17:56:14.372 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.372 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.372 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:14.372 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:37.372816+0000", 2024-02-09T17:56:14.372 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0024236700000000002 2024-02-09T17:56:14.373 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.373 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.373 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:14.373 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:37.466222+0000", 2024-02-09T17:56:14.373 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.093405375999999998 2024-02-09T17:56:14.373 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.373 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.373 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:14.373 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:37.466263+0000", 2024-02-09T17:56:14.373 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1020999999999999e-05 2024-02-09T17:56:14.373 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.373 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.373 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:14.373 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:37.466292+0000", 2024-02-09T17:56:14.373 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.9601e-05 2024-02-09T17:56:14.374 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:14.374 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:14.374 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:14.374 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.374 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.374 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.16440.0:210607 7.a 7:537cfc84:::benchmark_data_smithi179_135804_object102816:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e769)", 2024-02-09T17:56:14.374 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:54:39.685778+0000", 2024-02-09T17:56:14.374 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 94.582294026, 2024-02-09T17:56:14.374 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.080938066000000003, 2024-02-09T17:56:14.374 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:14.374 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:14.374 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:14.374 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.16440", 2024-02-09T17:56:14.374 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:56:14.374 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 210607 2024-02-09T17:56:14.374 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.375 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:14.375 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.375 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:14.375 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:39.685778+0000", 2024-02-09T17:56:14.375 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:14.375 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.375 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.375 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:14.375 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:39.685778+0000", 2024-02-09T17:56:14.375 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:14.375 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.375 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.375 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:14.375 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:39.685780+0000", 2024-02-09T17:56:14.375 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3149999999999999e-06 2024-02-09T17:56:14.376 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.376 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.376 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:14.376 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:39.685784+0000", 2024-02-09T17:56:14.376 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3440000000000003e-06 2024-02-09T17:56:14.376 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.376 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.376 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:14.376 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:39.685797+0000", 2024-02-09T17:56:14.376 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2670000000000001e-05 2024-02-09T17:56:14.376 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.376 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.376 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:14.376 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:39.685804+0000", 2024-02-09T17:56:14.376 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8480000000000001e-06 2024-02-09T17:56:14.377 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.377 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.377 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:14.377 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:39.685858+0000", 2024-02-09T17:56:14.377 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3727999999999998e-05 2024-02-09T17:56:14.377 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.377 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.377 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:14.377 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:39.685934+0000", 2024-02-09T17:56:14.377 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6039999999999997e-05 2024-02-09T17:56:14.377 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.377 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.377 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,11", 2024-02-09T17:56:14.378 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:56:14.379 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:39.686005+0000", 2024-02-09T17:56:14.379 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.0505000000000003e-05 2024-02-09T17:56:14.379 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.379 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.379 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:14.379 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:39.693670+0000", 2024-02-09T17:56:14.379 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0076653650000000004 2024-02-09T17:56:14.379 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.379 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.379 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:14.379 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:39.693695+0000", 2024-02-09T17:56:14.380 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5038e-05 2024-02-09T17:56:14.380 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.380 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.380 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:14.380 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:39.766622+0000", 2024-02-09T17:56:14.380 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.072927264000000006 2024-02-09T17:56:14.380 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.380 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.380 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:14.380 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:39.766674+0000", 2024-02-09T17:56:14.380 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1730000000000001e-05 2024-02-09T17:56:14.380 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.380 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.380 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:14.381 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:39.766716+0000", 2024-02-09T17:56:14.381 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2218999999999997e-05 2024-02-09T17:56:14.381 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:14.381 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:14.381 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:14.381 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.381 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.381 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.16440.0:210609 7.a 7:5ab1a799:::benchmark_data_smithi179_135804_object102818:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e769)", 2024-02-09T17:56:14.381 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:54:39.689418+0000", 2024-02-09T17:56:14.381 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 94.578654374999999, 2024-02-09T17:56:14.381 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.077426747000000004, 2024-02-09T17:56:14.381 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:14.381 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:14.381 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:14.381 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.16440", 2024-02-09T17:56:14.382 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:56:14.382 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 210609 2024-02-09T17:56:14.382 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.382 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:14.382 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.382 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:14.382 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:39.689418+0000", 2024-02-09T17:56:14.382 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:14.382 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.382 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.382 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:14.382 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:39.689418+0000", 2024-02-09T17:56:14.382 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:14.382 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.382 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.383 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:14.383 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:39.689419+0000", 2024-02-09T17:56:14.383 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.218e-06 2024-02-09T17:56:14.383 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.383 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.383 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:14.383 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:39.689425+0000", 2024-02-09T17:56:14.383 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9139999999999996e-06 2024-02-09T17:56:14.383 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.383 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.383 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:14.383 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:39.689438+0000", 2024-02-09T17:56:14.383 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3198e-05 2024-02-09T17:56:14.383 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.383 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.384 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:14.384 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:39.689446+0000", 2024-02-09T17:56:14.384 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.9789999999999993e-06 2024-02-09T17:56:14.384 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.384 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.384 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:14.384 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:39.689834+0000", 2024-02-09T17:56:14.384 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00038810100000000002 2024-02-09T17:56:14.384 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.384 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.384 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:14.384 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:39.689948+0000", 2024-02-09T17:56:14.384 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011369 2024-02-09T17:56:14.384 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.385 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.385 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,11", 2024-02-09T17:56:14.385 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:39.690033+0000", 2024-02-09T17:56:14.385 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5017000000000003e-05 2024-02-09T17:56:14.385 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.385 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.385 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:14.385 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:39.693829+0000", 2024-02-09T17:56:14.385 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0037964330000000001 2024-02-09T17:56:14.385 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.385 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.385 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:14.385 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:39.696913+0000", 2024-02-09T17:56:14.385 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0030838620000000001 2024-02-09T17:56:14.386 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.386 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.386 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:14.386 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:39.766758+0000", 2024-02-09T17:56:14.386 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.069844495000000006 2024-02-09T17:56:14.386 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.386 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.386 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:14.386 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:39.766809+0000", 2024-02-09T17:56:14.386 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0878000000000003e-05 2024-02-09T17:56:14.386 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.386 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.386 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:14.386 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:39.766845+0000", 2024-02-09T17:56:14.386 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.5961999999999999e-05 2024-02-09T17:56:14.387 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:14.387 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:14.387 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:14.387 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.387 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.387 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:81738 8.4 8:2163f94d:::benchmark_data_smithi179_148205_object81737:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e809)", 2024-02-09T17:56:14.387 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:56:12.737379+0000", 2024-02-09T17:56:14.387 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 1.5306930969999999, 2024-02-09T17:56:14.387 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.11189160300000001, 2024-02-09T17:56:14.387 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:14.387 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:14.387 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:14.387 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T17:56:14.387 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T17:56:14.387 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 81738 2024-02-09T17:56:14.388 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.388 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:14.388 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.388 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:14.388 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.737379+0000", 2024-02-09T17:56:14.388 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:14.388 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.388 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.388 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:14.388 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.737379+0000", 2024-02-09T17:56:14.388 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:14.388 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.388 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.388 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:14.389 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.737380+0000", 2024-02-09T17:56:14.389 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.195e-06 2024-02-09T17:56:14.389 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.389 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.389 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:14.389 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.737408+0000", 2024-02-09T17:56:14.389 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7755000000000001e-05 2024-02-09T17:56:14.389 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.389 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.389 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:14.389 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.737434+0000", 2024-02-09T17:56:14.389 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6299999999999999e-05 2024-02-09T17:56:14.389 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.389 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.389 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:14.390 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.737442+0000", 2024-02-09T17:56:14.390 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8960000000000003e-06 2024-02-09T17:56:14.390 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.390 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.390 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:14.390 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.737491+0000", 2024-02-09T17:56:14.390 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8851999999999999e-05 2024-02-09T17:56:14.390 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.390 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.390 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:14.390 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.737611+0000", 2024-02-09T17:56:14.390 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012021199999999999 2024-02-09T17:56:14.390 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.390 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.390 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,10", 2024-02-09T17:56:14.390 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.737695+0000", 2024-02-09T17:56:14.391 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4031999999999998e-05 2024-02-09T17:56:14.391 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.391 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.391 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:14.391 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.739908+0000", 2024-02-09T17:56:14.391 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0022125449999999998 2024-02-09T17:56:14.391 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.391 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.391 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:14.391 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.740208+0000", 2024-02-09T17:56:14.391 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000299685 2024-02-09T17:56:14.391 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.391 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.391 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:14.392 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.849176+0000", 2024-02-09T17:56:14.392 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.108967905 2024-02-09T17:56:14.392 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.392 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.392 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:14.392 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.849233+0000", 2024-02-09T17:56:14.392 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7785000000000003e-05 2024-02-09T17:56:14.392 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:14.392 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:14.392 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:14.392 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.849271+0000", 2024-02-09T17:56:14.392 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7441000000000003e-05 2024-02-09T17:56:14.392 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:14.392 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:14.393 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:14.393 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:14.393 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:14.393 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:56:14.407 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:14.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:14 smithi179 ceph-mon[43975]: pgmap v1993: 47 pgs: 47 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-02-09T17:56:14.425 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:14 smithi099 ceph-mon[39420]: pgmap v1993: 47 pgs: 47 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-02-09T17:56:14.429 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:56:14.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:14.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:56:14.536 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:56:14.591 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:14.602 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 48 16 83777 83761 109.029 69.5625 0.0102644 0.00916728 2024-02-09T17:56:14.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:14 smithi162 ceph-mon[44304]: pgmap v1993: 47 pgs: 47 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-02-09T17:56:14.960 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:56:14.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:14.964 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:56:15.065 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:56:15.114 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:15.167 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:56:15.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:15.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:56:15.172 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:56:15.271 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:56:15.278 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:56:15.297 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:15 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/894384329' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:56:15.300 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:15.323 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:56:15.324 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:56:15.328 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:15.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:56:15.371 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:56:15.405 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:56:15.429 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:56:15.430 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:56:15.450 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:56:15.456 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:15.476 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:56:15.477 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:56:15.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:15.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:56:15.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:15 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/894384329' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:56:15.517 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:56:15.530 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:56:15.530 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:56:15.530 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 809, 2024-02-09T17:56:15.530 INFO:teuthology.orchestra.run.smithi099.stdout: "fsid": "a69c870a-c76e-11ee-95b7-87774f69a715", 2024-02-09T17:56:15.530 INFO:teuthology.orchestra.run.smithi099.stdout: "created": "2024-02-09T17:15:26.113606+0000", 2024-02-09T17:56:15.530 INFO:teuthology.orchestra.run.smithi099.stdout: "modified": "2024-02-09T17:56:07.253898+0000", 2024-02-09T17:56:15.530 INFO:teuthology.orchestra.run.smithi099.stdout: "last_up_change": "2024-02-09T17:28:17.711044+0000", 2024-02-09T17:56:15.530 INFO:teuthology.orchestra.run.smithi099.stdout: "last_in_change": "2024-02-09T17:43:24.138763+0000", 2024-02-09T17:56:15.530 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-09T17:56:15.530 INFO:teuthology.orchestra.run.smithi099.stdout: "flags_num": 5801984, 2024-02-09T17:56:15.530 INFO:teuthology.orchestra.run.smithi099.stdout: "flags_set": [ 2024-02-09T17:56:15.530 INFO:teuthology.orchestra.run.smithi099.stdout: "noscrub", 2024-02-09T17:56:15.531 INFO:teuthology.orchestra.run.smithi099.stdout: "pglog_hardlimit", 2024-02-09T17:56:15.531 INFO:teuthology.orchestra.run.smithi099.stdout: "purged_snapdirs", 2024-02-09T17:56:15.531 INFO:teuthology.orchestra.run.smithi099.stdout: "recovery_deletes", 2024-02-09T17:56:15.531 INFO:teuthology.orchestra.run.smithi099.stdout: "sortbitwise" 2024-02-09T17:56:15.531 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T17:56:15.531 INFO:teuthology.orchestra.run.smithi099.stdout: "crush_version": 27, 2024-02-09T17:56:15.531 INFO:teuthology.orchestra.run.smithi099.stdout: "full_ratio": 0.94999998807907104, 2024-02-09T17:56:15.531 INFO:teuthology.orchestra.run.smithi099.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-09T17:56:15.531 INFO:teuthology.orchestra.run.smithi099.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-09T17:56:15.531 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_snapshot": "", 2024-02-09T17:56:15.531 INFO:teuthology.orchestra.run.smithi099.stdout: "pool_max": 8, 2024-02-09T17:56:15.531 INFO:teuthology.orchestra.run.smithi099.stdout: "max_osd": 12, 2024-02-09T17:56:15.531 INFO:teuthology.orchestra.run.smithi099.stdout: "require_min_compat_client": "luminous", 2024-02-09T17:56:15.531 INFO:teuthology.orchestra.run.smithi099.stdout: "min_compat_client": "luminous", 2024-02-09T17:56:15.532 INFO:teuthology.orchestra.run.smithi099.stdout: "require_osd_release": "squid", 2024-02-09T17:56:15.532 INFO:teuthology.orchestra.run.smithi099.stdout: "allow_crimson": false, 2024-02-09T17:56:15.532 INFO:teuthology.orchestra.run.smithi099.stdout: "pools": [ 2024-02-09T17:56:15.532 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.532 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 1, 2024-02-09T17:56:15.532 INFO:teuthology.orchestra.run.smithi099.stdout: "pool_name": ".mgr", 2024-02-09T17:56:15.532 INFO:teuthology.orchestra.run.smithi099.stdout: "create_time": "2024-02-09T17:18:56.552286+0000", 2024-02-09T17:56:15.532 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 1, 2024-02-09T17:56:15.532 INFO:teuthology.orchestra.run.smithi099.stdout: "flags_names": "hashpspool", 2024-02-09T17:56:15.532 INFO:teuthology.orchestra.run.smithi099.stdout: "type": 1, 2024-02-09T17:56:15.532 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-02-09T17:56:15.532 INFO:teuthology.orchestra.run.smithi099.stdout: "min_size": 2, 2024-02-09T17:56:15.532 INFO:teuthology.orchestra.run.smithi099.stdout: "crush_rule": 0, 2024-02-09T17:56:15.532 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_count": 0, 2024-02-09T17:56:15.533 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_target": 0, 2024-02-09T17:56:15.533 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_barrier": 0, 2024-02-09T17:56:15.533 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-09T17:56:15.533 INFO:teuthology.orchestra.run.smithi099.stdout: "object_hash": 2, 2024-02-09T17:56:15.533 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_autoscale_mode": "off", 2024-02-09T17:56:15.533 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num": 31, 2024-02-09T17:56:15.533 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_placement_num": 31, 2024-02-09T17:56:15.533 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_placement_num_target": 31, 2024-02-09T17:56:15.533 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num_target": 31, 2024-02-09T17:56:15.533 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num_pending": 31, 2024-02-09T17:56:15.533 INFO:teuthology.orchestra.run.smithi099.stdout: "last_pg_merge_meta": { 2024-02-09T17:56:15.533 INFO:teuthology.orchestra.run.smithi099.stdout: "source_pgid": "0.0", 2024-02-09T17:56:15.533 INFO:teuthology.orchestra.run.smithi099.stdout: "ready_epoch": 0, 2024-02-09T17:56:15.533 INFO:teuthology.orchestra.run.smithi099.stdout: "last_epoch_started": 0, 2024-02-09T17:56:15.533 INFO:teuthology.orchestra.run.smithi099.stdout: "last_epoch_clean": 0, 2024-02-09T17:56:15.534 INFO:teuthology.orchestra.run.smithi099.stdout: "source_version": "0'0", 2024-02-09T17:56:15.534 INFO:teuthology.orchestra.run.smithi099.stdout: "target_version": "0'0" 2024-02-09T17:56:15.534 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.534 INFO:teuthology.orchestra.run.smithi099.stdout: "last_change": "544", 2024-02-09T17:56:15.534 INFO:teuthology.orchestra.run.smithi099.stdout: "last_force_op_resend": "0", 2024-02-09T17:56:15.534 INFO:teuthology.orchestra.run.smithi099.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-09T17:56:15.534 INFO:teuthology.orchestra.run.smithi099.stdout: "last_force_op_resend_preluminous": "514", 2024-02-09T17:56:15.534 INFO:teuthology.orchestra.run.smithi099.stdout: "auid": 0, 2024-02-09T17:56:15.534 INFO:teuthology.orchestra.run.smithi099.stdout: "snap_mode": "selfmanaged", 2024-02-09T17:56:15.534 INFO:teuthology.orchestra.run.smithi099.stdout: "snap_seq": 0, 2024-02-09T17:56:15.534 INFO:teuthology.orchestra.run.smithi099.stdout: "snap_epoch": 0, 2024-02-09T17:56:15.534 INFO:teuthology.orchestra.run.smithi099.stdout: "pool_snaps": [], 2024-02-09T17:56:15.534 INFO:teuthology.orchestra.run.smithi099.stdout: "removed_snaps": "[]", 2024-02-09T17:56:15.534 INFO:teuthology.orchestra.run.smithi099.stdout: "quota_max_bytes": 0, 2024-02-09T17:56:15.534 INFO:teuthology.orchestra.run.smithi099.stdout: "quota_max_objects": 0, 2024-02-09T17:56:15.535 INFO:teuthology.orchestra.run.smithi099.stdout: "tiers": [], 2024-02-09T17:56:15.535 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_of": -1, 2024-02-09T17:56:15.535 INFO:teuthology.orchestra.run.smithi099.stdout: "read_tier": -1, 2024-02-09T17:56:15.535 INFO:teuthology.orchestra.run.smithi099.stdout: "write_tier": -1, 2024-02-09T17:56:15.535 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_mode": "none", 2024-02-09T17:56:15.535 INFO:teuthology.orchestra.run.smithi099.stdout: "target_max_bytes": 0, 2024-02-09T17:56:15.535 INFO:teuthology.orchestra.run.smithi099.stdout: "target_max_objects": 0, 2024-02-09T17:56:15.535 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-09T17:56:15.535 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-09T17:56:15.535 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-09T17:56:15.535 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_min_flush_age": 0, 2024-02-09T17:56:15.535 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_min_evict_age": 0, 2024-02-09T17:56:15.535 INFO:teuthology.orchestra.run.smithi099.stdout: "erasure_code_profile": "", 2024-02-09T17:56:15.535 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_params": { 2024-02-09T17:56:15.535 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "none" 2024-02-09T17:56:15.536 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.536 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_period": 0, 2024-02-09T17:56:15.536 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_count": 0, 2024-02-09T17:56:15.536 INFO:teuthology.orchestra.run.smithi099.stdout: "use_gmt_hitset": true, 2024-02-09T17:56:15.536 INFO:teuthology.orchestra.run.smithi099.stdout: "min_read_recency_for_promote": 0, 2024-02-09T17:56:15.536 INFO:teuthology.orchestra.run.smithi099.stdout: "min_write_recency_for_promote": 0, 2024-02-09T17:56:15.536 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_grade_decay_rate": 0, 2024-02-09T17:56:15.536 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_search_last_n": 0, 2024-02-09T17:56:15.536 INFO:teuthology.orchestra.run.smithi099.stdout: "grade_table": [], 2024-02-09T17:56:15.536 INFO:teuthology.orchestra.run.smithi099.stdout: "stripe_width": 0, 2024-02-09T17:56:15.536 INFO:teuthology.orchestra.run.smithi099.stdout: "expected_num_objects": 0, 2024-02-09T17:56:15.536 INFO:teuthology.orchestra.run.smithi099.stdout: "fast_read": false, 2024-02-09T17:56:15.536 INFO:teuthology.orchestra.run.smithi099.stdout: "options": { 2024-02-09T17:56:15.536 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num_max": 32, 2024-02-09T17:56:15.537 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num_min": 1 2024-02-09T17:56:15.537 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.537 INFO:teuthology.orchestra.run.smithi099.stdout: "application_metadata": { 2024-02-09T17:56:15.537 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": {} 2024-02-09T17:56:15.537 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.537 INFO:teuthology.orchestra.run.smithi099.stdout: "read_balance": { 2024-02-09T17:56:15.537 INFO:teuthology.orchestra.run.smithi099.stdout: "score_type": "Fair distribution", 2024-02-09T17:56:15.537 INFO:teuthology.orchestra.run.smithi099.stdout: "score_acting": 5.9899997711181641, 2024-02-09T17:56:15.537 INFO:teuthology.orchestra.run.smithi099.stdout: "score_stable": 2.130000114440918, 2024-02-09T17:56:15.537 INFO:teuthology.orchestra.run.smithi099.stdout: "optimal_score": 1.0900000333786011, 2024-02-09T17:56:15.537 INFO:teuthology.orchestra.run.smithi099.stdout: "raw_score_acting": 6.5300002098083496, 2024-02-09T17:56:15.537 INFO:teuthology.orchestra.run.smithi099.stdout: "raw_score_stable": 2.3199999332427979, 2024-02-09T17:56:15.537 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity_weighted": 0.78079599142074585, 2024-02-09T17:56:15.537 INFO:teuthology.orchestra.run.smithi099.stdout: "average_primary_affinity": 0.85000002384185791, 2024-02-09T17:56:15.537 INFO:teuthology.orchestra.run.smithi099.stdout: "average_primary_affinity_weighted": 0.77999997138977051, 2024-02-09T17:56:15.538 INFO:teuthology.orchestra.run.smithi099.stdout: "error_message": "pool 1 has acting primaries on OSD(s) with primary affinity 0, read balance score is not accurate" 2024-02-09T17:56:15.538 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.538 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.538 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.538 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 8, 2024-02-09T17:56:15.538 INFO:teuthology.orchestra.run.smithi099.stdout: "pool_name": "unique_pool_6", 2024-02-09T17:56:15.538 INFO:teuthology.orchestra.run.smithi099.stdout: "create_time": "2024-02-09T17:55:15.183490+0000", 2024-02-09T17:56:15.538 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 1, 2024-02-09T17:56:15.538 INFO:teuthology.orchestra.run.smithi099.stdout: "flags_names": "hashpspool", 2024-02-09T17:56:15.538 INFO:teuthology.orchestra.run.smithi099.stdout: "type": 1, 2024-02-09T17:56:15.538 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-02-09T17:56:15.538 INFO:teuthology.orchestra.run.smithi099.stdout: "min_size": 2, 2024-02-09T17:56:15.538 INFO:teuthology.orchestra.run.smithi099.stdout: "crush_rule": 0, 2024-02-09T17:56:15.538 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_count": 0, 2024-02-09T17:56:15.538 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_target": 0, 2024-02-09T17:56:15.539 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_barrier": 0, 2024-02-09T17:56:15.539 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-09T17:56:15.539 INFO:teuthology.orchestra.run.smithi099.stdout: "object_hash": 2, 2024-02-09T17:56:15.539 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_autoscale_mode": "off", 2024-02-09T17:56:15.539 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num": 16, 2024-02-09T17:56:15.539 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_placement_num": 16, 2024-02-09T17:56:15.539 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_placement_num_target": 16, 2024-02-09T17:56:15.539 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num_target": 16, 2024-02-09T17:56:15.539 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num_pending": 16, 2024-02-09T17:56:15.539 INFO:teuthology.orchestra.run.smithi099.stdout: "last_pg_merge_meta": { 2024-02-09T17:56:15.539 INFO:teuthology.orchestra.run.smithi099.stdout: "source_pgid": "0.0", 2024-02-09T17:56:15.539 INFO:teuthology.orchestra.run.smithi099.stdout: "ready_epoch": 0, 2024-02-09T17:56:15.539 INFO:teuthology.orchestra.run.smithi099.stdout: "last_epoch_started": 0, 2024-02-09T17:56:15.539 INFO:teuthology.orchestra.run.smithi099.stdout: "last_epoch_clean": 0, 2024-02-09T17:56:15.539 INFO:teuthology.orchestra.run.smithi099.stdout: "source_version": "0'0", 2024-02-09T17:56:15.540 INFO:teuthology.orchestra.run.smithi099.stdout: "target_version": "0'0" 2024-02-09T17:56:15.540 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.540 INFO:teuthology.orchestra.run.smithi099.stdout: "last_change": "793", 2024-02-09T17:56:15.540 INFO:teuthology.orchestra.run.smithi099.stdout: "last_force_op_resend": "0", 2024-02-09T17:56:15.540 INFO:teuthology.orchestra.run.smithi099.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-09T17:56:15.540 INFO:teuthology.orchestra.run.smithi099.stdout: "last_force_op_resend_preluminous": "0", 2024-02-09T17:56:15.540 INFO:teuthology.orchestra.run.smithi099.stdout: "auid": 0, 2024-02-09T17:56:15.540 INFO:teuthology.orchestra.run.smithi099.stdout: "snap_mode": "selfmanaged", 2024-02-09T17:56:15.540 INFO:teuthology.orchestra.run.smithi099.stdout: "snap_seq": 0, 2024-02-09T17:56:15.540 INFO:teuthology.orchestra.run.smithi099.stdout: "snap_epoch": 0, 2024-02-09T17:56:15.540 INFO:teuthology.orchestra.run.smithi099.stdout: "pool_snaps": [], 2024-02-09T17:56:15.540 INFO:teuthology.orchestra.run.smithi099.stdout: "removed_snaps": "[]", 2024-02-09T17:56:15.540 INFO:teuthology.orchestra.run.smithi099.stdout: "quota_max_bytes": 0, 2024-02-09T17:56:15.540 INFO:teuthology.orchestra.run.smithi099.stdout: "quota_max_objects": 0, 2024-02-09T17:56:15.541 INFO:teuthology.orchestra.run.smithi099.stdout: "tiers": [], 2024-02-09T17:56:15.541 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_of": -1, 2024-02-09T17:56:15.541 INFO:teuthology.orchestra.run.smithi099.stdout: "read_tier": -1, 2024-02-09T17:56:15.541 INFO:teuthology.orchestra.run.smithi099.stdout: "write_tier": -1, 2024-02-09T17:56:15.541 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_mode": "none", 2024-02-09T17:56:15.541 INFO:teuthology.orchestra.run.smithi099.stdout: "target_max_bytes": 0, 2024-02-09T17:56:15.541 INFO:teuthology.orchestra.run.smithi099.stdout: "target_max_objects": 0, 2024-02-09T17:56:15.541 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-09T17:56:15.541 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-09T17:56:15.541 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-09T17:56:15.541 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_min_flush_age": 0, 2024-02-09T17:56:15.541 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_min_evict_age": 0, 2024-02-09T17:56:15.541 INFO:teuthology.orchestra.run.smithi099.stdout: "erasure_code_profile": "", 2024-02-09T17:56:15.541 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_params": { 2024-02-09T17:56:15.541 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "none" 2024-02-09T17:56:15.542 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.542 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_period": 0, 2024-02-09T17:56:15.542 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_count": 0, 2024-02-09T17:56:15.542 INFO:teuthology.orchestra.run.smithi099.stdout: "use_gmt_hitset": true, 2024-02-09T17:56:15.542 INFO:teuthology.orchestra.run.smithi099.stdout: "min_read_recency_for_promote": 0, 2024-02-09T17:56:15.542 INFO:teuthology.orchestra.run.smithi099.stdout: "min_write_recency_for_promote": 0, 2024-02-09T17:56:15.542 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_grade_decay_rate": 0, 2024-02-09T17:56:15.542 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_search_last_n": 0, 2024-02-09T17:56:15.542 INFO:teuthology.orchestra.run.smithi099.stdout: "grade_table": [], 2024-02-09T17:56:15.542 INFO:teuthology.orchestra.run.smithi099.stdout: "stripe_width": 0, 2024-02-09T17:56:15.542 INFO:teuthology.orchestra.run.smithi099.stdout: "expected_num_objects": 0, 2024-02-09T17:56:15.542 INFO:teuthology.orchestra.run.smithi099.stdout: "fast_read": false, 2024-02-09T17:56:15.542 INFO:teuthology.orchestra.run.smithi099.stdout: "options": {}, 2024-02-09T17:56:15.542 INFO:teuthology.orchestra.run.smithi099.stdout: "application_metadata": { 2024-02-09T17:56:15.543 INFO:teuthology.orchestra.run.smithi099.stdout: "rados": {} 2024-02-09T17:56:15.543 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.543 INFO:teuthology.orchestra.run.smithi099.stdout: "read_balance": { 2024-02-09T17:56:15.543 INFO:teuthology.orchestra.run.smithi099.stdout: "score_type": "Fair distribution", 2024-02-09T17:56:15.543 INFO:teuthology.orchestra.run.smithi099.stdout: "score_acting": 2.75, 2024-02-09T17:56:15.543 INFO:teuthology.orchestra.run.smithi099.stdout: "score_stable": 2.75, 2024-02-09T17:56:15.543 INFO:teuthology.orchestra.run.smithi099.stdout: "optimal_score": 1, 2024-02-09T17:56:15.543 INFO:teuthology.orchestra.run.smithi099.stdout: "raw_score_acting": 2.75, 2024-02-09T17:56:15.543 INFO:teuthology.orchestra.run.smithi099.stdout: "raw_score_stable": 2.75, 2024-02-09T17:56:15.543 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity_weighted": 0.78079599142074585, 2024-02-09T17:56:15.543 INFO:teuthology.orchestra.run.smithi099.stdout: "average_primary_affinity": 0.85000002384185791, 2024-02-09T17:56:15.543 INFO:teuthology.orchestra.run.smithi099.stdout: "average_primary_affinity_weighted": 0.77999997138977051 2024-02-09T17:56:15.543 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.543 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.543 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T17:56:15.543 INFO:teuthology.orchestra.run.smithi099.stdout: "osds": [ 2024-02-09T17:56:15.544 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.544 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 0, 2024-02-09T17:56:15.544 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "21739aa7-e278-45da-9a3f-867385f01baf", 2024-02-09T17:56:15.544 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:56:15.544 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:56:15.544 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:56:15.544 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 0.609619140625, 2024-02-09T17:56:15.544 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:56:15.544 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:56:15.544 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 8, 2024-02-09T17:56:15.544 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 785, 2024-02-09T17:56:15.544 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:56:15.544 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:56:15.544 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:56:15.544 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:56:15.545 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.545 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:56:15.545 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6802", 2024-02-09T17:56:15.545 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T17:56:15.545 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.545 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.545 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:56:15.545 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6803", 2024-02-09T17:56:15.545 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T17:56:15.545 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.545 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.545 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.545 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:56:15.545 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:56:15.546 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.546 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:56:15.546 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6804", 2024-02-09T17:56:15.546 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T17:56:15.546 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.546 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.546 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:56:15.546 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6805", 2024-02-09T17:56:15.546 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T17:56:15.546 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.546 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.546 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.546 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:56:15.546 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:56:15.546 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.546 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:56:15.547 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6808", 2024-02-09T17:56:15.547 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T17:56:15.547 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.547 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.547 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:56:15.547 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6809", 2024-02-09T17:56:15.547 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T17:56:15.547 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.547 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.547 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.547 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:56:15.547 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:56:15.547 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.547 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:56:15.547 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6806", 2024-02-09T17:56:15.548 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T17:56:15.548 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.548 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.548 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:56:15.548 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6807", 2024-02-09T17:56:15.548 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T17:56:15.548 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.548 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.548 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.548 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.99:6803/3859868923", 2024-02-09T17:56:15.548 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.99:6805/3859868923", 2024-02-09T17:56:15.548 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.99:6809/3859868923", 2024-02-09T17:56:15.548 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.99:6807/3859868923", 2024-02-09T17:56:15.548 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:56:15.548 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:56:15.549 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:56:15.549 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.549 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.549 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.549 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 1, 2024-02-09T17:56:15.549 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "8eab347c-9065-4de1-93ae-e2eb7e11cab7", 2024-02-09T17:56:15.549 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:56:15.549 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:56:15.549 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:56:15.549 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T17:56:15.549 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:56:15.549 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:56:15.549 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 13, 2024-02-09T17:56:15.549 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 785, 2024-02-09T17:56:15.549 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:56:15.550 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:56:15.550 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:56:15.550 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:56:15.550 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.550 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:56:15.550 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6810", 2024-02-09T17:56:15.550 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T17:56:15.550 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.550 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.550 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:56:15.550 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6811", 2024-02-09T17:56:15.550 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T17:56:15.550 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.550 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.550 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.551 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:56:15.551 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:56:15.551 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.551 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:56:15.551 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6812", 2024-02-09T17:56:15.551 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T17:56:15.551 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.551 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.551 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:56:15.551 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6813", 2024-02-09T17:56:15.551 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T17:56:15.551 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.551 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.551 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.551 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:56:15.551 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:56:15.552 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.552 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:56:15.552 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6816", 2024-02-09T17:56:15.552 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T17:56:15.552 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.552 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.552 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:56:15.552 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6817", 2024-02-09T17:56:15.552 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T17:56:15.552 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.552 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.552 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.552 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:56:15.552 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:56:15.552 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.553 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:56:15.553 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6814", 2024-02-09T17:56:15.553 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T17:56:15.553 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.553 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.553 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:56:15.553 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6815", 2024-02-09T17:56:15.553 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T17:56:15.553 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.553 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.553 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.553 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.99:6811/3868328452", 2024-02-09T17:56:15.553 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.99:6813/3868328452", 2024-02-09T17:56:15.553 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.99:6817/3868328452", 2024-02-09T17:56:15.553 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.99:6815/3868328452", 2024-02-09T17:56:15.554 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:56:15.554 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:56:15.554 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:56:15.554 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.554 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.554 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.554 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 2, 2024-02-09T17:56:15.554 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "575f4257-7e34-485e-963c-2607b69e1220", 2024-02-09T17:56:15.554 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:56:15.554 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:56:15.554 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:56:15.554 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 0.3601837158203125, 2024-02-09T17:56:15.554 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:56:15.554 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:56:15.554 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 18, 2024-02-09T17:56:15.555 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 661, 2024-02-09T17:56:15.555 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:56:15.555 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:56:15.555 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:56:15.555 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:56:15.555 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.555 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:56:15.555 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6818", 2024-02-09T17:56:15.555 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T17:56:15.555 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.555 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.555 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:56:15.555 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6819", 2024-02-09T17:56:15.555 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T17:56:15.555 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.556 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.556 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.556 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:56:15.556 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:56:15.556 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.556 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:56:15.556 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6820", 2024-02-09T17:56:15.556 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T17:56:15.556 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.556 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.556 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:56:15.556 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6821", 2024-02-09T17:56:15.556 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T17:56:15.556 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.557 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.557 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.557 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:56:15.557 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:56:15.557 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.557 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:56:15.557 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6824", 2024-02-09T17:56:15.557 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T17:56:15.557 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.557 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.557 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:56:15.557 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6825", 2024-02-09T17:56:15.557 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T17:56:15.558 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.558 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.558 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.558 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:56:15.558 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:56:15.558 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.558 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:56:15.558 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6822", 2024-02-09T17:56:15.558 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T17:56:15.558 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.558 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.558 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:56:15.558 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6823", 2024-02-09T17:56:15.558 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T17:56:15.558 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.559 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.559 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.559 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.99:6819/537506237", 2024-02-09T17:56:15.559 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.99:6821/537506237", 2024-02-09T17:56:15.559 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.99:6825/537506237", 2024-02-09T17:56:15.559 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.99:6823/537506237", 2024-02-09T17:56:15.559 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:56:15.559 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:56:15.559 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:56:15.559 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.559 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.559 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.559 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 3, 2024-02-09T17:56:15.559 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "1a520bcf-81bb-4f1d-80fb-11e98008e429", 2024-02-09T17:56:15.559 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:56:15.559 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:56:15.560 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:56:15.560 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 0.8014984130859375, 2024-02-09T17:56:15.560 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:56:15.560 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:56:15.560 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 27, 2024-02-09T17:56:15.560 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 785, 2024-02-09T17:56:15.560 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:56:15.560 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:56:15.560 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:56:15.560 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:56:15.560 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.560 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:56:15.560 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6826", 2024-02-09T17:56:15.560 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 762784793 2024-02-09T17:56:15.560 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.561 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.561 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:56:15.561 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6827", 2024-02-09T17:56:15.561 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 762784793 2024-02-09T17:56:15.561 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.561 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.561 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.561 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:56:15.561 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:56:15.561 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.561 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:56:15.561 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6828", 2024-02-09T17:56:15.561 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 762784793 2024-02-09T17:56:15.561 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.562 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.562 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:56:15.562 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6829", 2024-02-09T17:56:15.562 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 762784793 2024-02-09T17:56:15.562 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.562 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.562 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.562 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:56:15.562 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:56:15.562 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.562 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:56:15.562 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6832", 2024-02-09T17:56:15.562 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 762784793 2024-02-09T17:56:15.562 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.562 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.563 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:56:15.563 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6833", 2024-02-09T17:56:15.563 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 762784793 2024-02-09T17:56:15.563 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.563 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.563 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.563 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:56:15.563 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:56:15.563 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.563 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:56:15.563 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6830", 2024-02-09T17:56:15.563 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 762784793 2024-02-09T17:56:15.563 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.563 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.564 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:56:15.564 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6831", 2024-02-09T17:56:15.564 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 762784793 2024-02-09T17:56:15.564 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.564 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.564 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.564 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.99:6827/762784793", 2024-02-09T17:56:15.564 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.99:6829/762784793", 2024-02-09T17:56:15.564 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.99:6833/762784793", 2024-02-09T17:56:15.564 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.99:6831/762784793", 2024-02-09T17:56:15.564 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:56:15.564 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:56:15.564 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:56:15.564 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.565 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.565 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.565 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 4, 2024-02-09T17:56:15.565 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "4617c2fe-c9d8-40cd-887b-303e3215bdf8", 2024-02-09T17:56:15.565 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:56:15.565 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:56:15.565 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:56:15.565 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T17:56:15.565 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:56:15.565 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:56:15.565 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 32, 2024-02-09T17:56:15.565 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 776, 2024-02-09T17:56:15.565 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:56:15.565 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:56:15.565 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:56:15.566 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:56:15.566 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.566 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:56:15.566 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6800", 2024-02-09T17:56:15.566 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T17:56:15.566 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.566 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.566 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:56:15.566 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6801", 2024-02-09T17:56:15.566 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T17:56:15.566 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.566 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.566 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.566 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:56:15.566 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:56:15.567 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.567 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:56:15.567 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6802", 2024-02-09T17:56:15.567 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T17:56:15.567 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.567 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.567 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:56:15.567 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6803", 2024-02-09T17:56:15.567 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T17:56:15.567 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.567 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.567 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.567 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:56:15.567 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:56:15.567 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.568 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:56:15.568 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6806", 2024-02-09T17:56:15.568 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T17:56:15.568 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.568 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.568 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:56:15.568 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6807", 2024-02-09T17:56:15.568 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T17:56:15.568 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.568 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.568 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.568 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:56:15.568 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:56:15.568 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.568 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:56:15.569 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6804", 2024-02-09T17:56:15.569 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T17:56:15.569 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.569 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.569 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:56:15.569 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6805", 2024-02-09T17:56:15.569 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T17:56:15.569 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.569 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.569 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.569 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.162:6801/4178676504", 2024-02-09T17:56:15.569 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.162:6803/4178676504", 2024-02-09T17:56:15.569 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.162:6807/4178676504", 2024-02-09T17:56:15.569 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.162:6805/4178676504", 2024-02-09T17:56:15.569 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:56:15.570 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:56:15.570 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:56:15.570 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.570 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.570 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.570 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 5, 2024-02-09T17:56:15.570 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "da98e3f3-173e-46fe-be8b-1ccbe8b0dabc", 2024-02-09T17:56:15.570 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:56:15.570 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:56:15.570 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:56:15.570 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 0, 2024-02-09T17:56:15.570 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:56:15.570 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:56:15.570 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 40, 2024-02-09T17:56:15.570 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 715, 2024-02-09T17:56:15.571 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:56:15.571 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:56:15.571 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:56:15.571 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:56:15.571 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.571 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:56:15.571 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6808", 2024-02-09T17:56:15.571 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T17:56:15.571 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.571 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.571 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:56:15.571 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6809", 2024-02-09T17:56:15.571 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T17:56:15.571 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.571 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.572 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.572 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:56:15.572 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:56:15.572 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.572 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:56:15.572 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6810", 2024-02-09T17:56:15.572 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T17:56:15.572 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.572 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.572 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:56:15.572 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6811", 2024-02-09T17:56:15.572 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T17:56:15.572 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.572 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.572 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.573 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:56:15.573 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:56:15.573 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.573 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:56:15.573 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6814", 2024-02-09T17:56:15.573 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T17:56:15.573 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.573 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.573 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:56:15.573 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6815", 2024-02-09T17:56:15.573 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T17:56:15.573 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.573 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.573 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.573 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:56:15.574 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:56:15.574 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.574 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:56:15.574 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6812", 2024-02-09T17:56:15.574 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T17:56:15.574 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.574 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.574 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:56:15.574 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6813", 2024-02-09T17:56:15.574 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T17:56:15.574 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.574 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.574 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.574 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.162:6809/1489371339", 2024-02-09T17:56:15.574 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.162:6811/1489371339", 2024-02-09T17:56:15.575 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.162:6815/1489371339", 2024-02-09T17:56:15.575 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.162:6813/1489371339", 2024-02-09T17:56:15.575 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:56:15.575 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:56:15.575 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:56:15.575 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.575 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.575 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.575 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 6, 2024-02-09T17:56:15.575 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "ff50382d-5ddf-4cda-bb0c-ff055feeaa40", 2024-02-09T17:56:15.575 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:56:15.575 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:56:15.575 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:56:15.575 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T17:56:15.575 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:56:15.575 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:56:15.576 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 49, 2024-02-09T17:56:15.576 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 776, 2024-02-09T17:56:15.576 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:56:15.576 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:56:15.576 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:56:15.576 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:56:15.576 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.576 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:56:15.576 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6816", 2024-02-09T17:56:15.576 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T17:56:15.576 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.576 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.576 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:56:15.576 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6817", 2024-02-09T17:56:15.576 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T17:56:15.577 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.577 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.577 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.577 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:56:15.577 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:56:15.577 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.577 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:56:15.577 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6818", 2024-02-09T17:56:15.577 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T17:56:15.577 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.577 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.577 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:56:15.577 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6819", 2024-02-09T17:56:15.577 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T17:56:15.577 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.578 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.578 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.578 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:56:15.578 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:56:15.578 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.578 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:56:15.578 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6822", 2024-02-09T17:56:15.578 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T17:56:15.578 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.578 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.578 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:56:15.578 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6823", 2024-02-09T17:56:15.578 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T17:56:15.578 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.578 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.578 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.579 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:56:15.579 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:56:15.579 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.579 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:56:15.579 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6820", 2024-02-09T17:56:15.579 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T17:56:15.579 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.579 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.579 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:56:15.579 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6821", 2024-02-09T17:56:15.579 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T17:56:15.579 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.579 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.579 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.579 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.162:6817/821169663", 2024-02-09T17:56:15.580 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.162:6819/821169663", 2024-02-09T17:56:15.580 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.162:6823/821169663", 2024-02-09T17:56:15.580 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.162:6821/821169663", 2024-02-09T17:56:15.580 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:56:15.580 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:56:15.580 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:56:15.580 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.580 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.580 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.580 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 7, 2024-02-09T17:56:15.580 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "a82984dc-5e22-424b-a446-ae2ee71eee19", 2024-02-09T17:56:15.580 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:56:15.580 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:56:15.580 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:56:15.581 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 0.5982513427734375, 2024-02-09T17:56:15.581 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:56:15.581 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:56:15.581 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 57, 2024-02-09T17:56:15.581 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 661, 2024-02-09T17:56:15.581 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:56:15.581 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:56:15.581 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:56:15.581 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:56:15.581 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.581 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:56:15.581 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6824", 2024-02-09T17:56:15.581 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T17:56:15.581 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.581 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.582 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:56:15.582 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6825", 2024-02-09T17:56:15.582 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T17:56:15.582 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.582 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.582 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.582 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:56:15.582 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:56:15.582 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.582 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:56:15.582 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6826", 2024-02-09T17:56:15.582 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T17:56:15.582 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.582 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.582 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:56:15.582 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6827", 2024-02-09T17:56:15.583 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T17:56:15.583 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.583 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.583 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.583 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:56:15.583 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:56:15.583 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.583 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:56:15.583 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6830", 2024-02-09T17:56:15.583 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T17:56:15.583 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.583 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.583 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:56:15.583 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6831", 2024-02-09T17:56:15.583 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T17:56:15.584 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.584 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.584 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.584 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:56:15.584 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:56:15.584 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.584 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:56:15.584 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6828", 2024-02-09T17:56:15.584 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T17:56:15.584 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.584 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.584 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:56:15.584 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6829", 2024-02-09T17:56:15.584 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T17:56:15.584 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.585 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.585 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.585 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.162:6825/965264369", 2024-02-09T17:56:15.585 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.162:6827/965264369", 2024-02-09T17:56:15.585 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.162:6831/965264369", 2024-02-09T17:56:15.585 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.162:6829/965264369", 2024-02-09T17:56:15.585 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:56:15.585 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:56:15.585 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:56:15.585 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.585 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.585 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.585 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 8, 2024-02-09T17:56:15.585 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "63f9d79f-0400-44dc-b388-df00fefb5820", 2024-02-09T17:56:15.585 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:56:15.586 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:56:15.586 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:56:15.586 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T17:56:15.586 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 66, 2024-02-09T17:56:15.586 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 145, 2024-02-09T17:56:15.586 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 171, 2024-02-09T17:56:15.586 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 790, 2024-02-09T17:56:15.586 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 146, 2024-02-09T17:56:15.586 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:56:15.586 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:56:15.586 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:56:15.586 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.586 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:56:15.586 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6800", 2024-02-09T17:56:15.586 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T17:56:15.587 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.587 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.587 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:56:15.587 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6801", 2024-02-09T17:56:15.587 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T17:56:15.587 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.587 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.587 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.587 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:56:15.587 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:56:15.587 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.587 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:56:15.587 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6802", 2024-02-09T17:56:15.587 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T17:56:15.587 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.588 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.588 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:56:15.588 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6803", 2024-02-09T17:56:15.588 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T17:56:15.588 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.588 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.588 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.588 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:56:15.588 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:56:15.588 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.588 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:56:15.588 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6806", 2024-02-09T17:56:15.588 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T17:56:15.588 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.588 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.589 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:56:15.589 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6807", 2024-02-09T17:56:15.589 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T17:56:15.589 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.589 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.589 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.589 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:56:15.589 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:56:15.589 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.589 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:56:15.589 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6804", 2024-02-09T17:56:15.589 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T17:56:15.589 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.589 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.589 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:56:15.590 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6805", 2024-02-09T17:56:15.590 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T17:56:15.590 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.590 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.590 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.590 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.179:6801/2558294495", 2024-02-09T17:56:15.590 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.179:6803/2558294495", 2024-02-09T17:56:15.590 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.179:6807/2558294495", 2024-02-09T17:56:15.590 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.179:6805/2558294495", 2024-02-09T17:56:15.590 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:56:15.590 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:56:15.590 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:56:15.590 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.590 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.590 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.590 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 9, 2024-02-09T17:56:15.591 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "7d995854-7328-4dfc-b1a9-0edca8d74f46", 2024-02-09T17:56:15.591 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:56:15.591 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:56:15.591 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:56:15.591 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T17:56:15.591 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:56:15.591 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:56:15.591 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 77, 2024-02-09T17:56:15.591 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 785, 2024-02-09T17:56:15.591 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:56:15.591 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:56:15.591 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:56:15.591 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:56:15.591 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.591 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:56:15.591 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6808", 2024-02-09T17:56:15.592 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T17:56:15.592 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.592 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.592 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:56:15.592 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6809", 2024-02-09T17:56:15.592 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T17:56:15.592 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.592 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.592 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.592 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:56:15.592 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:56:15.592 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.592 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:56:15.592 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6810", 2024-02-09T17:56:15.593 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T17:56:15.593 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.593 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.593 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:56:15.593 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6811", 2024-02-09T17:56:15.593 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T17:56:15.593 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.593 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.593 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.593 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:56:15.593 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:56:15.593 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.593 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:56:15.593 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6814", 2024-02-09T17:56:15.593 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T17:56:15.593 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.594 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.594 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:56:15.594 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6815", 2024-02-09T17:56:15.594 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T17:56:15.594 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.594 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.594 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.594 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:56:15.594 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:56:15.594 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.594 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:56:15.594 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6812", 2024-02-09T17:56:15.594 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T17:56:15.594 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.594 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.595 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:56:15.595 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6813", 2024-02-09T17:56:15.595 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T17:56:15.595 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.595 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.595 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.595 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.179:6809/504427183", 2024-02-09T17:56:15.595 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.179:6811/504427183", 2024-02-09T17:56:15.595 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.179:6815/504427183", 2024-02-09T17:56:15.595 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.179:6813/504427183", 2024-02-09T17:56:15.595 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:56:15.595 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:56:15.595 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:56:15.595 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.595 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.596 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.596 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 10, 2024-02-09T17:56:15.596 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "151fbedd-08e6-46e6-a4e5-a14248399f61", 2024-02-09T17:56:15.596 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:56:15.596 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:56:15.596 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:56:15.596 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T17:56:15.596 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:56:15.596 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:56:15.596 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 84, 2024-02-09T17:56:15.596 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 790, 2024-02-09T17:56:15.596 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:56:15.596 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:56:15.596 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:56:15.596 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:56:15.597 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.597 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:56:15.597 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6816", 2024-02-09T17:56:15.597 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T17:56:15.597 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.597 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.597 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:56:15.597 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6817", 2024-02-09T17:56:15.597 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T17:56:15.597 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.597 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.597 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.597 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:56:15.597 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:56:15.597 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.598 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:56:15.598 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6818", 2024-02-09T17:56:15.598 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T17:56:15.598 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.598 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.598 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:56:15.598 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6819", 2024-02-09T17:56:15.598 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T17:56:15.598 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.598 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.598 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.598 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:56:15.598 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:56:15.598 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.598 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:56:15.599 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6822", 2024-02-09T17:56:15.599 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T17:56:15.599 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.599 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.599 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:56:15.599 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6823", 2024-02-09T17:56:15.599 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T17:56:15.599 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.599 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.599 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.599 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:56:15.599 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:56:15.599 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.599 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:56:15.599 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6820", 2024-02-09T17:56:15.600 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T17:56:15.600 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.600 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.600 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:56:15.600 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6821", 2024-02-09T17:56:15.600 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T17:56:15.600 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.600 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.600 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.600 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.179:6817/2403345164", 2024-02-09T17:56:15.600 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.179:6819/2403345164", 2024-02-09T17:56:15.600 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.179:6823/2403345164", 2024-02-09T17:56:15.600 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.179:6821/2403345164", 2024-02-09T17:56:15.600 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:56:15.601 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:56:15.601 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:56:15.601 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.601 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.601 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.601 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 11, 2024-02-09T17:56:15.601 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "6d3a3ad8-417a-483f-991c-52549dbe7d73", 2024-02-09T17:56:15.601 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:56:15.601 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:56:15.601 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:56:15.601 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T17:56:15.601 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:56:15.601 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:56:15.601 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 90, 2024-02-09T17:56:15.601 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 785, 2024-02-09T17:56:15.601 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:56:15.602 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:56:15.602 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:56:15.602 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:56:15.602 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.602 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:56:15.602 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6824", 2024-02-09T17:56:15.602 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T17:56:15.602 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.602 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.602 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:56:15.602 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6825", 2024-02-09T17:56:15.602 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T17:56:15.602 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.602 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.602 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.603 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:56:15.603 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:56:15.603 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.603 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:56:15.603 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6826", 2024-02-09T17:56:15.603 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T17:56:15.603 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.603 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.603 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:56:15.603 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6827", 2024-02-09T17:56:15.603 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T17:56:15.603 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.603 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.603 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.603 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:56:15.604 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:56:15.604 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.604 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:56:15.604 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6830", 2024-02-09T17:56:15.604 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T17:56:15.604 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.604 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.604 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:56:15.604 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6831", 2024-02-09T17:56:15.604 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T17:56:15.604 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.604 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.604 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.604 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:56:15.604 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:56:15.605 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.605 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:56:15.605 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6828", 2024-02-09T17:56:15.605 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T17:56:15.605 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.605 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.605 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:56:15.605 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6829", 2024-02-09T17:56:15.605 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T17:56:15.605 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.605 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.605 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.605 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.179:6825/2899402373", 2024-02-09T17:56:15.605 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.179:6827/2899402373", 2024-02-09T17:56:15.605 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.179:6831/2899402373", 2024-02-09T17:56:15.606 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.179:6829/2899402373", 2024-02-09T17:56:15.606 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:56:15.606 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:56:15.606 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:56:15.606 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.606 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.606 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T17:56:15.606 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_xinfo": [ 2024-02-09T17:56:15.606 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.606 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 0, 2024-02-09T17:56:15.606 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:56:15.606 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:56:15.606 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:56:15.606 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:56:15.606 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:56:15.607 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:55:31.124968+0000", 2024-02-09T17:56:15.607 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:56:15.607 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.607 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.607 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 1, 2024-02-09T17:56:15.607 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:56:15.607 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:56:15.607 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:56:15.607 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:56:15.607 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:56:15.607 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:55:34.085748+0000", 2024-02-09T17:56:15.607 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:56:15.607 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.607 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.607 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 2, 2024-02-09T17:56:15.608 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:56:15.608 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:56:15.608 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:56:15.608 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:56:15.608 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:56:15.608 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:55:14.894019+0000", 2024-02-09T17:56:15.608 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:56:15.608 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.608 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.608 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 3, 2024-02-09T17:56:15.608 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:56:15.608 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:56:15.608 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:56:15.608 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:56:15.608 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:56:15.609 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:55:39.048234+0000", 2024-02-09T17:56:15.609 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:56:15.609 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.609 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.609 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 4, 2024-02-09T17:56:15.609 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:56:15.609 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:56:15.609 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:56:15.609 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:56:15.609 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:56:15.609 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:55:33.806511+0000", 2024-02-09T17:56:15.609 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:56:15.609 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.609 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.609 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 5, 2024-02-09T17:56:15.609 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:56:15.610 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:56:15.610 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:56:15.610 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:56:15.610 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:56:15.610 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:55:58.161842+0000", 2024-02-09T17:56:15.610 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:56:15.610 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.610 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.610 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 6, 2024-02-09T17:56:15.610 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:56:15.610 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:56:15.610 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:56:15.610 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:56:15.610 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:56:15.610 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:55:28.194418+0000", 2024-02-09T17:56:15.611 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:56:15.611 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.611 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.611 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 7, 2024-02-09T17:56:15.611 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:56:15.611 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:56:15.611 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:56:15.611 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:56:15.611 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:56:15.611 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:55:28.355488+0000", 2024-02-09T17:56:15.611 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:56:15.611 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.611 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.611 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 8, 2024-02-09T17:56:15.611 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:56:15.611 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:56:15.612 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:56:15.612 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:56:15.612 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:56:15.612 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:55:23.821377+0000", 2024-02-09T17:56:15.612 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 145 2024-02-09T17:56:15.612 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.612 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.612 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 9, 2024-02-09T17:56:15.612 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:56:15.612 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:56:15.612 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:56:15.612 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:56:15.612 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:56:15.612 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:55:26.565122+0000", 2024-02-09T17:56:15.613 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:56:15.613 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.613 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.613 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 10, 2024-02-09T17:56:15.613 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:56:15.613 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:56:15.613 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:56:15.613 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:56:15.613 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:56:15.613 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:55:41.413332+0000", 2024-02-09T17:56:15.613 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:56:15.613 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.613 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.613 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 11, 2024-02-09T17:56:15.613 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:56:15.614 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:56:15.614 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:56:15.614 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:56:15.614 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:56:15.614 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:55:52.237074+0000", 2024-02-09T17:56:15.614 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:56:15.614 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.614 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T17:56:15.614 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_upmap": [ 2024-02-09T17:56:15.614 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.614 INFO:teuthology.orchestra.run.smithi099.stdout: "pgid": "1.7", 2024-02-09T17:56:15.614 INFO:teuthology.orchestra.run.smithi099.stdout: "osds": [ 2024-02-09T17:56:15.614 INFO:teuthology.orchestra.run.smithi099.stdout: 1, 2024-02-09T17:56:15.614 INFO:teuthology.orchestra.run.smithi099.stdout: 8, 2024-02-09T17:56:15.614 INFO:teuthology.orchestra.run.smithi099.stdout: 6 2024-02-09T17:56:15.614 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.615 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.615 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T17:56:15.615 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_upmap_items": [ 2024-02-09T17:56:15.615 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.615 INFO:teuthology.orchestra.run.smithi099.stdout: "pgid": "1.1e", 2024-02-09T17:56:15.615 INFO:teuthology.orchestra.run.smithi099.stdout: "mappings": [ 2024-02-09T17:56:15.615 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:56:15.615 INFO:teuthology.orchestra.run.smithi099.stdout: "from": 8, 2024-02-09T17:56:15.615 INFO:teuthology.orchestra.run.smithi099.stdout: "to": 3 2024-02-09T17:56:15.615 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.615 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:56:15.615 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.615 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T17:56:15.615 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_upmap_primaries": [], 2024-02-09T17:56:15.615 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_temp": [], 2024-02-09T17:56:15.616 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_temp": [], 2024-02-09T17:56:15.616 INFO:teuthology.orchestra.run.smithi099.stdout: "blocklist": { 2024-02-09T17:56:15.616 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:0/3904172782": "2024-02-10T17:16:24.229365+0000", 2024-02-09T17:56:15.616 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:6801/1586034576": "2024-02-10T17:16:24.229365+0000", 2024-02-09T17:56:15.616 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:6800/746248609": "2024-02-10T17:15:54.399201+0000", 2024-02-09T17:56:15.616 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:0/4092784197": "2024-02-10T17:16:24.229365+0000", 2024-02-09T17:56:15.616 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:0/2844598101": "2024-02-10T17:15:54.399201+0000", 2024-02-09T17:56:15.616 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:6801/746248609": "2024-02-10T17:15:54.399201+0000", 2024-02-09T17:56:15.616 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:0/2528750050": "2024-02-10T17:15:54.399201+0000", 2024-02-09T17:56:15.616 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:6800/1586034576": "2024-02-10T17:16:24.229365+0000", 2024-02-09T17:56:15.616 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:0/2214140078": "2024-02-10T17:15:54.399201+0000", 2024-02-09T17:56:15.616 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:0/4217821965": "2024-02-10T17:16:24.229365+0000" 2024-02-09T17:56:15.616 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.616 INFO:teuthology.orchestra.run.smithi099.stdout: "range_blocklist": {}, 2024-02-09T17:56:15.616 INFO:teuthology.orchestra.run.smithi099.stdout: "erasure_code_profiles": { 2024-02-09T17:56:15.617 INFO:teuthology.orchestra.run.smithi099.stdout: "default": { 2024-02-09T17:56:15.617 INFO:teuthology.orchestra.run.smithi099.stdout: "crush-failure-domain": "osd", 2024-02-09T17:56:15.617 INFO:teuthology.orchestra.run.smithi099.stdout: "k": "2", 2024-02-09T17:56:15.617 INFO:teuthology.orchestra.run.smithi099.stdout: "m": "1", 2024-02-09T17:56:15.617 INFO:teuthology.orchestra.run.smithi099.stdout: "plugin": "jerasure", 2024-02-09T17:56:15.617 INFO:teuthology.orchestra.run.smithi099.stdout: "technique": "reed_sol_van" 2024-02-09T17:56:15.617 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.617 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:56:15.617 INFO:teuthology.orchestra.run.smithi099.stdout: "removed_snaps_queue": [], 2024-02-09T17:56:15.617 INFO:teuthology.orchestra.run.smithi099.stdout: "new_removed_snaps": [], 2024-02-09T17:56:15.617 INFO:teuthology.orchestra.run.smithi099.stdout: "new_purged_snaps": [], 2024-02-09T17:56:15.617 INFO:teuthology.orchestra.run.smithi099.stdout: "crush_node_flags": {}, 2024-02-09T17:56:15.617 INFO:teuthology.orchestra.run.smithi099.stdout: "device_class_flags": {}, 2024-02-09T17:56:15.617 INFO:teuthology.orchestra.run.smithi099.stdout: "stretch_mode": { 2024-02-09T17:56:15.617 INFO:teuthology.orchestra.run.smithi099.stdout: "stretch_mode_enabled": false, 2024-02-09T17:56:15.618 INFO:teuthology.orchestra.run.smithi099.stdout: "stretch_bucket_count": 0, 2024-02-09T17:56:15.618 INFO:teuthology.orchestra.run.smithi099.stdout: "degraded_stretch_mode": 0, 2024-02-09T17:56:15.618 INFO:teuthology.orchestra.run.smithi099.stdout: "recovering_stretch_mode": 0, 2024-02-09T17:56:15.618 INFO:teuthology.orchestra.run.smithi099.stdout: "stretch_mode_bucket": 0 2024-02-09T17:56:15.618 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:56:15.618 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:56:15.618 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:56:15.619 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 49 16 84894 84878 108.227 69.8125 0.0143558 0.00923544 2024-02-09T17:56:15.620 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:56:15.620 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:56:15.632 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:56:15.645 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:15 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/894384329' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:56:15.647 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:15.663 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:56:15.668 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:56:15.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:15.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:56:15.676 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:56:15.676 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:56:15.676 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:56:15.676 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:56:15.676 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:56:15.676 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:56:15.676 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:56:15.676 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:56:15.676 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:56:15.676 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:56:15.677 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:56:15.677 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:56:15.772 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:56:15.806 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:16.123 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.10 dump_ops_in_flight 2024-02-09T17:56:16.163 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:56:16.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:16.168 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:56:16.269 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:56:16.298 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:16.319 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:56:16.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:16.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:56:16.329 INFO:tasks.thrashosds.thrasher:j is {'epoch': 809, 'fsid': 'a69c870a-c76e-11ee-95b7-87774f69a715', 'created': '2024-02-09T17:15:26.113606+0000', 'modified': '2024-02-09T17:56:07.253898+0000', 'last_up_change': '2024-02-09T17:28:17.711044+0000', 'last_in_change': '2024-02-09T17:43:24.138763+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 27, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 8, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-09T17:18:56.552286+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '544', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '514', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 5.989999771118164, 'score_stable': 2.130000114440918, 'optimal_score': 1.090000033378601, 'raw_score_acting': 6.53000020980835, 'raw_score_stable': 2.319999933242798, 'primary_affinity_weighted': 0.7807959914207458, 'average_primary_affinity': 0.8500000238418579, 'average_primary_affinity_weighted': 0.7799999713897705, 'error_message': 'pool 1 has acting primaries on OSD(s) with primary affinity 0, read balance score is not accurate'}}, {'pool': 8, 'pool_name': 'unique_pool_6', 'create_time': '2024-02-09T17:55:15.183490+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '793', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.75, 'score_stable': 2.75, 'optimal_score': 1, 'raw_score_acting': 2.75, 'raw_score_stable': 2.75, 'primary_affinity_weighted': 0.7807959914207458, 'average_primary_affinity': 0.8500000238418579, 'average_primary_affinity_weighted': 0.7799999713897705}}], 'osds': [{'osd': 0, 'uuid': '21739aa7-e278-45da-9a3f-867385f01baf', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.609619140625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 785, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6802', 'nonce': 3859868923}, {'type': 'v1', 'addr': '172.21.15.99:6803', 'nonce': 3859868923}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6804', 'nonce': 3859868923}, {'type': 'v1', 'addr': '172.21.15.99:6805', 'nonce': 3859868923}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6808', 'nonce': 3859868923}, {'type': 'v1', 'addr': '172.21.15.99:6809', 'nonce': 3859868923}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6806', 'nonce': 3859868923}, {'type': 'v1', 'addr': '172.21.15.99:6807', 'nonce': 3859868923}]}, 'public_addr': '172.21.15.99:6803/3859868923', 'cluster_addr': '172.21.15.99:6805/3859868923', 'heartbeat_back_addr': '172.21.15.99:6809/3859868923', 'heartbeat_front_addr': '172.21.15.99:6807/3859868923', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '8eab347c-9065-4de1-93ae-e2eb7e11cab7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 785, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6810', 'nonce': 3868328452}, {'type': 'v1', 'addr': '172.21.15.99:6811', 'nonce': 3868328452}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6812', 'nonce': 3868328452}, {'type': 'v1', 'addr': '172.21.15.99:6813', 'nonce': 3868328452}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6816', 'nonce': 3868328452}, {'type': 'v1', 'addr': '172.21.15.99:6817', 'nonce': 3868328452}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6814', 'nonce': 3868328452}, {'type': 'v1', 'addr': '172.21.15.99:6815', 'nonce': 3868328452}]}, 'public_addr': '172.21.15.99:6811/3868328452', 'cluster_addr': '172.21.15.99:6813/3868328452', 'heartbeat_back_addr': '172.21.15.99:6817/3868328452', 'heartbeat_front_addr': '172.21.15.99:6815/3868328452', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '575f4257-7e34-485e-963c-2607b69e1220', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.3601837158203125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 661, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6818', 'nonce': 537506237}, {'type': 'v1', 'addr': '172.21.15.99:6819', 'nonce': 537506237}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6820', 'nonce': 537506237}, {'type': 'v1', 'addr': '172.21.15.99:6821', 'nonce': 537506237}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6824', 'nonce': 537506237}, {'type': 'v1', 'addr': '172.21.15.99:6825', 'nonce': 537506237}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6822', 'nonce': 537506237}, {'type': 'v1', 'addr': '172.21.15.99:6823', 'nonce': 537506237}]}, 'public_addr': '172.21.15.99:6819/537506237', 'cluster_addr': '172.21.15.99:6821/537506237', 'heartbeat_back_addr': '172.21.15.99:6825/537506237', 'heartbeat_front_addr': '172.21.15.99:6823/537506237', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '1a520bcf-81bb-4f1d-80fb-11e98008e429', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.8014984130859375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 785, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6826', 'nonce': 762784793}, {'type': 'v1', 'addr': '172.21.15.99:6827', 'nonce': 762784793}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6828', 'nonce': 762784793}, {'type': 'v1', 'addr': '172.21.15.99:6829', 'nonce': 762784793}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6832', 'nonce': 762784793}, {'type': 'v1', 'addr': '172.21.15.99:6833', 'nonce': 762784793}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6830', 'nonce': 762784793}, {'type': 'v1', 'addr': '172.21.15.99:6831', 'nonce': 762784793}]}, 'public_addr': '172.21.15.99:6827/762784793', 'cluster_addr': '172.21.15.99:6829/762784793', 'heartbeat_back_addr': '172.21.15.99:6833/762784793', 'heartbeat_front_addr': '172.21.15.99:6831/762784793', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '4617c2fe-c9d8-40cd-887b-303e3215bdf8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 776, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6800', 'nonce': 4178676504}, {'type': 'v1', 'addr': '172.21.15.162:6801', 'nonce': 4178676504}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6802', 'nonce': 4178676504}, {'type': 'v1', 'addr': '172.21.15.162:6803', 'nonce': 4178676504}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6806', 'nonce': 4178676504}, {'type': 'v1', 'addr': '172.21.15.162:6807', 'nonce': 4178676504}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6804', 'nonce': 4178676504}, {'type': 'v1', 'addr': '172.21.15.162:6805', 'nonce': 4178676504}]}, 'public_addr': '172.21.15.162:6801/4178676504', 'cluster_addr': '172.21.15.162:6803/4178676504', 'heartbeat_back_addr': '172.21.15.162:6807/4178676504', 'heartbeat_front_addr': '172.21.15.162:6805/4178676504', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'da98e3f3-173e-46fe-be8b-1ccbe8b0dabc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 715, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6808', 'nonce': 1489371339}, {'type': 'v1', 'addr': '172.21.15.162:6809', 'nonce': 1489371339}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6810', 'nonce': 1489371339}, {'type': 'v1', 'addr': '172.21.15.162:6811', 'nonce': 1489371339}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6814', 'nonce': 1489371339}, {'type': 'v1', 'addr': '172.21.15.162:6815', 'nonce': 1489371339}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6812', 'nonce': 1489371339}, {'type': 'v1', 'addr': '172.21.15.162:6813', 'nonce': 1489371339}]}, 'public_addr': '172.21.15.162:6809/1489371339', 'cluster_addr': '172.21.15.162:6811/1489371339', 'heartbeat_back_addr': '172.21.15.162:6815/1489371339', 'heartbeat_front_addr': '172.21.15.162:6813/1489371339', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'ff50382d-5ddf-4cda-bb0c-ff055feeaa40', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 49, 'up_thru': 776, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6816', 'nonce': 821169663}, {'type': 'v1', 'addr': '172.21.15.162:6817', 'nonce': 821169663}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6818', 'nonce': 821169663}, {'type': 'v1', 'addr': '172.21.15.162:6819', 'nonce': 821169663}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6822', 'nonce': 821169663}, {'type': 'v1', 'addr': '172.21.15.162:6823', 'nonce': 821169663}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6820', 'nonce': 821169663}, {'type': 'v1', 'addr': '172.21.15.162:6821', 'nonce': 821169663}]}, 'public_addr': '172.21.15.162:6817/821169663', 'cluster_addr': '172.21.15.162:6819/821169663', 'heartbeat_back_addr': '172.21.15.162:6823/821169663', 'heartbeat_front_addr': '172.21.15.162:6821/821169663', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'a82984dc-5e22-424b-a446-ae2ee71eee19', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.5982513427734375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 57, 'up_thru': 661, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6824', 'nonce': 965264369}, {'type': 'v1', 'addr': '172.21.15.162:6825', 'nonce': 965264369}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6826', 'nonce': 965264369}, {'type': 'v1', 'addr': '172.21.15.162:6827', 'nonce': 965264369}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6830', 'nonce': 965264369}, {'type': 'v1', 'addr': '172.21.15.162:6831', 'nonce': 965264369}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6828', 'nonce': 965264369}, {'type': 'v1', 'addr': '172.21.15.162:6829', 'nonce': 965264369}]}, 'public_addr': '172.21.15.162:6825/965264369', 'cluster_addr': '172.21.15.162:6827/965264369', 'heartbeat_back_addr': '172.21.15.162:6831/965264369', 'heartbeat_front_addr': '172.21.15.162:6829/965264369', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '63f9d79f-0400-44dc-b388-df00fefb5820', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 66, 'last_clean_end': 145, 'up_from': 171, 'up_thru': 790, 'down_at': 146, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6800', 'nonce': 2558294495}, {'type': 'v1', 'addr': '172.21.15.179:6801', 'nonce': 2558294495}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6802', 'nonce': 2558294495}, {'type': 'v1', 'addr': '172.21.15.179:6803', 'nonce': 2558294495}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6806', 'nonce': 2558294495}, {'type': 'v1', 'addr': '172.21.15.179:6807', 'nonce': 2558294495}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6804', 'nonce': 2558294495}, {'type': 'v1', 'addr': '172.21.15.179:6805', 'nonce': 2558294495}]}, 'public_addr': '172.21.15.179:6801/2558294495', 'cluster_addr': '172.21.15.179:6803/2558294495', 'heartbeat_back_addr': '172.21.15.179:6807/2558294495', 'heartbeat_front_addr': '172.21.15.179:6805/2558294495', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '7d995854-7328-4dfc-b1a9-0edca8d74f46', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 77, 'up_thru': 785, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6808', 'nonce': 504427183}, {'type': 'v1', 'addr': '172.21.15.179:6809', 'nonce': 504427183}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6810', 'nonce': 504427183}, {'type': 'v1', 'addr': '172.21.15.179:6811', 'nonce': 504427183}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6814', 'nonce': 504427183}, {'type': 'v1', 'addr': '172.21.15.179:6815', 'nonce': 504427183}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6812', 'nonce': 504427183}, {'type': 'v1', 'addr': '172.21.15.179:6813', 'nonce': 504427183}]}, 'public_addr': '172.21.15.179:6809/504427183', 'cluster_addr': '172.21.15.179:6811/504427183', 'heartbeat_back_addr': '172.21.15.179:6815/504427183', 'heartbeat_front_addr': '172.21.15.179:6813/504427183', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '151fbedd-08e6-46e6-a4e5-a14248399f61', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 84, 'up_thru': 790, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6816', 'nonce': 2403345164}, {'type': 'v1', 'addr': '172.21.15.179:6817', 'nonce': 2403345164}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6818', 'nonce': 2403345164}, {'type': 'v1', 'addr': '172.21.15.179:6819', 'nonce': 2403345164}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6822', 'nonce': 2403345164}, {'type': 'v1', 'addr': '172.21.15.179:6823', 'nonce': 2403345164}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6820', 'nonce': 2403345164}, {'type': 'v1', 'addr': '172.21.15.179:6821', 'nonce': 2403345164}]}, 'public_addr': '172.21.15.179:6817/2403345164', 'cluster_addr': '172.21.15.179:6819/2403345164', 'heartbeat_back_addr': '172.21.15.179:6823/2403345164', 'heartbeat_front_addr': '172.21.15.179:6821/2403345164', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '6d3a3ad8-417a-483f-991c-52549dbe7d73', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 90, 'up_thru': 785, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6824', 'nonce': 2899402373}, {'type': 'v1', 'addr': '172.21.15.179:6825', 'nonce': 2899402373}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6826', 'nonce': 2899402373}, {'type': 'v1', 'addr': '172.21.15.179:6827', 'nonce': 2899402373}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6830', 'nonce': 2899402373}, {'type': 'v1', 'addr': '172.21.15.179:6831', 'nonce': 2899402373}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6828', 'nonce': 2899402373}, {'type': 'v1', 'addr': '172.21.15.179:6829', 'nonce': 2899402373}]}, 'public_addr': '172.21.15.179:6825/2899402373', 'cluster_addr': '172.21.15.179:6827/2899402373', 'heartbeat_back_addr': '172.21.15.179:6831/2899402373', 'heartbeat_front_addr': '172.21.15.179:6829/2899402373', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:55:31.124968+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:55:34.085748+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:55:14.894019+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:55:39.048234+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:55:33.806511+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:55:58.161842+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:55:28.194418+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:55:28.355488+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:55:23.821377+0000', 'dead_epoch': 145}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:55:26.565122+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:55:41.413332+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:55:52.237074+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.7', 'osds': [1, 8, 6]}], 'pg_upmap_items': [{'pgid': '1.1e', 'mappings': [{'from': 8, 'to': 3}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.99:0/3904172782': '2024-02-10T17:16:24.229365+0000', '172.21.15.99:6801/1586034576': '2024-02-10T17:16:24.229365+0000', '172.21.15.99:6800/746248609': '2024-02-10T17:15:54.399201+0000', '172.21.15.99:0/4092784197': '2024-02-10T17:16:24.229365+0000', '172.21.15.99:0/2844598101': '2024-02-10T17:15:54.399201+0000', '172.21.15.99:6801/746248609': '2024-02-10T17:15:54.399201+0000', '172.21.15.99:0/2528750050': '2024-02-10T17:15:54.399201+0000', '172.21.15.99:6800/1586034576': '2024-02-10T17:16:24.229365+0000', '172.21.15.99:0/2214140078': '2024-02-10T17:15:54.399201+0000', '172.21.15.99:0/4217821965': '2024-02-10T17:16:24.229365+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-09T17:56:16.329 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph pg dump --format=json 2024-02-09T17:56:16.423 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:56:16.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:16 smithi179 ceph-mon[43975]: pgmap v1994: 47 pgs: 47 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.24k op/s 2024-02-09T17:56:16.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:16 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3861515837' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-09T17:56:16.488 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:16.522 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:56:16.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:16.527 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:56:16.603 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 50 16 85905 85889 107.327 63.1875 0.014142 0.00931234 2024-02-09T17:56:16.607 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:56:16.628 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:56:16.656 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:16 smithi162 ceph-mon[44304]: pgmap v1994: 47 pgs: 47 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.24k op/s 2024-02-09T17:56:16.656 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:16 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3861515837' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-09T17:56:16.658 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:16.684 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:56:16.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:16.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:56:16.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:16 smithi099 ceph-mon[39420]: pgmap v1994: 47 pgs: 47 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.24k op/s 2024-02-09T17:56:16.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:16 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3861515837' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-09T17:56:16.790 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:56:16.844 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:17.603 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 51 16 87125 87109 106.717 76.25 0.0138276 0.00936619 2024-02-09T17:56:17.606 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:56:17.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:17.610 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:56:17.659 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:56:17.711 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:56:17.759 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:17.922 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:56:17.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:17.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:56:18.029 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:56:18.055 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:18.075 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:56:18.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:18.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:56:18.178 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:56:18.222 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:18.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:18 smithi099 ceph-mon[39420]: pgmap v1995: 47 pgs: 47 active+clean; 5.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-02-09T17:56:18.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:18 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:56:18.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:18 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:56:18.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:18 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:56:18.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:18 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:56:18.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:18 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:56:18.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:18 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:56:18.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:18 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:56:18.603 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 52 16 88269 88253 106.04 71.5 0.0194154 0.00942584 2024-02-09T17:56:18.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:18 smithi179 ceph-mon[43975]: pgmap v1995: 47 pgs: 47 active+clean; 5.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-02-09T17:56:18.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:18 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:56:18.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:18 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:56:18.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:18 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:56:18.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:18 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:56:18.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:18 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:56:18.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:18 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:56:18.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:18 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:56:18.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:18 smithi162 ceph-mon[44304]: pgmap v1995: 47 pgs: 47 active+clean; 5.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-02-09T17:56:18.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:18 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:56:18.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:18 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:56:18.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:18 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:56:18.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:18 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:56:18.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:18 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:56:18.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:18 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:56:18.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:18 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:56:18.887 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:56:18.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:18.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:56:18.992 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:56:19.064 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:19.326 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:56:19.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:19.331 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:56:19.432 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:56:19.475 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:19.603 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 53 16 89438 89422 105.417 73.0625 0.00917401 0.00948244 2024-02-09T17:56:19.661 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:56:19.699 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:56:19.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:19.710 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:56:19.811 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:56:19.885 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:20.093 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:56:20.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:20.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:56:20.151 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T17:56:20.151 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:56:20.197 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:56:20.232 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:20.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:20 smithi099 ceph-mon[39420]: pgmap v1996: 47 pgs: 47 active+clean; 5.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.23k op/s 2024-02-09T17:56:20.566 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:20 smithi179 ceph-mon[43975]: pgmap v1996: 47 pgs: 47 active+clean; 5.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.23k op/s 2024-02-09T17:56:20.603 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 54 16 90543 90527 104.744 69.0625 0.00753269 0.00954249 2024-02-09T17:56:20.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:20 smithi162 ceph-mon[44304]: pgmap v1996: 47 pgs: 47 active+clean; 5.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.23k op/s 2024-02-09T17:56:20.786 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:56:20.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:20.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:56:20.891 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:56:20.921 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:20.943 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:56:20.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:20.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:56:21.047 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:56:21.069 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:56:21.131 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:21.139 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:56:21.149 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-02-09T17:56:21.344 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is set 2024-02-09T17:56:21.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:21 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2527320910' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:56:21.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:21 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:56:21.604 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 55 16 91656 91640 104.104 69.5625 0.0114352 0.00960111 2024-02-09T17:56:21.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:21 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2527320910' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:56:21.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:21 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:56:21.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:21 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2527320910' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:56:21.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:21 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:56:22.055 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.10 dump_blocked_ops 2024-02-09T17:56:22.504 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:56:22.604 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 56 16 92741 92725 103.455 67.8125 0.0120567 0.00966192 2024-02-09T17:56:22.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:22 smithi179 ceph-mon[43975]: from='client.26342 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:56:22.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:22 smithi179 ceph-mon[43975]: pgmap v1997: 47 pgs: 47 active+clean; 5.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s 2024-02-09T17:56:22.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:22 smithi179 ceph-mon[43975]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:56:22.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:22 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:56:22.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:22 smithi179 ceph-mon[43975]: osdmap e810: 12 total, 12 up, 12 in 2024-02-09T17:56:22.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:22 smithi162 ceph-mon[44304]: from='client.26342 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:56:22.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:22 smithi162 ceph-mon[44304]: pgmap v1997: 47 pgs: 47 active+clean; 5.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s 2024-02-09T17:56:22.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:22 smithi162 ceph-mon[44304]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:56:22.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:22 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:56:22.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:22 smithi162 ceph-mon[44304]: osdmap e810: 12 total, 12 up, 12 in 2024-02-09T17:56:22.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:22 smithi099 ceph-mon[39420]: from='client.26342 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:56:22.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:22 smithi099 ceph-mon[39420]: pgmap v1997: 47 pgs: 47 active+clean; 5.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s 2024-02-09T17:56:22.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:22 smithi099 ceph-mon[39420]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:56:22.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:22 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:56:22.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:22 smithi099 ceph-mon[39420]: osdmap e810: 12 total, 12 up, 12 in 2024-02-09T17:56:23.604 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 57 16 93878 93862 102.887 71.0625 0.0160076 0.00971512 2024-02-09T17:56:24.127 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:56:24.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:24.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:56:24.235 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:56:24.284 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:24.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:24 smithi099 ceph-mon[39420]: pgmap v1999: 47 pgs: 47 active+clean; 5.6 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.13k op/s 2024-02-09T17:56:24.604 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 58 16 95043 95027 102.368 72.8125 0.0138809 0.00976367 2024-02-09T17:56:24.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:24 smithi179 ceph-mon[43975]: pgmap v1999: 47 pgs: 47 active+clean; 5.6 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.13k op/s 2024-02-09T17:56:24.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:24 smithi162 ceph-mon[44304]: pgmap v1999: 47 pgs: 47 active+clean; 5.6 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.13k op/s 2024-02-09T17:56:25.254 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:56:25.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:25.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:56:25.364 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:56:25.427 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:25.549 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:56:25.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:25.554 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:56:25.604 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 59 16 96046 96030 101.696 62.6875 0.0120776 0.00982781 2024-02-09T17:56:25.655 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:56:25.685 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:25.709 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:56:25.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:25.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:56:25.795 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":1996,"stamp":"2024-02-09T17:56:19.186925+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5681381920,"num_objects":86684,"num_object_clones":0,"num_object_copies":260052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86682,"num_write_kb":5547648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90991,"ondisk_log_size":90991,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":25251824,"kb_used_data":16432152,"kb_used_omap":229,"kb_used_meta":8819162,"kb_avail":1099690000,"statfs":{"total":1151940427776,"available":1126082560000,"internally_reserved":0,"allocated":16826523648,"data_stored":16821529704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235127,"internal_metadata":9030822281},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":960823296,"num_objects":14661,"num_object_clones":0,"num_object_copies":43983,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14661,"num_write_kb":938304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.919600"},"pg_stats":[{"pgid":"1.1e","version":"63'139","reported_seq":912,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-02-09T17:56:07.276972+0000","last_change":"2024-02-09T17:55:17.114116+0000","last_active":"2024-02-09T17:56:07.276972+0000","last_peered":"2024-02-09T17:56:07.276972+0000","last_clean":"2024-02-09T17:56:07.276972+0000","last_became_active":"2024-02-09T17:55:08.938117+0000","last_became_peered":"2024-02-09T17:55:08.938117+0000","last_unstale":"2024-02-09T17:56:07.276972+0000","last_undegraded":"2024-02-09T17:56:07.276972+0000","last_fullsized":"2024-02-09T17:56:07.276972+0000","mapping_epoch":781,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":782,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:17.113960+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:55:17.113960+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:31.972827+0000","scrub_duration":125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,7],"acting":[11,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"63'139","reported_seq":925,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-02-09T17:56:07.273120+0000","last_change":"2024-02-09T17:54:59.163874+0000","last_active":"2024-02-09T17:56:07.273120+0000","last_peered":"2024-02-09T17:56:07.273120+0000","last_clean":"2024-02-09T17:56:07.273120+0000","last_became_active":"2024-02-09T17:43:28.594296+0000","last_became_peered":"2024-02-09T17:43:28.594296+0000","last_unstale":"2024-02-09T17:56:07.273120+0000","last_undegraded":"2024-02-09T17:56:07.273120+0000","last_fullsized":"2024-02-09T17:56:07.273120+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:54:59.163739+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:48:24.634092+0000","last_clean_scrub_stamp":"2024-02-09T17:54:59.163739+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:26.933451+0000","scrub_duration":123,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"63'139","reported_seq":1445,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-02-09T17:56:07.278286+0000","last_change":"2024-02-09T17:55:50.323999+0000","last_active":"2024-02-09T17:56:07.278286+0000","last_peered":"2024-02-09T17:56:07.278286+0000","last_clean":"2024-02-09T17:56:07.278286+0000","last_became_active":"2024-02-09T17:43:19.103584+0000","last_became_peered":"2024-02-09T17:43:19.103584+0000","last_unstale":"2024-02-09T17:56:07.278286+0000","last_undegraded":"2024-02-09T17:56:07.278286+0000","last_fullsized":"2024-02-09T17:56:07.278286+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:50.323880+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:23.912504+0000","last_clean_scrub_stamp":"2024-02-09T17:55:50.323880+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:57:14.267470+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"63'139","reported_seq":1243,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-02-09T17:56:07.278555+0000","last_change":"2024-02-09T17:55:11.475048+0000","last_active":"2024-02-09T17:56:07.278555+0000","last_peered":"2024-02-09T17:56:07.278555+0000","last_clean":"2024-02-09T17:56:07.278555+0000","last_became_active":"2024-02-09T17:43:19.102729+0000","last_became_peered":"2024-02-09T17:43:19.102729+0000","last_unstale":"2024-02-09T17:56:07.278555+0000","last_undegraded":"2024-02-09T17:56:07.278555+0000","last_fullsized":"2024-02-09T17:56:07.278555+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:11.474947+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:45:44.988633+0000","last_clean_scrub_stamp":"2024-02-09T17:55:11.474947+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:38.756171+0000","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"63'139","reported_seq":898,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-02-09T17:56:07.282152+0000","last_change":"2024-02-09T17:55:03.984081+0000","last_active":"2024-02-09T17:56:07.282152+0000","last_peered":"2024-02-09T17:56:07.282152+0000","last_clean":"2024-02-09T17:56:07.282152+0000","last_became_active":"2024-02-09T17:43:19.120814+0000","last_became_peered":"2024-02-09T17:43:19.120814+0000","last_unstale":"2024-02-09T17:56:07.282152+0000","last_undegraded":"2024-02-09T17:56:07.282152+0000","last_fullsized":"2024-02-09T17:56:07.282152+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:03.983998+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:55:03.983998+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:29.536408+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"63'139","reported_seq":1752,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-02-09T17:56:07.277181+0000","last_change":"2024-02-09T17:55:21.231646+0000","last_active":"2024-02-09T17:56:07.277181+0000","last_peered":"2024-02-09T17:56:07.277181+0000","last_clean":"2024-02-09T17:56:07.277181+0000","last_became_active":"2024-02-09T17:55:21.231291+0000","last_became_peered":"2024-02-09T17:55:21.231291+0000","last_unstale":"2024-02-09T17:56:07.277181+0000","last_undegraded":"2024-02-09T17:56:07.277181+0000","last_fullsized":"2024-02-09T17:56:07.277181+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":791,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:15.085677+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:30.193194+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.085677+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:27.587337+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"63'139","reported_seq":1589,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-02-09T17:56:07.279316+0000","last_change":"2024-02-09T17:54:50.053517+0000","last_active":"2024-02-09T17:56:07.279316+0000","last_peered":"2024-02-09T17:56:07.279316+0000","last_clean":"2024-02-09T17:56:07.279316+0000","last_became_active":"2024-02-09T17:43:28.585573+0000","last_became_peered":"2024-02-09T17:43:28.585573+0000","last_unstale":"2024-02-09T17:56:07.279316+0000","last_undegraded":"2024-02-09T17:56:07.279316+0000","last_fullsized":"2024-02-09T17:56:07.279316+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:54:50.053414+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:54:50.053414+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:13.442075+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"63'139","reported_seq":1035,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-02-09T17:56:07.278286+0000","last_change":"2024-02-09T17:55:04.470767+0000","last_active":"2024-02-09T17:56:07.278286+0000","last_peered":"2024-02-09T17:56:07.278286+0000","last_clean":"2024-02-09T17:56:07.278286+0000","last_became_active":"2024-02-09T17:48:46.184419+0000","last_became_peered":"2024-02-09T17:48:46.184419+0000","last_unstale":"2024-02-09T17:56:07.278286+0000","last_undegraded":"2024-02-09T17:56:07.278286+0000","last_fullsized":"2024-02-09T17:56:07.278286+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:04.470661+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:44:50.214686+0000","last_clean_scrub_stamp":"2024-02-09T17:55:04.470661+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:29.156638+0000","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"63'139","reported_seq":926,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-02-09T17:56:07.273000+0000","last_change":"2024-02-09T17:54:56.134959+0000","last_active":"2024-02-09T17:56:07.273000+0000","last_peered":"2024-02-09T17:56:07.273000+0000","last_clean":"2024-02-09T17:56:07.273000+0000","last_became_active":"2024-02-09T17:53:02.869767+0000","last_became_peered":"2024-02-09T17:53:02.869767+0000","last_unstale":"2024-02-09T17:56:07.273000+0000","last_undegraded":"2024-02-09T17:56:07.273000+0000","last_fullsized":"2024-02-09T17:56:07.273000+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:54:56.134826+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:11.970338+0000","last_clean_scrub_stamp":"2024-02-09T17:54:56.134826+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:19.204901+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"63'139","reported_seq":1347,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-02-09T17:56:07.285357+0000","last_change":"2024-02-09T17:55:51.743478+0000","last_active":"2024-02-09T17:56:07.285357+0000","last_peered":"2024-02-09T17:56:07.285357+0000","last_clean":"2024-02-09T17:56:07.285357+0000","last_became_active":"2024-02-09T17:43:19.138398+0000","last_became_peered":"2024-02-09T17:43:19.138398+0000","last_unstale":"2024-02-09T17:56:07.285357+0000","last_undegraded":"2024-02-09T17:56:07.285357+0000","last_fullsized":"2024-02-09T17:56:07.285357+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:51.743366+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:55:51.743366+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:57:15.219835+0000","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"63'139","reported_seq":1417,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-02-09T17:56:07.268647+0000","last_change":"2024-02-09T17:55:09.116245+0000","last_active":"2024-02-09T17:56:07.268647+0000","last_peered":"2024-02-09T17:56:07.268647+0000","last_clean":"2024-02-09T17:56:07.268647+0000","last_became_active":"2024-02-09T17:43:28.589307+0000","last_became_peered":"2024-02-09T17:43:28.589307+0000","last_unstale":"2024-02-09T17:56:07.268647+0000","last_undegraded":"2024-02-09T17:56:07.268647+0000","last_fullsized":"2024-02-09T17:56:07.268647+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:09.116128+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:38:26.968892+0000","last_clean_scrub_stamp":"2024-02-09T17:55:09.116128+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:31.309149+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"809'5483","reported_seq":5541,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-02-09T17:56:18.492472+0000","last_change":"2024-02-09T17:55:16.444774+0000","last_active":"2024-02-09T17:56:18.492472+0000","last_peered":"2024-02-09T17:56:18.492472+0000","last_clean":"2024-02-09T17:56:18.492472+0000","last_became_active":"2024-02-09T17:55:16.444412+0000","last_became_peered":"2024-02-09T17:55:16.444412+0000","last_unstale":"2024-02-09T17:56:18.492472+0000","last_undegraded":"2024-02-09T17:56:18.492472+0000","last_fullsized":"2024-02-09T17:56:18.492472+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":5483,"log_dups_size":0,"ondisk_log_size":5483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:39.138631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":359333888,"num_objects":5483,"num_object_clones":0,"num_object_copies":16449,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5483,"num_write_kb":350912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,0],"acting":[10,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"63'139","reported_seq":1996,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-02-09T17:56:07.285284+0000","last_change":"2024-02-09T17:55:19.636501+0000","last_active":"2024-02-09T17:56:07.285284+0000","last_peered":"2024-02-09T17:56:07.285284+0000","last_clean":"2024-02-09T17:56:07.285284+0000","last_became_active":"2024-02-09T17:41:14.521079+0000","last_became_peered":"2024-02-09T17:41:14.521079+0000","last_unstale":"2024-02-09T17:56:07.285284+0000","last_undegraded":"2024-02-09T17:56:07.285284+0000","last_fullsized":"2024-02-09T17:56:07.285284+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:19.636423+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:55:19.636423+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:27.811356+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"809'5393","reported_seq":5451,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-02-09T17:56:17.413820+0000","last_change":"2024-02-09T17:55:16.913627+0000","last_active":"2024-02-09T17:56:17.413820+0000","last_peered":"2024-02-09T17:56:17.413820+0000","last_clean":"2024-02-09T17:56:17.413820+0000","last_became_active":"2024-02-09T17:55:16.913155+0000","last_became_peered":"2024-02-09T17:55:16.913155+0000","last_unstale":"2024-02-09T17:56:17.413820+0000","last_undegraded":"2024-02-09T17:56:17.413820+0000","last_fullsized":"2024-02-09T17:56:17.413820+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":5393,"log_dups_size":0,"ondisk_log_size":5393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:36.474670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353435648,"num_objects":5393,"num_object_clones":0,"num_object_copies":16179,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5393,"num_write_kb":345152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,1],"acting":[8,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.2","version":"63'139","reported_seq":1562,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-02-09T17:56:07.279673+0000","last_change":"2024-02-09T17:55:09.292488+0000","last_active":"2024-02-09T17:56:07.279673+0000","last_peered":"2024-02-09T17:56:07.279673+0000","last_clean":"2024-02-09T17:56:07.279673+0000","last_became_active":"2024-02-09T17:43:39.981054+0000","last_became_peered":"2024-02-09T17:43:39.981054+0000","last_unstale":"2024-02-09T17:56:07.279673+0000","last_undegraded":"2024-02-09T17:56:07.279673+0000","last_fullsized":"2024-02-09T17:56:07.279673+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:09.292403+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:55:09.292403+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"809'5380","reported_seq":5438,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-02-09T17:56:17.436344+0000","last_change":"2024-02-09T17:55:16.204365+0000","last_active":"2024-02-09T17:56:17.436344+0000","last_peered":"2024-02-09T17:56:17.436344+0000","last_clean":"2024-02-09T17:56:17.436344+0000","last_became_active":"2024-02-09T17:55:16.204001+0000","last_became_peered":"2024-02-09T17:55:16.204001+0000","last_unstale":"2024-02-09T17:56:17.436344+0000","last_undegraded":"2024-02-09T17:56:17.436344+0000","last_fullsized":"2024-02-09T17:56:17.436344+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":5380,"log_dups_size":0,"ondisk_log_size":5380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:19.806947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352583680,"num_objects":5380,"num_object_clones":0,"num_object_copies":16140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5380,"num_write_kb":344320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,9],"acting":[8,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"63'139","reported_seq":1193,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-02-09T17:56:07.272995+0000","last_change":"2024-02-09T17:54:51.201051+0000","last_active":"2024-02-09T17:56:07.272995+0000","last_peered":"2024-02-09T17:56:07.272995+0000","last_clean":"2024-02-09T17:56:07.272995+0000","last_became_active":"2024-02-09T17:40:13.501711+0000","last_became_peered":"2024-02-09T17:40:13.501711+0000","last_unstale":"2024-02-09T17:56:07.272995+0000","last_undegraded":"2024-02-09T17:56:07.272995+0000","last_fullsized":"2024-02-09T17:56:07.272995+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:54:51.200917+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:46:40.684762+0000","last_clean_scrub_stamp":"2024-02-09T17:54:51.200917+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:10.063418+0000","scrub_duration":164,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"809'5485","reported_seq":5543,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-02-09T17:56:18.128320+0000","last_change":"2024-02-09T17:55:16.445066+0000","last_active":"2024-02-09T17:56:18.128320+0000","last_peered":"2024-02-09T17:56:18.128320+0000","last_clean":"2024-02-09T17:56:18.128320+0000","last_became_active":"2024-02-09T17:55:16.444743+0000","last_became_peered":"2024-02-09T17:55:16.444743+0000","last_unstale":"2024-02-09T17:56:18.128320+0000","last_undegraded":"2024-02-09T17:56:18.128320+0000","last_fullsized":"2024-02-09T17:56:18.128320+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":5485,"log_dups_size":0,"ondisk_log_size":5485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:27.459931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":359464960,"num_objects":5485,"num_object_clones":0,"num_object_copies":16455,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5485,"num_write_kb":351040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,9],"acting":[3,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"63'139","reported_seq":729,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-02-09T17:56:07.277253+0000","last_change":"2024-02-09T17:55:13.048648+0000","last_active":"2024-02-09T17:56:07.277253+0000","last_peered":"2024-02-09T17:56:07.277253+0000","last_clean":"2024-02-09T17:56:07.277253+0000","last_became_active":"2024-02-09T17:54:32.415032+0000","last_became_peered":"2024-02-09T17:54:32.415032+0000","last_unstale":"2024-02-09T17:56:07.277253+0000","last_undegraded":"2024-02-09T17:56:07.277253+0000","last_fullsized":"2024-02-09T17:56:07.277253+0000","mapping_epoch":766,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":767,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:13.048497+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:55:13.048497+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:35.168553+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.f","version":"809'5442","reported_seq":5500,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-02-09T17:56:16.350539+0000","last_change":"2024-02-09T17:55:16.205753+0000","last_active":"2024-02-09T17:56:16.350539+0000","last_peered":"2024-02-09T17:56:16.350539+0000","last_clean":"2024-02-09T17:56:16.350539+0000","last_became_active":"2024-02-09T17:55:16.205394+0000","last_became_peered":"2024-02-09T17:55:16.205394+0000","last_unstale":"2024-02-09T17:56:16.350539+0000","last_undegraded":"2024-02-09T17:56:16.350539+0000","last_fullsized":"2024-02-09T17:56:16.350539+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":5442,"log_dups_size":0,"ondisk_log_size":5442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:24.386617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356646912,"num_objects":5442,"num_object_clones":0,"num_object_copies":16326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5442,"num_write_kb":348288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,1],"acting":[11,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"63'139","reported_seq":1496,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-02-09T17:56:07.282790+0000","last_change":"2024-02-09T17:55:11.015552+0000","last_active":"2024-02-09T17:56:07.282790+0000","last_peered":"2024-02-09T17:56:07.282790+0000","last_clean":"2024-02-09T17:56:07.282790+0000","last_became_active":"2024-02-09T17:43:14.024894+0000","last_became_peered":"2024-02-09T17:43:14.024894+0000","last_unstale":"2024-02-09T17:56:07.282790+0000","last_undegraded":"2024-02-09T17:56:07.282790+0000","last_fullsized":"2024-02-09T17:56:07.282790+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:11.015439+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:11.970338+0000","last_clean_scrub_stamp":"2024-02-09T17:55:11.015439+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:11.829031+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.e","version":"809'5514","reported_seq":5572,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-02-09T17:56:18.794664+0000","last_change":"2024-02-09T17:55:16.913062+0000","last_active":"2024-02-09T17:56:18.794664+0000","last_peered":"2024-02-09T17:56:18.794664+0000","last_clean":"2024-02-09T17:56:18.794664+0000","last_became_active":"2024-02-09T17:55:16.912769+0000","last_became_peered":"2024-02-09T17:55:16.912769+0000","last_unstale":"2024-02-09T17:56:18.794664+0000","last_undegraded":"2024-02-09T17:56:18.794664+0000","last_fullsized":"2024-02-09T17:56:18.794664+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":5514,"log_dups_size":0,"ondisk_log_size":5514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:27.866364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":361365504,"num_objects":5514,"num_object_clones":0,"num_object_copies":16542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5514,"num_write_kb":352896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"63'139","reported_seq":1120,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-02-09T17:56:07.273204+0000","last_change":"2024-02-09T17:55:05.394372+0000","last_active":"2024-02-09T17:56:07.273204+0000","last_peered":"2024-02-09T17:56:07.273204+0000","last_clean":"2024-02-09T17:56:07.273204+0000","last_became_active":"2024-02-09T17:52:32.292519+0000","last_became_peered":"2024-02-09T17:52:32.292519+0000","last_unstale":"2024-02-09T17:56:07.273204+0000","last_undegraded":"2024-02-09T17:56:07.273204+0000","last_fullsized":"2024-02-09T17:56:07.273204+0000","mapping_epoch":716,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":717,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:05.394288+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:55:05.394288+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:30.532392+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.d","version":"809'5361","reported_seq":5419,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-02-09T17:56:16.324470+0000","last_change":"2024-02-09T17:55:16.445197+0000","last_active":"2024-02-09T17:56:16.324470+0000","last_peered":"2024-02-09T17:56:16.324470+0000","last_clean":"2024-02-09T17:56:16.324470+0000","last_became_active":"2024-02-09T17:55:16.444812+0000","last_became_peered":"2024-02-09T17:55:16.444812+0000","last_unstale":"2024-02-09T17:56:16.324470+0000","last_undegraded":"2024-02-09T17:56:16.324470+0000","last_fullsized":"2024-02-09T17:56:16.324470+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":5361,"log_dups_size":0,"ondisk_log_size":5361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:43.526067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":351338496,"num_objects":5361,"num_object_clones":0,"num_object_copies":16083,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5361,"num_write_kb":343104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,2],"acting":[11,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"63'139","reported_seq":1575,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-02-09T17:56:07.276728+0000","last_change":"2024-02-09T17:55:13.976245+0000","last_active":"2024-02-09T17:56:07.276728+0000","last_peered":"2024-02-09T17:56:07.276728+0000","last_clean":"2024-02-09T17:56:07.276728+0000","last_became_active":"2024-02-09T17:33:00.284924+0000","last_became_peered":"2024-02-09T17:33:00.284924+0000","last_unstale":"2024-02-09T17:56:07.276728+0000","last_undegraded":"2024-02-09T17:56:07.276728+0000","last_fullsized":"2024-02-09T17:56:07.276728+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:13.976095+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:56.105792+0000","last_clean_scrub_stamp":"2024-02-09T17:55:13.976095+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:25.434647+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.c","version":"809'5399","reported_seq":5457,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-02-09T17:56:17.033730+0000","last_change":"2024-02-09T17:55:16.207610+0000","last_active":"2024-02-09T17:56:17.033730+0000","last_peered":"2024-02-09T17:56:17.033730+0000","last_clean":"2024-02-09T17:56:17.033730+0000","last_became_active":"2024-02-09T17:55:16.207033+0000","last_became_peered":"2024-02-09T17:55:16.207033+0000","last_unstale":"2024-02-09T17:56:17.033730+0000","last_undegraded":"2024-02-09T17:56:17.033730+0000","last_fullsized":"2024-02-09T17:56:17.033730+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":5399,"log_dups_size":0,"ondisk_log_size":5399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:18.430656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353828864,"num_objects":5399,"num_object_clones":0,"num_object_copies":16197,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5399,"num_write_kb":345536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.5","version":"63'139","reported_seq":1416,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-02-09T17:56:07.284520+0000","last_change":"2024-02-09T17:55:06.072812+0000","last_active":"2024-02-09T17:56:07.284520+0000","last_peered":"2024-02-09T17:56:07.284520+0000","last_clean":"2024-02-09T17:56:07.284520+0000","last_became_active":"2024-02-09T17:48:46.184307+0000","last_became_peered":"2024-02-09T17:48:46.184307+0000","last_unstale":"2024-02-09T17:56:07.284520+0000","last_undegraded":"2024-02-09T17:56:07.284520+0000","last_fullsized":"2024-02-09T17:56:07.284520+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:06.072671+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:55:06.072671+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:13.050346+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.3","version":"809'5312","reported_seq":5370,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-02-09T17:56:16.328316+0000","last_change":"2024-02-09T17:55:16.445395+0000","last_active":"2024-02-09T17:56:16.328316+0000","last_peered":"2024-02-09T17:56:16.328316+0000","last_clean":"2024-02-09T17:56:16.328316+0000","last_became_active":"2024-02-09T17:55:16.445048+0000","last_became_peered":"2024-02-09T17:55:16.445048+0000","last_unstale":"2024-02-09T17:56:16.328316+0000","last_undegraded":"2024-02-09T17:56:16.328316+0000","last_fullsized":"2024-02-09T17:56:16.328316+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":5312,"log_dups_size":0,"ondisk_log_size":5312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:31.073410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":348127232,"num_objects":5312,"num_object_clones":0,"num_object_copies":15936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5312,"num_write_kb":339968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"63'139","reported_seq":1497,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-02-09T17:56:07.280307+0000","last_change":"2024-02-09T17:54:58.972744+0000","last_active":"2024-02-09T17:56:07.280307+0000","last_peered":"2024-02-09T17:56:07.280307+0000","last_clean":"2024-02-09T17:56:07.280307+0000","last_became_active":"2024-02-09T17:43:14.041291+0000","last_became_peered":"2024-02-09T17:43:14.041291+0000","last_unstale":"2024-02-09T17:56:07.280307+0000","last_undegraded":"2024-02-09T17:56:07.280307+0000","last_fullsized":"2024-02-09T17:56:07.280307+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:54:58.972617+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:49:39.575571+0000","last_clean_scrub_stamp":"2024-02-09T17:54:58.972617+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:09.245500+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.2","version":"809'5560","reported_seq":5618,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-02-09T17:56:18.457231+0000","last_change":"2024-02-09T17:55:16.914203+0000","last_active":"2024-02-09T17:56:18.457231+0000","last_peered":"2024-02-09T17:56:18.457231+0000","last_clean":"2024-02-09T17:56:18.457231+0000","last_became_active":"2024-02-09T17:55:16.913786+0000","last_became_peered":"2024-02-09T17:55:16.913786+0000","last_unstale":"2024-02-09T17:56:18.457231+0000","last_undegraded":"2024-02-09T17:56:18.457231+0000","last_fullsized":"2024-02-09T17:56:18.457231+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":5560,"log_dups_size":0,"ondisk_log_size":5560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364380160,"num_objects":5560,"num_object_clones":0,"num_object_copies":16680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5560,"num_write_kb":355840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,4],"acting":[10,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.b","version":"63'139","reported_seq":865,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-02-09T17:56:07.276507+0000","last_change":"2024-02-09T17:55:10.144538+0000","last_active":"2024-02-09T17:56:07.276507+0000","last_peered":"2024-02-09T17:56:07.276507+0000","last_clean":"2024-02-09T17:56:07.276507+0000","last_became_active":"2024-02-09T17:55:08.938203+0000","last_became_peered":"2024-02-09T17:55:08.938203+0000","last_unstale":"2024-02-09T17:56:07.276507+0000","last_undegraded":"2024-02-09T17:56:07.276507+0000","last_fullsized":"2024-02-09T17:56:07.276507+0000","mapping_epoch":781,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":782,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:10.144404+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:55:10.144404+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:25.671268+0000","scrub_duration":181,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.1","version":"809'5189","reported_seq":5247,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-02-09T17:56:15.401603+0000","last_change":"2024-02-09T17:55:16.205903+0000","last_active":"2024-02-09T17:56:15.401603+0000","last_peered":"2024-02-09T17:56:15.401603+0000","last_clean":"2024-02-09T17:56:15.401603+0000","last_became_active":"2024-02-09T17:55:16.205574+0000","last_became_peered":"2024-02-09T17:55:16.205574+0000","last_unstale":"2024-02-09T17:56:15.401603+0000","last_undegraded":"2024-02-09T17:56:15.401603+0000","last_fullsized":"2024-02-09T17:56:15.401603+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":5189,"log_dups_size":0,"ondisk_log_size":5189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:24.914406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":340066304,"num_objects":5189,"num_object_clones":0,"num_object_copies":15567,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5189,"num_write_kb":332096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,2],"acting":[1,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"63'139","reported_seq":1147,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-02-09T17:56:07.268837+0000","last_change":"2024-02-09T17:55:10.313343+0000","last_active":"2024-02-09T17:56:07.268837+0000","last_peered":"2024-02-09T17:56:07.268837+0000","last_clean":"2024-02-09T17:56:07.268837+0000","last_became_active":"2024-02-09T17:46:47.013853+0000","last_became_peered":"2024-02-09T17:46:47.013853+0000","last_unstale":"2024-02-09T17:56:07.268837+0000","last_undegraded":"2024-02-09T17:56:07.268837+0000","last_fullsized":"2024-02-09T17:56:07.268837+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:10.313268+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:55:10.313268+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:24.198160+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"809'5582","reported_seq":5640,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-02-09T17:56:18.076637+0000","last_change":"2024-02-09T17:55:16.444927+0000","last_active":"2024-02-09T17:56:18.076637+0000","last_peered":"2024-02-09T17:56:18.076637+0000","last_clean":"2024-02-09T17:56:18.076637+0000","last_became_active":"2024-02-09T17:55:16.444683+0000","last_became_peered":"2024-02-09T17:55:16.444683+0000","last_unstale":"2024-02-09T17:56:18.076637+0000","last_undegraded":"2024-02-09T17:56:18.076637+0000","last_fullsized":"2024-02-09T17:56:18.076637+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":5582,"log_dups_size":0,"ondisk_log_size":5582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:37.733967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365821952,"num_objects":5582,"num_object_clones":0,"num_object_copies":16746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5582,"num_write_kb":357248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,9],"acting":[3,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"63'139","reported_seq":2002,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-02-09T17:56:07.285332+0000","last_change":"2024-02-09T17:55:13.823711+0000","last_active":"2024-02-09T17:56:07.285332+0000","last_peered":"2024-02-09T17:56:07.285332+0000","last_clean":"2024-02-09T17:56:07.285332+0000","last_became_active":"2024-02-09T17:43:28.590237+0000","last_became_peered":"2024-02-09T17:43:28.590237+0000","last_unstale":"2024-02-09T17:56:07.285332+0000","last_undegraded":"2024-02-09T17:56:07.285332+0000","last_fullsized":"2024-02-09T17:56:07.285332+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:13.823598+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:45:47.710252+0000","last_clean_scrub_stamp":"2024-02-09T17:55:13.823598+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:41.832421+0000","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.7","version":"809'5282","reported_seq":5340,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-02-09T17:56:15.393861+0000","last_change":"2024-02-09T17:55:16.206804+0000","last_active":"2024-02-09T17:56:15.393861+0000","last_peered":"2024-02-09T17:56:15.393861+0000","last_clean":"2024-02-09T17:56:15.393861+0000","last_became_active":"2024-02-09T17:55:16.206561+0000","last_became_peered":"2024-02-09T17:55:16.206561+0000","last_unstale":"2024-02-09T17:56:15.393861+0000","last_undegraded":"2024-02-09T17:56:15.393861+0000","last_fullsized":"2024-02-09T17:56:15.393861+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":5282,"log_dups_size":0,"ondisk_log_size":5282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:41.254300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":346161152,"num_objects":5282,"num_object_clones":0,"num_object_copies":15846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5282,"num_write_kb":338048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.e","version":"63'139","reported_seq":1459,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-02-09T17:56:07.282868+0000","last_change":"2024-02-09T17:55:08.044579+0000","last_active":"2024-02-09T17:56:07.282868+0000","last_peered":"2024-02-09T17:56:07.282868+0000","last_clean":"2024-02-09T17:56:07.282868+0000","last_became_active":"2024-02-09T17:43:14.036583+0000","last_became_peered":"2024-02-09T17:43:14.036583+0000","last_unstale":"2024-02-09T17:56:07.282868+0000","last_undegraded":"2024-02-09T17:56:07.282868+0000","last_fullsized":"2024-02-09T17:56:07.282868+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:08.044477+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:14.981453+0000","last_clean_scrub_stamp":"2024-02-09T17:55:08.044477+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:28.800003+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"809'5423","reported_seq":5481,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-02-09T17:56:16.308842+0000","last_change":"2024-02-09T17:55:16.446161+0000","last_active":"2024-02-09T17:56:16.308842+0000","last_peered":"2024-02-09T17:56:16.308842+0000","last_clean":"2024-02-09T17:56:16.308842+0000","last_became_active":"2024-02-09T17:55:16.445702+0000","last_became_peered":"2024-02-09T17:55:16.445702+0000","last_unstale":"2024-02-09T17:56:16.308842+0000","last_undegraded":"2024-02-09T17:56:16.308842+0000","last_fullsized":"2024-02-09T17:56:16.308842+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":5423,"log_dups_size":0,"ondisk_log_size":5423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:18.722073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":355401728,"num_objects":5423,"num_object_clones":0,"num_object_copies":16269,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5423,"num_write_kb":347072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,7],"acting":[11,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.f","version":"63'139","reported_seq":1519,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-02-09T17:56:07.564417+0000","last_change":"2024-02-09T17:55:08.355359+0000","last_active":"2024-02-09T17:56:07.564417+0000","last_peered":"2024-02-09T17:56:07.564417+0000","last_clean":"2024-02-09T17:56:07.564417+0000","last_became_active":"2024-02-09T17:33:05.316100+0000","last_became_peered":"2024-02-09T17:33:05.316100+0000","last_unstale":"2024-02-09T17:56:07.564417+0000","last_undegraded":"2024-02-09T17:56:07.564417+0000","last_fullsized":"2024-02-09T17:56:07.564417+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:08.355221+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:48:34.651647+0000","last_clean_scrub_stamp":"2024-02-09T17:55:08.355221+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:11.673721+0000","scrub_duration":144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.5","version":"809'5558","reported_seq":5614,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-02-09T17:56:18.497128+0000","last_change":"2024-02-09T17:55:16.913593+0000","last_active":"2024-02-09T17:56:18.497128+0000","last_peered":"2024-02-09T17:56:18.497128+0000","last_clean":"2024-02-09T17:56:18.497128+0000","last_became_active":"2024-02-09T17:55:16.913201+0000","last_became_peered":"2024-02-09T17:55:16.913201+0000","last_unstale":"2024-02-09T17:56:18.497128+0000","last_undegraded":"2024-02-09T17:56:18.497128+0000","last_fullsized":"2024-02-09T17:56:18.497128+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:26.249705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364249088,"num_objects":5558,"num_object_clones":0,"num_object_copies":16674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5558,"num_write_kb":355712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,1],"acting":[10,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.c","version":"63'139","reported_seq":1646,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-02-09T17:56:07.268900+0000","last_change":"2024-02-09T17:55:07.304327+0000","last_active":"2024-02-09T17:56:07.268900+0000","last_peered":"2024-02-09T17:56:07.268900+0000","last_clean":"2024-02-09T17:56:07.268900+0000","last_became_active":"2024-02-09T17:43:14.027715+0000","last_became_peered":"2024-02-09T17:43:14.027715+0000","last_unstale":"2024-02-09T17:56:07.268900+0000","last_undegraded":"2024-02-09T17:56:07.268900+0000","last_fullsized":"2024-02-09T17:56:07.268900+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:07.304236+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:14.135321+0000","last_clean_scrub_stamp":"2024-02-09T17:55:07.304236+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:16.017635+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"809'5319","reported_seq":5377,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-02-09T17:56:16.973721+0000","last_change":"2024-02-09T17:55:16.445202+0000","last_active":"2024-02-09T17:56:16.973721+0000","last_peered":"2024-02-09T17:56:16.973721+0000","last_clean":"2024-02-09T17:56:16.973721+0000","last_became_active":"2024-02-09T17:55:16.444827+0000","last_became_peered":"2024-02-09T17:55:16.444827+0000","last_unstale":"2024-02-09T17:56:16.973721+0000","last_undegraded":"2024-02-09T17:56:16.973721+0000","last_fullsized":"2024-02-09T17:56:16.973721+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":5319,"log_dups_size":0,"ondisk_log_size":5319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:20.836494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":348585984,"num_objects":5319,"num_object_clones":0,"num_object_copies":15957,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5319,"num_write_kb":340416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.d","version":"63'139","reported_seq":1051,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-02-09T17:56:07.285512+0000","last_change":"2024-02-09T17:55:08.898009+0000","last_active":"2024-02-09T17:56:07.285512+0000","last_peered":"2024-02-09T17:56:07.285512+0000","last_clean":"2024-02-09T17:56:07.285512+0000","last_became_active":"2024-02-09T17:43:14.036759+0000","last_became_peered":"2024-02-09T17:43:14.036759+0000","last_unstale":"2024-02-09T17:56:07.285512+0000","last_undegraded":"2024-02-09T17:56:07.285512+0000","last_fullsized":"2024-02-09T17:56:07.285512+0000","mapping_epoch":454,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:08.897882+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:55:08.897882+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:15.119150+0000","scrub_duration":145,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"63'139","reported_seq":1223,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-02-09T17:56:07.285204+0000","last_change":"2024-02-09T17:55:10.810070+0000","last_active":"2024-02-09T17:56:07.285204+0000","last_peered":"2024-02-09T17:56:07.285204+0000","last_clean":"2024-02-09T17:56:07.285204+0000","last_became_active":"2024-02-09T17:46:47.013920+0000","last_became_peered":"2024-02-09T17:46:47.013920+0000","last_unstale":"2024-02-09T17:56:07.285204+0000","last_undegraded":"2024-02-09T17:56:07.285204+0000","last_fullsized":"2024-02-09T17:56:07.285204+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:10.809935+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:48:35.170638+0000","last_clean_scrub_stamp":"2024-02-09T17:55:10.809935+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:31.681608+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"63'139","reported_seq":1492,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-02-09T17:56:07.277999+0000","last_change":"2024-02-09T17:55:21.232444+0000","last_active":"2024-02-09T17:56:07.277999+0000","last_peered":"2024-02-09T17:56:07.277999+0000","last_clean":"2024-02-09T17:56:07.277999+0000","last_became_active":"2024-02-09T17:55:21.231902+0000","last_became_peered":"2024-02-09T17:55:21.231902+0000","last_unstale":"2024-02-09T17:56:07.277999+0000","last_undegraded":"2024-02-09T17:56:07.277999+0000","last_fullsized":"2024-02-09T17:56:07.277999+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":791,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:01.456654+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:44:51.190789+0000","last_clean_scrub_stamp":"2024-02-09T17:55:01.456654+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:22.186440+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.12","version":"63'139","reported_seq":1576,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-02-09T17:56:07.272980+0000","last_change":"2024-02-09T17:56:02.485340+0000","last_active":"2024-02-09T17:56:07.272980+0000","last_peered":"2024-02-09T17:56:07.272980+0000","last_clean":"2024-02-09T17:56:07.272980+0000","last_became_active":"2024-02-09T17:46:47.015622+0000","last_became_peered":"2024-02-09T17:46:47.015622+0000","last_unstale":"2024-02-09T17:56:07.272980+0000","last_undegraded":"2024-02-09T17:56:07.272980+0000","last_fullsized":"2024-02-09T17:56:07.272980+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:56:02.485208+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:56:02.485208+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:57:04.696191+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.13","version":"63'139","reported_seq":1187,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-02-09T17:56:07.273155+0000","last_change":"2024-02-09T17:54:54.282606+0000","last_active":"2024-02-09T17:56:07.273155+0000","last_peered":"2024-02-09T17:56:07.273155+0000","last_clean":"2024-02-09T17:56:07.273155+0000","last_became_active":"2024-02-09T17:41:14.521672+0000","last_became_peered":"2024-02-09T17:41:14.521672+0000","last_unstale":"2024-02-09T17:56:07.273155+0000","last_undegraded":"2024-02-09T17:56:07.273155+0000","last_fullsized":"2024-02-09T17:56:07.273155+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:54:54.282472+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:54:54.282472+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:56:19.415044+0000","scrub_duration":214,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":5680791552,"num_objects":86682,"num_object_clones":0,"num_object_copies":260046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86682,"num_write_kb":5547648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16919101440,"data_stored":16919101440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86682,"ondisk_log_size":86682,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547057048,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2531912,"kb_used_data":1702464,"kb_used_omap":1,"kb_used_meta":829438,"kb_avail":91213240,"statfs":{"total":95995035648,"available":93402357760,"internally_reserved":0,"allocated":1743323136,"data_stored":1742907862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":849344964},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:56:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":2.306,"15min":1.518},"min":{"1min":0.28999999999999998,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":4.6950000000000003,"5min":20.003,"15min":20.003},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.95499999999999996,"5min":1.821,"15min":1.3089999999999999},"min":{"1min":0.29199999999999998,"5min":0.248,"15min":0.23100000000000001},"max":{"1min":3.1789999999999998,"5min":9.6259999999999994,"15min":9.6259999999999994},"last":0.34799999999999998}]},{"osd":1,"last update":"Fri Feb 9 17:56:01 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":1.762,"15min":1.484},"min":{"1min":0.34100000000000003,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":2.4119999999999999,"5min":9.532,"15min":11.576000000000001},"last":0.309},{"interface":"front","average":{"1min":1.407,"5min":1.8420000000000001,"15min":1.6120000000000001},"min":{"1min":0.33000000000000002,"5min":0.25,"15min":0.25},"max":{"1min":5.0410000000000004,"5min":14.287000000000001,"15min":14.287000000000001},"last":2.4249999999999998}]},{"osd":2,"last update":"Fri Feb 9 17:56:05 2024","interfaces":[{"interface":"back","average":{"1min":1.486,"5min":1.486,"15min":1.486},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":4.9989999999999997,"5min":4.9989999999999997,"15min":4.9989999999999997},"last":3.8159999999999998},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":5.0789999999999997,"5min":5.0789999999999997,"15min":5.0789999999999997},"last":2.3690000000000002}]},{"osd":3,"last update":"Fri Feb 9 17:56:01 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":2.5,"15min":1.8740000000000001},"min":{"1min":0.47999999999999998,"5min":0.32200000000000001,"15min":0.248},"max":{"1min":5.1509999999999998,"5min":43.344000000000001,"15min":43.344000000000001},"last":4.5599999999999996},{"interface":"front","average":{"1min":1.556,"5min":2.5379999999999998,"15min":1.954},"min":{"1min":0.45300000000000001,"5min":0.375,"15min":0.33600000000000002},"max":{"1min":5.1769999999999996,"5min":43.378,"15min":43.378},"last":4.524}]},{"osd":4,"last update":"Fri Feb 9 17:54:59 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":1.746,"15min":1.762},"min":{"1min":0.254,"5min":0.254,"15min":0.246},"max":{"1min":4.7460000000000004,"5min":19.956,"15min":65.109999999999999},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.89000000000000001,"5min":1.496,"15min":1.7110000000000001},"min":{"1min":0.309,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":3.3490000000000002,"5min":9.5690000000000008,"15min":65.099000000000004},"last":0.50700000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:56:01 2024","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.601,"15min":1.5980000000000001},"min":{"1min":0.34899999999999998,"5min":0.29699999999999999,"15min":0.252},"max":{"1min":4.6139999999999999,"5min":11.119999999999999,"15min":18.878},"last":0.44},{"interface":"front","average":{"1min":1.5369999999999999,"5min":1.762,"15min":1.6160000000000001},"min":{"1min":0.41999999999999998,"5min":0.313,"15min":0.28999999999999998},"max":{"1min":5.069,"5min":20.038,"15min":20.038},"last":2.4079999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:56:01 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":1.6439999999999999,"15min":1.629},"min":{"1min":0.40899999999999997,"5min":0.315,"15min":0.30399999999999999},"max":{"1min":2.3769999999999998,"5min":9.5980000000000008,"15min":12.362},"last":0.53800000000000003},{"interface":"front","average":{"1min":1.5669999999999999,"5min":1.659,"15min":1.7070000000000001},"min":{"1min":0.45800000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":5.125,"5min":15.409000000000001,"15min":15.409000000000001},"last":2.387}]},{"osd":7,"last update":"Fri Feb 9 17:56:01 2024","interfaces":[{"interface":"back","average":{"1min":1.587,"5min":1.635,"15min":1.696},"min":{"1min":0.39000000000000001,"5min":0.33000000000000002,"15min":0.317},"max":{"1min":5.1269999999999998,"5min":15.51,"15min":18.846},"last":2.4460000000000002},{"interface":"front","average":{"1min":1.603,"5min":2.2869999999999999,"15min":1.897},"min":{"1min":0.44,"5min":0.35699999999999998,"15min":0.318},"max":{"1min":8.0440000000000005,"5min":24.373999999999999,"15min":24.373999999999999},"last":0.87}]},{"osd":8,"last update":"Fri Feb 9 17:56:13 2024","interfaces":[{"interface":"back","average":{"1min":1.6659999999999999,"5min":2.2719999999999998,"15min":1.4930000000000001},"min":{"1min":0.379,"5min":0.36599999999999999,"15min":0.27900000000000003},"max":{"1min":11.944000000000001,"5min":13.574,"15min":13.574},"last":8.9090000000000007},{"interface":"front","average":{"1min":1.8740000000000001,"5min":2.3239999999999998,"15min":1.5820000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.32700000000000001},"max":{"1min":11.922000000000001,"5min":13.606,"15min":13.606},"last":0.51200000000000001}]},{"osd":9,"last update":"Fri Feb 9 17:55:45 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":2.4510000000000001,"15min":1.6259999999999999},"min":{"1min":0.47099999999999997,"5min":0.32900000000000001,"15min":0.29399999999999998},"max":{"1min":5.2960000000000003,"5min":22.613,"15min":22.613},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.76900000000000002,"5min":1.845,"15min":1.3740000000000001},"min":{"1min":0.49099999999999999,"5min":0.38300000000000001,"15min":0.34399999999999997},"max":{"1min":1.488,"5min":12.289,"15min":12.313000000000001},"last":8.9849999999999994}]},{"osd":10,"last update":"Fri Feb 9 17:56:01 2024","interfaces":[{"interface":"back","average":{"1min":1.706,"5min":2.641,"15min":1.718},"min":{"1min":0.54900000000000004,"5min":0.41199999999999998,"15min":0.32700000000000001},"max":{"1min":6.2359999999999998,"5min":17.373000000000001,"15min":17.373000000000001},"last":0.77000000000000002},{"interface":"front","average":{"1min":2.2559999999999998,"5min":2.105,"15min":1.5940000000000001},"min":{"1min":0.75,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":5.2370000000000001,"5min":20.233000000000001,"15min":20.233000000000001},"last":2.3530000000000002}]}]},{"osd":10,"up_from":84,"seq":360777253280,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2778440,"kb_used_data":2093376,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":90966712,"statfs":{"total":95995035648,"available":93149913088,"internally_reserved":0,"allocated":2143617024,"data_stored":2143201750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":701495744},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:55:46 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":1.544,"15min":1.1240000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.20599999999999999},"max":{"1min":5.5609999999999999,"5min":10.532,"15min":10.532},"last":2.7370000000000001},{"interface":"front","average":{"1min":0.56100000000000005,"5min":1.601,"15min":1.1399999999999999},"min":{"1min":0.29199999999999998,"5min":0.26400000000000001,"15min":0.248},"max":{"1min":1.3109999999999999,"5min":10.573,"15min":10.573},"last":0.33000000000000002}]},{"osd":1,"last update":"Fri Feb 9 17:56:16 2024","interfaces":[{"interface":"back","average":{"1min":1.5660000000000001,"5min":1.5660000000000001,"15min":1.5660000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":7.1360000000000001,"5min":7.1360000000000001,"15min":7.1360000000000001},"last":7.1360000000000001},{"interface":"front","average":{"1min":1.5680000000000001,"5min":1.5680000000000001,"15min":1.5680000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":7.1050000000000004,"5min":7.1050000000000004,"15min":7.1050000000000004},"last":7.1050000000000004}]},{"osd":2,"last update":"Fri Feb 9 17:55:20 2024","interfaces":[{"interface":"back","average":{"1min":1.8400000000000001,"5min":1.96,"15min":1.5620000000000001},"min":{"1min":0.27800000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":7.399,"5min":13.215999999999999,"15min":13.215999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":1.661,"5min":2.101,"15min":1.6870000000000001},"min":{"1min":0.46500000000000002,"5min":0.371,"15min":0.25600000000000001},"max":{"1min":5.8239999999999998,"5min":13.259,"15min":38.868000000000002},"last":0.28899999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:55:09 2024","interfaces":[{"interface":"back","average":{"1min":2.282,"5min":2.1549999999999998,"15min":1.796},"min":{"1min":0.29999999999999999,"5min":0.27700000000000002,"15min":0.26100000000000001},"max":{"1min":6.9909999999999997,"5min":14.689,"15min":35.113},"last":0.65700000000000003},{"interface":"front","average":{"1min":1.296,"5min":2.4550000000000001,"15min":2.077},"min":{"1min":0.27800000000000002,"5min":0.249,"15min":0.249},"max":{"1min":3.5449999999999999,"5min":11.209,"15min":34.359000000000002},"last":0.54400000000000004}]},{"osd":4,"last update":"Fri Feb 9 17:55:42 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":1.1220000000000001,"15min":1.181},"min":{"1min":0.43099999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":5.4509999999999996,"5min":11.705,"15min":11.705},"last":0.35699999999999998},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.2669999999999999,"15min":1.2549999999999999},"min":{"1min":0.36499999999999999,"5min":0.33900000000000002,"15min":0.27000000000000002},"max":{"1min":5.5039999999999996,"5min":11.744,"15min":11.744},"last":0.379}]},{"osd":5,"last update":"Fri Feb 9 17:55:20 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":1.292,"15min":1.2929999999999999},"min":{"1min":0.33200000000000002,"5min":0.30099999999999999,"15min":0.29099999999999998},"max":{"1min":0.82899999999999996,"5min":10.759,"15min":10.759},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.97499999999999998,"5min":1.4039999999999999,"15min":1.504},"min":{"1min":0.46700000000000003,"5min":0.373,"15min":0.33500000000000002},"max":{"1min":5.5389999999999997,"5min":11.779,"15min":11.779},"last":0.40600000000000003}]},{"osd":6,"last update":"Fri Feb 9 17:55:24 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":1.272,"15min":1.4259999999999999},"min":{"1min":0.34599999999999997,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.77000000000000002,"5min":10.776999999999999,"15min":10.962},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.5780000000000001,"15min":1.5129999999999999},"min":{"1min":0.434,"5min":0.35099999999999998,"15min":0.32600000000000001},"max":{"1min":5.5910000000000002,"5min":12.414,"15min":12.414},"last":0.432}]},{"osd":7,"last update":"Fri Feb 9 17:56:16 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":1.5509999999999999,"15min":2.8370000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":3.7320000000000002,"5min":11.288,"15min":11.288},"last":0.88},{"interface":"front","average":{"1min":0.95799999999999996,"5min":1.583,"15min":2.863},"min":{"1min":0.442,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":3.762,"5min":10.736000000000001,"15min":10.736000000000001},"last":0.91100000000000003}]},{"osd":8,"last update":"Fri Feb 9 17:56:11 2024","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.798,"15min":1.222},"min":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.31},"max":{"1min":3.5409999999999999,"5min":10.792999999999999,"15min":10.792999999999999},"last":7.0609999999999999},{"interface":"front","average":{"1min":1.111,"5min":1.843,"15min":1.359},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.30599999999999999},"max":{"1min":4.7300000000000004,"5min":12.249000000000001,"15min":12.249000000000001},"last":0.69799999999999995}]},{"osd":9,"last update":"Fri Feb 9 17:55:20 2024","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":2.3220000000000001,"15min":1.419},"min":{"1min":0.373,"5min":0.34300000000000003,"15min":0.33300000000000002},"max":{"1min":4.2350000000000003,"5min":13.298,"15min":13.298},"last":1.387},{"interface":"front","average":{"1min":1.1140000000000001,"5min":2.206,"15min":1.5329999999999999},"min":{"1min":0.44,"5min":0.39600000000000002,"15min":0.36099999999999999},"max":{"1min":4.7080000000000002,"5min":10.473000000000001,"15min":10.622999999999999},"last":1.3049999999999999}]},{"osd":11,"last update":"Fri Feb 9 17:55:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":1.8240000000000001,"15min":1.2350000000000001},"min":{"1min":0.44700000000000001,"5min":0.36799999999999999,"15min":0.35399999999999998},"max":{"1min":1.135,"5min":23.579000000000001,"15min":23.579000000000001},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.93899999999999995,"5min":1.883,"15min":1.2949999999999999},"min":{"1min":0.58399999999999996,"5min":0.45000000000000001,"15min":0.438},"max":{"1min":2.2120000000000002,"5min":23.619,"15min":23.619},"last":0.45400000000000001}]}]},{"osd":9,"up_from":77,"seq":330712482211,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2854088,"kb_used_data":2060736,"kb_used_omap":1,"kb_used_meta":793342,"kb_avail":90891064,"statfs":{"total":95995035648,"available":93072449536,"internally_reserved":0,"allocated":2110193664,"data_stored":2109778390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":812382656},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:55:17 2024","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.8919999999999999,"15min":1.3340000000000001},"min":{"1min":0.27900000000000003,"5min":0.23899999999999999,"15min":0.215},"max":{"1min":9.718,"5min":11.539999999999999,"15min":11.539999999999999},"last":1.5209999999999999},{"interface":"front","average":{"1min":0.91200000000000003,"5min":1.454,"15min":1.359},"min":{"1min":0.28999999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.5640000000000001,"5min":10.566000000000001,"15min":11.596},"last":0.57999999999999996}]},{"osd":1,"last update":"Fri Feb 9 17:56:05 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":2.2509999999999999,"15min":1.621},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":2.8319999999999999,"5min":61.031999999999996,"15min":61.031999999999996},"last":1.1579999999999999},{"interface":"front","average":{"1min":0.84199999999999997,"5min":2.7509999999999999,"15min":1.7010000000000001},"min":{"1min":0.41599999999999998,"5min":0.28699999999999998,"15min":0.25},"max":{"1min":2.278,"5min":61.970999999999997,"15min":61.970999999999997},"last":0.496}]},{"osd":2,"last update":"Fri Feb 9 17:55:35 2024","interfaces":[{"interface":"back","average":{"1min":1.4730000000000001,"5min":1.994,"15min":1.8380000000000001},"min":{"1min":0.32600000000000001,"5min":0.27600000000000002,"15min":0.26900000000000002},"max":{"1min":9.8689999999999998,"5min":10.657999999999999,"15min":10.657999999999999},"last":1.575},{"interface":"front","average":{"1min":1.292,"5min":1.74,"15min":1.6890000000000001},"min":{"1min":0.32100000000000001,"5min":0.248,"15min":0.248},"max":{"1min":8.0389999999999997,"5min":11.35,"15min":11.35},"last":2.508}]},{"osd":3,"last update":"Fri Feb 9 17:55:31 2024","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":2.1320000000000001,"15min":1.496},"min":{"1min":0.40899999999999997,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":10.064,"5min":10.537000000000001,"15min":10.537000000000001},"last":0.371},{"interface":"front","average":{"1min":1.581,"5min":2.2090000000000001,"15min":1.149},"min":{"1min":0.377,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":10.093,"5min":10.388,"15min":10.388},"last":0.46600000000000003}]},{"osd":4,"last update":"Fri Feb 9 17:56:05 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":1.8899999999999999,"15min":1.552},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.26800000000000002},"max":{"1min":2.1890000000000001,"5min":19.414000000000001,"15min":19.414000000000001},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.77300000000000002,"5min":1.5680000000000001,"15min":1.2989999999999999},"min":{"1min":0.309,"5min":0.30299999999999999,"15min":0.26900000000000002},"max":{"1min":2.7810000000000001,"5min":19.224,"15min":19.224},"last":1.1100000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:56:15 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":1.4910000000000001,"15min":1.6990000000000001},"min":{"1min":0.436,"5min":0.35399999999999998,"15min":0.26900000000000002},"max":{"1min":2.2189999999999999,"5min":9.7729999999999997,"15min":77.924000000000007},"last":0.52000000000000002},{"interface":"front","average":{"1min":1.046,"5min":1.288,"15min":1.6779999999999999},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.28000000000000003},"max":{"1min":5.0259999999999998,"5min":10.643000000000001,"15min":77.899000000000001},"last":0.69199999999999995}]},{"osd":6,"last update":"Fri Feb 9 17:56:05 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":1.518,"15min":1.5129999999999999},"min":{"1min":0.36899999999999999,"5min":0.33400000000000002,"15min":0.30199999999999999},"max":{"1min":3.0019999999999998,"5min":19.375,"15min":19.375},"last":0.96499999999999997},{"interface":"front","average":{"1min":0.92000000000000004,"5min":1.532,"15min":1.698},"min":{"1min":0.42399999999999999,"5min":0.34999999999999998,"15min":0.33500000000000002},"max":{"1min":2.9470000000000001,"5min":19.178000000000001,"15min":19.178000000000001},"last":0.66600000000000004}]},{"osd":7,"last update":"Fri Feb 9 17:49:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.9670000000000001,"15min":1.8400000000000001},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.33400000000000002},"max":{"1min":3.7909999999999999,"5min":29.001999999999999,"15min":29.001999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":1.127,"5min":1.919,"15min":1.841},"min":{"1min":0.39400000000000002,"5min":0.373,"15min":0.32200000000000001},"max":{"1min":3.7749999999999999,"5min":28.872,"15min":28.872},"last":0.47399999999999998}]},{"osd":8,"last update":"Fri Feb 9 17:55:17 2024","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.677,"15min":1.1879999999999999},"min":{"1min":0.38600000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":9.4930000000000003,"5min":10.744,"15min":10.744},"last":0.753},{"interface":"front","average":{"1min":1.46,"5min":1.7430000000000001,"15min":1.3799999999999999},"min":{"1min":0.52500000000000002,"5min":0.35499999999999998,"15min":0.32200000000000001},"max":{"1min":9.5289999999999999,"5min":10.768000000000001,"15min":12.885999999999999},"last":0.78100000000000003}]},{"osd":10,"last update":"Fri Feb 9 17:55:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":2.3279999999999998,"15min":1.675},"min":{"1min":0.50800000000000001,"5min":0.372,"15min":0.312},"max":{"1min":9.8109999999999999,"5min":20.773,"15min":20.773},"last":1.1890000000000001},{"interface":"front","average":{"1min":1.284,"5min":1.78,"15min":1.2889999999999999},"min":{"1min":0.39000000000000001,"5min":0.34699999999999998,"15min":0.34300000000000003},"max":{"1min":9.5879999999999992,"5min":20.637,"15min":20.637},"last":0.85699999999999998}]},{"osd":11,"last update":"Fri Feb 9 17:55:31 2024","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":2.1190000000000002,"15min":1.4870000000000001},"min":{"1min":0.47799999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":9.9789999999999992,"5min":11.324,"15min":11.324},"last":0.54200000000000004},{"interface":"front","average":{"1min":1.524,"5min":1.7829999999999999,"15min":1.331},"min":{"1min":0.58799999999999997,"5min":0.39000000000000001,"15min":0.36499999999999999},"max":{"1min":6.101,"5min":11.388,"15min":11.388},"last":0.63700000000000001}]}]},{"osd":8,"up_from":171,"seq":734439407954,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1488264,"kb_used_data":1037504,"kb_used_omap":212,"kb_used_meta":450539,"kb_avail":92256888,"statfs":{"total":95995035648,"available":94471053312,"internally_reserved":0,"allocated":1062404096,"data_stored":1061988822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217579,"internal_metadata":461352469},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:56:14 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":1.879,"15min":1.2390000000000001},"min":{"1min":0.26700000000000002,"5min":0.254,"15min":0.217},"max":{"1min":2.3069999999999999,"5min":10.832000000000001,"15min":12.712999999999999},"last":3.113},{"interface":"front","average":{"1min":1.0269999999999999,"5min":1.4930000000000001,"15min":1.3540000000000001},"min":{"1min":0.30599999999999999,"5min":0.23999999999999999,"15min":0.23100000000000001},"max":{"1min":5.5899999999999999,"5min":9.25,"15min":12.725},"last":0.23699999999999999}]},{"osd":1,"last update":"Fri Feb 9 17:56:14 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":1.8700000000000001,"15min":1.3859999999999999},"min":{"1min":0.32300000000000001,"5min":0.23799999999999999,"15min":0.22700000000000001},"max":{"1min":3.9399999999999999,"5min":8.2550000000000008,"15min":8.718},"last":3.1899999999999999},{"interface":"front","average":{"1min":1.4330000000000001,"5min":2.1499999999999999,"15min":1.542},"min":{"1min":0.36399999999999999,"5min":0.27100000000000002,"15min":0.24099999999999999},"max":{"1min":5.7249999999999996,"5min":21.399000000000001,"15min":21.399000000000001},"last":0.27200000000000002}]},{"osd":2,"last update":"Fri Feb 9 17:56:15 2024","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.099,"15min":1.099},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":3.3730000000000002,"5min":3.3730000000000002,"15min":3.3730000000000002},"last":3.2210000000000001},{"interface":"front","average":{"1min":1.7509999999999999,"5min":1.7509999999999999,"15min":1.7509999999999999},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":5.7350000000000003,"5min":5.7350000000000003,"15min":5.7350000000000003},"last":0.29999999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:55:19 2024","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":2.8889999999999998,"15min":2.9159999999999999},"min":{"1min":0.32400000000000001,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":4.5490000000000004,"5min":15.654,"15min":15.654},"last":3.3610000000000002},{"interface":"front","average":{"1min":2.0790000000000002,"5min":2.427,"15min":2.0529999999999999},"min":{"1min":0.378,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":14.35,"5min":15.609999999999999,"15min":15.609999999999999},"last":3.3839999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:56:14 2024","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.206,"15min":1.206},"min":{"1min":0.38500000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":5.5449999999999999,"5min":8.7959999999999994,"15min":8.7959999999999994},"last":0.35899999999999999},{"interface":"front","average":{"1min":1.175,"5min":1.1200000000000001,"15min":1.3839999999999999},"min":{"1min":0.40300000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":5.6890000000000001,"5min":8.8780000000000001,"15min":9.6140000000000008},"last":0.32600000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:55:36 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":1.6739999999999999,"15min":1.6879999999999999},"min":{"1min":0.28399999999999997,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":2.625,"5min":9.657,"15min":9.657},"last":3.2829999999999999},{"interface":"front","average":{"1min":0.71999999999999997,"5min":1.2190000000000001,"15min":1.47},"min":{"1min":0.40100000000000002,"5min":0.30299999999999999,"15min":0.29899999999999999},"max":{"1min":1.887,"5min":9.375,"15min":9.375},"last":0.39000000000000001}]},{"osd":6,"last update":"Fri Feb 9 17:56:14 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":1.74,"15min":1.577},"min":{"1min":0.36599999999999999,"5min":0.313,"15min":0.29899999999999999},"max":{"1min":2.4049999999999998,"5min":9.7539999999999996,"15min":13.031000000000001},"last":3.3079999999999998},{"interface":"front","average":{"1min":0.871,"5min":1.5069999999999999,"15min":1.4850000000000001},"min":{"1min":0.35499999999999998,"5min":0.311,"15min":0.311},"max":{"1min":2.2669999999999999,"5min":10.178000000000001,"15min":12.157999999999999},"last":3.2490000000000001}]},{"osd":7,"last update":"Fri Feb 9 17:56:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.7,"15min":1.6779999999999999},"min":{"1min":0.50800000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":5.7069999999999999,"5min":9.3130000000000006,"15min":11.278},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.90800000000000003,"5min":1.373,"15min":1.6359999999999999},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":2.3570000000000002,"5min":9.7110000000000003,"15min":11.313000000000001},"last":3.407}]},{"osd":9,"last update":"Fri Feb 9 17:56:14 2024","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":2.23,"15min":1.589},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.26800000000000002},"max":{"1min":3.0920000000000001,"5min":16.358000000000001,"15min":45.616},"last":3.331},{"interface":"front","average":{"1min":1.236,"5min":2.3170000000000002,"15min":1.6240000000000001},"min":{"1min":0.40600000000000003,"5min":0.376,"15min":0.314},"max":{"1min":3.1280000000000001,"5min":16.422999999999998,"15min":45.718000000000004},"last":3.431}]},{"osd":10,"last update":"Fri Feb 9 17:56:14 2024","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.7490000000000001,"15min":1.212},"min":{"1min":0.50700000000000001,"5min":0.34599999999999997,"15min":0.29699999999999999},"max":{"1min":4.3250000000000002,"5min":9.157,"15min":9.157},"last":3.9590000000000001},{"interface":"front","average":{"1min":1.4490000000000001,"5min":2.2690000000000001,"15min":1.47},"min":{"1min":0.41799999999999998,"5min":0.39400000000000002,"15min":0.34300000000000003},"max":{"1min":5.6820000000000004,"5min":13.096,"15min":13.096},"last":3.1640000000000001}]},{"osd":11,"last update":"Fri Feb 9 17:55:09 2024","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":2.262,"15min":1.5},"min":{"1min":0.46000000000000002,"5min":0.36099999999999999,"15min":0.34000000000000002},"max":{"1min":5.3029999999999999,"5min":16.466999999999999,"15min":16.466999999999999},"last":0.68799999999999994},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.696,"15min":1.357},"min":{"1min":0.39700000000000002,"5min":0.33900000000000002,"15min":0.308},"max":{"1min":5.2850000000000001,"5min":20.831,"15min":20.831},"last":0.437}]}]},{"osd":7,"up_from":57,"seq":244813136299,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1428496,"kb_used_data":685128,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":92316656,"statfs":{"total":95995035648,"available":94532255744,"internally_reserved":0,"allocated":701571072,"data_stored":701152246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":761199050},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:55:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":2.0499999999999998,"15min":1.6799999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.249},"max":{"1min":3.3650000000000002,"5min":101.167,"15min":101.167},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.71199999999999997,"5min":2.024,"15min":1.6539999999999999},"min":{"1min":0.376,"5min":0.29399999999999998,"15min":0.24099999999999999},"max":{"1min":3.3759999999999999,"5min":101.07599999999999,"15min":101.07599999999999},"last":0.495}]},{"osd":1,"last update":"Fri Feb 9 17:55:45 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":1.641,"15min":1.6739999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.24199999999999999},"max":{"1min":3.6320000000000001,"5min":17.314,"15min":17.314},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.70599999999999996,"5min":1.3100000000000001,"15min":1.7470000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.1180000000000003,"5min":8.9960000000000004,"15min":15.276},"last":0.46100000000000002}]},{"osd":2,"last update":"Fri Feb 9 17:55:45 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":1.319,"15min":1.6200000000000001},"min":{"1min":0.34899999999999998,"5min":0.30599999999999999,"15min":0.25},"max":{"1min":1.1200000000000001,"5min":9.8279999999999994,"15min":9.8279999999999994},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.90500000000000003,"5min":1.546,"15min":1.851},"min":{"1min":0.371,"5min":0.33800000000000002,"15min":0.29499999999999998},"max":{"1min":5.5670000000000002,"5min":8.3889999999999993,"15min":11.968},"last":0.35399999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:56:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":1.7529999999999999,"15min":2.1960000000000002},"min":{"1min":0.41599999999999998,"5min":0.32800000000000001,"15min":0.25600000000000001},"max":{"1min":2.0179999999999998,"5min":11.112,"15min":78.644999999999996},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.68200000000000005,"5min":1.7130000000000001,"15min":2.3170000000000002},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.33800000000000002},"max":{"1min":2.0830000000000002,"5min":14.592000000000001,"15min":78.414000000000001},"last":0.503}]},{"osd":4,"last update":"Fri Feb 9 17:55:45 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.83799999999999997,"15min":1.4450000000000001},"min":{"1min":0.29499999999999998,"5min":0.28100000000000003,"15min":0.23699999999999999},"max":{"1min":1.1819999999999999,"5min":4.7270000000000003,"15min":86.813000000000002},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.85699999999999998,"15min":1.6639999999999999},"min":{"1min":0.315,"5min":0.26300000000000001,"15min":0.23899999999999999},"max":{"1min":1.085,"5min":8.2880000000000003,"15min":86.751000000000005},"last":0.56200000000000006}]},{"osd":5,"last update":"Fri Feb 9 17:55:45 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.93500000000000005,"15min":1.478},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.65200000000000002,"5min":4.6539999999999999,"15min":9.6590000000000007},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.53800000000000003,"5min":1.022,"15min":1.633},"min":{"1min":0.40899999999999997,"5min":0.35199999999999998,"15min":0.31900000000000001},"max":{"1min":0.74199999999999999,"5min":6.9379999999999997,"15min":15.228999999999999},"last":0.60499999999999998}]},{"osd":6,"last update":"Fri Feb 9 17:56:11 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":1.0109999999999999,"15min":1.629},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.913,"5min":6.3520000000000003,"15min":15.275},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.623,"5min":0.874,"15min":1.448},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.29199999999999998},"max":{"1min":1.931,"5min":4.6980000000000004,"15min":18.236999999999998},"last":0.64400000000000002}]},{"osd":8,"last update":"Fri Feb 9 17:56:08 2024","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.3069999999999999,"15min":1.472},"min":{"1min":0.52000000000000002,"5min":0.39600000000000002,"15min":0.33100000000000002},"max":{"1min":2.7829999999999999,"5min":9.2249999999999996,"15min":11.81},"last":0.56999999999999995},{"interface":"front","average":{"1min":1.073,"5min":1.3560000000000001,"15min":1.4970000000000001},"min":{"1min":0.432,"5min":0.37,"15min":0.37},"max":{"1min":2.8109999999999999,"5min":8.3949999999999996,"15min":12.058},"last":0.67600000000000005}]},{"osd":9,"last update":"Fri Feb 9 17:55:58 2024","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.8120000000000001,"15min":1.681},"min":{"1min":0.47299999999999998,"5min":0.44800000000000001,"15min":0.32700000000000001},"max":{"1min":3.4910000000000001,"5min":8.8119999999999994,"15min":11.941000000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":1.012,"5min":1.704,"15min":1.74},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40000000000000002},"max":{"1min":2.4590000000000001,"5min":15.852,"15min":15.852},"last":0.68899999999999995}]},{"osd":10,"last update":"Fri Feb 9 17:55:25 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":1.516,"15min":1.458},"min":{"1min":0.498,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":3.0139999999999998,"5min":9.5350000000000001,"15min":9.5350000000000001},"last":0.59499999999999997},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.889,"15min":1.7969999999999999},"min":{"1min":0.59999999999999998,"5min":0.40400000000000003,"15min":0.38900000000000001},"max":{"1min":4.4240000000000004,"5min":10.263999999999999,"15min":11.878},"last":0.627}]},{"osd":11,"last update":"Fri Feb 9 17:55:45 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":1.5469999999999999,"15min":1.5669999999999999},"min":{"1min":0.56000000000000005,"5min":0.375,"15min":0.35999999999999999},"max":{"1min":1.7270000000000001,"5min":11.961,"15min":11.961},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.79300000000000004,"5min":1.786,"15min":1.6950000000000001},"min":{"1min":0.52500000000000002,"5min":0.48199999999999998,"15min":0.438},"max":{"1min":1.5640000000000001,"5min":12.243,"15min":12.423999999999999},"last":0.70899999999999996}]}]},{"osd":6,"up_from":49,"seq":210453397938,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":678336,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":676734,"kb_avail":93066816,"statfs":{"total":95995035648,"available":95300419584,"internally_reserved":0,"allocated":1441792,"data_stored":1026518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":692976070},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:55:36 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":1.3899999999999999,"15min":1.4850000000000001},"min":{"1min":0.29699999999999999,"5min":0.23999999999999999,"15min":0.22800000000000001},"max":{"1min":6.048,"5min":10.356,"15min":10.481999999999999},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.84599999999999997,"5min":1.4370000000000001,"15min":1.583},"min":{"1min":0.36599999999999999,"5min":0.27400000000000002,"15min":0.24399999999999999},"max":{"1min":6.8440000000000003,"5min":10.436,"15min":10.436},"last":1.1220000000000001}]},{"osd":1,"last update":"Fri Feb 9 17:55:36 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":1.377,"15min":2.218},"min":{"1min":0.37,"5min":0.25,"15min":0.25},"max":{"1min":0.94499999999999995,"5min":9.593,"15min":79.945999999999998},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.67800000000000005,"5min":1.429,"15min":2.298},"min":{"1min":0.46300000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.97399999999999998,"5min":9.6329999999999991,"15min":80.003},"last":0.442}]},{"osd":2,"last update":"Fri Feb 9 17:55:34 2024","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":1.4950000000000001,"15min":2.0899999999999999},"min":{"1min":0.436,"5min":0.29999999999999999,"15min":0.28100000000000003},"max":{"1min":3.569,"5min":9.1080000000000005,"15min":84.912999999999997},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.81899999999999995,"5min":1.536,"15min":2.129},"min":{"1min":0.35099999999999998,"5min":0.34599999999999997,"15min":0.34000000000000002},"max":{"1min":3.6019999999999999,"5min":9.0850000000000009,"15min":84.942999999999998},"last":0.57999999999999996}]},{"osd":3,"last update":"Fri Feb 9 17:55:07 2024","interfaces":[{"interface":"back","average":{"1min":1.8240000000000001,"5min":1.5389999999999999,"15min":1.4910000000000001},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":9.0129999999999999,"5min":9.0129999999999999,"15min":9.0129999999999999},"last":0.54500000000000004},{"interface":"front","average":{"1min":1.841,"5min":1.8169999999999999,"15min":1.8129999999999999},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":6.6769999999999996,"5min":8.9410000000000007,"15min":8.9410000000000007},"last":0.48399999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:55:34 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.97799999999999998,"15min":1.7190000000000001},"min":{"1min":0.33200000000000002,"5min":0.26200000000000001,"15min":0.252},"max":{"1min":0.68200000000000005,"5min":8.1349999999999998,"15min":15.061999999999999},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.51200000000000001,"5min":1.0409999999999999,"15min":1.8300000000000001},"min":{"1min":0.39300000000000002,"5min":0.314,"15min":0.26400000000000001},"max":{"1min":0.70499999999999996,"5min":8.1809999999999992,"15min":15.118},"last":0.53100000000000003}]},{"osd":5,"last update":"Fri Feb 9 17:55:34 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":1.1579999999999999,"15min":1.8129999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":1.294,"5min":7.4930000000000003,"15min":14.151999999999999},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":1.139,"15min":1.76},"min":{"1min":0.35999999999999999,"5min":0.32800000000000001,"15min":0.26100000000000001},"max":{"1min":0.84699999999999998,"5min":8.9629999999999992,"15min":18.315999999999999},"last":0.51600000000000001}]},{"osd":7,"last update":"Fri Feb 9 17:56:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":1.0660000000000001,"15min":1.77},"min":{"1min":0.33900000000000002,"5min":0.29499999999999998,"15min":0.28699999999999998},"max":{"1min":2.1819999999999999,"5min":6.343,"15min":23.265999999999998},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.91400000000000003,"15min":1.5309999999999999},"min":{"1min":0.36799999999999999,"5min":0.34699999999999998,"15min":0.30499999999999999},"max":{"1min":1.2949999999999999,"5min":5.0670000000000002,"15min":12.967000000000001},"last":0.42199999999999999}]},{"osd":8,"last update":"Fri Feb 9 17:55:24 2024","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":1.258,"15min":1.048},"min":{"1min":0.438,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":4.6280000000000001,"5min":9.9480000000000004,"15min":9.9480000000000004},"last":1.95},{"interface":"front","average":{"1min":0.80000000000000004,"5min":1.7789999999999999,"15min":1.0629999999999999},"min":{"1min":0.54200000000000004,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.3,"5min":11.316000000000001,"15min":11.316000000000001},"last":1.923}]},{"osd":9,"last update":"Fri Feb 9 17:55:41 2024","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":1.679,"15min":1.629},"min":{"1min":0.46600000000000003,"5min":0.41799999999999998,"15min":0.32600000000000001},"max":{"1min":2.867,"5min":10.58,"15min":10.58},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.83499999999999996,"5min":2.0139999999999998,"15min":1.774},"min":{"1min":0.499,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.792,"5min":10.609,"15min":10.609},"last":0.61599999999999999}]},{"osd":10,"last update":"Fri Feb 9 17:56:11 2024","interfaces":[{"interface":"back","average":{"1min":1.877,"5min":1.9039999999999999,"15min":1.849},"min":{"1min":0.44500000000000001,"5min":0.36599999999999999,"15min":0.34300000000000003},"max":{"1min":21.364999999999998,"5min":21.364999999999998,"15min":25.370000000000001},"last":0.502},{"interface":"front","average":{"1min":1.9219999999999999,"5min":1.8029999999999999,"15min":1.897},"min":{"1min":0.47999999999999998,"5min":0.40100000000000002,"15min":0.35099999999999998},"max":{"1min":21.408999999999999,"5min":21.408999999999999,"15min":26.027999999999999},"last":0.54500000000000004}]},{"osd":11,"last update":"Fri Feb 9 17:55:34 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":1.6060000000000001,"15min":1.7629999999999999},"min":{"1min":0.51800000000000002,"5min":0.40200000000000002,"15min":0.372},"max":{"1min":1.359,"5min":8.4969999999999999,"15min":13.109},"last":1.0900000000000001},{"interface":"front","average":{"1min":0.82799999999999996,"5min":1.734,"15min":1.7629999999999999},"min":{"1min":0.53900000000000003,"5min":0.44500000000000001,"15min":0.373},"max":{"1min":1.383,"5min":10.788,"15min":13.164999999999999},"last":1.1379999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575306,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2946504,"kb_used_data":2030976,"kb_used_omap":1,"kb_used_meta":915518,"kb_avail":90798648,"statfs":{"total":95995035648,"available":92977815552,"internally_reserved":0,"allocated":2079719424,"data_stored":2079304150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":937490885},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:55:51 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":1.835,"15min":1.476},"min":{"1min":0.251,"5min":0.19,"15min":0.19},"max":{"1min":3.524,"5min":14.885999999999999,"15min":14.885999999999999},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.66600000000000004,"5min":1.8919999999999999,"15min":1.6839999999999999},"min":{"1min":0.246,"5min":0.22,"15min":0.214},"max":{"1min":3.5569999999999999,"5min":15.411,"15min":33.389000000000003},"last":0.38800000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:55:15 2024","interfaces":[{"interface":"back","average":{"1min":2.1709999999999998,"5min":1.617,"15min":1.734},"min":{"1min":0.33400000000000002,"5min":0.222,"15min":0.222},"max":{"1min":9.2240000000000002,"5min":13.173,"15min":14.83},"last":0.53400000000000003},{"interface":"front","average":{"1min":2.2890000000000001,"5min":1.5109999999999999,"15min":1.5800000000000001},"min":{"1min":0.378,"5min":0.29299999999999998,"15min":0.222},"max":{"1min":10.4,"5min":11.103,"15min":11.103},"last":0.51400000000000001}]},{"osd":3,"last update":"Fri Feb 9 17:56:07 2024","interfaces":[{"interface":"back","average":{"1min":1.5009999999999999,"5min":3.1970000000000001,"15min":3.5990000000000002},"min":{"1min":0.28899999999999998,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":19.917999999999999,"5min":19.917999999999999,"15min":19.917999999999999},"last":0.34899999999999998},{"interface":"front","average":{"1min":1.7,"5min":3.6589999999999998,"15min":4.1470000000000002},"min":{"1min":0.313,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":20.175999999999998,"5min":20.175999999999998,"15min":20.175999999999998},"last":0.81399999999999995}]},{"osd":4,"last update":"Fri Feb 9 17:55:51 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":1.069,"15min":1.5549999999999999},"min":{"1min":0.35299999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.2,"5min":9.032,"15min":45.137999999999998},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.64600000000000002,"5min":1.77,"15min":1.8660000000000001},"min":{"1min":0.38600000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.5669999999999999,"5min":15.331,"15min":44.950000000000003},"last":0.40100000000000002}]},{"osd":5,"last update":"Fri Feb 9 17:55:29 2024","interfaces":[{"interface":"back","average":{"1min":1.669,"5min":1.5740000000000001,"15min":1.786},"min":{"1min":0.41999999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":15.106999999999999,"5min":15.106999999999999,"15min":45.171999999999997},"last":0.36399999999999999},{"interface":"front","average":{"1min":1.742,"5min":1.8819999999999999,"15min":2.036},"min":{"1min":0.34999999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":15.355,"5min":15.355,"15min":45.127000000000002},"last":0.46000000000000002}]},{"osd":6,"last update":"Fri Feb 9 17:55:42 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":1.5669999999999999,"15min":1.913},"min":{"1min":0.313,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.6749999999999998,"5min":15.154999999999999,"15min":33.338000000000001},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.85099999999999998,"5min":1.625,"15min":1.9770000000000001},"min":{"1min":0.371,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":5.7279999999999998,"5min":15.253,"15min":33.286999999999999},"last":0.442}]},{"osd":7,"last update":"Fri Feb 9 17:55:42 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":1.681,"15min":2.0449999999999999},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.34399999999999997},"max":{"1min":5.7000000000000002,"5min":15.287000000000001,"15min":45.740000000000002},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.7250000000000001,"15min":2.1200000000000001},"min":{"1min":0.48799999999999999,"5min":0.46899999999999997,"15min":0.309},"max":{"1min":5.782,"5min":15.311,"15min":45.792000000000002},"last":0.84199999999999997}]},{"osd":8,"last update":"Fri Feb 9 17:55:51 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":1.7170000000000001,"15min":1.4590000000000001},"min":{"1min":0.46000000000000002,"5min":0.42699999999999999,"15min":0.35599999999999998},"max":{"1min":2.423,"5min":9.1999999999999993,"15min":9.2520000000000007},"last":2.996},{"interface":"front","average":{"1min":0.90400000000000003,"5min":2.1720000000000002,"15min":1.631},"min":{"1min":0.48799999999999999,"5min":0.45500000000000002,"15min":0.35599999999999998},"max":{"1min":2.4119999999999999,"5min":12.464,"15min":12.69},"last":0.78900000000000003}]},{"osd":9,"last update":"Fri Feb 9 17:55:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":2.117,"15min":1.589},"min":{"1min":0.622,"5min":0.42799999999999999,"15min":0.33900000000000002},"max":{"1min":3.343,"5min":20.789000000000001,"15min":20.789000000000001},"last":4.4630000000000001},{"interface":"front","average":{"1min":1.071,"5min":1.982,"15min":1.698},"min":{"1min":0.52100000000000002,"5min":0.39400000000000002,"15min":0.34899999999999998},"max":{"1min":2.9689999999999999,"5min":9.5239999999999991,"15min":9.5239999999999991},"last":3.6800000000000002}]},{"osd":10,"last update":"Fri Feb 9 17:55:29 2024","interfaces":[{"interface":"back","average":{"1min":1.7090000000000001,"5min":2.0910000000000002,"15min":1.879},"min":{"1min":0.55800000000000005,"5min":0.47099999999999997,"15min":0.39200000000000002},"max":{"1min":15.211,"5min":15.211,"15min":33.213999999999999},"last":0.86199999999999999},{"interface":"front","average":{"1min":2.04,"5min":2.4009999999999998,"15min":1.841},"min":{"1min":0.55000000000000004,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":15.271000000000001,"5min":15.271000000000001,"15min":15.271000000000001},"last":0.77700000000000002}]},{"osd":11,"last update":"Fri Feb 9 17:55:42 2024","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":2.9529999999999998,"15min":2.0790000000000002},"min":{"1min":0.57099999999999995,"5min":0.42899999999999999,"15min":0.39100000000000001},"max":{"1min":7.234,"5min":19.648,"15min":19.648},"last":0.67100000000000004},{"interface":"front","average":{"1min":1.099,"5min":2.5209999999999999,"15min":2.0699999999999998},"min":{"1min":0.58299999999999996,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":5.7549999999999999,"5min":19.077000000000002,"15min":33.161000000000001},"last":3.2959999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738832,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1882952,"kb_used_data":1057664,"kb_used_omap":1,"kb_used_meta":825278,"kb_avail":91862200,"statfs":{"total":95995035648,"available":94066892800,"internally_reserved":0,"allocated":1083047936,"data_stored":1082632662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":845085125},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 9 17:55:38 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":1.6599999999999999,"15min":1.4550000000000001},"min":{"1min":0.28199999999999997,"5min":0.23699999999999999,"15min":0.20599999999999999},"max":{"1min":0.73999999999999999,"5min":18.698,"15min":18.698},"last":6.1740000000000004},{"interface":"front","average":{"1min":0.503,"5min":1.736,"15min":1.5469999999999999},"min":{"1min":0.38600000000000001,"5min":0.32100000000000001,"15min":0.24399999999999999},"max":{"1min":0.77800000000000002,"5min":18.742999999999999,"15min":18.742999999999999},"last":6.2169999999999996}]},{"osd":2,"last update":"Fri Feb 9 17:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":1.903,"15min":1.9279999999999999},"min":{"1min":0.317,"5min":0.29499999999999998,"15min":0.245},"max":{"1min":2,"5min":20.013999999999999,"15min":20.013999999999999},"last":13.457000000000001},{"interface":"front","average":{"1min":0.71499999999999997,"5min":1.9990000000000001,"15min":1.681},"min":{"1min":0.307,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":2.7109999999999999,"5min":20.103999999999999,"15min":20.103999999999999},"last":16.085000000000001}]},{"osd":3,"last update":"Fri Feb 9 17:55:22 2024","interfaces":[{"interface":"back","average":{"1min":2.1400000000000001,"5min":2.4700000000000002,"15min":1.9470000000000001},"min":{"1min":0.442,"5min":0.26100000000000001,"15min":0.23100000000000001},"max":{"1min":8.827,"5min":20.469999999999999,"15min":36.052},"last":2.0089999999999999},{"interface":"front","average":{"1min":2.2669999999999999,"5min":2.6429999999999998,"15min":2.1640000000000001},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.217},"max":{"1min":10.414999999999999,"5min":21.920000000000002,"15min":38.383000000000003},"last":6.3380000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:56:08 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":1.619,"15min":1.4730000000000001},"min":{"1min":0.38800000000000001,"5min":0.28100000000000003,"15min":0.27000000000000002},"max":{"1min":1.7490000000000001,"5min":12.757,"15min":12.757},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.627,"5min":1.764,"15min":1.756},"min":{"1min":0.28499999999999998,"5min":0.249,"15min":0.249},"max":{"1min":2.1560000000000001,"5min":17.091999999999999,"15min":17.091999999999999},"last":11.039}]},{"osd":5,"last update":"Fri Feb 9 17:55:22 2024","interfaces":[{"interface":"back","average":{"1min":1.746,"5min":1.407,"15min":1.79},"min":{"1min":0.45700000000000002,"5min":0.34300000000000003,"15min":0.30299999999999999},"max":{"1min":8.7710000000000008,"5min":9.7929999999999993,"15min":48.924999999999997},"last":0.504},{"interface":"front","average":{"1min":1.782,"5min":1.46,"15min":1.8180000000000001},"min":{"1min":0.52400000000000002,"5min":0.38200000000000001,"15min":0.29599999999999999},"max":{"1min":8.798,"5min":9.8230000000000004,"15min":48.969999999999999},"last":0.57299999999999995}]},{"osd":6,"last update":"Fri Feb 9 17:55:47 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":1.9370000000000001,"15min":1.716},"min":{"1min":0.41299999999999998,"5min":0.307,"15min":0.28699999999999998},"max":{"1min":1.7330000000000001,"5min":15.667,"15min":15.667},"last":2.0499999999999998},{"interface":"front","average":{"1min":0.72599999999999998,"5min":2.0099999999999998,"15min":1.8120000000000001},"min":{"1min":0.48399999999999999,"5min":0.34799999999999998,"15min":0.33300000000000002},"max":{"1min":1.7809999999999999,"5min":15.702999999999999,"15min":15.702999999999999},"last":2.0680000000000001}]},{"osd":7,"last update":"Fri Feb 9 17:56:17 2024","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.7789999999999999,"15min":1.7789999999999999},"min":{"1min":0.45300000000000001,"5min":0.33100000000000002,"15min":0.29199999999999998},"max":{"1min":11.089,"5min":11.436,"15min":15.271000000000001},"last":11.089},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.6479999999999999,"15min":1.8420000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.28299999999999997},"max":{"1min":11.119,"5min":17.152999999999999,"15min":17.152999999999999},"last":11.119}]},{"osd":8,"last update":"Fri Feb 9 17:56:16 2024","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":2.3399999999999999,"15min":1.7849999999999999},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.313},"max":{"1min":4.5439999999999996,"5min":17.189,"15min":17.189},"last":11.227},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.911,"15min":1.6499999999999999},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.35099999999999998},"max":{"1min":3.6789999999999998,"5min":17.27,"15min":17.27},"last":11.202999999999999}]},{"osd":9,"last update":"Fri Feb 9 17:55:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":2.5870000000000002,"15min":2.0739999999999998},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.33800000000000002},"max":{"1min":5.3869999999999996,"5min":24.757000000000001,"15min":24.757000000000001},"last":11.148999999999999},{"interface":"front","average":{"1min":1.284,"5min":2.4500000000000002,"15min":2.0019999999999998},"min":{"1min":0.56499999999999995,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":5.4050000000000002,"5min":24.681999999999999,"15min":24.681999999999999},"last":0.59999999999999998}]},{"osd":10,"last update":"Fri Feb 9 17:55:22 2024","interfaces":[{"interface":"back","average":{"1min":2.9929999999999999,"5min":2.5590000000000002,"15min":2.097},"min":{"1min":0.42499999999999999,"5min":0.41999999999999998,"15min":0.35499999999999998},"max":{"1min":15.614000000000001,"5min":15.614000000000001,"15min":31.766999999999999},"last":2.0310000000000001},{"interface":"front","average":{"1min":2.1379999999999999,"5min":2.7679999999999998,"15min":1.905},"min":{"1min":0.51800000000000002,"5min":0.40400000000000003,"15min":0.34200000000000003},"max":{"1min":11.096,"5min":17.242000000000001,"15min":28.048999999999999},"last":11.175000000000001}]},{"osd":11,"last update":"Fri Feb 9 17:55:47 2024","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":2.5550000000000002,"15min":1.835},"min":{"1min":0.51400000000000001,"5min":0.45300000000000001,"15min":0.41599999999999998},"max":{"1min":2.5830000000000002,"5min":25.994,"15min":25.994},"last":0.621},{"interface":"front","average":{"1min":1.1100000000000001,"5min":2.5680000000000001,"15min":1.929},"min":{"1min":0.65600000000000003,"5min":0.441,"15min":0.441},"max":{"1min":3.6549999999999998,"5min":25.515999999999998,"15min":25.515999999999998},"last":0.64400000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411780,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2267152,"kb_used_data":1680648,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":91478000,"statfs":{"total":95995035648,"available":93673472000,"internally_reserved":0,"allocated":1720983552,"data_stored":1720564726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":600570309},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":1.4510000000000001,"15min":1.006},"min":{"1min":0.29999999999999999,"5min":0.23699999999999999,"15min":0.221},"max":{"1min":4.9089999999999998,"5min":17.251000000000001,"15min":17.251000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.74399999999999999,"5min":1.611,"15min":1.113},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.22900000000000001},"max":{"1min":4.7720000000000002,"5min":23.097999999999999,"15min":23.097999999999999},"last":0.875}]},{"osd":1,"last update":"Fri Feb 9 17:56:13 2024","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":1.786,"15min":1.9299999999999999},"min":{"1min":0.253,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":3.8420000000000001,"5min":15.378,"15min":52.646000000000001},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.94799999999999995,"5min":1.895,"15min":1.9279999999999999},"min":{"1min":0.308,"5min":0.248,"15min":0.248},"max":{"1min":3.8719999999999999,"5min":15.092000000000001,"15min":52.683999999999997},"last":0.36699999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:55:40 2024","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":2.8069999999999999,"15min":2.1920000000000002},"min":{"1min":0.29799999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":8.7200000000000006,"5min":46.927999999999997,"15min":46.927999999999997},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.92000000000000004,"5min":2.6110000000000002,"15min":2.1970000000000001},"min":{"1min":0.28000000000000003,"5min":0.192,"15min":0.192},"max":{"1min":5.8140000000000001,"5min":45.798999999999999,"15min":45.798999999999999},"last":0.29299999999999998}]},{"osd":4,"last update":"Fri Feb 9 17:55:40 2024","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":1.198,"15min":1.5569999999999999},"min":{"1min":0.36599999999999999,"5min":0.245,"15min":0.245},"max":{"1min":8.6099999999999994,"5min":10.365,"15min":12.398},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.69099999999999995,"5min":1.4079999999999999,"15min":1.536},"min":{"1min":0.26600000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":3.8149999999999999,"5min":10.529,"15min":10.888999999999999},"last":0.42199999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:55:35 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":1.542,"15min":2.0209999999999999},"min":{"1min":0.34200000000000003,"5min":0.245,"15min":0.245},"max":{"1min":5.742,"5min":12.305999999999999,"15min":42.664000000000001},"last":0.439},{"interface":"front","average":{"1min":0.68000000000000005,"5min":1.4119999999999999,"15min":2.1760000000000002},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":3.718,"5min":10.603,"15min":43.311999999999998},"last":0.46500000000000002}]},{"osd":6,"last update":"Fri Feb 9 17:55:40 2024","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":1.5629999999999999,"15min":1.891},"min":{"1min":0.312,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":5.8879999999999999,"5min":10.573,"15min":18.518999999999998},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.97099999999999997,"5min":1.597,"15min":1.905},"min":{"1min":0.34799999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":5.915,"5min":10.731,"15min":18.481999999999999},"last":0.54300000000000004}]},{"osd":7,"last update":"Fri Feb 9 17:56:13 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":1.5580000000000001,"15min":1.845},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.32400000000000001},"max":{"1min":1.5609999999999999,"5min":10.933,"15min":16.666},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.94799999999999995,"5min":1.3919999999999999,"15min":1.9650000000000001},"min":{"1min":0.33000000000000002,"5min":0.313,"15min":0.313},"max":{"1min":2.79,"5min":10.316000000000001,"15min":16.638999999999999},"last":0.501}]},{"osd":8,"last update":"Fri Feb 9 17:55:40 2024","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":2.2589999999999999,"15min":1.754},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.316},"max":{"1min":3.7839999999999998,"5min":12.500999999999999,"15min":12.500999999999999},"last":2.0019999999999998},{"interface":"front","average":{"1min":0.82399999999999995,"5min":2.5449999999999999,"15min":1.857},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.34399999999999997},"max":{"1min":3.8370000000000002,"5min":54.759999999999998,"15min":54.759999999999998},"last":0.57999999999999996}]},{"osd":9,"last update":"Fri Feb 9 17:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":2.8599999999999999,"15min":2.0510000000000002},"min":{"1min":0.504,"5min":0.318,"15min":0.318},"max":{"1min":2.8900000000000001,"5min":53.935000000000002,"15min":53.935000000000002},"last":0.621},{"interface":"front","average":{"1min":0.93500000000000005,"5min":2.7730000000000001,"15min":1.9510000000000001},"min":{"1min":0.54800000000000004,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":2.1230000000000002,"5min":54.670999999999999,"15min":54.670999999999999},"last":0.64200000000000002}]},{"osd":10,"last update":"Fri Feb 9 17:55:26 2024","interfaces":[{"interface":"back","average":{"1min":1.1279999999999999,"5min":2.4670000000000001,"15min":1.9319999999999999},"min":{"1min":0.46200000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":8.6720000000000006,"5min":54.039999999999999,"15min":54.039999999999999},"last":0.73999999999999999},{"interface":"front","average":{"1min":1.165,"5min":2.5049999999999999,"15min":1.972},"min":{"1min":0.505,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":8.6959999999999997,"5min":54.118000000000002,"15min":54.118000000000002},"last":0.76100000000000001}]},{"osd":11,"last update":"Fri Feb 9 17:55:40 2024","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":2.7519999999999998,"15min":2.0600000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.40600000000000003},"max":{"1min":3.8660000000000001,"5min":54.799999999999997,"15min":54.799999999999997},"last":2.0859999999999999},{"interface":"front","average":{"1min":1.0780000000000001,"5min":2.8490000000000002,"15min":2.0609999999999999},"min":{"1min":0.51300000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":5.7869999999999999,"5min":54.103000000000002,"15min":54.103000000000002},"last":1.4510000000000001}]}]},{"osd":3,"up_from":27,"seq":115964117442,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1896712,"kb_used_data":1057536,"kb_used_omap":1,"kb_used_meta":839166,"kb_avail":91848440,"statfs":{"total":95995035648,"available":94052802560,"internally_reserved":0,"allocated":1082916864,"data_stored":1082501590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":859306439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:55:33 2024","interfaces":[{"interface":"back","average":{"1min":1.8620000000000001,"5min":2.4260000000000002,"15min":1.8999999999999999},"min":{"1min":0.26500000000000001,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":12.391,"5min":17.006,"15min":17.006},"last":10.628},{"interface":"front","average":{"1min":1.3580000000000001,"5min":1.694,"15min":1.9430000000000001},"min":{"1min":0.24099999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":6.343,"5min":12.288,"15min":16.637},"last":6.016}]},{"osd":1,"last update":"Fri Feb 9 17:56:03 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":1.8460000000000001,"15min":2.512},"min":{"1min":0.23499999999999999,"5min":0.221,"15min":0.221},"max":{"1min":2.0779999999999998,"5min":12.609,"15min":53.152999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.68000000000000005,"5min":1.4379999999999999,"15min":1.859},"min":{"1min":0.29799999999999999,"5min":0.24299999999999999,"15min":0.23200000000000001},"max":{"1min":2.1419999999999999,"5min":12.237,"15min":53.118000000000002},"last":8.3960000000000008}]},{"osd":2,"last update":"Fri Feb 9 17:55:33 2024","interfaces":[{"interface":"back","average":{"1min":1.903,"5min":2.2080000000000002,"15min":1.9330000000000001},"min":{"1min":0.27600000000000002,"5min":0.26700000000000002,"15min":0.248},"max":{"1min":12.349,"5min":17.477,"15min":17.477},"last":10.353999999999999},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.9730000000000001,"15min":1.8720000000000001},"min":{"1min":0.26600000000000001,"5min":0.23699999999999999,"15min":0.23499999999999999},"max":{"1min":7.4059999999999997,"5min":21.103000000000002,"15min":21.103000000000002},"last":0.20399999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:55:33 2024","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":1.4430000000000001,"15min":1.724},"min":{"1min":0.29299999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":3.7149999999999999,"5min":12.718,"15min":12.718},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.77200000000000002,"5min":1.2210000000000001,"15min":1.591},"min":{"1min":0.28699999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":5.6280000000000001,"5min":5.8849999999999998,"15min":19.713999999999999},"last":4.7990000000000004}]},{"osd":5,"last update":"Fri Feb 9 17:56:17 2024","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.7889999999999999,"15min":1.98},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.28000000000000003},"max":{"1min":8.1920000000000002,"5min":13.321,"15min":14.106},"last":8.1920000000000002},{"interface":"front","average":{"1min":1.4399999999999999,"5min":1.8440000000000001,"15min":1.968},"min":{"1min":0.44,"5min":0.44,"15min":0.316},"max":{"1min":8.3789999999999996,"5min":13.347,"15min":14.138999999999999},"last":8.3789999999999996}]},{"osd":6,"last update":"Fri Feb 9 17:56:03 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":1.829,"15min":1.8580000000000001},"min":{"1min":0.41499999999999998,"5min":0.35799999999999998,"15min":0.28899999999999998},"max":{"1min":1.385,"5min":13.207000000000001,"15min":15.061999999999999},"last":8.3469999999999995},{"interface":"front","average":{"1min":0.68300000000000005,"5min":1.988,"15min":1.9339999999999999},"min":{"1min":0.44900000000000001,"5min":0.375,"15min":0.36399999999999999},"max":{"1min":1.448,"5min":13.298,"15min":13.298},"last":8.4160000000000004}]},{"osd":7,"last update":"Fri Feb 9 17:55:50 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":1.6359999999999999,"15min":1.869},"min":{"1min":0.374,"5min":0.313,"15min":0.313},"max":{"1min":1.1140000000000001,"5min":18.587,"15min":19.788},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":1.8879999999999999,"15min":1.921},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.34899999999999998},"max":{"1min":1.4139999999999999,"5min":13.239000000000001,"15min":13.239000000000001},"last":8.3040000000000003}]},{"osd":8,"last update":"Fri Feb 9 17:45:16 2024","interfaces":[{"interface":"back","average":{"1min":1.714,"5min":1.4450000000000001,"15min":1.498},"min":{"1min":0.377,"5min":0.377,"15min":0.26800000000000002},"max":{"1min":8.6020000000000003,"5min":11.821,"15min":11.821},"last":0.67700000000000005},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.1719999999999999,"15min":1.351},"min":{"1min":0.46500000000000002,"5min":0.36199999999999999,"15min":0.34499999999999997},"max":{"1min":6.8390000000000004,"5min":6.8390000000000004,"15min":14.129},"last":0.78200000000000003}]},{"osd":9,"last update":"Fri Feb 9 17:56:03 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.9379999999999999,"15min":1.804},"min":{"1min":0.497,"5min":0.312,"15min":0.312},"max":{"1min":3.0270000000000001,"5min":11.039,"15min":11.851000000000001},"last":4.8479999999999999},{"interface":"front","average":{"1min":1.026,"5min":1.8500000000000001,"15min":1.7969999999999999},"min":{"1min":0.376,"5min":0.318,"15min":0.318},"max":{"1min":3.2400000000000002,"5min":10.462999999999999,"15min":11.907},"last":4.8810000000000002}]},{"osd":10,"last update":"Fri Feb 9 17:55:50 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":2.3580000000000001,"15min":1.75},"min":{"1min":0.52400000000000002,"5min":0.371,"15min":0.371},"max":{"1min":1.5449999999999999,"5min":18.632999999999999,"15min":18.632999999999999},"last":0.437},{"interface":"front","average":{"1min":0.82399999999999995,"5min":2.2549999999999999,"15min":1.798},"min":{"1min":0.60699999999999998,"5min":0.46800000000000003,"15min":0.34100000000000003},"max":{"1min":1.5760000000000001,"5min":13.272,"15min":19.859000000000002},"last":8.2530000000000001}]},{"osd":11,"last update":"Fri Feb 9 17:55:33 2024","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":2.6070000000000002,"15min":1.899},"min":{"1min":0.44,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":7.2679999999999998,"5min":18.66,"15min":19.824000000000002},"last":0.45800000000000002},{"interface":"front","average":{"1min":1.4019999999999999,"5min":2.6589999999999998,"15min":1.7889999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":7.3739999999999997,"5min":18.687999999999999,"15min":18.687999999999999},"last":0.47399999999999998}]}]},{"osd":4,"up_from":32,"seq":137438953916,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2148424,"kb_used_data":1363392,"kb_used_omap":1,"kb_used_meta":785022,"kb_avail":91596728,"statfs":{"total":95995035648,"available":93795049472,"internally_reserved":0,"allocated":1396113408,"data_stored":1395698134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":803862981},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:55:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":1.26,"15min":1.3260000000000001},"min":{"1min":0.27200000000000002,"5min":0.246,"15min":0.222},"max":{"1min":0.67400000000000004,"5min":10.005000000000001,"15min":10.301},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.47699999999999998,"5min":1.3320000000000001,"15min":1.3959999999999999},"min":{"1min":0.34799999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.69699999999999995,"5min":10.036,"15min":10.407999999999999},"last":0.42999999999999999}]},{"osd":1,"last update":"Fri Feb 9 17:55:52 2024","interfaces":[{"interface":"back","average":{"1min":2.8050000000000002,"5min":1.9279999999999999,"15min":1.8600000000000001},"min":{"1min":0.27600000000000002,"5min":0.26100000000000001,"15min":0.23699999999999999},"max":{"1min":50.255000000000003,"5min":50.255000000000003,"15min":50.255000000000003},"last":0.32300000000000001},{"interface":"front","average":{"1min":2.903,"5min":2.258,"15min":2.1800000000000002},"min":{"1min":0.36599999999999999,"5min":0.23699999999999999,"15min":0.23100000000000001},"max":{"1min":49.991999999999997,"5min":49.991999999999997,"15min":49.991999999999997},"last":0.76500000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.663,"15min":2.6150000000000002},"min":{"1min":0.38700000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":4.2220000000000004,"5min":8.5139999999999993,"15min":17.494},"last":0.46000000000000002},{"interface":"front","average":{"1min":1.351,"5min":1.865,"15min":2.6110000000000002},"min":{"1min":0.38,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.6609999999999996,"5min":17.283999999999999,"15min":17.283999999999999},"last":0.44800000000000001}]},{"osd":3,"last update":"Fri Feb 9 17:55:52 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":1.248,"15min":1.569},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.71799999999999997,"5min":10.484999999999999,"15min":14.448},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.4419999999999999,"15min":1.655},"min":{"1min":0.315,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.93000000000000005,"5min":10.455,"15min":10.455},"last":0.379}]},{"osd":5,"last update":"Fri Feb 9 17:56:11 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.81399999999999995,"15min":1.8009999999999999},"min":{"1min":0.27300000000000002,"5min":0.253,"15min":0.22600000000000001},"max":{"1min":2.0259999999999998,"5min":4.8019999999999996,"15min":18.887},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.88100000000000001,"15min":1.881},"min":{"1min":0.33100000000000002,"5min":0.31900000000000001,"15min":0.31},"max":{"1min":2.0609999999999999,"5min":4.8639999999999999,"15min":18.917000000000002},"last":0.442}]},{"osd":6,"last update":"Fri Feb 9 17:55:52 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.0620000000000001,"15min":1.4730000000000001},"min":{"1min":0.31900000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.83999999999999997,"5min":23.343,"15min":23.343},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.50700000000000001,"5min":1.0189999999999999,"15min":1.601},"min":{"1min":0.30599999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.80900000000000005,"5min":21.861000000000001,"15min":21.861000000000001},"last":0.70699999999999996}]},{"osd":7,"last update":"Fri Feb 9 17:55:28 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":1.0129999999999999,"15min":1.536},"min":{"1min":0.41599999999999998,"5min":0.32900000000000001,"15min":0.29899999999999999},"max":{"1min":0.77200000000000002,"5min":6.9550000000000001,"15min":11.875},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.51100000000000001,"5min":1.0389999999999999,"15min":1.6890000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.28599999999999998},"max":{"1min":0.748,"5min":8.6950000000000003,"15min":12.696},"last":0.496}]},{"osd":8,"last update":"Fri Feb 9 17:56:11 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.7110000000000001,"15min":1.829},"min":{"1min":0.41399999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":9.7059999999999995,"5min":9.7059999999999995,"15min":17.335999999999999},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.92900000000000005,"5min":1.419,"15min":1.2869999999999999},"min":{"1min":0.39200000000000002,"5min":0.373,"15min":0.34699999999999998},"max":{"1min":2.3580000000000001,"5min":5.6470000000000002,"15min":10.433},"last":1.0249999999999999}]},{"osd":9,"last update":"Fri Feb 9 17:56:14 2024","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":1.77,"15min":2.4409999999999998},"min":{"1min":0.46400000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":3.214,"5min":7.665,"15min":7.665},"last":0.46400000000000002},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.3999999999999999,"15min":1.619},"min":{"1min":0.47299999999999998,"5min":0.44,"15min":0.44},"max":{"1min":3.585,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":2.468}]},{"osd":10,"last update":"Fri Feb 9 17:55:28 2024","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":1.264,"15min":1.5329999999999999},"min":{"1min":0.48899999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":6.9509999999999996,"5min":6.9509999999999996,"15min":17.367000000000001},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.83699999999999997,"5min":1.3680000000000001,"15min":1.587},"min":{"1min":0.47199999999999998,"5min":0.38,"15min":0.38},"max":{"1min":3.028,"5min":5.7439999999999998,"15min":12.888},"last":0.57599999999999996}]},{"osd":11,"last update":"Fri Feb 9 17:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":1.7869999999999999,"15min":1.5429999999999999},"min":{"1min":0.56999999999999995,"5min":0.46500000000000002,"15min":0.39900000000000002},"max":{"1min":1.9119999999999999,"5min":7.8410000000000002,"15min":10.521000000000001},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.76700000000000002,"5min":1.669,"15min":1.5860000000000001},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.40999999999999998},"max":{"1min":1.5760000000000001,"5min":8.4420000000000002,"15min":13.154999999999999},"last":0.60099999999999998}]}]},{"osd":5,"up_from":40,"seq":171798692278,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2350544,"kb_used_data":1661320,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":91394608,"statfs":{"total":95995035648,"available":93588078592,"internally_reserved":0,"allocated":1701191680,"data_stored":1700772854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:52:23 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":2.4900000000000002,"15min":1.71},"min":{"1min":0.27600000000000002,"5min":0.26600000000000001,"15min":0.23599999999999999},"max":{"1min":2.613,"5min":116.77800000000001,"15min":116.77800000000001},"last":0.748},{"interface":"front","average":{"1min":1.246,"5min":2.867,"15min":2.0529999999999999},"min":{"1min":0.36299999999999999,"5min":0.252,"15min":0.249},"max":{"1min":3.2959999999999998,"5min":116.959,"15min":116.959},"last":0.36599999999999999}]},{"osd":1,"last update":"Fri Feb 9 17:55:15 2024","interfaces":[{"interface":"back","average":{"1min":1.552,"5min":1.0820000000000001,"15min":1.6779999999999999},"min":{"1min":0.30099999999999999,"5min":0.27900000000000003,"15min":0.24199999999999999},"max":{"1min":9.532,"5min":9.532,"15min":19.079999999999998},"last":0.318},{"interface":"front","average":{"1min":1.1379999999999999,"5min":0.89600000000000002,"15min":1.6679999999999999},"min":{"1min":0.28299999999999997,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":6.2190000000000003,"5min":6.2190000000000003,"15min":19.065999999999999},"last":0.28699999999999998}]},{"osd":2,"last update":"Fri Feb 9 17:55:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":1.087,"15min":1.73},"min":{"1min":0.26300000000000001,"5min":0.26100000000000001,"15min":0.249},"max":{"1min":1.804,"5min":7.5670000000000002,"15min":13.929},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.81699999999999995,"5min":1.3049999999999999,"15min":1.9650000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":5.1189999999999998,"5min":6.8929999999999998,"15min":15.945},"last":0.378}]},{"osd":3,"last update":"Fri Feb 9 17:55:15 2024","interfaces":[{"interface":"back","average":{"1min":2.0219999999999998,"5min":1.8779999999999999,"15min":2.1179999999999999},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":20.690000000000001,"5min":20.690000000000001,"15min":20.690000000000001},"last":0.46700000000000003},{"interface":"front","average":{"1min":1.591,"5min":1.5069999999999999,"15min":2.036},"min":{"1min":0.35499999999999998,"5min":0.316,"15min":0.313},"max":{"1min":6.9269999999999996,"5min":17.209,"15min":17.209},"last":0.48199999999999998}]},{"osd":4,"last update":"Fri Feb 9 17:55:15 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.88,"15min":1.5609999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.249},"max":{"1min":3.8719999999999999,"5min":4.3849999999999998,"15min":28.481999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.89700000000000002,"15min":1.6339999999999999},"min":{"1min":0.32900000000000001,"5min":0.22500000000000001,"15min":0.218},"max":{"1min":1.3129999999999999,"5min":4.9189999999999996,"15min":15.843},"last":0.502}]},{"osd":6,"last update":"Fri Feb 9 17:55:50 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.90400000000000003,"15min":1.6020000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.27900000000000003},"max":{"1min":0.748,"5min":6.0869999999999997,"15min":16.657},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":1.131,"15min":1.7290000000000001},"min":{"1min":0.39400000000000002,"5min":0.33900000000000002,"15min":0.26200000000000001},"max":{"1min":1.179,"5min":9.7200000000000006,"15min":20.192},"last":0.40899999999999997}]},{"osd":7,"last update":"Fri Feb 9 17:55:19 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":1.2589999999999999,"15min":1.704},"min":{"1min":0.39000000000000001,"5min":0.29499999999999998,"15min":0.27900000000000003},"max":{"1min":3.6880000000000002,"5min":18.134,"15min":18.134},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.59199999999999997,"5min":1.1830000000000001,"15min":1.8859999999999999},"min":{"1min":0.39800000000000002,"5min":0.32600000000000001,"15min":0.31},"max":{"1min":1.0760000000000001,"5min":19.300000000000001,"15min":19.300000000000001},"last":0.57899999999999996}]},{"osd":8,"last update":"Fri Feb 9 17:55:38 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":1.4410000000000001,"15min":1.734},"min":{"1min":0.56799999999999995,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.577,"5min":9.6110000000000007,"15min":9.6110000000000007},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.90600000000000003,"5min":1.526,"15min":1.8080000000000001},"min":{"1min":0.39600000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":4.1849999999999996,"5min":8.1829999999999998,"15min":8.1829999999999998},"last":0.55000000000000004}]},{"osd":9,"last update":"Fri Feb 9 17:55:50 2024","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":1.879,"15min":1.7190000000000001},"min":{"1min":0.48299999999999998,"5min":0.41499999999999998,"15min":0.35299999999999998},"max":{"1min":2.5379999999999998,"5min":10.987,"15min":16.411000000000001},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.82199999999999995,"5min":1.347,"15min":1.73},"min":{"1min":0.46500000000000002,"5min":0.373,"15min":0.35299999999999998},"max":{"1min":1.9099999999999999,"5min":9.7050000000000001,"15min":16.417999999999999},"last":2.3620000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:55:15 2024","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":1.5549999999999999,"15min":2},"min":{"1min":0.52100000000000002,"5min":0.373,"15min":0.317},"max":{"1min":4.0140000000000002,"5min":16.292000000000002,"15min":80.042000000000002},"last":3.0710000000000002},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.502,"15min":1.9019999999999999},"min":{"1min":0.54800000000000004,"5min":0.41199999999999998,"15min":0.38200000000000001},"max":{"1min":4.4960000000000004,"5min":12.426,"15min":79.807000000000002},"last":0.69599999999999995}]},{"osd":11,"last update":"Fri Feb 9 17:55:32 2024","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.548,"15min":1.9790000000000001},"min":{"1min":0.51500000000000001,"5min":0.44700000000000001,"15min":0.38400000000000001},"max":{"1min":7.5270000000000001,"5min":9.7680000000000007,"15min":16.390000000000001},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.78600000000000003,"5min":1.9330000000000001,"15min":1.835},"min":{"1min":0.434,"5min":0.434,"15min":0.42199999999999999},"max":{"1min":2.101,"5min":20.981000000000002,"15min":20.981000000000002},"last":0.52500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1087569920,"data_stored":1087569920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2078605312,"data_stored":2078605312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1728053248,"data_stored":1728053248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1084162048,"data_stored":1084162048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1407188992,"data_stored":1407188992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1718026240,"data_stored":1718026240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":707788800,"data_stored":707788800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1061617664,"data_stored":1061617664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2115174400,"data_stored":2115174400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2168717312,"data_stored":2168717312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1762197504,"data_stored":1762197504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-09T17:56:25.800 INFO:tasks.thrashosds.thrasher:Setting 8.7 to [8, 5, 11, 10, 0, 9] 2024-02-09T17:56:25.800 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '8.7', '8', '5', '11', '10', '0', '9'] 2024-02-09T17:56:25.801 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd pg-upmap-items 8.7 8 5 11 10 0 9 2024-02-09T17:56:25.815 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:56:25.854 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:25.943 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:56:25.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:25.951 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:56:26.042 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T17:56:26.042 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:56:26.051 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:56:26.119 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:26.351 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:26 smithi099 ceph-mon[39420]: pgmap v2000: 47 pgs: 47 active+clean; 5.6 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.13k op/s 2024-02-09T17:56:26.351 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:26 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:56:26.351 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:26 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:56:26.351 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:26 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:56:26.351 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:26 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:56:26.479 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:56:26.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:26.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:56:26.515 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:56:26.578 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:56:26.588 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:56:26.604 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:2024-02-09T17:56:26.604129+0000 min lat: 0.00109523 max lat: 0.219811 avg lat: 0.00988664 2024-02-09T17:56:26.605 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-09T17:56:26.605 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 60 16 97109 97093 101.108 66.4375 0.013082 0.00988664 2024-02-09T17:56:26.614 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:56:26.630 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:26.658 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:56:26.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:26 smithi179 ceph-mon[43975]: pgmap v2000: 47 pgs: 47 active+clean; 5.6 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.13k op/s 2024-02-09T17:56:26.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:26 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:56:26.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:26 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:56:26.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:26 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:56:26.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:26 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:56:26.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:26 smithi162 ceph-mon[44304]: pgmap v2000: 47 pgs: 47 active+clean; 5.6 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.13k op/s 2024-02-09T17:56:26.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:26 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:56:26.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:26 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:56:26.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:26 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:56:26.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:26 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:56:26.696 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:56:26.721 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:56:26.744 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:56:26.773 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:56:26.805 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:56:26.809 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:56:26.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:26.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:56:26.846 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:56:26.887 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:56:26.914 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:56:26.917 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset noscrub 2024-02-09T17:56:26.921 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:56:26.949 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:56:26.951 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:56:26.951 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:56:26.951 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:56:26.951 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:56:26.951 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:56:26.951 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:56:26.952 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:56:26.952 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:56:26.952 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:56:26.952 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:56:26.952 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:56:26.952 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:56:26.965 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:27.605 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 61 16 98182 98166 100.55 67.0625 0.0118626 0.00994091 2024-02-09T17:56:27.624 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:27 smithi099 ceph-mon[39420]: osdmap e811: 12 total, 12 up, 12 in 2024-02-09T17:56:27.624 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:27 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/37497103' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:56:27.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:27 smithi179 ceph-mon[43975]: osdmap e811: 12 total, 12 up, 12 in 2024-02-09T17:56:27.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:27 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/37497103' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:56:27.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:27 smithi162 ceph-mon[44304]: osdmap e811: 12 total, 12 up, 12 in 2024-02-09T17:56:27.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:27 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/37497103' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:56:27.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:27.834 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:56:27.835 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:56:27.871 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.10 dump_historic_ops 2024-02-09T17:56:27.936 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:56:27.999 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:28.026 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:56:28.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:28.030 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:56:28.131 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:56:28.167 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:28.310 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:56:28.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:28 smithi099 ceph-mon[39420]: pgmap v2002: 47 pgs: 47 active+clean; 5.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.06k op/s 2024-02-09T17:56:28.605 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 62 16 99200 99184 99.9537 63.625 0.0187343 0.00999975 2024-02-09T17:56:28.626 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:28 smithi179 ceph-mon[43975]: pgmap v2002: 47 pgs: 47 active+clean; 5.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.06k op/s 2024-02-09T17:56:28.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:28 smithi162 ceph-mon[44304]: pgmap v2002: 47 pgs: 47 active+clean; 5.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.06k op/s 2024-02-09T17:56:29.127 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:56:29.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:29.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:56:29.231 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:56:29.279 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:29.301 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:56:29.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:29.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:56:29.406 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:56:29.436 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:29.458 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:56:29.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:29.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:56:29.564 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:56:29.594 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:29.605 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 63 16 100276 100260 99.4346 67.25 0.00560855 0.0100523 2024-02-09T17:56:29.616 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:56:29.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:29.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:56:29.720 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:56:29.750 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:29.774 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:56:29.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:29.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:56:29.878 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:56:29.934 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:30.194 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:56:30.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:30.206 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:56:30.307 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:56:30.354 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:30.605 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 64 16 101358 101342 98.9376 67.625 0.00957392 0.0101032 2024-02-09T17:56:30.616 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:56:30.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:30 smithi179 ceph-mon[43975]: pgmap v2003: 47 pgs: 47 active+clean; 6.0 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-02-09T17:56:30.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:30 smithi162 ceph-mon[44304]: pgmap v2003: 47 pgs: 47 active+clean; 6.0 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-02-09T17:56:30.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:30 smithi099 ceph-mon[39420]: pgmap v2003: 47 pgs: 47 active+clean; 6.0 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-02-09T17:56:30.813 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:56:30.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:30.822 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:56:30.923 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:56:30.953 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:30.978 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:56:30.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:30.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:56:31.084 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:56:31.151 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:31.178 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:56:31.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:31.184 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:56:31.285 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:56:31.338 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:31.390 INFO:teuthology.orchestra.run.smithi099.stderr:set 8.7 pg_upmap_items mapping to [8->5,11->10,0->9] 2024-02-09T17:56:31.479 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:56:31.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:31.487 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:56:31.589 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:56:31.599 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:56:31.599 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-02-09T17:56:31.600 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-02-09T17:56:31.600 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-02-09T17:56:31.600 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.600 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25837.0:202623 6.f 6:fc8a4001:::benchmark_data_smithi179_126206_object72242:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e638)", 2024-02-09T17:56:31.600 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:48:47.488364+0000", 2024-02-09T17:56:31.600 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 464.10712147599997, 2024-02-09T17:56:31.600 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.092512484000000006, 2024-02-09T17:56:31.600 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:31.600 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:31.600 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:31.600 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25837", 2024-02-09T17:56:31.600 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:56:31.600 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 202623 2024-02-09T17:56:31.601 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.601 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:31.601 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.601 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:31.601 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.488364+0000", 2024-02-09T17:56:31.601 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:31.601 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.601 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.601 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:31.601 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.488364+0000", 2024-02-09T17:56:31.601 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:31.601 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.601 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.601 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:31.602 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.488365+0000", 2024-02-09T17:56:31.602 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.47e-07 2024-02-09T17:56:31.602 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.602 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.602 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:31.602 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.488370+0000", 2024-02-09T17:56:31.602 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4120000000000003e-06 2024-02-09T17:56:31.602 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.602 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.602 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:31.602 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.488388+0000", 2024-02-09T17:56:31.602 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8479e-05 2024-02-09T17:56:31.602 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.602 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.603 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:31.603 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.488396+0000", 2024-02-09T17:56:31.603 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.661e-06 2024-02-09T17:56:31.603 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.603 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.603 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:31.603 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.497037+0000", 2024-02-09T17:56:31.603 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0086414300000000003 2024-02-09T17:56:31.603 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.603 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.603 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:31.603 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.497173+0000", 2024-02-09T17:56:31.603 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013531099999999999 2024-02-09T17:56:31.603 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.603 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.604 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,9", 2024-02-09T17:56:31.604 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.497293+0000", 2024-02-09T17:56:31.604 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000120549 2024-02-09T17:56:31.604 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.604 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.604 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:31.604 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.501982+0000", 2024-02-09T17:56:31.604 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0046894010000000002 2024-02-09T17:56:31.604 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.604 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.604 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:31.604 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.506566+0000", 2024-02-09T17:56:31.605 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00458334 2024-02-09T17:56:31.605 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.605 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.605 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:31.605 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.580816+0000", 2024-02-09T17:56:31.605 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.074250110999999994 2024-02-09T17:56:31.605 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.605 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.605 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:31.605 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.580852+0000", 2024-02-09T17:56:31.605 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.5929000000000001e-05 2024-02-09T17:56:31.605 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.605 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.605 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:31.606 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.580877+0000", 2024-02-09T17:56:31.606 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4913999999999999e-05 2024-02-09T17:56:31.606 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:31.606 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:31.606 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:31.606 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.606 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.606 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25837.0:202637 6.f 6:f258b00e:::benchmark_data_smithi179_126206_object72256:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e638)", 2024-02-09T17:56:31.606 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:48:47.501920+0000", 2024-02-09T17:56:31.606 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 464.09356530700001, 2024-02-09T17:56:31.606 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.079088620999999998, 2024-02-09T17:56:31.606 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:31.606 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:31.606 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:31.606 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25837", 2024-02-09T17:56:31.607 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:56:31.607 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 202637 2024-02-09T17:56:31.607 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.607 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:31.607 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.607 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:31.607 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.501920+0000", 2024-02-09T17:56:31.607 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:31.607 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.607 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.607 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:31.607 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.501920+0000", 2024-02-09T17:56:31.607 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:31.607 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.607 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.607 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:31.608 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.501922+0000", 2024-02-09T17:56:31.608 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0190000000000001e-06 2024-02-09T17:56:31.608 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.608 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.608 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:31.608 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.501927+0000", 2024-02-09T17:56:31.608 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0010000000000001e-06 2024-02-09T17:56:31.608 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.608 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.608 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:31.608 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.501939+0000", 2024-02-09T17:56:31.608 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.146e-05 2024-02-09T17:56:31.608 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.608 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.608 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:31.608 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.501946+0000", 2024-02-09T17:56:31.609 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4320000000000002e-06 2024-02-09T17:56:31.609 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.609 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.609 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:31.609 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.502019+0000", 2024-02-09T17:56:31.609 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2898000000000006e-05 2024-02-09T17:56:31.609 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.609 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.609 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:31.609 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.502156+0000", 2024-02-09T17:56:31.609 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013662400000000001 2024-02-09T17:56:31.609 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.610 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.610 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,9", 2024-02-09T17:56:31.610 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.502280+0000", 2024-02-09T17:56:31.610 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012392800000000001 2024-02-09T17:56:31.610 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.610 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.610 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:31.610 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.506713+0000", 2024-02-09T17:56:31.610 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.004433607 2024-02-09T17:56:31.610 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.610 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.610 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:31.610 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.515170+0000", 2024-02-09T17:56:31.611 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0084569550000000004 2024-02-09T17:56:31.611 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.611 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.611 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:31.611 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.580924+0000", 2024-02-09T17:56:31.611 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.065753354 2024-02-09T17:56:31.611 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.611 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.611 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:31.611 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.580972+0000", 2024-02-09T17:56:31.611 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8059000000000002e-05 2024-02-09T17:56:31.611 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.612 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.612 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:31.612 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.581009+0000", 2024-02-09T17:56:31.612 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7283999999999997e-05 2024-02-09T17:56:31.612 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:31.612 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:31.612 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:31.612 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.612 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.612 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25837.0:202638 6.f 6:fc3bb8d9:::benchmark_data_smithi179_126206_object72257:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e638)", 2024-02-09T17:56:31.612 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:48:47.504642+0000", 2024-02-09T17:56:31.612 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 464.09084358699999, 2024-02-09T17:56:31.612 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.076473263999999999, 2024-02-09T17:56:31.613 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:31.613 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:31.613 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:31.613 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25837", 2024-02-09T17:56:31.613 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:56:31.613 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 202638 2024-02-09T17:56:31.613 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.613 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:31.613 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.613 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:31.613 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.504642+0000", 2024-02-09T17:56:31.613 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:31.613 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.614 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.614 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:31.614 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.504642+0000", 2024-02-09T17:56:31.614 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:31.614 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.614 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.614 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:31.614 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.504643+0000", 2024-02-09T17:56:31.614 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.879999999999999e-07 2024-02-09T17:56:31.614 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.614 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.614 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:31.614 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.504648+0000", 2024-02-09T17:56:31.614 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3810000000000001e-06 2024-02-09T17:56:31.614 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.615 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.615 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:31.615 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.504658+0000", 2024-02-09T17:56:31.615 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0549e-05 2024-02-09T17:56:31.615 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.615 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.615 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:31.615 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.504664+0000", 2024-02-09T17:56:31.615 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1689999999999997e-06 2024-02-09T17:56:31.615 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.615 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.615 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:31.615 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.506739+0000", 2024-02-09T17:56:31.615 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00207435 2024-02-09T17:56:31.616 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.616 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.616 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:31.616 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.506833+0000", 2024-02-09T17:56:31.616 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.4765999999999994e-05 2024-02-09T17:56:31.616 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.616 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.616 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,9", 2024-02-09T17:56:31.616 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.506907+0000", 2024-02-09T17:56:31.616 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.3178999999999998e-05 2024-02-09T17:56:31.616 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.616 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.616 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:31.617 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.511798+0000", 2024-02-09T17:56:31.617 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0048913070000000001 2024-02-09T17:56:31.617 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.617 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.617 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:31.617 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.515176+0000", 2024-02-09T17:56:31.617 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0033782669999999999 2024-02-09T17:56:31.617 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.617 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.617 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:31.617 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.581037+0000", 2024-02-09T17:56:31.617 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.065861123999999993 2024-02-09T17:56:31.617 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.618 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.618 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:31.618 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.581081+0000", 2024-02-09T17:56:31.618 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3374999999999997e-05 2024-02-09T17:56:31.618 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.618 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.618 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:31.618 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:47.581115+0000", 2024-02-09T17:56:31.618 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4808999999999998e-05 2024-02-09T17:56:31.618 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:31.618 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:31.618 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:31.618 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.618 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.618 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:76563 8.5 8:a2fcd537:::benchmark_data_smithi179_148205_object76562:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e809)", 2024-02-09T17:56:31.619 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:56:08.330120+0000", 2024-02-09T17:56:31.619 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 23.265365686999999, 2024-02-09T17:56:31.619 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.075664743000000007, 2024-02-09T17:56:31.619 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:31.619 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:31.619 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:31.619 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T17:56:31.619 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T17:56:31.619 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 76563 2024-02-09T17:56:31.619 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.619 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:31.619 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.619 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:31.619 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.330120+0000", 2024-02-09T17:56:31.620 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:31.620 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.620 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.620 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:31.620 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.330120+0000", 2024-02-09T17:56:31.620 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:31.620 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.620 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.620 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:31.620 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.330121+0000", 2024-02-09T17:56:31.620 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1500000000000003e-07 2024-02-09T17:56:31.620 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.620 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.620 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:31.621 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.330165+0000", 2024-02-09T17:56:31.621 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3900999999999999e-05 2024-02-09T17:56:31.621 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.621 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.621 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:31.621 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.330174+0000", 2024-02-09T17:56:31.621 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.789e-06 2024-02-09T17:56:31.621 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.621 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.621 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:31.621 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.330179+0000", 2024-02-09T17:56:31.621 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7119999999999997e-06 2024-02-09T17:56:31.621 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.621 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.622 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:31.622 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.332569+0000", 2024-02-09T17:56:31.622 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002389281 2024-02-09T17:56:31.622 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.622 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.622 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:31.622 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.332671+0000", 2024-02-09T17:56:31.622 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000102328 2024-02-09T17:56:31.622 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.622 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.622 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:56:31.622 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.332793+0000", 2024-02-09T17:56:31.622 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000121903 2024-02-09T17:56:31.622 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.623 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.623 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:31.623 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.335693+0000", 2024-02-09T17:56:31.623 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0029004149999999999 2024-02-09T17:56:31.623 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.623 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.623 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:31.623 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.337739+0000", 2024-02-09T17:56:31.623 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0020461260000000001 2024-02-09T17:56:31.623 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.623 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.623 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:31.623 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.405691+0000", 2024-02-09T17:56:31.623 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.067951479999999995 2024-02-09T17:56:31.624 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.624 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.624 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:31.624 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.405746+0000", 2024-02-09T17:56:31.624 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5384999999999999e-05 2024-02-09T17:56:31.624 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.624 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.624 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:31.624 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.405785+0000", 2024-02-09T17:56:31.624 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8608e-05 2024-02-09T17:56:31.624 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:31.624 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:31.624 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:31.624 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.625 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.625 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:76565 8.2 8:450dee5e:::benchmark_data_smithi179_148205_object76564:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e809)", 2024-02-09T17:56:31.625 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:56:08.332068+0000", 2024-02-09T17:56:31.625 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 23.263417458999999, 2024-02-09T17:56:31.625 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.084815183000000002, 2024-02-09T17:56:31.625 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:31.625 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:31.625 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:31.625 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T17:56:31.625 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T17:56:31.625 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 76565 2024-02-09T17:56:31.625 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.625 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:31.625 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.626 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:31.626 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.332068+0000", 2024-02-09T17:56:31.626 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:31.626 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.626 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.626 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:31.626 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.332068+0000", 2024-02-09T17:56:31.626 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:31.626 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.626 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.626 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:31.626 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.332069+0000", 2024-02-09T17:56:31.626 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2699999999999998e-07 2024-02-09T17:56:31.626 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.626 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.627 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:31.627 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.332085+0000", 2024-02-09T17:56:31.627 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5523e-05 2024-02-09T17:56:31.627 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.627 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.627 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:31.627 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.332093+0000", 2024-02-09T17:56:31.627 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5669999999999995e-06 2024-02-09T17:56:31.627 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.627 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.627 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:31.627 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.332099+0000", 2024-02-09T17:56:31.627 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3990000000000003e-06 2024-02-09T17:56:31.627 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.628 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.628 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:31.628 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.332154+0000", 2024-02-09T17:56:31.628 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5404e-05 2024-02-09T17:56:31.628 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.628 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.628 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:31.628 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.332288+0000", 2024-02-09T17:56:31.628 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000134084 2024-02-09T17:56:31.628 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.628 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.628 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,4", 2024-02-09T17:56:31.628 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.332417+0000", 2024-02-09T17:56:31.629 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012870899999999999 2024-02-09T17:56:31.629 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.629 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.629 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:31.629 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.405143+0000", 2024-02-09T17:56:31.629 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.072725889000000002 2024-02-09T17:56:31.629 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.629 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.629 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:31.629 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.405206+0000", 2024-02-09T17:56:31.629 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2965000000000004e-05 2024-02-09T17:56:31.629 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.629 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.629 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:31.629 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.416789+0000", 2024-02-09T17:56:31.630 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011583579 2024-02-09T17:56:31.630 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.630 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.630 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:31.630 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.416844+0000", 2024-02-09T17:56:31.630 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4573000000000001e-05 2024-02-09T17:56:31.630 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.630 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.630 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:31.630 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.416883+0000", 2024-02-09T17:56:31.630 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9663999999999998e-05 2024-02-09T17:56:31.630 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:31.630 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:31.631 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:31.631 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.631 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.631 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:76568 8.2 8:4027815f:::benchmark_data_smithi179_148205_object76567:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e809)", 2024-02-09T17:56:31.631 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:56:08.332178+0000", 2024-02-09T17:56:31.631 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 23.263307811000001, 2024-02-09T17:56:31.631 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.084929631000000005, 2024-02-09T17:56:31.631 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:31.631 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:31.631 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:31.631 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T17:56:31.631 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T17:56:31.631 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 76568 2024-02-09T17:56:31.631 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.632 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:31.632 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.632 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:31.632 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.332178+0000", 2024-02-09T17:56:31.632 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:31.632 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.632 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.632 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:31.632 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.332178+0000", 2024-02-09T17:56:31.632 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:31.632 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.632 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.632 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:31.633 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.332179+0000", 2024-02-09T17:56:31.633 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.16e-07 2024-02-09T17:56:31.633 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.633 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.633 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:31.633 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.332198+0000", 2024-02-09T17:56:31.633 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8839999999999999e-05 2024-02-09T17:56:31.633 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.633 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.633 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:31.633 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.332205+0000", 2024-02-09T17:56:31.633 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.0720000000000002e-06 2024-02-09T17:56:31.633 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.634 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.634 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:31.634 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.332210+0000", 2024-02-09T17:56:31.634 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1240000000000004e-06 2024-02-09T17:56:31.634 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.634 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.634 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:31.634 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.335363+0000", 2024-02-09T17:56:31.634 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0031535209999999998 2024-02-09T17:56:31.634 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.634 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.634 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:31.634 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.335499+0000", 2024-02-09T17:56:31.635 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013521200000000001 2024-02-09T17:56:31.635 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.635 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.635 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,4", 2024-02-09T17:56:31.635 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.335606+0000", 2024-02-09T17:56:31.635 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000107371 2024-02-09T17:56:31.635 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.635 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.635 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:31.635 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.411204+0000", 2024-02-09T17:56:31.635 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.075598355000000006 2024-02-09T17:56:31.635 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.635 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.636 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:31.636 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.411244+0000", 2024-02-09T17:56:31.636 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9656000000000001e-05 2024-02-09T17:56:31.636 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.636 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.636 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:31.636 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.417014+0000", 2024-02-09T17:56:31.636 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0057697030000000002 2024-02-09T17:56:31.636 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.636 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.636 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:31.636 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.417067+0000", 2024-02-09T17:56:31.636 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3440000000000003e-05 2024-02-09T17:56:31.636 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.637 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.637 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:31.637 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.417108+0000", 2024-02-09T17:56:31.637 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0420999999999998e-05 2024-02-09T17:56:31.637 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:31.637 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:31.637 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:31.637 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.637 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.637 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop_reply(client.26305.0:76565 8.2 e809/785 ondisk, result = 0)", 2024-02-09T17:56:31.637 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:56:08.335288+0000", 2024-02-09T17:56:31.637 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 23.260198182, 2024-02-09T17:56:31.637 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.069870758000000005, 2024-02-09T17:56:31.637 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:31.638 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-02-09T17:56:31.638 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:31.638 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.638 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:31.638 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.335288+0000", 2024-02-09T17:56:31.638 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:31.638 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.638 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.638 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:31.638 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.335288+0000", 2024-02-09T17:56:31.638 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:31.638 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.638 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.638 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:31.639 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.335288+0000", 2024-02-09T17:56:31.639 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3799999999999996e-07 2024-02-09T17:56:31.639 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.639 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.639 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:31.639 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.335293+0000", 2024-02-09T17:56:31.639 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.211e-06 2024-02-09T17:56:31.639 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.639 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.639 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:31.639 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.335303+0000", 2024-02-09T17:56:31.639 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.9639999999999998e-06 2024-02-09T17:56:31.639 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.639 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.639 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:31.639 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.335309+0000", 2024-02-09T17:56:31.640 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7360000000000004e-06 2024-02-09T17:56:31.640 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.640 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.640 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:31.640 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.405110+0000", 2024-02-09T17:56:31.640 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.069800366000000003 2024-02-09T17:56:31.640 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.640 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.640 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:31.640 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.405127+0000", 2024-02-09T17:56:31.640 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7056999999999999e-05 2024-02-09T17:56:31.640 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.640 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.640 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:31.641 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.405158+0000", 2024-02-09T17:56:31.641 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1585999999999999e-05 2024-02-09T17:56:31.641 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:31.641 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:31.641 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:31.641 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.641 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.641 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:76569 8.2 8:4a41ef4d:::benchmark_data_smithi179_148205_object76568:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e809)", 2024-02-09T17:56:31.641 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:56:08.335549+0000", 2024-02-09T17:56:31.641 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 23.259936319000001, 2024-02-09T17:56:31.641 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.081649264999999999, 2024-02-09T17:56:31.641 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:31.641 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:31.642 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:31.642 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T17:56:31.642 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T17:56:31.642 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 76569 2024-02-09T17:56:31.642 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.642 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:31.642 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.642 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:31.642 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.335549+0000", 2024-02-09T17:56:31.642 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:31.642 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.642 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.642 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:31.642 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.335549+0000", 2024-02-09T17:56:31.642 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:31.643 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.643 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.643 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:31.643 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.335551+0000", 2024-02-09T17:56:31.643 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1489999999999999e-06 2024-02-09T17:56:31.643 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.643 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.643 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:31.643 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.335601+0000", 2024-02-09T17:56:31.643 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0377999999999998e-05 2024-02-09T17:56:31.643 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.643 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.643 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:31.643 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.335611+0000", 2024-02-09T17:56:31.644 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0186999999999999e-05 2024-02-09T17:56:31.644 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.644 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.644 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:31.644 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.335619+0000", 2024-02-09T17:56:31.644 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1300000000000001e-06 2024-02-09T17:56:31.644 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.644 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.644 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:31.644 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.405239+0000", 2024-02-09T17:56:31.644 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.069619723999999994 2024-02-09T17:56:31.644 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.644 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.644 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:31.645 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.405392+0000", 2024-02-09T17:56:31.645 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015252899999999999 2024-02-09T17:56:31.645 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.645 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.645 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,4", 2024-02-09T17:56:31.645 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.405541+0000", 2024-02-09T17:56:31.645 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014960599999999999 2024-02-09T17:56:31.645 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.645 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.645 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:31.645 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.411281+0000", 2024-02-09T17:56:31.645 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0057398479999999997 2024-02-09T17:56:31.645 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.645 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.646 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:31.646 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.411321+0000", 2024-02-09T17:56:31.646 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9904999999999998e-05 2024-02-09T17:56:31.646 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.646 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.646 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:31.646 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.417116+0000", 2024-02-09T17:56:31.646 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0057946289999999999 2024-02-09T17:56:31.646 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.646 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.646 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:31.646 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.417163+0000", 2024-02-09T17:56:31.646 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7679999999999998e-05 2024-02-09T17:56:31.646 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.646 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.646 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:31.647 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.417199+0000", 2024-02-09T17:56:31.647 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.5500000000000002e-05 2024-02-09T17:56:31.647 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:31.647 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:31.647 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:31.647 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.647 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.647 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:76577 8.2 8:42349304:::benchmark_data_smithi179_148205_object76576:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e809)", 2024-02-09T17:56:31.647 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:56:08.338364+0000", 2024-02-09T17:56:31.647 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 23.257122108000001, 2024-02-09T17:56:31.647 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.078934032000000001, 2024-02-09T17:56:31.647 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:31.647 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:31.647 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:31.648 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T17:56:31.648 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T17:56:31.648 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 76577 2024-02-09T17:56:31.648 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.648 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:31.648 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.648 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:31.648 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.338364+0000", 2024-02-09T17:56:31.648 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:31.648 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.648 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.648 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:31.648 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.338364+0000", 2024-02-09T17:56:31.648 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:31.648 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.649 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.649 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:31.649 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.338366+0000", 2024-02-09T17:56:31.649 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8640000000000001e-06 2024-02-09T17:56:31.649 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.649 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.649 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:31.649 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.338414+0000", 2024-02-09T17:56:31.649 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8032999999999999e-05 2024-02-09T17:56:31.649 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.649 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.649 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:31.649 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.405115+0000", 2024-02-09T17:56:31.649 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.066701163999999993 2024-02-09T17:56:31.649 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.650 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.650 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:31.650 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.405138+0000", 2024-02-09T17:56:31.650 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3057999999999998e-05 2024-02-09T17:56:31.650 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.650 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.650 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:31.650 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.408321+0000", 2024-02-09T17:56:31.650 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0031836260000000002 2024-02-09T17:56:31.650 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.650 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.650 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:31.650 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.408442+0000", 2024-02-09T17:56:31.650 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000120449 2024-02-09T17:56:31.650 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.651 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.651 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,4", 2024-02-09T17:56:31.651 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.408573+0000", 2024-02-09T17:56:31.651 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000131493 2024-02-09T17:56:31.651 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.651 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.651 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:31.651 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.416937+0000", 2024-02-09T17:56:31.651 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0083633909999999995 2024-02-09T17:56:31.651 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.651 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.651 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:31.651 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.416983+0000", 2024-02-09T17:56:31.651 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6393000000000001e-05 2024-02-09T17:56:31.651 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.652 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.652 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:31.652 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.417214+0000", 2024-02-09T17:56:31.652 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000230764 2024-02-09T17:56:31.652 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.652 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.652 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:31.652 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.417261+0000", 2024-02-09T17:56:31.652 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6896e-05 2024-02-09T17:56:31.652 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.652 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.652 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:31.652 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.417298+0000", 2024-02-09T17:56:31.652 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6900999999999998e-05 2024-02-09T17:56:31.653 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:31.653 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:31.653 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:31.653 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.653 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.653 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop_reply(client.26305.0:76568 8.2 e809/785 ondisk, result = 0)", 2024-02-09T17:56:31.653 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:56:08.338455+0000", 2024-02-09T17:56:31.653 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 23.257031187999999, 2024-02-09T17:56:31.653 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.072795622000000004, 2024-02-09T17:56:31.653 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:31.653 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-02-09T17:56:31.653 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:31.653 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.653 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:31.653 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.338455+0000", 2024-02-09T17:56:31.654 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:31.654 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.654 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.654 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:31.654 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.338455+0000", 2024-02-09T17:56:31.654 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:31.654 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.654 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.654 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:31.654 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.338462+0000", 2024-02-09T17:56:31.654 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5259999999999996e-06 2024-02-09T17:56:31.654 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.654 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.654 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:31.655 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.338467+0000", 2024-02-09T17:56:31.655 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7859999999999996e-06 2024-02-09T17:56:31.655 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.655 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.655 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:31.655 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.405543+0000", 2024-02-09T17:56:31.655 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.067076419999999998 2024-02-09T17:56:31.655 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.655 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.655 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:31.655 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.405555+0000", 2024-02-09T17:56:31.655 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1474999999999999e-05 2024-02-09T17:56:31.655 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.656 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.656 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:31.656 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.411232+0000", 2024-02-09T17:56:31.656 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0056775200000000001 2024-02-09T17:56:31.656 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.656 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.656 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:31.656 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.411238+0000", 2024-02-09T17:56:31.656 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9939999999999997e-06 2024-02-09T17:56:31.656 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.656 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.656 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:31.656 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.411250+0000", 2024-02-09T17:56:31.656 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1901e-05 2024-02-09T17:56:31.657 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:31.657 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:31.657 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:31.657 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.657 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.657 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop_reply(client.26305.0:76568 8.2 e809/785 ondisk, result = 0)", 2024-02-09T17:56:31.657 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:56:08.338891+0000", 2024-02-09T17:56:31.657 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 23.256594649, 2024-02-09T17:56:31.657 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.072320544, 2024-02-09T17:56:31.657 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:31.657 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-02-09T17:56:31.657 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:31.657 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.657 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:31.658 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.338891+0000", 2024-02-09T17:56:31.658 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:31.658 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.658 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.658 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:31.658 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.338891+0000", 2024-02-09T17:56:31.658 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:31.658 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.658 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.658 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:31.658 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.338892+0000", 2024-02-09T17:56:31.658 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.92e-07 2024-02-09T17:56:31.658 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.658 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.658 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:31.659 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.338897+0000", 2024-02-09T17:56:31.659 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8420000000000004e-06 2024-02-09T17:56:31.659 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.659 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.659 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:31.659 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.405271+0000", 2024-02-09T17:56:31.659 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.066374293000000001 2024-02-09T17:56:31.659 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.659 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.659 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:31.659 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.405284+0000", 2024-02-09T17:56:31.659 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2726e-05 2024-02-09T17:56:31.659 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.659 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.660 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:31.660 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.411190+0000", 2024-02-09T17:56:31.660 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0059060700000000002 2024-02-09T17:56:31.660 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.660 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.660 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:31.660 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.411197+0000", 2024-02-09T17:56:31.660 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1709999999999996e-06 2024-02-09T17:56:31.660 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.660 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.660 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:31.660 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:08.411212+0000", 2024-02-09T17:56:31.660 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4749999999999999e-05 2024-02-09T17:56:31.660 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:31.661 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:31.661 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:31.661 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.661 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.661 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:78418 8.5 8:aa8566a1:::benchmark_data_smithi179_148205_object78417:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e809)", 2024-02-09T17:56:31.661 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:56:09.842338+0000", 2024-02-09T17:56:31.661 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 21.753147799000001, 2024-02-09T17:56:31.661 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.11411647, 2024-02-09T17:56:31.661 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:31.661 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:31.661 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:31.661 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T17:56:31.661 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T17:56:31.661 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 78418 2024-02-09T17:56:31.661 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.662 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:31.662 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.662 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:31.662 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:09.842338+0000", 2024-02-09T17:56:31.662 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:31.662 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.662 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.662 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:31.662 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:09.842338+0000", 2024-02-09T17:56:31.662 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:31.662 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.662 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.662 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:31.662 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:09.842340+0000", 2024-02-09T17:56:31.662 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.561e-06 2024-02-09T17:56:31.663 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.663 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.663 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:31.663 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:09.842373+0000", 2024-02-09T17:56:31.663 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.3624000000000002e-05 2024-02-09T17:56:31.663 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.663 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.663 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:31.663 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:09.842383+0000", 2024-02-09T17:56:31.663 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.8900000000000002e-06 2024-02-09T17:56:31.663 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.663 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.663 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:31.663 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:09.842396+0000", 2024-02-09T17:56:31.663 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2566e-05 2024-02-09T17:56:31.664 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.664 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.664 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:31.664 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:09.842541+0000", 2024-02-09T17:56:31.664 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014532 2024-02-09T17:56:31.664 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.664 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.664 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:31.664 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:09.842684+0000", 2024-02-09T17:56:31.664 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000142757 2024-02-09T17:56:31.664 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.664 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.664 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:56:31.664 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:09.842840+0000", 2024-02-09T17:56:31.665 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000155864 2024-02-09T17:56:31.665 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.665 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.665 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:31.665 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:09.845518+0000", 2024-02-09T17:56:31.665 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002678387 2024-02-09T17:56:31.665 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.665 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.665 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:31.665 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:09.846332+0000", 2024-02-09T17:56:31.665 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00081387899999999999 2024-02-09T17:56:31.665 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.665 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.665 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:31.665 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:09.956360+0000", 2024-02-09T17:56:31.666 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.110028057 2024-02-09T17:56:31.666 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.666 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.666 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:31.666 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:09.956418+0000", 2024-02-09T17:56:31.666 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8057999999999999e-05 2024-02-09T17:56:31.666 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.666 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.666 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:31.666 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:09.956454+0000", 2024-02-09T17:56:31.666 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6507000000000001e-05 2024-02-09T17:56:31.666 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:31.666 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:31.666 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:31.666 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.666 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.667 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:78441 8.5 8:a86c53ac:::benchmark_data_smithi179_148205_object78440:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e809)", 2024-02-09T17:56:31.667 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:56:09.862945+0000", 2024-02-09T17:56:31.667 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 21.732540606000001, 2024-02-09T17:56:31.667 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.096827603999999998, 2024-02-09T17:56:31.667 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:31.667 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:31.667 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:31.667 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T17:56:31.667 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T17:56:31.667 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 78441 2024-02-09T17:56:31.667 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.667 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:31.667 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.668 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:31.668 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:09.862945+0000", 2024-02-09T17:56:31.668 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:31.668 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.668 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.668 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:31.668 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:09.862945+0000", 2024-02-09T17:56:31.668 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:31.668 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.668 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.668 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:31.668 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:09.862947+0000", 2024-02-09T17:56:31.668 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3799999999999999e-06 2024-02-09T17:56:31.668 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.668 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.669 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:31.669 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:09.862967+0000", 2024-02-09T17:56:31.669 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.069e-05 2024-02-09T17:56:31.669 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.669 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.669 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:31.669 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:09.862981+0000", 2024-02-09T17:56:31.669 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.377e-05 2024-02-09T17:56:31.669 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.669 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.669 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:31.669 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:09.862990+0000", 2024-02-09T17:56:31.669 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5399999999999996e-06 2024-02-09T17:56:31.669 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.669 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.670 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:31.670 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:09.863040+0000", 2024-02-09T17:56:31.670 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0362999999999999e-05 2024-02-09T17:56:31.670 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.670 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.670 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:31.670 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:09.863157+0000", 2024-02-09T17:56:31.670 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000117593 2024-02-09T17:56:31.670 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.670 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.670 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:56:31.670 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:09.863249+0000", 2024-02-09T17:56:31.670 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.1917999999999996e-05 2024-02-09T17:56:31.670 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.670 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.671 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:31.671 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:09.865758+0000", 2024-02-09T17:56:31.671 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0025082799999999999 2024-02-09T17:56:31.671 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.671 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.671 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:31.671 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:09.868017+0000", 2024-02-09T17:56:31.671 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002259306 2024-02-09T17:56:31.671 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.671 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.671 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:31.671 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:09.959675+0000", 2024-02-09T17:56:31.671 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.091657823999999999 2024-02-09T17:56:31.671 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.671 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.672 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:31.672 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:09.959736+0000", 2024-02-09T17:56:31.672 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1050999999999995e-05 2024-02-09T17:56:31.672 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.672 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.672 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:31.672 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:09.959773+0000", 2024-02-09T17:56:31.672 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6888999999999999e-05 2024-02-09T17:56:31.672 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:31.672 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:31.672 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:31.672 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.672 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.672 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:78448 8.5 8:a3cef99d:::benchmark_data_smithi179_148205_object78447:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e809)", 2024-02-09T17:56:31.673 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:56:09.870399+0000", 2024-02-09T17:56:31.673 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 21.725086994000002, 2024-02-09T17:56:31.673 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.089493699999999995, 2024-02-09T17:56:31.673 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:31.673 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:31.673 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:31.673 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T17:56:31.673 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T17:56:31.673 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 78448 2024-02-09T17:56:31.673 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.673 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:31.673 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.673 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:31.673 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:09.870399+0000", 2024-02-09T17:56:31.674 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:31.674 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.674 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.674 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:31.674 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:09.870399+0000", 2024-02-09T17:56:31.674 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:31.674 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.674 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.674 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:31.674 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:09.870400+0000", 2024-02-09T17:56:31.674 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.451e-06 2024-02-09T17:56:31.674 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.674 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.674 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:31.674 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:09.870420+0000", 2024-02-09T17:56:31.675 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9741e-05 2024-02-09T17:56:31.675 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.675 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.675 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:31.675 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:09.870431+0000", 2024-02-09T17:56:31.675 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0686999999999999e-05 2024-02-09T17:56:31.675 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.675 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.675 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:31.675 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:09.870436+0000", 2024-02-09T17:56:31.675 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.592e-06 2024-02-09T17:56:31.675 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.675 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.675 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:31.676 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:09.870477+0000", 2024-02-09T17:56:31.676 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1019000000000001e-05 2024-02-09T17:56:31.676 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.676 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.676 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:31.676 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:09.870569+0000", 2024-02-09T17:56:31.676 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.1837999999999997e-05 2024-02-09T17:56:31.676 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.676 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.676 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:56:31.676 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:09.870660+0000", 2024-02-09T17:56:31.676 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.0573000000000002e-05 2024-02-09T17:56:31.676 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.676 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.677 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:31.677 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:09.872700+0000", 2024-02-09T17:56:31.677 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0020404580000000002 2024-02-09T17:56:31.677 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.677 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.677 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:31.677 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:09.872841+0000", 2024-02-09T17:56:31.677 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0001404 2024-02-09T17:56:31.677 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.677 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.677 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:31.677 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:09.959810+0000", 2024-02-09T17:56:31.677 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.086969542999999996 2024-02-09T17:56:31.677 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.677 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.678 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:31.678 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:09.959861+0000", 2024-02-09T17:56:31.678 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1334e-05 2024-02-09T17:56:31.678 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.678 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.678 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:31.678 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:09.959892+0000", 2024-02-09T17:56:31.678 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1063999999999999e-05 2024-02-09T17:56:31.678 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:31.678 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:31.678 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:31.678 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.678 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.678 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:81697 8.2 8:49e2fa2f:::benchmark_data_smithi179_148205_object81696:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e809)", 2024-02-09T17:56:31.678 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:56:12.708398+0000", 2024-02-09T17:56:31.679 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 18.887088092999999, 2024-02-09T17:56:31.679 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.13639375400000001, 2024-02-09T17:56:31.679 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:31.679 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:31.679 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:31.679 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T17:56:31.679 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T17:56:31.679 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 81697 2024-02-09T17:56:31.679 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.679 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:31.679 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.679 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:31.679 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.708398+0000", 2024-02-09T17:56:31.679 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:31.680 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.680 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.680 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:31.680 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.708398+0000", 2024-02-09T17:56:31.680 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:31.680 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.680 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.680 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:31.680 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.708400+0000", 2024-02-09T17:56:31.680 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9920000000000002e-06 2024-02-09T17:56:31.680 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.680 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.680 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:31.681 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.708419+0000", 2024-02-09T17:56:31.681 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9639e-05 2024-02-09T17:56:31.681 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.681 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.681 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:31.681 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.708427+0000", 2024-02-09T17:56:31.681 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.0399999999999993e-06 2024-02-09T17:56:31.681 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.681 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.681 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:31.681 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.708432+0000", 2024-02-09T17:56:31.681 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6600000000000003e-06 2024-02-09T17:56:31.681 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.681 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.682 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:31.682 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.708483+0000", 2024-02-09T17:56:31.682 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0889e-05 2024-02-09T17:56:31.682 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.682 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.682 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:31.682 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.708618+0000", 2024-02-09T17:56:31.682 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013486699999999999 2024-02-09T17:56:31.682 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.682 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.682 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,4", 2024-02-09T17:56:31.682 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.708747+0000", 2024-02-09T17:56:31.682 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012905800000000001 2024-02-09T17:56:31.682 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.682 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.683 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:31.683 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.712230+0000", 2024-02-09T17:56:31.683 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0034827320000000001 2024-02-09T17:56:31.683 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.683 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.683 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:31.683 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.712620+0000", 2024-02-09T17:56:31.683 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00039090300000000001 2024-02-09T17:56:31.683 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.683 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.683 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:31.683 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.844685+0000", 2024-02-09T17:56:31.683 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.132064704 2024-02-09T17:56:31.683 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.684 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.684 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:31.684 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.844748+0000", 2024-02-09T17:56:31.684 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2977000000000004e-05 2024-02-09T17:56:31.684 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.684 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.684 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:31.684 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.844791+0000", 2024-02-09T17:56:31.684 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3293e-05 2024-02-09T17:56:31.684 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:31.684 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:31.684 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:31.684 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.685 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.685 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:81701 8.9 8:960965d9:::benchmark_data_smithi179_148205_object81700:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e809)", 2024-02-09T17:56:31.685 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:56:12.708448+0000", 2024-02-09T17:56:31.685 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 18.887037418999999, 2024-02-09T17:56:31.685 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.13639743600000001, 2024-02-09T17:56:31.685 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:31.685 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:31.685 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:31.685 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T17:56:31.685 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T17:56:31.685 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 81701 2024-02-09T17:56:31.685 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.685 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:31.685 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.686 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:31.686 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.708448+0000", 2024-02-09T17:56:31.686 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:31.686 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.686 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.686 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:31.686 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.708448+0000", 2024-02-09T17:56:31.686 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:31.686 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.686 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.686 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:31.686 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.708449+0000", 2024-02-09T17:56:31.686 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2699999999999999e-07 2024-02-09T17:56:31.686 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.686 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.687 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:31.687 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.708469+0000", 2024-02-09T17:56:31.687 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0140999999999999e-05 2024-02-09T17:56:31.687 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.687 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.687 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:31.687 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.708476+0000", 2024-02-09T17:56:31.687 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.0119999999999999e-06 2024-02-09T17:56:31.687 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.687 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.687 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:31.687 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.708482+0000", 2024-02-09T17:56:31.687 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3709999999999996e-06 2024-02-09T17:56:31.687 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.687 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.688 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:31.688 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.708769+0000", 2024-02-09T17:56:31.688 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00028648999999999998 2024-02-09T17:56:31.688 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.688 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.688 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:31.688 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.708880+0000", 2024-02-09T17:56:31.688 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000111387 2024-02-09T17:56:31.688 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.688 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.688 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,5", 2024-02-09T17:56:31.688 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.709003+0000", 2024-02-09T17:56:31.688 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000122695 2024-02-09T17:56:31.688 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.689 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.689 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:31.689 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.712206+0000", 2024-02-09T17:56:31.689 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0032034009999999998 2024-02-09T17:56:31.689 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.689 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.689 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:31.689 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.714253+0000", 2024-02-09T17:56:31.689 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0020461839999999999 2024-02-09T17:56:31.689 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.689 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.689 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:31.689 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.844764+0000", 2024-02-09T17:56:31.689 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.130511236 2024-02-09T17:56:31.689 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.690 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.690 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:31.690 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.844813+0000", 2024-02-09T17:56:31.690 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8962999999999999e-05 2024-02-09T17:56:31.690 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.690 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.690 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:31.690 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.844846+0000", 2024-02-09T17:56:31.690 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.3028999999999998e-05 2024-02-09T17:56:31.690 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:31.690 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:31.690 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:31.690 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.690 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.690 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:81702 8.5 8:a7cb69df:::benchmark_data_smithi179_148205_object81701:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e809)", 2024-02-09T17:56:31.691 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:56:12.708494+0000", 2024-02-09T17:56:31.691 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 18.886991512000002, 2024-02-09T17:56:31.691 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.13625372899999999, 2024-02-09T17:56:31.691 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:31.691 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:31.691 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:31.691 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T17:56:31.691 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T17:56:31.691 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 81702 2024-02-09T17:56:31.691 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.691 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:31.691 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.691 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:31.691 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.708494+0000", 2024-02-09T17:56:31.691 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:31.692 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.692 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.692 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:31.692 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.708494+0000", 2024-02-09T17:56:31.692 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:31.692 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.692 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.692 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:31.692 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.708495+0000", 2024-02-09T17:56:31.692 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4000000000000002e-07 2024-02-09T17:56:31.692 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.692 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.692 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:31.692 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.708514+0000", 2024-02-09T17:56:31.693 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9093000000000002e-05 2024-02-09T17:56:31.693 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.693 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.693 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:31.693 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.708519+0000", 2024-02-09T17:56:31.693 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0860000000000001e-06 2024-02-09T17:56:31.693 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.693 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.693 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:31.693 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.708523+0000", 2024-02-09T17:56:31.693 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.692e-06 2024-02-09T17:56:31.693 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.693 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.693 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:31.694 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.708568+0000", 2024-02-09T17:56:31.694 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5349999999999998e-05 2024-02-09T17:56:31.694 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.694 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.694 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:31.694 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.708705+0000", 2024-02-09T17:56:31.694 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013731099999999999 2024-02-09T17:56:31.694 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.694 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.694 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:56:31.694 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.708836+0000", 2024-02-09T17:56:31.694 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013066099999999999 2024-02-09T17:56:31.694 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.695 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.695 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:31.695 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.712199+0000", 2024-02-09T17:56:31.695 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0033626709999999998 2024-02-09T17:56:31.695 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.695 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.695 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:31.695 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.712267+0000", 2024-02-09T17:56:31.695 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8140999999999997e-05 2024-02-09T17:56:31.695 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.695 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.695 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:31.695 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.844652+0000", 2024-02-09T17:56:31.695 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.13238517799999999 2024-02-09T17:56:31.695 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.695 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.696 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:31.696 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.844709+0000", 2024-02-09T17:56:31.696 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6858999999999999e-05 2024-02-09T17:56:31.696 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.696 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.696 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:31.696 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.844748+0000", 2024-02-09T17:56:31.696 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9147000000000003e-05 2024-02-09T17:56:31.696 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:31.696 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:31.696 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:31.696 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.697 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.697 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop_reply(client.26305.0:81702 8.5 e809/785 ondisk, result = 0)", 2024-02-09T17:56:31.697 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:56:12.717332+0000", 2024-02-09T17:56:31.697 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 18.878153521000002, 2024-02-09T17:56:31.697 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.127422851, 2024-02-09T17:56:31.697 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:31.697 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-02-09T17:56:31.697 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:31.697 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.697 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:31.697 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.717332+0000", 2024-02-09T17:56:31.697 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:31.697 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.698 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.698 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:31.698 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.717332+0000", 2024-02-09T17:56:31.698 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:31.698 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.698 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.698 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:31.698 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.717333+0000", 2024-02-09T17:56:31.698 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5700000000000001e-07 2024-02-09T17:56:31.698 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.698 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.698 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:31.698 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.717338+0000", 2024-02-09T17:56:31.698 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1050000000000003e-06 2024-02-09T17:56:31.698 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.699 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.699 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:31.699 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.844571+0000", 2024-02-09T17:56:31.699 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.127232873 2024-02-09T17:56:31.699 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.699 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.699 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:31.699 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.844580+0000", 2024-02-09T17:56:31.699 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.3060000000000003e-06 2024-02-09T17:56:31.699 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.699 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.699 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:31.699 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.844635+0000", 2024-02-09T17:56:31.699 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5009999999999997e-05 2024-02-09T17:56:31.700 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.700 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.700 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:31.700 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.844646+0000", 2024-02-09T17:56:31.700 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0183000000000001e-05 2024-02-09T17:56:31.700 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.700 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.700 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:31.700 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.844755+0000", 2024-02-09T17:56:31.700 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000109517 2024-02-09T17:56:31.700 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:31.700 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:31.700 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:31.700 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.700 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.701 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop_reply(client.26305.0:81697 8.2 e809/785 ondisk, result = 0)", 2024-02-09T17:56:31.701 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:56:12.720214+0000", 2024-02-09T17:56:31.701 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 18.875272118000002, 2024-02-09T17:56:31.701 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.124583517, 2024-02-09T17:56:31.701 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:31.701 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-02-09T17:56:31.701 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:31.701 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.701 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:31.701 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.720214+0000", 2024-02-09T17:56:31.701 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:31.701 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.701 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.701 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:31.702 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.720214+0000", 2024-02-09T17:56:31.702 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:31.702 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.702 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.702 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:31.702 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.720215+0000", 2024-02-09T17:56:31.702 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.235e-06 2024-02-09T17:56:31.702 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.702 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.702 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:31.702 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.720222+0000", 2024-02-09T17:56:31.702 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6470000000000001e-06 2024-02-09T17:56:31.702 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.703 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.703 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:31.703 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.844575+0000", 2024-02-09T17:56:31.703 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.124353221 2024-02-09T17:56:31.703 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.703 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.703 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:31.703 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.844583+0000", 2024-02-09T17:56:31.703 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2470000000000008e-06 2024-02-09T17:56:31.703 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.703 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.703 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:31.703 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.844671+0000", 2024-02-09T17:56:31.703 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7675000000000003e-05 2024-02-09T17:56:31.703 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.704 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.704 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:31.704 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.844679+0000", 2024-02-09T17:56:31.704 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3639999999999999e-06 2024-02-09T17:56:31.704 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.704 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.704 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:31.704 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.844797+0000", 2024-02-09T17:56:31.704 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000118128 2024-02-09T17:56:31.704 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:31.704 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:31.704 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:31.704 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.704 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.705 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.26305.0:81738 8.4 e809/785 8:2163f94d:::benchmark_data_smithi179_148205_object81737:head v 809'5043, mlcod=809'5041)", 2024-02-09T17:56:31.705 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:56:12.737800+0000", 2024-02-09T17:56:31.705 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 18.857685523000001, 2024-02-09T17:56:31.705 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.11111293, 2024-02-09T17:56:31.705 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:31.705 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:31.705 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:31.705 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.705 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:31.705 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.737800+0000", 2024-02-09T17:56:31.705 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:31.705 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.705 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.705 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:31.705 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.737800+0000", 2024-02-09T17:56:31.706 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:31.706 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.706 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.706 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:31.706 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.737802+0000", 2024-02-09T17:56:31.706 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.378e-06 2024-02-09T17:56:31.706 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.706 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.706 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:31.706 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.737830+0000", 2024-02-09T17:56:31.706 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.8411e-05 2024-02-09T17:56:31.706 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.706 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.706 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:31.706 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.844554+0000", 2024-02-09T17:56:31.706 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.106723926 2024-02-09T17:56:31.707 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.707 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.707 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:31.707 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.844569+0000", 2024-02-09T17:56:31.707 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.524e-05 2024-02-09T17:56:31.707 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.707 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.707 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:31.707 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.844635+0000", 2024-02-09T17:56:31.707 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5378999999999999e-05 2024-02-09T17:56:31.707 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.707 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.707 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:31.707 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.844663+0000", 2024-02-09T17:56:31.708 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.8311e-05 2024-02-09T17:56:31.708 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.708 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.708 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:31.708 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.848879+0000", 2024-02-09T17:56:31.708 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0042160019999999999 2024-02-09T17:56:31.708 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:31.708 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:31.708 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:31.708 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.848913+0000", 2024-02-09T17:56:31.708 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4282999999999997e-05 2024-02-09T17:56:31.708 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:31.708 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:31.708 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:31.709 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:31.709 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:31.709 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:56:31.710 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:31 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2639392009' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "8.7", "id": [8, 5, 11, 10, 0, 9]}]: dispatch 2024-02-09T17:56:31.710 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:31 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "8.7", "id": [8, 5, 11, 10, 0, 9]}]: dispatch 2024-02-09T17:56:31.710 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:31.710 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 65 16 102349 102333 98.3684 61.9375 0.010853 0.0101613 2024-02-09T17:56:31.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:31 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2639392009' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "8.7", "id": [8, 5, 11, 10, 0, 9]}]: dispatch 2024-02-09T17:56:31.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:31 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "8.7", "id": [8, 5, 11, 10, 0, 9]}]: dispatch 2024-02-09T17:56:31.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:31 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2639392009' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "8.7", "id": [8, 5, 11, 10, 0, 9]}]: dispatch 2024-02-09T17:56:31.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:31 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "8.7", "id": [8, 5, 11, 10, 0, 9]}]: dispatch 2024-02-09T17:56:32.039 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:56:32.607 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 66 16 103396 103380 97.8695 65.4375 0.0147184 0.0102128 2024-02-09T17:56:32.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:32 smithi179 ceph-mon[43975]: pgmap v2004: 47 pgs: 47 active+clean; 6.0 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.11k op/s 2024-02-09T17:56:32.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:32 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "8.7", "id": [8, 5, 11, 10, 0, 9]}]': finished 2024-02-09T17:56:32.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:32 smithi179 ceph-mon[43975]: osdmap e812: 12 total, 12 up, 12 in 2024-02-09T17:56:32.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:32 smithi162 ceph-mon[44304]: pgmap v2004: 47 pgs: 47 active+clean; 6.0 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.11k op/s 2024-02-09T17:56:32.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:32 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "8.7", "id": [8, 5, 11, 10, 0, 9]}]': finished 2024-02-09T17:56:32.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:32 smithi162 ceph-mon[44304]: osdmap e812: 12 total, 12 up, 12 in 2024-02-09T17:56:32.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:32 smithi099 ceph-mon[39420]: pgmap v2004: 47 pgs: 47 active+clean; 6.0 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.11k op/s 2024-02-09T17:56:32.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:32 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "8.7", "id": [8, 5, 11, 10, 0, 9]}]': finished 2024-02-09T17:56:32.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:32 smithi099 ceph-mon[39420]: osdmap e812: 12 total, 12 up, 12 in 2024-02-09T17:56:32.796 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:56:32.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:32.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:56:32.913 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:56:32.963 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:33.201 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:56:33.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:33.205 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:56:33.253 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.11 dump_ops_in_flight 2024-02-09T17:56:33.307 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:56:33.341 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:33.450 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:56:33.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:33.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:56:33.556 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:56:33.606 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 67 16 104394 104378 97.3397 62.375 0.0089898 0.0102679 2024-02-09T17:56:33.627 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:33.657 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:56:33.659 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:33.660 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:56:33.705 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:56:33.762 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:56:33.796 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:33.817 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:56:33.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:33.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:56:33.921 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:56:33.960 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:33.975 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:56:33.989 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:56:33.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:33.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:56:34.098 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:56:34.128 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:34.173 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:56:34.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:34.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:56:34.280 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:56:34.339 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:34.606 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 68 16 105385 105369 96.8191 61.9375 0.0158256 0.0103236 2024-02-09T17:56:34.613 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:34 smithi099 ceph-mon[39420]: pgmap v2006: 47 pgs: 47 active+clean; 6.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-02-09T17:56:34.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:34 smithi179 ceph-mon[43975]: pgmap v2006: 47 pgs: 47 active+clean; 6.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-02-09T17:56:34.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:34 smithi162 ceph-mon[44304]: pgmap v2006: 47 pgs: 47 active+clean; 6.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-02-09T17:56:35.505 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:56:35.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:35.513 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:56:35.606 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 69 16 106380 106364 96.3173 62.1875 0.016693 0.010378 2024-02-09T17:56:35.615 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:56:35.643 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:35.667 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:56:35.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:35.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:56:35.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:35 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:56:35.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:35 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:56:35.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:35 smithi162 ceph-mon[44304]: osdmap e813: 12 total, 12 up, 12 in 2024-02-09T17:56:35.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:35 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:56:35.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:35 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:56:35.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:35 smithi099 ceph-mon[39420]: osdmap e813: 12 total, 12 up, 12 in 2024-02-09T17:56:35.772 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:56:35.828 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:35.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:35 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:56:35.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:35 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:56:35.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:35 smithi179 ceph-mon[43975]: osdmap e813: 12 total, 12 up, 12 in 2024-02-09T17:56:35.936 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:56:35.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:35.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:56:36.045 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:56:36.106 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:36.147 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:56:36.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:36.157 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:56:36.258 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:56:36.293 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:36.315 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:56:36.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:36.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:56:36.422 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:56:36.452 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:36.460 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is unset 2024-02-09T17:56:36.473 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:56:36.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:36.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:56:36.579 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:56:36.610 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 70 16 107395 107379 95.8425 63.4375 0.0149498 0.0104299 2024-02-09T17:56:36.634 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:36 smithi099 ceph-mon[39420]: pgmap v2008: 47 pgs: 47 active+clean; 6.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 988 op/s 2024-02-09T17:56:36.634 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:36 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/513584221' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:56:36.644 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:36.666 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:56:36.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:36 smithi162 ceph-mon[44304]: pgmap v2008: 47 pgs: 47 active+clean; 6.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 988 op/s 2024-02-09T17:56:36.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:36 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/513584221' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:56:36.699 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:56:36.728 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:56:36.759 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:56:36.790 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:56:36.813 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:56:36.836 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:56:36.857 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:56:36.857 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-02-09T17:56:36.857 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 0 2024-02-09T17:56:36.858 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:56:36.860 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:56:36.880 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:56:36.886 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:36 smithi179 ceph-mon[43975]: pgmap v2008: 47 pgs: 47 active+clean; 6.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 988 op/s 2024-02-09T17:56:36.886 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:36 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/513584221' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:56:36.900 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:56:36.919 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:56:36.940 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:56:36.941 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:56:36.941 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:56:36.941 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:56:36.941 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:56:36.942 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:56:36.942 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:56:36.942 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:56:36.942 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:56:36.942 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:56:36.942 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:56:36.942 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:56:36.942 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:56:37.401 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:56:37.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:37.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:56:37.508 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:56:37.537 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:37.560 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:56:37.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:37.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:56:37.611 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 71 16 108244 108228 95.2384 53.0625 0.0109218 0.010495 2024-02-09T17:56:37.664 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:56:37.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:37 smithi179 ceph-mon[43975]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:56:37.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:37 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/513584221' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:56:37.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:37 smithi179 ceph-mon[43975]: osdmap e814: 12 total, 12 up, 12 in 2024-02-09T17:56:37.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:37 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1763394531' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:56:37.706 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:37.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:37 smithi099 ceph-mon[39420]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:56:37.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:37 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/513584221' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:56:37.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:37 smithi099 ceph-mon[39420]: osdmap e814: 12 total, 12 up, 12 in 2024-02-09T17:56:37.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:37 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1763394531' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:56:37.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:37 smithi162 ceph-mon[44304]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:56:37.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:37 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/513584221' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:56:37.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:37 smithi162 ceph-mon[44304]: osdmap e814: 12 total, 12 up, 12 in 2024-02-09T17:56:37.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:37 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1763394531' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:56:38.611 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 72 16 109273 109257 94.8089 64.3125 0.00487721 0.0105408 2024-02-09T17:56:38.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:38 smithi099 ceph-mon[39420]: pgmap v2010: 47 pgs: 47 active+clean; 6.4 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-02-09T17:56:38.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:38 smithi099 ceph-mon[39420]: 1.d scrub starts 2024-02-09T17:56:38.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:38 smithi099 ceph-mon[39420]: 1.6 scrub starts 2024-02-09T17:56:38.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:38 smithi099 ceph-mon[39420]: 1.6 scrub ok 2024-02-09T17:56:38.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:38 smithi099 ceph-mon[39420]: 1.d scrub ok 2024-02-09T17:56:38.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:38 smithi179 ceph-mon[43975]: pgmap v2010: 47 pgs: 47 active+clean; 6.4 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-02-09T17:56:38.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:38 smithi179 ceph-mon[43975]: 1.d scrub starts 2024-02-09T17:56:38.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:38 smithi179 ceph-mon[43975]: 1.6 scrub starts 2024-02-09T17:56:38.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:38 smithi179 ceph-mon[43975]: 1.6 scrub ok 2024-02-09T17:56:38.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:38 smithi179 ceph-mon[43975]: 1.d scrub ok 2024-02-09T17:56:38.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:38 smithi162 ceph-mon[44304]: pgmap v2010: 47 pgs: 47 active+clean; 6.4 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-02-09T17:56:38.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:38 smithi162 ceph-mon[44304]: 1.d scrub starts 2024-02-09T17:56:38.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:38 smithi162 ceph-mon[44304]: 1.6 scrub starts 2024-02-09T17:56:38.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:38 smithi162 ceph-mon[44304]: 1.6 scrub ok 2024-02-09T17:56:38.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:38 smithi162 ceph-mon[44304]: 1.d scrub ok 2024-02-09T17:56:38.973 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:56:38.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:38.978 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:56:39.008 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.11 dump_blocked_ops 2024-02-09T17:56:39.025 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] 2024-02-09T17:56:39.025 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:56:39.026 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] 2024-02-09T17:56:39.026 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-02-09T17:56:39.026 DEBUG:teuthology.orchestra.run.smithi099:> sudo systemctl stop ceph-a69c870a-c76e-11ee-95b7-87774f69a715@osd.3 2024-02-09T17:56:39.078 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:56:39.138 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:39.171 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:56:39.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:39.179 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:56:39.280 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:56:39.308 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:39.330 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:56:39.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:39.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:56:39.400 INFO:journalctl@ceph.osd.3.smithi099.stdout:Feb 09 17:56:39 smithi099 systemd[1]: Stopping Ceph osd.3 for a69c870a-c76e-11ee-95b7-87774f69a715... 2024-02-09T17:56:39.400 INFO:journalctl@ceph.osd.3.smithi099.stdout:Feb 09 17:56:39 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-3[80449]: 2024-02-09T17:56:39.395+0000 7fe43d3f8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T17:56:39.400 INFO:journalctl@ceph.osd.3.smithi099.stdout:Feb 09 17:56:39 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-3[80449]: 2024-02-09T17:56:39.395+0000 7fe43d3f8700 -1 osd.3 814 *** Got signal Terminated *** 2024-02-09T17:56:39.400 INFO:journalctl@ceph.osd.3.smithi099.stdout:Feb 09 17:56:39 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-3[80449]: 2024-02-09T17:56:39.395+0000 7fe43d3f8700 -1 osd.3 814 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-09T17:56:39.428 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-09T17:56:39.429 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:56:39.433 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:56:39.506 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:39.532 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:56:39.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:39.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:56:39.611 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 73 16 109918 109902 94.0626 40.3125 0.00433527 0.0106105 2024-02-09T17:56:39.636 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:56:39.660 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:39 smithi162 ceph-mon[44304]: 8.a scrub starts 2024-02-09T17:56:39.660 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:39 smithi162 ceph-mon[44304]: 1.3 scrub starts 2024-02-09T17:56:39.660 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:39 smithi162 ceph-mon[44304]: 1.3 scrub ok 2024-02-09T17:56:39.661 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:39 smithi162 ceph-mon[44304]: osd.3 marked itself down and dead 2024-02-09T17:56:39.662 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:39.681 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:56:39.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:39.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:56:39.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:39 smithi179 ceph-mon[43975]: 8.a scrub starts 2024-02-09T17:56:39.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:39 smithi179 ceph-mon[43975]: 1.3 scrub starts 2024-02-09T17:56:39.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:39 smithi179 ceph-mon[43975]: 1.3 scrub ok 2024-02-09T17:56:39.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:39 smithi179 ceph-mon[43975]: osd.3 marked itself down and dead 2024-02-09T17:56:39.720 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:56:39.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:39 smithi099 ceph-mon[39420]: 8.a scrub starts 2024-02-09T17:56:39.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:39 smithi099 ceph-mon[39420]: 1.3 scrub starts 2024-02-09T17:56:39.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:39 smithi099 ceph-mon[39420]: 1.3 scrub ok 2024-02-09T17:56:39.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:39 smithi099 ceph-mon[39420]: osd.3 marked itself down and dead 2024-02-09T17:56:39.785 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:56:39.811 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:39.831 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:56:39.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:39.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:56:39.935 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:56:39.965 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:39.988 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:56:39.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:39.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:56:40.092 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:56:40.120 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:40.165 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:56:40.318 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:56:40.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:40.322 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:56:40.423 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:56:40.450 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:40.611 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 74 15 110047 110032 92.9015 8.125 0.0199876 0.0107594 2024-02-09T17:56:40.639 INFO:journalctl@ceph.osd.3.smithi099.stdout:Feb 09 17:56:40 smithi099 bash[265084]: ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-3 2024-02-09T17:56:40.643 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:40 smithi099 ceph-mon[39420]: pgmap v2011: 47 pgs: 47 active+clean; 6.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-02-09T17:56:40.643 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:40 smithi099 ceph-mon[39420]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-09T17:56:40.643 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:40 smithi099 ceph-mon[39420]: osdmap e815: 12 total, 11 up, 12 in 2024-02-09T17:56:40.643 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:40 smithi099 ceph-mon[39420]: 8.a scrub ok 2024-02-09T17:56:40.643 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:40 smithi099 ceph-mon[39420]: 1.14 scrub starts 2024-02-09T17:56:40.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:40 smithi179 ceph-mon[43975]: pgmap v2011: 47 pgs: 47 active+clean; 6.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-02-09T17:56:40.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:40 smithi179 ceph-mon[43975]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-09T17:56:40.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:40 smithi179 ceph-mon[43975]: osdmap e815: 12 total, 11 up, 12 in 2024-02-09T17:56:40.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:40 smithi179 ceph-mon[43975]: 8.a scrub ok 2024-02-09T17:56:40.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:40 smithi179 ceph-mon[43975]: 1.14 scrub starts 2024-02-09T17:56:40.910 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:56:40.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:40.923 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:56:40.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:40 smithi162 ceph-mon[44304]: pgmap v2011: 47 pgs: 47 active+clean; 6.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-02-09T17:56:40.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:40 smithi162 ceph-mon[44304]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-09T17:56:40.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:40 smithi162 ceph-mon[44304]: osdmap e815: 12 total, 11 up, 12 in 2024-02-09T17:56:40.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:40 smithi162 ceph-mon[44304]: 8.a scrub ok 2024-02-09T17:56:40.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:40 smithi162 ceph-mon[44304]: 1.14 scrub starts 2024-02-09T17:56:41.023 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:56:41.051 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:41.073 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:56:41.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:41.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:56:41.179 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:56:41.208 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:41.232 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:56:41.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:41.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:56:41.336 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:56:41.393 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:41.460 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:56:41.461 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:41 smithi099 ceph-mon[39420]: 1.1a scrub starts 2024-02-09T17:56:41.461 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:41 smithi099 ceph-mon[39420]: 1.1a scrub ok 2024-02-09T17:56:41.461 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:41 smithi099 ceph-mon[39420]: 1.14 scrub ok 2024-02-09T17:56:41.461 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:41 smithi099 ceph-mon[39420]: 1.f scrub starts 2024-02-09T17:56:41.461 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:41 smithi099 ceph-mon[39420]: 1.f scrub ok 2024-02-09T17:56:41.461 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:41 smithi099 ceph-mon[39420]: 1.5 scrub starts 2024-02-09T17:56:41.461 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:41 smithi099 ceph-mon[39420]: osdmap e816: 12 total, 11 up, 12 in 2024-02-09T17:56:41.461 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:41 smithi099 ceph-mon[39420]: 1.5 scrub ok 2024-02-09T17:56:41.508 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:56:41.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:41.511 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:56:41.611 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 75 16 111014 110998 92.4678 60.375 0.0124666 0.0108101 2024-02-09T17:56:41.611 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:56:41.667 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:41.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:41 smithi179 ceph-mon[43975]: 1.1a scrub starts 2024-02-09T17:56:41.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:41 smithi179 ceph-mon[43975]: 1.1a scrub ok 2024-02-09T17:56:41.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:41 smithi179 ceph-mon[43975]: 1.14 scrub ok 2024-02-09T17:56:41.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:41 smithi179 ceph-mon[43975]: 1.f scrub starts 2024-02-09T17:56:41.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:41 smithi179 ceph-mon[43975]: 1.f scrub ok 2024-02-09T17:56:41.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:41 smithi179 ceph-mon[43975]: 1.5 scrub starts 2024-02-09T17:56:41.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:41 smithi179 ceph-mon[43975]: osdmap e816: 12 total, 11 up, 12 in 2024-02-09T17:56:41.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:41 smithi179 ceph-mon[43975]: 1.5 scrub ok 2024-02-09T17:56:41.821 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T17:56:41.822 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:56:41.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:41 smithi162 ceph-mon[44304]: 1.1a scrub starts 2024-02-09T17:56:41.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:41 smithi162 ceph-mon[44304]: 1.1a scrub ok 2024-02-09T17:56:41.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:41 smithi162 ceph-mon[44304]: 1.14 scrub ok 2024-02-09T17:56:41.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:41 smithi162 ceph-mon[44304]: 1.f scrub starts 2024-02-09T17:56:41.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:41 smithi162 ceph-mon[44304]: 1.f scrub ok 2024-02-09T17:56:41.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:41 smithi162 ceph-mon[44304]: 1.5 scrub starts 2024-02-09T17:56:41.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:41 smithi162 ceph-mon[44304]: osdmap e816: 12 total, 11 up, 12 in 2024-02-09T17:56:41.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:41 smithi162 ceph-mon[44304]: 1.5 scrub ok 2024-02-09T17:56:42.222 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:56:42.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:42.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:56:42.329 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:56:42.360 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:42.383 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:56:42.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:42.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:56:42.487 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:56:42.489 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:42 smithi099 ceph-mon[39420]: pgmap v2014: 47 pgs: 1 activating+undersized, 1 active+undersized, 1 peering, 2 stale+active+clean, 42 active+clean; 6.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 891 op/s 2024-02-09T17:56:42.489 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:42 smithi099 ceph-mon[39420]: 1.2 scrub starts 2024-02-09T17:56:42.489 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:42 smithi099 ceph-mon[39420]: 1.2 scrub ok 2024-02-09T17:56:42.489 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:42 smithi099 ceph-mon[39420]: 8.6 scrub starts 2024-02-09T17:56:42.489 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:42 smithi099 ceph-mon[39420]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-09T17:56:42.490 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:42 smithi099 ceph-mon[39420]: 1.13 scrub starts 2024-02-09T17:56:42.490 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:42 smithi099 ceph-mon[39420]: 1.13 scrub ok 2024-02-09T17:56:42.550 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:42.612 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 76 15 111812 111797 91.9083 49.9375 0.0180185 0.0108749 2024-02-09T17:56:42.630 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:56:42.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:42.634 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:56:42.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:42 smithi162 ceph-mon[44304]: pgmap v2014: 47 pgs: 1 activating+undersized, 1 active+undersized, 1 peering, 2 stale+active+clean, 42 active+clean; 6.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 891 op/s 2024-02-09T17:56:42.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:42 smithi162 ceph-mon[44304]: 1.2 scrub starts 2024-02-09T17:56:42.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:42 smithi162 ceph-mon[44304]: 1.2 scrub ok 2024-02-09T17:56:42.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:42 smithi162 ceph-mon[44304]: 8.6 scrub starts 2024-02-09T17:56:42.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:42 smithi162 ceph-mon[44304]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-09T17:56:42.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:42 smithi162 ceph-mon[44304]: 1.13 scrub starts 2024-02-09T17:56:42.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:42 smithi162 ceph-mon[44304]: 1.13 scrub ok 2024-02-09T17:56:42.735 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:56:42.764 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:42.786 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:56:42.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:42.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:56:42.891 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:56:42.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:42 smithi179 ceph-mon[43975]: pgmap v2014: 47 pgs: 1 activating+undersized, 1 active+undersized, 1 peering, 2 stale+active+clean, 42 active+clean; 6.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 891 op/s 2024-02-09T17:56:42.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:42 smithi179 ceph-mon[43975]: 1.2 scrub starts 2024-02-09T17:56:42.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:42 smithi179 ceph-mon[43975]: 1.2 scrub ok 2024-02-09T17:56:42.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:42 smithi179 ceph-mon[43975]: 8.6 scrub starts 2024-02-09T17:56:42.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:42 smithi179 ceph-mon[43975]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-09T17:56:42.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:42 smithi179 ceph-mon[43975]: 1.13 scrub starts 2024-02-09T17:56:42.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:42 smithi179 ceph-mon[43975]: 1.13 scrub ok 2024-02-09T17:56:42.930 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:42.957 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:56:42.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:42.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:56:43.065 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:56:43.097 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:43.611 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 77 15 112641 112626 91.3877 51.8125 0.0120143 0.0109382 2024-02-09T17:56:43.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:43 smithi099 ceph-mon[39420]: 1.18 scrub starts 2024-02-09T17:56:43.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:43 smithi099 ceph-mon[39420]: 1.18 scrub ok 2024-02-09T17:56:43.860 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:43 smithi179 ceph-mon[43975]: 1.18 scrub starts 2024-02-09T17:56:43.860 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:43 smithi179 ceph-mon[43975]: 1.18 scrub ok 2024-02-09T17:56:43.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:43 smithi162 ceph-mon[44304]: 1.18 scrub starts 2024-02-09T17:56:43.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:43 smithi162 ceph-mon[44304]: 1.18 scrub ok 2024-02-09T17:56:44.257 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:56:44.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:44.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:56:44.363 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:56:44.421 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:44.488 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is unset 2024-02-09T17:56:44.516 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:44 smithi099 ceph-mon[39420]: pgmap v2015: 47 pgs: 3 active+undersized+degraded, 1 activating+undersized, 3 active+undersized, 1 peering, 39 active+clean; 6.7 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 840 op/s; 21061/331647 objects degraded (6.350%) 2024-02-09T17:56:44.516 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:44 smithi099 ceph-mon[39420]: 8.f scrub starts 2024-02-09T17:56:44.516 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:44 smithi099 ceph-mon[39420]: 8.6 scrub ok 2024-02-09T17:56:44.516 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:44 smithi099 ceph-mon[39420]: Health check failed: Degraded data redundancy: 21061/331647 objects degraded (6.350%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:56:44.516 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:44 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/150383416' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:56:44.611 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 78 16 113471 113455 90.8804 51.8125 0.00942099 0.0109989 2024-02-09T17:56:44.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:44 smithi179 ceph-mon[43975]: pgmap v2015: 47 pgs: 3 active+undersized+degraded, 1 activating+undersized, 3 active+undersized, 1 peering, 39 active+clean; 6.7 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 840 op/s; 21061/331647 objects degraded (6.350%) 2024-02-09T17:56:44.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:44 smithi179 ceph-mon[43975]: 8.f scrub starts 2024-02-09T17:56:44.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:44 smithi179 ceph-mon[43975]: 8.6 scrub ok 2024-02-09T17:56:44.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:44 smithi179 ceph-mon[43975]: Health check failed: Degraded data redundancy: 21061/331647 objects degraded (6.350%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:56:44.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:44 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/150383416' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:56:44.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:44 smithi162 ceph-mon[44304]: pgmap v2015: 47 pgs: 3 active+undersized+degraded, 1 activating+undersized, 3 active+undersized, 1 peering, 39 active+clean; 6.7 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 840 op/s; 21061/331647 objects degraded (6.350%) 2024-02-09T17:56:44.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:44 smithi162 ceph-mon[44304]: 8.f scrub starts 2024-02-09T17:56:44.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:44 smithi162 ceph-mon[44304]: 8.6 scrub ok 2024-02-09T17:56:44.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:44 smithi162 ceph-mon[44304]: Health check failed: Degraded data redundancy: 21061/331647 objects degraded (6.350%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:56:44.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:44 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/150383416' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:56:44.965 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:56:44.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:44.979 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:56:44.996 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.11 dump_historic_ops 2024-02-09T17:56:45.079 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:56:45.134 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:45.177 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:56:45.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:45.184 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:56:45.260 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:56:45.285 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:56:45.306 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:56:45.314 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:45.338 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:56:45.338 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:56:45.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:45.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:56:45.374 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:56:45.408 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:56:45.445 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:56:45.454 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:56:45.491 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:56:45.510 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:45.510 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:56:45.539 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:56:45.572 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:56:45.610 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:56:45.611 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 79 16 114145 114129 90.2633 42.125 0.0209936 0.011074 2024-02-09T17:56:45.639 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:56:45.639 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:56:45.639 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:56:45.639 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:56:45.640 INFO:teuthology.orchestra.run.smithi099.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-09T17:56:45.640 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:56:45.640 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:56:45.640 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:56:45.640 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:56:45.640 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:56:45.640 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:56:45.640 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:56:45.640 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:56:45.657 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:45 smithi099 ceph-mon[39420]: 8.e scrub starts 2024-02-09T17:56:45.657 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:45 smithi099 ceph-mon[39420]: 8.2 scrub starts 2024-02-09T17:56:45.658 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:45 smithi099 ceph-mon[39420]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:56:45.658 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:45 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/150383416' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:56:45.658 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:45 smithi099 ceph-mon[39420]: osdmap e817: 12 total, 11 up, 12 in 2024-02-09T17:56:45.658 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:45 smithi099 ceph-mon[39420]: 8.1 scrub starts 2024-02-09T17:56:45.658 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:45 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2866851244' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:56:45.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:45 smithi162 ceph-mon[44304]: 8.e scrub starts 2024-02-09T17:56:45.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:45 smithi162 ceph-mon[44304]: 8.2 scrub starts 2024-02-09T17:56:45.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:45 smithi162 ceph-mon[44304]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:56:45.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:45 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/150383416' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:56:45.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:45 smithi162 ceph-mon[44304]: osdmap e817: 12 total, 11 up, 12 in 2024-02-09T17:56:45.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:45 smithi162 ceph-mon[44304]: 8.1 scrub starts 2024-02-09T17:56:45.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:45 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2866851244' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:56:45.818 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:45 smithi179 ceph-mon[43975]: 8.e scrub starts 2024-02-09T17:56:45.818 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:45 smithi179 ceph-mon[43975]: 8.2 scrub starts 2024-02-09T17:56:45.818 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:45 smithi179 ceph-mon[43975]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:56:45.818 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:45 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/150383416' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:56:45.818 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:45 smithi179 ceph-mon[43975]: osdmap e817: 12 total, 11 up, 12 in 2024-02-09T17:56:45.818 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:45 smithi179 ceph-mon[43975]: 8.1 scrub starts 2024-02-09T17:56:45.818 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:45 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2866851244' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:56:46.612 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:2024-02-09T17:56:46.612157+0000 min lat: 0.00109523 max lat: 1.06242 avg lat: 0.0111269 2024-02-09T17:56:46.613 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-09T17:56:46.613 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 80 15 115038 115023 89.8324 55.875 0.0110627 0.0111269 2024-02-09T17:56:46.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:46 smithi099 ceph-mon[39420]: 1.a deep-scrub starts 2024-02-09T17:56:46.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:46 smithi099 ceph-mon[39420]: 1.a deep-scrub ok 2024-02-09T17:56:46.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:46 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:56:46.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:46 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:56:46.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:46 smithi099 ceph-mon[39420]: pgmap v2017: 47 pgs: 3 active+undersized+degraded, 1 activating+undersized, 4 active+undersized, 39 active+clean; 6.8 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 570 op/s; 21061/334146 objects degraded (6.303%) 2024-02-09T17:56:46.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:46 smithi099 ceph-mon[39420]: 1.11 scrub starts 2024-02-09T17:56:46.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:46 smithi099 ceph-mon[39420]: 1.4 deep-scrub starts 2024-02-09T17:56:46.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:46 smithi099 ceph-mon[39420]: 1.4 deep-scrub ok 2024-02-09T17:56:46.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:46 smithi099 ceph-mon[39420]: 1.11 scrub ok 2024-02-09T17:56:46.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:46 smithi099 ceph-mon[39420]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-09T17:56:46.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:46 smithi099 ceph-mon[39420]: osdmap e818: 12 total, 11 up, 12 in 2024-02-09T17:56:46.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:46 smithi099 ceph-mon[39420]: 8.f scrub ok 2024-02-09T17:56:46.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:46 smithi099 ceph-mon[39420]: 1.1d scrub starts 2024-02-09T17:56:46.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:46 smithi099 ceph-mon[39420]: 1.1d scrub ok 2024-02-09T17:56:46.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:46 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:56:46.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:46 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:56:46.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:46 smithi099 ceph-mon[39420]: 8.2 scrub ok 2024-02-09T17:56:46.823 DEBUG:teuthology.orchestra.run.smithi099:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a69c870a-c76e-11ee-95b7-87774f69a715@osd.3.service' 2024-02-09T17:56:46.926 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:46 smithi179 ceph-mon[43975]: 1.a deep-scrub starts 2024-02-09T17:56:46.926 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:46 smithi179 ceph-mon[43975]: 1.a deep-scrub ok 2024-02-09T17:56:46.926 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:46 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:56:46.926 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:46 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:56:46.926 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:46 smithi179 ceph-mon[43975]: pgmap v2017: 47 pgs: 3 active+undersized+degraded, 1 activating+undersized, 4 active+undersized, 39 active+clean; 6.8 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 570 op/s; 21061/334146 objects degraded (6.303%) 2024-02-09T17:56:46.926 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:46 smithi179 ceph-mon[43975]: 1.11 scrub starts 2024-02-09T17:56:46.926 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:46 smithi179 ceph-mon[43975]: 1.4 deep-scrub starts 2024-02-09T17:56:46.926 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:46 smithi179 ceph-mon[43975]: 1.4 deep-scrub ok 2024-02-09T17:56:46.926 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:46 smithi179 ceph-mon[43975]: 1.11 scrub ok 2024-02-09T17:56:46.927 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:46 smithi179 ceph-mon[43975]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-09T17:56:46.927 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:46 smithi179 ceph-mon[43975]: osdmap e818: 12 total, 11 up, 12 in 2024-02-09T17:56:46.927 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:46 smithi179 ceph-mon[43975]: 8.f scrub ok 2024-02-09T17:56:46.927 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:46 smithi179 ceph-mon[43975]: 1.1d scrub starts 2024-02-09T17:56:46.927 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:46 smithi179 ceph-mon[43975]: 1.1d scrub ok 2024-02-09T17:56:46.927 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:46 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:56:46.927 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:46 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:56:46.927 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:46 smithi179 ceph-mon[43975]: 8.2 scrub ok 2024-02-09T17:56:46.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:46 smithi162 ceph-mon[44304]: 1.a deep-scrub starts 2024-02-09T17:56:46.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:46 smithi162 ceph-mon[44304]: 1.a deep-scrub ok 2024-02-09T17:56:46.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:46 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:56:46.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:46 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:56:46.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:46 smithi162 ceph-mon[44304]: pgmap v2017: 47 pgs: 3 active+undersized+degraded, 1 activating+undersized, 4 active+undersized, 39 active+clean; 6.8 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 570 op/s; 21061/334146 objects degraded (6.303%) 2024-02-09T17:56:46.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:46 smithi162 ceph-mon[44304]: 1.11 scrub starts 2024-02-09T17:56:46.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:46 smithi162 ceph-mon[44304]: 1.4 deep-scrub starts 2024-02-09T17:56:46.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:46 smithi162 ceph-mon[44304]: 1.4 deep-scrub ok 2024-02-09T17:56:46.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:46 smithi162 ceph-mon[44304]: 1.11 scrub ok 2024-02-09T17:56:46.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:46 smithi162 ceph-mon[44304]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-09T17:56:46.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:46 smithi162 ceph-mon[44304]: osdmap e818: 12 total, 11 up, 12 in 2024-02-09T17:56:46.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:46 smithi162 ceph-mon[44304]: 8.f scrub ok 2024-02-09T17:56:46.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:46 smithi162 ceph-mon[44304]: 1.1d scrub starts 2024-02-09T17:56:46.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:46 smithi162 ceph-mon[44304]: 1.1d scrub ok 2024-02-09T17:56:46.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:46 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:56:46.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:46 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:56:46.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:46 smithi162 ceph-mon[44304]: 8.2 scrub ok 2024-02-09T17:56:46.955 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T17:56:46.955 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-02-09T17:56:46.955 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.3 2024-02-09T17:56:46.955 INFO:tasks.thrashosds.thrasher:remote for osd 3 is ubuntu@smithi099.front.sepia.ceph.com 2024-02-09T17:56:46.955 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell -n osd.3 --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --op list-pgs 2024-02-09T17:56:47.092 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:56:47.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:47.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:56:47.198 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:56:47.226 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:47.262 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:56:47.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:47.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:56:47.368 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:56:47.407 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:47.521 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:47 smithi162 ceph-mon[44304]: 8.e scrub ok 2024-02-09T17:56:47.522 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:47 smithi162 ceph-mon[44304]: 8.1 scrub ok 2024-02-09T17:56:47.522 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:47 smithi162 ceph-mon[44304]: 1.0 scrub starts 2024-02-09T17:56:47.522 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:47 smithi162 ceph-mon[44304]: 1.0 scrub ok 2024-02-09T17:56:47.522 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:47 smithi162 ceph-mon[44304]: osdmap e819: 12 total, 11 up, 12 in 2024-02-09T17:56:47.613 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 81 16 115985 115969 89.4523 59.125 0.0267808 0.0111745 2024-02-09T17:56:47.662 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:56:47.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:47.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:56:47.666 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:47 smithi099 ceph-mon[39420]: 8.e scrub ok 2024-02-09T17:56:47.667 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:47 smithi099 ceph-mon[39420]: 8.1 scrub ok 2024-02-09T17:56:47.667 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:47 smithi099 ceph-mon[39420]: 1.0 scrub starts 2024-02-09T17:56:47.667 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:47 smithi099 ceph-mon[39420]: 1.0 scrub ok 2024-02-09T17:56:47.667 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:47 smithi099 ceph-mon[39420]: osdmap e819: 12 total, 11 up, 12 in 2024-02-09T17:56:47.670 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-09T17:56:47.670 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-09T17:56:47.699 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/osd.3/config 2024-02-09T17:56:47.766 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:56:47.794 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:47.816 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:56:47.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:47.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:56:47.921 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:56:47.926 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:47 smithi179 ceph-mon[43975]: 8.e scrub ok 2024-02-09T17:56:47.926 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:47 smithi179 ceph-mon[43975]: 8.1 scrub ok 2024-02-09T17:56:47.926 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:47 smithi179 ceph-mon[43975]: 1.0 scrub starts 2024-02-09T17:56:47.926 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:47 smithi179 ceph-mon[43975]: 1.0 scrub ok 2024-02-09T17:56:47.926 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:47 smithi179 ceph-mon[43975]: osdmap e819: 12 total, 11 up, 12 in 2024-02-09T17:56:47.989 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:48.015 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:56:48.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:48.020 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:56:48.115 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:56:48.116 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-02-09T17:56:48.116 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-02-09T17:56:48.116 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-02-09T17:56:48.116 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.116 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25837.0:212109 6.8 6:12d751a2:::benchmark_data_smithi179_126206_object81728:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e641)", 2024-02-09T17:56:48.116 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:48:58.903495+0000", 2024-02-09T17:56:48.116 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 469.20339620099998, 2024-02-09T17:56:48.116 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14948397099999999, 2024-02-09T17:56:48.116 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:48.116 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:48.116 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:48.116 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25837", 2024-02-09T17:56:48.117 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:56:48.117 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 212109 2024-02-09T17:56:48.117 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.117 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:48.117 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.117 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:48.117 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:58.903495+0000", 2024-02-09T17:56:48.117 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:48.117 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.117 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.117 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:48.117 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:58.903495+0000", 2024-02-09T17:56:48.117 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:48.117 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.117 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.118 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:48.118 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:58.903497+0000", 2024-02-09T17:56:48.118 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5900000000000002e-06 2024-02-09T17:56:48.118 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.118 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.118 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:48.118 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:58.903509+0000", 2024-02-09T17:56:48.118 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1603000000000001e-05 2024-02-09T17:56:48.118 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.118 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.118 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:48.118 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:58.903540+0000", 2024-02-09T17:56:48.118 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0956e-05 2024-02-09T17:56:48.119 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.119 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.119 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:48.119 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:58.903558+0000", 2024-02-09T17:56:48.119 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8257000000000002e-05 2024-02-09T17:56:48.119 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.119 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.119 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:48.119 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:58.903671+0000", 2024-02-09T17:56:48.119 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011299 2024-02-09T17:56:48.119 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.119 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.119 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:48.119 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:58.903788+0000", 2024-02-09T17:56:48.120 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000116396 2024-02-09T17:56:48.120 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.120 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.120 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,7", 2024-02-09T17:56:48.120 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:58.903865+0000", 2024-02-09T17:56:48.120 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7911000000000007e-05 2024-02-09T17:56:48.120 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.120 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.120 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:48.120 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:58.907956+0000", 2024-02-09T17:56:48.120 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0040901039999999998 2024-02-09T17:56:48.120 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.120 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.120 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:48.121 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:58.908920+0000", 2024-02-09T17:56:48.121 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000964233 2024-02-09T17:56:48.121 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.121 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.121 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:48.121 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:59.052875+0000", 2024-02-09T17:56:48.121 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14395539099999999 2024-02-09T17:56:48.121 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.121 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.121 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:48.121 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:59.052938+0000", 2024-02-09T17:56:48.121 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3196999999999998e-05 2024-02-09T17:56:48.121 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.121 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.121 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:48.122 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:48:59.052979+0000", 2024-02-09T17:56:48.122 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0343000000000003e-05 2024-02-09T17:56:48.122 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:48.122 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:48.122 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:48.122 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.122 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.122 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25837.0:256618 6.8 6:10514f6e:::benchmark_data_smithi179_126206_object126237:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:56:48.122 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:49:55.307984+0000", 2024-02-09T17:56:48.122 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 412.79890703199999, 2024-02-09T17:56:48.122 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.16505310500000001, 2024-02-09T17:56:48.122 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:48.122 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:48.122 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:48.123 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25837", 2024-02-09T17:56:48.123 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:56:48.123 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 256618 2024-02-09T17:56:48.123 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.123 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:48.123 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.123 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:48.123 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.307984+0000", 2024-02-09T17:56:48.123 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:48.123 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.123 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.123 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:48.123 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.307984+0000", 2024-02-09T17:56:48.123 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:48.123 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.124 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.124 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:48.124 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.307985+0000", 2024-02-09T17:56:48.124 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1319999999999999e-06 2024-02-09T17:56:48.124 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.124 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.124 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:48.124 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.307990+0000", 2024-02-09T17:56:48.124 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5870000000000003e-06 2024-02-09T17:56:48.124 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.124 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.124 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:48.124 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.308003+0000", 2024-02-09T17:56:48.125 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3020000000000001e-05 2024-02-09T17:56:48.125 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.125 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.125 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:48.125 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.308010+0000", 2024-02-09T17:56:48.125 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6429999999999995e-06 2024-02-09T17:56:48.125 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.125 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.125 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:48.125 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.308054+0000", 2024-02-09T17:56:48.125 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3430000000000003e-05 2024-02-09T17:56:48.125 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.125 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.125 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:48.126 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.308162+0000", 2024-02-09T17:56:48.126 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000107821 2024-02-09T17:56:48.126 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.126 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.126 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,7", 2024-02-09T17:56:48.126 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.308229+0000", 2024-02-09T17:56:48.126 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7150999999999995e-05 2024-02-09T17:56:48.126 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.126 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.126 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:48.126 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.312154+0000", 2024-02-09T17:56:48.126 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0039250910000000003 2024-02-09T17:56:48.126 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.126 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.126 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:48.127 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.317879+0000", 2024-02-09T17:56:48.127 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0057249930000000003 2024-02-09T17:56:48.127 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.127 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.127 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:48.127 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.472974+0000", 2024-02-09T17:56:48.127 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.155095488 2024-02-09T17:56:48.127 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.127 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.127 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:48.127 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.473011+0000", 2024-02-09T17:56:48.127 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6575000000000001e-05 2024-02-09T17:56:48.127 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.127 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.127 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:48.128 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.473037+0000", 2024-02-09T17:56:48.128 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6174e-05 2024-02-09T17:56:48.128 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:48.128 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:48.128 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:48.128 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.128 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.128 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25837.0:256647 6.8 6:100d7332:::benchmark_data_smithi179_126206_object126266:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:56:48.128 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:49:55.338986+0000", 2024-02-09T17:56:48.128 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 412.767904794, 2024-02-09T17:56:48.128 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.140859136, 2024-02-09T17:56:48.128 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:48.128 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:48.128 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:48.128 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25837", 2024-02-09T17:56:48.129 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:56:48.129 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 256647 2024-02-09T17:56:48.129 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.129 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:48.129 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.129 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:48.129 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.338986+0000", 2024-02-09T17:56:48.129 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:48.129 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.129 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.129 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:48.129 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.338986+0000", 2024-02-09T17:56:48.129 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:48.129 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.130 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.130 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:48.130 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.338988+0000", 2024-02-09T17:56:48.130 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6440000000000001e-06 2024-02-09T17:56:48.130 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.130 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.130 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:48.130 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.338995+0000", 2024-02-09T17:56:48.130 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7909999999999999e-06 2024-02-09T17:56:48.130 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.130 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.130 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:48.130 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.339009+0000", 2024-02-09T17:56:48.130 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4881e-05 2024-02-09T17:56:48.130 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.131 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.131 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:48.131 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.339018+0000", 2024-02-09T17:56:48.131 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3049999999999993e-06 2024-02-09T17:56:48.131 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.131 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.131 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:48.131 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.339100+0000", 2024-02-09T17:56:48.131 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2579999999999999e-05 2024-02-09T17:56:48.131 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.131 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.131 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:48.131 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.339204+0000", 2024-02-09T17:56:48.131 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010393800000000001 2024-02-09T17:56:48.131 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.131 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.132 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,7", 2024-02-09T17:56:48.132 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.339281+0000", 2024-02-09T17:56:48.132 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.716e-05 2024-02-09T17:56:48.132 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.132 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.132 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:48.132 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.343222+0000", 2024-02-09T17:56:48.132 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0039405159999999998 2024-02-09T17:56:48.132 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.132 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.132 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:48.132 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.343582+0000", 2024-02-09T17:56:48.132 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000359599 2024-02-09T17:56:48.132 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.133 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.133 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:48.133 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.479787+0000", 2024-02-09T17:56:48.133 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.13620548399999999 2024-02-09T17:56:48.133 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.133 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.133 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:48.133 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.479822+0000", 2024-02-09T17:56:48.133 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4641999999999998e-05 2024-02-09T17:56:48.133 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.133 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.133 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:48.133 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:49:55.479845+0000", 2024-02-09T17:56:48.133 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3595999999999999e-05 2024-02-09T17:56:48.134 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:48.134 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:48.134 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:48.134 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.134 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.134 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.16440.0:61006 7.5 7:ad455a63:::benchmark_data_smithi179_135804_object61005:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e674)", 2024-02-09T17:56:48.134 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:50:55.020008+0000", 2024-02-09T17:56:48.134 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 353.08688287899997, 2024-02-09T17:56:48.134 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.196220321, 2024-02-09T17:56:48.134 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:48.134 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:48.134 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:48.134 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.16440", 2024-02-09T17:56:48.134 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:56:48.134 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 61006 2024-02-09T17:56:48.135 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.135 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:48.135 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.135 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:48.135 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.020008+0000", 2024-02-09T17:56:48.135 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:48.135 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.135 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.135 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:48.135 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.020008+0000", 2024-02-09T17:56:48.135 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:48.135 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.135 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.135 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:48.135 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.020010+0000", 2024-02-09T17:56:48.136 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.885e-06 2024-02-09T17:56:48.136 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.136 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.136 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:48.136 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.020036+0000", 2024-02-09T17:56:48.136 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.561e-05 2024-02-09T17:56:48.136 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.136 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.136 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:48.136 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.020050+0000", 2024-02-09T17:56:48.136 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4465e-05 2024-02-09T17:56:48.136 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.136 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.136 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:48.136 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.020058+0000", 2024-02-09T17:56:48.137 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4270000000000008e-06 2024-02-09T17:56:48.137 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.137 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.137 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:48.137 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.020116+0000", 2024-02-09T17:56:48.137 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7695999999999998e-05 2024-02-09T17:56:48.137 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.137 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.137 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:48.137 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.020257+0000", 2024-02-09T17:56:48.137 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014065400000000001 2024-02-09T17:56:48.137 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.137 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.137 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,9", 2024-02-09T17:56:48.138 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.020354+0000", 2024-02-09T17:56:48.138 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7031e-05 2024-02-09T17:56:48.138 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.138 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.138 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:48.138 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.022998+0000", 2024-02-09T17:56:48.138 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0026436900000000002 2024-02-09T17:56:48.138 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.138 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.138 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:48.138 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.023073+0000", 2024-02-09T17:56:48.138 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5365999999999996e-05 2024-02-09T17:56:48.138 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.138 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.138 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:48.139 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.216125+0000", 2024-02-09T17:56:48.139 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.193051797 2024-02-09T17:56:48.139 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.139 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.139 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:48.139 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.216189+0000", 2024-02-09T17:56:48.139 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3959000000000002e-05 2024-02-09T17:56:48.139 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.139 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.139 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:48.139 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.216228+0000", 2024-02-09T17:56:48.139 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9740999999999998e-05 2024-02-09T17:56:48.139 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:48.139 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:48.139 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:48.140 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.140 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.140 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.16440.0:61016 7.5 7:a0e287c6:::benchmark_data_smithi179_135804_object61015:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e674)", 2024-02-09T17:56:48.140 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:50:55.027713+0000", 2024-02-09T17:56:48.140 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 353.07917751500003, 2024-02-09T17:56:48.140 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.188635202, 2024-02-09T17:56:48.140 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:48.140 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:48.140 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:48.140 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.16440", 2024-02-09T17:56:48.140 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:56:48.140 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 61016 2024-02-09T17:56:48.140 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.140 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:48.140 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.141 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:48.141 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.027713+0000", 2024-02-09T17:56:48.141 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:48.141 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.141 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.141 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:48.141 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.027713+0000", 2024-02-09T17:56:48.141 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:48.141 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.141 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.141 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:48.141 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.027715+0000", 2024-02-09T17:56:48.141 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.668e-06 2024-02-09T17:56:48.141 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.141 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.141 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:48.142 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.027736+0000", 2024-02-09T17:56:48.142 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0661000000000001e-05 2024-02-09T17:56:48.142 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.142 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.142 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:48.142 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.027747+0000", 2024-02-09T17:56:48.142 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1085e-05 2024-02-09T17:56:48.142 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.142 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.142 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:48.142 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.027754+0000", 2024-02-09T17:56:48.142 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7920000000000004e-06 2024-02-09T17:56:48.142 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.142 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.143 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:48.143 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.027798+0000", 2024-02-09T17:56:48.143 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3971000000000003e-05 2024-02-09T17:56:48.143 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.143 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.143 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:48.143 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.027895+0000", 2024-02-09T17:56:48.143 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7120999999999994e-05 2024-02-09T17:56:48.143 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.143 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.143 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,9", 2024-02-09T17:56:48.143 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.028002+0000", 2024-02-09T17:56:48.143 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000106792 2024-02-09T17:56:48.143 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.144 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.144 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:48.144 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.030226+0000", 2024-02-09T17:56:48.144 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0022244550000000002 2024-02-09T17:56:48.144 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.144 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.144 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:48.144 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.030810+0000", 2024-02-09T17:56:48.144 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000583608 2024-02-09T17:56:48.144 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.144 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.144 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:48.144 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.216269+0000", 2024-02-09T17:56:48.144 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.18545963100000001 2024-02-09T17:56:48.144 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.145 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.145 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:48.145 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.216316+0000", 2024-02-09T17:56:48.145 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6542000000000003e-05 2024-02-09T17:56:48.145 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.145 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.145 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:48.145 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.216349+0000", 2024-02-09T17:56:48.145 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2876000000000002e-05 2024-02-09T17:56:48.145 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:48.145 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:48.145 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:48.145 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.145 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.145 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.16440.0:61018 7.5 7:aabe8c5f:::benchmark_data_smithi179_135804_object61017:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e674)", 2024-02-09T17:56:48.146 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:50:55.030684+0000", 2024-02-09T17:56:48.146 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 353.07620709499997, 2024-02-09T17:56:48.146 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.185782171, 2024-02-09T17:56:48.146 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:48.146 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:48.146 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:48.146 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.16440", 2024-02-09T17:56:48.146 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:56:48.146 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 61018 2024-02-09T17:56:48.146 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.146 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:48.146 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.146 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:48.146 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.030684+0000", 2024-02-09T17:56:48.146 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:48.147 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.147 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.147 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:48.147 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.030684+0000", 2024-02-09T17:56:48.147 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:48.147 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.147 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.147 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:48.147 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.030685+0000", 2024-02-09T17:56:48.147 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5349999999999999e-06 2024-02-09T17:56:48.147 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.147 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.147 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:48.147 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.030712+0000", 2024-02-09T17:56:48.148 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7016e-05 2024-02-09T17:56:48.148 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.148 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.148 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:48.148 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.030725+0000", 2024-02-09T17:56:48.148 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2259000000000001e-05 2024-02-09T17:56:48.148 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.148 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.148 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:48.148 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.030732+0000", 2024-02-09T17:56:48.148 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7940000000000003e-06 2024-02-09T17:56:48.148 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.148 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.148 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:48.149 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.030827+0000", 2024-02-09T17:56:48.149 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.4452999999999998e-05 2024-02-09T17:56:48.149 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.149 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.149 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:48.149 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.030927+0000", 2024-02-09T17:56:48.149 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000100083 2024-02-09T17:56:48.149 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.149 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.149 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,9", 2024-02-09T17:56:48.149 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.031017+0000", 2024-02-09T17:56:48.149 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.9994999999999995e-05 2024-02-09T17:56:48.149 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.149 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.149 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:48.150 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.037230+0000", 2024-02-09T17:56:48.150 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0062132890000000003 2024-02-09T17:56:48.150 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.150 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.150 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:48.150 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.037574+0000", 2024-02-09T17:56:48.150 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000343228 2024-02-09T17:56:48.150 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.150 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.150 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:48.150 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.216386+0000", 2024-02-09T17:56:48.150 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.17881296099999999 2024-02-09T17:56:48.150 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.150 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.151 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:48.151 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.216437+0000", 2024-02-09T17:56:48.151 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0027000000000001e-05 2024-02-09T17:56:48.151 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.151 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.151 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:48.151 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.216466+0000", 2024-02-09T17:56:48.151 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.9530999999999999e-05 2024-02-09T17:56:48.151 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:48.151 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:48.151 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:48.151 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.151 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.151 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.16440.0:61020 7.5 7:af0a572b:::benchmark_data_smithi179_135804_object61019:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e674)", 2024-02-09T17:56:48.151 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:50:55.030745+0000", 2024-02-09T17:56:48.152 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 353.07614620200002, 2024-02-09T17:56:48.152 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.18584134399999999, 2024-02-09T17:56:48.152 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:48.152 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:48.152 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:48.152 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.16440", 2024-02-09T17:56:48.152 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:56:48.152 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 61020 2024-02-09T17:56:48.152 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.152 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:48.152 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.152 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:48.152 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.030745+0000", 2024-02-09T17:56:48.152 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:48.153 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.153 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.153 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:48.153 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.030745+0000", 2024-02-09T17:56:48.153 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:48.153 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.153 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.153 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:48.153 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.030745+0000", 2024-02-09T17:56:48.153 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7599999999999997e-07 2024-02-09T17:56:48.153 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.153 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.153 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:48.153 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.030762+0000", 2024-02-09T17:56:48.153 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6869000000000001e-05 2024-02-09T17:56:48.154 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.154 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.154 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:48.154 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.030767+0000", 2024-02-09T17:56:48.154 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9169999999999997e-06 2024-02-09T17:56:48.154 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.154 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.154 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:48.154 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.030771+0000", 2024-02-09T17:56:48.154 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2890000000000002e-06 2024-02-09T17:56:48.154 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.154 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.154 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:48.154 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.033207+0000", 2024-02-09T17:56:48.155 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0024362950000000002 2024-02-09T17:56:48.155 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.155 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.155 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:48.155 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.033351+0000", 2024-02-09T17:56:48.155 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014458299999999999 2024-02-09T17:56:48.155 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.155 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.155 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,9", 2024-02-09T17:56:48.155 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.033507+0000", 2024-02-09T17:56:48.155 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015522800000000001 2024-02-09T17:56:48.155 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.155 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.155 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:48.155 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.037237+0000", 2024-02-09T17:56:48.156 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0037301830000000002 2024-02-09T17:56:48.156 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.156 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.156 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:48.156 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.037639+0000", 2024-02-09T17:56:48.156 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00040180700000000001 2024-02-09T17:56:48.156 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.156 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.156 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:48.156 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.216500+0000", 2024-02-09T17:56:48.156 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.178861613 2024-02-09T17:56:48.156 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.156 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.156 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:48.157 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.216547+0000", 2024-02-09T17:56:48.157 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6845999999999998e-05 2024-02-09T17:56:48.157 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.157 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.157 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:48.157 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.216586+0000", 2024-02-09T17:56:48.157 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9038000000000001e-05 2024-02-09T17:56:48.157 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:48.157 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:48.157 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:48.157 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.157 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.157 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.16440.0:61023 7.5 7:a4a9218b:::benchmark_data_smithi179_135804_object61022:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e674)", 2024-02-09T17:56:48.157 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:50:55.035155+0000", 2024-02-09T17:56:48.157 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 353.07173555899999, 2024-02-09T17:56:48.158 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.18159081199999999, 2024-02-09T17:56:48.158 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:48.158 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:48.158 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:48.158 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.16440", 2024-02-09T17:56:48.158 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:56:48.158 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 61023 2024-02-09T17:56:48.158 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.158 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:48.158 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.158 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:48.158 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.035155+0000", 2024-02-09T17:56:48.158 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:48.158 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.158 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.159 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:48.159 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.035155+0000", 2024-02-09T17:56:48.159 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:48.159 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.159 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.159 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:48.159 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.035157+0000", 2024-02-09T17:56:48.159 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.378e-06 2024-02-09T17:56:48.159 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.159 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.159 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:48.159 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.035182+0000", 2024-02-09T17:56:48.159 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5622e-05 2024-02-09T17:56:48.159 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.160 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.160 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:48.160 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.035198+0000", 2024-02-09T17:56:48.160 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.52e-05 2024-02-09T17:56:48.160 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.160 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.160 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:48.160 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.035206+0000", 2024-02-09T17:56:48.160 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3510000000000002e-06 2024-02-09T17:56:48.160 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.160 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.160 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:48.160 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.035372+0000", 2024-02-09T17:56:48.160 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016588599999999999 2024-02-09T17:56:48.161 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.161 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.161 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:48.161 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.035480+0000", 2024-02-09T17:56:48.161 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000107996 2024-02-09T17:56:48.161 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.161 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.161 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,9", 2024-02-09T17:56:48.161 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.035577+0000", 2024-02-09T17:56:48.161 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6902000000000002e-05 2024-02-09T17:56:48.161 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.161 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.161 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:48.161 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.037304+0000", 2024-02-09T17:56:48.161 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0017267739999999999 2024-02-09T17:56:48.161 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.162 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.162 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:48.162 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.037800+0000", 2024-02-09T17:56:48.162 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000496866 2024-02-09T17:56:48.162 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.162 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.162 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:48.162 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.216654+0000", 2024-02-09T17:56:48.162 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.17885345999999999 2024-02-09T17:56:48.162 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.162 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.162 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:48.162 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.216710+0000", 2024-02-09T17:56:48.162 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6313000000000001e-05 2024-02-09T17:56:48.163 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.163 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.163 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:48.163 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.216746+0000", 2024-02-09T17:56:48.163 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6063999999999999e-05 2024-02-09T17:56:48.163 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:48.163 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:48.163 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:48.163 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.163 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.163 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.16440.0:61029 7.5 7:af86aa1e:::benchmark_data_smithi179_135804_object61028:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e674)", 2024-02-09T17:56:48.163 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:50:55.040695+0000", 2024-02-09T17:56:48.163 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 353.06619632299999, 2024-02-09T17:56:48.163 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.17622264900000001, 2024-02-09T17:56:48.164 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:48.164 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:48.164 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:48.164 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.16440", 2024-02-09T17:56:48.164 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:56:48.164 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 61029 2024-02-09T17:56:48.164 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.164 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:48.164 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.164 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:48.164 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.040695+0000", 2024-02-09T17:56:48.164 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:48.164 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.164 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.165 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:48.165 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.040695+0000", 2024-02-09T17:56:48.165 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:48.165 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.165 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.165 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:48.165 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.040696+0000", 2024-02-09T17:56:48.165 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2929999999999999e-06 2024-02-09T17:56:48.165 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.165 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.165 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:48.165 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.040717+0000", 2024-02-09T17:56:48.165 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1535999999999998e-05 2024-02-09T17:56:48.165 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.165 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.166 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:48.166 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.040731+0000", 2024-02-09T17:56:48.166 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3207000000000001e-05 2024-02-09T17:56:48.166 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.166 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.166 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:48.166 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.040738+0000", 2024-02-09T17:56:48.166 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4850000000000003e-06 2024-02-09T17:56:48.166 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.166 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.166 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:48.166 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.040779+0000", 2024-02-09T17:56:48.166 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0958000000000003e-05 2024-02-09T17:56:48.166 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.166 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.167 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:48.167 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.040876+0000", 2024-02-09T17:56:48.167 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6836999999999995e-05 2024-02-09T17:56:48.167 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.167 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.167 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,9", 2024-02-09T17:56:48.167 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.040972+0000", 2024-02-09T17:56:48.167 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5735000000000004e-05 2024-02-09T17:56:48.167 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.167 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.167 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:48.167 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.042563+0000", 2024-02-09T17:56:48.167 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001591271 2024-02-09T17:56:48.167 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.167 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.168 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:48.168 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.042649+0000", 2024-02-09T17:56:48.168 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5702e-05 2024-02-09T17:56:48.168 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.168 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.168 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:48.168 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.216796+0000", 2024-02-09T17:56:48.168 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.17414739100000001 2024-02-09T17:56:48.168 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.168 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.168 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:48.168 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.216864+0000", 2024-02-09T17:56:48.168 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7535999999999998e-05 2024-02-09T17:56:48.168 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.168 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.169 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:48.169 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:50:55.216917+0000", 2024-02-09T17:56:48.169 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3698e-05 2024-02-09T17:56:48.169 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:48.169 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:48.169 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:48.169 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.169 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.169 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:81716 8.6 8:6a5241ef:::benchmark_data_smithi179_148205_object81715:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e809)", 2024-02-09T17:56:48.169 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:56:12.720413+0000", 2024-02-09T17:56:48.169 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 35.386478121000003, 2024-02-09T17:56:48.169 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.12902585499999999, 2024-02-09T17:56:48.169 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:48.169 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:48.170 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:48.170 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T17:56:48.170 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T17:56:48.170 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 81716 2024-02-09T17:56:48.170 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.170 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:48.170 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.170 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:48.170 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.720413+0000", 2024-02-09T17:56:48.170 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:48.170 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.170 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.170 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:48.170 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.720413+0000", 2024-02-09T17:56:48.170 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:48.171 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.171 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.171 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:48.171 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.720414+0000", 2024-02-09T17:56:48.171 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0890000000000001e-06 2024-02-09T17:56:48.171 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.171 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.171 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:48.171 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.720461+0000", 2024-02-09T17:56:48.171 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7259999999999998e-05 2024-02-09T17:56:48.171 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.171 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.171 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:48.171 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.720468+0000", 2024-02-09T17:56:48.171 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4790000000000001e-06 2024-02-09T17:56:48.172 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.172 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.172 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:48.172 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.720473+0000", 2024-02-09T17:56:48.172 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0259999999999998e-06 2024-02-09T17:56:48.172 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.172 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.172 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:48.172 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.720546+0000", 2024-02-09T17:56:48.172 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.3392999999999993e-05 2024-02-09T17:56:48.172 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.172 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.172 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:48.172 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.720700+0000", 2024-02-09T17:56:48.172 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015412100000000001 2024-02-09T17:56:48.173 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.173 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.173 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 7,10", 2024-02-09T17:56:48.173 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.720790+0000", 2024-02-09T17:56:48.173 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.9452999999999999e-05 2024-02-09T17:56:48.173 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.173 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.173 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:48.173 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.723314+0000", 2024-02-09T17:56:48.173 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002524058 2024-02-09T17:56:48.173 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.173 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.173 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:48.173 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.723691+0000", 2024-02-09T17:56:48.173 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00037683200000000001 2024-02-09T17:56:48.174 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.174 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.174 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:48.174 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.849333+0000", 2024-02-09T17:56:48.174 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.12564265799999999 2024-02-09T17:56:48.174 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.174 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.174 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:48.174 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.849390+0000", 2024-02-09T17:56:48.174 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6344999999999997e-05 2024-02-09T17:56:48.174 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.174 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.174 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:48.174 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:12.849439+0000", 2024-02-09T17:56:48.175 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9141000000000003e-05 2024-02-09T17:56:48.175 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:48.175 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:48.175 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:48.175 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.175 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.175 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:105629 8.3 8:cd87d712:::benchmark_data_smithi179_148205_object105628:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e813)", 2024-02-09T17:56:48.175 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:56:34.837661+0000", 2024-02-09T17:56:48.175 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 13.269229899999999, 2024-02-09T17:56:48.175 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.20996332200000001, 2024-02-09T17:56:48.175 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:48.175 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:48.175 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:48.175 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T17:56:48.175 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T17:56:48.176 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 105629 2024-02-09T17:56:48.176 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.176 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:48.176 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.176 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:48.176 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.837661+0000", 2024-02-09T17:56:48.176 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:48.176 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.176 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.176 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:48.176 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.837661+0000", 2024-02-09T17:56:48.176 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:48.176 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.176 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.176 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:48.177 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.837661+0000", 2024-02-09T17:56:48.177 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8099999999999998e-07 2024-02-09T17:56:48.177 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.177 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.177 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:48.177 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.837681+0000", 2024-02-09T17:56:48.177 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9480999999999999e-05 2024-02-09T17:56:48.177 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.177 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.177 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:48.177 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.837687+0000", 2024-02-09T17:56:48.177 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0419999999999999e-06 2024-02-09T17:56:48.177 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.177 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.177 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:48.178 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.837692+0000", 2024-02-09T17:56:48.178 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.617e-06 2024-02-09T17:56:48.178 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.178 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.178 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:48.178 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.837784+0000", 2024-02-09T17:56:48.178 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2528999999999994e-05 2024-02-09T17:56:48.178 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.178 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.178 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:48.178 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.837913+0000", 2024-02-09T17:56:48.178 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012892700000000001 2024-02-09T17:56:48.178 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.178 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.178 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5", 2024-02-09T17:56:48.179 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.838037+0000", 2024-02-09T17:56:48.179 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000123934 2024-02-09T17:56:48.179 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.179 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.179 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:48.179 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.841579+0000", 2024-02-09T17:56:48.179 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0035419710000000001 2024-02-09T17:56:48.179 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.179 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.179 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:48.179 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.844085+0000", 2024-02-09T17:56:48.179 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002506458 2024-02-09T17:56:48.179 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.179 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.179 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:48.179 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:35.047529+0000", 2024-02-09T17:56:48.180 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.20344384800000001 2024-02-09T17:56:48.180 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.180 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.180 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:48.180 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:35.047588+0000", 2024-02-09T17:56:48.180 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9030999999999998e-05 2024-02-09T17:56:48.180 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.180 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.180 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:48.180 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:35.047624+0000", 2024-02-09T17:56:48.180 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6103000000000003e-05 2024-02-09T17:56:48.180 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:48.180 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:48.180 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:48.181 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.181 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.181 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:105635 8.3 8:c539ab04:::benchmark_data_smithi179_148205_object105634:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e813)", 2024-02-09T17:56:48.181 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:56:34.844055+0000", 2024-02-09T17:56:48.181 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 13.262836159999999, 2024-02-09T17:56:48.181 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.2139199, 2024-02-09T17:56:48.181 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:48.181 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:48.181 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:48.181 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T17:56:48.181 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T17:56:48.181 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 105635 2024-02-09T17:56:48.181 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.181 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:48.181 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.182 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:48.182 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.844055+0000", 2024-02-09T17:56:48.182 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:48.182 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.182 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.182 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:48.182 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.844055+0000", 2024-02-09T17:56:48.182 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:48.182 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.182 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.182 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:48.182 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.844056+0000", 2024-02-09T17:56:48.182 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.581e-06 2024-02-09T17:56:48.182 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.182 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.183 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:48.183 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.844079+0000", 2024-02-09T17:56:48.183 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2778000000000001e-05 2024-02-09T17:56:48.183 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.183 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.183 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:48.183 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.844100+0000", 2024-02-09T17:56:48.183 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0360000000000002e-05 2024-02-09T17:56:48.183 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.183 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.183 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:48.183 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.844106+0000", 2024-02-09T17:56:48.183 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5599999999999999e-06 2024-02-09T17:56:48.183 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.183 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.184 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:48.184 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.844165+0000", 2024-02-09T17:56:48.184 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8675000000000003e-05 2024-02-09T17:56:48.184 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.184 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.184 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:48.184 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.844277+0000", 2024-02-09T17:56:48.184 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000111801 2024-02-09T17:56:48.184 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.184 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.184 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5", 2024-02-09T17:56:48.184 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.844408+0000", 2024-02-09T17:56:48.184 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000131772 2024-02-09T17:56:48.184 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.185 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.185 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:48.185 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.880948+0000", 2024-02-09T17:56:48.185 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.036539598999999999 2024-02-09T17:56:48.185 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.185 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.185 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:48.185 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.884819+0000", 2024-02-09T17:56:48.185 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0038705829999999999 2024-02-09T17:56:48.185 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.185 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.185 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:48.185 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:35.057908+0000", 2024-02-09T17:56:48.185 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.17308984899999999 2024-02-09T17:56:48.185 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.186 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.186 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:48.186 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:35.057952+0000", 2024-02-09T17:56:48.186 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3711000000000002e-05 2024-02-09T17:56:48.186 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.186 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.186 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:48.186 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:35.057975+0000", 2024-02-09T17:56:48.186 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2631000000000001e-05 2024-02-09T17:56:48.186 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:48.186 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:48.186 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:48.186 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.186 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.186 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:105639 8.3 8:c0ddb084:::benchmark_data_smithi179_148205_object105638:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e813)", 2024-02-09T17:56:48.187 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:56:34.848960+0000", 2024-02-09T17:56:48.187 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 13.257930757, 2024-02-09T17:56:48.187 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.20911170300000001, 2024-02-09T17:56:48.187 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:48.187 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:48.187 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:48.187 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T17:56:48.187 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T17:56:48.187 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 105639 2024-02-09T17:56:48.187 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.187 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:48.187 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.187 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:48.187 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.848960+0000", 2024-02-09T17:56:48.187 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:48.188 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.188 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.188 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:48.188 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.848960+0000", 2024-02-09T17:56:48.188 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:48.188 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.188 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.188 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:48.188 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.848962+0000", 2024-02-09T17:56:48.188 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5990000000000001e-06 2024-02-09T17:56:48.188 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.188 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.188 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:48.188 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.848997+0000", 2024-02-09T17:56:48.188 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4958e-05 2024-02-09T17:56:48.189 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.189 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.189 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:48.189 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.880848+0000", 2024-02-09T17:56:48.189 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.031851434999999997 2024-02-09T17:56:48.189 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.189 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.189 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:48.189 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.880891+0000", 2024-02-09T17:56:48.189 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3041999999999999e-05 2024-02-09T17:56:48.189 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.189 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.189 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:48.189 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.881034+0000", 2024-02-09T17:56:48.189 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014289000000000001 2024-02-09T17:56:48.190 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.190 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.190 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:48.190 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.881289+0000", 2024-02-09T17:56:48.190 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00025442999999999999 2024-02-09T17:56:48.190 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.190 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.190 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5", 2024-02-09T17:56:48.190 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.881469+0000", 2024-02-09T17:56:48.190 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00018017 2024-02-09T17:56:48.190 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.190 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.190 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:48.190 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.885350+0000", 2024-02-09T17:56:48.190 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.003881387 2024-02-09T17:56:48.191 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.191 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.191 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:48.191 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.890147+0000", 2024-02-09T17:56:48.191 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0047972620000000001 2024-02-09T17:56:48.191 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.191 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.191 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:48.191 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:35.058011+0000", 2024-02-09T17:56:48.191 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.16786406700000001 2024-02-09T17:56:48.191 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.191 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.191 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:48.191 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:35.058045+0000", 2024-02-09T17:56:48.191 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.3838999999999999e-05 2024-02-09T17:56:48.192 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.192 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.192 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:48.192 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:35.058072+0000", 2024-02-09T17:56:48.192 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6624000000000001e-05 2024-02-09T17:56:48.192 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:48.192 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:48.192 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:48.192 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.192 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.192 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:105646 8.3 8:ca3559b0:::benchmark_data_smithi179_148205_object105645:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e813)", 2024-02-09T17:56:48.192 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:56:34.881228+0000", 2024-02-09T17:56:48.192 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 13.225663007, 2024-02-09T17:56:48.192 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.176949463, 2024-02-09T17:56:48.193 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:48.193 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:48.193 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:48.193 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T17:56:48.193 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T17:56:48.193 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 105646 2024-02-09T17:56:48.193 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.193 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:48.193 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.193 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:48.193 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.881228+0000", 2024-02-09T17:56:48.193 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:48.193 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.193 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.194 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:48.194 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.881228+0000", 2024-02-09T17:56:48.194 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:48.194 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.194 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.194 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:48.194 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.881228+0000", 2024-02-09T17:56:48.194 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2499999999999995e-07 2024-02-09T17:56:48.194 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.194 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.194 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:48.194 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.881255+0000", 2024-02-09T17:56:48.194 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6769000000000001e-05 2024-02-09T17:56:48.194 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.194 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.195 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:48.195 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.881262+0000", 2024-02-09T17:56:48.195 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1749999999999999e-06 2024-02-09T17:56:48.195 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.195 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.195 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:48.195 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.881268+0000", 2024-02-09T17:56:48.195 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9290000000000003e-06 2024-02-09T17:56:48.195 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.195 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.195 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:48.195 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.885578+0000", 2024-02-09T17:56:48.195 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0043097070000000003 2024-02-09T17:56:48.195 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.195 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.196 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:48.196 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.885704+0000", 2024-02-09T17:56:48.196 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000125912 2024-02-09T17:56:48.196 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.196 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.196 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5", 2024-02-09T17:56:48.196 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.885850+0000", 2024-02-09T17:56:48.196 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000145976 2024-02-09T17:56:48.196 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.196 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.196 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:48.196 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.891295+0000", 2024-02-09T17:56:48.196 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0054448730000000002 2024-02-09T17:56:48.196 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.197 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.197 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:48.197 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.892998+0000", 2024-02-09T17:56:48.197 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0017029009999999999 2024-02-09T17:56:48.197 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.197 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.197 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:48.197 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:35.058115+0000", 2024-02-09T17:56:48.197 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.165117025 2024-02-09T17:56:48.197 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.197 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.197 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:48.197 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:35.058157+0000", 2024-02-09T17:56:48.197 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1869999999999997e-05 2024-02-09T17:56:48.198 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.198 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.198 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:48.198 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:35.058177+0000", 2024-02-09T17:56:48.198 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0801e-05 2024-02-09T17:56:48.198 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:48.198 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:48.198 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:48.198 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.198 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.198 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:105676 8.3 8:ca880744:::benchmark_data_smithi179_148205_object105675:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e813)", 2024-02-09T17:56:48.198 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:56:34.905656+0000", 2024-02-09T17:56:48.198 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 13.201235378, 2024-02-09T17:56:48.198 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15641833399999999, 2024-02-09T17:56:48.198 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:48.199 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:48.199 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:48.199 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T17:56:48.199 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T17:56:48.199 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 105676 2024-02-09T17:56:48.199 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.199 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:48.199 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.199 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:48.199 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.905656+0000", 2024-02-09T17:56:48.199 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:48.199 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.199 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.199 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:48.199 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.905656+0000", 2024-02-09T17:56:48.200 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:48.200 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.200 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.200 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:48.200 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.905657+0000", 2024-02-09T17:56:48.200 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.635e-06 2024-02-09T17:56:48.200 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.200 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.200 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:48.200 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.905682+0000", 2024-02-09T17:56:48.200 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4624e-05 2024-02-09T17:56:48.200 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.200 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.200 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:48.201 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.905697+0000", 2024-02-09T17:56:48.201 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5569999999999998e-05 2024-02-09T17:56:48.201 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.201 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.201 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:48.201 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.905706+0000", 2024-02-09T17:56:48.201 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2300000000000008e-06 2024-02-09T17:56:48.201 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.201 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.201 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:48.201 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.905767+0000", 2024-02-09T17:56:48.201 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1601999999999997e-05 2024-02-09T17:56:48.201 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.201 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.201 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:48.202 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.905947+0000", 2024-02-09T17:56:48.202 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00017951299999999999 2024-02-09T17:56:48.202 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.202 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.202 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5", 2024-02-09T17:56:48.202 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.906104+0000", 2024-02-09T17:56:48.202 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015701300000000001 2024-02-09T17:56:48.202 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.202 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.202 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:48.202 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.909742+0000", 2024-02-09T17:56:48.202 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0036384519999999999 2024-02-09T17:56:48.202 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.202 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.202 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:48.203 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.911323+0000", 2024-02-09T17:56:48.203 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001581163 2024-02-09T17:56:48.203 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.203 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.203 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:48.203 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:35.062012+0000", 2024-02-09T17:56:48.203 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15068821399999999 2024-02-09T17:56:48.203 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.203 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.203 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:48.203 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:35.062050+0000", 2024-02-09T17:56:48.203 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8328999999999998e-05 2024-02-09T17:56:48.203 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.203 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.203 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:48.204 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:35.062074+0000", 2024-02-09T17:56:48.204 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3989e-05 2024-02-09T17:56:48.204 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:48.204 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:48.204 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:48.204 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.204 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.204 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:105691 8.3 8:cb441aa7:::benchmark_data_smithi179_148205_object105690:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e813)", 2024-02-09T17:56:48.204 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:56:34.920256+0000", 2024-02-09T17:56:48.204 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 13.186635322000001, 2024-02-09T17:56:48.204 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14192108000000001, 2024-02-09T17:56:48.204 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:48.204 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:48.204 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:48.204 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T17:56:48.205 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T17:56:48.205 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 105691 2024-02-09T17:56:48.205 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.205 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:48.205 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.205 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:48.205 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.920256+0000", 2024-02-09T17:56:48.205 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:48.205 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.205 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.205 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:48.205 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.920256+0000", 2024-02-09T17:56:48.205 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:48.205 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.205 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.206 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:48.206 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.920257+0000", 2024-02-09T17:56:48.206 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0279999999999999e-06 2024-02-09T17:56:48.206 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.206 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.206 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:48.206 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.920288+0000", 2024-02-09T17:56:48.206 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1723000000000001e-05 2024-02-09T17:56:48.206 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.206 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.206 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:48.206 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.920302+0000", 2024-02-09T17:56:48.206 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3481e-05 2024-02-09T17:56:48.206 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.206 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.207 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:48.207 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.920310+0000", 2024-02-09T17:56:48.207 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.0879999999999995e-06 2024-02-09T17:56:48.207 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.207 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.207 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:48.207 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.920375+0000", 2024-02-09T17:56:48.207 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5501999999999997e-05 2024-02-09T17:56:48.207 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.207 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.207 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:48.207 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.920519+0000", 2024-02-09T17:56:48.207 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000143552 2024-02-09T17:56:48.207 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.207 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.208 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5", 2024-02-09T17:56:48.208 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.920655+0000", 2024-02-09T17:56:48.208 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000136303 2024-02-09T17:56:48.208 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.208 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.208 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:48.208 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.924818+0000", 2024-02-09T17:56:48.208 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0041626789999999999 2024-02-09T17:56:48.208 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.208 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.208 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:48.208 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.926321+0000", 2024-02-09T17:56:48.208 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001502729 2024-02-09T17:56:48.208 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.209 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.209 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:48.209 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:35.062110+0000", 2024-02-09T17:56:48.209 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.135789413 2024-02-09T17:56:48.209 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.209 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.209 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:48.209 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:35.062148+0000", 2024-02-09T17:56:48.209 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8019999999999999e-05 2024-02-09T17:56:48.209 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.209 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.209 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:48.209 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:35.062177+0000", 2024-02-09T17:56:48.209 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.8561999999999999e-05 2024-02-09T17:56:48.209 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:48.209 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:48.210 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:48.210 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.210 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.210 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:105701 8.3 8:c4c9abb8:::benchmark_data_smithi179_148205_object105700:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e813)", 2024-02-09T17:56:48.210 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:56:34.929318+0000", 2024-02-09T17:56:48.210 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 13.177572622, 2024-02-09T17:56:48.210 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.13296940700000001, 2024-02-09T17:56:48.210 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:48.210 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:48.210 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:48.210 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T17:56:48.210 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T17:56:48.210 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 105701 2024-02-09T17:56:48.210 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.210 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:48.211 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.211 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:48.211 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.929318+0000", 2024-02-09T17:56:48.211 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:48.211 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.211 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.211 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:48.211 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.929318+0000", 2024-02-09T17:56:48.211 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:48.211 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.211 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.211 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:48.211 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.929320+0000", 2024-02-09T17:56:48.211 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4699999999999999e-06 2024-02-09T17:56:48.211 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.212 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.212 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:48.212 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.929338+0000", 2024-02-09T17:56:48.212 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8425000000000001e-05 2024-02-09T17:56:48.212 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.212 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.212 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:48.212 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.929352+0000", 2024-02-09T17:56:48.212 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4055e-05 2024-02-09T17:56:48.212 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.212 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.212 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:48.212 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.929361+0000", 2024-02-09T17:56:48.212 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3250000000000008e-06 2024-02-09T17:56:48.212 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.213 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.213 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:48.213 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.929413+0000", 2024-02-09T17:56:48.213 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2323e-05 2024-02-09T17:56:48.213 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.213 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.213 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:48.213 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.929525+0000", 2024-02-09T17:56:48.213 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000112507 2024-02-09T17:56:48.213 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.213 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.213 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5", 2024-02-09T17:56:48.213 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.929619+0000", 2024-02-09T17:56:48.213 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.3877999999999998e-05 2024-02-09T17:56:48.213 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.214 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.214 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:48.214 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.932543+0000", 2024-02-09T17:56:48.214 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0029232210000000002 2024-02-09T17:56:48.214 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.214 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.214 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:48.214 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.934913+0000", 2024-02-09T17:56:48.214 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0023709529999999999 2024-02-09T17:56:48.214 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.214 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.214 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:48.214 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:35.062208+0000", 2024-02-09T17:56:48.214 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.12729458099999999 2024-02-09T17:56:48.214 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.215 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.215 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:48.215 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:35.062257+0000", 2024-02-09T17:56:48.215 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9218999999999998e-05 2024-02-09T17:56:48.215 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.215 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.215 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:48.215 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:35.062288+0000", 2024-02-09T17:56:48.215 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0450000000000002e-05 2024-02-09T17:56:48.215 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:48.215 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:48.215 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:48.215 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.216 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.216 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:105712 8.3 8:c0b52f6a:::benchmark_data_smithi179_148205_object105711:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e813)", 2024-02-09T17:56:48.216 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:56:34.942737+0000", 2024-02-09T17:56:48.216 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 13.164154009000001, 2024-02-09T17:56:48.216 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.142697937, 2024-02-09T17:56:48.216 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:48.216 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:48.216 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:48.216 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T17:56:48.216 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T17:56:48.216 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 105712 2024-02-09T17:56:48.216 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.216 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:48.216 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.216 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:48.217 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.942737+0000", 2024-02-09T17:56:48.217 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:48.217 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.217 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.217 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:48.217 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.942737+0000", 2024-02-09T17:56:48.217 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:48.217 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.217 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.217 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:48.217 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.942739+0000", 2024-02-09T17:56:48.217 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0020000000000001e-06 2024-02-09T17:56:48.217 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.217 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.217 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:48.218 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.942793+0000", 2024-02-09T17:56:48.218 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4536000000000001e-05 2024-02-09T17:56:48.218 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.218 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.218 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:48.218 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.942815+0000", 2024-02-09T17:56:48.218 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1563e-05 2024-02-09T17:56:48.218 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.218 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.218 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:48.218 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.942825+0000", 2024-02-09T17:56:48.218 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6590000000000002e-06 2024-02-09T17:56:48.218 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.218 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.218 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:48.219 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.943007+0000", 2024-02-09T17:56:48.219 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000182067 2024-02-09T17:56:48.219 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.219 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.219 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:48.219 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.943160+0000", 2024-02-09T17:56:48.219 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000153685 2024-02-09T17:56:48.219 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.219 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.219 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5", 2024-02-09T17:56:48.219 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.943297+0000", 2024-02-09T17:56:48.219 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013692899999999999 2024-02-09T17:56:48.219 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.219 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.219 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:48.220 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.947110+0000", 2024-02-09T17:56:48.220 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0038121850000000001 2024-02-09T17:56:48.220 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.220 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.220 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:48.220 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.949304+0000", 2024-02-09T17:56:48.220 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002194587 2024-02-09T17:56:48.220 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.220 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.220 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:48.220 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:35.085350+0000", 2024-02-09T17:56:48.220 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.13604601099999999 2024-02-09T17:56:48.220 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.220 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.220 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:48.221 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:35.085398+0000", 2024-02-09T17:56:48.221 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7648999999999997e-05 2024-02-09T17:56:48.221 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.221 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.221 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:48.221 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:35.085435+0000", 2024-02-09T17:56:48.221 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7064000000000003e-05 2024-02-09T17:56:48.221 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:48.221 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:48.221 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:48.221 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.221 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.221 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:105717 8.3 8:c7a68182:::benchmark_data_smithi179_148205_object105716:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e813)", 2024-02-09T17:56:48.221 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:56:34.949968+0000", 2024-02-09T17:56:48.221 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 13.156922751, 2024-02-09T17:56:48.222 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.13557548, 2024-02-09T17:56:48.222 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:48.222 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:48.222 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:48.222 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T17:56:48.222 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T17:56:48.222 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 105717 2024-02-09T17:56:48.222 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.222 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:48.222 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.222 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:48.222 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.949968+0000", 2024-02-09T17:56:48.222 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:48.222 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.222 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.223 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:48.223 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.949968+0000", 2024-02-09T17:56:48.223 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:48.223 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.223 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.223 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:48.223 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.949970+0000", 2024-02-09T17:56:48.223 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.782e-06 2024-02-09T17:56:48.223 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.223 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.223 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:48.223 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.950019+0000", 2024-02-09T17:56:48.223 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8946999999999997e-05 2024-02-09T17:56:48.223 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.223 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.224 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:48.224 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.950029+0000", 2024-02-09T17:56:48.224 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0161e-05 2024-02-09T17:56:48.224 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.224 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.224 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:48.224 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.950035+0000", 2024-02-09T17:56:48.224 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6080000000000003e-06 2024-02-09T17:56:48.224 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.224 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.224 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:48.224 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.950080+0000", 2024-02-09T17:56:48.224 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5667000000000002e-05 2024-02-09T17:56:48.224 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.224 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.225 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:48.225 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.950200+0000", 2024-02-09T17:56:48.225 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011971700000000001 2024-02-09T17:56:48.225 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.225 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.225 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5", 2024-02-09T17:56:48.225 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.950348+0000", 2024-02-09T17:56:48.225 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014774699999999999 2024-02-09T17:56:48.225 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.225 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.225 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:48.225 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.953463+0000", 2024-02-09T17:56:48.225 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0031156090000000001 2024-02-09T17:56:48.225 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.225 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.226 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:48.226 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.953936+0000", 2024-02-09T17:56:48.226 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00047283900000000002 2024-02-09T17:56:48.226 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.226 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.226 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:48.226 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:35.085472+0000", 2024-02-09T17:56:48.226 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.131535237 2024-02-09T17:56:48.226 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.226 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.226 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:48.226 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:35.085514+0000", 2024-02-09T17:56:48.226 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2290999999999999e-05 2024-02-09T17:56:48.226 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.226 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.227 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:48.227 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:35.085544+0000", 2024-02-09T17:56:48.227 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.9875000000000001e-05 2024-02-09T17:56:48.227 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:48.227 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:48.227 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:48.227 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.227 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.227 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:105723 8.3 8:c4a73867:::benchmark_data_smithi179_148205_object105722:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e813)", 2024-02-09T17:56:48.227 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:56:34.956198+0000", 2024-02-09T17:56:48.227 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 13.150692849, 2024-02-09T17:56:48.227 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.12945121100000001, 2024-02-09T17:56:48.227 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:56:48.227 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:56:48.227 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:56:48.228 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T17:56:48.228 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T17:56:48.228 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 105723 2024-02-09T17:56:48.228 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.228 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:56:48.228 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.228 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:56:48.228 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.956198+0000", 2024-02-09T17:56:48.228 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:48.228 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.228 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.228 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:56:48.228 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.956198+0000", 2024-02-09T17:56:48.228 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:56:48.229 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.229 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.229 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:56:48.229 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.956200+0000", 2024-02-09T17:56:48.229 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7489999999999999e-06 2024-02-09T17:56:48.229 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.229 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.229 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:56:48.229 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.956221+0000", 2024-02-09T17:56:48.229 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1084e-05 2024-02-09T17:56:48.229 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.229 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.229 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:56:48.229 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.956235+0000", 2024-02-09T17:56:48.229 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4277000000000001e-05 2024-02-09T17:56:48.229 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.230 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.230 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:56:48.230 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.956243+0000", 2024-02-09T17:56:48.230 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1100000000000003e-06 2024-02-09T17:56:48.230 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.230 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.230 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:56:48.230 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.956294+0000", 2024-02-09T17:56:48.230 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0269000000000003e-05 2024-02-09T17:56:48.230 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.230 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.230 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:56:48.230 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.956463+0000", 2024-02-09T17:56:48.230 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016910300000000001 2024-02-09T17:56:48.230 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.231 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.231 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5", 2024-02-09T17:56:48.231 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.956584+0000", 2024-02-09T17:56:48.231 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000120821 2024-02-09T17:56:48.231 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.231 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.231 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:56:48.231 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.959805+0000", 2024-02-09T17:56:48.231 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.003221893 2024-02-09T17:56:48.231 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.231 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.231 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:48.231 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:34.960263+0000", 2024-02-09T17:56:48.231 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00045756600000000002 2024-02-09T17:56:48.231 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.232 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.232 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:56:48.232 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:35.085578+0000", 2024-02-09T17:56:48.232 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.12531534999999999 2024-02-09T17:56:48.233 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.233 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.233 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:56:48.233 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:35.085620+0000", 2024-02-09T17:56:48.233 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1955000000000001e-05 2024-02-09T17:56:48.233 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:56:48.233 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:56:48.233 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:56:48.234 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:35.085649+0000", 2024-02-09T17:56:48.234 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.9034e-05 2024-02-09T17:56:48.234 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:48.234 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:48.234 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:48.234 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:56:48.234 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:56:48.234 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:56:48.235 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:56:48.264 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:48.288 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:56:48.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:48.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:56:48.392 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:56:48.467 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:48.509 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:56:48.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:48.512 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:56:48.613 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:56:48.614 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 82 16 116738 116722 88.9355 47.0625 0.0209298 0.0112394 2024-02-09T17:56:48.659 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:48.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:48 smithi179 ceph-mon[43975]: pgmap v2020: 47 pgs: 1 active+clean+scrubbing, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 7.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 922 op/s; 21876/341838 objects degraded (6.400%) 2024-02-09T17:56:48.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:48 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:56:48.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:48 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:56:48.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:48 smithi179 ceph-mon[43975]: 1.7 scrub starts 2024-02-09T17:56:48.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:48 smithi179 ceph-mon[43975]: 1.7 scrub ok 2024-02-09T17:56:48.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:48 smithi179 ceph-mon[43975]: 8.4 scrub starts 2024-02-09T17:56:48.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:48 smithi179 ceph-mon[43975]: osdmap e820: 12 total, 11 up, 12 in 2024-02-09T17:56:48.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:48 smithi179 ceph-mon[43975]: 1.b scrub starts 2024-02-09T17:56:48.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:48 smithi179 ceph-mon[43975]: 1.10 scrub starts 2024-02-09T17:56:48.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:48 smithi179 ceph-mon[43975]: 1.10 scrub ok 2024-02-09T17:56:48.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:48 smithi162 ceph-mon[44304]: pgmap v2020: 47 pgs: 1 active+clean+scrubbing, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 7.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 922 op/s; 21876/341838 objects degraded (6.400%) 2024-02-09T17:56:48.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:48 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:56:48.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:48 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:56:48.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:48 smithi162 ceph-mon[44304]: 1.7 scrub starts 2024-02-09T17:56:48.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:48 smithi162 ceph-mon[44304]: 1.7 scrub ok 2024-02-09T17:56:48.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:48 smithi162 ceph-mon[44304]: 8.4 scrub starts 2024-02-09T17:56:48.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:48 smithi162 ceph-mon[44304]: osdmap e820: 12 total, 11 up, 12 in 2024-02-09T17:56:48.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:48 smithi162 ceph-mon[44304]: 1.b scrub starts 2024-02-09T17:56:48.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:48 smithi162 ceph-mon[44304]: 1.10 scrub starts 2024-02-09T17:56:48.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:48 smithi162 ceph-mon[44304]: 1.10 scrub ok 2024-02-09T17:56:48.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:48 smithi099 ceph-mon[39420]: pgmap v2020: 47 pgs: 1 active+clean+scrubbing, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 7.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 922 op/s; 21876/341838 objects degraded (6.400%) 2024-02-09T17:56:48.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:48 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:56:48.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:48 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:56:48.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:48 smithi099 ceph-mon[39420]: 1.7 scrub starts 2024-02-09T17:56:48.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:48 smithi099 ceph-mon[39420]: 1.7 scrub ok 2024-02-09T17:56:48.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:48 smithi099 ceph-mon[39420]: 8.4 scrub starts 2024-02-09T17:56:48.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:48 smithi099 ceph-mon[39420]: osdmap e820: 12 total, 11 up, 12 in 2024-02-09T17:56:48.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:48 smithi099 ceph-mon[39420]: 1.b scrub starts 2024-02-09T17:56:48.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:48 smithi099 ceph-mon[39420]: 1.10 scrub starts 2024-02-09T17:56:48.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:48 smithi099 ceph-mon[39420]: 1.10 scrub ok 2024-02-09T17:56:49.124 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set noscrub 2024-02-09T17:56:49.181 INFO:teuthology.orchestra.run.smithi099.stderr:Creating an OSD daemon form without an OSD FSID value 2024-02-09T17:56:49.614 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 83 16 117539 117523 88.4667 50.0625 0.00505269 0.0112984 2024-02-09T17:56:49.671 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:56:49.705 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:49 smithi099 ceph-mon[39420]: 1.b scrub ok 2024-02-09T17:56:49.705 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:49 smithi099 ceph-mon[39420]: 8.3 scrub starts 2024-02-09T17:56:49.705 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:49 smithi099 ceph-mon[39420]: 1.9 deep-scrub starts 2024-02-09T17:56:49.911 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:56:49.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:49 smithi179 ceph-mon[43975]: 1.b scrub ok 2024-02-09T17:56:49.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:49 smithi179 ceph-mon[43975]: 8.3 scrub starts 2024-02-09T17:56:49.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:49 smithi179 ceph-mon[43975]: 1.9 deep-scrub starts 2024-02-09T17:56:49.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:49 smithi162 ceph-mon[44304]: 1.b scrub ok 2024-02-09T17:56:49.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:49 smithi162 ceph-mon[44304]: 8.3 scrub starts 2024-02-09T17:56:49.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:49 smithi162 ceph-mon[44304]: 1.9 deep-scrub starts 2024-02-09T17:56:50.614 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 84 16 118253 118237 87.9449 44.625 0.00345982 0.0113571 2024-02-09T17:56:50.810 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:56:50.841 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:56:50.844 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:50 smithi179 ceph-mon[43975]: pgmap v2022: 47 pgs: 1 active+clean+scrubbing, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 7.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 873 op/s; 21876/346491 objects degraded (6.314%) 2024-02-09T17:56:50.844 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:50 smithi179 ceph-mon[43975]: 8.7 scrub starts 2024-02-09T17:56:50.844 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:50 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 21876/346491 objects degraded (6.314%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:56:50.844 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:50 smithi179 ceph-mon[43975]: 1.9 deep-scrub ok 2024-02-09T17:56:50.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:50.845 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:56:50.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:50 smithi162 ceph-mon[44304]: pgmap v2022: 47 pgs: 1 active+clean+scrubbing, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 7.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 873 op/s; 21876/346491 objects degraded (6.314%) 2024-02-09T17:56:50.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:50 smithi162 ceph-mon[44304]: 8.7 scrub starts 2024-02-09T17:56:50.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:50 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 21876/346491 objects degraded (6.314%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:56:50.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:50 smithi162 ceph-mon[44304]: 1.9 deep-scrub ok 2024-02-09T17:56:50.945 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:56:50.962 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:50 smithi099 ceph-mon[39420]: pgmap v2022: 47 pgs: 1 active+clean+scrubbing, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 7.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 873 op/s; 21876/346491 objects degraded (6.314%) 2024-02-09T17:56:50.962 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:50 smithi099 ceph-mon[39420]: 8.7 scrub starts 2024-02-09T17:56:50.962 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:50 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 21876/346491 objects degraded (6.314%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:56:50.963 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:50 smithi099 ceph-mon[39420]: 1.9 deep-scrub ok 2024-02-09T17:56:50.973 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:50.995 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:56:50.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:50.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:56:51.100 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:56:51.127 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:51.146 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:56:51.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:51.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:56:51.253 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:56:51.301 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:51.565 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:51 smithi179 ceph-mon[43975]: 8.4 scrub ok 2024-02-09T17:56:51.616 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 85 16 119007 118991 87.4627 47.125 0.00456276 0.0114229 2024-02-09T17:56:51.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:51 smithi162 ceph-mon[44304]: 8.4 scrub ok 2024-02-09T17:56:51.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:51 smithi099 ceph-mon[39420]: 8.4 scrub ok 2024-02-09T17:56:52.617 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 86 16 119786 119770 87.0119 48.6875 0.0304451 0.0114881 2024-02-09T17:56:52.674 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-09T17:56:52.714 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:56:52.715 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:52 smithi099 ceph-mon[39420]: pgmap v2023: 47 pgs: 1 active+clean+scrubbing, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 7.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 770 op/s; 21876/348027 objects degraded (6.286%) 2024-02-09T17:56:52.715 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:52 smithi099 ceph-mon[39420]: 1.17 scrub starts 2024-02-09T17:56:52.715 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:52 smithi099 ceph-mon[39420]: 8.5 scrub starts 2024-02-09T17:56:52.715 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:52 smithi099 ceph-mon[39420]: 8.3 scrub ok 2024-02-09T17:56:52.715 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:52 smithi099 ceph-mon[39420]: 1.17 scrub ok 2024-02-09T17:56:52.715 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:52 smithi099 ceph-mon[39420]: 8.7 scrub ok 2024-02-09T17:56:52.715 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:52 smithi099 ceph-mon[39420]: 8.d scrub starts 2024-02-09T17:56:52.716 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:52 smithi099 ceph-mon[39420]: 8.b scrub starts 2024-02-09T17:56:52.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:52.719 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:56:52.820 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:56:52.872 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:52 smithi179 ceph-mon[43975]: pgmap v2023: 47 pgs: 1 active+clean+scrubbing, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 7.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 770 op/s; 21876/348027 objects degraded (6.286%) 2024-02-09T17:56:52.873 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:52 smithi179 ceph-mon[43975]: 1.17 scrub starts 2024-02-09T17:56:52.873 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:52 smithi179 ceph-mon[43975]: 8.5 scrub starts 2024-02-09T17:56:52.873 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:52 smithi179 ceph-mon[43975]: 8.3 scrub ok 2024-02-09T17:56:52.873 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:52 smithi179 ceph-mon[43975]: 1.17 scrub ok 2024-02-09T17:56:52.873 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:52 smithi179 ceph-mon[43975]: 8.7 scrub ok 2024-02-09T17:56:52.873 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:52 smithi179 ceph-mon[43975]: 8.d scrub starts 2024-02-09T17:56:52.873 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:52 smithi179 ceph-mon[43975]: 8.b scrub starts 2024-02-09T17:56:52.883 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:52.905 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:56:52.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:52.908 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:56:52.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:52 smithi162 ceph-mon[44304]: pgmap v2023: 47 pgs: 1 active+clean+scrubbing, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 7.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 770 op/s; 21876/348027 objects degraded (6.286%) 2024-02-09T17:56:52.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:52 smithi162 ceph-mon[44304]: 1.17 scrub starts 2024-02-09T17:56:52.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:52 smithi162 ceph-mon[44304]: 8.5 scrub starts 2024-02-09T17:56:52.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:52 smithi162 ceph-mon[44304]: 8.3 scrub ok 2024-02-09T17:56:52.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:52 smithi162 ceph-mon[44304]: 1.17 scrub ok 2024-02-09T17:56:52.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:52 smithi162 ceph-mon[44304]: 8.7 scrub ok 2024-02-09T17:56:52.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:52 smithi162 ceph-mon[44304]: 8.d scrub starts 2024-02-09T17:56:52.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:52 smithi162 ceph-mon[44304]: 8.b scrub starts 2024-02-09T17:56:53.009 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:56:53.062 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:53.081 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:56:53.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:53.085 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:56:53.186 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:56:53.214 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:53.239 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:56:53.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:53.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:56:53.344 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:56:53.372 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:53.395 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:56:53.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:53.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:56:53.502 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:56:53.580 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:53.593 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is set 2024-02-09T17:56:53.618 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:56:53.618 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 87 15 120427 120412 86.4712 40.125 0.0162546 0.0115596 2024-02-09T17:56:53.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:53.623 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:56:53.724 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:56:53.762 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:53.865 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:56:53.868 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:53 smithi099 ceph-mon[39420]: 1.19 scrub starts 2024-02-09T17:56:53.868 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:53 smithi099 ceph-mon[39420]: 1.19 scrub ok 2024-02-09T17:56:53.868 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:53 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/263383573' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:56:53.868 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:53 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:56:53.868 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:53 smithi099 ceph-mon[39420]: 1.1b scrub starts 2024-02-09T17:56:53.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:53.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:56:53.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:53 smithi162 ceph-mon[44304]: 1.19 scrub starts 2024-02-09T17:56:53.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:53 smithi162 ceph-mon[44304]: 1.19 scrub ok 2024-02-09T17:56:53.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:53 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/263383573' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:56:53.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:53 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:56:53.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:53 smithi162 ceph-mon[44304]: 1.1b scrub starts 2024-02-09T17:56:53.930 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:53 smithi179 ceph-mon[43975]: 1.19 scrub starts 2024-02-09T17:56:53.930 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:53 smithi179 ceph-mon[43975]: 1.19 scrub ok 2024-02-09T17:56:53.930 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:53 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/263383573' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:56:53.930 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:53 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:56:53.930 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:53 smithi179 ceph-mon[43975]: 1.1b scrub starts 2024-02-09T17:56:53.970 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:56:54.087 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:54.115 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:56:54.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:54.121 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:56:54.222 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:56:54.285 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:54.618 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 88 16 121343 121327 86.1385 57.1875 0.00894909 0.0116047 2024-02-09T17:56:54.878 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:54 smithi099 ceph-mon[39420]: pgmap v2024: 47 pgs: 3 active+clean+scrubbing, 3 active+undersized+degraded, 5 active+undersized, 36 active+clean; 7.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 854 op/s; 22562/355341 objects degraded (6.349%) 2024-02-09T17:56:54.879 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:54 smithi099 ceph-mon[39420]: 8.5 scrub ok 2024-02-09T17:56:54.879 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:54 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:56:54.879 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:54 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:56:54.879 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:54 smithi099 ceph-mon[39420]: 1.1 scrub starts 2024-02-09T17:56:54.879 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:54 smithi099 ceph-mon[39420]: 1.1b scrub ok 2024-02-09T17:56:54.879 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:54 smithi099 ceph-mon[39420]: 1.1 scrub ok 2024-02-09T17:56:54.879 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:54 smithi099 ceph-mon[39420]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:56:54.879 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:54 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:56:54.879 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:54 smithi099 ceph-mon[39420]: osdmap e821: 12 total, 11 up, 12 in 2024-02-09T17:56:54.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:54 smithi179 ceph-mon[43975]: pgmap v2024: 47 pgs: 3 active+clean+scrubbing, 3 active+undersized+degraded, 5 active+undersized, 36 active+clean; 7.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 854 op/s; 22562/355341 objects degraded (6.349%) 2024-02-09T17:56:54.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:54 smithi179 ceph-mon[43975]: 8.5 scrub ok 2024-02-09T17:56:54.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:54 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:56:54.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:54 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:56:54.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:54 smithi179 ceph-mon[43975]: 1.1 scrub starts 2024-02-09T17:56:54.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:54 smithi179 ceph-mon[43975]: 1.1b scrub ok 2024-02-09T17:56:54.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:54 smithi179 ceph-mon[43975]: 1.1 scrub ok 2024-02-09T17:56:54.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:54 smithi179 ceph-mon[43975]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:56:54.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:54 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:56:54.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:54 smithi179 ceph-mon[43975]: osdmap e821: 12 total, 11 up, 12 in 2024-02-09T17:56:54.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:54 smithi162 ceph-mon[44304]: pgmap v2024: 47 pgs: 3 active+clean+scrubbing, 3 active+undersized+degraded, 5 active+undersized, 36 active+clean; 7.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 854 op/s; 22562/355341 objects degraded (6.349%) 2024-02-09T17:56:54.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:54 smithi162 ceph-mon[44304]: 8.5 scrub ok 2024-02-09T17:56:54.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:54 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:56:54.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:54 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:56:54.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:54 smithi162 ceph-mon[44304]: 1.1 scrub starts 2024-02-09T17:56:54.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:54 smithi162 ceph-mon[44304]: 1.1b scrub ok 2024-02-09T17:56:54.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:54 smithi162 ceph-mon[44304]: 1.1 scrub ok 2024-02-09T17:56:54.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:54 smithi162 ceph-mon[44304]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:56:54.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:54 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:56:54.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:54 smithi162 ceph-mon[44304]: osdmap e821: 12 total, 11 up, 12 in 2024-02-09T17:56:55.618 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 89 16 122171 122155 85.7522 51.75 0.0061189 0.0116573 2024-02-09T17:56:55.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:55 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 22562/358164 objects degraded (6.299%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:56:55.925 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:56:55.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:55 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 22562/358164 objects degraded (6.299%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:56:55.934 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:56:55.939 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:56:55.942 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:55 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 22562/358164 objects degraded (6.299%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:56:55.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:55.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:56:55.952 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:56:55.983 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:56:56.010 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:56:56.042 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:56:56.044 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:56:56.081 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:56:56.121 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:56:56.122 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:56.159 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:56:56.195 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:56:56.206 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:56:56.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:56.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:56:56.244 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:56:56.282 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:56:56.283 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:56:56.283 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:56:56.283 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:56:56.283 INFO:teuthology.orchestra.run.smithi099.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-09T17:56:56.283 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:56:56.283 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:56:56.283 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:56:56.283 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:56:56.283 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:56:56.283 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:56:56.283 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:56:56.284 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:56:56.315 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:56:56.341 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:56.364 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:56:56.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:56.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:56:56.469 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:56:56.512 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:56.619 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Total time run: 90.0088 2024-02-09T17:56:56.619 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Total writes made: 123069 2024-02-09T17:56:56.619 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Write size: 65536 2024-02-09T17:56:56.619 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Object size: 65536 2024-02-09T17:56:56.619 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Bandwidth (MB/sec): 85.4562 2024-02-09T17:56:56.619 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Stddev Bandwidth: 37.903 2024-02-09T17:56:56.619 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Max bandwidth (MB/sec): 214.625 2024-02-09T17:56:56.619 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Min bandwidth (MB/sec): 8.125 2024-02-09T17:56:56.619 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Average IOPS: 1367 2024-02-09T17:56:56.619 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Stddev IOPS: 606.449 2024-02-09T17:56:56.619 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Max IOPS: 3434 2024-02-09T17:56:56.619 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Min IOPS: 130 2024-02-09T17:56:56.620 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Average Latency(s): 0.0116984 2024-02-09T17:56:56.620 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Stddev Latency(s): 0.0161659 2024-02-09T17:56:56.620 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Max latency(s): 1.06242 2024-02-09T17:56:56.620 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Min latency(s): 0.00109523 2024-02-09T17:56:56.624 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Cleaning up (deleting benchmark objects) 2024-02-09T17:56:56.655 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:56:56.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:56.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:56:56.760 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:56:56.797 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:56 smithi099 ceph-mon[39420]: pgmap v2026: 47 pgs: 4 active+clean+scrubbing, 3 active+undersized+degraded, 5 active+undersized, 35 active+clean; 7.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 MiB/s wr, 680 op/s; 22562/358164 objects degraded (6.299%) 2024-02-09T17:56:56.797 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:56 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3347146671' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:56:56.815 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:56.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:56 smithi179 ceph-mon[43975]: pgmap v2026: 47 pgs: 4 active+clean+scrubbing, 3 active+undersized+degraded, 5 active+undersized, 35 active+clean; 7.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 MiB/s wr, 680 op/s; 22562/358164 objects degraded (6.299%) 2024-02-09T17:56:56.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:56 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3347146671' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:56:56.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:56 smithi162 ceph-mon[44304]: pgmap v2026: 47 pgs: 4 active+clean+scrubbing, 3 active+undersized+degraded, 5 active+undersized, 35 active+clean; 7.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 MiB/s wr, 680 op/s; 22562/358164 objects degraded (6.299%) 2024-02-09T17:56:56.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:56 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3347146671' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:56:57.166 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-09T17:56:57.167 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-09T17:56:57.173 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:56:57.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:57.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:56:57.280 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:56:57.326 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:57.827 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-09T17:56:57.960 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:56:57 smithi099 ceph-mon[39420]: pgmap v2027: 47 pgs: 2 active+clean+scrubbing, 3 active+undersized+degraded, 5 active+undersized, 37 active+clean; 7.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s wr, 805 op/s; 23209/365136 objects degraded (6.356%) 2024-02-09T17:56:58.091 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:56:58.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:58.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:56:58.199 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:56:58.224 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:56:57 smithi162 ceph-mon[44304]: pgmap v2027: 47 pgs: 2 active+clean+scrubbing, 3 active+undersized+degraded, 5 active+undersized, 37 active+clean; 7.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s wr, 805 op/s; 23209/365136 objects degraded (6.356%) 2024-02-09T17:56:58.226 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:58.251 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:56:58.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:58.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:56:58.356 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:56:58.406 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:56:57 smithi179 ceph-mon[43975]: pgmap v2027: 47 pgs: 2 active+clean+scrubbing, 3 active+undersized+degraded, 5 active+undersized, 37 active+clean; 7.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s wr, 805 op/s; 23209/365136 objects degraded (6.356%) 2024-02-09T17:56:58.407 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:58.448 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:56:58.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:58.455 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:56:58.556 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:56:58.569 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:56:58.605 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:58.702 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:56:58.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:58.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:56:58.811 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:56:58.847 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:58.880 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:56:58.880 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:56:58.890 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:56:58.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:58.894 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:56:58.996 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:56:59.023 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:59.047 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:56:59.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:59.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:56:59.154 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:56:59.168 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:56:59.199 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:59.316 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:56:59.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:59.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:56:59.428 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:56:59.458 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:59.478 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:56:59.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:59.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:56:59.582 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:56:59.618 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:56:59.758 INFO:teuthology.orchestra.run.smithi099.stdout:8.c 2024-02-09T17:56:59.758 INFO:teuthology.orchestra.run.smithi099.stdout:8.8 2024-02-09T17:56:59.758 INFO:teuthology.orchestra.run.smithi099.stdout:1.1e 2024-02-09T17:56:59.759 INFO:teuthology.orchestra.run.smithi099.stdout:1.c 2024-02-09T17:56:59.759 INFO:teuthology.orchestra.run.smithi099.stdout:1.e 2024-02-09T17:56:59.759 INFO:teuthology.orchestra.run.smithi099.stdout:1.16 2024-02-09T17:56:59.759 INFO:teuthology.orchestra.run.smithi099.stdout:1.8 2024-02-09T17:56:59.759 INFO:teuthology.orchestra.run.smithi099.stdout:8.0 2024-02-09T17:56:59.977 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:56:59.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:56:59.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:57:00.011 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-09T17:57:00.085 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:57:00.149 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:00.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:00 smithi179 ceph-mon[43975]: pgmap v2028: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 7.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 46 MiB/s wr, 805 op/s; 23209/366708 objects degraded (6.329%) 2024-02-09T17:57:00.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:00 smithi162 ceph-mon[44304]: pgmap v2028: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 7.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 46 MiB/s wr, 805 op/s; 23209/366708 objects degraded (6.329%) 2024-02-09T17:57:00.729 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:57:00.734 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:00 smithi099 ceph-mon[39420]: pgmap v2028: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 7.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 46 MiB/s wr, 805 op/s; 23209/366708 objects degraded (6.329%) 2024-02-09T17:57:00.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:00.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:57:00.837 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:57:00.899 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:00.937 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:57:00.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:00.949 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:57:01.050 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:57:01.101 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:01.126 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:57:01.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:01.136 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:57:01.237 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:57:01.284 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:01.314 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:57:01.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:01.323 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:57:01.424 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:57:01.460 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:01.480 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:57:01.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:01.483 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:57:01.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:01 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 23209/365679 objects degraded (6.347%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:57:01.583 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:57:01.608 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:01 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 23209/365679 objects degraded (6.347%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:57:01.610 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:01.632 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:57:01.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:01.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:57:01.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:01 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 23209/365679 objects degraded (6.347%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:57:01.737 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:57:01.768 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:01.790 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:57:01.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:01.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:57:01.809 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell -n osd.3 --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --op export --pgid 8.8 --file /var/log/ceph/exp.8.8.3 2024-02-09T17:57:01.894 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:57:01.933 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:02.064 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:57:02.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:02.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:57:02.081 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:57:02.113 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/osd.3/config 2024-02-09T17:57:02.170 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:57:02.214 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:02.245 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:57:02.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:02.259 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:57:02.360 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:57:02.410 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:02 smithi099 ceph-mon[39420]: pgmap v2029: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 7.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 814 op/s; 23209/365679 objects degraded (6.347%) 2024-02-09T17:57:02.431 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:02.467 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:57:02.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:02.470 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:57:02.471 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:02 smithi179 ceph-mon[43975]: pgmap v2029: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 7.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 814 op/s; 23209/365679 objects degraded (6.347%) 2024-02-09T17:57:02.572 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:57:02.635 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:02.670 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:57:02.671 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:57:02.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:02.674 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:57:02.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:02 smithi162 ceph-mon[44304]: pgmap v2029: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 7.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 814 op/s; 23209/365679 objects degraded (6.347%) 2024-02-09T17:57:02.775 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:57:02.812 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:03.029 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:57:03.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:03.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:57:03.137 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:57:03.177 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:03.342 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is set 2024-02-09T17:57:03.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:03 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1834388242' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:57:03.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:03 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1834388242' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:57:03.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:03 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1834388242' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:57:04.445 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:57:04.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:04.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:57:04.548 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:57:04.586 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:04.587 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:04 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:57:04.587 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:04 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:57:04.587 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:04 smithi179 ceph-mon[43975]: pgmap v2030: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 7.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 29 MiB/s wr, 872 op/s; 22245/356913 objects degraded (6.233%) 2024-02-09T17:57:04.587 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:04 smithi179 ceph-mon[43975]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:57:04.587 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:04 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1834388242' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:57:04.587 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:04 smithi179 ceph-mon[43975]: osdmap e822: 12 total, 11 up, 12 in 2024-02-09T17:57:04.614 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:57:04.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:04.618 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:57:04.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:04 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:57:04.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:04 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:57:04.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:04 smithi162 ceph-mon[44304]: pgmap v2030: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 7.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 29 MiB/s wr, 872 op/s; 22245/356913 objects degraded (6.233%) 2024-02-09T17:57:04.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:04 smithi162 ceph-mon[44304]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:57:04.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:04 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1834388242' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:57:04.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:04 smithi162 ceph-mon[44304]: osdmap e822: 12 total, 11 up, 12 in 2024-02-09T17:57:04.719 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:57:04.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:04 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:57:04.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:04 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:57:04.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:04 smithi099 ceph-mon[39420]: pgmap v2030: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 7.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 29 MiB/s wr, 872 op/s; 22245/356913 objects degraded (6.233%) 2024-02-09T17:57:04.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:04 smithi099 ceph-mon[39420]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:57:04.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:04 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1834388242' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:57:04.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:04 smithi099 ceph-mon[39420]: osdmap e822: 12 total, 11 up, 12 in 2024-02-09T17:57:04.750 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:04.775 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:57:04.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:04.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:57:04.879 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:57:04.943 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:04.978 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:57:04.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:04.983 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:57:05.084 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:57:05.122 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:05.147 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:57:05.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:05.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:57:05.253 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:57:05.281 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:05.306 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:57:05.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:05.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:57:05.410 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:57:05.438 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:05.463 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:57:05.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:05.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:57:05.568 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:57:05.614 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:05.650 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:57:05.657 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:05.657 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:57:05.758 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:57:05.788 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:05.808 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:57:05.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:05.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:57:05.914 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:57:05.968 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:06.001 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:57:06.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:06.011 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:57:06.112 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:57:06.141 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:06.165 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:57:06.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:06.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:57:06.269 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:57:06.307 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:06.335 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:57:06.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:06.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:57:06.354 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:06 smithi179 ceph-mon[43975]: pgmap v2032: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 7.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 23 MiB/s wr, 899 op/s; 22245/353280 objects degraded (6.297%) 2024-02-09T17:57:06.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:06 smithi162 ceph-mon[44304]: pgmap v2032: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 7.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 23 MiB/s wr, 899 op/s; 22245/353280 objects degraded (6.297%) 2024-02-09T17:57:06.439 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:57:06.492 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:06.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:06.526 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:57:06.526 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:57:06.589 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:57:06.589 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:57:06.616 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:06 smithi099 ceph-mon[39420]: pgmap v2032: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 7.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 23 MiB/s wr, 899 op/s; 22245/353280 objects degraded (6.297%) 2024-02-09T17:57:06.628 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:57:06.657 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:06.683 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:57:06.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:06.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:57:06.789 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:57:06.818 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:06.840 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:57:06.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:06.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:57:06.944 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:57:06.972 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:06.997 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:57:07.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:07.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:57:07.101 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:57:07.151 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:07.172 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset noscrub 2024-02-09T17:57:07.173 INFO:teuthology.orchestra.run.smithi099.stderr:Creating an OSD daemon form without an OSD FSID value 2024-02-09T17:57:07.196 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:57:07.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:07.209 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:57:07.309 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:57:07.346 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:07.429 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:07 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:57:07.429 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:07 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:57:07.429 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:07 smithi179 ceph-mon[43975]: osdmap e823: 12 total, 11 up, 12 in 2024-02-09T17:57:07.429 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:07 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 22245/353280 objects degraded (6.297%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:57:07.466 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:57:07.472 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:07 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:57:07.472 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:07 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:57:07.472 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:07 smithi099 ceph-mon[39420]: osdmap e823: 12 total, 11 up, 12 in 2024-02-09T17:57:07.472 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:07 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 22245/353280 objects degraded (6.297%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:57:07.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:07.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:57:07.574 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:57:07.624 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:07.651 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:57:07.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:07.655 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:57:07.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:07 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:57:07.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:07 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:57:07.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:07 smithi162 ceph-mon[44304]: osdmap e823: 12 total, 11 up, 12 in 2024-02-09T17:57:07.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:07 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 22245/353280 objects degraded (6.297%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:57:07.755 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:57:07.783 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:07.805 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:57:07.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:07.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:57:07.913 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:57:07.972 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:08.004 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:57:08.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:08.013 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:57:08.114 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:57:08.158 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:08.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:08 smithi179 ceph-mon[43975]: pgmap v2034: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 7.0 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 1.4 MiB/s wr, 967 op/s; 21384/344541 objects degraded (6.207%) 2024-02-09T17:57:08.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:08 smithi162 ceph-mon[44304]: pgmap v2034: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 7.0 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 1.4 MiB/s wr, 967 op/s; 21384/344541 objects degraded (6.207%) 2024-02-09T17:57:08.708 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:57:08.717 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:57:08.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:08.727 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:08 smithi099 ceph-mon[39420]: pgmap v2034: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 7.0 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 1.4 MiB/s wr, 967 op/s; 21384/344541 objects degraded (6.207%) 2024-02-09T17:57:08.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:57:08.829 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:57:08.857 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:57:08.857 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:57:08.858 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:57:08.867 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:08.867 INFO:teuthology.orchestra.run.smithi099.stderr:Exporting 8.8 info 8.8( v 814'6895 (0'0,814'6895] local-lis/les=785/786 n=6895 ec=785/785 lis/c=785/785 les/c/f=786/786/0 sis=785) 2024-02-09T17:57:08.885 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:57:08.892 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:57:08.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:08.899 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:57:08.919 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10004783:::benchmark_data_smithi179_148205_object16613:head# 2024-02-09T17:57:08.920 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1000de04:::benchmark_data_smithi179_148205_object3158:head# 2024-02-09T17:57:08.920 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1001065b:::benchmark_data_smithi179_148205_object2013:head# 2024-02-09T17:57:08.921 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:100124c5:::benchmark_data_smithi179_148205_object16676:head# 2024-02-09T17:57:08.922 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1001d046:::benchmark_data_smithi179_148205_object91041:head# 2024-02-09T17:57:08.922 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1001d6b6:::benchmark_data_smithi179_148205_object34114:head# 2024-02-09T17:57:08.923 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1001e676:::benchmark_data_smithi179_148205_object27293:head# 2024-02-09T17:57:08.923 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1002eb7a:::benchmark_data_smithi179_148205_object50076:head# 2024-02-09T17:57:08.924 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:100323ca:::benchmark_data_smithi179_148205_object79498:head# 2024-02-09T17:57:08.924 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1003442b:::benchmark_data_smithi179_148205_object76905:head# 2024-02-09T17:57:08.925 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10037eae:::benchmark_data_smithi179_148205_object73374:head# 2024-02-09T17:57:08.925 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1003dd15:::benchmark_data_smithi179_148205_object28408:head# 2024-02-09T17:57:08.929 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10040b3f:::benchmark_data_smithi179_148205_object57136:head# 2024-02-09T17:57:08.929 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1004f176:::benchmark_data_smithi179_148205_object95899:head# 2024-02-09T17:57:08.929 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1005d34f:::benchmark_data_smithi179_148205_object60589:head# 2024-02-09T17:57:08.929 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:100653e4:::benchmark_data_smithi179_148205_object71293:head# 2024-02-09T17:57:08.930 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10069535:::benchmark_data_smithi179_148205_object2813:head# 2024-02-09T17:57:08.930 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1006f8d4:::benchmark_data_smithi179_148205_object104236:head# 2024-02-09T17:57:08.930 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1007f142:::benchmark_data_smithi179_148205_object78765:head# 2024-02-09T17:57:08.930 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10091873:::benchmark_data_smithi179_148205_object1283:head# 2024-02-09T17:57:08.930 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10092b16:::benchmark_data_smithi179_148205_object66859:head# 2024-02-09T17:57:08.931 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1009fae6:::benchmark_data_smithi179_148205_object8034:head# 2024-02-09T17:57:08.931 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:100a9dec:::benchmark_data_smithi179_148205_object58994:head# 2024-02-09T17:57:08.932 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:100aa9a6:::benchmark_data_smithi179_148205_object26963:head# 2024-02-09T17:57:08.932 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:100ad409:::benchmark_data_smithi179_148205_object67709:head# 2024-02-09T17:57:08.933 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:100bc17e:::benchmark_data_smithi179_148205_object24850:head# 2024-02-09T17:57:08.933 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:100bf449:::benchmark_data_smithi179_148205_object70834:head# 2024-02-09T17:57:08.936 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:100d880b:::benchmark_data_smithi179_148205_object56082:head# 2024-02-09T17:57:08.936 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:57:08.937 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:100eb35f:::benchmark_data_smithi179_148205_object53188:head# 2024-02-09T17:57:08.937 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:100f07ed:::benchmark_data_smithi179_148205_object85245:head# 2024-02-09T17:57:08.937 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:100f6489:::benchmark_data_smithi179_148205_object68676:head# 2024-02-09T17:57:08.941 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:100f9034:::benchmark_data_smithi179_148205_object27839:head# 2024-02-09T17:57:08.945 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:101117ef:::benchmark_data_smithi179_148205_object109673:head# 2024-02-09T17:57:08.954 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1011b636:::benchmark_data_smithi179_148205_object83822:head# 2024-02-09T17:57:08.955 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1012b51a:::benchmark_data_smithi179_148205_object50843:head# 2024-02-09T17:57:08.955 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1012c2b1:::benchmark_data_smithi179_148205_object109741:head# 2024-02-09T17:57:08.955 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10141a16:::benchmark_data_smithi179_148205_object25510:head# 2024-02-09T17:57:08.956 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10144508:::benchmark_data_smithi179_148205_object18840:head# 2024-02-09T17:57:08.956 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1014e8f4:::benchmark_data_smithi179_148205_object7068:head# 2024-02-09T17:57:08.957 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10162356:::benchmark_data_smithi179_148205_object105892:head# 2024-02-09T17:57:08.957 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1016a829:::benchmark_data_smithi179_148205_object7573:head# 2024-02-09T17:57:08.957 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1016b6b7:::benchmark_data_smithi179_148205_object65338:head# 2024-02-09T17:57:08.958 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10176357:::benchmark_data_smithi179_148205_object22758:head# 2024-02-09T17:57:08.958 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10191fac:::benchmark_data_smithi179_148205_object82356:head# 2024-02-09T17:57:08.959 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:101a336a:::benchmark_data_smithi179_148205_object58970:head# 2024-02-09T17:57:08.959 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:101bdd4f:::benchmark_data_smithi179_148205_object46665:head# 2024-02-09T17:57:08.964 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:101c5fe7:::benchmark_data_smithi179_148205_object70054:head# 2024-02-09T17:57:08.964 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:101cab72:::benchmark_data_smithi179_148205_object29448:head# 2024-02-09T17:57:08.964 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:101cfbd8:::benchmark_data_smithi179_148205_object23037:head# 2024-02-09T17:57:08.964 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:101d55b4:::benchmark_data_smithi179_148205_object104880:head# 2024-02-09T17:57:08.964 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:101e8e09:::benchmark_data_smithi179_148205_object9173:head# 2024-02-09T17:57:08.964 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:101fb866:::benchmark_data_smithi179_148205_object60498:head# 2024-02-09T17:57:08.964 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:101fc0d2:::benchmark_data_smithi179_148205_object23908:head# 2024-02-09T17:57:08.964 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:101fe793:::benchmark_data_smithi179_148205_object74821:head# 2024-02-09T17:57:08.964 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10210286:::benchmark_data_smithi179_148205_object24991:head# 2024-02-09T17:57:08.964 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10222c8e:::benchmark_data_smithi179_148205_object100270:head# 2024-02-09T17:57:08.967 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1022584b:::benchmark_data_smithi179_148205_object60307:head# 2024-02-09T17:57:08.968 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1022bbd8:::benchmark_data_smithi179_148205_object4322:head# 2024-02-09T17:57:08.968 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1022d752:::benchmark_data_smithi179_148205_object95520:head# 2024-02-09T17:57:08.968 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1022de95:::benchmark_data_smithi179_148205_object1005:head# 2024-02-09T17:57:08.968 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1022ee2c:::benchmark_data_smithi179_148205_object69312:head# 2024-02-09T17:57:08.968 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1024f965:::benchmark_data_smithi179_148205_object56847:head# 2024-02-09T17:57:08.968 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:57:08.972 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1026beef:::benchmark_data_smithi179_148205_object55363:head# 2024-02-09T17:57:08.972 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:102777dc:::benchmark_data_smithi179_148205_object32244:head# 2024-02-09T17:57:08.972 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1027eba9:::benchmark_data_smithi179_148205_object108510:head# 2024-02-09T17:57:08.980 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1028a1ce:::benchmark_data_smithi179_148205_object87474:head# 2024-02-09T17:57:08.980 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1028a347:::benchmark_data_smithi179_148205_object9557:head# 2024-02-09T17:57:08.980 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1029b40e:::benchmark_data_smithi179_148205_object5067:head# 2024-02-09T17:57:08.985 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1029f6e2:::benchmark_data_smithi179_148205_object92328:head# 2024-02-09T17:57:08.985 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:102a04d0:::benchmark_data_smithi179_148205_object66697:head# 2024-02-09T17:57:08.985 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:102a40c5:::benchmark_data_smithi179_148205_object79668:head# 2024-02-09T17:57:08.988 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:102b18b6:::benchmark_data_smithi179_148205_object66935:head# 2024-02-09T17:57:08.988 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:102b56a7:::benchmark_data_smithi179_148205_object62012:head# 2024-02-09T17:57:08.988 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:102c6c37:::benchmark_data_smithi179_148205_object80291:head# 2024-02-09T17:57:08.988 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:102c752e:::benchmark_data_smithi179_148205_object73103:head# 2024-02-09T17:57:08.988 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:102d4a7b:::benchmark_data_smithi179_148205_object81704:head# 2024-02-09T17:57:08.989 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:102ea5da:::benchmark_data_smithi179_148205_object51319:head# 2024-02-09T17:57:08.989 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:102edfb6:::benchmark_data_smithi179_148205_object32950:head# 2024-02-09T17:57:08.989 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:102ef2c1:::benchmark_data_smithi179_148205_object39771:head# 2024-02-09T17:57:08.995 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:102fa2cb:::benchmark_data_smithi179_148205_object16628:head# 2024-02-09T17:57:08.995 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1031e579:::benchmark_data_smithi179_148205_object18064:head# 2024-02-09T17:57:08.995 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10325019:::benchmark_data_smithi179_148205_object41333:head# 2024-02-09T17:57:08.995 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10339215:::benchmark_data_smithi179_148205_object67260:head# 2024-02-09T17:57:08.995 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10347cb3:::benchmark_data_smithi179_148205_object61276:head# 2024-02-09T17:57:08.995 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1034c57e:::benchmark_data_smithi179_148205_object108915:head# 2024-02-09T17:57:08.996 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1035939c:::benchmark_data_smithi179_148205_object20019:head# 2024-02-09T17:57:08.996 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10359e88:::benchmark_data_smithi179_148205_object1776:head# 2024-02-09T17:57:08.996 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1035cc6d:::benchmark_data_smithi179_148205_object87594:head# 2024-02-09T17:57:08.996 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1036af69:::benchmark_data_smithi179_148205_object37875:head# 2024-02-09T17:57:08.996 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10374f61:::benchmark_data_smithi179_148205_object73757:head# 2024-02-09T17:57:08.999 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1037d519:::benchmark_data_smithi179_148205_object75655:head# 2024-02-09T17:57:08.999 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:103865f5:::benchmark_data_smithi179_148205_object90685:head# 2024-02-09T17:57:09.000 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1038aeca:::benchmark_data_smithi179_148205_object30801:head# 2024-02-09T17:57:09.000 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:57:09.001 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1038d441:::benchmark_data_smithi179_148205_object83824:head# 2024-02-09T17:57:09.004 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:57:09.005 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10392df8:::benchmark_data_smithi179_148205_object78758:head# 2024-02-09T17:57:09.005 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1039d25c:::benchmark_data_smithi179_148205_object100148:head# 2024-02-09T17:57:09.006 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1039ea26:::benchmark_data_smithi179_148205_object11149:head# 2024-02-09T17:57:09.006 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:103a878a:::benchmark_data_smithi179_148205_object56688:head# 2024-02-09T17:57:09.007 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:103a9213:::benchmark_data_smithi179_148205_object7825:head# 2024-02-09T17:57:09.007 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:103b696d:::benchmark_data_smithi179_148205_object23459:head# 2024-02-09T17:57:09.008 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:103b8264:::benchmark_data_smithi179_148205_object4343:head# 2024-02-09T17:57:09.008 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:103c9e21:::benchmark_data_smithi179_148205_object37100:head# 2024-02-09T17:57:09.008 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:103ce2e3:::benchmark_data_smithi179_148205_object14788:head# 2024-02-09T17:57:09.009 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:103cee49:::benchmark_data_smithi179_148205_object61480:head# 2024-02-09T17:57:09.009 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:103d4a90:::benchmark_data_smithi179_148205_object25633:head# 2024-02-09T17:57:09.010 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:103ef0d0:::benchmark_data_smithi179_148205_object32666:head# 2024-02-09T17:57:09.010 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:103fd84b:::benchmark_data_smithi179_148205_object12285:head# 2024-02-09T17:57:09.010 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:104017bd:::benchmark_data_smithi179_148205_object56716:head# 2024-02-09T17:57:09.011 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10411776:::benchmark_data_smithi179_148205_object24118:head# 2024-02-09T17:57:09.011 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10424a51:::benchmark_data_smithi179_148205_object9427:head# 2024-02-09T17:57:09.012 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10427451:::benchmark_data_smithi179_148205_object52220:head# 2024-02-09T17:57:09.012 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1042a47f:::benchmark_data_smithi179_148205_object65893:head# 2024-02-09T17:57:09.014 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10430f7f:::benchmark_data_smithi179_148205_object61983:head# 2024-02-09T17:57:09.014 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1043c57f:::benchmark_data_smithi179_148205_object103854:head# 2024-02-09T17:57:09.015 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10447c3a:::benchmark_data_smithi179_148205_object82266:head# 2024-02-09T17:57:09.015 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:104570cd:::benchmark_data_smithi179_148205_object74631:head# 2024-02-09T17:57:09.015 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1045d2df:::benchmark_data_smithi179_148205_object27264:head# 2024-02-09T17:57:09.020 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10461e34:::benchmark_data_smithi179_148205_object453:head# 2024-02-09T17:57:09.020 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1046e4f3:::benchmark_data_smithi179_148205_object41531:head# 2024-02-09T17:57:09.020 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:104747db:::benchmark_data_smithi179_148205_object73802:head# 2024-02-09T17:57:09.021 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:104753b7:::benchmark_data_smithi179_148205_object53745:head# 2024-02-09T17:57:09.021 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10482bd4:::benchmark_data_smithi179_148205_object102188:head# 2024-02-09T17:57:09.022 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:104891ef:::benchmark_data_smithi179_148205_object106174:head# 2024-02-09T17:57:09.027 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1048ced1:::benchmark_data_smithi179_148205_object2651:head# 2024-02-09T17:57:09.027 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:104920f1:::benchmark_data_smithi179_148205_object71269:head# 2024-02-09T17:57:09.027 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10496222:::benchmark_data_smithi179_148205_object84063:head# 2024-02-09T17:57:09.027 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:104972d0:::benchmark_data_smithi179_148205_object70456:head# 2024-02-09T17:57:09.032 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:104982b6:::benchmark_data_smithi179_148205_object91837:head# 2024-02-09T17:57:09.032 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1049f39b:::benchmark_data_smithi179_148205_object72988:head# 2024-02-09T17:57:09.032 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:104b5e63:::benchmark_data_smithi179_148205_object48336:head# 2024-02-09T17:57:09.033 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:104bca89:::benchmark_data_smithi179_148205_object431:head# 2024-02-09T17:57:09.033 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:104c313d:::benchmark_data_smithi179_148205_object85967:head# 2024-02-09T17:57:09.035 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:104c50d6:::benchmark_data_smithi179_148205_object87166:head# 2024-02-09T17:57:09.035 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:104cf713:::benchmark_data_smithi179_148205_object39444:head# 2024-02-09T17:57:09.035 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1050a65a:::benchmark_data_smithi179_148205_object43010:head# 2024-02-09T17:57:09.035 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1052518f:::benchmark_data_smithi179_148205_object9822:head# 2024-02-09T17:57:09.036 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10525ce5:::benchmark_data_smithi179_148205_object46551:head# 2024-02-09T17:57:09.037 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10529835:::benchmark_data_smithi179_148205_object46770:head# 2024-02-09T17:57:09.037 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1053daa2:::benchmark_data_smithi179_148205_object40058:head# 2024-02-09T17:57:09.037 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:105419c1:::benchmark_data_smithi179_148205_object2883:head# 2024-02-09T17:57:09.038 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1054c604:::benchmark_data_smithi179_148205_object64401:head# 2024-02-09T17:57:09.038 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1054e742:::benchmark_data_smithi179_148205_object71580:head# 2024-02-09T17:57:09.038 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10555fc9:::benchmark_data_smithi179_148205_object20681:head# 2024-02-09T17:57:09.039 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1055f033:::benchmark_data_smithi179_148205_object85593:head# 2024-02-09T17:57:09.039 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1055f5c5:::benchmark_data_smithi179_148205_object99768:head# 2024-02-09T17:57:09.040 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10567d14:::benchmark_data_smithi179_148205_object9973:head# 2024-02-09T17:57:09.040 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10568111:::benchmark_data_smithi179_148205_object90527:head# 2024-02-09T17:57:09.041 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1056f1aa:::benchmark_data_smithi179_148205_object52615:head# 2024-02-09T17:57:09.041 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1057dacd:::benchmark_data_smithi179_148205_object103853:head# 2024-02-09T17:57:09.041 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:105ac0da:::benchmark_data_smithi179_148205_object76012:head# 2024-02-09T17:57:09.042 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:105b27ef:::benchmark_data_smithi179_148205_object66053:head# 2024-02-09T17:57:09.042 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:105b6259:::benchmark_data_smithi179_148205_object46670:head# 2024-02-09T17:57:09.043 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:105bd60f:::benchmark_data_smithi179_148205_object68809:head# 2024-02-09T17:57:09.047 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:105c013f:::benchmark_data_smithi179_148205_object17747:head# 2024-02-09T17:57:09.047 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:105c499b:::benchmark_data_smithi179_148205_object106921:head# 2024-02-09T17:57:09.047 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:105ce8b3:::benchmark_data_smithi179_148205_object48425:head# 2024-02-09T17:57:09.048 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:105d0f6d:::benchmark_data_smithi179_148205_object21913:head# 2024-02-09T17:57:09.048 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:57:09.049 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:105d725c:::benchmark_data_smithi179_148205_object13775:head# 2024-02-09T17:57:09.049 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:105ec253:::benchmark_data_smithi179_148205_object108904:head# 2024-02-09T17:57:09.049 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1060156d:::benchmark_data_smithi179_148205_object38098:head# 2024-02-09T17:57:09.049 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10607892:::benchmark_data_smithi179_148205_object87694:head# 2024-02-09T17:57:09.049 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1060d002:::benchmark_data_smithi179_148205_object94000:head# 2024-02-09T17:57:09.049 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10611667:::benchmark_data_smithi179_148205_object95558:head# 2024-02-09T17:57:09.049 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10612699:::benchmark_data_smithi179_148205_object89583:head# 2024-02-09T17:57:09.049 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1061487a:::benchmark_data_smithi179_148205_object22704:head# 2024-02-09T17:57:09.049 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10616ad3:::benchmark_data_smithi179_148205_object106523:head# 2024-02-09T17:57:09.050 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1061adb6:::benchmark_data_smithi179_148205_object14373:head# 2024-02-09T17:57:09.051 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:106368fa:::benchmark_data_smithi179_148205_object33970:head# 2024-02-09T17:57:09.051 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10637098:::benchmark_data_smithi179_148205_object68275:head# 2024-02-09T17:57:09.052 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:106381d4:::benchmark_data_smithi179_148205_object46427:head# 2024-02-09T17:57:09.052 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:09.052 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1063ce4c:::benchmark_data_smithi179_148205_object11998:head# 2024-02-09T17:57:09.053 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10654a10:::benchmark_data_smithi179_148205_object109359:head# 2024-02-09T17:57:09.053 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10662cc2:::benchmark_data_smithi179_148205_object54016:head# 2024-02-09T17:57:09.053 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1066592a:::benchmark_data_smithi179_148205_object4623:head# 2024-02-09T17:57:09.054 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:106704d1:::benchmark_data_smithi179_148205_object85380:head# 2024-02-09T17:57:09.054 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10677445:::benchmark_data_smithi179_148205_object66933:head# 2024-02-09T17:57:09.055 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1068a47b:::benchmark_data_smithi179_148205_object68114:head# 2024-02-09T17:57:09.055 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:106a42e8:::benchmark_data_smithi179_148205_object65700:head# 2024-02-09T17:57:09.055 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:106ae7c9:::benchmark_data_smithi179_148205_object50645:head# 2024-02-09T17:57:09.056 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:106b5997:::benchmark_data_smithi179_148205_object96328:head# 2024-02-09T17:57:09.056 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:106b5f0a:::benchmark_data_smithi179_148205_object57424:head# 2024-02-09T17:57:09.061 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:106b8e3e:::benchmark_data_smithi179_148205_object31886:head# 2024-02-09T17:57:09.061 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:106b8efc:::benchmark_data_smithi179_148205_object73480:head# 2024-02-09T17:57:09.061 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:106bb4f5:::benchmark_data_smithi179_148205_object49843:head# 2024-02-09T17:57:09.061 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:106bb8ff:::benchmark_data_smithi179_148205_object57307:head# 2024-02-09T17:57:09.061 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:106bf104:::benchmark_data_smithi179_148205_object30940:head# 2024-02-09T17:57:09.061 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:106cc65c:::benchmark_data_smithi179_148205_object74645:head# 2024-02-09T17:57:09.061 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:106e85de:::benchmark_data_smithi179_148205_object10913:head# 2024-02-09T17:57:09.061 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:106eefb8:::benchmark_data_smithi179_148205_object52819:head# 2024-02-09T17:57:09.061 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:106f82ca:::benchmark_data_smithi179_148205_object87056:head# 2024-02-09T17:57:09.062 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10717cea:::benchmark_data_smithi179_148205_object60180:head# 2024-02-09T17:57:09.062 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10718e63:::benchmark_data_smithi179_148205_object3263:head# 2024-02-09T17:57:09.062 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:107218f4:::benchmark_data_smithi179_148205_object12297:head# 2024-02-09T17:57:09.065 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10721be6:::benchmark_data_smithi179_148205_object81332:head# 2024-02-09T17:57:09.066 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1072496f:::benchmark_data_smithi179_148205_object60725:head# 2024-02-09T17:57:09.066 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1072611a:::benchmark_data_smithi179_148205_object5901:head# 2024-02-09T17:57:09.067 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1072ed65:::benchmark_data_smithi179_148205_object99653:head# 2024-02-09T17:57:09.068 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1073d06c:::benchmark_data_smithi179_148205_object65377:head# 2024-02-09T17:57:09.068 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:107413e3:::benchmark_data_smithi179_148205_object73497:head# 2024-02-09T17:57:09.068 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10749031:::benchmark_data_smithi179_148205_object71156:head# 2024-02-09T17:57:09.074 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1074e88e:::benchmark_data_smithi179_148205_object50143:head# 2024-02-09T17:57:09.074 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1075a3f5:::benchmark_data_smithi179_148205_object14269:head# 2024-02-09T17:57:09.074 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1078155e:::benchmark_data_smithi179_148205_object41120:head# 2024-02-09T17:57:09.074 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:107a183e:::benchmark_data_smithi179_148205_object92303:head# 2024-02-09T17:57:09.074 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:107b2273:::benchmark_data_smithi179_148205_object44822:head# 2024-02-09T17:57:09.074 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:107b45ce:::benchmark_data_smithi179_148205_object75334:head# 2024-02-09T17:57:09.074 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:107c71ed:::benchmark_data_smithi179_148205_object83786:head# 2024-02-09T17:57:09.074 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:107c896a:::benchmark_data_smithi179_148205_object7464:head# 2024-02-09T17:57:09.074 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:107ca97a:::benchmark_data_smithi179_148205_object4984:head# 2024-02-09T17:57:09.075 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:107cd204:::benchmark_data_smithi179_148205_object35595:head# 2024-02-09T17:57:09.075 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:107ef565:::benchmark_data_smithi179_148205_object90016:head# 2024-02-09T17:57:09.079 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:107fd742:::benchmark_data_smithi179_148205_object89218:head# 2024-02-09T17:57:09.083 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:108007f2:::benchmark_data_smithi179_148205_object28218:head# 2024-02-09T17:57:09.083 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10803eb6:::benchmark_data_smithi179_148205_object45345:head# 2024-02-09T17:57:09.083 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10806f6b:::benchmark_data_smithi179_148205_object8734:head# 2024-02-09T17:57:09.083 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:57:09.084 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1080ca19:::benchmark_data_smithi179_148205_object51899:head# 2024-02-09T17:57:09.084 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:108145e7:::benchmark_data_smithi179_148205_object12677:head# 2024-02-09T17:57:09.085 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10817a65:::benchmark_data_smithi179_148205_object99356:head# 2024-02-09T17:57:09.090 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1081ad02:::benchmark_data_smithi179_148205_object50075:head# 2024-02-09T17:57:09.090 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1081c184:::benchmark_data_smithi179_148205_object5756:head# 2024-02-09T17:57:09.090 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1082abda:::benchmark_data_smithi179_148205_object7240:head# 2024-02-09T17:57:09.090 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1082e05e:::benchmark_data_smithi179_148205_object96560:head# 2024-02-09T17:57:09.090 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:108388a6:::benchmark_data_smithi179_148205_object101610:head# 2024-02-09T17:57:09.090 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10845389:::benchmark_data_smithi179_148205_object26551:head# 2024-02-09T17:57:09.090 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10853df4:::benchmark_data_smithi179_148205_object51648:head# 2024-02-09T17:57:09.090 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:108542b7:::benchmark_data_smithi179_148205_object85591:head# 2024-02-09T17:57:09.090 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10855bf9:::benchmark_data_smithi179_148205_object20833:head# 2024-02-09T17:57:09.090 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1086e651:::benchmark_data_smithi179_148205_object85837:head# 2024-02-09T17:57:09.090 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1086fe2e:::benchmark_data_smithi179_148205_object42800:head# 2024-02-09T17:57:09.091 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10874a23:::benchmark_data_smithi179_148205_object34469:head# 2024-02-09T17:57:09.091 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1088e4cb:::benchmark_data_smithi179_148205_object11791:head# 2024-02-09T17:57:09.093 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:57:09.097 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1089446e:::benchmark_data_smithi179_148205_object19582:head# 2024-02-09T17:57:09.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:09.098 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:57:09.099 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10895fde:::benchmark_data_smithi179_148205_object61768:head# 2024-02-09T17:57:09.099 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:108ab7db:::benchmark_data_smithi179_148205_object11376:head# 2024-02-09T17:57:09.099 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:108c0bad:::benchmark_data_smithi179_148205_object23710:head# 2024-02-09T17:57:09.099 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:108c76c9:::benchmark_data_smithi179_148205_object70355:head# 2024-02-09T17:57:09.100 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:108d2575:::benchmark_data_smithi179_148205_object103391:head# 2024-02-09T17:57:09.101 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:108dc0d5:::benchmark_data_smithi179_148205_object11852:head# 2024-02-09T17:57:09.101 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:108f3ba0:::benchmark_data_smithi179_148205_object18854:head# 2024-02-09T17:57:09.101 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:108fa92b:::benchmark_data_smithi179_148205_object102341:head# 2024-02-09T17:57:09.102 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:108ff225:::benchmark_data_smithi179_148205_object92867:head# 2024-02-09T17:57:09.102 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10900120:::benchmark_data_smithi179_148205_object86251:head# 2024-02-09T17:57:09.103 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10913344:::benchmark_data_smithi179_148205_object62825:head# 2024-02-09T17:57:09.103 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1092ea6c:::benchmark_data_smithi179_148205_object53762:head# 2024-02-09T17:57:09.103 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1092eb7e:::benchmark_data_smithi179_148205_object12260:head# 2024-02-09T17:57:09.104 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10936ade:::benchmark_data_smithi179_148205_object6103:head# 2024-02-09T17:57:09.107 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:109475b7:::benchmark_data_smithi179_148205_object58938:head# 2024-02-09T17:57:09.107 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:109539db:::benchmark_data_smithi179_148205_object66877:head# 2024-02-09T17:57:09.107 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1097acfa:::benchmark_data_smithi179_148205_object64959:head# 2024-02-09T17:57:09.108 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1097fdf3:::benchmark_data_smithi179_148205_object96434:head# 2024-02-09T17:57:09.109 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1098eb7e:::benchmark_data_smithi179_148205_object64398:head# 2024-02-09T17:57:09.110 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:109a09b5:::benchmark_data_smithi179_148205_object5274:head# 2024-02-09T17:57:09.110 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:109b3396:::benchmark_data_smithi179_148205_object8418:head# 2024-02-09T17:57:09.111 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:109b33a3:::benchmark_data_smithi179_148205_object48852:head# 2024-02-09T17:57:09.111 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:109b56d6:::benchmark_data_smithi179_148205_object83409:head# 2024-02-09T17:57:09.112 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:109b7044:::benchmark_data_smithi179_148205_object107282:head# 2024-02-09T17:57:09.112 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:109bdc35:::benchmark_data_smithi179_148205_object38901:head# 2024-02-09T17:57:09.112 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:109beefd:::benchmark_data_smithi179_148205_object81947:head# 2024-02-09T17:57:09.113 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:109ca9f5:::benchmark_data_smithi179_148205_object89204:head# 2024-02-09T17:57:09.113 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:109d6f71:::benchmark_data_smithi179_148205_object29228:head# 2024-02-09T17:57:09.116 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:57:09.116 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:109decb3:::benchmark_data_smithi179_148205_object61896:head# 2024-02-09T17:57:09.117 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:109e920d:::benchmark_data_smithi179_148205_object47277:head# 2024-02-09T17:57:09.117 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:109f4ea9:::benchmark_data_smithi179_148205_object63301:head# 2024-02-09T17:57:09.119 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:109f7abe:::benchmark_data_smithi179_148205_object51094:head# 2024-02-09T17:57:09.120 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10a0181f:::benchmark_data_smithi179_148205_object62732:head# 2024-02-09T17:57:09.120 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10a03fce:::benchmark_data_smithi179_148205_object51257:head# 2024-02-09T17:57:09.120 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10a0f889:::benchmark_data_smithi179_148205_object103887:head# 2024-02-09T17:57:09.121 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10a13615:::benchmark_data_smithi179_148205_object81580:head# 2024-02-09T17:57:09.126 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10a1aa40:::benchmark_data_smithi179_148205_object58434:head# 2024-02-09T17:57:09.126 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10a1bc9b:::benchmark_data_smithi179_148205_object89823:head# 2024-02-09T17:57:09.126 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10a2054c:::benchmark_data_smithi179_148205_object11142:head# 2024-02-09T17:57:09.126 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10a276ac:::benchmark_data_smithi179_148205_object46498:head# 2024-02-09T17:57:09.126 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10a2c45b:::benchmark_data_smithi179_148205_object8568:head# 2024-02-09T17:57:09.127 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10a3dc6b:::benchmark_data_smithi179_148205_object109851:head# 2024-02-09T17:57:09.127 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10a53152:::benchmark_data_smithi179_148205_object61307:head# 2024-02-09T17:57:09.127 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10a5bdc6:::benchmark_data_smithi179_148205_object92287:head# 2024-02-09T17:57:09.127 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10a64b08:::benchmark_data_smithi179_148205_object89429:head# 2024-02-09T17:57:09.127 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10a693be:::benchmark_data_smithi179_148205_object71264:head# 2024-02-09T17:57:09.127 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10a6f7ee:::benchmark_data_smithi179_148205_object27129:head# 2024-02-09T17:57:09.129 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10a7ec68:::benchmark_data_smithi179_148205_object73166:head# 2024-02-09T17:57:09.129 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10a80091:::benchmark_data_smithi179_148205_object50881:head# 2024-02-09T17:57:09.129 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10a82969:::benchmark_data_smithi179_148205_object109797:head# 2024-02-09T17:57:09.130 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10a8a0e5:::benchmark_data_smithi179_148205_object70817:head# 2024-02-09T17:57:09.130 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10a8abe0:::benchmark_data_smithi179_148205_object82520:head# 2024-02-09T17:57:09.131 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10abf644:::benchmark_data_smithi179_148205_object13397:head# 2024-02-09T17:57:09.141 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10acf62d:::benchmark_data_smithi179_148205_object41596:head# 2024-02-09T17:57:09.141 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10ad7e71:::benchmark_data_smithi179_148205_object48759:head# 2024-02-09T17:57:09.141 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10ae55fb:::benchmark_data_smithi179_148205_object84430:head# 2024-02-09T17:57:09.141 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10ae805e:::benchmark_data_smithi179_148205_object109139:head# 2024-02-09T17:57:09.141 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10aef786:::benchmark_data_smithi179_148205_object42192:head# 2024-02-09T17:57:09.141 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10b09dd3:::benchmark_data_smithi179_148205_object14230:head# 2024-02-09T17:57:09.141 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10b10acc:::benchmark_data_smithi179_148205_object105209:head# 2024-02-09T17:57:09.141 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10b17fe3:::benchmark_data_smithi179_148205_object58130:head# 2024-02-09T17:57:09.141 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10b1febe:::benchmark_data_smithi179_148205_object103739:head# 2024-02-09T17:57:09.142 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10b258ab:::benchmark_data_smithi179_148205_object75953:head# 2024-02-09T17:57:09.142 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10b26864:::benchmark_data_smithi179_148205_object108376:head# 2024-02-09T17:57:09.143 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10b41c26:::benchmark_data_smithi179_148205_object49922:head# 2024-02-09T17:57:09.143 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10b4da9f:::benchmark_data_smithi179_148205_object92573:head# 2024-02-09T17:57:09.143 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10b500f2:::benchmark_data_smithi179_148205_object78854:head# 2024-02-09T17:57:09.145 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10b5c9c5:::benchmark_data_smithi179_148205_object74274:head# 2024-02-09T17:57:09.145 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10b6417c:::benchmark_data_smithi179_148205_object91195:head# 2024-02-09T17:57:09.146 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10b6652b:::benchmark_data_smithi179_148205_object85509:head# 2024-02-09T17:57:09.146 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10b9eff0:::benchmark_data_smithi179_148205_object31660:head# 2024-02-09T17:57:09.147 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10ba7484:::benchmark_data_smithi179_148205_object99107:head# 2024-02-09T17:57:09.147 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10bac11b:::benchmark_data_smithi179_148205_object79808:head# 2024-02-09T17:57:09.147 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10bb00d6:::benchmark_data_smithi179_148205_object94595:head# 2024-02-09T17:57:09.148 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10bb6133:::benchmark_data_smithi179_148205_object42599:head# 2024-02-09T17:57:09.149 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10bbbec6:::benchmark_data_smithi179_148205_object48946:head# 2024-02-09T17:57:09.154 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10bc2606:::benchmark_data_smithi179_148205_object36601:head# 2024-02-09T17:57:09.154 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10bc6969:::benchmark_data_smithi179_148205_object32003:head# 2024-02-09T17:57:09.154 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10bd9e4d:::benchmark_data_smithi179_148205_object85234:head# 2024-02-09T17:57:09.154 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10be32e9:::benchmark_data_smithi179_148205_object107409:head# 2024-02-09T17:57:09.154 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10bf54f1:::benchmark_data_smithi179_148205_object65518:head# 2024-02-09T17:57:09.154 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10bfc914:::benchmark_data_smithi179_148205_object103955:head# 2024-02-09T17:57:09.154 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10bfefa3:::benchmark_data_smithi179_148205_object106215:head# 2024-02-09T17:57:09.154 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10c0b970:::benchmark_data_smithi179_148205_object946:head# 2024-02-09T17:57:09.155 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:57:09.155 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:57:09.155 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:57:09.155 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:57:09.155 INFO:teuthology.orchestra.run.smithi099.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-09T17:57:09.155 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:57:09.155 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:57:09.156 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:57:09.156 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:57:09.156 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:57:09.156 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:57:09.156 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:57:09.156 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:57:09.156 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10c1841a:::benchmark_data_smithi179_148205_object31438:head# 2024-02-09T17:57:09.156 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10c1b9e6:::benchmark_data_smithi179_148205_object1186:head# 2024-02-09T17:57:09.157 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10c1f2a1:::benchmark_data_smithi179_148205_object74819:head# 2024-02-09T17:57:09.157 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10c2381a:::benchmark_data_smithi179_148205_object46721:head# 2024-02-09T17:57:09.157 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10c4e28f:::benchmark_data_smithi179_148205_object46228:head# 2024-02-09T17:57:09.157 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10c528d9:::benchmark_data_smithi179_148205_object15017:head# 2024-02-09T17:57:09.157 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10c56b11:::benchmark_data_smithi179_148205_object75965:head# 2024-02-09T17:57:09.158 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10c582e7:::benchmark_data_smithi179_148205_object89625:head# 2024-02-09T17:57:09.158 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10c68a2c:::benchmark_data_smithi179_148205_object66439:head# 2024-02-09T17:57:09.158 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10c6d490:::benchmark_data_smithi179_148205_object55428:head# 2024-02-09T17:57:09.158 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10c6eee7:::benchmark_data_smithi179_148205_object97334:head# 2024-02-09T17:57:09.159 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10c73d46:::benchmark_data_smithi179_148205_object12647:head# 2024-02-09T17:57:09.162 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10c74bba:::benchmark_data_smithi179_148205_object104084:head# 2024-02-09T17:57:09.163 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10c93e07:::benchmark_data_smithi179_148205_object107670:head# 2024-02-09T17:57:09.163 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10c98330:::benchmark_data_smithi179_148205_object11309:head# 2024-02-09T17:57:09.163 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10c9853a:::benchmark_data_smithi179_148205_object79318:head# 2024-02-09T17:57:09.165 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10c9b069:::benchmark_data_smithi179_148205_object71956:head# 2024-02-09T17:57:09.167 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10ca8297:::benchmark_data_smithi179_148205_object99066:head# 2024-02-09T17:57:09.167 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10ca9390:::benchmark_data_smithi179_148205_object107220:head# 2024-02-09T17:57:09.167 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10cb2132:::benchmark_data_smithi179_148205_object107015:head# 2024-02-09T17:57:09.168 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10cb278b:::benchmark_data_smithi179_148205_object17896:head# 2024-02-09T17:57:09.169 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10cb4d59:::benchmark_data_smithi179_148205_object175:head# 2024-02-09T17:57:09.169 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10cbc38d:::benchmark_data_smithi179_148205_object30958:head# 2024-02-09T17:57:09.169 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10cbeeee:::benchmark_data_smithi179_148205_object24190:head# 2024-02-09T17:57:09.171 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10ccd00d:::benchmark_data_smithi179_148205_object103072:head# 2024-02-09T17:57:09.171 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10cd08ec:::benchmark_data_smithi179_148205_object96280:head# 2024-02-09T17:57:09.171 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10cf0924:::benchmark_data_smithi179_148205_object15580:head# 2024-02-09T17:57:09.172 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10cf933c:::benchmark_data_smithi179_148205_object28430:head# 2024-02-09T17:57:09.173 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10cffccd:::benchmark_data_smithi179_148205_object43809:head# 2024-02-09T17:57:09.173 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10d0c9b8:::benchmark_data_smithi179_148205_object58534:head# 2024-02-09T17:57:09.177 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10d16dc7:::benchmark_data_smithi179_148205_object26671:head# 2024-02-09T17:57:09.178 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10d206e8:::benchmark_data_smithi179_148205_object77862:head# 2024-02-09T17:57:09.178 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10d2c566:::benchmark_data_smithi179_148205_object9030:head# 2024-02-09T17:57:09.178 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10d3180d:::benchmark_data_smithi179_148205_object84426:head# 2024-02-09T17:57:09.178 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10d3c3db:::benchmark_data_smithi179_148205_object43798:head# 2024-02-09T17:57:09.179 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10d45002:::benchmark_data_smithi179_148205_object62478:head# 2024-02-09T17:57:09.185 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10d46f54:::benchmark_data_smithi179_148205_object39360:head# 2024-02-09T17:57:09.185 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10d4f5f5:::benchmark_data_smithi179_148205_object33551:head# 2024-02-09T17:57:09.185 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10d65d83:::benchmark_data_smithi179_148205_object9659:head# 2024-02-09T17:57:09.185 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10d696e5:::benchmark_data_smithi179_148205_object8775:head# 2024-02-09T17:57:09.185 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10d7dc1a:::benchmark_data_smithi179_148205_object11201:head# 2024-02-09T17:57:09.185 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10d7f2f5:::benchmark_data_smithi179_148205_object67439:head# 2024-02-09T17:57:09.185 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10d83354:::benchmark_data_smithi179_148205_object74142:head# 2024-02-09T17:57:09.185 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10d88c40:::benchmark_data_smithi179_148205_object37985:head# 2024-02-09T17:57:09.186 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10d89730:::benchmark_data_smithi179_148205_object6547:head# 2024-02-09T17:57:09.186 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10d8a9a2:::benchmark_data_smithi179_148205_object55556:head# 2024-02-09T17:57:09.186 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10d8c303:::benchmark_data_smithi179_148205_object106341:head# 2024-02-09T17:57:09.193 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10d8d1bb:::benchmark_data_smithi179_148205_object50390:head# 2024-02-09T17:57:09.193 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10d97a2c:::benchmark_data_smithi179_148205_object36737:head# 2024-02-09T17:57:09.193 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10d995ac:::benchmark_data_smithi179_148205_object21537:head# 2024-02-09T17:57:09.194 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10dbd862:::benchmark_data_smithi179_148205_object47422:head# 2024-02-09T17:57:09.194 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10dbeedd:::benchmark_data_smithi179_148205_object47497:head# 2024-02-09T17:57:09.195 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10dcf7f0:::benchmark_data_smithi179_148205_object21374:head# 2024-02-09T17:57:09.198 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10dd8857:::benchmark_data_smithi179_148205_object102608:head# 2024-02-09T17:57:09.199 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:57:09.202 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10de5f9e:::benchmark_data_smithi179_148205_object60334:head# 2024-02-09T17:57:09.202 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10deed6f:::benchmark_data_smithi179_148205_object48374:head# 2024-02-09T17:57:09.212 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10e2273a:::benchmark_data_smithi179_148205_object97683:head# 2024-02-09T17:57:09.217 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10e2d216:::benchmark_data_smithi179_148205_object91759:head# 2024-02-09T17:57:09.217 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10e2f130:::benchmark_data_smithi179_148205_object38911:head# 2024-02-09T17:57:09.218 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10e31acc:::benchmark_data_smithi179_148205_object7311:head# 2024-02-09T17:57:09.218 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10e3a161:::benchmark_data_smithi179_148205_object53866:head# 2024-02-09T17:57:09.219 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10e3d7cd:::benchmark_data_smithi179_148205_object68977:head# 2024-02-09T17:57:09.227 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:09.229 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10e47723:::benchmark_data_smithi179_148205_object38511:head# 2024-02-09T17:57:09.230 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10e48e5d:::benchmark_data_smithi179_148205_object24423:head# 2024-02-09T17:57:09.230 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10e62598:::benchmark_data_smithi179_148205_object87030:head# 2024-02-09T17:57:09.230 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10e6b220:::benchmark_data_smithi179_148205_object46578:head# 2024-02-09T17:57:09.230 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10e969fd:::benchmark_data_smithi179_148205_object18853:head# 2024-02-09T17:57:09.230 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10e9d0d4:::benchmark_data_smithi179_148205_object86610:head# 2024-02-09T17:57:09.230 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10eaa187:::benchmark_data_smithi179_148205_object47208:head# 2024-02-09T17:57:09.230 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10eaf8b2:::benchmark_data_smithi179_148205_object17374:head# 2024-02-09T17:57:09.231 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10eb09bb:::benchmark_data_smithi179_148205_object92938:head# 2024-02-09T17:57:09.231 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10eb82ef:::benchmark_data_smithi179_148205_object9370:head# 2024-02-09T17:57:09.231 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10ebf49a:::benchmark_data_smithi179_148205_object68116:head# 2024-02-09T17:57:09.231 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10ec5941:::benchmark_data_smithi179_148205_object101979:head# 2024-02-09T17:57:09.231 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10ed80d4:::benchmark_data_smithi179_148205_object75820:head# 2024-02-09T17:57:09.232 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10ede2fc:::benchmark_data_smithi179_148205_object52626:head# 2024-02-09T17:57:09.235 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10edfed3:::benchmark_data_smithi179_148205_object11605:head# 2024-02-09T17:57:09.237 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10ee0f2c:::benchmark_data_smithi179_148205_object10546:head# 2024-02-09T17:57:09.241 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10ee24ad:::benchmark_data_smithi179_148205_object54643:head# 2024-02-09T17:57:09.242 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10eff587:::benchmark_data_smithi179_148205_object36723:head# 2024-02-09T17:57:09.242 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10f034f0:::benchmark_data_smithi179_148205_object41661:head# 2024-02-09T17:57:09.242 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10f1f59e:::benchmark_data_smithi179_148205_object4549:head# 2024-02-09T17:57:09.242 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10f26ea5:::benchmark_data_smithi179_148205_object35311:head# 2024-02-09T17:57:09.249 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:57:09.250 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10f28256:::benchmark_data_smithi179_148205_object12193:head# 2024-02-09T17:57:09.250 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10f2ee23:::benchmark_data_smithi179_148205_object2379:head# 2024-02-09T17:57:09.250 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10f33731:::benchmark_data_smithi179_148205_object68834:head# 2024-02-09T17:57:09.250 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10f3571f:::benchmark_data_smithi179_148205_object68715:head# 2024-02-09T17:57:09.250 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10f3a92c:::benchmark_data_smithi179_148205_object86300:head# 2024-02-09T17:57:09.250 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10f44e89:::benchmark_data_smithi179_148205_object96901:head# 2024-02-09T17:57:09.251 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10f4b93c:::benchmark_data_smithi179_148205_object103360:head# 2024-02-09T17:57:09.251 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10f5774f:::benchmark_data_smithi179_148205_object35005:head# 2024-02-09T17:57:09.251 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10f59dd0:::benchmark_data_smithi179_148205_object91474:head# 2024-02-09T17:57:09.252 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10f5d9a6:::benchmark_data_smithi179_148205_object43359:head# 2024-02-09T17:57:09.252 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10f6a394:::benchmark_data_smithi179_148205_object45164:head# 2024-02-09T17:57:09.252 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10f6c2f3:::benchmark_data_smithi179_148205_object79464:head# 2024-02-09T17:57:09.253 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10f7040a:::benchmark_data_smithi179_148205_object60583:head# 2024-02-09T17:57:09.253 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10f7094c:::benchmark_data_smithi179_148205_object95545:head# 2024-02-09T17:57:09.254 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10f7383c:::benchmark_data_smithi179_148205_object79620:head# 2024-02-09T17:57:09.254 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10f753e3:::benchmark_data_smithi179_148205_object103727:head# 2024-02-09T17:57:09.254 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10f78df6:::benchmark_data_smithi179_148205_object45838:head# 2024-02-09T17:57:09.255 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10f7b068:::benchmark_data_smithi179_148205_object41202:head# 2024-02-09T17:57:09.255 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10f85199:::benchmark_data_smithi179_148205_object22460:head# 2024-02-09T17:57:09.256 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:09.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:57:09.259 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10f8f0d2:::benchmark_data_smithi179_148205_object76461:head# 2024-02-09T17:57:09.260 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10f8fc26:::benchmark_data_smithi179_148205_object75423:head# 2024-02-09T17:57:09.260 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10f92960:::benchmark_data_smithi179_148205_object67682:head# 2024-02-09T17:57:09.260 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10f9b176:::benchmark_data_smithi179_148205_object49991:head# 2024-02-09T17:57:09.261 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10fa1cca:::benchmark_data_smithi179_148205_object40623:head# 2024-02-09T17:57:09.261 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10fa64d3:::benchmark_data_smithi179_148205_object47594:head# 2024-02-09T17:57:09.262 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10fb139b:::benchmark_data_smithi179_148205_object63352:head# 2024-02-09T17:57:09.262 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10fb4ecd:::benchmark_data_smithi179_148205_object105535:head# 2024-02-09T17:57:09.263 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10fbb9f0:::benchmark_data_smithi179_148205_object25541:head# 2024-02-09T17:57:09.263 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10fc416d:::benchmark_data_smithi179_148205_object104996:head# 2024-02-09T17:57:09.263 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10fc5581:::benchmark_data_smithi179_148205_object102747:head# 2024-02-09T17:57:09.264 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10fca2fa:::benchmark_data_smithi179_148205_object17813:head# 2024-02-09T17:57:09.264 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10fd1171:::benchmark_data_smithi179_148205_object29594:head# 2024-02-09T17:57:09.264 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10fd8a09:::benchmark_data_smithi179_148205_object62422:head# 2024-02-09T17:57:09.265 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10fda1ab:::benchmark_data_smithi179_148205_object19538:head# 2024-02-09T17:57:09.265 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10fdcbaa:::benchmark_data_smithi179_148205_object79655:head# 2024-02-09T17:57:09.265 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10feb25f:::benchmark_data_smithi179_148205_object92954:head# 2024-02-09T17:57:09.266 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10fed228:::benchmark_data_smithi179_148205_object42250:head# 2024-02-09T17:57:09.266 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10ff4e38:::benchmark_data_smithi179_148205_object48115:head# 2024-02-09T17:57:09.267 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:10ff5e58:::benchmark_data_smithi179_148205_object83587:head# 2024-02-09T17:57:09.267 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:110030e2:::benchmark_data_smithi179_148205_object85731:head# 2024-02-09T17:57:09.268 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:110040aa:::benchmark_data_smithi179_148205_object1860:head# 2024-02-09T17:57:09.268 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1102145f:::benchmark_data_smithi179_148205_object74788:head# 2024-02-09T17:57:09.268 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1102b1a3:::benchmark_data_smithi179_148205_object89106:head# 2024-02-09T17:57:09.269 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1102b59c:::benchmark_data_smithi179_148205_object51707:head# 2024-02-09T17:57:09.270 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1103177c:::benchmark_data_smithi179_148205_object13217:head# 2024-02-09T17:57:09.270 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1104e4a4:::benchmark_data_smithi179_148205_object99792:head# 2024-02-09T17:57:09.270 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11054a86:::benchmark_data_smithi179_148205_object52163:head# 2024-02-09T17:57:09.270 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11080117:::benchmark_data_smithi179_148205_object60084:head# 2024-02-09T17:57:09.271 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:110915a4:::benchmark_data_smithi179_148205_object45873:head# 2024-02-09T17:57:09.271 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:110929c2:::benchmark_data_smithi179_148205_object26587:head# 2024-02-09T17:57:09.271 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:110aaf56:::benchmark_data_smithi179_148205_object26241:head# 2024-02-09T17:57:09.272 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:110abba6:::benchmark_data_smithi179_148205_object69986:head# 2024-02-09T17:57:09.272 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:110abdee:::benchmark_data_smithi179_148205_object47365:head# 2024-02-09T17:57:09.273 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:110ac61a:::benchmark_data_smithi179_148205_object41873:head# 2024-02-09T17:57:09.273 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:110af88b:::benchmark_data_smithi179_148205_object55045:head# 2024-02-09T17:57:09.273 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:110b63aa:::benchmark_data_smithi179_148205_object104426:head# 2024-02-09T17:57:09.274 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:110c5423:::benchmark_data_smithi179_148205_object57574:head# 2024-02-09T17:57:09.274 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:110c9009:::benchmark_data_smithi179_148205_object37590:head# 2024-02-09T17:57:09.275 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:110d3160:::benchmark_data_smithi179_148205_object100147:head# 2024-02-09T17:57:09.275 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:110d914c:::benchmark_data_smithi179_148205_object74836:head# 2024-02-09T17:57:09.275 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:110e22f8:::benchmark_data_smithi179_148205_object879:head# 2024-02-09T17:57:09.276 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:110eca51:::benchmark_data_smithi179_148205_object11238:head# 2024-02-09T17:57:09.276 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1110810a:::benchmark_data_smithi179_148205_object95155:head# 2024-02-09T17:57:09.277 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1110969b:::benchmark_data_smithi179_148205_object18078:head# 2024-02-09T17:57:09.277 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11111dc9:::benchmark_data_smithi179_148205_object63145:head# 2024-02-09T17:57:09.277 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11112230:::benchmark_data_smithi179_148205_object34225:head# 2024-02-09T17:57:09.278 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11113154:::benchmark_data_smithi179_148205_object73947:head# 2024-02-09T17:57:09.279 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:111256dd:::benchmark_data_smithi179_148205_object65351:head# 2024-02-09T17:57:09.279 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11126eb2:::benchmark_data_smithi179_148205_object62634:head# 2024-02-09T17:57:09.279 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1112af7e:::benchmark_data_smithi179_148205_object48404:head# 2024-02-09T17:57:09.282 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1112f18d:::benchmark_data_smithi179_148205_object94826:head# 2024-02-09T17:57:09.282 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11135edb:::benchmark_data_smithi179_148205_object68291:head# 2024-02-09T17:57:09.282 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11172859:::benchmark_data_smithi179_148205_object13024:head# 2024-02-09T17:57:09.282 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:111780e6:::benchmark_data_smithi179_148205_object9940:head# 2024-02-09T17:57:09.282 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1117e03a:::benchmark_data_smithi179_148205_object80333:head# 2024-02-09T17:57:09.282 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11182ff4:::benchmark_data_smithi179_148205_object98285:head# 2024-02-09T17:57:09.283 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11183d46:::benchmark_data_smithi179_148205_object58990:head# 2024-02-09T17:57:09.283 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1118b073:::benchmark_data_smithi179_148205_object94139:head# 2024-02-09T17:57:09.284 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1118bacb:::benchmark_data_smithi179_148205_object27743:head# 2024-02-09T17:57:09.284 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11196763:::benchmark_data_smithi179_148205_object95457:head# 2024-02-09T17:57:09.284 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11199bcc:::benchmark_data_smithi179_148205_object91887:head# 2024-02-09T17:57:09.285 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:111acbab:::benchmark_data_smithi179_148205_object97294:head# 2024-02-09T17:57:09.286 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:111c1f7a:::benchmark_data_smithi179_148205_object99877:head# 2024-02-09T17:57:09.287 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:111d480d:::benchmark_data_smithi179_148205_object35236:head# 2024-02-09T17:57:09.287 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:111d502f:::benchmark_data_smithi179_148205_object5297:head# 2024-02-09T17:57:09.287 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:111e52b4:::benchmark_data_smithi179_148205_object109261:head# 2024-02-09T17:57:09.288 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:111f01cd:::benchmark_data_smithi179_148205_object27136:head# 2024-02-09T17:57:09.288 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:111f47e2:::benchmark_data_smithi179_148205_object75531:head# 2024-02-09T17:57:09.293 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:111f49ab:::benchmark_data_smithi179_148205_object97914:head# 2024-02-09T17:57:09.293 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:111f6546:::benchmark_data_smithi179_148205_object27237:head# 2024-02-09T17:57:09.293 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11209ab0:::benchmark_data_smithi179_148205_object80054:head# 2024-02-09T17:57:09.293 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1122de6a:::benchmark_data_smithi179_148205_object95424:head# 2024-02-09T17:57:09.293 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1123c76b:::benchmark_data_smithi179_148205_object10802:head# 2024-02-09T17:57:09.293 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11245da5:::benchmark_data_smithi179_148205_object7862:head# 2024-02-09T17:57:09.293 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:112531f2:::benchmark_data_smithi179_148205_object44909:head# 2024-02-09T17:57:09.294 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11256f38:::benchmark_data_smithi179_148205_object55429:head# 2024-02-09T17:57:09.294 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1125dfda:::benchmark_data_smithi179_148205_object92669:head# 2024-02-09T17:57:09.294 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1126bad6:::benchmark_data_smithi179_148205_object77060:head# 2024-02-09T17:57:09.294 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1127971b:::benchmark_data_smithi179_148205_object29405:head# 2024-02-09T17:57:09.294 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1127f806:::benchmark_data_smithi179_148205_object21388:head# 2024-02-09T17:57:09.294 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11280cf0:::benchmark_data_smithi179_148205_object58871:head# 2024-02-09T17:57:09.294 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:112857f9:::benchmark_data_smithi179_148205_object4430:head# 2024-02-09T17:57:09.299 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1129853b:::benchmark_data_smithi179_148205_object47164:head# 2024-02-09T17:57:09.300 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:112ac687:::benchmark_data_smithi179_148205_object43825:head# 2024-02-09T17:57:09.300 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:112b87a9:::benchmark_data_smithi179_148205_object105134:head# 2024-02-09T17:57:09.301 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:112b995e:::benchmark_data_smithi179_148205_object9841:head# 2024-02-09T17:57:09.301 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:112bb98f:::benchmark_data_smithi179_148205_object57829:head# 2024-02-09T17:57:09.302 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:112bd5ac:::benchmark_data_smithi179_148205_object24139:head# 2024-02-09T17:57:09.302 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:112e8e05:::benchmark_data_smithi179_148205_object102326:head# 2024-02-09T17:57:09.303 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:112ef252:::benchmark_data_smithi179_148205_object83024:head# 2024-02-09T17:57:09.303 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11302398:::benchmark_data_smithi179_148205_object27119:head# 2024-02-09T17:57:09.303 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11303576:::benchmark_data_smithi179_148205_object82284:head# 2024-02-09T17:57:09.304 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1130be34:::benchmark_data_smithi179_148205_object69610:head# 2024-02-09T17:57:09.304 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:113150b2:::benchmark_data_smithi179_148205_object21214:head# 2024-02-09T17:57:09.305 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11316a22:::benchmark_data_smithi179_148205_object28486:head# 2024-02-09T17:57:09.305 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1132429a:::benchmark_data_smithi179_148205_object335:head# 2024-02-09T17:57:09.305 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:113260e3:::benchmark_data_smithi179_148205_object41549:head# 2024-02-09T17:57:09.306 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11336a25:::benchmark_data_smithi179_148205_object102078:head# 2024-02-09T17:57:09.306 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:113424a9:::benchmark_data_smithi179_148205_object74710:head# 2024-02-09T17:57:09.306 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1136332e:::benchmark_data_smithi179_148205_object109392:head# 2024-02-09T17:57:09.307 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:113640b3:::benchmark_data_smithi179_148205_object78581:head# 2024-02-09T17:57:09.307 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1136f0a3:::benchmark_data_smithi179_148205_object10874:head# 2024-02-09T17:57:09.307 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:113788a9:::benchmark_data_smithi179_148205_object48944:head# 2024-02-09T17:57:09.308 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1138ac80:::benchmark_data_smithi179_148205_object78650:head# 2024-02-09T17:57:09.308 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11394547:::benchmark_data_smithi179_148205_object73370:head# 2024-02-09T17:57:09.309 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1139cd37:::benchmark_data_smithi179_148205_object84694:head# 2024-02-09T17:57:09.309 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:113b498c:::benchmark_data_smithi179_148205_object107036:head# 2024-02-09T17:57:09.310 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:113bfa6e:::benchmark_data_smithi179_148205_object37287:head# 2024-02-09T17:57:09.310 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:113c089b:::benchmark_data_smithi179_148205_object38384:head# 2024-02-09T17:57:09.311 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:113c0c37:::benchmark_data_smithi179_148205_object16495:head# 2024-02-09T17:57:09.311 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:113c5002:::benchmark_data_smithi179_148205_object19624:head# 2024-02-09T17:57:09.312 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:113db01c:::benchmark_data_smithi179_148205_object29022:head# 2024-02-09T17:57:09.312 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:113dcc97:::benchmark_data_smithi179_148205_object74672:head# 2024-02-09T17:57:09.312 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:113e4f44:::benchmark_data_smithi179_148205_object93761:head# 2024-02-09T17:57:09.313 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:113edea4:::benchmark_data_smithi179_148205_object40147:head# 2024-02-09T17:57:09.313 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:113edff0:::benchmark_data_smithi179_148205_object15963:head# 2024-02-09T17:57:09.313 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:113fbd27:::benchmark_data_smithi179_148205_object2863:head# 2024-02-09T17:57:09.314 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11402bd9:::benchmark_data_smithi179_148205_object93264:head# 2024-02-09T17:57:09.314 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11405dc6:::benchmark_data_smithi179_148205_object71406:head# 2024-02-09T17:57:09.314 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11406c00:::benchmark_data_smithi179_148205_object962:head# 2024-02-09T17:57:09.315 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1140b48b:::benchmark_data_smithi179_148205_object108584:head# 2024-02-09T17:57:09.316 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1140bcf9:::benchmark_data_smithi179_148205_object74328:head# 2024-02-09T17:57:09.316 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1140e741:::benchmark_data_smithi179_148205_object15236:head# 2024-02-09T17:57:09.316 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11416b11:::benchmark_data_smithi179_148205_object2811:head# 2024-02-09T17:57:09.317 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1141aab7:::benchmark_data_smithi179_148205_object70728:head# 2024-02-09T17:57:09.317 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1141ce9f:::benchmark_data_smithi179_148205_object16528:head# 2024-02-09T17:57:09.318 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1142026b:::benchmark_data_smithi179_148205_object47848:head# 2024-02-09T17:57:09.322 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11424aaf:::benchmark_data_smithi179_148205_object54267:head# 2024-02-09T17:57:09.322 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:114299d9:::benchmark_data_smithi179_148205_object84632:head# 2024-02-09T17:57:09.322 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11437ec0:::benchmark_data_smithi179_148205_object50563:head# 2024-02-09T17:57:09.322 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1145af0c:::benchmark_data_smithi179_148205_object103030:head# 2024-02-09T17:57:09.322 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1146301f:::benchmark_data_smithi179_148205_object25322:head# 2024-02-09T17:57:09.322 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11464ae1:::benchmark_data_smithi179_148205_object59460:head# 2024-02-09T17:57:09.322 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1147e546:::benchmark_data_smithi179_148205_object80954:head# 2024-02-09T17:57:09.323 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1147fa1f:::benchmark_data_smithi179_148205_object32139:head# 2024-02-09T17:57:09.326 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1148ec57:::benchmark_data_smithi179_148205_object87304:head# 2024-02-09T17:57:09.326 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:114975df:::benchmark_data_smithi179_148205_object29763:head# 2024-02-09T17:57:09.326 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:114acfc1:::benchmark_data_smithi179_148205_object56005:head# 2024-02-09T17:57:09.326 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:114b5a66:::benchmark_data_smithi179_148205_object2789:head# 2024-02-09T17:57:09.327 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:114ba30e:::benchmark_data_smithi179_148205_object40538:head# 2024-02-09T17:57:09.327 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:114bccc3:::benchmark_data_smithi179_148205_object1447:head# 2024-02-09T17:57:09.327 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:114db412:::benchmark_data_smithi179_148205_object91059:head# 2024-02-09T17:57:09.329 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:115015d8:::benchmark_data_smithi179_148205_object37000:head# 2024-02-09T17:57:09.330 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11511e10:::benchmark_data_smithi179_148205_object19405:head# 2024-02-09T17:57:09.332 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1153430e:::benchmark_data_smithi179_148205_object98031:head# 2024-02-09T17:57:09.332 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11555a9e:::benchmark_data_smithi179_148205_object79519:head# 2024-02-09T17:57:09.332 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:115591bc:::benchmark_data_smithi179_148205_object109121:head# 2024-02-09T17:57:09.332 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1155c14c:::benchmark_data_smithi179_148205_object82909:head# 2024-02-09T17:57:09.333 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1155c946:::benchmark_data_smithi179_148205_object13815:head# 2024-02-09T17:57:09.333 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1155e19f:::benchmark_data_smithi179_148205_object14391:head# 2024-02-09T17:57:09.333 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1156d307:::benchmark_data_smithi179_148205_object58686:head# 2024-02-09T17:57:09.333 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1156f2c8:::benchmark_data_smithi179_148205_object84625:head# 2024-02-09T17:57:09.333 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11571bc8:::benchmark_data_smithi179_148205_object55435:head# 2024-02-09T17:57:09.335 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:115765c2:::benchmark_data_smithi179_148205_object69588:head# 2024-02-09T17:57:09.335 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:115773ca:::benchmark_data_smithi179_148205_object31411:head# 2024-02-09T17:57:09.335 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:115903be:::benchmark_data_smithi179_148205_object1838:head# 2024-02-09T17:57:09.336 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:115958ca:::benchmark_data_smithi179_148205_object43601:head# 2024-02-09T17:57:09.336 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:115ad276:::benchmark_data_smithi179_148205_object11482:head# 2024-02-09T17:57:09.337 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:115b7b2f:::benchmark_data_smithi179_148205_object37423:head# 2024-02-09T17:57:09.337 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:115b9418:::benchmark_data_smithi179_148205_object108055:head# 2024-02-09T17:57:09.338 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:115c6171:::benchmark_data_smithi179_148205_object49897:head# 2024-02-09T17:57:09.338 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:115cae0f:::benchmark_data_smithi179_148205_object8421:head# 2024-02-09T17:57:09.338 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:115d4fb2:::benchmark_data_smithi179_148205_object104015:head# 2024-02-09T17:57:09.339 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:115dd861:::benchmark_data_smithi179_148205_object98008:head# 2024-02-09T17:57:09.339 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:115de1ab:::benchmark_data_smithi179_148205_object83961:head# 2024-02-09T17:57:09.340 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:115f0929:::benchmark_data_smithi179_148205_object95694:head# 2024-02-09T17:57:09.342 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:115f2b5b:::benchmark_data_smithi179_148205_object108482:head# 2024-02-09T17:57:09.342 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:115f2b8c:::benchmark_data_smithi179_148205_object38903:head# 2024-02-09T17:57:09.345 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:115f5781:::benchmark_data_smithi179_148205_object64635:head# 2024-02-09T17:57:09.345 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:115fdaba:::benchmark_data_smithi179_148205_object104813:head# 2024-02-09T17:57:09.346 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:115fe738:::benchmark_data_smithi179_148205_object5886:head# 2024-02-09T17:57:09.346 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1162597a:::benchmark_data_smithi179_148205_object104333:head# 2024-02-09T17:57:09.347 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1162c901:::benchmark_data_smithi179_148205_object36554:head# 2024-02-09T17:57:09.347 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11633842:::benchmark_data_smithi179_148205_object89860:head# 2024-02-09T17:57:09.348 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:116367c4:::benchmark_data_smithi179_148205_object11628:head# 2024-02-09T17:57:09.348 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1163d3dd:::benchmark_data_smithi179_148205_object48768:head# 2024-02-09T17:57:09.348 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11640dbe:::benchmark_data_smithi179_148205_object20900:head# 2024-02-09T17:57:09.349 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1164454a:::benchmark_data_smithi179_148205_object91044:head# 2024-02-09T17:57:09.349 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11649350:::benchmark_data_smithi179_148205_object40258:head# 2024-02-09T17:57:09.349 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1164ec82:::benchmark_data_smithi179_148205_object67298:head# 2024-02-09T17:57:09.351 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1165353c:::benchmark_data_smithi179_148205_object26152:head# 2024-02-09T17:57:09.352 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11654c14:::benchmark_data_smithi179_148205_object88494:head# 2024-02-09T17:57:09.352 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1165747c:::benchmark_data_smithi179_148205_object15701:head# 2024-02-09T17:57:09.352 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11665e69:::benchmark_data_smithi179_148205_object38864:head# 2024-02-09T17:57:09.353 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1166db00:::benchmark_data_smithi179_148205_object50946:head# 2024-02-09T17:57:09.353 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1167db3a:::benchmark_data_smithi179_148205_object91744:head# 2024-02-09T17:57:09.354 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1168fc3b:::benchmark_data_smithi179_148205_object6859:head# 2024-02-09T17:57:09.354 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1169f78f:::benchmark_data_smithi179_148205_object22922:head# 2024-02-09T17:57:09.354 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:116a1897:::benchmark_data_smithi179_148205_object28850:head# 2024-02-09T17:57:09.356 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:116a915c:::benchmark_data_smithi179_148205_object84760:head# 2024-02-09T17:57:09.357 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:116acc7e:::benchmark_data_smithi179_148205_object36169:head# 2024-02-09T17:57:09.358 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:57:09.358 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:116b103c:::benchmark_data_smithi179_148205_object4757:head# 2024-02-09T17:57:09.358 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:116c3ddf:::benchmark_data_smithi179_148205_object75234:head# 2024-02-09T17:57:09.359 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:116db63a:::benchmark_data_smithi179_148205_object100559:head# 2024-02-09T17:57:09.359 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:116ea077:::benchmark_data_smithi179_148205_object43374:head# 2024-02-09T17:57:09.360 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:116eb5da:::benchmark_data_smithi179_148205_object71844:head# 2024-02-09T17:57:09.360 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:116fce7e:::benchmark_data_smithi179_148205_object103875:head# 2024-02-09T17:57:09.361 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:116fdc1c:::benchmark_data_smithi179_148205_object2800:head# 2024-02-09T17:57:09.361 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:117200a9:::benchmark_data_smithi179_148205_object49581:head# 2024-02-09T17:57:09.362 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11735236:::benchmark_data_smithi179_148205_object17491:head# 2024-02-09T17:57:09.362 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1173f01d:::benchmark_data_smithi179_148205_object47715:head# 2024-02-09T17:57:09.362 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11749f39:::benchmark_data_smithi179_148205_object54279:head# 2024-02-09T17:57:09.363 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1174c540:::benchmark_data_smithi179_148205_object18706:head# 2024-02-09T17:57:09.363 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1174ec4e:::benchmark_data_smithi179_148205_object46445:head# 2024-02-09T17:57:09.363 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:117648b6:::benchmark_data_smithi179_148205_object54048:head# 2024-02-09T17:57:09.364 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1177892e:::benchmark_data_smithi179_148205_object7383:head# 2024-02-09T17:57:09.364 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:117846e8:::benchmark_data_smithi179_148205_object64562:head# 2024-02-09T17:57:09.365 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:117899a3:::benchmark_data_smithi179_148205_object105876:head# 2024-02-09T17:57:09.365 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1178ee5d:::benchmark_data_smithi179_148205_object40807:head# 2024-02-09T17:57:09.366 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:117b7924:::benchmark_data_smithi179_148205_object105694:head# 2024-02-09T17:57:09.366 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:117c229e:::benchmark_data_smithi179_148205_object56275:head# 2024-02-09T17:57:09.367 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:117c8a0c:::benchmark_data_smithi179_148205_object3405:head# 2024-02-09T17:57:09.367 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:117c9b74:::benchmark_data_smithi179_148205_object32472:head# 2024-02-09T17:57:09.368 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:117d2ac3:::benchmark_data_smithi179_148205_object42992:head# 2024-02-09T17:57:09.368 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:117dacee:::benchmark_data_smithi179_148205_object29862:head# 2024-02-09T17:57:09.368 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:117def43:::benchmark_data_smithi179_148205_object71103:head# 2024-02-09T17:57:09.369 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:117e3471:::benchmark_data_smithi179_148205_object108573:head# 2024-02-09T17:57:09.370 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:117e5c35:::benchmark_data_smithi179_148205_object64667:head# 2024-02-09T17:57:09.370 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:117ea23a:::benchmark_data_smithi179_148205_object11117:head# 2024-02-09T17:57:09.370 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:117ec4b1:::benchmark_data_smithi179_148205_object54018:head# 2024-02-09T17:57:09.371 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11803ee1:::benchmark_data_smithi179_148205_object36377:head# 2024-02-09T17:57:09.371 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11812ca8:::benchmark_data_smithi179_148205_object87045:head# 2024-02-09T17:57:09.372 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1181a993:::benchmark_data_smithi179_148205_object40953:head# 2024-02-09T17:57:09.372 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1182ff4a:::benchmark_data_smithi179_148205_object65968:head# 2024-02-09T17:57:09.372 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11834618:::benchmark_data_smithi179_148205_object42768:head# 2024-02-09T17:57:09.373 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11841ccd:::benchmark_data_smithi179_148205_object30981:head# 2024-02-09T17:57:09.373 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1185a37f:::benchmark_data_smithi179_148205_object89896:head# 2024-02-09T17:57:09.374 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11864ca1:::benchmark_data_smithi179_148205_object22230:head# 2024-02-09T17:57:09.374 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11868bf8:::benchmark_data_smithi179_148205_object7312:head# 2024-02-09T17:57:09.375 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1186a1a5:::benchmark_data_smithi179_148205_object34143:head# 2024-02-09T17:57:09.375 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1186a8be:::benchmark_data_smithi179_148205_object67066:head# 2024-02-09T17:57:09.376 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:118720d0:::benchmark_data_smithi179_148205_object39388:head# 2024-02-09T17:57:09.376 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11884039:::benchmark_data_smithi179_148205_object84248:head# 2024-02-09T17:57:09.376 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11886977:::benchmark_data_smithi179_148205_object42645:head# 2024-02-09T17:57:09.377 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1188bc3d:::benchmark_data_smithi179_148205_object31047:head# 2024-02-09T17:57:09.377 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:118a5584:::benchmark_data_smithi179_148205_object81977:head# 2024-02-09T17:57:09.378 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:118a850f:::benchmark_data_smithi179_148205_object106714:head# 2024-02-09T17:57:09.378 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:118b9fe1:::benchmark_data_smithi179_148205_object8289:head# 2024-02-09T17:57:09.378 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:118beedb:::benchmark_data_smithi179_148205_object95782:head# 2024-02-09T17:57:09.379 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:118bf7ae:::benchmark_data_smithi179_148205_object80677:head# 2024-02-09T17:57:09.379 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:118c262c:::benchmark_data_smithi179_148205_object25794:head# 2024-02-09T17:57:09.380 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:118cded9:::benchmark_data_smithi179_148205_object6324:head# 2024-02-09T17:57:09.380 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:118d6a42:::benchmark_data_smithi179_148205_object84849:head# 2024-02-09T17:57:09.381 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:118d8c2f:::benchmark_data_smithi179_148205_object10046:head# 2024-02-09T17:57:09.381 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:118dc565:::benchmark_data_smithi179_148205_object51884:head# 2024-02-09T17:57:09.382 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:118e2e84:::benchmark_data_smithi179_148205_object3802:head# 2024-02-09T17:57:09.382 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:118e6bea:::benchmark_data_smithi179_148205_object16564:head# 2024-02-09T17:57:09.382 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1190f6a2:::benchmark_data_smithi179_148205_object12000:head# 2024-02-09T17:57:09.383 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11910b86:::benchmark_data_smithi179_148205_object56790:head# 2024-02-09T17:57:09.383 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1191155b:::benchmark_data_smithi179_148205_object97602:head# 2024-02-09T17:57:09.384 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11925534:::benchmark_data_smithi179_148205_object45013:head# 2024-02-09T17:57:09.384 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:119278ee:::benchmark_data_smithi179_148205_object80424:head# 2024-02-09T17:57:09.384 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1193d348:::benchmark_data_smithi179_148205_object1211:head# 2024-02-09T17:57:09.385 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1194bb2d:::benchmark_data_smithi179_148205_object3080:head# 2024-02-09T17:57:09.385 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11953463:::benchmark_data_smithi179_148205_object52380:head# 2024-02-09T17:57:09.386 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11955345:::benchmark_data_smithi179_148205_object99393:head# 2024-02-09T17:57:09.386 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11972d76:::benchmark_data_smithi179_148205_object74781:head# 2024-02-09T17:57:09.386 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11973135:::benchmark_data_smithi179_148205_object54861:head# 2024-02-09T17:57:09.387 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1198217a:::benchmark_data_smithi179_148205_object6726:head# 2024-02-09T17:57:09.387 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11987cdb:::benchmark_data_smithi179_148205_object47039:head# 2024-02-09T17:57:09.388 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11988875:::benchmark_data_smithi179_148205_object8913:head# 2024-02-09T17:57:09.388 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1198d91d:::benchmark_data_smithi179_148205_object51706:head# 2024-02-09T17:57:09.389 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1198d93e:::benchmark_data_smithi179_148205_object82883:head# 2024-02-09T17:57:09.389 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1198ee71:::benchmark_data_smithi179_148205_object27495:head# 2024-02-09T17:57:09.389 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:119b4ae1:::benchmark_data_smithi179_148205_object57237:head# 2024-02-09T17:57:09.390 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:119bd590:::benchmark_data_smithi179_148205_object99422:head# 2024-02-09T17:57:09.390 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:119bdad5:::benchmark_data_smithi179_148205_object98176:head# 2024-02-09T17:57:09.391 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:119be395:::benchmark_data_smithi179_148205_object66287:head# 2024-02-09T17:57:09.391 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:119c7ccf:::benchmark_data_smithi179_148205_object21420:head# 2024-02-09T17:57:09.391 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:119ca9a8:::benchmark_data_smithi179_148205_object337:head# 2024-02-09T17:57:09.392 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:119d15b6:::benchmark_data_smithi179_148205_object34805:head# 2024-02-09T17:57:09.392 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:119d2cfa:::benchmark_data_smithi179_148205_object18120:head# 2024-02-09T17:57:09.393 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:119d3696:::benchmark_data_smithi179_148205_object8733:head# 2024-02-09T17:57:09.393 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:119d42b5:::benchmark_data_smithi179_148205_object19456:head# 2024-02-09T17:57:09.394 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:119e01b1:::benchmark_data_smithi179_148205_object86737:head# 2024-02-09T17:57:09.394 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:119eeb65:::benchmark_data_smithi179_148205_object54886:head# 2024-02-09T17:57:09.395 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:119f880d:::benchmark_data_smithi179_148205_object93329:head# 2024-02-09T17:57:09.395 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11a0100d:::benchmark_data_smithi179_148205_object98026:head# 2024-02-09T17:57:09.399 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11a051f8:::benchmark_data_smithi179_148205_object23414:head# 2024-02-09T17:57:09.399 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11a05dfe:::benchmark_data_smithi179_148205_object59013:head# 2024-02-09T17:57:09.399 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11a15b92:::benchmark_data_smithi179_148205_object33440:head# 2024-02-09T17:57:09.399 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11a1baa6:::benchmark_data_smithi179_148205_object28721:head# 2024-02-09T17:57:09.399 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11a1d88c:::benchmark_data_smithi179_148205_object74197:head# 2024-02-09T17:57:09.400 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11a1fa9f:::benchmark_data_smithi179_148205_object38031:head# 2024-02-09T17:57:09.400 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11a26c68:::benchmark_data_smithi179_148205_object75799:head# 2024-02-09T17:57:09.400 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11a26c9f:::benchmark_data_smithi179_148205_object31636:head# 2024-02-09T17:57:09.400 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11a279ff:::benchmark_data_smithi179_148205_object79756:head# 2024-02-09T17:57:09.401 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11a397e6:::benchmark_data_smithi179_148205_object91746:head# 2024-02-09T17:57:09.401 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11a3ff3e:::benchmark_data_smithi179_148205_object21350:head# 2024-02-09T17:57:09.401 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11a4fd88:::benchmark_data_smithi179_148205_object91213:head# 2024-02-09T17:57:09.401 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11a56827:::benchmark_data_smithi179_148205_object76991:head# 2024-02-09T17:57:09.402 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11a5e934:::benchmark_data_smithi179_148205_object93637:head# 2024-02-09T17:57:09.402 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11a6fdb3:::benchmark_data_smithi179_148205_object38279:head# 2024-02-09T17:57:09.403 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11abdfe9:::benchmark_data_smithi179_148205_object87127:head# 2024-02-09T17:57:09.403 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11ac4d07:::benchmark_data_smithi179_148205_object6552:head# 2024-02-09T17:57:09.403 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11ad553d:::benchmark_data_smithi179_148205_object19365:head# 2024-02-09T17:57:09.404 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11adf77a:::benchmark_data_smithi179_148205_object1662:head# 2024-02-09T17:57:09.405 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11ae413d:::benchmark_data_smithi179_148205_object101887:head# 2024-02-09T17:57:09.405 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11ae8478:::benchmark_data_smithi179_148205_object98810:head# 2024-02-09T17:57:09.406 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11ae8ec0:::benchmark_data_smithi179_148205_object75303:head# 2024-02-09T17:57:09.406 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11af4014:::benchmark_data_smithi179_148205_object102408:head# 2024-02-09T17:57:09.407 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11af7124:::benchmark_data_smithi179_148205_object66227:head# 2024-02-09T17:57:09.407 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11afba3d:::benchmark_data_smithi179_148205_object27375:head# 2024-02-09T17:57:09.408 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11afcd3a:::benchmark_data_smithi179_148205_object64713:head# 2024-02-09T17:57:09.412 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11b01c69:::benchmark_data_smithi179_148205_object33723:head# 2024-02-09T17:57:09.413 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11b0731d:::benchmark_data_smithi179_148205_object81522:head# 2024-02-09T17:57:09.413 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11b2bd84:::benchmark_data_smithi179_148205_object105310:head# 2024-02-09T17:57:09.413 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11b55f91:::benchmark_data_smithi179_148205_object63158:head# 2024-02-09T17:57:09.413 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11b5a1fa:::benchmark_data_smithi179_148205_object11332:head# 2024-02-09T17:57:09.413 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11b64e2d:::benchmark_data_smithi179_148205_object89196:head# 2024-02-09T17:57:09.413 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11b6949f:::benchmark_data_smithi179_148205_object72597:head# 2024-02-09T17:57:09.413 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11b740c2:::benchmark_data_smithi179_148205_object467:head# 2024-02-09T17:57:09.413 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11b741be:::benchmark_data_smithi179_148205_object36501:head# 2024-02-09T17:57:09.413 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11b74773:::benchmark_data_smithi179_148205_object105127:head# 2024-02-09T17:57:09.414 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:09.414 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11b83b23:::benchmark_data_smithi179_148205_object96530:head# 2024-02-09T17:57:09.414 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11b8db68:::benchmark_data_smithi179_148205_object48777:head# 2024-02-09T17:57:09.418 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11b8f9e4:::benchmark_data_smithi179_148205_object15607:head# 2024-02-09T17:57:09.418 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11b9c75e:::benchmark_data_smithi179_148205_object25554:head# 2024-02-09T17:57:09.418 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11bb5561:::benchmark_data_smithi179_148205_object49569:head# 2024-02-09T17:57:09.418 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11bb6a7d:::benchmark_data_smithi179_148205_object76805:head# 2024-02-09T17:57:09.418 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11bbfc4b:::benchmark_data_smithi179_148205_object23931:head# 2024-02-09T17:57:09.418 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11be33d3:::benchmark_data_smithi179_148205_object17726:head# 2024-02-09T17:57:09.418 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11be4213:::benchmark_data_smithi179_148205_object97958:head# 2024-02-09T17:57:09.418 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11be5a51:::benchmark_data_smithi179_148205_object65033:head# 2024-02-09T17:57:09.418 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11be72c1:::benchmark_data_smithi179_148205_object103993:head# 2024-02-09T17:57:09.419 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11c01424:::benchmark_data_smithi179_148205_object104820:head# 2024-02-09T17:57:09.419 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11c03ba1:::benchmark_data_smithi179_148205_object86204:head# 2024-02-09T17:57:09.419 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11c0461b:::benchmark_data_smithi179_148205_object86316:head# 2024-02-09T17:57:09.419 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11c1027b:::benchmark_data_smithi179_148205_object81794:head# 2024-02-09T17:57:09.420 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11c1101e:::benchmark_data_smithi179_148205_object108882:head# 2024-02-09T17:57:09.420 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11c150f5:::benchmark_data_smithi179_148205_object62333:head# 2024-02-09T17:57:09.420 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11c1e2a9:::benchmark_data_smithi179_148205_object1203:head# 2024-02-09T17:57:09.420 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11c21b7b:::benchmark_data_smithi179_148205_object87945:head# 2024-02-09T17:57:09.421 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11c39c7d:::benchmark_data_smithi179_148205_object17333:head# 2024-02-09T17:57:09.421 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11c3ce29:::benchmark_data_smithi179_148205_object105121:head# 2024-02-09T17:57:09.421 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11c3e250:::benchmark_data_smithi179_148205_object61578:head# 2024-02-09T17:57:09.421 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11c52778:::benchmark_data_smithi179_148205_object101423:head# 2024-02-09T17:57:09.422 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11c5901f:::benchmark_data_smithi179_148205_object20819:head# 2024-02-09T17:57:09.422 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11c64d27:::benchmark_data_smithi179_148205_object77027:head# 2024-02-09T17:57:09.422 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11c7354d:::benchmark_data_smithi179_148205_object4783:head# 2024-02-09T17:57:09.423 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11c81f80:::benchmark_data_smithi179_148205_object67366:head# 2024-02-09T17:57:09.423 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11c85357:::benchmark_data_smithi179_148205_object34957:head# 2024-02-09T17:57:09.424 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11c860de:::benchmark_data_smithi179_148205_object82712:head# 2024-02-09T17:57:09.424 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11c90b64:::benchmark_data_smithi179_148205_object25781:head# 2024-02-09T17:57:09.425 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11ca843a:::benchmark_data_smithi179_148205_object65593:head# 2024-02-09T17:57:09.425 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11cafc3d:::benchmark_data_smithi179_148205_object587:head# 2024-02-09T17:57:09.425 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11cba501:::benchmark_data_smithi179_148205_object58568:head# 2024-02-09T17:57:09.426 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11cbce00:::benchmark_data_smithi179_148205_object51513:head# 2024-02-09T17:57:09.426 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11cbd368:::benchmark_data_smithi179_148205_object12008:head# 2024-02-09T17:57:09.426 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11cc7011:::benchmark_data_smithi179_148205_object14560:head# 2024-02-09T17:57:09.427 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11cdb54c:::benchmark_data_smithi179_148205_object85389:head# 2024-02-09T17:57:09.428 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11cdc5a2:::benchmark_data_smithi179_148205_object79948:head# 2024-02-09T17:57:09.428 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11cddfd3:::benchmark_data_smithi179_148205_object88940:head# 2024-02-09T17:57:09.429 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11cdec59:::benchmark_data_smithi179_148205_object35126:head# 2024-02-09T17:57:09.429 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11ce08b5:::benchmark_data_smithi179_148205_object28240:head# 2024-02-09T17:57:09.429 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11ce8664:::benchmark_data_smithi179_148205_object14421:head# 2024-02-09T17:57:09.429 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11ceef18:::benchmark_data_smithi179_148205_object42916:head# 2024-02-09T17:57:09.430 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11cfeab4:::benchmark_data_smithi179_148205_object92618:head# 2024-02-09T17:57:09.430 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11d0f0be:::benchmark_data_smithi179_148205_object60604:head# 2024-02-09T17:57:09.431 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11d0f801:::benchmark_data_smithi179_148205_object74270:head# 2024-02-09T17:57:09.431 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11d231c5:::benchmark_data_smithi179_148205_object36635:head# 2024-02-09T17:57:09.431 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11d36596:::benchmark_data_smithi179_148205_object64858:head# 2024-02-09T17:57:09.432 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11d38819:::benchmark_data_smithi179_148205_object60359:head# 2024-02-09T17:57:09.432 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11d592b5:::benchmark_data_smithi179_148205_object76056:head# 2024-02-09T17:57:09.432 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11d59a4b:::benchmark_data_smithi179_148205_object46731:head# 2024-02-09T17:57:09.433 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11d5ba1d:::benchmark_data_smithi179_148205_object35232:head# 2024-02-09T17:57:09.433 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11d5d8c7:::benchmark_data_smithi179_148205_object72026:head# 2024-02-09T17:57:09.433 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11d600aa:::benchmark_data_smithi179_148205_object87574:head# 2024-02-09T17:57:09.434 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11d615e9:::benchmark_data_smithi179_148205_object17260:head# 2024-02-09T17:57:09.434 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:57:09.435 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11d635d8:::benchmark_data_smithi179_148205_object80551:head# 2024-02-09T17:57:09.435 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11d814cb:::benchmark_data_smithi179_148205_object57800:head# 2024-02-09T17:57:09.435 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11d85345:::benchmark_data_smithi179_148205_object54425:head# 2024-02-09T17:57:09.436 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11d868e2:::benchmark_data_smithi179_148205_object15621:head# 2024-02-09T17:57:09.436 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11d87b3f:::benchmark_data_smithi179_148205_object7148:head# 2024-02-09T17:57:09.437 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11d946ae:::benchmark_data_smithi179_148205_object58948:head# 2024-02-09T17:57:09.437 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11da8626:::benchmark_data_smithi179_148205_object36916:head# 2024-02-09T17:57:09.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:09.438 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:57:09.438 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11daa3ab:::benchmark_data_smithi179_148205_object109447:head# 2024-02-09T17:57:09.439 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11dad86b:::benchmark_data_smithi179_148205_object9221:head# 2024-02-09T17:57:09.439 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11db3b8a:::benchmark_data_smithi179_148205_object38375:head# 2024-02-09T17:57:09.439 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11dbf52a:::benchmark_data_smithi179_148205_object24647:head# 2024-02-09T17:57:09.439 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11ddf785:::benchmark_data_smithi179_148205_object44779:head# 2024-02-09T17:57:09.440 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11de0bcd:::benchmark_data_smithi179_148205_object64261:head# 2024-02-09T17:57:09.440 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11de43e6:::benchmark_data_smithi179_148205_object75663:head# 2024-02-09T17:57:09.440 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11de9186:::benchmark_data_smithi179_148205_object104009:head# 2024-02-09T17:57:09.441 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11dfacbe:::benchmark_data_smithi179_148205_object60831:head# 2024-02-09T17:57:09.446 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11e09377:::benchmark_data_smithi179_148205_object42247:head# 2024-02-09T17:57:09.446 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11e0e65b:::benchmark_data_smithi179_148205_object89400:head# 2024-02-09T17:57:09.446 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11e12cdd:::benchmark_data_smithi179_148205_object109608:head# 2024-02-09T17:57:09.446 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11e130ed:::benchmark_data_smithi179_148205_object26998:head# 2024-02-09T17:57:09.446 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11e14acd:::benchmark_data_smithi179_148205_object1306:head# 2024-02-09T17:57:09.446 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11e1821c:::benchmark_data_smithi179_148205_object8010:head# 2024-02-09T17:57:09.446 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11e18a36:::benchmark_data_smithi179_148205_object77122:head# 2024-02-09T17:57:09.446 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11e19b29:::benchmark_data_smithi179_148205_object63101:head# 2024-02-09T17:57:09.447 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11e1bf75:::benchmark_data_smithi179_148205_object72265:head# 2024-02-09T17:57:09.450 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11e1e756:::benchmark_data_smithi179_148205_object1620:head# 2024-02-09T17:57:09.450 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11e20aff:::benchmark_data_smithi179_148205_object104854:head# 2024-02-09T17:57:09.450 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11e23cb7:::benchmark_data_smithi179_148205_object4807:head# 2024-02-09T17:57:09.450 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11e25e6a:::benchmark_data_smithi179_148205_object14409:head# 2024-02-09T17:57:09.450 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11e291d7:::benchmark_data_smithi179_148205_object13453:head# 2024-02-09T17:57:09.450 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11e50f32:::benchmark_data_smithi179_148205_object25254:head# 2024-02-09T17:57:09.450 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11e6635f:::benchmark_data_smithi179_148205_object50198:head# 2024-02-09T17:57:09.450 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11e67f54:::benchmark_data_smithi179_148205_object17375:head# 2024-02-09T17:57:09.451 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11e6b847:::benchmark_data_smithi179_148205_object38263:head# 2024-02-09T17:57:09.451 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11e7128d:::benchmark_data_smithi179_148205_object108579:head# 2024-02-09T17:57:09.451 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11e86c10:::benchmark_data_smithi179_148205_object85058:head# 2024-02-09T17:57:09.451 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11e87e6b:::benchmark_data_smithi179_148205_object24444:head# 2024-02-09T17:57:09.451 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11e89dc8:::benchmark_data_smithi179_148205_object4101:head# 2024-02-09T17:57:09.452 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11e94237:::benchmark_data_smithi179_148205_object1449:head# 2024-02-09T17:57:09.452 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11e9532c:::benchmark_data_smithi179_148205_object79037:head# 2024-02-09T17:57:09.452 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11e9a9db:::benchmark_data_smithi179_148205_object37518:head# 2024-02-09T17:57:09.453 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11e9acb5:::benchmark_data_smithi179_148205_object43810:head# 2024-02-09T17:57:09.453 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11ea4e65:::benchmark_data_smithi179_148205_object22010:head# 2024-02-09T17:57:09.454 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11eba9cf:::benchmark_data_smithi179_148205_object66130:head# 2024-02-09T17:57:09.454 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11ebab85:::benchmark_data_smithi179_148205_object96385:head# 2024-02-09T17:57:09.454 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11ec4319:::benchmark_data_smithi179_148205_object68182:head# 2024-02-09T17:57:09.455 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11ecec6c:::benchmark_data_smithi179_148205_object63725:head# 2024-02-09T17:57:09.455 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11ed0196:::benchmark_data_smithi179_148205_object17399:head# 2024-02-09T17:57:09.456 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11ed7dd5:::benchmark_data_smithi179_148205_object102911:head# 2024-02-09T17:57:09.456 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11edef3f:::benchmark_data_smithi179_148205_object21629:head# 2024-02-09T17:57:09.460 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11ee7027:::benchmark_data_smithi179_148205_object21791:head# 2024-02-09T17:57:09.460 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11eee562:::benchmark_data_smithi179_148205_object87363:head# 2024-02-09T17:57:09.460 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11f284a8:::benchmark_data_smithi179_148205_object10734:head# 2024-02-09T17:57:09.460 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11f2b28b:::benchmark_data_smithi179_148205_object91335:head# 2024-02-09T17:57:09.461 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11f4111f:::benchmark_data_smithi179_148205_object68432:head# 2024-02-09T17:57:09.461 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11f42fb9:::benchmark_data_smithi179_148205_object34193:head# 2024-02-09T17:57:09.461 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11f46b65:::benchmark_data_smithi179_148205_object78337:head# 2024-02-09T17:57:09.461 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11f5452f:::benchmark_data_smithi179_148205_object46022:head# 2024-02-09T17:57:09.461 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11f55ec3:::benchmark_data_smithi179_148205_object107194:head# 2024-02-09T17:57:09.461 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11f61119:::benchmark_data_smithi179_148205_object48660:head# 2024-02-09T17:57:09.461 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11f62677:::benchmark_data_smithi179_148205_object74032:head# 2024-02-09T17:57:09.461 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11f6fc40:::benchmark_data_smithi179_148205_object108519:head# 2024-02-09T17:57:09.462 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11f703c9:::benchmark_data_smithi179_148205_object97298:head# 2024-02-09T17:57:09.462 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11f729ef:::benchmark_data_smithi179_148205_object106899:head# 2024-02-09T17:57:09.463 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11f87c26:::benchmark_data_smithi179_148205_object92721:head# 2024-02-09T17:57:09.467 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11f9033f:::benchmark_data_smithi179_148205_object104430:head# 2024-02-09T17:57:09.467 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11f9be39:::benchmark_data_smithi179_148205_object23876:head# 2024-02-09T17:57:09.467 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11fd573b:::benchmark_data_smithi179_148205_object55328:head# 2024-02-09T17:57:09.468 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11fddc54:::benchmark_data_smithi179_148205_object71902:head# 2024-02-09T17:57:09.468 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11fe3338:::benchmark_data_smithi179_148205_object69536:head# 2024-02-09T17:57:09.469 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11fe71ad:::benchmark_data_smithi179_148205_object33744:head# 2024-02-09T17:57:09.469 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11feb186:::benchmark_data_smithi179_148205_object24312:head# 2024-02-09T17:57:09.469 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11feb4a8:::benchmark_data_smithi179_148205_object37229:head# 2024-02-09T17:57:09.470 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:11ffcef1:::benchmark_data_smithi179_148205_object47323:head# 2024-02-09T17:57:09.470 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12008ad6:::benchmark_data_smithi179_148205_object28515:head# 2024-02-09T17:57:09.471 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12009cec:::benchmark_data_smithi179_148205_object51955:head# 2024-02-09T17:57:09.471 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12013148:::benchmark_data_smithi179_148205_object29589:head# 2024-02-09T17:57:09.473 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:120160a8:::benchmark_data_smithi179_148205_object79317:head# 2024-02-09T17:57:09.474 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1201e963:::benchmark_data_smithi179_148205_object77263:head# 2024-02-09T17:57:09.474 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1203da0e:::benchmark_data_smithi179_148205_object14463:head# 2024-02-09T17:57:09.475 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1204a58a:::benchmark_data_smithi179_148205_object104768:head# 2024-02-09T17:57:09.475 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12055e4f:::benchmark_data_smithi179_148205_object31951:head# 2024-02-09T17:57:09.476 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:120611e9:::benchmark_data_smithi179_148205_object96353:head# 2024-02-09T17:57:09.476 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12063d8c:::benchmark_data_smithi179_148205_object100232:head# 2024-02-09T17:57:09.476 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1206a7e4:::benchmark_data_smithi179_148205_object11457:head# 2024-02-09T17:57:09.477 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1206eaf1:::benchmark_data_smithi179_148205_object45710:head# 2024-02-09T17:57:09.477 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12071325:::benchmark_data_smithi179_148205_object77400:head# 2024-02-09T17:57:09.478 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1207f0f2:::benchmark_data_smithi179_148205_object88273:head# 2024-02-09T17:57:09.478 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12080e5a:::benchmark_data_smithi179_148205_object60721:head# 2024-02-09T17:57:09.479 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1209510d:::benchmark_data_smithi179_148205_object16536:head# 2024-02-09T17:57:09.479 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:120a46fc:::benchmark_data_smithi179_148205_object101555:head# 2024-02-09T17:57:09.480 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:120a9f1e:::benchmark_data_smithi179_148205_object14085:head# 2024-02-09T17:57:09.480 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:120aada3:::benchmark_data_smithi179_148205_object28234:head# 2024-02-09T17:57:09.480 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:120afc91:::benchmark_data_smithi179_148205_object83753:head# 2024-02-09T17:57:09.480 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:120b4e1b:::benchmark_data_smithi179_148205_object80632:head# 2024-02-09T17:57:09.485 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:120b6495:::benchmark_data_smithi179_148205_object89875:head# 2024-02-09T17:57:09.485 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:120e62c0:::benchmark_data_smithi179_148205_object7391:head# 2024-02-09T17:57:09.485 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:120e73b8:::benchmark_data_smithi179_148205_object48414:head# 2024-02-09T17:57:09.485 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:120f0885:::benchmark_data_smithi179_148205_object83480:head# 2024-02-09T17:57:09.485 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12103b83:::benchmark_data_smithi179_148205_object103230:head# 2024-02-09T17:57:09.485 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12120bbc:::benchmark_data_smithi179_148205_object2896:head# 2024-02-09T17:57:09.485 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1212ea75:::benchmark_data_smithi179_148205_object68037:head# 2024-02-09T17:57:09.485 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1213a890:::benchmark_data_smithi179_148205_object88223:head# 2024-02-09T17:57:09.486 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1214381e:::benchmark_data_smithi179_148205_object24418:head# 2024-02-09T17:57:09.486 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1214e1cc:::benchmark_data_smithi179_148205_object104208:head# 2024-02-09T17:57:09.486 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1214f7d5:::benchmark_data_smithi179_148205_object40545:head# 2024-02-09T17:57:09.486 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12158c90:::benchmark_data_smithi179_148205_object11904:head# 2024-02-09T17:57:09.487 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:121591d8:::benchmark_data_smithi179_148205_object29889:head# 2024-02-09T17:57:09.487 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1215e7e1:::benchmark_data_smithi179_148205_object60542:head# 2024-02-09T17:57:09.487 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1216006e:::benchmark_data_smithi179_148205_object96203:head# 2024-02-09T17:57:09.487 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1219d0f6:::benchmark_data_smithi179_148205_object58638:head# 2024-02-09T17:57:09.492 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:121a8fb7:::benchmark_data_smithi179_148205_object30786:head# 2024-02-09T17:57:09.492 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:121ab3a4:::benchmark_data_smithi179_148205_object12980:head# 2024-02-09T17:57:09.492 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:121abd4b:::benchmark_data_smithi179_148205_object46444:head# 2024-02-09T17:57:09.492 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:121c3359:::benchmark_data_smithi179_148205_object20804:head# 2024-02-09T17:57:09.492 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:121d00a0:::benchmark_data_smithi179_148205_object41853:head# 2024-02-09T17:57:09.492 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:121d45ec:::benchmark_data_smithi179_148205_object4812:head# 2024-02-09T17:57:09.492 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:121d7394:::benchmark_data_smithi179_148205_object46697:head# 2024-02-09T17:57:09.493 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:121d75cc:::benchmark_data_smithi179_148205_object56539:head# 2024-02-09T17:57:09.493 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:121db17e:::benchmark_data_smithi179_148205_object93572:head# 2024-02-09T17:57:09.493 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:121dc1c4:::benchmark_data_smithi179_148205_object107929:head# 2024-02-09T17:57:09.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:09 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3363208437' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:57:09.494 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:121e139e:::benchmark_data_smithi179_148205_object29560:head# 2024-02-09T17:57:09.494 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:121e2874:::benchmark_data_smithi179_148205_object22928:head# 2024-02-09T17:57:09.495 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:121f568a:::benchmark_data_smithi179_148205_object37889:head# 2024-02-09T17:57:09.495 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:121fd13b:::benchmark_data_smithi179_148205_object103928:head# 2024-02-09T17:57:09.496 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:121fd9fb:::benchmark_data_smithi179_148205_object99005:head# 2024-02-09T17:57:09.496 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12208c71:::benchmark_data_smithi179_148205_object32297:head# 2024-02-09T17:57:09.497 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1220bb1c:::benchmark_data_smithi179_148205_object101857:head# 2024-02-09T17:57:09.497 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1220daeb:::benchmark_data_smithi179_148205_object100170:head# 2024-02-09T17:57:09.502 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:122108ed:::benchmark_data_smithi179_148205_object94673:head# 2024-02-09T17:57:09.502 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1223ee03:::benchmark_data_smithi179_148205_object9679:head# 2024-02-09T17:57:09.503 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1223faa4:::benchmark_data_smithi179_148205_object15310:head# 2024-02-09T17:57:09.503 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12258d72:::benchmark_data_smithi179_148205_object52657:head# 2024-02-09T17:57:09.504 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12267825:::benchmark_data_smithi179_148205_object251:head# 2024-02-09T17:57:09.504 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1226feb8:::benchmark_data_smithi179_148205_object52085:head# 2024-02-09T17:57:09.505 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1227bc2a:::benchmark_data_smithi179_148205_object53172:head# 2024-02-09T17:57:09.507 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1227f3fd:::benchmark_data_smithi179_148205_object79557:head# 2024-02-09T17:57:09.507 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12285f2e:::benchmark_data_smithi179_148205_object62783:head# 2024-02-09T17:57:09.507 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1228cad6:::benchmark_data_smithi179_148205_object30052:head# 2024-02-09T17:57:09.508 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:122c4782:::benchmark_data_smithi179_148205_object79015:head# 2024-02-09T17:57:09.508 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:122c4f87:::benchmark_data_smithi179_148205_object96521:head# 2024-02-09T17:57:09.508 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:122d06b7:::benchmark_data_smithi179_148205_object8237:head# 2024-02-09T17:57:09.509 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:122db251:::benchmark_data_smithi179_148205_object63056:head# 2024-02-09T17:57:09.509 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:122e2704:::benchmark_data_smithi179_148205_object52962:head# 2024-02-09T17:57:09.510 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:122e79e5:::benchmark_data_smithi179_148205_object91683:head# 2024-02-09T17:57:09.510 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:122f61a8:::benchmark_data_smithi179_148205_object32483:head# 2024-02-09T17:57:09.510 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:122fdce6:::benchmark_data_smithi179_148205_object59100:head# 2024-02-09T17:57:09.511 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1232dffb:::benchmark_data_smithi179_148205_object66343:head# 2024-02-09T17:57:09.511 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:123344c1:::benchmark_data_smithi179_148205_object59587:head# 2024-02-09T17:57:09.512 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12335d15:::benchmark_data_smithi179_148205_object83735:head# 2024-02-09T17:57:09.512 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1233747a:::benchmark_data_smithi179_148205_object24029:head# 2024-02-09T17:57:09.513 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1234552f:::benchmark_data_smithi179_148205_object43348:head# 2024-02-09T17:57:09.513 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1234a3f9:::benchmark_data_smithi179_148205_object45644:head# 2024-02-09T17:57:09.514 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12350688:::benchmark_data_smithi179_148205_object1736:head# 2024-02-09T17:57:09.514 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12355983:::benchmark_data_smithi179_148205_object103232:head# 2024-02-09T17:57:09.514 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1236d29f:::benchmark_data_smithi179_148205_object19869:head# 2024-02-09T17:57:09.515 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1237018f:::benchmark_data_smithi179_148205_object97424:head# 2024-02-09T17:57:09.515 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1237bfb4:::benchmark_data_smithi179_148205_object35132:head# 2024-02-09T17:57:09.516 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1239600a:::benchmark_data_smithi179_148205_object106245:head# 2024-02-09T17:57:09.516 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1239dc76:::benchmark_data_smithi179_148205_object45899:head# 2024-02-09T17:57:09.517 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:123a4d1f:::benchmark_data_smithi179_148205_object54730:head# 2024-02-09T17:57:09.517 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:123b6092:::benchmark_data_smithi179_148205_object76077:head# 2024-02-09T17:57:09.517 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:123c6c29:::benchmark_data_smithi179_148205_object58603:head# 2024-02-09T17:57:09.518 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:123c86db:::benchmark_data_smithi179_148205_object31008:head# 2024-02-09T17:57:09.518 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:123d217f:::benchmark_data_smithi179_148205_object105109:head# 2024-02-09T17:57:09.519 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:123f6e78:::benchmark_data_smithi179_148205_object100988:head# 2024-02-09T17:57:09.519 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1240ffa5:::benchmark_data_smithi179_148205_object90660:head# 2024-02-09T17:57:09.520 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:124113b5:::benchmark_data_smithi179_148205_object54005:head# 2024-02-09T17:57:09.520 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1241555d:::benchmark_data_smithi179_148205_object97499:head# 2024-02-09T17:57:09.520 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1242166c:::benchmark_data_smithi179_148205_object67544:head# 2024-02-09T17:57:09.521 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1242d1a8:::benchmark_data_smithi179_148205_object59022:head# 2024-02-09T17:57:09.521 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12432d4f:::benchmark_data_smithi179_148205_object26074:head# 2024-02-09T17:57:09.522 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12447422:::benchmark_data_smithi179_148205_object79946:head# 2024-02-09T17:57:09.522 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1245f60b:::benchmark_data_smithi179_148205_object14757:head# 2024-02-09T17:57:09.522 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:124757cf:::benchmark_data_smithi179_148205_object97487:head# 2024-02-09T17:57:09.523 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12476cfb:::benchmark_data_smithi179_148205_object58443:head# 2024-02-09T17:57:09.523 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12488781:::benchmark_data_smithi179_148205_object6014:head# 2024-02-09T17:57:09.524 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1249643e:::benchmark_data_smithi179_148205_object38339:head# 2024-02-09T17:57:09.524 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1249b880:::benchmark_data_smithi179_148205_object10115:head# 2024-02-09T17:57:09.524 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1249bee6:::benchmark_data_smithi179_148205_object73093:head# 2024-02-09T17:57:09.525 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:124b5b38:::benchmark_data_smithi179_148205_object43814:head# 2024-02-09T17:57:09.525 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:124ba1dd:::benchmark_data_smithi179_148205_object75597:head# 2024-02-09T17:57:09.530 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:124c34e4:::benchmark_data_smithi179_148205_object74296:head# 2024-02-09T17:57:09.530 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:124c4a42:::benchmark_data_smithi179_148205_object88313:head# 2024-02-09T17:57:09.530 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:124c9d58:::benchmark_data_smithi179_148205_object25005:head# 2024-02-09T17:57:09.530 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:124ca57a:::benchmark_data_smithi179_148205_object96426:head# 2024-02-09T17:57:09.530 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:124d92a0:::benchmark_data_smithi179_148205_object101999:head# 2024-02-09T17:57:09.530 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:124fa142:::benchmark_data_smithi179_148205_object39195:head# 2024-02-09T17:57:09.530 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:124fb2fb:::benchmark_data_smithi179_148205_object1557:head# 2024-02-09T17:57:09.530 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:124fee28:::benchmark_data_smithi179_148205_object107908:head# 2024-02-09T17:57:09.533 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1250557b:::benchmark_data_smithi179_148205_object63204:head# 2024-02-09T17:57:09.533 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12518f61:::benchmark_data_smithi179_148205_object102209:head# 2024-02-09T17:57:09.536 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12523434:::benchmark_data_smithi179_148205_object34408:head# 2024-02-09T17:57:09.536 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:125248ce:::benchmark_data_smithi179_148205_object23121:head# 2024-02-09T17:57:09.537 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:125310b3:::benchmark_data_smithi179_148205_object70688:head# 2024-02-09T17:57:09.537 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12549554:::benchmark_data_smithi179_148205_object13570:head# 2024-02-09T17:57:09.537 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12575046:::benchmark_data_smithi179_148205_object101982:head# 2024-02-09T17:57:09.538 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1257673e:::benchmark_data_smithi179_148205_object90369:head# 2024-02-09T17:57:09.538 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12590999:::benchmark_data_smithi179_148205_object65581:head# 2024-02-09T17:57:09.539 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:57:09.539 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1259b382:::benchmark_data_smithi179_148205_object7392:head# 2024-02-09T17:57:09.539 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1259f051:::benchmark_data_smithi179_148205_object18453:head# 2024-02-09T17:57:09.540 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:125a82bc:::benchmark_data_smithi179_148205_object22108:head# 2024-02-09T17:57:09.540 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:125c3516:::benchmark_data_smithi179_148205_object106266:head# 2024-02-09T17:57:09.541 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:125d590d:::benchmark_data_smithi179_148205_object107985:head# 2024-02-09T17:57:09.541 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:125dca70:::benchmark_data_smithi179_148205_object58528:head# 2024-02-09T17:57:09.542 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:125e32e3:::benchmark_data_smithi179_148205_object87586:head# 2024-02-09T17:57:09.542 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:125ef662:::benchmark_data_smithi179_148205_object100647:head# 2024-02-09T17:57:09.543 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:125f38b0:::benchmark_data_smithi179_148205_object41764:head# 2024-02-09T17:57:09.543 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12606394:::benchmark_data_smithi179_148205_object12265:head# 2024-02-09T17:57:09.543 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1260e713:::benchmark_data_smithi179_148205_object10275:head# 2024-02-09T17:57:09.547 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12612593:::benchmark_data_smithi179_148205_object45265:head# 2024-02-09T17:57:09.547 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1261be7a:::benchmark_data_smithi179_148205_object26301:head# 2024-02-09T17:57:09.547 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12631380:::benchmark_data_smithi179_148205_object14755:head# 2024-02-09T17:57:09.547 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1264162f:::benchmark_data_smithi179_148205_object101405:head# 2024-02-09T17:57:09.547 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12643118:::benchmark_data_smithi179_148205_object104067:head# 2024-02-09T17:57:09.548 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12644c46:::benchmark_data_smithi179_148205_object42392:head# 2024-02-09T17:57:09.548 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12648356:::benchmark_data_smithi179_148205_object99573:head# 2024-02-09T17:57:09.548 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1265dfba:::benchmark_data_smithi179_148205_object100856:head# 2024-02-09T17:57:09.548 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:126654ea:::benchmark_data_smithi179_148205_object31606:head# 2024-02-09T17:57:09.548 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1266ab63:::benchmark_data_smithi179_148205_object106356:head# 2024-02-09T17:57:09.548 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:126741d0:::benchmark_data_smithi179_148205_object58741:head# 2024-02-09T17:57:09.548 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12683aa9:::benchmark_data_smithi179_148205_object78704:head# 2024-02-09T17:57:09.549 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12687130:::benchmark_data_smithi179_148205_object49425:head# 2024-02-09T17:57:09.549 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1268f603:::benchmark_data_smithi179_148205_object61682:head# 2024-02-09T17:57:09.549 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:126a3465:::benchmark_data_smithi179_148205_object104026:head# 2024-02-09T17:57:09.550 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:126a362c:::benchmark_data_smithi179_148205_object65609:head# 2024-02-09T17:57:09.550 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:126a4b65:::benchmark_data_smithi179_148205_object58504:head# 2024-02-09T17:57:09.551 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:126ac55c:::benchmark_data_smithi179_148205_object97070:head# 2024-02-09T17:57:09.551 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:126b0ed6:::benchmark_data_smithi179_148205_object17210:head# 2024-02-09T17:57:09.552 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:126b3e18:::benchmark_data_smithi179_148205_object68161:head# 2024-02-09T17:57:09.552 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:126b896b:::benchmark_data_smithi179_148205_object64244:head# 2024-02-09T17:57:09.553 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:126bce3e:::benchmark_data_smithi179_148205_object59730:head# 2024-02-09T17:57:09.553 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:126c06ca:::benchmark_data_smithi179_148205_object74340:head# 2024-02-09T17:57:09.554 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:126c1eb9:::benchmark_data_smithi179_148205_object44026:head# 2024-02-09T17:57:09.554 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:126cd756:::benchmark_data_smithi179_148205_object31088:head# 2024-02-09T17:57:09.554 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:126d0743:::benchmark_data_smithi179_148205_object94277:head# 2024-02-09T17:57:09.557 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:126d6935:::benchmark_data_smithi179_148205_object29424:head# 2024-02-09T17:57:09.557 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:126da9e0:::benchmark_data_smithi179_148205_object23314:head# 2024-02-09T17:57:09.557 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:126e43e6:::benchmark_data_smithi179_148205_object66198:head# 2024-02-09T17:57:09.557 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:126e8822:::benchmark_data_smithi179_148205_object50017:head# 2024-02-09T17:57:09.558 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:126eb615:::benchmark_data_smithi179_148205_object29347:head# 2024-02-09T17:57:09.558 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:126f12b9:::benchmark_data_smithi179_148205_object36563:head# 2024-02-09T17:57:09.558 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:126f493b:::benchmark_data_smithi179_148205_object49311:head# 2024-02-09T17:57:09.558 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:126fa632:::benchmark_data_smithi179_148205_object85518:head# 2024-02-09T17:57:09.559 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:127001d7:::benchmark_data_smithi179_148205_object57137:head# 2024-02-09T17:57:09.559 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12716676:::benchmark_data_smithi179_148205_object100454:head# 2024-02-09T17:57:09.559 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1271e76e:::benchmark_data_smithi179_148205_object97335:head# 2024-02-09T17:57:09.559 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12726dad:::benchmark_data_smithi179_148205_object67101:head# 2024-02-09T17:57:09.559 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1272ff2b:::benchmark_data_smithi179_148205_object69019:head# 2024-02-09T17:57:09.560 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12749f7e:::benchmark_data_smithi179_148205_object10674:head# 2024-02-09T17:57:09.560 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1276af5e:::benchmark_data_smithi179_148205_object91100:head# 2024-02-09T17:57:09.561 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1277374b:::benchmark_data_smithi179_148205_object18181:head# 2024-02-09T17:57:09.561 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:127750c4:::benchmark_data_smithi179_148205_object48391:head# 2024-02-09T17:57:09.562 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1277d4cd:::benchmark_data_smithi179_148205_object27750:head# 2024-02-09T17:57:09.562 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12782b21:::benchmark_data_smithi179_148205_object18872:head# 2024-02-09T17:57:09.562 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12788949:::benchmark_data_smithi179_148205_object51065:head# 2024-02-09T17:57:09.563 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:127921c9:::benchmark_data_smithi179_148205_object4940:head# 2024-02-09T17:57:09.563 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:127a00b3:::benchmark_data_smithi179_148205_object65844:head# 2024-02-09T17:57:09.564 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:127ab06b:::benchmark_data_smithi179_148205_object22103:head# 2024-02-09T17:57:09.564 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:127af979:::benchmark_data_smithi179_148205_object38247:head# 2024-02-09T17:57:09.567 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:127b62f0:::benchmark_data_smithi179_148205_object48999:head# 2024-02-09T17:57:09.567 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:127c3e10:::benchmark_data_smithi179_148205_object46492:head# 2024-02-09T17:57:09.567 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:127c5948:::benchmark_data_smithi179_148205_object21137:head# 2024-02-09T17:57:09.567 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:127d300e:::benchmark_data_smithi179_148205_object30040:head# 2024-02-09T17:57:09.567 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:127d325c:::benchmark_data_smithi179_148205_object40381:head# 2024-02-09T17:57:09.567 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:127dca9d:::benchmark_data_smithi179_148205_object44982:head# 2024-02-09T17:57:09.567 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:127e359c:::benchmark_data_smithi179_148205_object4005:head# 2024-02-09T17:57:09.568 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:127ea8ad:::benchmark_data_smithi179_148205_object39581:head# 2024-02-09T17:57:09.568 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:127ee740:::benchmark_data_smithi179_148205_object26235:head# 2024-02-09T17:57:09.569 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:128008c7:::benchmark_data_smithi179_148205_object16880:head# 2024-02-09T17:57:09.569 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1280362e:::benchmark_data_smithi179_148205_object105018:head# 2024-02-09T17:57:09.569 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:128040e5:::benchmark_data_smithi179_148205_object28290:head# 2024-02-09T17:57:09.570 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12809ccb:::benchmark_data_smithi179_148205_object48371:head# 2024-02-09T17:57:09.570 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1280d548:::benchmark_data_smithi179_148205_object48327:head# 2024-02-09T17:57:09.571 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1280ee8a:::benchmark_data_smithi179_148205_object21101:head# 2024-02-09T17:57:09.571 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1281d20d:::benchmark_data_smithi179_148205_object67948:head# 2024-02-09T17:57:09.576 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:128259d8:::benchmark_data_smithi179_148205_object84399:head# 2024-02-09T17:57:09.577 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1282cabc:::benchmark_data_smithi179_148205_object100402:head# 2024-02-09T17:57:09.577 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12835ed9:::benchmark_data_smithi179_148205_object107640:head# 2024-02-09T17:57:09.577 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12857a75:::benchmark_data_smithi179_148205_object56569:head# 2024-02-09T17:57:09.577 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1285c2f2:::benchmark_data_smithi179_148205_object5181:head# 2024-02-09T17:57:09.577 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1285e5a3:::benchmark_data_smithi179_148205_object91597:head# 2024-02-09T17:57:09.577 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12876757:::benchmark_data_smithi179_148205_object19951:head# 2024-02-09T17:57:09.577 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:128769ef:::benchmark_data_smithi179_148205_object100217:head# 2024-02-09T17:57:09.577 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1287b6d0:::benchmark_data_smithi179_148205_object78436:head# 2024-02-09T17:57:09.577 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12895267:::benchmark_data_smithi179_148205_object10302:head# 2024-02-09T17:57:09.577 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1289ddd2:::benchmark_data_smithi179_148205_object81962:head# 2024-02-09T17:57:09.577 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:128aba66:::benchmark_data_smithi179_148205_object46020:head# 2024-02-09T17:57:09.577 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:128ace04:::benchmark_data_smithi179_148205_object37056:head# 2024-02-09T17:57:09.578 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:128bc92e:::benchmark_data_smithi179_148205_object41039:head# 2024-02-09T17:57:09.578 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:128c5cd5:::benchmark_data_smithi179_148205_object1603:head# 2024-02-09T17:57:09.578 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:128cb753:::benchmark_data_smithi179_148205_object86886:head# 2024-02-09T17:57:09.578 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:128d5431:::benchmark_data_smithi179_148205_object47077:head# 2024-02-09T17:57:09.579 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:128e58d9:::benchmark_data_smithi179_148205_object70087:head# 2024-02-09T17:57:09.579 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:128ee542:::benchmark_data_smithi179_148205_object61272:head# 2024-02-09T17:57:09.579 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:128f776d:::benchmark_data_smithi179_148205_object106970:head# 2024-02-09T17:57:09.580 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:128f7b4b:::benchmark_data_smithi179_148205_object93127:head# 2024-02-09T17:57:09.581 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:128ffe79:::benchmark_data_smithi179_148205_object84735:head# 2024-02-09T17:57:09.581 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:129006ed:::benchmark_data_smithi179_148205_object77382:head# 2024-02-09T17:57:09.581 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12900706:::benchmark_data_smithi179_148205_object32715:head# 2024-02-09T17:57:09.582 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:09.585 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12914830:::benchmark_data_smithi179_148205_object2057:head# 2024-02-09T17:57:09.586 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12914e54:::benchmark_data_smithi179_148205_object43143:head# 2024-02-09T17:57:09.586 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12919cac:::benchmark_data_smithi179_148205_object47847:head# 2024-02-09T17:57:09.586 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12923dde:::benchmark_data_smithi179_148205_object69129:head# 2024-02-09T17:57:09.587 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1292582d:::benchmark_data_smithi179_148205_object38745:head# 2024-02-09T17:57:09.587 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1293f8b4:::benchmark_data_smithi179_148205_object53672:head# 2024-02-09T17:57:09.587 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1294dffc:::benchmark_data_smithi179_148205_object50672:head# 2024-02-09T17:57:09.588 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12952dad:::benchmark_data_smithi179_148205_object29394:head# 2024-02-09T17:57:09.588 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12966a9b:::benchmark_data_smithi179_148205_object1902:head# 2024-02-09T17:57:09.589 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1297c90f:::benchmark_data_smithi179_148205_object39351:head# 2024-02-09T17:57:09.589 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:129827a9:::benchmark_data_smithi179_148205_object64057:head# 2024-02-09T17:57:09.589 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1298412e:::benchmark_data_smithi179_148205_object75740:head# 2024-02-09T17:57:09.590 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12985efa:::benchmark_data_smithi179_148205_object66717:head# 2024-02-09T17:57:09.590 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:129a0616:::benchmark_data_smithi179_148205_object13530:head# 2024-02-09T17:57:09.592 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:129a80d7:::benchmark_data_smithi179_148205_object44754:head# 2024-02-09T17:57:09.593 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:129af014:::benchmark_data_smithi179_148205_object28043:head# 2024-02-09T17:57:09.593 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:129af619:::benchmark_data_smithi179_148205_object31080:head# 2024-02-09T17:57:09.593 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:129b51c6:::benchmark_data_smithi179_148205_object41663:head# 2024-02-09T17:57:09.593 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:129be3b6:::benchmark_data_smithi179_148205_object49610:head# 2024-02-09T17:57:09.593 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:129c37d1:::benchmark_data_smithi179_148205_object64621:head# 2024-02-09T17:57:09.594 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:129c6a9b:::benchmark_data_smithi179_148205_object97421:head# 2024-02-09T17:57:09.594 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:129ce399:::benchmark_data_smithi179_148205_object77307:head# 2024-02-09T17:57:09.594 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:129e808f:::benchmark_data_smithi179_148205_object22116:head# 2024-02-09T17:57:09.594 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:129fa649:::benchmark_data_smithi179_148205_object7431:head# 2024-02-09T17:57:09.595 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12a0276a:::benchmark_data_smithi179_148205_object103013:head# 2024-02-09T17:57:09.595 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12a0977e:::benchmark_data_smithi179_148205_object85233:head# 2024-02-09T17:57:09.596 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12a22520:::benchmark_data_smithi179_148205_object39338:head# 2024-02-09T17:57:09.596 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12a22f18:::benchmark_data_smithi179_148205_object89056:head# 2024-02-09T17:57:09.597 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12a2551f:::benchmark_data_smithi179_148205_object34409:head# 2024-02-09T17:57:09.597 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12a3e397:::benchmark_data_smithi179_148205_object86180:head# 2024-02-09T17:57:09.597 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12a4746b:::benchmark_data_smithi179_148205_object89073:head# 2024-02-09T17:57:09.598 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12a4df43:::benchmark_data_smithi179_148205_object67453:head# 2024-02-09T17:57:09.598 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12a7a2b9:::benchmark_data_smithi179_148205_object33869:head# 2024-02-09T17:57:09.599 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12a8a821:::benchmark_data_smithi179_148205_object42134:head# 2024-02-09T17:57:09.600 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12a97188:::benchmark_data_smithi179_148205_object6215:head# 2024-02-09T17:57:09.602 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12aa4b2a:::benchmark_data_smithi179_148205_object58515:head# 2024-02-09T17:57:09.603 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12aad64f:::benchmark_data_smithi179_148205_object7493:head# 2024-02-09T17:57:09.603 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12aaf7c4:::benchmark_data_smithi179_148205_object20637:head# 2024-02-09T17:57:09.603 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12acbe79:::benchmark_data_smithi179_148205_object52129:head# 2024-02-09T17:57:09.603 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12ae2255:::benchmark_data_smithi179_148205_object46655:head# 2024-02-09T17:57:09.604 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12aeed36:::benchmark_data_smithi179_148205_object18785:head# 2024-02-09T17:57:09.604 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12af425e:::benchmark_data_smithi179_148205_object7192:head# 2024-02-09T17:57:09.605 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12af570a:::benchmark_data_smithi179_148205_object43705:head# 2024-02-09T17:57:09.605 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12af954b:::benchmark_data_smithi179_148205_object75809:head# 2024-02-09T17:57:09.605 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12afcd83:::benchmark_data_smithi179_148205_object34591:head# 2024-02-09T17:57:09.606 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12b039f9:::benchmark_data_smithi179_148205_object37066:head# 2024-02-09T17:57:09.606 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12b0a143:::benchmark_data_smithi179_148205_object13120:head# 2024-02-09T17:57:09.607 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12b0f21f:::benchmark_data_smithi179_148205_object31160:head# 2024-02-09T17:57:09.607 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12b10aa6:::benchmark_data_smithi179_148205_object17928:head# 2024-02-09T17:57:09.607 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12b13cfc:::benchmark_data_smithi179_148205_object55109:head# 2024-02-09T17:57:09.608 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12b15e36:::benchmark_data_smithi179_148205_object34777:head# 2024-02-09T17:57:09.608 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12b1ea6b:::benchmark_data_smithi179_148205_object9153:head# 2024-02-09T17:57:09.609 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12b30aae:::benchmark_data_smithi179_148205_object107214:head# 2024-02-09T17:57:09.609 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12b435bb:::benchmark_data_smithi179_148205_object103503:head# 2024-02-09T17:57:09.610 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12b52140:::benchmark_data_smithi179_148205_object82148:head# 2024-02-09T17:57:09.610 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12b59d51:::benchmark_data_smithi179_148205_object109335:head# 2024-02-09T17:57:09.611 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12b6f1c5:::benchmark_data_smithi179_148205_object7139:head# 2024-02-09T17:57:09.611 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12b75dac:::benchmark_data_smithi179_148205_object86403:head# 2024-02-09T17:57:09.611 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12b7c414:::benchmark_data_smithi179_148205_object55388:head# 2024-02-09T17:57:09.613 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12b85048:::benchmark_data_smithi179_148205_object28000:head# 2024-02-09T17:57:09.615 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:57:09.617 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12b85d47:::benchmark_data_smithi179_148205_object52785:head# 2024-02-09T17:57:09.618 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12b8815d:::benchmark_data_smithi179_148205_object19905:head# 2024-02-09T17:57:09.618 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12b9a3c1:::benchmark_data_smithi179_148205_object85131:head# 2024-02-09T17:57:09.618 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12b9f1a2:::benchmark_data_smithi179_148205_object80282:head# 2024-02-09T17:57:09.619 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12ba8efa:::benchmark_data_smithi179_148205_object82430:head# 2024-02-09T17:57:09.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:09.621 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:57:09.626 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12bb0da9:::benchmark_data_smithi179_148205_object94199:head# 2024-02-09T17:57:09.626 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12bb422c:::benchmark_data_smithi179_148205_object78007:head# 2024-02-09T17:57:09.627 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12bc5a3e:::benchmark_data_smithi179_148205_object24180:head# 2024-02-09T17:57:09.627 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12bdb398:::benchmark_data_smithi179_148205_object81855:head# 2024-02-09T17:57:09.627 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12bf6215:::benchmark_data_smithi179_148205_object100911:head# 2024-02-09T17:57:09.628 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12bfbfb0:::benchmark_data_smithi179_148205_object47426:head# 2024-02-09T17:57:09.628 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12c1cb89:::benchmark_data_smithi179_148205_object92354:head# 2024-02-09T17:57:09.629 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12c1de95:::benchmark_data_smithi179_148205_object15464:head# 2024-02-09T17:57:09.630 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12c24e24:::benchmark_data_smithi179_148205_object97624:head# 2024-02-09T17:57:09.630 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12c26d40:::benchmark_data_smithi179_148205_object46755:head# 2024-02-09T17:57:09.630 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12c2d9ed:::benchmark_data_smithi179_148205_object83203:head# 2024-02-09T17:57:09.631 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12c30a15:::benchmark_data_smithi179_148205_object59799:head# 2024-02-09T17:57:09.631 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12c3968e:::benchmark_data_smithi179_148205_object105539:head# 2024-02-09T17:57:09.631 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12c3e6b9:::benchmark_data_smithi179_148205_object16878:head# 2024-02-09T17:57:09.632 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12c400ec:::benchmark_data_smithi179_148205_object34136:head# 2024-02-09T17:57:09.634 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12c41e71:::benchmark_data_smithi179_148205_object57753:head# 2024-02-09T17:57:09.634 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12c47328:::benchmark_data_smithi179_148205_object14318:head# 2024-02-09T17:57:09.634 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12c51a04:::benchmark_data_smithi179_148205_object102955:head# 2024-02-09T17:57:09.634 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12c5586c:::benchmark_data_smithi179_148205_object96860:head# 2024-02-09T17:57:09.634 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12c56141:::benchmark_data_smithi179_148205_object60979:head# 2024-02-09T17:57:09.635 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12c561e5:::benchmark_data_smithi179_148205_object38858:head# 2024-02-09T17:57:09.635 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12c58838:::benchmark_data_smithi179_148205_object109663:head# 2024-02-09T17:57:09.636 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12c6704b:::benchmark_data_smithi179_148205_object91315:head# 2024-02-09T17:57:09.636 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12c6a16e:::benchmark_data_smithi179_148205_object82125:head# 2024-02-09T17:57:09.636 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12c6e479:::benchmark_data_smithi179_148205_object19291:head# 2024-02-09T17:57:09.637 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12c80300:::benchmark_data_smithi179_148205_object108222:head# 2024-02-09T17:57:09.637 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12c869bd:::benchmark_data_smithi179_148205_object74531:head# 2024-02-09T17:57:09.637 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12c87d43:::benchmark_data_smithi179_148205_object47619:head# 2024-02-09T17:57:09.638 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12c895ae:::benchmark_data_smithi179_148205_object64097:head# 2024-02-09T17:57:09.638 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12c8a69d:::benchmark_data_smithi179_148205_object93188:head# 2024-02-09T17:57:09.639 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12c8b659:::benchmark_data_smithi179_148205_object30607:head# 2024-02-09T17:57:09.639 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12c94b10:::benchmark_data_smithi179_148205_object32711:head# 2024-02-09T17:57:09.640 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12c9a9bb:::benchmark_data_smithi179_148205_object23514:head# 2024-02-09T17:57:09.640 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12ca5d71:::benchmark_data_smithi179_148205_object83047:head# 2024-02-09T17:57:09.640 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12cb5cde:::benchmark_data_smithi179_148205_object21769:head# 2024-02-09T17:57:09.643 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12cbaccc:::benchmark_data_smithi179_148205_object76503:head# 2024-02-09T17:57:09.643 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12cbb201:::benchmark_data_smithi179_148205_object62366:head# 2024-02-09T17:57:09.643 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12cbb2e2:::benchmark_data_smithi179_148205_object107276:head# 2024-02-09T17:57:09.643 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12cbc1a2:::benchmark_data_smithi179_148205_object79844:head# 2024-02-09T17:57:09.643 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12cbef5e:::benchmark_data_smithi179_148205_object9635:head# 2024-02-09T17:57:09.643 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12ccab2a:::benchmark_data_smithi179_148205_object89757:head# 2024-02-09T17:57:09.643 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12ccc712:::benchmark_data_smithi179_148205_object92602:head# 2024-02-09T17:57:09.644 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12cdcfd7:::benchmark_data_smithi179_148205_object25840:head# 2024-02-09T17:57:09.644 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12cf8620:::benchmark_data_smithi179_148205_object64780:head# 2024-02-09T17:57:09.645 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12cf98c0:::benchmark_data_smithi179_148205_object104622:head# 2024-02-09T17:57:09.645 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12cf9de8:::benchmark_data_smithi179_148205_object53317:head# 2024-02-09T17:57:09.647 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12d09007:::benchmark_data_smithi179_148205_object55839:head# 2024-02-09T17:57:09.647 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12d0fffd:::benchmark_data_smithi179_148205_object26607:head# 2024-02-09T17:57:09.648 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12d23db8:::benchmark_data_smithi179_148205_object75396:head# 2024-02-09T17:57:09.648 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12d34506:::benchmark_data_smithi179_148205_object7866:head# 2024-02-09T17:57:09.648 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12d40aba:::benchmark_data_smithi179_148205_object29430:head# 2024-02-09T17:57:09.649 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12d45357:::benchmark_data_smithi179_148205_object74055:head# 2024-02-09T17:57:09.649 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12d5172d:::benchmark_data_smithi179_148205_object40382:head# 2024-02-09T17:57:09.650 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12d69baf:::benchmark_data_smithi179_148205_object69654:head# 2024-02-09T17:57:09.650 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12d6b7fc:::benchmark_data_smithi179_148205_object61134:head# 2024-02-09T17:57:09.651 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12d6fc31:::benchmark_data_smithi179_148205_object13646:head# 2024-02-09T17:57:09.651 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12d8fefa:::benchmark_data_smithi179_148205_object55634:head# 2024-02-09T17:57:09.652 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12d98202:::benchmark_data_smithi179_148205_object57462:head# 2024-02-09T17:57:09.652 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12da7254:::benchmark_data_smithi179_148205_object65479:head# 2024-02-09T17:57:09.653 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12daf8fb:::benchmark_data_smithi179_148205_object82903:head# 2024-02-09T17:57:09.653 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12dcbc7e:::benchmark_data_smithi179_148205_object87667:head# 2024-02-09T17:57:09.653 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12dcf565:::benchmark_data_smithi179_148205_object107411:head# 2024-02-09T17:57:09.654 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12dd517c:::benchmark_data_smithi179_148205_object22304:head# 2024-02-09T17:57:09.655 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12de16e3:::benchmark_data_smithi179_148205_object72913:head# 2024-02-09T17:57:09.655 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12df4224:::benchmark_data_smithi179_148205_object52536:head# 2024-02-09T17:57:09.655 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12df9a25:::benchmark_data_smithi179_148205_object25248:head# 2024-02-09T17:57:09.656 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12dfb847:::benchmark_data_smithi179_148205_object106297:head# 2024-02-09T17:57:09.656 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12dfd150:::benchmark_data_smithi179_148205_object45026:head# 2024-02-09T17:57:09.656 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12e01c71:::benchmark_data_smithi179_148205_object36419:head# 2024-02-09T17:57:09.657 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12e09697:::benchmark_data_smithi179_148205_object75583:head# 2024-02-09T17:57:09.658 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12e0b81d:::benchmark_data_smithi179_148205_object3033:head# 2024-02-09T17:57:09.658 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12e13467:::benchmark_data_smithi179_148205_object3787:head# 2024-02-09T17:57:09.658 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12e279ba:::benchmark_data_smithi179_148205_object60316:head# 2024-02-09T17:57:09.659 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12e2fd4a:::benchmark_data_smithi179_148205_object45694:head# 2024-02-09T17:57:09.659 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12e3124b:::benchmark_data_smithi179_148205_object59126:head# 2024-02-09T17:57:09.659 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12e384d7:::benchmark_data_smithi179_148205_object84907:head# 2024-02-09T17:57:09.660 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12e532fb:::benchmark_data_smithi179_148205_object24842:head# 2024-02-09T17:57:09.660 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12e58840:::benchmark_data_smithi179_148205_object27860:head# 2024-02-09T17:57:09.661 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12e5baa8:::benchmark_data_smithi179_148205_object35122:head# 2024-02-09T17:57:09.661 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12e5e5d4:::benchmark_data_smithi179_148205_object59681:head# 2024-02-09T17:57:09.661 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12e6f01a:::benchmark_data_smithi179_148205_object105203:head# 2024-02-09T17:57:09.662 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12e94ec3:::benchmark_data_smithi179_148205_object35341:head# 2024-02-09T17:57:09.663 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12e967a5:::benchmark_data_smithi179_148205_object79391:head# 2024-02-09T17:57:09.663 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12ea36e6:::benchmark_data_smithi179_148205_object79572:head# 2024-02-09T17:57:09.664 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12ea42da:::benchmark_data_smithi179_148205_object96456:head# 2024-02-09T17:57:09.664 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12ea45c9:::benchmark_data_smithi179_148205_object89752:head# 2024-02-09T17:57:09.665 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12eaa2c7:::benchmark_data_smithi179_148205_object20203:head# 2024-02-09T17:57:09.665 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12eafedb:::benchmark_data_smithi179_148205_object89809:head# 2024-02-09T17:57:09.666 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12ebf682:::benchmark_data_smithi179_148205_object1628:head# 2024-02-09T17:57:09.666 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12ec18a5:::benchmark_data_smithi179_148205_object57528:head# 2024-02-09T17:57:09.667 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12ec476f:::benchmark_data_smithi179_148205_object105927:head# 2024-02-09T17:57:09.667 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12ec61ca:::benchmark_data_smithi179_148205_object4479:head# 2024-02-09T17:57:09.668 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12ed2836:::benchmark_data_smithi179_148205_object12709:head# 2024-02-09T17:57:09.668 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12eeb65e:::benchmark_data_smithi179_148205_object23133:head# 2024-02-09T17:57:09.669 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12efcd54:::benchmark_data_smithi179_148205_object63275:head# 2024-02-09T17:57:09.671 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12f0992b:::benchmark_data_smithi179_148205_object97156:head# 2024-02-09T17:57:09.671 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12f09d43:::benchmark_data_smithi179_148205_object106863:head# 2024-02-09T17:57:09.671 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12f0e3fc:::benchmark_data_smithi179_148205_object63429:head# 2024-02-09T17:57:09.671 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12f1e64d:::benchmark_data_smithi179_148205_object18941:head# 2024-02-09T17:57:09.671 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12f1fcbe:::benchmark_data_smithi179_148205_object36701:head# 2024-02-09T17:57:09.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:09 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3363208437' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:57:09.672 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12f22dd5:::benchmark_data_smithi179_148205_object5433:head# 2024-02-09T17:57:09.672 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12f22fc0:::benchmark_data_smithi179_148205_object52952:head# 2024-02-09T17:57:09.672 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12f40847:::benchmark_data_smithi179_148205_object19034:head# 2024-02-09T17:57:09.673 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12f471b1:::benchmark_data_smithi179_148205_object87630:head# 2024-02-09T17:57:09.673 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12f587da:::benchmark_data_smithi179_148205_object41584:head# 2024-02-09T17:57:09.673 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12f5b57e:::benchmark_data_smithi179_148205_object1742:head# 2024-02-09T17:57:09.674 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12f68670:::benchmark_data_smithi179_148205_object31360:head# 2024-02-09T17:57:09.674 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12f6bad2:::benchmark_data_smithi179_148205_object11489:head# 2024-02-09T17:57:09.674 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12f7259a:::benchmark_data_smithi179_148205_object72014:head# 2024-02-09T17:57:09.675 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12f72a82:::benchmark_data_smithi179_148205_object32255:head# 2024-02-09T17:57:09.675 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12f75678:::benchmark_data_smithi179_148205_object69954:head# 2024-02-09T17:57:09.676 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12f7d7e8:::benchmark_data_smithi179_148205_object29092:head# 2024-02-09T17:57:09.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:09 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3363208437' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:57:09.677 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12f86730:::benchmark_data_smithi179_148205_object22543:head# 2024-02-09T17:57:09.677 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12f921e0:::benchmark_data_smithi179_148205_object31028:head# 2024-02-09T17:57:09.677 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12f94fbf:::benchmark_data_smithi179_148205_object36290:head# 2024-02-09T17:57:09.678 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12f96606:::benchmark_data_smithi179_148205_object53669:head# 2024-02-09T17:57:09.678 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12fa7269:::benchmark_data_smithi179_148205_object6143:head# 2024-02-09T17:57:09.679 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12faf90c:::benchmark_data_smithi179_148205_object21185:head# 2024-02-09T17:57:09.679 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12fbed47:::benchmark_data_smithi179_148205_object81535:head# 2024-02-09T17:57:09.679 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12fc69cb:::benchmark_data_smithi179_148205_object45924:head# 2024-02-09T17:57:09.680 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12fc79c5:::benchmark_data_smithi179_148205_object29124:head# 2024-02-09T17:57:09.680 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12fcbe30:::benchmark_data_smithi179_148205_object17363:head# 2024-02-09T17:57:09.681 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12fd9112:::benchmark_data_smithi179_148205_object105337:head# 2024-02-09T17:57:09.685 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12fe2fc9:::benchmark_data_smithi179_148205_object44651:head# 2024-02-09T17:57:09.685 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12fe38ac:::benchmark_data_smithi179_148205_object3338:head# 2024-02-09T17:57:09.685 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12fe4ada:::benchmark_data_smithi179_148205_object101364:head# 2024-02-09T17:57:09.686 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:12feeacb:::benchmark_data_smithi179_148205_object93371:head# 2024-02-09T17:57:09.686 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1300e95b:::benchmark_data_smithi179_148205_object67671:head# 2024-02-09T17:57:09.686 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13010c54:::benchmark_data_smithi179_148205_object70768:head# 2024-02-09T17:57:09.686 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1301aae5:::benchmark_data_smithi179_148205_object24807:head# 2024-02-09T17:57:09.687 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1302ee88:::benchmark_data_smithi179_148205_object31681:head# 2024-02-09T17:57:09.687 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:130383f4:::benchmark_data_smithi179_148205_object60143:head# 2024-02-09T17:57:09.687 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13038822:::benchmark_data_smithi179_148205_object95216:head# 2024-02-09T17:57:09.688 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1304be46:::benchmark_data_smithi179_148205_object56323:head# 2024-02-09T17:57:09.688 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1305b414:::benchmark_data_smithi179_148205_object85951:head# 2024-02-09T17:57:09.689 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1305bb4f:::benchmark_data_smithi179_148205_object21724:head# 2024-02-09T17:57:09.689 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13060308:::benchmark_data_smithi179_148205_object18516:head# 2024-02-09T17:57:09.690 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13067e0e:::benchmark_data_smithi179_148205_object39365:head# 2024-02-09T17:57:09.695 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:130752c4:::benchmark_data_smithi179_148205_object50806:head# 2024-02-09T17:57:09.696 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13085f57:::benchmark_data_smithi179_148205_object89777:head# 2024-02-09T17:57:09.696 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13090e12:::benchmark_data_smithi179_148205_object29272:head# 2024-02-09T17:57:09.697 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13099313:::benchmark_data_smithi179_148205_object2765:head# 2024-02-09T17:57:09.697 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1309abb0:::benchmark_data_smithi179_148205_object40708:head# 2024-02-09T17:57:09.698 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1309dcea:::benchmark_data_smithi179_148205_object89238:head# 2024-02-09T17:57:09.698 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:130b446d:::benchmark_data_smithi179_148205_object43787:head# 2024-02-09T17:57:09.699 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:130c70b8:::benchmark_data_smithi179_148205_object39300:head# 2024-02-09T17:57:09.699 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:130c9b64:::benchmark_data_smithi179_148205_object52845:head# 2024-02-09T17:57:09.699 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:130ca56d:::benchmark_data_smithi179_148205_object58310:head# 2024-02-09T17:57:09.700 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:130ca64f:::benchmark_data_smithi179_148205_object77591:head# 2024-02-09T17:57:09.700 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:130cf7e7:::benchmark_data_smithi179_148205_object22210:head# 2024-02-09T17:57:09.701 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:130cf8e1:::benchmark_data_smithi179_148205_object37910:head# 2024-02-09T17:57:09.701 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:130d15e0:::benchmark_data_smithi179_148205_object62809:head# 2024-02-09T17:57:09.701 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:130e9f29:::benchmark_data_smithi179_148205_object57852:head# 2024-02-09T17:57:09.702 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13102830:::benchmark_data_smithi179_148205_object34701:head# 2024-02-09T17:57:09.702 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1311dddf:::benchmark_data_smithi179_148205_object2516:head# 2024-02-09T17:57:09.702 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13130211:::benchmark_data_smithi179_148205_object92989:head# 2024-02-09T17:57:09.703 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1313606f:::benchmark_data_smithi179_148205_object54270:head# 2024-02-09T17:57:09.703 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1313b9cf:::benchmark_data_smithi179_148205_object97814:head# 2024-02-09T17:57:09.704 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1315ba69:::benchmark_data_smithi179_148205_object46965:head# 2024-02-09T17:57:09.704 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1316f50a:::benchmark_data_smithi179_148205_object75580:head# 2024-02-09T17:57:09.705 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1316ffd9:::benchmark_data_smithi179_148205_object103183:head# 2024-02-09T17:57:09.705 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13186b3f:::benchmark_data_smithi179_148205_object80115:head# 2024-02-09T17:57:09.709 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1318a619:::benchmark_data_smithi179_148205_object41533:head# 2024-02-09T17:57:09.710 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:131902e1:::benchmark_data_smithi179_148205_object71784:head# 2024-02-09T17:57:09.710 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13194868:::benchmark_data_smithi179_148205_object14127:head# 2024-02-09T17:57:09.710 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1319fab2:::benchmark_data_smithi179_148205_object82272:head# 2024-02-09T17:57:09.710 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:131a39d4:::benchmark_data_smithi179_148205_object81428:head# 2024-02-09T17:57:09.710 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:131ae5c1:::benchmark_data_smithi179_148205_object5299:head# 2024-02-09T17:57:09.710 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:131b6bf6:::benchmark_data_smithi179_148205_object46711:head# 2024-02-09T17:57:09.710 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:131c4aa3:::benchmark_data_smithi179_148205_object74738:head# 2024-02-09T17:57:09.710 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:131c803d:::benchmark_data_smithi179_148205_object69790:head# 2024-02-09T17:57:09.710 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:131ca151:::benchmark_data_smithi179_148205_object15947:head# 2024-02-09T17:57:09.711 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:131d00d1:::benchmark_data_smithi179_148205_object97789:head# 2024-02-09T17:57:09.711 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:131d433c:::benchmark_data_smithi179_148205_object27985:head# 2024-02-09T17:57:09.711 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:131d47ef:::benchmark_data_smithi179_148205_object34414:head# 2024-02-09T17:57:09.711 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:131eea16:::benchmark_data_smithi179_148205_object46652:head# 2024-02-09T17:57:09.712 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:131f1ab4:::benchmark_data_smithi179_148205_object16738:head# 2024-02-09T17:57:09.712 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:131f2005:::benchmark_data_smithi179_148205_object21691:head# 2024-02-09T17:57:09.713 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:131f394d:::benchmark_data_smithi179_148205_object108488:head# 2024-02-09T17:57:09.713 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:131f9c5e:::benchmark_data_smithi179_148205_object40892:head# 2024-02-09T17:57:09.714 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1320ab8e:::benchmark_data_smithi179_148205_object41960:head# 2024-02-09T17:57:09.715 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:132106cf:::benchmark_data_smithi179_148205_object94709:head# 2024-02-09T17:57:09.715 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:132343fa:::benchmark_data_smithi179_148205_object32630:head# 2024-02-09T17:57:09.716 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1323b77e:::benchmark_data_smithi179_148205_object35425:head# 2024-02-09T17:57:09.716 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13243f89:::benchmark_data_smithi179_148205_object107266:head# 2024-02-09T17:57:09.716 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13246f06:::benchmark_data_smithi179_148205_object44912:head# 2024-02-09T17:57:09.716 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13250171:::benchmark_data_smithi179_148205_object106840:head# 2024-02-09T17:57:09.717 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:132619f5:::benchmark_data_smithi179_148205_object53814:head# 2024-02-09T17:57:09.717 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13264035:::benchmark_data_smithi179_148205_object20782:head# 2024-02-09T17:57:09.717 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:132652ea:::benchmark_data_smithi179_148205_object90544:head# 2024-02-09T17:57:09.718 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1326b748:::benchmark_data_smithi179_148205_object11768:head# 2024-02-09T17:57:09.718 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13277a98:::benchmark_data_smithi179_148205_object49969:head# 2024-02-09T17:57:09.719 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1327885b:::benchmark_data_smithi179_148205_object20841:head# 2024-02-09T17:57:09.719 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13279337:::benchmark_data_smithi179_148205_object13145:head# 2024-02-09T17:57:09.721 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:57:09.724 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13285918:::benchmark_data_smithi179_148205_object9903:head# 2024-02-09T17:57:09.724 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:132940a0:::benchmark_data_smithi179_148205_object40321:head# 2024-02-09T17:57:09.724 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:132ae190:::benchmark_data_smithi179_148205_object64378:head# 2024-02-09T17:57:09.724 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:132ba8ea:::benchmark_data_smithi179_148205_object1433:head# 2024-02-09T17:57:09.724 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:132bee62:::benchmark_data_smithi179_148205_object75175:head# 2024-02-09T17:57:09.724 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:132bf5af:::benchmark_data_smithi179_148205_object87800:head# 2024-02-09T17:57:09.724 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:132c5ff5:::benchmark_data_smithi179_148205_object76233:head# 2024-02-09T17:57:09.724 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:132c6b35:::benchmark_data_smithi179_148205_object72559:head# 2024-02-09T17:57:09.724 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:132d5d24:::benchmark_data_smithi179_148205_object32865:head# 2024-02-09T17:57:09.724 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:132dad37:::benchmark_data_smithi179_148205_object46598:head# 2024-02-09T17:57:09.725 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:132dd5bf:::benchmark_data_smithi179_148205_object22075:head# 2024-02-09T17:57:09.729 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:132e65d1:::benchmark_data_smithi179_148205_object93343:head# 2024-02-09T17:57:09.730 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:132f1433:::benchmark_data_smithi179_148205_object24555:head# 2024-02-09T17:57:09.730 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1330213c:::benchmark_data_smithi179_148205_object47670:head# 2024-02-09T17:57:09.730 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13303a66:::benchmark_data_smithi179_148205_object37986:head# 2024-02-09T17:57:09.730 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:133162bb:::benchmark_data_smithi179_148205_object72592:head# 2024-02-09T17:57:09.730 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1331f06e:::benchmark_data_smithi179_148205_object13352:head# 2024-02-09T17:57:09.730 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1331f267:::benchmark_data_smithi179_148205_object95275:head# 2024-02-09T17:57:09.730 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13320f8d:::benchmark_data_smithi179_148205_object78949:head# 2024-02-09T17:57:09.730 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:133212e4:::benchmark_data_smithi179_148205_object107408:head# 2024-02-09T17:57:09.731 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:133279ec:::benchmark_data_smithi179_148205_object75398:head# 2024-02-09T17:57:09.731 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1332998a:::benchmark_data_smithi179_148205_object7744:head# 2024-02-09T17:57:09.732 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1332d77e:::benchmark_data_smithi179_148205_object52280:head# 2024-02-09T17:57:09.732 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13347a09:::benchmark_data_smithi179_148205_object59647:head# 2024-02-09T17:57:09.732 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:133491aa:::benchmark_data_smithi179_148205_object71299:head# 2024-02-09T17:57:09.733 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13357928:::benchmark_data_smithi179_148205_object44655:head# 2024-02-09T17:57:09.733 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1335b079:::benchmark_data_smithi179_148205_object107155:head# 2024-02-09T17:57:09.733 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1337b189:::benchmark_data_smithi179_148205_object4978:head# 2024-02-09T17:57:09.734 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13388214:::benchmark_data_smithi179_148205_object19692:head# 2024-02-09T17:57:09.734 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:133897a8:::benchmark_data_smithi179_148205_object62584:head# 2024-02-09T17:57:09.735 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13394841:::benchmark_data_smithi179_148205_object63993:head# 2024-02-09T17:57:09.735 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:133998f8:::benchmark_data_smithi179_148205_object8666:head# 2024-02-09T17:57:09.735 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1339eddb:::benchmark_data_smithi179_148205_object77036:head# 2024-02-09T17:57:09.736 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:133be0ca:::benchmark_data_smithi179_148205_object106815:head# 2024-02-09T17:57:09.736 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:133ce46d:::benchmark_data_smithi179_148205_object81819:head# 2024-02-09T17:57:09.737 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:133cea49:::benchmark_data_smithi179_148205_object8001:head# 2024-02-09T17:57:09.737 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:133e203d:::benchmark_data_smithi179_148205_object77599:head# 2024-02-09T17:57:09.738 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:133e688d:::benchmark_data_smithi179_148205_object90480:head# 2024-02-09T17:57:09.738 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:133f6d29:::benchmark_data_smithi179_148205_object13108:head# 2024-02-09T17:57:09.739 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13407e98:::benchmark_data_smithi179_148205_object12554:head# 2024-02-09T17:57:09.740 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1340aef2:::benchmark_data_smithi179_148205_object5146:head# 2024-02-09T17:57:09.740 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1341058d:::benchmark_data_smithi179_148205_object87644:head# 2024-02-09T17:57:09.741 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1341b318:::benchmark_data_smithi179_148205_object101661:head# 2024-02-09T17:57:09.741 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13421b7b:::benchmark_data_smithi179_148205_object33261:head# 2024-02-09T17:57:09.741 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1344eaa0:::benchmark_data_smithi179_148205_object81911:head# 2024-02-09T17:57:09.742 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13458684:::benchmark_data_smithi179_148205_object44811:head# 2024-02-09T17:57:09.745 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1345cead:::benchmark_data_smithi179_148205_object70900:head# 2024-02-09T17:57:09.745 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13476078:::benchmark_data_smithi179_148205_object65302:head# 2024-02-09T17:57:09.745 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1348a4c9:::benchmark_data_smithi179_148205_object70466:head# 2024-02-09T17:57:09.745 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:134a1d88:::benchmark_data_smithi179_148205_object2048:head# 2024-02-09T17:57:09.745 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:134a81b5:::benchmark_data_smithi179_148205_object91693:head# 2024-02-09T17:57:09.745 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:134a84f1:::benchmark_data_smithi179_148205_object63979:head# 2024-02-09T17:57:09.745 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:134a8696:::benchmark_data_smithi179_148205_object11271:head# 2024-02-09T17:57:09.746 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:134c80e5:::benchmark_data_smithi179_148205_object67254:head# 2024-02-09T17:57:09.746 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:134d3fd5:::benchmark_data_smithi179_148205_object21118:head# 2024-02-09T17:57:09.747 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:134db95f:::benchmark_data_smithi179_148205_object28378:head# 2024-02-09T17:57:09.747 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:134ddb6d:::benchmark_data_smithi179_148205_object75244:head# 2024-02-09T17:57:09.748 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:134e0fc9:::benchmark_data_smithi179_148205_object1227:head# 2024-02-09T17:57:09.748 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:134e8b16:::benchmark_data_smithi179_148205_object4919:head# 2024-02-09T17:57:09.749 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:134e9bb3:::benchmark_data_smithi179_148205_object71327:head# 2024-02-09T17:57:09.749 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:134f0db1:::benchmark_data_smithi179_148205_object102963:head# 2024-02-09T17:57:09.750 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:134fedd7:::benchmark_data_smithi179_148205_object76520:head# 2024-02-09T17:57:09.750 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:135128a3:::benchmark_data_smithi179_148205_object106306:head# 2024-02-09T17:57:09.751 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13527daf:::benchmark_data_smithi179_148205_object9721:head# 2024-02-09T17:57:09.751 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1352c515:::benchmark_data_smithi179_148205_object47523:head# 2024-02-09T17:57:09.752 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1353116b:::benchmark_data_smithi179_148205_object32062:head# 2024-02-09T17:57:09.752 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13531d56:::benchmark_data_smithi179_148205_object104219:head# 2024-02-09T17:57:09.753 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13534c67:::benchmark_data_smithi179_148205_object11561:head# 2024-02-09T17:57:09.753 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13546731:::benchmark_data_smithi179_148205_object52957:head# 2024-02-09T17:57:09.753 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:135499f2:::benchmark_data_smithi179_148205_object34867:head# 2024-02-09T17:57:09.754 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1354ee05:::benchmark_data_smithi179_148205_object33043:head# 2024-02-09T17:57:09.754 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1354f439:::benchmark_data_smithi179_148205_object82964:head# 2024-02-09T17:57:09.755 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13561498:::benchmark_data_smithi179_148205_object92816:head# 2024-02-09T17:57:09.755 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1357b0ab:::benchmark_data_smithi179_148205_object101325:head# 2024-02-09T17:57:09.756 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1358e196:::benchmark_data_smithi179_148205_object81360:head# 2024-02-09T17:57:09.756 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:135ab23b:::benchmark_data_smithi179_148205_object4955:head# 2024-02-09T17:57:09.757 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:135b064b:::benchmark_data_smithi179_148205_object35204:head# 2024-02-09T17:57:09.757 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:135c4a5a:::benchmark_data_smithi179_148205_object103094:head# 2024-02-09T17:57:09.757 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:135ea975:::benchmark_data_smithi179_148205_object98184:head# 2024-02-09T17:57:09.758 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:135f7e82:::benchmark_data_smithi179_148205_object34693:head# 2024-02-09T17:57:09.758 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:135fb210:::benchmark_data_smithi179_148205_object84599:head# 2024-02-09T17:57:09.759 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:136190eb:::benchmark_data_smithi179_148205_object67988:head# 2024-02-09T17:57:09.759 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13619513:::benchmark_data_smithi179_148205_object90926:head# 2024-02-09T17:57:09.760 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1362d224:::benchmark_data_smithi179_148205_object66777:head# 2024-02-09T17:57:09.760 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1362e5c3:::benchmark_data_smithi179_148205_object63512:head# 2024-02-09T17:57:09.760 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:136452ac:::benchmark_data_smithi179_148205_object3603:head# 2024-02-09T17:57:09.761 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:136490a5:::benchmark_data_smithi179_148205_object7882:head# 2024-02-09T17:57:09.763 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13654dac:::benchmark_data_smithi179_148205_object96139:head# 2024-02-09T17:57:09.763 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1366c925:::benchmark_data_smithi179_148205_object44536:head# 2024-02-09T17:57:09.763 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1366ceb9:::benchmark_data_smithi179_148205_object64262:head# 2024-02-09T17:57:09.764 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:136995bc:::benchmark_data_smithi179_148205_object36947:head# 2024-02-09T17:57:09.764 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1369dce7:::benchmark_data_smithi179_148205_object82151:head# 2024-02-09T17:57:09.765 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:136a2b3d:::benchmark_data_smithi179_148205_object62331:head# 2024-02-09T17:57:09.765 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:136b9260:::benchmark_data_smithi179_148205_object78327:head# 2024-02-09T17:57:09.766 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:136bf937:::benchmark_data_smithi179_148205_object58713:head# 2024-02-09T17:57:09.767 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:136c917e:::benchmark_data_smithi179_148205_object74135:head# 2024-02-09T17:57:09.767 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:136cc01d:::benchmark_data_smithi179_148205_object52734:head# 2024-02-09T17:57:09.767 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:136d8935:::benchmark_data_smithi179_148205_object50578:head# 2024-02-09T17:57:09.768 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:136e42fb:::benchmark_data_smithi179_148205_object38232:head# 2024-02-09T17:57:09.768 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:136ec800:::benchmark_data_smithi179_148205_object77320:head# 2024-02-09T17:57:09.769 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:136fadf5:::benchmark_data_smithi179_148205_object25520:head# 2024-02-09T17:57:09.769 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13708711:::benchmark_data_smithi179_148205_object99769:head# 2024-02-09T17:57:09.770 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1370c4c0:::benchmark_data_smithi179_148205_object106339:head# 2024-02-09T17:57:09.770 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1370cc50:::benchmark_data_smithi179_148205_object4440:head# 2024-02-09T17:57:09.771 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1370dbe5:::benchmark_data_smithi179_148205_object83402:head# 2024-02-09T17:57:09.771 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1370e1a1:::benchmark_data_smithi179_148205_object8134:head# 2024-02-09T17:57:09.771 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:137109a9:::benchmark_data_smithi179_148205_object14465:head# 2024-02-09T17:57:09.772 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13718bee:::benchmark_data_smithi179_148205_object76082:head# 2024-02-09T17:57:09.772 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1371a407:::benchmark_data_smithi179_148205_object73028:head# 2024-02-09T17:57:09.773 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1371f2e9:::benchmark_data_smithi179_148205_object11859:head# 2024-02-09T17:57:09.773 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13720c81:::benchmark_data_smithi179_148205_object50496:head# 2024-02-09T17:57:09.773 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13739380:::benchmark_data_smithi179_148205_object47561:head# 2024-02-09T17:57:09.774 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13740f8a:::benchmark_data_smithi179_148205_object50928:head# 2024-02-09T17:57:09.774 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13741543:::benchmark_data_smithi179_148205_object102926:head# 2024-02-09T17:57:09.775 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13744dfb:::benchmark_data_smithi179_148205_object49758:head# 2024-02-09T17:57:09.775 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1374eb4e:::benchmark_data_smithi179_148205_object107995:head# 2024-02-09T17:57:09.775 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13750aa4:::benchmark_data_smithi179_148205_object50939:head# 2024-02-09T17:57:09.776 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13751328:::benchmark_data_smithi179_148205_object68456:head# 2024-02-09T17:57:09.776 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13760b44:::benchmark_data_smithi179_148205_object71663:head# 2024-02-09T17:57:09.777 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13766a1c:::benchmark_data_smithi179_148205_object26038:head# 2024-02-09T17:57:09.777 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1376ebbc:::benchmark_data_smithi179_148205_object105473:head# 2024-02-09T17:57:09.777 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13771a6f:::benchmark_data_smithi179_148205_object30841:head# 2024-02-09T17:57:09.778 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1377671e:::benchmark_data_smithi179_148205_object18610:head# 2024-02-09T17:57:09.779 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13780370:::benchmark_data_smithi179_148205_object33134:head# 2024-02-09T17:57:09.779 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1378060c:::benchmark_data_smithi179_148205_object88639:head# 2024-02-09T17:57:09.780 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13788879:::benchmark_data_smithi179_148205_object54120:head# 2024-02-09T17:57:09.781 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1378c8a5:::benchmark_data_smithi179_148205_object69794:head# 2024-02-09T17:57:09.781 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13790832:::benchmark_data_smithi179_148205_object72187:head# 2024-02-09T17:57:09.781 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:137919c4:::benchmark_data_smithi179_148205_object90460:head# 2024-02-09T17:57:09.781 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13792f32:::benchmark_data_smithi179_148205_object66819:head# 2024-02-09T17:57:09.782 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13795a61:::benchmark_data_smithi179_148205_object68619:head# 2024-02-09T17:57:09.782 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1379b233:::benchmark_data_smithi179_148205_object98647:head# 2024-02-09T17:57:09.783 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1379c7a1:::benchmark_data_smithi179_148205_object55145:head# 2024-02-09T17:57:09.783 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:137ade47:::benchmark_data_smithi179_148205_object28767:head# 2024-02-09T17:57:09.783 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:137b2665:::benchmark_data_smithi179_148205_object66111:head# 2024-02-09T17:57:09.784 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:137b89a1:::benchmark_data_smithi179_148205_object94907:head# 2024-02-09T17:57:09.784 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:137bbad0:::benchmark_data_smithi179_148205_object88450:head# 2024-02-09T17:57:09.785 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:137bf544:::benchmark_data_smithi179_148205_object39574:head# 2024-02-09T17:57:09.785 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:137cac2e:::benchmark_data_smithi179_148205_object2173:head# 2024-02-09T17:57:09.785 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:137d2f95:::benchmark_data_smithi179_148205_object31961:head# 2024-02-09T17:57:09.786 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:137dc4c3:::benchmark_data_smithi179_148205_object104869:head# 2024-02-09T17:57:09.786 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:137e0818:::benchmark_data_smithi179_148205_object71867:head# 2024-02-09T17:57:09.787 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13800afb:::benchmark_data_smithi179_148205_object93827:head# 2024-02-09T17:57:09.787 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1383639d:::benchmark_data_smithi179_148205_object90310:head# 2024-02-09T17:57:09.789 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1383dbc6:::benchmark_data_smithi179_148205_object42866:head# 2024-02-09T17:57:09.789 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1383e751:::benchmark_data_smithi179_148205_object26629:head# 2024-02-09T17:57:09.790 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:138459b1:::benchmark_data_smithi179_148205_object42016:head# 2024-02-09T17:57:09.790 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:09.790 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1385d306:::benchmark_data_smithi179_148205_object69227:head# 2024-02-09T17:57:09.791 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13868368:::benchmark_data_smithi179_148205_object44402:head# 2024-02-09T17:57:09.791 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13868f33:::benchmark_data_smithi179_148205_object101099:head# 2024-02-09T17:57:09.791 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1386ac9b:::benchmark_data_smithi179_148205_object16900:head# 2024-02-09T17:57:09.792 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13876e14:::benchmark_data_smithi179_148205_object64029:head# 2024-02-09T17:57:09.793 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:138773c5:::benchmark_data_smithi179_148205_object79974:head# 2024-02-09T17:57:09.794 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1387b137:::benchmark_data_smithi179_148205_object37905:head# 2024-02-09T17:57:09.794 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1387ca2f:::benchmark_data_smithi179_148205_object78181:head# 2024-02-09T17:57:09.795 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1387d2ae:::benchmark_data_smithi179_148205_object84622:head# 2024-02-09T17:57:09.795 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13888d3b:::benchmark_data_smithi179_148205_object14128:head# 2024-02-09T17:57:09.795 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1388d433:::benchmark_data_smithi179_148205_object23862:head# 2024-02-09T17:57:09.796 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:138974c3:::benchmark_data_smithi179_148205_object18191:head# 2024-02-09T17:57:09.796 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13897fca:::benchmark_data_smithi179_148205_object48040:head# 2024-02-09T17:57:09.796 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1389a9e1:::benchmark_data_smithi179_148205_object1381:head# 2024-02-09T17:57:09.797 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1389bccd:::benchmark_data_smithi179_148205_object49981:head# 2024-02-09T17:57:09.797 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:138a138a:::benchmark_data_smithi179_148205_object49587:head# 2024-02-09T17:57:09.798 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:138a7904:::benchmark_data_smithi179_148205_object71715:head# 2024-02-09T17:57:09.798 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:138b04a3:::benchmark_data_smithi179_148205_object64516:head# 2024-02-09T17:57:09.799 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:138bd40b:::benchmark_data_smithi179_148205_object46818:head# 2024-02-09T17:57:09.799 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:138d1077:::benchmark_data_smithi179_148205_object29295:head# 2024-02-09T17:57:09.800 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:138d1b85:::benchmark_data_smithi179_148205_object29510:head# 2024-02-09T17:57:09.800 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:138d61c6:::benchmark_data_smithi179_148205_object62520:head# 2024-02-09T17:57:09.800 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:138d8b63:::benchmark_data_smithi179_148205_object27474:head# 2024-02-09T17:57:09.801 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:138e82e3:::benchmark_data_smithi179_148205_object24342:head# 2024-02-09T17:57:09.801 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:139044ae:::benchmark_data_smithi179_148205_object96325:head# 2024-02-09T17:57:09.802 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13907b18:::benchmark_data_smithi179_148205_object45240:head# 2024-02-09T17:57:09.802 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1390ec0a:::benchmark_data_smithi179_148205_object17585:head# 2024-02-09T17:57:09.803 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:139104e8:::benchmark_data_smithi179_148205_object62213:head# 2024-02-09T17:57:09.803 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13920b00:::benchmark_data_smithi179_148205_object11082:head# 2024-02-09T17:57:09.804 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13924913:::benchmark_data_smithi179_148205_object47182:head# 2024-02-09T17:57:09.804 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:139289bf:::benchmark_data_smithi179_148205_object58637:head# 2024-02-09T17:57:09.804 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1392ae0a:::benchmark_data_smithi179_148205_object3894:head# 2024-02-09T17:57:09.805 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13946b74:::benchmark_data_smithi179_148205_object40164:head# 2024-02-09T17:57:09.805 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13948f15:::benchmark_data_smithi179_148205_object108448:head# 2024-02-09T17:57:09.806 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13949ee6:::benchmark_data_smithi179_148205_object109282:head# 2024-02-09T17:57:09.810 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13950de3:::benchmark_data_smithi179_148205_object89051:head# 2024-02-09T17:57:09.810 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13963ecd:::benchmark_data_smithi179_148205_object59159:head# 2024-02-09T17:57:09.810 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13966917:::benchmark_data_smithi179_148205_object74914:head# 2024-02-09T17:57:09.811 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1396956b:::benchmark_data_smithi179_148205_object60823:head# 2024-02-09T17:57:09.812 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13971fb6:::benchmark_data_smithi179_148205_object7529:head# 2024-02-09T17:57:09.812 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:139802a4:::benchmark_data_smithi179_148205_object3669:head# 2024-02-09T17:57:09.812 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1398268d:::benchmark_data_smithi179_148205_object107306:head# 2024-02-09T17:57:09.813 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1398b7bc:::benchmark_data_smithi179_148205_object50995:head# 2024-02-09T17:57:09.813 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:57:09.814 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:139907ea:::benchmark_data_smithi179_148205_object79878:head# 2024-02-09T17:57:09.814 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13996863:::benchmark_data_smithi179_148205_object38599:head# 2024-02-09T17:57:09.814 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1399aa37:::benchmark_data_smithi179_148205_object77518:head# 2024-02-09T17:57:09.815 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1399cf68:::benchmark_data_smithi179_148205_object67267:head# 2024-02-09T17:57:09.815 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:139b2e82:::benchmark_data_smithi179_148205_object54237:head# 2024-02-09T17:57:09.816 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:139c6bd0:::benchmark_data_smithi179_148205_object2794:head# 2024-02-09T17:57:09.820 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:139cf945:::benchmark_data_smithi179_148205_object90166:head# 2024-02-09T17:57:09.820 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:139d9969:::benchmark_data_smithi179_148205_object57900:head# 2024-02-09T17:57:09.820 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:139fc554:::benchmark_data_smithi179_148205_object3274:head# 2024-02-09T17:57:09.820 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13a031e4:::benchmark_data_smithi179_148205_object43206:head# 2024-02-09T17:57:09.820 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13a1c956:::benchmark_data_smithi179_148205_object38950:head# 2024-02-09T17:57:09.820 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13a28ae6:::benchmark_data_smithi179_148205_object2217:head# 2024-02-09T17:57:09.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:09.821 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:57:09.823 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13a31fa1:::benchmark_data_smithi179_148205_object39790:head# 2024-02-09T17:57:09.823 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13a392fd:::benchmark_data_smithi179_148205_object30984:head# 2024-02-09T17:57:09.823 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13a3eee8:::benchmark_data_smithi179_148205_object9712:head# 2024-02-09T17:57:09.823 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13a45556:::benchmark_data_smithi179_148205_object15149:head# 2024-02-09T17:57:09.823 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13a48b4f:::benchmark_data_smithi179_148205_object65832:head# 2024-02-09T17:57:09.823 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13a519ee:::benchmark_data_smithi179_148205_object28997:head# 2024-02-09T17:57:09.823 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13a65256:::benchmark_data_smithi179_148205_object8017:head# 2024-02-09T17:57:09.823 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13a6ad6a:::benchmark_data_smithi179_148205_object48321:head# 2024-02-09T17:57:09.824 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13a714ae:::benchmark_data_smithi179_148205_object96926:head# 2024-02-09T17:57:09.830 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13a7a707:::benchmark_data_smithi179_148205_object98637:head# 2024-02-09T17:57:09.831 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13a90e24:::benchmark_data_smithi179_148205_object89906:head# 2024-02-09T17:57:09.831 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13a9122f:::benchmark_data_smithi179_148205_object75252:head# 2024-02-09T17:57:09.831 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13aa873d:::benchmark_data_smithi179_148205_object85633:head# 2024-02-09T17:57:09.831 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13aaa0e6:::benchmark_data_smithi179_148205_object42903:head# 2024-02-09T17:57:09.831 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13aac4ae:::benchmark_data_smithi179_148205_object33189:head# 2024-02-09T17:57:09.835 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13ac2dac:::benchmark_data_smithi179_148205_object37638:head# 2024-02-09T17:57:09.836 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13ac6a3d:::benchmark_data_smithi179_148205_object98011:head# 2024-02-09T17:57:09.836 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13ac9342:::benchmark_data_smithi179_148205_object65245:head# 2024-02-09T17:57:09.837 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13acf9eb:::benchmark_data_smithi179_148205_object21752:head# 2024-02-09T17:57:09.837 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13ad76b5:::benchmark_data_smithi179_148205_object74790:head# 2024-02-09T17:57:09.838 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13adc855:::benchmark_data_smithi179_148205_object107877:head# 2024-02-09T17:57:09.838 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13ae1cc3:::benchmark_data_smithi179_148205_object12494:head# 2024-02-09T17:57:09.838 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13ae6384:::benchmark_data_smithi179_148205_object86788:head# 2024-02-09T17:57:09.839 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13ae9349:::benchmark_data_smithi179_148205_object7593:head# 2024-02-09T17:57:09.840 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13af011a:::benchmark_data_smithi179_148205_object86175:head# 2024-02-09T17:57:09.840 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13af32ed:::benchmark_data_smithi179_148205_object55928:head# 2024-02-09T17:57:09.841 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13afc84a:::benchmark_data_smithi179_148205_object17845:head# 2024-02-09T17:57:09.841 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13b05664:::benchmark_data_smithi179_148205_object109085:head# 2024-02-09T17:57:09.841 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13b0de9d:::benchmark_data_smithi179_148205_object22060:head# 2024-02-09T17:57:09.842 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13b3c943:::benchmark_data_smithi179_148205_object100215:head# 2024-02-09T17:57:09.842 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13b53ec6:::benchmark_data_smithi179_148205_object3681:head# 2024-02-09T17:57:09.842 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13b567e2:::benchmark_data_smithi179_148205_object433:head# 2024-02-09T17:57:09.843 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13b5bfda:::benchmark_data_smithi179_148205_object100554:head# 2024-02-09T17:57:09.843 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13b68b23:::benchmark_data_smithi179_148205_object50850:head# 2024-02-09T17:57:09.844 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13b6ba3d:::benchmark_data_smithi179_148205_object46204:head# 2024-02-09T17:57:09.844 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13b8306f:::benchmark_data_smithi179_148205_object67873:head# 2024-02-09T17:57:09.845 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13b9935c:::benchmark_data_smithi179_148205_object85649:head# 2024-02-09T17:57:09.845 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13baa9a8:::benchmark_data_smithi179_148205_object76775:head# 2024-02-09T17:57:09.845 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13bd148a:::benchmark_data_smithi179_148205_object81638:head# 2024-02-09T17:57:09.846 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13bd6564:::benchmark_data_smithi179_148205_object3718:head# 2024-02-09T17:57:09.846 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13be5663:::benchmark_data_smithi179_148205_object94570:head# 2024-02-09T17:57:09.847 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13be6590:::benchmark_data_smithi179_148205_object38861:head# 2024-02-09T17:57:09.847 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13bed526:::benchmark_data_smithi179_148205_object85835:head# 2024-02-09T17:57:09.848 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13bef649:::benchmark_data_smithi179_148205_object46183:head# 2024-02-09T17:57:09.848 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13bfad62:::benchmark_data_smithi179_148205_object16112:head# 2024-02-09T17:57:09.848 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13c0bc44:::benchmark_data_smithi179_148205_object67235:head# 2024-02-09T17:57:09.849 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13c14b8a:::benchmark_data_smithi179_148205_object91534:head# 2024-02-09T17:57:09.849 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13c24aca:::benchmark_data_smithi179_148205_object50989:head# 2024-02-09T17:57:09.853 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13c25827:::benchmark_data_smithi179_148205_object27845:head# 2024-02-09T17:57:09.853 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13c294b4:::benchmark_data_smithi179_148205_object8336:head# 2024-02-09T17:57:09.853 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13c2c713:::benchmark_data_smithi179_148205_object57450:head# 2024-02-09T17:57:09.853 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13c2de63:::benchmark_data_smithi179_148205_object91111:head# 2024-02-09T17:57:09.854 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13c3f979:::benchmark_data_smithi179_148205_object98708:head# 2024-02-09T17:57:09.854 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13c42786:::benchmark_data_smithi179_148205_object61373:head# 2024-02-09T17:57:09.855 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13c51491:::benchmark_data_smithi179_148205_object101572:head# 2024-02-09T17:57:09.855 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13c54847:::benchmark_data_smithi179_148205_object47364:head# 2024-02-09T17:57:09.856 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13c6a3b9:::benchmark_data_smithi179_148205_object71666:head# 2024-02-09T17:57:09.856 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13c6d19a:::benchmark_data_smithi179_148205_object83884:head# 2024-02-09T17:57:09.857 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13c71fef:::benchmark_data_smithi179_148205_object62392:head# 2024-02-09T17:57:09.857 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13c792c3:::benchmark_data_smithi179_148205_object40066:head# 2024-02-09T17:57:09.858 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13c8a350:::benchmark_data_smithi179_148205_object64337:head# 2024-02-09T17:57:09.858 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13c8af12:::benchmark_data_smithi179_148205_object105148:head# 2024-02-09T17:57:09.858 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13c99005:::benchmark_data_smithi179_148205_object19432:head# 2024-02-09T17:57:09.859 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13ca4549:::benchmark_data_smithi179_148205_object8787:head# 2024-02-09T17:57:09.860 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13cb06f9:::benchmark_data_smithi179_148205_object41787:head# 2024-02-09T17:57:09.860 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13cb26a8:::benchmark_data_smithi179_148205_object80540:head# 2024-02-09T17:57:09.861 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13cf39e0:::benchmark_data_smithi179_148205_object8170:head# 2024-02-09T17:57:09.861 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13d1715d:::benchmark_data_smithi179_148205_object27154:head# 2024-02-09T17:57:09.862 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13d3099f:::benchmark_data_smithi179_148205_object51534:head# 2024-02-09T17:57:09.862 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13d39801:::benchmark_data_smithi179_148205_object78001:head# 2024-02-09T17:57:09.862 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13d7f59e:::benchmark_data_smithi179_148205_object38792:head# 2024-02-09T17:57:09.863 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13d8ac94:::benchmark_data_smithi179_148205_object61495:head# 2024-02-09T17:57:09.865 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13d92232:::benchmark_data_smithi179_148205_object48494:head# 2024-02-09T17:57:09.866 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13daee21:::benchmark_data_smithi179_148205_object106415:head# 2024-02-09T17:57:09.866 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13db1c87:::benchmark_data_smithi179_148205_object105136:head# 2024-02-09T17:57:09.866 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13dbf28d:::benchmark_data_smithi179_148205_object42258:head# 2024-02-09T17:57:09.866 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13dd87fc:::benchmark_data_smithi179_148205_object60875:head# 2024-02-09T17:57:09.866 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13ddaadf:::benchmark_data_smithi179_148205_object30521:head# 2024-02-09T17:57:09.866 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13ddb9e8:::benchmark_data_smithi179_148205_object52326:head# 2024-02-09T17:57:09.867 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13ddbd43:::benchmark_data_smithi179_148205_object9832:head# 2024-02-09T17:57:09.868 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13dee46e:::benchmark_data_smithi179_148205_object56248:head# 2024-02-09T17:57:09.868 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13dfad8a:::benchmark_data_smithi179_148205_object90417:head# 2024-02-09T17:57:09.868 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13e09713:::benchmark_data_smithi179_148205_object49914:head# 2024-02-09T17:57:09.869 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13e15381:::benchmark_data_smithi179_148205_object65181:head# 2024-02-09T17:57:09.873 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13e15aff:::benchmark_data_smithi179_148205_object65505:head# 2024-02-09T17:57:09.874 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13e1cce1:::benchmark_data_smithi179_148205_object55075:head# 2024-02-09T17:57:09.874 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13e323a7:::benchmark_data_smithi179_148205_object57482:head# 2024-02-09T17:57:09.875 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13e36bde:::benchmark_data_smithi179_148205_object61526:head# 2024-02-09T17:57:09.875 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13e38aaa:::benchmark_data_smithi179_148205_object105702:head# 2024-02-09T17:57:09.876 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13e3e7b7:::benchmark_data_smithi179_148205_object81796:head# 2024-02-09T17:57:09.877 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13e52a73:::benchmark_data_smithi179_148205_object93933:head# 2024-02-09T17:57:09.877 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13e53a6f:::benchmark_data_smithi179_148205_object21382:head# 2024-02-09T17:57:09.878 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13e53bdd:::benchmark_data_smithi179_148205_object78385:head# 2024-02-09T17:57:09.878 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13e7511f:::benchmark_data_smithi179_148205_object96357:head# 2024-02-09T17:57:09.878 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13e7e845:::benchmark_data_smithi179_148205_object59432:head# 2024-02-09T17:57:09.879 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13e8cdf4:::benchmark_data_smithi179_148205_object63899:head# 2024-02-09T17:57:09.879 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13e8d20a:::benchmark_data_smithi179_148205_object104851:head# 2024-02-09T17:57:09.882 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13ea082f:::benchmark_data_smithi179_148205_object46640:head# 2024-02-09T17:57:09.883 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13ea4fc5:::benchmark_data_smithi179_148205_object23962:head# 2024-02-09T17:57:09.883 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13eb35e6:::benchmark_data_smithi179_148205_object10828:head# 2024-02-09T17:57:09.884 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13ec1ea9:::benchmark_data_smithi179_148205_object14494:head# 2024-02-09T17:57:09.885 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13ecae3a:::benchmark_data_smithi179_148205_object51147:head# 2024-02-09T17:57:09.886 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13ecdbe2:::benchmark_data_smithi179_148205_object39165:head# 2024-02-09T17:57:09.886 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13edaaee:::benchmark_data_smithi179_148205_object2885:head# 2024-02-09T17:57:09.887 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13ee4eee:::benchmark_data_smithi179_148205_object11367:head# 2024-02-09T17:57:09.887 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13eeea75:::benchmark_data_smithi179_148205_object7990:head# 2024-02-09T17:57:09.888 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13ef2c6c:::benchmark_data_smithi179_148205_object35035:head# 2024-02-09T17:57:09.888 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13ef8cb4:::benchmark_data_smithi179_148205_object15698:head# 2024-02-09T17:57:09.888 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13ef972c:::benchmark_data_smithi179_148205_object52688:head# 2024-02-09T17:57:09.890 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13ef9c12:::benchmark_data_smithi179_148205_object55178:head# 2024-02-09T17:57:09.890 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13efb14d:::benchmark_data_smithi179_148205_object58302:head# 2024-02-09T17:57:09.891 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13efdda3:::benchmark_data_smithi179_148205_object59018:head# 2024-02-09T17:57:09.891 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13efddf2:::benchmark_data_smithi179_148205_object7525:head# 2024-02-09T17:57:09.891 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13f05648:::benchmark_data_smithi179_148205_object39244:head# 2024-02-09T17:57:09.892 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13f06a63:::benchmark_data_smithi179_148205_object55462:head# 2024-02-09T17:57:09.892 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13f1b5d7:::benchmark_data_smithi179_148205_object3968:head# 2024-02-09T17:57:09.893 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13f1d6dc:::benchmark_data_smithi179_148205_object10656:head# 2024-02-09T17:57:09.893 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13f29c1b:::benchmark_data_smithi179_148205_object23104:head# 2024-02-09T17:57:09.898 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13f2ec85:::benchmark_data_smithi179_148205_object30815:head# 2024-02-09T17:57:09.898 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13f39606:::benchmark_data_smithi179_148205_object38322:head# 2024-02-09T17:57:09.898 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13f3adde:::benchmark_data_smithi179_148205_object84871:head# 2024-02-09T17:57:09.898 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13f5ee86:::benchmark_data_smithi179_148205_object71143:head# 2024-02-09T17:57:09.898 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13f79bd4:::benchmark_data_smithi179_148205_object37153:head# 2024-02-09T17:57:09.898 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13f7abe1:::benchmark_data_smithi179_148205_object62267:head# 2024-02-09T17:57:09.898 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13f9853c:::benchmark_data_smithi179_148205_object57082:head# 2024-02-09T17:57:09.898 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13fc4dbf:::benchmark_data_smithi179_148205_object24747:head# 2024-02-09T17:57:09.899 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13fcdd53:::benchmark_data_smithi179_148205_object11641:head# 2024-02-09T17:57:09.900 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13fde6ce:::benchmark_data_smithi179_148205_object44108:head# 2024-02-09T17:57:09.900 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:13fffc62:::benchmark_data_smithi179_148205_object88623:head# 2024-02-09T17:57:09.900 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14009b5f:::benchmark_data_smithi179_148205_object100437:head# 2024-02-09T17:57:09.900 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1400c361:::benchmark_data_smithi179_148205_object47139:head# 2024-02-09T17:57:09.901 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1402bfd2:::benchmark_data_smithi179_148205_object30768:head# 2024-02-09T17:57:09.901 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1402db17:::benchmark_data_smithi179_148205_object5881:head# 2024-02-09T17:57:09.901 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1402f7b1:::benchmark_data_smithi179_148205_object75088:head# 2024-02-09T17:57:09.902 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14031f3a:::benchmark_data_smithi179_148205_object83268:head# 2024-02-09T17:57:09.902 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1403e545:::benchmark_data_smithi179_148205_object49169:head# 2024-02-09T17:57:09.903 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14075e4a:::benchmark_data_smithi179_148205_object86620:head# 2024-02-09T17:57:09.903 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:140975fb:::benchmark_data_smithi179_148205_object52827:head# 2024-02-09T17:57:09.903 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:140addbd:::benchmark_data_smithi179_148205_object77219:head# 2024-02-09T17:57:09.904 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:140bc90f:::benchmark_data_smithi179_148205_object133:head# 2024-02-09T17:57:09.904 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1410113f:::benchmark_data_smithi179_148205_object90964:head# 2024-02-09T17:57:09.905 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14102580:::benchmark_data_smithi179_148205_object13859:head# 2024-02-09T17:57:09.906 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14118702:::benchmark_data_smithi179_148205_object101787:head# 2024-02-09T17:57:09.909 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1411adda:::benchmark_data_smithi179_148205_object59485:head# 2024-02-09T17:57:09.909 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1411e9a6:::benchmark_data_smithi179_148205_object92121:head# 2024-02-09T17:57:09.910 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1412c53c:::benchmark_data_smithi179_148205_object57512:head# 2024-02-09T17:57:09.910 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1413dec0:::benchmark_data_smithi179_148205_object3821:head# 2024-02-09T17:57:09.916 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14149f4d:::benchmark_data_smithi179_148205_object102871:head# 2024-02-09T17:57:09.916 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14156eeb:::benchmark_data_smithi179_148205_object81840:head# 2024-02-09T17:57:09.918 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14158e94:::benchmark_data_smithi179_148205_object37142:head# 2024-02-09T17:57:09.919 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1415e7f2:::benchmark_data_smithi179_148205_object75701:head# 2024-02-09T17:57:09.919 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:141661dc:::benchmark_data_smithi179_148205_object47175:head# 2024-02-09T17:57:09.920 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14167abc:::benchmark_data_smithi179_148205_object32002:head# 2024-02-09T17:57:09.920 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14171fe4:::benchmark_data_smithi179_148205_object108992:head# 2024-02-09T17:57:09.921 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1418368e:::benchmark_data_smithi179_148205_object3849:head# 2024-02-09T17:57:09.921 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1419281d:::benchmark_data_smithi179_148205_object74985:head# 2024-02-09T17:57:09.921 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:141a0d0d:::benchmark_data_smithi179_148205_object89094:head# 2024-02-09T17:57:09.922 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:57:09.923 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:141a48c8:::benchmark_data_smithi179_148205_object30723:head# 2024-02-09T17:57:09.923 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:141a863c:::benchmark_data_smithi179_148205_object36957:head# 2024-02-09T17:57:09.926 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:141ac5cb:::benchmark_data_smithi179_148205_object98371:head# 2024-02-09T17:57:09.927 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:141ae3a2:::benchmark_data_smithi179_148205_object73088:head# 2024-02-09T17:57:09.927 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:141b4ff4:::benchmark_data_smithi179_148205_object99566:head# 2024-02-09T17:57:09.927 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:141c5c68:::benchmark_data_smithi179_148205_object39115:head# 2024-02-09T17:57:09.927 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:141ceaa5:::benchmark_data_smithi179_148205_object101180:head# 2024-02-09T17:57:09.927 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:141d1a27:::benchmark_data_smithi179_148205_object52505:head# 2024-02-09T17:57:09.927 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:141dc613:::benchmark_data_smithi179_148205_object104566:head# 2024-02-09T17:57:09.927 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:141e63d1:::benchmark_data_smithi179_148205_object57716:head# 2024-02-09T17:57:09.927 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:141e9703:::benchmark_data_smithi179_148205_object55913:head# 2024-02-09T17:57:09.928 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:141efbd6:::benchmark_data_smithi179_148205_object108052:head# 2024-02-09T17:57:09.928 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:142020e9:::benchmark_data_smithi179_148205_object25708:head# 2024-02-09T17:57:09.929 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:142244ae:::benchmark_data_smithi179_148205_object1135:head# 2024-02-09T17:57:09.929 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:142245fc:::benchmark_data_smithi179_148205_object105019:head# 2024-02-09T17:57:09.929 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14224e32:::benchmark_data_smithi179_148205_object36521:head# 2024-02-09T17:57:09.930 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14227893:::benchmark_data_smithi179_148205_object36989:head# 2024-02-09T17:57:09.930 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1422cdf0:::benchmark_data_smithi179_148205_object35303:head# 2024-02-09T17:57:09.931 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14244684:::benchmark_data_smithi179_148205_object49284:head# 2024-02-09T17:57:09.931 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14247d51:::benchmark_data_smithi179_148205_object87720:head# 2024-02-09T17:57:09.932 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1424df7d:::benchmark_data_smithi179_148205_object54943:head# 2024-02-09T17:57:09.932 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:142510c1:::benchmark_data_smithi179_148205_object32723:head# 2024-02-09T17:57:09.933 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14251f60:::benchmark_data_smithi179_148205_object99025:head# 2024-02-09T17:57:09.933 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1425919a:::benchmark_data_smithi179_148205_object62474:head# 2024-02-09T17:57:09.934 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1425a130:::benchmark_data_smithi179_148205_object61823:head# 2024-02-09T17:57:09.934 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1425f912:::benchmark_data_smithi179_148205_object27947:head# 2024-02-09T17:57:09.935 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1427c3ab:::benchmark_data_smithi179_148205_object95986:head# 2024-02-09T17:57:09.935 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:142831c5:::benchmark_data_smithi179_148205_object31604:head# 2024-02-09T17:57:09.935 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:142a335c:::benchmark_data_smithi179_148205_object83954:head# 2024-02-09T17:57:09.936 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:142a6614:::benchmark_data_smithi179_148205_object24875:head# 2024-02-09T17:57:09.936 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:142a72bf:::benchmark_data_smithi179_148205_object45154:head# 2024-02-09T17:57:09.937 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:142ae9c5:::benchmark_data_smithi179_148205_object40888:head# 2024-02-09T17:57:09.937 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:142bb66c:::benchmark_data_smithi179_148205_object42430:head# 2024-02-09T17:57:09.941 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:142bd7de:::benchmark_data_smithi179_148205_object29887:head# 2024-02-09T17:57:09.941 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:142c1b66:::benchmark_data_smithi179_148205_object97024:head# 2024-02-09T17:57:09.942 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:142c832f:::benchmark_data_smithi179_148205_object450:head# 2024-02-09T17:57:09.942 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:142cc259:::benchmark_data_smithi179_148205_object55573:head# 2024-02-09T17:57:09.943 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:142d1388:::benchmark_data_smithi179_148205_object101593:head# 2024-02-09T17:57:09.943 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:142d2081:::benchmark_data_smithi179_148205_object69247:head# 2024-02-09T17:57:09.943 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:142d40f7:::benchmark_data_smithi179_148205_object24188:head# 2024-02-09T17:57:09.944 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:142d4f70:::benchmark_data_smithi179_148205_object42466:head# 2024-02-09T17:57:09.944 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:142dbd23:::benchmark_data_smithi179_148205_object87530:head# 2024-02-09T17:57:09.945 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:142e2383:::benchmark_data_smithi179_148205_object94937:head# 2024-02-09T17:57:09.945 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:142eaef6:::benchmark_data_smithi179_148205_object16521:head# 2024-02-09T17:57:09.946 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:142fb037:::benchmark_data_smithi179_148205_object85994:head# 2024-02-09T17:57:09.946 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14304b05:::benchmark_data_smithi179_148205_object16277:head# 2024-02-09T17:57:09.947 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14310fb9:::benchmark_data_smithi179_148205_object37632:head# 2024-02-09T17:57:09.947 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14316484:::benchmark_data_smithi179_148205_object36873:head# 2024-02-09T17:57:09.947 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14317ca6:::benchmark_data_smithi179_148205_object17837:head# 2024-02-09T17:57:09.949 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1431804d:::benchmark_data_smithi179_148205_object41793:head# 2024-02-09T17:57:09.949 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:143253f3:::benchmark_data_smithi179_148205_object44490:head# 2024-02-09T17:57:09.950 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:143267dd:::benchmark_data_smithi179_148205_object69590:head# 2024-02-09T17:57:09.950 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1432e33d:::benchmark_data_smithi179_148205_object29337:head# 2024-02-09T17:57:09.950 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1432f504:::benchmark_data_smithi179_148205_object14392:head# 2024-02-09T17:57:09.953 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14347362:::benchmark_data_smithi179_148205_object49492:head# 2024-02-09T17:57:09.953 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1434e2b9:::benchmark_data_smithi179_148205_object65199:head# 2024-02-09T17:57:09.953 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14373820:::benchmark_data_smithi179_148205_object74549:head# 2024-02-09T17:57:09.953 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1437d2c9:::benchmark_data_smithi179_148205_object95968:head# 2024-02-09T17:57:09.953 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1439a840:::benchmark_data_smithi179_148205_object33516:head# 2024-02-09T17:57:09.954 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:143a031c:::benchmark_data_smithi179_148205_object101750:head# 2024-02-09T17:57:09.954 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:143a591f:::benchmark_data_smithi179_148205_object692:head# 2024-02-09T17:57:09.954 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:143a7d17:::benchmark_data_smithi179_148205_object37019:head# 2024-02-09T17:57:09.954 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:143aed33:::benchmark_data_smithi179_148205_object43915:head# 2024-02-09T17:57:09.955 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:143afba0:::benchmark_data_smithi179_148205_object84238:head# 2024-02-09T17:57:09.955 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:143b237c:::benchmark_data_smithi179_148205_object11042:head# 2024-02-09T17:57:09.955 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:143b317f:::benchmark_data_smithi179_148205_object99343:head# 2024-02-09T17:57:09.956 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:143c5c11:::benchmark_data_smithi179_148205_object43387:head# 2024-02-09T17:57:09.956 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:143caa15:::benchmark_data_smithi179_148205_object8833:head# 2024-02-09T17:57:09.957 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:143ccedf:::benchmark_data_smithi179_148205_object31554:head# 2024-02-09T17:57:09.957 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:143d69d5:::benchmark_data_smithi179_148205_object45428:head# 2024-02-09T17:57:09.957 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:143e145a:::benchmark_data_smithi179_148205_object7428:head# 2024-02-09T17:57:09.958 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:143f5ab7:::benchmark_data_smithi179_148205_object46374:head# 2024-02-09T17:57:09.958 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14409ef6:::benchmark_data_smithi179_148205_object48959:head# 2024-02-09T17:57:09.963 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14417cd1:::benchmark_data_smithi179_148205_object88719:head# 2024-02-09T17:57:09.963 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:09.963 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1441d701:::benchmark_data_smithi179_148205_object6943:head# 2024-02-09T17:57:09.963 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14424d4e:::benchmark_data_smithi179_148205_object79447:head# 2024-02-09T17:57:09.964 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14426d9d:::benchmark_data_smithi179_148205_object101254:head# 2024-02-09T17:57:09.964 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14432f1d:::benchmark_data_smithi179_148205_object53221:head# 2024-02-09T17:57:09.964 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14436268:::benchmark_data_smithi179_148205_object86278:head# 2024-02-09T17:57:09.965 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:144370c7:::benchmark_data_smithi179_148205_object31619:head# 2024-02-09T17:57:09.965 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:144377e9:::benchmark_data_smithi179_148205_object89542:head# 2024-02-09T17:57:09.966 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:144517c7:::benchmark_data_smithi179_148205_object72695:head# 2024-02-09T17:57:09.966 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1445384f:::benchmark_data_smithi179_148205_object103418:head# 2024-02-09T17:57:09.967 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1445ca1a:::benchmark_data_smithi179_148205_object25869:head# 2024-02-09T17:57:09.967 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1445ea37:::benchmark_data_smithi179_148205_object63318:head# 2024-02-09T17:57:09.968 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:144667f8:::benchmark_data_smithi179_148205_object82792:head# 2024-02-09T17:57:09.968 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:144687d3:::benchmark_data_smithi179_148205_object107685:head# 2024-02-09T17:57:09.969 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1446a322:::benchmark_data_smithi179_148205_object101953:head# 2024-02-09T17:57:09.969 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1448a2f5:::benchmark_data_smithi179_148205_object43608:head# 2024-02-09T17:57:09.970 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14498f1e:::benchmark_data_smithi179_148205_object81999:head# 2024-02-09T17:57:09.970 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:144a3d71:::benchmark_data_smithi179_148205_object96865:head# 2024-02-09T17:57:09.971 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:144b1371:::benchmark_data_smithi179_148205_object18667:head# 2024-02-09T17:57:09.971 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:144b9cda:::benchmark_data_smithi179_148205_object109528:head# 2024-02-09T17:57:09.973 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:144bc959:::benchmark_data_smithi179_148205_object46470:head# 2024-02-09T17:57:09.974 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:144c6376:::benchmark_data_smithi179_148205_object13791:head# 2024-02-09T17:57:09.975 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:144cad77:::benchmark_data_smithi179_148205_object72910:head# 2024-02-09T17:57:09.975 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:144cc4e7:::benchmark_data_smithi179_148205_object107672:head# 2024-02-09T17:57:09.975 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:144d12b2:::benchmark_data_smithi179_148205_object23134:head# 2024-02-09T17:57:09.976 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:144d66b2:::benchmark_data_smithi179_148205_object77449:head# 2024-02-09T17:57:09.976 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:144e2139:::benchmark_data_smithi179_148205_object43862:head# 2024-02-09T17:57:09.976 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:144f28ed:::benchmark_data_smithi179_148205_object74201:head# 2024-02-09T17:57:09.977 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:144f7801:::benchmark_data_smithi179_148205_object2862:head# 2024-02-09T17:57:09.977 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1450bfc2:::benchmark_data_smithi179_148205_object8052:head# 2024-02-09T17:57:09.977 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14514b0c:::benchmark_data_smithi179_148205_object39131:head# 2024-02-09T17:57:09.978 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:145180e6:::benchmark_data_smithi179_148205_object19266:head# 2024-02-09T17:57:09.978 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14518f43:::benchmark_data_smithi179_148205_object62852:head# 2024-02-09T17:57:09.979 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1452307d:::benchmark_data_smithi179_148205_object92311:head# 2024-02-09T17:57:09.979 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1452371d:::benchmark_data_smithi179_148205_object64370:head# 2024-02-09T17:57:09.979 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:145278aa:::benchmark_data_smithi179_148205_object30166:head# 2024-02-09T17:57:09.980 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:145361b1:::benchmark_data_smithi179_148205_object75519:head# 2024-02-09T17:57:09.983 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14543d90:::benchmark_data_smithi179_148205_object60791:head# 2024-02-09T17:57:09.984 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:145441d0:::benchmark_data_smithi179_148205_object64287:head# 2024-02-09T17:57:09.984 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14548175:::benchmark_data_smithi179_148205_object9421:head# 2024-02-09T17:57:09.984 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14554ab0:::benchmark_data_smithi179_148205_object96524:head# 2024-02-09T17:57:09.984 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14558cc8:::benchmark_data_smithi179_148205_object3271:head# 2024-02-09T17:57:09.984 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1456ff9e:::benchmark_data_smithi179_148205_object51067:head# 2024-02-09T17:57:09.984 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1457535a:::benchmark_data_smithi179_148205_object81668:head# 2024-02-09T17:57:09.987 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1457cde3:::benchmark_data_smithi179_148205_object56166:head# 2024-02-09T17:57:09.992 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14587a4b:::benchmark_data_smithi179_148205_object76438:head# 2024-02-09T17:57:09.992 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1458d603:::benchmark_data_smithi179_148205_object57239:head# 2024-02-09T17:57:09.993 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1459071a:::benchmark_data_smithi179_148205_object7445:head# 2024-02-09T17:57:09.993 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14591aab:::benchmark_data_smithi179_148205_object23762:head# 2024-02-09T17:57:09.994 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1459a76d:::benchmark_data_smithi179_148205_object21896:head# 2024-02-09T17:57:09.994 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:145a6e63:::benchmark_data_smithi179_148205_object32543:head# 2024-02-09T17:57:09.995 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:145aa347:::benchmark_data_smithi179_148205_object83981:head# 2024-02-09T17:57:09.995 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:145b1cfb:::benchmark_data_smithi179_148205_object38272:head# 2024-02-09T17:57:09.995 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:145d6cd2:::benchmark_data_smithi179_148205_object23838:head# 2024-02-09T17:57:09.995 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:145d799d:::benchmark_data_smithi179_148205_object63752:head# 2024-02-09T17:57:09.996 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:145da65d:::benchmark_data_smithi179_148205_object41326:head# 2024-02-09T17:57:09.996 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:145de6d0:::benchmark_data_smithi179_148205_object19991:head# 2024-02-09T17:57:09.997 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:145e628e:::benchmark_data_smithi179_148205_object23995:head# 2024-02-09T17:57:09.997 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:145ef8aa:::benchmark_data_smithi179_148205_object77262:head# 2024-02-09T17:57:09.998 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:145f8455:::benchmark_data_smithi179_148205_object45680:head# 2024-02-09T17:57:09.998 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:146065f9:::benchmark_data_smithi179_148205_object46406:head# 2024-02-09T17:57:09.999 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1460a90e:::benchmark_data_smithi179_148205_object23869:head# 2024-02-09T17:57:09.999 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1460bac6:::benchmark_data_smithi179_148205_object71586:head# 2024-02-09T17:57:09.999 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:146200bb:::benchmark_data_smithi179_148205_object10888:head# 2024-02-09T17:57:10.000 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14626056:::benchmark_data_smithi179_148205_object51248:head# 2024-02-09T17:57:10.001 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14629673:::benchmark_data_smithi179_148205_object80073:head# 2024-02-09T17:57:10.001 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1462d40a:::benchmark_data_smithi179_148205_object57145:head# 2024-02-09T17:57:10.001 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1462fbc4:::benchmark_data_smithi179_148205_object101967:head# 2024-02-09T17:57:10.002 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1463ef96:::benchmark_data_smithi179_148205_object31583:head# 2024-02-09T17:57:10.002 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14642362:::benchmark_data_smithi179_148205_object14739:head# 2024-02-09T17:57:10.002 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1464cd10:::benchmark_data_smithi179_148205_object11674:head# 2024-02-09T17:57:10.003 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1464de42:::benchmark_data_smithi179_148205_object12371:head# 2024-02-09T17:57:10.003 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14651c4c:::benchmark_data_smithi179_148205_object36319:head# 2024-02-09T17:57:10.004 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14654b70:::benchmark_data_smithi179_148205_object99416:head# 2024-02-09T17:57:10.008 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14656255:::benchmark_data_smithi179_148205_object43044:head# 2024-02-09T17:57:10.009 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14658239:::benchmark_data_smithi179_148205_object90434:head# 2024-02-09T17:57:10.009 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14683d9d:::benchmark_data_smithi179_148205_object68727:head# 2024-02-09T17:57:10.010 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1468912c:::benchmark_data_smithi179_148205_object41705:head# 2024-02-09T17:57:10.010 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1468b24a:::benchmark_data_smithi179_148205_object51213:head# 2024-02-09T17:57:10.010 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1468c1c0:::benchmark_data_smithi179_148205_object2633:head# 2024-02-09T17:57:10.011 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1469fefd:::benchmark_data_smithi179_148205_object76065:head# 2024-02-09T17:57:10.011 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:146b16c2:::benchmark_data_smithi179_148205_object67599:head# 2024-02-09T17:57:10.011 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:146c35f5:::benchmark_data_smithi179_148205_object14994:head# 2024-02-09T17:57:10.012 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:146d95b8:::benchmark_data_smithi179_148205_object20040:head# 2024-02-09T17:57:10.012 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:146df7bd:::benchmark_data_smithi179_148205_object6071:head# 2024-02-09T17:57:10.013 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:146ec2ba:::benchmark_data_smithi179_148205_object49966:head# 2024-02-09T17:57:10.013 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:146ee4bd:::benchmark_data_smithi179_148205_object41258:head# 2024-02-09T17:57:10.014 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:146f5460:::benchmark_data_smithi179_148205_object36155:head# 2024-02-09T17:57:10.014 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:146f5b55:::benchmark_data_smithi179_148205_object1177:head# 2024-02-09T17:57:10.014 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:146f63c2:::benchmark_data_smithi179_148205_object36354:head# 2024-02-09T17:57:10.015 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:146f7a93:::benchmark_data_smithi179_148205_object96414:head# 2024-02-09T17:57:10.015 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14718e9c:::benchmark_data_smithi179_148205_object51492:head# 2024-02-09T17:57:10.016 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1472d7dc:::benchmark_data_smithi179_148205_object90457:head# 2024-02-09T17:57:10.017 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1472da04:::benchmark_data_smithi179_148205_object75328:head# 2024-02-09T17:57:10.017 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1472f477:::benchmark_data_smithi179_148205_object77714:head# 2024-02-09T17:57:10.018 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1472f4c9:::benchmark_data_smithi179_148205_object12267:head# 2024-02-09T17:57:10.018 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14737aa5:::benchmark_data_smithi179_148205_object21784:head# 2024-02-09T17:57:10.018 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14746b49:::benchmark_data_smithi179_148205_object93717:head# 2024-02-09T17:57:10.019 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1474cf8f:::benchmark_data_smithi179_148205_object75541:head# 2024-02-09T17:57:10.019 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1474ee2a:::benchmark_data_smithi179_148205_object6654:head# 2024-02-09T17:57:10.020 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14772b4b:::benchmark_data_smithi179_148205_object86176:head# 2024-02-09T17:57:10.020 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14772bfa:::benchmark_data_smithi179_148205_object89956:head# 2024-02-09T17:57:10.021 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1477520c:::benchmark_data_smithi179_148205_object7926:head# 2024-02-09T17:57:10.026 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1477648d:::benchmark_data_smithi179_148205_object25263:head# 2024-02-09T17:57:10.026 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14778662:::benchmark_data_smithi179_148205_object69983:head# 2024-02-09T17:57:10.026 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1478a9d2:::benchmark_data_smithi179_148205_object24353:head# 2024-02-09T17:57:10.026 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1478eebf:::benchmark_data_smithi179_148205_object47880:head# 2024-02-09T17:57:10.026 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14790a22:::benchmark_data_smithi179_148205_object84903:head# 2024-02-09T17:57:10.027 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:147a67f8:::benchmark_data_smithi179_148205_object26:head# 2024-02-09T17:57:10.027 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:147b0c0c:::benchmark_data_smithi179_148205_object21936:head# 2024-02-09T17:57:10.029 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:147cfecd:::benchmark_data_smithi179_148205_object69197:head# 2024-02-09T17:57:10.029 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:147e90f9:::benchmark_data_smithi179_148205_object16156:head# 2024-02-09T17:57:10.029 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:147e99e9:::benchmark_data_smithi179_148205_object80067:head# 2024-02-09T17:57:10.029 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:147efd61:::benchmark_data_smithi179_148205_object58756:head# 2024-02-09T17:57:10.030 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1480470d:::benchmark_data_smithi179_148205_object9244:head# 2024-02-09T17:57:10.031 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14809c9a:::benchmark_data_smithi179_148205_object107805:head# 2024-02-09T17:57:10.032 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14816689:::benchmark_data_smithi179_148205_object7999:head# 2024-02-09T17:57:10.032 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1482a1df:::benchmark_data_smithi179_148205_object91019:head# 2024-02-09T17:57:10.033 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14839646:::benchmark_data_smithi179_148205_object12447:head# 2024-02-09T17:57:10.033 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1483b2ae:::benchmark_data_smithi179_148205_object83705:head# 2024-02-09T17:57:10.034 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1483b7b8:::benchmark_data_smithi179_148205_object8965:head# 2024-02-09T17:57:10.034 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:148415f6:::benchmark_data_smithi179_148205_object7149:head# 2024-02-09T17:57:10.035 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1484236b:::benchmark_data_smithi179_148205_object63895:head# 2024-02-09T17:57:10.035 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14860e10:::benchmark_data_smithi179_148205_object83120:head# 2024-02-09T17:57:10.036 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14864a74:::benchmark_data_smithi179_148205_object88366:head# 2024-02-09T17:57:10.036 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1487411b:::benchmark_data_smithi179_148205_object106556:head# 2024-02-09T17:57:10.036 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14896241:::benchmark_data_smithi179_148205_object33747:head# 2024-02-09T17:57:10.037 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1489c072:::benchmark_data_smithi179_148205_object13869:head# 2024-02-09T17:57:10.037 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:148a1ff8:::benchmark_data_smithi179_148205_object90467:head# 2024-02-09T17:57:10.037 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:148a400f:::benchmark_data_smithi179_148205_object88621:head# 2024-02-09T17:57:10.038 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:148a6c77:::benchmark_data_smithi179_148205_object102344:head# 2024-02-09T17:57:10.038 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:148ada81:::benchmark_data_smithi179_148205_object16499:head# 2024-02-09T17:57:10.039 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:148b4e5e:::benchmark_data_smithi179_148205_object95232:head# 2024-02-09T17:57:10.039 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:148c02c3:::benchmark_data_smithi179_148205_object29270:head# 2024-02-09T17:57:10.040 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:148cc532:::benchmark_data_smithi179_148205_object23755:head# 2024-02-09T17:57:10.040 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:148d488a:::benchmark_data_smithi179_148205_object33580:head# 2024-02-09T17:57:10.040 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:148e5136:::benchmark_data_smithi179_148205_object123:head# 2024-02-09T17:57:10.041 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:148ee7b1:::benchmark_data_smithi179_148205_object42069:head# 2024-02-09T17:57:10.041 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:148f75d8:::benchmark_data_smithi179_148205_object44551:head# 2024-02-09T17:57:10.042 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:148f9aed:::benchmark_data_smithi179_148205_object25125:head# 2024-02-09T17:57:10.042 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:149055e7:::benchmark_data_smithi179_148205_object19267:head# 2024-02-09T17:57:10.042 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1490ad5a:::benchmark_data_smithi179_148205_object9181:head# 2024-02-09T17:57:10.043 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1492ff83:::benchmark_data_smithi179_148205_object11167:head# 2024-02-09T17:57:10.043 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1493f970:::benchmark_data_smithi179_148205_object91374:head# 2024-02-09T17:57:10.044 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14941dd4:::benchmark_data_smithi179_148205_object1847:head# 2024-02-09T17:57:10.044 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14944091:::benchmark_data_smithi179_148205_object21812:head# 2024-02-09T17:57:10.045 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1494b1d6:::benchmark_data_smithi179_148205_object97690:head# 2024-02-09T17:57:10.045 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1494b4b2:::benchmark_data_smithi179_148205_object8186:head# 2024-02-09T17:57:10.046 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1494e550:::benchmark_data_smithi179_148205_object45892:head# 2024-02-09T17:57:10.046 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1494e90d:::benchmark_data_smithi179_148205_object100229:head# 2024-02-09T17:57:10.046 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:149550c1:::benchmark_data_smithi179_148205_object471:head# 2024-02-09T17:57:10.047 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1495ffc1:::benchmark_data_smithi179_148205_object9464:head# 2024-02-09T17:57:10.047 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:149838f4:::benchmark_data_smithi179_148205_object106312:head# 2024-02-09T17:57:10.048 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1498ba0b:::benchmark_data_smithi179_148205_object53842:head# 2024-02-09T17:57:10.048 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:149961a0:::benchmark_data_smithi179_148205_object23423:head# 2024-02-09T17:57:10.048 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1499add9:::benchmark_data_smithi179_148205_object78759:head# 2024-02-09T17:57:10.049 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:149bea83:::benchmark_data_smithi179_148205_object49635:head# 2024-02-09T17:57:10.049 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:149c3eb9:::benchmark_data_smithi179_148205_object61946:head# 2024-02-09T17:57:10.050 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:149c4cbb:::benchmark_data_smithi179_148205_object54398:head# 2024-02-09T17:57:10.050 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:149c5bb0:::benchmark_data_smithi179_148205_object41227:head# 2024-02-09T17:57:10.050 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:149de81c:::benchmark_data_smithi179_148205_object41857:head# 2024-02-09T17:57:10.051 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:149dfe47:::benchmark_data_smithi179_148205_object41195:head# 2024-02-09T17:57:10.051 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:149e0fac:::benchmark_data_smithi179_148205_object32086:head# 2024-02-09T17:57:10.052 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:149f0a0a:::benchmark_data_smithi179_148205_object15498:head# 2024-02-09T17:57:10.052 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:149f9ab2:::benchmark_data_smithi179_148205_object67550:head# 2024-02-09T17:57:10.053 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:149fab94:::benchmark_data_smithi179_148205_object5828:head# 2024-02-09T17:57:10.053 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:149fd124:::benchmark_data_smithi179_148205_object57733:head# 2024-02-09T17:57:10.054 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14a0ecb8:::benchmark_data_smithi179_148205_object92458:head# 2024-02-09T17:57:10.054 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14a19489:::benchmark_data_smithi179_148205_object9014:head# 2024-02-09T17:57:10.055 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14a2d03e:::benchmark_data_smithi179_148205_object31955:head# 2024-02-09T17:57:10.055 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14a2ee3a:::benchmark_data_smithi179_148205_object25484:head# 2024-02-09T17:57:10.055 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14a48174:::benchmark_data_smithi179_148205_object37579:head# 2024-02-09T17:57:10.056 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14a4ec1b:::benchmark_data_smithi179_148205_object18200:head# 2024-02-09T17:57:10.056 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14a50fe3:::benchmark_data_smithi179_148205_object104167:head# 2024-02-09T17:57:10.058 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14a54a53:::benchmark_data_smithi179_148205_object12615:head# 2024-02-09T17:57:10.058 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14a7083c:::benchmark_data_smithi179_148205_object34435:head# 2024-02-09T17:57:10.058 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14a82f5e:::benchmark_data_smithi179_148205_object21126:head# 2024-02-09T17:57:10.058 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14a8771c:::benchmark_data_smithi179_148205_object100491:head# 2024-02-09T17:57:10.059 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14a9453e:::benchmark_data_smithi179_148205_object20779:head# 2024-02-09T17:57:10.059 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14a945c5:::benchmark_data_smithi179_148205_object27127:head# 2024-02-09T17:57:10.059 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14ab2440:::benchmark_data_smithi179_148205_object15985:head# 2024-02-09T17:57:10.061 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14ac3b1e:::benchmark_data_smithi179_148205_object75973:head# 2024-02-09T17:57:10.061 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14ad4661:::benchmark_data_smithi179_148205_object17094:head# 2024-02-09T17:57:10.062 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14ad5985:::benchmark_data_smithi179_148205_object37685:head# 2024-02-09T17:57:10.062 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14ad8586:::benchmark_data_smithi179_148205_object7584:head# 2024-02-09T17:57:10.062 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14af0d52:::benchmark_data_smithi179_148205_object2215:head# 2024-02-09T17:57:10.063 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14afc3a7:::benchmark_data_smithi179_148205_object77485:head# 2024-02-09T17:57:10.063 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14b06048:::benchmark_data_smithi179_148205_object25858:head# 2024-02-09T17:57:10.064 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14b11fac:::benchmark_data_smithi179_148205_object61816:head# 2024-02-09T17:57:10.064 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14b150e2:::benchmark_data_smithi179_148205_object31632:head# 2024-02-09T17:57:10.065 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14b278d9:::benchmark_data_smithi179_148205_object77479:head# 2024-02-09T17:57:10.065 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14b53dd6:::benchmark_data_smithi179_148205_object105528:head# 2024-02-09T17:57:10.066 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14b5adf3:::benchmark_data_smithi179_148205_object102916:head# 2024-02-09T17:57:10.066 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14b5c049:::benchmark_data_smithi179_148205_object14799:head# 2024-02-09T17:57:10.068 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14b616b0:::benchmark_data_smithi179_148205_object99607:head# 2024-02-09T17:57:10.073 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14b6e1cc:::benchmark_data_smithi179_148205_object3812:head# 2024-02-09T17:57:10.073 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14b9cab3:::benchmark_data_smithi179_148205_object3505:head# 2024-02-09T17:57:10.073 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14ba0713:::benchmark_data_smithi179_148205_object90108:head# 2024-02-09T17:57:10.075 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14bb6c1a:::benchmark_data_smithi179_148205_object16298:head# 2024-02-09T17:57:10.075 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14bc5ee5:::benchmark_data_smithi179_148205_object107490:head# 2024-02-09T17:57:10.076 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14bc6481:::benchmark_data_smithi179_148205_object67617:head# 2024-02-09T17:57:10.076 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14bcb625:::benchmark_data_smithi179_148205_object94175:head# 2024-02-09T17:57:10.077 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14bd8206:::benchmark_data_smithi179_148205_object34566:head# 2024-02-09T17:57:10.077 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14be2f15:::benchmark_data_smithi179_148205_object31071:head# 2024-02-09T17:57:10.078 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14be71be:::benchmark_data_smithi179_148205_object2401:head# 2024-02-09T17:57:10.078 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14bed2d2:::benchmark_data_smithi179_148205_object84302:head# 2024-02-09T17:57:10.079 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14bf50d6:::benchmark_data_smithi179_148205_object95858:head# 2024-02-09T17:57:10.079 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14c29f37:::benchmark_data_smithi179_148205_object72151:head# 2024-02-09T17:57:10.079 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14c3f241:::benchmark_data_smithi179_148205_object96033:head# 2024-02-09T17:57:10.080 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14c42988:::benchmark_data_smithi179_148205_object84650:head# 2024-02-09T17:57:10.081 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14c42eaf:::benchmark_data_smithi179_148205_object107591:head# 2024-02-09T17:57:10.081 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14c4613f:::benchmark_data_smithi179_148205_object28978:head# 2024-02-09T17:57:10.082 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14c47e14:::benchmark_data_smithi179_148205_object78267:head# 2024-02-09T17:57:10.084 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14c55efd:::benchmark_data_smithi179_148205_object16839:head# 2024-02-09T17:57:10.085 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14c578fc:::benchmark_data_smithi179_148205_object54955:head# 2024-02-09T17:57:10.085 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14c7abd4:::benchmark_data_smithi179_148205_object26772:head# 2024-02-09T17:57:10.085 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14c88795:::benchmark_data_smithi179_148205_object80255:head# 2024-02-09T17:57:10.085 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14ca0ccc:::benchmark_data_smithi179_148205_object21815:head# 2024-02-09T17:57:10.086 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14ca94f6:::benchmark_data_smithi179_148205_object9058:head# 2024-02-09T17:57:10.091 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14cb3a06:::benchmark_data_smithi179_148205_object13159:head# 2024-02-09T17:57:10.091 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14cbcec8:::benchmark_data_smithi179_148205_object14039:head# 2024-02-09T17:57:10.091 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14cd05c7:::benchmark_data_smithi179_148205_object64075:head# 2024-02-09T17:57:10.094 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14cf703a:::benchmark_data_smithi179_148205_object89017:head# 2024-02-09T17:57:10.094 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14cf974d:::benchmark_data_smithi179_148205_object71734:head# 2024-02-09T17:57:10.094 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14cfd623:::benchmark_data_smithi179_148205_object19329:head# 2024-02-09T17:57:10.094 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14d02540:::benchmark_data_smithi179_148205_object15063:head# 2024-02-09T17:57:10.094 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14d0bba0:::benchmark_data_smithi179_148205_object80316:head# 2024-02-09T17:57:10.094 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14d0cd99:::benchmark_data_smithi179_148205_object92663:head# 2024-02-09T17:57:10.095 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14d22427:::benchmark_data_smithi179_148205_object55766:head# 2024-02-09T17:57:10.095 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14d2d3d9:::benchmark_data_smithi179_148205_object97650:head# 2024-02-09T17:57:10.095 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14d2f0a2:::benchmark_data_smithi179_148205_object88866:head# 2024-02-09T17:57:10.095 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14d3ca3e:::benchmark_data_smithi179_148205_object68550:head# 2024-02-09T17:57:10.096 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14d440a3:::benchmark_data_smithi179_148205_object60696:head# 2024-02-09T17:57:10.097 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14d4aae3:::benchmark_data_smithi179_148205_object3831:head# 2024-02-09T17:57:10.097 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14d5caa8:::benchmark_data_smithi179_148205_object1729:head# 2024-02-09T17:57:10.097 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14d5efb8:::benchmark_data_smithi179_148205_object63942:head# 2024-02-09T17:57:10.098 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14d663ad:::benchmark_data_smithi179_148205_object56072:head# 2024-02-09T17:57:10.098 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14d86ab8:::benchmark_data_smithi179_148205_object3621:head# 2024-02-09T17:57:10.099 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14d8735b:::benchmark_data_smithi179_148205_object77080:head# 2024-02-09T17:57:10.099 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14d96e1a:::benchmark_data_smithi179_148205_object57460:head# 2024-02-09T17:57:10.100 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14d9b0aa:::benchmark_data_smithi179_148205_object35807:head# 2024-02-09T17:57:10.100 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14d9e79a:::benchmark_data_smithi179_148205_object76741:head# 2024-02-09T17:57:10.101 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14dab963:::benchmark_data_smithi179_148205_object7845:head# 2024-02-09T17:57:10.101 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14db1d62:::benchmark_data_smithi179_148205_object43745:head# 2024-02-09T17:57:10.102 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14db22c2:::benchmark_data_smithi179_148205_object79352:head# 2024-02-09T17:57:10.102 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14dbf25f:::benchmark_data_smithi179_148205_object77717:head# 2024-02-09T17:57:10.103 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14dd206f:::benchmark_data_smithi179_148205_object20087:head# 2024-02-09T17:57:10.103 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14dd4c2f:::benchmark_data_smithi179_148205_object78442:head# 2024-02-09T17:57:10.104 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14dd9b80:::benchmark_data_smithi179_148205_object76332:head# 2024-02-09T17:57:10.104 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14ddff01:::benchmark_data_smithi179_148205_object56893:head# 2024-02-09T17:57:10.105 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14de3708:::benchmark_data_smithi179_148205_object101233:head# 2024-02-09T17:57:10.105 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14de68f6:::benchmark_data_smithi179_148205_object37855:head# 2024-02-09T17:57:10.106 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14deb3e0:::benchmark_data_smithi179_148205_object62568:head# 2024-02-09T17:57:10.106 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14df1cac:::benchmark_data_smithi179_148205_object101036:head# 2024-02-09T17:57:10.106 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14dfa42d:::benchmark_data_smithi179_148205_object49457:head# 2024-02-09T17:57:10.107 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14dfad46:::benchmark_data_smithi179_148205_object8899:head# 2024-02-09T17:57:10.107 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14e0b7b8:::benchmark_data_smithi179_148205_object70015:head# 2024-02-09T17:57:10.108 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14e103ef:::benchmark_data_smithi179_148205_object39330:head# 2024-02-09T17:57:10.108 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14e2422d:::benchmark_data_smithi179_148205_object71278:head# 2024-02-09T17:57:10.109 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14e2e759:::benchmark_data_smithi179_148205_object2361:head# 2024-02-09T17:57:10.109 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14e32673:::benchmark_data_smithi179_148205_object66924:head# 2024-02-09T17:57:10.110 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14e3d3c2:::benchmark_data_smithi179_148205_object2740:head# 2024-02-09T17:57:10.110 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14e4396c:::benchmark_data_smithi179_148205_object20589:head# 2024-02-09T17:57:10.110 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14e4e44e:::benchmark_data_smithi179_148205_object73964:head# 2024-02-09T17:57:10.111 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14e51f46:::benchmark_data_smithi179_148205_object71576:head# 2024-02-09T17:57:10.111 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14e5417a:::benchmark_data_smithi179_148205_object101653:head# 2024-02-09T17:57:10.112 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14e697ed:::benchmark_data_smithi179_148205_object7821:head# 2024-02-09T17:57:10.112 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14e802f0:::benchmark_data_smithi179_148205_object105399:head# 2024-02-09T17:57:10.113 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14e832fb:::benchmark_data_smithi179_148205_object80836:head# 2024-02-09T17:57:10.113 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14e8b44a:::benchmark_data_smithi179_148205_object20748:head# 2024-02-09T17:57:10.113 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14ea86a7:::benchmark_data_smithi179_148205_object9961:head# 2024-02-09T17:57:10.114 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14eb87ce:::benchmark_data_smithi179_148205_object9042:head# 2024-02-09T17:57:10.114 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14ebb42d:::benchmark_data_smithi179_148205_object49898:head# 2024-02-09T17:57:10.114 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14ebbd97:::benchmark_data_smithi179_148205_object66714:head# 2024-02-09T17:57:10.115 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14ec0c9b:::benchmark_data_smithi179_148205_object85042:head# 2024-02-09T17:57:10.115 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14ed1a82:::benchmark_data_smithi179_148205_object92908:head# 2024-02-09T17:57:10.116 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14ed2ceb:::benchmark_data_smithi179_148205_object78798:head# 2024-02-09T17:57:10.116 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14ed9e5f:::benchmark_data_smithi179_148205_object68119:head# 2024-02-09T17:57:10.117 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14edfc57:::benchmark_data_smithi179_148205_object19260:head# 2024-02-09T17:57:10.117 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14ee8166:::benchmark_data_smithi179_148205_object103922:head# 2024-02-09T17:57:10.117 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14eea409:::benchmark_data_smithi179_148205_object1727:head# 2024-02-09T17:57:10.118 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14ef5f3a:::benchmark_data_smithi179_148205_object90071:head# 2024-02-09T17:57:10.118 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14efc56b:::benchmark_data_smithi179_148205_object84563:head# 2024-02-09T17:57:10.119 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14efe785:::benchmark_data_smithi179_148205_object1489:head# 2024-02-09T17:57:10.119 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14f05fc9:::benchmark_data_smithi179_148205_object31548:head# 2024-02-09T17:57:10.120 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14f09b4d:::benchmark_data_smithi179_148205_object43486:head# 2024-02-09T17:57:10.121 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14f0a182:::benchmark_data_smithi179_148205_object38436:head# 2024-02-09T17:57:10.121 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14f13c4d:::benchmark_data_smithi179_148205_object73623:head# 2024-02-09T17:57:10.121 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14f2eaed:::benchmark_data_smithi179_148205_object97563:head# 2024-02-09T17:57:10.122 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14f38fe1:::benchmark_data_smithi179_148205_object63043:head# 2024-02-09T17:57:10.123 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14f427bc:::benchmark_data_smithi179_148205_object90686:head# 2024-02-09T17:57:10.123 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14f50686:::benchmark_data_smithi179_148205_object75218:head# 2024-02-09T17:57:10.124 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14f62c99:::benchmark_data_smithi179_148205_object43468:head# 2024-02-09T17:57:10.124 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14f6a739:::benchmark_data_smithi179_148205_object105246:head# 2024-02-09T17:57:10.125 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14f6e167:::benchmark_data_smithi179_148205_object41269:head# 2024-02-09T17:57:10.125 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14f7788e:::benchmark_data_smithi179_148205_object91089:head# 2024-02-09T17:57:10.129 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14f783d7:::benchmark_data_smithi179_148205_object80061:head# 2024-02-09T17:57:10.129 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14fc4569:::benchmark_data_smithi179_148205_object74268:head# 2024-02-09T17:57:10.129 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14fd2828:::benchmark_data_smithi179_148205_object18775:head# 2024-02-09T17:57:10.130 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14fe0421:::benchmark_data_smithi179_148205_object94697:head# 2024-02-09T17:57:10.130 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14fe66b3:::benchmark_data_smithi179_148205_object29652:head# 2024-02-09T17:57:10.131 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14ff6cc5:::benchmark_data_smithi179_148205_object3151:head# 2024-02-09T17:57:10.131 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:14ffa3b6:::benchmark_data_smithi179_148205_object1327:head# 2024-02-09T17:57:10.132 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:150049e1:::benchmark_data_smithi179_148205_object81902:head# 2024-02-09T17:57:10.132 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1500708e:::benchmark_data_smithi179_148205_object45082:head# 2024-02-09T17:57:10.133 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:150196f1:::benchmark_data_smithi179_148205_object89705:head# 2024-02-09T17:57:10.133 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15046b1f:::benchmark_data_smithi179_148205_object11064:head# 2024-02-09T17:57:10.134 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1504cc65:::benchmark_data_smithi179_148205_object86735:head# 2024-02-09T17:57:10.134 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1504d977:::benchmark_data_smithi179_148205_object72513:head# 2024-02-09T17:57:10.135 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1505080b:::benchmark_data_smithi179_148205_object91535:head# 2024-02-09T17:57:10.135 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15066671:::benchmark_data_smithi179_148205_object91979:head# 2024-02-09T17:57:10.135 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:150741df:::benchmark_data_smithi179_148205_object51250:head# 2024-02-09T17:57:10.137 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1507c434:::benchmark_data_smithi179_148205_object101981:head# 2024-02-09T17:57:10.137 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:150840ab:::benchmark_data_smithi179_148205_object57215:head# 2024-02-09T17:57:10.137 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15087652:::benchmark_data_smithi179_148205_object49682:head# 2024-02-09T17:57:10.137 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15089a9c:::benchmark_data_smithi179_148205_object6409:head# 2024-02-09T17:57:10.138 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15089bb3:::benchmark_data_smithi179_148205_object103140:head# 2024-02-09T17:57:10.138 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1509d003:::benchmark_data_smithi179_148205_object62717:head# 2024-02-09T17:57:10.139 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:150a5076:::benchmark_data_smithi179_148205_object1056:head# 2024-02-09T17:57:10.140 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:150ada69:::benchmark_data_smithi179_148205_object6163:head# 2024-02-09T17:57:10.141 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:150b5640:::benchmark_data_smithi179_148205_object6121:head# 2024-02-09T17:57:10.142 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:150d5109:::benchmark_data_smithi179_148205_object85308:head# 2024-02-09T17:57:10.143 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:150d5ad8:::benchmark_data_smithi179_148205_object7033:head# 2024-02-09T17:57:10.143 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1510f759:::benchmark_data_smithi179_148205_object52145:head# 2024-02-09T17:57:10.144 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1511a10d:::benchmark_data_smithi179_148205_object90474:head# 2024-02-09T17:57:10.148 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1511b065:::benchmark_data_smithi179_148205_object93502:head# 2024-02-09T17:57:10.148 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1511d17c:::benchmark_data_smithi179_148205_object68863:head# 2024-02-09T17:57:10.148 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15126d08:::benchmark_data_smithi179_148205_object24184:head# 2024-02-09T17:57:10.149 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15137c31:::benchmark_data_smithi179_148205_object96769:head# 2024-02-09T17:57:10.149 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15139796:::benchmark_data_smithi179_148205_object106701:head# 2024-02-09T17:57:10.149 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1513c395:::benchmark_data_smithi179_148205_object102990:head# 2024-02-09T17:57:10.149 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:151460f6:::benchmark_data_smithi179_148205_object55180:head# 2024-02-09T17:57:10.149 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1514e73e:::benchmark_data_smithi179_148205_object67165:head# 2024-02-09T17:57:10.150 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15150fef:::benchmark_data_smithi179_148205_object40671:head# 2024-02-09T17:57:10.150 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1515b92d:::benchmark_data_smithi179_148205_object45607:head# 2024-02-09T17:57:10.151 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1515d3aa:::benchmark_data_smithi179_148205_object108809:head# 2024-02-09T17:57:10.151 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1516cd85:::benchmark_data_smithi179_148205_object68664:head# 2024-02-09T17:57:10.151 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1518a74e:::benchmark_data_smithi179_148205_object81648:head# 2024-02-09T17:57:10.152 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1518ae37:::benchmark_data_smithi179_148205_object39829:head# 2024-02-09T17:57:10.152 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15196b18:::benchmark_data_smithi179_148205_object85118:head# 2024-02-09T17:57:10.153 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1519bfc4:::benchmark_data_smithi179_148205_object109321:head# 2024-02-09T17:57:10.153 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:151b2a6c:::benchmark_data_smithi179_148205_object70911:head# 2024-02-09T17:57:10.157 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:151b6117:::benchmark_data_smithi179_148205_object37725:head# 2024-02-09T17:57:10.160 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:151c1851:::benchmark_data_smithi179_148205_object59592:head# 2024-02-09T17:57:10.161 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:151c8ce1:::benchmark_data_smithi179_148205_object36729:head# 2024-02-09T17:57:10.161 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:151cfb45:::benchmark_data_smithi179_148205_object2663:head# 2024-02-09T17:57:10.161 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:151da07e:::benchmark_data_smithi179_148205_object102885:head# 2024-02-09T17:57:10.161 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:151dab6c:::benchmark_data_smithi179_148205_object79957:head# 2024-02-09T17:57:10.161 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:151ef5bc:::benchmark_data_smithi179_148205_object62157:head# 2024-02-09T17:57:10.166 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:151f9292:::benchmark_data_smithi179_148205_object12119:head# 2024-02-09T17:57:10.166 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:151ff42f:::benchmark_data_smithi179_148205_object65575:head# 2024-02-09T17:57:10.169 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1520a6fd:::benchmark_data_smithi179_148205_object100623:head# 2024-02-09T17:57:10.169 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1521f8a0:::benchmark_data_smithi179_148205_object48235:head# 2024-02-09T17:57:10.170 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1521f94c:::benchmark_data_smithi179_148205_object49936:head# 2024-02-09T17:57:10.170 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1522497b:::benchmark_data_smithi179_148205_object89669:head# 2024-02-09T17:57:10.170 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1522e634:::benchmark_data_smithi179_148205_object23836:head# 2024-02-09T17:57:10.175 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1522f8ef:::benchmark_data_smithi179_148205_object6106:head# 2024-02-09T17:57:10.175 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1524bd56:::benchmark_data_smithi179_148205_object67587:head# 2024-02-09T17:57:10.175 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1524c310:::benchmark_data_smithi179_148205_object3310:head# 2024-02-09T17:57:10.175 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:152553e0:::benchmark_data_smithi179_148205_object12087:head# 2024-02-09T17:57:10.177 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15256be7:::benchmark_data_smithi179_148205_object53594:head# 2024-02-09T17:57:10.178 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15268919:::benchmark_data_smithi179_148205_object5381:head# 2024-02-09T17:57:10.178 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1526e82f:::benchmark_data_smithi179_148205_object9797:head# 2024-02-09T17:57:10.178 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1527116b:::benchmark_data_smithi179_148205_object65261:head# 2024-02-09T17:57:10.178 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15275a62:::benchmark_data_smithi179_148205_object22017:head# 2024-02-09T17:57:10.178 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15279c90:::benchmark_data_smithi179_148205_object73577:head# 2024-02-09T17:57:10.179 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1528ae67:::benchmark_data_smithi179_148205_object67530:head# 2024-02-09T17:57:10.179 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:152996e8:::benchmark_data_smithi179_148205_object68560:head# 2024-02-09T17:57:10.179 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:152a0aa1:::benchmark_data_smithi179_148205_object65542:head# 2024-02-09T17:57:10.179 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:152a3761:::benchmark_data_smithi179_148205_object25489:head# 2024-02-09T17:57:10.179 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:152a892e:::benchmark_data_smithi179_148205_object34099:head# 2024-02-09T17:57:10.180 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:152b2e96:::benchmark_data_smithi179_148205_object53512:head# 2024-02-09T17:57:10.181 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:152b6a86:::benchmark_data_smithi179_148205_object10573:head# 2024-02-09T17:57:10.181 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:152c307f:::benchmark_data_smithi179_148205_object94590:head# 2024-02-09T17:57:10.181 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:152d9ca4:::benchmark_data_smithi179_148205_object33316:head# 2024-02-09T17:57:10.182 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:152da700:::benchmark_data_smithi179_148205_object36227:head# 2024-02-09T17:57:10.182 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:152e15fc:::benchmark_data_smithi179_148205_object48226:head# 2024-02-09T17:57:10.182 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:152e506b:::benchmark_data_smithi179_148205_object80556:head# 2024-02-09T17:57:10.183 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:152fc879:::benchmark_data_smithi179_148205_object18989:head# 2024-02-09T17:57:10.183 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1530096f:::benchmark_data_smithi179_148205_object65433:head# 2024-02-09T17:57:10.183 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1530efdc:::benchmark_data_smithi179_148205_object64234:head# 2024-02-09T17:57:10.184 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1531a7fa:::benchmark_data_smithi179_148205_object27090:head# 2024-02-09T17:57:10.184 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1531cdec:::benchmark_data_smithi179_148205_object46622:head# 2024-02-09T17:57:10.185 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:153201f0:::benchmark_data_smithi179_148205_object24626:head# 2024-02-09T17:57:10.185 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15343421:::benchmark_data_smithi179_148205_object36248:head# 2024-02-09T17:57:10.190 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15359eb9:::benchmark_data_smithi179_148205_object70092:head# 2024-02-09T17:57:10.190 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15362cd9:::benchmark_data_smithi179_148205_object99261:head# 2024-02-09T17:57:10.190 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15365528:::benchmark_data_smithi179_148205_object91523:head# 2024-02-09T17:57:10.190 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1536d6c2:::benchmark_data_smithi179_148205_object25587:head# 2024-02-09T17:57:10.190 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1536d7fe:::benchmark_data_smithi179_148205_object107319:head# 2024-02-09T17:57:10.190 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15372430:::benchmark_data_smithi179_148205_object1605:head# 2024-02-09T17:57:10.190 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15379203:::benchmark_data_smithi179_148205_object22016:head# 2024-02-09T17:57:10.190 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1538614e:::benchmark_data_smithi179_148205_object47242:head# 2024-02-09T17:57:10.190 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1538b687:::benchmark_data_smithi179_148205_object28190:head# 2024-02-09T17:57:10.190 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15392d9f:::benchmark_data_smithi179_148205_object116:head# 2024-02-09T17:57:10.193 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15397864:::benchmark_data_smithi179_148205_object96466:head# 2024-02-09T17:57:10.193 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1539802d:::benchmark_data_smithi179_148205_object74885:head# 2024-02-09T17:57:10.196 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:153a7c0e:::benchmark_data_smithi179_148205_object26376:head# 2024-02-09T17:57:10.196 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:153d6ef8:::benchmark_data_smithi179_148205_object29035:head# 2024-02-09T17:57:10.197 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:153e75ee:::benchmark_data_smithi179_148205_object13844:head# 2024-02-09T17:57:10.197 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:153eb687:::benchmark_data_smithi179_148205_object60266:head# 2024-02-09T17:57:10.197 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:153f7418:::benchmark_data_smithi179_148205_object602:head# 2024-02-09T17:57:10.198 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:153fd3f3:::benchmark_data_smithi179_148205_object3052:head# 2024-02-09T17:57:10.198 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15404b62:::benchmark_data_smithi179_148205_object73182:head# 2024-02-09T17:57:10.199 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1540df35:::benchmark_data_smithi179_148205_object5384:head# 2024-02-09T17:57:10.206 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1541c112:::benchmark_data_smithi179_148205_object29976:head# 2024-02-09T17:57:10.206 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1541cd73:::benchmark_data_smithi179_148205_object77981:head# 2024-02-09T17:57:10.206 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1543823e:::benchmark_data_smithi179_148205_object22133:head# 2024-02-09T17:57:10.206 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1543bcf6:::benchmark_data_smithi179_148205_object29645:head# 2024-02-09T17:57:10.206 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1543c8fe:::benchmark_data_smithi179_148205_object56438:head# 2024-02-09T17:57:10.206 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15441313:::benchmark_data_smithi179_148205_object6407:head# 2024-02-09T17:57:10.206 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15453b65:::benchmark_data_smithi179_148205_object73134:head# 2024-02-09T17:57:10.206 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15458216:::benchmark_data_smithi179_148205_object200:head# 2024-02-09T17:57:10.206 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1545b92a:::benchmark_data_smithi179_148205_object105693:head# 2024-02-09T17:57:10.206 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1545edf3:::benchmark_data_smithi179_148205_object104562:head# 2024-02-09T17:57:10.206 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15467156:::benchmark_data_smithi179_148205_object16237:head# 2024-02-09T17:57:10.206 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15470de7:::benchmark_data_smithi179_148205_object9723:head# 2024-02-09T17:57:10.206 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1547f23d:::benchmark_data_smithi179_148205_object72477:head# 2024-02-09T17:57:10.207 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1548d47d:::benchmark_data_smithi179_148205_object68765:head# 2024-02-09T17:57:10.207 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1549866e:::benchmark_data_smithi179_148205_object62677:head# 2024-02-09T17:57:10.207 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1549efe2:::benchmark_data_smithi179_148205_object58861:head# 2024-02-09T17:57:10.207 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:154a2096:::benchmark_data_smithi179_148205_object858:head# 2024-02-09T17:57:10.207 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:154a9777:::benchmark_data_smithi179_148205_object75454:head# 2024-02-09T17:57:10.208 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:154af210:::benchmark_data_smithi179_148205_object20542:head# 2024-02-09T17:57:10.208 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:154b3c98:::benchmark_data_smithi179_148205_object106191:head# 2024-02-09T17:57:10.209 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:154b63f1:::benchmark_data_smithi179_148205_object25044:head# 2024-02-09T17:57:10.209 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:154b74d0:::benchmark_data_smithi179_148205_object5276:head# 2024-02-09T17:57:10.209 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:154c16f1:::benchmark_data_smithi179_148205_object63428:head# 2024-02-09T17:57:10.210 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:154c5b3c:::benchmark_data_smithi179_148205_object64763:head# 2024-02-09T17:57:10.210 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:154d910e:::benchmark_data_smithi179_148205_object16137:head# 2024-02-09T17:57:10.211 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:154e80fb:::benchmark_data_smithi179_148205_object68086:head# 2024-02-09T17:57:10.211 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:154f6315:::benchmark_data_smithi179_148205_object45668:head# 2024-02-09T17:57:10.215 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:154f691a:::benchmark_data_smithi179_148205_object44206:head# 2024-02-09T17:57:10.216 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:154f8bc5:::benchmark_data_smithi179_148205_object82366:head# 2024-02-09T17:57:10.219 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:155001c8:::benchmark_data_smithi179_148205_object23499:head# 2024-02-09T17:57:10.219 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:155139f6:::benchmark_data_smithi179_148205_object43274:head# 2024-02-09T17:57:10.219 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15515363:::benchmark_data_smithi179_148205_object9392:head# 2024-02-09T17:57:10.219 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15544aab:::benchmark_data_smithi179_148205_object74575:head# 2024-02-09T17:57:10.219 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:155462e8:::benchmark_data_smithi179_148205_object58700:head# 2024-02-09T17:57:10.219 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15546a12:::benchmark_data_smithi179_148205_object108341:head# 2024-02-09T17:57:10.221 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15566084:::benchmark_data_smithi179_148205_object62588:head# 2024-02-09T17:57:10.225 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:155770f0:::benchmark_data_smithi179_148205_object21425:head# 2024-02-09T17:57:10.227 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15592c34:::benchmark_data_smithi179_148205_object41641:head# 2024-02-09T17:57:10.227 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:155a2d1a:::benchmark_data_smithi179_148205_object29267:head# 2024-02-09T17:57:10.227 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:155b0ece:::benchmark_data_smithi179_148205_object20505:head# 2024-02-09T17:57:10.227 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:155b5eb2:::benchmark_data_smithi179_148205_object3531:head# 2024-02-09T17:57:10.227 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:155bbc9b:::benchmark_data_smithi179_148205_object90622:head# 2024-02-09T17:57:10.229 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:155ce3b3:::benchmark_data_smithi179_148205_object60076:head# 2024-02-09T17:57:10.229 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:155d4844:::benchmark_data_smithi179_148205_object85805:head# 2024-02-09T17:57:10.229 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:155dd46f:::benchmark_data_smithi179_148205_object92278:head# 2024-02-09T17:57:10.230 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:155ddd31:::benchmark_data_smithi179_148205_object70295:head# 2024-02-09T17:57:10.231 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:155f4127:::benchmark_data_smithi179_148205_object28846:head# 2024-02-09T17:57:10.231 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:156004a4:::benchmark_data_smithi179_148205_object76058:head# 2024-02-09T17:57:10.232 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1560dfd5:::benchmark_data_smithi179_148205_object76708:head# 2024-02-09T17:57:10.232 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1561e968:::benchmark_data_smithi179_148205_object38816:head# 2024-02-09T17:57:10.232 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1562c4c0:::benchmark_data_smithi179_148205_object100182:head# 2024-02-09T17:57:10.233 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15633cc4:::benchmark_data_smithi179_148205_object44886:head# 2024-02-09T17:57:10.233 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1563cc85:::benchmark_data_smithi179_148205_object46340:head# 2024-02-09T17:57:10.233 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1564cd77:::benchmark_data_smithi179_148205_object17654:head# 2024-02-09T17:57:10.234 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1564d7bf:::benchmark_data_smithi179_148205_object58417:head# 2024-02-09T17:57:10.234 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:156526b6:::benchmark_data_smithi179_148205_object9108:head# 2024-02-09T17:57:10.234 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15660e69:::benchmark_data_smithi179_148205_object27742:head# 2024-02-09T17:57:10.235 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15662894:::benchmark_data_smithi179_148205_object84732:head# 2024-02-09T17:57:10.235 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15663bbf:::benchmark_data_smithi179_148205_object39802:head# 2024-02-09T17:57:10.236 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:156774e7:::benchmark_data_smithi179_148205_object64538:head# 2024-02-09T17:57:10.236 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1567d340:::benchmark_data_smithi179_148205_object78663:head# 2024-02-09T17:57:10.236 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1567e16e:::benchmark_data_smithi179_148205_object81684:head# 2024-02-09T17:57:10.237 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1567f37f:::benchmark_data_smithi179_148205_object87389:head# 2024-02-09T17:57:10.237 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:156840db:::benchmark_data_smithi179_148205_object97331:head# 2024-02-09T17:57:10.238 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1569fe18:::benchmark_data_smithi179_148205_object22008:head# 2024-02-09T17:57:10.238 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:156af22c:::benchmark_data_smithi179_148205_object99917:head# 2024-02-09T17:57:10.238 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:156b7734:::benchmark_data_smithi179_148205_object26400:head# 2024-02-09T17:57:10.239 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:156c19a8:::benchmark_data_smithi179_148205_object14166:head# 2024-02-09T17:57:10.239 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:156c2247:::benchmark_data_smithi179_148205_object25862:head# 2024-02-09T17:57:10.239 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:156c8b4c:::benchmark_data_smithi179_148205_object1285:head# 2024-02-09T17:57:10.240 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15709512:::benchmark_data_smithi179_148205_object106708:head# 2024-02-09T17:57:10.240 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15725320:::benchmark_data_smithi179_148205_object69731:head# 2024-02-09T17:57:10.241 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1572cfae:::benchmark_data_smithi179_148205_object80004:head# 2024-02-09T17:57:10.241 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1573cdb5:::benchmark_data_smithi179_148205_object1488:head# 2024-02-09T17:57:10.242 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15740d34:::benchmark_data_smithi179_148205_object78848:head# 2024-02-09T17:57:10.242 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15753091:::benchmark_data_smithi179_148205_object57081:head# 2024-02-09T17:57:10.243 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15762c98:::benchmark_data_smithi179_148205_object89060:head# 2024-02-09T17:57:10.243 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15770a59:::benchmark_data_smithi179_148205_object76673:head# 2024-02-09T17:57:10.243 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1577fa11:::benchmark_data_smithi179_148205_object20200:head# 2024-02-09T17:57:10.244 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:157800dc:::benchmark_data_smithi179_148205_object62636:head# 2024-02-09T17:57:10.245 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15787aa9:::benchmark_data_smithi179_148205_object14662:head# 2024-02-09T17:57:10.245 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1579a9da:::benchmark_data_smithi179_148205_object103817:head# 2024-02-09T17:57:10.245 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:157acaa0:::benchmark_data_smithi179_148205_object81609:head# 2024-02-09T17:57:10.245 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:157d6f14:::benchmark_data_smithi179_148205_object54129:head# 2024-02-09T17:57:10.246 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:157e1431:::benchmark_data_smithi179_148205_object2968:head# 2024-02-09T17:57:10.247 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:157e4b31:::benchmark_data_smithi179_148205_object64061:head# 2024-02-09T17:57:10.247 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:157e8d76:::benchmark_data_smithi179_148205_object1080:head# 2024-02-09T17:57:10.247 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:157e972b:::benchmark_data_smithi179_148205_object42454:head# 2024-02-09T17:57:10.248 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:157eaa39:::benchmark_data_smithi179_148205_object102812:head# 2024-02-09T17:57:10.248 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:157ec073:::benchmark_data_smithi179_148205_object94674:head# 2024-02-09T17:57:10.249 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:157edc51:::benchmark_data_smithi179_148205_object94462:head# 2024-02-09T17:57:10.249 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:157f43d4:::benchmark_data_smithi179_148205_object38411:head# 2024-02-09T17:57:10.249 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:157fbb41:::benchmark_data_smithi179_148205_object95294:head# 2024-02-09T17:57:10.250 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1580cd3f:::benchmark_data_smithi179_148205_object82602:head# 2024-02-09T17:57:10.250 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1581d3e3:::benchmark_data_smithi179_148205_object54192:head# 2024-02-09T17:57:10.251 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1581eaf2:::benchmark_data_smithi179_148205_object48375:head# 2024-02-09T17:57:10.251 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1582330b:::benchmark_data_smithi179_148205_object58113:head# 2024-02-09T17:57:10.255 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1582bb75:::benchmark_data_smithi179_148205_object90960:head# 2024-02-09T17:57:10.256 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:158410e7:::benchmark_data_smithi179_148205_object93893:head# 2024-02-09T17:57:10.256 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1584173b:::benchmark_data_smithi179_148205_object48341:head# 2024-02-09T17:57:10.256 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1584c9e7:::benchmark_data_smithi179_148205_object87839:head# 2024-02-09T17:57:10.256 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:158539c0:::benchmark_data_smithi179_148205_object61582:head# 2024-02-09T17:57:10.256 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1585ac8c:::benchmark_data_smithi179_148205_object48170:head# 2024-02-09T17:57:10.256 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:158674dc:::benchmark_data_smithi179_148205_object20754:head# 2024-02-09T17:57:10.256 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15879624:::benchmark_data_smithi179_148205_object86563:head# 2024-02-09T17:57:10.257 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1587b5d5:::benchmark_data_smithi179_148205_object7804:head# 2024-02-09T17:57:10.257 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1587bde1:::benchmark_data_smithi179_148205_object72107:head# 2024-02-09T17:57:10.257 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1587d35f:::benchmark_data_smithi179_148205_object53422:head# 2024-02-09T17:57:10.258 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15880380:::benchmark_data_smithi179_148205_object64255:head# 2024-02-09T17:57:10.258 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15887f57:::benchmark_data_smithi179_148205_object49858:head# 2024-02-09T17:57:10.258 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1588c8c9:::benchmark_data_smithi179_148205_object94652:head# 2024-02-09T17:57:10.258 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15895006:::benchmark_data_smithi179_148205_object19750:head# 2024-02-09T17:57:10.259 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15896212:::benchmark_data_smithi179_148205_object98116:head# 2024-02-09T17:57:10.259 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1589b3f6:::benchmark_data_smithi179_148205_object99340:head# 2024-02-09T17:57:10.259 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1589c986:::benchmark_data_smithi179_148205_object67103:head# 2024-02-09T17:57:10.260 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:158a663b:::benchmark_data_smithi179_148205_object80186:head# 2024-02-09T17:57:10.260 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:158b2368:::benchmark_data_smithi179_148205_object86140:head# 2024-02-09T17:57:10.264 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:158ccc81:::benchmark_data_smithi179_148205_object8487:head# 2024-02-09T17:57:10.266 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:158cddc1:::benchmark_data_smithi179_148205_object98401:head# 2024-02-09T17:57:10.266 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:158cf5e5:::benchmark_data_smithi179_148205_object77569:head# 2024-02-09T17:57:10.266 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:158d12b1:::benchmark_data_smithi179_148205_object108183:head# 2024-02-09T17:57:10.266 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:158d1554:::benchmark_data_smithi179_148205_object69337:head# 2024-02-09T17:57:10.266 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:158d5e29:::benchmark_data_smithi179_148205_object83978:head# 2024-02-09T17:57:10.266 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:158db77e:::benchmark_data_smithi179_148205_object58005:head# 2024-02-09T17:57:10.266 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:158e42b3:::benchmark_data_smithi179_148205_object96896:head# 2024-02-09T17:57:10.266 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:158ec2cb:::benchmark_data_smithi179_148205_object5215:head# 2024-02-09T17:57:10.269 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:158edfc2:::benchmark_data_smithi179_148205_object102535:head# 2024-02-09T17:57:10.269 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:158f5caa:::benchmark_data_smithi179_148205_object69609:head# 2024-02-09T17:57:10.269 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15907948:::benchmark_data_smithi179_148205_object101655:head# 2024-02-09T17:57:10.269 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1590f2f3:::benchmark_data_smithi179_148205_object48988:head# 2024-02-09T17:57:10.269 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1591ecb2:::benchmark_data_smithi179_148205_object67701:head# 2024-02-09T17:57:10.269 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15929660:::benchmark_data_smithi179_148205_object39107:head# 2024-02-09T17:57:10.269 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1592b997:::benchmark_data_smithi179_148205_object19176:head# 2024-02-09T17:57:10.269 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15935835:::benchmark_data_smithi179_148205_object106636:head# 2024-02-09T17:57:10.273 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1594002f:::benchmark_data_smithi179_148205_object29730:head# 2024-02-09T17:57:10.273 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15946ca2:::benchmark_data_smithi179_148205_object23981:head# 2024-02-09T17:57:10.273 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1597dbdc:::benchmark_data_smithi179_148205_object93060:head# 2024-02-09T17:57:10.273 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1599309a:::benchmark_data_smithi179_148205_object13457:head# 2024-02-09T17:57:10.273 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1599a884:::benchmark_data_smithi179_148205_object76833:head# 2024-02-09T17:57:10.274 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:159ab323:::benchmark_data_smithi179_148205_object74118:head# 2024-02-09T17:57:10.274 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:159adab2:::benchmark_data_smithi179_148205_object61743:head# 2024-02-09T17:57:10.274 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:159be6f6:::benchmark_data_smithi179_148205_object84653:head# 2024-02-09T17:57:10.274 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:159e1854:::benchmark_data_smithi179_148205_object90258:head# 2024-02-09T17:57:10.274 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:159e4c81:::benchmark_data_smithi179_148205_object74086:head# 2024-02-09T17:57:10.274 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:159eaffd:::benchmark_data_smithi179_148205_object107696:head# 2024-02-09T17:57:10.277 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:159f21a9:::benchmark_data_smithi179_148205_object37097:head# 2024-02-09T17:57:10.278 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:159f9946:::benchmark_data_smithi179_148205_object13384:head# 2024-02-09T17:57:10.278 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15a0191f:::benchmark_data_smithi179_148205_object97102:head# 2024-02-09T17:57:10.279 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15a03ecc:::benchmark_data_smithi179_148205_object3297:head# 2024-02-09T17:57:10.279 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15a092a5:::benchmark_data_smithi179_148205_object23851:head# 2024-02-09T17:57:10.280 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15a0b65e:::benchmark_data_smithi179_148205_object88967:head# 2024-02-09T17:57:10.280 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15a11593:::benchmark_data_smithi179_148205_object5918:head# 2024-02-09T17:57:10.281 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15a3b558:::benchmark_data_smithi179_148205_object56575:head# 2024-02-09T17:57:10.281 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15a580b0:::benchmark_data_smithi179_148205_object104992:head# 2024-02-09T17:57:10.281 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15a6f111:::benchmark_data_smithi179_148205_object92450:head# 2024-02-09T17:57:10.282 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15a6f4c8:::benchmark_data_smithi179_148205_object54353:head# 2024-02-09T17:57:10.282 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15a7414a:::benchmark_data_smithi179_148205_object35681:head# 2024-02-09T17:57:10.283 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15a7aac8:::benchmark_data_smithi179_148205_object106981:head# 2024-02-09T17:57:10.283 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15a7b80c:::benchmark_data_smithi179_148205_object73597:head# 2024-02-09T17:57:10.283 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15a7c885:::benchmark_data_smithi179_148205_object77215:head# 2024-02-09T17:57:10.284 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15a822e3:::benchmark_data_smithi179_148205_object52109:head# 2024-02-09T17:57:10.284 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15a866fb:::benchmark_data_smithi179_148205_object105270:head# 2024-02-09T17:57:10.285 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15a89305:::benchmark_data_smithi179_148205_object102045:head# 2024-02-09T17:57:10.285 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15a896ad:::benchmark_data_smithi179_148205_object63750:head# 2024-02-09T17:57:10.285 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15a8ac0d:::benchmark_data_smithi179_148205_object28736:head# 2024-02-09T17:57:10.286 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15a8dfe5:::benchmark_data_smithi179_148205_object105937:head# 2024-02-09T17:57:10.286 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15a950e9:::benchmark_data_smithi179_148205_object103389:head# 2024-02-09T17:57:10.286 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15a9c6b7:::benchmark_data_smithi179_148205_object60201:head# 2024-02-09T17:57:10.287 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15a9c959:::benchmark_data_smithi179_148205_object7711:head# 2024-02-09T17:57:10.287 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15aa96c9:::benchmark_data_smithi179_148205_object10695:head# 2024-02-09T17:57:10.288 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15aaf948:::benchmark_data_smithi179_148205_object62649:head# 2024-02-09T17:57:10.288 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15ad4215:::benchmark_data_smithi179_148205_object97952:head# 2024-02-09T17:57:10.289 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15af5269:::benchmark_data_smithi179_148205_object90374:head# 2024-02-09T17:57:10.289 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15af603e:::benchmark_data_smithi179_148205_object71749:head# 2024-02-09T17:57:10.290 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15af8a3c:::benchmark_data_smithi179_148205_object105360:head# 2024-02-09T17:57:10.290 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15b08f61:::benchmark_data_smithi179_148205_object78747:head# 2024-02-09T17:57:10.291 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15b0ce87:::benchmark_data_smithi179_148205_object104689:head# 2024-02-09T17:57:10.295 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15b14412:::benchmark_data_smithi179_148205_object24374:head# 2024-02-09T17:57:10.296 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15b19ec6:::benchmark_data_smithi179_148205_object33484:head# 2024-02-09T17:57:10.296 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15b1e434:::benchmark_data_smithi179_148205_object30062:head# 2024-02-09T17:57:10.296 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15b1e968:::benchmark_data_smithi179_148205_object81774:head# 2024-02-09T17:57:10.296 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15b1f334:::benchmark_data_smithi179_148205_object98642:head# 2024-02-09T17:57:10.296 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15b207a6:::benchmark_data_smithi179_148205_object90602:head# 2024-02-09T17:57:10.296 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15b23ba1:::benchmark_data_smithi179_148205_object108852:head# 2024-02-09T17:57:10.296 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15b2b15d:::benchmark_data_smithi179_148205_object30927:head# 2024-02-09T17:57:10.296 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15b3695d:::benchmark_data_smithi179_148205_object100021:head# 2024-02-09T17:57:10.296 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15b3bff0:::benchmark_data_smithi179_148205_object7073:head# 2024-02-09T17:57:10.297 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15b43f61:::benchmark_data_smithi179_148205_object36412:head# 2024-02-09T17:57:10.297 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15b4accb:::benchmark_data_smithi179_148205_object92860:head# 2024-02-09T17:57:10.299 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15b72bde:::benchmark_data_smithi179_148205_object80975:head# 2024-02-09T17:57:10.300 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15b79c89:::benchmark_data_smithi179_148205_object56012:head# 2024-02-09T17:57:10.300 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15b8b427:::benchmark_data_smithi179_148205_object39138:head# 2024-02-09T17:57:10.300 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15b9ba3f:::benchmark_data_smithi179_148205_object109738:head# 2024-02-09T17:57:10.300 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15b9e70e:::benchmark_data_smithi179_148205_object13014:head# 2024-02-09T17:57:10.300 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15b9fa1f:::benchmark_data_smithi179_148205_object73520:head# 2024-02-09T17:57:10.304 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15ba0054:::benchmark_data_smithi179_148205_object96683:head# 2024-02-09T17:57:10.304 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15bb4a7f:::benchmark_data_smithi179_148205_object39739:head# 2024-02-09T17:57:10.305 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15bbf44e:::benchmark_data_smithi179_148205_object57190:head# 2024-02-09T17:57:10.305 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15bc4d81:::benchmark_data_smithi179_148205_object37722:head# 2024-02-09T17:57:10.306 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15bd6584:::benchmark_data_smithi179_148205_object70080:head# 2024-02-09T17:57:10.306 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15bdd6c6:::benchmark_data_smithi179_148205_object80184:head# 2024-02-09T17:57:10.307 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15bddf7c:::benchmark_data_smithi179_148205_object771:head# 2024-02-09T17:57:10.307 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15bfb5eb:::benchmark_data_smithi179_148205_object4481:head# 2024-02-09T17:57:10.308 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15bfdca4:::benchmark_data_smithi179_148205_object48981:head# 2024-02-09T17:57:10.310 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15c0a6d4:::benchmark_data_smithi179_148205_object15274:head# 2024-02-09T17:57:10.310 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15c0b751:::benchmark_data_smithi179_148205_object61236:head# 2024-02-09T17:57:10.310 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15c1157b:::benchmark_data_smithi179_148205_object18072:head# 2024-02-09T17:57:10.310 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15c146d5:::benchmark_data_smithi179_148205_object59491:head# 2024-02-09T17:57:10.310 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15c1b2c3:::benchmark_data_smithi179_148205_object39409:head# 2024-02-09T17:57:10.313 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15c3a66e:::benchmark_data_smithi179_148205_object6256:head# 2024-02-09T17:57:10.318 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15c45a9b:::benchmark_data_smithi179_148205_object89029:head# 2024-02-09T17:57:10.318 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15c4607f:::benchmark_data_smithi179_148205_object87986:head# 2024-02-09T17:57:10.318 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15c55684:::benchmark_data_smithi179_148205_object75183:head# 2024-02-09T17:57:10.318 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15c55e8c:::benchmark_data_smithi179_148205_object68495:head# 2024-02-09T17:57:10.319 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15c5b293:::benchmark_data_smithi179_148205_object10666:head# 2024-02-09T17:57:10.321 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15c5cfaa:::benchmark_data_smithi179_148205_object28522:head# 2024-02-09T17:57:10.321 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15c61f47:::benchmark_data_smithi179_148205_object102692:head# 2024-02-09T17:57:10.321 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15c6e54d:::benchmark_data_smithi179_148205_object10379:head# 2024-02-09T17:57:10.322 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15c71d1b:::benchmark_data_smithi179_148205_object30875:head# 2024-02-09T17:57:10.322 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15c73990:::benchmark_data_smithi179_148205_object49718:head# 2024-02-09T17:57:10.323 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15c7754f:::benchmark_data_smithi179_148205_object45669:head# 2024-02-09T17:57:10.324 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15c84719:::benchmark_data_smithi179_148205_object22102:head# 2024-02-09T17:57:10.324 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15c8ca3e:::benchmark_data_smithi179_148205_object39595:head# 2024-02-09T17:57:10.325 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15c8fd40:::benchmark_data_smithi179_148205_object29301:head# 2024-02-09T17:57:10.325 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15c90387:::benchmark_data_smithi179_148205_object34903:head# 2024-02-09T17:57:10.327 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15c90ccf:::benchmark_data_smithi179_148205_object10772:head# 2024-02-09T17:57:10.332 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15c93901:::benchmark_data_smithi179_148205_object31455:head# 2024-02-09T17:57:10.332 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15c9472c:::benchmark_data_smithi179_148205_object4585:head# 2024-02-09T17:57:10.332 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15c976a3:::benchmark_data_smithi179_148205_object10612:head# 2024-02-09T17:57:10.332 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15c99de1:::benchmark_data_smithi179_148205_object21365:head# 2024-02-09T17:57:10.333 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15ca1d76:::benchmark_data_smithi179_148205_object10248:head# 2024-02-09T17:57:10.333 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15ca47d8:::benchmark_data_smithi179_148205_object59979:head# 2024-02-09T17:57:10.333 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15ca991b:::benchmark_data_smithi179_148205_object23031:head# 2024-02-09T17:57:10.334 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15cc2fed:::benchmark_data_smithi179_148205_object2502:head# 2024-02-09T17:57:10.334 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15cc6778:::benchmark_data_smithi179_148205_object103456:head# 2024-02-09T17:57:10.335 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15ccc77d:::benchmark_data_smithi179_148205_object41115:head# 2024-02-09T17:57:10.335 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15ccd81f:::benchmark_data_smithi179_148205_object74665:head# 2024-02-09T17:57:10.335 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15cd8289:::benchmark_data_smithi179_148205_object98172:head# 2024-02-09T17:57:10.336 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15ce6432:::benchmark_data_smithi179_148205_object27156:head# 2024-02-09T17:57:10.336 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15cea4a4:::benchmark_data_smithi179_148205_object54581:head# 2024-02-09T17:57:10.337 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15cf1932:::benchmark_data_smithi179_148205_object1387:head# 2024-02-09T17:57:10.342 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15cfdbb7:::benchmark_data_smithi179_148205_object75608:head# 2024-02-09T17:57:10.342 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15d05fb2:::benchmark_data_smithi179_148205_object17329:head# 2024-02-09T17:57:10.342 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15d09a01:::benchmark_data_smithi179_148205_object81442:head# 2024-02-09T17:57:10.342 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15d0aa62:::benchmark_data_smithi179_148205_object98688:head# 2024-02-09T17:57:10.342 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15d1923a:::benchmark_data_smithi179_148205_object83914:head# 2024-02-09T17:57:10.342 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15d201de:::benchmark_data_smithi179_148205_object20879:head# 2024-02-09T17:57:10.342 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15d2aef5:::benchmark_data_smithi179_148205_object20610:head# 2024-02-09T17:57:10.347 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15d3eb7d:::benchmark_data_smithi179_148205_object80227:head# 2024-02-09T17:57:10.348 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15d4f59a:::benchmark_data_smithi179_148205_object81888:head# 2024-02-09T17:57:10.348 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15d5104d:::benchmark_data_smithi179_148205_object66912:head# 2024-02-09T17:57:10.348 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15d6a3bf:::benchmark_data_smithi179_148205_object90950:head# 2024-02-09T17:57:10.348 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15d75e17:::benchmark_data_smithi179_148205_object96674:head# 2024-02-09T17:57:10.348 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15d88981:::benchmark_data_smithi179_148205_object45011:head# 2024-02-09T17:57:10.348 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15d921be:::benchmark_data_smithi179_148205_object22326:head# 2024-02-09T17:57:10.351 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15d981a5:::benchmark_data_smithi179_148205_object26537:head# 2024-02-09T17:57:10.352 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15d9b2a2:::benchmark_data_smithi179_148205_object85081:head# 2024-02-09T17:57:10.352 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15da8ab8:::benchmark_data_smithi179_148205_object55696:head# 2024-02-09T17:57:10.352 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15dada58:::benchmark_data_smithi179_148205_object12404:head# 2024-02-09T17:57:10.352 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15dba0e8:::benchmark_data_smithi179_148205_object76898:head# 2024-02-09T17:57:10.352 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15dbc851:::benchmark_data_smithi179_148205_object13897:head# 2024-02-09T17:57:10.352 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15dc0dc5:::benchmark_data_smithi179_148205_object18567:head# 2024-02-09T17:57:10.353 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15dc7d67:::benchmark_data_smithi179_148205_object34345:head# 2024-02-09T17:57:10.353 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15dd7e2e:::benchmark_data_smithi179_148205_object61766:head# 2024-02-09T17:57:10.353 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15de3a67:::benchmark_data_smithi179_148205_object61246:head# 2024-02-09T17:57:10.353 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15de4745:::benchmark_data_smithi179_148205_object99622:head# 2024-02-09T17:57:10.353 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15de5910:::benchmark_data_smithi179_148205_object92006:head# 2024-02-09T17:57:10.354 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15de611d:::benchmark_data_smithi179_148205_object104569:head# 2024-02-09T17:57:10.354 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15de9fde:::benchmark_data_smithi179_148205_object57441:head# 2024-02-09T17:57:10.355 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15dfb183:::benchmark_data_smithi179_148205_object85213:head# 2024-02-09T17:57:10.355 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15dfd3e1:::benchmark_data_smithi179_148205_object69509:head# 2024-02-09T17:57:10.356 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15e0efe0:::benchmark_data_smithi179_148205_object39375:head# 2024-02-09T17:57:10.356 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15e19dd4:::benchmark_data_smithi179_148205_object23810:head# 2024-02-09T17:57:10.357 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15e1bc27:::benchmark_data_smithi179_148205_object9951:head# 2024-02-09T17:57:10.357 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15e30a3d:::benchmark_data_smithi179_148205_object53777:head# 2024-02-09T17:57:10.357 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15e3bfb2:::benchmark_data_smithi179_148205_object28158:head# 2024-02-09T17:57:10.358 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15e3da40:::benchmark_data_smithi179_148205_object69064:head# 2024-02-09T17:57:10.358 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15e3ec14:::benchmark_data_smithi179_148205_object70879:head# 2024-02-09T17:57:10.359 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15e4e58c:::benchmark_data_smithi179_148205_object52244:head# 2024-02-09T17:57:10.359 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15e4e6bd:::benchmark_data_smithi179_148205_object54674:head# 2024-02-09T17:57:10.360 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15e56873:::benchmark_data_smithi179_148205_object83363:head# 2024-02-09T17:57:10.360 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15e6deba:::benchmark_data_smithi179_148205_object2966:head# 2024-02-09T17:57:10.361 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15e6f59f:::benchmark_data_smithi179_148205_object91914:head# 2024-02-09T17:57:10.361 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15e704fc:::benchmark_data_smithi179_148205_object96364:head# 2024-02-09T17:57:10.362 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15e79345:::benchmark_data_smithi179_148205_object87449:head# 2024-02-09T17:57:10.362 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15e91edd:::benchmark_data_smithi179_148205_object33751:head# 2024-02-09T17:57:10.363 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15e99900:::benchmark_data_smithi179_148205_object72827:head# 2024-02-09T17:57:10.365 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15e9cc4e:::benchmark_data_smithi179_148205_object94657:head# 2024-02-09T17:57:10.365 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15ea671c:::benchmark_data_smithi179_148205_object44767:head# 2024-02-09T17:57:10.365 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15ebb2cf:::benchmark_data_smithi179_148205_object60689:head# 2024-02-09T17:57:10.365 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15ec301f:::benchmark_data_smithi179_148205_object74865:head# 2024-02-09T17:57:10.365 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15ed5a90:::benchmark_data_smithi179_148205_object31458:head# 2024-02-09T17:57:10.365 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15ed5bd6:::benchmark_data_smithi179_148205_object83189:head# 2024-02-09T17:57:10.368 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15ed95b2:::benchmark_data_smithi179_148205_object20574:head# 2024-02-09T17:57:10.369 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15ef52df:::benchmark_data_smithi179_148205_object21735:head# 2024-02-09T17:57:10.369 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15ef95d5:::benchmark_data_smithi179_148205_object6627:head# 2024-02-09T17:57:10.369 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15f078d2:::benchmark_data_smithi179_148205_object32526:head# 2024-02-09T17:57:10.369 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15f0bd0f:::benchmark_data_smithi179_148205_object10176:head# 2024-02-09T17:57:10.369 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15f10d91:::benchmark_data_smithi179_148205_object67903:head# 2024-02-09T17:57:10.369 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15f13196:::benchmark_data_smithi179_148205_object99428:head# 2024-02-09T17:57:10.369 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15f16303:::benchmark_data_smithi179_148205_object15618:head# 2024-02-09T17:57:10.370 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15f2a6ce:::benchmark_data_smithi179_148205_object22100:head# 2024-02-09T17:57:10.370 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15f2dcf4:::benchmark_data_smithi179_148205_object77839:head# 2024-02-09T17:57:10.370 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15f2e599:::benchmark_data_smithi179_148205_object38814:head# 2024-02-09T17:57:10.373 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15f35242:::benchmark_data_smithi179_148205_object81791:head# 2024-02-09T17:57:10.373 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15f39c4d:::benchmark_data_smithi179_148205_object78855:head# 2024-02-09T17:57:10.374 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15f4f0d1:::benchmark_data_smithi179_148205_object6221:head# 2024-02-09T17:57:10.374 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15f4f20f:::benchmark_data_smithi179_148205_object107795:head# 2024-02-09T17:57:10.375 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15f57cab:::benchmark_data_smithi179_148205_object37663:head# 2024-02-09T17:57:10.375 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15f6ca0e:::benchmark_data_smithi179_148205_object86033:head# 2024-02-09T17:57:10.376 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15f6ead3:::benchmark_data_smithi179_148205_object92010:head# 2024-02-09T17:57:10.376 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15f6f684:::benchmark_data_smithi179_148205_object64297:head# 2024-02-09T17:57:10.376 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15f785a2:::benchmark_data_smithi179_148205_object35259:head# 2024-02-09T17:57:10.377 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15f80c64:::benchmark_data_smithi179_148205_object42225:head# 2024-02-09T17:57:10.377 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15f81717:::benchmark_data_smithi179_148205_object43786:head# 2024-02-09T17:57:10.377 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15f8b8c3:::benchmark_data_smithi179_148205_object33092:head# 2024-02-09T17:57:10.378 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15f9dbaf:::benchmark_data_smithi179_148205_object53806:head# 2024-02-09T17:57:10.378 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15fa4bf8:::benchmark_data_smithi179_148205_object61316:head# 2024-02-09T17:57:10.381 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15fa9776:::benchmark_data_smithi179_148205_object77935:head# 2024-02-09T17:57:10.381 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15fb9ae1:::benchmark_data_smithi179_148205_object109146:head# 2024-02-09T17:57:10.381 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15fc2649:::benchmark_data_smithi179_148205_object34517:head# 2024-02-09T17:57:10.381 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15fc5abe:::benchmark_data_smithi179_148205_object96506:head# 2024-02-09T17:57:10.383 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15fcd35a:::benchmark_data_smithi179_148205_object56260:head# 2024-02-09T17:57:10.383 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15fdb80b:::benchmark_data_smithi179_148205_object103452:head# 2024-02-09T17:57:10.383 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15febe69:::benchmark_data_smithi179_148205_object15375:head# 2024-02-09T17:57:10.383 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15ff12cb:::benchmark_data_smithi179_148205_object92060:head# 2024-02-09T17:57:10.383 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15ff3005:::benchmark_data_smithi179_148205_object104280:head# 2024-02-09T17:57:10.383 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15ff4abc:::benchmark_data_smithi179_148205_object64182:head# 2024-02-09T17:57:10.384 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15ff67c0:::benchmark_data_smithi179_148205_object77469:head# 2024-02-09T17:57:10.385 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15ff8934:::benchmark_data_smithi179_148205_object3805:head# 2024-02-09T17:57:10.385 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:15ff945c:::benchmark_data_smithi179_148205_object89890:head# 2024-02-09T17:57:10.385 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16005d66:::benchmark_data_smithi179_148205_object5672:head# 2024-02-09T17:57:10.388 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1600b0b3:::benchmark_data_smithi179_148205_object57212:head# 2024-02-09T17:57:10.388 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16018b17:::benchmark_data_smithi179_148205_object25995:head# 2024-02-09T17:57:10.388 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1601c01d:::benchmark_data_smithi179_148205_object15771:head# 2024-02-09T17:57:10.388 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1601c97a:::benchmark_data_smithi179_148205_object101225:head# 2024-02-09T17:57:10.388 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16023959:::benchmark_data_smithi179_148205_object100791:head# 2024-02-09T17:57:10.388 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16026fa2:::benchmark_data_smithi179_148205_object103026:head# 2024-02-09T17:57:10.388 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:160339c0:::benchmark_data_smithi179_148205_object18940:head# 2024-02-09T17:57:10.389 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16037d73:::benchmark_data_smithi179_148205_object35150:head# 2024-02-09T17:57:10.389 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1603f31e:::benchmark_data_smithi179_148205_object84834:head# 2024-02-09T17:57:10.389 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16040932:::benchmark_data_smithi179_148205_object96797:head# 2024-02-09T17:57:10.389 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16049dc3:::benchmark_data_smithi179_148205_object37915:head# 2024-02-09T17:57:10.390 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1604b4d6:::benchmark_data_smithi179_148205_object45297:head# 2024-02-09T17:57:10.390 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1604d976:::benchmark_data_smithi179_148205_object109118:head# 2024-02-09T17:57:10.391 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1605013d:::benchmark_data_smithi179_148205_object107593:head# 2024-02-09T17:57:10.391 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16053840:::benchmark_data_smithi179_148205_object4742:head# 2024-02-09T17:57:10.392 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1605d3fb:::benchmark_data_smithi179_148205_object32348:head# 2024-02-09T17:57:10.392 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:160833e6:::benchmark_data_smithi179_148205_object32376:head# 2024-02-09T17:57:10.392 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16099cac:::benchmark_data_smithi179_148205_object36921:head# 2024-02-09T17:57:10.393 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1609b37c:::benchmark_data_smithi179_148205_object58505:head# 2024-02-09T17:57:10.393 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1609b61f:::benchmark_data_smithi179_148205_object78042:head# 2024-02-09T17:57:10.394 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1609fcd6:::benchmark_data_smithi179_148205_object49600:head# 2024-02-09T17:57:10.394 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:160a395d:::benchmark_data_smithi179_148205_object109186:head# 2024-02-09T17:57:10.394 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:160c3e6a:::benchmark_data_smithi179_148205_object102258:head# 2024-02-09T17:57:10.395 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:160c7af3:::benchmark_data_smithi179_148205_object103857:head# 2024-02-09T17:57:10.395 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:160cf2c0:::benchmark_data_smithi179_148205_object51421:head# 2024-02-09T17:57:10.400 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:160d20c6:::benchmark_data_smithi179_148205_object55522:head# 2024-02-09T17:57:10.400 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:160d6284:::benchmark_data_smithi179_148205_object68395:head# 2024-02-09T17:57:10.400 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:160ed043:::benchmark_data_smithi179_148205_object41479:head# 2024-02-09T17:57:10.401 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:160f3229:::benchmark_data_smithi179_148205_object86588:head# 2024-02-09T17:57:10.401 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:160f97f8:::benchmark_data_smithi179_148205_object37612:head# 2024-02-09T17:57:10.402 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1610a14b:::benchmark_data_smithi179_148205_object17646:head# 2024-02-09T17:57:10.402 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1610a264:::benchmark_data_smithi179_148205_object87171:head# 2024-02-09T17:57:10.403 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16127f6c:::benchmark_data_smithi179_148205_object19071:head# 2024-02-09T17:57:10.403 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16161f2a:::benchmark_data_smithi179_148205_object67861:head# 2024-02-09T17:57:10.403 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:161672c2:::benchmark_data_smithi179_148205_object24839:head# 2024-02-09T17:57:10.404 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1616c919:::benchmark_data_smithi179_148205_object78157:head# 2024-02-09T17:57:10.404 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16173444:::benchmark_data_smithi179_148205_object66963:head# 2024-02-09T17:57:10.405 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1617af13:::benchmark_data_smithi179_148205_object53220:head# 2024-02-09T17:57:10.405 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1618cb1b:::benchmark_data_smithi179_148205_object16534:head# 2024-02-09T17:57:10.405 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:161a36af:::benchmark_data_smithi179_148205_object80192:head# 2024-02-09T17:57:10.406 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:161a8a2c:::benchmark_data_smithi179_148205_object52076:head# 2024-02-09T17:57:10.406 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:161b2c99:::benchmark_data_smithi179_148205_object14769:head# 2024-02-09T17:57:10.407 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:161baf01:::benchmark_data_smithi179_148205_object108569:head# 2024-02-09T17:57:10.407 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:161e7aa5:::benchmark_data_smithi179_148205_object73902:head# 2024-02-09T17:57:10.412 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:161edd5c:::benchmark_data_smithi179_148205_object68971:head# 2024-02-09T17:57:10.412 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:161f5bc7:::benchmark_data_smithi179_148205_object14797:head# 2024-02-09T17:57:10.412 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:161fb430:::benchmark_data_smithi179_148205_object17776:head# 2024-02-09T17:57:10.413 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16207bce:::benchmark_data_smithi179_148205_object64687:head# 2024-02-09T17:57:10.413 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1620c134:::benchmark_data_smithi179_148205_object36940:head# 2024-02-09T17:57:10.413 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1621b3ca:::benchmark_data_smithi179_148205_object96040:head# 2024-02-09T17:57:10.413 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16238b54:::benchmark_data_smithi179_148205_object36875:head# 2024-02-09T17:57:10.413 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1623ee40:::benchmark_data_smithi179_148205_object25004:head# 2024-02-09T17:57:10.413 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1625ac3e:::benchmark_data_smithi179_148205_object36293:head# 2024-02-09T17:57:10.413 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:162610eb:::benchmark_data_smithi179_148205_object100999:head# 2024-02-09T17:57:10.414 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16271e9d:::benchmark_data_smithi179_148205_object22604:head# 2024-02-09T17:57:10.414 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:162782ec:::benchmark_data_smithi179_148205_object58363:head# 2024-02-09T17:57:10.415 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16280dfb:::benchmark_data_smithi179_148205_object21128:head# 2024-02-09T17:57:10.415 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1628a2ef:::benchmark_data_smithi179_148205_object64164:head# 2024-02-09T17:57:10.415 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16292367:::benchmark_data_smithi179_148205_object97086:head# 2024-02-09T17:57:10.416 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:162ae4fd:::benchmark_data_smithi179_148205_object55812:head# 2024-02-09T17:57:10.416 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:162b0243:::benchmark_data_smithi179_148205_object66267:head# 2024-02-09T17:57:10.417 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:162b1739:::benchmark_data_smithi179_148205_object90341:head# 2024-02-09T17:57:10.418 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:162b50d4:::benchmark_data_smithi179_148205_object51887:head# 2024-02-09T17:57:10.421 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:57:10.422 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-09T17:57:10.422 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-09T17:57:10.423 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:162cbb43:::benchmark_data_smithi179_148205_object91677:head# 2024-02-09T17:57:10.423 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:162d3a14:::benchmark_data_smithi179_148205_object69501:head# 2024-02-09T17:57:10.423 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:162d3d1e:::benchmark_data_smithi179_148205_object88825:head# 2024-02-09T17:57:10.423 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:162dd8f1:::benchmark_data_smithi179_148205_object84459:head# 2024-02-09T17:57:10.424 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:162f0217:::benchmark_data_smithi179_148205_object18505:head# 2024-02-09T17:57:10.424 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:162f6f66:::benchmark_data_smithi179_148205_object57266:head# 2024-02-09T17:57:10.424 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:162f82d2:::benchmark_data_smithi179_148205_object13430:head# 2024-02-09T17:57:10.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:10.431 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:162f9f49:::benchmark_data_smithi179_148205_object107783:head# 2024-02-09T17:57:10.432 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:162fa377:::benchmark_data_smithi179_148205_object103671:head# 2024-02-09T17:57:10.432 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16301cc0:::benchmark_data_smithi179_148205_object84662:head# 2024-02-09T17:57:10.432 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1630959b:::benchmark_data_smithi179_148205_object46434:head# 2024-02-09T17:57:10.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:57:10.433 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16316113:::benchmark_data_smithi179_148205_object62840:head# 2024-02-09T17:57:10.434 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1631c48e:::benchmark_data_smithi179_148205_object17814:head# 2024-02-09T17:57:10.434 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16331d7d:::benchmark_data_smithi179_148205_object38610:head# 2024-02-09T17:57:10.434 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16353336:::benchmark_data_smithi179_148205_object106934:head# 2024-02-09T17:57:10.434 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:163558c6:::benchmark_data_smithi179_148205_object87715:head# 2024-02-09T17:57:10.434 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16363880:::benchmark_data_smithi179_148205_object18364:head# 2024-02-09T17:57:10.434 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1636e527:::benchmark_data_smithi179_148205_object104075:head# 2024-02-09T17:57:10.434 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1636e61d:::benchmark_data_smithi179_148205_object32884:head# 2024-02-09T17:57:10.434 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16388247:::benchmark_data_smithi179_148205_object8162:head# 2024-02-09T17:57:10.435 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:163967f5:::benchmark_data_smithi179_148205_object29209:head# 2024-02-09T17:57:10.435 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16398b3b:::benchmark_data_smithi179_148205_object6879:head# 2024-02-09T17:57:10.435 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1639d0b8:::benchmark_data_smithi179_148205_object27832:head# 2024-02-09T17:57:10.436 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:163a46c9:::benchmark_data_smithi179_148205_object65455:head# 2024-02-09T17:57:10.436 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:163a70a2:::benchmark_data_smithi179_148205_object52746:head# 2024-02-09T17:57:10.437 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:163a87fb:::benchmark_data_smithi179_148205_object60028:head# 2024-02-09T17:57:10.437 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:163ab82c:::benchmark_data_smithi179_148205_object96212:head# 2024-02-09T17:57:10.437 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:163abf74:::benchmark_data_smithi179_148205_object52620:head# 2024-02-09T17:57:10.438 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:163b968c:::benchmark_data_smithi179_148205_object89305:head# 2024-02-09T17:57:10.439 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:163be316:::benchmark_data_smithi179_148205_object13330:head# 2024-02-09T17:57:10.439 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:163cb019:::benchmark_data_smithi179_148205_object67238:head# 2024-02-09T17:57:10.440 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:163d5c07:::benchmark_data_smithi179_148205_object33461:head# 2024-02-09T17:57:10.440 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:163d8ec4:::benchmark_data_smithi179_148205_object42002:head# 2024-02-09T17:57:10.440 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:163da1e0:::benchmark_data_smithi179_148205_object66463:head# 2024-02-09T17:57:10.441 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:163e0b96:::benchmark_data_smithi179_148205_object62981:head# 2024-02-09T17:57:10.441 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:163e4134:::benchmark_data_smithi179_148205_object6514:head# 2024-02-09T17:57:10.442 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:163eab20:::benchmark_data_smithi179_148205_object85722:head# 2024-02-09T17:57:10.442 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:163f147d:::benchmark_data_smithi179_148205_object40063:head# 2024-02-09T17:57:10.444 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:163f596e:::benchmark_data_smithi179_148205_object97364:head# 2024-02-09T17:57:10.444 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:163fc46a:::benchmark_data_smithi179_148205_object37488:head# 2024-02-09T17:57:10.444 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1640438e:::benchmark_data_smithi179_148205_object82288:head# 2024-02-09T17:57:10.444 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1640a9ff:::benchmark_data_smithi179_148205_object46382:head# 2024-02-09T17:57:10.445 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1641a8a8:::benchmark_data_smithi179_148205_object21560:head# 2024-02-09T17:57:10.445 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1641b89f:::benchmark_data_smithi179_148205_object81709:head# 2024-02-09T17:57:10.445 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1642e5a2:::benchmark_data_smithi179_148205_object6892:head# 2024-02-09T17:57:10.446 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16439ee0:::benchmark_data_smithi179_148205_object97604:head# 2024-02-09T17:57:10.446 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1643f2ba:::benchmark_data_smithi179_148205_object89791:head# 2024-02-09T17:57:10.447 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16443bdf:::benchmark_data_smithi179_148205_object5775:head# 2024-02-09T17:57:10.447 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16451ab1:::benchmark_data_smithi179_148205_object90533:head# 2024-02-09T17:57:10.447 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1647576e:::benchmark_data_smithi179_148205_object8909:head# 2024-02-09T17:57:10.448 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:164815b6:::benchmark_data_smithi179_148205_object34046:head# 2024-02-09T17:57:10.448 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1648c421:::benchmark_data_smithi179_148205_object38061:head# 2024-02-09T17:57:10.451 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1648e5d9:::benchmark_data_smithi179_148205_object19668:head# 2024-02-09T17:57:10.451 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:164a61c1:::benchmark_data_smithi179_148205_object74469:head# 2024-02-09T17:57:10.452 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:164a893b:::benchmark_data_smithi179_148205_object28349:head# 2024-02-09T17:57:10.452 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:164b2be9:::benchmark_data_smithi179_148205_object101121:head# 2024-02-09T17:57:10.452 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:164bba3e:::benchmark_data_smithi179_148205_object46437:head# 2024-02-09T17:57:10.452 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:164c1236:::benchmark_data_smithi179_148205_object47815:head# 2024-02-09T17:57:10.453 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:164cff24:::benchmark_data_smithi179_148205_object40966:head# 2024-02-09T17:57:10.453 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:164d21de:::benchmark_data_smithi179_148205_object46084:head# 2024-02-09T17:57:10.455 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:164dfaa3:::benchmark_data_smithi179_148205_object67014:head# 2024-02-09T17:57:10.455 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:164f5da0:::benchmark_data_smithi179_148205_object48468:head# 2024-02-09T17:57:10.455 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:164f8aa1:::benchmark_data_smithi179_148205_object23337:head# 2024-02-09T17:57:10.455 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1650bda4:::benchmark_data_smithi179_148205_object3053:head# 2024-02-09T17:57:10.455 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1650fe05:::benchmark_data_smithi179_148205_object66159:head# 2024-02-09T17:57:10.456 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16519744:::benchmark_data_smithi179_148205_object64241:head# 2024-02-09T17:57:10.456 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16524874:::benchmark_data_smithi179_148205_object36264:head# 2024-02-09T17:57:10.456 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16526808:::benchmark_data_smithi179_148205_object52358:head# 2024-02-09T17:57:10.457 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16535834:::benchmark_data_smithi179_148205_object39620:head# 2024-02-09T17:57:10.457 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:165508d8:::benchmark_data_smithi179_148205_object97078:head# 2024-02-09T17:57:10.458 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1655a78a:::benchmark_data_smithi179_148205_object89321:head# 2024-02-09T17:57:10.458 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16563f18:::benchmark_data_smithi179_148205_object68263:head# 2024-02-09T17:57:10.459 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16568b9f:::benchmark_data_smithi179_148205_object30896:head# 2024-02-09T17:57:10.460 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:165752da:::benchmark_data_smithi179_148205_object95746:head# 2024-02-09T17:57:10.460 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1657b86e:::benchmark_data_smithi179_148205_object99254:head# 2024-02-09T17:57:10.461 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1657c3a3:::benchmark_data_smithi179_148205_object102216:head# 2024-02-09T17:57:10.461 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:165815c4:::benchmark_data_smithi179_148205_object25776:head# 2024-02-09T17:57:10.461 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1658a350:::benchmark_data_smithi179_148205_object94929:head# 2024-02-09T17:57:10.462 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1658aa35:::benchmark_data_smithi179_148205_object98328:head# 2024-02-09T17:57:10.462 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16593abb:::benchmark_data_smithi179_148205_object43236:head# 2024-02-09T17:57:10.463 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16597762:::benchmark_data_smithi179_148205_object9362:head# 2024-02-09T17:57:10.463 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:165997b3:::benchmark_data_smithi179_148205_object49255:head# 2024-02-09T17:57:10.463 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:165b2e45:::benchmark_data_smithi179_148205_object47685:head# 2024-02-09T17:57:10.464 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:165c3f71:::benchmark_data_smithi179_148205_object52989:head# 2024-02-09T17:57:10.464 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:165c8e33:::benchmark_data_smithi179_148205_object25515:head# 2024-02-09T17:57:10.465 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:165cc5c4:::benchmark_data_smithi179_148205_object33972:head# 2024-02-09T17:57:10.465 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:165d1b9f:::benchmark_data_smithi179_148205_object21070:head# 2024-02-09T17:57:10.466 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:165dff31:::benchmark_data_smithi179_148205_object3902:head# 2024-02-09T17:57:10.466 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:165e1994:::benchmark_data_smithi179_148205_object38409:head# 2024-02-09T17:57:10.468 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:165ef024:::benchmark_data_smithi179_148205_object85329:head# 2024-02-09T17:57:10.469 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:165fbd36:::benchmark_data_smithi179_148205_object60999:head# 2024-02-09T17:57:10.469 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:165fcff6:::benchmark_data_smithi179_148205_object8039:head# 2024-02-09T17:57:10.470 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:166047c5:::benchmark_data_smithi179_148205_object57065:head# 2024-02-09T17:57:10.470 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16605722:::benchmark_data_smithi179_148205_object38484:head# 2024-02-09T17:57:10.471 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16611b52:::benchmark_data_smithi179_148205_object106280:head# 2024-02-09T17:57:10.471 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16613580:::benchmark_data_smithi179_148205_object22490:head# 2024-02-09T17:57:10.471 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16618795:::benchmark_data_smithi179_148205_object107051:head# 2024-02-09T17:57:10.472 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1662616e:::benchmark_data_smithi179_148205_object96863:head# 2024-02-09T17:57:10.472 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1662f75e:::benchmark_data_smithi179_148205_object48136:head# 2024-02-09T17:57:10.473 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:166311eb:::benchmark_data_smithi179_148205_object82975:head# 2024-02-09T17:57:10.474 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16633183:::benchmark_data_smithi179_148205_object61693:head# 2024-02-09T17:57:10.476 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16634f8c:::benchmark_data_smithi179_148205_object11816:head# 2024-02-09T17:57:10.477 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1663ab69:::benchmark_data_smithi179_148205_object5241:head# 2024-02-09T17:57:10.477 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16646282:::benchmark_data_smithi179_148205_object72941:head# 2024-02-09T17:57:10.477 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16651ee1:::benchmark_data_smithi179_148205_object31663:head# 2024-02-09T17:57:10.477 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16663a2b:::benchmark_data_smithi179_148205_object10019:head# 2024-02-09T17:57:10.477 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1667458f:::benchmark_data_smithi179_148205_object26029:head# 2024-02-09T17:57:10.477 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1668b824:::benchmark_data_smithi179_148205_object100811:head# 2024-02-09T17:57:10.477 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16693f94:::benchmark_data_smithi179_148205_object21195:head# 2024-02-09T17:57:10.478 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:166b21e6:::benchmark_data_smithi179_148205_object108219:head# 2024-02-09T17:57:10.478 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:166d6e4e:::benchmark_data_smithi179_148205_object24880:head# 2024-02-09T17:57:10.479 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:166d8b04:::benchmark_data_smithi179_148205_object78397:head# 2024-02-09T17:57:10.479 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:166e7c59:::benchmark_data_smithi179_148205_object4017:head# 2024-02-09T17:57:10.479 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:166fe1a9:::benchmark_data_smithi179_148205_object101991:head# 2024-02-09T17:57:10.480 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1670425b:::benchmark_data_smithi179_148205_object78273:head# 2024-02-09T17:57:10.480 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:167084c5:::benchmark_data_smithi179_148205_object22464:head# 2024-02-09T17:57:10.481 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1670e2d8:::benchmark_data_smithi179_148205_object23425:head# 2024-02-09T17:57:10.481 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16717df3:::benchmark_data_smithi179_148205_object44690:head# 2024-02-09T17:57:10.483 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1671fa57:::benchmark_data_smithi179_148205_object9715:head# 2024-02-09T17:57:10.483 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16736d40:::benchmark_data_smithi179_148205_object107501:head# 2024-02-09T17:57:10.485 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1673abe0:::benchmark_data_smithi179_148205_object106344:head# 2024-02-09T17:57:10.485 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1673b6d7:::benchmark_data_smithi179_148205_object107975:head# 2024-02-09T17:57:10.485 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16741f17:::benchmark_data_smithi179_148205_object28096:head# 2024-02-09T17:57:10.486 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16742f21:::benchmark_data_smithi179_148205_object48841:head# 2024-02-09T17:57:10.486 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1674e4f3:::benchmark_data_smithi179_148205_object106684:head# 2024-02-09T17:57:10.487 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1674e5a8:::benchmark_data_smithi179_148205_object51079:head# 2024-02-09T17:57:10.487 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16762fed:::benchmark_data_smithi179_148205_object102745:head# 2024-02-09T17:57:10.487 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:167663a1:::benchmark_data_smithi179_148205_object20231:head# 2024-02-09T17:57:10.488 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16771c44:::benchmark_data_smithi179_148205_object90827:head# 2024-02-09T17:57:10.488 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16779b73:::benchmark_data_smithi179_148205_object20670:head# 2024-02-09T17:57:10.489 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1677fc4f:::benchmark_data_smithi179_148205_object19515:head# 2024-02-09T17:57:10.489 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16782894:::benchmark_data_smithi179_148205_object108305:head# 2024-02-09T17:57:10.490 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:167a369c:::benchmark_data_smithi179_148205_object89230:head# 2024-02-09T17:57:10.492 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:167cc4b8:::benchmark_data_smithi179_148205_object69534:head# 2024-02-09T17:57:10.493 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:167e0243:::benchmark_data_smithi179_148205_object48135:head# 2024-02-09T17:57:10.493 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:167e9e65:::benchmark_data_smithi179_148205_object77907:head# 2024-02-09T17:57:10.494 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:167eb98e:::benchmark_data_smithi179_148205_object18110:head# 2024-02-09T17:57:10.497 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:167f0f38:::benchmark_data_smithi179_148205_object75959:head# 2024-02-09T17:57:10.497 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16801d65:::benchmark_data_smithi179_148205_object18144:head# 2024-02-09T17:57:10.497 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:168074af:::benchmark_data_smithi179_148205_object80592:head# 2024-02-09T17:57:10.497 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1680f068:::benchmark_data_smithi179_148205_object85406:head# 2024-02-09T17:57:10.497 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1680fcdb:::benchmark_data_smithi179_148205_object70455:head# 2024-02-09T17:57:10.497 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16815af1:::benchmark_data_smithi179_148205_object78083:head# 2024-02-09T17:57:10.497 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1682a5d8:::benchmark_data_smithi179_148205_object109208:head# 2024-02-09T17:57:10.498 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1682b24c:::benchmark_data_smithi179_148205_object4454:head# 2024-02-09T17:57:10.498 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16832a4a:::benchmark_data_smithi179_148205_object56949:head# 2024-02-09T17:57:10.498 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16834c0b:::benchmark_data_smithi179_148205_object104868:head# 2024-02-09T17:57:10.498 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1684532b:::benchmark_data_smithi179_148205_object59020:head# 2024-02-09T17:57:10.499 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1684d95d:::benchmark_data_smithi179_148205_object107299:head# 2024-02-09T17:57:10.499 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:168560d5:::benchmark_data_smithi179_148205_object73777:head# 2024-02-09T17:57:10.500 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16856bca:::benchmark_data_smithi179_148205_object109268:head# 2024-02-09T17:57:10.500 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1685ef92:::benchmark_data_smithi179_148205_object55038:head# 2024-02-09T17:57:10.502 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:168608ce:::benchmark_data_smithi179_148205_object11009:head# 2024-02-09T17:57:10.503 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1686ed2d:::benchmark_data_smithi179_148205_object36468:head# 2024-02-09T17:57:10.503 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1686fc55:::benchmark_data_smithi179_148205_object92688:head# 2024-02-09T17:57:10.503 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16873ba2:::benchmark_data_smithi179_148205_object45956:head# 2024-02-09T17:57:10.504 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16888e3c:::benchmark_data_smithi179_148205_object43380:head# 2024-02-09T17:57:10.504 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16892914:::benchmark_data_smithi179_148205_object60065:head# 2024-02-09T17:57:10.505 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16895ce5:::benchmark_data_smithi179_148205_object16045:head# 2024-02-09T17:57:10.505 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:168ad50d:::benchmark_data_smithi179_148205_object93589:head# 2024-02-09T17:57:10.506 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:168b635d:::benchmark_data_smithi179_148205_object64451:head# 2024-02-09T17:57:10.506 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:168b6b71:::benchmark_data_smithi179_148205_object45963:head# 2024-02-09T17:57:10.506 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:168b86d9:::benchmark_data_smithi179_148205_object16365:head# 2024-02-09T17:57:10.507 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:168bf227:::benchmark_data_smithi179_148205_object105697:head# 2024-02-09T17:57:10.507 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:168bf6ae:::benchmark_data_smithi179_148205_object48077:head# 2024-02-09T17:57:10.508 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:168c5564:::benchmark_data_smithi179_148205_object43570:head# 2024-02-09T17:57:10.512 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:168cae19:::benchmark_data_smithi179_148205_object86539:head# 2024-02-09T17:57:10.512 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:168cb445:::benchmark_data_smithi179_148205_object93856:head# 2024-02-09T17:57:10.512 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:168cf23a:::benchmark_data_smithi179_148205_object5150:head# 2024-02-09T17:57:10.513 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:168d19cb:::benchmark_data_smithi179_148205_object1354:head# 2024-02-09T17:57:10.513 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:168df668:::benchmark_data_smithi179_148205_object97978:head# 2024-02-09T17:57:10.513 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:168dfbf5:::benchmark_data_smithi179_148205_object45927:head# 2024-02-09T17:57:10.513 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:168e40d3:::benchmark_data_smithi179_148205_object15233:head# 2024-02-09T17:57:10.513 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:168f217a:::benchmark_data_smithi179_148205_object31690:head# 2024-02-09T17:57:10.513 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:168f80a4:::benchmark_data_smithi179_148205_object83580:head# 2024-02-09T17:57:10.513 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:168ffdf6:::benchmark_data_smithi179_148205_object72629:head# 2024-02-09T17:57:10.513 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16905a5a:::benchmark_data_smithi179_148205_object25775:head# 2024-02-09T17:57:10.514 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1690869d:::benchmark_data_smithi179_148205_object53907:head# 2024-02-09T17:57:10.514 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:169114fc:::benchmark_data_smithi179_148205_object85827:head# 2024-02-09T17:57:10.514 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16932f15:::benchmark_data_smithi179_148205_object92054:head# 2024-02-09T17:57:10.515 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16934629:::benchmark_data_smithi179_148205_object77104:head# 2024-02-09T17:57:10.515 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16944622:::benchmark_data_smithi179_148205_object104253:head# 2024-02-09T17:57:10.515 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1694f5e3:::benchmark_data_smithi179_148205_object26905:head# 2024-02-09T17:57:10.516 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16951f48:::benchmark_data_smithi179_148205_object72805:head# 2024-02-09T17:57:10.516 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16976b1d:::benchmark_data_smithi179_148205_object95085:head# 2024-02-09T17:57:10.517 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1697f5b9:::benchmark_data_smithi179_148205_object76907:head# 2024-02-09T17:57:10.520 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16987a68:::benchmark_data_smithi179_148205_object11090:head# 2024-02-09T17:57:10.520 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1698a682:::benchmark_data_smithi179_148205_object48006:head# 2024-02-09T17:57:10.520 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1698ad28:::benchmark_data_smithi179_148205_object2592:head# 2024-02-09T17:57:10.521 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1699c90f:::benchmark_data_smithi179_148205_object106092:head# 2024-02-09T17:57:10.521 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1699e29e:::benchmark_data_smithi179_148205_object85607:head# 2024-02-09T17:57:10.522 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:169b7a71:::benchmark_data_smithi179_148205_object105446:head# 2024-02-09T17:57:10.522 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:169bf7ca:::benchmark_data_smithi179_148205_object52985:head# 2024-02-09T17:57:10.522 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:169cd3d9:::benchmark_data_smithi179_148205_object34952:head# 2024-02-09T17:57:10.523 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:169e03a6:::benchmark_data_smithi179_148205_object71434:head# 2024-02-09T17:57:10.523 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:169e8aab:::benchmark_data_smithi179_148205_object76197:head# 2024-02-09T17:57:10.524 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16a02eff:::benchmark_data_smithi179_148205_object25042:head# 2024-02-09T17:57:10.524 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16a10711:::benchmark_data_smithi179_148205_object13700:head# 2024-02-09T17:57:10.525 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16a1aa9a:::benchmark_data_smithi179_148205_object11880:head# 2024-02-09T17:57:10.525 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16a23f5f:::benchmark_data_smithi179_148205_object37421:head# 2024-02-09T17:57:10.526 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16a24d41:::benchmark_data_smithi179_148205_object143:head# 2024-02-09T17:57:10.526 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16a2bcad:::benchmark_data_smithi179_148205_object41268:head# 2024-02-09T17:57:10.526 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16a310d3:::benchmark_data_smithi179_148205_object21114:head# 2024-02-09T17:57:10.527 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16a36491:::benchmark_data_smithi179_148205_object33255:head# 2024-02-09T17:57:10.527 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16a3b017:::benchmark_data_smithi179_148205_object99615:head# 2024-02-09T17:57:10.528 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16a3b8b4:::benchmark_data_smithi179_148205_object27453:head# 2024-02-09T17:57:10.531 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16a3e85c:::benchmark_data_smithi179_148205_object106396:head# 2024-02-09T17:57:10.531 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16a4a5b9:::benchmark_data_smithi179_148205_object31765:head# 2024-02-09T17:57:10.532 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16a5a9ab:::benchmark_data_smithi179_148205_object81490:head# 2024-02-09T17:57:10.532 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16a658f8:::benchmark_data_smithi179_148205_object80634:head# 2024-02-09T17:57:10.533 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16a6e1e9:::benchmark_data_smithi179_148205_object56703:head# 2024-02-09T17:57:10.533 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16a7c72a:::benchmark_data_smithi179_148205_object29719:head# 2024-02-09T17:57:10.533 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:57:10.534 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16a8128d:::benchmark_data_smithi179_148205_object31495:head# 2024-02-09T17:57:10.534 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16a82ad7:::benchmark_data_smithi179_148205_object97619:head# 2024-02-09T17:57:10.535 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16a8eaa4:::benchmark_data_smithi179_148205_object34785:head# 2024-02-09T17:57:10.535 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16a9cc37:::benchmark_data_smithi179_148205_object23622:head# 2024-02-09T17:57:10.535 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16aa6e06:::benchmark_data_smithi179_148205_object96996:head# 2024-02-09T17:57:10.535 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16aa7c46:::benchmark_data_smithi179_148205_object87160:head# 2024-02-09T17:57:10.536 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16ab0dc5:::benchmark_data_smithi179_148205_object85600:head# 2024-02-09T17:57:10.536 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16ab26cc:::benchmark_data_smithi179_148205_object107392:head# 2024-02-09T17:57:10.537 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16abe2e6:::benchmark_data_smithi179_148205_object91261:head# 2024-02-09T17:57:10.537 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16ac24c3:::benchmark_data_smithi179_148205_object107933:head# 2024-02-09T17:57:10.537 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16ac78cb:::benchmark_data_smithi179_148205_object77073:head# 2024-02-09T17:57:10.538 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16ac9512:::benchmark_data_smithi179_148205_object19596:head# 2024-02-09T17:57:10.539 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16adbf49:::benchmark_data_smithi179_148205_object77454:head# 2024-02-09T17:57:10.540 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16af54f5:::benchmark_data_smithi179_148205_object24048:head# 2024-02-09T17:57:10.541 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16b07c0e:::benchmark_data_smithi179_148205_object95942:head# 2024-02-09T17:57:10.541 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16b084dc:::benchmark_data_smithi179_148205_object49363:head# 2024-02-09T17:57:10.541 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16b14727:::benchmark_data_smithi179_148205_object26159:head# 2024-02-09T17:57:10.541 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16b233a0:::benchmark_data_smithi179_148205_object61034:head# 2024-02-09T17:57:10.541 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16b2f9a5:::benchmark_data_smithi179_148205_object66816:head# 2024-02-09T17:57:10.542 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16b31afb:::benchmark_data_smithi179_148205_object58160:head# 2024-02-09T17:57:10.542 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16b3b38c:::benchmark_data_smithi179_148205_object2585:head# 2024-02-09T17:57:10.543 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16b3e447:::benchmark_data_smithi179_148205_object58165:head# 2024-02-09T17:57:10.543 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16b3ea4f:::benchmark_data_smithi179_148205_object30570:head# 2024-02-09T17:57:10.543 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16b3fbc4:::benchmark_data_smithi179_148205_object54462:head# 2024-02-09T17:57:10.544 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16b5735a:::benchmark_data_smithi179_148205_object95939:head# 2024-02-09T17:57:10.544 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16b59c30:::benchmark_data_smithi179_148205_object34425:head# 2024-02-09T17:57:10.549 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16b5a19b:::benchmark_data_smithi179_148205_object92503:head# 2024-02-09T17:57:10.549 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16b61108:::benchmark_data_smithi179_148205_object63036:head# 2024-02-09T17:57:10.550 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16b666a4:::benchmark_data_smithi179_148205_object79114:head# 2024-02-09T17:57:10.551 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16b721d2:::benchmark_data_smithi179_148205_object12100:head# 2024-02-09T17:57:10.552 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16b74c9f:::benchmark_data_smithi179_148205_object80641:head# 2024-02-09T17:57:10.552 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16b7f31e:::benchmark_data_smithi179_148205_object11547:head# 2024-02-09T17:57:10.553 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16b9b6cd:::benchmark_data_smithi179_148205_object81121:head# 2024-02-09T17:57:10.554 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16ba8602:::benchmark_data_smithi179_148205_object8878:head# 2024-02-09T17:57:10.555 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16bb7266:::benchmark_data_smithi179_148205_object62209:head# 2024-02-09T17:57:10.555 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16bbc163:::benchmark_data_smithi179_148205_object46480:head# 2024-02-09T17:57:10.555 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16bc2fdc:::benchmark_data_smithi179_148205_object91505:head# 2024-02-09T17:57:10.555 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16bc6472:::benchmark_data_smithi179_148205_object57106:head# 2024-02-09T17:57:10.556 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16bd0323:::benchmark_data_smithi179_148205_object94189:head# 2024-02-09T17:57:10.556 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16bd1469:::benchmark_data_smithi179_148205_object22005:head# 2024-02-09T17:57:10.556 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16bd6ccf:::benchmark_data_smithi179_148205_object88530:head# 2024-02-09T17:57:10.557 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16c07672:::benchmark_data_smithi179_148205_object81070:head# 2024-02-09T17:57:10.558 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16c0f93f:::benchmark_data_smithi179_148205_object49347:head# 2024-02-09T17:57:10.558 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16c13efd:::benchmark_data_smithi179_148205_object85952:head# 2024-02-09T17:57:10.559 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16c142f6:::benchmark_data_smithi179_148205_object5491:head# 2024-02-09T17:57:10.559 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16c17d77:::benchmark_data_smithi179_148205_object17123:head# 2024-02-09T17:57:10.561 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16c2829c:::benchmark_data_smithi179_148205_object28352:head# 2024-02-09T17:57:10.561 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16c2b3a3:::benchmark_data_smithi179_148205_object73402:head# 2024-02-09T17:57:10.561 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16c2d189:::benchmark_data_smithi179_148205_object35994:head# 2024-02-09T17:57:10.564 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:10.569 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16c373e8:::benchmark_data_smithi179_148205_object109581:head# 2024-02-09T17:57:10.569 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16c39613:::benchmark_data_smithi179_148205_object38652:head# 2024-02-09T17:57:10.569 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16c5ed6b:::benchmark_data_smithi179_148205_object73633:head# 2024-02-09T17:57:10.569 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16c5fb7c:::benchmark_data_smithi179_148205_object13198:head# 2024-02-09T17:57:10.569 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16c676b7:::benchmark_data_smithi179_148205_object84122:head# 2024-02-09T17:57:10.569 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16c7a930:::benchmark_data_smithi179_148205_object66866:head# 2024-02-09T17:57:10.569 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16c7c98e:::benchmark_data_smithi179_148205_object36395:head# 2024-02-09T17:57:10.569 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16c7d811:::benchmark_data_smithi179_148205_object1716:head# 2024-02-09T17:57:10.569 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16c7f54c:::benchmark_data_smithi179_148205_object14550:head# 2024-02-09T17:57:10.570 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16c80c6c:::benchmark_data_smithi179_148205_object108956:head# 2024-02-09T17:57:10.570 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16c82bbe:::benchmark_data_smithi179_148205_object95776:head# 2024-02-09T17:57:10.570 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16c83130:::benchmark_data_smithi179_148205_object75706:head# 2024-02-09T17:57:10.570 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16c87cf8:::benchmark_data_smithi179_148205_object108933:head# 2024-02-09T17:57:10.574 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16c91b8e:::benchmark_data_smithi179_148205_object93535:head# 2024-02-09T17:57:10.574 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16c99595:::benchmark_data_smithi179_148205_object58340:head# 2024-02-09T17:57:10.574 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16ca2fff:::benchmark_data_smithi179_148205_object81302:head# 2024-02-09T17:57:10.574 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16cc499c:::benchmark_data_smithi179_148205_object51507:head# 2024-02-09T17:57:10.574 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16ccacf7:::benchmark_data_smithi179_148205_object8184:head# 2024-02-09T17:57:10.574 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16cd9360:::benchmark_data_smithi179_148205_object96175:head# 2024-02-09T17:57:10.574 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16ce7c85:::benchmark_data_smithi179_148205_object89940:head# 2024-02-09T17:57:10.574 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16cf8f0e:::benchmark_data_smithi179_148205_object96605:head# 2024-02-09T17:57:10.574 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16d024b4:::benchmark_data_smithi179_148205_object47086:head# 2024-02-09T17:57:10.575 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16d034fc:::benchmark_data_smithi179_148205_object28622:head# 2024-02-09T17:57:10.576 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16d0506a:::benchmark_data_smithi179_148205_object83414:head# 2024-02-09T17:57:10.576 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16d1313c:::benchmark_data_smithi179_148205_object43969:head# 2024-02-09T17:57:10.576 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16d14c3e:::benchmark_data_smithi179_148205_object29043:head# 2024-02-09T17:57:10.577 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16d181d0:::benchmark_data_smithi179_148205_object20773:head# 2024-02-09T17:57:10.578 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16d18b7b:::benchmark_data_smithi179_148205_object26616:head# 2024-02-09T17:57:10.580 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16d281e3:::benchmark_data_smithi179_148205_object18562:head# 2024-02-09T17:57:10.580 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16d2f1e6:::benchmark_data_smithi179_148205_object38318:head# 2024-02-09T17:57:10.580 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16d31f23:::benchmark_data_smithi179_148205_object31493:head# 2024-02-09T17:57:10.580 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16d38a53:::benchmark_data_smithi179_148205_object77106:head# 2024-02-09T17:57:10.580 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16d3ce7b:::benchmark_data_smithi179_148205_object46207:head# 2024-02-09T17:57:10.583 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16d40b22:::benchmark_data_smithi179_148205_object32574:head# 2024-02-09T17:57:10.583 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16d414c6:::benchmark_data_smithi179_148205_object17404:head# 2024-02-09T17:57:10.584 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16d418f9:::benchmark_data_smithi179_148205_object87257:head# 2024-02-09T17:57:10.584 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16d4b481:::benchmark_data_smithi179_148205_object10252:head# 2024-02-09T17:57:10.584 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16d57aa6:::benchmark_data_smithi179_148205_object64157:head# 2024-02-09T17:57:10.587 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:57:10.587 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16d64f16:::benchmark_data_smithi179_148205_object58835:head# 2024-02-09T17:57:10.588 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16d7330b:::benchmark_data_smithi179_148205_object59358:head# 2024-02-09T17:57:10.588 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16d88221:::benchmark_data_smithi179_148205_object97256:head# 2024-02-09T17:57:10.588 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16d9a889:::benchmark_data_smithi179_148205_object54096:head# 2024-02-09T17:57:10.588 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16dab3c8:::benchmark_data_smithi179_148205_object37533:head# 2024-02-09T17:57:10.588 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16dacad8:::benchmark_data_smithi179_148205_object36046:head# 2024-02-09T17:57:10.589 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16db5b63:::benchmark_data_smithi179_148205_object45030:head# 2024-02-09T17:57:10.589 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16dbc5ec:::benchmark_data_smithi179_148205_object2944:head# 2024-02-09T17:57:10.589 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16dc27a8:::benchmark_data_smithi179_148205_object19102:head# 2024-02-09T17:57:10.590 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16dc3043:::benchmark_data_smithi179_148205_object62720:head# 2024-02-09T17:57:10.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:10.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:57:10.592 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16dc674d:::benchmark_data_smithi179_148205_object17692:head# 2024-02-09T17:57:10.592 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16dc8b72:::benchmark_data_smithi179_148205_object106968:head# 2024-02-09T17:57:10.593 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16dca2b3:::benchmark_data_smithi179_148205_object15545:head# 2024-02-09T17:57:10.593 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16dcdc8a:::benchmark_data_smithi179_148205_object13121:head# 2024-02-09T17:57:10.593 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16dd3bd7:::benchmark_data_smithi179_148205_object58546:head# 2024-02-09T17:57:10.594 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16dd4317:::benchmark_data_smithi179_148205_object82522:head# 2024-02-09T17:57:10.594 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16ddfee1:::benchmark_data_smithi179_148205_object42804:head# 2024-02-09T17:57:10.595 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16df0787:::benchmark_data_smithi179_148205_object13199:head# 2024-02-09T17:57:10.595 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16df5189:::benchmark_data_smithi179_148205_object100860:head# 2024-02-09T17:57:10.596 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16df775a:::benchmark_data_smithi179_148205_object62702:head# 2024-02-09T17:57:10.598 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16e0c060:::benchmark_data_smithi179_148205_object38079:head# 2024-02-09T17:57:10.598 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16e28613:::benchmark_data_smithi179_148205_object30264:head# 2024-02-09T17:57:10.602 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16e28dec:::benchmark_data_smithi179_148205_object101608:head# 2024-02-09T17:57:10.602 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16e37339:::benchmark_data_smithi179_148205_object76807:head# 2024-02-09T17:57:10.604 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16e3b2c7:::benchmark_data_smithi179_148205_object77745:head# 2024-02-09T17:57:10.604 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16e49ef6:::benchmark_data_smithi179_148205_object94905:head# 2024-02-09T17:57:10.604 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16e54048:::benchmark_data_smithi179_148205_object805:head# 2024-02-09T17:57:10.605 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16e63431:::benchmark_data_smithi179_148205_object88811:head# 2024-02-09T17:57:10.605 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16e6ae4e:::benchmark_data_smithi179_148205_object80682:head# 2024-02-09T17:57:10.605 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16e6ffee:::benchmark_data_smithi179_148205_object103965:head# 2024-02-09T17:57:10.606 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16e7008d:::benchmark_data_smithi179_148205_object4626:head# 2024-02-09T17:57:10.606 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16e71061:::benchmark_data_smithi179_148205_object31164:head# 2024-02-09T17:57:10.607 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16e74d30:::benchmark_data_smithi179_148205_object49162:head# 2024-02-09T17:57:10.607 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16e7c74b:::benchmark_data_smithi179_148205_object85844:head# 2024-02-09T17:57:10.607 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16e7ca8a:::benchmark_data_smithi179_148205_object60438:head# 2024-02-09T17:57:10.608 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16e8d79d:::benchmark_data_smithi179_148205_object77154:head# 2024-02-09T17:57:10.608 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16e93fa1:::benchmark_data_smithi179_148205_object37504:head# 2024-02-09T17:57:10.609 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16e969c8:::benchmark_data_smithi179_148205_object29450:head# 2024-02-09T17:57:10.609 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16ea0c5f:::benchmark_data_smithi179_148205_object34740:head# 2024-02-09T17:57:10.610 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16ea484b:::benchmark_data_smithi179_148205_object99782:head# 2024-02-09T17:57:10.610 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16eb6825:::benchmark_data_smithi179_148205_object1249:head# 2024-02-09T17:57:10.613 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16ec9a17:::benchmark_data_smithi179_148205_object63485:head# 2024-02-09T17:57:10.613 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16eda0bb:::benchmark_data_smithi179_148205_object9722:head# 2024-02-09T17:57:10.613 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16eef504:::benchmark_data_smithi179_148205_object97887:head# 2024-02-09T17:57:10.614 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16ef4825:::benchmark_data_smithi179_148205_object90546:head# 2024-02-09T17:57:10.615 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16efe9e8:::benchmark_data_smithi179_148205_object79331:head# 2024-02-09T17:57:10.615 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16eff14e:::benchmark_data_smithi179_148205_object94040:head# 2024-02-09T17:57:10.615 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16f09e31:::benchmark_data_smithi179_148205_object56963:head# 2024-02-09T17:57:10.616 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16f2532c:::benchmark_data_smithi179_148205_object24535:head# 2024-02-09T17:57:10.616 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16f27de7:::benchmark_data_smithi179_148205_object4699:head# 2024-02-09T17:57:10.625 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16f2deb1:::benchmark_data_smithi179_148205_object21465:head# 2024-02-09T17:57:10.625 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16f424f8:::benchmark_data_smithi179_148205_object13173:head# 2024-02-09T17:57:10.625 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16f5c14f:::benchmark_data_smithi179_148205_object14343:head# 2024-02-09T17:57:10.625 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16f64141:::benchmark_data_smithi179_148205_object64730:head# 2024-02-09T17:57:10.625 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16f7373d:::benchmark_data_smithi179_148205_object90407:head# 2024-02-09T17:57:10.625 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16f7fb7a:::benchmark_data_smithi179_148205_object40901:head# 2024-02-09T17:57:10.625 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16f7fe53:::benchmark_data_smithi179_148205_object1737:head# 2024-02-09T17:57:10.626 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16f8b7e2:::benchmark_data_smithi179_148205_object39765:head# 2024-02-09T17:57:10.626 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16f8cccb:::benchmark_data_smithi179_148205_object149:head# 2024-02-09T17:57:10.626 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16f91abf:::benchmark_data_smithi179_148205_object27813:head# 2024-02-09T17:57:10.627 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16f93d2d:::benchmark_data_smithi179_148205_object38922:head# 2024-02-09T17:57:10.627 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16f966fa:::benchmark_data_smithi179_148205_object80922:head# 2024-02-09T17:57:10.628 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16fa3bd3:::benchmark_data_smithi179_148205_object105961:head# 2024-02-09T17:57:10.628 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16fa69fb:::benchmark_data_smithi179_148205_object5249:head# 2024-02-09T17:57:10.629 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16fb2655:::benchmark_data_smithi179_148205_object31753:head# 2024-02-09T17:57:10.629 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16fb3d39:::benchmark_data_smithi179_148205_object56129:head# 2024-02-09T17:57:10.630 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16fb59f7:::benchmark_data_smithi179_148205_object28671:head# 2024-02-09T17:57:10.630 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16fbfd5c:::benchmark_data_smithi179_148205_object32737:head# 2024-02-09T17:57:10.631 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16fc0835:::benchmark_data_smithi179_148205_object31633:head# 2024-02-09T17:57:10.631 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16fc3a93:::benchmark_data_smithi179_148205_object16257:head# 2024-02-09T17:57:10.632 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16fc8d30:::benchmark_data_smithi179_148205_object5055:head# 2024-02-09T17:57:10.633 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16fc9a9e:::benchmark_data_smithi179_148205_object108725:head# 2024-02-09T17:57:10.633 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16fcdc97:::benchmark_data_smithi179_148205_object56155:head# 2024-02-09T17:57:10.633 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16fd032c:::benchmark_data_smithi179_148205_object88003:head# 2024-02-09T17:57:10.633 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16ff2849:::benchmark_data_smithi179_148205_object40884:head# 2024-02-09T17:57:10.634 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16ff38a6:::benchmark_data_smithi179_148205_object104808:head# 2024-02-09T17:57:10.634 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16ff57ed:::benchmark_data_smithi179_148205_object24382:head# 2024-02-09T17:57:10.635 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16ff7431:::benchmark_data_smithi179_148205_object57321:head# 2024-02-09T17:57:10.635 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16ffa09d:::benchmark_data_smithi179_148205_object72662:head# 2024-02-09T17:57:10.635 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:16ffb74c:::benchmark_data_smithi179_148205_object2667:head# 2024-02-09T17:57:10.636 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17006a7e:::benchmark_data_smithi179_148205_object49841:head# 2024-02-09T17:57:10.637 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1700a3dd:::benchmark_data_smithi179_148205_object95511:head# 2024-02-09T17:57:10.637 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1701818a:::benchmark_data_smithi179_148205_object56144:head# 2024-02-09T17:57:10.637 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17021fdf:::benchmark_data_smithi179_148205_object69684:head# 2024-02-09T17:57:10.638 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17027baf:::benchmark_data_smithi179_148205_object76184:head# 2024-02-09T17:57:10.638 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:170299a2:::benchmark_data_smithi179_148205_object20299:head# 2024-02-09T17:57:10.639 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17030c64:::benchmark_data_smithi179_148205_object89178:head# 2024-02-09T17:57:10.639 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17031a9f:::benchmark_data_smithi179_148205_object67088:head# 2024-02-09T17:57:10.640 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17036475:::benchmark_data_smithi179_148205_object71665:head# 2024-02-09T17:57:10.640 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1703743f:::benchmark_data_smithi179_148205_object86298:head# 2024-02-09T17:57:10.641 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:170449ad:::benchmark_data_smithi179_148205_object27648:head# 2024-02-09T17:57:10.641 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1704c948:::benchmark_data_smithi179_148205_object68360:head# 2024-02-09T17:57:10.642 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1705c338:::benchmark_data_smithi179_148205_object54314:head# 2024-02-09T17:57:10.642 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1707ffb6:::benchmark_data_smithi179_148205_object20883:head# 2024-02-09T17:57:10.642 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17081ca1:::benchmark_data_smithi179_148205_object51349:head# 2024-02-09T17:57:10.643 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17083e30:::benchmark_data_smithi179_148205_object41848:head# 2024-02-09T17:57:10.644 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1709431d:::benchmark_data_smithi179_148205_object28280:head# 2024-02-09T17:57:10.644 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17099e54:::benchmark_data_smithi179_148205_object80164:head# 2024-02-09T17:57:10.645 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1709c9b1:::benchmark_data_smithi179_148205_object81591:head# 2024-02-09T17:57:10.645 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:170a2058:::benchmark_data_smithi179_148205_object83830:head# 2024-02-09T17:57:10.646 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:170b179d:::benchmark_data_smithi179_148205_object81286:head# 2024-02-09T17:57:10.646 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:170b627a:::benchmark_data_smithi179_148205_object97867:head# 2024-02-09T17:57:10.646 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:170c1cac:::benchmark_data_smithi179_148205_object104656:head# 2024-02-09T17:57:10.647 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:170c42bb:::benchmark_data_smithi179_148205_object31347:head# 2024-02-09T17:57:10.647 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:170d26da:::benchmark_data_smithi179_148205_object87013:head# 2024-02-09T17:57:10.648 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:170da87f:::benchmark_data_smithi179_148205_object82494:head# 2024-02-09T17:57:10.648 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:170dbc96:::benchmark_data_smithi179_148205_object34098:head# 2024-02-09T17:57:10.648 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:170e1b6a:::benchmark_data_smithi179_148205_object87573:head# 2024-02-09T17:57:10.649 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:170e87bf:::benchmark_data_smithi179_148205_object22224:head# 2024-02-09T17:57:10.649 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:170e9208:::benchmark_data_smithi179_148205_object72230:head# 2024-02-09T17:57:10.649 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:170eb9ca:::benchmark_data_smithi179_148205_object6745:head# 2024-02-09T17:57:10.650 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:170ece86:::benchmark_data_smithi179_148205_object50756:head# 2024-02-09T17:57:10.650 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:170f707f:::benchmark_data_smithi179_148205_object101690:head# 2024-02-09T17:57:10.651 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17102862:::benchmark_data_smithi179_148205_object61697:head# 2024-02-09T17:57:10.652 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1710971e:::benchmark_data_smithi179_148205_object95982:head# 2024-02-09T17:57:10.652 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1710fd46:::benchmark_data_smithi179_148205_object82763:head# 2024-02-09T17:57:10.652 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17126274:::benchmark_data_smithi179_148205_object37879:head# 2024-02-09T17:57:10.653 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17127a56:::benchmark_data_smithi179_148205_object24518:head# 2024-02-09T17:57:10.653 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1712af45:::benchmark_data_smithi179_148205_object91140:head# 2024-02-09T17:57:10.653 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1712b73f:::benchmark_data_smithi179_148205_object36832:head# 2024-02-09T17:57:10.654 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1712e475:::benchmark_data_smithi179_148205_object66669:head# 2024-02-09T17:57:10.656 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:171303f9:::benchmark_data_smithi179_148205_object53800:head# 2024-02-09T17:57:10.656 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:171342f8:::benchmark_data_smithi179_148205_object16869:head# 2024-02-09T17:57:10.657 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17147c6b:::benchmark_data_smithi179_148205_object64122:head# 2024-02-09T17:57:10.657 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1714809d:::benchmark_data_smithi179_148205_object57025:head# 2024-02-09T17:57:10.657 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17152a00:::benchmark_data_smithi179_148205_object83945:head# 2024-02-09T17:57:10.658 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17157dc8:::benchmark_data_smithi179_148205_object24865:head# 2024-02-09T17:57:10.658 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1715ee71:::benchmark_data_smithi179_148205_object27995:head# 2024-02-09T17:57:10.659 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17160ba4:::benchmark_data_smithi179_148205_object14719:head# 2024-02-09T17:57:10.659 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1718b06d:::benchmark_data_smithi179_148205_object70837:head# 2024-02-09T17:57:10.659 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1718f480:::benchmark_data_smithi179_148205_object103520:head# 2024-02-09T17:57:10.660 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17190053:::benchmark_data_smithi179_148205_object4260:head# 2024-02-09T17:57:10.660 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17190540:::benchmark_data_smithi179_148205_object23391:head# 2024-02-09T17:57:10.661 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1719e6d7:::benchmark_data_smithi179_148205_object68839:head# 2024-02-09T17:57:10.661 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:171aae29:::benchmark_data_smithi179_148205_object103158:head# 2024-02-09T17:57:10.662 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:171ab4df:::benchmark_data_smithi179_148205_object83749:head# 2024-02-09T17:57:10.663 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:171cef14:::benchmark_data_smithi179_148205_object20615:head# 2024-02-09T17:57:10.663 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:171e0ac1:::benchmark_data_smithi179_148205_object57937:head# 2024-02-09T17:57:10.664 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:171f33d0:::benchmark_data_smithi179_148205_object58914:head# 2024-02-09T17:57:10.664 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:171fa230:::benchmark_data_smithi179_148205_object21690:head# 2024-02-09T17:57:10.665 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:171fdce8:::benchmark_data_smithi179_148205_object82944:head# 2024-02-09T17:57:10.665 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17200644:::benchmark_data_smithi179_148205_object54242:head# 2024-02-09T17:57:10.666 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:172012c9:::benchmark_data_smithi179_148205_object20865:head# 2024-02-09T17:57:10.666 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17202591:::benchmark_data_smithi179_148205_object16579:head# 2024-02-09T17:57:10.666 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17205657:::benchmark_data_smithi179_148205_object6757:head# 2024-02-09T17:57:10.667 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17209430:::benchmark_data_smithi179_148205_object2461:head# 2024-02-09T17:57:10.667 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1721974c:::benchmark_data_smithi179_148205_object46010:head# 2024-02-09T17:57:10.668 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17219aa9:::benchmark_data_smithi179_148205_object20082:head# 2024-02-09T17:57:10.668 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1722088e:::benchmark_data_smithi179_148205_object73990:head# 2024-02-09T17:57:10.669 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17220f46:::benchmark_data_smithi179_148205_object56236:head# 2024-02-09T17:57:10.669 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17223278:::benchmark_data_smithi179_148205_object41521:head# 2024-02-09T17:57:10.670 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17229514:::benchmark_data_smithi179_148205_object38606:head# 2024-02-09T17:57:10.670 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1722d3cd:::benchmark_data_smithi179_148205_object18141:head# 2024-02-09T17:57:10.670 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17245a14:::benchmark_data_smithi179_148205_object87677:head# 2024-02-09T17:57:10.671 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1724a35f:::benchmark_data_smithi179_148205_object3325:head# 2024-02-09T17:57:10.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:10 smithi179 ceph-mon[43975]: pgmap v2035: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 6.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 360 KiB/s wr, 1.12k op/s; 21384/339138 objects degraded (6.305%) 2024-02-09T17:57:10.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:10 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:57:10.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:10 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:57:10.672 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1724bcf7:::benchmark_data_smithi179_148205_object55053:head# 2024-02-09T17:57:10.673 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17252b95:::benchmark_data_smithi179_148205_object988:head# 2024-02-09T17:57:10.673 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1725e126:::benchmark_data_smithi179_148205_object51961:head# 2024-02-09T17:57:10.673 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17263b66:::benchmark_data_smithi179_148205_object66878:head# 2024-02-09T17:57:10.673 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1726d75a:::benchmark_data_smithi179_148205_object103491:head# 2024-02-09T17:57:10.673 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1728fdcf:::benchmark_data_smithi179_148205_object487:head# 2024-02-09T17:57:10.674 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:172916f9:::benchmark_data_smithi179_148205_object94157:head# 2024-02-09T17:57:10.674 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17295968:::benchmark_data_smithi179_148205_object5391:head# 2024-02-09T17:57:10.675 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17295a9c:::benchmark_data_smithi179_148205_object51366:head# 2024-02-09T17:57:10.675 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1729d6af:::benchmark_data_smithi179_148205_object92558:head# 2024-02-09T17:57:10.675 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:172a5512:::benchmark_data_smithi179_148205_object3567:head# 2024-02-09T17:57:10.676 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:172ab45c:::benchmark_data_smithi179_148205_object94552:head# 2024-02-09T17:57:10.676 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:172b1544:::benchmark_data_smithi179_148205_object93161:head# 2024-02-09T17:57:10.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:10 smithi162 ceph-mon[44304]: pgmap v2035: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 6.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 360 KiB/s wr, 1.12k op/s; 21384/339138 objects degraded (6.305%) 2024-02-09T17:57:10.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:10 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:57:10.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:10 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:57:10.677 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:172b96ca:::benchmark_data_smithi179_148205_object98955:head# 2024-02-09T17:57:10.678 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:172c2b7a:::benchmark_data_smithi179_148205_object8779:head# 2024-02-09T17:57:10.679 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:172ceaca:::benchmark_data_smithi179_148205_object55851:head# 2024-02-09T17:57:10.679 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:172e70af:::benchmark_data_smithi179_148205_object109200:head# 2024-02-09T17:57:10.679 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17304635:::benchmark_data_smithi179_148205_object35323:head# 2024-02-09T17:57:10.680 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:173214d2:::benchmark_data_smithi179_148205_object29037:head# 2024-02-09T17:57:10.680 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:173241a7:::benchmark_data_smithi179_148205_object74931:head# 2024-02-09T17:57:10.680 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17327ade:::benchmark_data_smithi179_148205_object19222:head# 2024-02-09T17:57:10.681 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:173297e0:::benchmark_data_smithi179_148205_object4877:head# 2024-02-09T17:57:10.681 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1732aea9:::benchmark_data_smithi179_148205_object9804:head# 2024-02-09T17:57:10.682 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1732cbae:::benchmark_data_smithi179_148205_object50766:head# 2024-02-09T17:57:10.682 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1733bb06:::benchmark_data_smithi179_148205_object15686:head# 2024-02-09T17:57:10.682 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17341877:::benchmark_data_smithi179_148205_object61117:head# 2024-02-09T17:57:10.683 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17342b3d:::benchmark_data_smithi179_148205_object61493:head# 2024-02-09T17:57:10.683 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17346be3:::benchmark_data_smithi179_148205_object64630:head# 2024-02-09T17:57:10.684 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1734efa0:::benchmark_data_smithi179_148205_object98531:head# 2024-02-09T17:57:10.684 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17352e66:::benchmark_data_smithi179_148205_object23253:head# 2024-02-09T17:57:10.684 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17367b95:::benchmark_data_smithi179_148205_object64400:head# 2024-02-09T17:57:10.685 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17375331:::benchmark_data_smithi179_148205_object56027:head# 2024-02-09T17:57:10.685 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1737827e:::benchmark_data_smithi179_148205_object104985:head# 2024-02-09T17:57:10.686 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1737c56b:::benchmark_data_smithi179_148205_object18112:head# 2024-02-09T17:57:10.687 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1738bcc2:::benchmark_data_smithi179_148205_object40640:head# 2024-02-09T17:57:10.687 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1739189b:::benchmark_data_smithi179_148205_object51021:head# 2024-02-09T17:57:10.687 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:173a1042:::benchmark_data_smithi179_148205_object17241:head# 2024-02-09T17:57:10.687 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:173a8f04:::benchmark_data_smithi179_148205_object13921:head# 2024-02-09T17:57:10.688 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:173adc08:::benchmark_data_smithi179_148205_object78800:head# 2024-02-09T17:57:10.689 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:173b4ea9:::benchmark_data_smithi179_148205_object90668:head# 2024-02-09T17:57:10.689 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:173b947d:::benchmark_data_smithi179_148205_object92415:head# 2024-02-09T17:57:10.689 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:173c5fd6:::benchmark_data_smithi179_148205_object7290:head# 2024-02-09T17:57:10.690 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:173cb74d:::benchmark_data_smithi179_148205_object74404:head# 2024-02-09T17:57:10.692 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:57:10.694 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:173dc4c6:::benchmark_data_smithi179_148205_object7546:head# 2024-02-09T17:57:10.694 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:173ea3b3:::benchmark_data_smithi179_148205_object62980:head# 2024-02-09T17:57:10.694 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:173ede16:::benchmark_data_smithi179_148205_object101737:head# 2024-02-09T17:57:10.694 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17414a5a:::benchmark_data_smithi179_148205_object13764:head# 2024-02-09T17:57:10.694 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1742eb69:::benchmark_data_smithi179_148205_object17835:head# 2024-02-09T17:57:10.694 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17432d66:::benchmark_data_smithi179_148205_object94744:head# 2024-02-09T17:57:10.695 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17441315:::benchmark_data_smithi179_148205_object70269:head# 2024-02-09T17:57:10.695 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1744227d:::benchmark_data_smithi179_148205_object64001:head# 2024-02-09T17:57:10.695 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17449ddd:::benchmark_data_smithi179_148205_object76243:head# 2024-02-09T17:57:10.695 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1745afc4:::benchmark_data_smithi179_148205_object78197:head# 2024-02-09T17:57:10.696 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:174808d0:::benchmark_data_smithi179_148205_object14301:head# 2024-02-09T17:57:10.696 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:174816c0:::benchmark_data_smithi179_148205_object18040:head# 2024-02-09T17:57:10.697 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:174885a0:::benchmark_data_smithi179_148205_object69887:head# 2024-02-09T17:57:10.698 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1748eadc:::benchmark_data_smithi179_148205_object31275:head# 2024-02-09T17:57:10.698 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17499f6e:::benchmark_data_smithi179_148205_object17786:head# 2024-02-09T17:57:10.698 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1749baf0:::benchmark_data_smithi179_148205_object16971:head# 2024-02-09T17:57:10.699 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1749be57:::benchmark_data_smithi179_148205_object76387:head# 2024-02-09T17:57:10.699 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1749db62:::benchmark_data_smithi179_148205_object80962:head# 2024-02-09T17:57:10.699 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:174a937e:::benchmark_data_smithi179_148205_object83686:head# 2024-02-09T17:57:10.700 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:174b4e8d:::benchmark_data_smithi179_148205_object103703:head# 2024-02-09T17:57:10.704 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:174bba5a:::benchmark_data_smithi179_148205_object57678:head# 2024-02-09T17:57:10.704 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:174bce94:::benchmark_data_smithi179_148205_object90490:head# 2024-02-09T17:57:10.704 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:174c5a1c:::benchmark_data_smithi179_148205_object95178:head# 2024-02-09T17:57:10.705 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:174c8858:::benchmark_data_smithi179_148205_object61041:head# 2024-02-09T17:57:10.705 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:174ce64e:::benchmark_data_smithi179_148205_object28597:head# 2024-02-09T17:57:10.706 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:174d235f:::benchmark_data_smithi179_148205_object16209:head# 2024-02-09T17:57:10.706 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:174d7bb0:::benchmark_data_smithi179_148205_object9594:head# 2024-02-09T17:57:10.706 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:174d87c4:::benchmark_data_smithi179_148205_object41987:head# 2024-02-09T17:57:10.707 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:174da99e:::benchmark_data_smithi179_148205_object72703:head# 2024-02-09T17:57:10.708 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:174db10d:::benchmark_data_smithi179_148205_object44056:head# 2024-02-09T17:57:10.708 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:174e107c:::benchmark_data_smithi179_148205_object107525:head# 2024-02-09T17:57:10.713 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:174e12f7:::benchmark_data_smithi179_148205_object24396:head# 2024-02-09T17:57:10.713 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:174fb8f0:::benchmark_data_smithi179_148205_object9695:head# 2024-02-09T17:57:10.713 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:174fc418:::benchmark_data_smithi179_148205_object96314:head# 2024-02-09T17:57:10.713 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1751507e:::benchmark_data_smithi179_148205_object38390:head# 2024-02-09T17:57:10.713 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1751f532:::benchmark_data_smithi179_148205_object106156:head# 2024-02-09T17:57:10.713 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17521a73:::benchmark_data_smithi179_148205_object37393:head# 2024-02-09T17:57:10.713 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:175223af:::benchmark_data_smithi179_148205_object28005:head# 2024-02-09T17:57:10.713 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:175245b1:::benchmark_data_smithi179_148205_object63592:head# 2024-02-09T17:57:10.713 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1752ca86:::benchmark_data_smithi179_148205_object21240:head# 2024-02-09T17:57:10.714 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1754698e:::benchmark_data_smithi179_148205_object64620:head# 2024-02-09T17:57:10.714 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17547af1:::benchmark_data_smithi179_148205_object109706:head# 2024-02-09T17:57:10.714 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17552f88:::benchmark_data_smithi179_148205_object68722:head# 2024-02-09T17:57:10.714 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17553543:::benchmark_data_smithi179_148205_object1606:head# 2024-02-09T17:57:10.715 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:175616cd:::benchmark_data_smithi179_148205_object4709:head# 2024-02-09T17:57:10.715 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17585e61:::benchmark_data_smithi179_148205_object107453:head# 2024-02-09T17:57:10.715 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:175a633a:::benchmark_data_smithi179_148205_object41160:head# 2024-02-09T17:57:10.716 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:175a6a30:::benchmark_data_smithi179_148205_object92102:head# 2024-02-09T17:57:10.716 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:175a9936:::benchmark_data_smithi179_148205_object53546:head# 2024-02-09T17:57:10.717 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:175ae97c:::benchmark_data_smithi179_148205_object92308:head# 2024-02-09T17:57:10.717 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:175b05d8:::benchmark_data_smithi179_148205_object31732:head# 2024-02-09T17:57:10.718 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:175b143f:::benchmark_data_smithi179_148205_object106702:head# 2024-02-09T17:57:10.718 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:175b796a:::benchmark_data_smithi179_148205_object105752:head# 2024-02-09T17:57:10.718 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:175ba2a4:::benchmark_data_smithi179_148205_object46333:head# 2024-02-09T17:57:10.719 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:175cb2b2:::benchmark_data_smithi179_148205_object19209:head# 2024-02-09T17:57:10.719 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:175d6d22:::benchmark_data_smithi179_148205_object102277:head# 2024-02-09T17:57:10.720 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:175d7c18:::benchmark_data_smithi179_148205_object15927:head# 2024-02-09T17:57:10.720 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:175d8fdd:::benchmark_data_smithi179_148205_object8794:head# 2024-02-09T17:57:10.720 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:175da43f:::benchmark_data_smithi179_148205_object56714:head# 2024-02-09T17:57:10.721 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:175e1db1:::benchmark_data_smithi179_148205_object37336:head# 2024-02-09T17:57:10.721 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:175e922a:::benchmark_data_smithi179_148205_object35629:head# 2024-02-09T17:57:10.722 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:176003b6:::benchmark_data_smithi179_148205_object12758:head# 2024-02-09T17:57:10.727 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17600fd1:::benchmark_data_smithi179_148205_object31585:head# 2024-02-09T17:57:10.727 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1760a8ee:::benchmark_data_smithi179_148205_object74537:head# 2024-02-09T17:57:10.727 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1760ff00:::benchmark_data_smithi179_148205_object6758:head# 2024-02-09T17:57:10.727 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17612bc3:::benchmark_data_smithi179_148205_object38759:head# 2024-02-09T17:57:10.727 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17616d5a:::benchmark_data_smithi179_148205_object94658:head# 2024-02-09T17:57:10.728 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17624859:::benchmark_data_smithi179_148205_object77766:head# 2024-02-09T17:57:10.728 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17633b5a:::benchmark_data_smithi179_148205_object90020:head# 2024-02-09T17:57:10.729 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17644a45:::benchmark_data_smithi179_148205_object17849:head# 2024-02-09T17:57:10.729 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:176623e4:::benchmark_data_smithi179_148205_object55490:head# 2024-02-09T17:57:10.729 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17666eec:::benchmark_data_smithi179_148205_object24676:head# 2024-02-09T17:57:10.730 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1766f325:::benchmark_data_smithi179_148205_object66362:head# 2024-02-09T17:57:10.730 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17675e48:::benchmark_data_smithi179_148205_object8089:head# 2024-02-09T17:57:10.730 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1767b330:::benchmark_data_smithi179_148205_object51741:head# 2024-02-09T17:57:10.731 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17680387:::benchmark_data_smithi179_148205_object93645:head# 2024-02-09T17:57:10.733 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17689847:::benchmark_data_smithi179_148205_object51592:head# 2024-02-09T17:57:10.734 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:176955cb:::benchmark_data_smithi179_148205_object6715:head# 2024-02-09T17:57:10.734 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1769df6a:::benchmark_data_smithi179_148205_object39828:head# 2024-02-09T17:57:10.735 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:176ac753:::benchmark_data_smithi179_148205_object100171:head# 2024-02-09T17:57:10.736 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:176b24d2:::benchmark_data_smithi179_148205_object75813:head# 2024-02-09T17:57:10.736 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:176b39b0:::benchmark_data_smithi179_148205_object1288:head# 2024-02-09T17:57:10.738 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:176b7505:::benchmark_data_smithi179_148205_object34912:head# 2024-02-09T17:57:10.738 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:176b7907:::benchmark_data_smithi179_148205_object60750:head# 2024-02-09T17:57:10.738 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:176c5ce7:::benchmark_data_smithi179_148205_object44150:head# 2024-02-09T17:57:10.739 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:176c88d6:::benchmark_data_smithi179_148205_object42398:head# 2024-02-09T17:57:10.739 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:176cf6d6:::benchmark_data_smithi179_148205_object59248:head# 2024-02-09T17:57:10.740 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:176e7656:::benchmark_data_smithi179_148205_object36032:head# 2024-02-09T17:57:10.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:10 smithi099 ceph-mon[39420]: pgmap v2035: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 6.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 360 KiB/s wr, 1.12k op/s; 21384/339138 objects degraded (6.305%) 2024-02-09T17:57:10.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:10 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:57:10.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:10 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:57:10.747 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:176ec385:::benchmark_data_smithi179_148205_object103106:head# 2024-02-09T17:57:10.747 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:176edb6f:::benchmark_data_smithi179_148205_object26480:head# 2024-02-09T17:57:10.748 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:176f342a:::benchmark_data_smithi179_148205_object51515:head# 2024-02-09T17:57:10.748 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:176f4543:::benchmark_data_smithi179_148205_object38292:head# 2024-02-09T17:57:10.748 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1770615a:::benchmark_data_smithi179_148205_object8570:head# 2024-02-09T17:57:10.748 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:177113bf:::benchmark_data_smithi179_148205_object26540:head# 2024-02-09T17:57:10.749 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17729f4f:::benchmark_data_smithi179_148205_object87977:head# 2024-02-09T17:57:10.749 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17736106:::benchmark_data_smithi179_148205_object74103:head# 2024-02-09T17:57:10.749 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17746bdc:::benchmark_data_smithi179_148205_object19735:head# 2024-02-09T17:57:10.749 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1774ba66:::benchmark_data_smithi179_148205_object2473:head# 2024-02-09T17:57:10.749 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1774c5d9:::benchmark_data_smithi179_148205_object87654:head# 2024-02-09T17:57:10.750 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:177515ee:::benchmark_data_smithi179_148205_object100263:head# 2024-02-09T17:57:10.750 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17751d30:::benchmark_data_smithi179_148205_object33716:head# 2024-02-09T17:57:10.750 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:177527f3:::benchmark_data_smithi179_148205_object26386:head# 2024-02-09T17:57:10.750 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17756fbe:::benchmark_data_smithi179_148205_object1033:head# 2024-02-09T17:57:10.750 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1775845d:::benchmark_data_smithi179_148205_object109242:head# 2024-02-09T17:57:10.750 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17766244:::benchmark_data_smithi179_148205_object33643:head# 2024-02-09T17:57:10.751 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17781b6a:::benchmark_data_smithi179_148205_object104425:head# 2024-02-09T17:57:10.751 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1778cbc3:::benchmark_data_smithi179_148205_object49399:head# 2024-02-09T17:57:10.752 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:10.755 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1778fdbe:::benchmark_data_smithi179_148205_object94636:head# 2024-02-09T17:57:10.755 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17795b73:::benchmark_data_smithi179_148205_object55880:head# 2024-02-09T17:57:10.755 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:177a68e9:::benchmark_data_smithi179_148205_object22842:head# 2024-02-09T17:57:10.756 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:177ab1d0:::benchmark_data_smithi179_148205_object99846:head# 2024-02-09T17:57:10.759 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:177b2f29:::benchmark_data_smithi179_148205_object80343:head# 2024-02-09T17:57:10.759 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:177b6853:::benchmark_data_smithi179_148205_object16117:head# 2024-02-09T17:57:10.760 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:177b7f54:::benchmark_data_smithi179_148205_object69466:head# 2024-02-09T17:57:10.760 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:177c3d24:::benchmark_data_smithi179_148205_object92436:head# 2024-02-09T17:57:10.760 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:177d20ba:::benchmark_data_smithi179_148205_object42474:head# 2024-02-09T17:57:10.760 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:177d9ec6:::benchmark_data_smithi179_148205_object87206:head# 2024-02-09T17:57:10.760 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:178039a3:::benchmark_data_smithi179_148205_object57650:head# 2024-02-09T17:57:10.761 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1780af72:::benchmark_data_smithi179_148205_object77751:head# 2024-02-09T17:57:10.761 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17813074:::benchmark_data_smithi179_148205_object84613:head# 2024-02-09T17:57:10.761 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17819bc4:::benchmark_data_smithi179_148205_object106988:head# 2024-02-09T17:57:10.762 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1781b879:::benchmark_data_smithi179_148205_object68561:head# 2024-02-09T17:57:10.762 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1781d88a:::benchmark_data_smithi179_148205_object106786:head# 2024-02-09T17:57:10.762 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17850d79:::benchmark_data_smithi179_148205_object87100:head# 2024-02-09T17:57:10.763 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:178525e7:::benchmark_data_smithi179_148205_object32012:head# 2024-02-09T17:57:10.763 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:178551f3:::benchmark_data_smithi179_148205_object105682:head# 2024-02-09T17:57:10.764 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17859e26:::benchmark_data_smithi179_148205_object71658:head# 2024-02-09T17:57:10.764 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1785d81d:::benchmark_data_smithi179_148205_object80081:head# 2024-02-09T17:57:10.765 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1786842a:::benchmark_data_smithi179_148205_object101688:head# 2024-02-09T17:57:10.765 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1786a6b2:::benchmark_data_smithi179_148205_object33422:head# 2024-02-09T17:57:10.765 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1787fea4:::benchmark_data_smithi179_148205_object45273:head# 2024-02-09T17:57:10.766 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17881c6f:::benchmark_data_smithi179_148205_object47864:head# 2024-02-09T17:57:10.766 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1788708a:::benchmark_data_smithi179_148205_object47257:head# 2024-02-09T17:57:10.767 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1789797d:::benchmark_data_smithi179_148205_object61154:head# 2024-02-09T17:57:10.767 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1789b591:::benchmark_data_smithi179_148205_object84729:head# 2024-02-09T17:57:10.767 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1789ec5b:::benchmark_data_smithi179_148205_object52042:head# 2024-02-09T17:57:10.768 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:178a4141:::benchmark_data_smithi179_148205_object86476:head# 2024-02-09T17:57:10.768 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:178a4fff:::benchmark_data_smithi179_148205_object24992:head# 2024-02-09T17:57:10.769 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:178a9a0d:::benchmark_data_smithi179_148205_object55770:head# 2024-02-09T17:57:10.769 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:178ab210:::benchmark_data_smithi179_148205_object48664:head# 2024-02-09T17:57:10.769 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:178b9b94:::benchmark_data_smithi179_148205_object28953:head# 2024-02-09T17:57:10.770 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:178cdab1:::benchmark_data_smithi179_148205_object14873:head# 2024-02-09T17:57:10.770 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:178ce018:::benchmark_data_smithi179_148205_object38653:head# 2024-02-09T17:57:10.771 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:178cf493:::benchmark_data_smithi179_148205_object19028:head# 2024-02-09T17:57:10.772 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:178d8c57:::benchmark_data_smithi179_148205_object55599:head# 2024-02-09T17:57:10.772 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:178e3be3:::benchmark_data_smithi179_148205_object106566:head# 2024-02-09T17:57:10.772 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:178e9707:::benchmark_data_smithi179_148205_object32939:head# 2024-02-09T17:57:10.773 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:178f2476:::benchmark_data_smithi179_148205_object19101:head# 2024-02-09T17:57:10.773 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:178f57d1:::benchmark_data_smithi179_148205_object351:head# 2024-02-09T17:57:10.774 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1790537f:::benchmark_data_smithi179_148205_object109416:head# 2024-02-09T17:57:10.774 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17910f0d:::benchmark_data_smithi179_148205_object42458:head# 2024-02-09T17:57:10.774 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17911b71:::benchmark_data_smithi179_148205_object46980:head# 2024-02-09T17:57:10.774 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17911e10:::benchmark_data_smithi179_148205_object92479:head# 2024-02-09T17:57:10.775 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1791b010:::benchmark_data_smithi179_148205_object29105:head# 2024-02-09T17:57:10.775 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1791d66c:::benchmark_data_smithi179_148205_object3048:head# 2024-02-09T17:57:10.775 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1791fd1f:::benchmark_data_smithi179_148205_object60419:head# 2024-02-09T17:57:10.776 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1792eeff:::benchmark_data_smithi179_148205_object25111:head# 2024-02-09T17:57:10.776 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1794de58:::benchmark_data_smithi179_148205_object35643:head# 2024-02-09T17:57:10.777 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17956e2b:::benchmark_data_smithi179_148205_object92775:head# 2024-02-09T17:57:10.777 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17994e5d:::benchmark_data_smithi179_148205_object15765:head# 2024-02-09T17:57:10.778 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1799d140:::benchmark_data_smithi179_148205_object74377:head# 2024-02-09T17:57:10.778 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:179af6e0:::benchmark_data_smithi179_148205_object104983:head# 2024-02-09T17:57:10.779 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:179b7fc1:::benchmark_data_smithi179_148205_object67248:head# 2024-02-09T17:57:10.780 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:179c7398:::benchmark_data_smithi179_148205_object79681:head# 2024-02-09T17:57:10.780 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:179dc373:::benchmark_data_smithi179_148205_object67945:head# 2024-02-09T17:57:10.780 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:179df9cc:::benchmark_data_smithi179_148205_object2925:head# 2024-02-09T17:57:10.781 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:179e202b:::benchmark_data_smithi179_148205_object49749:head# 2024-02-09T17:57:10.781 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:179e8f47:::benchmark_data_smithi179_148205_object96919:head# 2024-02-09T17:57:10.782 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:179ee127:::benchmark_data_smithi179_148205_object65840:head# 2024-02-09T17:57:10.782 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17a007e3:::benchmark_data_smithi179_148205_object22616:head# 2024-02-09T17:57:10.782 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17a08b73:::benchmark_data_smithi179_148205_object73104:head# 2024-02-09T17:57:10.783 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17a1e284:::benchmark_data_smithi179_148205_object104276:head# 2024-02-09T17:57:10.783 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17a3345d:::benchmark_data_smithi179_148205_object17428:head# 2024-02-09T17:57:10.784 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17a3de41:::benchmark_data_smithi179_148205_object21249:head# 2024-02-09T17:57:10.784 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17a3f22f:::benchmark_data_smithi179_148205_object107169:head# 2024-02-09T17:57:10.784 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17a485f4:::benchmark_data_smithi179_148205_object54503:head# 2024-02-09T17:57:10.785 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17a4ea2c:::benchmark_data_smithi179_148205_object57591:head# 2024-02-09T17:57:10.785 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17a56def:::benchmark_data_smithi179_148205_object23602:head# 2024-02-09T17:57:10.786 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17a5b774:::benchmark_data_smithi179_148205_object9492:head# 2024-02-09T17:57:10.786 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17a61e83:::benchmark_data_smithi179_148205_object93865:head# 2024-02-09T17:57:10.787 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17a8b22d:::benchmark_data_smithi179_148205_object29304:head# 2024-02-09T17:57:10.787 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17a95d83:::benchmark_data_smithi179_148205_object8294:head# 2024-02-09T17:57:10.788 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:57:10.788 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17a9ca20:::benchmark_data_smithi179_148205_object34156:head# 2024-02-09T17:57:10.788 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17aa5780:::benchmark_data_smithi179_148205_object19988:head# 2024-02-09T17:57:10.789 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17aa8028:::benchmark_data_smithi179_148205_object81055:head# 2024-02-09T17:57:10.789 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17ab7d2c:::benchmark_data_smithi179_148205_object3814:head# 2024-02-09T17:57:10.789 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17ab8f99:::benchmark_data_smithi179_148205_object99125:head# 2024-02-09T17:57:10.790 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17ac47ed:::benchmark_data_smithi179_148205_object99016:head# 2024-02-09T17:57:10.790 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17ac75f3:::benchmark_data_smithi179_148205_object107054:head# 2024-02-09T17:57:10.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:10.792 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:57:10.796 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17ad7dfc:::benchmark_data_smithi179_148205_object30957:head# 2024-02-09T17:57:10.796 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17add35a:::benchmark_data_smithi179_148205_object87197:head# 2024-02-09T17:57:10.796 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17ae607f:::benchmark_data_smithi179_148205_object90837:head# 2024-02-09T17:57:10.796 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17af6ff2:::benchmark_data_smithi179_148205_object9872:head# 2024-02-09T17:57:10.798 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17afa79f:::benchmark_data_smithi179_148205_object87245:head# 2024-02-09T17:57:10.799 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17afd7d9:::benchmark_data_smithi179_148205_object81485:head# 2024-02-09T17:57:10.799 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17b143e4:::benchmark_data_smithi179_148205_object86741:head# 2024-02-09T17:57:10.807 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17b17c44:::benchmark_data_smithi179_148205_object70805:head# 2024-02-09T17:57:10.807 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17b1a39b:::benchmark_data_smithi179_148205_object90234:head# 2024-02-09T17:57:10.807 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17b208ae:::benchmark_data_smithi179_148205_object37800:head# 2024-02-09T17:57:10.807 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17b2bcd6:::benchmark_data_smithi179_148205_object102507:head# 2024-02-09T17:57:10.807 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17b2e198:::benchmark_data_smithi179_148205_object73198:head# 2024-02-09T17:57:10.807 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17b6825a:::benchmark_data_smithi179_148205_object86738:head# 2024-02-09T17:57:10.807 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17b70c8c:::benchmark_data_smithi179_148205_object34948:head# 2024-02-09T17:57:10.807 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17b74979:::benchmark_data_smithi179_148205_object95199:head# 2024-02-09T17:57:10.807 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17b75b3e:::benchmark_data_smithi179_148205_object58585:head# 2024-02-09T17:57:10.808 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17b7c2b7:::benchmark_data_smithi179_148205_object93524:head# 2024-02-09T17:57:10.808 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17b9354f:::benchmark_data_smithi179_148205_object107144:head# 2024-02-09T17:57:10.808 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17b94e5a:::benchmark_data_smithi179_148205_object33350:head# 2024-02-09T17:57:10.808 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17b9c90e:::benchmark_data_smithi179_148205_object74352:head# 2024-02-09T17:57:10.808 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17ba371c:::benchmark_data_smithi179_148205_object88741:head# 2024-02-09T17:57:10.809 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17bacd84:::benchmark_data_smithi179_148205_object58058:head# 2024-02-09T17:57:10.809 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17bbdb7f:::benchmark_data_smithi179_148205_object66784:head# 2024-02-09T17:57:10.809 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17bbec04:::benchmark_data_smithi179_148205_object90267:head# 2024-02-09T17:57:10.809 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17bc20d1:::benchmark_data_smithi179_148205_object84533:head# 2024-02-09T17:57:10.810 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17bc36da:::benchmark_data_smithi179_148205_object26345:head# 2024-02-09T17:57:10.810 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17bcb66b:::benchmark_data_smithi179_148205_object29703:head# 2024-02-09T17:57:10.811 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17bcbd61:::benchmark_data_smithi179_148205_object90589:head# 2024-02-09T17:57:10.811 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17bdf01b:::benchmark_data_smithi179_148205_object28467:head# 2024-02-09T17:57:10.812 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17bdf208:::benchmark_data_smithi179_148205_object35695:head# 2024-02-09T17:57:10.812 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17bdfdad:::benchmark_data_smithi179_148205_object74538:head# 2024-02-09T17:57:10.813 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17bfa99b:::benchmark_data_smithi179_148205_object74378:head# 2024-02-09T17:57:10.813 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17bfb3e9:::benchmark_data_smithi179_148205_object29339:head# 2024-02-09T17:57:10.814 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17c14fae:::benchmark_data_smithi179_148205_object22422:head# 2024-02-09T17:57:10.815 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17c171ac:::benchmark_data_smithi179_148205_object57256:head# 2024-02-09T17:57:10.816 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17c18f52:::benchmark_data_smithi179_148205_object68880:head# 2024-02-09T17:57:10.816 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17c28cbc:::benchmark_data_smithi179_148205_object47784:head# 2024-02-09T17:57:10.817 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17c364d5:::benchmark_data_smithi179_148205_object43804:head# 2024-02-09T17:57:10.817 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17c46e67:::benchmark_data_smithi179_148205_object10362:head# 2024-02-09T17:57:10.818 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17c4b43c:::benchmark_data_smithi179_148205_object28436:head# 2024-02-09T17:57:10.818 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17c4f9d4:::benchmark_data_smithi179_148205_object18983:head# 2024-02-09T17:57:10.819 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17c58065:::benchmark_data_smithi179_148205_object2577:head# 2024-02-09T17:57:10.821 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17c58365:::benchmark_data_smithi179_148205_object45448:head# 2024-02-09T17:57:10.821 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17c67264:::benchmark_data_smithi179_148205_object22250:head# 2024-02-09T17:57:10.822 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17c6f733:::benchmark_data_smithi179_148205_object53852:head# 2024-02-09T17:57:10.822 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17c85056:::benchmark_data_smithi179_148205_object87807:head# 2024-02-09T17:57:10.823 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17caaa2f:::benchmark_data_smithi179_148205_object61261:head# 2024-02-09T17:57:10.824 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17cbce81:::benchmark_data_smithi179_148205_object12413:head# 2024-02-09T17:57:10.824 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17cbead9:::benchmark_data_smithi179_148205_object14521:head# 2024-02-09T17:57:10.825 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17cc961c:::benchmark_data_smithi179_148205_object106762:head# 2024-02-09T17:57:10.829 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17cc96c6:::benchmark_data_smithi179_148205_object26063:head# 2024-02-09T17:57:10.829 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17cdac2e:::benchmark_data_smithi179_148205_object94:head# 2024-02-09T17:57:10.829 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17ce9456:::benchmark_data_smithi179_148205_object109101:head# 2024-02-09T17:57:10.829 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17cec884:::benchmark_data_smithi179_148205_object79612:head# 2024-02-09T17:57:10.829 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17cf05eb:::benchmark_data_smithi179_148205_object48567:head# 2024-02-09T17:57:10.829 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17cf36c6:::benchmark_data_smithi179_148205_object79111:head# 2024-02-09T17:57:10.829 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17d02e8b:::benchmark_data_smithi179_148205_object95614:head# 2024-02-09T17:57:10.829 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17d18237:::benchmark_data_smithi179_148205_object58746:head# 2024-02-09T17:57:10.829 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17d29286:::benchmark_data_smithi179_148205_object74239:head# 2024-02-09T17:57:10.831 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17d2bb6e:::benchmark_data_smithi179_148205_object85427:head# 2024-02-09T17:57:10.831 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17d3158d:::benchmark_data_smithi179_148205_object41809:head# 2024-02-09T17:57:10.831 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17d42289:::benchmark_data_smithi179_148205_object60236:head# 2024-02-09T17:57:10.832 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17d42df6:::benchmark_data_smithi179_148205_object75316:head# 2024-02-09T17:57:10.832 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17d45536:::benchmark_data_smithi179_148205_object93257:head# 2024-02-09T17:57:10.833 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17d4d85d:::benchmark_data_smithi179_148205_object88238:head# 2024-02-09T17:57:10.833 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17d6458e:::benchmark_data_smithi179_148205_object61958:head# 2024-02-09T17:57:10.834 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17d6a0cf:::benchmark_data_smithi179_148205_object97587:head# 2024-02-09T17:57:10.834 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17d6b160:::benchmark_data_smithi179_148205_object29532:head# 2024-02-09T17:57:10.835 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17d72f10:::benchmark_data_smithi179_148205_object98597:head# 2024-02-09T17:57:10.835 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17d938df:::benchmark_data_smithi179_148205_object3668:head# 2024-02-09T17:57:10.835 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17d971de:::benchmark_data_smithi179_148205_object15535:head# 2024-02-09T17:57:10.836 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17d9b1da:::benchmark_data_smithi179_148205_object15397:head# 2024-02-09T17:57:10.837 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17db2df6:::benchmark_data_smithi179_148205_object90551:head# 2024-02-09T17:57:10.841 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17db55f3:::benchmark_data_smithi179_148205_object15787:head# 2024-02-09T17:57:10.841 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17dc196d:::benchmark_data_smithi179_148205_object65219:head# 2024-02-09T17:57:10.841 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17dcac07:::benchmark_data_smithi179_148205_object54624:head# 2024-02-09T17:57:10.841 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17dcc930:::benchmark_data_smithi179_148205_object70864:head# 2024-02-09T17:57:10.841 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17dcf281:::benchmark_data_smithi179_148205_object53468:head# 2024-02-09T17:57:10.841 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17de874b:::benchmark_data_smithi179_148205_object57619:head# 2024-02-09T17:57:10.841 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17dee328:::benchmark_data_smithi179_148205_object92101:head# 2024-02-09T17:57:10.841 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17deeeda:::benchmark_data_smithi179_148205_object47087:head# 2024-02-09T17:57:10.841 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17df2ff3:::benchmark_data_smithi179_148205_object27393:head# 2024-02-09T17:57:10.842 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17e09846:::benchmark_data_smithi179_148205_object106398:head# 2024-02-09T17:57:10.842 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17e0b855:::benchmark_data_smithi179_148205_object100310:head# 2024-02-09T17:57:10.842 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17e1b22e:::benchmark_data_smithi179_148205_object30458:head# 2024-02-09T17:57:10.843 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17e1eedb:::benchmark_data_smithi179_148205_object83685:head# 2024-02-09T17:57:10.843 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17e1f52a:::benchmark_data_smithi179_148205_object108298:head# 2024-02-09T17:57:10.847 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17e447cc:::benchmark_data_smithi179_148205_object21723:head# 2024-02-09T17:57:10.848 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17e50d5d:::benchmark_data_smithi179_148205_object15683:head# 2024-02-09T17:57:10.848 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17e5190d:::benchmark_data_smithi179_148205_object42352:head# 2024-02-09T17:57:10.848 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17e5b517:::benchmark_data_smithi179_148205_object26010:head# 2024-02-09T17:57:10.848 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17e5bcfe:::benchmark_data_smithi179_148205_object42241:head# 2024-02-09T17:57:10.848 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17e60c62:::benchmark_data_smithi179_148205_object77605:head# 2024-02-09T17:57:10.848 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17e73f87:::benchmark_data_smithi179_148205_object98868:head# 2024-02-09T17:57:10.848 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17e8e028:::benchmark_data_smithi179_148205_object89059:head# 2024-02-09T17:57:10.849 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17e94873:::benchmark_data_smithi179_148205_object70559:head# 2024-02-09T17:57:10.849 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17ea0c97:::benchmark_data_smithi179_148205_object22799:head# 2024-02-09T17:57:10.849 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17ea8955:::benchmark_data_smithi179_148205_object104005:head# 2024-02-09T17:57:10.850 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17eacc1a:::benchmark_data_smithi179_148205_object70601:head# 2024-02-09T17:57:10.851 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17eb8894:::benchmark_data_smithi179_148205_object37434:head# 2024-02-09T17:57:10.851 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17ecf46c:::benchmark_data_smithi179_148205_object21276:head# 2024-02-09T17:57:10.851 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17edbeb8:::benchmark_data_smithi179_148205_object22003:head# 2024-02-09T17:57:10.856 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17ee241a:::benchmark_data_smithi179_148205_object88436:head# 2024-02-09T17:57:10.856 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17eef69d:::benchmark_data_smithi179_148205_object84915:head# 2024-02-09T17:57:10.857 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17ef0941:::benchmark_data_smithi179_148205_object97562:head# 2024-02-09T17:57:10.857 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17ef7417:::benchmark_data_smithi179_148205_object78577:head# 2024-02-09T17:57:10.861 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17efb25e:::benchmark_data_smithi179_148205_object17968:head# 2024-02-09T17:57:10.861 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17efd216:::benchmark_data_smithi179_148205_object12025:head# 2024-02-09T17:57:10.861 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17f01abc:::benchmark_data_smithi179_148205_object77438:head# 2024-02-09T17:57:10.861 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17f035c7:::benchmark_data_smithi179_148205_object44074:head# 2024-02-09T17:57:10.861 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17f144f0:::benchmark_data_smithi179_148205_object80025:head# 2024-02-09T17:57:10.861 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17f35fa1:::benchmark_data_smithi179_148205_object96501:head# 2024-02-09T17:57:10.862 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17f42d7a:::benchmark_data_smithi179_148205_object8980:head# 2024-02-09T17:57:10.862 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17f4e6f5:::benchmark_data_smithi179_148205_object73693:head# 2024-02-09T17:57:10.862 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17f4e8f0:::benchmark_data_smithi179_148205_object69249:head# 2024-02-09T17:57:10.862 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17f58638:::benchmark_data_smithi179_148205_object61319:head# 2024-02-09T17:57:10.863 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17f58a3e:::benchmark_data_smithi179_148205_object51583:head# 2024-02-09T17:57:10.863 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17f5f5c0:::benchmark_data_smithi179_148205_object64915:head# 2024-02-09T17:57:10.863 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17f63590:::benchmark_data_smithi179_148205_object8353:head# 2024-02-09T17:57:10.863 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17f65ca9:::benchmark_data_smithi179_148205_object7699:head# 2024-02-09T17:57:10.863 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17f66598:::benchmark_data_smithi179_148205_object63067:head# 2024-02-09T17:57:10.864 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17f6e6c9:::benchmark_data_smithi179_148205_object33464:head# 2024-02-09T17:57:10.864 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17f79ba7:::benchmark_data_smithi179_148205_object109452:head# 2024-02-09T17:57:10.864 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17fa9c7f:::benchmark_data_smithi179_148205_object45225:head# 2024-02-09T17:57:10.865 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17faf9e3:::benchmark_data_smithi179_148205_object82420:head# 2024-02-09T17:57:10.865 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17fb862e:::benchmark_data_smithi179_148205_object54856:head# 2024-02-09T17:57:10.866 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17fb98ba:::benchmark_data_smithi179_148205_object2872:head# 2024-02-09T17:57:10.866 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17fc5cf9:::benchmark_data_smithi179_148205_object16356:head# 2024-02-09T17:57:10.867 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17fcbdff:::benchmark_data_smithi179_148205_object66066:head# 2024-02-09T17:57:10.867 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17fcf5ca:::benchmark_data_smithi179_148205_object89320:head# 2024-02-09T17:57:10.868 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17fd4299:::benchmark_data_smithi179_148205_object45935:head# 2024-02-09T17:57:10.868 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17fd9800:::benchmark_data_smithi179_148205_object26373:head# 2024-02-09T17:57:10.868 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17fdbf8d:::benchmark_data_smithi179_148205_object36513:head# 2024-02-09T17:57:10.869 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17fdd301:::benchmark_data_smithi179_148205_object50159:head# 2024-02-09T17:57:10.869 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17fde6dc:::benchmark_data_smithi179_148205_object50891:head# 2024-02-09T17:57:10.870 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17fe170c:::benchmark_data_smithi179_148205_object94733:head# 2024-02-09T17:57:10.870 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17ff2b4e:::benchmark_data_smithi179_148205_object33118:head# 2024-02-09T17:57:10.870 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17ff2c64:::benchmark_data_smithi179_148205_object29361:head# 2024-02-09T17:57:10.871 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17ff5d45:::benchmark_data_smithi179_148205_object88602:head# 2024-02-09T17:57:10.871 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17ffe287:::benchmark_data_smithi179_148205_object101170:head# 2024-02-09T17:57:10.872 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:17ffef4b:::benchmark_data_smithi179_148205_object48930:head# 2024-02-09T17:57:10.872 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18001750:::benchmark_data_smithi179_148205_object87478:head# 2024-02-09T17:57:10.873 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18005118:::benchmark_data_smithi179_148205_object24833:head# 2024-02-09T17:57:10.873 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1800e829:::benchmark_data_smithi179_148205_object103075:head# 2024-02-09T17:57:10.873 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:180130fe:::benchmark_data_smithi179_148205_object96842:head# 2024-02-09T17:57:10.874 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1801837f:::benchmark_data_smithi179_148205_object46951:head# 2024-02-09T17:57:10.874 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:180207f3:::benchmark_data_smithi179_148205_object50950:head# 2024-02-09T17:57:10.874 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1802c3ac:::benchmark_data_smithi179_148205_object102311:head# 2024-02-09T17:57:10.875 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:180685c5:::benchmark_data_smithi179_148205_object74144:head# 2024-02-09T17:57:10.875 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18076014:::benchmark_data_smithi179_148205_object3298:head# 2024-02-09T17:57:10.876 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1807b4c0:::benchmark_data_smithi179_148205_object61340:head# 2024-02-09T17:57:10.876 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18082348:::benchmark_data_smithi179_148205_object4094:head# 2024-02-09T17:57:10.876 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1808da3f:::benchmark_data_smithi179_148205_object76595:head# 2024-02-09T17:57:10.877 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:180923b0:::benchmark_data_smithi179_148205_object16681:head# 2024-02-09T17:57:10.877 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:180933c0:::benchmark_data_smithi179_148205_object11066:head# 2024-02-09T17:57:10.877 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:180aa7dd:::benchmark_data_smithi179_148205_object74331:head# 2024-02-09T17:57:10.878 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:180acbe9:::benchmark_data_smithi179_148205_object47869:head# 2024-02-09T17:57:10.878 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:180b1d2c:::benchmark_data_smithi179_148205_object29924:head# 2024-02-09T17:57:10.879 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:180b5e68:::benchmark_data_smithi179_148205_object102095:head# 2024-02-09T17:57:10.879 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:180cfe80:::benchmark_data_smithi179_148205_object820:head# 2024-02-09T17:57:10.879 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:180d1585:::benchmark_data_smithi179_148205_object86261:head# 2024-02-09T17:57:10.880 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:180d2d3b:::benchmark_data_smithi179_148205_object67356:head# 2024-02-09T17:57:10.880 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:180d6ae5:::benchmark_data_smithi179_148205_object97711:head# 2024-02-09T17:57:10.881 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:180db7fa:::benchmark_data_smithi179_148205_object43314:head# 2024-02-09T17:57:10.884 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:180dfcfc:::benchmark_data_smithi179_148205_object68806:head# 2024-02-09T17:57:10.884 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:180e4159:::benchmark_data_smithi179_148205_object69244:head# 2024-02-09T17:57:10.884 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:180e5a38:::benchmark_data_smithi179_148205_object55895:head# 2024-02-09T17:57:10.884 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:180f37cf:::benchmark_data_smithi179_148205_object488:head# 2024-02-09T17:57:10.884 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:180f4f60:::benchmark_data_smithi179_148205_object97041:head# 2024-02-09T17:57:10.884 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:180fa37f:::benchmark_data_smithi179_148205_object98530:head# 2024-02-09T17:57:10.884 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:180fe1dc:::benchmark_data_smithi179_148205_object86383:head# 2024-02-09T17:57:10.884 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18106143:::benchmark_data_smithi179_148205_object9558:head# 2024-02-09T17:57:10.885 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18108300:::benchmark_data_smithi179_148205_object109802:head# 2024-02-09T17:57:10.885 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1811e1b0:::benchmark_data_smithi179_148205_object25728:head# 2024-02-09T17:57:10.885 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18123b74:::benchmark_data_smithi179_148205_object28577:head# 2024-02-09T17:57:10.885 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:181278f7:::benchmark_data_smithi179_148205_object70924:head# 2024-02-09T17:57:10.891 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18138867:::benchmark_data_smithi179_148205_object62372:head# 2024-02-09T17:57:10.891 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1813ef4c:::benchmark_data_smithi179_148205_object12942:head# 2024-02-09T17:57:10.891 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18146a32:::benchmark_data_smithi179_148205_object51639:head# 2024-02-09T17:57:10.893 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:57:10.894 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18150b56:::benchmark_data_smithi179_148205_object51578:head# 2024-02-09T17:57:10.894 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1815989d:::benchmark_data_smithi179_148205_object74987:head# 2024-02-09T17:57:10.896 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18175efc:::benchmark_data_smithi179_148205_object75009:head# 2024-02-09T17:57:10.896 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:181810bf:::benchmark_data_smithi179_148205_object63594:head# 2024-02-09T17:57:10.896 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:181828d3:::benchmark_data_smithi179_148205_object32017:head# 2024-02-09T17:57:10.896 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18186ce8:::benchmark_data_smithi179_148205_object71332:head# 2024-02-09T17:57:10.897 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18186ffe:::benchmark_data_smithi179_148205_object64619:head# 2024-02-09T17:57:10.897 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18191bf0:::benchmark_data_smithi179_148205_object99707:head# 2024-02-09T17:57:10.898 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18196008:::benchmark_data_smithi179_148205_object8161:head# 2024-02-09T17:57:10.898 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1819a016:::benchmark_data_smithi179_148205_object31768:head# 2024-02-09T17:57:10.899 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:181a7820:::benchmark_data_smithi179_148205_object39646:head# 2024-02-09T17:57:10.899 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:181b6b21:::benchmark_data_smithi179_148205_object26066:head# 2024-02-09T17:57:10.899 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:181b82ea:::benchmark_data_smithi179_148205_object55108:head# 2024-02-09T17:57:10.900 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:181b8cac:::benchmark_data_smithi179_148205_object43615:head# 2024-02-09T17:57:10.900 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:181bd935:::benchmark_data_smithi179_148205_object72660:head# 2024-02-09T17:57:10.901 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:181cb8e1:::benchmark_data_smithi179_148205_object9347:head# 2024-02-09T17:57:10.901 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:181d1250:::benchmark_data_smithi179_148205_object546:head# 2024-02-09T17:57:10.902 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:181d7a19:::benchmark_data_smithi179_148205_object6706:head# 2024-02-09T17:57:10.903 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:181d9e63:::benchmark_data_smithi179_148205_object65933:head# 2024-02-09T17:57:10.904 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:181e04fe:::benchmark_data_smithi179_148205_object45810:head# 2024-02-09T17:57:10.905 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:181efd93:::benchmark_data_smithi179_148205_object50293:head# 2024-02-09T17:57:10.905 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:181efe0f:::benchmark_data_smithi179_148205_object40216:head# 2024-02-09T17:57:10.906 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:181f13d9:::benchmark_data_smithi179_148205_object74189:head# 2024-02-09T17:57:10.906 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:181f30ed:::benchmark_data_smithi179_148205_object79806:head# 2024-02-09T17:57:10.910 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18202ac1:::benchmark_data_smithi179_148205_object25339:head# 2024-02-09T17:57:10.910 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1820585b:::benchmark_data_smithi179_148205_object40896:head# 2024-02-09T17:57:10.910 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1820ceac:::benchmark_data_smithi179_148205_object96218:head# 2024-02-09T17:57:10.910 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:182246bb:::benchmark_data_smithi179_148205_object42143:head# 2024-02-09T17:57:10.910 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1822d890:::benchmark_data_smithi179_148205_object84608:head# 2024-02-09T17:57:10.910 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18237687:::benchmark_data_smithi179_148205_object28390:head# 2024-02-09T17:57:10.911 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1823d7c0:::benchmark_data_smithi179_148205_object18795:head# 2024-02-09T17:57:10.911 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18249e65:::benchmark_data_smithi179_148205_object108472:head# 2024-02-09T17:57:10.912 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1826133e:::benchmark_data_smithi179_148205_object548:head# 2024-02-09T17:57:10.913 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18271167:::benchmark_data_smithi179_148205_object94812:head# 2024-02-09T17:57:10.913 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18280722:::benchmark_data_smithi179_148205_object10521:head# 2024-02-09T17:57:10.914 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18281a1a:::benchmark_data_smithi179_148205_object20520:head# 2024-02-09T17:57:10.914 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1828d111:::benchmark_data_smithi179_148205_object4678:head# 2024-02-09T17:57:10.915 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:182a33a5:::benchmark_data_smithi179_148205_object79104:head# 2024-02-09T17:57:10.915 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:182a9576:::benchmark_data_smithi179_148205_object63608:head# 2024-02-09T17:57:10.915 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:182b2c19:::benchmark_data_smithi179_148205_object103931:head# 2024-02-09T17:57:10.916 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:182c0e70:::benchmark_data_smithi179_148205_object94440:head# 2024-02-09T17:57:10.916 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:182c9a73:::benchmark_data_smithi179_148205_object25285:head# 2024-02-09T17:57:10.917 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:182d8bfc:::benchmark_data_smithi179_148205_object37082:head# 2024-02-09T17:57:10.924 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:182de895:::benchmark_data_smithi179_148205_object21819:head# 2024-02-09T17:57:10.924 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:182e3111:::benchmark_data_smithi179_148205_object37594:head# 2024-02-09T17:57:10.924 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:182eeafe:::benchmark_data_smithi179_148205_object59821:head# 2024-02-09T17:57:10.924 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:182f5e24:::benchmark_data_smithi179_148205_object81268:head# 2024-02-09T17:57:10.924 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:182f6cbd:::benchmark_data_smithi179_148205_object4521:head# 2024-02-09T17:57:10.924 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:182f7f8e:::benchmark_data_smithi179_148205_object3624:head# 2024-02-09T17:57:10.924 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:183026e0:::benchmark_data_smithi179_148205_object26267:head# 2024-02-09T17:57:10.924 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18303033:::benchmark_data_smithi179_148205_object84922:head# 2024-02-09T17:57:10.925 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1832b374:::benchmark_data_smithi179_148205_object101329:head# 2024-02-09T17:57:10.927 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1832ec8b:::benchmark_data_smithi179_148205_object75430:head# 2024-02-09T17:57:10.927 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18333b99:::benchmark_data_smithi179_148205_object27696:head# 2024-02-09T17:57:10.928 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1833fc52:::benchmark_data_smithi179_148205_object33640:head# 2024-02-09T17:57:10.928 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1834272c:::benchmark_data_smithi179_148205_object21597:head# 2024-02-09T17:57:10.928 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1834b1df:::benchmark_data_smithi179_148205_object23090:head# 2024-02-09T17:57:10.929 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18352040:::benchmark_data_smithi179_148205_object77424:head# 2024-02-09T17:57:10.929 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1836eda0:::benchmark_data_smithi179_148205_object74921:head# 2024-02-09T17:57:10.930 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1837269e:::benchmark_data_smithi179_148205_object74745:head# 2024-02-09T17:57:10.931 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:183734d9:::benchmark_data_smithi179_148205_object78170:head# 2024-02-09T17:57:10.931 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1837a208:::benchmark_data_smithi179_148205_object34220:head# 2024-02-09T17:57:10.932 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18394c25:::benchmark_data_smithi179_148205_object95868:head# 2024-02-09T17:57:10.932 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1839df46:::benchmark_data_smithi179_148205_object99764:head# 2024-02-09T17:57:10.933 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:10.933 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:183a37cf:::benchmark_data_smithi179_148205_object64610:head# 2024-02-09T17:57:10.933 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:183ae0c3:::benchmark_data_smithi179_148205_object76356:head# 2024-02-09T17:57:10.937 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:183afa00:::benchmark_data_smithi179_148205_object18004:head# 2024-02-09T17:57:10.938 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:183bc896:::benchmark_data_smithi179_148205_object20847:head# 2024-02-09T17:57:10.938 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:183c5e22:::benchmark_data_smithi179_148205_object31373:head# 2024-02-09T17:57:10.938 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:183ce12d:::benchmark_data_smithi179_148205_object60095:head# 2024-02-09T17:57:10.952 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:183da06d:::benchmark_data_smithi179_148205_object67281:head# 2024-02-09T17:57:10.953 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:183db0fd:::benchmark_data_smithi179_148205_object14369:head# 2024-02-09T17:57:10.953 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:183e0880:::benchmark_data_smithi179_148205_object44572:head# 2024-02-09T17:57:10.953 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:183fcca5:::benchmark_data_smithi179_148205_object17538:head# 2024-02-09T17:57:10.953 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1840241d:::benchmark_data_smithi179_148205_object19401:head# 2024-02-09T17:57:10.953 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1841cd0b:::benchmark_data_smithi179_148205_object18268:head# 2024-02-09T17:57:10.953 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18421157:::benchmark_data_smithi179_148205_object21439:head# 2024-02-09T17:57:10.953 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1842a299:::benchmark_data_smithi179_148205_object13499:head# 2024-02-09T17:57:10.953 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18430061:::benchmark_data_smithi179_148205_object69650:head# 2024-02-09T17:57:10.953 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1843aed1:::benchmark_data_smithi179_148205_object2374:head# 2024-02-09T17:57:10.953 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:184663d9:::benchmark_data_smithi179_148205_object80456:head# 2024-02-09T17:57:10.953 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:184663e7:::benchmark_data_smithi179_148205_object80844:head# 2024-02-09T17:57:10.953 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:184878f6:::benchmark_data_smithi179_148205_object13847:head# 2024-02-09T17:57:10.953 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18490b3f:::benchmark_data_smithi179_148205_object63562:head# 2024-02-09T17:57:10.953 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:184ae1bd:::benchmark_data_smithi179_148205_object97984:head# 2024-02-09T17:57:10.954 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:184ae6f3:::benchmark_data_smithi179_148205_object82567:head# 2024-02-09T17:57:10.954 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:184b18c8:::benchmark_data_smithi179_148205_object99794:head# 2024-02-09T17:57:10.954 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:184bd37b:::benchmark_data_smithi179_148205_object14000:head# 2024-02-09T17:57:10.954 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:184cb73b:::benchmark_data_smithi179_148205_object42728:head# 2024-02-09T17:57:10.957 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:184f224c:::benchmark_data_smithi179_148205_object33909:head# 2024-02-09T17:57:10.957 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:184f974b:::benchmark_data_smithi179_148205_object27734:head# 2024-02-09T17:57:10.958 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1850c8d4:::benchmark_data_smithi179_148205_object82889:head# 2024-02-09T17:57:10.958 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1851eb05:::benchmark_data_smithi179_148205_object38114:head# 2024-02-09T17:57:10.958 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1852aacd:::benchmark_data_smithi179_148205_object76681:head# 2024-02-09T17:57:10.958 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1852ac4e:::benchmark_data_smithi179_148205_object35427:head# 2024-02-09T17:57:10.958 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:185359a8:::benchmark_data_smithi179_148205_object52173:head# 2024-02-09T17:57:10.958 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:185407b5:::benchmark_data_smithi179_148205_object64683:head# 2024-02-09T17:57:10.959 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1854f781:::benchmark_data_smithi179_148205_object4170:head# 2024-02-09T17:57:10.959 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18552587:::benchmark_data_smithi179_148205_object58824:head# 2024-02-09T17:57:10.961 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:57:10.961 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18552bb0:::benchmark_data_smithi179_148205_object28034:head# 2024-02-09T17:57:10.962 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:185545f5:::benchmark_data_smithi179_148205_object33993:head# 2024-02-09T17:57:10.962 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18557824:::benchmark_data_smithi179_148205_object24620:head# 2024-02-09T17:57:10.962 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1855b9c3:::benchmark_data_smithi179_148205_object13161:head# 2024-02-09T17:57:10.966 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18562b79:::benchmark_data_smithi179_148205_object34492:head# 2024-02-09T17:57:10.967 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18567fd1:::benchmark_data_smithi179_148205_object31358:head# 2024-02-09T17:57:10.967 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1856b870:::benchmark_data_smithi179_148205_object4333:head# 2024-02-09T17:57:10.967 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1857f933:::benchmark_data_smithi179_148205_object46791:head# 2024-02-09T17:57:10.967 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18583e42:::benchmark_data_smithi179_148205_object102716:head# 2024-02-09T17:57:10.971 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18590b76:::benchmark_data_smithi179_148205_object82212:head# 2024-02-09T17:57:10.971 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:185947a6:::benchmark_data_smithi179_148205_object99394:head# 2024-02-09T17:57:10.971 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:185a25c8:::benchmark_data_smithi179_148205_object58899:head# 2024-02-09T17:57:10.971 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:185afb58:::benchmark_data_smithi179_148205_object76948:head# 2024-02-09T17:57:10.971 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:185b6ea5:::benchmark_data_smithi179_148205_object38035:head# 2024-02-09T17:57:10.972 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:185bfaae:::benchmark_data_smithi179_148205_object37418:head# 2024-02-09T17:57:10.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:10.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:57:10.973 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:185d12b9:::benchmark_data_smithi179_148205_object97708:head# 2024-02-09T17:57:10.974 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:185dae90:::benchmark_data_smithi179_148205_object72052:head# 2024-02-09T17:57:10.975 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:185eda89:::benchmark_data_smithi179_148205_object20695:head# 2024-02-09T17:57:10.975 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:185f6242:::benchmark_data_smithi179_148205_object96799:head# 2024-02-09T17:57:10.976 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:185fade1:::benchmark_data_smithi179_148205_object51209:head# 2024-02-09T17:57:10.976 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:185ffb25:::benchmark_data_smithi179_148205_object61633:head# 2024-02-09T17:57:10.977 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1860705c:::benchmark_data_smithi179_148205_object98790:head# 2024-02-09T17:57:10.977 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:186116f7:::benchmark_data_smithi179_148205_object72112:head# 2024-02-09T17:57:10.978 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18621809:::benchmark_data_smithi179_148205_object56105:head# 2024-02-09T17:57:10.978 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18638b28:::benchmark_data_smithi179_148205_object87018:head# 2024-02-09T17:57:10.978 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1863b7b9:::benchmark_data_smithi179_148205_object90352:head# 2024-02-09T17:57:10.984 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1863d7f9:::benchmark_data_smithi179_148205_object69533:head# 2024-02-09T17:57:10.985 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:186465ee:::benchmark_data_smithi179_148205_object97567:head# 2024-02-09T17:57:10.986 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1864f75f:::benchmark_data_smithi179_148205_object48787:head# 2024-02-09T17:57:10.986 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1864f789:::benchmark_data_smithi179_148205_object23443:head# 2024-02-09T17:57:10.987 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1865265e:::benchmark_data_smithi179_148205_object85466:head# 2024-02-09T17:57:10.989 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-09T17:57:10.992 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1865a0ab:::benchmark_data_smithi179_148205_object39924:head# 2024-02-09T17:57:10.992 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1865d50b:::benchmark_data_smithi179_148205_object70725:head# 2024-02-09T17:57:10.992 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1866d94c:::benchmark_data_smithi179_148205_object103220:head# 2024-02-09T17:57:10.992 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18684065:::benchmark_data_smithi179_148205_object99470:head# 2024-02-09T17:57:10.992 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1868512a:::benchmark_data_smithi179_148205_object22551:head# 2024-02-09T17:57:10.992 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18697866:::benchmark_data_smithi179_148205_object76852:head# 2024-02-09T17:57:10.992 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:186a700a:::benchmark_data_smithi179_148205_object34667:head# 2024-02-09T17:57:10.996 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:186aca31:::benchmark_data_smithi179_148205_object2278:head# 2024-02-09T17:57:10.996 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:186ae521:::benchmark_data_smithi179_148205_object40115:head# 2024-02-09T17:57:10.997 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:186b4475:::benchmark_data_smithi179_148205_object73339:head# 2024-02-09T17:57:10.997 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:186b965b:::benchmark_data_smithi179_148205_object1621:head# 2024-02-09T17:57:11.000 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:186bb3c8:::benchmark_data_smithi179_148205_object9813:head# 2024-02-09T17:57:11.000 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:186d7099:::benchmark_data_smithi179_148205_object12967:head# 2024-02-09T17:57:11.000 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:186e5ac6:::benchmark_data_smithi179_148205_object84094:head# 2024-02-09T17:57:11.000 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:186e9073:::benchmark_data_smithi179_148205_object98884:head# 2024-02-09T17:57:11.001 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:186ed0a7:::benchmark_data_smithi179_148205_object105034:head# 2024-02-09T17:57:11.001 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:186f1305:::benchmark_data_smithi179_148205_object57247:head# 2024-02-09T17:57:11.001 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18705510:::benchmark_data_smithi179_148205_object32490:head# 2024-02-09T17:57:11.002 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18713635:::benchmark_data_smithi179_148205_object13031:head# 2024-02-09T17:57:11.003 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1871eac9:::benchmark_data_smithi179_148205_object23437:head# 2024-02-09T17:57:11.006 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1872293f:::benchmark_data_smithi179_148205_object75589:head# 2024-02-09T17:57:11.006 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18723218:::benchmark_data_smithi179_148205_object16779:head# 2024-02-09T17:57:11.006 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1872e5ee:::benchmark_data_smithi179_148205_object4345:head# 2024-02-09T17:57:11.007 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1872f1e7:::benchmark_data_smithi179_148205_object37104:head# 2024-02-09T17:57:11.007 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:187399e7:::benchmark_data_smithi179_148205_object109702:head# 2024-02-09T17:57:11.007 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1873b5a2:::benchmark_data_smithi179_148205_object59687:head# 2024-02-09T17:57:11.008 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1874daf6:::benchmark_data_smithi179_148205_object29040:head# 2024-02-09T17:57:11.008 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18750918:::benchmark_data_smithi179_148205_object103661:head# 2024-02-09T17:57:11.009 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18764168:::benchmark_data_smithi179_148205_object45989:head# 2024-02-09T17:57:11.009 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:187767e3:::benchmark_data_smithi179_148205_object108355:head# 2024-02-09T17:57:11.009 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:187887d5:::benchmark_data_smithi179_148205_object615:head# 2024-02-09T17:57:11.010 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:187924e7:::benchmark_data_smithi179_148205_object109128:head# 2024-02-09T17:57:11.010 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:187950a3:::benchmark_data_smithi179_148205_object66157:head# 2024-02-09T17:57:11.011 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18797666:::benchmark_data_smithi179_148205_object85044:head# 2024-02-09T17:57:11.011 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:187a614e:::benchmark_data_smithi179_148205_object109184:head# 2024-02-09T17:57:11.012 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:187aaa95:::benchmark_data_smithi179_148205_object27679:head# 2024-02-09T17:57:11.012 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:187adc14:::benchmark_data_smithi179_148205_object102159:head# 2024-02-09T17:57:11.012 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:187b50c9:::benchmark_data_smithi179_148205_object17673:head# 2024-02-09T17:57:11.013 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:187def12:::benchmark_data_smithi179_148205_object14246:head# 2024-02-09T17:57:11.013 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:187ea22d:::benchmark_data_smithi179_148205_object18222:head# 2024-02-09T17:57:11.014 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:187ed942:::benchmark_data_smithi179_148205_object98259:head# 2024-02-09T17:57:11.014 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:187fedfe:::benchmark_data_smithi179_148205_object32691:head# 2024-02-09T17:57:11.015 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:188079af:::benchmark_data_smithi179_148205_object67168:head# 2024-02-09T17:57:11.015 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18817bdd:::benchmark_data_smithi179_148205_object74562:head# 2024-02-09T17:57:11.017 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18820b30:::benchmark_data_smithi179_148205_object37888:head# 2024-02-09T17:57:11.020 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:188273ac:::benchmark_data_smithi179_148205_object78864:head# 2024-02-09T17:57:11.021 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18833dad:::benchmark_data_smithi179_148205_object83186:head# 2024-02-09T17:57:11.021 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18841d0c:::benchmark_data_smithi179_148205_object909:head# 2024-02-09T17:57:11.021 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18843529:::benchmark_data_smithi179_148205_object28855:head# 2024-02-09T17:57:11.022 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18846811:::benchmark_data_smithi179_148205_object16865:head# 2024-02-09T17:57:11.022 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:188486b8:::benchmark_data_smithi179_148205_object17986:head# 2024-02-09T17:57:11.022 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1884d4a2:::benchmark_data_smithi179_148205_object25845:head# 2024-02-09T17:57:11.023 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1885eadf:::benchmark_data_smithi179_148205_object102028:head# 2024-02-09T17:57:11.023 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1885f9d9:::benchmark_data_smithi179_148205_object95773:head# 2024-02-09T17:57:11.023 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18863f0e:::benchmark_data_smithi179_148205_object104155:head# 2024-02-09T17:57:11.027 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:188671e1:::benchmark_data_smithi179_148205_object81460:head# 2024-02-09T17:57:11.027 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1886b417:::benchmark_data_smithi179_148205_object62096:head# 2024-02-09T17:57:11.027 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1886e6f1:::benchmark_data_smithi179_148205_object88312:head# 2024-02-09T17:57:11.028 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18872ae5:::benchmark_data_smithi179_148205_object108805:head# 2024-02-09T17:57:11.028 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:188848fc:::benchmark_data_smithi179_148205_object22393:head# 2024-02-09T17:57:11.028 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:188874fc:::benchmark_data_smithi179_148205_object2766:head# 2024-02-09T17:57:11.028 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1889887e:::benchmark_data_smithi179_148205_object80724:head# 2024-02-09T17:57:11.029 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1889e5e8:::benchmark_data_smithi179_148205_object83165:head# 2024-02-09T17:57:11.029 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1889f687:::benchmark_data_smithi179_148205_object36781:head# 2024-02-09T17:57:11.029 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:188a0625:::benchmark_data_smithi179_148205_object97644:head# 2024-02-09T17:57:11.029 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:188a3692:::benchmark_data_smithi179_148205_object43512:head# 2024-02-09T17:57:11.031 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:188a974d:::benchmark_data_smithi179_148205_object108601:head# 2024-02-09T17:57:11.031 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:188ac9cb:::benchmark_data_smithi179_148205_object83316:head# 2024-02-09T17:57:11.031 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:188cd356:::benchmark_data_smithi179_148205_object64271:head# 2024-02-09T17:57:11.031 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:188e4ece:::benchmark_data_smithi179_148205_object13866:head# 2024-02-09T17:57:11.035 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:188ea098:::benchmark_data_smithi179_148205_object54855:head# 2024-02-09T17:57:11.035 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:188f2744:::benchmark_data_smithi179_148205_object13882:head# 2024-02-09T17:57:11.035 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:188f3791:::benchmark_data_smithi179_148205_object2674:head# 2024-02-09T17:57:11.035 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18905192:::benchmark_data_smithi179_148205_object74954:head# 2024-02-09T17:57:11.035 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1890dd9c:::benchmark_data_smithi179_148205_object75087:head# 2024-02-09T17:57:11.036 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1890e41f:::benchmark_data_smithi179_148205_object29945:head# 2024-02-09T17:57:11.036 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:189190e8:::benchmark_data_smithi179_148205_object99224:head# 2024-02-09T17:57:11.037 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1891ad05:::benchmark_data_smithi179_148205_object30873:head# 2024-02-09T17:57:11.037 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18927253:::benchmark_data_smithi179_148205_object60385:head# 2024-02-09T17:57:11.038 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1892b5f5:::benchmark_data_smithi179_148205_object12486:head# 2024-02-09T17:57:11.038 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1892f085:::benchmark_data_smithi179_148205_object14478:head# 2024-02-09T17:57:11.039 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18941d41:::benchmark_data_smithi179_148205_object144:head# 2024-02-09T17:57:11.039 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1897a710:::benchmark_data_smithi179_148205_object36173:head# 2024-02-09T17:57:11.039 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1897e162:::benchmark_data_smithi179_148205_object75994:head# 2024-02-09T17:57:11.040 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1898968c:::benchmark_data_smithi179_148205_object106678:head# 2024-02-09T17:57:11.040 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18999900:::benchmark_data_smithi179_148205_object93114:head# 2024-02-09T17:57:11.041 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:189b6e81:::benchmark_data_smithi179_148205_object2053:head# 2024-02-09T17:57:11.041 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:189bd4e9:::benchmark_data_smithi179_148205_object105369:head# 2024-02-09T17:57:11.042 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:189bde72:::benchmark_data_smithi179_148205_object11858:head# 2024-02-09T17:57:11.042 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:189c284f:::benchmark_data_smithi179_148205_object23748:head# 2024-02-09T17:57:11.043 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:189c766c:::benchmark_data_smithi179_148205_object28353:head# 2024-02-09T17:57:11.043 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:189c8b80:::benchmark_data_smithi179_148205_object17981:head# 2024-02-09T17:57:11.044 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:189ee248:::benchmark_data_smithi179_148205_object77909:head# 2024-02-09T17:57:11.044 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:189fe74b:::benchmark_data_smithi179_148205_object94700:head# 2024-02-09T17:57:11.044 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18a06072:::benchmark_data_smithi179_148205_object82044:head# 2024-02-09T17:57:11.045 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18a163e2:::benchmark_data_smithi179_148205_object51167:head# 2024-02-09T17:57:11.045 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18a20c69:::benchmark_data_smithi179_148205_object13323:head# 2024-02-09T17:57:11.046 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18a2beab:::benchmark_data_smithi179_148205_object56370:head# 2024-02-09T17:57:11.046 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18a2fdcf:::benchmark_data_smithi179_148205_object101625:head# 2024-02-09T17:57:11.047 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18a3e072:::benchmark_data_smithi179_148205_object10624:head# 2024-02-09T17:57:11.047 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18a3e593:::benchmark_data_smithi179_148205_object9509:head# 2024-02-09T17:57:11.048 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18a528d6:::benchmark_data_smithi179_148205_object4269:head# 2024-02-09T17:57:11.048 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18a59063:::benchmark_data_smithi179_148205_object63590:head# 2024-02-09T17:57:11.049 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18a6083d:::benchmark_data_smithi179_148205_object71077:head# 2024-02-09T17:57:11.054 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18a6fce5:::benchmark_data_smithi179_148205_object24840:head# 2024-02-09T17:57:11.056 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18a716ce:::benchmark_data_smithi179_148205_object39410:head# 2024-02-09T17:57:11.056 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18a72504:::benchmark_data_smithi179_148205_object88357:head# 2024-02-09T17:57:11.062 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18a78cd9:::benchmark_data_smithi179_148205_object88361:head# 2024-02-09T17:57:11.062 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18a7c6e5:::benchmark_data_smithi179_148205_object99312:head# 2024-02-09T17:57:11.062 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18a7cc3b:::benchmark_data_smithi179_148205_object82026:head# 2024-02-09T17:57:11.062 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18a84edd:::benchmark_data_smithi179_148205_object43964:head# 2024-02-09T17:57:11.062 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18a8de8d:::benchmark_data_smithi179_148205_object46093:head# 2024-02-09T17:57:11.062 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18a8efe6:::benchmark_data_smithi179_148205_object61069:head# 2024-02-09T17:57:11.062 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18a8f35a:::benchmark_data_smithi179_148205_object62457:head# 2024-02-09T17:57:11.062 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18aa0183:::benchmark_data_smithi179_148205_object17816:head# 2024-02-09T17:57:11.062 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18aa917d:::benchmark_data_smithi179_148205_object66264:head# 2024-02-09T17:57:11.062 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18aa94b6:::benchmark_data_smithi179_148205_object71209:head# 2024-02-09T17:57:11.062 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18ac35fd:::benchmark_data_smithi179_148205_object44785:head# 2024-02-09T17:57:11.066 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18adad27:::benchmark_data_smithi179_148205_object51052:head# 2024-02-09T17:57:11.066 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18adc138:::benchmark_data_smithi179_148205_object15984:head# 2024-02-09T17:57:11.069 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18ade227:::benchmark_data_smithi179_148205_object20679:head# 2024-02-09T17:57:11.070 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18ae1cbf:::benchmark_data_smithi179_148205_object83442:head# 2024-02-09T17:57:11.070 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18af4632:::benchmark_data_smithi179_148205_object52249:head# 2024-02-09T17:57:11.070 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18b18f44:::benchmark_data_smithi179_148205_object32162:head# 2024-02-09T17:57:11.070 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18b358d6:::benchmark_data_smithi179_148205_object14458:head# 2024-02-09T17:57:11.070 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18b4143d:::benchmark_data_smithi179_148205_object77189:head# 2024-02-09T17:57:11.071 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18b478c8:::benchmark_data_smithi179_148205_object44711:head# 2024-02-09T17:57:11.071 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18b48357:::benchmark_data_smithi179_148205_object49373:head# 2024-02-09T17:57:11.072 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18b4e9da:::benchmark_data_smithi179_148205_object81666:head# 2024-02-09T17:57:11.073 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:57:11.075 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18b51da6:::benchmark_data_smithi179_148205_object96495:head# 2024-02-09T17:57:11.075 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18b652d1:::benchmark_data_smithi179_148205_object95281:head# 2024-02-09T17:57:11.075 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18b7ea8c:::benchmark_data_smithi179_148205_object55970:head# 2024-02-09T17:57:11.075 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18b8cb4d:::benchmark_data_smithi179_148205_object3791:head# 2024-02-09T17:57:11.075 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18b92b1e:::benchmark_data_smithi179_148205_object20774:head# 2024-02-09T17:57:11.075 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18b9f909:::benchmark_data_smithi179_148205_object73452:head# 2024-02-09T17:57:11.075 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18ba08bf:::benchmark_data_smithi179_148205_object50466:head# 2024-02-09T17:57:11.075 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18bb0726:::benchmark_data_smithi179_148205_object32634:head# 2024-02-09T17:57:11.076 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18bb5003:::benchmark_data_smithi179_148205_object47185:head# 2024-02-09T17:57:11.077 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18bb5065:::benchmark_data_smithi179_148205_object108495:head# 2024-02-09T17:57:11.077 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18bc1a9b:::benchmark_data_smithi179_148205_object100690:head# 2024-02-09T17:57:11.077 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18bc8d1a:::benchmark_data_smithi179_148205_object28916:head# 2024-02-09T17:57:11.077 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18bcd3c6:::benchmark_data_smithi179_148205_object105233:head# 2024-02-09T17:57:11.077 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18bcd3f3:::benchmark_data_smithi179_148205_object76178:head# 2024-02-09T17:57:11.078 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18bcfd00:::benchmark_data_smithi179_148205_object107066:head# 2024-02-09T17:57:11.078 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18bd536d:::benchmark_data_smithi179_148205_object44827:head# 2024-02-09T17:57:11.078 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18bda184:::benchmark_data_smithi179_148205_object58116:head# 2024-02-09T17:57:11.079 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18bdbd73:::benchmark_data_smithi179_148205_object67024:head# 2024-02-09T17:57:11.083 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18bdeadd:::benchmark_data_smithi179_148205_object44580:head# 2024-02-09T17:57:11.083 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18bdf1a2:::benchmark_data_smithi179_148205_object4245:head# 2024-02-09T17:57:11.083 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18c00895:::benchmark_data_smithi179_148205_object35696:head# 2024-02-09T17:57:11.083 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18c0c54b:::benchmark_data_smithi179_148205_object80867:head# 2024-02-09T17:57:11.083 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18c0ee35:::benchmark_data_smithi179_148205_object24531:head# 2024-02-09T17:57:11.083 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18c0f1c8:::benchmark_data_smithi179_148205_object76296:head# 2024-02-09T17:57:11.084 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18c13b52:::benchmark_data_smithi179_148205_object84737:head# 2024-02-09T17:57:11.084 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18c24e05:::benchmark_data_smithi179_148205_object56897:head# 2024-02-09T17:57:11.084 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18c2ace0:::benchmark_data_smithi179_148205_object7631:head# 2024-02-09T17:57:11.084 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18c2f714:::benchmark_data_smithi179_148205_object94542:head# 2024-02-09T17:57:11.084 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18c41347:::benchmark_data_smithi179_148205_object89766:head# 2024-02-09T17:57:11.084 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18c42808:::benchmark_data_smithi179_148205_object81997:head# 2024-02-09T17:57:11.085 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18c4bb84:::benchmark_data_smithi179_148205_object95798:head# 2024-02-09T17:57:11.085 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18c50c94:::benchmark_data_smithi179_148205_object87607:head# 2024-02-09T17:57:11.085 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18c58162:::benchmark_data_smithi179_148205_object41040:head# 2024-02-09T17:57:11.086 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18c5a36b:::benchmark_data_smithi179_148205_object74212:head# 2024-02-09T17:57:11.086 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18c64abe:::benchmark_data_smithi179_148205_object90935:head# 2024-02-09T17:57:11.087 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18c695e3:::benchmark_data_smithi179_148205_object93588:head# 2024-02-09T17:57:11.087 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18c7f0a0:::benchmark_data_smithi179_148205_object29606:head# 2024-02-09T17:57:11.088 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18c89b29:::benchmark_data_smithi179_148205_object66150:head# 2024-02-09T17:57:11.089 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18c8c689:::benchmark_data_smithi179_148205_object79521:head# 2024-02-09T17:57:11.089 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18c998a9:::benchmark_data_smithi179_148205_object31429:head# 2024-02-09T17:57:11.089 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18ca9acf:::benchmark_data_smithi179_148205_object12523:head# 2024-02-09T17:57:11.090 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18cb12f6:::benchmark_data_smithi179_148205_object35754:head# 2024-02-09T17:57:11.090 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18cb82f1:::benchmark_data_smithi179_148205_object67848:head# 2024-02-09T17:57:11.091 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18cc4a4e:::benchmark_data_smithi179_148205_object76286:head# 2024-02-09T17:57:11.091 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18cc4f12:::benchmark_data_smithi179_148205_object103092:head# 2024-02-09T17:57:11.091 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18ccbcc0:::benchmark_data_smithi179_148205_object78689:head# 2024-02-09T17:57:11.092 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18cd556a:::benchmark_data_smithi179_148205_object52805:head# 2024-02-09T17:57:11.096 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18cda2b5:::benchmark_data_smithi179_148205_object22529:head# 2024-02-09T17:57:11.096 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18ce5ccf:::benchmark_data_smithi179_148205_object107080:head# 2024-02-09T17:57:11.096 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18cea345:::benchmark_data_smithi179_148205_object31789:head# 2024-02-09T17:57:11.096 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18cfccdf:::benchmark_data_smithi179_148205_object63968:head# 2024-02-09T17:57:11.096 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18d0734b:::benchmark_data_smithi179_148205_object64686:head# 2024-02-09T17:57:11.096 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18d0ff89:::benchmark_data_smithi179_148205_object1507:head# 2024-02-09T17:57:11.097 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18d13575:::benchmark_data_smithi179_148205_object54939:head# 2024-02-09T17:57:11.097 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18d1b027:::benchmark_data_smithi179_148205_object16587:head# 2024-02-09T17:57:11.097 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18d1deab:::benchmark_data_smithi179_148205_object91673:head# 2024-02-09T17:57:11.098 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18d23c61:::benchmark_data_smithi179_148205_object77956:head# 2024-02-09T17:57:11.098 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18d295d9:::benchmark_data_smithi179_148205_object80889:head# 2024-02-09T17:57:11.098 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18d30bf6:::benchmark_data_smithi179_148205_object101879:head# 2024-02-09T17:57:11.099 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18d31bce:::benchmark_data_smithi179_148205_object51889:head# 2024-02-09T17:57:11.100 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18d674ba:::benchmark_data_smithi179_148205_object49260:head# 2024-02-09T17:57:11.100 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18d7aec3:::benchmark_data_smithi179_148205_object81811:head# 2024-02-09T17:57:11.100 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18d8186e:::benchmark_data_smithi179_148205_object31503:head# 2024-02-09T17:57:11.104 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18d825b3:::benchmark_data_smithi179_148205_object22067:head# 2024-02-09T17:57:11.104 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18d84d89:::benchmark_data_smithi179_148205_object16290:head# 2024-02-09T17:57:11.106 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18d9a644:::benchmark_data_smithi179_148205_object52776:head# 2024-02-09T17:57:11.107 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18da29f7:::benchmark_data_smithi179_148205_object24669:head# 2024-02-09T17:57:11.107 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18da4645:::benchmark_data_smithi179_148205_object74155:head# 2024-02-09T17:57:11.107 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18daa348:::benchmark_data_smithi179_148205_object10365:head# 2024-02-09T17:57:11.109 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18dacb9c:::benchmark_data_smithi179_148205_object76189:head# 2024-02-09T17:57:11.109 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18db470a:::benchmark_data_smithi179_148205_object34275:head# 2024-02-09T17:57:11.109 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18dc5be6:::benchmark_data_smithi179_148205_object32698:head# 2024-02-09T17:57:11.109 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18dd8d35:::benchmark_data_smithi179_148205_object4502:head# 2024-02-09T17:57:11.109 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18dda735:::benchmark_data_smithi179_148205_object102554:head# 2024-02-09T17:57:11.111 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18de7cdd:::benchmark_data_smithi179_148205_object43506:head# 2024-02-09T17:57:11.112 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18df02b1:::benchmark_data_smithi179_148205_object49727:head# 2024-02-09T17:57:11.112 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18df27da:::benchmark_data_smithi179_148205_object101942:head# 2024-02-09T17:57:11.112 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18df28ca:::benchmark_data_smithi179_148205_object15988:head# 2024-02-09T17:57:11.113 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18df2931:::benchmark_data_smithi179_148205_object39543:head# 2024-02-09T17:57:11.113 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18df6840:::benchmark_data_smithi179_148205_object63107:head# 2024-02-09T17:57:11.121 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:11.124 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18df8b96:::benchmark_data_smithi179_148205_object80685:head# 2024-02-09T17:57:11.124 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18e01e0e:::benchmark_data_smithi179_148205_object88161:head# 2024-02-09T17:57:11.124 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18e136a4:::benchmark_data_smithi179_148205_object1852:head# 2024-02-09T17:57:11.124 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18e234ab:::benchmark_data_smithi179_148205_object90210:head# 2024-02-09T17:57:11.124 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18e2aa31:::benchmark_data_smithi179_148205_object56202:head# 2024-02-09T17:57:11.126 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18e32fc3:::benchmark_data_smithi179_148205_object1553:head# 2024-02-09T17:57:11.126 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18e39bdf:::benchmark_data_smithi179_148205_object27770:head# 2024-02-09T17:57:11.130 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18e4660d:::benchmark_data_smithi179_148205_object42152:head# 2024-02-09T17:57:11.130 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18e48cfc:::benchmark_data_smithi179_148205_object68012:head# 2024-02-09T17:57:11.130 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18e49a51:::benchmark_data_smithi179_148205_object83568:head# 2024-02-09T17:57:11.130 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18e4a4f4:::benchmark_data_smithi179_148205_object68517:head# 2024-02-09T17:57:11.132 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18e4ad58:::benchmark_data_smithi179_148205_object5453:head# 2024-02-09T17:57:11.133 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18e4b1e0:::benchmark_data_smithi179_148205_object41804:head# 2024-02-09T17:57:11.134 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18e5500c:::benchmark_data_smithi179_148205_object109366:head# 2024-02-09T17:57:11.134 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18e5db8a:::benchmark_data_smithi179_148205_object18426:head# 2024-02-09T17:57:11.134 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18e63417:::benchmark_data_smithi179_148205_object84645:head# 2024-02-09T17:57:11.135 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18e6bd0f:::benchmark_data_smithi179_148205_object104247:head# 2024-02-09T17:57:11.135 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18e837ee:::benchmark_data_smithi179_148205_object38361:head# 2024-02-09T17:57:11.135 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18e938c6:::benchmark_data_smithi179_148205_object97572:head# 2024-02-09T17:57:11.140 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18e9b867:::benchmark_data_smithi179_148205_object19665:head# 2024-02-09T17:57:11.140 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18e9bdc9:::benchmark_data_smithi179_148205_object87638:head# 2024-02-09T17:57:11.140 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18ea371b:::benchmark_data_smithi179_148205_object82585:head# 2024-02-09T17:57:11.140 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18ea8385:::benchmark_data_smithi179_148205_object91052:head# 2024-02-09T17:57:11.141 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18eaaf28:::benchmark_data_smithi179_148205_object80797:head# 2024-02-09T17:57:11.143 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18eb7200:::benchmark_data_smithi179_148205_object26037:head# 2024-02-09T17:57:11.143 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18eb73d9:::benchmark_data_smithi179_148205_object70398:head# 2024-02-09T17:57:11.144 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18ebff87:::benchmark_data_smithi179_148205_object76960:head# 2024-02-09T17:57:11.144 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18ec9105:::benchmark_data_smithi179_148205_object63044:head# 2024-02-09T17:57:11.144 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18ecafb3:::benchmark_data_smithi179_148205_object85544:head# 2024-02-09T17:57:11.145 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18ecf122:::benchmark_data_smithi179_148205_object78256:head# 2024-02-09T17:57:11.145 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18edbf3c:::benchmark_data_smithi179_148205_object3955:head# 2024-02-09T17:57:11.146 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18ede791:::benchmark_data_smithi179_148205_object10782:head# 2024-02-09T17:57:11.146 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18ee1cb3:::benchmark_data_smithi179_148205_object48150:head# 2024-02-09T17:57:11.147 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18ee2445:::benchmark_data_smithi179_148205_object28206:head# 2024-02-09T17:57:11.151 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18ef48e2:::benchmark_data_smithi179_148205_object97367:head# 2024-02-09T17:57:11.151 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18efd52e:::benchmark_data_smithi179_148205_object101749:head# 2024-02-09T17:57:11.151 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18f0046e:::benchmark_data_smithi179_148205_object43668:head# 2024-02-09T17:57:11.151 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18f16029:::benchmark_data_smithi179_148205_object78646:head# 2024-02-09T17:57:11.151 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18f188f1:::benchmark_data_smithi179_148205_object100750:head# 2024-02-09T17:57:11.151 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18f2f257:::benchmark_data_smithi179_148205_object89743:head# 2024-02-09T17:57:11.151 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18f3fb84:::benchmark_data_smithi179_148205_object64625:head# 2024-02-09T17:57:11.152 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18f44db5:::benchmark_data_smithi179_148205_object17998:head# 2024-02-09T17:57:11.152 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18f460b1:::benchmark_data_smithi179_148205_object14297:head# 2024-02-09T17:57:11.152 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18f4ce39:::benchmark_data_smithi179_148205_object27481:head# 2024-02-09T17:57:11.152 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18f540f4:::benchmark_data_smithi179_148205_object84038:head# 2024-02-09T17:57:11.163 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18f56575:::benchmark_data_smithi179_148205_object108672:head# 2024-02-09T17:57:11.164 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18f5fbf5:::benchmark_data_smithi179_148205_object20810:head# 2024-02-09T17:57:11.168 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18f845dc:::benchmark_data_smithi179_148205_object28807:head# 2024-02-09T17:57:11.168 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18f8fc42:::benchmark_data_smithi179_148205_object25116:head# 2024-02-09T17:57:11.169 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18f8fd70:::benchmark_data_smithi179_148205_object1481:head# 2024-02-09T17:57:11.169 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18f96aac:::benchmark_data_smithi179_148205_object107062:head# 2024-02-09T17:57:11.170 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18f9a5c4:::benchmark_data_smithi179_148205_object107500:head# 2024-02-09T17:57:11.173 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18f9fc07:::benchmark_data_smithi179_148205_object49173:head# 2024-02-09T17:57:11.173 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18fa83b9:::benchmark_data_smithi179_148205_object39838:head# 2024-02-09T17:57:11.173 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18fb5d23:::benchmark_data_smithi179_148205_object48204:head# 2024-02-09T17:57:11.173 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18fb5f61:::benchmark_data_smithi179_148205_object2521:head# 2024-02-09T17:57:11.173 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18fb885a:::benchmark_data_smithi179_148205_object23182:head# 2024-02-09T17:57:11.173 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18fc1088:::benchmark_data_smithi179_148205_object20675:head# 2024-02-09T17:57:11.176 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18fd69d1:::benchmark_data_smithi179_148205_object66861:head# 2024-02-09T17:57:11.177 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18fdc4d5:::benchmark_data_smithi179_148205_object12881:head# 2024-02-09T17:57:11.177 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18fdef66:::benchmark_data_smithi179_148205_object49541:head# 2024-02-09T17:57:11.177 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18ff48ba:::benchmark_data_smithi179_148205_object93918:head# 2024-02-09T17:57:11.177 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18ff8cc7:::benchmark_data_smithi179_148205_object4593:head# 2024-02-09T17:57:11.177 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:18ffd79a:::benchmark_data_smithi179_148205_object58805:head# 2024-02-09T17:57:11.177 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:190135de:::benchmark_data_smithi179_148205_object103317:head# 2024-02-09T17:57:11.177 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:190195e7:::benchmark_data_smithi179_148205_object36638:head# 2024-02-09T17:57:11.177 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19021b3d:::benchmark_data_smithi179_148205_object81123:head# 2024-02-09T17:57:11.178 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19022eb9:::benchmark_data_smithi179_148205_object18780:head# 2024-02-09T17:57:11.178 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:190383e6:::benchmark_data_smithi179_148205_object31137:head# 2024-02-09T17:57:11.178 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1903cb7b:::benchmark_data_smithi179_148205_object43925:head# 2024-02-09T17:57:11.178 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:190415b1:::benchmark_data_smithi179_148205_object108792:head# 2024-02-09T17:57:11.179 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:190537e0:::benchmark_data_smithi179_148205_object39044:head# 2024-02-09T17:57:11.179 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19058bbd:::benchmark_data_smithi179_148205_object17905:head# 2024-02-09T17:57:11.180 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1906e3ff:::benchmark_data_smithi179_148205_object43260:head# 2024-02-09T17:57:11.180 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19070393:::benchmark_data_smithi179_148205_object80309:head# 2024-02-09T17:57:11.180 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1907431d:::benchmark_data_smithi179_148205_object56047:head# 2024-02-09T17:57:11.181 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19074578:::benchmark_data_smithi179_148205_object49821:head# 2024-02-09T17:57:11.181 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19076e8e:::benchmark_data_smithi179_148205_object98518:head# 2024-02-09T17:57:11.182 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19076f14:::benchmark_data_smithi179_148205_object89563:head# 2024-02-09T17:57:11.182 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19095db3:::benchmark_data_smithi179_148205_object45702:head# 2024-02-09T17:57:11.183 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1909634a:::benchmark_data_smithi179_148205_object104719:head# 2024-02-09T17:57:11.183 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19099acd:::benchmark_data_smithi179_148205_object69456:head# 2024-02-09T17:57:11.184 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:190ab7c3:::benchmark_data_smithi179_148205_object30304:head# 2024-02-09T17:57:11.185 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:190b7619:::benchmark_data_smithi179_148205_object108379:head# 2024-02-09T17:57:11.185 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:190bded1:::benchmark_data_smithi179_148205_object40690:head# 2024-02-09T17:57:11.188 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:190c20c7:::benchmark_data_smithi179_148205_object33528:head# 2024-02-09T17:57:11.189 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:190e12e2:::benchmark_data_smithi179_148205_object67718:head# 2024-02-09T17:57:11.189 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:190facad:::benchmark_data_smithi179_148205_object94829:head# 2024-02-09T17:57:11.189 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1910ca60:::benchmark_data_smithi179_148205_object12820:head# 2024-02-09T17:57:11.189 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19124533:::benchmark_data_smithi179_148205_object37235:head# 2024-02-09T17:57:11.189 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:191265ed:::benchmark_data_smithi179_148205_object44999:head# 2024-02-09T17:57:11.189 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1914a310:::benchmark_data_smithi179_148205_object46102:head# 2024-02-09T17:57:11.189 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1916df60:::benchmark_data_smithi179_148205_object3964:head# 2024-02-09T17:57:11.189 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19171ba1:::benchmark_data_smithi179_148205_object9138:head# 2024-02-09T17:57:11.189 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1917da5a:::benchmark_data_smithi179_148205_object27072:head# 2024-02-09T17:57:11.190 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1917e09a:::benchmark_data_smithi179_148205_object24270:head# 2024-02-09T17:57:11.190 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19184fc0:::benchmark_data_smithi179_148205_object85323:head# 2024-02-09T17:57:11.190 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19193d6d:::benchmark_data_smithi179_148205_object9730:head# 2024-02-09T17:57:11.190 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:191a32f3:::benchmark_data_smithi179_148205_object8907:head# 2024-02-09T17:57:11.190 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:191a3985:::benchmark_data_smithi179_148205_object19554:head# 2024-02-09T17:57:11.191 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:191a90dc:::benchmark_data_smithi179_148205_object41534:head# 2024-02-09T17:57:11.191 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:191c126c:::benchmark_data_smithi179_148205_object71448:head# 2024-02-09T17:57:11.196 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:191c2482:::benchmark_data_smithi179_148205_object76174:head# 2024-02-09T17:57:11.197 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:191cf588:::benchmark_data_smithi179_148205_object104343:head# 2024-02-09T17:57:11.197 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:191e0b7e:::benchmark_data_smithi179_148205_object51681:head# 2024-02-09T17:57:11.197 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:191eb6d2:::benchmark_data_smithi179_148205_object58945:head# 2024-02-09T17:57:11.197 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:191eccac:::benchmark_data_smithi179_148205_object106617:head# 2024-02-09T17:57:11.197 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:191ee27c:::benchmark_data_smithi179_148205_object62198:head# 2024-02-09T17:57:11.197 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:191eeebc:::benchmark_data_smithi179_148205_object39285:head# 2024-02-09T17:57:11.197 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:191f770d:::benchmark_data_smithi179_148205_object69195:head# 2024-02-09T17:57:11.197 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1920f40a:::benchmark_data_smithi179_148205_object24324:head# 2024-02-09T17:57:11.198 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1921b116:::benchmark_data_smithi179_148205_object12355:head# 2024-02-09T17:57:11.198 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1921be01:::benchmark_data_smithi179_148205_object52384:head# 2024-02-09T17:57:11.199 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1921ed26:::benchmark_data_smithi179_148205_object24790:head# 2024-02-09T17:57:11.199 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19223279:::benchmark_data_smithi179_148205_object74486:head# 2024-02-09T17:57:11.199 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1922f566:::benchmark_data_smithi179_148205_object25140:head# 2024-02-09T17:57:11.201 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:192303d4:::benchmark_data_smithi179_148205_object6664:head# 2024-02-09T17:57:11.202 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19236aa9:::benchmark_data_smithi179_148205_object76736:head# 2024-02-09T17:57:11.202 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1923bda7:::benchmark_data_smithi179_148205_object66108:head# 2024-02-09T17:57:11.202 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:192598cc:::benchmark_data_smithi179_148205_object4779:head# 2024-02-09T17:57:11.203 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19268679:::benchmark_data_smithi179_148205_object61591:head# 2024-02-09T17:57:11.203 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1926c55e:::benchmark_data_smithi179_148205_object6720:head# 2024-02-09T17:57:11.204 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1927151a:::benchmark_data_smithi179_148205_object76892:head# 2024-02-09T17:57:11.204 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1928292c:::benchmark_data_smithi179_148205_object32661:head# 2024-02-09T17:57:11.205 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19289bfa:::benchmark_data_smithi179_148205_object94850:head# 2024-02-09T17:57:11.205 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1929e902:::benchmark_data_smithi179_148205_object101837:head# 2024-02-09T17:57:11.209 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:192a8bd0:::benchmark_data_smithi179_148205_object27367:head# 2024-02-09T17:57:11.209 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:192ad364:::benchmark_data_smithi179_148205_object47875:head# 2024-02-09T17:57:11.209 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:192b7e3b:::benchmark_data_smithi179_148205_object68932:head# 2024-02-09T17:57:11.209 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:192c45bc:::benchmark_data_smithi179_148205_object88841:head# 2024-02-09T17:57:11.209 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:192d297c:::benchmark_data_smithi179_148205_object42423:head# 2024-02-09T17:57:11.209 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:192e567a:::benchmark_data_smithi179_148205_object83237:head# 2024-02-09T17:57:11.209 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19306ce3:::benchmark_data_smithi179_148205_object16031:head# 2024-02-09T17:57:11.209 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1931d9b0:::benchmark_data_smithi179_148205_object35386:head# 2024-02-09T17:57:11.209 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19327025:::benchmark_data_smithi179_148205_object24002:head# 2024-02-09T17:57:11.210 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19327236:::benchmark_data_smithi179_148205_object4703:head# 2024-02-09T17:57:11.210 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1932ca64:::benchmark_data_smithi179_148205_object96826:head# 2024-02-09T17:57:11.210 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:193308e4:::benchmark_data_smithi179_148205_object108346:head# 2024-02-09T17:57:11.210 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19331262:::benchmark_data_smithi179_148205_object51952:head# 2024-02-09T17:57:11.211 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19332ee8:::benchmark_data_smithi179_148205_object8914:head# 2024-02-09T17:57:11.211 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1933d4cd:::benchmark_data_smithi179_148205_object38752:head# 2024-02-09T17:57:11.212 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19344c06:::benchmark_data_smithi179_148205_object26285:head# 2024-02-09T17:57:11.212 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1935c98b:::benchmark_data_smithi179_148205_object72363:head# 2024-02-09T17:57:11.212 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:193696af:::benchmark_data_smithi179_148205_object80425:head# 2024-02-09T17:57:11.213 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19388603:::benchmark_data_smithi179_148205_object88716:head# 2024-02-09T17:57:11.213 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1938f21a:::benchmark_data_smithi179_148205_object72581:head# 2024-02-09T17:57:11.214 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:193923cf:::benchmark_data_smithi179_148205_object39421:head# 2024-02-09T17:57:11.214 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:193bb003:::benchmark_data_smithi179_148205_object34182:head# 2024-02-09T17:57:11.215 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:193bb430:::benchmark_data_smithi179_148205_object88086:head# 2024-02-09T17:57:11.215 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:193d3875:::benchmark_data_smithi179_148205_object49287:head# 2024-02-09T17:57:11.215 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:193d75da:::benchmark_data_smithi179_148205_object21594:head# 2024-02-09T17:57:11.216 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:193dd204:::benchmark_data_smithi179_148205_object14088:head# 2024-02-09T17:57:11.216 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:193e1657:::benchmark_data_smithi179_148205_object31214:head# 2024-02-09T17:57:11.217 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:193ea658:::benchmark_data_smithi179_148205_object15830:head# 2024-02-09T17:57:11.217 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:193f88ff:::benchmark_data_smithi179_148205_object9909:head# 2024-02-09T17:57:11.217 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1940ed04:::benchmark_data_smithi179_148205_object67417:head# 2024-02-09T17:57:11.218 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1941a5d4:::benchmark_data_smithi179_148205_object7998:head# 2024-02-09T17:57:11.218 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19422ee9:::benchmark_data_smithi179_148205_object97379:head# 2024-02-09T17:57:11.218 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:194247aa:::benchmark_data_smithi179_148205_object60256:head# 2024-02-09T17:57:11.219 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1942b350:::benchmark_data_smithi179_148205_object67917:head# 2024-02-09T17:57:11.219 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1942f14b:::benchmark_data_smithi179_148205_object92132:head# 2024-02-09T17:57:11.219 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19432a44:::benchmark_data_smithi179_148205_object87044:head# 2024-02-09T17:57:11.220 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:194467ea:::benchmark_data_smithi179_148205_object103210:head# 2024-02-09T17:57:11.220 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1944b762:::benchmark_data_smithi179_148205_object73382:head# 2024-02-09T17:57:11.221 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1944d258:::benchmark_data_smithi179_148205_object86259:head# 2024-02-09T17:57:11.221 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:194556c2:::benchmark_data_smithi179_148205_object74983:head# 2024-02-09T17:57:11.222 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19457eeb:::benchmark_data_smithi179_148205_object6312:head# 2024-02-09T17:57:11.222 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1945ced0:::benchmark_data_smithi179_148205_object27337:head# 2024-02-09T17:57:11.223 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1947faff:::benchmark_data_smithi179_148205_object69668:head# 2024-02-09T17:57:11.223 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19481773:::benchmark_data_smithi179_148205_object34538:head# 2024-02-09T17:57:11.224 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1948d98b:::benchmark_data_smithi179_148205_object47063:head# 2024-02-09T17:57:11.224 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1949ccdf:::benchmark_data_smithi179_148205_object39226:head# 2024-02-09T17:57:11.225 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:194a49d5:::benchmark_data_smithi179_148205_object40304:head# 2024-02-09T17:57:11.225 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:194aec15:::benchmark_data_smithi179_148205_object29455:head# 2024-02-09T17:57:11.225 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:194b75d6:::benchmark_data_smithi179_148205_object24861:head# 2024-02-09T17:57:11.225 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:194ca55c:::benchmark_data_smithi179_148205_object97177:head# 2024-02-09T17:57:11.229 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:194ca863:::benchmark_data_smithi179_148205_object5724:head# 2024-02-09T17:57:11.229 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:194d2adb:::benchmark_data_smithi179_148205_object43441:head# 2024-02-09T17:57:11.230 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:194d8dd2:::benchmark_data_smithi179_148205_object58263:head# 2024-02-09T17:57:11.230 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:194e2cb2:::benchmark_data_smithi179_148205_object13456:head# 2024-02-09T17:57:11.230 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:194ecf78:::benchmark_data_smithi179_148205_object22084:head# 2024-02-09T17:57:11.231 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:194f452d:::benchmark_data_smithi179_148205_object13705:head# 2024-02-09T17:57:11.232 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:194fb301:::benchmark_data_smithi179_148205_object73106:head# 2024-02-09T17:57:11.232 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:194ff6ab:::benchmark_data_smithi179_148205_object65618:head# 2024-02-09T17:57:11.236 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19501621:::benchmark_data_smithi179_148205_object86150:head# 2024-02-09T17:57:11.237 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1950342c:::benchmark_data_smithi179_148205_object75755:head# 2024-02-09T17:57:11.237 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1950fa1f:::benchmark_data_smithi179_148205_object97351:head# 2024-02-09T17:57:11.237 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1951d7cf:::benchmark_data_smithi179_148205_object25296:head# 2024-02-09T17:57:11.237 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19524a57:::benchmark_data_smithi179_148205_object56530:head# 2024-02-09T17:57:11.237 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1952bd80:::benchmark_data_smithi179_148205_object87529:head# 2024-02-09T17:57:11.237 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19534ffb:::benchmark_data_smithi179_148205_object67574:head# 2024-02-09T17:57:11.240 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19546716:::benchmark_data_smithi179_148205_object87633:head# 2024-02-09T17:57:11.240 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1955cc64:::benchmark_data_smithi179_148205_object89755:head# 2024-02-09T17:57:11.244 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:195650da:::benchmark_data_smithi179_148205_object71142:head# 2024-02-09T17:57:11.244 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1956efa4:::benchmark_data_smithi179_148205_object41146:head# 2024-02-09T17:57:11.244 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:195707bb:::benchmark_data_smithi179_148205_object41889:head# 2024-02-09T17:57:11.248 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19576b6a:::benchmark_data_smithi179_148205_object106142:head# 2024-02-09T17:57:11.248 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1957af63:::benchmark_data_smithi179_148205_object13650:head# 2024-02-09T17:57:11.249 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1957f91f:::benchmark_data_smithi179_148205_object61712:head# 2024-02-09T17:57:11.249 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:195808c3:::benchmark_data_smithi179_148205_object49462:head# 2024-02-09T17:57:11.249 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19582273:::benchmark_data_smithi179_148205_object17597:head# 2024-02-09T17:57:11.249 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19582b7f:::benchmark_data_smithi179_148205_object74992:head# 2024-02-09T17:57:11.249 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19585744:::benchmark_data_smithi179_148205_object71553:head# 2024-02-09T17:57:11.249 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:195869dd:::benchmark_data_smithi179_148205_object33613:head# 2024-02-09T17:57:11.249 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:195949b7:::benchmark_data_smithi179_148205_object95208:head# 2024-02-09T17:57:11.250 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:195a3afb:::benchmark_data_smithi179_148205_object27277:head# 2024-02-09T17:57:11.250 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:195a848c:::benchmark_data_smithi179_148205_object77883:head# 2024-02-09T17:57:11.251 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:195e10fa:::benchmark_data_smithi179_148205_object102869:head# 2024-02-09T17:57:11.251 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:195e4dc8:::benchmark_data_smithi179_148205_object91751:head# 2024-02-09T17:57:11.251 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:195f18e8:::benchmark_data_smithi179_148205_object79531:head# 2024-02-09T17:57:11.252 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1960171f:::benchmark_data_smithi179_148205_object63786:head# 2024-02-09T17:57:11.252 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19607c28:::benchmark_data_smithi179_148205_object6713:head# 2024-02-09T17:57:11.253 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1960eb6e:::benchmark_data_smithi179_148205_object20890:head# 2024-02-09T17:57:11.253 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19610833:::benchmark_data_smithi179_148205_object54227:head# 2024-02-09T17:57:11.253 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1964d83a:::benchmark_data_smithi179_148205_object102785:head# 2024-02-09T17:57:11.254 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19655461:::benchmark_data_smithi179_148205_object70956:head# 2024-02-09T17:57:11.259 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19659927:::benchmark_data_smithi179_148205_object38117:head# 2024-02-09T17:57:11.262 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19666b70:::benchmark_data_smithi179_148205_object57110:head# 2024-02-09T17:57:11.264 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:196698b6:::benchmark_data_smithi179_148205_object71204:head# 2024-02-09T17:57:11.264 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:196711f4:::benchmark_data_smithi179_148205_object27751:head# 2024-02-09T17:57:11.265 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19677e44:::benchmark_data_smithi179_148205_object62605:head# 2024-02-09T17:57:11.265 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19688e44:::benchmark_data_smithi179_148205_object5043:head# 2024-02-09T17:57:11.265 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1968e3aa:::benchmark_data_smithi179_148205_object95255:head# 2024-02-09T17:57:11.270 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1968f60d:::benchmark_data_smithi179_148205_object105985:head# 2024-02-09T17:57:11.281 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1968fdf3:::benchmark_data_smithi179_148205_object75525:head# 2024-02-09T17:57:11.282 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19691a6b:::benchmark_data_smithi179_148205_object74412:head# 2024-02-09T17:57:11.282 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19696246:::benchmark_data_smithi179_148205_object10742:head# 2024-02-09T17:57:11.283 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:196968e4:::benchmark_data_smithi179_148205_object45076:head# 2024-02-09T17:57:11.285 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:196a2ea6:::benchmark_data_smithi179_148205_object30919:head# 2024-02-09T17:57:11.285 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:196ae57b:::benchmark_data_smithi179_148205_object4118:head# 2024-02-09T17:57:11.285 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:196bbead:::benchmark_data_smithi179_148205_object69746:head# 2024-02-09T17:57:11.286 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:196c0919:::benchmark_data_smithi179_148205_object45415:head# 2024-02-09T17:57:11.287 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:196c5689:::benchmark_data_smithi179_148205_object81693:head# 2024-02-09T17:57:11.287 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:196e3486:::benchmark_data_smithi179_148205_object62473:head# 2024-02-09T17:57:11.287 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:196e3b1a:::benchmark_data_smithi179_148205_object35495:head# 2024-02-09T17:57:11.287 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:196e76f4:::benchmark_data_smithi179_148205_object104700:head# 2024-02-09T17:57:11.288 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:196ee438:::benchmark_data_smithi179_148205_object20011:head# 2024-02-09T17:57:11.288 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19701812:::benchmark_data_smithi179_148205_object63253:head# 2024-02-09T17:57:11.289 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1970d98a:::benchmark_data_smithi179_148205_object87752:head# 2024-02-09T17:57:11.289 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1970e28b:::benchmark_data_smithi179_148205_object42062:head# 2024-02-09T17:57:11.289 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19710bbb:::benchmark_data_smithi179_148205_object16522:head# 2024-02-09T17:57:11.294 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19713bc6:::benchmark_data_smithi179_148205_object87539:head# 2024-02-09T17:57:11.294 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:197177a0:::benchmark_data_smithi179_148205_object85707:head# 2024-02-09T17:57:11.294 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1971b5bf:::benchmark_data_smithi179_148205_object42775:head# 2024-02-09T17:57:11.294 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1972c7f5:::benchmark_data_smithi179_148205_object51796:head# 2024-02-09T17:57:11.294 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1972f1ea:::benchmark_data_smithi179_148205_object93908:head# 2024-02-09T17:57:11.294 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19731524:::benchmark_data_smithi179_148205_object41838:head# 2024-02-09T17:57:11.294 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:197333c7:::benchmark_data_smithi179_148205_object13746:head# 2024-02-09T17:57:11.294 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1973a94c:::benchmark_data_smithi179_148205_object43805:head# 2024-02-09T17:57:11.294 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1973e6b3:::benchmark_data_smithi179_148205_object92049:head# 2024-02-09T17:57:11.294 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1973e949:::benchmark_data_smithi179_148205_object27056:head# 2024-02-09T17:57:11.295 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:197486a7:::benchmark_data_smithi179_148205_object21862:head# 2024-02-09T17:57:11.295 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1974bce0:::benchmark_data_smithi179_148205_object94274:head# 2024-02-09T17:57:11.295 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1974ca2b:::benchmark_data_smithi179_148205_object44733:head# 2024-02-09T17:57:11.296 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1975511e:::benchmark_data_smithi179_148205_object77823:head# 2024-02-09T17:57:11.296 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19757ac6:::benchmark_data_smithi179_148205_object52187:head# 2024-02-09T17:57:11.296 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1975ce22:::benchmark_data_smithi179_148205_object53145:head# 2024-02-09T17:57:11.296 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19761987:::benchmark_data_smithi179_148205_object80701:head# 2024-02-09T17:57:11.297 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19771f7e:::benchmark_data_smithi179_148205_object41841:head# 2024-02-09T17:57:11.297 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:197725f6:::benchmark_data_smithi179_148205_object89048:head# 2024-02-09T17:57:11.298 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19778401:::benchmark_data_smithi179_148205_object18638:head# 2024-02-09T17:57:11.298 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1977b34c:::benchmark_data_smithi179_148205_object43941:head# 2024-02-09T17:57:11.298 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1977fc2a:::benchmark_data_smithi179_148205_object41546:head# 2024-02-09T17:57:11.299 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1978b16e:::benchmark_data_smithi179_148205_object99111:head# 2024-02-09T17:57:11.299 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1978c20a:::benchmark_data_smithi179_148205_object75294:head# 2024-02-09T17:57:11.299 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1978c74b:::benchmark_data_smithi179_148205_object88247:head# 2024-02-09T17:57:11.300 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:197941b2:::benchmark_data_smithi179_148205_object8166:head# 2024-02-09T17:57:11.301 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:197a122a:::benchmark_data_smithi179_148205_object70918:head# 2024-02-09T17:57:11.301 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:197ad6dd:::benchmark_data_smithi179_148205_object19155:head# 2024-02-09T17:57:11.301 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:197c6d2a:::benchmark_data_smithi179_148205_object60370:head# 2024-02-09T17:57:11.302 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:197cd524:::benchmark_data_smithi179_148205_object31451:head# 2024-02-09T17:57:11.302 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:197ddf01:::benchmark_data_smithi179_148205_object83488:head# 2024-02-09T17:57:11.303 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:197e91de:::benchmark_data_smithi179_148205_object13793:head# 2024-02-09T17:57:11.303 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:197f4c9d:::benchmark_data_smithi179_148205_object67887:head# 2024-02-09T17:57:11.304 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:197f8167:::benchmark_data_smithi179_148205_object69015:head# 2024-02-09T17:57:11.304 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:197f8a15:::benchmark_data_smithi179_148205_object69290:head# 2024-02-09T17:57:11.304 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19822997:::benchmark_data_smithi179_148205_object44221:head# 2024-02-09T17:57:11.305 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1982422b:::benchmark_data_smithi179_148205_object106126:head# 2024-02-09T17:57:11.305 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19826d0f:::benchmark_data_smithi179_148205_object84847:head# 2024-02-09T17:57:11.306 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1983175c:::benchmark_data_smithi179_148205_object86159:head# 2024-02-09T17:57:11.306 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:198344f3:::benchmark_data_smithi179_148205_object103455:head# 2024-02-09T17:57:11.306 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1983979e:::benchmark_data_smithi179_148205_object41989:head# 2024-02-09T17:57:11.322 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1985b485:::benchmark_data_smithi179_148205_object90963:head# 2024-02-09T17:57:11.322 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:198681d8:::benchmark_data_smithi179_148205_object48630:head# 2024-02-09T17:57:11.322 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19869305:::benchmark_data_smithi179_148205_object1030:head# 2024-02-09T17:57:11.322 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1986a72a:::benchmark_data_smithi179_148205_object12665:head# 2024-02-09T17:57:11.322 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1986fe9b:::benchmark_data_smithi179_148205_object22101:head# 2024-02-09T17:57:11.322 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1987aaf9:::benchmark_data_smithi179_148205_object34364:head# 2024-02-09T17:57:11.322 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1987dbcf:::benchmark_data_smithi179_148205_object39109:head# 2024-02-09T17:57:11.323 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19889422:::benchmark_data_smithi179_148205_object69707:head# 2024-02-09T17:57:11.323 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1988aebe:::benchmark_data_smithi179_148205_object51397:head# 2024-02-09T17:57:11.323 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1988cc95:::benchmark_data_smithi179_148205_object96080:head# 2024-02-09T17:57:11.324 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1989ad84:::benchmark_data_smithi179_148205_object103973:head# 2024-02-09T17:57:11.324 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:198a1413:::benchmark_data_smithi179_148205_object95959:head# 2024-02-09T17:57:11.324 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:198bf64f:::benchmark_data_smithi179_148205_object57162:head# 2024-02-09T17:57:11.325 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:198d2fae:::benchmark_data_smithi179_148205_object74767:head# 2024-02-09T17:57:11.325 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:198d74cd:::benchmark_data_smithi179_148205_object27550:head# 2024-02-09T17:57:11.325 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:198d91f8:::benchmark_data_smithi179_148205_object10968:head# 2024-02-09T17:57:11.326 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:198ee1d7:::benchmark_data_smithi179_148205_object36653:head# 2024-02-09T17:57:11.326 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:198f68a8:::benchmark_data_smithi179_148205_object44744:head# 2024-02-09T17:57:11.327 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:198f6ae5:::benchmark_data_smithi179_148205_object54285:head# 2024-02-09T17:57:11.327 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:198f9f95:::benchmark_data_smithi179_148205_object81013:head# 2024-02-09T17:57:11.327 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:199030b4:::benchmark_data_smithi179_148205_object100326:head# 2024-02-09T17:57:11.328 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19903e4f:::benchmark_data_smithi179_148205_object95027:head# 2024-02-09T17:57:11.328 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1990654a:::benchmark_data_smithi179_148205_object80181:head# 2024-02-09T17:57:11.328 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19914666:::benchmark_data_smithi179_148205_object109193:head# 2024-02-09T17:57:11.329 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1992924e:::benchmark_data_smithi179_148205_object24257:head# 2024-02-09T17:57:11.329 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1992e51b:::benchmark_data_smithi179_148205_object58331:head# 2024-02-09T17:57:11.330 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19932733:::benchmark_data_smithi179_148205_object92607:head# 2024-02-09T17:57:11.330 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19934af3:::benchmark_data_smithi179_148205_object89277:head# 2024-02-09T17:57:11.331 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19935868:::benchmark_data_smithi179_148205_object65449:head# 2024-02-09T17:57:11.331 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1994115e:::benchmark_data_smithi179_148205_object85599:head# 2024-02-09T17:57:11.335 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19951ecd:::benchmark_data_smithi179_148205_object47533:head# 2024-02-09T17:57:11.335 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19953e28:::benchmark_data_smithi179_148205_object47667:head# 2024-02-09T17:57:11.336 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1995cfd4:::benchmark_data_smithi179_148205_object84152:head# 2024-02-09T17:57:11.336 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19965fb9:::benchmark_data_smithi179_148205_object94021:head# 2024-02-09T17:57:11.336 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1996614d:::benchmark_data_smithi179_148205_object104394:head# 2024-02-09T17:57:11.336 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:199680dc:::benchmark_data_smithi179_148205_object100035:head# 2024-02-09T17:57:11.337 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1999be82:::benchmark_data_smithi179_148205_object103103:head# 2024-02-09T17:57:11.337 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1999cef0:::benchmark_data_smithi179_148205_object50338:head# 2024-02-09T17:57:11.337 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:199a454c:::benchmark_data_smithi179_148205_object14315:head# 2024-02-09T17:57:11.337 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:199bbad1:::benchmark_data_smithi179_148205_object31158:head# 2024-02-09T17:57:11.337 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:199c0141:::benchmark_data_smithi179_148205_object23392:head# 2024-02-09T17:57:11.337 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:199c1b32:::benchmark_data_smithi179_148205_object15693:head# 2024-02-09T17:57:11.337 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:199ca859:::benchmark_data_smithi179_148205_object84218:head# 2024-02-09T17:57:11.341 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:199d1713:::benchmark_data_smithi179_148205_object109636:head# 2024-02-09T17:57:11.342 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:199d41f6:::benchmark_data_smithi179_148205_object65127:head# 2024-02-09T17:57:11.342 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:199d8e5c:::benchmark_data_smithi179_148205_object73439:head# 2024-02-09T17:57:11.342 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:199e863b:::benchmark_data_smithi179_148205_object54491:head# 2024-02-09T17:57:11.342 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:199e9c6e:::benchmark_data_smithi179_148205_object52872:head# 2024-02-09T17:57:11.342 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:199ea8fa:::benchmark_data_smithi179_148205_object2555:head# 2024-02-09T17:57:11.342 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:199f20aa:::benchmark_data_smithi179_148205_object37896:head# 2024-02-09T17:57:11.342 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:199fc431:::benchmark_data_smithi179_148205_object3100:head# 2024-02-09T17:57:11.343 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19a03094:::benchmark_data_smithi179_148205_object41441:head# 2024-02-09T17:57:11.343 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19a0441d:::benchmark_data_smithi179_148205_object83057:head# 2024-02-09T17:57:11.343 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19a100b4:::benchmark_data_smithi179_148205_object27253:head# 2024-02-09T17:57:11.344 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19a15c31:::benchmark_data_smithi179_148205_object61502:head# 2024-02-09T17:57:11.344 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19a1a840:::benchmark_data_smithi179_148205_object99975:head# 2024-02-09T17:57:11.344 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19a20e7a:::benchmark_data_smithi179_148205_object86239:head# 2024-02-09T17:57:11.344 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19a30cdb:::benchmark_data_smithi179_148205_object12452:head# 2024-02-09T17:57:11.345 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19a33837:::benchmark_data_smithi179_148205_object94027:head# 2024-02-09T17:57:11.345 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19a43265:::benchmark_data_smithi179_148205_object90623:head# 2024-02-09T17:57:11.346 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19a4758c:::benchmark_data_smithi179_148205_object48127:head# 2024-02-09T17:57:11.346 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19a55721:::benchmark_data_smithi179_148205_object44201:head# 2024-02-09T17:57:11.346 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19a55ca5:::benchmark_data_smithi179_148205_object43435:head# 2024-02-09T17:57:11.348 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19a590e7:::benchmark_data_smithi179_148205_object24673:head# 2024-02-09T17:57:11.348 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19a66961:::benchmark_data_smithi179_148205_object98110:head# 2024-02-09T17:57:11.349 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19a6ab23:::benchmark_data_smithi179_148205_object22069:head# 2024-02-09T17:57:11.349 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19a6b59c:::benchmark_data_smithi179_148205_object78835:head# 2024-02-09T17:57:11.349 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19a76cfa:::benchmark_data_smithi179_148205_object16431:head# 2024-02-09T17:57:11.350 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19a85166:::benchmark_data_smithi179_148205_object96071:head# 2024-02-09T17:57:11.350 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19a8b0d0:::benchmark_data_smithi179_148205_object61208:head# 2024-02-09T17:57:11.351 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19a8d80e:::benchmark_data_smithi179_148205_object43670:head# 2024-02-09T17:57:11.352 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19a9ac5c:::benchmark_data_smithi179_148205_object48624:head# 2024-02-09T17:57:11.352 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19aa4afb:::benchmark_data_smithi179_148205_object15398:head# 2024-02-09T17:57:11.352 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19ab8d97:::benchmark_data_smithi179_148205_object84442:head# 2024-02-09T17:57:11.355 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19ae6592:::benchmark_data_smithi179_148205_object38627:head# 2024-02-09T17:57:11.357 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19af2f39:::benchmark_data_smithi179_148205_object99432:head# 2024-02-09T17:57:11.357 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19afdb02:::benchmark_data_smithi179_148205_object109160:head# 2024-02-09T17:57:11.358 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19b00896:::benchmark_data_smithi179_148205_object16069:head# 2024-02-09T17:57:11.360 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19b0a8d8:::benchmark_data_smithi179_148205_object14244:head# 2024-02-09T17:57:11.360 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19b138d9:::benchmark_data_smithi179_148205_object27093:head# 2024-02-09T17:57:11.360 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19b15003:::benchmark_data_smithi179_148205_object73159:head# 2024-02-09T17:57:11.361 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19b2b332:::benchmark_data_smithi179_148205_object86674:head# 2024-02-09T17:57:11.361 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19b3035e:::benchmark_data_smithi179_148205_object11464:head# 2024-02-09T17:57:11.362 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19b37004:::benchmark_data_smithi179_148205_object19860:head# 2024-02-09T17:57:11.362 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19b371c0:::benchmark_data_smithi179_148205_object98373:head# 2024-02-09T17:57:11.367 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19b411dd:::benchmark_data_smithi179_148205_object107293:head# 2024-02-09T17:57:11.367 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19b50498:::benchmark_data_smithi179_148205_object14062:head# 2024-02-09T17:57:11.367 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19b51e13:::benchmark_data_smithi179_148205_object35951:head# 2024-02-09T17:57:11.367 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19b54b16:::benchmark_data_smithi179_148205_object53727:head# 2024-02-09T17:57:11.367 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19b55ed1:::benchmark_data_smithi179_148205_object95516:head# 2024-02-09T17:57:11.368 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19b5abc2:::benchmark_data_smithi179_148205_object31547:head# 2024-02-09T17:57:11.368 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19b5db3f:::benchmark_data_smithi179_148205_object80615:head# 2024-02-09T17:57:11.368 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19b70a73:::benchmark_data_smithi179_148205_object49830:head# 2024-02-09T17:57:11.368 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19b73dbe:::benchmark_data_smithi179_148205_object25276:head# 2024-02-09T17:57:11.368 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19b84e5d:::benchmark_data_smithi179_148205_object69294:head# 2024-02-09T17:57:11.368 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19ba2502:::benchmark_data_smithi179_148205_object85694:head# 2024-02-09T17:57:11.368 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19ba7c56:::benchmark_data_smithi179_148205_object64975:head# 2024-02-09T17:57:11.372 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19bb26df:::benchmark_data_smithi179_148205_object71993:head# 2024-02-09T17:57:11.372 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19bb7b88:::benchmark_data_smithi179_148205_object38733:head# 2024-02-09T17:57:11.376 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19bb7e99:::benchmark_data_smithi179_148205_object91689:head# 2024-02-09T17:57:11.376 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19bc79af:::benchmark_data_smithi179_148205_object80823:head# 2024-02-09T17:57:11.376 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19bd4fd4:::benchmark_data_smithi179_148205_object92099:head# 2024-02-09T17:57:11.377 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19bd6174:::benchmark_data_smithi179_148205_object99327:head# 2024-02-09T17:57:11.377 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19be2a15:::benchmark_data_smithi179_148205_object1656:head# 2024-02-09T17:57:11.377 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19be5eab:::benchmark_data_smithi179_148205_object45122:head# 2024-02-09T17:57:11.379 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19bebc2d:::benchmark_data_smithi179_148205_object92389:head# 2024-02-09T17:57:11.379 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19bebe62:::benchmark_data_smithi179_148205_object25048:head# 2024-02-09T17:57:11.379 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19bf1f41:::benchmark_data_smithi179_148205_object79443:head# 2024-02-09T17:57:11.379 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19bfa769:::benchmark_data_smithi179_148205_object61620:head# 2024-02-09T17:57:11.380 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19c04d6a:::benchmark_data_smithi179_148205_object95061:head# 2024-02-09T17:57:11.380 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19c2750f:::benchmark_data_smithi179_148205_object69126:head# 2024-02-09T17:57:11.381 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19c31712:::benchmark_data_smithi179_148205_object7873:head# 2024-02-09T17:57:11.381 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19c344f3:::benchmark_data_smithi179_148205_object23970:head# 2024-02-09T17:57:11.385 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19c3de40:::benchmark_data_smithi179_148205_object101222:head# 2024-02-09T17:57:11.385 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19c58b5b:::benchmark_data_smithi179_148205_object30579:head# 2024-02-09T17:57:11.386 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19c666a2:::benchmark_data_smithi179_148205_object29602:head# 2024-02-09T17:57:11.386 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19c6b82a:::benchmark_data_smithi179_148205_object58015:head# 2024-02-09T17:57:11.386 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19c74721:::benchmark_data_smithi179_148205_object84224:head# 2024-02-09T17:57:11.386 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19c74fea:::benchmark_data_smithi179_148205_object8195:head# 2024-02-09T17:57:11.386 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19c75a48:::benchmark_data_smithi179_148205_object46397:head# 2024-02-09T17:57:11.388 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19c7f5ef:::benchmark_data_smithi179_148205_object49472:head# 2024-02-09T17:57:11.388 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19c9fb02:::benchmark_data_smithi179_148205_object33368:head# 2024-02-09T17:57:11.389 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19cae890:::benchmark_data_smithi179_148205_object73963:head# 2024-02-09T17:57:11.389 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19cba619:::benchmark_data_smithi179_148205_object84252:head# 2024-02-09T17:57:11.389 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19cc3b74:::benchmark_data_smithi179_148205_object74018:head# 2024-02-09T17:57:11.390 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19cd8667:::benchmark_data_smithi179_148205_object37671:head# 2024-02-09T17:57:11.390 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19ce3708:::benchmark_data_smithi179_148205_object8890:head# 2024-02-09T17:57:11.390 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19ce6683:::benchmark_data_smithi179_148205_object71803:head# 2024-02-09T17:57:11.391 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19cf7e0a:::benchmark_data_smithi179_148205_object5682:head# 2024-02-09T17:57:11.391 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19d04f37:::benchmark_data_smithi179_148205_object27792:head# 2024-02-09T17:57:11.392 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19d14025:::benchmark_data_smithi179_148205_object8778:head# 2024-02-09T17:57:11.393 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19d18713:::benchmark_data_smithi179_148205_object2705:head# 2024-02-09T17:57:11.393 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19d21a64:::benchmark_data_smithi179_148205_object43340:head# 2024-02-09T17:57:11.393 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19d27867:::benchmark_data_smithi179_148205_object33889:head# 2024-02-09T17:57:11.395 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19d38fe9:::benchmark_data_smithi179_148205_object60192:head# 2024-02-09T17:57:11.397 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19d4aa8d:::benchmark_data_smithi179_148205_object62207:head# 2024-02-09T17:57:11.397 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19d4d263:::benchmark_data_smithi179_148205_object82211:head# 2024-02-09T17:57:11.398 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19d50d8e:::benchmark_data_smithi179_148205_object45195:head# 2024-02-09T17:57:11.398 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19d544ca:::benchmark_data_smithi179_148205_object59215:head# 2024-02-09T17:57:11.399 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19d5f5ab:::benchmark_data_smithi179_148205_object9241:head# 2024-02-09T17:57:11.399 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19d69686:::benchmark_data_smithi179_148205_object34357:head# 2024-02-09T17:57:11.401 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19d6abd5:::benchmark_data_smithi179_148205_object94284:head# 2024-02-09T17:57:11.402 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19d79cc8:::benchmark_data_smithi179_148205_object97507:head# 2024-02-09T17:57:11.402 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19d79e6d:::benchmark_data_smithi179_148205_object47941:head# 2024-02-09T17:57:11.403 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19d83780:::benchmark_data_smithi179_148205_object26234:head# 2024-02-09T17:57:11.403 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19d86dd3:::benchmark_data_smithi179_148205_object53406:head# 2024-02-09T17:57:11.403 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19d8e34c:::benchmark_data_smithi179_148205_object28835:head# 2024-02-09T17:57:11.404 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19d954b1:::benchmark_data_smithi179_148205_object104446:head# 2024-02-09T17:57:11.404 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19db6ebd:::benchmark_data_smithi179_148205_object14989:head# 2024-02-09T17:57:11.405 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19db79ce:::benchmark_data_smithi179_148205_object88922:head# 2024-02-09T17:57:11.407 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19dc6d0a:::benchmark_data_smithi179_148205_object100022:head# 2024-02-09T17:57:11.408 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19dd105b:::benchmark_data_smithi179_148205_object47264:head# 2024-02-09T17:57:11.408 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19ddc889:::benchmark_data_smithi179_148205_object87646:head# 2024-02-09T17:57:11.408 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19de783a:::benchmark_data_smithi179_148205_object14309:head# 2024-02-09T17:57:11.409 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19e01507:::benchmark_data_smithi179_148205_object10346:head# 2024-02-09T17:57:11.409 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19e02010:::benchmark_data_smithi179_148205_object41791:head# 2024-02-09T17:57:11.409 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19e09234:::benchmark_data_smithi179_148205_object6788:head# 2024-02-09T17:57:11.410 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19e0a55c:::benchmark_data_smithi179_148205_object86549:head# 2024-02-09T17:57:11.410 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19e14730:::benchmark_data_smithi179_148205_object27226:head# 2024-02-09T17:57:11.410 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19e14bcc:::benchmark_data_smithi179_148205_object20228:head# 2024-02-09T17:57:11.411 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19e1bde9:::benchmark_data_smithi179_148205_object76285:head# 2024-02-09T17:57:11.411 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19e258b4:::benchmark_data_smithi179_148205_object13870:head# 2024-02-09T17:57:11.412 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19e30a14:::benchmark_data_smithi179_148205_object41510:head# 2024-02-09T17:57:11.412 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19e37b6f:::benchmark_data_smithi179_148205_object30114:head# 2024-02-09T17:57:11.413 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19e3a62d:::benchmark_data_smithi179_148205_object43740:head# 2024-02-09T17:57:11.413 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19e3e3e7:::benchmark_data_smithi179_148205_object94257:head# 2024-02-09T17:57:11.413 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19e3f165:::benchmark_data_smithi179_148205_object103498:head# 2024-02-09T17:57:11.414 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19e4d263:::benchmark_data_smithi179_148205_object78741:head# 2024-02-09T17:57:11.414 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19e4ea71:::benchmark_data_smithi179_148205_object89872:head# 2024-02-09T17:57:11.414 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19e50ef9:::benchmark_data_smithi179_148205_object27372:head# 2024-02-09T17:57:11.415 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19e51a0c:::benchmark_data_smithi179_148205_object31278:head# 2024-02-09T17:57:11.415 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19e56b65:::benchmark_data_smithi179_148205_object93996:head# 2024-02-09T17:57:11.416 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19e6deb9:::benchmark_data_smithi179_148205_object101643:head# 2024-02-09T17:57:11.416 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19e78aa0:::benchmark_data_smithi179_148205_object20069:head# 2024-02-09T17:57:11.416 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19e86533:::benchmark_data_smithi179_148205_object35734:head# 2024-02-09T17:57:11.417 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19e8b33d:::benchmark_data_smithi179_148205_object55514:head# 2024-02-09T17:57:11.417 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19e94552:::benchmark_data_smithi179_148205_object54766:head# 2024-02-09T17:57:11.417 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19ea8a8a:::benchmark_data_smithi179_148205_object88920:head# 2024-02-09T17:57:11.418 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19ea9374:::benchmark_data_smithi179_148205_object1867:head# 2024-02-09T17:57:11.418 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19eb91d8:::benchmark_data_smithi179_148205_object37647:head# 2024-02-09T17:57:11.419 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19ebfca6:::benchmark_data_smithi179_148205_object106808:head# 2024-02-09T17:57:11.419 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19eca4a1:::benchmark_data_smithi179_148205_object5917:head# 2024-02-09T17:57:11.420 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19ecd338:::benchmark_data_smithi179_148205_object91883:head# 2024-02-09T17:57:11.420 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19ed0a7a:::benchmark_data_smithi179_148205_object75385:head# 2024-02-09T17:57:11.421 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19ed4191:::benchmark_data_smithi179_148205_object70763:head# 2024-02-09T17:57:11.421 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19edcf15:::benchmark_data_smithi179_148205_object87578:head# 2024-02-09T17:57:11.421 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19ef5b40:::benchmark_data_smithi179_148205_object23494:head# 2024-02-09T17:57:11.422 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19efa1fe:::benchmark_data_smithi179_148205_object51193:head# 2024-02-09T17:57:11.423 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19efd461:::benchmark_data_smithi179_148205_object91167:head# 2024-02-09T17:57:11.423 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19f00a03:::benchmark_data_smithi179_148205_object96997:head# 2024-02-09T17:57:11.423 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19f11dac:::benchmark_data_smithi179_148205_object79945:head# 2024-02-09T17:57:11.424 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19f18a20:::benchmark_data_smithi179_148205_object39243:head# 2024-02-09T17:57:11.424 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19f199d3:::benchmark_data_smithi179_148205_object368:head# 2024-02-09T17:57:11.425 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19f1e9cb:::benchmark_data_smithi179_148205_object28965:head# 2024-02-09T17:57:11.426 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19f2ffbd:::benchmark_data_smithi179_148205_object93034:head# 2024-02-09T17:57:11.426 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19f31297:::benchmark_data_smithi179_148205_object96815:head# 2024-02-09T17:57:11.426 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19f54982:::benchmark_data_smithi179_148205_object106423:head# 2024-02-09T17:57:11.427 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19f5a61b:::benchmark_data_smithi179_148205_object47215:head# 2024-02-09T17:57:11.427 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19f5f3d5:::benchmark_data_smithi179_148205_object54840:head# 2024-02-09T17:57:11.427 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19f7474f:::benchmark_data_smithi179_148205_object54232:head# 2024-02-09T17:57:11.430 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19f76067:::benchmark_data_smithi179_148205_object9745:head# 2024-02-09T17:57:11.430 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19f93178:::benchmark_data_smithi179_148205_object46939:head# 2024-02-09T17:57:11.431 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19f93588:::benchmark_data_smithi179_148205_object77901:head# 2024-02-09T17:57:11.432 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19f94867:::benchmark_data_smithi179_148205_object109519:head# 2024-02-09T17:57:11.432 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19f96ed1:::benchmark_data_smithi179_148205_object24580:head# 2024-02-09T17:57:11.432 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19f998c5:::benchmark_data_smithi179_148205_object17973:head# 2024-02-09T17:57:11.433 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19fa1f51:::benchmark_data_smithi179_148205_object6201:head# 2024-02-09T17:57:11.433 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19fb00fa:::benchmark_data_smithi179_148205_object958:head# 2024-02-09T17:57:11.433 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19fb35cf:::benchmark_data_smithi179_148205_object9507:head# 2024-02-09T17:57:11.434 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19fcb7ee:::benchmark_data_smithi179_148205_object39795:head# 2024-02-09T17:57:11.434 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19fcf994:::benchmark_data_smithi179_148205_object31702:head# 2024-02-09T17:57:11.435 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19fd23ad:::benchmark_data_smithi179_148205_object109089:head# 2024-02-09T17:57:11.435 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19fd4a28:::benchmark_data_smithi179_148205_object105908:head# 2024-02-09T17:57:11.435 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19fd7ca1:::benchmark_data_smithi179_148205_object78124:head# 2024-02-09T17:57:11.436 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19fde30f:::benchmark_data_smithi179_148205_object69085:head# 2024-02-09T17:57:11.436 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19fe35a1:::benchmark_data_smithi179_148205_object49023:head# 2024-02-09T17:57:11.437 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:19fef2a8:::benchmark_data_smithi179_148205_object104114:head# 2024-02-09T17:57:11.437 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a0040d1:::benchmark_data_smithi179_148205_object97292:head# 2024-02-09T17:57:11.437 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a00dcbd:::benchmark_data_smithi179_148205_object80925:head# 2024-02-09T17:57:11.438 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a01c0ed:::benchmark_data_smithi179_148205_object101014:head# 2024-02-09T17:57:11.438 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a02b82e:::benchmark_data_smithi179_148205_object14561:head# 2024-02-09T17:57:11.439 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a02c65d:::benchmark_data_smithi179_148205_object23883:head# 2024-02-09T17:57:11.439 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a02c7c7:::benchmark_data_smithi179_148205_object81215:head# 2024-02-09T17:57:11.440 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a045b23:::benchmark_data_smithi179_148205_object43139:head# 2024-02-09T17:57:11.440 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a045d7d:::benchmark_data_smithi179_148205_object104686:head# 2024-02-09T17:57:11.441 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a056c63:::benchmark_data_smithi179_148205_object92885:head# 2024-02-09T17:57:11.441 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a0571bc:::benchmark_data_smithi179_148205_object6508:head# 2024-02-09T17:57:11.441 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a0623a5:::benchmark_data_smithi179_148205_object66616:head# 2024-02-09T17:57:11.442 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a06b3cf:::benchmark_data_smithi179_148205_object96502:head# 2024-02-09T17:57:11.442 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a06c851:::benchmark_data_smithi179_148205_object105789:head# 2024-02-09T17:57:11.443 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a06d7b2:::benchmark_data_smithi179_148205_object99039:head# 2024-02-09T17:57:11.443 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a085aaf:::benchmark_data_smithi179_148205_object86826:head# 2024-02-09T17:57:11.444 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a085f27:::benchmark_data_smithi179_148205_object40700:head# 2024-02-09T17:57:11.444 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a096b14:::benchmark_data_smithi179_148205_object30109:head# 2024-02-09T17:57:11.445 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a09e024:::benchmark_data_smithi179_148205_object34009:head# 2024-02-09T17:57:11.445 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a0aef1a:::benchmark_data_smithi179_148205_object2208:head# 2024-02-09T17:57:11.445 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a0be67b:::benchmark_data_smithi179_148205_object60648:head# 2024-02-09T17:57:11.446 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a0cc13e:::benchmark_data_smithi179_148205_object56480:head# 2024-02-09T17:57:11.446 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a0cc79f:::benchmark_data_smithi179_148205_object105639:head# 2024-02-09T17:57:11.446 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a0dac7d:::benchmark_data_smithi179_148205_object84103:head# 2024-02-09T17:57:11.447 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a0db117:::benchmark_data_smithi179_148205_object55814:head# 2024-02-09T17:57:11.447 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a0e661b:::benchmark_data_smithi179_148205_object27068:head# 2024-02-09T17:57:11.452 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a1162aa:::benchmark_data_smithi179_148205_object43415:head# 2024-02-09T17:57:11.452 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a119316:::benchmark_data_smithi179_148205_object29116:head# 2024-02-09T17:57:11.452 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a11cd08:::benchmark_data_smithi179_148205_object12696:head# 2024-02-09T17:57:11.453 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a120f8c:::benchmark_data_smithi179_148205_object53566:head# 2024-02-09T17:57:11.453 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a130207:::benchmark_data_smithi179_148205_object1257:head# 2024-02-09T17:57:11.454 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a13409f:::benchmark_data_smithi179_148205_object93468:head# 2024-02-09T17:57:11.454 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a1360b6:::benchmark_data_smithi179_148205_object13151:head# 2024-02-09T17:57:11.454 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a13962a:::benchmark_data_smithi179_148205_object22059:head# 2024-02-09T17:57:11.455 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a13b6c4:::benchmark_data_smithi179_148205_object4416:head# 2024-02-09T17:57:11.458 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a14f2c8:::benchmark_data_smithi179_148205_object68135:head# 2024-02-09T17:57:11.458 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a16847a:::benchmark_data_smithi179_148205_object88031:head# 2024-02-09T17:57:11.459 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a168591:::benchmark_data_smithi179_148205_object57940:head# 2024-02-09T17:57:11.459 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a16b4c5:::benchmark_data_smithi179_148205_object79868:head# 2024-02-09T17:57:11.459 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a16b94e:::benchmark_data_smithi179_148205_object61411:head# 2024-02-09T17:57:11.460 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a17f42d:::benchmark_data_smithi179_148205_object7756:head# 2024-02-09T17:57:11.460 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a189e1b:::benchmark_data_smithi179_148205_object73097:head# 2024-02-09T17:57:11.462 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a197f3d:::benchmark_data_smithi179_148205_object98201:head# 2024-02-09T17:57:11.462 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a19fa9e:::benchmark_data_smithi179_148205_object34173:head# 2024-02-09T17:57:11.462 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a1a2e6a:::benchmark_data_smithi179_148205_object25378:head# 2024-02-09T17:57:11.462 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a1b47f0:::benchmark_data_smithi179_148205_object92658:head# 2024-02-09T17:57:11.462 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a1ca8a3:::benchmark_data_smithi179_148205_object104803:head# 2024-02-09T17:57:11.463 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a1e75ae:::benchmark_data_smithi179_148205_object99823:head# 2024-02-09T17:57:11.463 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a1eaeaf:::benchmark_data_smithi179_148205_object97739:head# 2024-02-09T17:57:11.464 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a1f1d5c:::benchmark_data_smithi179_148205_object70448:head# 2024-02-09T17:57:11.464 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a1f93f5:::benchmark_data_smithi179_148205_object23538:head# 2024-02-09T17:57:11.465 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a1fed06:::benchmark_data_smithi179_148205_object96973:head# 2024-02-09T17:57:11.467 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a217b4a:::benchmark_data_smithi179_148205_object8339:head# 2024-02-09T17:57:11.468 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a21ee56:::benchmark_data_smithi179_148205_object13135:head# 2024-02-09T17:57:11.468 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a22002f:::benchmark_data_smithi179_148205_object81099:head# 2024-02-09T17:57:11.469 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a227986:::benchmark_data_smithi179_148205_object16254:head# 2024-02-09T17:57:11.469 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a238d0a:::benchmark_data_smithi179_148205_object28129:head# 2024-02-09T17:57:11.470 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a23bfa7:::benchmark_data_smithi179_148205_object87081:head# 2024-02-09T17:57:11.470 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a2494cd:::benchmark_data_smithi179_148205_object19594:head# 2024-02-09T17:57:11.470 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a24f6a5:::benchmark_data_smithi179_148205_object73168:head# 2024-02-09T17:57:11.476 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a2568d7:::benchmark_data_smithi179_148205_object84798:head# 2024-02-09T17:57:11.477 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a256b98:::benchmark_data_smithi179_148205_object78746:head# 2024-02-09T17:57:11.477 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a25c033:::benchmark_data_smithi179_148205_object20842:head# 2024-02-09T17:57:11.482 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a26b834:::benchmark_data_smithi179_148205_object3655:head# 2024-02-09T17:57:11.482 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a26c1d4:::benchmark_data_smithi179_148205_object109382:head# 2024-02-09T17:57:11.482 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a26e190:::benchmark_data_smithi179_148205_object50066:head# 2024-02-09T17:57:11.482 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a28882b:::benchmark_data_smithi179_148205_object54573:head# 2024-02-09T17:57:11.482 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a2945ea:::benchmark_data_smithi179_148205_object33184:head# 2024-02-09T17:57:11.482 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a295900:::benchmark_data_smithi179_148205_object102862:head# 2024-02-09T17:57:11.482 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a2a0300:::benchmark_data_smithi179_148205_object37721:head# 2024-02-09T17:57:11.482 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a2a533b:::benchmark_data_smithi179_148205_object12226:head# 2024-02-09T17:57:11.483 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a2b0c37:::benchmark_data_smithi179_148205_object25778:head# 2024-02-09T17:57:11.483 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a2c31e5:::benchmark_data_smithi179_148205_object82415:head# 2024-02-09T17:57:11.483 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a2dad94:::benchmark_data_smithi179_148205_object63098:head# 2024-02-09T17:57:11.483 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a2e67e0:::benchmark_data_smithi179_148205_object44939:head# 2024-02-09T17:57:11.484 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a2ed273:::benchmark_data_smithi179_148205_object2471:head# 2024-02-09T17:57:11.485 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a2f5ab9:::benchmark_data_smithi179_148205_object27197:head# 2024-02-09T17:57:11.485 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a2f7d49:::benchmark_data_smithi179_148205_object44295:head# 2024-02-09T17:57:11.485 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a3071a6:::benchmark_data_smithi179_148205_object7463:head# 2024-02-09T17:57:11.487 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a316158:::benchmark_data_smithi179_148205_object26431:head# 2024-02-09T17:57:11.487 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a3187f6:::benchmark_data_smithi179_148205_object61626:head# 2024-02-09T17:57:11.488 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a319b47:::benchmark_data_smithi179_148205_object54961:head# 2024-02-09T17:57:11.488 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a321889:::benchmark_data_smithi179_148205_object55062:head# 2024-02-09T17:57:11.493 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a32a71b:::benchmark_data_smithi179_148205_object105504:head# 2024-02-09T17:57:11.493 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a32be43:::benchmark_data_smithi179_148205_object21504:head# 2024-02-09T17:57:11.493 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a32f16d:::benchmark_data_smithi179_148205_object10450:head# 2024-02-09T17:57:11.493 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a3349ea:::benchmark_data_smithi179_148205_object35690:head# 2024-02-09T17:57:11.493 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a33f824:::benchmark_data_smithi179_148205_object12372:head# 2024-02-09T17:57:11.493 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a340c16:::benchmark_data_smithi179_148205_object107238:head# 2024-02-09T17:57:11.493 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a3413c0:::benchmark_data_smithi179_148205_object20732:head# 2024-02-09T17:57:11.496 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a344676:::benchmark_data_smithi179_148205_object79132:head# 2024-02-09T17:57:11.496 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a35826c:::benchmark_data_smithi179_148205_object63672:head# 2024-02-09T17:57:11.496 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a35c774:::benchmark_data_smithi179_148205_object87954:head# 2024-02-09T17:57:11.496 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a3759b4:::benchmark_data_smithi179_148205_object21864:head# 2024-02-09T17:57:11.496 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a38b0f4:::benchmark_data_smithi179_148205_object19334:head# 2024-02-09T17:57:11.496 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a390bc1:::benchmark_data_smithi179_148205_object75320:head# 2024-02-09T17:57:11.497 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a393808:::benchmark_data_smithi179_148205_object35540:head# 2024-02-09T17:57:11.497 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a3a12dc:::benchmark_data_smithi179_148205_object81328:head# 2024-02-09T17:57:11.497 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a3a5f17:::benchmark_data_smithi179_148205_object87926:head# 2024-02-09T17:57:11.497 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a3ad11c:::benchmark_data_smithi179_148205_object9479:head# 2024-02-09T17:57:11.497 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a3bad73:::benchmark_data_smithi179_148205_object33821:head# 2024-02-09T17:57:11.498 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a3bbe63:::benchmark_data_smithi179_148205_object72826:head# 2024-02-09T17:57:11.498 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a3ce91a:::benchmark_data_smithi179_148205_object953:head# 2024-02-09T17:57:11.498 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a3e3f9d:::benchmark_data_smithi179_148205_object18999:head# 2024-02-09T17:57:11.499 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a3f87da:::benchmark_data_smithi179_148205_object17875:head# 2024-02-09T17:57:11.499 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a3fc3d0:::benchmark_data_smithi179_148205_object85030:head# 2024-02-09T17:57:11.499 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a402bd4:::benchmark_data_smithi179_148205_object37616:head# 2024-02-09T17:57:11.500 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a408b79:::benchmark_data_smithi179_148205_object60167:head# 2024-02-09T17:57:11.501 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a413b0e:::benchmark_data_smithi179_148205_object84727:head# 2024-02-09T17:57:11.501 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a41bdf4:::benchmark_data_smithi179_148205_object49346:head# 2024-02-09T17:57:11.501 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a41cabd:::benchmark_data_smithi179_148205_object54434:head# 2024-02-09T17:57:11.503 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a4212b1:::benchmark_data_smithi179_148205_object34771:head# 2024-02-09T17:57:11.503 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a433b8d:::benchmark_data_smithi179_148205_object26094:head# 2024-02-09T17:57:11.503 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a43477b:::benchmark_data_smithi179_148205_object84900:head# 2024-02-09T17:57:11.503 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a44a354:::benchmark_data_smithi179_148205_object3294:head# 2024-02-09T17:57:11.503 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a44f2e0:::benchmark_data_smithi179_148205_object66709:head# 2024-02-09T17:57:11.503 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a451c7a:::benchmark_data_smithi179_148205_object74511:head# 2024-02-09T17:57:11.503 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a466cc4:::benchmark_data_smithi179_148205_object54200:head# 2024-02-09T17:57:11.504 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a46afe3:::benchmark_data_smithi179_148205_object89290:head# 2024-02-09T17:57:11.504 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a46e8e9:::benchmark_data_smithi179_148205_object80569:head# 2024-02-09T17:57:11.504 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a47f4bf:::benchmark_data_smithi179_148205_object99434:head# 2024-02-09T17:57:11.505 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a481a19:::benchmark_data_smithi179_148205_object12824:head# 2024-02-09T17:57:11.505 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a48d541:::benchmark_data_smithi179_148205_object31103:head# 2024-02-09T17:57:11.505 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a4a3e47:::benchmark_data_smithi179_148205_object22239:head# 2024-02-09T17:57:11.506 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a4a7cab:::benchmark_data_smithi179_148205_object69774:head# 2024-02-09T17:57:11.506 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a4af1ea:::benchmark_data_smithi179_148205_object102865:head# 2024-02-09T17:57:11.506 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a4af4e5:::benchmark_data_smithi179_148205_object87732:head# 2024-02-09T17:57:11.507 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a4b4afb:::benchmark_data_smithi179_148205_object80236:head# 2024-02-09T17:57:11.508 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a4b71a8:::benchmark_data_smithi179_148205_object19871:head# 2024-02-09T17:57:11.509 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a4bc847:::benchmark_data_smithi179_148205_object53476:head# 2024-02-09T17:57:11.509 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a4ca14b:::benchmark_data_smithi179_148205_object41175:head# 2024-02-09T17:57:11.509 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a4e0d9b:::benchmark_data_smithi179_148205_object16989:head# 2024-02-09T17:57:11.510 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a51acf2:::benchmark_data_smithi179_148205_object100820:head# 2024-02-09T17:57:11.510 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a521c6d:::benchmark_data_smithi179_148205_object62527:head# 2024-02-09T17:57:11.510 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a52e059:::benchmark_data_smithi179_148205_object12279:head# 2024-02-09T17:57:11.511 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a535f57:::benchmark_data_smithi179_148205_object36028:head# 2024-02-09T17:57:11.511 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a53b812:::benchmark_data_smithi179_148205_object32319:head# 2024-02-09T17:57:11.511 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a546759:::benchmark_data_smithi179_148205_object39889:head# 2024-02-09T17:57:11.512 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a54c94a:::benchmark_data_smithi179_148205_object105617:head# 2024-02-09T17:57:11.513 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a5581e2:::benchmark_data_smithi179_148205_object22447:head# 2024-02-09T17:57:11.513 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a55ac90:::benchmark_data_smithi179_148205_object103560:head# 2024-02-09T17:57:11.513 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a5635ef:::benchmark_data_smithi179_148205_object7034:head# 2024-02-09T17:57:11.514 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a578d5d:::benchmark_data_smithi179_148205_object23268:head# 2024-02-09T17:57:11.514 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a579531:::benchmark_data_smithi179_148205_object73897:head# 2024-02-09T17:57:11.514 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a584abc:::benchmark_data_smithi179_148205_object7190:head# 2024-02-09T17:57:11.515 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a588bf4:::benchmark_data_smithi179_148205_object19374:head# 2024-02-09T17:57:11.515 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a59ed48:::benchmark_data_smithi179_148205_object106834:head# 2024-02-09T17:57:11.515 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a5a1fe9:::benchmark_data_smithi179_148205_object73552:head# 2024-02-09T17:57:11.516 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a5a7cb5:::benchmark_data_smithi179_148205_object70330:head# 2024-02-09T17:57:11.516 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a5aac68:::benchmark_data_smithi179_148205_object75752:head# 2024-02-09T17:57:11.517 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a5ae393:::benchmark_data_smithi179_148205_object62550:head# 2024-02-09T17:57:11.517 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a5b1167:::benchmark_data_smithi179_148205_object65337:head# 2024-02-09T17:57:11.519 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a5b6cdf:::benchmark_data_smithi179_148205_object46503:head# 2024-02-09T17:57:11.519 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a5ba8a5:::benchmark_data_smithi179_148205_object90163:head# 2024-02-09T17:57:11.524 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a5ddd07:::benchmark_data_smithi179_148205_object33205:head# 2024-02-09T17:57:11.524 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a5ea1ee:::benchmark_data_smithi179_148205_object67068:head# 2024-02-09T17:57:11.524 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a5fa907:::benchmark_data_smithi179_148205_object34638:head# 2024-02-09T17:57:11.524 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a60b8b7:::benchmark_data_smithi179_148205_object93168:head# 2024-02-09T17:57:11.524 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a619be4:::benchmark_data_smithi179_148205_object5356:head# 2024-02-09T17:57:11.525 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a63ca41:::benchmark_data_smithi179_148205_object105214:head# 2024-02-09T17:57:11.525 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a63f5f1:::benchmark_data_smithi179_148205_object15760:head# 2024-02-09T17:57:11.525 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a64d34f:::benchmark_data_smithi179_148205_object102907:head# 2024-02-09T17:57:11.525 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a653d37:::benchmark_data_smithi179_148205_object46276:head# 2024-02-09T17:57:11.525 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a65c1e4:::benchmark_data_smithi179_148205_object23823:head# 2024-02-09T17:57:11.525 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a65fac5:::benchmark_data_smithi179_148205_object47912:head# 2024-02-09T17:57:11.526 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a66140f:::benchmark_data_smithi179_148205_object9626:head# 2024-02-09T17:57:11.526 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a66cfda:::benchmark_data_smithi179_148205_object83015:head# 2024-02-09T17:57:11.526 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a66daff:::benchmark_data_smithi179_148205_object27701:head# 2024-02-09T17:57:11.526 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a6819ae:::benchmark_data_smithi179_148205_object10281:head# 2024-02-09T17:57:11.526 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a687d85:::benchmark_data_smithi179_148205_object85778:head# 2024-02-09T17:57:11.527 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a68b406:::benchmark_data_smithi179_148205_object17005:head# 2024-02-09T17:57:11.527 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a68de9d:::benchmark_data_smithi179_148205_object21096:head# 2024-02-09T17:57:11.528 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a6943e7:::benchmark_data_smithi179_148205_object84449:head# 2024-02-09T17:57:11.528 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a69528d:::benchmark_data_smithi179_148205_object11034:head# 2024-02-09T17:57:11.528 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a6a0703:::benchmark_data_smithi179_148205_object11527:head# 2024-02-09T17:57:11.529 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a6a4630:::benchmark_data_smithi179_148205_object34112:head# 2024-02-09T17:57:11.529 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a6a5570:::benchmark_data_smithi179_148205_object67385:head# 2024-02-09T17:57:11.530 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a6a7902:::benchmark_data_smithi179_148205_object78930:head# 2024-02-09T17:57:11.530 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a6aa159:::benchmark_data_smithi179_148205_object8629:head# 2024-02-09T17:57:11.530 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a6c634c:::benchmark_data_smithi179_148205_object36345:head# 2024-02-09T17:57:11.531 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a6d5dd2:::benchmark_data_smithi179_148205_object105367:head# 2024-02-09T17:57:11.531 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a6d8702:::benchmark_data_smithi179_148205_object97985:head# 2024-02-09T17:57:11.531 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a6da8e6:::benchmark_data_smithi179_148205_object83394:head# 2024-02-09T17:57:11.532 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a6dadfb:::benchmark_data_smithi179_148205_object25940:head# 2024-02-09T17:57:11.532 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a6dbd22:::benchmark_data_smithi179_148205_object66073:head# 2024-02-09T17:57:11.533 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a6ef16d:::benchmark_data_smithi179_148205_object555:head# 2024-02-09T17:57:11.533 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a6f5348:::benchmark_data_smithi179_148205_object35891:head# 2024-02-09T17:57:11.533 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a6f7e70:::benchmark_data_smithi179_148205_object102261:head# 2024-02-09T17:57:11.534 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a70d698:::benchmark_data_smithi179_148205_object48455:head# 2024-02-09T17:57:11.535 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a73d0a7:::benchmark_data_smithi179_148205_object6021:head# 2024-02-09T17:57:11.535 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a74495a:::benchmark_data_smithi179_148205_object92063:head# 2024-02-09T17:57:11.536 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a7472d8:::benchmark_data_smithi179_148205_object46563:head# 2024-02-09T17:57:11.536 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a757cd3:::benchmark_data_smithi179_148205_object102620:head# 2024-02-09T17:57:11.536 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a75c9af:::benchmark_data_smithi179_148205_object48570:head# 2024-02-09T17:57:11.537 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a76379f:::benchmark_data_smithi179_148205_object107119:head# 2024-02-09T17:57:11.537 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a770797:::benchmark_data_smithi179_148205_object4520:head# 2024-02-09T17:57:11.538 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a779395:::benchmark_data_smithi179_148205_object78788:head# 2024-02-09T17:57:11.538 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a77e77e:::benchmark_data_smithi179_148205_object23451:head# 2024-02-09T17:57:11.539 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a79e4bd:::benchmark_data_smithi179_148205_object95245:head# 2024-02-09T17:57:11.539 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a7bda73:::benchmark_data_smithi179_148205_object79781:head# 2024-02-09T17:57:11.540 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a7cacfa:::benchmark_data_smithi179_148205_object45144:head# 2024-02-09T17:57:11.540 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a7cdd7f:::benchmark_data_smithi179_148205_object29364:head# 2024-02-09T17:57:11.540 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a7d3d10:::benchmark_data_smithi179_148205_object49526:head# 2024-02-09T17:57:11.540 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a7dbb72:::benchmark_data_smithi179_148205_object55284:head# 2024-02-09T17:57:11.541 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a7e36d5:::benchmark_data_smithi179_148205_object28307:head# 2024-02-09T17:57:11.541 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a7ec137:::benchmark_data_smithi179_148205_object14370:head# 2024-02-09T17:57:11.544 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a7f49e1:::benchmark_data_smithi179_148205_object5507:head# 2024-02-09T17:57:11.545 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a802a8d:::benchmark_data_smithi179_148205_object91319:head# 2024-02-09T17:57:11.545 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a80de2f:::benchmark_data_smithi179_148205_object82523:head# 2024-02-09T17:57:11.545 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a82ff1e:::benchmark_data_smithi179_148205_object91433:head# 2024-02-09T17:57:11.545 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a83673c:::benchmark_data_smithi179_148205_object102402:head# 2024-02-09T17:57:11.545 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a851517:::benchmark_data_smithi179_148205_object31816:head# 2024-02-09T17:57:11.547 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a86bd96:::benchmark_data_smithi179_148205_object30133:head# 2024-02-09T17:57:11.549 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a88ec5b:::benchmark_data_smithi179_148205_object36949:head# 2024-02-09T17:57:11.549 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a890a1a:::benchmark_data_smithi179_148205_object31385:head# 2024-02-09T17:57:11.550 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a897778:::benchmark_data_smithi179_148205_object82002:head# 2024-02-09T17:57:11.550 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a89ef11:::benchmark_data_smithi179_148205_object13110:head# 2024-02-09T17:57:11.552 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a8b244a:::benchmark_data_smithi179_148205_object70172:head# 2024-02-09T17:57:11.554 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a8b26b2:::benchmark_data_smithi179_148205_object65513:head# 2024-02-09T17:57:11.560 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a8bea57:::benchmark_data_smithi179_148205_object86659:head# 2024-02-09T17:57:11.560 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a8c1db9:::benchmark_data_smithi179_148205_object105315:head# 2024-02-09T17:57:11.560 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a8c3e9a:::benchmark_data_smithi179_148205_object46519:head# 2024-02-09T17:57:11.560 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a8c6c2f:::benchmark_data_smithi179_148205_object32890:head# 2024-02-09T17:57:11.560 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a8c86fe:::benchmark_data_smithi179_148205_object20431:head# 2024-02-09T17:57:11.561 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a8c9b4f:::benchmark_data_smithi179_148205_object59915:head# 2024-02-09T17:57:11.562 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a8cd2eb:::benchmark_data_smithi179_148205_object61985:head# 2024-02-09T17:57:11.563 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a8cd6fc:::benchmark_data_smithi179_148205_object87679:head# 2024-02-09T17:57:11.563 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a8dbbc6:::benchmark_data_smithi179_148205_object53411:head# 2024-02-09T17:57:11.564 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a8e9781:::benchmark_data_smithi179_148205_object94734:head# 2024-02-09T17:57:11.564 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a8f4808:::benchmark_data_smithi179_148205_object31860:head# 2024-02-09T17:57:11.564 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a8fec8d:::benchmark_data_smithi179_148205_object68857:head# 2024-02-09T17:57:11.565 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a903d67:::benchmark_data_smithi179_148205_object5890:head# 2024-02-09T17:57:11.565 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a90862a:::benchmark_data_smithi179_148205_object20731:head# 2024-02-09T17:57:11.566 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a91518b:::benchmark_data_smithi179_148205_object67351:head# 2024-02-09T17:57:11.569 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a916fd8:::benchmark_data_smithi179_148205_object39142:head# 2024-02-09T17:57:11.593 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a929bb5:::benchmark_data_smithi179_148205_object48830:head# 2024-02-09T17:57:11.593 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a94a8aa:::benchmark_data_smithi179_148205_object89747:head# 2024-02-09T17:57:11.593 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a94e928:::benchmark_data_smithi179_148205_object52429:head# 2024-02-09T17:57:11.593 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a94f691:::benchmark_data_smithi179_148205_object5048:head# 2024-02-09T17:57:11.593 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a9503c2:::benchmark_data_smithi179_148205_object76241:head# 2024-02-09T17:57:11.593 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a954c6c:::benchmark_data_smithi179_148205_object63510:head# 2024-02-09T17:57:11.593 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a96234c:::benchmark_data_smithi179_148205_object29778:head# 2024-02-09T17:57:11.593 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a966b97:::benchmark_data_smithi179_148205_object66899:head# 2024-02-09T17:57:11.593 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a9720ca:::benchmark_data_smithi179_148205_object70643:head# 2024-02-09T17:57:11.593 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a977e31:::benchmark_data_smithi179_148205_object22205:head# 2024-02-09T17:57:11.594 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a97a224:::benchmark_data_smithi179_148205_object89576:head# 2024-02-09T17:57:11.594 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a980ac3:::benchmark_data_smithi179_148205_object18391:head# 2024-02-09T17:57:11.594 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a982cbb:::benchmark_data_smithi179_148205_object53721:head# 2024-02-09T17:57:11.594 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a983d39:::benchmark_data_smithi179_148205_object81528:head# 2024-02-09T17:57:11.594 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a9888d2:::benchmark_data_smithi179_148205_object55666:head# 2024-02-09T17:57:11.594 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a98f090:::benchmark_data_smithi179_148205_object28947:head# 2024-02-09T17:57:11.594 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a991353:::benchmark_data_smithi179_148205_object60740:head# 2024-02-09T17:57:11.594 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a996e4f:::benchmark_data_smithi179_148205_object50520:head# 2024-02-09T17:57:11.594 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a9a6430:::benchmark_data_smithi179_148205_object99216:head# 2024-02-09T17:57:11.594 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a9b43cc:::benchmark_data_smithi179_148205_object85016:head# 2024-02-09T17:57:11.594 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a9ccb93:::benchmark_data_smithi179_148205_object100769:head# 2024-02-09T17:57:11.594 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a9d5acd:::benchmark_data_smithi179_148205_object3321:head# 2024-02-09T17:57:11.596 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a9d9d22:::benchmark_data_smithi179_148205_object52242:head# 2024-02-09T17:57:11.596 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a9dd2d7:::benchmark_data_smithi179_148205_object34998:head# 2024-02-09T17:57:11.596 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a9de330:::benchmark_data_smithi179_148205_object10093:head# 2024-02-09T17:57:11.596 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a9ec621:::benchmark_data_smithi179_148205_object101349:head# 2024-02-09T17:57:11.596 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a9f5a2d:::benchmark_data_smithi179_148205_object106879:head# 2024-02-09T17:57:11.596 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1a9f99ca:::benchmark_data_smithi179_148205_object52118:head# 2024-02-09T17:57:11.596 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1aa1dc13:::benchmark_data_smithi179_148205_object96389:head# 2024-02-09T17:57:11.596 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1aa23468:::benchmark_data_smithi179_148205_object103125:head# 2024-02-09T17:57:11.597 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1aa25c7c:::benchmark_data_smithi179_148205_object51444:head# 2024-02-09T17:57:11.598 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1aa3d7f5:::benchmark_data_smithi179_148205_object92735:head# 2024-02-09T17:57:11.598 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1aa4857f:::benchmark_data_smithi179_148205_object40793:head# 2024-02-09T17:57:11.598 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1aa49170:::benchmark_data_smithi179_148205_object26035:head# 2024-02-09T17:57:11.599 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1aa4d6b5:::benchmark_data_smithi179_148205_object9969:head# 2024-02-09T17:57:11.599 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1aa55df3:::benchmark_data_smithi179_148205_object41426:head# 2024-02-09T17:57:11.600 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1aa5f7eb:::benchmark_data_smithi179_148205_object97450:head# 2024-02-09T17:57:11.600 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1aa65b16:::benchmark_data_smithi179_148205_object33114:head# 2024-02-09T17:57:11.605 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1aa747f0:::benchmark_data_smithi179_148205_object66661:head# 2024-02-09T17:57:11.605 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1aa76383:::benchmark_data_smithi179_148205_object48152:head# 2024-02-09T17:57:11.605 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1aa78783:::benchmark_data_smithi179_148205_object23877:head# 2024-02-09T17:57:11.605 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1aa7bea3:::benchmark_data_smithi179_148205_object76981:head# 2024-02-09T17:57:11.606 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1aa7eaa3:::benchmark_data_smithi179_148205_object50559:head# 2024-02-09T17:57:11.606 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1aa8324b:::benchmark_data_smithi179_148205_object42122:head# 2024-02-09T17:57:11.607 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1aa880d6:::benchmark_data_smithi179_148205_object81231:head# 2024-02-09T17:57:11.610 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1aa90b8a:::benchmark_data_smithi179_148205_object88002:head# 2024-02-09T17:57:11.610 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1aa91f05:::benchmark_data_smithi179_148205_object71404:head# 2024-02-09T17:57:11.610 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1aabb6f9:::benchmark_data_smithi179_148205_object63814:head# 2024-02-09T17:57:11.611 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1aac3f5d:::benchmark_data_smithi179_148205_object70038:head# 2024-02-09T17:57:11.611 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1aac57e5:::benchmark_data_smithi179_148205_object26113:head# 2024-02-09T17:57:11.611 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1aac5e1e:::benchmark_data_smithi179_148205_object4008:head# 2024-02-09T17:57:11.616 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1aac8a7a:::benchmark_data_smithi179_148205_object20285:head# 2024-02-09T17:57:11.616 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1aaca766:::benchmark_data_smithi179_148205_object10484:head# 2024-02-09T17:57:11.616 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1aacc1af:::benchmark_data_smithi179_148205_object13454:head# 2024-02-09T17:57:11.620 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1aafb978:::benchmark_data_smithi179_148205_object33385:head# 2024-02-09T17:57:11.620 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ab022f3:::benchmark_data_smithi179_148205_object42511:head# 2024-02-09T17:57:11.620 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ab0eec1:::benchmark_data_smithi179_148205_object14559:head# 2024-02-09T17:57:11.620 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ab43cf0:::benchmark_data_smithi179_148205_object7137:head# 2024-02-09T17:57:11.621 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ab4e85e:::benchmark_data_smithi179_148205_object1592:head# 2024-02-09T17:57:11.621 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ab564dd:::benchmark_data_smithi179_148205_object25165:head# 2024-02-09T17:57:11.621 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ab69107:::benchmark_data_smithi179_148205_object44797:head# 2024-02-09T17:57:11.621 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ab6cbd0:::benchmark_data_smithi179_148205_object21296:head# 2024-02-09T17:57:11.621 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ab713f6:::benchmark_data_smithi179_148205_object72829:head# 2024-02-09T17:57:11.624 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ab93b25:::benchmark_data_smithi179_148205_object106225:head# 2024-02-09T17:57:11.626 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ab9e7cf:::benchmark_data_smithi179_148205_object99580:head# 2024-02-09T17:57:11.626 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1aba2587:::benchmark_data_smithi179_148205_object54161:head# 2024-02-09T17:57:11.626 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1abb1131:::benchmark_data_smithi179_148205_object83390:head# 2024-02-09T17:57:11.626 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1abb7a1d:::benchmark_data_smithi179_148205_object940:head# 2024-02-09T17:57:11.626 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1abc0b44:::benchmark_data_smithi179_148205_object92457:head# 2024-02-09T17:57:11.627 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1abc28cc:::benchmark_data_smithi179_148205_object70151:head# 2024-02-09T17:57:11.627 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1abd25ac:::benchmark_data_smithi179_148205_object14579:head# 2024-02-09T17:57:11.627 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1abd3b65:::benchmark_data_smithi179_148205_object102740:head# 2024-02-09T17:57:11.627 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1abd46b0:::benchmark_data_smithi179_148205_object1862:head# 2024-02-09T17:57:11.628 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1abd894e:::benchmark_data_smithi179_148205_object25630:head# 2024-02-09T17:57:11.628 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1abdbac2:::benchmark_data_smithi179_148205_object89527:head# 2024-02-09T17:57:11.628 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1abe4c2e:::benchmark_data_smithi179_148205_object44699:head# 2024-02-09T17:57:11.629 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1abf3c0b:::benchmark_data_smithi179_148205_object59780:head# 2024-02-09T17:57:11.630 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1abfa864:::benchmark_data_smithi179_148205_object70137:head# 2024-02-09T17:57:11.630 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1abfca1c:::benchmark_data_smithi179_148205_object38303:head# 2024-02-09T17:57:11.631 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ac08325:::benchmark_data_smithi179_148205_object58440:head# 2024-02-09T17:57:11.631 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ac08e14:::benchmark_data_smithi179_148205_object109678:head# 2024-02-09T17:57:11.631 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ac1a74b:::benchmark_data_smithi179_148205_object21477:head# 2024-02-09T17:57:11.632 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ac1bbe4:::benchmark_data_smithi179_148205_object12072:head# 2024-02-09T17:57:11.633 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ac314a5:::benchmark_data_smithi179_148205_object59840:head# 2024-02-09T17:57:11.633 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ac32cc9:::benchmark_data_smithi179_148205_object99893:head# 2024-02-09T17:57:11.634 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ac3f57e:::benchmark_data_smithi179_148205_object87204:head# 2024-02-09T17:57:11.634 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ac4876b:::benchmark_data_smithi179_148205_object57442:head# 2024-02-09T17:57:11.635 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ac4baf2:::benchmark_data_smithi179_148205_object28205:head# 2024-02-09T17:57:11.635 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ac4de13:::benchmark_data_smithi179_148205_object14813:head# 2024-02-09T17:57:11.635 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ac67a49:::benchmark_data_smithi179_148205_object28126:head# 2024-02-09T17:57:11.636 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ac7b6a1:::benchmark_data_smithi179_148205_object97791:head# 2024-02-09T17:57:11.636 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ac8393b:::benchmark_data_smithi179_148205_object74390:head# 2024-02-09T17:57:11.637 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1aca1424:::benchmark_data_smithi179_148205_object55171:head# 2024-02-09T17:57:11.637 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1acb3483:::benchmark_data_smithi179_148205_object14759:head# 2024-02-09T17:57:11.637 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1acb5001:::benchmark_data_smithi179_148205_object24730:head# 2024-02-09T17:57:11.638 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1acb9043:::benchmark_data_smithi179_148205_object66680:head# 2024-02-09T17:57:11.638 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1acba662:::benchmark_data_smithi179_148205_object108917:head# 2024-02-09T17:57:11.639 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1acbd1a1:::benchmark_data_smithi179_148205_object12638:head# 2024-02-09T17:57:11.642 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1acc3d39:::benchmark_data_smithi179_148205_object36113:head# 2024-02-09T17:57:11.642 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1acd47c8:::benchmark_data_smithi179_148205_object98906:head# 2024-02-09T17:57:11.642 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1acd5cb7:::benchmark_data_smithi179_148205_object30164:head# 2024-02-09T17:57:11.642 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ace0106:::benchmark_data_smithi179_148205_object51476:head# 2024-02-09T17:57:11.642 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ace6226:::benchmark_data_smithi179_148205_object16577:head# 2024-02-09T17:57:11.642 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ace72be:::benchmark_data_smithi179_148205_object62093:head# 2024-02-09T17:57:11.642 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1acf0d89:::benchmark_data_smithi179_148205_object100703:head# 2024-02-09T17:57:11.643 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1acfcb48:::benchmark_data_smithi179_148205_object20089:head# 2024-02-09T17:57:11.643 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1acfda36:::benchmark_data_smithi179_148205_object5318:head# 2024-02-09T17:57:11.643 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ad0754d:::benchmark_data_smithi179_148205_object82558:head# 2024-02-09T17:57:11.644 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ad0f99f:::benchmark_data_smithi179_148205_object84214:head# 2024-02-09T17:57:11.646 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ad1f819:::benchmark_data_smithi179_148205_object102887:head# 2024-02-09T17:57:11.646 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ad1fecc:::benchmark_data_smithi179_148205_object53926:head# 2024-02-09T17:57:11.646 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ad452d6:::benchmark_data_smithi179_148205_object11511:head# 2024-02-09T17:57:11.649 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ad4fefe:::benchmark_data_smithi179_148205_object30694:head# 2024-02-09T17:57:11.650 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ad51533:::benchmark_data_smithi179_148205_object74416:head# 2024-02-09T17:57:11.650 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ad601c4:::benchmark_data_smithi179_148205_object49602:head# 2024-02-09T17:57:11.650 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ad64303:::benchmark_data_smithi179_148205_object36453:head# 2024-02-09T17:57:11.650 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ad74ea9:::benchmark_data_smithi179_148205_object44390:head# 2024-02-09T17:57:11.650 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ad85907:::benchmark_data_smithi179_148205_object70519:head# 2024-02-09T17:57:11.650 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ad8e0e0:::benchmark_data_smithi179_148205_object16054:head# 2024-02-09T17:57:11.650 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ad9917e:::benchmark_data_smithi179_148205_object85662:head# 2024-02-09T17:57:11.650 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ada0d0b:::benchmark_data_smithi179_148205_object67526:head# 2024-02-09T17:57:11.650 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ada0e62:::benchmark_data_smithi179_148205_object5895:head# 2024-02-09T17:57:11.651 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ada1620:::benchmark_data_smithi179_148205_object46378:head# 2024-02-09T17:57:11.651 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1adaedce:::benchmark_data_smithi179_148205_object67807:head# 2024-02-09T17:57:11.651 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1adc1433:::benchmark_data_smithi179_148205_object49166:head# 2024-02-09T17:57:11.651 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1adc2022:::benchmark_data_smithi179_148205_object6712:head# 2024-02-09T17:57:11.652 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1adcb204:::benchmark_data_smithi179_148205_object6033:head# 2024-02-09T17:57:11.652 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1add78a0:::benchmark_data_smithi179_148205_object96653:head# 2024-02-09T17:57:11.653 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1addcc6d:::benchmark_data_smithi179_148205_object37496:head# 2024-02-09T17:57:11.653 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ade6b17:::benchmark_data_smithi179_148205_object15646:head# 2024-02-09T17:57:11.653 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1adea294:::benchmark_data_smithi179_148205_object14415:head# 2024-02-09T17:57:11.654 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1adee34a:::benchmark_data_smithi179_148205_object99635:head# 2024-02-09T17:57:11.654 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1adf7604:::benchmark_data_smithi179_148205_object33888:head# 2024-02-09T17:57:11.654 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ae08287:::benchmark_data_smithi179_148205_object10779:head# 2024-02-09T17:57:11.655 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ae08607:::benchmark_data_smithi179_148205_object82030:head# 2024-02-09T17:57:11.655 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ae2a9f0:::benchmark_data_smithi179_148205_object12600:head# 2024-02-09T17:57:11.656 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ae3b24b:::benchmark_data_smithi179_148205_object82937:head# 2024-02-09T17:57:11.656 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ae3f5a4:::benchmark_data_smithi179_148205_object14221:head# 2024-02-09T17:57:11.656 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ae4e743:::benchmark_data_smithi179_148205_object3759:head# 2024-02-09T17:57:11.657 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ae547e4:::benchmark_data_smithi179_148205_object47844:head# 2024-02-09T17:57:11.657 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ae5bc55:::benchmark_data_smithi179_148205_object13585:head# 2024-02-09T17:57:11.657 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ae61fda:::benchmark_data_smithi179_148205_object77783:head# 2024-02-09T17:57:11.658 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ae625e4:::benchmark_data_smithi179_148205_object3424:head# 2024-02-09T17:57:11.658 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ae6398e:::benchmark_data_smithi179_148205_object97790:head# 2024-02-09T17:57:11.659 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ae6d79d:::benchmark_data_smithi179_148205_object80583:head# 2024-02-09T17:57:11.659 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ae79b90:::benchmark_data_smithi179_148205_object67983:head# 2024-02-09T17:57:11.659 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ae87f35:::benchmark_data_smithi179_148205_object39181:head# 2024-02-09T17:57:11.660 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ae8ea23:::benchmark_data_smithi179_148205_object35818:head# 2024-02-09T17:57:11.660 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ae9f3ed:::benchmark_data_smithi179_148205_object104629:head# 2024-02-09T17:57:11.660 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1aea3cc7:::benchmark_data_smithi179_148205_object41780:head# 2024-02-09T17:57:11.661 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1aea9d97:::benchmark_data_smithi179_148205_object46774:head# 2024-02-09T17:57:11.661 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1aeabcf2:::benchmark_data_smithi179_148205_object55458:head# 2024-02-09T17:57:11.661 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1aeefa25:::benchmark_data_smithi179_148205_object104919:head# 2024-02-09T17:57:11.662 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1aef9c81:::benchmark_data_smithi179_148205_object56925:head# 2024-02-09T17:57:11.662 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1af0cfad:::benchmark_data_smithi179_148205_object100074:head# 2024-02-09T17:57:11.663 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1af15b8c:::benchmark_data_smithi179_148205_object65391:head# 2024-02-09T17:57:11.664 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1af1be32:::benchmark_data_smithi179_148205_object76872:head# 2024-02-09T17:57:11.664 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1af2e849:::benchmark_data_smithi179_148205_object89954:head# 2024-02-09T17:57:11.664 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1af331ee:::benchmark_data_smithi179_148205_object67030:head# 2024-02-09T17:57:11.664 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1af3dd81:::benchmark_data_smithi179_148205_object57735:head# 2024-02-09T17:57:11.664 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1af4846a:::benchmark_data_smithi179_148205_object83064:head# 2024-02-09T17:57:11.665 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1af50d3a:::benchmark_data_smithi179_148205_object47160:head# 2024-02-09T17:57:11.665 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1af53714:::benchmark_data_smithi179_148205_object4382:head# 2024-02-09T17:57:11.665 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1af590b6:::benchmark_data_smithi179_148205_object4752:head# 2024-02-09T17:57:11.666 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1af5e5ae:::benchmark_data_smithi179_148205_object16240:head# 2024-02-09T17:57:11.666 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1af69eed:::benchmark_data_smithi179_148205_object2286:head# 2024-02-09T17:57:11.666 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1af6a90b:::benchmark_data_smithi179_148205_object30905:head# 2024-02-09T17:57:11.667 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1af71c81:::benchmark_data_smithi179_148205_object12213:head# 2024-02-09T17:57:11.667 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1af7cbfa:::benchmark_data_smithi179_148205_object25018:head# 2024-02-09T17:57:11.667 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1af810b0:::benchmark_data_smithi179_148205_object30631:head# 2024-02-09T17:57:11.668 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1af84146:::benchmark_data_smithi179_148205_object46121:head# 2024-02-09T17:57:11.668 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1af84768:::benchmark_data_smithi179_148205_object45708:head# 2024-02-09T17:57:11.669 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1af8b6c9:::benchmark_data_smithi179_148205_object10700:head# 2024-02-09T17:57:11.669 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1af9418c:::benchmark_data_smithi179_148205_object37516:head# 2024-02-09T17:57:11.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:11 smithi162 ceph-mon[44304]: osdmap e824: 12 total, 11 up, 12 in 2024-02-09T17:57:11.677 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1afbac93:::benchmark_data_smithi179_148205_object87704:head# 2024-02-09T17:57:11.677 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1afcf328:::benchmark_data_smithi179_148205_object35147:head# 2024-02-09T17:57:11.677 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1afe1a35:::benchmark_data_smithi179_148205_object17812:head# 2024-02-09T17:57:11.677 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1afe6aec:::benchmark_data_smithi179_148205_object4057:head# 2024-02-09T17:57:11.677 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1afe9ca6:::benchmark_data_smithi179_148205_object109142:head# 2024-02-09T17:57:11.677 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b00318c:::benchmark_data_smithi179_148205_object34202:head# 2024-02-09T17:57:11.677 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b00f26b:::benchmark_data_smithi179_148205_object46523:head# 2024-02-09T17:57:11.678 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b0103be:::benchmark_data_smithi179_148205_object74072:head# 2024-02-09T17:57:11.678 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b02ff45:::benchmark_data_smithi179_148205_object44913:head# 2024-02-09T17:57:11.678 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b03481b:::benchmark_data_smithi179_148205_object36126:head# 2024-02-09T17:57:11.678 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b037e5e:::benchmark_data_smithi179_148205_object91765:head# 2024-02-09T17:57:11.678 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b038250:::benchmark_data_smithi179_148205_object29431:head# 2024-02-09T17:57:11.678 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b0403a6:::benchmark_data_smithi179_148205_object26722:head# 2024-02-09T17:57:11.678 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b046f86:::benchmark_data_smithi179_148205_object13184:head# 2024-02-09T17:57:11.679 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b0547a9:::benchmark_data_smithi179_148205_object107134:head# 2024-02-09T17:57:11.679 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b05d297:::benchmark_data_smithi179_148205_object86136:head# 2024-02-09T17:57:11.679 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b069651:::benchmark_data_smithi179_148205_object70843:head# 2024-02-09T17:57:11.679 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b0838f2:::benchmark_data_smithi179_148205_object31463:head# 2024-02-09T17:57:11.683 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b0867e1:::benchmark_data_smithi179_148205_object73255:head# 2024-02-09T17:57:11.683 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b089028:::benchmark_data_smithi179_148205_object22391:head# 2024-02-09T17:57:11.683 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b08ed04:::benchmark_data_smithi179_148205_object50749:head# 2024-02-09T17:57:11.683 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b09689b:::benchmark_data_smithi179_148205_object31108:head# 2024-02-09T17:57:11.683 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b096ef3:::benchmark_data_smithi179_148205_object31445:head# 2024-02-09T17:57:11.683 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b0a330f:::benchmark_data_smithi179_148205_object59593:head# 2024-02-09T17:57:11.683 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b0abef0:::benchmark_data_smithi179_148205_object3896:head# 2024-02-09T17:57:11.683 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b0ac07a:::benchmark_data_smithi179_148205_object62655:head# 2024-02-09T17:57:11.683 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b0ca1bd:::benchmark_data_smithi179_148205_object97817:head# 2024-02-09T17:57:11.685 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b0d1ba4:::benchmark_data_smithi179_148205_object85325:head# 2024-02-09T17:57:11.685 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b0e45d1:::benchmark_data_smithi179_148205_object102360:head# 2024-02-09T17:57:11.685 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b102cd9:::benchmark_data_smithi179_148205_object83360:head# 2024-02-09T17:57:11.686 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b12354f:::benchmark_data_smithi179_148205_object9843:head# 2024-02-09T17:57:11.686 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b13a624:::benchmark_data_smithi179_148205_object50241:head# 2024-02-09T17:57:11.687 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b13f85a:::benchmark_data_smithi179_148205_object72388:head# 2024-02-09T17:57:11.687 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b149a08:::benchmark_data_smithi179_148205_object89137:head# 2024-02-09T17:57:11.687 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b14e3bb:::benchmark_data_smithi179_148205_object101891:head# 2024-02-09T17:57:11.688 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b165ecb:::benchmark_data_smithi179_148205_object108039:head# 2024-02-09T17:57:11.689 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b17b18f:::benchmark_data_smithi179_148205_object60009:head# 2024-02-09T17:57:11.689 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b17ee64:::benchmark_data_smithi179_148205_object72331:head# 2024-02-09T17:57:11.689 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b18098e:::benchmark_data_smithi179_148205_object78858:head# 2024-02-09T17:57:11.690 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b18c072:::benchmark_data_smithi179_148205_object64724:head# 2024-02-09T17:57:11.690 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b18e0fe:::benchmark_data_smithi179_148205_object50234:head# 2024-02-09T17:57:11.691 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b199325:::benchmark_data_smithi179_148205_object103528:head# 2024-02-09T17:57:11.691 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b1ad085:::benchmark_data_smithi179_148205_object71365:head# 2024-02-09T17:57:11.692 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b1b0cd6:::benchmark_data_smithi179_148205_object18339:head# 2024-02-09T17:57:11.692 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b1b3f4c:::benchmark_data_smithi179_148205_object104662:head# 2024-02-09T17:57:11.693 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b1beaac:::benchmark_data_smithi179_148205_object33596:head# 2024-02-09T17:57:11.693 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b1ce623:::benchmark_data_smithi179_148205_object34947:head# 2024-02-09T17:57:11.694 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b1d0b38:::benchmark_data_smithi179_148205_object31832:head# 2024-02-09T17:57:11.695 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b1d12ea:::benchmark_data_smithi179_148205_object6611:head# 2024-02-09T17:57:11.696 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b1d2360:::benchmark_data_smithi179_148205_object30863:head# 2024-02-09T17:57:11.699 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b1d7c16:::benchmark_data_smithi179_148205_object55327:head# 2024-02-09T17:57:11.700 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b1e21de:::benchmark_data_smithi179_148205_object14974:head# 2024-02-09T17:57:11.700 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b1e59c8:::benchmark_data_smithi179_148205_object4141:head# 2024-02-09T17:57:11.700 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b1f5577:::benchmark_data_smithi179_148205_object101671:head# 2024-02-09T17:57:11.701 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b1fb475:::benchmark_data_smithi179_148205_object22966:head# 2024-02-09T17:57:11.701 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b20d7c4:::benchmark_data_smithi179_148205_object30500:head# 2024-02-09T17:57:11.706 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b20eb40:::benchmark_data_smithi179_148205_object89215:head# 2024-02-09T17:57:11.706 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b2105fc:::benchmark_data_smithi179_148205_object274:head# 2024-02-09T17:57:11.706 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b21a6eb:::benchmark_data_smithi179_148205_object6382:head# 2024-02-09T17:57:11.706 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b21e9d1:::benchmark_data_smithi179_148205_object104360:head# 2024-02-09T17:57:11.706 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b225d23:::benchmark_data_smithi179_148205_object36759:head# 2024-02-09T17:57:11.706 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b22f0ea:::benchmark_data_smithi179_148205_object20427:head# 2024-02-09T17:57:11.706 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b231ded:::benchmark_data_smithi179_148205_object38748:head# 2024-02-09T17:57:11.706 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b233669:::benchmark_data_smithi179_148205_object22487:head# 2024-02-09T17:57:11.706 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b24fad6:::benchmark_data_smithi179_148205_object98568:head# 2024-02-09T17:57:11.707 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b256e8b:::benchmark_data_smithi179_148205_object75511:head# 2024-02-09T17:57:11.707 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b268ae8:::benchmark_data_smithi179_148205_object71532:head# 2024-02-09T17:57:11.707 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b26f0e0:::benchmark_data_smithi179_148205_object24143:head# 2024-02-09T17:57:11.709 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b286ac6:::benchmark_data_smithi179_148205_object87063:head# 2024-02-09T17:57:11.710 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b2881c5:::benchmark_data_smithi179_148205_object3979:head# 2024-02-09T17:57:11.710 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b2a1497:::benchmark_data_smithi179_148205_object44349:head# 2024-02-09T17:57:11.711 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b2a6491:::benchmark_data_smithi179_148205_object5716:head# 2024-02-09T17:57:11.711 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b2a7bd1:::benchmark_data_smithi179_148205_object70809:head# 2024-02-09T17:57:11.712 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b2a9f1c:::benchmark_data_smithi179_148205_object79513:head# 2024-02-09T17:57:11.712 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b2b013f:::benchmark_data_smithi179_148205_object23688:head# 2024-02-09T17:57:11.713 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b2b0e92:::benchmark_data_smithi179_148205_object41499:head# 2024-02-09T17:57:11.713 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b2c650b:::benchmark_data_smithi179_148205_object43855:head# 2024-02-09T17:57:11.714 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b2d9b39:::benchmark_data_smithi179_148205_object55545:head# 2024-02-09T17:57:11.714 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b2e1a66:::benchmark_data_smithi179_148205_object18409:head# 2024-02-09T17:57:11.714 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b2f65e9:::benchmark_data_smithi179_148205_object101333:head# 2024-02-09T17:57:11.715 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b30b1d1:::benchmark_data_smithi179_148205_object11948:head# 2024-02-09T17:57:11.715 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b30f72a:::benchmark_data_smithi179_148205_object92405:head# 2024-02-09T17:57:11.715 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b325739:::benchmark_data_smithi179_148205_object33490:head# 2024-02-09T17:57:11.716 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b334157:::benchmark_data_smithi179_148205_object97093:head# 2024-02-09T17:57:11.716 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b33c2f9:::benchmark_data_smithi179_148205_object109038:head# 2024-02-09T17:57:11.717 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b33d214:::benchmark_data_smithi179_148205_object100027:head# 2024-02-09T17:57:11.720 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b341acb:::benchmark_data_smithi179_148205_object85614:head# 2024-02-09T17:57:11.721 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b3609cc:::benchmark_data_smithi179_148205_object82336:head# 2024-02-09T17:57:11.721 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b36a055:::benchmark_data_smithi179_148205_object32814:head# 2024-02-09T17:57:11.721 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b36ae02:::benchmark_data_smithi179_148205_object4039:head# 2024-02-09T17:57:11.721 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b385791:::benchmark_data_smithi179_148205_object90862:head# 2024-02-09T17:57:11.721 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b397237:::benchmark_data_smithi179_148205_object14751:head# 2024-02-09T17:57:11.721 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b3991d9:::benchmark_data_smithi179_148205_object19625:head# 2024-02-09T17:57:11.721 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b3a2ed3:::benchmark_data_smithi179_148205_object11298:head# 2024-02-09T17:57:11.721 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b3a7261:::benchmark_data_smithi179_148205_object84424:head# 2024-02-09T17:57:11.722 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b3a9068:::benchmark_data_smithi179_148205_object15688:head# 2024-02-09T17:57:11.722 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b3aac2f:::benchmark_data_smithi179_148205_object7563:head# 2024-02-09T17:57:11.722 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b3afb13:::benchmark_data_smithi179_148205_object60269:head# 2024-02-09T17:57:11.723 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b3afd0b:::benchmark_data_smithi179_148205_object81980:head# 2024-02-09T17:57:11.723 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b3b319e:::benchmark_data_smithi179_148205_object89311:head# 2024-02-09T17:57:11.723 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b3b9e64:::benchmark_data_smithi179_148205_object39820:head# 2024-02-09T17:57:11.723 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b3c8dec:::benchmark_data_smithi179_148205_object8812:head# 2024-02-09T17:57:11.724 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b3d308b:::benchmark_data_smithi179_148205_object44796:head# 2024-02-09T17:57:11.724 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b3d48e6:::benchmark_data_smithi179_148205_object17771:head# 2024-02-09T17:57:11.724 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b3d7b0c:::benchmark_data_smithi179_148205_object2089:head# 2024-02-09T17:57:11.727 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b3d865c:::benchmark_data_smithi179_148205_object35576:head# 2024-02-09T17:57:11.729 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b3f1d42:::benchmark_data_smithi179_148205_object45512:head# 2024-02-09T17:57:11.729 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b3f8ca8:::benchmark_data_smithi179_148205_object56649:head# 2024-02-09T17:57:11.729 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b400a58:::benchmark_data_smithi179_148205_object25741:head# 2024-02-09T17:57:11.730 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b4011d4:::benchmark_data_smithi179_148205_object55548:head# 2024-02-09T17:57:11.730 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b4046f6:::benchmark_data_smithi179_148205_object24616:head# 2024-02-09T17:57:11.730 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b406863:::benchmark_data_smithi179_148205_object61386:head# 2024-02-09T17:57:11.730 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b409561:::benchmark_data_smithi179_148205_object7189:head# 2024-02-09T17:57:11.731 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b40dc4e:::benchmark_data_smithi179_148205_object39430:head# 2024-02-09T17:57:11.731 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b414b7b:::benchmark_data_smithi179_148205_object22409:head# 2024-02-09T17:57:11.732 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b416018:::benchmark_data_smithi179_148205_object71667:head# 2024-02-09T17:57:11.732 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b4214cd:::benchmark_data_smithi179_148205_object5705:head# 2024-02-09T17:57:11.732 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b421a55:::benchmark_data_smithi179_148205_object29807:head# 2024-02-09T17:57:11.733 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b4228b6:::benchmark_data_smithi179_148205_object18944:head# 2024-02-09T17:57:11.733 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b42f84b:::benchmark_data_smithi179_148205_object58458:head# 2024-02-09T17:57:11.733 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b438936:::benchmark_data_smithi179_148205_object97923:head# 2024-02-09T17:57:11.734 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b44afdf:::benchmark_data_smithi179_148205_object97149:head# 2024-02-09T17:57:11.734 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b4502ed:::benchmark_data_smithi179_148205_object93977:head# 2024-02-09T17:57:11.737 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b46305e:::benchmark_data_smithi179_148205_object84295:head# 2024-02-09T17:57:11.738 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b4702f3:::benchmark_data_smithi179_148205_object76337:head# 2024-02-09T17:57:11.738 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b47388d:::benchmark_data_smithi179_148205_object94769:head# 2024-02-09T17:57:11.738 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b47b886:::benchmark_data_smithi179_148205_object108313:head# 2024-02-09T17:57:11.739 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b485603:::benchmark_data_smithi179_148205_object44644:head# 2024-02-09T17:57:11.739 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b49491d:::benchmark_data_smithi179_148205_object74471:head# 2024-02-09T17:57:11.740 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b4a00e3:::benchmark_data_smithi179_148205_object35832:head# 2024-02-09T17:57:11.740 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b4a6926:::benchmark_data_smithi179_148205_object4393:head# 2024-02-09T17:57:11.740 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b4aa13c:::benchmark_data_smithi179_148205_object106479:head# 2024-02-09T17:57:11.741 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b4c928f:::benchmark_data_smithi179_148205_object79424:head# 2024-02-09T17:57:11.745 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b4cbfdd:::benchmark_data_smithi179_148205_object108681:head# 2024-02-09T17:57:11.746 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b4ccb03:::benchmark_data_smithi179_148205_object91497:head# 2024-02-09T17:57:11.746 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b4e2fbb:::benchmark_data_smithi179_148205_object16173:head# 2024-02-09T17:57:11.746 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b4eb52b:::benchmark_data_smithi179_148205_object16084:head# 2024-02-09T17:57:11.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:11 smithi099 ceph-mon[39420]: osdmap e824: 12 total, 11 up, 12 in 2024-02-09T17:57:11.747 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b4ebbc8:::benchmark_data_smithi179_148205_object22781:head# 2024-02-09T17:57:11.748 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b4ede4c:::benchmark_data_smithi179_148205_object61045:head# 2024-02-09T17:57:11.748 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b4f15a5:::benchmark_data_smithi179_148205_object104008:head# 2024-02-09T17:57:11.749 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b502157:::benchmark_data_smithi179_148205_object96925:head# 2024-02-09T17:57:11.749 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b507795:::benchmark_data_smithi179_148205_object103537:head# 2024-02-09T17:57:11.750 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b507aa2:::benchmark_data_smithi179_148205_object26710:head# 2024-02-09T17:57:11.750 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b50df2a:::benchmark_data_smithi179_148205_object89662:head# 2024-02-09T17:57:11.751 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b51a695:::benchmark_data_smithi179_148205_object25168:head# 2024-02-09T17:57:11.751 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b51fef0:::benchmark_data_smithi179_148205_object58951:head# 2024-02-09T17:57:11.751 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b525455:::benchmark_data_smithi179_148205_object52699:head# 2024-02-09T17:57:11.753 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b5278b1:::benchmark_data_smithi179_148205_object2443:head# 2024-02-09T17:57:11.753 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b5349a0:::benchmark_data_smithi179_148205_object5617:head# 2024-02-09T17:57:11.753 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b548d9c:::benchmark_data_smithi179_148205_object59628:head# 2024-02-09T17:57:11.754 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b54976a:::benchmark_data_smithi179_148205_object19501:head# 2024-02-09T17:57:11.754 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b54d41f:::benchmark_data_smithi179_148205_object9535:head# 2024-02-09T17:57:11.755 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b557c7a:::benchmark_data_smithi179_148205_object70357:head# 2024-02-09T17:57:11.755 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b57a1ea:::benchmark_data_smithi179_148205_object86693:head# 2024-02-09T17:57:11.757 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b58962d:::benchmark_data_smithi179_148205_object34174:head# 2024-02-09T17:57:11.757 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b5995e6:::benchmark_data_smithi179_148205_object102765:head# 2024-02-09T17:57:11.758 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b59e669:::benchmark_data_smithi179_148205_object10949:head# 2024-02-09T17:57:11.758 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b5a0a36:::benchmark_data_smithi179_148205_object87944:head# 2024-02-09T17:57:11.759 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b5a7a3d:::benchmark_data_smithi179_148205_object40297:head# 2024-02-09T17:57:11.759 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b5a8d1f:::benchmark_data_smithi179_148205_object31014:head# 2024-02-09T17:57:11.760 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b5bc69a:::benchmark_data_smithi179_148205_object105548:head# 2024-02-09T17:57:11.760 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b5be7a0:::benchmark_data_smithi179_148205_object27364:head# 2024-02-09T17:57:11.761 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b5bf930:::benchmark_data_smithi179_148205_object71166:head# 2024-02-09T17:57:11.761 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b5bfc2f:::benchmark_data_smithi179_148205_object69835:head# 2024-02-09T17:57:11.769 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b5c75d0:::benchmark_data_smithi179_148205_object40131:head# 2024-02-09T17:57:11.769 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b5d1389:::benchmark_data_smithi179_148205_object22559:head# 2024-02-09T17:57:11.769 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b5de830:::benchmark_data_smithi179_148205_object1473:head# 2024-02-09T17:57:11.770 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b5ec04c:::benchmark_data_smithi179_148205_object69780:head# 2024-02-09T17:57:11.770 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b5ec16d:::benchmark_data_smithi179_148205_object22847:head# 2024-02-09T17:57:11.770 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b5fee45:::benchmark_data_smithi179_148205_object99666:head# 2024-02-09T17:57:11.770 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b5ffbd5:::benchmark_data_smithi179_148205_object70093:head# 2024-02-09T17:57:11.771 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b60d0ad:::benchmark_data_smithi179_148205_object23469:head# 2024-02-09T17:57:11.771 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b6190e9:::benchmark_data_smithi179_148205_object71180:head# 2024-02-09T17:57:11.771 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b6280d2:::benchmark_data_smithi179_148205_object107576:head# 2024-02-09T17:57:11.772 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b629737:::benchmark_data_smithi179_148205_object60378:head# 2024-02-09T17:57:11.772 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b6303a2:::benchmark_data_smithi179_148205_object11648:head# 2024-02-09T17:57:11.772 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b643e3f:::benchmark_data_smithi179_148205_object89812:head# 2024-02-09T17:57:11.773 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b64fe57:::benchmark_data_smithi179_148205_object47964:head# 2024-02-09T17:57:11.773 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b659e56:::benchmark_data_smithi179_148205_object14863:head# 2024-02-09T17:57:11.773 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b6794a9:::benchmark_data_smithi179_148205_object57952:head# 2024-02-09T17:57:11.774 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b685b3c:::benchmark_data_smithi179_148205_object28672:head# 2024-02-09T17:57:11.775 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b687b5d:::benchmark_data_smithi179_148205_object58289:head# 2024-02-09T17:57:11.775 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b691c87:::benchmark_data_smithi179_148205_object92625:head# 2024-02-09T17:57:11.775 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b69f505:::benchmark_data_smithi179_148205_object22291:head# 2024-02-09T17:57:11.776 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b6b452f:::benchmark_data_smithi179_148205_object87843:head# 2024-02-09T17:57:11.776 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b6cd6a2:::benchmark_data_smithi179_148205_object25023:head# 2024-02-09T17:57:11.781 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b6d26f4:::benchmark_data_smithi179_148205_object61342:head# 2024-02-09T17:57:11.781 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b6e38c0:::benchmark_data_smithi179_148205_object24471:head# 2024-02-09T17:57:11.786 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b6eb5df:::benchmark_data_smithi179_148205_object53249:head# 2024-02-09T17:57:11.786 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b6ff71f:::benchmark_data_smithi179_148205_object24360:head# 2024-02-09T17:57:11.786 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b706fc5:::benchmark_data_smithi179_148205_object28088:head# 2024-02-09T17:57:11.787 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b710147:::benchmark_data_smithi179_148205_object46782:head# 2024-02-09T17:57:11.787 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b716961:::benchmark_data_smithi179_148205_object30982:head# 2024-02-09T17:57:11.788 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b71c58a:::benchmark_data_smithi179_148205_object65165:head# 2024-02-09T17:57:11.788 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b72421a:::benchmark_data_smithi179_148205_object30978:head# 2024-02-09T17:57:11.788 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b7327fa:::benchmark_data_smithi179_148205_object81191:head# 2024-02-09T17:57:11.789 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b732935:::benchmark_data_smithi179_148205_object47754:head# 2024-02-09T17:57:11.789 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b734d25:::benchmark_data_smithi179_148205_object101794:head# 2024-02-09T17:57:11.790 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b747b86:::benchmark_data_smithi179_148205_object45346:head# 2024-02-09T17:57:11.790 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b74e05e:::benchmark_data_smithi179_148205_object75353:head# 2024-02-09T17:57:11.791 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b76405f:::benchmark_data_smithi179_148205_object74649:head# 2024-02-09T17:57:11.795 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b764329:::benchmark_data_smithi179_148205_object20413:head# 2024-02-09T17:57:11.795 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b767ad9:::benchmark_data_smithi179_148205_object44498:head# 2024-02-09T17:57:11.795 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b76d21a:::benchmark_data_smithi179_148205_object29686:head# 2024-02-09T17:57:11.795 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b770529:::benchmark_data_smithi179_148205_object75413:head# 2024-02-09T17:57:11.796 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b77be67:::benchmark_data_smithi179_148205_object76275:head# 2024-02-09T17:57:11.796 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b780caf:::benchmark_data_smithi179_148205_object65408:head# 2024-02-09T17:57:11.796 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b7907e3:::benchmark_data_smithi179_148205_object38723:head# 2024-02-09T17:57:11.796 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b795551:::benchmark_data_smithi179_148205_object84987:head# 2024-02-09T17:57:11.796 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b79be9e:::benchmark_data_smithi179_148205_object28453:head# 2024-02-09T17:57:11.797 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b7a6c8f:::benchmark_data_smithi179_148205_object27289:head# 2024-02-09T17:57:11.797 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b7bda30:::benchmark_data_smithi179_148205_object40577:head# 2024-02-09T17:57:11.797 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b7de689:::benchmark_data_smithi179_148205_object22502:head# 2024-02-09T17:57:11.797 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b7f0922:::benchmark_data_smithi179_148205_object37947:head# 2024-02-09T17:57:11.797 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b7f123e:::benchmark_data_smithi179_148205_object36177:head# 2024-02-09T17:57:11.798 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b7f1dc2:::benchmark_data_smithi179_148205_object17239:head# 2024-02-09T17:57:11.798 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b802f8e:::benchmark_data_smithi179_148205_object99213:head# 2024-02-09T17:57:11.799 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b80d2aa:::benchmark_data_smithi179_148205_object106426:head# 2024-02-09T17:57:11.799 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b814af5:::benchmark_data_smithi179_148205_object19001:head# 2024-02-09T17:57:11.799 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b81c49d:::benchmark_data_smithi179_148205_object79477:head# 2024-02-09T17:57:11.800 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b82a913:::benchmark_data_smithi179_148205_object93147:head# 2024-02-09T17:57:11.800 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b8336fc:::benchmark_data_smithi179_148205_object17616:head# 2024-02-09T17:57:11.800 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b839db8:::benchmark_data_smithi179_148205_object99651:head# 2024-02-09T17:57:11.801 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b8488b4:::benchmark_data_smithi179_148205_object94126:head# 2024-02-09T17:57:11.802 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b84d860:::benchmark_data_smithi179_148205_object31881:head# 2024-02-09T17:57:11.802 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b862c78:::benchmark_data_smithi179_148205_object9094:head# 2024-02-09T17:57:11.802 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b87c5db:::benchmark_data_smithi179_148205_object92204:head# 2024-02-09T17:57:11.803 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b887729:::benchmark_data_smithi179_148205_object58490:head# 2024-02-09T17:57:11.803 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b88925d:::benchmark_data_smithi179_148205_object79712:head# 2024-02-09T17:57:11.804 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b88e97b:::benchmark_data_smithi179_148205_object6101:head# 2024-02-09T17:57:11.804 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b894fd6:::benchmark_data_smithi179_148205_object32351:head# 2024-02-09T17:57:11.804 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b89f2d6:::benchmark_data_smithi179_148205_object10658:head# 2024-02-09T17:57:11.807 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b8a4fd2:::benchmark_data_smithi179_148205_object86352:head# 2024-02-09T17:57:11.808 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b8a91d7:::benchmark_data_smithi179_148205_object69477:head# 2024-02-09T17:57:11.808 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b8c67a9:::benchmark_data_smithi179_148205_object18588:head# 2024-02-09T17:57:11.808 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b8c7034:::benchmark_data_smithi179_148205_object85303:head# 2024-02-09T17:57:11.809 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b8e5f67:::benchmark_data_smithi179_148205_object13296:head# 2024-02-09T17:57:11.809 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b8e78f3:::benchmark_data_smithi179_148205_object89876:head# 2024-02-09T17:57:11.809 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b8ea129:::benchmark_data_smithi179_148205_object95137:head# 2024-02-09T17:57:11.810 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b8ef2d8:::benchmark_data_smithi179_148205_object6158:head# 2024-02-09T17:57:11.810 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b904751:::benchmark_data_smithi179_148205_object81282:head# 2024-02-09T17:57:11.811 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b91dda0:::benchmark_data_smithi179_148205_object101189:head# 2024-02-09T17:57:11.811 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b9207c9:::benchmark_data_smithi179_148205_object90497:head# 2024-02-09T17:57:11.812 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b92a6d7:::benchmark_data_smithi179_148205_object13041:head# 2024-02-09T17:57:11.813 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b949e9d:::benchmark_data_smithi179_148205_object9292:head# 2024-02-09T17:57:11.814 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b955e13:::benchmark_data_smithi179_148205_object98611:head# 2024-02-09T17:57:11.814 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b961ac7:::benchmark_data_smithi179_148205_object79062:head# 2024-02-09T17:57:11.815 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b96d90f:::benchmark_data_smithi179_148205_object94684:head# 2024-02-09T17:57:11.815 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b97d02f:::benchmark_data_smithi179_148205_object79924:head# 2024-02-09T17:57:11.819 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b97dd58:::benchmark_data_smithi179_148205_object5521:head# 2024-02-09T17:57:11.819 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b9863d8:::benchmark_data_smithi179_148205_object97785:head# 2024-02-09T17:57:11.819 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b98ae97:::benchmark_data_smithi179_148205_object103679:head# 2024-02-09T17:57:11.819 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b98fc70:::benchmark_data_smithi179_148205_object25810:head# 2024-02-09T17:57:11.819 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b9913f2:::benchmark_data_smithi179_148205_object4850:head# 2024-02-09T17:57:11.819 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b9ab1f6:::benchmark_data_smithi179_148205_object56197:head# 2024-02-09T17:57:11.819 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b9b47ad:::benchmark_data_smithi179_148205_object99322:head# 2024-02-09T17:57:11.819 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b9b506f:::benchmark_data_smithi179_148205_object25085:head# 2024-02-09T17:57:11.820 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b9bc2ed:::benchmark_data_smithi179_148205_object94049:head# 2024-02-09T17:57:11.820 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b9cf175:::benchmark_data_smithi179_148205_object5254:head# 2024-02-09T17:57:11.820 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b9d8f10:::benchmark_data_smithi179_148205_object45674:head# 2024-02-09T17:57:11.821 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b9e06a7:::benchmark_data_smithi179_148205_object89510:head# 2024-02-09T17:57:11.825 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b9efc90:::benchmark_data_smithi179_148205_object2087:head# 2024-02-09T17:57:11.826 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b9f07d2:::benchmark_data_smithi179_148205_object61707:head# 2024-02-09T17:57:11.831 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1b9f12bc:::benchmark_data_smithi179_148205_object60657:head# 2024-02-09T17:57:11.831 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ba04c75:::benchmark_data_smithi179_148205_object21755:head# 2024-02-09T17:57:11.831 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ba12dea:::benchmark_data_smithi179_148205_object5644:head# 2024-02-09T17:57:11.831 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ba16b04:::benchmark_data_smithi179_148205_object52065:head# 2024-02-09T17:57:11.831 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ba20836:::benchmark_data_smithi179_148205_object73146:head# 2024-02-09T17:57:11.831 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ba3077f:::benchmark_data_smithi179_148205_object72797:head# 2024-02-09T17:57:11.831 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ba3cfd0:::benchmark_data_smithi179_148205_object80279:head# 2024-02-09T17:57:11.831 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ba3da26:::benchmark_data_smithi179_148205_object61998:head# 2024-02-09T17:57:11.831 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ba443f5:::benchmark_data_smithi179_148205_object30210:head# 2024-02-09T17:57:11.834 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ba50564:::benchmark_data_smithi179_148205_object42478:head# 2024-02-09T17:57:11.834 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ba64be9:::benchmark_data_smithi179_148205_object74219:head# 2024-02-09T17:57:11.839 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ba65709:::benchmark_data_smithi179_148205_object57943:head# 2024-02-09T17:57:11.839 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ba739e5:::benchmark_data_smithi179_148205_object40380:head# 2024-02-09T17:57:11.839 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ba73bbd:::benchmark_data_smithi179_148205_object86655:head# 2024-02-09T17:57:11.839 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ba816c3:::benchmark_data_smithi179_148205_object91980:head# 2024-02-09T17:57:11.839 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ba82fe1:::benchmark_data_smithi179_148205_object82836:head# 2024-02-09T17:57:11.839 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ba84f40:::benchmark_data_smithi179_148205_object47583:head# 2024-02-09T17:57:11.840 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ba8543f:::benchmark_data_smithi179_148205_object6311:head# 2024-02-09T17:57:11.840 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ba955df:::benchmark_data_smithi179_148205_object28283:head# 2024-02-09T17:57:11.840 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ba984c6:::benchmark_data_smithi179_148205_object3707:head# 2024-02-09T17:57:11.841 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ba9e507:::benchmark_data_smithi179_148205_object86840:head# 2024-02-09T17:57:11.841 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1baa09b1:::benchmark_data_smithi179_148205_object60541:head# 2024-02-09T17:57:11.842 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1baa7cf5:::benchmark_data_smithi179_148205_object48197:head# 2024-02-09T17:57:11.842 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bab9ec4:::benchmark_data_smithi179_148205_object30249:head# 2024-02-09T17:57:11.842 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1babc862:::benchmark_data_smithi179_148205_object107524:head# 2024-02-09T17:57:11.843 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bac2c0b:::benchmark_data_smithi179_148205_object19132:head# 2024-02-09T17:57:11.843 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bac6c2b:::benchmark_data_smithi179_148205_object48267:head# 2024-02-09T17:57:11.843 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bad4b3a:::benchmark_data_smithi179_148205_object8500:head# 2024-02-09T17:57:11.844 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bada82b:::benchmark_data_smithi179_148205_object86713:head# 2024-02-09T17:57:11.844 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bae09cd:::benchmark_data_smithi179_148205_object48480:head# 2024-02-09T17:57:11.845 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bae1ace:::benchmark_data_smithi179_148205_object77642:head# 2024-02-09T17:57:11.845 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bae22e2:::benchmark_data_smithi179_148205_object96630:head# 2024-02-09T17:57:11.845 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bae2421:::benchmark_data_smithi179_148205_object27535:head# 2024-02-09T17:57:11.850 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bb0f816:::benchmark_data_smithi179_148205_object92355:head# 2024-02-09T17:57:11.850 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bb18898:::benchmark_data_smithi179_148205_object97641:head# 2024-02-09T17:57:11.850 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bb18cb7:::benchmark_data_smithi179_148205_object1317:head# 2024-02-09T17:57:11.850 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bb231b9:::benchmark_data_smithi179_148205_object44556:head# 2024-02-09T17:57:11.850 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bb26871:::benchmark_data_smithi179_148205_object72419:head# 2024-02-09T17:57:11.850 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bb3bf2d:::benchmark_data_smithi179_148205_object84818:head# 2024-02-09T17:57:11.850 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bb3f43d:::benchmark_data_smithi179_148205_object30357:head# 2024-02-09T17:57:11.851 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bb46ffe:::benchmark_data_smithi179_148205_object39910:head# 2024-02-09T17:57:11.852 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bb5c56d:::benchmark_data_smithi179_148205_object66209:head# 2024-02-09T17:57:11.852 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bb6f3ad:::benchmark_data_smithi179_148205_object94803:head# 2024-02-09T17:57:11.852 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bb742fb:::benchmark_data_smithi179_148205_object51789:head# 2024-02-09T17:57:11.853 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bb75fae:::benchmark_data_smithi179_148205_object68331:head# 2024-02-09T17:57:11.853 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bb84038:::benchmark_data_smithi179_148205_object57210:head# 2024-02-09T17:57:11.854 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bb84233:::benchmark_data_smithi179_148205_object99840:head# 2024-02-09T17:57:11.854 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bb90ddc:::benchmark_data_smithi179_148205_object60626:head# 2024-02-09T17:57:11.854 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bba230f:::benchmark_data_smithi179_148205_object79283:head# 2024-02-09T17:57:11.855 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bba3f38:::benchmark_data_smithi179_148205_object88386:head# 2024-02-09T17:57:11.855 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bbd5dad:::benchmark_data_smithi179_148205_object75020:head# 2024-02-09T17:57:11.856 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bbd7ffd:::benchmark_data_smithi179_148205_object57292:head# 2024-02-09T17:57:11.856 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bbe7f8f:::benchmark_data_smithi179_148205_object13583:head# 2024-02-09T17:57:11.856 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bbe9651:::benchmark_data_smithi179_148205_object48906:head# 2024-02-09T17:57:11.857 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bbfbe4c:::benchmark_data_smithi179_148205_object25857:head# 2024-02-09T17:57:11.858 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bc02813:::benchmark_data_smithi179_148205_object6670:head# 2024-02-09T17:57:11.864 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bc0387c:::benchmark_data_smithi179_148205_object64961:head# 2024-02-09T17:57:11.864 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bc11ca2:::benchmark_data_smithi179_148205_object12459:head# 2024-02-09T17:57:11.864 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bc13249:::benchmark_data_smithi179_148205_object98113:head# 2024-02-09T17:57:11.864 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bc14f23:::benchmark_data_smithi179_148205_object99335:head# 2024-02-09T17:57:11.864 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bc1e9f2:::benchmark_data_smithi179_148205_object81317:head# 2024-02-09T17:57:11.864 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bc26cc9:::benchmark_data_smithi179_148205_object42721:head# 2024-02-09T17:57:11.864 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bc55ebb:::benchmark_data_smithi179_148205_object91046:head# 2024-02-09T17:57:11.864 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bc5a96a:::benchmark_data_smithi179_148205_object3905:head# 2024-02-09T17:57:11.864 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bc5ba18:::benchmark_data_smithi179_148205_object11242:head# 2024-02-09T17:57:11.864 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bc7626f:::benchmark_data_smithi179_148205_object35357:head# 2024-02-09T17:57:11.864 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bc7cde7:::benchmark_data_smithi179_148205_object38203:head# 2024-02-09T17:57:11.865 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bc83123:::benchmark_data_smithi179_148205_object64087:head# 2024-02-09T17:57:11.865 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bc856cb:::benchmark_data_smithi179_148205_object14687:head# 2024-02-09T17:57:11.865 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bc9328a:::benchmark_data_smithi179_148205_object18778:head# 2024-02-09T17:57:11.866 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bca4d04:::benchmark_data_smithi179_148205_object33383:head# 2024-02-09T17:57:11.866 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bcab590:::benchmark_data_smithi179_148205_object81081:head# 2024-02-09T17:57:11.867 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bcaf594:::benchmark_data_smithi179_148205_object33103:head# 2024-02-09T17:57:11.867 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bcb87e7:::benchmark_data_smithi179_148205_object53785:head# 2024-02-09T17:57:11.867 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bcb9fb2:::benchmark_data_smithi179_148205_object106715:head# 2024-02-09T17:57:11.869 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bcbc0cd:::benchmark_data_smithi179_148205_object42948:head# 2024-02-09T17:57:11.870 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bcc0e07:::benchmark_data_smithi179_148205_object16215:head# 2024-02-09T17:57:11.870 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bccd19a:::benchmark_data_smithi179_148205_object21409:head# 2024-02-09T17:57:11.870 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bccf2fe:::benchmark_data_smithi179_148205_object19834:head# 2024-02-09T17:57:11.871 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bce0136:::benchmark_data_smithi179_148205_object35144:head# 2024-02-09T17:57:11.871 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bce083d:::benchmark_data_smithi179_148205_object12225:head# 2024-02-09T17:57:11.872 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bce3c3f:::benchmark_data_smithi179_148205_object107124:head# 2024-02-09T17:57:11.874 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bce6b0c:::benchmark_data_smithi179_148205_object17436:head# 2024-02-09T17:57:11.874 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bd04123:::benchmark_data_smithi179_148205_object37648:head# 2024-02-09T17:57:11.874 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bd0dd8c:::benchmark_data_smithi179_148205_object12760:head# 2024-02-09T17:57:11.875 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bd1b56e:::benchmark_data_smithi179_148205_object109106:head# 2024-02-09T17:57:11.875 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bd2550d:::benchmark_data_smithi179_148205_object73466:head# 2024-02-09T17:57:11.875 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bd2b98b:::benchmark_data_smithi179_148205_object54867:head# 2024-02-09T17:57:11.876 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bd303be:::benchmark_data_smithi179_148205_object6197:head# 2024-02-09T17:57:11.876 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bd30646:::benchmark_data_smithi179_148205_object72113:head# 2024-02-09T17:57:11.876 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bd3d716:::benchmark_data_smithi179_148205_object96103:head# 2024-02-09T17:57:11.877 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bd470f1:::benchmark_data_smithi179_148205_object35384:head# 2024-02-09T17:57:11.877 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bd4c583:::benchmark_data_smithi179_148205_object67494:head# 2024-02-09T17:57:11.878 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bd4f7ec:::benchmark_data_smithi179_148205_object103239:head# 2024-02-09T17:57:11.878 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bd51d49:::benchmark_data_smithi179_148205_object89840:head# 2024-02-09T17:57:11.879 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bd5cf90:::benchmark_data_smithi179_148205_object98788:head# 2024-02-09T17:57:11.879 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bd6b693:::benchmark_data_smithi179_148205_object28075:head# 2024-02-09T17:57:11.880 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bd7a770:::benchmark_data_smithi179_148205_object8191:head# 2024-02-09T17:57:11.880 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bd84d27:::benchmark_data_smithi179_148205_object104096:head# 2024-02-09T17:57:11.880 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bd89d5e:::benchmark_data_smithi179_148205_object14380:head# 2024-02-09T17:57:11.881 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bd8ce5e:::benchmark_data_smithi179_148205_object75182:head# 2024-02-09T17:57:11.881 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bd955e5:::benchmark_data_smithi179_148205_object44031:head# 2024-02-09T17:57:11.882 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bda4013:::benchmark_data_smithi179_148205_object18307:head# 2024-02-09T17:57:11.882 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bdbd51a:::benchmark_data_smithi179_148205_object96241:head# 2024-02-09T17:57:11.882 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bdc1d97:::benchmark_data_smithi179_148205_object37981:head# 2024-02-09T17:57:11.883 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bdc583b:::benchmark_data_smithi179_148205_object71431:head# 2024-02-09T17:57:11.883 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bdd7a97:::benchmark_data_smithi179_148205_object65292:head# 2024-02-09T17:57:11.884 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bdf6c7f:::benchmark_data_smithi179_148205_object60220:head# 2024-02-09T17:57:11.884 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1be0a298:::benchmark_data_smithi179_148205_object40988:head# 2024-02-09T17:57:11.885 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1be0eff6:::benchmark_data_smithi179_148205_object35284:head# 2024-02-09T17:57:11.885 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1be15b84:::benchmark_data_smithi179_148205_object106882:head# 2024-02-09T17:57:11.885 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1be2b80f:::benchmark_data_smithi179_148205_object95052:head# 2024-02-09T17:57:11.886 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1be2f74f:::benchmark_data_smithi179_148205_object40135:head# 2024-02-09T17:57:11.886 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1be37395:::benchmark_data_smithi179_148205_object54660:head# 2024-02-09T17:57:11.886 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1be3eab2:::benchmark_data_smithi179_148205_object100756:head# 2024-02-09T17:57:11.887 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1be402a1:::benchmark_data_smithi179_148205_object108443:head# 2024-02-09T17:57:11.887 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1be5fd80:::benchmark_data_smithi179_148205_object29378:head# 2024-02-09T17:57:11.888 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1be63f44:::benchmark_data_smithi179_148205_object68285:head# 2024-02-09T17:57:11.888 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1be67821:::benchmark_data_smithi179_148205_object42034:head# 2024-02-09T17:57:11.889 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1be7b29d:::benchmark_data_smithi179_148205_object69494:head# 2024-02-09T17:57:11.889 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1be80dd0:::benchmark_data_smithi179_148205_object21583:head# 2024-02-09T17:57:11.889 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1be83219:::benchmark_data_smithi179_148205_object84491:head# 2024-02-09T17:57:11.889 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1be84891:::benchmark_data_smithi179_148205_object53938:head# 2024-02-09T17:57:11.890 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1be8fec6:::benchmark_data_smithi179_148205_object95536:head# 2024-02-09T17:57:11.891 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1be92728:::benchmark_data_smithi179_148205_object67534:head# 2024-02-09T17:57:11.891 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bea6c3c:::benchmark_data_smithi179_148205_object43499:head# 2024-02-09T17:57:11.892 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bea965e:::benchmark_data_smithi179_148205_object83421:head# 2024-02-09T17:57:11.892 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bec500a:::benchmark_data_smithi179_148205_object98798:head# 2024-02-09T17:57:11.893 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bec966a:::benchmark_data_smithi179_148205_object77724:head# 2024-02-09T17:57:11.893 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bed1362:::benchmark_data_smithi179_148205_object34033:head# 2024-02-09T17:57:11.894 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bee60b2:::benchmark_data_smithi179_148205_object24574:head# 2024-02-09T17:57:11.894 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1beed777:::benchmark_data_smithi179_148205_object13898:head# 2024-02-09T17:57:11.894 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bef1c49:::benchmark_data_smithi179_148205_object66723:head# 2024-02-09T17:57:11.895 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bef2fee:::benchmark_data_smithi179_148205_object100473:head# 2024-02-09T17:57:11.895 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1befa7c5:::benchmark_data_smithi179_148205_object3959:head# 2024-02-09T17:57:11.896 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bf0ded5:::benchmark_data_smithi179_148205_object98268:head# 2024-02-09T17:57:11.896 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bf22501:::benchmark_data_smithi179_148205_object108020:head# 2024-02-09T17:57:11.896 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bf29824:::benchmark_data_smithi179_148205_object26129:head# 2024-02-09T17:57:11.897 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bf2a166:::benchmark_data_smithi179_148205_object86064:head# 2024-02-09T17:57:11.897 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bf38a28:::benchmark_data_smithi179_148205_object69251:head# 2024-02-09T17:57:11.898 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bf3ff77:::benchmark_data_smithi179_148205_object65912:head# 2024-02-09T17:57:11.898 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bf42e83:::benchmark_data_smithi179_148205_object106674:head# 2024-02-09T17:57:11.899 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bf461c0:::benchmark_data_smithi179_148205_object41902:head# 2024-02-09T17:57:11.899 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bf6174f:::benchmark_data_smithi179_148205_object68461:head# 2024-02-09T17:57:11.899 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bf96800:::benchmark_data_smithi179_148205_object87131:head# 2024-02-09T17:57:11.900 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bfa06d7:::benchmark_data_smithi179_148205_object102570:head# 2024-02-09T17:57:11.901 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bfa4f4c:::benchmark_data_smithi179_148205_object24108:head# 2024-02-09T17:57:11.901 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bfaef07:::benchmark_data_smithi179_148205_object4214:head# 2024-02-09T17:57:11.902 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bfb0419:::benchmark_data_smithi179_148205_object53912:head# 2024-02-09T17:57:11.902 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bfbd9fa:::benchmark_data_smithi179_148205_object45027:head# 2024-02-09T17:57:11.902 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bfc6381:::benchmark_data_smithi179_148205_object98883:head# 2024-02-09T17:57:11.903 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bfd982f:::benchmark_data_smithi179_148205_object53086:head# 2024-02-09T17:57:11.903 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bfde6c5:::benchmark_data_smithi179_148205_object49230:head# 2024-02-09T17:57:11.904 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bfe4d81:::benchmark_data_smithi179_148205_object1043:head# 2024-02-09T17:57:11.904 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1bffe540:::benchmark_data_smithi179_148205_object81802:head# 2024-02-09T17:57:11.905 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c00190e:::benchmark_data_smithi179_148205_object61711:head# 2024-02-09T17:57:11.905 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c014070:::benchmark_data_smithi179_148205_object73057:head# 2024-02-09T17:57:11.905 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c01a9dc:::benchmark_data_smithi179_148205_object62314:head# 2024-02-09T17:57:11.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:11 smithi179 ceph-mon[43975]: osdmap e824: 12 total, 11 up, 12 in 2024-02-09T17:57:11.926 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c022a32:::benchmark_data_smithi179_148205_object39768:head# 2024-02-09T17:57:11.926 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c02abd0:::benchmark_data_smithi179_148205_object89505:head# 2024-02-09T17:57:11.926 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c047dd5:::benchmark_data_smithi179_148205_object59680:head# 2024-02-09T17:57:11.926 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c05fd57:::benchmark_data_smithi179_148205_object104240:head# 2024-02-09T17:57:11.926 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c06635e:::benchmark_data_smithi179_148205_object86690:head# 2024-02-09T17:57:11.927 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c0695e3:::benchmark_data_smithi179_148205_object10111:head# 2024-02-09T17:57:11.927 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c06d87f:::benchmark_data_smithi179_148205_object31590:head# 2024-02-09T17:57:11.927 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c06fa4a:::benchmark_data_smithi179_148205_object108659:head# 2024-02-09T17:57:11.927 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c071ee7:::benchmark_data_smithi179_148205_object25243:head# 2024-02-09T17:57:11.927 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c07714c:::benchmark_data_smithi179_148205_object30489:head# 2024-02-09T17:57:11.927 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c086024:::benchmark_data_smithi179_148205_object100334:head# 2024-02-09T17:57:11.927 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c089995:::benchmark_data_smithi179_148205_object83550:head# 2024-02-09T17:57:11.927 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c0916bb:::benchmark_data_smithi179_148205_object90328:head# 2024-02-09T17:57:11.927 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c09bef4:::benchmark_data_smithi179_148205_object9884:head# 2024-02-09T17:57:11.927 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c0be9f4:::benchmark_data_smithi179_148205_object7410:head# 2024-02-09T17:57:11.927 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c0cdb40:::benchmark_data_smithi179_148205_object16007:head# 2024-02-09T17:57:11.928 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c0d8d84:::benchmark_data_smithi179_148205_object109482:head# 2024-02-09T17:57:11.929 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c0e16a6:::benchmark_data_smithi179_148205_object98336:head# 2024-02-09T17:57:11.929 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c0e43fb:::benchmark_data_smithi179_148205_object95658:head# 2024-02-09T17:57:11.929 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c0edfb6:::benchmark_data_smithi179_148205_object44970:head# 2024-02-09T17:57:11.929 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c0ee4d6:::benchmark_data_smithi179_148205_object24657:head# 2024-02-09T17:57:11.929 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c10553f:::benchmark_data_smithi179_148205_object95930:head# 2024-02-09T17:57:11.929 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c12363a:::benchmark_data_smithi179_148205_object27373:head# 2024-02-09T17:57:11.930 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c12c838:::benchmark_data_smithi179_148205_object27013:head# 2024-02-09T17:57:11.930 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c1375df:::benchmark_data_smithi179_148205_object80662:head# 2024-02-09T17:57:11.930 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c138b79:::benchmark_data_smithi179_148205_object92325:head# 2024-02-09T17:57:11.930 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c14fc1f:::benchmark_data_smithi179_148205_object61640:head# 2024-02-09T17:57:11.931 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c162499:::benchmark_data_smithi179_148205_object36109:head# 2024-02-09T17:57:11.931 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c16fc34:::benchmark_data_smithi179_148205_object40972:head# 2024-02-09T17:57:11.931 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c180ecf:::benchmark_data_smithi179_148205_object22570:head# 2024-02-09T17:57:11.932 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c1860f3:::benchmark_data_smithi179_148205_object106527:head# 2024-02-09T17:57:11.934 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c1912d8:::benchmark_data_smithi179_148205_object81260:head# 2024-02-09T17:57:11.934 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c19258f:::benchmark_data_smithi179_148205_object63555:head# 2024-02-09T17:57:11.934 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c1a49d4:::benchmark_data_smithi179_148205_object27125:head# 2024-02-09T17:57:11.934 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c1a548f:::benchmark_data_smithi179_148205_object100704:head# 2024-02-09T17:57:11.934 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c1b37c9:::benchmark_data_smithi179_148205_object45403:head# 2024-02-09T17:57:11.935 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c1bb4cf:::benchmark_data_smithi179_148205_object101878:head# 2024-02-09T17:57:11.935 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c1c0b8e:::benchmark_data_smithi179_148205_object95608:head# 2024-02-09T17:57:11.935 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c1e7f66:::benchmark_data_smithi179_148205_object36019:head# 2024-02-09T17:57:11.937 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c2024f3:::benchmark_data_smithi179_148205_object56699:head# 2024-02-09T17:57:11.937 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c2160f3:::benchmark_data_smithi179_148205_object16712:head# 2024-02-09T17:57:11.937 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c216146:::benchmark_data_smithi179_148205_object100801:head# 2024-02-09T17:57:11.938 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c228eae:::benchmark_data_smithi179_148205_object64800:head# 2024-02-09T17:57:11.938 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c231ba3:::benchmark_data_smithi179_148205_object106385:head# 2024-02-09T17:57:11.938 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c23505d:::benchmark_data_smithi179_148205_object50415:head# 2024-02-09T17:57:11.939 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c2354c6:::benchmark_data_smithi179_148205_object108420:head# 2024-02-09T17:57:11.939 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c2404f5:::benchmark_data_smithi179_148205_object31397:head# 2024-02-09T17:57:11.940 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c24e100:::benchmark_data_smithi179_148205_object7681:head# 2024-02-09T17:57:11.940 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c255362:::benchmark_data_smithi179_148205_object63844:head# 2024-02-09T17:57:11.941 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c256e5f:::benchmark_data_smithi179_148205_object60348:head# 2024-02-09T17:57:11.943 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c25b072:::benchmark_data_smithi179_148205_object108637:head# 2024-02-09T17:57:11.943 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c25b128:::benchmark_data_smithi179_148205_object11287:head# 2024-02-09T17:57:11.943 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c26b9db:::benchmark_data_smithi179_148205_object47464:head# 2024-02-09T17:57:11.943 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c26d724:::benchmark_data_smithi179_148205_object66946:head# 2024-02-09T17:57:11.943 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c271130:::benchmark_data_smithi179_148205_object81862:head# 2024-02-09T17:57:11.944 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c288888:::benchmark_data_smithi179_148205_object75171:head# 2024-02-09T17:57:11.944 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c28f9c4:::benchmark_data_smithi179_148205_object94886:head# 2024-02-09T17:57:11.944 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c29097f:::benchmark_data_smithi179_148205_object21965:head# 2024-02-09T17:57:11.944 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c2979aa:::benchmark_data_smithi179_148205_object45781:head# 2024-02-09T17:57:11.944 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c2bb0a8:::benchmark_data_smithi179_148205_object21971:head# 2024-02-09T17:57:11.945 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c2c11f2:::benchmark_data_smithi179_148205_object29962:head# 2024-02-09T17:57:11.945 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c2c8d15:::benchmark_data_smithi179_148205_object61087:head# 2024-02-09T17:57:11.945 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c2d6cd3:::benchmark_data_smithi179_148205_object93274:head# 2024-02-09T17:57:11.946 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c2dc84f:::benchmark_data_smithi179_148205_object62715:head# 2024-02-09T17:57:11.946 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c2dd4d8:::benchmark_data_smithi179_148205_object44844:head# 2024-02-09T17:57:11.950 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c2ebd84:::benchmark_data_smithi179_148205_object74245:head# 2024-02-09T17:57:11.950 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c2fb1a0:::benchmark_data_smithi179_148205_object71279:head# 2024-02-09T17:57:11.950 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c30161d:::benchmark_data_smithi179_148205_object70097:head# 2024-02-09T17:57:11.950 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c3035f2:::benchmark_data_smithi179_148205_object2973:head# 2024-02-09T17:57:11.950 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c30e131:::benchmark_data_smithi179_148205_object103930:head# 2024-02-09T17:57:11.950 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c30f858:::benchmark_data_smithi179_148205_object52453:head# 2024-02-09T17:57:11.951 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c314b4c:::benchmark_data_smithi179_148205_object50463:head# 2024-02-09T17:57:11.951 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c32c633:::benchmark_data_smithi179_148205_object23539:head# 2024-02-09T17:57:11.952 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c32dd9f:::benchmark_data_smithi179_148205_object54617:head# 2024-02-09T17:57:11.952 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c32ea50:::benchmark_data_smithi179_148205_object101382:head# 2024-02-09T17:57:11.953 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c32fd1f:::benchmark_data_smithi179_148205_object94339:head# 2024-02-09T17:57:11.953 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c33b06c:::benchmark_data_smithi179_148205_object96236:head# 2024-02-09T17:57:11.957 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c348e83:::benchmark_data_smithi179_148205_object64841:head# 2024-02-09T17:57:11.957 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c35783c:::benchmark_data_smithi179_148205_object56933:head# 2024-02-09T17:57:11.957 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c35a3f4:::benchmark_data_smithi179_148205_object637:head# 2024-02-09T17:57:11.957 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c35d628:::benchmark_data_smithi179_148205_object55635:head# 2024-02-09T17:57:11.957 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c360187:::benchmark_data_smithi179_148205_object103777:head# 2024-02-09T17:57:11.957 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c37aacf:::benchmark_data_smithi179_148205_object23979:head# 2024-02-09T17:57:11.957 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c384509:::benchmark_data_smithi179_148205_object40386:head# 2024-02-09T17:57:11.957 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c389fc5:::benchmark_data_smithi179_148205_object69465:head# 2024-02-09T17:57:11.957 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c399e3c:::benchmark_data_smithi179_148205_object109062:head# 2024-02-09T17:57:11.958 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c39f17d:::benchmark_data_smithi179_148205_object8342:head# 2024-02-09T17:57:11.960 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c3a3d1a:::benchmark_data_smithi179_148205_object72423:head# 2024-02-09T17:57:11.960 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c3a52e6:::benchmark_data_smithi179_148205_object63802:head# 2024-02-09T17:57:11.963 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c3a6771:::benchmark_data_smithi179_148205_object87850:head# 2024-02-09T17:57:11.964 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c3a7d33:::benchmark_data_smithi179_148205_object91880:head# 2024-02-09T17:57:11.965 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c3b0da3:::benchmark_data_smithi179_148205_object42633:head# 2024-02-09T17:57:11.967 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c3b8001:::benchmark_data_smithi179_148205_object23569:head# 2024-02-09T17:57:11.968 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c3cbcfe:::benchmark_data_smithi179_148205_object5703:head# 2024-02-09T17:57:11.968 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c3d4971:::benchmark_data_smithi179_148205_object101613:head# 2024-02-09T17:57:11.968 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c3ef799:::benchmark_data_smithi179_148205_object36796:head# 2024-02-09T17:57:11.969 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c3f9ee1:::benchmark_data_smithi179_148205_object33455:head# 2024-02-09T17:57:11.969 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c4019c8:::benchmark_data_smithi179_148205_object91524:head# 2024-02-09T17:57:11.970 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c405952:::benchmark_data_smithi179_148205_object46100:head# 2024-02-09T17:57:11.970 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c409074:::benchmark_data_smithi179_148205_object85456:head# 2024-02-09T17:57:11.971 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c416a94:::benchmark_data_smithi179_148205_object30445:head# 2024-02-09T17:57:11.971 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c41a821:::benchmark_data_smithi179_148205_object36391:head# 2024-02-09T17:57:11.972 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c41baec:::benchmark_data_smithi179_148205_object95765:head# 2024-02-09T17:57:11.972 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c4219ed:::benchmark_data_smithi179_148205_object71714:head# 2024-02-09T17:57:11.972 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c422b3c:::benchmark_data_smithi179_148205_object93511:head# 2024-02-09T17:57:11.973 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c42cb42:::benchmark_data_smithi179_148205_object67702:head# 2024-02-09T17:57:11.973 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c432985:::benchmark_data_smithi179_148205_object95066:head# 2024-02-09T17:57:11.974 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c43d63b:::benchmark_data_smithi179_148205_object100359:head# 2024-02-09T17:57:11.974 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c43f667:::benchmark_data_smithi179_148205_object15215:head# 2024-02-09T17:57:11.979 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c44e061:::benchmark_data_smithi179_148205_object50416:head# 2024-02-09T17:57:11.979 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c44ea44:::benchmark_data_smithi179_148205_object106461:head# 2024-02-09T17:57:11.979 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c459a2a:::benchmark_data_smithi179_148205_object75739:head# 2024-02-09T17:57:11.979 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c463a32:::benchmark_data_smithi179_148205_object59379:head# 2024-02-09T17:57:11.980 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c46b736:::benchmark_data_smithi179_148205_object27983:head# 2024-02-09T17:57:11.980 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c46e2d3:::benchmark_data_smithi179_148205_object58342:head# 2024-02-09T17:57:11.980 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c475697:::benchmark_data_smithi179_148205_object27677:head# 2024-02-09T17:57:11.980 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c4884a0:::benchmark_data_smithi179_148205_object12806:head# 2024-02-09T17:57:11.980 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c48c432:::benchmark_data_smithi179_148205_object21279:head# 2024-02-09T17:57:11.980 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c48e85b:::benchmark_data_smithi179_148205_object85098:head# 2024-02-09T17:57:11.980 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c4a223c:::benchmark_data_smithi179_148205_object51649:head# 2024-02-09T17:57:11.980 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c4b4966:::benchmark_data_smithi179_148205_object51273:head# 2024-02-09T17:57:11.980 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c4b7d0a:::benchmark_data_smithi179_148205_object23055:head# 2024-02-09T17:57:11.981 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c4bd2b9:::benchmark_data_smithi179_148205_object79069:head# 2024-02-09T17:57:11.984 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c4bf428:::benchmark_data_smithi179_148205_object55704:head# 2024-02-09T17:57:11.984 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c4c07bb:::benchmark_data_smithi179_148205_object1048:head# 2024-02-09T17:57:11.984 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c4c17ae:::benchmark_data_smithi179_148205_object73510:head# 2024-02-09T17:57:11.984 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c4c5107:::benchmark_data_smithi179_148205_object46752:head# 2024-02-09T17:57:11.984 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c4d33c3:::benchmark_data_smithi179_148205_object11143:head# 2024-02-09T17:57:11.984 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c4d58bb:::benchmark_data_smithi179_148205_object63916:head# 2024-02-09T17:57:11.984 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c4de096:::benchmark_data_smithi179_148205_object25476:head# 2024-02-09T17:57:11.985 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c4e721a:::benchmark_data_smithi179_148205_object6126:head# 2024-02-09T17:57:11.985 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c504293:::benchmark_data_smithi179_148205_object641:head# 2024-02-09T17:57:11.985 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c509b63:::benchmark_data_smithi179_148205_object12853:head# 2024-02-09T17:57:11.985 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c50be70:::benchmark_data_smithi179_148205_object102631:head# 2024-02-09T17:57:11.985 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c50dfb3:::benchmark_data_smithi179_148205_object49455:head# 2024-02-09T17:57:11.986 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c51b4eb:::benchmark_data_smithi179_148205_object57204:head# 2024-02-09T17:57:11.986 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c51f8bb:::benchmark_data_smithi179_148205_object104859:head# 2024-02-09T17:57:11.986 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c524378:::benchmark_data_smithi179_148205_object63940:head# 2024-02-09T17:57:11.987 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c528137:::benchmark_data_smithi179_148205_object53189:head# 2024-02-09T17:57:11.987 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c532dad:::benchmark_data_smithi179_148205_object863:head# 2024-02-09T17:57:11.987 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c54a92d:::benchmark_data_smithi179_148205_object30952:head# 2024-02-09T17:57:11.988 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c5504d8:::benchmark_data_smithi179_148205_object63382:head# 2024-02-09T17:57:11.988 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c55b003:::benchmark_data_smithi179_148205_object27968:head# 2024-02-09T17:57:11.990 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c5654f3:::benchmark_data_smithi179_148205_object21661:head# 2024-02-09T17:57:11.991 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c568910:::benchmark_data_smithi179_148205_object58879:head# 2024-02-09T17:57:11.991 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c57090a:::benchmark_data_smithi179_148205_object3874:head# 2024-02-09T17:57:11.991 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c5859e7:::benchmark_data_smithi179_148205_object25765:head# 2024-02-09T17:57:11.991 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c5920da:::benchmark_data_smithi179_148205_object50610:head# 2024-02-09T17:57:11.991 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c592721:::benchmark_data_smithi179_148205_object56362:head# 2024-02-09T17:57:11.992 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c5a5ced:::benchmark_data_smithi179_148205_object103272:head# 2024-02-09T17:57:11.992 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c5a8f98:::benchmark_data_smithi179_148205_object68777:head# 2024-02-09T17:57:11.993 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c5b5423:::benchmark_data_smithi179_148205_object36567:head# 2024-02-09T17:57:11.993 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c5c81a3:::benchmark_data_smithi179_148205_object108851:head# 2024-02-09T17:57:11.993 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c5cc6dd:::benchmark_data_smithi179_148205_object108127:head# 2024-02-09T17:57:11.997 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c5dfd34:::benchmark_data_smithi179_148205_object13055:head# 2024-02-09T17:57:11.997 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c5e69db:::benchmark_data_smithi179_148205_object100726:head# 2024-02-09T17:57:11.997 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c5eb6b8:::benchmark_data_smithi179_148205_object104755:head# 2024-02-09T17:57:11.997 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c5f2a66:::benchmark_data_smithi179_148205_object34651:head# 2024-02-09T17:57:11.997 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c6018fa:::benchmark_data_smithi179_148205_object14909:head# 2024-02-09T17:57:11.998 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c603db5:::benchmark_data_smithi179_148205_object15756:head# 2024-02-09T17:57:11.998 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c60bc62:::benchmark_data_smithi179_148205_object66864:head# 2024-02-09T17:57:11.998 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c60bf07:::benchmark_data_smithi179_148205_object63409:head# 2024-02-09T17:57:11.998 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c616e24:::benchmark_data_smithi179_148205_object88115:head# 2024-02-09T17:57:11.999 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c618966:::benchmark_data_smithi179_148205_object69098:head# 2024-02-09T17:57:11.999 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c61aad3:::benchmark_data_smithi179_148205_object11609:head# 2024-02-09T17:57:12.000 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c632b36:::benchmark_data_smithi179_148205_object10704:head# 2024-02-09T17:57:12.000 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c6469ba:::benchmark_data_smithi179_148205_object5090:head# 2024-02-09T17:57:12.000 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c65a19c:::benchmark_data_smithi179_148205_object3986:head# 2024-02-09T17:57:12.001 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c661886:::benchmark_data_smithi179_148205_object74401:head# 2024-02-09T17:57:12.002 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c666ce0:::benchmark_data_smithi179_148205_object46577:head# 2024-02-09T17:57:12.002 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c66cc58:::benchmark_data_smithi179_148205_object55285:head# 2024-02-09T17:57:12.003 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c6701fd:::benchmark_data_smithi179_148205_object59520:head# 2024-02-09T17:57:12.008 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c67c068:::benchmark_data_smithi179_148205_object30810:head# 2024-02-09T17:57:12.008 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c688620:::benchmark_data_smithi179_148205_object5959:head# 2024-02-09T17:57:12.008 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c697208:::benchmark_data_smithi179_148205_object58144:head# 2024-02-09T17:57:12.008 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c6a69ea:::benchmark_data_smithi179_148205_object82330:head# 2024-02-09T17:57:12.008 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c6c3f1c:::benchmark_data_smithi179_148205_object20875:head# 2024-02-09T17:57:12.008 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c6c5c56:::benchmark_data_smithi179_148205_object93664:head# 2024-02-09T17:57:12.008 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c6d2854:::benchmark_data_smithi179_148205_object21837:head# 2024-02-09T17:57:12.008 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c6e7059:::benchmark_data_smithi179_148205_object51494:head# 2024-02-09T17:57:12.008 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c6ed402:::benchmark_data_smithi179_148205_object73254:head# 2024-02-09T17:57:12.008 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c6f51b6:::benchmark_data_smithi179_148205_object90084:head# 2024-02-09T17:57:12.009 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c6fc658:::benchmark_data_smithi179_148205_object101443:head# 2024-02-09T17:57:12.009 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c6fec6a:::benchmark_data_smithi179_148205_object76565:head# 2024-02-09T17:57:12.009 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c708c3a:::benchmark_data_smithi179_148205_object33620:head# 2024-02-09T17:57:12.009 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c7135a7:::benchmark_data_smithi179_148205_object72408:head# 2024-02-09T17:57:12.010 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c71640e:::benchmark_data_smithi179_148205_object11251:head# 2024-02-09T17:57:12.014 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c71f73a:::benchmark_data_smithi179_148205_object12359:head# 2024-02-09T17:57:12.014 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c723297:::benchmark_data_smithi179_148205_object21832:head# 2024-02-09T17:57:12.014 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c728bd1:::benchmark_data_smithi179_148205_object22972:head# 2024-02-09T17:57:12.014 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c72ee95:::benchmark_data_smithi179_148205_object54937:head# 2024-02-09T17:57:12.014 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c748fc2:::benchmark_data_smithi179_148205_object52068:head# 2024-02-09T17:57:12.014 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c74b3f8:::benchmark_data_smithi179_148205_object62611:head# 2024-02-09T17:57:12.014 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c74c745:::benchmark_data_smithi179_148205_object8672:head# 2024-02-09T17:57:12.015 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c757830:::benchmark_data_smithi179_148205_object64678:head# 2024-02-09T17:57:12.015 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c759b4b:::benchmark_data_smithi179_148205_object33624:head# 2024-02-09T17:57:12.015 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c77bfad:::benchmark_data_smithi179_148205_object45000:head# 2024-02-09T17:57:12.015 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c77ddd7:::benchmark_data_smithi179_148205_object67863:head# 2024-02-09T17:57:12.015 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c782704:::benchmark_data_smithi179_148205_object82835:head# 2024-02-09T17:57:12.015 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c78406d:::benchmark_data_smithi179_148205_object50267:head# 2024-02-09T17:57:12.016 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c78e515:::benchmark_data_smithi179_148205_object9349:head# 2024-02-09T17:57:12.016 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c7975d9:::benchmark_data_smithi179_148205_object60682:head# 2024-02-09T17:57:12.016 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c79f210:::benchmark_data_smithi179_148205_object105273:head# 2024-02-09T17:57:12.016 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c7a070e:::benchmark_data_smithi179_148205_object54551:head# 2024-02-09T17:57:12.016 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c7a6d3d:::benchmark_data_smithi179_148205_object13970:head# 2024-02-09T17:57:12.017 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c7a7c2d:::benchmark_data_smithi179_148205_object31853:head# 2024-02-09T17:57:12.017 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c7aeca0:::benchmark_data_smithi179_148205_object90891:head# 2024-02-09T17:57:12.018 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c7b0099:::benchmark_data_smithi179_148205_object32909:head# 2024-02-09T17:57:12.018 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c7be76a:::benchmark_data_smithi179_148205_object88062:head# 2024-02-09T17:57:12.018 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c7ce86a:::benchmark_data_smithi179_148205_object10846:head# 2024-02-09T17:57:12.019 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c7cf68c:::benchmark_data_smithi179_148205_object41090:head# 2024-02-09T17:57:12.023 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c7d05f1:::benchmark_data_smithi179_148205_object11663:head# 2024-02-09T17:57:12.023 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c7e6737:::benchmark_data_smithi179_148205_object103255:head# 2024-02-09T17:57:12.024 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c7f3d62:::benchmark_data_smithi179_148205_object945:head# 2024-02-09T17:57:12.024 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c7fbb42:::benchmark_data_smithi179_148205_object65643:head# 2024-02-09T17:57:12.024 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c8019db:::benchmark_data_smithi179_148205_object77054:head# 2024-02-09T17:57:12.024 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c804fd6:::benchmark_data_smithi179_148205_object70606:head# 2024-02-09T17:57:12.025 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c8100f6:::benchmark_data_smithi179_148205_object2474:head# 2024-02-09T17:57:12.026 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c81ab6a:::benchmark_data_smithi179_148205_object45351:head# 2024-02-09T17:57:12.026 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c830fde:::benchmark_data_smithi179_148205_object12148:head# 2024-02-09T17:57:12.026 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c832fe8:::benchmark_data_smithi179_148205_object104069:head# 2024-02-09T17:57:12.026 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c83b103:::benchmark_data_smithi179_148205_object104625:head# 2024-02-09T17:57:12.027 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c84a242:::benchmark_data_smithi179_148205_object23117:head# 2024-02-09T17:57:12.027 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c84c9e4:::benchmark_data_smithi179_148205_object106488:head# 2024-02-09T17:57:12.027 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c84ce70:::benchmark_data_smithi179_148205_object40033:head# 2024-02-09T17:57:12.031 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c85de18:::benchmark_data_smithi179_148205_object38844:head# 2024-02-09T17:57:12.031 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c868640:::benchmark_data_smithi179_148205_object7081:head# 2024-02-09T17:57:12.031 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c870566:::benchmark_data_smithi179_148205_object70042:head# 2024-02-09T17:57:12.031 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c8736cf:::benchmark_data_smithi179_148205_object939:head# 2024-02-09T17:57:12.032 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c87377c:::benchmark_data_smithi179_148205_object14670:head# 2024-02-09T17:57:12.032 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c876222:::benchmark_data_smithi179_148205_object30796:head# 2024-02-09T17:57:12.032 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c87c6f4:::benchmark_data_smithi179_148205_object57184:head# 2024-02-09T17:57:12.032 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c87dfce:::benchmark_data_smithi179_148205_object102026:head# 2024-02-09T17:57:12.032 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c881a4d:::benchmark_data_smithi179_148205_object83930:head# 2024-02-09T17:57:12.033 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c88a8dc:::benchmark_data_smithi179_148205_object109421:head# 2024-02-09T17:57:12.033 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c89c54c:::benchmark_data_smithi179_148205_object42095:head# 2024-02-09T17:57:12.033 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c8a826a:::benchmark_data_smithi179_148205_object7026:head# 2024-02-09T17:57:12.033 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c8b2b85:::benchmark_data_smithi179_148205_object108378:head# 2024-02-09T17:57:12.033 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c8bc0ae:::benchmark_data_smithi179_148205_object66124:head# 2024-02-09T17:57:12.033 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c8bc19b:::benchmark_data_smithi179_148205_object10680:head# 2024-02-09T17:57:12.034 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c8ccc2b:::benchmark_data_smithi179_148205_object42912:head# 2024-02-09T17:57:12.034 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c8ce8b0:::benchmark_data_smithi179_148205_object83647:head# 2024-02-09T17:57:12.035 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c8d2d1b:::benchmark_data_smithi179_148205_object20987:head# 2024-02-09T17:57:12.035 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c8d5261:::benchmark_data_smithi179_148205_object101464:head# 2024-02-09T17:57:12.035 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c8d80d4:::benchmark_data_smithi179_148205_object105654:head# 2024-02-09T17:57:12.036 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c8deabc:::benchmark_data_smithi179_148205_object7283:head# 2024-02-09T17:57:12.036 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c8f0b7c:::benchmark_data_smithi179_148205_object42551:head# 2024-02-09T17:57:12.037 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c8f693c:::benchmark_data_smithi179_148205_object47622:head# 2024-02-09T17:57:12.037 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c8f91dc:::benchmark_data_smithi179_148205_object598:head# 2024-02-09T17:57:12.038 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c8f9b48:::benchmark_data_smithi179_148205_object66352:head# 2024-02-09T17:57:12.038 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c8fc051:::benchmark_data_smithi179_148205_object99576:head# 2024-02-09T17:57:12.038 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c8fcc90:::benchmark_data_smithi179_148205_object95780:head# 2024-02-09T17:57:12.039 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c904ce8:::benchmark_data_smithi179_148205_object108316:head# 2024-02-09T17:57:12.039 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c905587:::benchmark_data_smithi179_148205_object83905:head# 2024-02-09T17:57:12.040 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c90e9c0:::benchmark_data_smithi179_148205_object101213:head# 2024-02-09T17:57:12.040 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c90ecbe:::benchmark_data_smithi179_148205_object72651:head# 2024-02-09T17:57:12.042 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c913a79:::benchmark_data_smithi179_148205_object98595:head# 2024-02-09T17:57:12.043 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c915676:::benchmark_data_smithi179_148205_object15788:head# 2024-02-09T17:57:12.043 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c917059:::benchmark_data_smithi179_148205_object31679:head# 2024-02-09T17:57:12.044 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c931385:::benchmark_data_smithi179_148205_object91551:head# 2024-02-09T17:57:12.044 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c938044:::benchmark_data_smithi179_148205_object109752:head# 2024-02-09T17:57:12.045 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c939feb:::benchmark_data_smithi179_148205_object45720:head# 2024-02-09T17:57:12.045 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c93c338:::benchmark_data_smithi179_148205_object109656:head# 2024-02-09T17:57:12.045 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c945de7:::benchmark_data_smithi179_148205_object10361:head# 2024-02-09T17:57:12.050 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c946e7a:::benchmark_data_smithi179_148205_object3643:head# 2024-02-09T17:57:12.050 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c94a5e8:::benchmark_data_smithi179_148205_object14118:head# 2024-02-09T17:57:12.050 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c94c19d:::benchmark_data_smithi179_148205_object6501:head# 2024-02-09T17:57:12.050 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c94e9da:::benchmark_data_smithi179_148205_object2781:head# 2024-02-09T17:57:12.050 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c95ccd8:::benchmark_data_smithi179_148205_object101211:head# 2024-02-09T17:57:12.050 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c9618e7:::benchmark_data_smithi179_148205_object43918:head# 2024-02-09T17:57:12.050 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c968c6a:::benchmark_data_smithi179_148205_object98216:head# 2024-02-09T17:57:12.050 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c96ac5c:::benchmark_data_smithi179_148205_object30328:head# 2024-02-09T17:57:12.050 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c96eff6:::benchmark_data_smithi179_148205_object101834:head# 2024-02-09T17:57:12.050 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c97c1c8:::benchmark_data_smithi179_148205_object43006:head# 2024-02-09T17:57:12.051 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c991237:::benchmark_data_smithi179_148205_object29821:head# 2024-02-09T17:57:12.051 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c9ae06e:::benchmark_data_smithi179_148205_object30121:head# 2024-02-09T17:57:12.051 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c9b3ade:::benchmark_data_smithi179_148205_object41644:head# 2024-02-09T17:57:12.051 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c9b6a6d:::benchmark_data_smithi179_148205_object63531:head# 2024-02-09T17:57:12.052 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c9c244a:::benchmark_data_smithi179_148205_object88008:head# 2024-02-09T17:57:12.052 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c9cbcb8:::benchmark_data_smithi179_148205_object107986:head# 2024-02-09T17:57:12.052 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c9e93be:::benchmark_data_smithi179_148205_object103348:head# 2024-02-09T17:57:12.053 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c9ef04e:::benchmark_data_smithi179_148205_object32657:head# 2024-02-09T17:57:12.053 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c9f048d:::benchmark_data_smithi179_148205_object8713:head# 2024-02-09T17:57:12.054 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1c9f12ee:::benchmark_data_smithi179_148205_object105958:head# 2024-02-09T17:57:12.055 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ca1587d:::benchmark_data_smithi179_148205_object49350:head# 2024-02-09T17:57:12.055 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ca22388:::benchmark_data_smithi179_148205_object25978:head# 2024-02-09T17:57:12.056 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ca31f28:::benchmark_data_smithi179_148205_object48753:head# 2024-02-09T17:57:12.056 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ca328c5:::benchmark_data_smithi179_148205_object15061:head# 2024-02-09T17:57:12.060 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ca37889:::benchmark_data_smithi179_148205_object84017:head# 2024-02-09T17:57:12.060 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ca3e0f6:::benchmark_data_smithi179_148205_object102487:head# 2024-02-09T17:57:12.061 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ca3f687:::benchmark_data_smithi179_148205_object24465:head# 2024-02-09T17:57:12.061 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ca41add:::benchmark_data_smithi179_148205_object109626:head# 2024-02-09T17:57:12.063 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ca497ac:::benchmark_data_smithi179_148205_object99717:head# 2024-02-09T17:57:12.064 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ca4bf59:::benchmark_data_smithi179_148205_object12398:head# 2024-02-09T17:57:12.064 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ca65968:::benchmark_data_smithi179_148205_object29840:head# 2024-02-09T17:57:12.064 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ca78518:::benchmark_data_smithi179_148205_object93184:head# 2024-02-09T17:57:12.065 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ca85339:::benchmark_data_smithi179_148205_object59741:head# 2024-02-09T17:57:12.065 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ca867ba:::benchmark_data_smithi179_148205_object62855:head# 2024-02-09T17:57:12.065 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ca87052:::benchmark_data_smithi179_148205_object58911:head# 2024-02-09T17:57:12.066 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ca8d647:::benchmark_data_smithi179_148205_object47697:head# 2024-02-09T17:57:12.066 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ca93397:::benchmark_data_smithi179_148205_object69552:head# 2024-02-09T17:57:12.067 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ca9470d:::benchmark_data_smithi179_148205_object89599:head# 2024-02-09T17:57:12.067 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ca98e7f:::benchmark_data_smithi179_148205_object41813:head# 2024-02-09T17:57:12.067 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ca9b048:::benchmark_data_smithi179_148205_object78799:head# 2024-02-09T17:57:12.072 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ca9f852:::benchmark_data_smithi179_148205_object49074:head# 2024-02-09T17:57:12.072 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cab5173:::benchmark_data_smithi179_148205_object33991:head# 2024-02-09T17:57:12.072 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cac3dbc:::benchmark_data_smithi179_148205_object83723:head# 2024-02-09T17:57:12.072 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cac9759:::benchmark_data_smithi179_148205_object11955:head# 2024-02-09T17:57:12.072 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cad2442:::benchmark_data_smithi179_148205_object53982:head# 2024-02-09T17:57:12.072 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cad7d6d:::benchmark_data_smithi179_148205_object95621:head# 2024-02-09T17:57:12.073 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cad9d88:::benchmark_data_smithi179_148205_object38807:head# 2024-02-09T17:57:12.073 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cade143:::benchmark_data_smithi179_148205_object74218:head# 2024-02-09T17:57:12.073 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1caecfce:::benchmark_data_smithi179_148205_object70706:head# 2024-02-09T17:57:12.076 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1caf53f2:::benchmark_data_smithi179_148205_object80736:head# 2024-02-09T17:57:12.076 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cafd67d:::benchmark_data_smithi179_148205_object1420:head# 2024-02-09T17:57:12.076 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cb02464:::benchmark_data_smithi179_148205_object61137:head# 2024-02-09T17:57:12.077 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cb02cc4:::benchmark_data_smithi179_148205_object6144:head# 2024-02-09T17:57:12.077 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cb06646:::benchmark_data_smithi179_148205_object37938:head# 2024-02-09T17:57:12.077 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cb093dc:::benchmark_data_smithi179_148205_object54603:head# 2024-02-09T17:57:12.078 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cb0b0fa:::benchmark_data_smithi179_148205_object42054:head# 2024-02-09T17:57:12.078 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cb1698c:::benchmark_data_smithi179_148205_object75202:head# 2024-02-09T17:57:12.079 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cb2179b:::benchmark_data_smithi179_148205_object96390:head# 2024-02-09T17:57:12.079 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cb218d4:::benchmark_data_smithi179_148205_object93313:head# 2024-02-09T17:57:12.080 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cb24848:::benchmark_data_smithi179_148205_object14901:head# 2024-02-09T17:57:12.080 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cb2bbbb:::benchmark_data_smithi179_148205_object83003:head# 2024-02-09T17:57:12.080 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cb3bcc5:::benchmark_data_smithi179_148205_object98352:head# 2024-02-09T17:57:12.080 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cb3e221:::benchmark_data_smithi179_148205_object99013:head# 2024-02-09T17:57:12.081 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cb3ff65:::benchmark_data_smithi179_148205_object28490:head# 2024-02-09T17:57:12.081 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cb425b6:::benchmark_data_smithi179_148205_object65668:head# 2024-02-09T17:57:12.082 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cb4478a:::benchmark_data_smithi179_148205_object47340:head# 2024-02-09T17:57:12.082 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cb475d9:::benchmark_data_smithi179_148205_object84870:head# 2024-02-09T17:57:12.083 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cb4ae68:::benchmark_data_smithi179_148205_object109784:head# 2024-02-09T17:57:12.083 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cb4eb8e:::benchmark_data_smithi179_148205_object96679:head# 2024-02-09T17:57:12.084 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cb5b6a3:::benchmark_data_smithi179_148205_object41850:head# 2024-02-09T17:57:12.084 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cb64ba3:::benchmark_data_smithi179_148205_object55875:head# 2024-02-09T17:57:12.084 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cb64dcf:::benchmark_data_smithi179_148205_object47837:head# 2024-02-09T17:57:12.085 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cb809ad:::benchmark_data_smithi179_148205_object56342:head# 2024-02-09T17:57:12.085 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cb86038:::benchmark_data_smithi179_148205_object86790:head# 2024-02-09T17:57:12.085 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cb8db16:::benchmark_data_smithi179_148205_object90944:head# 2024-02-09T17:57:12.086 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cb92275:::benchmark_data_smithi179_148205_object101037:head# 2024-02-09T17:57:12.086 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cb92bb1:::benchmark_data_smithi179_148205_object105133:head# 2024-02-09T17:57:12.087 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cb99a6e:::benchmark_data_smithi179_148205_object69933:head# 2024-02-09T17:57:12.087 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cb9e9f7:::benchmark_data_smithi179_148205_object33546:head# 2024-02-09T17:57:12.087 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cb9f1d9:::benchmark_data_smithi179_148205_object14936:head# 2024-02-09T17:57:12.087 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cbb42df:::benchmark_data_smithi179_148205_object39144:head# 2024-02-09T17:57:12.088 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cbb60ef:::benchmark_data_smithi179_148205_object64369:head# 2024-02-09T17:57:12.088 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cbbae25:::benchmark_data_smithi179_148205_object77800:head# 2024-02-09T17:57:12.091 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cbbd473:::benchmark_data_smithi179_148205_object55049:head# 2024-02-09T17:57:12.091 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cbc178b:::benchmark_data_smithi179_148205_object22534:head# 2024-02-09T17:57:12.091 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cbd437c:::benchmark_data_smithi179_148205_object76889:head# 2024-02-09T17:57:12.092 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cbd49af:::benchmark_data_smithi179_148205_object50800:head# 2024-02-09T17:57:12.092 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cbe5cd3:::benchmark_data_smithi179_148205_object12616:head# 2024-02-09T17:57:12.093 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cbe7129:::benchmark_data_smithi179_148205_object94250:head# 2024-02-09T17:57:12.093 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cbff31a:::benchmark_data_smithi179_148205_object54219:head# 2024-02-09T17:57:12.093 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cc12d82:::benchmark_data_smithi179_148205_object1533:head# 2024-02-09T17:57:12.094 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cc133f9:::benchmark_data_smithi179_148205_object81277:head# 2024-02-09T17:57:12.094 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cc191b3:::benchmark_data_smithi179_148205_object13618:head# 2024-02-09T17:57:12.095 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cc1c35d:::benchmark_data_smithi179_148205_object64251:head# 2024-02-09T17:57:12.095 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cc338b8:::benchmark_data_smithi179_148205_object39641:head# 2024-02-09T17:57:12.096 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cc371e4:::benchmark_data_smithi179_148205_object50823:head# 2024-02-09T17:57:12.096 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cc3c4f9:::benchmark_data_smithi179_148205_object65232:head# 2024-02-09T17:57:12.096 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cc4470e:::benchmark_data_smithi179_148205_object42904:head# 2024-02-09T17:57:12.097 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cc482bc:::benchmark_data_smithi179_148205_object50531:head# 2024-02-09T17:57:12.097 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cc4e3b9:::benchmark_data_smithi179_148205_object62385:head# 2024-02-09T17:57:12.098 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cc50680:::benchmark_data_smithi179_148205_object32415:head# 2024-02-09T17:57:12.098 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cc5e193:::benchmark_data_smithi179_148205_object40455:head# 2024-02-09T17:57:12.099 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cc6f8e6:::benchmark_data_smithi179_148205_object57242:head# 2024-02-09T17:57:12.101 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ccadd50:::benchmark_data_smithi179_148205_object66897:head# 2024-02-09T17:57:12.102 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ccc1e84:::benchmark_data_smithi179_148205_object92047:head# 2024-02-09T17:57:12.105 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ccd82ad:::benchmark_data_smithi179_148205_object27542:head# 2024-02-09T17:57:12.105 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ccde5a7:::benchmark_data_smithi179_148205_object66031:head# 2024-02-09T17:57:12.105 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cced164:::benchmark_data_smithi179_148205_object3089:head# 2024-02-09T17:57:12.106 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ccef793:::benchmark_data_smithi179_148205_object69543:head# 2024-02-09T17:57:12.109 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ccf20e3:::benchmark_data_smithi179_148205_object13890:head# 2024-02-09T17:57:12.109 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ccff622:::benchmark_data_smithi179_148205_object50648:head# 2024-02-09T17:57:12.110 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cd035e1:::benchmark_data_smithi179_148205_object71908:head# 2024-02-09T17:57:12.110 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cd146df:::benchmark_data_smithi179_148205_object66009:head# 2024-02-09T17:57:12.111 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cd15140:::benchmark_data_smithi179_148205_object52974:head# 2024-02-09T17:57:12.111 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cd179db:::benchmark_data_smithi179_148205_object14736:head# 2024-02-09T17:57:12.111 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cd1fc45:::benchmark_data_smithi179_148205_object91218:head# 2024-02-09T17:57:12.112 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cd2bb8d:::benchmark_data_smithi179_148205_object48069:head# 2024-02-09T17:57:12.112 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cd3083e:::benchmark_data_smithi179_148205_object75721:head# 2024-02-09T17:57:12.112 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cd32b67:::benchmark_data_smithi179_148205_object10597:head# 2024-02-09T17:57:12.113 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cd3c0f3:::benchmark_data_smithi179_148205_object47610:head# 2024-02-09T17:57:12.114 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cd3d126:::benchmark_data_smithi179_148205_object33677:head# 2024-02-09T17:57:12.117 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cd47611:::benchmark_data_smithi179_148205_object91163:head# 2024-02-09T17:57:12.117 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cd4bf15:::benchmark_data_smithi179_148205_object43537:head# 2024-02-09T17:57:12.117 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cd4deb2:::benchmark_data_smithi179_148205_object50562:head# 2024-02-09T17:57:12.117 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cd52391:::benchmark_data_smithi179_148205_object8282:head# 2024-02-09T17:57:12.117 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cd55e75:::benchmark_data_smithi179_148205_object70583:head# 2024-02-09T17:57:12.117 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cd65cf6:::benchmark_data_smithi179_148205_object91725:head# 2024-02-09T17:57:12.117 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cd7341b:::benchmark_data_smithi179_148205_object107751:head# 2024-02-09T17:57:12.118 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cd73d0e:::benchmark_data_smithi179_148205_object87891:head# 2024-02-09T17:57:12.118 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cd7af4e:::benchmark_data_smithi179_148205_object40101:head# 2024-02-09T17:57:12.118 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cda6137:::benchmark_data_smithi179_148205_object52725:head# 2024-02-09T17:57:12.118 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cdaaf82:::benchmark_data_smithi179_148205_object104171:head# 2024-02-09T17:57:12.119 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cdaed5d:::benchmark_data_smithi179_148205_object94750:head# 2024-02-09T17:57:12.119 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cdbb59a:::benchmark_data_smithi179_148205_object23293:head# 2024-02-09T17:57:12.120 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cdbfe81:::benchmark_data_smithi179_148205_object94395:head# 2024-02-09T17:57:12.120 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cdc016f:::benchmark_data_smithi179_148205_object61726:head# 2024-02-09T17:57:12.121 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cdc4889:::benchmark_data_smithi179_148205_object57475:head# 2024-02-09T17:57:12.121 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cdccf54:::benchmark_data_smithi179_148205_object70823:head# 2024-02-09T17:57:12.121 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cdd8a27:::benchmark_data_smithi179_148205_object26039:head# 2024-02-09T17:57:12.122 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cde1247:::benchmark_data_smithi179_148205_object109289:head# 2024-02-09T17:57:12.122 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cdf0ed7:::benchmark_data_smithi179_148205_object44793:head# 2024-02-09T17:57:12.122 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cdfc8aa:::benchmark_data_smithi179_148205_object2680:head# 2024-02-09T17:57:12.123 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ce0b0f3:::benchmark_data_smithi179_148205_object22234:head# 2024-02-09T17:57:12.126 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ce171cf:::benchmark_data_smithi179_148205_object74499:head# 2024-02-09T17:57:12.126 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ce1779b:::benchmark_data_smithi179_148205_object1189:head# 2024-02-09T17:57:12.127 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ce188ed:::benchmark_data_smithi179_148205_object82114:head# 2024-02-09T17:57:12.128 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ce21779:::benchmark_data_smithi179_148205_object76006:head# 2024-02-09T17:57:12.128 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ce25b4a:::benchmark_data_smithi179_148205_object80812:head# 2024-02-09T17:57:12.128 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ce28301:::benchmark_data_smithi179_148205_object6679:head# 2024-02-09T17:57:12.128 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ce2bb10:::benchmark_data_smithi179_148205_object8480:head# 2024-02-09T17:57:12.128 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ce34e9b:::benchmark_data_smithi179_148205_object47710:head# 2024-02-09T17:57:12.129 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ce3b0e8:::benchmark_data_smithi179_148205_object43716:head# 2024-02-09T17:57:12.131 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ce3bde4:::benchmark_data_smithi179_148205_object47031:head# 2024-02-09T17:57:12.131 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ce3f0c5:::benchmark_data_smithi179_148205_object100561:head# 2024-02-09T17:57:12.132 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ce47306:::benchmark_data_smithi179_148205_object95443:head# 2024-02-09T17:57:12.132 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ce47d6e:::benchmark_data_smithi179_148205_object20022:head# 2024-02-09T17:57:12.133 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ce55c80:::benchmark_data_smithi179_148205_object21705:head# 2024-02-09T17:57:12.133 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ce579d3:::benchmark_data_smithi179_148205_object96599:head# 2024-02-09T17:57:12.133 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ce5acdc:::benchmark_data_smithi179_148205_object34635:head# 2024-02-09T17:57:12.134 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ce607ab:::benchmark_data_smithi179_148205_object3657:head# 2024-02-09T17:57:12.137 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ce6233b:::benchmark_data_smithi179_148205_object38367:head# 2024-02-09T17:57:12.137 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ce6296a:::benchmark_data_smithi179_148205_object78755:head# 2024-02-09T17:57:12.137 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ce6421e:::benchmark_data_smithi179_148205_object56360:head# 2024-02-09T17:57:12.138 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ce75a5e:::benchmark_data_smithi179_148205_object70830:head# 2024-02-09T17:57:12.138 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ce78a5b:::benchmark_data_smithi179_148205_object4621:head# 2024-02-09T17:57:12.140 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cea2855:::benchmark_data_smithi179_148205_object23651:head# 2024-02-09T17:57:12.141 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ceac4b6:::benchmark_data_smithi179_148205_object78836:head# 2024-02-09T17:57:12.141 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ceb3509:::benchmark_data_smithi179_148205_object30340:head# 2024-02-09T17:57:12.142 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ceb8714:::benchmark_data_smithi179_148205_object101408:head# 2024-02-09T17:57:12.142 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cec0988:::benchmark_data_smithi179_148205_object77444:head# 2024-02-09T17:57:12.143 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cec30f8:::benchmark_data_smithi179_148205_object39294:head# 2024-02-09T17:57:12.143 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ced0ca3:::benchmark_data_smithi179_148205_object71457:head# 2024-02-09T17:57:12.143 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cee19bf:::benchmark_data_smithi179_148205_object39666:head# 2024-02-09T17:57:12.146 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cee29d8:::benchmark_data_smithi179_148205_object74027:head# 2024-02-09T17:57:12.146 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cef4156:::benchmark_data_smithi179_148205_object41505:head# 2024-02-09T17:57:12.147 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cef8e84:::benchmark_data_smithi179_148205_object29911:head# 2024-02-09T17:57:12.147 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cf11d3c:::benchmark_data_smithi179_148205_object39373:head# 2024-02-09T17:57:12.148 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cf2ab8f:::benchmark_data_smithi179_148205_object32953:head# 2024-02-09T17:57:12.148 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cf3c3a7:::benchmark_data_smithi179_148205_object41910:head# 2024-02-09T17:57:12.148 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cf42e8e:::benchmark_data_smithi179_148205_object20414:head# 2024-02-09T17:57:12.149 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cf439b9:::benchmark_data_smithi179_148205_object99918:head# 2024-02-09T17:57:12.149 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cf470dc:::benchmark_data_smithi179_148205_object60464:head# 2024-02-09T17:57:12.150 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cf4a9d4:::benchmark_data_smithi179_148205_object89063:head# 2024-02-09T17:57:12.150 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cf53a19:::benchmark_data_smithi179_148205_object93840:head# 2024-02-09T17:57:12.151 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cf5a218:::benchmark_data_smithi179_148205_object10457:head# 2024-02-09T17:57:12.151 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cf604e6:::benchmark_data_smithi179_148205_object16188:head# 2024-02-09T17:57:12.154 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cf69b01:::benchmark_data_smithi179_148205_object78997:head# 2024-02-09T17:57:12.154 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cf6dd81:::benchmark_data_smithi179_148205_object97127:head# 2024-02-09T17:57:12.154 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cf79353:::benchmark_data_smithi179_148205_object57452:head# 2024-02-09T17:57:12.155 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cf97d63:::benchmark_data_smithi179_148205_object14564:head# 2024-02-09T17:57:12.155 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cfafd4f:::benchmark_data_smithi179_148205_object16296:head# 2024-02-09T17:57:12.156 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cfb38b3:::benchmark_data_smithi179_148205_object79929:head# 2024-02-09T17:57:12.157 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cfc3b13:::benchmark_data_smithi179_148205_object69971:head# 2024-02-09T17:57:12.157 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cfc69af:::benchmark_data_smithi179_148205_object84605:head# 2024-02-09T17:57:12.158 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cfcb00a:::benchmark_data_smithi179_148205_object2874:head# 2024-02-09T17:57:12.158 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cfcccb7:::benchmark_data_smithi179_148205_object11958:head# 2024-02-09T17:57:12.159 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cfd8576:::benchmark_data_smithi179_148205_object85848:head# 2024-02-09T17:57:12.163 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cfe0f5b:::benchmark_data_smithi179_148205_object57961:head# 2024-02-09T17:57:12.163 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cfe10c0:::benchmark_data_smithi179_148205_object3113:head# 2024-02-09T17:57:12.164 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cfeaff8:::benchmark_data_smithi179_148205_object56091:head# 2024-02-09T17:57:12.180 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cff236d:::benchmark_data_smithi179_148205_object41009:head# 2024-02-09T17:57:12.180 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1cffda1e:::benchmark_data_smithi179_148205_object82259:head# 2024-02-09T17:57:12.182 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d005334:::benchmark_data_smithi179_148205_object22614:head# 2024-02-09T17:57:12.182 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d0053be:::benchmark_data_smithi179_148205_object24820:head# 2024-02-09T17:57:12.183 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d00ac55:::benchmark_data_smithi179_148205_object104731:head# 2024-02-09T17:57:12.183 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d00d7ce:::benchmark_data_smithi179_148205_object86196:head# 2024-02-09T17:57:12.188 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d013b71:::benchmark_data_smithi179_148205_object64733:head# 2024-02-09T17:57:12.188 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d021951:::benchmark_data_smithi179_148205_object51133:head# 2024-02-09T17:57:12.188 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d021f60:::benchmark_data_smithi179_148205_object71033:head# 2024-02-09T17:57:12.188 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d029d75:::benchmark_data_smithi179_148205_object43764:head# 2024-02-09T17:57:12.189 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d034517:::benchmark_data_smithi179_148205_object52482:head# 2024-02-09T17:57:12.189 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d039149:::benchmark_data_smithi179_148205_object55084:head# 2024-02-09T17:57:12.189 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d04eaaa:::benchmark_data_smithi179_148205_object74740:head# 2024-02-09T17:57:12.189 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d0551d4:::benchmark_data_smithi179_148205_object48809:head# 2024-02-09T17:57:12.189 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d061745:::benchmark_data_smithi179_148205_object4761:head# 2024-02-09T17:57:12.190 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d066284:::benchmark_data_smithi179_148205_object20966:head# 2024-02-09T17:57:12.190 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d066b04:::benchmark_data_smithi179_148205_object103475:head# 2024-02-09T17:57:12.190 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d07fd12:::benchmark_data_smithi179_148205_object71482:head# 2024-02-09T17:57:12.191 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d095265:::benchmark_data_smithi179_148205_object32771:head# 2024-02-09T17:57:12.191 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d0a3f2d:::benchmark_data_smithi179_148205_object94748:head# 2024-02-09T17:57:12.195 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d0a5b18:::benchmark_data_smithi179_148205_object92624:head# 2024-02-09T17:57:12.195 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d0adc0e:::benchmark_data_smithi179_148205_object83254:head# 2024-02-09T17:57:12.195 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d0b8983:::benchmark_data_smithi179_148205_object43792:head# 2024-02-09T17:57:12.196 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d0bc784:::benchmark_data_smithi179_148205_object93356:head# 2024-02-09T17:57:12.196 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d0be810:::benchmark_data_smithi179_148205_object9290:head# 2024-02-09T17:57:12.196 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d0c43d3:::benchmark_data_smithi179_148205_object91504:head# 2024-02-09T17:57:12.196 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d0c7b8e:::benchmark_data_smithi179_148205_object47645:head# 2024-02-09T17:57:12.200 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d0c95b9:::benchmark_data_smithi179_148205_object28384:head# 2024-02-09T17:57:12.200 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d0d44c4:::benchmark_data_smithi179_148205_object49068:head# 2024-02-09T17:57:12.200 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d0df689:::benchmark_data_smithi179_148205_object61359:head# 2024-02-09T17:57:12.200 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d105903:::benchmark_data_smithi179_148205_object39652:head# 2024-02-09T17:57:12.200 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d11ea2d:::benchmark_data_smithi179_148205_object85090:head# 2024-02-09T17:57:12.200 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d120bd5:::benchmark_data_smithi179_148205_object14837:head# 2024-02-09T17:57:12.200 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d127d3b:::benchmark_data_smithi179_148205_object68002:head# 2024-02-09T17:57:12.200 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d129c48:::benchmark_data_smithi179_148205_object47032:head# 2024-02-09T17:57:12.200 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d12dd61:::benchmark_data_smithi179_148205_object17117:head# 2024-02-09T17:57:12.200 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d14411e:::benchmark_data_smithi179_148205_object58233:head# 2024-02-09T17:57:12.200 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d14f384:::benchmark_data_smithi179_148205_object45312:head# 2024-02-09T17:57:12.200 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d163d80:::benchmark_data_smithi179_148205_object56648:head# 2024-02-09T17:57:12.201 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d16d994:::benchmark_data_smithi179_148205_object36910:head# 2024-02-09T17:57:12.205 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d1782d4:::benchmark_data_smithi179_148205_object48629:head# 2024-02-09T17:57:12.205 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d185aeb:::benchmark_data_smithi179_148205_object23275:head# 2024-02-09T17:57:12.206 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d18b03d:::benchmark_data_smithi179_148205_object72853:head# 2024-02-09T17:57:12.206 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d18fe8e:::benchmark_data_smithi179_148205_object44264:head# 2024-02-09T17:57:12.206 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d19ae4e:::benchmark_data_smithi179_148205_object59519:head# 2024-02-09T17:57:12.207 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d19f606:::benchmark_data_smithi179_148205_object80284:head# 2024-02-09T17:57:12.207 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d1d190b:::benchmark_data_smithi179_148205_object18784:head# 2024-02-09T17:57:12.208 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d1d841a:::benchmark_data_smithi179_148205_object63914:head# 2024-02-09T17:57:12.208 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d1db7b8:::benchmark_data_smithi179_148205_object27256:head# 2024-02-09T17:57:12.215 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d1e9edd:::benchmark_data_smithi179_148205_object99909:head# 2024-02-09T17:57:12.215 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d1ec410:::benchmark_data_smithi179_148205_object22242:head# 2024-02-09T17:57:12.215 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d205154:::benchmark_data_smithi179_148205_object50528:head# 2024-02-09T17:57:12.215 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d22e455:::benchmark_data_smithi179_148205_object58814:head# 2024-02-09T17:57:12.215 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d231618:::benchmark_data_smithi179_148205_object86962:head# 2024-02-09T17:57:12.215 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d232ae8:::benchmark_data_smithi179_148205_object50475:head# 2024-02-09T17:57:12.216 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d234e1e:::benchmark_data_smithi179_148205_object19263:head# 2024-02-09T17:57:12.216 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d25e137:::benchmark_data_smithi179_148205_object54001:head# 2024-02-09T17:57:12.216 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d2633c3:::benchmark_data_smithi179_148205_object40919:head# 2024-02-09T17:57:12.216 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d265d00:::benchmark_data_smithi179_148205_object86665:head# 2024-02-09T17:57:12.216 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d26c1d1:::benchmark_data_smithi179_148205_object100246:head# 2024-02-09T17:57:12.216 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d27342c:::benchmark_data_smithi179_148205_object73155:head# 2024-02-09T17:57:12.216 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d2761f0:::benchmark_data_smithi179_148205_object46197:head# 2024-02-09T17:57:12.216 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d2801b8:::benchmark_data_smithi179_148205_object57769:head# 2024-02-09T17:57:12.217 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d298f71:::benchmark_data_smithi179_148205_object109680:head# 2024-02-09T17:57:12.217 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d2a8ae1:::benchmark_data_smithi179_148205_object56132:head# 2024-02-09T17:57:12.218 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d2ad1b6:::benchmark_data_smithi179_148205_object16051:head# 2024-02-09T17:57:12.218 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d2b2634:::benchmark_data_smithi179_148205_object35987:head# 2024-02-09T17:57:12.219 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d2cda30:::benchmark_data_smithi179_148205_object4414:head# 2024-02-09T17:57:12.219 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d2e20f8:::benchmark_data_smithi179_148205_object24422:head# 2024-02-09T17:57:12.219 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d2e3512:::benchmark_data_smithi179_148205_object26238:head# 2024-02-09T17:57:12.223 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d2ef66b:::benchmark_data_smithi179_148205_object106275:head# 2024-02-09T17:57:12.223 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d2f3c99:::benchmark_data_smithi179_148205_object58934:head# 2024-02-09T17:57:12.223 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d2fabf9:::benchmark_data_smithi179_148205_object1289:head# 2024-02-09T17:57:12.223 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d2fc6fe:::benchmark_data_smithi179_148205_object25686:head# 2024-02-09T17:57:12.223 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d3077cf:::benchmark_data_smithi179_148205_object56227:head# 2024-02-09T17:57:12.223 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d30ce37:::benchmark_data_smithi179_148205_object33649:head# 2024-02-09T17:57:12.223 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d30d998:::benchmark_data_smithi179_148205_object10539:head# 2024-02-09T17:57:12.224 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d321f09:::benchmark_data_smithi179_148205_object53711:head# 2024-02-09T17:57:12.224 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d33746a:::benchmark_data_smithi179_148205_object76015:head# 2024-02-09T17:57:12.225 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d351d94:::benchmark_data_smithi179_148205_object39908:head# 2024-02-09T17:57:12.225 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d358a7e:::benchmark_data_smithi179_148205_object94773:head# 2024-02-09T17:57:12.226 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d358b87:::benchmark_data_smithi179_148205_object39920:head# 2024-02-09T17:57:12.226 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d35fcaa:::benchmark_data_smithi179_148205_object107742:head# 2024-02-09T17:57:12.227 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d362e7f:::benchmark_data_smithi179_148205_object107020:head# 2024-02-09T17:57:12.227 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d370005:::benchmark_data_smithi179_148205_object258:head# 2024-02-09T17:57:12.227 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d37894c:::benchmark_data_smithi179_148205_object66713:head# 2024-02-09T17:57:12.228 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d388c30:::benchmark_data_smithi179_148205_object108884:head# 2024-02-09T17:57:12.228 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d38a53f:::benchmark_data_smithi179_148205_object47657:head# 2024-02-09T17:57:12.228 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d39a361:::benchmark_data_smithi179_148205_object37322:head# 2024-02-09T17:57:12.229 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d3a07b0:::benchmark_data_smithi179_148205_object87488:head# 2024-02-09T17:57:12.229 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d3b8607:::benchmark_data_smithi179_148205_object64084:head# 2024-02-09T17:57:12.229 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d3c94da:::benchmark_data_smithi179_148205_object9311:head# 2024-02-09T17:57:12.230 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d3d7853:::benchmark_data_smithi179_148205_object109607:head# 2024-02-09T17:57:12.230 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d3dc2f3:::benchmark_data_smithi179_148205_object62394:head# 2024-02-09T17:57:12.231 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d3e5989:::benchmark_data_smithi179_148205_object71710:head# 2024-02-09T17:57:12.231 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d3e8633:::benchmark_data_smithi179_148205_object78236:head# 2024-02-09T17:57:12.231 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d3e9064:::benchmark_data_smithi179_148205_object25190:head# 2024-02-09T17:57:12.232 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d3f3cd2:::benchmark_data_smithi179_148205_object16064:head# 2024-02-09T17:57:12.232 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d3f92f1:::benchmark_data_smithi179_148205_object81451:head# 2024-02-09T17:57:12.233 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d3f95c0:::benchmark_data_smithi179_148205_object103481:head# 2024-02-09T17:57:12.233 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d3ff45c:::benchmark_data_smithi179_148205_object85645:head# 2024-02-09T17:57:12.233 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d401c27:::benchmark_data_smithi179_148205_object8430:head# 2024-02-09T17:57:12.234 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d40ac21:::benchmark_data_smithi179_148205_object24014:head# 2024-02-09T17:57:12.237 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d41f4d5:::benchmark_data_smithi179_148205_object63941:head# 2024-02-09T17:57:12.237 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d42312d:::benchmark_data_smithi179_148205_object67993:head# 2024-02-09T17:57:12.239 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d435512:::benchmark_data_smithi179_148205_object60635:head# 2024-02-09T17:57:12.239 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d4359d8:::benchmark_data_smithi179_148205_object102944:head# 2024-02-09T17:57:12.240 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d439916:::benchmark_data_smithi179_148205_object61597:head# 2024-02-09T17:57:12.241 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d446270:::benchmark_data_smithi179_148205_object104308:head# 2024-02-09T17:57:12.241 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d446632:::benchmark_data_smithi179_148205_object28282:head# 2024-02-09T17:57:12.242 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d45c58f:::benchmark_data_smithi179_148205_object6929:head# 2024-02-09T17:57:12.242 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d466e9f:::benchmark_data_smithi179_148205_object38596:head# 2024-02-09T17:57:12.242 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d468bfb:::benchmark_data_smithi179_148205_object59081:head# 2024-02-09T17:57:12.243 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d477f48:::benchmark_data_smithi179_148205_object77677:head# 2024-02-09T17:57:12.243 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d4821d6:::benchmark_data_smithi179_148205_object70988:head# 2024-02-09T17:57:12.244 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d48f27a:::benchmark_data_smithi179_148205_object36392:head# 2024-02-09T17:57:12.244 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d4923fe:::benchmark_data_smithi179_148205_object79941:head# 2024-02-09T17:57:12.244 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d496f3f:::benchmark_data_smithi179_148205_object72655:head# 2024-02-09T17:57:12.245 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d497c86:::benchmark_data_smithi179_148205_object7564:head# 2024-02-09T17:57:12.245 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d49e46d:::benchmark_data_smithi179_148205_object97098:head# 2024-02-09T17:57:12.245 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d4abfc3:::benchmark_data_smithi179_148205_object50419:head# 2024-02-09T17:57:12.246 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d4bcdc3:::benchmark_data_smithi179_148205_object65890:head# 2024-02-09T17:57:12.246 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d4c0583:::benchmark_data_smithi179_148205_object9404:head# 2024-02-09T17:57:12.247 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d4c5532:::benchmark_data_smithi179_148205_object104514:head# 2024-02-09T17:57:12.248 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d4c920b:::benchmark_data_smithi179_148205_object77994:head# 2024-02-09T17:57:12.248 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d4cea0a:::benchmark_data_smithi179_148205_object57345:head# 2024-02-09T17:57:12.249 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d4d0c0c:::benchmark_data_smithi179_148205_object106019:head# 2024-02-09T17:57:12.249 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d4d2291:::benchmark_data_smithi179_148205_object82224:head# 2024-02-09T17:57:12.249 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d4d6b8f:::benchmark_data_smithi179_148205_object5909:head# 2024-02-09T17:57:12.252 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d4f6b07:::benchmark_data_smithi179_148205_object13664:head# 2024-02-09T17:57:12.252 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d4f93ba:::benchmark_data_smithi179_148205_object54595:head# 2024-02-09T17:57:12.253 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d514e15:::benchmark_data_smithi179_148205_object91364:head# 2024-02-09T17:57:12.253 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d51612d:::benchmark_data_smithi179_148205_object5670:head# 2024-02-09T17:57:12.253 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d51f3a1:::benchmark_data_smithi179_148205_object3287:head# 2024-02-09T17:57:12.257 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d525be1:::benchmark_data_smithi179_148205_object70983:head# 2024-02-09T17:57:12.257 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d5395a1:::benchmark_data_smithi179_148205_object59715:head# 2024-02-09T17:57:12.257 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d54cdfb:::benchmark_data_smithi179_148205_object6661:head# 2024-02-09T17:57:12.257 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d553a6b:::benchmark_data_smithi179_148205_object57104:head# 2024-02-09T17:57:12.258 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d554bab:::benchmark_data_smithi179_148205_object77379:head# 2024-02-09T17:57:12.258 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d5624fc:::benchmark_data_smithi179_148205_object13645:head# 2024-02-09T17:57:12.259 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d5665d2:::benchmark_data_smithi179_148205_object45303:head# 2024-02-09T17:57:12.259 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d56a533:::benchmark_data_smithi179_148205_object7884:head# 2024-02-09T17:57:12.259 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d5750f7:::benchmark_data_smithi179_148205_object86454:head# 2024-02-09T17:57:12.262 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d577f0b:::benchmark_data_smithi179_148205_object91664:head# 2024-02-09T17:57:12.263 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d57f626:::benchmark_data_smithi179_148205_object61753:head# 2024-02-09T17:57:12.263 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d5a09eb:::benchmark_data_smithi179_148205_object23582:head# 2024-02-09T17:57:12.263 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d5b9b52:::benchmark_data_smithi179_148205_object65876:head# 2024-02-09T17:57:12.263 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d5cda47:::benchmark_data_smithi179_148205_object87319:head# 2024-02-09T17:57:12.263 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d5d3cdd:::benchmark_data_smithi179_148205_object29110:head# 2024-02-09T17:57:12.263 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d5d70ef:::benchmark_data_smithi179_148205_object89399:head# 2024-02-09T17:57:12.264 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d5de3ff:::benchmark_data_smithi179_148205_object9758:head# 2024-02-09T17:57:12.264 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d5eb4db:::benchmark_data_smithi179_148205_object64292:head# 2024-02-09T17:57:12.264 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d5ef1b7:::benchmark_data_smithi179_148205_object29753:head# 2024-02-09T17:57:12.264 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d5fe2ab:::benchmark_data_smithi179_148205_object81771:head# 2024-02-09T17:57:12.266 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d5ff0a8:::benchmark_data_smithi179_148205_object66738:head# 2024-02-09T17:57:12.266 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d60717b:::benchmark_data_smithi179_148205_object95675:head# 2024-02-09T17:57:12.267 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d60e86b:::benchmark_data_smithi179_148205_object1837:head# 2024-02-09T17:57:12.267 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d64aa95:::benchmark_data_smithi179_148205_object97097:head# 2024-02-09T17:57:12.267 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d64c688:::benchmark_data_smithi179_148205_object45168:head# 2024-02-09T17:57:12.271 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d654e6d:::benchmark_data_smithi179_148205_object94324:head# 2024-02-09T17:57:12.271 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d65985a:::benchmark_data_smithi179_148205_object13164:head# 2024-02-09T17:57:12.271 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d66a367:::benchmark_data_smithi179_148205_object92825:head# 2024-02-09T17:57:12.272 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d67b877:::benchmark_data_smithi179_148205_object46426:head# 2024-02-09T17:57:12.272 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d689b16:::benchmark_data_smithi179_148205_object59581:head# 2024-02-09T17:57:12.273 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d6931f8:::benchmark_data_smithi179_148205_object84941:head# 2024-02-09T17:57:12.276 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d6a4ff5:::benchmark_data_smithi179_148205_object82741:head# 2024-02-09T17:57:12.276 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d6a7080:::benchmark_data_smithi179_148205_object32041:head# 2024-02-09T17:57:12.276 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d6b1271:::benchmark_data_smithi179_148205_object380:head# 2024-02-09T17:57:12.276 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d6b7345:::benchmark_data_smithi179_148205_object72140:head# 2024-02-09T17:57:12.276 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d6b9641:::benchmark_data_smithi179_148205_object36876:head# 2024-02-09T17:57:12.276 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d6c0480:::benchmark_data_smithi179_148205_object70927:head# 2024-02-09T17:57:12.277 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d6c279a:::benchmark_data_smithi179_148205_object35500:head# 2024-02-09T17:57:12.277 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d6c9519:::benchmark_data_smithi179_148205_object39769:head# 2024-02-09T17:57:12.278 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d6d9a0b:::benchmark_data_smithi179_148205_object85709:head# 2024-02-09T17:57:12.278 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d6dc205:::benchmark_data_smithi179_148205_object29107:head# 2024-02-09T17:57:12.279 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d6e1f10:::benchmark_data_smithi179_148205_object92383:head# 2024-02-09T17:57:12.279 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d6f1830:::benchmark_data_smithi179_148205_object83992:head# 2024-02-09T17:57:12.279 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d6f23ae:::benchmark_data_smithi179_148205_object2868:head# 2024-02-09T17:57:12.280 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d6f83ed:::benchmark_data_smithi179_148205_object94790:head# 2024-02-09T17:57:12.280 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d6fafa5:::benchmark_data_smithi179_148205_object59184:head# 2024-02-09T17:57:12.281 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d6ffbcf:::benchmark_data_smithi179_148205_object109060:head# 2024-02-09T17:57:12.281 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d70dd05:::benchmark_data_smithi179_148205_object40523:head# 2024-02-09T17:57:12.282 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d712fca:::benchmark_data_smithi179_148205_object37995:head# 2024-02-09T17:57:12.283 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d740eb6:::benchmark_data_smithi179_148205_object57348:head# 2024-02-09T17:57:12.283 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d742adb:::benchmark_data_smithi179_148205_object88460:head# 2024-02-09T17:57:12.283 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d745cb8:::benchmark_data_smithi179_148205_object8445:head# 2024-02-09T17:57:12.284 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d74a6a5:::benchmark_data_smithi179_148205_object22457:head# 2024-02-09T17:57:12.284 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d74b248:::benchmark_data_smithi179_148205_object34285:head# 2024-02-09T17:57:12.284 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d751c0f:::benchmark_data_smithi179_148205_object38246:head# 2024-02-09T17:57:12.285 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d7598a0:::benchmark_data_smithi179_148205_object99255:head# 2024-02-09T17:57:12.285 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d7631ea:::benchmark_data_smithi179_148205_object95261:head# 2024-02-09T17:57:12.286 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d76f2d4:::benchmark_data_smithi179_148205_object30737:head# 2024-02-09T17:57:12.286 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d77f1b4:::benchmark_data_smithi179_148205_object109021:head# 2024-02-09T17:57:12.286 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d780b30:::benchmark_data_smithi179_148205_object77195:head# 2024-02-09T17:57:12.287 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d793101:::benchmark_data_smithi179_148205_object43276:head# 2024-02-09T17:57:12.287 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d7a010c:::benchmark_data_smithi179_148205_object78033:head# 2024-02-09T17:57:12.288 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d7a12fc:::benchmark_data_smithi179_148205_object34502:head# 2024-02-09T17:57:12.288 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d7bfb9e:::benchmark_data_smithi179_148205_object90321:head# 2024-02-09T17:57:12.288 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d7c3b2e:::benchmark_data_smithi179_148205_object80496:head# 2024-02-09T17:57:12.292 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d7c53f5:::benchmark_data_smithi179_148205_object67465:head# 2024-02-09T17:57:12.292 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d7c7a79:::benchmark_data_smithi179_148205_object56720:head# 2024-02-09T17:57:12.293 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d7c95cb:::benchmark_data_smithi179_148205_object83279:head# 2024-02-09T17:57:12.293 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d7d9139:::benchmark_data_smithi179_148205_object49105:head# 2024-02-09T17:57:12.293 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d7dc70f:::benchmark_data_smithi179_148205_object45324:head# 2024-02-09T17:57:12.293 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d7e26b6:::benchmark_data_smithi179_148205_object35911:head# 2024-02-09T17:57:12.293 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d7e2b8e:::benchmark_data_smithi179_148205_object45114:head# 2024-02-09T17:57:12.293 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d7f895a:::benchmark_data_smithi179_148205_object21575:head# 2024-02-09T17:57:12.294 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d7f9f09:::benchmark_data_smithi179_148205_object11521:head# 2024-02-09T17:57:12.294 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d819f52:::benchmark_data_smithi179_148205_object5111:head# 2024-02-09T17:57:12.294 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d81cc0d:::benchmark_data_smithi179_148205_object68047:head# 2024-02-09T17:57:12.294 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d821741:::benchmark_data_smithi179_148205_object36020:head# 2024-02-09T17:57:12.297 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d828b76:::benchmark_data_smithi179_148205_object80931:head# 2024-02-09T17:57:12.297 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d82c540:::benchmark_data_smithi179_148205_object51058:head# 2024-02-09T17:57:12.298 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d82d0a2:::benchmark_data_smithi179_148205_object41609:head# 2024-02-09T17:57:12.298 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d835942:::benchmark_data_smithi179_148205_object74756:head# 2024-02-09T17:57:12.298 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d84041d:::benchmark_data_smithi179_148205_object22632:head# 2024-02-09T17:57:12.300 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d84b71c:::benchmark_data_smithi179_148205_object10502:head# 2024-02-09T17:57:12.300 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d84e470:::benchmark_data_smithi179_148205_object35795:head# 2024-02-09T17:57:12.301 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d856f42:::benchmark_data_smithi179_148205_object22167:head# 2024-02-09T17:57:12.301 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d85cd3e:::benchmark_data_smithi179_148205_object35871:head# 2024-02-09T17:57:12.302 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d85d8cf:::benchmark_data_smithi179_148205_object23105:head# 2024-02-09T17:57:12.303 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d85f751:::benchmark_data_smithi179_148205_object10406:head# 2024-02-09T17:57:12.304 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d863735:::benchmark_data_smithi179_148205_object87714:head# 2024-02-09T17:57:12.304 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d874725:::benchmark_data_smithi179_148205_object15673:head# 2024-02-09T17:57:12.305 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d880397:::benchmark_data_smithi179_148205_object48752:head# 2024-02-09T17:57:12.305 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d88611c:::benchmark_data_smithi179_148205_object97071:head# 2024-02-09T17:57:12.306 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d8a375d:::benchmark_data_smithi179_148205_object23680:head# 2024-02-09T17:57:12.306 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d8a9e34:::benchmark_data_smithi179_148205_object40701:head# 2024-02-09T17:57:12.306 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d8b6b74:::benchmark_data_smithi179_148205_object102165:head# 2024-02-09T17:57:12.307 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d8c5a15:::benchmark_data_smithi179_148205_object52828:head# 2024-02-09T17:57:12.307 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d8c5d04:::benchmark_data_smithi179_148205_object45509:head# 2024-02-09T17:57:12.308 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d8c990e:::benchmark_data_smithi179_148205_object38868:head# 2024-02-09T17:57:12.308 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d8f2e8c:::benchmark_data_smithi179_148205_object70577:head# 2024-02-09T17:57:12.308 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d8f7ac8:::benchmark_data_smithi179_148205_object79416:head# 2024-02-09T17:57:12.309 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d8fb4ea:::benchmark_data_smithi179_148205_object14552:head# 2024-02-09T17:57:12.309 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d924a9d:::benchmark_data_smithi179_148205_object63380:head# 2024-02-09T17:57:12.310 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d92ffeb:::benchmark_data_smithi179_148205_object3457:head# 2024-02-09T17:57:12.310 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d934399:::benchmark_data_smithi179_148205_object47488:head# 2024-02-09T17:57:12.311 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d944caa:::benchmark_data_smithi179_148205_object72744:head# 2024-02-09T17:57:12.311 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d960852:::benchmark_data_smithi179_148205_object54341:head# 2024-02-09T17:57:12.311 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d96f4ed:::benchmark_data_smithi179_148205_object57838:head# 2024-02-09T17:57:12.312 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d97f81e:::benchmark_data_smithi179_148205_object28579:head# 2024-02-09T17:57:12.313 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d984492:::benchmark_data_smithi179_148205_object61549:head# 2024-02-09T17:57:12.313 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d985f5a:::benchmark_data_smithi179_148205_object43726:head# 2024-02-09T17:57:12.313 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d991253:::benchmark_data_smithi179_148205_object8006:head# 2024-02-09T17:57:12.313 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d9aa80a:::benchmark_data_smithi179_148205_object74353:head# 2024-02-09T17:57:12.314 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d9ab6df:::benchmark_data_smithi179_148205_object87987:head# 2024-02-09T17:57:12.314 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d9b2642:::benchmark_data_smithi179_148205_object16212:head# 2024-02-09T17:57:12.315 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d9b2961:::benchmark_data_smithi179_148205_object2698:head# 2024-02-09T17:57:12.315 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d9c2ddd:::benchmark_data_smithi179_148205_object56471:head# 2024-02-09T17:57:12.315 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d9c6317:::benchmark_data_smithi179_148205_object65301:head# 2024-02-09T17:57:12.316 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d9c7ed3:::benchmark_data_smithi179_148205_object62951:head# 2024-02-09T17:57:12.316 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d9cd890:::benchmark_data_smithi179_148205_object22337:head# 2024-02-09T17:57:12.316 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d9cde67:::benchmark_data_smithi179_148205_object50341:head# 2024-02-09T17:57:12.317 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d9d5b6f:::benchmark_data_smithi179_148205_object44685:head# 2024-02-09T17:57:12.317 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d9dc4bc:::benchmark_data_smithi179_148205_object96308:head# 2024-02-09T17:57:12.318 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d9e2714:::benchmark_data_smithi179_148205_object42742:head# 2024-02-09T17:57:12.318 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d9f31b7:::benchmark_data_smithi179_148205_object108971:head# 2024-02-09T17:57:12.322 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d9f6ac5:::benchmark_data_smithi179_148205_object38540:head# 2024-02-09T17:57:12.322 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d9fdb61:::benchmark_data_smithi179_148205_object48241:head# 2024-02-09T17:57:12.322 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1d9fe1ed:::benchmark_data_smithi179_148205_object13012:head# 2024-02-09T17:57:12.322 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1da0192b:::benchmark_data_smithi179_148205_object52956:head# 2024-02-09T17:57:12.322 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1da0d946:::benchmark_data_smithi179_148205_object102687:head# 2024-02-09T17:57:12.322 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1da12144:::benchmark_data_smithi179_148205_object17229:head# 2024-02-09T17:57:12.322 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1da19d36:::benchmark_data_smithi179_148205_object45191:head# 2024-02-09T17:57:12.322 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1da1f12d:::benchmark_data_smithi179_148205_object1095:head# 2024-02-09T17:57:12.323 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1da2169d:::benchmark_data_smithi179_148205_object48024:head# 2024-02-09T17:57:12.323 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1da41416:::benchmark_data_smithi179_148205_object67696:head# 2024-02-09T17:57:12.323 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1da54c69:::benchmark_data_smithi179_148205_object19106:head# 2024-02-09T17:57:12.323 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1da55909:::benchmark_data_smithi179_148205_object34869:head# 2024-02-09T17:57:12.324 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1da5bb6f:::benchmark_data_smithi179_148205_object24138:head# 2024-02-09T17:57:12.324 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1da6813d:::benchmark_data_smithi179_148205_object74517:head# 2024-02-09T17:57:12.325 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1da6c033:::benchmark_data_smithi179_148205_object95145:head# 2024-02-09T17:57:12.325 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1da6e4e6:::benchmark_data_smithi179_148205_object38694:head# 2024-02-09T17:57:12.326 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1da7978e:::benchmark_data_smithi179_148205_object87135:head# 2024-02-09T17:57:12.326 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1da799a0:::benchmark_data_smithi179_148205_object14913:head# 2024-02-09T17:57:12.327 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1da7ccda:::benchmark_data_smithi179_148205_object6270:head# 2024-02-09T17:57:12.327 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1da905e7:::benchmark_data_smithi179_148205_object29891:head# 2024-02-09T17:57:12.328 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1da9b0df:::benchmark_data_smithi179_148205_object80485:head# 2024-02-09T17:57:12.328 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1da9b37d:::benchmark_data_smithi179_148205_object58468:head# 2024-02-09T17:57:12.333 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1da9c371:::benchmark_data_smithi179_148205_object78446:head# 2024-02-09T17:57:12.333 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1da9ff8b:::benchmark_data_smithi179_148205_object3680:head# 2024-02-09T17:57:12.333 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dab0448:::benchmark_data_smithi179_148205_object29875:head# 2024-02-09T17:57:12.333 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dabaa3f:::benchmark_data_smithi179_148205_object17547:head# 2024-02-09T17:57:12.335 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dabb08b:::benchmark_data_smithi179_148205_object34513:head# 2024-02-09T17:57:12.336 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dac23d1:::benchmark_data_smithi179_148205_object9489:head# 2024-02-09T17:57:12.336 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dac3135:::benchmark_data_smithi179_148205_object43410:head# 2024-02-09T17:57:12.336 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dad3a53:::benchmark_data_smithi179_148205_object37643:head# 2024-02-09T17:57:12.337 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dae004f:::benchmark_data_smithi179_148205_object54785:head# 2024-02-09T17:57:12.337 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dafa2c3:::benchmark_data_smithi179_148205_object57612:head# 2024-02-09T17:57:12.338 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1db000c3:::benchmark_data_smithi179_148205_object70528:head# 2024-02-09T17:57:12.338 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1db096fd:::benchmark_data_smithi179_148205_object102102:head# 2024-02-09T17:57:12.339 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1db0ddea:::benchmark_data_smithi179_148205_object16944:head# 2024-02-09T17:57:12.340 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1db13cc0:::benchmark_data_smithi179_148205_object16560:head# 2024-02-09T17:57:12.343 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1db2108d:::benchmark_data_smithi179_148205_object71015:head# 2024-02-09T17:57:12.343 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1db3880a:::benchmark_data_smithi179_148205_object2595:head# 2024-02-09T17:57:12.344 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1db3f9f6:::benchmark_data_smithi179_148205_object60937:head# 2024-02-09T17:57:12.349 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1db44d0d:::benchmark_data_smithi179_148205_object10348:head# 2024-02-09T17:57:12.349 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1db555f3:::benchmark_data_smithi179_148205_object101779:head# 2024-02-09T17:57:12.349 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1db64dd4:::benchmark_data_smithi179_148205_object23325:head# 2024-02-09T17:57:12.349 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1db66a23:::benchmark_data_smithi179_148205_object108272:head# 2024-02-09T17:57:12.349 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1db6ea0d:::benchmark_data_smithi179_148205_object79465:head# 2024-02-09T17:57:12.349 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1db6fa9b:::benchmark_data_smithi179_148205_object58283:head# 2024-02-09T17:57:12.349 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1db73bff:::benchmark_data_smithi179_148205_object100552:head# 2024-02-09T17:57:12.353 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1db766fd:::benchmark_data_smithi179_148205_object59276:head# 2024-02-09T17:57:12.354 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1db7e9d8:::benchmark_data_smithi179_148205_object26500:head# 2024-02-09T17:57:12.354 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1db91d96:::benchmark_data_smithi179_148205_object95795:head# 2024-02-09T17:57:12.355 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1db99d2e:::benchmark_data_smithi179_148205_object25614:head# 2024-02-09T17:57:12.355 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dba8e61:::benchmark_data_smithi179_148205_object42259:head# 2024-02-09T17:57:12.355 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dbb3464:::benchmark_data_smithi179_148205_object70002:head# 2024-02-09T17:57:12.356 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dbbdfcc:::benchmark_data_smithi179_148205_object7997:head# 2024-02-09T17:57:12.356 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dbc2a1e:::benchmark_data_smithi179_148205_object69539:head# 2024-02-09T17:57:12.356 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dbe4f28:::benchmark_data_smithi179_148205_object20003:head# 2024-02-09T17:57:12.357 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dbf90d7:::benchmark_data_smithi179_148205_object44361:head# 2024-02-09T17:57:12.357 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dbff703:::benchmark_data_smithi179_148205_object101212:head# 2024-02-09T17:57:12.358 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dc07f23:::benchmark_data_smithi179_148205_object96847:head# 2024-02-09T17:57:12.358 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dc0b4fd:::benchmark_data_smithi179_148205_object19450:head# 2024-02-09T17:57:12.358 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dc0fc16:::benchmark_data_smithi179_148205_object53629:head# 2024-02-09T17:57:12.359 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dc140de:::benchmark_data_smithi179_148205_object58296:head# 2024-02-09T17:57:12.359 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dc1a362:::benchmark_data_smithi179_148205_object17680:head# 2024-02-09T17:57:12.360 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dc1bdf4:::benchmark_data_smithi179_148205_object107698:head# 2024-02-09T17:57:12.360 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dc204aa:::benchmark_data_smithi179_148205_object4033:head# 2024-02-09T17:57:12.360 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dc42a34:::benchmark_data_smithi179_148205_object61628:head# 2024-02-09T17:57:12.361 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dc4b908:::benchmark_data_smithi179_148205_object109328:head# 2024-02-09T17:57:12.361 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dc4dce2:::benchmark_data_smithi179_148205_object58138:head# 2024-02-09T17:57:12.361 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dc53a82:::benchmark_data_smithi179_148205_object50001:head# 2024-02-09T17:57:12.362 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dc58c94:::benchmark_data_smithi179_148205_object14325:head# 2024-02-09T17:57:12.362 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dc5a9c1:::benchmark_data_smithi179_148205_object25036:head# 2024-02-09T17:57:12.363 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dc66b29:::benchmark_data_smithi179_148205_object97113:head# 2024-02-09T17:57:12.363 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dc73276:::benchmark_data_smithi179_148205_object11796:head# 2024-02-09T17:57:12.363 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dc754ce:::benchmark_data_smithi179_148205_object71408:head# 2024-02-09T17:57:12.364 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dc7cac9:::benchmark_data_smithi179_148205_object104822:head# 2024-02-09T17:57:12.364 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dc8746a:::benchmark_data_smithi179_148205_object39359:head# 2024-02-09T17:57:12.365 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dc8cf4f:::benchmark_data_smithi179_148205_object77167:head# 2024-02-09T17:57:12.365 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dc99465:::benchmark_data_smithi179_148205_object92917:head# 2024-02-09T17:57:12.366 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dc9ec66:::benchmark_data_smithi179_148205_object39267:head# 2024-02-09T17:57:12.367 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dcb7241:::benchmark_data_smithi179_148205_object76677:head# 2024-02-09T17:57:12.367 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dcc2a07:::benchmark_data_smithi179_148205_object89770:head# 2024-02-09T17:57:12.368 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dcc76a7:::benchmark_data_smithi179_148205_object59354:head# 2024-02-09T17:57:12.368 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dcd826f:::benchmark_data_smithi179_148205_object68706:head# 2024-02-09T17:57:12.368 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dce866c:::benchmark_data_smithi179_148205_object3932:head# 2024-02-09T17:57:12.369 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dced314:::benchmark_data_smithi179_148205_object8917:head# 2024-02-09T17:57:12.369 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dcf213e:::benchmark_data_smithi179_148205_object76429:head# 2024-02-09T17:57:12.369 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dcfdea3:::benchmark_data_smithi179_148205_object2039:head# 2024-02-09T17:57:12.370 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dd0115e:::benchmark_data_smithi179_148205_object81414:head# 2024-02-09T17:57:12.370 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dd0b2f3:::benchmark_data_smithi179_148205_object45543:head# 2024-02-09T17:57:12.370 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dd1a813:::benchmark_data_smithi179_148205_object103002:head# 2024-02-09T17:57:12.371 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dd37388:::benchmark_data_smithi179_148205_object49246:head# 2024-02-09T17:57:12.371 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dd40510:::benchmark_data_smithi179_148205_object36051:head# 2024-02-09T17:57:12.371 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dd44e28:::benchmark_data_smithi179_148205_object16004:head# 2024-02-09T17:57:12.372 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dd44ea5:::benchmark_data_smithi179_148205_object81692:head# 2024-02-09T17:57:12.372 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dd4d348:::benchmark_data_smithi179_148205_object25658:head# 2024-02-09T17:57:12.372 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dd66e14:::benchmark_data_smithi179_148205_object44849:head# 2024-02-09T17:57:12.373 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dd71188:::benchmark_data_smithi179_148205_object2531:head# 2024-02-09T17:57:12.373 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dd776a7:::benchmark_data_smithi179_148205_object236:head# 2024-02-09T17:57:12.373 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dd7805b:::benchmark_data_smithi179_148205_object19075:head# 2024-02-09T17:57:12.374 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dd92d5b:::benchmark_data_smithi179_148205_object77552:head# 2024-02-09T17:57:12.374 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dd9c138:::benchmark_data_smithi179_148205_object23225:head# 2024-02-09T17:57:12.375 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dda04f4:::benchmark_data_smithi179_148205_object85790:head# 2024-02-09T17:57:12.375 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dda0632:::benchmark_data_smithi179_148205_object21573:head# 2024-02-09T17:57:12.375 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dda6dbd:::benchmark_data_smithi179_148205_object68440:head# 2024-02-09T17:57:12.376 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ddb1582:::benchmark_data_smithi179_148205_object68533:head# 2024-02-09T17:57:12.376 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ddc03ea:::benchmark_data_smithi179_148205_object46390:head# 2024-02-09T17:57:12.377 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ddc383a:::benchmark_data_smithi179_148205_object32749:head# 2024-02-09T17:57:12.377 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dde30ce:::benchmark_data_smithi179_148205_object98857:head# 2024-02-09T17:57:12.377 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ddefe90:::benchmark_data_smithi179_148205_object70188:head# 2024-02-09T17:57:12.378 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ddf4a19:::benchmark_data_smithi179_148205_object32567:head# 2024-02-09T17:57:12.378 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ddfa9db:::benchmark_data_smithi179_148205_object63539:head# 2024-02-09T17:57:12.379 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ddfbcf7:::benchmark_data_smithi179_148205_object102952:head# 2024-02-09T17:57:12.379 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ddff8ee:::benchmark_data_smithi179_148205_object107884:head# 2024-02-09T17:57:12.379 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1de009ca:::benchmark_data_smithi179_148205_object106788:head# 2024-02-09T17:57:12.380 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1de095b2:::benchmark_data_smithi179_148205_object57067:head# 2024-02-09T17:57:12.380 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1de13b7d:::benchmark_data_smithi179_148205_object79700:head# 2024-02-09T17:57:12.381 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1de202f9:::benchmark_data_smithi179_148205_object26082:head# 2024-02-09T17:57:12.382 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1de32cef:::benchmark_data_smithi179_148205_object11011:head# 2024-02-09T17:57:12.382 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1de3b830:::benchmark_data_smithi179_148205_object35188:head# 2024-02-09T17:57:12.382 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1de3f64c:::benchmark_data_smithi179_148205_object11324:head# 2024-02-09T17:57:12.383 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1de414bd:::benchmark_data_smithi179_148205_object54940:head# 2024-02-09T17:57:12.383 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1de62c5c:::benchmark_data_smithi179_148205_object51083:head# 2024-02-09T17:57:12.384 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1de68fb9:::benchmark_data_smithi179_148205_object22796:head# 2024-02-09T17:57:12.384 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1de6ad10:::benchmark_data_smithi179_148205_object83447:head# 2024-02-09T17:57:12.384 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1de70347:::benchmark_data_smithi179_148205_object37629:head# 2024-02-09T17:57:12.385 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1de7aa0b:::benchmark_data_smithi179_148205_object68918:head# 2024-02-09T17:57:12.385 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1de875c3:::benchmark_data_smithi179_148205_object80445:head# 2024-02-09T17:57:12.386 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1de8ecc7:::benchmark_data_smithi179_148205_object98272:head# 2024-02-09T17:57:12.387 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1de8f636:::benchmark_data_smithi179_148205_object13995:head# 2024-02-09T17:57:12.388 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1de97feb:::benchmark_data_smithi179_148205_object29488:head# 2024-02-09T17:57:12.388 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1de9f123:::benchmark_data_smithi179_148205_object26183:head# 2024-02-09T17:57:12.388 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dea8370:::benchmark_data_smithi179_148205_object43245:head# 2024-02-09T17:57:12.389 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1deaa93d:::benchmark_data_smithi179_148205_object89918:head# 2024-02-09T17:57:12.389 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1deaccfb:::benchmark_data_smithi179_148205_object31066:head# 2024-02-09T17:57:12.390 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1deb1c6e:::benchmark_data_smithi179_148205_object64674:head# 2024-02-09T17:57:12.390 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1decfa3a:::benchmark_data_smithi179_148205_object31195:head# 2024-02-09T17:57:12.392 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1deeba76:::benchmark_data_smithi179_148205_object65531:head# 2024-02-09T17:57:12.392 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1deed56a:::benchmark_data_smithi179_148205_object21047:head# 2024-02-09T17:57:12.392 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1deefc7d:::benchmark_data_smithi179_148205_object74592:head# 2024-02-09T17:57:12.393 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1defd44c:::benchmark_data_smithi179_148205_object42260:head# 2024-02-09T17:57:12.393 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1defe04d:::benchmark_data_smithi179_148205_object49002:head# 2024-02-09T17:57:12.394 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1df034b9:::benchmark_data_smithi179_148205_object25932:head# 2024-02-09T17:57:12.397 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1df0e0c0:::benchmark_data_smithi179_148205_object17465:head# 2024-02-09T17:57:12.397 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1df116f5:::benchmark_data_smithi179_148205_object74701:head# 2024-02-09T17:57:12.397 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1df1acfb:::benchmark_data_smithi179_148205_object68286:head# 2024-02-09T17:57:12.397 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1df21082:::benchmark_data_smithi179_148205_object28260:head# 2024-02-09T17:57:12.397 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1df2b155:::benchmark_data_smithi179_148205_object5528:head# 2024-02-09T17:57:12.397 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1df31ec7:::benchmark_data_smithi179_148205_object87143:head# 2024-02-09T17:57:12.397 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1df35e9d:::benchmark_data_smithi179_148205_object4721:head# 2024-02-09T17:57:12.397 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1df3daaa:::benchmark_data_smithi179_148205_object84551:head# 2024-02-09T17:57:12.398 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1df47144:::benchmark_data_smithi179_148205_object64439:head# 2024-02-09T17:57:12.399 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1df4c7be:::benchmark_data_smithi179_148205_object64580:head# 2024-02-09T17:57:12.399 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1df539ae:::benchmark_data_smithi179_148205_object64911:head# 2024-02-09T17:57:12.399 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1df54935:::benchmark_data_smithi179_148205_object74398:head# 2024-02-09T17:57:12.402 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1df556f9:::benchmark_data_smithi179_148205_object72507:head# 2024-02-09T17:57:12.403 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1df56bcd:::benchmark_data_smithi179_148205_object96760:head# 2024-02-09T17:57:12.403 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1df585c6:::benchmark_data_smithi179_148205_object6821:head# 2024-02-09T17:57:12.403 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1df63e98:::benchmark_data_smithi179_148205_object24727:head# 2024-02-09T17:57:12.404 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1df6a27b:::benchmark_data_smithi179_148205_object106270:head# 2024-02-09T17:57:12.404 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1df6d941:::benchmark_data_smithi179_148205_object52544:head# 2024-02-09T17:57:12.405 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1df70d46:::benchmark_data_smithi179_148205_object61022:head# 2024-02-09T17:57:12.405 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1df78e59:::benchmark_data_smithi179_148205_object34223:head# 2024-02-09T17:57:12.407 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1df96a77:::benchmark_data_smithi179_148205_object54009:head# 2024-02-09T17:57:12.407 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dfb1c54:::benchmark_data_smithi179_148205_object56406:head# 2024-02-09T17:57:12.408 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dfb210e:::benchmark_data_smithi179_148205_object76623:head# 2024-02-09T17:57:12.408 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dfc8ac7:::benchmark_data_smithi179_148205_object16547:head# 2024-02-09T17:57:12.408 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dfd3308:::benchmark_data_smithi179_148205_object5996:head# 2024-02-09T17:57:12.410 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dfd669f:::benchmark_data_smithi179_148205_object94444:head# 2024-02-09T17:57:12.411 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dfe5001:::benchmark_data_smithi179_148205_object54142:head# 2024-02-09T17:57:12.411 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dfe6e90:::benchmark_data_smithi179_148205_object15653:head# 2024-02-09T17:57:12.412 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dfe87b3:::benchmark_data_smithi179_148205_object30567:head# 2024-02-09T17:57:12.412 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1dfecec8:::benchmark_data_smithi179_148205_object87908:head# 2024-02-09T17:57:12.413 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e0099cc:::benchmark_data_smithi179_148205_object511:head# 2024-02-09T17:57:12.413 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e00d08e:::benchmark_data_smithi179_148205_object94427:head# 2024-02-09T17:57:12.413 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e0121ff:::benchmark_data_smithi179_148205_object54686:head# 2024-02-09T17:57:12.414 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e02530d:::benchmark_data_smithi179_148205_object66504:head# 2024-02-09T17:57:12.414 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e03d70d:::benchmark_data_smithi179_148205_object63615:head# 2024-02-09T17:57:12.417 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e041a14:::benchmark_data_smithi179_148205_object88794:head# 2024-02-09T17:57:12.418 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e04d07f:::benchmark_data_smithi179_148205_object35491:head# 2024-02-09T17:57:12.418 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e06c849:::benchmark_data_smithi179_148205_object11049:head# 2024-02-09T17:57:12.418 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e07653f:::benchmark_data_smithi179_148205_object30480:head# 2024-02-09T17:57:12.418 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e0836b7:::benchmark_data_smithi179_148205_object33098:head# 2024-02-09T17:57:12.418 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e092341:::benchmark_data_smithi179_148205_object103458:head# 2024-02-09T17:57:12.418 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e09adf9:::benchmark_data_smithi179_148205_object15333:head# 2024-02-09T17:57:12.419 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e09e4bc:::benchmark_data_smithi179_148205_object29157:head# 2024-02-09T17:57:12.419 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e09e587:::benchmark_data_smithi179_148205_object81232:head# 2024-02-09T17:57:12.419 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e0aebc7:::benchmark_data_smithi179_148205_object49157:head# 2024-02-09T17:57:12.419 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e0bcdb7:::benchmark_data_smithi179_148205_object69834:head# 2024-02-09T17:57:12.419 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e0c7e56:::benchmark_data_smithi179_148205_object60252:head# 2024-02-09T17:57:12.420 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e0e0ca1:::benchmark_data_smithi179_148205_object50740:head# 2024-02-09T17:57:12.420 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e0ee7cc:::benchmark_data_smithi179_148205_object9581:head# 2024-02-09T17:57:12.420 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e0f5103:::benchmark_data_smithi179_148205_object3045:head# 2024-02-09T17:57:12.421 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e0fb8da:::benchmark_data_smithi179_148205_object6029:head# 2024-02-09T17:57:12.421 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e10f4fb:::benchmark_data_smithi179_148205_object18002:head# 2024-02-09T17:57:12.422 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e11d59a:::benchmark_data_smithi179_148205_object51552:head# 2024-02-09T17:57:12.422 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e1216a1:::benchmark_data_smithi179_148205_object71046:head# 2024-02-09T17:57:12.422 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e127dae:::benchmark_data_smithi179_148205_object78720:head# 2024-02-09T17:57:12.423 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e1359fe:::benchmark_data_smithi179_148205_object43678:head# 2024-02-09T17:57:12.423 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:57:12.424 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e145e4b:::benchmark_data_smithi179_148205_object104397:head# 2024-02-09T17:57:12.424 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e167fd6:::benchmark_data_smithi179_148205_object11682:head# 2024-02-09T17:57:12.425 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e169081:::benchmark_data_smithi179_148205_object26002:head# 2024-02-09T17:57:12.425 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e1709ce:::benchmark_data_smithi179_148205_object28362:head# 2024-02-09T17:57:12.426 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e1715de:::benchmark_data_smithi179_148205_object88643:head# 2024-02-09T17:57:12.427 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e18265a:::benchmark_data_smithi179_148205_object13737:head# 2024-02-09T17:57:12.427 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e186bad:::benchmark_data_smithi179_148205_object25895:head# 2024-02-09T17:57:12.427 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e19e516:::benchmark_data_smithi179_148205_object104091:head# 2024-02-09T17:57:12.427 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e1a80bb:::benchmark_data_smithi179_148205_object44060:head# 2024-02-09T17:57:12.427 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e1b6ee9:::benchmark_data_smithi179_148205_object81159:head# 2024-02-09T17:57:12.427 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e1b7bcf:::benchmark_data_smithi179_148205_object20241:head# 2024-02-09T17:57:12.428 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e1b9743:::benchmark_data_smithi179_148205_object12151:head# 2024-02-09T17:57:12.430 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:12 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:57:12.430 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:12 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:57:12.430 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:12 smithi099 ceph-mon[39420]: pgmap v2037: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 6.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 826 op/s; 21384/337380 objects degraded (6.338%) 2024-02-09T17:57:12.430 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:12 smithi099 ceph-mon[39420]: osdmap e825: 12 total, 11 up, 12 in 2024-02-09T17:57:12.431 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:12 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 21384/337380 objects degraded (6.338%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:57:12.433 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e1c94b6:::benchmark_data_smithi179_148205_object104568:head# 2024-02-09T17:57:12.433 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e1d851d:::benchmark_data_smithi179_148205_object18714:head# 2024-02-09T17:57:12.434 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e1dea5d:::benchmark_data_smithi179_148205_object42565:head# 2024-02-09T17:57:12.434 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e1e3868:::benchmark_data_smithi179_148205_object24889:head# 2024-02-09T17:57:12.434 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e1e6fe1:::benchmark_data_smithi179_148205_object57738:head# 2024-02-09T17:57:12.435 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e1ea25f:::benchmark_data_smithi179_148205_object74608:head# 2024-02-09T17:57:12.435 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e1ec169:::benchmark_data_smithi179_148205_object56816:head# 2024-02-09T17:57:12.436 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e1f4dca:::benchmark_data_smithi179_148205_object57066:head# 2024-02-09T17:57:12.436 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e202797:::benchmark_data_smithi179_148205_object85237:head# 2024-02-09T17:57:12.437 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e203e7b:::benchmark_data_smithi179_148205_object55356:head# 2024-02-09T17:57:12.437 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e20463b:::benchmark_data_smithi179_148205_object38825:head# 2024-02-09T17:57:12.437 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e20661d:::benchmark_data_smithi179_148205_object88085:head# 2024-02-09T17:57:12.438 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e207061:::benchmark_data_smithi179_148205_object85735:head# 2024-02-09T17:57:12.438 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e20f2a6:::benchmark_data_smithi179_148205_object32730:head# 2024-02-09T17:57:12.439 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e218863:::benchmark_data_smithi179_148205_object102855:head# 2024-02-09T17:57:12.439 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e22a1a0:::benchmark_data_smithi179_148205_object47035:head# 2024-02-09T17:57:12.440 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e230f41:::benchmark_data_smithi179_148205_object33060:head# 2024-02-09T17:57:12.441 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e235107:::benchmark_data_smithi179_148205_object90899:head# 2024-02-09T17:57:12.441 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e25780e:::benchmark_data_smithi179_148205_object77368:head# 2024-02-09T17:57:12.441 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e25d668:::benchmark_data_smithi179_148205_object66149:head# 2024-02-09T17:57:12.442 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e269df0:::benchmark_data_smithi179_148205_object65147:head# 2024-02-09T17:57:12.442 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e284e33:::benchmark_data_smithi179_148205_object66107:head# 2024-02-09T17:57:12.442 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e2862e6:::benchmark_data_smithi179_148205_object27213:head# 2024-02-09T17:57:12.443 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e28a280:::benchmark_data_smithi179_148205_object98746:head# 2024-02-09T17:57:12.443 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e29cd4c:::benchmark_data_smithi179_148205_object22261:head# 2024-02-09T17:57:12.444 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e2a13f0:::benchmark_data_smithi179_148205_object5599:head# 2024-02-09T17:57:12.448 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e2a56c3:::benchmark_data_smithi179_148205_object83051:head# 2024-02-09T17:57:12.448 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e2b3844:::benchmark_data_smithi179_148205_object59135:head# 2024-02-09T17:57:12.448 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e2d9091:::benchmark_data_smithi179_148205_object30096:head# 2024-02-09T17:57:12.448 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e2e03b6:::benchmark_data_smithi179_148205_object78499:head# 2024-02-09T17:57:12.448 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e2eb4ab:::benchmark_data_smithi179_148205_object24500:head# 2024-02-09T17:57:12.448 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e2ee4c9:::benchmark_data_smithi179_148205_object57910:head# 2024-02-09T17:57:12.448 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e2f19f4:::benchmark_data_smithi179_148205_object106291:head# 2024-02-09T17:57:12.448 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e2f45e8:::benchmark_data_smithi179_148205_object70716:head# 2024-02-09T17:57:12.450 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e2f65c4:::benchmark_data_smithi179_148205_object106475:head# 2024-02-09T17:57:12.453 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e2f8a4e:::benchmark_data_smithi179_148205_object98616:head# 2024-02-09T17:57:12.454 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e2ffb6c:::benchmark_data_smithi179_148205_object11529:head# 2024-02-09T17:57:12.454 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e30801c:::benchmark_data_smithi179_148205_object47207:head# 2024-02-09T17:57:12.455 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e30a6a5:::benchmark_data_smithi179_148205_object19678:head# 2024-02-09T17:57:12.455 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e30b42f:::benchmark_data_smithi179_148205_object63348:head# 2024-02-09T17:57:12.455 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e313f33:::benchmark_data_smithi179_148205_object91311:head# 2024-02-09T17:57:12.456 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e323bab:::benchmark_data_smithi179_148205_object91515:head# 2024-02-09T17:57:12.472 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e326231:::benchmark_data_smithi179_148205_object6864:head# 2024-02-09T17:57:12.473 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e33af57:::benchmark_data_smithi179_148205_object74814:head# 2024-02-09T17:57:12.479 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e3413b2:::benchmark_data_smithi179_148205_object22082:head# 2024-02-09T17:57:12.480 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e34cc9d:::benchmark_data_smithi179_148205_object59663:head# 2024-02-09T17:57:12.480 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e34ed72:::benchmark_data_smithi179_148205_object42067:head# 2024-02-09T17:57:12.481 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e350e9b:::benchmark_data_smithi179_148205_object87394:head# 2024-02-09T17:57:12.486 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e362671:::benchmark_data_smithi179_148205_object37536:head# 2024-02-09T17:57:12.486 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e36927b:::benchmark_data_smithi179_148205_object42415:head# 2024-02-09T17:57:12.487 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e36e541:::benchmark_data_smithi179_148205_object86676:head# 2024-02-09T17:57:12.487 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e36e583:::benchmark_data_smithi179_148205_object41022:head# 2024-02-09T17:57:12.490 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e377fa3:::benchmark_data_smithi179_148205_object18568:head# 2024-02-09T17:57:12.490 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e379506:::benchmark_data_smithi179_148205_object91904:head# 2024-02-09T17:57:12.490 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e37e096:::benchmark_data_smithi179_148205_object104059:head# 2024-02-09T17:57:12.490 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e3848eb:::benchmark_data_smithi179_148205_object23131:head# 2024-02-09T17:57:12.491 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e385779:::benchmark_data_smithi179_148205_object15426:head# 2024-02-09T17:57:12.491 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e391fd2:::benchmark_data_smithi179_148205_object26305:head# 2024-02-09T17:57:12.492 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e3a02fd:::benchmark_data_smithi179_148205_object79272:head# 2024-02-09T17:57:12.492 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e3a2268:::benchmark_data_smithi179_148205_object52641:head# 2024-02-09T17:57:12.493 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e3bedcd:::benchmark_data_smithi179_148205_object50667:head# 2024-02-09T17:57:12.493 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e3bf446:::benchmark_data_smithi179_148205_object45991:head# 2024-02-09T17:57:12.494 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e3c3039:::benchmark_data_smithi179_148205_object51724:head# 2024-02-09T17:57:12.494 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e400ebe:::benchmark_data_smithi179_148205_object108326:head# 2024-02-09T17:57:12.494 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e4027b6:::benchmark_data_smithi179_148205_object50668:head# 2024-02-09T17:57:12.495 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e40daa3:::benchmark_data_smithi179_148205_object20952:head# 2024-02-09T17:57:12.495 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e40e2c0:::benchmark_data_smithi179_148205_object50354:head# 2024-02-09T17:57:12.495 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e416f50:::benchmark_data_smithi179_148205_object57983:head# 2024-02-09T17:57:12.495 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e4202ff:::benchmark_data_smithi179_148205_object16924:head# 2024-02-09T17:57:12.496 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e424ee1:::benchmark_data_smithi179_148205_object31863:head# 2024-02-09T17:57:12.496 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e434a34:::benchmark_data_smithi179_148205_object93352:head# 2024-02-09T17:57:12.497 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e43c003:::benchmark_data_smithi179_148205_object29776:head# 2024-02-09T17:57:12.497 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e443502:::benchmark_data_smithi179_148205_object105678:head# 2024-02-09T17:57:12.498 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e444c6a:::benchmark_data_smithi179_148205_object70566:head# 2024-02-09T17:57:12.498 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e44f23b:::benchmark_data_smithi179_148205_object90153:head# 2024-02-09T17:57:12.498 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e457174:::benchmark_data_smithi179_148205_object104076:head# 2024-02-09T17:57:12.499 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e45b848:::benchmark_data_smithi179_148205_object68284:head# 2024-02-09T17:57:12.503 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e48630c:::benchmark_data_smithi179_148205_object94955:head# 2024-02-09T17:57:12.504 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e487094:::benchmark_data_smithi179_148205_object48301:head# 2024-02-09T17:57:12.509 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e48a7d6:::benchmark_data_smithi179_148205_object27026:head# 2024-02-09T17:57:12.510 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e497f7b:::benchmark_data_smithi179_148205_object36813:head# 2024-02-09T17:57:12.510 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e4a82a7:::benchmark_data_smithi179_148205_object91962:head# 2024-02-09T17:57:12.510 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e4d7664:::benchmark_data_smithi179_148205_object18566:head# 2024-02-09T17:57:12.510 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e4f081b:::benchmark_data_smithi179_148205_object75851:head# 2024-02-09T17:57:12.510 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e4f1423:::benchmark_data_smithi179_148205_object29151:head# 2024-02-09T17:57:12.510 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e4f26b1:::benchmark_data_smithi179_148205_object3266:head# 2024-02-09T17:57:12.510 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e4f77b5:::benchmark_data_smithi179_148205_object27608:head# 2024-02-09T17:57:12.511 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e4f8062:::benchmark_data_smithi179_148205_object107396:head# 2024-02-09T17:57:12.511 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e4fec44:::benchmark_data_smithi179_148205_object25067:head# 2024-02-09T17:57:12.512 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e4fed89:::benchmark_data_smithi179_148205_object69982:head# 2024-02-09T17:57:12.512 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e503423:::benchmark_data_smithi179_148205_object54543:head# 2024-02-09T17:57:12.512 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e506801:::benchmark_data_smithi179_148205_object108371:head# 2024-02-09T17:57:12.513 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e50eb3c:::benchmark_data_smithi179_148205_object50582:head# 2024-02-09T17:57:12.513 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e512788:::benchmark_data_smithi179_148205_object108128:head# 2024-02-09T17:57:12.513 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e51a39d:::benchmark_data_smithi179_148205_object106577:head# 2024-02-09T17:57:12.514 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e5244c9:::benchmark_data_smithi179_148205_object11315:head# 2024-02-09T17:57:12.514 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e5297ac:::benchmark_data_smithi179_148205_object3088:head# 2024-02-09T17:57:12.515 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e5321c9:::benchmark_data_smithi179_148205_object78182:head# 2024-02-09T17:57:12.515 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e53977a:::benchmark_data_smithi179_148205_object109023:head# 2024-02-09T17:57:12.515 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e53b0f9:::benchmark_data_smithi179_148205_object74715:head# 2024-02-09T17:57:12.516 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e548127:::benchmark_data_smithi179_148205_object10170:head# 2024-02-09T17:57:12.516 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e55e11e:::benchmark_data_smithi179_148205_object77623:head# 2024-02-09T17:57:12.517 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e567064:::benchmark_data_smithi179_148205_object70393:head# 2024-02-09T17:57:12.517 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e5674fc:::benchmark_data_smithi179_148205_object94401:head# 2024-02-09T17:57:12.518 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e56b975:::benchmark_data_smithi179_148205_object36405:head# 2024-02-09T17:57:12.518 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e576624:::benchmark_data_smithi179_148205_object24020:head# 2024-02-09T17:57:12.519 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e57da6f:::benchmark_data_smithi179_148205_object87739:head# 2024-02-09T17:57:12.519 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e57f012:::benchmark_data_smithi179_148205_object3858:head# 2024-02-09T17:57:12.519 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e583720:::benchmark_data_smithi179_148205_object91423:head# 2024-02-09T17:57:12.520 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e58b90c:::benchmark_data_smithi179_148205_object10430:head# 2024-02-09T17:57:12.520 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e58fe2e:::benchmark_data_smithi179_148205_object85551:head# 2024-02-09T17:57:12.521 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e5b0371:::benchmark_data_smithi179_148205_object28070:head# 2024-02-09T17:57:12.521 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e5b430c:::benchmark_data_smithi179_148205_object19121:head# 2024-02-09T17:57:12.522 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e5b8502:::benchmark_data_smithi179_148205_object34657:head# 2024-02-09T17:57:12.522 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e5bf3e9:::benchmark_data_smithi179_148205_object10166:head# 2024-02-09T17:57:12.522 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e5de9af:::benchmark_data_smithi179_148205_object49125:head# 2024-02-09T17:57:12.523 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e5ed86b:::benchmark_data_smithi179_148205_object37863:head# 2024-02-09T17:57:12.523 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e603a55:::benchmark_data_smithi179_148205_object4460:head# 2024-02-09T17:57:12.524 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e60a91d:::benchmark_data_smithi179_148205_object38233:head# 2024-02-09T17:57:12.524 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e615f63:::benchmark_data_smithi179_148205_object37984:head# 2024-02-09T17:57:12.525 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e619d8f:::benchmark_data_smithi179_148205_object10247:head# 2024-02-09T17:57:12.525 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e62a416:::benchmark_data_smithi179_148205_object94202:head# 2024-02-09T17:57:12.526 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e62cfd9:::benchmark_data_smithi179_148205_object38724:head# 2024-02-09T17:57:12.526 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e63318e:::benchmark_data_smithi179_148205_object72534:head# 2024-02-09T17:57:12.526 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e633ea6:::benchmark_data_smithi179_148205_object13074:head# 2024-02-09T17:57:12.527 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e64ce8a:::benchmark_data_smithi179_148205_object53999:head# 2024-02-09T17:57:12.527 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e6538c1:::benchmark_data_smithi179_148205_object57147:head# 2024-02-09T17:57:12.528 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e65e75e:::benchmark_data_smithi179_148205_object89845:head# 2024-02-09T17:57:12.529 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e679874:::benchmark_data_smithi179_148205_object20503:head# 2024-02-09T17:57:12.529 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e6906e2:::benchmark_data_smithi179_148205_object76153:head# 2024-02-09T17:57:12.529 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e6913a4:::benchmark_data_smithi179_148205_object80289:head# 2024-02-09T17:57:12.530 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e6984fd:::benchmark_data_smithi179_148205_object39913:head# 2024-02-09T17:57:12.530 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e6a5e99:::benchmark_data_smithi179_148205_object37813:head# 2024-02-09T17:57:12.531 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e6ac0e5:::benchmark_data_smithi179_148205_object31091:head# 2024-02-09T17:57:12.531 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e6b7fe2:::benchmark_data_smithi179_148205_object66501:head# 2024-02-09T17:57:12.532 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e6bb5ad:::benchmark_data_smithi179_148205_object83240:head# 2024-02-09T17:57:12.532 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e6c993e:::benchmark_data_smithi179_148205_object31286:head# 2024-02-09T17:57:12.532 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e6d73b2:::benchmark_data_smithi179_148205_object47429:head# 2024-02-09T17:57:12.533 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e6e4f9b:::benchmark_data_smithi179_148205_object102748:head# 2024-02-09T17:57:12.533 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e6e6479:::benchmark_data_smithi179_148205_object6206:head# 2024-02-09T17:57:12.533 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e70480b:::benchmark_data_smithi179_148205_object60584:head# 2024-02-09T17:57:12.534 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e71f4bb:::benchmark_data_smithi179_148205_object103314:head# 2024-02-09T17:57:12.534 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e7225fe:::benchmark_data_smithi179_148205_object48799:head# 2024-02-09T17:57:12.535 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e72fb92:::benchmark_data_smithi179_148205_object87506:head# 2024-02-09T17:57:12.535 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e72fe62:::benchmark_data_smithi179_148205_object43165:head# 2024-02-09T17:57:12.537 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e732cf4:::benchmark_data_smithi179_148205_object60619:head# 2024-02-09T17:57:12.538 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e73830d:::benchmark_data_smithi179_148205_object21199:head# 2024-02-09T17:57:12.538 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e73f1a1:::benchmark_data_smithi179_148205_object56761:head# 2024-02-09T17:57:12.538 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e74b569:::benchmark_data_smithi179_148205_object41088:head# 2024-02-09T17:57:12.538 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e7579f1:::benchmark_data_smithi179_148205_object82085:head# 2024-02-09T17:57:12.540 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e769199:::benchmark_data_smithi179_148205_object17134:head# 2024-02-09T17:57:12.540 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e779259:::benchmark_data_smithi179_148205_object27448:head# 2024-02-09T17:57:12.540 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e788894:::benchmark_data_smithi179_148205_object105242:head# 2024-02-09T17:57:12.540 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e79ee31:::benchmark_data_smithi179_148205_object16819:head# 2024-02-09T17:57:12.540 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e7a304b:::benchmark_data_smithi179_148205_object18381:head# 2024-02-09T17:57:12.540 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e7a8eee:::benchmark_data_smithi179_148205_object106861:head# 2024-02-09T17:57:12.541 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e7aaa5c:::benchmark_data_smithi179_148205_object87333:head# 2024-02-09T17:57:12.541 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e7cc635:::benchmark_data_smithi179_148205_object84767:head# 2024-02-09T17:57:12.541 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e7d68cb:::benchmark_data_smithi179_148205_object53163:head# 2024-02-09T17:57:12.543 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e7df707:::benchmark_data_smithi179_148205_object95777:head# 2024-02-09T17:57:12.543 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e7ecb6c:::benchmark_data_smithi179_148205_object15126:head# 2024-02-09T17:57:12.543 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e7fb9d4:::benchmark_data_smithi179_148205_object33831:head# 2024-02-09T17:57:12.545 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e7fdf68:::benchmark_data_smithi179_148205_object41830:head# 2024-02-09T17:57:12.546 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e801cc8:::benchmark_data_smithi179_148205_object12079:head# 2024-02-09T17:57:12.547 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e803ae3:::benchmark_data_smithi179_148205_object24142:head# 2024-02-09T17:57:12.547 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e805c79:::benchmark_data_smithi179_148205_object33907:head# 2024-02-09T17:57:12.550 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e80e83c:::benchmark_data_smithi179_148205_object95764:head# 2024-02-09T17:57:12.550 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e817a86:::benchmark_data_smithi179_148205_object78105:head# 2024-02-09T17:57:12.551 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e8421da:::benchmark_data_smithi179_148205_object84049:head# 2024-02-09T17:57:12.551 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e844e13:::benchmark_data_smithi179_148205_object72179:head# 2024-02-09T17:57:12.552 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e866510:::benchmark_data_smithi179_148205_object74407:head# 2024-02-09T17:57:12.552 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e86b199:::benchmark_data_smithi179_148205_object18124:head# 2024-02-09T17:57:12.552 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e880a64:::benchmark_data_smithi179_148205_object77360:head# 2024-02-09T17:57:12.553 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e880d61:::benchmark_data_smithi179_148205_object48233:head# 2024-02-09T17:57:12.553 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e884298:::benchmark_data_smithi179_148205_object44076:head# 2024-02-09T17:57:12.554 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e894dfc:::benchmark_data_smithi179_148205_object19142:head# 2024-02-09T17:57:12.554 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e8a0548:::benchmark_data_smithi179_148205_object102813:head# 2024-02-09T17:57:12.555 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e8a13bb:::benchmark_data_smithi179_148205_object28001:head# 2024-02-09T17:57:12.555 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e8b62e1:::benchmark_data_smithi179_148205_object107353:head# 2024-02-09T17:57:12.556 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e8b91f1:::benchmark_data_smithi179_148205_object93131:head# 2024-02-09T17:57:12.556 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e8bb19d:::benchmark_data_smithi179_148205_object58047:head# 2024-02-09T17:57:12.557 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e8ce784:::benchmark_data_smithi179_148205_object79538:head# 2024-02-09T17:57:12.557 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e8db18f:::benchmark_data_smithi179_148205_object79707:head# 2024-02-09T17:57:12.558 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e8db5fd:::benchmark_data_smithi179_148205_object63100:head# 2024-02-09T17:57:12.562 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e8dcaff:::benchmark_data_smithi179_148205_object46938:head# 2024-02-09T17:57:12.562 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e8df989:::benchmark_data_smithi179_148205_object19886:head# 2024-02-09T17:57:12.562 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e8e0452:::benchmark_data_smithi179_148205_object6482:head# 2024-02-09T17:57:12.562 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e8e1a4b:::benchmark_data_smithi179_148205_object55858:head# 2024-02-09T17:57:12.562 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e8e52ab:::benchmark_data_smithi179_148205_object64259:head# 2024-02-09T17:57:12.563 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e8f7990:::benchmark_data_smithi179_148205_object24250:head# 2024-02-09T17:57:12.563 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e8f8730:::benchmark_data_smithi179_148205_object41132:head# 2024-02-09T17:57:12.563 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e8ff804:::benchmark_data_smithi179_148205_object70999:head# 2024-02-09T17:57:12.563 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e8ff8db:::benchmark_data_smithi179_148205_object74474:head# 2024-02-09T17:57:12.564 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e908935:::benchmark_data_smithi179_148205_object43210:head# 2024-02-09T17:57:12.564 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e90e2c1:::benchmark_data_smithi179_148205_object103935:head# 2024-02-09T17:57:12.564 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e91288d:::benchmark_data_smithi179_148205_object90477:head# 2024-02-09T17:57:12.568 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e91b887:::benchmark_data_smithi179_148205_object107445:head# 2024-02-09T17:57:12.569 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e9238ad:::benchmark_data_smithi179_148205_object44835:head# 2024-02-09T17:57:12.569 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e934e48:::benchmark_data_smithi179_148205_object57850:head# 2024-02-09T17:57:12.569 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e9362a3:::benchmark_data_smithi179_148205_object83949:head# 2024-02-09T17:57:12.569 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e937328:::benchmark_data_smithi179_148205_object10171:head# 2024-02-09T17:57:12.569 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e949f6b:::benchmark_data_smithi179_148205_object72793:head# 2024-02-09T17:57:12.569 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e952eff:::benchmark_data_smithi179_148205_object32046:head# 2024-02-09T17:57:12.569 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e9550af:::benchmark_data_smithi179_148205_object23541:head# 2024-02-09T17:57:12.569 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e967859:::benchmark_data_smithi179_148205_object102835:head# 2024-02-09T17:57:12.569 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e9689c1:::benchmark_data_smithi179_148205_object15109:head# 2024-02-09T17:57:12.570 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e96c6a3:::benchmark_data_smithi179_148205_object75076:head# 2024-02-09T17:57:12.570 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e96d9f4:::benchmark_data_smithi179_148205_object105885:head# 2024-02-09T17:57:12.571 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e9787df:::benchmark_data_smithi179_148205_object100572:head# 2024-02-09T17:57:12.571 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e97d509:::benchmark_data_smithi179_148205_object70175:head# 2024-02-09T17:57:12.571 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e9820a4:::benchmark_data_smithi179_148205_object108576:head# 2024-02-09T17:57:12.572 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e99266b:::benchmark_data_smithi179_148205_object85315:head# 2024-02-09T17:57:12.572 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e9994eb:::benchmark_data_smithi179_148205_object41766:head# 2024-02-09T17:57:12.575 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e99aca7:::benchmark_data_smithi179_148205_object94423:head# 2024-02-09T17:57:12.575 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e9b6295:::benchmark_data_smithi179_148205_object94617:head# 2024-02-09T17:57:12.575 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e9b8725:::benchmark_data_smithi179_148205_object2129:head# 2024-02-09T17:57:12.576 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e9be4c9:::benchmark_data_smithi179_148205_object78847:head# 2024-02-09T17:57:12.576 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e9c0eff:::benchmark_data_smithi179_148205_object16455:head# 2024-02-09T17:57:12.577 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e9c5e49:::benchmark_data_smithi179_148205_object64903:head# 2024-02-09T17:57:12.577 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e9c907e:::benchmark_data_smithi179_148205_object87612:head# 2024-02-09T17:57:12.578 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e9e7c3f:::benchmark_data_smithi179_148205_object103561:head# 2024-02-09T17:57:12.578 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e9e9772:::benchmark_data_smithi179_148205_object73646:head# 2024-02-09T17:57:12.578 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e9eebf6:::benchmark_data_smithi179_148205_object594:head# 2024-02-09T17:57:12.579 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e9f0b85:::benchmark_data_smithi179_148205_object18519:head# 2024-02-09T17:57:12.579 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1e9f3347:::benchmark_data_smithi179_148205_object70991:head# 2024-02-09T17:57:12.580 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ea01446:::benchmark_data_smithi179_148205_object61058:head# 2024-02-09T17:57:12.580 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ea04e5d:::benchmark_data_smithi179_148205_object5235:head# 2024-02-09T17:57:12.580 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ea10762:::benchmark_data_smithi179_148205_object84480:head# 2024-02-09T17:57:12.581 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ea1811d:::benchmark_data_smithi179_148205_object67239:head# 2024-02-09T17:57:12.582 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ea1a148:::benchmark_data_smithi179_148205_object17541:head# 2024-02-09T17:57:12.582 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ea1f2ff:::benchmark_data_smithi179_148205_object20348:head# 2024-02-09T17:57:12.582 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ea256f4:::benchmark_data_smithi179_148205_object59419:head# 2024-02-09T17:57:12.582 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ea2bb23:::benchmark_data_smithi179_148205_object99990:head# 2024-02-09T17:57:12.583 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ea3359c:::benchmark_data_smithi179_148205_object95890:head# 2024-02-09T17:57:12.583 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ea37551:::benchmark_data_smithi179_148205_object88074:head# 2024-02-09T17:57:12.583 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ea4112d:::benchmark_data_smithi179_148205_object69644:head# 2024-02-09T17:57:12.584 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ea41a1f:::benchmark_data_smithi179_148205_object71724:head# 2024-02-09T17:57:12.584 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ea4895d:::benchmark_data_smithi179_148205_object38304:head# 2024-02-09T17:57:12.585 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ea49576:::benchmark_data_smithi179_148205_object65042:head# 2024-02-09T17:57:12.589 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ea4bd51:::benchmark_data_smithi179_148205_object51472:head# 2024-02-09T17:57:12.589 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ea6fb3a:::benchmark_data_smithi179_148205_object48969:head# 2024-02-09T17:57:12.589 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ea7391b:::benchmark_data_smithi179_148205_object30659:head# 2024-02-09T17:57:12.589 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ea78feb:::benchmark_data_smithi179_148205_object89706:head# 2024-02-09T17:57:12.589 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ea7c4c9:::benchmark_data_smithi179_148205_object72461:head# 2024-02-09T17:57:12.590 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ea8d46e:::benchmark_data_smithi179_148205_object1655:head# 2024-02-09T17:57:12.590 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ea94494:::benchmark_data_smithi179_148205_object88797:head# 2024-02-09T17:57:12.590 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ea9abf9:::benchmark_data_smithi179_148205_object33279:head# 2024-02-09T17:57:12.590 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ea9d438:::benchmark_data_smithi179_148205_object32801:head# 2024-02-09T17:57:12.590 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1eabff0e:::benchmark_data_smithi179_148205_object63437:head# 2024-02-09T17:57:12.590 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1eac1663:::benchmark_data_smithi179_148205_object46449:head# 2024-02-09T17:57:12.590 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1eac5860:::benchmark_data_smithi179_148205_object108198:head# 2024-02-09T17:57:12.591 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1eade22f:::benchmark_data_smithi179_148205_object8600:head# 2024-02-09T17:57:12.591 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1eae0238:::benchmark_data_smithi179_148205_object96162:head# 2024-02-09T17:57:12.591 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1eae40f3:::benchmark_data_smithi179_148205_object65153:head# 2024-02-09T17:57:12.592 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1eaea6ce:::benchmark_data_smithi179_148205_object95331:head# 2024-02-09T17:57:12.592 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1eaebca0:::benchmark_data_smithi179_148205_object75287:head# 2024-02-09T17:57:12.592 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1eb0081f:::benchmark_data_smithi179_148205_object76345:head# 2024-02-09T17:57:12.594 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1eb08155:::benchmark_data_smithi179_148205_object66590:head# 2024-02-09T17:57:12.594 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1eb10a0c:::benchmark_data_smithi179_148205_object45481:head# 2024-02-09T17:57:12.598 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1eb1e1c3:::benchmark_data_smithi179_148205_object97396:head# 2024-02-09T17:57:12.598 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1eb208f9:::benchmark_data_smithi179_148205_object94224:head# 2024-02-09T17:57:12.598 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1eb213bf:::benchmark_data_smithi179_148205_object58758:head# 2024-02-09T17:57:12.598 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1eb3f223:::benchmark_data_smithi179_148205_object105374:head# 2024-02-09T17:57:12.598 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1eb3f88f:::benchmark_data_smithi179_148205_object21833:head# 2024-02-09T17:57:12.598 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1eb4a18c:::benchmark_data_smithi179_148205_object40504:head# 2024-02-09T17:57:12.598 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1eb4b289:::benchmark_data_smithi179_148205_object1391:head# 2024-02-09T17:57:12.598 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1eb4d4ac:::benchmark_data_smithi179_148205_object10875:head# 2024-02-09T17:57:12.598 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1eb58902:::benchmark_data_smithi179_148205_object24828:head# 2024-02-09T17:57:12.599 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1eb704cc:::benchmark_data_smithi179_148205_object92606:head# 2024-02-09T17:57:12.599 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1eb7fef0:::benchmark_data_smithi179_148205_object86573:head# 2024-02-09T17:57:12.599 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1eb8cbe9:::benchmark_data_smithi179_148205_object18512:head# 2024-02-09T17:57:12.599 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1eb9322c:::benchmark_data_smithi179_148205_object92283:head# 2024-02-09T17:57:12.599 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1eb9adb9:::benchmark_data_smithi179_148205_object2005:head# 2024-02-09T17:57:12.600 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1eba6760:::benchmark_data_smithi179_148205_object4114:head# 2024-02-09T17:57:12.600 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1eba7d41:::benchmark_data_smithi179_148205_object17522:head# 2024-02-09T17:57:12.601 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1eba96d1:::benchmark_data_smithi179_148205_object75939:head# 2024-02-09T17:57:12.601 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ebabd9b:::benchmark_data_smithi179_148205_object46931:head# 2024-02-09T17:57:12.602 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ebacadd:::benchmark_data_smithi179_148205_object39057:head# 2024-02-09T17:57:12.602 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ebb73e4:::benchmark_data_smithi179_148205_object25592:head# 2024-02-09T17:57:12.602 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ebb7b36:::benchmark_data_smithi179_148205_object24092:head# 2024-02-09T17:57:12.603 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ebca771:::benchmark_data_smithi179_148205_object79589:head# 2024-02-09T17:57:12.603 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ebcbb25:::benchmark_data_smithi179_148205_object15902:head# 2024-02-09T17:57:12.603 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ebda2b5:::benchmark_data_smithi179_148205_object35932:head# 2024-02-09T17:57:12.604 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ebe7745:::benchmark_data_smithi179_148205_object25197:head# 2024-02-09T17:57:12.604 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ebfc1d4:::benchmark_data_smithi179_148205_object55302:head# 2024-02-09T17:57:12.605 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ec083de:::benchmark_data_smithi179_148205_object108351:head# 2024-02-09T17:57:12.605 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ec0d941:::benchmark_data_smithi179_148205_object92256:head# 2024-02-09T17:57:12.605 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ec0e382:::benchmark_data_smithi179_148205_object31700:head# 2024-02-09T17:57:12.606 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ec1175b:::benchmark_data_smithi179_148205_object69216:head# 2024-02-09T17:57:12.606 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ec2f9ac:::benchmark_data_smithi179_148205_object19083:head# 2024-02-09T17:57:12.606 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ec34da1:::benchmark_data_smithi179_148205_object8636:head# 2024-02-09T17:57:12.612 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ec53537:::benchmark_data_smithi179_148205_object97705:head# 2024-02-09T17:57:12.612 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ec57795:::benchmark_data_smithi179_148205_object102063:head# 2024-02-09T17:57:12.612 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ec58d42:::benchmark_data_smithi179_148205_object64849:head# 2024-02-09T17:57:12.612 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ec59d60:::benchmark_data_smithi179_148205_object39993:head# 2024-02-09T17:57:12.612 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ec5da0d:::benchmark_data_smithi179_148205_object26887:head# 2024-02-09T17:57:12.612 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ec650ec:::benchmark_data_smithi179_148205_object12813:head# 2024-02-09T17:57:12.612 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ec6513c:::benchmark_data_smithi179_148205_object26985:head# 2024-02-09T17:57:12.612 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ec6cd29:::benchmark_data_smithi179_148205_object9205:head# 2024-02-09T17:57:12.612 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ec76f34:::benchmark_data_smithi179_148205_object38284:head# 2024-02-09T17:57:12.618 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ec77f10:::benchmark_data_smithi179_148205_object68501:head# 2024-02-09T17:57:12.619 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ec7f2bf:::benchmark_data_smithi179_148205_object107527:head# 2024-02-09T17:57:12.619 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ec84d64:::benchmark_data_smithi179_148205_object58429:head# 2024-02-09T17:57:12.619 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ec8a491:::benchmark_data_smithi179_148205_object18089:head# 2024-02-09T17:57:12.619 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ec8d40a:::benchmark_data_smithi179_148205_object90594:head# 2024-02-09T17:57:12.619 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ec9578a:::benchmark_data_smithi179_148205_object79130:head# 2024-02-09T17:57:12.619 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ec9d84b:::benchmark_data_smithi179_148205_object58258:head# 2024-02-09T17:57:12.619 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1eca226b:::benchmark_data_smithi179_148205_object2406:head# 2024-02-09T17:57:12.619 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1eca48f8:::benchmark_data_smithi179_148205_object56749:head# 2024-02-09T17:57:12.619 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ecad3a3:::benchmark_data_smithi179_148205_object62847:head# 2024-02-09T17:57:12.620 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ecb20f2:::benchmark_data_smithi179_148205_object24879:head# 2024-02-09T17:57:12.620 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ecb30ca:::benchmark_data_smithi179_148205_object7784:head# 2024-02-09T17:57:12.620 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ecc28a8:::benchmark_data_smithi179_148205_object14447:head# 2024-02-09T17:57:12.624 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ecd1184:::benchmark_data_smithi179_148205_object82274:head# 2024-02-09T17:57:12.625 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ecd2de1:::benchmark_data_smithi179_148205_object2035:head# 2024-02-09T17:57:12.625 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ecd74f7:::benchmark_data_smithi179_148205_object14427:head# 2024-02-09T17:57:12.625 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ece0ed8:::benchmark_data_smithi179_148205_object71905:head# 2024-02-09T17:57:12.625 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ed0289f:::benchmark_data_smithi179_148205_object42169:head# 2024-02-09T17:57:12.625 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ed17b05:::benchmark_data_smithi179_148205_object59926:head# 2024-02-09T17:57:12.625 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ed1e555:::benchmark_data_smithi179_148205_object885:head# 2024-02-09T17:57:12.625 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ed208f0:::benchmark_data_smithi179_148205_object22140:head# 2024-02-09T17:57:12.625 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ed220a4:::benchmark_data_smithi179_148205_object91634:head# 2024-02-09T17:57:12.625 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ed23710:::benchmark_data_smithi179_148205_object91995:head# 2024-02-09T17:57:12.626 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ed25b60:::benchmark_data_smithi179_148205_object56568:head# 2024-02-09T17:57:12.626 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ed2a13e:::benchmark_data_smithi179_148205_object64086:head# 2024-02-09T17:57:12.626 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ed34161:::benchmark_data_smithi179_148205_object64714:head# 2024-02-09T17:57:12.627 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ed3ca67:::benchmark_data_smithi179_148205_object21695:head# 2024-02-09T17:57:12.627 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ed42f3c:::benchmark_data_smithi179_148205_object14290:head# 2024-02-09T17:57:12.627 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ed44378:::benchmark_data_smithi179_148205_object62687:head# 2024-02-09T17:57:12.628 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ed4c105:::benchmark_data_smithi179_148205_object65063:head# 2024-02-09T17:57:12.628 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ed54ee0:::benchmark_data_smithi179_148205_object93744:head# 2024-02-09T17:57:12.628 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ed5e8fa:::benchmark_data_smithi179_148205_object59735:head# 2024-02-09T17:57:12.629 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ed64436:::benchmark_data_smithi179_148205_object32728:head# 2024-02-09T17:57:12.629 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ed763cf:::benchmark_data_smithi179_148205_object89425:head# 2024-02-09T17:57:12.630 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ed7a46f:::benchmark_data_smithi179_148205_object35373:head# 2024-02-09T17:57:12.630 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ed8308c:::benchmark_data_smithi179_148205_object85425:head# 2024-02-09T17:57:12.633 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ed935f2:::benchmark_data_smithi179_148205_object81117:head# 2024-02-09T17:57:12.633 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1eda17f2:::benchmark_data_smithi179_148205_object91899:head# 2024-02-09T17:57:12.633 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1eda5c85:::benchmark_data_smithi179_148205_object53701:head# 2024-02-09T17:57:12.634 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1edaa214:::benchmark_data_smithi179_148205_object76868:head# 2024-02-09T17:57:12.634 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1edaa360:::benchmark_data_smithi179_148205_object7843:head# 2024-02-09T17:57:12.639 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1edb4379:::benchmark_data_smithi179_148205_object33586:head# 2024-02-09T17:57:12.639 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1edc51db:::benchmark_data_smithi179_148205_object83904:head# 2024-02-09T17:57:12.639 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1edd0044:::benchmark_data_smithi179_148205_object14344:head# 2024-02-09T17:57:12.640 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1edd374d:::benchmark_data_smithi179_148205_object108138:head# 2024-02-09T17:57:12.640 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1edd7ec5:::benchmark_data_smithi179_148205_object65576:head# 2024-02-09T17:57:12.640 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1edda94f:::benchmark_data_smithi179_148205_object9275:head# 2024-02-09T17:57:12.641 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1edeb23b:::benchmark_data_smithi179_148205_object89910:head# 2024-02-09T17:57:12.646 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ee04639:::benchmark_data_smithi179_148205_object102460:head# 2024-02-09T17:57:12.646 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ee0a28a:::benchmark_data_smithi179_148205_object30844:head# 2024-02-09T17:57:12.646 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ee1d4df:::benchmark_data_smithi179_148205_object29466:head# 2024-02-09T17:57:12.646 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ee2676e:::benchmark_data_smithi179_148205_object13053:head# 2024-02-09T17:57:12.647 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ee294d8:::benchmark_data_smithi179_148205_object71320:head# 2024-02-09T17:57:12.647 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ee30c38:::benchmark_data_smithi179_148205_object40765:head# 2024-02-09T17:57:12.647 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ee35d6d:::benchmark_data_smithi179_148205_object1491:head# 2024-02-09T17:57:12.647 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ee381cf:::benchmark_data_smithi179_148205_object19850:head# 2024-02-09T17:57:12.647 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ee3f399:::benchmark_data_smithi179_148205_object24608:head# 2024-02-09T17:57:12.647 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ee61857:::benchmark_data_smithi179_148205_object4500:head# 2024-02-09T17:57:12.647 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ee6364d:::benchmark_data_smithi179_148205_object100225:head# 2024-02-09T17:57:12.650 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ee657fa:::benchmark_data_smithi179_148205_object52892:head# 2024-02-09T17:57:12.650 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ee6f9cc:::benchmark_data_smithi179_148205_object80659:head# 2024-02-09T17:57:12.650 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ee97e30:::benchmark_data_smithi179_148205_object90793:head# 2024-02-09T17:57:12.650 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ee994c4:::benchmark_data_smithi179_148205_object106995:head# 2024-02-09T17:57:12.650 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1eeaa37c:::benchmark_data_smithi179_148205_object35313:head# 2024-02-09T17:57:12.650 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1eebaef5:::benchmark_data_smithi179_148205_object41448:head# 2024-02-09T17:57:12.650 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1eebb5ec:::benchmark_data_smithi179_148205_object66293:head# 2024-02-09T17:57:12.651 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1eebd916:::benchmark_data_smithi179_148205_object82462:head# 2024-02-09T17:57:12.651 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1eed917c:::benchmark_data_smithi179_148205_object45959:head# 2024-02-09T17:57:12.674 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1eed960a:::benchmark_data_smithi179_148205_object36718:head# 2024-02-09T17:57:12.674 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1eee6b6e:::benchmark_data_smithi179_148205_object62021:head# 2024-02-09T17:57:12.674 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1eeeda66:::benchmark_data_smithi179_148205_object70935:head# 2024-02-09T17:57:12.674 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1eeee6d5:::benchmark_data_smithi179_148205_object42366:head# 2024-02-09T17:57:12.674 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1eef6aed:::benchmark_data_smithi179_148205_object8561:head# 2024-02-09T17:57:12.674 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1eefd676:::benchmark_data_smithi179_148205_object68972:head# 2024-02-09T17:57:12.674 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ef0154b:::benchmark_data_smithi179_148205_object20434:head# 2024-02-09T17:57:12.674 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ef1b55f:::benchmark_data_smithi179_148205_object105159:head# 2024-02-09T17:57:12.674 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ef329b0:::benchmark_data_smithi179_148205_object56560:head# 2024-02-09T17:57:12.674 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ef3503a:::benchmark_data_smithi179_148205_object2230:head# 2024-02-09T17:57:12.674 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ef382f7:::benchmark_data_smithi179_148205_object82239:head# 2024-02-09T17:57:12.674 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ef40536:::benchmark_data_smithi179_148205_object105160:head# 2024-02-09T17:57:12.674 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ef42ce3:::benchmark_data_smithi179_148205_object103915:head# 2024-02-09T17:57:12.674 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ef5a3f9:::benchmark_data_smithi179_148205_object94547:head# 2024-02-09T17:57:12.675 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ef6ebc8:::benchmark_data_smithi179_148205_object25626:head# 2024-02-09T17:57:12.675 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ef78ecf:::benchmark_data_smithi179_148205_object50736:head# 2024-02-09T17:57:12.675 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ef7dd7e:::benchmark_data_smithi179_148205_object57072:head# 2024-02-09T17:57:12.680 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ef95172:::benchmark_data_smithi179_148205_object85021:head# 2024-02-09T17:57:12.680 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ef98bee:::benchmark_data_smithi179_148205_object32299:head# 2024-02-09T17:57:12.681 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ef9bd7b:::benchmark_data_smithi179_148205_object99183:head# 2024-02-09T17:57:12.681 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1efa12d3:::benchmark_data_smithi179_148205_object77727:head# 2024-02-09T17:57:12.681 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1efb788d:::benchmark_data_smithi179_148205_object26700:head# 2024-02-09T17:57:12.684 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1efb9d67:::benchmark_data_smithi179_148205_object81477:head# 2024-02-09T17:57:12.684 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1efbea54:::benchmark_data_smithi179_148205_object69976:head# 2024-02-09T17:57:12.684 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1efc6e94:::benchmark_data_smithi179_148205_object103944:head# 2024-02-09T17:57:12.685 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1efd2873:::benchmark_data_smithi179_148205_object43283:head# 2024-02-09T17:57:12.685 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1efd4581:::benchmark_data_smithi179_148205_object30830:head# 2024-02-09T17:57:12.685 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1efe63c0:::benchmark_data_smithi179_148205_object18604:head# 2024-02-09T17:57:12.686 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1efe6ee8:::benchmark_data_smithi179_148205_object13792:head# 2024-02-09T17:57:12.687 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1efee0d4:::benchmark_data_smithi179_148205_object18211:head# 2024-02-09T17:57:12.687 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1effe678:::benchmark_data_smithi179_148205_object103441:head# 2024-02-09T17:57:12.687 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f0006ba:::benchmark_data_smithi179_148205_object28642:head# 2024-02-09T17:57:12.688 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f008812:::benchmark_data_smithi179_148205_object32769:head# 2024-02-09T17:57:12.689 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f008f77:::benchmark_data_smithi179_148205_object44439:head# 2024-02-09T17:57:12.689 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f00c5c9:::benchmark_data_smithi179_148205_object25070:head# 2024-02-09T17:57:12.690 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f011fcc:::benchmark_data_smithi179_148205_object102351:head# 2024-02-09T17:57:12.690 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f012992:::benchmark_data_smithi179_148205_object63237:head# 2024-02-09T17:57:12.690 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f01aa84:::benchmark_data_smithi179_148205_object22455:head# 2024-02-09T17:57:12.691 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f01ffad:::benchmark_data_smithi179_148205_object7341:head# 2024-02-09T17:57:12.691 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f02bea7:::benchmark_data_smithi179_148205_object2697:head# 2024-02-09T17:57:12.695 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f03104b:::benchmark_data_smithi179_148205_object22642:head# 2024-02-09T17:57:12.695 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f031f25:::benchmark_data_smithi179_148205_object11282:head# 2024-02-09T17:57:12.695 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f039c8c:::benchmark_data_smithi179_148205_object30149:head# 2024-02-09T17:57:12.696 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f03f7c9:::benchmark_data_smithi179_148205_object25176:head# 2024-02-09T17:57:12.696 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f03fad1:::benchmark_data_smithi179_148205_object57140:head# 2024-02-09T17:57:12.697 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f045315:::benchmark_data_smithi179_148205_object67460:head# 2024-02-09T17:57:12.697 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f04d4d2:::benchmark_data_smithi179_148205_object25025:head# 2024-02-09T17:57:12.698 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f053ab7:::benchmark_data_smithi179_148205_object94311:head# 2024-02-09T17:57:12.698 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f05a515:::benchmark_data_smithi179_148205_object69624:head# 2024-02-09T17:57:12.698 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f05d733:::benchmark_data_smithi179_148205_object97483:head# 2024-02-09T17:57:12.699 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f06647a:::benchmark_data_smithi179_148205_object55305:head# 2024-02-09T17:57:12.699 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f070453:::benchmark_data_smithi179_148205_object96947:head# 2024-02-09T17:57:12.699 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f0738e7:::benchmark_data_smithi179_148205_object22379:head# 2024-02-09T17:57:12.700 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f078188:::benchmark_data_smithi179_148205_object44321:head# 2024-02-09T17:57:12.700 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f0804fb:::benchmark_data_smithi179_148205_object101576:head# 2024-02-09T17:57:12.701 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f0805d9:::benchmark_data_smithi179_148205_object69831:head# 2024-02-09T17:57:12.701 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f09bf0a:::benchmark_data_smithi179_148205_object87730:head# 2024-02-09T17:57:12.702 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f09cdd7:::benchmark_data_smithi179_148205_object57916:head# 2024-02-09T17:57:12.702 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f0a37e3:::benchmark_data_smithi179_148205_object10311:head# 2024-02-09T17:57:12.702 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f0a4cf0:::benchmark_data_smithi179_148205_object7305:head# 2024-02-09T17:57:12.705 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f0a4cfc:::benchmark_data_smithi179_148205_object90949:head# 2024-02-09T17:57:12.705 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f0a8004:::benchmark_data_smithi179_148205_object58844:head# 2024-02-09T17:57:12.706 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f0aa79a:::benchmark_data_smithi179_148205_object82182:head# 2024-02-09T17:57:12.706 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f0ab95c:::benchmark_data_smithi179_148205_object77325:head# 2024-02-09T17:57:12.707 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f0b56fc:::benchmark_data_smithi179_148205_object21672:head# 2024-02-09T17:57:12.707 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f0b7b68:::benchmark_data_smithi179_148205_object40096:head# 2024-02-09T17:57:12.708 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f0c301c:::benchmark_data_smithi179_148205_object62436:head# 2024-02-09T17:57:12.708 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f0cf0ca:::benchmark_data_smithi179_148205_object87713:head# 2024-02-09T17:57:12.708 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f0d61f9:::benchmark_data_smithi179_148205_object82568:head# 2024-02-09T17:57:12.709 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f0d8570:::benchmark_data_smithi179_148205_object69663:head# 2024-02-09T17:57:12.709 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f0d8833:::benchmark_data_smithi179_148205_object2827:head# 2024-02-09T17:57:12.712 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f0d894c:::benchmark_data_smithi179_148205_object50263:head# 2024-02-09T17:57:12.712 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f0e143d:::benchmark_data_smithi179_148205_object87991:head# 2024-02-09T17:57:12.712 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f0e2bfc:::benchmark_data_smithi179_148205_object22622:head# 2024-02-09T17:57:12.713 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f0eb140:::benchmark_data_smithi179_148205_object66068:head# 2024-02-09T17:57:12.713 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f0fc4a1:::benchmark_data_smithi179_148205_object57207:head# 2024-02-09T17:57:12.715 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f0fd53b:::benchmark_data_smithi179_148205_object83915:head# 2024-02-09T17:57:12.716 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f10a9c3:::benchmark_data_smithi179_148205_object36397:head# 2024-02-09T17:57:12.719 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f118211:::benchmark_data_smithi179_148205_object47486:head# 2024-02-09T17:57:12.719 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f125c07:::benchmark_data_smithi179_148205_object109220:head# 2024-02-09T17:57:12.719 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f13811d:::benchmark_data_smithi179_148205_object34486:head# 2024-02-09T17:57:12.720 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f1476c2:::benchmark_data_smithi179_148205_object107940:head# 2024-02-09T17:57:12.720 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f167636:::benchmark_data_smithi179_148205_object98552:head# 2024-02-09T17:57:12.720 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f16eb2b:::benchmark_data_smithi179_148205_object32268:head# 2024-02-09T17:57:12.720 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f16f43a:::benchmark_data_smithi179_148205_object97350:head# 2024-02-09T17:57:12.721 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f17093a:::benchmark_data_smithi179_148205_object44448:head# 2024-02-09T17:57:12.721 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f181d15:::benchmark_data_smithi179_148205_object101877:head# 2024-02-09T17:57:12.722 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f1919ba:::benchmark_data_smithi179_148205_object107863:head# 2024-02-09T17:57:12.722 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f1c2c08:::benchmark_data_smithi179_148205_object26006:head# 2024-02-09T17:57:12.726 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f1d5fc6:::benchmark_data_smithi179_148205_object87857:head# 2024-02-09T17:57:12.726 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f1e458a:::benchmark_data_smithi179_148205_object101646:head# 2024-02-09T17:57:12.726 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f1e7887:::benchmark_data_smithi179_148205_object78178:head# 2024-02-09T17:57:12.727 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f1ee9f6:::benchmark_data_smithi179_148205_object105684:head# 2024-02-09T17:57:12.727 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f20b3ed:::benchmark_data_smithi179_148205_object99820:head# 2024-02-09T17:57:12.727 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f21fbe4:::benchmark_data_smithi179_148205_object11318:head# 2024-02-09T17:57:12.728 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f229fa4:::benchmark_data_smithi179_148205_object53941:head# 2024-02-09T17:57:12.728 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f230b61:::benchmark_data_smithi179_148205_object59421:head# 2024-02-09T17:57:12.728 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f232158:::benchmark_data_smithi179_148205_object81076:head# 2024-02-09T17:57:12.730 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f239762:::benchmark_data_smithi179_148205_object100267:head# 2024-02-09T17:57:12.730 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f242739:::benchmark_data_smithi179_148205_object61449:head# 2024-02-09T17:57:12.730 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f257b36:::benchmark_data_smithi179_148205_object66778:head# 2024-02-09T17:57:12.731 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f2627d0:::benchmark_data_smithi179_148205_object103735:head# 2024-02-09T17:57:12.731 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f26f8f8:::benchmark_data_smithi179_148205_object9694:head# 2024-02-09T17:57:12.731 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f27c63c:::benchmark_data_smithi179_148205_object88973:head# 2024-02-09T17:57:12.735 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f27d139:::benchmark_data_smithi179_148205_object54397:head# 2024-02-09T17:57:12.735 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f2874ab:::benchmark_data_smithi179_148205_object36335:head# 2024-02-09T17:57:12.736 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f28a87f:::benchmark_data_smithi179_148205_object85056:head# 2024-02-09T17:57:12.736 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f291cbb:::benchmark_data_smithi179_148205_object49430:head# 2024-02-09T17:57:12.746 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f292e3c:::benchmark_data_smithi179_148205_object108398:head# 2024-02-09T17:57:12.746 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f29cf09:::benchmark_data_smithi179_148205_object57791:head# 2024-02-09T17:57:12.746 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f29f06b:::benchmark_data_smithi179_148205_object34507:head# 2024-02-09T17:57:12.746 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f2a38ed:::benchmark_data_smithi179_148205_object64647:head# 2024-02-09T17:57:12.746 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f2ab987:::benchmark_data_smithi179_148205_object81897:head# 2024-02-09T17:57:12.746 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f2ef862:::benchmark_data_smithi179_148205_object12383:head# 2024-02-09T17:57:12.746 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f2f47d1:::benchmark_data_smithi179_148205_object67495:head# 2024-02-09T17:57:12.746 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f2f7de0:::benchmark_data_smithi179_148205_object104677:head# 2024-02-09T17:57:12.750 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f2fb749:::benchmark_data_smithi179_148205_object78282:head# 2024-02-09T17:57:12.750 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f2fba62:::benchmark_data_smithi179_148205_object11254:head# 2024-02-09T17:57:12.750 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f3033b3:::benchmark_data_smithi179_148205_object101140:head# 2024-02-09T17:57:12.750 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f315ea7:::benchmark_data_smithi179_148205_object11981:head# 2024-02-09T17:57:12.750 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f31a190:::benchmark_data_smithi179_148205_object73047:head# 2024-02-09T17:57:12.751 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f31bda2:::benchmark_data_smithi179_148205_object89055:head# 2024-02-09T17:57:12.751 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f31f8c1:::benchmark_data_smithi179_148205_object72167:head# 2024-02-09T17:57:12.751 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f3286f9:::benchmark_data_smithi179_148205_object94640:head# 2024-02-09T17:57:12.751 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f32fec3:::benchmark_data_smithi179_148205_object103284:head# 2024-02-09T17:57:12.751 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f33dfb9:::benchmark_data_smithi179_148205_object74723:head# 2024-02-09T17:57:12.751 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f35076b:::benchmark_data_smithi179_148205_object95026:head# 2024-02-09T17:57:12.751 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f35a6c5:::benchmark_data_smithi179_148205_object23213:head# 2024-02-09T17:57:12.752 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f35b80c:::benchmark_data_smithi179_148205_object59053:head# 2024-02-09T17:57:12.752 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f35ed93:::benchmark_data_smithi179_148205_object63404:head# 2024-02-09T17:57:12.753 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f3673b8:::benchmark_data_smithi179_148205_object59887:head# 2024-02-09T17:57:12.753 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f3705c4:::benchmark_data_smithi179_148205_object83820:head# 2024-02-09T17:57:12.754 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f3734f1:::benchmark_data_smithi179_148205_object98770:head# 2024-02-09T17:57:12.754 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f37b50b:::benchmark_data_smithi179_148205_object35452:head# 2024-02-09T17:57:12.755 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f37b6d5:::benchmark_data_smithi179_148205_object69862:head# 2024-02-09T17:57:12.755 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f3add9b:::benchmark_data_smithi179_148205_object51980:head# 2024-02-09T17:57:12.756 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f3c3ad5:::benchmark_data_smithi179_148205_object81915:head# 2024-02-09T17:57:12.756 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f3cbbe2:::benchmark_data_smithi179_148205_object40527:head# 2024-02-09T17:57:12.756 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f3d0695:::benchmark_data_smithi179_148205_object66411:head# 2024-02-09T17:57:12.757 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f3d2c40:::benchmark_data_smithi179_148205_object102522:head# 2024-02-09T17:57:12.757 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f3d57b3:::benchmark_data_smithi179_148205_object79927:head# 2024-02-09T17:57:12.758 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f3d5e03:::benchmark_data_smithi179_148205_object27112:head# 2024-02-09T17:57:12.758 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f3e2ac0:::benchmark_data_smithi179_148205_object9501:head# 2024-02-09T17:57:12.759 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f3e3633:::benchmark_data_smithi179_148205_object104643:head# 2024-02-09T17:57:12.759 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f3e6049:::benchmark_data_smithi179_148205_object93390:head# 2024-02-09T17:57:12.761 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f407bab:::benchmark_data_smithi179_148205_object106546:head# 2024-02-09T17:57:12.761 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f40c9cb:::benchmark_data_smithi179_148205_object66867:head# 2024-02-09T17:57:12.761 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f40dcc9:::benchmark_data_smithi179_148205_object65406:head# 2024-02-09T17:57:12.762 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f42e116:::benchmark_data_smithi179_148205_object50026:head# 2024-02-09T17:57:12.762 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f42ecb2:::benchmark_data_smithi179_148205_object8592:head# 2024-02-09T17:57:12.763 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f4378ab:::benchmark_data_smithi179_148205_object107732:head# 2024-02-09T17:57:12.763 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f43b811:::benchmark_data_smithi179_148205_object101352:head# 2024-02-09T17:57:12.763 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f4418bb:::benchmark_data_smithi179_148205_object12263:head# 2024-02-09T17:57:12.764 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f44df47:::benchmark_data_smithi179_148205_object17922:head# 2024-02-09T17:57:12.764 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f451567:::benchmark_data_smithi179_148205_object76458:head# 2024-02-09T17:57:12.765 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f454298:::benchmark_data_smithi179_148205_object67247:head# 2024-02-09T17:57:12.765 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f4547bf:::benchmark_data_smithi179_148205_object64078:head# 2024-02-09T17:57:12.765 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f45d798:::benchmark_data_smithi179_148205_object73420:head# 2024-02-09T17:57:12.766 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f45eb89:::benchmark_data_smithi179_148205_object69169:head# 2024-02-09T17:57:12.766 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f4606db:::benchmark_data_smithi179_148205_object96192:head# 2024-02-09T17:57:12.766 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f4663ae:::benchmark_data_smithi179_148205_object53670:head# 2024-02-09T17:57:12.767 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f4672f8:::benchmark_data_smithi179_148205_object46522:head# 2024-02-09T17:57:12.767 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f46a3ac:::benchmark_data_smithi179_148205_object70394:head# 2024-02-09T17:57:12.767 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f472d63:::benchmark_data_smithi179_148205_object93530:head# 2024-02-09T17:57:12.768 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f4735f0:::benchmark_data_smithi179_148205_object74835:head# 2024-02-09T17:57:12.768 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f485040:::benchmark_data_smithi179_148205_object92777:head# 2024-02-09T17:57:12.768 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f48994e:::benchmark_data_smithi179_148205_object86635:head# 2024-02-09T17:57:12.769 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f4a232f:::benchmark_data_smithi179_148205_object81465:head# 2024-02-09T17:57:12.769 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f4a24e4:::benchmark_data_smithi179_148205_object40177:head# 2024-02-09T17:57:12.770 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f4a79bf:::benchmark_data_smithi179_148205_object56622:head# 2024-02-09T17:57:12.770 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f4a7a22:::benchmark_data_smithi179_148205_object65954:head# 2024-02-09T17:57:12.773 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f4b3500:::benchmark_data_smithi179_148205_object31828:head# 2024-02-09T17:57:12.773 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f4b36ce:::benchmark_data_smithi179_148205_object31775:head# 2024-02-09T17:57:12.773 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f4be3ce:::benchmark_data_smithi179_148205_object77514:head# 2024-02-09T17:57:12.773 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f4bf6df:::benchmark_data_smithi179_148205_object27115:head# 2024-02-09T17:57:12.773 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f4c264b:::benchmark_data_smithi179_148205_object81595:head# 2024-02-09T17:57:12.773 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f4c7c0b:::benchmark_data_smithi179_148205_object61498:head# 2024-02-09T17:57:12.774 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f4cdbed:::benchmark_data_smithi179_148205_object88700:head# 2024-02-09T17:57:12.774 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f4d0b0f:::benchmark_data_smithi179_148205_object85148:head# 2024-02-09T17:57:12.774 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f4d435a:::benchmark_data_smithi179_148205_object12384:head# 2024-02-09T17:57:12.774 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f4d9c51:::benchmark_data_smithi179_148205_object2112:head# 2024-02-09T17:57:12.775 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f4e2a72:::benchmark_data_smithi179_148205_object35594:head# 2024-02-09T17:57:12.778 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f4ebe1d:::benchmark_data_smithi179_148205_object100248:head# 2024-02-09T17:57:12.778 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f4f37cd:::benchmark_data_smithi179_148205_object109272:head# 2024-02-09T17:57:12.779 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f4f72cd:::benchmark_data_smithi179_148205_object30134:head# 2024-02-09T17:57:12.779 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f505b8f:::benchmark_data_smithi179_148205_object77612:head# 2024-02-09T17:57:12.779 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f50d820:::benchmark_data_smithi179_148205_object107963:head# 2024-02-09T17:57:12.783 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f513929:::benchmark_data_smithi179_148205_object76938:head# 2024-02-09T17:57:12.783 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f5143fc:::benchmark_data_smithi179_148205_object45039:head# 2024-02-09T17:57:12.783 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f527228:::benchmark_data_smithi179_148205_object42450:head# 2024-02-09T17:57:12.783 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f5344c3:::benchmark_data_smithi179_148205_object33801:head# 2024-02-09T17:57:12.783 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f537c70:::benchmark_data_smithi179_148205_object100331:head# 2024-02-09T17:57:12.783 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f53b57a:::benchmark_data_smithi179_148205_object21145:head# 2024-02-09T17:57:12.784 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f53efdc:::benchmark_data_smithi179_148205_object63368:head# 2024-02-09T17:57:12.784 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f540992:::benchmark_data_smithi179_148205_object88627:head# 2024-02-09T17:57:12.784 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f542764:::benchmark_data_smithi179_148205_object82164:head# 2024-02-09T17:57:12.785 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f558632:::benchmark_data_smithi179_148205_object21377:head# 2024-02-09T17:57:12.785 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f5613bf:::benchmark_data_smithi179_148205_object17951:head# 2024-02-09T17:57:12.786 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f572385:::benchmark_data_smithi179_148205_object43049:head# 2024-02-09T17:57:12.786 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f575425:::benchmark_data_smithi179_148205_object62662:head# 2024-02-09T17:57:12.786 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f578504:::benchmark_data_smithi179_148205_object28745:head# 2024-02-09T17:57:12.787 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f581f63:::benchmark_data_smithi179_148205_object69603:head# 2024-02-09T17:57:12.787 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f58a69c:::benchmark_data_smithi179_148205_object10716:head# 2024-02-09T17:57:12.787 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f58aa85:::benchmark_data_smithi179_148205_object83123:head# 2024-02-09T17:57:12.788 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f59c0b0:::benchmark_data_smithi179_148205_object94158:head# 2024-02-09T17:57:12.788 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f5a1a12:::benchmark_data_smithi179_148205_object13321:head# 2024-02-09T17:57:12.788 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f5a9932:::benchmark_data_smithi179_148205_object44992:head# 2024-02-09T17:57:12.789 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f5b3b06:::benchmark_data_smithi179_148205_object66368:head# 2024-02-09T17:57:12.789 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f5ce654:::benchmark_data_smithi179_148205_object104742:head# 2024-02-09T17:57:12.790 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f5d38dc:::benchmark_data_smithi179_148205_object66132:head# 2024-02-09T17:57:12.794 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f5d6676:::benchmark_data_smithi179_148205_object86857:head# 2024-02-09T17:57:12.794 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f5e84cc:::benchmark_data_smithi179_148205_object65752:head# 2024-02-09T17:57:12.795 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f5e89f2:::benchmark_data_smithi179_148205_object85203:head# 2024-02-09T17:57:12.795 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f5fc005:::benchmark_data_smithi179_148205_object39979:head# 2024-02-09T17:57:12.795 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f60db5f:::benchmark_data_smithi179_148205_object53085:head# 2024-02-09T17:57:12.795 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f619634:::benchmark_data_smithi179_148205_object12887:head# 2024-02-09T17:57:12.795 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f61a50c:::benchmark_data_smithi179_148205_object38163:head# 2024-02-09T17:57:12.795 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f61ba98:::benchmark_data_smithi179_148205_object103675:head# 2024-02-09T17:57:12.795 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f62c884:::benchmark_data_smithi179_148205_object105308:head# 2024-02-09T17:57:12.795 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f631c51:::benchmark_data_smithi179_148205_object103789:head# 2024-02-09T17:57:12.795 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f636d28:::benchmark_data_smithi179_148205_object88060:head# 2024-02-09T17:57:12.795 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f63ddc8:::benchmark_data_smithi179_148205_object71164:head# 2024-02-09T17:57:12.796 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f642b2d:::benchmark_data_smithi179_148205_object99230:head# 2024-02-09T17:57:12.796 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f655365:::benchmark_data_smithi179_148205_object5442:head# 2024-02-09T17:57:12.796 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f65779b:::benchmark_data_smithi179_148205_object55879:head# 2024-02-09T17:57:12.796 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f65da3c:::benchmark_data_smithi179_148205_object86464:head# 2024-02-09T17:57:12.799 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f6680c9:::benchmark_data_smithi179_148205_object11115:head# 2024-02-09T17:57:12.799 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f66b5e4:::benchmark_data_smithi179_148205_object14904:head# 2024-02-09T17:57:12.802 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f66cc79:::benchmark_data_smithi179_148205_object28173:head# 2024-02-09T17:57:12.802 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f673267:::benchmark_data_smithi179_148205_object34102:head# 2024-02-09T17:57:12.802 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f674311:::benchmark_data_smithi179_148205_object30042:head# 2024-02-09T17:57:12.802 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f6820e3:::benchmark_data_smithi179_148205_object46928:head# 2024-02-09T17:57:12.802 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f683015:::benchmark_data_smithi179_148205_object35824:head# 2024-02-09T17:57:12.802 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f68add6:::benchmark_data_smithi179_148205_object91324:head# 2024-02-09T17:57:12.803 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f6957f0:::benchmark_data_smithi179_148205_object76350:head# 2024-02-09T17:57:12.803 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f699754:::benchmark_data_smithi179_148205_object22445:head# 2024-02-09T17:57:12.803 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f69a120:::benchmark_data_smithi179_148205_object9552:head# 2024-02-09T17:57:12.803 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f6a9b2c:::benchmark_data_smithi179_148205_object19953:head# 2024-02-09T17:57:12.804 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f6b7234:::benchmark_data_smithi179_148205_object48547:head# 2024-02-09T17:57:12.804 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f6b96e0:::benchmark_data_smithi179_148205_object44649:head# 2024-02-09T17:57:12.804 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f6bca7e:::benchmark_data_smithi179_148205_object35628:head# 2024-02-09T17:57:12.805 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f6c1346:::benchmark_data_smithi179_148205_object87015:head# 2024-02-09T17:57:12.805 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f6c5f05:::benchmark_data_smithi179_148205_object70630:head# 2024-02-09T17:57:12.805 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f6ca6f2:::benchmark_data_smithi179_148205_object105197:head# 2024-02-09T17:57:12.806 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f6cd2e3:::benchmark_data_smithi179_148205_object94207:head# 2024-02-09T17:57:12.807 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f6cffb2:::benchmark_data_smithi179_148205_object56066:head# 2024-02-09T17:57:12.807 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f6d4f8e:::benchmark_data_smithi179_148205_object20824:head# 2024-02-09T17:57:12.807 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f6dec2f:::benchmark_data_smithi179_148205_object43596:head# 2024-02-09T17:57:12.807 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f6edf97:::benchmark_data_smithi179_148205_object50422:head# 2024-02-09T17:57:12.808 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f6fccee:::benchmark_data_smithi179_148205_object109764:head# 2024-02-09T17:57:12.808 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f71b3e7:::benchmark_data_smithi179_148205_object56615:head# 2024-02-09T17:57:12.809 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f71bfd7:::benchmark_data_smithi179_148205_object10687:head# 2024-02-09T17:57:12.809 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f7245ad:::benchmark_data_smithi179_148205_object39893:head# 2024-02-09T17:57:12.809 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f72e48c:::benchmark_data_smithi179_148205_object105942:head# 2024-02-09T17:57:12.810 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f7323c8:::benchmark_data_smithi179_148205_object31720:head# 2024-02-09T17:57:12.810 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f73330e:::benchmark_data_smithi179_148205_object54973:head# 2024-02-09T17:57:12.811 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f733d16:::benchmark_data_smithi179_148205_object21753:head# 2024-02-09T17:57:12.811 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f734fb8:::benchmark_data_smithi179_148205_object45518:head# 2024-02-09T17:57:12.812 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f73ba32:::benchmark_data_smithi179_148205_object36404:head# 2024-02-09T17:57:12.812 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f740aef:::benchmark_data_smithi179_148205_object22905:head# 2024-02-09T17:57:12.812 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f7499e1:::benchmark_data_smithi179_148205_object90410:head# 2024-02-09T17:57:12.812 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f74e28a:::benchmark_data_smithi179_148205_object98731:head# 2024-02-09T17:57:12.813 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f750640:::benchmark_data_smithi179_148205_object66248:head# 2024-02-09T17:57:12.814 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f754120:::benchmark_data_smithi179_148205_object17221:head# 2024-02-09T17:57:12.814 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f759719:::benchmark_data_smithi179_148205_object107040:head# 2024-02-09T17:57:12.815 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f75a902:::benchmark_data_smithi179_148205_object3206:head# 2024-02-09T17:57:12.815 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f75bf7e:::benchmark_data_smithi179_148205_object98766:head# 2024-02-09T17:57:12.819 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f762212:::benchmark_data_smithi179_148205_object90426:head# 2024-02-09T17:57:12.819 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f7683b4:::benchmark_data_smithi179_148205_object20400:head# 2024-02-09T17:57:12.819 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f787ae7:::benchmark_data_smithi179_148205_object65784:head# 2024-02-09T17:57:12.819 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f795f00:::benchmark_data_smithi179_148205_object102822:head# 2024-02-09T17:57:12.819 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f7ad434:::benchmark_data_smithi179_148205_object1784:head# 2024-02-09T17:57:12.819 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f7b8886:::benchmark_data_smithi179_148205_object68010:head# 2024-02-09T17:57:12.819 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f7c279c:::benchmark_data_smithi179_148205_object24049:head# 2024-02-09T17:57:12.819 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f7c4bd0:::benchmark_data_smithi179_148205_object100349:head# 2024-02-09T17:57:12.820 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f7cac7b:::benchmark_data_smithi179_148205_object3155:head# 2024-02-09T17:57:12.821 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f7cfc87:::benchmark_data_smithi179_148205_object42745:head# 2024-02-09T17:57:12.821 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f7d80cc:::benchmark_data_smithi179_148205_object36128:head# 2024-02-09T17:57:12.821 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f7eace2:::benchmark_data_smithi179_148205_object3443:head# 2024-02-09T17:57:12.821 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f7f0423:::benchmark_data_smithi179_148205_object47295:head# 2024-02-09T17:57:12.821 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f7f5a28:::benchmark_data_smithi179_148205_object68724:head# 2024-02-09T17:57:12.822 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f80bca6:::benchmark_data_smithi179_148205_object62931:head# 2024-02-09T17:57:12.822 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f8156f0:::benchmark_data_smithi179_148205_object40163:head# 2024-02-09T17:57:12.822 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f826e56:::benchmark_data_smithi179_148205_object65152:head# 2024-02-09T17:57:12.822 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f82a88d:::benchmark_data_smithi179_148205_object68938:head# 2024-02-09T17:57:12.823 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f833da7:::benchmark_data_smithi179_148205_object70854:head# 2024-02-09T17:57:12.823 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f834fa1:::benchmark_data_smithi179_148205_object101960:head# 2024-02-09T17:57:12.824 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f83c3ac:::benchmark_data_smithi179_148205_object75132:head# 2024-02-09T17:57:12.824 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f840596:::benchmark_data_smithi179_148205_object8083:head# 2024-02-09T17:57:12.824 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f840c34:::benchmark_data_smithi179_148205_object10962:head# 2024-02-09T17:57:12.825 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f8422c9:::benchmark_data_smithi179_148205_object43997:head# 2024-02-09T17:57:12.825 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f8442e3:::benchmark_data_smithi179_148205_object17288:head# 2024-02-09T17:57:12.825 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f84b23a:::benchmark_data_smithi179_148205_object10885:head# 2024-02-09T17:57:12.826 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f858726:::benchmark_data_smithi179_148205_object15549:head# 2024-02-09T17:57:12.826 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f868391:::benchmark_data_smithi179_148205_object62322:head# 2024-02-09T17:57:12.826 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f86c492:::benchmark_data_smithi179_148205_object103062:head# 2024-02-09T17:57:12.827 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f874fa3:::benchmark_data_smithi179_148205_object63790:head# 2024-02-09T17:57:12.827 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f878286:::benchmark_data_smithi179_148205_object9559:head# 2024-02-09T17:57:12.828 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f880c7a:::benchmark_data_smithi179_148205_object44985:head# 2024-02-09T17:57:12.828 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f884ebe:::benchmark_data_smithi179_148205_object71464:head# 2024-02-09T17:57:12.828 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f88c079:::benchmark_data_smithi179_148205_object86838:head# 2024-02-09T17:57:12.829 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f88de42:::benchmark_data_smithi179_148205_object48395:head# 2024-02-09T17:57:12.829 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f8964cb:::benchmark_data_smithi179_148205_object15874:head# 2024-02-09T17:57:12.829 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f896871:::benchmark_data_smithi179_148205_object23992:head# 2024-02-09T17:57:12.830 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f899896:::benchmark_data_smithi179_148205_object104251:head# 2024-02-09T17:57:12.830 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f8ca92b:::benchmark_data_smithi179_148205_object6688:head# 2024-02-09T17:57:12.831 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f8cd6e5:::benchmark_data_smithi179_148205_object101997:head# 2024-02-09T17:57:12.831 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f8d8135:::benchmark_data_smithi179_148205_object36110:head# 2024-02-09T17:57:12.831 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f8d98c4:::benchmark_data_smithi179_148205_object32813:head# 2024-02-09T17:57:12.832 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f8ea928:::benchmark_data_smithi179_148205_object88799:head# 2024-02-09T17:57:12.832 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f8fc0b0:::benchmark_data_smithi179_148205_object2428:head# 2024-02-09T17:57:12.832 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f90bd0b:::benchmark_data_smithi179_148205_object104053:head# 2024-02-09T17:57:12.833 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f90c7f8:::benchmark_data_smithi179_148205_object106644:head# 2024-02-09T17:57:12.833 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f928889:::benchmark_data_smithi179_148205_object72509:head# 2024-02-09T17:57:12.834 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f928d3c:::benchmark_data_smithi179_148205_object50796:head# 2024-02-09T17:57:12.834 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f93b51a:::benchmark_data_smithi179_148205_object99202:head# 2024-02-09T17:57:12.840 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f93e827:::benchmark_data_smithi179_148205_object7892:head# 2024-02-09T17:57:12.840 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f93fdf2:::benchmark_data_smithi179_148205_object72103:head# 2024-02-09T17:57:12.840 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f948a69:::benchmark_data_smithi179_148205_object66434:head# 2024-02-09T17:57:12.840 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f94a35b:::benchmark_data_smithi179_148205_object25298:head# 2024-02-09T17:57:12.841 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f94f174:::benchmark_data_smithi179_148205_object66022:head# 2024-02-09T17:57:12.841 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f950ca4:::benchmark_data_smithi179_148205_object12090:head# 2024-02-09T17:57:12.842 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f9640d9:::benchmark_data_smithi179_148205_object39082:head# 2024-02-09T17:57:12.842 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f96d095:::benchmark_data_smithi179_148205_object15103:head# 2024-02-09T17:57:12.843 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f970c99:::benchmark_data_smithi179_148205_object73652:head# 2024-02-09T17:57:12.845 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f976246:::benchmark_data_smithi179_148205_object69235:head# 2024-02-09T17:57:12.845 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f97f674:::benchmark_data_smithi179_148205_object55488:head# 2024-02-09T17:57:12.845 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f9a42a1:::benchmark_data_smithi179_148205_object26916:head# 2024-02-09T17:57:12.845 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f9a5b16:::benchmark_data_smithi179_148205_object107309:head# 2024-02-09T17:57:12.845 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f9a6acb:::benchmark_data_smithi179_148205_object45716:head# 2024-02-09T17:57:12.847 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f9c3223:::benchmark_data_smithi179_148205_object79385:head# 2024-02-09T17:57:12.848 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f9cbe43:::benchmark_data_smithi179_148205_object27361:head# 2024-02-09T17:57:12.848 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f9d03b2:::benchmark_data_smithi179_148205_object40154:head# 2024-02-09T17:57:12.848 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f9d60c8:::benchmark_data_smithi179_148205_object50579:head# 2024-02-09T17:57:12.850 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f9e501f:::benchmark_data_smithi179_148205_object34067:head# 2024-02-09T17:57:12.851 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f9ecac0:::benchmark_data_smithi179_148205_object107105:head# 2024-02-09T17:57:12.851 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f9f1cd6:::benchmark_data_smithi179_148205_object33128:head# 2024-02-09T17:57:12.851 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f9fb05e:::benchmark_data_smithi179_148205_object67846:head# 2024-02-09T17:57:12.851 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1f9ff99e:::benchmark_data_smithi179_148205_object99320:head# 2024-02-09T17:57:12.854 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fa0d4e6:::benchmark_data_smithi179_148205_object32710:head# 2024-02-09T17:57:12.855 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fa0d9e9:::benchmark_data_smithi179_148205_object102619:head# 2024-02-09T17:57:12.858 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fa0f434:::benchmark_data_smithi179_148205_object5638:head# 2024-02-09T17:57:12.858 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fa140fc:::benchmark_data_smithi179_148205_object49551:head# 2024-02-09T17:57:12.858 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fa1d157:::benchmark_data_smithi179_148205_object87500:head# 2024-02-09T17:57:12.858 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fa22a8e:::benchmark_data_smithi179_148205_object41785:head# 2024-02-09T17:57:12.858 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fa28040:::benchmark_data_smithi179_148205_object12979:head# 2024-02-09T17:57:12.859 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fa36e09:::benchmark_data_smithi179_148205_object81203:head# 2024-02-09T17:57:12.859 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fa3af85:::benchmark_data_smithi179_148205_object11691:head# 2024-02-09T17:57:12.859 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fa562b0:::benchmark_data_smithi179_148205_object50914:head# 2024-02-09T17:57:12.860 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fa5dded:::benchmark_data_smithi179_148205_object36517:head# 2024-02-09T17:57:12.860 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fa729b3:::benchmark_data_smithi179_148205_object15068:head# 2024-02-09T17:57:12.860 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fa76489:::benchmark_data_smithi179_148205_object7516:head# 2024-02-09T17:57:12.861 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fa7ba94:::benchmark_data_smithi179_148205_object91955:head# 2024-02-09T17:57:12.861 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fa8a5da:::benchmark_data_smithi179_148205_object73434:head# 2024-02-09T17:57:12.862 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fa8de60:::benchmark_data_smithi179_148205_object79177:head# 2024-02-09T17:57:12.862 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fa96a72:::benchmark_data_smithi179_148205_object101861:head# 2024-02-09T17:57:12.862 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1faa33be:::benchmark_data_smithi179_148205_object70375:head# 2024-02-09T17:57:12.863 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1faab63b:::benchmark_data_smithi179_148205_object26581:head# 2024-02-09T17:57:12.863 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fab60f7:::benchmark_data_smithi179_148205_object14613:head# 2024-02-09T17:57:12.863 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fabe7e5:::benchmark_data_smithi179_148205_object89824:head# 2024-02-09T17:57:12.864 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fabf431:::benchmark_data_smithi179_148205_object20938:head# 2024-02-09T17:57:12.864 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fac0fb1:::benchmark_data_smithi179_148205_object93807:head# 2024-02-09T17:57:12.865 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fac4be3:::benchmark_data_smithi179_148205_object37766:head# 2024-02-09T17:57:12.865 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fac5821:::benchmark_data_smithi179_148205_object15514:head# 2024-02-09T17:57:12.866 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1facc0e4:::benchmark_data_smithi179_148205_object72531:head# 2024-02-09T17:57:12.866 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fadea8b:::benchmark_data_smithi179_148205_object2512:head# 2024-02-09T17:57:12.866 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fae2fef:::benchmark_data_smithi179_148205_object55447:head# 2024-02-09T17:57:12.867 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fae83d9:::benchmark_data_smithi179_148205_object50295:head# 2024-02-09T17:57:12.867 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1faeb8a5:::benchmark_data_smithi179_148205_object63689:head# 2024-02-09T17:57:12.869 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fafe9c8:::benchmark_data_smithi179_148205_object41655:head# 2024-02-09T17:57:12.869 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fb1550d:::benchmark_data_smithi179_148205_object39389:head# 2024-02-09T17:57:12.869 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fb20b3a:::benchmark_data_smithi179_148205_object71943:head# 2024-02-09T17:57:12.871 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fb2303a:::benchmark_data_smithi179_148205_object94476:head# 2024-02-09T17:57:12.872 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fb31cb2:::benchmark_data_smithi179_148205_object84555:head# 2024-02-09T17:57:12.872 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fb7c578:::benchmark_data_smithi179_148205_object7666:head# 2024-02-09T17:57:12.872 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fb9dc55:::benchmark_data_smithi179_148205_object22880:head# 2024-02-09T17:57:12.873 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fbb8060:::benchmark_data_smithi179_148205_object70083:head# 2024-02-09T17:57:12.873 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fbba84d:::benchmark_data_smithi179_148205_object66195:head# 2024-02-09T17:57:12.874 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fbbc05d:::benchmark_data_smithi179_148205_object50209:head# 2024-02-09T17:57:12.876 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fbbcf09:::benchmark_data_smithi179_148205_object2543:head# 2024-02-09T17:57:12.876 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fbc32e0:::benchmark_data_smithi179_148205_object102511:head# 2024-02-09T17:57:12.877 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fbc6844:::benchmark_data_smithi179_148205_object2444:head# 2024-02-09T17:57:12.877 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fbd1332:::benchmark_data_smithi179_148205_object36182:head# 2024-02-09T17:57:12.878 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fbd758b:::benchmark_data_smithi179_148205_object69271:head# 2024-02-09T17:57:12.878 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fbdd3f3:::benchmark_data_smithi179_148205_object26186:head# 2024-02-09T17:57:12.879 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fbf4bd7:::benchmark_data_smithi179_148205_object86731:head# 2024-02-09T17:57:12.879 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fbfd75c:::benchmark_data_smithi179_148205_object69419:head# 2024-02-09T17:57:12.880 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fc1664c:::benchmark_data_smithi179_148205_object41847:head# 2024-02-09T17:57:12.885 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fc23531:::benchmark_data_smithi179_148205_object48217:head# 2024-02-09T17:57:12.885 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fc2afc2:::benchmark_data_smithi179_148205_object42465:head# 2024-02-09T17:57:12.885 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fc342b4:::benchmark_data_smithi179_148205_object54002:head# 2024-02-09T17:57:12.885 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fc43f3e:::benchmark_data_smithi179_148205_object26189:head# 2024-02-09T17:57:12.885 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fc4ca04:::benchmark_data_smithi179_148205_object66330:head# 2024-02-09T17:57:12.885 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fc60866:::benchmark_data_smithi179_148205_object74718:head# 2024-02-09T17:57:12.885 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fc751ff:::benchmark_data_smithi179_148205_object32408:head# 2024-02-09T17:57:12.892 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fc83753:::benchmark_data_smithi179_148205_object100787:head# 2024-02-09T17:57:12.892 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fc901c4:::benchmark_data_smithi179_148205_object103149:head# 2024-02-09T17:57:12.892 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fc95bf7:::benchmark_data_smithi179_148205_object33790:head# 2024-02-09T17:57:12.892 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fc97a52:::benchmark_data_smithi179_148205_object83883:head# 2024-02-09T17:57:12.892 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fca003a:::benchmark_data_smithi179_148205_object15716:head# 2024-02-09T17:57:12.892 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fcbba18:::benchmark_data_smithi179_148205_object74099:head# 2024-02-09T17:57:12.892 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fcc1b8e:::benchmark_data_smithi179_148205_object72334:head# 2024-02-09T17:57:12.892 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fccfd3b:::benchmark_data_smithi179_148205_object67070:head# 2024-02-09T17:57:12.892 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fcf5178:::benchmark_data_smithi179_148205_object69370:head# 2024-02-09T17:57:12.894 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fcfe09b:::benchmark_data_smithi179_148205_object86214:head# 2024-02-09T17:57:12.894 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fcfe1dd:::benchmark_data_smithi179_148205_object6173:head# 2024-02-09T17:57:12.894 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fd00327:::benchmark_data_smithi179_148205_object94217:head# 2024-02-09T17:57:12.894 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fd0b4ac:::benchmark_data_smithi179_148205_object54357:head# 2024-02-09T17:57:12.897 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fd0c764:::benchmark_data_smithi179_148205_object18655:head# 2024-02-09T17:57:12.897 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fd12554:::benchmark_data_smithi179_148205_object47065:head# 2024-02-09T17:57:12.897 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fd15bc8:::benchmark_data_smithi179_148205_object76746:head# 2024-02-09T17:57:12.898 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fd1a7ed:::benchmark_data_smithi179_148205_object32513:head# 2024-02-09T17:57:12.899 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fd26f97:::benchmark_data_smithi179_148205_object89088:head# 2024-02-09T17:57:12.899 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fd28548:::benchmark_data_smithi179_148205_object36892:head# 2024-02-09T17:57:12.899 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fd32c20:::benchmark_data_smithi179_148205_object99172:head# 2024-02-09T17:57:12.899 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fd3bdc0:::benchmark_data_smithi179_148205_object32364:head# 2024-02-09T17:57:12.900 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fd45b0d:::benchmark_data_smithi179_148205_object34843:head# 2024-02-09T17:57:12.902 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fd46819:::benchmark_data_smithi179_148205_object20606:head# 2024-02-09T17:57:12.902 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fd47d57:::benchmark_data_smithi179_148205_object81527:head# 2024-02-09T17:57:12.902 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fd52860:::benchmark_data_smithi179_148205_object7786:head# 2024-02-09T17:57:12.902 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fd53a8e:::benchmark_data_smithi179_148205_object35647:head# 2024-02-09T17:57:12.902 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fd54e16:::benchmark_data_smithi179_148205_object11732:head# 2024-02-09T17:57:12.902 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fd5a84d:::benchmark_data_smithi179_148205_object66318:head# 2024-02-09T17:57:12.903 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fd6dd31:::benchmark_data_smithi179_148205_object61050:head# 2024-02-09T17:57:12.903 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fd71dbc:::benchmark_data_smithi179_148205_object32587:head# 2024-02-09T17:57:12.903 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fd90895:::benchmark_data_smithi179_148205_object87378:head# 2024-02-09T17:57:12.904 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fd9a7b2:::benchmark_data_smithi179_148205_object54335:head# 2024-02-09T17:57:12.904 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fd9a990:::benchmark_data_smithi179_148205_object97000:head# 2024-02-09T17:57:12.904 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fd9dcc4:::benchmark_data_smithi179_148205_object59098:head# 2024-02-09T17:57:12.905 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fdad2bb:::benchmark_data_smithi179_148205_object48998:head# 2024-02-09T17:57:12.905 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fdd6310:::benchmark_data_smithi179_148205_object92341:head# 2024-02-09T17:57:12.906 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fddd1e4:::benchmark_data_smithi179_148205_object90340:head# 2024-02-09T17:57:12.906 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fddfa3f:::benchmark_data_smithi179_148205_object18432:head# 2024-02-09T17:57:12.906 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fde49b8:::benchmark_data_smithi179_148205_object39100:head# 2024-02-09T17:57:12.907 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fe32f5d:::benchmark_data_smithi179_148205_object29002:head# 2024-02-09T17:57:12.908 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fe365ca:::benchmark_data_smithi179_148205_object46893:head# 2024-02-09T17:57:12.908 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fe42f2b:::benchmark_data_smithi179_148205_object61936:head# 2024-02-09T17:57:12.908 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fe472a3:::benchmark_data_smithi179_148205_object90618:head# 2024-02-09T17:57:12.908 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fe5c0a3:::benchmark_data_smithi179_148205_object12799:head# 2024-02-09T17:57:12.909 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fe64d79:::benchmark_data_smithi179_148205_object87938:head# 2024-02-09T17:57:12.909 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fe68aaf:::benchmark_data_smithi179_148205_object41908:head# 2024-02-09T17:57:12.910 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fe79160:::benchmark_data_smithi179_148205_object104427:head# 2024-02-09T17:57:12.913 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fe7aa6b:::benchmark_data_smithi179_148205_object43561:head# 2024-02-09T17:57:12.915 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fe7cade:::benchmark_data_smithi179_148205_object11625:head# 2024-02-09T17:57:12.915 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fe864fe:::benchmark_data_smithi179_148205_object81645:head# 2024-02-09T17:57:12.917 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fe9785a:::benchmark_data_smithi179_148205_object79204:head# 2024-02-09T17:57:12.918 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fe987f8:::benchmark_data_smithi179_148205_object88384:head# 2024-02-09T17:57:12.918 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1feb0b1e:::benchmark_data_smithi179_148205_object109483:head# 2024-02-09T17:57:12.918 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1febc932:::benchmark_data_smithi179_148205_object579:head# 2024-02-09T17:57:12.919 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1febe262:::benchmark_data_smithi179_148205_object5676:head# 2024-02-09T17:57:12.919 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1febf714:::benchmark_data_smithi179_148205_object3139:head# 2024-02-09T17:57:12.920 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fecc590:::benchmark_data_smithi179_148205_object21441:head# 2024-02-09T17:57:12.920 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fed7031:::benchmark_data_smithi179_148205_object48038:head# 2024-02-09T17:57:12.920 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fedd264:::benchmark_data_smithi179_148205_object69957:head# 2024-02-09T17:57:12.921 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fee008a:::benchmark_data_smithi179_148205_object88023:head# 2024-02-09T17:57:12.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:12 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:57:12.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:12 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:57:12.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:12 smithi179 ceph-mon[43975]: pgmap v2037: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 6.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 826 op/s; 21384/337380 objects degraded (6.338%) 2024-02-09T17:57:12.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:12 smithi179 ceph-mon[43975]: osdmap e825: 12 total, 11 up, 12 in 2024-02-09T17:57:12.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:12 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 21384/337380 objects degraded (6.338%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:57:12.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:12 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:57:12.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:12 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:57:12.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:12 smithi162 ceph-mon[44304]: pgmap v2037: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 6.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 826 op/s; 21384/337380 objects degraded (6.338%) 2024-02-09T17:57:12.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:12 smithi162 ceph-mon[44304]: osdmap e825: 12 total, 11 up, 12 in 2024-02-09T17:57:12.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:12 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 21384/337380 objects degraded (6.338%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:57:12.927 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1feee0ac:::benchmark_data_smithi179_148205_object108620:head# 2024-02-09T17:57:12.927 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fef3eca:::benchmark_data_smithi179_148205_object44922:head# 2024-02-09T17:57:12.927 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fef44dd:::benchmark_data_smithi179_148205_object39209:head# 2024-02-09T17:57:12.928 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fefef90:::benchmark_data_smithi179_148205_object62908:head# 2024-02-09T17:57:12.928 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ff0eee5:::benchmark_data_smithi179_148205_object42207:head# 2024-02-09T17:57:12.928 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ff19144:::benchmark_data_smithi179_148205_object71424:head# 2024-02-09T17:57:12.928 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ff19783:::benchmark_data_smithi179_148205_object14074:head# 2024-02-09T17:57:12.928 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ff24023:::benchmark_data_smithi179_148205_object70420:head# 2024-02-09T17:57:12.928 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ff29cc8:::benchmark_data_smithi179_148205_object6665:head# 2024-02-09T17:57:12.928 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ff38f0c:::benchmark_data_smithi179_148205_object50719:head# 2024-02-09T17:57:12.928 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ff39a47:::benchmark_data_smithi179_148205_object107915:head# 2024-02-09T17:57:12.931 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ff3be2f:::benchmark_data_smithi179_148205_object57602:head# 2024-02-09T17:57:12.931 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ff4f715:::benchmark_data_smithi179_148205_object76107:head# 2024-02-09T17:57:12.931 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ff579e3:::benchmark_data_smithi179_148205_object82640:head# 2024-02-09T17:57:12.931 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ff58ec0:::benchmark_data_smithi179_148205_object11016:head# 2024-02-09T17:57:12.931 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ff5adab:::benchmark_data_smithi179_148205_object60756:head# 2024-02-09T17:57:12.931 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ff5f0d4:::benchmark_data_smithi179_148205_object25575:head# 2024-02-09T17:57:12.931 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ff6ccf3:::benchmark_data_smithi179_148205_object102406:head# 2024-02-09T17:57:12.931 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ff6ceb3:::benchmark_data_smithi179_148205_object10211:head# 2024-02-09T17:57:12.932 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ff75ef5:::benchmark_data_smithi179_148205_object48082:head# 2024-02-09T17:57:12.932 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ff7b773:::benchmark_data_smithi179_148205_object45327:head# 2024-02-09T17:57:12.933 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ff9ca37:::benchmark_data_smithi179_148205_object23237:head# 2024-02-09T17:57:12.933 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ffa6fb3:::benchmark_data_smithi179_148205_object8469:head# 2024-02-09T17:57:12.934 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ffaa3ab:::benchmark_data_smithi179_148205_object36793:head# 2024-02-09T17:57:12.934 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ffadb98:::benchmark_data_smithi179_148205_object36760:head# 2024-02-09T17:57:12.934 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ffb3ee2:::benchmark_data_smithi179_148205_object75221:head# 2024-02-09T17:57:12.935 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ffb69ad:::benchmark_data_smithi179_148205_object19628:head# 2024-02-09T17:57:12.935 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ffca75e:::benchmark_data_smithi179_148205_object66871:head# 2024-02-09T17:57:12.935 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1ffe0bd9:::benchmark_data_smithi179_148205_object7120:head# 2024-02-09T17:57:12.936 INFO:teuthology.orchestra.run.smithi099.stderr:Read #8:1fff10fd:::benchmark_data_smithi179_148205_object47895:head# 2024-02-09T17:57:12.942 INFO:teuthology.orchestra.run.smithi099.stderr:Export successful 2024-02-09T17:57:14.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:14 smithi099 ceph-mon[39420]: pgmap v2039: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 6.7 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s; 20509/328638 objects degraded (6.241%) 2024-02-09T17:57:14.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:14 smithi179 ceph-mon[43975]: pgmap v2039: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 6.7 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s; 20509/328638 objects degraded (6.241%) 2024-02-09T17:57:14.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:14 smithi162 ceph-mon[44304]: pgmap v2039: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 6.7 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s; 20509/328638 objects degraded (6.241%) 2024-02-09T17:57:16.154 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:57:16.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:16.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:57:16.261 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:57:16.310 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:16.338 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:57:16.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:16.343 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:57:16.440 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:16 smithi099 ceph-mon[39420]: pgmap v2040: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 6.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 807 op/s; 20509/325164 objects degraded (6.307%) 2024-02-09T17:57:16.443 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:57:16.499 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:16.532 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:57:16.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:16.541 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:57:16.642 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:57:16.671 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:16.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:16 smithi179 ceph-mon[43975]: pgmap v2040: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 6.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 807 op/s; 20509/325164 objects degraded (6.307%) 2024-02-09T17:57:16.695 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:57:16.698 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:16 smithi162 ceph-mon[44304]: pgmap v2040: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 6.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 807 op/s; 20509/325164 objects degraded (6.307%) 2024-02-09T17:57:16.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:16.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:57:16.800 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:57:16.828 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:16.853 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:57:16.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:16.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:57:16.959 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:57:16.994 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:17.259 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:57:17.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:17.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:57:17.367 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:57:17.435 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:17.483 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:57:17.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:17.487 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:57:17.520 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:57:17.587 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:57:17.616 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:17.638 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:57:17.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:17.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:57:17.742 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:57:17.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:17 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 20509/325164 objects degraded (6.307%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:57:17.769 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:17 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 20509/325164 objects degraded (6.307%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:57:17.771 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:17.795 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:57:17.803 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:17.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:57:17.905 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:57:17.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:17 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 20509/325164 objects degraded (6.307%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:57:17.934 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:17.959 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:57:17.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:17.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:57:18.065 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:57:18.094 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:18.121 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:57:18.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:18.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:57:18.226 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:57:18.255 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:18.286 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:57:18.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:18.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:57:18.394 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:57:18.438 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:18.457 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is unset 2024-02-09T17:57:18.483 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:57:18.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:18.491 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:57:18.496 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell -n osd.3 --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --force --op remove --pgid 8.8 2024-02-09T17:57:18.592 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:57:18.623 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:18.643 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:57:18.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:18.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:57:18.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:18 smithi099 ceph-mon[39420]: pgmap v2041: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 6.4 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 940 op/s; 19668/316566 objects degraded (6.213%) 2024-02-09T17:57:18.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:18 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3406968932' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:57:18.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:18 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:57:18.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:18 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:57:18.750 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:57:18.800 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:18 smithi179 ceph-mon[43975]: pgmap v2041: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 6.4 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 940 op/s; 19668/316566 objects degraded (6.213%) 2024-02-09T17:57:18.801 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:18 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3406968932' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:57:18.801 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:18 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:57:18.801 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:18 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:57:18.803 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:18.850 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:57:18.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:18.857 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:57:18.900 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/osd.3/config 2024-02-09T17:57:18.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:18 smithi162 ceph-mon[44304]: pgmap v2041: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 6.4 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 940 op/s; 19668/316566 objects degraded (6.213%) 2024-02-09T17:57:18.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:18 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3406968932' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:57:18.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:18 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:57:18.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:18 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:57:18.958 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:57:18.986 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:19.009 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:57:19.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:19.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:57:19.115 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:57:19.145 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:19.169 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:57:19.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:19.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:57:19.273 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:57:19.307 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:19.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:19 smithi099 ceph-mon[39420]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:57:19.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:19 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:57:19.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:19 smithi099 ceph-mon[39420]: osdmap e826: 12 total, 11 up, 12 in 2024-02-09T17:57:19.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:19 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:57:19.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:19 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:57:19.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:19 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:57:19.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:19 smithi179 ceph-mon[43975]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:57:19.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:19 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:57:19.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:19 smithi179 ceph-mon[43975]: osdmap e826: 12 total, 11 up, 12 in 2024-02-09T17:57:19.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:19 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:57:19.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:19 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:57:19.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:19 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:57:19.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:19 smithi162 ceph-mon[44304]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:57:19.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:19 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:57:19.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:19 smithi162 ceph-mon[44304]: osdmap e826: 12 total, 11 up, 12 in 2024-02-09T17:57:19.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:19 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:57:19.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:19 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:57:19.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:19 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:57:20.161 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:57:20.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:20.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:57:20.266 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:57:20.308 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:20.421 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:57:20.429 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:20.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:57:20.530 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:57:20.577 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:20.616 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:57:20.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:20.621 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:57:20.722 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:57:20.730 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:20 smithi099 ceph-mon[39420]: pgmap v2043: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 6.3 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 19668/311478 objects degraded (6.314%) 2024-02-09T17:57:20.752 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:20 smithi179 ceph-mon[43975]: pgmap v2043: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 6.3 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 19668/311478 objects degraded (6.314%) 2024-02-09T17:57:20.754 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:20.781 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:57:20.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:20.784 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:57:20.885 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:57:20.919 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:20.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:20 smithi162 ceph-mon[44304]: pgmap v2043: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 6.3 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 19668/311478 objects degraded (6.314%) 2024-02-09T17:57:21.035 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:57:21.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:21.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:57:21.144 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:57:21.171 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:21.195 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:57:21.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:21.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:57:21.259 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-09T17:57:21.299 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:57:21.327 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:21.349 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:57:21.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:21.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:57:21.455 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:57:21.508 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:21.554 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:57:21.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:21.564 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:57:21.636 INFO:teuthology.orchestra.run.smithi099.stderr:Creating an OSD daemon form without an OSD FSID value 2024-02-09T17:57:21.664 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:57:21.720 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:21.744 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:57:21.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:21.753 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:57:21.854 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:57:21.914 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:21.926 INFO:journalctl@ceph.osd.4.smithi162.stdout:Feb 09 17:57:21 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-4[49654]: 2024-02-09T17:57:21.589+0000 7f1a0bbdc700 -1 log_channel(cluster) log [ERR] : osd.4 pg[8.b]: reservation requested while still reserved 2024-02-09T17:57:21.938 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:57:21.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:21.945 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:57:22.046 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:57:22.073 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:22.095 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:57:22.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:22.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:57:22.200 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:57:22.227 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:22.251 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:57:22.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:22.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:57:22.347 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:57:22.348 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 20, 2024-02-09T17:57:22.348 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 600, 2024-02-09T17:57:22.348 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [ 2024-02-09T17:57:22.348 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.348 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25837.0:180603 6.a e630/600 6:57888cf7:::benchmark_data_smithi179_126206_object50222:head v 630'11416, mlcod=630'11414)", 2024-02-09T17:57:22.348 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:48:19.898551+0000", 2024-02-09T17:57:22.348 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 542.39165678999996, 2024-02-09T17:57:22.348 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.169202886, 2024-02-09T17:57:22.348 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:57:22.348 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:57:22.348 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:57:22.349 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.349 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:57:22.349 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:19.898551+0000", 2024-02-09T17:57:22.349 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:22.349 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.349 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.349 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:57:22.349 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:19.898551+0000", 2024-02-09T17:57:22.349 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:22.349 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.349 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.349 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:57:22.349 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:19.898552+0000", 2024-02-09T17:57:22.349 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.9899999999999999e-07 2024-02-09T17:57:22.350 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.350 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.350 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:57:22.350 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:19.898560+0000", 2024-02-09T17:57:22.350 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.3490000000000003e-06 2024-02-09T17:57:22.350 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.350 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.350 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:57:22.350 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:19.898574+0000", 2024-02-09T17:57:22.350 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4508000000000001e-05 2024-02-09T17:57:22.350 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.350 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.350 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:57:22.350 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:19.898582+0000", 2024-02-09T17:57:22.350 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.2519999999999999e-06 2024-02-09T17:57:22.351 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.351 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.351 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:57:22.351 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:20.055533+0000", 2024-02-09T17:57:22.351 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.15695081299999999 2024-02-09T17:57:22.351 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.351 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.351 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:57:22.351 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:20.055562+0000", 2024-02-09T17:57:22.351 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.9283000000000001e-05 2024-02-09T17:57:22.351 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.351 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.351 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:57:22.351 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:20.067711+0000", 2024-02-09T17:57:22.352 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.012148871 2024-02-09T17:57:22.352 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.352 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.352 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:57:22.352 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:20.067754+0000", 2024-02-09T17:57:22.352 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2911000000000003e-05 2024-02-09T17:57:22.352 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:22.352 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:22.352 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:22.352 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.352 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.352 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25837.0:180607 6.a e630/600 6:5c83d64d:::benchmark_data_smithi179_126206_object50226:head v 630'11417, mlcod=630'11414)", 2024-02-09T17:57:22.352 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:48:19.907823+0000", 2024-02-09T17:57:22.352 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 542.382384581, 2024-02-09T17:57:22.353 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.172944088, 2024-02-09T17:57:22.353 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:57:22.353 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:57:22.353 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:57:22.353 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.353 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:57:22.353 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:19.907823+0000", 2024-02-09T17:57:22.353 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:22.353 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.353 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.353 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:57:22.353 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:19.907823+0000", 2024-02-09T17:57:22.353 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:22.353 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.354 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.354 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:57:22.354 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:19.907825+0000", 2024-02-09T17:57:22.354 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.404e-06 2024-02-09T17:57:22.354 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.354 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.354 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:57:22.354 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:19.907837+0000", 2024-02-09T17:57:22.354 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1936000000000001e-05 2024-02-09T17:57:22.354 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.354 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.354 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:57:22.354 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:20.055528+0000", 2024-02-09T17:57:22.354 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.14769067999999999 2024-02-09T17:57:22.355 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.355 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.355 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:57:22.355 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:20.055538+0000", 2024-02-09T17:57:22.355 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0548e-05 2024-02-09T17:57:22.355 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.355 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.355 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:57:22.355 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:20.060269+0000", 2024-02-09T17:57:22.355 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.004731257 2024-02-09T17:57:22.355 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.355 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.355 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:57:22.355 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:20.060284+0000", 2024-02-09T17:57:22.355 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4518999999999999e-05 2024-02-09T17:57:22.356 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.356 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.356 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:57:22.356 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:20.080736+0000", 2024-02-09T17:57:22.356 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.020452436000000001 2024-02-09T17:57:22.356 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.356 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.356 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:57:22.356 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:20.080768+0000", 2024-02-09T17:57:22.356 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.1307999999999999e-05 2024-02-09T17:57:22.356 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:22.356 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:22.356 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:22.356 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.357 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.357 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.25837.0:180601 6.0 e630/576 6:03594e9d:::benchmark_data_smithi179_126206_object50220:head v 630'11216, mlcod=630'11214)", 2024-02-09T17:57:22.357 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:48:19.913305+0000", 2024-02-09T17:57:22.357 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 542.37690338699997, 2024-02-09T17:57:22.357 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.16802792599999999, 2024-02-09T17:57:22.357 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:57:22.357 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:57:22.357 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:57:22.357 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.357 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:57:22.357 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:19.913305+0000", 2024-02-09T17:57:22.357 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:22.357 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.357 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.357 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:57:22.358 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:19.913305+0000", 2024-02-09T17:57:22.358 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:22.358 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.358 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.358 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:57:22.358 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:19.913306+0000", 2024-02-09T17:57:22.358 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.164e-06 2024-02-09T17:57:22.358 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.358 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.358 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:57:22.358 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:19.913316+0000", 2024-02-09T17:57:22.358 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.9049999999999992e-06 2024-02-09T17:57:22.358 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.358 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.358 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:57:22.359 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:20.055524+0000", 2024-02-09T17:57:22.359 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.14220781499999999 2024-02-09T17:57:22.359 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.359 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.359 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:57:22.359 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:20.055538+0000", 2024-02-09T17:57:22.359 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4231e-05 2024-02-09T17:57:22.359 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.359 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.359 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:57:22.359 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:20.055593+0000", 2024-02-09T17:57:22.359 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.4960000000000002e-05 2024-02-09T17:57:22.359 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.359 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.359 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:57:22.360 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:20.055616+0000", 2024-02-09T17:57:22.360 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.281e-05 2024-02-09T17:57:22.360 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.360 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.360 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:57:22.360 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:20.081299+0000", 2024-02-09T17:57:22.360 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.025683113 2024-02-09T17:57:22.360 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.360 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.360 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:57:22.360 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:20.081333+0000", 2024-02-09T17:57:22.360 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.3927999999999998e-05 2024-02-09T17:57:22.360 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:22.360 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:22.361 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:22.361 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.361 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.361 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.16440.0:60983 7.f e674/661 7:ff502c2e:::benchmark_data_smithi179_135804_object60982:head v 674'3853, mlcod=674'3850)", 2024-02-09T17:57:22.361 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:50:55.011600+0000", 2024-02-09T17:57:22.361 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 387.27860845100003, 2024-02-09T17:57:22.361 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.19042357100000001, 2024-02-09T17:57:22.361 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:57:22.361 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:57:22.361 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:57:22.361 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.361 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:57:22.361 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.011600+0000", 2024-02-09T17:57:22.361 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:22.361 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.362 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.362 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:57:22.362 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.011600+0000", 2024-02-09T17:57:22.362 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:22.362 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.362 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.362 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:57:22.362 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.011600+0000", 2024-02-09T17:57:22.362 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.13e-07 2024-02-09T17:57:22.362 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.362 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.362 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:57:22.362 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.011635+0000", 2024-02-09T17:57:22.363 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.4453999999999999e-05 2024-02-09T17:57:22.363 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.363 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.363 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:57:22.363 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.011650+0000", 2024-02-09T17:57:22.363 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.488e-05 2024-02-09T17:57:22.363 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.363 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.363 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:57:22.363 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.011656+0000", 2024-02-09T17:57:22.363 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.5649999999999998e-06 2024-02-09T17:57:22.363 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.363 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.363 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:57:22.363 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.011764+0000", 2024-02-09T17:57:22.364 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000107988 2024-02-09T17:57:22.364 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.364 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.364 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:57:22.364 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.011781+0000", 2024-02-09T17:57:22.364 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.6806000000000001e-05 2024-02-09T17:57:22.364 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.364 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.364 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:57:22.364 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.013991+0000", 2024-02-09T17:57:22.364 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0022102889999999998 2024-02-09T17:57:22.364 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.364 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.365 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:57:22.365 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.202023+0000", 2024-02-09T17:57:22.365 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.18803197599999999 2024-02-09T17:57:22.365 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:22.365 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:22.365 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:22.365 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.365 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.365 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.16440.0:60992 7.4 7:25bdd397:::benchmark_data_smithi179_135804_object60991:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e674)", 2024-02-09T17:57:22.365 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:50:55.012006+0000", 2024-02-09T17:57:22.365 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 387.27820254400001, 2024-02-09T17:57:22.365 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.19498057899999999, 2024-02-09T17:57:22.365 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:57:22.365 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:57:22.365 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:57:22.366 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.16440", 2024-02-09T17:57:22.366 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:57:22.366 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 60992 2024-02-09T17:57:22.366 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.366 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:57:22.366 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.366 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:57:22.366 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.012006+0000", 2024-02-09T17:57:22.366 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:22.366 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.366 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.366 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:57:22.366 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.012006+0000", 2024-02-09T17:57:22.366 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:22.366 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.367 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.367 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:57:22.367 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.012007+0000", 2024-02-09T17:57:22.367 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.319e-06 2024-02-09T17:57:22.367 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.367 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.367 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:57:22.367 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.012071+0000", 2024-02-09T17:57:22.367 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.4219000000000003e-05 2024-02-09T17:57:22.367 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.367 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.367 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:57:22.367 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.012081+0000", 2024-02-09T17:57:22.367 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.9019999999999994e-06 2024-02-09T17:57:22.367 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.367 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.368 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:57:22.368 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.012088+0000", 2024-02-09T17:57:22.368 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.1749999999999999e-06 2024-02-09T17:57:22.368 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.368 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.368 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:57:22.368 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.012139+0000", 2024-02-09T17:57:22.368 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.0408999999999998e-05 2024-02-09T17:57:22.368 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.368 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.368 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:57:22.368 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.012262+0000", 2024-02-09T17:57:22.368 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012394000000000001 2024-02-09T17:57:22.368 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.369 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.369 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:57:22.369 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.012402+0000", 2024-02-09T17:57:22.369 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013949500000000001 2024-02-09T17:57:22.369 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.369 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.369 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:57:22.369 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.204409+0000", 2024-02-09T17:57:22.369 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.192007386 2024-02-09T17:57:22.369 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.369 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.369 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:22.369 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.206652+0000", 2024-02-09T17:57:22.369 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0022421519999999999 2024-02-09T17:57:22.369 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.369 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.370 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:22.370 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.206898+0000", 2024-02-09T17:57:22.370 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00024642899999999999 2024-02-09T17:57:22.370 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.370 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.370 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:57:22.370 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.206950+0000", 2024-02-09T17:57:22.370 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2176000000000003e-05 2024-02-09T17:57:22.370 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.370 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.370 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:57:22.370 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.206986+0000", 2024-02-09T17:57:22.370 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.5976999999999998e-05 2024-02-09T17:57:22.370 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:22.370 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:22.371 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:22.371 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.371 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.371 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.16440.0:60993 7.4 7:2a843e6d:::benchmark_data_smithi179_135804_object60992:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e674)", 2024-02-09T17:57:22.371 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:50:55.012103+0000", 2024-02-09T17:57:22.371 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 387.27810551300001, 2024-02-09T17:57:22.371 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20198994000000001, 2024-02-09T17:57:22.371 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:57:22.371 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:57:22.371 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:57:22.371 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.16440", 2024-02-09T17:57:22.371 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:57:22.371 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 60993 2024-02-09T17:57:22.371 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.371 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:57:22.372 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.372 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:57:22.372 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.012103+0000", 2024-02-09T17:57:22.372 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:22.372 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.372 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.372 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:57:22.372 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.012103+0000", 2024-02-09T17:57:22.372 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:22.372 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.372 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.372 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:57:22.372 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.012103+0000", 2024-02-09T17:57:22.372 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1900000000000003e-07 2024-02-09T17:57:22.373 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.373 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.373 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:57:22.373 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.012138+0000", 2024-02-09T17:57:22.373 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.5407999999999997e-05 2024-02-09T17:57:22.373 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.373 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.373 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:57:22.373 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.012146+0000", 2024-02-09T17:57:22.373 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.4189999999999996e-06 2024-02-09T17:57:22.373 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.373 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.373 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:57:22.373 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.012151+0000", 2024-02-09T17:57:22.373 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3970000000000002e-06 2024-02-09T17:57:22.374 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.374 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.374 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:57:22.374 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.202126+0000", 2024-02-09T17:57:22.374 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.18997508199999999 2024-02-09T17:57:22.374 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.374 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.374 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:57:22.374 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.202292+0000", 2024-02-09T17:57:22.374 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000166012 2024-02-09T17:57:22.374 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.374 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.374 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:57:22.374 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.202428+0000", 2024-02-09T17:57:22.374 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000135711 2024-02-09T17:57:22.375 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.375 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.375 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:57:22.375 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.206666+0000", 2024-02-09T17:57:22.375 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.004237879 2024-02-09T17:57:22.375 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.375 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.375 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:22.375 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.207053+0000", 2024-02-09T17:57:22.375 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00038739900000000002 2024-02-09T17:57:22.375 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.375 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.375 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:22.375 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.214006+0000", 2024-02-09T17:57:22.376 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0069521499999999998 2024-02-09T17:57:22.376 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.376 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.376 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:57:22.376 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.214057+0000", 2024-02-09T17:57:22.376 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1211999999999997e-05 2024-02-09T17:57:22.376 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.376 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.376 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:57:22.376 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.214092+0000", 2024-02-09T17:57:22.376 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.5751999999999999e-05 2024-02-09T17:57:22.376 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:22.376 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:22.376 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:22.376 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.377 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.377 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.16440.0:60994 7.4 7:28e19133:::benchmark_data_smithi179_135804_object60993:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e674)", 2024-02-09T17:57:22.377 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:50:55.012158+0000", 2024-02-09T17:57:22.377 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 387.27804971500001, 2024-02-09T17:57:22.377 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20203644600000001, 2024-02-09T17:57:22.377 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:57:22.377 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:57:22.377 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:57:22.377 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.16440", 2024-02-09T17:57:22.377 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:57:22.377 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 60994 2024-02-09T17:57:22.377 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.377 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:57:22.377 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.377 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:57:22.378 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.012158+0000", 2024-02-09T17:57:22.378 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:22.378 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.378 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.378 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:57:22.378 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.012158+0000", 2024-02-09T17:57:22.378 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:22.378 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.378 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.378 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:57:22.378 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.012159+0000", 2024-02-09T17:57:22.378 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2600000000000002e-07 2024-02-09T17:57:22.378 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.378 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.379 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:57:22.379 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.012191+0000", 2024-02-09T17:57:22.379 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.1888999999999999e-05 2024-02-09T17:57:22.379 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.379 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.379 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:57:22.379 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.012197+0000", 2024-02-09T17:57:22.379 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.4989999999999999e-06 2024-02-09T17:57:22.379 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.379 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.379 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:57:22.379 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.012202+0000", 2024-02-09T17:57:22.379 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.6630000000000001e-06 2024-02-09T17:57:22.379 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.379 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.380 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:57:22.380 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.204440+0000", 2024-02-09T17:57:22.380 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.19223841899999999 2024-02-09T17:57:22.380 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.380 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.380 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:57:22.380 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.204583+0000", 2024-02-09T17:57:22.380 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000142483 2024-02-09T17:57:22.380 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.380 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.380 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:57:22.380 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.204727+0000", 2024-02-09T17:57:22.380 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000144413 2024-02-09T17:57:22.380 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.380 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.381 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:22.381 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.207102+0000", 2024-02-09T17:57:22.381 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0023749190000000001 2024-02-09T17:57:22.381 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.381 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.381 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:57:22.381 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.211602+0000", 2024-02-09T17:57:22.381 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0044995649999999996 2024-02-09T17:57:22.381 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.381 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.381 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:22.381 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.214129+0000", 2024-02-09T17:57:22.381 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0025270620000000001 2024-02-09T17:57:22.381 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.381 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.382 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:57:22.382 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.214160+0000", 2024-02-09T17:57:22.382 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.1038000000000003e-05 2024-02-09T17:57:22.382 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.382 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.382 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:57:22.382 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.214195+0000", 2024-02-09T17:57:22.382 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.4969999999999999e-05 2024-02-09T17:57:22.382 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:22.382 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:22.382 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:22.382 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.382 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.382 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.16440.0:60995 7.d e674/661 7:be52cce4:::benchmark_data_smithi179_135804_object60994:head v 674'3776, mlcod=674'3774)", 2024-02-09T17:57:22.382 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:50:55.016422+0000", 2024-02-09T17:57:22.383 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 387.27378639800003, 2024-02-09T17:57:22.383 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.196075797, 2024-02-09T17:57:22.383 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:57:22.383 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:57:22.383 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:57:22.383 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.383 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:57:22.383 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.016422+0000", 2024-02-09T17:57:22.383 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:22.383 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.383 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.383 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:57:22.383 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.016422+0000", 2024-02-09T17:57:22.383 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:22.383 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.384 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.384 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:57:22.384 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.016422+0000", 2024-02-09T17:57:22.384 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.0400000000000005e-07 2024-02-09T17:57:22.384 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.384 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.384 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:57:22.384 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.016557+0000", 2024-02-09T17:57:22.384 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000135017 2024-02-09T17:57:22.384 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.384 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.384 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:57:22.384 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.202022+0000", 2024-02-09T17:57:22.384 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.185464823 2024-02-09T17:57:22.384 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.385 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.385 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:57:22.385 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.202037+0000", 2024-02-09T17:57:22.385 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4538e-05 2024-02-09T17:57:22.385 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.385 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.385 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:57:22.385 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.202113+0000", 2024-02-09T17:57:22.385 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.6124000000000005e-05 2024-02-09T17:57:22.385 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.385 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.385 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:57:22.385 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.202138+0000", 2024-02-09T17:57:22.385 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.5415e-05 2024-02-09T17:57:22.386 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.386 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.386 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:57:22.386 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.208662+0000", 2024-02-09T17:57:22.386 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0065240009999999998 2024-02-09T17:57:22.386 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.386 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.386 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:57:22.386 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.212497+0000", 2024-02-09T17:57:22.386 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0038350749999999999 2024-02-09T17:57:22.386 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:22.386 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:22.386 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:22.386 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.387 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.387 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.16440.0:61011 7.f e674/661 7:f12e60e5:::benchmark_data_smithi179_135804_object61010:head v 674'3854, mlcod=674'3852)", 2024-02-09T17:57:22.387 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:50:55.026760+0000", 2024-02-09T17:57:22.387 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 387.26344847000001, 2024-02-09T17:57:22.387 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.177944042, 2024-02-09T17:57:22.387 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:57:22.387 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:57:22.387 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:57:22.387 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.387 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:57:22.387 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.026760+0000", 2024-02-09T17:57:22.387 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:22.387 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.387 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.387 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:57:22.388 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.026760+0000", 2024-02-09T17:57:22.388 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:22.388 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.388 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.388 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:57:22.388 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.026761+0000", 2024-02-09T17:57:22.388 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.128e-06 2024-02-09T17:57:22.388 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.388 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.388 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:57:22.388 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.026821+0000", 2024-02-09T17:57:22.388 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.0739999999999998e-05 2024-02-09T17:57:22.388 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.388 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.388 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:57:22.389 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.202043+0000", 2024-02-09T17:57:22.389 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.17522120699999999 2024-02-09T17:57:22.389 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.389 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.389 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:57:22.389 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.202053+0000", 2024-02-09T17:57:22.389 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.8649999999999996e-06 2024-02-09T17:57:22.389 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.389 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.389 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:57:22.389 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.202183+0000", 2024-02-09T17:57:22.389 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013011000000000001 2024-02-09T17:57:22.389 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.389 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.389 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:57:22.390 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.202201+0000", 2024-02-09T17:57:22.390 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.8499999999999999e-05 2024-02-09T17:57:22.390 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.390 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.390 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:57:22.390 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.204620+0000", 2024-02-09T17:57:22.390 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00241881 2024-02-09T17:57:22.390 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.390 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.390 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:57:22.390 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:55.204704+0000", 2024-02-09T17:57:22.390 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.3682000000000003e-05 2024-02-09T17:57:22.390 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:22.390 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:22.390 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:22.391 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.391 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.391 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.16440.0:64896 7.4 7:22763377:::benchmark_data_smithi179_135804_object64895:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e677)", 2024-02-09T17:57:22.391 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:50:57.781322+0000", 2024-02-09T17:57:22.391 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 384.50888616700001, 2024-02-09T17:57:22.391 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.21076742700000001, 2024-02-09T17:57:22.391 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:57:22.391 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:57:22.391 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:57:22.391 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.16440", 2024-02-09T17:57:22.391 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:57:22.391 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 64896 2024-02-09T17:57:22.391 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.391 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:57:22.391 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.392 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:57:22.392 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.781322+0000", 2024-02-09T17:57:22.392 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:22.392 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.392 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.392 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:57:22.392 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.781322+0000", 2024-02-09T17:57:22.392 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:22.392 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.392 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.392 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:57:22.392 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.781322+0000", 2024-02-09T17:57:22.392 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.5699999999999998e-07 2024-02-09T17:57:22.392 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.392 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.393 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:57:22.393 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.781365+0000", 2024-02-09T17:57:22.393 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2533000000000001e-05 2024-02-09T17:57:22.393 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.393 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.393 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:57:22.393 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.781372+0000", 2024-02-09T17:57:22.393 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.391e-06 2024-02-09T17:57:22.393 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.393 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.393 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:57:22.393 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.781378+0000", 2024-02-09T17:57:22.393 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.5350000000000004e-06 2024-02-09T17:57:22.393 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.393 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.394 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:57:22.394 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.781431+0000", 2024-02-09T17:57:22.394 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3520000000000003e-05 2024-02-09T17:57:22.394 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.394 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.394 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:57:22.394 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.781545+0000", 2024-02-09T17:57:22.394 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000113267 2024-02-09T17:57:22.394 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.394 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.394 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:57:22.394 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.781663+0000", 2024-02-09T17:57:22.394 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000118776 2024-02-09T17:57:22.394 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.394 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.395 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:57:22.395 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.785902+0000", 2024-02-09T17:57:22.395 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0042387529999999996 2024-02-09T17:57:22.395 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.395 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.395 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:22.395 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.792579+0000", 2024-02-09T17:57:22.395 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0066770900000000001 2024-02-09T17:57:22.395 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.395 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.395 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:22.395 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.992022+0000", 2024-02-09T17:57:22.395 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.19944258600000001 2024-02-09T17:57:22.395 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.395 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.396 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:57:22.396 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.992062+0000", 2024-02-09T17:57:22.396 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9737999999999998e-05 2024-02-09T17:57:22.396 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.396 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.396 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:57:22.396 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.992089+0000", 2024-02-09T17:57:22.396 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.7781e-05 2024-02-09T17:57:22.396 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:22.396 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:22.396 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:22.396 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.396 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.396 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.16440.0:64918 7.4 7:21f41282:::benchmark_data_smithi179_135804_object64917:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e677)", 2024-02-09T17:57:22.397 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:50:57.796365+0000", 2024-02-09T17:57:22.397 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 384.49384348699999, 2024-02-09T17:57:22.397 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.198345465, 2024-02-09T17:57:22.397 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:57:22.397 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:57:22.397 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:57:22.397 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.16440", 2024-02-09T17:57:22.397 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:57:22.397 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 64918 2024-02-09T17:57:22.397 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.397 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:57:22.397 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.397 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:57:22.397 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.796365+0000", 2024-02-09T17:57:22.397 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:22.398 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.398 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.398 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:57:22.398 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.796365+0000", 2024-02-09T17:57:22.398 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:22.398 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.398 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.398 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:57:22.398 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.796365+0000", 2024-02-09T17:57:22.398 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.0599999999999999e-07 2024-02-09T17:57:22.398 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.398 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.398 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:57:22.398 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.796386+0000", 2024-02-09T17:57:22.398 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0540999999999999e-05 2024-02-09T17:57:22.399 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.399 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.399 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:57:22.399 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.796401+0000", 2024-02-09T17:57:22.399 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5362e-05 2024-02-09T17:57:22.399 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.399 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.399 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:57:22.399 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.796408+0000", 2024-02-09T17:57:22.399 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.861e-06 2024-02-09T17:57:22.399 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.399 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.399 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:57:22.399 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.796452+0000", 2024-02-09T17:57:22.399 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.4135999999999999e-05 2024-02-09T17:57:22.400 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.400 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.400 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:57:22.400 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.796609+0000", 2024-02-09T17:57:22.400 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00015707899999999999 2024-02-09T17:57:22.400 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.400 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.400 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:57:22.400 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.796764+0000", 2024-02-09T17:57:22.400 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00015433499999999999 2024-02-09T17:57:22.400 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.400 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.400 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:57:22.400 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.798793+0000", 2024-02-09T17:57:22.401 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0020289610000000001 2024-02-09T17:57:22.401 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.401 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.401 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:22.401 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.801981+0000", 2024-02-09T17:57:22.401 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.003188115 2024-02-09T17:57:22.401 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.401 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.401 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:22.401 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.994653+0000", 2024-02-09T17:57:22.401 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.192671912 2024-02-09T17:57:22.401 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.401 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.401 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:57:22.401 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.994687+0000", 2024-02-09T17:57:22.402 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.4570000000000003e-05 2024-02-09T17:57:22.402 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.402 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.402 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:57:22.402 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.994710+0000", 2024-02-09T17:57:22.402 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2687000000000001e-05 2024-02-09T17:57:22.402 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:22.402 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:22.402 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:22.402 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.402 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.402 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.16440.0:64934 7.4 7:2d16648d:::benchmark_data_smithi179_135804_object64933:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e677)", 2024-02-09T17:57:22.402 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:50:57.805129+0000", 2024-02-09T17:57:22.402 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 384.485079058, 2024-02-09T17:57:22.402 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.18967294900000001, 2024-02-09T17:57:22.403 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:57:22.403 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:57:22.403 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:57:22.403 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.16440", 2024-02-09T17:57:22.403 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:57:22.403 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 64934 2024-02-09T17:57:22.403 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.403 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:57:22.403 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.403 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:57:22.403 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.805129+0000", 2024-02-09T17:57:22.403 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:22.403 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.403 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.404 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:57:22.404 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.805129+0000", 2024-02-09T17:57:22.404 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:22.404 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.404 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.404 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:57:22.404 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.805131+0000", 2024-02-09T17:57:22.404 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.745e-06 2024-02-09T17:57:22.404 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.404 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.404 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:57:22.404 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.805159+0000", 2024-02-09T17:57:22.404 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.8586e-05 2024-02-09T17:57:22.405 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.405 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.405 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:57:22.405 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.805172+0000", 2024-02-09T17:57:22.405 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2452999999999999e-05 2024-02-09T17:57:22.405 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.405 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.405 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:57:22.405 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.805180+0000", 2024-02-09T17:57:22.405 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.4610000000000008e-06 2024-02-09T17:57:22.405 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.405 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.405 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:57:22.405 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.805237+0000", 2024-02-09T17:57:22.405 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6345999999999999e-05 2024-02-09T17:57:22.406 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.406 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.406 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:57:22.406 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.805374+0000", 2024-02-09T17:57:22.406 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000137844 2024-02-09T17:57:22.406 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.406 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.406 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:57:22.406 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.805513+0000", 2024-02-09T17:57:22.406 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013850099999999999 2024-02-09T17:57:22.406 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.406 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.406 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:57:22.406 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.810399+0000", 2024-02-09T17:57:22.407 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0048859159999999997 2024-02-09T17:57:22.407 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.407 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.407 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:22.407 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.811445+0000", 2024-02-09T17:57:22.407 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0010456949999999999 2024-02-09T17:57:22.407 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.407 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.407 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:22.407 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.994744+0000", 2024-02-09T17:57:22.407 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.18329941699999999 2024-02-09T17:57:22.407 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.407 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.407 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:57:22.407 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.994778+0000", 2024-02-09T17:57:22.407 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.3612000000000003e-05 2024-02-09T17:57:22.408 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.408 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.408 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:57:22.408 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.994802+0000", 2024-02-09T17:57:22.408 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.4372999999999999e-05 2024-02-09T17:57:22.408 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:22.408 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:22.408 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:22.408 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.408 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.408 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.16440.0:64958 7.4 7:26eaf9c4:::benchmark_data_smithi179_135804_object64957:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e677)", 2024-02-09T17:57:22.408 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:50:57.820974+0000", 2024-02-09T17:57:22.408 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 384.46923406100001, 2024-02-09T17:57:22.408 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.17518472700000001, 2024-02-09T17:57:22.409 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:57:22.409 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:57:22.409 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:57:22.409 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.16440", 2024-02-09T17:57:22.409 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:57:22.409 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 64958 2024-02-09T17:57:22.409 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.409 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:57:22.409 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.409 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:57:22.409 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.820974+0000", 2024-02-09T17:57:22.409 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:22.409 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.409 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.409 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:57:22.410 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.820974+0000", 2024-02-09T17:57:22.410 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:22.410 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.410 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.410 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:57:22.410 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.820975+0000", 2024-02-09T17:57:22.410 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.328e-06 2024-02-09T17:57:22.410 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.410 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.410 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:57:22.410 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.821048+0000", 2024-02-09T17:57:22.410 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.2988e-05 2024-02-09T17:57:22.410 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.410 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.410 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:57:22.411 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.821060+0000", 2024-02-09T17:57:22.411 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1438e-05 2024-02-09T17:57:22.411 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.411 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.411 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:57:22.411 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.821068+0000", 2024-02-09T17:57:22.411 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.3310000000000004e-06 2024-02-09T17:57:22.411 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.411 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.411 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:57:22.411 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.821127+0000", 2024-02-09T17:57:22.411 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.8465999999999999e-05 2024-02-09T17:57:22.411 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.411 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.411 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:57:22.412 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.821259+0000", 2024-02-09T17:57:22.412 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013286400000000001 2024-02-09T17:57:22.412 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.412 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.412 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,4", 2024-02-09T17:57:22.412 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.821421+0000", 2024-02-09T17:57:22.412 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00016158399999999999 2024-02-09T17:57:22.412 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.412 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.412 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:57:22.412 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.826247+0000", 2024-02-09T17:57:22.412 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0048264249999999996 2024-02-09T17:57:22.412 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.412 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.412 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:22.413 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.831024+0000", 2024-02-09T17:57:22.413 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.004776443 2024-02-09T17:57:22.413 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.413 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.413 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:22.413 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.996071+0000", 2024-02-09T17:57:22.413 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.16504697900000001 2024-02-09T17:57:22.413 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.413 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.413 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:57:22.413 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.996124+0000", 2024-02-09T17:57:22.413 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3575999999999997e-05 2024-02-09T17:57:22.413 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.413 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.413 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:57:22.414 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:50:57.996159+0000", 2024-02-09T17:57:22.414 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.4304999999999998e-05 2024-02-09T17:57:22.414 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:22.414 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:22.414 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:22.414 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.414 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.414 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.16440.0:164321 7.6 7:6f4e36c1:::benchmark_data_smithi179_135804_object56530:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e746)", 2024-02-09T17:57:22.414 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:53:31.119520+0000", 2024-02-09T17:57:22.414 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 231.170688138, 2024-02-09T17:57:22.414 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20402888799999999, 2024-02-09T17:57:22.414 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:57:22.414 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:57:22.414 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:57:22.414 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.16440", 2024-02-09T17:57:22.414 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:57:22.415 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 164321 2024-02-09T17:57:22.415 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.415 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:57:22.415 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.415 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:57:22.415 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:31.119520+0000", 2024-02-09T17:57:22.415 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:22.415 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.415 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.415 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:57:22.415 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:31.119520+0000", 2024-02-09T17:57:22.415 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:22.415 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.415 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.415 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:57:22.416 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:31.119522+0000", 2024-02-09T17:57:22.416 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.703e-06 2024-02-09T17:57:22.416 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.416 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.416 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:57:22.416 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:31.119529+0000", 2024-02-09T17:57:22.416 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.6650000000000003e-06 2024-02-09T17:57:22.416 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.416 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.416 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:57:22.416 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:31.119545+0000", 2024-02-09T17:57:22.416 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5333999999999999e-05 2024-02-09T17:57:22.416 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.416 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.417 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:57:22.417 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:31.119555+0000", 2024-02-09T17:57:22.417 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.8949999999999993e-06 2024-02-09T17:57:22.417 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.417 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.417 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:57:22.417 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:31.131944+0000", 2024-02-09T17:57:22.417 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.012389555 2024-02-09T17:57:22.417 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.417 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.417 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for rw locks", 2024-02-09T17:57:22.417 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:31.132106+0000", 2024-02-09T17:57:22.417 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00016233600000000001 2024-02-09T17:57:22.417 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.417 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.418 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:57:22.418 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:31.274184+0000", 2024-02-09T17:57:22.418 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.142078077 2024-02-09T17:57:22.418 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.418 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.418 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:57:22.418 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:31.274278+0000", 2024-02-09T17:57:22.418 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.3511000000000006e-05 2024-02-09T17:57:22.418 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.418 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.418 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,7", 2024-02-09T17:57:22.418 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:31.274427+0000", 2024-02-09T17:57:22.418 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000149047 2024-02-09T17:57:22.418 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.418 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.419 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:57:22.419 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:31.296756+0000", 2024-02-09T17:57:22.419 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.022329324000000001 2024-02-09T17:57:22.419 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.419 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.419 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:22.419 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:31.296870+0000", 2024-02-09T17:57:22.419 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00011324600000000001 2024-02-09T17:57:22.419 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.419 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.419 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:22.419 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:31.320158+0000", 2024-02-09T17:57:22.419 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.023288297999999999 2024-02-09T17:57:22.419 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.419 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.420 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:57:22.420 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:31.320226+0000", 2024-02-09T17:57:22.420 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.8573999999999998e-05 2024-02-09T17:57:22.420 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.420 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.420 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:57:22.420 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:31.323549+0000", 2024-02-09T17:57:22.420 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0033223229999999999 2024-02-09T17:57:22.420 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:22.420 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:22.420 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:22.420 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.420 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.420 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.16440.0:182836 7.6 7:623452ae:::benchmark_data_smithi179_135804_object75045:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e749)", 2024-02-09T17:57:22.421 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:53:59.320963+0000", 2024-02-09T17:57:22.421 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 202.96924528, 2024-02-09T17:57:22.421 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 11.242320245, 2024-02-09T17:57:22.421 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:57:22.421 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "reached pg", 2024-02-09T17:57:22.421 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:57:22.421 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.16440", 2024-02-09T17:57:22.421 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:57:22.421 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 182836 2024-02-09T17:57:22.421 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.421 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:57:22.421 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.421 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:57:22.421 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:59.320963+0000", 2024-02-09T17:57:22.421 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:22.422 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.422 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.422 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:57:22.422 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:59.320963+0000", 2024-02-09T17:57:22.422 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:22.422 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.422 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.422 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:57:22.422 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:59.320963+0000", 2024-02-09T17:57:22.422 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.1e-07 2024-02-09T17:57:22.422 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.422 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.422 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:57:22.422 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:59.320967+0000", 2024-02-09T17:57:22.422 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.143e-06 2024-02-09T17:57:22.422 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.423 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.423 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:57:22.423 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:59.320973+0000", 2024-02-09T17:57:22.423 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.1770000000000003e-06 2024-02-09T17:57:22.423 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.423 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.423 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:57:22.423 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:59.320978+0000", 2024-02-09T17:57:22.423 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.6079999999999998e-06 2024-02-09T17:57:22.423 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.423 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.423 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:57:22.423 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:59.329864+0000", 2024-02-09T17:57:22.423 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0088855419999999997 2024-02-09T17:57:22.423 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.424 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.424 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for missing object", 2024-02-09T17:57:22.424 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:53:59.329968+0000", 2024-02-09T17:57:22.424 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000104076 2024-02-09T17:57:22.424 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.424 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.424 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:57:22.424 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:10.563249+0000", 2024-02-09T17:57:22.424 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 11.233281245000001 2024-02-09T17:57:22.424 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.424 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.424 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:57:22.424 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:10.563283+0000", 2024-02-09T17:57:22.424 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.4143999999999997e-05 2024-02-09T17:57:22.424 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:22.425 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:22.425 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:22.425 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.425 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.425 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.16440.0:207749 7.6 7:67e7152a:::benchmark_data_smithi179_135804_object99958:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e768)", 2024-02-09T17:57:22.425 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:54:35.617411+0000", 2024-02-09T17:57:22.425 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 166.67279714599999, 2024-02-09T17:57:22.425 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.127240563, 2024-02-09T17:57:22.425 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:57:22.425 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:57:22.425 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:57:22.425 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.16440", 2024-02-09T17:57:22.425 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:57:22.425 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 207749 2024-02-09T17:57:22.425 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.426 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:57:22.426 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.426 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:57:22.426 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:35.617411+0000", 2024-02-09T17:57:22.426 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:22.426 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.426 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.426 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:57:22.426 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:35.617411+0000", 2024-02-09T17:57:22.426 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:22.426 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.426 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.426 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:57:22.426 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:35.617413+0000", 2024-02-09T17:57:22.426 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.784e-06 2024-02-09T17:57:22.426 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.427 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.427 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:57:22.427 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:35.617420+0000", 2024-02-09T17:57:22.427 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.2030000000000003e-06 2024-02-09T17:57:22.427 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.427 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.427 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:57:22.427 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:35.624582+0000", 2024-02-09T17:57:22.427 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0071616099999999997 2024-02-09T17:57:22.427 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.427 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.427 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:57:22.427 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:35.624591+0000", 2024-02-09T17:57:22.427 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.0399999999999998e-06 2024-02-09T17:57:22.427 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.428 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.428 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:57:22.428 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:35.626216+0000", 2024-02-09T17:57:22.428 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.001625009 2024-02-09T17:57:22.428 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.428 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.428 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for rw locks", 2024-02-09T17:57:22.428 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:35.626308+0000", 2024-02-09T17:57:22.428 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.2748e-05 2024-02-09T17:57:22.428 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.428 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.428 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:57:22.428 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:35.678518+0000", 2024-02-09T17:57:22.428 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.052209401000000003 2024-02-09T17:57:22.429 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.429 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.429 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:57:22.429 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:35.678619+0000", 2024-02-09T17:57:22.429 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000101228 2024-02-09T17:57:22.429 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.429 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.429 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,7", 2024-02-09T17:57:22.429 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:35.678765+0000", 2024-02-09T17:57:22.429 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014620400000000001 2024-02-09T17:57:22.429 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.429 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.429 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:57:22.429 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:35.686369+0000", 2024-02-09T17:57:22.429 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0076035690000000001 2024-02-09T17:57:22.430 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.430 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.430 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:22.430 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:35.702079+0000", 2024-02-09T17:57:22.430 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.015710253 2024-02-09T17:57:22.430 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.430 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.430 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:22.430 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:35.743245+0000", 2024-02-09T17:57:22.430 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.041166139999999997 2024-02-09T17:57:22.430 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.430 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.430 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:57:22.430 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:35.743310+0000", 2024-02-09T17:57:22.430 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.4697999999999996e-05 2024-02-09T17:57:22.431 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.431 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.431 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:57:22.431 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:35.744651+0000", 2024-02-09T17:57:22.431 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.001341676 2024-02-09T17:57:22.431 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:22.431 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:22.431 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:22.431 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.431 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.431 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.16440.0:208979 7.6 7:6a3db65c:::benchmark_data_smithi179_135804_object101188:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e768)", 2024-02-09T17:57:22.431 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:54:37.362049+0000", 2024-02-09T17:57:22.431 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 164.92815907799999, 2024-02-09T17:57:22.431 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.13556977100000001, 2024-02-09T17:57:22.431 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:57:22.431 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:57:22.432 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:57:22.432 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.16440", 2024-02-09T17:57:22.432 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:57:22.432 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 208979 2024-02-09T17:57:22.432 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.432 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:57:22.432 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.432 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:57:22.432 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:37.362049+0000", 2024-02-09T17:57:22.432 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:22.432 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.432 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.432 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:57:22.432 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:37.362049+0000", 2024-02-09T17:57:22.433 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:22.433 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.433 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.433 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:57:22.433 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:37.362050+0000", 2024-02-09T17:57:22.433 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.186e-06 2024-02-09T17:57:22.433 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.433 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.433 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:57:22.433 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:37.362055+0000", 2024-02-09T17:57:22.433 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.0250000000000002e-06 2024-02-09T17:57:22.433 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.433 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.433 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:57:22.433 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:37.362066+0000", 2024-02-09T17:57:22.434 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0584000000000001e-05 2024-02-09T17:57:22.434 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.434 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.434 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:57:22.434 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:37.362073+0000", 2024-02-09T17:57:22.434 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.1269999999999997e-06 2024-02-09T17:57:22.434 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.434 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.434 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:57:22.434 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:37.373675+0000", 2024-02-09T17:57:22.434 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.01160222 2024-02-09T17:57:22.434 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.434 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.434 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for rw locks", 2024-02-09T17:57:22.434 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:37.373804+0000", 2024-02-09T17:57:22.435 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000128892 2024-02-09T17:57:22.435 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.435 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.435 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:57:22.435 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:37.483035+0000", 2024-02-09T17:57:22.435 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.109231129 2024-02-09T17:57:22.435 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.435 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.435 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:57:22.435 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:37.483149+0000", 2024-02-09T17:57:22.435 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000113744 2024-02-09T17:57:22.435 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.435 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.435 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,7", 2024-02-09T17:57:22.435 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:37.483310+0000", 2024-02-09T17:57:22.435 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00016147700000000001 2024-02-09T17:57:22.436 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.436 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.436 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:22.436 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:37.496176+0000", 2024-02-09T17:57:22.436 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.012865143000000001 2024-02-09T17:57:22.436 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.436 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.436 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:22.436 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:37.496222+0000", 2024-02-09T17:57:22.436 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.6094999999999998e-05 2024-02-09T17:57:22.436 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.436 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.436 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:57:22.436 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:37.496238+0000", 2024-02-09T17:57:22.437 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.6314999999999999e-05 2024-02-09T17:57:22.437 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.437 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.437 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:57:22.437 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:37.496292+0000", 2024-02-09T17:57:22.437 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.4313e-05 2024-02-09T17:57:22.437 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.437 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.437 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:57:22.437 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:37.497619+0000", 2024-02-09T17:57:22.437 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.001326521 2024-02-09T17:57:22.437 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:22.437 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:22.437 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:22.437 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.438 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.438 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26305.0:116097 8.e 8:74ccbe31:::benchmark_data_smithi179_148205_object116096:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e819)", 2024-02-09T17:57:22.438 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:56:47.748905+0000", 2024-02-09T17:57:22.438 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 34.541302588999997, 2024-02-09T17:57:22.438 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.217008865, 2024-02-09T17:57:22.438 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:57:22.438 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:57:22.438 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:57:22.438 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26305", 2024-02-09T17:57:22.438 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T17:57:22.438 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 116097 2024-02-09T17:57:22.438 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.438 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:57:22.438 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.438 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:57:22.439 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.748905+0000", 2024-02-09T17:57:22.439 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:22.439 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.439 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.439 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:57:22.439 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.748905+0000", 2024-02-09T17:57:22.439 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:22.439 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.439 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.439 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:57:22.439 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.748907+0000", 2024-02-09T17:57:22.439 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.217e-06 2024-02-09T17:57:22.439 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.439 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.439 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:57:22.440 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.748962+0000", 2024-02-09T17:57:22.440 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.5415999999999999e-05 2024-02-09T17:57:22.440 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.440 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.440 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:57:22.440 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.748975+0000", 2024-02-09T17:57:22.440 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.326e-05 2024-02-09T17:57:22.440 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.440 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.440 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:57:22.440 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.748984+0000", 2024-02-09T17:57:22.440 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.7280000000000001e-06 2024-02-09T17:57:22.440 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.440 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.441 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:57:22.441 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.749029+0000", 2024-02-09T17:57:22.441 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.5045e-05 2024-02-09T17:57:22.441 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.441 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.441 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:57:22.441 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.749174+0000", 2024-02-09T17:57:22.441 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014466199999999999 2024-02-09T17:57:22.441 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.441 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.441 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 2,4", 2024-02-09T17:57:22.441 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.749323+0000", 2024-02-09T17:57:22.441 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014889999999999999 2024-02-09T17:57:22.441 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.441 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.442 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:22.442 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.756474+0000", 2024-02-09T17:57:22.442 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0071517459999999996 2024-02-09T17:57:22.442 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.442 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.442 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:57:22.442 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.756500+0000", 2024-02-09T17:57:22.442 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.5803999999999999e-05 2024-02-09T17:57:22.442 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.442 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.442 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:22.442 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.965826+0000", 2024-02-09T17:57:22.442 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20932557500000001 2024-02-09T17:57:22.442 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.442 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.443 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:57:22.443 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.965872+0000", 2024-02-09T17:57:22.443 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.5720999999999998e-05 2024-02-09T17:57:22.443 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.443 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.443 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:57:22.443 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.965914+0000", 2024-02-09T17:57:22.443 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2790999999999997e-05 2024-02-09T17:57:22.443 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:22.443 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:22.443 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:22.443 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.443 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.443 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26305.0:116161 8.e 8:72a3408f:::benchmark_data_smithi179_148205_object116160:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e819)", 2024-02-09T17:57:22.443 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:56:47.828489+0000", 2024-02-09T17:57:22.444 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 34.461719361999997, 2024-02-09T17:57:22.444 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.14249688999999999, 2024-02-09T17:57:22.444 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:57:22.444 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:57:22.444 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:57:22.444 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26305", 2024-02-09T17:57:22.444 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T17:57:22.444 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 116161 2024-02-09T17:57:22.444 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.444 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:57:22.444 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.444 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:57:22.444 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.828489+0000", 2024-02-09T17:57:22.444 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:22.445 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.445 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.445 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:57:22.445 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.828489+0000", 2024-02-09T17:57:22.445 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:22.445 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.445 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.445 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:57:22.445 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.828490+0000", 2024-02-09T17:57:22.445 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.212e-06 2024-02-09T17:57:22.445 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.445 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.445 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:57:22.445 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.828527+0000", 2024-02-09T17:57:22.445 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.6764000000000002e-05 2024-02-09T17:57:22.446 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.446 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.446 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:57:22.446 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.828540+0000", 2024-02-09T17:57:22.446 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3393e-05 2024-02-09T17:57:22.446 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.446 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.446 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:57:22.446 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.828549+0000", 2024-02-09T17:57:22.446 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.3279999999999995e-06 2024-02-09T17:57:22.446 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.446 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.446 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:57:22.446 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.828602+0000", 2024-02-09T17:57:22.447 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2120000000000002e-05 2024-02-09T17:57:22.447 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.447 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.447 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:57:22.447 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.828717+0000", 2024-02-09T17:57:22.447 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00011510099999999999 2024-02-09T17:57:22.447 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.447 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.447 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 2,4", 2024-02-09T17:57:22.447 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.828817+0000", 2024-02-09T17:57:22.447 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.9947999999999999e-05 2024-02-09T17:57:22.447 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.447 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.447 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:57:22.447 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.831874+0000", 2024-02-09T17:57:22.447 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0030575099999999998 2024-02-09T17:57:22.448 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.448 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.448 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:22.448 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.832050+0000", 2024-02-09T17:57:22.448 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00017553999999999999 2024-02-09T17:57:22.448 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.448 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.448 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:22.448 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.970875+0000", 2024-02-09T17:57:22.448 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.138825536 2024-02-09T17:57:22.448 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.448 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.448 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:57:22.448 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.970942+0000", 2024-02-09T17:57:22.448 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.6546999999999998e-05 2024-02-09T17:57:22.449 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.449 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.449 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:57:22.449 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.970986+0000", 2024-02-09T17:57:22.449 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.3890999999999997e-05 2024-02-09T17:57:22.449 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:22.449 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:22.449 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:22.449 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.449 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.449 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26305.0:116173 8.e 8:7776389d:::benchmark_data_smithi179_148205_object116172:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e819)", 2024-02-09T17:57:22.449 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:56:47.836251+0000", 2024-02-09T17:57:22.449 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 34.453957469000002, 2024-02-09T17:57:22.449 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.13491573700000001, 2024-02-09T17:57:22.449 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:57:22.450 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:57:22.450 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:57:22.450 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26305", 2024-02-09T17:57:22.450 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T17:57:22.450 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 116173 2024-02-09T17:57:22.450 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.450 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:57:22.450 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.450 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:57:22.450 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.836251+0000", 2024-02-09T17:57:22.450 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:22.450 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.450 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.450 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:57:22.450 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.836251+0000", 2024-02-09T17:57:22.450 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:22.451 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.451 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.451 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:57:22.451 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.836252+0000", 2024-02-09T17:57:22.451 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2109999999999999e-06 2024-02-09T17:57:22.451 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.451 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.451 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:57:22.451 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.836298+0000", 2024-02-09T17:57:22.451 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.6226000000000001e-05 2024-02-09T17:57:22.451 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.451 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.451 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:57:22.451 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.836310+0000", 2024-02-09T17:57:22.452 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2221999999999999e-05 2024-02-09T17:57:22.452 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.452 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.452 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:57:22.452 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.836317+0000", 2024-02-09T17:57:22.452 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.72e-06 2024-02-09T17:57:22.452 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.452 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.452 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:57:22.452 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.836364+0000", 2024-02-09T17:57:22.452 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.6582000000000002e-05 2024-02-09T17:57:22.452 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.452 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.452 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:57:22.452 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.836522+0000", 2024-02-09T17:57:22.453 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000158553 2024-02-09T17:57:22.453 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.453 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.453 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 2,4", 2024-02-09T17:57:22.453 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.836667+0000", 2024-02-09T17:57:22.453 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014528699999999999 2024-02-09T17:57:22.453 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.453 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.453 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:57:22.453 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.839800+0000", 2024-02-09T17:57:22.453 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.003132957 2024-02-09T17:57:22.453 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.453 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.453 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:22.454 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.839996+0000", 2024-02-09T17:57:22.454 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000195293 2024-02-09T17:57:22.454 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.454 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.454 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:22.454 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.971058+0000", 2024-02-09T17:57:22.454 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.13106238200000001 2024-02-09T17:57:22.454 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.454 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.454 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:57:22.454 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.971124+0000", 2024-02-09T17:57:22.454 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.5988000000000006e-05 2024-02-09T17:57:22.454 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:22.454 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:22.454 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:57:22.455 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.971166+0000", 2024-02-09T17:57:22.455 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2315999999999999e-05 2024-02-09T17:57:22.455 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:22.455 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:22.455 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:22.455 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:22.455 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:22.455 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:57:22.455 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:57:22.456 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:57:22.485 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:22.510 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:57:22.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:22.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:57:22.576 INFO:teuthology.orchestra.run.smithi099.stdout: marking collection for removal 2024-02-09T17:57:22.576 INFO:teuthology.orchestra.run.smithi099.stdout:setting '_remove' omap key 2024-02-09T17:57:22.576 INFO:teuthology.orchestra.run.smithi099.stdout:finish_remove_pgs 8.8_head removing 8.8 2024-02-09T17:57:22.618 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:57:22.653 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:22 smithi099 ceph-mon[39420]: pgmap v2044: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 6.3 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 938 op/s; 19668/309840 objects degraded (6.348%) 2024-02-09T17:57:22.653 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:22 smithi099 ceph-mon[39420]: 8.b scrub starts 2024-02-09T17:57:22.653 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:22 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 19668/309840 objects degraded (6.348%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:57:22.656 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:22.718 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:57:22.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:22.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:57:22.770 INFO:teuthology.orchestra.run.smithi099.stdout:Remove successful 2024-02-09T17:57:22.825 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:57:22.890 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:22 smithi179 ceph-mon[43975]: pgmap v2044: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 6.3 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 938 op/s; 19668/309840 objects degraded (6.348%) 2024-02-09T17:57:22.890 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:22 smithi179 ceph-mon[43975]: 8.b scrub starts 2024-02-09T17:57:22.890 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:22 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 19668/309840 objects degraded (6.348%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:57:22.892 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:22.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:22 smithi162 ceph-mon[44304]: pgmap v2044: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 6.3 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 938 op/s; 19668/309840 objects degraded (6.348%) 2024-02-09T17:57:22.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:22 smithi162 ceph-mon[44304]: 8.b scrub starts 2024-02-09T17:57:22.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:22 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 19668/309840 objects degraded (6.348%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:57:22.933 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:57:22.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:22.940 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:57:23.041 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:57:23.069 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:23.093 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:57:23.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:23.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:57:23.197 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:57:23.246 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:23.279 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:57:23.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:23.288 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:57:23.389 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:57:23.417 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:23.438 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:57:23.441 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:23.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:57:23.499 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:23 smithi179 ceph-mon[43975]: osd.4 pg[8.b]: reservation requested while still reserved 2024-02-09T17:57:23.500 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:23 smithi179 ceph-mon[43975]: 1.15 scrub starts 2024-02-09T17:57:23.500 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:23 smithi179 ceph-mon[43975]: 8.9 scrub starts 2024-02-09T17:57:23.500 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:23 smithi179 ceph-mon[43975]: 8.b scrub ok 2024-02-09T17:57:23.542 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:57:23.583 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-09T17:57:23.589 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:23.623 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:57:23.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:23.627 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:57:23.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:23 smithi162 ceph-mon[44304]: osd.4 pg[8.b]: reservation requested while still reserved 2024-02-09T17:57:23.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:23 smithi162 ceph-mon[44304]: 1.15 scrub starts 2024-02-09T17:57:23.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:23 smithi162 ceph-mon[44304]: 8.9 scrub starts 2024-02-09T17:57:23.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:23 smithi162 ceph-mon[44304]: 8.b scrub ok 2024-02-09T17:57:23.728 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:57:23.757 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:23.778 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:57:23.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:23.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:57:23.883 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:57:23.911 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:23.933 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:57:23.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:23.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:57:23.952 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:23 smithi099 ceph-mon[39420]: osd.4 pg[8.b]: reservation requested while still reserved 2024-02-09T17:57:23.952 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:23 smithi099 ceph-mon[39420]: 1.15 scrub starts 2024-02-09T17:57:23.952 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:23 smithi099 ceph-mon[39420]: 8.9 scrub starts 2024-02-09T17:57:23.952 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:23 smithi099 ceph-mon[39420]: 8.b scrub ok 2024-02-09T17:57:24.040 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:57:24.067 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:24.091 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:57:24.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:24.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:57:24.197 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:57:24.268 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:24.296 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:57:24.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:24.301 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:57:24.402 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:57:24.433 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:24.508 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:24 smithi179 ceph-mon[43975]: 1.15 scrub ok 2024-02-09T17:57:24.508 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:24 smithi179 ceph-mon[43975]: pgmap v2045: 47 pgs: 1 active+clean+scrubbing, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 6.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 892 op/s; 18838/301842 objects degraded (6.241%) 2024-02-09T17:57:24.508 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:24 smithi179 ceph-mon[43975]: 8.9 scrub ok 2024-02-09T17:57:24.702 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:24 smithi099 ceph-mon[39420]: 1.15 scrub ok 2024-02-09T17:57:24.703 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:24 smithi099 ceph-mon[39420]: pgmap v2045: 47 pgs: 1 active+clean+scrubbing, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 6.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 892 op/s; 18838/301842 objects degraded (6.241%) 2024-02-09T17:57:24.703 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:24 smithi099 ceph-mon[39420]: 8.9 scrub ok 2024-02-09T17:57:24.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:24 smithi162 ceph-mon[44304]: 1.15 scrub ok 2024-02-09T17:57:24.960 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:24 smithi162 ceph-mon[44304]: pgmap v2045: 47 pgs: 1 active+clean+scrubbing, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 6.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 892 op/s; 18838/301842 objects degraded (6.241%) 2024-02-09T17:57:24.960 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:24 smithi162 ceph-mon[44304]: 8.9 scrub ok 2024-02-09T17:57:25.251 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell -n osd.3 --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.8.8.3 2024-02-09T17:57:25.390 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:57:25.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:25.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:57:25.404 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:57:25.405 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:57:25.496 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:57:25.531 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:25.644 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:57:25.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:25.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:57:25.648 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/osd.3/config 2024-02-09T17:57:25.749 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:57:25.784 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:25 smithi099 ceph-mon[39420]: 1.1c scrub starts 2024-02-09T17:57:25.784 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:25 smithi099 ceph-mon[39420]: 1.1c scrub ok 2024-02-09T17:57:25.786 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:25.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:25 smithi179 ceph-mon[43975]: 1.1c scrub starts 2024-02-09T17:57:25.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:25 smithi179 ceph-mon[43975]: 1.1c scrub ok 2024-02-09T17:57:25.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:25 smithi162 ceph-mon[44304]: 1.1c scrub starts 2024-02-09T17:57:25.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:25 smithi162 ceph-mon[44304]: 1.1c scrub ok 2024-02-09T17:57:25.998 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:57:26.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:26.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:57:26.102 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:57:26.143 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:26.167 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:57:26.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:26.173 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:57:26.274 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:57:26.301 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:26.323 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:57:26.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:26.327 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:57:26.428 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:57:26.477 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:26.505 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:57:26.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:26.515 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:57:26.616 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:57:26.662 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:26.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:26 smithi099 ceph-mon[39420]: pgmap v2046: 47 pgs: 2 active+clean+scrubbing, 3 active+undersized+degraded, 5 active+undersized, 37 active+clean; 6.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 920 op/s; 18838/297537 objects degraded (6.331%) 2024-02-09T17:57:26.765 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:57:26.771 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:57:26.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:26.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:57:26.803 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:57:26.831 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:57:26.862 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:57:26.877 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:57:26.884 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:57:26.917 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:26.918 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:57:26.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:26 smithi179 ceph-mon[43975]: pgmap v2046: 47 pgs: 2 active+clean+scrubbing, 3 active+undersized+degraded, 5 active+undersized, 37 active+clean; 6.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 920 op/s; 18838/297537 objects degraded (6.331%) 2024-02-09T17:57:26.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:26 smithi162 ceph-mon[44304]: pgmap v2046: 47 pgs: 2 active+clean+scrubbing, 3 active+undersized+degraded, 5 active+undersized, 37 active+clean; 6.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 920 op/s; 18838/297537 objects degraded (6.331%) 2024-02-09T17:57:26.954 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:57:27.000 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:57:27.033 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:57:27.062 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:57:27.080 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:57:27.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:27.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:57:27.107 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:57:27.107 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:57:27.107 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:57:27.107 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:57:27.107 INFO:teuthology.orchestra.run.smithi099.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-09T17:57:27.107 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:57:27.107 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:57:27.108 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:57:27.108 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:57:27.108 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:57:27.108 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:57:27.108 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:57:27.108 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:57:27.198 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:57:27.283 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:27.325 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:57:27.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:27.331 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:57:27.432 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:57:27.473 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:27.513 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:57:27.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:27.519 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:57:27.537 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:27 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3327112353' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:57:27.537 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:27 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 18838/297537 objects degraded (6.331%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:57:27.620 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:57:27.648 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:27.671 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:57:27.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:27.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:57:27.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:27 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3327112353' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:57:27.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:27 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 18838/297537 objects degraded (6.331%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:57:27.775 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:57:27.811 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:27.853 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:57:27.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:27.863 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:57:27.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:27 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3327112353' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:57:27.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:27 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 18838/297537 objects degraded (6.331%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:57:27.964 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:57:27.994 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:28.016 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:57:28.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:28.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:57:28.122 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:57:28.150 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:28.174 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:57:28.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:28.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:57:28.280 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:57:28.328 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:28.364 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:57:28.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:28.375 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:57:28.476 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:57:28.542 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:28.573 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:57:28.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:28.585 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:57:28.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:28 smithi179 ceph-mon[43975]: pgmap v2047: 47 pgs: 2 active+clean+scrubbing, 3 active+undersized+degraded, 5 active+undersized, 37 active+clean; 5.9 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 832 op/s; 18054/291597 objects degraded (6.191%) 2024-02-09T17:57:28.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:28 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:57:28.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:28 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:57:28.686 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:57:28.747 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:28.796 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:57:28.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:28.808 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:57:28.909 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:57:28.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:28 smithi162 ceph-mon[44304]: pgmap v2047: 47 pgs: 2 active+clean+scrubbing, 3 active+undersized+degraded, 5 active+undersized, 37 active+clean; 5.9 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 832 op/s; 18054/291597 objects degraded (6.191%) 2024-02-09T17:57:28.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:28 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:57:28.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:28 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:57:28.939 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:28.965 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:57:28.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:28.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:57:28.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:28 smithi099 ceph-mon[39420]: pgmap v2047: 47 pgs: 2 active+clean+scrubbing, 3 active+undersized+degraded, 5 active+undersized, 37 active+clean; 5.9 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 832 op/s; 18054/291597 objects degraded (6.191%) 2024-02-09T17:57:28.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:28 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:57:28.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:28 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:57:29.073 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:57:29.127 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:29.174 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:57:29.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:29.183 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:57:29.284 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:57:29.337 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:29.371 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:57:29.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:29.377 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:57:29.478 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:57:29.509 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:29.720 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:57:29.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:29.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:57:29.758 INFO:teuthology.orchestra.run.smithi099.stderr:Creating an OSD daemon form without an OSD FSID value 2024-02-09T17:57:29.825 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:57:29.854 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:29.875 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:57:29.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:29.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:57:29.979 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:57:29.988 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: osdmap e827: 12 total, 11 up, 12 in 2024-02-09T17:57:29.988 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=56,logm=4} crc {auth=1008950465,config=2679774077,health=2486118068,kv=3617071472,logm=2324930738}) 2024-02-09T17:57:29.988 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=362255561}) 2024-02-09T17:57:29.988 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3525439184}) 2024-02-09T17:57:29.988 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=443347420}) 2024-02-09T17:57:29.988 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=186762696}) 2024-02-09T17:57:29.988 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=827224866}) 2024-02-09T17:57:29.988 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1369593453}) 2024-02-09T17:57:29.989 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1654872594}) 2024-02-09T17:57:29.989 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3524462290}) 2024-02-09T17:57:29.989 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4268791596}) 2024-02-09T17:57:29.989 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3795342545}) 2024-02-09T17:57:29.989 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=699952833}) 2024-02-09T17:57:29.989 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1558938610}) 2024-02-09T17:57:29.989 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=462306450}) 2024-02-09T17:57:29.989 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=190232446}) 2024-02-09T17:57:29.989 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2852635266}) 2024-02-09T17:57:29.989 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2289411114}) 2024-02-09T17:57:29.989 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2682676271}) 2024-02-09T17:57:29.989 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2417113247}) 2024-02-09T17:57:29.989 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3929715150}) 2024-02-09T17:57:29.989 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3751127582}) 2024-02-09T17:57:29.990 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2704738656}) 2024-02-09T17:57:29.990 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3745290731}) 2024-02-09T17:57:29.990 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2846596824}) 2024-02-09T17:57:29.990 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2245405383}) 2024-02-09T17:57:29.990 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2365217504}) 2024-02-09T17:57:29.990 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3358551447}) 2024-02-09T17:57:29.990 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2295501097}) 2024-02-09T17:57:29.990 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=960342832}) 2024-02-09T17:57:29.990 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3471338695}) 2024-02-09T17:57:29.990 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2580214025}) 2024-02-09T17:57:29.990 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1830558432}) 2024-02-09T17:57:29.990 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4221318844}) 2024-02-09T17:57:29.990 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1595354161}) 2024-02-09T17:57:29.990 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3515603617}) 2024-02-09T17:57:29.990 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2896387672}) 2024-02-09T17:57:29.991 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2735581928}) 2024-02-09T17:57:29.991 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3825307009}) 2024-02-09T17:57:29.991 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=446580327}) 2024-02-09T17:57:29.991 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2195592347}) 2024-02-09T17:57:29.991 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1697903394}) 2024-02-09T17:57:29.991 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2289368509}) 2024-02-09T17:57:29.991 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3254805643}) 2024-02-09T17:57:29.991 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1532924102}) 2024-02-09T17:57:29.991 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1347976385}) 2024-02-09T17:57:29.991 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=80081959}) 2024-02-09T17:57:29.991 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3577531502}) 2024-02-09T17:57:29.991 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2972872464}) 2024-02-09T17:57:29.991 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2852693351}) 2024-02-09T17:57:29.991 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1642622810}) 2024-02-09T17:57:29.991 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3576606303}) 2024-02-09T17:57:29.992 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=57802564}) 2024-02-09T17:57:29.992 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=453231023}) 2024-02-09T17:57:29.992 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=351402302}) 2024-02-09T17:57:29.992 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1978580590}) 2024-02-09T17:57:29.992 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2359268144}) 2024-02-09T17:57:29.992 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=293998308}) 2024-02-09T17:57:29.992 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1165491957}) 2024-02-09T17:57:29.992 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=50823483}) 2024-02-09T17:57:29.992 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2600437287}) 2024-02-09T17:57:29.992 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1291899047}) 2024-02-09T17:57:29.992 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=692500883}) 2024-02-09T17:57:29.992 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4279179219}) 2024-02-09T17:57:29.992 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=445735787}) 2024-02-09T17:57:29.992 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2816064096}) 2024-02-09T17:57:29.992 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=913257386}) 2024-02-09T17:57:29.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=882128662}) 2024-02-09T17:57:29.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=984663121}) 2024-02-09T17:57:29.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=548239527}) 2024-02-09T17:57:29.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4245115727}) 2024-02-09T17:57:29.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2477902454}) 2024-02-09T17:57:29.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3966451065}) 2024-02-09T17:57:29.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=75974147}) 2024-02-09T17:57:29.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=31,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=9,mon_config_key=36} crc {logm=3060281975,mdsmap=3984425268,mgr=1708625883,mgr_command_descs=708342470,mgr_metadata=670763949,mgrstat=443051121,mon_config_key=2956859279}) 2024-02-09T17:57:29.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1416769658}) 2024-02-09T17:57:29.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3017999775}) 2024-02-09T17:57:29.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=45,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=37} crc {mon_config_key=1629038428,monmap=423865542,osd_metadata=545325961,osd_pg_creating=4112596473,osdmap=2231144161}) 2024-02-09T17:57:29.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=84} crc {osdmap=394159926}) 2024-02-09T17:57:29.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:29 smithi099 ceph-mon[39420]: pgmap v2049: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 5.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 811 op/s; 18054/287139 objects degraded (6.288%) 2024-02-09T17:57:30.014 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:30.036 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:57:30.039 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:30.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:57:30.140 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:57:30.173 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:30.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: osdmap e827: 12 total, 11 up, 12 in 2024-02-09T17:57:30.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=56,logm=4} crc {auth=1008950465,config=2679774077,health=2486118068,kv=3617071472,logm=2324930738}) 2024-02-09T17:57:30.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=362255561}) 2024-02-09T17:57:30.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3525439184}) 2024-02-09T17:57:30.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=443347420}) 2024-02-09T17:57:30.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=186762696}) 2024-02-09T17:57:30.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=827224866}) 2024-02-09T17:57:30.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1369593453}) 2024-02-09T17:57:30.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1654872594}) 2024-02-09T17:57:30.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3524462290}) 2024-02-09T17:57:30.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4268791596}) 2024-02-09T17:57:30.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3795342545}) 2024-02-09T17:57:30.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=699952833}) 2024-02-09T17:57:30.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1558938610}) 2024-02-09T17:57:30.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=462306450}) 2024-02-09T17:57:30.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=190232446}) 2024-02-09T17:57:30.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2852635266}) 2024-02-09T17:57:30.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2289411114}) 2024-02-09T17:57:30.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2682676271}) 2024-02-09T17:57:30.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2417113247}) 2024-02-09T17:57:30.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3929715150}) 2024-02-09T17:57:30.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3751127582}) 2024-02-09T17:57:30.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2704738656}) 2024-02-09T17:57:30.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3745290731}) 2024-02-09T17:57:30.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2846596824}) 2024-02-09T17:57:30.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2245405383}) 2024-02-09T17:57:30.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2365217504}) 2024-02-09T17:57:30.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3358551447}) 2024-02-09T17:57:30.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2295501097}) 2024-02-09T17:57:30.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=960342832}) 2024-02-09T17:57:30.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3471338695}) 2024-02-09T17:57:30.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2580214025}) 2024-02-09T17:57:30.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1830558432}) 2024-02-09T17:57:30.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4221318844}) 2024-02-09T17:57:30.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1595354161}) 2024-02-09T17:57:30.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3515603617}) 2024-02-09T17:57:30.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2896387672}) 2024-02-09T17:57:30.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2735581928}) 2024-02-09T17:57:30.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3825307009}) 2024-02-09T17:57:30.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=446580327}) 2024-02-09T17:57:30.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2195592347}) 2024-02-09T17:57:30.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1697903394}) 2024-02-09T17:57:30.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2289368509}) 2024-02-09T17:57:30.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3254805643}) 2024-02-09T17:57:30.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1532924102}) 2024-02-09T17:57:30.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1347976385}) 2024-02-09T17:57:30.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=80081959}) 2024-02-09T17:57:30.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3577531502}) 2024-02-09T17:57:30.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2972872464}) 2024-02-09T17:57:30.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2852693351}) 2024-02-09T17:57:30.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1642622810}) 2024-02-09T17:57:30.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3576606303}) 2024-02-09T17:57:30.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=57802564}) 2024-02-09T17:57:30.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=453231023}) 2024-02-09T17:57:30.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=351402302}) 2024-02-09T17:57:30.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1978580590}) 2024-02-09T17:57:30.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2359268144}) 2024-02-09T17:57:30.181 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=293998308}) 2024-02-09T17:57:30.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1165491957}) 2024-02-09T17:57:30.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=50823483}) 2024-02-09T17:57:30.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2600437287}) 2024-02-09T17:57:30.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1291899047}) 2024-02-09T17:57:30.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=692500883}) 2024-02-09T17:57:30.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4279179219}) 2024-02-09T17:57:30.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=445735787}) 2024-02-09T17:57:30.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2816064096}) 2024-02-09T17:57:30.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=913257386}) 2024-02-09T17:57:30.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=882128662}) 2024-02-09T17:57:30.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=984663121}) 2024-02-09T17:57:30.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=548239527}) 2024-02-09T17:57:30.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4245115727}) 2024-02-09T17:57:30.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2477902454}) 2024-02-09T17:57:30.182 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3966451065}) 2024-02-09T17:57:30.183 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=75974147}) 2024-02-09T17:57:30.183 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=31,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=9,mon_config_key=36} crc {logm=3060281975,mdsmap=3984425268,mgr=1708625883,mgr_command_descs=708342470,mgr_metadata=670763949,mgrstat=443051121,mon_config_key=2956859279}) 2024-02-09T17:57:30.183 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1416769658}) 2024-02-09T17:57:30.183 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3017999775}) 2024-02-09T17:57:30.183 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=45,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=37} crc {mon_config_key=1629038428,monmap=423865542,osd_metadata=545325961,osd_pg_creating=4112596473,osdmap=2231144161}) 2024-02-09T17:57:30.183 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=84} crc {osdmap=394159926}) 2024-02-09T17:57:30.183 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:29 smithi162 ceph-mon[44304]: pgmap v2049: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 5.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 811 op/s; 18054/287139 objects degraded (6.288%) 2024-02-09T17:57:30.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: osdmap e827: 12 total, 11 up, 12 in 2024-02-09T17:57:30.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=56,logm=4} crc {auth=1008950465,config=2679774077,health=2486118068,kv=3617071472,logm=2324930738}) 2024-02-09T17:57:30.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=362255561}) 2024-02-09T17:57:30.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3525439184}) 2024-02-09T17:57:30.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=443347420}) 2024-02-09T17:57:30.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=186762696}) 2024-02-09T17:57:30.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=827224866}) 2024-02-09T17:57:30.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1369593453}) 2024-02-09T17:57:30.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1654872594}) 2024-02-09T17:57:30.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3524462290}) 2024-02-09T17:57:30.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4268791596}) 2024-02-09T17:57:30.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3795342545}) 2024-02-09T17:57:30.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=699952833}) 2024-02-09T17:57:30.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1558938610}) 2024-02-09T17:57:30.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=462306450}) 2024-02-09T17:57:30.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=190232446}) 2024-02-09T17:57:30.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2852635266}) 2024-02-09T17:57:30.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2289411114}) 2024-02-09T17:57:30.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2682676271}) 2024-02-09T17:57:30.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2417113247}) 2024-02-09T17:57:30.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3929715150}) 2024-02-09T17:57:30.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3751127582}) 2024-02-09T17:57:30.426 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2704738656}) 2024-02-09T17:57:30.426 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3745290731}) 2024-02-09T17:57:30.426 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2846596824}) 2024-02-09T17:57:30.426 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2245405383}) 2024-02-09T17:57:30.426 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2365217504}) 2024-02-09T17:57:30.426 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3358551447}) 2024-02-09T17:57:30.426 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2295501097}) 2024-02-09T17:57:30.426 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=960342832}) 2024-02-09T17:57:30.426 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3471338695}) 2024-02-09T17:57:30.426 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2580214025}) 2024-02-09T17:57:30.426 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1830558432}) 2024-02-09T17:57:30.426 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4221318844}) 2024-02-09T17:57:30.426 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1595354161}) 2024-02-09T17:57:30.427 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3515603617}) 2024-02-09T17:57:30.427 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2896387672}) 2024-02-09T17:57:30.427 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2735581928}) 2024-02-09T17:57:30.427 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3825307009}) 2024-02-09T17:57:30.427 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=446580327}) 2024-02-09T17:57:30.427 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2195592347}) 2024-02-09T17:57:30.427 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1697903394}) 2024-02-09T17:57:30.427 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2289368509}) 2024-02-09T17:57:30.427 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3254805643}) 2024-02-09T17:57:30.427 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1532924102}) 2024-02-09T17:57:30.427 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1347976385}) 2024-02-09T17:57:30.427 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=80081959}) 2024-02-09T17:57:30.427 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3577531502}) 2024-02-09T17:57:30.428 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2972872464}) 2024-02-09T17:57:30.428 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2852693351}) 2024-02-09T17:57:30.428 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1642622810}) 2024-02-09T17:57:30.428 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3576606303}) 2024-02-09T17:57:30.428 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=57802564}) 2024-02-09T17:57:30.428 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=453231023}) 2024-02-09T17:57:30.428 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=351402302}) 2024-02-09T17:57:30.428 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1978580590}) 2024-02-09T17:57:30.428 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2359268144}) 2024-02-09T17:57:30.428 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=293998308}) 2024-02-09T17:57:30.428 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1165491957}) 2024-02-09T17:57:30.428 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=50823483}) 2024-02-09T17:57:30.428 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2600437287}) 2024-02-09T17:57:30.428 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1291899047}) 2024-02-09T17:57:30.429 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=692500883}) 2024-02-09T17:57:30.429 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4279179219}) 2024-02-09T17:57:30.429 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=445735787}) 2024-02-09T17:57:30.429 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2816064096}) 2024-02-09T17:57:30.429 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=913257386}) 2024-02-09T17:57:30.429 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=882128662}) 2024-02-09T17:57:30.429 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=984663121}) 2024-02-09T17:57:30.429 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=548239527}) 2024-02-09T17:57:30.429 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4245115727}) 2024-02-09T17:57:30.429 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2477902454}) 2024-02-09T17:57:30.429 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3966451065}) 2024-02-09T17:57:30.430 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=75974147}) 2024-02-09T17:57:30.430 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=31,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=9,mon_config_key=36} crc {logm=3060281975,mdsmap=3984425268,mgr=1708625883,mgr_command_descs=708342470,mgr_metadata=670763949,mgrstat=443051121,mon_config_key=2956859279}) 2024-02-09T17:57:30.430 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1416769658}) 2024-02-09T17:57:30.430 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3017999775}) 2024-02-09T17:57:30.430 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=45,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=37} crc {mon_config_key=1629038428,monmap=423865542,osd_metadata=545325961,osd_pg_creating=4112596473,osdmap=2231144161}) 2024-02-09T17:57:30.430 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=84} crc {osdmap=394159926}) 2024-02-09T17:57:30.430 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:29 smithi179 ceph-mon[43975]: pgmap v2049: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 5.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 811 op/s; 18054/287139 objects degraded (6.288%) 2024-02-09T17:57:30.645 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:57:30.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:30.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:57:30.749 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:57:30.780 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:30.807 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:57:30.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:30.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:57:30.911 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:57:30.942 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:30.963 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:57:30.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:30.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:57:31.071 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:57:31.123 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:31.124 INFO:teuthology.orchestra.run.smithi099.stdout:get_pg_num_history pg_num_history pg_num_history(e814 pg_nums {1={514=31},7={661=16},8={785=16}} deleted_pools 780,7) 2024-02-09T17:57:31.124 INFO:teuthology.orchestra.run.smithi099.stdout:Importing pgid 8.8 2024-02-09T17:57:31.130 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10004783:::benchmark_data_smithi179_148205_object16613:head# 2024-02-09T17:57:31.130 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.132 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1000de04:::benchmark_data_smithi179_148205_object3158:head# 2024-02-09T17:57:31.132 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.132 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1001065b:::benchmark_data_smithi179_148205_object2013:head# 2024-02-09T17:57:31.132 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.134 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:100124c5:::benchmark_data_smithi179_148205_object16676:head# 2024-02-09T17:57:31.134 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.134 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1001d046:::benchmark_data_smithi179_148205_object91041:head# 2024-02-09T17:57:31.134 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.134 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1001d6b6:::benchmark_data_smithi179_148205_object34114:head# 2024-02-09T17:57:31.134 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.134 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1001e676:::benchmark_data_smithi179_148205_object27293:head# 2024-02-09T17:57:31.134 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.134 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1002eb7a:::benchmark_data_smithi179_148205_object50076:head# 2024-02-09T17:57:31.134 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.134 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:100323ca:::benchmark_data_smithi179_148205_object79498:head# 2024-02-09T17:57:31.134 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.135 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1003442b:::benchmark_data_smithi179_148205_object76905:head# 2024-02-09T17:57:31.135 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.135 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10037eae:::benchmark_data_smithi179_148205_object73374:head# 2024-02-09T17:57:31.135 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.135 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1003dd15:::benchmark_data_smithi179_148205_object28408:head# 2024-02-09T17:57:31.136 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.136 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10040b3f:::benchmark_data_smithi179_148205_object57136:head# 2024-02-09T17:57:31.136 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.138 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1004f176:::benchmark_data_smithi179_148205_object95899:head# 2024-02-09T17:57:31.138 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.138 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1005d34f:::benchmark_data_smithi179_148205_object60589:head# 2024-02-09T17:57:31.138 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.139 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:100653e4:::benchmark_data_smithi179_148205_object71293:head# 2024-02-09T17:57:31.139 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.139 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10069535:::benchmark_data_smithi179_148205_object2813:head# 2024-02-09T17:57:31.139 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.140 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1006f8d4:::benchmark_data_smithi179_148205_object104236:head# 2024-02-09T17:57:31.140 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.140 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1007f142:::benchmark_data_smithi179_148205_object78765:head# 2024-02-09T17:57:31.140 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.140 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10091873:::benchmark_data_smithi179_148205_object1283:head# 2024-02-09T17:57:31.140 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.141 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10092b16:::benchmark_data_smithi179_148205_object66859:head# 2024-02-09T17:57:31.141 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.141 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1009fae6:::benchmark_data_smithi179_148205_object8034:head# 2024-02-09T17:57:31.141 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.141 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:100a9dec:::benchmark_data_smithi179_148205_object58994:head# 2024-02-09T17:57:31.141 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.142 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:100aa9a6:::benchmark_data_smithi179_148205_object26963:head# 2024-02-09T17:57:31.142 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.142 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:100ad409:::benchmark_data_smithi179_148205_object67709:head# 2024-02-09T17:57:31.142 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.142 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:100bc17e:::benchmark_data_smithi179_148205_object24850:head# 2024-02-09T17:57:31.142 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.146 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:100bf449:::benchmark_data_smithi179_148205_object70834:head# 2024-02-09T17:57:31.146 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.148 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:100d880b:::benchmark_data_smithi179_148205_object56082:head# 2024-02-09T17:57:31.148 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.152 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:100eb35f:::benchmark_data_smithi179_148205_object53188:head# 2024-02-09T17:57:31.152 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.152 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:100f07ed:::benchmark_data_smithi179_148205_object85245:head# 2024-02-09T17:57:31.152 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.153 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:100f6489:::benchmark_data_smithi179_148205_object68676:head# 2024-02-09T17:57:31.153 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.159 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:100f9034:::benchmark_data_smithi179_148205_object27839:head# 2024-02-09T17:57:31.159 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.160 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:101117ef:::benchmark_data_smithi179_148205_object109673:head# 2024-02-09T17:57:31.160 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.160 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1011b636:::benchmark_data_smithi179_148205_object83822:head# 2024-02-09T17:57:31.160 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.161 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1012b51a:::benchmark_data_smithi179_148205_object50843:head# 2024-02-09T17:57:31.161 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.161 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1012c2b1:::benchmark_data_smithi179_148205_object109741:head# 2024-02-09T17:57:31.161 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.162 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10141a16:::benchmark_data_smithi179_148205_object25510:head# 2024-02-09T17:57:31.162 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.162 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10144508:::benchmark_data_smithi179_148205_object18840:head# 2024-02-09T17:57:31.162 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.162 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1014e8f4:::benchmark_data_smithi179_148205_object7068:head# 2024-02-09T17:57:31.162 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.163 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10162356:::benchmark_data_smithi179_148205_object105892:head# 2024-02-09T17:57:31.163 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.163 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1016a829:::benchmark_data_smithi179_148205_object7573:head# 2024-02-09T17:57:31.163 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.163 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1016b6b7:::benchmark_data_smithi179_148205_object65338:head# 2024-02-09T17:57:31.163 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.163 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10176357:::benchmark_data_smithi179_148205_object22758:head# 2024-02-09T17:57:31.164 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.164 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10191fac:::benchmark_data_smithi179_148205_object82356:head# 2024-02-09T17:57:31.164 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.164 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:101a336a:::benchmark_data_smithi179_148205_object58970:head# 2024-02-09T17:57:31.164 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.165 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:101bdd4f:::benchmark_data_smithi179_148205_object46665:head# 2024-02-09T17:57:31.165 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.165 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:101c5fe7:::benchmark_data_smithi179_148205_object70054:head# 2024-02-09T17:57:31.165 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.165 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:101cab72:::benchmark_data_smithi179_148205_object29448:head# 2024-02-09T17:57:31.165 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.166 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:101cfbd8:::benchmark_data_smithi179_148205_object23037:head# 2024-02-09T17:57:31.166 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.166 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:101d55b4:::benchmark_data_smithi179_148205_object104880:head# 2024-02-09T17:57:31.166 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.166 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:101e8e09:::benchmark_data_smithi179_148205_object9173:head# 2024-02-09T17:57:31.167 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.167 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:101fb866:::benchmark_data_smithi179_148205_object60498:head# 2024-02-09T17:57:31.167 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.167 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:101fc0d2:::benchmark_data_smithi179_148205_object23908:head# 2024-02-09T17:57:31.168 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.168 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:101fe793:::benchmark_data_smithi179_148205_object74821:head# 2024-02-09T17:57:31.168 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.168 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10210286:::benchmark_data_smithi179_148205_object24991:head# 2024-02-09T17:57:31.168 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.168 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10222c8e:::benchmark_data_smithi179_148205_object100270:head# 2024-02-09T17:57:31.168 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.169 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1022584b:::benchmark_data_smithi179_148205_object60307:head# 2024-02-09T17:57:31.169 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.169 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1022bbd8:::benchmark_data_smithi179_148205_object4322:head# 2024-02-09T17:57:31.169 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.169 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1022d752:::benchmark_data_smithi179_148205_object95520:head# 2024-02-09T17:57:31.169 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.169 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1022de95:::benchmark_data_smithi179_148205_object1005:head# 2024-02-09T17:57:31.169 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.170 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1022ee2c:::benchmark_data_smithi179_148205_object69312:head# 2024-02-09T17:57:31.170 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.170 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1024f965:::benchmark_data_smithi179_148205_object56847:head# 2024-02-09T17:57:31.170 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.171 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1026beef:::benchmark_data_smithi179_148205_object55363:head# 2024-02-09T17:57:31.171 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.171 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:102777dc:::benchmark_data_smithi179_148205_object32244:head# 2024-02-09T17:57:31.171 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.171 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1027eba9:::benchmark_data_smithi179_148205_object108510:head# 2024-02-09T17:57:31.171 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.172 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1028a1ce:::benchmark_data_smithi179_148205_object87474:head# 2024-02-09T17:57:31.172 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.172 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1028a347:::benchmark_data_smithi179_148205_object9557:head# 2024-02-09T17:57:31.172 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.173 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1029b40e:::benchmark_data_smithi179_148205_object5067:head# 2024-02-09T17:57:31.173 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.173 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1029f6e2:::benchmark_data_smithi179_148205_object92328:head# 2024-02-09T17:57:31.173 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.173 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:102a04d0:::benchmark_data_smithi179_148205_object66697:head# 2024-02-09T17:57:31.173 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.173 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:102a40c5:::benchmark_data_smithi179_148205_object79668:head# 2024-02-09T17:57:31.173 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.174 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:102b18b6:::benchmark_data_smithi179_148205_object66935:head# 2024-02-09T17:57:31.174 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.174 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:102b56a7:::benchmark_data_smithi179_148205_object62012:head# 2024-02-09T17:57:31.174 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.174 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:102c6c37:::benchmark_data_smithi179_148205_object80291:head# 2024-02-09T17:57:31.174 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.174 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:102c752e:::benchmark_data_smithi179_148205_object73103:head# 2024-02-09T17:57:31.174 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.175 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:102d4a7b:::benchmark_data_smithi179_148205_object81704:head# 2024-02-09T17:57:31.175 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.175 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:102ea5da:::benchmark_data_smithi179_148205_object51319:head# 2024-02-09T17:57:31.175 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.175 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:102edfb6:::benchmark_data_smithi179_148205_object32950:head# 2024-02-09T17:57:31.175 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.176 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:102ef2c1:::benchmark_data_smithi179_148205_object39771:head# 2024-02-09T17:57:31.176 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.176 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:102fa2cb:::benchmark_data_smithi179_148205_object16628:head# 2024-02-09T17:57:31.176 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.177 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1031e579:::benchmark_data_smithi179_148205_object18064:head# 2024-02-09T17:57:31.177 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.177 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10325019:::benchmark_data_smithi179_148205_object41333:head# 2024-02-09T17:57:31.177 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:30 smithi162 ceph-mon[44304]: 8.d scrub starts 2024-02-09T17:57:31.178 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10339215:::benchmark_data_smithi179_148205_object67260:head# 2024-02-09T17:57:31.178 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.178 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10347cb3:::benchmark_data_smithi179_148205_object61276:head# 2024-02-09T17:57:31.178 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.179 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1034c57e:::benchmark_data_smithi179_148205_object108915:head# 2024-02-09T17:57:31.179 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.179 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1035939c:::benchmark_data_smithi179_148205_object20019:head# 2024-02-09T17:57:31.179 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.179 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10359e88:::benchmark_data_smithi179_148205_object1776:head# 2024-02-09T17:57:31.179 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.179 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1035cc6d:::benchmark_data_smithi179_148205_object87594:head# 2024-02-09T17:57:31.179 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.180 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1036af69:::benchmark_data_smithi179_148205_object37875:head# 2024-02-09T17:57:31.180 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.180 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10374f61:::benchmark_data_smithi179_148205_object73757:head# 2024-02-09T17:57:31.180 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.180 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1037d519:::benchmark_data_smithi179_148205_object75655:head# 2024-02-09T17:57:31.180 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.181 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:103865f5:::benchmark_data_smithi179_148205_object90685:head# 2024-02-09T17:57:31.181 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.181 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1038aeca:::benchmark_data_smithi179_148205_object30801:head# 2024-02-09T17:57:31.181 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.181 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1038d441:::benchmark_data_smithi179_148205_object83824:head# 2024-02-09T17:57:31.181 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.182 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10392df8:::benchmark_data_smithi179_148205_object78758:head# 2024-02-09T17:57:31.182 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.182 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1039d25c:::benchmark_data_smithi179_148205_object100148:head# 2024-02-09T17:57:31.182 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.182 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1039ea26:::benchmark_data_smithi179_148205_object11149:head# 2024-02-09T17:57:31.182 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.182 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:103a878a:::benchmark_data_smithi179_148205_object56688:head# 2024-02-09T17:57:31.182 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.182 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:103a9213:::benchmark_data_smithi179_148205_object7825:head# 2024-02-09T17:57:31.182 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.183 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:103b696d:::benchmark_data_smithi179_148205_object23459:head# 2024-02-09T17:57:31.183 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.183 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:103b8264:::benchmark_data_smithi179_148205_object4343:head# 2024-02-09T17:57:31.183 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.183 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:103c9e21:::benchmark_data_smithi179_148205_object37100:head# 2024-02-09T17:57:31.183 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.184 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:103ce2e3:::benchmark_data_smithi179_148205_object14788:head# 2024-02-09T17:57:31.184 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.184 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:103cee49:::benchmark_data_smithi179_148205_object61480:head# 2024-02-09T17:57:31.184 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.185 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:103d4a90:::benchmark_data_smithi179_148205_object25633:head# 2024-02-09T17:57:31.185 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.185 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:103ef0d0:::benchmark_data_smithi179_148205_object32666:head# 2024-02-09T17:57:31.185 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.185 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:103fd84b:::benchmark_data_smithi179_148205_object12285:head# 2024-02-09T17:57:31.185 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.186 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:104017bd:::benchmark_data_smithi179_148205_object56716:head# 2024-02-09T17:57:31.186 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.186 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10411776:::benchmark_data_smithi179_148205_object24118:head# 2024-02-09T17:57:31.186 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.186 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10424a51:::benchmark_data_smithi179_148205_object9427:head# 2024-02-09T17:57:31.186 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.186 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10427451:::benchmark_data_smithi179_148205_object52220:head# 2024-02-09T17:57:31.186 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.186 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1042a47f:::benchmark_data_smithi179_148205_object65893:head# 2024-02-09T17:57:31.186 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.187 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10430f7f:::benchmark_data_smithi179_148205_object61983:head# 2024-02-09T17:57:31.187 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.187 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1043c57f:::benchmark_data_smithi179_148205_object103854:head# 2024-02-09T17:57:31.187 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.187 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10447c3a:::benchmark_data_smithi179_148205_object82266:head# 2024-02-09T17:57:31.188 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.188 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:104570cd:::benchmark_data_smithi179_148205_object74631:head# 2024-02-09T17:57:31.188 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.188 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1045d2df:::benchmark_data_smithi179_148205_object27264:head# 2024-02-09T17:57:31.188 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.188 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10461e34:::benchmark_data_smithi179_148205_object453:head# 2024-02-09T17:57:31.189 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.189 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1046e4f3:::benchmark_data_smithi179_148205_object41531:head# 2024-02-09T17:57:31.189 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.193 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:104747db:::benchmark_data_smithi179_148205_object73802:head# 2024-02-09T17:57:31.193 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.193 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:104753b7:::benchmark_data_smithi179_148205_object53745:head# 2024-02-09T17:57:31.194 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.194 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10482bd4:::benchmark_data_smithi179_148205_object102188:head# 2024-02-09T17:57:31.194 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.194 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:104891ef:::benchmark_data_smithi179_148205_object106174:head# 2024-02-09T17:57:31.194 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.194 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1048ced1:::benchmark_data_smithi179_148205_object2651:head# 2024-02-09T17:57:31.194 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.195 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:104920f1:::benchmark_data_smithi179_148205_object71269:head# 2024-02-09T17:57:31.195 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.195 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10496222:::benchmark_data_smithi179_148205_object84063:head# 2024-02-09T17:57:31.195 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.195 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:104972d0:::benchmark_data_smithi179_148205_object70456:head# 2024-02-09T17:57:31.195 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.195 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:104982b6:::benchmark_data_smithi179_148205_object91837:head# 2024-02-09T17:57:31.195 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.196 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1049f39b:::benchmark_data_smithi179_148205_object72988:head# 2024-02-09T17:57:31.196 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.196 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:104b5e63:::benchmark_data_smithi179_148205_object48336:head# 2024-02-09T17:57:31.196 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.196 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:104bca89:::benchmark_data_smithi179_148205_object431:head# 2024-02-09T17:57:31.196 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.197 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:104c313d:::benchmark_data_smithi179_148205_object85967:head# 2024-02-09T17:57:31.197 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.199 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:104c50d6:::benchmark_data_smithi179_148205_object87166:head# 2024-02-09T17:57:31.199 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.203 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:104cf713:::benchmark_data_smithi179_148205_object39444:head# 2024-02-09T17:57:31.203 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.203 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1050a65a:::benchmark_data_smithi179_148205_object43010:head# 2024-02-09T17:57:31.203 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.204 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1052518f:::benchmark_data_smithi179_148205_object9822:head# 2024-02-09T17:57:31.204 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.204 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10525ce5:::benchmark_data_smithi179_148205_object46551:head# 2024-02-09T17:57:31.204 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.204 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10529835:::benchmark_data_smithi179_148205_object46770:head# 2024-02-09T17:57:31.204 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.205 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1053daa2:::benchmark_data_smithi179_148205_object40058:head# 2024-02-09T17:57:31.205 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.205 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:105419c1:::benchmark_data_smithi179_148205_object2883:head# 2024-02-09T17:57:31.205 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.205 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1054c604:::benchmark_data_smithi179_148205_object64401:head# 2024-02-09T17:57:31.205 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.205 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1054e742:::benchmark_data_smithi179_148205_object71580:head# 2024-02-09T17:57:31.206 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.206 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10555fc9:::benchmark_data_smithi179_148205_object20681:head# 2024-02-09T17:57:31.206 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.206 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1055f033:::benchmark_data_smithi179_148205_object85593:head# 2024-02-09T17:57:31.206 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.206 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1055f5c5:::benchmark_data_smithi179_148205_object99768:head# 2024-02-09T17:57:31.206 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.209 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10567d14:::benchmark_data_smithi179_148205_object9973:head# 2024-02-09T17:57:31.209 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.209 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10568111:::benchmark_data_smithi179_148205_object90527:head# 2024-02-09T17:57:31.209 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.210 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1056f1aa:::benchmark_data_smithi179_148205_object52615:head# 2024-02-09T17:57:31.210 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.210 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1057dacd:::benchmark_data_smithi179_148205_object103853:head# 2024-02-09T17:57:31.210 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.210 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:105ac0da:::benchmark_data_smithi179_148205_object76012:head# 2024-02-09T17:57:31.210 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.211 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:105b27ef:::benchmark_data_smithi179_148205_object66053:head# 2024-02-09T17:57:31.211 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.212 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:105b6259:::benchmark_data_smithi179_148205_object46670:head# 2024-02-09T17:57:31.212 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.212 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:105bd60f:::benchmark_data_smithi179_148205_object68809:head# 2024-02-09T17:57:31.213 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.213 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:105c013f:::benchmark_data_smithi179_148205_object17747:head# 2024-02-09T17:57:31.213 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.213 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:105c499b:::benchmark_data_smithi179_148205_object106921:head# 2024-02-09T17:57:31.213 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.213 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:105ce8b3:::benchmark_data_smithi179_148205_object48425:head# 2024-02-09T17:57:31.213 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.213 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:105d0f6d:::benchmark_data_smithi179_148205_object21913:head# 2024-02-09T17:57:31.213 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.213 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:105d725c:::benchmark_data_smithi179_148205_object13775:head# 2024-02-09T17:57:31.213 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.213 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:105ec253:::benchmark_data_smithi179_148205_object108904:head# 2024-02-09T17:57:31.213 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.213 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1060156d:::benchmark_data_smithi179_148205_object38098:head# 2024-02-09T17:57:31.214 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.216 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10607892:::benchmark_data_smithi179_148205_object87694:head# 2024-02-09T17:57:31.216 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.216 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1060d002:::benchmark_data_smithi179_148205_object94000:head# 2024-02-09T17:57:31.216 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.216 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10611667:::benchmark_data_smithi179_148205_object95558:head# 2024-02-09T17:57:31.216 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.216 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10612699:::benchmark_data_smithi179_148205_object89583:head# 2024-02-09T17:57:31.216 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.216 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1061487a:::benchmark_data_smithi179_148205_object22704:head# 2024-02-09T17:57:31.217 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.217 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10616ad3:::benchmark_data_smithi179_148205_object106523:head# 2024-02-09T17:57:31.217 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.217 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1061adb6:::benchmark_data_smithi179_148205_object14373:head# 2024-02-09T17:57:31.217 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.217 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:106368fa:::benchmark_data_smithi179_148205_object33970:head# 2024-02-09T17:57:31.217 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.217 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10637098:::benchmark_data_smithi179_148205_object68275:head# 2024-02-09T17:57:31.217 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.217 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:106381d4:::benchmark_data_smithi179_148205_object46427:head# 2024-02-09T17:57:31.217 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.217 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1063ce4c:::benchmark_data_smithi179_148205_object11998:head# 2024-02-09T17:57:31.217 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.217 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10654a10:::benchmark_data_smithi179_148205_object109359:head# 2024-02-09T17:57:31.218 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.218 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10662cc2:::benchmark_data_smithi179_148205_object54016:head# 2024-02-09T17:57:31.218 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.220 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1066592a:::benchmark_data_smithi179_148205_object4623:head# 2024-02-09T17:57:31.220 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.220 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:106704d1:::benchmark_data_smithi179_148205_object85380:head# 2024-02-09T17:57:31.220 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.220 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10677445:::benchmark_data_smithi179_148205_object66933:head# 2024-02-09T17:57:31.220 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.220 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1068a47b:::benchmark_data_smithi179_148205_object68114:head# 2024-02-09T17:57:31.221 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.221 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:106a42e8:::benchmark_data_smithi179_148205_object65700:head# 2024-02-09T17:57:31.221 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.221 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:106ae7c9:::benchmark_data_smithi179_148205_object50645:head# 2024-02-09T17:57:31.221 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.221 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:106b5997:::benchmark_data_smithi179_148205_object96328:head# 2024-02-09T17:57:31.221 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.221 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:106b5f0a:::benchmark_data_smithi179_148205_object57424:head# 2024-02-09T17:57:31.221 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.221 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:106b8e3e:::benchmark_data_smithi179_148205_object31886:head# 2024-02-09T17:57:31.221 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.221 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:106b8efc:::benchmark_data_smithi179_148205_object73480:head# 2024-02-09T17:57:31.221 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.222 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:106bb4f5:::benchmark_data_smithi179_148205_object49843:head# 2024-02-09T17:57:31.222 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.222 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:106bb8ff:::benchmark_data_smithi179_148205_object57307:head# 2024-02-09T17:57:31.222 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.222 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:106bf104:::benchmark_data_smithi179_148205_object30940:head# 2024-02-09T17:57:31.222 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.223 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:106cc65c:::benchmark_data_smithi179_148205_object74645:head# 2024-02-09T17:57:31.223 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.223 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:106e85de:::benchmark_data_smithi179_148205_object10913:head# 2024-02-09T17:57:31.223 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.223 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:106eefb8:::benchmark_data_smithi179_148205_object52819:head# 2024-02-09T17:57:31.223 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.223 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:106f82ca:::benchmark_data_smithi179_148205_object87056:head# 2024-02-09T17:57:31.223 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.228 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10717cea:::benchmark_data_smithi179_148205_object60180:head# 2024-02-09T17:57:31.228 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.228 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10718e63:::benchmark_data_smithi179_148205_object3263:head# 2024-02-09T17:57:31.228 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.229 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:107218f4:::benchmark_data_smithi179_148205_object12297:head# 2024-02-09T17:57:31.229 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.229 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10721be6:::benchmark_data_smithi179_148205_object81332:head# 2024-02-09T17:57:31.229 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.229 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1072496f:::benchmark_data_smithi179_148205_object60725:head# 2024-02-09T17:57:31.229 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.230 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1072611a:::benchmark_data_smithi179_148205_object5901:head# 2024-02-09T17:57:31.230 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.230 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1072ed65:::benchmark_data_smithi179_148205_object99653:head# 2024-02-09T17:57:31.230 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.230 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1073d06c:::benchmark_data_smithi179_148205_object65377:head# 2024-02-09T17:57:31.230 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.231 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:107413e3:::benchmark_data_smithi179_148205_object73497:head# 2024-02-09T17:57:31.231 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.231 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10749031:::benchmark_data_smithi179_148205_object71156:head# 2024-02-09T17:57:31.231 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.232 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1074e88e:::benchmark_data_smithi179_148205_object50143:head# 2024-02-09T17:57:31.232 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.232 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1075a3f5:::benchmark_data_smithi179_148205_object14269:head# 2024-02-09T17:57:31.232 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.233 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1078155e:::benchmark_data_smithi179_148205_object41120:head# 2024-02-09T17:57:31.233 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.233 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:107a183e:::benchmark_data_smithi179_148205_object92303:head# 2024-02-09T17:57:31.233 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.233 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:107b2273:::benchmark_data_smithi179_148205_object44822:head# 2024-02-09T17:57:31.233 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.233 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:107b45ce:::benchmark_data_smithi179_148205_object75334:head# 2024-02-09T17:57:31.233 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.233 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:107c71ed:::benchmark_data_smithi179_148205_object83786:head# 2024-02-09T17:57:31.234 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.234 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:107c896a:::benchmark_data_smithi179_148205_object7464:head# 2024-02-09T17:57:31.234 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.236 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:107ca97a:::benchmark_data_smithi179_148205_object4984:head# 2024-02-09T17:57:31.236 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.236 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:107cd204:::benchmark_data_smithi179_148205_object35595:head# 2024-02-09T17:57:31.236 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.237 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:107ef565:::benchmark_data_smithi179_148205_object90016:head# 2024-02-09T17:57:31.237 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.237 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:107fd742:::benchmark_data_smithi179_148205_object89218:head# 2024-02-09T17:57:31.237 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.237 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:108007f2:::benchmark_data_smithi179_148205_object28218:head# 2024-02-09T17:57:31.237 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.237 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10803eb6:::benchmark_data_smithi179_148205_object45345:head# 2024-02-09T17:57:31.237 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.237 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10806f6b:::benchmark_data_smithi179_148205_object8734:head# 2024-02-09T17:57:31.237 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.237 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1080ca19:::benchmark_data_smithi179_148205_object51899:head# 2024-02-09T17:57:31.237 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.237 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:108145e7:::benchmark_data_smithi179_148205_object12677:head# 2024-02-09T17:57:31.237 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.238 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10817a65:::benchmark_data_smithi179_148205_object99356:head# 2024-02-09T17:57:31.238 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.238 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1081ad02:::benchmark_data_smithi179_148205_object50075:head# 2024-02-09T17:57:31.238 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.238 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1081c184:::benchmark_data_smithi179_148205_object5756:head# 2024-02-09T17:57:31.238 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.238 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1082abda:::benchmark_data_smithi179_148205_object7240:head# 2024-02-09T17:57:31.238 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.241 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1082e05e:::benchmark_data_smithi179_148205_object96560:head# 2024-02-09T17:57:31.241 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.241 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:108388a6:::benchmark_data_smithi179_148205_object101610:head# 2024-02-09T17:57:31.241 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.241 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10845389:::benchmark_data_smithi179_148205_object26551:head# 2024-02-09T17:57:31.241 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.241 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10853df4:::benchmark_data_smithi179_148205_object51648:head# 2024-02-09T17:57:31.241 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.241 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:108542b7:::benchmark_data_smithi179_148205_object85591:head# 2024-02-09T17:57:31.241 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.241 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10855bf9:::benchmark_data_smithi179_148205_object20833:head# 2024-02-09T17:57:31.241 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.242 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1086e651:::benchmark_data_smithi179_148205_object85837:head# 2024-02-09T17:57:31.242 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.242 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1086fe2e:::benchmark_data_smithi179_148205_object42800:head# 2024-02-09T17:57:31.242 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.242 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10874a23:::benchmark_data_smithi179_148205_object34469:head# 2024-02-09T17:57:31.242 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.242 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1088e4cb:::benchmark_data_smithi179_148205_object11791:head# 2024-02-09T17:57:31.242 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.242 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1089446e:::benchmark_data_smithi179_148205_object19582:head# 2024-02-09T17:57:31.242 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.242 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10895fde:::benchmark_data_smithi179_148205_object61768:head# 2024-02-09T17:57:31.242 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.242 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:108ab7db:::benchmark_data_smithi179_148205_object11376:head# 2024-02-09T17:57:31.242 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.243 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:108c0bad:::benchmark_data_smithi179_148205_object23710:head# 2024-02-09T17:57:31.243 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.243 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:108c76c9:::benchmark_data_smithi179_148205_object70355:head# 2024-02-09T17:57:31.243 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.246 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:108d2575:::benchmark_data_smithi179_148205_object103391:head# 2024-02-09T17:57:31.246 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:30 smithi099 ceph-mon[39420]: 8.d scrub starts 2024-02-09T17:57:31.247 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:108dc0d5:::benchmark_data_smithi179_148205_object11852:head# 2024-02-09T17:57:31.247 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.247 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:108f3ba0:::benchmark_data_smithi179_148205_object18854:head# 2024-02-09T17:57:31.247 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.247 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:108fa92b:::benchmark_data_smithi179_148205_object102341:head# 2024-02-09T17:57:31.248 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.248 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:108ff225:::benchmark_data_smithi179_148205_object92867:head# 2024-02-09T17:57:31.248 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.249 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10900120:::benchmark_data_smithi179_148205_object86251:head# 2024-02-09T17:57:31.249 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.249 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10913344:::benchmark_data_smithi179_148205_object62825:head# 2024-02-09T17:57:31.249 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.250 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1092ea6c:::benchmark_data_smithi179_148205_object53762:head# 2024-02-09T17:57:31.250 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.250 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1092eb7e:::benchmark_data_smithi179_148205_object12260:head# 2024-02-09T17:57:31.250 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.250 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10936ade:::benchmark_data_smithi179_148205_object6103:head# 2024-02-09T17:57:31.250 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.250 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:109475b7:::benchmark_data_smithi179_148205_object58938:head# 2024-02-09T17:57:31.250 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.250 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:109539db:::benchmark_data_smithi179_148205_object66877:head# 2024-02-09T17:57:31.250 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.250 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1097acfa:::benchmark_data_smithi179_148205_object64959:head# 2024-02-09T17:57:31.250 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.252 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1097fdf3:::benchmark_data_smithi179_148205_object96434:head# 2024-02-09T17:57:31.252 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.252 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1098eb7e:::benchmark_data_smithi179_148205_object64398:head# 2024-02-09T17:57:31.252 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.252 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:109a09b5:::benchmark_data_smithi179_148205_object5274:head# 2024-02-09T17:57:31.252 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.252 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:109b3396:::benchmark_data_smithi179_148205_object8418:head# 2024-02-09T17:57:31.252 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.253 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:109b33a3:::benchmark_data_smithi179_148205_object48852:head# 2024-02-09T17:57:31.253 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.253 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:109b56d6:::benchmark_data_smithi179_148205_object83409:head# 2024-02-09T17:57:31.253 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.253 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:109b7044:::benchmark_data_smithi179_148205_object107282:head# 2024-02-09T17:57:31.253 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.253 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:109bdc35:::benchmark_data_smithi179_148205_object38901:head# 2024-02-09T17:57:31.253 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.258 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:109beefd:::benchmark_data_smithi179_148205_object81947:head# 2024-02-09T17:57:31.258 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.258 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:109ca9f5:::benchmark_data_smithi179_148205_object89204:head# 2024-02-09T17:57:31.258 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.258 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:109d6f71:::benchmark_data_smithi179_148205_object29228:head# 2024-02-09T17:57:31.258 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.258 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:109decb3:::benchmark_data_smithi179_148205_object61896:head# 2024-02-09T17:57:31.258 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.258 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:109e920d:::benchmark_data_smithi179_148205_object47277:head# 2024-02-09T17:57:31.259 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.259 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:109f4ea9:::benchmark_data_smithi179_148205_object63301:head# 2024-02-09T17:57:31.259 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.263 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:109f7abe:::benchmark_data_smithi179_148205_object51094:head# 2024-02-09T17:57:31.263 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.263 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10a0181f:::benchmark_data_smithi179_148205_object62732:head# 2024-02-09T17:57:31.263 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.264 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10a03fce:::benchmark_data_smithi179_148205_object51257:head# 2024-02-09T17:57:31.264 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.264 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10a0f889:::benchmark_data_smithi179_148205_object103887:head# 2024-02-09T17:57:31.264 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.264 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10a13615:::benchmark_data_smithi179_148205_object81580:head# 2024-02-09T17:57:31.264 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.264 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10a1aa40:::benchmark_data_smithi179_148205_object58434:head# 2024-02-09T17:57:31.264 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.264 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10a1bc9b:::benchmark_data_smithi179_148205_object89823:head# 2024-02-09T17:57:31.264 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.264 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10a2054c:::benchmark_data_smithi179_148205_object11142:head# 2024-02-09T17:57:31.264 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.264 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10a276ac:::benchmark_data_smithi179_148205_object46498:head# 2024-02-09T17:57:31.264 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.265 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10a2c45b:::benchmark_data_smithi179_148205_object8568:head# 2024-02-09T17:57:31.265 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.265 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10a3dc6b:::benchmark_data_smithi179_148205_object109851:head# 2024-02-09T17:57:31.265 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.265 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10a53152:::benchmark_data_smithi179_148205_object61307:head# 2024-02-09T17:57:31.265 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.265 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10a5bdc6:::benchmark_data_smithi179_148205_object92287:head# 2024-02-09T17:57:31.265 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.265 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10a64b08:::benchmark_data_smithi179_148205_object89429:head# 2024-02-09T17:57:31.265 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.265 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10a693be:::benchmark_data_smithi179_148205_object71264:head# 2024-02-09T17:57:31.265 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.265 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10a6f7ee:::benchmark_data_smithi179_148205_object27129:head# 2024-02-09T17:57:31.266 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.266 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10a7ec68:::benchmark_data_smithi179_148205_object73166:head# 2024-02-09T17:57:31.266 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.267 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10a80091:::benchmark_data_smithi179_148205_object50881:head# 2024-02-09T17:57:31.267 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.268 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10a82969:::benchmark_data_smithi179_148205_object109797:head# 2024-02-09T17:57:31.268 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.268 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10a8a0e5:::benchmark_data_smithi179_148205_object70817:head# 2024-02-09T17:57:31.268 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.268 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10a8abe0:::benchmark_data_smithi179_148205_object82520:head# 2024-02-09T17:57:31.268 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.268 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10abf644:::benchmark_data_smithi179_148205_object13397:head# 2024-02-09T17:57:31.268 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.268 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10acf62d:::benchmark_data_smithi179_148205_object41596:head# 2024-02-09T17:57:31.268 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.268 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10ad7e71:::benchmark_data_smithi179_148205_object48759:head# 2024-02-09T17:57:31.268 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.269 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10ae55fb:::benchmark_data_smithi179_148205_object84430:head# 2024-02-09T17:57:31.269 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.269 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10ae805e:::benchmark_data_smithi179_148205_object109139:head# 2024-02-09T17:57:31.269 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.269 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10aef786:::benchmark_data_smithi179_148205_object42192:head# 2024-02-09T17:57:31.269 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.269 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10b09dd3:::benchmark_data_smithi179_148205_object14230:head# 2024-02-09T17:57:31.269 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.269 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10b10acc:::benchmark_data_smithi179_148205_object105209:head# 2024-02-09T17:57:31.269 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.269 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10b17fe3:::benchmark_data_smithi179_148205_object58130:head# 2024-02-09T17:57:31.269 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.269 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10b1febe:::benchmark_data_smithi179_148205_object103739:head# 2024-02-09T17:57:31.270 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.272 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10b258ab:::benchmark_data_smithi179_148205_object75953:head# 2024-02-09T17:57:31.272 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.272 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10b26864:::benchmark_data_smithi179_148205_object108376:head# 2024-02-09T17:57:31.272 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.272 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10b41c26:::benchmark_data_smithi179_148205_object49922:head# 2024-02-09T17:57:31.272 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.272 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10b4da9f:::benchmark_data_smithi179_148205_object92573:head# 2024-02-09T17:57:31.273 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.273 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10b500f2:::benchmark_data_smithi179_148205_object78854:head# 2024-02-09T17:57:31.273 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.273 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10b5c9c5:::benchmark_data_smithi179_148205_object74274:head# 2024-02-09T17:57:31.273 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.274 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10b6417c:::benchmark_data_smithi179_148205_object91195:head# 2024-02-09T17:57:31.274 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.274 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10b6652b:::benchmark_data_smithi179_148205_object85509:head# 2024-02-09T17:57:31.274 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.274 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10b9eff0:::benchmark_data_smithi179_148205_object31660:head# 2024-02-09T17:57:31.274 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.274 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10ba7484:::benchmark_data_smithi179_148205_object99107:head# 2024-02-09T17:57:31.274 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.275 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10bac11b:::benchmark_data_smithi179_148205_object79808:head# 2024-02-09T17:57:31.275 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.275 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10bb00d6:::benchmark_data_smithi179_148205_object94595:head# 2024-02-09T17:57:31.275 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.275 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10bb6133:::benchmark_data_smithi179_148205_object42599:head# 2024-02-09T17:57:31.275 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.275 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10bbbec6:::benchmark_data_smithi179_148205_object48946:head# 2024-02-09T17:57:31.275 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.275 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10bc2606:::benchmark_data_smithi179_148205_object36601:head# 2024-02-09T17:57:31.275 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.275 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10bc6969:::benchmark_data_smithi179_148205_object32003:head# 2024-02-09T17:57:31.275 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.275 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10bd9e4d:::benchmark_data_smithi179_148205_object85234:head# 2024-02-09T17:57:31.275 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.275 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10be32e9:::benchmark_data_smithi179_148205_object107409:head# 2024-02-09T17:57:31.276 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.276 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10bf54f1:::benchmark_data_smithi179_148205_object65518:head# 2024-02-09T17:57:31.276 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.276 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10bfc914:::benchmark_data_smithi179_148205_object103955:head# 2024-02-09T17:57:31.276 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.276 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10bfefa3:::benchmark_data_smithi179_148205_object106215:head# 2024-02-09T17:57:31.276 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.278 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10c0b970:::benchmark_data_smithi179_148205_object946:head# 2024-02-09T17:57:31.278 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.278 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10c1841a:::benchmark_data_smithi179_148205_object31438:head# 2024-02-09T17:57:31.278 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.278 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10c1b9e6:::benchmark_data_smithi179_148205_object1186:head# 2024-02-09T17:57:31.278 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.278 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10c1f2a1:::benchmark_data_smithi179_148205_object74819:head# 2024-02-09T17:57:31.279 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.279 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10c2381a:::benchmark_data_smithi179_148205_object46721:head# 2024-02-09T17:57:31.279 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.279 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10c4e28f:::benchmark_data_smithi179_148205_object46228:head# 2024-02-09T17:57:31.279 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.279 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10c528d9:::benchmark_data_smithi179_148205_object15017:head# 2024-02-09T17:57:31.279 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.279 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10c56b11:::benchmark_data_smithi179_148205_object75965:head# 2024-02-09T17:57:31.279 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.279 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10c582e7:::benchmark_data_smithi179_148205_object89625:head# 2024-02-09T17:57:31.279 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.279 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10c68a2c:::benchmark_data_smithi179_148205_object66439:head# 2024-02-09T17:57:31.279 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.280 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10c6d490:::benchmark_data_smithi179_148205_object55428:head# 2024-02-09T17:57:31.280 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.280 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10c6eee7:::benchmark_data_smithi179_148205_object97334:head# 2024-02-09T17:57:31.280 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.280 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10c73d46:::benchmark_data_smithi179_148205_object12647:head# 2024-02-09T17:57:31.280 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.280 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10c74bba:::benchmark_data_smithi179_148205_object104084:head# 2024-02-09T17:57:31.280 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.282 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10c93e07:::benchmark_data_smithi179_148205_object107670:head# 2024-02-09T17:57:31.282 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.282 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10c98330:::benchmark_data_smithi179_148205_object11309:head# 2024-02-09T17:57:31.282 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.282 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10c9853a:::benchmark_data_smithi179_148205_object79318:head# 2024-02-09T17:57:31.282 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.282 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10c9b069:::benchmark_data_smithi179_148205_object71956:head# 2024-02-09T17:57:31.282 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.282 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10ca8297:::benchmark_data_smithi179_148205_object99066:head# 2024-02-09T17:57:31.282 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.282 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10ca9390:::benchmark_data_smithi179_148205_object107220:head# 2024-02-09T17:57:31.283 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.283 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10cb2132:::benchmark_data_smithi179_148205_object107015:head# 2024-02-09T17:57:31.283 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.283 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10cb278b:::benchmark_data_smithi179_148205_object17896:head# 2024-02-09T17:57:31.283 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.283 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10cb4d59:::benchmark_data_smithi179_148205_object175:head# 2024-02-09T17:57:31.283 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.283 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10cbc38d:::benchmark_data_smithi179_148205_object30958:head# 2024-02-09T17:57:31.283 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.283 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10cbeeee:::benchmark_data_smithi179_148205_object24190:head# 2024-02-09T17:57:31.283 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.287 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10ccd00d:::benchmark_data_smithi179_148205_object103072:head# 2024-02-09T17:57:31.287 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.287 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10cd08ec:::benchmark_data_smithi179_148205_object96280:head# 2024-02-09T17:57:31.287 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.288 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10cf0924:::benchmark_data_smithi179_148205_object15580:head# 2024-02-09T17:57:31.288 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.288 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10cf933c:::benchmark_data_smithi179_148205_object28430:head# 2024-02-09T17:57:31.288 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.288 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10cffccd:::benchmark_data_smithi179_148205_object43809:head# 2024-02-09T17:57:31.288 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.288 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10d0c9b8:::benchmark_data_smithi179_148205_object58534:head# 2024-02-09T17:57:31.288 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.288 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10d16dc7:::benchmark_data_smithi179_148205_object26671:head# 2024-02-09T17:57:31.288 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.288 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10d206e8:::benchmark_data_smithi179_148205_object77862:head# 2024-02-09T17:57:31.288 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.288 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10d2c566:::benchmark_data_smithi179_148205_object9030:head# 2024-02-09T17:57:31.288 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.289 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10d3180d:::benchmark_data_smithi179_148205_object84426:head# 2024-02-09T17:57:31.289 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.289 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10d3c3db:::benchmark_data_smithi179_148205_object43798:head# 2024-02-09T17:57:31.289 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.289 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10d45002:::benchmark_data_smithi179_148205_object62478:head# 2024-02-09T17:57:31.289 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.289 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10d46f54:::benchmark_data_smithi179_148205_object39360:head# 2024-02-09T17:57:31.289 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.289 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10d4f5f5:::benchmark_data_smithi179_148205_object33551:head# 2024-02-09T17:57:31.289 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.289 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10d65d83:::benchmark_data_smithi179_148205_object9659:head# 2024-02-09T17:57:31.289 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.289 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10d696e5:::benchmark_data_smithi179_148205_object8775:head# 2024-02-09T17:57:31.289 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.289 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10d7dc1a:::benchmark_data_smithi179_148205_object11201:head# 2024-02-09T17:57:31.290 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.290 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10d7f2f5:::benchmark_data_smithi179_148205_object67439:head# 2024-02-09T17:57:31.290 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.291 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10d83354:::benchmark_data_smithi179_148205_object74142:head# 2024-02-09T17:57:31.291 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.292 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10d88c40:::benchmark_data_smithi179_148205_object37985:head# 2024-02-09T17:57:31.292 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.292 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10d89730:::benchmark_data_smithi179_148205_object6547:head# 2024-02-09T17:57:31.292 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.292 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10d8a9a2:::benchmark_data_smithi179_148205_object55556:head# 2024-02-09T17:57:31.292 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.292 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10d8c303:::benchmark_data_smithi179_148205_object106341:head# 2024-02-09T17:57:31.292 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.292 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10d8d1bb:::benchmark_data_smithi179_148205_object50390:head# 2024-02-09T17:57:31.292 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.292 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10d97a2c:::benchmark_data_smithi179_148205_object36737:head# 2024-02-09T17:57:31.292 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.292 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10d995ac:::benchmark_data_smithi179_148205_object21537:head# 2024-02-09T17:57:31.292 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.293 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10dbd862:::benchmark_data_smithi179_148205_object47422:head# 2024-02-09T17:57:31.293 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.293 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10dbeedd:::benchmark_data_smithi179_148205_object47497:head# 2024-02-09T17:57:31.293 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.293 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10dcf7f0:::benchmark_data_smithi179_148205_object21374:head# 2024-02-09T17:57:31.293 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.293 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10dd8857:::benchmark_data_smithi179_148205_object102608:head# 2024-02-09T17:57:31.293 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.293 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10de5f9e:::benchmark_data_smithi179_148205_object60334:head# 2024-02-09T17:57:31.293 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.295 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10deed6f:::benchmark_data_smithi179_148205_object48374:head# 2024-02-09T17:57:31.295 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.296 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10e2273a:::benchmark_data_smithi179_148205_object97683:head# 2024-02-09T17:57:31.296 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.296 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10e2d216:::benchmark_data_smithi179_148205_object91759:head# 2024-02-09T17:57:31.296 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.301 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10e2f130:::benchmark_data_smithi179_148205_object38911:head# 2024-02-09T17:57:31.301 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.301 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10e31acc:::benchmark_data_smithi179_148205_object7311:head# 2024-02-09T17:57:31.301 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.301 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10e3a161:::benchmark_data_smithi179_148205_object53866:head# 2024-02-09T17:57:31.301 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.301 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10e3d7cd:::benchmark_data_smithi179_148205_object68977:head# 2024-02-09T17:57:31.301 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.301 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10e47723:::benchmark_data_smithi179_148205_object38511:head# 2024-02-09T17:57:31.301 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.301 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10e48e5d:::benchmark_data_smithi179_148205_object24423:head# 2024-02-09T17:57:31.301 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.302 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10e62598:::benchmark_data_smithi179_148205_object87030:head# 2024-02-09T17:57:31.302 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.302 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10e6b220:::benchmark_data_smithi179_148205_object46578:head# 2024-02-09T17:57:31.302 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.302 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10e969fd:::benchmark_data_smithi179_148205_object18853:head# 2024-02-09T17:57:31.302 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.302 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10e9d0d4:::benchmark_data_smithi179_148205_object86610:head# 2024-02-09T17:57:31.302 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.302 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10eaa187:::benchmark_data_smithi179_148205_object47208:head# 2024-02-09T17:57:31.302 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.302 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10eaf8b2:::benchmark_data_smithi179_148205_object17374:head# 2024-02-09T17:57:31.302 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.310 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10eb09bb:::benchmark_data_smithi179_148205_object92938:head# 2024-02-09T17:57:31.310 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.313 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10eb82ef:::benchmark_data_smithi179_148205_object9370:head# 2024-02-09T17:57:31.313 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.314 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10ebf49a:::benchmark_data_smithi179_148205_object68116:head# 2024-02-09T17:57:31.314 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.314 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10ec5941:::benchmark_data_smithi179_148205_object101979:head# 2024-02-09T17:57:31.314 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.314 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10ed80d4:::benchmark_data_smithi179_148205_object75820:head# 2024-02-09T17:57:31.314 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.314 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10ede2fc:::benchmark_data_smithi179_148205_object52626:head# 2024-02-09T17:57:31.315 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.315 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10edfed3:::benchmark_data_smithi179_148205_object11605:head# 2024-02-09T17:57:31.315 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.315 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10ee0f2c:::benchmark_data_smithi179_148205_object10546:head# 2024-02-09T17:57:31.315 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.316 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10ee24ad:::benchmark_data_smithi179_148205_object54643:head# 2024-02-09T17:57:31.316 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.316 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10eff587:::benchmark_data_smithi179_148205_object36723:head# 2024-02-09T17:57:31.316 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.316 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10f034f0:::benchmark_data_smithi179_148205_object41661:head# 2024-02-09T17:57:31.316 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.316 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10f1f59e:::benchmark_data_smithi179_148205_object4549:head# 2024-02-09T17:57:31.317 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.317 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:57:31.317 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10f26ea5:::benchmark_data_smithi179_148205_object35311:head# 2024-02-09T17:57:31.317 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.317 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10f28256:::benchmark_data_smithi179_148205_object12193:head# 2024-02-09T17:57:31.317 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.318 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10f2ee23:::benchmark_data_smithi179_148205_object2379:head# 2024-02-09T17:57:31.318 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.320 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10f33731:::benchmark_data_smithi179_148205_object68834:head# 2024-02-09T17:57:31.320 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.320 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10f3571f:::benchmark_data_smithi179_148205_object68715:head# 2024-02-09T17:57:31.320 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.320 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10f3a92c:::benchmark_data_smithi179_148205_object86300:head# 2024-02-09T17:57:31.320 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.320 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10f44e89:::benchmark_data_smithi179_148205_object96901:head# 2024-02-09T17:57:31.320 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.320 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10f4b93c:::benchmark_data_smithi179_148205_object103360:head# 2024-02-09T17:57:31.320 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.320 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10f5774f:::benchmark_data_smithi179_148205_object35005:head# 2024-02-09T17:57:31.321 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.321 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10f59dd0:::benchmark_data_smithi179_148205_object91474:head# 2024-02-09T17:57:31.321 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:31.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:57:31.326 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10f5d9a6:::benchmark_data_smithi179_148205_object43359:head# 2024-02-09T17:57:31.326 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.326 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10f6a394:::benchmark_data_smithi179_148205_object45164:head# 2024-02-09T17:57:31.326 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.326 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10f6c2f3:::benchmark_data_smithi179_148205_object79464:head# 2024-02-09T17:57:31.326 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.327 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10f7040a:::benchmark_data_smithi179_148205_object60583:head# 2024-02-09T17:57:31.327 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.327 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10f7094c:::benchmark_data_smithi179_148205_object95545:head# 2024-02-09T17:57:31.327 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.327 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10f7383c:::benchmark_data_smithi179_148205_object79620:head# 2024-02-09T17:57:31.327 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.327 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10f753e3:::benchmark_data_smithi179_148205_object103727:head# 2024-02-09T17:57:31.327 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.328 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10f78df6:::benchmark_data_smithi179_148205_object45838:head# 2024-02-09T17:57:31.328 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.329 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10f7b068:::benchmark_data_smithi179_148205_object41202:head# 2024-02-09T17:57:31.329 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.329 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10f85199:::benchmark_data_smithi179_148205_object22460:head# 2024-02-09T17:57:31.329 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.329 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10f8f0d2:::benchmark_data_smithi179_148205_object76461:head# 2024-02-09T17:57:31.329 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.329 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10f8fc26:::benchmark_data_smithi179_148205_object75423:head# 2024-02-09T17:57:31.330 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.330 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10f92960:::benchmark_data_smithi179_148205_object67682:head# 2024-02-09T17:57:31.330 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.330 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10f9b176:::benchmark_data_smithi179_148205_object49991:head# 2024-02-09T17:57:31.330 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.330 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10fa1cca:::benchmark_data_smithi179_148205_object40623:head# 2024-02-09T17:57:31.330 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.330 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10fa64d3:::benchmark_data_smithi179_148205_object47594:head# 2024-02-09T17:57:31.330 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.331 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10fb139b:::benchmark_data_smithi179_148205_object63352:head# 2024-02-09T17:57:31.331 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.331 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10fb4ecd:::benchmark_data_smithi179_148205_object105535:head# 2024-02-09T17:57:31.331 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.331 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10fbb9f0:::benchmark_data_smithi179_148205_object25541:head# 2024-02-09T17:57:31.331 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.331 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10fc416d:::benchmark_data_smithi179_148205_object104996:head# 2024-02-09T17:57:31.331 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.331 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10fc5581:::benchmark_data_smithi179_148205_object102747:head# 2024-02-09T17:57:31.331 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.332 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10fca2fa:::benchmark_data_smithi179_148205_object17813:head# 2024-02-09T17:57:31.332 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.332 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10fd1171:::benchmark_data_smithi179_148205_object29594:head# 2024-02-09T17:57:31.332 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.332 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10fd8a09:::benchmark_data_smithi179_148205_object62422:head# 2024-02-09T17:57:31.333 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.333 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10fda1ab:::benchmark_data_smithi179_148205_object19538:head# 2024-02-09T17:57:31.333 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.333 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10fdcbaa:::benchmark_data_smithi179_148205_object79655:head# 2024-02-09T17:57:31.333 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.333 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10feb25f:::benchmark_data_smithi179_148205_object92954:head# 2024-02-09T17:57:31.333 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.333 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10fed228:::benchmark_data_smithi179_148205_object42250:head# 2024-02-09T17:57:31.333 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.334 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10ff4e38:::benchmark_data_smithi179_148205_object48115:head# 2024-02-09T17:57:31.334 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.334 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:10ff5e58:::benchmark_data_smithi179_148205_object83587:head# 2024-02-09T17:57:31.334 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.334 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:110030e2:::benchmark_data_smithi179_148205_object85731:head# 2024-02-09T17:57:31.334 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.334 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:110040aa:::benchmark_data_smithi179_148205_object1860:head# 2024-02-09T17:57:31.335 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.335 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1102145f:::benchmark_data_smithi179_148205_object74788:head# 2024-02-09T17:57:31.335 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.335 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1102b1a3:::benchmark_data_smithi179_148205_object89106:head# 2024-02-09T17:57:31.335 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.335 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1102b59c:::benchmark_data_smithi179_148205_object51707:head# 2024-02-09T17:57:31.335 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.336 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1103177c:::benchmark_data_smithi179_148205_object13217:head# 2024-02-09T17:57:31.336 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.336 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1104e4a4:::benchmark_data_smithi179_148205_object99792:head# 2024-02-09T17:57:31.336 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.336 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11054a86:::benchmark_data_smithi179_148205_object52163:head# 2024-02-09T17:57:31.339 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.339 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11080117:::benchmark_data_smithi179_148205_object60084:head# 2024-02-09T17:57:31.339 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.339 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:110915a4:::benchmark_data_smithi179_148205_object45873:head# 2024-02-09T17:57:31.340 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.340 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:110929c2:::benchmark_data_smithi179_148205_object26587:head# 2024-02-09T17:57:31.340 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.340 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:110aaf56:::benchmark_data_smithi179_148205_object26241:head# 2024-02-09T17:57:31.340 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.340 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:110abba6:::benchmark_data_smithi179_148205_object69986:head# 2024-02-09T17:57:31.340 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.340 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:110abdee:::benchmark_data_smithi179_148205_object47365:head# 2024-02-09T17:57:31.340 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.340 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:110ac61a:::benchmark_data_smithi179_148205_object41873:head# 2024-02-09T17:57:31.340 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.340 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:110af88b:::benchmark_data_smithi179_148205_object55045:head# 2024-02-09T17:57:31.340 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.340 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:110b63aa:::benchmark_data_smithi179_148205_object104426:head# 2024-02-09T17:57:31.341 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.341 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:110c5423:::benchmark_data_smithi179_148205_object57574:head# 2024-02-09T17:57:31.341 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.341 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:110c9009:::benchmark_data_smithi179_148205_object37590:head# 2024-02-09T17:57:31.341 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.341 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:110d3160:::benchmark_data_smithi179_148205_object100147:head# 2024-02-09T17:57:31.341 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.341 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:110d914c:::benchmark_data_smithi179_148205_object74836:head# 2024-02-09T17:57:31.341 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.341 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:110e22f8:::benchmark_data_smithi179_148205_object879:head# 2024-02-09T17:57:31.341 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.341 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:110eca51:::benchmark_data_smithi179_148205_object11238:head# 2024-02-09T17:57:31.341 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.341 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1110810a:::benchmark_data_smithi179_148205_object95155:head# 2024-02-09T17:57:31.342 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.342 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1110969b:::benchmark_data_smithi179_148205_object18078:head# 2024-02-09T17:57:31.342 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.342 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11111dc9:::benchmark_data_smithi179_148205_object63145:head# 2024-02-09T17:57:31.342 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.343 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11112230:::benchmark_data_smithi179_148205_object34225:head# 2024-02-09T17:57:31.344 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.344 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11113154:::benchmark_data_smithi179_148205_object73947:head# 2024-02-09T17:57:31.344 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.344 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:111256dd:::benchmark_data_smithi179_148205_object65351:head# 2024-02-09T17:57:31.344 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.344 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11126eb2:::benchmark_data_smithi179_148205_object62634:head# 2024-02-09T17:57:31.344 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.344 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1112af7e:::benchmark_data_smithi179_148205_object48404:head# 2024-02-09T17:57:31.344 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.344 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1112f18d:::benchmark_data_smithi179_148205_object94826:head# 2024-02-09T17:57:31.344 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.344 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11135edb:::benchmark_data_smithi179_148205_object68291:head# 2024-02-09T17:57:31.344 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.344 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11172859:::benchmark_data_smithi179_148205_object13024:head# 2024-02-09T17:57:31.345 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.345 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:111780e6:::benchmark_data_smithi179_148205_object9940:head# 2024-02-09T17:57:31.345 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.345 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1117e03a:::benchmark_data_smithi179_148205_object80333:head# 2024-02-09T17:57:31.345 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.345 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11182ff4:::benchmark_data_smithi179_148205_object98285:head# 2024-02-09T17:57:31.345 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.345 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11183d46:::benchmark_data_smithi179_148205_object58990:head# 2024-02-09T17:57:31.345 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.346 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1118b073:::benchmark_data_smithi179_148205_object94139:head# 2024-02-09T17:57:31.346 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.346 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1118bacb:::benchmark_data_smithi179_148205_object27743:head# 2024-02-09T17:57:31.346 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.346 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11196763:::benchmark_data_smithi179_148205_object95457:head# 2024-02-09T17:57:31.347 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.347 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11199bcc:::benchmark_data_smithi179_148205_object91887:head# 2024-02-09T17:57:31.347 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.347 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:111acbab:::benchmark_data_smithi179_148205_object97294:head# 2024-02-09T17:57:31.347 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.347 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:111c1f7a:::benchmark_data_smithi179_148205_object99877:head# 2024-02-09T17:57:31.347 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.347 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:111d480d:::benchmark_data_smithi179_148205_object35236:head# 2024-02-09T17:57:31.347 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.347 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:111d502f:::benchmark_data_smithi179_148205_object5297:head# 2024-02-09T17:57:31.347 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.347 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:111e52b4:::benchmark_data_smithi179_148205_object109261:head# 2024-02-09T17:57:31.347 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.347 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:111f01cd:::benchmark_data_smithi179_148205_object27136:head# 2024-02-09T17:57:31.348 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.348 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:111f47e2:::benchmark_data_smithi179_148205_object75531:head# 2024-02-09T17:57:31.348 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.348 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:111f49ab:::benchmark_data_smithi179_148205_object97914:head# 2024-02-09T17:57:31.348 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.349 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:111f6546:::benchmark_data_smithi179_148205_object27237:head# 2024-02-09T17:57:31.349 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.349 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11209ab0:::benchmark_data_smithi179_148205_object80054:head# 2024-02-09T17:57:31.349 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.353 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1122de6a:::benchmark_data_smithi179_148205_object95424:head# 2024-02-09T17:57:31.353 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.353 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1123c76b:::benchmark_data_smithi179_148205_object10802:head# 2024-02-09T17:57:31.353 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.353 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11245da5:::benchmark_data_smithi179_148205_object7862:head# 2024-02-09T17:57:31.353 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.353 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:112531f2:::benchmark_data_smithi179_148205_object44909:head# 2024-02-09T17:57:31.353 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.353 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11256f38:::benchmark_data_smithi179_148205_object55429:head# 2024-02-09T17:57:31.354 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.354 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1125dfda:::benchmark_data_smithi179_148205_object92669:head# 2024-02-09T17:57:31.354 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.354 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1126bad6:::benchmark_data_smithi179_148205_object77060:head# 2024-02-09T17:57:31.354 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.354 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1127971b:::benchmark_data_smithi179_148205_object29405:head# 2024-02-09T17:57:31.354 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.354 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1127f806:::benchmark_data_smithi179_148205_object21388:head# 2024-02-09T17:57:31.354 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.354 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11280cf0:::benchmark_data_smithi179_148205_object58871:head# 2024-02-09T17:57:31.354 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.354 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:112857f9:::benchmark_data_smithi179_148205_object4430:head# 2024-02-09T17:57:31.354 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.354 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1129853b:::benchmark_data_smithi179_148205_object47164:head# 2024-02-09T17:57:31.354 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.355 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:112ac687:::benchmark_data_smithi179_148205_object43825:head# 2024-02-09T17:57:31.355 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.355 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:112b87a9:::benchmark_data_smithi179_148205_object105134:head# 2024-02-09T17:57:31.355 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.355 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:112b995e:::benchmark_data_smithi179_148205_object9841:head# 2024-02-09T17:57:31.355 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.355 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:112bb98f:::benchmark_data_smithi179_148205_object57829:head# 2024-02-09T17:57:31.355 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.355 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:112bd5ac:::benchmark_data_smithi179_148205_object24139:head# 2024-02-09T17:57:31.355 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.355 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:112e8e05:::benchmark_data_smithi179_148205_object102326:head# 2024-02-09T17:57:31.355 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.355 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:112ef252:::benchmark_data_smithi179_148205_object83024:head# 2024-02-09T17:57:31.355 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.355 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11302398:::benchmark_data_smithi179_148205_object27119:head# 2024-02-09T17:57:31.356 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.356 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11303576:::benchmark_data_smithi179_148205_object82284:head# 2024-02-09T17:57:31.356 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.358 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1130be34:::benchmark_data_smithi179_148205_object69610:head# 2024-02-09T17:57:31.358 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.358 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:113150b2:::benchmark_data_smithi179_148205_object21214:head# 2024-02-09T17:57:31.358 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.358 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11316a22:::benchmark_data_smithi179_148205_object28486:head# 2024-02-09T17:57:31.358 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.358 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1132429a:::benchmark_data_smithi179_148205_object335:head# 2024-02-09T17:57:31.358 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.359 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:113260e3:::benchmark_data_smithi179_148205_object41549:head# 2024-02-09T17:57:31.359 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.359 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11336a25:::benchmark_data_smithi179_148205_object102078:head# 2024-02-09T17:57:31.359 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.359 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:113424a9:::benchmark_data_smithi179_148205_object74710:head# 2024-02-09T17:57:31.359 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.359 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1136332e:::benchmark_data_smithi179_148205_object109392:head# 2024-02-09T17:57:31.359 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.359 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:113640b3:::benchmark_data_smithi179_148205_object78581:head# 2024-02-09T17:57:31.359 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.359 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1136f0a3:::benchmark_data_smithi179_148205_object10874:head# 2024-02-09T17:57:31.359 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.359 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:113788a9:::benchmark_data_smithi179_148205_object48944:head# 2024-02-09T17:57:31.359 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.360 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1138ac80:::benchmark_data_smithi179_148205_object78650:head# 2024-02-09T17:57:31.360 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.360 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11394547:::benchmark_data_smithi179_148205_object73370:head# 2024-02-09T17:57:31.360 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.360 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1139cd37:::benchmark_data_smithi179_148205_object84694:head# 2024-02-09T17:57:31.360 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.360 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:113b498c:::benchmark_data_smithi179_148205_object107036:head# 2024-02-09T17:57:31.360 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.360 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:113bfa6e:::benchmark_data_smithi179_148205_object37287:head# 2024-02-09T17:57:31.360 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.360 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:113c089b:::benchmark_data_smithi179_148205_object38384:head# 2024-02-09T17:57:31.360 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.363 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:113c0c37:::benchmark_data_smithi179_148205_object16495:head# 2024-02-09T17:57:31.363 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.363 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:113c5002:::benchmark_data_smithi179_148205_object19624:head# 2024-02-09T17:57:31.363 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.363 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:113db01c:::benchmark_data_smithi179_148205_object29022:head# 2024-02-09T17:57:31.364 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.364 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:113dcc97:::benchmark_data_smithi179_148205_object74672:head# 2024-02-09T17:57:31.364 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.364 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:113e4f44:::benchmark_data_smithi179_148205_object93761:head# 2024-02-09T17:57:31.364 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.364 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:113edea4:::benchmark_data_smithi179_148205_object40147:head# 2024-02-09T17:57:31.364 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.364 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:113edff0:::benchmark_data_smithi179_148205_object15963:head# 2024-02-09T17:57:31.364 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.364 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:113fbd27:::benchmark_data_smithi179_148205_object2863:head# 2024-02-09T17:57:31.364 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.364 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11402bd9:::benchmark_data_smithi179_148205_object93264:head# 2024-02-09T17:57:31.364 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.364 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11405dc6:::benchmark_data_smithi179_148205_object71406:head# 2024-02-09T17:57:31.364 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.365 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11406c00:::benchmark_data_smithi179_148205_object962:head# 2024-02-09T17:57:31.365 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.365 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1140b48b:::benchmark_data_smithi179_148205_object108584:head# 2024-02-09T17:57:31.365 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.365 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1140bcf9:::benchmark_data_smithi179_148205_object74328:head# 2024-02-09T17:57:31.365 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.365 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1140e741:::benchmark_data_smithi179_148205_object15236:head# 2024-02-09T17:57:31.365 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.365 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11416b11:::benchmark_data_smithi179_148205_object2811:head# 2024-02-09T17:57:31.365 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.365 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1141aab7:::benchmark_data_smithi179_148205_object70728:head# 2024-02-09T17:57:31.365 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.365 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1141ce9f:::benchmark_data_smithi179_148205_object16528:head# 2024-02-09T17:57:31.365 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.365 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1142026b:::benchmark_data_smithi179_148205_object47848:head# 2024-02-09T17:57:31.366 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.366 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11424aaf:::benchmark_data_smithi179_148205_object54267:head# 2024-02-09T17:57:31.366 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.366 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:114299d9:::benchmark_data_smithi179_148205_object84632:head# 2024-02-09T17:57:31.366 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.368 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11437ec0:::benchmark_data_smithi179_148205_object50563:head# 2024-02-09T17:57:31.368 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.368 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1145af0c:::benchmark_data_smithi179_148205_object103030:head# 2024-02-09T17:57:31.368 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.369 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1146301f:::benchmark_data_smithi179_148205_object25322:head# 2024-02-09T17:57:31.369 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.369 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11464ae1:::benchmark_data_smithi179_148205_object59460:head# 2024-02-09T17:57:31.369 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.369 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1147e546:::benchmark_data_smithi179_148205_object80954:head# 2024-02-09T17:57:31.369 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.369 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1147fa1f:::benchmark_data_smithi179_148205_object32139:head# 2024-02-09T17:57:31.369 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.369 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1148ec57:::benchmark_data_smithi179_148205_object87304:head# 2024-02-09T17:57:31.369 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.369 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:114975df:::benchmark_data_smithi179_148205_object29763:head# 2024-02-09T17:57:31.369 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.369 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:114acfc1:::benchmark_data_smithi179_148205_object56005:head# 2024-02-09T17:57:31.369 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.370 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:114b5a66:::benchmark_data_smithi179_148205_object2789:head# 2024-02-09T17:57:31.370 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.370 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:114ba30e:::benchmark_data_smithi179_148205_object40538:head# 2024-02-09T17:57:31.370 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.370 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:114bccc3:::benchmark_data_smithi179_148205_object1447:head# 2024-02-09T17:57:31.370 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.370 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:114db412:::benchmark_data_smithi179_148205_object91059:head# 2024-02-09T17:57:31.370 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.373 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:115015d8:::benchmark_data_smithi179_148205_object37000:head# 2024-02-09T17:57:31.373 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.373 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11511e10:::benchmark_data_smithi179_148205_object19405:head# 2024-02-09T17:57:31.373 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.373 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1153430e:::benchmark_data_smithi179_148205_object98031:head# 2024-02-09T17:57:31.373 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.373 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11555a9e:::benchmark_data_smithi179_148205_object79519:head# 2024-02-09T17:57:31.373 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.373 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:115591bc:::benchmark_data_smithi179_148205_object109121:head# 2024-02-09T17:57:31.373 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.373 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1155c14c:::benchmark_data_smithi179_148205_object82909:head# 2024-02-09T17:57:31.373 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.374 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1155c946:::benchmark_data_smithi179_148205_object13815:head# 2024-02-09T17:57:31.374 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.374 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1155e19f:::benchmark_data_smithi179_148205_object14391:head# 2024-02-09T17:57:31.374 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.374 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1156d307:::benchmark_data_smithi179_148205_object58686:head# 2024-02-09T17:57:31.374 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.374 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1156f2c8:::benchmark_data_smithi179_148205_object84625:head# 2024-02-09T17:57:31.374 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.374 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11571bc8:::benchmark_data_smithi179_148205_object55435:head# 2024-02-09T17:57:31.374 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.374 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:115765c2:::benchmark_data_smithi179_148205_object69588:head# 2024-02-09T17:57:31.374 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.374 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:115773ca:::benchmark_data_smithi179_148205_object31411:head# 2024-02-09T17:57:31.374 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.374 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:115903be:::benchmark_data_smithi179_148205_object1838:head# 2024-02-09T17:57:31.375 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.375 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:115958ca:::benchmark_data_smithi179_148205_object43601:head# 2024-02-09T17:57:31.375 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.375 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:115ad276:::benchmark_data_smithi179_148205_object11482:head# 2024-02-09T17:57:31.375 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.375 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:115b7b2f:::benchmark_data_smithi179_148205_object37423:head# 2024-02-09T17:57:31.375 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.375 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:115b9418:::benchmark_data_smithi179_148205_object108055:head# 2024-02-09T17:57:31.375 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.377 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:115c6171:::benchmark_data_smithi179_148205_object49897:head# 2024-02-09T17:57:31.377 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.377 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:115cae0f:::benchmark_data_smithi179_148205_object8421:head# 2024-02-09T17:57:31.377 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.377 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:115d4fb2:::benchmark_data_smithi179_148205_object104015:head# 2024-02-09T17:57:31.377 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.377 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:115dd861:::benchmark_data_smithi179_148205_object98008:head# 2024-02-09T17:57:31.377 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.377 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:115de1ab:::benchmark_data_smithi179_148205_object83961:head# 2024-02-09T17:57:31.377 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.378 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:115f0929:::benchmark_data_smithi179_148205_object95694:head# 2024-02-09T17:57:31.378 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.378 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:115f2b5b:::benchmark_data_smithi179_148205_object108482:head# 2024-02-09T17:57:31.378 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.378 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:115f2b8c:::benchmark_data_smithi179_148205_object38903:head# 2024-02-09T17:57:31.378 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.378 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:115f5781:::benchmark_data_smithi179_148205_object64635:head# 2024-02-09T17:57:31.378 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.378 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:115fdaba:::benchmark_data_smithi179_148205_object104813:head# 2024-02-09T17:57:31.378 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.378 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:115fe738:::benchmark_data_smithi179_148205_object5886:head# 2024-02-09T17:57:31.378 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.378 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1162597a:::benchmark_data_smithi179_148205_object104333:head# 2024-02-09T17:57:31.378 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.379 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1162c901:::benchmark_data_smithi179_148205_object36554:head# 2024-02-09T17:57:31.379 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.379 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11633842:::benchmark_data_smithi179_148205_object89860:head# 2024-02-09T17:57:31.379 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.380 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:116367c4:::benchmark_data_smithi179_148205_object11628:head# 2024-02-09T17:57:31.380 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.380 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1163d3dd:::benchmark_data_smithi179_148205_object48768:head# 2024-02-09T17:57:31.380 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.381 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11640dbe:::benchmark_data_smithi179_148205_object20900:head# 2024-02-09T17:57:31.381 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.381 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1164454a:::benchmark_data_smithi179_148205_object91044:head# 2024-02-09T17:57:31.381 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.381 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11649350:::benchmark_data_smithi179_148205_object40258:head# 2024-02-09T17:57:31.381 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.381 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1164ec82:::benchmark_data_smithi179_148205_object67298:head# 2024-02-09T17:57:31.381 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.381 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1165353c:::benchmark_data_smithi179_148205_object26152:head# 2024-02-09T17:57:31.381 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.381 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11654c14:::benchmark_data_smithi179_148205_object88494:head# 2024-02-09T17:57:31.381 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.381 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1165747c:::benchmark_data_smithi179_148205_object15701:head# 2024-02-09T17:57:31.381 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.382 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11665e69:::benchmark_data_smithi179_148205_object38864:head# 2024-02-09T17:57:31.382 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.384 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1166db00:::benchmark_data_smithi179_148205_object50946:head# 2024-02-09T17:57:31.385 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.385 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1167db3a:::benchmark_data_smithi179_148205_object91744:head# 2024-02-09T17:57:31.385 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.385 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1168fc3b:::benchmark_data_smithi179_148205_object6859:head# 2024-02-09T17:57:31.385 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.385 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1169f78f:::benchmark_data_smithi179_148205_object22922:head# 2024-02-09T17:57:31.385 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.385 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:116a1897:::benchmark_data_smithi179_148205_object28850:head# 2024-02-09T17:57:31.385 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.385 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:116a915c:::benchmark_data_smithi179_148205_object84760:head# 2024-02-09T17:57:31.385 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.385 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:116acc7e:::benchmark_data_smithi179_148205_object36169:head# 2024-02-09T17:57:31.385 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.385 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:116b103c:::benchmark_data_smithi179_148205_object4757:head# 2024-02-09T17:57:31.385 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.386 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:116c3ddf:::benchmark_data_smithi179_148205_object75234:head# 2024-02-09T17:57:31.386 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.386 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:116db63a:::benchmark_data_smithi179_148205_object100559:head# 2024-02-09T17:57:31.386 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.386 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:116ea077:::benchmark_data_smithi179_148205_object43374:head# 2024-02-09T17:57:31.386 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.388 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:116eb5da:::benchmark_data_smithi179_148205_object71844:head# 2024-02-09T17:57:31.388 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.388 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:116fce7e:::benchmark_data_smithi179_148205_object103875:head# 2024-02-09T17:57:31.388 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.388 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:116fdc1c:::benchmark_data_smithi179_148205_object2800:head# 2024-02-09T17:57:31.388 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.388 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:117200a9:::benchmark_data_smithi179_148205_object49581:head# 2024-02-09T17:57:31.389 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.389 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11735236:::benchmark_data_smithi179_148205_object17491:head# 2024-02-09T17:57:31.389 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.389 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1173f01d:::benchmark_data_smithi179_148205_object47715:head# 2024-02-09T17:57:31.389 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.389 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11749f39:::benchmark_data_smithi179_148205_object54279:head# 2024-02-09T17:57:31.389 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.389 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1174c540:::benchmark_data_smithi179_148205_object18706:head# 2024-02-09T17:57:31.389 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.389 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1174ec4e:::benchmark_data_smithi179_148205_object46445:head# 2024-02-09T17:57:31.389 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.389 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:117648b6:::benchmark_data_smithi179_148205_object54048:head# 2024-02-09T17:57:31.389 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.390 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1177892e:::benchmark_data_smithi179_148205_object7383:head# 2024-02-09T17:57:31.390 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.390 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:117846e8:::benchmark_data_smithi179_148205_object64562:head# 2024-02-09T17:57:31.390 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.390 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:117899a3:::benchmark_data_smithi179_148205_object105876:head# 2024-02-09T17:57:31.390 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.390 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1178ee5d:::benchmark_data_smithi179_148205_object40807:head# 2024-02-09T17:57:31.390 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.390 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:117b7924:::benchmark_data_smithi179_148205_object105694:head# 2024-02-09T17:57:31.390 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.390 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:117c229e:::benchmark_data_smithi179_148205_object56275:head# 2024-02-09T17:57:31.390 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.390 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:117c8a0c:::benchmark_data_smithi179_148205_object3405:head# 2024-02-09T17:57:31.390 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.390 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:117c9b74:::benchmark_data_smithi179_148205_object32472:head# 2024-02-09T17:57:31.391 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.391 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:117d2ac3:::benchmark_data_smithi179_148205_object42992:head# 2024-02-09T17:57:31.391 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.392 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:117dacee:::benchmark_data_smithi179_148205_object29862:head# 2024-02-09T17:57:31.392 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.392 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:117def43:::benchmark_data_smithi179_148205_object71103:head# 2024-02-09T17:57:31.392 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.392 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:117e3471:::benchmark_data_smithi179_148205_object108573:head# 2024-02-09T17:57:31.392 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.392 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:117e5c35:::benchmark_data_smithi179_148205_object64667:head# 2024-02-09T17:57:31.392 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.392 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:117ea23a:::benchmark_data_smithi179_148205_object11117:head# 2024-02-09T17:57:31.392 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.393 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:117ec4b1:::benchmark_data_smithi179_148205_object54018:head# 2024-02-09T17:57:31.393 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.395 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11803ee1:::benchmark_data_smithi179_148205_object36377:head# 2024-02-09T17:57:31.395 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.395 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11812ca8:::benchmark_data_smithi179_148205_object87045:head# 2024-02-09T17:57:31.395 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.395 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1181a993:::benchmark_data_smithi179_148205_object40953:head# 2024-02-09T17:57:31.396 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.396 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1182ff4a:::benchmark_data_smithi179_148205_object65968:head# 2024-02-09T17:57:31.396 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.396 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11834618:::benchmark_data_smithi179_148205_object42768:head# 2024-02-09T17:57:31.396 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.398 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11841ccd:::benchmark_data_smithi179_148205_object30981:head# 2024-02-09T17:57:31.398 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.398 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1185a37f:::benchmark_data_smithi179_148205_object89896:head# 2024-02-09T17:57:31.398 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.398 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11864ca1:::benchmark_data_smithi179_148205_object22230:head# 2024-02-09T17:57:31.398 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.398 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11868bf8:::benchmark_data_smithi179_148205_object7312:head# 2024-02-09T17:57:31.398 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.398 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1186a1a5:::benchmark_data_smithi179_148205_object34143:head# 2024-02-09T17:57:31.398 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.398 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1186a8be:::benchmark_data_smithi179_148205_object67066:head# 2024-02-09T17:57:31.399 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.399 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:118720d0:::benchmark_data_smithi179_148205_object39388:head# 2024-02-09T17:57:31.399 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.399 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11884039:::benchmark_data_smithi179_148205_object84248:head# 2024-02-09T17:57:31.399 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.400 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11886977:::benchmark_data_smithi179_148205_object42645:head# 2024-02-09T17:57:31.400 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.400 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1188bc3d:::benchmark_data_smithi179_148205_object31047:head# 2024-02-09T17:57:31.400 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.400 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:118a5584:::benchmark_data_smithi179_148205_object81977:head# 2024-02-09T17:57:31.400 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.400 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:118a850f:::benchmark_data_smithi179_148205_object106714:head# 2024-02-09T17:57:31.400 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.404 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:118b9fe1:::benchmark_data_smithi179_148205_object8289:head# 2024-02-09T17:57:31.404 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.404 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:118beedb:::benchmark_data_smithi179_148205_object95782:head# 2024-02-09T17:57:31.404 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.404 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:118bf7ae:::benchmark_data_smithi179_148205_object80677:head# 2024-02-09T17:57:31.405 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.408 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:118c262c:::benchmark_data_smithi179_148205_object25794:head# 2024-02-09T17:57:31.408 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.409 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:118cded9:::benchmark_data_smithi179_148205_object6324:head# 2024-02-09T17:57:31.410 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.410 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:118d6a42:::benchmark_data_smithi179_148205_object84849:head# 2024-02-09T17:57:31.411 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.411 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:118d8c2f:::benchmark_data_smithi179_148205_object10046:head# 2024-02-09T17:57:31.411 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.411 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:118dc565:::benchmark_data_smithi179_148205_object51884:head# 2024-02-09T17:57:31.411 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.412 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:118e2e84:::benchmark_data_smithi179_148205_object3802:head# 2024-02-09T17:57:31.412 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.412 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:118e6bea:::benchmark_data_smithi179_148205_object16564:head# 2024-02-09T17:57:31.412 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.412 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1190f6a2:::benchmark_data_smithi179_148205_object12000:head# 2024-02-09T17:57:31.413 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.413 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11910b86:::benchmark_data_smithi179_148205_object56790:head# 2024-02-09T17:57:31.413 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.413 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1191155b:::benchmark_data_smithi179_148205_object97602:head# 2024-02-09T17:57:31.413 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.413 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11925534:::benchmark_data_smithi179_148205_object45013:head# 2024-02-09T17:57:31.413 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.414 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:119278ee:::benchmark_data_smithi179_148205_object80424:head# 2024-02-09T17:57:31.414 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.414 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1193d348:::benchmark_data_smithi179_148205_object1211:head# 2024-02-09T17:57:31.414 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.414 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1194bb2d:::benchmark_data_smithi179_148205_object3080:head# 2024-02-09T17:57:31.414 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.414 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11953463:::benchmark_data_smithi179_148205_object52380:head# 2024-02-09T17:57:31.415 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.415 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11955345:::benchmark_data_smithi179_148205_object99393:head# 2024-02-09T17:57:31.415 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.415 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11972d76:::benchmark_data_smithi179_148205_object74781:head# 2024-02-09T17:57:31.415 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.415 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11973135:::benchmark_data_smithi179_148205_object54861:head# 2024-02-09T17:57:31.415 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.417 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1198217a:::benchmark_data_smithi179_148205_object6726:head# 2024-02-09T17:57:31.417 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.417 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11987cdb:::benchmark_data_smithi179_148205_object47039:head# 2024-02-09T17:57:31.417 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.417 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11988875:::benchmark_data_smithi179_148205_object8913:head# 2024-02-09T17:57:31.417 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.417 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1198d91d:::benchmark_data_smithi179_148205_object51706:head# 2024-02-09T17:57:31.417 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.417 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1198d93e:::benchmark_data_smithi179_148205_object82883:head# 2024-02-09T17:57:31.418 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.418 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1198ee71:::benchmark_data_smithi179_148205_object27495:head# 2024-02-09T17:57:31.418 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.418 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:119b4ae1:::benchmark_data_smithi179_148205_object57237:head# 2024-02-09T17:57:31.418 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.418 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:119bd590:::benchmark_data_smithi179_148205_object99422:head# 2024-02-09T17:57:31.418 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.418 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:119bdad5:::benchmark_data_smithi179_148205_object98176:head# 2024-02-09T17:57:31.418 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.418 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:119be395:::benchmark_data_smithi179_148205_object66287:head# 2024-02-09T17:57:31.418 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.418 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:119c7ccf:::benchmark_data_smithi179_148205_object21420:head# 2024-02-09T17:57:31.419 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.421 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:119ca9a8:::benchmark_data_smithi179_148205_object337:head# 2024-02-09T17:57:31.421 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.421 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:119d15b6:::benchmark_data_smithi179_148205_object34805:head# 2024-02-09T17:57:31.421 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.421 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:119d2cfa:::benchmark_data_smithi179_148205_object18120:head# 2024-02-09T17:57:31.421 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.421 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:119d3696:::benchmark_data_smithi179_148205_object8733:head# 2024-02-09T17:57:31.421 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.421 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:119d42b5:::benchmark_data_smithi179_148205_object19456:head# 2024-02-09T17:57:31.421 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.421 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:119e01b1:::benchmark_data_smithi179_148205_object86737:head# 2024-02-09T17:57:31.421 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.421 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:119eeb65:::benchmark_data_smithi179_148205_object54886:head# 2024-02-09T17:57:31.422 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.422 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:119f880d:::benchmark_data_smithi179_148205_object93329:head# 2024-02-09T17:57:31.422 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.422 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11a0100d:::benchmark_data_smithi179_148205_object98026:head# 2024-02-09T17:57:31.422 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.422 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11a051f8:::benchmark_data_smithi179_148205_object23414:head# 2024-02-09T17:57:31.422 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.422 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11a05dfe:::benchmark_data_smithi179_148205_object59013:head# 2024-02-09T17:57:31.422 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.422 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11a15b92:::benchmark_data_smithi179_148205_object33440:head# 2024-02-09T17:57:31.422 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.422 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11a1baa6:::benchmark_data_smithi179_148205_object28721:head# 2024-02-09T17:57:31.422 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.422 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11a1d88c:::benchmark_data_smithi179_148205_object74197:head# 2024-02-09T17:57:31.423 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.423 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:57:31.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:30 smithi179 ceph-mon[43975]: 8.d scrub starts 2024-02-09T17:57:31.426 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11a1fa9f:::benchmark_data_smithi179_148205_object38031:head# 2024-02-09T17:57:31.426 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.426 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11a26c68:::benchmark_data_smithi179_148205_object75799:head# 2024-02-09T17:57:31.426 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.426 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11a26c9f:::benchmark_data_smithi179_148205_object31636:head# 2024-02-09T17:57:31.426 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.426 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11a279ff:::benchmark_data_smithi179_148205_object79756:head# 2024-02-09T17:57:31.426 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.426 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11a397e6:::benchmark_data_smithi179_148205_object91746:head# 2024-02-09T17:57:31.426 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.426 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11a3ff3e:::benchmark_data_smithi179_148205_object21350:head# 2024-02-09T17:57:31.426 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.426 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11a4fd88:::benchmark_data_smithi179_148205_object91213:head# 2024-02-09T17:57:31.427 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.427 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11a56827:::benchmark_data_smithi179_148205_object76991:head# 2024-02-09T17:57:31.427 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.427 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11a5e934:::benchmark_data_smithi179_148205_object93637:head# 2024-02-09T17:57:31.427 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.427 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11a6fdb3:::benchmark_data_smithi179_148205_object38279:head# 2024-02-09T17:57:31.427 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.427 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11abdfe9:::benchmark_data_smithi179_148205_object87127:head# 2024-02-09T17:57:31.427 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.427 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11ac4d07:::benchmark_data_smithi179_148205_object6552:head# 2024-02-09T17:57:31.427 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.427 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11ad553d:::benchmark_data_smithi179_148205_object19365:head# 2024-02-09T17:57:31.427 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.428 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11adf77a:::benchmark_data_smithi179_148205_object1662:head# 2024-02-09T17:57:31.428 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.428 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11ae413d:::benchmark_data_smithi179_148205_object101887:head# 2024-02-09T17:57:31.428 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.428 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11ae8478:::benchmark_data_smithi179_148205_object98810:head# 2024-02-09T17:57:31.428 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.428 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11ae8ec0:::benchmark_data_smithi179_148205_object75303:head# 2024-02-09T17:57:31.428 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.428 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11af4014:::benchmark_data_smithi179_148205_object102408:head# 2024-02-09T17:57:31.428 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.429 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11af7124:::benchmark_data_smithi179_148205_object66227:head# 2024-02-09T17:57:31.429 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.429 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11afba3d:::benchmark_data_smithi179_148205_object27375:head# 2024-02-09T17:57:31.430 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.430 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11afcd3a:::benchmark_data_smithi179_148205_object64713:head# 2024-02-09T17:57:31.430 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.430 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11b01c69:::benchmark_data_smithi179_148205_object33723:head# 2024-02-09T17:57:31.430 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.430 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11b0731d:::benchmark_data_smithi179_148205_object81522:head# 2024-02-09T17:57:31.430 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.430 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11b2bd84:::benchmark_data_smithi179_148205_object105310:head# 2024-02-09T17:57:31.430 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.430 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11b55f91:::benchmark_data_smithi179_148205_object63158:head# 2024-02-09T17:57:31.430 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.430 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11b5a1fa:::benchmark_data_smithi179_148205_object11332:head# 2024-02-09T17:57:31.430 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.434 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11b64e2d:::benchmark_data_smithi179_148205_object89196:head# 2024-02-09T17:57:31.434 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.435 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11b6949f:::benchmark_data_smithi179_148205_object72597:head# 2024-02-09T17:57:31.435 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.435 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11b740c2:::benchmark_data_smithi179_148205_object467:head# 2024-02-09T17:57:31.435 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.435 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11b741be:::benchmark_data_smithi179_148205_object36501:head# 2024-02-09T17:57:31.435 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.436 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11b74773:::benchmark_data_smithi179_148205_object105127:head# 2024-02-09T17:57:31.436 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.436 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11b83b23:::benchmark_data_smithi179_148205_object96530:head# 2024-02-09T17:57:31.436 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.436 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11b8db68:::benchmark_data_smithi179_148205_object48777:head# 2024-02-09T17:57:31.436 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.437 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11b8f9e4:::benchmark_data_smithi179_148205_object15607:head# 2024-02-09T17:57:31.437 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.443 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11b9c75e:::benchmark_data_smithi179_148205_object25554:head# 2024-02-09T17:57:31.443 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.443 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11bb5561:::benchmark_data_smithi179_148205_object49569:head# 2024-02-09T17:57:31.443 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.443 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11bb6a7d:::benchmark_data_smithi179_148205_object76805:head# 2024-02-09T17:57:31.443 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.443 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11bbfc4b:::benchmark_data_smithi179_148205_object23931:head# 2024-02-09T17:57:31.443 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.443 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11be33d3:::benchmark_data_smithi179_148205_object17726:head# 2024-02-09T17:57:31.443 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.444 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11be4213:::benchmark_data_smithi179_148205_object97958:head# 2024-02-09T17:57:31.444 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.444 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11be5a51:::benchmark_data_smithi179_148205_object65033:head# 2024-02-09T17:57:31.444 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.444 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11be72c1:::benchmark_data_smithi179_148205_object103993:head# 2024-02-09T17:57:31.444 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.444 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11c01424:::benchmark_data_smithi179_148205_object104820:head# 2024-02-09T17:57:31.444 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.444 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11c03ba1:::benchmark_data_smithi179_148205_object86204:head# 2024-02-09T17:57:31.444 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.444 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11c0461b:::benchmark_data_smithi179_148205_object86316:head# 2024-02-09T17:57:31.444 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.444 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11c1027b:::benchmark_data_smithi179_148205_object81794:head# 2024-02-09T17:57:31.444 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.445 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11c1101e:::benchmark_data_smithi179_148205_object108882:head# 2024-02-09T17:57:31.445 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.445 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11c150f5:::benchmark_data_smithi179_148205_object62333:head# 2024-02-09T17:57:31.445 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.445 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11c1e2a9:::benchmark_data_smithi179_148205_object1203:head# 2024-02-09T17:57:31.445 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.445 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11c21b7b:::benchmark_data_smithi179_148205_object87945:head# 2024-02-09T17:57:31.445 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.445 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11c39c7d:::benchmark_data_smithi179_148205_object17333:head# 2024-02-09T17:57:31.445 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.445 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11c3ce29:::benchmark_data_smithi179_148205_object105121:head# 2024-02-09T17:57:31.445 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.447 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11c3e250:::benchmark_data_smithi179_148205_object61578:head# 2024-02-09T17:57:31.448 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.449 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11c52778:::benchmark_data_smithi179_148205_object101423:head# 2024-02-09T17:57:31.449 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.450 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11c5901f:::benchmark_data_smithi179_148205_object20819:head# 2024-02-09T17:57:31.450 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.450 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11c64d27:::benchmark_data_smithi179_148205_object77027:head# 2024-02-09T17:57:31.450 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.451 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11c7354d:::benchmark_data_smithi179_148205_object4783:head# 2024-02-09T17:57:31.451 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.455 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11c81f80:::benchmark_data_smithi179_148205_object67366:head# 2024-02-09T17:57:31.455 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.457 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11c85357:::benchmark_data_smithi179_148205_object34957:head# 2024-02-09T17:57:31.457 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.458 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11c860de:::benchmark_data_smithi179_148205_object82712:head# 2024-02-09T17:57:31.458 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.458 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11c90b64:::benchmark_data_smithi179_148205_object25781:head# 2024-02-09T17:57:31.458 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.459 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11ca843a:::benchmark_data_smithi179_148205_object65593:head# 2024-02-09T17:57:31.459 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.459 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11cafc3d:::benchmark_data_smithi179_148205_object587:head# 2024-02-09T17:57:31.459 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.459 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11cba501:::benchmark_data_smithi179_148205_object58568:head# 2024-02-09T17:57:31.459 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.460 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11cbce00:::benchmark_data_smithi179_148205_object51513:head# 2024-02-09T17:57:31.460 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.460 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11cbd368:::benchmark_data_smithi179_148205_object12008:head# 2024-02-09T17:57:31.460 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.461 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11cc7011:::benchmark_data_smithi179_148205_object14560:head# 2024-02-09T17:57:31.461 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.461 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11cdb54c:::benchmark_data_smithi179_148205_object85389:head# 2024-02-09T17:57:31.461 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.461 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11cdc5a2:::benchmark_data_smithi179_148205_object79948:head# 2024-02-09T17:57:31.461 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.462 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11cddfd3:::benchmark_data_smithi179_148205_object88940:head# 2024-02-09T17:57:31.462 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.462 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11cdec59:::benchmark_data_smithi179_148205_object35126:head# 2024-02-09T17:57:31.462 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.462 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11ce08b5:::benchmark_data_smithi179_148205_object28240:head# 2024-02-09T17:57:31.462 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.462 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11ce8664:::benchmark_data_smithi179_148205_object14421:head# 2024-02-09T17:57:31.462 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.463 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11ceef18:::benchmark_data_smithi179_148205_object42916:head# 2024-02-09T17:57:31.463 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.463 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11cfeab4:::benchmark_data_smithi179_148205_object92618:head# 2024-02-09T17:57:31.463 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.464 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11d0f0be:::benchmark_data_smithi179_148205_object60604:head# 2024-02-09T17:57:31.464 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.464 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11d0f801:::benchmark_data_smithi179_148205_object74270:head# 2024-02-09T17:57:31.464 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.464 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11d231c5:::benchmark_data_smithi179_148205_object36635:head# 2024-02-09T17:57:31.464 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.464 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11d36596:::benchmark_data_smithi179_148205_object64858:head# 2024-02-09T17:57:31.464 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.464 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11d38819:::benchmark_data_smithi179_148205_object60359:head# 2024-02-09T17:57:31.464 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.464 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11d592b5:::benchmark_data_smithi179_148205_object76056:head# 2024-02-09T17:57:31.465 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.465 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11d59a4b:::benchmark_data_smithi179_148205_object46731:head# 2024-02-09T17:57:31.465 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.466 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11d5ba1d:::benchmark_data_smithi179_148205_object35232:head# 2024-02-09T17:57:31.466 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.466 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11d5d8c7:::benchmark_data_smithi179_148205_object72026:head# 2024-02-09T17:57:31.466 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.466 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11d600aa:::benchmark_data_smithi179_148205_object87574:head# 2024-02-09T17:57:31.466 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.466 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11d615e9:::benchmark_data_smithi179_148205_object17260:head# 2024-02-09T17:57:31.466 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.466 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11d635d8:::benchmark_data_smithi179_148205_object80551:head# 2024-02-09T17:57:31.466 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.467 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11d814cb:::benchmark_data_smithi179_148205_object57800:head# 2024-02-09T17:57:31.467 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.467 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11d85345:::benchmark_data_smithi179_148205_object54425:head# 2024-02-09T17:57:31.467 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.467 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11d868e2:::benchmark_data_smithi179_148205_object15621:head# 2024-02-09T17:57:31.467 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.467 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11d87b3f:::benchmark_data_smithi179_148205_object7148:head# 2024-02-09T17:57:31.468 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.471 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11d946ae:::benchmark_data_smithi179_148205_object58948:head# 2024-02-09T17:57:31.472 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.472 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11da8626:::benchmark_data_smithi179_148205_object36916:head# 2024-02-09T17:57:31.472 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.472 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11daa3ab:::benchmark_data_smithi179_148205_object109447:head# 2024-02-09T17:57:31.472 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.472 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11dad86b:::benchmark_data_smithi179_148205_object9221:head# 2024-02-09T17:57:31.472 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.472 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11db3b8a:::benchmark_data_smithi179_148205_object38375:head# 2024-02-09T17:57:31.472 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.472 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11dbf52a:::benchmark_data_smithi179_148205_object24647:head# 2024-02-09T17:57:31.472 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.472 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11ddf785:::benchmark_data_smithi179_148205_object44779:head# 2024-02-09T17:57:31.473 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.473 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11de0bcd:::benchmark_data_smithi179_148205_object64261:head# 2024-02-09T17:57:31.473 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.473 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11de43e6:::benchmark_data_smithi179_148205_object75663:head# 2024-02-09T17:57:31.473 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.475 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:31.480 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11de9186:::benchmark_data_smithi179_148205_object104009:head# 2024-02-09T17:57:31.480 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.486 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11dfacbe:::benchmark_data_smithi179_148205_object60831:head# 2024-02-09T17:57:31.486 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.493 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11e09377:::benchmark_data_smithi179_148205_object42247:head# 2024-02-09T17:57:31.494 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.494 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11e0e65b:::benchmark_data_smithi179_148205_object89400:head# 2024-02-09T17:57:31.494 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.494 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11e12cdd:::benchmark_data_smithi179_148205_object109608:head# 2024-02-09T17:57:31.494 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.494 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11e130ed:::benchmark_data_smithi179_148205_object26998:head# 2024-02-09T17:57:31.494 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.494 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11e14acd:::benchmark_data_smithi179_148205_object1306:head# 2024-02-09T17:57:31.494 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.494 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11e1821c:::benchmark_data_smithi179_148205_object8010:head# 2024-02-09T17:57:31.494 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.494 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11e18a36:::benchmark_data_smithi179_148205_object77122:head# 2024-02-09T17:57:31.494 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.494 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11e19b29:::benchmark_data_smithi179_148205_object63101:head# 2024-02-09T17:57:31.495 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.495 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11e1bf75:::benchmark_data_smithi179_148205_object72265:head# 2024-02-09T17:57:31.495 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.495 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11e1e756:::benchmark_data_smithi179_148205_object1620:head# 2024-02-09T17:57:31.495 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.495 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11e20aff:::benchmark_data_smithi179_148205_object104854:head# 2024-02-09T17:57:31.495 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.495 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11e23cb7:::benchmark_data_smithi179_148205_object4807:head# 2024-02-09T17:57:31.495 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.495 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11e25e6a:::benchmark_data_smithi179_148205_object14409:head# 2024-02-09T17:57:31.495 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.495 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11e291d7:::benchmark_data_smithi179_148205_object13453:head# 2024-02-09T17:57:31.495 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.497 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11e50f32:::benchmark_data_smithi179_148205_object25254:head# 2024-02-09T17:57:31.497 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.497 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11e6635f:::benchmark_data_smithi179_148205_object50198:head# 2024-02-09T17:57:31.497 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.497 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11e67f54:::benchmark_data_smithi179_148205_object17375:head# 2024-02-09T17:57:31.497 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.497 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11e6b847:::benchmark_data_smithi179_148205_object38263:head# 2024-02-09T17:57:31.497 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.497 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11e7128d:::benchmark_data_smithi179_148205_object108579:head# 2024-02-09T17:57:31.497 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.497 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11e86c10:::benchmark_data_smithi179_148205_object85058:head# 2024-02-09T17:57:31.498 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.498 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11e87e6b:::benchmark_data_smithi179_148205_object24444:head# 2024-02-09T17:57:31.498 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.498 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11e89dc8:::benchmark_data_smithi179_148205_object4101:head# 2024-02-09T17:57:31.498 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.498 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11e94237:::benchmark_data_smithi179_148205_object1449:head# 2024-02-09T17:57:31.498 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.498 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11e9532c:::benchmark_data_smithi179_148205_object79037:head# 2024-02-09T17:57:31.498 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.503 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11e9a9db:::benchmark_data_smithi179_148205_object37518:head# 2024-02-09T17:57:31.503 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.503 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:57:31.504 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:57:31.504 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11e9acb5:::benchmark_data_smithi179_148205_object43810:head# 2024-02-09T17:57:31.504 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.505 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11ea4e65:::benchmark_data_smithi179_148205_object22010:head# 2024-02-09T17:57:31.505 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.505 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11eba9cf:::benchmark_data_smithi179_148205_object66130:head# 2024-02-09T17:57:31.505 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.506 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11ebab85:::benchmark_data_smithi179_148205_object96385:head# 2024-02-09T17:57:31.506 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.506 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11ec4319:::benchmark_data_smithi179_148205_object68182:head# 2024-02-09T17:57:31.506 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.507 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11ecec6c:::benchmark_data_smithi179_148205_object63725:head# 2024-02-09T17:57:31.507 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.507 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11ed0196:::benchmark_data_smithi179_148205_object17399:head# 2024-02-09T17:57:31.507 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.507 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11ed7dd5:::benchmark_data_smithi179_148205_object102911:head# 2024-02-09T17:57:31.507 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.507 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11edef3f:::benchmark_data_smithi179_148205_object21629:head# 2024-02-09T17:57:31.508 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.508 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11ee7027:::benchmark_data_smithi179_148205_object21791:head# 2024-02-09T17:57:31.508 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.508 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11eee562:::benchmark_data_smithi179_148205_object87363:head# 2024-02-09T17:57:31.508 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.511 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11f284a8:::benchmark_data_smithi179_148205_object10734:head# 2024-02-09T17:57:31.511 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.511 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11f2b28b:::benchmark_data_smithi179_148205_object91335:head# 2024-02-09T17:57:31.511 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.511 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11f4111f:::benchmark_data_smithi179_148205_object68432:head# 2024-02-09T17:57:31.511 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.511 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11f42fb9:::benchmark_data_smithi179_148205_object34193:head# 2024-02-09T17:57:31.511 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.511 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11f46b65:::benchmark_data_smithi179_148205_object78337:head# 2024-02-09T17:57:31.511 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.511 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11f5452f:::benchmark_data_smithi179_148205_object46022:head# 2024-02-09T17:57:31.511 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.512 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11f55ec3:::benchmark_data_smithi179_148205_object107194:head# 2024-02-09T17:57:31.512 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.512 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11f61119:::benchmark_data_smithi179_148205_object48660:head# 2024-02-09T17:57:31.512 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.512 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11f62677:::benchmark_data_smithi179_148205_object74032:head# 2024-02-09T17:57:31.512 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.514 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11f6fc40:::benchmark_data_smithi179_148205_object108519:head# 2024-02-09T17:57:31.514 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.514 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11f703c9:::benchmark_data_smithi179_148205_object97298:head# 2024-02-09T17:57:31.514 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.514 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11f729ef:::benchmark_data_smithi179_148205_object106899:head# 2024-02-09T17:57:31.514 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.514 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11f87c26:::benchmark_data_smithi179_148205_object92721:head# 2024-02-09T17:57:31.514 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.514 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11f9033f:::benchmark_data_smithi179_148205_object104430:head# 2024-02-09T17:57:31.514 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.514 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11f9be39:::benchmark_data_smithi179_148205_object23876:head# 2024-02-09T17:57:31.515 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.515 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11fd573b:::benchmark_data_smithi179_148205_object55328:head# 2024-02-09T17:57:31.515 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.515 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11fddc54:::benchmark_data_smithi179_148205_object71902:head# 2024-02-09T17:57:31.515 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.515 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11fe3338:::benchmark_data_smithi179_148205_object69536:head# 2024-02-09T17:57:31.515 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.515 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11fe71ad:::benchmark_data_smithi179_148205_object33744:head# 2024-02-09T17:57:31.515 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.515 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11feb186:::benchmark_data_smithi179_148205_object24312:head# 2024-02-09T17:57:31.515 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.515 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11feb4a8:::benchmark_data_smithi179_148205_object37229:head# 2024-02-09T17:57:31.515 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.517 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:11ffcef1:::benchmark_data_smithi179_148205_object47323:head# 2024-02-09T17:57:31.517 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.517 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12008ad6:::benchmark_data_smithi179_148205_object28515:head# 2024-02-09T17:57:31.517 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.517 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12009cec:::benchmark_data_smithi179_148205_object51955:head# 2024-02-09T17:57:31.517 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.517 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12013148:::benchmark_data_smithi179_148205_object29589:head# 2024-02-09T17:57:31.517 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.518 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:120160a8:::benchmark_data_smithi179_148205_object79317:head# 2024-02-09T17:57:31.518 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.518 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1201e963:::benchmark_data_smithi179_148205_object77263:head# 2024-02-09T17:57:31.518 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.518 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1203da0e:::benchmark_data_smithi179_148205_object14463:head# 2024-02-09T17:57:31.518 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.518 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1204a58a:::benchmark_data_smithi179_148205_object104768:head# 2024-02-09T17:57:31.518 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.518 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12055e4f:::benchmark_data_smithi179_148205_object31951:head# 2024-02-09T17:57:31.518 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.518 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:120611e9:::benchmark_data_smithi179_148205_object96353:head# 2024-02-09T17:57:31.518 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.518 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12063d8c:::benchmark_data_smithi179_148205_object100232:head# 2024-02-09T17:57:31.518 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.522 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1206a7e4:::benchmark_data_smithi179_148205_object11457:head# 2024-02-09T17:57:31.522 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.522 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1206eaf1:::benchmark_data_smithi179_148205_object45710:head# 2024-02-09T17:57:31.522 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.522 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12071325:::benchmark_data_smithi179_148205_object77400:head# 2024-02-09T17:57:31.522 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.522 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1207f0f2:::benchmark_data_smithi179_148205_object88273:head# 2024-02-09T17:57:31.522 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.522 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12080e5a:::benchmark_data_smithi179_148205_object60721:head# 2024-02-09T17:57:31.522 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.522 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1209510d:::benchmark_data_smithi179_148205_object16536:head# 2024-02-09T17:57:31.522 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.522 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:120a46fc:::benchmark_data_smithi179_148205_object101555:head# 2024-02-09T17:57:31.523 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.523 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:120a9f1e:::benchmark_data_smithi179_148205_object14085:head# 2024-02-09T17:57:31.523 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.523 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:120aada3:::benchmark_data_smithi179_148205_object28234:head# 2024-02-09T17:57:31.523 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.523 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:120afc91:::benchmark_data_smithi179_148205_object83753:head# 2024-02-09T17:57:31.523 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.523 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:120b4e1b:::benchmark_data_smithi179_148205_object80632:head# 2024-02-09T17:57:31.523 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.523 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:120b6495:::benchmark_data_smithi179_148205_object89875:head# 2024-02-09T17:57:31.523 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.523 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:120e62c0:::benchmark_data_smithi179_148205_object7391:head# 2024-02-09T17:57:31.523 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.523 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:120e73b8:::benchmark_data_smithi179_148205_object48414:head# 2024-02-09T17:57:31.523 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.524 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:120f0885:::benchmark_data_smithi179_148205_object83480:head# 2024-02-09T17:57:31.524 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.528 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12103b83:::benchmark_data_smithi179_148205_object103230:head# 2024-02-09T17:57:31.528 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.528 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12120bbc:::benchmark_data_smithi179_148205_object2896:head# 2024-02-09T17:57:31.528 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.528 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1212ea75:::benchmark_data_smithi179_148205_object68037:head# 2024-02-09T17:57:31.528 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.528 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1213a890:::benchmark_data_smithi179_148205_object88223:head# 2024-02-09T17:57:31.528 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.528 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1214381e:::benchmark_data_smithi179_148205_object24418:head# 2024-02-09T17:57:31.528 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.529 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1214e1cc:::benchmark_data_smithi179_148205_object104208:head# 2024-02-09T17:57:31.529 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.529 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1214f7d5:::benchmark_data_smithi179_148205_object40545:head# 2024-02-09T17:57:31.529 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.529 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12158c90:::benchmark_data_smithi179_148205_object11904:head# 2024-02-09T17:57:31.529 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.529 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:121591d8:::benchmark_data_smithi179_148205_object29889:head# 2024-02-09T17:57:31.529 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.529 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1215e7e1:::benchmark_data_smithi179_148205_object60542:head# 2024-02-09T17:57:31.529 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.529 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1216006e:::benchmark_data_smithi179_148205_object96203:head# 2024-02-09T17:57:31.529 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.529 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1219d0f6:::benchmark_data_smithi179_148205_object58638:head# 2024-02-09T17:57:31.529 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.529 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:121a8fb7:::benchmark_data_smithi179_148205_object30786:head# 2024-02-09T17:57:31.530 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.530 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:121ab3a4:::benchmark_data_smithi179_148205_object12980:head# 2024-02-09T17:57:31.530 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.530 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:121abd4b:::benchmark_data_smithi179_148205_object46444:head# 2024-02-09T17:57:31.530 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.530 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:121c3359:::benchmark_data_smithi179_148205_object20804:head# 2024-02-09T17:57:31.530 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.530 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:121d00a0:::benchmark_data_smithi179_148205_object41853:head# 2024-02-09T17:57:31.530 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.530 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:121d45ec:::benchmark_data_smithi179_148205_object4812:head# 2024-02-09T17:57:31.530 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.530 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:121d7394:::benchmark_data_smithi179_148205_object46697:head# 2024-02-09T17:57:31.530 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.530 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:121d75cc:::benchmark_data_smithi179_148205_object56539:head# 2024-02-09T17:57:31.531 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.531 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:121db17e:::benchmark_data_smithi179_148205_object93572:head# 2024-02-09T17:57:31.531 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.531 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:121dc1c4:::benchmark_data_smithi179_148205_object107929:head# 2024-02-09T17:57:31.531 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.531 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:121e139e:::benchmark_data_smithi179_148205_object29560:head# 2024-02-09T17:57:31.531 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.531 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:121e2874:::benchmark_data_smithi179_148205_object22928:head# 2024-02-09T17:57:31.531 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.531 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:121f568a:::benchmark_data_smithi179_148205_object37889:head# 2024-02-09T17:57:31.531 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.536 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:121fd13b:::benchmark_data_smithi179_148205_object103928:head# 2024-02-09T17:57:31.537 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.537 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:121fd9fb:::benchmark_data_smithi179_148205_object99005:head# 2024-02-09T17:57:31.537 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.537 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12208c71:::benchmark_data_smithi179_148205_object32297:head# 2024-02-09T17:57:31.537 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.537 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1220bb1c:::benchmark_data_smithi179_148205_object101857:head# 2024-02-09T17:57:31.537 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.537 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1220daeb:::benchmark_data_smithi179_148205_object100170:head# 2024-02-09T17:57:31.537 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.537 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:122108ed:::benchmark_data_smithi179_148205_object94673:head# 2024-02-09T17:57:31.537 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.537 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1223ee03:::benchmark_data_smithi179_148205_object9679:head# 2024-02-09T17:57:31.537 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.538 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1223faa4:::benchmark_data_smithi179_148205_object15310:head# 2024-02-09T17:57:31.538 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.538 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12258d72:::benchmark_data_smithi179_148205_object52657:head# 2024-02-09T17:57:31.538 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.538 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12267825:::benchmark_data_smithi179_148205_object251:head# 2024-02-09T17:57:31.538 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.538 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1226feb8:::benchmark_data_smithi179_148205_object52085:head# 2024-02-09T17:57:31.538 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.538 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1227bc2a:::benchmark_data_smithi179_148205_object53172:head# 2024-02-09T17:57:31.538 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.538 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1227f3fd:::benchmark_data_smithi179_148205_object79557:head# 2024-02-09T17:57:31.538 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.538 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12285f2e:::benchmark_data_smithi179_148205_object62783:head# 2024-02-09T17:57:31.538 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.539 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1228cad6:::benchmark_data_smithi179_148205_object30052:head# 2024-02-09T17:57:31.539 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.539 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:122c4782:::benchmark_data_smithi179_148205_object79015:head# 2024-02-09T17:57:31.539 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.539 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:122c4f87:::benchmark_data_smithi179_148205_object96521:head# 2024-02-09T17:57:31.539 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.539 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:122d06b7:::benchmark_data_smithi179_148205_object8237:head# 2024-02-09T17:57:31.539 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.539 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:122db251:::benchmark_data_smithi179_148205_object63056:head# 2024-02-09T17:57:31.539 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.539 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:122e2704:::benchmark_data_smithi179_148205_object52962:head# 2024-02-09T17:57:31.539 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.539 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:122e79e5:::benchmark_data_smithi179_148205_object91683:head# 2024-02-09T17:57:31.539 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.540 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:122f61a8:::benchmark_data_smithi179_148205_object32483:head# 2024-02-09T17:57:31.540 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.540 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:122fdce6:::benchmark_data_smithi179_148205_object59100:head# 2024-02-09T17:57:31.540 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.540 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1232dffb:::benchmark_data_smithi179_148205_object66343:head# 2024-02-09T17:57:31.540 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.540 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:123344c1:::benchmark_data_smithi179_148205_object59587:head# 2024-02-09T17:57:31.540 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.540 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12335d15:::benchmark_data_smithi179_148205_object83735:head# 2024-02-09T17:57:31.540 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.540 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1233747a:::benchmark_data_smithi179_148205_object24029:head# 2024-02-09T17:57:31.540 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.540 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1234552f:::benchmark_data_smithi179_148205_object43348:head# 2024-02-09T17:57:31.540 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.541 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1234a3f9:::benchmark_data_smithi179_148205_object45644:head# 2024-02-09T17:57:31.541 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.541 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12350688:::benchmark_data_smithi179_148205_object1736:head# 2024-02-09T17:57:31.541 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.541 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12355983:::benchmark_data_smithi179_148205_object103232:head# 2024-02-09T17:57:31.541 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.541 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1236d29f:::benchmark_data_smithi179_148205_object19869:head# 2024-02-09T17:57:31.541 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.541 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1237018f:::benchmark_data_smithi179_148205_object97424:head# 2024-02-09T17:57:31.541 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.541 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1237bfb4:::benchmark_data_smithi179_148205_object35132:head# 2024-02-09T17:57:31.541 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.541 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1239600a:::benchmark_data_smithi179_148205_object106245:head# 2024-02-09T17:57:31.541 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.542 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1239dc76:::benchmark_data_smithi179_148205_object45899:head# 2024-02-09T17:57:31.542 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.549 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:123a4d1f:::benchmark_data_smithi179_148205_object54730:head# 2024-02-09T17:57:31.549 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.549 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:123b6092:::benchmark_data_smithi179_148205_object76077:head# 2024-02-09T17:57:31.550 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.550 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:123c6c29:::benchmark_data_smithi179_148205_object58603:head# 2024-02-09T17:57:31.550 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.550 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:123c86db:::benchmark_data_smithi179_148205_object31008:head# 2024-02-09T17:57:31.550 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.550 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:123d217f:::benchmark_data_smithi179_148205_object105109:head# 2024-02-09T17:57:31.550 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.550 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:123f6e78:::benchmark_data_smithi179_148205_object100988:head# 2024-02-09T17:57:31.550 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.550 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1240ffa5:::benchmark_data_smithi179_148205_object90660:head# 2024-02-09T17:57:31.550 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.550 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:124113b5:::benchmark_data_smithi179_148205_object54005:head# 2024-02-09T17:57:31.550 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.550 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1241555d:::benchmark_data_smithi179_148205_object97499:head# 2024-02-09T17:57:31.551 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.551 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1242166c:::benchmark_data_smithi179_148205_object67544:head# 2024-02-09T17:57:31.551 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.551 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1242d1a8:::benchmark_data_smithi179_148205_object59022:head# 2024-02-09T17:57:31.551 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.551 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12432d4f:::benchmark_data_smithi179_148205_object26074:head# 2024-02-09T17:57:31.551 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.551 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12447422:::benchmark_data_smithi179_148205_object79946:head# 2024-02-09T17:57:31.551 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.551 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1245f60b:::benchmark_data_smithi179_148205_object14757:head# 2024-02-09T17:57:31.551 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.551 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:124757cf:::benchmark_data_smithi179_148205_object97487:head# 2024-02-09T17:57:31.551 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.552 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12476cfb:::benchmark_data_smithi179_148205_object58443:head# 2024-02-09T17:57:31.552 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.552 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12488781:::benchmark_data_smithi179_148205_object6014:head# 2024-02-09T17:57:31.552 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.552 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1249643e:::benchmark_data_smithi179_148205_object38339:head# 2024-02-09T17:57:31.552 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.552 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1249b880:::benchmark_data_smithi179_148205_object10115:head# 2024-02-09T17:57:31.552 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.552 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1249bee6:::benchmark_data_smithi179_148205_object73093:head# 2024-02-09T17:57:31.552 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.552 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:124b5b38:::benchmark_data_smithi179_148205_object43814:head# 2024-02-09T17:57:31.552 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.552 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:124ba1dd:::benchmark_data_smithi179_148205_object75597:head# 2024-02-09T17:57:31.552 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.552 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:124c34e4:::benchmark_data_smithi179_148205_object74296:head# 2024-02-09T17:57:31.553 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.553 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:124c4a42:::benchmark_data_smithi179_148205_object88313:head# 2024-02-09T17:57:31.553 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.553 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:124c9d58:::benchmark_data_smithi179_148205_object25005:head# 2024-02-09T17:57:31.553 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.553 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:124ca57a:::benchmark_data_smithi179_148205_object96426:head# 2024-02-09T17:57:31.553 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.553 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:124d92a0:::benchmark_data_smithi179_148205_object101999:head# 2024-02-09T17:57:31.553 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.553 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:124fa142:::benchmark_data_smithi179_148205_object39195:head# 2024-02-09T17:57:31.553 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.553 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:124fb2fb:::benchmark_data_smithi179_148205_object1557:head# 2024-02-09T17:57:31.553 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.553 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:124fee28:::benchmark_data_smithi179_148205_object107908:head# 2024-02-09T17:57:31.553 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.554 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1250557b:::benchmark_data_smithi179_148205_object63204:head# 2024-02-09T17:57:31.554 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.554 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12518f61:::benchmark_data_smithi179_148205_object102209:head# 2024-02-09T17:57:31.554 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.554 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12523434:::benchmark_data_smithi179_148205_object34408:head# 2024-02-09T17:57:31.554 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.554 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:125248ce:::benchmark_data_smithi179_148205_object23121:head# 2024-02-09T17:57:31.554 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.554 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:125310b3:::benchmark_data_smithi179_148205_object70688:head# 2024-02-09T17:57:31.554 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.554 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12549554:::benchmark_data_smithi179_148205_object13570:head# 2024-02-09T17:57:31.554 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.554 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12575046:::benchmark_data_smithi179_148205_object101982:head# 2024-02-09T17:57:31.554 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.554 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1257673e:::benchmark_data_smithi179_148205_object90369:head# 2024-02-09T17:57:31.555 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.555 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12590999:::benchmark_data_smithi179_148205_object65581:head# 2024-02-09T17:57:31.555 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.555 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1259b382:::benchmark_data_smithi179_148205_object7392:head# 2024-02-09T17:57:31.555 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.555 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1259f051:::benchmark_data_smithi179_148205_object18453:head# 2024-02-09T17:57:31.555 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.555 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:125a82bc:::benchmark_data_smithi179_148205_object22108:head# 2024-02-09T17:57:31.555 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.555 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:125c3516:::benchmark_data_smithi179_148205_object106266:head# 2024-02-09T17:57:31.555 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.555 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:125d590d:::benchmark_data_smithi179_148205_object107985:head# 2024-02-09T17:57:31.555 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.555 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:125dca70:::benchmark_data_smithi179_148205_object58528:head# 2024-02-09T17:57:31.555 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.556 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:125e32e3:::benchmark_data_smithi179_148205_object87586:head# 2024-02-09T17:57:31.556 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.556 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:125ef662:::benchmark_data_smithi179_148205_object100647:head# 2024-02-09T17:57:31.556 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.556 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:125f38b0:::benchmark_data_smithi179_148205_object41764:head# 2024-02-09T17:57:31.556 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.556 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12606394:::benchmark_data_smithi179_148205_object12265:head# 2024-02-09T17:57:31.556 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.556 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1260e713:::benchmark_data_smithi179_148205_object10275:head# 2024-02-09T17:57:31.556 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.556 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12612593:::benchmark_data_smithi179_148205_object45265:head# 2024-02-09T17:57:31.556 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.566 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1261be7a:::benchmark_data_smithi179_148205_object26301:head# 2024-02-09T17:57:31.566 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.566 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12631380:::benchmark_data_smithi179_148205_object14755:head# 2024-02-09T17:57:31.566 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.566 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1264162f:::benchmark_data_smithi179_148205_object101405:head# 2024-02-09T17:57:31.567 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.567 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12643118:::benchmark_data_smithi179_148205_object104067:head# 2024-02-09T17:57:31.567 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.567 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12644c46:::benchmark_data_smithi179_148205_object42392:head# 2024-02-09T17:57:31.567 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.567 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12648356:::benchmark_data_smithi179_148205_object99573:head# 2024-02-09T17:57:31.567 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.567 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1265dfba:::benchmark_data_smithi179_148205_object100856:head# 2024-02-09T17:57:31.567 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.567 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:126654ea:::benchmark_data_smithi179_148205_object31606:head# 2024-02-09T17:57:31.567 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.567 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1266ab63:::benchmark_data_smithi179_148205_object106356:head# 2024-02-09T17:57:31.567 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.568 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:126741d0:::benchmark_data_smithi179_148205_object58741:head# 2024-02-09T17:57:31.568 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.568 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12683aa9:::benchmark_data_smithi179_148205_object78704:head# 2024-02-09T17:57:31.568 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.568 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12687130:::benchmark_data_smithi179_148205_object49425:head# 2024-02-09T17:57:31.568 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.568 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1268f603:::benchmark_data_smithi179_148205_object61682:head# 2024-02-09T17:57:31.568 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.568 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:126a3465:::benchmark_data_smithi179_148205_object104026:head# 2024-02-09T17:57:31.568 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.568 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:126a362c:::benchmark_data_smithi179_148205_object65609:head# 2024-02-09T17:57:31.568 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.568 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:126a4b65:::benchmark_data_smithi179_148205_object58504:head# 2024-02-09T17:57:31.568 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.569 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:126ac55c:::benchmark_data_smithi179_148205_object97070:head# 2024-02-09T17:57:31.569 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.569 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:126b0ed6:::benchmark_data_smithi179_148205_object17210:head# 2024-02-09T17:57:31.569 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.569 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:126b3e18:::benchmark_data_smithi179_148205_object68161:head# 2024-02-09T17:57:31.569 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.569 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:126b896b:::benchmark_data_smithi179_148205_object64244:head# 2024-02-09T17:57:31.569 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.569 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:126bce3e:::benchmark_data_smithi179_148205_object59730:head# 2024-02-09T17:57:31.569 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.569 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:126c06ca:::benchmark_data_smithi179_148205_object74340:head# 2024-02-09T17:57:31.569 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.569 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:126c1eb9:::benchmark_data_smithi179_148205_object44026:head# 2024-02-09T17:57:31.569 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.569 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:126cd756:::benchmark_data_smithi179_148205_object31088:head# 2024-02-09T17:57:31.570 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.570 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:126d0743:::benchmark_data_smithi179_148205_object94277:head# 2024-02-09T17:57:31.570 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.570 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:126d6935:::benchmark_data_smithi179_148205_object29424:head# 2024-02-09T17:57:31.570 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.570 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:126da9e0:::benchmark_data_smithi179_148205_object23314:head# 2024-02-09T17:57:31.570 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.570 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:126e43e6:::benchmark_data_smithi179_148205_object66198:head# 2024-02-09T17:57:31.570 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.570 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:126e8822:::benchmark_data_smithi179_148205_object50017:head# 2024-02-09T17:57:31.570 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.571 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:126eb615:::benchmark_data_smithi179_148205_object29347:head# 2024-02-09T17:57:31.571 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.571 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:126f12b9:::benchmark_data_smithi179_148205_object36563:head# 2024-02-09T17:57:31.571 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.571 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:126f493b:::benchmark_data_smithi179_148205_object49311:head# 2024-02-09T17:57:31.571 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.571 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:126fa632:::benchmark_data_smithi179_148205_object85518:head# 2024-02-09T17:57:31.571 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.571 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:127001d7:::benchmark_data_smithi179_148205_object57137:head# 2024-02-09T17:57:31.571 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.571 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12716676:::benchmark_data_smithi179_148205_object100454:head# 2024-02-09T17:57:31.571 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.571 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1271e76e:::benchmark_data_smithi179_148205_object97335:head# 2024-02-09T17:57:31.572 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.572 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12726dad:::benchmark_data_smithi179_148205_object67101:head# 2024-02-09T17:57:31.572 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.572 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1272ff2b:::benchmark_data_smithi179_148205_object69019:head# 2024-02-09T17:57:31.572 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.572 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12749f7e:::benchmark_data_smithi179_148205_object10674:head# 2024-02-09T17:57:31.572 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.572 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1276af5e:::benchmark_data_smithi179_148205_object91100:head# 2024-02-09T17:57:31.572 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.572 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1277374b:::benchmark_data_smithi179_148205_object18181:head# 2024-02-09T17:57:31.572 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.572 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:127750c4:::benchmark_data_smithi179_148205_object48391:head# 2024-02-09T17:57:31.572 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.572 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1277d4cd:::benchmark_data_smithi179_148205_object27750:head# 2024-02-09T17:57:31.573 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.573 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12782b21:::benchmark_data_smithi179_148205_object18872:head# 2024-02-09T17:57:31.573 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.573 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12788949:::benchmark_data_smithi179_148205_object51065:head# 2024-02-09T17:57:31.573 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.573 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:127921c9:::benchmark_data_smithi179_148205_object4940:head# 2024-02-09T17:57:31.573 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.573 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:127a00b3:::benchmark_data_smithi179_148205_object65844:head# 2024-02-09T17:57:31.573 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.573 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:127ab06b:::benchmark_data_smithi179_148205_object22103:head# 2024-02-09T17:57:31.573 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.573 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:127af979:::benchmark_data_smithi179_148205_object38247:head# 2024-02-09T17:57:31.573 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.573 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:127b62f0:::benchmark_data_smithi179_148205_object48999:head# 2024-02-09T17:57:31.574 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.574 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:127c3e10:::benchmark_data_smithi179_148205_object46492:head# 2024-02-09T17:57:31.574 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.574 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:127c5948:::benchmark_data_smithi179_148205_object21137:head# 2024-02-09T17:57:31.574 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.574 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:127d300e:::benchmark_data_smithi179_148205_object30040:head# 2024-02-09T17:57:31.574 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.574 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:127d325c:::benchmark_data_smithi179_148205_object40381:head# 2024-02-09T17:57:31.574 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.574 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:127dca9d:::benchmark_data_smithi179_148205_object44982:head# 2024-02-09T17:57:31.574 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.574 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:127e359c:::benchmark_data_smithi179_148205_object4005:head# 2024-02-09T17:57:31.574 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.574 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:127ea8ad:::benchmark_data_smithi179_148205_object39581:head# 2024-02-09T17:57:31.574 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.575 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:127ee740:::benchmark_data_smithi179_148205_object26235:head# 2024-02-09T17:57:31.575 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.575 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:128008c7:::benchmark_data_smithi179_148205_object16880:head# 2024-02-09T17:57:31.575 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.577 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1280362e:::benchmark_data_smithi179_148205_object105018:head# 2024-02-09T17:57:31.577 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.577 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:128040e5:::benchmark_data_smithi179_148205_object28290:head# 2024-02-09T17:57:31.578 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.578 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12809ccb:::benchmark_data_smithi179_148205_object48371:head# 2024-02-09T17:57:31.578 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.578 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1280d548:::benchmark_data_smithi179_148205_object48327:head# 2024-02-09T17:57:31.578 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.578 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1280ee8a:::benchmark_data_smithi179_148205_object21101:head# 2024-02-09T17:57:31.578 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.578 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1281d20d:::benchmark_data_smithi179_148205_object67948:head# 2024-02-09T17:57:31.578 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.578 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:128259d8:::benchmark_data_smithi179_148205_object84399:head# 2024-02-09T17:57:31.578 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.578 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1282cabc:::benchmark_data_smithi179_148205_object100402:head# 2024-02-09T17:57:31.578 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.578 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12835ed9:::benchmark_data_smithi179_148205_object107640:head# 2024-02-09T17:57:31.578 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.579 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12857a75:::benchmark_data_smithi179_148205_object56569:head# 2024-02-09T17:57:31.579 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.579 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1285c2f2:::benchmark_data_smithi179_148205_object5181:head# 2024-02-09T17:57:31.579 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.579 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1285e5a3:::benchmark_data_smithi179_148205_object91597:head# 2024-02-09T17:57:31.579 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.579 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12876757:::benchmark_data_smithi179_148205_object19951:head# 2024-02-09T17:57:31.579 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.579 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:128769ef:::benchmark_data_smithi179_148205_object100217:head# 2024-02-09T17:57:31.579 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.579 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1287b6d0:::benchmark_data_smithi179_148205_object78436:head# 2024-02-09T17:57:31.579 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.579 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12895267:::benchmark_data_smithi179_148205_object10302:head# 2024-02-09T17:57:31.579 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.580 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1289ddd2:::benchmark_data_smithi179_148205_object81962:head# 2024-02-09T17:57:31.580 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.582 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:128aba66:::benchmark_data_smithi179_148205_object46020:head# 2024-02-09T17:57:31.582 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.582 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:128ace04:::benchmark_data_smithi179_148205_object37056:head# 2024-02-09T17:57:31.582 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.582 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:128bc92e:::benchmark_data_smithi179_148205_object41039:head# 2024-02-09T17:57:31.582 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.583 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:128c5cd5:::benchmark_data_smithi179_148205_object1603:head# 2024-02-09T17:57:31.583 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.583 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:128cb753:::benchmark_data_smithi179_148205_object86886:head# 2024-02-09T17:57:31.583 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.583 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:128d5431:::benchmark_data_smithi179_148205_object47077:head# 2024-02-09T17:57:31.583 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.583 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:128e58d9:::benchmark_data_smithi179_148205_object70087:head# 2024-02-09T17:57:31.583 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.583 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:128ee542:::benchmark_data_smithi179_148205_object61272:head# 2024-02-09T17:57:31.583 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.583 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:128f776d:::benchmark_data_smithi179_148205_object106970:head# 2024-02-09T17:57:31.583 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.583 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:128f7b4b:::benchmark_data_smithi179_148205_object93127:head# 2024-02-09T17:57:31.584 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.584 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:128ffe79:::benchmark_data_smithi179_148205_object84735:head# 2024-02-09T17:57:31.584 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.584 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:129006ed:::benchmark_data_smithi179_148205_object77382:head# 2024-02-09T17:57:31.584 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.584 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12900706:::benchmark_data_smithi179_148205_object32715:head# 2024-02-09T17:57:31.584 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.584 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12914830:::benchmark_data_smithi179_148205_object2057:head# 2024-02-09T17:57:31.584 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.584 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12914e54:::benchmark_data_smithi179_148205_object43143:head# 2024-02-09T17:57:31.584 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.584 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12919cac:::benchmark_data_smithi179_148205_object47847:head# 2024-02-09T17:57:31.584 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.584 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12923dde:::benchmark_data_smithi179_148205_object69129:head# 2024-02-09T17:57:31.585 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.585 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1292582d:::benchmark_data_smithi179_148205_object38745:head# 2024-02-09T17:57:31.585 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.585 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1293f8b4:::benchmark_data_smithi179_148205_object53672:head# 2024-02-09T17:57:31.585 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.587 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1294dffc:::benchmark_data_smithi179_148205_object50672:head# 2024-02-09T17:57:31.587 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.587 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12952dad:::benchmark_data_smithi179_148205_object29394:head# 2024-02-09T17:57:31.587 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.587 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12966a9b:::benchmark_data_smithi179_148205_object1902:head# 2024-02-09T17:57:31.587 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.587 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1297c90f:::benchmark_data_smithi179_148205_object39351:head# 2024-02-09T17:57:31.587 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.587 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:129827a9:::benchmark_data_smithi179_148205_object64057:head# 2024-02-09T17:57:31.587 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.587 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1298412e:::benchmark_data_smithi179_148205_object75740:head# 2024-02-09T17:57:31.587 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.588 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12985efa:::benchmark_data_smithi179_148205_object66717:head# 2024-02-09T17:57:31.588 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.588 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:129a0616:::benchmark_data_smithi179_148205_object13530:head# 2024-02-09T17:57:31.588 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.588 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:129a80d7:::benchmark_data_smithi179_148205_object44754:head# 2024-02-09T17:57:31.588 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.588 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:129af014:::benchmark_data_smithi179_148205_object28043:head# 2024-02-09T17:57:31.588 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.588 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:129af619:::benchmark_data_smithi179_148205_object31080:head# 2024-02-09T17:57:31.588 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.588 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:129b51c6:::benchmark_data_smithi179_148205_object41663:head# 2024-02-09T17:57:31.588 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.588 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:129be3b6:::benchmark_data_smithi179_148205_object49610:head# 2024-02-09T17:57:31.589 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.589 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:129c37d1:::benchmark_data_smithi179_148205_object64621:head# 2024-02-09T17:57:31.589 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.589 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:129c6a9b:::benchmark_data_smithi179_148205_object97421:head# 2024-02-09T17:57:31.589 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.589 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:129ce399:::benchmark_data_smithi179_148205_object77307:head# 2024-02-09T17:57:31.589 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.589 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:129e808f:::benchmark_data_smithi179_148205_object22116:head# 2024-02-09T17:57:31.589 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.591 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:129fa649:::benchmark_data_smithi179_148205_object7431:head# 2024-02-09T17:57:31.591 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.591 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12a0276a:::benchmark_data_smithi179_148205_object103013:head# 2024-02-09T17:57:31.591 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.591 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12a0977e:::benchmark_data_smithi179_148205_object85233:head# 2024-02-09T17:57:31.591 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.591 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12a22520:::benchmark_data_smithi179_148205_object39338:head# 2024-02-09T17:57:31.591 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.591 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12a22f18:::benchmark_data_smithi179_148205_object89056:head# 2024-02-09T17:57:31.592 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.592 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12a2551f:::benchmark_data_smithi179_148205_object34409:head# 2024-02-09T17:57:31.592 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.592 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12a3e397:::benchmark_data_smithi179_148205_object86180:head# 2024-02-09T17:57:31.592 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.592 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12a4746b:::benchmark_data_smithi179_148205_object89073:head# 2024-02-09T17:57:31.592 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.592 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12a4df43:::benchmark_data_smithi179_148205_object67453:head# 2024-02-09T17:57:31.592 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.592 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12a7a2b9:::benchmark_data_smithi179_148205_object33869:head# 2024-02-09T17:57:31.592 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.592 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12a8a821:::benchmark_data_smithi179_148205_object42134:head# 2024-02-09T17:57:31.592 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.592 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12a97188:::benchmark_data_smithi179_148205_object6215:head# 2024-02-09T17:57:31.593 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.593 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12aa4b2a:::benchmark_data_smithi179_148205_object58515:head# 2024-02-09T17:57:31.593 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.593 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12aad64f:::benchmark_data_smithi179_148205_object7493:head# 2024-02-09T17:57:31.593 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.593 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12aaf7c4:::benchmark_data_smithi179_148205_object20637:head# 2024-02-09T17:57:31.593 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.593 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12acbe79:::benchmark_data_smithi179_148205_object52129:head# 2024-02-09T17:57:31.593 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.596 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12ae2255:::benchmark_data_smithi179_148205_object46655:head# 2024-02-09T17:57:31.597 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.597 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12aeed36:::benchmark_data_smithi179_148205_object18785:head# 2024-02-09T17:57:31.597 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.597 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12af425e:::benchmark_data_smithi179_148205_object7192:head# 2024-02-09T17:57:31.597 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.597 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12af570a:::benchmark_data_smithi179_148205_object43705:head# 2024-02-09T17:57:31.597 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.597 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12af954b:::benchmark_data_smithi179_148205_object75809:head# 2024-02-09T17:57:31.597 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.597 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12afcd83:::benchmark_data_smithi179_148205_object34591:head# 2024-02-09T17:57:31.597 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.597 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12b039f9:::benchmark_data_smithi179_148205_object37066:head# 2024-02-09T17:57:31.597 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.598 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12b0a143:::benchmark_data_smithi179_148205_object13120:head# 2024-02-09T17:57:31.598 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.598 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12b0f21f:::benchmark_data_smithi179_148205_object31160:head# 2024-02-09T17:57:31.598 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.598 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12b10aa6:::benchmark_data_smithi179_148205_object17928:head# 2024-02-09T17:57:31.598 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.598 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12b13cfc:::benchmark_data_smithi179_148205_object55109:head# 2024-02-09T17:57:31.598 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.598 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12b15e36:::benchmark_data_smithi179_148205_object34777:head# 2024-02-09T17:57:31.598 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.598 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12b1ea6b:::benchmark_data_smithi179_148205_object9153:head# 2024-02-09T17:57:31.598 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.598 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12b30aae:::benchmark_data_smithi179_148205_object107214:head# 2024-02-09T17:57:31.598 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.599 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12b435bb:::benchmark_data_smithi179_148205_object103503:head# 2024-02-09T17:57:31.599 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.599 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12b52140:::benchmark_data_smithi179_148205_object82148:head# 2024-02-09T17:57:31.599 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.599 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12b59d51:::benchmark_data_smithi179_148205_object109335:head# 2024-02-09T17:57:31.599 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.599 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12b6f1c5:::benchmark_data_smithi179_148205_object7139:head# 2024-02-09T17:57:31.599 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.599 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12b75dac:::benchmark_data_smithi179_148205_object86403:head# 2024-02-09T17:57:31.599 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.599 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12b7c414:::benchmark_data_smithi179_148205_object55388:head# 2024-02-09T17:57:31.599 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.602 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12b85048:::benchmark_data_smithi179_148205_object28000:head# 2024-02-09T17:57:31.602 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.602 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12b85d47:::benchmark_data_smithi179_148205_object52785:head# 2024-02-09T17:57:31.602 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.602 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12b8815d:::benchmark_data_smithi179_148205_object19905:head# 2024-02-09T17:57:31.602 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.602 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12b9a3c1:::benchmark_data_smithi179_148205_object85131:head# 2024-02-09T17:57:31.602 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.602 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12b9f1a2:::benchmark_data_smithi179_148205_object80282:head# 2024-02-09T17:57:31.602 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.603 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12ba8efa:::benchmark_data_smithi179_148205_object82430:head# 2024-02-09T17:57:31.603 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.603 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12bb0da9:::benchmark_data_smithi179_148205_object94199:head# 2024-02-09T17:57:31.603 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.603 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12bb422c:::benchmark_data_smithi179_148205_object78007:head# 2024-02-09T17:57:31.603 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.603 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12bc5a3e:::benchmark_data_smithi179_148205_object24180:head# 2024-02-09T17:57:31.603 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.603 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12bdb398:::benchmark_data_smithi179_148205_object81855:head# 2024-02-09T17:57:31.603 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.603 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12bf6215:::benchmark_data_smithi179_148205_object100911:head# 2024-02-09T17:57:31.603 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.603 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12bfbfb0:::benchmark_data_smithi179_148205_object47426:head# 2024-02-09T17:57:31.603 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.604 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12c1cb89:::benchmark_data_smithi179_148205_object92354:head# 2024-02-09T17:57:31.604 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.604 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12c1de95:::benchmark_data_smithi179_148205_object15464:head# 2024-02-09T17:57:31.604 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.604 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12c24e24:::benchmark_data_smithi179_148205_object97624:head# 2024-02-09T17:57:31.604 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.604 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12c26d40:::benchmark_data_smithi179_148205_object46755:head# 2024-02-09T17:57:31.604 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.608 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12c2d9ed:::benchmark_data_smithi179_148205_object83203:head# 2024-02-09T17:57:31.608 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.608 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12c30a15:::benchmark_data_smithi179_148205_object59799:head# 2024-02-09T17:57:31.608 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.608 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12c3968e:::benchmark_data_smithi179_148205_object105539:head# 2024-02-09T17:57:31.608 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.609 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12c3e6b9:::benchmark_data_smithi179_148205_object16878:head# 2024-02-09T17:57:31.609 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.609 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12c400ec:::benchmark_data_smithi179_148205_object34136:head# 2024-02-09T17:57:31.609 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.609 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12c41e71:::benchmark_data_smithi179_148205_object57753:head# 2024-02-09T17:57:31.609 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.609 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12c47328:::benchmark_data_smithi179_148205_object14318:head# 2024-02-09T17:57:31.609 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.609 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12c51a04:::benchmark_data_smithi179_148205_object102955:head# 2024-02-09T17:57:31.609 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.609 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12c5586c:::benchmark_data_smithi179_148205_object96860:head# 2024-02-09T17:57:31.609 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.609 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12c56141:::benchmark_data_smithi179_148205_object60979:head# 2024-02-09T17:57:31.609 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.610 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12c561e5:::benchmark_data_smithi179_148205_object38858:head# 2024-02-09T17:57:31.610 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.610 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12c58838:::benchmark_data_smithi179_148205_object109663:head# 2024-02-09T17:57:31.610 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.610 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12c6704b:::benchmark_data_smithi179_148205_object91315:head# 2024-02-09T17:57:31.610 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.610 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12c6a16e:::benchmark_data_smithi179_148205_object82125:head# 2024-02-09T17:57:31.610 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.610 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12c6e479:::benchmark_data_smithi179_148205_object19291:head# 2024-02-09T17:57:31.610 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.610 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12c80300:::benchmark_data_smithi179_148205_object108222:head# 2024-02-09T17:57:31.610 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.610 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12c869bd:::benchmark_data_smithi179_148205_object74531:head# 2024-02-09T17:57:31.610 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.611 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12c87d43:::benchmark_data_smithi179_148205_object47619:head# 2024-02-09T17:57:31.611 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.611 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12c895ae:::benchmark_data_smithi179_148205_object64097:head# 2024-02-09T17:57:31.611 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.611 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12c8a69d:::benchmark_data_smithi179_148205_object93188:head# 2024-02-09T17:57:31.611 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.611 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12c8b659:::benchmark_data_smithi179_148205_object30607:head# 2024-02-09T17:57:31.611 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.611 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12c94b10:::benchmark_data_smithi179_148205_object32711:head# 2024-02-09T17:57:31.611 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.611 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12c9a9bb:::benchmark_data_smithi179_148205_object23514:head# 2024-02-09T17:57:31.611 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.611 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12ca5d71:::benchmark_data_smithi179_148205_object83047:head# 2024-02-09T17:57:31.611 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.612 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12cb5cde:::benchmark_data_smithi179_148205_object21769:head# 2024-02-09T17:57:31.612 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.614 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12cbaccc:::benchmark_data_smithi179_148205_object76503:head# 2024-02-09T17:57:31.614 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.614 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12cbb201:::benchmark_data_smithi179_148205_object62366:head# 2024-02-09T17:57:31.614 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.614 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12cbb2e2:::benchmark_data_smithi179_148205_object107276:head# 2024-02-09T17:57:31.614 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.614 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12cbc1a2:::benchmark_data_smithi179_148205_object79844:head# 2024-02-09T17:57:31.614 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.615 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12cbef5e:::benchmark_data_smithi179_148205_object9635:head# 2024-02-09T17:57:31.615 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.615 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12ccab2a:::benchmark_data_smithi179_148205_object89757:head# 2024-02-09T17:57:31.615 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.615 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12ccc712:::benchmark_data_smithi179_148205_object92602:head# 2024-02-09T17:57:31.615 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.615 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12cdcfd7:::benchmark_data_smithi179_148205_object25840:head# 2024-02-09T17:57:31.615 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.615 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12cf8620:::benchmark_data_smithi179_148205_object64780:head# 2024-02-09T17:57:31.615 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.615 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12cf98c0:::benchmark_data_smithi179_148205_object104622:head# 2024-02-09T17:57:31.615 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.615 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12cf9de8:::benchmark_data_smithi179_148205_object53317:head# 2024-02-09T17:57:31.615 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.616 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12d09007:::benchmark_data_smithi179_148205_object55839:head# 2024-02-09T17:57:31.616 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.616 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12d0fffd:::benchmark_data_smithi179_148205_object26607:head# 2024-02-09T17:57:31.616 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.616 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12d23db8:::benchmark_data_smithi179_148205_object75396:head# 2024-02-09T17:57:31.616 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.616 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12d34506:::benchmark_data_smithi179_148205_object7866:head# 2024-02-09T17:57:31.616 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.618 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12d40aba:::benchmark_data_smithi179_148205_object29430:head# 2024-02-09T17:57:31.618 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.618 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12d45357:::benchmark_data_smithi179_148205_object74055:head# 2024-02-09T17:57:31.618 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.619 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12d5172d:::benchmark_data_smithi179_148205_object40382:head# 2024-02-09T17:57:31.619 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.619 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12d69baf:::benchmark_data_smithi179_148205_object69654:head# 2024-02-09T17:57:31.619 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.619 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12d6b7fc:::benchmark_data_smithi179_148205_object61134:head# 2024-02-09T17:57:31.619 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.619 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12d6fc31:::benchmark_data_smithi179_148205_object13646:head# 2024-02-09T17:57:31.619 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.619 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12d8fefa:::benchmark_data_smithi179_148205_object55634:head# 2024-02-09T17:57:31.619 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.619 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12d98202:::benchmark_data_smithi179_148205_object57462:head# 2024-02-09T17:57:31.619 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.623 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12da7254:::benchmark_data_smithi179_148205_object65479:head# 2024-02-09T17:57:31.623 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.623 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12daf8fb:::benchmark_data_smithi179_148205_object82903:head# 2024-02-09T17:57:31.623 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.623 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12dcbc7e:::benchmark_data_smithi179_148205_object87667:head# 2024-02-09T17:57:31.623 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.623 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12dcf565:::benchmark_data_smithi179_148205_object107411:head# 2024-02-09T17:57:31.623 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.623 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12dd517c:::benchmark_data_smithi179_148205_object22304:head# 2024-02-09T17:57:31.624 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.624 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12de16e3:::benchmark_data_smithi179_148205_object72913:head# 2024-02-09T17:57:31.624 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.624 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12df4224:::benchmark_data_smithi179_148205_object52536:head# 2024-02-09T17:57:31.624 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.624 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12df9a25:::benchmark_data_smithi179_148205_object25248:head# 2024-02-09T17:57:31.624 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.624 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12dfb847:::benchmark_data_smithi179_148205_object106297:head# 2024-02-09T17:57:31.624 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.624 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12dfd150:::benchmark_data_smithi179_148205_object45026:head# 2024-02-09T17:57:31.624 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.624 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12e01c71:::benchmark_data_smithi179_148205_object36419:head# 2024-02-09T17:57:31.624 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.624 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12e09697:::benchmark_data_smithi179_148205_object75583:head# 2024-02-09T17:57:31.625 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.625 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12e0b81d:::benchmark_data_smithi179_148205_object3033:head# 2024-02-09T17:57:31.626 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.626 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12e13467:::benchmark_data_smithi179_148205_object3787:head# 2024-02-09T17:57:31.626 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.626 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12e279ba:::benchmark_data_smithi179_148205_object60316:head# 2024-02-09T17:57:31.626 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.626 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12e2fd4a:::benchmark_data_smithi179_148205_object45694:head# 2024-02-09T17:57:31.626 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.626 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12e3124b:::benchmark_data_smithi179_148205_object59126:head# 2024-02-09T17:57:31.626 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.628 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12e384d7:::benchmark_data_smithi179_148205_object84907:head# 2024-02-09T17:57:31.629 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.631 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12e532fb:::benchmark_data_smithi179_148205_object24842:head# 2024-02-09T17:57:31.631 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.631 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12e58840:::benchmark_data_smithi179_148205_object27860:head# 2024-02-09T17:57:31.631 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.631 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12e5baa8:::benchmark_data_smithi179_148205_object35122:head# 2024-02-09T17:57:31.631 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.632 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12e5e5d4:::benchmark_data_smithi179_148205_object59681:head# 2024-02-09T17:57:31.632 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.632 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12e6f01a:::benchmark_data_smithi179_148205_object105203:head# 2024-02-09T17:57:31.632 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.632 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12e94ec3:::benchmark_data_smithi179_148205_object35341:head# 2024-02-09T17:57:31.632 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.633 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12e967a5:::benchmark_data_smithi179_148205_object79391:head# 2024-02-09T17:57:31.633 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.633 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12ea36e6:::benchmark_data_smithi179_148205_object79572:head# 2024-02-09T17:57:31.633 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.634 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12ea42da:::benchmark_data_smithi179_148205_object96456:head# 2024-02-09T17:57:31.634 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.634 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12ea45c9:::benchmark_data_smithi179_148205_object89752:head# 2024-02-09T17:57:31.634 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.634 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12eaa2c7:::benchmark_data_smithi179_148205_object20203:head# 2024-02-09T17:57:31.634 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.634 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12eafedb:::benchmark_data_smithi179_148205_object89809:head# 2024-02-09T17:57:31.634 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.635 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12ebf682:::benchmark_data_smithi179_148205_object1628:head# 2024-02-09T17:57:31.635 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.635 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12ec18a5:::benchmark_data_smithi179_148205_object57528:head# 2024-02-09T17:57:31.635 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.635 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12ec476f:::benchmark_data_smithi179_148205_object105927:head# 2024-02-09T17:57:31.635 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.637 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12ec61ca:::benchmark_data_smithi179_148205_object4479:head# 2024-02-09T17:57:31.637 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.637 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12ed2836:::benchmark_data_smithi179_148205_object12709:head# 2024-02-09T17:57:31.637 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.637 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12eeb65e:::benchmark_data_smithi179_148205_object23133:head# 2024-02-09T17:57:31.637 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.637 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12efcd54:::benchmark_data_smithi179_148205_object63275:head# 2024-02-09T17:57:31.637 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.637 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12f0992b:::benchmark_data_smithi179_148205_object97156:head# 2024-02-09T17:57:31.637 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.637 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12f09d43:::benchmark_data_smithi179_148205_object106863:head# 2024-02-09T17:57:31.637 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.637 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12f0e3fc:::benchmark_data_smithi179_148205_object63429:head# 2024-02-09T17:57:31.637 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.638 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12f1e64d:::benchmark_data_smithi179_148205_object18941:head# 2024-02-09T17:57:31.638 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.638 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12f1fcbe:::benchmark_data_smithi179_148205_object36701:head# 2024-02-09T17:57:31.638 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.638 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12f22dd5:::benchmark_data_smithi179_148205_object5433:head# 2024-02-09T17:57:31.638 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.638 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12f22fc0:::benchmark_data_smithi179_148205_object52952:head# 2024-02-09T17:57:31.638 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.642 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12f40847:::benchmark_data_smithi179_148205_object19034:head# 2024-02-09T17:57:31.642 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.642 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12f471b1:::benchmark_data_smithi179_148205_object87630:head# 2024-02-09T17:57:31.642 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.642 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12f587da:::benchmark_data_smithi179_148205_object41584:head# 2024-02-09T17:57:31.642 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.642 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12f5b57e:::benchmark_data_smithi179_148205_object1742:head# 2024-02-09T17:57:31.642 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.642 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12f68670:::benchmark_data_smithi179_148205_object31360:head# 2024-02-09T17:57:31.642 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.642 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12f6bad2:::benchmark_data_smithi179_148205_object11489:head# 2024-02-09T17:57:31.642 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.642 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12f7259a:::benchmark_data_smithi179_148205_object72014:head# 2024-02-09T17:57:31.643 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.643 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12f72a82:::benchmark_data_smithi179_148205_object32255:head# 2024-02-09T17:57:31.643 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.643 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12f75678:::benchmark_data_smithi179_148205_object69954:head# 2024-02-09T17:57:31.643 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.643 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12f7d7e8:::benchmark_data_smithi179_148205_object29092:head# 2024-02-09T17:57:31.643 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.643 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12f86730:::benchmark_data_smithi179_148205_object22543:head# 2024-02-09T17:57:31.643 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.643 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12f921e0:::benchmark_data_smithi179_148205_object31028:head# 2024-02-09T17:57:31.643 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.643 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12f94fbf:::benchmark_data_smithi179_148205_object36290:head# 2024-02-09T17:57:31.643 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.643 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12f96606:::benchmark_data_smithi179_148205_object53669:head# 2024-02-09T17:57:31.644 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.644 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12fa7269:::benchmark_data_smithi179_148205_object6143:head# 2024-02-09T17:57:31.644 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.644 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12faf90c:::benchmark_data_smithi179_148205_object21185:head# 2024-02-09T17:57:31.644 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.644 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12fbed47:::benchmark_data_smithi179_148205_object81535:head# 2024-02-09T17:57:31.644 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.644 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12fc69cb:::benchmark_data_smithi179_148205_object45924:head# 2024-02-09T17:57:31.644 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.644 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12fc79c5:::benchmark_data_smithi179_148205_object29124:head# 2024-02-09T17:57:31.644 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.644 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12fcbe30:::benchmark_data_smithi179_148205_object17363:head# 2024-02-09T17:57:31.644 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.646 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12fd9112:::benchmark_data_smithi179_148205_object105337:head# 2024-02-09T17:57:31.646 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.646 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12fe2fc9:::benchmark_data_smithi179_148205_object44651:head# 2024-02-09T17:57:31.646 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.646 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12fe38ac:::benchmark_data_smithi179_148205_object3338:head# 2024-02-09T17:57:31.646 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.646 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12fe4ada:::benchmark_data_smithi179_148205_object101364:head# 2024-02-09T17:57:31.646 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.647 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:12feeacb:::benchmark_data_smithi179_148205_object93371:head# 2024-02-09T17:57:31.647 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.647 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1300e95b:::benchmark_data_smithi179_148205_object67671:head# 2024-02-09T17:57:31.647 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.647 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13010c54:::benchmark_data_smithi179_148205_object70768:head# 2024-02-09T17:57:31.647 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.647 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1301aae5:::benchmark_data_smithi179_148205_object24807:head# 2024-02-09T17:57:31.647 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.647 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1302ee88:::benchmark_data_smithi179_148205_object31681:head# 2024-02-09T17:57:31.647 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.647 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:130383f4:::benchmark_data_smithi179_148205_object60143:head# 2024-02-09T17:57:31.647 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.649 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13038822:::benchmark_data_smithi179_148205_object95216:head# 2024-02-09T17:57:31.649 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.650 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1304be46:::benchmark_data_smithi179_148205_object56323:head# 2024-02-09T17:57:31.650 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.650 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1305b414:::benchmark_data_smithi179_148205_object85951:head# 2024-02-09T17:57:31.651 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.651 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1305bb4f:::benchmark_data_smithi179_148205_object21724:head# 2024-02-09T17:57:31.651 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.651 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13060308:::benchmark_data_smithi179_148205_object18516:head# 2024-02-09T17:57:31.651 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.651 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13067e0e:::benchmark_data_smithi179_148205_object39365:head# 2024-02-09T17:57:31.651 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.653 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:130752c4:::benchmark_data_smithi179_148205_object50806:head# 2024-02-09T17:57:31.653 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.653 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13085f57:::benchmark_data_smithi179_148205_object89777:head# 2024-02-09T17:57:31.653 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.653 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13090e12:::benchmark_data_smithi179_148205_object29272:head# 2024-02-09T17:57:31.653 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.653 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13099313:::benchmark_data_smithi179_148205_object2765:head# 2024-02-09T17:57:31.653 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.653 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1309abb0:::benchmark_data_smithi179_148205_object40708:head# 2024-02-09T17:57:31.653 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.654 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1309dcea:::benchmark_data_smithi179_148205_object89238:head# 2024-02-09T17:57:31.654 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.654 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:130b446d:::benchmark_data_smithi179_148205_object43787:head# 2024-02-09T17:57:31.654 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.654 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:130c70b8:::benchmark_data_smithi179_148205_object39300:head# 2024-02-09T17:57:31.654 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.654 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:130c9b64:::benchmark_data_smithi179_148205_object52845:head# 2024-02-09T17:57:31.654 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.654 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:130ca56d:::benchmark_data_smithi179_148205_object58310:head# 2024-02-09T17:57:31.654 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.658 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:130ca64f:::benchmark_data_smithi179_148205_object77591:head# 2024-02-09T17:57:31.658 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.658 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:130cf7e7:::benchmark_data_smithi179_148205_object22210:head# 2024-02-09T17:57:31.658 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.658 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:130cf8e1:::benchmark_data_smithi179_148205_object37910:head# 2024-02-09T17:57:31.658 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.658 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:130d15e0:::benchmark_data_smithi179_148205_object62809:head# 2024-02-09T17:57:31.658 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.658 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:130e9f29:::benchmark_data_smithi179_148205_object57852:head# 2024-02-09T17:57:31.658 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.658 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13102830:::benchmark_data_smithi179_148205_object34701:head# 2024-02-09T17:57:31.659 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.659 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1311dddf:::benchmark_data_smithi179_148205_object2516:head# 2024-02-09T17:57:31.659 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.659 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13130211:::benchmark_data_smithi179_148205_object92989:head# 2024-02-09T17:57:31.659 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.659 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1313606f:::benchmark_data_smithi179_148205_object54270:head# 2024-02-09T17:57:31.659 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.659 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1313b9cf:::benchmark_data_smithi179_148205_object97814:head# 2024-02-09T17:57:31.659 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.659 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1315ba69:::benchmark_data_smithi179_148205_object46965:head# 2024-02-09T17:57:31.659 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.659 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1316f50a:::benchmark_data_smithi179_148205_object75580:head# 2024-02-09T17:57:31.659 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.659 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1316ffd9:::benchmark_data_smithi179_148205_object103183:head# 2024-02-09T17:57:31.660 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.660 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13186b3f:::benchmark_data_smithi179_148205_object80115:head# 2024-02-09T17:57:31.660 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.660 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1318a619:::benchmark_data_smithi179_148205_object41533:head# 2024-02-09T17:57:31.660 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.660 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:131902e1:::benchmark_data_smithi179_148205_object71784:head# 2024-02-09T17:57:31.660 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.660 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13194868:::benchmark_data_smithi179_148205_object14127:head# 2024-02-09T17:57:31.660 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.660 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1319fab2:::benchmark_data_smithi179_148205_object82272:head# 2024-02-09T17:57:31.660 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.660 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:131a39d4:::benchmark_data_smithi179_148205_object81428:head# 2024-02-09T17:57:31.660 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.661 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:131ae5c1:::benchmark_data_smithi179_148205_object5299:head# 2024-02-09T17:57:31.661 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.661 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:131b6bf6:::benchmark_data_smithi179_148205_object46711:head# 2024-02-09T17:57:31.661 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.662 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:131c4aa3:::benchmark_data_smithi179_148205_object74738:head# 2024-02-09T17:57:31.662 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.662 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:131c803d:::benchmark_data_smithi179_148205_object69790:head# 2024-02-09T17:57:31.662 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.662 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:131ca151:::benchmark_data_smithi179_148205_object15947:head# 2024-02-09T17:57:31.662 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.662 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:131d00d1:::benchmark_data_smithi179_148205_object97789:head# 2024-02-09T17:57:31.662 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.662 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:131d433c:::benchmark_data_smithi179_148205_object27985:head# 2024-02-09T17:57:31.662 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.662 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:131d47ef:::benchmark_data_smithi179_148205_object34414:head# 2024-02-09T17:57:31.662 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.662 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:131eea16:::benchmark_data_smithi179_148205_object46652:head# 2024-02-09T17:57:31.663 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.663 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:131f1ab4:::benchmark_data_smithi179_148205_object16738:head# 2024-02-09T17:57:31.663 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.663 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:131f2005:::benchmark_data_smithi179_148205_object21691:head# 2024-02-09T17:57:31.663 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.663 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:131f394d:::benchmark_data_smithi179_148205_object108488:head# 2024-02-09T17:57:31.663 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.663 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:131f9c5e:::benchmark_data_smithi179_148205_object40892:head# 2024-02-09T17:57:31.663 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.665 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1320ab8e:::benchmark_data_smithi179_148205_object41960:head# 2024-02-09T17:57:31.665 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.665 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:132106cf:::benchmark_data_smithi179_148205_object94709:head# 2024-02-09T17:57:31.665 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.665 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:132343fa:::benchmark_data_smithi179_148205_object32630:head# 2024-02-09T17:57:31.665 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.666 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1323b77e:::benchmark_data_smithi179_148205_object35425:head# 2024-02-09T17:57:31.666 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.666 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13243f89:::benchmark_data_smithi179_148205_object107266:head# 2024-02-09T17:57:31.666 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.666 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13246f06:::benchmark_data_smithi179_148205_object44912:head# 2024-02-09T17:57:31.666 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.666 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13250171:::benchmark_data_smithi179_148205_object106840:head# 2024-02-09T17:57:31.666 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.666 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:132619f5:::benchmark_data_smithi179_148205_object53814:head# 2024-02-09T17:57:31.666 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.666 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13264035:::benchmark_data_smithi179_148205_object20782:head# 2024-02-09T17:57:31.666 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.666 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:132652ea:::benchmark_data_smithi179_148205_object90544:head# 2024-02-09T17:57:31.666 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.666 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1326b748:::benchmark_data_smithi179_148205_object11768:head# 2024-02-09T17:57:31.667 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.667 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13277a98:::benchmark_data_smithi179_148205_object49969:head# 2024-02-09T17:57:31.667 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.667 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1327885b:::benchmark_data_smithi179_148205_object20841:head# 2024-02-09T17:57:31.667 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.672 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13279337:::benchmark_data_smithi179_148205_object13145:head# 2024-02-09T17:57:31.672 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.672 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13285918:::benchmark_data_smithi179_148205_object9903:head# 2024-02-09T17:57:31.672 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.672 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:132940a0:::benchmark_data_smithi179_148205_object40321:head# 2024-02-09T17:57:31.672 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.672 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:132ae190:::benchmark_data_smithi179_148205_object64378:head# 2024-02-09T17:57:31.672 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.672 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:132ba8ea:::benchmark_data_smithi179_148205_object1433:head# 2024-02-09T17:57:31.672 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.672 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:132bee62:::benchmark_data_smithi179_148205_object75175:head# 2024-02-09T17:57:31.672 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.673 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:132bf5af:::benchmark_data_smithi179_148205_object87800:head# 2024-02-09T17:57:31.673 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.673 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:132c5ff5:::benchmark_data_smithi179_148205_object76233:head# 2024-02-09T17:57:31.673 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.673 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:132c6b35:::benchmark_data_smithi179_148205_object72559:head# 2024-02-09T17:57:31.673 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.673 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:132d5d24:::benchmark_data_smithi179_148205_object32865:head# 2024-02-09T17:57:31.673 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.673 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:132dad37:::benchmark_data_smithi179_148205_object46598:head# 2024-02-09T17:57:31.673 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.673 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:132dd5bf:::benchmark_data_smithi179_148205_object22075:head# 2024-02-09T17:57:31.673 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.673 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:132e65d1:::benchmark_data_smithi179_148205_object93343:head# 2024-02-09T17:57:31.673 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.674 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:132f1433:::benchmark_data_smithi179_148205_object24555:head# 2024-02-09T17:57:31.674 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.674 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1330213c:::benchmark_data_smithi179_148205_object47670:head# 2024-02-09T17:57:31.674 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.674 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13303a66:::benchmark_data_smithi179_148205_object37986:head# 2024-02-09T17:57:31.674 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.674 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:133162bb:::benchmark_data_smithi179_148205_object72592:head# 2024-02-09T17:57:31.674 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.674 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1331f06e:::benchmark_data_smithi179_148205_object13352:head# 2024-02-09T17:57:31.674 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.674 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1331f267:::benchmark_data_smithi179_148205_object95275:head# 2024-02-09T17:57:31.674 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.674 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13320f8d:::benchmark_data_smithi179_148205_object78949:head# 2024-02-09T17:57:31.674 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.675 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:133212e4:::benchmark_data_smithi179_148205_object107408:head# 2024-02-09T17:57:31.675 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.675 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:133279ec:::benchmark_data_smithi179_148205_object75398:head# 2024-02-09T17:57:31.675 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.675 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1332998a:::benchmark_data_smithi179_148205_object7744:head# 2024-02-09T17:57:31.675 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.675 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1332d77e:::benchmark_data_smithi179_148205_object52280:head# 2024-02-09T17:57:31.675 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.675 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13347a09:::benchmark_data_smithi179_148205_object59647:head# 2024-02-09T17:57:31.675 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.675 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:133491aa:::benchmark_data_smithi179_148205_object71299:head# 2024-02-09T17:57:31.675 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.676 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13357928:::benchmark_data_smithi179_148205_object44655:head# 2024-02-09T17:57:31.676 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.676 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1335b079:::benchmark_data_smithi179_148205_object107155:head# 2024-02-09T17:57:31.677 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.677 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1337b189:::benchmark_data_smithi179_148205_object4978:head# 2024-02-09T17:57:31.677 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.677 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13388214:::benchmark_data_smithi179_148205_object19692:head# 2024-02-09T17:57:31.677 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.678 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:133897a8:::benchmark_data_smithi179_148205_object62584:head# 2024-02-09T17:57:31.678 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.678 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13394841:::benchmark_data_smithi179_148205_object63993:head# 2024-02-09T17:57:31.678 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.679 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:133998f8:::benchmark_data_smithi179_148205_object8666:head# 2024-02-09T17:57:31.679 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.679 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1339eddb:::benchmark_data_smithi179_148205_object77036:head# 2024-02-09T17:57:31.679 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.679 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:133be0ca:::benchmark_data_smithi179_148205_object106815:head# 2024-02-09T17:57:31.679 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.679 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:133ce46d:::benchmark_data_smithi179_148205_object81819:head# 2024-02-09T17:57:31.679 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.679 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:133cea49:::benchmark_data_smithi179_148205_object8001:head# 2024-02-09T17:57:31.679 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.683 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:133e203d:::benchmark_data_smithi179_148205_object77599:head# 2024-02-09T17:57:31.683 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.683 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:133e688d:::benchmark_data_smithi179_148205_object90480:head# 2024-02-09T17:57:31.683 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.683 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:133f6d29:::benchmark_data_smithi179_148205_object13108:head# 2024-02-09T17:57:31.684 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.684 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13407e98:::benchmark_data_smithi179_148205_object12554:head# 2024-02-09T17:57:31.684 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.684 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1340aef2:::benchmark_data_smithi179_148205_object5146:head# 2024-02-09T17:57:31.684 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.684 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1341058d:::benchmark_data_smithi179_148205_object87644:head# 2024-02-09T17:57:31.684 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.684 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1341b318:::benchmark_data_smithi179_148205_object101661:head# 2024-02-09T17:57:31.684 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.684 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13421b7b:::benchmark_data_smithi179_148205_object33261:head# 2024-02-09T17:57:31.684 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.684 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1344eaa0:::benchmark_data_smithi179_148205_object81911:head# 2024-02-09T17:57:31.684 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.685 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13458684:::benchmark_data_smithi179_148205_object44811:head# 2024-02-09T17:57:31.685 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.685 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1345cead:::benchmark_data_smithi179_148205_object70900:head# 2024-02-09T17:57:31.685 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.685 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13476078:::benchmark_data_smithi179_148205_object65302:head# 2024-02-09T17:57:31.685 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.685 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1348a4c9:::benchmark_data_smithi179_148205_object70466:head# 2024-02-09T17:57:31.685 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.685 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:134a1d88:::benchmark_data_smithi179_148205_object2048:head# 2024-02-09T17:57:31.685 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.685 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:134a81b5:::benchmark_data_smithi179_148205_object91693:head# 2024-02-09T17:57:31.685 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.685 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:134a84f1:::benchmark_data_smithi179_148205_object63979:head# 2024-02-09T17:57:31.685 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.685 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:134a8696:::benchmark_data_smithi179_148205_object11271:head# 2024-02-09T17:57:31.686 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.686 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:134c80e5:::benchmark_data_smithi179_148205_object67254:head# 2024-02-09T17:57:31.686 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.689 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:134d3fd5:::benchmark_data_smithi179_148205_object21118:head# 2024-02-09T17:57:31.689 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.694 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:134db95f:::benchmark_data_smithi179_148205_object28378:head# 2024-02-09T17:57:31.694 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.694 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:134ddb6d:::benchmark_data_smithi179_148205_object75244:head# 2024-02-09T17:57:31.695 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.695 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:134e0fc9:::benchmark_data_smithi179_148205_object1227:head# 2024-02-09T17:57:31.695 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.695 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:134e8b16:::benchmark_data_smithi179_148205_object4919:head# 2024-02-09T17:57:31.695 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.695 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:134e9bb3:::benchmark_data_smithi179_148205_object71327:head# 2024-02-09T17:57:31.695 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.695 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:134f0db1:::benchmark_data_smithi179_148205_object102963:head# 2024-02-09T17:57:31.695 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.695 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:134fedd7:::benchmark_data_smithi179_148205_object76520:head# 2024-02-09T17:57:31.695 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.695 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:135128a3:::benchmark_data_smithi179_148205_object106306:head# 2024-02-09T17:57:31.695 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.695 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13527daf:::benchmark_data_smithi179_148205_object9721:head# 2024-02-09T17:57:31.695 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.695 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1352c515:::benchmark_data_smithi179_148205_object47523:head# 2024-02-09T17:57:31.696 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.696 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1353116b:::benchmark_data_smithi179_148205_object32062:head# 2024-02-09T17:57:31.696 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.696 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13531d56:::benchmark_data_smithi179_148205_object104219:head# 2024-02-09T17:57:31.696 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.696 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13534c67:::benchmark_data_smithi179_148205_object11561:head# 2024-02-09T17:57:31.696 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.696 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13546731:::benchmark_data_smithi179_148205_object52957:head# 2024-02-09T17:57:31.696 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.696 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:135499f2:::benchmark_data_smithi179_148205_object34867:head# 2024-02-09T17:57:31.696 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.696 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1354ee05:::benchmark_data_smithi179_148205_object33043:head# 2024-02-09T17:57:31.696 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.697 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1354f439:::benchmark_data_smithi179_148205_object82964:head# 2024-02-09T17:57:31.697 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.700 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13561498:::benchmark_data_smithi179_148205_object92816:head# 2024-02-09T17:57:31.700 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.700 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1357b0ab:::benchmark_data_smithi179_148205_object101325:head# 2024-02-09T17:57:31.700 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.700 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1358e196:::benchmark_data_smithi179_148205_object81360:head# 2024-02-09T17:57:31.701 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.701 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:135ab23b:::benchmark_data_smithi179_148205_object4955:head# 2024-02-09T17:57:31.701 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.701 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:135b064b:::benchmark_data_smithi179_148205_object35204:head# 2024-02-09T17:57:31.701 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.701 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:135c4a5a:::benchmark_data_smithi179_148205_object103094:head# 2024-02-09T17:57:31.701 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.701 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:135ea975:::benchmark_data_smithi179_148205_object98184:head# 2024-02-09T17:57:31.701 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.701 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:135f7e82:::benchmark_data_smithi179_148205_object34693:head# 2024-02-09T17:57:31.701 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.701 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:135fb210:::benchmark_data_smithi179_148205_object84599:head# 2024-02-09T17:57:31.702 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.702 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:136190eb:::benchmark_data_smithi179_148205_object67988:head# 2024-02-09T17:57:31.702 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.702 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13619513:::benchmark_data_smithi179_148205_object90926:head# 2024-02-09T17:57:31.702 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.702 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1362d224:::benchmark_data_smithi179_148205_object66777:head# 2024-02-09T17:57:31.702 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.702 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1362e5c3:::benchmark_data_smithi179_148205_object63512:head# 2024-02-09T17:57:31.702 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.702 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:136452ac:::benchmark_data_smithi179_148205_object3603:head# 2024-02-09T17:57:31.702 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.702 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:136490a5:::benchmark_data_smithi179_148205_object7882:head# 2024-02-09T17:57:31.702 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.702 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13654dac:::benchmark_data_smithi179_148205_object96139:head# 2024-02-09T17:57:31.702 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.702 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1366c925:::benchmark_data_smithi179_148205_object44536:head# 2024-02-09T17:57:31.703 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.703 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1366ceb9:::benchmark_data_smithi179_148205_object64262:head# 2024-02-09T17:57:31.703 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.703 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:136995bc:::benchmark_data_smithi179_148205_object36947:head# 2024-02-09T17:57:31.703 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.703 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1369dce7:::benchmark_data_smithi179_148205_object82151:head# 2024-02-09T17:57:31.703 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.703 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:136a2b3d:::benchmark_data_smithi179_148205_object62331:head# 2024-02-09T17:57:31.703 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.703 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:136b9260:::benchmark_data_smithi179_148205_object78327:head# 2024-02-09T17:57:31.703 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.703 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:136bf937:::benchmark_data_smithi179_148205_object58713:head# 2024-02-09T17:57:31.703 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.704 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:136c917e:::benchmark_data_smithi179_148205_object74135:head# 2024-02-09T17:57:31.704 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.704 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:136cc01d:::benchmark_data_smithi179_148205_object52734:head# 2024-02-09T17:57:31.704 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.704 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:136d8935:::benchmark_data_smithi179_148205_object50578:head# 2024-02-09T17:57:31.705 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.705 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:136e42fb:::benchmark_data_smithi179_148205_object38232:head# 2024-02-09T17:57:31.707 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.707 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:136ec800:::benchmark_data_smithi179_148205_object77320:head# 2024-02-09T17:57:31.707 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.708 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:136fadf5:::benchmark_data_smithi179_148205_object25520:head# 2024-02-09T17:57:31.708 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.709 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13708711:::benchmark_data_smithi179_148205_object99769:head# 2024-02-09T17:57:31.709 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.709 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1370c4c0:::benchmark_data_smithi179_148205_object106339:head# 2024-02-09T17:57:31.709 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.709 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1370cc50:::benchmark_data_smithi179_148205_object4440:head# 2024-02-09T17:57:31.709 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.709 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1370dbe5:::benchmark_data_smithi179_148205_object83402:head# 2024-02-09T17:57:31.709 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.710 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1370e1a1:::benchmark_data_smithi179_148205_object8134:head# 2024-02-09T17:57:31.710 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.710 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:137109a9:::benchmark_data_smithi179_148205_object14465:head# 2024-02-09T17:57:31.710 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.710 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13718bee:::benchmark_data_smithi179_148205_object76082:head# 2024-02-09T17:57:31.710 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.713 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1371a407:::benchmark_data_smithi179_148205_object73028:head# 2024-02-09T17:57:31.713 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.713 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1371f2e9:::benchmark_data_smithi179_148205_object11859:head# 2024-02-09T17:57:31.713 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.714 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13720c81:::benchmark_data_smithi179_148205_object50496:head# 2024-02-09T17:57:31.714 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.714 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13739380:::benchmark_data_smithi179_148205_object47561:head# 2024-02-09T17:57:31.714 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.715 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13740f8a:::benchmark_data_smithi179_148205_object50928:head# 2024-02-09T17:57:31.715 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.715 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13741543:::benchmark_data_smithi179_148205_object102926:head# 2024-02-09T17:57:31.715 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.715 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13744dfb:::benchmark_data_smithi179_148205_object49758:head# 2024-02-09T17:57:31.715 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.716 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1374eb4e:::benchmark_data_smithi179_148205_object107995:head# 2024-02-09T17:57:31.716 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.716 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13750aa4:::benchmark_data_smithi179_148205_object50939:head# 2024-02-09T17:57:31.716 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.717 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13751328:::benchmark_data_smithi179_148205_object68456:head# 2024-02-09T17:57:31.717 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.717 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13760b44:::benchmark_data_smithi179_148205_object71663:head# 2024-02-09T17:57:31.717 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.717 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13766a1c:::benchmark_data_smithi179_148205_object26038:head# 2024-02-09T17:57:31.717 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.719 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1376ebbc:::benchmark_data_smithi179_148205_object105473:head# 2024-02-09T17:57:31.719 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.719 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13771a6f:::benchmark_data_smithi179_148205_object30841:head# 2024-02-09T17:57:31.719 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.719 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1377671e:::benchmark_data_smithi179_148205_object18610:head# 2024-02-09T17:57:31.719 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.719 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13780370:::benchmark_data_smithi179_148205_object33134:head# 2024-02-09T17:57:31.719 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.719 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1378060c:::benchmark_data_smithi179_148205_object88639:head# 2024-02-09T17:57:31.719 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.720 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13788879:::benchmark_data_smithi179_148205_object54120:head# 2024-02-09T17:57:31.720 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.720 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1378c8a5:::benchmark_data_smithi179_148205_object69794:head# 2024-02-09T17:57:31.720 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.720 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13790832:::benchmark_data_smithi179_148205_object72187:head# 2024-02-09T17:57:31.720 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.720 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:137919c4:::benchmark_data_smithi179_148205_object90460:head# 2024-02-09T17:57:31.720 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.720 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13792f32:::benchmark_data_smithi179_148205_object66819:head# 2024-02-09T17:57:31.720 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.722 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13795a61:::benchmark_data_smithi179_148205_object68619:head# 2024-02-09T17:57:31.722 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.722 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1379b233:::benchmark_data_smithi179_148205_object98647:head# 2024-02-09T17:57:31.723 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.723 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1379c7a1:::benchmark_data_smithi179_148205_object55145:head# 2024-02-09T17:57:31.723 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.723 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:137ade47:::benchmark_data_smithi179_148205_object28767:head# 2024-02-09T17:57:31.723 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.723 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:137b2665:::benchmark_data_smithi179_148205_object66111:head# 2024-02-09T17:57:31.723 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.725 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:137b89a1:::benchmark_data_smithi179_148205_object94907:head# 2024-02-09T17:57:31.725 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.725 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:137bbad0:::benchmark_data_smithi179_148205_object88450:head# 2024-02-09T17:57:31.725 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.725 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:137bf544:::benchmark_data_smithi179_148205_object39574:head# 2024-02-09T17:57:31.725 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.725 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:137cac2e:::benchmark_data_smithi179_148205_object2173:head# 2024-02-09T17:57:31.725 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.725 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:137d2f95:::benchmark_data_smithi179_148205_object31961:head# 2024-02-09T17:57:31.726 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.726 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:137dc4c3:::benchmark_data_smithi179_148205_object104869:head# 2024-02-09T17:57:31.726 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.726 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:137e0818:::benchmark_data_smithi179_148205_object71867:head# 2024-02-09T17:57:31.726 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.726 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13800afb:::benchmark_data_smithi179_148205_object93827:head# 2024-02-09T17:57:31.726 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.728 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1383639d:::benchmark_data_smithi179_148205_object90310:head# 2024-02-09T17:57:31.728 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.728 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1383dbc6:::benchmark_data_smithi179_148205_object42866:head# 2024-02-09T17:57:31.728 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.728 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1383e751:::benchmark_data_smithi179_148205_object26629:head# 2024-02-09T17:57:31.728 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.728 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:138459b1:::benchmark_data_smithi179_148205_object42016:head# 2024-02-09T17:57:31.728 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.728 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1385d306:::benchmark_data_smithi179_148205_object69227:head# 2024-02-09T17:57:31.729 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.729 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13868368:::benchmark_data_smithi179_148205_object44402:head# 2024-02-09T17:57:31.729 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.729 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13868f33:::benchmark_data_smithi179_148205_object101099:head# 2024-02-09T17:57:31.729 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.729 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1386ac9b:::benchmark_data_smithi179_148205_object16900:head# 2024-02-09T17:57:31.729 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.729 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13876e14:::benchmark_data_smithi179_148205_object64029:head# 2024-02-09T17:57:31.729 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.729 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:138773c5:::benchmark_data_smithi179_148205_object79974:head# 2024-02-09T17:57:31.729 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.729 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1387b137:::benchmark_data_smithi179_148205_object37905:head# 2024-02-09T17:57:31.729 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.729 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1387ca2f:::benchmark_data_smithi179_148205_object78181:head# 2024-02-09T17:57:31.729 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.729 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1387d2ae:::benchmark_data_smithi179_148205_object84622:head# 2024-02-09T17:57:31.730 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.733 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13888d3b:::benchmark_data_smithi179_148205_object14128:head# 2024-02-09T17:57:31.733 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.733 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1388d433:::benchmark_data_smithi179_148205_object23862:head# 2024-02-09T17:57:31.733 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.733 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:138974c3:::benchmark_data_smithi179_148205_object18191:head# 2024-02-09T17:57:31.733 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.734 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13897fca:::benchmark_data_smithi179_148205_object48040:head# 2024-02-09T17:57:31.734 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.734 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1389a9e1:::benchmark_data_smithi179_148205_object1381:head# 2024-02-09T17:57:31.734 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.734 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1389bccd:::benchmark_data_smithi179_148205_object49981:head# 2024-02-09T17:57:31.734 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.734 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:138a138a:::benchmark_data_smithi179_148205_object49587:head# 2024-02-09T17:57:31.734 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.734 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:138a7904:::benchmark_data_smithi179_148205_object71715:head# 2024-02-09T17:57:31.734 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.734 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:138b04a3:::benchmark_data_smithi179_148205_object64516:head# 2024-02-09T17:57:31.734 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.734 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:138bd40b:::benchmark_data_smithi179_148205_object46818:head# 2024-02-09T17:57:31.734 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.734 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:138d1077:::benchmark_data_smithi179_148205_object29295:head# 2024-02-09T17:57:31.735 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.735 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:138d1b85:::benchmark_data_smithi179_148205_object29510:head# 2024-02-09T17:57:31.735 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.735 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:138d61c6:::benchmark_data_smithi179_148205_object62520:head# 2024-02-09T17:57:31.735 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.735 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:138d8b63:::benchmark_data_smithi179_148205_object27474:head# 2024-02-09T17:57:31.735 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.735 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:138e82e3:::benchmark_data_smithi179_148205_object24342:head# 2024-02-09T17:57:31.735 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.735 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:139044ae:::benchmark_data_smithi179_148205_object96325:head# 2024-02-09T17:57:31.735 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.735 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13907b18:::benchmark_data_smithi179_148205_object45240:head# 2024-02-09T17:57:31.735 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.735 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1390ec0a:::benchmark_data_smithi179_148205_object17585:head# 2024-02-09T17:57:31.735 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.736 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:139104e8:::benchmark_data_smithi179_148205_object62213:head# 2024-02-09T17:57:31.736 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.736 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13920b00:::benchmark_data_smithi179_148205_object11082:head# 2024-02-09T17:57:31.736 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.737 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13924913:::benchmark_data_smithi179_148205_object47182:head# 2024-02-09T17:57:31.737 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.737 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:139289bf:::benchmark_data_smithi179_148205_object58637:head# 2024-02-09T17:57:31.737 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.737 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1392ae0a:::benchmark_data_smithi179_148205_object3894:head# 2024-02-09T17:57:31.737 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.737 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13946b74:::benchmark_data_smithi179_148205_object40164:head# 2024-02-09T17:57:31.737 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.737 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13948f15:::benchmark_data_smithi179_148205_object108448:head# 2024-02-09T17:57:31.737 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.745 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13949ee6:::benchmark_data_smithi179_148205_object109282:head# 2024-02-09T17:57:31.745 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.745 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13950de3:::benchmark_data_smithi179_148205_object89051:head# 2024-02-09T17:57:31.745 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.745 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13963ecd:::benchmark_data_smithi179_148205_object59159:head# 2024-02-09T17:57:31.745 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.746 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13966917:::benchmark_data_smithi179_148205_object74914:head# 2024-02-09T17:57:31.746 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.746 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1396956b:::benchmark_data_smithi179_148205_object60823:head# 2024-02-09T17:57:31.746 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.746 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13971fb6:::benchmark_data_smithi179_148205_object7529:head# 2024-02-09T17:57:31.746 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.746 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:139802a4:::benchmark_data_smithi179_148205_object3669:head# 2024-02-09T17:57:31.746 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.746 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1398268d:::benchmark_data_smithi179_148205_object107306:head# 2024-02-09T17:57:31.746 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.746 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1398b7bc:::benchmark_data_smithi179_148205_object50995:head# 2024-02-09T17:57:31.746 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.747 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:139907ea:::benchmark_data_smithi179_148205_object79878:head# 2024-02-09T17:57:31.747 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.747 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13996863:::benchmark_data_smithi179_148205_object38599:head# 2024-02-09T17:57:31.747 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.747 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1399aa37:::benchmark_data_smithi179_148205_object77518:head# 2024-02-09T17:57:31.747 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.747 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1399cf68:::benchmark_data_smithi179_148205_object67267:head# 2024-02-09T17:57:31.747 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.747 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:139b2e82:::benchmark_data_smithi179_148205_object54237:head# 2024-02-09T17:57:31.747 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.747 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:139c6bd0:::benchmark_data_smithi179_148205_object2794:head# 2024-02-09T17:57:31.747 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.747 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:139cf945:::benchmark_data_smithi179_148205_object90166:head# 2024-02-09T17:57:31.747 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.747 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:139d9969:::benchmark_data_smithi179_148205_object57900:head# 2024-02-09T17:57:31.748 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.748 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:139fc554:::benchmark_data_smithi179_148205_object3274:head# 2024-02-09T17:57:31.748 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.748 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13a031e4:::benchmark_data_smithi179_148205_object43206:head# 2024-02-09T17:57:31.748 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.748 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13a1c956:::benchmark_data_smithi179_148205_object38950:head# 2024-02-09T17:57:31.748 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.748 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13a28ae6:::benchmark_data_smithi179_148205_object2217:head# 2024-02-09T17:57:31.748 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.748 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13a31fa1:::benchmark_data_smithi179_148205_object39790:head# 2024-02-09T17:57:31.748 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.748 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13a392fd:::benchmark_data_smithi179_148205_object30984:head# 2024-02-09T17:57:31.748 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.748 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13a3eee8:::benchmark_data_smithi179_148205_object9712:head# 2024-02-09T17:57:31.749 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.749 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13a45556:::benchmark_data_smithi179_148205_object15149:head# 2024-02-09T17:57:31.749 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.749 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13a48b4f:::benchmark_data_smithi179_148205_object65832:head# 2024-02-09T17:57:31.749 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.749 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13a519ee:::benchmark_data_smithi179_148205_object28997:head# 2024-02-09T17:57:31.749 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.749 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13a65256:::benchmark_data_smithi179_148205_object8017:head# 2024-02-09T17:57:31.749 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.749 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13a6ad6a:::benchmark_data_smithi179_148205_object48321:head# 2024-02-09T17:57:31.749 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.749 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13a714ae:::benchmark_data_smithi179_148205_object96926:head# 2024-02-09T17:57:31.749 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.753 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13a7a707:::benchmark_data_smithi179_148205_object98637:head# 2024-02-09T17:57:31.753 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.753 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13a90e24:::benchmark_data_smithi179_148205_object89906:head# 2024-02-09T17:57:31.753 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.753 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13a9122f:::benchmark_data_smithi179_148205_object75252:head# 2024-02-09T17:57:31.753 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.753 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13aa873d:::benchmark_data_smithi179_148205_object85633:head# 2024-02-09T17:57:31.753 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.753 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13aaa0e6:::benchmark_data_smithi179_148205_object42903:head# 2024-02-09T17:57:31.753 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.753 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13aac4ae:::benchmark_data_smithi179_148205_object33189:head# 2024-02-09T17:57:31.754 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.754 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13ac2dac:::benchmark_data_smithi179_148205_object37638:head# 2024-02-09T17:57:31.754 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.754 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13ac6a3d:::benchmark_data_smithi179_148205_object98011:head# 2024-02-09T17:57:31.754 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.754 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13ac9342:::benchmark_data_smithi179_148205_object65245:head# 2024-02-09T17:57:31.754 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.754 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13acf9eb:::benchmark_data_smithi179_148205_object21752:head# 2024-02-09T17:57:31.754 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.754 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13ad76b5:::benchmark_data_smithi179_148205_object74790:head# 2024-02-09T17:57:31.754 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.754 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13adc855:::benchmark_data_smithi179_148205_object107877:head# 2024-02-09T17:57:31.754 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.754 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13ae1cc3:::benchmark_data_smithi179_148205_object12494:head# 2024-02-09T17:57:31.754 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.755 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13ae6384:::benchmark_data_smithi179_148205_object86788:head# 2024-02-09T17:57:31.755 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.755 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13ae9349:::benchmark_data_smithi179_148205_object7593:head# 2024-02-09T17:57:31.755 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.755 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13af011a:::benchmark_data_smithi179_148205_object86175:head# 2024-02-09T17:57:31.755 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.755 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13af32ed:::benchmark_data_smithi179_148205_object55928:head# 2024-02-09T17:57:31.755 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.755 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13afc84a:::benchmark_data_smithi179_148205_object17845:head# 2024-02-09T17:57:31.755 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.755 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13b05664:::benchmark_data_smithi179_148205_object109085:head# 2024-02-09T17:57:31.755 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.755 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13b0de9d:::benchmark_data_smithi179_148205_object22060:head# 2024-02-09T17:57:31.755 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.755 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13b3c943:::benchmark_data_smithi179_148205_object100215:head# 2024-02-09T17:57:31.755 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.756 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13b53ec6:::benchmark_data_smithi179_148205_object3681:head# 2024-02-09T17:57:31.756 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.756 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13b567e2:::benchmark_data_smithi179_148205_object433:head# 2024-02-09T17:57:31.756 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.756 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13b5bfda:::benchmark_data_smithi179_148205_object100554:head# 2024-02-09T17:57:31.756 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.756 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13b68b23:::benchmark_data_smithi179_148205_object50850:head# 2024-02-09T17:57:31.756 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.756 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13b6ba3d:::benchmark_data_smithi179_148205_object46204:head# 2024-02-09T17:57:31.756 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.756 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13b8306f:::benchmark_data_smithi179_148205_object67873:head# 2024-02-09T17:57:31.756 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.756 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13b9935c:::benchmark_data_smithi179_148205_object85649:head# 2024-02-09T17:57:31.756 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.756 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13baa9a8:::benchmark_data_smithi179_148205_object76775:head# 2024-02-09T17:57:31.757 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.757 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13bd148a:::benchmark_data_smithi179_148205_object81638:head# 2024-02-09T17:57:31.757 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.757 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13bd6564:::benchmark_data_smithi179_148205_object3718:head# 2024-02-09T17:57:31.757 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.757 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13be5663:::benchmark_data_smithi179_148205_object94570:head# 2024-02-09T17:57:31.757 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.757 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13be6590:::benchmark_data_smithi179_148205_object38861:head# 2024-02-09T17:57:31.757 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.767 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13bed526:::benchmark_data_smithi179_148205_object85835:head# 2024-02-09T17:57:31.767 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.767 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13bef649:::benchmark_data_smithi179_148205_object46183:head# 2024-02-09T17:57:31.767 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.767 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13bfad62:::benchmark_data_smithi179_148205_object16112:head# 2024-02-09T17:57:31.767 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.767 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13c0bc44:::benchmark_data_smithi179_148205_object67235:head# 2024-02-09T17:57:31.767 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.767 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13c14b8a:::benchmark_data_smithi179_148205_object91534:head# 2024-02-09T17:57:31.767 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.767 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13c24aca:::benchmark_data_smithi179_148205_object50989:head# 2024-02-09T17:57:31.767 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.768 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13c25827:::benchmark_data_smithi179_148205_object27845:head# 2024-02-09T17:57:31.768 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.768 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13c294b4:::benchmark_data_smithi179_148205_object8336:head# 2024-02-09T17:57:31.768 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.768 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13c2c713:::benchmark_data_smithi179_148205_object57450:head# 2024-02-09T17:57:31.768 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.768 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13c2de63:::benchmark_data_smithi179_148205_object91111:head# 2024-02-09T17:57:31.768 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.768 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13c3f979:::benchmark_data_smithi179_148205_object98708:head# 2024-02-09T17:57:31.768 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.768 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13c42786:::benchmark_data_smithi179_148205_object61373:head# 2024-02-09T17:57:31.768 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.768 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13c51491:::benchmark_data_smithi179_148205_object101572:head# 2024-02-09T17:57:31.769 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.769 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13c54847:::benchmark_data_smithi179_148205_object47364:head# 2024-02-09T17:57:31.769 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.769 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13c6a3b9:::benchmark_data_smithi179_148205_object71666:head# 2024-02-09T17:57:31.769 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.769 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13c6d19a:::benchmark_data_smithi179_148205_object83884:head# 2024-02-09T17:57:31.769 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.769 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13c71fef:::benchmark_data_smithi179_148205_object62392:head# 2024-02-09T17:57:31.769 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.769 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13c792c3:::benchmark_data_smithi179_148205_object40066:head# 2024-02-09T17:57:31.769 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.769 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13c8a350:::benchmark_data_smithi179_148205_object64337:head# 2024-02-09T17:57:31.769 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.769 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13c8af12:::benchmark_data_smithi179_148205_object105148:head# 2024-02-09T17:57:31.769 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.769 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13c99005:::benchmark_data_smithi179_148205_object19432:head# 2024-02-09T17:57:31.770 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.770 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13ca4549:::benchmark_data_smithi179_148205_object8787:head# 2024-02-09T17:57:31.770 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.770 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13cb06f9:::benchmark_data_smithi179_148205_object41787:head# 2024-02-09T17:57:31.770 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.770 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13cb26a8:::benchmark_data_smithi179_148205_object80540:head# 2024-02-09T17:57:31.770 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.770 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13cf39e0:::benchmark_data_smithi179_148205_object8170:head# 2024-02-09T17:57:31.770 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.770 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13d1715d:::benchmark_data_smithi179_148205_object27154:head# 2024-02-09T17:57:31.770 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.770 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13d3099f:::benchmark_data_smithi179_148205_object51534:head# 2024-02-09T17:57:31.770 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.770 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13d39801:::benchmark_data_smithi179_148205_object78001:head# 2024-02-09T17:57:31.770 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.771 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13d7f59e:::benchmark_data_smithi179_148205_object38792:head# 2024-02-09T17:57:31.771 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.771 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13d8ac94:::benchmark_data_smithi179_148205_object61495:head# 2024-02-09T17:57:31.771 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.771 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13d92232:::benchmark_data_smithi179_148205_object48494:head# 2024-02-09T17:57:31.771 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.771 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13daee21:::benchmark_data_smithi179_148205_object106415:head# 2024-02-09T17:57:31.771 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.771 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13db1c87:::benchmark_data_smithi179_148205_object105136:head# 2024-02-09T17:57:31.771 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.771 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13dbf28d:::benchmark_data_smithi179_148205_object42258:head# 2024-02-09T17:57:31.771 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.771 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13dd87fc:::benchmark_data_smithi179_148205_object60875:head# 2024-02-09T17:57:31.771 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.771 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13ddaadf:::benchmark_data_smithi179_148205_object30521:head# 2024-02-09T17:57:31.771 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.772 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13ddb9e8:::benchmark_data_smithi179_148205_object52326:head# 2024-02-09T17:57:31.772 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.772 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13ddbd43:::benchmark_data_smithi179_148205_object9832:head# 2024-02-09T17:57:31.772 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.772 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13dee46e:::benchmark_data_smithi179_148205_object56248:head# 2024-02-09T17:57:31.772 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.772 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13dfad8a:::benchmark_data_smithi179_148205_object90417:head# 2024-02-09T17:57:31.772 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.772 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13e09713:::benchmark_data_smithi179_148205_object49914:head# 2024-02-09T17:57:31.772 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.772 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13e15381:::benchmark_data_smithi179_148205_object65181:head# 2024-02-09T17:57:31.772 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.772 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13e15aff:::benchmark_data_smithi179_148205_object65505:head# 2024-02-09T17:57:31.772 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.772 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13e1cce1:::benchmark_data_smithi179_148205_object55075:head# 2024-02-09T17:57:31.772 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.773 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13e323a7:::benchmark_data_smithi179_148205_object57482:head# 2024-02-09T17:57:31.773 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.773 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13e36bde:::benchmark_data_smithi179_148205_object61526:head# 2024-02-09T17:57:31.773 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.773 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13e38aaa:::benchmark_data_smithi179_148205_object105702:head# 2024-02-09T17:57:31.773 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.773 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13e3e7b7:::benchmark_data_smithi179_148205_object81796:head# 2024-02-09T17:57:31.773 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.773 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13e52a73:::benchmark_data_smithi179_148205_object93933:head# 2024-02-09T17:57:31.773 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.773 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13e53a6f:::benchmark_data_smithi179_148205_object21382:head# 2024-02-09T17:57:31.773 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.773 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13e53bdd:::benchmark_data_smithi179_148205_object78385:head# 2024-02-09T17:57:31.773 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.773 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13e7511f:::benchmark_data_smithi179_148205_object96357:head# 2024-02-09T17:57:31.773 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.774 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13e7e845:::benchmark_data_smithi179_148205_object59432:head# 2024-02-09T17:57:31.774 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.774 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13e8cdf4:::benchmark_data_smithi179_148205_object63899:head# 2024-02-09T17:57:31.774 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.774 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13e8d20a:::benchmark_data_smithi179_148205_object104851:head# 2024-02-09T17:57:31.774 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.774 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13ea082f:::benchmark_data_smithi179_148205_object46640:head# 2024-02-09T17:57:31.774 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.774 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13ea4fc5:::benchmark_data_smithi179_148205_object23962:head# 2024-02-09T17:57:31.774 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.777 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13eb35e6:::benchmark_data_smithi179_148205_object10828:head# 2024-02-09T17:57:31.778 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.778 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13ec1ea9:::benchmark_data_smithi179_148205_object14494:head# 2024-02-09T17:57:31.778 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.778 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13ecae3a:::benchmark_data_smithi179_148205_object51147:head# 2024-02-09T17:57:31.778 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.778 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13ecdbe2:::benchmark_data_smithi179_148205_object39165:head# 2024-02-09T17:57:31.778 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.778 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13edaaee:::benchmark_data_smithi179_148205_object2885:head# 2024-02-09T17:57:31.778 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.778 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13ee4eee:::benchmark_data_smithi179_148205_object11367:head# 2024-02-09T17:57:31.778 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.778 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13eeea75:::benchmark_data_smithi179_148205_object7990:head# 2024-02-09T17:57:31.778 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.778 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13ef2c6c:::benchmark_data_smithi179_148205_object35035:head# 2024-02-09T17:57:31.778 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.778 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13ef8cb4:::benchmark_data_smithi179_148205_object15698:head# 2024-02-09T17:57:31.779 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.779 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13ef972c:::benchmark_data_smithi179_148205_object52688:head# 2024-02-09T17:57:31.779 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.779 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13ef9c12:::benchmark_data_smithi179_148205_object55178:head# 2024-02-09T17:57:31.779 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.779 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13efb14d:::benchmark_data_smithi179_148205_object58302:head# 2024-02-09T17:57:31.779 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.779 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13efdda3:::benchmark_data_smithi179_148205_object59018:head# 2024-02-09T17:57:31.779 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.779 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13efddf2:::benchmark_data_smithi179_148205_object7525:head# 2024-02-09T17:57:31.779 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.779 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13f05648:::benchmark_data_smithi179_148205_object39244:head# 2024-02-09T17:57:31.779 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.779 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13f06a63:::benchmark_data_smithi179_148205_object55462:head# 2024-02-09T17:57:31.780 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.780 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13f1b5d7:::benchmark_data_smithi179_148205_object3968:head# 2024-02-09T17:57:31.780 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.780 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13f1d6dc:::benchmark_data_smithi179_148205_object10656:head# 2024-02-09T17:57:31.780 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.780 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13f29c1b:::benchmark_data_smithi179_148205_object23104:head# 2024-02-09T17:57:31.780 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.780 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13f2ec85:::benchmark_data_smithi179_148205_object30815:head# 2024-02-09T17:57:31.780 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.780 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13f39606:::benchmark_data_smithi179_148205_object38322:head# 2024-02-09T17:57:31.780 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.780 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13f3adde:::benchmark_data_smithi179_148205_object84871:head# 2024-02-09T17:57:31.780 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.781 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13f5ee86:::benchmark_data_smithi179_148205_object71143:head# 2024-02-09T17:57:31.781 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.781 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13f79bd4:::benchmark_data_smithi179_148205_object37153:head# 2024-02-09T17:57:31.781 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.781 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13f7abe1:::benchmark_data_smithi179_148205_object62267:head# 2024-02-09T17:57:31.781 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.785 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13f9853c:::benchmark_data_smithi179_148205_object57082:head# 2024-02-09T17:57:31.785 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.785 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13fc4dbf:::benchmark_data_smithi179_148205_object24747:head# 2024-02-09T17:57:31.785 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.785 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13fcdd53:::benchmark_data_smithi179_148205_object11641:head# 2024-02-09T17:57:31.785 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.785 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13fde6ce:::benchmark_data_smithi179_148205_object44108:head# 2024-02-09T17:57:31.785 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.785 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:13fffc62:::benchmark_data_smithi179_148205_object88623:head# 2024-02-09T17:57:31.785 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.785 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14009b5f:::benchmark_data_smithi179_148205_object100437:head# 2024-02-09T17:57:31.785 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.786 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1400c361:::benchmark_data_smithi179_148205_object47139:head# 2024-02-09T17:57:31.786 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.786 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1402bfd2:::benchmark_data_smithi179_148205_object30768:head# 2024-02-09T17:57:31.786 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.786 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1402db17:::benchmark_data_smithi179_148205_object5881:head# 2024-02-09T17:57:31.786 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.786 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1402f7b1:::benchmark_data_smithi179_148205_object75088:head# 2024-02-09T17:57:31.786 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.786 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14031f3a:::benchmark_data_smithi179_148205_object83268:head# 2024-02-09T17:57:31.786 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.786 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1403e545:::benchmark_data_smithi179_148205_object49169:head# 2024-02-09T17:57:31.786 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.786 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14075e4a:::benchmark_data_smithi179_148205_object86620:head# 2024-02-09T17:57:31.786 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.786 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:140975fb:::benchmark_data_smithi179_148205_object52827:head# 2024-02-09T17:57:31.786 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.787 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:140addbd:::benchmark_data_smithi179_148205_object77219:head# 2024-02-09T17:57:31.787 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.787 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:140bc90f:::benchmark_data_smithi179_148205_object133:head# 2024-02-09T17:57:31.787 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.787 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1410113f:::benchmark_data_smithi179_148205_object90964:head# 2024-02-09T17:57:31.787 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.787 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14102580:::benchmark_data_smithi179_148205_object13859:head# 2024-02-09T17:57:31.787 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.787 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14118702:::benchmark_data_smithi179_148205_object101787:head# 2024-02-09T17:57:31.787 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.787 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1411adda:::benchmark_data_smithi179_148205_object59485:head# 2024-02-09T17:57:31.787 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.787 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1411e9a6:::benchmark_data_smithi179_148205_object92121:head# 2024-02-09T17:57:31.787 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.787 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1412c53c:::benchmark_data_smithi179_148205_object57512:head# 2024-02-09T17:57:31.788 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.788 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1413dec0:::benchmark_data_smithi179_148205_object3821:head# 2024-02-09T17:57:31.788 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.788 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14149f4d:::benchmark_data_smithi179_148205_object102871:head# 2024-02-09T17:57:31.788 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.788 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14156eeb:::benchmark_data_smithi179_148205_object81840:head# 2024-02-09T17:57:31.788 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.788 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14158e94:::benchmark_data_smithi179_148205_object37142:head# 2024-02-09T17:57:31.788 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.788 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1415e7f2:::benchmark_data_smithi179_148205_object75701:head# 2024-02-09T17:57:31.788 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.788 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:141661dc:::benchmark_data_smithi179_148205_object47175:head# 2024-02-09T17:57:31.788 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.788 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14167abc:::benchmark_data_smithi179_148205_object32002:head# 2024-02-09T17:57:31.788 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.789 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14171fe4:::benchmark_data_smithi179_148205_object108992:head# 2024-02-09T17:57:31.789 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.789 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1418368e:::benchmark_data_smithi179_148205_object3849:head# 2024-02-09T17:57:31.789 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.789 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1419281d:::benchmark_data_smithi179_148205_object74985:head# 2024-02-09T17:57:31.789 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.789 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:141a0d0d:::benchmark_data_smithi179_148205_object89094:head# 2024-02-09T17:57:31.789 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.790 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:141a48c8:::benchmark_data_smithi179_148205_object30723:head# 2024-02-09T17:57:31.790 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.790 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:141a863c:::benchmark_data_smithi179_148205_object36957:head# 2024-02-09T17:57:31.791 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.791 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:141ac5cb:::benchmark_data_smithi179_148205_object98371:head# 2024-02-09T17:57:31.791 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.791 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:141ae3a2:::benchmark_data_smithi179_148205_object73088:head# 2024-02-09T17:57:31.791 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.791 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:141b4ff4:::benchmark_data_smithi179_148205_object99566:head# 2024-02-09T17:57:31.792 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.793 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:141c5c68:::benchmark_data_smithi179_148205_object39115:head# 2024-02-09T17:57:31.793 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.793 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:141ceaa5:::benchmark_data_smithi179_148205_object101180:head# 2024-02-09T17:57:31.793 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.793 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:141d1a27:::benchmark_data_smithi179_148205_object52505:head# 2024-02-09T17:57:31.793 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.793 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:141dc613:::benchmark_data_smithi179_148205_object104566:head# 2024-02-09T17:57:31.793 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.793 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:141e63d1:::benchmark_data_smithi179_148205_object57716:head# 2024-02-09T17:57:31.793 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.793 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:141e9703:::benchmark_data_smithi179_148205_object55913:head# 2024-02-09T17:57:31.793 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.794 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:141efbd6:::benchmark_data_smithi179_148205_object108052:head# 2024-02-09T17:57:31.794 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.794 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:142020e9:::benchmark_data_smithi179_148205_object25708:head# 2024-02-09T17:57:31.794 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.795 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:142244ae:::benchmark_data_smithi179_148205_object1135:head# 2024-02-09T17:57:31.795 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.795 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:142245fc:::benchmark_data_smithi179_148205_object105019:head# 2024-02-09T17:57:31.795 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.795 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14224e32:::benchmark_data_smithi179_148205_object36521:head# 2024-02-09T17:57:31.795 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.795 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14227893:::benchmark_data_smithi179_148205_object36989:head# 2024-02-09T17:57:31.795 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.796 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1422cdf0:::benchmark_data_smithi179_148205_object35303:head# 2024-02-09T17:57:31.796 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.796 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14244684:::benchmark_data_smithi179_148205_object49284:head# 2024-02-09T17:57:31.796 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.796 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14247d51:::benchmark_data_smithi179_148205_object87720:head# 2024-02-09T17:57:31.796 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.796 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1424df7d:::benchmark_data_smithi179_148205_object54943:head# 2024-02-09T17:57:31.796 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.797 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:142510c1:::benchmark_data_smithi179_148205_object32723:head# 2024-02-09T17:57:31.797 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.797 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14251f60:::benchmark_data_smithi179_148205_object99025:head# 2024-02-09T17:57:31.797 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.798 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1425919a:::benchmark_data_smithi179_148205_object62474:head# 2024-02-09T17:57:31.798 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.798 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1425a130:::benchmark_data_smithi179_148205_object61823:head# 2024-02-09T17:57:31.798 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.798 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1425f912:::benchmark_data_smithi179_148205_object27947:head# 2024-02-09T17:57:31.798 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.798 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1427c3ab:::benchmark_data_smithi179_148205_object95986:head# 2024-02-09T17:57:31.798 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.799 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:142831c5:::benchmark_data_smithi179_148205_object31604:head# 2024-02-09T17:57:31.799 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.799 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:142a335c:::benchmark_data_smithi179_148205_object83954:head# 2024-02-09T17:57:31.799 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.799 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:142a6614:::benchmark_data_smithi179_148205_object24875:head# 2024-02-09T17:57:31.799 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.801 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:142a72bf:::benchmark_data_smithi179_148205_object45154:head# 2024-02-09T17:57:31.801 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.801 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:142ae9c5:::benchmark_data_smithi179_148205_object40888:head# 2024-02-09T17:57:31.802 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.802 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:142bb66c:::benchmark_data_smithi179_148205_object42430:head# 2024-02-09T17:57:31.802 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.802 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:142bd7de:::benchmark_data_smithi179_148205_object29887:head# 2024-02-09T17:57:31.802 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.802 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:142c1b66:::benchmark_data_smithi179_148205_object97024:head# 2024-02-09T17:57:31.802 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.802 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:142c832f:::benchmark_data_smithi179_148205_object450:head# 2024-02-09T17:57:31.802 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.803 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:142cc259:::benchmark_data_smithi179_148205_object55573:head# 2024-02-09T17:57:31.803 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.803 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:142d1388:::benchmark_data_smithi179_148205_object101593:head# 2024-02-09T17:57:31.803 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.803 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:142d2081:::benchmark_data_smithi179_148205_object69247:head# 2024-02-09T17:57:31.803 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.803 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:142d40f7:::benchmark_data_smithi179_148205_object24188:head# 2024-02-09T17:57:31.803 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.803 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:142d4f70:::benchmark_data_smithi179_148205_object42466:head# 2024-02-09T17:57:31.803 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.805 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:142dbd23:::benchmark_data_smithi179_148205_object87530:head# 2024-02-09T17:57:31.805 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.805 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:142e2383:::benchmark_data_smithi179_148205_object94937:head# 2024-02-09T17:57:31.805 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.806 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:142eaef6:::benchmark_data_smithi179_148205_object16521:head# 2024-02-09T17:57:31.806 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.806 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:142fb037:::benchmark_data_smithi179_148205_object85994:head# 2024-02-09T17:57:31.806 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.806 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14304b05:::benchmark_data_smithi179_148205_object16277:head# 2024-02-09T17:57:31.806 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.806 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14310fb9:::benchmark_data_smithi179_148205_object37632:head# 2024-02-09T17:57:31.806 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.806 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14316484:::benchmark_data_smithi179_148205_object36873:head# 2024-02-09T17:57:31.806 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.806 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14317ca6:::benchmark_data_smithi179_148205_object17837:head# 2024-02-09T17:57:31.806 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.806 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1431804d:::benchmark_data_smithi179_148205_object41793:head# 2024-02-09T17:57:31.806 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.806 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:143253f3:::benchmark_data_smithi179_148205_object44490:head# 2024-02-09T17:57:31.807 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.807 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:143267dd:::benchmark_data_smithi179_148205_object69590:head# 2024-02-09T17:57:31.807 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.807 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1432e33d:::benchmark_data_smithi179_148205_object29337:head# 2024-02-09T17:57:31.807 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.807 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1432f504:::benchmark_data_smithi179_148205_object14392:head# 2024-02-09T17:57:31.807 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.807 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14347362:::benchmark_data_smithi179_148205_object49492:head# 2024-02-09T17:57:31.807 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.807 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1434e2b9:::benchmark_data_smithi179_148205_object65199:head# 2024-02-09T17:57:31.807 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.807 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14373820:::benchmark_data_smithi179_148205_object74549:head# 2024-02-09T17:57:31.807 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.809 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1437d2c9:::benchmark_data_smithi179_148205_object95968:head# 2024-02-09T17:57:31.809 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.809 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1439a840:::benchmark_data_smithi179_148205_object33516:head# 2024-02-09T17:57:31.810 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.810 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:143a031c:::benchmark_data_smithi179_148205_object101750:head# 2024-02-09T17:57:31.810 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.810 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:143a591f:::benchmark_data_smithi179_148205_object692:head# 2024-02-09T17:57:31.810 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.810 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:143a7d17:::benchmark_data_smithi179_148205_object37019:head# 2024-02-09T17:57:31.810 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.810 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:143aed33:::benchmark_data_smithi179_148205_object43915:head# 2024-02-09T17:57:31.810 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.810 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:143afba0:::benchmark_data_smithi179_148205_object84238:head# 2024-02-09T17:57:31.810 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.810 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:143b237c:::benchmark_data_smithi179_148205_object11042:head# 2024-02-09T17:57:31.810 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.810 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:143b317f:::benchmark_data_smithi179_148205_object99343:head# 2024-02-09T17:57:31.810 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.811 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:143c5c11:::benchmark_data_smithi179_148205_object43387:head# 2024-02-09T17:57:31.811 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.811 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:143caa15:::benchmark_data_smithi179_148205_object8833:head# 2024-02-09T17:57:31.811 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.811 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:143ccedf:::benchmark_data_smithi179_148205_object31554:head# 2024-02-09T17:57:31.811 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.811 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:143d69d5:::benchmark_data_smithi179_148205_object45428:head# 2024-02-09T17:57:31.811 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.812 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:143e145a:::benchmark_data_smithi179_148205_object7428:head# 2024-02-09T17:57:31.812 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.812 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:143f5ab7:::benchmark_data_smithi179_148205_object46374:head# 2024-02-09T17:57:31.812 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.812 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14409ef6:::benchmark_data_smithi179_148205_object48959:head# 2024-02-09T17:57:31.812 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.813 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14417cd1:::benchmark_data_smithi179_148205_object88719:head# 2024-02-09T17:57:31.813 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.813 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1441d701:::benchmark_data_smithi179_148205_object6943:head# 2024-02-09T17:57:31.813 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.813 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14424d4e:::benchmark_data_smithi179_148205_object79447:head# 2024-02-09T17:57:31.814 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.816 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14426d9d:::benchmark_data_smithi179_148205_object101254:head# 2024-02-09T17:57:31.816 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.816 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14432f1d:::benchmark_data_smithi179_148205_object53221:head# 2024-02-09T17:57:31.816 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.817 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14436268:::benchmark_data_smithi179_148205_object86278:head# 2024-02-09T17:57:31.817 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.817 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:144370c7:::benchmark_data_smithi179_148205_object31619:head# 2024-02-09T17:57:31.817 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.818 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:144377e9:::benchmark_data_smithi179_148205_object89542:head# 2024-02-09T17:57:31.818 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.818 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:144517c7:::benchmark_data_smithi179_148205_object72695:head# 2024-02-09T17:57:31.818 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.818 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1445384f:::benchmark_data_smithi179_148205_object103418:head# 2024-02-09T17:57:31.818 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.818 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1445ca1a:::benchmark_data_smithi179_148205_object25869:head# 2024-02-09T17:57:31.818 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.819 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1445ea37:::benchmark_data_smithi179_148205_object63318:head# 2024-02-09T17:57:31.819 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.819 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:144667f8:::benchmark_data_smithi179_148205_object82792:head# 2024-02-09T17:57:31.820 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.820 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:144687d3:::benchmark_data_smithi179_148205_object107685:head# 2024-02-09T17:57:31.820 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.820 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1446a322:::benchmark_data_smithi179_148205_object101953:head# 2024-02-09T17:57:31.820 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.820 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1448a2f5:::benchmark_data_smithi179_148205_object43608:head# 2024-02-09T17:57:31.821 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.821 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14498f1e:::benchmark_data_smithi179_148205_object81999:head# 2024-02-09T17:57:31.821 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.821 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:144a3d71:::benchmark_data_smithi179_148205_object96865:head# 2024-02-09T17:57:31.821 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.821 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:144b1371:::benchmark_data_smithi179_148205_object18667:head# 2024-02-09T17:57:31.821 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.821 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:144b9cda:::benchmark_data_smithi179_148205_object109528:head# 2024-02-09T17:57:31.821 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.822 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:144bc959:::benchmark_data_smithi179_148205_object46470:head# 2024-02-09T17:57:31.823 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.823 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:144c6376:::benchmark_data_smithi179_148205_object13791:head# 2024-02-09T17:57:31.823 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.823 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:144cad77:::benchmark_data_smithi179_148205_object72910:head# 2024-02-09T17:57:31.823 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.824 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:144cc4e7:::benchmark_data_smithi179_148205_object107672:head# 2024-02-09T17:57:31.824 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.824 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:144d12b2:::benchmark_data_smithi179_148205_object23134:head# 2024-02-09T17:57:31.824 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.825 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:144d66b2:::benchmark_data_smithi179_148205_object77449:head# 2024-02-09T17:57:31.825 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.825 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:144e2139:::benchmark_data_smithi179_148205_object43862:head# 2024-02-09T17:57:31.825 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.825 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:144f28ed:::benchmark_data_smithi179_148205_object74201:head# 2024-02-09T17:57:31.825 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.825 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:144f7801:::benchmark_data_smithi179_148205_object2862:head# 2024-02-09T17:57:31.825 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.826 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1450bfc2:::benchmark_data_smithi179_148205_object8052:head# 2024-02-09T17:57:31.826 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.826 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14514b0c:::benchmark_data_smithi179_148205_object39131:head# 2024-02-09T17:57:31.826 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.826 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:145180e6:::benchmark_data_smithi179_148205_object19266:head# 2024-02-09T17:57:31.826 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.826 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14518f43:::benchmark_data_smithi179_148205_object62852:head# 2024-02-09T17:57:31.826 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.826 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1452307d:::benchmark_data_smithi179_148205_object92311:head# 2024-02-09T17:57:31.826 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.826 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1452371d:::benchmark_data_smithi179_148205_object64370:head# 2024-02-09T17:57:31.826 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.826 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:145278aa:::benchmark_data_smithi179_148205_object30166:head# 2024-02-09T17:57:31.826 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.826 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:145361b1:::benchmark_data_smithi179_148205_object75519:head# 2024-02-09T17:57:31.827 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.827 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14543d90:::benchmark_data_smithi179_148205_object60791:head# 2024-02-09T17:57:31.827 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.832 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:145441d0:::benchmark_data_smithi179_148205_object64287:head# 2024-02-09T17:57:31.832 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.832 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14548175:::benchmark_data_smithi179_148205_object9421:head# 2024-02-09T17:57:31.832 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.832 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14554ab0:::benchmark_data_smithi179_148205_object96524:head# 2024-02-09T17:57:31.832 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.832 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14558cc8:::benchmark_data_smithi179_148205_object3271:head# 2024-02-09T17:57:31.832 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.840 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1456ff9e:::benchmark_data_smithi179_148205_object51067:head# 2024-02-09T17:57:31.840 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.840 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1457535a:::benchmark_data_smithi179_148205_object81668:head# 2024-02-09T17:57:31.840 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.841 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1457cde3:::benchmark_data_smithi179_148205_object56166:head# 2024-02-09T17:57:31.841 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.841 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14587a4b:::benchmark_data_smithi179_148205_object76438:head# 2024-02-09T17:57:31.841 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.841 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1458d603:::benchmark_data_smithi179_148205_object57239:head# 2024-02-09T17:57:31.841 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.841 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1459071a:::benchmark_data_smithi179_148205_object7445:head# 2024-02-09T17:57:31.841 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.841 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14591aab:::benchmark_data_smithi179_148205_object23762:head# 2024-02-09T17:57:31.841 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.841 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1459a76d:::benchmark_data_smithi179_148205_object21896:head# 2024-02-09T17:57:31.841 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.841 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:145a6e63:::benchmark_data_smithi179_148205_object32543:head# 2024-02-09T17:57:31.841 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.841 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:145aa347:::benchmark_data_smithi179_148205_object83981:head# 2024-02-09T17:57:31.841 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.842 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:145b1cfb:::benchmark_data_smithi179_148205_object38272:head# 2024-02-09T17:57:31.842 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.843 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:145d6cd2:::benchmark_data_smithi179_148205_object23838:head# 2024-02-09T17:57:31.843 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.843 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:145d799d:::benchmark_data_smithi179_148205_object63752:head# 2024-02-09T17:57:31.843 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.843 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:145da65d:::benchmark_data_smithi179_148205_object41326:head# 2024-02-09T17:57:31.843 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.843 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:145de6d0:::benchmark_data_smithi179_148205_object19991:head# 2024-02-09T17:57:31.843 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.843 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:145e628e:::benchmark_data_smithi179_148205_object23995:head# 2024-02-09T17:57:31.843 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.843 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:145ef8aa:::benchmark_data_smithi179_148205_object77262:head# 2024-02-09T17:57:31.843 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.844 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:145f8455:::benchmark_data_smithi179_148205_object45680:head# 2024-02-09T17:57:31.844 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.844 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:146065f9:::benchmark_data_smithi179_148205_object46406:head# 2024-02-09T17:57:31.844 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.845 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1460a90e:::benchmark_data_smithi179_148205_object23869:head# 2024-02-09T17:57:31.845 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.849 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1460bac6:::benchmark_data_smithi179_148205_object71586:head# 2024-02-09T17:57:31.849 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.850 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:146200bb:::benchmark_data_smithi179_148205_object10888:head# 2024-02-09T17:57:31.850 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.850 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14626056:::benchmark_data_smithi179_148205_object51248:head# 2024-02-09T17:57:31.850 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.850 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14629673:::benchmark_data_smithi179_148205_object80073:head# 2024-02-09T17:57:31.850 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.851 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1462d40a:::benchmark_data_smithi179_148205_object57145:head# 2024-02-09T17:57:31.851 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.851 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1462fbc4:::benchmark_data_smithi179_148205_object101967:head# 2024-02-09T17:57:31.851 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.851 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1463ef96:::benchmark_data_smithi179_148205_object31583:head# 2024-02-09T17:57:31.851 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.851 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14642362:::benchmark_data_smithi179_148205_object14739:head# 2024-02-09T17:57:31.851 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.854 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1464cd10:::benchmark_data_smithi179_148205_object11674:head# 2024-02-09T17:57:31.854 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.854 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1464de42:::benchmark_data_smithi179_148205_object12371:head# 2024-02-09T17:57:31.854 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.854 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14651c4c:::benchmark_data_smithi179_148205_object36319:head# 2024-02-09T17:57:31.855 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.855 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14654b70:::benchmark_data_smithi179_148205_object99416:head# 2024-02-09T17:57:31.855 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.855 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14656255:::benchmark_data_smithi179_148205_object43044:head# 2024-02-09T17:57:31.855 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.855 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14658239:::benchmark_data_smithi179_148205_object90434:head# 2024-02-09T17:57:31.855 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.855 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14683d9d:::benchmark_data_smithi179_148205_object68727:head# 2024-02-09T17:57:31.855 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.855 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1468912c:::benchmark_data_smithi179_148205_object41705:head# 2024-02-09T17:57:31.855 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.855 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1468b24a:::benchmark_data_smithi179_148205_object51213:head# 2024-02-09T17:57:31.855 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.856 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1468c1c0:::benchmark_data_smithi179_148205_object2633:head# 2024-02-09T17:57:31.856 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.856 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1469fefd:::benchmark_data_smithi179_148205_object76065:head# 2024-02-09T17:57:31.856 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.856 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:146b16c2:::benchmark_data_smithi179_148205_object67599:head# 2024-02-09T17:57:31.856 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.856 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:146c35f5:::benchmark_data_smithi179_148205_object14994:head# 2024-02-09T17:57:31.856 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.857 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:146d95b8:::benchmark_data_smithi179_148205_object20040:head# 2024-02-09T17:57:31.857 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.858 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:146df7bd:::benchmark_data_smithi179_148205_object6071:head# 2024-02-09T17:57:31.858 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.858 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:146ec2ba:::benchmark_data_smithi179_148205_object49966:head# 2024-02-09T17:57:31.859 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.859 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:146ee4bd:::benchmark_data_smithi179_148205_object41258:head# 2024-02-09T17:57:31.859 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.859 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:146f5460:::benchmark_data_smithi179_148205_object36155:head# 2024-02-09T17:57:31.859 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.859 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:146f5b55:::benchmark_data_smithi179_148205_object1177:head# 2024-02-09T17:57:31.859 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.859 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:146f63c2:::benchmark_data_smithi179_148205_object36354:head# 2024-02-09T17:57:31.859 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.859 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:146f7a93:::benchmark_data_smithi179_148205_object96414:head# 2024-02-09T17:57:31.859 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.859 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14718e9c:::benchmark_data_smithi179_148205_object51492:head# 2024-02-09T17:57:31.859 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.859 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1472d7dc:::benchmark_data_smithi179_148205_object90457:head# 2024-02-09T17:57:31.859 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.859 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1472da04:::benchmark_data_smithi179_148205_object75328:head# 2024-02-09T17:57:31.860 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.860 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1472f477:::benchmark_data_smithi179_148205_object77714:head# 2024-02-09T17:57:31.860 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.860 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1472f4c9:::benchmark_data_smithi179_148205_object12267:head# 2024-02-09T17:57:31.860 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.860 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14737aa5:::benchmark_data_smithi179_148205_object21784:head# 2024-02-09T17:57:31.860 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.860 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14746b49:::benchmark_data_smithi179_148205_object93717:head# 2024-02-09T17:57:31.860 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.860 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1474cf8f:::benchmark_data_smithi179_148205_object75541:head# 2024-02-09T17:57:31.860 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.860 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1474ee2a:::benchmark_data_smithi179_148205_object6654:head# 2024-02-09T17:57:31.860 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.860 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14772b4b:::benchmark_data_smithi179_148205_object86176:head# 2024-02-09T17:57:31.861 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.868 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14772bfa:::benchmark_data_smithi179_148205_object89956:head# 2024-02-09T17:57:31.868 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.868 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1477520c:::benchmark_data_smithi179_148205_object7926:head# 2024-02-09T17:57:31.868 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.868 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1477648d:::benchmark_data_smithi179_148205_object25263:head# 2024-02-09T17:57:31.868 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.868 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14778662:::benchmark_data_smithi179_148205_object69983:head# 2024-02-09T17:57:31.868 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.868 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1478a9d2:::benchmark_data_smithi179_148205_object24353:head# 2024-02-09T17:57:31.868 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.868 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1478eebf:::benchmark_data_smithi179_148205_object47880:head# 2024-02-09T17:57:31.868 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.868 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14790a22:::benchmark_data_smithi179_148205_object84903:head# 2024-02-09T17:57:31.869 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.869 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:147a67f8:::benchmark_data_smithi179_148205_object26:head# 2024-02-09T17:57:31.869 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.869 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:147b0c0c:::benchmark_data_smithi179_148205_object21936:head# 2024-02-09T17:57:31.869 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.869 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:147cfecd:::benchmark_data_smithi179_148205_object69197:head# 2024-02-09T17:57:31.869 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.869 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:147e90f9:::benchmark_data_smithi179_148205_object16156:head# 2024-02-09T17:57:31.869 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.869 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:147e99e9:::benchmark_data_smithi179_148205_object80067:head# 2024-02-09T17:57:31.869 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.869 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:147efd61:::benchmark_data_smithi179_148205_object58756:head# 2024-02-09T17:57:31.869 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.869 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1480470d:::benchmark_data_smithi179_148205_object9244:head# 2024-02-09T17:57:31.869 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.870 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14809c9a:::benchmark_data_smithi179_148205_object107805:head# 2024-02-09T17:57:31.870 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.870 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14816689:::benchmark_data_smithi179_148205_object7999:head# 2024-02-09T17:57:31.870 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.870 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1482a1df:::benchmark_data_smithi179_148205_object91019:head# 2024-02-09T17:57:31.870 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.870 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14839646:::benchmark_data_smithi179_148205_object12447:head# 2024-02-09T17:57:31.870 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.870 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1483b2ae:::benchmark_data_smithi179_148205_object83705:head# 2024-02-09T17:57:31.870 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.870 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1483b7b8:::benchmark_data_smithi179_148205_object8965:head# 2024-02-09T17:57:31.870 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.870 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:148415f6:::benchmark_data_smithi179_148205_object7149:head# 2024-02-09T17:57:31.870 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.870 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1484236b:::benchmark_data_smithi179_148205_object63895:head# 2024-02-09T17:57:31.871 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.871 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14860e10:::benchmark_data_smithi179_148205_object83120:head# 2024-02-09T17:57:31.871 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.871 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14864a74:::benchmark_data_smithi179_148205_object88366:head# 2024-02-09T17:57:31.871 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.871 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1487411b:::benchmark_data_smithi179_148205_object106556:head# 2024-02-09T17:57:31.871 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.871 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14896241:::benchmark_data_smithi179_148205_object33747:head# 2024-02-09T17:57:31.871 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.871 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1489c072:::benchmark_data_smithi179_148205_object13869:head# 2024-02-09T17:57:31.871 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.871 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:148a1ff8:::benchmark_data_smithi179_148205_object90467:head# 2024-02-09T17:57:31.871 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.871 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:148a400f:::benchmark_data_smithi179_148205_object88621:head# 2024-02-09T17:57:31.871 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.872 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:148a6c77:::benchmark_data_smithi179_148205_object102344:head# 2024-02-09T17:57:31.872 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.872 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:148ada81:::benchmark_data_smithi179_148205_object16499:head# 2024-02-09T17:57:31.872 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.872 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:148b4e5e:::benchmark_data_smithi179_148205_object95232:head# 2024-02-09T17:57:31.872 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.872 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:148c02c3:::benchmark_data_smithi179_148205_object29270:head# 2024-02-09T17:57:31.872 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.872 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:148cc532:::benchmark_data_smithi179_148205_object23755:head# 2024-02-09T17:57:31.872 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.872 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:148d488a:::benchmark_data_smithi179_148205_object33580:head# 2024-02-09T17:57:31.872 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.872 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:148e5136:::benchmark_data_smithi179_148205_object123:head# 2024-02-09T17:57:31.872 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.873 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:148ee7b1:::benchmark_data_smithi179_148205_object42069:head# 2024-02-09T17:57:31.873 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.873 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:148f75d8:::benchmark_data_smithi179_148205_object44551:head# 2024-02-09T17:57:31.873 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.873 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:148f9aed:::benchmark_data_smithi179_148205_object25125:head# 2024-02-09T17:57:31.873 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.873 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:149055e7:::benchmark_data_smithi179_148205_object19267:head# 2024-02-09T17:57:31.873 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.873 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1490ad5a:::benchmark_data_smithi179_148205_object9181:head# 2024-02-09T17:57:31.873 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.873 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1492ff83:::benchmark_data_smithi179_148205_object11167:head# 2024-02-09T17:57:31.873 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.876 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1493f970:::benchmark_data_smithi179_148205_object91374:head# 2024-02-09T17:57:31.876 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.876 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14941dd4:::benchmark_data_smithi179_148205_object1847:head# 2024-02-09T17:57:31.876 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.876 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14944091:::benchmark_data_smithi179_148205_object21812:head# 2024-02-09T17:57:31.876 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.876 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1494b1d6:::benchmark_data_smithi179_148205_object97690:head# 2024-02-09T17:57:31.877 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.877 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1494b4b2:::benchmark_data_smithi179_148205_object8186:head# 2024-02-09T17:57:31.877 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.877 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1494e550:::benchmark_data_smithi179_148205_object45892:head# 2024-02-09T17:57:31.877 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.877 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1494e90d:::benchmark_data_smithi179_148205_object100229:head# 2024-02-09T17:57:31.877 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.877 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:149550c1:::benchmark_data_smithi179_148205_object471:head# 2024-02-09T17:57:31.877 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.877 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1495ffc1:::benchmark_data_smithi179_148205_object9464:head# 2024-02-09T17:57:31.877 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.877 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:149838f4:::benchmark_data_smithi179_148205_object106312:head# 2024-02-09T17:57:31.877 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.877 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1498ba0b:::benchmark_data_smithi179_148205_object53842:head# 2024-02-09T17:57:31.878 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.878 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:149961a0:::benchmark_data_smithi179_148205_object23423:head# 2024-02-09T17:57:31.878 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.878 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1499add9:::benchmark_data_smithi179_148205_object78759:head# 2024-02-09T17:57:31.878 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.878 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:149bea83:::benchmark_data_smithi179_148205_object49635:head# 2024-02-09T17:57:31.878 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.878 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:149c3eb9:::benchmark_data_smithi179_148205_object61946:head# 2024-02-09T17:57:31.878 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.879 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:149c4cbb:::benchmark_data_smithi179_148205_object54398:head# 2024-02-09T17:57:31.879 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.879 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:149c5bb0:::benchmark_data_smithi179_148205_object41227:head# 2024-02-09T17:57:31.879 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.879 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:149de81c:::benchmark_data_smithi179_148205_object41857:head# 2024-02-09T17:57:31.879 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.879 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:149dfe47:::benchmark_data_smithi179_148205_object41195:head# 2024-02-09T17:57:31.879 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.879 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:149e0fac:::benchmark_data_smithi179_148205_object32086:head# 2024-02-09T17:57:31.879 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.880 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:149f0a0a:::benchmark_data_smithi179_148205_object15498:head# 2024-02-09T17:57:31.880 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.880 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:149f9ab2:::benchmark_data_smithi179_148205_object67550:head# 2024-02-09T17:57:31.880 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.880 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:149fab94:::benchmark_data_smithi179_148205_object5828:head# 2024-02-09T17:57:31.880 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.885 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:149fd124:::benchmark_data_smithi179_148205_object57733:head# 2024-02-09T17:57:31.885 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.885 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14a0ecb8:::benchmark_data_smithi179_148205_object92458:head# 2024-02-09T17:57:31.885 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.885 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14a19489:::benchmark_data_smithi179_148205_object9014:head# 2024-02-09T17:57:31.885 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.885 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14a2d03e:::benchmark_data_smithi179_148205_object31955:head# 2024-02-09T17:57:31.885 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.885 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14a2ee3a:::benchmark_data_smithi179_148205_object25484:head# 2024-02-09T17:57:31.885 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.885 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14a48174:::benchmark_data_smithi179_148205_object37579:head# 2024-02-09T17:57:31.886 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.886 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14a4ec1b:::benchmark_data_smithi179_148205_object18200:head# 2024-02-09T17:57:31.886 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.886 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14a50fe3:::benchmark_data_smithi179_148205_object104167:head# 2024-02-09T17:57:31.886 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.886 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14a54a53:::benchmark_data_smithi179_148205_object12615:head# 2024-02-09T17:57:31.886 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.886 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14a7083c:::benchmark_data_smithi179_148205_object34435:head# 2024-02-09T17:57:31.886 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.886 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14a82f5e:::benchmark_data_smithi179_148205_object21126:head# 2024-02-09T17:57:31.886 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.886 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14a8771c:::benchmark_data_smithi179_148205_object100491:head# 2024-02-09T17:57:31.886 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.887 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14a9453e:::benchmark_data_smithi179_148205_object20779:head# 2024-02-09T17:57:31.887 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.887 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14a945c5:::benchmark_data_smithi179_148205_object27127:head# 2024-02-09T17:57:31.887 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.887 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14ab2440:::benchmark_data_smithi179_148205_object15985:head# 2024-02-09T17:57:31.887 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.887 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14ac3b1e:::benchmark_data_smithi179_148205_object75973:head# 2024-02-09T17:57:31.887 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.887 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14ad4661:::benchmark_data_smithi179_148205_object17094:head# 2024-02-09T17:57:31.887 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.887 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14ad5985:::benchmark_data_smithi179_148205_object37685:head# 2024-02-09T17:57:31.887 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.887 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14ad8586:::benchmark_data_smithi179_148205_object7584:head# 2024-02-09T17:57:31.887 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.888 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14af0d52:::benchmark_data_smithi179_148205_object2215:head# 2024-02-09T17:57:31.888 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.888 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14afc3a7:::benchmark_data_smithi179_148205_object77485:head# 2024-02-09T17:57:31.888 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.890 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14b06048:::benchmark_data_smithi179_148205_object25858:head# 2024-02-09T17:57:31.890 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.890 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14b11fac:::benchmark_data_smithi179_148205_object61816:head# 2024-02-09T17:57:31.890 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.890 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14b150e2:::benchmark_data_smithi179_148205_object31632:head# 2024-02-09T17:57:31.890 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.891 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14b278d9:::benchmark_data_smithi179_148205_object77479:head# 2024-02-09T17:57:31.891 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.891 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14b53dd6:::benchmark_data_smithi179_148205_object105528:head# 2024-02-09T17:57:31.891 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.891 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14b5adf3:::benchmark_data_smithi179_148205_object102916:head# 2024-02-09T17:57:31.891 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.891 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14b5c049:::benchmark_data_smithi179_148205_object14799:head# 2024-02-09T17:57:31.892 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.894 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14b616b0:::benchmark_data_smithi179_148205_object99607:head# 2024-02-09T17:57:31.894 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.894 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14b6e1cc:::benchmark_data_smithi179_148205_object3812:head# 2024-02-09T17:57:31.894 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.894 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14b9cab3:::benchmark_data_smithi179_148205_object3505:head# 2024-02-09T17:57:31.894 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.894 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14ba0713:::benchmark_data_smithi179_148205_object90108:head# 2024-02-09T17:57:31.894 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.894 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14bb6c1a:::benchmark_data_smithi179_148205_object16298:head# 2024-02-09T17:57:31.894 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.895 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14bc5ee5:::benchmark_data_smithi179_148205_object107490:head# 2024-02-09T17:57:31.895 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.895 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14bc6481:::benchmark_data_smithi179_148205_object67617:head# 2024-02-09T17:57:31.895 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.895 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14bcb625:::benchmark_data_smithi179_148205_object94175:head# 2024-02-09T17:57:31.895 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.895 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14bd8206:::benchmark_data_smithi179_148205_object34566:head# 2024-02-09T17:57:31.895 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.895 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14be2f15:::benchmark_data_smithi179_148205_object31071:head# 2024-02-09T17:57:31.895 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.895 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14be71be:::benchmark_data_smithi179_148205_object2401:head# 2024-02-09T17:57:31.895 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.898 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14bed2d2:::benchmark_data_smithi179_148205_object84302:head# 2024-02-09T17:57:31.898 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.898 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14bf50d6:::benchmark_data_smithi179_148205_object95858:head# 2024-02-09T17:57:31.898 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.898 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14c29f37:::benchmark_data_smithi179_148205_object72151:head# 2024-02-09T17:57:31.898 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.898 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14c3f241:::benchmark_data_smithi179_148205_object96033:head# 2024-02-09T17:57:31.898 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.898 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14c42988:::benchmark_data_smithi179_148205_object84650:head# 2024-02-09T17:57:31.898 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.898 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14c42eaf:::benchmark_data_smithi179_148205_object107591:head# 2024-02-09T17:57:31.898 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.898 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14c4613f:::benchmark_data_smithi179_148205_object28978:head# 2024-02-09T17:57:31.899 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.899 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14c47e14:::benchmark_data_smithi179_148205_object78267:head# 2024-02-09T17:57:31.899 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.899 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14c55efd:::benchmark_data_smithi179_148205_object16839:head# 2024-02-09T17:57:31.899 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.899 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14c578fc:::benchmark_data_smithi179_148205_object54955:head# 2024-02-09T17:57:31.899 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.899 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14c7abd4:::benchmark_data_smithi179_148205_object26772:head# 2024-02-09T17:57:31.899 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.899 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14c88795:::benchmark_data_smithi179_148205_object80255:head# 2024-02-09T17:57:31.899 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.899 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14ca0ccc:::benchmark_data_smithi179_148205_object21815:head# 2024-02-09T17:57:31.900 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.900 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14ca94f6:::benchmark_data_smithi179_148205_object9058:head# 2024-02-09T17:57:31.900 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.900 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14cb3a06:::benchmark_data_smithi179_148205_object13159:head# 2024-02-09T17:57:31.900 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.900 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14cbcec8:::benchmark_data_smithi179_148205_object14039:head# 2024-02-09T17:57:31.900 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.901 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14cd05c7:::benchmark_data_smithi179_148205_object64075:head# 2024-02-09T17:57:31.902 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.902 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14cf703a:::benchmark_data_smithi179_148205_object89017:head# 2024-02-09T17:57:31.902 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.902 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14cf974d:::benchmark_data_smithi179_148205_object71734:head# 2024-02-09T17:57:31.902 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.902 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14cfd623:::benchmark_data_smithi179_148205_object19329:head# 2024-02-09T17:57:31.902 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.902 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14d02540:::benchmark_data_smithi179_148205_object15063:head# 2024-02-09T17:57:31.902 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.902 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14d0bba0:::benchmark_data_smithi179_148205_object80316:head# 2024-02-09T17:57:31.902 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.902 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14d0cd99:::benchmark_data_smithi179_148205_object92663:head# 2024-02-09T17:57:31.903 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.903 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14d22427:::benchmark_data_smithi179_148205_object55766:head# 2024-02-09T17:57:31.903 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.905 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14d2d3d9:::benchmark_data_smithi179_148205_object97650:head# 2024-02-09T17:57:31.905 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.905 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14d2f0a2:::benchmark_data_smithi179_148205_object88866:head# 2024-02-09T17:57:31.905 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.907 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14d3ca3e:::benchmark_data_smithi179_148205_object68550:head# 2024-02-09T17:57:31.907 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.907 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14d440a3:::benchmark_data_smithi179_148205_object60696:head# 2024-02-09T17:57:31.907 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.907 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14d4aae3:::benchmark_data_smithi179_148205_object3831:head# 2024-02-09T17:57:31.907 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.907 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14d5caa8:::benchmark_data_smithi179_148205_object1729:head# 2024-02-09T17:57:31.907 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.907 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14d5efb8:::benchmark_data_smithi179_148205_object63942:head# 2024-02-09T17:57:31.907 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.908 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14d663ad:::benchmark_data_smithi179_148205_object56072:head# 2024-02-09T17:57:31.908 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.908 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14d86ab8:::benchmark_data_smithi179_148205_object3621:head# 2024-02-09T17:57:31.908 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.912 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14d8735b:::benchmark_data_smithi179_148205_object77080:head# 2024-02-09T17:57:31.912 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.912 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14d96e1a:::benchmark_data_smithi179_148205_object57460:head# 2024-02-09T17:57:31.912 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.913 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14d9b0aa:::benchmark_data_smithi179_148205_object35807:head# 2024-02-09T17:57:31.913 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.913 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14d9e79a:::benchmark_data_smithi179_148205_object76741:head# 2024-02-09T17:57:31.913 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.913 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14dab963:::benchmark_data_smithi179_148205_object7845:head# 2024-02-09T17:57:31.913 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.913 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14db1d62:::benchmark_data_smithi179_148205_object43745:head# 2024-02-09T17:57:31.913 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.913 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14db22c2:::benchmark_data_smithi179_148205_object79352:head# 2024-02-09T17:57:31.913 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.913 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14dbf25f:::benchmark_data_smithi179_148205_object77717:head# 2024-02-09T17:57:31.913 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.913 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14dd206f:::benchmark_data_smithi179_148205_object20087:head# 2024-02-09T17:57:31.913 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.913 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14dd4c2f:::benchmark_data_smithi179_148205_object78442:head# 2024-02-09T17:57:31.914 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.915 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14dd9b80:::benchmark_data_smithi179_148205_object76332:head# 2024-02-09T17:57:31.915 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.915 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14ddff01:::benchmark_data_smithi179_148205_object56893:head# 2024-02-09T17:57:31.915 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.915 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14de3708:::benchmark_data_smithi179_148205_object101233:head# 2024-02-09T17:57:31.915 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.915 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14de68f6:::benchmark_data_smithi179_148205_object37855:head# 2024-02-09T17:57:31.915 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.915 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14deb3e0:::benchmark_data_smithi179_148205_object62568:head# 2024-02-09T17:57:31.916 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.916 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14df1cac:::benchmark_data_smithi179_148205_object101036:head# 2024-02-09T17:57:31.916 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.916 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14dfa42d:::benchmark_data_smithi179_148205_object49457:head# 2024-02-09T17:57:31.916 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.916 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14dfad46:::benchmark_data_smithi179_148205_object8899:head# 2024-02-09T17:57:31.916 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.916 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14e0b7b8:::benchmark_data_smithi179_148205_object70015:head# 2024-02-09T17:57:31.916 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.916 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14e103ef:::benchmark_data_smithi179_148205_object39330:head# 2024-02-09T17:57:31.916 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.916 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14e2422d:::benchmark_data_smithi179_148205_object71278:head# 2024-02-09T17:57:31.916 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.917 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14e2e759:::benchmark_data_smithi179_148205_object2361:head# 2024-02-09T17:57:31.917 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.917 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14e32673:::benchmark_data_smithi179_148205_object66924:head# 2024-02-09T17:57:31.917 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.917 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14e3d3c2:::benchmark_data_smithi179_148205_object2740:head# 2024-02-09T17:57:31.917 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.917 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14e4396c:::benchmark_data_smithi179_148205_object20589:head# 2024-02-09T17:57:31.917 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.917 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14e4e44e:::benchmark_data_smithi179_148205_object73964:head# 2024-02-09T17:57:31.917 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.920 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14e51f46:::benchmark_data_smithi179_148205_object71576:head# 2024-02-09T17:57:31.920 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.920 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14e5417a:::benchmark_data_smithi179_148205_object101653:head# 2024-02-09T17:57:31.920 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.920 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14e697ed:::benchmark_data_smithi179_148205_object7821:head# 2024-02-09T17:57:31.920 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.920 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14e802f0:::benchmark_data_smithi179_148205_object105399:head# 2024-02-09T17:57:31.920 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.921 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14e832fb:::benchmark_data_smithi179_148205_object80836:head# 2024-02-09T17:57:31.921 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.921 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14e8b44a:::benchmark_data_smithi179_148205_object20748:head# 2024-02-09T17:57:31.922 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.922 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14ea86a7:::benchmark_data_smithi179_148205_object9961:head# 2024-02-09T17:57:31.922 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.922 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14eb87ce:::benchmark_data_smithi179_148205_object9042:head# 2024-02-09T17:57:31.922 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.922 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14ebb42d:::benchmark_data_smithi179_148205_object49898:head# 2024-02-09T17:57:31.922 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.923 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14ebbd97:::benchmark_data_smithi179_148205_object66714:head# 2024-02-09T17:57:31.923 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.923 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14ec0c9b:::benchmark_data_smithi179_148205_object85042:head# 2024-02-09T17:57:31.923 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.923 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14ed1a82:::benchmark_data_smithi179_148205_object92908:head# 2024-02-09T17:57:31.924 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.924 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14ed2ceb:::benchmark_data_smithi179_148205_object78798:head# 2024-02-09T17:57:31.924 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.924 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14ed9e5f:::benchmark_data_smithi179_148205_object68119:head# 2024-02-09T17:57:31.924 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.924 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14edfc57:::benchmark_data_smithi179_148205_object19260:head# 2024-02-09T17:57:31.924 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.924 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14ee8166:::benchmark_data_smithi179_148205_object103922:head# 2024-02-09T17:57:31.924 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.925 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14eea409:::benchmark_data_smithi179_148205_object1727:head# 2024-02-09T17:57:31.925 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.930 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14ef5f3a:::benchmark_data_smithi179_148205_object90071:head# 2024-02-09T17:57:31.930 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.930 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14efc56b:::benchmark_data_smithi179_148205_object84563:head# 2024-02-09T17:57:31.930 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.930 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14efe785:::benchmark_data_smithi179_148205_object1489:head# 2024-02-09T17:57:31.930 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.930 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14f05fc9:::benchmark_data_smithi179_148205_object31548:head# 2024-02-09T17:57:31.930 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.930 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14f09b4d:::benchmark_data_smithi179_148205_object43486:head# 2024-02-09T17:57:31.930 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.930 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14f0a182:::benchmark_data_smithi179_148205_object38436:head# 2024-02-09T17:57:31.931 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.931 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14f13c4d:::benchmark_data_smithi179_148205_object73623:head# 2024-02-09T17:57:31.931 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.931 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14f2eaed:::benchmark_data_smithi179_148205_object97563:head# 2024-02-09T17:57:31.931 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.931 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14f38fe1:::benchmark_data_smithi179_148205_object63043:head# 2024-02-09T17:57:31.931 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.931 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14f427bc:::benchmark_data_smithi179_148205_object90686:head# 2024-02-09T17:57:31.931 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.931 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14f50686:::benchmark_data_smithi179_148205_object75218:head# 2024-02-09T17:57:31.931 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.932 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14f62c99:::benchmark_data_smithi179_148205_object43468:head# 2024-02-09T17:57:31.932 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.932 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14f6a739:::benchmark_data_smithi179_148205_object105246:head# 2024-02-09T17:57:31.932 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.932 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14f6e167:::benchmark_data_smithi179_148205_object41269:head# 2024-02-09T17:57:31.932 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.932 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14f7788e:::benchmark_data_smithi179_148205_object91089:head# 2024-02-09T17:57:31.932 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.932 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14f783d7:::benchmark_data_smithi179_148205_object80061:head# 2024-02-09T17:57:31.932 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.932 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14fc4569:::benchmark_data_smithi179_148205_object74268:head# 2024-02-09T17:57:31.932 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.935 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14fd2828:::benchmark_data_smithi179_148205_object18775:head# 2024-02-09T17:57:31.935 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.935 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14fe0421:::benchmark_data_smithi179_148205_object94697:head# 2024-02-09T17:57:31.935 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.936 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14fe66b3:::benchmark_data_smithi179_148205_object29652:head# 2024-02-09T17:57:31.936 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.936 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14ff6cc5:::benchmark_data_smithi179_148205_object3151:head# 2024-02-09T17:57:31.936 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.936 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:14ffa3b6:::benchmark_data_smithi179_148205_object1327:head# 2024-02-09T17:57:31.936 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.936 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:150049e1:::benchmark_data_smithi179_148205_object81902:head# 2024-02-09T17:57:31.936 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.936 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1500708e:::benchmark_data_smithi179_148205_object45082:head# 2024-02-09T17:57:31.936 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.936 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:150196f1:::benchmark_data_smithi179_148205_object89705:head# 2024-02-09T17:57:31.936 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.936 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15046b1f:::benchmark_data_smithi179_148205_object11064:head# 2024-02-09T17:57:31.936 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.937 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1504cc65:::benchmark_data_smithi179_148205_object86735:head# 2024-02-09T17:57:31.937 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.937 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1504d977:::benchmark_data_smithi179_148205_object72513:head# 2024-02-09T17:57:31.937 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.937 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1505080b:::benchmark_data_smithi179_148205_object91535:head# 2024-02-09T17:57:31.937 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.937 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15066671:::benchmark_data_smithi179_148205_object91979:head# 2024-02-09T17:57:31.937 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.937 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:150741df:::benchmark_data_smithi179_148205_object51250:head# 2024-02-09T17:57:31.937 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.937 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1507c434:::benchmark_data_smithi179_148205_object101981:head# 2024-02-09T17:57:31.937 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.937 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:150840ab:::benchmark_data_smithi179_148205_object57215:head# 2024-02-09T17:57:31.937 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.937 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15087652:::benchmark_data_smithi179_148205_object49682:head# 2024-02-09T17:57:31.938 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.938 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15089a9c:::benchmark_data_smithi179_148205_object6409:head# 2024-02-09T17:57:31.938 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.938 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15089bb3:::benchmark_data_smithi179_148205_object103140:head# 2024-02-09T17:57:31.938 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.938 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1509d003:::benchmark_data_smithi179_148205_object62717:head# 2024-02-09T17:57:31.938 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.938 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:150a5076:::benchmark_data_smithi179_148205_object1056:head# 2024-02-09T17:57:31.938 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.938 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:150ada69:::benchmark_data_smithi179_148205_object6163:head# 2024-02-09T17:57:31.938 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.941 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:150b5640:::benchmark_data_smithi179_148205_object6121:head# 2024-02-09T17:57:31.941 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.942 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:150d5109:::benchmark_data_smithi179_148205_object85308:head# 2024-02-09T17:57:31.942 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.942 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:150d5ad8:::benchmark_data_smithi179_148205_object7033:head# 2024-02-09T17:57:31.942 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.942 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1510f759:::benchmark_data_smithi179_148205_object52145:head# 2024-02-09T17:57:31.942 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.942 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1511a10d:::benchmark_data_smithi179_148205_object90474:head# 2024-02-09T17:57:31.942 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.942 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1511b065:::benchmark_data_smithi179_148205_object93502:head# 2024-02-09T17:57:31.942 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.942 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1511d17c:::benchmark_data_smithi179_148205_object68863:head# 2024-02-09T17:57:31.942 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.942 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15126d08:::benchmark_data_smithi179_148205_object24184:head# 2024-02-09T17:57:31.942 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.942 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15137c31:::benchmark_data_smithi179_148205_object96769:head# 2024-02-09T17:57:31.943 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.943 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15139796:::benchmark_data_smithi179_148205_object106701:head# 2024-02-09T17:57:31.943 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.943 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1513c395:::benchmark_data_smithi179_148205_object102990:head# 2024-02-09T17:57:31.943 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.943 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:151460f6:::benchmark_data_smithi179_148205_object55180:head# 2024-02-09T17:57:31.943 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.943 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1514e73e:::benchmark_data_smithi179_148205_object67165:head# 2024-02-09T17:57:31.943 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.943 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15150fef:::benchmark_data_smithi179_148205_object40671:head# 2024-02-09T17:57:31.943 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.943 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1515b92d:::benchmark_data_smithi179_148205_object45607:head# 2024-02-09T17:57:31.944 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.944 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1515d3aa:::benchmark_data_smithi179_148205_object108809:head# 2024-02-09T17:57:31.944 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.944 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1516cd85:::benchmark_data_smithi179_148205_object68664:head# 2024-02-09T17:57:31.944 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.944 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1518a74e:::benchmark_data_smithi179_148205_object81648:head# 2024-02-09T17:57:31.944 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.944 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1518ae37:::benchmark_data_smithi179_148205_object39829:head# 2024-02-09T17:57:31.944 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.944 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15196b18:::benchmark_data_smithi179_148205_object85118:head# 2024-02-09T17:57:31.944 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.945 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1519bfc4:::benchmark_data_smithi179_148205_object109321:head# 2024-02-09T17:57:31.945 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.945 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:151b2a6c:::benchmark_data_smithi179_148205_object70911:head# 2024-02-09T17:57:31.945 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.954 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:151b6117:::benchmark_data_smithi179_148205_object37725:head# 2024-02-09T17:57:31.955 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.961 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:151c1851:::benchmark_data_smithi179_148205_object59592:head# 2024-02-09T17:57:31.961 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.965 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:151c8ce1:::benchmark_data_smithi179_148205_object36729:head# 2024-02-09T17:57:31.966 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.966 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:151cfb45:::benchmark_data_smithi179_148205_object2663:head# 2024-02-09T17:57:31.966 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.966 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:151da07e:::benchmark_data_smithi179_148205_object102885:head# 2024-02-09T17:57:31.966 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.974 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:151dab6c:::benchmark_data_smithi179_148205_object79957:head# 2024-02-09T17:57:31.974 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.979 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:151ef5bc:::benchmark_data_smithi179_148205_object62157:head# 2024-02-09T17:57:31.979 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.980 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-09T17:57:31.981 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-09T17:57:31.984 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:151f9292:::benchmark_data_smithi179_148205_object12119:head# 2024-02-09T17:57:31.984 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.986 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:151ff42f:::benchmark_data_smithi179_148205_object65575:head# 2024-02-09T17:57:31.986 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.986 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1520a6fd:::benchmark_data_smithi179_148205_object100623:head# 2024-02-09T17:57:31.986 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.993 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1521f8a0:::benchmark_data_smithi179_148205_object48235:head# 2024-02-09T17:57:31.993 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.993 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1521f94c:::benchmark_data_smithi179_148205_object49936:head# 2024-02-09T17:57:31.994 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.994 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1522497b:::benchmark_data_smithi179_148205_object89669:head# 2024-02-09T17:57:31.994 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.994 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1522e634:::benchmark_data_smithi179_148205_object23836:head# 2024-02-09T17:57:31.994 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.994 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1522f8ef:::benchmark_data_smithi179_148205_object6106:head# 2024-02-09T17:57:31.994 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.994 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1524bd56:::benchmark_data_smithi179_148205_object67587:head# 2024-02-09T17:57:31.994 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.994 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1524c310:::benchmark_data_smithi179_148205_object3310:head# 2024-02-09T17:57:31.994 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.994 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:152553e0:::benchmark_data_smithi179_148205_object12087:head# 2024-02-09T17:57:31.994 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.994 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15256be7:::benchmark_data_smithi179_148205_object53594:head# 2024-02-09T17:57:31.994 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.995 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15268919:::benchmark_data_smithi179_148205_object5381:head# 2024-02-09T17:57:31.995 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.995 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1526e82f:::benchmark_data_smithi179_148205_object9797:head# 2024-02-09T17:57:31.995 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.998 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1527116b:::benchmark_data_smithi179_148205_object65261:head# 2024-02-09T17:57:31.998 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.998 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15275a62:::benchmark_data_smithi179_148205_object22017:head# 2024-02-09T17:57:31.998 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.998 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15279c90:::benchmark_data_smithi179_148205_object73577:head# 2024-02-09T17:57:31.998 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.998 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1528ae67:::benchmark_data_smithi179_148205_object67530:head# 2024-02-09T17:57:31.998 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.998 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:152996e8:::benchmark_data_smithi179_148205_object68560:head# 2024-02-09T17:57:31.998 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.998 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:152a0aa1:::benchmark_data_smithi179_148205_object65542:head# 2024-02-09T17:57:31.999 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.999 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:152a3761:::benchmark_data_smithi179_148205_object25489:head# 2024-02-09T17:57:31.999 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.999 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:152a892e:::benchmark_data_smithi179_148205_object34099:head# 2024-02-09T17:57:31.999 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.999 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:152b2e96:::benchmark_data_smithi179_148205_object53512:head# 2024-02-09T17:57:31.999 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.999 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:152b6a86:::benchmark_data_smithi179_148205_object10573:head# 2024-02-09T17:57:31.999 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.999 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:152c307f:::benchmark_data_smithi179_148205_object94590:head# 2024-02-09T17:57:31.999 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.999 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:152d9ca4:::benchmark_data_smithi179_148205_object33316:head# 2024-02-09T17:57:31.999 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:31.999 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:152da700:::benchmark_data_smithi179_148205_object36227:head# 2024-02-09T17:57:31.999 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.000 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:152e15fc:::benchmark_data_smithi179_148205_object48226:head# 2024-02-09T17:57:32.000 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.000 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:152e506b:::benchmark_data_smithi179_148205_object80556:head# 2024-02-09T17:57:32.000 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.000 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:152fc879:::benchmark_data_smithi179_148205_object18989:head# 2024-02-09T17:57:32.000 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.000 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1530096f:::benchmark_data_smithi179_148205_object65433:head# 2024-02-09T17:57:32.000 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.000 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1530efdc:::benchmark_data_smithi179_148205_object64234:head# 2024-02-09T17:57:32.000 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.003 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1531a7fa:::benchmark_data_smithi179_148205_object27090:head# 2024-02-09T17:57:32.003 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.003 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1531cdec:::benchmark_data_smithi179_148205_object46622:head# 2024-02-09T17:57:32.003 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.003 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:153201f0:::benchmark_data_smithi179_148205_object24626:head# 2024-02-09T17:57:32.003 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.003 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15343421:::benchmark_data_smithi179_148205_object36248:head# 2024-02-09T17:57:32.003 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.003 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15359eb9:::benchmark_data_smithi179_148205_object70092:head# 2024-02-09T17:57:32.003 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.003 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15362cd9:::benchmark_data_smithi179_148205_object99261:head# 2024-02-09T17:57:32.003 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.003 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15365528:::benchmark_data_smithi179_148205_object91523:head# 2024-02-09T17:57:32.003 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.004 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1536d6c2:::benchmark_data_smithi179_148205_object25587:head# 2024-02-09T17:57:32.004 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.004 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1536d7fe:::benchmark_data_smithi179_148205_object107319:head# 2024-02-09T17:57:32.004 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.004 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15372430:::benchmark_data_smithi179_148205_object1605:head# 2024-02-09T17:57:32.004 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.004 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15379203:::benchmark_data_smithi179_148205_object22016:head# 2024-02-09T17:57:32.004 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.004 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1538614e:::benchmark_data_smithi179_148205_object47242:head# 2024-02-09T17:57:32.004 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.004 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1538b687:::benchmark_data_smithi179_148205_object28190:head# 2024-02-09T17:57:32.004 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.004 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15392d9f:::benchmark_data_smithi179_148205_object116:head# 2024-02-09T17:57:32.004 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.005 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15397864:::benchmark_data_smithi179_148205_object96466:head# 2024-02-09T17:57:32.005 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.005 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1539802d:::benchmark_data_smithi179_148205_object74885:head# 2024-02-09T17:57:32.005 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.005 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:153a7c0e:::benchmark_data_smithi179_148205_object26376:head# 2024-02-09T17:57:32.005 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.006 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:153d6ef8:::benchmark_data_smithi179_148205_object29035:head# 2024-02-09T17:57:32.006 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.006 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:153e75ee:::benchmark_data_smithi179_148205_object13844:head# 2024-02-09T17:57:32.006 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.006 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:153eb687:::benchmark_data_smithi179_148205_object60266:head# 2024-02-09T17:57:32.006 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.006 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:153f7418:::benchmark_data_smithi179_148205_object602:head# 2024-02-09T17:57:32.006 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.007 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:153fd3f3:::benchmark_data_smithi179_148205_object3052:head# 2024-02-09T17:57:32.007 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.008 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15404b62:::benchmark_data_smithi179_148205_object73182:head# 2024-02-09T17:57:32.008 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.008 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1540df35:::benchmark_data_smithi179_148205_object5384:head# 2024-02-09T17:57:32.008 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.008 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1541c112:::benchmark_data_smithi179_148205_object29976:head# 2024-02-09T17:57:32.008 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.009 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1541cd73:::benchmark_data_smithi179_148205_object77981:head# 2024-02-09T17:57:32.009 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.009 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1543823e:::benchmark_data_smithi179_148205_object22133:head# 2024-02-09T17:57:32.009 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.009 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1543bcf6:::benchmark_data_smithi179_148205_object29645:head# 2024-02-09T17:57:32.009 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.009 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1543c8fe:::benchmark_data_smithi179_148205_object56438:head# 2024-02-09T17:57:32.009 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.010 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15441313:::benchmark_data_smithi179_148205_object6407:head# 2024-02-09T17:57:32.010 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.010 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15453b65:::benchmark_data_smithi179_148205_object73134:head# 2024-02-09T17:57:32.010 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.011 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15458216:::benchmark_data_smithi179_148205_object200:head# 2024-02-09T17:57:32.011 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.011 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1545b92a:::benchmark_data_smithi179_148205_object105693:head# 2024-02-09T17:57:32.011 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.011 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1545edf3:::benchmark_data_smithi179_148205_object104562:head# 2024-02-09T17:57:32.011 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.011 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15467156:::benchmark_data_smithi179_148205_object16237:head# 2024-02-09T17:57:32.011 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.012 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15470de7:::benchmark_data_smithi179_148205_object9723:head# 2024-02-09T17:57:32.012 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.012 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1547f23d:::benchmark_data_smithi179_148205_object72477:head# 2024-02-09T17:57:32.012 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.012 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1548d47d:::benchmark_data_smithi179_148205_object68765:head# 2024-02-09T17:57:32.013 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.013 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1549866e:::benchmark_data_smithi179_148205_object62677:head# 2024-02-09T17:57:32.013 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.016 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1549efe2:::benchmark_data_smithi179_148205_object58861:head# 2024-02-09T17:57:32.016 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.017 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:154a2096:::benchmark_data_smithi179_148205_object858:head# 2024-02-09T17:57:32.017 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.017 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:154a9777:::benchmark_data_smithi179_148205_object75454:head# 2024-02-09T17:57:32.018 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.018 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:154af210:::benchmark_data_smithi179_148205_object20542:head# 2024-02-09T17:57:32.018 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.018 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:154b3c98:::benchmark_data_smithi179_148205_object106191:head# 2024-02-09T17:57:32.018 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.019 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:154b63f1:::benchmark_data_smithi179_148205_object25044:head# 2024-02-09T17:57:32.019 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.019 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:154b74d0:::benchmark_data_smithi179_148205_object5276:head# 2024-02-09T17:57:32.020 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.020 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:154c16f1:::benchmark_data_smithi179_148205_object63428:head# 2024-02-09T17:57:32.020 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.020 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:154c5b3c:::benchmark_data_smithi179_148205_object64763:head# 2024-02-09T17:57:32.020 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.020 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:154d910e:::benchmark_data_smithi179_148205_object16137:head# 2024-02-09T17:57:32.020 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.020 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:154e80fb:::benchmark_data_smithi179_148205_object68086:head# 2024-02-09T17:57:32.020 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.020 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:154f6315:::benchmark_data_smithi179_148205_object45668:head# 2024-02-09T17:57:32.020 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.022 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:154f691a:::benchmark_data_smithi179_148205_object44206:head# 2024-02-09T17:57:32.022 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.022 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:154f8bc5:::benchmark_data_smithi179_148205_object82366:head# 2024-02-09T17:57:32.022 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.022 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:155001c8:::benchmark_data_smithi179_148205_object23499:head# 2024-02-09T17:57:32.022 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.022 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:155139f6:::benchmark_data_smithi179_148205_object43274:head# 2024-02-09T17:57:32.022 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.023 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15515363:::benchmark_data_smithi179_148205_object9392:head# 2024-02-09T17:57:32.023 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.023 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15544aab:::benchmark_data_smithi179_148205_object74575:head# 2024-02-09T17:57:32.023 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.023 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:155462e8:::benchmark_data_smithi179_148205_object58700:head# 2024-02-09T17:57:32.023 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.023 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15546a12:::benchmark_data_smithi179_148205_object108341:head# 2024-02-09T17:57:32.023 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.023 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15566084:::benchmark_data_smithi179_148205_object62588:head# 2024-02-09T17:57:32.023 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.024 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:155770f0:::benchmark_data_smithi179_148205_object21425:head# 2024-02-09T17:57:32.024 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.024 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is unset 2024-02-09T17:57:32.027 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15592c34:::benchmark_data_smithi179_148205_object41641:head# 2024-02-09T17:57:32.027 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.027 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:155a2d1a:::benchmark_data_smithi179_148205_object29267:head# 2024-02-09T17:57:32.027 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.028 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:155b0ece:::benchmark_data_smithi179_148205_object20505:head# 2024-02-09T17:57:32.028 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.028 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:155b5eb2:::benchmark_data_smithi179_148205_object3531:head# 2024-02-09T17:57:32.028 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.028 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:155bbc9b:::benchmark_data_smithi179_148205_object90622:head# 2024-02-09T17:57:32.028 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.028 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:155ce3b3:::benchmark_data_smithi179_148205_object60076:head# 2024-02-09T17:57:32.028 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.028 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:155d4844:::benchmark_data_smithi179_148205_object85805:head# 2024-02-09T17:57:32.028 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.028 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:155dd46f:::benchmark_data_smithi179_148205_object92278:head# 2024-02-09T17:57:32.028 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.028 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:155ddd31:::benchmark_data_smithi179_148205_object70295:head# 2024-02-09T17:57:32.029 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.029 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:155f4127:::benchmark_data_smithi179_148205_object28846:head# 2024-02-09T17:57:32.029 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.029 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:156004a4:::benchmark_data_smithi179_148205_object76058:head# 2024-02-09T17:57:32.029 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.029 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1560dfd5:::benchmark_data_smithi179_148205_object76708:head# 2024-02-09T17:57:32.029 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.029 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1561e968:::benchmark_data_smithi179_148205_object38816:head# 2024-02-09T17:57:32.029 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.029 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1562c4c0:::benchmark_data_smithi179_148205_object100182:head# 2024-02-09T17:57:32.029 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.029 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15633cc4:::benchmark_data_smithi179_148205_object44886:head# 2024-02-09T17:57:32.029 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.029 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1563cc85:::benchmark_data_smithi179_148205_object46340:head# 2024-02-09T17:57:32.029 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.030 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1564cd77:::benchmark_data_smithi179_148205_object17654:head# 2024-02-09T17:57:32.030 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.030 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1564d7bf:::benchmark_data_smithi179_148205_object58417:head# 2024-02-09T17:57:32.030 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.030 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:156526b6:::benchmark_data_smithi179_148205_object9108:head# 2024-02-09T17:57:32.030 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.030 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15660e69:::benchmark_data_smithi179_148205_object27742:head# 2024-02-09T17:57:32.030 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.030 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15662894:::benchmark_data_smithi179_148205_object84732:head# 2024-02-09T17:57:32.030 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.032 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15663bbf:::benchmark_data_smithi179_148205_object39802:head# 2024-02-09T17:57:32.032 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.032 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:156774e7:::benchmark_data_smithi179_148205_object64538:head# 2024-02-09T17:57:32.033 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.033 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1567d340:::benchmark_data_smithi179_148205_object78663:head# 2024-02-09T17:57:32.033 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.033 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1567e16e:::benchmark_data_smithi179_148205_object81684:head# 2024-02-09T17:57:32.033 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.033 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1567f37f:::benchmark_data_smithi179_148205_object87389:head# 2024-02-09T17:57:32.033 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.033 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:156840db:::benchmark_data_smithi179_148205_object97331:head# 2024-02-09T17:57:32.033 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.033 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1569fe18:::benchmark_data_smithi179_148205_object22008:head# 2024-02-09T17:57:32.033 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.033 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:156af22c:::benchmark_data_smithi179_148205_object99917:head# 2024-02-09T17:57:32.033 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.033 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:156b7734:::benchmark_data_smithi179_148205_object26400:head# 2024-02-09T17:57:32.033 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.034 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:156c19a8:::benchmark_data_smithi179_148205_object14166:head# 2024-02-09T17:57:32.034 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.034 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:156c2247:::benchmark_data_smithi179_148205_object25862:head# 2024-02-09T17:57:32.034 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.034 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:156c8b4c:::benchmark_data_smithi179_148205_object1285:head# 2024-02-09T17:57:32.034 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.034 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15709512:::benchmark_data_smithi179_148205_object106708:head# 2024-02-09T17:57:32.034 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.034 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15725320:::benchmark_data_smithi179_148205_object69731:head# 2024-02-09T17:57:32.034 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.034 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1572cfae:::benchmark_data_smithi179_148205_object80004:head# 2024-02-09T17:57:32.034 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.037 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1573cdb5:::benchmark_data_smithi179_148205_object1488:head# 2024-02-09T17:57:32.037 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.037 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15740d34:::benchmark_data_smithi179_148205_object78848:head# 2024-02-09T17:57:32.038 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.038 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15753091:::benchmark_data_smithi179_148205_object57081:head# 2024-02-09T17:57:32.038 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.038 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15762c98:::benchmark_data_smithi179_148205_object89060:head# 2024-02-09T17:57:32.038 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.038 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15770a59:::benchmark_data_smithi179_148205_object76673:head# 2024-02-09T17:57:32.038 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.038 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1577fa11:::benchmark_data_smithi179_148205_object20200:head# 2024-02-09T17:57:32.038 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.038 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:157800dc:::benchmark_data_smithi179_148205_object62636:head# 2024-02-09T17:57:32.038 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.038 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15787aa9:::benchmark_data_smithi179_148205_object14662:head# 2024-02-09T17:57:32.038 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.038 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1579a9da:::benchmark_data_smithi179_148205_object103817:head# 2024-02-09T17:57:32.038 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.039 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:157acaa0:::benchmark_data_smithi179_148205_object81609:head# 2024-02-09T17:57:32.039 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.039 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:157d6f14:::benchmark_data_smithi179_148205_object54129:head# 2024-02-09T17:57:32.039 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.039 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:157e1431:::benchmark_data_smithi179_148205_object2968:head# 2024-02-09T17:57:32.039 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.039 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:157e4b31:::benchmark_data_smithi179_148205_object64061:head# 2024-02-09T17:57:32.039 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.039 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:157e8d76:::benchmark_data_smithi179_148205_object1080:head# 2024-02-09T17:57:32.039 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.039 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:157e972b:::benchmark_data_smithi179_148205_object42454:head# 2024-02-09T17:57:32.039 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.039 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:157eaa39:::benchmark_data_smithi179_148205_object102812:head# 2024-02-09T17:57:32.039 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.039 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:157ec073:::benchmark_data_smithi179_148205_object94674:head# 2024-02-09T17:57:32.040 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.040 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:157edc51:::benchmark_data_smithi179_148205_object94462:head# 2024-02-09T17:57:32.040 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.040 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:157f43d4:::benchmark_data_smithi179_148205_object38411:head# 2024-02-09T17:57:32.040 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.040 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:157fbb41:::benchmark_data_smithi179_148205_object95294:head# 2024-02-09T17:57:32.040 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.043 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1580cd3f:::benchmark_data_smithi179_148205_object82602:head# 2024-02-09T17:57:32.044 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.044 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1581d3e3:::benchmark_data_smithi179_148205_object54192:head# 2024-02-09T17:57:32.044 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.044 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1581eaf2:::benchmark_data_smithi179_148205_object48375:head# 2024-02-09T17:57:32.044 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.044 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1582330b:::benchmark_data_smithi179_148205_object58113:head# 2024-02-09T17:57:32.044 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.044 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1582bb75:::benchmark_data_smithi179_148205_object90960:head# 2024-02-09T17:57:32.044 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.044 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:158410e7:::benchmark_data_smithi179_148205_object93893:head# 2024-02-09T17:57:32.044 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.044 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1584173b:::benchmark_data_smithi179_148205_object48341:head# 2024-02-09T17:57:32.044 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.044 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1584c9e7:::benchmark_data_smithi179_148205_object87839:head# 2024-02-09T17:57:32.045 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.045 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:158539c0:::benchmark_data_smithi179_148205_object61582:head# 2024-02-09T17:57:32.045 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.045 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1585ac8c:::benchmark_data_smithi179_148205_object48170:head# 2024-02-09T17:57:32.045 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.045 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:158674dc:::benchmark_data_smithi179_148205_object20754:head# 2024-02-09T17:57:32.045 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.045 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15879624:::benchmark_data_smithi179_148205_object86563:head# 2024-02-09T17:57:32.045 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.045 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1587b5d5:::benchmark_data_smithi179_148205_object7804:head# 2024-02-09T17:57:32.045 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.045 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1587bde1:::benchmark_data_smithi179_148205_object72107:head# 2024-02-09T17:57:32.045 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.045 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1587d35f:::benchmark_data_smithi179_148205_object53422:head# 2024-02-09T17:57:32.045 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.046 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15880380:::benchmark_data_smithi179_148205_object64255:head# 2024-02-09T17:57:32.046 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.046 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15887f57:::benchmark_data_smithi179_148205_object49858:head# 2024-02-09T17:57:32.046 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.046 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1588c8c9:::benchmark_data_smithi179_148205_object94652:head# 2024-02-09T17:57:32.046 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.046 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15895006:::benchmark_data_smithi179_148205_object19750:head# 2024-02-09T17:57:32.046 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.046 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15896212:::benchmark_data_smithi179_148205_object98116:head# 2024-02-09T17:57:32.046 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.046 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1589b3f6:::benchmark_data_smithi179_148205_object99340:head# 2024-02-09T17:57:32.046 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.046 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1589c986:::benchmark_data_smithi179_148205_object67103:head# 2024-02-09T17:57:32.046 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.046 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:158a663b:::benchmark_data_smithi179_148205_object80186:head# 2024-02-09T17:57:32.047 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.049 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:158b2368:::benchmark_data_smithi179_148205_object86140:head# 2024-02-09T17:57:32.049 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.049 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:158ccc81:::benchmark_data_smithi179_148205_object8487:head# 2024-02-09T17:57:32.049 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.049 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:158cddc1:::benchmark_data_smithi179_148205_object98401:head# 2024-02-09T17:57:32.049 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.049 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:158cf5e5:::benchmark_data_smithi179_148205_object77569:head# 2024-02-09T17:57:32.049 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.050 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:158d12b1:::benchmark_data_smithi179_148205_object108183:head# 2024-02-09T17:57:32.050 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.050 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:158d1554:::benchmark_data_smithi179_148205_object69337:head# 2024-02-09T17:57:32.050 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.050 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:158d5e29:::benchmark_data_smithi179_148205_object83978:head# 2024-02-09T17:57:32.050 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.050 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:158db77e:::benchmark_data_smithi179_148205_object58005:head# 2024-02-09T17:57:32.050 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.050 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:158e42b3:::benchmark_data_smithi179_148205_object96896:head# 2024-02-09T17:57:32.050 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.050 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:158ec2cb:::benchmark_data_smithi179_148205_object5215:head# 2024-02-09T17:57:32.050 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.050 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:158edfc2:::benchmark_data_smithi179_148205_object102535:head# 2024-02-09T17:57:32.050 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.051 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:158f5caa:::benchmark_data_smithi179_148205_object69609:head# 2024-02-09T17:57:32.051 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.051 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15907948:::benchmark_data_smithi179_148205_object101655:head# 2024-02-09T17:57:32.051 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.051 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1590f2f3:::benchmark_data_smithi179_148205_object48988:head# 2024-02-09T17:57:32.051 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.051 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1591ecb2:::benchmark_data_smithi179_148205_object67701:head# 2024-02-09T17:57:32.051 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.051 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15929660:::benchmark_data_smithi179_148205_object39107:head# 2024-02-09T17:57:32.051 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.051 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1592b997:::benchmark_data_smithi179_148205_object19176:head# 2024-02-09T17:57:32.051 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.051 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15935835:::benchmark_data_smithi179_148205_object106636:head# 2024-02-09T17:57:32.051 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.051 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1594002f:::benchmark_data_smithi179_148205_object29730:head# 2024-02-09T17:57:32.052 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.052 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15946ca2:::benchmark_data_smithi179_148205_object23981:head# 2024-02-09T17:57:32.052 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.052 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1597dbdc:::benchmark_data_smithi179_148205_object93060:head# 2024-02-09T17:57:32.052 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.055 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1599309a:::benchmark_data_smithi179_148205_object13457:head# 2024-02-09T17:57:32.055 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.055 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1599a884:::benchmark_data_smithi179_148205_object76833:head# 2024-02-09T17:57:32.055 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.056 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:159ab323:::benchmark_data_smithi179_148205_object74118:head# 2024-02-09T17:57:32.056 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.056 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:159adab2:::benchmark_data_smithi179_148205_object61743:head# 2024-02-09T17:57:32.056 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.056 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:159be6f6:::benchmark_data_smithi179_148205_object84653:head# 2024-02-09T17:57:32.056 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.056 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:159e1854:::benchmark_data_smithi179_148205_object90258:head# 2024-02-09T17:57:32.056 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.056 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:159e4c81:::benchmark_data_smithi179_148205_object74086:head# 2024-02-09T17:57:32.056 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.056 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:159eaffd:::benchmark_data_smithi179_148205_object107696:head# 2024-02-09T17:57:32.056 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.056 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:159f21a9:::benchmark_data_smithi179_148205_object37097:head# 2024-02-09T17:57:32.056 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.057 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:159f9946:::benchmark_data_smithi179_148205_object13384:head# 2024-02-09T17:57:32.057 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.057 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15a0191f:::benchmark_data_smithi179_148205_object97102:head# 2024-02-09T17:57:32.057 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.057 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15a03ecc:::benchmark_data_smithi179_148205_object3297:head# 2024-02-09T17:57:32.057 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.057 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15a092a5:::benchmark_data_smithi179_148205_object23851:head# 2024-02-09T17:57:32.057 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.057 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15a0b65e:::benchmark_data_smithi179_148205_object88967:head# 2024-02-09T17:57:32.057 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.057 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15a11593:::benchmark_data_smithi179_148205_object5918:head# 2024-02-09T17:57:32.057 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.057 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15a3b558:::benchmark_data_smithi179_148205_object56575:head# 2024-02-09T17:57:32.057 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.057 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15a580b0:::benchmark_data_smithi179_148205_object104992:head# 2024-02-09T17:57:32.057 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.058 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15a6f111:::benchmark_data_smithi179_148205_object92450:head# 2024-02-09T17:57:32.058 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.058 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15a6f4c8:::benchmark_data_smithi179_148205_object54353:head# 2024-02-09T17:57:32.058 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.058 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15a7414a:::benchmark_data_smithi179_148205_object35681:head# 2024-02-09T17:57:32.058 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.058 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15a7aac8:::benchmark_data_smithi179_148205_object106981:head# 2024-02-09T17:57:32.058 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.058 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15a7b80c:::benchmark_data_smithi179_148205_object73597:head# 2024-02-09T17:57:32.058 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.059 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15a7c885:::benchmark_data_smithi179_148205_object77215:head# 2024-02-09T17:57:32.059 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.059 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15a822e3:::benchmark_data_smithi179_148205_object52109:head# 2024-02-09T17:57:32.059 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.059 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15a866fb:::benchmark_data_smithi179_148205_object105270:head# 2024-02-09T17:57:32.060 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.064 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15a89305:::benchmark_data_smithi179_148205_object102045:head# 2024-02-09T17:57:32.064 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.064 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15a896ad:::benchmark_data_smithi179_148205_object63750:head# 2024-02-09T17:57:32.064 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.065 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15a8ac0d:::benchmark_data_smithi179_148205_object28736:head# 2024-02-09T17:57:32.066 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.066 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15a8dfe5:::benchmark_data_smithi179_148205_object105937:head# 2024-02-09T17:57:32.066 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.066 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15a950e9:::benchmark_data_smithi179_148205_object103389:head# 2024-02-09T17:57:32.066 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.066 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15a9c6b7:::benchmark_data_smithi179_148205_object60201:head# 2024-02-09T17:57:32.066 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.067 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15a9c959:::benchmark_data_smithi179_148205_object7711:head# 2024-02-09T17:57:32.067 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.067 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15aa96c9:::benchmark_data_smithi179_148205_object10695:head# 2024-02-09T17:57:32.067 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.067 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15aaf948:::benchmark_data_smithi179_148205_object62649:head# 2024-02-09T17:57:32.067 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.067 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15ad4215:::benchmark_data_smithi179_148205_object97952:head# 2024-02-09T17:57:32.067 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.067 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15af5269:::benchmark_data_smithi179_148205_object90374:head# 2024-02-09T17:57:32.067 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.068 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15af603e:::benchmark_data_smithi179_148205_object71749:head# 2024-02-09T17:57:32.069 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.069 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15af8a3c:::benchmark_data_smithi179_148205_object105360:head# 2024-02-09T17:57:32.069 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.069 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15b08f61:::benchmark_data_smithi179_148205_object78747:head# 2024-02-09T17:57:32.069 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.069 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15b0ce87:::benchmark_data_smithi179_148205_object104689:head# 2024-02-09T17:57:32.069 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.070 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15b14412:::benchmark_data_smithi179_148205_object24374:head# 2024-02-09T17:57:32.070 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.070 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15b19ec6:::benchmark_data_smithi179_148205_object33484:head# 2024-02-09T17:57:32.070 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.070 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15b1e434:::benchmark_data_smithi179_148205_object30062:head# 2024-02-09T17:57:32.070 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.070 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15b1e968:::benchmark_data_smithi179_148205_object81774:head# 2024-02-09T17:57:32.071 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.071 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15b1f334:::benchmark_data_smithi179_148205_object98642:head# 2024-02-09T17:57:32.071 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.071 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15b207a6:::benchmark_data_smithi179_148205_object90602:head# 2024-02-09T17:57:32.071 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.072 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15b23ba1:::benchmark_data_smithi179_148205_object108852:head# 2024-02-09T17:57:32.072 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.072 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15b2b15d:::benchmark_data_smithi179_148205_object30927:head# 2024-02-09T17:57:32.072 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.072 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15b3695d:::benchmark_data_smithi179_148205_object100021:head# 2024-02-09T17:57:32.072 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.072 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15b3bff0:::benchmark_data_smithi179_148205_object7073:head# 2024-02-09T17:57:32.072 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.072 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15b43f61:::benchmark_data_smithi179_148205_object36412:head# 2024-02-09T17:57:32.072 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.073 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15b4accb:::benchmark_data_smithi179_148205_object92860:head# 2024-02-09T17:57:32.073 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.074 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15b72bde:::benchmark_data_smithi179_148205_object80975:head# 2024-02-09T17:57:32.074 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.074 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15b79c89:::benchmark_data_smithi179_148205_object56012:head# 2024-02-09T17:57:32.074 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.074 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15b8b427:::benchmark_data_smithi179_148205_object39138:head# 2024-02-09T17:57:32.074 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.074 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15b9ba3f:::benchmark_data_smithi179_148205_object109738:head# 2024-02-09T17:57:32.074 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.074 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15b9e70e:::benchmark_data_smithi179_148205_object13014:head# 2024-02-09T17:57:32.074 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.074 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15b9fa1f:::benchmark_data_smithi179_148205_object73520:head# 2024-02-09T17:57:32.074 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.076 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15ba0054:::benchmark_data_smithi179_148205_object96683:head# 2024-02-09T17:57:32.076 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.076 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15bb4a7f:::benchmark_data_smithi179_148205_object39739:head# 2024-02-09T17:57:32.077 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.077 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15bbf44e:::benchmark_data_smithi179_148205_object57190:head# 2024-02-09T17:57:32.077 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.077 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15bc4d81:::benchmark_data_smithi179_148205_object37722:head# 2024-02-09T17:57:32.077 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.077 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15bd6584:::benchmark_data_smithi179_148205_object70080:head# 2024-02-09T17:57:32.077 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.077 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15bdd6c6:::benchmark_data_smithi179_148205_object80184:head# 2024-02-09T17:57:32.077 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.077 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15bddf7c:::benchmark_data_smithi179_148205_object771:head# 2024-02-09T17:57:32.077 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.077 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15bfb5eb:::benchmark_data_smithi179_148205_object4481:head# 2024-02-09T17:57:32.077 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.078 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15bfdca4:::benchmark_data_smithi179_148205_object48981:head# 2024-02-09T17:57:32.078 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.078 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15c0a6d4:::benchmark_data_smithi179_148205_object15274:head# 2024-02-09T17:57:32.078 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.078 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15c0b751:::benchmark_data_smithi179_148205_object61236:head# 2024-02-09T17:57:32.078 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.079 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15c1157b:::benchmark_data_smithi179_148205_object18072:head# 2024-02-09T17:57:32.079 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.079 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15c146d5:::benchmark_data_smithi179_148205_object59491:head# 2024-02-09T17:57:32.079 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.079 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15c1b2c3:::benchmark_data_smithi179_148205_object39409:head# 2024-02-09T17:57:32.079 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.079 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15c3a66e:::benchmark_data_smithi179_148205_object6256:head# 2024-02-09T17:57:32.079 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.081 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15c45a9b:::benchmark_data_smithi179_148205_object89029:head# 2024-02-09T17:57:32.081 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.081 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15c4607f:::benchmark_data_smithi179_148205_object87986:head# 2024-02-09T17:57:32.081 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.081 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15c55684:::benchmark_data_smithi179_148205_object75183:head# 2024-02-09T17:57:32.081 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.081 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15c55e8c:::benchmark_data_smithi179_148205_object68495:head# 2024-02-09T17:57:32.081 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.081 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15c5b293:::benchmark_data_smithi179_148205_object10666:head# 2024-02-09T17:57:32.081 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.081 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15c5cfaa:::benchmark_data_smithi179_148205_object28522:head# 2024-02-09T17:57:32.081 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.081 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15c61f47:::benchmark_data_smithi179_148205_object102692:head# 2024-02-09T17:57:32.081 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.082 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15c6e54d:::benchmark_data_smithi179_148205_object10379:head# 2024-02-09T17:57:32.082 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.082 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15c71d1b:::benchmark_data_smithi179_148205_object30875:head# 2024-02-09T17:57:32.083 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.083 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15c73990:::benchmark_data_smithi179_148205_object49718:head# 2024-02-09T17:57:32.083 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.083 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15c7754f:::benchmark_data_smithi179_148205_object45669:head# 2024-02-09T17:57:32.083 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.083 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15c84719:::benchmark_data_smithi179_148205_object22102:head# 2024-02-09T17:57:32.083 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.087 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15c8ca3e:::benchmark_data_smithi179_148205_object39595:head# 2024-02-09T17:57:32.088 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.088 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15c8fd40:::benchmark_data_smithi179_148205_object29301:head# 2024-02-09T17:57:32.088 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.089 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15c90387:::benchmark_data_smithi179_148205_object34903:head# 2024-02-09T17:57:32.089 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.089 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15c90ccf:::benchmark_data_smithi179_148205_object10772:head# 2024-02-09T17:57:32.089 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.089 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15c93901:::benchmark_data_smithi179_148205_object31455:head# 2024-02-09T17:57:32.089 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.089 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15c9472c:::benchmark_data_smithi179_148205_object4585:head# 2024-02-09T17:57:32.089 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.090 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15c976a3:::benchmark_data_smithi179_148205_object10612:head# 2024-02-09T17:57:32.090 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.091 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15c99de1:::benchmark_data_smithi179_148205_object21365:head# 2024-02-09T17:57:32.091 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.091 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15ca1d76:::benchmark_data_smithi179_148205_object10248:head# 2024-02-09T17:57:32.091 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.091 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15ca47d8:::benchmark_data_smithi179_148205_object59979:head# 2024-02-09T17:57:32.091 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.091 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15ca991b:::benchmark_data_smithi179_148205_object23031:head# 2024-02-09T17:57:32.091 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.091 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15cc2fed:::benchmark_data_smithi179_148205_object2502:head# 2024-02-09T17:57:32.091 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.092 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15cc6778:::benchmark_data_smithi179_148205_object103456:head# 2024-02-09T17:57:32.092 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.093 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15ccc77d:::benchmark_data_smithi179_148205_object41115:head# 2024-02-09T17:57:32.093 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.093 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15ccd81f:::benchmark_data_smithi179_148205_object74665:head# 2024-02-09T17:57:32.093 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.093 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15cd8289:::benchmark_data_smithi179_148205_object98172:head# 2024-02-09T17:57:32.093 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.093 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15ce6432:::benchmark_data_smithi179_148205_object27156:head# 2024-02-09T17:57:32.094 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.094 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15cea4a4:::benchmark_data_smithi179_148205_object54581:head# 2024-02-09T17:57:32.094 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.094 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15cf1932:::benchmark_data_smithi179_148205_object1387:head# 2024-02-09T17:57:32.094 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.094 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15cfdbb7:::benchmark_data_smithi179_148205_object75608:head# 2024-02-09T17:57:32.094 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.098 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15d05fb2:::benchmark_data_smithi179_148205_object17329:head# 2024-02-09T17:57:32.098 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.102 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15d09a01:::benchmark_data_smithi179_148205_object81442:head# 2024-02-09T17:57:32.102 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.103 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15d0aa62:::benchmark_data_smithi179_148205_object98688:head# 2024-02-09T17:57:32.103 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.103 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15d1923a:::benchmark_data_smithi179_148205_object83914:head# 2024-02-09T17:57:32.103 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.103 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15d201de:::benchmark_data_smithi179_148205_object20879:head# 2024-02-09T17:57:32.103 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.103 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15d2aef5:::benchmark_data_smithi179_148205_object20610:head# 2024-02-09T17:57:32.103 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.103 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15d3eb7d:::benchmark_data_smithi179_148205_object80227:head# 2024-02-09T17:57:32.103 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.103 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15d4f59a:::benchmark_data_smithi179_148205_object81888:head# 2024-02-09T17:57:32.103 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.107 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15d5104d:::benchmark_data_smithi179_148205_object66912:head# 2024-02-09T17:57:32.107 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.112 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15d6a3bf:::benchmark_data_smithi179_148205_object90950:head# 2024-02-09T17:57:32.112 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.112 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15d75e17:::benchmark_data_smithi179_148205_object96674:head# 2024-02-09T17:57:32.112 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.112 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15d88981:::benchmark_data_smithi179_148205_object45011:head# 2024-02-09T17:57:32.113 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.113 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15d921be:::benchmark_data_smithi179_148205_object22326:head# 2024-02-09T17:57:32.113 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.113 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15d981a5:::benchmark_data_smithi179_148205_object26537:head# 2024-02-09T17:57:32.114 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.114 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15d9b2a2:::benchmark_data_smithi179_148205_object85081:head# 2024-02-09T17:57:32.114 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.114 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15da8ab8:::benchmark_data_smithi179_148205_object55696:head# 2024-02-09T17:57:32.114 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.114 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15dada58:::benchmark_data_smithi179_148205_object12404:head# 2024-02-09T17:57:32.115 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.115 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15dba0e8:::benchmark_data_smithi179_148205_object76898:head# 2024-02-09T17:57:32.115 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.115 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15dbc851:::benchmark_data_smithi179_148205_object13897:head# 2024-02-09T17:57:32.115 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.115 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15dc0dc5:::benchmark_data_smithi179_148205_object18567:head# 2024-02-09T17:57:32.115 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.116 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15dc7d67:::benchmark_data_smithi179_148205_object34345:head# 2024-02-09T17:57:32.116 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.116 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15dd7e2e:::benchmark_data_smithi179_148205_object61766:head# 2024-02-09T17:57:32.116 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.116 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15de3a67:::benchmark_data_smithi179_148205_object61246:head# 2024-02-09T17:57:32.116 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.116 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15de4745:::benchmark_data_smithi179_148205_object99622:head# 2024-02-09T17:57:32.116 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.116 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15de5910:::benchmark_data_smithi179_148205_object92006:head# 2024-02-09T17:57:32.117 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.117 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15de611d:::benchmark_data_smithi179_148205_object104569:head# 2024-02-09T17:57:32.117 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.117 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15de9fde:::benchmark_data_smithi179_148205_object57441:head# 2024-02-09T17:57:32.118 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.118 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15dfb183:::benchmark_data_smithi179_148205_object85213:head# 2024-02-09T17:57:32.118 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.118 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15dfd3e1:::benchmark_data_smithi179_148205_object69509:head# 2024-02-09T17:57:32.118 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.118 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15e0efe0:::benchmark_data_smithi179_148205_object39375:head# 2024-02-09T17:57:32.118 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.119 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15e19dd4:::benchmark_data_smithi179_148205_object23810:head# 2024-02-09T17:57:32.119 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.119 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15e1bc27:::benchmark_data_smithi179_148205_object9951:head# 2024-02-09T17:57:32.119 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.119 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15e30a3d:::benchmark_data_smithi179_148205_object53777:head# 2024-02-09T17:57:32.119 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.119 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15e3bfb2:::benchmark_data_smithi179_148205_object28158:head# 2024-02-09T17:57:32.119 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.122 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15e3da40:::benchmark_data_smithi179_148205_object69064:head# 2024-02-09T17:57:32.122 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.122 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15e3ec14:::benchmark_data_smithi179_148205_object70879:head# 2024-02-09T17:57:32.122 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.122 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15e4e58c:::benchmark_data_smithi179_148205_object52244:head# 2024-02-09T17:57:32.122 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.122 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15e4e6bd:::benchmark_data_smithi179_148205_object54674:head# 2024-02-09T17:57:32.122 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.122 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15e56873:::benchmark_data_smithi179_148205_object83363:head# 2024-02-09T17:57:32.122 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.122 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15e6deba:::benchmark_data_smithi179_148205_object2966:head# 2024-02-09T17:57:32.123 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.123 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15e6f59f:::benchmark_data_smithi179_148205_object91914:head# 2024-02-09T17:57:32.123 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.123 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15e704fc:::benchmark_data_smithi179_148205_object96364:head# 2024-02-09T17:57:32.123 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.123 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15e79345:::benchmark_data_smithi179_148205_object87449:head# 2024-02-09T17:57:32.123 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.123 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15e91edd:::benchmark_data_smithi179_148205_object33751:head# 2024-02-09T17:57:32.123 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.123 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15e99900:::benchmark_data_smithi179_148205_object72827:head# 2024-02-09T17:57:32.123 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.123 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15e9cc4e:::benchmark_data_smithi179_148205_object94657:head# 2024-02-09T17:57:32.123 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.123 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15ea671c:::benchmark_data_smithi179_148205_object44767:head# 2024-02-09T17:57:32.123 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.128 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15ebb2cf:::benchmark_data_smithi179_148205_object60689:head# 2024-02-09T17:57:32.128 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.128 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15ec301f:::benchmark_data_smithi179_148205_object74865:head# 2024-02-09T17:57:32.128 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.128 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15ed5a90:::benchmark_data_smithi179_148205_object31458:head# 2024-02-09T17:57:32.128 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.128 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15ed5bd6:::benchmark_data_smithi179_148205_object83189:head# 2024-02-09T17:57:32.128 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.129 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15ed95b2:::benchmark_data_smithi179_148205_object20574:head# 2024-02-09T17:57:32.129 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.129 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15ef52df:::benchmark_data_smithi179_148205_object21735:head# 2024-02-09T17:57:32.129 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.129 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15ef95d5:::benchmark_data_smithi179_148205_object6627:head# 2024-02-09T17:57:32.129 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.129 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15f078d2:::benchmark_data_smithi179_148205_object32526:head# 2024-02-09T17:57:32.129 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.129 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15f0bd0f:::benchmark_data_smithi179_148205_object10176:head# 2024-02-09T17:57:32.129 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.129 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15f10d91:::benchmark_data_smithi179_148205_object67903:head# 2024-02-09T17:57:32.129 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.129 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15f13196:::benchmark_data_smithi179_148205_object99428:head# 2024-02-09T17:57:32.129 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.130 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15f16303:::benchmark_data_smithi179_148205_object15618:head# 2024-02-09T17:57:32.130 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.130 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15f2a6ce:::benchmark_data_smithi179_148205_object22100:head# 2024-02-09T17:57:32.130 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.130 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15f2dcf4:::benchmark_data_smithi179_148205_object77839:head# 2024-02-09T17:57:32.130 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.130 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15f2e599:::benchmark_data_smithi179_148205_object38814:head# 2024-02-09T17:57:32.130 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.130 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15f35242:::benchmark_data_smithi179_148205_object81791:head# 2024-02-09T17:57:32.130 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.130 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15f39c4d:::benchmark_data_smithi179_148205_object78855:head# 2024-02-09T17:57:32.130 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.130 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15f4f0d1:::benchmark_data_smithi179_148205_object6221:head# 2024-02-09T17:57:32.130 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.130 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15f4f20f:::benchmark_data_smithi179_148205_object107795:head# 2024-02-09T17:57:32.130 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.131 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15f57cab:::benchmark_data_smithi179_148205_object37663:head# 2024-02-09T17:57:32.131 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.131 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15f6ca0e:::benchmark_data_smithi179_148205_object86033:head# 2024-02-09T17:57:32.131 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.131 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15f6ead3:::benchmark_data_smithi179_148205_object92010:head# 2024-02-09T17:57:32.131 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.131 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15f6f684:::benchmark_data_smithi179_148205_object64297:head# 2024-02-09T17:57:32.131 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.131 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15f785a2:::benchmark_data_smithi179_148205_object35259:head# 2024-02-09T17:57:32.131 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.131 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15f80c64:::benchmark_data_smithi179_148205_object42225:head# 2024-02-09T17:57:32.131 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.131 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15f81717:::benchmark_data_smithi179_148205_object43786:head# 2024-02-09T17:57:32.131 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.131 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15f8b8c3:::benchmark_data_smithi179_148205_object33092:head# 2024-02-09T17:57:32.132 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.141 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15f9dbaf:::benchmark_data_smithi179_148205_object53806:head# 2024-02-09T17:57:32.141 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.141 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15fa4bf8:::benchmark_data_smithi179_148205_object61316:head# 2024-02-09T17:57:32.141 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.141 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15fa9776:::benchmark_data_smithi179_148205_object77935:head# 2024-02-09T17:57:32.141 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.142 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15fb9ae1:::benchmark_data_smithi179_148205_object109146:head# 2024-02-09T17:57:32.142 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.142 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15fc2649:::benchmark_data_smithi179_148205_object34517:head# 2024-02-09T17:57:32.142 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.142 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15fc5abe:::benchmark_data_smithi179_148205_object96506:head# 2024-02-09T17:57:32.142 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.142 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15fcd35a:::benchmark_data_smithi179_148205_object56260:head# 2024-02-09T17:57:32.142 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.142 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15fdb80b:::benchmark_data_smithi179_148205_object103452:head# 2024-02-09T17:57:32.142 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.142 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15febe69:::benchmark_data_smithi179_148205_object15375:head# 2024-02-09T17:57:32.142 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.142 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15ff12cb:::benchmark_data_smithi179_148205_object92060:head# 2024-02-09T17:57:32.142 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.143 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15ff3005:::benchmark_data_smithi179_148205_object104280:head# 2024-02-09T17:57:32.143 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.143 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15ff4abc:::benchmark_data_smithi179_148205_object64182:head# 2024-02-09T17:57:32.143 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.143 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15ff67c0:::benchmark_data_smithi179_148205_object77469:head# 2024-02-09T17:57:32.143 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.143 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15ff8934:::benchmark_data_smithi179_148205_object3805:head# 2024-02-09T17:57:32.143 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.143 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:15ff945c:::benchmark_data_smithi179_148205_object89890:head# 2024-02-09T17:57:32.143 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.143 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16005d66:::benchmark_data_smithi179_148205_object5672:head# 2024-02-09T17:57:32.143 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.143 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1600b0b3:::benchmark_data_smithi179_148205_object57212:head# 2024-02-09T17:57:32.143 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.143 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16018b17:::benchmark_data_smithi179_148205_object25995:head# 2024-02-09T17:57:32.144 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.144 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1601c01d:::benchmark_data_smithi179_148205_object15771:head# 2024-02-09T17:57:32.144 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.144 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1601c97a:::benchmark_data_smithi179_148205_object101225:head# 2024-02-09T17:57:32.144 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.144 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16023959:::benchmark_data_smithi179_148205_object100791:head# 2024-02-09T17:57:32.144 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.144 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16026fa2:::benchmark_data_smithi179_148205_object103026:head# 2024-02-09T17:57:32.144 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.144 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:160339c0:::benchmark_data_smithi179_148205_object18940:head# 2024-02-09T17:57:32.144 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.144 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16037d73:::benchmark_data_smithi179_148205_object35150:head# 2024-02-09T17:57:32.144 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.144 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1603f31e:::benchmark_data_smithi179_148205_object84834:head# 2024-02-09T17:57:32.145 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.145 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16040932:::benchmark_data_smithi179_148205_object96797:head# 2024-02-09T17:57:32.145 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.145 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16049dc3:::benchmark_data_smithi179_148205_object37915:head# 2024-02-09T17:57:32.145 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.145 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1604b4d6:::benchmark_data_smithi179_148205_object45297:head# 2024-02-09T17:57:32.145 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.145 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1604d976:::benchmark_data_smithi179_148205_object109118:head# 2024-02-09T17:57:32.145 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.145 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1605013d:::benchmark_data_smithi179_148205_object107593:head# 2024-02-09T17:57:32.145 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.145 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16053840:::benchmark_data_smithi179_148205_object4742:head# 2024-02-09T17:57:32.145 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.145 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1605d3fb:::benchmark_data_smithi179_148205_object32348:head# 2024-02-09T17:57:32.145 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.145 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:160833e6:::benchmark_data_smithi179_148205_object32376:head# 2024-02-09T17:57:32.146 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.146 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16099cac:::benchmark_data_smithi179_148205_object36921:head# 2024-02-09T17:57:32.146 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.146 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1609b37c:::benchmark_data_smithi179_148205_object58505:head# 2024-02-09T17:57:32.146 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.146 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1609b61f:::benchmark_data_smithi179_148205_object78042:head# 2024-02-09T17:57:32.146 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.146 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1609fcd6:::benchmark_data_smithi179_148205_object49600:head# 2024-02-09T17:57:32.146 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.146 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:160a395d:::benchmark_data_smithi179_148205_object109186:head# 2024-02-09T17:57:32.146 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.146 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:160c3e6a:::benchmark_data_smithi179_148205_object102258:head# 2024-02-09T17:57:32.146 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.146 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:160c7af3:::benchmark_data_smithi179_148205_object103857:head# 2024-02-09T17:57:32.147 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.147 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:160cf2c0:::benchmark_data_smithi179_148205_object51421:head# 2024-02-09T17:57:32.147 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.147 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:160d20c6:::benchmark_data_smithi179_148205_object55522:head# 2024-02-09T17:57:32.147 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.147 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:160d6284:::benchmark_data_smithi179_148205_object68395:head# 2024-02-09T17:57:32.147 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.147 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:160ed043:::benchmark_data_smithi179_148205_object41479:head# 2024-02-09T17:57:32.147 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.147 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:160f3229:::benchmark_data_smithi179_148205_object86588:head# 2024-02-09T17:57:32.147 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.147 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:160f97f8:::benchmark_data_smithi179_148205_object37612:head# 2024-02-09T17:57:32.147 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.147 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1610a14b:::benchmark_data_smithi179_148205_object17646:head# 2024-02-09T17:57:32.147 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.148 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1610a264:::benchmark_data_smithi179_148205_object87171:head# 2024-02-09T17:57:32.148 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.148 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16127f6c:::benchmark_data_smithi179_148205_object19071:head# 2024-02-09T17:57:32.148 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.148 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16161f2a:::benchmark_data_smithi179_148205_object67861:head# 2024-02-09T17:57:32.148 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.148 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:161672c2:::benchmark_data_smithi179_148205_object24839:head# 2024-02-09T17:57:32.148 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.148 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1616c919:::benchmark_data_smithi179_148205_object78157:head# 2024-02-09T17:57:32.148 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.148 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16173444:::benchmark_data_smithi179_148205_object66963:head# 2024-02-09T17:57:32.148 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.148 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1617af13:::benchmark_data_smithi179_148205_object53220:head# 2024-02-09T17:57:32.148 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.149 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1618cb1b:::benchmark_data_smithi179_148205_object16534:head# 2024-02-09T17:57:32.149 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.161 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:161a36af:::benchmark_data_smithi179_148205_object80192:head# 2024-02-09T17:57:32.161 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.161 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:161a8a2c:::benchmark_data_smithi179_148205_object52076:head# 2024-02-09T17:57:32.161 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.161 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:161b2c99:::benchmark_data_smithi179_148205_object14769:head# 2024-02-09T17:57:32.161 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.161 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:161baf01:::benchmark_data_smithi179_148205_object108569:head# 2024-02-09T17:57:32.161 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.161 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:161e7aa5:::benchmark_data_smithi179_148205_object73902:head# 2024-02-09T17:57:32.161 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.161 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:161edd5c:::benchmark_data_smithi179_148205_object68971:head# 2024-02-09T17:57:32.161 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.161 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:161f5bc7:::benchmark_data_smithi179_148205_object14797:head# 2024-02-09T17:57:32.162 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.162 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:161fb430:::benchmark_data_smithi179_148205_object17776:head# 2024-02-09T17:57:32.162 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.162 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16207bce:::benchmark_data_smithi179_148205_object64687:head# 2024-02-09T17:57:32.162 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.162 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1620c134:::benchmark_data_smithi179_148205_object36940:head# 2024-02-09T17:57:32.162 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.162 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1621b3ca:::benchmark_data_smithi179_148205_object96040:head# 2024-02-09T17:57:32.162 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.162 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16238b54:::benchmark_data_smithi179_148205_object36875:head# 2024-02-09T17:57:32.162 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.162 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1623ee40:::benchmark_data_smithi179_148205_object25004:head# 2024-02-09T17:57:32.162 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.162 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1625ac3e:::benchmark_data_smithi179_148205_object36293:head# 2024-02-09T17:57:32.163 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.163 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:162610eb:::benchmark_data_smithi179_148205_object100999:head# 2024-02-09T17:57:32.163 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.163 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16271e9d:::benchmark_data_smithi179_148205_object22604:head# 2024-02-09T17:57:32.163 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.163 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:162782ec:::benchmark_data_smithi179_148205_object58363:head# 2024-02-09T17:57:32.163 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.163 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16280dfb:::benchmark_data_smithi179_148205_object21128:head# 2024-02-09T17:57:32.163 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.163 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1628a2ef:::benchmark_data_smithi179_148205_object64164:head# 2024-02-09T17:57:32.163 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.163 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16292367:::benchmark_data_smithi179_148205_object97086:head# 2024-02-09T17:57:32.163 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.163 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:162ae4fd:::benchmark_data_smithi179_148205_object55812:head# 2024-02-09T17:57:32.163 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.164 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:162b0243:::benchmark_data_smithi179_148205_object66267:head# 2024-02-09T17:57:32.164 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.164 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:162b1739:::benchmark_data_smithi179_148205_object90341:head# 2024-02-09T17:57:32.164 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.164 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:162b50d4:::benchmark_data_smithi179_148205_object51887:head# 2024-02-09T17:57:32.164 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.164 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:162cbb43:::benchmark_data_smithi179_148205_object91677:head# 2024-02-09T17:57:32.164 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.164 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:162d3a14:::benchmark_data_smithi179_148205_object69501:head# 2024-02-09T17:57:32.164 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.164 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:162d3d1e:::benchmark_data_smithi179_148205_object88825:head# 2024-02-09T17:57:32.164 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.164 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:162dd8f1:::benchmark_data_smithi179_148205_object84459:head# 2024-02-09T17:57:32.164 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.164 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:162f0217:::benchmark_data_smithi179_148205_object18505:head# 2024-02-09T17:57:32.165 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.165 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:162f6f66:::benchmark_data_smithi179_148205_object57266:head# 2024-02-09T17:57:32.165 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.165 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:162f82d2:::benchmark_data_smithi179_148205_object13430:head# 2024-02-09T17:57:32.165 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.165 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:162f9f49:::benchmark_data_smithi179_148205_object107783:head# 2024-02-09T17:57:32.165 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.165 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:162fa377:::benchmark_data_smithi179_148205_object103671:head# 2024-02-09T17:57:32.165 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.165 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16301cc0:::benchmark_data_smithi179_148205_object84662:head# 2024-02-09T17:57:32.165 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.165 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1630959b:::benchmark_data_smithi179_148205_object46434:head# 2024-02-09T17:57:32.165 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.165 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16316113:::benchmark_data_smithi179_148205_object62840:head# 2024-02-09T17:57:32.165 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.166 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1631c48e:::benchmark_data_smithi179_148205_object17814:head# 2024-02-09T17:57:32.166 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.166 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16331d7d:::benchmark_data_smithi179_148205_object38610:head# 2024-02-09T17:57:32.166 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.166 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16353336:::benchmark_data_smithi179_148205_object106934:head# 2024-02-09T17:57:32.166 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.166 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:163558c6:::benchmark_data_smithi179_148205_object87715:head# 2024-02-09T17:57:32.166 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.166 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16363880:::benchmark_data_smithi179_148205_object18364:head# 2024-02-09T17:57:32.166 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.166 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1636e527:::benchmark_data_smithi179_148205_object104075:head# 2024-02-09T17:57:32.166 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.166 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1636e61d:::benchmark_data_smithi179_148205_object32884:head# 2024-02-09T17:57:32.166 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.166 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16388247:::benchmark_data_smithi179_148205_object8162:head# 2024-02-09T17:57:32.167 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.167 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:163967f5:::benchmark_data_smithi179_148205_object29209:head# 2024-02-09T17:57:32.167 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.167 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16398b3b:::benchmark_data_smithi179_148205_object6879:head# 2024-02-09T17:57:32.167 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.167 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1639d0b8:::benchmark_data_smithi179_148205_object27832:head# 2024-02-09T17:57:32.167 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.167 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:163a46c9:::benchmark_data_smithi179_148205_object65455:head# 2024-02-09T17:57:32.167 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.167 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:163a70a2:::benchmark_data_smithi179_148205_object52746:head# 2024-02-09T17:57:32.167 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.167 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:163a87fb:::benchmark_data_smithi179_148205_object60028:head# 2024-02-09T17:57:32.167 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.167 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:163ab82c:::benchmark_data_smithi179_148205_object96212:head# 2024-02-09T17:57:32.168 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.168 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:163abf74:::benchmark_data_smithi179_148205_object52620:head# 2024-02-09T17:57:32.168 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.168 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:163b968c:::benchmark_data_smithi179_148205_object89305:head# 2024-02-09T17:57:32.168 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.168 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:163be316:::benchmark_data_smithi179_148205_object13330:head# 2024-02-09T17:57:32.168 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.168 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:163cb019:::benchmark_data_smithi179_148205_object67238:head# 2024-02-09T17:57:32.168 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.168 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:163d5c07:::benchmark_data_smithi179_148205_object33461:head# 2024-02-09T17:57:32.168 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.168 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:163d8ec4:::benchmark_data_smithi179_148205_object42002:head# 2024-02-09T17:57:32.168 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.168 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:163da1e0:::benchmark_data_smithi179_148205_object66463:head# 2024-02-09T17:57:32.168 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.169 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:163e0b96:::benchmark_data_smithi179_148205_object62981:head# 2024-02-09T17:57:32.169 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.169 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:163e4134:::benchmark_data_smithi179_148205_object6514:head# 2024-02-09T17:57:32.169 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.169 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:163eab20:::benchmark_data_smithi179_148205_object85722:head# 2024-02-09T17:57:32.169 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.169 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:163f147d:::benchmark_data_smithi179_148205_object40063:head# 2024-02-09T17:57:32.169 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.169 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:163f596e:::benchmark_data_smithi179_148205_object97364:head# 2024-02-09T17:57:32.169 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.169 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:163fc46a:::benchmark_data_smithi179_148205_object37488:head# 2024-02-09T17:57:32.169 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.169 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1640438e:::benchmark_data_smithi179_148205_object82288:head# 2024-02-09T17:57:32.169 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.169 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1640a9ff:::benchmark_data_smithi179_148205_object46382:head# 2024-02-09T17:57:32.170 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.170 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1641a8a8:::benchmark_data_smithi179_148205_object21560:head# 2024-02-09T17:57:32.170 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.170 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1641b89f:::benchmark_data_smithi179_148205_object81709:head# 2024-02-09T17:57:32.170 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.170 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1642e5a2:::benchmark_data_smithi179_148205_object6892:head# 2024-02-09T17:57:32.170 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.170 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16439ee0:::benchmark_data_smithi179_148205_object97604:head# 2024-02-09T17:57:32.170 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.170 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1643f2ba:::benchmark_data_smithi179_148205_object89791:head# 2024-02-09T17:57:32.170 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.170 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16443bdf:::benchmark_data_smithi179_148205_object5775:head# 2024-02-09T17:57:32.170 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.170 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16451ab1:::benchmark_data_smithi179_148205_object90533:head# 2024-02-09T17:57:32.170 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.171 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1647576e:::benchmark_data_smithi179_148205_object8909:head# 2024-02-09T17:57:32.171 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.171 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:164815b6:::benchmark_data_smithi179_148205_object34046:head# 2024-02-09T17:57:32.171 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.180 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1648c421:::benchmark_data_smithi179_148205_object38061:head# 2024-02-09T17:57:32.180 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.180 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1648e5d9:::benchmark_data_smithi179_148205_object19668:head# 2024-02-09T17:57:32.180 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.180 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:164a61c1:::benchmark_data_smithi179_148205_object74469:head# 2024-02-09T17:57:32.180 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.180 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:164a893b:::benchmark_data_smithi179_148205_object28349:head# 2024-02-09T17:57:32.180 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.181 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:164b2be9:::benchmark_data_smithi179_148205_object101121:head# 2024-02-09T17:57:32.181 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.181 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:164bba3e:::benchmark_data_smithi179_148205_object46437:head# 2024-02-09T17:57:32.181 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.181 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:164c1236:::benchmark_data_smithi179_148205_object47815:head# 2024-02-09T17:57:32.181 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.181 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:164cff24:::benchmark_data_smithi179_148205_object40966:head# 2024-02-09T17:57:32.181 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.181 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:164d21de:::benchmark_data_smithi179_148205_object46084:head# 2024-02-09T17:57:32.181 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.181 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:164dfaa3:::benchmark_data_smithi179_148205_object67014:head# 2024-02-09T17:57:32.181 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.181 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:164f5da0:::benchmark_data_smithi179_148205_object48468:head# 2024-02-09T17:57:32.181 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.181 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:164f8aa1:::benchmark_data_smithi179_148205_object23337:head# 2024-02-09T17:57:32.182 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.182 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1650bda4:::benchmark_data_smithi179_148205_object3053:head# 2024-02-09T17:57:32.182 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.182 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1650fe05:::benchmark_data_smithi179_148205_object66159:head# 2024-02-09T17:57:32.182 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.182 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16519744:::benchmark_data_smithi179_148205_object64241:head# 2024-02-09T17:57:32.182 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.182 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16524874:::benchmark_data_smithi179_148205_object36264:head# 2024-02-09T17:57:32.182 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.182 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16526808:::benchmark_data_smithi179_148205_object52358:head# 2024-02-09T17:57:32.182 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.182 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16535834:::benchmark_data_smithi179_148205_object39620:head# 2024-02-09T17:57:32.182 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.182 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:165508d8:::benchmark_data_smithi179_148205_object97078:head# 2024-02-09T17:57:32.182 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.183 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1655a78a:::benchmark_data_smithi179_148205_object89321:head# 2024-02-09T17:57:32.183 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.183 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16563f18:::benchmark_data_smithi179_148205_object68263:head# 2024-02-09T17:57:32.183 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.183 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16568b9f:::benchmark_data_smithi179_148205_object30896:head# 2024-02-09T17:57:32.183 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.183 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:165752da:::benchmark_data_smithi179_148205_object95746:head# 2024-02-09T17:57:32.183 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.183 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1657b86e:::benchmark_data_smithi179_148205_object99254:head# 2024-02-09T17:57:32.183 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.183 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1657c3a3:::benchmark_data_smithi179_148205_object102216:head# 2024-02-09T17:57:32.183 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.183 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:165815c4:::benchmark_data_smithi179_148205_object25776:head# 2024-02-09T17:57:32.183 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.184 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1658a350:::benchmark_data_smithi179_148205_object94929:head# 2024-02-09T17:57:32.184 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.184 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1658aa35:::benchmark_data_smithi179_148205_object98328:head# 2024-02-09T17:57:32.184 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.184 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16593abb:::benchmark_data_smithi179_148205_object43236:head# 2024-02-09T17:57:32.184 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.184 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16597762:::benchmark_data_smithi179_148205_object9362:head# 2024-02-09T17:57:32.184 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.184 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:165997b3:::benchmark_data_smithi179_148205_object49255:head# 2024-02-09T17:57:32.184 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.184 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:165b2e45:::benchmark_data_smithi179_148205_object47685:head# 2024-02-09T17:57:32.184 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.184 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:165c3f71:::benchmark_data_smithi179_148205_object52989:head# 2024-02-09T17:57:32.184 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.185 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:165c8e33:::benchmark_data_smithi179_148205_object25515:head# 2024-02-09T17:57:32.185 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.185 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:165cc5c4:::benchmark_data_smithi179_148205_object33972:head# 2024-02-09T17:57:32.185 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.185 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:165d1b9f:::benchmark_data_smithi179_148205_object21070:head# 2024-02-09T17:57:32.185 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.185 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:165dff31:::benchmark_data_smithi179_148205_object3902:head# 2024-02-09T17:57:32.185 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.185 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:165e1994:::benchmark_data_smithi179_148205_object38409:head# 2024-02-09T17:57:32.185 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.185 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:165ef024:::benchmark_data_smithi179_148205_object85329:head# 2024-02-09T17:57:32.185 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.185 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:165fbd36:::benchmark_data_smithi179_148205_object60999:head# 2024-02-09T17:57:32.185 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.186 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:165fcff6:::benchmark_data_smithi179_148205_object8039:head# 2024-02-09T17:57:32.186 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.186 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:166047c5:::benchmark_data_smithi179_148205_object57065:head# 2024-02-09T17:57:32.186 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.186 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16605722:::benchmark_data_smithi179_148205_object38484:head# 2024-02-09T17:57:32.186 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.186 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16611b52:::benchmark_data_smithi179_148205_object106280:head# 2024-02-09T17:57:32.186 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.186 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16613580:::benchmark_data_smithi179_148205_object22490:head# 2024-02-09T17:57:32.186 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.186 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16618795:::benchmark_data_smithi179_148205_object107051:head# 2024-02-09T17:57:32.186 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.186 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1662616e:::benchmark_data_smithi179_148205_object96863:head# 2024-02-09T17:57:32.186 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.186 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1662f75e:::benchmark_data_smithi179_148205_object48136:head# 2024-02-09T17:57:32.187 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.187 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:166311eb:::benchmark_data_smithi179_148205_object82975:head# 2024-02-09T17:57:32.187 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.187 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16633183:::benchmark_data_smithi179_148205_object61693:head# 2024-02-09T17:57:32.187 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.187 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16634f8c:::benchmark_data_smithi179_148205_object11816:head# 2024-02-09T17:57:32.187 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.187 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1663ab69:::benchmark_data_smithi179_148205_object5241:head# 2024-02-09T17:57:32.187 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.187 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16646282:::benchmark_data_smithi179_148205_object72941:head# 2024-02-09T17:57:32.187 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.187 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16651ee1:::benchmark_data_smithi179_148205_object31663:head# 2024-02-09T17:57:32.187 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.187 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16663a2b:::benchmark_data_smithi179_148205_object10019:head# 2024-02-09T17:57:32.188 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.188 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1667458f:::benchmark_data_smithi179_148205_object26029:head# 2024-02-09T17:57:32.188 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.188 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1668b824:::benchmark_data_smithi179_148205_object100811:head# 2024-02-09T17:57:32.188 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.188 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16693f94:::benchmark_data_smithi179_148205_object21195:head# 2024-02-09T17:57:32.188 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.188 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:166b21e6:::benchmark_data_smithi179_148205_object108219:head# 2024-02-09T17:57:32.188 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.188 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:166d6e4e:::benchmark_data_smithi179_148205_object24880:head# 2024-02-09T17:57:32.188 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.188 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:166d8b04:::benchmark_data_smithi179_148205_object78397:head# 2024-02-09T17:57:32.188 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.188 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:166e7c59:::benchmark_data_smithi179_148205_object4017:head# 2024-02-09T17:57:32.189 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.189 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:166fe1a9:::benchmark_data_smithi179_148205_object101991:head# 2024-02-09T17:57:32.189 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.189 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1670425b:::benchmark_data_smithi179_148205_object78273:head# 2024-02-09T17:57:32.189 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.189 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:167084c5:::benchmark_data_smithi179_148205_object22464:head# 2024-02-09T17:57:32.189 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.189 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1670e2d8:::benchmark_data_smithi179_148205_object23425:head# 2024-02-09T17:57:32.189 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.189 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16717df3:::benchmark_data_smithi179_148205_object44690:head# 2024-02-09T17:57:32.189 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.189 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1671fa57:::benchmark_data_smithi179_148205_object9715:head# 2024-02-09T17:57:32.189 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.189 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16736d40:::benchmark_data_smithi179_148205_object107501:head# 2024-02-09T17:57:32.189 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.196 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1673abe0:::benchmark_data_smithi179_148205_object106344:head# 2024-02-09T17:57:32.196 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.196 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1673b6d7:::benchmark_data_smithi179_148205_object107975:head# 2024-02-09T17:57:32.196 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.196 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16741f17:::benchmark_data_smithi179_148205_object28096:head# 2024-02-09T17:57:32.196 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.196 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16742f21:::benchmark_data_smithi179_148205_object48841:head# 2024-02-09T17:57:32.196 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.196 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1674e4f3:::benchmark_data_smithi179_148205_object106684:head# 2024-02-09T17:57:32.196 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.196 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1674e5a8:::benchmark_data_smithi179_148205_object51079:head# 2024-02-09T17:57:32.196 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.196 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16762fed:::benchmark_data_smithi179_148205_object102745:head# 2024-02-09T17:57:32.197 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.197 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:167663a1:::benchmark_data_smithi179_148205_object20231:head# 2024-02-09T17:57:32.197 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.197 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16771c44:::benchmark_data_smithi179_148205_object90827:head# 2024-02-09T17:57:32.197 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.197 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16779b73:::benchmark_data_smithi179_148205_object20670:head# 2024-02-09T17:57:32.197 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.197 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1677fc4f:::benchmark_data_smithi179_148205_object19515:head# 2024-02-09T17:57:32.197 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.197 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16782894:::benchmark_data_smithi179_148205_object108305:head# 2024-02-09T17:57:32.197 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.197 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:167a369c:::benchmark_data_smithi179_148205_object89230:head# 2024-02-09T17:57:32.197 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.197 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:167cc4b8:::benchmark_data_smithi179_148205_object69534:head# 2024-02-09T17:57:32.197 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.198 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:167e0243:::benchmark_data_smithi179_148205_object48135:head# 2024-02-09T17:57:32.198 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.198 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:167e9e65:::benchmark_data_smithi179_148205_object77907:head# 2024-02-09T17:57:32.198 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.198 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:167eb98e:::benchmark_data_smithi179_148205_object18110:head# 2024-02-09T17:57:32.198 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.198 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:167f0f38:::benchmark_data_smithi179_148205_object75959:head# 2024-02-09T17:57:32.198 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.198 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16801d65:::benchmark_data_smithi179_148205_object18144:head# 2024-02-09T17:57:32.198 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.198 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:168074af:::benchmark_data_smithi179_148205_object80592:head# 2024-02-09T17:57:32.198 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.198 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1680f068:::benchmark_data_smithi179_148205_object85406:head# 2024-02-09T17:57:32.198 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.198 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1680fcdb:::benchmark_data_smithi179_148205_object70455:head# 2024-02-09T17:57:32.199 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.199 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16815af1:::benchmark_data_smithi179_148205_object78083:head# 2024-02-09T17:57:32.199 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.199 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1682a5d8:::benchmark_data_smithi179_148205_object109208:head# 2024-02-09T17:57:32.199 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.199 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1682b24c:::benchmark_data_smithi179_148205_object4454:head# 2024-02-09T17:57:32.199 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.199 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16832a4a:::benchmark_data_smithi179_148205_object56949:head# 2024-02-09T17:57:32.199 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.199 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16834c0b:::benchmark_data_smithi179_148205_object104868:head# 2024-02-09T17:57:32.199 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.199 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1684532b:::benchmark_data_smithi179_148205_object59020:head# 2024-02-09T17:57:32.199 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.199 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1684d95d:::benchmark_data_smithi179_148205_object107299:head# 2024-02-09T17:57:32.200 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.200 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:168560d5:::benchmark_data_smithi179_148205_object73777:head# 2024-02-09T17:57:32.200 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.200 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16856bca:::benchmark_data_smithi179_148205_object109268:head# 2024-02-09T17:57:32.200 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.200 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1685ef92:::benchmark_data_smithi179_148205_object55038:head# 2024-02-09T17:57:32.200 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.200 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:168608ce:::benchmark_data_smithi179_148205_object11009:head# 2024-02-09T17:57:32.200 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.200 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1686ed2d:::benchmark_data_smithi179_148205_object36468:head# 2024-02-09T17:57:32.200 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.200 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1686fc55:::benchmark_data_smithi179_148205_object92688:head# 2024-02-09T17:57:32.200 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.200 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16873ba2:::benchmark_data_smithi179_148205_object45956:head# 2024-02-09T17:57:32.200 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.201 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16888e3c:::benchmark_data_smithi179_148205_object43380:head# 2024-02-09T17:57:32.201 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.201 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16892914:::benchmark_data_smithi179_148205_object60065:head# 2024-02-09T17:57:32.201 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.201 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16895ce5:::benchmark_data_smithi179_148205_object16045:head# 2024-02-09T17:57:32.201 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.201 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:168ad50d:::benchmark_data_smithi179_148205_object93589:head# 2024-02-09T17:57:32.201 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.207 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:168b635d:::benchmark_data_smithi179_148205_object64451:head# 2024-02-09T17:57:32.207 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.207 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:168b6b71:::benchmark_data_smithi179_148205_object45963:head# 2024-02-09T17:57:32.207 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.207 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:168b86d9:::benchmark_data_smithi179_148205_object16365:head# 2024-02-09T17:57:32.207 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.207 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:168bf227:::benchmark_data_smithi179_148205_object105697:head# 2024-02-09T17:57:32.207 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.207 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:168bf6ae:::benchmark_data_smithi179_148205_object48077:head# 2024-02-09T17:57:32.207 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.207 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:168c5564:::benchmark_data_smithi179_148205_object43570:head# 2024-02-09T17:57:32.207 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.208 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:168cae19:::benchmark_data_smithi179_148205_object86539:head# 2024-02-09T17:57:32.208 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.208 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:168cb445:::benchmark_data_smithi179_148205_object93856:head# 2024-02-09T17:57:32.208 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.208 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:168cf23a:::benchmark_data_smithi179_148205_object5150:head# 2024-02-09T17:57:32.208 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.208 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:168d19cb:::benchmark_data_smithi179_148205_object1354:head# 2024-02-09T17:57:32.208 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.208 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:168df668:::benchmark_data_smithi179_148205_object97978:head# 2024-02-09T17:57:32.208 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.208 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:168dfbf5:::benchmark_data_smithi179_148205_object45927:head# 2024-02-09T17:57:32.208 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.208 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:168e40d3:::benchmark_data_smithi179_148205_object15233:head# 2024-02-09T17:57:32.208 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.209 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:168f217a:::benchmark_data_smithi179_148205_object31690:head# 2024-02-09T17:57:32.209 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.209 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:168f80a4:::benchmark_data_smithi179_148205_object83580:head# 2024-02-09T17:57:32.209 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.209 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:168ffdf6:::benchmark_data_smithi179_148205_object72629:head# 2024-02-09T17:57:32.209 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.209 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16905a5a:::benchmark_data_smithi179_148205_object25775:head# 2024-02-09T17:57:32.209 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.209 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1690869d:::benchmark_data_smithi179_148205_object53907:head# 2024-02-09T17:57:32.209 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.209 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:169114fc:::benchmark_data_smithi179_148205_object85827:head# 2024-02-09T17:57:32.209 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.209 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16932f15:::benchmark_data_smithi179_148205_object92054:head# 2024-02-09T17:57:32.209 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.209 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16934629:::benchmark_data_smithi179_148205_object77104:head# 2024-02-09T17:57:32.210 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.210 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16944622:::benchmark_data_smithi179_148205_object104253:head# 2024-02-09T17:57:32.210 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.210 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1694f5e3:::benchmark_data_smithi179_148205_object26905:head# 2024-02-09T17:57:32.210 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.210 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16951f48:::benchmark_data_smithi179_148205_object72805:head# 2024-02-09T17:57:32.210 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.210 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16976b1d:::benchmark_data_smithi179_148205_object95085:head# 2024-02-09T17:57:32.210 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.210 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1697f5b9:::benchmark_data_smithi179_148205_object76907:head# 2024-02-09T17:57:32.210 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.210 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16987a68:::benchmark_data_smithi179_148205_object11090:head# 2024-02-09T17:57:32.210 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.210 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1698a682:::benchmark_data_smithi179_148205_object48006:head# 2024-02-09T17:57:32.210 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.211 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1698ad28:::benchmark_data_smithi179_148205_object2592:head# 2024-02-09T17:57:32.211 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.211 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1699c90f:::benchmark_data_smithi179_148205_object106092:head# 2024-02-09T17:57:32.211 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.211 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1699e29e:::benchmark_data_smithi179_148205_object85607:head# 2024-02-09T17:57:32.211 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.211 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:169b7a71:::benchmark_data_smithi179_148205_object105446:head# 2024-02-09T17:57:32.211 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.211 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:169bf7ca:::benchmark_data_smithi179_148205_object52985:head# 2024-02-09T17:57:32.211 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.211 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:169cd3d9:::benchmark_data_smithi179_148205_object34952:head# 2024-02-09T17:57:32.211 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.211 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:169e03a6:::benchmark_data_smithi179_148205_object71434:head# 2024-02-09T17:57:32.211 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.211 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:169e8aab:::benchmark_data_smithi179_148205_object76197:head# 2024-02-09T17:57:32.212 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.212 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16a02eff:::benchmark_data_smithi179_148205_object25042:head# 2024-02-09T17:57:32.212 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.212 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16a10711:::benchmark_data_smithi179_148205_object13700:head# 2024-02-09T17:57:32.212 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.218 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16a1aa9a:::benchmark_data_smithi179_148205_object11880:head# 2024-02-09T17:57:32.218 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.218 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16a23f5f:::benchmark_data_smithi179_148205_object37421:head# 2024-02-09T17:57:32.218 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.218 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16a24d41:::benchmark_data_smithi179_148205_object143:head# 2024-02-09T17:57:32.218 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.218 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16a2bcad:::benchmark_data_smithi179_148205_object41268:head# 2024-02-09T17:57:32.218 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.218 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16a310d3:::benchmark_data_smithi179_148205_object21114:head# 2024-02-09T17:57:32.218 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.219 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16a36491:::benchmark_data_smithi179_148205_object33255:head# 2024-02-09T17:57:32.219 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.219 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16a3b017:::benchmark_data_smithi179_148205_object99615:head# 2024-02-09T17:57:32.219 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.219 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16a3b8b4:::benchmark_data_smithi179_148205_object27453:head# 2024-02-09T17:57:32.219 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.219 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16a3e85c:::benchmark_data_smithi179_148205_object106396:head# 2024-02-09T17:57:32.219 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.219 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16a4a5b9:::benchmark_data_smithi179_148205_object31765:head# 2024-02-09T17:57:32.219 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.219 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16a5a9ab:::benchmark_data_smithi179_148205_object81490:head# 2024-02-09T17:57:32.219 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.219 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16a658f8:::benchmark_data_smithi179_148205_object80634:head# 2024-02-09T17:57:32.219 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.219 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16a6e1e9:::benchmark_data_smithi179_148205_object56703:head# 2024-02-09T17:57:32.220 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.220 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16a7c72a:::benchmark_data_smithi179_148205_object29719:head# 2024-02-09T17:57:32.220 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.220 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16a8128d:::benchmark_data_smithi179_148205_object31495:head# 2024-02-09T17:57:32.220 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.220 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16a82ad7:::benchmark_data_smithi179_148205_object97619:head# 2024-02-09T17:57:32.220 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.220 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16a8eaa4:::benchmark_data_smithi179_148205_object34785:head# 2024-02-09T17:57:32.220 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.220 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16a9cc37:::benchmark_data_smithi179_148205_object23622:head# 2024-02-09T17:57:32.220 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.220 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16aa6e06:::benchmark_data_smithi179_148205_object96996:head# 2024-02-09T17:57:32.220 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.220 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16aa7c46:::benchmark_data_smithi179_148205_object87160:head# 2024-02-09T17:57:32.221 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.221 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16ab0dc5:::benchmark_data_smithi179_148205_object85600:head# 2024-02-09T17:57:32.221 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.221 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16ab26cc:::benchmark_data_smithi179_148205_object107392:head# 2024-02-09T17:57:32.221 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.221 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16abe2e6:::benchmark_data_smithi179_148205_object91261:head# 2024-02-09T17:57:32.221 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.221 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16ac24c3:::benchmark_data_smithi179_148205_object107933:head# 2024-02-09T17:57:32.221 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.221 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16ac78cb:::benchmark_data_smithi179_148205_object77073:head# 2024-02-09T17:57:32.221 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.221 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16ac9512:::benchmark_data_smithi179_148205_object19596:head# 2024-02-09T17:57:32.221 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.221 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16adbf49:::benchmark_data_smithi179_148205_object77454:head# 2024-02-09T17:57:32.221 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.221 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16af54f5:::benchmark_data_smithi179_148205_object24048:head# 2024-02-09T17:57:32.222 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.222 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16b07c0e:::benchmark_data_smithi179_148205_object95942:head# 2024-02-09T17:57:32.222 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.222 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16b084dc:::benchmark_data_smithi179_148205_object49363:head# 2024-02-09T17:57:32.222 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.222 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16b14727:::benchmark_data_smithi179_148205_object26159:head# 2024-02-09T17:57:32.222 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.222 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16b233a0:::benchmark_data_smithi179_148205_object61034:head# 2024-02-09T17:57:32.222 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.222 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16b2f9a5:::benchmark_data_smithi179_148205_object66816:head# 2024-02-09T17:57:32.222 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.222 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16b31afb:::benchmark_data_smithi179_148205_object58160:head# 2024-02-09T17:57:32.222 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.222 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16b3b38c:::benchmark_data_smithi179_148205_object2585:head# 2024-02-09T17:57:32.222 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.223 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16b3e447:::benchmark_data_smithi179_148205_object58165:head# 2024-02-09T17:57:32.223 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.223 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16b3ea4f:::benchmark_data_smithi179_148205_object30570:head# 2024-02-09T17:57:32.223 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.223 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16b3fbc4:::benchmark_data_smithi179_148205_object54462:head# 2024-02-09T17:57:32.223 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.223 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16b5735a:::benchmark_data_smithi179_148205_object95939:head# 2024-02-09T17:57:32.223 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.223 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16b59c30:::benchmark_data_smithi179_148205_object34425:head# 2024-02-09T17:57:32.223 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.223 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16b5a19b:::benchmark_data_smithi179_148205_object92503:head# 2024-02-09T17:57:32.223 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.223 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16b61108:::benchmark_data_smithi179_148205_object63036:head# 2024-02-09T17:57:32.223 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.223 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16b666a4:::benchmark_data_smithi179_148205_object79114:head# 2024-02-09T17:57:32.224 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.224 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16b721d2:::benchmark_data_smithi179_148205_object12100:head# 2024-02-09T17:57:32.224 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.227 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16b74c9f:::benchmark_data_smithi179_148205_object80641:head# 2024-02-09T17:57:32.227 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.227 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16b7f31e:::benchmark_data_smithi179_148205_object11547:head# 2024-02-09T17:57:32.227 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.227 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16b9b6cd:::benchmark_data_smithi179_148205_object81121:head# 2024-02-09T17:57:32.227 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.227 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16ba8602:::benchmark_data_smithi179_148205_object8878:head# 2024-02-09T17:57:32.227 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.228 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16bb7266:::benchmark_data_smithi179_148205_object62209:head# 2024-02-09T17:57:32.228 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.228 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16bbc163:::benchmark_data_smithi179_148205_object46480:head# 2024-02-09T17:57:32.228 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.228 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16bc2fdc:::benchmark_data_smithi179_148205_object91505:head# 2024-02-09T17:57:32.228 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.228 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16bc6472:::benchmark_data_smithi179_148205_object57106:head# 2024-02-09T17:57:32.228 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.228 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16bd0323:::benchmark_data_smithi179_148205_object94189:head# 2024-02-09T17:57:32.228 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.228 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16bd1469:::benchmark_data_smithi179_148205_object22005:head# 2024-02-09T17:57:32.228 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.228 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16bd6ccf:::benchmark_data_smithi179_148205_object88530:head# 2024-02-09T17:57:32.228 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.229 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16c07672:::benchmark_data_smithi179_148205_object81070:head# 2024-02-09T17:57:32.229 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.229 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16c0f93f:::benchmark_data_smithi179_148205_object49347:head# 2024-02-09T17:57:32.229 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.229 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16c13efd:::benchmark_data_smithi179_148205_object85952:head# 2024-02-09T17:57:32.229 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.229 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16c142f6:::benchmark_data_smithi179_148205_object5491:head# 2024-02-09T17:57:32.229 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.229 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16c17d77:::benchmark_data_smithi179_148205_object17123:head# 2024-02-09T17:57:32.229 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.229 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16c2829c:::benchmark_data_smithi179_148205_object28352:head# 2024-02-09T17:57:32.229 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.229 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16c2b3a3:::benchmark_data_smithi179_148205_object73402:head# 2024-02-09T17:57:32.229 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.229 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16c2d189:::benchmark_data_smithi179_148205_object35994:head# 2024-02-09T17:57:32.230 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.230 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16c373e8:::benchmark_data_smithi179_148205_object109581:head# 2024-02-09T17:57:32.230 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.230 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16c39613:::benchmark_data_smithi179_148205_object38652:head# 2024-02-09T17:57:32.230 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.230 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16c5ed6b:::benchmark_data_smithi179_148205_object73633:head# 2024-02-09T17:57:32.230 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.230 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16c5fb7c:::benchmark_data_smithi179_148205_object13198:head# 2024-02-09T17:57:32.230 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.230 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16c676b7:::benchmark_data_smithi179_148205_object84122:head# 2024-02-09T17:57:32.230 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.232 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16c7a930:::benchmark_data_smithi179_148205_object66866:head# 2024-02-09T17:57:32.232 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.232 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16c7c98e:::benchmark_data_smithi179_148205_object36395:head# 2024-02-09T17:57:32.232 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.233 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16c7d811:::benchmark_data_smithi179_148205_object1716:head# 2024-02-09T17:57:32.233 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.233 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16c7f54c:::benchmark_data_smithi179_148205_object14550:head# 2024-02-09T17:57:32.233 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.233 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16c80c6c:::benchmark_data_smithi179_148205_object108956:head# 2024-02-09T17:57:32.233 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.233 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16c82bbe:::benchmark_data_smithi179_148205_object95776:head# 2024-02-09T17:57:32.233 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.233 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16c83130:::benchmark_data_smithi179_148205_object75706:head# 2024-02-09T17:57:32.233 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.233 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16c87cf8:::benchmark_data_smithi179_148205_object108933:head# 2024-02-09T17:57:32.233 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.233 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16c91b8e:::benchmark_data_smithi179_148205_object93535:head# 2024-02-09T17:57:32.233 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.233 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16c99595:::benchmark_data_smithi179_148205_object58340:head# 2024-02-09T17:57:32.234 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.234 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16ca2fff:::benchmark_data_smithi179_148205_object81302:head# 2024-02-09T17:57:32.234 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.234 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16cc499c:::benchmark_data_smithi179_148205_object51507:head# 2024-02-09T17:57:32.234 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.234 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16ccacf7:::benchmark_data_smithi179_148205_object8184:head# 2024-02-09T17:57:32.234 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.235 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16cd9360:::benchmark_data_smithi179_148205_object96175:head# 2024-02-09T17:57:32.235 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.235 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16ce7c85:::benchmark_data_smithi179_148205_object89940:head# 2024-02-09T17:57:32.235 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.235 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16cf8f0e:::benchmark_data_smithi179_148205_object96605:head# 2024-02-09T17:57:32.235 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.235 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16d024b4:::benchmark_data_smithi179_148205_object47086:head# 2024-02-09T17:57:32.235 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.236 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16d034fc:::benchmark_data_smithi179_148205_object28622:head# 2024-02-09T17:57:32.236 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.236 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16d0506a:::benchmark_data_smithi179_148205_object83414:head# 2024-02-09T17:57:32.236 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.236 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16d1313c:::benchmark_data_smithi179_148205_object43969:head# 2024-02-09T17:57:32.236 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.236 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:57:32.237 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16d14c3e:::benchmark_data_smithi179_148205_object29043:head# 2024-02-09T17:57:32.237 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.237 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16d181d0:::benchmark_data_smithi179_148205_object20773:head# 2024-02-09T17:57:32.237 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:32.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:57:32.241 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:32 smithi099 ceph-mon[39420]: pgmap v2050: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 5.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 808 op/s; 18054/285570 objects degraded (6.322%) 2024-02-09T17:57:32.241 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:32 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1014253901' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:57:32.241 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:32 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:57:32.241 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:32 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 18054/287139 objects degraded (6.288%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:57:32.242 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16d18b7b:::benchmark_data_smithi179_148205_object26616:head# 2024-02-09T17:57:32.242 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.242 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16d281e3:::benchmark_data_smithi179_148205_object18562:head# 2024-02-09T17:57:32.242 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.242 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16d2f1e6:::benchmark_data_smithi179_148205_object38318:head# 2024-02-09T17:57:32.242 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.243 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16d31f23:::benchmark_data_smithi179_148205_object31493:head# 2024-02-09T17:57:32.243 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.243 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16d38a53:::benchmark_data_smithi179_148205_object77106:head# 2024-02-09T17:57:32.243 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.243 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16d3ce7b:::benchmark_data_smithi179_148205_object46207:head# 2024-02-09T17:57:32.243 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.244 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16d40b22:::benchmark_data_smithi179_148205_object32574:head# 2024-02-09T17:57:32.244 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.244 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16d414c6:::benchmark_data_smithi179_148205_object17404:head# 2024-02-09T17:57:32.244 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.244 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16d418f9:::benchmark_data_smithi179_148205_object87257:head# 2024-02-09T17:57:32.244 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.245 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16d4b481:::benchmark_data_smithi179_148205_object10252:head# 2024-02-09T17:57:32.245 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.245 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16d57aa6:::benchmark_data_smithi179_148205_object64157:head# 2024-02-09T17:57:32.245 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.245 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16d64f16:::benchmark_data_smithi179_148205_object58835:head# 2024-02-09T17:57:32.245 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.247 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16d7330b:::benchmark_data_smithi179_148205_object59358:head# 2024-02-09T17:57:32.247 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.247 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16d88221:::benchmark_data_smithi179_148205_object97256:head# 2024-02-09T17:57:32.248 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.248 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16d9a889:::benchmark_data_smithi179_148205_object54096:head# 2024-02-09T17:57:32.248 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.248 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16dab3c8:::benchmark_data_smithi179_148205_object37533:head# 2024-02-09T17:57:32.248 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.248 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16dacad8:::benchmark_data_smithi179_148205_object36046:head# 2024-02-09T17:57:32.248 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.248 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16db5b63:::benchmark_data_smithi179_148205_object45030:head# 2024-02-09T17:57:32.248 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.248 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16dbc5ec:::benchmark_data_smithi179_148205_object2944:head# 2024-02-09T17:57:32.248 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.248 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16dc27a8:::benchmark_data_smithi179_148205_object19102:head# 2024-02-09T17:57:32.248 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.248 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16dc3043:::benchmark_data_smithi179_148205_object62720:head# 2024-02-09T17:57:32.249 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.249 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16dc674d:::benchmark_data_smithi179_148205_object17692:head# 2024-02-09T17:57:32.249 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.249 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16dc8b72:::benchmark_data_smithi179_148205_object106968:head# 2024-02-09T17:57:32.249 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.252 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16dca2b3:::benchmark_data_smithi179_148205_object15545:head# 2024-02-09T17:57:32.252 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.252 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16dcdc8a:::benchmark_data_smithi179_148205_object13121:head# 2024-02-09T17:57:32.252 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.252 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16dd3bd7:::benchmark_data_smithi179_148205_object58546:head# 2024-02-09T17:57:32.252 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.252 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16dd4317:::benchmark_data_smithi179_148205_object82522:head# 2024-02-09T17:57:32.252 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.253 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16ddfee1:::benchmark_data_smithi179_148205_object42804:head# 2024-02-09T17:57:32.253 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.253 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16df0787:::benchmark_data_smithi179_148205_object13199:head# 2024-02-09T17:57:32.253 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.253 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16df5189:::benchmark_data_smithi179_148205_object100860:head# 2024-02-09T17:57:32.253 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.253 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16df775a:::benchmark_data_smithi179_148205_object62702:head# 2024-02-09T17:57:32.253 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.253 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16e0c060:::benchmark_data_smithi179_148205_object38079:head# 2024-02-09T17:57:32.253 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.253 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16e28613:::benchmark_data_smithi179_148205_object30264:head# 2024-02-09T17:57:32.253 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.253 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16e28dec:::benchmark_data_smithi179_148205_object101608:head# 2024-02-09T17:57:32.253 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.254 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16e37339:::benchmark_data_smithi179_148205_object76807:head# 2024-02-09T17:57:32.254 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.254 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16e3b2c7:::benchmark_data_smithi179_148205_object77745:head# 2024-02-09T17:57:32.254 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.254 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16e49ef6:::benchmark_data_smithi179_148205_object94905:head# 2024-02-09T17:57:32.254 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.254 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16e54048:::benchmark_data_smithi179_148205_object805:head# 2024-02-09T17:57:32.254 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.254 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16e63431:::benchmark_data_smithi179_148205_object88811:head# 2024-02-09T17:57:32.254 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.254 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16e6ae4e:::benchmark_data_smithi179_148205_object80682:head# 2024-02-09T17:57:32.254 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.254 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16e6ffee:::benchmark_data_smithi179_148205_object103965:head# 2024-02-09T17:57:32.254 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.256 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16e7008d:::benchmark_data_smithi179_148205_object4626:head# 2024-02-09T17:57:32.256 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.257 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16e71061:::benchmark_data_smithi179_148205_object31164:head# 2024-02-09T17:57:32.257 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.257 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16e74d30:::benchmark_data_smithi179_148205_object49162:head# 2024-02-09T17:57:32.257 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.257 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16e7c74b:::benchmark_data_smithi179_148205_object85844:head# 2024-02-09T17:57:32.257 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.257 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16e7ca8a:::benchmark_data_smithi179_148205_object60438:head# 2024-02-09T17:57:32.257 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.257 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16e8d79d:::benchmark_data_smithi179_148205_object77154:head# 2024-02-09T17:57:32.257 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.257 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16e93fa1:::benchmark_data_smithi179_148205_object37504:head# 2024-02-09T17:57:32.257 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.257 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16e969c8:::benchmark_data_smithi179_148205_object29450:head# 2024-02-09T17:57:32.257 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.257 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16ea0c5f:::benchmark_data_smithi179_148205_object34740:head# 2024-02-09T17:57:32.258 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.258 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16ea484b:::benchmark_data_smithi179_148205_object99782:head# 2024-02-09T17:57:32.258 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.258 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16eb6825:::benchmark_data_smithi179_148205_object1249:head# 2024-02-09T17:57:32.258 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.258 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16ec9a17:::benchmark_data_smithi179_148205_object63485:head# 2024-02-09T17:57:32.258 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.258 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16eda0bb:::benchmark_data_smithi179_148205_object9722:head# 2024-02-09T17:57:32.258 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.258 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16eef504:::benchmark_data_smithi179_148205_object97887:head# 2024-02-09T17:57:32.258 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.258 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16ef4825:::benchmark_data_smithi179_148205_object90546:head# 2024-02-09T17:57:32.258 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.258 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16efe9e8:::benchmark_data_smithi179_148205_object79331:head# 2024-02-09T17:57:32.258 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.262 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16eff14e:::benchmark_data_smithi179_148205_object94040:head# 2024-02-09T17:57:32.262 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.262 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16f09e31:::benchmark_data_smithi179_148205_object56963:head# 2024-02-09T17:57:32.262 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.262 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16f2532c:::benchmark_data_smithi179_148205_object24535:head# 2024-02-09T17:57:32.262 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.262 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16f27de7:::benchmark_data_smithi179_148205_object4699:head# 2024-02-09T17:57:32.262 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.262 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16f2deb1:::benchmark_data_smithi179_148205_object21465:head# 2024-02-09T17:57:32.262 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.263 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16f424f8:::benchmark_data_smithi179_148205_object13173:head# 2024-02-09T17:57:32.263 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.263 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16f5c14f:::benchmark_data_smithi179_148205_object14343:head# 2024-02-09T17:57:32.263 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.263 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16f64141:::benchmark_data_smithi179_148205_object64730:head# 2024-02-09T17:57:32.263 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.263 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16f7373d:::benchmark_data_smithi179_148205_object90407:head# 2024-02-09T17:57:32.263 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.263 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16f7fb7a:::benchmark_data_smithi179_148205_object40901:head# 2024-02-09T17:57:32.263 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.263 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16f7fe53:::benchmark_data_smithi179_148205_object1737:head# 2024-02-09T17:57:32.263 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.263 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16f8b7e2:::benchmark_data_smithi179_148205_object39765:head# 2024-02-09T17:57:32.263 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.263 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16f8cccb:::benchmark_data_smithi179_148205_object149:head# 2024-02-09T17:57:32.264 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.264 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16f91abf:::benchmark_data_smithi179_148205_object27813:head# 2024-02-09T17:57:32.264 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.264 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16f93d2d:::benchmark_data_smithi179_148205_object38922:head# 2024-02-09T17:57:32.264 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.264 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16f966fa:::benchmark_data_smithi179_148205_object80922:head# 2024-02-09T17:57:32.264 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.264 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16fa3bd3:::benchmark_data_smithi179_148205_object105961:head# 2024-02-09T17:57:32.264 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.264 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16fa69fb:::benchmark_data_smithi179_148205_object5249:head# 2024-02-09T17:57:32.264 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.264 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16fb2655:::benchmark_data_smithi179_148205_object31753:head# 2024-02-09T17:57:32.264 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.264 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16fb3d39:::benchmark_data_smithi179_148205_object56129:head# 2024-02-09T17:57:32.264 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.265 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16fb59f7:::benchmark_data_smithi179_148205_object28671:head# 2024-02-09T17:57:32.265 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.265 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16fbfd5c:::benchmark_data_smithi179_148205_object32737:head# 2024-02-09T17:57:32.265 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.265 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16fc0835:::benchmark_data_smithi179_148205_object31633:head# 2024-02-09T17:57:32.265 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.271 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16fc3a93:::benchmark_data_smithi179_148205_object16257:head# 2024-02-09T17:57:32.271 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.271 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16fc8d30:::benchmark_data_smithi179_148205_object5055:head# 2024-02-09T17:57:32.271 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.271 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16fc9a9e:::benchmark_data_smithi179_148205_object108725:head# 2024-02-09T17:57:32.272 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.272 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16fcdc97:::benchmark_data_smithi179_148205_object56155:head# 2024-02-09T17:57:32.272 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.272 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16fd032c:::benchmark_data_smithi179_148205_object88003:head# 2024-02-09T17:57:32.272 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.272 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16ff2849:::benchmark_data_smithi179_148205_object40884:head# 2024-02-09T17:57:32.272 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.272 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16ff38a6:::benchmark_data_smithi179_148205_object104808:head# 2024-02-09T17:57:32.272 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.272 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16ff57ed:::benchmark_data_smithi179_148205_object24382:head# 2024-02-09T17:57:32.272 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.272 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16ff7431:::benchmark_data_smithi179_148205_object57321:head# 2024-02-09T17:57:32.272 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.273 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16ffa09d:::benchmark_data_smithi179_148205_object72662:head# 2024-02-09T17:57:32.273 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.273 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:16ffb74c:::benchmark_data_smithi179_148205_object2667:head# 2024-02-09T17:57:32.273 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.273 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17006a7e:::benchmark_data_smithi179_148205_object49841:head# 2024-02-09T17:57:32.273 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.273 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1700a3dd:::benchmark_data_smithi179_148205_object95511:head# 2024-02-09T17:57:32.273 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.273 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1701818a:::benchmark_data_smithi179_148205_object56144:head# 2024-02-09T17:57:32.273 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.273 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17021fdf:::benchmark_data_smithi179_148205_object69684:head# 2024-02-09T17:57:32.273 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.273 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17027baf:::benchmark_data_smithi179_148205_object76184:head# 2024-02-09T17:57:32.273 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.273 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:170299a2:::benchmark_data_smithi179_148205_object20299:head# 2024-02-09T17:57:32.274 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.274 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17030c64:::benchmark_data_smithi179_148205_object89178:head# 2024-02-09T17:57:32.274 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.274 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17031a9f:::benchmark_data_smithi179_148205_object67088:head# 2024-02-09T17:57:32.274 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.274 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17036475:::benchmark_data_smithi179_148205_object71665:head# 2024-02-09T17:57:32.274 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.274 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1703743f:::benchmark_data_smithi179_148205_object86298:head# 2024-02-09T17:57:32.274 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.274 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:170449ad:::benchmark_data_smithi179_148205_object27648:head# 2024-02-09T17:57:32.274 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.274 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1704c948:::benchmark_data_smithi179_148205_object68360:head# 2024-02-09T17:57:32.274 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.274 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1705c338:::benchmark_data_smithi179_148205_object54314:head# 2024-02-09T17:57:32.274 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.275 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1707ffb6:::benchmark_data_smithi179_148205_object20883:head# 2024-02-09T17:57:32.275 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.275 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17081ca1:::benchmark_data_smithi179_148205_object51349:head# 2024-02-09T17:57:32.275 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.275 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17083e30:::benchmark_data_smithi179_148205_object41848:head# 2024-02-09T17:57:32.275 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.275 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1709431d:::benchmark_data_smithi179_148205_object28280:head# 2024-02-09T17:57:32.275 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.275 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17099e54:::benchmark_data_smithi179_148205_object80164:head# 2024-02-09T17:57:32.275 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.275 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1709c9b1:::benchmark_data_smithi179_148205_object81591:head# 2024-02-09T17:57:32.275 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.275 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:170a2058:::benchmark_data_smithi179_148205_object83830:head# 2024-02-09T17:57:32.275 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.276 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:170b179d:::benchmark_data_smithi179_148205_object81286:head# 2024-02-09T17:57:32.276 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.276 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:170b627a:::benchmark_data_smithi179_148205_object97867:head# 2024-02-09T17:57:32.276 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.276 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:170c1cac:::benchmark_data_smithi179_148205_object104656:head# 2024-02-09T17:57:32.276 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.276 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:170c42bb:::benchmark_data_smithi179_148205_object31347:head# 2024-02-09T17:57:32.276 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.276 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:170d26da:::benchmark_data_smithi179_148205_object87013:head# 2024-02-09T17:57:32.276 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.276 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:170da87f:::benchmark_data_smithi179_148205_object82494:head# 2024-02-09T17:57:32.276 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.276 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:170dbc96:::benchmark_data_smithi179_148205_object34098:head# 2024-02-09T17:57:32.276 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.276 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:170e1b6a:::benchmark_data_smithi179_148205_object87573:head# 2024-02-09T17:57:32.277 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.277 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:170e87bf:::benchmark_data_smithi179_148205_object22224:head# 2024-02-09T17:57:32.277 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.277 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:170e9208:::benchmark_data_smithi179_148205_object72230:head# 2024-02-09T17:57:32.277 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.277 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:170eb9ca:::benchmark_data_smithi179_148205_object6745:head# 2024-02-09T17:57:32.277 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.277 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:170ece86:::benchmark_data_smithi179_148205_object50756:head# 2024-02-09T17:57:32.277 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.289 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:170f707f:::benchmark_data_smithi179_148205_object101690:head# 2024-02-09T17:57:32.289 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.289 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17102862:::benchmark_data_smithi179_148205_object61697:head# 2024-02-09T17:57:32.289 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.289 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1710971e:::benchmark_data_smithi179_148205_object95982:head# 2024-02-09T17:57:32.289 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.289 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1710fd46:::benchmark_data_smithi179_148205_object82763:head# 2024-02-09T17:57:32.289 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.290 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17126274:::benchmark_data_smithi179_148205_object37879:head# 2024-02-09T17:57:32.290 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.290 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17127a56:::benchmark_data_smithi179_148205_object24518:head# 2024-02-09T17:57:32.290 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.290 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1712af45:::benchmark_data_smithi179_148205_object91140:head# 2024-02-09T17:57:32.290 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.290 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1712b73f:::benchmark_data_smithi179_148205_object36832:head# 2024-02-09T17:57:32.290 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.290 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1712e475:::benchmark_data_smithi179_148205_object66669:head# 2024-02-09T17:57:32.290 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.290 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:171303f9:::benchmark_data_smithi179_148205_object53800:head# 2024-02-09T17:57:32.290 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.290 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:171342f8:::benchmark_data_smithi179_148205_object16869:head# 2024-02-09T17:57:32.290 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.291 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17147c6b:::benchmark_data_smithi179_148205_object64122:head# 2024-02-09T17:57:32.291 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.291 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1714809d:::benchmark_data_smithi179_148205_object57025:head# 2024-02-09T17:57:32.291 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.291 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17152a00:::benchmark_data_smithi179_148205_object83945:head# 2024-02-09T17:57:32.291 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.291 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17157dc8:::benchmark_data_smithi179_148205_object24865:head# 2024-02-09T17:57:32.291 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.291 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1715ee71:::benchmark_data_smithi179_148205_object27995:head# 2024-02-09T17:57:32.291 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.291 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17160ba4:::benchmark_data_smithi179_148205_object14719:head# 2024-02-09T17:57:32.291 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.291 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1718b06d:::benchmark_data_smithi179_148205_object70837:head# 2024-02-09T17:57:32.291 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.292 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1718f480:::benchmark_data_smithi179_148205_object103520:head# 2024-02-09T17:57:32.292 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.292 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17190053:::benchmark_data_smithi179_148205_object4260:head# 2024-02-09T17:57:32.292 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.292 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17190540:::benchmark_data_smithi179_148205_object23391:head# 2024-02-09T17:57:32.292 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.292 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1719e6d7:::benchmark_data_smithi179_148205_object68839:head# 2024-02-09T17:57:32.292 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.292 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:171aae29:::benchmark_data_smithi179_148205_object103158:head# 2024-02-09T17:57:32.292 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.292 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:171ab4df:::benchmark_data_smithi179_148205_object83749:head# 2024-02-09T17:57:32.292 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.292 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:171cef14:::benchmark_data_smithi179_148205_object20615:head# 2024-02-09T17:57:32.293 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.293 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:171e0ac1:::benchmark_data_smithi179_148205_object57937:head# 2024-02-09T17:57:32.293 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.293 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:171f33d0:::benchmark_data_smithi179_148205_object58914:head# 2024-02-09T17:57:32.293 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.293 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:171fa230:::benchmark_data_smithi179_148205_object21690:head# 2024-02-09T17:57:32.293 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.293 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:171fdce8:::benchmark_data_smithi179_148205_object82944:head# 2024-02-09T17:57:32.293 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.293 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17200644:::benchmark_data_smithi179_148205_object54242:head# 2024-02-09T17:57:32.293 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.293 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:172012c9:::benchmark_data_smithi179_148205_object20865:head# 2024-02-09T17:57:32.293 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.293 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17202591:::benchmark_data_smithi179_148205_object16579:head# 2024-02-09T17:57:32.294 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.294 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17205657:::benchmark_data_smithi179_148205_object6757:head# 2024-02-09T17:57:32.294 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.294 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17209430:::benchmark_data_smithi179_148205_object2461:head# 2024-02-09T17:57:32.294 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.294 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1721974c:::benchmark_data_smithi179_148205_object46010:head# 2024-02-09T17:57:32.294 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.294 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17219aa9:::benchmark_data_smithi179_148205_object20082:head# 2024-02-09T17:57:32.294 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.294 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1722088e:::benchmark_data_smithi179_148205_object73990:head# 2024-02-09T17:57:32.294 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.294 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17220f46:::benchmark_data_smithi179_148205_object56236:head# 2024-02-09T17:57:32.294 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.294 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17223278:::benchmark_data_smithi179_148205_object41521:head# 2024-02-09T17:57:32.295 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.295 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17229514:::benchmark_data_smithi179_148205_object38606:head# 2024-02-09T17:57:32.295 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.295 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1722d3cd:::benchmark_data_smithi179_148205_object18141:head# 2024-02-09T17:57:32.295 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.295 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17245a14:::benchmark_data_smithi179_148205_object87677:head# 2024-02-09T17:57:32.295 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.295 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1724a35f:::benchmark_data_smithi179_148205_object3325:head# 2024-02-09T17:57:32.295 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.295 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1724bcf7:::benchmark_data_smithi179_148205_object55053:head# 2024-02-09T17:57:32.295 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.295 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17252b95:::benchmark_data_smithi179_148205_object988:head# 2024-02-09T17:57:32.295 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.295 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1725e126:::benchmark_data_smithi179_148205_object51961:head# 2024-02-09T17:57:32.295 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.296 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17263b66:::benchmark_data_smithi179_148205_object66878:head# 2024-02-09T17:57:32.296 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.296 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1726d75a:::benchmark_data_smithi179_148205_object103491:head# 2024-02-09T17:57:32.296 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.296 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1728fdcf:::benchmark_data_smithi179_148205_object487:head# 2024-02-09T17:57:32.296 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.296 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:172916f9:::benchmark_data_smithi179_148205_object94157:head# 2024-02-09T17:57:32.296 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.296 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17295968:::benchmark_data_smithi179_148205_object5391:head# 2024-02-09T17:57:32.296 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.296 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17295a9c:::benchmark_data_smithi179_148205_object51366:head# 2024-02-09T17:57:32.296 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.296 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1729d6af:::benchmark_data_smithi179_148205_object92558:head# 2024-02-09T17:57:32.296 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.297 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:172a5512:::benchmark_data_smithi179_148205_object3567:head# 2024-02-09T17:57:32.297 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.297 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:172ab45c:::benchmark_data_smithi179_148205_object94552:head# 2024-02-09T17:57:32.297 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.297 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:172b1544:::benchmark_data_smithi179_148205_object93161:head# 2024-02-09T17:57:32.297 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.297 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:172b96ca:::benchmark_data_smithi179_148205_object98955:head# 2024-02-09T17:57:32.297 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.297 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:172c2b7a:::benchmark_data_smithi179_148205_object8779:head# 2024-02-09T17:57:32.297 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.297 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:172ceaca:::benchmark_data_smithi179_148205_object55851:head# 2024-02-09T17:57:32.297 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.297 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:172e70af:::benchmark_data_smithi179_148205_object109200:head# 2024-02-09T17:57:32.297 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.297 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17304635:::benchmark_data_smithi179_148205_object35323:head# 2024-02-09T17:57:32.298 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.298 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:173214d2:::benchmark_data_smithi179_148205_object29037:head# 2024-02-09T17:57:32.298 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.298 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:173241a7:::benchmark_data_smithi179_148205_object74931:head# 2024-02-09T17:57:32.298 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.298 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17327ade:::benchmark_data_smithi179_148205_object19222:head# 2024-02-09T17:57:32.298 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.298 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:173297e0:::benchmark_data_smithi179_148205_object4877:head# 2024-02-09T17:57:32.298 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.298 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1732aea9:::benchmark_data_smithi179_148205_object9804:head# 2024-02-09T17:57:32.298 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.298 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1732cbae:::benchmark_data_smithi179_148205_object50766:head# 2024-02-09T17:57:32.298 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.298 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1733bb06:::benchmark_data_smithi179_148205_object15686:head# 2024-02-09T17:57:32.299 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.299 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17341877:::benchmark_data_smithi179_148205_object61117:head# 2024-02-09T17:57:32.299 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.299 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17342b3d:::benchmark_data_smithi179_148205_object61493:head# 2024-02-09T17:57:32.299 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.299 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17346be3:::benchmark_data_smithi179_148205_object64630:head# 2024-02-09T17:57:32.299 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.299 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1734efa0:::benchmark_data_smithi179_148205_object98531:head# 2024-02-09T17:57:32.299 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.299 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17352e66:::benchmark_data_smithi179_148205_object23253:head# 2024-02-09T17:57:32.299 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.299 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17367b95:::benchmark_data_smithi179_148205_object64400:head# 2024-02-09T17:57:32.299 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.299 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17375331:::benchmark_data_smithi179_148205_object56027:head# 2024-02-09T17:57:32.299 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.300 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1737827e:::benchmark_data_smithi179_148205_object104985:head# 2024-02-09T17:57:32.300 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.305 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1737c56b:::benchmark_data_smithi179_148205_object18112:head# 2024-02-09T17:57:32.305 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.305 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1738bcc2:::benchmark_data_smithi179_148205_object40640:head# 2024-02-09T17:57:32.305 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.305 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1739189b:::benchmark_data_smithi179_148205_object51021:head# 2024-02-09T17:57:32.305 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.305 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:173a1042:::benchmark_data_smithi179_148205_object17241:head# 2024-02-09T17:57:32.305 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.306 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:173a8f04:::benchmark_data_smithi179_148205_object13921:head# 2024-02-09T17:57:32.306 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.306 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:173adc08:::benchmark_data_smithi179_148205_object78800:head# 2024-02-09T17:57:32.306 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.306 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:173b4ea9:::benchmark_data_smithi179_148205_object90668:head# 2024-02-09T17:57:32.306 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.306 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:173b947d:::benchmark_data_smithi179_148205_object92415:head# 2024-02-09T17:57:32.306 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.306 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:173c5fd6:::benchmark_data_smithi179_148205_object7290:head# 2024-02-09T17:57:32.306 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.306 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:173cb74d:::benchmark_data_smithi179_148205_object74404:head# 2024-02-09T17:57:32.306 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.306 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:173dc4c6:::benchmark_data_smithi179_148205_object7546:head# 2024-02-09T17:57:32.306 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.306 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:173ea3b3:::benchmark_data_smithi179_148205_object62980:head# 2024-02-09T17:57:32.307 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.307 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:173ede16:::benchmark_data_smithi179_148205_object101737:head# 2024-02-09T17:57:32.307 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.307 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17414a5a:::benchmark_data_smithi179_148205_object13764:head# 2024-02-09T17:57:32.307 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.307 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1742eb69:::benchmark_data_smithi179_148205_object17835:head# 2024-02-09T17:57:32.307 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.307 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17432d66:::benchmark_data_smithi179_148205_object94744:head# 2024-02-09T17:57:32.307 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.307 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17441315:::benchmark_data_smithi179_148205_object70269:head# 2024-02-09T17:57:32.307 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.307 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1744227d:::benchmark_data_smithi179_148205_object64001:head# 2024-02-09T17:57:32.307 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.307 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17449ddd:::benchmark_data_smithi179_148205_object76243:head# 2024-02-09T17:57:32.308 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.308 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1745afc4:::benchmark_data_smithi179_148205_object78197:head# 2024-02-09T17:57:32.308 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.308 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:174808d0:::benchmark_data_smithi179_148205_object14301:head# 2024-02-09T17:57:32.308 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.308 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:174816c0:::benchmark_data_smithi179_148205_object18040:head# 2024-02-09T17:57:32.308 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.308 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:174885a0:::benchmark_data_smithi179_148205_object69887:head# 2024-02-09T17:57:32.308 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.308 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1748eadc:::benchmark_data_smithi179_148205_object31275:head# 2024-02-09T17:57:32.308 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.308 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17499f6e:::benchmark_data_smithi179_148205_object17786:head# 2024-02-09T17:57:32.308 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.308 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1749baf0:::benchmark_data_smithi179_148205_object16971:head# 2024-02-09T17:57:32.308 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.309 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1749be57:::benchmark_data_smithi179_148205_object76387:head# 2024-02-09T17:57:32.309 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.309 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1749db62:::benchmark_data_smithi179_148205_object80962:head# 2024-02-09T17:57:32.309 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.309 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:174a937e:::benchmark_data_smithi179_148205_object83686:head# 2024-02-09T17:57:32.309 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.309 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:174b4e8d:::benchmark_data_smithi179_148205_object103703:head# 2024-02-09T17:57:32.309 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.309 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:174bba5a:::benchmark_data_smithi179_148205_object57678:head# 2024-02-09T17:57:32.309 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.309 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:174bce94:::benchmark_data_smithi179_148205_object90490:head# 2024-02-09T17:57:32.309 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.309 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:174c5a1c:::benchmark_data_smithi179_148205_object95178:head# 2024-02-09T17:57:32.309 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.310 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:174c8858:::benchmark_data_smithi179_148205_object61041:head# 2024-02-09T17:57:32.310 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.310 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:174ce64e:::benchmark_data_smithi179_148205_object28597:head# 2024-02-09T17:57:32.310 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.310 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:174d235f:::benchmark_data_smithi179_148205_object16209:head# 2024-02-09T17:57:32.310 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.310 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:174d7bb0:::benchmark_data_smithi179_148205_object9594:head# 2024-02-09T17:57:32.310 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.310 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:174d87c4:::benchmark_data_smithi179_148205_object41987:head# 2024-02-09T17:57:32.310 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.310 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:174da99e:::benchmark_data_smithi179_148205_object72703:head# 2024-02-09T17:57:32.310 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.310 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:174db10d:::benchmark_data_smithi179_148205_object44056:head# 2024-02-09T17:57:32.310 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.310 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:174e107c:::benchmark_data_smithi179_148205_object107525:head# 2024-02-09T17:57:32.311 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.311 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:174e12f7:::benchmark_data_smithi179_148205_object24396:head# 2024-02-09T17:57:32.311 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.311 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:174fb8f0:::benchmark_data_smithi179_148205_object9695:head# 2024-02-09T17:57:32.311 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.311 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:174fc418:::benchmark_data_smithi179_148205_object96314:head# 2024-02-09T17:57:32.311 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.311 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1751507e:::benchmark_data_smithi179_148205_object38390:head# 2024-02-09T17:57:32.311 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.311 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1751f532:::benchmark_data_smithi179_148205_object106156:head# 2024-02-09T17:57:32.311 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.311 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17521a73:::benchmark_data_smithi179_148205_object37393:head# 2024-02-09T17:57:32.311 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.311 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:175223af:::benchmark_data_smithi179_148205_object28005:head# 2024-02-09T17:57:32.311 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.312 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:175245b1:::benchmark_data_smithi179_148205_object63592:head# 2024-02-09T17:57:32.312 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.312 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1752ca86:::benchmark_data_smithi179_148205_object21240:head# 2024-02-09T17:57:32.312 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.312 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1754698e:::benchmark_data_smithi179_148205_object64620:head# 2024-02-09T17:57:32.312 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.312 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17547af1:::benchmark_data_smithi179_148205_object109706:head# 2024-02-09T17:57:32.312 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.312 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17552f88:::benchmark_data_smithi179_148205_object68722:head# 2024-02-09T17:57:32.312 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.321 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17553543:::benchmark_data_smithi179_148205_object1606:head# 2024-02-09T17:57:32.321 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.321 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:175616cd:::benchmark_data_smithi179_148205_object4709:head# 2024-02-09T17:57:32.321 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.321 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17585e61:::benchmark_data_smithi179_148205_object107453:head# 2024-02-09T17:57:32.321 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.321 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:175a633a:::benchmark_data_smithi179_148205_object41160:head# 2024-02-09T17:57:32.321 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.321 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:175a6a30:::benchmark_data_smithi179_148205_object92102:head# 2024-02-09T17:57:32.321 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.321 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:175a9936:::benchmark_data_smithi179_148205_object53546:head# 2024-02-09T17:57:32.321 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.322 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:175ae97c:::benchmark_data_smithi179_148205_object92308:head# 2024-02-09T17:57:32.322 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.322 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:175b05d8:::benchmark_data_smithi179_148205_object31732:head# 2024-02-09T17:57:32.322 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.322 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:175b143f:::benchmark_data_smithi179_148205_object106702:head# 2024-02-09T17:57:32.322 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.322 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:175b796a:::benchmark_data_smithi179_148205_object105752:head# 2024-02-09T17:57:32.322 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.322 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:175ba2a4:::benchmark_data_smithi179_148205_object46333:head# 2024-02-09T17:57:32.322 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.322 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:175cb2b2:::benchmark_data_smithi179_148205_object19209:head# 2024-02-09T17:57:32.322 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.322 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:175d6d22:::benchmark_data_smithi179_148205_object102277:head# 2024-02-09T17:57:32.322 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.323 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:175d7c18:::benchmark_data_smithi179_148205_object15927:head# 2024-02-09T17:57:32.323 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.323 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:175d8fdd:::benchmark_data_smithi179_148205_object8794:head# 2024-02-09T17:57:32.323 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.323 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:175da43f:::benchmark_data_smithi179_148205_object56714:head# 2024-02-09T17:57:32.323 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.323 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:175e1db1:::benchmark_data_smithi179_148205_object37336:head# 2024-02-09T17:57:32.323 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.323 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:175e922a:::benchmark_data_smithi179_148205_object35629:head# 2024-02-09T17:57:32.323 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.323 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:176003b6:::benchmark_data_smithi179_148205_object12758:head# 2024-02-09T17:57:32.323 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.323 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17600fd1:::benchmark_data_smithi179_148205_object31585:head# 2024-02-09T17:57:32.323 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.323 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1760a8ee:::benchmark_data_smithi179_148205_object74537:head# 2024-02-09T17:57:32.324 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.324 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1760ff00:::benchmark_data_smithi179_148205_object6758:head# 2024-02-09T17:57:32.324 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.324 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17612bc3:::benchmark_data_smithi179_148205_object38759:head# 2024-02-09T17:57:32.324 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.324 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17616d5a:::benchmark_data_smithi179_148205_object94658:head# 2024-02-09T17:57:32.324 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.324 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17624859:::benchmark_data_smithi179_148205_object77766:head# 2024-02-09T17:57:32.324 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.324 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17633b5a:::benchmark_data_smithi179_148205_object90020:head# 2024-02-09T17:57:32.324 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.324 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17644a45:::benchmark_data_smithi179_148205_object17849:head# 2024-02-09T17:57:32.324 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.324 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:176623e4:::benchmark_data_smithi179_148205_object55490:head# 2024-02-09T17:57:32.325 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.325 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17666eec:::benchmark_data_smithi179_148205_object24676:head# 2024-02-09T17:57:32.325 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.325 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1766f325:::benchmark_data_smithi179_148205_object66362:head# 2024-02-09T17:57:32.325 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.325 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17675e48:::benchmark_data_smithi179_148205_object8089:head# 2024-02-09T17:57:32.325 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.325 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1767b330:::benchmark_data_smithi179_148205_object51741:head# 2024-02-09T17:57:32.325 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.325 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17680387:::benchmark_data_smithi179_148205_object93645:head# 2024-02-09T17:57:32.325 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.325 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17689847:::benchmark_data_smithi179_148205_object51592:head# 2024-02-09T17:57:32.325 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.325 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:176955cb:::benchmark_data_smithi179_148205_object6715:head# 2024-02-09T17:57:32.325 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.326 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1769df6a:::benchmark_data_smithi179_148205_object39828:head# 2024-02-09T17:57:32.326 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.326 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:176ac753:::benchmark_data_smithi179_148205_object100171:head# 2024-02-09T17:57:32.326 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.326 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:176b24d2:::benchmark_data_smithi179_148205_object75813:head# 2024-02-09T17:57:32.326 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.326 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:176b39b0:::benchmark_data_smithi179_148205_object1288:head# 2024-02-09T17:57:32.326 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.326 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:176b7505:::benchmark_data_smithi179_148205_object34912:head# 2024-02-09T17:57:32.326 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.326 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:176b7907:::benchmark_data_smithi179_148205_object60750:head# 2024-02-09T17:57:32.326 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.326 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:176c5ce7:::benchmark_data_smithi179_148205_object44150:head# 2024-02-09T17:57:32.327 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.327 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:176c88d6:::benchmark_data_smithi179_148205_object42398:head# 2024-02-09T17:57:32.327 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.327 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:176cf6d6:::benchmark_data_smithi179_148205_object59248:head# 2024-02-09T17:57:32.327 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.327 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:176e7656:::benchmark_data_smithi179_148205_object36032:head# 2024-02-09T17:57:32.327 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.327 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:176ec385:::benchmark_data_smithi179_148205_object103106:head# 2024-02-09T17:57:32.327 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.327 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:176edb6f:::benchmark_data_smithi179_148205_object26480:head# 2024-02-09T17:57:32.327 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.327 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:176f342a:::benchmark_data_smithi179_148205_object51515:head# 2024-02-09T17:57:32.327 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.327 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:176f4543:::benchmark_data_smithi179_148205_object38292:head# 2024-02-09T17:57:32.327 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.327 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1770615a:::benchmark_data_smithi179_148205_object8570:head# 2024-02-09T17:57:32.327 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.328 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:177113bf:::benchmark_data_smithi179_148205_object26540:head# 2024-02-09T17:57:32.328 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.328 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17729f4f:::benchmark_data_smithi179_148205_object87977:head# 2024-02-09T17:57:32.328 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.332 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17736106:::benchmark_data_smithi179_148205_object74103:head# 2024-02-09T17:57:32.332 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.332 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17746bdc:::benchmark_data_smithi179_148205_object19735:head# 2024-02-09T17:57:32.332 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.332 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1774ba66:::benchmark_data_smithi179_148205_object2473:head# 2024-02-09T17:57:32.332 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.332 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1774c5d9:::benchmark_data_smithi179_148205_object87654:head# 2024-02-09T17:57:32.332 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.332 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:177515ee:::benchmark_data_smithi179_148205_object100263:head# 2024-02-09T17:57:32.332 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.332 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17751d30:::benchmark_data_smithi179_148205_object33716:head# 2024-02-09T17:57:32.333 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.333 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:177527f3:::benchmark_data_smithi179_148205_object26386:head# 2024-02-09T17:57:32.333 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.333 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17756fbe:::benchmark_data_smithi179_148205_object1033:head# 2024-02-09T17:57:32.333 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.333 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1775845d:::benchmark_data_smithi179_148205_object109242:head# 2024-02-09T17:57:32.333 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.333 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17766244:::benchmark_data_smithi179_148205_object33643:head# 2024-02-09T17:57:32.333 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.333 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17781b6a:::benchmark_data_smithi179_148205_object104425:head# 2024-02-09T17:57:32.333 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.333 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1778cbc3:::benchmark_data_smithi179_148205_object49399:head# 2024-02-09T17:57:32.333 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.334 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1778fdbe:::benchmark_data_smithi179_148205_object94636:head# 2024-02-09T17:57:32.334 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.334 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17795b73:::benchmark_data_smithi179_148205_object55880:head# 2024-02-09T17:57:32.334 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.334 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:177a68e9:::benchmark_data_smithi179_148205_object22842:head# 2024-02-09T17:57:32.334 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.334 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:177ab1d0:::benchmark_data_smithi179_148205_object99846:head# 2024-02-09T17:57:32.334 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.334 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:177b2f29:::benchmark_data_smithi179_148205_object80343:head# 2024-02-09T17:57:32.334 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.334 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:177b6853:::benchmark_data_smithi179_148205_object16117:head# 2024-02-09T17:57:32.334 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.334 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:177b7f54:::benchmark_data_smithi179_148205_object69466:head# 2024-02-09T17:57:32.334 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.335 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:177c3d24:::benchmark_data_smithi179_148205_object92436:head# 2024-02-09T17:57:32.335 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.335 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:177d20ba:::benchmark_data_smithi179_148205_object42474:head# 2024-02-09T17:57:32.335 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.335 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:177d9ec6:::benchmark_data_smithi179_148205_object87206:head# 2024-02-09T17:57:32.335 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.335 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:178039a3:::benchmark_data_smithi179_148205_object57650:head# 2024-02-09T17:57:32.335 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.335 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1780af72:::benchmark_data_smithi179_148205_object77751:head# 2024-02-09T17:57:32.335 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.335 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17813074:::benchmark_data_smithi179_148205_object84613:head# 2024-02-09T17:57:32.335 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.335 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17819bc4:::benchmark_data_smithi179_148205_object106988:head# 2024-02-09T17:57:32.335 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.335 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1781b879:::benchmark_data_smithi179_148205_object68561:head# 2024-02-09T17:57:32.336 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.336 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1781d88a:::benchmark_data_smithi179_148205_object106786:head# 2024-02-09T17:57:32.336 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.336 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17850d79:::benchmark_data_smithi179_148205_object87100:head# 2024-02-09T17:57:32.336 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.336 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:178525e7:::benchmark_data_smithi179_148205_object32012:head# 2024-02-09T17:57:32.336 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.336 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:178551f3:::benchmark_data_smithi179_148205_object105682:head# 2024-02-09T17:57:32.336 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.336 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17859e26:::benchmark_data_smithi179_148205_object71658:head# 2024-02-09T17:57:32.336 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.336 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1785d81d:::benchmark_data_smithi179_148205_object80081:head# 2024-02-09T17:57:32.336 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.336 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1786842a:::benchmark_data_smithi179_148205_object101688:head# 2024-02-09T17:57:32.336 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.337 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1786a6b2:::benchmark_data_smithi179_148205_object33422:head# 2024-02-09T17:57:32.337 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.337 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1787fea4:::benchmark_data_smithi179_148205_object45273:head# 2024-02-09T17:57:32.337 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.337 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17881c6f:::benchmark_data_smithi179_148205_object47864:head# 2024-02-09T17:57:32.337 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.337 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1788708a:::benchmark_data_smithi179_148205_object47257:head# 2024-02-09T17:57:32.337 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.339 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1789797d:::benchmark_data_smithi179_148205_object61154:head# 2024-02-09T17:57:32.339 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.339 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1789b591:::benchmark_data_smithi179_148205_object84729:head# 2024-02-09T17:57:32.339 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.339 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1789ec5b:::benchmark_data_smithi179_148205_object52042:head# 2024-02-09T17:57:32.339 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.339 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:178a4141:::benchmark_data_smithi179_148205_object86476:head# 2024-02-09T17:57:32.339 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.339 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:178a4fff:::benchmark_data_smithi179_148205_object24992:head# 2024-02-09T17:57:32.339 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.339 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:178a9a0d:::benchmark_data_smithi179_148205_object55770:head# 2024-02-09T17:57:32.339 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.339 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:178ab210:::benchmark_data_smithi179_148205_object48664:head# 2024-02-09T17:57:32.339 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.340 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:178b9b94:::benchmark_data_smithi179_148205_object28953:head# 2024-02-09T17:57:32.340 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.340 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:178cdab1:::benchmark_data_smithi179_148205_object14873:head# 2024-02-09T17:57:32.340 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.340 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:178ce018:::benchmark_data_smithi179_148205_object38653:head# 2024-02-09T17:57:32.340 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.340 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:178cf493:::benchmark_data_smithi179_148205_object19028:head# 2024-02-09T17:57:32.340 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.340 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:178d8c57:::benchmark_data_smithi179_148205_object55599:head# 2024-02-09T17:57:32.340 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.340 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:178e3be3:::benchmark_data_smithi179_148205_object106566:head# 2024-02-09T17:57:32.340 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.340 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:178e9707:::benchmark_data_smithi179_148205_object32939:head# 2024-02-09T17:57:32.340 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.340 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:178f2476:::benchmark_data_smithi179_148205_object19101:head# 2024-02-09T17:57:32.341 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.341 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:178f57d1:::benchmark_data_smithi179_148205_object351:head# 2024-02-09T17:57:32.341 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.341 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1790537f:::benchmark_data_smithi179_148205_object109416:head# 2024-02-09T17:57:32.341 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.341 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17910f0d:::benchmark_data_smithi179_148205_object42458:head# 2024-02-09T17:57:32.341 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.341 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17911b71:::benchmark_data_smithi179_148205_object46980:head# 2024-02-09T17:57:32.341 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.341 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17911e10:::benchmark_data_smithi179_148205_object92479:head# 2024-02-09T17:57:32.341 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.341 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1791b010:::benchmark_data_smithi179_148205_object29105:head# 2024-02-09T17:57:32.341 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.341 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:57:32.345 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1791d66c:::benchmark_data_smithi179_148205_object3048:head# 2024-02-09T17:57:32.345 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.345 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1791fd1f:::benchmark_data_smithi179_148205_object60419:head# 2024-02-09T17:57:32.345 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.345 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1792eeff:::benchmark_data_smithi179_148205_object25111:head# 2024-02-09T17:57:32.345 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.346 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1794de58:::benchmark_data_smithi179_148205_object35643:head# 2024-02-09T17:57:32.346 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.346 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17956e2b:::benchmark_data_smithi179_148205_object92775:head# 2024-02-09T17:57:32.346 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.346 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17994e5d:::benchmark_data_smithi179_148205_object15765:head# 2024-02-09T17:57:32.346 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.346 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1799d140:::benchmark_data_smithi179_148205_object74377:head# 2024-02-09T17:57:32.346 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.347 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:179af6e0:::benchmark_data_smithi179_148205_object104983:head# 2024-02-09T17:57:32.347 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.348 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:179b7fc1:::benchmark_data_smithi179_148205_object67248:head# 2024-02-09T17:57:32.348 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.348 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:179c7398:::benchmark_data_smithi179_148205_object79681:head# 2024-02-09T17:57:32.348 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.348 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:179dc373:::benchmark_data_smithi179_148205_object67945:head# 2024-02-09T17:57:32.348 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.348 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:179df9cc:::benchmark_data_smithi179_148205_object2925:head# 2024-02-09T17:57:32.348 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.348 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:179e202b:::benchmark_data_smithi179_148205_object49749:head# 2024-02-09T17:57:32.348 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.348 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:179e8f47:::benchmark_data_smithi179_148205_object96919:head# 2024-02-09T17:57:32.348 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.348 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:179ee127:::benchmark_data_smithi179_148205_object65840:head# 2024-02-09T17:57:32.348 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.349 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17a007e3:::benchmark_data_smithi179_148205_object22616:head# 2024-02-09T17:57:32.349 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.349 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17a08b73:::benchmark_data_smithi179_148205_object73104:head# 2024-02-09T17:57:32.349 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.349 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17a1e284:::benchmark_data_smithi179_148205_object104276:head# 2024-02-09T17:57:32.349 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.351 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17a3345d:::benchmark_data_smithi179_148205_object17428:head# 2024-02-09T17:57:32.351 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.351 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17a3de41:::benchmark_data_smithi179_148205_object21249:head# 2024-02-09T17:57:32.351 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.351 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17a3f22f:::benchmark_data_smithi179_148205_object107169:head# 2024-02-09T17:57:32.351 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.351 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17a485f4:::benchmark_data_smithi179_148205_object54503:head# 2024-02-09T17:57:32.351 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.351 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17a4ea2c:::benchmark_data_smithi179_148205_object57591:head# 2024-02-09T17:57:32.351 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.351 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17a56def:::benchmark_data_smithi179_148205_object23602:head# 2024-02-09T17:57:32.351 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.351 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17a5b774:::benchmark_data_smithi179_148205_object9492:head# 2024-02-09T17:57:32.351 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.352 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17a61e83:::benchmark_data_smithi179_148205_object93865:head# 2024-02-09T17:57:32.352 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.352 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17a8b22d:::benchmark_data_smithi179_148205_object29304:head# 2024-02-09T17:57:32.352 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.354 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17a95d83:::benchmark_data_smithi179_148205_object8294:head# 2024-02-09T17:57:32.354 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.355 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17a9ca20:::benchmark_data_smithi179_148205_object34156:head# 2024-02-09T17:57:32.355 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.355 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17aa5780:::benchmark_data_smithi179_148205_object19988:head# 2024-02-09T17:57:32.355 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.355 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17aa8028:::benchmark_data_smithi179_148205_object81055:head# 2024-02-09T17:57:32.355 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.355 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17ab7d2c:::benchmark_data_smithi179_148205_object3814:head# 2024-02-09T17:57:32.355 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.355 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17ab8f99:::benchmark_data_smithi179_148205_object99125:head# 2024-02-09T17:57:32.355 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.355 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17ac47ed:::benchmark_data_smithi179_148205_object99016:head# 2024-02-09T17:57:32.355 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.355 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17ac75f3:::benchmark_data_smithi179_148205_object107054:head# 2024-02-09T17:57:32.355 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.356 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17ad7dfc:::benchmark_data_smithi179_148205_object30957:head# 2024-02-09T17:57:32.356 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.356 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17add35a:::benchmark_data_smithi179_148205_object87197:head# 2024-02-09T17:57:32.356 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.356 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17ae607f:::benchmark_data_smithi179_148205_object90837:head# 2024-02-09T17:57:32.356 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.356 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17af6ff2:::benchmark_data_smithi179_148205_object9872:head# 2024-02-09T17:57:32.356 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.356 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17afa79f:::benchmark_data_smithi179_148205_object87245:head# 2024-02-09T17:57:32.356 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.356 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17afd7d9:::benchmark_data_smithi179_148205_object81485:head# 2024-02-09T17:57:32.356 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.356 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17b143e4:::benchmark_data_smithi179_148205_object86741:head# 2024-02-09T17:57:32.356 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.357 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17b17c44:::benchmark_data_smithi179_148205_object70805:head# 2024-02-09T17:57:32.357 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.357 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17b1a39b:::benchmark_data_smithi179_148205_object90234:head# 2024-02-09T17:57:32.357 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.359 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17b208ae:::benchmark_data_smithi179_148205_object37800:head# 2024-02-09T17:57:32.360 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.360 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17b2bcd6:::benchmark_data_smithi179_148205_object102507:head# 2024-02-09T17:57:32.360 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.360 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17b2e198:::benchmark_data_smithi179_148205_object73198:head# 2024-02-09T17:57:32.360 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.360 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17b6825a:::benchmark_data_smithi179_148205_object86738:head# 2024-02-09T17:57:32.360 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.360 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17b70c8c:::benchmark_data_smithi179_148205_object34948:head# 2024-02-09T17:57:32.360 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.360 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17b74979:::benchmark_data_smithi179_148205_object95199:head# 2024-02-09T17:57:32.360 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.360 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17b75b3e:::benchmark_data_smithi179_148205_object58585:head# 2024-02-09T17:57:32.360 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.361 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17b7c2b7:::benchmark_data_smithi179_148205_object93524:head# 2024-02-09T17:57:32.361 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.361 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17b9354f:::benchmark_data_smithi179_148205_object107144:head# 2024-02-09T17:57:32.361 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.361 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17b94e5a:::benchmark_data_smithi179_148205_object33350:head# 2024-02-09T17:57:32.361 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.361 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17b9c90e:::benchmark_data_smithi179_148205_object74352:head# 2024-02-09T17:57:32.361 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.361 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17ba371c:::benchmark_data_smithi179_148205_object88741:head# 2024-02-09T17:57:32.361 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.361 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17bacd84:::benchmark_data_smithi179_148205_object58058:head# 2024-02-09T17:57:32.361 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.361 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17bbdb7f:::benchmark_data_smithi179_148205_object66784:head# 2024-02-09T17:57:32.361 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.361 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17bbec04:::benchmark_data_smithi179_148205_object90267:head# 2024-02-09T17:57:32.361 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.362 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17bc20d1:::benchmark_data_smithi179_148205_object84533:head# 2024-02-09T17:57:32.362 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.362 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17bc36da:::benchmark_data_smithi179_148205_object26345:head# 2024-02-09T17:57:32.362 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.362 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17bcb66b:::benchmark_data_smithi179_148205_object29703:head# 2024-02-09T17:57:32.362 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.362 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17bcbd61:::benchmark_data_smithi179_148205_object90589:head# 2024-02-09T17:57:32.362 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.362 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17bdf01b:::benchmark_data_smithi179_148205_object28467:head# 2024-02-09T17:57:32.362 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.362 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17bdf208:::benchmark_data_smithi179_148205_object35695:head# 2024-02-09T17:57:32.362 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.363 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17bdfdad:::benchmark_data_smithi179_148205_object74538:head# 2024-02-09T17:57:32.363 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.364 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17bfa99b:::benchmark_data_smithi179_148205_object74378:head# 2024-02-09T17:57:32.364 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.364 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17bfb3e9:::benchmark_data_smithi179_148205_object29339:head# 2024-02-09T17:57:32.364 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.364 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17c14fae:::benchmark_data_smithi179_148205_object22422:head# 2024-02-09T17:57:32.364 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.365 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17c171ac:::benchmark_data_smithi179_148205_object57256:head# 2024-02-09T17:57:32.365 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.365 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17c18f52:::benchmark_data_smithi179_148205_object68880:head# 2024-02-09T17:57:32.365 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.365 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17c28cbc:::benchmark_data_smithi179_148205_object47784:head# 2024-02-09T17:57:32.365 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.366 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17c364d5:::benchmark_data_smithi179_148205_object43804:head# 2024-02-09T17:57:32.366 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.366 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17c46e67:::benchmark_data_smithi179_148205_object10362:head# 2024-02-09T17:57:32.366 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.366 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17c4b43c:::benchmark_data_smithi179_148205_object28436:head# 2024-02-09T17:57:32.366 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.367 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17c4f9d4:::benchmark_data_smithi179_148205_object18983:head# 2024-02-09T17:57:32.367 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.368 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17c58065:::benchmark_data_smithi179_148205_object2577:head# 2024-02-09T17:57:32.368 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.368 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17c58365:::benchmark_data_smithi179_148205_object45448:head# 2024-02-09T17:57:32.368 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.368 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17c67264:::benchmark_data_smithi179_148205_object22250:head# 2024-02-09T17:57:32.368 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.368 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17c6f733:::benchmark_data_smithi179_148205_object53852:head# 2024-02-09T17:57:32.368 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.368 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17c85056:::benchmark_data_smithi179_148205_object87807:head# 2024-02-09T17:57:32.368 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.368 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17caaa2f:::benchmark_data_smithi179_148205_object61261:head# 2024-02-09T17:57:32.368 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.368 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17cbce81:::benchmark_data_smithi179_148205_object12413:head# 2024-02-09T17:57:32.368 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.369 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17cbead9:::benchmark_data_smithi179_148205_object14521:head# 2024-02-09T17:57:32.369 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.370 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17cc961c:::benchmark_data_smithi179_148205_object106762:head# 2024-02-09T17:57:32.370 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.370 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17cc96c6:::benchmark_data_smithi179_148205_object26063:head# 2024-02-09T17:57:32.370 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.370 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17cdac2e:::benchmark_data_smithi179_148205_object94:head# 2024-02-09T17:57:32.370 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.371 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17ce9456:::benchmark_data_smithi179_148205_object109101:head# 2024-02-09T17:57:32.371 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.371 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17cec884:::benchmark_data_smithi179_148205_object79612:head# 2024-02-09T17:57:32.371 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.371 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17cf05eb:::benchmark_data_smithi179_148205_object48567:head# 2024-02-09T17:57:32.371 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.371 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17cf36c6:::benchmark_data_smithi179_148205_object79111:head# 2024-02-09T17:57:32.371 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.371 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17d02e8b:::benchmark_data_smithi179_148205_object95614:head# 2024-02-09T17:57:32.371 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.371 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17d18237:::benchmark_data_smithi179_148205_object58746:head# 2024-02-09T17:57:32.371 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.371 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17d29286:::benchmark_data_smithi179_148205_object74239:head# 2024-02-09T17:57:32.371 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.371 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17d2bb6e:::benchmark_data_smithi179_148205_object85427:head# 2024-02-09T17:57:32.371 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.372 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17d3158d:::benchmark_data_smithi179_148205_object41809:head# 2024-02-09T17:57:32.372 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.372 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17d42289:::benchmark_data_smithi179_148205_object60236:head# 2024-02-09T17:57:32.372 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.372 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17d42df6:::benchmark_data_smithi179_148205_object75316:head# 2024-02-09T17:57:32.372 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.376 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17d45536:::benchmark_data_smithi179_148205_object93257:head# 2024-02-09T17:57:32.376 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.376 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17d4d85d:::benchmark_data_smithi179_148205_object88238:head# 2024-02-09T17:57:32.376 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.376 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17d6458e:::benchmark_data_smithi179_148205_object61958:head# 2024-02-09T17:57:32.376 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.376 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17d6a0cf:::benchmark_data_smithi179_148205_object97587:head# 2024-02-09T17:57:32.376 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.376 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17d6b160:::benchmark_data_smithi179_148205_object29532:head# 2024-02-09T17:57:32.376 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.377 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17d72f10:::benchmark_data_smithi179_148205_object98597:head# 2024-02-09T17:57:32.377 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.377 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17d938df:::benchmark_data_smithi179_148205_object3668:head# 2024-02-09T17:57:32.377 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.377 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17d971de:::benchmark_data_smithi179_148205_object15535:head# 2024-02-09T17:57:32.377 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.377 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17d9b1da:::benchmark_data_smithi179_148205_object15397:head# 2024-02-09T17:57:32.377 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.377 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17db2df6:::benchmark_data_smithi179_148205_object90551:head# 2024-02-09T17:57:32.377 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.377 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17db55f3:::benchmark_data_smithi179_148205_object15787:head# 2024-02-09T17:57:32.377 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.377 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17dc196d:::benchmark_data_smithi179_148205_object65219:head# 2024-02-09T17:57:32.377 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.378 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17dcac07:::benchmark_data_smithi179_148205_object54624:head# 2024-02-09T17:57:32.378 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.379 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17dcc930:::benchmark_data_smithi179_148205_object70864:head# 2024-02-09T17:57:32.379 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.379 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17dcf281:::benchmark_data_smithi179_148205_object53468:head# 2024-02-09T17:57:32.379 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.379 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17de874b:::benchmark_data_smithi179_148205_object57619:head# 2024-02-09T17:57:32.379 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.379 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17dee328:::benchmark_data_smithi179_148205_object92101:head# 2024-02-09T17:57:32.379 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.379 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17deeeda:::benchmark_data_smithi179_148205_object47087:head# 2024-02-09T17:57:32.379 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.379 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17df2ff3:::benchmark_data_smithi179_148205_object27393:head# 2024-02-09T17:57:32.379 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.379 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17e09846:::benchmark_data_smithi179_148205_object106398:head# 2024-02-09T17:57:32.379 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.379 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17e0b855:::benchmark_data_smithi179_148205_object100310:head# 2024-02-09T17:57:32.380 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.381 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17e1b22e:::benchmark_data_smithi179_148205_object30458:head# 2024-02-09T17:57:32.382 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.382 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17e1eedb:::benchmark_data_smithi179_148205_object83685:head# 2024-02-09T17:57:32.382 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.382 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17e1f52a:::benchmark_data_smithi179_148205_object108298:head# 2024-02-09T17:57:32.382 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.382 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17e447cc:::benchmark_data_smithi179_148205_object21723:head# 2024-02-09T17:57:32.382 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.382 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17e50d5d:::benchmark_data_smithi179_148205_object15683:head# 2024-02-09T17:57:32.382 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.382 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17e5190d:::benchmark_data_smithi179_148205_object42352:head# 2024-02-09T17:57:32.382 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.382 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17e5b517:::benchmark_data_smithi179_148205_object26010:head# 2024-02-09T17:57:32.382 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.382 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17e5bcfe:::benchmark_data_smithi179_148205_object42241:head# 2024-02-09T17:57:32.383 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.383 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17e60c62:::benchmark_data_smithi179_148205_object77605:head# 2024-02-09T17:57:32.383 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.383 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17e73f87:::benchmark_data_smithi179_148205_object98868:head# 2024-02-09T17:57:32.383 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.383 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17e8e028:::benchmark_data_smithi179_148205_object89059:head# 2024-02-09T17:57:32.383 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.383 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17e94873:::benchmark_data_smithi179_148205_object70559:head# 2024-02-09T17:57:32.383 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.386 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17ea0c97:::benchmark_data_smithi179_148205_object22799:head# 2024-02-09T17:57:32.386 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.386 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17ea8955:::benchmark_data_smithi179_148205_object104005:head# 2024-02-09T17:57:32.386 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.387 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17eacc1a:::benchmark_data_smithi179_148205_object70601:head# 2024-02-09T17:57:32.387 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.387 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17eb8894:::benchmark_data_smithi179_148205_object37434:head# 2024-02-09T17:57:32.387 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.387 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17ecf46c:::benchmark_data_smithi179_148205_object21276:head# 2024-02-09T17:57:32.387 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.387 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17edbeb8:::benchmark_data_smithi179_148205_object22003:head# 2024-02-09T17:57:32.387 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.387 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17ee241a:::benchmark_data_smithi179_148205_object88436:head# 2024-02-09T17:57:32.387 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.387 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17eef69d:::benchmark_data_smithi179_148205_object84915:head# 2024-02-09T17:57:32.387 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.387 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17ef0941:::benchmark_data_smithi179_148205_object97562:head# 2024-02-09T17:57:32.387 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.387 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17ef7417:::benchmark_data_smithi179_148205_object78577:head# 2024-02-09T17:57:32.388 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.388 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17efb25e:::benchmark_data_smithi179_148205_object17968:head# 2024-02-09T17:57:32.388 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.388 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17efd216:::benchmark_data_smithi179_148205_object12025:head# 2024-02-09T17:57:32.388 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.388 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17f01abc:::benchmark_data_smithi179_148205_object77438:head# 2024-02-09T17:57:32.388 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.388 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17f035c7:::benchmark_data_smithi179_148205_object44074:head# 2024-02-09T17:57:32.388 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.388 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17f144f0:::benchmark_data_smithi179_148205_object80025:head# 2024-02-09T17:57:32.388 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.388 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17f35fa1:::benchmark_data_smithi179_148205_object96501:head# 2024-02-09T17:57:32.388 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.388 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17f42d7a:::benchmark_data_smithi179_148205_object8980:head# 2024-02-09T17:57:32.389 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.389 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17f4e6f5:::benchmark_data_smithi179_148205_object73693:head# 2024-02-09T17:57:32.389 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.389 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17f4e8f0:::benchmark_data_smithi179_148205_object69249:head# 2024-02-09T17:57:32.389 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.389 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17f58638:::benchmark_data_smithi179_148205_object61319:head# 2024-02-09T17:57:32.389 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.392 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17f58a3e:::benchmark_data_smithi179_148205_object51583:head# 2024-02-09T17:57:32.393 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.393 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17f5f5c0:::benchmark_data_smithi179_148205_object64915:head# 2024-02-09T17:57:32.393 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.393 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17f63590:::benchmark_data_smithi179_148205_object8353:head# 2024-02-09T17:57:32.393 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.393 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17f65ca9:::benchmark_data_smithi179_148205_object7699:head# 2024-02-09T17:57:32.393 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.393 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17f66598:::benchmark_data_smithi179_148205_object63067:head# 2024-02-09T17:57:32.393 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.393 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17f6e6c9:::benchmark_data_smithi179_148205_object33464:head# 2024-02-09T17:57:32.393 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.393 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17f79ba7:::benchmark_data_smithi179_148205_object109452:head# 2024-02-09T17:57:32.393 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.393 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17fa9c7f:::benchmark_data_smithi179_148205_object45225:head# 2024-02-09T17:57:32.394 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.394 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17faf9e3:::benchmark_data_smithi179_148205_object82420:head# 2024-02-09T17:57:32.394 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.394 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17fb862e:::benchmark_data_smithi179_148205_object54856:head# 2024-02-09T17:57:32.394 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.394 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17fb98ba:::benchmark_data_smithi179_148205_object2872:head# 2024-02-09T17:57:32.394 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.394 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17fc5cf9:::benchmark_data_smithi179_148205_object16356:head# 2024-02-09T17:57:32.394 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.394 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17fcbdff:::benchmark_data_smithi179_148205_object66066:head# 2024-02-09T17:57:32.394 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.394 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17fcf5ca:::benchmark_data_smithi179_148205_object89320:head# 2024-02-09T17:57:32.394 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.395 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17fd4299:::benchmark_data_smithi179_148205_object45935:head# 2024-02-09T17:57:32.395 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.395 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17fd9800:::benchmark_data_smithi179_148205_object26373:head# 2024-02-09T17:57:32.395 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.395 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17fdbf8d:::benchmark_data_smithi179_148205_object36513:head# 2024-02-09T17:57:32.395 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.395 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17fdd301:::benchmark_data_smithi179_148205_object50159:head# 2024-02-09T17:57:32.395 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.395 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17fde6dc:::benchmark_data_smithi179_148205_object50891:head# 2024-02-09T17:57:32.395 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.395 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17fe170c:::benchmark_data_smithi179_148205_object94733:head# 2024-02-09T17:57:32.395 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.395 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17ff2b4e:::benchmark_data_smithi179_148205_object33118:head# 2024-02-09T17:57:32.395 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.395 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17ff2c64:::benchmark_data_smithi179_148205_object29361:head# 2024-02-09T17:57:32.396 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.396 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17ff5d45:::benchmark_data_smithi179_148205_object88602:head# 2024-02-09T17:57:32.396 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.396 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17ffe287:::benchmark_data_smithi179_148205_object101170:head# 2024-02-09T17:57:32.396 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.396 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:17ffef4b:::benchmark_data_smithi179_148205_object48930:head# 2024-02-09T17:57:32.396 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.397 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:32 smithi179 ceph-mon[43975]: pgmap v2050: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 5.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 808 op/s; 18054/285570 objects degraded (6.322%) 2024-02-09T17:57:32.397 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:32 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1014253901' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:57:32.397 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:32 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:57:32.397 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:32 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 18054/287139 objects degraded (6.288%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:57:32.402 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18001750:::benchmark_data_smithi179_148205_object87478:head# 2024-02-09T17:57:32.402 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.402 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18005118:::benchmark_data_smithi179_148205_object24833:head# 2024-02-09T17:57:32.402 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.402 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1800e829:::benchmark_data_smithi179_148205_object103075:head# 2024-02-09T17:57:32.402 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.403 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:180130fe:::benchmark_data_smithi179_148205_object96842:head# 2024-02-09T17:57:32.403 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.403 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1801837f:::benchmark_data_smithi179_148205_object46951:head# 2024-02-09T17:57:32.403 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.403 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:180207f3:::benchmark_data_smithi179_148205_object50950:head# 2024-02-09T17:57:32.403 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.403 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1802c3ac:::benchmark_data_smithi179_148205_object102311:head# 2024-02-09T17:57:32.403 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.403 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:180685c5:::benchmark_data_smithi179_148205_object74144:head# 2024-02-09T17:57:32.403 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.403 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18076014:::benchmark_data_smithi179_148205_object3298:head# 2024-02-09T17:57:32.403 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.403 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1807b4c0:::benchmark_data_smithi179_148205_object61340:head# 2024-02-09T17:57:32.403 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.403 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18082348:::benchmark_data_smithi179_148205_object4094:head# 2024-02-09T17:57:32.404 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.404 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1808da3f:::benchmark_data_smithi179_148205_object76595:head# 2024-02-09T17:57:32.404 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.404 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:180923b0:::benchmark_data_smithi179_148205_object16681:head# 2024-02-09T17:57:32.404 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.404 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:180933c0:::benchmark_data_smithi179_148205_object11066:head# 2024-02-09T17:57:32.404 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.404 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:180aa7dd:::benchmark_data_smithi179_148205_object74331:head# 2024-02-09T17:57:32.404 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.404 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:180acbe9:::benchmark_data_smithi179_148205_object47869:head# 2024-02-09T17:57:32.404 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.404 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:180b1d2c:::benchmark_data_smithi179_148205_object29924:head# 2024-02-09T17:57:32.404 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.404 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:180b5e68:::benchmark_data_smithi179_148205_object102095:head# 2024-02-09T17:57:32.404 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.405 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:180cfe80:::benchmark_data_smithi179_148205_object820:head# 2024-02-09T17:57:32.405 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.405 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:180d1585:::benchmark_data_smithi179_148205_object86261:head# 2024-02-09T17:57:32.405 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.405 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:180d2d3b:::benchmark_data_smithi179_148205_object67356:head# 2024-02-09T17:57:32.405 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.405 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:180d6ae5:::benchmark_data_smithi179_148205_object97711:head# 2024-02-09T17:57:32.405 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.405 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:180db7fa:::benchmark_data_smithi179_148205_object43314:head# 2024-02-09T17:57:32.405 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.405 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:180dfcfc:::benchmark_data_smithi179_148205_object68806:head# 2024-02-09T17:57:32.405 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.405 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:180e4159:::benchmark_data_smithi179_148205_object69244:head# 2024-02-09T17:57:32.405 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.405 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:180e5a38:::benchmark_data_smithi179_148205_object55895:head# 2024-02-09T17:57:32.405 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.406 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:180f37cf:::benchmark_data_smithi179_148205_object488:head# 2024-02-09T17:57:32.406 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.406 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:180f4f60:::benchmark_data_smithi179_148205_object97041:head# 2024-02-09T17:57:32.406 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.406 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:180fa37f:::benchmark_data_smithi179_148205_object98530:head# 2024-02-09T17:57:32.406 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.406 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:180fe1dc:::benchmark_data_smithi179_148205_object86383:head# 2024-02-09T17:57:32.406 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.406 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18106143:::benchmark_data_smithi179_148205_object9558:head# 2024-02-09T17:57:32.406 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.406 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18108300:::benchmark_data_smithi179_148205_object109802:head# 2024-02-09T17:57:32.406 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.406 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1811e1b0:::benchmark_data_smithi179_148205_object25728:head# 2024-02-09T17:57:32.406 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.407 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18123b74:::benchmark_data_smithi179_148205_object28577:head# 2024-02-09T17:57:32.407 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.407 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:181278f7:::benchmark_data_smithi179_148205_object70924:head# 2024-02-09T17:57:32.407 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.407 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18138867:::benchmark_data_smithi179_148205_object62372:head# 2024-02-09T17:57:32.407 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.407 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1813ef4c:::benchmark_data_smithi179_148205_object12942:head# 2024-02-09T17:57:32.407 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.407 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:32.409 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18146a32:::benchmark_data_smithi179_148205_object51639:head# 2024-02-09T17:57:32.409 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.409 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18150b56:::benchmark_data_smithi179_148205_object51578:head# 2024-02-09T17:57:32.409 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.409 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1815989d:::benchmark_data_smithi179_148205_object74987:head# 2024-02-09T17:57:32.409 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.409 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18175efc:::benchmark_data_smithi179_148205_object75009:head# 2024-02-09T17:57:32.409 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.409 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:181810bf:::benchmark_data_smithi179_148205_object63594:head# 2024-02-09T17:57:32.409 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.410 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:181828d3:::benchmark_data_smithi179_148205_object32017:head# 2024-02-09T17:57:32.410 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.410 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18186ce8:::benchmark_data_smithi179_148205_object71332:head# 2024-02-09T17:57:32.410 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.410 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18186ffe:::benchmark_data_smithi179_148205_object64619:head# 2024-02-09T17:57:32.410 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.410 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18191bf0:::benchmark_data_smithi179_148205_object99707:head# 2024-02-09T17:57:32.410 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.410 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18196008:::benchmark_data_smithi179_148205_object8161:head# 2024-02-09T17:57:32.410 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.412 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1819a016:::benchmark_data_smithi179_148205_object31768:head# 2024-02-09T17:57:32.412 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.413 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:181a7820:::benchmark_data_smithi179_148205_object39646:head# 2024-02-09T17:57:32.413 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.413 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:181b6b21:::benchmark_data_smithi179_148205_object26066:head# 2024-02-09T17:57:32.413 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.413 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:181b82ea:::benchmark_data_smithi179_148205_object55108:head# 2024-02-09T17:57:32.413 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.413 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:181b8cac:::benchmark_data_smithi179_148205_object43615:head# 2024-02-09T17:57:32.413 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.413 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:181bd935:::benchmark_data_smithi179_148205_object72660:head# 2024-02-09T17:57:32.413 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.413 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:181cb8e1:::benchmark_data_smithi179_148205_object9347:head# 2024-02-09T17:57:32.413 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.413 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:181d1250:::benchmark_data_smithi179_148205_object546:head# 2024-02-09T17:57:32.413 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.414 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:181d7a19:::benchmark_data_smithi179_148205_object6706:head# 2024-02-09T17:57:32.414 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.414 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:181d9e63:::benchmark_data_smithi179_148205_object65933:head# 2024-02-09T17:57:32.414 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.414 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:181e04fe:::benchmark_data_smithi179_148205_object45810:head# 2024-02-09T17:57:32.414 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.414 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:181efd93:::benchmark_data_smithi179_148205_object50293:head# 2024-02-09T17:57:32.414 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.414 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:181efe0f:::benchmark_data_smithi179_148205_object40216:head# 2024-02-09T17:57:32.414 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.414 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:181f13d9:::benchmark_data_smithi179_148205_object74189:head# 2024-02-09T17:57:32.414 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.416 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:181f30ed:::benchmark_data_smithi179_148205_object79806:head# 2024-02-09T17:57:32.416 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.416 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18202ac1:::benchmark_data_smithi179_148205_object25339:head# 2024-02-09T17:57:32.416 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.416 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1820585b:::benchmark_data_smithi179_148205_object40896:head# 2024-02-09T17:57:32.417 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.417 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1820ceac:::benchmark_data_smithi179_148205_object96218:head# 2024-02-09T17:57:32.417 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.417 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:182246bb:::benchmark_data_smithi179_148205_object42143:head# 2024-02-09T17:57:32.417 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.417 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1822d890:::benchmark_data_smithi179_148205_object84608:head# 2024-02-09T17:57:32.417 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.417 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18237687:::benchmark_data_smithi179_148205_object28390:head# 2024-02-09T17:57:32.417 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.417 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1823d7c0:::benchmark_data_smithi179_148205_object18795:head# 2024-02-09T17:57:32.417 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.417 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18249e65:::benchmark_data_smithi179_148205_object108472:head# 2024-02-09T17:57:32.417 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.420 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1826133e:::benchmark_data_smithi179_148205_object548:head# 2024-02-09T17:57:32.420 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.420 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18271167:::benchmark_data_smithi179_148205_object94812:head# 2024-02-09T17:57:32.421 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.421 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18280722:::benchmark_data_smithi179_148205_object10521:head# 2024-02-09T17:57:32.421 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.422 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18281a1a:::benchmark_data_smithi179_148205_object20520:head# 2024-02-09T17:57:32.422 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.422 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1828d111:::benchmark_data_smithi179_148205_object4678:head# 2024-02-09T17:57:32.422 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.422 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:182a33a5:::benchmark_data_smithi179_148205_object79104:head# 2024-02-09T17:57:32.422 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.423 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:182a9576:::benchmark_data_smithi179_148205_object63608:head# 2024-02-09T17:57:32.424 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.424 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:182b2c19:::benchmark_data_smithi179_148205_object103931:head# 2024-02-09T17:57:32.424 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.424 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:182c0e70:::benchmark_data_smithi179_148205_object94440:head# 2024-02-09T17:57:32.424 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.424 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:182c9a73:::benchmark_data_smithi179_148205_object25285:head# 2024-02-09T17:57:32.424 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.424 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:182d8bfc:::benchmark_data_smithi179_148205_object37082:head# 2024-02-09T17:57:32.424 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.424 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:182de895:::benchmark_data_smithi179_148205_object21819:head# 2024-02-09T17:57:32.424 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.424 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:182e3111:::benchmark_data_smithi179_148205_object37594:head# 2024-02-09T17:57:32.424 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.424 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:182eeafe:::benchmark_data_smithi179_148205_object59821:head# 2024-02-09T17:57:32.425 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.425 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:182f5e24:::benchmark_data_smithi179_148205_object81268:head# 2024-02-09T17:57:32.425 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.428 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:182f6cbd:::benchmark_data_smithi179_148205_object4521:head# 2024-02-09T17:57:32.428 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.428 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:182f7f8e:::benchmark_data_smithi179_148205_object3624:head# 2024-02-09T17:57:32.428 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.428 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:183026e0:::benchmark_data_smithi179_148205_object26267:head# 2024-02-09T17:57:32.428 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.428 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18303033:::benchmark_data_smithi179_148205_object84922:head# 2024-02-09T17:57:32.428 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.428 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1832b374:::benchmark_data_smithi179_148205_object101329:head# 2024-02-09T17:57:32.428 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.428 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1832ec8b:::benchmark_data_smithi179_148205_object75430:head# 2024-02-09T17:57:32.428 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.428 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18333b99:::benchmark_data_smithi179_148205_object27696:head# 2024-02-09T17:57:32.428 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.429 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1833fc52:::benchmark_data_smithi179_148205_object33640:head# 2024-02-09T17:57:32.429 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.429 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1834272c:::benchmark_data_smithi179_148205_object21597:head# 2024-02-09T17:57:32.429 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.429 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1834b1df:::benchmark_data_smithi179_148205_object23090:head# 2024-02-09T17:57:32.429 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.429 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18352040:::benchmark_data_smithi179_148205_object77424:head# 2024-02-09T17:57:32.429 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.429 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1836eda0:::benchmark_data_smithi179_148205_object74921:head# 2024-02-09T17:57:32.429 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.429 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1837269e:::benchmark_data_smithi179_148205_object74745:head# 2024-02-09T17:57:32.429 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.429 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:183734d9:::benchmark_data_smithi179_148205_object78170:head# 2024-02-09T17:57:32.429 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.429 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1837a208:::benchmark_data_smithi179_148205_object34220:head# 2024-02-09T17:57:32.430 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.430 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18394c25:::benchmark_data_smithi179_148205_object95868:head# 2024-02-09T17:57:32.430 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.430 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1839df46:::benchmark_data_smithi179_148205_object99764:head# 2024-02-09T17:57:32.430 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:32 smithi162 ceph-mon[44304]: pgmap v2050: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 5.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 808 op/s; 18054/285570 objects degraded (6.322%) 2024-02-09T17:57:32.430 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:32 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1014253901' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:57:32.431 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:32 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:57:32.431 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:32 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 18054/287139 objects degraded (6.288%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:57:32.436 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:183a37cf:::benchmark_data_smithi179_148205_object64610:head# 2024-02-09T17:57:32.436 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.436 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:183ae0c3:::benchmark_data_smithi179_148205_object76356:head# 2024-02-09T17:57:32.436 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.436 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:183afa00:::benchmark_data_smithi179_148205_object18004:head# 2024-02-09T17:57:32.436 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.437 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:183bc896:::benchmark_data_smithi179_148205_object20847:head# 2024-02-09T17:57:32.437 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.437 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:183c5e22:::benchmark_data_smithi179_148205_object31373:head# 2024-02-09T17:57:32.437 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.437 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:183ce12d:::benchmark_data_smithi179_148205_object60095:head# 2024-02-09T17:57:32.437 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.437 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:183da06d:::benchmark_data_smithi179_148205_object67281:head# 2024-02-09T17:57:32.437 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.437 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:183db0fd:::benchmark_data_smithi179_148205_object14369:head# 2024-02-09T17:57:32.437 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.437 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:183e0880:::benchmark_data_smithi179_148205_object44572:head# 2024-02-09T17:57:32.437 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.437 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:183fcca5:::benchmark_data_smithi179_148205_object17538:head# 2024-02-09T17:57:32.438 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.438 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1840241d:::benchmark_data_smithi179_148205_object19401:head# 2024-02-09T17:57:32.438 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.438 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1841cd0b:::benchmark_data_smithi179_148205_object18268:head# 2024-02-09T17:57:32.438 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.438 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18421157:::benchmark_data_smithi179_148205_object21439:head# 2024-02-09T17:57:32.438 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.438 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1842a299:::benchmark_data_smithi179_148205_object13499:head# 2024-02-09T17:57:32.438 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.438 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18430061:::benchmark_data_smithi179_148205_object69650:head# 2024-02-09T17:57:32.438 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.438 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1843aed1:::benchmark_data_smithi179_148205_object2374:head# 2024-02-09T17:57:32.438 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.438 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:184663d9:::benchmark_data_smithi179_148205_object80456:head# 2024-02-09T17:57:32.438 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.439 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:184663e7:::benchmark_data_smithi179_148205_object80844:head# 2024-02-09T17:57:32.439 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.439 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:184878f6:::benchmark_data_smithi179_148205_object13847:head# 2024-02-09T17:57:32.439 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.439 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18490b3f:::benchmark_data_smithi179_148205_object63562:head# 2024-02-09T17:57:32.439 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.439 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:184ae1bd:::benchmark_data_smithi179_148205_object97984:head# 2024-02-09T17:57:32.439 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.439 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:184ae6f3:::benchmark_data_smithi179_148205_object82567:head# 2024-02-09T17:57:32.439 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.439 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:184b18c8:::benchmark_data_smithi179_148205_object99794:head# 2024-02-09T17:57:32.439 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.439 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:184bd37b:::benchmark_data_smithi179_148205_object14000:head# 2024-02-09T17:57:32.439 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.439 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:184cb73b:::benchmark_data_smithi179_148205_object42728:head# 2024-02-09T17:57:32.439 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.440 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:184f224c:::benchmark_data_smithi179_148205_object33909:head# 2024-02-09T17:57:32.440 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.440 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:184f974b:::benchmark_data_smithi179_148205_object27734:head# 2024-02-09T17:57:32.440 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.440 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1850c8d4:::benchmark_data_smithi179_148205_object82889:head# 2024-02-09T17:57:32.440 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.440 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1851eb05:::benchmark_data_smithi179_148205_object38114:head# 2024-02-09T17:57:32.440 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.440 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1852aacd:::benchmark_data_smithi179_148205_object76681:head# 2024-02-09T17:57:32.440 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.440 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1852ac4e:::benchmark_data_smithi179_148205_object35427:head# 2024-02-09T17:57:32.440 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.440 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:185359a8:::benchmark_data_smithi179_148205_object52173:head# 2024-02-09T17:57:32.440 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.440 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:185407b5:::benchmark_data_smithi179_148205_object64683:head# 2024-02-09T17:57:32.440 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.441 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1854f781:::benchmark_data_smithi179_148205_object4170:head# 2024-02-09T17:57:32.441 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.441 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18552587:::benchmark_data_smithi179_148205_object58824:head# 2024-02-09T17:57:32.441 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.441 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:57:32.446 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18552bb0:::benchmark_data_smithi179_148205_object28034:head# 2024-02-09T17:57:32.446 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.446 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:185545f5:::benchmark_data_smithi179_148205_object33993:head# 2024-02-09T17:57:32.446 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.447 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18557824:::benchmark_data_smithi179_148205_object24620:head# 2024-02-09T17:57:32.447 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.447 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1855b9c3:::benchmark_data_smithi179_148205_object13161:head# 2024-02-09T17:57:32.447 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.447 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18562b79:::benchmark_data_smithi179_148205_object34492:head# 2024-02-09T17:57:32.447 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.447 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18567fd1:::benchmark_data_smithi179_148205_object31358:head# 2024-02-09T17:57:32.447 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.447 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1856b870:::benchmark_data_smithi179_148205_object4333:head# 2024-02-09T17:57:32.447 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.447 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1857f933:::benchmark_data_smithi179_148205_object46791:head# 2024-02-09T17:57:32.447 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.447 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18583e42:::benchmark_data_smithi179_148205_object102716:head# 2024-02-09T17:57:32.447 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.447 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18590b76:::benchmark_data_smithi179_148205_object82212:head# 2024-02-09T17:57:32.447 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.448 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:185947a6:::benchmark_data_smithi179_148205_object99394:head# 2024-02-09T17:57:32.448 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.448 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:185a25c8:::benchmark_data_smithi179_148205_object58899:head# 2024-02-09T17:57:32.448 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.448 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:185afb58:::benchmark_data_smithi179_148205_object76948:head# 2024-02-09T17:57:32.448 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.448 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:185b6ea5:::benchmark_data_smithi179_148205_object38035:head# 2024-02-09T17:57:32.448 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.448 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:185bfaae:::benchmark_data_smithi179_148205_object37418:head# 2024-02-09T17:57:32.448 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.448 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:185d12b9:::benchmark_data_smithi179_148205_object97708:head# 2024-02-09T17:57:32.448 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.448 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:185dae90:::benchmark_data_smithi179_148205_object72052:head# 2024-02-09T17:57:32.449 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.449 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:185eda89:::benchmark_data_smithi179_148205_object20695:head# 2024-02-09T17:57:32.449 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.449 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:185f6242:::benchmark_data_smithi179_148205_object96799:head# 2024-02-09T17:57:32.449 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.449 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:185fade1:::benchmark_data_smithi179_148205_object51209:head# 2024-02-09T17:57:32.449 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.449 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:185ffb25:::benchmark_data_smithi179_148205_object61633:head# 2024-02-09T17:57:32.449 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.449 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1860705c:::benchmark_data_smithi179_148205_object98790:head# 2024-02-09T17:57:32.449 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.449 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:186116f7:::benchmark_data_smithi179_148205_object72112:head# 2024-02-09T17:57:32.449 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.449 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18621809:::benchmark_data_smithi179_148205_object56105:head# 2024-02-09T17:57:32.449 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.450 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18638b28:::benchmark_data_smithi179_148205_object87018:head# 2024-02-09T17:57:32.450 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.450 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1863b7b9:::benchmark_data_smithi179_148205_object90352:head# 2024-02-09T17:57:32.450 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.450 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1863d7f9:::benchmark_data_smithi179_148205_object69533:head# 2024-02-09T17:57:32.450 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.450 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:186465ee:::benchmark_data_smithi179_148205_object97567:head# 2024-02-09T17:57:32.450 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.450 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1864f75f:::benchmark_data_smithi179_148205_object48787:head# 2024-02-09T17:57:32.450 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.450 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1864f789:::benchmark_data_smithi179_148205_object23443:head# 2024-02-09T17:57:32.450 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.450 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1865265e:::benchmark_data_smithi179_148205_object85466:head# 2024-02-09T17:57:32.450 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.450 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1865a0ab:::benchmark_data_smithi179_148205_object39924:head# 2024-02-09T17:57:32.451 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.451 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1865d50b:::benchmark_data_smithi179_148205_object70725:head# 2024-02-09T17:57:32.451 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.451 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1866d94c:::benchmark_data_smithi179_148205_object103220:head# 2024-02-09T17:57:32.451 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.451 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18684065:::benchmark_data_smithi179_148205_object99470:head# 2024-02-09T17:57:32.451 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.451 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1868512a:::benchmark_data_smithi179_148205_object22551:head# 2024-02-09T17:57:32.451 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.452 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18697866:::benchmark_data_smithi179_148205_object76852:head# 2024-02-09T17:57:32.452 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.452 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:186a700a:::benchmark_data_smithi179_148205_object34667:head# 2024-02-09T17:57:32.452 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.452 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:186aca31:::benchmark_data_smithi179_148205_object2278:head# 2024-02-09T17:57:32.452 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.452 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:186ae521:::benchmark_data_smithi179_148205_object40115:head# 2024-02-09T17:57:32.452 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:32.453 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:57:32.454 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:186b4475:::benchmark_data_smithi179_148205_object73339:head# 2024-02-09T17:57:32.454 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.454 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:186b965b:::benchmark_data_smithi179_148205_object1621:head# 2024-02-09T17:57:32.454 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.454 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:186bb3c8:::benchmark_data_smithi179_148205_object9813:head# 2024-02-09T17:57:32.454 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.458 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:186d7099:::benchmark_data_smithi179_148205_object12967:head# 2024-02-09T17:57:32.458 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.458 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:186e5ac6:::benchmark_data_smithi179_148205_object84094:head# 2024-02-09T17:57:32.458 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.458 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:186e9073:::benchmark_data_smithi179_148205_object98884:head# 2024-02-09T17:57:32.458 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.458 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:186ed0a7:::benchmark_data_smithi179_148205_object105034:head# 2024-02-09T17:57:32.458 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.458 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:186f1305:::benchmark_data_smithi179_148205_object57247:head# 2024-02-09T17:57:32.458 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.458 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18705510:::benchmark_data_smithi179_148205_object32490:head# 2024-02-09T17:57:32.458 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.458 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18713635:::benchmark_data_smithi179_148205_object13031:head# 2024-02-09T17:57:32.458 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.459 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1871eac9:::benchmark_data_smithi179_148205_object23437:head# 2024-02-09T17:57:32.459 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.459 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1872293f:::benchmark_data_smithi179_148205_object75589:head# 2024-02-09T17:57:32.459 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.459 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18723218:::benchmark_data_smithi179_148205_object16779:head# 2024-02-09T17:57:32.459 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.459 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1872e5ee:::benchmark_data_smithi179_148205_object4345:head# 2024-02-09T17:57:32.459 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.459 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1872f1e7:::benchmark_data_smithi179_148205_object37104:head# 2024-02-09T17:57:32.459 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.459 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:187399e7:::benchmark_data_smithi179_148205_object109702:head# 2024-02-09T17:57:32.459 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.459 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1873b5a2:::benchmark_data_smithi179_148205_object59687:head# 2024-02-09T17:57:32.459 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.459 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1874daf6:::benchmark_data_smithi179_148205_object29040:head# 2024-02-09T17:57:32.460 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.460 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18750918:::benchmark_data_smithi179_148205_object103661:head# 2024-02-09T17:57:32.460 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.460 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18764168:::benchmark_data_smithi179_148205_object45989:head# 2024-02-09T17:57:32.460 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.460 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:187767e3:::benchmark_data_smithi179_148205_object108355:head# 2024-02-09T17:57:32.460 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.460 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:187887d5:::benchmark_data_smithi179_148205_object615:head# 2024-02-09T17:57:32.460 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.460 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:187924e7:::benchmark_data_smithi179_148205_object109128:head# 2024-02-09T17:57:32.460 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.460 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:187950a3:::benchmark_data_smithi179_148205_object66157:head# 2024-02-09T17:57:32.460 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.460 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18797666:::benchmark_data_smithi179_148205_object85044:head# 2024-02-09T17:57:32.461 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.461 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:187a614e:::benchmark_data_smithi179_148205_object109184:head# 2024-02-09T17:57:32.461 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.461 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:187aaa95:::benchmark_data_smithi179_148205_object27679:head# 2024-02-09T17:57:32.462 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.462 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:187adc14:::benchmark_data_smithi179_148205_object102159:head# 2024-02-09T17:57:32.462 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.465 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:187b50c9:::benchmark_data_smithi179_148205_object17673:head# 2024-02-09T17:57:32.465 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.465 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:187def12:::benchmark_data_smithi179_148205_object14246:head# 2024-02-09T17:57:32.466 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.466 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:187ea22d:::benchmark_data_smithi179_148205_object18222:head# 2024-02-09T17:57:32.466 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.467 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:187ed942:::benchmark_data_smithi179_148205_object98259:head# 2024-02-09T17:57:32.467 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.467 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:187fedfe:::benchmark_data_smithi179_148205_object32691:head# 2024-02-09T17:57:32.467 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.467 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:188079af:::benchmark_data_smithi179_148205_object67168:head# 2024-02-09T17:57:32.467 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.468 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18817bdd:::benchmark_data_smithi179_148205_object74562:head# 2024-02-09T17:57:32.468 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.468 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18820b30:::benchmark_data_smithi179_148205_object37888:head# 2024-02-09T17:57:32.468 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.468 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:188273ac:::benchmark_data_smithi179_148205_object78864:head# 2024-02-09T17:57:32.468 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.470 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18833dad:::benchmark_data_smithi179_148205_object83186:head# 2024-02-09T17:57:32.470 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.470 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18841d0c:::benchmark_data_smithi179_148205_object909:head# 2024-02-09T17:57:32.470 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.470 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18843529:::benchmark_data_smithi179_148205_object28855:head# 2024-02-09T17:57:32.470 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.470 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18846811:::benchmark_data_smithi179_148205_object16865:head# 2024-02-09T17:57:32.470 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.470 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:188486b8:::benchmark_data_smithi179_148205_object17986:head# 2024-02-09T17:57:32.470 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.470 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1884d4a2:::benchmark_data_smithi179_148205_object25845:head# 2024-02-09T17:57:32.470 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.470 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1885eadf:::benchmark_data_smithi179_148205_object102028:head# 2024-02-09T17:57:32.470 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.471 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1885f9d9:::benchmark_data_smithi179_148205_object95773:head# 2024-02-09T17:57:32.471 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.471 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18863f0e:::benchmark_data_smithi179_148205_object104155:head# 2024-02-09T17:57:32.471 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.471 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:188671e1:::benchmark_data_smithi179_148205_object81460:head# 2024-02-09T17:57:32.471 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.473 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1886b417:::benchmark_data_smithi179_148205_object62096:head# 2024-02-09T17:57:32.473 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.473 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1886e6f1:::benchmark_data_smithi179_148205_object88312:head# 2024-02-09T17:57:32.473 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.473 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18872ae5:::benchmark_data_smithi179_148205_object108805:head# 2024-02-09T17:57:32.474 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.474 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:188848fc:::benchmark_data_smithi179_148205_object22393:head# 2024-02-09T17:57:32.474 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.474 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:188874fc:::benchmark_data_smithi179_148205_object2766:head# 2024-02-09T17:57:32.474 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.474 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1889887e:::benchmark_data_smithi179_148205_object80724:head# 2024-02-09T17:57:32.474 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.474 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1889e5e8:::benchmark_data_smithi179_148205_object83165:head# 2024-02-09T17:57:32.474 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.474 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1889f687:::benchmark_data_smithi179_148205_object36781:head# 2024-02-09T17:57:32.474 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.474 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:188a0625:::benchmark_data_smithi179_148205_object97644:head# 2024-02-09T17:57:32.474 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.474 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:188a3692:::benchmark_data_smithi179_148205_object43512:head# 2024-02-09T17:57:32.474 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.474 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:188a974d:::benchmark_data_smithi179_148205_object108601:head# 2024-02-09T17:57:32.475 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.475 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:188ac9cb:::benchmark_data_smithi179_148205_object83316:head# 2024-02-09T17:57:32.475 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.475 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:188cd356:::benchmark_data_smithi179_148205_object64271:head# 2024-02-09T17:57:32.475 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.475 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:188e4ece:::benchmark_data_smithi179_148205_object13866:head# 2024-02-09T17:57:32.475 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.475 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:188ea098:::benchmark_data_smithi179_148205_object54855:head# 2024-02-09T17:57:32.475 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.478 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:188f2744:::benchmark_data_smithi179_148205_object13882:head# 2024-02-09T17:57:32.478 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.478 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:188f3791:::benchmark_data_smithi179_148205_object2674:head# 2024-02-09T17:57:32.478 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.478 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18905192:::benchmark_data_smithi179_148205_object74954:head# 2024-02-09T17:57:32.478 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.478 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1890dd9c:::benchmark_data_smithi179_148205_object75087:head# 2024-02-09T17:57:32.478 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.478 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1890e41f:::benchmark_data_smithi179_148205_object29945:head# 2024-02-09T17:57:32.478 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.478 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:189190e8:::benchmark_data_smithi179_148205_object99224:head# 2024-02-09T17:57:32.478 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.479 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1891ad05:::benchmark_data_smithi179_148205_object30873:head# 2024-02-09T17:57:32.479 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.479 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18927253:::benchmark_data_smithi179_148205_object60385:head# 2024-02-09T17:57:32.479 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.479 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1892b5f5:::benchmark_data_smithi179_148205_object12486:head# 2024-02-09T17:57:32.479 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.479 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1892f085:::benchmark_data_smithi179_148205_object14478:head# 2024-02-09T17:57:32.479 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.479 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18941d41:::benchmark_data_smithi179_148205_object144:head# 2024-02-09T17:57:32.479 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.479 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1897a710:::benchmark_data_smithi179_148205_object36173:head# 2024-02-09T17:57:32.479 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.479 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1897e162:::benchmark_data_smithi179_148205_object75994:head# 2024-02-09T17:57:32.479 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.479 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1898968c:::benchmark_data_smithi179_148205_object106678:head# 2024-02-09T17:57:32.479 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.480 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18999900:::benchmark_data_smithi179_148205_object93114:head# 2024-02-09T17:57:32.480 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.480 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:189b6e81:::benchmark_data_smithi179_148205_object2053:head# 2024-02-09T17:57:32.480 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.480 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:189bd4e9:::benchmark_data_smithi179_148205_object105369:head# 2024-02-09T17:57:32.480 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.480 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:189bde72:::benchmark_data_smithi179_148205_object11858:head# 2024-02-09T17:57:32.480 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.480 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:189c284f:::benchmark_data_smithi179_148205_object23748:head# 2024-02-09T17:57:32.480 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.482 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:189c766c:::benchmark_data_smithi179_148205_object28353:head# 2024-02-09T17:57:32.482 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.482 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:189c8b80:::benchmark_data_smithi179_148205_object17981:head# 2024-02-09T17:57:32.482 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.482 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:189ee248:::benchmark_data_smithi179_148205_object77909:head# 2024-02-09T17:57:32.482 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.482 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:189fe74b:::benchmark_data_smithi179_148205_object94700:head# 2024-02-09T17:57:32.482 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.482 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18a06072:::benchmark_data_smithi179_148205_object82044:head# 2024-02-09T17:57:32.482 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.482 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18a163e2:::benchmark_data_smithi179_148205_object51167:head# 2024-02-09T17:57:32.482 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.483 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18a20c69:::benchmark_data_smithi179_148205_object13323:head# 2024-02-09T17:57:32.483 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.485 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18a2beab:::benchmark_data_smithi179_148205_object56370:head# 2024-02-09T17:57:32.485 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.486 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18a2fdcf:::benchmark_data_smithi179_148205_object101625:head# 2024-02-09T17:57:32.486 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.486 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18a3e072:::benchmark_data_smithi179_148205_object10624:head# 2024-02-09T17:57:32.486 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.487 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18a3e593:::benchmark_data_smithi179_148205_object9509:head# 2024-02-09T17:57:32.487 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.487 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18a528d6:::benchmark_data_smithi179_148205_object4269:head# 2024-02-09T17:57:32.487 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.487 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18a59063:::benchmark_data_smithi179_148205_object63590:head# 2024-02-09T17:57:32.487 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.487 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18a6083d:::benchmark_data_smithi179_148205_object71077:head# 2024-02-09T17:57:32.487 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.489 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18a6fce5:::benchmark_data_smithi179_148205_object24840:head# 2024-02-09T17:57:32.489 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.489 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18a716ce:::benchmark_data_smithi179_148205_object39410:head# 2024-02-09T17:57:32.489 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.489 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18a72504:::benchmark_data_smithi179_148205_object88357:head# 2024-02-09T17:57:32.489 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.489 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18a78cd9:::benchmark_data_smithi179_148205_object88361:head# 2024-02-09T17:57:32.489 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.489 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18a7c6e5:::benchmark_data_smithi179_148205_object99312:head# 2024-02-09T17:57:32.489 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.489 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18a7cc3b:::benchmark_data_smithi179_148205_object82026:head# 2024-02-09T17:57:32.490 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.490 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18a84edd:::benchmark_data_smithi179_148205_object43964:head# 2024-02-09T17:57:32.490 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.490 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18a8de8d:::benchmark_data_smithi179_148205_object46093:head# 2024-02-09T17:57:32.490 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.490 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18a8efe6:::benchmark_data_smithi179_148205_object61069:head# 2024-02-09T17:57:32.490 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.491 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18a8f35a:::benchmark_data_smithi179_148205_object62457:head# 2024-02-09T17:57:32.491 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.491 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18aa0183:::benchmark_data_smithi179_148205_object17816:head# 2024-02-09T17:57:32.491 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.495 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18aa917d:::benchmark_data_smithi179_148205_object66264:head# 2024-02-09T17:57:32.495 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.495 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18aa94b6:::benchmark_data_smithi179_148205_object71209:head# 2024-02-09T17:57:32.495 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.495 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18ac35fd:::benchmark_data_smithi179_148205_object44785:head# 2024-02-09T17:57:32.495 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.495 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18adad27:::benchmark_data_smithi179_148205_object51052:head# 2024-02-09T17:57:32.495 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.495 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18adc138:::benchmark_data_smithi179_148205_object15984:head# 2024-02-09T17:57:32.495 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.495 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18ade227:::benchmark_data_smithi179_148205_object20679:head# 2024-02-09T17:57:32.496 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.496 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18ae1cbf:::benchmark_data_smithi179_148205_object83442:head# 2024-02-09T17:57:32.496 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.496 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18af4632:::benchmark_data_smithi179_148205_object52249:head# 2024-02-09T17:57:32.496 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.496 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18b18f44:::benchmark_data_smithi179_148205_object32162:head# 2024-02-09T17:57:32.496 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.496 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18b358d6:::benchmark_data_smithi179_148205_object14458:head# 2024-02-09T17:57:32.496 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.496 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18b4143d:::benchmark_data_smithi179_148205_object77189:head# 2024-02-09T17:57:32.496 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.496 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18b478c8:::benchmark_data_smithi179_148205_object44711:head# 2024-02-09T17:57:32.496 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.496 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18b48357:::benchmark_data_smithi179_148205_object49373:head# 2024-02-09T17:57:32.497 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.498 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18b4e9da:::benchmark_data_smithi179_148205_object81666:head# 2024-02-09T17:57:32.499 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.499 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18b51da6:::benchmark_data_smithi179_148205_object96495:head# 2024-02-09T17:57:32.499 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.499 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18b652d1:::benchmark_data_smithi179_148205_object95281:head# 2024-02-09T17:57:32.499 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.499 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18b7ea8c:::benchmark_data_smithi179_148205_object55970:head# 2024-02-09T17:57:32.499 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.499 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18b8cb4d:::benchmark_data_smithi179_148205_object3791:head# 2024-02-09T17:57:32.499 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.499 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18b92b1e:::benchmark_data_smithi179_148205_object20774:head# 2024-02-09T17:57:32.499 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.499 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18b9f909:::benchmark_data_smithi179_148205_object73452:head# 2024-02-09T17:57:32.499 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.500 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18ba08bf:::benchmark_data_smithi179_148205_object50466:head# 2024-02-09T17:57:32.500 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.500 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18bb0726:::benchmark_data_smithi179_148205_object32634:head# 2024-02-09T17:57:32.500 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.500 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18bb5003:::benchmark_data_smithi179_148205_object47185:head# 2024-02-09T17:57:32.500 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.500 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18bb5065:::benchmark_data_smithi179_148205_object108495:head# 2024-02-09T17:57:32.500 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.500 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18bc1a9b:::benchmark_data_smithi179_148205_object100690:head# 2024-02-09T17:57:32.500 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.500 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18bc8d1a:::benchmark_data_smithi179_148205_object28916:head# 2024-02-09T17:57:32.500 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.501 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18bcd3c6:::benchmark_data_smithi179_148205_object105233:head# 2024-02-09T17:57:32.501 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.504 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18bcd3f3:::benchmark_data_smithi179_148205_object76178:head# 2024-02-09T17:57:32.504 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.504 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18bcfd00:::benchmark_data_smithi179_148205_object107066:head# 2024-02-09T17:57:32.504 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.504 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18bd536d:::benchmark_data_smithi179_148205_object44827:head# 2024-02-09T17:57:32.504 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.504 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18bda184:::benchmark_data_smithi179_148205_object58116:head# 2024-02-09T17:57:32.504 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.504 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18bdbd73:::benchmark_data_smithi179_148205_object67024:head# 2024-02-09T17:57:32.504 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.504 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18bdeadd:::benchmark_data_smithi179_148205_object44580:head# 2024-02-09T17:57:32.505 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.505 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18bdf1a2:::benchmark_data_smithi179_148205_object4245:head# 2024-02-09T17:57:32.505 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.505 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18c00895:::benchmark_data_smithi179_148205_object35696:head# 2024-02-09T17:57:32.505 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.506 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18c0c54b:::benchmark_data_smithi179_148205_object80867:head# 2024-02-09T17:57:32.506 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.506 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18c0ee35:::benchmark_data_smithi179_148205_object24531:head# 2024-02-09T17:57:32.506 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.506 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18c0f1c8:::benchmark_data_smithi179_148205_object76296:head# 2024-02-09T17:57:32.506 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.506 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18c13b52:::benchmark_data_smithi179_148205_object84737:head# 2024-02-09T17:57:32.506 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.506 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18c24e05:::benchmark_data_smithi179_148205_object56897:head# 2024-02-09T17:57:32.506 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.506 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18c2ace0:::benchmark_data_smithi179_148205_object7631:head# 2024-02-09T17:57:32.506 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.508 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18c2f714:::benchmark_data_smithi179_148205_object94542:head# 2024-02-09T17:57:32.508 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.508 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18c41347:::benchmark_data_smithi179_148205_object89766:head# 2024-02-09T17:57:32.508 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.508 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18c42808:::benchmark_data_smithi179_148205_object81997:head# 2024-02-09T17:57:32.508 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.508 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18c4bb84:::benchmark_data_smithi179_148205_object95798:head# 2024-02-09T17:57:32.508 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.508 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18c50c94:::benchmark_data_smithi179_148205_object87607:head# 2024-02-09T17:57:32.508 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.508 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18c58162:::benchmark_data_smithi179_148205_object41040:head# 2024-02-09T17:57:32.508 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.509 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18c5a36b:::benchmark_data_smithi179_148205_object74212:head# 2024-02-09T17:57:32.509 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.509 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18c64abe:::benchmark_data_smithi179_148205_object90935:head# 2024-02-09T17:57:32.509 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.511 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18c695e3:::benchmark_data_smithi179_148205_object93588:head# 2024-02-09T17:57:32.511 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.511 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18c7f0a0:::benchmark_data_smithi179_148205_object29606:head# 2024-02-09T17:57:32.511 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.511 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18c89b29:::benchmark_data_smithi179_148205_object66150:head# 2024-02-09T17:57:32.511 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.511 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18c8c689:::benchmark_data_smithi179_148205_object79521:head# 2024-02-09T17:57:32.511 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.511 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18c998a9:::benchmark_data_smithi179_148205_object31429:head# 2024-02-09T17:57:32.511 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.511 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18ca9acf:::benchmark_data_smithi179_148205_object12523:head# 2024-02-09T17:57:32.512 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.512 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18cb12f6:::benchmark_data_smithi179_148205_object35754:head# 2024-02-09T17:57:32.512 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.512 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18cb82f1:::benchmark_data_smithi179_148205_object67848:head# 2024-02-09T17:57:32.512 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.512 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18cc4a4e:::benchmark_data_smithi179_148205_object76286:head# 2024-02-09T17:57:32.512 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.512 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18cc4f12:::benchmark_data_smithi179_148205_object103092:head# 2024-02-09T17:57:32.512 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.512 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18ccbcc0:::benchmark_data_smithi179_148205_object78689:head# 2024-02-09T17:57:32.512 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.515 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18cd556a:::benchmark_data_smithi179_148205_object52805:head# 2024-02-09T17:57:32.516 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.516 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18cda2b5:::benchmark_data_smithi179_148205_object22529:head# 2024-02-09T17:57:32.516 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.516 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18ce5ccf:::benchmark_data_smithi179_148205_object107080:head# 2024-02-09T17:57:32.516 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.516 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18cea345:::benchmark_data_smithi179_148205_object31789:head# 2024-02-09T17:57:32.516 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.516 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18cfccdf:::benchmark_data_smithi179_148205_object63968:head# 2024-02-09T17:57:32.516 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.516 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18d0734b:::benchmark_data_smithi179_148205_object64686:head# 2024-02-09T17:57:32.516 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.516 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18d0ff89:::benchmark_data_smithi179_148205_object1507:head# 2024-02-09T17:57:32.516 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.516 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18d13575:::benchmark_data_smithi179_148205_object54939:head# 2024-02-09T17:57:32.517 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.517 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18d1b027:::benchmark_data_smithi179_148205_object16587:head# 2024-02-09T17:57:32.517 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.517 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18d1deab:::benchmark_data_smithi179_148205_object91673:head# 2024-02-09T17:57:32.517 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.517 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18d23c61:::benchmark_data_smithi179_148205_object77956:head# 2024-02-09T17:57:32.517 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.517 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18d295d9:::benchmark_data_smithi179_148205_object80889:head# 2024-02-09T17:57:32.517 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.518 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18d30bf6:::benchmark_data_smithi179_148205_object101879:head# 2024-02-09T17:57:32.518 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.518 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18d31bce:::benchmark_data_smithi179_148205_object51889:head# 2024-02-09T17:57:32.518 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.518 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18d674ba:::benchmark_data_smithi179_148205_object49260:head# 2024-02-09T17:57:32.518 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.520 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18d7aec3:::benchmark_data_smithi179_148205_object81811:head# 2024-02-09T17:57:32.520 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.521 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18d8186e:::benchmark_data_smithi179_148205_object31503:head# 2024-02-09T17:57:32.521 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.521 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18d825b3:::benchmark_data_smithi179_148205_object22067:head# 2024-02-09T17:57:32.521 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.522 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18d84d89:::benchmark_data_smithi179_148205_object16290:head# 2024-02-09T17:57:32.522 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.522 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18d9a644:::benchmark_data_smithi179_148205_object52776:head# 2024-02-09T17:57:32.522 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.522 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18da29f7:::benchmark_data_smithi179_148205_object24669:head# 2024-02-09T17:57:32.522 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.523 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18da4645:::benchmark_data_smithi179_148205_object74155:head# 2024-02-09T17:57:32.523 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.523 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18daa348:::benchmark_data_smithi179_148205_object10365:head# 2024-02-09T17:57:32.523 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.523 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18dacb9c:::benchmark_data_smithi179_148205_object76189:head# 2024-02-09T17:57:32.523 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.523 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18db470a:::benchmark_data_smithi179_148205_object34275:head# 2024-02-09T17:57:32.523 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.524 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18dc5be6:::benchmark_data_smithi179_148205_object32698:head# 2024-02-09T17:57:32.524 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.525 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18dd8d35:::benchmark_data_smithi179_148205_object4502:head# 2024-02-09T17:57:32.525 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.525 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18dda735:::benchmark_data_smithi179_148205_object102554:head# 2024-02-09T17:57:32.525 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.525 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18de7cdd:::benchmark_data_smithi179_148205_object43506:head# 2024-02-09T17:57:32.525 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.525 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18df02b1:::benchmark_data_smithi179_148205_object49727:head# 2024-02-09T17:57:32.525 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.525 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18df27da:::benchmark_data_smithi179_148205_object101942:head# 2024-02-09T17:57:32.525 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.526 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18df28ca:::benchmark_data_smithi179_148205_object15988:head# 2024-02-09T17:57:32.526 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.526 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18df2931:::benchmark_data_smithi179_148205_object39543:head# 2024-02-09T17:57:32.526 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.527 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18df6840:::benchmark_data_smithi179_148205_object63107:head# 2024-02-09T17:57:32.527 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.527 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18df8b96:::benchmark_data_smithi179_148205_object80685:head# 2024-02-09T17:57:32.528 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.528 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18e01e0e:::benchmark_data_smithi179_148205_object88161:head# 2024-02-09T17:57:32.528 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.529 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18e136a4:::benchmark_data_smithi179_148205_object1852:head# 2024-02-09T17:57:32.529 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.529 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18e234ab:::benchmark_data_smithi179_148205_object90210:head# 2024-02-09T17:57:32.529 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.529 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18e2aa31:::benchmark_data_smithi179_148205_object56202:head# 2024-02-09T17:57:32.529 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.529 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18e32fc3:::benchmark_data_smithi179_148205_object1553:head# 2024-02-09T17:57:32.529 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.530 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18e39bdf:::benchmark_data_smithi179_148205_object27770:head# 2024-02-09T17:57:32.530 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.530 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18e4660d:::benchmark_data_smithi179_148205_object42152:head# 2024-02-09T17:57:32.530 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.530 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18e48cfc:::benchmark_data_smithi179_148205_object68012:head# 2024-02-09T17:57:32.530 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.530 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18e49a51:::benchmark_data_smithi179_148205_object83568:head# 2024-02-09T17:57:32.530 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.530 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18e4a4f4:::benchmark_data_smithi179_148205_object68517:head# 2024-02-09T17:57:32.530 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.534 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18e4ad58:::benchmark_data_smithi179_148205_object5453:head# 2024-02-09T17:57:32.534 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.534 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18e4b1e0:::benchmark_data_smithi179_148205_object41804:head# 2024-02-09T17:57:32.534 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.534 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18e5500c:::benchmark_data_smithi179_148205_object109366:head# 2024-02-09T17:57:32.535 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.535 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18e5db8a:::benchmark_data_smithi179_148205_object18426:head# 2024-02-09T17:57:32.535 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.535 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18e63417:::benchmark_data_smithi179_148205_object84645:head# 2024-02-09T17:57:32.535 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.535 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18e6bd0f:::benchmark_data_smithi179_148205_object104247:head# 2024-02-09T17:57:32.535 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.535 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18e837ee:::benchmark_data_smithi179_148205_object38361:head# 2024-02-09T17:57:32.535 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.535 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18e938c6:::benchmark_data_smithi179_148205_object97572:head# 2024-02-09T17:57:32.535 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.535 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18e9b867:::benchmark_data_smithi179_148205_object19665:head# 2024-02-09T17:57:32.535 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.535 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18e9bdc9:::benchmark_data_smithi179_148205_object87638:head# 2024-02-09T17:57:32.535 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.535 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18ea371b:::benchmark_data_smithi179_148205_object82585:head# 2024-02-09T17:57:32.536 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.536 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18ea8385:::benchmark_data_smithi179_148205_object91052:head# 2024-02-09T17:57:32.536 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.536 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18eaaf28:::benchmark_data_smithi179_148205_object80797:head# 2024-02-09T17:57:32.536 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.536 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18eb7200:::benchmark_data_smithi179_148205_object26037:head# 2024-02-09T17:57:32.536 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.536 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18eb73d9:::benchmark_data_smithi179_148205_object70398:head# 2024-02-09T17:57:32.536 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.536 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18ebff87:::benchmark_data_smithi179_148205_object76960:head# 2024-02-09T17:57:32.536 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.536 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18ec9105:::benchmark_data_smithi179_148205_object63044:head# 2024-02-09T17:57:32.536 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.538 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18ecafb3:::benchmark_data_smithi179_148205_object85544:head# 2024-02-09T17:57:32.539 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.539 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18ecf122:::benchmark_data_smithi179_148205_object78256:head# 2024-02-09T17:57:32.539 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.539 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18edbf3c:::benchmark_data_smithi179_148205_object3955:head# 2024-02-09T17:57:32.539 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.539 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18ede791:::benchmark_data_smithi179_148205_object10782:head# 2024-02-09T17:57:32.539 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.539 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18ee1cb3:::benchmark_data_smithi179_148205_object48150:head# 2024-02-09T17:57:32.539 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.539 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18ee2445:::benchmark_data_smithi179_148205_object28206:head# 2024-02-09T17:57:32.539 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.539 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18ef48e2:::benchmark_data_smithi179_148205_object97367:head# 2024-02-09T17:57:32.539 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.539 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18efd52e:::benchmark_data_smithi179_148205_object101749:head# 2024-02-09T17:57:32.539 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.539 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18f0046e:::benchmark_data_smithi179_148205_object43668:head# 2024-02-09T17:57:32.540 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.540 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18f16029:::benchmark_data_smithi179_148205_object78646:head# 2024-02-09T17:57:32.540 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.540 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18f188f1:::benchmark_data_smithi179_148205_object100750:head# 2024-02-09T17:57:32.540 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.540 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18f2f257:::benchmark_data_smithi179_148205_object89743:head# 2024-02-09T17:57:32.540 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.541 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18f3fb84:::benchmark_data_smithi179_148205_object64625:head# 2024-02-09T17:57:32.541 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.541 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18f44db5:::benchmark_data_smithi179_148205_object17998:head# 2024-02-09T17:57:32.541 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.541 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18f460b1:::benchmark_data_smithi179_148205_object14297:head# 2024-02-09T17:57:32.541 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.541 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18f4ce39:::benchmark_data_smithi179_148205_object27481:head# 2024-02-09T17:57:32.541 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.541 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18f540f4:::benchmark_data_smithi179_148205_object84038:head# 2024-02-09T17:57:32.541 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.543 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18f56575:::benchmark_data_smithi179_148205_object108672:head# 2024-02-09T17:57:32.543 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.543 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18f5fbf5:::benchmark_data_smithi179_148205_object20810:head# 2024-02-09T17:57:32.543 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.543 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18f845dc:::benchmark_data_smithi179_148205_object28807:head# 2024-02-09T17:57:32.543 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.543 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18f8fc42:::benchmark_data_smithi179_148205_object25116:head# 2024-02-09T17:57:32.543 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.544 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18f8fd70:::benchmark_data_smithi179_148205_object1481:head# 2024-02-09T17:57:32.544 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.544 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18f96aac:::benchmark_data_smithi179_148205_object107062:head# 2024-02-09T17:57:32.544 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.544 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18f9a5c4:::benchmark_data_smithi179_148205_object107500:head# 2024-02-09T17:57:32.544 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.544 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18f9fc07:::benchmark_data_smithi179_148205_object49173:head# 2024-02-09T17:57:32.544 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.547 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18fa83b9:::benchmark_data_smithi179_148205_object39838:head# 2024-02-09T17:57:32.547 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.547 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18fb5d23:::benchmark_data_smithi179_148205_object48204:head# 2024-02-09T17:57:32.547 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.548 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18fb5f61:::benchmark_data_smithi179_148205_object2521:head# 2024-02-09T17:57:32.549 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.549 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18fb885a:::benchmark_data_smithi179_148205_object23182:head# 2024-02-09T17:57:32.549 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.549 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18fc1088:::benchmark_data_smithi179_148205_object20675:head# 2024-02-09T17:57:32.549 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.549 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18fd69d1:::benchmark_data_smithi179_148205_object66861:head# 2024-02-09T17:57:32.549 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.549 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18fdc4d5:::benchmark_data_smithi179_148205_object12881:head# 2024-02-09T17:57:32.549 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.549 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18fdef66:::benchmark_data_smithi179_148205_object49541:head# 2024-02-09T17:57:32.549 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.549 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18ff48ba:::benchmark_data_smithi179_148205_object93918:head# 2024-02-09T17:57:32.549 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.549 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18ff8cc7:::benchmark_data_smithi179_148205_object4593:head# 2024-02-09T17:57:32.550 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.551 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:18ffd79a:::benchmark_data_smithi179_148205_object58805:head# 2024-02-09T17:57:32.551 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.551 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:190135de:::benchmark_data_smithi179_148205_object103317:head# 2024-02-09T17:57:32.551 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.551 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:190195e7:::benchmark_data_smithi179_148205_object36638:head# 2024-02-09T17:57:32.551 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.551 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19021b3d:::benchmark_data_smithi179_148205_object81123:head# 2024-02-09T17:57:32.551 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.551 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19022eb9:::benchmark_data_smithi179_148205_object18780:head# 2024-02-09T17:57:32.551 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.554 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:57:32.555 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:190383e6:::benchmark_data_smithi179_148205_object31137:head# 2024-02-09T17:57:32.555 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.557 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1903cb7b:::benchmark_data_smithi179_148205_object43925:head# 2024-02-09T17:57:32.557 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.557 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:190415b1:::benchmark_data_smithi179_148205_object108792:head# 2024-02-09T17:57:32.557 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.557 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:190537e0:::benchmark_data_smithi179_148205_object39044:head# 2024-02-09T17:57:32.557 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.557 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19058bbd:::benchmark_data_smithi179_148205_object17905:head# 2024-02-09T17:57:32.558 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.558 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1906e3ff:::benchmark_data_smithi179_148205_object43260:head# 2024-02-09T17:57:32.558 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.558 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19070393:::benchmark_data_smithi179_148205_object80309:head# 2024-02-09T17:57:32.558 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.558 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1907431d:::benchmark_data_smithi179_148205_object56047:head# 2024-02-09T17:57:32.558 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.558 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19074578:::benchmark_data_smithi179_148205_object49821:head# 2024-02-09T17:57:32.558 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.558 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19076e8e:::benchmark_data_smithi179_148205_object98518:head# 2024-02-09T17:57:32.558 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.558 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19076f14:::benchmark_data_smithi179_148205_object89563:head# 2024-02-09T17:57:32.558 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.558 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19095db3:::benchmark_data_smithi179_148205_object45702:head# 2024-02-09T17:57:32.558 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.562 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1909634a:::benchmark_data_smithi179_148205_object104719:head# 2024-02-09T17:57:32.562 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.562 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19099acd:::benchmark_data_smithi179_148205_object69456:head# 2024-02-09T17:57:32.562 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.562 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:190ab7c3:::benchmark_data_smithi179_148205_object30304:head# 2024-02-09T17:57:32.562 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.562 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:190b7619:::benchmark_data_smithi179_148205_object108379:head# 2024-02-09T17:57:32.562 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.562 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:190bded1:::benchmark_data_smithi179_148205_object40690:head# 2024-02-09T17:57:32.563 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.563 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:190c20c7:::benchmark_data_smithi179_148205_object33528:head# 2024-02-09T17:57:32.563 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.563 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:190e12e2:::benchmark_data_smithi179_148205_object67718:head# 2024-02-09T17:57:32.563 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.563 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:190facad:::benchmark_data_smithi179_148205_object94829:head# 2024-02-09T17:57:32.563 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.563 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1910ca60:::benchmark_data_smithi179_148205_object12820:head# 2024-02-09T17:57:32.563 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.563 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19124533:::benchmark_data_smithi179_148205_object37235:head# 2024-02-09T17:57:32.563 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.563 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:191265ed:::benchmark_data_smithi179_148205_object44999:head# 2024-02-09T17:57:32.563 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.563 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1914a310:::benchmark_data_smithi179_148205_object46102:head# 2024-02-09T17:57:32.564 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.564 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1916df60:::benchmark_data_smithi179_148205_object3964:head# 2024-02-09T17:57:32.564 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.564 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19171ba1:::benchmark_data_smithi179_148205_object9138:head# 2024-02-09T17:57:32.564 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.564 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1917da5a:::benchmark_data_smithi179_148205_object27072:head# 2024-02-09T17:57:32.564 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.564 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1917e09a:::benchmark_data_smithi179_148205_object24270:head# 2024-02-09T17:57:32.564 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.564 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19184fc0:::benchmark_data_smithi179_148205_object85323:head# 2024-02-09T17:57:32.564 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.564 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19193d6d:::benchmark_data_smithi179_148205_object9730:head# 2024-02-09T17:57:32.564 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.566 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:191a32f3:::benchmark_data_smithi179_148205_object8907:head# 2024-02-09T17:57:32.566 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.566 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:191a3985:::benchmark_data_smithi179_148205_object19554:head# 2024-02-09T17:57:32.566 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.572 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:191a90dc:::benchmark_data_smithi179_148205_object41534:head# 2024-02-09T17:57:32.572 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.575 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:191c126c:::benchmark_data_smithi179_148205_object71448:head# 2024-02-09T17:57:32.575 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.575 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:191c2482:::benchmark_data_smithi179_148205_object76174:head# 2024-02-09T17:57:32.575 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.575 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:191cf588:::benchmark_data_smithi179_148205_object104343:head# 2024-02-09T17:57:32.575 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.576 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:191e0b7e:::benchmark_data_smithi179_148205_object51681:head# 2024-02-09T17:57:32.576 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.576 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:191eb6d2:::benchmark_data_smithi179_148205_object58945:head# 2024-02-09T17:57:32.576 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.576 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:191eccac:::benchmark_data_smithi179_148205_object106617:head# 2024-02-09T17:57:32.577 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.577 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:191ee27c:::benchmark_data_smithi179_148205_object62198:head# 2024-02-09T17:57:32.577 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.577 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:191eeebc:::benchmark_data_smithi179_148205_object39285:head# 2024-02-09T17:57:32.577 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.578 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:191f770d:::benchmark_data_smithi179_148205_object69195:head# 2024-02-09T17:57:32.578 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.578 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1920f40a:::benchmark_data_smithi179_148205_object24324:head# 2024-02-09T17:57:32.578 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.578 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1921b116:::benchmark_data_smithi179_148205_object12355:head# 2024-02-09T17:57:32.578 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.578 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1921be01:::benchmark_data_smithi179_148205_object52384:head# 2024-02-09T17:57:32.578 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.578 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1921ed26:::benchmark_data_smithi179_148205_object24790:head# 2024-02-09T17:57:32.578 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.578 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19223279:::benchmark_data_smithi179_148205_object74486:head# 2024-02-09T17:57:32.579 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.579 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1922f566:::benchmark_data_smithi179_148205_object25140:head# 2024-02-09T17:57:32.579 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.579 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:192303d4:::benchmark_data_smithi179_148205_object6664:head# 2024-02-09T17:57:32.579 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.583 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19236aa9:::benchmark_data_smithi179_148205_object76736:head# 2024-02-09T17:57:32.584 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.584 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1923bda7:::benchmark_data_smithi179_148205_object66108:head# 2024-02-09T17:57:32.584 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.584 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:192598cc:::benchmark_data_smithi179_148205_object4779:head# 2024-02-09T17:57:32.584 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.584 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19268679:::benchmark_data_smithi179_148205_object61591:head# 2024-02-09T17:57:32.584 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.585 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1926c55e:::benchmark_data_smithi179_148205_object6720:head# 2024-02-09T17:57:32.585 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.585 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1927151a:::benchmark_data_smithi179_148205_object76892:head# 2024-02-09T17:57:32.585 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.585 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1928292c:::benchmark_data_smithi179_148205_object32661:head# 2024-02-09T17:57:32.585 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.585 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19289bfa:::benchmark_data_smithi179_148205_object94850:head# 2024-02-09T17:57:32.585 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.586 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1929e902:::benchmark_data_smithi179_148205_object101837:head# 2024-02-09T17:57:32.586 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.586 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:192a8bd0:::benchmark_data_smithi179_148205_object27367:head# 2024-02-09T17:57:32.586 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.586 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:192ad364:::benchmark_data_smithi179_148205_object47875:head# 2024-02-09T17:57:32.586 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.586 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:192b7e3b:::benchmark_data_smithi179_148205_object68932:head# 2024-02-09T17:57:32.586 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.587 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:192c45bc:::benchmark_data_smithi179_148205_object88841:head# 2024-02-09T17:57:32.587 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.587 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:192d297c:::benchmark_data_smithi179_148205_object42423:head# 2024-02-09T17:57:32.587 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.587 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:192e567a:::benchmark_data_smithi179_148205_object83237:head# 2024-02-09T17:57:32.587 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.587 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19306ce3:::benchmark_data_smithi179_148205_object16031:head# 2024-02-09T17:57:32.588 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.588 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1931d9b0:::benchmark_data_smithi179_148205_object35386:head# 2024-02-09T17:57:32.588 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.588 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19327025:::benchmark_data_smithi179_148205_object24002:head# 2024-02-09T17:57:32.588 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.588 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19327236:::benchmark_data_smithi179_148205_object4703:head# 2024-02-09T17:57:32.588 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.588 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1932ca64:::benchmark_data_smithi179_148205_object96826:head# 2024-02-09T17:57:32.589 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.589 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:193308e4:::benchmark_data_smithi179_148205_object108346:head# 2024-02-09T17:57:32.589 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.589 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19331262:::benchmark_data_smithi179_148205_object51952:head# 2024-02-09T17:57:32.589 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.589 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19332ee8:::benchmark_data_smithi179_148205_object8914:head# 2024-02-09T17:57:32.589 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.589 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1933d4cd:::benchmark_data_smithi179_148205_object38752:head# 2024-02-09T17:57:32.590 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.590 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19344c06:::benchmark_data_smithi179_148205_object26285:head# 2024-02-09T17:57:32.590 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.597 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:32.599 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1935c98b:::benchmark_data_smithi179_148205_object72363:head# 2024-02-09T17:57:32.599 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.599 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:193696af:::benchmark_data_smithi179_148205_object80425:head# 2024-02-09T17:57:32.599 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.599 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19388603:::benchmark_data_smithi179_148205_object88716:head# 2024-02-09T17:57:32.599 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.599 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1938f21a:::benchmark_data_smithi179_148205_object72581:head# 2024-02-09T17:57:32.600 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.600 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:193923cf:::benchmark_data_smithi179_148205_object39421:head# 2024-02-09T17:57:32.600 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.600 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:193bb003:::benchmark_data_smithi179_148205_object34182:head# 2024-02-09T17:57:32.600 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.600 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:193bb430:::benchmark_data_smithi179_148205_object88086:head# 2024-02-09T17:57:32.600 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.600 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:193d3875:::benchmark_data_smithi179_148205_object49287:head# 2024-02-09T17:57:32.600 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.601 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:193d75da:::benchmark_data_smithi179_148205_object21594:head# 2024-02-09T17:57:32.601 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.601 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:193dd204:::benchmark_data_smithi179_148205_object14088:head# 2024-02-09T17:57:32.601 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.601 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:193e1657:::benchmark_data_smithi179_148205_object31214:head# 2024-02-09T17:57:32.601 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.601 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:193ea658:::benchmark_data_smithi179_148205_object15830:head# 2024-02-09T17:57:32.601 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.601 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:193f88ff:::benchmark_data_smithi179_148205_object9909:head# 2024-02-09T17:57:32.601 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.601 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1940ed04:::benchmark_data_smithi179_148205_object67417:head# 2024-02-09T17:57:32.601 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.601 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1941a5d4:::benchmark_data_smithi179_148205_object7998:head# 2024-02-09T17:57:32.601 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.601 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19422ee9:::benchmark_data_smithi179_148205_object97379:head# 2024-02-09T17:57:32.602 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.602 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:194247aa:::benchmark_data_smithi179_148205_object60256:head# 2024-02-09T17:57:32.602 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.602 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1942b350:::benchmark_data_smithi179_148205_object67917:head# 2024-02-09T17:57:32.602 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.602 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1942f14b:::benchmark_data_smithi179_148205_object92132:head# 2024-02-09T17:57:32.602 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.602 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19432a44:::benchmark_data_smithi179_148205_object87044:head# 2024-02-09T17:57:32.602 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.602 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:194467ea:::benchmark_data_smithi179_148205_object103210:head# 2024-02-09T17:57:32.602 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.602 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1944b762:::benchmark_data_smithi179_148205_object73382:head# 2024-02-09T17:57:32.602 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.602 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1944d258:::benchmark_data_smithi179_148205_object86259:head# 2024-02-09T17:57:32.602 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.602 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:194556c2:::benchmark_data_smithi179_148205_object74983:head# 2024-02-09T17:57:32.603 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.603 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19457eeb:::benchmark_data_smithi179_148205_object6312:head# 2024-02-09T17:57:32.603 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.603 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1945ced0:::benchmark_data_smithi179_148205_object27337:head# 2024-02-09T17:57:32.603 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.603 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1947faff:::benchmark_data_smithi179_148205_object69668:head# 2024-02-09T17:57:32.603 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.603 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19481773:::benchmark_data_smithi179_148205_object34538:head# 2024-02-09T17:57:32.603 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.603 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1948d98b:::benchmark_data_smithi179_148205_object47063:head# 2024-02-09T17:57:32.603 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.603 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1949ccdf:::benchmark_data_smithi179_148205_object39226:head# 2024-02-09T17:57:32.603 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.603 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:194a49d5:::benchmark_data_smithi179_148205_object40304:head# 2024-02-09T17:57:32.603 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.603 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:194aec15:::benchmark_data_smithi179_148205_object29455:head# 2024-02-09T17:57:32.604 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.604 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:194b75d6:::benchmark_data_smithi179_148205_object24861:head# 2024-02-09T17:57:32.604 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.604 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:194ca55c:::benchmark_data_smithi179_148205_object97177:head# 2024-02-09T17:57:32.604 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.604 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:194ca863:::benchmark_data_smithi179_148205_object5724:head# 2024-02-09T17:57:32.604 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.604 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:194d2adb:::benchmark_data_smithi179_148205_object43441:head# 2024-02-09T17:57:32.604 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.604 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:194d8dd2:::benchmark_data_smithi179_148205_object58263:head# 2024-02-09T17:57:32.604 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.604 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:194e2cb2:::benchmark_data_smithi179_148205_object13456:head# 2024-02-09T17:57:32.604 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.604 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:194ecf78:::benchmark_data_smithi179_148205_object22084:head# 2024-02-09T17:57:32.604 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.605 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:194f452d:::benchmark_data_smithi179_148205_object13705:head# 2024-02-09T17:57:32.605 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.605 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:194fb301:::benchmark_data_smithi179_148205_object73106:head# 2024-02-09T17:57:32.605 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.605 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:194ff6ab:::benchmark_data_smithi179_148205_object65618:head# 2024-02-09T17:57:32.605 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.605 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19501621:::benchmark_data_smithi179_148205_object86150:head# 2024-02-09T17:57:32.605 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.605 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1950342c:::benchmark_data_smithi179_148205_object75755:head# 2024-02-09T17:57:32.605 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.605 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1950fa1f:::benchmark_data_smithi179_148205_object97351:head# 2024-02-09T17:57:32.605 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.605 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1951d7cf:::benchmark_data_smithi179_148205_object25296:head# 2024-02-09T17:57:32.605 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.605 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19524a57:::benchmark_data_smithi179_148205_object56530:head# 2024-02-09T17:57:32.605 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.606 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1952bd80:::benchmark_data_smithi179_148205_object87529:head# 2024-02-09T17:57:32.606 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.606 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19534ffb:::benchmark_data_smithi179_148205_object67574:head# 2024-02-09T17:57:32.606 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.606 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19546716:::benchmark_data_smithi179_148205_object87633:head# 2024-02-09T17:57:32.606 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.606 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1955cc64:::benchmark_data_smithi179_148205_object89755:head# 2024-02-09T17:57:32.606 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.606 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:195650da:::benchmark_data_smithi179_148205_object71142:head# 2024-02-09T17:57:32.606 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.606 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1956efa4:::benchmark_data_smithi179_148205_object41146:head# 2024-02-09T17:57:32.606 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.606 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:195707bb:::benchmark_data_smithi179_148205_object41889:head# 2024-02-09T17:57:32.606 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.606 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19576b6a:::benchmark_data_smithi179_148205_object106142:head# 2024-02-09T17:57:32.606 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.607 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1957af63:::benchmark_data_smithi179_148205_object13650:head# 2024-02-09T17:57:32.607 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.607 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1957f91f:::benchmark_data_smithi179_148205_object61712:head# 2024-02-09T17:57:32.607 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.612 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:195808c3:::benchmark_data_smithi179_148205_object49462:head# 2024-02-09T17:57:32.613 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.613 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19582273:::benchmark_data_smithi179_148205_object17597:head# 2024-02-09T17:57:32.613 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.613 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19582b7f:::benchmark_data_smithi179_148205_object74992:head# 2024-02-09T17:57:32.613 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.613 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19585744:::benchmark_data_smithi179_148205_object71553:head# 2024-02-09T17:57:32.613 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.613 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:195869dd:::benchmark_data_smithi179_148205_object33613:head# 2024-02-09T17:57:32.613 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.613 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:195949b7:::benchmark_data_smithi179_148205_object95208:head# 2024-02-09T17:57:32.613 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.613 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:195a3afb:::benchmark_data_smithi179_148205_object27277:head# 2024-02-09T17:57:32.613 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.613 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:195a848c:::benchmark_data_smithi179_148205_object77883:head# 2024-02-09T17:57:32.614 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.614 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:195e10fa:::benchmark_data_smithi179_148205_object102869:head# 2024-02-09T17:57:32.614 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.614 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:195e4dc8:::benchmark_data_smithi179_148205_object91751:head# 2024-02-09T17:57:32.614 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.614 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:195f18e8:::benchmark_data_smithi179_148205_object79531:head# 2024-02-09T17:57:32.614 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.614 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1960171f:::benchmark_data_smithi179_148205_object63786:head# 2024-02-09T17:57:32.614 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.614 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19607c28:::benchmark_data_smithi179_148205_object6713:head# 2024-02-09T17:57:32.614 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.614 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1960eb6e:::benchmark_data_smithi179_148205_object20890:head# 2024-02-09T17:57:32.614 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.614 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19610833:::benchmark_data_smithi179_148205_object54227:head# 2024-02-09T17:57:32.614 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.615 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1964d83a:::benchmark_data_smithi179_148205_object102785:head# 2024-02-09T17:57:32.615 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.615 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19655461:::benchmark_data_smithi179_148205_object70956:head# 2024-02-09T17:57:32.615 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.615 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19659927:::benchmark_data_smithi179_148205_object38117:head# 2024-02-09T17:57:32.615 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.615 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19666b70:::benchmark_data_smithi179_148205_object57110:head# 2024-02-09T17:57:32.615 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.615 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:196698b6:::benchmark_data_smithi179_148205_object71204:head# 2024-02-09T17:57:32.615 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.615 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:196711f4:::benchmark_data_smithi179_148205_object27751:head# 2024-02-09T17:57:32.615 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.615 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19677e44:::benchmark_data_smithi179_148205_object62605:head# 2024-02-09T17:57:32.615 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.616 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19688e44:::benchmark_data_smithi179_148205_object5043:head# 2024-02-09T17:57:32.616 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.616 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1968e3aa:::benchmark_data_smithi179_148205_object95255:head# 2024-02-09T17:57:32.616 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.616 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1968f60d:::benchmark_data_smithi179_148205_object105985:head# 2024-02-09T17:57:32.616 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.616 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1968fdf3:::benchmark_data_smithi179_148205_object75525:head# 2024-02-09T17:57:32.616 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.616 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19691a6b:::benchmark_data_smithi179_148205_object74412:head# 2024-02-09T17:57:32.616 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.616 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19696246:::benchmark_data_smithi179_148205_object10742:head# 2024-02-09T17:57:32.616 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.616 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:196968e4:::benchmark_data_smithi179_148205_object45076:head# 2024-02-09T17:57:32.616 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.617 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:196a2ea6:::benchmark_data_smithi179_148205_object30919:head# 2024-02-09T17:57:32.617 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.617 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:196ae57b:::benchmark_data_smithi179_148205_object4118:head# 2024-02-09T17:57:32.617 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.617 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:196bbead:::benchmark_data_smithi179_148205_object69746:head# 2024-02-09T17:57:32.617 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.617 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:196c0919:::benchmark_data_smithi179_148205_object45415:head# 2024-02-09T17:57:32.617 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.617 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:196c5689:::benchmark_data_smithi179_148205_object81693:head# 2024-02-09T17:57:32.617 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.617 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:196e3486:::benchmark_data_smithi179_148205_object62473:head# 2024-02-09T17:57:32.617 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.617 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:196e3b1a:::benchmark_data_smithi179_148205_object35495:head# 2024-02-09T17:57:32.617 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.618 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:196e76f4:::benchmark_data_smithi179_148205_object104700:head# 2024-02-09T17:57:32.618 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.618 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:196ee438:::benchmark_data_smithi179_148205_object20011:head# 2024-02-09T17:57:32.618 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.618 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19701812:::benchmark_data_smithi179_148205_object63253:head# 2024-02-09T17:57:32.618 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.619 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1970d98a:::benchmark_data_smithi179_148205_object87752:head# 2024-02-09T17:57:32.619 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.619 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1970e28b:::benchmark_data_smithi179_148205_object42062:head# 2024-02-09T17:57:32.619 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.620 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19710bbb:::benchmark_data_smithi179_148205_object16522:head# 2024-02-09T17:57:32.620 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.620 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19713bc6:::benchmark_data_smithi179_148205_object87539:head# 2024-02-09T17:57:32.620 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.620 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:197177a0:::benchmark_data_smithi179_148205_object85707:head# 2024-02-09T17:57:32.620 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.620 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1971b5bf:::benchmark_data_smithi179_148205_object42775:head# 2024-02-09T17:57:32.620 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.620 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1972c7f5:::benchmark_data_smithi179_148205_object51796:head# 2024-02-09T17:57:32.620 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.620 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1972f1ea:::benchmark_data_smithi179_148205_object93908:head# 2024-02-09T17:57:32.620 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.621 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19731524:::benchmark_data_smithi179_148205_object41838:head# 2024-02-09T17:57:32.621 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.622 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:197333c7:::benchmark_data_smithi179_148205_object13746:head# 2024-02-09T17:57:32.622 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.623 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1973a94c:::benchmark_data_smithi179_148205_object43805:head# 2024-02-09T17:57:32.623 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.623 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1973e6b3:::benchmark_data_smithi179_148205_object92049:head# 2024-02-09T17:57:32.623 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.623 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1973e949:::benchmark_data_smithi179_148205_object27056:head# 2024-02-09T17:57:32.623 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.623 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:197486a7:::benchmark_data_smithi179_148205_object21862:head# 2024-02-09T17:57:32.623 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.623 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1974bce0:::benchmark_data_smithi179_148205_object94274:head# 2024-02-09T17:57:32.623 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.623 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1974ca2b:::benchmark_data_smithi179_148205_object44733:head# 2024-02-09T17:57:32.623 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.623 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1975511e:::benchmark_data_smithi179_148205_object77823:head# 2024-02-09T17:57:32.623 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.624 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19757ac6:::benchmark_data_smithi179_148205_object52187:head# 2024-02-09T17:57:32.624 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.624 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1975ce22:::benchmark_data_smithi179_148205_object53145:head# 2024-02-09T17:57:32.624 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.624 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19761987:::benchmark_data_smithi179_148205_object80701:head# 2024-02-09T17:57:32.624 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.624 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19771f7e:::benchmark_data_smithi179_148205_object41841:head# 2024-02-09T17:57:32.624 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.625 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:57:32.627 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:197725f6:::benchmark_data_smithi179_148205_object89048:head# 2024-02-09T17:57:32.627 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.627 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19778401:::benchmark_data_smithi179_148205_object18638:head# 2024-02-09T17:57:32.627 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.627 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1977b34c:::benchmark_data_smithi179_148205_object43941:head# 2024-02-09T17:57:32.627 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.627 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1977fc2a:::benchmark_data_smithi179_148205_object41546:head# 2024-02-09T17:57:32.627 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.627 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1978b16e:::benchmark_data_smithi179_148205_object99111:head# 2024-02-09T17:57:32.628 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.628 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1978c20a:::benchmark_data_smithi179_148205_object75294:head# 2024-02-09T17:57:32.628 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.628 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1978c74b:::benchmark_data_smithi179_148205_object88247:head# 2024-02-09T17:57:32.628 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.628 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:197941b2:::benchmark_data_smithi179_148205_object8166:head# 2024-02-09T17:57:32.628 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.628 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:197a122a:::benchmark_data_smithi179_148205_object70918:head# 2024-02-09T17:57:32.628 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.628 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:197ad6dd:::benchmark_data_smithi179_148205_object19155:head# 2024-02-09T17:57:32.628 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.628 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:197c6d2a:::benchmark_data_smithi179_148205_object60370:head# 2024-02-09T17:57:32.628 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.628 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:197cd524:::benchmark_data_smithi179_148205_object31451:head# 2024-02-09T17:57:32.629 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.629 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:197ddf01:::benchmark_data_smithi179_148205_object83488:head# 2024-02-09T17:57:32.629 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.629 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:197e91de:::benchmark_data_smithi179_148205_object13793:head# 2024-02-09T17:57:32.629 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.629 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:197f4c9d:::benchmark_data_smithi179_148205_object67887:head# 2024-02-09T17:57:32.629 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.629 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:197f8167:::benchmark_data_smithi179_148205_object69015:head# 2024-02-09T17:57:32.629 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:32.630 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:57:32.632 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:197f8a15:::benchmark_data_smithi179_148205_object69290:head# 2024-02-09T17:57:32.632 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.632 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19822997:::benchmark_data_smithi179_148205_object44221:head# 2024-02-09T17:57:32.632 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.632 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1982422b:::benchmark_data_smithi179_148205_object106126:head# 2024-02-09T17:57:32.632 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.632 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19826d0f:::benchmark_data_smithi179_148205_object84847:head# 2024-02-09T17:57:32.632 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.633 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1983175c:::benchmark_data_smithi179_148205_object86159:head# 2024-02-09T17:57:32.633 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.636 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:198344f3:::benchmark_data_smithi179_148205_object103455:head# 2024-02-09T17:57:32.636 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.636 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1983979e:::benchmark_data_smithi179_148205_object41989:head# 2024-02-09T17:57:32.636 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.637 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1985b485:::benchmark_data_smithi179_148205_object90963:head# 2024-02-09T17:57:32.637 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.637 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:198681d8:::benchmark_data_smithi179_148205_object48630:head# 2024-02-09T17:57:32.637 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.638 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19869305:::benchmark_data_smithi179_148205_object1030:head# 2024-02-09T17:57:32.638 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.638 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1986a72a:::benchmark_data_smithi179_148205_object12665:head# 2024-02-09T17:57:32.638 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.638 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1986fe9b:::benchmark_data_smithi179_148205_object22101:head# 2024-02-09T17:57:32.638 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.638 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1987aaf9:::benchmark_data_smithi179_148205_object34364:head# 2024-02-09T17:57:32.638 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.639 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1987dbcf:::benchmark_data_smithi179_148205_object39109:head# 2024-02-09T17:57:32.639 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.639 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19889422:::benchmark_data_smithi179_148205_object69707:head# 2024-02-09T17:57:32.639 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.641 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1988aebe:::benchmark_data_smithi179_148205_object51397:head# 2024-02-09T17:57:32.641 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.641 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1988cc95:::benchmark_data_smithi179_148205_object96080:head# 2024-02-09T17:57:32.641 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.641 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1989ad84:::benchmark_data_smithi179_148205_object103973:head# 2024-02-09T17:57:32.641 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.641 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:198a1413:::benchmark_data_smithi179_148205_object95959:head# 2024-02-09T17:57:32.641 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.641 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:198bf64f:::benchmark_data_smithi179_148205_object57162:head# 2024-02-09T17:57:32.641 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.641 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:198d2fae:::benchmark_data_smithi179_148205_object74767:head# 2024-02-09T17:57:32.641 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.641 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:198d74cd:::benchmark_data_smithi179_148205_object27550:head# 2024-02-09T17:57:32.641 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.642 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:198d91f8:::benchmark_data_smithi179_148205_object10968:head# 2024-02-09T17:57:32.642 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.642 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:198ee1d7:::benchmark_data_smithi179_148205_object36653:head# 2024-02-09T17:57:32.642 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.642 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:198f68a8:::benchmark_data_smithi179_148205_object44744:head# 2024-02-09T17:57:32.642 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.642 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:198f6ae5:::benchmark_data_smithi179_148205_object54285:head# 2024-02-09T17:57:32.642 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.645 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:198f9f95:::benchmark_data_smithi179_148205_object81013:head# 2024-02-09T17:57:32.646 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.646 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:199030b4:::benchmark_data_smithi179_148205_object100326:head# 2024-02-09T17:57:32.646 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.646 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19903e4f:::benchmark_data_smithi179_148205_object95027:head# 2024-02-09T17:57:32.646 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.646 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1990654a:::benchmark_data_smithi179_148205_object80181:head# 2024-02-09T17:57:32.646 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.646 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19914666:::benchmark_data_smithi179_148205_object109193:head# 2024-02-09T17:57:32.646 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.646 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1992924e:::benchmark_data_smithi179_148205_object24257:head# 2024-02-09T17:57:32.646 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.646 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1992e51b:::benchmark_data_smithi179_148205_object58331:head# 2024-02-09T17:57:32.646 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.646 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19932733:::benchmark_data_smithi179_148205_object92607:head# 2024-02-09T17:57:32.646 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.647 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19934af3:::benchmark_data_smithi179_148205_object89277:head# 2024-02-09T17:57:32.647 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.647 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19935868:::benchmark_data_smithi179_148205_object65449:head# 2024-02-09T17:57:32.647 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.647 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1994115e:::benchmark_data_smithi179_148205_object85599:head# 2024-02-09T17:57:32.647 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.647 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19951ecd:::benchmark_data_smithi179_148205_object47533:head# 2024-02-09T17:57:32.647 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.647 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19953e28:::benchmark_data_smithi179_148205_object47667:head# 2024-02-09T17:57:32.647 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.647 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1995cfd4:::benchmark_data_smithi179_148205_object84152:head# 2024-02-09T17:57:32.647 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.647 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19965fb9:::benchmark_data_smithi179_148205_object94021:head# 2024-02-09T17:57:32.647 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.647 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1996614d:::benchmark_data_smithi179_148205_object104394:head# 2024-02-09T17:57:32.647 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.648 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:199680dc:::benchmark_data_smithi179_148205_object100035:head# 2024-02-09T17:57:32.648 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.648 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1999be82:::benchmark_data_smithi179_148205_object103103:head# 2024-02-09T17:57:32.648 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.648 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1999cef0:::benchmark_data_smithi179_148205_object50338:head# 2024-02-09T17:57:32.648 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.648 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:199a454c:::benchmark_data_smithi179_148205_object14315:head# 2024-02-09T17:57:32.648 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.649 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:199bbad1:::benchmark_data_smithi179_148205_object31158:head# 2024-02-09T17:57:32.649 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.649 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:199c0141:::benchmark_data_smithi179_148205_object23392:head# 2024-02-09T17:57:32.649 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.649 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:199c1b32:::benchmark_data_smithi179_148205_object15693:head# 2024-02-09T17:57:32.649 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.652 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:199ca859:::benchmark_data_smithi179_148205_object84218:head# 2024-02-09T17:57:32.652 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.652 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:199d1713:::benchmark_data_smithi179_148205_object109636:head# 2024-02-09T17:57:32.652 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.652 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:199d41f6:::benchmark_data_smithi179_148205_object65127:head# 2024-02-09T17:57:32.652 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.652 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:199d8e5c:::benchmark_data_smithi179_148205_object73439:head# 2024-02-09T17:57:32.652 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.652 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:199e863b:::benchmark_data_smithi179_148205_object54491:head# 2024-02-09T17:57:32.653 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.653 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:199e9c6e:::benchmark_data_smithi179_148205_object52872:head# 2024-02-09T17:57:32.653 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.653 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:199ea8fa:::benchmark_data_smithi179_148205_object2555:head# 2024-02-09T17:57:32.653 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.653 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:199f20aa:::benchmark_data_smithi179_148205_object37896:head# 2024-02-09T17:57:32.653 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.653 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:199fc431:::benchmark_data_smithi179_148205_object3100:head# 2024-02-09T17:57:32.653 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.653 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19a03094:::benchmark_data_smithi179_148205_object41441:head# 2024-02-09T17:57:32.653 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.653 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19a0441d:::benchmark_data_smithi179_148205_object83057:head# 2024-02-09T17:57:32.653 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.653 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19a100b4:::benchmark_data_smithi179_148205_object27253:head# 2024-02-09T17:57:32.653 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.654 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19a15c31:::benchmark_data_smithi179_148205_object61502:head# 2024-02-09T17:57:32.654 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.654 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19a1a840:::benchmark_data_smithi179_148205_object99975:head# 2024-02-09T17:57:32.654 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.654 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19a20e7a:::benchmark_data_smithi179_148205_object86239:head# 2024-02-09T17:57:32.654 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.654 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19a30cdb:::benchmark_data_smithi179_148205_object12452:head# 2024-02-09T17:57:32.654 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.654 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19a33837:::benchmark_data_smithi179_148205_object94027:head# 2024-02-09T17:57:32.654 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.654 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19a43265:::benchmark_data_smithi179_148205_object90623:head# 2024-02-09T17:57:32.654 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.654 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19a4758c:::benchmark_data_smithi179_148205_object48127:head# 2024-02-09T17:57:32.654 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.654 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19a55721:::benchmark_data_smithi179_148205_object44201:head# 2024-02-09T17:57:32.655 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.655 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19a55ca5:::benchmark_data_smithi179_148205_object43435:head# 2024-02-09T17:57:32.655 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.655 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19a590e7:::benchmark_data_smithi179_148205_object24673:head# 2024-02-09T17:57:32.655 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.657 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19a66961:::benchmark_data_smithi179_148205_object98110:head# 2024-02-09T17:57:32.657 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.657 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19a6ab23:::benchmark_data_smithi179_148205_object22069:head# 2024-02-09T17:57:32.657 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.657 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19a6b59c:::benchmark_data_smithi179_148205_object78835:head# 2024-02-09T17:57:32.657 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.657 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19a76cfa:::benchmark_data_smithi179_148205_object16431:head# 2024-02-09T17:57:32.657 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.657 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19a85166:::benchmark_data_smithi179_148205_object96071:head# 2024-02-09T17:57:32.657 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.657 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19a8b0d0:::benchmark_data_smithi179_148205_object61208:head# 2024-02-09T17:57:32.658 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.658 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19a8d80e:::benchmark_data_smithi179_148205_object43670:head# 2024-02-09T17:57:32.658 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.658 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19a9ac5c:::benchmark_data_smithi179_148205_object48624:head# 2024-02-09T17:57:32.658 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.658 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19aa4afb:::benchmark_data_smithi179_148205_object15398:head# 2024-02-09T17:57:32.658 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.658 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19ab8d97:::benchmark_data_smithi179_148205_object84442:head# 2024-02-09T17:57:32.658 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.658 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19ae6592:::benchmark_data_smithi179_148205_object38627:head# 2024-02-09T17:57:32.658 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.662 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19af2f39:::benchmark_data_smithi179_148205_object99432:head# 2024-02-09T17:57:32.662 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.667 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19afdb02:::benchmark_data_smithi179_148205_object109160:head# 2024-02-09T17:57:32.667 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.668 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19b00896:::benchmark_data_smithi179_148205_object16069:head# 2024-02-09T17:57:32.668 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.668 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19b0a8d8:::benchmark_data_smithi179_148205_object14244:head# 2024-02-09T17:57:32.668 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.669 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19b138d9:::benchmark_data_smithi179_148205_object27093:head# 2024-02-09T17:57:32.669 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.671 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19b15003:::benchmark_data_smithi179_148205_object73159:head# 2024-02-09T17:57:32.672 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.672 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19b2b332:::benchmark_data_smithi179_148205_object86674:head# 2024-02-09T17:57:32.672 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.677 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19b3035e:::benchmark_data_smithi179_148205_object11464:head# 2024-02-09T17:57:32.677 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.678 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19b37004:::benchmark_data_smithi179_148205_object19860:head# 2024-02-09T17:57:32.678 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.678 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19b371c0:::benchmark_data_smithi179_148205_object98373:head# 2024-02-09T17:57:32.678 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.678 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19b411dd:::benchmark_data_smithi179_148205_object107293:head# 2024-02-09T17:57:32.678 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.678 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19b50498:::benchmark_data_smithi179_148205_object14062:head# 2024-02-09T17:57:32.678 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.678 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19b51e13:::benchmark_data_smithi179_148205_object35951:head# 2024-02-09T17:57:32.678 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.678 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19b54b16:::benchmark_data_smithi179_148205_object53727:head# 2024-02-09T17:57:32.678 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.678 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19b55ed1:::benchmark_data_smithi179_148205_object95516:head# 2024-02-09T17:57:32.678 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.679 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19b5abc2:::benchmark_data_smithi179_148205_object31547:head# 2024-02-09T17:57:32.679 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.679 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19b5db3f:::benchmark_data_smithi179_148205_object80615:head# 2024-02-09T17:57:32.679 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.679 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19b70a73:::benchmark_data_smithi179_148205_object49830:head# 2024-02-09T17:57:32.679 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.679 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19b73dbe:::benchmark_data_smithi179_148205_object25276:head# 2024-02-09T17:57:32.679 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.679 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19b84e5d:::benchmark_data_smithi179_148205_object69294:head# 2024-02-09T17:57:32.679 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.679 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19ba2502:::benchmark_data_smithi179_148205_object85694:head# 2024-02-09T17:57:32.679 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.679 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19ba7c56:::benchmark_data_smithi179_148205_object64975:head# 2024-02-09T17:57:32.679 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.680 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19bb26df:::benchmark_data_smithi179_148205_object71993:head# 2024-02-09T17:57:32.680 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.682 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19bb7b88:::benchmark_data_smithi179_148205_object38733:head# 2024-02-09T17:57:32.682 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.682 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19bb7e99:::benchmark_data_smithi179_148205_object91689:head# 2024-02-09T17:57:32.683 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.683 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19bc79af:::benchmark_data_smithi179_148205_object80823:head# 2024-02-09T17:57:32.683 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.683 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19bd4fd4:::benchmark_data_smithi179_148205_object92099:head# 2024-02-09T17:57:32.683 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.683 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19bd6174:::benchmark_data_smithi179_148205_object99327:head# 2024-02-09T17:57:32.683 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.683 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19be2a15:::benchmark_data_smithi179_148205_object1656:head# 2024-02-09T17:57:32.683 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.683 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19be5eab:::benchmark_data_smithi179_148205_object45122:head# 2024-02-09T17:57:32.683 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.683 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19bebc2d:::benchmark_data_smithi179_148205_object92389:head# 2024-02-09T17:57:32.683 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.683 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19bebe62:::benchmark_data_smithi179_148205_object25048:head# 2024-02-09T17:57:32.684 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.684 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19bf1f41:::benchmark_data_smithi179_148205_object79443:head# 2024-02-09T17:57:32.684 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.684 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19bfa769:::benchmark_data_smithi179_148205_object61620:head# 2024-02-09T17:57:32.684 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.684 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19c04d6a:::benchmark_data_smithi179_148205_object95061:head# 2024-02-09T17:57:32.684 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.684 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19c2750f:::benchmark_data_smithi179_148205_object69126:head# 2024-02-09T17:57:32.684 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.684 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19c31712:::benchmark_data_smithi179_148205_object7873:head# 2024-02-09T17:57:32.684 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.684 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19c344f3:::benchmark_data_smithi179_148205_object23970:head# 2024-02-09T17:57:32.684 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.684 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19c3de40:::benchmark_data_smithi179_148205_object101222:head# 2024-02-09T17:57:32.684 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.685 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19c58b5b:::benchmark_data_smithi179_148205_object30579:head# 2024-02-09T17:57:32.685 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.685 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19c666a2:::benchmark_data_smithi179_148205_object29602:head# 2024-02-09T17:57:32.685 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.687 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19c6b82a:::benchmark_data_smithi179_148205_object58015:head# 2024-02-09T17:57:32.687 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.687 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19c74721:::benchmark_data_smithi179_148205_object84224:head# 2024-02-09T17:57:32.687 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.687 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19c74fea:::benchmark_data_smithi179_148205_object8195:head# 2024-02-09T17:57:32.687 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.687 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19c75a48:::benchmark_data_smithi179_148205_object46397:head# 2024-02-09T17:57:32.687 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.688 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19c7f5ef:::benchmark_data_smithi179_148205_object49472:head# 2024-02-09T17:57:32.688 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.688 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19c9fb02:::benchmark_data_smithi179_148205_object33368:head# 2024-02-09T17:57:32.688 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.688 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19cae890:::benchmark_data_smithi179_148205_object73963:head# 2024-02-09T17:57:32.688 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.688 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19cba619:::benchmark_data_smithi179_148205_object84252:head# 2024-02-09T17:57:32.688 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.688 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19cc3b74:::benchmark_data_smithi179_148205_object74018:head# 2024-02-09T17:57:32.688 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.688 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19cd8667:::benchmark_data_smithi179_148205_object37671:head# 2024-02-09T17:57:32.688 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.688 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19ce3708:::benchmark_data_smithi179_148205_object8890:head# 2024-02-09T17:57:32.689 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.689 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19ce6683:::benchmark_data_smithi179_148205_object71803:head# 2024-02-09T17:57:32.689 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.689 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19cf7e0a:::benchmark_data_smithi179_148205_object5682:head# 2024-02-09T17:57:32.689 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.689 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19d04f37:::benchmark_data_smithi179_148205_object27792:head# 2024-02-09T17:57:32.689 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.689 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19d14025:::benchmark_data_smithi179_148205_object8778:head# 2024-02-09T17:57:32.689 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.689 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19d18713:::benchmark_data_smithi179_148205_object2705:head# 2024-02-09T17:57:32.689 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.689 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19d21a64:::benchmark_data_smithi179_148205_object43340:head# 2024-02-09T17:57:32.689 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.692 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19d27867:::benchmark_data_smithi179_148205_object33889:head# 2024-02-09T17:57:32.692 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.692 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19d38fe9:::benchmark_data_smithi179_148205_object60192:head# 2024-02-09T17:57:32.692 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.692 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19d4aa8d:::benchmark_data_smithi179_148205_object62207:head# 2024-02-09T17:57:32.692 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.692 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19d4d263:::benchmark_data_smithi179_148205_object82211:head# 2024-02-09T17:57:32.692 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.692 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19d50d8e:::benchmark_data_smithi179_148205_object45195:head# 2024-02-09T17:57:32.693 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.693 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19d544ca:::benchmark_data_smithi179_148205_object59215:head# 2024-02-09T17:57:32.693 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.693 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19d5f5ab:::benchmark_data_smithi179_148205_object9241:head# 2024-02-09T17:57:32.693 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.693 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19d69686:::benchmark_data_smithi179_148205_object34357:head# 2024-02-09T17:57:32.693 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.693 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19d6abd5:::benchmark_data_smithi179_148205_object94284:head# 2024-02-09T17:57:32.693 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.693 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19d79cc8:::benchmark_data_smithi179_148205_object97507:head# 2024-02-09T17:57:32.693 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.693 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19d79e6d:::benchmark_data_smithi179_148205_object47941:head# 2024-02-09T17:57:32.693 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.693 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19d83780:::benchmark_data_smithi179_148205_object26234:head# 2024-02-09T17:57:32.693 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.693 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19d86dd3:::benchmark_data_smithi179_148205_object53406:head# 2024-02-09T17:57:32.694 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.694 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19d8e34c:::benchmark_data_smithi179_148205_object28835:head# 2024-02-09T17:57:32.694 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.694 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19d954b1:::benchmark_data_smithi179_148205_object104446:head# 2024-02-09T17:57:32.694 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.694 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19db6ebd:::benchmark_data_smithi179_148205_object14989:head# 2024-02-09T17:57:32.694 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.694 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19db79ce:::benchmark_data_smithi179_148205_object88922:head# 2024-02-09T17:57:32.694 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.694 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19dc6d0a:::benchmark_data_smithi179_148205_object100022:head# 2024-02-09T17:57:32.694 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.697 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19dd105b:::benchmark_data_smithi179_148205_object47264:head# 2024-02-09T17:57:32.697 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.697 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19ddc889:::benchmark_data_smithi179_148205_object87646:head# 2024-02-09T17:57:32.697 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.697 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19de783a:::benchmark_data_smithi179_148205_object14309:head# 2024-02-09T17:57:32.698 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.698 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19e01507:::benchmark_data_smithi179_148205_object10346:head# 2024-02-09T17:57:32.698 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.698 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19e02010:::benchmark_data_smithi179_148205_object41791:head# 2024-02-09T17:57:32.698 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.698 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19e09234:::benchmark_data_smithi179_148205_object6788:head# 2024-02-09T17:57:32.698 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.698 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19e0a55c:::benchmark_data_smithi179_148205_object86549:head# 2024-02-09T17:57:32.698 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.698 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19e14730:::benchmark_data_smithi179_148205_object27226:head# 2024-02-09T17:57:32.698 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.698 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19e14bcc:::benchmark_data_smithi179_148205_object20228:head# 2024-02-09T17:57:32.698 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.698 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19e1bde9:::benchmark_data_smithi179_148205_object76285:head# 2024-02-09T17:57:32.698 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.699 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19e258b4:::benchmark_data_smithi179_148205_object13870:head# 2024-02-09T17:57:32.699 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.699 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19e30a14:::benchmark_data_smithi179_148205_object41510:head# 2024-02-09T17:57:32.699 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.699 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19e37b6f:::benchmark_data_smithi179_148205_object30114:head# 2024-02-09T17:57:32.699 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.699 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19e3a62d:::benchmark_data_smithi179_148205_object43740:head# 2024-02-09T17:57:32.699 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.699 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19e3e3e7:::benchmark_data_smithi179_148205_object94257:head# 2024-02-09T17:57:32.699 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.699 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19e3f165:::benchmark_data_smithi179_148205_object103498:head# 2024-02-09T17:57:32.699 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.699 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19e4d263:::benchmark_data_smithi179_148205_object78741:head# 2024-02-09T17:57:32.699 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.699 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19e4ea71:::benchmark_data_smithi179_148205_object89872:head# 2024-02-09T17:57:32.699 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.699 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19e50ef9:::benchmark_data_smithi179_148205_object27372:head# 2024-02-09T17:57:32.700 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.700 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19e51a0c:::benchmark_data_smithi179_148205_object31278:head# 2024-02-09T17:57:32.700 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.700 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19e56b65:::benchmark_data_smithi179_148205_object93996:head# 2024-02-09T17:57:32.700 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.703 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19e6deb9:::benchmark_data_smithi179_148205_object101643:head# 2024-02-09T17:57:32.703 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.704 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19e78aa0:::benchmark_data_smithi179_148205_object20069:head# 2024-02-09T17:57:32.704 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.704 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19e86533:::benchmark_data_smithi179_148205_object35734:head# 2024-02-09T17:57:32.704 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.704 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19e8b33d:::benchmark_data_smithi179_148205_object55514:head# 2024-02-09T17:57:32.704 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.705 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19e94552:::benchmark_data_smithi179_148205_object54766:head# 2024-02-09T17:57:32.705 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.706 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19ea8a8a:::benchmark_data_smithi179_148205_object88920:head# 2024-02-09T17:57:32.706 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.706 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19ea9374:::benchmark_data_smithi179_148205_object1867:head# 2024-02-09T17:57:32.706 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.706 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19eb91d8:::benchmark_data_smithi179_148205_object37647:head# 2024-02-09T17:57:32.706 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.706 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19ebfca6:::benchmark_data_smithi179_148205_object106808:head# 2024-02-09T17:57:32.706 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.706 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19eca4a1:::benchmark_data_smithi179_148205_object5917:head# 2024-02-09T17:57:32.706 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.706 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19ecd338:::benchmark_data_smithi179_148205_object91883:head# 2024-02-09T17:57:32.706 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.706 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19ed0a7a:::benchmark_data_smithi179_148205_object75385:head# 2024-02-09T17:57:32.706 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.710 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19ed4191:::benchmark_data_smithi179_148205_object70763:head# 2024-02-09T17:57:32.710 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.710 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19edcf15:::benchmark_data_smithi179_148205_object87578:head# 2024-02-09T17:57:32.710 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.710 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19ef5b40:::benchmark_data_smithi179_148205_object23494:head# 2024-02-09T17:57:32.710 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.710 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19efa1fe:::benchmark_data_smithi179_148205_object51193:head# 2024-02-09T17:57:32.710 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.710 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19efd461:::benchmark_data_smithi179_148205_object91167:head# 2024-02-09T17:57:32.710 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.710 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19f00a03:::benchmark_data_smithi179_148205_object96997:head# 2024-02-09T17:57:32.710 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.710 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19f11dac:::benchmark_data_smithi179_148205_object79945:head# 2024-02-09T17:57:32.710 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.711 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19f18a20:::benchmark_data_smithi179_148205_object39243:head# 2024-02-09T17:57:32.711 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.711 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19f199d3:::benchmark_data_smithi179_148205_object368:head# 2024-02-09T17:57:32.711 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.711 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19f1e9cb:::benchmark_data_smithi179_148205_object28965:head# 2024-02-09T17:57:32.711 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.711 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19f2ffbd:::benchmark_data_smithi179_148205_object93034:head# 2024-02-09T17:57:32.711 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.711 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19f31297:::benchmark_data_smithi179_148205_object96815:head# 2024-02-09T17:57:32.711 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.711 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19f54982:::benchmark_data_smithi179_148205_object106423:head# 2024-02-09T17:57:32.711 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.711 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19f5a61b:::benchmark_data_smithi179_148205_object47215:head# 2024-02-09T17:57:32.711 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.712 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19f5f3d5:::benchmark_data_smithi179_148205_object54840:head# 2024-02-09T17:57:32.712 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.712 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19f7474f:::benchmark_data_smithi179_148205_object54232:head# 2024-02-09T17:57:32.712 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.712 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19f76067:::benchmark_data_smithi179_148205_object9745:head# 2024-02-09T17:57:32.712 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.713 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19f93178:::benchmark_data_smithi179_148205_object46939:head# 2024-02-09T17:57:32.713 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.713 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19f93588:::benchmark_data_smithi179_148205_object77901:head# 2024-02-09T17:57:32.713 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.713 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19f94867:::benchmark_data_smithi179_148205_object109519:head# 2024-02-09T17:57:32.713 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.713 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19f96ed1:::benchmark_data_smithi179_148205_object24580:head# 2024-02-09T17:57:32.713 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.713 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19f998c5:::benchmark_data_smithi179_148205_object17973:head# 2024-02-09T17:57:32.713 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.715 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19fa1f51:::benchmark_data_smithi179_148205_object6201:head# 2024-02-09T17:57:32.715 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.715 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19fb00fa:::benchmark_data_smithi179_148205_object958:head# 2024-02-09T17:57:32.715 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.715 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19fb35cf:::benchmark_data_smithi179_148205_object9507:head# 2024-02-09T17:57:32.715 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.715 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19fcb7ee:::benchmark_data_smithi179_148205_object39795:head# 2024-02-09T17:57:32.716 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.716 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19fcf994:::benchmark_data_smithi179_148205_object31702:head# 2024-02-09T17:57:32.716 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.716 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19fd23ad:::benchmark_data_smithi179_148205_object109089:head# 2024-02-09T17:57:32.716 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.716 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19fd4a28:::benchmark_data_smithi179_148205_object105908:head# 2024-02-09T17:57:32.716 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.716 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19fd7ca1:::benchmark_data_smithi179_148205_object78124:head# 2024-02-09T17:57:32.716 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.716 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19fde30f:::benchmark_data_smithi179_148205_object69085:head# 2024-02-09T17:57:32.716 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.716 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19fe35a1:::benchmark_data_smithi179_148205_object49023:head# 2024-02-09T17:57:32.716 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.716 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:19fef2a8:::benchmark_data_smithi179_148205_object104114:head# 2024-02-09T17:57:32.717 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.718 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a0040d1:::benchmark_data_smithi179_148205_object97292:head# 2024-02-09T17:57:32.718 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.718 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a00dcbd:::benchmark_data_smithi179_148205_object80925:head# 2024-02-09T17:57:32.718 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.718 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a01c0ed:::benchmark_data_smithi179_148205_object101014:head# 2024-02-09T17:57:32.718 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.718 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a02b82e:::benchmark_data_smithi179_148205_object14561:head# 2024-02-09T17:57:32.718 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.718 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a02c65d:::benchmark_data_smithi179_148205_object23883:head# 2024-02-09T17:57:32.719 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.719 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a02c7c7:::benchmark_data_smithi179_148205_object81215:head# 2024-02-09T17:57:32.719 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.719 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a045b23:::benchmark_data_smithi179_148205_object43139:head# 2024-02-09T17:57:32.719 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.719 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a045d7d:::benchmark_data_smithi179_148205_object104686:head# 2024-02-09T17:57:32.719 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.719 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a056c63:::benchmark_data_smithi179_148205_object92885:head# 2024-02-09T17:57:32.719 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.728 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a0571bc:::benchmark_data_smithi179_148205_object6508:head# 2024-02-09T17:57:32.728 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.728 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a0623a5:::benchmark_data_smithi179_148205_object66616:head# 2024-02-09T17:57:32.728 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.728 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a06b3cf:::benchmark_data_smithi179_148205_object96502:head# 2024-02-09T17:57:32.728 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.729 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a06c851:::benchmark_data_smithi179_148205_object105789:head# 2024-02-09T17:57:32.729 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.729 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a06d7b2:::benchmark_data_smithi179_148205_object99039:head# 2024-02-09T17:57:32.729 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.729 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a085aaf:::benchmark_data_smithi179_148205_object86826:head# 2024-02-09T17:57:32.729 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.730 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a085f27:::benchmark_data_smithi179_148205_object40700:head# 2024-02-09T17:57:32.730 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.730 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a096b14:::benchmark_data_smithi179_148205_object30109:head# 2024-02-09T17:57:32.730 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.730 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a09e024:::benchmark_data_smithi179_148205_object34009:head# 2024-02-09T17:57:32.731 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.731 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a0aef1a:::benchmark_data_smithi179_148205_object2208:head# 2024-02-09T17:57:32.731 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.731 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a0be67b:::benchmark_data_smithi179_148205_object60648:head# 2024-02-09T17:57:32.731 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.731 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:57:32.732 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a0cc13e:::benchmark_data_smithi179_148205_object56480:head# 2024-02-09T17:57:32.732 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.732 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a0cc79f:::benchmark_data_smithi179_148205_object105639:head# 2024-02-09T17:57:32.732 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.732 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a0dac7d:::benchmark_data_smithi179_148205_object84103:head# 2024-02-09T17:57:32.732 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.733 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a0db117:::benchmark_data_smithi179_148205_object55814:head# 2024-02-09T17:57:32.733 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.734 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a0e661b:::benchmark_data_smithi179_148205_object27068:head# 2024-02-09T17:57:32.734 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.734 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a1162aa:::benchmark_data_smithi179_148205_object43415:head# 2024-02-09T17:57:32.734 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.734 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a119316:::benchmark_data_smithi179_148205_object29116:head# 2024-02-09T17:57:32.734 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.734 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a11cd08:::benchmark_data_smithi179_148205_object12696:head# 2024-02-09T17:57:32.734 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.735 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a120f8c:::benchmark_data_smithi179_148205_object53566:head# 2024-02-09T17:57:32.735 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.735 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a130207:::benchmark_data_smithi179_148205_object1257:head# 2024-02-09T17:57:32.735 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.735 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a13409f:::benchmark_data_smithi179_148205_object93468:head# 2024-02-09T17:57:32.735 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.736 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a1360b6:::benchmark_data_smithi179_148205_object13151:head# 2024-02-09T17:57:32.736 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.737 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a13962a:::benchmark_data_smithi179_148205_object22059:head# 2024-02-09T17:57:32.737 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.737 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a13b6c4:::benchmark_data_smithi179_148205_object4416:head# 2024-02-09T17:57:32.737 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.737 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a14f2c8:::benchmark_data_smithi179_148205_object68135:head# 2024-02-09T17:57:32.737 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.737 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a16847a:::benchmark_data_smithi179_148205_object88031:head# 2024-02-09T17:57:32.737 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.740 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a168591:::benchmark_data_smithi179_148205_object57940:head# 2024-02-09T17:57:32.740 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.740 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a16b4c5:::benchmark_data_smithi179_148205_object79868:head# 2024-02-09T17:57:32.741 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.741 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a16b94e:::benchmark_data_smithi179_148205_object61411:head# 2024-02-09T17:57:32.741 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.741 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a17f42d:::benchmark_data_smithi179_148205_object7756:head# 2024-02-09T17:57:32.741 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.741 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a189e1b:::benchmark_data_smithi179_148205_object73097:head# 2024-02-09T17:57:32.741 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.741 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a197f3d:::benchmark_data_smithi179_148205_object98201:head# 2024-02-09T17:57:32.741 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.741 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a19fa9e:::benchmark_data_smithi179_148205_object34173:head# 2024-02-09T17:57:32.741 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.742 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a1a2e6a:::benchmark_data_smithi179_148205_object25378:head# 2024-02-09T17:57:32.742 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.742 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a1b47f0:::benchmark_data_smithi179_148205_object92658:head# 2024-02-09T17:57:32.742 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.742 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a1ca8a3:::benchmark_data_smithi179_148205_object104803:head# 2024-02-09T17:57:32.742 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.742 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a1e75ae:::benchmark_data_smithi179_148205_object99823:head# 2024-02-09T17:57:32.742 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.743 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a1eaeaf:::benchmark_data_smithi179_148205_object97739:head# 2024-02-09T17:57:32.743 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.743 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a1f1d5c:::benchmark_data_smithi179_148205_object70448:head# 2024-02-09T17:57:32.743 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.747 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a1f93f5:::benchmark_data_smithi179_148205_object23538:head# 2024-02-09T17:57:32.747 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.747 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a1fed06:::benchmark_data_smithi179_148205_object96973:head# 2024-02-09T17:57:32.747 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.747 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a217b4a:::benchmark_data_smithi179_148205_object8339:head# 2024-02-09T17:57:32.747 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.747 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a21ee56:::benchmark_data_smithi179_148205_object13135:head# 2024-02-09T17:57:32.748 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.748 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a22002f:::benchmark_data_smithi179_148205_object81099:head# 2024-02-09T17:57:32.748 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.748 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a227986:::benchmark_data_smithi179_148205_object16254:head# 2024-02-09T17:57:32.748 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.748 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a238d0a:::benchmark_data_smithi179_148205_object28129:head# 2024-02-09T17:57:32.748 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.748 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a23bfa7:::benchmark_data_smithi179_148205_object87081:head# 2024-02-09T17:57:32.748 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.748 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a2494cd:::benchmark_data_smithi179_148205_object19594:head# 2024-02-09T17:57:32.748 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.748 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a24f6a5:::benchmark_data_smithi179_148205_object73168:head# 2024-02-09T17:57:32.749 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.749 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a2568d7:::benchmark_data_smithi179_148205_object84798:head# 2024-02-09T17:57:32.749 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.749 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a256b98:::benchmark_data_smithi179_148205_object78746:head# 2024-02-09T17:57:32.749 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.749 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a25c033:::benchmark_data_smithi179_148205_object20842:head# 2024-02-09T17:57:32.749 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.749 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a26b834:::benchmark_data_smithi179_148205_object3655:head# 2024-02-09T17:57:32.749 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.749 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a26c1d4:::benchmark_data_smithi179_148205_object109382:head# 2024-02-09T17:57:32.749 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.749 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a26e190:::benchmark_data_smithi179_148205_object50066:head# 2024-02-09T17:57:32.749 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.751 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a28882b:::benchmark_data_smithi179_148205_object54573:head# 2024-02-09T17:57:32.752 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.752 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a2945ea:::benchmark_data_smithi179_148205_object33184:head# 2024-02-09T17:57:32.752 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.753 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a295900:::benchmark_data_smithi179_148205_object102862:head# 2024-02-09T17:57:32.753 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.753 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a2a0300:::benchmark_data_smithi179_148205_object37721:head# 2024-02-09T17:57:32.753 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.753 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a2a533b:::benchmark_data_smithi179_148205_object12226:head# 2024-02-09T17:57:32.753 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.754 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a2b0c37:::benchmark_data_smithi179_148205_object25778:head# 2024-02-09T17:57:32.754 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.754 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a2c31e5:::benchmark_data_smithi179_148205_object82415:head# 2024-02-09T17:57:32.755 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.755 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a2dad94:::benchmark_data_smithi179_148205_object63098:head# 2024-02-09T17:57:32.755 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.755 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a2e67e0:::benchmark_data_smithi179_148205_object44939:head# 2024-02-09T17:57:32.755 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.755 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a2ed273:::benchmark_data_smithi179_148205_object2471:head# 2024-02-09T17:57:32.755 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.755 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a2f5ab9:::benchmark_data_smithi179_148205_object27197:head# 2024-02-09T17:57:32.755 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.755 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a2f7d49:::benchmark_data_smithi179_148205_object44295:head# 2024-02-09T17:57:32.755 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.756 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a3071a6:::benchmark_data_smithi179_148205_object7463:head# 2024-02-09T17:57:32.756 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.756 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a316158:::benchmark_data_smithi179_148205_object26431:head# 2024-02-09T17:57:32.756 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.756 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a3187f6:::benchmark_data_smithi179_148205_object61626:head# 2024-02-09T17:57:32.757 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.757 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a319b47:::benchmark_data_smithi179_148205_object54961:head# 2024-02-09T17:57:32.757 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.762 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a321889:::benchmark_data_smithi179_148205_object55062:head# 2024-02-09T17:57:32.762 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.762 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a32a71b:::benchmark_data_smithi179_148205_object105504:head# 2024-02-09T17:57:32.762 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.762 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a32be43:::benchmark_data_smithi179_148205_object21504:head# 2024-02-09T17:57:32.762 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.762 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a32f16d:::benchmark_data_smithi179_148205_object10450:head# 2024-02-09T17:57:32.762 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.763 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:32.765 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a3349ea:::benchmark_data_smithi179_148205_object35690:head# 2024-02-09T17:57:32.765 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.765 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a33f824:::benchmark_data_smithi179_148205_object12372:head# 2024-02-09T17:57:32.766 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.766 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a340c16:::benchmark_data_smithi179_148205_object107238:head# 2024-02-09T17:57:32.766 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.766 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a3413c0:::benchmark_data_smithi179_148205_object20732:head# 2024-02-09T17:57:32.766 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.766 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a344676:::benchmark_data_smithi179_148205_object79132:head# 2024-02-09T17:57:32.766 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.766 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a35826c:::benchmark_data_smithi179_148205_object63672:head# 2024-02-09T17:57:32.766 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.766 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a35c774:::benchmark_data_smithi179_148205_object87954:head# 2024-02-09T17:57:32.766 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.766 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a3759b4:::benchmark_data_smithi179_148205_object21864:head# 2024-02-09T17:57:32.766 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.766 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a38b0f4:::benchmark_data_smithi179_148205_object19334:head# 2024-02-09T17:57:32.767 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.767 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a390bc1:::benchmark_data_smithi179_148205_object75320:head# 2024-02-09T17:57:32.767 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.767 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a393808:::benchmark_data_smithi179_148205_object35540:head# 2024-02-09T17:57:32.767 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.767 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a3a12dc:::benchmark_data_smithi179_148205_object81328:head# 2024-02-09T17:57:32.767 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.767 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a3a5f17:::benchmark_data_smithi179_148205_object87926:head# 2024-02-09T17:57:32.767 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.767 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a3ad11c:::benchmark_data_smithi179_148205_object9479:head# 2024-02-09T17:57:32.767 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.767 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a3bad73:::benchmark_data_smithi179_148205_object33821:head# 2024-02-09T17:57:32.767 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.770 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a3bbe63:::benchmark_data_smithi179_148205_object72826:head# 2024-02-09T17:57:32.770 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.771 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a3ce91a:::benchmark_data_smithi179_148205_object953:head# 2024-02-09T17:57:32.771 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.771 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a3e3f9d:::benchmark_data_smithi179_148205_object18999:head# 2024-02-09T17:57:32.771 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.771 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a3f87da:::benchmark_data_smithi179_148205_object17875:head# 2024-02-09T17:57:32.771 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.771 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a3fc3d0:::benchmark_data_smithi179_148205_object85030:head# 2024-02-09T17:57:32.771 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.771 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a402bd4:::benchmark_data_smithi179_148205_object37616:head# 2024-02-09T17:57:32.771 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.771 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a408b79:::benchmark_data_smithi179_148205_object60167:head# 2024-02-09T17:57:32.771 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.771 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a413b0e:::benchmark_data_smithi179_148205_object84727:head# 2024-02-09T17:57:32.771 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.772 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a41bdf4:::benchmark_data_smithi179_148205_object49346:head# 2024-02-09T17:57:32.772 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.772 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a41cabd:::benchmark_data_smithi179_148205_object54434:head# 2024-02-09T17:57:32.772 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.772 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a4212b1:::benchmark_data_smithi179_148205_object34771:head# 2024-02-09T17:57:32.772 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.772 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a433b8d:::benchmark_data_smithi179_148205_object26094:head# 2024-02-09T17:57:32.772 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.772 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a43477b:::benchmark_data_smithi179_148205_object84900:head# 2024-02-09T17:57:32.772 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.772 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a44a354:::benchmark_data_smithi179_148205_object3294:head# 2024-02-09T17:57:32.773 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.773 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a44f2e0:::benchmark_data_smithi179_148205_object66709:head# 2024-02-09T17:57:32.773 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.773 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a451c7a:::benchmark_data_smithi179_148205_object74511:head# 2024-02-09T17:57:32.773 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.773 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a466cc4:::benchmark_data_smithi179_148205_object54200:head# 2024-02-09T17:57:32.773 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.773 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a46afe3:::benchmark_data_smithi179_148205_object89290:head# 2024-02-09T17:57:32.773 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.773 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a46e8e9:::benchmark_data_smithi179_148205_object80569:head# 2024-02-09T17:57:32.773 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.773 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a47f4bf:::benchmark_data_smithi179_148205_object99434:head# 2024-02-09T17:57:32.773 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.773 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a481a19:::benchmark_data_smithi179_148205_object12824:head# 2024-02-09T17:57:32.773 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.777 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a48d541:::benchmark_data_smithi179_148205_object31103:head# 2024-02-09T17:57:32.778 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.778 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a4a3e47:::benchmark_data_smithi179_148205_object22239:head# 2024-02-09T17:57:32.778 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.778 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a4a7cab:::benchmark_data_smithi179_148205_object69774:head# 2024-02-09T17:57:32.778 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.778 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a4af1ea:::benchmark_data_smithi179_148205_object102865:head# 2024-02-09T17:57:32.778 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.778 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a4af4e5:::benchmark_data_smithi179_148205_object87732:head# 2024-02-09T17:57:32.778 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.778 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a4b4afb:::benchmark_data_smithi179_148205_object80236:head# 2024-02-09T17:57:32.778 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.778 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a4b71a8:::benchmark_data_smithi179_148205_object19871:head# 2024-02-09T17:57:32.778 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.779 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a4bc847:::benchmark_data_smithi179_148205_object53476:head# 2024-02-09T17:57:32.779 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.779 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a4ca14b:::benchmark_data_smithi179_148205_object41175:head# 2024-02-09T17:57:32.779 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.779 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a4e0d9b:::benchmark_data_smithi179_148205_object16989:head# 2024-02-09T17:57:32.779 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.779 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a51acf2:::benchmark_data_smithi179_148205_object100820:head# 2024-02-09T17:57:32.779 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.779 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a521c6d:::benchmark_data_smithi179_148205_object62527:head# 2024-02-09T17:57:32.779 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.779 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a52e059:::benchmark_data_smithi179_148205_object12279:head# 2024-02-09T17:57:32.779 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.779 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a535f57:::benchmark_data_smithi179_148205_object36028:head# 2024-02-09T17:57:32.779 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.779 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a53b812:::benchmark_data_smithi179_148205_object32319:head# 2024-02-09T17:57:32.780 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.780 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a546759:::benchmark_data_smithi179_148205_object39889:head# 2024-02-09T17:57:32.780 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.780 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a54c94a:::benchmark_data_smithi179_148205_object105617:head# 2024-02-09T17:57:32.780 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.780 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a5581e2:::benchmark_data_smithi179_148205_object22447:head# 2024-02-09T17:57:32.780 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.780 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a55ac90:::benchmark_data_smithi179_148205_object103560:head# 2024-02-09T17:57:32.780 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.780 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a5635ef:::benchmark_data_smithi179_148205_object7034:head# 2024-02-09T17:57:32.780 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.780 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a578d5d:::benchmark_data_smithi179_148205_object23268:head# 2024-02-09T17:57:32.780 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.780 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a579531:::benchmark_data_smithi179_148205_object73897:head# 2024-02-09T17:57:32.781 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.781 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a584abc:::benchmark_data_smithi179_148205_object7190:head# 2024-02-09T17:57:32.781 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.781 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a588bf4:::benchmark_data_smithi179_148205_object19374:head# 2024-02-09T17:57:32.781 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.781 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a59ed48:::benchmark_data_smithi179_148205_object106834:head# 2024-02-09T17:57:32.781 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.781 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a5a1fe9:::benchmark_data_smithi179_148205_object73552:head# 2024-02-09T17:57:32.781 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.781 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a5a7cb5:::benchmark_data_smithi179_148205_object70330:head# 2024-02-09T17:57:32.781 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.784 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a5aac68:::benchmark_data_smithi179_148205_object75752:head# 2024-02-09T17:57:32.784 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.785 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a5ae393:::benchmark_data_smithi179_148205_object62550:head# 2024-02-09T17:57:32.785 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.785 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a5b1167:::benchmark_data_smithi179_148205_object65337:head# 2024-02-09T17:57:32.785 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.785 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a5b6cdf:::benchmark_data_smithi179_148205_object46503:head# 2024-02-09T17:57:32.785 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.785 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a5ba8a5:::benchmark_data_smithi179_148205_object90163:head# 2024-02-09T17:57:32.785 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.785 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a5ddd07:::benchmark_data_smithi179_148205_object33205:head# 2024-02-09T17:57:32.785 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.785 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a5ea1ee:::benchmark_data_smithi179_148205_object67068:head# 2024-02-09T17:57:32.785 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.785 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a5fa907:::benchmark_data_smithi179_148205_object34638:head# 2024-02-09T17:57:32.785 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.786 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a60b8b7:::benchmark_data_smithi179_148205_object93168:head# 2024-02-09T17:57:32.786 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.786 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a619be4:::benchmark_data_smithi179_148205_object5356:head# 2024-02-09T17:57:32.786 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.786 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a63ca41:::benchmark_data_smithi179_148205_object105214:head# 2024-02-09T17:57:32.786 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.786 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a63f5f1:::benchmark_data_smithi179_148205_object15760:head# 2024-02-09T17:57:32.786 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.786 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a64d34f:::benchmark_data_smithi179_148205_object102907:head# 2024-02-09T17:57:32.786 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.786 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a653d37:::benchmark_data_smithi179_148205_object46276:head# 2024-02-09T17:57:32.786 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.786 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a65c1e4:::benchmark_data_smithi179_148205_object23823:head# 2024-02-09T17:57:32.786 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.786 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a65fac5:::benchmark_data_smithi179_148205_object47912:head# 2024-02-09T17:57:32.787 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.787 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a66140f:::benchmark_data_smithi179_148205_object9626:head# 2024-02-09T17:57:32.787 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.787 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:57:32.788 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a66cfda:::benchmark_data_smithi179_148205_object83015:head# 2024-02-09T17:57:32.788 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.788 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a66daff:::benchmark_data_smithi179_148205_object27701:head# 2024-02-09T17:57:32.788 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.788 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a6819ae:::benchmark_data_smithi179_148205_object10281:head# 2024-02-09T17:57:32.788 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.788 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a687d85:::benchmark_data_smithi179_148205_object85778:head# 2024-02-09T17:57:32.788 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.788 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a68b406:::benchmark_data_smithi179_148205_object17005:head# 2024-02-09T17:57:32.788 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.788 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a68de9d:::benchmark_data_smithi179_148205_object21096:head# 2024-02-09T17:57:32.789 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.789 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a6943e7:::benchmark_data_smithi179_148205_object84449:head# 2024-02-09T17:57:32.789 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.789 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a69528d:::benchmark_data_smithi179_148205_object11034:head# 2024-02-09T17:57:32.789 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.789 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a6a0703:::benchmark_data_smithi179_148205_object11527:head# 2024-02-09T17:57:32.789 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.789 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a6a4630:::benchmark_data_smithi179_148205_object34112:head# 2024-02-09T17:57:32.789 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.789 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a6a5570:::benchmark_data_smithi179_148205_object67385:head# 2024-02-09T17:57:32.789 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.789 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a6a7902:::benchmark_data_smithi179_148205_object78930:head# 2024-02-09T17:57:32.789 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.789 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a6aa159:::benchmark_data_smithi179_148205_object8629:head# 2024-02-09T17:57:32.789 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.790 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a6c634c:::benchmark_data_smithi179_148205_object36345:head# 2024-02-09T17:57:32.790 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.790 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a6d5dd2:::benchmark_data_smithi179_148205_object105367:head# 2024-02-09T17:57:32.790 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:32.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:57:32.796 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a6d8702:::benchmark_data_smithi179_148205_object97985:head# 2024-02-09T17:57:32.796 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.796 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a6da8e6:::benchmark_data_smithi179_148205_object83394:head# 2024-02-09T17:57:32.796 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.796 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a6dadfb:::benchmark_data_smithi179_148205_object25940:head# 2024-02-09T17:57:32.796 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.796 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a6dbd22:::benchmark_data_smithi179_148205_object66073:head# 2024-02-09T17:57:32.796 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.796 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a6ef16d:::benchmark_data_smithi179_148205_object555:head# 2024-02-09T17:57:32.796 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.796 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a6f5348:::benchmark_data_smithi179_148205_object35891:head# 2024-02-09T17:57:32.796 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.797 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a6f7e70:::benchmark_data_smithi179_148205_object102261:head# 2024-02-09T17:57:32.797 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.797 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a70d698:::benchmark_data_smithi179_148205_object48455:head# 2024-02-09T17:57:32.797 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.797 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a73d0a7:::benchmark_data_smithi179_148205_object6021:head# 2024-02-09T17:57:32.797 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.797 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a74495a:::benchmark_data_smithi179_148205_object92063:head# 2024-02-09T17:57:32.797 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.797 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a7472d8:::benchmark_data_smithi179_148205_object46563:head# 2024-02-09T17:57:32.797 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.797 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a757cd3:::benchmark_data_smithi179_148205_object102620:head# 2024-02-09T17:57:32.797 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.797 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a75c9af:::benchmark_data_smithi179_148205_object48570:head# 2024-02-09T17:57:32.797 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.798 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a76379f:::benchmark_data_smithi179_148205_object107119:head# 2024-02-09T17:57:32.798 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.798 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a770797:::benchmark_data_smithi179_148205_object4520:head# 2024-02-09T17:57:32.798 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.798 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a779395:::benchmark_data_smithi179_148205_object78788:head# 2024-02-09T17:57:32.798 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.798 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a77e77e:::benchmark_data_smithi179_148205_object23451:head# 2024-02-09T17:57:32.798 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.798 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a79e4bd:::benchmark_data_smithi179_148205_object95245:head# 2024-02-09T17:57:32.798 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.798 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a7bda73:::benchmark_data_smithi179_148205_object79781:head# 2024-02-09T17:57:32.798 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.798 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a7cacfa:::benchmark_data_smithi179_148205_object45144:head# 2024-02-09T17:57:32.798 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.798 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a7cdd7f:::benchmark_data_smithi179_148205_object29364:head# 2024-02-09T17:57:32.799 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.799 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a7d3d10:::benchmark_data_smithi179_148205_object49526:head# 2024-02-09T17:57:32.799 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.799 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a7dbb72:::benchmark_data_smithi179_148205_object55284:head# 2024-02-09T17:57:32.799 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.799 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a7e36d5:::benchmark_data_smithi179_148205_object28307:head# 2024-02-09T17:57:32.799 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.799 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a7ec137:::benchmark_data_smithi179_148205_object14370:head# 2024-02-09T17:57:32.799 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.799 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a7f49e1:::benchmark_data_smithi179_148205_object5507:head# 2024-02-09T17:57:32.799 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.799 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a802a8d:::benchmark_data_smithi179_148205_object91319:head# 2024-02-09T17:57:32.799 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.799 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a80de2f:::benchmark_data_smithi179_148205_object82523:head# 2024-02-09T17:57:32.799 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.800 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a82ff1e:::benchmark_data_smithi179_148205_object91433:head# 2024-02-09T17:57:32.800 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.800 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a83673c:::benchmark_data_smithi179_148205_object102402:head# 2024-02-09T17:57:32.800 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.807 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a851517:::benchmark_data_smithi179_148205_object31816:head# 2024-02-09T17:57:32.807 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.807 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a86bd96:::benchmark_data_smithi179_148205_object30133:head# 2024-02-09T17:57:32.807 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.807 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a88ec5b:::benchmark_data_smithi179_148205_object36949:head# 2024-02-09T17:57:32.807 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.807 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a890a1a:::benchmark_data_smithi179_148205_object31385:head# 2024-02-09T17:57:32.807 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.807 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a897778:::benchmark_data_smithi179_148205_object82002:head# 2024-02-09T17:57:32.807 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.807 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a89ef11:::benchmark_data_smithi179_148205_object13110:head# 2024-02-09T17:57:32.807 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.808 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a8b244a:::benchmark_data_smithi179_148205_object70172:head# 2024-02-09T17:57:32.808 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.808 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a8b26b2:::benchmark_data_smithi179_148205_object65513:head# 2024-02-09T17:57:32.808 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.808 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a8bea57:::benchmark_data_smithi179_148205_object86659:head# 2024-02-09T17:57:32.808 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.808 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a8c1db9:::benchmark_data_smithi179_148205_object105315:head# 2024-02-09T17:57:32.808 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.808 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a8c3e9a:::benchmark_data_smithi179_148205_object46519:head# 2024-02-09T17:57:32.808 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.808 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a8c6c2f:::benchmark_data_smithi179_148205_object32890:head# 2024-02-09T17:57:32.808 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.808 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a8c86fe:::benchmark_data_smithi179_148205_object20431:head# 2024-02-09T17:57:32.808 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.809 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a8c9b4f:::benchmark_data_smithi179_148205_object59915:head# 2024-02-09T17:57:32.809 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.809 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a8cd2eb:::benchmark_data_smithi179_148205_object61985:head# 2024-02-09T17:57:32.809 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.809 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a8cd6fc:::benchmark_data_smithi179_148205_object87679:head# 2024-02-09T17:57:32.809 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.809 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a8dbbc6:::benchmark_data_smithi179_148205_object53411:head# 2024-02-09T17:57:32.809 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.809 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a8e9781:::benchmark_data_smithi179_148205_object94734:head# 2024-02-09T17:57:32.809 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.809 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a8f4808:::benchmark_data_smithi179_148205_object31860:head# 2024-02-09T17:57:32.809 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.809 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a8fec8d:::benchmark_data_smithi179_148205_object68857:head# 2024-02-09T17:57:32.809 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.810 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a903d67:::benchmark_data_smithi179_148205_object5890:head# 2024-02-09T17:57:32.810 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.810 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a90862a:::benchmark_data_smithi179_148205_object20731:head# 2024-02-09T17:57:32.810 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.810 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a91518b:::benchmark_data_smithi179_148205_object67351:head# 2024-02-09T17:57:32.810 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.810 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a916fd8:::benchmark_data_smithi179_148205_object39142:head# 2024-02-09T17:57:32.810 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.810 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a929bb5:::benchmark_data_smithi179_148205_object48830:head# 2024-02-09T17:57:32.810 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.810 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a94a8aa:::benchmark_data_smithi179_148205_object89747:head# 2024-02-09T17:57:32.810 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.810 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a94e928:::benchmark_data_smithi179_148205_object52429:head# 2024-02-09T17:57:32.810 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.810 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a94f691:::benchmark_data_smithi179_148205_object5048:head# 2024-02-09T17:57:32.811 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.811 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a9503c2:::benchmark_data_smithi179_148205_object76241:head# 2024-02-09T17:57:32.811 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.811 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a954c6c:::benchmark_data_smithi179_148205_object63510:head# 2024-02-09T17:57:32.811 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.811 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a96234c:::benchmark_data_smithi179_148205_object29778:head# 2024-02-09T17:57:32.811 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.811 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a966b97:::benchmark_data_smithi179_148205_object66899:head# 2024-02-09T17:57:32.811 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.811 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a9720ca:::benchmark_data_smithi179_148205_object70643:head# 2024-02-09T17:57:32.811 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.811 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a977e31:::benchmark_data_smithi179_148205_object22205:head# 2024-02-09T17:57:32.811 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.811 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a97a224:::benchmark_data_smithi179_148205_object89576:head# 2024-02-09T17:57:32.812 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.812 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a980ac3:::benchmark_data_smithi179_148205_object18391:head# 2024-02-09T17:57:32.812 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.812 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a982cbb:::benchmark_data_smithi179_148205_object53721:head# 2024-02-09T17:57:32.812 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.812 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a983d39:::benchmark_data_smithi179_148205_object81528:head# 2024-02-09T17:57:32.812 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.812 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a9888d2:::benchmark_data_smithi179_148205_object55666:head# 2024-02-09T17:57:32.812 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.812 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a98f090:::benchmark_data_smithi179_148205_object28947:head# 2024-02-09T17:57:32.812 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.812 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a991353:::benchmark_data_smithi179_148205_object60740:head# 2024-02-09T17:57:32.812 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.812 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a996e4f:::benchmark_data_smithi179_148205_object50520:head# 2024-02-09T17:57:32.813 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.813 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a9a6430:::benchmark_data_smithi179_148205_object99216:head# 2024-02-09T17:57:32.813 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.813 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a9b43cc:::benchmark_data_smithi179_148205_object85016:head# 2024-02-09T17:57:32.813 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.813 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a9ccb93:::benchmark_data_smithi179_148205_object100769:head# 2024-02-09T17:57:32.813 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.813 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a9d5acd:::benchmark_data_smithi179_148205_object3321:head# 2024-02-09T17:57:32.813 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.819 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a9d9d22:::benchmark_data_smithi179_148205_object52242:head# 2024-02-09T17:57:32.819 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.819 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a9dd2d7:::benchmark_data_smithi179_148205_object34998:head# 2024-02-09T17:57:32.819 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.819 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a9de330:::benchmark_data_smithi179_148205_object10093:head# 2024-02-09T17:57:32.819 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.819 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a9ec621:::benchmark_data_smithi179_148205_object101349:head# 2024-02-09T17:57:32.819 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.819 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a9f5a2d:::benchmark_data_smithi179_148205_object106879:head# 2024-02-09T17:57:32.819 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.819 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1a9f99ca:::benchmark_data_smithi179_148205_object52118:head# 2024-02-09T17:57:32.819 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.820 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1aa1dc13:::benchmark_data_smithi179_148205_object96389:head# 2024-02-09T17:57:32.820 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.820 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1aa23468:::benchmark_data_smithi179_148205_object103125:head# 2024-02-09T17:57:32.820 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.820 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1aa25c7c:::benchmark_data_smithi179_148205_object51444:head# 2024-02-09T17:57:32.820 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.820 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1aa3d7f5:::benchmark_data_smithi179_148205_object92735:head# 2024-02-09T17:57:32.820 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.820 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1aa4857f:::benchmark_data_smithi179_148205_object40793:head# 2024-02-09T17:57:32.820 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.820 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1aa49170:::benchmark_data_smithi179_148205_object26035:head# 2024-02-09T17:57:32.821 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.821 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1aa4d6b5:::benchmark_data_smithi179_148205_object9969:head# 2024-02-09T17:57:32.821 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.821 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1aa55df3:::benchmark_data_smithi179_148205_object41426:head# 2024-02-09T17:57:32.821 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.821 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1aa5f7eb:::benchmark_data_smithi179_148205_object97450:head# 2024-02-09T17:57:32.821 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.821 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1aa65b16:::benchmark_data_smithi179_148205_object33114:head# 2024-02-09T17:57:32.821 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.821 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1aa747f0:::benchmark_data_smithi179_148205_object66661:head# 2024-02-09T17:57:32.821 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.821 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1aa76383:::benchmark_data_smithi179_148205_object48152:head# 2024-02-09T17:57:32.821 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.821 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1aa78783:::benchmark_data_smithi179_148205_object23877:head# 2024-02-09T17:57:32.822 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.822 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1aa7bea3:::benchmark_data_smithi179_148205_object76981:head# 2024-02-09T17:57:32.822 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.822 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1aa7eaa3:::benchmark_data_smithi179_148205_object50559:head# 2024-02-09T17:57:32.822 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.822 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1aa8324b:::benchmark_data_smithi179_148205_object42122:head# 2024-02-09T17:57:32.822 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.822 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1aa880d6:::benchmark_data_smithi179_148205_object81231:head# 2024-02-09T17:57:32.822 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.822 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1aa90b8a:::benchmark_data_smithi179_148205_object88002:head# 2024-02-09T17:57:32.822 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.822 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1aa91f05:::benchmark_data_smithi179_148205_object71404:head# 2024-02-09T17:57:32.822 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.822 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1aabb6f9:::benchmark_data_smithi179_148205_object63814:head# 2024-02-09T17:57:32.822 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.822 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1aac3f5d:::benchmark_data_smithi179_148205_object70038:head# 2024-02-09T17:57:32.823 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.823 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1aac57e5:::benchmark_data_smithi179_148205_object26113:head# 2024-02-09T17:57:32.823 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.823 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1aac5e1e:::benchmark_data_smithi179_148205_object4008:head# 2024-02-09T17:57:32.823 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.823 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1aac8a7a:::benchmark_data_smithi179_148205_object20285:head# 2024-02-09T17:57:32.823 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.823 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1aaca766:::benchmark_data_smithi179_148205_object10484:head# 2024-02-09T17:57:32.823 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.823 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1aacc1af:::benchmark_data_smithi179_148205_object13454:head# 2024-02-09T17:57:32.823 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.823 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1aafb978:::benchmark_data_smithi179_148205_object33385:head# 2024-02-09T17:57:32.823 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.823 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ab022f3:::benchmark_data_smithi179_148205_object42511:head# 2024-02-09T17:57:32.823 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.823 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ab0eec1:::benchmark_data_smithi179_148205_object14559:head# 2024-02-09T17:57:32.824 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.824 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ab43cf0:::benchmark_data_smithi179_148205_object7137:head# 2024-02-09T17:57:32.824 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.824 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ab4e85e:::benchmark_data_smithi179_148205_object1592:head# 2024-02-09T17:57:32.824 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.824 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ab564dd:::benchmark_data_smithi179_148205_object25165:head# 2024-02-09T17:57:32.824 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.824 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ab69107:::benchmark_data_smithi179_148205_object44797:head# 2024-02-09T17:57:32.824 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.824 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ab6cbd0:::benchmark_data_smithi179_148205_object21296:head# 2024-02-09T17:57:32.824 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.832 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ab713f6:::benchmark_data_smithi179_148205_object72829:head# 2024-02-09T17:57:32.832 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.832 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ab93b25:::benchmark_data_smithi179_148205_object106225:head# 2024-02-09T17:57:32.832 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.832 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ab9e7cf:::benchmark_data_smithi179_148205_object99580:head# 2024-02-09T17:57:32.832 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.832 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1aba2587:::benchmark_data_smithi179_148205_object54161:head# 2024-02-09T17:57:32.832 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.833 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1abb1131:::benchmark_data_smithi179_148205_object83390:head# 2024-02-09T17:57:32.833 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.833 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1abb7a1d:::benchmark_data_smithi179_148205_object940:head# 2024-02-09T17:57:32.833 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.833 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1abc0b44:::benchmark_data_smithi179_148205_object92457:head# 2024-02-09T17:57:32.833 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.833 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1abc28cc:::benchmark_data_smithi179_148205_object70151:head# 2024-02-09T17:57:32.833 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.833 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1abd25ac:::benchmark_data_smithi179_148205_object14579:head# 2024-02-09T17:57:32.833 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.833 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1abd3b65:::benchmark_data_smithi179_148205_object102740:head# 2024-02-09T17:57:32.833 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.833 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1abd46b0:::benchmark_data_smithi179_148205_object1862:head# 2024-02-09T17:57:32.833 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.834 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1abd894e:::benchmark_data_smithi179_148205_object25630:head# 2024-02-09T17:57:32.834 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.834 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1abdbac2:::benchmark_data_smithi179_148205_object89527:head# 2024-02-09T17:57:32.834 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.834 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1abe4c2e:::benchmark_data_smithi179_148205_object44699:head# 2024-02-09T17:57:32.834 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.834 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1abf3c0b:::benchmark_data_smithi179_148205_object59780:head# 2024-02-09T17:57:32.834 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.834 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1abfa864:::benchmark_data_smithi179_148205_object70137:head# 2024-02-09T17:57:32.834 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.834 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1abfca1c:::benchmark_data_smithi179_148205_object38303:head# 2024-02-09T17:57:32.834 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.834 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ac08325:::benchmark_data_smithi179_148205_object58440:head# 2024-02-09T17:57:32.834 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.834 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ac08e14:::benchmark_data_smithi179_148205_object109678:head# 2024-02-09T17:57:32.835 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.835 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ac1a74b:::benchmark_data_smithi179_148205_object21477:head# 2024-02-09T17:57:32.835 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.835 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ac1bbe4:::benchmark_data_smithi179_148205_object12072:head# 2024-02-09T17:57:32.835 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.835 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ac314a5:::benchmark_data_smithi179_148205_object59840:head# 2024-02-09T17:57:32.835 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.835 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ac32cc9:::benchmark_data_smithi179_148205_object99893:head# 2024-02-09T17:57:32.835 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.835 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ac3f57e:::benchmark_data_smithi179_148205_object87204:head# 2024-02-09T17:57:32.835 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.835 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ac4876b:::benchmark_data_smithi179_148205_object57442:head# 2024-02-09T17:57:32.835 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.835 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ac4baf2:::benchmark_data_smithi179_148205_object28205:head# 2024-02-09T17:57:32.835 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.836 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ac4de13:::benchmark_data_smithi179_148205_object14813:head# 2024-02-09T17:57:32.836 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.836 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ac67a49:::benchmark_data_smithi179_148205_object28126:head# 2024-02-09T17:57:32.836 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.836 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ac7b6a1:::benchmark_data_smithi179_148205_object97791:head# 2024-02-09T17:57:32.836 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.836 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ac8393b:::benchmark_data_smithi179_148205_object74390:head# 2024-02-09T17:57:32.836 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.836 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1aca1424:::benchmark_data_smithi179_148205_object55171:head# 2024-02-09T17:57:32.836 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.836 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1acb3483:::benchmark_data_smithi179_148205_object14759:head# 2024-02-09T17:57:32.836 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.836 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1acb5001:::benchmark_data_smithi179_148205_object24730:head# 2024-02-09T17:57:32.837 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.837 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1acb9043:::benchmark_data_smithi179_148205_object66680:head# 2024-02-09T17:57:32.837 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.837 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1acba662:::benchmark_data_smithi179_148205_object108917:head# 2024-02-09T17:57:32.837 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.837 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1acbd1a1:::benchmark_data_smithi179_148205_object12638:head# 2024-02-09T17:57:32.837 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.837 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1acc3d39:::benchmark_data_smithi179_148205_object36113:head# 2024-02-09T17:57:32.837 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.837 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1acd47c8:::benchmark_data_smithi179_148205_object98906:head# 2024-02-09T17:57:32.837 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.837 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1acd5cb7:::benchmark_data_smithi179_148205_object30164:head# 2024-02-09T17:57:32.837 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.837 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ace0106:::benchmark_data_smithi179_148205_object51476:head# 2024-02-09T17:57:32.838 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.838 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ace6226:::benchmark_data_smithi179_148205_object16577:head# 2024-02-09T17:57:32.838 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.838 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ace72be:::benchmark_data_smithi179_148205_object62093:head# 2024-02-09T17:57:32.838 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.838 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1acf0d89:::benchmark_data_smithi179_148205_object100703:head# 2024-02-09T17:57:32.838 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.838 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1acfcb48:::benchmark_data_smithi179_148205_object20089:head# 2024-02-09T17:57:32.838 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.838 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1acfda36:::benchmark_data_smithi179_148205_object5318:head# 2024-02-09T17:57:32.838 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.838 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ad0754d:::benchmark_data_smithi179_148205_object82558:head# 2024-02-09T17:57:32.838 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.838 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ad0f99f:::benchmark_data_smithi179_148205_object84214:head# 2024-02-09T17:57:32.838 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.839 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ad1f819:::benchmark_data_smithi179_148205_object102887:head# 2024-02-09T17:57:32.839 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.839 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ad1fecc:::benchmark_data_smithi179_148205_object53926:head# 2024-02-09T17:57:32.839 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.839 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ad452d6:::benchmark_data_smithi179_148205_object11511:head# 2024-02-09T17:57:32.839 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.843 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ad4fefe:::benchmark_data_smithi179_148205_object30694:head# 2024-02-09T17:57:32.843 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.843 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ad51533:::benchmark_data_smithi179_148205_object74416:head# 2024-02-09T17:57:32.844 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.844 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ad601c4:::benchmark_data_smithi179_148205_object49602:head# 2024-02-09T17:57:32.844 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.844 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ad64303:::benchmark_data_smithi179_148205_object36453:head# 2024-02-09T17:57:32.844 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.844 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ad74ea9:::benchmark_data_smithi179_148205_object44390:head# 2024-02-09T17:57:32.844 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.844 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ad85907:::benchmark_data_smithi179_148205_object70519:head# 2024-02-09T17:57:32.844 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.844 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ad8e0e0:::benchmark_data_smithi179_148205_object16054:head# 2024-02-09T17:57:32.844 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.844 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ad9917e:::benchmark_data_smithi179_148205_object85662:head# 2024-02-09T17:57:32.844 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.844 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ada0d0b:::benchmark_data_smithi179_148205_object67526:head# 2024-02-09T17:57:32.845 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.845 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ada0e62:::benchmark_data_smithi179_148205_object5895:head# 2024-02-09T17:57:32.845 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.845 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ada1620:::benchmark_data_smithi179_148205_object46378:head# 2024-02-09T17:57:32.845 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.845 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1adaedce:::benchmark_data_smithi179_148205_object67807:head# 2024-02-09T17:57:32.845 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.845 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1adc1433:::benchmark_data_smithi179_148205_object49166:head# 2024-02-09T17:57:32.845 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.845 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1adc2022:::benchmark_data_smithi179_148205_object6712:head# 2024-02-09T17:57:32.845 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.845 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1adcb204:::benchmark_data_smithi179_148205_object6033:head# 2024-02-09T17:57:32.845 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.845 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1add78a0:::benchmark_data_smithi179_148205_object96653:head# 2024-02-09T17:57:32.845 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.846 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1addcc6d:::benchmark_data_smithi179_148205_object37496:head# 2024-02-09T17:57:32.846 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.846 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ade6b17:::benchmark_data_smithi179_148205_object15646:head# 2024-02-09T17:57:32.846 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.846 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1adea294:::benchmark_data_smithi179_148205_object14415:head# 2024-02-09T17:57:32.846 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.846 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1adee34a:::benchmark_data_smithi179_148205_object99635:head# 2024-02-09T17:57:32.846 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.846 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1adf7604:::benchmark_data_smithi179_148205_object33888:head# 2024-02-09T17:57:32.846 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.846 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ae08287:::benchmark_data_smithi179_148205_object10779:head# 2024-02-09T17:57:32.846 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.846 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ae08607:::benchmark_data_smithi179_148205_object82030:head# 2024-02-09T17:57:32.846 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.846 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ae2a9f0:::benchmark_data_smithi179_148205_object12600:head# 2024-02-09T17:57:32.847 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.847 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ae3b24b:::benchmark_data_smithi179_148205_object82937:head# 2024-02-09T17:57:32.847 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.847 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ae3f5a4:::benchmark_data_smithi179_148205_object14221:head# 2024-02-09T17:57:32.847 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.847 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ae4e743:::benchmark_data_smithi179_148205_object3759:head# 2024-02-09T17:57:32.847 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.847 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ae547e4:::benchmark_data_smithi179_148205_object47844:head# 2024-02-09T17:57:32.847 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.848 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ae5bc55:::benchmark_data_smithi179_148205_object13585:head# 2024-02-09T17:57:32.849 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.849 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ae61fda:::benchmark_data_smithi179_148205_object77783:head# 2024-02-09T17:57:32.849 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.849 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ae625e4:::benchmark_data_smithi179_148205_object3424:head# 2024-02-09T17:57:32.849 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.849 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ae6398e:::benchmark_data_smithi179_148205_object97790:head# 2024-02-09T17:57:32.849 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.849 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ae6d79d:::benchmark_data_smithi179_148205_object80583:head# 2024-02-09T17:57:32.849 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.849 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ae79b90:::benchmark_data_smithi179_148205_object67983:head# 2024-02-09T17:57:32.849 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.850 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ae87f35:::benchmark_data_smithi179_148205_object39181:head# 2024-02-09T17:57:32.850 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.850 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ae8ea23:::benchmark_data_smithi179_148205_object35818:head# 2024-02-09T17:57:32.850 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.850 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ae9f3ed:::benchmark_data_smithi179_148205_object104629:head# 2024-02-09T17:57:32.850 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.850 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1aea3cc7:::benchmark_data_smithi179_148205_object41780:head# 2024-02-09T17:57:32.850 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.851 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1aea9d97:::benchmark_data_smithi179_148205_object46774:head# 2024-02-09T17:57:32.851 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.851 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1aeabcf2:::benchmark_data_smithi179_148205_object55458:head# 2024-02-09T17:57:32.852 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.852 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1aeefa25:::benchmark_data_smithi179_148205_object104919:head# 2024-02-09T17:57:32.852 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.852 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1aef9c81:::benchmark_data_smithi179_148205_object56925:head# 2024-02-09T17:57:32.852 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.854 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1af0cfad:::benchmark_data_smithi179_148205_object100074:head# 2024-02-09T17:57:32.854 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.854 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1af15b8c:::benchmark_data_smithi179_148205_object65391:head# 2024-02-09T17:57:32.854 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.854 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1af1be32:::benchmark_data_smithi179_148205_object76872:head# 2024-02-09T17:57:32.854 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.854 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1af2e849:::benchmark_data_smithi179_148205_object89954:head# 2024-02-09T17:57:32.854 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.854 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1af331ee:::benchmark_data_smithi179_148205_object67030:head# 2024-02-09T17:57:32.854 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.854 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1af3dd81:::benchmark_data_smithi179_148205_object57735:head# 2024-02-09T17:57:32.854 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.854 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1af4846a:::benchmark_data_smithi179_148205_object83064:head# 2024-02-09T17:57:32.855 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.855 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1af50d3a:::benchmark_data_smithi179_148205_object47160:head# 2024-02-09T17:57:32.855 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.855 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1af53714:::benchmark_data_smithi179_148205_object4382:head# 2024-02-09T17:57:32.855 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.856 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1af590b6:::benchmark_data_smithi179_148205_object4752:head# 2024-02-09T17:57:32.856 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.856 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1af5e5ae:::benchmark_data_smithi179_148205_object16240:head# 2024-02-09T17:57:32.857 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.857 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1af69eed:::benchmark_data_smithi179_148205_object2286:head# 2024-02-09T17:57:32.857 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.857 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1af6a90b:::benchmark_data_smithi179_148205_object30905:head# 2024-02-09T17:57:32.857 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.857 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1af71c81:::benchmark_data_smithi179_148205_object12213:head# 2024-02-09T17:57:32.857 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.857 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1af7cbfa:::benchmark_data_smithi179_148205_object25018:head# 2024-02-09T17:57:32.857 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.857 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1af810b0:::benchmark_data_smithi179_148205_object30631:head# 2024-02-09T17:57:32.857 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.857 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1af84146:::benchmark_data_smithi179_148205_object46121:head# 2024-02-09T17:57:32.857 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.857 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1af84768:::benchmark_data_smithi179_148205_object45708:head# 2024-02-09T17:57:32.857 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.861 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1af8b6c9:::benchmark_data_smithi179_148205_object10700:head# 2024-02-09T17:57:32.861 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.861 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1af9418c:::benchmark_data_smithi179_148205_object37516:head# 2024-02-09T17:57:32.861 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.861 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1afbac93:::benchmark_data_smithi179_148205_object87704:head# 2024-02-09T17:57:32.861 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.861 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1afcf328:::benchmark_data_smithi179_148205_object35147:head# 2024-02-09T17:57:32.861 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.861 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1afe1a35:::benchmark_data_smithi179_148205_object17812:head# 2024-02-09T17:57:32.862 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.862 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1afe6aec:::benchmark_data_smithi179_148205_object4057:head# 2024-02-09T17:57:32.862 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.862 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1afe9ca6:::benchmark_data_smithi179_148205_object109142:head# 2024-02-09T17:57:32.862 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.862 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b00318c:::benchmark_data_smithi179_148205_object34202:head# 2024-02-09T17:57:32.862 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.862 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b00f26b:::benchmark_data_smithi179_148205_object46523:head# 2024-02-09T17:57:32.862 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.862 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b0103be:::benchmark_data_smithi179_148205_object74072:head# 2024-02-09T17:57:32.862 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.862 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b02ff45:::benchmark_data_smithi179_148205_object44913:head# 2024-02-09T17:57:32.862 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.862 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b03481b:::benchmark_data_smithi179_148205_object36126:head# 2024-02-09T17:57:32.862 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.863 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b037e5e:::benchmark_data_smithi179_148205_object91765:head# 2024-02-09T17:57:32.863 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.863 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b038250:::benchmark_data_smithi179_148205_object29431:head# 2024-02-09T17:57:32.863 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.863 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b0403a6:::benchmark_data_smithi179_148205_object26722:head# 2024-02-09T17:57:32.863 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.863 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b046f86:::benchmark_data_smithi179_148205_object13184:head# 2024-02-09T17:57:32.863 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.863 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b0547a9:::benchmark_data_smithi179_148205_object107134:head# 2024-02-09T17:57:32.863 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.867 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b05d297:::benchmark_data_smithi179_148205_object86136:head# 2024-02-09T17:57:32.867 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.867 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b069651:::benchmark_data_smithi179_148205_object70843:head# 2024-02-09T17:57:32.867 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.867 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b0838f2:::benchmark_data_smithi179_148205_object31463:head# 2024-02-09T17:57:32.867 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.868 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b0867e1:::benchmark_data_smithi179_148205_object73255:head# 2024-02-09T17:57:32.868 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.868 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b089028:::benchmark_data_smithi179_148205_object22391:head# 2024-02-09T17:57:32.868 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.868 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b08ed04:::benchmark_data_smithi179_148205_object50749:head# 2024-02-09T17:57:32.868 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.868 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b09689b:::benchmark_data_smithi179_148205_object31108:head# 2024-02-09T17:57:32.868 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.868 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b096ef3:::benchmark_data_smithi179_148205_object31445:head# 2024-02-09T17:57:32.869 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.869 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b0a330f:::benchmark_data_smithi179_148205_object59593:head# 2024-02-09T17:57:32.870 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.870 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b0abef0:::benchmark_data_smithi179_148205_object3896:head# 2024-02-09T17:57:32.870 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.870 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b0ac07a:::benchmark_data_smithi179_148205_object62655:head# 2024-02-09T17:57:32.870 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.870 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b0ca1bd:::benchmark_data_smithi179_148205_object97817:head# 2024-02-09T17:57:32.870 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.870 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b0d1ba4:::benchmark_data_smithi179_148205_object85325:head# 2024-02-09T17:57:32.870 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.870 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b0e45d1:::benchmark_data_smithi179_148205_object102360:head# 2024-02-09T17:57:32.870 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.871 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b102cd9:::benchmark_data_smithi179_148205_object83360:head# 2024-02-09T17:57:32.871 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.872 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b12354f:::benchmark_data_smithi179_148205_object9843:head# 2024-02-09T17:57:32.872 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.872 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b13a624:::benchmark_data_smithi179_148205_object50241:head# 2024-02-09T17:57:32.872 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.872 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b13f85a:::benchmark_data_smithi179_148205_object72388:head# 2024-02-09T17:57:32.872 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.872 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b149a08:::benchmark_data_smithi179_148205_object89137:head# 2024-02-09T17:57:32.872 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.872 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b14e3bb:::benchmark_data_smithi179_148205_object101891:head# 2024-02-09T17:57:32.872 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.872 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b165ecb:::benchmark_data_smithi179_148205_object108039:head# 2024-02-09T17:57:32.872 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.873 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b17b18f:::benchmark_data_smithi179_148205_object60009:head# 2024-02-09T17:57:32.874 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.874 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b17ee64:::benchmark_data_smithi179_148205_object72331:head# 2024-02-09T17:57:32.874 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.874 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b18098e:::benchmark_data_smithi179_148205_object78858:head# 2024-02-09T17:57:32.874 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.874 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b18c072:::benchmark_data_smithi179_148205_object64724:head# 2024-02-09T17:57:32.874 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.874 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b18e0fe:::benchmark_data_smithi179_148205_object50234:head# 2024-02-09T17:57:32.874 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.874 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b199325:::benchmark_data_smithi179_148205_object103528:head# 2024-02-09T17:57:32.874 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.874 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b1ad085:::benchmark_data_smithi179_148205_object71365:head# 2024-02-09T17:57:32.874 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.874 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b1b0cd6:::benchmark_data_smithi179_148205_object18339:head# 2024-02-09T17:57:32.874 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.876 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b1b3f4c:::benchmark_data_smithi179_148205_object104662:head# 2024-02-09T17:57:32.876 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.876 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b1beaac:::benchmark_data_smithi179_148205_object33596:head# 2024-02-09T17:57:32.876 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.876 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b1ce623:::benchmark_data_smithi179_148205_object34947:head# 2024-02-09T17:57:32.876 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.876 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b1d0b38:::benchmark_data_smithi179_148205_object31832:head# 2024-02-09T17:57:32.876 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.876 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b1d12ea:::benchmark_data_smithi179_148205_object6611:head# 2024-02-09T17:57:32.877 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.877 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b1d2360:::benchmark_data_smithi179_148205_object30863:head# 2024-02-09T17:57:32.877 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.877 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b1d7c16:::benchmark_data_smithi179_148205_object55327:head# 2024-02-09T17:57:32.877 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.877 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b1e21de:::benchmark_data_smithi179_148205_object14974:head# 2024-02-09T17:57:32.877 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.877 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b1e59c8:::benchmark_data_smithi179_148205_object4141:head# 2024-02-09T17:57:32.877 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.879 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b1f5577:::benchmark_data_smithi179_148205_object101671:head# 2024-02-09T17:57:32.879 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.879 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b1fb475:::benchmark_data_smithi179_148205_object22966:head# 2024-02-09T17:57:32.879 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.879 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b20d7c4:::benchmark_data_smithi179_148205_object30500:head# 2024-02-09T17:57:32.879 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.879 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b20eb40:::benchmark_data_smithi179_148205_object89215:head# 2024-02-09T17:57:32.879 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.880 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b2105fc:::benchmark_data_smithi179_148205_object274:head# 2024-02-09T17:57:32.880 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.880 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b21a6eb:::benchmark_data_smithi179_148205_object6382:head# 2024-02-09T17:57:32.880 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.880 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b21e9d1:::benchmark_data_smithi179_148205_object104360:head# 2024-02-09T17:57:32.880 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.880 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b225d23:::benchmark_data_smithi179_148205_object36759:head# 2024-02-09T17:57:32.880 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.880 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b22f0ea:::benchmark_data_smithi179_148205_object20427:head# 2024-02-09T17:57:32.880 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.880 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b231ded:::benchmark_data_smithi179_148205_object38748:head# 2024-02-09T17:57:32.880 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.880 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b233669:::benchmark_data_smithi179_148205_object22487:head# 2024-02-09T17:57:32.880 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.880 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b24fad6:::benchmark_data_smithi179_148205_object98568:head# 2024-02-09T17:57:32.881 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.882 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b256e8b:::benchmark_data_smithi179_148205_object75511:head# 2024-02-09T17:57:32.883 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.883 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b268ae8:::benchmark_data_smithi179_148205_object71532:head# 2024-02-09T17:57:32.883 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.883 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b26f0e0:::benchmark_data_smithi179_148205_object24143:head# 2024-02-09T17:57:32.883 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.883 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b286ac6:::benchmark_data_smithi179_148205_object87063:head# 2024-02-09T17:57:32.883 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.883 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b2881c5:::benchmark_data_smithi179_148205_object3979:head# 2024-02-09T17:57:32.883 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.883 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b2a1497:::benchmark_data_smithi179_148205_object44349:head# 2024-02-09T17:57:32.883 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.883 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b2a6491:::benchmark_data_smithi179_148205_object5716:head# 2024-02-09T17:57:32.884 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.884 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b2a7bd1:::benchmark_data_smithi179_148205_object70809:head# 2024-02-09T17:57:32.884 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.884 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b2a9f1c:::benchmark_data_smithi179_148205_object79513:head# 2024-02-09T17:57:32.884 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.884 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b2b013f:::benchmark_data_smithi179_148205_object23688:head# 2024-02-09T17:57:32.884 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.884 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b2b0e92:::benchmark_data_smithi179_148205_object41499:head# 2024-02-09T17:57:32.884 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.884 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b2c650b:::benchmark_data_smithi179_148205_object43855:head# 2024-02-09T17:57:32.884 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.884 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b2d9b39:::benchmark_data_smithi179_148205_object55545:head# 2024-02-09T17:57:32.884 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.884 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b2e1a66:::benchmark_data_smithi179_148205_object18409:head# 2024-02-09T17:57:32.885 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.886 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b2f65e9:::benchmark_data_smithi179_148205_object101333:head# 2024-02-09T17:57:32.886 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.887 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b30b1d1:::benchmark_data_smithi179_148205_object11948:head# 2024-02-09T17:57:32.887 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.887 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b30f72a:::benchmark_data_smithi179_148205_object92405:head# 2024-02-09T17:57:32.887 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.887 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b325739:::benchmark_data_smithi179_148205_object33490:head# 2024-02-09T17:57:32.887 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.887 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b334157:::benchmark_data_smithi179_148205_object97093:head# 2024-02-09T17:57:32.887 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.887 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b33c2f9:::benchmark_data_smithi179_148205_object109038:head# 2024-02-09T17:57:32.887 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.887 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b33d214:::benchmark_data_smithi179_148205_object100027:head# 2024-02-09T17:57:32.887 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.887 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b341acb:::benchmark_data_smithi179_148205_object85614:head# 2024-02-09T17:57:32.887 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.887 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b3609cc:::benchmark_data_smithi179_148205_object82336:head# 2024-02-09T17:57:32.888 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.888 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b36a055:::benchmark_data_smithi179_148205_object32814:head# 2024-02-09T17:57:32.888 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.888 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b36ae02:::benchmark_data_smithi179_148205_object4039:head# 2024-02-09T17:57:32.888 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.888 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b385791:::benchmark_data_smithi179_148205_object90862:head# 2024-02-09T17:57:32.888 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.888 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b397237:::benchmark_data_smithi179_148205_object14751:head# 2024-02-09T17:57:32.888 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.888 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b3991d9:::benchmark_data_smithi179_148205_object19625:head# 2024-02-09T17:57:32.888 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.891 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b3a2ed3:::benchmark_data_smithi179_148205_object11298:head# 2024-02-09T17:57:32.891 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.891 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b3a7261:::benchmark_data_smithi179_148205_object84424:head# 2024-02-09T17:57:32.891 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.891 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b3a9068:::benchmark_data_smithi179_148205_object15688:head# 2024-02-09T17:57:32.891 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.891 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b3aac2f:::benchmark_data_smithi179_148205_object7563:head# 2024-02-09T17:57:32.891 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.891 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b3afb13:::benchmark_data_smithi179_148205_object60269:head# 2024-02-09T17:57:32.891 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.892 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b3afd0b:::benchmark_data_smithi179_148205_object81980:head# 2024-02-09T17:57:32.892 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.892 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b3b319e:::benchmark_data_smithi179_148205_object89311:head# 2024-02-09T17:57:32.892 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.892 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b3b9e64:::benchmark_data_smithi179_148205_object39820:head# 2024-02-09T17:57:32.892 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.892 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b3c8dec:::benchmark_data_smithi179_148205_object8812:head# 2024-02-09T17:57:32.892 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.892 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b3d308b:::benchmark_data_smithi179_148205_object44796:head# 2024-02-09T17:57:32.892 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.892 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b3d48e6:::benchmark_data_smithi179_148205_object17771:head# 2024-02-09T17:57:32.892 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.892 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b3d7b0c:::benchmark_data_smithi179_148205_object2089:head# 2024-02-09T17:57:32.892 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.892 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b3d865c:::benchmark_data_smithi179_148205_object35576:head# 2024-02-09T17:57:32.892 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.893 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b3f1d42:::benchmark_data_smithi179_148205_object45512:head# 2024-02-09T17:57:32.893 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.893 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b3f8ca8:::benchmark_data_smithi179_148205_object56649:head# 2024-02-09T17:57:32.893 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.893 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b400a58:::benchmark_data_smithi179_148205_object25741:head# 2024-02-09T17:57:32.893 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.893 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:57:32.894 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b4011d4:::benchmark_data_smithi179_148205_object55548:head# 2024-02-09T17:57:32.894 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.895 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b4046f6:::benchmark_data_smithi179_148205_object24616:head# 2024-02-09T17:57:32.895 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.895 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b406863:::benchmark_data_smithi179_148205_object61386:head# 2024-02-09T17:57:32.895 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.895 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b409561:::benchmark_data_smithi179_148205_object7189:head# 2024-02-09T17:57:32.895 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.895 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b40dc4e:::benchmark_data_smithi179_148205_object39430:head# 2024-02-09T17:57:32.895 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.895 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b414b7b:::benchmark_data_smithi179_148205_object22409:head# 2024-02-09T17:57:32.895 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.896 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b416018:::benchmark_data_smithi179_148205_object71667:head# 2024-02-09T17:57:32.897 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.897 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b4214cd:::benchmark_data_smithi179_148205_object5705:head# 2024-02-09T17:57:32.897 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.897 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b421a55:::benchmark_data_smithi179_148205_object29807:head# 2024-02-09T17:57:32.897 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.897 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b4228b6:::benchmark_data_smithi179_148205_object18944:head# 2024-02-09T17:57:32.897 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.897 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b42f84b:::benchmark_data_smithi179_148205_object58458:head# 2024-02-09T17:57:32.897 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.897 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b438936:::benchmark_data_smithi179_148205_object97923:head# 2024-02-09T17:57:32.897 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.897 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b44afdf:::benchmark_data_smithi179_148205_object97149:head# 2024-02-09T17:57:32.897 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.897 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b4502ed:::benchmark_data_smithi179_148205_object93977:head# 2024-02-09T17:57:32.898 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.898 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b46305e:::benchmark_data_smithi179_148205_object84295:head# 2024-02-09T17:57:32.898 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.898 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b4702f3:::benchmark_data_smithi179_148205_object76337:head# 2024-02-09T17:57:32.898 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.898 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b47388d:::benchmark_data_smithi179_148205_object94769:head# 2024-02-09T17:57:32.898 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.898 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b47b886:::benchmark_data_smithi179_148205_object108313:head# 2024-02-09T17:57:32.898 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.898 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b485603:::benchmark_data_smithi179_148205_object44644:head# 2024-02-09T17:57:32.898 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.898 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b49491d:::benchmark_data_smithi179_148205_object74471:head# 2024-02-09T17:57:32.898 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.898 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b4a00e3:::benchmark_data_smithi179_148205_object35832:head# 2024-02-09T17:57:32.898 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.901 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b4a6926:::benchmark_data_smithi179_148205_object4393:head# 2024-02-09T17:57:32.901 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.916 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b4aa13c:::benchmark_data_smithi179_148205_object106479:head# 2024-02-09T17:57:32.916 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.916 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b4c928f:::benchmark_data_smithi179_148205_object79424:head# 2024-02-09T17:57:32.916 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.916 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b4cbfdd:::benchmark_data_smithi179_148205_object108681:head# 2024-02-09T17:57:32.916 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.916 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b4ccb03:::benchmark_data_smithi179_148205_object91497:head# 2024-02-09T17:57:32.916 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.916 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b4e2fbb:::benchmark_data_smithi179_148205_object16173:head# 2024-02-09T17:57:32.916 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.916 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b4eb52b:::benchmark_data_smithi179_148205_object16084:head# 2024-02-09T17:57:32.916 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.916 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b4ebbc8:::benchmark_data_smithi179_148205_object22781:head# 2024-02-09T17:57:32.917 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.917 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b4ede4c:::benchmark_data_smithi179_148205_object61045:head# 2024-02-09T17:57:32.917 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.920 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b4f15a5:::benchmark_data_smithi179_148205_object104008:head# 2024-02-09T17:57:32.920 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.920 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b502157:::benchmark_data_smithi179_148205_object96925:head# 2024-02-09T17:57:32.920 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.921 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b507795:::benchmark_data_smithi179_148205_object103537:head# 2024-02-09T17:57:32.921 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.921 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b507aa2:::benchmark_data_smithi179_148205_object26710:head# 2024-02-09T17:57:32.921 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.921 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b50df2a:::benchmark_data_smithi179_148205_object89662:head# 2024-02-09T17:57:32.921 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.921 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b51a695:::benchmark_data_smithi179_148205_object25168:head# 2024-02-09T17:57:32.922 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.922 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b51fef0:::benchmark_data_smithi179_148205_object58951:head# 2024-02-09T17:57:32.922 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.922 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b525455:::benchmark_data_smithi179_148205_object52699:head# 2024-02-09T17:57:32.922 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.922 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:32.927 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b5278b1:::benchmark_data_smithi179_148205_object2443:head# 2024-02-09T17:57:32.927 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.927 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b5349a0:::benchmark_data_smithi179_148205_object5617:head# 2024-02-09T17:57:32.927 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.927 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b548d9c:::benchmark_data_smithi179_148205_object59628:head# 2024-02-09T17:57:32.927 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.927 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b54976a:::benchmark_data_smithi179_148205_object19501:head# 2024-02-09T17:57:32.927 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.927 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b54d41f:::benchmark_data_smithi179_148205_object9535:head# 2024-02-09T17:57:32.927 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.927 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b557c7a:::benchmark_data_smithi179_148205_object70357:head# 2024-02-09T17:57:32.927 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.927 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b57a1ea:::benchmark_data_smithi179_148205_object86693:head# 2024-02-09T17:57:32.927 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.927 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b58962d:::benchmark_data_smithi179_148205_object34174:head# 2024-02-09T17:57:32.928 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.928 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b5995e6:::benchmark_data_smithi179_148205_object102765:head# 2024-02-09T17:57:32.928 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.928 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b59e669:::benchmark_data_smithi179_148205_object10949:head# 2024-02-09T17:57:32.928 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.928 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b5a0a36:::benchmark_data_smithi179_148205_object87944:head# 2024-02-09T17:57:32.928 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.928 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b5a7a3d:::benchmark_data_smithi179_148205_object40297:head# 2024-02-09T17:57:32.928 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.928 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b5a8d1f:::benchmark_data_smithi179_148205_object31014:head# 2024-02-09T17:57:32.928 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.928 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b5bc69a:::benchmark_data_smithi179_148205_object105548:head# 2024-02-09T17:57:32.928 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.928 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b5be7a0:::benchmark_data_smithi179_148205_object27364:head# 2024-02-09T17:57:32.929 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.929 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b5bf930:::benchmark_data_smithi179_148205_object71166:head# 2024-02-09T17:57:32.929 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.929 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b5bfc2f:::benchmark_data_smithi179_148205_object69835:head# 2024-02-09T17:57:32.929 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.929 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b5c75d0:::benchmark_data_smithi179_148205_object40131:head# 2024-02-09T17:57:32.929 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.929 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b5d1389:::benchmark_data_smithi179_148205_object22559:head# 2024-02-09T17:57:32.929 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.929 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b5de830:::benchmark_data_smithi179_148205_object1473:head# 2024-02-09T17:57:32.929 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.929 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b5ec04c:::benchmark_data_smithi179_148205_object69780:head# 2024-02-09T17:57:32.929 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.929 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b5ec16d:::benchmark_data_smithi179_148205_object22847:head# 2024-02-09T17:57:32.931 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.931 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b5fee45:::benchmark_data_smithi179_148205_object99666:head# 2024-02-09T17:57:32.931 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.931 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b5ffbd5:::benchmark_data_smithi179_148205_object70093:head# 2024-02-09T17:57:32.931 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.931 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b60d0ad:::benchmark_data_smithi179_148205_object23469:head# 2024-02-09T17:57:32.931 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.931 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b6190e9:::benchmark_data_smithi179_148205_object71180:head# 2024-02-09T17:57:32.931 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.935 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b6280d2:::benchmark_data_smithi179_148205_object107576:head# 2024-02-09T17:57:32.935 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.935 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b629737:::benchmark_data_smithi179_148205_object60378:head# 2024-02-09T17:57:32.936 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.936 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b6303a2:::benchmark_data_smithi179_148205_object11648:head# 2024-02-09T17:57:32.936 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.936 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b643e3f:::benchmark_data_smithi179_148205_object89812:head# 2024-02-09T17:57:32.936 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.936 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b64fe57:::benchmark_data_smithi179_148205_object47964:head# 2024-02-09T17:57:32.936 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.936 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b659e56:::benchmark_data_smithi179_148205_object14863:head# 2024-02-09T17:57:32.936 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.936 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b6794a9:::benchmark_data_smithi179_148205_object57952:head# 2024-02-09T17:57:32.936 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.936 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b685b3c:::benchmark_data_smithi179_148205_object28672:head# 2024-02-09T17:57:32.936 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.936 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b687b5d:::benchmark_data_smithi179_148205_object58289:head# 2024-02-09T17:57:32.937 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.937 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b691c87:::benchmark_data_smithi179_148205_object92625:head# 2024-02-09T17:57:32.937 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.937 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b69f505:::benchmark_data_smithi179_148205_object22291:head# 2024-02-09T17:57:32.937 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.937 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b6b452f:::benchmark_data_smithi179_148205_object87843:head# 2024-02-09T17:57:32.937 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.937 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b6cd6a2:::benchmark_data_smithi179_148205_object25023:head# 2024-02-09T17:57:32.937 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.937 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b6d26f4:::benchmark_data_smithi179_148205_object61342:head# 2024-02-09T17:57:32.937 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.937 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b6e38c0:::benchmark_data_smithi179_148205_object24471:head# 2024-02-09T17:57:32.937 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.937 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b6eb5df:::benchmark_data_smithi179_148205_object53249:head# 2024-02-09T17:57:32.938 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.938 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b6ff71f:::benchmark_data_smithi179_148205_object24360:head# 2024-02-09T17:57:32.938 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.938 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b706fc5:::benchmark_data_smithi179_148205_object28088:head# 2024-02-09T17:57:32.938 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.938 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b710147:::benchmark_data_smithi179_148205_object46782:head# 2024-02-09T17:57:32.938 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.938 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b716961:::benchmark_data_smithi179_148205_object30982:head# 2024-02-09T17:57:32.938 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.938 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b71c58a:::benchmark_data_smithi179_148205_object65165:head# 2024-02-09T17:57:32.938 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.938 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b72421a:::benchmark_data_smithi179_148205_object30978:head# 2024-02-09T17:57:32.938 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.938 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b7327fa:::benchmark_data_smithi179_148205_object81191:head# 2024-02-09T17:57:32.938 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.939 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b732935:::benchmark_data_smithi179_148205_object47754:head# 2024-02-09T17:57:32.939 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.939 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b734d25:::benchmark_data_smithi179_148205_object101794:head# 2024-02-09T17:57:32.939 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.939 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b747b86:::benchmark_data_smithi179_148205_object45346:head# 2024-02-09T17:57:32.939 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.939 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b74e05e:::benchmark_data_smithi179_148205_object75353:head# 2024-02-09T17:57:32.939 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.939 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b76405f:::benchmark_data_smithi179_148205_object74649:head# 2024-02-09T17:57:32.939 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.939 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b764329:::benchmark_data_smithi179_148205_object20413:head# 2024-02-09T17:57:32.939 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.939 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b767ad9:::benchmark_data_smithi179_148205_object44498:head# 2024-02-09T17:57:32.939 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.940 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b76d21a:::benchmark_data_smithi179_148205_object29686:head# 2024-02-09T17:57:32.940 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.940 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b770529:::benchmark_data_smithi179_148205_object75413:head# 2024-02-09T17:57:32.940 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.943 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b77be67:::benchmark_data_smithi179_148205_object76275:head# 2024-02-09T17:57:32.943 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.943 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b780caf:::benchmark_data_smithi179_148205_object65408:head# 2024-02-09T17:57:32.943 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.943 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b7907e3:::benchmark_data_smithi179_148205_object38723:head# 2024-02-09T17:57:32.944 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.944 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b795551:::benchmark_data_smithi179_148205_object84987:head# 2024-02-09T17:57:32.944 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.944 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b79be9e:::benchmark_data_smithi179_148205_object28453:head# 2024-02-09T17:57:32.944 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.944 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b7a6c8f:::benchmark_data_smithi179_148205_object27289:head# 2024-02-09T17:57:32.944 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.944 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b7bda30:::benchmark_data_smithi179_148205_object40577:head# 2024-02-09T17:57:32.944 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.944 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b7de689:::benchmark_data_smithi179_148205_object22502:head# 2024-02-09T17:57:32.944 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.944 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b7f0922:::benchmark_data_smithi179_148205_object37947:head# 2024-02-09T17:57:32.944 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.944 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b7f123e:::benchmark_data_smithi179_148205_object36177:head# 2024-02-09T17:57:32.945 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.945 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b7f1dc2:::benchmark_data_smithi179_148205_object17239:head# 2024-02-09T17:57:32.945 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.945 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b802f8e:::benchmark_data_smithi179_148205_object99213:head# 2024-02-09T17:57:32.945 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.945 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b80d2aa:::benchmark_data_smithi179_148205_object106426:head# 2024-02-09T17:57:32.945 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.945 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b814af5:::benchmark_data_smithi179_148205_object19001:head# 2024-02-09T17:57:32.945 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.945 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b81c49d:::benchmark_data_smithi179_148205_object79477:head# 2024-02-09T17:57:32.945 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.945 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b82a913:::benchmark_data_smithi179_148205_object93147:head# 2024-02-09T17:57:32.945 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.945 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b8336fc:::benchmark_data_smithi179_148205_object17616:head# 2024-02-09T17:57:32.946 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.946 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b839db8:::benchmark_data_smithi179_148205_object99651:head# 2024-02-09T17:57:32.946 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.946 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b8488b4:::benchmark_data_smithi179_148205_object94126:head# 2024-02-09T17:57:32.946 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.946 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b84d860:::benchmark_data_smithi179_148205_object31881:head# 2024-02-09T17:57:32.946 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.946 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b862c78:::benchmark_data_smithi179_148205_object9094:head# 2024-02-09T17:57:32.946 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.946 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b87c5db:::benchmark_data_smithi179_148205_object92204:head# 2024-02-09T17:57:32.946 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.948 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b887729:::benchmark_data_smithi179_148205_object58490:head# 2024-02-09T17:57:32.948 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.948 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b88925d:::benchmark_data_smithi179_148205_object79712:head# 2024-02-09T17:57:32.948 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.949 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b88e97b:::benchmark_data_smithi179_148205_object6101:head# 2024-02-09T17:57:32.949 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.949 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b894fd6:::benchmark_data_smithi179_148205_object32351:head# 2024-02-09T17:57:32.949 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.949 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b89f2d6:::benchmark_data_smithi179_148205_object10658:head# 2024-02-09T17:57:32.949 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.949 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b8a4fd2:::benchmark_data_smithi179_148205_object86352:head# 2024-02-09T17:57:32.949 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.949 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b8a91d7:::benchmark_data_smithi179_148205_object69477:head# 2024-02-09T17:57:32.949 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.950 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:57:32.957 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:32.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:57:32.963 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b8c67a9:::benchmark_data_smithi179_148205_object18588:head# 2024-02-09T17:57:32.963 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.963 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b8c7034:::benchmark_data_smithi179_148205_object85303:head# 2024-02-09T17:57:32.963 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.963 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b8e5f67:::benchmark_data_smithi179_148205_object13296:head# 2024-02-09T17:57:32.963 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.963 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b8e78f3:::benchmark_data_smithi179_148205_object89876:head# 2024-02-09T17:57:32.963 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.963 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b8ea129:::benchmark_data_smithi179_148205_object95137:head# 2024-02-09T17:57:32.964 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.964 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b8ef2d8:::benchmark_data_smithi179_148205_object6158:head# 2024-02-09T17:57:32.964 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.964 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b904751:::benchmark_data_smithi179_148205_object81282:head# 2024-02-09T17:57:32.964 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.964 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b91dda0:::benchmark_data_smithi179_148205_object101189:head# 2024-02-09T17:57:32.964 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.964 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b9207c9:::benchmark_data_smithi179_148205_object90497:head# 2024-02-09T17:57:32.964 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.964 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b92a6d7:::benchmark_data_smithi179_148205_object13041:head# 2024-02-09T17:57:32.964 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.964 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b949e9d:::benchmark_data_smithi179_148205_object9292:head# 2024-02-09T17:57:32.965 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.965 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b955e13:::benchmark_data_smithi179_148205_object98611:head# 2024-02-09T17:57:32.965 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.965 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b961ac7:::benchmark_data_smithi179_148205_object79062:head# 2024-02-09T17:57:32.965 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.965 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b96d90f:::benchmark_data_smithi179_148205_object94684:head# 2024-02-09T17:57:32.965 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.965 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b97d02f:::benchmark_data_smithi179_148205_object79924:head# 2024-02-09T17:57:32.965 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.965 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b97dd58:::benchmark_data_smithi179_148205_object5521:head# 2024-02-09T17:57:32.965 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.965 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b9863d8:::benchmark_data_smithi179_148205_object97785:head# 2024-02-09T17:57:32.965 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.965 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b98ae97:::benchmark_data_smithi179_148205_object103679:head# 2024-02-09T17:57:32.965 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.966 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b98fc70:::benchmark_data_smithi179_148205_object25810:head# 2024-02-09T17:57:32.966 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.966 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b9913f2:::benchmark_data_smithi179_148205_object4850:head# 2024-02-09T17:57:32.966 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.966 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b9ab1f6:::benchmark_data_smithi179_148205_object56197:head# 2024-02-09T17:57:32.966 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.966 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b9b47ad:::benchmark_data_smithi179_148205_object99322:head# 2024-02-09T17:57:32.966 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.966 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b9b506f:::benchmark_data_smithi179_148205_object25085:head# 2024-02-09T17:57:32.966 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.966 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b9bc2ed:::benchmark_data_smithi179_148205_object94049:head# 2024-02-09T17:57:32.966 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.966 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b9cf175:::benchmark_data_smithi179_148205_object5254:head# 2024-02-09T17:57:32.966 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.967 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b9d8f10:::benchmark_data_smithi179_148205_object45674:head# 2024-02-09T17:57:32.967 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.967 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b9e06a7:::benchmark_data_smithi179_148205_object89510:head# 2024-02-09T17:57:32.967 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.967 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b9efc90:::benchmark_data_smithi179_148205_object2087:head# 2024-02-09T17:57:32.967 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.967 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b9f07d2:::benchmark_data_smithi179_148205_object61707:head# 2024-02-09T17:57:32.967 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.967 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1b9f12bc:::benchmark_data_smithi179_148205_object60657:head# 2024-02-09T17:57:32.967 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.967 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ba04c75:::benchmark_data_smithi179_148205_object21755:head# 2024-02-09T17:57:32.967 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.967 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ba12dea:::benchmark_data_smithi179_148205_object5644:head# 2024-02-09T17:57:32.968 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.968 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ba16b04:::benchmark_data_smithi179_148205_object52065:head# 2024-02-09T17:57:32.968 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.968 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ba20836:::benchmark_data_smithi179_148205_object73146:head# 2024-02-09T17:57:32.968 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.968 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ba3077f:::benchmark_data_smithi179_148205_object72797:head# 2024-02-09T17:57:32.968 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.968 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ba3cfd0:::benchmark_data_smithi179_148205_object80279:head# 2024-02-09T17:57:32.968 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.968 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ba3da26:::benchmark_data_smithi179_148205_object61998:head# 2024-02-09T17:57:32.968 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.968 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ba443f5:::benchmark_data_smithi179_148205_object30210:head# 2024-02-09T17:57:32.968 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.968 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ba50564:::benchmark_data_smithi179_148205_object42478:head# 2024-02-09T17:57:32.969 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.969 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ba64be9:::benchmark_data_smithi179_148205_object74219:head# 2024-02-09T17:57:32.969 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.969 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ba65709:::benchmark_data_smithi179_148205_object57943:head# 2024-02-09T17:57:32.969 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.969 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ba739e5:::benchmark_data_smithi179_148205_object40380:head# 2024-02-09T17:57:32.969 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.969 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ba73bbd:::benchmark_data_smithi179_148205_object86655:head# 2024-02-09T17:57:32.969 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.969 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ba816c3:::benchmark_data_smithi179_148205_object91980:head# 2024-02-09T17:57:32.969 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.969 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ba82fe1:::benchmark_data_smithi179_148205_object82836:head# 2024-02-09T17:57:32.969 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.969 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ba84f40:::benchmark_data_smithi179_148205_object47583:head# 2024-02-09T17:57:32.969 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.969 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ba8543f:::benchmark_data_smithi179_148205_object6311:head# 2024-02-09T17:57:32.970 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.970 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ba955df:::benchmark_data_smithi179_148205_object28283:head# 2024-02-09T17:57:32.970 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.970 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ba984c6:::benchmark_data_smithi179_148205_object3707:head# 2024-02-09T17:57:32.970 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.970 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ba9e507:::benchmark_data_smithi179_148205_object86840:head# 2024-02-09T17:57:32.970 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.970 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1baa09b1:::benchmark_data_smithi179_148205_object60541:head# 2024-02-09T17:57:32.970 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.970 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1baa7cf5:::benchmark_data_smithi179_148205_object48197:head# 2024-02-09T17:57:32.970 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.970 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bab9ec4:::benchmark_data_smithi179_148205_object30249:head# 2024-02-09T17:57:32.970 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.970 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1babc862:::benchmark_data_smithi179_148205_object107524:head# 2024-02-09T17:57:32.970 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.971 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bac2c0b:::benchmark_data_smithi179_148205_object19132:head# 2024-02-09T17:57:32.971 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.971 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bac6c2b:::benchmark_data_smithi179_148205_object48267:head# 2024-02-09T17:57:32.971 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.971 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bad4b3a:::benchmark_data_smithi179_148205_object8500:head# 2024-02-09T17:57:32.971 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.971 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bada82b:::benchmark_data_smithi179_148205_object86713:head# 2024-02-09T17:57:32.971 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.971 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bae09cd:::benchmark_data_smithi179_148205_object48480:head# 2024-02-09T17:57:32.971 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.971 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bae1ace:::benchmark_data_smithi179_148205_object77642:head# 2024-02-09T17:57:32.971 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.971 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bae22e2:::benchmark_data_smithi179_148205_object96630:head# 2024-02-09T17:57:32.971 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.971 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bae2421:::benchmark_data_smithi179_148205_object27535:head# 2024-02-09T17:57:32.972 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.972 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bb0f816:::benchmark_data_smithi179_148205_object92355:head# 2024-02-09T17:57:32.972 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.972 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bb18898:::benchmark_data_smithi179_148205_object97641:head# 2024-02-09T17:57:32.972 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.972 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bb18cb7:::benchmark_data_smithi179_148205_object1317:head# 2024-02-09T17:57:32.972 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.972 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bb231b9:::benchmark_data_smithi179_148205_object44556:head# 2024-02-09T17:57:32.972 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.972 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bb26871:::benchmark_data_smithi179_148205_object72419:head# 2024-02-09T17:57:32.972 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.972 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bb3bf2d:::benchmark_data_smithi179_148205_object84818:head# 2024-02-09T17:57:32.972 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.972 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bb3f43d:::benchmark_data_smithi179_148205_object30357:head# 2024-02-09T17:57:32.973 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.973 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bb46ffe:::benchmark_data_smithi179_148205_object39910:head# 2024-02-09T17:57:32.973 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.973 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bb5c56d:::benchmark_data_smithi179_148205_object66209:head# 2024-02-09T17:57:32.973 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.978 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bb6f3ad:::benchmark_data_smithi179_148205_object94803:head# 2024-02-09T17:57:32.978 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.978 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bb742fb:::benchmark_data_smithi179_148205_object51789:head# 2024-02-09T17:57:32.978 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.978 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bb75fae:::benchmark_data_smithi179_148205_object68331:head# 2024-02-09T17:57:32.978 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.978 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bb84038:::benchmark_data_smithi179_148205_object57210:head# 2024-02-09T17:57:32.978 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.978 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bb84233:::benchmark_data_smithi179_148205_object99840:head# 2024-02-09T17:57:32.978 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.978 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bb90ddc:::benchmark_data_smithi179_148205_object60626:head# 2024-02-09T17:57:32.978 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.978 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bba230f:::benchmark_data_smithi179_148205_object79283:head# 2024-02-09T17:57:32.979 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.979 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bba3f38:::benchmark_data_smithi179_148205_object88386:head# 2024-02-09T17:57:32.979 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.979 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bbd5dad:::benchmark_data_smithi179_148205_object75020:head# 2024-02-09T17:57:32.979 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.979 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bbd7ffd:::benchmark_data_smithi179_148205_object57292:head# 2024-02-09T17:57:32.979 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.979 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bbe7f8f:::benchmark_data_smithi179_148205_object13583:head# 2024-02-09T17:57:32.979 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.979 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bbe9651:::benchmark_data_smithi179_148205_object48906:head# 2024-02-09T17:57:32.979 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.979 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bbfbe4c:::benchmark_data_smithi179_148205_object25857:head# 2024-02-09T17:57:32.979 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.979 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bc02813:::benchmark_data_smithi179_148205_object6670:head# 2024-02-09T17:57:32.979 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.980 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bc0387c:::benchmark_data_smithi179_148205_object64961:head# 2024-02-09T17:57:32.980 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.980 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bc11ca2:::benchmark_data_smithi179_148205_object12459:head# 2024-02-09T17:57:32.980 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.980 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bc13249:::benchmark_data_smithi179_148205_object98113:head# 2024-02-09T17:57:32.980 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.980 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bc14f23:::benchmark_data_smithi179_148205_object99335:head# 2024-02-09T17:57:32.980 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.980 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bc1e9f2:::benchmark_data_smithi179_148205_object81317:head# 2024-02-09T17:57:32.980 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.980 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bc26cc9:::benchmark_data_smithi179_148205_object42721:head# 2024-02-09T17:57:32.980 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.980 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bc55ebb:::benchmark_data_smithi179_148205_object91046:head# 2024-02-09T17:57:32.980 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.981 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bc5a96a:::benchmark_data_smithi179_148205_object3905:head# 2024-02-09T17:57:32.981 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.981 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bc5ba18:::benchmark_data_smithi179_148205_object11242:head# 2024-02-09T17:57:32.981 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.981 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bc7626f:::benchmark_data_smithi179_148205_object35357:head# 2024-02-09T17:57:32.981 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.981 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bc7cde7:::benchmark_data_smithi179_148205_object38203:head# 2024-02-09T17:57:32.981 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.981 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bc83123:::benchmark_data_smithi179_148205_object64087:head# 2024-02-09T17:57:32.981 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.981 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bc856cb:::benchmark_data_smithi179_148205_object14687:head# 2024-02-09T17:57:32.981 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.981 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bc9328a:::benchmark_data_smithi179_148205_object18778:head# 2024-02-09T17:57:32.981 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.981 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bca4d04:::benchmark_data_smithi179_148205_object33383:head# 2024-02-09T17:57:32.982 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.982 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bcab590:::benchmark_data_smithi179_148205_object81081:head# 2024-02-09T17:57:32.982 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.982 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bcaf594:::benchmark_data_smithi179_148205_object33103:head# 2024-02-09T17:57:32.982 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.982 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bcb87e7:::benchmark_data_smithi179_148205_object53785:head# 2024-02-09T17:57:32.982 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.982 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bcb9fb2:::benchmark_data_smithi179_148205_object106715:head# 2024-02-09T17:57:32.982 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.982 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bcbc0cd:::benchmark_data_smithi179_148205_object42948:head# 2024-02-09T17:57:32.982 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.982 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bcc0e07:::benchmark_data_smithi179_148205_object16215:head# 2024-02-09T17:57:32.982 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.983 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bccd19a:::benchmark_data_smithi179_148205_object21409:head# 2024-02-09T17:57:32.983 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.983 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bccf2fe:::benchmark_data_smithi179_148205_object19834:head# 2024-02-09T17:57:32.983 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.983 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bce0136:::benchmark_data_smithi179_148205_object35144:head# 2024-02-09T17:57:32.983 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.983 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bce083d:::benchmark_data_smithi179_148205_object12225:head# 2024-02-09T17:57:32.983 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.983 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bce3c3f:::benchmark_data_smithi179_148205_object107124:head# 2024-02-09T17:57:32.983 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.983 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bce6b0c:::benchmark_data_smithi179_148205_object17436:head# 2024-02-09T17:57:32.983 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.984 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bd04123:::benchmark_data_smithi179_148205_object37648:head# 2024-02-09T17:57:32.984 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.984 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bd0dd8c:::benchmark_data_smithi179_148205_object12760:head# 2024-02-09T17:57:32.984 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.984 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bd1b56e:::benchmark_data_smithi179_148205_object109106:head# 2024-02-09T17:57:32.984 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.984 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bd2550d:::benchmark_data_smithi179_148205_object73466:head# 2024-02-09T17:57:32.984 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.984 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bd2b98b:::benchmark_data_smithi179_148205_object54867:head# 2024-02-09T17:57:32.984 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.984 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bd303be:::benchmark_data_smithi179_148205_object6197:head# 2024-02-09T17:57:32.984 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.984 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bd30646:::benchmark_data_smithi179_148205_object72113:head# 2024-02-09T17:57:32.984 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.984 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bd3d716:::benchmark_data_smithi179_148205_object96103:head# 2024-02-09T17:57:32.985 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.985 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bd470f1:::benchmark_data_smithi179_148205_object35384:head# 2024-02-09T17:57:32.985 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.985 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bd4c583:::benchmark_data_smithi179_148205_object67494:head# 2024-02-09T17:57:32.985 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.985 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bd4f7ec:::benchmark_data_smithi179_148205_object103239:head# 2024-02-09T17:57:32.985 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.985 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bd51d49:::benchmark_data_smithi179_148205_object89840:head# 2024-02-09T17:57:32.985 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.985 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bd5cf90:::benchmark_data_smithi179_148205_object98788:head# 2024-02-09T17:57:32.986 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.986 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bd6b693:::benchmark_data_smithi179_148205_object28075:head# 2024-02-09T17:57:32.986 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.987 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bd7a770:::benchmark_data_smithi179_148205_object8191:head# 2024-02-09T17:57:32.987 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.988 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bd84d27:::benchmark_data_smithi179_148205_object104096:head# 2024-02-09T17:57:32.988 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.988 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bd89d5e:::benchmark_data_smithi179_148205_object14380:head# 2024-02-09T17:57:32.988 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.988 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bd8ce5e:::benchmark_data_smithi179_148205_object75182:head# 2024-02-09T17:57:32.988 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.989 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bd955e5:::benchmark_data_smithi179_148205_object44031:head# 2024-02-09T17:57:32.989 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.990 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bda4013:::benchmark_data_smithi179_148205_object18307:head# 2024-02-09T17:57:32.990 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.990 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bdbd51a:::benchmark_data_smithi179_148205_object96241:head# 2024-02-09T17:57:32.990 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.990 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bdc1d97:::benchmark_data_smithi179_148205_object37981:head# 2024-02-09T17:57:32.990 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.990 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bdc583b:::benchmark_data_smithi179_148205_object71431:head# 2024-02-09T17:57:32.990 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.990 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bdd7a97:::benchmark_data_smithi179_148205_object65292:head# 2024-02-09T17:57:32.990 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.990 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bdf6c7f:::benchmark_data_smithi179_148205_object60220:head# 2024-02-09T17:57:32.990 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.990 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1be0a298:::benchmark_data_smithi179_148205_object40988:head# 2024-02-09T17:57:32.990 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.992 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1be0eff6:::benchmark_data_smithi179_148205_object35284:head# 2024-02-09T17:57:32.992 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.992 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1be15b84:::benchmark_data_smithi179_148205_object106882:head# 2024-02-09T17:57:32.992 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.992 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1be2b80f:::benchmark_data_smithi179_148205_object95052:head# 2024-02-09T17:57:32.992 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.993 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1be2f74f:::benchmark_data_smithi179_148205_object40135:head# 2024-02-09T17:57:32.993 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.993 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1be37395:::benchmark_data_smithi179_148205_object54660:head# 2024-02-09T17:57:32.993 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.993 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1be3eab2:::benchmark_data_smithi179_148205_object100756:head# 2024-02-09T17:57:32.993 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.993 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1be402a1:::benchmark_data_smithi179_148205_object108443:head# 2024-02-09T17:57:32.993 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.993 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1be5fd80:::benchmark_data_smithi179_148205_object29378:head# 2024-02-09T17:57:32.993 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.993 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1be63f44:::benchmark_data_smithi179_148205_object68285:head# 2024-02-09T17:57:32.993 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.993 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1be67821:::benchmark_data_smithi179_148205_object42034:head# 2024-02-09T17:57:32.993 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.998 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1be7b29d:::benchmark_data_smithi179_148205_object69494:head# 2024-02-09T17:57:32.998 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.998 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1be80dd0:::benchmark_data_smithi179_148205_object21583:head# 2024-02-09T17:57:32.999 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.999 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1be83219:::benchmark_data_smithi179_148205_object84491:head# 2024-02-09T17:57:32.999 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.999 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1be84891:::benchmark_data_smithi179_148205_object53938:head# 2024-02-09T17:57:32.999 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.999 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1be8fec6:::benchmark_data_smithi179_148205_object95536:head# 2024-02-09T17:57:32.999 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.999 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1be92728:::benchmark_data_smithi179_148205_object67534:head# 2024-02-09T17:57:32.999 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.999 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bea6c3c:::benchmark_data_smithi179_148205_object43499:head# 2024-02-09T17:57:32.999 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.999 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bea965e:::benchmark_data_smithi179_148205_object83421:head# 2024-02-09T17:57:32.999 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:32.999 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bec500a:::benchmark_data_smithi179_148205_object98798:head# 2024-02-09T17:57:33.000 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.000 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bec966a:::benchmark_data_smithi179_148205_object77724:head# 2024-02-09T17:57:33.000 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.000 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bed1362:::benchmark_data_smithi179_148205_object34033:head# 2024-02-09T17:57:33.000 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.000 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bee60b2:::benchmark_data_smithi179_148205_object24574:head# 2024-02-09T17:57:33.000 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.000 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1beed777:::benchmark_data_smithi179_148205_object13898:head# 2024-02-09T17:57:33.000 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.000 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bef1c49:::benchmark_data_smithi179_148205_object66723:head# 2024-02-09T17:57:33.000 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.000 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bef2fee:::benchmark_data_smithi179_148205_object100473:head# 2024-02-09T17:57:33.000 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.000 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1befa7c5:::benchmark_data_smithi179_148205_object3959:head# 2024-02-09T17:57:33.001 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.001 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bf0ded5:::benchmark_data_smithi179_148205_object98268:head# 2024-02-09T17:57:33.001 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.001 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bf22501:::benchmark_data_smithi179_148205_object108020:head# 2024-02-09T17:57:33.001 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.001 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bf29824:::benchmark_data_smithi179_148205_object26129:head# 2024-02-09T17:57:33.001 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.001 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bf2a166:::benchmark_data_smithi179_148205_object86064:head# 2024-02-09T17:57:33.001 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.001 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bf38a28:::benchmark_data_smithi179_148205_object69251:head# 2024-02-09T17:57:33.001 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.001 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bf3ff77:::benchmark_data_smithi179_148205_object65912:head# 2024-02-09T17:57:33.001 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.001 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bf42e83:::benchmark_data_smithi179_148205_object106674:head# 2024-02-09T17:57:33.002 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.002 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bf461c0:::benchmark_data_smithi179_148205_object41902:head# 2024-02-09T17:57:33.002 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.002 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bf6174f:::benchmark_data_smithi179_148205_object68461:head# 2024-02-09T17:57:33.002 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.002 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bf96800:::benchmark_data_smithi179_148205_object87131:head# 2024-02-09T17:57:33.002 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.007 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bfa06d7:::benchmark_data_smithi179_148205_object102570:head# 2024-02-09T17:57:33.008 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.008 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bfa4f4c:::benchmark_data_smithi179_148205_object24108:head# 2024-02-09T17:57:33.008 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.008 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bfaef07:::benchmark_data_smithi179_148205_object4214:head# 2024-02-09T17:57:33.008 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.008 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bfb0419:::benchmark_data_smithi179_148205_object53912:head# 2024-02-09T17:57:33.008 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.008 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bfbd9fa:::benchmark_data_smithi179_148205_object45027:head# 2024-02-09T17:57:33.008 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.008 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bfc6381:::benchmark_data_smithi179_148205_object98883:head# 2024-02-09T17:57:33.008 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.008 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bfd982f:::benchmark_data_smithi179_148205_object53086:head# 2024-02-09T17:57:33.008 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.008 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bfde6c5:::benchmark_data_smithi179_148205_object49230:head# 2024-02-09T17:57:33.009 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.009 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bfe4d81:::benchmark_data_smithi179_148205_object1043:head# 2024-02-09T17:57:33.009 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.009 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1bffe540:::benchmark_data_smithi179_148205_object81802:head# 2024-02-09T17:57:33.009 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.009 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c00190e:::benchmark_data_smithi179_148205_object61711:head# 2024-02-09T17:57:33.009 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.009 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c014070:::benchmark_data_smithi179_148205_object73057:head# 2024-02-09T17:57:33.009 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.009 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c01a9dc:::benchmark_data_smithi179_148205_object62314:head# 2024-02-09T17:57:33.009 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.009 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c022a32:::benchmark_data_smithi179_148205_object39768:head# 2024-02-09T17:57:33.009 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.009 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c02abd0:::benchmark_data_smithi179_148205_object89505:head# 2024-02-09T17:57:33.010 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.010 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c047dd5:::benchmark_data_smithi179_148205_object59680:head# 2024-02-09T17:57:33.010 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.010 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c05fd57:::benchmark_data_smithi179_148205_object104240:head# 2024-02-09T17:57:33.010 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.010 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c06635e:::benchmark_data_smithi179_148205_object86690:head# 2024-02-09T17:57:33.010 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.010 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c0695e3:::benchmark_data_smithi179_148205_object10111:head# 2024-02-09T17:57:33.010 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.010 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c06d87f:::benchmark_data_smithi179_148205_object31590:head# 2024-02-09T17:57:33.010 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.010 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c06fa4a:::benchmark_data_smithi179_148205_object108659:head# 2024-02-09T17:57:33.010 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.010 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c071ee7:::benchmark_data_smithi179_148205_object25243:head# 2024-02-09T17:57:33.011 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.011 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c07714c:::benchmark_data_smithi179_148205_object30489:head# 2024-02-09T17:57:33.011 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.011 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c086024:::benchmark_data_smithi179_148205_object100334:head# 2024-02-09T17:57:33.011 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.011 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c089995:::benchmark_data_smithi179_148205_object83550:head# 2024-02-09T17:57:33.011 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.011 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c0916bb:::benchmark_data_smithi179_148205_object90328:head# 2024-02-09T17:57:33.011 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.011 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c09bef4:::benchmark_data_smithi179_148205_object9884:head# 2024-02-09T17:57:33.011 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.011 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c0be9f4:::benchmark_data_smithi179_148205_object7410:head# 2024-02-09T17:57:33.011 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.011 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c0cdb40:::benchmark_data_smithi179_148205_object16007:head# 2024-02-09T17:57:33.011 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.011 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c0d8d84:::benchmark_data_smithi179_148205_object109482:head# 2024-02-09T17:57:33.012 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.012 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c0e16a6:::benchmark_data_smithi179_148205_object98336:head# 2024-02-09T17:57:33.012 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.012 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c0e43fb:::benchmark_data_smithi179_148205_object95658:head# 2024-02-09T17:57:33.012 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.012 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c0edfb6:::benchmark_data_smithi179_148205_object44970:head# 2024-02-09T17:57:33.012 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.012 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c0ee4d6:::benchmark_data_smithi179_148205_object24657:head# 2024-02-09T17:57:33.012 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.012 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c10553f:::benchmark_data_smithi179_148205_object95930:head# 2024-02-09T17:57:33.012 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.016 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c12363a:::benchmark_data_smithi179_148205_object27373:head# 2024-02-09T17:57:33.016 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.016 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c12c838:::benchmark_data_smithi179_148205_object27013:head# 2024-02-09T17:57:33.017 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.017 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c1375df:::benchmark_data_smithi179_148205_object80662:head# 2024-02-09T17:57:33.017 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.017 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c138b79:::benchmark_data_smithi179_148205_object92325:head# 2024-02-09T17:57:33.017 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.017 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c14fc1f:::benchmark_data_smithi179_148205_object61640:head# 2024-02-09T17:57:33.017 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.017 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c162499:::benchmark_data_smithi179_148205_object36109:head# 2024-02-09T17:57:33.017 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.017 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c16fc34:::benchmark_data_smithi179_148205_object40972:head# 2024-02-09T17:57:33.017 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.017 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c180ecf:::benchmark_data_smithi179_148205_object22570:head# 2024-02-09T17:57:33.018 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.018 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c1860f3:::benchmark_data_smithi179_148205_object106527:head# 2024-02-09T17:57:33.018 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.018 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c1912d8:::benchmark_data_smithi179_148205_object81260:head# 2024-02-09T17:57:33.018 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.018 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c19258f:::benchmark_data_smithi179_148205_object63555:head# 2024-02-09T17:57:33.018 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.018 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c1a49d4:::benchmark_data_smithi179_148205_object27125:head# 2024-02-09T17:57:33.019 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.019 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c1a548f:::benchmark_data_smithi179_148205_object100704:head# 2024-02-09T17:57:33.019 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.019 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c1b37c9:::benchmark_data_smithi179_148205_object45403:head# 2024-02-09T17:57:33.019 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.019 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c1bb4cf:::benchmark_data_smithi179_148205_object101878:head# 2024-02-09T17:57:33.019 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.019 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c1c0b8e:::benchmark_data_smithi179_148205_object95608:head# 2024-02-09T17:57:33.020 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.020 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c1e7f66:::benchmark_data_smithi179_148205_object36019:head# 2024-02-09T17:57:33.020 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.020 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c2024f3:::benchmark_data_smithi179_148205_object56699:head# 2024-02-09T17:57:33.020 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.020 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c2160f3:::benchmark_data_smithi179_148205_object16712:head# 2024-02-09T17:57:33.020 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.021 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c216146:::benchmark_data_smithi179_148205_object100801:head# 2024-02-09T17:57:33.021 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.021 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c228eae:::benchmark_data_smithi179_148205_object64800:head# 2024-02-09T17:57:33.021 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.021 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c231ba3:::benchmark_data_smithi179_148205_object106385:head# 2024-02-09T17:57:33.021 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.021 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c23505d:::benchmark_data_smithi179_148205_object50415:head# 2024-02-09T17:57:33.022 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.022 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c2354c6:::benchmark_data_smithi179_148205_object108420:head# 2024-02-09T17:57:33.022 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.022 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c2404f5:::benchmark_data_smithi179_148205_object31397:head# 2024-02-09T17:57:33.022 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.022 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c24e100:::benchmark_data_smithi179_148205_object7681:head# 2024-02-09T17:57:33.022 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.022 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c255362:::benchmark_data_smithi179_148205_object63844:head# 2024-02-09T17:57:33.022 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.023 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c256e5f:::benchmark_data_smithi179_148205_object60348:head# 2024-02-09T17:57:33.023 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.023 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c25b072:::benchmark_data_smithi179_148205_object108637:head# 2024-02-09T17:57:33.023 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.023 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c25b128:::benchmark_data_smithi179_148205_object11287:head# 2024-02-09T17:57:33.023 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.023 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c26b9db:::benchmark_data_smithi179_148205_object47464:head# 2024-02-09T17:57:33.023 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.023 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c26d724:::benchmark_data_smithi179_148205_object66946:head# 2024-02-09T17:57:33.023 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.027 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c271130:::benchmark_data_smithi179_148205_object81862:head# 2024-02-09T17:57:33.027 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.027 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c288888:::benchmark_data_smithi179_148205_object75171:head# 2024-02-09T17:57:33.027 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.027 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c28f9c4:::benchmark_data_smithi179_148205_object94886:head# 2024-02-09T17:57:33.028 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.028 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c29097f:::benchmark_data_smithi179_148205_object21965:head# 2024-02-09T17:57:33.028 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.028 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c2979aa:::benchmark_data_smithi179_148205_object45781:head# 2024-02-09T17:57:33.028 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.028 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c2bb0a8:::benchmark_data_smithi179_148205_object21971:head# 2024-02-09T17:57:33.028 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.028 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c2c11f2:::benchmark_data_smithi179_148205_object29962:head# 2024-02-09T17:57:33.028 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.028 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c2c8d15:::benchmark_data_smithi179_148205_object61087:head# 2024-02-09T17:57:33.028 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.028 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c2d6cd3:::benchmark_data_smithi179_148205_object93274:head# 2024-02-09T17:57:33.028 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.029 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c2dc84f:::benchmark_data_smithi179_148205_object62715:head# 2024-02-09T17:57:33.029 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.029 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c2dd4d8:::benchmark_data_smithi179_148205_object44844:head# 2024-02-09T17:57:33.029 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.029 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c2ebd84:::benchmark_data_smithi179_148205_object74245:head# 2024-02-09T17:57:33.029 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.029 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c2fb1a0:::benchmark_data_smithi179_148205_object71279:head# 2024-02-09T17:57:33.029 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.029 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c30161d:::benchmark_data_smithi179_148205_object70097:head# 2024-02-09T17:57:33.029 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.029 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c3035f2:::benchmark_data_smithi179_148205_object2973:head# 2024-02-09T17:57:33.029 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.029 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c30e131:::benchmark_data_smithi179_148205_object103930:head# 2024-02-09T17:57:33.029 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.029 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c30f858:::benchmark_data_smithi179_148205_object52453:head# 2024-02-09T17:57:33.030 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.030 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c314b4c:::benchmark_data_smithi179_148205_object50463:head# 2024-02-09T17:57:33.030 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.030 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c32c633:::benchmark_data_smithi179_148205_object23539:head# 2024-02-09T17:57:33.030 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.030 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c32dd9f:::benchmark_data_smithi179_148205_object54617:head# 2024-02-09T17:57:33.030 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.030 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c32ea50:::benchmark_data_smithi179_148205_object101382:head# 2024-02-09T17:57:33.030 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.030 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c32fd1f:::benchmark_data_smithi179_148205_object94339:head# 2024-02-09T17:57:33.030 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.030 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c33b06c:::benchmark_data_smithi179_148205_object96236:head# 2024-02-09T17:57:33.030 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.030 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c348e83:::benchmark_data_smithi179_148205_object64841:head# 2024-02-09T17:57:33.030 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.031 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c35783c:::benchmark_data_smithi179_148205_object56933:head# 2024-02-09T17:57:33.031 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.031 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c35a3f4:::benchmark_data_smithi179_148205_object637:head# 2024-02-09T17:57:33.031 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.031 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c35d628:::benchmark_data_smithi179_148205_object55635:head# 2024-02-09T17:57:33.031 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.031 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c360187:::benchmark_data_smithi179_148205_object103777:head# 2024-02-09T17:57:33.031 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.031 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c37aacf:::benchmark_data_smithi179_148205_object23979:head# 2024-02-09T17:57:33.031 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.033 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c384509:::benchmark_data_smithi179_148205_object40386:head# 2024-02-09T17:57:33.033 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.033 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c389fc5:::benchmark_data_smithi179_148205_object69465:head# 2024-02-09T17:57:33.033 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.033 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c399e3c:::benchmark_data_smithi179_148205_object109062:head# 2024-02-09T17:57:33.033 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.033 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c39f17d:::benchmark_data_smithi179_148205_object8342:head# 2024-02-09T17:57:33.033 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.034 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c3a3d1a:::benchmark_data_smithi179_148205_object72423:head# 2024-02-09T17:57:33.034 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.034 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c3a52e6:::benchmark_data_smithi179_148205_object63802:head# 2024-02-09T17:57:33.034 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.034 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c3a6771:::benchmark_data_smithi179_148205_object87850:head# 2024-02-09T17:57:33.034 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.034 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c3a7d33:::benchmark_data_smithi179_148205_object91880:head# 2024-02-09T17:57:33.034 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.034 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c3b0da3:::benchmark_data_smithi179_148205_object42633:head# 2024-02-09T17:57:33.034 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.034 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c3b8001:::benchmark_data_smithi179_148205_object23569:head# 2024-02-09T17:57:33.034 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.034 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c3cbcfe:::benchmark_data_smithi179_148205_object5703:head# 2024-02-09T17:57:33.034 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.035 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c3d4971:::benchmark_data_smithi179_148205_object101613:head# 2024-02-09T17:57:33.035 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.035 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c3ef799:::benchmark_data_smithi179_148205_object36796:head# 2024-02-09T17:57:33.035 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.035 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c3f9ee1:::benchmark_data_smithi179_148205_object33455:head# 2024-02-09T17:57:33.035 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.036 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c4019c8:::benchmark_data_smithi179_148205_object91524:head# 2024-02-09T17:57:33.036 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.036 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c405952:::benchmark_data_smithi179_148205_object46100:head# 2024-02-09T17:57:33.036 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.036 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c409074:::benchmark_data_smithi179_148205_object85456:head# 2024-02-09T17:57:33.036 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.036 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c416a94:::benchmark_data_smithi179_148205_object30445:head# 2024-02-09T17:57:33.036 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.037 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c41a821:::benchmark_data_smithi179_148205_object36391:head# 2024-02-09T17:57:33.037 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.037 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c41baec:::benchmark_data_smithi179_148205_object95765:head# 2024-02-09T17:57:33.037 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.040 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c4219ed:::benchmark_data_smithi179_148205_object71714:head# 2024-02-09T17:57:33.040 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.040 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c422b3c:::benchmark_data_smithi179_148205_object93511:head# 2024-02-09T17:57:33.040 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.040 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c42cb42:::benchmark_data_smithi179_148205_object67702:head# 2024-02-09T17:57:33.040 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.040 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c432985:::benchmark_data_smithi179_148205_object95066:head# 2024-02-09T17:57:33.040 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.040 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c43d63b:::benchmark_data_smithi179_148205_object100359:head# 2024-02-09T17:57:33.040 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.040 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c43f667:::benchmark_data_smithi179_148205_object15215:head# 2024-02-09T17:57:33.040 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.048 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c44e061:::benchmark_data_smithi179_148205_object50416:head# 2024-02-09T17:57:33.048 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.051 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c44ea44:::benchmark_data_smithi179_148205_object106461:head# 2024-02-09T17:57:33.051 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.057 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c459a2a:::benchmark_data_smithi179_148205_object75739:head# 2024-02-09T17:57:33.057 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.057 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c463a32:::benchmark_data_smithi179_148205_object59379:head# 2024-02-09T17:57:33.057 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.057 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c46b736:::benchmark_data_smithi179_148205_object27983:head# 2024-02-09T17:57:33.058 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.059 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c46e2d3:::benchmark_data_smithi179_148205_object58342:head# 2024-02-09T17:57:33.059 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.059 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c475697:::benchmark_data_smithi179_148205_object27677:head# 2024-02-09T17:57:33.059 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.059 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c4884a0:::benchmark_data_smithi179_148205_object12806:head# 2024-02-09T17:57:33.059 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.059 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c48c432:::benchmark_data_smithi179_148205_object21279:head# 2024-02-09T17:57:33.059 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.059 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c48e85b:::benchmark_data_smithi179_148205_object85098:head# 2024-02-09T17:57:33.059 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.060 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:57:33.061 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c4a223c:::benchmark_data_smithi179_148205_object51649:head# 2024-02-09T17:57:33.061 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.061 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c4b4966:::benchmark_data_smithi179_148205_object51273:head# 2024-02-09T17:57:33.061 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.061 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c4b7d0a:::benchmark_data_smithi179_148205_object23055:head# 2024-02-09T17:57:33.061 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.061 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c4bd2b9:::benchmark_data_smithi179_148205_object79069:head# 2024-02-09T17:57:33.062 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.062 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c4bf428:::benchmark_data_smithi179_148205_object55704:head# 2024-02-09T17:57:33.062 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.064 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c4c07bb:::benchmark_data_smithi179_148205_object1048:head# 2024-02-09T17:57:33.065 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.065 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c4c17ae:::benchmark_data_smithi179_148205_object73510:head# 2024-02-09T17:57:33.065 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.065 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c4c5107:::benchmark_data_smithi179_148205_object46752:head# 2024-02-09T17:57:33.065 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.065 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c4d33c3:::benchmark_data_smithi179_148205_object11143:head# 2024-02-09T17:57:33.065 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.065 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c4d58bb:::benchmark_data_smithi179_148205_object63916:head# 2024-02-09T17:57:33.065 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.065 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c4de096:::benchmark_data_smithi179_148205_object25476:head# 2024-02-09T17:57:33.065 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.065 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c4e721a:::benchmark_data_smithi179_148205_object6126:head# 2024-02-09T17:57:33.065 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.065 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c504293:::benchmark_data_smithi179_148205_object641:head# 2024-02-09T17:57:33.065 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.066 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c509b63:::benchmark_data_smithi179_148205_object12853:head# 2024-02-09T17:57:33.066 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.066 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c50be70:::benchmark_data_smithi179_148205_object102631:head# 2024-02-09T17:57:33.066 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.066 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c50dfb3:::benchmark_data_smithi179_148205_object49455:head# 2024-02-09T17:57:33.066 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.066 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c51b4eb:::benchmark_data_smithi179_148205_object57204:head# 2024-02-09T17:57:33.066 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.066 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c51f8bb:::benchmark_data_smithi179_148205_object104859:head# 2024-02-09T17:57:33.066 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.069 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c524378:::benchmark_data_smithi179_148205_object63940:head# 2024-02-09T17:57:33.069 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.069 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c528137:::benchmark_data_smithi179_148205_object53189:head# 2024-02-09T17:57:33.069 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.069 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c532dad:::benchmark_data_smithi179_148205_object863:head# 2024-02-09T17:57:33.069 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.069 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c54a92d:::benchmark_data_smithi179_148205_object30952:head# 2024-02-09T17:57:33.070 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.070 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c5504d8:::benchmark_data_smithi179_148205_object63382:head# 2024-02-09T17:57:33.070 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.070 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c55b003:::benchmark_data_smithi179_148205_object27968:head# 2024-02-09T17:57:33.070 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.070 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c5654f3:::benchmark_data_smithi179_148205_object21661:head# 2024-02-09T17:57:33.070 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.070 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c568910:::benchmark_data_smithi179_148205_object58879:head# 2024-02-09T17:57:33.070 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.070 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c57090a:::benchmark_data_smithi179_148205_object3874:head# 2024-02-09T17:57:33.070 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.070 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c5859e7:::benchmark_data_smithi179_148205_object25765:head# 2024-02-09T17:57:33.070 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.070 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c5920da:::benchmark_data_smithi179_148205_object50610:head# 2024-02-09T17:57:33.070 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.071 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c592721:::benchmark_data_smithi179_148205_object56362:head# 2024-02-09T17:57:33.071 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.071 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c5a5ced:::benchmark_data_smithi179_148205_object103272:head# 2024-02-09T17:57:33.071 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.071 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c5a8f98:::benchmark_data_smithi179_148205_object68777:head# 2024-02-09T17:57:33.071 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.071 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c5b5423:::benchmark_data_smithi179_148205_object36567:head# 2024-02-09T17:57:33.071 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.071 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c5c81a3:::benchmark_data_smithi179_148205_object108851:head# 2024-02-09T17:57:33.071 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.071 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c5cc6dd:::benchmark_data_smithi179_148205_object108127:head# 2024-02-09T17:57:33.071 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.071 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c5dfd34:::benchmark_data_smithi179_148205_object13055:head# 2024-02-09T17:57:33.071 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.071 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c5e69db:::benchmark_data_smithi179_148205_object100726:head# 2024-02-09T17:57:33.071 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.075 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c5eb6b8:::benchmark_data_smithi179_148205_object104755:head# 2024-02-09T17:57:33.075 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.076 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c5f2a66:::benchmark_data_smithi179_148205_object34651:head# 2024-02-09T17:57:33.076 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.076 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c6018fa:::benchmark_data_smithi179_148205_object14909:head# 2024-02-09T17:57:33.076 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.076 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c603db5:::benchmark_data_smithi179_148205_object15756:head# 2024-02-09T17:57:33.076 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.076 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c60bc62:::benchmark_data_smithi179_148205_object66864:head# 2024-02-09T17:57:33.076 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.076 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c60bf07:::benchmark_data_smithi179_148205_object63409:head# 2024-02-09T17:57:33.076 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.076 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c616e24:::benchmark_data_smithi179_148205_object88115:head# 2024-02-09T17:57:33.076 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.076 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c618966:::benchmark_data_smithi179_148205_object69098:head# 2024-02-09T17:57:33.077 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.077 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c61aad3:::benchmark_data_smithi179_148205_object11609:head# 2024-02-09T17:57:33.077 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.077 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c632b36:::benchmark_data_smithi179_148205_object10704:head# 2024-02-09T17:57:33.077 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.077 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c6469ba:::benchmark_data_smithi179_148205_object5090:head# 2024-02-09T17:57:33.077 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.077 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c65a19c:::benchmark_data_smithi179_148205_object3986:head# 2024-02-09T17:57:33.077 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.077 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c661886:::benchmark_data_smithi179_148205_object74401:head# 2024-02-09T17:57:33.077 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.077 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c666ce0:::benchmark_data_smithi179_148205_object46577:head# 2024-02-09T17:57:33.077 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.077 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c66cc58:::benchmark_data_smithi179_148205_object55285:head# 2024-02-09T17:57:33.077 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.077 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c6701fd:::benchmark_data_smithi179_148205_object59520:head# 2024-02-09T17:57:33.078 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.078 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c67c068:::benchmark_data_smithi179_148205_object30810:head# 2024-02-09T17:57:33.078 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.078 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c688620:::benchmark_data_smithi179_148205_object5959:head# 2024-02-09T17:57:33.078 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.078 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c697208:::benchmark_data_smithi179_148205_object58144:head# 2024-02-09T17:57:33.078 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.078 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c6a69ea:::benchmark_data_smithi179_148205_object82330:head# 2024-02-09T17:57:33.078 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.078 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c6c3f1c:::benchmark_data_smithi179_148205_object20875:head# 2024-02-09T17:57:33.078 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.078 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c6c5c56:::benchmark_data_smithi179_148205_object93664:head# 2024-02-09T17:57:33.078 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.078 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c6d2854:::benchmark_data_smithi179_148205_object21837:head# 2024-02-09T17:57:33.079 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.079 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c6e7059:::benchmark_data_smithi179_148205_object51494:head# 2024-02-09T17:57:33.079 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.079 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c6ed402:::benchmark_data_smithi179_148205_object73254:head# 2024-02-09T17:57:33.079 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.084 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c6f51b6:::benchmark_data_smithi179_148205_object90084:head# 2024-02-09T17:57:33.084 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.085 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c6fc658:::benchmark_data_smithi179_148205_object101443:head# 2024-02-09T17:57:33.085 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.085 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c6fec6a:::benchmark_data_smithi179_148205_object76565:head# 2024-02-09T17:57:33.085 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.085 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c708c3a:::benchmark_data_smithi179_148205_object33620:head# 2024-02-09T17:57:33.085 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.085 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c7135a7:::benchmark_data_smithi179_148205_object72408:head# 2024-02-09T17:57:33.085 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.085 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c71640e:::benchmark_data_smithi179_148205_object11251:head# 2024-02-09T17:57:33.085 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.085 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c71f73a:::benchmark_data_smithi179_148205_object12359:head# 2024-02-09T17:57:33.085 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.085 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c723297:::benchmark_data_smithi179_148205_object21832:head# 2024-02-09T17:57:33.085 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.086 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c728bd1:::benchmark_data_smithi179_148205_object22972:head# 2024-02-09T17:57:33.086 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.086 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c72ee95:::benchmark_data_smithi179_148205_object54937:head# 2024-02-09T17:57:33.086 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.086 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c748fc2:::benchmark_data_smithi179_148205_object52068:head# 2024-02-09T17:57:33.086 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.086 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c74b3f8:::benchmark_data_smithi179_148205_object62611:head# 2024-02-09T17:57:33.086 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.086 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c74c745:::benchmark_data_smithi179_148205_object8672:head# 2024-02-09T17:57:33.086 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.086 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c757830:::benchmark_data_smithi179_148205_object64678:head# 2024-02-09T17:57:33.086 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.086 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c759b4b:::benchmark_data_smithi179_148205_object33624:head# 2024-02-09T17:57:33.086 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.086 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c77bfad:::benchmark_data_smithi179_148205_object45000:head# 2024-02-09T17:57:33.087 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.087 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c77ddd7:::benchmark_data_smithi179_148205_object67863:head# 2024-02-09T17:57:33.087 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.087 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c782704:::benchmark_data_smithi179_148205_object82835:head# 2024-02-09T17:57:33.087 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.087 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c78406d:::benchmark_data_smithi179_148205_object50267:head# 2024-02-09T17:57:33.087 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.087 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c78e515:::benchmark_data_smithi179_148205_object9349:head# 2024-02-09T17:57:33.087 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.087 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c7975d9:::benchmark_data_smithi179_148205_object60682:head# 2024-02-09T17:57:33.087 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.087 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c79f210:::benchmark_data_smithi179_148205_object105273:head# 2024-02-09T17:57:33.087 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.087 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c7a070e:::benchmark_data_smithi179_148205_object54551:head# 2024-02-09T17:57:33.087 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.088 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c7a6d3d:::benchmark_data_smithi179_148205_object13970:head# 2024-02-09T17:57:33.088 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.088 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c7a7c2d:::benchmark_data_smithi179_148205_object31853:head# 2024-02-09T17:57:33.088 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.088 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c7aeca0:::benchmark_data_smithi179_148205_object90891:head# 2024-02-09T17:57:33.088 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.088 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c7b0099:::benchmark_data_smithi179_148205_object32909:head# 2024-02-09T17:57:33.088 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.088 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c7be76a:::benchmark_data_smithi179_148205_object88062:head# 2024-02-09T17:57:33.088 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.088 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c7ce86a:::benchmark_data_smithi179_148205_object10846:head# 2024-02-09T17:57:33.088 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.088 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c7cf68c:::benchmark_data_smithi179_148205_object41090:head# 2024-02-09T17:57:33.088 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.088 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c7d05f1:::benchmark_data_smithi179_148205_object11663:head# 2024-02-09T17:57:33.089 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.089 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c7e6737:::benchmark_data_smithi179_148205_object103255:head# 2024-02-09T17:57:33.089 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.089 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c7f3d62:::benchmark_data_smithi179_148205_object945:head# 2024-02-09T17:57:33.089 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.089 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c7fbb42:::benchmark_data_smithi179_148205_object65643:head# 2024-02-09T17:57:33.089 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.089 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c8019db:::benchmark_data_smithi179_148205_object77054:head# 2024-02-09T17:57:33.089 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.089 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c804fd6:::benchmark_data_smithi179_148205_object70606:head# 2024-02-09T17:57:33.089 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.089 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c8100f6:::benchmark_data_smithi179_148205_object2474:head# 2024-02-09T17:57:33.089 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.089 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c81ab6a:::benchmark_data_smithi179_148205_object45351:head# 2024-02-09T17:57:33.090 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.090 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c830fde:::benchmark_data_smithi179_148205_object12148:head# 2024-02-09T17:57:33.090 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.090 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c832fe8:::benchmark_data_smithi179_148205_object104069:head# 2024-02-09T17:57:33.090 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.090 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c83b103:::benchmark_data_smithi179_148205_object104625:head# 2024-02-09T17:57:33.090 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.091 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:33 smithi162 ceph-mon[44304]: 1.12 scrub starts 2024-02-09T17:57:33.091 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:33 smithi162 ceph-mon[44304]: 1.12 scrub ok 2024-02-09T17:57:33.091 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:33 smithi162 ceph-mon[44304]: 8.d scrub ok 2024-02-09T17:57:33.091 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:33 smithi162 ceph-mon[44304]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:57:33.091 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:33 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:57:33.091 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:33 smithi162 ceph-mon[44304]: osdmap e828: 12 total, 11 up, 12 in 2024-02-09T17:57:33.091 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:33.092 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c84a242:::benchmark_data_smithi179_148205_object23117:head# 2024-02-09T17:57:33.092 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.092 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c84c9e4:::benchmark_data_smithi179_148205_object106488:head# 2024-02-09T17:57:33.092 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.092 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c84ce70:::benchmark_data_smithi179_148205_object40033:head# 2024-02-09T17:57:33.092 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.092 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c85de18:::benchmark_data_smithi179_148205_object38844:head# 2024-02-09T17:57:33.092 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.095 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c868640:::benchmark_data_smithi179_148205_object7081:head# 2024-02-09T17:57:33.095 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.096 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c870566:::benchmark_data_smithi179_148205_object70042:head# 2024-02-09T17:57:33.096 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.096 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c8736cf:::benchmark_data_smithi179_148205_object939:head# 2024-02-09T17:57:33.096 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.097 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c87377c:::benchmark_data_smithi179_148205_object14670:head# 2024-02-09T17:57:33.098 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.098 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c876222:::benchmark_data_smithi179_148205_object30796:head# 2024-02-09T17:57:33.098 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.098 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c87c6f4:::benchmark_data_smithi179_148205_object57184:head# 2024-02-09T17:57:33.098 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.098 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c87dfce:::benchmark_data_smithi179_148205_object102026:head# 2024-02-09T17:57:33.098 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.098 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c881a4d:::benchmark_data_smithi179_148205_object83930:head# 2024-02-09T17:57:33.098 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.098 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c88a8dc:::benchmark_data_smithi179_148205_object109421:head# 2024-02-09T17:57:33.098 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.098 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c89c54c:::benchmark_data_smithi179_148205_object42095:head# 2024-02-09T17:57:33.098 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.098 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c8a826a:::benchmark_data_smithi179_148205_object7026:head# 2024-02-09T17:57:33.099 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.101 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c8b2b85:::benchmark_data_smithi179_148205_object108378:head# 2024-02-09T17:57:33.101 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.101 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c8bc0ae:::benchmark_data_smithi179_148205_object66124:head# 2024-02-09T17:57:33.101 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.101 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c8bc19b:::benchmark_data_smithi179_148205_object10680:head# 2024-02-09T17:57:33.101 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.101 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c8ccc2b:::benchmark_data_smithi179_148205_object42912:head# 2024-02-09T17:57:33.101 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.101 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c8ce8b0:::benchmark_data_smithi179_148205_object83647:head# 2024-02-09T17:57:33.102 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.102 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c8d2d1b:::benchmark_data_smithi179_148205_object20987:head# 2024-02-09T17:57:33.102 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.102 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c8d5261:::benchmark_data_smithi179_148205_object101464:head# 2024-02-09T17:57:33.102 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.102 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c8d80d4:::benchmark_data_smithi179_148205_object105654:head# 2024-02-09T17:57:33.102 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.102 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c8deabc:::benchmark_data_smithi179_148205_object7283:head# 2024-02-09T17:57:33.102 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.102 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c8f0b7c:::benchmark_data_smithi179_148205_object42551:head# 2024-02-09T17:57:33.102 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.102 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c8f693c:::benchmark_data_smithi179_148205_object47622:head# 2024-02-09T17:57:33.102 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.102 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c8f91dc:::benchmark_data_smithi179_148205_object598:head# 2024-02-09T17:57:33.103 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.103 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c8f9b48:::benchmark_data_smithi179_148205_object66352:head# 2024-02-09T17:57:33.103 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.105 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c8fc051:::benchmark_data_smithi179_148205_object99576:head# 2024-02-09T17:57:33.106 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.106 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c8fcc90:::benchmark_data_smithi179_148205_object95780:head# 2024-02-09T17:57:33.106 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.106 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c904ce8:::benchmark_data_smithi179_148205_object108316:head# 2024-02-09T17:57:33.106 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.106 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c905587:::benchmark_data_smithi179_148205_object83905:head# 2024-02-09T17:57:33.106 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.106 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c90e9c0:::benchmark_data_smithi179_148205_object101213:head# 2024-02-09T17:57:33.106 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.106 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c90ecbe:::benchmark_data_smithi179_148205_object72651:head# 2024-02-09T17:57:33.106 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.106 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c913a79:::benchmark_data_smithi179_148205_object98595:head# 2024-02-09T17:57:33.106 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.107 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c915676:::benchmark_data_smithi179_148205_object15788:head# 2024-02-09T17:57:33.107 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.107 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c917059:::benchmark_data_smithi179_148205_object31679:head# 2024-02-09T17:57:33.107 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.110 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c931385:::benchmark_data_smithi179_148205_object91551:head# 2024-02-09T17:57:33.110 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.110 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c938044:::benchmark_data_smithi179_148205_object109752:head# 2024-02-09T17:57:33.110 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.111 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c939feb:::benchmark_data_smithi179_148205_object45720:head# 2024-02-09T17:57:33.111 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.111 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c93c338:::benchmark_data_smithi179_148205_object109656:head# 2024-02-09T17:57:33.111 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.111 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c945de7:::benchmark_data_smithi179_148205_object10361:head# 2024-02-09T17:57:33.111 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.111 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c946e7a:::benchmark_data_smithi179_148205_object3643:head# 2024-02-09T17:57:33.111 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.111 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c94a5e8:::benchmark_data_smithi179_148205_object14118:head# 2024-02-09T17:57:33.111 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.111 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c94c19d:::benchmark_data_smithi179_148205_object6501:head# 2024-02-09T17:57:33.111 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.111 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c94e9da:::benchmark_data_smithi179_148205_object2781:head# 2024-02-09T17:57:33.111 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.111 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c95ccd8:::benchmark_data_smithi179_148205_object101211:head# 2024-02-09T17:57:33.112 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.112 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c9618e7:::benchmark_data_smithi179_148205_object43918:head# 2024-02-09T17:57:33.112 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.112 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c968c6a:::benchmark_data_smithi179_148205_object98216:head# 2024-02-09T17:57:33.112 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.112 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c96ac5c:::benchmark_data_smithi179_148205_object30328:head# 2024-02-09T17:57:33.112 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.112 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c96eff6:::benchmark_data_smithi179_148205_object101834:head# 2024-02-09T17:57:33.112 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.112 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c97c1c8:::benchmark_data_smithi179_148205_object43006:head# 2024-02-09T17:57:33.112 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.112 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c991237:::benchmark_data_smithi179_148205_object29821:head# 2024-02-09T17:57:33.112 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.113 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c9ae06e:::benchmark_data_smithi179_148205_object30121:head# 2024-02-09T17:57:33.113 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.113 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c9b3ade:::benchmark_data_smithi179_148205_object41644:head# 2024-02-09T17:57:33.113 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.113 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c9b6a6d:::benchmark_data_smithi179_148205_object63531:head# 2024-02-09T17:57:33.113 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:33.114 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:57:33.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:57:33.119 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c9c244a:::benchmark_data_smithi179_148205_object88008:head# 2024-02-09T17:57:33.119 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.119 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c9cbcb8:::benchmark_data_smithi179_148205_object107986:head# 2024-02-09T17:57:33.119 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.120 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c9e93be:::benchmark_data_smithi179_148205_object103348:head# 2024-02-09T17:57:33.120 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.120 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c9ef04e:::benchmark_data_smithi179_148205_object32657:head# 2024-02-09T17:57:33.120 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.120 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c9f048d:::benchmark_data_smithi179_148205_object8713:head# 2024-02-09T17:57:33.120 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.120 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1c9f12ee:::benchmark_data_smithi179_148205_object105958:head# 2024-02-09T17:57:33.120 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.120 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ca1587d:::benchmark_data_smithi179_148205_object49350:head# 2024-02-09T17:57:33.120 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.120 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ca22388:::benchmark_data_smithi179_148205_object25978:head# 2024-02-09T17:57:33.120 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.120 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ca31f28:::benchmark_data_smithi179_148205_object48753:head# 2024-02-09T17:57:33.120 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.121 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ca328c5:::benchmark_data_smithi179_148205_object15061:head# 2024-02-09T17:57:33.121 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.121 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ca37889:::benchmark_data_smithi179_148205_object84017:head# 2024-02-09T17:57:33.121 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.121 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ca3e0f6:::benchmark_data_smithi179_148205_object102487:head# 2024-02-09T17:57:33.121 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.121 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ca3f687:::benchmark_data_smithi179_148205_object24465:head# 2024-02-09T17:57:33.121 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.121 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ca41add:::benchmark_data_smithi179_148205_object109626:head# 2024-02-09T17:57:33.121 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.121 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ca497ac:::benchmark_data_smithi179_148205_object99717:head# 2024-02-09T17:57:33.121 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.121 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ca4bf59:::benchmark_data_smithi179_148205_object12398:head# 2024-02-09T17:57:33.121 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.122 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ca65968:::benchmark_data_smithi179_148205_object29840:head# 2024-02-09T17:57:33.122 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.122 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ca78518:::benchmark_data_smithi179_148205_object93184:head# 2024-02-09T17:57:33.122 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.122 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ca85339:::benchmark_data_smithi179_148205_object59741:head# 2024-02-09T17:57:33.122 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.122 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ca867ba:::benchmark_data_smithi179_148205_object62855:head# 2024-02-09T17:57:33.122 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.122 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ca87052:::benchmark_data_smithi179_148205_object58911:head# 2024-02-09T17:57:33.122 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.122 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ca8d647:::benchmark_data_smithi179_148205_object47697:head# 2024-02-09T17:57:33.122 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.122 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ca93397:::benchmark_data_smithi179_148205_object69552:head# 2024-02-09T17:57:33.122 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.122 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ca9470d:::benchmark_data_smithi179_148205_object89599:head# 2024-02-09T17:57:33.123 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.123 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ca98e7f:::benchmark_data_smithi179_148205_object41813:head# 2024-02-09T17:57:33.123 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.123 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ca9b048:::benchmark_data_smithi179_148205_object78799:head# 2024-02-09T17:57:33.123 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.126 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ca9f852:::benchmark_data_smithi179_148205_object49074:head# 2024-02-09T17:57:33.126 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.126 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cab5173:::benchmark_data_smithi179_148205_object33991:head# 2024-02-09T17:57:33.126 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.126 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cac3dbc:::benchmark_data_smithi179_148205_object83723:head# 2024-02-09T17:57:33.126 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.126 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cac9759:::benchmark_data_smithi179_148205_object11955:head# 2024-02-09T17:57:33.126 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.126 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cad2442:::benchmark_data_smithi179_148205_object53982:head# 2024-02-09T17:57:33.126 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.126 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cad7d6d:::benchmark_data_smithi179_148205_object95621:head# 2024-02-09T17:57:33.126 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.126 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cad9d88:::benchmark_data_smithi179_148205_object38807:head# 2024-02-09T17:57:33.127 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.127 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cade143:::benchmark_data_smithi179_148205_object74218:head# 2024-02-09T17:57:33.127 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.127 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1caecfce:::benchmark_data_smithi179_148205_object70706:head# 2024-02-09T17:57:33.127 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.127 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1caf53f2:::benchmark_data_smithi179_148205_object80736:head# 2024-02-09T17:57:33.127 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.127 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cafd67d:::benchmark_data_smithi179_148205_object1420:head# 2024-02-09T17:57:33.127 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.127 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cb02464:::benchmark_data_smithi179_148205_object61137:head# 2024-02-09T17:57:33.127 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.127 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cb02cc4:::benchmark_data_smithi179_148205_object6144:head# 2024-02-09T17:57:33.127 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.127 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cb06646:::benchmark_data_smithi179_148205_object37938:head# 2024-02-09T17:57:33.128 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.128 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cb093dc:::benchmark_data_smithi179_148205_object54603:head# 2024-02-09T17:57:33.128 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.128 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cb0b0fa:::benchmark_data_smithi179_148205_object42054:head# 2024-02-09T17:57:33.128 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.128 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cb1698c:::benchmark_data_smithi179_148205_object75202:head# 2024-02-09T17:57:33.128 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.128 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cb2179b:::benchmark_data_smithi179_148205_object96390:head# 2024-02-09T17:57:33.128 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.128 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cb218d4:::benchmark_data_smithi179_148205_object93313:head# 2024-02-09T17:57:33.128 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.129 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cb24848:::benchmark_data_smithi179_148205_object14901:head# 2024-02-09T17:57:33.130 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.130 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cb2bbbb:::benchmark_data_smithi179_148205_object83003:head# 2024-02-09T17:57:33.130 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.130 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cb3bcc5:::benchmark_data_smithi179_148205_object98352:head# 2024-02-09T17:57:33.130 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.130 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cb3e221:::benchmark_data_smithi179_148205_object99013:head# 2024-02-09T17:57:33.130 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.130 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cb3ff65:::benchmark_data_smithi179_148205_object28490:head# 2024-02-09T17:57:33.130 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.130 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cb425b6:::benchmark_data_smithi179_148205_object65668:head# 2024-02-09T17:57:33.130 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.130 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cb4478a:::benchmark_data_smithi179_148205_object47340:head# 2024-02-09T17:57:33.130 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.130 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cb475d9:::benchmark_data_smithi179_148205_object84870:head# 2024-02-09T17:57:33.131 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.131 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cb4ae68:::benchmark_data_smithi179_148205_object109784:head# 2024-02-09T17:57:33.131 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.131 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cb4eb8e:::benchmark_data_smithi179_148205_object96679:head# 2024-02-09T17:57:33.131 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.131 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cb5b6a3:::benchmark_data_smithi179_148205_object41850:head# 2024-02-09T17:57:33.131 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.131 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cb64ba3:::benchmark_data_smithi179_148205_object55875:head# 2024-02-09T17:57:33.131 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.131 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cb64dcf:::benchmark_data_smithi179_148205_object47837:head# 2024-02-09T17:57:33.131 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.133 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cb809ad:::benchmark_data_smithi179_148205_object56342:head# 2024-02-09T17:57:33.133 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.133 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cb86038:::benchmark_data_smithi179_148205_object86790:head# 2024-02-09T17:57:33.133 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.133 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cb8db16:::benchmark_data_smithi179_148205_object90944:head# 2024-02-09T17:57:33.133 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.133 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cb92275:::benchmark_data_smithi179_148205_object101037:head# 2024-02-09T17:57:33.133 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.134 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cb92bb1:::benchmark_data_smithi179_148205_object105133:head# 2024-02-09T17:57:33.134 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.134 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cb99a6e:::benchmark_data_smithi179_148205_object69933:head# 2024-02-09T17:57:33.134 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.134 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cb9e9f7:::benchmark_data_smithi179_148205_object33546:head# 2024-02-09T17:57:33.134 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.134 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cb9f1d9:::benchmark_data_smithi179_148205_object14936:head# 2024-02-09T17:57:33.134 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.134 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cbb42df:::benchmark_data_smithi179_148205_object39144:head# 2024-02-09T17:57:33.134 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.134 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cbb60ef:::benchmark_data_smithi179_148205_object64369:head# 2024-02-09T17:57:33.134 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.134 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cbbae25:::benchmark_data_smithi179_148205_object77800:head# 2024-02-09T17:57:33.134 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.136 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cbbd473:::benchmark_data_smithi179_148205_object55049:head# 2024-02-09T17:57:33.136 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.136 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cbc178b:::benchmark_data_smithi179_148205_object22534:head# 2024-02-09T17:57:33.136 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.136 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cbd437c:::benchmark_data_smithi179_148205_object76889:head# 2024-02-09T17:57:33.136 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.136 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cbd49af:::benchmark_data_smithi179_148205_object50800:head# 2024-02-09T17:57:33.136 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.137 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cbe5cd3:::benchmark_data_smithi179_148205_object12616:head# 2024-02-09T17:57:33.137 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.137 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cbe7129:::benchmark_data_smithi179_148205_object94250:head# 2024-02-09T17:57:33.137 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.137 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cbff31a:::benchmark_data_smithi179_148205_object54219:head# 2024-02-09T17:57:33.137 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.137 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cc12d82:::benchmark_data_smithi179_148205_object1533:head# 2024-02-09T17:57:33.137 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.137 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cc133f9:::benchmark_data_smithi179_148205_object81277:head# 2024-02-09T17:57:33.137 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.137 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cc191b3:::benchmark_data_smithi179_148205_object13618:head# 2024-02-09T17:57:33.137 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.137 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cc1c35d:::benchmark_data_smithi179_148205_object64251:head# 2024-02-09T17:57:33.137 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.138 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cc338b8:::benchmark_data_smithi179_148205_object39641:head# 2024-02-09T17:57:33.138 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.138 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cc371e4:::benchmark_data_smithi179_148205_object50823:head# 2024-02-09T17:57:33.138 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.138 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cc3c4f9:::benchmark_data_smithi179_148205_object65232:head# 2024-02-09T17:57:33.138 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.138 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cc4470e:::benchmark_data_smithi179_148205_object42904:head# 2024-02-09T17:57:33.138 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.141 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cc482bc:::benchmark_data_smithi179_148205_object50531:head# 2024-02-09T17:57:33.141 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.141 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cc4e3b9:::benchmark_data_smithi179_148205_object62385:head# 2024-02-09T17:57:33.141 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.141 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cc50680:::benchmark_data_smithi179_148205_object32415:head# 2024-02-09T17:57:33.141 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.141 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cc5e193:::benchmark_data_smithi179_148205_object40455:head# 2024-02-09T17:57:33.141 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.141 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cc6f8e6:::benchmark_data_smithi179_148205_object57242:head# 2024-02-09T17:57:33.141 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.142 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ccadd50:::benchmark_data_smithi179_148205_object66897:head# 2024-02-09T17:57:33.142 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.144 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ccc1e84:::benchmark_data_smithi179_148205_object92047:head# 2024-02-09T17:57:33.144 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.144 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ccd82ad:::benchmark_data_smithi179_148205_object27542:head# 2024-02-09T17:57:33.144 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.145 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ccde5a7:::benchmark_data_smithi179_148205_object66031:head# 2024-02-09T17:57:33.146 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.146 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cced164:::benchmark_data_smithi179_148205_object3089:head# 2024-02-09T17:57:33.146 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.146 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ccef793:::benchmark_data_smithi179_148205_object69543:head# 2024-02-09T17:57:33.146 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.146 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ccf20e3:::benchmark_data_smithi179_148205_object13890:head# 2024-02-09T17:57:33.146 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.146 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ccff622:::benchmark_data_smithi179_148205_object50648:head# 2024-02-09T17:57:33.146 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.146 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cd035e1:::benchmark_data_smithi179_148205_object71908:head# 2024-02-09T17:57:33.146 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.147 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cd146df:::benchmark_data_smithi179_148205_object66009:head# 2024-02-09T17:57:33.147 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.147 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cd15140:::benchmark_data_smithi179_148205_object52974:head# 2024-02-09T17:57:33.148 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.148 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cd179db:::benchmark_data_smithi179_148205_object14736:head# 2024-02-09T17:57:33.148 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.148 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cd1fc45:::benchmark_data_smithi179_148205_object91218:head# 2024-02-09T17:57:33.148 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.148 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cd2bb8d:::benchmark_data_smithi179_148205_object48069:head# 2024-02-09T17:57:33.148 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.148 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cd3083e:::benchmark_data_smithi179_148205_object75721:head# 2024-02-09T17:57:33.148 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.148 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cd32b67:::benchmark_data_smithi179_148205_object10597:head# 2024-02-09T17:57:33.148 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.150 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cd3c0f3:::benchmark_data_smithi179_148205_object47610:head# 2024-02-09T17:57:33.150 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.151 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cd3d126:::benchmark_data_smithi179_148205_object33677:head# 2024-02-09T17:57:33.151 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.151 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cd47611:::benchmark_data_smithi179_148205_object91163:head# 2024-02-09T17:57:33.151 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.151 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cd4bf15:::benchmark_data_smithi179_148205_object43537:head# 2024-02-09T17:57:33.151 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.151 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cd4deb2:::benchmark_data_smithi179_148205_object50562:head# 2024-02-09T17:57:33.151 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.151 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cd52391:::benchmark_data_smithi179_148205_object8282:head# 2024-02-09T17:57:33.151 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.151 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cd55e75:::benchmark_data_smithi179_148205_object70583:head# 2024-02-09T17:57:33.151 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.151 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cd65cf6:::benchmark_data_smithi179_148205_object91725:head# 2024-02-09T17:57:33.151 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.152 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cd7341b:::benchmark_data_smithi179_148205_object107751:head# 2024-02-09T17:57:33.152 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.152 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cd73d0e:::benchmark_data_smithi179_148205_object87891:head# 2024-02-09T17:57:33.152 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.152 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cd7af4e:::benchmark_data_smithi179_148205_object40101:head# 2024-02-09T17:57:33.152 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.153 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cda6137:::benchmark_data_smithi179_148205_object52725:head# 2024-02-09T17:57:33.153 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.153 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cdaaf82:::benchmark_data_smithi179_148205_object104171:head# 2024-02-09T17:57:33.153 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.153 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cdaed5d:::benchmark_data_smithi179_148205_object94750:head# 2024-02-09T17:57:33.153 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.153 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cdbb59a:::benchmark_data_smithi179_148205_object23293:head# 2024-02-09T17:57:33.154 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.154 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cdbfe81:::benchmark_data_smithi179_148205_object94395:head# 2024-02-09T17:57:33.154 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.156 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-09T17:57:33.159 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cdc016f:::benchmark_data_smithi179_148205_object61726:head# 2024-02-09T17:57:33.159 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.161 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cdc4889:::benchmark_data_smithi179_148205_object57475:head# 2024-02-09T17:57:33.161 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.162 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cdccf54:::benchmark_data_smithi179_148205_object70823:head# 2024-02-09T17:57:33.162 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.162 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cdd8a27:::benchmark_data_smithi179_148205_object26039:head# 2024-02-09T17:57:33.162 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.162 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cde1247:::benchmark_data_smithi179_148205_object109289:head# 2024-02-09T17:57:33.162 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.162 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cdf0ed7:::benchmark_data_smithi179_148205_object44793:head# 2024-02-09T17:57:33.162 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.162 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cdfc8aa:::benchmark_data_smithi179_148205_object2680:head# 2024-02-09T17:57:33.162 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.162 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ce0b0f3:::benchmark_data_smithi179_148205_object22234:head# 2024-02-09T17:57:33.163 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.164 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ce171cf:::benchmark_data_smithi179_148205_object74499:head# 2024-02-09T17:57:33.164 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.164 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ce1779b:::benchmark_data_smithi179_148205_object1189:head# 2024-02-09T17:57:33.164 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.164 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ce188ed:::benchmark_data_smithi179_148205_object82114:head# 2024-02-09T17:57:33.164 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.164 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ce21779:::benchmark_data_smithi179_148205_object76006:head# 2024-02-09T17:57:33.164 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.165 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ce25b4a:::benchmark_data_smithi179_148205_object80812:head# 2024-02-09T17:57:33.165 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.165 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ce28301:::benchmark_data_smithi179_148205_object6679:head# 2024-02-09T17:57:33.165 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.165 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ce2bb10:::benchmark_data_smithi179_148205_object8480:head# 2024-02-09T17:57:33.165 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.166 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ce34e9b:::benchmark_data_smithi179_148205_object47710:head# 2024-02-09T17:57:33.166 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.166 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ce3b0e8:::benchmark_data_smithi179_148205_object43716:head# 2024-02-09T17:57:33.166 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.166 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ce3bde4:::benchmark_data_smithi179_148205_object47031:head# 2024-02-09T17:57:33.166 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.166 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ce3f0c5:::benchmark_data_smithi179_148205_object100561:head# 2024-02-09T17:57:33.166 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.167 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ce47306:::benchmark_data_smithi179_148205_object95443:head# 2024-02-09T17:57:33.167 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.167 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ce47d6e:::benchmark_data_smithi179_148205_object20022:head# 2024-02-09T17:57:33.167 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.167 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ce55c80:::benchmark_data_smithi179_148205_object21705:head# 2024-02-09T17:57:33.167 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.170 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ce579d3:::benchmark_data_smithi179_148205_object96599:head# 2024-02-09T17:57:33.170 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.170 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ce5acdc:::benchmark_data_smithi179_148205_object34635:head# 2024-02-09T17:57:33.170 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.170 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ce607ab:::benchmark_data_smithi179_148205_object3657:head# 2024-02-09T17:57:33.170 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.170 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ce6233b:::benchmark_data_smithi179_148205_object38367:head# 2024-02-09T17:57:33.170 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.170 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ce6296a:::benchmark_data_smithi179_148205_object78755:head# 2024-02-09T17:57:33.170 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.172 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ce6421e:::benchmark_data_smithi179_148205_object56360:head# 2024-02-09T17:57:33.172 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.172 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ce75a5e:::benchmark_data_smithi179_148205_object70830:head# 2024-02-09T17:57:33.172 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.173 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ce78a5b:::benchmark_data_smithi179_148205_object4621:head# 2024-02-09T17:57:33.173 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.173 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cea2855:::benchmark_data_smithi179_148205_object23651:head# 2024-02-09T17:57:33.173 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.173 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ceac4b6:::benchmark_data_smithi179_148205_object78836:head# 2024-02-09T17:57:33.174 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.174 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ceb3509:::benchmark_data_smithi179_148205_object30340:head# 2024-02-09T17:57:33.174 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.174 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ceb8714:::benchmark_data_smithi179_148205_object101408:head# 2024-02-09T17:57:33.174 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.175 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cec0988:::benchmark_data_smithi179_148205_object77444:head# 2024-02-09T17:57:33.175 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.175 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cec30f8:::benchmark_data_smithi179_148205_object39294:head# 2024-02-09T17:57:33.175 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.175 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ced0ca3:::benchmark_data_smithi179_148205_object71457:head# 2024-02-09T17:57:33.175 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.175 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cee19bf:::benchmark_data_smithi179_148205_object39666:head# 2024-02-09T17:57:33.176 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.177 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cee29d8:::benchmark_data_smithi179_148205_object74027:head# 2024-02-09T17:57:33.177 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.177 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cef4156:::benchmark_data_smithi179_148205_object41505:head# 2024-02-09T17:57:33.177 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.177 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cef8e84:::benchmark_data_smithi179_148205_object29911:head# 2024-02-09T17:57:33.177 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.177 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cf11d3c:::benchmark_data_smithi179_148205_object39373:head# 2024-02-09T17:57:33.178 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.178 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cf2ab8f:::benchmark_data_smithi179_148205_object32953:head# 2024-02-09T17:57:33.178 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.178 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cf3c3a7:::benchmark_data_smithi179_148205_object41910:head# 2024-02-09T17:57:33.178 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.179 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cf42e8e:::benchmark_data_smithi179_148205_object20414:head# 2024-02-09T17:57:33.179 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.179 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cf439b9:::benchmark_data_smithi179_148205_object99918:head# 2024-02-09T17:57:33.179 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.179 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cf470dc:::benchmark_data_smithi179_148205_object60464:head# 2024-02-09T17:57:33.179 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.179 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cf4a9d4:::benchmark_data_smithi179_148205_object89063:head# 2024-02-09T17:57:33.179 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.179 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cf53a19:::benchmark_data_smithi179_148205_object93840:head# 2024-02-09T17:57:33.179 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.179 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cf5a218:::benchmark_data_smithi179_148205_object10457:head# 2024-02-09T17:57:33.180 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.180 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cf604e6:::benchmark_data_smithi179_148205_object16188:head# 2024-02-09T17:57:33.181 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.181 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cf69b01:::benchmark_data_smithi179_148205_object78997:head# 2024-02-09T17:57:33.181 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.181 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cf6dd81:::benchmark_data_smithi179_148205_object97127:head# 2024-02-09T17:57:33.181 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.181 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cf79353:::benchmark_data_smithi179_148205_object57452:head# 2024-02-09T17:57:33.181 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.181 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cf97d63:::benchmark_data_smithi179_148205_object14564:head# 2024-02-09T17:57:33.181 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.181 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cfafd4f:::benchmark_data_smithi179_148205_object16296:head# 2024-02-09T17:57:33.181 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.182 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cfb38b3:::benchmark_data_smithi179_148205_object79929:head# 2024-02-09T17:57:33.182 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.182 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cfc3b13:::benchmark_data_smithi179_148205_object69971:head# 2024-02-09T17:57:33.182 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.182 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cfc69af:::benchmark_data_smithi179_148205_object84605:head# 2024-02-09T17:57:33.182 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.187 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cfcb00a:::benchmark_data_smithi179_148205_object2874:head# 2024-02-09T17:57:33.187 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.187 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cfcccb7:::benchmark_data_smithi179_148205_object11958:head# 2024-02-09T17:57:33.188 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.188 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cfd8576:::benchmark_data_smithi179_148205_object85848:head# 2024-02-09T17:57:33.188 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.189 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cfe0f5b:::benchmark_data_smithi179_148205_object57961:head# 2024-02-09T17:57:33.189 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.189 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cfe10c0:::benchmark_data_smithi179_148205_object3113:head# 2024-02-09T17:57:33.189 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.189 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cfeaff8:::benchmark_data_smithi179_148205_object56091:head# 2024-02-09T17:57:33.189 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.193 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cff236d:::benchmark_data_smithi179_148205_object41009:head# 2024-02-09T17:57:33.194 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.194 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1cffda1e:::benchmark_data_smithi179_148205_object82259:head# 2024-02-09T17:57:33.194 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.194 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d005334:::benchmark_data_smithi179_148205_object22614:head# 2024-02-09T17:57:33.194 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.194 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d0053be:::benchmark_data_smithi179_148205_object24820:head# 2024-02-09T17:57:33.194 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.194 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d00ac55:::benchmark_data_smithi179_148205_object104731:head# 2024-02-09T17:57:33.194 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.194 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d00d7ce:::benchmark_data_smithi179_148205_object86196:head# 2024-02-09T17:57:33.194 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.194 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d013b71:::benchmark_data_smithi179_148205_object64733:head# 2024-02-09T17:57:33.194 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.194 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d021951:::benchmark_data_smithi179_148205_object51133:head# 2024-02-09T17:57:33.195 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.195 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d021f60:::benchmark_data_smithi179_148205_object71033:head# 2024-02-09T17:57:33.195 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.198 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d029d75:::benchmark_data_smithi179_148205_object43764:head# 2024-02-09T17:57:33.198 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.198 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d034517:::benchmark_data_smithi179_148205_object52482:head# 2024-02-09T17:57:33.198 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.199 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d039149:::benchmark_data_smithi179_148205_object55084:head# 2024-02-09T17:57:33.199 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.199 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d04eaaa:::benchmark_data_smithi179_148205_object74740:head# 2024-02-09T17:57:33.199 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.199 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d0551d4:::benchmark_data_smithi179_148205_object48809:head# 2024-02-09T17:57:33.199 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.200 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d061745:::benchmark_data_smithi179_148205_object4761:head# 2024-02-09T17:57:33.200 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.200 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d066284:::benchmark_data_smithi179_148205_object20966:head# 2024-02-09T17:57:33.200 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.200 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d066b04:::benchmark_data_smithi179_148205_object103475:head# 2024-02-09T17:57:33.200 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.204 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d07fd12:::benchmark_data_smithi179_148205_object71482:head# 2024-02-09T17:57:33.204 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.214 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:57:33.215 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d095265:::benchmark_data_smithi179_148205_object32771:head# 2024-02-09T17:57:33.215 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.219 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d0a3f2d:::benchmark_data_smithi179_148205_object94748:head# 2024-02-09T17:57:33.219 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.220 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d0a5b18:::benchmark_data_smithi179_148205_object92624:head# 2024-02-09T17:57:33.220 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.222 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d0adc0e:::benchmark_data_smithi179_148205_object83254:head# 2024-02-09T17:57:33.222 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.222 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d0b8983:::benchmark_data_smithi179_148205_object43792:head# 2024-02-09T17:57:33.222 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.223 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d0bc784:::benchmark_data_smithi179_148205_object93356:head# 2024-02-09T17:57:33.224 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.224 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d0be810:::benchmark_data_smithi179_148205_object9290:head# 2024-02-09T17:57:33.224 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.224 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d0c43d3:::benchmark_data_smithi179_148205_object91504:head# 2024-02-09T17:57:33.224 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.225 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d0c7b8e:::benchmark_data_smithi179_148205_object47645:head# 2024-02-09T17:57:33.225 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.225 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d0c95b9:::benchmark_data_smithi179_148205_object28384:head# 2024-02-09T17:57:33.225 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.226 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d0d44c4:::benchmark_data_smithi179_148205_object49068:head# 2024-02-09T17:57:33.226 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.226 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d0df689:::benchmark_data_smithi179_148205_object61359:head# 2024-02-09T17:57:33.226 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.226 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d105903:::benchmark_data_smithi179_148205_object39652:head# 2024-02-09T17:57:33.226 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.226 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d11ea2d:::benchmark_data_smithi179_148205_object85090:head# 2024-02-09T17:57:33.227 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.227 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d120bd5:::benchmark_data_smithi179_148205_object14837:head# 2024-02-09T17:57:33.227 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.227 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d127d3b:::benchmark_data_smithi179_148205_object68002:head# 2024-02-09T17:57:33.227 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.227 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d129c48:::benchmark_data_smithi179_148205_object47032:head# 2024-02-09T17:57:33.227 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.232 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d12dd61:::benchmark_data_smithi179_148205_object17117:head# 2024-02-09T17:57:33.232 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.235 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d14411e:::benchmark_data_smithi179_148205_object58233:head# 2024-02-09T17:57:33.235 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.235 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d14f384:::benchmark_data_smithi179_148205_object45312:head# 2024-02-09T17:57:33.235 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.235 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d163d80:::benchmark_data_smithi179_148205_object56648:head# 2024-02-09T17:57:33.235 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.236 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d16d994:::benchmark_data_smithi179_148205_object36910:head# 2024-02-09T17:57:33.236 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.236 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d1782d4:::benchmark_data_smithi179_148205_object48629:head# 2024-02-09T17:57:33.236 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.236 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d185aeb:::benchmark_data_smithi179_148205_object23275:head# 2024-02-09T17:57:33.236 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.237 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d18b03d:::benchmark_data_smithi179_148205_object72853:head# 2024-02-09T17:57:33.237 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.237 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d18fe8e:::benchmark_data_smithi179_148205_object44264:head# 2024-02-09T17:57:33.237 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.238 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d19ae4e:::benchmark_data_smithi179_148205_object59519:head# 2024-02-09T17:57:33.238 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.242 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d19f606:::benchmark_data_smithi179_148205_object80284:head# 2024-02-09T17:57:33.242 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.242 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d1d190b:::benchmark_data_smithi179_148205_object18784:head# 2024-02-09T17:57:33.242 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.242 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d1d841a:::benchmark_data_smithi179_148205_object63914:head# 2024-02-09T17:57:33.242 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.242 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d1db7b8:::benchmark_data_smithi179_148205_object27256:head# 2024-02-09T17:57:33.242 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.242 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d1e9edd:::benchmark_data_smithi179_148205_object99909:head# 2024-02-09T17:57:33.242 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.242 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d1ec410:::benchmark_data_smithi179_148205_object22242:head# 2024-02-09T17:57:33.242 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.243 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d205154:::benchmark_data_smithi179_148205_object50528:head# 2024-02-09T17:57:33.243 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.243 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d22e455:::benchmark_data_smithi179_148205_object58814:head# 2024-02-09T17:57:33.243 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.243 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d231618:::benchmark_data_smithi179_148205_object86962:head# 2024-02-09T17:57:33.243 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.243 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d232ae8:::benchmark_data_smithi179_148205_object50475:head# 2024-02-09T17:57:33.243 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.243 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d234e1e:::benchmark_data_smithi179_148205_object19263:head# 2024-02-09T17:57:33.243 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.243 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d25e137:::benchmark_data_smithi179_148205_object54001:head# 2024-02-09T17:57:33.243 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.243 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:33.244 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d2633c3:::benchmark_data_smithi179_148205_object40919:head# 2024-02-09T17:57:33.244 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.244 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d265d00:::benchmark_data_smithi179_148205_object86665:head# 2024-02-09T17:57:33.245 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.245 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d26c1d1:::benchmark_data_smithi179_148205_object100246:head# 2024-02-09T17:57:33.245 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.245 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d27342c:::benchmark_data_smithi179_148205_object73155:head# 2024-02-09T17:57:33.245 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.251 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d2761f0:::benchmark_data_smithi179_148205_object46197:head# 2024-02-09T17:57:33.251 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.251 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d2801b8:::benchmark_data_smithi179_148205_object57769:head# 2024-02-09T17:57:33.251 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.251 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d298f71:::benchmark_data_smithi179_148205_object109680:head# 2024-02-09T17:57:33.251 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.251 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d2a8ae1:::benchmark_data_smithi179_148205_object56132:head# 2024-02-09T17:57:33.251 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.258 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d2ad1b6:::benchmark_data_smithi179_148205_object16051:head# 2024-02-09T17:57:33.258 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.258 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d2b2634:::benchmark_data_smithi179_148205_object35987:head# 2024-02-09T17:57:33.258 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.258 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d2cda30:::benchmark_data_smithi179_148205_object4414:head# 2024-02-09T17:57:33.258 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.258 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d2e20f8:::benchmark_data_smithi179_148205_object24422:head# 2024-02-09T17:57:33.258 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.259 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d2e3512:::benchmark_data_smithi179_148205_object26238:head# 2024-02-09T17:57:33.259 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.259 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d2ef66b:::benchmark_data_smithi179_148205_object106275:head# 2024-02-09T17:57:33.259 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.259 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d2f3c99:::benchmark_data_smithi179_148205_object58934:head# 2024-02-09T17:57:33.260 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.260 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d2fabf9:::benchmark_data_smithi179_148205_object1289:head# 2024-02-09T17:57:33.261 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.261 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d2fc6fe:::benchmark_data_smithi179_148205_object25686:head# 2024-02-09T17:57:33.261 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.261 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d3077cf:::benchmark_data_smithi179_148205_object56227:head# 2024-02-09T17:57:33.261 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.261 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d30ce37:::benchmark_data_smithi179_148205_object33649:head# 2024-02-09T17:57:33.261 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.261 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d30d998:::benchmark_data_smithi179_148205_object10539:head# 2024-02-09T17:57:33.261 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.261 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d321f09:::benchmark_data_smithi179_148205_object53711:head# 2024-02-09T17:57:33.261 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.263 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:57:33.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:33.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:57:33.268 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d33746a:::benchmark_data_smithi179_148205_object76015:head# 2024-02-09T17:57:33.268 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.268 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d351d94:::benchmark_data_smithi179_148205_object39908:head# 2024-02-09T17:57:33.268 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.268 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d358a7e:::benchmark_data_smithi179_148205_object94773:head# 2024-02-09T17:57:33.268 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.268 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d358b87:::benchmark_data_smithi179_148205_object39920:head# 2024-02-09T17:57:33.269 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.269 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d35fcaa:::benchmark_data_smithi179_148205_object107742:head# 2024-02-09T17:57:33.269 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.269 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d362e7f:::benchmark_data_smithi179_148205_object107020:head# 2024-02-09T17:57:33.269 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.270 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d370005:::benchmark_data_smithi179_148205_object258:head# 2024-02-09T17:57:33.270 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.270 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d37894c:::benchmark_data_smithi179_148205_object66713:head# 2024-02-09T17:57:33.271 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.273 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d388c30:::benchmark_data_smithi179_148205_object108884:head# 2024-02-09T17:57:33.273 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.279 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d38a53f:::benchmark_data_smithi179_148205_object47657:head# 2024-02-09T17:57:33.279 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.280 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d39a361:::benchmark_data_smithi179_148205_object37322:head# 2024-02-09T17:57:33.280 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.281 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d3a07b0:::benchmark_data_smithi179_148205_object87488:head# 2024-02-09T17:57:33.281 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.281 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d3b8607:::benchmark_data_smithi179_148205_object64084:head# 2024-02-09T17:57:33.281 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.281 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d3c94da:::benchmark_data_smithi179_148205_object9311:head# 2024-02-09T17:57:33.282 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.282 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d3d7853:::benchmark_data_smithi179_148205_object109607:head# 2024-02-09T17:57:33.282 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.282 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d3dc2f3:::benchmark_data_smithi179_148205_object62394:head# 2024-02-09T17:57:33.282 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.282 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d3e5989:::benchmark_data_smithi179_148205_object71710:head# 2024-02-09T17:57:33.283 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.283 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d3e8633:::benchmark_data_smithi179_148205_object78236:head# 2024-02-09T17:57:33.283 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.283 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d3e9064:::benchmark_data_smithi179_148205_object25190:head# 2024-02-09T17:57:33.283 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.283 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d3f3cd2:::benchmark_data_smithi179_148205_object16064:head# 2024-02-09T17:57:33.283 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.284 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d3f92f1:::benchmark_data_smithi179_148205_object81451:head# 2024-02-09T17:57:33.284 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.284 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d3f95c0:::benchmark_data_smithi179_148205_object103481:head# 2024-02-09T17:57:33.284 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.284 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d3ff45c:::benchmark_data_smithi179_148205_object85645:head# 2024-02-09T17:57:33.284 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.284 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d401c27:::benchmark_data_smithi179_148205_object8430:head# 2024-02-09T17:57:33.284 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.286 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d40ac21:::benchmark_data_smithi179_148205_object24014:head# 2024-02-09T17:57:33.286 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.286 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d41f4d5:::benchmark_data_smithi179_148205_object63941:head# 2024-02-09T17:57:33.286 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.286 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d42312d:::benchmark_data_smithi179_148205_object67993:head# 2024-02-09T17:57:33.286 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.286 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d435512:::benchmark_data_smithi179_148205_object60635:head# 2024-02-09T17:57:33.286 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.286 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d4359d8:::benchmark_data_smithi179_148205_object102944:head# 2024-02-09T17:57:33.286 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.287 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d439916:::benchmark_data_smithi179_148205_object61597:head# 2024-02-09T17:57:33.287 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.287 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d446270:::benchmark_data_smithi179_148205_object104308:head# 2024-02-09T17:57:33.287 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.288 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d446632:::benchmark_data_smithi179_148205_object28282:head# 2024-02-09T17:57:33.288 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.288 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d45c58f:::benchmark_data_smithi179_148205_object6929:head# 2024-02-09T17:57:33.288 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.288 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d466e9f:::benchmark_data_smithi179_148205_object38596:head# 2024-02-09T17:57:33.288 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.288 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d468bfb:::benchmark_data_smithi179_148205_object59081:head# 2024-02-09T17:57:33.288 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.289 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d477f48:::benchmark_data_smithi179_148205_object77677:head# 2024-02-09T17:57:33.289 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.289 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d4821d6:::benchmark_data_smithi179_148205_object70988:head# 2024-02-09T17:57:33.289 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.289 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d48f27a:::benchmark_data_smithi179_148205_object36392:head# 2024-02-09T17:57:33.289 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.289 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d4923fe:::benchmark_data_smithi179_148205_object79941:head# 2024-02-09T17:57:33.289 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.292 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d496f3f:::benchmark_data_smithi179_148205_object72655:head# 2024-02-09T17:57:33.293 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.296 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d497c86:::benchmark_data_smithi179_148205_object7564:head# 2024-02-09T17:57:33.296 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.296 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d49e46d:::benchmark_data_smithi179_148205_object97098:head# 2024-02-09T17:57:33.296 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.296 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d4abfc3:::benchmark_data_smithi179_148205_object50419:head# 2024-02-09T17:57:33.297 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.297 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d4bcdc3:::benchmark_data_smithi179_148205_object65890:head# 2024-02-09T17:57:33.297 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.297 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d4c0583:::benchmark_data_smithi179_148205_object9404:head# 2024-02-09T17:57:33.297 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.297 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d4c5532:::benchmark_data_smithi179_148205_object104514:head# 2024-02-09T17:57:33.297 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.298 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d4c920b:::benchmark_data_smithi179_148205_object77994:head# 2024-02-09T17:57:33.298 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.298 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d4cea0a:::benchmark_data_smithi179_148205_object57345:head# 2024-02-09T17:57:33.298 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.298 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d4d0c0c:::benchmark_data_smithi179_148205_object106019:head# 2024-02-09T17:57:33.299 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.299 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d4d2291:::benchmark_data_smithi179_148205_object82224:head# 2024-02-09T17:57:33.299 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.299 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d4d6b8f:::benchmark_data_smithi179_148205_object5909:head# 2024-02-09T17:57:33.299 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.300 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d4f6b07:::benchmark_data_smithi179_148205_object13664:head# 2024-02-09T17:57:33.300 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.300 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d4f93ba:::benchmark_data_smithi179_148205_object54595:head# 2024-02-09T17:57:33.300 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.300 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d514e15:::benchmark_data_smithi179_148205_object91364:head# 2024-02-09T17:57:33.300 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.300 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d51612d:::benchmark_data_smithi179_148205_object5670:head# 2024-02-09T17:57:33.300 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.301 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d51f3a1:::benchmark_data_smithi179_148205_object3287:head# 2024-02-09T17:57:33.301 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.301 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d525be1:::benchmark_data_smithi179_148205_object70983:head# 2024-02-09T17:57:33.301 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.301 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d5395a1:::benchmark_data_smithi179_148205_object59715:head# 2024-02-09T17:57:33.301 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.301 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d54cdfb:::benchmark_data_smithi179_148205_object6661:head# 2024-02-09T17:57:33.301 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.304 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d553a6b:::benchmark_data_smithi179_148205_object57104:head# 2024-02-09T17:57:33.305 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.306 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d554bab:::benchmark_data_smithi179_148205_object77379:head# 2024-02-09T17:57:33.306 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.311 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d5624fc:::benchmark_data_smithi179_148205_object13645:head# 2024-02-09T17:57:33.312 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.314 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d5665d2:::benchmark_data_smithi179_148205_object45303:head# 2024-02-09T17:57:33.314 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.315 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d56a533:::benchmark_data_smithi179_148205_object7884:head# 2024-02-09T17:57:33.315 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.315 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d5750f7:::benchmark_data_smithi179_148205_object86454:head# 2024-02-09T17:57:33.315 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.315 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d577f0b:::benchmark_data_smithi179_148205_object91664:head# 2024-02-09T17:57:33.316 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.316 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d57f626:::benchmark_data_smithi179_148205_object61753:head# 2024-02-09T17:57:33.316 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.319 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d5a09eb:::benchmark_data_smithi179_148205_object23582:head# 2024-02-09T17:57:33.319 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.320 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d5b9b52:::benchmark_data_smithi179_148205_object65876:head# 2024-02-09T17:57:33.320 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.320 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d5cda47:::benchmark_data_smithi179_148205_object87319:head# 2024-02-09T17:57:33.320 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.323 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d5d3cdd:::benchmark_data_smithi179_148205_object29110:head# 2024-02-09T17:57:33.323 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.323 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d5d70ef:::benchmark_data_smithi179_148205_object89399:head# 2024-02-09T17:57:33.324 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.324 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d5de3ff:::benchmark_data_smithi179_148205_object9758:head# 2024-02-09T17:57:33.324 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.325 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d5eb4db:::benchmark_data_smithi179_148205_object64292:head# 2024-02-09T17:57:33.325 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.325 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d5ef1b7:::benchmark_data_smithi179_148205_object29753:head# 2024-02-09T17:57:33.325 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.325 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d5fe2ab:::benchmark_data_smithi179_148205_object81771:head# 2024-02-09T17:57:33.325 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.325 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d5ff0a8:::benchmark_data_smithi179_148205_object66738:head# 2024-02-09T17:57:33.325 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.326 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d60717b:::benchmark_data_smithi179_148205_object95675:head# 2024-02-09T17:57:33.326 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.326 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d60e86b:::benchmark_data_smithi179_148205_object1837:head# 2024-02-09T17:57:33.326 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.329 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d64aa95:::benchmark_data_smithi179_148205_object97097:head# 2024-02-09T17:57:33.329 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.339 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d64c688:::benchmark_data_smithi179_148205_object45168:head# 2024-02-09T17:57:33.339 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.341 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d654e6d:::benchmark_data_smithi179_148205_object94324:head# 2024-02-09T17:57:33.341 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.342 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d65985a:::benchmark_data_smithi179_148205_object13164:head# 2024-02-09T17:57:33.342 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.342 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d66a367:::benchmark_data_smithi179_148205_object92825:head# 2024-02-09T17:57:33.342 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.342 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d67b877:::benchmark_data_smithi179_148205_object46426:head# 2024-02-09T17:57:33.343 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.343 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d689b16:::benchmark_data_smithi179_148205_object59581:head# 2024-02-09T17:57:33.343 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.343 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d6931f8:::benchmark_data_smithi179_148205_object84941:head# 2024-02-09T17:57:33.344 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.344 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d6a4ff5:::benchmark_data_smithi179_148205_object82741:head# 2024-02-09T17:57:33.344 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.344 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d6a7080:::benchmark_data_smithi179_148205_object32041:head# 2024-02-09T17:57:33.344 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.345 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d6b1271:::benchmark_data_smithi179_148205_object380:head# 2024-02-09T17:57:33.345 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.345 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d6b7345:::benchmark_data_smithi179_148205_object72140:head# 2024-02-09T17:57:33.345 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.345 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d6b9641:::benchmark_data_smithi179_148205_object36876:head# 2024-02-09T17:57:33.345 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.345 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d6c0480:::benchmark_data_smithi179_148205_object70927:head# 2024-02-09T17:57:33.345 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.345 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d6c279a:::benchmark_data_smithi179_148205_object35500:head# 2024-02-09T17:57:33.345 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.346 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d6c9519:::benchmark_data_smithi179_148205_object39769:head# 2024-02-09T17:57:33.346 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.346 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d6d9a0b:::benchmark_data_smithi179_148205_object85709:head# 2024-02-09T17:57:33.346 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.346 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d6dc205:::benchmark_data_smithi179_148205_object29107:head# 2024-02-09T17:57:33.347 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.347 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d6e1f10:::benchmark_data_smithi179_148205_object92383:head# 2024-02-09T17:57:33.347 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.351 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d6f1830:::benchmark_data_smithi179_148205_object83992:head# 2024-02-09T17:57:33.352 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.358 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d6f23ae:::benchmark_data_smithi179_148205_object2868:head# 2024-02-09T17:57:33.358 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.358 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d6f83ed:::benchmark_data_smithi179_148205_object94790:head# 2024-02-09T17:57:33.358 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.358 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d6fafa5:::benchmark_data_smithi179_148205_object59184:head# 2024-02-09T17:57:33.358 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.359 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d6ffbcf:::benchmark_data_smithi179_148205_object109060:head# 2024-02-09T17:57:33.359 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.359 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d70dd05:::benchmark_data_smithi179_148205_object40523:head# 2024-02-09T17:57:33.359 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.360 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d712fca:::benchmark_data_smithi179_148205_object37995:head# 2024-02-09T17:57:33.360 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.360 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d740eb6:::benchmark_data_smithi179_148205_object57348:head# 2024-02-09T17:57:33.360 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.361 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d742adb:::benchmark_data_smithi179_148205_object88460:head# 2024-02-09T17:57:33.361 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.361 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d745cb8:::benchmark_data_smithi179_148205_object8445:head# 2024-02-09T17:57:33.361 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.361 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d74a6a5:::benchmark_data_smithi179_148205_object22457:head# 2024-02-09T17:57:33.361 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.361 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d74b248:::benchmark_data_smithi179_148205_object34285:head# 2024-02-09T17:57:33.361 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.361 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d751c0f:::benchmark_data_smithi179_148205_object38246:head# 2024-02-09T17:57:33.361 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.361 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d7598a0:::benchmark_data_smithi179_148205_object99255:head# 2024-02-09T17:57:33.361 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.363 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d7631ea:::benchmark_data_smithi179_148205_object95261:head# 2024-02-09T17:57:33.363 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.363 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d76f2d4:::benchmark_data_smithi179_148205_object30737:head# 2024-02-09T17:57:33.363 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.363 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d77f1b4:::benchmark_data_smithi179_148205_object109021:head# 2024-02-09T17:57:33.363 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.363 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d780b30:::benchmark_data_smithi179_148205_object77195:head# 2024-02-09T17:57:33.363 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.363 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d793101:::benchmark_data_smithi179_148205_object43276:head# 2024-02-09T17:57:33.363 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.363 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d7a010c:::benchmark_data_smithi179_148205_object78033:head# 2024-02-09T17:57:33.363 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.363 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d7a12fc:::benchmark_data_smithi179_148205_object34502:head# 2024-02-09T17:57:33.363 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.365 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d7bfb9e:::benchmark_data_smithi179_148205_object90321:head# 2024-02-09T17:57:33.365 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.365 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d7c3b2e:::benchmark_data_smithi179_148205_object80496:head# 2024-02-09T17:57:33.366 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.366 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d7c53f5:::benchmark_data_smithi179_148205_object67465:head# 2024-02-09T17:57:33.366 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.366 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d7c7a79:::benchmark_data_smithi179_148205_object56720:head# 2024-02-09T17:57:33.366 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.366 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d7c95cb:::benchmark_data_smithi179_148205_object83279:head# 2024-02-09T17:57:33.366 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.366 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d7d9139:::benchmark_data_smithi179_148205_object49105:head# 2024-02-09T17:57:33.366 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.366 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d7dc70f:::benchmark_data_smithi179_148205_object45324:head# 2024-02-09T17:57:33.366 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.366 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d7e26b6:::benchmark_data_smithi179_148205_object35911:head# 2024-02-09T17:57:33.366 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.366 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d7e2b8e:::benchmark_data_smithi179_148205_object45114:head# 2024-02-09T17:57:33.366 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.366 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d7f895a:::benchmark_data_smithi179_148205_object21575:head# 2024-02-09T17:57:33.367 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.367 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d7f9f09:::benchmark_data_smithi179_148205_object11521:head# 2024-02-09T17:57:33.367 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.368 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:57:33.371 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d819f52:::benchmark_data_smithi179_148205_object5111:head# 2024-02-09T17:57:33.372 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.372 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d81cc0d:::benchmark_data_smithi179_148205_object68047:head# 2024-02-09T17:57:33.372 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.372 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d821741:::benchmark_data_smithi179_148205_object36020:head# 2024-02-09T17:57:33.372 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.372 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d828b76:::benchmark_data_smithi179_148205_object80931:head# 2024-02-09T17:57:33.372 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.372 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d82c540:::benchmark_data_smithi179_148205_object51058:head# 2024-02-09T17:57:33.372 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.372 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d82d0a2:::benchmark_data_smithi179_148205_object41609:head# 2024-02-09T17:57:33.372 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.372 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d835942:::benchmark_data_smithi179_148205_object74756:head# 2024-02-09T17:57:33.373 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.373 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d84041d:::benchmark_data_smithi179_148205_object22632:head# 2024-02-09T17:57:33.373 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.373 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d84b71c:::benchmark_data_smithi179_148205_object10502:head# 2024-02-09T17:57:33.373 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.373 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d84e470:::benchmark_data_smithi179_148205_object35795:head# 2024-02-09T17:57:33.373 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.373 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d856f42:::benchmark_data_smithi179_148205_object22167:head# 2024-02-09T17:57:33.373 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.373 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d85cd3e:::benchmark_data_smithi179_148205_object35871:head# 2024-02-09T17:57:33.373 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.373 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d85d8cf:::benchmark_data_smithi179_148205_object23105:head# 2024-02-09T17:57:33.373 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.374 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d85f751:::benchmark_data_smithi179_148205_object10406:head# 2024-02-09T17:57:33.374 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.374 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d863735:::benchmark_data_smithi179_148205_object87714:head# 2024-02-09T17:57:33.374 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.374 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d874725:::benchmark_data_smithi179_148205_object15673:head# 2024-02-09T17:57:33.374 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.374 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d880397:::benchmark_data_smithi179_148205_object48752:head# 2024-02-09T17:57:33.374 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.374 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d88611c:::benchmark_data_smithi179_148205_object97071:head# 2024-02-09T17:57:33.374 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.374 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d8a375d:::benchmark_data_smithi179_148205_object23680:head# 2024-02-09T17:57:33.374 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.374 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d8a9e34:::benchmark_data_smithi179_148205_object40701:head# 2024-02-09T17:57:33.375 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.375 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d8b6b74:::benchmark_data_smithi179_148205_object102165:head# 2024-02-09T17:57:33.375 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.375 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d8c5a15:::benchmark_data_smithi179_148205_object52828:head# 2024-02-09T17:57:33.375 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.375 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d8c5d04:::benchmark_data_smithi179_148205_object45509:head# 2024-02-09T17:57:33.375 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.377 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d8c990e:::benchmark_data_smithi179_148205_object38868:head# 2024-02-09T17:57:33.377 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.378 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d8f2e8c:::benchmark_data_smithi179_148205_object70577:head# 2024-02-09T17:57:33.378 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.378 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d8f7ac8:::benchmark_data_smithi179_148205_object79416:head# 2024-02-09T17:57:33.378 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.378 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d8fb4ea:::benchmark_data_smithi179_148205_object14552:head# 2024-02-09T17:57:33.378 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.378 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d924a9d:::benchmark_data_smithi179_148205_object63380:head# 2024-02-09T17:57:33.378 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.378 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d92ffeb:::benchmark_data_smithi179_148205_object3457:head# 2024-02-09T17:57:33.378 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.378 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d934399:::benchmark_data_smithi179_148205_object47488:head# 2024-02-09T17:57:33.378 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.378 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d944caa:::benchmark_data_smithi179_148205_object72744:head# 2024-02-09T17:57:33.378 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.381 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d960852:::benchmark_data_smithi179_148205_object54341:head# 2024-02-09T17:57:33.382 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.382 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d96f4ed:::benchmark_data_smithi179_148205_object57838:head# 2024-02-09T17:57:33.382 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.388 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d97f81e:::benchmark_data_smithi179_148205_object28579:head# 2024-02-09T17:57:33.388 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.388 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d984492:::benchmark_data_smithi179_148205_object61549:head# 2024-02-09T17:57:33.388 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.388 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d985f5a:::benchmark_data_smithi179_148205_object43726:head# 2024-02-09T17:57:33.388 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.388 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d991253:::benchmark_data_smithi179_148205_object8006:head# 2024-02-09T17:57:33.388 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.388 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d9aa80a:::benchmark_data_smithi179_148205_object74353:head# 2024-02-09T17:57:33.388 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.388 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d9ab6df:::benchmark_data_smithi179_148205_object87987:head# 2024-02-09T17:57:33.388 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.388 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d9b2642:::benchmark_data_smithi179_148205_object16212:head# 2024-02-09T17:57:33.389 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.389 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d9b2961:::benchmark_data_smithi179_148205_object2698:head# 2024-02-09T17:57:33.389 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.389 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d9c2ddd:::benchmark_data_smithi179_148205_object56471:head# 2024-02-09T17:57:33.389 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.389 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d9c6317:::benchmark_data_smithi179_148205_object65301:head# 2024-02-09T17:57:33.389 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.389 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d9c7ed3:::benchmark_data_smithi179_148205_object62951:head# 2024-02-09T17:57:33.389 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.389 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d9cd890:::benchmark_data_smithi179_148205_object22337:head# 2024-02-09T17:57:33.389 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.389 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d9cde67:::benchmark_data_smithi179_148205_object50341:head# 2024-02-09T17:57:33.389 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.389 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d9d5b6f:::benchmark_data_smithi179_148205_object44685:head# 2024-02-09T17:57:33.389 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.389 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d9dc4bc:::benchmark_data_smithi179_148205_object96308:head# 2024-02-09T17:57:33.390 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.390 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d9e2714:::benchmark_data_smithi179_148205_object42742:head# 2024-02-09T17:57:33.390 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.390 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d9f31b7:::benchmark_data_smithi179_148205_object108971:head# 2024-02-09T17:57:33.390 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.390 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d9f6ac5:::benchmark_data_smithi179_148205_object38540:head# 2024-02-09T17:57:33.390 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.390 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d9fdb61:::benchmark_data_smithi179_148205_object48241:head# 2024-02-09T17:57:33.390 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.390 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1d9fe1ed:::benchmark_data_smithi179_148205_object13012:head# 2024-02-09T17:57:33.390 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.390 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1da0192b:::benchmark_data_smithi179_148205_object52956:head# 2024-02-09T17:57:33.390 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.390 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1da0d946:::benchmark_data_smithi179_148205_object102687:head# 2024-02-09T17:57:33.390 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.391 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1da12144:::benchmark_data_smithi179_148205_object17229:head# 2024-02-09T17:57:33.391 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.391 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1da19d36:::benchmark_data_smithi179_148205_object45191:head# 2024-02-09T17:57:33.391 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.391 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1da1f12d:::benchmark_data_smithi179_148205_object1095:head# 2024-02-09T17:57:33.391 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.391 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1da2169d:::benchmark_data_smithi179_148205_object48024:head# 2024-02-09T17:57:33.391 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.393 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1da41416:::benchmark_data_smithi179_148205_object67696:head# 2024-02-09T17:57:33.393 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.393 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1da54c69:::benchmark_data_smithi179_148205_object19106:head# 2024-02-09T17:57:33.393 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.393 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1da55909:::benchmark_data_smithi179_148205_object34869:head# 2024-02-09T17:57:33.393 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.393 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1da5bb6f:::benchmark_data_smithi179_148205_object24138:head# 2024-02-09T17:57:33.393 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.393 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1da6813d:::benchmark_data_smithi179_148205_object74517:head# 2024-02-09T17:57:33.393 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.393 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1da6c033:::benchmark_data_smithi179_148205_object95145:head# 2024-02-09T17:57:33.394 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.394 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1da6e4e6:::benchmark_data_smithi179_148205_object38694:head# 2024-02-09T17:57:33.394 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.394 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1da7978e:::benchmark_data_smithi179_148205_object87135:head# 2024-02-09T17:57:33.394 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.394 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1da799a0:::benchmark_data_smithi179_148205_object14913:head# 2024-02-09T17:57:33.394 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.394 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1da7ccda:::benchmark_data_smithi179_148205_object6270:head# 2024-02-09T17:57:33.394 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.394 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1da905e7:::benchmark_data_smithi179_148205_object29891:head# 2024-02-09T17:57:33.394 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.394 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1da9b0df:::benchmark_data_smithi179_148205_object80485:head# 2024-02-09T17:57:33.394 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.394 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1da9b37d:::benchmark_data_smithi179_148205_object58468:head# 2024-02-09T17:57:33.394 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.395 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1da9c371:::benchmark_data_smithi179_148205_object78446:head# 2024-02-09T17:57:33.395 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.395 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1da9ff8b:::benchmark_data_smithi179_148205_object3680:head# 2024-02-09T17:57:33.395 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.395 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dab0448:::benchmark_data_smithi179_148205_object29875:head# 2024-02-09T17:57:33.395 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.395 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dabaa3f:::benchmark_data_smithi179_148205_object17547:head# 2024-02-09T17:57:33.395 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.398 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dabb08b:::benchmark_data_smithi179_148205_object34513:head# 2024-02-09T17:57:33.398 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.398 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dac23d1:::benchmark_data_smithi179_148205_object9489:head# 2024-02-09T17:57:33.398 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.398 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dac3135:::benchmark_data_smithi179_148205_object43410:head# 2024-02-09T17:57:33.398 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.399 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dad3a53:::benchmark_data_smithi179_148205_object37643:head# 2024-02-09T17:57:33.399 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.399 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dae004f:::benchmark_data_smithi179_148205_object54785:head# 2024-02-09T17:57:33.399 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.399 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dafa2c3:::benchmark_data_smithi179_148205_object57612:head# 2024-02-09T17:57:33.399 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.399 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1db000c3:::benchmark_data_smithi179_148205_object70528:head# 2024-02-09T17:57:33.399 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.399 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1db096fd:::benchmark_data_smithi179_148205_object102102:head# 2024-02-09T17:57:33.399 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.399 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1db0ddea:::benchmark_data_smithi179_148205_object16944:head# 2024-02-09T17:57:33.399 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.399 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1db13cc0:::benchmark_data_smithi179_148205_object16560:head# 2024-02-09T17:57:33.399 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.399 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1db2108d:::benchmark_data_smithi179_148205_object71015:head# 2024-02-09T17:57:33.399 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.399 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1db3880a:::benchmark_data_smithi179_148205_object2595:head# 2024-02-09T17:57:33.400 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.400 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1db3f9f6:::benchmark_data_smithi179_148205_object60937:head# 2024-02-09T17:57:33.400 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.400 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1db44d0d:::benchmark_data_smithi179_148205_object10348:head# 2024-02-09T17:57:33.400 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.400 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1db555f3:::benchmark_data_smithi179_148205_object101779:head# 2024-02-09T17:57:33.400 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.400 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1db64dd4:::benchmark_data_smithi179_148205_object23325:head# 2024-02-09T17:57:33.400 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.400 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1db66a23:::benchmark_data_smithi179_148205_object108272:head# 2024-02-09T17:57:33.400 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.400 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1db6ea0d:::benchmark_data_smithi179_148205_object79465:head# 2024-02-09T17:57:33.400 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.401 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1db6fa9b:::benchmark_data_smithi179_148205_object58283:head# 2024-02-09T17:57:33.401 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.401 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1db73bff:::benchmark_data_smithi179_148205_object100552:head# 2024-02-09T17:57:33.401 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.402 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1db766fd:::benchmark_data_smithi179_148205_object59276:head# 2024-02-09T17:57:33.402 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.403 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1db7e9d8:::benchmark_data_smithi179_148205_object26500:head# 2024-02-09T17:57:33.403 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.403 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1db91d96:::benchmark_data_smithi179_148205_object95795:head# 2024-02-09T17:57:33.403 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.403 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1db99d2e:::benchmark_data_smithi179_148205_object25614:head# 2024-02-09T17:57:33.403 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.403 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dba8e61:::benchmark_data_smithi179_148205_object42259:head# 2024-02-09T17:57:33.404 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.404 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dbb3464:::benchmark_data_smithi179_148205_object70002:head# 2024-02-09T17:57:33.404 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.405 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dbbdfcc:::benchmark_data_smithi179_148205_object7997:head# 2024-02-09T17:57:33.405 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.405 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dbc2a1e:::benchmark_data_smithi179_148205_object69539:head# 2024-02-09T17:57:33.405 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.405 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dbe4f28:::benchmark_data_smithi179_148205_object20003:head# 2024-02-09T17:57:33.406 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.406 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dbf90d7:::benchmark_data_smithi179_148205_object44361:head# 2024-02-09T17:57:33.406 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.406 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dbff703:::benchmark_data_smithi179_148205_object101212:head# 2024-02-09T17:57:33.406 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.406 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dc07f23:::benchmark_data_smithi179_148205_object96847:head# 2024-02-09T17:57:33.406 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.406 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dc0b4fd:::benchmark_data_smithi179_148205_object19450:head# 2024-02-09T17:57:33.406 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.406 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dc0fc16:::benchmark_data_smithi179_148205_object53629:head# 2024-02-09T17:57:33.406 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.407 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:33 smithi179 ceph-mon[43975]: 1.12 scrub starts 2024-02-09T17:57:33.407 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:33 smithi179 ceph-mon[43975]: 1.12 scrub ok 2024-02-09T17:57:33.407 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:33 smithi179 ceph-mon[43975]: 8.d scrub ok 2024-02-09T17:57:33.407 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:33 smithi179 ceph-mon[43975]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:57:33.407 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:33 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:57:33.407 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:33 smithi179 ceph-mon[43975]: osdmap e828: 12 total, 11 up, 12 in 2024-02-09T17:57:33.410 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dc140de:::benchmark_data_smithi179_148205_object58296:head# 2024-02-09T17:57:33.410 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.410 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dc1a362:::benchmark_data_smithi179_148205_object17680:head# 2024-02-09T17:57:33.410 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.410 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dc1bdf4:::benchmark_data_smithi179_148205_object107698:head# 2024-02-09T17:57:33.410 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.410 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dc204aa:::benchmark_data_smithi179_148205_object4033:head# 2024-02-09T17:57:33.410 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.411 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dc42a34:::benchmark_data_smithi179_148205_object61628:head# 2024-02-09T17:57:33.411 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.411 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dc4b908:::benchmark_data_smithi179_148205_object109328:head# 2024-02-09T17:57:33.411 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.411 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dc4dce2:::benchmark_data_smithi179_148205_object58138:head# 2024-02-09T17:57:33.411 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.411 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dc53a82:::benchmark_data_smithi179_148205_object50001:head# 2024-02-09T17:57:33.411 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.411 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dc58c94:::benchmark_data_smithi179_148205_object14325:head# 2024-02-09T17:57:33.411 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.411 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dc5a9c1:::benchmark_data_smithi179_148205_object25036:head# 2024-02-09T17:57:33.411 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.411 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dc66b29:::benchmark_data_smithi179_148205_object97113:head# 2024-02-09T17:57:33.411 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.411 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dc73276:::benchmark_data_smithi179_148205_object11796:head# 2024-02-09T17:57:33.411 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.412 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dc754ce:::benchmark_data_smithi179_148205_object71408:head# 2024-02-09T17:57:33.412 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.412 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dc7cac9:::benchmark_data_smithi179_148205_object104822:head# 2024-02-09T17:57:33.412 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.412 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dc8746a:::benchmark_data_smithi179_148205_object39359:head# 2024-02-09T17:57:33.412 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.412 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dc8cf4f:::benchmark_data_smithi179_148205_object77167:head# 2024-02-09T17:57:33.412 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.412 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dc99465:::benchmark_data_smithi179_148205_object92917:head# 2024-02-09T17:57:33.412 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.412 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dc9ec66:::benchmark_data_smithi179_148205_object39267:head# 2024-02-09T17:57:33.412 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.412 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dcb7241:::benchmark_data_smithi179_148205_object76677:head# 2024-02-09T17:57:33.412 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.413 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:33.415 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dcc2a07:::benchmark_data_smithi179_148205_object89770:head# 2024-02-09T17:57:33.415 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.415 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dcc76a7:::benchmark_data_smithi179_148205_object59354:head# 2024-02-09T17:57:33.416 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.416 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dcd826f:::benchmark_data_smithi179_148205_object68706:head# 2024-02-09T17:57:33.416 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.416 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dce866c:::benchmark_data_smithi179_148205_object3932:head# 2024-02-09T17:57:33.416 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.416 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dced314:::benchmark_data_smithi179_148205_object8917:head# 2024-02-09T17:57:33.416 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.416 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dcf213e:::benchmark_data_smithi179_148205_object76429:head# 2024-02-09T17:57:33.416 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.416 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dcfdea3:::benchmark_data_smithi179_148205_object2039:head# 2024-02-09T17:57:33.416 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.416 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dd0115e:::benchmark_data_smithi179_148205_object81414:head# 2024-02-09T17:57:33.416 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.416 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dd0b2f3:::benchmark_data_smithi179_148205_object45543:head# 2024-02-09T17:57:33.417 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.417 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dd1a813:::benchmark_data_smithi179_148205_object103002:head# 2024-02-09T17:57:33.417 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.417 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dd37388:::benchmark_data_smithi179_148205_object49246:head# 2024-02-09T17:57:33.417 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.417 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dd40510:::benchmark_data_smithi179_148205_object36051:head# 2024-02-09T17:57:33.417 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.417 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dd44e28:::benchmark_data_smithi179_148205_object16004:head# 2024-02-09T17:57:33.417 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.417 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dd44ea5:::benchmark_data_smithi179_148205_object81692:head# 2024-02-09T17:57:33.417 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.417 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dd4d348:::benchmark_data_smithi179_148205_object25658:head# 2024-02-09T17:57:33.417 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.417 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dd66e14:::benchmark_data_smithi179_148205_object44849:head# 2024-02-09T17:57:33.417 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.418 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dd71188:::benchmark_data_smithi179_148205_object2531:head# 2024-02-09T17:57:33.418 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.418 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dd776a7:::benchmark_data_smithi179_148205_object236:head# 2024-02-09T17:57:33.418 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.418 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dd7805b:::benchmark_data_smithi179_148205_object19075:head# 2024-02-09T17:57:33.418 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.418 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dd92d5b:::benchmark_data_smithi179_148205_object77552:head# 2024-02-09T17:57:33.418 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.423 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dd9c138:::benchmark_data_smithi179_148205_object23225:head# 2024-02-09T17:57:33.424 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.424 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dda04f4:::benchmark_data_smithi179_148205_object85790:head# 2024-02-09T17:57:33.424 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.424 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dda0632:::benchmark_data_smithi179_148205_object21573:head# 2024-02-09T17:57:33.424 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.424 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dda6dbd:::benchmark_data_smithi179_148205_object68440:head# 2024-02-09T17:57:33.424 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.424 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ddb1582:::benchmark_data_smithi179_148205_object68533:head# 2024-02-09T17:57:33.424 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.424 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ddc03ea:::benchmark_data_smithi179_148205_object46390:head# 2024-02-09T17:57:33.424 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.424 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ddc383a:::benchmark_data_smithi179_148205_object32749:head# 2024-02-09T17:57:33.424 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.424 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dde30ce:::benchmark_data_smithi179_148205_object98857:head# 2024-02-09T17:57:33.425 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.425 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ddefe90:::benchmark_data_smithi179_148205_object70188:head# 2024-02-09T17:57:33.425 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.425 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ddf4a19:::benchmark_data_smithi179_148205_object32567:head# 2024-02-09T17:57:33.425 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.425 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ddfa9db:::benchmark_data_smithi179_148205_object63539:head# 2024-02-09T17:57:33.425 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.425 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ddfbcf7:::benchmark_data_smithi179_148205_object102952:head# 2024-02-09T17:57:33.425 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.425 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ddff8ee:::benchmark_data_smithi179_148205_object107884:head# 2024-02-09T17:57:33.425 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.425 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1de009ca:::benchmark_data_smithi179_148205_object106788:head# 2024-02-09T17:57:33.425 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.425 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1de095b2:::benchmark_data_smithi179_148205_object57067:head# 2024-02-09T17:57:33.425 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.426 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1de13b7d:::benchmark_data_smithi179_148205_object79700:head# 2024-02-09T17:57:33.426 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.426 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1de202f9:::benchmark_data_smithi179_148205_object26082:head# 2024-02-09T17:57:33.426 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.426 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1de32cef:::benchmark_data_smithi179_148205_object11011:head# 2024-02-09T17:57:33.426 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.426 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1de3b830:::benchmark_data_smithi179_148205_object35188:head# 2024-02-09T17:57:33.426 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.426 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1de3f64c:::benchmark_data_smithi179_148205_object11324:head# 2024-02-09T17:57:33.426 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.426 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1de414bd:::benchmark_data_smithi179_148205_object54940:head# 2024-02-09T17:57:33.426 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.426 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1de62c5c:::benchmark_data_smithi179_148205_object51083:head# 2024-02-09T17:57:33.426 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.426 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1de68fb9:::benchmark_data_smithi179_148205_object22796:head# 2024-02-09T17:57:33.426 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.427 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1de6ad10:::benchmark_data_smithi179_148205_object83447:head# 2024-02-09T17:57:33.427 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.427 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1de70347:::benchmark_data_smithi179_148205_object37629:head# 2024-02-09T17:57:33.427 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.427 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1de7aa0b:::benchmark_data_smithi179_148205_object68918:head# 2024-02-09T17:57:33.427 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.427 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1de875c3:::benchmark_data_smithi179_148205_object80445:head# 2024-02-09T17:57:33.427 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.427 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1de8ecc7:::benchmark_data_smithi179_148205_object98272:head# 2024-02-09T17:57:33.427 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.427 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1de8f636:::benchmark_data_smithi179_148205_object13995:head# 2024-02-09T17:57:33.427 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.427 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1de97feb:::benchmark_data_smithi179_148205_object29488:head# 2024-02-09T17:57:33.427 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.427 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1de9f123:::benchmark_data_smithi179_148205_object26183:head# 2024-02-09T17:57:33.427 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.428 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dea8370:::benchmark_data_smithi179_148205_object43245:head# 2024-02-09T17:57:33.428 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.433 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1deaa93d:::benchmark_data_smithi179_148205_object89918:head# 2024-02-09T17:57:33.433 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.433 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1deaccfb:::benchmark_data_smithi179_148205_object31066:head# 2024-02-09T17:57:33.433 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.433 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1deb1c6e:::benchmark_data_smithi179_148205_object64674:head# 2024-02-09T17:57:33.433 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.433 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1decfa3a:::benchmark_data_smithi179_148205_object31195:head# 2024-02-09T17:57:33.434 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.434 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1deeba76:::benchmark_data_smithi179_148205_object65531:head# 2024-02-09T17:57:33.434 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.434 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1deed56a:::benchmark_data_smithi179_148205_object21047:head# 2024-02-09T17:57:33.434 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.434 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1deefc7d:::benchmark_data_smithi179_148205_object74592:head# 2024-02-09T17:57:33.434 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.434 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1defd44c:::benchmark_data_smithi179_148205_object42260:head# 2024-02-09T17:57:33.434 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.434 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1defe04d:::benchmark_data_smithi179_148205_object49002:head# 2024-02-09T17:57:33.434 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.434 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1df034b9:::benchmark_data_smithi179_148205_object25932:head# 2024-02-09T17:57:33.434 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.434 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1df0e0c0:::benchmark_data_smithi179_148205_object17465:head# 2024-02-09T17:57:33.435 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.435 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1df116f5:::benchmark_data_smithi179_148205_object74701:head# 2024-02-09T17:57:33.435 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.435 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1df1acfb:::benchmark_data_smithi179_148205_object68286:head# 2024-02-09T17:57:33.435 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.435 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1df21082:::benchmark_data_smithi179_148205_object28260:head# 2024-02-09T17:57:33.435 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.435 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1df2b155:::benchmark_data_smithi179_148205_object5528:head# 2024-02-09T17:57:33.435 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.435 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1df31ec7:::benchmark_data_smithi179_148205_object87143:head# 2024-02-09T17:57:33.435 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.435 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1df35e9d:::benchmark_data_smithi179_148205_object4721:head# 2024-02-09T17:57:33.435 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.435 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1df3daaa:::benchmark_data_smithi179_148205_object84551:head# 2024-02-09T17:57:33.436 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.436 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1df47144:::benchmark_data_smithi179_148205_object64439:head# 2024-02-09T17:57:33.436 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.436 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1df4c7be:::benchmark_data_smithi179_148205_object64580:head# 2024-02-09T17:57:33.436 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.436 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1df539ae:::benchmark_data_smithi179_148205_object64911:head# 2024-02-09T17:57:33.436 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.436 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1df54935:::benchmark_data_smithi179_148205_object74398:head# 2024-02-09T17:57:33.436 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.436 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1df556f9:::benchmark_data_smithi179_148205_object72507:head# 2024-02-09T17:57:33.436 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.436 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1df56bcd:::benchmark_data_smithi179_148205_object96760:head# 2024-02-09T17:57:33.436 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.436 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1df585c6:::benchmark_data_smithi179_148205_object6821:head# 2024-02-09T17:57:33.437 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.437 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1df63e98:::benchmark_data_smithi179_148205_object24727:head# 2024-02-09T17:57:33.437 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.437 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1df6a27b:::benchmark_data_smithi179_148205_object106270:head# 2024-02-09T17:57:33.437 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.437 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1df6d941:::benchmark_data_smithi179_148205_object52544:head# 2024-02-09T17:57:33.437 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.437 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1df70d46:::benchmark_data_smithi179_148205_object61022:head# 2024-02-09T17:57:33.437 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.437 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1df78e59:::benchmark_data_smithi179_148205_object34223:head# 2024-02-09T17:57:33.437 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.437 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1df96a77:::benchmark_data_smithi179_148205_object54009:head# 2024-02-09T17:57:33.437 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.437 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dfb1c54:::benchmark_data_smithi179_148205_object56406:head# 2024-02-09T17:57:33.437 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.438 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dfb210e:::benchmark_data_smithi179_148205_object76623:head# 2024-02-09T17:57:33.438 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.438 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dfc8ac7:::benchmark_data_smithi179_148205_object16547:head# 2024-02-09T17:57:33.438 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.438 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dfd3308:::benchmark_data_smithi179_148205_object5996:head# 2024-02-09T17:57:33.438 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.438 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dfd669f:::benchmark_data_smithi179_148205_object94444:head# 2024-02-09T17:57:33.438 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.438 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dfe5001:::benchmark_data_smithi179_148205_object54142:head# 2024-02-09T17:57:33.438 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.438 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dfe6e90:::benchmark_data_smithi179_148205_object15653:head# 2024-02-09T17:57:33.438 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.438 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dfe87b3:::benchmark_data_smithi179_148205_object30567:head# 2024-02-09T17:57:33.439 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.439 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1dfecec8:::benchmark_data_smithi179_148205_object87908:head# 2024-02-09T17:57:33.439 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.440 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:33 smithi099 ceph-mon[39420]: 1.12 scrub starts 2024-02-09T17:57:33.440 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:33 smithi099 ceph-mon[39420]: 1.12 scrub ok 2024-02-09T17:57:33.440 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:33 smithi099 ceph-mon[39420]: 8.d scrub ok 2024-02-09T17:57:33.440 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:33 smithi099 ceph-mon[39420]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:57:33.440 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:33 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:57:33.440 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:33 smithi099 ceph-mon[39420]: osdmap e828: 12 total, 11 up, 12 in 2024-02-09T17:57:33.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:33.441 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:57:33.441 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:57:33.443 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e0099cc:::benchmark_data_smithi179_148205_object511:head# 2024-02-09T17:57:33.443 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.443 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e00d08e:::benchmark_data_smithi179_148205_object94427:head# 2024-02-09T17:57:33.443 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.444 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e0121ff:::benchmark_data_smithi179_148205_object54686:head# 2024-02-09T17:57:33.444 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.444 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e02530d:::benchmark_data_smithi179_148205_object66504:head# 2024-02-09T17:57:33.444 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.444 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e03d70d:::benchmark_data_smithi179_148205_object63615:head# 2024-02-09T17:57:33.444 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.444 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e041a14:::benchmark_data_smithi179_148205_object88794:head# 2024-02-09T17:57:33.444 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.444 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e04d07f:::benchmark_data_smithi179_148205_object35491:head# 2024-02-09T17:57:33.444 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.444 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e06c849:::benchmark_data_smithi179_148205_object11049:head# 2024-02-09T17:57:33.444 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.444 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e07653f:::benchmark_data_smithi179_148205_object30480:head# 2024-02-09T17:57:33.444 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.445 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e0836b7:::benchmark_data_smithi179_148205_object33098:head# 2024-02-09T17:57:33.445 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.445 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e092341:::benchmark_data_smithi179_148205_object103458:head# 2024-02-09T17:57:33.445 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.445 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e09adf9:::benchmark_data_smithi179_148205_object15333:head# 2024-02-09T17:57:33.445 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.445 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e09e4bc:::benchmark_data_smithi179_148205_object29157:head# 2024-02-09T17:57:33.445 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.445 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e09e587:::benchmark_data_smithi179_148205_object81232:head# 2024-02-09T17:57:33.445 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.445 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e0aebc7:::benchmark_data_smithi179_148205_object49157:head# 2024-02-09T17:57:33.445 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.445 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e0bcdb7:::benchmark_data_smithi179_148205_object69834:head# 2024-02-09T17:57:33.445 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.445 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e0c7e56:::benchmark_data_smithi179_148205_object60252:head# 2024-02-09T17:57:33.445 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.446 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e0e0ca1:::benchmark_data_smithi179_148205_object50740:head# 2024-02-09T17:57:33.446 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.446 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e0ee7cc:::benchmark_data_smithi179_148205_object9581:head# 2024-02-09T17:57:33.446 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.446 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e0f5103:::benchmark_data_smithi179_148205_object3045:head# 2024-02-09T17:57:33.446 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.448 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e0fb8da:::benchmark_data_smithi179_148205_object6029:head# 2024-02-09T17:57:33.448 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.448 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e10f4fb:::benchmark_data_smithi179_148205_object18002:head# 2024-02-09T17:57:33.448 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.448 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e11d59a:::benchmark_data_smithi179_148205_object51552:head# 2024-02-09T17:57:33.448 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.448 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e1216a1:::benchmark_data_smithi179_148205_object71046:head# 2024-02-09T17:57:33.448 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.449 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e127dae:::benchmark_data_smithi179_148205_object78720:head# 2024-02-09T17:57:33.449 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.449 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e1359fe:::benchmark_data_smithi179_148205_object43678:head# 2024-02-09T17:57:33.449 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.449 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e145e4b:::benchmark_data_smithi179_148205_object104397:head# 2024-02-09T17:57:33.449 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.449 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e167fd6:::benchmark_data_smithi179_148205_object11682:head# 2024-02-09T17:57:33.449 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.449 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e169081:::benchmark_data_smithi179_148205_object26002:head# 2024-02-09T17:57:33.449 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.449 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e1709ce:::benchmark_data_smithi179_148205_object28362:head# 2024-02-09T17:57:33.449 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.449 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e1715de:::benchmark_data_smithi179_148205_object88643:head# 2024-02-09T17:57:33.449 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.449 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e18265a:::benchmark_data_smithi179_148205_object13737:head# 2024-02-09T17:57:33.450 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.450 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e186bad:::benchmark_data_smithi179_148205_object25895:head# 2024-02-09T17:57:33.450 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.450 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e19e516:::benchmark_data_smithi179_148205_object104091:head# 2024-02-09T17:57:33.450 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.450 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e1a80bb:::benchmark_data_smithi179_148205_object44060:head# 2024-02-09T17:57:33.450 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.450 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e1b6ee9:::benchmark_data_smithi179_148205_object81159:head# 2024-02-09T17:57:33.450 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.450 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e1b7bcf:::benchmark_data_smithi179_148205_object20241:head# 2024-02-09T17:57:33.450 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.450 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e1b9743:::benchmark_data_smithi179_148205_object12151:head# 2024-02-09T17:57:33.450 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.455 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e1c94b6:::benchmark_data_smithi179_148205_object104568:head# 2024-02-09T17:57:33.455 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.456 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e1d851d:::benchmark_data_smithi179_148205_object18714:head# 2024-02-09T17:57:33.456 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.456 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e1dea5d:::benchmark_data_smithi179_148205_object42565:head# 2024-02-09T17:57:33.456 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.456 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e1e3868:::benchmark_data_smithi179_148205_object24889:head# 2024-02-09T17:57:33.456 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.456 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e1e6fe1:::benchmark_data_smithi179_148205_object57738:head# 2024-02-09T17:57:33.456 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.456 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e1ea25f:::benchmark_data_smithi179_148205_object74608:head# 2024-02-09T17:57:33.456 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.456 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e1ec169:::benchmark_data_smithi179_148205_object56816:head# 2024-02-09T17:57:33.456 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.456 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e1f4dca:::benchmark_data_smithi179_148205_object57066:head# 2024-02-09T17:57:33.456 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.457 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e202797:::benchmark_data_smithi179_148205_object85237:head# 2024-02-09T17:57:33.457 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.457 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e203e7b:::benchmark_data_smithi179_148205_object55356:head# 2024-02-09T17:57:33.457 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.457 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e20463b:::benchmark_data_smithi179_148205_object38825:head# 2024-02-09T17:57:33.457 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.457 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e20661d:::benchmark_data_smithi179_148205_object88085:head# 2024-02-09T17:57:33.457 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.457 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e207061:::benchmark_data_smithi179_148205_object85735:head# 2024-02-09T17:57:33.457 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.457 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e20f2a6:::benchmark_data_smithi179_148205_object32730:head# 2024-02-09T17:57:33.457 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.457 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e218863:::benchmark_data_smithi179_148205_object102855:head# 2024-02-09T17:57:33.457 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.457 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e22a1a0:::benchmark_data_smithi179_148205_object47035:head# 2024-02-09T17:57:33.457 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.458 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e230f41:::benchmark_data_smithi179_148205_object33060:head# 2024-02-09T17:57:33.458 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.458 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e235107:::benchmark_data_smithi179_148205_object90899:head# 2024-02-09T17:57:33.458 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.458 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e25780e:::benchmark_data_smithi179_148205_object77368:head# 2024-02-09T17:57:33.458 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.458 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e25d668:::benchmark_data_smithi179_148205_object66149:head# 2024-02-09T17:57:33.458 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.458 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e269df0:::benchmark_data_smithi179_148205_object65147:head# 2024-02-09T17:57:33.458 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.458 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e284e33:::benchmark_data_smithi179_148205_object66107:head# 2024-02-09T17:57:33.458 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.458 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e2862e6:::benchmark_data_smithi179_148205_object27213:head# 2024-02-09T17:57:33.458 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.458 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e28a280:::benchmark_data_smithi179_148205_object98746:head# 2024-02-09T17:57:33.458 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.459 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e29cd4c:::benchmark_data_smithi179_148205_object22261:head# 2024-02-09T17:57:33.459 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.459 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e2a13f0:::benchmark_data_smithi179_148205_object5599:head# 2024-02-09T17:57:33.459 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.459 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e2a56c3:::benchmark_data_smithi179_148205_object83051:head# 2024-02-09T17:57:33.459 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.459 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e2b3844:::benchmark_data_smithi179_148205_object59135:head# 2024-02-09T17:57:33.459 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.459 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e2d9091:::benchmark_data_smithi179_148205_object30096:head# 2024-02-09T17:57:33.459 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.459 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e2e03b6:::benchmark_data_smithi179_148205_object78499:head# 2024-02-09T17:57:33.459 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.459 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e2eb4ab:::benchmark_data_smithi179_148205_object24500:head# 2024-02-09T17:57:33.459 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.460 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e2ee4c9:::benchmark_data_smithi179_148205_object57910:head# 2024-02-09T17:57:33.460 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.469 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e2f19f4:::benchmark_data_smithi179_148205_object106291:head# 2024-02-09T17:57:33.469 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.469 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e2f45e8:::benchmark_data_smithi179_148205_object70716:head# 2024-02-09T17:57:33.469 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.469 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e2f65c4:::benchmark_data_smithi179_148205_object106475:head# 2024-02-09T17:57:33.469 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.469 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e2f8a4e:::benchmark_data_smithi179_148205_object98616:head# 2024-02-09T17:57:33.469 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.469 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e2ffb6c:::benchmark_data_smithi179_148205_object11529:head# 2024-02-09T17:57:33.470 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.470 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e30801c:::benchmark_data_smithi179_148205_object47207:head# 2024-02-09T17:57:33.470 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.470 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e30a6a5:::benchmark_data_smithi179_148205_object19678:head# 2024-02-09T17:57:33.470 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.470 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e30b42f:::benchmark_data_smithi179_148205_object63348:head# 2024-02-09T17:57:33.470 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.470 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e313f33:::benchmark_data_smithi179_148205_object91311:head# 2024-02-09T17:57:33.470 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.470 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e323bab:::benchmark_data_smithi179_148205_object91515:head# 2024-02-09T17:57:33.470 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.470 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e326231:::benchmark_data_smithi179_148205_object6864:head# 2024-02-09T17:57:33.470 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.470 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e33af57:::benchmark_data_smithi179_148205_object74814:head# 2024-02-09T17:57:33.470 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.471 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e3413b2:::benchmark_data_smithi179_148205_object22082:head# 2024-02-09T17:57:33.471 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.471 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e34cc9d:::benchmark_data_smithi179_148205_object59663:head# 2024-02-09T17:57:33.471 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.471 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e34ed72:::benchmark_data_smithi179_148205_object42067:head# 2024-02-09T17:57:33.471 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.471 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e350e9b:::benchmark_data_smithi179_148205_object87394:head# 2024-02-09T17:57:33.471 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.471 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e362671:::benchmark_data_smithi179_148205_object37536:head# 2024-02-09T17:57:33.471 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.471 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e36927b:::benchmark_data_smithi179_148205_object42415:head# 2024-02-09T17:57:33.471 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.471 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e36e541:::benchmark_data_smithi179_148205_object86676:head# 2024-02-09T17:57:33.471 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.471 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e36e583:::benchmark_data_smithi179_148205_object41022:head# 2024-02-09T17:57:33.471 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.472 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e377fa3:::benchmark_data_smithi179_148205_object18568:head# 2024-02-09T17:57:33.472 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.472 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e379506:::benchmark_data_smithi179_148205_object91904:head# 2024-02-09T17:57:33.472 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.472 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e37e096:::benchmark_data_smithi179_148205_object104059:head# 2024-02-09T17:57:33.472 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.472 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e3848eb:::benchmark_data_smithi179_148205_object23131:head# 2024-02-09T17:57:33.472 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.472 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e385779:::benchmark_data_smithi179_148205_object15426:head# 2024-02-09T17:57:33.472 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.472 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e391fd2:::benchmark_data_smithi179_148205_object26305:head# 2024-02-09T17:57:33.472 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.472 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e3a02fd:::benchmark_data_smithi179_148205_object79272:head# 2024-02-09T17:57:33.472 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.473 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e3a2268:::benchmark_data_smithi179_148205_object52641:head# 2024-02-09T17:57:33.473 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.473 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e3bedcd:::benchmark_data_smithi179_148205_object50667:head# 2024-02-09T17:57:33.473 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.473 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e3bf446:::benchmark_data_smithi179_148205_object45991:head# 2024-02-09T17:57:33.473 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.473 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e3c3039:::benchmark_data_smithi179_148205_object51724:head# 2024-02-09T17:57:33.473 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.473 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e400ebe:::benchmark_data_smithi179_148205_object108326:head# 2024-02-09T17:57:33.473 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.473 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e4027b6:::benchmark_data_smithi179_148205_object50668:head# 2024-02-09T17:57:33.473 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.473 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e40daa3:::benchmark_data_smithi179_148205_object20952:head# 2024-02-09T17:57:33.473 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.474 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e40e2c0:::benchmark_data_smithi179_148205_object50354:head# 2024-02-09T17:57:33.474 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.474 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e416f50:::benchmark_data_smithi179_148205_object57983:head# 2024-02-09T17:57:33.474 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.474 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e4202ff:::benchmark_data_smithi179_148205_object16924:head# 2024-02-09T17:57:33.474 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.474 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e424ee1:::benchmark_data_smithi179_148205_object31863:head# 2024-02-09T17:57:33.474 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.474 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e434a34:::benchmark_data_smithi179_148205_object93352:head# 2024-02-09T17:57:33.474 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.474 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e43c003:::benchmark_data_smithi179_148205_object29776:head# 2024-02-09T17:57:33.474 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.474 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e443502:::benchmark_data_smithi179_148205_object105678:head# 2024-02-09T17:57:33.474 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.474 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e444c6a:::benchmark_data_smithi179_148205_object70566:head# 2024-02-09T17:57:33.475 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.475 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e44f23b:::benchmark_data_smithi179_148205_object90153:head# 2024-02-09T17:57:33.475 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.475 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e457174:::benchmark_data_smithi179_148205_object104076:head# 2024-02-09T17:57:33.475 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.475 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e45b848:::benchmark_data_smithi179_148205_object68284:head# 2024-02-09T17:57:33.475 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.475 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e48630c:::benchmark_data_smithi179_148205_object94955:head# 2024-02-09T17:57:33.475 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.475 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e487094:::benchmark_data_smithi179_148205_object48301:head# 2024-02-09T17:57:33.475 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.475 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e48a7d6:::benchmark_data_smithi179_148205_object27026:head# 2024-02-09T17:57:33.475 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.475 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e497f7b:::benchmark_data_smithi179_148205_object36813:head# 2024-02-09T17:57:33.475 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.475 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e4a82a7:::benchmark_data_smithi179_148205_object91962:head# 2024-02-09T17:57:33.476 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.476 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e4d7664:::benchmark_data_smithi179_148205_object18566:head# 2024-02-09T17:57:33.476 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.476 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e4f081b:::benchmark_data_smithi179_148205_object75851:head# 2024-02-09T17:57:33.476 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.476 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e4f1423:::benchmark_data_smithi179_148205_object29151:head# 2024-02-09T17:57:33.476 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.476 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e4f26b1:::benchmark_data_smithi179_148205_object3266:head# 2024-02-09T17:57:33.476 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.491 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e4f77b5:::benchmark_data_smithi179_148205_object27608:head# 2024-02-09T17:57:33.491 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.491 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e4f8062:::benchmark_data_smithi179_148205_object107396:head# 2024-02-09T17:57:33.491 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.491 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e4fec44:::benchmark_data_smithi179_148205_object25067:head# 2024-02-09T17:57:33.491 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.491 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e4fed89:::benchmark_data_smithi179_148205_object69982:head# 2024-02-09T17:57:33.491 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.491 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e503423:::benchmark_data_smithi179_148205_object54543:head# 2024-02-09T17:57:33.491 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.491 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e506801:::benchmark_data_smithi179_148205_object108371:head# 2024-02-09T17:57:33.492 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.492 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e50eb3c:::benchmark_data_smithi179_148205_object50582:head# 2024-02-09T17:57:33.492 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.492 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e512788:::benchmark_data_smithi179_148205_object108128:head# 2024-02-09T17:57:33.492 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.492 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e51a39d:::benchmark_data_smithi179_148205_object106577:head# 2024-02-09T17:57:33.492 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.492 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e5244c9:::benchmark_data_smithi179_148205_object11315:head# 2024-02-09T17:57:33.492 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.492 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e5297ac:::benchmark_data_smithi179_148205_object3088:head# 2024-02-09T17:57:33.492 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.492 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e5321c9:::benchmark_data_smithi179_148205_object78182:head# 2024-02-09T17:57:33.492 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.492 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e53977a:::benchmark_data_smithi179_148205_object109023:head# 2024-02-09T17:57:33.493 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.493 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e53b0f9:::benchmark_data_smithi179_148205_object74715:head# 2024-02-09T17:57:33.493 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.493 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e548127:::benchmark_data_smithi179_148205_object10170:head# 2024-02-09T17:57:33.493 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.493 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e55e11e:::benchmark_data_smithi179_148205_object77623:head# 2024-02-09T17:57:33.493 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.493 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e567064:::benchmark_data_smithi179_148205_object70393:head# 2024-02-09T17:57:33.493 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.493 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e5674fc:::benchmark_data_smithi179_148205_object94401:head# 2024-02-09T17:57:33.493 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.493 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e56b975:::benchmark_data_smithi179_148205_object36405:head# 2024-02-09T17:57:33.493 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.493 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e576624:::benchmark_data_smithi179_148205_object24020:head# 2024-02-09T17:57:33.493 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.494 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e57da6f:::benchmark_data_smithi179_148205_object87739:head# 2024-02-09T17:57:33.494 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.494 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e57f012:::benchmark_data_smithi179_148205_object3858:head# 2024-02-09T17:57:33.494 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.494 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e583720:::benchmark_data_smithi179_148205_object91423:head# 2024-02-09T17:57:33.494 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.494 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e58b90c:::benchmark_data_smithi179_148205_object10430:head# 2024-02-09T17:57:33.494 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.494 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e58fe2e:::benchmark_data_smithi179_148205_object85551:head# 2024-02-09T17:57:33.494 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.494 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e5b0371:::benchmark_data_smithi179_148205_object28070:head# 2024-02-09T17:57:33.494 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.494 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e5b430c:::benchmark_data_smithi179_148205_object19121:head# 2024-02-09T17:57:33.494 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.495 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e5b8502:::benchmark_data_smithi179_148205_object34657:head# 2024-02-09T17:57:33.495 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.495 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e5bf3e9:::benchmark_data_smithi179_148205_object10166:head# 2024-02-09T17:57:33.495 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.495 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e5de9af:::benchmark_data_smithi179_148205_object49125:head# 2024-02-09T17:57:33.495 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.495 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e5ed86b:::benchmark_data_smithi179_148205_object37863:head# 2024-02-09T17:57:33.495 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.495 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e603a55:::benchmark_data_smithi179_148205_object4460:head# 2024-02-09T17:57:33.495 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.495 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e60a91d:::benchmark_data_smithi179_148205_object38233:head# 2024-02-09T17:57:33.495 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.495 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e615f63:::benchmark_data_smithi179_148205_object37984:head# 2024-02-09T17:57:33.495 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.496 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e619d8f:::benchmark_data_smithi179_148205_object10247:head# 2024-02-09T17:57:33.496 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.496 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e62a416:::benchmark_data_smithi179_148205_object94202:head# 2024-02-09T17:57:33.496 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.496 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e62cfd9:::benchmark_data_smithi179_148205_object38724:head# 2024-02-09T17:57:33.496 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.496 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e63318e:::benchmark_data_smithi179_148205_object72534:head# 2024-02-09T17:57:33.496 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.496 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e633ea6:::benchmark_data_smithi179_148205_object13074:head# 2024-02-09T17:57:33.496 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.496 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e64ce8a:::benchmark_data_smithi179_148205_object53999:head# 2024-02-09T17:57:33.496 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.496 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e6538c1:::benchmark_data_smithi179_148205_object57147:head# 2024-02-09T17:57:33.496 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.497 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e65e75e:::benchmark_data_smithi179_148205_object89845:head# 2024-02-09T17:57:33.497 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.497 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e679874:::benchmark_data_smithi179_148205_object20503:head# 2024-02-09T17:57:33.497 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.497 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e6906e2:::benchmark_data_smithi179_148205_object76153:head# 2024-02-09T17:57:33.497 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.497 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e6913a4:::benchmark_data_smithi179_148205_object80289:head# 2024-02-09T17:57:33.497 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.497 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e6984fd:::benchmark_data_smithi179_148205_object39913:head# 2024-02-09T17:57:33.497 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.497 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e6a5e99:::benchmark_data_smithi179_148205_object37813:head# 2024-02-09T17:57:33.497 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.497 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e6ac0e5:::benchmark_data_smithi179_148205_object31091:head# 2024-02-09T17:57:33.497 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.498 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e6b7fe2:::benchmark_data_smithi179_148205_object66501:head# 2024-02-09T17:57:33.498 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.498 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e6bb5ad:::benchmark_data_smithi179_148205_object83240:head# 2024-02-09T17:57:33.498 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.498 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e6c993e:::benchmark_data_smithi179_148205_object31286:head# 2024-02-09T17:57:33.498 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.498 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e6d73b2:::benchmark_data_smithi179_148205_object47429:head# 2024-02-09T17:57:33.498 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.498 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e6e4f9b:::benchmark_data_smithi179_148205_object102748:head# 2024-02-09T17:57:33.498 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.498 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e6e6479:::benchmark_data_smithi179_148205_object6206:head# 2024-02-09T17:57:33.498 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.498 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e70480b:::benchmark_data_smithi179_148205_object60584:head# 2024-02-09T17:57:33.498 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.498 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e71f4bb:::benchmark_data_smithi179_148205_object103314:head# 2024-02-09T17:57:33.499 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.499 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e7225fe:::benchmark_data_smithi179_148205_object48799:head# 2024-02-09T17:57:33.499 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.499 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e72fb92:::benchmark_data_smithi179_148205_object87506:head# 2024-02-09T17:57:33.499 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.499 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e72fe62:::benchmark_data_smithi179_148205_object43165:head# 2024-02-09T17:57:33.499 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.499 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e732cf4:::benchmark_data_smithi179_148205_object60619:head# 2024-02-09T17:57:33.499 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.499 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e73830d:::benchmark_data_smithi179_148205_object21199:head# 2024-02-09T17:57:33.499 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.499 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e73f1a1:::benchmark_data_smithi179_148205_object56761:head# 2024-02-09T17:57:33.499 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.499 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e74b569:::benchmark_data_smithi179_148205_object41088:head# 2024-02-09T17:57:33.499 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.500 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e7579f1:::benchmark_data_smithi179_148205_object82085:head# 2024-02-09T17:57:33.500 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.500 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e769199:::benchmark_data_smithi179_148205_object17134:head# 2024-02-09T17:57:33.500 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.500 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e779259:::benchmark_data_smithi179_148205_object27448:head# 2024-02-09T17:57:33.500 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.500 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e788894:::benchmark_data_smithi179_148205_object105242:head# 2024-02-09T17:57:33.500 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.500 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e79ee31:::benchmark_data_smithi179_148205_object16819:head# 2024-02-09T17:57:33.500 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.500 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e7a304b:::benchmark_data_smithi179_148205_object18381:head# 2024-02-09T17:57:33.500 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.500 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e7a8eee:::benchmark_data_smithi179_148205_object106861:head# 2024-02-09T17:57:33.501 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.501 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e7aaa5c:::benchmark_data_smithi179_148205_object87333:head# 2024-02-09T17:57:33.501 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.501 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e7cc635:::benchmark_data_smithi179_148205_object84767:head# 2024-02-09T17:57:33.501 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.501 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e7d68cb:::benchmark_data_smithi179_148205_object53163:head# 2024-02-09T17:57:33.501 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.501 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e7df707:::benchmark_data_smithi179_148205_object95777:head# 2024-02-09T17:57:33.501 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.501 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e7ecb6c:::benchmark_data_smithi179_148205_object15126:head# 2024-02-09T17:57:33.501 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.501 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e7fb9d4:::benchmark_data_smithi179_148205_object33831:head# 2024-02-09T17:57:33.501 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.501 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e7fdf68:::benchmark_data_smithi179_148205_object41830:head# 2024-02-09T17:57:33.502 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.502 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e801cc8:::benchmark_data_smithi179_148205_object12079:head# 2024-02-09T17:57:33.502 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.502 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e803ae3:::benchmark_data_smithi179_148205_object24142:head# 2024-02-09T17:57:33.502 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.502 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e805c79:::benchmark_data_smithi179_148205_object33907:head# 2024-02-09T17:57:33.502 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.502 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e80e83c:::benchmark_data_smithi179_148205_object95764:head# 2024-02-09T17:57:33.502 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.502 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e817a86:::benchmark_data_smithi179_148205_object78105:head# 2024-02-09T17:57:33.502 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.502 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e8421da:::benchmark_data_smithi179_148205_object84049:head# 2024-02-09T17:57:33.502 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.502 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e844e13:::benchmark_data_smithi179_148205_object72179:head# 2024-02-09T17:57:33.502 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.502 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e866510:::benchmark_data_smithi179_148205_object74407:head# 2024-02-09T17:57:33.503 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.503 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e86b199:::benchmark_data_smithi179_148205_object18124:head# 2024-02-09T17:57:33.503 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.503 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e880a64:::benchmark_data_smithi179_148205_object77360:head# 2024-02-09T17:57:33.503 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.503 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e880d61:::benchmark_data_smithi179_148205_object48233:head# 2024-02-09T17:57:33.503 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.503 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e884298:::benchmark_data_smithi179_148205_object44076:head# 2024-02-09T17:57:33.503 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.503 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e894dfc:::benchmark_data_smithi179_148205_object19142:head# 2024-02-09T17:57:33.503 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.503 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e8a0548:::benchmark_data_smithi179_148205_object102813:head# 2024-02-09T17:57:33.503 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.503 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e8a13bb:::benchmark_data_smithi179_148205_object28001:head# 2024-02-09T17:57:33.504 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.504 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e8b62e1:::benchmark_data_smithi179_148205_object107353:head# 2024-02-09T17:57:33.504 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.504 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e8b91f1:::benchmark_data_smithi179_148205_object93131:head# 2024-02-09T17:57:33.504 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.504 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e8bb19d:::benchmark_data_smithi179_148205_object58047:head# 2024-02-09T17:57:33.504 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.504 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e8ce784:::benchmark_data_smithi179_148205_object79538:head# 2024-02-09T17:57:33.504 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.504 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e8db18f:::benchmark_data_smithi179_148205_object79707:head# 2024-02-09T17:57:33.504 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.504 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e8db5fd:::benchmark_data_smithi179_148205_object63100:head# 2024-02-09T17:57:33.504 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.504 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e8dcaff:::benchmark_data_smithi179_148205_object46938:head# 2024-02-09T17:57:33.505 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.505 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e8df989:::benchmark_data_smithi179_148205_object19886:head# 2024-02-09T17:57:33.505 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.505 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e8e0452:::benchmark_data_smithi179_148205_object6482:head# 2024-02-09T17:57:33.505 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.505 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e8e1a4b:::benchmark_data_smithi179_148205_object55858:head# 2024-02-09T17:57:33.505 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.541 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e8e52ab:::benchmark_data_smithi179_148205_object64259:head# 2024-02-09T17:57:33.541 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.541 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e8f7990:::benchmark_data_smithi179_148205_object24250:head# 2024-02-09T17:57:33.541 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.541 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e8f8730:::benchmark_data_smithi179_148205_object41132:head# 2024-02-09T17:57:33.542 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.542 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e8ff804:::benchmark_data_smithi179_148205_object70999:head# 2024-02-09T17:57:33.542 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.542 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e8ff8db:::benchmark_data_smithi179_148205_object74474:head# 2024-02-09T17:57:33.542 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.542 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e908935:::benchmark_data_smithi179_148205_object43210:head# 2024-02-09T17:57:33.542 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.542 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e90e2c1:::benchmark_data_smithi179_148205_object103935:head# 2024-02-09T17:57:33.542 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.542 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e91288d:::benchmark_data_smithi179_148205_object90477:head# 2024-02-09T17:57:33.542 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.542 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e91b887:::benchmark_data_smithi179_148205_object107445:head# 2024-02-09T17:57:33.542 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.542 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e9238ad:::benchmark_data_smithi179_148205_object44835:head# 2024-02-09T17:57:33.543 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.543 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e934e48:::benchmark_data_smithi179_148205_object57850:head# 2024-02-09T17:57:33.543 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.543 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e9362a3:::benchmark_data_smithi179_148205_object83949:head# 2024-02-09T17:57:33.543 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.543 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e937328:::benchmark_data_smithi179_148205_object10171:head# 2024-02-09T17:57:33.543 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.543 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e949f6b:::benchmark_data_smithi179_148205_object72793:head# 2024-02-09T17:57:33.543 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.543 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e952eff:::benchmark_data_smithi179_148205_object32046:head# 2024-02-09T17:57:33.543 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.543 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e9550af:::benchmark_data_smithi179_148205_object23541:head# 2024-02-09T17:57:33.543 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.543 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e967859:::benchmark_data_smithi179_148205_object102835:head# 2024-02-09T17:57:33.544 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.544 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e9689c1:::benchmark_data_smithi179_148205_object15109:head# 2024-02-09T17:57:33.544 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.544 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e96c6a3:::benchmark_data_smithi179_148205_object75076:head# 2024-02-09T17:57:33.544 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.544 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e96d9f4:::benchmark_data_smithi179_148205_object105885:head# 2024-02-09T17:57:33.544 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.544 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e9787df:::benchmark_data_smithi179_148205_object100572:head# 2024-02-09T17:57:33.544 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.544 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e97d509:::benchmark_data_smithi179_148205_object70175:head# 2024-02-09T17:57:33.544 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.544 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e9820a4:::benchmark_data_smithi179_148205_object108576:head# 2024-02-09T17:57:33.544 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.544 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e99266b:::benchmark_data_smithi179_148205_object85315:head# 2024-02-09T17:57:33.545 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.545 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e9994eb:::benchmark_data_smithi179_148205_object41766:head# 2024-02-09T17:57:33.545 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.545 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e99aca7:::benchmark_data_smithi179_148205_object94423:head# 2024-02-09T17:57:33.545 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.545 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e9b6295:::benchmark_data_smithi179_148205_object94617:head# 2024-02-09T17:57:33.545 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.545 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e9b8725:::benchmark_data_smithi179_148205_object2129:head# 2024-02-09T17:57:33.545 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.545 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e9be4c9:::benchmark_data_smithi179_148205_object78847:head# 2024-02-09T17:57:33.545 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.545 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e9c0eff:::benchmark_data_smithi179_148205_object16455:head# 2024-02-09T17:57:33.545 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.546 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e9c5e49:::benchmark_data_smithi179_148205_object64903:head# 2024-02-09T17:57:33.546 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.546 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e9c907e:::benchmark_data_smithi179_148205_object87612:head# 2024-02-09T17:57:33.546 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.546 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e9e7c3f:::benchmark_data_smithi179_148205_object103561:head# 2024-02-09T17:57:33.546 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.546 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e9e9772:::benchmark_data_smithi179_148205_object73646:head# 2024-02-09T17:57:33.546 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.546 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e9eebf6:::benchmark_data_smithi179_148205_object594:head# 2024-02-09T17:57:33.546 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.546 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e9f0b85:::benchmark_data_smithi179_148205_object18519:head# 2024-02-09T17:57:33.546 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.546 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1e9f3347:::benchmark_data_smithi179_148205_object70991:head# 2024-02-09T17:57:33.546 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.546 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ea01446:::benchmark_data_smithi179_148205_object61058:head# 2024-02-09T17:57:33.547 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.547 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ea04e5d:::benchmark_data_smithi179_148205_object5235:head# 2024-02-09T17:57:33.547 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.547 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ea10762:::benchmark_data_smithi179_148205_object84480:head# 2024-02-09T17:57:33.547 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.547 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ea1811d:::benchmark_data_smithi179_148205_object67239:head# 2024-02-09T17:57:33.547 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.547 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ea1a148:::benchmark_data_smithi179_148205_object17541:head# 2024-02-09T17:57:33.547 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.547 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ea1f2ff:::benchmark_data_smithi179_148205_object20348:head# 2024-02-09T17:57:33.547 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.547 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ea256f4:::benchmark_data_smithi179_148205_object59419:head# 2024-02-09T17:57:33.547 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.547 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ea2bb23:::benchmark_data_smithi179_148205_object99990:head# 2024-02-09T17:57:33.547 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.548 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ea3359c:::benchmark_data_smithi179_148205_object95890:head# 2024-02-09T17:57:33.548 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.548 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ea37551:::benchmark_data_smithi179_148205_object88074:head# 2024-02-09T17:57:33.548 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.548 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ea4112d:::benchmark_data_smithi179_148205_object69644:head# 2024-02-09T17:57:33.548 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.548 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ea41a1f:::benchmark_data_smithi179_148205_object71724:head# 2024-02-09T17:57:33.548 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.548 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ea4895d:::benchmark_data_smithi179_148205_object38304:head# 2024-02-09T17:57:33.548 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.548 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ea49576:::benchmark_data_smithi179_148205_object65042:head# 2024-02-09T17:57:33.548 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.548 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ea4bd51:::benchmark_data_smithi179_148205_object51472:head# 2024-02-09T17:57:33.549 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.549 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ea6fb3a:::benchmark_data_smithi179_148205_object48969:head# 2024-02-09T17:57:33.549 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.549 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ea7391b:::benchmark_data_smithi179_148205_object30659:head# 2024-02-09T17:57:33.549 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.549 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ea78feb:::benchmark_data_smithi179_148205_object89706:head# 2024-02-09T17:57:33.549 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.549 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ea7c4c9:::benchmark_data_smithi179_148205_object72461:head# 2024-02-09T17:57:33.549 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.549 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ea8d46e:::benchmark_data_smithi179_148205_object1655:head# 2024-02-09T17:57:33.549 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.549 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ea94494:::benchmark_data_smithi179_148205_object88797:head# 2024-02-09T17:57:33.549 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.550 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ea9abf9:::benchmark_data_smithi179_148205_object33279:head# 2024-02-09T17:57:33.550 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.550 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ea9d438:::benchmark_data_smithi179_148205_object32801:head# 2024-02-09T17:57:33.550 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.550 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1eabff0e:::benchmark_data_smithi179_148205_object63437:head# 2024-02-09T17:57:33.550 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.550 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1eac1663:::benchmark_data_smithi179_148205_object46449:head# 2024-02-09T17:57:33.550 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.550 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1eac5860:::benchmark_data_smithi179_148205_object108198:head# 2024-02-09T17:57:33.550 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.550 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1eade22f:::benchmark_data_smithi179_148205_object8600:head# 2024-02-09T17:57:33.550 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.550 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1eae0238:::benchmark_data_smithi179_148205_object96162:head# 2024-02-09T17:57:33.550 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.551 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1eae40f3:::benchmark_data_smithi179_148205_object65153:head# 2024-02-09T17:57:33.551 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.551 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1eaea6ce:::benchmark_data_smithi179_148205_object95331:head# 2024-02-09T17:57:33.551 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.551 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1eaebca0:::benchmark_data_smithi179_148205_object75287:head# 2024-02-09T17:57:33.551 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.551 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1eb0081f:::benchmark_data_smithi179_148205_object76345:head# 2024-02-09T17:57:33.551 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.551 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1eb08155:::benchmark_data_smithi179_148205_object66590:head# 2024-02-09T17:57:33.551 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.551 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1eb10a0c:::benchmark_data_smithi179_148205_object45481:head# 2024-02-09T17:57:33.551 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.551 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1eb1e1c3:::benchmark_data_smithi179_148205_object97396:head# 2024-02-09T17:57:33.551 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.551 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1eb208f9:::benchmark_data_smithi179_148205_object94224:head# 2024-02-09T17:57:33.552 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.552 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1eb213bf:::benchmark_data_smithi179_148205_object58758:head# 2024-02-09T17:57:33.552 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.552 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1eb3f223:::benchmark_data_smithi179_148205_object105374:head# 2024-02-09T17:57:33.552 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.552 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1eb3f88f:::benchmark_data_smithi179_148205_object21833:head# 2024-02-09T17:57:33.552 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.552 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1eb4a18c:::benchmark_data_smithi179_148205_object40504:head# 2024-02-09T17:57:33.552 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.552 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1eb4b289:::benchmark_data_smithi179_148205_object1391:head# 2024-02-09T17:57:33.552 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.552 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1eb4d4ac:::benchmark_data_smithi179_148205_object10875:head# 2024-02-09T17:57:33.552 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.552 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1eb58902:::benchmark_data_smithi179_148205_object24828:head# 2024-02-09T17:57:33.553 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.553 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1eb704cc:::benchmark_data_smithi179_148205_object92606:head# 2024-02-09T17:57:33.553 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.553 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1eb7fef0:::benchmark_data_smithi179_148205_object86573:head# 2024-02-09T17:57:33.553 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.553 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1eb8cbe9:::benchmark_data_smithi179_148205_object18512:head# 2024-02-09T17:57:33.553 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.553 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1eb9322c:::benchmark_data_smithi179_148205_object92283:head# 2024-02-09T17:57:33.553 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.553 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1eb9adb9:::benchmark_data_smithi179_148205_object2005:head# 2024-02-09T17:57:33.553 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.553 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1eba6760:::benchmark_data_smithi179_148205_object4114:head# 2024-02-09T17:57:33.553 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.553 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1eba7d41:::benchmark_data_smithi179_148205_object17522:head# 2024-02-09T17:57:33.554 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.554 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1eba96d1:::benchmark_data_smithi179_148205_object75939:head# 2024-02-09T17:57:33.554 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.554 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ebabd9b:::benchmark_data_smithi179_148205_object46931:head# 2024-02-09T17:57:33.554 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.554 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ebacadd:::benchmark_data_smithi179_148205_object39057:head# 2024-02-09T17:57:33.554 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.554 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ebb73e4:::benchmark_data_smithi179_148205_object25592:head# 2024-02-09T17:57:33.554 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.554 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ebb7b36:::benchmark_data_smithi179_148205_object24092:head# 2024-02-09T17:57:33.554 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.554 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ebca771:::benchmark_data_smithi179_148205_object79589:head# 2024-02-09T17:57:33.554 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.554 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ebcbb25:::benchmark_data_smithi179_148205_object15902:head# 2024-02-09T17:57:33.554 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.554 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ebda2b5:::benchmark_data_smithi179_148205_object35932:head# 2024-02-09T17:57:33.555 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.555 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ebe7745:::benchmark_data_smithi179_148205_object25197:head# 2024-02-09T17:57:33.555 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.555 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ebfc1d4:::benchmark_data_smithi179_148205_object55302:head# 2024-02-09T17:57:33.555 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.555 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ec083de:::benchmark_data_smithi179_148205_object108351:head# 2024-02-09T17:57:33.555 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.555 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ec0d941:::benchmark_data_smithi179_148205_object92256:head# 2024-02-09T17:57:33.555 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.555 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ec0e382:::benchmark_data_smithi179_148205_object31700:head# 2024-02-09T17:57:33.555 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.555 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ec1175b:::benchmark_data_smithi179_148205_object69216:head# 2024-02-09T17:57:33.555 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.555 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ec2f9ac:::benchmark_data_smithi179_148205_object19083:head# 2024-02-09T17:57:33.555 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.556 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ec34da1:::benchmark_data_smithi179_148205_object8636:head# 2024-02-09T17:57:33.556 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.556 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ec53537:::benchmark_data_smithi179_148205_object97705:head# 2024-02-09T17:57:33.556 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.556 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ec57795:::benchmark_data_smithi179_148205_object102063:head# 2024-02-09T17:57:33.556 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.556 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ec58d42:::benchmark_data_smithi179_148205_object64849:head# 2024-02-09T17:57:33.556 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.556 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ec59d60:::benchmark_data_smithi179_148205_object39993:head# 2024-02-09T17:57:33.556 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.556 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ec5da0d:::benchmark_data_smithi179_148205_object26887:head# 2024-02-09T17:57:33.556 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.556 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ec650ec:::benchmark_data_smithi179_148205_object12813:head# 2024-02-09T17:57:33.557 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.557 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ec6513c:::benchmark_data_smithi179_148205_object26985:head# 2024-02-09T17:57:33.557 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.557 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ec6cd29:::benchmark_data_smithi179_148205_object9205:head# 2024-02-09T17:57:33.557 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.557 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ec76f34:::benchmark_data_smithi179_148205_object38284:head# 2024-02-09T17:57:33.557 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.557 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ec77f10:::benchmark_data_smithi179_148205_object68501:head# 2024-02-09T17:57:33.557 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.557 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ec7f2bf:::benchmark_data_smithi179_148205_object107527:head# 2024-02-09T17:57:33.557 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.557 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ec84d64:::benchmark_data_smithi179_148205_object58429:head# 2024-02-09T17:57:33.558 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.558 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ec8a491:::benchmark_data_smithi179_148205_object18089:head# 2024-02-09T17:57:33.558 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.558 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ec8d40a:::benchmark_data_smithi179_148205_object90594:head# 2024-02-09T17:57:33.558 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.558 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ec9578a:::benchmark_data_smithi179_148205_object79130:head# 2024-02-09T17:57:33.558 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.558 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ec9d84b:::benchmark_data_smithi179_148205_object58258:head# 2024-02-09T17:57:33.558 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.558 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1eca226b:::benchmark_data_smithi179_148205_object2406:head# 2024-02-09T17:57:33.558 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.558 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1eca48f8:::benchmark_data_smithi179_148205_object56749:head# 2024-02-09T17:57:33.559 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.559 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ecad3a3:::benchmark_data_smithi179_148205_object62847:head# 2024-02-09T17:57:33.559 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.559 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ecb20f2:::benchmark_data_smithi179_148205_object24879:head# 2024-02-09T17:57:33.559 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.559 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ecb30ca:::benchmark_data_smithi179_148205_object7784:head# 2024-02-09T17:57:33.559 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.559 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ecc28a8:::benchmark_data_smithi179_148205_object14447:head# 2024-02-09T17:57:33.559 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.559 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ecd1184:::benchmark_data_smithi179_148205_object82274:head# 2024-02-09T17:57:33.559 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.559 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ecd2de1:::benchmark_data_smithi179_148205_object2035:head# 2024-02-09T17:57:33.559 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.560 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ecd74f7:::benchmark_data_smithi179_148205_object14427:head# 2024-02-09T17:57:33.560 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.560 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ece0ed8:::benchmark_data_smithi179_148205_object71905:head# 2024-02-09T17:57:33.560 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.560 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ed0289f:::benchmark_data_smithi179_148205_object42169:head# 2024-02-09T17:57:33.560 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.560 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ed17b05:::benchmark_data_smithi179_148205_object59926:head# 2024-02-09T17:57:33.560 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.560 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ed1e555:::benchmark_data_smithi179_148205_object885:head# 2024-02-09T17:57:33.560 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.560 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ed208f0:::benchmark_data_smithi179_148205_object22140:head# 2024-02-09T17:57:33.560 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.560 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ed220a4:::benchmark_data_smithi179_148205_object91634:head# 2024-02-09T17:57:33.560 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.561 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ed23710:::benchmark_data_smithi179_148205_object91995:head# 2024-02-09T17:57:33.561 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.561 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ed25b60:::benchmark_data_smithi179_148205_object56568:head# 2024-02-09T17:57:33.561 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.561 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ed2a13e:::benchmark_data_smithi179_148205_object64086:head# 2024-02-09T17:57:33.561 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.561 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ed34161:::benchmark_data_smithi179_148205_object64714:head# 2024-02-09T17:57:33.561 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.561 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ed3ca67:::benchmark_data_smithi179_148205_object21695:head# 2024-02-09T17:57:33.561 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.561 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ed42f3c:::benchmark_data_smithi179_148205_object14290:head# 2024-02-09T17:57:33.561 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.561 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ed44378:::benchmark_data_smithi179_148205_object62687:head# 2024-02-09T17:57:33.562 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.562 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ed4c105:::benchmark_data_smithi179_148205_object65063:head# 2024-02-09T17:57:33.562 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.562 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ed54ee0:::benchmark_data_smithi179_148205_object93744:head# 2024-02-09T17:57:33.562 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.562 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ed5e8fa:::benchmark_data_smithi179_148205_object59735:head# 2024-02-09T17:57:33.562 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.562 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ed64436:::benchmark_data_smithi179_148205_object32728:head# 2024-02-09T17:57:33.562 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.562 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ed763cf:::benchmark_data_smithi179_148205_object89425:head# 2024-02-09T17:57:33.562 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.562 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ed7a46f:::benchmark_data_smithi179_148205_object35373:head# 2024-02-09T17:57:33.562 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.562 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ed8308c:::benchmark_data_smithi179_148205_object85425:head# 2024-02-09T17:57:33.563 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.563 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ed935f2:::benchmark_data_smithi179_148205_object81117:head# 2024-02-09T17:57:33.563 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.563 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1eda17f2:::benchmark_data_smithi179_148205_object91899:head# 2024-02-09T17:57:33.563 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.563 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1eda5c85:::benchmark_data_smithi179_148205_object53701:head# 2024-02-09T17:57:33.563 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.563 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1edaa214:::benchmark_data_smithi179_148205_object76868:head# 2024-02-09T17:57:33.563 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.564 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1edaa360:::benchmark_data_smithi179_148205_object7843:head# 2024-02-09T17:57:33.564 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.564 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1edb4379:::benchmark_data_smithi179_148205_object33586:head# 2024-02-09T17:57:33.564 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.564 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1edc51db:::benchmark_data_smithi179_148205_object83904:head# 2024-02-09T17:57:33.564 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.564 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1edd0044:::benchmark_data_smithi179_148205_object14344:head# 2024-02-09T17:57:33.564 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.564 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1edd374d:::benchmark_data_smithi179_148205_object108138:head# 2024-02-09T17:57:33.564 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.564 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1edd7ec5:::benchmark_data_smithi179_148205_object65576:head# 2024-02-09T17:57:33.564 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.564 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1edda94f:::benchmark_data_smithi179_148205_object9275:head# 2024-02-09T17:57:33.565 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.565 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1edeb23b:::benchmark_data_smithi179_148205_object89910:head# 2024-02-09T17:57:33.565 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.565 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ee04639:::benchmark_data_smithi179_148205_object102460:head# 2024-02-09T17:57:33.565 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.565 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ee0a28a:::benchmark_data_smithi179_148205_object30844:head# 2024-02-09T17:57:33.565 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.565 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ee1d4df:::benchmark_data_smithi179_148205_object29466:head# 2024-02-09T17:57:33.565 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.565 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ee2676e:::benchmark_data_smithi179_148205_object13053:head# 2024-02-09T17:57:33.565 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.565 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ee294d8:::benchmark_data_smithi179_148205_object71320:head# 2024-02-09T17:57:33.565 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.566 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ee30c38:::benchmark_data_smithi179_148205_object40765:head# 2024-02-09T17:57:33.566 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.566 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ee35d6d:::benchmark_data_smithi179_148205_object1491:head# 2024-02-09T17:57:33.566 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.566 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ee381cf:::benchmark_data_smithi179_148205_object19850:head# 2024-02-09T17:57:33.566 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.566 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ee3f399:::benchmark_data_smithi179_148205_object24608:head# 2024-02-09T17:57:33.566 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.566 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ee61857:::benchmark_data_smithi179_148205_object4500:head# 2024-02-09T17:57:33.566 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.566 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ee6364d:::benchmark_data_smithi179_148205_object100225:head# 2024-02-09T17:57:33.566 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.566 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ee657fa:::benchmark_data_smithi179_148205_object52892:head# 2024-02-09T17:57:33.567 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.567 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ee6f9cc:::benchmark_data_smithi179_148205_object80659:head# 2024-02-09T17:57:33.567 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.567 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ee97e30:::benchmark_data_smithi179_148205_object90793:head# 2024-02-09T17:57:33.567 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.567 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ee994c4:::benchmark_data_smithi179_148205_object106995:head# 2024-02-09T17:57:33.567 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.567 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1eeaa37c:::benchmark_data_smithi179_148205_object35313:head# 2024-02-09T17:57:33.567 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.567 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1eebaef5:::benchmark_data_smithi179_148205_object41448:head# 2024-02-09T17:57:33.567 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.567 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1eebb5ec:::benchmark_data_smithi179_148205_object66293:head# 2024-02-09T17:57:33.567 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.567 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1eebd916:::benchmark_data_smithi179_148205_object82462:head# 2024-02-09T17:57:33.567 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.568 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1eed917c:::benchmark_data_smithi179_148205_object45959:head# 2024-02-09T17:57:33.568 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.568 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1eed960a:::benchmark_data_smithi179_148205_object36718:head# 2024-02-09T17:57:33.568 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.568 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1eee6b6e:::benchmark_data_smithi179_148205_object62021:head# 2024-02-09T17:57:33.568 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.568 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1eeeda66:::benchmark_data_smithi179_148205_object70935:head# 2024-02-09T17:57:33.568 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.568 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1eeee6d5:::benchmark_data_smithi179_148205_object42366:head# 2024-02-09T17:57:33.568 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.568 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1eef6aed:::benchmark_data_smithi179_148205_object8561:head# 2024-02-09T17:57:33.568 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.568 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1eefd676:::benchmark_data_smithi179_148205_object68972:head# 2024-02-09T17:57:33.568 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.568 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ef0154b:::benchmark_data_smithi179_148205_object20434:head# 2024-02-09T17:57:33.569 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.569 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ef1b55f:::benchmark_data_smithi179_148205_object105159:head# 2024-02-09T17:57:33.569 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.569 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ef329b0:::benchmark_data_smithi179_148205_object56560:head# 2024-02-09T17:57:33.569 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.569 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ef3503a:::benchmark_data_smithi179_148205_object2230:head# 2024-02-09T17:57:33.569 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.569 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ef382f7:::benchmark_data_smithi179_148205_object82239:head# 2024-02-09T17:57:33.569 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.569 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ef40536:::benchmark_data_smithi179_148205_object105160:head# 2024-02-09T17:57:33.569 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.569 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ef42ce3:::benchmark_data_smithi179_148205_object103915:head# 2024-02-09T17:57:33.569 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.569 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ef5a3f9:::benchmark_data_smithi179_148205_object94547:head# 2024-02-09T17:57:33.570 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.570 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ef6ebc8:::benchmark_data_smithi179_148205_object25626:head# 2024-02-09T17:57:33.570 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.570 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ef78ecf:::benchmark_data_smithi179_148205_object50736:head# 2024-02-09T17:57:33.570 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.570 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ef7dd7e:::benchmark_data_smithi179_148205_object57072:head# 2024-02-09T17:57:33.570 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.570 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ef95172:::benchmark_data_smithi179_148205_object85021:head# 2024-02-09T17:57:33.570 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.570 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ef98bee:::benchmark_data_smithi179_148205_object32299:head# 2024-02-09T17:57:33.570 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.570 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ef9bd7b:::benchmark_data_smithi179_148205_object99183:head# 2024-02-09T17:57:33.570 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.570 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1efa12d3:::benchmark_data_smithi179_148205_object77727:head# 2024-02-09T17:57:33.570 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.570 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1efb788d:::benchmark_data_smithi179_148205_object26700:head# 2024-02-09T17:57:33.571 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.571 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1efb9d67:::benchmark_data_smithi179_148205_object81477:head# 2024-02-09T17:57:33.571 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.571 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1efbea54:::benchmark_data_smithi179_148205_object69976:head# 2024-02-09T17:57:33.571 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.571 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1efc6e94:::benchmark_data_smithi179_148205_object103944:head# 2024-02-09T17:57:33.571 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.571 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1efd2873:::benchmark_data_smithi179_148205_object43283:head# 2024-02-09T17:57:33.571 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.571 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1efd4581:::benchmark_data_smithi179_148205_object30830:head# 2024-02-09T17:57:33.571 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.571 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1efe63c0:::benchmark_data_smithi179_148205_object18604:head# 2024-02-09T17:57:33.571 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.571 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1efe6ee8:::benchmark_data_smithi179_148205_object13792:head# 2024-02-09T17:57:33.571 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.572 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1efee0d4:::benchmark_data_smithi179_148205_object18211:head# 2024-02-09T17:57:33.572 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.572 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1effe678:::benchmark_data_smithi179_148205_object103441:head# 2024-02-09T17:57:33.572 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.572 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f0006ba:::benchmark_data_smithi179_148205_object28642:head# 2024-02-09T17:57:33.572 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.572 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f008812:::benchmark_data_smithi179_148205_object32769:head# 2024-02-09T17:57:33.572 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.572 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f008f77:::benchmark_data_smithi179_148205_object44439:head# 2024-02-09T17:57:33.572 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.572 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f00c5c9:::benchmark_data_smithi179_148205_object25070:head# 2024-02-09T17:57:33.572 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.572 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f011fcc:::benchmark_data_smithi179_148205_object102351:head# 2024-02-09T17:57:33.572 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.572 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f012992:::benchmark_data_smithi179_148205_object63237:head# 2024-02-09T17:57:33.573 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.573 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f01aa84:::benchmark_data_smithi179_148205_object22455:head# 2024-02-09T17:57:33.573 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.573 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f01ffad:::benchmark_data_smithi179_148205_object7341:head# 2024-02-09T17:57:33.573 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.573 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f02bea7:::benchmark_data_smithi179_148205_object2697:head# 2024-02-09T17:57:33.573 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.573 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f03104b:::benchmark_data_smithi179_148205_object22642:head# 2024-02-09T17:57:33.573 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.573 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f031f25:::benchmark_data_smithi179_148205_object11282:head# 2024-02-09T17:57:33.573 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.573 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f039c8c:::benchmark_data_smithi179_148205_object30149:head# 2024-02-09T17:57:33.573 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.573 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f03f7c9:::benchmark_data_smithi179_148205_object25176:head# 2024-02-09T17:57:33.573 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.574 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f03fad1:::benchmark_data_smithi179_148205_object57140:head# 2024-02-09T17:57:33.574 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.574 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f045315:::benchmark_data_smithi179_148205_object67460:head# 2024-02-09T17:57:33.574 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.574 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f04d4d2:::benchmark_data_smithi179_148205_object25025:head# 2024-02-09T17:57:33.574 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.574 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f053ab7:::benchmark_data_smithi179_148205_object94311:head# 2024-02-09T17:57:33.574 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.574 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f05a515:::benchmark_data_smithi179_148205_object69624:head# 2024-02-09T17:57:33.574 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.574 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f05d733:::benchmark_data_smithi179_148205_object97483:head# 2024-02-09T17:57:33.574 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.574 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f06647a:::benchmark_data_smithi179_148205_object55305:head# 2024-02-09T17:57:33.574 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.574 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f070453:::benchmark_data_smithi179_148205_object96947:head# 2024-02-09T17:57:33.575 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.575 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f0738e7:::benchmark_data_smithi179_148205_object22379:head# 2024-02-09T17:57:33.575 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.575 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f078188:::benchmark_data_smithi179_148205_object44321:head# 2024-02-09T17:57:33.575 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.575 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f0804fb:::benchmark_data_smithi179_148205_object101576:head# 2024-02-09T17:57:33.575 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.575 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f0805d9:::benchmark_data_smithi179_148205_object69831:head# 2024-02-09T17:57:33.575 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.575 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f09bf0a:::benchmark_data_smithi179_148205_object87730:head# 2024-02-09T17:57:33.575 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.575 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:57:33.618 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:33.620 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f09cdd7:::benchmark_data_smithi179_148205_object57916:head# 2024-02-09T17:57:33.620 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.620 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f0a37e3:::benchmark_data_smithi179_148205_object10311:head# 2024-02-09T17:57:33.620 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.620 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f0a4cf0:::benchmark_data_smithi179_148205_object7305:head# 2024-02-09T17:57:33.620 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.620 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f0a4cfc:::benchmark_data_smithi179_148205_object90949:head# 2024-02-09T17:57:33.620 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.620 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f0a8004:::benchmark_data_smithi179_148205_object58844:head# 2024-02-09T17:57:33.620 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.620 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f0aa79a:::benchmark_data_smithi179_148205_object82182:head# 2024-02-09T17:57:33.620 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.621 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f0ab95c:::benchmark_data_smithi179_148205_object77325:head# 2024-02-09T17:57:33.621 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.621 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f0b56fc:::benchmark_data_smithi179_148205_object21672:head# 2024-02-09T17:57:33.621 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.621 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f0b7b68:::benchmark_data_smithi179_148205_object40096:head# 2024-02-09T17:57:33.621 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.621 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f0c301c:::benchmark_data_smithi179_148205_object62436:head# 2024-02-09T17:57:33.621 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.621 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f0cf0ca:::benchmark_data_smithi179_148205_object87713:head# 2024-02-09T17:57:33.621 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.621 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f0d61f9:::benchmark_data_smithi179_148205_object82568:head# 2024-02-09T17:57:33.621 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.621 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f0d8570:::benchmark_data_smithi179_148205_object69663:head# 2024-02-09T17:57:33.622 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.622 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f0d8833:::benchmark_data_smithi179_148205_object2827:head# 2024-02-09T17:57:33.622 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.622 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f0d894c:::benchmark_data_smithi179_148205_object50263:head# 2024-02-09T17:57:33.622 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.622 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f0e143d:::benchmark_data_smithi179_148205_object87991:head# 2024-02-09T17:57:33.622 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.622 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f0e2bfc:::benchmark_data_smithi179_148205_object22622:head# 2024-02-09T17:57:33.622 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.622 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f0eb140:::benchmark_data_smithi179_148205_object66068:head# 2024-02-09T17:57:33.622 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.622 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f0fc4a1:::benchmark_data_smithi179_148205_object57207:head# 2024-02-09T17:57:33.622 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.622 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f0fd53b:::benchmark_data_smithi179_148205_object83915:head# 2024-02-09T17:57:33.622 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.623 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f10a9c3:::benchmark_data_smithi179_148205_object36397:head# 2024-02-09T17:57:33.623 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.623 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f118211:::benchmark_data_smithi179_148205_object47486:head# 2024-02-09T17:57:33.623 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.623 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f125c07:::benchmark_data_smithi179_148205_object109220:head# 2024-02-09T17:57:33.623 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.623 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f13811d:::benchmark_data_smithi179_148205_object34486:head# 2024-02-09T17:57:33.623 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.623 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f1476c2:::benchmark_data_smithi179_148205_object107940:head# 2024-02-09T17:57:33.623 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.623 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f167636:::benchmark_data_smithi179_148205_object98552:head# 2024-02-09T17:57:33.623 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.623 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f16eb2b:::benchmark_data_smithi179_148205_object32268:head# 2024-02-09T17:57:33.623 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.623 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f16f43a:::benchmark_data_smithi179_148205_object97350:head# 2024-02-09T17:57:33.624 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.624 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f17093a:::benchmark_data_smithi179_148205_object44448:head# 2024-02-09T17:57:33.624 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.624 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f181d15:::benchmark_data_smithi179_148205_object101877:head# 2024-02-09T17:57:33.624 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.624 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f1919ba:::benchmark_data_smithi179_148205_object107863:head# 2024-02-09T17:57:33.624 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.624 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f1c2c08:::benchmark_data_smithi179_148205_object26006:head# 2024-02-09T17:57:33.624 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.624 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f1d5fc6:::benchmark_data_smithi179_148205_object87857:head# 2024-02-09T17:57:33.624 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.624 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f1e458a:::benchmark_data_smithi179_148205_object101646:head# 2024-02-09T17:57:33.624 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.624 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f1e7887:::benchmark_data_smithi179_148205_object78178:head# 2024-02-09T17:57:33.625 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.625 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f1ee9f6:::benchmark_data_smithi179_148205_object105684:head# 2024-02-09T17:57:33.625 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.625 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f20b3ed:::benchmark_data_smithi179_148205_object99820:head# 2024-02-09T17:57:33.625 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.625 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f21fbe4:::benchmark_data_smithi179_148205_object11318:head# 2024-02-09T17:57:33.625 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.625 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f229fa4:::benchmark_data_smithi179_148205_object53941:head# 2024-02-09T17:57:33.625 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.625 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f230b61:::benchmark_data_smithi179_148205_object59421:head# 2024-02-09T17:57:33.625 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.625 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f232158:::benchmark_data_smithi179_148205_object81076:head# 2024-02-09T17:57:33.625 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.625 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f239762:::benchmark_data_smithi179_148205_object100267:head# 2024-02-09T17:57:33.625 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.626 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f242739:::benchmark_data_smithi179_148205_object61449:head# 2024-02-09T17:57:33.626 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.626 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f257b36:::benchmark_data_smithi179_148205_object66778:head# 2024-02-09T17:57:33.626 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.626 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f2627d0:::benchmark_data_smithi179_148205_object103735:head# 2024-02-09T17:57:33.626 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.626 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f26f8f8:::benchmark_data_smithi179_148205_object9694:head# 2024-02-09T17:57:33.626 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.626 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f27c63c:::benchmark_data_smithi179_148205_object88973:head# 2024-02-09T17:57:33.626 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.626 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f27d139:::benchmark_data_smithi179_148205_object54397:head# 2024-02-09T17:57:33.626 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.626 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f2874ab:::benchmark_data_smithi179_148205_object36335:head# 2024-02-09T17:57:33.626 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.626 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f28a87f:::benchmark_data_smithi179_148205_object85056:head# 2024-02-09T17:57:33.627 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.627 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f291cbb:::benchmark_data_smithi179_148205_object49430:head# 2024-02-09T17:57:33.627 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.627 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f292e3c:::benchmark_data_smithi179_148205_object108398:head# 2024-02-09T17:57:33.627 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.627 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f29cf09:::benchmark_data_smithi179_148205_object57791:head# 2024-02-09T17:57:33.627 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.627 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f29f06b:::benchmark_data_smithi179_148205_object34507:head# 2024-02-09T17:57:33.627 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.627 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f2a38ed:::benchmark_data_smithi179_148205_object64647:head# 2024-02-09T17:57:33.627 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.627 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f2ab987:::benchmark_data_smithi179_148205_object81897:head# 2024-02-09T17:57:33.627 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.627 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f2ef862:::benchmark_data_smithi179_148205_object12383:head# 2024-02-09T17:57:33.628 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.628 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f2f47d1:::benchmark_data_smithi179_148205_object67495:head# 2024-02-09T17:57:33.628 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.628 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f2f7de0:::benchmark_data_smithi179_148205_object104677:head# 2024-02-09T17:57:33.628 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.628 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f2fb749:::benchmark_data_smithi179_148205_object78282:head# 2024-02-09T17:57:33.628 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.628 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f2fba62:::benchmark_data_smithi179_148205_object11254:head# 2024-02-09T17:57:33.628 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.628 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f3033b3:::benchmark_data_smithi179_148205_object101140:head# 2024-02-09T17:57:33.628 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.628 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f315ea7:::benchmark_data_smithi179_148205_object11981:head# 2024-02-09T17:57:33.628 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.628 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f31a190:::benchmark_data_smithi179_148205_object73047:head# 2024-02-09T17:57:33.629 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.629 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f31bda2:::benchmark_data_smithi179_148205_object89055:head# 2024-02-09T17:57:33.629 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.629 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f31f8c1:::benchmark_data_smithi179_148205_object72167:head# 2024-02-09T17:57:33.629 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.629 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f3286f9:::benchmark_data_smithi179_148205_object94640:head# 2024-02-09T17:57:33.629 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.629 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f32fec3:::benchmark_data_smithi179_148205_object103284:head# 2024-02-09T17:57:33.629 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.629 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f33dfb9:::benchmark_data_smithi179_148205_object74723:head# 2024-02-09T17:57:33.629 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.629 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f35076b:::benchmark_data_smithi179_148205_object95026:head# 2024-02-09T17:57:33.629 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.629 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f35a6c5:::benchmark_data_smithi179_148205_object23213:head# 2024-02-09T17:57:33.629 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.630 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f35b80c:::benchmark_data_smithi179_148205_object59053:head# 2024-02-09T17:57:33.630 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.630 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f35ed93:::benchmark_data_smithi179_148205_object63404:head# 2024-02-09T17:57:33.630 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.630 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f3673b8:::benchmark_data_smithi179_148205_object59887:head# 2024-02-09T17:57:33.630 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.630 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f3705c4:::benchmark_data_smithi179_148205_object83820:head# 2024-02-09T17:57:33.630 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.630 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f3734f1:::benchmark_data_smithi179_148205_object98770:head# 2024-02-09T17:57:33.630 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.630 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f37b50b:::benchmark_data_smithi179_148205_object35452:head# 2024-02-09T17:57:33.630 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.630 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f37b6d5:::benchmark_data_smithi179_148205_object69862:head# 2024-02-09T17:57:33.630 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.630 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f3add9b:::benchmark_data_smithi179_148205_object51980:head# 2024-02-09T17:57:33.630 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.631 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f3c3ad5:::benchmark_data_smithi179_148205_object81915:head# 2024-02-09T17:57:33.631 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.631 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f3cbbe2:::benchmark_data_smithi179_148205_object40527:head# 2024-02-09T17:57:33.631 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.631 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f3d0695:::benchmark_data_smithi179_148205_object66411:head# 2024-02-09T17:57:33.631 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.631 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f3d2c40:::benchmark_data_smithi179_148205_object102522:head# 2024-02-09T17:57:33.631 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.631 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f3d57b3:::benchmark_data_smithi179_148205_object79927:head# 2024-02-09T17:57:33.631 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.631 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f3d5e03:::benchmark_data_smithi179_148205_object27112:head# 2024-02-09T17:57:33.631 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.631 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f3e2ac0:::benchmark_data_smithi179_148205_object9501:head# 2024-02-09T17:57:33.631 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.632 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f3e3633:::benchmark_data_smithi179_148205_object104643:head# 2024-02-09T17:57:33.632 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.632 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f3e6049:::benchmark_data_smithi179_148205_object93390:head# 2024-02-09T17:57:33.632 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.632 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f407bab:::benchmark_data_smithi179_148205_object106546:head# 2024-02-09T17:57:33.632 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.632 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f40c9cb:::benchmark_data_smithi179_148205_object66867:head# 2024-02-09T17:57:33.632 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.632 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f40dcc9:::benchmark_data_smithi179_148205_object65406:head# 2024-02-09T17:57:33.632 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.632 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f42e116:::benchmark_data_smithi179_148205_object50026:head# 2024-02-09T17:57:33.632 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.632 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f42ecb2:::benchmark_data_smithi179_148205_object8592:head# 2024-02-09T17:57:33.632 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.632 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f4378ab:::benchmark_data_smithi179_148205_object107732:head# 2024-02-09T17:57:33.633 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.633 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f43b811:::benchmark_data_smithi179_148205_object101352:head# 2024-02-09T17:57:33.633 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.633 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f4418bb:::benchmark_data_smithi179_148205_object12263:head# 2024-02-09T17:57:33.633 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.633 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f44df47:::benchmark_data_smithi179_148205_object17922:head# 2024-02-09T17:57:33.633 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.633 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f451567:::benchmark_data_smithi179_148205_object76458:head# 2024-02-09T17:57:33.633 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.633 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f454298:::benchmark_data_smithi179_148205_object67247:head# 2024-02-09T17:57:33.633 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.633 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f4547bf:::benchmark_data_smithi179_148205_object64078:head# 2024-02-09T17:57:33.633 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.633 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f45d798:::benchmark_data_smithi179_148205_object73420:head# 2024-02-09T17:57:33.633 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.634 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f45eb89:::benchmark_data_smithi179_148205_object69169:head# 2024-02-09T17:57:33.634 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.634 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f4606db:::benchmark_data_smithi179_148205_object96192:head# 2024-02-09T17:57:33.634 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.634 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f4663ae:::benchmark_data_smithi179_148205_object53670:head# 2024-02-09T17:57:33.634 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.634 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f4672f8:::benchmark_data_smithi179_148205_object46522:head# 2024-02-09T17:57:33.634 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.634 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f46a3ac:::benchmark_data_smithi179_148205_object70394:head# 2024-02-09T17:57:33.634 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.634 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f472d63:::benchmark_data_smithi179_148205_object93530:head# 2024-02-09T17:57:33.634 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.634 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f4735f0:::benchmark_data_smithi179_148205_object74835:head# 2024-02-09T17:57:33.634 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.634 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f485040:::benchmark_data_smithi179_148205_object92777:head# 2024-02-09T17:57:33.635 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.635 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f48994e:::benchmark_data_smithi179_148205_object86635:head# 2024-02-09T17:57:33.635 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.635 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f4a232f:::benchmark_data_smithi179_148205_object81465:head# 2024-02-09T17:57:33.635 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.635 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f4a24e4:::benchmark_data_smithi179_148205_object40177:head# 2024-02-09T17:57:33.635 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.635 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f4a79bf:::benchmark_data_smithi179_148205_object56622:head# 2024-02-09T17:57:33.635 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.635 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f4a7a22:::benchmark_data_smithi179_148205_object65954:head# 2024-02-09T17:57:33.635 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.635 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f4b3500:::benchmark_data_smithi179_148205_object31828:head# 2024-02-09T17:57:33.635 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.635 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f4b36ce:::benchmark_data_smithi179_148205_object31775:head# 2024-02-09T17:57:33.636 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.636 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f4be3ce:::benchmark_data_smithi179_148205_object77514:head# 2024-02-09T17:57:33.636 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.636 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f4bf6df:::benchmark_data_smithi179_148205_object27115:head# 2024-02-09T17:57:33.636 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.636 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f4c264b:::benchmark_data_smithi179_148205_object81595:head# 2024-02-09T17:57:33.636 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.636 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f4c7c0b:::benchmark_data_smithi179_148205_object61498:head# 2024-02-09T17:57:33.636 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.636 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f4cdbed:::benchmark_data_smithi179_148205_object88700:head# 2024-02-09T17:57:33.636 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.636 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f4d0b0f:::benchmark_data_smithi179_148205_object85148:head# 2024-02-09T17:57:33.636 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.636 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f4d435a:::benchmark_data_smithi179_148205_object12384:head# 2024-02-09T17:57:33.636 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.637 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f4d9c51:::benchmark_data_smithi179_148205_object2112:head# 2024-02-09T17:57:33.637 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.637 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f4e2a72:::benchmark_data_smithi179_148205_object35594:head# 2024-02-09T17:57:33.637 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.637 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f4ebe1d:::benchmark_data_smithi179_148205_object100248:head# 2024-02-09T17:57:33.637 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.637 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f4f37cd:::benchmark_data_smithi179_148205_object109272:head# 2024-02-09T17:57:33.637 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.637 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f4f72cd:::benchmark_data_smithi179_148205_object30134:head# 2024-02-09T17:57:33.637 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.637 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f505b8f:::benchmark_data_smithi179_148205_object77612:head# 2024-02-09T17:57:33.637 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.637 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f50d820:::benchmark_data_smithi179_148205_object107963:head# 2024-02-09T17:57:33.637 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.637 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f513929:::benchmark_data_smithi179_148205_object76938:head# 2024-02-09T17:57:33.638 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.638 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f5143fc:::benchmark_data_smithi179_148205_object45039:head# 2024-02-09T17:57:33.638 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.638 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f527228:::benchmark_data_smithi179_148205_object42450:head# 2024-02-09T17:57:33.638 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.638 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f5344c3:::benchmark_data_smithi179_148205_object33801:head# 2024-02-09T17:57:33.638 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.638 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f537c70:::benchmark_data_smithi179_148205_object100331:head# 2024-02-09T17:57:33.638 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.638 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f53b57a:::benchmark_data_smithi179_148205_object21145:head# 2024-02-09T17:57:33.638 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.638 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f53efdc:::benchmark_data_smithi179_148205_object63368:head# 2024-02-09T17:57:33.638 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.638 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f540992:::benchmark_data_smithi179_148205_object88627:head# 2024-02-09T17:57:33.639 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.639 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f542764:::benchmark_data_smithi179_148205_object82164:head# 2024-02-09T17:57:33.639 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.639 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f558632:::benchmark_data_smithi179_148205_object21377:head# 2024-02-09T17:57:33.639 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.639 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f5613bf:::benchmark_data_smithi179_148205_object17951:head# 2024-02-09T17:57:33.639 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.639 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f572385:::benchmark_data_smithi179_148205_object43049:head# 2024-02-09T17:57:33.639 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.639 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f575425:::benchmark_data_smithi179_148205_object62662:head# 2024-02-09T17:57:33.639 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.639 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f578504:::benchmark_data_smithi179_148205_object28745:head# 2024-02-09T17:57:33.639 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.639 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f581f63:::benchmark_data_smithi179_148205_object69603:head# 2024-02-09T17:57:33.639 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.640 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f58a69c:::benchmark_data_smithi179_148205_object10716:head# 2024-02-09T17:57:33.640 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.640 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f58aa85:::benchmark_data_smithi179_148205_object83123:head# 2024-02-09T17:57:33.640 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.640 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f59c0b0:::benchmark_data_smithi179_148205_object94158:head# 2024-02-09T17:57:33.640 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.640 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f5a1a12:::benchmark_data_smithi179_148205_object13321:head# 2024-02-09T17:57:33.640 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.640 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f5a9932:::benchmark_data_smithi179_148205_object44992:head# 2024-02-09T17:57:33.640 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.640 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f5b3b06:::benchmark_data_smithi179_148205_object66368:head# 2024-02-09T17:57:33.640 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.640 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f5ce654:::benchmark_data_smithi179_148205_object104742:head# 2024-02-09T17:57:33.640 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.641 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f5d38dc:::benchmark_data_smithi179_148205_object66132:head# 2024-02-09T17:57:33.641 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.641 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f5d6676:::benchmark_data_smithi179_148205_object86857:head# 2024-02-09T17:57:33.641 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.641 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f5e84cc:::benchmark_data_smithi179_148205_object65752:head# 2024-02-09T17:57:33.641 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.641 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f5e89f2:::benchmark_data_smithi179_148205_object85203:head# 2024-02-09T17:57:33.641 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.641 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f5fc005:::benchmark_data_smithi179_148205_object39979:head# 2024-02-09T17:57:33.641 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.641 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f60db5f:::benchmark_data_smithi179_148205_object53085:head# 2024-02-09T17:57:33.641 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.641 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f619634:::benchmark_data_smithi179_148205_object12887:head# 2024-02-09T17:57:33.641 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.641 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f61a50c:::benchmark_data_smithi179_148205_object38163:head# 2024-02-09T17:57:33.642 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.642 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f61ba98:::benchmark_data_smithi179_148205_object103675:head# 2024-02-09T17:57:33.642 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.642 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f62c884:::benchmark_data_smithi179_148205_object105308:head# 2024-02-09T17:57:33.642 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.642 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f631c51:::benchmark_data_smithi179_148205_object103789:head# 2024-02-09T17:57:33.642 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.642 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f636d28:::benchmark_data_smithi179_148205_object88060:head# 2024-02-09T17:57:33.642 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.642 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f63ddc8:::benchmark_data_smithi179_148205_object71164:head# 2024-02-09T17:57:33.642 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.642 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f642b2d:::benchmark_data_smithi179_148205_object99230:head# 2024-02-09T17:57:33.642 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.642 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f655365:::benchmark_data_smithi179_148205_object5442:head# 2024-02-09T17:57:33.642 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.643 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f65779b:::benchmark_data_smithi179_148205_object55879:head# 2024-02-09T17:57:33.643 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.643 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f65da3c:::benchmark_data_smithi179_148205_object86464:head# 2024-02-09T17:57:33.643 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.643 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f6680c9:::benchmark_data_smithi179_148205_object11115:head# 2024-02-09T17:57:33.643 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.643 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f66b5e4:::benchmark_data_smithi179_148205_object14904:head# 2024-02-09T17:57:33.643 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.643 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f66cc79:::benchmark_data_smithi179_148205_object28173:head# 2024-02-09T17:57:33.643 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.643 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f673267:::benchmark_data_smithi179_148205_object34102:head# 2024-02-09T17:57:33.643 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.644 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f674311:::benchmark_data_smithi179_148205_object30042:head# 2024-02-09T17:57:33.644 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.644 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f6820e3:::benchmark_data_smithi179_148205_object46928:head# 2024-02-09T17:57:33.644 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.644 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f683015:::benchmark_data_smithi179_148205_object35824:head# 2024-02-09T17:57:33.644 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.644 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f68add6:::benchmark_data_smithi179_148205_object91324:head# 2024-02-09T17:57:33.644 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.644 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f6957f0:::benchmark_data_smithi179_148205_object76350:head# 2024-02-09T17:57:33.644 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.644 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f699754:::benchmark_data_smithi179_148205_object22445:head# 2024-02-09T17:57:33.644 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.644 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f69a120:::benchmark_data_smithi179_148205_object9552:head# 2024-02-09T17:57:33.644 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.645 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f6a9b2c:::benchmark_data_smithi179_148205_object19953:head# 2024-02-09T17:57:33.645 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.645 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f6b7234:::benchmark_data_smithi179_148205_object48547:head# 2024-02-09T17:57:33.645 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.645 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f6b96e0:::benchmark_data_smithi179_148205_object44649:head# 2024-02-09T17:57:33.645 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.645 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f6bca7e:::benchmark_data_smithi179_148205_object35628:head# 2024-02-09T17:57:33.645 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.645 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f6c1346:::benchmark_data_smithi179_148205_object87015:head# 2024-02-09T17:57:33.645 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.645 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f6c5f05:::benchmark_data_smithi179_148205_object70630:head# 2024-02-09T17:57:33.645 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.645 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f6ca6f2:::benchmark_data_smithi179_148205_object105197:head# 2024-02-09T17:57:33.646 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.646 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f6cd2e3:::benchmark_data_smithi179_148205_object94207:head# 2024-02-09T17:57:33.646 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.646 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f6cffb2:::benchmark_data_smithi179_148205_object56066:head# 2024-02-09T17:57:33.646 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.646 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f6d4f8e:::benchmark_data_smithi179_148205_object20824:head# 2024-02-09T17:57:33.646 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.646 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f6dec2f:::benchmark_data_smithi179_148205_object43596:head# 2024-02-09T17:57:33.646 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.646 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f6edf97:::benchmark_data_smithi179_148205_object50422:head# 2024-02-09T17:57:33.646 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.646 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f6fccee:::benchmark_data_smithi179_148205_object109764:head# 2024-02-09T17:57:33.646 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.646 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f71b3e7:::benchmark_data_smithi179_148205_object56615:head# 2024-02-09T17:57:33.646 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.647 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f71bfd7:::benchmark_data_smithi179_148205_object10687:head# 2024-02-09T17:57:33.647 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.647 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f7245ad:::benchmark_data_smithi179_148205_object39893:head# 2024-02-09T17:57:33.647 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.647 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f72e48c:::benchmark_data_smithi179_148205_object105942:head# 2024-02-09T17:57:33.647 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.647 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f7323c8:::benchmark_data_smithi179_148205_object31720:head# 2024-02-09T17:57:33.647 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.647 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f73330e:::benchmark_data_smithi179_148205_object54973:head# 2024-02-09T17:57:33.647 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.647 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f733d16:::benchmark_data_smithi179_148205_object21753:head# 2024-02-09T17:57:33.647 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.647 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f734fb8:::benchmark_data_smithi179_148205_object45518:head# 2024-02-09T17:57:33.648 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.648 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f73ba32:::benchmark_data_smithi179_148205_object36404:head# 2024-02-09T17:57:33.648 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.648 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f740aef:::benchmark_data_smithi179_148205_object22905:head# 2024-02-09T17:57:33.648 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.648 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f7499e1:::benchmark_data_smithi179_148205_object90410:head# 2024-02-09T17:57:33.648 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.648 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f74e28a:::benchmark_data_smithi179_148205_object98731:head# 2024-02-09T17:57:33.648 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.648 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f750640:::benchmark_data_smithi179_148205_object66248:head# 2024-02-09T17:57:33.648 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.648 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f754120:::benchmark_data_smithi179_148205_object17221:head# 2024-02-09T17:57:33.648 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.648 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f759719:::benchmark_data_smithi179_148205_object107040:head# 2024-02-09T17:57:33.648 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.649 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f75a902:::benchmark_data_smithi179_148205_object3206:head# 2024-02-09T17:57:33.649 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.649 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f75bf7e:::benchmark_data_smithi179_148205_object98766:head# 2024-02-09T17:57:33.649 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.649 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f762212:::benchmark_data_smithi179_148205_object90426:head# 2024-02-09T17:57:33.649 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.649 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f7683b4:::benchmark_data_smithi179_148205_object20400:head# 2024-02-09T17:57:33.649 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.649 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f787ae7:::benchmark_data_smithi179_148205_object65784:head# 2024-02-09T17:57:33.649 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.649 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f795f00:::benchmark_data_smithi179_148205_object102822:head# 2024-02-09T17:57:33.649 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.649 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f7ad434:::benchmark_data_smithi179_148205_object1784:head# 2024-02-09T17:57:33.649 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.649 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f7b8886:::benchmark_data_smithi179_148205_object68010:head# 2024-02-09T17:57:33.650 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.650 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f7c279c:::benchmark_data_smithi179_148205_object24049:head# 2024-02-09T17:57:33.650 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.650 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f7c4bd0:::benchmark_data_smithi179_148205_object100349:head# 2024-02-09T17:57:33.650 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.650 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f7cac7b:::benchmark_data_smithi179_148205_object3155:head# 2024-02-09T17:57:33.650 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.650 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f7cfc87:::benchmark_data_smithi179_148205_object42745:head# 2024-02-09T17:57:33.650 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.650 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f7d80cc:::benchmark_data_smithi179_148205_object36128:head# 2024-02-09T17:57:33.650 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.650 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f7eace2:::benchmark_data_smithi179_148205_object3443:head# 2024-02-09T17:57:33.650 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.650 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f7f0423:::benchmark_data_smithi179_148205_object47295:head# 2024-02-09T17:57:33.651 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.651 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f7f5a28:::benchmark_data_smithi179_148205_object68724:head# 2024-02-09T17:57:33.651 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.651 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f80bca6:::benchmark_data_smithi179_148205_object62931:head# 2024-02-09T17:57:33.651 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.651 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f8156f0:::benchmark_data_smithi179_148205_object40163:head# 2024-02-09T17:57:33.651 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.651 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f826e56:::benchmark_data_smithi179_148205_object65152:head# 2024-02-09T17:57:33.651 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.651 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f82a88d:::benchmark_data_smithi179_148205_object68938:head# 2024-02-09T17:57:33.651 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.651 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f833da7:::benchmark_data_smithi179_148205_object70854:head# 2024-02-09T17:57:33.651 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.651 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f834fa1:::benchmark_data_smithi179_148205_object101960:head# 2024-02-09T17:57:33.651 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.652 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f83c3ac:::benchmark_data_smithi179_148205_object75132:head# 2024-02-09T17:57:33.652 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.652 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f840596:::benchmark_data_smithi179_148205_object8083:head# 2024-02-09T17:57:33.652 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.652 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f840c34:::benchmark_data_smithi179_148205_object10962:head# 2024-02-09T17:57:33.652 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.652 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f8422c9:::benchmark_data_smithi179_148205_object43997:head# 2024-02-09T17:57:33.652 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.652 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f8442e3:::benchmark_data_smithi179_148205_object17288:head# 2024-02-09T17:57:33.652 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.652 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f84b23a:::benchmark_data_smithi179_148205_object10885:head# 2024-02-09T17:57:33.652 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.652 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f858726:::benchmark_data_smithi179_148205_object15549:head# 2024-02-09T17:57:33.652 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.653 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f868391:::benchmark_data_smithi179_148205_object62322:head# 2024-02-09T17:57:33.653 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.653 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f86c492:::benchmark_data_smithi179_148205_object103062:head# 2024-02-09T17:57:33.653 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.653 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f874fa3:::benchmark_data_smithi179_148205_object63790:head# 2024-02-09T17:57:33.653 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.653 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f878286:::benchmark_data_smithi179_148205_object9559:head# 2024-02-09T17:57:33.653 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.653 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f880c7a:::benchmark_data_smithi179_148205_object44985:head# 2024-02-09T17:57:33.653 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.653 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f884ebe:::benchmark_data_smithi179_148205_object71464:head# 2024-02-09T17:57:33.653 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.653 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f88c079:::benchmark_data_smithi179_148205_object86838:head# 2024-02-09T17:57:33.653 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.653 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f88de42:::benchmark_data_smithi179_148205_object48395:head# 2024-02-09T17:57:33.654 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.654 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f8964cb:::benchmark_data_smithi179_148205_object15874:head# 2024-02-09T17:57:33.654 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.654 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f896871:::benchmark_data_smithi179_148205_object23992:head# 2024-02-09T17:57:33.654 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.654 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f899896:::benchmark_data_smithi179_148205_object104251:head# 2024-02-09T17:57:33.654 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.654 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f8ca92b:::benchmark_data_smithi179_148205_object6688:head# 2024-02-09T17:57:33.654 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.654 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f8cd6e5:::benchmark_data_smithi179_148205_object101997:head# 2024-02-09T17:57:33.654 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.654 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f8d8135:::benchmark_data_smithi179_148205_object36110:head# 2024-02-09T17:57:33.654 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.654 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f8d98c4:::benchmark_data_smithi179_148205_object32813:head# 2024-02-09T17:57:33.655 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.655 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f8ea928:::benchmark_data_smithi179_148205_object88799:head# 2024-02-09T17:57:33.655 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.655 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f8fc0b0:::benchmark_data_smithi179_148205_object2428:head# 2024-02-09T17:57:33.655 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.655 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f90bd0b:::benchmark_data_smithi179_148205_object104053:head# 2024-02-09T17:57:33.655 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.655 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f90c7f8:::benchmark_data_smithi179_148205_object106644:head# 2024-02-09T17:57:33.655 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.655 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f928889:::benchmark_data_smithi179_148205_object72509:head# 2024-02-09T17:57:33.655 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.655 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f928d3c:::benchmark_data_smithi179_148205_object50796:head# 2024-02-09T17:57:33.655 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.655 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f93b51a:::benchmark_data_smithi179_148205_object99202:head# 2024-02-09T17:57:33.655 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.656 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f93e827:::benchmark_data_smithi179_148205_object7892:head# 2024-02-09T17:57:33.656 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.656 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f93fdf2:::benchmark_data_smithi179_148205_object72103:head# 2024-02-09T17:57:33.656 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.656 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f948a69:::benchmark_data_smithi179_148205_object66434:head# 2024-02-09T17:57:33.656 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.656 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f94a35b:::benchmark_data_smithi179_148205_object25298:head# 2024-02-09T17:57:33.656 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.656 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f94f174:::benchmark_data_smithi179_148205_object66022:head# 2024-02-09T17:57:33.656 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.656 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f950ca4:::benchmark_data_smithi179_148205_object12090:head# 2024-02-09T17:57:33.656 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.656 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f9640d9:::benchmark_data_smithi179_148205_object39082:head# 2024-02-09T17:57:33.656 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.657 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f96d095:::benchmark_data_smithi179_148205_object15103:head# 2024-02-09T17:57:33.657 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.657 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f970c99:::benchmark_data_smithi179_148205_object73652:head# 2024-02-09T17:57:33.657 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.657 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f976246:::benchmark_data_smithi179_148205_object69235:head# 2024-02-09T17:57:33.657 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.657 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f97f674:::benchmark_data_smithi179_148205_object55488:head# 2024-02-09T17:57:33.657 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.657 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f9a42a1:::benchmark_data_smithi179_148205_object26916:head# 2024-02-09T17:57:33.657 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.657 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f9a5b16:::benchmark_data_smithi179_148205_object107309:head# 2024-02-09T17:57:33.657 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.657 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f9a6acb:::benchmark_data_smithi179_148205_object45716:head# 2024-02-09T17:57:33.657 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.657 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f9c3223:::benchmark_data_smithi179_148205_object79385:head# 2024-02-09T17:57:33.658 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.658 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f9cbe43:::benchmark_data_smithi179_148205_object27361:head# 2024-02-09T17:57:33.658 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.658 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f9d03b2:::benchmark_data_smithi179_148205_object40154:head# 2024-02-09T17:57:33.658 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.658 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f9d60c8:::benchmark_data_smithi179_148205_object50579:head# 2024-02-09T17:57:33.658 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.658 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f9e501f:::benchmark_data_smithi179_148205_object34067:head# 2024-02-09T17:57:33.658 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.658 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f9ecac0:::benchmark_data_smithi179_148205_object107105:head# 2024-02-09T17:57:33.658 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.658 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f9f1cd6:::benchmark_data_smithi179_148205_object33128:head# 2024-02-09T17:57:33.658 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.658 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f9fb05e:::benchmark_data_smithi179_148205_object67846:head# 2024-02-09T17:57:33.658 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.658 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1f9ff99e:::benchmark_data_smithi179_148205_object99320:head# 2024-02-09T17:57:33.659 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.659 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fa0d4e6:::benchmark_data_smithi179_148205_object32710:head# 2024-02-09T17:57:33.659 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.659 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fa0d9e9:::benchmark_data_smithi179_148205_object102619:head# 2024-02-09T17:57:33.659 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.659 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fa0f434:::benchmark_data_smithi179_148205_object5638:head# 2024-02-09T17:57:33.659 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.659 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fa140fc:::benchmark_data_smithi179_148205_object49551:head# 2024-02-09T17:57:33.659 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.659 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fa1d157:::benchmark_data_smithi179_148205_object87500:head# 2024-02-09T17:57:33.659 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.659 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fa22a8e:::benchmark_data_smithi179_148205_object41785:head# 2024-02-09T17:57:33.659 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.659 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fa28040:::benchmark_data_smithi179_148205_object12979:head# 2024-02-09T17:57:33.660 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.660 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fa36e09:::benchmark_data_smithi179_148205_object81203:head# 2024-02-09T17:57:33.660 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.660 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fa3af85:::benchmark_data_smithi179_148205_object11691:head# 2024-02-09T17:57:33.660 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.660 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fa562b0:::benchmark_data_smithi179_148205_object50914:head# 2024-02-09T17:57:33.660 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.660 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fa5dded:::benchmark_data_smithi179_148205_object36517:head# 2024-02-09T17:57:33.660 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.660 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fa729b3:::benchmark_data_smithi179_148205_object15068:head# 2024-02-09T17:57:33.660 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.660 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fa76489:::benchmark_data_smithi179_148205_object7516:head# 2024-02-09T17:57:33.660 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.660 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fa7ba94:::benchmark_data_smithi179_148205_object91955:head# 2024-02-09T17:57:33.660 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.661 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fa8a5da:::benchmark_data_smithi179_148205_object73434:head# 2024-02-09T17:57:33.661 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.661 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fa8de60:::benchmark_data_smithi179_148205_object79177:head# 2024-02-09T17:57:33.661 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.661 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fa96a72:::benchmark_data_smithi179_148205_object101861:head# 2024-02-09T17:57:33.661 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.661 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1faa33be:::benchmark_data_smithi179_148205_object70375:head# 2024-02-09T17:57:33.661 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.661 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1faab63b:::benchmark_data_smithi179_148205_object26581:head# 2024-02-09T17:57:33.661 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.661 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fab60f7:::benchmark_data_smithi179_148205_object14613:head# 2024-02-09T17:57:33.661 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.661 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fabe7e5:::benchmark_data_smithi179_148205_object89824:head# 2024-02-09T17:57:33.661 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.662 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fabf431:::benchmark_data_smithi179_148205_object20938:head# 2024-02-09T17:57:33.662 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.662 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fac0fb1:::benchmark_data_smithi179_148205_object93807:head# 2024-02-09T17:57:33.662 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.662 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fac4be3:::benchmark_data_smithi179_148205_object37766:head# 2024-02-09T17:57:33.662 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.662 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fac5821:::benchmark_data_smithi179_148205_object15514:head# 2024-02-09T17:57:33.662 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.662 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1facc0e4:::benchmark_data_smithi179_148205_object72531:head# 2024-02-09T17:57:33.662 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.662 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fadea8b:::benchmark_data_smithi179_148205_object2512:head# 2024-02-09T17:57:33.662 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.662 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fae2fef:::benchmark_data_smithi179_148205_object55447:head# 2024-02-09T17:57:33.662 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.662 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fae83d9:::benchmark_data_smithi179_148205_object50295:head# 2024-02-09T17:57:33.662 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.663 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1faeb8a5:::benchmark_data_smithi179_148205_object63689:head# 2024-02-09T17:57:33.663 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.663 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fafe9c8:::benchmark_data_smithi179_148205_object41655:head# 2024-02-09T17:57:33.663 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.663 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fb1550d:::benchmark_data_smithi179_148205_object39389:head# 2024-02-09T17:57:33.663 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.663 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fb20b3a:::benchmark_data_smithi179_148205_object71943:head# 2024-02-09T17:57:33.663 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.663 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fb2303a:::benchmark_data_smithi179_148205_object94476:head# 2024-02-09T17:57:33.663 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.663 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fb31cb2:::benchmark_data_smithi179_148205_object84555:head# 2024-02-09T17:57:33.663 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.664 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fb7c578:::benchmark_data_smithi179_148205_object7666:head# 2024-02-09T17:57:33.664 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.664 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fb9dc55:::benchmark_data_smithi179_148205_object22880:head# 2024-02-09T17:57:33.664 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.664 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fbb8060:::benchmark_data_smithi179_148205_object70083:head# 2024-02-09T17:57:33.664 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:33.666 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:57:33.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:57:33.681 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fbba84d:::benchmark_data_smithi179_148205_object66195:head# 2024-02-09T17:57:33.682 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.682 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fbbc05d:::benchmark_data_smithi179_148205_object50209:head# 2024-02-09T17:57:33.682 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.682 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fbbcf09:::benchmark_data_smithi179_148205_object2543:head# 2024-02-09T17:57:33.682 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.682 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fbc32e0:::benchmark_data_smithi179_148205_object102511:head# 2024-02-09T17:57:33.682 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.682 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fbc6844:::benchmark_data_smithi179_148205_object2444:head# 2024-02-09T17:57:33.682 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.682 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fbd1332:::benchmark_data_smithi179_148205_object36182:head# 2024-02-09T17:57:33.682 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.682 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fbd758b:::benchmark_data_smithi179_148205_object69271:head# 2024-02-09T17:57:33.682 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.683 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fbdd3f3:::benchmark_data_smithi179_148205_object26186:head# 2024-02-09T17:57:33.683 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.683 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fbf4bd7:::benchmark_data_smithi179_148205_object86731:head# 2024-02-09T17:57:33.683 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.683 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fbfd75c:::benchmark_data_smithi179_148205_object69419:head# 2024-02-09T17:57:33.683 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.683 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fc1664c:::benchmark_data_smithi179_148205_object41847:head# 2024-02-09T17:57:33.683 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.683 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fc23531:::benchmark_data_smithi179_148205_object48217:head# 2024-02-09T17:57:33.683 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.683 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fc2afc2:::benchmark_data_smithi179_148205_object42465:head# 2024-02-09T17:57:33.683 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.683 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fc342b4:::benchmark_data_smithi179_148205_object54002:head# 2024-02-09T17:57:33.683 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.684 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fc43f3e:::benchmark_data_smithi179_148205_object26189:head# 2024-02-09T17:57:33.684 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.684 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fc4ca04:::benchmark_data_smithi179_148205_object66330:head# 2024-02-09T17:57:33.684 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.684 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fc60866:::benchmark_data_smithi179_148205_object74718:head# 2024-02-09T17:57:33.684 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.684 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fc751ff:::benchmark_data_smithi179_148205_object32408:head# 2024-02-09T17:57:33.684 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.684 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fc83753:::benchmark_data_smithi179_148205_object100787:head# 2024-02-09T17:57:33.684 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.684 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fc901c4:::benchmark_data_smithi179_148205_object103149:head# 2024-02-09T17:57:33.684 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.685 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fc95bf7:::benchmark_data_smithi179_148205_object33790:head# 2024-02-09T17:57:33.685 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.685 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fc97a52:::benchmark_data_smithi179_148205_object83883:head# 2024-02-09T17:57:33.685 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.685 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fca003a:::benchmark_data_smithi179_148205_object15716:head# 2024-02-09T17:57:33.685 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.685 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fcbba18:::benchmark_data_smithi179_148205_object74099:head# 2024-02-09T17:57:33.685 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.685 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fcc1b8e:::benchmark_data_smithi179_148205_object72334:head# 2024-02-09T17:57:33.685 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.685 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fccfd3b:::benchmark_data_smithi179_148205_object67070:head# 2024-02-09T17:57:33.685 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.685 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fcf5178:::benchmark_data_smithi179_148205_object69370:head# 2024-02-09T17:57:33.685 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.686 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fcfe09b:::benchmark_data_smithi179_148205_object86214:head# 2024-02-09T17:57:33.686 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.686 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fcfe1dd:::benchmark_data_smithi179_148205_object6173:head# 2024-02-09T17:57:33.686 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.686 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fd00327:::benchmark_data_smithi179_148205_object94217:head# 2024-02-09T17:57:33.686 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.686 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fd0b4ac:::benchmark_data_smithi179_148205_object54357:head# 2024-02-09T17:57:33.686 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.686 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fd0c764:::benchmark_data_smithi179_148205_object18655:head# 2024-02-09T17:57:33.686 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.686 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fd12554:::benchmark_data_smithi179_148205_object47065:head# 2024-02-09T17:57:33.686 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.686 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fd15bc8:::benchmark_data_smithi179_148205_object76746:head# 2024-02-09T17:57:33.686 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.686 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fd1a7ed:::benchmark_data_smithi179_148205_object32513:head# 2024-02-09T17:57:33.687 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.687 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fd26f97:::benchmark_data_smithi179_148205_object89088:head# 2024-02-09T17:57:33.687 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.687 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fd28548:::benchmark_data_smithi179_148205_object36892:head# 2024-02-09T17:57:33.687 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.687 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fd32c20:::benchmark_data_smithi179_148205_object99172:head# 2024-02-09T17:57:33.687 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.687 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fd3bdc0:::benchmark_data_smithi179_148205_object32364:head# 2024-02-09T17:57:33.687 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.687 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fd45b0d:::benchmark_data_smithi179_148205_object34843:head# 2024-02-09T17:57:33.687 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.687 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fd46819:::benchmark_data_smithi179_148205_object20606:head# 2024-02-09T17:57:33.687 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.688 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fd47d57:::benchmark_data_smithi179_148205_object81527:head# 2024-02-09T17:57:33.688 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.688 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fd52860:::benchmark_data_smithi179_148205_object7786:head# 2024-02-09T17:57:33.688 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.688 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fd53a8e:::benchmark_data_smithi179_148205_object35647:head# 2024-02-09T17:57:33.688 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.688 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fd54e16:::benchmark_data_smithi179_148205_object11732:head# 2024-02-09T17:57:33.688 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.688 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fd5a84d:::benchmark_data_smithi179_148205_object66318:head# 2024-02-09T17:57:33.688 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.688 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fd6dd31:::benchmark_data_smithi179_148205_object61050:head# 2024-02-09T17:57:33.688 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.688 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fd71dbc:::benchmark_data_smithi179_148205_object32587:head# 2024-02-09T17:57:33.688 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.689 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fd90895:::benchmark_data_smithi179_148205_object87378:head# 2024-02-09T17:57:33.689 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.689 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fd9a7b2:::benchmark_data_smithi179_148205_object54335:head# 2024-02-09T17:57:33.689 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.689 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fd9a990:::benchmark_data_smithi179_148205_object97000:head# 2024-02-09T17:57:33.689 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.689 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fd9dcc4:::benchmark_data_smithi179_148205_object59098:head# 2024-02-09T17:57:33.689 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.689 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fdad2bb:::benchmark_data_smithi179_148205_object48998:head# 2024-02-09T17:57:33.689 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.689 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fdd6310:::benchmark_data_smithi179_148205_object92341:head# 2024-02-09T17:57:33.689 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.689 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fddd1e4:::benchmark_data_smithi179_148205_object90340:head# 2024-02-09T17:57:33.689 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.690 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fddfa3f:::benchmark_data_smithi179_148205_object18432:head# 2024-02-09T17:57:33.690 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.690 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fde49b8:::benchmark_data_smithi179_148205_object39100:head# 2024-02-09T17:57:33.690 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.690 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fe32f5d:::benchmark_data_smithi179_148205_object29002:head# 2024-02-09T17:57:33.690 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.690 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fe365ca:::benchmark_data_smithi179_148205_object46893:head# 2024-02-09T17:57:33.690 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.690 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fe42f2b:::benchmark_data_smithi179_148205_object61936:head# 2024-02-09T17:57:33.690 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.690 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fe472a3:::benchmark_data_smithi179_148205_object90618:head# 2024-02-09T17:57:33.690 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.690 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fe5c0a3:::benchmark_data_smithi179_148205_object12799:head# 2024-02-09T17:57:33.690 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.690 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fe64d79:::benchmark_data_smithi179_148205_object87938:head# 2024-02-09T17:57:33.691 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.691 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fe68aaf:::benchmark_data_smithi179_148205_object41908:head# 2024-02-09T17:57:33.691 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.691 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fe79160:::benchmark_data_smithi179_148205_object104427:head# 2024-02-09T17:57:33.691 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.691 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fe7aa6b:::benchmark_data_smithi179_148205_object43561:head# 2024-02-09T17:57:33.691 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.691 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fe7cade:::benchmark_data_smithi179_148205_object11625:head# 2024-02-09T17:57:33.691 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.691 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fe864fe:::benchmark_data_smithi179_148205_object81645:head# 2024-02-09T17:57:33.691 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.691 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fe9785a:::benchmark_data_smithi179_148205_object79204:head# 2024-02-09T17:57:33.691 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.691 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fe987f8:::benchmark_data_smithi179_148205_object88384:head# 2024-02-09T17:57:33.692 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.692 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1feb0b1e:::benchmark_data_smithi179_148205_object109483:head# 2024-02-09T17:57:33.692 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.692 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1febc932:::benchmark_data_smithi179_148205_object579:head# 2024-02-09T17:57:33.692 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.692 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1febe262:::benchmark_data_smithi179_148205_object5676:head# 2024-02-09T17:57:33.692 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.692 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1febf714:::benchmark_data_smithi179_148205_object3139:head# 2024-02-09T17:57:33.692 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.692 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fecc590:::benchmark_data_smithi179_148205_object21441:head# 2024-02-09T17:57:33.692 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.692 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fed7031:::benchmark_data_smithi179_148205_object48038:head# 2024-02-09T17:57:33.692 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.692 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fedd264:::benchmark_data_smithi179_148205_object69957:head# 2024-02-09T17:57:33.693 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.693 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fee008a:::benchmark_data_smithi179_148205_object88023:head# 2024-02-09T17:57:33.693 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.693 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1feee0ac:::benchmark_data_smithi179_148205_object108620:head# 2024-02-09T17:57:33.693 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.693 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fef3eca:::benchmark_data_smithi179_148205_object44922:head# 2024-02-09T17:57:33.693 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.693 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fef44dd:::benchmark_data_smithi179_148205_object39209:head# 2024-02-09T17:57:33.693 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.693 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fefef90:::benchmark_data_smithi179_148205_object62908:head# 2024-02-09T17:57:33.693 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.693 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ff0eee5:::benchmark_data_smithi179_148205_object42207:head# 2024-02-09T17:57:33.693 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.693 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ff19144:::benchmark_data_smithi179_148205_object71424:head# 2024-02-09T17:57:33.693 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.694 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ff19783:::benchmark_data_smithi179_148205_object14074:head# 2024-02-09T17:57:33.694 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.694 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ff24023:::benchmark_data_smithi179_148205_object70420:head# 2024-02-09T17:57:33.694 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.694 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ff29cc8:::benchmark_data_smithi179_148205_object6665:head# 2024-02-09T17:57:33.694 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.694 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ff38f0c:::benchmark_data_smithi179_148205_object50719:head# 2024-02-09T17:57:33.694 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.694 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ff39a47:::benchmark_data_smithi179_148205_object107915:head# 2024-02-09T17:57:33.694 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.694 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ff3be2f:::benchmark_data_smithi179_148205_object57602:head# 2024-02-09T17:57:33.694 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.694 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ff4f715:::benchmark_data_smithi179_148205_object76107:head# 2024-02-09T17:57:33.694 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.695 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ff579e3:::benchmark_data_smithi179_148205_object82640:head# 2024-02-09T17:57:33.695 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.695 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ff58ec0:::benchmark_data_smithi179_148205_object11016:head# 2024-02-09T17:57:33.695 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.695 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ff5adab:::benchmark_data_smithi179_148205_object60756:head# 2024-02-09T17:57:33.695 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.695 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ff5f0d4:::benchmark_data_smithi179_148205_object25575:head# 2024-02-09T17:57:33.695 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.695 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ff6ccf3:::benchmark_data_smithi179_148205_object102406:head# 2024-02-09T17:57:33.695 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.695 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ff6ceb3:::benchmark_data_smithi179_148205_object10211:head# 2024-02-09T17:57:33.695 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.695 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ff75ef5:::benchmark_data_smithi179_148205_object48082:head# 2024-02-09T17:57:33.695 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.695 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ff7b773:::benchmark_data_smithi179_148205_object45327:head# 2024-02-09T17:57:33.696 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.696 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ff9ca37:::benchmark_data_smithi179_148205_object23237:head# 2024-02-09T17:57:33.696 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.696 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ffa6fb3:::benchmark_data_smithi179_148205_object8469:head# 2024-02-09T17:57:33.696 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.696 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ffaa3ab:::benchmark_data_smithi179_148205_object36793:head# 2024-02-09T17:57:33.696 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.696 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ffadb98:::benchmark_data_smithi179_148205_object36760:head# 2024-02-09T17:57:33.696 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.696 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ffb3ee2:::benchmark_data_smithi179_148205_object75221:head# 2024-02-09T17:57:33.696 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.696 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ffb69ad:::benchmark_data_smithi179_148205_object19628:head# 2024-02-09T17:57:33.696 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.696 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ffca75e:::benchmark_data_smithi179_148205_object66871:head# 2024-02-09T17:57:33.696 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.697 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1ffe0bd9:::benchmark_data_smithi179_148205_object7120:head# 2024-02-09T17:57:33.697 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.697 INFO:teuthology.orchestra.run.smithi099.stdout:Write #8:1fff10fd:::benchmark_data_smithi179_148205_object47895:head# 2024-02-09T17:57:33.697 INFO:teuthology.orchestra.run.smithi099.stdout:snapset 0=[]:{} 2024-02-09T17:57:33.697 INFO:teuthology.orchestra.run.smithi099.stdout:write_pg epoch 814 info 8.8( v 814'6895 (0'0,814'6895] local-lis/les=785/786 n=6895 ec=785/785 lis/c=785/785 les/c/f=786/786/0 sis=785) 2024-02-09T17:57:33.705 INFO:teuthology.orchestra.run.smithi099.stdout:Import successful 2024-02-09T17:57:33.767 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:57:33.807 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:33.884 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:57:33.888 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:33.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:57:33.950 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:57:33.982 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:57:33.989 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:57:34.019 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:34.059 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:57:34.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:34.068 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:57:34.169 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:57:34.195 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:34.196 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:34 smithi099 ceph-mon[39420]: pgmap v2052: 47 pgs: 1 active+clean+scrubbing, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 5.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 814 op/s; 17283/277986 objects degraded (6.217%) 2024-02-09T17:57:34.221 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:57:34.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:34.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:57:34.325 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:57:34.359 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:34.427 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:34 smithi179 ceph-mon[43975]: pgmap v2052: 47 pgs: 1 active+clean+scrubbing, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 5.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 814 op/s; 17283/277986 objects degraded (6.217%) 2024-02-09T17:57:34.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:34 smithi162 ceph-mon[44304]: pgmap v2052: 47 pgs: 1 active+clean+scrubbing, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 5.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 814 op/s; 17283/277986 objects degraded (6.217%) 2024-02-09T17:57:35.319 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:57:35.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:35.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:57:35.426 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:57:35.479 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:35.531 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:57:35.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:35.535 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:57:35.636 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:57:35.706 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:35.740 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:57:35.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:35.752 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:57:35.846 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -f /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/exp.8.8.3 2024-02-09T17:57:35.852 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:57:35.911 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set noscrub 2024-02-09T17:57:35.914 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:35.966 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:57:35.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:35.970 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:57:36.071 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:57:36.109 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:57:36.136 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:36.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:36 smithi099 ceph-mon[39420]: pgmap v2053: 47 pgs: 1 active+clean+scrubbing, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 5.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 750 op/s; 17283/273564 objects degraded (6.318%) 2024-02-09T17:57:36.618 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:57:36.618 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:57:36.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:36 smithi179 ceph-mon[43975]: pgmap v2053: 47 pgs: 1 active+clean+scrubbing, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 5.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 750 op/s; 17283/273564 objects degraded (6.318%) 2024-02-09T17:57:36.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:36 smithi162 ceph-mon[44304]: pgmap v2053: 47 pgs: 1 active+clean+scrubbing, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 5.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 750 op/s; 17283/273564 objects degraded (6.318%) 2024-02-09T17:57:37.294 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:57:37.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:37.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:57:37.401 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:57:37.429 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:37.452 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:57:37.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:37.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:57:37.560 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:57:37.601 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:37 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 17283/273564 objects degraded (6.318%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:57:37.605 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:37.660 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:57:37.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:37.667 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:57:37.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:37 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 17283/273564 objects degraded (6.318%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:57:37.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:37 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 17283/273564 objects degraded (6.318%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:57:37.768 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:57:37.798 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:37.821 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:57:37.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:37.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:57:37.927 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:57:37.956 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:37.981 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:57:37.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:37.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:57:38.085 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:57:38.130 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:38.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:38 smithi099 ceph-mon[39420]: pgmap v2054: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 5.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 788 op/s; 16551/267639 objects degraded (6.184%) 2024-02-09T17:57:38.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:38 smithi179 ceph-mon[43975]: pgmap v2054: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 5.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 788 op/s; 16551/267639 objects degraded (6.184%) 2024-02-09T17:57:38.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:38 smithi162 ceph-mon[44304]: pgmap v2054: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 5.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 788 op/s; 16551/267639 objects degraded (6.184%) 2024-02-09T17:57:40.476 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-09T17:57:40.488 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:57:40.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:40.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:57:40.593 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:57:40.619 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:40 smithi162 ceph-mon[44304]: pgmap v2055: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 5.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 803 op/s; 16551/263013 objects degraded (6.293%) 2024-02-09T17:57:40.622 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:40.643 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:57:40.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:40.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:57:40.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:40 smithi179 ceph-mon[43975]: pgmap v2055: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 5.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 803 op/s; 16551/263013 objects degraded (6.293%) 2024-02-09T17:57:40.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:40 smithi099 ceph-mon[39420]: pgmap v2055: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 5.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 803 op/s; 16551/263013 objects degraded (6.293%) 2024-02-09T17:57:40.747 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:57:40.777 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:40.798 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:57:40.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:40.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:57:40.903 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:57:40.931 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:40.958 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:57:40.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:40.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:57:41.007 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 2] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2, 4, 5, 6, 7, 9, 10, 11, 8] 2024-02-09T17:57:41.007 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:57:41.007 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-02-09T17:57:41.008 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-09T17:57:41.061 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:57:41.113 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:41.132 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:57:41.136 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:57:41.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:41.143 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:57:41.150 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:57:41.155 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:57:41.166 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:57:41.190 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:57:41.242 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:57:41.243 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:57:41.271 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:41.273 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:57:41.296 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:57:41.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:41.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:57:41.308 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:57:41.314 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:41 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3871813429' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:57:41.337 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:57:41.372 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:57:41.400 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:57:41.405 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:57:41.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:41 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3871813429' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:57:41.428 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:41.441 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:57:41.450 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:57:41.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:41.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:57:41.474 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:57:41.474 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:57:41.474 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:57:41.474 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:57:41.474 INFO:teuthology.orchestra.run.smithi099.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-09T17:57:41.474 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:57:41.475 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:57:41.475 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:57:41.475 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:57:41.475 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:57:41.475 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:57:41.475 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:57:41.475 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:57:41.555 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:57:41.589 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:41.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:41 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3871813429' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:57:41.836 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:57:41.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:41.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:57:41.944 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:57:41.974 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:42.000 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:57:42.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:42.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:57:42.104 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:57:42.143 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:42.321 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:42 smithi162 ceph-mon[44304]: pgmap v2056: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 5.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 802 op/s; 16551/261471 objects degraded (6.330%) 2024-02-09T17:57:42.321 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:42 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 16551/261471 objects degraded (6.330%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:57:42.398 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:57:42.402 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:42 smithi099 ceph-mon[39420]: pgmap v2056: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 5.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 802 op/s; 16551/261471 objects degraded (6.330%) 2024-02-09T17:57:42.402 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:42 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 16551/261471 objects degraded (6.330%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:57:42.402 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:42.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:57:42.418 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-09T17:57:42.418 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-09T17:57:42.504 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:57:42.538 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:42.621 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:57:42.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:42.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:57:42.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:42 smithi179 ceph-mon[43975]: pgmap v2056: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 5.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 802 op/s; 16551/261471 objects degraded (6.330%) 2024-02-09T17:57:42.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:42 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 16551/261471 objects degraded (6.330%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:57:42.724 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:57:42.783 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:42.827 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:57:42.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:42.835 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:57:42.936 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:57:43.003 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:43.040 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:57:43.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:43.047 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:57:43.148 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:57:43.207 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:43.236 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:57:43.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:43.239 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:57:43.340 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:57:43.404 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:43.447 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:57:43.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:43.453 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:57:43.554 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:57:43.581 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:43.604 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:57:43.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:43.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:57:43.710 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:57:43.786 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:43.828 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:57:43.831 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:43.832 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:57:43.933 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:57:43.987 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:44.019 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:57:44.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:44.022 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:57:44.123 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:57:44.152 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:44.174 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:57:44.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:44.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:57:44.282 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:57:44.317 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:44.345 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:57:44.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:44.348 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:57:44.420 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:57:44.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:44 smithi162 ceph-mon[44304]: 1.1a scrub starts 2024-02-09T17:57:44.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:44 smithi162 ceph-mon[44304]: 1.1a scrub ok 2024-02-09T17:57:44.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:44 smithi162 ceph-mon[44304]: pgmap v2057: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 851 op/s; 15771/253827 objects degraded (6.213%) 2024-02-09T17:57:44.449 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:57:44.487 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:44.505 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:57:44.532 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:57:44.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:44.536 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:57:44.637 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:57:44.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:44 smithi179 ceph-mon[43975]: 1.1a scrub starts 2024-02-09T17:57:44.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:44 smithi179 ceph-mon[43975]: 1.1a scrub ok 2024-02-09T17:57:44.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:44 smithi179 ceph-mon[43975]: pgmap v2057: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 851 op/s; 15771/253827 objects degraded (6.213%) 2024-02-09T17:57:44.704 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:44.732 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:57:44.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:44.735 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:57:44.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:44 smithi099 ceph-mon[39420]: 1.1a scrub starts 2024-02-09T17:57:44.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:44 smithi099 ceph-mon[39420]: 1.1a scrub ok 2024-02-09T17:57:44.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:44 smithi099 ceph-mon[39420]: pgmap v2057: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 851 op/s; 15771/253827 objects degraded (6.213%) 2024-02-09T17:57:44.836 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:57:44.898 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:44.923 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:57:44.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:44.926 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:57:45.027 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:57:45.055 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:45.079 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:57:45.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:45.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:57:45.187 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:57:45.251 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:45.279 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:57:45.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:45.283 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:57:45.383 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:57:45.411 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:45.440 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:57:45.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:45.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:57:45.546 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:57:45.604 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:45.605 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:45 smithi179 ceph-mon[43975]: 1.13 scrub starts 2024-02-09T17:57:45.605 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:45 smithi179 ceph-mon[43975]: 1.2 scrub starts 2024-02-09T17:57:45.605 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:45 smithi179 ceph-mon[43975]: 1.13 scrub ok 2024-02-09T17:57:45.605 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:45 smithi179 ceph-mon[43975]: 1.2 scrub ok 2024-02-09T17:57:45.605 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:45 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:57:45.606 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:45 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:57:45.611 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:57:45.611 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 20, 2024-02-09T17:57:45.611 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 600, 2024-02-09T17:57:45.611 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [ 2024-02-09T17:57:45.611 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.611 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25837.0:203763 6.4 6:2535f01d:::benchmark_data_smithi179_126206_object73382:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e638)", 2024-02-09T17:57:45.611 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:48:48.769263+0000", 2024-02-09T17:57:45.611 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 536.52514483499999, 2024-02-09T17:57:45.611 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.221425328, 2024-02-09T17:57:45.611 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:57:45.611 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:57:45.611 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:57:45.611 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25837", 2024-02-09T17:57:45.612 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:57:45.612 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 203763 2024-02-09T17:57:45.612 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.612 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:57:45.612 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.612 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:57:45.612 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.769263+0000", 2024-02-09T17:57:45.612 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:45.612 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.612 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.612 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:57:45.612 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.769263+0000", 2024-02-09T17:57:45.612 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:45.612 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.613 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.613 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:57:45.613 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.769264+0000", 2024-02-09T17:57:45.613 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.751e-06 2024-02-09T17:57:45.613 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.613 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.613 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:57:45.613 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.769270+0000", 2024-02-09T17:57:45.613 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.5099999999999998e-06 2024-02-09T17:57:45.613 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.613 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.613 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:57:45.613 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.769280+0000", 2024-02-09T17:57:45.613 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.7790000000000007e-06 2024-02-09T17:57:45.613 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.614 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.614 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:57:45.614 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.769286+0000", 2024-02-09T17:57:45.614 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.2740000000000004e-06 2024-02-09T17:57:45.614 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.614 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.614 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:57:45.614 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.769333+0000", 2024-02-09T17:57:45.614 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.7534999999999998e-05 2024-02-09T17:57:45.614 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.614 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.614 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:57:45.614 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.769453+0000", 2024-02-09T17:57:45.614 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000119601 2024-02-09T17:57:45.614 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.615 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.615 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,5", 2024-02-09T17:57:45.615 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.769565+0000", 2024-02-09T17:57:45.615 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000111505 2024-02-09T17:57:45.615 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.615 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.615 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:57:45.615 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.790278+0000", 2024-02-09T17:57:45.615 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.020713809999999999 2024-02-09T17:57:45.615 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.615 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.615 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:45.616 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.811239+0000", 2024-02-09T17:57:45.616 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.020960138999999999 2024-02-09T17:57:45.616 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.616 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.616 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:45.616 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.990621+0000", 2024-02-09T17:57:45.616 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.17938268199999999 2024-02-09T17:57:45.616 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.616 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.616 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:57:45.616 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.990660+0000", 2024-02-09T17:57:45.616 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9014999999999998e-05 2024-02-09T17:57:45.616 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.616 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.617 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:57:45.617 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.990688+0000", 2024-02-09T17:57:45.617 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.7727000000000001e-05 2024-02-09T17:57:45.617 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:45.617 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:45.617 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:45.617 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.617 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.617 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25837.0:203764 6.4 6:282276a4:::benchmark_data_smithi179_126206_object73383:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e638)", 2024-02-09T17:57:45.617 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:48:48.769301+0000", 2024-02-09T17:57:45.617 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 536.52510604600002, 2024-02-09T17:57:45.617 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.22147773000000001, 2024-02-09T17:57:45.618 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:57:45.618 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:57:45.618 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:57:45.618 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25837", 2024-02-09T17:57:45.618 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:57:45.618 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 203764 2024-02-09T17:57:45.618 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.618 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:57:45.618 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.618 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:57:45.618 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.769301+0000", 2024-02-09T17:57:45.618 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:45.618 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.618 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.618 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:57:45.619 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.769301+0000", 2024-02-09T17:57:45.619 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:45.619 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.619 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.619 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:57:45.619 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.769302+0000", 2024-02-09T17:57:45.619 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.5900000000000002e-07 2024-02-09T17:57:45.619 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.619 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.619 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:57:45.619 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.769304+0000", 2024-02-09T17:57:45.619 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.5639999999999999e-06 2024-02-09T17:57:45.619 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.619 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.619 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:57:45.620 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.769311+0000", 2024-02-09T17:57:45.620 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.6950000000000003e-06 2024-02-09T17:57:45.620 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.620 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.620 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:57:45.620 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.769317+0000", 2024-02-09T17:57:45.620 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.4219999999999999e-06 2024-02-09T17:57:45.620 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.620 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.620 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:57:45.620 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.774400+0000", 2024-02-09T17:57:45.620 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0050838890000000003 2024-02-09T17:57:45.620 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.620 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.620 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:57:45.621 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.774528+0000", 2024-02-09T17:57:45.621 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012776199999999999 2024-02-09T17:57:45.621 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.621 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.621 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,5", 2024-02-09T17:57:45.621 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.774657+0000", 2024-02-09T17:57:45.621 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000129123 2024-02-09T17:57:45.621 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.621 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.621 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:57:45.621 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.790498+0000", 2024-02-09T17:57:45.621 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.015840559000000001 2024-02-09T17:57:45.621 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.621 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.622 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:45.622 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.811292+0000", 2024-02-09T17:57:45.622 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.020793736 2024-02-09T17:57:45.622 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.622 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.622 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:45.622 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.990722+0000", 2024-02-09T17:57:45.622 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.179430535 2024-02-09T17:57:45.622 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.622 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.622 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:57:45.622 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.990755+0000", 2024-02-09T17:57:45.622 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.2574999999999999e-05 2024-02-09T17:57:45.623 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.623 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.623 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:57:45.623 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.990779+0000", 2024-02-09T17:57:45.623 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.4411000000000001e-05 2024-02-09T17:57:45.623 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:45.623 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:45.623 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:45.623 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.623 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.623 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25837.0:203767 6.4 6:2ef8b14c:::benchmark_data_smithi179_126206_object73386:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e638)", 2024-02-09T17:57:45.623 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:48:48.771761+0000", 2024-02-09T17:57:45.623 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 536.52264627600005, 2024-02-09T17:57:45.623 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.229900145, 2024-02-09T17:57:45.623 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:57:45.623 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:57:45.624 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:57:45.624 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25837", 2024-02-09T17:57:45.624 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:57:45.624 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 203767 2024-02-09T17:57:45.624 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.624 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:57:45.624 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.624 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:57:45.624 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.771761+0000", 2024-02-09T17:57:45.624 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:45.624 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.624 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.625 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:57:45.625 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.771761+0000", 2024-02-09T17:57:45.625 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:45.625 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.625 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.625 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:57:45.625 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.771762+0000", 2024-02-09T17:57:45.625 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1960000000000001e-06 2024-02-09T17:57:45.625 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.625 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.625 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:57:45.625 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.771767+0000", 2024-02-09T17:57:45.625 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.7090000000000001e-06 2024-02-09T17:57:45.625 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.625 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.625 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:57:45.626 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.771780+0000", 2024-02-09T17:57:45.626 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2500999999999999e-05 2024-02-09T17:57:45.626 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.626 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.626 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:57:45.626 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.771788+0000", 2024-02-09T17:57:45.626 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.2560000000000002e-06 2024-02-09T17:57:45.626 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.626 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.626 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:57:45.626 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.784186+0000", 2024-02-09T17:57:45.626 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.012397874 2024-02-09T17:57:45.626 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.626 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.626 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:57:45.626 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.784322+0000", 2024-02-09T17:57:45.626 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000136038 2024-02-09T17:57:45.627 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.627 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.627 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,5", 2024-02-09T17:57:45.627 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.784437+0000", 2024-02-09T17:57:45.627 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00011572400000000001 2024-02-09T17:57:45.627 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.627 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.627 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:57:45.627 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.801411+0000", 2024-02-09T17:57:45.627 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.016973722 2024-02-09T17:57:45.627 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.627 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.628 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:45.628 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.811405+0000", 2024-02-09T17:57:45.628 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0099935340000000001 2024-02-09T17:57:45.628 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.628 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.628 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:45.628 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:49.001602+0000", 2024-02-09T17:57:45.629 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.19019723099999999 2024-02-09T17:57:45.629 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.629 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.629 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:57:45.629 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:49.001637+0000", 2024-02-09T17:57:45.629 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.5113000000000001e-05 2024-02-09T17:57:45.629 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.629 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.629 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:57:45.629 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:49.001661+0000", 2024-02-09T17:57:45.629 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.4247e-05 2024-02-09T17:57:45.630 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:45.630 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:45.630 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:45.630 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.630 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.630 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25837.0:203782 6.4 6:29d9fa0d:::benchmark_data_smithi179_126206_object73401:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e638)", 2024-02-09T17:57:45.630 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:48:48.791451+0000", 2024-02-09T17:57:45.631 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 536.50295682000001, 2024-02-09T17:57:45.631 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.23181099899999999, 2024-02-09T17:57:45.631 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:57:45.631 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:57:45.631 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:57:45.631 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25837", 2024-02-09T17:57:45.631 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:57:45.631 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 203782 2024-02-09T17:57:45.631 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.631 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:57:45.631 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.631 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:57:45.631 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.791451+0000", 2024-02-09T17:57:45.631 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:45.632 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.632 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.632 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:57:45.632 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.791451+0000", 2024-02-09T17:57:45.632 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:45.632 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.632 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.632 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:57:45.632 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.791452+0000", 2024-02-09T17:57:45.633 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5200000000000001e-06 2024-02-09T17:57:45.633 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.633 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.633 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:57:45.633 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.791457+0000", 2024-02-09T17:57:45.634 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.9049999999999996e-06 2024-02-09T17:57:45.634 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.634 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.634 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:57:45.634 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.791470+0000", 2024-02-09T17:57:45.634 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2860999999999999e-05 2024-02-09T17:57:45.634 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.635 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.635 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:57:45.635 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.791479+0000", 2024-02-09T17:57:45.635 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.8589999999999994e-06 2024-02-09T17:57:45.635 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.635 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.635 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:57:45.636 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.791534+0000", 2024-02-09T17:57:45.636 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.5389000000000001e-05 2024-02-09T17:57:45.636 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.636 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.636 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:57:45.636 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.791658+0000", 2024-02-09T17:57:45.636 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000123638 2024-02-09T17:57:45.637 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.637 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.637 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,5", 2024-02-09T17:57:45.637 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.791780+0000", 2024-02-09T17:57:45.637 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000121746 2024-02-09T17:57:45.637 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.637 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.637 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:57:45.637 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.801416+0000", 2024-02-09T17:57:45.637 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0096366400000000001 2024-02-09T17:57:45.637 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.637 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.637 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:45.638 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.811443+0000", 2024-02-09T17:57:45.638 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.010027230999999999 2024-02-09T17:57:45.638 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.638 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.638 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:45.638 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:49.023163+0000", 2024-02-09T17:57:45.638 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.211719871 2024-02-09T17:57:45.638 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.638 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.638 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:57:45.638 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:49.023220+0000", 2024-02-09T17:57:45.638 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6338999999999997e-05 2024-02-09T17:57:45.638 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.638 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.638 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:57:45.638 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:49.023262+0000", 2024-02-09T17:57:45.639 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.1999999999999998e-05 2024-02-09T17:57:45.639 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:45.639 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:45.639 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:45.639 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.639 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.639 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25837.0:203783 6.4 6:2118d7a2:::benchmark_data_smithi179_126206_object73402:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e638)", 2024-02-09T17:57:45.639 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:48:48.791775+0000", 2024-02-09T17:57:45.639 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 536.50263250499995, 2024-02-09T17:57:45.639 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.237107436, 2024-02-09T17:57:45.639 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:57:45.639 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:57:45.639 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:57:45.639 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25837", 2024-02-09T17:57:45.639 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:57:45.639 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 203783 2024-02-09T17:57:45.640 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.640 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:57:45.640 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.640 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:57:45.640 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.791775+0000", 2024-02-09T17:57:45.640 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:45.640 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.640 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.640 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:57:45.640 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.791775+0000", 2024-02-09T17:57:45.640 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:45.640 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.640 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.640 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:57:45.641 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.791776+0000", 2024-02-09T17:57:45.641 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.8500000000000001e-07 2024-02-09T17:57:45.641 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.641 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.641 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:57:45.641 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.791779+0000", 2024-02-09T17:57:45.641 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.2440000000000002e-06 2024-02-09T17:57:45.641 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.641 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.641 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:57:45.641 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.791794+0000", 2024-02-09T17:57:45.641 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4722999999999999e-05 2024-02-09T17:57:45.641 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.641 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.642 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:57:45.642 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.791800+0000", 2024-02-09T17:57:45.642 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.8519999999999998e-06 2024-02-09T17:57:45.642 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.642 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.642 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:57:45.642 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.797008+0000", 2024-02-09T17:57:45.642 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0052077060000000003 2024-02-09T17:57:45.642 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.642 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.642 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:57:45.642 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.797134+0000", 2024-02-09T17:57:45.642 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012541700000000001 2024-02-09T17:57:45.642 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.643 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.643 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,5", 2024-02-09T17:57:45.643 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.797259+0000", 2024-02-09T17:57:45.643 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012533999999999999 2024-02-09T17:57:45.643 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.643 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.643 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:57:45.643 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.802889+0000", 2024-02-09T17:57:45.643 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0056301090000000003 2024-02-09T17:57:45.643 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.643 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.643 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:45.643 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.811505+0000", 2024-02-09T17:57:45.644 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0086157820000000006 2024-02-09T17:57:45.644 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.644 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.644 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:45.644 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:49.028781+0000", 2024-02-09T17:57:45.644 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.21727664999999999 2024-02-09T17:57:45.644 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.644 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.644 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:57:45.644 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:49.028839+0000", 2024-02-09T17:57:45.644 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7933999999999998e-05 2024-02-09T17:57:45.644 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.644 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.644 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:57:45.645 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:49.028882+0000", 2024-02-09T17:57:45.645 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2994000000000002e-05 2024-02-09T17:57:45.645 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:45.645 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:45.645 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:45.645 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.645 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.645 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25837.0:203801 6.4 6:217ce34f:::benchmark_data_smithi179_126206_object73420:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e638)", 2024-02-09T17:57:45.645 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:48:48.813837+0000", 2024-02-09T17:57:45.645 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 536.48057043300003, 2024-02-09T17:57:45.645 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.215159932, 2024-02-09T17:57:45.645 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:57:45.645 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:57:45.645 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:57:45.645 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25837", 2024-02-09T17:57:45.646 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:57:45.646 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 203801 2024-02-09T17:57:45.646 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.646 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:57:45.646 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.646 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:57:45.646 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.813837+0000", 2024-02-09T17:57:45.646 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:45.646 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.646 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.646 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:57:45.646 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.813837+0000", 2024-02-09T17:57:45.646 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:45.646 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.646 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.646 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:57:45.647 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.813838+0000", 2024-02-09T17:57:45.647 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.077e-06 2024-02-09T17:57:45.647 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.647 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.647 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:57:45.647 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.813842+0000", 2024-02-09T17:57:45.647 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2259999999999999e-06 2024-02-09T17:57:45.647 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.647 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.647 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:57:45.647 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.813853+0000", 2024-02-09T17:57:45.647 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0304e-05 2024-02-09T17:57:45.647 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.647 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.647 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:57:45.647 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.813863+0000", 2024-02-09T17:57:45.648 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.049e-05 2024-02-09T17:57:45.648 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.648 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.648 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:57:45.648 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.813917+0000", 2024-02-09T17:57:45.648 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3625000000000002e-05 2024-02-09T17:57:45.648 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.648 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.648 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:57:45.648 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.814067+0000", 2024-02-09T17:57:45.648 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00015053700000000001 2024-02-09T17:57:45.648 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.648 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.648 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,5", 2024-02-09T17:57:45.648 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.814191+0000", 2024-02-09T17:57:45.649 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012389500000000001 2024-02-09T17:57:45.649 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.649 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.649 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:45.649 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.819437+0000", 2024-02-09T17:57:45.649 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0052463240000000001 2024-02-09T17:57:45.649 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.649 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.649 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:57:45.649 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.819577+0000", 2024-02-09T17:57:45.649 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013955700000000001 2024-02-09T17:57:45.649 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.649 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.649 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:45.649 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:49.028918+0000", 2024-02-09T17:57:45.649 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20934101599999999 2024-02-09T17:57:45.649 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.649 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.650 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:57:45.650 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:49.028964+0000", 2024-02-09T17:57:45.650 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.6236999999999998e-05 2024-02-09T17:57:45.650 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.650 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.650 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:57:45.650 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:49.028997+0000", 2024-02-09T17:57:45.650 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.2644000000000001e-05 2024-02-09T17:57:45.650 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:45.650 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:45.650 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:45.650 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.650 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.650 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25837.0:203816 6.4 6:22e11e42:::benchmark_data_smithi179_126206_object73435:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e638)", 2024-02-09T17:57:45.650 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:48:48.836412+0000", 2024-02-09T17:57:45.650 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 536.45799573700003, 2024-02-09T17:57:45.651 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.19269656800000001, 2024-02-09T17:57:45.651 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:57:45.651 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:57:45.651 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:57:45.651 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25837", 2024-02-09T17:57:45.651 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:57:45.651 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 203816 2024-02-09T17:57:45.651 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.651 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:57:45.651 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.651 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:57:45.651 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.836412+0000", 2024-02-09T17:57:45.651 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:45.651 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.651 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.651 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:57:45.652 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.836412+0000", 2024-02-09T17:57:45.652 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:45.652 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.652 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.652 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:57:45.652 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.836412+0000", 2024-02-09T17:57:45.652 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.0200000000000002e-07 2024-02-09T17:57:45.652 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.652 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.652 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:57:45.652 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.836415+0000", 2024-02-09T17:57:45.652 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.3700000000000002e-06 2024-02-09T17:57:45.652 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.652 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.652 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:57:45.653 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.836420+0000", 2024-02-09T17:57:45.653 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3900000000000001e-06 2024-02-09T17:57:45.653 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.653 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.653 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:57:45.653 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.836424+0000", 2024-02-09T17:57:45.653 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.805e-06 2024-02-09T17:57:45.653 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.653 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.653 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:57:45.653 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.836475+0000", 2024-02-09T17:57:45.653 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1493000000000003e-05 2024-02-09T17:57:45.653 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.653 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.653 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:57:45.653 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.836585+0000", 2024-02-09T17:57:45.654 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00010932199999999999 2024-02-09T17:57:45.654 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.654 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.654 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,5", 2024-02-09T17:57:45.654 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.836696+0000", 2024-02-09T17:57:45.654 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000111456 2024-02-09T17:57:45.654 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.654 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.654 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:57:45.654 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.844807+0000", 2024-02-09T17:57:45.654 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0081107969999999995 2024-02-09T17:57:45.654 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.654 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.654 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:45.655 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:48.853708+0000", 2024-02-09T17:57:45.655 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0089015420000000001 2024-02-09T17:57:45.655 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.655 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.655 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:45.655 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:49.029030+0000", 2024-02-09T17:57:45.655 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.17532159899999999 2024-02-09T17:57:45.655 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.655 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.655 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:57:45.655 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:49.029074+0000", 2024-02-09T17:57:45.655 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.4195e-05 2024-02-09T17:57:45.655 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.655 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.655 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:57:45.655 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:48:49.029108+0000", 2024-02-09T17:57:45.656 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.4097000000000002e-05 2024-02-09T17:57:45.656 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:45.656 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:45.656 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:45.656 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.656 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.656 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25837.0:250108 6.4 6:2d609273:::benchmark_data_smithi179_126206_object119727:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:57:45.656 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:49:47.518298+0000", 2024-02-09T17:57:45.656 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 477.77610897800002, 2024-02-09T17:57:45.656 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.242809575, 2024-02-09T17:57:45.656 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:57:45.656 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:57:45.656 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:57:45.656 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25837", 2024-02-09T17:57:45.657 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:57:45.657 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 250108 2024-02-09T17:57:45.657 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.657 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:57:45.657 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.657 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:57:45.657 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.518298+0000", 2024-02-09T17:57:45.657 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:45.657 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.657 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.657 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:57:45.657 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.518298+0000", 2024-02-09T17:57:45.657 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:45.657 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.657 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.657 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:57:45.658 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.518300+0000", 2024-02-09T17:57:45.658 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5260000000000001e-06 2024-02-09T17:57:45.658 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.658 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.658 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:57:45.658 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.518305+0000", 2024-02-09T17:57:45.658 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.5279999999999997e-06 2024-02-09T17:57:45.658 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.658 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.658 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:57:45.658 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.518316+0000", 2024-02-09T17:57:45.658 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1148000000000001e-05 2024-02-09T17:57:45.658 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.658 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.658 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:57:45.659 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.518323+0000", 2024-02-09T17:57:45.659 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.5140000000000004e-06 2024-02-09T17:57:45.659 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.659 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.659 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:57:45.659 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.518367+0000", 2024-02-09T17:57:45.659 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.3683000000000002e-05 2024-02-09T17:57:45.659 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.659 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.659 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:57:45.659 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.518467+0000", 2024-02-09T17:57:45.659 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.9852999999999994e-05 2024-02-09T17:57:45.659 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.659 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.659 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,5", 2024-02-09T17:57:45.660 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.518542+0000", 2024-02-09T17:57:45.660 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.5304999999999998e-05 2024-02-09T17:57:45.660 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.660 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.660 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:57:45.660 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.522701+0000", 2024-02-09T17:57:45.660 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0041589640000000002 2024-02-09T17:57:45.660 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.660 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.660 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:45.660 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.533905+0000", 2024-02-09T17:57:45.660 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.011204492 2024-02-09T17:57:45.660 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.661 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.661 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:45.661 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.760994+0000", 2024-02-09T17:57:45.661 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.22708893099999999 2024-02-09T17:57:45.661 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.661 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.661 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:57:45.661 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.761061+0000", 2024-02-09T17:57:45.661 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.6198000000000006e-05 2024-02-09T17:57:45.661 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.661 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.661 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:57:45.661 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.761108+0000", 2024-02-09T17:57:45.661 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.7432999999999998e-05 2024-02-09T17:57:45.661 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:45.662 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:45.662 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:45.662 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.662 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.662 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25837.0:250121 6.4 6:264967d5:::benchmark_data_smithi179_126206_object119740:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:57:45.662 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:49:47.538505+0000", 2024-02-09T17:57:45.662 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 477.75590256700002, 2024-02-09T17:57:45.662 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.222748849, 2024-02-09T17:57:45.662 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:57:45.662 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:57:45.662 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:57:45.662 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25837", 2024-02-09T17:57:45.662 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:57:45.662 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 250121 2024-02-09T17:57:45.662 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.663 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:57:45.663 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.663 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:57:45.664 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.538505+0000", 2024-02-09T17:57:45.664 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:45.664 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.664 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.664 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:57:45.664 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.538505+0000", 2024-02-09T17:57:45.664 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:45.664 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.664 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.664 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:57:45.664 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.538507+0000", 2024-02-09T17:57:45.664 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.7400000000000001e-06 2024-02-09T17:57:45.664 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.665 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.665 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:57:45.665 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.538515+0000", 2024-02-09T17:57:45.665 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.0190000000000006e-06 2024-02-09T17:57:45.665 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.665 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.665 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:57:45.665 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.538530+0000", 2024-02-09T17:57:45.665 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5335000000000001e-05 2024-02-09T17:57:45.665 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.665 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.665 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:57:45.665 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.538539+0000", 2024-02-09T17:57:45.665 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.3489999999999998e-06 2024-02-09T17:57:45.665 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.666 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.666 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:57:45.666 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.538609+0000", 2024-02-09T17:57:45.666 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.957e-05 2024-02-09T17:57:45.666 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.666 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.666 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:57:45.666 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.538757+0000", 2024-02-09T17:57:45.666 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014793200000000001 2024-02-09T17:57:45.666 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.666 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.666 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,5", 2024-02-09T17:57:45.666 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.538883+0000", 2024-02-09T17:57:45.666 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000125711 2024-02-09T17:57:45.667 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.667 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.667 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:57:45.667 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.543299+0000", 2024-02-09T17:57:45.667 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.004416906 2024-02-09T17:57:45.667 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.667 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.667 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:45.667 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.554527+0000", 2024-02-09T17:57:45.667 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.011227391 2024-02-09T17:57:45.667 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.667 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.667 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:45.667 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.761161+0000", 2024-02-09T17:57:45.668 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.206634028 2024-02-09T17:57:45.668 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.668 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.668 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:57:45.668 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.761217+0000", 2024-02-09T17:57:45.668 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6282000000000001e-05 2024-02-09T17:57:45.668 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.668 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.668 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:57:45.668 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.761254+0000", 2024-02-09T17:57:45.668 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.6585999999999998e-05 2024-02-09T17:57:45.668 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:45.668 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:45.669 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:45.669 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.669 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.669 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25837.0:250138 6.4 6:263dfb23:::benchmark_data_smithi179_126206_object119757:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:57:45.669 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:49:47.556378+0000", 2024-02-09T17:57:45.669 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 477.73802953500001, 2024-02-09T17:57:45.669 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20504225100000001, 2024-02-09T17:57:45.669 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:57:45.669 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:57:45.669 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:57:45.669 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25837", 2024-02-09T17:57:45.669 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:57:45.669 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 250138 2024-02-09T17:57:45.669 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.670 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:57:45.670 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.670 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:57:45.670 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.556378+0000", 2024-02-09T17:57:45.670 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:45.670 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.670 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.670 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:57:45.670 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.556378+0000", 2024-02-09T17:57:45.670 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:45.670 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.670 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.670 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:57:45.670 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.556379+0000", 2024-02-09T17:57:45.671 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.217e-06 2024-02-09T17:57:45.671 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.671 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.671 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:57:45.671 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.556384+0000", 2024-02-09T17:57:45.671 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.4039999999999997e-06 2024-02-09T17:57:45.671 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.671 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.671 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:57:45.671 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.556399+0000", 2024-02-09T17:57:45.671 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5355000000000001e-05 2024-02-09T17:57:45.671 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.671 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.672 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:57:45.672 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.556407+0000", 2024-02-09T17:57:45.672 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.0409999999999998e-06 2024-02-09T17:57:45.672 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.672 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.672 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:57:45.672 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.556461+0000", 2024-02-09T17:57:45.672 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.4159000000000001e-05 2024-02-09T17:57:45.672 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.672 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.672 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:57:45.672 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.556584+0000", 2024-02-09T17:57:45.673 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012253899999999999 2024-02-09T17:57:45.673 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.673 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.673 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,5", 2024-02-09T17:57:45.673 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.556680+0000", 2024-02-09T17:57:45.673 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.6075000000000004e-05 2024-02-09T17:57:45.673 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.673 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.673 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:57:45.673 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.561115+0000", 2024-02-09T17:57:45.673 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0044352219999999999 2024-02-09T17:57:45.673 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.674 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.674 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:45.674 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.570136+0000", 2024-02-09T17:57:45.674 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0090209369999999997 2024-02-09T17:57:45.674 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.674 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.674 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:45.674 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.761321+0000", 2024-02-09T17:57:45.674 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.191185144 2024-02-09T17:57:45.674 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.674 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.674 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:57:45.674 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.761371+0000", 2024-02-09T17:57:45.674 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.9793999999999998e-05 2024-02-09T17:57:45.674 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.675 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.675 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:57:45.675 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.761420+0000", 2024-02-09T17:57:45.675 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.9363999999999997e-05 2024-02-09T17:57:45.675 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:45.675 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:45.675 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:45.675 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.675 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.675 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.25837.0:250144 6.4 6:2d888af1:::benchmark_data_smithi179_126206_object119763:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:57:45.675 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:49:47.562632+0000", 2024-02-09T17:57:45.675 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 477.73177514600002, 2024-02-09T17:57:45.675 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20241316000000001, 2024-02-09T17:57:45.675 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:57:45.676 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:57:45.676 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:57:45.676 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.25837", 2024-02-09T17:57:45.676 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:57:45.676 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 250144 2024-02-09T17:57:45.676 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.676 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:57:45.676 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.676 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:57:45.676 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.562632+0000", 2024-02-09T17:57:45.676 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:45.676 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.676 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.676 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:57:45.676 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.562632+0000", 2024-02-09T17:57:45.677 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:45.677 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.677 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.677 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:57:45.677 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.562634+0000", 2024-02-09T17:57:45.677 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2929999999999999e-06 2024-02-09T17:57:45.677 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.677 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.677 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:57:45.677 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.562637+0000", 2024-02-09T17:57:45.677 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.8410000000000002e-06 2024-02-09T17:57:45.677 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.677 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.677 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:57:45.678 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.562644+0000", 2024-02-09T17:57:45.678 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.4330000000000001e-06 2024-02-09T17:57:45.678 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.678 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.678 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:57:45.678 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.562648+0000", 2024-02-09T17:57:45.678 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2679999999999996e-06 2024-02-09T17:57:45.678 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.678 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.678 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:57:45.678 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.562708+0000", 2024-02-09T17:57:45.678 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.0024e-05 2024-02-09T17:57:45.678 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.679 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.679 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:57:45.679 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.562842+0000", 2024-02-09T17:57:45.679 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013413699999999999 2024-02-09T17:57:45.679 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.679 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.679 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 3,5", 2024-02-09T17:57:45.679 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.562971+0000", 2024-02-09T17:57:45.679 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012912000000000001 2024-02-09T17:57:45.679 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.679 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.679 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:57:45.679 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.567206+0000", 2024-02-09T17:57:45.679 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0042345400000000002 2024-02-09T17:57:45.679 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.680 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.680 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:45.680 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.571339+0000", 2024-02-09T17:57:45.680 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0041335349999999998 2024-02-09T17:57:45.680 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.680 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.680 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:45.680 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.764960+0000", 2024-02-09T17:57:45.680 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.19362069500000001 2024-02-09T17:57:45.680 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.680 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.680 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:57:45.680 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.765013+0000", 2024-02-09T17:57:45.680 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2379000000000001e-05 2024-02-09T17:57:45.680 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.681 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.681 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:57:45.681 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:49:47.765045+0000", 2024-02-09T17:57:45.681 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.2895000000000003e-05 2024-02-09T17:57:45.681 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:45.681 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:45.681 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:45.681 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.681 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.681 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.16440.0:191383 7.b 7:d3ae1e92:::benchmark_data_smithi179_135804_object83592:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e754)", 2024-02-09T17:57:45.681 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:54:11.829256+0000", 2024-02-09T17:57:45.681 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 213.46515169200001, 2024-02-09T17:57:45.681 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.206107082, 2024-02-09T17:57:45.681 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:57:45.682 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:57:45.682 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:57:45.682 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.16440", 2024-02-09T17:57:45.682 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:57:45.682 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 191383 2024-02-09T17:57:45.682 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.682 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:57:45.682 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.682 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:57:45.682 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:11.829256+0000", 2024-02-09T17:57:45.682 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:45.682 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.683 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.683 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:57:45.683 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:11.829256+0000", 2024-02-09T17:57:45.683 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:45.683 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.683 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.683 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:57:45.683 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:11.829258+0000", 2024-02-09T17:57:45.683 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.787e-06 2024-02-09T17:57:45.683 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.683 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.683 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:57:45.683 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:11.829265+0000", 2024-02-09T17:57:45.683 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.9419999999999995e-06 2024-02-09T17:57:45.683 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.684 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.684 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:57:45.684 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:11.829286+0000", 2024-02-09T17:57:45.684 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0516000000000001e-05 2024-02-09T17:57:45.684 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.684 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.684 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:57:45.684 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:11.829296+0000", 2024-02-09T17:57:45.684 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.9340000000000001e-06 2024-02-09T17:57:45.684 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.684 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.684 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:57:45.684 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:11.829362+0000", 2024-02-09T17:57:45.685 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.6328000000000006e-05 2024-02-09T17:57:45.685 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.685 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.685 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:57:45.685 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:11.829494+0000", 2024-02-09T17:57:45.685 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000131524 2024-02-09T17:57:45.685 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.685 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.685 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 5,11", 2024-02-09T17:57:45.685 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:11.829605+0000", 2024-02-09T17:57:45.685 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00011152100000000001 2024-02-09T17:57:45.685 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.685 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.685 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:57:45.685 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:11.834985+0000", 2024-02-09T17:57:45.686 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0053798500000000003 2024-02-09T17:57:45.686 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.686 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.686 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:45.686 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:11.838110+0000", 2024-02-09T17:57:45.686 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0031245460000000002 2024-02-09T17:57:45.686 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.686 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.686 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:45.686 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:12.035266+0000", 2024-02-09T17:57:45.686 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.197156628 2024-02-09T17:57:45.686 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.686 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.686 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:57:45.686 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:12.035324+0000", 2024-02-09T17:57:45.687 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7428e-05 2024-02-09T17:57:45.687 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.687 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.687 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:57:45.687 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:12.035363+0000", 2024-02-09T17:57:45.687 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9078000000000001e-05 2024-02-09T17:57:45.687 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:45.687 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:45.687 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:45.687 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.687 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.687 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.16440.0:191393 7.b 7:dcf28c3e:::benchmark_data_smithi179_135804_object83602:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e754)", 2024-02-09T17:57:45.687 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:54:11.840739+0000", 2024-02-09T17:57:45.687 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 213.453668158, 2024-02-09T17:57:45.688 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.214610569, 2024-02-09T17:57:45.688 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:57:45.688 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:57:45.688 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:57:45.688 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.16440", 2024-02-09T17:57:45.688 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:57:45.688 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 191393 2024-02-09T17:57:45.688 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.688 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:57:45.688 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.688 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:57:45.688 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:11.840739+0000", 2024-02-09T17:57:45.688 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:45.688 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.689 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.689 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:57:45.689 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:11.840739+0000", 2024-02-09T17:57:45.689 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:45.689 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.689 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.689 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:57:45.689 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:11.840741+0000", 2024-02-09T17:57:45.689 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.9539999999999998e-06 2024-02-09T17:57:45.689 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.689 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.689 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:57:45.689 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:11.840748+0000", 2024-02-09T17:57:45.689 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.4609999999999996e-06 2024-02-09T17:57:45.689 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.690 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.690 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:57:45.690 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:11.840761+0000", 2024-02-09T17:57:45.690 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3655000000000001e-05 2024-02-09T17:57:45.690 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.690 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.690 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:57:45.690 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:11.840770+0000", 2024-02-09T17:57:45.690 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.3529999999999995e-06 2024-02-09T17:57:45.690 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.690 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.690 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:57:45.690 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:11.841406+0000", 2024-02-09T17:57:45.690 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00063584100000000005 2024-02-09T17:57:45.690 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.690 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.690 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:57:45.691 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:11.841535+0000", 2024-02-09T17:57:45.691 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000129713 2024-02-09T17:57:45.691 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.691 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.691 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 5,11", 2024-02-09T17:57:45.691 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:11.841653+0000", 2024-02-09T17:57:45.691 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00011736 2024-02-09T17:57:45.691 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.691 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.691 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:57:45.691 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:11.852713+0000", 2024-02-09T17:57:45.691 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.011060716 2024-02-09T17:57:45.691 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.691 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.691 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:45.691 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:11.855438+0000", 2024-02-09T17:57:45.691 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.002724602 2024-02-09T17:57:45.692 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.692 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.692 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:45.692 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:12.055246+0000", 2024-02-09T17:57:45.692 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.19980847099999999 2024-02-09T17:57:45.692 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.692 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.692 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:57:45.692 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:12.055306+0000", 2024-02-09T17:57:45.692 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.9982999999999998e-05 2024-02-09T17:57:45.692 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.692 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.692 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:57:45.692 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:12.055350+0000", 2024-02-09T17:57:45.693 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.3460000000000001e-05 2024-02-09T17:57:45.693 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:45.693 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:45.693 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:45.693 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.693 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.693 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.16440.0:191395 7.b 7:dcd175db:::benchmark_data_smithi179_135804_object83604:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e754)", 2024-02-09T17:57:45.693 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:54:11.841043+0000", 2024-02-09T17:57:45.693 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 213.453364717, 2024-02-09T17:57:45.693 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.214434281, 2024-02-09T17:57:45.693 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:57:45.693 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:57:45.693 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:57:45.693 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.16440", 2024-02-09T17:57:45.693 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:57:45.694 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 191395 2024-02-09T17:57:45.694 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.694 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:57:45.694 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.694 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:57:45.694 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:11.841043+0000", 2024-02-09T17:57:45.694 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:45.694 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.694 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.694 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:57:45.694 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:11.841043+0000", 2024-02-09T17:57:45.694 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:45.694 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.694 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.694 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:57:45.694 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:11.841043+0000", 2024-02-09T17:57:45.695 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.3900000000000004e-07 2024-02-09T17:57:45.695 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.695 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.695 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:57:45.695 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:11.841047+0000", 2024-02-09T17:57:45.695 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.7680000000000002e-06 2024-02-09T17:57:45.695 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.695 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.695 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:57:45.695 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:11.841054+0000", 2024-02-09T17:57:45.695 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.3010000000000001e-06 2024-02-09T17:57:45.695 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.695 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.695 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:57:45.696 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:11.841060+0000", 2024-02-09T17:57:45.696 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.8440000000000003e-06 2024-02-09T17:57:45.696 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.696 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.696 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:57:45.696 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:11.846061+0000", 2024-02-09T17:57:45.696 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.005000422 2024-02-09T17:57:45.696 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.696 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.696 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:57:45.696 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:11.846201+0000", 2024-02-09T17:57:45.696 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000140643 2024-02-09T17:57:45.696 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.697 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.697 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 5,11", 2024-02-09T17:57:45.697 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:11.846332+0000", 2024-02-09T17:57:45.697 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000130727 2024-02-09T17:57:45.697 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.697 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.697 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:57:45.697 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:11.852805+0000", 2024-02-09T17:57:45.697 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0064724939999999996 2024-02-09T17:57:45.697 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.697 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.697 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:45.697 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:11.855478+0000", 2024-02-09T17:57:45.697 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0026737829999999999 2024-02-09T17:57:45.697 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.698 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.698 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:57:45.698 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:12.055387+0000", 2024-02-09T17:57:45.698 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.19990860899999999 2024-02-09T17:57:45.698 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.698 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.698 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:57:45.698 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:12.055440+0000", 2024-02-09T17:57:45.698 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3047999999999998e-05 2024-02-09T17:57:45.698 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.698 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.698 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:57:45.698 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:12.055477+0000", 2024-02-09T17:57:45.699 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.7002999999999998e-05 2024-02-09T17:57:45.699 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:45.699 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:45.699 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:45.699 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.699 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.699 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.26305.0:164145 8.b e828/785 8:dc04903d:::benchmark_data_smithi179_148205_object41073:head v 828'10144, mlcod=828'10142)", 2024-02-09T17:57:45.699 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:57:44.188675+0000", 2024-02-09T17:57:45.699 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 1.105732044, 2024-02-09T17:57:45.699 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.21037414600000001, 2024-02-09T17:57:45.699 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:57:45.699 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:57:45.699 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:57:45.699 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.700 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:57:45.700 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.188675+0000", 2024-02-09T17:57:45.700 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:45.700 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.700 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.700 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:57:45.700 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.188675+0000", 2024-02-09T17:57:45.700 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:45.700 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.700 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.700 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:57:45.700 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.188676+0000", 2024-02-09T17:57:45.700 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.3300000000000002e-07 2024-02-09T17:57:45.700 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.701 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.701 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:57:45.701 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.188681+0000", 2024-02-09T17:57:45.701 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.0390000000000004e-06 2024-02-09T17:57:45.701 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.701 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.701 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:57:45.701 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.188691+0000", 2024-02-09T17:57:45.701 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.8409999999999995e-06 2024-02-09T17:57:45.701 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.701 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.701 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:57:45.701 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.188696+0000", 2024-02-09T17:57:45.701 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.8570000000000003e-06 2024-02-09T17:57:45.702 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.702 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.702 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:57:45.702 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.188776+0000", 2024-02-09T17:57:45.702 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.0110000000000004e-05 2024-02-09T17:57:45.702 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.702 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.702 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:57:45.702 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.188790+0000", 2024-02-09T17:57:45.702 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4416000000000001e-05 2024-02-09T17:57:45.702 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.702 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.702 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:57:45.702 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.399017+0000", 2024-02-09T17:57:45.703 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.210226722 2024-02-09T17:57:45.703 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.703 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.703 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:57:45.703 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.399050+0000", 2024-02-09T17:57:45.703 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.2527999999999998e-05 2024-02-09T17:57:45.703 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:45.703 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:45.703 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:45.703 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.703 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.703 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.26305.0:164151 8.3 e828/785 8:c4ea51f3:::benchmark_data_smithi179_148205_object41079:head v 828'10316, mlcod=828'10314)", 2024-02-09T17:57:45.703 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:57:44.201574+0000", 2024-02-09T17:57:45.703 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 1.0928333480000001, 2024-02-09T17:57:45.703 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.19749834999999999, 2024-02-09T17:57:45.704 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:57:45.704 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:57:45.704 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:57:45.704 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.704 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:57:45.704 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.201574+0000", 2024-02-09T17:57:45.704 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:45.704 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.704 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.704 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:57:45.704 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.201574+0000", 2024-02-09T17:57:45.704 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:45.704 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.704 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.705 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:57:45.705 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.201575+0000", 2024-02-09T17:57:45.705 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.5099999999999999e-07 2024-02-09T17:57:45.705 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.705 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.705 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:57:45.705 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.201580+0000", 2024-02-09T17:57:45.705 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6160000000000001e-06 2024-02-09T17:57:45.705 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.705 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.705 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:57:45.705 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.201590+0000", 2024-02-09T17:57:45.705 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.9990000000000003e-06 2024-02-09T17:57:45.705 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.706 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.706 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:57:45.706 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.201597+0000", 2024-02-09T17:57:45.706 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.1619999999999996e-06 2024-02-09T17:57:45.706 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.706 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.706 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:57:45.706 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.201646+0000", 2024-02-09T17:57:45.706 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.9722000000000003e-05 2024-02-09T17:57:45.706 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.706 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.706 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:57:45.706 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.201666+0000", 2024-02-09T17:57:45.707 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.9936e-05 2024-02-09T17:57:45.707 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.707 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.707 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:57:45.707 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.399053+0000", 2024-02-09T17:57:45.707 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.19738677700000001 2024-02-09T17:57:45.707 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.707 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.707 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:57:45.707 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.399072+0000", 2024-02-09T17:57:45.707 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.9386999999999999e-05 2024-02-09T17:57:45.707 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:45.707 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:45.707 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:45.707 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.708 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.708 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.26305.0:164167 8.b e828/785 8:ddd29c13:::benchmark_data_smithi179_148205_object41095:head v 828'10145, mlcod=828'10142)", 2024-02-09T17:57:45.708 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:57:44.217911+0000", 2024-02-09T17:57:45.708 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 1.0764968020000001, 2024-02-09T17:57:45.708 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.18118167900000001, 2024-02-09T17:57:45.708 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:57:45.708 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:57:45.708 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:57:45.708 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.708 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:57:45.708 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.217911+0000", 2024-02-09T17:57:45.708 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:45.708 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.709 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.709 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:57:45.709 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.217911+0000", 2024-02-09T17:57:45.709 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:45.709 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.709 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.709 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:57:45.709 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.217911+0000", 2024-02-09T17:57:45.709 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.68e-07 2024-02-09T17:57:45.709 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.709 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.709 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:57:45.709 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.217915+0000", 2024-02-09T17:57:45.709 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2549999999999999e-06 2024-02-09T17:57:45.709 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.710 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.710 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:57:45.710 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.217925+0000", 2024-02-09T17:57:45.710 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.4560000000000006e-06 2024-02-09T17:57:45.710 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.710 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.710 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:57:45.710 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.217930+0000", 2024-02-09T17:57:45.710 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.6829999999999999e-06 2024-02-09T17:57:45.710 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.710 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.710 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:57:45.711 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.217995+0000", 2024-02-09T17:57:45.711 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.5001999999999999e-05 2024-02-09T17:57:45.711 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.711 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.711 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:57:45.711 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.218015+0000", 2024-02-09T17:57:45.711 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0598000000000002e-05 2024-02-09T17:57:45.711 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.711 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.711 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:57:45.711 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.399075+0000", 2024-02-09T17:57:45.711 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.18105960400000001 2024-02-09T17:57:45.711 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.712 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.712 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:57:45.712 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.399092+0000", 2024-02-09T17:57:45.712 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.7513e-05 2024-02-09T17:57:45.712 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:45.712 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:45.712 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:45.712 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.712 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.712 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.26305.0:164179 8.3 e828/785 8:c5848782:::benchmark_data_smithi179_148205_object41107:head v 828'10317, mlcod=828'10314)", 2024-02-09T17:57:45.712 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:57:44.230199+0000", 2024-02-09T17:57:45.712 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 1.064208713, 2024-02-09T17:57:45.712 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.16890950499999999, 2024-02-09T17:57:45.712 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:57:45.713 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:57:45.713 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:57:45.713 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.713 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:57:45.713 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.230199+0000", 2024-02-09T17:57:45.713 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:45.713 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.713 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.713 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:57:45.713 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.230199+0000", 2024-02-09T17:57:45.713 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:45.713 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.713 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.713 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:57:45.713 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.230200+0000", 2024-02-09T17:57:45.714 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.0599999999999999e-07 2024-02-09T17:57:45.714 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.714 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.714 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:57:45.714 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.230205+0000", 2024-02-09T17:57:45.714 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.4970000000000001e-06 2024-02-09T17:57:45.714 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.714 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.714 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:57:45.714 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.230213+0000", 2024-02-09T17:57:45.714 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.3380000000000005e-06 2024-02-09T17:57:45.714 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.714 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.715 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:57:45.715 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.230219+0000", 2024-02-09T17:57:45.715 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.9000000000000003e-06 2024-02-09T17:57:45.715 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.715 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.715 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:57:45.715 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.230262+0000", 2024-02-09T17:57:45.715 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2765000000000001e-05 2024-02-09T17:57:45.715 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.715 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.715 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:57:45.715 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.230281+0000", 2024-02-09T17:57:45.715 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.8941999999999999e-05 2024-02-09T17:57:45.715 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.716 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.716 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:57:45.716 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.399094+0000", 2024-02-09T17:57:45.716 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.168813357 2024-02-09T17:57:45.716 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.716 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.716 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:57:45.716 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.399108+0000", 2024-02-09T17:57:45.716 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3900000000000001e-05 2024-02-09T17:57:45.716 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:45.716 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:45.716 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:45.716 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.716 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.717 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.26305.0:164177 8.b e828/785 8:d8ae48a4:::benchmark_data_smithi179_148205_object41105:head v 828'10146, mlcod=828'10142)", 2024-02-09T17:57:45.717 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:57:44.235456+0000", 2024-02-09T17:57:45.717 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 1.0589517850000001, 2024-02-09T17:57:45.717 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.16367093999999999, 2024-02-09T17:57:45.717 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:57:45.717 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:57:45.717 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:57:45.717 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.717 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:57:45.717 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.235456+0000", 2024-02-09T17:57:45.717 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:45.717 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.717 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.717 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:57:45.718 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.235456+0000", 2024-02-09T17:57:45.718 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:45.718 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.718 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.718 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:57:45.718 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.235456+0000", 2024-02-09T17:57:45.718 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3399999999999999e-07 2024-02-09T17:57:45.718 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.718 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.718 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:57:45.718 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.235460+0000", 2024-02-09T17:57:45.718 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.3019999999999996e-06 2024-02-09T17:57:45.718 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.718 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.718 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:57:45.718 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.235469+0000", 2024-02-09T17:57:45.719 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.4940000000000003e-06 2024-02-09T17:57:45.719 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.719 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.719 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:57:45.719 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.235473+0000", 2024-02-09T17:57:45.719 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.1910000000000002e-06 2024-02-09T17:57:45.719 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.719 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.719 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:57:45.719 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.235512+0000", 2024-02-09T17:57:45.719 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.8494000000000001e-05 2024-02-09T17:57:45.719 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.719 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.719 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:57:45.719 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.235527+0000", 2024-02-09T17:57:45.720 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5786000000000001e-05 2024-02-09T17:57:45.720 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.720 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.720 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:57:45.720 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.399112+0000", 2024-02-09T17:57:45.720 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.16358472900000001 2024-02-09T17:57:45.720 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.720 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.720 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:57:45.720 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.399127+0000", 2024-02-09T17:57:45.720 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4409999999999999e-05 2024-02-09T17:57:45.720 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:45.720 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:45.720 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:45.721 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.721 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.721 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.26305.0:164183 8.3 e828/785 8:c76f053f:::benchmark_data_smithi179_148205_object41111:head v 828'10318, mlcod=828'10314)", 2024-02-09T17:57:45.721 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:57:44.240476+0000", 2024-02-09T17:57:45.721 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 1.0539310909999999, 2024-02-09T17:57:45.721 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.15866620400000001, 2024-02-09T17:57:45.721 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:57:45.721 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:57:45.721 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:57:45.721 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.721 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:57:45.721 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.240476+0000", 2024-02-09T17:57:45.721 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:45.721 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.722 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.722 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:57:45.722 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.240476+0000", 2024-02-09T17:57:45.722 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:57:45.722 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.722 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.722 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:57:45.722 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.240477+0000", 2024-02-09T17:57:45.722 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6899999999999997e-07 2024-02-09T17:57:45.722 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.722 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.722 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:57:45.722 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.240482+0000", 2024-02-09T17:57:45.722 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3909999999999997e-06 2024-02-09T17:57:45.722 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.723 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.723 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:57:45.723 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.240493+0000", 2024-02-09T17:57:45.723 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1022999999999999e-05 2024-02-09T17:57:45.723 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.723 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.723 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:57:45.723 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.240500+0000", 2024-02-09T17:57:45.723 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.0690000000000004e-06 2024-02-09T17:57:45.723 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.723 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.723 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:57:45.723 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.240605+0000", 2024-02-09T17:57:45.723 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00010450600000000001 2024-02-09T17:57:45.724 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.724 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.724 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:57:45.724 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.240623+0000", 2024-02-09T17:57:45.724 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.8559e-05 2024-02-09T17:57:45.724 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.724 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.724 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:57:45.724 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.399129+0000", 2024-02-09T17:57:45.724 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.15850510500000001 2024-02-09T17:57:45.724 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:45.724 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:45.724 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:57:45.724 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:44.399143+0000", 2024-02-09T17:57:45.724 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3981999999999999e-05 2024-02-09T17:57:45.725 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:45.725 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:45.725 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:45.725 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:45.725 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:45.725 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:57:45.726 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:45 smithi162 ceph-mon[44304]: 1.13 scrub starts 2024-02-09T17:57:45.726 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:45 smithi162 ceph-mon[44304]: 1.2 scrub starts 2024-02-09T17:57:45.726 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:45 smithi162 ceph-mon[44304]: 1.13 scrub ok 2024-02-09T17:57:45.726 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:45 smithi162 ceph-mon[44304]: 1.2 scrub ok 2024-02-09T17:57:45.726 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:45 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:57:45.726 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:45 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:57:45.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:45.727 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:57:45.727 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:57:45.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:45 smithi099 ceph-mon[39420]: 1.13 scrub starts 2024-02-09T17:57:45.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:45 smithi099 ceph-mon[39420]: 1.2 scrub starts 2024-02-09T17:57:45.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:45 smithi099 ceph-mon[39420]: 1.13 scrub ok 2024-02-09T17:57:45.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:45 smithi099 ceph-mon[39420]: 1.2 scrub ok 2024-02-09T17:57:45.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:45 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:57:45.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:45 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:57:45.828 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:57:45.867 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:45.898 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:57:45.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:45.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:57:46.021 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:57:46.052 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:46.073 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:57:46.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:46.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:57:46.179 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:57:46.207 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:46.228 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:57:46.234 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:46.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:57:46.335 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:57:46.364 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:46.367 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is set 2024-02-09T17:57:46.389 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:57:46.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:46.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:57:46.493 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:57:46.559 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-09T17:57:46.565 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:46.587 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:46 smithi099 ceph-mon[39420]: pgmap v2058: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 5.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 791 op/s; 15771/249489 objects degraded (6.321%) 2024-02-09T17:57:46.587 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:46 smithi099 ceph-mon[39420]: osdmap e829: 12 total, 11 up, 12 in 2024-02-09T17:57:46.587 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:46 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2725949199' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:57:46.587 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:46 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:57:46.593 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:57:46.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:46.601 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:46 smithi179 ceph-mon[43975]: pgmap v2058: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 5.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 791 op/s; 15771/249489 objects degraded (6.321%) 2024-02-09T17:57:46.601 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:46 smithi179 ceph-mon[43975]: osdmap e829: 12 total, 11 up, 12 in 2024-02-09T17:57:46.601 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:46 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2725949199' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:57:46.601 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:46 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:57:46.601 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:57:46.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:46 smithi162 ceph-mon[44304]: pgmap v2058: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 5.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 791 op/s; 15771/249489 objects degraded (6.321%) 2024-02-09T17:57:46.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:46 smithi162 ceph-mon[44304]: osdmap e829: 12 total, 11 up, 12 in 2024-02-09T17:57:46.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:46 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2725949199' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:57:46.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:46 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:57:46.702 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:57:46.740 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:46.953 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:57:46.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:46.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:57:47.060 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:57:47.100 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:47.536 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:57:47.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:47.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:57:47.647 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:57:47.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:47 smithi179 ceph-mon[43975]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:57:47.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:47 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:57:47.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:47 smithi179 ceph-mon[43975]: osdmap e830: 12 total, 11 up, 12 in 2024-02-09T17:57:47.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:47 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:57:47.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:47 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:57:47.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:47 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 15771/249489 objects degraded (6.321%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:57:47.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:47 smithi162 ceph-mon[44304]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:57:47.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:47 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:57:47.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:47 smithi162 ceph-mon[44304]: osdmap e830: 12 total, 11 up, 12 in 2024-02-09T17:57:47.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:47 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:57:47.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:47 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:57:47.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:47 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 15771/249489 objects degraded (6.321%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:57:47.692 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:47.722 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:57:47.731 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:47.732 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:57:47.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:47 smithi099 ceph-mon[39420]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:57:47.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:47 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:57:47.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:47 smithi099 ceph-mon[39420]: osdmap e830: 12 total, 11 up, 12 in 2024-02-09T17:57:47.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:47 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:57:47.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:47 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:57:47.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:47 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 15771/249489 objects degraded (6.321%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:57:47.833 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:57:47.894 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:47.895 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:57:47.925 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:57:47.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:47.932 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:57:48.033 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:57:48.064 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:48.603 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:48 smithi099 ceph-mon[39420]: pgmap v2061: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 804 op/s; 15012/243696 objects degraded (6.160%) 2024-02-09T17:57:48.604 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:48 smithi099 ceph-mon[39420]: osdmap e831: 12 total, 11 up, 12 in 2024-02-09T17:57:48.607 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:57:48.616 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:57:48.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:48.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:57:48.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:48 smithi179 ceph-mon[43975]: pgmap v2061: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 804 op/s; 15012/243696 objects degraded (6.160%) 2024-02-09T17:57:48.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:48 smithi179 ceph-mon[43975]: osdmap e831: 12 total, 11 up, 12 in 2024-02-09T17:57:48.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:48 smithi162 ceph-mon[44304]: pgmap v2061: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 804 op/s; 15012/243696 objects degraded (6.160%) 2024-02-09T17:57:48.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:48 smithi162 ceph-mon[44304]: osdmap e831: 12 total, 11 up, 12 in 2024-02-09T17:57:48.723 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:57:48.751 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:48.772 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:57:48.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:48.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:57:48.877 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:57:48.927 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:48.977 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:57:48.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:48.987 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:57:49.088 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:57:49.117 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:49.137 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:57:49.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:49.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:57:49.243 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:57:49.271 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:49.295 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:57:49.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:49.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:57:49.401 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:57:49.446 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:49.477 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:57:49.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:49.484 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:57:49.547 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-09T17:57:49.558 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:57:49.558 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:57:49.558 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 831, 2024-02-09T17:57:49.558 INFO:teuthology.orchestra.run.smithi099.stdout: "fsid": "a69c870a-c76e-11ee-95b7-87774f69a715", 2024-02-09T17:57:49.558 INFO:teuthology.orchestra.run.smithi099.stdout: "created": "2024-02-09T17:15:26.113606+0000", 2024-02-09T17:57:49.558 INFO:teuthology.orchestra.run.smithi099.stdout: "modified": "2024-02-09T17:57:47.345312+0000", 2024-02-09T17:57:49.558 INFO:teuthology.orchestra.run.smithi099.stdout: "last_up_change": "2024-02-09T17:56:39.428689+0000", 2024-02-09T17:57:49.558 INFO:teuthology.orchestra.run.smithi099.stdout: "last_in_change": "2024-02-09T17:43:24.138763+0000", 2024-02-09T17:57:49.558 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-09T17:57:49.558 INFO:teuthology.orchestra.run.smithi099.stdout: "flags_num": 5801984, 2024-02-09T17:57:49.559 INFO:teuthology.orchestra.run.smithi099.stdout: "flags_set": [ 2024-02-09T17:57:49.559 INFO:teuthology.orchestra.run.smithi099.stdout: "noscrub", 2024-02-09T17:57:49.559 INFO:teuthology.orchestra.run.smithi099.stdout: "pglog_hardlimit", 2024-02-09T17:57:49.559 INFO:teuthology.orchestra.run.smithi099.stdout: "purged_snapdirs", 2024-02-09T17:57:49.559 INFO:teuthology.orchestra.run.smithi099.stdout: "recovery_deletes", 2024-02-09T17:57:49.559 INFO:teuthology.orchestra.run.smithi099.stdout: "sortbitwise" 2024-02-09T17:57:49.559 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T17:57:49.559 INFO:teuthology.orchestra.run.smithi099.stdout: "crush_version": 27, 2024-02-09T17:57:49.559 INFO:teuthology.orchestra.run.smithi099.stdout: "full_ratio": 0.94999998807907104, 2024-02-09T17:57:49.559 INFO:teuthology.orchestra.run.smithi099.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-09T17:57:49.559 INFO:teuthology.orchestra.run.smithi099.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-09T17:57:49.559 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_snapshot": "", 2024-02-09T17:57:49.559 INFO:teuthology.orchestra.run.smithi099.stdout: "pool_max": 8, 2024-02-09T17:57:49.560 INFO:teuthology.orchestra.run.smithi099.stdout: "max_osd": 12, 2024-02-09T17:57:49.560 INFO:teuthology.orchestra.run.smithi099.stdout: "require_min_compat_client": "luminous", 2024-02-09T17:57:49.560 INFO:teuthology.orchestra.run.smithi099.stdout: "min_compat_client": "luminous", 2024-02-09T17:57:49.560 INFO:teuthology.orchestra.run.smithi099.stdout: "require_osd_release": "squid", 2024-02-09T17:57:49.560 INFO:teuthology.orchestra.run.smithi099.stdout: "allow_crimson": false, 2024-02-09T17:57:49.560 INFO:teuthology.orchestra.run.smithi099.stdout: "pools": [ 2024-02-09T17:57:49.560 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.560 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 1, 2024-02-09T17:57:49.560 INFO:teuthology.orchestra.run.smithi099.stdout: "pool_name": ".mgr", 2024-02-09T17:57:49.560 INFO:teuthology.orchestra.run.smithi099.stdout: "create_time": "2024-02-09T17:18:56.552286+0000", 2024-02-09T17:57:49.560 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 1, 2024-02-09T17:57:49.560 INFO:teuthology.orchestra.run.smithi099.stdout: "flags_names": "hashpspool", 2024-02-09T17:57:49.560 INFO:teuthology.orchestra.run.smithi099.stdout: "type": 1, 2024-02-09T17:57:49.561 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-02-09T17:57:49.561 INFO:teuthology.orchestra.run.smithi099.stdout: "min_size": 2, 2024-02-09T17:57:49.561 INFO:teuthology.orchestra.run.smithi099.stdout: "crush_rule": 0, 2024-02-09T17:57:49.561 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_count": 0, 2024-02-09T17:57:49.561 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_target": 0, 2024-02-09T17:57:49.561 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_barrier": 0, 2024-02-09T17:57:49.561 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-09T17:57:49.561 INFO:teuthology.orchestra.run.smithi099.stdout: "object_hash": 2, 2024-02-09T17:57:49.561 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_autoscale_mode": "off", 2024-02-09T17:57:49.561 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num": 31, 2024-02-09T17:57:49.561 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_placement_num": 31, 2024-02-09T17:57:49.561 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_placement_num_target": 31, 2024-02-09T17:57:49.561 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num_target": 31, 2024-02-09T17:57:49.561 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num_pending": 31, 2024-02-09T17:57:49.562 INFO:teuthology.orchestra.run.smithi099.stdout: "last_pg_merge_meta": { 2024-02-09T17:57:49.562 INFO:teuthology.orchestra.run.smithi099.stdout: "source_pgid": "0.0", 2024-02-09T17:57:49.562 INFO:teuthology.orchestra.run.smithi099.stdout: "ready_epoch": 0, 2024-02-09T17:57:49.562 INFO:teuthology.orchestra.run.smithi099.stdout: "last_epoch_started": 0, 2024-02-09T17:57:49.562 INFO:teuthology.orchestra.run.smithi099.stdout: "last_epoch_clean": 0, 2024-02-09T17:57:49.562 INFO:teuthology.orchestra.run.smithi099.stdout: "source_version": "0'0", 2024-02-09T17:57:49.562 INFO:teuthology.orchestra.run.smithi099.stdout: "target_version": "0'0" 2024-02-09T17:57:49.562 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.562 INFO:teuthology.orchestra.run.smithi099.stdout: "last_change": "544", 2024-02-09T17:57:49.562 INFO:teuthology.orchestra.run.smithi099.stdout: "last_force_op_resend": "0", 2024-02-09T17:57:49.562 INFO:teuthology.orchestra.run.smithi099.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-09T17:57:49.562 INFO:teuthology.orchestra.run.smithi099.stdout: "last_force_op_resend_preluminous": "514", 2024-02-09T17:57:49.562 INFO:teuthology.orchestra.run.smithi099.stdout: "auid": 0, 2024-02-09T17:57:49.562 INFO:teuthology.orchestra.run.smithi099.stdout: "snap_mode": "selfmanaged", 2024-02-09T17:57:49.562 INFO:teuthology.orchestra.run.smithi099.stdout: "snap_seq": 0, 2024-02-09T17:57:49.563 INFO:teuthology.orchestra.run.smithi099.stdout: "snap_epoch": 0, 2024-02-09T17:57:49.563 INFO:teuthology.orchestra.run.smithi099.stdout: "pool_snaps": [], 2024-02-09T17:57:49.563 INFO:teuthology.orchestra.run.smithi099.stdout: "removed_snaps": "[]", 2024-02-09T17:57:49.563 INFO:teuthology.orchestra.run.smithi099.stdout: "quota_max_bytes": 0, 2024-02-09T17:57:49.563 INFO:teuthology.orchestra.run.smithi099.stdout: "quota_max_objects": 0, 2024-02-09T17:57:49.563 INFO:teuthology.orchestra.run.smithi099.stdout: "tiers": [], 2024-02-09T17:57:49.563 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_of": -1, 2024-02-09T17:57:49.563 INFO:teuthology.orchestra.run.smithi099.stdout: "read_tier": -1, 2024-02-09T17:57:49.563 INFO:teuthology.orchestra.run.smithi099.stdout: "write_tier": -1, 2024-02-09T17:57:49.563 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_mode": "none", 2024-02-09T17:57:49.563 INFO:teuthology.orchestra.run.smithi099.stdout: "target_max_bytes": 0, 2024-02-09T17:57:49.563 INFO:teuthology.orchestra.run.smithi099.stdout: "target_max_objects": 0, 2024-02-09T17:57:49.563 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-09T17:57:49.563 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-09T17:57:49.564 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-09T17:57:49.564 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_min_flush_age": 0, 2024-02-09T17:57:49.564 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_min_evict_age": 0, 2024-02-09T17:57:49.564 INFO:teuthology.orchestra.run.smithi099.stdout: "erasure_code_profile": "", 2024-02-09T17:57:49.564 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_params": { 2024-02-09T17:57:49.564 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "none" 2024-02-09T17:57:49.564 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.564 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_period": 0, 2024-02-09T17:57:49.564 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_count": 0, 2024-02-09T17:57:49.564 INFO:teuthology.orchestra.run.smithi099.stdout: "use_gmt_hitset": true, 2024-02-09T17:57:49.564 INFO:teuthology.orchestra.run.smithi099.stdout: "min_read_recency_for_promote": 0, 2024-02-09T17:57:49.564 INFO:teuthology.orchestra.run.smithi099.stdout: "min_write_recency_for_promote": 0, 2024-02-09T17:57:49.564 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_grade_decay_rate": 0, 2024-02-09T17:57:49.564 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_search_last_n": 0, 2024-02-09T17:57:49.564 INFO:teuthology.orchestra.run.smithi099.stdout: "grade_table": [], 2024-02-09T17:57:49.565 INFO:teuthology.orchestra.run.smithi099.stdout: "stripe_width": 0, 2024-02-09T17:57:49.565 INFO:teuthology.orchestra.run.smithi099.stdout: "expected_num_objects": 0, 2024-02-09T17:57:49.565 INFO:teuthology.orchestra.run.smithi099.stdout: "fast_read": false, 2024-02-09T17:57:49.565 INFO:teuthology.orchestra.run.smithi099.stdout: "options": { 2024-02-09T17:57:49.565 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num_max": 32, 2024-02-09T17:57:49.565 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num_min": 1 2024-02-09T17:57:49.565 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.565 INFO:teuthology.orchestra.run.smithi099.stdout: "application_metadata": { 2024-02-09T17:57:49.565 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": {} 2024-02-09T17:57:49.565 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.565 INFO:teuthology.orchestra.run.smithi099.stdout: "read_balance": { 2024-02-09T17:57:49.565 INFO:teuthology.orchestra.run.smithi099.stdout: "score_type": "Fair distribution", 2024-02-09T17:57:49.565 INFO:teuthology.orchestra.run.smithi099.stdout: "score_acting": 7.119999885559082, 2024-02-09T17:57:49.565 INFO:teuthology.orchestra.run.smithi099.stdout: "score_stable": 2.130000114440918, 2024-02-09T17:57:49.566 INFO:teuthology.orchestra.run.smithi099.stdout: "optimal_score": 1, 2024-02-09T17:57:49.566 INFO:teuthology.orchestra.run.smithi099.stdout: "raw_score_acting": 7.119999885559082, 2024-02-09T17:57:49.566 INFO:teuthology.orchestra.run.smithi099.stdout: "raw_score_stable": 2.130000114440918, 2024-02-09T17:57:49.566 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity_weighted": 0.78079599142074585, 2024-02-09T17:57:49.566 INFO:teuthology.orchestra.run.smithi099.stdout: "average_primary_affinity": 0.85000002384185791, 2024-02-09T17:57:49.566 INFO:teuthology.orchestra.run.smithi099.stdout: "average_primary_affinity_weighted": 0.77999997138977051, 2024-02-09T17:57:49.566 INFO:teuthology.orchestra.run.smithi099.stdout: "error_message": "pool 1 has acting primaries on OSD(s) with primary affinity 0, read balance score is not accurate" 2024-02-09T17:57:49.566 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.566 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.566 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.566 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 8, 2024-02-09T17:57:49.566 INFO:teuthology.orchestra.run.smithi099.stdout: "pool_name": "unique_pool_6", 2024-02-09T17:57:49.566 INFO:teuthology.orchestra.run.smithi099.stdout: "create_time": "2024-02-09T17:55:15.183490+0000", 2024-02-09T17:57:49.566 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 1, 2024-02-09T17:57:49.566 INFO:teuthology.orchestra.run.smithi099.stdout: "flags_names": "hashpspool", 2024-02-09T17:57:49.567 INFO:teuthology.orchestra.run.smithi099.stdout: "type": 1, 2024-02-09T17:57:49.567 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-02-09T17:57:49.567 INFO:teuthology.orchestra.run.smithi099.stdout: "min_size": 2, 2024-02-09T17:57:49.567 INFO:teuthology.orchestra.run.smithi099.stdout: "crush_rule": 0, 2024-02-09T17:57:49.567 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_count": 0, 2024-02-09T17:57:49.567 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_target": 0, 2024-02-09T17:57:49.567 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_barrier": 0, 2024-02-09T17:57:49.567 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-09T17:57:49.567 INFO:teuthology.orchestra.run.smithi099.stdout: "object_hash": 2, 2024-02-09T17:57:49.567 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_autoscale_mode": "off", 2024-02-09T17:57:49.567 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num": 16, 2024-02-09T17:57:49.567 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_placement_num": 16, 2024-02-09T17:57:49.567 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_placement_num_target": 16, 2024-02-09T17:57:49.567 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num_target": 16, 2024-02-09T17:57:49.567 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num_pending": 16, 2024-02-09T17:57:49.568 INFO:teuthology.orchestra.run.smithi099.stdout: "last_pg_merge_meta": { 2024-02-09T17:57:49.568 INFO:teuthology.orchestra.run.smithi099.stdout: "source_pgid": "0.0", 2024-02-09T17:57:49.568 INFO:teuthology.orchestra.run.smithi099.stdout: "ready_epoch": 0, 2024-02-09T17:57:49.568 INFO:teuthology.orchestra.run.smithi099.stdout: "last_epoch_started": 0, 2024-02-09T17:57:49.568 INFO:teuthology.orchestra.run.smithi099.stdout: "last_epoch_clean": 0, 2024-02-09T17:57:49.568 INFO:teuthology.orchestra.run.smithi099.stdout: "source_version": "0'0", 2024-02-09T17:57:49.568 INFO:teuthology.orchestra.run.smithi099.stdout: "target_version": "0'0" 2024-02-09T17:57:49.568 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.568 INFO:teuthology.orchestra.run.smithi099.stdout: "last_change": "793", 2024-02-09T17:57:49.568 INFO:teuthology.orchestra.run.smithi099.stdout: "last_force_op_resend": "0", 2024-02-09T17:57:49.568 INFO:teuthology.orchestra.run.smithi099.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-09T17:57:49.568 INFO:teuthology.orchestra.run.smithi099.stdout: "last_force_op_resend_preluminous": "0", 2024-02-09T17:57:49.568 INFO:teuthology.orchestra.run.smithi099.stdout: "auid": 0, 2024-02-09T17:57:49.568 INFO:teuthology.orchestra.run.smithi099.stdout: "snap_mode": "selfmanaged", 2024-02-09T17:57:49.568 INFO:teuthology.orchestra.run.smithi099.stdout: "snap_seq": 0, 2024-02-09T17:57:49.569 INFO:teuthology.orchestra.run.smithi099.stdout: "snap_epoch": 0, 2024-02-09T17:57:49.569 INFO:teuthology.orchestra.run.smithi099.stdout: "pool_snaps": [], 2024-02-09T17:57:49.569 INFO:teuthology.orchestra.run.smithi099.stdout: "removed_snaps": "[]", 2024-02-09T17:57:49.569 INFO:teuthology.orchestra.run.smithi099.stdout: "quota_max_bytes": 0, 2024-02-09T17:57:49.569 INFO:teuthology.orchestra.run.smithi099.stdout: "quota_max_objects": 0, 2024-02-09T17:57:49.569 INFO:teuthology.orchestra.run.smithi099.stdout: "tiers": [], 2024-02-09T17:57:49.569 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_of": -1, 2024-02-09T17:57:49.569 INFO:teuthology.orchestra.run.smithi099.stdout: "read_tier": -1, 2024-02-09T17:57:49.569 INFO:teuthology.orchestra.run.smithi099.stdout: "write_tier": -1, 2024-02-09T17:57:49.569 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_mode": "none", 2024-02-09T17:57:49.569 INFO:teuthology.orchestra.run.smithi099.stdout: "target_max_bytes": 0, 2024-02-09T17:57:49.569 INFO:teuthology.orchestra.run.smithi099.stdout: "target_max_objects": 0, 2024-02-09T17:57:49.569 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-09T17:57:49.569 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-09T17:57:49.569 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-09T17:57:49.570 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_min_flush_age": 0, 2024-02-09T17:57:49.570 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_min_evict_age": 0, 2024-02-09T17:57:49.570 INFO:teuthology.orchestra.run.smithi099.stdout: "erasure_code_profile": "", 2024-02-09T17:57:49.570 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_params": { 2024-02-09T17:57:49.570 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "none" 2024-02-09T17:57:49.570 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.570 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_period": 0, 2024-02-09T17:57:49.570 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_count": 0, 2024-02-09T17:57:49.570 INFO:teuthology.orchestra.run.smithi099.stdout: "use_gmt_hitset": true, 2024-02-09T17:57:49.570 INFO:teuthology.orchestra.run.smithi099.stdout: "min_read_recency_for_promote": 0, 2024-02-09T17:57:49.570 INFO:teuthology.orchestra.run.smithi099.stdout: "min_write_recency_for_promote": 0, 2024-02-09T17:57:49.570 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_grade_decay_rate": 0, 2024-02-09T17:57:49.570 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_search_last_n": 0, 2024-02-09T17:57:49.570 INFO:teuthology.orchestra.run.smithi099.stdout: "grade_table": [], 2024-02-09T17:57:49.571 INFO:teuthology.orchestra.run.smithi099.stdout: "stripe_width": 0, 2024-02-09T17:57:49.571 INFO:teuthology.orchestra.run.smithi099.stdout: "expected_num_objects": 0, 2024-02-09T17:57:49.571 INFO:teuthology.orchestra.run.smithi099.stdout: "fast_read": false, 2024-02-09T17:57:49.571 INFO:teuthology.orchestra.run.smithi099.stdout: "options": {}, 2024-02-09T17:57:49.571 INFO:teuthology.orchestra.run.smithi099.stdout: "application_metadata": { 2024-02-09T17:57:49.571 INFO:teuthology.orchestra.run.smithi099.stdout: "rados": {} 2024-02-09T17:57:49.571 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.571 INFO:teuthology.orchestra.run.smithi099.stdout: "read_balance": { 2024-02-09T17:57:49.571 INFO:teuthology.orchestra.run.smithi099.stdout: "score_type": "Fair distribution", 2024-02-09T17:57:49.571 INFO:teuthology.orchestra.run.smithi099.stdout: "score_acting": 2.75, 2024-02-09T17:57:49.571 INFO:teuthology.orchestra.run.smithi099.stdout: "score_stable": 2.75, 2024-02-09T17:57:49.571 INFO:teuthology.orchestra.run.smithi099.stdout: "optimal_score": 0.9100000262260437, 2024-02-09T17:57:49.571 INFO:teuthology.orchestra.run.smithi099.stdout: "raw_score_acting": 2.5, 2024-02-09T17:57:49.571 INFO:teuthology.orchestra.run.smithi099.stdout: "raw_score_stable": 2.5, 2024-02-09T17:57:49.572 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity_weighted": 0.78079599142074585, 2024-02-09T17:57:49.572 INFO:teuthology.orchestra.run.smithi099.stdout: "average_primary_affinity": 0.85000002384185791, 2024-02-09T17:57:49.572 INFO:teuthology.orchestra.run.smithi099.stdout: "average_primary_affinity_weighted": 0.77999997138977051 2024-02-09T17:57:49.572 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.572 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.572 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T17:57:49.572 INFO:teuthology.orchestra.run.smithi099.stdout: "osds": [ 2024-02-09T17:57:49.572 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.572 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 0, 2024-02-09T17:57:49.572 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "21739aa7-e278-45da-9a3f-867385f01baf", 2024-02-09T17:57:49.572 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:57:49.572 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:57:49.572 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:57:49.572 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 0.609619140625, 2024-02-09T17:57:49.572 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:57:49.573 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:57:49.573 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 8, 2024-02-09T17:57:49.573 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 785, 2024-02-09T17:57:49.573 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:57:49.573 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:57:49.573 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:57:49.573 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:57:49.573 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.573 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:57:49.573 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6802", 2024-02-09T17:57:49.573 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T17:57:49.573 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.573 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.573 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:57:49.574 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6803", 2024-02-09T17:57:49.574 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T17:57:49.574 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.574 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.574 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.574 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:57:49.574 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:57:49.574 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.574 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:57:49.574 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6804", 2024-02-09T17:57:49.574 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T17:57:49.574 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.574 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.575 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:57:49.575 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6805", 2024-02-09T17:57:49.575 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T17:57:49.575 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.575 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.575 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.575 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:57:49.575 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:57:49.575 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.575 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:57:49.575 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6808", 2024-02-09T17:57:49.575 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T17:57:49.575 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.576 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.576 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:57:49.576 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6809", 2024-02-09T17:57:49.576 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T17:57:49.576 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.576 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.576 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.576 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:57:49.576 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:57:49.576 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.576 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:57:49.576 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6806", 2024-02-09T17:57:49.576 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T17:57:49.576 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.577 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.577 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:57:49.577 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6807", 2024-02-09T17:57:49.577 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T17:57:49.577 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.577 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.577 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.577 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.99:6803/3859868923", 2024-02-09T17:57:49.577 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.99:6805/3859868923", 2024-02-09T17:57:49.577 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.99:6809/3859868923", 2024-02-09T17:57:49.577 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.99:6807/3859868923", 2024-02-09T17:57:49.577 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:57:49.577 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:57:49.577 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:57:49.578 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.578 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.578 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.578 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 1, 2024-02-09T17:57:49.578 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "8eab347c-9065-4de1-93ae-e2eb7e11cab7", 2024-02-09T17:57:49.578 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:57:49.578 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:57:49.578 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:57:49.578 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T17:57:49.578 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:57:49.578 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:57:49.578 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 13, 2024-02-09T17:57:49.578 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 815, 2024-02-09T17:57:49.578 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:57:49.578 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:57:49.578 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:57:49.579 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:57:49.579 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.579 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:57:49.579 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6810", 2024-02-09T17:57:49.579 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T17:57:49.579 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.579 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.579 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:57:49.579 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6811", 2024-02-09T17:57:49.579 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T17:57:49.579 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.579 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.579 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.579 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:57:49.579 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:57:49.579 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.580 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:57:49.580 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6812", 2024-02-09T17:57:49.580 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T17:57:49.580 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.580 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.580 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:57:49.580 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6813", 2024-02-09T17:57:49.580 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T17:57:49.580 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.580 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.580 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.580 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:57:49.580 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:57:49.580 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.581 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:57:49.581 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6816", 2024-02-09T17:57:49.581 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T17:57:49.581 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.581 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.581 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:57:49.581 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6817", 2024-02-09T17:57:49.581 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T17:57:49.581 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.581 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.581 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.581 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:57:49.581 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:57:49.582 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.582 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:57:49.582 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6814", 2024-02-09T17:57:49.582 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T17:57:49.582 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.582 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.582 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:57:49.582 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6815", 2024-02-09T17:57:49.582 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T17:57:49.582 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.582 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.582 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.582 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.99:6811/3868328452", 2024-02-09T17:57:49.582 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.99:6813/3868328452", 2024-02-09T17:57:49.582 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.99:6817/3868328452", 2024-02-09T17:57:49.582 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.99:6815/3868328452", 2024-02-09T17:57:49.583 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:57:49.583 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:57:49.583 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:57:49.583 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.583 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.583 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.583 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 2, 2024-02-09T17:57:49.583 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "575f4257-7e34-485e-963c-2607b69e1220", 2024-02-09T17:57:49.583 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:57:49.583 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:57:49.583 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:57:49.583 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 0.3601837158203125, 2024-02-09T17:57:49.583 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:57:49.583 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:57:49.584 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 18, 2024-02-09T17:57:49.584 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 661, 2024-02-09T17:57:49.584 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:57:49.584 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:57:49.584 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:57:49.584 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:57:49.584 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.584 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:57:49.584 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6818", 2024-02-09T17:57:49.584 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T17:57:49.584 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.584 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.584 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:57:49.584 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6819", 2024-02-09T17:57:49.585 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T17:57:49.585 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.585 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.585 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.585 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:57:49.585 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:57:49.585 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.585 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:57:49.585 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6820", 2024-02-09T17:57:49.585 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T17:57:49.585 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.585 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.585 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:57:49.585 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6821", 2024-02-09T17:57:49.585 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T17:57:49.586 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.586 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.586 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.586 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:57:49.586 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:57:49.586 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.586 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:57:49.586 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6824", 2024-02-09T17:57:49.586 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T17:57:49.586 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.586 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.586 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:57:49.586 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6825", 2024-02-09T17:57:49.586 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T17:57:49.586 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.587 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.587 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.587 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:57:49.587 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:57:49.587 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.587 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:57:49.587 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6822", 2024-02-09T17:57:49.587 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T17:57:49.587 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.587 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.587 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:57:49.587 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6823", 2024-02-09T17:57:49.587 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T17:57:49.587 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.587 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.588 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.588 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.99:6819/537506237", 2024-02-09T17:57:49.588 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.99:6821/537506237", 2024-02-09T17:57:49.588 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.99:6825/537506237", 2024-02-09T17:57:49.588 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.99:6823/537506237", 2024-02-09T17:57:49.588 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:57:49.588 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:57:49.588 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:57:49.588 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.588 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.588 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.588 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 3, 2024-02-09T17:57:49.588 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "1a520bcf-81bb-4f1d-80fb-11e98008e429", 2024-02-09T17:57:49.588 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 0, 2024-02-09T17:57:49.589 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:57:49.589 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:57:49.589 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 0.8014984130859375, 2024-02-09T17:57:49.589 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:57:49.589 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:57:49.589 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 27, 2024-02-09T17:57:49.589 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 785, 2024-02-09T17:57:49.589 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 815, 2024-02-09T17:57:49.589 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:57:49.589 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:57:49.589 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:57:49.589 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.589 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:57:49.589 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6826", 2024-02-09T17:57:49.589 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 762784793 2024-02-09T17:57:49.589 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.590 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.590 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:57:49.590 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6827", 2024-02-09T17:57:49.590 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 762784793 2024-02-09T17:57:49.590 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.590 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.590 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.590 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:57:49.590 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:57:49.590 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.590 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:57:49.590 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6828", 2024-02-09T17:57:49.590 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 762784793 2024-02-09T17:57:49.590 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.590 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.591 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:57:49.591 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6829", 2024-02-09T17:57:49.591 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 762784793 2024-02-09T17:57:49.591 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.591 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.591 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.591 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:57:49.591 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:57:49.591 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.591 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:57:49.591 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6832", 2024-02-09T17:57:49.591 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 762784793 2024-02-09T17:57:49.591 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.591 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.591 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:57:49.592 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6833", 2024-02-09T17:57:49.592 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 762784793 2024-02-09T17:57:49.592 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.592 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.592 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.592 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:57:49.592 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:57:49.592 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.592 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:57:49.592 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6830", 2024-02-09T17:57:49.592 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 762784793 2024-02-09T17:57:49.592 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.592 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.592 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:57:49.593 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6831", 2024-02-09T17:57:49.593 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 762784793 2024-02-09T17:57:49.593 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.593 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.593 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.593 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.99:6827/762784793", 2024-02-09T17:57:49.593 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.99:6829/762784793", 2024-02-09T17:57:49.593 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.99:6833/762784793", 2024-02-09T17:57:49.593 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.99:6831/762784793", 2024-02-09T17:57:49.593 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:57:49.593 INFO:teuthology.orchestra.run.smithi099.stdout: "exists" 2024-02-09T17:57:49.593 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.593 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.593 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.594 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 4, 2024-02-09T17:57:49.594 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "4617c2fe-c9d8-40cd-887b-303e3215bdf8", 2024-02-09T17:57:49.594 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:57:49.594 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:57:49.594 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:57:49.594 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T17:57:49.594 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:57:49.594 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:57:49.594 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 32, 2024-02-09T17:57:49.594 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 815, 2024-02-09T17:57:49.594 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:57:49.594 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:57:49.594 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:57:49.594 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:57:49.594 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.595 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:57:49.595 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6800", 2024-02-09T17:57:49.595 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T17:57:49.595 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.595 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.595 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:57:49.595 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6801", 2024-02-09T17:57:49.595 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T17:57:49.595 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.595 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.595 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.595 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:57:49.595 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:57:49.595 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.595 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:57:49.595 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6802", 2024-02-09T17:57:49.596 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T17:57:49.596 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.596 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.596 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:57:49.596 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6803", 2024-02-09T17:57:49.596 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T17:57:49.596 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.596 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.596 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.596 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:57:49.596 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:57:49.596 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.596 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:57:49.596 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6806", 2024-02-09T17:57:49.597 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T17:57:49.597 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.597 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.597 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:57:49.597 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6807", 2024-02-09T17:57:49.597 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T17:57:49.597 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.597 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.597 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.597 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:57:49.597 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:57:49.597 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.597 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:57:49.597 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6804", 2024-02-09T17:57:49.598 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T17:57:49.598 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.598 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.598 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:57:49.598 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6805", 2024-02-09T17:57:49.598 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T17:57:49.598 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.598 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.598 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.598 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.162:6801/4178676504", 2024-02-09T17:57:49.598 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.162:6803/4178676504", 2024-02-09T17:57:49.598 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.162:6807/4178676504", 2024-02-09T17:57:49.598 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.162:6805/4178676504", 2024-02-09T17:57:49.599 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:57:49.599 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:57:49.599 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:57:49.599 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.599 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.599 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.599 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 5, 2024-02-09T17:57:49.599 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "da98e3f3-173e-46fe-be8b-1ccbe8b0dabc", 2024-02-09T17:57:49.599 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:57:49.599 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:57:49.599 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:57:49.599 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 0, 2024-02-09T17:57:49.599 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:57:49.599 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:57:49.599 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 40, 2024-02-09T17:57:49.600 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 815, 2024-02-09T17:57:49.600 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:57:49.600 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:57:49.600 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:57:49.600 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:57:49.600 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.600 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:57:49.600 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6808", 2024-02-09T17:57:49.600 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T17:57:49.600 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.600 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.600 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:57:49.600 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6809", 2024-02-09T17:57:49.601 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T17:57:49.601 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.601 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.601 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.601 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:57:49.601 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:57:49.601 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.601 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:57:49.601 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6810", 2024-02-09T17:57:49.601 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T17:57:49.601 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.601 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.601 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:57:49.601 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6811", 2024-02-09T17:57:49.601 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T17:57:49.602 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.602 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.602 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.602 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:57:49.602 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:57:49.602 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.602 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:57:49.602 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6814", 2024-02-09T17:57:49.602 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T17:57:49.602 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.602 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.602 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:57:49.602 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6815", 2024-02-09T17:57:49.603 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T17:57:49.603 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.603 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.603 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.603 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:57:49.603 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:57:49.603 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.603 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:57:49.603 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6812", 2024-02-09T17:57:49.603 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T17:57:49.603 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.603 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.603 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:57:49.604 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6813", 2024-02-09T17:57:49.604 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T17:57:49.604 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.604 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.604 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.604 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.162:6809/1489371339", 2024-02-09T17:57:49.604 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.162:6811/1489371339", 2024-02-09T17:57:49.604 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.162:6815/1489371339", 2024-02-09T17:57:49.604 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.162:6813/1489371339", 2024-02-09T17:57:49.604 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:57:49.604 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:57:49.604 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:57:49.604 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.605 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.605 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.605 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 6, 2024-02-09T17:57:49.605 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "ff50382d-5ddf-4cda-bb0c-ff055feeaa40", 2024-02-09T17:57:49.605 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:57:49.605 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:57:49.605 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:57:49.605 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T17:57:49.605 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:57:49.605 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:57:49.605 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 49, 2024-02-09T17:57:49.605 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 815, 2024-02-09T17:57:49.605 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:57:49.605 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:57:49.606 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:57:49.606 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:57:49.606 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.606 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:57:49.606 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6816", 2024-02-09T17:57:49.606 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T17:57:49.606 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.606 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.606 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:57:49.606 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6817", 2024-02-09T17:57:49.606 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T17:57:49.606 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.606 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.606 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.607 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:57:49.607 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:57:49.607 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.607 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:57:49.607 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6818", 2024-02-09T17:57:49.607 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T17:57:49.607 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.607 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.607 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:57:49.607 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6819", 2024-02-09T17:57:49.607 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T17:57:49.607 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.607 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.607 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.607 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:57:49.608 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:57:49.608 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.608 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:57:49.608 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6822", 2024-02-09T17:57:49.608 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T17:57:49.608 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.608 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.608 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:57:49.608 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6823", 2024-02-09T17:57:49.608 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T17:57:49.608 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.608 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.608 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.608 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:57:49.608 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:57:49.609 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.609 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:57:49.609 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6820", 2024-02-09T17:57:49.609 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T17:57:49.609 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.609 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.609 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:57:49.609 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6821", 2024-02-09T17:57:49.609 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T17:57:49.609 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.609 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.609 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.609 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.162:6817/821169663", 2024-02-09T17:57:49.609 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.162:6819/821169663", 2024-02-09T17:57:49.610 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.162:6823/821169663", 2024-02-09T17:57:49.610 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.162:6821/821169663", 2024-02-09T17:57:49.610 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:57:49.610 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:57:49.610 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:57:49.610 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.610 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.610 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.610 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 7, 2024-02-09T17:57:49.610 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "a82984dc-5e22-424b-a446-ae2ee71eee19", 2024-02-09T17:57:49.610 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:57:49.610 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:57:49.610 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:57:49.610 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 0.5982513427734375, 2024-02-09T17:57:49.611 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:57:49.611 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:57:49.611 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 57, 2024-02-09T17:57:49.611 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 661, 2024-02-09T17:57:49.611 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:57:49.611 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:57:49.611 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:57:49.611 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:57:49.611 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.611 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:57:49.611 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6824", 2024-02-09T17:57:49.611 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T17:57:49.611 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.611 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.611 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:57:49.612 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6825", 2024-02-09T17:57:49.612 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T17:57:49.612 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.612 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.612 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.612 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:57:49.612 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:57:49.612 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.612 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:57:49.612 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6826", 2024-02-09T17:57:49.612 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T17:57:49.612 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.612 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.612 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:57:49.612 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6827", 2024-02-09T17:57:49.613 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T17:57:49.613 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.613 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.613 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.613 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:57:49.613 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:57:49.613 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.613 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:57:49.613 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6830", 2024-02-09T17:57:49.613 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T17:57:49.613 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.613 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.613 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:57:49.613 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6831", 2024-02-09T17:57:49.613 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T17:57:49.614 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.614 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.614 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.614 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:57:49.614 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:57:49.614 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.614 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:57:49.614 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6828", 2024-02-09T17:57:49.614 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T17:57:49.614 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.614 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.614 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:57:49.614 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6829", 2024-02-09T17:57:49.614 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T17:57:49.614 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.615 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.615 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.615 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.162:6825/965264369", 2024-02-09T17:57:49.615 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.162:6827/965264369", 2024-02-09T17:57:49.615 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.162:6831/965264369", 2024-02-09T17:57:49.615 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.162:6829/965264369", 2024-02-09T17:57:49.615 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:57:49.615 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:57:49.615 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:57:49.615 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.615 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.615 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.615 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 8, 2024-02-09T17:57:49.615 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "63f9d79f-0400-44dc-b388-df00fefb5820", 2024-02-09T17:57:49.615 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:57:49.615 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:57:49.615 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:57:49.616 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T17:57:49.616 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 66, 2024-02-09T17:57:49.616 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 145, 2024-02-09T17:57:49.616 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 171, 2024-02-09T17:57:49.616 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 790, 2024-02-09T17:57:49.616 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 146, 2024-02-09T17:57:49.616 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:57:49.616 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:57:49.616 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:57:49.616 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.616 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:57:49.616 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6800", 2024-02-09T17:57:49.616 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T17:57:49.616 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.616 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.616 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:57:49.616 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6801", 2024-02-09T17:57:49.617 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T17:57:49.617 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.617 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.617 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.617 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:57:49.617 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:57:49.617 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.617 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:57:49.617 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6802", 2024-02-09T17:57:49.617 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T17:57:49.617 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.617 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.617 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:57:49.617 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6803", 2024-02-09T17:57:49.618 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T17:57:49.618 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.618 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.618 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.618 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:57:49.618 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:57:49.618 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.618 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:57:49.618 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6806", 2024-02-09T17:57:49.618 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T17:57:49.618 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.618 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.618 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:57:49.619 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6807", 2024-02-09T17:57:49.619 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T17:57:49.619 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.619 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.619 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.619 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:57:49.619 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:57:49.619 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.619 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:57:49.619 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6804", 2024-02-09T17:57:49.619 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T17:57:49.619 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.619 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.619 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:57:49.619 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6805", 2024-02-09T17:57:49.620 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T17:57:49.620 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.620 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.620 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.620 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.179:6801/2558294495", 2024-02-09T17:57:49.620 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.179:6803/2558294495", 2024-02-09T17:57:49.620 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.179:6807/2558294495", 2024-02-09T17:57:49.620 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.179:6805/2558294495", 2024-02-09T17:57:49.620 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:57:49.620 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:57:49.620 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:57:49.620 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.620 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.620 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.620 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 9, 2024-02-09T17:57:49.621 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "7d995854-7328-4dfc-b1a9-0edca8d74f46", 2024-02-09T17:57:49.621 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:57:49.621 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:57:49.621 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:57:49.621 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T17:57:49.621 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:57:49.621 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:57:49.621 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 77, 2024-02-09T17:57:49.621 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 815, 2024-02-09T17:57:49.621 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:57:49.621 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:57:49.621 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:57:49.621 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:57:49.621 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.621 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:57:49.622 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6808", 2024-02-09T17:57:49.622 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T17:57:49.622 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.622 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.622 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:57:49.622 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6809", 2024-02-09T17:57:49.622 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T17:57:49.622 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.622 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.622 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.622 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:57:49.622 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:57:49.622 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.622 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:57:49.622 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6810", 2024-02-09T17:57:49.622 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T17:57:49.623 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.623 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.623 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:57:49.623 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6811", 2024-02-09T17:57:49.623 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T17:57:49.623 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.623 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.623 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.623 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:57:49.623 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:57:49.623 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.623 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:57:49.623 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6814", 2024-02-09T17:57:49.623 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T17:57:49.624 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.624 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.624 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:57:49.624 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6815", 2024-02-09T17:57:49.624 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T17:57:49.624 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.624 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.624 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.624 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:57:49.624 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:57:49.624 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.624 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:57:49.624 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6812", 2024-02-09T17:57:49.624 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T17:57:49.624 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.625 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.625 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:57:49.625 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6813", 2024-02-09T17:57:49.625 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T17:57:49.625 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.625 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.625 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.625 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.179:6809/504427183", 2024-02-09T17:57:49.625 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.179:6811/504427183", 2024-02-09T17:57:49.625 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.179:6815/504427183", 2024-02-09T17:57:49.625 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.179:6813/504427183", 2024-02-09T17:57:49.625 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:57:49.625 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:57:49.625 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:57:49.625 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.626 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.626 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.626 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 10, 2024-02-09T17:57:49.626 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "151fbedd-08e6-46e6-a4e5-a14248399f61", 2024-02-09T17:57:49.626 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:57:49.626 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:57:49.626 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:57:49.626 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T17:57:49.626 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:57:49.626 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:57:49.626 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 84, 2024-02-09T17:57:49.626 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 790, 2024-02-09T17:57:49.626 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:57:49.626 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:57:49.626 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:57:49.627 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:57:49.627 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.627 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:57:49.627 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6816", 2024-02-09T17:57:49.627 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T17:57:49.627 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.627 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.627 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:57:49.627 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6817", 2024-02-09T17:57:49.627 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T17:57:49.627 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.627 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.627 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.627 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:57:49.627 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:57:49.628 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.628 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:57:49.628 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6818", 2024-02-09T17:57:49.628 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T17:57:49.628 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.628 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.628 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:57:49.628 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6819", 2024-02-09T17:57:49.628 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T17:57:49.628 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.628 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.628 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.628 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:57:49.628 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:57:49.628 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.629 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:57:49.629 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6822", 2024-02-09T17:57:49.629 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T17:57:49.629 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.629 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.629 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:57:49.629 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6823", 2024-02-09T17:57:49.629 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T17:57:49.629 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.629 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.629 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.629 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:57:49.629 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:57:49.629 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.629 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:57:49.630 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6820", 2024-02-09T17:57:49.630 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T17:57:49.630 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.630 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.630 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:57:49.630 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6821", 2024-02-09T17:57:49.630 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T17:57:49.630 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.630 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.630 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.630 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.179:6817/2403345164", 2024-02-09T17:57:49.630 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.179:6819/2403345164", 2024-02-09T17:57:49.630 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.179:6823/2403345164", 2024-02-09T17:57:49.630 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.179:6821/2403345164", 2024-02-09T17:57:49.630 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:57:49.631 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:57:49.631 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:57:49.631 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.631 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.631 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.631 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 11, 2024-02-09T17:57:49.631 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "6d3a3ad8-417a-483f-991c-52549dbe7d73", 2024-02-09T17:57:49.631 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T17:57:49.631 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T17:57:49.631 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T17:57:49.631 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T17:57:49.631 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T17:57:49.631 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T17:57:49.631 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 90, 2024-02-09T17:57:49.631 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 815, 2024-02-09T17:57:49.632 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T17:57:49.632 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T17:57:49.632 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T17:57:49.632 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:57:49.632 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.632 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:57:49.632 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6824", 2024-02-09T17:57:49.632 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T17:57:49.632 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.632 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.632 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:57:49.632 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6825", 2024-02-09T17:57:49.632 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T17:57:49.632 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.632 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.633 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.633 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T17:57:49.633 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:57:49.633 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.633 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:57:49.633 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6826", 2024-02-09T17:57:49.633 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T17:57:49.633 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.633 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.633 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:57:49.633 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6827", 2024-02-09T17:57:49.633 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T17:57:49.633 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.634 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.634 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.634 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T17:57:49.634 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:57:49.634 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.634 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:57:49.634 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6830", 2024-02-09T17:57:49.634 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T17:57:49.634 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.634 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.634 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:57:49.634 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6831", 2024-02-09T17:57:49.634 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T17:57:49.634 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.634 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.635 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.635 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T17:57:49.635 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T17:57:49.635 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.635 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T17:57:49.635 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6828", 2024-02-09T17:57:49.635 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T17:57:49.635 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.635 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.635 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T17:57:49.635 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6829", 2024-02-09T17:57:49.635 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T17:57:49.635 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.635 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.635 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.636 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.179:6825/2899402373", 2024-02-09T17:57:49.636 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.179:6827/2899402373", 2024-02-09T17:57:49.636 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.179:6831/2899402373", 2024-02-09T17:57:49.636 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.179:6829/2899402373", 2024-02-09T17:57:49.636 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T17:57:49.636 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T17:57:49.636 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T17:57:49.636 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.636 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.636 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T17:57:49.637 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_xinfo": [ 2024-02-09T17:57:49.637 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.637 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 0, 2024-02-09T17:57:49.637 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:57:49.637 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:57:49.637 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:57:49.637 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:57:49.637 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:57:49.637 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:56:33.967744+0000", 2024-02-09T17:57:49.637 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:57:49.637 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.637 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.637 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 1, 2024-02-09T17:57:49.637 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:57:49.638 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:57:49.638 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:57:49.638 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:57:49.638 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:57:49.638 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:56:53.308946+0000", 2024-02-09T17:57:49.638 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:57:49.638 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.638 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.638 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 2, 2024-02-09T17:57:49.638 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:57:49.638 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:57:49.638 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:57:49.638 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:57:49.638 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:57:49.639 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:57:44.470547+0000", 2024-02-09T17:57:49.639 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:57:49.639 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.639 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.639 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 3, 2024-02-09T17:57:49.639 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:57:49.639 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:57:49.639 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:57:49.639 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:57:49.639 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:57:49.639 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:55:39.048234+0000", 2024-02-09T17:57:49.639 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 814 2024-02-09T17:57:49.639 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.639 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.640 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 4, 2024-02-09T17:57:49.640 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:57:49.640 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:57:49.640 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:57:49.640 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:57:49.640 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:57:49.640 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:57:02.826463+0000", 2024-02-09T17:57:49.640 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:57:49.640 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.640 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.640 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 5, 2024-02-09T17:57:49.640 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:57:49.640 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:57:49.640 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:57:49.641 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:57:49.641 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:57:49.641 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:57:10.792025+0000", 2024-02-09T17:57:49.641 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:57:49.641 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.641 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.641 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 6, 2024-02-09T17:57:49.641 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:57:49.641 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:57:49.641 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:57:49.641 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:57:49.641 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:57:49.641 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:57:46.462636+0000", 2024-02-09T17:57:49.642 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:57:49.642 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.642 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.642 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 7, 2024-02-09T17:57:49.642 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:57:49.642 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:57:49.642 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:57:49.642 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:57:49.642 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:57:49.642 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:56:47.199190+0000", 2024-02-09T17:57:49.642 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:57:49.642 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.642 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.642 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 8, 2024-02-09T17:57:49.642 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:57:49.643 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:57:49.643 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:57:49.643 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:57:49.643 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:57:49.643 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:57:28.367312+0000", 2024-02-09T17:57:49.643 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 145 2024-02-09T17:57:49.643 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.643 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.643 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 9, 2024-02-09T17:57:49.643 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:57:49.643 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:57:49.643 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:57:49.643 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:57:49.643 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:57:49.643 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:56:45.805121+0000", 2024-02-09T17:57:49.644 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:57:49.644 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.644 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.644 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 10, 2024-02-09T17:57:49.644 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:57:49.644 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:57:49.644 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:57:49.644 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:57:49.644 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:57:49.644 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:57:09.652358+0000", 2024-02-09T17:57:49.644 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:57:49.644 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.644 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.645 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 11, 2024-02-09T17:57:49.645 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T17:57:49.645 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T17:57:49.645 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T17:57:49.645 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T17:57:49.645 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T17:57:49.645 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T17:57:05.442904+0000", 2024-02-09T17:57:49.645 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T17:57:49.645 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.645 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T17:57:49.645 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_upmap": [ 2024-02-09T17:57:49.645 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.645 INFO:teuthology.orchestra.run.smithi099.stdout: "pgid": "1.7", 2024-02-09T17:57:49.645 INFO:teuthology.orchestra.run.smithi099.stdout: "osds": [ 2024-02-09T17:57:49.646 INFO:teuthology.orchestra.run.smithi099.stdout: 1, 2024-02-09T17:57:49.646 INFO:teuthology.orchestra.run.smithi099.stdout: 8, 2024-02-09T17:57:49.646 INFO:teuthology.orchestra.run.smithi099.stdout: 6 2024-02-09T17:57:49.646 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.646 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.646 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T17:57:49.646 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_upmap_items": [ 2024-02-09T17:57:49.646 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.646 INFO:teuthology.orchestra.run.smithi099.stdout: "pgid": "1.1e", 2024-02-09T17:57:49.646 INFO:teuthology.orchestra.run.smithi099.stdout: "mappings": [ 2024-02-09T17:57:49.646 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.646 INFO:teuthology.orchestra.run.smithi099.stdout: "from": 8, 2024-02-09T17:57:49.646 INFO:teuthology.orchestra.run.smithi099.stdout: "to": 3 2024-02-09T17:57:49.646 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.647 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.647 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.647 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.647 INFO:teuthology.orchestra.run.smithi099.stdout: "pgid": "8.7", 2024-02-09T17:57:49.647 INFO:teuthology.orchestra.run.smithi099.stdout: "mappings": [ 2024-02-09T17:57:49.647 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:57:49.647 INFO:teuthology.orchestra.run.smithi099.stdout: "from": 11, 2024-02-09T17:57:49.647 INFO:teuthology.orchestra.run.smithi099.stdout: "to": 10 2024-02-09T17:57:49.647 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.647 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:57:49.647 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.647 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T17:57:49.647 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_upmap_primaries": [], 2024-02-09T17:57:49.647 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_temp": [], 2024-02-09T17:57:49.647 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_temp": [], 2024-02-09T17:57:49.647 INFO:teuthology.orchestra.run.smithi099.stdout: "blocklist": { 2024-02-09T17:57:49.648 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:0/3904172782": "2024-02-10T17:16:24.229365+0000", 2024-02-09T17:57:49.648 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:6801/1586034576": "2024-02-10T17:16:24.229365+0000", 2024-02-09T17:57:49.648 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:6800/746248609": "2024-02-10T17:15:54.399201+0000", 2024-02-09T17:57:49.648 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:0/4092784197": "2024-02-10T17:16:24.229365+0000", 2024-02-09T17:57:49.648 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:0/2844598101": "2024-02-10T17:15:54.399201+0000", 2024-02-09T17:57:49.648 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:6801/746248609": "2024-02-10T17:15:54.399201+0000", 2024-02-09T17:57:49.648 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:0/2528750050": "2024-02-10T17:15:54.399201+0000", 2024-02-09T17:57:49.648 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:6800/1586034576": "2024-02-10T17:16:24.229365+0000", 2024-02-09T17:57:49.648 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:0/2214140078": "2024-02-10T17:15:54.399201+0000", 2024-02-09T17:57:49.648 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:0/4217821965": "2024-02-10T17:16:24.229365+0000" 2024-02-09T17:57:49.648 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.648 INFO:teuthology.orchestra.run.smithi099.stdout: "range_blocklist": {}, 2024-02-09T17:57:49.648 INFO:teuthology.orchestra.run.smithi099.stdout: "erasure_code_profiles": { 2024-02-09T17:57:49.648 INFO:teuthology.orchestra.run.smithi099.stdout: "default": { 2024-02-09T17:57:49.649 INFO:teuthology.orchestra.run.smithi099.stdout: "crush-failure-domain": "osd", 2024-02-09T17:57:49.649 INFO:teuthology.orchestra.run.smithi099.stdout: "k": "2", 2024-02-09T17:57:49.649 INFO:teuthology.orchestra.run.smithi099.stdout: "m": "1", 2024-02-09T17:57:49.649 INFO:teuthology.orchestra.run.smithi099.stdout: "plugin": "jerasure", 2024-02-09T17:57:49.649 INFO:teuthology.orchestra.run.smithi099.stdout: "technique": "reed_sol_van" 2024-02-09T17:57:49.649 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.649 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:57:49.649 INFO:teuthology.orchestra.run.smithi099.stdout: "removed_snaps_queue": [], 2024-02-09T17:57:49.649 INFO:teuthology.orchestra.run.smithi099.stdout: "new_removed_snaps": [], 2024-02-09T17:57:49.649 INFO:teuthology.orchestra.run.smithi099.stdout: "new_purged_snaps": [], 2024-02-09T17:57:49.649 INFO:teuthology.orchestra.run.smithi099.stdout: "crush_node_flags": {}, 2024-02-09T17:57:49.649 INFO:teuthology.orchestra.run.smithi099.stdout: "device_class_flags": {}, 2024-02-09T17:57:49.649 INFO:teuthology.orchestra.run.smithi099.stdout: "stretch_mode": { 2024-02-09T17:57:49.649 INFO:teuthology.orchestra.run.smithi099.stdout: "stretch_mode_enabled": false, 2024-02-09T17:57:49.649 INFO:teuthology.orchestra.run.smithi099.stdout: "stretch_bucket_count": 0, 2024-02-09T17:57:49.650 INFO:teuthology.orchestra.run.smithi099.stdout: "degraded_stretch_mode": 0, 2024-02-09T17:57:49.650 INFO:teuthology.orchestra.run.smithi099.stdout: "recovering_stretch_mode": 0, 2024-02-09T17:57:49.650 INFO:teuthology.orchestra.run.smithi099.stdout: "stretch_mode_bucket": 0 2024-02-09T17:57:49.650 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:57:49.650 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:57:49.650 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:57:49.703 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:49.727 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:57:49.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:49.729 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:57:49.831 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:57:49.865 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:49.972 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:57:49.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:49.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:57:50.093 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:57:50.127 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:50.427 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:50 smithi099 ceph-mon[39420]: pgmap v2063: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 4.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 820 op/s; 15012/239055 objects degraded (6.280%) 2024-02-09T17:57:50.427 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:50 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1581489464' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-09T17:57:50.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:50 smithi179 ceph-mon[43975]: pgmap v2063: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 4.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 820 op/s; 15012/239055 objects degraded (6.280%) 2024-02-09T17:57:50.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:50 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1581489464' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-09T17:57:50.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:50 smithi162 ceph-mon[44304]: pgmap v2063: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 4.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 820 op/s; 15012/239055 objects degraded (6.280%) 2024-02-09T17:57:50.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:50 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1581489464' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-09T17:57:50.688 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:57:50.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:50.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:57:50.796 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:57:50.851 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:51.305 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:57:51.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:51.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:57:51.409 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:57:51.484 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:51.525 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:57:51.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:51.530 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:57:51.631 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:57:51.659 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:51.678 INFO:tasks.thrashosds.thrasher:j is {'epoch': 831, 'fsid': 'a69c870a-c76e-11ee-95b7-87774f69a715', 'created': '2024-02-09T17:15:26.113606+0000', 'modified': '2024-02-09T17:57:47.345312+0000', 'last_up_change': '2024-02-09T17:56:39.428689+0000', 'last_in_change': '2024-02-09T17:43:24.138763+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 27, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 8, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-09T17:18:56.552286+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '544', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '514', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.119999885559082, 'score_stable': 2.130000114440918, 'optimal_score': 1, 'raw_score_acting': 7.119999885559082, 'raw_score_stable': 2.130000114440918, 'primary_affinity_weighted': 0.7807959914207458, 'average_primary_affinity': 0.8500000238418579, 'average_primary_affinity_weighted': 0.7799999713897705, 'error_message': 'pool 1 has acting primaries on OSD(s) with primary affinity 0, read balance score is not accurate'}}, {'pool': 8, 'pool_name': 'unique_pool_6', 'create_time': '2024-02-09T17:55:15.183490+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '793', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.75, 'score_stable': 2.75, 'optimal_score': 0.9100000262260437, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.7807959914207458, 'average_primary_affinity': 0.8500000238418579, 'average_primary_affinity_weighted': 0.7799999713897705}}], 'osds': [{'osd': 0, 'uuid': '21739aa7-e278-45da-9a3f-867385f01baf', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.609619140625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 785, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6802', 'nonce': 3859868923}, {'type': 'v1', 'addr': '172.21.15.99:6803', 'nonce': 3859868923}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6804', 'nonce': 3859868923}, {'type': 'v1', 'addr': '172.21.15.99:6805', 'nonce': 3859868923}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6808', 'nonce': 3859868923}, {'type': 'v1', 'addr': '172.21.15.99:6809', 'nonce': 3859868923}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6806', 'nonce': 3859868923}, {'type': 'v1', 'addr': '172.21.15.99:6807', 'nonce': 3859868923}]}, 'public_addr': '172.21.15.99:6803/3859868923', 'cluster_addr': '172.21.15.99:6805/3859868923', 'heartbeat_back_addr': '172.21.15.99:6809/3859868923', 'heartbeat_front_addr': '172.21.15.99:6807/3859868923', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '8eab347c-9065-4de1-93ae-e2eb7e11cab7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 815, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6810', 'nonce': 3868328452}, {'type': 'v1', 'addr': '172.21.15.99:6811', 'nonce': 3868328452}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6812', 'nonce': 3868328452}, {'type': 'v1', 'addr': '172.21.15.99:6813', 'nonce': 3868328452}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6816', 'nonce': 3868328452}, {'type': 'v1', 'addr': '172.21.15.99:6817', 'nonce': 3868328452}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6814', 'nonce': 3868328452}, {'type': 'v1', 'addr': '172.21.15.99:6815', 'nonce': 3868328452}]}, 'public_addr': '172.21.15.99:6811/3868328452', 'cluster_addr': '172.21.15.99:6813/3868328452', 'heartbeat_back_addr': '172.21.15.99:6817/3868328452', 'heartbeat_front_addr': '172.21.15.99:6815/3868328452', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '575f4257-7e34-485e-963c-2607b69e1220', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.3601837158203125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 661, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6818', 'nonce': 537506237}, {'type': 'v1', 'addr': '172.21.15.99:6819', 'nonce': 537506237}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6820', 'nonce': 537506237}, {'type': 'v1', 'addr': '172.21.15.99:6821', 'nonce': 537506237}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6824', 'nonce': 537506237}, {'type': 'v1', 'addr': '172.21.15.99:6825', 'nonce': 537506237}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6822', 'nonce': 537506237}, {'type': 'v1', 'addr': '172.21.15.99:6823', 'nonce': 537506237}]}, 'public_addr': '172.21.15.99:6819/537506237', 'cluster_addr': '172.21.15.99:6821/537506237', 'heartbeat_back_addr': '172.21.15.99:6825/537506237', 'heartbeat_front_addr': '172.21.15.99:6823/537506237', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '1a520bcf-81bb-4f1d-80fb-11e98008e429', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 0.8014984130859375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 785, 'down_at': 815, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6826', 'nonce': 762784793}, {'type': 'v1', 'addr': '172.21.15.99:6827', 'nonce': 762784793}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6828', 'nonce': 762784793}, {'type': 'v1', 'addr': '172.21.15.99:6829', 'nonce': 762784793}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6832', 'nonce': 762784793}, {'type': 'v1', 'addr': '172.21.15.99:6833', 'nonce': 762784793}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6830', 'nonce': 762784793}, {'type': 'v1', 'addr': '172.21.15.99:6831', 'nonce': 762784793}]}, 'public_addr': '172.21.15.99:6827/762784793', 'cluster_addr': '172.21.15.99:6829/762784793', 'heartbeat_back_addr': '172.21.15.99:6833/762784793', 'heartbeat_front_addr': '172.21.15.99:6831/762784793', 'state': ['exists']}, {'osd': 4, 'uuid': '4617c2fe-c9d8-40cd-887b-303e3215bdf8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 815, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6800', 'nonce': 4178676504}, {'type': 'v1', 'addr': '172.21.15.162:6801', 'nonce': 4178676504}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6802', 'nonce': 4178676504}, {'type': 'v1', 'addr': '172.21.15.162:6803', 'nonce': 4178676504}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6806', 'nonce': 4178676504}, {'type': 'v1', 'addr': '172.21.15.162:6807', 'nonce': 4178676504}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6804', 'nonce': 4178676504}, {'type': 'v1', 'addr': '172.21.15.162:6805', 'nonce': 4178676504}]}, 'public_addr': '172.21.15.162:6801/4178676504', 'cluster_addr': '172.21.15.162:6803/4178676504', 'heartbeat_back_addr': '172.21.15.162:6807/4178676504', 'heartbeat_front_addr': '172.21.15.162:6805/4178676504', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'da98e3f3-173e-46fe-be8b-1ccbe8b0dabc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 815, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6808', 'nonce': 1489371339}, {'type': 'v1', 'addr': '172.21.15.162:6809', 'nonce': 1489371339}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6810', 'nonce': 1489371339}, {'type': 'v1', 'addr': '172.21.15.162:6811', 'nonce': 1489371339}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6814', 'nonce': 1489371339}, {'type': 'v1', 'addr': '172.21.15.162:6815', 'nonce': 1489371339}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6812', 'nonce': 1489371339}, {'type': 'v1', 'addr': '172.21.15.162:6813', 'nonce': 1489371339}]}, 'public_addr': '172.21.15.162:6809/1489371339', 'cluster_addr': '172.21.15.162:6811/1489371339', 'heartbeat_back_addr': '172.21.15.162:6815/1489371339', 'heartbeat_front_addr': '172.21.15.162:6813/1489371339', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'ff50382d-5ddf-4cda-bb0c-ff055feeaa40', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 49, 'up_thru': 815, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6816', 'nonce': 821169663}, {'type': 'v1', 'addr': '172.21.15.162:6817', 'nonce': 821169663}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6818', 'nonce': 821169663}, {'type': 'v1', 'addr': '172.21.15.162:6819', 'nonce': 821169663}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6822', 'nonce': 821169663}, {'type': 'v1', 'addr': '172.21.15.162:6823', 'nonce': 821169663}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6820', 'nonce': 821169663}, {'type': 'v1', 'addr': '172.21.15.162:6821', 'nonce': 821169663}]}, 'public_addr': '172.21.15.162:6817/821169663', 'cluster_addr': '172.21.15.162:6819/821169663', 'heartbeat_back_addr': '172.21.15.162:6823/821169663', 'heartbeat_front_addr': '172.21.15.162:6821/821169663', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'a82984dc-5e22-424b-a446-ae2ee71eee19', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.5982513427734375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 57, 'up_thru': 661, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6824', 'nonce': 965264369}, {'type': 'v1', 'addr': '172.21.15.162:6825', 'nonce': 965264369}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6826', 'nonce': 965264369}, {'type': 'v1', 'addr': '172.21.15.162:6827', 'nonce': 965264369}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6830', 'nonce': 965264369}, {'type': 'v1', 'addr': '172.21.15.162:6831', 'nonce': 965264369}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6828', 'nonce': 965264369}, {'type': 'v1', 'addr': '172.21.15.162:6829', 'nonce': 965264369}]}, 'public_addr': '172.21.15.162:6825/965264369', 'cluster_addr': '172.21.15.162:6827/965264369', 'heartbeat_back_addr': '172.21.15.162:6831/965264369', 'heartbeat_front_addr': '172.21.15.162:6829/965264369', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '63f9d79f-0400-44dc-b388-df00fefb5820', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 66, 'last_clean_end': 145, 'up_from': 171, 'up_thru': 790, 'down_at': 146, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6800', 'nonce': 2558294495}, {'type': 'v1', 'addr': '172.21.15.179:6801', 'nonce': 2558294495}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6802', 'nonce': 2558294495}, {'type': 'v1', 'addr': '172.21.15.179:6803', 'nonce': 2558294495}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6806', 'nonce': 2558294495}, {'type': 'v1', 'addr': '172.21.15.179:6807', 'nonce': 2558294495}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6804', 'nonce': 2558294495}, {'type': 'v1', 'addr': '172.21.15.179:6805', 'nonce': 2558294495}]}, 'public_addr': '172.21.15.179:6801/2558294495', 'cluster_addr': '172.21.15.179:6803/2558294495', 'heartbeat_back_addr': '172.21.15.179:6807/2558294495', 'heartbeat_front_addr': '172.21.15.179:6805/2558294495', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '7d995854-7328-4dfc-b1a9-0edca8d74f46', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 77, 'up_thru': 815, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6808', 'nonce': 504427183}, {'type': 'v1', 'addr': '172.21.15.179:6809', 'nonce': 504427183}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6810', 'nonce': 504427183}, {'type': 'v1', 'addr': '172.21.15.179:6811', 'nonce': 504427183}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6814', 'nonce': 504427183}, {'type': 'v1', 'addr': '172.21.15.179:6815', 'nonce': 504427183}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6812', 'nonce': 504427183}, {'type': 'v1', 'addr': '172.21.15.179:6813', 'nonce': 504427183}]}, 'public_addr': '172.21.15.179:6809/504427183', 'cluster_addr': '172.21.15.179:6811/504427183', 'heartbeat_back_addr': '172.21.15.179:6815/504427183', 'heartbeat_front_addr': '172.21.15.179:6813/504427183', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '151fbedd-08e6-46e6-a4e5-a14248399f61', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 84, 'up_thru': 790, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6816', 'nonce': 2403345164}, {'type': 'v1', 'addr': '172.21.15.179:6817', 'nonce': 2403345164}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6818', 'nonce': 2403345164}, {'type': 'v1', 'addr': '172.21.15.179:6819', 'nonce': 2403345164}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6822', 'nonce': 2403345164}, {'type': 'v1', 'addr': '172.21.15.179:6823', 'nonce': 2403345164}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6820', 'nonce': 2403345164}, {'type': 'v1', 'addr': '172.21.15.179:6821', 'nonce': 2403345164}]}, 'public_addr': '172.21.15.179:6817/2403345164', 'cluster_addr': '172.21.15.179:6819/2403345164', 'heartbeat_back_addr': '172.21.15.179:6823/2403345164', 'heartbeat_front_addr': '172.21.15.179:6821/2403345164', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '6d3a3ad8-417a-483f-991c-52549dbe7d73', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 90, 'up_thru': 815, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6824', 'nonce': 2899402373}, {'type': 'v1', 'addr': '172.21.15.179:6825', 'nonce': 2899402373}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6826', 'nonce': 2899402373}, {'type': 'v1', 'addr': '172.21.15.179:6827', 'nonce': 2899402373}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6830', 'nonce': 2899402373}, {'type': 'v1', 'addr': '172.21.15.179:6831', 'nonce': 2899402373}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6828', 'nonce': 2899402373}, {'type': 'v1', 'addr': '172.21.15.179:6829', 'nonce': 2899402373}]}, 'public_addr': '172.21.15.179:6825/2899402373', 'cluster_addr': '172.21.15.179:6827/2899402373', 'heartbeat_back_addr': '172.21.15.179:6831/2899402373', 'heartbeat_front_addr': '172.21.15.179:6829/2899402373', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:56:33.967744+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:56:53.308946+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:57:44.470547+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:55:39.048234+0000', 'dead_epoch': 814}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:57:02.826463+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:57:10.792025+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:57:46.462636+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:56:47.199190+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:57:28.367312+0000', 'dead_epoch': 145}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:56:45.805121+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:57:09.652358+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T17:57:05.442904+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.7', 'osds': [1, 8, 6]}], 'pg_upmap_items': [{'pgid': '1.1e', 'mappings': [{'from': 8, 'to': 3}]}, {'pgid': '8.7', 'mappings': [{'from': 11, 'to': 10}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.99:0/3904172782': '2024-02-10T17:16:24.229365+0000', '172.21.15.99:6801/1586034576': '2024-02-10T17:16:24.229365+0000', '172.21.15.99:6800/746248609': '2024-02-10T17:15:54.399201+0000', '172.21.15.99:0/4092784197': '2024-02-10T17:16:24.229365+0000', '172.21.15.99:0/2844598101': '2024-02-10T17:15:54.399201+0000', '172.21.15.99:6801/746248609': '2024-02-10T17:15:54.399201+0000', '172.21.15.99:0/2528750050': '2024-02-10T17:15:54.399201+0000', '172.21.15.99:6800/1586034576': '2024-02-10T17:16:24.229365+0000', '172.21.15.99:0/2214140078': '2024-02-10T17:15:54.399201+0000', '172.21.15.99:0/4217821965': '2024-02-10T17:16:24.229365+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-09T17:57:51.678 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph pg dump --format=json 2024-02-09T17:57:51.684 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:57:51.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:51.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:57:51.788 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:57:51.816 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:51.839 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:57:51.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:51.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:57:51.943 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:57:51.971 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:51.997 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:57:52.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:52.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:57:52.102 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:57:52.149 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:52.179 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:57:52.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:52.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:57:52.214 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:57:52.243 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:57:52.268 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:57:52.287 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:57:52.302 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:57:52.328 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:57:52.353 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:57:52.354 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:52.379 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:57:52.392 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:57:52.405 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:57:52.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:52.406 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:57:52.432 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:57:52.471 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:57:52.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:52 smithi099 ceph-mon[39420]: pgmap v2064: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 4.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 663 op/s; 15012/237549 objects degraded (6.320%) 2024-02-09T17:57:52.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:52 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 15012/237549 objects degraded (6.320%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:57:52.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:52 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2546366111' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:57:52.499 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:57:52.499 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:57:52.499 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:57:52.499 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:57:52.499 INFO:teuthology.orchestra.run.smithi099.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-09T17:57:52.499 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:57:52.499 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:57:52.499 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:57:52.499 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:57:52.499 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:57:52.500 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:57:52.500 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:57:52.500 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:57:52.507 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:57:52.534 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:52.556 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:57:52.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:52.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:57:52.662 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:57:52.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:52 smithi179 ceph-mon[43975]: pgmap v2064: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 4.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 663 op/s; 15012/237549 objects degraded (6.320%) 2024-02-09T17:57:52.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:52 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 15012/237549 objects degraded (6.320%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:57:52.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:52 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2546366111' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:57:52.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:52 smithi162 ceph-mon[44304]: pgmap v2064: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 4.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 663 op/s; 15012/237549 objects degraded (6.320%) 2024-02-09T17:57:52.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:52 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 15012/237549 objects degraded (6.320%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:57:52.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:52 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2546366111' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:57:52.720 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:52.751 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:57:52.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:52.756 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:57:52.856 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:57:52.883 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:52.891 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:57:52.905 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:57:52.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:52.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:57:53.009 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:57:53.037 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:53.059 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:57:53.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:53.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:57:53.163 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:57:53.191 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:53.213 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:57:53.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:53.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:57:53.317 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:57:53.347 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:53.371 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:57:53.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:53.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:57:53.456 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:57:53.475 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:57:53.481 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-09T17:57:53.482 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-09T17:57:53.511 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:53.652 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:57:53.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:53.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:57:53.757 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:57:53.787 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:53.811 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:57:53.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:53.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:57:53.916 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:57:53.944 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:53.965 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:57:53.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:53.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:57:54.069 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:57:54.134 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:54.172 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:57:54.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:54.179 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:57:54.280 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:57:54.307 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:54.328 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:57:54.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:54.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:57:54.433 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:57:54.496 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:54.543 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:57:54.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:54.546 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:57:54.647 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:57:54.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:54 smithi179 ceph-mon[43975]: pgmap v2065: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 4.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 822 op/s; 14245/230031 objects degraded (6.193%) 2024-02-09T17:57:54.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:54 smithi162 ceph-mon[44304]: pgmap v2065: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 4.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 822 op/s; 14245/230031 objects degraded (6.193%) 2024-02-09T17:57:54.716 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:54.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:54 smithi099 ceph-mon[39420]: pgmap v2065: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 4.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 822 op/s; 14245/230031 objects degraded (6.193%) 2024-02-09T17:57:54.771 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:57:54.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:54.773 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:57:54.873 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:57:54.903 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:54.928 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:57:54.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:54.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:57:55.013 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:57:55.013 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:57:55.033 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:57:55.076 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:55.117 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:57:55.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:55.123 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:57:55.224 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:57:55.254 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:55.276 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:57:55.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:55.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:57:55.381 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:57:55.415 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:55.436 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:57:55.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:55.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:57:55.483 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:57:55.543 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:57:55.592 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:55.645 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:57:55.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:55.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:57:55.812 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:57:55.850 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:56.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:56 smithi099 ceph-mon[39420]: pgmap v2066: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 4.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 904 op/s; 14245/225417 objects degraded (6.319%) 2024-02-09T17:57:56.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:56 smithi179 ceph-mon[43975]: pgmap v2066: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 4.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 904 op/s; 14245/225417 objects degraded (6.319%) 2024-02-09T17:57:56.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:56 smithi162 ceph-mon[44304]: pgmap v2066: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 4.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 904 op/s; 14245/225417 objects degraded (6.319%) 2024-02-09T17:57:57.420 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:57 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 14245/225417 objects degraded (6.319%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:57:57.515 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:57:57.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:57.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:57:57.623 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:57:57.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:57 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 14245/225417 objects degraded (6.319%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:57:57.675 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:57.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:57 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 14245/225417 objects degraded (6.319%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:57:57.714 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:57:57.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:57.722 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:57:57.823 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:57:57.875 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:57.919 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:57:57.923 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:57.924 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:57:57.972 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:57:57.983 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-02-09T17:57:58.025 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:57:58.056 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:58.356 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-09T17:57:58.607 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:57:58.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:57:58.619 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:58 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:57:58.619 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:58 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:57:58.619 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:58 smithi099 ceph-mon[39420]: pgmap v2067: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 4.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 808 op/s; 13460/219438 objects degraded (6.134%) 2024-02-09T17:57:58.619 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:58 smithi099 ceph-mon[39420]: osdmap e832: 12 total, 11 up, 12 in 2024-02-09T17:57:58.619 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:58 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1852713742' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:57:58.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:57:58.647 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:57:58.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:58 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:57:58.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:58 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:57:58.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:58 smithi179 ceph-mon[43975]: pgmap v2067: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 4.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 808 op/s; 13460/219438 objects degraded (6.134%) 2024-02-09T17:57:58.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:58 smithi179 ceph-mon[43975]: osdmap e832: 12 total, 11 up, 12 in 2024-02-09T17:57:58.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:58 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1852713742' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:57:58.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:58 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:57:58.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:58 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:57:58.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:58 smithi162 ceph-mon[44304]: pgmap v2067: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 4.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 808 op/s; 13460/219438 objects degraded (6.134%) 2024-02-09T17:57:58.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:58 smithi162 ceph-mon[44304]: osdmap e832: 12 total, 11 up, 12 in 2024-02-09T17:57:58.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:58 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1852713742' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:57:58.721 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:57:58.760 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:57:59.003 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is set 2024-02-09T17:57:59.168 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":2067,"stamp":"2024-02-09T17:57:57.222458+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4793991736,"num_objects":73146,"num_object_clones":0,"num_object_copies":219438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13460,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73146,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172995,"num_write_kb":7876417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":165404,"ondisk_log_size":165404,"up":133,"acting":133,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":23639636,"kb_used_data":14557600,"kb_used_omap":229,"kb_used_meta":9081562,"kb_avail":1101302188,"statfs":{"total":1151940427776,"available":1127733440512,"internally_reserved":0,"allocated":14906982400,"data_stored":14903802520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235127,"internal_metadata":9299519881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,2,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-529924096,"num_objects":-8086,"num_object_clones":0,"num_object_copies":-24258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8086,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002336"},"pg_stats":[{"pgid":"1.1e","version":"63'139","reported_seq":960,"reported_epoch":831,"state":"active+undersized","last_fresh":"2024-02-09T17:57:47.372330+0000","last_change":"2024-02-09T17:56:40.457389+0000","last_active":"2024-02-09T17:57:47.372330+0000","last_peered":"2024-02-09T17:57:47.372330+0000","last_clean":"2024-02-09T17:56:36.455001+0000","last_became_active":"2024-02-09T17:56:40.457389+0000","last_became_peered":"2024-02-09T17:56:40.457389+0000","last_unstale":"2024-02-09T17:57:47.372330+0000","last_undegraded":"2024-02-09T17:57:47.372330+0000","last_fullsized":"2024-02-09T17:56:40.455311+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":782,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:17.113960+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:55:17.113960+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:57:47.547522+0000","scrub_duration":125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":["11","7"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"63'139","reported_seq":977,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-02-09T17:57:47.371134+0000","last_change":"2024-02-09T17:56:45.783380+0000","last_active":"2024-02-09T17:57:47.371134+0000","last_peered":"2024-02-09T17:57:47.371134+0000","last_clean":"2024-02-09T17:57:47.371134+0000","last_became_active":"2024-02-09T17:43:28.594296+0000","last_became_peered":"2024-02-09T17:43:28.594296+0000","last_unstale":"2024-02-09T17:57:47.371134+0000","last_undegraded":"2024-02-09T17:57:47.371134+0000","last_fullsized":"2024-02-09T17:57:47.371134+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:56:45.783240+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:48:24.634092+0000","last_clean_scrub_stamp":"2024-02-09T17:56:45.783240+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:58:09.331513+0000","scrub_duration":279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"63'139","reported_seq":1497,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-02-09T17:57:47.365137+0000","last_change":"2024-02-09T17:57:24.725091+0000","last_active":"2024-02-09T17:57:47.365137+0000","last_peered":"2024-02-09T17:57:47.365137+0000","last_clean":"2024-02-09T17:57:47.365137+0000","last_became_active":"2024-02-09T17:43:19.103584+0000","last_became_peered":"2024-02-09T17:43:19.103584+0000","last_unstale":"2024-02-09T17:57:47.365137+0000","last_undegraded":"2024-02-09T17:57:47.365137+0000","last_fullsized":"2024-02-09T17:57:47.365137+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:57:24.724984+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:23.912504+0000","last_clean_scrub_stamp":"2024-02-09T17:57:24.724984+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:58:34.802203+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"63'139","reported_seq":1295,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-02-09T17:57:47.361577+0000","last_change":"2024-02-09T17:56:53.473660+0000","last_active":"2024-02-09T17:57:47.361577+0000","last_peered":"2024-02-09T17:57:47.361577+0000","last_clean":"2024-02-09T17:57:47.361577+0000","last_became_active":"2024-02-09T17:43:19.102729+0000","last_became_peered":"2024-02-09T17:43:19.102729+0000","last_unstale":"2024-02-09T17:57:47.361577+0000","last_undegraded":"2024-02-09T17:57:47.361577+0000","last_fullsized":"2024-02-09T17:57:47.361577+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:56:53.473532+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:45:44.988633+0000","last_clean_scrub_stamp":"2024-02-09T17:56:53.473532+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:58:21.805992+0000","scrub_duration":205,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"63'139","reported_seq":958,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-02-09T17:57:47.365324+0000","last_change":"2024-02-09T17:57:42.734318+0000","last_active":"2024-02-09T17:57:47.365324+0000","last_peered":"2024-02-09T17:57:47.365324+0000","last_clean":"2024-02-09T17:57:47.365324+0000","last_became_active":"2024-02-09T17:43:19.120814+0000","last_became_peered":"2024-02-09T17:43:19.120814+0000","last_unstale":"2024-02-09T17:57:47.365324+0000","last_undegraded":"2024-02-09T17:57:47.365324+0000","last_fullsized":"2024-02-09T17:57:47.365324+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:57:42.734241+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:57:42.734241+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:59:02.094445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"63'139","reported_seq":1804,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-02-09T17:57:47.372729+0000","last_change":"2024-02-09T17:56:52.547225+0000","last_active":"2024-02-09T17:57:47.372729+0000","last_peered":"2024-02-09T17:57:47.372729+0000","last_clean":"2024-02-09T17:57:47.372729+0000","last_became_active":"2024-02-09T17:55:21.231291+0000","last_became_peered":"2024-02-09T17:55:21.231291+0000","last_unstale":"2024-02-09T17:57:47.372729+0000","last_undegraded":"2024-02-09T17:57:47.372729+0000","last_fullsized":"2024-02-09T17:57:47.372729+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":791,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:56:52.547098+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:30.193194+0000","last_clean_scrub_stamp":"2024-02-09T17:56:52.547098+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:58:17.022541+0000","scrub_duration":130,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"63'139","reported_seq":1642,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-02-09T17:57:47.371618+0000","last_change":"2024-02-09T17:56:42.154442+0000","last_active":"2024-02-09T17:57:47.371618+0000","last_peered":"2024-02-09T17:57:47.371618+0000","last_clean":"2024-02-09T17:57:47.371618+0000","last_became_active":"2024-02-09T17:43:28.585573+0000","last_became_peered":"2024-02-09T17:43:28.585573+0000","last_unstale":"2024-02-09T17:57:47.371618+0000","last_undegraded":"2024-02-09T17:57:47.371618+0000","last_fullsized":"2024-02-09T17:57:47.371618+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:56:42.154328+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:56:42.154328+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:58:08.168085+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"63'139","reported_seq":1088,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-02-09T17:57:47.360614+0000","last_change":"2024-02-09T17:56:51.533433+0000","last_active":"2024-02-09T17:57:47.360614+0000","last_peered":"2024-02-09T17:57:47.360614+0000","last_clean":"2024-02-09T17:57:47.360614+0000","last_became_active":"2024-02-09T17:48:46.184419+0000","last_became_peered":"2024-02-09T17:48:46.184419+0000","last_unstale":"2024-02-09T17:57:47.360614+0000","last_undegraded":"2024-02-09T17:57:47.360614+0000","last_fullsized":"2024-02-09T17:57:47.360614+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:56:51.533308+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:44:50.214686+0000","last_clean_scrub_stamp":"2024-02-09T17:56:51.533308+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:57:58.467145+0000","scrub_duration":194,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"63'139","reported_seq":974,"reported_epoch":831,"state":"active+undersized","last_fresh":"2024-02-09T17:57:47.367096+0000","last_change":"2024-02-09T17:56:40.452951+0000","last_active":"2024-02-09T17:57:47.367096+0000","last_peered":"2024-02-09T17:57:47.367096+0000","last_clean":"2024-02-09T17:56:36.452114+0000","last_became_active":"2024-02-09T17:56:40.452951+0000","last_became_peered":"2024-02-09T17:56:40.452951+0000","last_unstale":"2024-02-09T17:57:47.367096+0000","last_undegraded":"2024-02-09T17:57:47.367096+0000","last_fullsized":"2024-02-09T17:56:40.451051+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:54:56.134826+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:11.970338+0000","last_clean_scrub_stamp":"2024-02-09T17:54:56.134826+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:57:54.214921+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"63'139","reported_seq":1400,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-02-09T17:57:47.373490+0000","last_change":"2024-02-09T17:57:22.528295+0000","last_active":"2024-02-09T17:57:47.373490+0000","last_peered":"2024-02-09T17:57:47.373490+0000","last_clean":"2024-02-09T17:57:47.373490+0000","last_became_active":"2024-02-09T17:43:19.138398+0000","last_became_peered":"2024-02-09T17:43:19.138398+0000","last_unstale":"2024-02-09T17:57:47.373490+0000","last_undegraded":"2024-02-09T17:57:47.373490+0000","last_fullsized":"2024-02-09T17:57:47.373490+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:57:22.528147+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:57:22.528147+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:58:51.521146+0000","scrub_duration":169,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"63'139","reported_seq":1470,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-02-09T17:57:47.357406+0000","last_change":"2024-02-09T17:56:39.761291+0000","last_active":"2024-02-09T17:57:47.357406+0000","last_peered":"2024-02-09T17:57:47.357406+0000","last_clean":"2024-02-09T17:57:47.357406+0000","last_became_active":"2024-02-09T17:43:28.589307+0000","last_became_peered":"2024-02-09T17:43:28.589307+0000","last_unstale":"2024-02-09T17:57:47.357406+0000","last_undegraded":"2024-02-09T17:57:47.357406+0000","last_fullsized":"2024-02-09T17:57:47.357406+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:56:39.761191+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:38:26.968892+0000","last_clean_scrub_stamp":"2024-02-09T17:56:39.761191+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:57:54.003074+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"831'10673","reported_seq":10845,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-02-09T17:57:53.553632+0000","last_change":"2024-02-09T17:57:24.093468+0000","last_active":"2024-02-09T17:57:53.553632+0000","last_peered":"2024-02-09T17:57:53.553632+0000","last_clean":"2024-02-09T17:57:53.553632+0000","last_became_active":"2024-02-09T17:55:16.444412+0000","last_became_peered":"2024-02-09T17:55:16.444412+0000","last_unstale":"2024-02-09T17:57:53.553632+0000","last_undegraded":"2024-02-09T17:57:53.553632+0000","last_fullsized":"2024-02-09T17:57:53.553632+0000","mapping_epoch":785,"log_start":"796'600","ondisk_log_start":"796'600","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"826'9206","last_scrub_stamp":"2024-02-09T17:57:24.093337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:57:24.093337+0000","objects_scrubbed":6053,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:58:36.402300+0000","scrub_duration":1551,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298123264,"num_objects":4549,"num_object_clones":0,"num_object_copies":13647,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10673,"num_write_kb":487104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,0],"acting":[10,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"63'139","reported_seq":2049,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-02-09T17:57:47.373294+0000","last_change":"2024-02-09T17:56:46.401757+0000","last_active":"2024-02-09T17:57:47.373294+0000","last_peered":"2024-02-09T17:57:47.373294+0000","last_clean":"2024-02-09T17:57:47.373294+0000","last_became_active":"2024-02-09T17:41:14.521079+0000","last_became_peered":"2024-02-09T17:41:14.521079+0000","last_unstale":"2024-02-09T17:57:47.373294+0000","last_undegraded":"2024-02-09T17:57:47.373294+0000","last_fullsized":"2024-02-09T17:57:47.373294+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:56:46.401666+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:56:46.401666+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:58:11.562575+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"831'10597","reported_seq":10824,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-02-09T17:57:52.434215+0000","last_change":"2024-02-09T17:57:22.913985+0000","last_active":"2024-02-09T17:57:52.434215+0000","last_peered":"2024-02-09T17:57:52.434215+0000","last_clean":"2024-02-09T17:57:52.434215+0000","last_became_active":"2024-02-09T17:55:16.913155+0000","last_became_peered":"2024-02-09T17:55:16.913155+0000","last_unstale":"2024-02-09T17:57:52.434215+0000","last_undegraded":"2024-02-09T17:57:52.434215+0000","last_fullsized":"2024-02-09T17:57:52.434215+0000","mapping_epoch":785,"log_start":"796'500","ondisk_log_start":"796'500","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"826'9085","last_scrub_stamp":"2024-02-09T17:57:22.913841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:57:22.913841+0000","objects_scrubbed":6226,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:58:31.626831+0000","scrub_duration":1323,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306642944,"num_objects":4679,"num_object_clones":0,"num_object_copies":14037,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10597,"num_write_kb":488832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,1],"acting":[8,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.2","version":"63'139","reported_seq":1622,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-02-09T17:57:47.361227+0000","last_change":"2024-02-09T17:57:44.411868+0000","last_active":"2024-02-09T17:57:47.361227+0000","last_peered":"2024-02-09T17:57:47.361227+0000","last_clean":"2024-02-09T17:57:47.361227+0000","last_became_active":"2024-02-09T17:43:39.981054+0000","last_became_peered":"2024-02-09T17:43:39.981054+0000","last_unstale":"2024-02-09T17:57:47.361227+0000","last_undegraded":"2024-02-09T17:57:47.361227+0000","last_fullsized":"2024-02-09T17:57:47.361227+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:57:44.411749+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:57:44.411749+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:58:55.408216+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"831'10573","reported_seq":10751,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-02-09T17:57:52.497721+0000","last_change":"2024-02-09T17:56:39.649560+0000","last_active":"2024-02-09T17:57:52.497721+0000","last_peered":"2024-02-09T17:57:52.497721+0000","last_clean":"2024-02-09T17:57:52.497721+0000","last_became_active":"2024-02-09T17:55:16.204001+0000","last_became_peered":"2024-02-09T17:55:16.204001+0000","last_unstale":"2024-02-09T17:57:52.497721+0000","last_undegraded":"2024-02-09T17:57:52.497721+0000","last_fullsized":"2024-02-09T17:57:52.497721+0000","mapping_epoch":785,"log_start":"796'500","ondisk_log_start":"796'500","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"815'6775","last_scrub_stamp":"2024-02-09T17:56:39.649389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:56:39.649389+0000","objects_scrubbed":6759,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1192,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302317568,"num_objects":4613,"num_object_clones":0,"num_object_copies":13839,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10573,"num_write_kb":485952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,9],"acting":[8,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"63'139","reported_seq":1246,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-02-09T17:57:47.366981+0000","last_change":"2024-02-09T17:56:38.744322+0000","last_active":"2024-02-09T17:57:47.366981+0000","last_peered":"2024-02-09T17:57:47.366981+0000","last_clean":"2024-02-09T17:57:47.366981+0000","last_became_active":"2024-02-09T17:40:13.501711+0000","last_became_peered":"2024-02-09T17:40:13.501711+0000","last_unstale":"2024-02-09T17:57:47.366981+0000","last_undegraded":"2024-02-09T17:57:47.366981+0000","last_fullsized":"2024-02-09T17:57:47.366981+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:56:38.744215+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:46:40.684762+0000","last_clean_scrub_stamp":"2024-02-09T17:56:38.744215+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:57:49.842672+0000","scrub_duration":187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"831'10993","reported_seq":11098,"reported_epoch":831,"state":"active+undersized+degraded","last_fresh":"2024-02-09T17:57:57.175921+0000","last_change":"2024-02-09T17:56:40.456165+0000","last_active":"2024-02-09T17:57:57.175921+0000","last_peered":"2024-02-09T17:57:57.175921+0000","last_clean":"2024-02-09T17:56:39.401692+0000","last_became_active":"2024-02-09T17:56:40.456165+0000","last_became_peered":"2024-02-09T17:56:40.456165+0000","last_unstale":"2024-02-09T17:57:57.175921+0000","last_undegraded":"2024-02-09T17:56:40.454957+0000","last_fullsized":"2024-02-09T17:56:40.454832+0000","mapping_epoch":815,"log_start":"797'900","ondisk_log_start":"797'900","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:57:58.952326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290521088,"num_objects":4433,"num_object_clones":0,"num_object_copies":13299,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4433,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10993,"num_write_kb":493632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":["9","5"],"object_location_counts":[{"shards":"5,9","objects":4433}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.1","version":"63'139","reported_seq":782,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-02-09T17:57:47.372828+0000","last_change":"2024-02-09T17:56:53.563519+0000","last_active":"2024-02-09T17:57:47.372828+0000","last_peered":"2024-02-09T17:57:47.372828+0000","last_clean":"2024-02-09T17:57:47.372828+0000","last_became_active":"2024-02-09T17:54:32.415032+0000","last_became_peered":"2024-02-09T17:54:32.415032+0000","last_unstale":"2024-02-09T17:57:47.372828+0000","last_undegraded":"2024-02-09T17:57:47.372828+0000","last_fullsized":"2024-02-09T17:57:47.372828+0000","mapping_epoch":766,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":767,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:56:53.563396+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:56:53.563396+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:58:02.251227+0000","scrub_duration":117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.f","version":"831'10985","reported_seq":11167,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-02-09T17:57:56.464820+0000","last_change":"2024-02-09T17:56:45.650597+0000","last_active":"2024-02-09T17:57:56.464820+0000","last_peered":"2024-02-09T17:57:56.464820+0000","last_clean":"2024-02-09T17:57:56.464820+0000","last_became_active":"2024-02-09T17:55:16.205394+0000","last_became_peered":"2024-02-09T17:55:16.205394+0000","last_unstale":"2024-02-09T17:57:56.464820+0000","last_undegraded":"2024-02-09T17:57:56.464820+0000","last_fullsized":"2024-02-09T17:57:56.464820+0000","mapping_epoch":785,"log_start":"797'900","ondisk_log_start":"797'900","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"818'7248","last_scrub_stamp":"2024-02-09T17:56:45.650360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:56:45.650360+0000","objects_scrubbed":7196,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304939008,"num_objects":4653,"num_object_clones":0,"num_object_copies":13959,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10985,"num_write_kb":500416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,1],"acting":[11,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"63'139","reported_seq":1549,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-02-09T17:57:47.373036+0000","last_change":"2024-02-09T17:56:37.453550+0000","last_active":"2024-02-09T17:57:47.373036+0000","last_peered":"2024-02-09T17:57:47.373036+0000","last_clean":"2024-02-09T17:57:47.373036+0000","last_became_active":"2024-02-09T17:43:14.024894+0000","last_became_peered":"2024-02-09T17:43:14.024894+0000","last_unstale":"2024-02-09T17:57:47.373036+0000","last_undegraded":"2024-02-09T17:57:47.373036+0000","last_fullsized":"2024-02-09T17:57:47.373036+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:56:37.453473+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:11.970338+0000","last_clean_scrub_stamp":"2024-02-09T17:56:37.453473+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:57:51.003049+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.e","version":"831'10770","reported_seq":10952,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-02-09T17:57:53.794142+0000","last_change":"2024-02-09T17:56:45.795007+0000","last_active":"2024-02-09T17:57:53.794142+0000","last_peered":"2024-02-09T17:57:53.794142+0000","last_clean":"2024-02-09T17:57:53.794142+0000","last_became_active":"2024-02-09T17:55:16.912769+0000","last_became_peered":"2024-02-09T17:55:16.912769+0000","last_unstale":"2024-02-09T17:57:53.794142+0000","last_undegraded":"2024-02-09T17:57:53.794142+0000","last_fullsized":"2024-02-09T17:57:53.794142+0000","mapping_epoch":785,"log_start":"796'700","ondisk_log_start":"796'700","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"818'7152","last_scrub_stamp":"2024-02-09T17:56:45.794798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:56:45.794798+0000","objects_scrubbed":7113,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:58:05.577423+0000","scrub_duration":1380,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306446336,"num_objects":4676,"num_object_clones":0,"num_object_copies":14028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10770,"num_write_kb":494272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"63'139","reported_seq":1173,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-02-09T17:57:47.367147+0000","last_change":"2024-02-09T17:56:47.746888+0000","last_active":"2024-02-09T17:57:47.367147+0000","last_peered":"2024-02-09T17:57:47.367147+0000","last_clean":"2024-02-09T17:57:47.367147+0000","last_became_active":"2024-02-09T17:52:32.292519+0000","last_became_peered":"2024-02-09T17:52:32.292519+0000","last_unstale":"2024-02-09T17:57:47.367147+0000","last_undegraded":"2024-02-09T17:57:47.367147+0000","last_fullsized":"2024-02-09T17:57:47.367147+0000","mapping_epoch":716,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":717,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:56:47.746792+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:56:47.746792+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:58:10.055951+0000","scrub_duration":272,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.d","version":"831'10759","reported_seq":10978,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-02-09T17:57:56.546998+0000","last_change":"2024-02-09T17:57:31.713043+0000","last_active":"2024-02-09T17:57:56.546998+0000","last_peered":"2024-02-09T17:57:56.546998+0000","last_clean":"2024-02-09T17:57:56.546998+0000","last_became_active":"2024-02-09T17:55:16.444812+0000","last_became_peered":"2024-02-09T17:55:16.444812+0000","last_unstale":"2024-02-09T17:57:56.546998+0000","last_undegraded":"2024-02-09T17:57:56.546998+0000","last_fullsized":"2024-02-09T17:57:56.546998+0000","mapping_epoch":785,"log_start":"796'700","ondisk_log_start":"796'700","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"827'9558","last_scrub_stamp":"2024-02-09T17:57:31.712823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:57:31.712823+0000","objects_scrubbed":5744,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:58:53.881399+0000","scrub_duration":1339,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295698456,"num_objects":4513,"num_object_clones":0,"num_object_copies":13539,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4513,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10759,"num_write_kb":488641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,2],"acting":[11,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"63'139","reported_seq":1628,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-02-09T17:57:47.371695+0000","last_change":"2024-02-09T17:56:45.406886+0000","last_active":"2024-02-09T17:57:47.371695+0000","last_peered":"2024-02-09T17:57:47.371695+0000","last_clean":"2024-02-09T17:57:47.371695+0000","last_became_active":"2024-02-09T17:33:00.284924+0000","last_became_peered":"2024-02-09T17:33:00.284924+0000","last_unstale":"2024-02-09T17:57:47.371695+0000","last_undegraded":"2024-02-09T17:57:47.371695+0000","last_fullsized":"2024-02-09T17:57:47.371695+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:56:45.406767+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:56:45.406767+0000","last_clean_scrub_stamp":"2024-02-09T17:56:45.406767+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.c","version":"831'10923","reported_seq":11029,"reported_epoch":831,"state":"active+undersized+degraded","last_fresh":"2024-02-09T17:57:57.188142+0000","last_change":"2024-02-09T17:56:40.460851+0000","last_active":"2024-02-09T17:57:57.188142+0000","last_peered":"2024-02-09T17:57:57.188142+0000","last_clean":"2024-02-09T17:56:39.437320+0000","last_became_active":"2024-02-09T17:56:40.460851+0000","last_became_peered":"2024-02-09T17:56:40.460851+0000","last_unstale":"2024-02-09T17:57:57.188142+0000","last_undegraded":"2024-02-09T17:56:40.455527+0000","last_fullsized":"2024-02-09T17:56:40.455414+0000","mapping_epoch":815,"log_start":"797'900","ondisk_log_start":"797'900","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:57:58.952369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295370752,"num_objects":4507,"num_object_clones":0,"num_object_copies":13521,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4507,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10923,"num_write_kb":493760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":["9","8"],"object_location_counts":[{"shards":"8,9","objects":4507}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.5","version":"63'139","reported_seq":1469,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-02-09T17:57:47.372709+0000","last_change":"2024-02-09T17:56:40.524548+0000","last_active":"2024-02-09T17:57:47.372709+0000","last_peered":"2024-02-09T17:57:47.372709+0000","last_clean":"2024-02-09T17:57:47.372709+0000","last_became_active":"2024-02-09T17:48:46.184307+0000","last_became_peered":"2024-02-09T17:48:46.184307+0000","last_unstale":"2024-02-09T17:57:47.372709+0000","last_undegraded":"2024-02-09T17:57:47.372709+0000","last_fullsized":"2024-02-09T17:57:47.372709+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:56:40.524448+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:56:40.524448+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:57:59.812883+0000","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.3","version":"831'10885","reported_seq":11070,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-02-09T17:57:56.537123+0000","last_change":"2024-02-09T17:56:51.477774+0000","last_active":"2024-02-09T17:57:56.537123+0000","last_peered":"2024-02-09T17:57:56.537123+0000","last_clean":"2024-02-09T17:57:56.537123+0000","last_became_active":"2024-02-09T17:55:16.445048+0000","last_became_peered":"2024-02-09T17:55:16.445048+0000","last_unstale":"2024-02-09T17:57:56.537123+0000","last_undegraded":"2024-02-09T17:57:56.537123+0000","last_fullsized":"2024-02-09T17:57:56.537123+0000","mapping_epoch":785,"log_start":"796'800","ondisk_log_start":"796'800","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"820'7500","last_scrub_stamp":"2024-02-09T17:56:51.477633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:56:51.477633+0000","objects_scrubbed":7455,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:57:59.028115+0000","scrub_duration":2218,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303366144,"num_objects":4629,"num_object_clones":0,"num_object_copies":13887,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10885,"num_write_kb":496448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"63'139","reported_seq":1550,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-02-09T17:57:47.370986+0000","last_change":"2024-02-09T17:56:44.962532+0000","last_active":"2024-02-09T17:57:47.370986+0000","last_peered":"2024-02-09T17:57:47.370986+0000","last_clean":"2024-02-09T17:57:47.370986+0000","last_became_active":"2024-02-09T17:43:14.041291+0000","last_became_peered":"2024-02-09T17:43:14.041291+0000","last_unstale":"2024-02-09T17:57:47.370986+0000","last_undegraded":"2024-02-09T17:57:47.370986+0000","last_fullsized":"2024-02-09T17:57:47.370986+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:56:44.962386+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:56:44.962386+0000","last_clean_scrub_stamp":"2024-02-09T17:56:44.962386+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:58:01.148751+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.2","version":"831'10794","reported_seq":10976,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-02-09T17:57:53.593352+0000","last_change":"2024-02-09T17:56:46.079856+0000","last_active":"2024-02-09T17:57:53.593352+0000","last_peered":"2024-02-09T17:57:53.593352+0000","last_clean":"2024-02-09T17:57:53.593352+0000","last_became_active":"2024-02-09T17:55:16.913786+0000","last_became_peered":"2024-02-09T17:55:16.913786+0000","last_unstale":"2024-02-09T17:57:53.593352+0000","last_undegraded":"2024-02-09T17:57:53.593352+0000","last_fullsized":"2024-02-09T17:57:53.593352+0000","mapping_epoch":785,"log_start":"796'700","ondisk_log_start":"796'700","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"818'7186","last_scrub_stamp":"2024-02-09T17:56:46.079720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:56:46.079720+0000","objects_scrubbed":7152,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:57:56.349464+0000","scrub_duration":1659,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":307494912,"num_objects":4692,"num_object_clones":0,"num_object_copies":14076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10794,"num_write_kb":495552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,4],"acting":[10,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.b","version":"63'139","reported_seq":918,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-02-09T17:57:47.371670+0000","last_change":"2024-02-09T17:56:48.571204+0000","last_active":"2024-02-09T17:57:47.371670+0000","last_peered":"2024-02-09T17:57:47.371670+0000","last_clean":"2024-02-09T17:57:47.371670+0000","last_became_active":"2024-02-09T17:55:08.938203+0000","last_became_peered":"2024-02-09T17:55:08.938203+0000","last_unstale":"2024-02-09T17:57:47.371670+0000","last_undegraded":"2024-02-09T17:57:47.371670+0000","last_fullsized":"2024-02-09T17:57:47.371670+0000","mapping_epoch":781,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":782,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:56:48.571091+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:56:48.571091+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:57:51.562070+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.1","version":"831'10662","reported_seq":10843,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-02-09T17:57:54.686363+0000","last_change":"2024-02-09T17:56:46.138229+0000","last_active":"2024-02-09T17:57:54.686363+0000","last_peered":"2024-02-09T17:57:54.686363+0000","last_clean":"2024-02-09T17:57:54.686363+0000","last_became_active":"2024-02-09T17:55:16.205574+0000","last_became_peered":"2024-02-09T17:55:16.205574+0000","last_unstale":"2024-02-09T17:57:54.686363+0000","last_undegraded":"2024-02-09T17:57:54.686363+0000","last_fullsized":"2024-02-09T17:57:54.686363+0000","mapping_epoch":785,"log_start":"796'600","ondisk_log_start":"796'600","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"818'7081","last_scrub_stamp":"2024-02-09T17:56:46.137952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:56:46.137952+0000","objects_scrubbed":7048,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:58:14.615465+0000","scrub_duration":1634,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298713088,"num_objects":4558,"num_object_clones":0,"num_object_copies":13674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10662,"num_write_kb":487040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,2],"acting":[1,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"63'139","reported_seq":1138,"reported_epoch":831,"state":"active+undersized","last_fresh":"2024-02-09T17:57:47.371676+0000","last_change":"2024-02-09T17:56:40.455724+0000","last_active":"2024-02-09T17:57:47.371676+0000","last_peered":"2024-02-09T17:57:47.371676+0000","last_clean":"2024-02-09T17:46:43.974099+0000","last_became_active":"2024-02-09T17:56:40.455724+0000","last_became_peered":"2024-02-09T17:56:40.455724+0000","last_unstale":"2024-02-09T17:57:47.371676+0000","last_undegraded":"2024-02-09T17:57:47.371676+0000","last_fullsized":"2024-02-09T17:56:40.454196+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:10.313268+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:55:10.313268+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.0","version":"831'11194","reported_seq":11299,"reported_epoch":831,"state":"active+undersized+degraded","last_fresh":"2024-02-09T17:57:57.169914+0000","last_change":"2024-02-09T17:56:40.456476+0000","last_active":"2024-02-09T17:57:57.169914+0000","last_peered":"2024-02-09T17:57:57.169914+0000","last_clean":"2024-02-09T17:56:39.422623+0000","last_became_active":"2024-02-09T17:56:40.456476+0000","last_became_peered":"2024-02-09T17:56:40.456476+0000","last_unstale":"2024-02-09T17:57:57.169914+0000","last_undegraded":"2024-02-09T17:56:40.455373+0000","last_fullsized":"2024-02-09T17:56:40.455263+0000","mapping_epoch":815,"log_start":"798'1100","ondisk_log_start":"798'1100","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:57:58.952354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296222720,"num_objects":4520,"num_object_clones":0,"num_object_copies":13560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4520,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11194,"num_write_kb":502848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":["9","5"],"object_location_counts":[{"shards":"5,9","objects":4520}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.9","version":"63'139","reported_seq":2055,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-02-09T17:57:47.373247+0000","last_change":"2024-02-09T17:56:49.660561+0000","last_active":"2024-02-09T17:57:47.373247+0000","last_peered":"2024-02-09T17:57:47.373247+0000","last_clean":"2024-02-09T17:57:47.373247+0000","last_became_active":"2024-02-09T17:43:28.590237+0000","last_became_peered":"2024-02-09T17:43:28.590237+0000","last_unstale":"2024-02-09T17:57:47.373247+0000","last_undegraded":"2024-02-09T17:57:47.373247+0000","last_fullsized":"2024-02-09T17:57:47.373247+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:56:49.660423+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:56:49.660423+0000","last_clean_scrub_stamp":"2024-02-09T17:56:49.660423+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:58:19.063463+0000","scrub_duration":194,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.7","version":"831'10652","reported_seq":10836,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-02-09T17:57:54.666698+0000","last_change":"2024-02-09T17:56:51.664188+0000","last_active":"2024-02-09T17:57:54.666698+0000","last_peered":"2024-02-09T17:57:54.666698+0000","last_clean":"2024-02-09T17:57:54.666698+0000","last_became_active":"2024-02-09T17:55:16.206561+0000","last_became_peered":"2024-02-09T17:55:16.206561+0000","last_unstale":"2024-02-09T17:57:54.666698+0000","last_undegraded":"2024-02-09T17:57:54.666698+0000","last_fullsized":"2024-02-09T17:57:54.666698+0000","mapping_epoch":785,"log_start":"796'600","ondisk_log_start":"796'600","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"820'7359","last_scrub_stamp":"2024-02-09T17:56:51.664038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:56:51.664038+0000","objects_scrubbed":7309,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297271296,"num_objects":4536,"num_object_clones":0,"num_object_copies":13608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10652,"num_write_kb":486016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.e","version":"63'139","reported_seq":1507,"reported_epoch":831,"state":"active+undersized","last_fresh":"2024-02-09T17:57:47.365514+0000","last_change":"2024-02-09T17:56:40.457215+0000","last_active":"2024-02-09T17:57:47.365514+0000","last_peered":"2024-02-09T17:57:47.365514+0000","last_clean":"2024-02-09T17:56:36.466317+0000","last_became_active":"2024-02-09T17:56:40.457215+0000","last_became_peered":"2024-02-09T17:56:40.457215+0000","last_unstale":"2024-02-09T17:57:47.365514+0000","last_undegraded":"2024-02-09T17:57:47.365514+0000","last_fullsized":"2024-02-09T17:56:40.455199+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:08.044477+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:14.981453+0000","last_clean_scrub_stamp":"2024-02-09T17:55:08.044477+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:57:53.699576+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"831'11031","reported_seq":11214,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-02-09T17:57:56.519829+0000","last_change":"2024-02-09T17:56:43.410734+0000","last_active":"2024-02-09T17:57:56.519829+0000","last_peered":"2024-02-09T17:57:56.519829+0000","last_clean":"2024-02-09T17:57:56.519829+0000","last_became_active":"2024-02-09T17:55:16.445702+0000","last_became_peered":"2024-02-09T17:55:16.445702+0000","last_unstale":"2024-02-09T17:57:56.519829+0000","last_undegraded":"2024-02-09T17:57:56.519829+0000","last_fullsized":"2024-02-09T17:57:56.519829+0000","mapping_epoch":785,"log_start":"798'1000","ondisk_log_start":"798'1000","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"816'7145","last_scrub_stamp":"2024-02-09T17:56:43.410521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:56:43.410521+0000","objects_scrubbed":7092,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:57:57.523697+0000","scrub_duration":1999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302972928,"num_objects":4623,"num_object_clones":0,"num_object_copies":13869,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11031,"num_write_kb":500928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,7],"acting":[11,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.f","version":"63'139","reported_seq":1572,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-02-09T17:57:47.620773+0000","last_change":"2024-02-09T17:56:39.938089+0000","last_active":"2024-02-09T17:57:47.620773+0000","last_peered":"2024-02-09T17:57:47.620773+0000","last_clean":"2024-02-09T17:57:47.620773+0000","last_became_active":"2024-02-09T17:33:05.316100+0000","last_became_peered":"2024-02-09T17:33:05.316100+0000","last_unstale":"2024-02-09T17:57:47.620773+0000","last_undegraded":"2024-02-09T17:57:47.620773+0000","last_fullsized":"2024-02-09T17:57:47.620773+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:56:39.937959+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:48:34.651647+0000","last_clean_scrub_stamp":"2024-02-09T17:56:39.937959+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:57:49.103445+0000","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.5","version":"831'10809","reported_seq":10995,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-02-09T17:57:53.588614+0000","last_change":"2024-02-09T17:56:53.269202+0000","last_active":"2024-02-09T17:57:53.588614+0000","last_peered":"2024-02-09T17:57:53.588614+0000","last_clean":"2024-02-09T17:57:53.588614+0000","last_became_active":"2024-02-09T17:55:16.913201+0000","last_became_peered":"2024-02-09T17:55:16.913201+0000","last_unstale":"2024-02-09T17:57:53.588614+0000","last_undegraded":"2024-02-09T17:57:53.588614+0000","last_fullsized":"2024-02-09T17:57:53.588614+0000","mapping_epoch":785,"log_start":"796'800","ondisk_log_start":"796'800","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"820'7539","last_scrub_stamp":"2024-02-09T17:56:53.269036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:56:53.269036+0000","objects_scrubbed":7492,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:58:01.778816+0000","scrub_duration":1882,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304283648,"num_objects":4643,"num_object_clones":0,"num_object_copies":13929,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10809,"num_write_kb":494464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,1],"acting":[10,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.c","version":"63'139","reported_seq":846,"reported_epoch":831,"state":"active+undersized","last_fresh":"2024-02-09T17:57:47.620617+0000","last_change":"2024-02-09T17:56:40.452699+0000","last_active":"2024-02-09T17:57:47.620617+0000","last_peered":"2024-02-09T17:57:47.620617+0000","last_clean":"2024-02-09T17:39:31.468742+0000","last_became_active":"2024-02-09T17:56:40.452699+0000","last_became_peered":"2024-02-09T17:56:40.452699+0000","last_unstale":"2024-02-09T17:57:47.620617+0000","last_undegraded":"2024-02-09T17:57:47.620617+0000","last_fullsized":"2024-02-09T17:56:40.451638+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:07.304236+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:14.135321+0000","last_clean_scrub_stamp":"2024-02-09T17:55:07.304236+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:57:51.907425+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163840,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"831'10695","reported_seq":10878,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-02-09T17:57:57.112199+0000","last_change":"2024-02-09T17:56:50.221320+0000","last_active":"2024-02-09T17:57:57.112199+0000","last_peered":"2024-02-09T17:57:57.112199+0000","last_clean":"2024-02-09T17:57:57.112199+0000","last_became_active":"2024-02-09T17:55:16.444827+0000","last_became_peered":"2024-02-09T17:55:16.444827+0000","last_unstale":"2024-02-09T17:57:57.112199+0000","last_undegraded":"2024-02-09T17:57:57.112199+0000","last_fullsized":"2024-02-09T17:57:57.112199+0000","mapping_epoch":785,"log_start":"796'600","ondisk_log_start":"796'600","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"820'7206","last_scrub_stamp":"2024-02-09T17:56:50.221123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:56:50.221123+0000","objects_scrubbed":7150,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:58:08.990507+0000","scrub_duration":2292,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283181056,"num_objects":4321,"num_object_clones":0,"num_object_copies":12963,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10695,"num_write_kb":480512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.d","version":"63'139","reported_seq":1104,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-02-09T17:57:47.376371+0000","last_change":"2024-02-09T17:56:37.505698+0000","last_active":"2024-02-09T17:57:47.376371+0000","last_peered":"2024-02-09T17:57:47.376371+0000","last_clean":"2024-02-09T17:57:47.376371+0000","last_became_active":"2024-02-09T17:43:14.036759+0000","last_became_peered":"2024-02-09T17:43:14.036759+0000","last_unstale":"2024-02-09T17:57:47.376371+0000","last_undegraded":"2024-02-09T17:57:47.376371+0000","last_fullsized":"2024-02-09T17:57:47.376371+0000","mapping_epoch":454,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:56:37.505562+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:56:37.505562+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:57:51.107636+0000","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"63'139","reported_seq":1275,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-02-09T17:57:47.373234+0000","last_change":"2024-02-09T17:56:48.462272+0000","last_active":"2024-02-09T17:57:47.373234+0000","last_peered":"2024-02-09T17:57:47.373234+0000","last_clean":"2024-02-09T17:57:47.373234+0000","last_became_active":"2024-02-09T17:46:47.013920+0000","last_became_peered":"2024-02-09T17:46:47.013920+0000","last_unstale":"2024-02-09T17:57:47.373234+0000","last_undegraded":"2024-02-09T17:57:47.373234+0000","last_fullsized":"2024-02-09T17:57:47.373234+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:56:48.462167+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:48:35.170638+0000","last_clean_scrub_stamp":"2024-02-09T17:56:48.462167+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:57:55.792789+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"63'139","reported_seq":1545,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-02-09T17:57:47.360341+0000","last_change":"2024-02-09T17:56:45.479998+0000","last_active":"2024-02-09T17:57:47.360341+0000","last_peered":"2024-02-09T17:57:47.360341+0000","last_clean":"2024-02-09T17:57:47.360341+0000","last_became_active":"2024-02-09T17:55:21.231902+0000","last_became_peered":"2024-02-09T17:55:21.231902+0000","last_unstale":"2024-02-09T17:57:47.360341+0000","last_undegraded":"2024-02-09T17:57:47.360341+0000","last_fullsized":"2024-02-09T17:57:47.360341+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":791,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:56:45.479870+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:44:51.190789+0000","last_clean_scrub_stamp":"2024-02-09T17:56:45.479870+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:57:59.756499+0000","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.12","version":"63'139","reported_seq":1628,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-02-09T17:57:47.366878+0000","last_change":"2024-02-09T17:57:31.202700+0000","last_active":"2024-02-09T17:57:47.366878+0000","last_peered":"2024-02-09T17:57:47.366878+0000","last_clean":"2024-02-09T17:57:47.366878+0000","last_became_active":"2024-02-09T17:46:47.015622+0000","last_became_peered":"2024-02-09T17:46:47.015622+0000","last_unstale":"2024-02-09T17:57:47.366878+0000","last_undegraded":"2024-02-09T17:57:47.366878+0000","last_fullsized":"2024-02-09T17:57:47.366878+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:57:31.202562+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:57:31.202562+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:58:35.866210+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.13","version":"63'139","reported_seq":1247,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-02-09T17:57:47.367040+0000","last_change":"2024-02-09T17:57:44.410517+0000","last_active":"2024-02-09T17:57:47.367040+0000","last_peered":"2024-02-09T17:57:47.367040+0000","last_clean":"2024-02-09T17:57:47.367040+0000","last_became_active":"2024-02-09T17:41:14.521672+0000","last_became_peered":"2024-02-09T17:41:14.521672+0000","last_unstale":"2024-02-09T17:57:47.367040+0000","last_undegraded":"2024-02-09T17:57:47.367040+0000","last_fullsized":"2024-02-09T17:57:47.367040+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:57:44.410378+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:57:44.410378+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:58:56.127838+0000","scrub_duration":224,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":4793565208,"num_objects":73145,"num_object_clones":0,"num_object_copies":219435,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13460,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73145,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172995,"num_write_kb":7876417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14823534592,"data_stored":14823522376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161095,"ondisk_log_size":161095,"up":45,"acting":45,"num_store_stats":11},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":426528,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":88,"acting":88,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547057068,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2307548,"kb_used_data":1464212,"kb_used_omap":1,"kb_used_meta":843326,"kb_avail":91437604,"statfs":{"total":95995035648,"available":93632106496,"internally_reserved":0,"allocated":1499353088,"data_stored":1499087647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":863566276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:57:01 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.8819999999999999,"15min":1.5680000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.21199999999999999},"max":{"1min":4.1269999999999998,"5min":10.704000000000001,"15min":20.003},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.4990000000000001,"5min":1.851,"15min":1.3720000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":5.4539999999999997,"5min":9.6259999999999994,"15min":9.6259999999999994},"last":8.2249999999999996}]},{"osd":1,"last update":"Fri Feb 9 17:57:01 2024","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.748,"15min":1.5309999999999999},"min":{"1min":0.27400000000000002,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":3.8359999999999999,"5min":9.532,"15min":11.576000000000001},"last":8.2789999999999999},{"interface":"front","average":{"1min":2.0899999999999999,"5min":1.8520000000000001,"15min":1.712},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.25},"max":{"1min":12.765000000000001,"5min":12.765000000000001,"15min":14.287000000000001},"last":2.5}]},{"osd":2,"last update":"Fri Feb 9 17:57:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4009999999999998,"5min":1.669,"15min":1.5469999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":12.721,"5min":12.721,"15min":12.721},"last":0.40400000000000003},{"interface":"front","average":{"1min":1.984,"5min":1.343,"15min":1.236},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":5.0110000000000001,"5min":5.0789999999999997,"15min":5.0789999999999997},"last":0.379}]},{"osd":3,"last update":"Fri Feb 9 17:56:01 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":2.5,"15min":1.8740000000000001},"min":{"1min":0.47999999999999998,"5min":0.32200000000000001,"15min":0.248},"max":{"1min":5.1509999999999998,"5min":43.344000000000001,"15min":43.344000000000001},"last":4.5949999999999998},{"interface":"front","average":{"1min":1.556,"5min":2.5379999999999998,"15min":1.954},"min":{"1min":0.45300000000000001,"5min":0.375,"15min":0.33600000000000002},"max":{"1min":5.1769999999999996,"5min":43.378,"15min":43.378},"last":4.6189999999999998}]},{"osd":4,"last update":"Fri Feb 9 17:57:41 2024","interfaces":[{"interface":"back","average":{"1min":2.4300000000000002,"5min":2.4300000000000002,"15min":2.4300000000000002},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":12.818,"5min":12.818,"15min":12.818},"last":2.5259999999999998},{"interface":"front","average":{"1min":2.3969999999999998,"5min":2.3969999999999998,"15min":2.3969999999999998},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":12.893000000000001,"5min":12.893000000000001,"15min":12.893000000000001},"last":2.5750000000000002}]},{"osd":5,"last update":"Fri Feb 9 17:57:01 2024","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":1.45,"15min":1.6060000000000001},"min":{"1min":0.30499999999999999,"5min":0.29699999999999999,"15min":0.252},"max":{"1min":4.1589999999999998,"5min":11.119999999999999,"15min":18.878},"last":1.373},{"interface":"front","average":{"1min":1.9570000000000001,"5min":1.4390000000000001,"15min":1.702},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.28999999999999998},"max":{"1min":12.868,"5min":12.868,"15min":20.038},"last":2.6040000000000001}]},{"osd":6,"last update":"Fri Feb 9 17:57:01 2024","interfaces":[{"interface":"back","average":{"1min":1.8500000000000001,"5min":1.637,"15min":1.6950000000000001},"min":{"1min":0.39200000000000002,"5min":0.315,"15min":0.30399999999999999},"max":{"1min":11.885,"5min":11.885,"15min":12.362},"last":8.3439999999999994},{"interface":"front","average":{"1min":2.1070000000000002,"5min":1.633,"15min":1.798},"min":{"1min":0.36899999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":12.792,"5min":15.409000000000001,"15min":15.409000000000001},"last":2.6299999999999999}]},{"osd":7,"last update":"Fri Feb 9 17:57:01 2024","interfaces":[{"interface":"back","average":{"1min":2.0710000000000002,"5min":1.583,"15min":1.786},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.317},"max":{"1min":12.842000000000001,"5min":15.51,"15min":18.846},"last":2.5510000000000002},{"interface":"front","average":{"1min":0.98499999999999999,"5min":1.6120000000000001,"15min":1.9039999999999999},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.318},"max":{"1min":2.9260000000000002,"5min":10.911,"15min":24.373999999999999},"last":8.3789999999999996}]},{"osd":8,"last update":"Fri Feb 9 17:57:18 2024","interfaces":[{"interface":"back","average":{"1min":2.5649999999999999,"5min":2.1539999999999999,"15min":1.603},"min":{"1min":0.307,"5min":0.307,"15min":0.27900000000000003},"max":{"1min":8.9090000000000007,"5min":11.944000000000001,"15min":13.574},"last":8.3130000000000006},{"interface":"front","average":{"1min":2.4489999999999998,"5min":2.177,"15min":1.663},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.32700000000000001},"max":{"1min":6.8070000000000004,"5min":11.922000000000001,"15min":13.606},"last":8.4100000000000001}]},{"osd":9,"last update":"Fri Feb 9 17:57:50 2024","interfaces":[{"interface":"back","average":{"1min":1.9730000000000001,"5min":1.8680000000000001,"15min":1.7749999999999999},"min":{"1min":0.39000000000000001,"5min":0.32900000000000001,"15min":0.29399999999999998},"max":{"1min":10.247999999999999,"5min":22.613,"15min":22.613},"last":2.238},{"interface":"front","average":{"1min":2.0129999999999999,"5min":1.7030000000000001,"15min":1.5429999999999999},"min":{"1min":0.434,"5min":0.38300000000000001,"15min":0.34399999999999997},"max":{"1min":10.279,"5min":10.279,"15min":12.313000000000001},"last":2.274}]},{"osd":10,"last update":"Fri Feb 9 17:57:01 2024","interfaces":[{"interface":"back","average":{"1min":2.6520000000000001,"5min":2.5739999999999998,"15min":1.843},"min":{"1min":0.42899999999999999,"5min":0.41199999999999998,"15min":0.32700000000000001},"max":{"1min":11.962999999999999,"5min":17.373000000000001,"15min":17.373000000000001},"last":8.5269999999999992},{"interface":"front","average":{"1min":2.5680000000000001,"5min":2.0299999999999998,"15min":1.698},"min":{"1min":0.437,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":12.917,"5min":20.233000000000001,"15min":20.233000000000001},"last":3.1019999999999999}]}]},{"osd":10,"up_from":84,"seq":360777253300,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2514968,"kb_used_data":1789648,"kb_used_omap":1,"kb_used_meta":725310,"kb_avail":91230184,"statfs":{"total":95995035648,"available":93419708416,"internally_reserved":0,"allocated":1832599552,"data_stored":1832338183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":742717888},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:57:52 2024","interfaces":[{"interface":"back","average":{"1min":2.4809999999999999,"5min":1.9350000000000001,"15min":1.3600000000000001},"min":{"1min":0.27300000000000002,"5min":0.24199999999999999,"15min":0.23999999999999999},"max":{"1min":8.8040000000000003,"5min":9.5020000000000007,"15min":10.532},"last":8.8040000000000003},{"interface":"front","average":{"1min":2.3580000000000001,"5min":1.9950000000000001,"15min":1.3839999999999999},"min":{"1min":0.313,"5min":0.27100000000000002,"15min":0.248},"max":{"1min":8.8989999999999991,"5min":12.111000000000001,"15min":12.111000000000001},"last":4.694}]},{"osd":1,"last update":"Fri Feb 9 17:57:17 2024","interfaces":[{"interface":"back","average":{"1min":2.012,"5min":1.655,"15min":1.595},"min":{"1min":0.29099999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":7.0350000000000001,"5min":7.1360000000000001,"15min":7.1360000000000001},"last":0.26200000000000001},{"interface":"front","average":{"1min":2.2730000000000001,"5min":1.7090000000000001,"15min":1.615},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":7.077,"5min":7.1050000000000004,"15min":7.1050000000000004},"last":0.32800000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:57:22 2024","interfaces":[{"interface":"back","average":{"1min":3.5409999999999999,"5min":2.0659999999999998,"15min":1.7729999999999999},"min":{"1min":0.36899999999999999,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":22.652999999999999,"5min":22.652999999999999,"15min":22.652999999999999},"last":5.2350000000000003},{"interface":"front","average":{"1min":2.2440000000000002,"5min":1.958,"15min":1.7829999999999999},"min":{"1min":0.41299999999999998,"5min":0.28899999999999998,"15min":0.25600000000000001},"max":{"1min":9.532,"5min":9.532,"15min":38.868000000000002},"last":4.835}]},{"osd":3,"last update":"Fri Feb 9 17:55:09 2024","interfaces":[{"interface":"back","average":{"1min":2.282,"5min":2.1549999999999998,"15min":1.796},"min":{"1min":0.29999999999999999,"5min":0.27700000000000002,"15min":0.26100000000000001},"max":{"1min":6.9909999999999997,"5min":14.689,"15min":35.113},"last":0.65700000000000003},{"interface":"front","average":{"1min":1.296,"5min":2.4550000000000001,"15min":2.077},"min":{"1min":0.27800000000000002,"5min":0.249,"15min":0.249},"max":{"1min":3.5449999999999999,"5min":11.209,"15min":34.359000000000002},"last":0.54400000000000004}]},{"osd":4,"last update":"Fri Feb 9 17:57:48 2024","interfaces":[{"interface":"back","average":{"1min":1.603,"5min":1.1319999999999999,"15min":1.2889999999999999},"min":{"1min":0.39800000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":5.673,"5min":9.5850000000000009,"15min":11.705},"last":0.39300000000000002},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.1259999999999999,"15min":1.353},"min":{"1min":0.47499999999999998,"5min":0.33900000000000002,"15min":0.27000000000000002},"max":{"1min":5.7850000000000001,"5min":9.6630000000000003,"15min":11.744},"last":4.7350000000000003}]},{"osd":5,"last update":"Fri Feb 9 17:57:22 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":0.98899999999999999,"15min":1.353},"min":{"1min":0.32100000000000001,"5min":0.30099999999999999,"15min":0.29099999999999998},"max":{"1min":7.3380000000000001,"5min":7.3380000000000001,"15min":10.759},"last":0.35399999999999998},{"interface":"front","average":{"1min":2.0840000000000001,"5min":1.125,"15min":1.5549999999999999},"min":{"1min":0.40899999999999997,"5min":0.373,"15min":0.33500000000000002},"max":{"1min":9.6370000000000005,"5min":9.6370000000000005,"15min":11.779},"last":4.7699999999999996}]},{"osd":6,"last update":"Fri Feb 9 17:57:27 2024","interfaces":[{"interface":"back","average":{"1min":1.641,"5min":1.1299999999999999,"15min":1.492},"min":{"1min":0.40500000000000003,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":7.3929999999999998,"5min":9.2599999999999998,"15min":10.962},"last":0.49199999999999999},{"interface":"front","average":{"1min":2.2000000000000002,"5min":1.3069999999999999,"15min":1.5760000000000001},"min":{"1min":0.45800000000000002,"5min":0.35099999999999998,"15min":0.32600000000000001},"max":{"1min":9.6099999999999994,"5min":9.6099999999999994,"15min":12.414},"last":4.8029999999999999}]},{"osd":7,"last update":"Fri Feb 9 17:57:17 2024","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.1499999999999999,"15min":2.7040000000000002},"min":{"1min":0.35399999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":5.859,"5min":7.3250000000000002,"15min":11.288},"last":4.1349999999999998},{"interface":"front","average":{"1min":1.514,"5min":1.1850000000000001,"15min":2.73},"min":{"1min":0.432,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":5.891,"5min":7.3630000000000004,"15min":10.736000000000001},"last":4.109}]},{"osd":8,"last update":"Fri Feb 9 17:57:17 2024","interfaces":[{"interface":"back","average":{"1min":4.6920000000000002,"5min":2.3220000000000001,"15min":1.4730000000000001},"min":{"1min":0.44400000000000001,"5min":0.32400000000000001,"15min":0.31},"max":{"1min":22.454000000000001,"5min":22.454000000000001,"15min":22.454000000000001},"last":0.38100000000000001},{"interface":"front","average":{"1min":3.2120000000000002,"5min":1.72,"15min":1.506},"min":{"1min":0.60199999999999998,"5min":0.35299999999999998,"15min":0.30599999999999999},"max":{"1min":22.824999999999999,"5min":22.824999999999999,"15min":22.824999999999999},"last":0.42399999999999999}]},{"osd":9,"last update":"Fri Feb 9 17:57:22 2024","interfaces":[{"interface":"back","average":{"1min":2.9060000000000001,"5min":2.218,"15min":1.6240000000000001},"min":{"1min":0.40999999999999998,"5min":0.34300000000000003,"15min":0.33300000000000002},"max":{"1min":7.9720000000000004,"5min":13.298,"15min":13.298},"last":1.238},{"interface":"front","average":{"1min":2.5,"5min":1.988,"15min":1.71},"min":{"1min":0.54900000000000004,"5min":0.39600000000000002,"15min":0.36099999999999999},"max":{"1min":7.5119999999999996,"5min":8.3420000000000005,"15min":10.622999999999999},"last":0.41399999999999998}]},{"osd":11,"last update":"Fri Feb 9 17:57:52 2024","interfaces":[{"interface":"back","average":{"1min":2.7650000000000001,"5min":1.867,"15min":1.5269999999999999},"min":{"1min":0.46300000000000002,"5min":0.36799999999999999,"15min":0.35399999999999998},"max":{"1min":10.625999999999999,"5min":10.625999999999999,"15min":23.579000000000001},"last":4.9589999999999996},{"interface":"front","average":{"1min":2.7890000000000001,"5min":1.9390000000000001,"15min":1.587},"min":{"1min":0.59699999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":8.9260000000000002,"5min":22.725999999999999,"15min":23.619},"last":4.9820000000000002}]}]},{"osd":9,"up_from":77,"seq":330712482231,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2526296,"kb_used_data":1705168,"kb_used_omap":1,"kb_used_meta":821118,"kb_avail":91218856,"statfs":{"total":95995035648,"available":93408108544,"internally_reserved":0,"allocated":1746092032,"data_stored":1745830663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":840825280},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:57:23 2024","interfaces":[{"interface":"back","average":{"1min":1.7569999999999999,"5min":1.6739999999999999,"15min":1.4419999999999999},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.215},"max":{"1min":18.623999999999999,"5min":18.623999999999999,"15min":18.623999999999999},"last":5.931},{"interface":"front","average":{"1min":1.744,"5min":1.474,"15min":1.417},"min":{"1min":0.27300000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.375,"5min":7.6619999999999999,"15min":11.596},"last":0.39400000000000002}]},{"osd":1,"last update":"Fri Feb 9 17:57:06 2024","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":2.2469999999999999,"15min":1.6299999999999999},"min":{"1min":0.247,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":4.2009999999999996,"5min":61.031999999999996,"15min":61.031999999999996},"last":0.255},{"interface":"front","average":{"1min":1.169,"5min":2.4820000000000002,"15min":1.7310000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.25},"max":{"1min":6.3019999999999996,"5min":61.970999999999997,"15min":61.970999999999997},"last":5.9630000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:57:42 2024","interfaces":[{"interface":"back","average":{"1min":1.573,"5min":1.6619999999999999,"15min":1.782},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.26900000000000002},"max":{"1min":6.335,"5min":9.8689999999999998,"15min":10.657999999999999},"last":0.42499999999999999},{"interface":"front","average":{"1min":2.3050000000000002,"5min":1.9419999999999999,"15min":1.7270000000000001},"min":{"1min":0.373,"5min":0.29499999999999998,"15min":0.248},"max":{"1min":9.9239999999999995,"5min":11.35,"15min":11.35},"last":5.883}]},{"osd":3,"last update":"Fri Feb 9 17:56:32 2024","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":2.1299999999999999,"15min":1.496},"min":{"1min":0.35799999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.5289999999999999,"5min":10.537000000000001,"15min":10.537000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.86599999999999999,"5min":2.258,"15min":1.1659999999999999},"min":{"1min":0.40999999999999998,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":2.891,"5min":10.388,"15min":10.388},"last":0.53400000000000003}]},{"osd":4,"last update":"Fri Feb 9 17:57:06 2024","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":1.5669999999999999,"15min":1.5509999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26800000000000002},"max":{"1min":6.2729999999999997,"5min":19.414000000000001,"15min":19.414000000000001},"last":5.9960000000000004},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.415,"15min":1.323},"min":{"1min":0.31,"5min":0.30399999999999999,"15min":0.26900000000000002},"max":{"1min":4.1289999999999996,"5min":19.224,"15min":19.224},"last":0.30599999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:57:17 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.341,"15min":1.7270000000000001},"min":{"1min":0.40600000000000003,"5min":0.35499999999999998,"15min":0.26900000000000002},"max":{"1min":6.407,"5min":9.7729999999999997,"15min":77.924000000000007},"last":0.44900000000000001},{"interface":"front","average":{"1min":1.26,"5min":1.0960000000000001,"15min":1.704},"min":{"1min":0.316,"5min":0.316,"15min":0.28000000000000003},"max":{"1min":6.1980000000000004,"5min":7.4909999999999997,"15min":77.899000000000001},"last":6.0629999999999997}]},{"osd":6,"last update":"Fri Feb 9 17:57:06 2024","interfaces":[{"interface":"back","average":{"1min":1.6599999999999999,"5min":1.514,"15min":1.5660000000000001},"min":{"1min":0.42999999999999999,"5min":0.33400000000000002,"15min":0.30199999999999999},"max":{"1min":6.4370000000000003,"5min":19.375,"15min":19.375},"last":0.54000000000000004},{"interface":"front","average":{"1min":1.643,"5min":1.452,"15min":1.744},"min":{"1min":0.45800000000000002,"5min":0.34999999999999998,"15min":0.33500000000000002},"max":{"1min":6.5199999999999996,"5min":19.178000000000001,"15min":19.178000000000001},"last":0.51800000000000002}]},{"osd":7,"last update":"Fri Feb 9 17:57:42 2024","interfaces":[{"interface":"back","average":{"1min":2.1480000000000001,"5min":2.1480000000000001,"15min":2.1480000000000001},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":10.021000000000001,"5min":10.021000000000001,"15min":10.021000000000001},"last":0.49299999999999999},{"interface":"front","average":{"1min":1.9570000000000001,"5min":1.9570000000000001,"15min":1.9570000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":11.138,"5min":11.138,"15min":11.138},"last":6.0259999999999998}]},{"osd":8,"last update":"Fri Feb 9 17:57:23 2024","interfaces":[{"interface":"back","average":{"1min":1.6419999999999999,"5min":1.4319999999999999,"15min":1.2250000000000001},"min":{"1min":0.377,"5min":0.308,"15min":0.29599999999999999},"max":{"1min":6.5469999999999997,"5min":9.4930000000000003,"15min":10.744},"last":0.56899999999999995},{"interface":"front","average":{"1min":1.7370000000000001,"5min":1.5189999999999999,"15min":1.409},"min":{"1min":0.50600000000000001,"5min":0.35999999999999999,"15min":0.32200000000000001},"max":{"1min":6.5750000000000002,"5min":9.5289999999999999,"15min":12.885999999999999},"last":0.59099999999999997}]},{"osd":10,"last update":"Fri Feb 9 17:57:32 2024","interfaces":[{"interface":"back","average":{"1min":2.964,"5min":1.931,"15min":1.847},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.312},"max":{"1min":8.6969999999999992,"5min":10.333,"15min":20.773},"last":2.6779999999999999},{"interface":"front","average":{"1min":4.0999999999999996,"5min":1.998,"15min":1.5609999999999999},"min":{"1min":0.43099999999999999,"5min":0.35999999999999999,"15min":0.34300000000000003},"max":{"1min":18.579999999999998,"5min":18.579999999999998,"15min":20.637},"last":11.586}]},{"osd":11,"last update":"Fri Feb 9 17:57:32 2024","interfaces":[{"interface":"back","average":{"1min":2.738,"5min":1.917,"15min":1.617},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.33000000000000002},"max":{"1min":11.195,"5min":11.324,"15min":11.324},"last":0.65400000000000003},{"interface":"front","average":{"1min":3.133,"5min":2.1019999999999999,"15min":1.5800000000000001},"min":{"1min":0.624,"5min":0.41599999999999998,"15min":0.36499999999999999},"max":{"1min":11.737,"5min":11.737,"15min":11.737},"last":11.627000000000001}]}]},{"osd":8,"up_from":171,"seq":734439407973,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1364952,"kb_used_data":904464,"kb_used_omap":212,"kb_used_meta":460267,"kb_avail":92380200,"statfs":{"total":95995035648,"available":94597324800,"internally_reserved":0,"allocated":926171136,"data_stored":925909767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217579,"internal_metadata":471313941},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:57:16 2024","interfaces":[{"interface":"back","average":{"1min":3.0099999999999998,"5min":2.0640000000000001,"15min":1.3979999999999999},"min":{"1min":0.26400000000000001,"5min":0.254,"15min":0.224},"max":{"1min":9.9390000000000001,"5min":10.832000000000001,"15min":12.712999999999999},"last":4.0289999999999999},{"interface":"front","average":{"1min":1.7250000000000001,"5min":1.6020000000000001,"15min":1.401},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23100000000000001},"max":{"1min":6.617,"5min":9.25,"15min":12.725},"last":4.319}]},{"osd":1,"last update":"Fri Feb 9 17:57:15 2024","interfaces":[{"interface":"back","average":{"1min":2.6360000000000001,"5min":2.044,"15min":1.504},"min":{"1min":0.27000000000000002,"5min":0.23799999999999999,"15min":0.22700000000000001},"max":{"1min":6.8970000000000002,"5min":8.2550000000000008,"15min":8.718},"last":4.5},{"interface":"front","average":{"1min":1.367,"5min":2.1240000000000001,"15min":1.593},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.24099999999999999},"max":{"1min":6.5819999999999999,"5min":21.399000000000001,"15min":21.399000000000001},"last":3.9950000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:57:16 2024","interfaces":[{"interface":"back","average":{"1min":3.0299999999999998,"5min":1.4850000000000001,"15min":1.2270000000000001},"min":{"1min":0.33800000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.9260000000000002,"5min":6.9260000000000002,"15min":6.9260000000000002},"last":4.5670000000000002},{"interface":"front","average":{"1min":2.024,"5min":1.8049999999999999,"15min":1.7689999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":10.026999999999999,"5min":10.026999999999999,"15min":10.026999999999999},"last":3.956}]},{"osd":3,"last update":"Fri Feb 9 17:56:20 2024","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":2.552,"15min":2.8039999999999998},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":5.2560000000000002,"5min":15.654,"15min":15.654},"last":6.5350000000000001},{"interface":"front","average":{"1min":1.306,"5min":2.3149999999999999,"15min":2.0150000000000001},"min":{"1min":0.34300000000000003,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":5.2759999999999998,"5min":15.609999999999999,"15min":15.609999999999999},"last":6.5599999999999996}]},{"osd":4,"last update":"Fri Feb 9 17:57:15 2024","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.153,"15min":1.244},"min":{"1min":0.29499999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":6.6710000000000003,"5min":8.7959999999999994,"15min":8.7959999999999994},"last":4.1689999999999996},{"interface":"front","average":{"1min":1.403,"5min":1.1579999999999999,"15min":1.409},"min":{"1min":0.32600000000000001,"5min":0.32300000000000001,"15min":0.28000000000000003},"max":{"1min":6.7439999999999998,"5min":8.8780000000000001,"15min":9.6140000000000008},"last":4.1139999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:57:39 2024","interfaces":[{"interface":"back","average":{"1min":1.794,"5min":1.411,"15min":1.5609999999999999},"min":{"1min":0.312,"5min":0.28399999999999997,"15min":0.27400000000000002},"max":{"1min":6.2690000000000001,"5min":9.3550000000000004,"15min":9.657},"last":4.2839999999999998},{"interface":"front","average":{"1min":1.619,"5min":1.298,"15min":1.401},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.29899999999999999},"max":{"1min":10.093,"5min":10.093,"15min":10.093},"last":4.1420000000000003}]},{"osd":6,"last update":"Fri Feb 9 17:57:15 2024","interfaces":[{"interface":"back","average":{"1min":2.0670000000000002,"5min":1.6200000000000001,"15min":1.665},"min":{"1min":0.37,"5min":0.32700000000000001,"15min":0.29899999999999999},"max":{"1min":6.2960000000000003,"5min":9.7539999999999996,"15min":13.031000000000001},"last":4.5380000000000003},{"interface":"front","average":{"1min":2.7040000000000002,"5min":1.613,"15min":1.6160000000000001},"min":{"1min":0.371,"5min":0.311,"15min":0.311},"max":{"1min":6.9560000000000004,"5min":10.178000000000001,"15min":12.157999999999999},"last":4.3760000000000003}]},{"osd":7,"last update":"Fri Feb 9 17:57:15 2024","interfaces":[{"interface":"back","average":{"1min":1.708,"5min":1.4770000000000001,"15min":1.726},"min":{"1min":0.40699999999999997,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":6.6950000000000003,"5min":9.3130000000000006,"15min":11.278},"last":4.0880000000000001},{"interface":"front","average":{"1min":2.1080000000000001,"5min":1.5289999999999999,"15min":1.708},"min":{"1min":0.39600000000000002,"5min":0.309,"15min":0.309},"max":{"1min":6.3209999999999997,"5min":9.7110000000000003,"15min":11.313000000000001},"last":4.3479999999999999}]},{"osd":9,"last update":"Fri Feb 9 17:57:15 2024","interfaces":[{"interface":"back","average":{"1min":3.7120000000000002,"5min":2.3300000000000001,"15min":1.7629999999999999},"min":{"1min":0.42199999999999999,"5min":0.32700000000000001,"15min":0.26800000000000002},"max":{"1min":10.483000000000001,"5min":16.358000000000001,"15min":45.616},"last":4.6529999999999996},{"interface":"front","average":{"1min":2.7440000000000002,"5min":2.2149999999999999,"15min":1.7529999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.314},"max":{"1min":6.8449999999999998,"5min":16.422999999999998,"15min":45.718000000000004},"last":4.0590000000000002}]},{"osd":10,"last update":"Fri Feb 9 17:57:15 2024","interfaces":[{"interface":"back","average":{"1min":3.762,"5min":2.0190000000000001,"15min":1.3979999999999999},"min":{"1min":0.41899999999999998,"5min":0.34599999999999997,"15min":0.32300000000000001},"max":{"1min":16.356000000000002,"5min":16.356000000000002,"15min":16.356000000000002},"last":5.117},{"interface":"front","average":{"1min":4.1840000000000002,"5min":2.3959999999999999,"15min":1.6759999999999999},"min":{"1min":0.498,"5min":0.41799999999999998,"15min":0.34300000000000003},"max":{"1min":16.309999999999999,"5min":16.309999999999999,"15min":16.309999999999999},"last":5.6660000000000004}]},{"osd":11,"last update":"Fri Feb 9 17:57:49 2024","interfaces":[{"interface":"back","average":{"1min":1.95,"5min":1.95,"15min":1.95},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":5.5739999999999998,"5min":5.5739999999999998,"15min":5.5739999999999998},"last":4.4050000000000002},{"interface":"front","average":{"1min":1.9870000000000001,"5min":1.9870000000000001,"15min":1.9870000000000001},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":5.6120000000000001,"5min":5.6120000000000001,"15min":5.6120000000000001},"last":4.4359999999999999}]}]},{"osd":7,"up_from":57,"seq":244813136319,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1348900,"kb_used_data":591644,"kb_used_omap":1,"kb_used_meta":757246,"kb_avail":92396252,"statfs":{"total":95995035648,"available":94613762048,"internally_reserved":0,"allocated":605843456,"data_stored":605574463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":775420362},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:57:50 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":1.079,"15min":1.6870000000000001},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.249},"max":{"1min":4.3769999999999998,"5min":7.032,"15min":101.167},"last":0.318},{"interface":"front","average":{"1min":0.51800000000000002,"5min":1.0369999999999999,"15min":1.6160000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.24099999999999999},"max":{"1min":2.1829999999999998,"5min":7.0170000000000003,"15min":101.07599999999999},"last":0.50900000000000001}]},{"osd":1,"last update":"Fri Feb 9 17:57:50 2024","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":1.498,"15min":1.635},"min":{"1min":0.30299999999999999,"5min":0.28299999999999997,"15min":0.24299999999999999},"max":{"1min":8.7050000000000001,"5min":17.314,"15min":17.314},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.39000000000000001,"5min":1.1799999999999999,"15min":1.6659999999999999},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.65900000000000003,"5min":7.5949999999999998,"15min":12.359},"last":0.33800000000000002}]},{"osd":2,"last update":"Fri Feb 9 17:57:50 2024","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":1.268,"15min":1.48},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.25},"max":{"1min":2.9359999999999999,"5min":9.8279999999999994,"15min":9.8279999999999994},"last":0.44400000000000001},{"interface":"front","average":{"1min":1.194,"5min":1.359,"15min":1.6619999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.29499999999999998},"max":{"1min":8.5690000000000008,"5min":8.5690000000000008,"15min":11.968},"last":0.52700000000000002}]},{"osd":3,"last update":"Fri Feb 9 17:56:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":1.7529999999999999,"15min":2.1960000000000002},"min":{"1min":0.41599999999999998,"5min":0.32800000000000001,"15min":0.25600000000000001},"max":{"1min":2.0179999999999998,"5min":11.112,"15min":78.644999999999996},"last":1.72},{"interface":"front","average":{"1min":0.68200000000000005,"5min":1.7130000000000001,"15min":2.3170000000000002},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.33800000000000002},"max":{"1min":2.0830000000000002,"5min":14.592000000000001,"15min":78.414000000000001},"last":3.1070000000000002}]},{"osd":4,"last update":"Fri Feb 9 17:57:50 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.61399999999999999,"15min":1.385},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.25600000000000001},"max":{"1min":4.3330000000000002,"5min":4.3330000000000002,"15min":86.813000000000002},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.54600000000000004,"15min":1.5449999999999999},"min":{"1min":0.26600000000000001,"5min":0.26300000000000001,"15min":0.23899999999999999},"max":{"1min":1.202,"5min":2.028,"15min":86.751000000000005},"last":0.46400000000000002}]},{"osd":5,"last update":"Fri Feb 9 17:57:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.65600000000000003,"15min":1.3420000000000001},"min":{"1min":0.313,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.91400000000000003,"5min":4.0469999999999997,"15min":9.6590000000000007},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.72099999999999997,"15min":1.5269999999999999},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.31900000000000001},"max":{"1min":1.3220000000000001,"5min":4.0679999999999996,"15min":15.228999999999999},"last":0.48499999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:57:16 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.77400000000000002,"15min":1.603},"min":{"1min":0.36099999999999999,"5min":0.311,"15min":0.311},"max":{"1min":0.98699999999999999,"5min":5.8550000000000004,"15min":15.275},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.71199999999999997,"15min":1.4299999999999999},"min":{"1min":0.44500000000000001,"5min":0.35899999999999999,"15min":0.29199999999999998},"max":{"1min":0.93700000000000006,"5min":3.383,"15min":18.236999999999998},"last":0.54400000000000004}]},{"osd":8,"last update":"Fri Feb 9 17:57:10 2024","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.2889999999999999,"15min":1.514},"min":{"1min":0.43099999999999999,"5min":0.39600000000000002,"15min":0.33100000000000002},"max":{"1min":7.5410000000000004,"5min":9.2249999999999996,"15min":11.81},"last":4.8630000000000004},{"interface":"front","average":{"1min":1.694,"5min":1.3049999999999999,"15min":1.538},"min":{"1min":0.432,"5min":0.375,"15min":0.37},"max":{"1min":7.532,"5min":7.532,"15min":12.058},"last":4.774}]},{"osd":9,"last update":"Fri Feb 9 17:57:01 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":1.542,"15min":1.6830000000000001},"min":{"1min":0.50800000000000001,"5min":0.44800000000000001,"15min":0.32700000000000001},"max":{"1min":2.9060000000000001,"5min":7.3369999999999997,"15min":11.941000000000001},"last":4.9379999999999997},{"interface":"front","average":{"1min":1.089,"5min":1.4970000000000001,"15min":1.7370000000000001},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40000000000000002},"max":{"1min":5.5609999999999999,"5min":9.8670000000000009,"15min":15.852},"last":6.4169999999999998}]},{"osd":10,"last update":"Fri Feb 9 17:57:30 2024","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.304,"15min":1.4410000000000001},"min":{"1min":0.48899999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":3.0139999999999998,"5min":8.4969999999999999,"15min":9.5350000000000001},"last":0.57899999999999996},{"interface":"front","average":{"1min":1.8220000000000001,"5min":1.552,"15min":1.829},"min":{"1min":0.64500000000000002,"5min":0.40400000000000003,"15min":0.38900000000000001},"max":{"1min":4.9249999999999998,"5min":7.423,"15min":11.878},"last":0.56499999999999995}]},{"osd":11,"last update":"Fri Feb 9 17:57:50 2024","interfaces":[{"interface":"back","average":{"1min":1.5249999999999999,"5min":1.1859999999999999,"15min":1.5640000000000001},"min":{"1min":0.49199999999999999,"5min":0.375,"15min":0.35999999999999999},"max":{"1min":4.8040000000000003,"5min":5.0650000000000004,"15min":11.961},"last":6.2640000000000002},{"interface":"front","average":{"1min":1.732,"5min":1.569,"15min":1.7050000000000001},"min":{"1min":0.58099999999999996,"5min":0.48999999999999999,"15min":0.438},"max":{"1min":6.1870000000000003,"5min":12.243,"15min":12.423999999999999},"last":0.60999999999999999}]}]},{"osd":6,"up_from":49,"seq":210453397958,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":682024,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":93063128,"statfs":{"total":95995035648,"available":95296643072,"internally_reserved":0,"allocated":999424,"data_stored":738055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":697235910},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:57:37 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":1.1539999999999999,"15min":1.4730000000000001},"min":{"1min":0.28399999999999997,"5min":0.23999999999999999,"15min":0.22800000000000001},"max":{"1min":4.5270000000000001,"5min":10.356,"15min":10.481999999999999},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.98299999999999998,"15min":1.532},"min":{"1min":0.307,"5min":0.28000000000000003,"15min":0.24399999999999999},"max":{"1min":3.137,"5min":8.9459999999999997,"15min":10.436},"last":0.35899999999999999}]},{"osd":1,"last update":"Fri Feb 9 17:57:37 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.0880000000000001,"15min":2.1899999999999999},"min":{"1min":0.32000000000000001,"5min":0.27600000000000002,"15min":0.25},"max":{"1min":3.9940000000000002,"5min":9.3200000000000003,"15min":79.945999999999998},"last":3.218},{"interface":"front","average":{"1min":1.276,"5min":1.1459999999999999,"15min":2.3130000000000002},"min":{"1min":0.41499999999999998,"5min":0.35799999999999998,"15min":0.28599999999999998},"max":{"1min":4.0170000000000003,"5min":9.3510000000000009,"15min":80.003},"last":3.2469999999999999}]},{"osd":2,"last update":"Fri Feb 9 17:57:37 2024","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":1.1419999999999999,"15min":2.0710000000000002},"min":{"1min":0.38800000000000001,"5min":0.31900000000000001,"15min":0.28100000000000003},"max":{"1min":4.0259999999999998,"5min":5.657,"15min":84.912999999999997},"last":1.1879999999999999},{"interface":"front","average":{"1min":0.93400000000000005,"5min":1.1799999999999999,"15min":2.1120000000000001},"min":{"1min":0.47399999999999998,"5min":0.34999999999999998,"15min":0.34000000000000002},"max":{"1min":4.0540000000000003,"5min":5.6849999999999996,"15min":84.942999999999998},"last":1.22}]},{"osd":3,"last update":"Fri Feb 9 17:55:07 2024","interfaces":[{"interface":"back","average":{"1min":1.8240000000000001,"5min":1.5389999999999999,"15min":1.4910000000000001},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":9.0129999999999999,"5min":9.0129999999999999,"15min":9.0129999999999999},"last":0.54500000000000004},{"interface":"front","average":{"1min":1.841,"5min":1.8169999999999999,"15min":1.8129999999999999},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":6.6769999999999996,"5min":8.9410000000000007,"15min":8.9410000000000007},"last":0.48399999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:57:37 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.62,"15min":1.631},"min":{"1min":0.28999999999999998,"5min":0.26600000000000001,"15min":0.26200000000000001},"max":{"1min":3.0299999999999998,"5min":8.1349999999999998,"15min":15.061999999999999},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.68500000000000005,"15min":1.7410000000000001},"min":{"1min":0.34100000000000003,"5min":0.33200000000000002,"15min":0.29299999999999998},"max":{"1min":3.0649999999999999,"5min":8.1809999999999992,"15min":15.118},"last":0.42599999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:57:37 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.67200000000000004,"15min":1.6930000000000001},"min":{"1min":0.312,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80600000000000005,"5min":6.5800000000000001,"15min":14.151999999999999},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.623,"15min":1.6599999999999999},"min":{"1min":0.378,"5min":0.35699999999999998,"15min":0.26100000000000001},"max":{"1min":0.83199999999999996,"5min":3.8450000000000002,"15min":18.315999999999999},"last":0.55000000000000004}]},{"osd":7,"last update":"Fri Feb 9 17:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.753,"15min":1.7270000000000001},"min":{"1min":0.31,"5min":0.29499999999999998,"15min":0.28699999999999998},"max":{"1min":0.96599999999999997,"5min":5.4969999999999999,"15min":23.265999999999998},"last":0.438},{"interface":"front","average":{"1min":0.497,"5min":0.72299999999999998,"15min":1.4870000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.71699999999999997,"5min":5.0670000000000002,"15min":12.967000000000001},"last":0.371}]},{"osd":8,"last update":"Fri Feb 9 17:57:30 2024","interfaces":[{"interface":"back","average":{"1min":2.25,"5min":1.2929999999999999,"15min":1.1459999999999999},"min":{"1min":0.36799999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":6.5620000000000003,"5min":8.1500000000000004,"15min":9.9480000000000004},"last":1.6080000000000001},{"interface":"front","average":{"1min":2.3220000000000001,"5min":1.4350000000000001,"15min":1.2},"min":{"1min":0.503,"5min":0.41399999999999998,"15min":0.40400000000000003},"max":{"1min":6.548,"5min":8.2490000000000006,"15min":11.316000000000001},"last":3.75}]},{"osd":9,"last update":"Fri Feb 9 17:57:47 2024","interfaces":[{"interface":"back","average":{"1min":1.8069999999999999,"5min":1.74,"15min":1.7789999999999999},"min":{"1min":0.52400000000000002,"5min":0.41799999999999998,"15min":0.32600000000000001},"max":{"1min":7.6669999999999998,"5min":36.606000000000002,"15min":36.606000000000002},"last":4.9550000000000001},{"interface":"front","average":{"1min":2.0329999999999999,"5min":1.974,"15min":1.956},"min":{"1min":0.59799999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":4.7629999999999999,"5min":34.408000000000001,"15min":34.408000000000001},"last":3.9239999999999999}]},{"osd":10,"last update":"Fri Feb 9 17:57:12 2024","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.494,"15min":1.8400000000000001},"min":{"1min":0.45000000000000001,"5min":0.36599999999999999,"15min":0.34399999999999997},"max":{"1min":5.7439999999999998,"5min":21.364999999999998,"15min":25.370000000000001},"last":4.2279999999999998},{"interface":"front","average":{"1min":1.337,"5min":1.478,"15min":1.905},"min":{"1min":0.48199999999999998,"5min":0.40100000000000002,"15min":0.35099999999999998},"max":{"1min":5.7759999999999998,"5min":21.408999999999999,"15min":26.027999999999999},"last":4.2469999999999999}]},{"osd":11,"last update":"Fri Feb 9 17:57:37 2024","interfaces":[{"interface":"back","average":{"1min":1.7190000000000001,"5min":1.9379999999999999,"15min":1.9359999999999999},"min":{"1min":0.48399999999999999,"5min":0.44400000000000001,"15min":0.372},"max":{"1min":3.996,"5min":35.912999999999997,"15min":35.912999999999997},"last":3.3700000000000001},{"interface":"front","average":{"1min":1.754,"5min":1.9950000000000001,"15min":1.956},"min":{"1min":0.51900000000000002,"5min":0.48699999999999999,"15min":0.38800000000000001},"max":{"1min":4.0380000000000003,"5min":36.156999999999996,"15min":36.156999999999996},"last":3.4100000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575326,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2720536,"kb_used_data":1782800,"kb_used_omap":1,"kb_used_meta":937726,"kb_avail":91024616,"statfs":{"total":95995035648,"available":93209206784,"internally_reserved":0,"allocated":1825587200,"data_stored":1825325831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":960231877},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:56:53 2024","interfaces":[{"interface":"back","average":{"1min":1.4399999999999999,"5min":1.849,"15min":1.5289999999999999},"min":{"1min":0.246,"5min":0.19,"15min":0.19},"max":{"1min":7.0990000000000002,"5min":14.885999999999999,"15min":14.885999999999999},"last":0.27900000000000003},{"interface":"front","average":{"1min":1.355,"5min":2.0409999999999999,"15min":1.7450000000000001},"min":{"1min":0.27400000000000002,"5min":0.246,"15min":0.214},"max":{"1min":5.391,"5min":15.411,"15min":33.389000000000003},"last":0.42599999999999999}]},{"osd":2,"last update":"Fri Feb 9 17:57:22 2024","interfaces":[{"interface":"back","average":{"1min":1.6020000000000001,"5min":1.6519999999999999,"15min":1.726},"min":{"1min":0.30099999999999999,"5min":0.28899999999999998,"15min":0.222},"max":{"1min":6.5110000000000001,"5min":13.173,"15min":14.83},"last":0.37},{"interface":"front","average":{"1min":1.2709999999999999,"5min":1.5089999999999999,"15min":1.5489999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.222},"max":{"1min":5.79,"5min":11.103,"15min":11.103},"last":0.51900000000000002}]},{"osd":3,"last update":"Fri Feb 9 17:56:07 2024","interfaces":[{"interface":"back","average":{"1min":1.5009999999999999,"5min":3.1970000000000001,"15min":3.5990000000000002},"min":{"1min":0.28899999999999998,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":19.917999999999999,"5min":19.917999999999999,"15min":19.917999999999999},"last":1.2789999999999999},{"interface":"front","average":{"1min":1.7,"5min":3.6589999999999998,"15min":4.1470000000000002},"min":{"1min":0.313,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":20.175999999999998,"5min":20.175999999999998,"15min":20.175999999999998},"last":0.41599999999999998}]},{"osd":4,"last update":"Fri Feb 9 17:56:59 2024","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.1739999999999999,"15min":1.556},"min":{"1min":0.38300000000000001,"5min":0.33600000000000002,"15min":0.26900000000000002},"max":{"1min":5.4249999999999998,"5min":9.032,"15min":45.137999999999998},"last":0.442},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.8180000000000001,"15min":1.8680000000000001},"min":{"1min":0.27400000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":5.1959999999999997,"5min":15.331,"15min":44.950000000000003},"last":1.331}]},{"osd":5,"last update":"Fri Feb 9 17:57:32 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":1.4910000000000001,"15min":1.736},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.26000000000000001},"max":{"1min":2.319,"5min":15.106999999999999,"15min":45.171999999999997},"last":1.3620000000000001},{"interface":"front","average":{"1min":0.95199999999999996,"5min":1.772,"15min":2.012},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":5.2370000000000001,"5min":15.355,"15min":45.127000000000002},"last":0.48699999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:57:45 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":1.5600000000000001,"15min":1.909},"min":{"1min":0.34499999999999997,"5min":0.313,"15min":0.24299999999999999},"max":{"1min":3.0419999999999998,"5min":15.154999999999999,"15min":33.338000000000001},"last":1.387},{"interface":"front","average":{"1min":0.98999999999999999,"5min":1.619,"15min":1.9770000000000001},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":3.0920000000000001,"5min":15.253,"15min":33.286999999999999},"last":1.411}]},{"osd":7,"last update":"Fri Feb 9 17:57:45 2024","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.6359999999999999,"15min":2.0579999999999998},"min":{"1min":0.40300000000000002,"5min":0.316,"15min":0.316},"max":{"1min":3.0680000000000001,"5min":15.287000000000001,"15min":45.740000000000002},"last":2.9830000000000001},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.694,"15min":2.1389999999999998},"min":{"1min":0.46899999999999997,"5min":0.45400000000000001,"15min":0.39600000000000002},"max":{"1min":3.117,"5min":15.311,"15min":45.792000000000002},"last":3.0150000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:56:53 2024","interfaces":[{"interface":"back","average":{"1min":2.0419999999999998,"5min":1.8160000000000001,"15min":1.5349999999999999},"min":{"1min":0.51100000000000001,"5min":0.42699999999999999,"15min":0.35599999999999998},"max":{"1min":5.4859999999999998,"5min":9.1999999999999993,"15min":9.2520000000000007},"last":0.56299999999999994},{"interface":"front","average":{"1min":1.9590000000000001,"5min":2.1339999999999999,"15min":1.7050000000000001},"min":{"1min":0.50600000000000001,"5min":0.45500000000000002,"15min":0.35599999999999998},"max":{"1min":5.5170000000000003,"5min":12.464,"15min":12.69},"last":4.3410000000000002}]},{"osd":9,"last update":"Fri Feb 9 17:56:53 2024","interfaces":[{"interface":"back","average":{"1min":2.7589999999999999,"5min":2.3500000000000001,"15min":1.7190000000000001},"min":{"1min":0.57499999999999996,"5min":0.42799999999999999,"15min":0.33900000000000002},"max":{"1min":5.4610000000000003,"5min":20.789000000000001,"15min":20.789000000000001},"last":0.59499999999999997},{"interface":"front","average":{"1min":2.2440000000000002,"5min":1.9890000000000001,"15min":1.7729999999999999},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.34899999999999998},"max":{"1min":5.351,"5min":9.5239999999999991,"15min":9.5239999999999991},"last":0.55300000000000005}]},{"osd":10,"last update":"Fri Feb 9 17:57:32 2024","interfaces":[{"interface":"back","average":{"1min":1.694,"5min":1.9770000000000001,"15min":1.994},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40699999999999997},"max":{"1min":7.1740000000000004,"5min":15.211,"15min":33.213999999999999},"last":1.581},{"interface":"front","average":{"1min":1.8759999999999999,"5min":2.2280000000000002,"15min":1.944},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":7.202,"5min":15.271000000000001,"15min":15.271000000000001},"last":0.65400000000000003}]},{"osd":11,"last update":"Fri Feb 9 17:57:45 2024","interfaces":[{"interface":"back","average":{"1min":3.0910000000000002,"5min":2.5369999999999999,"15min":2.2629999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":14.935,"5min":15.301,"15min":19.648},"last":0.71199999999999997},{"interface":"front","average":{"1min":3.613,"5min":2.9470000000000001,"15min":2.343},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":15.209,"5min":15.209,"15min":33.161000000000001},"last":1.663}]}]},{"osd":0,"up_from":8,"seq":34359738851,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1733080,"kb_used_data":893904,"kb_used_omap":1,"kb_used_meta":839166,"kb_avail":92012072,"statfs":{"total":95995035648,"available":94220361728,"internally_reserved":0,"allocated":915357696,"data_stored":915096327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":859306437},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 9 17:57:40 2024","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.696,"15min":1.4870000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.20599999999999999},"max":{"1min":5.2759999999999998,"5min":18.698,"15min":18.698},"last":0.27300000000000002},{"interface":"front","average":{"1min":1.28,"5min":1.7769999999999999,"15min":1.5700000000000001},"min":{"1min":0.38300000000000001,"5min":0.32000000000000001,"15min":0.28299999999999997},"max":{"1min":5.3099999999999996,"5min":18.742999999999999,"15min":18.742999999999999},"last":0.35999999999999999}]},{"osd":2,"last update":"Fri Feb 9 17:57:02 2024","interfaces":[{"interface":"back","average":{"1min":1.742,"5min":2.0470000000000002,"15min":1.907},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.245},"max":{"1min":13.457000000000001,"5min":20.013999999999999,"15min":20.013999999999999},"last":0.42399999999999999},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.9830000000000001,"15min":1.6759999999999999},"min":{"1min":0.246,"5min":0.246,"15min":0.23699999999999999},"max":{"1min":16.085000000000001,"5min":20.103999999999999,"15min":20.103999999999999},"last":0.30299999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:56:22 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":2.327,"15min":1.7929999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.23100000000000001},"max":{"1min":3.198,"5min":20.469999999999999,"15min":36.052},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.88600000000000001,"5min":2.4249999999999998,"15min":2.1030000000000002},"min":{"1min":0.26800000000000002,"5min":0.23899999999999999,"15min":0.217},"max":{"1min":6.3380000000000001,"5min":21.920000000000002,"15min":38.383000000000003},"last":3.117}]},{"osd":4,"last update":"Fri Feb 9 17:57:08 2024","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":1.5389999999999999,"15min":1.474},"min":{"1min":0.309,"5min":0.28100000000000003,"15min":0.27000000000000002},"max":{"1min":5.1509999999999998,"5min":12.757,"15min":12.757},"last":0.39700000000000002},{"interface":"front","average":{"1min":2.1160000000000001,"5min":1.8839999999999999,"15min":1.8400000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.249},"max":{"1min":11.039,"5min":17.091999999999999,"15min":17.091999999999999},"last":0.318}]},{"osd":5,"last update":"Fri Feb 9 17:57:22 2024","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":1.296,"15min":1.766},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.30299999999999999},"max":{"1min":5.1920000000000002,"5min":9.7929999999999993,"15min":48.924999999999997},"last":0.45700000000000002},{"interface":"front","average":{"1min":1.0209999999999999,"5min":1.345,"15min":1.8100000000000001},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.29599999999999999},"max":{"1min":5.2759999999999998,"5min":9.8230000000000004,"15min":48.969999999999999},"last":0.48799999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:57:52 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.617,"15min":1.6970000000000001},"min":{"1min":0.35399999999999998,"5min":0.307,"15min":0.28699999999999998},"max":{"1min":6.2240000000000002,"5min":15.667,"15min":15.667},"last":0.498},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.7050000000000001,"15min":1.8049999999999999},"min":{"1min":0.36399999999999999,"5min":0.34799999999999998,"15min":0.33300000000000002},"max":{"1min":6.2800000000000002,"5min":15.702999999999999,"15min":15.702999999999999},"last":0.56100000000000005}]},{"osd":7,"last update":"Fri Feb 9 17:57:18 2024","interfaces":[{"interface":"back","average":{"1min":1.544,"5min":1.8140000000000001,"15min":1.8300000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.29199999999999998},"max":{"1min":7.6230000000000002,"5min":11.436,"15min":15.271000000000001},"last":0.377},{"interface":"front","average":{"1min":1.5900000000000001,"5min":1.669,"15min":1.885},"min":{"1min":0.35899999999999999,"5min":0.32600000000000001,"15min":0.28299999999999997},"max":{"1min":7.585,"5min":17.152999999999999,"15min":17.152999999999999},"last":0.441}]},{"osd":8,"last update":"Fri Feb 9 17:57:22 2024","interfaces":[{"interface":"back","average":{"1min":2.9319999999999999,"5min":2.5129999999999999,"15min":1.9219999999999999},"min":{"1min":0.39600000000000002,"5min":0.34699999999999998,"15min":0.313},"max":{"1min":11.227,"5min":17.189,"15min":17.189},"last":1.1100000000000001},{"interface":"front","average":{"1min":3.9399999999999999,"5min":2.278,"15min":1.845},"min":{"1min":0.46600000000000003,"5min":0.36499999999999999,"15min":0.35099999999999998},"max":{"1min":18.600000000000001,"5min":18.600000000000001,"15min":18.600000000000001},"last":0.53100000000000003}]},{"osd":9,"last update":"Fri Feb 9 17:57:02 2024","interfaces":[{"interface":"back","average":{"1min":3.5819999999999999,"5min":2.5800000000000001,"15min":2.2509999999999999},"min":{"1min":0.44800000000000001,"5min":0.39000000000000001,"15min":0.33800000000000002},"max":{"1min":11.148999999999999,"5min":17.216000000000001,"15min":24.757000000000001},"last":0.40999999999999998},{"interface":"front","average":{"1min":2.3180000000000001,"5min":2.2000000000000002,"15min":2.089},"min":{"1min":0.51600000000000001,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":6.2830000000000004,"5min":19.986999999999998,"15min":24.681999999999999},"last":0.51800000000000002}]},{"osd":10,"last update":"Fri Feb 9 17:57:22 2024","interfaces":[{"interface":"back","average":{"1min":1.831,"5min":2.8340000000000001,"15min":2.2890000000000001},"min":{"1min":0.45000000000000001,"5min":0.42499999999999999,"15min":0.35499999999999998},"max":{"1min":5.0419999999999998,"5min":57.979999999999997,"15min":57.979999999999997},"last":0.58199999999999996},{"interface":"front","average":{"1min":3.3570000000000002,"5min":3.2559999999999998,"15min":2.298},"min":{"1min":0.498,"5min":0.41299999999999998,"15min":0.34200000000000003},"max":{"1min":8.2729999999999997,"5min":57.606999999999999,"15min":57.606999999999999},"last":1.194}]},{"osd":11,"last update":"Fri Feb 9 17:57:52 2024","interfaces":[{"interface":"back","average":{"1min":1.99,"5min":2.6640000000000001,"15min":2.1360000000000001},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.41599999999999998},"max":{"1min":5.5439999999999996,"5min":58.351999999999997,"15min":58.351999999999997},"last":3.355},{"interface":"front","average":{"1min":2.0979999999999999,"5min":2.7240000000000002,"15min":2.1920000000000002},"min":{"1min":0.51800000000000002,"5min":0.441,"15min":0.441},"max":{"1min":6.0099999999999998,"5min":57.314,"15min":57.314},"last":3.3900000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411800,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2106468,"kb_used_data":1479708,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":91638684,"statfs":{"total":95995035648,"available":93838012416,"internally_reserved":0,"allocated":1515220992,"data_stored":1514951999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[0,1,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:57:44 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":3.629,"5min":3.629,"15min":3.629},"last":0.313},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":3.6509999999999998,"5min":3.6509999999999998,"15min":3.6509999999999998},"last":0.76700000000000002}]},{"osd":1,"last update":"Fri Feb 9 17:57:16 2024","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.9870000000000001,"15min":1.9950000000000001},"min":{"1min":0.28100000000000003,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":6.867,"5min":15.378,"15min":52.646000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":1.552,"5min":1.9259999999999999,"15min":1.9930000000000001},"min":{"1min":0.376,"5min":0.248,"15min":0.248},"max":{"1min":6.8899999999999997,"5min":15.092000000000001,"15min":52.683999999999997},"last":0.498}]},{"osd":3,"last update":"Fri Feb 9 17:55:40 2024","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":2.8069999999999999,"15min":2.1920000000000002},"min":{"1min":0.29799999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":8.7200000000000006,"5min":46.927999999999997,"15min":46.927999999999997},"last":4.415},{"interface":"front","average":{"1min":0.92000000000000004,"5min":2.6110000000000002,"15min":2.1970000000000001},"min":{"1min":0.28000000000000003,"5min":0.192,"15min":0.192},"max":{"1min":5.8140000000000001,"5min":45.798999999999999,"15min":45.798999999999999},"last":4.0940000000000003}]},{"osd":4,"last update":"Fri Feb 9 17:57:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":1.2190000000000001,"15min":1.5009999999999999},"min":{"1min":0.35699999999999998,"5min":0.245,"15min":0.245},"max":{"1min":3.516,"5min":10.365,"15min":12.398},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":1.2849999999999999,"15min":1.5089999999999999},"min":{"1min":0.38200000000000001,"5min":0.26600000000000001,"15min":0.25700000000000001},"max":{"1min":2.4529999999999998,"5min":10.529,"15min":10.888999999999999},"last":0.437}]},{"osd":5,"last update":"Fri Feb 9 17:57:38 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":1.5009999999999999,"15min":2.0209999999999999},"min":{"1min":0.29599999999999999,"5min":0.245,"15min":0.245},"max":{"1min":2.5870000000000002,"5min":12.305999999999999,"15min":42.664000000000001},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.82199999999999995,"5min":1.3149999999999999,"15min":2.105},"min":{"1min":0.30399999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":4.8150000000000004,"5min":10.603,"15min":43.311999999999998},"last":0.94699999999999995}]},{"osd":6,"last update":"Fri Feb 9 17:57:49 2024","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":1.502,"15min":1.8899999999999999},"min":{"1min":0.36299999999999999,"5min":0.29299999999999998,"15min":0.28399999999999997},"max":{"1min":4.7949999999999999,"5min":10.573,"15min":18.518999999999998},"last":1.0620000000000001},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.5600000000000001,"15min":1.8879999999999999},"min":{"1min":0.317,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":4.8570000000000002,"5min":10.731,"15min":18.481999999999999},"last":1.083}]},{"osd":7,"last update":"Fri Feb 9 17:57:16 2024","interfaces":[{"interface":"back","average":{"1min":1.552,"5min":1.4630000000000001,"15min":1.887},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.32400000000000001},"max":{"1min":7.1109999999999998,"5min":10.403,"15min":16.666},"last":0.52900000000000003},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.3180000000000001,"15min":1.992},"min":{"1min":0.32600000000000001,"5min":0.313,"15min":0.313},"max":{"1min":4.157,"5min":10.316000000000001,"15min":16.638999999999999},"last":1.0289999999999999}]},{"osd":8,"last update":"Fri Feb 9 17:57:49 2024","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.956,"15min":1.79},"min":{"1min":0.40699999999999997,"5min":0.35599999999999998,"15min":0.316},"max":{"1min":10.095000000000001,"5min":10.755000000000001,"15min":12.500999999999999},"last":1.55},{"interface":"front","average":{"1min":1.6879999999999999,"5min":1.7929999999999999,"15min":1.984},"min":{"1min":0.40400000000000003,"5min":0.35699999999999998,"15min":0.34399999999999997},"max":{"1min":4.8339999999999996,"5min":10.629,"15min":54.759999999999998},"last":3.101}]},{"osd":9,"last update":"Fri Feb 9 17:57:53 2024","interfaces":[{"interface":"back","average":{"1min":1.702,"5min":2.1739999999999999,"15min":2.2000000000000002},"min":{"1min":0.372,"5min":0.318,"15min":0.318},"max":{"1min":9.2240000000000002,"5min":12.276,"15min":53.935000000000002},"last":0.41599999999999998},{"interface":"front","average":{"1min":1.9590000000000001,"5min":2.1499999999999999,"15min":2.1349999999999998},"min":{"1min":0.48199999999999998,"5min":0.36199999999999999,"15min":0.34899999999999998},"max":{"1min":9.1989999999999998,"5min":10.657,"15min":54.670999999999999},"last":0.98899999999999999}]},{"osd":10,"last update":"Fri Feb 9 17:57:30 2024","interfaces":[{"interface":"back","average":{"1min":2.254,"5min":2.0710000000000002,"15min":1.9710000000000001},"min":{"1min":0.56599999999999995,"5min":0.40799999999999997,"15min":0.33500000000000002},"max":{"1min":7.1529999999999996,"5min":12.313000000000001,"15min":54.039999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":2.2789999999999999,"5min":2.1030000000000002,"15min":2.0099999999999998},"min":{"1min":0.58399999999999996,"5min":0.44500000000000001,"15min":0.42599999999999999},"max":{"1min":7.2300000000000004,"5min":12.343999999999999,"15min":54.118000000000002},"last":0.63400000000000001}]},{"osd":11,"last update":"Fri Feb 9 17:57:49 2024","interfaces":[{"interface":"back","average":{"1min":1.958,"5min":1.9830000000000001,"15min":2.1899999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.40600000000000003},"max":{"1min":4.891,"5min":10.702,"15min":54.799999999999997},"last":3.2229999999999999},{"interface":"front","average":{"1min":1.403,"5min":2.1120000000000001,"15min":2.1859999999999999},"min":{"1min":0.40600000000000003,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":5.4109999999999996,"5min":12.243,"15min":54.103000000000002},"last":0.54000000000000004}]}]},{"osd":3,"up_from":27,"seq":115964117446,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2157336,"kb_used_data":1318156,"kb_used_omap":1,"kb_used_meta":839166,"kb_avail":91587816,"statfs":{"total":95995035648,"available":93785923584,"internally_reserved":0,"allocated":1349791744,"data_stored":1349509795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":859306439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:56:33 2024","interfaces":[{"interface":"back","average":{"1min":2.012,"5min":2.4329999999999998,"15min":1.9950000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.221},"max":{"1min":10.628,"5min":17.006,"15min":17.006},"last":0.27200000000000002},{"interface":"front","average":{"1min":1.724,"5min":1.821,"15min":2.028},"min":{"1min":0.252,"5min":0.24099999999999999,"15min":0.23400000000000001},"max":{"1min":6.3860000000000001,"5min":12.288,"15min":16.637},"last":0.40200000000000002}]},{"osd":1,"last update":"Fri Feb 9 17:56:03 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":1.8460000000000001,"15min":2.512},"min":{"1min":0.23499999999999999,"5min":0.221,"15min":0.221},"max":{"1min":2.0779999999999998,"5min":12.609,"15min":53.152999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.68000000000000005,"5min":1.4379999999999999,"15min":1.859},"min":{"1min":0.29799999999999999,"5min":0.24299999999999999,"15min":0.23200000000000001},"max":{"1min":2.1419999999999999,"5min":12.237,"15min":53.118000000000002},"last":0.54900000000000004}]},{"osd":2,"last update":"Fri Feb 9 17:56:33 2024","interfaces":[{"interface":"back","average":{"1min":1.609,"5min":2.169,"15min":1.913},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":10.353999999999999,"5min":17.477,"15min":17.477},"last":0.35499999999999998},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.9039999999999999,"15min":1.839},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":3.0750000000000002,"5min":21.103000000000002,"15min":21.103000000000002},"last":0.36799999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:56:33 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":1.413,"15min":1.73},"min":{"1min":0.28999999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":2.907,"5min":12.718,"15min":12.718},"last":0.71999999999999997},{"interface":"front","average":{"1min":1.532,"5min":1.2969999999999999,"15min":1.6259999999999999},"min":{"1min":0.316,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":6.157,"5min":6.157,"15min":19.713999999999999},"last":0.58399999999999996}]},{"osd":5,"last update":"Fri Feb 9 17:56:17 2024","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.7889999999999999,"15min":1.98},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.28000000000000003},"max":{"1min":8.1920000000000002,"5min":13.321,"15min":14.106},"last":0.317},{"interface":"front","average":{"1min":1.4399999999999999,"5min":1.8440000000000001,"15min":1.968},"min":{"1min":0.44,"5min":0.44,"15min":0.316},"max":{"1min":8.3789999999999996,"5min":13.347,"15min":14.138999999999999},"last":0.48899999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:56:03 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":1.829,"15min":1.8580000000000001},"min":{"1min":0.41499999999999998,"5min":0.35799999999999998,"15min":0.28899999999999998},"max":{"1min":1.385,"5min":13.207000000000001,"15min":15.061999999999999},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.68300000000000005,"5min":1.988,"15min":1.9339999999999999},"min":{"1min":0.44900000000000001,"5min":0.375,"15min":0.36399999999999999},"max":{"1min":1.448,"5min":13.298,"15min":13.298},"last":0.441}]},{"osd":7,"last update":"Fri Feb 9 17:55:50 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":1.6359999999999999,"15min":1.869},"min":{"1min":0.374,"5min":0.313,"15min":0.313},"max":{"1min":1.1140000000000001,"5min":18.587,"15min":19.788},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":1.8879999999999999,"15min":1.921},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.34899999999999998},"max":{"1min":1.4139999999999999,"5min":13.239000000000001,"15min":13.239000000000001},"last":0.59799999999999998}]},{"osd":8,"last update":"Fri Feb 9 17:45:16 2024","interfaces":[{"interface":"back","average":{"1min":1.714,"5min":1.4450000000000001,"15min":1.498},"min":{"1min":0.377,"5min":0.377,"15min":0.26800000000000002},"max":{"1min":8.6020000000000003,"5min":11.821,"15min":11.821},"last":0.67700000000000005},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.1719999999999999,"15min":1.351},"min":{"1min":0.46500000000000002,"5min":0.36199999999999999,"15min":0.34499999999999997},"max":{"1min":6.8390000000000004,"5min":6.8390000000000004,"15min":14.129},"last":0.78200000000000003}]},{"osd":9,"last update":"Fri Feb 9 17:56:03 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.9379999999999999,"15min":1.804},"min":{"1min":0.497,"5min":0.312,"15min":0.312},"max":{"1min":3.0270000000000001,"5min":11.039,"15min":11.851000000000001},"last":0.66100000000000003},{"interface":"front","average":{"1min":1.026,"5min":1.8500000000000001,"15min":1.7969999999999999},"min":{"1min":0.376,"5min":0.318,"15min":0.318},"max":{"1min":3.2400000000000002,"5min":10.462999999999999,"15min":11.907},"last":0.69999999999999996}]},{"osd":10,"last update":"Fri Feb 9 17:55:50 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":2.3580000000000001,"15min":1.75},"min":{"1min":0.52400000000000002,"5min":0.371,"15min":0.371},"max":{"1min":1.5449999999999999,"5min":18.632999999999999,"15min":18.632999999999999},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.82399999999999995,"5min":2.2549999999999999,"15min":1.798},"min":{"1min":0.60699999999999998,"5min":0.46800000000000003,"15min":0.34100000000000003},"max":{"1min":1.5760000000000001,"5min":13.272,"15min":19.859000000000002},"last":0.56399999999999995}]},{"osd":11,"last update":"Fri Feb 9 17:56:33 2024","interfaces":[{"interface":"back","average":{"1min":2.1379999999999999,"5min":2.7080000000000002,"15min":1.9850000000000001},"min":{"1min":0.45800000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":8.875,"5min":18.66,"15min":19.824000000000002},"last":0.54000000000000004},{"interface":"front","average":{"1min":2.1619999999999999,"5min":2.738,"15min":1.8839999999999999},"min":{"1min":0.47399999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":8.9179999999999993,"5min":18.687999999999999,"15min":18.687999999999999},"last":0.67600000000000005}]}]},{"osd":4,"up_from":32,"seq":137438953936,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1976344,"kb_used_data":1164944,"kb_used_omap":1,"kb_used_meta":811390,"kb_avail":91768808,"statfs":{"total":95995035648,"available":93971259392,"internally_reserved":0,"allocated":1192902656,"data_stored":1192641287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":830863813},"hb_peers":[0,1,2,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":1.1499999999999999,"15min":1.306},"min":{"1min":0.26800000000000002,"5min":0.246,"15min":0.22800000000000001},"max":{"1min":6.3949999999999996,"5min":10.005000000000001,"15min":10.301},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.78100000000000003,"5min":1.2190000000000001,"15min":1.3740000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.28599999999999998},"max":{"1min":6.4210000000000003,"5min":10.036,"15min":10.407999999999999},"last":0.51200000000000001}]},{"osd":1,"last update":"Fri Feb 9 17:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":1.7829999999999999,"15min":1.786},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.23699999999999999},"max":{"1min":2.2309999999999999,"5min":50.255000000000003,"15min":50.255000000000003},"last":0.38900000000000001},{"interface":"front","average":{"1min":1.0600000000000001,"5min":2.0699999999999998,"15min":2.1389999999999998},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.23100000000000001},"max":{"1min":2.569,"5min":49.991999999999997,"15min":49.991999999999997},"last":0.46999999999999997}]},{"osd":2,"last update":"Fri Feb 9 17:57:41 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":3.6739999999999999,"5min":3.6739999999999999,"15min":3.6739999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":3.581,"5min":3.581,"15min":3.581},"last":0.41599999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:55:52 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":1.248,"15min":1.569},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.71799999999999997,"5min":10.484999999999999,"15min":14.448},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.4419999999999999,"15min":1.655},"min":{"1min":0.315,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.93000000000000005,"5min":10.455,"15min":10.455},"last":0.438}]},{"osd":5,"last update":"Fri Feb 9 17:57:11 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.60199999999999998,"15min":1.7470000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.22600000000000001},"max":{"1min":0.48199999999999998,"5min":4.8019999999999996,"15min":18.887},"last":0.308},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.67100000000000004,"15min":1.827},"min":{"1min":0.34200000000000003,"5min":0.31900000000000001,"15min":0.31},"max":{"1min":0.60399999999999998,"5min":4.8639999999999999,"15min":18.917000000000002},"last":0.37}]},{"osd":6,"last update":"Fri Feb 9 17:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.64500000000000002,"15min":1.3919999999999999},"min":{"1min":0.28399999999999997,"5min":0.27800000000000002,"15min":0.24299999999999999},"max":{"1min":2.2759999999999998,"5min":6.7560000000000002,"15min":23.343},"last":0.432},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.66300000000000003,"15min":1.456},"min":{"1min":0.33900000000000002,"5min":0.30599999999999999,"15min":0.28799999999999998},"max":{"1min":1.3939999999999999,"5min":5.0060000000000002,"15min":21.861000000000001},"last":0.55300000000000005}]},{"osd":7,"last update":"Fri Feb 9 17:57:30 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.69699999999999995,"15min":1.448},"min":{"1min":0.39200000000000002,"5min":0.32900000000000001,"15min":0.29899999999999999},"max":{"1min":1.694,"5min":5.0640000000000001,"15min":11.875},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.69099999999999995,"15min":1.6140000000000001},"min":{"1min":0.438,"5min":0.34399999999999997,"15min":0.28599999999999998},"max":{"1min":0.69199999999999995,"5min":5.0460000000000003,"15min":12.696},"last":0.53200000000000003}]},{"osd":8,"last update":"Fri Feb 9 17:57:11 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":1.3999999999999999,"15min":1.8089999999999999},"min":{"1min":0.42899999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":3.605,"5min":9.7059999999999995,"15min":17.335999999999999},"last":0.40400000000000003},{"interface":"front","average":{"1min":1.004,"5min":1.246,"15min":1.2809999999999999},"min":{"1min":0.41199999999999998,"5min":0.373,"15min":0.34699999999999998},"max":{"1min":3.5550000000000002,"5min":5.6470000000000002,"15min":10.433},"last":4.2469999999999999}]},{"osd":9,"last update":"Fri Feb 9 17:57:17 2024","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.458,"15min":2.3370000000000002},"min":{"1min":0.39700000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":3.5870000000000002,"5min":7.665,"15min":7.665},"last":4.0410000000000004},{"interface":"front","average":{"1min":1.53,"5min":1.3600000000000001,"15min":1.6060000000000001},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":7.0999999999999996,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":5.5119999999999996}]},{"osd":10,"last update":"Fri Feb 9 17:57:30 2024","interfaces":[{"interface":"back","average":{"1min":1.7350000000000001,"5min":1.405,"15min":1.5609999999999999},"min":{"1min":0.55700000000000005,"5min":0.41299999999999998,"15min":0.40200000000000002},"max":{"1min":4.4379999999999997,"5min":6.9509999999999996,"15min":17.367000000000001},"last":4.1139999999999999},{"interface":"front","average":{"1min":1.393,"5min":1.3080000000000001,"15min":1.6279999999999999},"min":{"1min":0.42699999999999999,"5min":0.41299999999999998,"15min":0.38},"max":{"1min":4.7439999999999998,"5min":7.9189999999999996,"15min":12.888},"last":5.4669999999999996}]},{"osd":11,"last update":"Fri Feb 9 17:57:03 2024","interfaces":[{"interface":"back","average":{"1min":1.621,"5min":1.484,"15min":1.5740000000000001},"min":{"1min":0.5,"5min":0.46500000000000002,"15min":0.39900000000000002},"max":{"1min":7.6689999999999996,"5min":7.6689999999999996,"15min":10.521000000000001},"last":4.0640000000000001},{"interface":"front","average":{"1min":1.472,"5min":1.504,"15min":1.6240000000000001},"min":{"1min":0.498,"5min":0.48499999999999999,"15min":0.40999999999999998},"max":{"1min":4.8550000000000004,"5min":8.4420000000000002,"15min":13.154999999999999},"last":5.4279999999999999}]}]},{"osd":5,"up_from":40,"seq":171798692299,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2201184,"kb_used_data":1461976,"kb_used_omap":1,"kb_used_meta":739198,"kb_avail":91543968,"statfs":{"total":95995035648,"available":93741023232,"internally_reserved":0,"allocated":1497063424,"data_stored":1496798503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":756939205},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:57:42 2024","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":5.399,"5min":5.399,"15min":5.399},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.996,"5min":0.996,"15min":0.996},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":5.4329999999999998,"5min":5.4329999999999998,"15min":5.4329999999999998},"last":0.67400000000000004}]},{"osd":1,"last update":"Fri Feb 9 17:57:17 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.02,"15min":1.647},"min":{"1min":0.27200000000000002,"5min":0.26600000000000001,"15min":0.254},"max":{"1min":3.956,"5min":9.532,"15min":19.079999999999998},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.93600000000000005,"15min":1.5529999999999999},"min":{"1min":0.26900000000000002,"5min":0.23999999999999999,"15min":0.23499999999999999},"max":{"1min":5.4470000000000001,"5min":6.2190000000000003,"15min":19.065999999999999},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Feb 9 17:57:53 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":1.0069999999999999,"15min":1.6419999999999999},"min":{"1min":0.28199999999999997,"5min":0.254,"15min":0.249},"max":{"1min":2.9900000000000002,"5min":7.5670000000000002,"15min":13.929},"last":1.796},{"interface":"front","average":{"1min":0.72199999999999998,"5min":1.0649999999999999,"15min":1.7869999999999999},"min":{"1min":0.30199999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.7689999999999999,"5min":6.3230000000000004,"15min":15.945},"last":1.6990000000000001}]},{"osd":3,"last update":"Fri Feb 9 17:56:16 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":1.8080000000000001,"15min":2.04},"min":{"1min":0.35799999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.3240000000000001,"5min":20.690000000000001,"15min":20.690000000000001},"last":0.505},{"interface":"front","average":{"1min":0.65000000000000002,"5min":1.468,"15min":1.925},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.583,"5min":17.209,"15min":17.209},"last":2.2719999999999998}]},{"osd":4,"last update":"Fri Feb 9 17:57:17 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.65900000000000003,"15min":1.4770000000000001},"min":{"1min":0.26400000000000001,"5min":0.26100000000000001,"15min":0.249},"max":{"1min":2.0750000000000002,"5min":4.1360000000000001,"15min":28.481999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.629,"5min":0.73799999999999999,"15min":1.569},"min":{"1min":0.374,"5min":0.22500000000000001,"15min":0.218},"max":{"1min":2.069,"5min":4.9189999999999996,"15min":15.843},"last":0.70599999999999996}]},{"osd":6,"last update":"Fri Feb 9 17:57:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.65800000000000003,"15min":1.4370000000000001},"min":{"1min":0.34100000000000003,"5min":0.29999999999999999,"15min":0.27900000000000003},"max":{"1min":1.52,"5min":5.2569999999999997,"15min":16.657},"last":0.623},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.72899999999999998,"15min":1.5329999999999999},"min":{"1min":0.34200000000000003,"5min":0.33500000000000002,"15min":0.26200000000000001},"max":{"1min":1.9419999999999999,"5min":5.2290000000000001,"15min":20.192},"last":0.42699999999999999}]},{"osd":7,"last update":"Fri Feb 9 17:57:24 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.876,"15min":1.661},"min":{"1min":0.318,"5min":0.318,"15min":0.27900000000000003},"max":{"1min":1.601,"5min":6.6319999999999997,"15min":18.134},"last":1.635},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.76600000000000001,"15min":1.8280000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.31},"max":{"1min":2.2829999999999999,"5min":4.9000000000000004,"15min":19.300000000000001},"last":0.66100000000000003}]},{"osd":8,"last update":"Fri Feb 9 17:57:42 2024","interfaces":[{"interface":"back","average":{"1min":2.5600000000000001,"5min":1.4019999999999999,"15min":1.7210000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":12.227,"5min":12.227,"15min":12.227},"last":4.7969999999999997},{"interface":"front","average":{"1min":0.94599999999999995,"5min":1.131,"15min":1.6759999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":3.8570000000000002,"5min":8.1829999999999998,"15min":8.1829999999999998},"last":0.47299999999999998}]},{"osd":9,"last update":"Fri Feb 9 17:57:53 2024","interfaces":[{"interface":"back","average":{"1min":2.7280000000000002,"5min":1.6279999999999999,"15min":1.7749999999999999},"min":{"1min":0.42899999999999999,"5min":0.41499999999999998,"15min":0.35299999999999998},"max":{"1min":11.167999999999999,"5min":11.167999999999999,"15min":16.411000000000001},"last":11.167999999999999},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.0600000000000001,"15min":1.6950000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.35999999999999999},"max":{"1min":3.919,"5min":4.3129999999999997,"15min":16.417999999999999},"last":1.651}]},{"osd":10,"last update":"Fri Feb 9 17:57:17 2024","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":1.355,"15min":1.9850000000000001},"min":{"1min":0.46700000000000003,"5min":0.39300000000000002,"15min":0.317},"max":{"1min":3.722,"5min":16.292000000000002,"15min":80.042000000000002},"last":0.73299999999999998},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.4530000000000001,"15min":1.944},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.38200000000000001},"max":{"1min":4.6219999999999999,"5min":5.7830000000000004,"15min":79.807000000000002},"last":0.745}]},{"osd":11,"last update":"Fri Feb 9 17:57:38 2024","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.393,"15min":1.9390000000000001},"min":{"1min":0.434,"5min":0.434,"15min":0.38400000000000001},"max":{"1min":3.532,"5min":7.5270000000000001,"15min":16.390000000000001},"last":5.2770000000000001},{"interface":"front","average":{"1min":1.335,"5min":1.2350000000000001,"15min":1.782},"min":{"1min":0.53500000000000003,"5min":0.434,"15min":0.42199999999999999},"max":{"1min":4.6669999999999998,"5min":5.6529999999999996,"15min":20.981000000000002},"last":5.2110000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":910295040,"data_stored":910295040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1814102016,"data_stored":1814102016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1498812416,"data_stored":1498808344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1348993024,"data_stored":1348993024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1182269440,"data_stored":1182269440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1495465984,"data_stored":1495465984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":599134208,"data_stored":599130136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":918683648,"data_stored":918683648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1741357056,"data_stored":1741357056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1816395776,"data_stored":1816395776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1498025984,"data_stored":1498021912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-09T17:57:59.173 INFO:tasks.thrashosds.thrasher:Setting 1.5 to [1, 0, 6, 8, 7, 10] 2024-02-09T17:57:59.173 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '1.5', '1', '0', '6', '8', '7', '10'] 2024-02-09T17:57:59.173 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd pg-upmap-items 1.5 1 0 6 8 7 10 2024-02-09T17:57:59.400 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:59 smithi099 ceph-mon[39420]: from='client.16872 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:57:59.400 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:59 smithi099 ceph-mon[39420]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:57:59.400 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:59 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1852713742' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:57:59.400 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:57:59 smithi099 ceph-mon[39420]: osdmap e833: 12 total, 11 up, 12 in 2024-02-09T17:57:59.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:59 smithi179 ceph-mon[43975]: from='client.16872 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:57:59.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:59 smithi179 ceph-mon[43975]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:57:59.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:59 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1852713742' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:57:59.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:57:59 smithi179 ceph-mon[43975]: osdmap e833: 12 total, 11 up, 12 in 2024-02-09T17:57:59.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:59 smithi162 ceph-mon[44304]: from='client.16872 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:57:59.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:59 smithi162 ceph-mon[44304]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:57:59.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:59 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1852713742' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:57:59.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:57:59 smithi162 ceph-mon[44304]: osdmap e833: 12 total, 11 up, 12 in 2024-02-09T17:58:00.008 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:58:00.027 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:58:00.570 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:00 smithi099 ceph-mon[39420]: pgmap v2070: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 4.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 942 op/s; 13460/214935 objects degraded (6.262%) 2024-02-09T17:58:00.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:00 smithi179 ceph-mon[43975]: pgmap v2070: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 4.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 942 op/s; 13460/214935 objects degraded (6.262%) 2024-02-09T17:58:00.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:00 smithi162 ceph-mon[44304]: pgmap v2070: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 4.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 942 op/s; 13460/214935 objects degraded (6.262%) 2024-02-09T17:58:01.675 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:01 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:58:01.675 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:01 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:58:01.677 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:01 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:58:01.677 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:01 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:58:01.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:01 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:58:01.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:01 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:58:01.813 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:58:01.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:01.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:58:01.918 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:58:01.975 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:01.989 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset noscrub 2024-02-09T17:58:02.019 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:58:02.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:02.022 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:58:02.123 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:58:02.177 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:02.211 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:58:02.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:02.220 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:58:02.321 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:58:02.349 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:02.372 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:58:02.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:02.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:58:02.477 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:58:02.507 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:02.531 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:58:02.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:02.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:58:02.636 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:58:02.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:02 smithi179 ceph-mon[43975]: pgmap v2071: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 4.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 694 op/s; 13460/213354 objects degraded (6.309%) 2024-02-09T17:58:02.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:02 smithi179 ceph-mon[43975]: osdmap e834: 12 total, 11 up, 12 in 2024-02-09T17:58:02.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:02 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 13460/213354 objects degraded (6.309%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:58:02.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:02 smithi162 ceph-mon[44304]: pgmap v2071: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 4.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 694 op/s; 13460/213354 objects degraded (6.309%) 2024-02-09T17:58:02.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:02 smithi162 ceph-mon[44304]: osdmap e834: 12 total, 11 up, 12 in 2024-02-09T17:58:02.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:02 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 13460/213354 objects degraded (6.309%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:58:02.691 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:02.734 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:58:02.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:02.740 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:58:02.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:02 smithi099 ceph-mon[39420]: pgmap v2071: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 4.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 694 op/s; 13460/213354 objects degraded (6.309%) 2024-02-09T17:58:02.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:02 smithi099 ceph-mon[39420]: osdmap e834: 12 total, 11 up, 12 in 2024-02-09T17:58:02.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:02 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 13460/213354 objects degraded (6.309%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:58:02.840 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:58:02.885 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:02.987 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:58:03.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:03.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:58:03.101 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:58:03.154 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:03.180 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:58:03.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:03.188 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:58:03.289 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:58:03.363 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:03.410 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:58:03.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:03.418 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:58:03.519 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:58:03.548 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:03.575 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:58:03.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:03.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:58:03.669 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T17:58:03.669 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:58:03.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:03 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:58:03.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:03 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:58:03.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:03 smithi179 ceph-mon[43975]: osdmap e835: 12 total, 11 up, 12 in 2024-02-09T17:58:03.679 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:58:03.691 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:03 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:58:03.691 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:03 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:58:03.691 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:03 smithi099 ceph-mon[39420]: osdmap e835: 12 total, 11 up, 12 in 2024-02-09T17:58:03.705 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:03 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:58:03.705 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:03 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:58:03.705 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:03 smithi162 ceph-mon[44304]: osdmap e835: 12 total, 11 up, 12 in 2024-02-09T17:58:03.706 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:03.728 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:58:03.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:03.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:58:03.833 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:58:03.862 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:03.893 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:58:03.895 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:03.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:58:03.997 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:58:04.025 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:04.049 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:58:04.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:04.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:58:04.157 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:58:04.214 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:04.240 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:58:04.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:04.243 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:58:04.344 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:58:04.372 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:04.399 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:58:04.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:04.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:58:04.459 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:04 smithi099 ceph-mon[39420]: pgmap v2074: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 4.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 768 op/s; 12763/205890 objects degraded (6.199%) 2024-02-09T17:58:04.502 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:58:04.542 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:04.576 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:58:04.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:04.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:58:04.598 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-09T17:58:04.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:04 smithi179 ceph-mon[43975]: pgmap v2074: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 4.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 768 op/s; 12763/205890 objects degraded (6.199%) 2024-02-09T17:58:04.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:04 smithi162 ceph-mon[44304]: pgmap v2074: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 4.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 768 op/s; 12763/205890 objects degraded (6.199%) 2024-02-09T17:58:04.685 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:58:04.745 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:05.081 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:58:05.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:05.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:58:05.186 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:58:05.242 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:05.271 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:58:05.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:05.278 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:58:05.379 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:58:05.409 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:05.431 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:58:05.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:05.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:58:05.536 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:58:05.564 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:05.586 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:58:05.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:05.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:58:05.690 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:58:05.720 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:05.977 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:58:05.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:05.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:58:06.011 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:58:06.012 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:58:06.078 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:58:06.095 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:58:06.109 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:58:06.131 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:58:06.145 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:06.159 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:58:06.179 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:58:06.184 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:58:06.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:06.189 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:58:06.209 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:58:06.238 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:58:06.272 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:58:06.290 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:58:06.304 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:58:06.324 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:06.330 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:58:06.346 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:58:06.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:06.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:58:06.372 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:58:06.372 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:58:06.372 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:58:06.373 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:58:06.373 INFO:teuthology.orchestra.run.smithi099.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-09T17:58:06.373 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:58:06.373 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:58:06.373 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:58:06.373 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:58:06.373 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:58:06.373 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:58:06.373 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:58:06.373 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:58:06.450 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:58:06.468 INFO:teuthology.orchestra.run.smithi099.stderr:set 1.5 pg_upmap_items mapping to [1->0,6->8,7->10] 2024-02-09T17:58:06.479 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:06.507 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:58:06.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:06.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:58:06.613 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:58:06.648 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:06 smithi099 ceph-mon[39420]: pgmap v2075: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 4.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 723 op/s; 12763/201411 objects degraded (6.337%) 2024-02-09T17:58:06.648 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:06 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2895983954' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:58:06.648 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:06 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3741214452' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.5", "id": [1, 0, 6, 8, 7, 10]}]: dispatch 2024-02-09T17:58:06.654 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:06.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:06 smithi162 ceph-mon[44304]: pgmap v2075: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 4.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 723 op/s; 12763/201411 objects degraded (6.337%) 2024-02-09T17:58:06.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:06 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2895983954' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:58:06.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:06 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3741214452' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.5", "id": [1, 0, 6, 8, 7, 10]}]: dispatch 2024-02-09T17:58:06.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:06 smithi179 ceph-mon[43975]: pgmap v2075: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 4.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 723 op/s; 12763/201411 objects degraded (6.337%) 2024-02-09T17:58:06.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:06 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2895983954' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:58:06.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:06 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3741214452' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.5", "id": [1, 0, 6, 8, 7, 10]}]: dispatch 2024-02-09T17:58:07.636 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:58:07.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:07.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:58:07.743 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:58:07.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:07 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3741214452' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.5", "id": [1, 0, 6, 8, 7, 10]}]': finished 2024-02-09T17:58:07.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:07 smithi099 ceph-mon[39420]: osdmap e836: 12 total, 11 up, 12 in 2024-02-09T17:58:07.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:07 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 12763/201411 objects degraded (6.337%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:58:07.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:07 smithi099 ceph-mon[39420]: osdmap e837: 12 total, 11 up, 12 in 2024-02-09T17:58:07.769 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:07 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3741214452' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.5", "id": [1, 0, 6, 8, 7, 10]}]': finished 2024-02-09T17:58:07.769 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:07 smithi162 ceph-mon[44304]: osdmap e836: 12 total, 11 up, 12 in 2024-02-09T17:58:07.769 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:07 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 12763/201411 objects degraded (6.337%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:58:07.770 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:07 smithi162 ceph-mon[44304]: osdmap e837: 12 total, 11 up, 12 in 2024-02-09T17:58:07.771 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:07.792 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:58:07.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:07.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:58:07.899 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:58:07.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:07 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3741214452' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.5", "id": [1, 0, 6, 8, 7, 10]}]': finished 2024-02-09T17:58:07.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:07 smithi179 ceph-mon[43975]: osdmap e836: 12 total, 11 up, 12 in 2024-02-09T17:58:07.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:07 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 12763/201411 objects degraded (6.337%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:58:07.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:07 smithi179 ceph-mon[43975]: osdmap e837: 12 total, 11 up, 12 in 2024-02-09T17:58:07.937 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:08.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:08 smithi099 ceph-mon[39420]: pgmap v2078: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 4.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 860 op/s; 12763/198363 objects degraded (6.434%) 2024-02-09T17:58:08.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:08 smithi099 ceph-mon[39420]: osdmap e838: 12 total, 11 up, 12 in 2024-02-09T17:58:08.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:08 smithi179 ceph-mon[43975]: pgmap v2078: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 4.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 860 op/s; 12763/198363 objects degraded (6.434%) 2024-02-09T17:58:08.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:08 smithi179 ceph-mon[43975]: osdmap e838: 12 total, 11 up, 12 in 2024-02-09T17:58:08.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:08 smithi162 ceph-mon[44304]: pgmap v2078: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 4.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 860 op/s; 12763/198363 objects degraded (6.434%) 2024-02-09T17:58:08.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:08 smithi162 ceph-mon[44304]: osdmap e838: 12 total, 11 up, 12 in 2024-02-09T17:58:09.939 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-09T17:58:09.940 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-09T17:58:10.178 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:10 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:58:10.178 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:10 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:58:10.178 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:10 smithi099 ceph-mon[39420]: osdmap e839: 12 total, 11 up, 12 in 2024-02-09T17:58:10.178 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:10 smithi099 ceph-mon[39420]: pgmap v2081: 47 pgs: 1 active+undersized+remapped, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 3.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 829 op/s; 12079/191457 objects degraded (6.309%) 2024-02-09T17:58:10.295 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:58:10.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:10.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:58:10.403 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:58:10.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:10 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:58:10.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:10 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:58:10.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:10 smithi179 ceph-mon[43975]: osdmap e839: 12 total, 11 up, 12 in 2024-02-09T17:58:10.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:10 smithi179 ceph-mon[43975]: pgmap v2081: 47 pgs: 1 active+undersized+remapped, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 3.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 829 op/s; 12079/191457 objects degraded (6.309%) 2024-02-09T17:58:10.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:10 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:58:10.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:10 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:58:10.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:10 smithi162 ceph-mon[44304]: osdmap e839: 12 total, 11 up, 12 in 2024-02-09T17:58:10.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:10 smithi162 ceph-mon[44304]: pgmap v2081: 47 pgs: 1 active+undersized+remapped, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 3.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 829 op/s; 12079/191457 objects degraded (6.309%) 2024-02-09T17:58:10.432 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:10.455 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:58:10.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:10.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:58:10.563 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:58:10.626 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:10.662 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:58:10.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:10.666 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:58:10.766 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:58:10.774 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:58:10.774 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 20, 2024-02-09T17:58:10.774 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 600, 2024-02-09T17:58:10.774 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [ 2024-02-09T17:58:10.774 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.775 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.16440.0:83375 7.3 e686/661 7:c9c8e507:::benchmark_data_smithi179_135804_object83374:head v 686'5229, mlcod=686'5226)", 2024-02-09T17:58:10.775 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:51:18.489551+0000", 2024-02-09T17:58:10.775 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 412.107555193, 2024-02-09T17:58:10.775 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.099479135999999996, 2024-02-09T17:58:10.775 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:58:10.775 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:58:10.775 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:58:10.775 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.775 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:58:10.776 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.489551+0000", 2024-02-09T17:58:10.776 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:58:10.776 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.776 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.776 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:58:10.776 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.489551+0000", 2024-02-09T17:58:10.776 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:58:10.776 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.776 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.776 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:58:10.777 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.489552+0000", 2024-02-09T17:58:10.777 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0279999999999999e-06 2024-02-09T17:58:10.777 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.777 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.777 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:58:10.777 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.489574+0000", 2024-02-09T17:58:10.777 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2113999999999999e-05 2024-02-09T17:58:10.777 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.777 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.778 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:58:10.778 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.489584+0000", 2024-02-09T17:58:10.778 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.4269999999999997e-06 2024-02-09T17:58:10.778 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.778 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.778 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:58:10.778 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.489590+0000", 2024-02-09T17:58:10.778 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.3579999999999999e-06 2024-02-09T17:58:10.779 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.779 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.779 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:58:10.779 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.489644+0000", 2024-02-09T17:58:10.779 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3961000000000001e-05 2024-02-09T17:58:10.779 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.779 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.779 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:58:10.779 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.489665+0000", 2024-02-09T17:58:10.779 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0577e-05 2024-02-09T17:58:10.780 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.780 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.780 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:58:10.780 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.588987+0000", 2024-02-09T17:58:10.780 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.099321999999999994 2024-02-09T17:58:10.780 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.780 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.780 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:58:10.780 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.589030+0000", 2024-02-09T17:58:10.780 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.3671000000000003e-05 2024-02-09T17:58:10.781 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:58:10.781 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:58:10.781 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:58:10.781 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.781 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.781 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.16440.0:83376 7.8 e686/661 7:14268f94:::benchmark_data_smithi179_135804_object83375:head v 686'5198, mlcod=686'5194)", 2024-02-09T17:58:10.781 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:51:18.490376+0000", 2024-02-09T17:58:10.781 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 412.10673018300002, 2024-02-09T17:58:10.781 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.109104801, 2024-02-09T17:58:10.781 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:58:10.782 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:58:10.782 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:58:10.782 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.782 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:58:10.782 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.490376+0000", 2024-02-09T17:58:10.782 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:58:10.782 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.782 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.782 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:58:10.782 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.490376+0000", 2024-02-09T17:58:10.782 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:58:10.783 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.783 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.783 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:58:10.783 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.490377+0000", 2024-02-09T17:58:10.783 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.5499999999999997e-07 2024-02-09T17:58:10.783 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.783 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.783 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:58:10.783 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.490445+0000", 2024-02-09T17:58:10.783 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.8361999999999994e-05 2024-02-09T17:58:10.783 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.784 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.784 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:58:10.784 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.490455+0000", 2024-02-09T17:58:10.784 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.7999999999999993e-06 2024-02-09T17:58:10.784 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.784 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.784 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:58:10.784 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.490461+0000", 2024-02-09T17:58:10.784 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.28e-06 2024-02-09T17:58:10.784 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.784 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.785 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:58:10.785 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.589005+0000", 2024-02-09T17:58:10.785 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.098543447000000006 2024-02-09T17:58:10.785 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.785 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.785 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:58:10.785 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.589030+0000", 2024-02-09T17:58:10.785 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.4782000000000001e-05 2024-02-09T17:58:10.785 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.785 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.786 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:58:10.786 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.599426+0000", 2024-02-09T17:58:10.786 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.010396720999999999 2024-02-09T17:58:10.786 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.786 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.786 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:58:10.786 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.599481+0000", 2024-02-09T17:58:10.786 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.4654000000000002e-05 2024-02-09T17:58:10.786 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:58:10.786 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:58:10.786 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:58:10.786 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.786 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.787 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.16440.0:83381 7.8 e686/661 7:1d33db6a:::benchmark_data_smithi179_135804_object83380:head v 686'5199, mlcod=686'5196)", 2024-02-09T17:58:10.787 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:51:18.495720+0000", 2024-02-09T17:58:10.787 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 412.10138643200003, 2024-02-09T17:58:10.787 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.10379113299999999, 2024-02-09T17:58:10.787 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:58:10.787 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:58:10.787 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:58:10.787 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.787 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:58:10.787 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.495720+0000", 2024-02-09T17:58:10.787 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:58:10.788 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.788 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.788 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:58:10.788 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.495720+0000", 2024-02-09T17:58:10.788 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:58:10.788 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.788 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.788 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:58:10.788 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.495722+0000", 2024-02-09T17:58:10.788 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4890000000000001e-06 2024-02-09T17:58:10.788 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.788 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.789 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:58:10.789 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.495769+0000", 2024-02-09T17:58:10.789 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.7274000000000002e-05 2024-02-09T17:58:10.789 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.789 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.789 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:58:10.789 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.588978+0000", 2024-02-09T17:58:10.789 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.093209146000000007 2024-02-09T17:58:10.789 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.789 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.789 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:58:10.789 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.588992+0000", 2024-02-09T17:58:10.790 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4178e-05 2024-02-09T17:58:10.790 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.790 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.790 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:58:10.790 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.591707+0000", 2024-02-09T17:58:10.790 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0027146589999999999 2024-02-09T17:58:10.790 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.790 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.790 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:58:10.790 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.591728+0000", 2024-02-09T17:58:10.790 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0798999999999999e-05 2024-02-09T17:58:10.790 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.791 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.791 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:58:10.791 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.599487+0000", 2024-02-09T17:58:10.791 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0077592609999999999 2024-02-09T17:58:10.791 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.791 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.791 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:58:10.791 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:51:18.599511+0000", 2024-02-09T17:58:10.791 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.4326999999999999e-05 2024-02-09T17:58:10.791 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:58:10.791 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:58:10.791 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:58:10.792 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.792 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.792 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "MOSDPGPush(7.c 706/682 [PushOp(7:32f3405a:::benchmark_data_smithi179_135804_object46814:head, version: 671'2962, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:32f3405a:::benchmark_data_smithi179_135804_object46814:head@671'2962, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-02-09T17:58:10.792 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:52:10.321219+0000", 2024-02-09T17:58:10.792 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 360.27588704800002, 2024-02-09T17:58:10.792 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.151726903, 2024-02-09T17:58:10.792 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:58:10.792 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T17:58:10.792 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:58:10.792 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.792 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:58:10.793 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.321219+0000", 2024-02-09T17:58:10.793 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:58:10.793 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.793 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.793 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:58:10.793 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.321219+0000", 2024-02-09T17:58:10.793 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:58:10.793 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.793 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.793 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:58:10.793 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.321220+0000", 2024-02-09T17:58:10.793 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.9800000000000003e-07 2024-02-09T17:58:10.794 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.794 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.794 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:58:10.794 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.321276+0000", 2024-02-09T17:58:10.794 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.5442000000000002e-05 2024-02-09T17:58:10.794 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.794 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.794 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:58:10.794 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.321291+0000", 2024-02-09T17:58:10.794 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4966e-05 2024-02-09T17:58:10.794 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.794 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.795 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:58:10.795 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.321302+0000", 2024-02-09T17:58:10.795 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1117e-05 2024-02-09T17:58:10.795 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.795 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.795 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:58:10.795 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.321365+0000", 2024-02-09T17:58:10.795 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.3261000000000003e-05 2024-02-09T17:58:10.795 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.795 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.795 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:58:10.796 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.321377+0000", 2024-02-09T17:58:10.796 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1558e-05 2024-02-09T17:58:10.796 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.796 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.796 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:58:10.796 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.472946+0000", 2024-02-09T17:58:10.796 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.151569761 2024-02-09T17:58:10.796 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:58:10.796 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:58:10.796 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:58:10.796 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.796 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.797 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.16440.0:116779 7.3 e706/661 7:cf34c6cd:::benchmark_data_smithi179_135804_object8988:head v 706'7373, mlcod=706'7371)", 2024-02-09T17:58:10.797 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:52:10.322168+0000", 2024-02-09T17:58:10.797 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 360.27493803900001, 2024-02-09T17:58:10.797 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.16296088, 2024-02-09T17:58:10.797 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:58:10.797 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:58:10.797 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:58:10.797 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.797 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:58:10.797 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.322168+0000", 2024-02-09T17:58:10.797 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:58:10.798 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.798 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.798 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:58:10.798 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.322168+0000", 2024-02-09T17:58:10.798 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:58:10.798 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.798 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.798 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:58:10.798 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.322169+0000", 2024-02-09T17:58:10.798 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.0800000000000005e-07 2024-02-09T17:58:10.798 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.798 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.798 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:58:10.799 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.322174+0000", 2024-02-09T17:58:10.799 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.9570000000000001e-06 2024-02-09T17:58:10.799 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.799 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.799 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:58:10.799 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.322179+0000", 2024-02-09T17:58:10.799 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.0259999999999998e-06 2024-02-09T17:58:10.799 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.799 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.799 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:58:10.799 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.322183+0000", 2024-02-09T17:58:10.799 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.4549999999999997e-06 2024-02-09T17:58:10.799 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.799 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.800 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:58:10.800 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.472977+0000", 2024-02-09T17:58:10.800 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.15079350799999999 2024-02-09T17:58:10.800 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.800 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.800 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:58:10.800 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.473001+0000", 2024-02-09T17:58:10.800 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.4403999999999999e-05 2024-02-09T17:58:10.800 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.801 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.801 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:58:10.801 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.485098+0000", 2024-02-09T17:58:10.801 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.012096361999999999 2024-02-09T17:58:10.801 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.801 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.801 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:58:10.801 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.485129+0000", 2024-02-09T17:58:10.802 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.1659999999999998e-05 2024-02-09T17:58:10.802 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:58:10.802 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:58:10.802 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:58:10.802 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.802 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.802 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.16440.0:116799 7.a e706/661 7:5a8c05a2:::benchmark_data_smithi179_135804_object9008:head v 706'7334, mlcod=706'7332)", 2024-02-09T17:58:10.802 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:52:10.349258+0000", 2024-02-09T17:58:10.802 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 360.24784868099999, 2024-02-09T17:58:10.802 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.139991741, 2024-02-09T17:58:10.802 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:58:10.803 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:58:10.803 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:58:10.803 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.803 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:58:10.803 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.349258+0000", 2024-02-09T17:58:10.803 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:58:10.803 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.803 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.803 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:58:10.803 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.349258+0000", 2024-02-09T17:58:10.803 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:58:10.804 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.804 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.804 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:58:10.804 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.349259+0000", 2024-02-09T17:58:10.804 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2500000000000001e-06 2024-02-09T17:58:10.804 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.804 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.804 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:58:10.804 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.349269+0000", 2024-02-09T17:58:10.804 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.9119999999999993e-06 2024-02-09T17:58:10.804 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.804 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.805 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:58:10.805 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.472952+0000", 2024-02-09T17:58:10.805 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.123683038 2024-02-09T17:58:10.805 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.805 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.805 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:58:10.805 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.472962+0000", 2024-02-09T17:58:10.805 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.8830000000000001e-06 2024-02-09T17:58:10.805 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.805 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.805 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:58:10.805 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.473014+0000", 2024-02-09T17:58:10.806 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1987999999999998e-05 2024-02-09T17:58:10.806 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.806 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.806 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:58:10.806 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.473035+0000", 2024-02-09T17:58:10.806 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0667000000000001e-05 2024-02-09T17:58:10.806 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.806 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.806 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:58:10.806 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.489212+0000", 2024-02-09T17:58:10.806 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.016177252 2024-02-09T17:58:10.806 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.807 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.807 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:58:10.807 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.489250+0000", 2024-02-09T17:58:10.807 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.7750999999999998e-05 2024-02-09T17:58:10.807 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:58:10.807 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:58:10.807 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:58:10.807 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.807 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.807 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.16440.0:116805 7.8 e706/661 7:1a85ab55:::benchmark_data_smithi179_135804_object9014:head v 706'7283, mlcod=706'7281)", 2024-02-09T17:58:10.807 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:52:10.358900+0000", 2024-02-09T17:58:10.808 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 360.238206822, 2024-02-09T17:58:10.808 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.121220444, 2024-02-09T17:58:10.808 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:58:10.808 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:58:10.808 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:58:10.808 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.808 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:58:10.808 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.358900+0000", 2024-02-09T17:58:10.808 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:58:10.808 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.808 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.808 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:58:10.809 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.358900+0000", 2024-02-09T17:58:10.809 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:58:10.809 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.809 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.809 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:58:10.809 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.358901+0000", 2024-02-09T17:58:10.809 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.7370000000000001e-06 2024-02-09T17:58:10.809 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.809 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.809 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:58:10.809 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.358910+0000", 2024-02-09T17:58:10.809 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.4630000000000001e-06 2024-02-09T17:58:10.810 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.810 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.810 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:58:10.810 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.472964+0000", 2024-02-09T17:58:10.810 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.11405451799999999 2024-02-09T17:58:10.810 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.810 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.810 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:58:10.810 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.472974+0000", 2024-02-09T17:58:10.810 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.4129999999999995e-06 2024-02-09T17:58:10.810 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.810 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.810 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:58:10.811 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.473037+0000", 2024-02-09T17:58:10.811 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.3367999999999994e-05 2024-02-09T17:58:10.811 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.811 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.811 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:58:10.811 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.473059+0000", 2024-02-09T17:58:10.811 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2010000000000001e-05 2024-02-09T17:58:10.811 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.811 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.811 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:58:10.811 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.480071+0000", 2024-02-09T17:58:10.811 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0070118530000000002 2024-02-09T17:58:10.812 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.812 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.812 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:58:10.812 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:52:10.480120+0000", 2024-02-09T17:58:10.812 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.9082000000000002e-05 2024-02-09T17:58:10.812 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:58:10.812 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:58:10.812 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:58:10.812 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.812 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.812 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.16440.0:185183 7.3 e749/661 7:c456d19d:::benchmark_data_smithi179_135804_object77392:head v 749'11612, mlcod=749'11609)", 2024-02-09T17:58:10.813 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:54:02.778478+0000", 2024-02-09T17:58:10.813 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 247.81862881500001, 2024-02-09T17:58:10.813 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.069240145000000003, 2024-02-09T17:58:10.813 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:58:10.813 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:58:10.813 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:58:10.813 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.813 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:58:10.813 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:02.778478+0000", 2024-02-09T17:58:10.813 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:58:10.813 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.813 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.813 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:58:10.813 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:02.778478+0000", 2024-02-09T17:58:10.814 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:58:10.814 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.814 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.814 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:58:10.814 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:02.778479+0000", 2024-02-09T17:58:10.814 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.173e-06 2024-02-09T17:58:10.814 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.814 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.814 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:58:10.814 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:02.778489+0000", 2024-02-09T17:58:10.814 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.7620000000000007e-06 2024-02-09T17:58:10.814 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.815 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.815 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:58:10.815 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:02.778509+0000", 2024-02-09T17:58:10.815 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0548999999999999e-05 2024-02-09T17:58:10.815 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.815 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.815 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:58:10.815 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:02.778521+0000", 2024-02-09T17:58:10.815 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1935e-05 2024-02-09T17:58:10.815 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.815 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.816 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:58:10.816 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:02.780020+0000", 2024-02-09T17:58:10.816 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.001498634 2024-02-09T17:58:10.816 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.816 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.816 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:58:10.816 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:02.780041+0000", 2024-02-09T17:58:10.816 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.1308999999999999e-05 2024-02-09T17:58:10.816 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.816 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.816 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:58:10.816 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:02.847678+0000", 2024-02-09T17:58:10.816 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.067637016999999994 2024-02-09T17:58:10.817 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.817 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.817 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:58:10.817 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:02.847718+0000", 2024-02-09T17:58:10.817 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9765999999999998e-05 2024-02-09T17:58:10.817 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:58:10.817 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:58:10.817 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:58:10.817 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.817 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.817 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.16440.0:208623 7.3 e768/661 7:cdf746f9:::benchmark_data_smithi179_135804_object100832:head v 768'13134, mlcod=768'13131)", 2024-02-09T17:58:10.817 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:54:36.866423+0000", 2024-02-09T17:58:10.817 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 213.730683677, 2024-02-09T17:58:10.818 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.096561118000000001, 2024-02-09T17:58:10.818 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:58:10.818 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:58:10.818 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:58:10.818 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.818 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:58:10.818 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:36.866423+0000", 2024-02-09T17:58:10.818 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:58:10.818 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.818 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.818 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:58:10.819 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:36.866423+0000", 2024-02-09T17:58:10.819 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:58:10.819 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.819 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.819 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:58:10.819 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:36.866424+0000", 2024-02-09T17:58:10.819 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.141e-06 2024-02-09T17:58:10.819 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.819 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.819 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:58:10.819 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:36.866434+0000", 2024-02-09T17:58:10.820 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.9340000000000001e-06 2024-02-09T17:58:10.820 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.820 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.820 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:58:10.820 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:36.866449+0000", 2024-02-09T17:58:10.820 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4789e-05 2024-02-09T17:58:10.820 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.820 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.820 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:58:10.820 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:36.866457+0000", 2024-02-09T17:58:10.820 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.8259999999999999e-06 2024-02-09T17:58:10.820 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.821 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.821 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:58:10.821 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:36.936515+0000", 2024-02-09T17:58:10.821 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.070057369999999994 2024-02-09T17:58:10.821 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.821 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.821 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:58:10.821 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:36.936539+0000", 2024-02-09T17:58:10.821 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.4221999999999999e-05 2024-02-09T17:58:10.821 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.821 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.822 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:58:10.822 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:36.962944+0000", 2024-02-09T17:58:10.822 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.026404891 2024-02-09T17:58:10.822 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.822 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.822 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:58:10.822 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:36.962984+0000", 2024-02-09T17:58:10.822 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9944999999999998e-05 2024-02-09T17:58:10.822 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:58:10.822 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:58:10.822 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:58:10.823 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.823 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.823 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.16440.0:208642 7.e 7:70525f8e:::benchmark_data_smithi179_135804_object100851:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e768)", 2024-02-09T17:58:10.823 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:54:36.880425+0000", 2024-02-09T17:58:10.823 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 213.716681822, 2024-02-09T17:58:10.823 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.070062207000000001, 2024-02-09T17:58:10.823 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:58:10.823 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:58:10.823 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T17:58:10.823 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.16440", 2024-02-09T17:58:10.823 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:58:10.823 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 208642 2024-02-09T17:58:10.824 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.824 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:58:10.824 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.824 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:58:10.824 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:36.880425+0000", 2024-02-09T17:58:10.824 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:58:10.824 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.824 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.824 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:58:10.825 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:36.880425+0000", 2024-02-09T17:58:10.825 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:58:10.825 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.825 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.825 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:58:10.825 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:36.880427+0000", 2024-02-09T17:58:10.825 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.9599999999999999e-06 2024-02-09T17:58:10.825 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.825 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.825 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:58:10.826 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:36.880434+0000", 2024-02-09T17:58:10.826 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.6869999999999994e-06 2024-02-09T17:58:10.826 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.826 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.826 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:58:10.826 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:36.936476+0000", 2024-02-09T17:58:10.826 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.056041858 2024-02-09T17:58:10.826 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.826 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.826 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:58:10.826 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:36.936493+0000", 2024-02-09T17:58:10.827 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.7255e-05 2024-02-09T17:58:10.827 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.827 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.827 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:58:10.827 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:36.936603+0000", 2024-02-09T17:58:10.827 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000109925 2024-02-09T17:58:10.827 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.827 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.827 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:58:10.827 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:36.936697+0000", 2024-02-09T17:58:10.827 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.3706e-05 2024-02-09T17:58:10.827 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.827 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.828 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 9,10", 2024-02-09T17:58:10.828 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:36.936775+0000", 2024-02-09T17:58:10.828 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.8349000000000005e-05 2024-02-09T17:58:10.828 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.828 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.828 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:10.828 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:36.950345+0000", 2024-02-09T17:58:10.828 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.013569500999999999 2024-02-09T17:58:10.828 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.828 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.828 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T17:58:10.829 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:36.950357+0000", 2024-02-09T17:58:10.829 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1640999999999999e-05 2024-02-09T17:58:10.829 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.829 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.829 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:10.829 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:36.950398+0000", 2024-02-09T17:58:10.829 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.1186000000000002e-05 2024-02-09T17:58:10.829 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.829 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.829 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:58:10.829 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:36.950450+0000", 2024-02-09T17:58:10.829 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2104000000000001e-05 2024-02-09T17:58:10.829 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.829 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.830 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:58:10.830 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:54:36.950487+0000", 2024-02-09T17:58:10.830 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.7035e-05 2024-02-09T17:58:10.830 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:58:10.830 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:58:10.830 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:58:10.830 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.830 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.830 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.26305.0:121261 8.d e821/785 8:b7159e57:::benchmark_data_smithi179_148205_object121260:head v 821'7523, mlcod=821'7514)", 2024-02-09T17:58:10.830 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:56:54.577716+0000", 2024-02-09T17:58:10.830 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 76.019390901999998, 2024-02-09T17:58:10.831 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.032253903, 2024-02-09T17:58:10.831 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:58:10.831 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:58:10.831 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:58:10.831 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.831 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:58:10.831 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:54.577716+0000", 2024-02-09T17:58:10.831 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:58:10.831 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.831 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.831 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:58:10.831 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:54.577716+0000", 2024-02-09T17:58:10.832 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:58:10.832 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.832 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.832 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:58:10.832 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:54.577716+0000", 2024-02-09T17:58:10.832 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.4699999999999998e-07 2024-02-09T17:58:10.832 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.832 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.832 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:58:10.832 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:54.577790+0000", 2024-02-09T17:58:10.832 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.3700000000000002e-05 2024-02-09T17:58:10.833 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.833 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.833 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:58:10.833 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:54.577797+0000", 2024-02-09T17:58:10.833 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.4490000000000002e-06 2024-02-09T17:58:10.833 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.833 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.833 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:58:10.833 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:54.577802+0000", 2024-02-09T17:58:10.833 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.6489999999999999e-06 2024-02-09T17:58:10.833 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.833 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.833 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:58:10.834 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:54.597501+0000", 2024-02-09T17:58:10.834 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.019699455000000001 2024-02-09T17:58:10.834 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.834 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.834 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:58:10.834 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:54.597520+0000", 2024-02-09T17:58:10.834 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.9233999999999999e-05 2024-02-09T17:58:10.834 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.834 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.834 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:58:10.834 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:54.609914+0000", 2024-02-09T17:58:10.835 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.012393886999999999 2024-02-09T17:58:10.835 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.835 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.835 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:58:10.835 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:54.609969+0000", 2024-02-09T17:58:10.835 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.5281999999999997e-05 2024-02-09T17:58:10.835 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:58:10.835 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:58:10.835 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:58:10.835 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.835 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.835 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.26305.0:134702 8.1 e823/785 8:84712802:::benchmark_data_smithi179_148205_object11630:head v 823'8335, mlcod=823'8330)", 2024-02-09T17:58:10.835 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:57:08.867674+0000", 2024-02-09T17:58:10.836 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 61.729432383999999, 2024-02-09T17:58:10.836 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.033739643, 2024-02-09T17:58:10.836 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:58:10.836 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:58:10.836 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:58:10.836 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.836 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:58:10.836 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:08.867674+0000", 2024-02-09T17:58:10.836 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:58:10.836 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.836 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.837 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:58:10.837 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:08.867674+0000", 2024-02-09T17:58:10.837 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:58:10.837 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.837 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.837 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:58:10.837 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:08.867675+0000", 2024-02-09T17:58:10.837 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.7800000000000001e-07 2024-02-09T17:58:10.837 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.837 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.838 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:58:10.838 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:08.867681+0000", 2024-02-09T17:58:10.838 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.0889999999999996e-06 2024-02-09T17:58:10.838 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.838 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.838 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:58:10.838 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:08.867691+0000", 2024-02-09T17:58:10.838 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.039e-05 2024-02-09T17:58:10.838 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.838 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.839 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:58:10.839 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:08.867698+0000", 2024-02-09T17:58:10.839 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.5300000000000002e-06 2024-02-09T17:58:10.839 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.839 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.839 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:58:10.839 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:08.871012+0000", 2024-02-09T17:58:10.839 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0033144089999999999 2024-02-09T17:58:10.839 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.839 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.839 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:58:10.840 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:08.871032+0000", 2024-02-09T17:58:10.840 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0122000000000001e-05 2024-02-09T17:58:10.840 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.840 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.840 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:58:10.840 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:08.901366+0000", 2024-02-09T17:58:10.840 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.030333280000000001 2024-02-09T17:58:10.840 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.840 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.840 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:58:10.840 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:08.901414+0000", 2024-02-09T17:58:10.840 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.8145000000000001e-05 2024-02-09T17:58:10.841 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:58:10.841 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:58:10.841 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:58:10.841 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.841 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.841 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.26305.0:134705 8.1 e823/785 8:80c7e9c7:::benchmark_data_smithi179_148205_object11633:head v 823'8337, mlcod=823'8330)", 2024-02-09T17:58:10.841 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:57:08.876165+0000", 2024-02-09T17:58:10.841 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 61.720941588000002, 2024-02-09T17:58:10.841 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.038494659000000001, 2024-02-09T17:58:10.841 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:58:10.842 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:58:10.842 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:58:10.842 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.842 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:58:10.842 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:08.876165+0000", 2024-02-09T17:58:10.842 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:58:10.842 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.842 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.842 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:58:10.842 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:08.876165+0000", 2024-02-09T17:58:10.842 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:58:10.842 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.843 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.843 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:58:10.843 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:08.876166+0000", 2024-02-09T17:58:10.843 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.8700000000000005e-07 2024-02-09T17:58:10.843 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.843 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.843 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:58:10.843 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:08.876172+0000", 2024-02-09T17:58:10.843 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.314e-06 2024-02-09T17:58:10.843 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.843 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.844 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:58:10.844 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:08.876183+0000", 2024-02-09T17:58:10.844 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1476e-05 2024-02-09T17:58:10.844 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.844 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.844 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:58:10.844 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:08.876191+0000", 2024-02-09T17:58:10.844 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.9310000000000008e-06 2024-02-09T17:58:10.844 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.844 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.844 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:58:10.844 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:08.888759+0000", 2024-02-09T17:58:10.844 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.012567815 2024-02-09T17:58:10.845 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.845 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.845 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:58:10.845 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:08.888781+0000", 2024-02-09T17:58:10.845 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.1687000000000001e-05 2024-02-09T17:58:10.845 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.845 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.845 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:58:10.845 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:08.914620+0000", 2024-02-09T17:58:10.845 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.025839269000000002 2024-02-09T17:58:10.845 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.845 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.846 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:58:10.846 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:08.914660+0000", 2024-02-09T17:58:10.846 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9379999999999999e-05 2024-02-09T17:58:10.846 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:58:10.846 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:58:10.846 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:58:10.846 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.846 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.846 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.26305.0:134939 8.1 e823/785 8:864e2f43:::benchmark_data_smithi179_148205_object11867:head v 823'8354, mlcod=823'8352)", 2024-02-09T17:58:10.846 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:57:09.148880+0000", 2024-02-09T17:58:10.846 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 61.448226335000001, 2024-02-09T17:58:10.846 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.040197447999999997, 2024-02-09T17:58:10.847 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:58:10.847 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:58:10.847 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:58:10.847 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.847 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:58:10.847 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:09.148880+0000", 2024-02-09T17:58:10.847 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:58:10.847 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.847 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.847 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:58:10.847 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:09.148880+0000", 2024-02-09T17:58:10.847 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:58:10.847 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.848 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.848 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:58:10.848 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:09.148881+0000", 2024-02-09T17:58:10.848 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2399999999999998e-07 2024-02-09T17:58:10.848 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.848 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.848 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:58:10.848 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:09.148887+0000", 2024-02-09T17:58:10.848 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.2589999999999997e-06 2024-02-09T17:58:10.848 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.848 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.849 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:58:10.849 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:09.148896+0000", 2024-02-09T17:58:10.849 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.6969999999999999e-06 2024-02-09T17:58:10.849 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.849 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.849 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:58:10.849 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:09.148902+0000", 2024-02-09T17:58:10.849 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.9109999999999998e-06 2024-02-09T17:58:10.849 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.849 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.849 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:58:10.850 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:09.148973+0000", 2024-02-09T17:58:10.850 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.1917999999999998e-05 2024-02-09T17:58:10.850 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.850 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.850 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:58:10.850 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:09.148997+0000", 2024-02-09T17:58:10.850 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.3881999999999999e-05 2024-02-09T17:58:10.850 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.850 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.850 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:58:10.850 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:09.189049+0000", 2024-02-09T17:58:10.850 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.040051224000000003 2024-02-09T17:58:10.851 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.851 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.851 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:58:10.851 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:09.189078+0000", 2024-02-09T17:58:10.851 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.9033000000000001e-05 2024-02-09T17:58:10.851 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:58:10.851 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:58:10.851 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:58:10.851 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.851 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.851 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.26305.0:168509 8.e e831/785 8:7a3bd527:::benchmark_data_smithi179_148205_object45437:head v 831'10556, mlcod=831'10548)", 2024-02-09T17:58:10.851 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:57:49.796116+0000", 2024-02-09T17:58:10.851 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 20.800990122999998, 2024-02-09T17:58:10.852 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.033442184, 2024-02-09T17:58:10.852 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:58:10.852 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:58:10.852 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:58:10.852 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.852 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:58:10.852 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796116+0000", 2024-02-09T17:58:10.852 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:58:10.852 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.852 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.852 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:58:10.852 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796116+0000", 2024-02-09T17:58:10.853 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:58:10.853 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.853 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.853 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:58:10.853 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796117+0000", 2024-02-09T17:58:10.853 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.5800000000000001e-07 2024-02-09T17:58:10.853 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.853 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.853 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:58:10.853 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796120+0000", 2024-02-09T17:58:10.853 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.067e-06 2024-02-09T17:58:10.853 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.853 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.854 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:58:10.854 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796125+0000", 2024-02-09T17:58:10.854 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3240000000000002e-06 2024-02-09T17:58:10.854 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.854 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.854 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:58:10.854 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796129+0000", 2024-02-09T17:58:10.854 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.0350000000000003e-06 2024-02-09T17:58:10.854 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.854 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.854 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:58:10.854 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.816103+0000", 2024-02-09T17:58:10.854 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.019973666000000001 2024-02-09T17:58:10.855 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.855 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.855 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:58:10.855 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.816120+0000", 2024-02-09T17:58:10.855 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.7736000000000001e-05 2024-02-09T17:58:10.855 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.855 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.855 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:58:10.855 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.829521+0000", 2024-02-09T17:58:10.855 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.013401006 2024-02-09T17:58:10.855 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.855 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.856 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:58:10.856 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.829559+0000", 2024-02-09T17:58:10.856 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.7092000000000003e-05 2024-02-09T17:58:10.856 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:58:10.856 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:58:10.856 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:58:10.856 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.856 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.856 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.26305.0:168514 8.e e831/785 8:7ea3477d:::benchmark_data_smithi179_148205_object45442:head v 831'10557, mlcod=831'10548)", 2024-02-09T17:58:10.856 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:57:49.796134+0000", 2024-02-09T17:58:10.856 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 20.800972164000001, 2024-02-09T17:58:10.857 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.045335991999999999, 2024-02-09T17:58:10.857 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:58:10.857 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:58:10.857 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:58:10.857 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.857 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:58:10.857 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796134+0000", 2024-02-09T17:58:10.857 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:58:10.857 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.857 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.857 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:58:10.857 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796134+0000", 2024-02-09T17:58:10.857 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:58:10.858 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.858 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.858 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:58:10.858 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796135+0000", 2024-02-09T17:58:10.858 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.34e-07 2024-02-09T17:58:10.858 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.858 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.858 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:58:10.858 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796141+0000", 2024-02-09T17:58:10.858 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.049e-06 2024-02-09T17:58:10.858 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.858 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.858 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:58:10.859 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796146+0000", 2024-02-09T17:58:10.859 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6910000000000002e-06 2024-02-09T17:58:10.859 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.859 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.859 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:58:10.859 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796151+0000", 2024-02-09T17:58:10.859 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2930000000000002e-06 2024-02-09T17:58:10.859 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.859 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.859 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:58:10.859 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.820112+0000", 2024-02-09T17:58:10.859 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.023961469999999999 2024-02-09T17:58:10.859 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.860 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.860 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:58:10.860 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.820133+0000", 2024-02-09T17:58:10.860 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.1214000000000001e-05 2024-02-09T17:58:10.860 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.860 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.860 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:58:10.860 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.841426+0000", 2024-02-09T17:58:10.860 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.021292789999999999 2024-02-09T17:58:10.860 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.860 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.860 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:58:10.860 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.841470+0000", 2024-02-09T17:58:10.861 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.4251e-05 2024-02-09T17:58:10.861 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:58:10.861 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:58:10.861 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:58:10.861 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.861 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.861 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.26305.0:168516 8.e e831/785 8:7a9c00b8:::benchmark_data_smithi179_148205_object45444:head v 831'10558, mlcod=831'10548)", 2024-02-09T17:58:10.861 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:57:49.796156+0000", 2024-02-09T17:58:10.861 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 20.800950905000001, 2024-02-09T17:58:10.861 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.045346616999999999, 2024-02-09T17:58:10.862 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:58:10.862 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:58:10.862 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:58:10.862 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.862 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:58:10.862 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796156+0000", 2024-02-09T17:58:10.862 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:58:10.862 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.862 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.862 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:58:10.862 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796156+0000", 2024-02-09T17:58:10.862 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:58:10.863 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.863 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.863 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:58:10.863 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796156+0000", 2024-02-09T17:58:10.863 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0599999999999999e-07 2024-02-09T17:58:10.863 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.863 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.863 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:58:10.863 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796158+0000", 2024-02-09T17:58:10.863 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.593e-06 2024-02-09T17:58:10.863 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.863 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.864 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:58:10.864 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796164+0000", 2024-02-09T17:58:10.864 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.037e-06 2024-02-09T17:58:10.864 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.864 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.864 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:58:10.864 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796168+0000", 2024-02-09T17:58:10.864 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.6880000000000001e-06 2024-02-09T17:58:10.864 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.864 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.865 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:58:10.865 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.823414+0000", 2024-02-09T17:58:10.865 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.027245933 2024-02-09T17:58:10.865 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.865 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.865 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:58:10.865 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.823428+0000", 2024-02-09T17:58:10.865 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4035e-05 2024-02-09T17:58:10.866 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.866 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.866 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:58:10.866 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.841475+0000", 2024-02-09T17:58:10.866 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.018046627999999999 2024-02-09T17:58:10.866 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.866 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.866 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:58:10.866 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.841502+0000", 2024-02-09T17:58:10.866 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.7497000000000001e-05 2024-02-09T17:58:10.866 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:58:10.867 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:58:10.867 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:58:10.867 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.867 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.867 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.26305.0:168535 8.e e831/785 8:738351cd:::benchmark_data_smithi179_148205_object45463:head v 831'10559, mlcod=831'10548)", 2024-02-09T17:58:10.867 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:57:49.796173+0000", 2024-02-09T17:58:10.867 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 20.800933848, 2024-02-09T17:58:10.867 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.045364935000000002, 2024-02-09T17:58:10.867 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:58:10.867 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:58:10.867 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:58:10.867 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.868 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:58:10.868 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796173+0000", 2024-02-09T17:58:10.868 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:58:10.868 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.868 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.868 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:58:10.868 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796173+0000", 2024-02-09T17:58:10.868 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:58:10.868 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.868 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.868 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:58:10.868 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796173+0000", 2024-02-09T17:58:10.869 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.31e-07 2024-02-09T17:58:10.869 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.869 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.869 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:58:10.869 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796176+0000", 2024-02-09T17:58:10.869 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.0189999999999998e-06 2024-02-09T17:58:10.869 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.869 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.869 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:58:10.869 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796181+0000", 2024-02-09T17:58:10.869 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6280000000000002e-06 2024-02-09T17:58:10.869 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.869 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.870 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:58:10.870 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796185+0000", 2024-02-09T17:58:10.870 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.6899999999999998e-06 2024-02-09T17:58:10.870 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.870 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.870 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:58:10.870 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.826413+0000", 2024-02-09T17:58:10.870 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.030228034000000001 2024-02-09T17:58:10.870 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.870 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.870 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:58:10.870 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.826427+0000", 2024-02-09T17:58:10.870 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4259e-05 2024-02-09T17:58:10.871 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.871 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.871 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:58:10.871 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.841509+0000", 2024-02-09T17:58:10.871 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.015081762 2024-02-09T17:58:10.871 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.871 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.871 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:58:10.871 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.841538+0000", 2024-02-09T17:58:10.871 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.8411999999999998e-05 2024-02-09T17:58:10.871 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:58:10.871 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:58:10.871 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:58:10.872 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.872 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.872 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.26305.0:168570 8.e e831/785 8:7ca3aecf:::benchmark_data_smithi179_148205_object45498:head v 831'10560, mlcod=831'10548)", 2024-02-09T17:58:10.872 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:57:49.796190+0000", 2024-02-09T17:58:10.872 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 20.8009168, 2024-02-09T17:58:10.872 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.045370932000000003, 2024-02-09T17:58:10.872 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:58:10.872 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:58:10.872 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:58:10.872 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.872 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:58:10.873 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796190+0000", 2024-02-09T17:58:10.873 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:58:10.873 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.873 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.873 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:58:10.873 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796190+0000", 2024-02-09T17:58:10.873 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:58:10.873 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.873 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.873 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:58:10.873 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796190+0000", 2024-02-09T17:58:10.874 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4999999999999999e-07 2024-02-09T17:58:10.874 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.874 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.874 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:58:10.874 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796195+0000", 2024-02-09T17:58:10.874 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.203e-06 2024-02-09T17:58:10.874 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.874 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.874 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:58:10.874 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796201+0000", 2024-02-09T17:58:10.874 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6919999999999999e-06 2024-02-09T17:58:10.874 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.874 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.875 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:58:10.875 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796204+0000", 2024-02-09T17:58:10.875 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.7400000000000002e-06 2024-02-09T17:58:10.875 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.875 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.875 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:58:10.875 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.829598+0000", 2024-02-09T17:58:10.875 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.033393316999999999 2024-02-09T17:58:10.875 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.875 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.875 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:58:10.875 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.829619+0000", 2024-02-09T17:58:10.875 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.1075999999999999e-05 2024-02-09T17:58:10.876 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.876 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.876 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:58:10.876 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.841542+0000", 2024-02-09T17:58:10.876 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.011923023 2024-02-09T17:58:10.876 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.876 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.876 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:58:10.876 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.841561+0000", 2024-02-09T17:58:10.876 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.8731000000000001e-05 2024-02-09T17:58:10.876 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:58:10.877 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:58:10.877 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:58:10.877 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.877 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.877 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.26305.0:168592 8.e e831/785 8:72e426c3:::benchmark_data_smithi179_148205_object45520:head v 831'10561, mlcod=831'10548)", 2024-02-09T17:58:10.877 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:57:49.808546+0000", 2024-02-09T17:58:10.877 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 20.788560111999999, 2024-02-09T17:58:10.877 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.033030627999999999, 2024-02-09T17:58:10.877 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T17:58:10.877 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:58:10.877 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T17:58:10.877 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.877 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T17:58:10.878 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.808546+0000", 2024-02-09T17:58:10.878 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:58:10.878 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.878 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.878 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T17:58:10.878 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.808546+0000", 2024-02-09T17:58:10.878 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T17:58:10.878 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.878 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.878 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T17:58:10.878 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.808547+0000", 2024-02-09T17:58:10.878 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.5600000000000004e-07 2024-02-09T17:58:10.878 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.879 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.879 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T17:58:10.879 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.808554+0000", 2024-02-09T17:58:10.879 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.9750000000000001e-06 2024-02-09T17:58:10.879 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.879 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.879 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T17:58:10.879 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.808569+0000", 2024-02-09T17:58:10.879 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4421e-05 2024-02-09T17:58:10.879 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.879 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.879 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T17:58:10.880 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.808576+0000", 2024-02-09T17:58:10.880 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.5299999999999999e-06 2024-02-09T17:58:10.880 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.880 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.880 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T17:58:10.880 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.833756+0000", 2024-02-09T17:58:10.880 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.025180146 2024-02-09T17:58:10.880 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.880 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.880 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T17:58:10.881 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.833780+0000", 2024-02-09T17:58:10.881 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.349e-05 2024-02-09T17:58:10.881 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.881 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.881 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T17:58:10.881 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.841563+0000", 2024-02-09T17:58:10.881 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0077834769999999996 2024-02-09T17:58:10.881 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T17:58:10.881 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T17:58:10.881 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T17:58:10.881 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.841577+0000", 2024-02-09T17:58:10.881 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3733e-05 2024-02-09T17:58:10.882 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:58:10.882 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:58:10.882 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:58:10.882 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T17:58:10.882 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T17:58:10.882 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:58:10.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:10.884 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:10.884 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:58:10.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:58:10.985 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:58:11.033 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:11.062 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:58:11.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:11.070 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:58:11.171 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:58:11.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:11 smithi162 ceph-mon[44304]: osdmap e840: 12 total, 11 up, 12 in 2024-02-09T17:58:11.223 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:11.255 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:58:11.266 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:11.267 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:58:11.368 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:58:11.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:11 smithi179 ceph-mon[43975]: osdmap e840: 12 total, 11 up, 12 in 2024-02-09T17:58:11.432 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:11.458 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:58:11.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:11.464 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:58:11.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:11 smithi099 ceph-mon[39420]: osdmap e840: 12 total, 11 up, 12 in 2024-02-09T17:58:11.565 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:58:11.593 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:11.613 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:58:11.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:11.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:58:11.717 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:58:11.753 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:11.941 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:58:12.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:12 smithi099 ceph-mon[39420]: pgmap v2083: 47 pgs: 1 active+undersized+remapped, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 3.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 649 op/s; 12079/190083 objects degraded (6.355%) 2024-02-09T17:58:12.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:12 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 12079/191457 objects degraded (6.309%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:58:12.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:12 smithi179 ceph-mon[43975]: pgmap v2083: 47 pgs: 1 active+undersized+remapped, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 3.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 649 op/s; 12079/190083 objects degraded (6.355%) 2024-02-09T17:58:12.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:12 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 12079/191457 objects degraded (6.309%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:58:12.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:12 smithi162 ceph-mon[44304]: pgmap v2083: 47 pgs: 1 active+undersized+remapped, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 3.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 649 op/s; 12079/190083 objects degraded (6.355%) 2024-02-09T17:58:12.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:12 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 12079/191457 objects degraded (6.309%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:58:14.415 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:58:14.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:14.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:58:14.441 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-09T17:58:14.483 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 2] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2, 4, 5, 6, 7, 9, 10, 11, 8] 2024-02-09T17:58:14.484 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:58:14.484 INFO:tasks.thrashosds.thrasher:inject_pause on osd.9 2024-02-09T17:58:14.484 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-09T17:58:14.484 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-09T17:58:14.484 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.9 config set filestore_inject_stall 3 2024-02-09T17:58:14.522 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:58:14.550 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:14 smithi099 ceph-mon[39420]: pgmap v2084: 47 pgs: 1 active+undersized+remapped, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 3.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 861 op/s; 11375/182847 objects degraded (6.221%) 2024-02-09T17:58:14.555 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:14.659 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:58:14.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:14.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:58:14.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:14 smithi179 ceph-mon[43975]: pgmap v2084: 47 pgs: 1 active+undersized+remapped, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 3.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 861 op/s; 11375/182847 objects degraded (6.221%) 2024-02-09T17:58:14.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:14 smithi162 ceph-mon[44304]: pgmap v2084: 47 pgs: 1 active+undersized+remapped, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 3.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 861 op/s; 11375/182847 objects degraded (6.221%) 2024-02-09T17:58:14.757 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:58:14.764 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:58:14.829 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:14.851 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:58:14.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:14.857 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:58:14.908 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:58:14.958 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:58:14.987 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:15.025 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:58:15.241 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:58:15.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:15.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:58:15.333 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is unset 2024-02-09T17:58:15.348 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:58:15.424 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:15.532 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:58:15.539 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:15 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1871153186' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:58:15.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:15.540 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:58:15.564 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:15 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1871153186' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:58:15.640 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:58:15.672 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:15 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1871153186' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:58:15.676 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:16.317 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:16 smithi162 ceph-mon[44304]: pgmap v2085: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 3.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 723 op/s; 11375/178545 objects degraded (6.371%) 2024-02-09T17:58:16.318 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:16 smithi162 ceph-mon[44304]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:58:16.318 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:16 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1871153186' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:58:16.318 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:16 smithi162 ceph-mon[44304]: osdmap e841: 12 total, 11 up, 12 in 2024-02-09T17:58:16.318 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:16 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:58:16.318 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:16 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:58:16.318 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:16 smithi162 ceph-mon[44304]: 1.3 scrub starts 2024-02-09T17:58:16.318 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:16 smithi162 ceph-mon[44304]: 1.6 scrub starts 2024-02-09T17:58:16.318 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:16 smithi162 ceph-mon[44304]: 1.6 scrub ok 2024-02-09T17:58:16.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:16 smithi099 ceph-mon[39420]: pgmap v2085: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 3.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 723 op/s; 11375/178545 objects degraded (6.371%) 2024-02-09T17:58:16.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:16 smithi099 ceph-mon[39420]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:58:16.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:16 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1871153186' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:58:16.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:16 smithi099 ceph-mon[39420]: osdmap e841: 12 total, 11 up, 12 in 2024-02-09T17:58:16.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:16 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:58:16.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:16 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:58:16.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:16 smithi099 ceph-mon[39420]: 1.3 scrub starts 2024-02-09T17:58:16.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:16 smithi099 ceph-mon[39420]: 1.6 scrub starts 2024-02-09T17:58:16.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:16 smithi099 ceph-mon[39420]: 1.6 scrub ok 2024-02-09T17:58:16.631 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:58:16.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:16.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:58:16.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:16 smithi179 ceph-mon[43975]: pgmap v2085: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 3.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 723 op/s; 11375/178545 objects degraded (6.371%) 2024-02-09T17:58:16.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:16 smithi179 ceph-mon[43975]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:58:16.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:16 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1871153186' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:58:16.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:16 smithi179 ceph-mon[43975]: osdmap e841: 12 total, 11 up, 12 in 2024-02-09T17:58:16.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:16 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:58:16.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:16 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:58:16.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:16 smithi179 ceph-mon[43975]: 1.3 scrub starts 2024-02-09T17:58:16.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:16 smithi179 ceph-mon[43975]: 1.6 scrub starts 2024-02-09T17:58:16.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:16 smithi179 ceph-mon[43975]: 1.6 scrub ok 2024-02-09T17:58:16.738 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:58:16.770 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:16.815 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:58:16.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:16.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:58:16.928 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:58:16.962 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:17.025 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:58:17.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:17.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:58:17.131 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:58:17.132 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:58:17.132 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-09T17:58:17.133 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-02-09T17:58:17.133 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:58:17.209 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:17.258 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:58:17.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:17.272 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:58:17.335 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:17 smithi162 ceph-mon[44304]: 1.3 scrub ok 2024-02-09T17:58:17.335 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:17 smithi162 ceph-mon[44304]: osdmap e842: 12 total, 11 up, 12 in 2024-02-09T17:58:17.336 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:17 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 11375/178545 objects degraded (6.371%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:58:17.372 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:58:17.406 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:58:17.406 INFO:teuthology.orchestra.run.smithi179.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-02-09T17:58:17.406 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:58:17.439 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:17.446 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:17 smithi179 ceph-mon[43975]: 1.3 scrub ok 2024-02-09T17:58:17.446 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:17 smithi179 ceph-mon[43975]: osdmap e842: 12 total, 11 up, 12 in 2024-02-09T17:58:17.447 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:17 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 11375/178545 objects degraded (6.371%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:58:17.479 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:58:17.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:17.485 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:58:17.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:17 smithi099 ceph-mon[39420]: 1.3 scrub ok 2024-02-09T17:58:17.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:17 smithi099 ceph-mon[39420]: osdmap e842: 12 total, 11 up, 12 in 2024-02-09T17:58:17.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:17 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 11375/178545 objects degraded (6.371%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:58:17.585 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:58:17.615 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:17.875 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:58:17.916 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:58:17.922 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-09T17:58:17.950 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:58:17.967 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:58:17.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:17.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:58:17.983 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:58:18.023 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:58:18.050 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:58:18.074 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:58:18.079 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:58:18.121 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:18.124 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:58:18.159 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:58:18.185 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:58:18.207 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:58:18.207 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:58:18.207 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:58:18.208 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:58:18.208 INFO:teuthology.orchestra.run.smithi099.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-09T17:58:18.208 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:58:18.208 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:58:18.208 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:58:18.208 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:58:18.208 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:58:18.208 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:58:18.208 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:58:18.208 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:58:18.288 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:58:18.412 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:18 smithi179 ceph-mon[43975]: 1.f scrub starts 2024-02-09T17:58:18.412 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:18 smithi179 ceph-mon[43975]: pgmap v2088: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 3.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 670 op/s; 11375/175590 objects degraded (6.478%) 2024-02-09T17:58:18.412 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:18 smithi179 ceph-mon[43975]: 1.f scrub ok 2024-02-09T17:58:18.412 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:18 smithi179 ceph-mon[43975]: 8.e scrub starts 2024-02-09T17:58:18.412 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:18 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1985207871' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:58:18.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:18 smithi099 ceph-mon[39420]: 1.f scrub starts 2024-02-09T17:58:18.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:18 smithi099 ceph-mon[39420]: pgmap v2088: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 3.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 670 op/s; 11375/175590 objects degraded (6.478%) 2024-02-09T17:58:18.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:18 smithi099 ceph-mon[39420]: 1.f scrub ok 2024-02-09T17:58:18.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:18 smithi099 ceph-mon[39420]: 8.e scrub starts 2024-02-09T17:58:18.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:18 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1985207871' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:58:18.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:18 smithi162 ceph-mon[44304]: 1.f scrub starts 2024-02-09T17:58:18.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:18 smithi162 ceph-mon[44304]: pgmap v2088: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 3.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 670 op/s; 11375/175590 objects degraded (6.478%) 2024-02-09T17:58:18.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:18 smithi162 ceph-mon[44304]: 1.f scrub ok 2024-02-09T17:58:18.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:18 smithi162 ceph-mon[44304]: 8.e scrub starts 2024-02-09T17:58:18.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:18 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1985207871' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:58:19.129 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-09T17:58:19.130 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-09T17:58:19.205 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-09T17:58:19.378 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:19 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:58:19.378 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:19 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:58:19.378 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:19 smithi099 ceph-mon[39420]: 8.e scrub ok 2024-02-09T17:58:19.378 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:19 smithi099 ceph-mon[39420]: osdmap e843: 12 total, 11 up, 12 in 2024-02-09T17:58:19.378 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:19 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:58:19.511 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:19 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:58:19.511 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:19 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:58:19.512 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:19 smithi162 ceph-mon[44304]: 8.e scrub ok 2024-02-09T17:58:19.512 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:19 smithi162 ceph-mon[44304]: osdmap e843: 12 total, 11 up, 12 in 2024-02-09T17:58:19.512 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:19 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:58:19.584 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:58:19.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:19 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:58:19.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:19 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:58:19.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:19 smithi179 ceph-mon[43975]: 8.e scrub ok 2024-02-09T17:58:19.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:19 smithi179 ceph-mon[43975]: osdmap e843: 12 total, 11 up, 12 in 2024-02-09T17:58:19.677 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:19 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:58:19.992 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:58:20.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:20.002 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:58:20.102 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:58:20.135 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:20.179 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:58:20.179 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:58:20.481 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:20 smithi162 ceph-mon[44304]: 1.d scrub starts 2024-02-09T17:58:20.481 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:20 smithi162 ceph-mon[44304]: 1.d scrub ok 2024-02-09T17:58:20.481 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:20 smithi162 ceph-mon[44304]: pgmap v2090: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 3.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 806 op/s; 10662/168333 objects degraded (6.334%) 2024-02-09T17:58:20.481 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:20 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:58:20.482 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:20 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:58:20.482 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:20 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:58:20.482 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:20 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:58:20.482 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:20 smithi162 ceph-mon[44304]: 8.f scrub starts 2024-02-09T17:58:20.482 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:20 smithi162 ceph-mon[44304]: 8.a scrub starts 2024-02-09T17:58:20.483 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:20 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:58:20.483 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:20 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:58:20.516 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:58:20.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:20.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:58:20.620 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:58:20.708 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:20.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:20 smithi099 ceph-mon[39420]: 1.d scrub starts 2024-02-09T17:58:20.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:20 smithi099 ceph-mon[39420]: 1.d scrub ok 2024-02-09T17:58:20.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:20 smithi099 ceph-mon[39420]: pgmap v2090: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 3.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 806 op/s; 10662/168333 objects degraded (6.334%) 2024-02-09T17:58:20.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:20 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:58:20.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:20 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:58:20.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:20 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:58:20.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:20 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:58:20.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:20 smithi099 ceph-mon[39420]: 8.f scrub starts 2024-02-09T17:58:20.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:20 smithi099 ceph-mon[39420]: 8.a scrub starts 2024-02-09T17:58:20.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:20 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:58:20.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:20 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:58:20.927 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:20 smithi179 ceph-mon[43975]: 1.d scrub starts 2024-02-09T17:58:20.927 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:20 smithi179 ceph-mon[43975]: 1.d scrub ok 2024-02-09T17:58:20.927 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:20 smithi179 ceph-mon[43975]: pgmap v2090: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 3.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 806 op/s; 10662/168333 objects degraded (6.334%) 2024-02-09T17:58:20.927 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:20 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:58:20.927 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:20 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:58:20.927 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:20 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:58:20.927 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:20 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:58:20.927 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:20 smithi179 ceph-mon[43975]: 8.f scrub starts 2024-02-09T17:58:20.927 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:20 smithi179 ceph-mon[43975]: 8.a scrub starts 2024-02-09T17:58:20.927 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:20 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:58:20.928 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:20 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:58:20.945 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-09T17:58:21.131 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:58:21.261 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:58:21.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:21 smithi162 ceph-mon[44304]: 1.10 scrub starts 2024-02-09T17:58:21.798 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:21 smithi162 ceph-mon[44304]: 1.10 scrub ok 2024-02-09T17:58:21.798 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:21 smithi162 ceph-mon[44304]: 1.14 scrub starts 2024-02-09T17:58:21.798 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:21 smithi162 ceph-mon[44304]: 1.14 scrub ok 2024-02-09T17:58:21.798 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:21 smithi162 ceph-mon[44304]: 1.4 scrub starts 2024-02-09T17:58:21.798 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:21 smithi162 ceph-mon[44304]: 1.4 scrub ok 2024-02-09T17:58:21.798 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:21 smithi162 ceph-mon[44304]: 8.2 scrub starts 2024-02-09T17:58:21.798 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:21 smithi162 ceph-mon[44304]: 8.a scrub ok 2024-02-09T17:58:21.798 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:21 smithi162 ceph-mon[44304]: 8.7 scrub starts 2024-02-09T17:58:21.798 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:21 smithi162 ceph-mon[44304]: 8.f scrub ok 2024-02-09T17:58:21.798 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:58:21.798 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:58:21.798 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:21 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:58:21.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:21 smithi179 ceph-mon[43975]: 1.10 scrub starts 2024-02-09T17:58:21.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:21 smithi179 ceph-mon[43975]: 1.10 scrub ok 2024-02-09T17:58:21.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:21 smithi179 ceph-mon[43975]: 1.14 scrub starts 2024-02-09T17:58:21.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:21 smithi179 ceph-mon[43975]: 1.14 scrub ok 2024-02-09T17:58:21.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:21 smithi179 ceph-mon[43975]: 1.4 scrub starts 2024-02-09T17:58:21.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:21 smithi179 ceph-mon[43975]: 1.4 scrub ok 2024-02-09T17:58:21.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:21 smithi179 ceph-mon[43975]: 8.2 scrub starts 2024-02-09T17:58:21.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:21 smithi179 ceph-mon[43975]: 8.a scrub ok 2024-02-09T17:58:21.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:21 smithi179 ceph-mon[43975]: 8.7 scrub starts 2024-02-09T17:58:21.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:21 smithi179 ceph-mon[43975]: 8.f scrub ok 2024-02-09T17:58:21.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:21 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:58:21.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:21 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:58:21.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:21 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:58:21.927 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:58:21.999 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:21 smithi099 ceph-mon[39420]: 1.10 scrub starts 2024-02-09T17:58:21.999 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:21 smithi099 ceph-mon[39420]: 1.10 scrub ok 2024-02-09T17:58:21.999 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:21 smithi099 ceph-mon[39420]: 1.14 scrub starts 2024-02-09T17:58:21.999 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:21 smithi099 ceph-mon[39420]: 1.14 scrub ok 2024-02-09T17:58:21.999 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:21 smithi099 ceph-mon[39420]: 1.4 scrub starts 2024-02-09T17:58:21.999 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:21 smithi099 ceph-mon[39420]: 1.4 scrub ok 2024-02-09T17:58:21.999 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:21 smithi099 ceph-mon[39420]: 8.2 scrub starts 2024-02-09T17:58:21.999 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:21 smithi099 ceph-mon[39420]: 8.a scrub ok 2024-02-09T17:58:21.999 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:21 smithi099 ceph-mon[39420]: 8.7 scrub starts 2024-02-09T17:58:22.000 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:21 smithi099 ceph-mon[39420]: 8.f scrub ok 2024-02-09T17:58:22.000 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:21 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:58:22.000 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:21 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:58:22.000 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:21 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:58:22.539 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is unset 2024-02-09T17:58:22.636 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:22 smithi162 ceph-mon[44304]: 1.0 scrub starts 2024-02-09T17:58:22.636 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:22 smithi162 ceph-mon[44304]: 1.0 scrub ok 2024-02-09T17:58:22.636 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:22 smithi162 ceph-mon[44304]: 1.a scrub starts 2024-02-09T17:58:22.636 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:22 smithi162 ceph-mon[44304]: 1.a scrub ok 2024-02-09T17:58:22.636 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:22 smithi162 ceph-mon[44304]: 1.17 scrub starts 2024-02-09T17:58:22.636 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:22 smithi162 ceph-mon[44304]: 8.7 scrub ok 2024-02-09T17:58:22.636 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:22 smithi162 ceph-mon[44304]: 8.2 scrub ok 2024-02-09T17:58:22.636 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:22 smithi162 ceph-mon[44304]: 8.5 scrub starts 2024-02-09T17:58:22.636 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:22 smithi162 ceph-mon[44304]: 1.1d scrub starts 2024-02-09T17:58:22.636 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:22 smithi162 ceph-mon[44304]: 1.17 scrub ok 2024-02-09T17:58:22.636 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:22 smithi162 ceph-mon[44304]: 1.1d scrub ok 2024-02-09T17:58:22.636 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:22 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 10662/166881 objects degraded (6.389%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:58:22.637 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:22 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/983203606' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:58:22.637 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:22 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:58:22.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:22 smithi099 ceph-mon[39420]: 1.0 scrub starts 2024-02-09T17:58:22.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:22 smithi099 ceph-mon[39420]: 1.0 scrub ok 2024-02-09T17:58:22.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:22 smithi099 ceph-mon[39420]: 1.a scrub starts 2024-02-09T17:58:22.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:22 smithi099 ceph-mon[39420]: 1.a scrub ok 2024-02-09T17:58:22.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:22 smithi099 ceph-mon[39420]: 1.17 scrub starts 2024-02-09T17:58:22.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:22 smithi099 ceph-mon[39420]: 8.7 scrub ok 2024-02-09T17:58:22.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:22 smithi099 ceph-mon[39420]: 8.2 scrub ok 2024-02-09T17:58:22.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:22 smithi099 ceph-mon[39420]: 8.5 scrub starts 2024-02-09T17:58:22.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:22 smithi099 ceph-mon[39420]: 1.1d scrub starts 2024-02-09T17:58:22.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:22 smithi099 ceph-mon[39420]: 1.17 scrub ok 2024-02-09T17:58:22.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:22 smithi099 ceph-mon[39420]: 1.1d scrub ok 2024-02-09T17:58:22.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:22 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 10662/166881 objects degraded (6.389%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:58:22.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:22 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/983203606' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:58:22.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:22 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:58:22.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:22 smithi179 ceph-mon[43975]: 1.0 scrub starts 2024-02-09T17:58:22.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:22 smithi179 ceph-mon[43975]: 1.0 scrub ok 2024-02-09T17:58:22.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:22 smithi179 ceph-mon[43975]: 1.a scrub starts 2024-02-09T17:58:22.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:22 smithi179 ceph-mon[43975]: 1.a scrub ok 2024-02-09T17:58:22.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:22 smithi179 ceph-mon[43975]: 1.17 scrub starts 2024-02-09T17:58:22.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:22 smithi179 ceph-mon[43975]: 8.7 scrub ok 2024-02-09T17:58:22.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:22 smithi179 ceph-mon[43975]: 8.2 scrub ok 2024-02-09T17:58:22.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:22 smithi179 ceph-mon[43975]: 8.5 scrub starts 2024-02-09T17:58:22.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:22 smithi179 ceph-mon[43975]: 1.1d scrub starts 2024-02-09T17:58:22.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:22 smithi179 ceph-mon[43975]: 1.17 scrub ok 2024-02-09T17:58:22.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:22 smithi179 ceph-mon[43975]: 1.1d scrub ok 2024-02-09T17:58:22.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:22 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 10662/166881 objects degraded (6.389%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:58:22.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:22 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/983203606' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:58:22.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:22 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:58:23.253 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:58:23.253 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:58:23.627 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:23 smithi162 ceph-mon[44304]: 1.18 scrub starts 2024-02-09T17:58:23.627 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:23 smithi162 ceph-mon[44304]: 1.18 scrub ok 2024-02-09T17:58:23.627 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:23 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:58:23.627 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:23 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:58:23.627 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:23 smithi162 ceph-mon[44304]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:58:23.627 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:23 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:58:23.627 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:23 smithi162 ceph-mon[44304]: osdmap e844: 12 total, 11 up, 12 in 2024-02-09T17:58:23.627 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:23 smithi162 ceph-mon[44304]: 8.5 scrub ok 2024-02-09T17:58:23.627 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:23 smithi162 ceph-mon[44304]: 1.1 scrub starts 2024-02-09T17:58:23.628 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:23 smithi162 ceph-mon[44304]: 1.1 scrub ok 2024-02-09T17:58:23.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:23 smithi179 ceph-mon[43975]: 1.18 scrub starts 2024-02-09T17:58:23.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:23 smithi179 ceph-mon[43975]: 1.18 scrub ok 2024-02-09T17:58:23.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:23 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:58:23.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:23 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:58:23.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:23 smithi179 ceph-mon[43975]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:58:23.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:23 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:58:23.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:23 smithi179 ceph-mon[43975]: osdmap e844: 12 total, 11 up, 12 in 2024-02-09T17:58:23.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:23 smithi179 ceph-mon[43975]: 8.5 scrub ok 2024-02-09T17:58:23.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:23 smithi179 ceph-mon[43975]: 1.1 scrub starts 2024-02-09T17:58:23.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:23 smithi179 ceph-mon[43975]: 1.1 scrub ok 2024-02-09T17:58:23.984 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-09T17:58:23.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:23 smithi099 ceph-mon[39420]: 1.18 scrub starts 2024-02-09T17:58:23.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:23 smithi099 ceph-mon[39420]: 1.18 scrub ok 2024-02-09T17:58:23.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:23 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:58:23.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:23 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:58:23.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:23 smithi099 ceph-mon[39420]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:58:23.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:23 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:58:23.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:23 smithi099 ceph-mon[39420]: osdmap e844: 12 total, 11 up, 12 in 2024-02-09T17:58:23.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:23 smithi099 ceph-mon[39420]: 8.5 scrub ok 2024-02-09T17:58:23.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:23 smithi099 ceph-mon[39420]: 1.1 scrub starts 2024-02-09T17:58:23.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:23 smithi099 ceph-mon[39420]: 1.1 scrub ok 2024-02-09T17:58:24.286 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:58:24.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:24.301 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:58:24.312 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:58:24.402 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:58:24.466 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:24.504 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:58:24.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:24.508 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:58:24.609 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:58:24.642 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:24 smithi099 ceph-mon[39420]: 8.4 scrub starts 2024-02-09T17:58:24.642 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:24 smithi099 ceph-mon[39420]: pgmap v2093: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 3.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 881 op/s; 9994/160302 objects degraded (6.234%) 2024-02-09T17:58:24.642 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:24 smithi099 ceph-mon[39420]: 1.7 scrub starts 2024-02-09T17:58:24.642 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:24 smithi099 ceph-mon[39420]: 8.6 deep-scrub starts 2024-02-09T17:58:24.642 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:24 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:58:24.642 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:24 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:58:24.642 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:24 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:58:24.642 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:24 smithi099 ceph-mon[39420]: 1.7 scrub ok 2024-02-09T17:58:24.642 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:24 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:58:24.642 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:24 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:58:24.643 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:24 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:58:24.643 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:24 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:58:24.646 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:24.672 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:58:24.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:24.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:58:24.676 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:58:24.701 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:58:24.726 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:58:24.748 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:58:24.772 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:58:24.776 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:58:24.800 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:58:24.826 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:58:24.847 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:24 smithi179 ceph-mon[43975]: 8.4 scrub starts 2024-02-09T17:58:24.847 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:24 smithi179 ceph-mon[43975]: pgmap v2093: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 3.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 881 op/s; 9994/160302 objects degraded (6.234%) 2024-02-09T17:58:24.847 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:24 smithi179 ceph-mon[43975]: 1.7 scrub starts 2024-02-09T17:58:24.847 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:24 smithi179 ceph-mon[43975]: 8.6 deep-scrub starts 2024-02-09T17:58:24.847 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:24 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:58:24.847 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:24 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:58:24.847 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:24 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:58:24.847 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:24 smithi179 ceph-mon[43975]: 1.7 scrub ok 2024-02-09T17:58:24.848 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:24 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:58:24.848 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:24 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:58:24.848 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:24 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:58:24.848 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:24 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:58:24.849 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:24.865 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:58:24.869 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:24 smithi162 ceph-mon[44304]: 8.4 scrub starts 2024-02-09T17:58:24.869 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:24 smithi162 ceph-mon[44304]: pgmap v2093: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 3.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 881 op/s; 9994/160302 objects degraded (6.234%) 2024-02-09T17:58:24.869 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:24 smithi162 ceph-mon[44304]: 1.7 scrub starts 2024-02-09T17:58:24.869 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:24 smithi162 ceph-mon[44304]: 8.6 deep-scrub starts 2024-02-09T17:58:24.869 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:24 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:58:24.869 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:24 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:58:24.869 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:24 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:58:24.870 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:24 smithi162 ceph-mon[44304]: 1.7 scrub ok 2024-02-09T17:58:24.870 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:24 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:58:24.870 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:24 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:58:24.870 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:24 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:58:24.870 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:24 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:58:24.888 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:58:24.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:24.893 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:58:24.898 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:58:24.934 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:58:24.961 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:58:24.961 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:58:24.961 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:58:24.961 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:58:24.961 INFO:teuthology.orchestra.run.smithi099.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-09T17:58:24.961 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:58:24.962 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:58:24.962 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:58:24.962 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:58:24.962 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:58:24.962 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:58:24.962 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:58:24.962 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:58:24.994 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:58:25.024 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:25.126 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:58:25.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:25.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:58:25.230 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:58:25.258 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:25.388 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:58:25.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:25.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:58:25.493 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:58:25.523 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:25.530 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:25 smithi099 ceph-mon[39420]: 8.4 scrub ok 2024-02-09T17:58:25.530 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:25 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/154792394' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:58:25.568 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set noscrub 2024-02-09T17:58:25.650 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:58:25.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:25.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:58:25.757 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:58:25.790 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:25 smithi099 ceph-mon[39420]: 1.11 scrub starts 2024-02-09T17:58:25.791 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:25 smithi099 ceph-mon[39420]: 1.11 scrub ok 2024-02-09T17:58:25.821 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:25 smithi179 ceph-mon[43975]: 8.4 scrub ok 2024-02-09T17:58:25.821 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:25 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/154792394' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:58:25.821 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:25 smithi179 ceph-mon[43975]: 1.11 scrub starts 2024-02-09T17:58:25.821 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:25 smithi179 ceph-mon[43975]: 1.11 scrub ok 2024-02-09T17:58:25.824 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:25.833 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-09T17:58:25.833 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-09T17:58:25.861 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:58:25.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:25.864 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:58:25.925 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:58:25.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:25 smithi162 ceph-mon[44304]: 8.4 scrub ok 2024-02-09T17:58:25.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:25 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/154792394' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:58:25.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:25 smithi162 ceph-mon[44304]: 1.11 scrub starts 2024-02-09T17:58:25.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:25 smithi162 ceph-mon[44304]: 1.11 scrub ok 2024-02-09T17:58:25.965 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:58:26.001 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:26.024 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:58:26.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:26.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:58:26.129 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:58:26.164 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:26.268 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:58:26.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:26.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:58:26.302 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:58:26.302 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-09T17:58:26.303 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-02-09T17:58:26.303 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:58:26.377 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:58:26.460 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:26.486 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:58:26.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:26.489 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:58:26.589 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:58:26.670 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:26.698 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:58:26.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:26.704 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:58:26.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:26 smithi099 ceph-mon[39420]: pgmap v2094: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 3.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 795 op/s; 9994/156477 objects degraded (6.387%) 2024-02-09T17:58:26.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:26 smithi099 ceph-mon[39420]: 8.1 deep-scrub starts 2024-02-09T17:58:26.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:26 smithi099 ceph-mon[39420]: 1.b scrub starts 2024-02-09T17:58:26.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:26 smithi099 ceph-mon[39420]: 1.b scrub ok 2024-02-09T17:58:26.804 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:58:26.847 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:26.888 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:58:26.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:26.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:58:26.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:26 smithi162 ceph-mon[44304]: pgmap v2094: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 3.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 795 op/s; 9994/156477 objects degraded (6.387%) 2024-02-09T17:58:26.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:26 smithi162 ceph-mon[44304]: 8.1 deep-scrub starts 2024-02-09T17:58:26.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:26 smithi162 ceph-mon[44304]: 1.b scrub starts 2024-02-09T17:58:26.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:26 smithi162 ceph-mon[44304]: 1.b scrub ok 2024-02-09T17:58:26.931 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:26 smithi179 ceph-mon[43975]: pgmap v2094: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 3.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 795 op/s; 9994/156477 objects degraded (6.387%) 2024-02-09T17:58:26.931 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:26 smithi179 ceph-mon[43975]: 8.1 deep-scrub starts 2024-02-09T17:58:26.931 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:26 smithi179 ceph-mon[43975]: 1.b scrub starts 2024-02-09T17:58:26.931 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:26 smithi179 ceph-mon[43975]: 1.b scrub ok 2024-02-09T17:58:26.995 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:58:27.025 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:27.332 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-09T17:58:27.376 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:58:27.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:27.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:58:27.480 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:58:27.526 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:27.557 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:58:27.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:27.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:58:27.665 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:58:27.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:27 smithi162 ceph-mon[44304]: 1.1b scrub starts 2024-02-09T17:58:27.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:27 smithi162 ceph-mon[44304]: 1.9 scrub starts 2024-02-09T17:58:27.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:27 smithi162 ceph-mon[44304]: 1.1b scrub ok 2024-02-09T17:58:27.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:27 smithi162 ceph-mon[44304]: 1.5 scrub starts 2024-02-09T17:58:27.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:27 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 9994/156477 objects degraded (6.387%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:58:27.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:27 smithi162 ceph-mon[44304]: 1.9 scrub ok 2024-02-09T17:58:27.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:27 smithi162 ceph-mon[44304]: 1.5 scrub ok 2024-02-09T17:58:27.697 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:27.698 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:58:27.725 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:58:27.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:27.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:58:27.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:27 smithi099 ceph-mon[39420]: 1.1b scrub starts 2024-02-09T17:58:27.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:27 smithi099 ceph-mon[39420]: 1.9 scrub starts 2024-02-09T17:58:27.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:27 smithi099 ceph-mon[39420]: 1.1b scrub ok 2024-02-09T17:58:27.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:27 smithi099 ceph-mon[39420]: 1.5 scrub starts 2024-02-09T17:58:27.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:27 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 9994/156477 objects degraded (6.387%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:58:27.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:27 smithi099 ceph-mon[39420]: 1.9 scrub ok 2024-02-09T17:58:27.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:27 smithi099 ceph-mon[39420]: 1.5 scrub ok 2024-02-09T17:58:27.828 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:58:27.834 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:58:27.873 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:27.880 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:27 smithi179 ceph-mon[43975]: 1.1b scrub starts 2024-02-09T17:58:27.881 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:27 smithi179 ceph-mon[43975]: 1.9 scrub starts 2024-02-09T17:58:27.881 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:27 smithi179 ceph-mon[43975]: 1.1b scrub ok 2024-02-09T17:58:27.881 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:27 smithi179 ceph-mon[43975]: 1.5 scrub starts 2024-02-09T17:58:27.881 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:27 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 9994/156477 objects degraded (6.387%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:58:27.881 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:27 smithi179 ceph-mon[43975]: 1.9 scrub ok 2024-02-09T17:58:27.881 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:27 smithi179 ceph-mon[43975]: 1.5 scrub ok 2024-02-09T17:58:27.986 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:58:28.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:28.031 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:58:28.132 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:58:28.208 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:28.250 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:58:28.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:28.255 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:58:28.266 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:58:28.356 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:58:28.386 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:28.538 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:58:28.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:28.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:58:28.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:28 smithi179 ceph-mon[43975]: pgmap v2095: 47 pgs: 1 active+clean+scrubbing+deep, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 3.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 649 op/s; 9994/153981 objects degraded (6.490%) 2024-02-09T17:58:28.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:28 smithi179 ceph-mon[43975]: 1.19 scrub starts 2024-02-09T17:58:28.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:28 smithi179 ceph-mon[43975]: 1.19 scrub ok 2024-02-09T17:58:28.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:28 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/413202935' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:58:28.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:28 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:58:28.642 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:58:28.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:28 smithi162 ceph-mon[44304]: pgmap v2095: 47 pgs: 1 active+clean+scrubbing+deep, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 3.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 649 op/s; 9994/153981 objects degraded (6.490%) 2024-02-09T17:58:28.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:28 smithi162 ceph-mon[44304]: 1.19 scrub starts 2024-02-09T17:58:28.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:28 smithi162 ceph-mon[44304]: 1.19 scrub ok 2024-02-09T17:58:28.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:28 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/413202935' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:58:28.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:28 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:58:28.678 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:28.803 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:58:28.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:28.810 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:28 smithi099 ceph-mon[39420]: pgmap v2095: 47 pgs: 1 active+clean+scrubbing+deep, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 3.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 649 op/s; 9994/153981 objects degraded (6.490%) 2024-02-09T17:58:28.810 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:28 smithi099 ceph-mon[39420]: 1.19 scrub starts 2024-02-09T17:58:28.810 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:28 smithi099 ceph-mon[39420]: 1.19 scrub ok 2024-02-09T17:58:28.810 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:28 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/413202935' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:58:28.811 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:28 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:58:28.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:58:28.912 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:58:28.979 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:29.008 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is set 2024-02-09T17:58:29.036 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:58:29.039 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:29.040 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:58:29.141 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:58:29.178 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:29.299 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 2] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2, 4, 5, 6, 7, 9, 10, 11, 8] 2024-02-09T17:58:29.299 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:58:29.300 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph pg dump --format=json 2024-02-09T17:58:29.334 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:58:29.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:29.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:58:29.444 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:58:29.486 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:29.510 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:58:29.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:29.517 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:58:29.617 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:58:29.648 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:29.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:29 smithi162 ceph-mon[44304]: 8.3 scrub starts 2024-02-09T17:58:29.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:29 smithi162 ceph-mon[44304]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:58:29.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:29 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:58:29.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:29 smithi162 ceph-mon[44304]: osdmap e845: 12 total, 11 up, 12 in 2024-02-09T17:58:29.737 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:58:29.737 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-09T17:58:29.737 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-02-09T17:58:29.737 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-02-09T17:58:29.737 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:58:29.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:29 smithi099 ceph-mon[39420]: 8.3 scrub starts 2024-02-09T17:58:29.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:29 smithi099 ceph-mon[39420]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:58:29.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:29 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:58:29.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:29 smithi099 ceph-mon[39420]: osdmap e845: 12 total, 11 up, 12 in 2024-02-09T17:58:29.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:29 smithi179 ceph-mon[43975]: 8.3 scrub starts 2024-02-09T17:58:29.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:29 smithi179 ceph-mon[43975]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:58:29.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:29 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:58:29.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:29 smithi179 ceph-mon[43975]: osdmap e845: 12 total, 11 up, 12 in 2024-02-09T17:58:30.405 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:58:30.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:30.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:58:30.432 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:58:30.491 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-09T17:58:30.517 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:58:30.583 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:30.613 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:58:30.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:30.623 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:58:30.720 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:30 smithi099 ceph-mon[39420]: pgmap v2097: 47 pgs: 1 active+clean+scrubbing+deep, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 760 op/s; 9436/148623 objects degraded (6.349%) 2024-02-09T17:58:30.724 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:58:30.756 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:30 smithi162 ceph-mon[44304]: pgmap v2097: 47 pgs: 1 active+clean+scrubbing+deep, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 760 op/s; 9436/148623 objects degraded (6.349%) 2024-02-09T17:58:30.756 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:30.782 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:58:30.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:30.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:58:30.803 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:58:30.889 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:58:30.917 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:30.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:30 smithi179 ceph-mon[43975]: pgmap v2097: 47 pgs: 1 active+clean+scrubbing+deep, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 760 op/s; 9436/148623 objects degraded (6.349%) 2024-02-09T17:58:31.057 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:58:31.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:31.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:58:31.163 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:58:31.214 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:31.279 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:58:31.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:31.285 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:58:31.386 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:58:31.469 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:31.539 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:58:31.542 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:31.543 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:58:31.586 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:31 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:58:31.587 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:31 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:58:31.643 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:58:31.695 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:31.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:31 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:58:31.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:31 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:58:31.950 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-09T17:58:31.987 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:31 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:58:31.987 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:31 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:58:32.221 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:58:32.239 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:58:32.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:32.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:58:32.259 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:58:32.283 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:58:32.312 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:58:32.342 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:58:32.343 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:58:32.368 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:58:32.372 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:32.400 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:58:32.432 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:58:32.469 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:58:32.519 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:58:32.552 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:58:32.552 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:58:32.552 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:58:32.552 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:58:32.552 INFO:teuthology.orchestra.run.smithi099.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-09T17:58:32.552 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:58:32.552 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:58:32.552 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:58:32.552 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:58:32.553 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:58:32.553 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:58:32.553 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:58:32.553 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:58:32.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:32 smithi162 ceph-mon[44304]: osdmap e846: 12 total, 11 up, 12 in 2024-02-09T17:58:32.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:32 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:58:32.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:32 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:58:32.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:32 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:58:32.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:32 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:58:32.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:32 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:58:32.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:32 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:58:32.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:32 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:58:32.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:32 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 9436/147624 objects degraded (6.392%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:58:32.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:32 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/300088688' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:58:32.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:32 smithi179 ceph-mon[43975]: osdmap e846: 12 total, 11 up, 12 in 2024-02-09T17:58:32.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:32 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:58:32.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:32 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:58:32.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:32 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:58:32.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:32 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:58:32.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:32 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:58:32.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:32 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:58:32.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:32 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:58:32.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:32 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 9436/147624 objects degraded (6.392%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:58:32.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:32 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/300088688' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:58:32.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:32 smithi099 ceph-mon[39420]: osdmap e846: 12 total, 11 up, 12 in 2024-02-09T17:58:32.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:32 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:58:32.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:32 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:58:32.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:32 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:58:32.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:32 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:58:32.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:32 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:58:32.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:32 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:58:32.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:32 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:58:32.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:32 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 9436/147624 objects degraded (6.392%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:58:32.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:32 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/300088688' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:58:33.102 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T17:58:33.104 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-02-09T17:58:33.227 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:58:33.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:33.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:58:33.332 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:58:33.368 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:33.609 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-09T17:58:33.610 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-09T17:58:33.820 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:58:33.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:33.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:58:33.832 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:58:33.928 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:58:33.963 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:34.093 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:58:34.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:34.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:58:34.101 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:58:34.101 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:58:34.198 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:58:34.238 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:34.348 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:58:34.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:34.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:58:34.366 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":2099,"stamp":"2024-02-09T17:58:31.582814+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3225190968,"num_objects":49208,"num_object_clones":0,"num_object_copies":147624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49208,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":196933,"num_write_kb":7876417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":165114,"ondisk_log_size":165114,"up":133,"acting":133,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":19305192,"kb_used_data":10106484,"kb_used_omap":229,"kb_used_meta":9198170,"kb_avail":1105636632,"statfs":{"total":1151940427776,"available":1132171911168,"internally_reserved":0,"allocated":10349039616,"data_stored":10345704754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235127,"internal_metadata":9418926473},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,7,2,8,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-276955136,"num_objects":-4226,"num_object_clones":0,"num_object_copies":-12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-558,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4226,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.350749"},"pg_stats":[{"pgid":"1.1e","version":"63'139","reported_seq":986,"reported_epoch":844,"state":"active+undersized","last_fresh":"2024-02-09T17:58:22.507059+0000","last_change":"2024-02-09T17:56:40.457389+0000","last_active":"2024-02-09T17:58:22.507059+0000","last_peered":"2024-02-09T17:58:22.507059+0000","last_clean":"2024-02-09T17:56:36.455001+0000","last_became_active":"2024-02-09T17:56:40.457389+0000","last_became_peered":"2024-02-09T17:56:40.457389+0000","last_unstale":"2024-02-09T17:58:22.507059+0000","last_undegraded":"2024-02-09T17:58:22.507059+0000","last_fullsized":"2024-02-09T17:56:40.455311+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":782,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:17.113960+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:55:17.113960+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:58:24.673215+0000","scrub_duration":125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":["11","7"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"63'139","reported_seq":1014,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-02-09T17:58:29.011264+0000","last_change":"2024-02-09T17:58:21.932294+0000","last_active":"2024-02-09T17:58:29.011264+0000","last_peered":"2024-02-09T17:58:29.011264+0000","last_clean":"2024-02-09T17:58:29.011264+0000","last_became_active":"2024-02-09T17:43:28.594296+0000","last_became_peered":"2024-02-09T17:43:28.594296+0000","last_unstale":"2024-02-09T17:58:29.011264+0000","last_undegraded":"2024-02-09T17:58:29.011264+0000","last_fullsized":"2024-02-09T17:58:29.011264+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:58:21.932171+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:48:24.634092+0000","last_clean_scrub_stamp":"2024-02-09T17:58:21.932171+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:59:34.897886+0000","scrub_duration":125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"63'139","reported_seq":1524,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-02-09T17:58:22.520407+0000","last_change":"2024-02-09T17:57:24.725091+0000","last_active":"2024-02-09T17:58:22.520407+0000","last_peered":"2024-02-09T17:58:22.520407+0000","last_clean":"2024-02-09T17:58:22.520407+0000","last_became_active":"2024-02-09T17:43:19.103584+0000","last_became_peered":"2024-02-09T17:43:19.103584+0000","last_unstale":"2024-02-09T17:58:22.520407+0000","last_undegraded":"2024-02-09T17:58:22.520407+0000","last_fullsized":"2024-02-09T17:58:22.520407+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:57:24.724984+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:23.912504+0000","last_clean_scrub_stamp":"2024-02-09T17:57:24.724984+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:58:34.802203+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"63'139","reported_seq":1330,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-02-09T17:58:26.907363+0000","last_change":"2024-02-09T17:58:26.907405+0000","last_active":"2024-02-09T17:58:26.907363+0000","last_peered":"2024-02-09T17:58:26.907363+0000","last_clean":"2024-02-09T17:58:26.907363+0000","last_became_active":"2024-02-09T17:43:19.102729+0000","last_became_peered":"2024-02-09T17:43:19.102729+0000","last_unstale":"2024-02-09T17:58:26.907363+0000","last_undegraded":"2024-02-09T17:58:26.907363+0000","last_fullsized":"2024-02-09T17:58:26.907363+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:58:26.907265+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:45:44.988633+0000","last_clean_scrub_stamp":"2024-02-09T17:58:26.907265+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:59:45.374015+0000","scrub_duration":166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"63'139","reported_seq":987,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-02-09T17:58:29.024986+0000","last_change":"2024-02-09T17:57:42.734318+0000","last_active":"2024-02-09T17:58:29.024986+0000","last_peered":"2024-02-09T17:58:29.024986+0000","last_clean":"2024-02-09T17:58:29.024986+0000","last_became_active":"2024-02-09T17:43:19.120814+0000","last_became_peered":"2024-02-09T17:43:19.120814+0000","last_unstale":"2024-02-09T17:58:29.024986+0000","last_undegraded":"2024-02-09T17:58:29.024986+0000","last_fullsized":"2024-02-09T17:58:29.024986+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:57:42.734241+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:57:42.734241+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:59:02.094445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"63'139","reported_seq":1839,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-02-09T17:58:28.079645+0000","last_change":"2024-02-09T17:58:28.079682+0000","last_active":"2024-02-09T17:58:28.079645+0000","last_peered":"2024-02-09T17:58:28.079645+0000","last_clean":"2024-02-09T17:58:28.079645+0000","last_became_active":"2024-02-09T17:55:21.231291+0000","last_became_peered":"2024-02-09T17:55:21.231291+0000","last_unstale":"2024-02-09T17:58:28.079645+0000","last_undegraded":"2024-02-09T17:58:28.079645+0000","last_fullsized":"2024-02-09T17:58:28.079645+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":791,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:58:28.079578+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:30.193194+0000","last_clean_scrub_stamp":"2024-02-09T17:58:28.079578+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:59:34.458003+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"63'139","reported_seq":1678,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-02-09T17:58:29.021021+0000","last_change":"2024-02-09T17:58:21.789074+0000","last_active":"2024-02-09T17:58:29.021021+0000","last_peered":"2024-02-09T17:58:29.021021+0000","last_clean":"2024-02-09T17:58:29.021021+0000","last_became_active":"2024-02-09T17:43:28.585573+0000","last_became_peered":"2024-02-09T17:43:28.585573+0000","last_unstale":"2024-02-09T17:58:29.021021+0000","last_undegraded":"2024-02-09T17:58:29.021021+0000","last_fullsized":"2024-02-09T17:58:29.021021+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:58:21.788945+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:58:21.788945+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:59:22.772853+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"63'139","reported_seq":1122,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-02-09T17:58:22.515577+0000","last_change":"2024-02-09T17:58:21.915292+0000","last_active":"2024-02-09T17:58:22.515577+0000","last_peered":"2024-02-09T17:58:22.515577+0000","last_clean":"2024-02-09T17:58:22.515577+0000","last_became_active":"2024-02-09T17:48:46.184419+0000","last_became_peered":"2024-02-09T17:48:46.184419+0000","last_unstale":"2024-02-09T17:58:22.515577+0000","last_undegraded":"2024-02-09T17:58:22.515577+0000","last_fullsized":"2024-02-09T17:58:22.515577+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:58:21.915175+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:44:50.214686+0000","last_clean_scrub_stamp":"2024-02-09T17:58:21.915175+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:59:35.111009+0000","scrub_duration":208,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"63'139","reported_seq":1002,"reported_epoch":845,"state":"active+undersized","last_fresh":"2024-02-09T17:58:29.011209+0000","last_change":"2024-02-09T17:56:40.452951+0000","last_active":"2024-02-09T17:58:29.011209+0000","last_peered":"2024-02-09T17:58:29.011209+0000","last_clean":"2024-02-09T17:56:36.452114+0000","last_became_active":"2024-02-09T17:56:40.452951+0000","last_became_peered":"2024-02-09T17:56:40.452951+0000","last_unstale":"2024-02-09T17:58:29.011209+0000","last_undegraded":"2024-02-09T17:58:29.011209+0000","last_fullsized":"2024-02-09T17:56:40.451051+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:54:56.134826+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:11.970338+0000","last_clean_scrub_stamp":"2024-02-09T17:54:56.134826+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:58:33.677995+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"63'139","reported_seq":1428,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-02-09T17:58:29.019620+0000","last_change":"2024-02-09T17:57:22.528295+0000","last_active":"2024-02-09T17:58:29.019620+0000","last_peered":"2024-02-09T17:58:29.019620+0000","last_clean":"2024-02-09T17:58:29.019620+0000","last_became_active":"2024-02-09T17:43:19.138398+0000","last_became_peered":"2024-02-09T17:43:19.138398+0000","last_unstale":"2024-02-09T17:58:29.019620+0000","last_undegraded":"2024-02-09T17:58:29.019620+0000","last_fullsized":"2024-02-09T17:58:29.019620+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:57:22.528147+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:57:22.528147+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:58:51.521146+0000","scrub_duration":169,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"63'139","reported_seq":1506,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-02-09T17:58:29.003155+0000","last_change":"2024-02-09T17:58:20.599010+0000","last_active":"2024-02-09T17:58:29.003155+0000","last_peered":"2024-02-09T17:58:29.003155+0000","last_clean":"2024-02-09T17:58:29.003155+0000","last_became_active":"2024-02-09T17:43:28.589307+0000","last_became_peered":"2024-02-09T17:43:28.589307+0000","last_unstale":"2024-02-09T17:58:29.003155+0000","last_undegraded":"2024-02-09T17:58:29.003155+0000","last_fullsized":"2024-02-09T17:58:29.003155+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:58:20.598894+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:38:26.968892+0000","last_clean_scrub_stamp":"2024-02-09T17:58:20.598894+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:59:48.339132+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"844'12264","reported_seq":12462,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-02-09T17:58:28.525450+0000","last_change":"2024-02-09T17:57:24.093468+0000","last_active":"2024-02-09T17:58:28.525450+0000","last_peered":"2024-02-09T17:58:28.525450+0000","last_clean":"2024-02-09T17:58:28.525450+0000","last_became_active":"2024-02-09T17:55:16.444412+0000","last_became_peered":"2024-02-09T17:55:16.444412+0000","last_unstale":"2024-02-09T17:58:28.525450+0000","last_undegraded":"2024-02-09T17:58:28.525450+0000","last_fullsized":"2024-02-09T17:58:28.525450+0000","mapping_epoch":785,"log_start":"801'2200","ondisk_log_start":"801'2200","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"826'9206","last_scrub_stamp":"2024-02-09T17:57:24.093337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:57:24.093337+0000","objects_scrubbed":6053,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:58:36.402300+0000","scrub_duration":1551,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193855488,"num_objects":2958,"num_object_clones":0,"num_object_copies":8874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12264,"num_write_kb":487104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,0],"acting":[10,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"63'139","reported_seq":2085,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-02-09T17:58:29.019429+0000","last_change":"2024-02-09T17:58:20.815872+0000","last_active":"2024-02-09T17:58:29.019429+0000","last_peered":"2024-02-09T17:58:29.019429+0000","last_clean":"2024-02-09T17:58:29.019429+0000","last_became_active":"2024-02-09T17:41:14.521079+0000","last_became_peered":"2024-02-09T17:41:14.521079+0000","last_unstale":"2024-02-09T17:58:29.019429+0000","last_undegraded":"2024-02-09T17:58:29.019429+0000","last_fullsized":"2024-02-09T17:58:29.019429+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:58:20.815755+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:58:20.815755+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:59:33.006470+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"844'12223","reported_seq":12476,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-02-09T17:58:27.448549+0000","last_change":"2024-02-09T17:57:22.913985+0000","last_active":"2024-02-09T17:58:27.448549+0000","last_peered":"2024-02-09T17:58:27.448549+0000","last_clean":"2024-02-09T17:58:27.448549+0000","last_became_active":"2024-02-09T17:55:16.913155+0000","last_became_peered":"2024-02-09T17:55:16.913155+0000","last_unstale":"2024-02-09T17:58:27.448549+0000","last_undegraded":"2024-02-09T17:58:27.448549+0000","last_fullsized":"2024-02-09T17:58:27.448549+0000","mapping_epoch":785,"log_start":"801'2200","ondisk_log_start":"801'2200","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"826'9085","last_scrub_stamp":"2024-02-09T17:57:22.913841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:57:22.913841+0000","objects_scrubbed":6226,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:58:31.626831+0000","scrub_duration":1323,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200081408,"num_objects":3053,"num_object_clones":0,"num_object_copies":9159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12223,"num_write_kb":488832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,1],"acting":[8,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.2","version":"63'139","reported_seq":1649,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-02-09T17:58:22.517092+0000","last_change":"2024-02-09T17:57:44.411868+0000","last_active":"2024-02-09T17:58:22.517092+0000","last_peered":"2024-02-09T17:58:22.517092+0000","last_clean":"2024-02-09T17:58:22.517092+0000","last_became_active":"2024-02-09T17:43:39.981054+0000","last_became_peered":"2024-02-09T17:43:39.981054+0000","last_unstale":"2024-02-09T17:58:22.517092+0000","last_undegraded":"2024-02-09T17:58:22.517092+0000","last_fullsized":"2024-02-09T17:58:22.517092+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:57:44.411749+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:57:44.411749+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:58:55.408216+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"844'12161","reported_seq":12407,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-02-09T17:58:27.510761+0000","last_change":"2024-02-09T17:58:20.849649+0000","last_active":"2024-02-09T17:58:27.510761+0000","last_peered":"2024-02-09T17:58:27.510761+0000","last_clean":"2024-02-09T17:58:27.510761+0000","last_became_active":"2024-02-09T17:55:16.204001+0000","last_became_peered":"2024-02-09T17:55:16.204001+0000","last_unstale":"2024-02-09T17:58:27.510761+0000","last_undegraded":"2024-02-09T17:58:27.510761+0000","last_fullsized":"2024-02-09T17:58:27.510761+0000","mapping_epoch":785,"log_start":"801'2100","ondisk_log_start":"801'2100","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"843'11921","last_scrub_stamp":"2024-02-09T17:58:20.849360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:58:20.849360+0000","objects_scrubbed":3285,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:59:25.100582+0000","scrub_duration":1114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198246400,"num_objects":3025,"num_object_clones":0,"num_object_copies":9075,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12161,"num_write_kb":485952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,9],"acting":[8,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"63'139","reported_seq":1282,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-02-09T17:58:29.011063+0000","last_change":"2024-02-09T17:58:16.132954+0000","last_active":"2024-02-09T17:58:29.011063+0000","last_peered":"2024-02-09T17:58:29.011063+0000","last_clean":"2024-02-09T17:58:29.011063+0000","last_became_active":"2024-02-09T17:40:13.501711+0000","last_became_peered":"2024-02-09T17:40:13.501711+0000","last_unstale":"2024-02-09T17:58:29.011063+0000","last_undegraded":"2024-02-09T17:58:29.011063+0000","last_fullsized":"2024-02-09T17:58:29.011063+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:58:16.132824+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:46:40.684762+0000","last_clean_scrub_stamp":"2024-02-09T17:58:16.132824+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:59:31.886216+0000","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"844'12309","reported_seq":12440,"reported_epoch":844,"state":"active+undersized+degraded","last_fresh":"2024-02-09T17:58:27.255432+0000","last_change":"2024-02-09T17:56:40.456165+0000","last_active":"2024-02-09T17:58:27.255432+0000","last_peered":"2024-02-09T17:58:27.255432+0000","last_clean":"2024-02-09T17:56:39.401692+0000","last_became_active":"2024-02-09T17:56:40.456165+0000","last_became_peered":"2024-02-09T17:56:40.456165+0000","last_unstale":"2024-02-09T17:58:27.255432+0000","last_undegraded":"2024-02-09T17:56:40.454957+0000","last_fullsized":"2024-02-09T17:56:40.454832+0000","mapping_epoch":815,"log_start":"801'2300","ondisk_log_start":"801'2300","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:58:31.086123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204275712,"num_objects":3117,"num_object_clones":0,"num_object_copies":9351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3117,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12309,"num_write_kb":493632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":["9","5"],"object_location_counts":[{"shards":"5,9","objects":3117}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.1","version":"63'139","reported_seq":816,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-02-09T17:58:22.886819+0000","last_change":"2024-02-09T17:58:22.886862+0000","last_active":"2024-02-09T17:58:22.886819+0000","last_peered":"2024-02-09T17:58:22.886819+0000","last_clean":"2024-02-09T17:58:22.886819+0000","last_became_active":"2024-02-09T17:54:32.415032+0000","last_became_peered":"2024-02-09T17:54:32.415032+0000","last_unstale":"2024-02-09T17:58:22.886819+0000","last_undegraded":"2024-02-09T17:58:22.886819+0000","last_fullsized":"2024-02-09T17:58:22.886819+0000","mapping_epoch":766,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":767,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:58:22.886743+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:58:22.886743+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:59:36.422403+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.f","version":"844'12447","reported_seq":12697,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-02-09T17:58:26.561297+0000","last_change":"2024-02-09T17:58:21.092234+0000","last_active":"2024-02-09T17:58:26.561297+0000","last_peered":"2024-02-09T17:58:26.561297+0000","last_clean":"2024-02-09T17:58:26.561297+0000","last_became_active":"2024-02-09T17:55:16.205394+0000","last_became_peered":"2024-02-09T17:55:16.205394+0000","last_unstale":"2024-02-09T17:58:26.561297+0000","last_undegraded":"2024-02-09T17:58:26.561297+0000","last_fullsized":"2024-02-09T17:58:26.561297+0000","mapping_epoch":785,"log_start":"802'2400","ondisk_log_start":"802'2400","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"843'12215","last_scrub_stamp":"2024-02-09T17:58:21.090987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:58:21.090987+0000","objects_scrubbed":3447,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:59:21.833171+0000","scrub_duration":1416,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209125376,"num_objects":3191,"num_object_clones":0,"num_object_copies":9573,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12447,"num_write_kb":500416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,1],"acting":[11,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"63'139","reported_seq":1583,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-02-09T17:58:22.511194+0000","last_change":"2024-02-09T17:58:16.048610+0000","last_active":"2024-02-09T17:58:22.511194+0000","last_peered":"2024-02-09T17:58:22.511194+0000","last_clean":"2024-02-09T17:58:22.511194+0000","last_became_active":"2024-02-09T17:43:14.024894+0000","last_became_peered":"2024-02-09T17:43:14.024894+0000","last_unstale":"2024-02-09T17:58:22.511194+0000","last_undegraded":"2024-02-09T17:58:22.511194+0000","last_fullsized":"2024-02-09T17:58:22.511194+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:58:16.048489+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:11.970338+0000","last_clean_scrub_stamp":"2024-02-09T17:58:16.048489+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:59:36.932401+0000","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.e","version":"844'12372","reported_seq":12623,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-02-09T17:58:28.732024+0000","last_change":"2024-02-09T17:58:18.148809+0000","last_active":"2024-02-09T17:58:28.732024+0000","last_peered":"2024-02-09T17:58:28.732024+0000","last_clean":"2024-02-09T17:58:28.732024+0000","last_became_active":"2024-02-09T17:55:16.912769+0000","last_became_peered":"2024-02-09T17:55:16.912769+0000","last_unstale":"2024-02-09T17:58:28.732024+0000","last_undegraded":"2024-02-09T17:58:28.732024+0000","last_fullsized":"2024-02-09T17:58:28.732024+0000","mapping_epoch":785,"log_start":"801'2300","ondisk_log_start":"801'2300","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"842'11960","last_scrub_stamp":"2024-02-09T17:58:18.148684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:58:18.148684+0000","objects_scrubbed":3513,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:59:22.858753+0000","scrub_duration":737,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201457664,"num_objects":3074,"num_object_clones":0,"num_object_copies":9222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12372,"num_write_kb":494272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"63'139","reported_seq":1209,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-02-09T17:58:29.011290+0000","last_change":"2024-02-09T17:58:23.991019+0000","last_active":"2024-02-09T17:58:29.011290+0000","last_peered":"2024-02-09T17:58:29.011290+0000","last_clean":"2024-02-09T17:58:29.011290+0000","last_became_active":"2024-02-09T17:52:32.292519+0000","last_became_peered":"2024-02-09T17:52:32.292519+0000","last_unstale":"2024-02-09T17:58:29.011290+0000","last_undegraded":"2024-02-09T17:58:29.011290+0000","last_fullsized":"2024-02-09T17:58:29.011290+0000","mapping_epoch":716,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":717,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:58:23.990934+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:58:23.990934+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:59:37.449056+0000","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.d","version":"844'12193","reported_seq":12438,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-02-09T17:58:26.576732+0000","last_change":"2024-02-09T17:57:31.713043+0000","last_active":"2024-02-09T17:58:26.576732+0000","last_peered":"2024-02-09T17:58:26.576732+0000","last_clean":"2024-02-09T17:58:26.576732+0000","last_became_active":"2024-02-09T17:55:16.444812+0000","last_became_peered":"2024-02-09T17:55:16.444812+0000","last_unstale":"2024-02-09T17:58:26.576732+0000","last_undegraded":"2024-02-09T17:58:26.576732+0000","last_fullsized":"2024-02-09T17:58:26.576732+0000","mapping_epoch":785,"log_start":"801'2100","ondisk_log_start":"801'2100","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"827'9558","last_scrub_stamp":"2024-02-09T17:57:31.712823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:57:31.712823+0000","objects_scrubbed":5744,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:58:53.881399+0000","scrub_duration":1339,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201719832,"num_objects":3079,"num_object_clones":0,"num_object_copies":9237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3079,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12193,"num_write_kb":488641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,2],"acting":[11,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"63'139","reported_seq":1662,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-02-09T17:58:22.506926+0000","last_change":"2024-02-09T17:58:20.785142+0000","last_active":"2024-02-09T17:58:22.506926+0000","last_peered":"2024-02-09T17:58:22.506926+0000","last_clean":"2024-02-09T17:58:22.506926+0000","last_became_active":"2024-02-09T17:33:00.284924+0000","last_became_peered":"2024-02-09T17:33:00.284924+0000","last_unstale":"2024-02-09T17:58:22.506926+0000","last_undegraded":"2024-02-09T17:58:22.506926+0000","last_fullsized":"2024-02-09T17:58:22.506926+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:58:20.785047+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:56:45.406767+0000","last_clean_scrub_stamp":"2024-02-09T17:58:20.785047+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:59:48.005974+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.c","version":"844'12302","reported_seq":12434,"reported_epoch":844,"state":"active+undersized+degraded","last_fresh":"2024-02-09T17:58:27.247933+0000","last_change":"2024-02-09T17:56:40.460851+0000","last_active":"2024-02-09T17:58:27.247933+0000","last_peered":"2024-02-09T17:58:27.247933+0000","last_clean":"2024-02-09T17:56:39.437320+0000","last_became_active":"2024-02-09T17:56:40.460851+0000","last_became_peered":"2024-02-09T17:56:40.460851+0000","last_unstale":"2024-02-09T17:58:27.247933+0000","last_undegraded":"2024-02-09T17:56:40.455527+0000","last_fullsized":"2024-02-09T17:56:40.455414+0000","mapping_epoch":815,"log_start":"801'2300","ondisk_log_start":"801'2300","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204996608,"num_objects":3128,"num_object_clones":0,"num_object_copies":9384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3128,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12302,"num_write_kb":493760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":["9","8"],"object_location_counts":[{"shards":"8,9","objects":3128}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.5","version":"63'139","reported_seq":1512,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-02-09T17:58:27.081725+0000","last_change":"2024-02-09T17:58:27.081766+0000","last_active":"2024-02-09T17:58:27.081725+0000","last_peered":"2024-02-09T17:58:27.081725+0000","last_clean":"2024-02-09T17:58:27.081725+0000","last_became_active":"2024-02-09T17:58:10.037010+0000","last_became_peered":"2024-02-09T17:58:10.037010+0000","last_unstale":"2024-02-09T17:58:27.081725+0000","last_undegraded":"2024-02-09T17:58:27.081725+0000","last_fullsized":"2024-02-09T17:58:27.081725+0000","mapping_epoch":839,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":840,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:58:27.081656+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:58:27.081656+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:59:49.676839+0000","scrub_duration":169,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,8],"acting":[10,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.3","version":"844'12293","reported_seq":12506,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-02-09T17:58:26.541055+0000","last_change":"2024-02-09T17:56:51.477774+0000","last_active":"2024-02-09T17:58:26.541055+0000","last_peered":"2024-02-09T17:58:26.541055+0000","last_clean":"2024-02-09T17:58:26.541055+0000","last_became_active":"2024-02-09T17:55:16.445048+0000","last_became_peered":"2024-02-09T17:55:16.445048+0000","last_unstale":"2024-02-09T17:58:26.541055+0000","last_undegraded":"2024-02-09T17:58:26.541055+0000","last_fullsized":"2024-02-09T17:58:26.541055+0000","mapping_epoch":785,"log_start":"801'2200","ondisk_log_start":"801'2200","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"820'7500","last_scrub_stamp":"2024-02-09T17:56:51.477633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:56:51.477633+0000","objects_scrubbed":7455,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2218,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211091456,"num_objects":3221,"num_object_clones":0,"num_object_copies":9663,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12293,"num_write_kb":496448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"63'139","reported_seq":1584,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-02-09T17:58:22.592185+0000","last_change":"2024-02-09T17:58:21.118935+0000","last_active":"2024-02-09T17:58:22.592185+0000","last_peered":"2024-02-09T17:58:22.592185+0000","last_clean":"2024-02-09T17:58:22.592185+0000","last_became_active":"2024-02-09T17:43:14.041291+0000","last_became_peered":"2024-02-09T17:43:14.041291+0000","last_unstale":"2024-02-09T17:58:22.592185+0000","last_undegraded":"2024-02-09T17:58:22.592185+0000","last_fullsized":"2024-02-09T17:58:22.592185+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:58:21.118819+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:56:44.962386+0000","last_clean_scrub_stamp":"2024-02-09T17:58:21.118819+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:59:26.056285+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.2","version":"844'12472","reported_seq":12722,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-02-09T17:58:28.683545+0000","last_change":"2024-02-09T17:58:21.745154+0000","last_active":"2024-02-09T17:58:28.683545+0000","last_peered":"2024-02-09T17:58:28.683545+0000","last_clean":"2024-02-09T17:58:28.683545+0000","last_became_active":"2024-02-09T17:55:16.913786+0000","last_became_peered":"2024-02-09T17:55:16.913786+0000","last_unstale":"2024-02-09T17:58:28.683545+0000","last_undegraded":"2024-02-09T17:58:28.683545+0000","last_fullsized":"2024-02-09T17:58:28.683545+0000","mapping_epoch":785,"log_start":"802'2400","ondisk_log_start":"802'2400","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"843'12189","last_scrub_stamp":"2024-02-09T17:58:21.745000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:58:21.745000+0000","objects_scrubbed":3320,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:59:30.102786+0000","scrub_duration":937,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197525504,"num_objects":3014,"num_object_clones":0,"num_object_copies":9042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12472,"num_write_kb":495552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,4],"acting":[10,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.b","version":"63'139","reported_seq":952,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-02-09T17:58:26.084138+0000","last_change":"2024-02-09T17:58:26.084179+0000","last_active":"2024-02-09T17:58:26.084138+0000","last_peered":"2024-02-09T17:58:26.084138+0000","last_clean":"2024-02-09T17:58:26.084138+0000","last_became_active":"2024-02-09T17:55:08.938203+0000","last_became_peered":"2024-02-09T17:55:08.938203+0000","last_unstale":"2024-02-09T17:58:26.084138+0000","last_undegraded":"2024-02-09T17:58:26.084138+0000","last_fullsized":"2024-02-09T17:58:26.084138+0000","mapping_epoch":781,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":782,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:58:26.084047+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:58:26.084047+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:59:44.997234+0000","scrub_duration":270,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.1","version":"845'12749","reported_seq":12960,"reported_epoch":845,"state":"active+clean+scrubbing+deep","last_fresh":"2024-02-09T17:58:29.858382+0000","last_change":"2024-02-09T17:58:25.741494+0000","last_active":"2024-02-09T17:58:29.858382+0000","last_peered":"2024-02-09T17:58:29.858382+0000","last_clean":"2024-02-09T17:58:29.858382+0000","last_became_active":"2024-02-09T17:55:16.205574+0000","last_became_peered":"2024-02-09T17:55:16.205574+0000","last_unstale":"2024-02-09T17:58:29.858382+0000","last_undegraded":"2024-02-09T17:58:29.858382+0000","last_fullsized":"2024-02-09T17:58:29.858382+0000","mapping_epoch":785,"log_start":"802'2700","ondisk_log_start":"802'2700","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"818'7081","last_scrub_stamp":"2024-02-09T17:56:46.137952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:56:46.137952+0000","objects_scrubbed":549,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"deep scrubbing for 4s","scrub_duration":1634,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197918720,"num_objects":3020,"num_object_clones":0,"num_object_copies":9060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12200,"num_write_kb":487040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,2],"acting":[1,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"63'139","reported_seq":1166,"reported_epoch":845,"state":"active+undersized","last_fresh":"2024-02-09T17:58:29.021111+0000","last_change":"2024-02-09T17:56:40.455724+0000","last_active":"2024-02-09T17:58:29.021111+0000","last_peered":"2024-02-09T17:58:29.021111+0000","last_clean":"2024-02-09T17:46:43.974099+0000","last_became_active":"2024-02-09T17:56:40.455724+0000","last_became_peered":"2024-02-09T17:56:40.455724+0000","last_unstale":"2024-02-09T17:58:29.021111+0000","last_undegraded":"2024-02-09T17:58:29.021111+0000","last_fullsized":"2024-02-09T17:56:40.454196+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:10.313268+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:55:10.313268+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.0","version":"844'12523","reported_seq":12654,"reported_epoch":844,"state":"active+undersized+degraded","last_fresh":"2024-02-09T17:58:27.275349+0000","last_change":"2024-02-09T17:56:40.456476+0000","last_active":"2024-02-09T17:58:27.275349+0000","last_peered":"2024-02-09T17:58:27.275349+0000","last_clean":"2024-02-09T17:56:39.422623+0000","last_became_active":"2024-02-09T17:56:40.456476+0000","last_became_peered":"2024-02-09T17:56:40.456476+0000","last_unstale":"2024-02-09T17:58:27.275349+0000","last_undegraded":"2024-02-09T17:56:40.455373+0000","last_fullsized":"2024-02-09T17:56:40.455263+0000","mapping_epoch":815,"log_start":"802'2500","ondisk_log_start":"802'2500","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:55:15.183893+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:58:31.086150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209125376,"num_objects":3191,"num_object_clones":0,"num_object_copies":9573,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12523,"num_write_kb":502848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":["9","5"],"object_location_counts":[{"shards":"5,9","objects":3191}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.9","version":"63'139","reported_seq":2091,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-02-09T17:58:29.019445+0000","last_change":"2024-02-09T17:58:27.034176+0000","last_active":"2024-02-09T17:58:29.019445+0000","last_peered":"2024-02-09T17:58:29.019445+0000","last_clean":"2024-02-09T17:58:29.019445+0000","last_became_active":"2024-02-09T17:43:28.590237+0000","last_became_peered":"2024-02-09T17:43:28.590237+0000","last_unstale":"2024-02-09T17:58:29.019445+0000","last_undegraded":"2024-02-09T17:58:29.019445+0000","last_fullsized":"2024-02-09T17:58:29.019445+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:58:27.034081+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:56:49.660423+0000","last_clean_scrub_stamp":"2024-02-09T17:58:27.034081+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:59:43.919997+0000","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.7","version":"845'12233","reported_seq":12485,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-02-09T17:58:29.841322+0000","last_change":"2024-02-09T17:58:21.743579+0000","last_active":"2024-02-09T17:58:29.841322+0000","last_peered":"2024-02-09T17:58:29.841322+0000","last_clean":"2024-02-09T17:58:29.841322+0000","last_became_active":"2024-02-09T17:55:16.206561+0000","last_became_peered":"2024-02-09T17:55:16.206561+0000","last_unstale":"2024-02-09T17:58:29.841322+0000","last_undegraded":"2024-02-09T17:58:29.841322+0000","last_fullsized":"2024-02-09T17:58:29.841322+0000","mapping_epoch":785,"log_start":"801'2200","ondisk_log_start":"801'2200","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"843'11950","last_scrub_stamp":"2024-02-09T17:58:21.743354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:58:21.743354+0000","objects_scrubbed":3260,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:59:50.935864+0000","scrub_duration":901,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193658880,"num_objects":2955,"num_object_clones":0,"num_object_copies":8865,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12233,"num_write_kb":486016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.e","version":"63'139","reported_seq":1535,"reported_epoch":845,"state":"active+undersized","last_fresh":"2024-02-09T17:58:29.025160+0000","last_change":"2024-02-09T17:56:40.457215+0000","last_active":"2024-02-09T17:58:29.025160+0000","last_peered":"2024-02-09T17:58:29.025160+0000","last_clean":"2024-02-09T17:56:36.466317+0000","last_became_active":"2024-02-09T17:56:40.457215+0000","last_became_peered":"2024-02-09T17:56:40.457215+0000","last_unstale":"2024-02-09T17:58:29.025160+0000","last_undegraded":"2024-02-09T17:58:29.025160+0000","last_fullsized":"2024-02-09T17:56:40.455199+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:08.044477+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:14.981453+0000","last_clean_scrub_stamp":"2024-02-09T17:55:08.044477+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:58:29.586541+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"844'12787","reported_seq":13009,"reported_epoch":844,"state":"active+clean+scrubbing+deep","last_fresh":"2024-02-09T17:58:26.606628+0000","last_change":"2024-02-09T17:58:23.784414+0000","last_active":"2024-02-09T17:58:26.606628+0000","last_peered":"2024-02-09T17:58:26.606628+0000","last_clean":"2024-02-09T17:58:26.606628+0000","last_became_active":"2024-02-09T17:55:16.445702+0000","last_became_peered":"2024-02-09T17:55:16.445702+0000","last_unstale":"2024-02-09T17:58:26.606628+0000","last_undegraded":"2024-02-09T17:58:26.606628+0000","last_fullsized":"2024-02-09T17:58:26.606628+0000","mapping_epoch":785,"log_start":"802'2700","ondisk_log_start":"802'2700","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"816'7145","last_scrub_stamp":"2024-02-09T17:56:43.410521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:56:43.410521+0000","objects_scrubbed":323,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"deep scrubbing for 2s","scrub_duration":1999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209059840,"num_objects":3190,"num_object_clones":0,"num_object_copies":9570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12464,"num_write_kb":500928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,7],"acting":[11,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.f","version":"63'139","reported_seq":1608,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-02-09T17:58:29.183898+0000","last_change":"2024-02-09T17:58:17.299511+0000","last_active":"2024-02-09T17:58:29.183898+0000","last_peered":"2024-02-09T17:58:29.183898+0000","last_clean":"2024-02-09T17:58:29.183898+0000","last_became_active":"2024-02-09T17:33:05.316100+0000","last_became_peered":"2024-02-09T17:33:05.316100+0000","last_unstale":"2024-02-09T17:58:29.183898+0000","last_undegraded":"2024-02-09T17:58:29.183898+0000","last_fullsized":"2024-02-09T17:58:29.183898+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:58:17.299422+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:48:34.651647+0000","last_clean_scrub_stamp":"2024-02-09T17:58:17.299422+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:59:46.230910+0000","scrub_duration":193,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.5","version":"844'12428","reported_seq":12681,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-02-09T17:58:28.594667+0000","last_change":"2024-02-09T17:58:22.708962+0000","last_active":"2024-02-09T17:58:28.594667+0000","last_peered":"2024-02-09T17:58:28.594667+0000","last_clean":"2024-02-09T17:58:28.594667+0000","last_became_active":"2024-02-09T17:55:16.913201+0000","last_became_peered":"2024-02-09T17:55:16.913201+0000","last_unstale":"2024-02-09T17:58:28.594667+0000","last_undegraded":"2024-02-09T17:58:28.594667+0000","last_fullsized":"2024-02-09T17:58:28.594667+0000","mapping_epoch":785,"log_start":"802'2400","ondisk_log_start":"802'2400","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"844'12214","last_scrub_stamp":"2024-02-09T17:58:22.708031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:58:22.708031+0000","objects_scrubbed":3260,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:59:39.786074+0000","scrub_duration":946,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198180864,"num_objects":3024,"num_object_clones":0,"num_object_copies":9072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":494464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,1],"acting":[10,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.c","version":"63'139","reported_seq":874,"reported_epoch":845,"state":"active+undersized","last_fresh":"2024-02-09T17:58:29.183671+0000","last_change":"2024-02-09T17:56:40.452699+0000","last_active":"2024-02-09T17:58:29.183671+0000","last_peered":"2024-02-09T17:58:29.183671+0000","last_clean":"2024-02-09T17:39:31.468742+0000","last_became_active":"2024-02-09T17:56:40.452699+0000","last_became_peered":"2024-02-09T17:56:40.452699+0000","last_unstale":"2024-02-09T17:58:29.183671+0000","last_undegraded":"2024-02-09T17:58:29.183671+0000","last_fullsized":"2024-02-09T17:56:40.451638+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:55:07.304236+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:14.135321+0000","last_clean_scrub_stamp":"2024-02-09T17:55:07.304236+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163840,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"844'12049","reported_seq":12297,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-02-09T17:58:27.159393+0000","last_change":"2024-02-09T17:58:24.326067+0000","last_active":"2024-02-09T17:58:27.159393+0000","last_peered":"2024-02-09T17:58:27.159393+0000","last_clean":"2024-02-09T17:58:27.159393+0000","last_became_active":"2024-02-09T17:55:16.444827+0000","last_became_peered":"2024-02-09T17:55:16.444827+0000","last_unstale":"2024-02-09T17:58:27.159393+0000","last_undegraded":"2024-02-09T17:58:27.159393+0000","last_fullsized":"2024-02-09T17:58:27.159393+0000","mapping_epoch":785,"log_start":"801'2000","ondisk_log_start":"801'2000","created":785,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"844'11952","last_scrub_stamp":"2024-02-09T17:58:24.325901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T17:55:15.183893+0000","last_clean_scrub_stamp":"2024-02-09T17:58:24.325901+0000","objects_scrubbed":3092,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:59:50.116897+0000","scrub_duration":1184,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194445312,"num_objects":2967,"num_object_clones":0,"num_object_copies":8901,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12049,"num_write_kb":480512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.d","version":"63'139","reported_seq":1140,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-02-09T17:58:29.019650+0000","last_change":"2024-02-09T17:58:18.944283+0000","last_active":"2024-02-09T17:58:29.019650+0000","last_peered":"2024-02-09T17:58:29.019650+0000","last_clean":"2024-02-09T17:58:29.019650+0000","last_became_active":"2024-02-09T17:43:14.036759+0000","last_became_peered":"2024-02-09T17:43:14.036759+0000","last_unstale":"2024-02-09T17:58:29.019650+0000","last_undegraded":"2024-02-09T17:58:29.019650+0000","last_fullsized":"2024-02-09T17:58:29.019650+0000","mapping_epoch":454,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:58:18.944179+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:58:18.944179+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:59:44.758624+0000","scrub_duration":195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"63'139","reported_seq":1312,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-02-09T17:58:29.019321+0000","last_change":"2024-02-09T17:58:19.807512+0000","last_active":"2024-02-09T17:58:29.019321+0000","last_peered":"2024-02-09T17:58:29.019321+0000","last_clean":"2024-02-09T17:58:29.019321+0000","last_became_active":"2024-02-09T17:46:47.013920+0000","last_became_peered":"2024-02-09T17:46:47.013920+0000","last_unstale":"2024-02-09T17:58:29.019321+0000","last_undegraded":"2024-02-09T17:58:29.019321+0000","last_fullsized":"2024-02-09T17:58:29.019321+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:58:19.807398+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:48:35.170638+0000","last_clean_scrub_stamp":"2024-02-09T17:58:19.807398+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:59:23.755805+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"63'139","reported_seq":1579,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-02-09T17:58:24.768021+0000","last_change":"2024-02-09T17:58:24.768048+0000","last_active":"2024-02-09T17:58:24.768021+0000","last_peered":"2024-02-09T17:58:24.768021+0000","last_clean":"2024-02-09T17:58:24.768021+0000","last_became_active":"2024-02-09T17:55:21.231902+0000","last_became_peered":"2024-02-09T17:55:21.231902+0000","last_unstale":"2024-02-09T17:58:24.768021+0000","last_undegraded":"2024-02-09T17:58:24.768021+0000","last_fullsized":"2024-02-09T17:58:24.768021+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":791,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:58:24.767953+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:44:51.190789+0000","last_clean_scrub_stamp":"2024-02-09T17:58:24.767953+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:59:31.653534+0000","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.12","version":"63'139","reported_seq":1657,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-02-09T17:58:29.011075+0000","last_change":"2024-02-09T17:57:31.202700+0000","last_active":"2024-02-09T17:58:29.011075+0000","last_peered":"2024-02-09T17:58:29.011075+0000","last_clean":"2024-02-09T17:58:29.011075+0000","last_became_active":"2024-02-09T17:46:47.015622+0000","last_became_peered":"2024-02-09T17:46:47.015622+0000","last_unstale":"2024-02-09T17:58:29.011075+0000","last_undegraded":"2024-02-09T17:58:29.011075+0000","last_fullsized":"2024-02-09T17:58:29.011075+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:57:31.202562+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:57:31.202562+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:58:35.866210+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.13","version":"63'139","reported_seq":1276,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-02-09T17:58:29.011157+0000","last_change":"2024-02-09T17:57:44.410517+0000","last_active":"2024-02-09T17:58:29.011157+0000","last_peered":"2024-02-09T17:58:29.011157+0000","last_clean":"2024-02-09T17:58:29.011157+0000","last_became_active":"2024-02-09T17:41:14.521672+0000","last_became_peered":"2024-02-09T17:41:14.521672+0000","last_unstale":"2024-02-09T17:58:29.011157+0000","last_undegraded":"2024-02-09T17:58:29.011157+0000","last_fullsized":"2024-02-09T17:58:29.011157+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T17:57:44.410378+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T17:57:44.410378+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T17:58:56.127838+0000","scrub_duration":224,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":3224764440,"num_objects":49207,"num_object_clones":0,"num_object_copies":147621,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49207,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":196933,"num_write_kb":7876417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10285428736,"data_stored":10285416520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160805,"ondisk_log_size":160805,"up":45,"acting":45,"num_store_stats":11},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":426528,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":88,"acting":88,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547057074,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1887104,"kb_used_data":1017336,"kb_used_omap":1,"kb_used_meta":869694,"kb_avail":91858048,"statfs":{"total":95995035648,"available":94062641152,"internally_reserved":0,"allocated":1041752064,"data_stored":1041474949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":890567108},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,0,1,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:58:04 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.595,"15min":1.5920000000000001},"min":{"1min":0.22,"5min":0.22,"15min":0.21199999999999999},"max":{"1min":9.6379999999999999,"5min":9.8970000000000002,"15min":20.003},"last":1.6559999999999999},{"interface":"front","average":{"1min":2.6379999999999999,"5min":1.756,"15min":1.476},"min":{"1min":0.23300000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":9.0790000000000006,"5min":9.6259999999999994,"15min":9.6259999999999994},"last":1.899}]},{"osd":1,"last update":"Fri Feb 9 17:58:04 2024","interfaces":[{"interface":"back","average":{"1min":2.98,"5min":1.829,"15min":1.673},"min":{"1min":0.25600000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":9.1980000000000004,"5min":9.532,"15min":11.576000000000001},"last":1.9570000000000001},{"interface":"front","average":{"1min":3,"5min":2.0710000000000002,"15min":1.857},"min":{"1min":0.32600000000000001,"5min":0.26400000000000001,"15min":0.25},"max":{"1min":14.81,"5min":14.81,"15min":14.81},"last":2.391}]},{"osd":2,"last update":"Fri Feb 9 17:58:14 2024","interfaces":[{"interface":"back","average":{"1min":3.2839999999999998,"5min":2.028,"15min":1.6659999999999999},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":14.766999999999999,"5min":14.766999999999999,"15min":14.766999999999999},"last":2.3220000000000001},{"interface":"front","average":{"1min":2.0819999999999999,"5min":1.5229999999999999,"15min":1.296},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":9.1560000000000006,"5min":9.1560000000000006,"15min":9.1560000000000006},"last":1.595}]},{"osd":3,"last update":"Fri Feb 9 17:56:01 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":2.5,"15min":1.8740000000000001},"min":{"1min":0.47999999999999998,"5min":0.32200000000000001,"15min":0.248},"max":{"1min":5.1509999999999998,"5min":43.344000000000001,"15min":43.344000000000001},"last":4.5949999999999998},{"interface":"front","average":{"1min":1.556,"5min":2.5379999999999998,"15min":1.954},"min":{"1min":0.45300000000000001,"5min":0.375,"15min":0.33600000000000002},"max":{"1min":5.1769999999999996,"5min":43.378,"15min":43.378},"last":4.6189999999999998}]},{"osd":4,"last update":"Fri Feb 9 17:57:41 2024","interfaces":[{"interface":"back","average":{"1min":2.4300000000000002,"5min":2.4300000000000002,"15min":2.4300000000000002},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":12.818,"5min":12.818,"15min":12.818},"last":2.4180000000000001},{"interface":"front","average":{"1min":2.3969999999999998,"5min":2.3969999999999998,"15min":2.3969999999999998},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":12.893000000000001,"5min":12.893000000000001,"15min":12.893000000000001},"last":2.4969999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:58:04 2024","interfaces":[{"interface":"back","average":{"1min":1.5940000000000001,"5min":1.202,"15min":1.601},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.252},"max":{"1min":9.5990000000000002,"5min":9.5990000000000002,"15min":18.878},"last":1.627},{"interface":"front","average":{"1min":2.9279999999999999,"5min":1.651,"15min":1.7809999999999999},"min":{"1min":0.44500000000000001,"5min":0.30099999999999999,"15min":0.28999999999999998},"max":{"1min":14.914999999999999,"5min":14.914999999999999,"15min":20.038},"last":2.4689999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:58:04 2024","interfaces":[{"interface":"back","average":{"1min":2.8300000000000001,"5min":1.6379999999999999,"15min":1.75},"min":{"1min":0.36899999999999999,"5min":0.315,"15min":0.30399999999999999},"max":{"1min":9.1300000000000008,"5min":11.885,"15min":12.362},"last":1.9299999999999999},{"interface":"front","average":{"1min":3.101,"5min":1.7270000000000001,"15min":1.8180000000000001},"min":{"1min":0.53800000000000003,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":14.962,"5min":14.962,"15min":15.409000000000001},"last":2.4430000000000001}]},{"osd":7,"last update":"Fri Feb 9 17:58:04 2024","interfaces":[{"interface":"back","average":{"1min":3.0550000000000002,"5min":1.6619999999999999,"15min":1.865},"min":{"1min":0.51200000000000001,"5min":0.32800000000000001,"15min":0.317},"max":{"1min":14.890000000000001,"5min":14.890000000000001,"15min":18.846},"last":2.5219999999999998},{"interface":"front","average":{"1min":2.8170000000000002,"5min":1.4870000000000001,"15min":1.9530000000000001},"min":{"1min":0.40100000000000002,"5min":0.33400000000000002,"15min":0.318},"max":{"1min":9.1029999999999998,"5min":9.1029999999999998,"15min":24.373999999999999},"last":1.984}]},{"osd":8,"last update":"Fri Feb 9 17:58:20 2024","interfaces":[{"interface":"back","average":{"1min":4.3840000000000003,"5min":2.395,"15min":1.786},"min":{"1min":0.311,"5min":0.307,"15min":0.27900000000000003},"max":{"1min":29.233000000000001,"5min":29.233000000000001,"15min":29.233000000000001},"last":2.056},{"interface":"front","average":{"1min":4.6550000000000002,"5min":2.4590000000000001,"15min":1.8420000000000001},"min":{"1min":0.42499999999999999,"5min":0.34499999999999997,"15min":0.32700000000000001},"max":{"1min":29.091999999999999,"5min":29.091999999999999,"15min":29.091999999999999},"last":2.1080000000000001}]},{"osd":9,"last update":"Fri Feb 9 17:57:50 2024","interfaces":[{"interface":"back","average":{"1min":1.9730000000000001,"5min":1.8680000000000001,"15min":1.7749999999999999},"min":{"1min":0.39000000000000001,"5min":0.32900000000000001,"15min":0.29399999999999998},"max":{"1min":10.247999999999999,"5min":22.613,"15min":22.613},"last":1.6779999999999999},{"interface":"front","average":{"1min":2.0129999999999999,"5min":1.7030000000000001,"15min":1.5429999999999999},"min":{"1min":0.434,"5min":0.38300000000000001,"15min":0.34399999999999997},"max":{"1min":10.279,"5min":10.279,"15min":12.313000000000001},"last":1.6930000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:58:04 2024","interfaces":[{"interface":"back","average":{"1min":4.6040000000000001,"5min":2.5880000000000001,"15min":2.0459999999999998},"min":{"1min":0.57799999999999996,"5min":0.41199999999999998,"15min":0.32700000000000001},"max":{"1min":22.215,"5min":22.215,"15min":22.215},"last":2.1379999999999999},{"interface":"front","average":{"1min":3.5630000000000002,"5min":2.117,"15min":1.821},"min":{"1min":0.56299999999999994,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":14.836,"5min":14.836,"15min":20.233000000000001},"last":2.363}]}]},{"osd":10,"up_from":84,"seq":360777253307,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1877436,"kb_used_data":1164532,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":91867716,"statfs":{"total":95995035648,"available":94072541184,"internally_reserved":0,"allocated":1192480768,"data_stored":1192207725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":729938368},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:57:52 2024","interfaces":[{"interface":"back","average":{"1min":2.4809999999999999,"5min":1.9350000000000001,"15min":1.3600000000000001},"min":{"1min":0.27300000000000002,"5min":0.24199999999999999,"15min":0.23999999999999999},"max":{"1min":8.8040000000000003,"5min":9.5020000000000007,"15min":10.532},"last":0.65100000000000002},{"interface":"front","average":{"1min":2.3580000000000001,"5min":1.9950000000000001,"15min":1.3839999999999999},"min":{"1min":0.313,"5min":0.27100000000000002,"15min":0.248},"max":{"1min":8.8989999999999991,"5min":12.111000000000001,"15min":12.111000000000001},"last":0.54100000000000004}]},{"osd":1,"last update":"Fri Feb 9 17:58:20 2024","interfaces":[{"interface":"back","average":{"1min":3.407,"5min":2.0230000000000001,"15min":1.718},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":13.356,"5min":13.356,"15min":13.356},"last":0.72999999999999998},{"interface":"front","average":{"1min":3.7250000000000001,"5min":2.1400000000000001,"15min":1.758},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":13.420999999999999,"5min":13.420999999999999,"15min":13.420999999999999},"last":0.752}]},{"osd":2,"last update":"Fri Feb 9 17:58:23 2024","interfaces":[{"interface":"back","average":{"1min":2.9060000000000001,"5min":2.274,"15min":1.915},"min":{"1min":0.34200000000000003,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":12.071999999999999,"5min":22.652999999999999,"15min":22.652999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":3.1930000000000001,"5min":2.2130000000000001,"15min":1.919},"min":{"1min":0.46400000000000002,"5min":0.28899999999999998,"15min":0.25600000000000001},"max":{"1min":12.180999999999999,"5min":12.180999999999999,"15min":38.868000000000002},"last":0.67600000000000005}]},{"osd":3,"last update":"Fri Feb 9 17:55:09 2024","interfaces":[{"interface":"back","average":{"1min":2.282,"5min":2.1549999999999998,"15min":1.796},"min":{"1min":0.29999999999999999,"5min":0.27700000000000002,"15min":0.26100000000000001},"max":{"1min":6.9909999999999997,"5min":14.689,"15min":35.113},"last":0.65700000000000003},{"interface":"front","average":{"1min":1.296,"5min":2.4550000000000001,"15min":2.077},"min":{"1min":0.27800000000000002,"5min":0.249,"15min":0.249},"max":{"1min":3.5449999999999999,"5min":11.209,"15min":34.359000000000002},"last":0.54400000000000004}]},{"osd":4,"last update":"Fri Feb 9 17:57:48 2024","interfaces":[{"interface":"back","average":{"1min":1.603,"5min":1.1319999999999999,"15min":1.2889999999999999},"min":{"1min":0.39800000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":5.673,"5min":9.5850000000000009,"15min":11.705},"last":4.6079999999999997},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.1259999999999999,"15min":1.353},"min":{"1min":0.47499999999999998,"5min":0.33900000000000002,"15min":0.27000000000000002},"max":{"1min":5.7850000000000001,"5min":9.6630000000000003,"15min":11.744},"last":0.57899999999999996}]},{"osd":5,"last update":"Fri Feb 9 17:58:23 2024","interfaces":[{"interface":"back","average":{"1min":1.627,"5min":1.0880000000000001,"15min":1.381},"min":{"1min":0.32000000000000001,"5min":0.315,"15min":0.29099999999999998},"max":{"1min":4.226,"5min":7.3380000000000001,"15min":10.759},"last":4.5419999999999998},{"interface":"front","average":{"1min":2.5059999999999998,"5min":1.498,"15min":1.5489999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":12.157999999999999,"5min":12.157999999999999,"15min":12.157999999999999},"last":0.60199999999999998}]},{"osd":6,"last update":"Fri Feb 9 17:57:27 2024","interfaces":[{"interface":"back","average":{"1min":1.641,"5min":1.1299999999999999,"15min":1.492},"min":{"1min":0.40500000000000003,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":7.3929999999999998,"5min":9.2599999999999998,"15min":10.962},"last":4.6319999999999997},{"interface":"front","average":{"1min":2.2000000000000002,"5min":1.3069999999999999,"15min":1.5760000000000001},"min":{"1min":0.45800000000000002,"5min":0.35099999999999998,"15min":0.32600000000000001},"max":{"1min":9.6099999999999994,"5min":9.6099999999999994,"15min":12.414},"last":0.629}]},{"osd":7,"last update":"Fri Feb 9 17:58:20 2024","interfaces":[{"interface":"back","average":{"1min":2.0470000000000002,"5min":1.2050000000000001,"15min":2.6080000000000001},"min":{"1min":0.38300000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":7.4219999999999997,"5min":7.4219999999999997,"15min":11.288},"last":4.5819999999999999},{"interface":"front","average":{"1min":3.4729999999999999,"5min":1.593,"15min":2.7280000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":13.388999999999999,"5min":13.388999999999999,"15min":13.388999999999999},"last":0.70799999999999996}]},{"osd":8,"last update":"Fri Feb 9 17:58:20 2024","interfaces":[{"interface":"back","average":{"1min":5.8710000000000004,"5min":2.8210000000000002,"15min":1.782},"min":{"1min":0.38100000000000001,"5min":0.32400000000000001,"15min":0.31},"max":{"1min":22.559000000000001,"5min":22.559000000000001,"15min":22.559000000000001},"last":0.96299999999999997},{"interface":"front","average":{"1min":3.1880000000000002,"5min":1.8859999999999999,"15min":1.645},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.30599999999999999},"max":{"1min":11.986000000000001,"5min":22.824999999999999,"15min":22.824999999999999},"last":4.5069999999999997}]},{"osd":9,"last update":"Fri Feb 9 17:58:23 2024","interfaces":[{"interface":"back","average":{"1min":4.9790000000000001,"5min":2.4220000000000002,"15min":1.8919999999999999},"min":{"1min":0.39200000000000002,"5min":0.34300000000000003,"15min":0.33300000000000002},"max":{"1min":23.201000000000001,"5min":23.201000000000001,"15min":23.201000000000001},"last":1.458},{"interface":"front","average":{"1min":5.4980000000000002,"5min":2.4870000000000001,"15min":2.0110000000000001},"min":{"1min":0.41099999999999998,"5min":0.39600000000000002,"15min":0.36099999999999999},"max":{"1min":17.498999999999999,"5min":17.498999999999999,"15min":17.498999999999999},"last":2.605}]},{"osd":11,"last update":"Fri Feb 9 17:57:52 2024","interfaces":[{"interface":"back","average":{"1min":2.7650000000000001,"5min":1.867,"15min":1.5269999999999999},"min":{"1min":0.46300000000000002,"5min":0.36799999999999999,"15min":0.35399999999999998},"max":{"1min":10.625999999999999,"5min":10.625999999999999,"15min":23.579000000000001},"last":0.83899999999999997},{"interface":"front","average":{"1min":2.7890000000000001,"5min":1.9390000000000001,"15min":1.587},"min":{"1min":0.59699999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":8.9260000000000002,"5min":22.725999999999999,"15min":23.619},"last":0.48499999999999999}]}]},{"osd":9,"up_from":77,"seq":330712482237,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2040700,"kb_used_data":1188980,"kb_used_omap":1,"kb_used_meta":851646,"kb_avail":91704452,"statfs":{"total":95995035648,"available":93905358848,"internally_reserved":0,"allocated":1217515520,"data_stored":1217242477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":872085952},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:58:23 2024","interfaces":[{"interface":"back","average":{"1min":2.1259999999999999,"5min":1.698,"15min":1.494},"min":{"1min":0.253,"5min":0.23899999999999999,"15min":0.215},"max":{"1min":6.0110000000000001,"5min":18.623999999999999,"15min":18.623999999999999},"last":5.3929999999999998},{"interface":"front","average":{"1min":2.4980000000000002,"5min":1.673,"15min":1.4870000000000001},"min":{"1min":0.26000000000000001,"5min":0.25700000000000001,"15min":0.22700000000000001},"max":{"1min":9.9610000000000003,"5min":9.9610000000000003,"15min":11.596},"last":2.6560000000000001}]},{"osd":1,"last update":"Fri Feb 9 17:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.901,"5min":2.1269999999999998,"15min":1.6699999999999999},"min":{"1min":0.255,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":8.3190000000000008,"5min":61.031999999999996,"15min":61.031999999999996},"last":4.1219999999999999},{"interface":"front","average":{"1min":2.391,"5min":2.298,"15min":1.8169999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25},"max":{"1min":6.0570000000000004,"5min":61.970999999999997,"15min":61.970999999999997},"last":5.4400000000000004}]},{"osd":2,"last update":"Fri Feb 9 17:57:42 2024","interfaces":[{"interface":"back","average":{"1min":1.573,"5min":1.6619999999999999,"15min":1.782},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.26900000000000002},"max":{"1min":6.335,"5min":9.8689999999999998,"15min":10.657999999999999},"last":2.5710000000000002},{"interface":"front","average":{"1min":2.3050000000000002,"5min":1.9419999999999999,"15min":1.7270000000000001},"min":{"1min":0.373,"5min":0.29499999999999998,"15min":0.248},"max":{"1min":9.9239999999999995,"5min":11.35,"15min":11.35},"last":5.468}]},{"osd":3,"last update":"Fri Feb 9 17:56:32 2024","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":2.1299999999999999,"15min":1.496},"min":{"1min":0.35799999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.5289999999999999,"5min":10.537000000000001,"15min":10.537000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.86599999999999999,"5min":2.258,"15min":1.1659999999999999},"min":{"1min":0.40999999999999998,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":2.891,"5min":10.388,"15min":10.388},"last":0.53400000000000003}]},{"osd":4,"last update":"Fri Feb 9 17:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.734,"5min":1.4159999999999999,"15min":1.54},"min":{"1min":0.27300000000000002,"5min":0.26900000000000002,"15min":0.26800000000000002},"max":{"1min":6.1210000000000004,"5min":19.414000000000001,"15min":19.414000000000001},"last":5.5650000000000004},{"interface":"front","average":{"1min":1.5249999999999999,"5min":1.3939999999999999,"15min":1.3080000000000001},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.26900000000000002},"max":{"1min":8.3599999999999994,"5min":19.224,"15min":19.224},"last":4.1459999999999999}]},{"osd":5,"last update":"Fri Feb 9 17:58:18 2024","interfaces":[{"interface":"back","average":{"1min":1.708,"5min":1.298,"15min":1.6990000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.26900000000000002},"max":{"1min":10.047000000000001,"5min":10.047000000000001,"15min":77.924000000000007},"last":5.4960000000000004},{"interface":"front","average":{"1min":2.0299999999999998,"5min":1.3600000000000001,"15min":1.637},"min":{"1min":0.33500000000000002,"5min":0.316,"15min":0.28000000000000003},"max":{"1min":6.1479999999999997,"5min":7.4909999999999997,"15min":77.899000000000001},"last":5.6319999999999997}]},{"osd":6,"last update":"Fri Feb 9 17:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.8600000000000001,"5min":1.5509999999999999,"15min":1.53},"min":{"1min":0.40100000000000002,"5min":0.35999999999999999,"15min":0.30199999999999999},"max":{"1min":10.071,"5min":19.375,"15min":19.375},"last":2.7519999999999998},{"interface":"front","average":{"1min":1.839,"5min":1.583,"15min":1.702},"min":{"1min":0.42699999999999999,"5min":0.39600000000000002,"15min":0.33500000000000002},"max":{"1min":9.9900000000000002,"5min":19.178000000000001,"15min":19.178000000000001},"last":2.73}]},{"osd":7,"last update":"Fri Feb 9 17:57:42 2024","interfaces":[{"interface":"back","average":{"1min":2.1480000000000001,"5min":2.1480000000000001,"15min":2.1480000000000001},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":10.021000000000001,"5min":10.021000000000001,"15min":10.021000000000001},"last":2.6789999999999998},{"interface":"front","average":{"1min":1.9570000000000001,"5min":1.9570000000000001,"15min":1.9570000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":11.138,"5min":11.138,"15min":11.138},"last":5.6079999999999997}]},{"osd":8,"last update":"Fri Feb 9 17:58:23 2024","interfaces":[{"interface":"back","average":{"1min":3.0649999999999999,"5min":1.6870000000000001,"15min":1.331},"min":{"1min":0.45100000000000001,"5min":0.308,"15min":0.29599999999999999},"max":{"1min":10.097,"5min":10.097,"15min":10.744},"last":2.629},{"interface":"front","average":{"1min":3.137,"5min":1.776,"15min":1.486},"min":{"1min":0.53100000000000003,"5min":0.35999999999999999,"15min":0.32200000000000001},"max":{"1min":10.119,"5min":10.119,"15min":12.885999999999999},"last":2.7010000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:57:32 2024","interfaces":[{"interface":"back","average":{"1min":2.964,"5min":1.931,"15min":1.847},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.312},"max":{"1min":8.6969999999999992,"5min":10.333,"15min":20.773},"last":4.0919999999999996},{"interface":"front","average":{"1min":4.0999999999999996,"5min":1.998,"15min":1.5609999999999999},"min":{"1min":0.43099999999999999,"5min":0.35999999999999999,"15min":0.34300000000000003},"max":{"1min":18.579999999999998,"5min":18.579999999999998,"15min":20.637},"last":6.7789999999999999}]},{"osd":11,"last update":"Fri Feb 9 17:57:32 2024","interfaces":[{"interface":"back","average":{"1min":2.738,"5min":1.917,"15min":1.617},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.33000000000000002},"max":{"1min":11.195,"5min":11.324,"15min":11.324},"last":2.605},{"interface":"front","average":{"1min":3.133,"5min":2.1019999999999999,"15min":1.5800000000000001},"min":{"1min":0.624,"5min":0.41599999999999998,"15min":0.36499999999999999},"max":{"1min":11.737,"5min":11.737,"15min":11.737},"last":6.8120000000000003}]}]},{"osd":8,"up_from":171,"seq":734439407980,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1072188,"kb_used_data":593652,"kb_used_omap":212,"kb_used_meta":478315,"kb_avail":92672964,"statfs":{"total":95995035648,"available":94897115136,"internally_reserved":0,"allocated":607899648,"data_stored":607626605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217579,"internal_metadata":489795093},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:58:18 2024","interfaces":[{"interface":"back","average":{"1min":1.9039999999999999,"5min":1.946,"15min":1.49},"min":{"1min":0.249,"5min":0.249,"15min":0.224},"max":{"1min":7.79,"5min":10.832000000000001,"15min":12.712999999999999},"last":1.6639999999999999},{"interface":"front","average":{"1min":1.2,"5min":1.448,"15min":1.4119999999999999},"min":{"1min":0.252,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":4.319,"5min":9.25,"15min":12.725},"last":2.7879999999999998}]},{"osd":1,"last update":"Fri Feb 9 17:58:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.9399999999999999,"15min":1.5149999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":4.5,"5min":8.2550000000000008,"15min":8.718},"last":1.919},{"interface":"front","average":{"1min":2.04,"5min":2.0899999999999999,"15min":1.6899999999999999},"min":{"1min":0.30199999999999999,"5min":0.27100000000000002,"15min":0.24099999999999999},"max":{"1min":9.9900000000000002,"5min":21.399000000000001,"15min":21.399000000000001},"last":2.726}]},{"osd":2,"last update":"Fri Feb 9 17:58:18 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.5169999999999999,"15min":1.238},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":4.5670000000000002,"5min":6.9260000000000002,"15min":6.9260000000000002},"last":4.4530000000000003},{"interface":"front","average":{"1min":1.639,"5min":1.7829999999999999,"15min":1.7609999999999999},"min":{"1min":0.29999999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":7.7149999999999999,"5min":10.026999999999999,"15min":10.026999999999999},"last":2.7639999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:56:20 2024","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":2.552,"15min":2.8039999999999998},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":5.2560000000000002,"5min":15.654,"15min":15.654},"last":6.5350000000000001},{"interface":"front","average":{"1min":1.306,"5min":2.3149999999999999,"15min":2.0150000000000001},"min":{"1min":0.34300000000000003,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":5.2759999999999998,"5min":15.609999999999999,"15min":15.609999999999999},"last":6.5599999999999996}]},{"osd":4,"last update":"Fri Feb 9 17:58:18 2024","interfaces":[{"interface":"back","average":{"1min":2.0550000000000002,"5min":1.282,"15min":1.3149999999999999},"min":{"1min":0.36599999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":10.186999999999999,"5min":10.186999999999999,"15min":10.186999999999999},"last":2.9020000000000001},{"interface":"front","average":{"1min":2.1190000000000002,"5min":1.337,"15min":1.4650000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28000000000000003},"max":{"1min":10.061999999999999,"5min":10.061999999999999,"15min":10.061999999999999},"last":2.8119999999999998}]},{"osd":5,"last update":"Fri Feb 9 17:57:39 2024","interfaces":[{"interface":"back","average":{"1min":1.794,"5min":1.411,"15min":1.5609999999999999},"min":{"1min":0.312,"5min":0.28399999999999997,"15min":0.27400000000000002},"max":{"1min":6.2690000000000001,"5min":9.3550000000000004,"15min":9.657},"last":2.8799999999999999},{"interface":"front","average":{"1min":1.619,"5min":1.298,"15min":1.401},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.29899999999999999},"max":{"1min":10.093,"5min":10.093,"15min":10.093},"last":2.8580000000000001}]},{"osd":6,"last update":"Fri Feb 9 17:58:18 2024","interfaces":[{"interface":"back","average":{"1min":1.462,"5min":1.4099999999999999,"15min":1.629},"min":{"1min":0.34300000000000003,"5min":0.32700000000000001,"15min":0.29899999999999999},"max":{"1min":4.5380000000000003,"5min":9.7539999999999996,"15min":13.031000000000001},"last":1.893},{"interface":"front","average":{"1min":1.484,"5min":1.3819999999999999,"15min":1.5900000000000001},"min":{"1min":0.41899999999999998,"5min":0.311,"15min":0.311},"max":{"1min":6.1849999999999996,"5min":6.9560000000000004,"15min":12.157999999999999},"last":1.7330000000000001}]},{"osd":7,"last update":"Fri Feb 9 17:58:18 2024","interfaces":[{"interface":"back","average":{"1min":2.1070000000000002,"5min":1.581,"15min":1.76},"min":{"1min":0.38100000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":10.157999999999999,"5min":10.157999999999999,"15min":11.278},"last":2.8340000000000001},{"interface":"front","average":{"1min":1.671,"5min":1.478,"15min":1.712},"min":{"1min":0.40600000000000003,"5min":0.309,"15min":0.309},"max":{"1min":6.218,"5min":9.7110000000000003,"15min":11.313000000000001},"last":1.756}]},{"osd":9,"last update":"Fri Feb 9 17:58:18 2024","interfaces":[{"interface":"back","average":{"1min":3.0070000000000001,"5min":2.1389999999999998,"15min":1.883},"min":{"1min":0.39400000000000002,"5min":0.32700000000000001,"15min":0.26800000000000002},"max":{"1min":9.1280000000000001,"5min":10.483000000000001,"15min":45.616},"last":5.1360000000000001},{"interface":"front","average":{"1min":4.117,"5min":2.2320000000000002,"15min":1.978},"min":{"1min":0.41599999999999998,"5min":0.39000000000000001,"15min":0.314},"max":{"1min":12.351000000000001,"5min":12.351000000000001,"15min":45.718000000000004},"last":0.61899999999999999}]},{"osd":10,"last update":"Fri Feb 9 17:58:18 2024","interfaces":[{"interface":"back","average":{"1min":3.718,"5min":2.2450000000000001,"15min":1.5569999999999999},"min":{"1min":0.54700000000000004,"5min":0.34799999999999998,"15min":0.32300000000000001},"max":{"1min":10.128,"5min":16.356000000000002,"15min":16.356000000000002},"last":1.8540000000000001},{"interface":"front","average":{"1min":3.2050000000000001,"5min":2.331,"15min":1.8009999999999999},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.34300000000000003},"max":{"1min":8.5869999999999997,"5min":16.309999999999999,"15min":16.309999999999999},"last":2.077}]},{"osd":11,"last update":"Fri Feb 9 17:57:49 2024","interfaces":[{"interface":"back","average":{"1min":1.95,"5min":1.95,"15min":1.95},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":5.5739999999999998,"5min":5.5739999999999998,"15min":5.5739999999999998},"last":1.778},{"interface":"front","average":{"1min":1.9870000000000001,"5min":1.9870000000000001,"15min":1.9870000000000001},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":5.6120000000000001,"5min":5.6120000000000001,"15min":5.6120000000000001},"last":1.8109999999999999}]}]},{"osd":7,"up_from":57,"seq":244813136326,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1141332,"kb_used_data":384076,"kb_used_omap":1,"kb_used_meta":757246,"kb_avail":92603820,"statfs":{"total":95995035648,"available":94826311680,"internally_reserved":0,"allocated":393293824,"data_stored":393008729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":775420362},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:57:50 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":1.079,"15min":1.6870000000000001},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.249},"max":{"1min":4.3769999999999998,"5min":7.032,"15min":101.167},"last":0.371},{"interface":"front","average":{"1min":0.51800000000000002,"5min":1.0369999999999999,"15min":1.6160000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.24099999999999999},"max":{"1min":2.1829999999999998,"5min":7.0170000000000003,"15min":101.07599999999999},"last":0.51400000000000001}]},{"osd":1,"last update":"Fri Feb 9 17:57:50 2024","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":1.498,"15min":1.635},"min":{"1min":0.30299999999999999,"5min":0.28299999999999997,"15min":0.24299999999999999},"max":{"1min":8.7050000000000001,"5min":17.314,"15min":17.314},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.39000000000000001,"5min":1.1799999999999999,"15min":1.6659999999999999},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.65900000000000003,"5min":7.5949999999999998,"15min":12.359},"last":0.39100000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:57:50 2024","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":1.268,"15min":1.48},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.25},"max":{"1min":2.9359999999999999,"5min":9.8279999999999994,"15min":9.8279999999999994},"last":0.54500000000000004},{"interface":"front","average":{"1min":1.194,"5min":1.359,"15min":1.6619999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.29499999999999998},"max":{"1min":8.5690000000000008,"5min":8.5690000000000008,"15min":11.968},"last":0.48599999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:56:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":1.7529999999999999,"15min":2.1960000000000002},"min":{"1min":0.41599999999999998,"5min":0.32800000000000001,"15min":0.25600000000000001},"max":{"1min":2.0179999999999998,"5min":11.112,"15min":78.644999999999996},"last":1.72},{"interface":"front","average":{"1min":0.68200000000000005,"5min":1.7130000000000001,"15min":2.3170000000000002},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.33800000000000002},"max":{"1min":2.0830000000000002,"5min":14.592000000000001,"15min":78.414000000000001},"last":3.1070000000000002}]},{"osd":4,"last update":"Fri Feb 9 17:57:50 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.61399999999999999,"15min":1.385},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.25600000000000001},"max":{"1min":4.3330000000000002,"5min":4.3330000000000002,"15min":86.813000000000002},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.54600000000000004,"15min":1.5449999999999999},"min":{"1min":0.26600000000000001,"5min":0.26300000000000001,"15min":0.23899999999999999},"max":{"1min":1.202,"5min":2.028,"15min":86.751000000000005},"last":0.52800000000000002}]},{"osd":5,"last update":"Fri Feb 9 17:57:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.65600000000000003,"15min":1.3420000000000001},"min":{"1min":0.313,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.91400000000000003,"5min":4.0469999999999997,"15min":9.6590000000000007},"last":0.499},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.72099999999999997,"15min":1.5269999999999999},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.31900000000000001},"max":{"1min":1.3220000000000001,"5min":4.0679999999999996,"15min":15.228999999999999},"last":0.44600000000000001}]},{"osd":6,"last update":"Fri Feb 9 17:58:16 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.79100000000000004,"15min":1.5369999999999999},"min":{"1min":0.40300000000000002,"5min":0.311,"15min":0.311},"max":{"1min":5.0220000000000002,"5min":5.8550000000000004,"15min":15.275},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.69399999999999995,"15min":1.3300000000000001},"min":{"1min":0.432,"5min":0.35899999999999999,"15min":0.29199999999999998},"max":{"1min":2.6230000000000002,"5min":2.6230000000000002,"15min":18.236999999999998},"last":0.41399999999999998}]},{"osd":8,"last update":"Fri Feb 9 17:58:12 2024","interfaces":[{"interface":"back","average":{"1min":2.0640000000000001,"5min":1.3660000000000001,"15min":1.494},"min":{"1min":0.46200000000000002,"5min":0.43099999999999999,"15min":0.375},"max":{"1min":5.7729999999999997,"5min":7.5410000000000004,"15min":11.81},"last":11.648999999999999},{"interface":"front","average":{"1min":1.8340000000000001,"5min":1.385,"15min":1.5509999999999999},"min":{"1min":0.42399999999999999,"5min":0.375,"15min":0.37},"max":{"1min":5.79,"5min":7.532,"15min":12.058},"last":11.629}]},{"osd":9,"last update":"Fri Feb 9 17:58:02 2024","interfaces":[{"interface":"back","average":{"1min":1.7150000000000001,"5min":1.3540000000000001,"15min":1.6759999999999999},"min":{"1min":0.51400000000000001,"5min":0.44800000000000001,"15min":0.375},"max":{"1min":4.9379999999999997,"5min":5.5119999999999996,"15min":11.941000000000001},"last":1.127},{"interface":"front","average":{"1min":2.302,"5min":1.4199999999999999,"15min":1.7509999999999999},"min":{"1min":0.53000000000000003,"5min":0.40100000000000002,"15min":0.40000000000000002},"max":{"1min":8.0909999999999993,"5min":9.8670000000000009,"15min":15.852},"last":0.58499999999999996}]},{"osd":10,"last update":"Fri Feb 9 17:57:30 2024","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.304,"15min":1.4410000000000001},"min":{"1min":0.48899999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":3.0139999999999998,"5min":8.4969999999999999,"15min":9.5350000000000001},"last":8.3089999999999993},{"interface":"front","average":{"1min":1.8220000000000001,"5min":1.552,"15min":1.829},"min":{"1min":0.64500000000000002,"5min":0.40400000000000003,"15min":0.38900000000000001},"max":{"1min":4.9249999999999998,"5min":7.423,"15min":11.878},"last":0.84199999999999997}]},{"osd":11,"last update":"Fri Feb 9 17:57:50 2024","interfaces":[{"interface":"back","average":{"1min":1.5249999999999999,"5min":1.1859999999999999,"15min":1.5640000000000001},"min":{"1min":0.49199999999999999,"5min":0.375,"15min":0.35999999999999999},"max":{"1min":4.8040000000000003,"5min":5.0650000000000004,"15min":11.961},"last":2.1800000000000002},{"interface":"front","average":{"1min":1.732,"5min":1.569,"15min":1.7050000000000001},"min":{"1min":0.58099999999999996,"5min":0.48999999999999999,"15min":0.438},"max":{"1min":6.1870000000000003,"5min":12.243,"15min":12.423999999999999},"last":2.0910000000000002}]}]},{"osd":6,"up_from":49,"seq":210453397965,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":681932,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":93063220,"statfs":{"total":95995035648,"available":95296737280,"internally_reserved":0,"allocated":1048576,"data_stored":771105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":697235910},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:57:37 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":1.1539999999999999,"15min":1.4730000000000001},"min":{"1min":0.28399999999999997,"5min":0.23999999999999999,"15min":0.22800000000000001},"max":{"1min":4.5270000000000001,"5min":10.356,"15min":10.481999999999999},"last":0.63},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.98299999999999998,"15min":1.532},"min":{"1min":0.307,"5min":0.28000000000000003,"15min":0.24399999999999999},"max":{"1min":3.137,"5min":8.9459999999999997,"15min":10.436},"last":0.60299999999999998}]},{"osd":1,"last update":"Fri Feb 9 17:57:37 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.0880000000000001,"15min":2.1899999999999999},"min":{"1min":0.32000000000000001,"5min":0.27600000000000002,"15min":0.25},"max":{"1min":3.9940000000000002,"5min":9.3200000000000003,"15min":79.945999999999998},"last":0.65000000000000002},{"interface":"front","average":{"1min":1.276,"5min":1.1459999999999999,"15min":2.3130000000000002},"min":{"1min":0.41499999999999998,"5min":0.35799999999999998,"15min":0.28599999999999998},"max":{"1min":4.0170000000000003,"5min":9.3510000000000009,"15min":80.003},"last":0.67700000000000005}]},{"osd":2,"last update":"Fri Feb 9 17:57:37 2024","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":1.1419999999999999,"15min":2.0710000000000002},"min":{"1min":0.38800000000000001,"5min":0.31900000000000001,"15min":0.28100000000000003},"max":{"1min":4.0259999999999998,"5min":5.657,"15min":84.912999999999997},"last":0.747},{"interface":"front","average":{"1min":0.93400000000000005,"5min":1.1799999999999999,"15min":2.1120000000000001},"min":{"1min":0.47399999999999998,"5min":0.34999999999999998,"15min":0.34000000000000002},"max":{"1min":4.0540000000000003,"5min":5.6849999999999996,"15min":84.942999999999998},"last":0.77400000000000002}]},{"osd":3,"last update":"Fri Feb 9 17:55:07 2024","interfaces":[{"interface":"back","average":{"1min":1.8240000000000001,"5min":1.5389999999999999,"15min":1.4910000000000001},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":9.0129999999999999,"5min":9.0129999999999999,"15min":9.0129999999999999},"last":0.54500000000000004},{"interface":"front","average":{"1min":1.841,"5min":1.8169999999999999,"15min":1.8129999999999999},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":6.6769999999999996,"5min":8.9410000000000007,"15min":8.9410000000000007},"last":0.48399999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:57:37 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.62,"15min":1.631},"min":{"1min":0.28999999999999998,"5min":0.26600000000000001,"15min":0.26200000000000001},"max":{"1min":3.0299999999999998,"5min":8.1349999999999998,"15min":15.061999999999999},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.68500000000000005,"15min":1.7410000000000001},"min":{"1min":0.34100000000000003,"5min":0.33200000000000002,"15min":0.29299999999999998},"max":{"1min":3.0649999999999999,"5min":8.1809999999999992,"15min":15.118},"last":0.46000000000000002}]},{"osd":5,"last update":"Fri Feb 9 17:57:37 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.67200000000000004,"15min":1.6930000000000001},"min":{"1min":0.312,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80600000000000005,"5min":6.5800000000000001,"15min":14.151999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.623,"15min":1.6599999999999999},"min":{"1min":0.378,"5min":0.35699999999999998,"15min":0.26100000000000001},"max":{"1min":0.83199999999999996,"5min":3.8450000000000002,"15min":18.315999999999999},"last":0.70899999999999996}]},{"osd":7,"last update":"Fri Feb 9 17:58:15 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.74399999999999999,"15min":1.5509999999999999},"min":{"1min":0.374,"5min":0.31,"15min":0.28699999999999998},"max":{"1min":3.0339999999999998,"5min":5.4969999999999999,"15min":23.265999999999998},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.73299999999999998,"15min":1.357},"min":{"1min":0.35799999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.95299999999999996,"5min":5.0670000000000002,"15min":12.967000000000001},"last":0.41799999999999998}]},{"osd":8,"last update":"Fri Feb 9 17:57:30 2024","interfaces":[{"interface":"back","average":{"1min":2.25,"5min":1.2929999999999999,"15min":1.1459999999999999},"min":{"1min":0.36799999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":6.5620000000000003,"5min":8.1500000000000004,"15min":9.9480000000000004},"last":0.68999999999999995},{"interface":"front","average":{"1min":2.3220000000000001,"5min":1.4350000000000001,"15min":1.2},"min":{"1min":0.503,"5min":0.41399999999999998,"15min":0.40400000000000003},"max":{"1min":6.548,"5min":8.2490000000000006,"15min":11.316000000000001},"last":1.1200000000000001}]},{"osd":9,"last update":"Fri Feb 9 17:57:47 2024","interfaces":[{"interface":"back","average":{"1min":1.8069999999999999,"5min":1.74,"15min":1.7789999999999999},"min":{"1min":0.52400000000000002,"5min":0.41799999999999998,"15min":0.32600000000000001},"max":{"1min":7.6669999999999998,"5min":36.606000000000002,"15min":36.606000000000002},"last":1.9339999999999999},{"interface":"front","average":{"1min":2.0329999999999999,"5min":1.974,"15min":1.956},"min":{"1min":0.59799999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":4.7629999999999999,"5min":34.408000000000001,"15min":34.408000000000001},"last":2.98}]},{"osd":10,"last update":"Fri Feb 9 17:58:15 2024","interfaces":[{"interface":"back","average":{"1min":3.0899999999999999,"5min":1.734,"15min":1.857},"min":{"1min":0.441,"5min":0.36599999999999999,"15min":0.34399999999999997},"max":{"1min":9.8460000000000001,"5min":21.364999999999998,"15min":25.370000000000001},"last":0.54500000000000004},{"interface":"front","average":{"1min":3.129,"5min":1.8120000000000001,"15min":1.97},"min":{"1min":0.54200000000000004,"5min":0.40100000000000002,"15min":0.36399999999999999},"max":{"1min":9.7989999999999995,"5min":21.408999999999999,"15min":26.027999999999999},"last":0.58199999999999996}]},{"osd":11,"last update":"Fri Feb 9 17:57:37 2024","interfaces":[{"interface":"back","average":{"1min":1.7190000000000001,"5min":1.9379999999999999,"15min":1.9359999999999999},"min":{"1min":0.48399999999999999,"5min":0.44400000000000001,"15min":0.372},"max":{"1min":3.996,"5min":35.912999999999997,"15min":35.912999999999997},"last":0.50600000000000001},{"interface":"front","average":{"1min":1.754,"5min":1.9950000000000001,"15min":1.956},"min":{"1min":0.51900000000000002,"5min":0.48699999999999999,"15min":0.38800000000000001},"max":{"1min":4.0380000000000003,"5min":36.156999999999996,"15min":36.156999999999996},"last":0.61699999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575333,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2146508,"kb_used_data":1164352,"kb_used_omap":1,"kb_used_meta":982142,"kb_avail":91598644,"statfs":{"total":95995035648,"available":93797011456,"internally_reserved":0,"allocated":1192296448,"data_stored":1192018977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1005713861},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:57:54 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.9119999999999999,"15min":1.5780000000000001},"min":{"1min":0.248,"5min":0.23300000000000001,"15min":0.19},"max":{"1min":7.5259999999999998,"5min":14.885999999999999,"15min":14.885999999999999},"last":2.0339999999999998},{"interface":"front","average":{"1min":1.113,"5min":2.0489999999999999,"15min":1.792},"min":{"1min":0.30199999999999999,"5min":0.246,"15min":0.214},"max":{"1min":7.5780000000000003,"5min":15.411,"15min":33.389000000000003},"last":0.47399999999999998}]},{"osd":2,"last update":"Fri Feb 9 17:58:22 2024","interfaces":[{"interface":"back","average":{"1min":1.6910000000000001,"5min":1.702,"15min":1.8069999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.222},"max":{"1min":10.510999999999999,"5min":13.173,"15min":14.83},"last":0.66400000000000003},{"interface":"front","average":{"1min":1.385,"5min":1.585,"15min":1.5760000000000001},"min":{"1min":0.36799999999999999,"5min":0.28100000000000003,"15min":0.222},"max":{"1min":7.6070000000000002,"5min":11.103,"15min":11.103},"last":0.51400000000000001}]},{"osd":3,"last update":"Fri Feb 9 17:56:07 2024","interfaces":[{"interface":"back","average":{"1min":1.5009999999999999,"5min":3.1970000000000001,"15min":3.5990000000000002},"min":{"1min":0.28899999999999998,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":19.917999999999999,"5min":19.917999999999999,"15min":19.917999999999999},"last":1.2789999999999999},{"interface":"front","average":{"1min":1.7,"5min":3.6589999999999998,"15min":4.1470000000000002},"min":{"1min":0.313,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":20.175999999999998,"5min":20.175999999999998,"15min":20.175999999999998},"last":0.41599999999999998}]},{"osd":4,"last update":"Fri Feb 9 17:58:01 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":1.1060000000000001,"15min":1.524},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.26900000000000002},"max":{"1min":4.9660000000000002,"5min":9.032,"15min":45.137999999999998},"last":0.38},{"interface":"front","average":{"1min":1.321,"5min":1.8149999999999999,"15min":1.899},"min":{"1min":0.29799999999999999,"5min":0.27400000000000002,"15min":0.26400000000000001},"max":{"1min":7.7370000000000001,"5min":15.331,"15min":44.950000000000003},"last":3.3740000000000001}]},{"osd":5,"last update":"Fri Feb 9 17:57:32 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":1.4910000000000001,"15min":1.736},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.26000000000000001},"max":{"1min":2.319,"5min":15.106999999999999,"15min":45.171999999999997},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.95199999999999996,"5min":1.772,"15min":2.012},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":5.2370000000000001,"5min":15.355,"15min":45.127000000000002},"last":3.4169999999999998}]},{"osd":6,"last update":"Fri Feb 9 17:57:45 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":1.5600000000000001,"15min":1.909},"min":{"1min":0.34499999999999997,"5min":0.313,"15min":0.24299999999999999},"max":{"1min":3.0419999999999998,"5min":15.154999999999999,"15min":33.338000000000001},"last":0.5},{"interface":"front","average":{"1min":0.98999999999999999,"5min":1.619,"15min":1.9770000000000001},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":3.0920000000000001,"5min":15.253,"15min":33.286999999999999},"last":0.64500000000000002}]},{"osd":7,"last update":"Fri Feb 9 17:57:45 2024","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.6359999999999999,"15min":2.0579999999999998},"min":{"1min":0.40300000000000002,"5min":0.316,"15min":0.316},"max":{"1min":3.0680000000000001,"5min":15.287000000000001,"15min":45.740000000000002},"last":0.53200000000000003},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.694,"15min":2.1389999999999998},"min":{"1min":0.46899999999999997,"5min":0.45400000000000001,"15min":0.39600000000000002},"max":{"1min":3.117,"5min":15.311,"15min":45.792000000000002},"last":0.57199999999999995}]},{"osd":8,"last update":"Fri Feb 9 17:57:54 2024","interfaces":[{"interface":"back","average":{"1min":1.8100000000000001,"5min":1.6819999999999999,"15min":1.6160000000000001},"min":{"1min":0.46600000000000003,"5min":0.42699999999999999,"15min":0.35599999999999998},"max":{"1min":7.6299999999999999,"5min":9.1999999999999993,"15min":9.2520000000000007},"last":0.69099999999999995},{"interface":"front","average":{"1min":2.4350000000000001,"5min":1.9890000000000001,"15min":1.8240000000000001},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.35599999999999998},"max":{"1min":7.6600000000000001,"5min":12.464,"15min":12.69},"last":0.70999999999999996}]},{"osd":9,"last update":"Fri Feb 9 17:57:54 2024","interfaces":[{"interface":"back","average":{"1min":2.839,"5min":2.3540000000000001,"15min":1.8560000000000001},"min":{"1min":0.503,"5min":0.42799999999999999,"15min":0.34200000000000003},"max":{"1min":7.6849999999999996,"5min":20.789000000000001,"15min":20.789000000000001},"last":0.626},{"interface":"front","average":{"1min":2.4860000000000002,"5min":2.02,"15min":1.8500000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":7.2110000000000003,"5min":9.1210000000000004,"15min":9.5239999999999991},"last":3.4449999999999998}]},{"osd":10,"last update":"Fri Feb 9 17:57:32 2024","interfaces":[{"interface":"back","average":{"1min":1.694,"5min":1.9770000000000001,"15min":1.994},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40699999999999997},"max":{"1min":7.1740000000000004,"5min":15.211,"15min":33.213999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":1.8759999999999999,"5min":2.2280000000000002,"15min":1.944},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":7.202,"5min":15.271000000000001,"15min":15.271000000000001},"last":3.4769999999999999}]},{"osd":11,"last update":"Fri Feb 9 17:57:45 2024","interfaces":[{"interface":"back","average":{"1min":3.0910000000000002,"5min":2.5369999999999999,"15min":2.2629999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":14.935,"5min":15.301,"15min":19.648},"last":8},{"interface":"front","average":{"1min":3.613,"5min":2.9470000000000001,"15min":2.343},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":15.209,"5min":15.209,"15min":33.161000000000001},"last":0.67900000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738858,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1438204,"kb_used_data":580980,"kb_used_omap":1,"kb_used_meta":857214,"kb_avail":92306948,"statfs":{"total":95995035648,"available":94522314752,"internally_reserved":0,"allocated":594923520,"data_stored":594650477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":877787589},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 9 17:57:40 2024","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.696,"15min":1.4870000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.20599999999999999},"max":{"1min":5.2759999999999998,"5min":18.698,"15min":18.698},"last":0.253},{"interface":"front","average":{"1min":1.28,"5min":1.7769999999999999,"15min":1.5700000000000001},"min":{"1min":0.38300000000000001,"5min":0.32000000000000001,"15min":0.28299999999999997},"max":{"1min":5.3099999999999996,"5min":18.742999999999999,"15min":18.742999999999999},"last":0.29199999999999998}]},{"osd":2,"last update":"Fri Feb 9 17:58:05 2024","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.948,"15min":1.845},"min":{"1min":0.253,"5min":0.253,"15min":0.245},"max":{"1min":3.6099999999999999,"5min":20.013999999999999,"15min":20.013999999999999},"last":0.36799999999999999},{"interface":"front","average":{"1min":1.3109999999999999,"5min":2.0089999999999999,"15min":1.6619999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":6.1379999999999999,"5min":20.103999999999999,"15min":20.103999999999999},"last":0.57299999999999995}]},{"osd":3,"last update":"Fri Feb 9 17:56:22 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":2.327,"15min":1.7929999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.23100000000000001},"max":{"1min":3.198,"5min":20.469999999999999,"15min":36.052},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.88600000000000001,"5min":2.4249999999999998,"15min":2.1030000000000002},"min":{"1min":0.26800000000000002,"5min":0.23899999999999999,"15min":0.217},"max":{"1min":6.3380000000000001,"5min":21.920000000000002,"15min":38.383000000000003},"last":3.117}]},{"osd":4,"last update":"Fri Feb 9 17:58:08 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":1.4470000000000001,"15min":1.4199999999999999},"min":{"1min":0.29999999999999999,"5min":0.28899999999999998,"15min":0.27000000000000002},"max":{"1min":3.8119999999999998,"5min":12.757,"15min":12.757},"last":0.40999999999999998},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.792,"15min":1.821},"min":{"1min":0.312,"5min":0.25800000000000001,"15min":0.249},"max":{"1min":5.3090000000000002,"5min":17.091999999999999,"15min":17.091999999999999},"last":1.956}]},{"osd":5,"last update":"Fri Feb 9 17:58:23 2024","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":1.236,"15min":1.732},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.30299999999999999},"max":{"1min":3.8969999999999998,"5min":9.7929999999999993,"15min":48.924999999999997},"last":0.46400000000000002},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.3149999999999999,"15min":1.7949999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.29599999999999999},"max":{"1min":7.7489999999999997,"5min":9.8230000000000004,"15min":48.969999999999999},"last":0.439}]},{"osd":6,"last update":"Fri Feb 9 17:57:52 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.617,"15min":1.6970000000000001},"min":{"1min":0.35399999999999998,"5min":0.307,"15min":0.28699999999999998},"max":{"1min":6.2240000000000002,"5min":15.667,"15min":15.667},"last":0.47699999999999998},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.7050000000000001,"15min":1.8049999999999999},"min":{"1min":0.36399999999999999,"5min":0.34799999999999998,"15min":0.33300000000000002},"max":{"1min":6.2800000000000002,"5min":15.702999999999999,"15min":15.702999999999999},"last":0.54600000000000004}]},{"osd":7,"last update":"Fri Feb 9 17:58:19 2024","interfaces":[{"interface":"back","average":{"1min":1.5629999999999999,"5min":1.8480000000000001,"15min":1.8109999999999999},"min":{"1min":0.35499999999999998,"5min":0.307,"15min":0.29199999999999998},"max":{"1min":5.3460000000000001,"5min":11.436,"15min":15.271000000000001},"last":2},{"interface":"front","average":{"1min":1.6100000000000001,"5min":1.7250000000000001,"15min":1.9139999999999999},"min":{"1min":0.39400000000000002,"5min":0.32600000000000001,"15min":0.28299999999999997},"max":{"1min":5.3979999999999997,"5min":17.152999999999999,"15min":17.152999999999999},"last":1.9259999999999999}]},{"osd":8,"last update":"Fri Feb 9 17:58:23 2024","interfaces":[{"interface":"back","average":{"1min":1.728,"5min":2.157,"15min":1.956},"min":{"1min":0.39900000000000002,"5min":0.34699999999999998,"15min":0.313},"max":{"1min":5.3710000000000004,"5min":17.189,"15min":17.189},"last":0.52100000000000002},{"interface":"front","average":{"1min":2.3010000000000002,"5min":2.3660000000000001,"15min":1.9099999999999999},"min":{"1min":0.44900000000000001,"5min":0.36499999999999999,"15min":0.35099999999999998},"max":{"1min":7.7169999999999996,"5min":18.600000000000001,"15min":18.600000000000001},"last":2.4049999999999998}]},{"osd":9,"last update":"Fri Feb 9 17:58:05 2024","interfaces":[{"interface":"back","average":{"1min":2.254,"5min":2.6190000000000002,"15min":2.3319999999999999},"min":{"1min":0.40999999999999998,"5min":0.39000000000000001,"15min":0.38300000000000001},"max":{"1min":5.4489999999999998,"5min":17.216000000000001,"15min":24.757000000000001},"last":3.2440000000000002},{"interface":"front","average":{"1min":2.0910000000000002,"5min":2.129,"15min":2.1190000000000002},"min":{"1min":0.41899999999999998,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":5.5810000000000004,"5min":19.986999999999998,"15min":24.681999999999999},"last":1.7789999999999999}]},{"osd":10,"last update":"Fri Feb 9 17:58:23 2024","interfaces":[{"interface":"back","average":{"1min":2.7000000000000002,"5min":2.984,"15min":2.2810000000000001},"min":{"1min":0.438,"5min":0.42499999999999999,"15min":0.40699999999999997},"max":{"1min":7.8029999999999999,"5min":57.979999999999997,"15min":57.979999999999997},"last":2.766},{"interface":"front","average":{"1min":2.7869999999999999,"5min":3.141,"15min":2.3100000000000001},"min":{"1min":0.42099999999999999,"5min":0.41299999999999998,"15min":0.34200000000000003},"max":{"1min":7.8010000000000002,"5min":57.606999999999999,"15min":57.606999999999999},"last":5.9530000000000003}]},{"osd":11,"last update":"Fri Feb 9 17:57:52 2024","interfaces":[{"interface":"back","average":{"1min":1.99,"5min":2.6640000000000001,"15min":2.1360000000000001},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.41599999999999998},"max":{"1min":5.5439999999999996,"5min":58.351999999999997,"15min":58.351999999999997},"last":2.8900000000000001},{"interface":"front","average":{"1min":2.0979999999999999,"5min":2.7240000000000002,"15min":2.1920000000000002},"min":{"1min":0.51800000000000002,"5min":0.441,"15min":0.441},"max":{"1min":6.0099999999999998,"5min":57.314,"15min":57.314},"last":2.9279999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411807,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1596312,"kb_used_data":969548,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":92148840,"statfs":{"total":95995035648,"available":94360412160,"internally_reserved":0,"allocated":992817152,"data_stored":992532057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[0,1,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:57:44 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":3.629,"5min":3.629,"15min":3.629},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":3.6509999999999998,"5min":3.6509999999999998,"15min":3.6509999999999998},"last":0.60599999999999998}]},{"osd":1,"last update":"Fri Feb 9 17:58:19 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":1.903,"15min":1.9950000000000001},"min":{"1min":0.247,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":3.835,"5min":15.378,"15min":52.646000000000001},"last":1.389},{"interface":"front","average":{"1min":0.97499999999999998,"5min":1.8879999999999999,"15min":2.0059999999999998},"min":{"1min":0.30099999999999999,"5min":0.248,"15min":0.248},"max":{"1min":3.8679999999999999,"5min":15.092000000000001,"15min":52.683999999999997},"last":1.4099999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:55:40 2024","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":2.8069999999999999,"15min":2.1920000000000002},"min":{"1min":0.29799999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":8.7200000000000006,"5min":46.927999999999997,"15min":46.927999999999997},"last":4.415},{"interface":"front","average":{"1min":0.92000000000000004,"5min":2.6110000000000002,"15min":2.1970000000000001},"min":{"1min":0.28000000000000003,"5min":0.192,"15min":0.192},"max":{"1min":5.8140000000000001,"5min":45.798999999999999,"15min":45.798999999999999},"last":4.0940000000000003}]},{"osd":4,"last update":"Fri Feb 9 17:57:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":1.2190000000000001,"15min":1.5009999999999999},"min":{"1min":0.35699999999999998,"5min":0.245,"15min":0.245},"max":{"1min":3.516,"5min":10.365,"15min":12.398},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.56699999999999995,"5min":1.2849999999999999,"15min":1.5089999999999999},"min":{"1min":0.38200000000000001,"5min":0.26600000000000001,"15min":0.25700000000000001},"max":{"1min":2.4529999999999998,"5min":10.529,"15min":10.888999999999999},"last":0.55600000000000005}]},{"osd":5,"last update":"Fri Feb 9 17:57:38 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":1.5009999999999999,"15min":2.0209999999999999},"min":{"1min":0.29599999999999999,"5min":0.245,"15min":0.245},"max":{"1min":2.5870000000000002,"5min":12.305999999999999,"15min":42.664000000000001},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.82199999999999995,"5min":1.3149999999999999,"15min":2.105},"min":{"1min":0.30399999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":4.8150000000000004,"5min":10.603,"15min":43.311999999999998},"last":0.58999999999999997}]},{"osd":6,"last update":"Fri Feb 9 17:57:49 2024","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":1.502,"15min":1.8899999999999999},"min":{"1min":0.36299999999999999,"5min":0.29299999999999998,"15min":0.28399999999999997},"max":{"1min":4.7949999999999999,"5min":10.573,"15min":18.518999999999998},"last":0.625},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.5600000000000001,"15min":1.8879999999999999},"min":{"1min":0.317,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":4.8570000000000002,"5min":10.731,"15min":18.481999999999999},"last":0.64700000000000002}]},{"osd":7,"last update":"Fri Feb 9 17:58:19 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":1.3069999999999999,"15min":1.7669999999999999},"min":{"1min":0.33900000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":2.4140000000000001,"5min":10.403,"15min":16.666},"last":0.503},{"interface":"front","average":{"1min":1.21,"5min":1.3129999999999999,"15min":1.879},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":4.7510000000000003,"5min":10.316000000000001,"15min":16.638999999999999},"last":0.67000000000000004}]},{"osd":8,"last update":"Fri Feb 9 17:57:49 2024","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.956,"15min":1.79},"min":{"1min":0.40699999999999997,"5min":0.35599999999999998,"15min":0.316},"max":{"1min":10.095000000000001,"5min":10.755000000000001,"15min":12.500999999999999},"last":1.361},{"interface":"front","average":{"1min":1.6879999999999999,"5min":1.7929999999999999,"15min":1.984},"min":{"1min":0.40400000000000003,"5min":0.35699999999999998,"15min":0.34399999999999997},"max":{"1min":4.8339999999999996,"5min":10.629,"15min":54.759999999999998},"last":1.1359999999999999}]},{"osd":9,"last update":"Fri Feb 9 17:57:53 2024","interfaces":[{"interface":"back","average":{"1min":1.702,"5min":2.1739999999999999,"15min":2.2000000000000002},"min":{"1min":0.372,"5min":0.318,"15min":0.318},"max":{"1min":9.2240000000000002,"5min":12.276,"15min":53.935000000000002},"last":5.069},{"interface":"front","average":{"1min":1.9590000000000001,"5min":2.1499999999999999,"15min":2.1349999999999998},"min":{"1min":0.48199999999999998,"5min":0.36199999999999999,"15min":0.34899999999999998},"max":{"1min":9.1989999999999998,"5min":10.657,"15min":54.670999999999999},"last":5.0880000000000001}]},{"osd":10,"last update":"Fri Feb 9 17:57:30 2024","interfaces":[{"interface":"back","average":{"1min":2.254,"5min":2.0710000000000002,"15min":1.9710000000000001},"min":{"1min":0.56599999999999995,"5min":0.40799999999999997,"15min":0.33500000000000002},"max":{"1min":7.1529999999999996,"5min":12.313000000000001,"15min":54.039999999999999},"last":5.0140000000000002},{"interface":"front","average":{"1min":2.2789999999999999,"5min":2.1030000000000002,"15min":2.0099999999999998},"min":{"1min":0.58399999999999996,"5min":0.44500000000000001,"15min":0.42599999999999999},"max":{"1min":7.2300000000000004,"5min":12.343999999999999,"15min":54.118000000000002},"last":5.0410000000000004}]},{"osd":11,"last update":"Fri Feb 9 17:57:49 2024","interfaces":[{"interface":"back","average":{"1min":1.958,"5min":1.9830000000000001,"15min":2.1899999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.40600000000000003},"max":{"1min":4.891,"5min":10.702,"15min":54.799999999999997},"last":0.96299999999999997},{"interface":"front","average":{"1min":1.403,"5min":2.1120000000000001,"15min":2.1859999999999999},"min":{"1min":0.40600000000000003,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":5.4109999999999996,"5min":12.243,"15min":54.103000000000002},"last":3.4990000000000001}]}]},{"osd":3,"up_from":27,"seq":115964117446,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2157336,"kb_used_data":1318156,"kb_used_omap":1,"kb_used_meta":839166,"kb_avail":91587816,"statfs":{"total":95995035648,"available":93785923584,"internally_reserved":0,"allocated":1349791744,"data_stored":1349509795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":859306439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:56:33 2024","interfaces":[{"interface":"back","average":{"1min":2.012,"5min":2.4329999999999998,"15min":1.9950000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.221},"max":{"1min":10.628,"5min":17.006,"15min":17.006},"last":0.27200000000000002},{"interface":"front","average":{"1min":1.724,"5min":1.821,"15min":2.028},"min":{"1min":0.252,"5min":0.24099999999999999,"15min":0.23400000000000001},"max":{"1min":6.3860000000000001,"5min":12.288,"15min":16.637},"last":0.40200000000000002}]},{"osd":1,"last update":"Fri Feb 9 17:56:03 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":1.8460000000000001,"15min":2.512},"min":{"1min":0.23499999999999999,"5min":0.221,"15min":0.221},"max":{"1min":2.0779999999999998,"5min":12.609,"15min":53.152999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.68000000000000005,"5min":1.4379999999999999,"15min":1.859},"min":{"1min":0.29799999999999999,"5min":0.24299999999999999,"15min":0.23200000000000001},"max":{"1min":2.1419999999999999,"5min":12.237,"15min":53.118000000000002},"last":0.54900000000000004}]},{"osd":2,"last update":"Fri Feb 9 17:56:33 2024","interfaces":[{"interface":"back","average":{"1min":1.609,"5min":2.169,"15min":1.913},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":10.353999999999999,"5min":17.477,"15min":17.477},"last":0.35499999999999998},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.9039999999999999,"15min":1.839},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":3.0750000000000002,"5min":21.103000000000002,"15min":21.103000000000002},"last":0.36799999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:56:33 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":1.413,"15min":1.73},"min":{"1min":0.28999999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":2.907,"5min":12.718,"15min":12.718},"last":0.71999999999999997},{"interface":"front","average":{"1min":1.532,"5min":1.2969999999999999,"15min":1.6259999999999999},"min":{"1min":0.316,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":6.157,"5min":6.157,"15min":19.713999999999999},"last":0.58399999999999996}]},{"osd":5,"last update":"Fri Feb 9 17:56:17 2024","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.7889999999999999,"15min":1.98},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.28000000000000003},"max":{"1min":8.1920000000000002,"5min":13.321,"15min":14.106},"last":0.317},{"interface":"front","average":{"1min":1.4399999999999999,"5min":1.8440000000000001,"15min":1.968},"min":{"1min":0.44,"5min":0.44,"15min":0.316},"max":{"1min":8.3789999999999996,"5min":13.347,"15min":14.138999999999999},"last":0.48899999999999999}]},{"osd":6,"last update":"Fri Feb 9 17:56:03 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":1.829,"15min":1.8580000000000001},"min":{"1min":0.41499999999999998,"5min":0.35799999999999998,"15min":0.28899999999999998},"max":{"1min":1.385,"5min":13.207000000000001,"15min":15.061999999999999},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.68300000000000005,"5min":1.988,"15min":1.9339999999999999},"min":{"1min":0.44900000000000001,"5min":0.375,"15min":0.36399999999999999},"max":{"1min":1.448,"5min":13.298,"15min":13.298},"last":0.441}]},{"osd":7,"last update":"Fri Feb 9 17:55:50 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":1.6359999999999999,"15min":1.869},"min":{"1min":0.374,"5min":0.313,"15min":0.313},"max":{"1min":1.1140000000000001,"5min":18.587,"15min":19.788},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":1.8879999999999999,"15min":1.921},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.34899999999999998},"max":{"1min":1.4139999999999999,"5min":13.239000000000001,"15min":13.239000000000001},"last":0.59799999999999998}]},{"osd":8,"last update":"Fri Feb 9 17:45:16 2024","interfaces":[{"interface":"back","average":{"1min":1.714,"5min":1.4450000000000001,"15min":1.498},"min":{"1min":0.377,"5min":0.377,"15min":0.26800000000000002},"max":{"1min":8.6020000000000003,"5min":11.821,"15min":11.821},"last":0.67700000000000005},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.1719999999999999,"15min":1.351},"min":{"1min":0.46500000000000002,"5min":0.36199999999999999,"15min":0.34499999999999997},"max":{"1min":6.8390000000000004,"5min":6.8390000000000004,"15min":14.129},"last":0.78200000000000003}]},{"osd":9,"last update":"Fri Feb 9 17:56:03 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.9379999999999999,"15min":1.804},"min":{"1min":0.497,"5min":0.312,"15min":0.312},"max":{"1min":3.0270000000000001,"5min":11.039,"15min":11.851000000000001},"last":0.66100000000000003},{"interface":"front","average":{"1min":1.026,"5min":1.8500000000000001,"15min":1.7969999999999999},"min":{"1min":0.376,"5min":0.318,"15min":0.318},"max":{"1min":3.2400000000000002,"5min":10.462999999999999,"15min":11.907},"last":0.69999999999999996}]},{"osd":10,"last update":"Fri Feb 9 17:55:50 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":2.3580000000000001,"15min":1.75},"min":{"1min":0.52400000000000002,"5min":0.371,"15min":0.371},"max":{"1min":1.5449999999999999,"5min":18.632999999999999,"15min":18.632999999999999},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.82399999999999995,"5min":2.2549999999999999,"15min":1.798},"min":{"1min":0.60699999999999998,"5min":0.46800000000000003,"15min":0.34100000000000003},"max":{"1min":1.5760000000000001,"5min":13.272,"15min":19.859000000000002},"last":0.56399999999999995}]},{"osd":11,"last update":"Fri Feb 9 17:56:33 2024","interfaces":[{"interface":"back","average":{"1min":2.1379999999999999,"5min":2.7080000000000002,"15min":1.9850000000000001},"min":{"1min":0.45800000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":8.875,"5min":18.66,"15min":19.824000000000002},"last":0.54000000000000004},{"interface":"front","average":{"1min":2.1619999999999999,"5min":2.738,"15min":1.8839999999999999},"min":{"1min":0.47399999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":8.9179999999999993,"5min":18.687999999999999,"15min":18.687999999999999},"last":0.67600000000000005}]}]},{"osd":4,"up_from":32,"seq":137438953943,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1564876,"kb_used_data":761792,"kb_used_omap":1,"kb_used_meta":803070,"kb_avail":92180276,"statfs":{"total":95995035648,"available":94392602624,"internally_reserved":0,"allocated":780075008,"data_stored":779797537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":822344133},"hb_peers":[0,1,2,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":1.1499999999999999,"15min":1.306},"min":{"1min":0.26800000000000002,"5min":0.246,"15min":0.22800000000000001},"max":{"1min":6.3949999999999996,"5min":10.005000000000001,"15min":10.301},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.78100000000000003,"5min":1.2190000000000001,"15min":1.3740000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.28599999999999998},"max":{"1min":6.4210000000000003,"5min":10.036,"15min":10.407999999999999},"last":0.51600000000000001}]},{"osd":1,"last update":"Fri Feb 9 17:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":1.7829999999999999,"15min":1.786},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.23699999999999999},"max":{"1min":2.2309999999999999,"5min":50.255000000000003,"15min":50.255000000000003},"last":0.27600000000000002},{"interface":"front","average":{"1min":1.0600000000000001,"5min":2.0699999999999998,"15min":2.1389999999999998},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.23100000000000001},"max":{"1min":2.569,"5min":49.991999999999997,"15min":49.991999999999997},"last":0.35699999999999998}]},{"osd":2,"last update":"Fri Feb 9 17:57:41 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":3.6739999999999999,"5min":3.6739999999999999,"15min":3.6739999999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":3.581,"5min":3.581,"15min":3.581},"last":0.56599999999999995}]},{"osd":3,"last update":"Fri Feb 9 17:55:52 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":1.248,"15min":1.569},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.71799999999999997,"5min":10.484999999999999,"15min":14.448},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.4419999999999999,"15min":1.655},"min":{"1min":0.315,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.93000000000000005,"5min":10.455,"15min":10.455},"last":0.438}]},{"osd":5,"last update":"Fri Feb 9 17:58:14 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.48999999999999999,"15min":1.575},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.22600000000000001},"max":{"1min":0.68300000000000005,"5min":3.649,"15min":18.887},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.55800000000000005,"15min":1.641},"min":{"1min":0.314,"5min":0.314,"15min":0.31},"max":{"1min":0.71099999999999997,"5min":3.6800000000000002,"15min":18.917000000000002},"last":0.38300000000000001}]},{"osd":6,"last update":"Fri Feb 9 17:57:52 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.64500000000000002,"15min":1.3919999999999999},"min":{"1min":0.28399999999999997,"5min":0.27800000000000002,"15min":0.24299999999999999},"max":{"1min":2.2759999999999998,"5min":6.7560000000000002,"15min":23.343},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.66300000000000003,"15min":1.456},"min":{"1min":0.33900000000000002,"5min":0.30599999999999999,"15min":0.28799999999999998},"max":{"1min":1.3939999999999999,"5min":5.0060000000000002,"15min":21.861000000000001},"last":0.67800000000000005}]},{"osd":7,"last update":"Fri Feb 9 17:57:30 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.69699999999999995,"15min":1.448},"min":{"1min":0.39200000000000002,"5min":0.32900000000000001,"15min":0.29899999999999999},"max":{"1min":1.694,"5min":5.0640000000000001,"15min":11.875},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.69099999999999995,"15min":1.6140000000000001},"min":{"1min":0.438,"5min":0.34399999999999997,"15min":0.28599999999999998},"max":{"1min":0.69199999999999995,"5min":5.0460000000000003,"15min":12.696},"last":0.753}]},{"osd":8,"last update":"Fri Feb 9 17:58:14 2024","interfaces":[{"interface":"back","average":{"1min":1.6160000000000001,"5min":1.1619999999999999,"15min":1.7470000000000001},"min":{"1min":0.371,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":6.1849999999999996,"5min":9.7059999999999995,"15min":17.335999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":1.8089999999999999,"5min":1.095,"15min":1.3080000000000001},"min":{"1min":0.48999999999999999,"5min":0.373,"15min":0.36699999999999999},"max":{"1min":7.1829999999999998,"5min":7.1829999999999998,"15min":10.433},"last":0.81699999999999995}]},{"osd":9,"last update":"Fri Feb 9 17:58:18 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.2170000000000001,"15min":2.2429999999999999},"min":{"1min":0.40500000000000003,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":4.2629999999999999,"5min":7.665,"15min":7.665},"last":0.58399999999999996},{"interface":"front","average":{"1min":1.833,"5min":1.2729999999999999,"15min":1.613},"min":{"1min":0.42699999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":5.5119999999999996,"5min":7.0999999999999996,"15min":7.9850000000000003},"last":0.78800000000000003}]},{"osd":10,"last update":"Fri Feb 9 17:57:30 2024","interfaces":[{"interface":"back","average":{"1min":1.7350000000000001,"5min":1.405,"15min":1.5609999999999999},"min":{"1min":0.55700000000000005,"5min":0.41299999999999998,"15min":0.40200000000000002},"max":{"1min":4.4379999999999997,"5min":6.9509999999999996,"15min":17.367000000000001},"last":0.84399999999999997},{"interface":"front","average":{"1min":1.393,"5min":1.3080000000000001,"15min":1.6279999999999999},"min":{"1min":0.42699999999999999,"5min":0.41299999999999998,"15min":0.38},"max":{"1min":4.7439999999999998,"5min":7.9189999999999996,"15min":12.888},"last":0.47199999999999998}]},{"osd":11,"last update":"Fri Feb 9 17:58:03 2024","interfaces":[{"interface":"back","average":{"1min":2.141,"5min":1.4419999999999999,"15min":1.5569999999999999},"min":{"1min":0.54200000000000004,"5min":0.46500000000000002,"15min":0.39900000000000002},"max":{"1min":4.766,"5min":7.6689999999999996,"15min":10.521000000000001},"last":3.6339999999999999},{"interface":"front","average":{"1min":2.7320000000000002,"5min":1.444,"15min":1.7310000000000001},"min":{"1min":0.49099999999999999,"5min":0.48499999999999999,"15min":0.45100000000000001},"max":{"1min":9.5960000000000001,"5min":9.5960000000000001,"15min":13.154999999999999},"last":2.1589999999999998}]}]},{"osd":5,"up_from":40,"seq":171798692306,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1701264,"kb_used_data":962056,"kb_used_omap":1,"kb_used_meta":739198,"kb_avail":92043888,"statfs":{"total":95995035648,"available":94252941312,"internally_reserved":0,"allocated":985145344,"data_stored":984864321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":756939205},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:57:42 2024","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":5.399,"5min":5.399,"15min":5.399},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.996,"5min":0.996,"15min":0.996},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":5.4329999999999998,"5min":5.4329999999999998,"15min":5.4329999999999998},"last":0.59299999999999997}]},{"osd":1,"last update":"Fri Feb 9 17:58:18 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.98899999999999999,"15min":1.5569999999999999},"min":{"1min":0.30299999999999999,"5min":0.26600000000000001,"15min":0.254},"max":{"1min":3.0699999999999998,"5min":9.532,"15min":19.079999999999998},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.83999999999999997,"15min":1.4490000000000001},"min":{"1min":0.28499999999999998,"5min":0.25700000000000001,"15min":0.23499999999999999},"max":{"1min":2.5329999999999999,"5min":6.2190000000000003,"15min":19.065999999999999},"last":0.39000000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:57:53 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":1.0069999999999999,"15min":1.6419999999999999},"min":{"1min":0.28199999999999997,"5min":0.254,"15min":0.249},"max":{"1min":2.9900000000000002,"5min":7.5670000000000002,"15min":13.929},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.72199999999999998,"5min":1.0649999999999999,"15min":1.7869999999999999},"min":{"1min":0.30199999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.7689999999999999,"5min":6.3230000000000004,"15min":15.945},"last":0.34599999999999997}]},{"osd":3,"last update":"Fri Feb 9 17:56:16 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":1.8080000000000001,"15min":2.04},"min":{"1min":0.35799999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.3240000000000001,"5min":20.690000000000001,"15min":20.690000000000001},"last":0.505},{"interface":"front","average":{"1min":0.65000000000000002,"5min":1.468,"15min":1.925},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.583,"5min":17.209,"15min":17.209},"last":2.2719999999999998}]},{"osd":4,"last update":"Fri Feb 9 17:58:18 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.60099999999999998,"15min":1.3340000000000001},"min":{"1min":0.26200000000000001,"5min":0.26100000000000001,"15min":0.249},"max":{"1min":1.516,"5min":4.1360000000000001,"15min":28.481999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.747,"15min":1.429},"min":{"1min":0.378,"5min":0.317,"15min":0.22500000000000001},"max":{"1min":2.3370000000000002,"5min":4.9189999999999996,"15min":15.843},"last":0.626}]},{"osd":6,"last update":"Fri Feb 9 17:57:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.65800000000000003,"15min":1.4370000000000001},"min":{"1min":0.34100000000000003,"5min":0.29999999999999999,"15min":0.27900000000000003},"max":{"1min":1.52,"5min":5.2569999999999997,"15min":16.657},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.72899999999999998,"15min":1.5329999999999999},"min":{"1min":0.34200000000000003,"5min":0.33500000000000002,"15min":0.26200000000000001},"max":{"1min":1.9419999999999999,"5min":5.2290000000000001,"15min":20.192},"last":0.40899999999999997}]},{"osd":7,"last update":"Fri Feb 9 17:58:26 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.79900000000000004,"15min":1.508},"min":{"1min":0.35999999999999999,"5min":0.318,"15min":0.29499999999999998},"max":{"1min":2.6909999999999998,"5min":4.9749999999999996,"15min":18.134},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.73799999999999999,"15min":1.649},"min":{"1min":0.40699999999999997,"5min":0.32400000000000001,"15min":0.31},"max":{"1min":1.2410000000000001,"5min":4.9000000000000004,"15min":19.300000000000001},"last":0.44900000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:57:42 2024","interfaces":[{"interface":"back","average":{"1min":2.5600000000000001,"5min":1.4019999999999999,"15min":1.7210000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":12.227,"5min":12.227,"15min":12.227},"last":1.843},{"interface":"front","average":{"1min":0.94599999999999995,"5min":1.131,"15min":1.6759999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":3.8570000000000002,"5min":8.1829999999999998,"15min":8.1829999999999998},"last":5.266}]},{"osd":9,"last update":"Fri Feb 9 17:57:53 2024","interfaces":[{"interface":"back","average":{"1min":2.7280000000000002,"5min":1.6279999999999999,"15min":1.7749999999999999},"min":{"1min":0.42899999999999999,"5min":0.41499999999999998,"15min":0.35299999999999998},"max":{"1min":11.167999999999999,"5min":11.167999999999999,"15min":16.411000000000001},"last":5.8019999999999996},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.0600000000000001,"15min":1.6950000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.35999999999999999},"max":{"1min":3.919,"5min":4.3129999999999997,"15min":16.417999999999999},"last":3.8519999999999999}]},{"osd":10,"last update":"Fri Feb 9 17:58:18 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.1699999999999999,"15min":1.8640000000000001},"min":{"1min":0.47499999999999998,"5min":0.46700000000000003,"15min":0.373},"max":{"1min":3.9780000000000002,"5min":4.8769999999999998,"15min":80.042000000000002},"last":3.0950000000000002},{"interface":"front","average":{"1min":2.2589999999999999,"5min":1.554,"15min":1.917},"min":{"1min":0.48199999999999998,"5min":0.39900000000000002,"15min":0.38200000000000001},"max":{"1min":11.638,"5min":11.638,"15min":79.807000000000002},"last":0.56399999999999995}]},{"osd":11,"last update":"Fri Feb 9 17:57:38 2024","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.393,"15min":1.9390000000000001},"min":{"1min":0.434,"5min":0.434,"15min":0.38400000000000001},"max":{"1min":3.532,"5min":7.5270000000000001,"15min":16.390000000000001},"last":5.2990000000000004},{"interface":"front","average":{"1min":1.335,"5min":1.2350000000000001,"15min":1.782},"min":{"1min":0.53500000000000003,"5min":0.434,"15min":0.42199999999999999},"max":{"1min":4.6669999999999998,"5min":5.6529999999999996,"15min":20.981000000000002},"last":5.25}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":592052224,"data_stored":592052224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1189216256,"data_stored":1189216256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":982323200,"data_stored":982319128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1348993024,"data_stored":1348993024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":771031040,"data_stored":771031040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":983433216,"data_stored":983433216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":388698112,"data_stored":388694040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":602669056,"data_stored":602669056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1214447616,"data_stored":1214447616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1180958720,"data_stored":1180958720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1031606272,"data_stored":1031602200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-09T17:58:34.372 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph pg force-backfill 8.0 2024-02-09T17:58:34.454 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:58:34.495 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:34.648 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:58:34.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:34.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:58:34.754 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:58:34.784 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:34 smithi162 ceph-mon[44304]: from='client.16920 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:58:34.784 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:34 smithi162 ceph-mon[44304]: pgmap v2100: 47 pgs: 2 active+clean+scrubbing+deep, 3 active+undersized+degraded, 5 active+undersized, 37 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 579 op/s; 8904/142578 objects degraded (6.245%) 2024-02-09T17:58:34.787 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:34.813 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:58:34.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:34.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:58:34.832 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-09T17:58:34.918 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:58:34.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:34 smithi179 ceph-mon[43975]: from='client.16920 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:58:34.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:34 smithi179 ceph-mon[43975]: pgmap v2100: 47 pgs: 2 active+clean+scrubbing+deep, 3 active+undersized+degraded, 5 active+undersized, 37 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 579 op/s; 8904/142578 objects degraded (6.245%) 2024-02-09T17:58:34.956 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:34.971 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:34 smithi099 ceph-mon[39420]: from='client.16920 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T17:58:34.971 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:34 smithi099 ceph-mon[39420]: pgmap v2100: 47 pgs: 2 active+clean+scrubbing+deep, 3 active+undersized+degraded, 5 active+undersized, 37 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 579 op/s; 8904/142578 objects degraded (6.245%) 2024-02-09T17:58:34.986 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:58:34.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:34.990 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:58:35.091 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:58:35.120 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:35.148 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:58:35.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:35.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:58:35.181 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:58:35.255 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:58:35.310 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:35.390 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:58:35.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:35.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:58:35.502 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:58:35.534 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:35.611 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:58:35.849 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:58:35.897 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:58:35.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:35.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:58:36.004 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:58:36.039 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:36.194 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:58:36.199 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:36.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:58:36.301 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:58:36.334 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:36.411 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:58:36.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:36.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:58:36.517 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:58:36.560 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:36.791 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:36 smithi162 ceph-mon[44304]: pgmap v2101: 47 pgs: 2 active+clean+scrubbing+deep, 3 active+undersized+degraded, 5 active+undersized, 37 active+clean; 2.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 598 op/s; 8904/139602 objects degraded (6.378%) 2024-02-09T17:58:36.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:36 smithi179 ceph-mon[43975]: pgmap v2101: 47 pgs: 2 active+clean+scrubbing+deep, 3 active+undersized+degraded, 5 active+undersized, 37 active+clean; 2.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 598 op/s; 8904/139602 objects degraded (6.378%) 2024-02-09T17:58:36.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:36 smithi099 ceph-mon[39420]: pgmap v2101: 47 pgs: 2 active+clean+scrubbing+deep, 3 active+undersized+degraded, 5 active+undersized, 37 active+clean; 2.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 598 op/s; 8904/139602 objects degraded (6.378%) 2024-02-09T17:58:37.104 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:58:37.104 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:58:37.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:37 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 8904/139602 objects degraded (6.378%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:58:37.783 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:37 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 8904/139602 objects degraded (6.378%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:58:37.819 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-09T17:58:37.821 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:58:37.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:37.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:58:37.831 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:58:37.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:37 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 8904/139602 objects degraded (6.378%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:58:37.926 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:58:37.957 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:38.162 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:58:38.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:38.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:58:38.167 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:58:38.266 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:58:38.321 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:38.368 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:58:38.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:38.376 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:58:38.477 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:58:38.539 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:38.564 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:58:38.568 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:38.569 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:58:38.620 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is set 2024-02-09T17:58:38.669 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:58:38.690 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:38 smithi162 ceph-mon[44304]: pgmap v2102: 47 pgs: 2 active+clean+scrubbing+deep, 3 active+undersized+degraded, 5 active+undersized, 37 active+clean; 2.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 449 op/s; 8904/137511 objects degraded (6.475%) 2024-02-09T17:58:38.691 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:38 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2871241689' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:58:38.691 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:38 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:58:38.708 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:38.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:38 smithi099 ceph-mon[39420]: pgmap v2102: 47 pgs: 2 active+clean+scrubbing+deep, 3 active+undersized+degraded, 5 active+undersized, 37 active+clean; 2.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 449 op/s; 8904/137511 objects degraded (6.475%) 2024-02-09T17:58:38.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:38 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2871241689' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:58:38.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:38 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:58:38.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:38 smithi179 ceph-mon[43975]: pgmap v2102: 47 pgs: 2 active+clean+scrubbing+deep, 3 active+undersized+degraded, 5 active+undersized, 37 active+clean; 2.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 449 op/s; 8904/137511 objects degraded (6.475%) 2024-02-09T17:58:38.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:38 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2871241689' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:58:38.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:38 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:58:39.187 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:58:39.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:39.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:58:39.290 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:58:39.328 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:39.358 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:58:39.362 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:39.362 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:58:39.463 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:58:39.505 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:39.510 INFO:teuthology.orchestra.run.smithi099.stderr:instructing pg(s) [8.0] on osd.9 to force-backfill; 2024-02-09T17:58:39.510 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-02-09T17:58:39.543 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:58:39.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:39.552 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:58:39.614 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:39 smithi179 ceph-mon[43975]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:58:39.614 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:39 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:58:39.614 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:39 smithi179 ceph-mon[43975]: osdmap e847: 12 total, 11 up, 12 in 2024-02-09T17:58:39.653 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:58:39.712 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:39.755 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:58:39.761 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:39.762 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:58:39.781 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:39 smithi099 ceph-mon[39420]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:58:39.781 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:39 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:58:39.782 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:39 smithi099 ceph-mon[39420]: osdmap e847: 12 total, 11 up, 12 in 2024-02-09T17:58:39.862 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:58:39.899 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:39 smithi162 ceph-mon[44304]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:58:39.899 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:39 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:58:39.899 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:39 smithi162 ceph-mon[44304]: osdmap e847: 12 total, 11 up, 12 in 2024-02-09T17:58:39.909 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:40.177 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T17:58:40.177 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:58:40.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:40 smithi179 ceph-mon[43975]: pgmap v2104: 47 pgs: 2 active+clean+scrubbing+deep, 3 active+undersized+degraded, 5 active+undersized, 37 active+clean; 2.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 628 op/s; 8398/132525 objects degraded (6.337%) 2024-02-09T17:58:40.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:40 smithi179 ceph-mon[43975]: from='client.16929 -' entity='client.admin' cmd=[{"prefix": "pg force-backfill", "pgid": ["8.0"], "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:58:40.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:40 smithi162 ceph-mon[44304]: pgmap v2104: 47 pgs: 2 active+clean+scrubbing+deep, 3 active+undersized+degraded, 5 active+undersized, 37 active+clean; 2.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 628 op/s; 8398/132525 objects degraded (6.337%) 2024-02-09T17:58:40.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:40 smithi162 ceph-mon[44304]: from='client.16929 -' entity='client.admin' cmd=[{"prefix": "pg force-backfill", "pgid": ["8.0"], "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:58:40.992 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-09T17:58:40.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:40 smithi099 ceph-mon[39420]: pgmap v2104: 47 pgs: 2 active+clean+scrubbing+deep, 3 active+undersized+degraded, 5 active+undersized, 37 active+clean; 2.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 628 op/s; 8398/132525 objects degraded (6.337%) 2024-02-09T17:58:40.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:40 smithi099 ceph-mon[39420]: from='client.16929 -' entity='client.admin' cmd=[{"prefix": "pg force-backfill", "pgid": ["8.0"], "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T17:58:41.062 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:58:41.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:41.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:58:41.167 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:58:41.203 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:41.249 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:58:41.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:41.262 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:58:41.314 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:58:41.363 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:58:41.398 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:41.599 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:58:41.605 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:41.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:58:41.707 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:58:41.754 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:41.783 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:58:41.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:41.786 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:58:41.887 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:58:41.924 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:42.177 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:58:42.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:42.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:58:42.281 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:58:42.282 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:58:42.314 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:58:42.338 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:42.359 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:58:42.369 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:58:42.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:42.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:58:42.398 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:58:42.423 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:58:42.461 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:58:42.474 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:58:42.483 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:58:42.504 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:42.519 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:58:42.533 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:58:42.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:42.536 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:58:42.552 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:58:42.589 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:58:42.623 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:58:42.624 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:58:42.624 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:58:42.624 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:58:42.624 INFO:teuthology.orchestra.run.smithi099.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-09T17:58:42.624 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:58:42.624 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:58:42.624 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:58:42.624 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:58:42.624 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:58:42.624 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:58:42.625 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:58:42.625 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:58:42.637 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:58:42.666 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:42.693 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:58:42.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:42.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:58:42.802 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:58:42.842 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:42 smithi179 ceph-mon[43975]: pgmap v2105: 47 pgs: 1 active+clean+scrubbing+deep, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 2.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 561 op/s; 8398/131373 objects degraded (6.392%) 2024-02-09T17:58:42.842 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:42 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 8398/131373 objects degraded (6.392%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:58:42.842 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:42 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4251665313' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:58:42.844 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:42.888 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:58:42.891 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:42.891 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:58:42.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:42 smithi162 ceph-mon[44304]: pgmap v2105: 47 pgs: 1 active+clean+scrubbing+deep, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 2.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 561 op/s; 8398/131373 objects degraded (6.392%) 2024-02-09T17:58:42.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:42 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 8398/131373 objects degraded (6.392%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:58:42.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:42 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4251665313' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:58:42.982 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:42 smithi099 ceph-mon[39420]: pgmap v2105: 47 pgs: 1 active+clean+scrubbing+deep, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 2.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 561 op/s; 8398/131373 objects degraded (6.392%) 2024-02-09T17:58:42.983 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:42 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 8398/131373 objects degraded (6.392%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:58:42.983 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:42 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4251665313' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:58:42.992 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:58:43.068 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:43.101 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:58:43.112 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:43.113 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:58:43.214 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:58:43.251 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:43.280 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:58:43.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:43.290 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:58:43.299 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:58:43.299 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-02-09T17:58:43.300 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-02-09T17:58:43.300 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-02-09T17:58:43.300 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.300 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.16440.0:86129 7.7 7:e80ddecc:::benchmark_data_smithi179_135804_object86128:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e688)", 2024-02-09T17:58:43.300 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:51:22.873912+0000", 2024-02-09T17:58:43.300 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 440.41855610900001, 2024-02-09T17:58:43.300 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23220575099999999, 2024-02-09T17:58:43.300 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:58:43.300 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:58:43.300 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:58:43.300 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.16440", 2024-02-09T17:58:43.300 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:58:43.301 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 86129 2024-02-09T17:58:43.301 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.301 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:58:43.301 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.301 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:58:43.301 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:22.873912+0000", 2024-02-09T17:58:43.301 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:43.301 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.301 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.301 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:58:43.301 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:22.873912+0000", 2024-02-09T17:58:43.301 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:43.301 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.301 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.302 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:58:43.302 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:22.873914+0000", 2024-02-09T17:58:43.302 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0609999999999998e-06 2024-02-09T17:58:43.302 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.302 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.302 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:58:43.302 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:22.873954+0000", 2024-02-09T17:58:43.302 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0321999999999998e-05 2024-02-09T17:58:43.302 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.302 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.302 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:58:43.302 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:22.873969+0000", 2024-02-09T17:58:43.302 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4559000000000001e-05 2024-02-09T17:58:43.303 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.303 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.303 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:58:43.303 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:22.873978+0000", 2024-02-09T17:58:43.303 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2809999999999997e-06 2024-02-09T17:58:43.303 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.303 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.303 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:58:43.303 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:22.882828+0000", 2024-02-09T17:58:43.303 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0088500869999999995 2024-02-09T17:58:43.303 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.303 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.303 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:58:43.304 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:22.882987+0000", 2024-02-09T17:58:43.304 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015907399999999999 2024-02-09T17:58:43.304 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.304 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.304 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,8", 2024-02-09T17:58:43.304 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:22.883141+0000", 2024-02-09T17:58:43.304 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000154279 2024-02-09T17:58:43.304 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.304 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.304 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:43.304 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:22.895820+0000", 2024-02-09T17:58:43.304 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012678782 2024-02-09T17:58:43.305 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.305 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.305 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:58:43.305 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:22.895838+0000", 2024-02-09T17:58:43.305 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8408000000000001e-05 2024-02-09T17:58:43.305 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.305 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.305 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:43.305 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:23.102657+0000", 2024-02-09T17:58:43.305 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20681824700000001 2024-02-09T17:58:43.305 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.305 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.305 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:58:43.306 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:23.102727+0000", 2024-02-09T17:58:43.306 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0693999999999998e-05 2024-02-09T17:58:43.306 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.306 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.306 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:58:43.306 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:23.106117+0000", 2024-02-09T17:58:43.306 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0033899569999999999 2024-02-09T17:58:43.306 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:43.306 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:58:43.306 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:43.306 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.307 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.307 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.16440.0:86138 7.7 7:e6c09341:::benchmark_data_smithi179_135804_object86137:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e688)", 2024-02-09T17:58:43.307 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:51:22.881337+0000", 2024-02-09T17:58:43.307 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 440.41113021400002, 2024-02-09T17:58:43.307 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.227619815, 2024-02-09T17:58:43.307 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:58:43.307 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:58:43.307 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:58:43.307 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.16440", 2024-02-09T17:58:43.307 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:58:43.307 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 86138 2024-02-09T17:58:43.307 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.307 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:58:43.307 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.308 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:58:43.308 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:22.881337+0000", 2024-02-09T17:58:43.308 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:43.308 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.308 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.308 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:58:43.308 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:22.881337+0000", 2024-02-09T17:58:43.308 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:43.308 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.308 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.308 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:58:43.308 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:22.881339+0000", 2024-02-09T17:58:43.309 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7570000000000001e-06 2024-02-09T17:58:43.309 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.309 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.309 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:58:43.309 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:22.881364+0000", 2024-02-09T17:58:43.309 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4505e-05 2024-02-09T17:58:43.309 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.309 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.309 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:58:43.309 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:22.881376+0000", 2024-02-09T17:58:43.309 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2526e-05 2024-02-09T17:58:43.309 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.309 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.309 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:58:43.310 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:22.881385+0000", 2024-02-09T17:58:43.310 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9099999999999994e-06 2024-02-09T17:58:43.310 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.310 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.310 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:58:43.310 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:22.892979+0000", 2024-02-09T17:58:43.310 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011593453 2024-02-09T17:58:43.310 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.310 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.310 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:58:43.310 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:22.893120+0000", 2024-02-09T17:58:43.310 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000140937 2024-02-09T17:58:43.310 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.310 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.310 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,8", 2024-02-09T17:58:43.311 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:22.893266+0000", 2024-02-09T17:58:43.311 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000145933 2024-02-09T17:58:43.311 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.311 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.311 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:58:43.311 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:22.902490+0000", 2024-02-09T17:58:43.311 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0092243740000000005 2024-02-09T17:58:43.311 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.311 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.311 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:43.311 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:22.902532+0000", 2024-02-09T17:58:43.311 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2123000000000003e-05 2024-02-09T17:58:43.311 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.311 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.311 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:43.312 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:23.106189+0000", 2024-02-09T17:58:43.312 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.203656691 2024-02-09T17:58:43.312 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.312 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.312 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:58:43.312 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:23.106266+0000", 2024-02-09T17:58:43.312 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7393999999999998e-05 2024-02-09T17:58:43.312 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.312 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.312 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:58:43.312 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:23.108957+0000", 2024-02-09T17:58:43.312 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0026912120000000001 2024-02-09T17:58:43.312 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:43.313 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:58:43.313 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:43.313 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.313 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.313 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.16440.0:118352 7.7 7:eb03f5cc:::benchmark_data_smithi179_135804_object10561:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e706)", 2024-02-09T17:58:43.313 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:52:12.843021+0000", 2024-02-09T17:58:43.313 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 390.44944626199998, 2024-02-09T17:58:43.313 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.221836534, 2024-02-09T17:58:43.313 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:58:43.313 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:58:43.313 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:58:43.313 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.16440", 2024-02-09T17:58:43.313 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:58:43.313 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 118352 2024-02-09T17:58:43.314 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.314 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:58:43.314 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.314 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:58:43.314 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:12.843021+0000", 2024-02-09T17:58:43.314 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:43.314 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.314 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.314 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:58:43.314 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:12.843021+0000", 2024-02-09T17:58:43.314 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:43.314 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.314 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.314 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:58:43.315 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:12.843022+0000", 2024-02-09T17:58:43.315 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0440000000000001e-06 2024-02-09T17:58:43.315 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.315 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.315 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:58:43.315 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:12.843027+0000", 2024-02-09T17:58:43.315 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3200000000000001e-06 2024-02-09T17:58:43.315 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.315 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.315 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:58:43.315 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:12.843036+0000", 2024-02-09T17:58:43.315 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3170000000000008e-06 2024-02-09T17:58:43.315 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.315 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.315 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:58:43.316 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:12.843042+0000", 2024-02-09T17:58:43.316 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6919999999999999e-06 2024-02-09T17:58:43.316 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.316 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.316 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:58:43.316 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:12.843086+0000", 2024-02-09T17:58:43.316 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4416000000000003e-05 2024-02-09T17:58:43.316 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.316 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.316 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:58:43.316 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:12.843194+0000", 2024-02-09T17:58:43.316 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010805 2024-02-09T17:58:43.317 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.317 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.317 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,8", 2024-02-09T17:58:43.317 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:12.843308+0000", 2024-02-09T17:58:43.317 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011412399999999999 2024-02-09T17:58:43.317 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.317 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.317 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:58:43.317 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:12.847007+0000", 2024-02-09T17:58:43.317 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0036983480000000002 2024-02-09T17:58:43.317 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.317 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.317 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:43.317 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:12.863174+0000", 2024-02-09T17:58:43.318 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.016166903999999999 2024-02-09T17:58:43.318 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.318 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.318 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:43.318 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:13.063081+0000", 2024-02-09T17:58:43.318 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19990731 2024-02-09T17:58:43.318 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.318 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.318 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:58:43.318 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:13.063137+0000", 2024-02-09T17:58:43.318 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5908000000000001e-05 2024-02-09T17:58:43.318 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.318 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.318 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:58:43.319 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:13.064858+0000", 2024-02-09T17:58:43.319 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017211010000000001 2024-02-09T17:58:43.319 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:43.319 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:58:43.319 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:43.319 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.319 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.319 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.16440.0:139599 7.7 7:e0c74c24:::benchmark_data_smithi179_135804_object31808:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e721)", 2024-02-09T17:58:43.319 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:52:47.850299+0000", 2024-02-09T17:58:43.319 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 355.44216881599999, 2024-02-09T17:58:43.319 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26119968900000001, 2024-02-09T17:58:43.319 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:58:43.319 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:58:43.319 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:58:43.320 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.16440", 2024-02-09T17:58:43.320 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:58:43.320 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 139599 2024-02-09T17:58:43.320 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.320 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:58:43.320 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.320 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:58:43.320 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:47.850299+0000", 2024-02-09T17:58:43.320 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:43.320 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.320 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.320 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:58:43.321 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:47.850299+0000", 2024-02-09T17:58:43.321 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:43.321 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.321 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.321 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:58:43.321 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:47.850300+0000", 2024-02-09T17:58:43.321 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.176e-06 2024-02-09T17:58:43.321 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.321 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.321 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:58:43.321 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:47.850304+0000", 2024-02-09T17:58:43.321 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8689999999999997e-06 2024-02-09T17:58:43.321 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.321 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.322 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:58:43.322 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:47.850315+0000", 2024-02-09T17:58:43.322 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1336e-05 2024-02-09T17:58:43.322 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.322 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.322 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:58:43.322 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:47.850321+0000", 2024-02-09T17:58:43.322 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1659999999999999e-06 2024-02-09T17:58:43.322 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.322 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.322 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:58:43.322 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:47.931291+0000", 2024-02-09T17:58:43.322 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.080969181000000001 2024-02-09T17:58:43.323 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.323 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.323 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:58:43.323 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:47.931387+0000", 2024-02-09T17:58:43.323 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6551000000000004e-05 2024-02-09T17:58:43.323 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.323 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.323 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,8", 2024-02-09T17:58:43.323 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:47.931474+0000", 2024-02-09T17:58:43.323 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6781999999999997e-05 2024-02-09T17:58:43.323 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.323 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.323 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:58:43.323 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:47.939145+0000", 2024-02-09T17:58:43.323 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0076708380000000001 2024-02-09T17:58:43.323 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.324 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.324 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:43.324 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:47.939954+0000", 2024-02-09T17:58:43.324 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00080872500000000005 2024-02-09T17:58:43.324 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.324 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.324 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:43.324 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:48.111416+0000", 2024-02-09T17:58:43.324 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.171462957 2024-02-09T17:58:43.324 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.324 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.324 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:58:43.324 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:48.111460+0000", 2024-02-09T17:58:43.324 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3665999999999998e-05 2024-02-09T17:58:43.325 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.325 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.325 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:58:43.325 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:48.111499+0000", 2024-02-09T17:58:43.325 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8442000000000002e-05 2024-02-09T17:58:43.325 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:43.325 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:58:43.325 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:43.325 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.325 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.325 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.16440.0:139617 7.7 7:ec6733b0:::benchmark_data_smithi179_135804_object31826:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e721)", 2024-02-09T17:58:43.325 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:52:47.870916+0000", 2024-02-09T17:58:43.325 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 355.42155126, 2024-02-09T17:58:43.325 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24070385999999999, 2024-02-09T17:58:43.326 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:58:43.326 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:58:43.326 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:58:43.326 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.16440", 2024-02-09T17:58:43.326 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:58:43.326 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 139617 2024-02-09T17:58:43.326 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.326 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:58:43.326 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.326 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:58:43.326 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:47.870916+0000", 2024-02-09T17:58:43.326 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:43.326 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.326 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.327 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:58:43.327 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:47.870916+0000", 2024-02-09T17:58:43.327 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:43.327 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.327 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.327 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:58:43.327 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:47.870918+0000", 2024-02-09T17:58:43.327 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.26e-06 2024-02-09T17:58:43.327 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.327 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.327 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:58:43.327 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:47.870922+0000", 2024-02-09T17:58:43.327 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7690000000000004e-06 2024-02-09T17:58:43.327 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.328 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.328 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:58:43.328 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:47.870934+0000", 2024-02-09T17:58:43.328 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1921e-05 2024-02-09T17:58:43.328 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.328 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.328 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:58:43.328 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:47.870941+0000", 2024-02-09T17:58:43.328 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0110000000000003e-06 2024-02-09T17:58:43.328 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.328 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.328 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:58:43.328 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:47.935155+0000", 2024-02-09T17:58:43.328 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.064213817000000006 2024-02-09T17:58:43.329 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.329 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.329 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:58:43.329 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:47.935282+0000", 2024-02-09T17:58:43.329 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012711299999999999 2024-02-09T17:58:43.329 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.329 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.329 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,8", 2024-02-09T17:58:43.329 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:47.935365+0000", 2024-02-09T17:58:43.329 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2199000000000004e-05 2024-02-09T17:58:43.329 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.329 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.329 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:58:43.329 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:47.939368+0000", 2024-02-09T17:58:43.330 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0040034459999999999 2024-02-09T17:58:43.330 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.330 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.330 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:43.330 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:47.939980+0000", 2024-02-09T17:58:43.330 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00061224000000000001 2024-02-09T17:58:43.330 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.330 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.330 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:43.330 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:48.111538+0000", 2024-02-09T17:58:43.330 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17155783399999999 2024-02-09T17:58:43.330 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.331 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.331 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:58:43.331 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:48.111579+0000", 2024-02-09T17:58:43.331 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0985000000000001e-05 2024-02-09T17:58:43.331 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.331 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.331 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:58:43.331 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:48.111620+0000", 2024-02-09T17:58:43.331 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1264999999999999e-05 2024-02-09T17:58:43.331 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:43.331 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:58:43.331 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:43.331 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.331 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.331 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.16440.0:156824 7.c 7:321e6228:::benchmark_data_smithi179_135804_object49033:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e742)", 2024-02-09T17:58:43.332 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:53:19.256314+0000", 2024-02-09T17:58:43.332 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 324.03615320099999, 2024-02-09T17:58:43.332 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9669922509999997, 2024-02-09T17:58:43.332 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:58:43.332 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-09T17:58:43.332 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:58:43.332 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.16440", 2024-02-09T17:58:43.332 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:58:43.332 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 156824 2024-02-09T17:58:43.332 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.332 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:58:43.332 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.332 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:58:43.333 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:19.256314+0000", 2024-02-09T17:58:43.333 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:43.333 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.333 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.333 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:58:43.333 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:19.256314+0000", 2024-02-09T17:58:43.333 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:43.333 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.333 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.333 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:58:43.333 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:19.256315+0000", 2024-02-09T17:58:43.333 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7000000000000003e-07 2024-02-09T17:58:43.333 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.333 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.334 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:58:43.334 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:19.256319+0000", 2024-02-09T17:58:43.334 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7390000000000001e-06 2024-02-09T17:58:43.334 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.334 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.334 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:58:43.334 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:19.256327+0000", 2024-02-09T17:58:43.334 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6490000000000008e-06 2024-02-09T17:58:43.334 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.334 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.334 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:58:43.334 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:19.256336+0000", 2024-02-09T17:58:43.334 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1449999999999997e-06 2024-02-09T17:58:43.334 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.334 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.335 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:58:43.335 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:19.256389+0000", 2024-02-09T17:58:43.335 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3430999999999997e-05 2024-02-09T17:58:43.335 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.335 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.335 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for missing object", 2024-02-09T17:58:43.335 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:19.256452+0000", 2024-02-09T17:58:43.335 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3037e-05 2024-02-09T17:58:43.335 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.335 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.335 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:58:43.335 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:23.223291+0000", 2024-02-09T17:58:43.335 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9668390609999999 2024-02-09T17:58:43.335 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.335 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.336 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:58:43.336 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:23.223307+0000", 2024-02-09T17:58:43.336 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5319e-05 2024-02-09T17:58:43.336 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:43.336 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:58:43.336 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:43.336 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.336 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.336 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.16440.0:158824 7.c 7:3cb4d6c0:::benchmark_data_smithi179_135804_object51033:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e743)", 2024-02-09T17:58:43.336 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:53:22.174549+0000", 2024-02-09T17:58:43.336 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 321.11791846400001, 2024-02-09T17:58:43.336 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22753790300000001, 2024-02-09T17:58:43.336 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:58:43.336 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:58:43.337 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:58:43.337 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.16440", 2024-02-09T17:58:43.337 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:58:43.337 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 158824 2024-02-09T17:58:43.337 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.337 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:58:43.337 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.337 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:58:43.337 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:22.174549+0000", 2024-02-09T17:58:43.337 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:43.337 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.337 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.337 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:58:43.337 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:22.174549+0000", 2024-02-09T17:58:43.337 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:43.337 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.337 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.338 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:58:43.338 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:22.174551+0000", 2024-02-09T17:58:43.338 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.658e-06 2024-02-09T17:58:43.338 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.338 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.338 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:58:43.338 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:22.174556+0000", 2024-02-09T17:58:43.338 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0780000000000003e-06 2024-02-09T17:58:43.338 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.338 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.338 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:58:43.338 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:22.174567+0000", 2024-02-09T17:58:43.338 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1173e-05 2024-02-09T17:58:43.338 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.339 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.339 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:58:43.339 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:22.174575+0000", 2024-02-09T17:58:43.339 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9300000000000003e-06 2024-02-09T17:58:43.339 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.339 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.339 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:58:43.339 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:22.174658+0000", 2024-02-09T17:58:43.339 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3374999999999994e-05 2024-02-09T17:58:43.339 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.339 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.339 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:58:43.339 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:22.174749+0000", 2024-02-09T17:58:43.340 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0161000000000006e-05 2024-02-09T17:58:43.340 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.340 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.340 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,11", 2024-02-09T17:58:43.340 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:22.174893+0000", 2024-02-09T17:58:43.340 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014465600000000001 2024-02-09T17:58:43.340 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.340 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.340 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:58:43.340 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:22.179028+0000", 2024-02-09T17:58:43.340 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041342949999999996 2024-02-09T17:58:43.340 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.341 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.341 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:43.341 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:22.192467+0000", 2024-02-09T17:58:43.341 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013439536 2024-02-09T17:58:43.341 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.341 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.341 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:43.341 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:22.398661+0000", 2024-02-09T17:58:43.341 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.206193554 2024-02-09T17:58:43.341 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.341 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.341 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:58:43.342 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:22.398728+0000", 2024-02-09T17:58:43.342 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7787999999999995e-05 2024-02-09T17:58:43.342 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.342 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.342 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:58:43.342 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:22.402087+0000", 2024-02-09T17:58:43.342 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0033586990000000001 2024-02-09T17:58:43.342 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:43.342 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:58:43.342 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:43.342 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.343 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.343 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.16440.0:180042 7.c 7:3ed72c98:::benchmark_data_smithi179_135804_object72251:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e749)", 2024-02-09T17:58:43.343 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:53:54.742640+0000", 2024-02-09T17:58:43.343 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 288.54982720800001, 2024-02-09T17:58:43.343 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23305893699999999, 2024-02-09T17:58:43.343 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:58:43.343 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:58:43.343 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:58:43.343 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.16440", 2024-02-09T17:58:43.343 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:58:43.343 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 180042 2024-02-09T17:58:43.343 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.344 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:58:43.344 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.344 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:58:43.344 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.742640+0000", 2024-02-09T17:58:43.344 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:43.344 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.344 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.344 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:58:43.344 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.742640+0000", 2024-02-09T17:58:43.344 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:43.344 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.344 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.344 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:58:43.345 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.742641+0000", 2024-02-09T17:58:43.345 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4900000000000005e-07 2024-02-09T17:58:43.345 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.345 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.345 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:58:43.345 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.742645+0000", 2024-02-09T17:58:43.345 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.49e-06 2024-02-09T17:58:43.345 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.345 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.345 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:58:43.345 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.742653+0000", 2024-02-09T17:58:43.345 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2050000000000002e-06 2024-02-09T17:58:43.345 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.346 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.346 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:58:43.346 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.742660+0000", 2024-02-09T17:58:43.346 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8290000000000002e-06 2024-02-09T17:58:43.346 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.346 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.346 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:58:43.346 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.742721+0000", 2024-02-09T17:58:43.346 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1445999999999994e-05 2024-02-09T17:58:43.346 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.346 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.346 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:58:43.346 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.742800+0000", 2024-02-09T17:58:43.347 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9288000000000004e-05 2024-02-09T17:58:43.347 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.347 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.347 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,11", 2024-02-09T17:58:43.347 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.742910+0000", 2024-02-09T17:58:43.347 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010961199999999999 2024-02-09T17:58:43.347 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.347 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.347 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:43.347 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.748191+0000", 2024-02-09T17:58:43.347 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0052813269999999997 2024-02-09T17:58:43.347 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.347 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.348 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:58:43.348 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.749269+0000", 2024-02-09T17:58:43.348 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0010771050000000001 2024-02-09T17:58:43.348 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.348 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.348 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:43.348 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.973364+0000", 2024-02-09T17:58:43.348 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.224095869 2024-02-09T17:58:43.348 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.348 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.348 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:58:43.349 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.973414+0000", 2024-02-09T17:58:43.349 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.994e-05 2024-02-09T17:58:43.349 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.349 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.349 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:58:43.349 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.975699+0000", 2024-02-09T17:58:43.349 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0022850769999999999 2024-02-09T17:58:43.349 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:43.349 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:58:43.349 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:43.349 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.349 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.349 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.16440.0:180054 7.c 7:3547ab8c:::benchmark_data_smithi179_135804_object72263:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e749)", 2024-02-09T17:58:43.350 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:53:54.764783+0000", 2024-02-09T17:58:43.350 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 288.527684454, 2024-02-09T17:58:43.350 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22504629000000001, 2024-02-09T17:58:43.350 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:58:43.350 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:58:43.350 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:58:43.350 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.16440", 2024-02-09T17:58:43.350 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:58:43.350 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 180054 2024-02-09T17:58:43.350 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.350 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:58:43.350 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.350 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:58:43.351 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.764783+0000", 2024-02-09T17:58:43.351 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:43.351 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.351 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.351 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:58:43.351 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.764783+0000", 2024-02-09T17:58:43.351 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:43.351 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.351 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.351 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:58:43.351 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.764784+0000", 2024-02-09T17:58:43.351 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4099999999999997e-07 2024-02-09T17:58:43.351 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.351 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.352 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:58:43.352 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.764788+0000", 2024-02-09T17:58:43.352 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3710000000000002e-06 2024-02-09T17:58:43.352 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.352 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.352 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:58:43.352 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.764800+0000", 2024-02-09T17:58:43.352 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1493e-05 2024-02-09T17:58:43.352 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.352 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.352 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:58:43.352 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.764807+0000", 2024-02-09T17:58:43.352 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2690000000000002e-06 2024-02-09T17:58:43.352 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.352 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.353 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:58:43.353 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.764871+0000", 2024-02-09T17:58:43.353 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3392000000000006e-05 2024-02-09T17:58:43.353 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.353 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.353 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:58:43.353 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.764983+0000", 2024-02-09T17:58:43.353 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011234 2024-02-09T17:58:43.353 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.353 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.353 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,11", 2024-02-09T17:58:43.353 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.765154+0000", 2024-02-09T17:58:43.353 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000170951 2024-02-09T17:58:43.353 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.354 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.354 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:43.354 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.770510+0000", 2024-02-09T17:58:43.354 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0053557479999999996 2024-02-09T17:58:43.354 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.354 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.354 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:58:43.354 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.770818+0000", 2024-02-09T17:58:43.354 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00030799700000000001 2024-02-09T17:58:43.354 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.354 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.354 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:43.354 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.987184+0000", 2024-02-09T17:58:43.355 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.216366849 2024-02-09T17:58:43.355 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.355 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.355 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:58:43.355 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.987255+0000", 2024-02-09T17:58:43.355 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0888000000000003e-05 2024-02-09T17:58:43.355 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.355 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.355 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:58:43.355 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.989830+0000", 2024-02-09T17:58:43.355 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0025741510000000002 2024-02-09T17:58:43.355 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:43.355 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:58:43.356 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:43.356 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.356 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.356 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.16440.0:180063 7.c 7:34f65bea:::benchmark_data_smithi179_135804_object72272:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e749)", 2024-02-09T17:58:43.356 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:53:54.783269+0000", 2024-02-09T17:58:43.356 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 288.50919896800002, 2024-02-09T17:58:43.356 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.221613017, 2024-02-09T17:58:43.356 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:58:43.356 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:58:43.356 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:58:43.356 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.16440", 2024-02-09T17:58:43.356 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:58:43.356 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 180063 2024-02-09T17:58:43.357 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.357 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:58:43.357 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.357 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:58:43.357 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.783269+0000", 2024-02-09T17:58:43.357 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:43.357 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.357 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.357 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:58:43.357 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.783269+0000", 2024-02-09T17:58:43.357 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:43.357 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.358 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.358 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:58:43.358 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.783270+0000", 2024-02-09T17:58:43.358 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.562e-06 2024-02-09T17:58:43.358 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.358 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.358 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:58:43.358 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.783277+0000", 2024-02-09T17:58:43.358 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8920000000000003e-06 2024-02-09T17:58:43.358 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.358 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.358 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:58:43.359 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.783292+0000", 2024-02-09T17:58:43.359 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4965e-05 2024-02-09T17:58:43.359 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.359 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.359 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:58:43.359 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.783300+0000", 2024-02-09T17:58:43.359 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4340000000000003e-06 2024-02-09T17:58:43.359 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.359 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.359 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:58:43.359 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.783364+0000", 2024-02-09T17:58:43.359 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4844000000000005e-05 2024-02-09T17:58:43.360 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.360 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.360 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:58:43.360 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.783487+0000", 2024-02-09T17:58:43.360 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012261999999999999 2024-02-09T17:58:43.360 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.360 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.360 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,11", 2024-02-09T17:58:43.360 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.783646+0000", 2024-02-09T17:58:43.360 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015908099999999999 2024-02-09T17:58:43.360 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.360 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.360 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:43.361 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.788961+0000", 2024-02-09T17:58:43.361 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0053144259999999997 2024-02-09T17:58:43.361 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.361 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.361 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:58:43.361 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.791274+0000", 2024-02-09T17:58:43.361 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023135260000000002 2024-02-09T17:58:43.361 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.361 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.361 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:43.361 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:55.002510+0000", 2024-02-09T17:58:43.361 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21123641500000001 2024-02-09T17:58:43.361 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.361 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.362 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:58:43.362 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:55.002548+0000", 2024-02-09T17:58:43.362 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7036000000000003e-05 2024-02-09T17:58:43.362 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.362 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.362 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:58:43.362 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:55.004882+0000", 2024-02-09T17:58:43.362 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023342160000000001 2024-02-09T17:58:43.362 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:43.362 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:58:43.362 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:43.362 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.362 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.363 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.16440.0:180215 7.c 7:3d493c52:::benchmark_data_smithi179_135804_object72424:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e749)", 2024-02-09T17:58:43.363 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:53:55.124898+0000", 2024-02-09T17:58:43.363 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 288.167570044, 2024-02-09T17:58:43.363 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0195489210000002, 2024-02-09T17:58:43.363 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:58:43.363 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-09T17:58:43.363 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:58:43.363 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.16440", 2024-02-09T17:58:43.363 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:58:43.363 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 180215 2024-02-09T17:58:43.363 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.363 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:58:43.364 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.364 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:58:43.364 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:55.124898+0000", 2024-02-09T17:58:43.364 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:43.364 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.364 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.364 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:58:43.364 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:55.124898+0000", 2024-02-09T17:58:43.364 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:43.364 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.364 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.364 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:58:43.365 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:55.124898+0000", 2024-02-09T17:58:43.365 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9900000000000002e-07 2024-02-09T17:58:43.365 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.365 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.365 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:58:43.365 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:55.124900+0000", 2024-02-09T17:58:43.365 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8610000000000001e-06 2024-02-09T17:58:43.365 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.365 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.365 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:58:43.365 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:55.124905+0000", 2024-02-09T17:58:43.365 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6859999999999997e-06 2024-02-09T17:58:43.365 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.365 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.366 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:58:43.366 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:55.124908+0000", 2024-02-09T17:58:43.366 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1729999999999999e-06 2024-02-09T17:58:43.366 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.366 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.366 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:58:43.366 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:55.131925+0000", 2024-02-09T17:58:43.366 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0070172129999999996 2024-02-09T17:58:43.366 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.366 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.366 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for missing object", 2024-02-09T17:58:43.366 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:55.131983+0000", 2024-02-09T17:58:43.366 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8168000000000003e-05 2024-02-09T17:58:43.366 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.366 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.366 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:58:43.366 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:57.144431+0000", 2024-02-09T17:58:43.367 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0124483209999999 2024-02-09T17:58:43.367 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.367 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.367 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:58:43.367 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:57.144447+0000", 2024-02-09T17:58:43.367 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.52e-05 2024-02-09T17:58:43.367 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:43.367 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:58:43.367 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:43.367 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.367 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.367 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.16440.0:180223 7.c 7:3058aa28:::benchmark_data_smithi179_135804_object72432:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e749)", 2024-02-09T17:58:43.367 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:53:55.124924+0000", 2024-02-09T17:58:43.367 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 288.16754352800001, 2024-02-09T17:58:43.367 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0243867280000001, 2024-02-09T17:58:43.368 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:58:43.368 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-09T17:58:43.368 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:58:43.368 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.16440", 2024-02-09T17:58:43.368 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:58:43.368 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 180223 2024-02-09T17:58:43.368 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.368 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:58:43.368 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.368 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:58:43.368 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:55.124924+0000", 2024-02-09T17:58:43.368 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:43.368 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.368 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.368 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:58:43.368 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:55.124924+0000", 2024-02-09T17:58:43.369 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:43.369 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.369 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.369 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:58:43.369 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:55.124924+0000", 2024-02-09T17:58:43.369 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2399999999999999e-07 2024-02-09T17:58:43.369 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.369 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.369 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:58:43.369 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:55.124926+0000", 2024-02-09T17:58:43.369 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.232e-06 2024-02-09T17:58:43.369 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.369 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.369 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:58:43.369 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:55.124930+0000", 2024-02-09T17:58:43.369 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.617e-06 2024-02-09T17:58:43.369 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.370 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.370 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:58:43.370 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:55.124933+0000", 2024-02-09T17:58:43.370 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9189999999999999e-06 2024-02-09T17:58:43.370 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.370 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.370 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:58:43.370 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:55.132004+0000", 2024-02-09T17:58:43.370 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0070709099999999997 2024-02-09T17:58:43.370 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.370 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.370 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for missing object", 2024-02-09T17:58:43.370 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:55.132062+0000", 2024-02-09T17:58:43.370 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7766000000000002e-05 2024-02-09T17:58:43.370 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.371 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.371 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:58:43.371 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:57.149294+0000", 2024-02-09T17:58:43.371 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0172325710000001 2024-02-09T17:58:43.371 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.371 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.371 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:58:43.371 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:57.149311+0000", 2024-02-09T17:58:43.371 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6489000000000001e-05 2024-02-09T17:58:43.371 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:43.371 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:58:43.371 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:43.371 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.371 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.372 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.16440.0:188793 7.c 7:3c4b8ada:::benchmark_data_smithi179_135804_object81002:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e752)", 2024-02-09T17:58:43.372 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:54:08.154901+0000", 2024-02-09T17:58:43.372 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 275.137566268, 2024-02-09T17:58:43.372 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.31438855300000002, 2024-02-09T17:58:43.372 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:58:43.372 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-09T17:58:43.372 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:58:43.372 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.16440", 2024-02-09T17:58:43.372 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:58:43.372 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 188793 2024-02-09T17:58:43.372 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.372 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:58:43.372 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.372 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:58:43.373 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:08.154901+0000", 2024-02-09T17:58:43.373 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:43.373 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.373 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.373 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:58:43.373 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:08.154901+0000", 2024-02-09T17:58:43.373 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:43.373 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.373 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.373 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:58:43.373 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:08.154902+0000", 2024-02-09T17:58:43.373 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7199999999999999e-07 2024-02-09T17:58:43.373 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.374 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.374 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:58:43.374 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:08.154905+0000", 2024-02-09T17:58:43.374 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2590000000000001e-06 2024-02-09T17:58:43.374 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.374 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.374 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:58:43.374 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:08.154911+0000", 2024-02-09T17:58:43.374 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5090000000000001e-06 2024-02-09T17:58:43.374 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.374 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.374 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:58:43.374 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:08.154917+0000", 2024-02-09T17:58:43.374 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6749999999999996e-06 2024-02-09T17:58:43.375 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.375 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.375 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:58:43.375 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:08.155056+0000", 2024-02-09T17:58:43.375 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013835200000000001 2024-02-09T17:58:43.375 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.375 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.375 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for missing object", 2024-02-09T17:58:43.375 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:08.155116+0000", 2024-02-09T17:58:43.375 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0544000000000002e-05 2024-02-09T17:58:43.375 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.375 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.375 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:58:43.375 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:08.469274+0000", 2024-02-09T17:58:43.375 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.31415796899999998 2024-02-09T17:58:43.376 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.376 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.376 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:58:43.376 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:08.469290+0000", 2024-02-09T17:58:43.376 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5773e-05 2024-02-09T17:58:43.376 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:43.376 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:58:43.376 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:43.376 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.376 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.376 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.16440.0:188801 7.c 7:33962c26:::benchmark_data_smithi179_135804_object81010:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e752)", 2024-02-09T17:58:43.376 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:54:08.154928+0000", 2024-02-09T17:58:43.376 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 275.13753968200001, 2024-02-09T17:58:43.376 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.31720087699999999, 2024-02-09T17:58:43.377 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:58:43.377 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-09T17:58:43.377 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:58:43.377 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.16440", 2024-02-09T17:58:43.377 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:58:43.377 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 188801 2024-02-09T17:58:43.377 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.377 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:58:43.377 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.377 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:58:43.377 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:08.154928+0000", 2024-02-09T17:58:43.377 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:43.377 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.377 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.378 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:58:43.378 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:08.154928+0000", 2024-02-09T17:58:43.378 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:43.378 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.378 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.378 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:58:43.378 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:08.154928+0000", 2024-02-09T17:58:43.378 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.89e-07 2024-02-09T17:58:43.378 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.378 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.378 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:58:43.378 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:08.154931+0000", 2024-02-09T17:58:43.378 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5600000000000001e-06 2024-02-09T17:58:43.378 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.378 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.379 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:58:43.379 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:08.154938+0000", 2024-02-09T17:58:43.379 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5549999999999999e-06 2024-02-09T17:58:43.379 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.379 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.379 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:58:43.379 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:08.154943+0000", 2024-02-09T17:58:43.379 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7599999999999999e-06 2024-02-09T17:58:43.379 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.379 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.379 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:58:43.379 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:08.155149+0000", 2024-02-09T17:58:43.379 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020600300000000001 2024-02-09T17:58:43.380 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.380 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.380 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for missing object", 2024-02-09T17:58:43.380 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:08.155201+0000", 2024-02-09T17:58:43.380 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1898999999999999e-05 2024-02-09T17:58:43.380 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.380 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.380 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:58:43.380 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:08.472113+0000", 2024-02-09T17:58:43.380 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.31691208900000001 2024-02-09T17:58:43.380 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.380 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.380 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:58:43.381 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:08.472129+0000", 2024-02-09T17:58:43.381 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5622000000000001e-05 2024-02-09T17:58:43.381 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:43.381 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:58:43.381 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:43.381 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.381 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.381 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.16440.0:191308 7.c 7:3bb83098:::benchmark_data_smithi179_135804_object83517:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e754)", 2024-02-09T17:58:43.381 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:54:11.757068+0000", 2024-02-09T17:58:43.381 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 271.53539966400001, 2024-02-09T17:58:43.381 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38417522799999998, 2024-02-09T17:58:43.381 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:58:43.381 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-09T17:58:43.382 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:58:43.382 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.16440", 2024-02-09T17:58:43.382 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:58:43.382 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 191308 2024-02-09T17:58:43.382 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.382 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:58:43.382 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.382 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:58:43.382 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:11.757068+0000", 2024-02-09T17:58:43.382 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:43.382 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.382 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.382 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:58:43.382 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:11.757068+0000", 2024-02-09T17:58:43.383 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:43.383 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.383 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.383 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:58:43.383 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:11.757068+0000", 2024-02-09T17:58:43.383 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2100000000000002e-07 2024-02-09T17:58:43.383 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.383 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.383 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:58:43.383 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:11.757071+0000", 2024-02-09T17:58:43.383 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.238e-06 2024-02-09T17:58:43.383 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.383 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.384 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:58:43.384 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:11.757077+0000", 2024-02-09T17:58:43.384 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3520000000000003e-06 2024-02-09T17:58:43.384 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.384 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.384 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:58:43.384 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:11.757082+0000", 2024-02-09T17:58:43.384 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8280000000000002e-06 2024-02-09T17:58:43.384 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.384 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.384 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:58:43.384 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:11.758782+0000", 2024-02-09T17:58:43.384 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001699629 2024-02-09T17:58:43.385 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.385 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.385 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for missing object", 2024-02-09T17:58:43.385 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:11.758839+0000", 2024-02-09T17:58:43.385 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7797000000000002e-05 2024-02-09T17:58:43.385 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.385 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.385 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:58:43.385 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.141219+0000", 2024-02-09T17:58:43.385 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38237959700000002 2024-02-09T17:58:43.385 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.385 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.385 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:58:43.385 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.141243+0000", 2024-02-09T17:58:43.385 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4366e-05 2024-02-09T17:58:43.386 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:43.386 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:58:43.386 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:43.386 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.386 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.386 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDScrubReserve(1.0,RELEASE,e:818,from:7,reservation_nonce:0)", 2024-02-09T17:58:43.386 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:56:46.401901+0000", 2024-02-09T17:58:43.386 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 116.890566561, 2024-02-09T17:58:43.386 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017382300000000001, 2024-02-09T17:58:43.386 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:58:43.386 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-09T17:58:43.386 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:58:43.386 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.386 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:58:43.387 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:56:46.401901+0000", 2024-02-09T17:58:43.387 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:43.387 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.387 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.387 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:58:43.387 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:56:46.401901+0000", 2024-02-09T17:58:43.387 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:43.387 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.387 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.387 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:58:43.387 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:56:46.401902+0000", 2024-02-09T17:58:43.387 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.77e-07 2024-02-09T17:58:43.387 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.387 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.387 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:58:43.388 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:56:46.401904+0000", 2024-02-09T17:58:43.388 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9299999999999999e-06 2024-02-09T17:58:43.388 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.388 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.388 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:58:43.388 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:56:46.401911+0000", 2024-02-09T17:58:43.388 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.55e-06 2024-02-09T17:58:43.388 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.388 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.388 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:58:43.389 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:56:46.401918+0000", 2024-02-09T17:58:43.389 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5039999999999999e-06 2024-02-09T17:58:43.389 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.389 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.389 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:58:43.389 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:56:46.402027+0000", 2024-02-09T17:58:43.389 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010969899999999999 2024-02-09T17:58:43.389 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.389 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.389 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:58:43.389 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:56:46.402047+0000", 2024-02-09T17:58:43.389 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9746999999999999e-05 2024-02-09T17:58:43.389 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.389 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.390 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:58:43.390 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:56:46.402075+0000", 2024-02-09T17:58:43.390 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8016000000000001e-05 2024-02-09T17:58:43.390 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:43.390 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:58:43.390 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:43.390 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.390 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.390 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "replica_scrub(pg: 1.19,from:0'0,to:0'0,epoch:820/790,start:1:98000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-02-09T17:58:43.390 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:56:52.467549+0000", 2024-02-09T17:58:43.390 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 110.824919016, 2024-02-09T17:58:43.390 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000172409, 2024-02-09T17:58:43.390 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:58:43.390 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-09T17:58:43.391 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:58:43.391 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.391 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:58:43.391 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:56:52.467549+0000", 2024-02-09T17:58:43.391 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:43.391 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.391 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.391 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:58:43.391 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:56:52.467549+0000", 2024-02-09T17:58:43.391 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:43.391 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.391 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.391 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:58:43.391 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:56:52.467550+0000", 2024-02-09T17:58:43.391 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.141e-06 2024-02-09T17:58:43.392 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.392 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.392 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:58:43.392 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:56:52.467557+0000", 2024-02-09T17:58:43.392 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1570000000000003e-06 2024-02-09T17:58:43.392 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.392 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.392 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:58:43.392 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:56:52.467574+0000", 2024-02-09T17:58:43.392 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6969999999999998e-05 2024-02-09T17:58:43.392 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.392 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.392 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:58:43.392 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:56:52.467583+0000", 2024-02-09T17:58:43.393 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9279999999999999e-06 2024-02-09T17:58:43.393 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.393 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.393 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:58:43.393 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:56:52.467670+0000", 2024-02-09T17:58:43.393 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7049e-05 2024-02-09T17:58:43.393 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.393 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.393 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:58:43.393 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:56:52.467683+0000", 2024-02-09T17:58:43.393 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3233e-05 2024-02-09T17:58:43.393 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.393 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.393 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:58:43.394 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:56:52.467721+0000", 2024-02-09T17:58:43.394 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7931e-05 2024-02-09T17:58:43.394 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:43.394 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:58:43.394 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:43.394 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.394 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.394 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDScrubReserve(1.3,RELEASE,e:841,from:1,reservation_nonce:0)", 2024-02-09T17:58:43.394 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:58:16.133205+0000", 2024-02-09T17:58:43.394 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 27.159262842, 2024-02-09T17:58:43.394 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021980400000000001, 2024-02-09T17:58:43.394 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:58:43.394 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-09T17:58:43.394 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:58:43.394 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.394 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:58:43.395 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:58:16.133205+0000", 2024-02-09T17:58:43.395 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:43.395 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.395 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.395 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:58:43.395 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:58:16.133205+0000", 2024-02-09T17:58:43.395 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:43.395 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.395 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.395 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:58:43.395 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:58:16.133205+0000", 2024-02-09T17:58:43.395 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6899999999999998e-07 2024-02-09T17:58:43.395 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.395 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.395 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:58:43.395 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:58:16.133208+0000", 2024-02-09T17:58:43.395 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8880000000000001e-06 2024-02-09T17:58:43.396 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.396 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.396 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:58:43.396 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:58:16.133216+0000", 2024-02-09T17:58:43.396 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2770000000000005e-06 2024-02-09T17:58:43.396 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.396 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.396 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:58:43.396 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:58:16.133224+0000", 2024-02-09T17:58:43.396 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5950000000000001e-06 2024-02-09T17:58:43.396 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.396 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.396 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:58:43.396 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:58:16.133376+0000", 2024-02-09T17:58:43.396 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015204500000000001 2024-02-09T17:58:43.397 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.397 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.397 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:58:43.397 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:58:16.133396+0000", 2024-02-09T17:58:43.397 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0296999999999999e-05 2024-02-09T17:58:43.397 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.397 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.397 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:58:43.397 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:58:16.133425+0000", 2024-02-09T17:58:43.397 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8333000000000001e-05 2024-02-09T17:58:43.397 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:43.397 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:58:43.397 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:43.397 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.397 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.397 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDScrubReserve(1.f,GRANT,e:842,from:5,reservation_nonce:0)", 2024-02-09T17:58:43.398 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:58:17.106016+0000", 2024-02-09T17:58:43.398 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 26.186451250000001, 2024-02-09T17:58:43.398 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000174601, 2024-02-09T17:58:43.398 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:58:43.398 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-09T17:58:43.398 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:58:43.398 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.398 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:58:43.398 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:58:17.106016+0000", 2024-02-09T17:58:43.398 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:43.398 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.398 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.398 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:58:43.398 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:58:17.106016+0000", 2024-02-09T17:58:43.398 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:43.398 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.399 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.399 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:58:43.399 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:58:17.106017+0000", 2024-02-09T17:58:43.399 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4199999999999995e-07 2024-02-09T17:58:43.399 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.399 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.399 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:58:43.399 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:58:17.106020+0000", 2024-02-09T17:58:43.399 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7499999999999999e-06 2024-02-09T17:58:43.399 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.399 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.399 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:58:43.399 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:58:17.106029+0000", 2024-02-09T17:58:43.399 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9409999999999996e-06 2024-02-09T17:58:43.399 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.399 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.400 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:58:43.400 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:58:17.106034+0000", 2024-02-09T17:58:43.400 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1610000000000002e-06 2024-02-09T17:58:43.400 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.400 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.400 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:58:43.400 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:58:17.106076+0000", 2024-02-09T17:58:43.400 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1606000000000001e-05 2024-02-09T17:58:43.400 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.400 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.400 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:58:43.400 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:58:17.106087+0000", 2024-02-09T17:58:43.400 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1435e-05 2024-02-09T17:58:43.400 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.400 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.400 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:58:43.401 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:58:17.106191+0000", 2024-02-09T17:58:43.401 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000104066 2024-02-09T17:58:43.401 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:43.401 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:58:43.401 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:43.401 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.401 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.401 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "rep_scrubmap(1.f e266 from shard 5)", 2024-02-09T17:58:43.401 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:58:17.296127+0000", 2024-02-09T17:58:43.401 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 25.996341022999999, 2024-02-09T17:58:43.401 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0019751349999999998, 2024-02-09T17:58:43.402 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:58:43.402 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-09T17:58:43.402 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:58:43.402 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.402 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:58:43.402 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:58:17.296127+0000", 2024-02-09T17:58:43.402 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:43.402 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.402 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.402 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:58:43.402 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:58:17.296127+0000", 2024-02-09T17:58:43.402 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:43.403 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.403 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.403 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:58:43.403 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:58:17.296128+0000", 2024-02-09T17:58:43.403 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.279e-06 2024-02-09T17:58:43.403 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.403 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.403 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:58:43.403 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:58:17.296136+0000", 2024-02-09T17:58:43.403 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9170000000000006e-06 2024-02-09T17:58:43.403 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.403 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.403 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:58:43.403 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:58:17.296156+0000", 2024-02-09T17:58:43.404 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0336999999999999e-05 2024-02-09T17:58:43.404 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.404 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.404 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:58:43.404 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:58:17.296165+0000", 2024-02-09T17:58:43.404 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0839999999999998e-06 2024-02-09T17:58:43.404 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.404 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.404 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:58:43.404 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:58:17.298073+0000", 2024-02-09T17:58:43.404 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001907958 2024-02-09T17:58:43.404 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.405 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.405 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:58:43.405 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:58:17.298088+0000", 2024-02-09T17:58:43.405 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.449e-05 2024-02-09T17:58:43.405 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:43.405 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:43.405 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:58:43.405 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:58:17.298102+0000", 2024-02-09T17:58:43.405 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4070000000000001e-05 2024-02-09T17:58:43.405 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:43.405 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:58:43.405 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:43.405 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:43.406 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:58:43.406 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:58:43.406 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:58:43.407 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-09T17:58:43.407 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-09T17:58:43.475 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:43.501 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:58:43.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:43.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:58:43.615 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:58:43.616 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset noscrub 2024-02-09T17:58:43.650 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:43.752 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:58:43.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:43.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:58:43.860 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:58:43.890 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:43.966 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:58:44.095 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:58:44.097 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-09T17:58:44.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:44.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:58:44.199 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:58:44.234 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:44.259 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:58:44.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:44.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:58:44.366 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:58:44.400 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:44.426 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:58:44.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:44.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:58:44.484 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:58:44.530 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:58:44.564 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:44.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:44 smithi179 ceph-mon[43975]: pgmap v2106: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 2.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 593 op/s; 7736/124764 objects degraded (6.201%) 2024-02-09T17:58:44.924 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:58:44.926 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:44 smithi099 ceph-mon[39420]: pgmap v2106: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 2.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 593 op/s; 7736/124764 objects degraded (6.201%) 2024-02-09T17:58:44.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:44 smithi162 ceph-mon[44304]: pgmap v2106: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 2.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 593 op/s; 7736/124764 objects degraded (6.201%) 2024-02-09T17:58:44.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:44.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:58:45.028 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:58:45.059 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:45.315 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:58:45.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:45.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:58:45.409 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:58:45.421 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:58:45.471 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:45.502 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:58:45.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:45.505 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:58:45.606 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:58:45.639 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:45.891 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:58:45.896 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:45.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:58:45.997 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:58:46.062 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 2] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2, 4, 5, 6, 7, 9, 10, 11, 8] 2024-02-09T17:58:46.063 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:58:46.063 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-02-09T17:58:46.063 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-09T17:58:46.063 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-09T17:58:46.063 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-02-09T17:58:46.064 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:46.109 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:58:46.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:46.113 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:58:46.214 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:58:46.269 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:46.297 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:58:46.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:46.303 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:58:46.404 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:58:46.439 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:46.496 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:58:46.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:46.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:58:46.580 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:58:46.580 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-09T17:58:46.580 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-02-09T17:58:46.580 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:58:46.600 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:58:46.649 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:46.696 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:58:46.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:46.711 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:58:46.812 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:58:46.839 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:46 smithi162 ceph-mon[44304]: pgmap v2107: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 2.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 640 op/s; 7736/120393 objects degraded (6.426%) 2024-02-09T17:58:46.841 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:46.863 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:58:46.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:46.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:58:46.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:46 smithi179 ceph-mon[43975]: pgmap v2107: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 2.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 640 op/s; 7736/120393 objects degraded (6.426%) 2024-02-09T17:58:46.967 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:58:46.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:46 smithi099 ceph-mon[39420]: pgmap v2107: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 2.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 640 op/s; 7736/120393 objects degraded (6.426%) 2024-02-09T17:58:47.014 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:47.068 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:58:47.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:47.076 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:58:47.177 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:58:47.232 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:47.267 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:58:47.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:47.273 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:58:47.349 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-09T17:58:47.374 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:58:47.410 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:47.667 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:58:47.670 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:47 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 7736/120393 objects degraded (6.426%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:58:47.670 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:47 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 7736/120393 objects degraded (6.426%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:58:48.070 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:58:48.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:48.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:58:48.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:47 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 7736/120393 objects degraded (6.426%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:58:48.177 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:58:48.223 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:48.278 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:58:48.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:48.285 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:58:48.386 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:58:48.423 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:48.555 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:58:48.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:48.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:58:48.658 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:58:48.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:48 smithi179 ceph-mon[43975]: pgmap v2108: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 2.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 668 op/s; 7736/117438 objects degraded (6.587%) 2024-02-09T17:58:48.675 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:48 smithi099 ceph-mon[39420]: pgmap v2108: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 2.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 668 op/s; 7736/117438 objects degraded (6.587%) 2024-02-09T17:58:48.704 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:48.750 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:58:48.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:48.756 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:58:48.781 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:58:48.782 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:58:48.857 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:58:48.926 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:48.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:48 smithi162 ceph-mon[44304]: pgmap v2108: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 2.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 668 op/s; 7736/117438 objects degraded (6.587%) 2024-02-09T17:58:48.958 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:58:48.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:48.966 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:58:49.067 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:58:49.115 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:49.150 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:58:49.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:49.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:58:49.255 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:58:49.317 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:49.353 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:58:49.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:49.367 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:58:49.469 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:58:49.509 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:49.541 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:58:49.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:49.547 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:58:49.648 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:58:49.674 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:58:49.674 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-09T17:58:49.674 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-02-09T17:58:49.674 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-02-09T17:58:49.674 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:58:49.686 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:49.692 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:49 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1437388876' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:58:49.692 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:49 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:58:49.698 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is unset 2024-02-09T17:58:49.710 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:58:49.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:49.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:58:49.814 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:58:49.878 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:49.911 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:58:49.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:49.915 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:58:50.016 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:58:50.038 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:49 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1437388876' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:58:50.038 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:49 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:58:50.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:49 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1437388876' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:58:50.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:49 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:58:50.089 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:50.130 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:58:50.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:50.134 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:58:50.235 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:58:50.304 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:50.329 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:58:50.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:50.332 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:58:50.390 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-09T17:58:50.433 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:58:50.478 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:50.505 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:58:50.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:50.508 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:58:50.609 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:58:50.644 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:50.671 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:58:50.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:50.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:58:50.717 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T17:58:50.779 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:58:50.831 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:50 smithi179 ceph-mon[43975]: pgmap v2109: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 2.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 761 op/s; 7001/110178 objects degraded (6.354%) 2024-02-09T17:58:50.831 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:50 smithi179 ceph-mon[43975]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:58:50.831 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:50 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:58:50.831 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:50 smithi179 ceph-mon[43975]: osdmap e848: 12 total, 11 up, 12 in 2024-02-09T17:58:50.831 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:50 smithi179 ceph-mon[43975]: 8.9 scrub starts 2024-02-09T17:58:50.837 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:50.872 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:58:50.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:50.874 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:58:50.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:50 smithi162 ceph-mon[44304]: pgmap v2109: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 2.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 761 op/s; 7001/110178 objects degraded (6.354%) 2024-02-09T17:58:50.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:50 smithi162 ceph-mon[44304]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:58:50.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:50 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:58:50.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:50 smithi162 ceph-mon[44304]: osdmap e848: 12 total, 11 up, 12 in 2024-02-09T17:58:50.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:50 smithi162 ceph-mon[44304]: 8.9 scrub starts 2024-02-09T17:58:50.974 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:58:50.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:50 smithi099 ceph-mon[39420]: pgmap v2109: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 2.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 761 op/s; 7001/110178 objects degraded (6.354%) 2024-02-09T17:58:50.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:50 smithi099 ceph-mon[39420]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:58:50.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:50 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:58:50.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:50 smithi099 ceph-mon[39420]: osdmap e848: 12 total, 11 up, 12 in 2024-02-09T17:58:50.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:50 smithi099 ceph-mon[39420]: 8.9 scrub starts 2024-02-09T17:58:51.013 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:51.924 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:51 smithi162 ceph-mon[44304]: 8.9 scrub ok 2024-02-09T17:58:51.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:51 smithi099 ceph-mon[39420]: 8.9 scrub ok 2024-02-09T17:58:52.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:51 smithi179 ceph-mon[43975]: 8.9 scrub ok 2024-02-09T17:58:52.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:52 smithi099 ceph-mon[39420]: pgmap v2111: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 758 op/s; 7001/108615 objects degraded (6.446%) 2024-02-09T17:58:52.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:52 smithi099 ceph-mon[39420]: 8.b scrub starts 2024-02-09T17:58:52.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:52 smithi099 ceph-mon[39420]: 1.15 scrub starts 2024-02-09T17:58:52.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:52 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 7001/108615 objects degraded (6.446%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:58:52.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:52 smithi099 ceph-mon[39420]: 1.15 scrub ok 2024-02-09T17:58:52.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:52 smithi099 ceph-mon[39420]: 8.b scrub ok 2024-02-09T17:58:52.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:52 smithi162 ceph-mon[44304]: pgmap v2111: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 758 op/s; 7001/108615 objects degraded (6.446%) 2024-02-09T17:58:52.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:52 smithi162 ceph-mon[44304]: 8.b scrub starts 2024-02-09T17:58:52.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:52 smithi162 ceph-mon[44304]: 1.15 scrub starts 2024-02-09T17:58:52.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:52 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 7001/108615 objects degraded (6.446%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:58:52.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:52 smithi162 ceph-mon[44304]: 1.15 scrub ok 2024-02-09T17:58:52.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:52 smithi162 ceph-mon[44304]: 8.b scrub ok 2024-02-09T17:58:52.956 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T17:58:52.956 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-02-09T17:58:52.956 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-02-09T17:58:52.956 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-02-09T17:58:52.956 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.956 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25837.0:254297 6.3 6:cd45665f:::benchmark_data_smithi179_126206_object123916:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:58:52.956 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:49:52.578440+0000", 2024-02-09T17:58:52.956 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 540.37405843399995, 2024-02-09T17:58:52.956 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.218960764, 2024-02-09T17:58:52.957 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:58:52.957 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:58:52.957 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:58:52.957 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25837", 2024-02-09T17:58:52.957 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:58:52.957 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 254297 2024-02-09T17:58:52.957 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.957 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:58:52.957 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.957 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:58:52.957 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.578440+0000", 2024-02-09T17:58:52.957 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:52.957 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.957 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.957 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:58:52.958 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.578440+0000", 2024-02-09T17:58:52.958 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:52.958 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.958 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.958 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:58:52.958 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.578441+0000", 2024-02-09T17:58:52.958 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.42e-06 2024-02-09T17:58:52.958 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.958 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.958 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:58:52.958 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.578445+0000", 2024-02-09T17:58:52.958 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6119999999999999e-06 2024-02-09T17:58:52.958 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.958 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.958 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:58:52.958 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.578454+0000", 2024-02-09T17:58:52.959 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9279999999999999e-06 2024-02-09T17:58:52.959 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.959 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.959 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:58:52.959 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.578460+0000", 2024-02-09T17:58:52.959 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.472e-06 2024-02-09T17:58:52.959 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.959 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.959 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:58:52.959 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.578519+0000", 2024-02-09T17:58:52.959 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8715999999999998e-05 2024-02-09T17:58:52.959 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.959 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.959 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:58:52.959 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.578651+0000", 2024-02-09T17:58:52.960 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013221500000000001 2024-02-09T17:58:52.960 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.960 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.960 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-02-09T17:58:52.960 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.578764+0000", 2024-02-09T17:58:52.960 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000112608 2024-02-09T17:58:52.960 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.960 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.960 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:58:52.960 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.588646+0000", 2024-02-09T17:58:52.960 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0098819219999999996 2024-02-09T17:58:52.960 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.960 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.960 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:52.961 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.593408+0000", 2024-02-09T17:58:52.961 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0047621299999999998 2024-02-09T17:58:52.961 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.961 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.961 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:52.961 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.797292+0000", 2024-02-09T17:58:52.961 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20388404099999999 2024-02-09T17:58:52.961 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.961 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.961 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:58:52.961 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.797354+0000", 2024-02-09T17:58:52.961 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2098000000000001e-05 2024-02-09T17:58:52.961 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.962 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.962 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:58:52.962 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.797401+0000", 2024-02-09T17:58:52.962 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6601999999999999e-05 2024-02-09T17:58:52.962 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:52.962 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:58:52.962 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:52.962 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.962 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.962 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25837.0:254298 6.3 6:c9777fa5:::benchmark_data_smithi179_126206_object123917:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:58:52.962 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:49:52.578475+0000", 2024-02-09T17:58:52.962 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 540.37402362600005, 2024-02-09T17:58:52.962 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24596101500000001, 2024-02-09T17:58:52.962 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:58:52.963 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:58:52.963 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:58:52.963 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25837", 2024-02-09T17:58:52.963 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:58:52.963 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 254298 2024-02-09T17:58:52.963 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.963 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:58:52.963 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.963 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:58:52.963 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.578475+0000", 2024-02-09T17:58:52.963 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:52.963 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.963 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.964 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:58:52.964 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.578475+0000", 2024-02-09T17:58:52.964 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:52.964 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.964 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.964 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:58:52.964 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.578475+0000", 2024-02-09T17:58:52.964 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1399999999999997e-07 2024-02-09T17:58:52.964 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.964 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.964 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:58:52.964 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.578478+0000", 2024-02-09T17:58:52.964 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2759999999999999e-06 2024-02-09T17:58:52.965 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.965 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.965 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:58:52.965 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583629+0000", 2024-02-09T17:58:52.965 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0051514940000000004 2024-02-09T17:58:52.965 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.965 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.965 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:58:52.965 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583640+0000", 2024-02-09T17:58:52.965 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0791999999999999e-05 2024-02-09T17:58:52.965 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.965 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.965 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:58:52.965 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.584392+0000", 2024-02-09T17:58:52.965 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00075225900000000002 2024-02-09T17:58:52.966 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.966 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.966 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:58:52.966 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.584488+0000", 2024-02-09T17:58:52.966 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5838e-05 2024-02-09T17:58:52.966 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.966 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.966 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-02-09T17:58:52.966 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.584555+0000", 2024-02-09T17:58:52.966 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7478999999999995e-05 2024-02-09T17:58:52.966 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.966 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.966 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:58:52.967 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.593428+0000", 2024-02-09T17:58:52.967 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0088728939999999992 2024-02-09T17:58:52.967 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.967 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.967 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:52.967 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.596904+0000", 2024-02-09T17:58:52.967 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034754249999999999 2024-02-09T17:58:52.967 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.967 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.967 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:52.967 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.824355+0000", 2024-02-09T17:58:52.967 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.227451715 2024-02-09T17:58:52.967 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.967 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.967 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:58:52.968 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.824395+0000", 2024-02-09T17:58:52.968 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9900000000000001e-05 2024-02-09T17:58:52.968 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.968 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.968 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:58:52.968 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.824436+0000", 2024-02-09T17:58:52.968 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0528999999999998e-05 2024-02-09T17:58:52.968 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:52.968 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:58:52.968 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:52.968 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.968 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.968 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25837.0:254300 6.5 6:aac8f1b4:::benchmark_data_smithi179_126206_object123919:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:58:52.968 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:49:52.583663+0000", 2024-02-09T17:58:52.969 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 540.36883496999997, 2024-02-09T17:58:52.969 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.212262326, 2024-02-09T17:58:52.969 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:58:52.969 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:58:52.969 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:58:52.969 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25837", 2024-02-09T17:58:52.969 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:58:52.969 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 254300 2024-02-09T17:58:52.969 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.969 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:58:52.969 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.969 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:58:52.969 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583663+0000", 2024-02-09T17:58:52.969 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:52.970 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.970 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.970 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:58:52.970 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583663+0000", 2024-02-09T17:58:52.970 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:52.971 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.971 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.971 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:58:52.971 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583665+0000", 2024-02-09T17:58:52.971 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.217e-06 2024-02-09T17:58:52.971 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.971 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.971 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:58:52.971 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583669+0000", 2024-02-09T17:58:52.971 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4739999999999999e-06 2024-02-09T17:58:52.971 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.972 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.972 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:58:52.972 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583679+0000", 2024-02-09T17:58:52.972 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7559999999999994e-06 2024-02-09T17:58:52.972 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.972 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.972 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:58:52.972 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583685+0000", 2024-02-09T17:58:52.972 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6130000000000003e-06 2024-02-09T17:58:52.972 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.972 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.972 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:58:52.972 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583729+0000", 2024-02-09T17:58:52.972 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4237999999999999e-05 2024-02-09T17:58:52.973 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.973 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.973 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:58:52.973 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583847+0000", 2024-02-09T17:58:52.973 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000118322 2024-02-09T17:58:52.973 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.973 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.973 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-02-09T17:58:52.973 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583952+0000", 2024-02-09T17:58:52.973 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000105154 2024-02-09T17:58:52.974 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.974 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.974 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:52.974 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.593158+0000", 2024-02-09T17:58:52.974 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0092052690000000003 2024-02-09T17:58:52.974 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.974 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.974 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:58:52.974 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.593588+0000", 2024-02-09T17:58:52.974 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000430176 2024-02-09T17:58:52.974 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.974 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.974 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:52.975 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.795858+0000", 2024-02-09T17:58:52.975 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20227031700000001 2024-02-09T17:58:52.975 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.975 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.975 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:58:52.975 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.795896+0000", 2024-02-09T17:58:52.975 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8470999999999998e-05 2024-02-09T17:58:52.975 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.975 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.975 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:58:52.975 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.795926+0000", 2024-02-09T17:58:52.975 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9319000000000002e-05 2024-02-09T17:58:52.975 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:52.975 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:58:52.975 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:52.976 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.976 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.976 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25837.0:254302 6.5 6:ac558352:::benchmark_data_smithi179_126206_object123921:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:58:52.976 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:49:52.583698+0000", 2024-02-09T17:58:52.976 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 540.36880056300004, 2024-02-09T17:58:52.976 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22801817599999999, 2024-02-09T17:58:52.976 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:58:52.976 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:58:52.976 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:58:52.976 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25837", 2024-02-09T17:58:52.976 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:58:52.976 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 254302 2024-02-09T17:58:52.976 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.977 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:58:52.977 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.977 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:58:52.977 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583698+0000", 2024-02-09T17:58:52.977 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:52.977 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.977 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.977 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:58:52.977 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583698+0000", 2024-02-09T17:58:52.977 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:52.977 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.977 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.977 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:58:52.977 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583698+0000", 2024-02-09T17:58:52.977 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9499999999999998e-07 2024-02-09T17:58:52.978 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.978 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.978 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:58:52.978 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583701+0000", 2024-02-09T17:58:52.978 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3760000000000002e-06 2024-02-09T17:58:52.978 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.978 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.978 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:58:52.978 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583707+0000", 2024-02-09T17:58:52.978 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.241e-06 2024-02-09T17:58:52.978 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.978 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.978 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:58:52.978 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.583712+0000", 2024-02-09T17:58:52.979 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1800000000000004e-06 2024-02-09T17:58:52.979 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.979 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.979 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:58:52.979 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.589123+0000", 2024-02-09T17:58:52.979 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0054113909999999998 2024-02-09T17:58:52.979 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.979 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.979 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:58:52.979 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.589215+0000", 2024-02-09T17:58:52.979 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1038999999999994e-05 2024-02-09T17:58:52.979 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.979 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.979 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-02-09T17:58:52.979 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.589298+0000", 2024-02-09T17:58:52.980 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3084999999999995e-05 2024-02-09T17:58:52.980 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.980 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.980 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:58:52.980 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.593594+0000", 2024-02-09T17:58:52.980 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042961290000000001 2024-02-09T17:58:52.980 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.980 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.980 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:52.980 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.594360+0000", 2024-02-09T17:58:52.980 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00076610199999999995 2024-02-09T17:58:52.981 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.981 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.981 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:52.981 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.811612+0000", 2024-02-09T17:58:52.981 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21725206 2024-02-09T17:58:52.981 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.981 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.981 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:58:52.981 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.811672+0000", 2024-02-09T17:58:52.981 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9901000000000001e-05 2024-02-09T17:58:52.981 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.981 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.981 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:58:52.981 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.811716+0000", 2024-02-09T17:58:52.982 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4277000000000003e-05 2024-02-09T17:58:52.982 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:52.982 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:58:52.982 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:52.982 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.982 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.982 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25837.0:254319 6.3 6:c6f40c4d:::benchmark_data_smithi179_126206_object123938:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:58:52.982 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:49:52.603086+0000", 2024-02-09T17:58:52.982 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 540.34941261899996, 2024-02-09T17:58:52.982 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22144201199999999, 2024-02-09T17:58:52.982 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:58:52.982 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:58:52.982 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:58:52.982 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25837", 2024-02-09T17:58:52.983 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:58:52.983 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 254319 2024-02-09T17:58:52.983 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.983 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:58:52.983 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.983 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:58:52.983 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.603086+0000", 2024-02-09T17:58:52.983 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:52.983 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.983 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.983 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:58:52.983 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.603086+0000", 2024-02-09T17:58:52.983 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:52.983 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.983 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.983 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:58:52.984 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.603087+0000", 2024-02-09T17:58:52.984 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5409999999999999e-06 2024-02-09T17:58:52.984 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.984 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.984 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:58:52.984 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.603093+0000", 2024-02-09T17:58:52.984 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9599999999999997e-06 2024-02-09T17:58:52.984 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.984 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.984 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:58:52.984 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.603106+0000", 2024-02-09T17:58:52.984 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2849e-05 2024-02-09T17:58:52.984 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.984 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.984 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:58:52.984 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.603114+0000", 2024-02-09T17:58:52.985 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9689999999999994e-06 2024-02-09T17:58:52.985 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.985 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.985 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:58:52.985 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.603179+0000", 2024-02-09T17:58:52.985 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4348000000000001e-05 2024-02-09T17:58:52.985 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.985 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.985 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:58:52.985 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.603326+0000", 2024-02-09T17:58:52.985 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014761700000000001 2024-02-09T17:58:52.985 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.985 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.985 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-02-09T17:58:52.985 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.603460+0000", 2024-02-09T17:58:52.985 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013390799999999999 2024-02-09T17:58:52.985 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.986 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.986 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:52.986 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.609555+0000", 2024-02-09T17:58:52.986 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0060945360000000002 2024-02-09T17:58:52.986 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.986 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.986 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:58:52.986 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.610250+0000", 2024-02-09T17:58:52.986 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00069547599999999999 2024-02-09T17:58:52.986 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.986 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.986 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:52.986 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.824478+0000", 2024-02-09T17:58:52.986 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.214227852 2024-02-09T17:58:52.986 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.987 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.987 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:58:52.987 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.824507+0000", 2024-02-09T17:58:52.987 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9278e-05 2024-02-09T17:58:52.987 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.987 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.987 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:58:52.987 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.824528+0000", 2024-02-09T17:58:52.987 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0678000000000001e-05 2024-02-09T17:58:52.987 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:52.987 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:58:52.987 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:52.987 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.987 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.987 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25837.0:254323 6.5 6:aaac2b69:::benchmark_data_smithi179_126206_object123942:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:58:52.987 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:49:52.604437+0000", 2024-02-09T17:58:52.988 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 540.34806164400004, 2024-02-09T17:58:52.988 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.217027939, 2024-02-09T17:58:52.988 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:58:52.988 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:58:52.988 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:58:52.988 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25837", 2024-02-09T17:58:52.988 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:58:52.988 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 254323 2024-02-09T17:58:52.988 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.988 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:58:52.988 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.988 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:58:52.988 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.604437+0000", 2024-02-09T17:58:52.988 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:52.988 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.989 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.989 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:58:52.989 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.604437+0000", 2024-02-09T17:58:52.989 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:52.989 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.989 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.989 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:58:52.989 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.604438+0000", 2024-02-09T17:58:52.989 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.051e-06 2024-02-09T17:58:52.989 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.989 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.989 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:58:52.989 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.604443+0000", 2024-02-09T17:58:52.989 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.147e-06 2024-02-09T17:58:52.990 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.990 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.990 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:58:52.990 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.604456+0000", 2024-02-09T17:58:52.990 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3414000000000001e-05 2024-02-09T17:58:52.990 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.990 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.990 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:58:52.990 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.604465+0000", 2024-02-09T17:58:52.990 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5359999999999992e-06 2024-02-09T17:58:52.990 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.990 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.990 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:58:52.990 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.604517+0000", 2024-02-09T17:58:52.990 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2478999999999997e-05 2024-02-09T17:58:52.991 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.991 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.991 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:58:52.991 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.604630+0000", 2024-02-09T17:58:52.991 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011217099999999999 2024-02-09T17:58:52.991 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.991 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.991 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-02-09T17:58:52.991 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.604706+0000", 2024-02-09T17:58:52.991 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6616e-05 2024-02-09T17:58:52.991 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.991 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.991 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:52.991 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.609972+0000", 2024-02-09T17:58:52.991 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0052659459999999996 2024-02-09T17:58:52.992 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.992 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.992 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:58:52.992 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.610237+0000", 2024-02-09T17:58:52.992 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00026531299999999999 2024-02-09T17:58:52.992 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.992 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.992 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:52.992 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.821367+0000", 2024-02-09T17:58:52.992 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21112936800000001 2024-02-09T17:58:52.992 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.992 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.992 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:58:52.992 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.821423+0000", 2024-02-09T17:58:52.993 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6109000000000001e-05 2024-02-09T17:58:52.993 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.993 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.993 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:58:52.993 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.821465+0000", 2024-02-09T17:58:52.993 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1789000000000003e-05 2024-02-09T17:58:52.993 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:52.993 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:58:52.993 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:52.993 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.993 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.993 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25837.0:254329 6.5 6:abdecdae:::benchmark_data_smithi179_126206_object123948:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:58:52.993 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:49:52.613907+0000", 2024-02-09T17:58:52.993 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 540.33859127300002, 2024-02-09T17:58:52.993 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20766915699999999, 2024-02-09T17:58:52.994 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:58:52.994 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:58:52.994 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:58:52.994 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25837", 2024-02-09T17:58:52.994 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:58:52.994 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 254329 2024-02-09T17:58:52.994 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.994 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:58:52.994 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.994 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:58:52.994 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.613907+0000", 2024-02-09T17:58:52.994 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:52.994 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.994 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.994 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:58:52.995 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.613907+0000", 2024-02-09T17:58:52.995 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:52.995 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.995 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.995 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:58:52.995 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.613909+0000", 2024-02-09T17:58:52.995 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6929999999999999e-06 2024-02-09T17:58:52.995 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.995 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.995 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:58:52.995 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.613915+0000", 2024-02-09T17:58:52.995 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4230000000000002e-06 2024-02-09T17:58:52.995 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.995 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.995 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:58:52.996 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.613930+0000", 2024-02-09T17:58:52.996 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5007999999999999e-05 2024-02-09T17:58:52.996 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.996 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.996 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:58:52.996 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.613940+0000", 2024-02-09T17:58:52.996 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5000000000000005e-06 2024-02-09T17:58:52.996 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.996 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.996 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:58:52.996 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.614001+0000", 2024-02-09T17:58:52.996 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0888999999999999e-05 2024-02-09T17:58:52.996 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.996 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.996 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:58:52.997 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.614152+0000", 2024-02-09T17:58:52.997 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015174200000000001 2024-02-09T17:58:52.997 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.997 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.997 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-02-09T17:58:52.997 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.614283+0000", 2024-02-09T17:58:52.997 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013045099999999999 2024-02-09T17:58:52.997 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.997 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.997 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:58:52.997 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.618788+0000", 2024-02-09T17:58:52.997 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0045055859999999998 2024-02-09T17:58:52.997 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.997 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.998 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:52.998 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.620381+0000", 2024-02-09T17:58:52.998 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0015923070000000001 2024-02-09T17:58:52.998 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.998 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.998 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:52.998 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.821500+0000", 2024-02-09T17:58:52.998 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20111922300000001 2024-02-09T17:58:52.998 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.998 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.998 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:58:52.998 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.821545+0000", 2024-02-09T17:58:52.998 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4802000000000002e-05 2024-02-09T17:58:52.998 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.999 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.999 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:58:52.999 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.821576+0000", 2024-02-09T17:58:52.999 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1532999999999998e-05 2024-02-09T17:58:52.999 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:52.999 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:58:52.999 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:52.999 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:52.999 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:52.999 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25837.0:254336 6.3 6:c045a3b9:::benchmark_data_smithi179_126206_object123955:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:58:52.999 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:49:52.620572+0000", 2024-02-09T17:58:52.999 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 540.33192621800004, 2024-02-09T17:58:52.999 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.204024654, 2024-02-09T17:58:52.999 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:58:52.999 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:58:52.999 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:58:53.000 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25837", 2024-02-09T17:58:53.000 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:58:53.000 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 254336 2024-02-09T17:58:53.000 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.000 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:58:53.000 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.000 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:58:53.000 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.620572+0000", 2024-02-09T17:58:53.000 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:53.000 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.000 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.000 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:58:53.000 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.620572+0000", 2024-02-09T17:58:53.000 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:53.001 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.001 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.001 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:58:53.001 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.620574+0000", 2024-02-09T17:58:53.001 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.826e-06 2024-02-09T17:58:53.001 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.001 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.001 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:58:53.001 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.620579+0000", 2024-02-09T17:58:53.001 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2039999999999996e-06 2024-02-09T17:58:53.001 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.001 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.001 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:58:53.001 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.620592+0000", 2024-02-09T17:58:53.001 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3052e-05 2024-02-09T17:58:53.002 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.002 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.002 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:58:53.002 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.620600+0000", 2024-02-09T17:58:53.002 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5610000000000001e-06 2024-02-09T17:58:53.002 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.002 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.002 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:58:53.002 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.620656+0000", 2024-02-09T17:58:53.002 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5738000000000001e-05 2024-02-09T17:58:53.002 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.002 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.002 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:58:53.002 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.620773+0000", 2024-02-09T17:58:53.003 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000117471 2024-02-09T17:58:53.003 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.003 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.003 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-02-09T17:58:53.003 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.620885+0000", 2024-02-09T17:58:53.003 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000111544 2024-02-09T17:58:53.003 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.003 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.003 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:53.003 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.626153+0000", 2024-02-09T17:58:53.003 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0052686039999999996 2024-02-09T17:58:53.003 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.003 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.003 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:58:53.003 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.628281+0000", 2024-02-09T17:58:53.004 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021282380000000002 2024-02-09T17:58:53.004 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.004 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.004 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:53.004 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.824552+0000", 2024-02-09T17:58:53.004 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19627041200000001 2024-02-09T17:58:53.004 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.004 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.004 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:58:53.004 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.824579+0000", 2024-02-09T17:58:53.004 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6631e-05 2024-02-09T17:58:53.004 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.004 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.004 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:58:53.005 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.824597+0000", 2024-02-09T17:58:53.005 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8372999999999998e-05 2024-02-09T17:58:53.005 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:53.005 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:58:53.005 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:53.005 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.005 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.005 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25837.0:254361 6.5 6:addcacb0:::benchmark_data_smithi179_126206_object123980:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:58:53.005 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:49:52.655698+0000", 2024-02-09T17:58:53.005 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 540.29680044500003, 2024-02-09T17:58:53.005 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16599267300000001, 2024-02-09T17:58:53.005 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:58:53.005 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:58:53.005 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:58:53.005 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25837", 2024-02-09T17:58:53.006 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:58:53.006 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 254361 2024-02-09T17:58:53.006 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.006 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:58:53.006 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.006 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:58:53.006 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.655698+0000", 2024-02-09T17:58:53.006 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:53.006 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.006 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.006 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:58:53.006 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.655698+0000", 2024-02-09T17:58:53.006 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:53.006 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.007 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.007 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:58:53.007 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.655699+0000", 2024-02-09T17:58:53.007 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.283e-06 2024-02-09T17:58:53.007 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.007 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.007 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:58:53.007 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.655705+0000", 2024-02-09T17:58:53.007 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2789999999999998e-06 2024-02-09T17:58:53.007 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.007 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.007 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:58:53.007 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.655716+0000", 2024-02-09T17:58:53.007 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1877e-05 2024-02-09T17:58:53.007 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.007 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.008 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:58:53.008 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.655725+0000", 2024-02-09T17:58:53.008 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4419999999999998e-06 2024-02-09T17:58:53.008 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.008 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.008 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:58:53.008 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.655768+0000", 2024-02-09T17:58:53.008 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3458000000000003e-05 2024-02-09T17:58:53.008 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.008 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.008 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:58:53.008 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.655849+0000", 2024-02-09T17:58:53.008 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0966999999999999e-05 2024-02-09T17:58:53.008 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.009 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.009 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-02-09T17:58:53.009 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.655949+0000", 2024-02-09T17:58:53.009 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9517000000000003e-05 2024-02-09T17:58:53.009 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.009 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.009 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:58:53.009 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.659984+0000", 2024-02-09T17:58:53.009 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0040348299999999997 2024-02-09T17:58:53.009 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.009 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.009 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:53.009 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.668934+0000", 2024-02-09T17:58:53.009 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0089507319999999994 2024-02-09T17:58:53.010 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.010 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.010 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:53.010 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.821610+0000", 2024-02-09T17:58:53.010 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.15267540900000001 2024-02-09T17:58:53.010 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.010 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.010 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:58:53.010 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.821658+0000", 2024-02-09T17:58:53.010 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8593e-05 2024-02-09T17:58:53.010 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.010 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.010 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:58:53.010 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.821691+0000", 2024-02-09T17:58:53.011 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2286000000000002e-05 2024-02-09T17:58:53.011 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:53.011 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:58:53.011 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:53.011 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.011 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.011 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25837.0:254364 6.5 6:a9cce28d:::benchmark_data_smithi179_126206_object123983:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:58:53.011 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:49:52.661673+0000", 2024-02-09T17:58:53.011 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 540.29082571799995, 2024-02-09T17:58:53.011 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17981395999999999, 2024-02-09T17:58:53.011 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:58:53.011 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:58:53.011 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:58:53.011 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25837", 2024-02-09T17:58:53.011 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:58:53.012 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 254364 2024-02-09T17:58:53.012 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.012 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:58:53.012 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.012 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:58:53.012 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.661673+0000", 2024-02-09T17:58:53.012 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:53.012 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.012 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.012 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:58:53.012 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.661673+0000", 2024-02-09T17:58:53.012 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:53.012 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.012 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.013 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:58:53.013 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.661674+0000", 2024-02-09T17:58:53.013 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.063e-06 2024-02-09T17:58:53.013 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.013 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.013 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:58:53.013 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.661679+0000", 2024-02-09T17:58:53.013 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1800000000000004e-06 2024-02-09T17:58:53.013 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.013 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.013 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:58:53.013 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.661690+0000", 2024-02-09T17:58:53.013 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1096e-05 2024-02-09T17:58:53.013 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.013 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.014 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:58:53.014 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.661697+0000", 2024-02-09T17:58:53.014 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.596e-06 2024-02-09T17:58:53.014 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.014 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.014 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:58:53.014 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.661740+0000", 2024-02-09T17:58:53.014 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3582999999999999e-05 2024-02-09T17:58:53.014 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.014 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.014 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:58:53.014 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.661835+0000", 2024-02-09T17:58:53.014 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4518999999999994e-05 2024-02-09T17:58:53.014 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.014 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.015 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-02-09T17:58:53.015 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.661916+0000", 2024-02-09T17:58:53.015 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1212999999999996e-05 2024-02-09T17:58:53.015 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.015 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.015 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:58:53.015 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.666085+0000", 2024-02-09T17:58:53.015 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041686609999999997 2024-02-09T17:58:53.015 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.015 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.015 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:53.015 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.669054+0000", 2024-02-09T17:58:53.015 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0029690010000000002 2024-02-09T17:58:53.015 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.015 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.016 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:53.016 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.841347+0000", 2024-02-09T17:58:53.016 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17229327999999999 2024-02-09T17:58:53.016 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.016 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.016 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:58:53.016 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.841404+0000", 2024-02-09T17:58:53.016 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6719999999999999e-05 2024-02-09T17:58:53.016 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.016 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.016 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:58:53.016 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.841487+0000", 2024-02-09T17:58:53.016 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3047999999999995e-05 2024-02-09T17:58:53.016 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:53.016 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:58:53.017 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:53.017 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.017 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.017 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25837.0:254374 6.3 6:c071e061:::benchmark_data_smithi179_126206_object123993:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:58:53.017 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:49:52.675875+0000", 2024-02-09T17:58:53.017 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 540.27662361, 2024-02-09T17:58:53.017 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.151919742, 2024-02-09T17:58:53.017 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:58:53.017 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:58:53.017 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:58:53.017 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25837", 2024-02-09T17:58:53.017 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:58:53.017 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 254374 2024-02-09T17:58:53.017 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.018 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:58:53.018 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.018 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:58:53.018 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.675875+0000", 2024-02-09T17:58:53.018 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:53.018 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.018 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.018 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:58:53.018 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.675875+0000", 2024-02-09T17:58:53.018 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:53.018 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.018 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.018 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:58:53.018 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.675877+0000", 2024-02-09T17:58:53.018 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8780000000000001e-06 2024-02-09T17:58:53.019 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.019 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.019 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:58:53.019 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.675883+0000", 2024-02-09T17:58:53.019 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0560000000000001e-06 2024-02-09T17:58:53.019 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.019 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.019 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:58:53.019 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.675898+0000", 2024-02-09T17:58:53.019 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4976e-05 2024-02-09T17:58:53.019 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.019 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.019 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:58:53.019 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.675906+0000", 2024-02-09T17:58:53.019 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1210000000000007e-06 2024-02-09T17:58:53.020 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.020 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.020 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:58:53.020 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.675972+0000", 2024-02-09T17:58:53.020 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5710999999999995e-05 2024-02-09T17:58:53.020 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.020 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.020 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:58:53.020 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.676127+0000", 2024-02-09T17:58:53.020 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015538300000000001 2024-02-09T17:58:53.020 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.020 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.020 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-02-09T17:58:53.020 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.676274+0000", 2024-02-09T17:58:53.021 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000147078 2024-02-09T17:58:53.021 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.021 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.021 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:58:53.021 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.680551+0000", 2024-02-09T17:58:53.021 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042773500000000001 2024-02-09T17:58:53.021 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.021 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.021 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:53.021 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.680811+0000", 2024-02-09T17:58:53.021 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025923599999999998 2024-02-09T17:58:53.021 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.021 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.021 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:53.022 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.827702+0000", 2024-02-09T17:58:53.022 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.14689158099999999 2024-02-09T17:58:53.022 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.022 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.022 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:58:53.022 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.827755+0000", 2024-02-09T17:58:53.022 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2315000000000003e-05 2024-02-09T17:58:53.022 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.022 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.022 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:58:53.022 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.827795+0000", 2024-02-09T17:58:53.022 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0057e-05 2024-02-09T17:58:53.022 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:53.022 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:58:53.022 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:53.023 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.023 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.023 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25837.0:254406 6.3 6:c56758bf:::benchmark_data_smithi179_126206_object124025:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:58:53.023 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:49:52.728445+0000", 2024-02-09T17:58:53.023 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 540.22405386000003, 2024-02-09T17:58:53.023 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.10753573299999999, 2024-02-09T17:58:53.023 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:58:53.023 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:58:53.023 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:58:53.023 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25837", 2024-02-09T17:58:53.023 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:58:53.023 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 254406 2024-02-09T17:58:53.023 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.024 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:58:53.024 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.024 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:58:53.024 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.728445+0000", 2024-02-09T17:58:53.024 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:53.024 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.024 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.024 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:58:53.024 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.728445+0000", 2024-02-09T17:58:53.024 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:53.024 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.024 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.024 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:58:53.024 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.728445+0000", 2024-02-09T17:58:53.024 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6899999999999998e-07 2024-02-09T17:58:53.024 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.025 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.025 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:58:53.025 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.728447+0000", 2024-02-09T17:58:53.025 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9549999999999999e-06 2024-02-09T17:58:53.025 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.025 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.025 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:58:53.025 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.728452+0000", 2024-02-09T17:58:53.025 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.412e-06 2024-02-09T17:58:53.025 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.025 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.025 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:58:53.025 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.728456+0000", 2024-02-09T17:58:53.025 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8689999999999997e-06 2024-02-09T17:58:53.025 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.026 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.026 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:58:53.026 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.732702+0000", 2024-02-09T17:58:53.026 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042453489999999998 2024-02-09T17:58:53.026 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.026 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.026 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:58:53.026 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.732837+0000", 2024-02-09T17:58:53.026 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013582899999999999 2024-02-09T17:58:53.026 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.026 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.026 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-02-09T17:58:53.026 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.732964+0000", 2024-02-09T17:58:53.026 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012613300000000001 2024-02-09T17:58:53.026 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.026 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.026 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:53.027 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.738024+0000", 2024-02-09T17:58:53.027 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0050601520000000001 2024-02-09T17:58:53.027 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.027 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.027 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:58:53.027 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.738069+0000", 2024-02-09T17:58:53.027 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5358000000000002e-05 2024-02-09T17:58:53.027 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.027 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.027 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:53.027 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.835884+0000", 2024-02-09T17:58:53.027 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.097814605999999998 2024-02-09T17:58:53.027 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.027 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.027 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:58:53.027 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.835939+0000", 2024-02-09T17:58:53.028 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5676e-05 2024-02-09T17:58:53.028 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.028 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.028 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:58:53.028 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.835980+0000", 2024-02-09T17:58:53.028 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1025000000000001e-05 2024-02-09T17:58:53.028 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:53.028 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:58:53.028 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:53.028 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.028 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.028 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25837.0:254410 6.5 6:a137bf3f:::benchmark_data_smithi179_126206_object124029:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-02-09T17:58:53.028 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:49:52.733427+0000", 2024-02-09T17:58:53.028 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 540.21907127300005, 2024-02-09T17:58:53.028 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.108200144, 2024-02-09T17:58:53.029 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:58:53.029 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:58:53.029 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:58:53.029 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25837", 2024-02-09T17:58:53.029 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/929031851", 2024-02-09T17:58:53.029 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 254410 2024-02-09T17:58:53.029 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.029 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:58:53.029 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.029 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:58:53.029 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.733427+0000", 2024-02-09T17:58:53.029 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:53.029 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.029 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.029 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:58:53.029 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.733427+0000", 2024-02-09T17:58:53.029 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:53.030 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.030 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.030 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:58:53.030 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.733428+0000", 2024-02-09T17:58:53.030 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.22e-06 2024-02-09T17:58:53.030 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.030 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.030 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:58:53.030 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.733432+0000", 2024-02-09T17:58:53.030 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3459999999999998e-06 2024-02-09T17:58:53.030 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.030 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.030 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:58:53.030 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.733442+0000", 2024-02-09T17:58:53.030 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0719e-05 2024-02-09T17:58:53.030 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.031 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.031 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:58:53.031 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.733451+0000", 2024-02-09T17:58:53.031 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2609999999999993e-06 2024-02-09T17:58:53.031 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.031 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.031 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:58:53.031 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.733496+0000", 2024-02-09T17:58:53.031 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5549e-05 2024-02-09T17:58:53.031 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.031 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.031 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:58:53.031 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.733579+0000", 2024-02-09T17:58:53.031 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3036999999999998e-05 2024-02-09T17:58:53.031 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.032 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.032 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-02-09T17:58:53.032 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.733651+0000", 2024-02-09T17:58:53.032 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1594999999999995e-05 2024-02-09T17:58:53.032 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.032 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.032 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:58:53.032 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.737521+0000", 2024-02-09T17:58:53.032 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0038698840000000001 2024-02-09T17:58:53.032 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.032 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.032 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:53.032 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.737808+0000", 2024-02-09T17:58:53.032 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00028684499999999998 2024-02-09T17:58:53.032 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.033 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.033 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:53.033 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.841538+0000", 2024-02-09T17:58:53.033 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.103730278 2024-02-09T17:58:53.033 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.033 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.033 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:58:53.033 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.841591+0000", 2024-02-09T17:58:53.033 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2840999999999998e-05 2024-02-09T17:58:53.033 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.033 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.033 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:58:53.033 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:49:52.841627+0000", 2024-02-09T17:58:53.033 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6569000000000002e-05 2024-02-09T17:58:53.033 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:53.034 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:58:53.034 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:53.034 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.034 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.034 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.16440.0:83376 7.8 7:14268f94:::benchmark_data_smithi179_135804_object83375:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e686)", 2024-02-09T17:58:53.034 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:51:18.486683+0000", 2024-02-09T17:58:53.034 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 454.46581496499999, 2024-02-09T17:58:53.034 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.113171218, 2024-02-09T17:58:53.034 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:58:53.034 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:58:53.034 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:58:53.034 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.16440", 2024-02-09T17:58:53.034 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:58:53.034 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 83376 2024-02-09T17:58:53.034 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.034 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:58:53.035 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.035 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:58:53.035 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.486683+0000", 2024-02-09T17:58:53.035 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:53.035 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.035 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.035 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:58:53.035 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.486683+0000", 2024-02-09T17:58:53.035 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:53.035 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.035 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.035 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:58:53.035 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.486685+0000", 2024-02-09T17:58:53.036 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.271e-06 2024-02-09T17:58:53.036 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.036 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.036 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:58:53.036 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.486746+0000", 2024-02-09T17:58:53.036 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1264000000000002e-05 2024-02-09T17:58:53.036 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.036 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.036 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:58:53.036 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.486760+0000", 2024-02-09T17:58:53.036 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.362e-05 2024-02-09T17:58:53.036 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.036 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.036 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:58:53.036 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.486767+0000", 2024-02-09T17:58:53.037 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6159999999999996e-06 2024-02-09T17:58:53.037 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.037 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.037 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:58:53.037 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.486821+0000", 2024-02-09T17:58:53.037 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3344999999999998e-05 2024-02-09T17:58:53.037 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.037 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.037 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:58:53.037 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.486956+0000", 2024-02-09T17:58:53.037 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013512800000000001 2024-02-09T17:58:53.037 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.037 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.037 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,4", 2024-02-09T17:58:53.038 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.487087+0000", 2024-02-09T17:58:53.038 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000131371 2024-02-09T17:58:53.038 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.038 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.038 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:58:53.038 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.490391+0000", 2024-02-09T17:58:53.038 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003304319 2024-02-09T17:58:53.038 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.038 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.038 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:53.038 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.494399+0000", 2024-02-09T17:58:53.038 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0040079709999999999 2024-02-09T17:58:53.038 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.038 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.039 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:53.039 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.599744+0000", 2024-02-09T17:58:53.039 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.10534436799999999 2024-02-09T17:58:53.039 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.039 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.039 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:58:53.039 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.599810+0000", 2024-02-09T17:58:53.039 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6365999999999994e-05 2024-02-09T17:58:53.039 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.039 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.039 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:58:53.039 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.599855+0000", 2024-02-09T17:58:53.039 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4579000000000001e-05 2024-02-09T17:58:53.040 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:53.040 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:58:53.040 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:53.040 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.040 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.040 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.16440.0:83381 7.8 7:1d33db6a:::benchmark_data_smithi179_135804_object83380:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e686)", 2024-02-09T17:58:53.040 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:51:18.495214+0000", 2024-02-09T17:58:53.040 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 454.45728396200002, 2024-02-09T17:58:53.040 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.10478098600000001, 2024-02-09T17:58:53.040 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:58:53.040 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:58:53.040 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:58:53.040 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.16440", 2024-02-09T17:58:53.040 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:58:53.041 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 83381 2024-02-09T17:58:53.041 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.041 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:58:53.041 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.041 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:58:53.041 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.495214+0000", 2024-02-09T17:58:53.041 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:53.041 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.041 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.041 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:58:53.041 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.495214+0000", 2024-02-09T17:58:53.041 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:53.041 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.041 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.041 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:58:53.042 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.495216+0000", 2024-02-09T17:58:53.042 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.418e-06 2024-02-09T17:58:53.042 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.042 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.042 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:58:53.042 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.495267+0000", 2024-02-09T17:58:53.042 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1190999999999998e-05 2024-02-09T17:58:53.042 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.042 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.042 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:58:53.042 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.495282+0000", 2024-02-09T17:58:53.042 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5197999999999999e-05 2024-02-09T17:58:53.042 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.042 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.043 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:58:53.043 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.495290+0000", 2024-02-09T17:58:53.043 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9030000000000004e-06 2024-02-09T17:58:53.043 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.043 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.043 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:58:53.043 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.495343+0000", 2024-02-09T17:58:53.043 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2788999999999999e-05 2024-02-09T17:58:53.043 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.043 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.043 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:58:53.043 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.495486+0000", 2024-02-09T17:58:53.043 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014301599999999999 2024-02-09T17:58:53.043 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.043 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.044 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,4", 2024-02-09T17:58:53.044 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.495624+0000", 2024-02-09T17:58:53.044 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013796500000000001 2024-02-09T17:58:53.044 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.044 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.044 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:53.044 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.501850+0000", 2024-02-09T17:58:53.044 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.006226077 2024-02-09T17:58:53.044 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.044 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.044 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:58:53.044 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.501876+0000", 2024-02-09T17:58:53.044 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5675000000000001e-05 2024-02-09T17:58:53.044 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.045 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.045 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:53.045 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.599910+0000", 2024-02-09T17:58:53.045 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.098033911000000001 2024-02-09T17:58:53.045 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.045 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.045 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:58:53.045 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.599961+0000", 2024-02-09T17:58:53.045 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1347999999999997e-05 2024-02-09T17:58:53.045 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.045 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.045 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:58:53.045 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:51:18.599995+0000", 2024-02-09T17:58:53.045 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4495000000000001e-05 2024-02-09T17:58:53.045 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:53.045 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:58:53.046 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:53.046 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.046 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.046 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.16440.0:116805 7.8 7:1a85ab55:::benchmark_data_smithi179_135804_object9014:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e706)", 2024-02-09T17:58:53.046 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:52:10.358547+0000", 2024-02-09T17:58:53.046 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 402.59395125899999, 2024-02-09T17:58:53.046 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.12194231799999999, 2024-02-09T17:58:53.046 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:58:53.046 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:58:53.046 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T17:58:53.046 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.16440", 2024-02-09T17:58:53.046 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:58:53.046 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 116805 2024-02-09T17:58:53.046 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.046 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:58:53.047 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.047 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:58:53.047 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:10.358547+0000", 2024-02-09T17:58:53.047 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:53.047 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.047 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.047 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:58:53.047 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:10.358547+0000", 2024-02-09T17:58:53.047 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:53.047 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.047 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.047 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:58:53.047 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:10.358549+0000", 2024-02-09T17:58:53.047 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6199999999999999e-06 2024-02-09T17:58:53.048 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.048 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.048 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:58:53.048 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:10.358556+0000", 2024-02-09T17:58:53.048 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3490000000000003e-06 2024-02-09T17:58:53.048 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.048 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.048 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:58:53.048 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:10.358574+0000", 2024-02-09T17:58:53.048 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7382000000000001e-05 2024-02-09T17:58:53.048 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.048 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.048 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:58:53.048 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:10.358582+0000", 2024-02-09T17:58:53.049 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4200000000000007e-06 2024-02-09T17:58:53.049 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.049 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.049 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:58:53.049 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:10.358634+0000", 2024-02-09T17:58:53.049 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.164e-05 2024-02-09T17:58:53.049 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.049 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.049 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:58:53.049 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:10.358730+0000", 2024-02-09T17:58:53.049 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6274999999999996e-05 2024-02-09T17:58:53.049 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.049 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.049 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,4", 2024-02-09T17:58:53.050 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:10.358804+0000", 2024-02-09T17:58:53.050 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4564e-05 2024-02-09T17:58:53.050 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.050 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.050 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T17:58:53.050 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:10.362801+0000", 2024-02-09T17:58:53.050 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003996131 2024-02-09T17:58:53.050 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.050 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.050 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:53.050 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:10.362991+0000", 2024-02-09T17:58:53.050 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019082999999999999 2024-02-09T17:58:53.050 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.050 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.050 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:58:53.050 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:10.480375+0000", 2024-02-09T17:58:53.051 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.117383336 2024-02-09T17:58:53.051 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.051 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.051 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:58:53.051 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:10.480438+0000", 2024-02-09T17:58:53.051 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3260000000000001e-05 2024-02-09T17:58:53.051 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.051 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.051 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:58:53.051 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:10.480490+0000", 2024-02-09T17:58:53.051 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1511000000000002e-05 2024-02-09T17:58:53.051 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:53.051 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:58:53.051 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:53.052 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.052 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.052 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26305.0:53557 8.d e803/785 8:b0bf2330:::benchmark_data_smithi179_148205_object53556:head v 803'3287, mlcod=803'3285)", 2024-02-09T17:58:53.052 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:55:51.566769+0000", 2024-02-09T17:58:53.052 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 181.38572912800001, 2024-02-09T17:58:53.052 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.083624594999999996, 2024-02-09T17:58:53.052 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:58:53.052 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:58:53.052 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:58:53.052 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.052 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:58:53.052 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:55:51.566769+0000", 2024-02-09T17:58:53.052 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:53.053 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.053 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.053 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:58:53.053 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:55:51.566769+0000", 2024-02-09T17:58:53.053 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:53.053 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.053 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.053 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:58:53.053 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:55:51.566770+0000", 2024-02-09T17:58:53.053 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8999999999999997e-07 2024-02-09T17:58:53.053 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.053 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.053 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:58:53.053 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:55:51.566824+0000", 2024-02-09T17:58:53.053 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4432000000000003e-05 2024-02-09T17:58:53.054 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.054 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.054 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:58:53.054 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:55:51.566835+0000", 2024-02-09T17:58:53.054 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0305999999999999e-05 2024-02-09T17:58:53.054 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.054 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.054 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:58:53.054 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:55:51.566840+0000", 2024-02-09T17:58:53.054 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.434e-06 2024-02-09T17:58:53.054 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.054 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.054 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:58:53.054 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:55:51.566880+0000", 2024-02-09T17:58:53.055 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0349999999999998e-05 2024-02-09T17:58:53.055 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.055 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.055 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:58:53.055 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:55:51.566895+0000", 2024-02-09T17:58:53.055 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4572999999999999e-05 2024-02-09T17:58:53.055 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.055 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.055 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:58:53.055 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:55:51.650363+0000", 2024-02-09T17:58:53.055 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.083468202000000005 2024-02-09T17:58:53.055 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.055 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.055 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:58:53.055 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:55:51.650394+0000", 2024-02-09T17:58:53.056 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0808e-05 2024-02-09T17:58:53.056 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:53.056 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:58:53.056 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:53.056 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.056 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.056 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26305.0:53564 8.d e803/785 8:bfafec64:::benchmark_data_smithi179_148205_object53563:head v 803'3288, mlcod=803'3285)", 2024-02-09T17:58:53.056 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:55:51.570482+0000", 2024-02-09T17:58:53.056 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 181.38201611, 2024-02-09T17:58:53.056 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.079930868000000002, 2024-02-09T17:58:53.056 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:58:53.056 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:58:53.056 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:58:53.057 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.057 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:58:53.057 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:55:51.570482+0000", 2024-02-09T17:58:53.057 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:53.057 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.057 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.057 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:58:53.057 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:55:51.570482+0000", 2024-02-09T17:58:53.057 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:53.057 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.057 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.057 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:58:53.057 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:55:51.570483+0000", 2024-02-09T17:58:53.057 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0100000000000005e-07 2024-02-09T17:58:53.057 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.058 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.058 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:58:53.058 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:55:51.570545+0000", 2024-02-09T17:58:53.058 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1591e-05 2024-02-09T17:58:53.058 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.058 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.058 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:58:53.058 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:55:51.570554+0000", 2024-02-09T17:58:53.058 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2820000000000002e-06 2024-02-09T17:58:53.058 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.058 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.058 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:58:53.058 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:55:51.570558+0000", 2024-02-09T17:58:53.058 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6280000000000004e-06 2024-02-09T17:58:53.059 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.059 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.059 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:58:53.059 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:55:51.570596+0000", 2024-02-09T17:58:53.059 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7969000000000002e-05 2024-02-09T17:58:53.059 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.059 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.059 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:58:53.059 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:55:51.570609+0000", 2024-02-09T17:58:53.059 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2951e-05 2024-02-09T17:58:53.059 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.059 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.059 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:58:53.059 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:55:51.650397+0000", 2024-02-09T17:58:53.059 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.079787963000000003 2024-02-09T17:58:53.060 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.060 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.060 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:58:53.060 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:55:51.650413+0000", 2024-02-09T17:58:53.060 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5883e-05 2024-02-09T17:58:53.060 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:53.060 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:58:53.060 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:53.060 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.060 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.060 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26305.0:146716 8.d e826/785 8:b2723d4d:::benchmark_data_smithi179_148205_object23644:head v 826'9127, mlcod=826'9125)", 2024-02-09T17:58:53.060 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:57:22.465129+0000", 2024-02-09T17:58:53.060 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 90.487369620999999, 2024-02-09T17:58:53.061 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.066489187000000005, 2024-02-09T17:58:53.061 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:58:53.061 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:58:53.061 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:58:53.061 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.061 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:58:53.061 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:57:22.465129+0000", 2024-02-09T17:58:53.061 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:53.061 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.061 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.061 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:58:53.061 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:57:22.465129+0000", 2024-02-09T17:58:53.061 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:53.061 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.061 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.062 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:58:53.062 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:57:22.465130+0000", 2024-02-09T17:58:53.062 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7700000000000004e-07 2024-02-09T17:58:53.062 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.062 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.062 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:58:53.062 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:57:22.465135+0000", 2024-02-09T17:58:53.062 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9229999999999999e-06 2024-02-09T17:58:53.062 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.062 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.062 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:58:53.062 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:57:22.465146+0000", 2024-02-09T17:58:53.062 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0478e-05 2024-02-09T17:58:53.062 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.063 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.063 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:58:53.063 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:57:22.465151+0000", 2024-02-09T17:58:53.063 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1259999999999997e-06 2024-02-09T17:58:53.063 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.063 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.063 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:58:53.063 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:57:22.527821+0000", 2024-02-09T17:58:53.063 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.062670315000000004 2024-02-09T17:58:53.063 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.063 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.063 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:58:53.063 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:57:22.527841+0000", 2024-02-09T17:58:53.063 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9879999999999999e-05 2024-02-09T17:58:53.063 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.063 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.064 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:58:53.064 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:57:22.531588+0000", 2024-02-09T17:58:53.064 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003746334 2024-02-09T17:58:53.064 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.064 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.064 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:58:53.064 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:57:22.531618+0000", 2024-02-09T17:58:53.064 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0454e-05 2024-02-09T17:58:53.064 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:53.064 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:58:53.064 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:53.064 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.064 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.064 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26305.0:213142 8.d e848/785 8:be9efaa6:::benchmark_data_smithi179_148205_object90070:head v 848'13249, mlcod=848'13247)", 2024-02-09T17:58:53.065 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:58:51.962143+0000", 2024-02-09T17:58:53.065 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.990355184, 2024-02-09T17:58:53.065 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.065511949999999999, 2024-02-09T17:58:53.065 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T17:58:53.065 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:58:53.065 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T17:58:53.065 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.065 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T17:58:53.065 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:58:51.962143+0000", 2024-02-09T17:58:53.065 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:53.065 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.065 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.065 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T17:58:53.065 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:58:51.962143+0000", 2024-02-09T17:58:53.066 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T17:58:53.066 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.066 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.066 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T17:58:53.066 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:58:51.962144+0000", 2024-02-09T17:58:53.066 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.192e-06 2024-02-09T17:58:53.066 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.066 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.066 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T17:58:53.066 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:58:51.962156+0000", 2024-02-09T17:58:53.066 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1189000000000001e-05 2024-02-09T17:58:53.066 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.066 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.066 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T17:58:53.066 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:58:51.962174+0000", 2024-02-09T17:58:53.067 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8681999999999998e-05 2024-02-09T17:58:53.067 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.067 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.067 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T17:58:53.067 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:58:51.962184+0000", 2024-02-09T17:58:53.067 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0071000000000001e-05 2024-02-09T17:58:53.067 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.067 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.067 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T17:58:53.067 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:58:51.962266+0000", 2024-02-09T17:58:53.067 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1688999999999997e-05 2024-02-09T17:58:53.067 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.067 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.067 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T17:58:53.067 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:58:51.962292+0000", 2024-02-09T17:58:53.068 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6007e-05 2024-02-09T17:58:53.068 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.068 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.068 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T17:58:53.068 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:58:52.027628+0000", 2024-02-09T17:58:53.068 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.065335993999999994 2024-02-09T17:58:53.068 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T17:58:53.068 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T17:58:53.068 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T17:58:53.068 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:58:52.027655+0000", 2024-02-09T17:58:53.068 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7126000000000001e-05 2024-02-09T17:58:53.068 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:53.068 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:58:53.068 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:53.068 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T17:58:53.069 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T17:58:53.069 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T17:58:53.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:53.070 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:58:53.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:58:53.171 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:58:53.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:52 smithi179 ceph-mon[43975]: pgmap v2111: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 758 op/s; 7001/108615 objects degraded (6.446%) 2024-02-09T17:58:53.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:52 smithi179 ceph-mon[43975]: 8.b scrub starts 2024-02-09T17:58:53.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:52 smithi179 ceph-mon[43975]: 1.15 scrub starts 2024-02-09T17:58:53.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:52 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 7001/108615 objects degraded (6.446%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:58:53.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:52 smithi179 ceph-mon[43975]: 1.15 scrub ok 2024-02-09T17:58:53.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:52 smithi179 ceph-mon[43975]: 8.b scrub ok 2024-02-09T17:58:53.230 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:53.264 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:58:53.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:53.268 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:58:53.369 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:58:53.403 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:53.970 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:53 smithi099 ceph-mon[39420]: 1.1c scrub starts 2024-02-09T17:58:53.970 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:53 smithi099 ceph-mon[39420]: 1.1c scrub ok 2024-02-09T17:58:53.970 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:53 smithi099 ceph-mon[39420]: 8.1 scrub starts 2024-02-09T17:58:54.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:53 smithi162 ceph-mon[44304]: 1.1c scrub starts 2024-02-09T17:58:54.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:53 smithi162 ceph-mon[44304]: 1.1c scrub ok 2024-02-09T17:58:54.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:53 smithi162 ceph-mon[44304]: 8.1 scrub starts 2024-02-09T17:58:54.177 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:53 smithi179 ceph-mon[43975]: 1.1c scrub starts 2024-02-09T17:58:54.178 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:53 smithi179 ceph-mon[43975]: 1.1c scrub ok 2024-02-09T17:58:54.178 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:53 smithi179 ceph-mon[43975]: 8.1 scrub starts 2024-02-09T17:58:54.665 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:58:54.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:54.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:58:54.687 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.9 dump_ops_in_flight 2024-02-09T17:58:54.772 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:58:54.825 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:54.868 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:58:54.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:54.874 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:58:54.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:54 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:58:54.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:54 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:58:54.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:54 smithi162 ceph-mon[44304]: 8.1 scrub ok 2024-02-09T17:58:54.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:54 smithi162 ceph-mon[44304]: pgmap v2112: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 785 op/s; 6244/101202 objects degraded (6.170%) 2024-02-09T17:58:54.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:54 smithi162 ceph-mon[44304]: osdmap e849: 12 total, 11 up, 12 in 2024-02-09T17:58:54.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:54 smithi162 ceph-mon[44304]: 1.12 scrub starts 2024-02-09T17:58:54.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:54 smithi162 ceph-mon[44304]: 1.12 scrub ok 2024-02-09T17:58:54.970 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:58:54.970 INFO:teuthology.orchestra.run.smithi099.stdout: "success": "" 2024-02-09T17:58:54.970 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:58:54.976 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:58:54.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:54 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:58:54.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:54 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:58:54.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:54 smithi099 ceph-mon[39420]: 8.1 scrub ok 2024-02-09T17:58:54.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:54 smithi099 ceph-mon[39420]: pgmap v2112: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 785 op/s; 6244/101202 objects degraded (6.170%) 2024-02-09T17:58:54.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:54 smithi099 ceph-mon[39420]: osdmap e849: 12 total, 11 up, 12 in 2024-02-09T17:58:54.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:54 smithi099 ceph-mon[39420]: 1.12 scrub starts 2024-02-09T17:58:54.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:54 smithi099 ceph-mon[39420]: 1.12 scrub ok 2024-02-09T17:58:55.013 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:55.072 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-09T17:58:55.145 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:58:55.176 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:54 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:58:55.176 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:54 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:58:55.176 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:54 smithi179 ceph-mon[43975]: 8.1 scrub ok 2024-02-09T17:58:55.176 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:54 smithi179 ceph-mon[43975]: pgmap v2112: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 785 op/s; 6244/101202 objects degraded (6.170%) 2024-02-09T17:58:55.176 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:54 smithi179 ceph-mon[43975]: osdmap e849: 12 total, 11 up, 12 in 2024-02-09T17:58:55.176 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:54 smithi179 ceph-mon[43975]: 1.12 scrub starts 2024-02-09T17:58:55.177 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:54 smithi179 ceph-mon[43975]: 1.12 scrub ok 2024-02-09T17:58:55.670 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:58:55.698 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:58:55.721 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:58:55.765 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:58:55.803 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:58:55.822 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:58:55.847 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:58:55.887 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:58:55.915 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:58:55.944 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:58:55.976 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:58:55.976 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:58:55.977 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:58:55.977 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:58:55.977 INFO:teuthology.orchestra.run.smithi099.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-09T17:58:55.977 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:58:55.977 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:58:55.977 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:58:55.977 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:58:55.977 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:58:55.977 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:58:55.977 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:58:55.977 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:58:55.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:55 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1625647371' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:58:56.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:55 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1625647371' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:58:56.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:55 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1625647371' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:58:56.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:56 smithi099 ceph-mon[39420]: pgmap v2114: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 2.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 861 op/s; 6244/96762 objects degraded (6.453%) 2024-02-09T17:58:57.016 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:58:57.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:57.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:58:57.123 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:58:57.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:56 smithi179 ceph-mon[43975]: pgmap v2114: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 2.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 861 op/s; 6244/96762 objects degraded (6.453%) 2024-02-09T17:58:57.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:56 smithi162 ceph-mon[44304]: pgmap v2114: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 2.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 861 op/s; 6244/96762 objects degraded (6.453%) 2024-02-09T17:58:57.187 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:57.737 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:57 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 6244/96762 objects degraded (6.453%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:58:57.737 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:57 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:58:57.737 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:57 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:58:57.794 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:58:57.837 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-09T17:58:57.837 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-09T17:58:57.952 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:58:57.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:57.960 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:58:58.056 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:57 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 6244/96762 objects degraded (6.453%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:58:58.056 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:57 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:58:58.056 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:57 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:58:58.061 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:58:58.100 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:58.123 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:58:58.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:58.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:58:58.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:57 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 6244/96762 objects degraded (6.453%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:58:58.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:57 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:58:58.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:57 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:58:58.232 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:58:58.261 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:58.284 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:58:58.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:58.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:58:58.389 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:58:58.427 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:58.740 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:58 smithi099 ceph-mon[39420]: pgmap v2115: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 1.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 685 op/s; 6244/93720 objects degraded (6.662%) 2024-02-09T17:58:58.782 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:58:58.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:58.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:58:58.887 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:58:58.915 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:58.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:58 smithi179 ceph-mon[43975]: pgmap v2115: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 1.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 685 op/s; 6244/93720 objects degraded (6.662%) 2024-02-09T17:58:58.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:58 smithi179 ceph-mon[43975]: 1.13 scrub starts 2024-02-09T17:58:58.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:58 smithi179 ceph-mon[43975]: osdmap e850: 12 total, 11 up, 12 in 2024-02-09T17:58:58.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:58 smithi179 ceph-mon[43975]: 1.2 scrub starts 2024-02-09T17:58:58.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:58 smithi179 ceph-mon[43975]: 1.2 scrub ok 2024-02-09T17:58:58.940 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:58:58.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:58.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:58:58.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:58 smithi099 ceph-mon[39420]: 1.13 scrub starts 2024-02-09T17:58:58.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:58 smithi099 ceph-mon[39420]: osdmap e850: 12 total, 11 up, 12 in 2024-02-09T17:58:58.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:58 smithi099 ceph-mon[39420]: 1.2 scrub starts 2024-02-09T17:58:58.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:58 smithi099 ceph-mon[39420]: 1.2 scrub ok 2024-02-09T17:58:59.044 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:58:59.074 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:59.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:58 smithi162 ceph-mon[44304]: pgmap v2115: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 1.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 685 op/s; 6244/93720 objects degraded (6.662%) 2024-02-09T17:58:59.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:58 smithi162 ceph-mon[44304]: 1.13 scrub starts 2024-02-09T17:58:59.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:58 smithi162 ceph-mon[44304]: osdmap e850: 12 total, 11 up, 12 in 2024-02-09T17:58:59.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:58 smithi162 ceph-mon[44304]: 1.2 scrub starts 2024-02-09T17:58:59.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:58 smithi162 ceph-mon[44304]: 1.2 scrub ok 2024-02-09T17:58:59.228 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:58:59.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:59.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:58:59.335 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:58:59.372 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:59.627 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:58:59.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:59.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:58:59.729 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:58:59.763 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:58:59.839 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:58:59.882 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:58:59 smithi099 ceph-mon[39420]: 1.13 scrub ok 2024-02-09T17:58:59.892 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:58:59.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:58:59.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:58:59.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:58:59 smithi179 ceph-mon[43975]: 1.13 scrub ok 2024-02-09T17:58:59.999 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:59:00.025 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:58:59 smithi162 ceph-mon[44304]: 1.13 scrub ok 2024-02-09T17:59:00.027 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:00.050 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:59:00.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:00.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:59:00.154 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:59:00.200 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:00.205 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:59:00.229 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:59:00.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:00.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:59:00.336 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:59:00.370 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:00.400 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T17:59:00.400 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:59:00.496 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:59:00.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:00.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:59:00.602 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:59:00.673 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:00.763 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is unset 2024-02-09T17:59:00.771 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:00 smithi099 ceph-mon[39420]: pgmap v2117: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 1.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 923 op/s; 5496/86439 objects degraded (6.358%) 2024-02-09T17:59:00.772 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:00 smithi099 ceph-mon[39420]: 8.6 scrub starts 2024-02-09T17:59:00.772 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:00 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4239170247' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:59:00.772 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:00 smithi099 ceph-mon[39420]: 8.6 scrub ok 2024-02-09T17:59:01.175 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:00 smithi179 ceph-mon[43975]: pgmap v2117: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 1.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 923 op/s; 5496/86439 objects degraded (6.358%) 2024-02-09T17:59:01.175 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:00 smithi179 ceph-mon[43975]: 8.6 scrub starts 2024-02-09T17:59:01.175 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:00 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4239170247' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:59:01.175 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:00 smithi179 ceph-mon[43975]: 8.6 scrub ok 2024-02-09T17:59:01.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:00 smithi162 ceph-mon[44304]: pgmap v2117: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 1.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 923 op/s; 5496/86439 objects degraded (6.358%) 2024-02-09T17:59:01.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:00 smithi162 ceph-mon[44304]: 8.6 scrub starts 2024-02-09T17:59:01.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:00 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4239170247' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:59:01.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:00 smithi162 ceph-mon[44304]: 8.6 scrub ok 2024-02-09T17:59:01.776 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:01 smithi099 ceph-mon[39420]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:59:01.776 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:01 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4239170247' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:59:01.776 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:01 smithi099 ceph-mon[39420]: osdmap e851: 12 total, 11 up, 12 in 2024-02-09T17:59:01.776 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:01 smithi099 ceph-mon[39420]: 8.3 deep-scrub starts 2024-02-09T17:59:02.068 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 2] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2, 4, 5, 6, 7, 9, 10, 11, 8] 2024-02-09T17:59:02.069 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:59:02.069 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2024-02-09T17:59:02.069 INFO:tasks.cephadm.osd.3:Restarting osd.3 (starting--it wasn't running)... 2024-02-09T17:59:02.069 DEBUG:teuthology.orchestra.run.smithi099:osd.3> sudo journalctl -f -n 0 -u ceph-a69c870a-c76e-11ee-95b7-87774f69a715@osd.3.service 2024-02-09T17:59:02.072 DEBUG:teuthology.orchestra.run.smithi099:> sudo systemctl start ceph-a69c870a-c76e-11ee-95b7-87774f69a715@osd.3 2024-02-09T17:59:02.107 INFO:journalctl@ceph.osd.3.smithi099.stdout:-- Logs begin at Fri 2024-02-09 17:04:25 UTC. -- 2024-02-09T17:59:02.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:01 smithi179 ceph-mon[43975]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:59:02.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:01 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4239170247' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:59:02.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:01 smithi179 ceph-mon[43975]: osdmap e851: 12 total, 11 up, 12 in 2024-02-09T17:59:02.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:01 smithi179 ceph-mon[43975]: 8.3 deep-scrub starts 2024-02-09T17:59:02.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:01 smithi162 ceph-mon[44304]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:59:02.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:01 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4239170247' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:59:02.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:01 smithi162 ceph-mon[44304]: osdmap e851: 12 total, 11 up, 12 in 2024-02-09T17:59:02.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:01 smithi162 ceph-mon[44304]: 8.3 deep-scrub starts 2024-02-09T17:59:02.361 INFO:journalctl@ceph.osd.3.smithi099.stdout:Feb 09 17:59:02 smithi099 systemd[1]: Starting Ceph osd.3 for a69c870a-c76e-11ee-95b7-87774f69a715... 2024-02-09T17:59:02.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:02 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:59:02.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:02 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:59:02.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:02 smithi099 ceph-mon[39420]: pgmap v2119: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 1.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 716 op/s; 5496/85017 objects degraded (6.465%) 2024-02-09T17:59:02.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:02 smithi099 ceph-mon[39420]: osdmap e852: 12 total, 11 up, 12 in 2024-02-09T17:59:02.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:02 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 5496/85017 objects degraded (6.465%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:59:03.053 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:59:03.086 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:59:03.116 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:59:03.144 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:59:03.171 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:59:03.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:02 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:59:03.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:02 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:59:03.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:02 smithi179 ceph-mon[43975]: pgmap v2119: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 1.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 716 op/s; 5496/85017 objects degraded (6.465%) 2024-02-09T17:59:03.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:02 smithi179 ceph-mon[43975]: osdmap e852: 12 total, 11 up, 12 in 2024-02-09T17:59:03.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:02 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 5496/85017 objects degraded (6.465%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:59:03.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:02 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:59:03.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:02 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:59:03.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:02 smithi162 ceph-mon[44304]: pgmap v2119: 47 pgs: 3 active+undersized+degraded, 5 active+undersized, 39 active+clean; 1.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 716 op/s; 5496/85017 objects degraded (6.465%) 2024-02-09T17:59:03.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:02 smithi162 ceph-mon[44304]: osdmap e852: 12 total, 11 up, 12 in 2024-02-09T17:59:03.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:02 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 5496/85017 objects degraded (6.465%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:59:03.201 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:59:03.231 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:59:03.278 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:59:03.309 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:59:03.343 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:59:03.376 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:59:03.376 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:59:03.376 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:59:03.377 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:59:03.377 INFO:teuthology.orchestra.run.smithi099.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-09T17:59:03.377 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:59:03.377 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:59:03.377 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:59:03.377 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:59:03.377 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:59:03.377 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:59:03.377 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:59:03.377 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:59:04.033 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.9 dump_blocked_ops 2024-02-09T17:59:04.046 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:59:04.049 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:03 smithi179 ceph-mon[43975]: 1.1a scrub starts 2024-02-09T17:59:04.049 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:03 smithi179 ceph-mon[43975]: 1.1a scrub ok 2024-02-09T17:59:04.049 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:03 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3332696326' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:59:04.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:04.050 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:59:04.060 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:03 smithi099 ceph-mon[39420]: 1.1a scrub starts 2024-02-09T17:59:04.060 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:03 smithi099 ceph-mon[39420]: 1.1a scrub ok 2024-02-09T17:59:04.060 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:03 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3332696326' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:59:04.151 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:59:04.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:03 smithi162 ceph-mon[44304]: 1.1a scrub starts 2024-02-09T17:59:04.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:03 smithi162 ceph-mon[44304]: 1.1a scrub ok 2024-02-09T17:59:04.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:03 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3332696326' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:59:04.180 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:04.203 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:59:04.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:04.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:59:04.308 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:59:04.379 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:04.399 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set noscrub 2024-02-09T17:59:04.426 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:59:04.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:04.428 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:59:04.506 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:59:04.530 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:59:04.558 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:04.579 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:59:04.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:04.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:59:04.687 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:59:04.745 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:04.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:04 smithi162 ceph-mon[44304]: pgmap v2121: 47 pgs: 1 active+clean+scrubbing+deep, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 1.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 888 op/s; 4766/77715 objects degraded (6.133%) 2024-02-09T17:59:05.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:04 smithi179 ceph-mon[43975]: pgmap v2121: 47 pgs: 1 active+clean+scrubbing+deep, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 1.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 888 op/s; 4766/77715 objects degraded (6.133%) 2024-02-09T17:59:05.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:04 smithi099 ceph-mon[39420]: pgmap v2121: 47 pgs: 1 active+clean+scrubbing+deep, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 1.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 888 op/s; 4766/77715 objects degraded (6.133%) 2024-02-09T17:59:05.707 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:59:05.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:05.712 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:59:05.743 INFO:journalctl@ceph.osd.3.smithi099.stdout:Feb 09 17:59:05 smithi099 bash[279742]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-09T17:59:05.744 INFO:journalctl@ceph.osd.3.smithi099.stdout:Feb 09 17:59:05 smithi099 bash[279742]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-02-09T17:59:05.744 INFO:journalctl@ceph.osd.3.smithi099.stdout:Feb 09 17:59:05 smithi099 bash[279742]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-02-09T17:59:05.744 INFO:journalctl@ceph.osd.3.smithi099.stdout:Feb 09 17:59:05 smithi099 bash[279742]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-02-09T17:59:05.744 INFO:journalctl@ceph.osd.3.smithi099.stdout:Feb 09 17:59:05 smithi099 bash[279742]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-02-09T17:59:05.744 INFO:journalctl@ceph.osd.3.smithi099.stdout:Feb 09 17:59:05 smithi099 bash[279742]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-09T17:59:05.744 INFO:journalctl@ceph.osd.3.smithi099.stdout:Feb 09 17:59:05 smithi099 bash[279742]: --> ceph-volume raw activate successful for osd ID: 3 2024-02-09T17:59:05.813 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:59:05.853 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:05.892 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:59:05.899 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-09T17:59:05.900 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-09T17:59:05.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:05.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:59:06.001 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:59:06.067 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:06.366 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:59:06.999 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:06 smithi099 ceph-mon[39420]: pgmap v2122: 47 pgs: 1 active+clean+scrubbing+deep, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 712 op/s; 4766/73803 objects degraded (6.458%) 2024-02-09T17:59:06.999 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:06 smithi099 ceph-mon[39420]: 8.d scrub starts 2024-02-09T17:59:06.999 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:06 smithi099 ceph-mon[39420]: 8.d scrub ok 2024-02-09T17:59:07.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:06 smithi179 ceph-mon[43975]: pgmap v2122: 47 pgs: 1 active+clean+scrubbing+deep, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 712 op/s; 4766/73803 objects degraded (6.458%) 2024-02-09T17:59:07.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:06 smithi179 ceph-mon[43975]: 8.d scrub starts 2024-02-09T17:59:07.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:06 smithi179 ceph-mon[43975]: 8.d scrub ok 2024-02-09T17:59:07.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:06 smithi162 ceph-mon[44304]: pgmap v2122: 47 pgs: 1 active+clean+scrubbing+deep, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 712 op/s; 4766/73803 objects degraded (6.458%) 2024-02-09T17:59:07.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:06 smithi162 ceph-mon[44304]: 8.d scrub starts 2024-02-09T17:59:07.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:06 smithi162 ceph-mon[44304]: 8.d scrub ok 2024-02-09T17:59:07.424 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-09T17:59:07.668 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:59:07.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:07.678 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:59:07.744 INFO:journalctl@ceph.osd.3.smithi099.stdout:Feb 09 17:59:07 smithi099 bash[280190]: b334a71d60e3abb66662dfa0b451f2a75ace492046c6682131a1785256c26b2a 2024-02-09T17:59:07.744 INFO:journalctl@ceph.osd.3.smithi099.stdout:Feb 09 17:59:07 smithi099 systemd[1]: Started Ceph osd.3 for a69c870a-c76e-11ee-95b7-87774f69a715. 2024-02-09T17:59:07.778 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:59:07.813 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:07.836 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:59:07.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:07.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:59:07.900 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:59:07.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:07 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 4766/73803 objects degraded (6.458%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:59:07.942 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:59:07.972 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:07.994 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:59:07.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:07.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:59:08.099 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:59:08.131 INFO:journalctl@ceph.osd.3.smithi099.stdout:Feb 09 17:59:07 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-3[280202]: 2024-02-09T17:59:07.852+0000 7fb903c627c0 -1 Falling back to public interface 2024-02-09T17:59:08.134 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:08.135 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:07 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 4766/73803 objects degraded (6.458%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:59:08.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:07 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 4766/73803 objects degraded (6.458%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:59:08.549 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:59:08.549 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-02-09T17:59:08.549 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-02-09T17:59:08.549 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-02-09T17:59:08.549 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:59:08.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:08 smithi179 ceph-mon[43975]: pgmap v2123: 47 pgs: 1 active+clean+scrubbing+deep, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 639 op/s; 4766/71085 objects degraded (6.705%) 2024-02-09T17:59:08.997 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:59:09.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:09.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:59:09.045 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:59:09.104 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:59:09.134 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:08 smithi162 ceph-mon[44304]: pgmap v2123: 47 pgs: 1 active+clean+scrubbing+deep, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 639 op/s; 4766/71085 objects degraded (6.705%) 2024-02-09T17:59:09.136 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:09.163 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:59:09.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:09.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:59:09.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:08 smithi099 ceph-mon[39420]: pgmap v2123: 47 pgs: 1 active+clean+scrubbing+deep, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 639 op/s; 4766/71085 objects degraded (6.705%) 2024-02-09T17:59:09.268 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:59:09.343 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:09.363 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.9 dump_historic_ops 2024-02-09T17:59:09.372 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:59:09.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:09.376 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:59:09.477 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:59:09.562 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:09.599 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:59:09.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:09.610 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:59:09.711 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:59:09.786 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:09.817 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:59:09.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:09.827 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:59:09.831 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:59:09.928 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:59:09.990 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:10.025 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:59:10.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:10.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:59:10.130 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:59:10.162 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:09 smithi099 ceph-mon[39420]: pgmap v2124: 47 pgs: 1 active+clean+scrubbing+deep, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 1.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 788 op/s; 4162/64938 objects degraded (6.409%) 2024-02-09T17:59:10.167 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:10.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:09 smithi179 ceph-mon[43975]: pgmap v2124: 47 pgs: 1 active+clean+scrubbing+deep, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 1.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 788 op/s; 4162/64938 objects degraded (6.409%) 2024-02-09T17:59:10.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:09 smithi162 ceph-mon[44304]: pgmap v2124: 47 pgs: 1 active+clean+scrubbing+deep, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 1.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 788 op/s; 4162/64938 objects degraded (6.409%) 2024-02-09T17:59:10.841 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:59:10.884 INFO:journalctl@ceph.osd.3.smithi099.stdout:Feb 09 17:59:10 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-3[280202]: 2024-02-09T17:59:10.602+0000 7fb903c627c0 -1 osd.3 814 log_to_monitors true 2024-02-09T17:59:10.904 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is set 2024-02-09T17:59:11.149 INFO:journalctl@ceph.osd.3.smithi099.stdout:Feb 09 17:59:10 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-3[280202]: 2024-02-09T17:59:10.911+0000 7fb8f50c1700 -1 osd.3 814 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-09T17:59:11.150 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:10 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1588651846' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:59:11.150 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:10 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:59:11.150 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:10 smithi099 ceph-mon[39420]: from='osd.3 [v2:172.21.15.99:6826/4191781533,v1:172.21.15.99:6827/4191781533]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-09T17:59:11.150 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:10 smithi099 ceph-mon[39420]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-09T17:59:11.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:10 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1588651846' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:59:11.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:10 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:59:11.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:10 smithi179 ceph-mon[43975]: from='osd.3 [v2:172.21.15.99:6826/4191781533,v1:172.21.15.99:6827/4191781533]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-09T17:59:11.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:10 smithi179 ceph-mon[43975]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-09T17:59:11.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:10 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1588651846' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:59:11.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:10 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:59:11.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:10 smithi162 ceph-mon[44304]: from='osd.3 [v2:172.21.15.99:6826/4191781533,v1:172.21.15.99:6827/4191781533]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-09T17:59:11.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:10 smithi162 ceph-mon[44304]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-09T17:59:11.434 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:59:11.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:11.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:59:11.539 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:59:11.575 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:12.139 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:59:12.139 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-02-09T17:59:12.139 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-02-09T17:59:12.139 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-02-09T17:59:12.140 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.140 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.16440.0:185443 7.0 7:03c708bb:::benchmark_data_smithi179_135804_object77652:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e750)", 2024-02-09T17:59:12.140 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:54:03.197202+0000", 2024-02-09T17:59:12.140 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 308.93747671099999, 2024-02-09T17:59:12.140 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.21396905599999999, 2024-02-09T17:59:12.141 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:59:12.141 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:59:12.141 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:59:12.141 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.16440", 2024-02-09T17:59:12.141 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:59:12.141 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 185443 2024-02-09T17:59:12.142 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.142 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:59:12.142 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.142 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:59:12.142 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.197202+0000", 2024-02-09T17:59:12.142 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:59:12.142 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.142 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.142 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:59:12.143 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.197202+0000", 2024-02-09T17:59:12.143 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:59:12.143 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.143 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.143 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:59:12.144 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.197203+0000", 2024-02-09T17:59:12.144 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.672e-06 2024-02-09T17:59:12.144 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.144 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.144 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:59:12.144 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.197208+0000", 2024-02-09T17:59:12.144 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0139999999999998e-06 2024-02-09T17:59:12.144 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.144 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.144 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:59:12.144 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.197220+0000", 2024-02-09T17:59:12.144 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1234e-05 2024-02-09T17:59:12.144 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.145 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.145 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:59:12.145 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.197226+0000", 2024-02-09T17:59:12.145 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6120000000000004e-06 2024-02-09T17:59:12.145 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.145 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.145 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:59:12.145 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.197286+0000", 2024-02-09T17:59:12.145 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0189000000000003e-05 2024-02-09T17:59:12.145 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.145 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.145 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:59:12.145 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.197430+0000", 2024-02-09T17:59:12.146 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000143953 2024-02-09T17:59:12.146 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.146 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.146 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,5", 2024-02-09T17:59:12.146 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.197567+0000", 2024-02-09T17:59:12.146 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013683400000000001 2024-02-09T17:59:12.146 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.146 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.146 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:59:12.146 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.207675+0000", 2024-02-09T17:59:12.146 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010107649999999999 2024-02-09T17:59:12.146 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.146 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.147 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:59:12.147 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.219491+0000", 2024-02-09T17:59:12.147 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011816455999999999 2024-02-09T17:59:12.147 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.147 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.147 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:59:12.147 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.411090+0000", 2024-02-09T17:59:12.147 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.19159906800000001 2024-02-09T17:59:12.147 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.147 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.147 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:59:12.147 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.411139+0000", 2024-02-09T17:59:12.147 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8937000000000003e-05 2024-02-09T17:59:12.148 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.148 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.148 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:59:12.148 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.411171+0000", 2024-02-09T17:59:12.148 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1436999999999997e-05 2024-02-09T17:59:12.148 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:59:12.148 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:59:12.148 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:59:12.148 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.148 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.148 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.16440.0:185444 7.0 7:0d715ece:::benchmark_data_smithi179_135804_object77653:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e750)", 2024-02-09T17:59:12.148 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:54:03.198499+0000", 2024-02-09T17:59:12.148 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 308.93617958700003, 2024-02-09T17:59:12.149 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.22028868099999999, 2024-02-09T17:59:12.149 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:59:12.149 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:59:12.149 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:59:12.149 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.16440", 2024-02-09T17:59:12.149 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T17:59:12.149 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 185444 2024-02-09T17:59:12.149 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.149 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:59:12.150 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.150 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:59:12.150 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.198499+0000", 2024-02-09T17:59:12.150 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:59:12.150 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.150 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.150 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:59:12.150 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.198499+0000", 2024-02-09T17:59:12.150 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:59:12.150 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.150 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.151 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:59:12.151 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.198500+0000", 2024-02-09T17:59:12.151 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.9699999999999994e-07 2024-02-09T17:59:12.151 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.151 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.151 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:59:12.151 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.198504+0000", 2024-02-09T17:59:12.151 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4950000000000002e-06 2024-02-09T17:59:12.151 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.151 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.151 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:59:12.151 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.198514+0000", 2024-02-09T17:59:12.151 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0247e-05 2024-02-09T17:59:12.151 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.152 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.152 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:59:12.152 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.198521+0000", 2024-02-09T17:59:12.152 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.888e-06 2024-02-09T17:59:12.152 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.152 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.152 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:59:12.152 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.202421+0000", 2024-02-09T17:59:12.152 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0038995330000000002 2024-02-09T17:59:12.152 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.152 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.152 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:59:12.152 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.202583+0000", 2024-02-09T17:59:12.153 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016227 2024-02-09T17:59:12.153 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.153 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.153 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,5", 2024-02-09T17:59:12.153 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.202753+0000", 2024-02-09T17:59:12.153 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016971900000000001 2024-02-09T17:59:12.153 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.153 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.153 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:59:12.153 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.208757+0000", 2024-02-09T17:59:12.153 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0060043859999999996 2024-02-09T17:59:12.153 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.153 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.153 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:59:12.153 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.219534+0000", 2024-02-09T17:59:12.154 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010776737 2024-02-09T17:59:12.154 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.154 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.154 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:59:12.154 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.418698+0000", 2024-02-09T17:59:12.154 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.19916429699999999 2024-02-09T17:59:12.154 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.154 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.154 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:59:12.154 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.418749+0000", 2024-02-09T17:59:12.154 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0470999999999998e-05 2024-02-09T17:59:12.154 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.154 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.154 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:59:12.154 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:54:03.418787+0000", 2024-02-09T17:59:12.154 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8640999999999998e-05 2024-02-09T17:59:12.154 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:59:12.155 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:59:12.155 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:59:12.155 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.155 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.155 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:149659 8.8 8:110929c2:::benchmark_data_smithi179_148205_object26587:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e826)", 2024-02-09T17:59:12.155 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:57:26.089697+0000", 2024-02-09T17:59:12.155 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 106.044981173, 2024-02-09T17:59:12.155 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.22273084000000001, 2024-02-09T17:59:12.155 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:59:12.155 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:59:12.155 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:59:12.155 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T17:59:12.155 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T17:59:12.155 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 149659 2024-02-09T17:59:12.155 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.156 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:59:12.156 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.156 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:59:12.156 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.089697+0000", 2024-02-09T17:59:12.156 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:59:12.156 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.156 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.156 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:59:12.156 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.089697+0000", 2024-02-09T17:59:12.156 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:59:12.156 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.156 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.156 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:59:12.157 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.089697+0000", 2024-02-09T17:59:12.157 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.3200000000000001e-07 2024-02-09T17:59:12.157 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.157 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.157 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:59:12.157 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.089700+0000", 2024-02-09T17:59:12.157 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1679999999999998e-06 2024-02-09T17:59:12.157 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.157 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.157 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:59:12.157 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.089706+0000", 2024-02-09T17:59:12.157 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8310000000000003e-06 2024-02-09T17:59:12.157 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.157 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.157 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:59:12.157 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.089712+0000", 2024-02-09T17:59:12.158 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5840000000000002e-06 2024-02-09T17:59:12.158 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.158 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.158 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:59:12.158 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.090056+0000", 2024-02-09T17:59:12.158 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00034346799999999999 2024-02-09T17:59:12.158 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.158 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.158 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:59:12.158 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.090176+0000", 2024-02-09T17:59:12.158 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012014 2024-02-09T17:59:12.158 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.158 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.159 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-02-09T17:59:12.159 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.090294+0000", 2024-02-09T17:59:12.159 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011880200000000001 2024-02-09T17:59:12.159 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.159 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.159 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:59:12.159 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.117377+0000", 2024-02-09T17:59:12.159 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.027082812000000001 2024-02-09T17:59:12.159 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.159 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.159 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:59:12.159 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.312327+0000", 2024-02-09T17:59:12.159 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.194950186 2024-02-09T17:59:12.160 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.160 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.160 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:59:12.160 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.312377+0000", 2024-02-09T17:59:12.160 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9329999999999997e-05 2024-02-09T17:59:12.160 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.160 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.160 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:59:12.160 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.312428+0000", 2024-02-09T17:59:12.160 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1187000000000003e-05 2024-02-09T17:59:12.160 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:59:12.160 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:59:12.160 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:59:12.161 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.161 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.161 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:149670 8.4 8:2f78b43d:::benchmark_data_smithi179_148205_object26598:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e826)", 2024-02-09T17:59:12.161 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:57:26.089894+0000", 2024-02-09T17:59:12.161 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 106.044784311, 2024-02-09T17:59:12.161 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.22246386800000001, 2024-02-09T17:59:12.161 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:59:12.161 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:59:12.161 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:59:12.161 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T17:59:12.161 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T17:59:12.161 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 149670 2024-02-09T17:59:12.161 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.161 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:59:12.162 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.162 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:59:12.162 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.089894+0000", 2024-02-09T17:59:12.162 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:59:12.162 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.162 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.162 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:59:12.162 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.089894+0000", 2024-02-09T17:59:12.162 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:59:12.162 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.162 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.162 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:59:12.162 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.089894+0000", 2024-02-09T17:59:12.162 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6600000000000002e-07 2024-02-09T17:59:12.162 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.162 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.163 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:59:12.163 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.089897+0000", 2024-02-09T17:59:12.163 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5179999999999999e-06 2024-02-09T17:59:12.163 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.163 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.163 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:59:12.163 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.089903+0000", 2024-02-09T17:59:12.163 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1480000000000003e-06 2024-02-09T17:59:12.163 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.163 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.163 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:59:12.163 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.089908+0000", 2024-02-09T17:59:12.163 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3070000000000002e-06 2024-02-09T17:59:12.163 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.164 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.164 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:59:12.164 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.097177+0000", 2024-02-09T17:59:12.164 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0072688199999999996 2024-02-09T17:59:12.164 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.164 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.164 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:59:12.164 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.097255+0000", 2024-02-09T17:59:12.164 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7460999999999996e-05 2024-02-09T17:59:12.164 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.164 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.164 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,10", 2024-02-09T17:59:12.164 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.097316+0000", 2024-02-09T17:59:12.165 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1753000000000003e-05 2024-02-09T17:59:12.165 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.165 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.165 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:59:12.165 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.110956+0000", 2024-02-09T17:59:12.165 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013639347 2024-02-09T17:59:12.165 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.165 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.165 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:59:12.165 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.111147+0000", 2024-02-09T17:59:12.165 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00019126200000000001 2024-02-09T17:59:12.165 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.165 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.165 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:59:12.165 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.312274+0000", 2024-02-09T17:59:12.165 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.201127057 2024-02-09T17:59:12.166 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.166 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.166 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:59:12.166 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.312328+0000", 2024-02-09T17:59:12.166 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.376e-05 2024-02-09T17:59:12.166 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.166 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.166 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:59:12.166 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.312358+0000", 2024-02-09T17:59:12.166 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0069e-05 2024-02-09T17:59:12.166 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:59:12.166 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:59:12.166 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:59:12.166 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.167 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.167 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:149671 8.0 8:0e2f9c24:::benchmark_data_smithi179_148205_object26599:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e826)", 2024-02-09T17:59:12.167 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:57:26.089915+0000", 2024-02-09T17:59:12.167 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 106.044763488, 2024-02-09T17:59:12.167 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.22250051400000001, 2024-02-09T17:59:12.167 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:59:12.167 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:59:12.167 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:59:12.167 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T17:59:12.167 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T17:59:12.167 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 149671 2024-02-09T17:59:12.167 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.168 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:59:12.168 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.168 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:59:12.168 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.089915+0000", 2024-02-09T17:59:12.168 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:59:12.168 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.168 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.168 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:59:12.168 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.089915+0000", 2024-02-09T17:59:12.168 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:59:12.168 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.168 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.168 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:59:12.168 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.089915+0000", 2024-02-09T17:59:12.169 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7300000000000002e-07 2024-02-09T17:59:12.169 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.169 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.169 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:59:12.169 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.089917+0000", 2024-02-09T17:59:12.169 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.12e-06 2024-02-09T17:59:12.169 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.169 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.169 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:59:12.169 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.089923+0000", 2024-02-09T17:59:12.169 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8649999999999998e-06 2024-02-09T17:59:12.169 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.169 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.169 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:59:12.169 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.089928+0000", 2024-02-09T17:59:12.170 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1540000000000001e-06 2024-02-09T17:59:12.170 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.170 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.170 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:59:12.170 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.090406+0000", 2024-02-09T17:59:12.170 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00047733699999999999 2024-02-09T17:59:12.170 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.170 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.170 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:59:12.170 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.090515+0000", 2024-02-09T17:59:12.170 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000108859 2024-02-09T17:59:12.170 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.170 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.170 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-02-09T17:59:12.170 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.090613+0000", 2024-02-09T17:59:12.171 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.8345999999999997e-05 2024-02-09T17:59:12.171 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.171 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.171 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:59:12.171 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.117389+0000", 2024-02-09T17:59:12.171 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.026776399999999999 2024-02-09T17:59:12.171 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.171 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.171 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:59:12.171 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.312290+0000", 2024-02-09T17:59:12.171 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.19490034000000001 2024-02-09T17:59:12.171 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.172 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.172 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:59:12.172 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.312331+0000", 2024-02-09T17:59:12.172 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1572000000000001e-05 2024-02-09T17:59:12.172 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.172 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.172 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:59:12.172 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.312415+0000", 2024-02-09T17:59:12.172 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4147999999999995e-05 2024-02-09T17:59:12.172 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:59:12.172 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:59:12.172 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:59:12.172 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.172 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.172 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:149672 8.0 8:029caa0a:::benchmark_data_smithi179_148205_object26600:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e826)", 2024-02-09T17:59:12.173 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:57:26.092158+0000", 2024-02-09T17:59:12.173 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 106.04252047600001, 2024-02-09T17:59:12.173 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.22041646100000001, 2024-02-09T17:59:12.173 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:59:12.173 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:59:12.173 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:59:12.173 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T17:59:12.174 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T17:59:12.174 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 149672 2024-02-09T17:59:12.174 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.174 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:59:12.174 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.174 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:59:12.174 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.092158+0000", 2024-02-09T17:59:12.174 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:59:12.174 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.174 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.174 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:59:12.174 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.092158+0000", 2024-02-09T17:59:12.174 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:59:12.175 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.175 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.175 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:59:12.175 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.092159+0000", 2024-02-09T17:59:12.175 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.8299999999999995e-07 2024-02-09T17:59:12.175 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.175 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.175 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:59:12.176 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.092164+0000", 2024-02-09T17:59:12.176 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6619999999999996e-06 2024-02-09T17:59:12.176 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.176 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.176 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:59:12.176 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.092174+0000", 2024-02-09T17:59:12.176 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0008e-05 2024-02-09T17:59:12.176 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.176 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.176 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:59:12.177 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.092181+0000", 2024-02-09T17:59:12.177 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6459999999999993e-06 2024-02-09T17:59:12.177 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.177 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.177 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:59:12.177 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.099362+0000", 2024-02-09T17:59:12.177 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0071812400000000002 2024-02-09T17:59:12.177 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.177 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.177 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:59:12.177 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.099478+0000", 2024-02-09T17:59:12.177 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000115896 2024-02-09T17:59:12.177 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.177 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.178 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-02-09T17:59:12.178 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.099581+0000", 2024-02-09T17:59:12.178 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0001031 2024-02-09T17:59:12.178 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.178 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.178 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:59:12.178 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.117395+0000", 2024-02-09T17:59:12.179 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017813711999999999 2024-02-09T17:59:12.179 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.179 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.179 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:59:12.179 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.312450+0000", 2024-02-09T17:59:12.179 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.19505461199999999 2024-02-09T17:59:12.180 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.180 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.180 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:59:12.180 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.312496+0000", 2024-02-09T17:59:12.180 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6569000000000001e-05 2024-02-09T17:59:12.180 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.181 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.181 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:59:12.181 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.312574+0000", 2024-02-09T17:59:12.181 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8033000000000003e-05 2024-02-09T17:59:12.181 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:59:12.181 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:59:12.181 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:59:12.181 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.181 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.181 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:149675 8.4 8:274fa1cc:::benchmark_data_smithi179_148205_object26603:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e826)", 2024-02-09T17:59:12.181 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:57:26.099085+0000", 2024-02-09T17:59:12.181 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 106.035593808, 2024-02-09T17:59:12.182 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.22750145699999999, 2024-02-09T17:59:12.182 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:59:12.182 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:59:12.182 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:59:12.182 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T17:59:12.182 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T17:59:12.182 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 149675 2024-02-09T17:59:12.182 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.182 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:59:12.182 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.182 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:59:12.182 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.099085+0000", 2024-02-09T17:59:12.182 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:59:12.182 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.183 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.183 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:59:12.183 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.099085+0000", 2024-02-09T17:59:12.183 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:59:12.183 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.183 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.183 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:59:12.183 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.099086+0000", 2024-02-09T17:59:12.183 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.156e-06 2024-02-09T17:59:12.183 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.183 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.183 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:59:12.183 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.099090+0000", 2024-02-09T17:59:12.183 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7949999999999998e-06 2024-02-09T17:59:12.183 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.184 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.184 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:59:12.184 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.099101+0000", 2024-02-09T17:59:12.184 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0448999999999999e-05 2024-02-09T17:59:12.184 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.184 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.184 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:59:12.184 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.099108+0000", 2024-02-09T17:59:12.184 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.345e-06 2024-02-09T17:59:12.185 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.185 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.185 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:59:12.185 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.101512+0000", 2024-02-09T17:59:12.185 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0024040490000000001 2024-02-09T17:59:12.185 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.185 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.185 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:59:12.185 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.101623+0000", 2024-02-09T17:59:12.185 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000110485 2024-02-09T17:59:12.185 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.185 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.186 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,10", 2024-02-09T17:59:12.186 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.101729+0000", 2024-02-09T17:59:12.186 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000105788 2024-02-09T17:59:12.186 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.186 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.186 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:59:12.186 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.111034+0000", 2024-02-09T17:59:12.186 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0093058329999999995 2024-02-09T17:59:12.186 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.186 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.186 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:59:12.186 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.111185+0000", 2024-02-09T17:59:12.186 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015010199999999999 2024-02-09T17:59:12.187 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.187 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.187 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:59:12.187 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.326493+0000", 2024-02-09T17:59:12.187 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.215308465 2024-02-09T17:59:12.187 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.188 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.188 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:59:12.188 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.326548+0000", 2024-02-09T17:59:12.188 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4759999999999997e-05 2024-02-09T17:59:12.188 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.188 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.189 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:59:12.189 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.326586+0000", 2024-02-09T17:59:12.189 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8229999999999998e-05 2024-02-09T17:59:12.189 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:59:12.189 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:59:12.189 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:59:12.189 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.189 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.189 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:149677 8.0 8:0d59ad39:::benchmark_data_smithi179_148205_object26605:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e826)", 2024-02-09T17:59:12.190 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:57:26.099122+0000", 2024-02-09T17:59:12.190 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 106.03555653700001, 2024-02-09T17:59:12.190 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.224356316, 2024-02-09T17:59:12.190 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:59:12.190 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:59:12.190 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:59:12.190 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T17:59:12.190 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T17:59:12.190 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 149677 2024-02-09T17:59:12.190 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.190 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:59:12.191 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.191 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:59:12.191 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.099122+0000", 2024-02-09T17:59:12.191 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:59:12.191 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.191 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.191 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:59:12.191 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.099122+0000", 2024-02-09T17:59:12.191 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:59:12.191 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.192 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.192 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:59:12.192 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.099122+0000", 2024-02-09T17:59:12.192 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1399999999999997e-07 2024-02-09T17:59:12.192 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.192 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.192 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:59:12.192 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.099124+0000", 2024-02-09T17:59:12.192 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1900000000000002e-06 2024-02-09T17:59:12.192 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.192 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.192 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:59:12.192 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.099136+0000", 2024-02-09T17:59:12.192 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1308000000000001e-05 2024-02-09T17:59:12.193 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.193 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.193 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:59:12.193 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.099141+0000", 2024-02-09T17:59:12.193 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.446e-06 2024-02-09T17:59:12.193 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.193 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.193 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:59:12.193 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.104437+0000", 2024-02-09T17:59:12.193 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0052956260000000003 2024-02-09T17:59:12.193 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.193 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.193 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:59:12.193 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.104549+0000", 2024-02-09T17:59:12.193 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000111869 2024-02-09T17:59:12.193 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.194 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.194 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-02-09T17:59:12.194 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.104663+0000", 2024-02-09T17:59:12.194 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000114103 2024-02-09T17:59:12.194 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.194 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.194 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:59:12.194 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.117403+0000", 2024-02-09T17:59:12.194 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.012739968000000001 2024-02-09T17:59:12.194 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.194 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.194 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:59:12.194 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.323314+0000", 2024-02-09T17:59:12.194 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.205910818 2024-02-09T17:59:12.195 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.195 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.195 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:59:12.195 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.323374+0000", 2024-02-09T17:59:12.195 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0745000000000002e-05 2024-02-09T17:59:12.195 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.195 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.195 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:59:12.195 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.323478+0000", 2024-02-09T17:59:12.195 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000103829 2024-02-09T17:59:12.195 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:59:12.195 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:59:12.195 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:59:12.195 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.196 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.196 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:149679 8.8 8:12d0fffd:::benchmark_data_smithi179_148205_object26607:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e826)", 2024-02-09T17:59:12.196 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:57:26.099148+0000", 2024-02-09T17:59:12.196 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 106.035529997, 2024-02-09T17:59:12.196 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.22577888700000001, 2024-02-09T17:59:12.196 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:59:12.196 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:59:12.196 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:59:12.196 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T17:59:12.197 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T17:59:12.197 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 149679 2024-02-09T17:59:12.197 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.197 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:59:12.197 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.197 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:59:12.197 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.099148+0000", 2024-02-09T17:59:12.198 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:59:12.198 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.198 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.198 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:59:12.198 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.099148+0000", 2024-02-09T17:59:12.198 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:59:12.198 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.198 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.199 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:59:12.199 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.099149+0000", 2024-02-09T17:59:12.199 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1399999999999997e-07 2024-02-09T17:59:12.199 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.199 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.199 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:59:12.200 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.099151+0000", 2024-02-09T17:59:12.200 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5000000000000002e-06 2024-02-09T17:59:12.200 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.200 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.200 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:59:12.200 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.099157+0000", 2024-02-09T17:59:12.200 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4890000000000003e-06 2024-02-09T17:59:12.201 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.201 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.201 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:59:12.201 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.099161+0000", 2024-02-09T17:59:12.201 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4349999999999999e-06 2024-02-09T17:59:12.201 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.201 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.201 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:59:12.201 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.104420+0000", 2024-02-09T17:59:12.201 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0052591169999999998 2024-02-09T17:59:12.202 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.202 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.202 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:59:12.202 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.104559+0000", 2024-02-09T17:59:12.202 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013904999999999999 2024-02-09T17:59:12.202 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.202 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.202 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-02-09T17:59:12.202 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.104667+0000", 2024-02-09T17:59:12.202 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010785099999999999 2024-02-09T17:59:12.202 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.202 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.202 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:59:12.202 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.117746+0000", 2024-02-09T17:59:12.202 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013078601 2024-02-09T17:59:12.203 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.203 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.203 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:59:12.203 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.324784+0000", 2024-02-09T17:59:12.203 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.207038051 2024-02-09T17:59:12.203 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.203 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.203 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:59:12.203 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.324840+0000", 2024-02-09T17:59:12.203 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5949999999999998e-05 2024-02-09T17:59:12.203 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.203 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.203 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:59:12.203 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.324927+0000", 2024-02-09T17:59:12.203 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7429000000000006e-05 2024-02-09T17:59:12.204 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:59:12.204 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:59:12.204 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:59:12.204 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.204 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.204 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:149683 8.0 8:0a05b423:::benchmark_data_smithi179_148205_object26611:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e826)", 2024-02-09T17:59:12.204 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:57:26.100174+0000", 2024-02-09T17:59:12.204 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 106.03450480399999, 2024-02-09T17:59:12.204 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.232663442, 2024-02-09T17:59:12.204 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:59:12.204 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:59:12.204 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:59:12.205 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T17:59:12.205 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T17:59:12.205 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 149683 2024-02-09T17:59:12.205 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.205 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:59:12.205 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.205 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:59:12.205 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.100174+0000", 2024-02-09T17:59:12.205 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:59:12.205 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.205 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.205 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:59:12.205 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.100174+0000", 2024-02-09T17:59:12.205 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:59:12.206 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.206 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.206 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:59:12.206 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.100174+0000", 2024-02-09T17:59:12.206 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.6799999999999999e-07 2024-02-09T17:59:12.206 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.206 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.206 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:59:12.206 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.100179+0000", 2024-02-09T17:59:12.206 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6519999999999997e-06 2024-02-09T17:59:12.206 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.206 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.207 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:59:12.207 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.100193+0000", 2024-02-09T17:59:12.207 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4124000000000001e-05 2024-02-09T17:59:12.207 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.207 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.207 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:59:12.207 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.100201+0000", 2024-02-09T17:59:12.207 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.525e-06 2024-02-09T17:59:12.207 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.207 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.207 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:59:12.207 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.109190+0000", 2024-02-09T17:59:12.207 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0089890479999999995 2024-02-09T17:59:12.208 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.208 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.208 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:59:12.208 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.109299+0000", 2024-02-09T17:59:12.208 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000109423 2024-02-09T17:59:12.208 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.208 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.208 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-02-09T17:59:12.208 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.109403+0000", 2024-02-09T17:59:12.208 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000103639 2024-02-09T17:59:12.208 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.208 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.208 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:59:12.208 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.117761+0000", 2024-02-09T17:59:12.209 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0083584420000000006 2024-02-09T17:59:12.209 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.209 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.209 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:59:12.209 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.332703+0000", 2024-02-09T17:59:12.209 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.214942037 2024-02-09T17:59:12.209 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.209 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.209 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:59:12.209 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.332754+0000", 2024-02-09T17:59:12.209 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1082000000000003e-05 2024-02-09T17:59:12.209 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.209 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.209 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:59:12.210 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.332837+0000", 2024-02-09T17:59:12.210 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2602000000000006e-05 2024-02-09T17:59:12.210 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:59:12.210 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:59:12.210 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:59:12.210 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.210 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.210 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:149687 8.4 8:23f63abb:::benchmark_data_smithi179_148205_object26615:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e826)", 2024-02-09T17:59:12.210 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:57:26.106520+0000", 2024-02-09T17:59:12.210 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 106.02815794599999, 2024-02-09T17:59:12.210 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.22018114899999999, 2024-02-09T17:59:12.210 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:59:12.210 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:59:12.210 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:59:12.210 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T17:59:12.211 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T17:59:12.211 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 149687 2024-02-09T17:59:12.211 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.211 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:59:12.211 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.211 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:59:12.211 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.106520+0000", 2024-02-09T17:59:12.211 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:59:12.211 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.211 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.211 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:59:12.211 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.106520+0000", 2024-02-09T17:59:12.211 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:59:12.211 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.212 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.212 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:59:12.212 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.106522+0000", 2024-02-09T17:59:12.212 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.511e-06 2024-02-09T17:59:12.212 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.212 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.212 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:59:12.212 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.106527+0000", 2024-02-09T17:59:12.212 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5790000000000003e-06 2024-02-09T17:59:12.212 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.212 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.212 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:59:12.212 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.106539+0000", 2024-02-09T17:59:12.212 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1412e-05 2024-02-09T17:59:12.213 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.213 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.213 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:59:12.213 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.106548+0000", 2024-02-09T17:59:12.213 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.1190000000000003e-06 2024-02-09T17:59:12.213 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.213 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.213 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:59:12.213 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.106962+0000", 2024-02-09T17:59:12.213 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00041418999999999999 2024-02-09T17:59:12.213 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.213 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.213 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:59:12.213 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.107087+0000", 2024-02-09T17:59:12.213 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012501499999999999 2024-02-09T17:59:12.214 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.214 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.214 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,10", 2024-02-09T17:59:12.214 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.107204+0000", 2024-02-09T17:59:12.214 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011710299999999999 2024-02-09T17:59:12.214 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.214 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.214 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:59:12.214 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.111255+0000", 2024-02-09T17:59:12.214 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0040507959999999997 2024-02-09T17:59:12.214 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.215 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.215 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:59:12.215 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.117993+0000", 2024-02-09T17:59:12.215 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0067382750000000002 2024-02-09T17:59:12.215 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.215 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.215 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:59:12.215 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.326622+0000", 2024-02-09T17:59:12.215 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.20862879500000001 2024-02-09T17:59:12.215 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.215 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.215 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:59:12.215 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.326669+0000", 2024-02-09T17:59:12.215 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6864999999999999e-05 2024-02-09T17:59:12.216 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.216 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.216 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:59:12.216 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.326702+0000", 2024-02-09T17:59:12.216 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2489e-05 2024-02-09T17:59:12.216 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:59:12.216 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:59:12.216 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:59:12.216 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.216 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.216 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:149688 8.8 8:16d18b7b:::benchmark_data_smithi179_148205_object26616:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e826)", 2024-02-09T17:59:12.216 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:57:26.106564+0000", 2024-02-09T17:59:12.216 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 106.028114317, 2024-02-09T17:59:12.216 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.21854638300000001, 2024-02-09T17:59:12.217 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:59:12.217 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:59:12.217 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:59:12.217 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T17:59:12.217 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T17:59:12.217 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 149688 2024-02-09T17:59:12.217 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.217 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:59:12.217 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.217 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:59:12.217 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.106564+0000", 2024-02-09T17:59:12.217 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:59:12.217 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.217 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.218 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:59:12.218 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.106564+0000", 2024-02-09T17:59:12.218 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:59:12.218 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.218 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.218 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:59:12.218 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.106565+0000", 2024-02-09T17:59:12.218 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3799999999999997e-07 2024-02-09T17:59:12.218 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.218 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.218 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:59:12.218 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.106567+0000", 2024-02-09T17:59:12.218 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.666e-06 2024-02-09T17:59:12.218 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.218 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.219 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:59:12.219 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.106574+0000", 2024-02-09T17:59:12.219 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9450000000000004e-06 2024-02-09T17:59:12.219 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.219 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.219 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:59:12.219 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.106580+0000", 2024-02-09T17:59:12.219 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7049999999999996e-06 2024-02-09T17:59:12.219 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.219 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.219 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:59:12.219 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.112909+0000", 2024-02-09T17:59:12.219 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0063293580000000002 2024-02-09T17:59:12.219 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.220 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.220 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:59:12.220 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.113015+0000", 2024-02-09T17:59:12.220 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000105809 2024-02-09T17:59:12.220 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.220 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.220 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-02-09T17:59:12.220 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.113126+0000", 2024-02-09T17:59:12.220 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000110796 2024-02-09T17:59:12.220 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.220 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.220 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:59:12.220 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.117767+0000", 2024-02-09T17:59:12.220 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0046414200000000003 2024-02-09T17:59:12.220 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.221 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.221 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:59:12.221 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.324976+0000", 2024-02-09T17:59:12.221 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.207208697 2024-02-09T17:59:12.221 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.221 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.221 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:59:12.221 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.325029+0000", 2024-02-09T17:59:12.221 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2589000000000001e-05 2024-02-09T17:59:12.221 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.221 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.221 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:59:12.221 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.325110+0000", 2024-02-09T17:59:12.221 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1860000000000006e-05 2024-02-09T17:59:12.221 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:59:12.222 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:59:12.222 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:59:12.222 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.222 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.222 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:149694 8.0 8:03ca8ca8:::benchmark_data_smithi179_148205_object26622:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e826)", 2024-02-09T17:59:12.222 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:57:26.118252+0000", 2024-02-09T17:59:12.222 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 106.01642613999999, 2024-02-09T17:59:12.222 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.21477158800000001, 2024-02-09T17:59:12.222 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:59:12.222 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:59:12.222 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:59:12.222 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T17:59:12.222 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T17:59:12.223 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 149694 2024-02-09T17:59:12.223 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.223 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:59:12.223 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.223 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:59:12.223 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.118252+0000", 2024-02-09T17:59:12.223 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:59:12.223 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.223 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.223 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:59:12.223 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.118252+0000", 2024-02-09T17:59:12.223 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:59:12.223 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.223 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.224 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:59:12.224 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.118254+0000", 2024-02-09T17:59:12.224 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.404e-06 2024-02-09T17:59:12.224 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.224 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.224 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:59:12.224 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.118258+0000", 2024-02-09T17:59:12.224 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2760000000000002e-06 2024-02-09T17:59:12.224 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.224 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.224 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:59:12.224 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.118270+0000", 2024-02-09T17:59:12.224 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1809e-05 2024-02-09T17:59:12.225 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.225 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.225 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:59:12.225 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.118277+0000", 2024-02-09T17:59:12.225 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.3749999999999997e-06 2024-02-09T17:59:12.225 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.225 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.225 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:59:12.225 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.118338+0000", 2024-02-09T17:59:12.225 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0668000000000003e-05 2024-02-09T17:59:12.225 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.225 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.225 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:59:12.225 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.118472+0000", 2024-02-09T17:59:12.225 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013422099999999999 2024-02-09T17:59:12.226 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.226 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.226 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-02-09T17:59:12.226 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.118542+0000", 2024-02-09T17:59:12.226 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.0389000000000007e-05 2024-02-09T17:59:12.226 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.226 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.226 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:59:12.226 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.122532+0000", 2024-02-09T17:59:12.226 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0039895210000000002 2024-02-09T17:59:12.226 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.226 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.226 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:59:12.226 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.332892+0000", 2024-02-09T17:59:12.226 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.21036048099999999 2024-02-09T17:59:12.227 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.227 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.227 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:59:12.227 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.332942+0000", 2024-02-09T17:59:12.227 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9916000000000001e-05 2024-02-09T17:59:12.227 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.227 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.227 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:59:12.227 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.333024+0000", 2024-02-09T17:59:12.227 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1527999999999996e-05 2024-02-09T17:59:12.227 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:59:12.227 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:59:12.227 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:59:12.227 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.227 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.228 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:149699 8.4 8:22f38004:::benchmark_data_smithi179_148205_object26627:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e826)", 2024-02-09T17:59:12.228 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:57:26.128906+0000", 2024-02-09T17:59:12.228 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 106.005771841, 2024-02-09T17:59:12.228 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.19790675399999999, 2024-02-09T17:59:12.228 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:59:12.228 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:59:12.228 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:59:12.228 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T17:59:12.228 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T17:59:12.228 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 149699 2024-02-09T17:59:12.228 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.228 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:59:12.228 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.229 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:59:12.229 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.128906+0000", 2024-02-09T17:59:12.229 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:59:12.229 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.229 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.229 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:59:12.229 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.128906+0000", 2024-02-09T17:59:12.229 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:59:12.229 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.229 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.229 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:59:12.229 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.128908+0000", 2024-02-09T17:59:12.229 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2759999999999999e-06 2024-02-09T17:59:12.229 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.229 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.230 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:59:12.230 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.128913+0000", 2024-02-09T17:59:12.230 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3340000000000001e-06 2024-02-09T17:59:12.232 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.232 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.232 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:59:12.232 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.128926+0000", 2024-02-09T17:59:12.232 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3234000000000001e-05 2024-02-09T17:59:12.232 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.233 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.233 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:59:12.233 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.128936+0000", 2024-02-09T17:59:12.233 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.9170000000000001e-06 2024-02-09T17:59:12.233 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.233 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.234 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:59:12.234 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.131557+0000", 2024-02-09T17:59:12.234 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0026205880000000001 2024-02-09T17:59:12.234 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.234 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.235 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:59:12.235 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.131678+0000", 2024-02-09T17:59:12.235 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000120931 2024-02-09T17:59:12.235 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.235 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.235 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,10", 2024-02-09T17:59:12.236 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.131793+0000", 2024-02-09T17:59:12.236 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000115356 2024-02-09T17:59:12.236 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.236 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.236 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:59:12.237 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.136810+0000", 2024-02-09T17:59:12.237 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0050166560000000004 2024-02-09T17:59:12.237 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.237 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.237 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:59:12.237 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.137360+0000", 2024-02-09T17:59:12.237 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00055053100000000005 2024-02-09T17:59:12.237 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.237 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.237 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:59:12.237 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.326735+0000", 2024-02-09T17:59:12.237 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.189374604 2024-02-09T17:59:12.238 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.238 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.238 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:59:12.238 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.326781+0000", 2024-02-09T17:59:12.238 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6502000000000003e-05 2024-02-09T17:59:12.239 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.239 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.239 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:59:12.239 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.326813+0000", 2024-02-09T17:59:12.239 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1825000000000001e-05 2024-02-09T17:59:12.239 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:59:12.239 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:59:12.239 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:59:12.239 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.239 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.239 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:149701 8.8 8:1383e751:::benchmark_data_smithi179_148205_object26629:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e826)", 2024-02-09T17:59:12.239 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:57:26.136017+0000", 2024-02-09T17:59:12.239 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 105.99866158899999, 2024-02-09T17:59:12.239 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.19697245699999999, 2024-02-09T17:59:12.239 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:59:12.240 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:59:12.240 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:59:12.240 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T17:59:12.240 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T17:59:12.240 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 149701 2024-02-09T17:59:12.240 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.240 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:59:12.240 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.240 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:59:12.240 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.136017+0000", 2024-02-09T17:59:12.240 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:59:12.240 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.240 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.240 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:59:12.241 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.136017+0000", 2024-02-09T17:59:12.241 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:59:12.241 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.241 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.241 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:59:12.241 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.136019+0000", 2024-02-09T17:59:12.241 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8679999999999999e-06 2024-02-09T17:59:12.241 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.241 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.241 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:59:12.241 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.136026+0000", 2024-02-09T17:59:12.241 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1189999999999999e-06 2024-02-09T17:59:12.241 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.241 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.242 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:59:12.242 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.136047+0000", 2024-02-09T17:59:12.242 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1243e-05 2024-02-09T17:59:12.242 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.242 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.242 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:59:12.242 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.136058+0000", 2024-02-09T17:59:12.242 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0577000000000001e-05 2024-02-09T17:59:12.242 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.242 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.242 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:59:12.242 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.136140+0000", 2024-02-09T17:59:12.242 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2396000000000002e-05 2024-02-09T17:59:12.242 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.243 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.243 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:59:12.243 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.136277+0000", 2024-02-09T17:59:12.243 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000137305 2024-02-09T17:59:12.243 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.243 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.243 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-02-09T17:59:12.243 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.136363+0000", 2024-02-09T17:59:12.243 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.6223000000000004e-05 2024-02-09T17:59:12.243 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.243 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.243 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:59:12.243 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.143146+0000", 2024-02-09T17:59:12.243 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0067825200000000002 2024-02-09T17:59:12.243 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.244 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.244 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:59:12.244 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.332884+0000", 2024-02-09T17:59:12.244 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.18973847099999999 2024-02-09T17:59:12.244 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.244 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.244 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:59:12.244 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.332922+0000", 2024-02-09T17:59:12.244 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7107999999999998e-05 2024-02-09T17:59:12.244 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.244 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.244 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:59:12.244 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.332989+0000", 2024-02-09T17:59:12.244 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7626999999999995e-05 2024-02-09T17:59:12.245 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:59:12.245 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:59:12.245 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:59:12.245 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.245 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.245 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.26305.0:222773 8.a e852/785 8:5930edee:::benchmark_data_smithi179_148205_object99701:head v 852'13755, mlcod=852'13753)", 2024-02-09T17:59:12.245 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:59:04.773166+0000", 2024-02-09T17:59:12.245 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 7.361512007, 2024-02-09T17:59:12.245 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.069119420000000001, 2024-02-09T17:59:12.245 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:59:12.245 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:59:12.245 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:59:12.245 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.245 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:59:12.245 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.773166+0000", 2024-02-09T17:59:12.245 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:59:12.245 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.246 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.246 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:59:12.246 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.773166+0000", 2024-02-09T17:59:12.246 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:59:12.246 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.246 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.246 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:59:12.246 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.773168+0000", 2024-02-09T17:59:12.246 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2899999999999999e-06 2024-02-09T17:59:12.246 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.246 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.246 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:59:12.247 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.773178+0000", 2024-02-09T17:59:12.247 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0657e-05 2024-02-09T17:59:12.247 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.247 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.247 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:59:12.247 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.773192+0000", 2024-02-09T17:59:12.247 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4218e-05 2024-02-09T17:59:12.247 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.247 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.247 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:59:12.247 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.773201+0000", 2024-02-09T17:59:12.247 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8850000000000005e-06 2024-02-09T17:59:12.247 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.247 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.248 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:59:12.248 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.773280+0000", 2024-02-09T17:59:12.248 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8212000000000003e-05 2024-02-09T17:59:12.248 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.248 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.248 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:59:12.248 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.773305+0000", 2024-02-09T17:59:12.248 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5009999999999999e-05 2024-02-09T17:59:12.248 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.248 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.248 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:59:12.248 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.787353+0000", 2024-02-09T17:59:12.249 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014048102 2024-02-09T17:59:12.249 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.249 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.249 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:59:12.249 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.842286+0000", 2024-02-09T17:59:12.249 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.054933045999999999 2024-02-09T17:59:12.249 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:59:12.249 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:59:12.249 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:59:12.249 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.249 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.249 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:222778 8.0 8:096ead4b:::benchmark_data_smithi179_148205_object99706:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e852)", 2024-02-09T17:59:12.249 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:59:04.781545+0000", 2024-02-09T17:59:12.249 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 7.3531334509999997, 2024-02-09T17:59:12.249 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.081490172, 2024-02-09T17:59:12.250 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:59:12.250 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:59:12.250 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:59:12.250 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T17:59:12.250 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T17:59:12.250 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 222778 2024-02-09T17:59:12.250 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.250 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:59:12.250 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.250 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:59:12.250 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.781545+0000", 2024-02-09T17:59:12.250 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:59:12.250 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.250 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.251 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:59:12.251 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.781545+0000", 2024-02-09T17:59:12.251 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:59:12.251 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.251 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.251 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:59:12.251 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.781547+0000", 2024-02-09T17:59:12.251 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8330000000000001e-06 2024-02-09T17:59:12.251 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.251 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.251 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:59:12.252 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.781551+0000", 2024-02-09T17:59:12.252 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5299999999999998e-06 2024-02-09T17:59:12.252 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.252 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.252 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:59:12.252 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.781564+0000", 2024-02-09T17:59:12.252 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2445e-05 2024-02-09T17:59:12.252 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.252 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.252 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:59:12.252 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.781572+0000", 2024-02-09T17:59:12.252 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1899999999999995e-06 2024-02-09T17:59:12.252 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.252 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.253 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:59:12.253 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.781734+0000", 2024-02-09T17:59:12.253 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000162592 2024-02-09T17:59:12.253 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.253 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.253 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:59:12.253 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.781839+0000", 2024-02-09T17:59:12.253 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000104688 2024-02-09T17:59:12.253 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.253 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.253 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-02-09T17:59:12.253 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.842289+0000", 2024-02-09T17:59:12.253 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.060449952000000001 2024-02-09T17:59:12.253 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.254 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.254 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:59:12.254 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.847622+0000", 2024-02-09T17:59:12.254 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0053327870000000003 2024-02-09T17:59:12.254 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.254 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.254 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:59:12.254 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.862916+0000", 2024-02-09T17:59:12.254 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.015294128000000001 2024-02-09T17:59:12.254 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.254 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.254 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:59:12.254 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.862970+0000", 2024-02-09T17:59:12.254 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3613999999999999e-05 2024-02-09T17:59:12.255 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.255 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.255 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:59:12.255 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.863035+0000", 2024-02-09T17:59:12.255 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5413000000000005e-05 2024-02-09T17:59:12.255 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:59:12.255 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:59:12.255 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:59:12.255 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.255 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.255 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:222779 8.8 8:18191bf0:::benchmark_data_smithi179_148205_object99707:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e852)", 2024-02-09T17:59:12.255 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:59:04.781587+0000", 2024-02-09T17:59:12.255 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 7.353091107, 2024-02-09T17:59:12.255 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.081527152000000006, 2024-02-09T17:59:12.256 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:59:12.256 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:59:12.256 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:59:12.256 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T17:59:12.256 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T17:59:12.256 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 222779 2024-02-09T17:59:12.256 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.256 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:59:12.256 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.256 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:59:12.256 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.781587+0000", 2024-02-09T17:59:12.256 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:59:12.256 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.256 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.256 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:59:12.257 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.781587+0000", 2024-02-09T17:59:12.257 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:59:12.257 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.257 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.257 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:59:12.257 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.781588+0000", 2024-02-09T17:59:12.257 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8700000000000001e-07 2024-02-09T17:59:12.257 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.257 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.257 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:59:12.257 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.781590+0000", 2024-02-09T17:59:12.257 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7319999999999998e-06 2024-02-09T17:59:12.257 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.257 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.257 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:59:12.258 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.781596+0000", 2024-02-09T17:59:12.258 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1090000000000003e-06 2024-02-09T17:59:12.258 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.258 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.258 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:59:12.258 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.781602+0000", 2024-02-09T17:59:12.258 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1309999999999997e-06 2024-02-09T17:59:12.258 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.258 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.258 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:59:12.258 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.842295+0000", 2024-02-09T17:59:12.258 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.060693511999999998 2024-02-09T17:59:12.258 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.258 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.258 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:59:12.258 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.842435+0000", 2024-02-09T17:59:12.258 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000139746 2024-02-09T17:59:12.259 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.259 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.259 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-02-09T17:59:12.259 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.842566+0000", 2024-02-09T17:59:12.259 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013088800000000001 2024-02-09T17:59:12.259 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.259 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.259 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:59:12.259 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.847675+0000", 2024-02-09T17:59:12.259 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0051091130000000002 2024-02-09T17:59:12.259 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.259 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.259 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:59:12.259 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.863042+0000", 2024-02-09T17:59:12.259 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.015367621 2024-02-09T17:59:12.259 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.260 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.260 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:59:12.260 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.863069+0000", 2024-02-09T17:59:12.260 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6290000000000001e-05 2024-02-09T17:59:12.260 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.260 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.260 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:59:12.260 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.863114+0000", 2024-02-09T17:59:12.260 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5623e-05 2024-02-09T17:59:12.260 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:59:12.260 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:59:12.260 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:59:12.260 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.260 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.260 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:222780 8.c 8:304ad53c:::benchmark_data_smithi179_148205_object99708:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e852)", 2024-02-09T17:59:12.261 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:59:04.781608+0000", 2024-02-09T17:59:12.261 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 7.3530702000000003, 2024-02-09T17:59:12.261 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.082251355999999998, 2024-02-09T17:59:12.261 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:59:12.261 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:59:12.261 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:59:12.261 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T17:59:12.261 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T17:59:12.261 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 222780 2024-02-09T17:59:12.261 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.261 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:59:12.261 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.261 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:59:12.261 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.781608+0000", 2024-02-09T17:59:12.261 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:59:12.262 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.262 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.262 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:59:12.262 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.781608+0000", 2024-02-09T17:59:12.262 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:59:12.262 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.262 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.262 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:59:12.262 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.781608+0000", 2024-02-09T17:59:12.262 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.9200000000000002e-07 2024-02-09T17:59:12.262 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.262 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.262 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:59:12.262 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.781611+0000", 2024-02-09T17:59:12.262 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5069999999999999e-06 2024-02-09T17:59:12.262 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.263 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.263 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:59:12.263 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.781617+0000", 2024-02-09T17:59:12.263 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1410000000000002e-06 2024-02-09T17:59:12.263 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.263 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.263 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:59:12.263 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.781622+0000", 2024-02-09T17:59:12.263 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9660000000000004e-06 2024-02-09T17:59:12.263 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.263 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.263 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:59:12.263 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.781677+0000", 2024-02-09T17:59:12.263 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5234e-05 2024-02-09T17:59:12.263 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.264 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.264 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:59:12.264 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.781798+0000", 2024-02-09T17:59:12.264 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000121063 2024-02-09T17:59:12.264 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.264 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.264 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 8", 2024-02-09T17:59:12.264 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.781909+0000", 2024-02-09T17:59:12.264 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000111079 2024-02-09T17:59:12.264 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.264 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.264 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:59:12.264 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.852868+0000", 2024-02-09T17:59:12.265 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.070959072999999998 2024-02-09T17:59:12.265 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.265 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.265 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:59:12.265 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.863703+0000", 2024-02-09T17:59:12.265 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010834165 2024-02-09T17:59:12.265 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.265 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.265 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:59:12.265 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.863759+0000", 2024-02-09T17:59:12.265 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6536999999999998e-05 2024-02-09T17:59:12.265 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.265 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.265 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:59:12.266 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.863859+0000", 2024-02-09T17:59:12.266 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000100299 2024-02-09T17:59:12.266 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:59:12.266 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:59:12.266 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:59:12.266 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.266 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.266 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:222785 8.c 8:309a4711:::benchmark_data_smithi179_148205_object99713:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e852)", 2024-02-09T17:59:12.266 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:59:04.787470+0000", 2024-02-09T17:59:12.266 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 7.3472078590000001, 2024-02-09T17:59:12.266 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.076616835999999994, 2024-02-09T17:59:12.266 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T17:59:12.266 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T17:59:12.266 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T17:59:12.266 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T17:59:12.267 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T17:59:12.267 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 222785 2024-02-09T17:59:12.267 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.267 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T17:59:12.267 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.267 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T17:59:12.267 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.787470+0000", 2024-02-09T17:59:12.267 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:59:12.267 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.267 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.267 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T17:59:12.267 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.787470+0000", 2024-02-09T17:59:12.267 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T17:59:12.267 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.267 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.267 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T17:59:12.267 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.787472+0000", 2024-02-09T17:59:12.268 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.511e-06 2024-02-09T17:59:12.268 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.268 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.268 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T17:59:12.268 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.787479+0000", 2024-02-09T17:59:12.268 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6290000000000004e-06 2024-02-09T17:59:12.268 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.268 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.268 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T17:59:12.268 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.842307+0000", 2024-02-09T17:59:12.268 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.054828676 2024-02-09T17:59:12.268 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.268 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.268 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T17:59:12.269 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.842319+0000", 2024-02-09T17:59:12.269 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1693e-05 2024-02-09T17:59:12.269 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.269 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.269 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T17:59:12.269 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.847426+0000", 2024-02-09T17:59:12.269 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0051066139999999998 2024-02-09T17:59:12.269 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.269 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.269 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T17:59:12.269 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.847539+0000", 2024-02-09T17:59:12.269 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000113198 2024-02-09T17:59:12.269 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.269 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.269 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 8", 2024-02-09T17:59:12.269 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.847655+0000", 2024-02-09T17:59:12.270 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011595900000000001 2024-02-09T17:59:12.270 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.270 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.270 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T17:59:12.270 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.858966+0000", 2024-02-09T17:59:12.270 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011311231999999999 2024-02-09T17:59:12.270 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.270 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.270 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T17:59:12.270 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.863956+0000", 2024-02-09T17:59:12.270 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0049897810000000004 2024-02-09T17:59:12.270 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.270 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.270 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T17:59:12.270 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.864002+0000", 2024-02-09T17:59:12.270 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5899000000000002e-05 2024-02-09T17:59:12.271 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T17:59:12.271 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T17:59:12.271 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T17:59:12.271 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:04.864087+0000", 2024-02-09T17:59:12.271 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5643999999999995e-05 2024-02-09T17:59:12.271 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:59:12.271 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:59:12.271 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:59:12.271 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T17:59:12.271 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T17:59:12.271 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:59:12.272 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:11 smithi099 ceph-mon[39420]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:59:12.272 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:11 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:59:12.272 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:11 smithi099 ceph-mon[39420]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-09T17:59:12.272 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:11 smithi099 ceph-mon[39420]: osdmap e853: 12 total, 11 up, 12 in 2024-02-09T17:59:12.272 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:11 smithi099 ceph-mon[39420]: from='osd.3 [v2:172.21.15.99:6826/4191781533,v1:172.21.15.99:6827/4191781533]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-02-09T17:59:12.273 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:11 smithi099 ceph-mon[39420]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-02-09T17:59:12.273 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:11 smithi099 ceph-mon[39420]: pgmap v2126: 47 pgs: 1 active+clean+scrubbing+deep, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 1.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 746 op/s; 4162/63816 objects degraded (6.522%) 2024-02-09T17:59:12.273 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:11 smithi162 ceph-mon[44304]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:59:12.273 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:11 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:59:12.273 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:11 smithi162 ceph-mon[44304]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-09T17:59:12.273 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:11 smithi162 ceph-mon[44304]: osdmap e853: 12 total, 11 up, 12 in 2024-02-09T17:59:12.273 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:11 smithi162 ceph-mon[44304]: from='osd.3 [v2:172.21.15.99:6826/4191781533,v1:172.21.15.99:6827/4191781533]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-02-09T17:59:12.273 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:11 smithi162 ceph-mon[44304]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-02-09T17:59:12.273 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:11 smithi162 ceph-mon[44304]: pgmap v2126: 47 pgs: 1 active+clean+scrubbing+deep, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 1.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 746 op/s; 4162/63816 objects degraded (6.522%) 2024-02-09T17:59:12.273 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:11 smithi179 ceph-mon[43975]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:59:12.273 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:11 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:59:12.273 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:11 smithi179 ceph-mon[43975]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-09T17:59:12.273 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:11 smithi179 ceph-mon[43975]: osdmap e853: 12 total, 11 up, 12 in 2024-02-09T17:59:12.274 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:11 smithi179 ceph-mon[43975]: from='osd.3 [v2:172.21.15.99:6826/4191781533,v1:172.21.15.99:6827/4191781533]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-02-09T17:59:12.274 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:11 smithi179 ceph-mon[43975]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-02-09T17:59:12.274 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:11 smithi179 ceph-mon[43975]: pgmap v2126: 47 pgs: 1 active+clean+scrubbing+deep, 3 active+undersized+degraded, 5 active+undersized, 38 active+clean; 1.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 746 op/s; 4162/63816 objects degraded (6.522%) 2024-02-09T17:59:12.833 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:59:12.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:12.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:59:12.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:12 smithi179 ceph-mon[43975]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-09T17:59:12.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:12 smithi179 ceph-mon[43975]: osd.3 [v2:172.21.15.99:6826/4191781533,v1:172.21.15.99:6827/4191781533] boot 2024-02-09T17:59:12.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:12 smithi179 ceph-mon[43975]: osdmap e854: 12 total, 12 up, 12 in 2024-02-09T17:59:12.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:12 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T17:59:12.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:12 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 4162/63816 objects degraded (6.522%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:59:12.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:12 smithi179 ceph-mon[43975]: osdmap e855: 12 total, 12 up, 12 in 2024-02-09T17:59:12.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:12 smithi179 ceph-mon[43975]: 8.3 deep-scrub ok 2024-02-09T17:59:12.936 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:59:12.965 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:12.984 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:59:12.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:12.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:59:13.047 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.10 dump_ops_in_flight 2024-02-09T17:59:13.089 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:59:13.120 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:13.141 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:59:13.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:13.144 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:59:13.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:12 smithi162 ceph-mon[44304]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-09T17:59:13.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:12 smithi162 ceph-mon[44304]: osd.3 [v2:172.21.15.99:6826/4191781533,v1:172.21.15.99:6827/4191781533] boot 2024-02-09T17:59:13.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:12 smithi162 ceph-mon[44304]: osdmap e854: 12 total, 12 up, 12 in 2024-02-09T17:59:13.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:12 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T17:59:13.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:12 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 4162/63816 objects degraded (6.522%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:59:13.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:12 smithi162 ceph-mon[44304]: osdmap e855: 12 total, 12 up, 12 in 2024-02-09T17:59:13.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:12 smithi162 ceph-mon[44304]: 8.3 deep-scrub ok 2024-02-09T17:59:13.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:12 smithi099 ceph-mon[39420]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-09T17:59:13.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:12 smithi099 ceph-mon[39420]: osd.3 [v2:172.21.15.99:6826/4191781533,v1:172.21.15.99:6827/4191781533] boot 2024-02-09T17:59:13.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:12 smithi099 ceph-mon[39420]: osdmap e854: 12 total, 12 up, 12 in 2024-02-09T17:59:13.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:12 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T17:59:13.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:12 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 4162/63816 objects degraded (6.522%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-09T17:59:13.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:12 smithi099 ceph-mon[39420]: osdmap e855: 12 total, 12 up, 12 in 2024-02-09T17:59:13.245 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:12 smithi099 ceph-mon[39420]: 8.3 deep-scrub ok 2024-02-09T17:59:13.245 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:59:13.277 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:13.305 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:59:13.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:13.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:59:13.398 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:59:13.409 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:59:13.447 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:13.528 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:59:13.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:13.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:59:13.653 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:59:13.703 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:14.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:14 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:59:14.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:14 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:59:14.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:14 smithi179 ceph-mon[43975]: osdmap e856: 12 total, 12 up, 12 in 2024-02-09T17:59:14.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:14 smithi179 ceph-mon[43975]: pgmap v2130: 47 pgs: 1 remapped+peering, 1 active+clean+scrubbing+deep, 2 active+undersized+degraded, 4 active+undersized, 39 active+clean; 1.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 489 op/s; 2787/59064 objects degraded (4.719%) 2024-02-09T17:59:14.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:14 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:59:14.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:14 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:59:14.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:14 smithi162 ceph-mon[44304]: osdmap e856: 12 total, 12 up, 12 in 2024-02-09T17:59:14.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:14 smithi162 ceph-mon[44304]: pgmap v2130: 47 pgs: 1 remapped+peering, 1 active+clean+scrubbing+deep, 2 active+undersized+degraded, 4 active+undersized, 39 active+clean; 1.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 489 op/s; 2787/59064 objects degraded (4.719%) 2024-02-09T17:59:14.499 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:14 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:59:14.499 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:14 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:59:14.499 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:14 smithi099 ceph-mon[39420]: osdmap e856: 12 total, 12 up, 12 in 2024-02-09T17:59:14.499 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:14 smithi099 ceph-mon[39420]: pgmap v2130: 47 pgs: 1 remapped+peering, 1 active+clean+scrubbing+deep, 2 active+undersized+degraded, 4 active+undersized, 39 active+clean; 1.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 489 op/s; 2787/59064 objects degraded (4.719%) 2024-02-09T17:59:14.811 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-09T17:59:15.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:15 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:59:15.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:15 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:59:15.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:15 smithi099 ceph-mon[39420]: osdmap e857: 12 total, 12 up, 12 in 2024-02-09T17:59:15.273 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:15 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:59:15.273 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:15 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:59:15.273 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:15 smithi179 ceph-mon[43975]: osdmap e857: 12 total, 12 up, 12 in 2024-02-09T17:59:15.273 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:59:15.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:15.279 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:59:15.380 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:59:15.391 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T17:59:15.391 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [], 2024-02-09T17:59:15.391 INFO:teuthology.orchestra.run.smithi099.stdout: "num_ops": 0 2024-02-09T17:59:15.391 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T17:59:15.399 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:59:15.406 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:15 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:59:15.406 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:15 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:59:15.407 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:15 smithi162 ceph-mon[44304]: osdmap e857: 12 total, 12 up, 12 in 2024-02-09T17:59:15.409 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:15.433 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:59:15.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:15.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:59:15.451 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:59:15.508 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:59:15.536 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:59:15.539 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:59:15.565 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:59:15.579 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:15.600 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:59:15.606 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:59:15.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:15.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:59:15.646 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:59:15.671 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:59:15.694 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:59:15.713 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:59:15.727 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:59:15.763 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:15.777 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:59:15.818 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:59:15.861 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:59:15.869 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:59:15.869 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:59:15.869 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:59:15.869 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:59:15.869 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:59:15.869 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:59:15.869 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:59:15.869 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:59:15.869 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:59:15.869 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:59:15.869 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:59:15.869 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:59:16.303 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:16 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:59:16.303 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:16 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:59:16.303 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:16 smithi099 ceph-mon[39420]: pgmap v2132: 47 pgs: 1 remapped+peering, 2 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 3 active+undersized, 40 active+clean; 1.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 750 op/s; 1635/54813 objects degraded (2.983%); 688 KiB/s, 10 objects/s recovering 2024-02-09T17:59:16.303 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:16 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3957007023' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:59:16.361 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:59:16.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:16.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:59:16.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:16 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:59:16.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:16 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:59:16.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:16 smithi179 ceph-mon[43975]: pgmap v2132: 47 pgs: 1 remapped+peering, 2 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 3 active+undersized, 40 active+clean; 1.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 750 op/s; 1635/54813 objects degraded (2.983%); 688 KiB/s, 10 objects/s recovering 2024-02-09T17:59:16.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:16 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3957007023' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:59:16.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:16 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:59:16.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:16 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:59:16.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:16 smithi162 ceph-mon[44304]: pgmap v2132: 47 pgs: 1 remapped+peering, 2 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 3 active+undersized, 40 active+clean; 1.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 750 op/s; 1635/54813 objects degraded (2.983%); 688 KiB/s, 10 objects/s recovering 2024-02-09T17:59:16.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:16 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3957007023' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:59:16.466 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:59:16.540 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:16.578 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:59:16.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:16.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:59:16.683 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:59:16.729 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:16.843 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:59:16.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:16.846 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:59:16.947 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:59:16.974 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:16.998 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:59:17.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:17.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:59:17.103 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:59:17.158 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:17 smithi099 ceph-mon[39420]: osdmap e858: 12 total, 12 up, 12 in 2024-02-09T17:59:17.158 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:17 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 1635/54813 objects degraded (2.983%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-09T17:59:17.158 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:17 smithi099 ceph-mon[39420]: osdmap e859: 12 total, 12 up, 12 in 2024-02-09T17:59:17.161 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:17.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:17 smithi179 ceph-mon[43975]: osdmap e858: 12 total, 12 up, 12 in 2024-02-09T17:59:17.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:17 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 1635/54813 objects degraded (2.983%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-09T17:59:17.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:17 smithi179 ceph-mon[43975]: osdmap e859: 12 total, 12 up, 12 in 2024-02-09T17:59:17.334 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T17:59:17.334 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:59:17.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:17 smithi162 ceph-mon[44304]: osdmap e858: 12 total, 12 up, 12 in 2024-02-09T17:59:17.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:17 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 1635/54813 objects degraded (2.983%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-09T17:59:17.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:17 smithi162 ceph-mon[44304]: osdmap e859: 12 total, 12 up, 12 in 2024-02-09T17:59:17.617 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:59:17.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:17.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:59:17.722 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:59:17.751 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:17.784 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:59:17.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:17.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:59:17.888 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:59:17.918 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:17.947 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:59:17.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:17.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:59:18.051 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:59:18.105 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:18.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:18 smithi099 ceph-mon[39420]: pgmap v2135: 47 pgs: 1 remapped+peering, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 472 op/s; 1635/53040 objects degraded (3.083%); 685 KiB/s, 10 objects/s recovering 2024-02-09T17:59:18.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:18 smithi179 ceph-mon[43975]: pgmap v2135: 47 pgs: 1 remapped+peering, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 472 op/s; 1635/53040 objects degraded (3.083%); 685 KiB/s, 10 objects/s recovering 2024-02-09T17:59:18.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:18 smithi162 ceph-mon[44304]: pgmap v2135: 47 pgs: 1 remapped+peering, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 472 op/s; 1635/53040 objects degraded (3.083%); 685 KiB/s, 10 objects/s recovering 2024-02-09T17:59:19.531 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:59:19.966 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:59:20.370 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is set 2024-02-09T17:59:20.550 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:20 smithi099 ceph-mon[39420]: pgmap v2136: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 1012 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 583 op/s; 2332/48555 objects degraded (4.803%); 1.7 MiB/s, 26 objects/s recovering 2024-02-09T17:59:20.550 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:20 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:59:20.550 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:20 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:59:20.550 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:20 smithi099 ceph-mon[39420]: osdmap e860: 12 total, 12 up, 12 in 2024-02-09T17:59:20.550 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:20 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3565867860' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:59:20.550 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:20 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:59:20.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:20 smithi179 ceph-mon[43975]: pgmap v2136: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 1012 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 583 op/s; 2332/48555 objects degraded (4.803%); 1.7 MiB/s, 26 objects/s recovering 2024-02-09T17:59:20.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:20 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:59:20.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:20 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:59:20.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:20 smithi179 ceph-mon[43975]: osdmap e860: 12 total, 12 up, 12 in 2024-02-09T17:59:20.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:20 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3565867860' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:59:20.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:20 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:59:20.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:20 smithi162 ceph-mon[44304]: pgmap v2136: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 1012 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 583 op/s; 2332/48555 objects degraded (4.803%); 1.7 MiB/s, 26 objects/s recovering 2024-02-09T17:59:20.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:20 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:59:20.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:20 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:59:20.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:20 smithi162 ceph-mon[44304]: osdmap e860: 12 total, 12 up, 12 in 2024-02-09T17:59:20.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:20 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3565867860' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:59:20.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:20 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:59:21.341 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.10 dump_blocked_ops 2024-02-09T17:59:21.350 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 7, 9, 10, 11, 8, 3] 2024-02-09T17:59:21.350 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T17:59:21.350 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 2] 2024-02-09T17:59:21.351 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd out 4 2024-02-09T17:59:21.389 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:59:21.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:21.393 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:59:21.494 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:59:21.561 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:21 smithi179 ceph-mon[43975]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:59:21.561 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:21 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:59:21.561 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:21 smithi179 ceph-mon[43975]: osdmap e861: 12 total, 12 up, 12 in 2024-02-09T17:59:21.561 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:21.594 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:59:21.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:21.601 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:59:21.624 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:21 smithi099 ceph-mon[39420]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:59:21.624 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:21 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:59:21.624 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:21 smithi099 ceph-mon[39420]: osdmap e861: 12 total, 12 up, 12 in 2024-02-09T17:59:21.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:21 smithi162 ceph-mon[44304]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:59:21.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:21 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T17:59:21.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:21 smithi162 ceph-mon[44304]: osdmap e861: 12 total, 12 up, 12 in 2024-02-09T17:59:21.702 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:59:21.757 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:21.772 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:59:22.447 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:22 smithi099 ceph-mon[39420]: pgmap v2139: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 963 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 547 op/s; 2079/46215 objects degraded (4.499%); 4.4 MiB/s, 70 objects/s recovering 2024-02-09T17:59:22.447 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:22 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:59:22.447 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:22 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:59:22.447 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:22 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 2079/46215 objects degraded (4.499%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:59:22.448 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:22 smithi099 ceph-mon[39420]: osdmap e862: 12 total, 12 up, 12 in 2024-02-09T17:59:22.448 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:59:22.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:22.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:59:22.563 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:59:22.594 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:22.621 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:59:22.627 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:22 smithi162 ceph-mon[44304]: pgmap v2139: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 963 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 547 op/s; 2079/46215 objects degraded (4.499%); 4.4 MiB/s, 70 objects/s recovering 2024-02-09T17:59:22.627 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:22 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:59:22.627 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:22 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:59:22.627 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:22 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 2079/46215 objects degraded (4.499%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:59:22.627 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:22 smithi162 ceph-mon[44304]: osdmap e862: 12 total, 12 up, 12 in 2024-02-09T17:59:22.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:22.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:59:22.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:22 smithi179 ceph-mon[43975]: pgmap v2139: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 963 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 547 op/s; 2079/46215 objects degraded (4.499%); 4.4 MiB/s, 70 objects/s recovering 2024-02-09T17:59:22.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:22 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:59:22.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:22 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:59:22.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:22 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 2079/46215 objects degraded (4.499%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:59:22.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:22 smithi179 ceph-mon[43975]: osdmap e862: 12 total, 12 up, 12 in 2024-02-09T17:59:22.728 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:59:22.794 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:22.821 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:59:22.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:22.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:59:22.879 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:59:22.907 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:59:22.928 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:59:22.973 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:59:23.003 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:59:23.005 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:23.033 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:59:23.076 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:59:23.120 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:59:23.150 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:59:23.161 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:59:23.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:23.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:59:23.189 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:59:23.229 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:59:23.262 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:59:23.271 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:59:23.290 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:59:23.316 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:23.326 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:59:23.338 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:59:23.338 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:59:23.338 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:59:23.338 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:59:23.338 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:59:23.338 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:59:23.338 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:59:23.338 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:59:23.338 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:59:23.338 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:59:23.338 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:59:23.338 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:59:23.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:23 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2583067314' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:59:23.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:23 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2583067314' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:59:23.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:23 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2583067314' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:59:23.995 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:59:24.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:24.009 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:59:24.110 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:59:24.162 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:24.185 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset noscrub 2024-02-09T17:59:24.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:24 smithi099 ceph-mon[39420]: pgmap v2141: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 863 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 646 op/s; 1923/41400 objects degraded (4.645%); 5.5 MiB/s, 87 objects/s recovering 2024-02-09T17:59:24.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:24 smithi179 ceph-mon[43975]: pgmap v2141: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 863 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 646 op/s; 1923/41400 objects degraded (4.645%); 5.5 MiB/s, 87 objects/s recovering 2024-02-09T17:59:24.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:24 smithi162 ceph-mon[44304]: pgmap v2141: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 863 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 646 op/s; 1923/41400 objects degraded (4.645%); 5.5 MiB/s, 87 objects/s recovering 2024-02-09T17:59:24.730 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:59:24.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:24.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:59:24.842 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T17:59:24.887 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:25.321 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T17:59:25.325 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:25.326 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T17:59:25.427 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:59:25.458 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:25.485 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:59:25.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:25.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:59:25.589 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:59:25.629 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:25.701 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:59:25.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:25.705 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:59:25.806 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:59:25.849 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:25.880 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:59:25.945 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:59:25.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:25.949 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:59:25.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:25 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mon-a[39397]: 2024-02-09T17:59:25.768+0000 7feb57590700 -1 mon.a@0(leader).osd e862 definitely_dead 0 2024-02-09T17:59:26.049 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:59:26.081 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:26.104 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:59:26.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:26.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:59:26.211 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:59:26.271 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:26.389 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:26 smithi179 ceph-mon[43975]: pgmap v2142: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 759 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 674 op/s; 1701/36411 objects degraded (4.672%); 6.6 MiB/s, 105 objects/s recovering 2024-02-09T17:59:26.389 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:26 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/772271' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-02-09T17:59:26.389 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:26 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-02-09T17:59:26.389 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:26 smithi179 ceph-mon[43975]: Client client.admin marked osd.4 out, while it was still marked up 2024-02-09T17:59:26.409 INFO:teuthology.orchestra.run.smithi099.stderr:marked out osd.4. 2024-02-09T17:59:26.437 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:26 smithi099 ceph-mon[39420]: pgmap v2142: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 759 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 674 op/s; 1701/36411 objects degraded (4.672%); 6.6 MiB/s, 105 objects/s recovering 2024-02-09T17:59:26.437 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:26 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/772271' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-02-09T17:59:26.437 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:26 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-02-09T17:59:26.437 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:26 smithi099 ceph-mon[39420]: Client client.admin marked osd.4 out, while it was still marked up 2024-02-09T17:59:26.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:26 smithi162 ceph-mon[44304]: pgmap v2142: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 759 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 674 op/s; 1701/36411 objects degraded (4.672%); 6.6 MiB/s, 105 objects/s recovering 2024-02-09T17:59:26.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:26 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/772271' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-02-09T17:59:26.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:26 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-02-09T17:59:26.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:26 smithi162 ceph-mon[44304]: Client client.admin marked osd.4 out, while it was still marked up 2024-02-09T17:59:27.052 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:59:27.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:27.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:59:27.137 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:59:27.137 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-02-09T17:59:27.137 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-02-09T17:59:27.137 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-02-09T17:59:27.137 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:59:27.156 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:59:27.184 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:27.238 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:59:27.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:27.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:59:27.343 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:59:27.371 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:27.390 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:59:27.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:27.393 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:59:27.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:27 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-02-09T17:59:27.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:27 smithi179 ceph-mon[43975]: osdmap e863: 12 total, 12 up, 11 in 2024-02-09T17:59:27.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:27 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 1701/36411 objects degraded (4.672%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:59:27.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:27 smithi179 ceph-mon[43975]: osdmap e864: 12 total, 12 up, 11 in 2024-02-09T17:59:27.495 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:59:27.521 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:27.542 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:59:27.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:27.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:59:27.641 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:59:27.648 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:59:27.666 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:27 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-02-09T17:59:27.666 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:27 smithi099 ceph-mon[39420]: osdmap e863: 12 total, 12 up, 11 in 2024-02-09T17:59:27.666 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:27 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 1701/36411 objects degraded (4.672%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:59:27.667 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:27 smithi099 ceph-mon[39420]: osdmap e864: 12 total, 12 up, 11 in 2024-02-09T17:59:27.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:27 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-02-09T17:59:27.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:27 smithi162 ceph-mon[44304]: osdmap e863: 12 total, 12 up, 11 in 2024-02-09T17:59:27.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:27 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 1701/36411 objects degraded (4.672%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:59:27.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:27 smithi162 ceph-mon[44304]: osdmap e864: 12 total, 12 up, 11 in 2024-02-09T17:59:27.694 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:27.772 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:59:27.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:27.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:59:27.882 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:59:27.910 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:28.160 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:59:28.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:28 smithi179 ceph-mon[43975]: pgmap v2145: 47 pgs: 1 unknown, 1 peering, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 714 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 341 B/s rd, 11 KiB/s wr, 664 op/s; 1701/34266 objects degraded (4.964%); 3.9 MiB/s, 62 objects/s recovering 2024-02-09T17:59:28.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:28 smithi179 ceph-mon[43975]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-02-09T17:59:28.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:28 smithi179 ceph-mon[43975]: osdmap e865: 12 total, 12 up, 11 in 2024-02-09T17:59:28.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:28 smithi179 ceph-mon[43975]: 8.5 starting backfill to osd.11 from (0'0,0'0] MAX to 862'14824 2024-02-09T17:59:28.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:28 smithi179 ceph-mon[43975]: 8.2 starting backfill to osd.9 from (0'0,0'0] MAX to 862'14837 2024-02-09T17:59:28.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:28 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:59:28.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:28 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:59:28.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:28 smithi162 ceph-mon[44304]: pgmap v2145: 47 pgs: 1 unknown, 1 peering, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 714 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 341 B/s rd, 11 KiB/s wr, 664 op/s; 1701/34266 objects degraded (4.964%); 3.9 MiB/s, 62 objects/s recovering 2024-02-09T17:59:28.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:28 smithi162 ceph-mon[44304]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-02-09T17:59:28.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:28 smithi162 ceph-mon[44304]: osdmap e865: 12 total, 12 up, 11 in 2024-02-09T17:59:28.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:28 smithi162 ceph-mon[44304]: 8.5 starting backfill to osd.11 from (0'0,0'0] MAX to 862'14824 2024-02-09T17:59:28.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:28 smithi162 ceph-mon[44304]: 8.2 starting backfill to osd.9 from (0'0,0'0] MAX to 862'14837 2024-02-09T17:59:28.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:28 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:59:28.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:28 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:59:28.683 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:28 smithi099 ceph-mon[39420]: pgmap v2145: 47 pgs: 1 unknown, 1 peering, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 714 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 341 B/s rd, 11 KiB/s wr, 664 op/s; 1701/34266 objects degraded (4.964%); 3.9 MiB/s, 62 objects/s recovering 2024-02-09T17:59:28.683 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:28 smithi099 ceph-mon[39420]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-02-09T17:59:28.683 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:28 smithi099 ceph-mon[39420]: osdmap e865: 12 total, 12 up, 11 in 2024-02-09T17:59:28.683 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:28 smithi099 ceph-mon[39420]: 8.5 starting backfill to osd.11 from (0'0,0'0] MAX to 862'14824 2024-02-09T17:59:28.683 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:28 smithi099 ceph-mon[39420]: 8.2 starting backfill to osd.9 from (0'0,0'0] MAX to 862'14837 2024-02-09T17:59:28.683 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:28 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:59:28.684 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:28 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:59:28.859 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:59:28.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:28.865 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:59:28.966 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:59:28.994 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:29.024 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:59:29.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:29.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:59:29.131 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:59:29.178 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:29.364 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:59:29.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:29.367 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:59:29.388 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.10 dump_historic_ops 2024-02-09T17:59:29.403 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:29 smithi099 ceph-mon[39420]: 8.e starting backfill to osd.7 from (0'0,0'0] MAX to 862'14791 2024-02-09T17:59:29.403 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:29 smithi099 ceph-mon[39420]: 8.b starting backfill to osd.11 from (0'0,0'0] MAX to 862'14641 2024-02-09T17:59:29.403 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:29 smithi099 ceph-mon[39420]: osdmap e866: 12 total, 12 up, 11 in 2024-02-09T17:59:29.403 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:29 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/371666849' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:59:29.403 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:29 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:59:29.469 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:59:29.485 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:29 smithi179 ceph-mon[43975]: 8.e starting backfill to osd.7 from (0'0,0'0] MAX to 862'14791 2024-02-09T17:59:29.485 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:29 smithi179 ceph-mon[43975]: 8.b starting backfill to osd.11 from (0'0,0'0] MAX to 862'14641 2024-02-09T17:59:29.485 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:29 smithi179 ceph-mon[43975]: osdmap e866: 12 total, 12 up, 11 in 2024-02-09T17:59:29.485 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:29 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/371666849' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:59:29.485 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:29 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:59:29.515 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:29.560 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:59:29.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:29.570 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:59:29.671 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:59:29.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:29 smithi162 ceph-mon[44304]: 8.e starting backfill to osd.7 from (0'0,0'0] MAX to 862'14791 2024-02-09T17:59:29.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:29 smithi162 ceph-mon[44304]: 8.b starting backfill to osd.11 from (0'0,0'0] MAX to 862'14641 2024-02-09T17:59:29.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:29 smithi162 ceph-mon[44304]: osdmap e866: 12 total, 12 up, 11 in 2024-02-09T17:59:29.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:29 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/371666849' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:59:29.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:29 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T17:59:29.714 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:29.871 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:59:29.993 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:59:29.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:29.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:59:30.078 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is unset 2024-02-09T17:59:30.099 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:59:30.171 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:30.193 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:59:30.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:30.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:59:30.298 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:59:30.327 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:30.424 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:59:30.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:30 smithi162 ceph-mon[44304]: pgmap v2148: 47 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 remapped+peering, 1 unknown, 1 peering, 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 36 active+clean; 644 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 511 B/s rd, 17 KiB/s wr, 462 op/s; 1495/30885 objects degraded (4.841%); 1672/30885 objects misplaced (5.414%); 6.2 MiB/s, 99 objects/s recovering 2024-02-09T17:59:30.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:30 smithi162 ceph-mon[44304]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:59:30.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:30 smithi162 ceph-mon[44304]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:59:30.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:30 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:59:30.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:30 smithi162 ceph-mon[44304]: osdmap e867: 12 total, 12 up, 11 in 2024-02-09T17:59:30.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:30.428 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:59:30.470 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:30 smithi179 ceph-mon[43975]: pgmap v2148: 47 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 remapped+peering, 1 unknown, 1 peering, 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 36 active+clean; 644 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 511 B/s rd, 17 KiB/s wr, 462 op/s; 1495/30885 objects degraded (4.841%); 1672/30885 objects misplaced (5.414%); 6.2 MiB/s, 99 objects/s recovering 2024-02-09T17:59:30.471 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:30 smithi179 ceph-mon[43975]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:59:30.471 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:30 smithi179 ceph-mon[43975]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:59:30.471 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:30 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:59:30.471 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:30 smithi179 ceph-mon[43975]: osdmap e867: 12 total, 12 up, 11 in 2024-02-09T17:59:30.529 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:59:30.556 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:30.587 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:59:30.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:30.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:59:30.692 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:59:30.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:30 smithi099 ceph-mon[39420]: pgmap v2148: 47 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 remapped+peering, 1 unknown, 1 peering, 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 36 active+clean; 644 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 511 B/s rd, 17 KiB/s wr, 462 op/s; 1495/30885 objects degraded (4.841%); 1672/30885 objects misplaced (5.414%); 6.2 MiB/s, 99 objects/s recovering 2024-02-09T17:59:30.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:30 smithi099 ceph-mon[39420]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:59:30.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:30 smithi099 ceph-mon[39420]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:59:30.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:30 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T17:59:30.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:30 smithi099 ceph-mon[39420]: osdmap e867: 12 total, 12 up, 11 in 2024-02-09T17:59:30.759 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:30.804 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:59:30.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:30.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:59:30.900 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:59:30.911 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:59:30.929 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:59:30.942 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:30.957 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:59:30.966 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:59:30.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:30.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:59:30.989 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:59:31.015 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:59:31.045 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:59:31.072 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:59:31.072 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:59:31.125 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:31.300 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:59:31.326 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:59:31.330 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:59:31.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:31.341 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:59:31.357 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:59:31.387 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:59:31.425 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:59:31.432 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:59:31.432 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:59:31.433 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:59:31.433 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:59:31.433 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:59:31.433 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:59:31.433 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:59:31.433 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:59:31.433 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:59:31.433 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:59:31.433 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:59:31.433 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:59:31.442 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:59:31.468 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:31.491 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:59:31.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:31.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:59:31.596 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:59:31.623 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:31.637 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:31 smithi179 ceph-mon[43975]: 1.f scrub starts 2024-02-09T17:59:31.637 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:31 smithi179 ceph-mon[43975]: 1.f scrub ok 2024-02-09T17:59:31.637 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:31 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1916598866' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:59:31.637 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:31 smithi179 ceph-mon[43975]: 1.1a scrub starts 2024-02-09T17:59:31.638 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:31 smithi179 ceph-mon[43975]: 1.1a scrub ok 2024-02-09T17:59:31.638 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:31 smithi179 ceph-mon[43975]: osdmap e868: 12 total, 12 up, 11 in 2024-02-09T17:59:31.649 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:59:31.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:31.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:59:31.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:31 smithi162 ceph-mon[44304]: 1.f scrub starts 2024-02-09T17:59:31.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:31 smithi162 ceph-mon[44304]: 1.f scrub ok 2024-02-09T17:59:31.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:31 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1916598866' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:59:31.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:31 smithi162 ceph-mon[44304]: 1.1a scrub starts 2024-02-09T17:59:31.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:31 smithi162 ceph-mon[44304]: 1.1a scrub ok 2024-02-09T17:59:31.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:31 smithi162 ceph-mon[44304]: osdmap e868: 12 total, 12 up, 11 in 2024-02-09T17:59:31.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:31 smithi099 ceph-mon[39420]: 1.f scrub starts 2024-02-09T17:59:31.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:31 smithi099 ceph-mon[39420]: 1.f scrub ok 2024-02-09T17:59:31.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:31 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1916598866' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:59:31.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:31 smithi099 ceph-mon[39420]: 1.1a scrub starts 2024-02-09T17:59:31.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:31 smithi099 ceph-mon[39420]: 1.1a scrub ok 2024-02-09T17:59:31.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:31 smithi099 ceph-mon[39420]: osdmap e868: 12 total, 12 up, 11 in 2024-02-09T17:59:31.754 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:59:31.780 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:31.801 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:59:31.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:31.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:59:31.906 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:59:31.935 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:31.958 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:59:31.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:31.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:59:32.062 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:59:32.095 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:32.126 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:59:32.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:32.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:59:32.232 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:59:32.260 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:32.291 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:59:32.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:32.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:59:32.395 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:59:32.472 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:32.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:32 smithi179 ceph-mon[43975]: 1.e scrub starts 2024-02-09T17:59:32.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:32 smithi179 ceph-mon[43975]: 1.e scrub ok 2024-02-09T17:59:32.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:32 smithi179 ceph-mon[43975]: 1.1e scrub starts 2024-02-09T17:59:32.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:32 smithi179 ceph-mon[43975]: 1.1e scrub ok 2024-02-09T17:59:32.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:32 smithi179 ceph-mon[43975]: pgmap v2151: 47 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 remapped+peering, 1 unknown, 3 peering, 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 611 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 415 op/s; 1108/29322 objects degraded (3.779%); 1672/29322 objects misplaced (5.702%); 12 MiB/s, 195 objects/s recovering 2024-02-09T17:59:32.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:32 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 1495/30885 objects degraded (4.841%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:59:32.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:32 smithi179 ceph-mon[43975]: osdmap e869: 12 total, 12 up, 11 in 2024-02-09T17:59:32.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:32 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:59:32.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:32 smithi162 ceph-mon[44304]: 1.e scrub starts 2024-02-09T17:59:32.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:32 smithi162 ceph-mon[44304]: 1.e scrub ok 2024-02-09T17:59:32.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:32 smithi162 ceph-mon[44304]: 1.1e scrub starts 2024-02-09T17:59:32.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:32 smithi162 ceph-mon[44304]: 1.1e scrub ok 2024-02-09T17:59:32.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:32 smithi162 ceph-mon[44304]: pgmap v2151: 47 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 remapped+peering, 1 unknown, 3 peering, 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 611 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 415 op/s; 1108/29322 objects degraded (3.779%); 1672/29322 objects misplaced (5.702%); 12 MiB/s, 195 objects/s recovering 2024-02-09T17:59:32.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:32 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 1495/30885 objects degraded (4.841%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:59:32.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:32 smithi162 ceph-mon[44304]: osdmap e869: 12 total, 12 up, 11 in 2024-02-09T17:59:32.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:32 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:59:32.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:32 smithi099 ceph-mon[39420]: 1.e scrub starts 2024-02-09T17:59:32.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:32 smithi099 ceph-mon[39420]: 1.e scrub ok 2024-02-09T17:59:32.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:32 smithi099 ceph-mon[39420]: 1.1e scrub starts 2024-02-09T17:59:32.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:32 smithi099 ceph-mon[39420]: 1.1e scrub ok 2024-02-09T17:59:32.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:32 smithi099 ceph-mon[39420]: pgmap v2151: 47 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 remapped+peering, 1 unknown, 3 peering, 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 34 active+clean; 611 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 415 op/s; 1108/29322 objects degraded (3.779%); 1672/29322 objects misplaced (5.702%); 12 MiB/s, 195 objects/s recovering 2024-02-09T17:59:32.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:32 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 1495/30885 objects degraded (4.841%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:59:32.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:32 smithi099 ceph-mon[39420]: osdmap e869: 12 total, 12 up, 11 in 2024-02-09T17:59:32.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:32 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:59:32.872 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T17:59:32.872 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:59:33.108 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 5, 6, 7, 8, 9, 10, 11, 0, 2] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 7, 9, 10, 11, 8, 3] 2024-02-09T17:59:33.109 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 1 1 2024-02-09T17:59:33.417 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:33 smithi099 ceph-mon[39420]: 8.e scrub starts 2024-02-09T17:59:33.417 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:33 smithi099 ceph-mon[39420]: 8.e scrub ok 2024-02-09T17:59:33.468 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-09T17:59:33.514 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:59:33.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:33 smithi162 ceph-mon[44304]: 8.e scrub starts 2024-02-09T17:59:33.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:33 smithi162 ceph-mon[44304]: 8.e scrub ok 2024-02-09T17:59:33.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:33 smithi179 ceph-mon[43975]: 8.e scrub starts 2024-02-09T17:59:33.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:33 smithi179 ceph-mon[43975]: 8.e scrub ok 2024-02-09T17:59:34.327 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:59:34.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:34 smithi179 ceph-mon[43975]: pgmap v2153: 47 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 2 peering, 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 37 active+clean; 554 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 726 B/s rd, 16 KiB/s wr, 341 op/s; 850/26565 objects degraded (3.200%); 2184/26565 objects misplaced (8.221%); 9.5 MiB/s, 152 objects/s recovering 2024-02-09T17:59:34.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:34 smithi179 ceph-mon[43975]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-02-09T17:59:34.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:34 smithi179 ceph-mon[43975]: 1.16 scrub starts 2024-02-09T17:59:34.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:34 smithi179 ceph-mon[43975]: 1.16 scrub ok 2024-02-09T17:59:34.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:34 smithi162 ceph-mon[44304]: pgmap v2153: 47 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 2 peering, 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 37 active+clean; 554 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 726 B/s rd, 16 KiB/s wr, 341 op/s; 850/26565 objects degraded (3.200%); 2184/26565 objects misplaced (8.221%); 9.5 MiB/s, 152 objects/s recovering 2024-02-09T17:59:34.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:34 smithi162 ceph-mon[44304]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-02-09T17:59:34.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:34 smithi162 ceph-mon[44304]: 1.16 scrub starts 2024-02-09T17:59:34.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:34 smithi162 ceph-mon[44304]: 1.16 scrub ok 2024-02-09T17:59:34.748 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:34 smithi099 ceph-mon[39420]: pgmap v2153: 47 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 2 peering, 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 37 active+clean; 554 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 726 B/s rd, 16 KiB/s wr, 341 op/s; 850/26565 objects degraded (3.200%); 2184/26565 objects misplaced (8.221%); 9.5 MiB/s, 152 objects/s recovering 2024-02-09T17:59:34.748 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:34 smithi099 ceph-mon[39420]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-02-09T17:59:34.748 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:34 smithi099 ceph-mon[39420]: 1.16 scrub starts 2024-02-09T17:59:34.748 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:34 smithi099 ceph-mon[39420]: 1.16 scrub ok 2024-02-09T17:59:34.986 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:59:34.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:34.995 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:59:35.096 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:59:35.120 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:59:35.125 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:35.420 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:35 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:59:35.421 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:35 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:59:35.421 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:35 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:59:35.440 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:59:35.487 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:59:35.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:35.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:59:35.616 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:59:35.645 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:35.672 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:59:35.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:35.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:59:35.679 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:35 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:59:35.679 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:35 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:59:35.680 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:35 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:59:35.681 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:35 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:59:35.681 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:35 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:59:35.681 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:35 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:59:35.779 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:59:35.830 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:36.440 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:36 smithi099 ceph-mon[39420]: 1.10 scrub starts 2024-02-09T17:59:36.441 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:36 smithi099 ceph-mon[39420]: 1.10 scrub ok 2024-02-09T17:59:36.441 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:36 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:59:36.441 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:36 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:59:36.441 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:36 smithi099 ceph-mon[39420]: pgmap v2154: 47 pgs: 3 active+remapped+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 481 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 432 op/s; 472/23085 objects degraded (2.045%); 1461/23085 objects misplaced (6.329%); 15 MiB/s, 242 objects/s recovering 2024-02-09T17:59:36.441 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:36 smithi099 ceph-mon[39420]: osdmap e870: 12 total, 12 up, 11 in 2024-02-09T17:59:36.580 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.11 dump_ops_in_flight 2024-02-09T17:59:36.606 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:59:36.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:36.612 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:59:36.713 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:59:36.773 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:36 smithi179 ceph-mon[43975]: 1.10 scrub starts 2024-02-09T17:59:36.773 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:36 smithi179 ceph-mon[43975]: 1.10 scrub ok 2024-02-09T17:59:36.774 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:36 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:59:36.774 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:36 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:59:36.774 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:36 smithi179 ceph-mon[43975]: pgmap v2154: 47 pgs: 3 active+remapped+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 481 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 432 op/s; 472/23085 objects degraded (2.045%); 1461/23085 objects misplaced (6.329%); 15 MiB/s, 242 objects/s recovering 2024-02-09T17:59:36.774 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:36 smithi179 ceph-mon[43975]: osdmap e870: 12 total, 12 up, 11 in 2024-02-09T17:59:36.779 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:36.835 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:59:36.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:36.841 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:59:36.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:36 smithi162 ceph-mon[44304]: 1.10 scrub starts 2024-02-09T17:59:36.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:36 smithi162 ceph-mon[44304]: 1.10 scrub ok 2024-02-09T17:59:36.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:36 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:59:36.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:36 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:59:36.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:36 smithi162 ceph-mon[44304]: pgmap v2154: 47 pgs: 3 active+remapped+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 481 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 432 op/s; 472/23085 objects degraded (2.045%); 1461/23085 objects misplaced (6.329%); 15 MiB/s, 242 objects/s recovering 2024-02-09T17:59:36.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:36 smithi162 ceph-mon[44304]: osdmap e870: 12 total, 12 up, 11 in 2024-02-09T17:59:36.942 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:59:36.996 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:37.007 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.1 to 1 (10000) 2024-02-09T17:59:37.068 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:59:37.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:37.074 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:59:37.089 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:59:37.175 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:59:37.204 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:37.231 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:59:37.234 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:37.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:59:37.335 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:59:37.414 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:37.530 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:59:37.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:37.538 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:59:37.638 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:59:37.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:37 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2233424399' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-09T17:59:37.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:37 smithi179 ceph-mon[43975]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:59:37.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:37 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 472/23085 objects degraded (2.045%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:59:37.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:37 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2233424399' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-09T17:59:37.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:37 smithi179 ceph-mon[43975]: osdmap e871: 12 total, 12 up, 11 in 2024-02-09T17:59:37.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:37 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2233424399' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-09T17:59:37.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:37 smithi162 ceph-mon[44304]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:59:37.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:37 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 472/23085 objects degraded (2.045%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:59:37.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:37 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2233424399' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-09T17:59:37.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:37 smithi162 ceph-mon[44304]: osdmap e871: 12 total, 12 up, 11 in 2024-02-09T17:59:37.695 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:37.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:37 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2233424399' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-09T17:59:37.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:37 smithi099 ceph-mon[39420]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T17:59:37.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:37 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 472/23085 objects degraded (2.045%), 3 pgs degraded (PG_DEGRADED) 2024-02-09T17:59:37.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:37 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2233424399' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-09T17:59:37.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:37 smithi099 ceph-mon[39420]: osdmap e871: 12 total, 12 up, 11 in 2024-02-09T17:59:38.537 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 3 1 2024-02-09T17:59:38.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:38 smithi179 ceph-mon[43975]: 1.0 scrub starts 2024-02-09T17:59:38.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:38 smithi179 ceph-mon[43975]: 1.0 scrub ok 2024-02-09T17:59:38.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:38 smithi179 ceph-mon[43975]: pgmap v2157: 47 pgs: 3 active+remapped+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 432 MiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 475 op/s; 472/20724 objects degraded (2.278%); 1461/20724 objects misplaced (7.050%); 11 MiB/s, 177 objects/s recovering 2024-02-09T17:59:38.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:38 smithi179 ceph-mon[43975]: 1.19 scrub starts 2024-02-09T17:59:38.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:38 smithi179 ceph-mon[43975]: 1.19 scrub ok 2024-02-09T17:59:38.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:38 smithi099 ceph-mon[39420]: 1.0 scrub starts 2024-02-09T17:59:38.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:38 smithi099 ceph-mon[39420]: 1.0 scrub ok 2024-02-09T17:59:38.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:38 smithi099 ceph-mon[39420]: pgmap v2157: 47 pgs: 3 active+remapped+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 432 MiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 475 op/s; 472/20724 objects degraded (2.278%); 1461/20724 objects misplaced (7.050%); 11 MiB/s, 177 objects/s recovering 2024-02-09T17:59:38.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:38 smithi099 ceph-mon[39420]: 1.19 scrub starts 2024-02-09T17:59:38.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:38 smithi099 ceph-mon[39420]: 1.19 scrub ok 2024-02-09T17:59:38.860 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:59:38.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:38.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:59:38.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:38 smithi162 ceph-mon[44304]: 1.0 scrub starts 2024-02-09T17:59:38.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:38 smithi162 ceph-mon[44304]: 1.0 scrub ok 2024-02-09T17:59:38.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:38 smithi162 ceph-mon[44304]: pgmap v2157: 47 pgs: 3 active+remapped+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 432 MiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 475 op/s; 472/20724 objects degraded (2.278%); 1461/20724 objects misplaced (7.050%); 11 MiB/s, 177 objects/s recovering 2024-02-09T17:59:38.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:38 smithi162 ceph-mon[44304]: 1.19 scrub starts 2024-02-09T17:59:38.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:38 smithi162 ceph-mon[44304]: 1.19 scrub ok 2024-02-09T17:59:38.968 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:59:38.999 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:39.024 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:59:39.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:39.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:59:39.129 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:59:39.192 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:39.231 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:59:39.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:39.239 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:59:39.340 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:59:39.414 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:39.562 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:59:39.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:39.566 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:59:39.667 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:59:39.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:39 smithi179 ceph-mon[43975]: 8.f scrub starts 2024-02-09T17:59:39.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:39 smithi179 ceph-mon[43975]: 8.f scrub ok 2024-02-09T17:59:39.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:39 smithi179 ceph-mon[43975]: 1.1 scrub starts 2024-02-09T17:59:39.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:39 smithi179 ceph-mon[43975]: 1.1 scrub ok 2024-02-09T17:59:39.727 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:39.732 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:39 smithi099 ceph-mon[39420]: 8.f scrub starts 2024-02-09T17:59:39.732 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:39 smithi099 ceph-mon[39420]: 8.f scrub ok 2024-02-09T17:59:39.732 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:39 smithi099 ceph-mon[39420]: 1.1 scrub starts 2024-02-09T17:59:39.732 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:39 smithi099 ceph-mon[39420]: 1.1 scrub ok 2024-02-09T17:59:39.762 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:59:39.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:39.767 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:59:39.867 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T17:59:39.894 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:39 smithi162 ceph-mon[44304]: 8.f scrub starts 2024-02-09T17:59:39.894 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:39 smithi162 ceph-mon[44304]: 8.f scrub ok 2024-02-09T17:59:39.894 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:39 smithi162 ceph-mon[44304]: 1.1 scrub starts 2024-02-09T17:59:39.894 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:39 smithi162 ceph-mon[44304]: 1.1 scrub ok 2024-02-09T17:59:39.896 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:39.918 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T17:59:39.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:39.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T17:59:40.023 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:59:40.077 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:40.113 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:59:40.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:40.126 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:59:40.167 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T17:59:40.167 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T17:59:40.227 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:59:40.281 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:40.320 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:59:40.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:40.323 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:59:40.424 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:59:40.453 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:40.478 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:59:40.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:40.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:59:40.484 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is unset 2024-02-09T17:59:40.504 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:59:40.566 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:59:40.582 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:59:40.599 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:59:40.611 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:40.635 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:59:40.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:40.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:59:40.640 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:59:40.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:40 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:59:40.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:40 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:59:40.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:40 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:59:40.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:40 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:59:40.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:40 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1889796657' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:59:40.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:40 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:59:40.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:40 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:59:40.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:40 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:59:40.680 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:59:40.714 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:59:40.739 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:59:40.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:40 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:59:40.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:40 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:59:40.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:40 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:59:40.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:40 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:59:40.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:40 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1889796657' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:59:40.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:40 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:59:40.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:40 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:59:40.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:40 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:59:40.745 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:59:40.775 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:59:40.800 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:59:40.804 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:40.839 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:59:40.868 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:59:40.908 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:59:40.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:40 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T17:59:40.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:40 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T17:59:40.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:40 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T17:59:40.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:40 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T17:59:40.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:40 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1889796657' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T17:59:40.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:40 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T17:59:40.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:40 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T17:59:40.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:40 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T17:59:40.948 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:59:40.980 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:59:40.980 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:59:40.980 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:59:40.981 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:59:40.981 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:59:40.981 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:59:40.981 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:59:40.981 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:59:40.981 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:59:40.981 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:59:40.981 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:59:40.981 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:59:41.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:41 smithi099 ceph-mon[39420]: 1.d scrub starts 2024-02-09T17:59:41.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:41 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:59:41.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:41 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:59:41.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:41 smithi099 ceph-mon[39420]: 1.d scrub ok 2024-02-09T17:59:41.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:41 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T17:59:41.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:41 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T17:59:41.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:41 smithi099 ceph-mon[39420]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:59:41.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:41 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1889796657' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:59:41.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:41 smithi099 ceph-mon[39420]: osdmap e872: 12 total, 12 up, 11 in 2024-02-09T17:59:41.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:41 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/958440668' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:59:41.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:41 smithi099 ceph-mon[39420]: 1.6 scrub starts 2024-02-09T17:59:41.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:41 smithi099 ceph-mon[39420]: 1.6 scrub ok 2024-02-09T17:59:41.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:41 smithi099 ceph-mon[39420]: 1.3 scrub starts 2024-02-09T17:59:41.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:41 smithi179 ceph-mon[43975]: 1.d scrub starts 2024-02-09T17:59:41.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:41 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:59:41.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:41 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:59:41.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:41 smithi179 ceph-mon[43975]: 1.d scrub ok 2024-02-09T17:59:41.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:41 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T17:59:41.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:41 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T17:59:41.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:41 smithi179 ceph-mon[43975]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:59:41.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:41 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1889796657' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:59:41.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:41 smithi179 ceph-mon[43975]: osdmap e872: 12 total, 12 up, 11 in 2024-02-09T17:59:41.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:41 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/958440668' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:59:41.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:41 smithi179 ceph-mon[43975]: 1.6 scrub starts 2024-02-09T17:59:41.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:41 smithi179 ceph-mon[43975]: 1.6 scrub ok 2024-02-09T17:59:41.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:41 smithi179 ceph-mon[43975]: 1.3 scrub starts 2024-02-09T17:59:41.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:41 smithi162 ceph-mon[44304]: 1.d scrub starts 2024-02-09T17:59:41.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:41 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:59:41.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:41 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:59:41.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:41 smithi162 ceph-mon[44304]: 1.d scrub ok 2024-02-09T17:59:41.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:41 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T17:59:41.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:41 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T17:59:41.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:41 smithi162 ceph-mon[44304]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T17:59:41.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:41 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1889796657' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T17:59:41.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:41 smithi162 ceph-mon[44304]: osdmap e872: 12 total, 12 up, 11 in 2024-02-09T17:59:41.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:41 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/958440668' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:59:41.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:41 smithi162 ceph-mon[44304]: 1.6 scrub starts 2024-02-09T17:59:41.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:41 smithi162 ceph-mon[44304]: 1.6 scrub ok 2024-02-09T17:59:41.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:41 smithi162 ceph-mon[44304]: 1.3 scrub starts 2024-02-09T17:59:42.076 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:59:42.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:42.082 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:59:42.183 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:59:42.232 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:42.654 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.11 dump_blocked_ops 2024-02-09T17:59:42.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:42 smithi099 ceph-mon[39420]: 1.3 scrub ok 2024-02-09T17:59:42.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:42 smithi099 ceph-mon[39420]: pgmap v2160: 47 pgs: 2 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 41 active+clean; 289 MiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 513 op/s; 119/13842 objects degraded (0.860%); 856/13842 objects misplaced (6.184%); 3.7 MiB/s, 75 objects/s recovering 2024-02-09T17:59:42.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:42 smithi099 ceph-mon[39420]: Health check update: Degraded data redundancy: 119/13842 objects degraded (0.860%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:59:42.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:42 smithi099 ceph-mon[39420]: osdmap e873: 12 total, 12 up, 11 in 2024-02-09T17:59:42.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:42 smithi179 ceph-mon[43975]: 1.3 scrub ok 2024-02-09T17:59:42.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:42 smithi179 ceph-mon[43975]: pgmap v2160: 47 pgs: 2 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 41 active+clean; 289 MiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 513 op/s; 119/13842 objects degraded (0.860%); 856/13842 objects misplaced (6.184%); 3.7 MiB/s, 75 objects/s recovering 2024-02-09T17:59:42.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:42 smithi179 ceph-mon[43975]: Health check update: Degraded data redundancy: 119/13842 objects degraded (0.860%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:59:42.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:42 smithi179 ceph-mon[43975]: osdmap e873: 12 total, 12 up, 11 in 2024-02-09T17:59:42.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:42 smithi162 ceph-mon[44304]: 1.3 scrub ok 2024-02-09T17:59:42.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:42 smithi162 ceph-mon[44304]: pgmap v2160: 47 pgs: 2 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 41 active+clean; 289 MiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 513 op/s; 119/13842 objects degraded (0.860%); 856/13842 objects misplaced (6.184%); 3.7 MiB/s, 75 objects/s recovering 2024-02-09T17:59:42.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:42 smithi162 ceph-mon[44304]: Health check update: Degraded data redundancy: 119/13842 objects degraded (0.860%), 1 pg degraded (PG_DEGRADED) 2024-02-09T17:59:42.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:42 smithi162 ceph-mon[44304]: osdmap e873: 12 total, 12 up, 11 in 2024-02-09T17:59:43.061 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:59:44.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:44 smithi179 ceph-mon[43975]: pgmap v2162: 47 pgs: 2 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 41 active+clean; 193 MiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 737 op/s; 119/9264 objects degraded (1.285%); 673/9264 objects misplaced (7.265%); 3.5 MiB/s, 99 objects/s recovering 2024-02-09T17:59:44.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:44 smithi162 ceph-mon[44304]: pgmap v2162: 47 pgs: 2 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 41 active+clean; 193 MiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 737 op/s; 119/9264 objects degraded (1.285%); 673/9264 objects misplaced (7.265%); 3.5 MiB/s, 99 objects/s recovering 2024-02-09T17:59:44.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:44 smithi099 ceph-mon[39420]: pgmap v2162: 47 pgs: 2 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 41 active+clean; 193 MiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 737 op/s; 119/9264 objects degraded (1.285%); 673/9264 objects misplaced (7.265%); 3.5 MiB/s, 99 objects/s recovering 2024-02-09T17:59:45.102 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:59:45.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:45.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:59:45.208 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:59:45.243 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:45.270 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:59:45.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:45.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:59:45.379 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:59:45.407 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:45.427 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:59:45.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:45.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:59:45.532 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T17:59:45.567 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:45.741 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Removed 123069 objects 2024-02-09T17:59:45.742 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Clean up completed and total clean up time :169.118 2024-02-09T17:59:45.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:45 smithi179 ceph-mon[43975]: 1.1d scrub starts 2024-02-09T17:59:45.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:45 smithi179 ceph-mon[43975]: 1.1d scrub ok 2024-02-09T17:59:45.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:45 smithi162 ceph-mon[44304]: 1.1d scrub starts 2024-02-09T17:59:45.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:45 smithi162 ceph-mon[44304]: 1.1d scrub ok 2024-02-09T17:59:45.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:45 smithi099 ceph-mon[39420]: 1.1d scrub starts 2024-02-09T17:59:45.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:45 smithi099 ceph-mon[39420]: 1.1d scrub ok 2024-02-09T17:59:46.030 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T17:59:46.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:46.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T17:59:46.134 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:59:46.161 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:46.186 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:59:46.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:46.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:59:46.291 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:59:46.319 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:46.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:46 smithi099 ceph-mon[39420]: pgmap v2163: 47 pgs: 1 active+clean+scrubbing, 3 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 40 active+clean; 96 MiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 671 op/s; 305/4611 objects misplaced (6.615%); 3.7 MiB/s, 113 objects/s recovering 2024-02-09T17:59:46.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:46 smithi099 ceph-mon[39420]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 119/9264 objects degraded (1.285%), 1 pg degraded) 2024-02-09T17:59:46.769 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T17:59:46.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:46.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T17:59:46.873 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:59:46.899 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:46 smithi179 ceph-mon[43975]: pgmap v2163: 47 pgs: 1 active+clean+scrubbing, 3 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 40 active+clean; 96 MiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 671 op/s; 305/4611 objects misplaced (6.615%); 3.7 MiB/s, 113 objects/s recovering 2024-02-09T17:59:46.899 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:46 smithi179 ceph-mon[43975]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 119/9264 objects degraded (1.285%), 1 pg degraded) 2024-02-09T17:59:46.901 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:46.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:46 smithi162 ceph-mon[44304]: pgmap v2163: 47 pgs: 1 active+clean+scrubbing, 3 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 40 active+clean; 96 MiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 671 op/s; 305/4611 objects misplaced (6.615%); 3.7 MiB/s, 113 objects/s recovering 2024-02-09T17:59:46.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:46 smithi162 ceph-mon[44304]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 119/9264 objects degraded (1.285%), 1 pg degraded) 2024-02-09T17:59:47.120 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T17:59:47.405 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:59:47.580 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:59:47.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:47.584 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:59:47.684 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:59:47.713 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:47.733 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:59:47.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:47.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:59:47.837 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T17:59:47.865 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:48.017 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.3 to 1 (10000) 2024-02-09T17:59:48.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:48 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3865620194' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-09T17:59:48.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:48 smithi179 ceph-mon[43975]: 1.b scrub starts 2024-02-09T17:59:48.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:48 smithi179 ceph-mon[43975]: pgmap v2164: 47 pgs: 1 active+clean+scrubbing, 3 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 40 active+clean; 61 MiB data, 8.8 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 552 op/s; 305/2925 objects misplaced (10.427%); 3.1 MiB/s, 92 objects/s recovering 2024-02-09T17:59:48.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:48 smithi179 ceph-mon[43975]: 1.b scrub ok 2024-02-09T17:59:48.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:48 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3865620194' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-09T17:59:48.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:48 smithi099 ceph-mon[39420]: 1.b scrub starts 2024-02-09T17:59:48.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:48 smithi099 ceph-mon[39420]: pgmap v2164: 47 pgs: 1 active+clean+scrubbing, 3 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 40 active+clean; 61 MiB data, 8.8 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 552 op/s; 305/2925 objects misplaced (10.427%); 3.1 MiB/s, 92 objects/s recovering 2024-02-09T17:59:48.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:48 smithi099 ceph-mon[39420]: 1.b scrub ok 2024-02-09T17:59:48.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:48 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3865620194' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-09T17:59:48.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:48 smithi162 ceph-mon[44304]: 1.b scrub starts 2024-02-09T17:59:48.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:48 smithi162 ceph-mon[44304]: pgmap v2164: 47 pgs: 1 active+clean+scrubbing, 3 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 40 active+clean; 61 MiB data, 8.8 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 552 op/s; 305/2925 objects misplaced (10.427%); 3.1 MiB/s, 92 objects/s recovering 2024-02-09T17:59:48.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:48 smithi162 ceph-mon[44304]: 1.b scrub ok 2024-02-09T17:59:48.889 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_6 2024-02-09T17:59:48.889 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd pool rm unique_pool_6 unique_pool_6 --yes-i-really-really-mean-it 2024-02-09T17:59:49.227 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set noscrub 2024-02-09T17:59:49.405 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:49 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3865620194' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-09T17:59:49.405 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:49 smithi179 ceph-mon[43975]: osdmap e874: 12 total, 12 up, 11 in 2024-02-09T17:59:49.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:49 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3865620194' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-09T17:59:49.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:49 smithi162 ceph-mon[44304]: osdmap e874: 12 total, 12 up, 11 in 2024-02-09T17:59:49.437 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T17:59:49.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:49.442 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T17:59:49.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:49 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3865620194' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-09T17:59:49.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:49 smithi099 ceph-mon[39420]: osdmap e874: 12 total, 12 up, 11 in 2024-02-09T17:59:49.543 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:59:49.570 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:49.817 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 5 1 2024-02-09T17:59:49.822 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:59:49.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:49.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:59:49.913 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T17:59:49.914 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-02-09T17:59:49.914 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-02-09T17:59:49.914 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-02-09T17:59:49.914 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T17:59:49.926 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:59:49.953 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:49.973 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:59:49.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:49.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:59:50.077 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:59:50.105 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:50.306 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:59:50.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:50.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:59:50.410 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:59:50.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:50 smithi179 ceph-mon[43975]: 1.14 scrub starts 2024-02-09T17:59:50.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:50 smithi179 ceph-mon[43975]: 1.14 scrub ok 2024-02-09T17:59:50.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:50 smithi179 ceph-mon[43975]: pgmap v2166: 47 pgs: 1 active+clean+scrubbing, 3 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 40 active+clean; 7.8 MiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 561 op/s; 952 KiB/s, 89 objects/s recovering 2024-02-09T17:59:50.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:50 smithi179 ceph-mon[43975]: 1.7 scrub starts 2024-02-09T17:59:50.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:50 smithi162 ceph-mon[44304]: 1.14 scrub starts 2024-02-09T17:59:50.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:50 smithi162 ceph-mon[44304]: 1.14 scrub ok 2024-02-09T17:59:50.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:50 smithi162 ceph-mon[44304]: pgmap v2166: 47 pgs: 1 active+clean+scrubbing, 3 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 40 active+clean; 7.8 MiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 561 op/s; 952 KiB/s, 89 objects/s recovering 2024-02-09T17:59:50.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:50 smithi162 ceph-mon[44304]: 1.7 scrub starts 2024-02-09T17:59:50.437 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:50 smithi099 ceph-mon[39420]: 1.14 scrub starts 2024-02-09T17:59:50.437 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:50 smithi099 ceph-mon[39420]: 1.14 scrub ok 2024-02-09T17:59:50.437 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:50 smithi099 ceph-mon[39420]: pgmap v2166: 47 pgs: 1 active+clean+scrubbing, 3 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 40 active+clean; 7.8 MiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 561 op/s; 952 KiB/s, 89 objects/s recovering 2024-02-09T17:59:50.437 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:50 smithi099 ceph-mon[39420]: 1.7 scrub starts 2024-02-09T17:59:50.439 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:50.460 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:59:50.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:50.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:59:50.565 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:59:50.595 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:50.649 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T17:59:50.661 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:59:50.676 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T17:59:50.698 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:59:50.709 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T17:59:50.712 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:59:50.733 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T17:59:50.754 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T17:59:50.776 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T17:59:50.797 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T17:59:50.818 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T17:59:50.841 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T17:59:50.863 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T17:59:50.884 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T17:59:50.905 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T17:59:50.906 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T17:59:50.906 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T17:59:50.906 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T17:59:50.906 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T17:59:50.906 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T17:59:50.906 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T17:59:50.907 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T17:59:50.907 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T17:59:50.907 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T17:59:50.907 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T17:59:50.907 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T17:59:50.907 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T17:59:51.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:51 smithi099 ceph-mon[39420]: 1.9 deep-scrub starts 2024-02-09T17:59:51.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:51 smithi099 ceph-mon[39420]: 1.9 deep-scrub ok 2024-02-09T17:59:51.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:51 smithi099 ceph-mon[39420]: 1.7 scrub ok 2024-02-09T17:59:51.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:51 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/98934842' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:59:51.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:51 smithi179 ceph-mon[43975]: 1.9 deep-scrub starts 2024-02-09T17:59:51.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:51 smithi179 ceph-mon[43975]: 1.9 deep-scrub ok 2024-02-09T17:59:51.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:51 smithi179 ceph-mon[43975]: 1.7 scrub ok 2024-02-09T17:59:51.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:51 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/98934842' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:59:51.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:51 smithi162 ceph-mon[44304]: 1.9 deep-scrub starts 2024-02-09T17:59:51.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:51 smithi162 ceph-mon[44304]: 1.9 deep-scrub ok 2024-02-09T17:59:51.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:51 smithi162 ceph-mon[44304]: 1.7 scrub ok 2024-02-09T17:59:51.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:51 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/98934842' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T17:59:52.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:52 smithi179 ceph-mon[43975]: pgmap v2167: 47 pgs: 1 active+clean+scrubbing, 3 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 40 active+clean; 417 KiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 498 op/s; 824 KiB/s, 153 objects/s recovering 2024-02-09T17:59:52.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:52 smithi179 ceph-mon[43975]: 8.7 scrub starts 2024-02-09T17:59:52.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:52 smithi179 ceph-mon[43975]: 8.7 scrub ok 2024-02-09T17:59:52.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:52 smithi179 ceph-mon[43975]: osdmap e875: 12 total, 12 up, 11 in 2024-02-09T17:59:52.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:52 smithi162 ceph-mon[44304]: pgmap v2167: 47 pgs: 1 active+clean+scrubbing, 3 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 40 active+clean; 417 KiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 498 op/s; 824 KiB/s, 153 objects/s recovering 2024-02-09T17:59:52.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:52 smithi162 ceph-mon[44304]: 8.7 scrub starts 2024-02-09T17:59:52.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:52 smithi162 ceph-mon[44304]: 8.7 scrub ok 2024-02-09T17:59:52.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:52 smithi162 ceph-mon[44304]: osdmap e875: 12 total, 12 up, 11 in 2024-02-09T17:59:52.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:52 smithi099 ceph-mon[39420]: pgmap v2167: 47 pgs: 1 active+clean+scrubbing, 3 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 40 active+clean; 417 KiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 498 op/s; 824 KiB/s, 153 objects/s recovering 2024-02-09T17:59:52.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:52 smithi099 ceph-mon[39420]: 8.7 scrub starts 2024-02-09T17:59:52.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:52 smithi099 ceph-mon[39420]: 8.7 scrub ok 2024-02-09T17:59:52.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:52 smithi099 ceph-mon[39420]: osdmap e875: 12 total, 12 up, 11 in 2024-02-09T17:59:54.148 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:59:54.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:54.151 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:59:54.177 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.11 dump_historic_ops 2024-02-09T17:59:54.252 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T17:59:54.281 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:54.301 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T17:59:54.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:54.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T17:59:54.342 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T17:59:54.406 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:59:54.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:54 smithi179 ceph-mon[43975]: 1.4 scrub starts 2024-02-09T17:59:54.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:54 smithi179 ceph-mon[43975]: 1.4 scrub ok 2024-02-09T17:59:54.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:54 smithi179 ceph-mon[43975]: pgmap v2169: 47 pgs: 1 active+clean+scrubbing, 3 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 40 active+clean; 417 KiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 191 op/s; 0 B/s, 180 objects/s recovering 2024-02-09T17:59:54.433 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:54.452 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:59:54.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:54.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:59:54.523 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T17:59:54.556 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T17:59:54.583 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:54 smithi099 ceph-mon[39420]: 1.4 scrub starts 2024-02-09T17:59:54.583 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:54 smithi099 ceph-mon[39420]: 1.4 scrub ok 2024-02-09T17:59:54.583 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:54 smithi099 ceph-mon[39420]: pgmap v2169: 47 pgs: 1 active+clean+scrubbing, 3 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 40 active+clean; 417 KiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 191 op/s; 0 B/s, 180 objects/s recovering 2024-02-09T17:59:54.585 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:54.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:54 smithi162 ceph-mon[44304]: 1.4 scrub starts 2024-02-09T17:59:54.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:54 smithi162 ceph-mon[44304]: 1.4 scrub ok 2024-02-09T17:59:54.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:54 smithi162 ceph-mon[44304]: pgmap v2169: 47 pgs: 1 active+clean+scrubbing, 3 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 40 active+clean; 417 KiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 191 op/s; 0 B/s, 180 objects/s recovering 2024-02-09T17:59:54.836 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T17:59:54.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:54.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T17:59:54.944 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T17:59:54.971 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:55.319 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is set 2024-02-09T17:59:55.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:55 smithi179 ceph-mon[43975]: 1.5 scrub starts 2024-02-09T17:59:55.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:55 smithi179 ceph-mon[43975]: 1.5 scrub ok 2024-02-09T17:59:55.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:55 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1905499027' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:59:55.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:55 smithi162 ceph-mon[44304]: 1.5 scrub starts 2024-02-09T17:59:55.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:55 smithi162 ceph-mon[44304]: 1.5 scrub ok 2024-02-09T17:59:55.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:55 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1905499027' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:59:55.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:55 smithi099 ceph-mon[39420]: 1.5 scrub starts 2024-02-09T17:59:55.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:55 smithi099 ceph-mon[39420]: 1.5 scrub ok 2024-02-09T17:59:55.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:55 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1905499027' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T17:59:55.875 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T17:59:55.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:55.878 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T17:59:55.980 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:59:56.010 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:56.565 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:59:56.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:56.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:59:56.568 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:56 smithi099 ceph-mon[39420]: 1.15 scrub starts 2024-02-09T17:59:56.568 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:56 smithi099 ceph-mon[39420]: 1.15 scrub ok 2024-02-09T17:59:56.568 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:56 smithi099 ceph-mon[39420]: pgmap v2170: 47 pgs: 3 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 41 active+clean; 417 KiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 121 op/s; 0 B/s, 294 objects/s recovering 2024-02-09T17:59:56.568 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:56 smithi099 ceph-mon[39420]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:59:56.568 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:56 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1905499027' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:59:56.568 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:56 smithi099 ceph-mon[39420]: osdmap e876: 12 total, 12 up, 11 in 2024-02-09T17:59:56.578 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:59:56.669 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T17:59:56.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:56 smithi179 ceph-mon[43975]: 1.15 scrub starts 2024-02-09T17:59:56.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:56 smithi179 ceph-mon[43975]: 1.15 scrub ok 2024-02-09T17:59:56.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:56 smithi179 ceph-mon[43975]: pgmap v2170: 47 pgs: 3 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 41 active+clean; 417 KiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 121 op/s; 0 B/s, 294 objects/s recovering 2024-02-09T17:59:56.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:56 smithi179 ceph-mon[43975]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:59:56.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:56 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1905499027' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:59:56.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:56 smithi179 ceph-mon[43975]: osdmap e876: 12 total, 12 up, 11 in 2024-02-09T17:59:56.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:56 smithi162 ceph-mon[44304]: 1.15 scrub starts 2024-02-09T17:59:56.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:56 smithi162 ceph-mon[44304]: 1.15 scrub ok 2024-02-09T17:59:56.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:56 smithi162 ceph-mon[44304]: pgmap v2170: 47 pgs: 3 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 41 active+clean; 417 KiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 121 op/s; 0 B/s, 294 objects/s recovering 2024-02-09T17:59:56.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:56 smithi162 ceph-mon[44304]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T17:59:56.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:56 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1905499027' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T17:59:56.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:56 smithi162 ceph-mon[44304]: osdmap e876: 12 total, 12 up, 11 in 2024-02-09T17:59:56.698 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:58.051 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'unique_pool_6' removed 2024-02-09T17:59:58.399 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:58 smithi099 ceph-mon[39420]: osdmap e877: 12 total, 12 up, 11 in 2024-02-09T17:59:58.399 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:58 smithi099 ceph-mon[39420]: pgmap v2173: 47 pgs: 3 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 41 active+clean; 417 KiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 0 B/s, 232 objects/s recovering 2024-02-09T17:59:58.399 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:58 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2062581282' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-09T17:59:58.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:58 smithi179 ceph-mon[43975]: osdmap e877: 12 total, 12 up, 11 in 2024-02-09T17:59:58.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:58 smithi179 ceph-mon[43975]: pgmap v2173: 47 pgs: 3 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 41 active+clean; 417 KiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 0 B/s, 232 objects/s recovering 2024-02-09T17:59:58.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:58 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2062581282' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-09T17:59:58.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:58 smithi162 ceph-mon[44304]: osdmap e877: 12 total, 12 up, 11 in 2024-02-09T17:59:58.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:58 smithi162 ceph-mon[44304]: pgmap v2173: 47 pgs: 3 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 41 active+clean; 417 KiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 0 B/s, 232 objects/s recovering 2024-02-09T17:59:58.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:58 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2062581282' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-09T17:59:58.590 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-09T17:59:59.128 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T17:59:59.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:59.132 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T17:59:59.233 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T17:59:59.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:59 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2062581282' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-02-09T17:59:59.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 17:59:59 smithi099 ceph-mon[39420]: osdmap e878: 12 total, 12 up, 11 in 2024-02-09T17:59:59.357 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:59 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2062581282' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-02-09T17:59:59.358 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 17:59:59 smithi162 ceph-mon[44304]: osdmap e878: 12 total, 12 up, 11 in 2024-02-09T17:59:59.360 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:59.382 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T17:59:59.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:59.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T17:59:59.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:59 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2062581282' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-02-09T17:59:59.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 17:59:59 smithi179 ceph-mon[43975]: osdmap e878: 12 total, 12 up, 11 in 2024-02-09T17:59:59.486 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T17:59:59.517 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:59.520 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T17:59:59.539 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T17:59:59.542 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:59.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T17:59:59.643 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T17:59:59.672 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T17:59:59.771 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T17:59:59.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T17:59:59.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T17:59:59.875 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T17:59:59.903 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:00.025 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.5 to 1 (10000) 2024-02-09T18:00:00.281 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:00 smithi179 ceph-mon[43975]: pgmap v2175: 31 pgs: 31 active+clean; 417 KiB data, 8.5 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:00:00.281 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:00 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/308029244' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-09T18:00:00.281 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:00 smithi179 ceph-mon[43975]: Health detail: HEALTH_WARN noscrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull 2024-02-09T18:00:00.281 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:00 smithi179 ceph-mon[43975]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-02-09T18:00:00.281 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:00 smithi179 ceph-mon[43975]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull 2024-02-09T18:00:00.281 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:00 smithi179 ceph-mon[43975]: pg 8.2 is active+remapped+backfill_toofull, acting [10,0,4] 2024-02-09T18:00:00.281 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:00 smithi179 ceph-mon[43975]: pg 8.5 is active+remapped+backfill_toofull, acting [10,1,4] 2024-02-09T18:00:00.282 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:00 smithi179 ceph-mon[43975]: pg 8.b is active+remapped+backfill_toofull, acting [8,1,4] 2024-02-09T18:00:00.336 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:00 smithi099 ceph-mon[39420]: pgmap v2175: 31 pgs: 31 active+clean; 417 KiB data, 8.5 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:00:00.336 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:00 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/308029244' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-09T18:00:00.336 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:00 smithi099 ceph-mon[39420]: Health detail: HEALTH_WARN noscrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull 2024-02-09T18:00:00.337 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:00 smithi099 ceph-mon[39420]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-02-09T18:00:00.337 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:00 smithi099 ceph-mon[39420]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull 2024-02-09T18:00:00.337 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:00 smithi099 ceph-mon[39420]: pg 8.2 is active+remapped+backfill_toofull, acting [10,0,4] 2024-02-09T18:00:00.337 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:00 smithi099 ceph-mon[39420]: pg 8.5 is active+remapped+backfill_toofull, acting [10,1,4] 2024-02-09T18:00:00.337 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:00 smithi099 ceph-mon[39420]: pg 8.b is active+remapped+backfill_toofull, acting [8,1,4] 2024-02-09T18:00:00.361 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-02-09T18:00:00.361 INFO:tasks.radosbench:Beginning radosbench... 2024-02-09T18:00:00.361 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_7 2024-02-09T18:00:00.362 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd pool create unique_pool_7 16 2024-02-09T18:00:00.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:00 smithi162 ceph-mon[44304]: pgmap v2175: 31 pgs: 31 active+clean; 417 KiB data, 8.5 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:00:00.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:00 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/308029244' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-09T18:00:00.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:00 smithi162 ceph-mon[44304]: Health detail: HEALTH_WARN noscrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull 2024-02-09T18:00:00.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:00 smithi162 ceph-mon[44304]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-02-09T18:00:00.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:00 smithi162 ceph-mon[44304]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull 2024-02-09T18:00:00.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:00 smithi162 ceph-mon[44304]: pg 8.2 is active+remapped+backfill_toofull, acting [10,0,4] 2024-02-09T18:00:00.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:00 smithi162 ceph-mon[44304]: pg 8.5 is active+remapped+backfill_toofull, acting [10,1,4] 2024-02-09T18:00:00.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:00 smithi162 ceph-mon[44304]: pg 8.b is active+remapped+backfill_toofull, acting [8,1,4] 2024-02-09T18:00:00.552 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:00:00.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:00.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:00:00.656 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:00:00.692 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:01.026 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 6 1 2024-02-09T18:00:01.245 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:00:01.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:01.248 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:00:01.283 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:01 smithi179 ceph-mon[43975]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull) 2024-02-09T18:00:01.283 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:01 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/308029244' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-09T18:00:01.283 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:01 smithi179 ceph-mon[43975]: osdmap e879: 12 total, 12 up, 11 in 2024-02-09T18:00:01.350 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T18:00:01.382 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:01.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:01 smithi162 ceph-mon[44304]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull) 2024-02-09T18:00:01.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:01 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/308029244' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-09T18:00:01.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:01 smithi162 ceph-mon[44304]: osdmap e879: 12 total, 12 up, 11 in 2024-02-09T18:00:01.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:01 smithi099 ceph-mon[39420]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull) 2024-02-09T18:00:01.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:01 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/308029244' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-09T18:00:01.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:01 smithi099 ceph-mon[39420]: osdmap e879: 12 total, 12 up, 11 in 2024-02-09T18:00:01.508 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T18:00:01.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:01.511 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T18:00:01.612 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:00:01.640 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:01.738 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T18:00:01.738 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T18:00:01.993 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:00:01.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:01.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:00:02.033 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:00:02.098 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:00:02.125 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:02.154 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:00:02.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:02.161 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:00:02.235 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:02 smithi099 ceph-mon[39420]: pgmap v2177: 31 pgs: 31 active+clean; 417 KiB data, 8.4 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:00:02.262 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:00:02.291 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:02.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:02 smithi179 ceph-mon[43975]: pgmap v2177: 31 pgs: 31 active+clean; 417 KiB data, 8.4 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:00:02.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:02 smithi162 ceph-mon[44304]: pgmap v2177: 31 pgs: 31 active+clean; 417 KiB data, 8.4 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:00:02.448 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.8 dump_ops_in_flight 2024-02-09T18:00:02.708 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T18:00:03.642 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:00:03.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:03.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:00:03.746 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:00:03.775 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:03.848 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:00:03.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:03.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:00:03.968 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:00:03.995 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:04.034 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is set 2024-02-09T18:00:04.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:04 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:00:04.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:04 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:00:04.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:04 smithi099 ceph-mon[39420]: osdmap e880: 12 total, 12 up, 11 in 2024-02-09T18:00:04.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:04 smithi099 ceph-mon[39420]: pgmap v2179: 31 pgs: 31 active+clean; 417 KiB data, 8.5 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:00:04.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:04 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1574312608' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T18:00:04.288 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:04 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:00:04.288 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:04 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:00:04.288 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:04 smithi179 ceph-mon[43975]: osdmap e880: 12 total, 12 up, 11 in 2024-02-09T18:00:04.288 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:04 smithi179 ceph-mon[43975]: pgmap v2179: 31 pgs: 31 active+clean; 417 KiB data, 8.5 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:00:04.289 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:04 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1574312608' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T18:00:04.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:04 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:00:04.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:04 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:00:04.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:04 smithi162 ceph-mon[44304]: osdmap e880: 12 total, 12 up, 11 in 2024-02-09T18:00:04.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:04 smithi162 ceph-mon[44304]: pgmap v2179: 31 pgs: 31 active+clean; 417 KiB data, 8.5 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:00:04.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:04 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1574312608' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T18:00:04.567 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T18:00:04.567 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T18:00:04.944 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:00:04.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:04.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:00:05.049 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:00:05.082 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:05.083 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T18:00:05.090 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:00:05.104 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:00:05.105 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T18:00:05.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:05.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:00:05.124 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T18:00:05.144 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T18:00:05.164 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T18:00:05.184 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T18:00:05.204 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T18:00:05.208 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:00:05.224 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T18:00:05.230 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:05 smithi179 ceph-mon[43975]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:00:05.230 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:05 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1574312608' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T18:00:05.231 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:05 smithi179 ceph-mon[43975]: osdmap e881: 12 total, 12 up, 11 in 2024-02-09T18:00:05.232 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:05 smithi099 ceph-mon[39420]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:00:05.233 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:05 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1574312608' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T18:00:05.233 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:05 smithi099 ceph-mon[39420]: osdmap e881: 12 total, 12 up, 11 in 2024-02-09T18:00:05.235 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:05.247 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T18:00:05.258 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.8 dump_blocked_ops 2024-02-09T18:00:05.268 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T18:00:05.288 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T18:00:05.307 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T18:00:05.309 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T18:00:05.309 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T18:00:05.309 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T18:00:05.309 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T18:00:05.309 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T18:00:05.309 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T18:00:05.309 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T18:00:05.309 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T18:00:05.309 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T18:00:05.309 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T18:00:05.309 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T18:00:05.310 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T18:00:05.359 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:00:05.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:05.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:00:05.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:05 smithi162 ceph-mon[44304]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:00:05.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:05 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1574312608' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T18:00:05.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:05 smithi162 ceph-mon[44304]: osdmap e881: 12 total, 12 up, 11 in 2024-02-09T18:00:05.464 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:00:05.492 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:05.520 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T18:00:05.941 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:00:05.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:05.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:00:06.056 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:00:06.089 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:06.112 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:00:06.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:06.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:00:06.217 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:00:06.248 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:06.267 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:00:06.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:06.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:00:06.294 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'unique_pool_7' created 2024-02-09T18:00:06.371 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:00:06.398 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:06.406 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:06 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1183670206' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:00:06.406 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:06 smithi179 ceph-mon[43975]: pgmap v2181: 31 pgs: 31 active+clean; 417 KiB data, 8.5 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:00:06.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:06 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1183670206' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:00:06.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:06 smithi162 ceph-mon[44304]: pgmap v2181: 31 pgs: 31 active+clean; 417 KiB data, 8.5 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:00:06.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:06 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1183670206' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:00:06.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:06 smithi099 ceph-mon[39420]: pgmap v2181: 31 pgs: 31 active+clean; 417 KiB data, 8.5 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:00:06.546 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:00:06.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:06.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:00:06.651 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T18:00:06.682 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:06.720 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T18:00:06.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:06.724 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T18:00:06.825 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:00:06.853 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:06.878 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset noscrub 2024-02-09T18:00:07.201 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:00:07.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:07.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:00:07.216 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:00:07.302 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T18:00:07.303 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-02-09T18:00:07.303 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-02-09T18:00:07.303 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-02-09T18:00:07.303 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T18:00:07.305 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T18:00:07.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:07 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/127154320' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-02-09T18:00:07.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:07 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/127154320' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-02-09T18:00:07.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:07 smithi179 ceph-mon[43975]: osdmap e882: 12 total, 12 up, 11 in 2024-02-09T18:00:07.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:07 smithi179 ceph-mon[43975]: osdmap e883: 12 total, 12 up, 11 in 2024-02-09T18:00:07.335 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:07.354 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T18:00:07.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:07.357 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T18:00:07.456 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:07 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/127154320' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-02-09T18:00:07.456 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:07 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/127154320' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-02-09T18:00:07.456 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:07 smithi099 ceph-mon[39420]: osdmap e882: 12 total, 12 up, 11 in 2024-02-09T18:00:07.456 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:07 smithi099 ceph-mon[39420]: osdmap e883: 12 total, 12 up, 11 in 2024-02-09T18:00:07.458 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:00:07.490 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:07.511 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:00:07.514 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:07 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/127154320' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-02-09T18:00:07.514 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:07 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/127154320' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-02-09T18:00:07.514 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:07 smithi162 ceph-mon[44304]: osdmap e882: 12 total, 12 up, 11 in 2024-02-09T18:00:07.514 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:07 smithi162 ceph-mon[44304]: osdmap e883: 12 total, 12 up, 11 in 2024-02-09T18:00:07.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:07.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:00:07.616 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:00:07.644 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:07.973 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:00:07.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:07.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:00:07.989 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd pool application enable unique_pool_7 rados --yes-i-really-mean-it || true 2024-02-09T18:00:08.054 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.8 dump_historic_ops 2024-02-09T18:00:08.077 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:00:08.109 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:08.131 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:00:08.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:08.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:00:08.235 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T18:00:08.260 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T18:00:08.267 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:08.289 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T18:00:08.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:08.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T18:00:08.314 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T18:00:08.393 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:00:08.420 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:08.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:08 smithi179 ceph-mon[43975]: pgmap v2184: 47 pgs: 1 creating+activating, 15 unknown, 31 active+clean; 417 KiB data, 8.4 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:00:08.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:08 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:00:08.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:08 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:00:08.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:08 smithi179 ceph-mon[43975]: osdmap e884: 12 total, 12 up, 11 in 2024-02-09T18:00:08.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:08 smithi162 ceph-mon[44304]: pgmap v2184: 47 pgs: 1 creating+activating, 15 unknown, 31 active+clean; 417 KiB data, 8.4 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:00:08.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:08 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:00:08.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:08 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:00:08.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:08 smithi162 ceph-mon[44304]: osdmap e884: 12 total, 12 up, 11 in 2024-02-09T18:00:08.447 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:00:08.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:08.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:00:08.554 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:00:08.586 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:08.608 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:00:08.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:08.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:00:08.642 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:08 smithi099 ceph-mon[39420]: pgmap v2184: 47 pgs: 1 creating+activating, 15 unknown, 31 active+clean; 417 KiB data, 8.4 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:00:08.642 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:08 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:00:08.643 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:08 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:00:08.643 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:08 smithi099 ceph-mon[39420]: osdmap e884: 12 total, 12 up, 11 in 2024-02-09T18:00:08.712 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T18:00:08.739 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:08.813 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T18:00:08.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:08.815 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T18:00:08.916 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:00:08.948 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:09.502 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:00:09.507 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:09 smithi099 ceph-mon[39420]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-09T18:00:09.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:09.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:00:09.609 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:00:09.610 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:09 smithi179 ceph-mon[43975]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-09T18:00:09.639 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:09 smithi162 ceph-mon[44304]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-09T18:00:09.641 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:09.663 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:00:09.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:09.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:00:09.767 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:00:09.801 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:10.197 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T18:00:10.197 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-02-09T18:00:10.197 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-02-09T18:00:10.197 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-02-09T18:00:10.197 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.197 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.16440.0:160190 7.f 7:ff1ac3bf:::benchmark_data_smithi179_135804_object52399:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e744)", 2024-02-09T18:00:10.197 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:53:24.244691+0000", 2024-02-09T18:00:10.198 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 405.94813729999998, 2024-02-09T18:00:10.198 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.038466613, 2024-02-09T18:00:10.198 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:00:10.198 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:00:10.198 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:00:10.198 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.16440", 2024-02-09T18:00:10.198 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T18:00:10.198 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 160190 2024-02-09T18:00:10.198 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.198 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:00:10.198 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.198 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:00:10.199 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.244691+0000", 2024-02-09T18:00:10.199 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:00:10.199 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.199 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.199 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:00:10.199 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.244691+0000", 2024-02-09T18:00:10.199 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:00:10.199 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.199 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.199 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:00:10.199 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.244692+0000", 2024-02-09T18:00:10.199 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4339999999999999e-06 2024-02-09T18:00:10.199 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.199 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.200 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:00:10.200 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.244699+0000", 2024-02-09T18:00:10.200 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.153e-06 2024-02-09T18:00:10.200 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.200 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.200 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:00:10.200 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.244715+0000", 2024-02-09T18:00:10.200 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5498999999999998e-05 2024-02-09T18:00:10.200 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.200 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.200 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:00:10.200 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.244723+0000", 2024-02-09T18:00:10.200 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2090000000000005e-06 2024-02-09T18:00:10.201 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.201 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.201 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:00:10.201 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.244780+0000", 2024-02-09T18:00:10.201 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7321999999999998e-05 2024-02-09T18:00:10.201 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.201 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.201 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:00:10.201 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.244789+0000", 2024-02-09T18:00:10.201 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7770000000000008e-06 2024-02-09T18:00:10.201 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.201 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.202 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:00:10.202 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.273539+0000", 2024-02-09T18:00:10.202 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.028750082 2024-02-09T18:00:10.202 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.202 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.202 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:00:10.202 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.273683+0000", 2024-02-09T18:00:10.202 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014348 2024-02-09T18:00:10.202 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.202 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.202 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 9,11", 2024-02-09T18:00:10.202 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.273807+0000", 2024-02-09T18:00:10.202 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000124895 2024-02-09T18:00:10.202 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.203 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.203 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:00:10.203 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.282991+0000", 2024-02-09T18:00:10.203 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0091832379999999998 2024-02-09T18:00:10.203 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.203 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.203 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:00:10.203 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.283001+0000", 2024-02-09T18:00:10.203 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0054999999999999e-05 2024-02-09T18:00:10.203 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.203 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.203 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:00:10.203 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.283052+0000", 2024-02-09T18:00:10.203 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1022e-05 2024-02-09T18:00:10.204 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.204 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.204 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:00:10.204 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.283111+0000", 2024-02-09T18:00:10.204 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9327999999999998e-05 2024-02-09T18:00:10.204 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.204 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.204 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:00:10.204 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.283157+0000", 2024-02-09T18:00:10.204 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6118999999999997e-05 2024-02-09T18:00:10.204 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:00:10.204 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:00:10.204 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:00:10.204 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.205 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.205 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.16440.0:160193 7.f 7:f4b2d211:::benchmark_data_smithi179_135804_object52402:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e744)", 2024-02-09T18:00:10.205 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:53:24.248012+0000", 2024-02-09T18:00:10.205 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 405.944816323, 2024-02-09T18:00:10.205 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0440866170000001, 2024-02-09T18:00:10.205 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:00:10.205 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:00:10.205 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:00:10.205 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.16440", 2024-02-09T18:00:10.205 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T18:00:10.205 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 160193 2024-02-09T18:00:10.205 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.205 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:00:10.205 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.206 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:00:10.206 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.248012+0000", 2024-02-09T18:00:10.206 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:00:10.206 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.206 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.206 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:00:10.206 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.248012+0000", 2024-02-09T18:00:10.206 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:00:10.206 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.206 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.206 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:00:10.206 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.248012+0000", 2024-02-09T18:00:10.206 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.54e-07 2024-02-09T18:00:10.206 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.206 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.207 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:00:10.207 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.248017+0000", 2024-02-09T18:00:10.207 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2270000000000004e-06 2024-02-09T18:00:10.207 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.207 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.207 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:00:10.207 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.248027+0000", 2024-02-09T18:00:10.207 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0001e-05 2024-02-09T18:00:10.207 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.207 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.207 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:00:10.207 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.248032+0000", 2024-02-09T18:00:10.207 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4330000000000003e-06 2024-02-09T18:00:10.207 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.207 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.208 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:00:10.208 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.248078+0000", 2024-02-09T18:00:10.208 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6449000000000002e-05 2024-02-09T18:00:10.208 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.208 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.208 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:00:10.208 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.248086+0000", 2024-02-09T18:00:10.208 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.9759999999999995e-06 2024-02-09T18:00:10.208 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.208 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.208 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:00:10.208 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.278569+0000", 2024-02-09T18:00:10.208 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0304823679999999 2024-02-09T18:00:10.208 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.208 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.209 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:00:10.209 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.278707+0000", 2024-02-09T18:00:10.209 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013862100000000001 2024-02-09T18:00:10.209 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.209 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.209 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 9,11", 2024-02-09T18:00:10.209 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.278847+0000", 2024-02-09T18:00:10.209 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013958699999999999 2024-02-09T18:00:10.209 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.209 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.209 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:00:10.209 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.283188+0000", 2024-02-09T18:00:10.209 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0043405689999999999 2024-02-09T18:00:10.209 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.209 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.210 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:00:10.210 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.291953+0000", 2024-02-09T18:00:10.210 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0087656660000000001 2024-02-09T18:00:10.210 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.210 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.210 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:00:10.210 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.291999+0000", 2024-02-09T18:00:10.210 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5513999999999998e-05 2024-02-09T18:00:10.210 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.210 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.210 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:00:10.210 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.292049+0000", 2024-02-09T18:00:10.210 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0108000000000002e-05 2024-02-09T18:00:10.210 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.210 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.210 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:00:10.211 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.292098+0000", 2024-02-09T18:00:10.211 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9344000000000001e-05 2024-02-09T18:00:10.211 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:00:10.211 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:00:10.211 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:00:10.211 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.211 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.211 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.16440.0:160222 7.f 7:fd59ed18:::benchmark_data_smithi179_135804_object52431:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e744)", 2024-02-09T18:00:10.211 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:53:24.295295+0000", 2024-02-09T18:00:10.211 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 405.89753318099997, 2024-02-09T18:00:10.211 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.001343222, 2024-02-09T18:00:10.211 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:00:10.211 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:00:10.211 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:00:10.212 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.16440", 2024-02-09T18:00:10.212 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T18:00:10.212 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 160222 2024-02-09T18:00:10.212 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.212 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:00:10.212 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.212 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:00:10.212 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.295295+0000", 2024-02-09T18:00:10.212 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:00:10.212 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.212 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.212 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:00:10.212 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.295295+0000", 2024-02-09T18:00:10.212 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:00:10.213 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.213 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.213 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:00:10.213 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.295297+0000", 2024-02-09T18:00:10.213 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9910000000000001e-06 2024-02-09T18:00:10.213 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.213 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.213 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:00:10.213 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.295303+0000", 2024-02-09T18:00:10.213 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0889999999999996e-06 2024-02-09T18:00:10.213 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.213 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.213 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:00:10.213 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.295317+0000", 2024-02-09T18:00:10.213 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3995999999999999e-05 2024-02-09T18:00:10.214 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.214 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.214 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:00:10.214 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.295326+0000", 2024-02-09T18:00:10.214 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.3489999999999998e-06 2024-02-09T18:00:10.214 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.214 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.214 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:00:10.214 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.295392+0000", 2024-02-09T18:00:10.214 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5555000000000005e-05 2024-02-09T18:00:10.214 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.214 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.214 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:00:10.214 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.295402+0000", 2024-02-09T18:00:10.215 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.063e-05 2024-02-09T18:00:10.215 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.215 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.215 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:00:10.215 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.283317+0000", 2024-02-09T18:00:10.215 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.98791436399999999 2024-02-09T18:00:10.215 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.215 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.215 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:00:10.215 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.283422+0000", 2024-02-09T18:00:10.215 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000104972 2024-02-09T18:00:10.215 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.215 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.215 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 9,11", 2024-02-09T18:00:10.215 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.283500+0000", 2024-02-09T18:00:10.215 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8177999999999996e-05 2024-02-09T18:00:10.216 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.216 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.216 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:00:10.216 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.292110+0000", 2024-02-09T18:00:10.216 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0086103610000000004 2024-02-09T18:00:10.216 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.216 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.216 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:00:10.216 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.292184+0000", 2024-02-09T18:00:10.216 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4036999999999996e-05 2024-02-09T18:00:10.216 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.216 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.216 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:00:10.217 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.296562+0000", 2024-02-09T18:00:10.217 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0043781510000000003 2024-02-09T18:00:10.217 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.217 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.217 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:00:10.217 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.296609+0000", 2024-02-09T18:00:10.217 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6508999999999998e-05 2024-02-09T18:00:10.217 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.217 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.217 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:00:10.217 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.296638+0000", 2024-02-09T18:00:10.217 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.904e-05 2024-02-09T18:00:10.217 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:00:10.217 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:00:10.217 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:00:10.218 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.218 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.218 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.16440.0:160251 7.f 7:fb8ce13f:::benchmark_data_smithi179_135804_object52460:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e744)", 2024-02-09T18:00:10.218 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:53:24.353658+0000", 2024-02-09T18:00:10.218 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 405.83917016300001, 2024-02-09T18:00:10.218 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.96362845799999997, 2024-02-09T18:00:10.218 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:00:10.218 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:00:10.218 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:00:10.218 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.16440", 2024-02-09T18:00:10.218 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T18:00:10.218 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 160251 2024-02-09T18:00:10.218 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.218 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:00:10.219 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.219 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:00:10.219 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.353658+0000", 2024-02-09T18:00:10.219 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:00:10.219 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.219 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.219 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:00:10.219 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.353658+0000", 2024-02-09T18:00:10.219 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:00:10.219 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.219 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.219 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:00:10.219 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.353659+0000", 2024-02-09T18:00:10.219 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.491e-06 2024-02-09T18:00:10.219 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.220 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.220 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:00:10.220 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.353666+0000", 2024-02-09T18:00:10.220 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3910000000000003e-06 2024-02-09T18:00:10.220 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.220 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.220 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:00:10.220 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.353678+0000", 2024-02-09T18:00:10.220 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2813000000000001e-05 2024-02-09T18:00:10.220 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.220 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.220 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:00:10.221 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.353686+0000", 2024-02-09T18:00:10.221 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2840000000000001e-06 2024-02-09T18:00:10.221 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.221 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.221 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:00:10.221 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.353738+0000", 2024-02-09T18:00:10.221 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2274999999999997e-05 2024-02-09T18:00:10.221 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.221 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.221 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:00:10.221 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.353746+0000", 2024-02-09T18:00:10.221 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.9689999999999994e-06 2024-02-09T18:00:10.221 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.221 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.222 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:00:10.222 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.287806+0000", 2024-02-09T18:00:10.222 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.93406052100000003 2024-02-09T18:00:10.222 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.222 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.222 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:00:10.222 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.287917+0000", 2024-02-09T18:00:10.222 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000110024 2024-02-09T18:00:10.222 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.222 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.222 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 9,11", 2024-02-09T18:00:10.222 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.287997+0000", 2024-02-09T18:00:10.223 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.0907000000000003e-05 2024-02-09T18:00:10.223 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.223 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.223 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:00:10.223 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.296488+0000", 2024-02-09T18:00:10.223 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0084902290000000002 2024-02-09T18:00:10.223 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.223 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.223 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:00:10.223 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.296694+0000", 2024-02-09T18:00:10.223 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00020637100000000001 2024-02-09T18:00:10.223 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.223 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.223 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:00:10.224 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.317180+0000", 2024-02-09T18:00:10.224 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.020485961 2024-02-09T18:00:10.224 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.224 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.224 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:00:10.224 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.317243+0000", 2024-02-09T18:00:10.224 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2670999999999997e-05 2024-02-09T18:00:10.224 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.224 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.224 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:00:10.224 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.317286+0000", 2024-02-09T18:00:10.224 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3550999999999997e-05 2024-02-09T18:00:10.225 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:00:10.225 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:00:10.225 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:00:10.225 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.225 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.225 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.16440.0:160252 7.f 7:f2b592c6:::benchmark_data_smithi179_135804_object52461:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e744)", 2024-02-09T18:00:10.225 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:53:24.354825+0000", 2024-02-09T18:00:10.225 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 405.83800306099999, 2024-02-09T18:00:10.225 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.96261408000000004, 2024-02-09T18:00:10.225 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:00:10.225 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:00:10.225 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:00:10.225 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.16440", 2024-02-09T18:00:10.225 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T18:00:10.225 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 160252 2024-02-09T18:00:10.226 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.226 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:00:10.226 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.226 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:00:10.226 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.354825+0000", 2024-02-09T18:00:10.226 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:00:10.226 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.226 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.226 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:00:10.226 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.354825+0000", 2024-02-09T18:00:10.226 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:00:10.226 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.226 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.227 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:00:10.227 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.354826+0000", 2024-02-09T18:00:10.227 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2799999999999995e-07 2024-02-09T18:00:10.227 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.227 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.227 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:00:10.227 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.354830+0000", 2024-02-09T18:00:10.227 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1300000000000003e-06 2024-02-09T18:00:10.227 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.227 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.228 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:00:10.228 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.354839+0000", 2024-02-09T18:00:10.228 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.3360000000000001e-06 2024-02-09T18:00:10.228 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.228 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.228 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:00:10.228 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.354845+0000", 2024-02-09T18:00:10.228 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5389999999999998e-06 2024-02-09T18:00:10.229 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.229 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.229 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:00:10.229 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.354898+0000", 2024-02-09T18:00:10.229 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3600000000000002e-05 2024-02-09T18:00:10.229 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.229 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.229 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:00:10.229 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.354908+0000", 2024-02-09T18:00:10.229 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.4509999999999998e-06 2024-02-09T18:00:10.229 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.229 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.229 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:00:10.230 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.292221+0000", 2024-02-09T18:00:10.230 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.93731293800000004 2024-02-09T18:00:10.230 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.230 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.230 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:00:10.230 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.292315+0000", 2024-02-09T18:00:10.230 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.4906000000000002e-05 2024-02-09T18:00:10.230 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.230 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.230 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 9,11", 2024-02-09T18:00:10.230 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.292398+0000", 2024-02-09T18:00:10.230 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2027999999999995e-05 2024-02-09T18:00:10.230 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.230 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.231 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:00:10.231 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.296736+0000", 2024-02-09T18:00:10.231 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0043380040000000003 2024-02-09T18:00:10.231 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.231 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.231 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:00:10.231 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.305942+0000", 2024-02-09T18:00:10.231 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0092062299999999993 2024-02-09T18:00:10.231 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.231 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.231 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:00:10.231 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.317344+0000", 2024-02-09T18:00:10.231 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.01140265 2024-02-09T18:00:10.231 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.231 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.232 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:00:10.232 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.317411+0000", 2024-02-09T18:00:10.232 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6427999999999995e-05 2024-02-09T18:00:10.232 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.232 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.232 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:00:10.232 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.317439+0000", 2024-02-09T18:00:10.232 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.8112000000000001e-05 2024-02-09T18:00:10.232 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:00:10.232 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:00:10.232 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:00:10.232 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.232 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.232 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.16440.0:160262 7.f 7:f556bb46:::benchmark_data_smithi179_135804_object52471:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e744)", 2024-02-09T18:00:10.232 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:53:24.361835+0000", 2024-02-09T18:00:10.233 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 405.83099270399998, 2024-02-09T18:00:10.233 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.95574056100000004, 2024-02-09T18:00:10.233 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:00:10.233 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:00:10.233 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:00:10.233 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.16440", 2024-02-09T18:00:10.233 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T18:00:10.233 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 160262 2024-02-09T18:00:10.233 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.233 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:00:10.233 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.233 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:00:10.233 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.361835+0000", 2024-02-09T18:00:10.233 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:00:10.233 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.234 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.234 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:00:10.234 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.361835+0000", 2024-02-09T18:00:10.234 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:00:10.234 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.234 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.234 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:00:10.234 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.361837+0000", 2024-02-09T18:00:10.234 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4270000000000001e-06 2024-02-09T18:00:10.234 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.234 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.234 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:00:10.234 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.361843+0000", 2024-02-09T18:00:10.234 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5509999999999996e-06 2024-02-09T18:00:10.234 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.234 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.235 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:00:10.235 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.361858+0000", 2024-02-09T18:00:10.235 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5150999999999999e-05 2024-02-09T18:00:10.235 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.235 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.235 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:00:10.235 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.361865+0000", 2024-02-09T18:00:10.235 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.951e-06 2024-02-09T18:00:10.235 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.235 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.235 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:00:10.235 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.361928+0000", 2024-02-09T18:00:10.235 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2768999999999995e-05 2024-02-09T18:00:10.235 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.235 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.236 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:00:10.236 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.361935+0000", 2024-02-09T18:00:10.236 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4370000000000001e-06 2024-02-09T18:00:10.236 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.236 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.236 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:00:10.236 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.296770+0000", 2024-02-09T18:00:10.236 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.93483409699999997 2024-02-09T18:00:10.236 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.236 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.236 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:00:10.236 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.296895+0000", 2024-02-09T18:00:10.236 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012538800000000001 2024-02-09T18:00:10.236 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.236 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.236 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 9,11", 2024-02-09T18:00:10.237 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.297025+0000", 2024-02-09T18:00:10.237 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013035499999999999 2024-02-09T18:00:10.237 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.237 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.237 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:00:10.237 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.312356+0000", 2024-02-09T18:00:10.237 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.015330617 2024-02-09T18:00:10.237 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.237 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.237 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:00:10.237 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.317073+0000", 2024-02-09T18:00:10.237 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0047165849999999997 2024-02-09T18:00:10.237 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.237 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.237 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:00:10.238 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.317506+0000", 2024-02-09T18:00:10.238 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00043317300000000002 2024-02-09T18:00:10.238 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.238 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.238 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:00:10.238 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.317551+0000", 2024-02-09T18:00:10.238 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5713000000000001e-05 2024-02-09T18:00:10.238 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.238 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.238 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:00:10.238 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.317576+0000", 2024-02-09T18:00:10.238 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4346999999999999e-05 2024-02-09T18:00:10.238 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:00:10.238 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:00:10.238 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:00:10.239 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.239 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.239 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.16440.0:160284 7.f 7:f7cbf418:::benchmark_data_smithi179_135804_object52493:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e744)", 2024-02-09T18:00:10.239 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:53:24.398953+0000", 2024-02-09T18:00:10.239 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 405.79387477, 2024-02-09T18:00:10.239 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.91877487999999996, 2024-02-09T18:00:10.239 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:00:10.239 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:00:10.239 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:00:10.239 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.16440", 2024-02-09T18:00:10.239 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T18:00:10.239 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 160284 2024-02-09T18:00:10.239 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.239 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:00:10.239 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.240 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:00:10.240 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.398953+0000", 2024-02-09T18:00:10.240 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:00:10.240 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.240 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.240 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:00:10.240 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.398953+0000", 2024-02-09T18:00:10.240 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:00:10.240 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.240 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.240 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:00:10.240 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.398955+0000", 2024-02-09T18:00:10.240 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.655e-06 2024-02-09T18:00:10.240 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.240 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.241 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:00:10.241 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.398962+0000", 2024-02-09T18:00:10.241 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5079999999999999e-06 2024-02-09T18:00:10.241 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.241 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.241 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:00:10.241 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.398977+0000", 2024-02-09T18:00:10.241 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5167000000000001e-05 2024-02-09T18:00:10.241 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.241 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.241 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:00:10.241 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.398986+0000", 2024-02-09T18:00:10.241 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4130000000000006e-06 2024-02-09T18:00:10.241 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.242 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.242 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:00:10.242 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.399058+0000", 2024-02-09T18:00:10.242 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2138000000000006e-05 2024-02-09T18:00:10.242 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.242 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.242 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:00:10.242 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.399070+0000", 2024-02-09T18:00:10.242 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1846e-05 2024-02-09T18:00:10.242 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.242 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.242 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:00:10.242 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.301351+0000", 2024-02-09T18:00:10.242 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.90228114599999998 2024-02-09T18:00:10.242 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.243 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.243 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:00:10.243 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.301494+0000", 2024-02-09T18:00:10.243 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000142831 2024-02-09T18:00:10.243 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.243 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.243 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 9,11", 2024-02-09T18:00:10.243 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.301617+0000", 2024-02-09T18:00:10.243 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000123499 2024-02-09T18:00:10.243 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.243 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.243 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:00:10.243 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.317091+0000", 2024-02-09T18:00:10.243 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.015473618999999999 2024-02-09T18:00:10.244 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.244 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.244 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:00:10.244 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.317139+0000", 2024-02-09T18:00:10.244 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7611000000000002e-05 2024-02-09T18:00:10.244 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.244 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.244 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:00:10.244 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.317640+0000", 2024-02-09T18:00:10.244 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00050167199999999999 2024-02-09T18:00:10.244 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.244 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.244 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:00:10.244 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.317690+0000", 2024-02-09T18:00:10.245 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0040999999999997e-05 2024-02-09T18:00:10.245 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.245 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.245 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:00:10.245 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.317728+0000", 2024-02-09T18:00:10.245 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7734000000000002e-05 2024-02-09T18:00:10.245 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:00:10.245 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:00:10.245 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:00:10.245 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.245 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.245 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.16440.0:160287 7.f 7:f42da2f8:::benchmark_data_smithi179_135804_object52496:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e744)", 2024-02-09T18:00:10.245 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:53:24.399841+0000", 2024-02-09T18:00:10.245 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 405.79298717500001, 2024-02-09T18:00:10.245 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.918048223, 2024-02-09T18:00:10.245 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:00:10.246 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:00:10.246 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:00:10.246 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.16440", 2024-02-09T18:00:10.246 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T18:00:10.246 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 160287 2024-02-09T18:00:10.246 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.246 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:00:10.246 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.246 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:00:10.246 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.399841+0000", 2024-02-09T18:00:10.246 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:00:10.246 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.246 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.247 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:00:10.247 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.399841+0000", 2024-02-09T18:00:10.247 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:00:10.247 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.247 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.247 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:00:10.247 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.399842+0000", 2024-02-09T18:00:10.247 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.071e-06 2024-02-09T18:00:10.247 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.247 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.247 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:00:10.247 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.399847+0000", 2024-02-09T18:00:10.247 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4539999999999998e-06 2024-02-09T18:00:10.247 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.248 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.248 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:00:10.248 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.399861+0000", 2024-02-09T18:00:10.248 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.418e-05 2024-02-09T18:00:10.248 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.248 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.248 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:00:10.248 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.399871+0000", 2024-02-09T18:00:10.248 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2320000000000007e-06 2024-02-09T18:00:10.248 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.248 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.248 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:00:10.248 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.399937+0000", 2024-02-09T18:00:10.248 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5862000000000001e-05 2024-02-09T18:00:10.249 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.249 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.249 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:00:10.249 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.399947+0000", 2024-02-09T18:00:10.249 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0494e-05 2024-02-09T18:00:10.249 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.249 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.249 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:00:10.249 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.305976+0000", 2024-02-09T18:00:10.249 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.90602855100000002 2024-02-09T18:00:10.249 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.249 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.249 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:00:10.249 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.306093+0000", 2024-02-09T18:00:10.249 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000116946 2024-02-09T18:00:10.250 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.250 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.250 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 9,11", 2024-02-09T18:00:10.250 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.306842+0000", 2024-02-09T18:00:10.250 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00074927000000000001 2024-02-09T18:00:10.250 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.250 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.250 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:00:10.250 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.317456+0000", 2024-02-09T18:00:10.250 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010614142999999999 2024-02-09T18:00:10.250 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.250 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.250 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:00:10.250 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.317766+0000", 2024-02-09T18:00:10.250 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00030963300000000002 2024-02-09T18:00:10.251 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.251 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.251 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:00:10.251 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.317808+0000", 2024-02-09T18:00:10.251 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2517e-05 2024-02-09T18:00:10.251 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.251 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.251 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:00:10.251 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.317855+0000", 2024-02-09T18:00:10.251 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6650000000000002e-05 2024-02-09T18:00:10.251 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.251 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.251 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:00:10.251 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.317889+0000", 2024-02-09T18:00:10.252 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4220000000000001e-05 2024-02-09T18:00:10.252 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:00:10.252 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:00:10.252 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:00:10.252 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.252 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.252 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.16440.0:160325 7.f 7:f76aaeaf:::benchmark_data_smithi179_135804_object52534:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e744)", 2024-02-09T18:00:10.252 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:53:24.462218+0000", 2024-02-09T18:00:10.252 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 405.730609759, 2024-02-09T18:00:10.252 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.85581675499999998, 2024-02-09T18:00:10.252 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:00:10.252 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:00:10.252 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:00:10.253 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.16440", 2024-02-09T18:00:10.253 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T18:00:10.253 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 160325 2024-02-09T18:00:10.253 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.253 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:00:10.253 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.253 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:00:10.253 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.462218+0000", 2024-02-09T18:00:10.253 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:00:10.253 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.253 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.253 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:00:10.253 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.462218+0000", 2024-02-09T18:00:10.253 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:00:10.253 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.254 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.254 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:00:10.254 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.462220+0000", 2024-02-09T18:00:10.254 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0159999999999998e-06 2024-02-09T18:00:10.254 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.254 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.254 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:00:10.254 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.462227+0000", 2024-02-09T18:00:10.254 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6749999999999996e-06 2024-02-09T18:00:10.254 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.254 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.254 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:00:10.254 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.462242+0000", 2024-02-09T18:00:10.254 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5639000000000001e-05 2024-02-09T18:00:10.254 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.255 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.255 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:00:10.255 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.462251+0000", 2024-02-09T18:00:10.255 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2360000000000004e-06 2024-02-09T18:00:10.255 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.255 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.255 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:00:10.255 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.462341+0000", 2024-02-09T18:00:10.255 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.0113999999999998e-05 2024-02-09T18:00:10.255 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.255 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.255 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:00:10.255 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.462353+0000", 2024-02-09T18:00:10.255 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2357e-05 2024-02-09T18:00:10.256 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.256 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.256 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:00:10.256 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.312404+0000", 2024-02-09T18:00:10.256 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.85005111099999997 2024-02-09T18:00:10.256 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.256 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.256 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:00:10.256 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.312545+0000", 2024-02-09T18:00:10.256 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000140321 2024-02-09T18:00:10.256 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.256 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.256 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 9,11", 2024-02-09T18:00:10.256 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.312681+0000", 2024-02-09T18:00:10.256 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000136067 2024-02-09T18:00:10.257 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.257 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.257 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:00:10.257 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.317740+0000", 2024-02-09T18:00:10.257 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0050591129999999996 2024-02-09T18:00:10.257 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.257 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.257 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:00:10.257 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.317920+0000", 2024-02-09T18:00:10.257 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00017997299999999999 2024-02-09T18:00:10.257 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.257 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.257 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:00:10.257 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.317963+0000", 2024-02-09T18:00:10.257 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3545999999999999e-05 2024-02-09T18:00:10.258 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.258 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.258 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:00:10.258 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.318004+0000", 2024-02-09T18:00:10.258 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1014999999999999e-05 2024-02-09T18:00:10.258 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.258 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.258 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:00:10.258 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.318035+0000", 2024-02-09T18:00:10.258 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0571999999999998e-05 2024-02-09T18:00:10.258 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:00:10.258 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:00:10.258 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:00:10.258 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.259 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.259 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.16440.0:160331 7.f 7:f3f0e3af:::benchmark_data_smithi179_135804_object52540:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e744)", 2024-02-09T18:00:10.259 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:53:24.469782+0000", 2024-02-09T18:00:10.259 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 405.72304630999997, 2024-02-09T18:00:10.259 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.88471905100000003, 2024-02-09T18:00:10.259 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:00:10.259 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:00:10.259 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:00:10.259 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.16440", 2024-02-09T18:00:10.259 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T18:00:10.259 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 160331 2024-02-09T18:00:10.259 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.259 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:00:10.259 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.259 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:00:10.260 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.469782+0000", 2024-02-09T18:00:10.260 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:00:10.260 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.260 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.260 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:00:10.260 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.469782+0000", 2024-02-09T18:00:10.260 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:00:10.260 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.260 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.260 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:00:10.260 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.469783+0000", 2024-02-09T18:00:10.260 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3799999999999999e-06 2024-02-09T18:00:10.260 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.260 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.260 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:00:10.261 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.469788+0000", 2024-02-09T18:00:10.261 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3149999999999999e-06 2024-02-09T18:00:10.261 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.261 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.261 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:00:10.261 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.469800+0000", 2024-02-09T18:00:10.261 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2143e-05 2024-02-09T18:00:10.261 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.261 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.261 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:00:10.261 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.469808+0000", 2024-02-09T18:00:10.261 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2949999999999996e-06 2024-02-09T18:00:10.261 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.261 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.261 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:00:10.262 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.469861+0000", 2024-02-09T18:00:10.262 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3514000000000003e-05 2024-02-09T18:00:10.262 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.262 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.262 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:00:10.262 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.469869+0000", 2024-02-09T18:00:10.262 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1580000000000005e-06 2024-02-09T18:00:10.262 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.262 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.262 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:00:10.262 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.318064+0000", 2024-02-09T18:00:10.262 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.84819491899999999 2024-02-09T18:00:10.262 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.262 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.263 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:00:10.263 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.318159+0000", 2024-02-09T18:00:10.263 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5136999999999994e-05 2024-02-09T18:00:10.263 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.263 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.263 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 9,11", 2024-02-09T18:00:10.263 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.318261+0000", 2024-02-09T18:00:10.263 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000101952 2024-02-09T18:00:10.263 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.263 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.263 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:00:10.263 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.338031+0000", 2024-02-09T18:00:10.264 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.019769450000000001 2024-02-09T18:00:10.264 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.264 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.264 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:00:10.264 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.343577+0000", 2024-02-09T18:00:10.264 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0055456799999999999 2024-02-09T18:00:10.264 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.264 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.264 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:00:10.264 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.354412+0000", 2024-02-09T18:00:10.264 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010835239 2024-02-09T18:00:10.264 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.265 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.265 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:00:10.265 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.354462+0000", 2024-02-09T18:00:10.265 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0250000000000002e-05 2024-02-09T18:00:10.265 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.265 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.265 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:00:10.265 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.354501+0000", 2024-02-09T18:00:10.265 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8618999999999997e-05 2024-02-09T18:00:10.265 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:00:10.265 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:00:10.265 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:00:10.265 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.265 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.265 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.16440.0:160339 7.f 7:f4e1b0c0:::benchmark_data_smithi179_135804_object52548:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e744)", 2024-02-09T18:00:10.266 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:53:24.499344+0000", 2024-02-09T18:00:10.266 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 405.69348408100001, 2024-02-09T18:00:10.266 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.86118746000000002, 2024-02-09T18:00:10.266 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:00:10.266 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:00:10.266 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:00:10.266 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.16440", 2024-02-09T18:00:10.266 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T18:00:10.266 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 160339 2024-02-09T18:00:10.266 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.266 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:00:10.266 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.266 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:00:10.267 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.499344+0000", 2024-02-09T18:00:10.267 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:00:10.267 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.267 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.267 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:00:10.267 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.499344+0000", 2024-02-09T18:00:10.267 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:00:10.267 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.267 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.267 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:00:10.267 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.499345+0000", 2024-02-09T18:00:10.267 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5060000000000001e-06 2024-02-09T18:00:10.267 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.267 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.267 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:00:10.267 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.499351+0000", 2024-02-09T18:00:10.267 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6699999999999999e-06 2024-02-09T18:00:10.268 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.268 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.268 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:00:10.268 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.499366+0000", 2024-02-09T18:00:10.268 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4961e-05 2024-02-09T18:00:10.268 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.268 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.268 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:00:10.268 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.499377+0000", 2024-02-09T18:00:10.268 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0783e-05 2024-02-09T18:00:10.268 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.268 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.268 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:00:10.268 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.499448+0000", 2024-02-09T18:00:10.268 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1667000000000003e-05 2024-02-09T18:00:10.269 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.269 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.269 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:00:10.269 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.499459+0000", 2024-02-09T18:00:10.269 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0566e-05 2024-02-09T18:00:10.269 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.269 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.269 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:00:10.269 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.323620+0000", 2024-02-09T18:00:10.269 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.82416054900000002 2024-02-09T18:00:10.269 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.269 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.269 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:00:10.269 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.323739+0000", 2024-02-09T18:00:10.270 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000119857 2024-02-09T18:00:10.270 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.270 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.270 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 9,11", 2024-02-09T18:00:10.270 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.323860+0000", 2024-02-09T18:00:10.270 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000120183 2024-02-09T18:00:10.270 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.270 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.270 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:00:10.270 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.338067+0000", 2024-02-09T18:00:10.270 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014206943 2024-02-09T18:00:10.270 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.270 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.270 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:00:10.270 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.354538+0000", 2024-02-09T18:00:10.271 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.016471031000000001 2024-02-09T18:00:10.271 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.271 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.271 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:00:10.271 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.360458+0000", 2024-02-09T18:00:10.271 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0059200329999999999 2024-02-09T18:00:10.271 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.271 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.271 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:00:10.271 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.360490+0000", 2024-02-09T18:00:10.271 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2786000000000001e-05 2024-02-09T18:00:10.271 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.271 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.272 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:00:10.272 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.360531+0000", 2024-02-09T18:00:10.272 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0924999999999999e-05 2024-02-09T18:00:10.272 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:00:10.272 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:00:10.272 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:00:10.272 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.272 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.272 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.16440.0:160365 7.f 7:f5f679d1:::benchmark_data_smithi179_135804_object52574:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e744)", 2024-02-09T18:00:10.272 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:53:24.562932+0000", 2024-02-09T18:00:10.272 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 405.62989570000002, 2024-02-09T18:00:10.272 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.80744457400000003, 2024-02-09T18:00:10.272 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:00:10.272 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:00:10.273 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:00:10.273 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.16440", 2024-02-09T18:00:10.273 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T18:00:10.273 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 160365 2024-02-09T18:00:10.273 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.273 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:00:10.273 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.273 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:00:10.273 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.562932+0000", 2024-02-09T18:00:10.273 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:00:10.273 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.273 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.273 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:00:10.273 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.562932+0000", 2024-02-09T18:00:10.274 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:00:10.274 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.274 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.274 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:00:10.274 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.562934+0000", 2024-02-09T18:00:10.274 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1710000000000001e-06 2024-02-09T18:00:10.274 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.274 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.274 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:00:10.274 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.562940+0000", 2024-02-09T18:00:10.274 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9610000000000002e-06 2024-02-09T18:00:10.274 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.274 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.274 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:00:10.275 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.562956+0000", 2024-02-09T18:00:10.275 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6067e-05 2024-02-09T18:00:10.275 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.275 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.275 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:00:10.275 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.562966+0000", 2024-02-09T18:00:10.275 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6309999999999998e-06 2024-02-09T18:00:10.275 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.275 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.275 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:00:10.275 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.563038+0000", 2024-02-09T18:00:10.275 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1768999999999997e-05 2024-02-09T18:00:10.275 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.276 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.276 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:00:10.276 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:24.563051+0000", 2024-02-09T18:00:10.276 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3147e-05 2024-02-09T18:00:10.276 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.276 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.276 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:00:10.276 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.328499+0000", 2024-02-09T18:00:10.276 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.765447981 2024-02-09T18:00:10.276 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.276 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.276 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:00:10.276 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.328621+0000", 2024-02-09T18:00:10.276 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012197399999999999 2024-02-09T18:00:10.277 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.277 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.277 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 9,11", 2024-02-09T18:00:10.277 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.328747+0000", 2024-02-09T18:00:10.277 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012642799999999999 2024-02-09T18:00:10.277 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.277 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.277 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:00:10.277 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.354577+0000", 2024-02-09T18:00:10.277 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.025829648 2024-02-09T18:00:10.277 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.277 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.277 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:00:10.278 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.360536+0000", 2024-02-09T18:00:10.278 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0059594280000000001 2024-02-09T18:00:10.278 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.278 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.278 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:00:10.278 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.370268+0000", 2024-02-09T18:00:10.278 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0097315360000000007 2024-02-09T18:00:10.278 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.278 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.278 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:00:10.278 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.370330+0000", 2024-02-09T18:00:10.278 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1587000000000005e-05 2024-02-09T18:00:10.278 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.279 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.279 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:00:10.279 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:53:25.370377+0000", 2024-02-09T18:00:10.279 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7246000000000001e-05 2024-02-09T18:00:10.279 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:00:10.279 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:00:10.279 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:00:10.279 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.279 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.279 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:164145 8.b 8:dc04903d:::benchmark_data_smithi179_148205_object41073:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e828)", 2024-02-09T18:00:10.279 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:57:44.187886+0000", 2024-02-09T18:00:10.279 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 146.00494184600001, 2024-02-09T18:00:10.279 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.21149133000000001, 2024-02-09T18:00:10.279 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:00:10.279 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:00:10.280 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:00:10.280 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:00:10.280 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:00:10.280 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 164145 2024-02-09T18:00:10.280 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.280 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:00:10.280 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.280 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:00:10.280 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.187886+0000", 2024-02-09T18:00:10.280 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:00:10.280 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.280 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.280 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:00:10.281 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.187886+0000", 2024-02-09T18:00:10.281 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:00:10.281 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.281 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.281 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:00:10.281 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.187888+0000", 2024-02-09T18:00:10.281 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.81e-06 2024-02-09T18:00:10.281 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.281 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.281 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:00:10.281 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.187894+0000", 2024-02-09T18:00:10.281 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3720000000000001e-06 2024-02-09T18:00:10.281 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.281 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.282 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:00:10.282 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.187906+0000", 2024-02-09T18:00:10.282 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.205e-05 2024-02-09T18:00:10.282 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.282 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.282 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:00:10.282 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.187916+0000", 2024-02-09T18:00:10.282 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2909999999999996e-06 2024-02-09T18:00:10.282 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.282 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.282 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:00:10.282 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.187974+0000", 2024-02-09T18:00:10.282 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8207e-05 2024-02-09T18:00:10.282 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.282 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.283 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:00:10.283 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.188053+0000", 2024-02-09T18:00:10.283 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.9464999999999999e-05 2024-02-09T18:00:10.283 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.283 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.283 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4", 2024-02-09T18:00:10.283 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.188123+0000", 2024-02-09T18:00:10.283 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9374000000000004e-05 2024-02-09T18:00:10.283 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.283 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.283 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:00:10.283 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.192987+0000", 2024-02-09T18:00:10.283 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0048644279999999996 2024-02-09T18:00:10.283 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.283 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.284 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:00:10.284 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.194465+0000", 2024-02-09T18:00:10.284 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0014777919999999999 2024-02-09T18:00:10.284 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.284 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.284 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:00:10.284 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.399266+0000", 2024-02-09T18:00:10.284 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.20480121400000001 2024-02-09T18:00:10.284 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.284 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.284 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:00:10.284 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.399333+0000", 2024-02-09T18:00:10.284 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6987000000000001e-05 2024-02-09T18:00:10.284 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.284 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.285 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:00:10.285 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.399377+0000", 2024-02-09T18:00:10.285 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4339999999999999e-05 2024-02-09T18:00:10.285 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:00:10.285 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:00:10.285 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:00:10.285 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.285 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.285 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:164167 8.b 8:ddd29c13:::benchmark_data_smithi179_148205_object41095:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e828)", 2024-02-09T18:00:10.285 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:57:44.217437+0000", 2024-02-09T18:00:10.285 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 145.975390887, 2024-02-09T18:00:10.285 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.18209402799999999, 2024-02-09T18:00:10.285 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:00:10.285 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:00:10.285 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:00:10.286 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:00:10.286 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:00:10.286 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 164167 2024-02-09T18:00:10.286 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.286 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:00:10.286 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.286 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:00:10.286 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.217437+0000", 2024-02-09T18:00:10.286 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:00:10.286 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.286 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.286 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:00:10.286 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.217437+0000", 2024-02-09T18:00:10.286 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:00:10.286 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.287 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.287 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:00:10.287 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.217439+0000", 2024-02-09T18:00:10.287 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.632e-06 2024-02-09T18:00:10.287 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.287 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.287 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:00:10.287 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.217446+0000", 2024-02-09T18:00:10.287 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7600000000000002e-06 2024-02-09T18:00:10.287 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.287 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.287 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:00:10.287 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.217460+0000", 2024-02-09T18:00:10.287 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3624e-05 2024-02-09T18:00:10.287 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.287 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.288 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:00:10.288 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.217469+0000", 2024-02-09T18:00:10.288 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5390000000000007e-06 2024-02-09T18:00:10.288 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.288 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.288 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:00:10.288 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.217530+0000", 2024-02-09T18:00:10.288 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1326000000000002e-05 2024-02-09T18:00:10.288 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.288 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.288 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:00:10.288 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.217665+0000", 2024-02-09T18:00:10.288 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013489099999999999 2024-02-09T18:00:10.288 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.289 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.289 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4", 2024-02-09T18:00:10.289 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.217804+0000", 2024-02-09T18:00:10.289 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000139443 2024-02-09T18:00:10.289 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.289 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.289 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:00:10.289 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.221795+0000", 2024-02-09T18:00:10.289 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0039909719999999997 2024-02-09T18:00:10.289 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.289 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.289 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:00:10.289 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.222839+0000", 2024-02-09T18:00:10.289 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0010436499999999999 2024-02-09T18:00:10.289 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.290 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.290 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:00:10.290 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.399435+0000", 2024-02-09T18:00:10.290 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.17659582200000001 2024-02-09T18:00:10.290 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.290 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.290 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:00:10.290 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.399490+0000", 2024-02-09T18:00:10.290 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5275999999999997e-05 2024-02-09T18:00:10.290 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.290 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.290 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:00:10.290 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.399531+0000", 2024-02-09T18:00:10.290 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1093000000000001e-05 2024-02-09T18:00:10.290 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:00:10.291 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:00:10.291 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:00:10.291 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.291 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.291 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:164177 8.b 8:d8ae48a4:::benchmark_data_smithi179_148205_object41105:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e828)", 2024-02-09T18:00:10.291 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:57:44.232231+0000", 2024-02-09T18:00:10.291 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 145.960597194, 2024-02-09T18:00:10.291 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.16745369099999999, 2024-02-09T18:00:10.291 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:00:10.291 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:00:10.291 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:00:10.291 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:00:10.291 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:00:10.292 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 164177 2024-02-09T18:00:10.292 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.292 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:00:10.292 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.292 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:00:10.292 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.232231+0000", 2024-02-09T18:00:10.292 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:00:10.292 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.292 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.292 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:00:10.292 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.232231+0000", 2024-02-09T18:00:10.292 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:00:10.292 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.292 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.293 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:00:10.293 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.232231+0000", 2024-02-09T18:00:10.293 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4000000000000002e-07 2024-02-09T18:00:10.293 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.293 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.293 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:00:10.293 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.232234+0000", 2024-02-09T18:00:10.293 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.9890000000000001e-06 2024-02-09T18:00:10.293 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.293 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.293 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:00:10.293 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.232242+0000", 2024-02-09T18:00:10.293 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6909999999999997e-06 2024-02-09T18:00:10.293 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.293 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.294 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:00:10.294 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.232247+0000", 2024-02-09T18:00:10.294 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4589999999999997e-06 2024-02-09T18:00:10.294 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.294 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.294 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:00:10.294 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.232501+0000", 2024-02-09T18:00:10.294 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00025392600000000002 2024-02-09T18:00:10.294 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.294 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.294 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:00:10.294 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.232612+0000", 2024-02-09T18:00:10.294 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011082999999999999 2024-02-09T18:00:10.295 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.295 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.295 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4", 2024-02-09T18:00:10.295 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.232719+0000", 2024-02-09T18:00:10.295 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000106783 2024-02-09T18:00:10.295 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.295 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.295 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:00:10.295 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.237739+0000", 2024-02-09T18:00:10.295 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0050199240000000003 2024-02-09T18:00:10.295 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.295 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.295 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:00:10.295 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.241431+0000", 2024-02-09T18:00:10.296 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0036920410000000001 2024-02-09T18:00:10.296 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.296 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.296 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:00:10.296 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.399589+0000", 2024-02-09T18:00:10.296 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15815844600000001 2024-02-09T18:00:10.296 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.296 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.296 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:00:10.296 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.399646+0000", 2024-02-09T18:00:10.296 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6699e-05 2024-02-09T18:00:10.296 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.296 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.296 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:00:10.297 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.399684+0000", 2024-02-09T18:00:10.297 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8463000000000001e-05 2024-02-09T18:00:10.297 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:00:10.297 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:00:10.297 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:00:10.297 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.297 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.297 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:164188 8.b 8:dc76ac9f:::benchmark_data_smithi179_148205_object41116:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e828)", 2024-02-09T18:00:10.297 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:57:44.249029+0000", 2024-02-09T18:00:10.297 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 145.94379847600001, 2024-02-09T18:00:10.297 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15084728999999999, 2024-02-09T18:00:10.297 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:00:10.297 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:00:10.297 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:00:10.297 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:00:10.298 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:00:10.298 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 164188 2024-02-09T18:00:10.298 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.298 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:00:10.298 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.298 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:00:10.298 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.249029+0000", 2024-02-09T18:00:10.298 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:00:10.298 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.298 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.298 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:00:10.298 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.249029+0000", 2024-02-09T18:00:10.298 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:00:10.298 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.299 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.299 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:00:10.299 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.249031+0000", 2024-02-09T18:00:10.299 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4300000000000001e-06 2024-02-09T18:00:10.299 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.299 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.299 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:00:10.299 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.249036+0000", 2024-02-09T18:00:10.299 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4960000000000004e-06 2024-02-09T18:00:10.299 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.299 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.299 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:00:10.299 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.249053+0000", 2024-02-09T18:00:10.299 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6818000000000001e-05 2024-02-09T18:00:10.299 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.299 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.300 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:00:10.300 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.249062+0000", 2024-02-09T18:00:10.300 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.9069999999999996e-06 2024-02-09T18:00:10.300 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.300 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.300 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:00:10.300 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.249172+0000", 2024-02-09T18:00:10.300 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000110252 2024-02-09T18:00:10.300 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.300 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.300 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:00:10.300 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.249292+0000", 2024-02-09T18:00:10.300 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000119984 2024-02-09T18:00:10.300 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.301 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.301 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4", 2024-02-09T18:00:10.301 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.249410+0000", 2024-02-09T18:00:10.301 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000117864 2024-02-09T18:00:10.301 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.301 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.301 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:00:10.301 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.253682+0000", 2024-02-09T18:00:10.301 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0042714939999999998 2024-02-09T18:00:10.301 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.301 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.301 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:00:10.301 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.256181+0000", 2024-02-09T18:00:10.301 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0024988789999999999 2024-02-09T18:00:10.301 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.302 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.302 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:00:10.302 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.399776+0000", 2024-02-09T18:00:10.302 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14359556400000001 2024-02-09T18:00:10.302 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.302 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.302 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:00:10.302 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.399835+0000", 2024-02-09T18:00:10.302 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8817999999999998e-05 2024-02-09T18:00:10.302 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.302 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.302 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:00:10.302 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:44.399877+0000", 2024-02-09T18:00:10.302 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1783999999999998e-05 2024-02-09T18:00:10.302 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:00:10.303 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:00:10.303 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:00:10.303 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.303 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.303 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:236077 8.b 8:d02b57f2:::benchmark_data_smithi179_148205_object113005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e863)", 2024-02-09T18:00:10.303 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:59:26.606600+0000", 2024-02-09T18:00:10.303 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 43.586227530000002, 2024-02-09T18:00:10.303 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6539503770000001, 2024-02-09T18:00:10.303 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:00:10.303 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:00:10.303 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:00:10.303 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:00:10.303 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:00:10.303 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 236077 2024-02-09T18:00:10.303 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.304 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:00:10.304 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.304 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:00:10.304 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.606600+0000", 2024-02-09T18:00:10.304 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:00:10.304 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.304 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.304 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:00:10.304 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.606600+0000", 2024-02-09T18:00:10.304 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:00:10.304 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.304 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.304 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:00:10.304 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.606602+0000", 2024-02-09T18:00:10.305 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4389999999999999e-06 2024-02-09T18:00:10.305 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.305 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.305 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:00:10.305 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.606609+0000", 2024-02-09T18:00:10.305 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7139999999999996e-06 2024-02-09T18:00:10.305 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.305 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.305 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:00:10.305 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.606625+0000", 2024-02-09T18:00:10.305 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6365999999999999e-05 2024-02-09T18:00:10.305 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.305 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.306 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:00:10.306 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.606634+0000", 2024-02-09T18:00:10.306 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.3370000000000006e-06 2024-02-09T18:00:10.306 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.306 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.306 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:00:10.306 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.606693+0000", 2024-02-09T18:00:10.306 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9222000000000003e-05 2024-02-09T18:00:10.306 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.306 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.306 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:00:10.306 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.606703+0000", 2024-02-09T18:00:10.306 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.8060000000000006e-06 2024-02-09T18:00:10.306 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.307 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.307 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:00:10.307 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.006088+0000", 2024-02-09T18:00:10.307 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.399384348 2024-02-09T18:00:10.307 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.307 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.307 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:00:10.307 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.006102+0000", 2024-02-09T18:00:10.307 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4442e-05 2024-02-09T18:00:10.307 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.307 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.307 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:00:10.307 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.238871+0000", 2024-02-09T18:00:10.307 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2327688619999999 2024-02-09T18:00:10.307 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.308 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.308 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:00:10.308 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.239029+0000", 2024-02-09T18:00:10.308 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000158049 2024-02-09T18:00:10.308 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.308 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.308 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4,11", 2024-02-09T18:00:10.308 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.239206+0000", 2024-02-09T18:00:10.308 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00017735599999999999 2024-02-09T18:00:10.308 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.308 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.308 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:00:10.308 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.244222+0000", 2024-02-09T18:00:10.308 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00501565 2024-02-09T18:00:10.309 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.309 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.309 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:00:10.309 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.244302+0000", 2024-02-09T18:00:10.309 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.0358000000000005e-05 2024-02-09T18:00:10.309 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.309 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.309 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:00:10.309 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.249476+0000", 2024-02-09T18:00:10.309 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0051733550000000001 2024-02-09T18:00:10.309 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.309 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.309 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:00:10.309 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.260367+0000", 2024-02-09T18:00:10.309 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010891171999999999 2024-02-09T18:00:10.310 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.310 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.310 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:00:10.310 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.260433+0000", 2024-02-09T18:00:10.310 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6156999999999997e-05 2024-02-09T18:00:10.310 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.310 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.310 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:00:10.310 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.260551+0000", 2024-02-09T18:00:10.310 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000117744 2024-02-09T18:00:10.310 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:00:10.310 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:00:10.310 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:00:10.310 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.310 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.310 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:237144 8.b 8:d410fa72:::benchmark_data_smithi179_148205_object114072:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e867)", 2024-02-09T18:00:10.311 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:59:30.150092+0000", 2024-02-09T18:00:10.311 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 40.042736173000002, 2024-02-09T18:00:10.311 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.31223010400000001, 2024-02-09T18:00:10.311 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:00:10.311 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:00:10.311 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:00:10.311 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:00:10.311 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:00:10.311 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 237144 2024-02-09T18:00:10.311 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.311 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:00:10.311 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.311 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:00:10.311 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.150092+0000", 2024-02-09T18:00:10.312 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:00:10.312 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.312 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.312 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:00:10.312 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.150092+0000", 2024-02-09T18:00:10.312 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:00:10.312 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.312 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.312 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:00:10.312 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.150094+0000", 2024-02-09T18:00:10.312 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9230000000000001e-06 2024-02-09T18:00:10.312 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.312 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.312 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:00:10.312 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.150102+0000", 2024-02-09T18:00:10.313 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2060000000000007e-06 2024-02-09T18:00:10.313 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.313 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.313 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:00:10.313 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.150122+0000", 2024-02-09T18:00:10.313 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9930999999999999e-05 2024-02-09T18:00:10.313 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.313 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.313 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:00:10.313 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.150141+0000", 2024-02-09T18:00:10.313 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9142000000000001e-05 2024-02-09T18:00:10.313 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.313 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.313 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:00:10.314 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.150214+0000", 2024-02-09T18:00:10.314 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2846e-05 2024-02-09T18:00:10.314 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.314 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.314 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:00:10.314 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.150364+0000", 2024-02-09T18:00:10.314 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015035600000000001 2024-02-09T18:00:10.314 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.314 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.314 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4,11", 2024-02-09T18:00:10.314 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.151273+0000", 2024-02-09T18:00:10.314 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00090894200000000004 2024-02-09T18:00:10.314 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.314 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.315 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:00:10.315 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.156327+0000", 2024-02-09T18:00:10.315 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00505376 2024-02-09T18:00:10.315 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.315 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.315 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:00:10.315 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.156343+0000", 2024-02-09T18:00:10.315 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6002e-05 2024-02-09T18:00:10.315 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.315 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.315 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:00:10.315 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.162202+0000", 2024-02-09T18:00:10.315 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0058588900000000003 2024-02-09T18:00:10.315 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.315 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.316 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:00:10.316 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.462198+0000", 2024-02-09T18:00:10.316 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.29999659200000001 2024-02-09T18:00:10.316 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.316 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.316 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:00:10.316 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.462246+0000", 2024-02-09T18:00:10.316 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8016999999999998e-05 2024-02-09T18:00:10.316 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.316 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.316 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:00:10.316 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.462322+0000", 2024-02-09T18:00:10.316 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5496999999999999e-05 2024-02-09T18:00:10.317 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:00:10.317 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:00:10.317 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:00:10.317 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.317 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.317 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:237146 8.b 8:d6785256:::benchmark_data_smithi179_148205_object114074:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e867)", 2024-02-09T18:00:10.317 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:59:30.156416+0000", 2024-02-09T18:00:10.317 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 40.036411833999999, 2024-02-09T18:00:10.317 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.30608889900000003, 2024-02-09T18:00:10.317 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:00:10.317 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:00:10.317 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:00:10.317 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:00:10.318 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:00:10.318 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 237146 2024-02-09T18:00:10.318 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.318 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:00:10.318 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.318 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:00:10.318 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.156416+0000", 2024-02-09T18:00:10.318 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:00:10.318 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.318 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.318 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:00:10.318 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.156416+0000", 2024-02-09T18:00:10.318 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:00:10.318 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.319 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.319 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:00:10.319 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.156417+0000", 2024-02-09T18:00:10.319 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4499999999999997e-07 2024-02-09T18:00:10.319 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.319 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.319 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:00:10.319 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.156421+0000", 2024-02-09T18:00:10.319 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8820000000000003e-06 2024-02-09T18:00:10.319 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.319 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.319 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:00:10.319 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.156428+0000", 2024-02-09T18:00:10.320 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1860000000000003e-06 2024-02-09T18:00:10.320 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.320 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.320 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:00:10.320 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.156433+0000", 2024-02-09T18:00:10.320 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9289999999999997e-06 2024-02-09T18:00:10.320 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.320 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.320 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:00:10.320 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.156469+0000", 2024-02-09T18:00:10.320 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6652000000000001e-05 2024-02-09T18:00:10.320 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.320 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.320 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:00:10.321 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.156556+0000", 2024-02-09T18:00:10.321 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.6669000000000007e-05 2024-02-09T18:00:10.321 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.321 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.321 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4,11", 2024-02-09T18:00:10.321 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.156645+0000", 2024-02-09T18:00:10.321 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.9404e-05 2024-02-09T18:00:10.321 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.321 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.321 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:00:10.321 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.160816+0000", 2024-02-09T18:00:10.321 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0041709540000000002 2024-02-09T18:00:10.321 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.321 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.321 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:00:10.322 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.161215+0000", 2024-02-09T18:00:10.322 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00039865899999999999 2024-02-09T18:00:10.322 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.322 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.322 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:00:10.322 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.162258+0000", 2024-02-09T18:00:10.322 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0010426389999999999 2024-02-09T18:00:10.322 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.322 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.322 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:00:10.322 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.462359+0000", 2024-02-09T18:00:10.322 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.30010168399999998 2024-02-09T18:00:10.322 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.322 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.322 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:00:10.323 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.462404+0000", 2024-02-09T18:00:10.323 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4907999999999998e-05 2024-02-09T18:00:10.323 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.323 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.323 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:00:10.323 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.462505+0000", 2024-02-09T18:00:10.323 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000100688 2024-02-09T18:00:10.323 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:00:10.323 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:00:10.323 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:00:10.323 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.323 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.323 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:237151 8.b 8:d3f4c7c9:::benchmark_data_smithi179_148205_object114079:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e867)", 2024-02-09T18:00:10.323 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:59:30.174193+0000", 2024-02-09T18:00:10.323 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 40.018635209999999, 2024-02-09T18:00:10.324 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.28857026400000002, 2024-02-09T18:00:10.324 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:00:10.324 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:00:10.324 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:00:10.324 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:00:10.324 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:00:10.324 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 237151 2024-02-09T18:00:10.324 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.324 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:00:10.324 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.324 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:00:10.324 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.174193+0000", 2024-02-09T18:00:10.324 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:00:10.324 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.324 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.325 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:00:10.325 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.174193+0000", 2024-02-09T18:00:10.325 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:00:10.325 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.325 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.325 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:00:10.325 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.174194+0000", 2024-02-09T18:00:10.325 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7570000000000001e-06 2024-02-09T18:00:10.325 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.325 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.325 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:00:10.325 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.174202+0000", 2024-02-09T18:00:10.325 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5009999999999998e-06 2024-02-09T18:00:10.325 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.325 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.326 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:00:10.326 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.174218+0000", 2024-02-09T18:00:10.326 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6303e-05 2024-02-09T18:00:10.326 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.326 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.326 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:00:10.326 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.174226+0000", 2024-02-09T18:00:10.326 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1210000000000007e-06 2024-02-09T18:00:10.326 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.326 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.326 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:00:10.326 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.174281+0000", 2024-02-09T18:00:10.326 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4531999999999999e-05 2024-02-09T18:00:10.327 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.327 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.327 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:00:10.327 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.174426+0000", 2024-02-09T18:00:10.327 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014521799999999999 2024-02-09T18:00:10.327 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.327 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.327 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4,11", 2024-02-09T18:00:10.327 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.174522+0000", 2024-02-09T18:00:10.327 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6223000000000004e-05 2024-02-09T18:00:10.327 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.327 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.327 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:00:10.327 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.179694+0000", 2024-02-09T18:00:10.327 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0051714029999999998 2024-02-09T18:00:10.327 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.328 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.328 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:00:10.328 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.179755+0000", 2024-02-09T18:00:10.328 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1410999999999999e-05 2024-02-09T18:00:10.328 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.328 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.328 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:00:10.328 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.179834+0000", 2024-02-09T18:00:10.328 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8700000000000002e-05 2024-02-09T18:00:10.328 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.328 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.328 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:00:10.328 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.462589+0000", 2024-02-09T18:00:10.329 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.28275532799999997 2024-02-09T18:00:10.329 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.329 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.329 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:00:10.329 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.462648+0000", 2024-02-09T18:00:10.329 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8649e-05 2024-02-09T18:00:10.329 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:00:10.329 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:00:10.329 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:00:10.329 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.462763+0000", 2024-02-09T18:00:10.329 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000115118 2024-02-09T18:00:10.329 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:00:10.329 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:00:10.329 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:00:10.329 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:00:10.330 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:00:10.330 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T18:00:10.353 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:00:10.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:10.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:00:10.459 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:00:10.487 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:10.508 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:00:10.508 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:00:10.515 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:00:10.517 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:10 smithi099 ceph-mon[39420]: pgmap v2186: 47 pgs: 3 creating+activating, 7 unknown, 37 active+clean; 417 KiB data, 8.4 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:00:10.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:10.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:00:10.618 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:10 smithi179 ceph-mon[43975]: pgmap v2186: 47 pgs: 3 creating+activating, 7 unknown, 37 active+clean; 417 KiB data, 8.4 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:00:10.619 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T18:00:10.649 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:10.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:10 smithi162 ceph-mon[44304]: pgmap v2186: 47 pgs: 3 creating+activating, 7 unknown, 37 active+clean; 417 KiB data, 8.4 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:00:10.949 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-09T18:00:10.999 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T18:00:11.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:11.002 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T18:00:11.103 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:00:11.133 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:11.262 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.6 to 1 (10000) 2024-02-09T18:00:11.263 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is unset 2024-02-09T18:00:11.285 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:11 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4006328656' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-09T18:00:11.286 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:11 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4240020584' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T18:00:11.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:11 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4006328656' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-09T18:00:11.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:11 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4240020584' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T18:00:11.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:11 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4006328656' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-09T18:00:11.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:11 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4240020584' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T18:00:11.783 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:00:11.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:11.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:00:11.887 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:00:11.917 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:12.066 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:00:12.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:12.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:00:12.170 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:00:12.203 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:12.225 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:00:12.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:12.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:00:12.329 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:00:12.354 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:12 smithi099 ceph-mon[39420]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:00:12.354 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:12 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4006328656' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-09T18:00:12.354 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:12 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4240020584' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T18:00:12.354 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:12 smithi099 ceph-mon[39420]: osdmap e885: 12 total, 12 up, 11 in 2024-02-09T18:00:12.354 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:12 smithi099 ceph-mon[39420]: pgmap v2188: 47 pgs: 3 creating+activating, 44 active+clean; 417 KiB data, 8.4 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:00:12.356 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:12.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:12 smithi179 ceph-mon[43975]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:00:12.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:12 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4006328656' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-09T18:00:12.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:12 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4240020584' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T18:00:12.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:12 smithi179 ceph-mon[43975]: osdmap e885: 12 total, 12 up, 11 in 2024-02-09T18:00:12.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:12 smithi179 ceph-mon[43975]: pgmap v2188: 47 pgs: 3 creating+activating, 44 active+clean; 417 KiB data, 8.4 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:00:12.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:12 smithi162 ceph-mon[44304]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:00:12.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:12 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4006328656' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-09T18:00:12.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:12 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4240020584' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T18:00:12.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:12 smithi162 ceph-mon[44304]: osdmap e885: 12 total, 12 up, 11 in 2024-02-09T18:00:12.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:12 smithi162 ceph-mon[44304]: pgmap v2188: 47 pgs: 3 creating+activating, 44 active+clean; 417 KiB data, 8.4 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:00:13.106 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:00:13.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:13.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:00:13.210 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:00:13.237 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:13.359 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:00:13.362 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:13.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:00:13.464 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:00:13.499 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:13.520 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:00:13.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:13.524 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:00:13.625 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:00:13.658 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:13.680 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:00:13.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:13.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:00:13.784 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T18:00:13.816 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:13.838 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T18:00:13.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:13.841 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T18:00:13.942 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:00:13.971 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:14.086 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 7 1 2024-02-09T18:00:14.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:14 smithi179 ceph-mon[43975]: pgmap v2189: 47 pgs: 47 active+clean; 417 KiB data, 8.4 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:00:14.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:14 smithi162 ceph-mon[44304]: pgmap v2189: 47 pgs: 47 active+clean; 417 KiB data, 8.4 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:00:14.740 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:14 smithi099 ceph-mon[39420]: pgmap v2189: 47 pgs: 47 active+clean; 417 KiB data, 8.4 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:00:15.097 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-09T18:00:15.143 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:00:15.150 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:00:15.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:15.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:00:15.255 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:00:15.288 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:15.310 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:00:15.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:15.313 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:00:15.323 INFO:teuthology.orchestra.run.smithi099.stderr:enabled application 'rados' on pool 'unique_pool_7' 2024-02-09T18:00:15.391 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T18:00:15.412 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T18:00:15.414 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:00:15.434 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T18:00:15.440 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:15 smithi099 ceph-mon[39420]: 1.c scrub starts 2024-02-09T18:00:15.441 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:15 smithi099 ceph-mon[39420]: 1.c scrub ok 2024-02-09T18:00:15.441 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:15 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1011036336' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-09T18:00:15.443 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:15.464 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T18:00:15.484 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T18:00:15.504 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T18:00:15.524 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T18:00:15.546 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T18:00:15.566 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T18:00:15.586 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T18:00:15.606 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T18:00:15.626 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T18:00:15.628 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T18:00:15.628 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T18:00:15.628 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T18:00:15.628 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T18:00:15.628 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T18:00:15.628 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T18:00:15.628 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T18:00:15.628 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T18:00:15.628 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T18:00:15.628 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T18:00:15.628 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T18:00:15.629 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T18:00:15.647 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:00:15.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:15.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:00:15.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:15 smithi179 ceph-mon[43975]: 1.c scrub starts 2024-02-09T18:00:15.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:15 smithi179 ceph-mon[43975]: 1.c scrub ok 2024-02-09T18:00:15.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:15 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1011036336' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-09T18:00:15.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:15 smithi162 ceph-mon[44304]: 1.c scrub starts 2024-02-09T18:00:15.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:15 smithi162 ceph-mon[44304]: 1.c scrub ok 2024-02-09T18:00:15.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:15 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1011036336' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-09T18:00:15.753 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:00:15.785 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:15.808 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:00:15.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:15.811 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:00:15.913 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:00:15.941 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:16.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:16 smithi179 ceph-mon[43975]: pgmap v2190: 47 pgs: 47 active+clean; 417 KiB data, 8.4 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:00:16.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:16 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1011036336' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-09T18:00:16.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:16 smithi179 ceph-mon[43975]: osdmap e886: 12 total, 12 up, 11 in 2024-02-09T18:00:16.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:16 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3407523912' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:00:16.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:16 smithi179 ceph-mon[43975]: 1.12 scrub starts 2024-02-09T18:00:16.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:16 smithi179 ceph-mon[43975]: 1.12 scrub ok 2024-02-09T18:00:16.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:16 smithi162 ceph-mon[44304]: pgmap v2190: 47 pgs: 47 active+clean; 417 KiB data, 8.4 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:00:16.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:16 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1011036336' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-09T18:00:16.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:16 smithi162 ceph-mon[44304]: osdmap e886: 12 total, 12 up, 11 in 2024-02-09T18:00:16.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:16 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3407523912' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:00:16.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:16 smithi162 ceph-mon[44304]: 1.12 scrub starts 2024-02-09T18:00:16.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:16 smithi162 ceph-mon[44304]: 1.12 scrub ok 2024-02-09T18:00:16.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:16 smithi099 ceph-mon[39420]: pgmap v2190: 47 pgs: 47 active+clean; 417 KiB data, 8.4 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:00:16.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:16 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1011036336' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-09T18:00:16.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:16 smithi099 ceph-mon[39420]: osdmap e886: 12 total, 12 up, 11 in 2024-02-09T18:00:16.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:16 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3407523912' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:00:16.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:16 smithi099 ceph-mon[39420]: 1.12 scrub starts 2024-02-09T18:00:16.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:16 smithi099 ceph-mon[39420]: 1.12 scrub ok 2024-02-09T18:00:17.290 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:00:17.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:17.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:00:17.394 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:00:17.423 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:17.530 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:00:17.537 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:17 smithi099 ceph-mon[39420]: 1.11 scrub starts 2024-02-09T18:00:17.537 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:17 smithi099 ceph-mon[39420]: 1.11 scrub ok 2024-02-09T18:00:17.537 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:17 smithi099 ceph-mon[39420]: osdmap e887: 12 total, 12 up, 11 in 2024-02-09T18:00:17.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:17.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:00:17.617 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T18:00:17.618 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T18:00:17.639 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:00:17.667 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:17.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:17 smithi179 ceph-mon[43975]: 1.11 scrub starts 2024-02-09T18:00:17.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:17 smithi179 ceph-mon[43975]: 1.11 scrub ok 2024-02-09T18:00:17.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:17 smithi179 ceph-mon[43975]: osdmap e887: 12 total, 12 up, 11 in 2024-02-09T18:00:17.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:17 smithi162 ceph-mon[44304]: 1.11 scrub starts 2024-02-09T18:00:17.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:17 smithi162 ceph-mon[44304]: 1.11 scrub ok 2024-02-09T18:00:17.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:17 smithi162 ceph-mon[44304]: osdmap e887: 12 total, 12 up, 11 in 2024-02-09T18:00:17.815 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:00:17.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:17.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:00:17.919 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T18:00:17.946 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:17.964 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T18:00:17.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:17.967 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T18:00:18.068 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:00:18.095 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:18.115 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:00:18.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:18.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:00:18.219 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:00:18.246 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:18.264 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:00:18.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:18.267 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:00:18.368 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:00:18.393 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:18 smithi099 ceph-mon[39420]: pgmap v2193: 47 pgs: 47 active+clean; 577 KiB data, 8.4 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:00:18.393 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:18 smithi099 ceph-mon[39420]: 1.a scrub starts 2024-02-09T18:00:18.393 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:18 smithi099 ceph-mon[39420]: 1.a scrub ok 2024-02-09T18:00:18.394 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:18 smithi099 ceph-mon[39420]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-09T18:00:18.395 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:18.493 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:00:18.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:18.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:00:18.598 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:00:18.622 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:18 smithi179 ceph-mon[43975]: pgmap v2193: 47 pgs: 47 active+clean; 577 KiB data, 8.4 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:00:18.622 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:18 smithi179 ceph-mon[43975]: 1.a scrub starts 2024-02-09T18:00:18.622 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:18 smithi179 ceph-mon[43975]: 1.a scrub ok 2024-02-09T18:00:18.623 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:18 smithi179 ceph-mon[43975]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-09T18:00:18.624 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:18.643 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:00:18.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:18.646 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:00:18.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:18 smithi162 ceph-mon[44304]: pgmap v2193: 47 pgs: 47 active+clean; 577 KiB data, 8.4 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:00:18.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:18 smithi162 ceph-mon[44304]: 1.a scrub starts 2024-02-09T18:00:18.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:18 smithi162 ceph-mon[44304]: 1.a scrub ok 2024-02-09T18:00:18.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:18 smithi162 ceph-mon[44304]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-09T18:00:18.747 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:00:18.774 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:18.897 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:00:18.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:18.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:00:18.901 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-09T18:00:19.002 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:00:19.028 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:19.399 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:00:19.445 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:00:19.518 DEBUG:teuthology.orchestra.run.smithi179:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_7 bench 90 write' 2024-02-09T18:00:19.521 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-02-09T18:00:19.521 INFO:teuthology.orchestra.run:waiting for 3000 2024-02-09T18:00:19.575 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:hints = 1 2024-02-09T18:00:19.575 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-02-09T18:00:19.575 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Object prefix: benchmark_data_smithi179_161915 2024-02-09T18:00:19.575 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-09T18:00:19.575 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 0 0 0 0 0 0 - 0 2024-02-09T18:00:19.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:19 smithi179 ceph-mon[43975]: 1.17 scrub starts 2024-02-09T18:00:19.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:19 smithi179 ceph-mon[43975]: 1.17 scrub ok 2024-02-09T18:00:19.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:19 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:00:19.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:19 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:00:19.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:19 smithi162 ceph-mon[44304]: 1.17 scrub starts 2024-02-09T18:00:19.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:19 smithi162 ceph-mon[44304]: 1.17 scrub ok 2024-02-09T18:00:19.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:19 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:00:19.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:19 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:00:19.677 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:00:19.677 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:19 smithi099 ceph-mon[39420]: 1.17 scrub starts 2024-02-09T18:00:19.678 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:19 smithi099 ceph-mon[39420]: 1.17 scrub ok 2024-02-09T18:00:19.678 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:19 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:00:19.678 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:19 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:00:19.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:19.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:00:19.683 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:00:19.782 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:00:19.827 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:19.852 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:00:19.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:19.856 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:00:19.957 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:00:20.004 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:20.088 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:00:20.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:20.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:00:20.192 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T18:00:20.226 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:20.252 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T18:00:20.256 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:20.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T18:00:20.357 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T18:00:20.388 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:20.416 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T18:00:20.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:20.419 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T18:00:20.520 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T18:00:20.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:20 smithi162 ceph-mon[44304]: 1.1b scrub starts 2024-02-09T18:00:20.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:20 smithi162 ceph-mon[44304]: 1.1b scrub ok 2024-02-09T18:00:20.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:20 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:00:20.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:20 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:00:20.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:20 smithi162 ceph-mon[44304]: pgmap v2194: 47 pgs: 47 active+clean; 577 KiB data, 8.4 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:00:20.554 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:20 smithi162 ceph-mon[44304]: osdmap e888: 12 total, 12 up, 11 in 2024-02-09T18:00:20.556 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:20.575 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 1 16 3727 3711 231.852 231.938 0.00426558 0.00430226 2024-02-09T18:00:20.582 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T18:00:20.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:20.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T18:00:20.589 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:20 smithi099 ceph-mon[39420]: 1.1b scrub starts 2024-02-09T18:00:20.589 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:20 smithi099 ceph-mon[39420]: 1.1b scrub ok 2024-02-09T18:00:20.589 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:20 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:00:20.589 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:20 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:00:20.589 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:20 smithi099 ceph-mon[39420]: pgmap v2194: 47 pgs: 47 active+clean; 577 KiB data, 8.4 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:00:20.590 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:20 smithi099 ceph-mon[39420]: osdmap e888: 12 total, 12 up, 11 in 2024-02-09T18:00:20.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:20 smithi179 ceph-mon[43975]: 1.1b scrub starts 2024-02-09T18:00:20.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:20 smithi179 ceph-mon[43975]: 1.1b scrub ok 2024-02-09T18:00:20.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:20 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:00:20.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:20 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:00:20.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:20 smithi179 ceph-mon[43975]: pgmap v2194: 47 pgs: 47 active+clean; 577 KiB data, 8.4 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:00:20.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:20 smithi179 ceph-mon[43975]: osdmap e888: 12 total, 12 up, 11 in 2024-02-09T18:00:20.687 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T18:00:20.724 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:20.749 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T18:00:20.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:20.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T18:00:20.854 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:00:20.895 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:20.927 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:00:20.931 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:20.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:00:21.033 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:00:21.074 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:21.439 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T18:00:21.575 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 2 16 7661 7645 238.852 245.875 0.00304489 0.00417867 2024-02-09T18:00:21.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:21 smithi179 ceph-mon[43975]: 1.1c scrub starts 2024-02-09T18:00:21.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:21 smithi179 ceph-mon[43975]: 1.1c scrub ok 2024-02-09T18:00:21.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:21 smithi179 ceph-mon[43975]: 1.8 scrub starts 2024-02-09T18:00:21.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:21 smithi179 ceph-mon[43975]: 1.8 scrub ok 2024-02-09T18:00:21.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:21 smithi179 ceph-mon[43975]: 1.13 scrub starts 2024-02-09T18:00:21.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:21 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:00:21.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:21 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:00:21.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:21 smithi179 ceph-mon[43975]: osdmap e889: 12 total, 12 up, 11 in 2024-02-09T18:00:21.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:21 smithi179 ceph-mon[43975]: 1.13 scrub ok 2024-02-09T18:00:21.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:21 smithi162 ceph-mon[44304]: 1.1c scrub starts 2024-02-09T18:00:21.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:21 smithi162 ceph-mon[44304]: 1.1c scrub ok 2024-02-09T18:00:21.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:21 smithi162 ceph-mon[44304]: 1.8 scrub starts 2024-02-09T18:00:21.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:21 smithi162 ceph-mon[44304]: 1.8 scrub ok 2024-02-09T18:00:21.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:21 smithi162 ceph-mon[44304]: 1.13 scrub starts 2024-02-09T18:00:21.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:21 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:00:21.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:21 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:00:21.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:21 smithi162 ceph-mon[44304]: osdmap e889: 12 total, 12 up, 11 in 2024-02-09T18:00:21.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:21 smithi162 ceph-mon[44304]: 1.13 scrub ok 2024-02-09T18:00:21.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:21 smithi099 ceph-mon[39420]: 1.1c scrub starts 2024-02-09T18:00:21.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:21 smithi099 ceph-mon[39420]: 1.1c scrub ok 2024-02-09T18:00:21.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:21 smithi099 ceph-mon[39420]: 1.8 scrub starts 2024-02-09T18:00:21.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:21 smithi099 ceph-mon[39420]: 1.8 scrub ok 2024-02-09T18:00:21.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:21 smithi099 ceph-mon[39420]: 1.13 scrub starts 2024-02-09T18:00:21.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:21 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:00:21.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:21 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:00:21.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:21 smithi099 ceph-mon[39420]: osdmap e889: 12 total, 12 up, 11 in 2024-02-09T18:00:21.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:21 smithi099 ceph-mon[39420]: 1.13 scrub ok 2024-02-09T18:00:22.576 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 3 16 10535 10519 219.052 179.625 0.00394621 0.0045568 2024-02-09T18:00:22.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:22 smithi179 ceph-mon[43975]: 1.18 scrub starts 2024-02-09T18:00:22.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:22 smithi179 ceph-mon[43975]: 1.18 scrub ok 2024-02-09T18:00:22.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:22 smithi179 ceph-mon[43975]: pgmap v2197: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 98 MiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 16 MiB/s wr, 261 op/s 2024-02-09T18:00:22.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:22 smithi162 ceph-mon[44304]: 1.18 scrub starts 2024-02-09T18:00:22.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:22 smithi162 ceph-mon[44304]: 1.18 scrub ok 2024-02-09T18:00:22.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:22 smithi162 ceph-mon[44304]: pgmap v2197: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 98 MiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 16 MiB/s wr, 261 op/s 2024-02-09T18:00:22.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:22 smithi099 ceph-mon[39420]: 1.18 scrub starts 2024-02-09T18:00:22.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:22 smithi099 ceph-mon[39420]: 1.18 scrub ok 2024-02-09T18:00:22.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:22 smithi099 ceph-mon[39420]: pgmap v2197: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 98 MiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 16 MiB/s wr, 261 op/s 2024-02-09T18:00:23.576 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 4 16 13997 13981 218.379 216.375 0.00405578 0.00457455 2024-02-09T18:00:24.335 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:00:24.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:24.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:00:24.441 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T18:00:24.474 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:24.498 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T18:00:24.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:24.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T18:00:24.576 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 5 16 16984 16968 212.034 186.688 0.00142663 0.00459819 2024-02-09T18:00:24.602 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:00:24.638 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:24 smithi099 ceph-mon[39420]: pgmap v2198: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 294 MiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 47 MiB/s wr, 751 op/s 2024-02-09T18:00:24.640 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:24.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:24 smithi179 ceph-mon[43975]: pgmap v2198: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 294 MiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 47 MiB/s wr, 751 op/s 2024-02-09T18:00:24.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:24 smithi162 ceph-mon[44304]: pgmap v2198: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 294 MiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 47 MiB/s wr, 751 op/s 2024-02-09T18:00:25.504 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:00:25.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:25.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:00:25.576 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 6 16 19857 19841 206.621 179.562 0.00300726 0.00483349 2024-02-09T18:00:25.611 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:00:25.657 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:25.711 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:00:25.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:25.721 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:00:25.803 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:00:25.822 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:00:25.853 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:25.879 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:00:25.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:25.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:00:25.985 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:00:26.031 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:26.059 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:00:26.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:26.063 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:00:26.164 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:00:26.197 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:26.223 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:00:26.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:26.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:00:26.328 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:00:26.359 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:26.577 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 7 16 22198 22182 198.005 146.312 0.00292589 0.00504474 2024-02-09T18:00:26.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:26 smithi179 ceph-mon[43975]: pgmap v2199: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 522 MiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-02-09T18:00:26.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:26 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:00:26.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:26 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:00:26.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:26 smithi162 ceph-mon[44304]: pgmap v2199: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 522 MiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-02-09T18:00:26.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:26 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:00:26.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:26 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:00:26.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:26 smithi099 ceph-mon[39420]: pgmap v2199: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 522 MiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-02-09T18:00:26.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:26 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:00:26.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:26 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:00:27.577 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 8 16 25034 25018 195.41 177.25 0.00314967 0.005113 2024-02-09T18:00:27.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:27 smithi179 ceph-mon[43975]: osdmap e890: 12 total, 12 up, 11 in 2024-02-09T18:00:27.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:27 smithi179 ceph-mon[43975]: osdmap e891: 12 total, 12 up, 11 in 2024-02-09T18:00:27.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:27 smithi162 ceph-mon[44304]: osdmap e890: 12 total, 12 up, 11 in 2024-02-09T18:00:27.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:27 smithi162 ceph-mon[44304]: osdmap e891: 12 total, 12 up, 11 in 2024-02-09T18:00:27.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:27 smithi099 ceph-mon[39420]: osdmap e890: 12 total, 12 up, 11 in 2024-02-09T18:00:27.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:27 smithi099 ceph-mon[39420]: osdmap e891: 12 total, 12 up, 11 in 2024-02-09T18:00:28.384 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.7 to 1 (10000) 2024-02-09T18:00:28.577 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 9 16 27620 27604 191.655 161.625 0.00484486 0.00521368 2024-02-09T18:00:28.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:28 smithi179 ceph-mon[43975]: pgmap v2202: 47 pgs: 47 active+clean; 1.0 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 148 MiB/s wr, 2.36k op/s 2024-02-09T18:00:28.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:28 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/770091034' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-09T18:00:28.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:28 smithi162 ceph-mon[44304]: pgmap v2202: 47 pgs: 47 active+clean; 1.0 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 148 MiB/s wr, 2.36k op/s 2024-02-09T18:00:28.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:28 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/770091034' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-09T18:00:28.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:28 smithi099 ceph-mon[39420]: pgmap v2202: 47 pgs: 47 active+clean; 1.0 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 148 MiB/s wr, 2.36k op/s 2024-02-09T18:00:28.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:28 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/770091034' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-09T18:00:29.579 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 10 16 29954 29938 187.076 145.875 0.00742589 0.00534083 2024-02-09T18:00:29.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:29 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/770091034' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-09T18:00:29.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:29 smithi179 ceph-mon[43975]: osdmap e892: 12 total, 12 up, 11 in 2024-02-09T18:00:29.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:29 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2704996447' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T18:00:29.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:29 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T18:00:29.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:29 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/770091034' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-09T18:00:29.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:29 smithi162 ceph-mon[44304]: osdmap e892: 12 total, 12 up, 11 in 2024-02-09T18:00:29.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:29 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2704996447' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T18:00:29.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:29 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T18:00:29.717 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:00:29.723 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:29 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/770091034' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-09T18:00:29.723 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:29 smithi099 ceph-mon[39420]: osdmap e892: 12 total, 12 up, 11 in 2024-02-09T18:00:29.723 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:29 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2704996447' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T18:00:29.724 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:29 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T18:00:29.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:29.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:00:29.826 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:00:29.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-02-09T18:00:28.044+0000 7f20d66c9700 0 --2- 172.21.15.99:0/2398622747 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f20d01015c0 0x7f20d010daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-02-09T18:00:29.839 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is unset 2024-02-09T18:00:29.870 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:30.331 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:00:30.339 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:30.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:00:30.441 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:00:30.485 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:30.531 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:00:30.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:30.536 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:00:30.577 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 11 16 32416 32400 184.057 153.875 0.00867132 0.00542803 2024-02-09T18:00:30.597 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:30 smithi099 ceph-mon[39420]: pgmap v2204: 47 pgs: 47 active+clean; 1.4 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 195 MiB/s wr, 3.12k op/s 2024-02-09T18:00:30.597 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:30 smithi099 ceph-mon[39420]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T18:00:30.597 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:30 smithi099 ceph-mon[39420]: Cluster is now healthy 2024-02-09T18:00:30.597 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:30 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T18:00:30.597 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:30 smithi099 ceph-mon[39420]: osdmap e893: 12 total, 12 up, 11 in 2024-02-09T18:00:30.597 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:30 smithi099 ceph-mon[39420]: 1.2 scrub starts 2024-02-09T18:00:30.597 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:30 smithi099 ceph-mon[39420]: 1.2 scrub ok 2024-02-09T18:00:30.637 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:00:30.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:30 smithi179 ceph-mon[43975]: pgmap v2204: 47 pgs: 47 active+clean; 1.4 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 195 MiB/s wr, 3.12k op/s 2024-02-09T18:00:30.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:30 smithi179 ceph-mon[43975]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T18:00:30.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:30 smithi179 ceph-mon[43975]: Cluster is now healthy 2024-02-09T18:00:30.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:30 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T18:00:30.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:30 smithi179 ceph-mon[43975]: osdmap e893: 12 total, 12 up, 11 in 2024-02-09T18:00:30.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:30 smithi179 ceph-mon[43975]: 1.2 scrub starts 2024-02-09T18:00:30.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:30 smithi179 ceph-mon[43975]: 1.2 scrub ok 2024-02-09T18:00:30.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:30 smithi162 ceph-mon[44304]: pgmap v2204: 47 pgs: 47 active+clean; 1.4 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 195 MiB/s wr, 3.12k op/s 2024-02-09T18:00:30.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:30 smithi162 ceph-mon[44304]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T18:00:30.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:30 smithi162 ceph-mon[44304]: Cluster is now healthy 2024-02-09T18:00:30.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:30 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T18:00:30.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:30 smithi162 ceph-mon[44304]: osdmap e893: 12 total, 12 up, 11 in 2024-02-09T18:00:30.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:30 smithi162 ceph-mon[44304]: 1.2 scrub starts 2024-02-09T18:00:30.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:30 smithi162 ceph-mon[44304]: 1.2 scrub ok 2024-02-09T18:00:30.679 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:31.461 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 8 1 2024-02-09T18:00:31.577 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 12 16 34440 34424 179.26 126.5 0.00636642 0.0055738 2024-02-09T18:00:31.639 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:00:31.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:31.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:00:31.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:31 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:00:31.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:31 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:00:31.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:31 smithi179 ceph-mon[43975]: osdmap e894: 12 total, 12 up, 11 in 2024-02-09T18:00:31.672 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:31 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:00:31.673 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:31 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:00:31.673 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:31 smithi099 ceph-mon[39420]: osdmap e894: 12 total, 12 up, 11 in 2024-02-09T18:00:31.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:31 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:00:31.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:31 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:00:31.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:31 smithi162 ceph-mon[44304]: osdmap e894: 12 total, 12 up, 11 in 2024-02-09T18:00:31.749 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:00:31.790 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:31.822 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:00:31.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:31.826 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:00:31.927 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:00:31.973 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:32.577 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 13 16 36769 36753 176.666 145.562 0.00928948 0.00565596 2024-02-09T18:00:32.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:32 smithi179 ceph-mon[43975]: pgmap v2207: 47 pgs: 47 active+clean; 1.8 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 198 MiB/s wr, 3.16k op/s 2024-02-09T18:00:32.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:32 smithi179 ceph-mon[43975]: osdmap e895: 12 total, 12 up, 11 in 2024-02-09T18:00:32.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:32 smithi162 ceph-mon[44304]: pgmap v2207: 47 pgs: 47 active+clean; 1.8 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 198 MiB/s wr, 3.16k op/s 2024-02-09T18:00:32.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:32 smithi162 ceph-mon[44304]: osdmap e895: 12 total, 12 up, 11 in 2024-02-09T18:00:32.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:32 smithi099 ceph-mon[39420]: pgmap v2207: 47 pgs: 47 active+clean; 1.8 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 198 MiB/s wr, 3.16k op/s 2024-02-09T18:00:32.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:32 smithi099 ceph-mon[39420]: osdmap e895: 12 total, 12 up, 11 in 2024-02-09T18:00:33.359 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:00:33.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:33.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:00:33.464 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T18:00:33.498 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:33.524 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T18:00:33.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:33.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T18:00:33.577 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 14 16 39061 39045 174.278 143.25 0.00579168 0.00573269 2024-02-09T18:00:33.630 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T18:00:33.665 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:33.673 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set noscrub 2024-02-09T18:00:33.689 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T18:00:33.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:33.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T18:00:33.794 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:00:33.839 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:33.868 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:00:33.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:33.873 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:00:33.974 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:00:34.011 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:34.578 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 15 16 41184 41168 171.502 132.688 0.00966877 0.00582682 2024-02-09T18:00:34.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:34 smithi099 ceph-mon[39420]: pgmap v2209: 47 pgs: 47 active+clean; 2.0 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 159 MiB/s wr, 2.54k op/s 2024-02-09T18:00:34.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:34 smithi179 ceph-mon[43975]: pgmap v2209: 47 pgs: 47 active+clean; 2.0 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 159 MiB/s wr, 2.54k op/s 2024-02-09T18:00:34.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:34 smithi162 ceph-mon[44304]: pgmap v2209: 47 pgs: 47 active+clean; 2.0 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 159 MiB/s wr, 2.54k op/s 2024-02-09T18:00:35.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:35 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1280200839' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:00:35.578 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 16 16 43283 43267 168.982 131.188 0.00321253 0.00591279 2024-02-09T18:00:35.811 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T18:00:35.811 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T18:00:35.811 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T18:00:35.811 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T18:00:35.811 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T18:00:35.811 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T18:00:35.811 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T18:00:35.811 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T18:00:35.811 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T18:00:35.811 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T18:00:35.811 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T18:00:35.812 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T18:00:35.812 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T18:00:35.812 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T18:00:35.812 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T18:00:35.812 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T18:00:35.812 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T18:00:35.812 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T18:00:35.812 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T18:00:35.812 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T18:00:35.812 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T18:00:35.812 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T18:00:35.813 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T18:00:35.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:35 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1280200839' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:00:35.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:35 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1280200839' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:00:36.578 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 17 16 44951 44935 165.174 104.25 0.00749447 0.00604989 2024-02-09T18:00:36.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:36 smithi099 ceph-mon[39420]: pgmap v2210: 47 pgs: 47 active+clean; 2.2 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 125 MiB/s wr, 2k op/s 2024-02-09T18:00:36.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:36 smithi179 ceph-mon[43975]: pgmap v2210: 47 pgs: 47 active+clean; 2.2 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 125 MiB/s wr, 2k op/s 2024-02-09T18:00:36.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:36 smithi162 ceph-mon[44304]: pgmap v2210: 47 pgs: 47 active+clean; 2.2 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 125 MiB/s wr, 2k op/s 2024-02-09T18:00:37.072 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:00:37.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:37.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:00:37.179 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:00:37.180 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T18:00:37.181 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T18:00:37.207 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:37.228 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:00:37.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:37.231 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:00:37.332 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:00:37.376 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:37.412 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:00:37.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:37.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:00:37.519 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:00:37.552 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:37.578 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 18 16 46744 46728 162.223 112.062 0.00489254 0.00616049 2024-02-09T18:00:37.584 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:00:37.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:37.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:00:37.688 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T18:00:37.729 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:37.762 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T18:00:37.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:37.766 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T18:00:37.868 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T18:00:37.905 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:37.931 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T18:00:37.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:37.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T18:00:38.035 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:38 smithi179 ceph-mon[43975]: pgmap v2211: 47 pgs: 47 active+clean; 2.5 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 138 MiB/s wr, 2.21k op/s 2024-02-09T18:00:38.035 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:00:38.075 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:38.103 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:00:38.106 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:38.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:00:38.157 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-09T18:00:38.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:38 smithi162 ceph-mon[44304]: pgmap v2211: 47 pgs: 47 active+clean; 2.5 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 138 MiB/s wr, 2.21k op/s 2024-02-09T18:00:38.208 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:00:38.240 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:38.265 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:00:38.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:38.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:00:38.371 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:00:38.409 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:38 smithi099 ceph-mon[39420]: pgmap v2211: 47 pgs: 47 active+clean; 2.5 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 138 MiB/s wr, 2.21k op/s 2024-02-09T18:00:38.417 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:38.459 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:00:38.465 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:00:38.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:38.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:00:38.568 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:00:38.578 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 19 16 48660 48644 159.987 119.75 0.00807002 0.00624618 2024-02-09T18:00:38.607 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:38.743 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:00:38.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:38.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:00:38.847 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:00:38.895 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:38.997 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:00:39.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:39.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:00:39.103 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:00:39.145 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:39.177 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:00:39.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:39.184 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:00:39.285 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:00:39.330 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:39.404 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:00:39.409 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:00:39.412 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:00:39.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:39.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:00:39.523 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:00:39.567 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:39.578 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:2024-02-09T18:00:39.577790+0000 min lat: 0.00108495 max lat: 0.211521 avg lat: 0.00632423 2024-02-09T18:00:39.578 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-09T18:00:39.578 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 20 16 50593 50577 158.028 120.812 0.0066749 0.00632423 2024-02-09T18:00:40.228 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:00:40.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:40.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:00:40.334 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:00:40.385 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:40.422 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:00:40.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:40.427 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:00:40.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:40 smithi099 ceph-mon[39420]: pgmap v2212: 47 pgs: 47 active+clean; 2.8 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 130 MiB/s wr, 2.08k op/s 2024-02-09T18:00:40.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:40 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T18:00:40.523 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T18:00:40.528 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:00:40.580 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 21 16 52346 52330 155.703 109.562 0.00769012 0.00641794 2024-02-09T18:00:40.618 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:40.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:40 smithi179 ceph-mon[43975]: pgmap v2212: 47 pgs: 47 active+clean; 2.8 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 130 MiB/s wr, 2.08k op/s 2024-02-09T18:00:40.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:40 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T18:00:40.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:40 smithi162 ceph-mon[44304]: pgmap v2212: 47 pgs: 47 active+clean; 2.8 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 130 MiB/s wr, 2.08k op/s 2024-02-09T18:00:40.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:40 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T18:00:41.582 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 22 16 54167 54151 153.789 113.812 0.00699624 0.00649845 2024-02-09T18:00:41.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:41 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:00:41.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:41 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:00:41.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:41 smithi179 ceph-mon[43975]: osdmap e896: 12 total, 12 up, 11 in 2024-02-09T18:00:41.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:41 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:00:41.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:41 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:00:41.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:41 smithi162 ceph-mon[44304]: osdmap e896: 12 total, 12 up, 11 in 2024-02-09T18:00:41.712 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:41 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:00:41.712 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:41 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:00:41.712 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:41 smithi099 ceph-mon[39420]: osdmap e896: 12 total, 12 up, 11 in 2024-02-09T18:00:42.582 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 23 16 55896 55880 151.801 108.062 0.00595479 0.00658239 2024-02-09T18:00:42.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:42 smithi179 ceph-mon[43975]: 1.10 scrub starts 2024-02-09T18:00:42.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:42 smithi179 ceph-mon[43975]: 1.10 scrub ok 2024-02-09T18:00:42.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:42 smithi179 ceph-mon[43975]: pgmap v2214: 47 pgs: 47 active+clean; 3.0 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 141 MiB/s wr, 2.26k op/s 2024-02-09T18:00:42.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:42 smithi179 ceph-mon[43975]: osdmap e897: 12 total, 12 up, 11 in 2024-02-09T18:00:42.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:42 smithi162 ceph-mon[44304]: 1.10 scrub starts 2024-02-09T18:00:42.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:42 smithi162 ceph-mon[44304]: 1.10 scrub ok 2024-02-09T18:00:42.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:42 smithi162 ceph-mon[44304]: pgmap v2214: 47 pgs: 47 active+clean; 3.0 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 141 MiB/s wr, 2.26k op/s 2024-02-09T18:00:42.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:42 smithi162 ceph-mon[44304]: osdmap e897: 12 total, 12 up, 11 in 2024-02-09T18:00:42.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:42 smithi099 ceph-mon[39420]: 1.10 scrub starts 2024-02-09T18:00:42.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:42 smithi099 ceph-mon[39420]: 1.10 scrub ok 2024-02-09T18:00:42.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:42 smithi099 ceph-mon[39420]: pgmap v2214: 47 pgs: 47 active+clean; 3.0 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 141 MiB/s wr, 2.26k op/s 2024-02-09T18:00:42.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:42 smithi099 ceph-mon[39420]: osdmap e897: 12 total, 12 up, 11 in 2024-02-09T18:00:42.791 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:00:42.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:42.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:00:42.898 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:00:42.940 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:43.124 INFO:teuthology.orchestra.run.smithi099.stderr:2024-02-09T18:00:43.122+0000 7fe4effff700 0 --2- 172.21.15.99:0/2008050835 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4f010e7d0 0x7fe4f0103980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-02-09T18:00:43.584 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 24 16 57653 57637 150.052 109.812 0.00990044 0.00666088 2024-02-09T18:00:44.404 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.8 to 1 (10000) 2024-02-09T18:00:44.582 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 25 16 59210 59194 147.942 97.3125 0.207878 0.00672341 2024-02-09T18:00:44.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:44 smithi179 ceph-mon[43975]: pgmap v2216: 47 pgs: 47 active+clean; 3.2 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 134 MiB/s wr, 2.14k op/s 2024-02-09T18:00:44.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:44 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2008050835' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-02-09T18:00:44.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:44 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T18:00:44.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:44 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T18:00:44.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:44 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T18:00:44.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:44 smithi162 ceph-mon[44304]: pgmap v2216: 47 pgs: 47 active+clean; 3.2 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 134 MiB/s wr, 2.14k op/s 2024-02-09T18:00:44.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:44 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2008050835' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-02-09T18:00:44.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:44 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T18:00:44.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:44 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T18:00:44.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:44 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T18:00:44.738 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T18:00:44.738 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 20, 2024-02-09T18:00:44.738 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 600, 2024-02-09T18:00:44.738 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [ 2024-02-09T18:00:44.738 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.738 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "MOSDPGPush(8.0 856/855 [PushOp(8:0d6d36bd:::benchmark_data_smithi179_148205_object109854:head, version: 816'6998, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:0d6d36bd:::benchmark_data_smithi179_148205_object109854:head@816'6998, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-02-09T18:00:44.739 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:59:13.388879+0000", 2024-02-09T18:00:44.739 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 91.339334457000007, 2024-02-09T18:00:44.739 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.118568496, 2024-02-09T18:00:44.739 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:00:44.739 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T18:00:44.739 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:00:44.739 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.739 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:00:44.739 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.388879+0000", 2024-02-09T18:00:44.739 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:00:44.739 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.739 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.739 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:00:44.739 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.388879+0000", 2024-02-09T18:00:44.740 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:00:44.740 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.740 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.740 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:00:44.740 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.388880+0000", 2024-02-09T18:00:44.740 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.176e-06 2024-02-09T18:00:44.740 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.740 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.740 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:00:44.740 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.388931+0000", 2024-02-09T18:00:44.740 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1221999999999998e-05 2024-02-09T18:00:44.740 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.740 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.741 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:00:44.741 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.388956+0000", 2024-02-09T18:00:44.741 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.5211e-05 2024-02-09T18:00:44.741 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.741 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.741 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:00:44.741 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.388975+0000", 2024-02-09T18:00:44.741 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.8898000000000001e-05 2024-02-09T18:00:44.741 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.741 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.741 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:00:44.741 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.496732+0000", 2024-02-09T18:00:44.741 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.107756876 2024-02-09T18:00:44.741 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.742 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.742 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:00:44.742 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.496741+0000", 2024-02-09T18:00:44.742 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.6710000000000005e-06 2024-02-09T18:00:44.742 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.742 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.742 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:00:44.742 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.507447+0000", 2024-02-09T18:00:44.742 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.010706442 2024-02-09T18:00:44.742 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:00:44.742 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:00:44.742 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:00:44.742 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.742 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.743 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "MOSDPGPush(8.8 856/855 [PushOp(8:1fb9aeb6:::benchmark_data_smithi179_148205_object109887:head, version: 816'6896, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:1fb9aeb6:::benchmark_data_smithi179_148205_object109887:head@816'6896, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-02-09T18:00:44.743 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:59:13.413321+0000", 2024-02-09T18:00:44.743 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 91.314892768999997, 2024-02-09T18:00:44.743 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.12133558699999999, 2024-02-09T18:00:44.743 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:00:44.743 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T18:00:44.743 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:00:44.743 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.743 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:00:44.743 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.413321+0000", 2024-02-09T18:00:44.743 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:00:44.743 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.743 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.743 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:00:44.744 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.413321+0000", 2024-02-09T18:00:44.744 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:00:44.744 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.744 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.744 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:00:44.744 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.413322+0000", 2024-02-09T18:00:44.744 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0950000000000001e-06 2024-02-09T18:00:44.744 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.744 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.744 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:00:44.744 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.413379+0000", 2024-02-09T18:00:44.744 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7781999999999997e-05 2024-02-09T18:00:44.744 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.745 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.745 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:00:44.745 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.413418+0000", 2024-02-09T18:00:44.745 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.8306000000000002e-05 2024-02-09T18:00:44.745 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.745 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.745 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:00:44.745 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.413438+0000", 2024-02-09T18:00:44.745 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.9803999999999999e-05 2024-02-09T18:00:44.745 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.745 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.745 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:00:44.745 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.507503+0000", 2024-02-09T18:00:44.745 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.094065498999999997 2024-02-09T18:00:44.745 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.745 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.745 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:00:44.746 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.507514+0000", 2024-02-09T18:00:44.746 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0988999999999999e-05 2024-02-09T18:00:44.746 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.746 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.746 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:00:44.746 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.534656+0000", 2024-02-09T18:00:44.746 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.027142111999999999 2024-02-09T18:00:44.746 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:00:44.746 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:00:44.746 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:00:44.746 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.746 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.746 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "MOSDPGPush(8.8 856/855 [PushOp(8:13e0b31e:::benchmark_data_smithi179_148205_object110014:head, version: 816'6902, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:13e0b31e:::benchmark_data_smithi179_148205_object110014:head@816'6902, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-02-09T18:00:44.746 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:59:13.848740+0000", 2024-02-09T18:00:44.746 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 90.879473778000005, 2024-02-09T18:00:44.746 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.193517147, 2024-02-09T18:00:44.747 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:00:44.747 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T18:00:44.747 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:00:44.747 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.747 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:00:44.747 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.848740+0000", 2024-02-09T18:00:44.747 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:00:44.747 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.747 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.747 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:00:44.747 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.848740+0000", 2024-02-09T18:00:44.747 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:00:44.747 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.747 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.747 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:00:44.748 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.848741+0000", 2024-02-09T18:00:44.748 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.7900000000000007e-07 2024-02-09T18:00:44.748 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.748 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.748 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:00:44.748 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.848793+0000", 2024-02-09T18:00:44.748 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2145000000000003e-05 2024-02-09T18:00:44.748 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.748 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.748 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:00:44.748 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.848823+0000", 2024-02-09T18:00:44.748 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.9989999999999999e-05 2024-02-09T18:00:44.748 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.748 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.749 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:00:44.749 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.848841+0000", 2024-02-09T18:00:44.749 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.7978999999999998e-05 2024-02-09T18:00:44.749 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.749 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.749 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:00:44.749 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:14.038689+0000", 2024-02-09T18:00:44.749 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.18984825599999999 2024-02-09T18:00:44.749 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.749 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.749 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:00:44.749 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:14.038698+0000", 2024-02-09T18:00:44.749 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.9339999999999995e-06 2024-02-09T18:00:44.749 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.749 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.750 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:00:44.750 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:14.042257+0000", 2024-02-09T18:00:44.750 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0035588640000000001 2024-02-09T18:00:44.750 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:00:44.750 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:00:44.750 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:00:44.750 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.750 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.750 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.26305.0:227943 8.0 e856/855 8:08b9b718:::benchmark_data_smithi179_148205_object104871:head v 856'14549, mlcod=816'7003)", 2024-02-09T18:00:44.750 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:59:13.900087+0000", 2024-02-09T18:00:44.750 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 90.828125955999994, 2024-02-09T18:00:44.750 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.127519629, 2024-02-09T18:00:44.750 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:00:44.750 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:00:44.750 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:00:44.751 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.751 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:00:44.751 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.900087+0000", 2024-02-09T18:00:44.751 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:00:44.751 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.751 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.751 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:00:44.751 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.900087+0000", 2024-02-09T18:00:44.751 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:00:44.751 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.751 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.751 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:00:44.751 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.900088+0000", 2024-02-09T18:00:44.751 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.9100000000000002e-07 2024-02-09T18:00:44.751 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.751 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.752 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:00:44.752 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.900096+0000", 2024-02-09T18:00:44.752 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.9200000000000004e-06 2024-02-09T18:00:44.752 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.752 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.752 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:00:44.752 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.900112+0000", 2024-02-09T18:00:44.752 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5942000000000001e-05 2024-02-09T18:00:44.752 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.752 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.752 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:00:44.752 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.900120+0000", 2024-02-09T18:00:44.752 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.1370000000000002e-06 2024-02-09T18:00:44.752 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.753 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.753 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:00:44.753 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.956909+0000", 2024-02-09T18:00:44.753 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.056788473999999999 2024-02-09T18:00:44.753 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.753 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.753 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:00:44.753 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.956938+0000", 2024-02-09T18:00:44.753 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.9652999999999999e-05 2024-02-09T18:00:44.753 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.753 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.753 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:00:44.753 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:14.027556+0000", 2024-02-09T18:00:44.753 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.070617569000000005 2024-02-09T18:00:44.753 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.753 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.754 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:00:44.754 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:14.027607+0000", 2024-02-09T18:00:44.754 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1042999999999999e-05 2024-02-09T18:00:44.754 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:00:44.754 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:00:44.754 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:00:44.754 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.754 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.754 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.26305.0:227949 8.0 e856/855 8:09a6da4e:::benchmark_data_smithi179_148205_object104877:head v 856'14550, mlcod=816'7003)", 2024-02-09T18:00:44.754 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:59:13.906980+0000", 2024-02-09T18:00:44.754 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 90.821233663000001, 2024-02-09T18:00:44.754 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.120649152, 2024-02-09T18:00:44.754 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:00:44.754 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:00:44.755 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:00:44.755 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.755 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:00:44.755 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.906980+0000", 2024-02-09T18:00:44.755 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:00:44.755 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.755 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.755 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:00:44.755 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.906980+0000", 2024-02-09T18:00:44.755 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:00:44.755 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.755 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.755 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:00:44.755 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.906981+0000", 2024-02-09T18:00:44.755 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.8000000000000004e-07 2024-02-09T18:00:44.756 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.756 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.756 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:00:44.756 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.906988+0000", 2024-02-09T18:00:44.756 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.0790000000000003e-06 2024-02-09T18:00:44.756 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.756 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.756 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:00:44.756 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.907003+0000", 2024-02-09T18:00:44.756 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5242000000000001e-05 2024-02-09T18:00:44.756 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.756 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.756 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:00:44.756 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.907010+0000", 2024-02-09T18:00:44.756 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.7700000000000004e-06 2024-02-09T18:00:44.757 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.757 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.757 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:00:44.757 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.965586+0000", 2024-02-09T18:00:44.757 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.058576243 2024-02-09T18:00:44.757 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.757 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.757 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:00:44.757 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.965605+0000", 2024-02-09T18:00:44.757 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.8873999999999999e-05 2024-02-09T18:00:44.757 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.757 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.757 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:00:44.757 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:14.027610+0000", 2024-02-09T18:00:44.757 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.062005174000000003 2024-02-09T18:00:44.757 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.758 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.758 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:00:44.758 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:14.027629+0000", 2024-02-09T18:00:44.758 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.889e-05 2024-02-09T18:00:44.758 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:00:44.758 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:00:44.758 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:00:44.758 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.758 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.758 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.26305.0:227952 8.8 e856/855 8:101d55b4:::benchmark_data_smithi179_148205_object104880:head v 856'14282, mlcod=816'6901)", 2024-02-09T18:00:44.758 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:59:13.910856+0000", 2024-02-09T18:00:44.758 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 90.817357453, 2024-02-09T18:00:44.758 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.116794316, 2024-02-09T18:00:44.758 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:00:44.758 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:00:44.758 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:00:44.758 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.759 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:00:44.759 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.910856+0000", 2024-02-09T18:00:44.759 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:00:44.759 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.759 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.759 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:00:44.759 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.910856+0000", 2024-02-09T18:00:44.759 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:00:44.759 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.759 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.759 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:00:44.759 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.910857+0000", 2024-02-09T18:00:44.759 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.333e-06 2024-02-09T18:00:44.759 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.759 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.759 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:00:44.759 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.910867+0000", 2024-02-09T18:00:44.760 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.6409999999999997e-06 2024-02-09T18:00:44.760 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.760 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.760 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:00:44.760 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.910886+0000", 2024-02-09T18:00:44.760 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.9573e-05 2024-02-09T18:00:44.760 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.760 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.760 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:00:44.760 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.910897+0000", 2024-02-09T18:00:44.760 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0498e-05 2024-02-09T18:00:44.760 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.760 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.760 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:00:44.760 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.965575+0000", 2024-02-09T18:00:44.760 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.054677884000000003 2024-02-09T18:00:44.761 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.761 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.761 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:00:44.761 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.965593+0000", 2024-02-09T18:00:44.761 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.7929999999999999e-05 2024-02-09T18:00:44.761 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.761 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.761 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:00:44.761 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:14.027631+0000", 2024-02-09T18:00:44.761 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.062038015000000002 2024-02-09T18:00:44.761 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.761 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.761 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:00:44.761 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:14.027650+0000", 2024-02-09T18:00:44.761 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.9442000000000001e-05 2024-02-09T18:00:44.761 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:00:44.762 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:00:44.762 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:00:44.762 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.762 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.762 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.26305.0:227950 8.0 e856/855 8:0cf3d7e8:::benchmark_data_smithi179_148205_object104878:head v 856'14551, mlcod=816'7003)", 2024-02-09T18:00:44.762 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:59:13.910919+0000", 2024-02-09T18:00:44.762 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 90.817294545999999, 2024-02-09T18:00:44.762 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.116828416, 2024-02-09T18:00:44.762 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:00:44.762 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:00:44.762 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:00:44.762 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.762 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:00:44.762 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.910919+0000", 2024-02-09T18:00:44.762 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:00:44.762 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.763 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.763 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:00:44.763 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.910919+0000", 2024-02-09T18:00:44.763 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:00:44.763 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.763 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.763 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:00:44.763 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.910919+0000", 2024-02-09T18:00:44.763 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.9799999999999999e-07 2024-02-09T18:00:44.763 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.763 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.763 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:00:44.763 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.910924+0000", 2024-02-09T18:00:44.763 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.4479999999999996e-06 2024-02-09T18:00:44.763 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.763 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.764 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:00:44.764 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.910929+0000", 2024-02-09T18:00:44.764 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.9750000000000004e-06 2024-02-09T18:00:44.764 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.764 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.764 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:00:44.764 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.910934+0000", 2024-02-09T18:00:44.764 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.33e-06 2024-02-09T18:00:44.764 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.764 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.764 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:00:44.764 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.974570+0000", 2024-02-09T18:00:44.764 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.063635915000000001 2024-02-09T18:00:44.764 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.764 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.765 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:00:44.765 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.974593+0000", 2024-02-09T18:00:44.765 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.3255999999999999e-05 2024-02-09T18:00:44.765 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.765 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.765 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:00:44.765 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:14.027728+0000", 2024-02-09T18:00:44.765 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.053134828000000002 2024-02-09T18:00:44.765 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.765 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.765 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:00:44.765 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:14.027747+0000", 2024-02-09T18:00:44.765 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.9366000000000001e-05 2024-02-09T18:00:44.765 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:00:44.765 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:00:44.765 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:00:44.765 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.766 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.766 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "MOSDPGPush(8.0 856/855 [PushOp(8:08871cdd:::benchmark_data_smithi179_148205_object109917:head, version: 816'7005, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:08871cdd:::benchmark_data_smithi179_148205_object109917:head@816'7005, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-02-09T18:00:44.766 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:59:13.925800+0000", 2024-02-09T18:00:44.766 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 90.802413313000002, 2024-02-09T18:00:44.766 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.13107142799999999, 2024-02-09T18:00:44.766 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:00:44.766 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T18:00:44.766 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:00:44.766 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.766 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:00:44.766 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.925800+0000", 2024-02-09T18:00:44.766 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:00:44.766 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.766 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.766 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:00:44.766 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.925800+0000", 2024-02-09T18:00:44.767 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:00:44.767 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.767 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.767 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:00:44.767 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.925801+0000", 2024-02-09T18:00:44.767 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3370000000000001e-06 2024-02-09T18:00:44.767 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.767 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.767 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:00:44.767 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.934740+0000", 2024-02-09T18:00:44.767 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0089389369999999992 2024-02-09T18:00:44.767 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.767 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.767 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:00:44.767 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.934762+0000", 2024-02-09T18:00:44.767 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.1350000000000001e-05 2024-02-09T18:00:44.767 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.768 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.768 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:00:44.768 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:13.934773+0000", 2024-02-09T18:00:44.768 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1425e-05 2024-02-09T18:00:44.768 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.768 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.768 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:00:44.768 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:14.051623+0000", 2024-02-09T18:00:44.768 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.116849933 2024-02-09T18:00:44.768 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.768 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.768 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:00:44.768 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:14.051632+0000", 2024-02-09T18:00:44.768 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.0469999999999999e-06 2024-02-09T18:00:44.768 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.768 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.769 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:00:44.769 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:14.056871+0000", 2024-02-09T18:00:44.769 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0052393989999999996 2024-02-09T18:00:44.769 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:00:44.769 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:00:44.769 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:00:44.769 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.769 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.769 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "MOSDPGPush(8.8 857/855 [PushOp(8:14b3df0b:::benchmark_data_smithi179_148205_object110147:head, version: 816'6910, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:14b3df0b:::benchmark_data_smithi179_148205_object110147:head@816'6910, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-02-09T18:00:44.769 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:59:14.285375+0000", 2024-02-09T18:00:44.769 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 90.442838155000004, 2024-02-09T18:00:44.769 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.14172563199999999, 2024-02-09T18:00:44.769 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:00:44.769 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T18:00:44.769 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:00:44.770 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.770 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:00:44.770 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:14.285375+0000", 2024-02-09T18:00:44.770 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:00:44.770 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.770 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.770 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:00:44.770 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:14.285375+0000", 2024-02-09T18:00:44.770 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:00:44.770 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.770 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.770 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:00:44.770 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:14.285376+0000", 2024-02-09T18:00:44.770 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.5199999999999995e-07 2024-02-09T18:00:44.770 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.770 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.771 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:00:44.771 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:14.285430+0000", 2024-02-09T18:00:44.771 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.4008000000000002e-05 2024-02-09T18:00:44.771 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.771 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.771 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:00:44.771 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:14.285454+0000", 2024-02-09T18:00:44.771 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.3872000000000001e-05 2024-02-09T18:00:44.771 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.771 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.771 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:00:44.771 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:14.285475+0000", 2024-02-09T18:00:44.771 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0669000000000001e-05 2024-02-09T18:00:44.771 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.771 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.771 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:00:44.772 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:14.415950+0000", 2024-02-09T18:00:44.772 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.130475015 2024-02-09T18:00:44.772 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.772 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.772 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:00:44.772 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:14.415958+0000", 2024-02-09T18:00:44.772 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.4370000000000007e-06 2024-02-09T18:00:44.772 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.772 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.772 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:00:44.772 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:14.427101+0000", 2024-02-09T18:00:44.772 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.011142779 2024-02-09T18:00:44.772 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:00:44.772 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:00:44.773 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:00:44.773 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.773 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.773 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "MOSDPGPush(8.0 857/855 [PushOp(8:092fa69a:::benchmark_data_smithi179_148205_object109969:head, version: 816'7009, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:092fa69a:::benchmark_data_smithi179_148205_object109969:head@816'7009, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-02-09T18:00:44.773 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:59:14.288861+0000", 2024-02-09T18:00:44.773 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 90.439352217000007, 2024-02-09T18:00:44.773 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.132265456, 2024-02-09T18:00:44.773 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:00:44.773 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T18:00:44.773 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:00:44.773 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.773 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:00:44.773 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:14.288861+0000", 2024-02-09T18:00:44.773 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:00:44.773 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.774 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.774 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:00:44.774 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:14.288861+0000", 2024-02-09T18:00:44.774 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:00:44.774 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.774 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.774 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:00:44.774 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:14.288862+0000", 2024-02-09T18:00:44.774 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1370000000000001e-06 2024-02-09T18:00:44.774 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.774 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.774 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:00:44.774 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:14.292860+0000", 2024-02-09T18:00:44.774 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0039982139999999999 2024-02-09T18:00:44.775 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.775 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.775 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:00:44.775 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:14.292897+0000", 2024-02-09T18:00:44.775 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.7004e-05 2024-02-09T18:00:44.775 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.775 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.775 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:00:44.775 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:14.292918+0000", 2024-02-09T18:00:44.775 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0114000000000001e-05 2024-02-09T18:00:44.775 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.775 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.775 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:00:44.775 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:14.415937+0000", 2024-02-09T18:00:44.775 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.12301925900000001 2024-02-09T18:00:44.776 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.776 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.776 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:00:44.776 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:14.415960+0000", 2024-02-09T18:00:44.776 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.3034999999999999e-05 2024-02-09T18:00:44.776 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.776 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.776 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:00:44.776 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:14.421127+0000", 2024-02-09T18:00:44.776 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.005166693 2024-02-09T18:00:44.776 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:00:44.776 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:00:44.776 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:00:44.776 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.777 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.777 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "MOSDPGPush(8.8 857/855 [PushOp(8:14871f12:::benchmark_data_smithi179_148205_object110207:head, version: 816'6913, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:14871f12:::benchmark_data_smithi179_148205_object110207:head@816'6913, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-02-09T18:00:44.777 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:59:14.472838+0000", 2024-02-09T18:00:44.777 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 90.255375440999998, 2024-02-09T18:00:44.777 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.150691254, 2024-02-09T18:00:44.777 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:00:44.777 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T18:00:44.777 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:00:44.777 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.777 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:00:44.777 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:14.472838+0000", 2024-02-09T18:00:44.777 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:00:44.777 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.777 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.777 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:00:44.778 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:14.472838+0000", 2024-02-09T18:00:44.778 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:00:44.778 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.778 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.778 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:00:44.778 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:14.472840+0000", 2024-02-09T18:00:44.778 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.6619999999999999e-06 2024-02-09T18:00:44.778 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.778 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.778 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:00:44.778 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:14.472897+0000", 2024-02-09T18:00:44.778 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7250999999999998e-05 2024-02-09T18:00:44.778 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.778 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.779 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:00:44.779 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:14.472922+0000", 2024-02-09T18:00:44.779 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.5559e-05 2024-02-09T18:00:44.779 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.779 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.779 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:00:44.779 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:14.472950+0000", 2024-02-09T18:00:44.779 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.7806000000000001e-05 2024-02-09T18:00:44.779 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.779 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.779 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:00:44.779 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:14.614191+0000", 2024-02-09T18:00:44.779 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.14124035800000001 2024-02-09T18:00:44.779 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.779 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.779 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:00:44.780 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:14.614204+0000", 2024-02-09T18:00:44.780 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3888e-05 2024-02-09T18:00:44.780 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.780 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.780 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:00:44.780 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:14.623529+0000", 2024-02-09T18:00:44.780 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0093247299999999998 2024-02-09T18:00:44.780 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:00:44.780 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:00:44.780 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:00:44.780 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.780 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.780 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "MOSDPGPush(8.0 862/855 [PushOp(8:0713764c:::benchmark_data_smithi179_148205_object113182:head, version: 816'7248, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:0713764c:::benchmark_data_smithi179_148205_object113182:head@816'7248, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-02-09T18:00:44.780 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:59:24.669531+0000", 2024-02-09T18:00:44.780 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 80.058682501000007, 2024-02-09T18:00:44.781 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20158225599999999, 2024-02-09T18:00:44.781 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:00:44.781 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T18:00:44.781 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:00:44.781 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.781 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:00:44.781 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:24.669531+0000", 2024-02-09T18:00:44.781 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:00:44.781 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.781 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.781 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:00:44.781 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:24.669531+0000", 2024-02-09T18:00:44.781 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:00:44.781 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.781 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.781 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:00:44.782 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:24.669531+0000", 2024-02-09T18:00:44.782 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.5999999999999999e-07 2024-02-09T18:00:44.782 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.782 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.782 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:00:44.782 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:24.669566+0000", 2024-02-09T18:00:44.782 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.4529000000000001e-05 2024-02-09T18:00:44.782 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.782 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.782 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:00:44.782 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:24.669579+0000", 2024-02-09T18:00:44.782 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2866e-05 2024-02-09T18:00:44.782 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.782 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.783 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:00:44.783 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:24.669592+0000", 2024-02-09T18:00:44.783 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.306e-05 2024-02-09T18:00:44.783 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.783 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.783 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:00:44.783 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:24.867221+0000", 2024-02-09T18:00:44.783 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.19762964 2024-02-09T18:00:44.783 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.783 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.783 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:00:44.783 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:24.867232+0000", 2024-02-09T18:00:44.783 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0282000000000001e-05 2024-02-09T18:00:44.783 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.783 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.783 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:00:44.784 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:24.871113+0000", 2024-02-09T18:00:44.784 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0038815189999999999 2024-02-09T18:00:44.784 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:00:44.784 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:00:44.784 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:00:44.784 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.784 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.784 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "MOSDPGPush(8.8 862/855 [PushOp(8:15b11e3e:::benchmark_data_smithi179_148205_object113810:head, version: 817'7162, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:15b11e3e:::benchmark_data_smithi179_148205_object113810:head@817'7162, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-02-09T18:00:44.784 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:59:24.716592+0000", 2024-02-09T18:00:44.784 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 80.011620969000006, 2024-02-09T18:00:44.784 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.154337048, 2024-02-09T18:00:44.784 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:00:44.784 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T18:00:44.784 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:00:44.785 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.785 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:00:44.785 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:24.716592+0000", 2024-02-09T18:00:44.785 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:00:44.785 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.785 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.785 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:00:44.785 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:24.716592+0000", 2024-02-09T18:00:44.785 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:00:44.785 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.785 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.785 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:00:44.785 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:24.716594+0000", 2024-02-09T18:00:44.785 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5579999999999999e-06 2024-02-09T18:00:44.785 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.786 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.786 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:00:44.786 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:24.716650+0000", 2024-02-09T18:00:44.786 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6455000000000001e-05 2024-02-09T18:00:44.786 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.786 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.786 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:00:44.786 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:24.716673+0000", 2024-02-09T18:00:44.786 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.249e-05 2024-02-09T18:00:44.786 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.786 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.786 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:00:44.786 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:24.716687+0000", 2024-02-09T18:00:44.786 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3964999999999999e-05 2024-02-09T18:00:44.787 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.787 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.787 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:00:44.787 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:24.867204+0000", 2024-02-09T18:00:44.787 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.15051742800000001 2024-02-09T18:00:44.787 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.787 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.787 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:00:44.787 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:24.867219+0000", 2024-02-09T18:00:44.787 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4457999999999999e-05 2024-02-09T18:00:44.787 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.787 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.787 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:00:44.787 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:24.870929+0000", 2024-02-09T18:00:44.787 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0037106940000000001 2024-02-09T18:00:44.787 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:00:44.788 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:00:44.788 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:00:44.788 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.788 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.788 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "MOSDPGPush(8.8 862/855 [PushOp(8:1051e066:::benchmark_data_smithi179_148205_object114077:head, version: 818'7173, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:1051e066:::benchmark_data_smithi179_148205_object114077:head@818'7173, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-02-09T18:00:44.788 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:59:25.526864+0000", 2024-02-09T18:00:44.788 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 79.201349738999994, 2024-02-09T18:00:44.788 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.32089669799999998, 2024-02-09T18:00:44.788 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:00:44.788 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T18:00:44.788 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:00:44.788 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.788 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:00:44.788 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:25.526864+0000", 2024-02-09T18:00:44.788 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:00:44.788 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.789 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.789 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:00:44.789 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:25.526864+0000", 2024-02-09T18:00:44.789 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:00:44.789 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.789 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.789 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:00:44.789 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:25.526865+0000", 2024-02-09T18:00:44.789 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.705e-06 2024-02-09T18:00:44.789 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.789 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.789 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:00:44.789 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:25.526908+0000", 2024-02-09T18:00:44.789 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2385999999999997e-05 2024-02-09T18:00:44.789 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.789 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.789 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:00:44.790 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:25.526943+0000", 2024-02-09T18:00:44.790 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.5564e-05 2024-02-09T18:00:44.790 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.790 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.790 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:00:44.790 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:25.526963+0000", 2024-02-09T18:00:44.790 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.9371000000000001e-05 2024-02-09T18:00:44.790 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.790 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.790 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:00:44.790 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:25.838423+0000", 2024-02-09T18:00:44.790 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.31145991000000001 2024-02-09T18:00:44.790 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.790 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.790 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:00:44.790 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:25.838436+0000", 2024-02-09T18:00:44.791 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.394e-05 2024-02-09T18:00:44.791 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.791 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.791 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:00:44.791 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:25.847760+0000", 2024-02-09T18:00:44.791 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0093238220000000007 2024-02-09T18:00:44.791 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:00:44.791 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:00:44.791 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:00:44.791 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.791 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.791 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "MOSDPGPush(8.0 862/855 [PushOp(8:0ca54e91:::benchmark_data_smithi179_148205_object113481:head, version: 817'7260, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:0ca54e91:::benchmark_data_smithi179_148205_object113481:head@817'7260, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-02-09T18:00:44.791 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:59:25.543883+0000", 2024-02-09T18:00:44.791 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 79.184329903999995, 2024-02-09T18:00:44.791 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.366177213, 2024-02-09T18:00:44.791 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:00:44.792 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T18:00:44.792 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:00:44.792 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.792 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:00:44.792 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:25.543883+0000", 2024-02-09T18:00:44.792 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:00:44.792 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.792 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.792 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:00:44.792 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:25.543883+0000", 2024-02-09T18:00:44.792 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:00:44.792 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.792 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.792 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:00:44.792 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:25.543885+0000", 2024-02-09T18:00:44.792 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1260000000000001e-06 2024-02-09T18:00:44.793 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.793 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.793 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:00:44.793 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:25.543943+0000", 2024-02-09T18:00:44.793 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.8764000000000002e-05 2024-02-09T18:00:44.793 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.793 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.793 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:00:44.793 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:25.543971+0000", 2024-02-09T18:00:44.793 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.7291e-05 2024-02-09T18:00:44.793 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.793 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.793 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:00:44.793 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:25.543993+0000", 2024-02-09T18:00:44.794 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2283999999999999e-05 2024-02-09T18:00:44.794 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.794 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.794 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:00:44.794 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:25.906614+0000", 2024-02-09T18:00:44.794 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.36262160799999998 2024-02-09T18:00:44.794 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.794 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.794 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:00:44.794 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:25.906627+0000", 2024-02-09T18:00:44.794 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2627e-05 2024-02-09T18:00:44.794 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.794 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.794 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:00:44.794 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:25.910061+0000", 2024-02-09T18:00:44.794 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.003433513 2024-02-09T18:00:44.794 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:00:44.795 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:00:44.795 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:00:44.795 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.795 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.795 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "MOSDPGPush(8.8 862/855 [PushOp(8:105c9f3c:::benchmark_data_smithi179_148205_object114183:head, version: 818'7180, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:105c9f3c:::benchmark_data_smithi179_148205_object114183:head@818'7180, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-02-09T18:00:44.795 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:59:26.046908+0000", 2024-02-09T18:00:44.795 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 78.681305328999997, 2024-02-09T18:00:44.795 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.122674142, 2024-02-09T18:00:44.795 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:00:44.795 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T18:00:44.795 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:00:44.795 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.795 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:00:44.795 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.046908+0000", 2024-02-09T18:00:44.795 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:00:44.795 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.796 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.796 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:00:44.796 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.046908+0000", 2024-02-09T18:00:44.796 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:00:44.796 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.796 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.796 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:00:44.796 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.046909+0000", 2024-02-09T18:00:44.796 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.209e-06 2024-02-09T18:00:44.796 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.796 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.796 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:00:44.796 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.046960+0000", 2024-02-09T18:00:44.797 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1004999999999997e-05 2024-02-09T18:00:44.797 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.797 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.797 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:00:44.797 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.046991+0000", 2024-02-09T18:00:44.797 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.1303999999999997e-05 2024-02-09T18:00:44.797 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.797 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.797 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:00:44.797 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.047010+0000", 2024-02-09T18:00:44.797 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.8595000000000001e-05 2024-02-09T18:00:44.797 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.797 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.797 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:00:44.797 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.165479+0000", 2024-02-09T18:00:44.797 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.118468754 2024-02-09T18:00:44.797 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.798 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.798 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:00:44.798 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.165493+0000", 2024-02-09T18:00:44.798 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.456e-05 2024-02-09T18:00:44.798 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.798 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.798 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:00:44.798 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.169582+0000", 2024-02-09T18:00:44.798 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0040887149999999997 2024-02-09T18:00:44.798 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:00:44.798 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:00:44.798 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:00:44.798 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.798 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.798 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "MOSDPGPush(8.0 862/855 [PushOp(8:03e2b1d2:::benchmark_data_smithi179_148205_object113540:head, version: 817'7266, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:03e2b1d2:::benchmark_data_smithi179_148205_object113540:head@817'7266, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-02-09T18:00:44.798 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:59:26.092558+0000", 2024-02-09T18:00:44.799 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 78.635655627999995, 2024-02-09T18:00:44.799 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.10890089999999999, 2024-02-09T18:00:44.799 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:00:44.799 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T18:00:44.799 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:00:44.799 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.799 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:00:44.799 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.092558+0000", 2024-02-09T18:00:44.799 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:00:44.799 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.799 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.799 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:00:44.799 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.092558+0000", 2024-02-09T18:00:44.799 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:00:44.799 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.799 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.799 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:00:44.799 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.092558+0000", 2024-02-09T18:00:44.800 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.7800000000000001e-07 2024-02-09T18:00:44.800 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.800 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.800 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:00:44.800 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.092630+0000", 2024-02-09T18:00:44.800 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.1404999999999998e-05 2024-02-09T18:00:44.800 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.800 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.800 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:00:44.800 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.092650+0000", 2024-02-09T18:00:44.800 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0322e-05 2024-02-09T18:00:44.800 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.800 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.800 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:00:44.800 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.092666+0000", 2024-02-09T18:00:44.800 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5529e-05 2024-02-09T18:00:44.800 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.801 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.801 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:00:44.801 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.169638+0000", 2024-02-09T18:00:44.801 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.076972223000000006 2024-02-09T18:00:44.801 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.801 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.801 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:00:44.801 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.169651+0000", 2024-02-09T18:00:44.801 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.366e-05 2024-02-09T18:00:44.801 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.801 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.801 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:00:44.801 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.201459+0000", 2024-02-09T18:00:44.801 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.031807083 2024-02-09T18:00:44.801 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:00:44.802 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:00:44.802 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:00:44.802 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.802 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.802 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.26305.0:236421 8.8 e865/855 8:1bb471e3:::benchmark_data_smithi179_148205_object113349:head v 865'14834, mlcod=820'7351)", 2024-02-09T18:00:44.802 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:59:28.878079+0000", 2024-02-09T18:00:44.802 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 75.850134650000001, 2024-02-09T18:00:44.802 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.11038102900000001, 2024-02-09T18:00:44.802 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:00:44.802 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:00:44.802 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:00:44.802 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.802 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:00:44.802 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:28.878079+0000", 2024-02-09T18:00:44.802 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:00:44.802 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.802 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.803 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:00:44.803 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:28.878079+0000", 2024-02-09T18:00:44.803 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:00:44.803 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.803 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.803 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:00:44.803 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:28.878079+0000", 2024-02-09T18:00:44.803 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.2999999999999999e-07 2024-02-09T18:00:44.803 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.803 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.803 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:00:44.803 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:28.878086+0000", 2024-02-09T18:00:44.803 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.8040000000000004e-06 2024-02-09T18:00:44.803 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.803 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.804 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:00:44.804 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:28.878098+0000", 2024-02-09T18:00:44.804 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1219999999999999e-05 2024-02-09T18:00:44.804 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.804 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.804 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:00:44.804 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:28.878105+0000", 2024-02-09T18:00:44.804 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.267e-06 2024-02-09T18:00:44.804 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.804 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.804 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:00:44.804 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:28.880862+0000", 2024-02-09T18:00:44.804 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0027570310000000001 2024-02-09T18:00:44.804 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.804 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.805 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:00:44.805 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:28.880887+0000", 2024-02-09T18:00:44.805 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.4888e-05 2024-02-09T18:00:44.805 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.805 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.805 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:00:44.805 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:28.988437+0000", 2024-02-09T18:00:44.805 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.107550749 2024-02-09T18:00:44.805 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.805 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.805 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:00:44.805 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:28.988460+0000", 2024-02-09T18:00:44.805 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2240000000000001e-05 2024-02-09T18:00:44.805 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:00:44.805 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:00:44.806 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:00:44.806 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.806 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.806 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "MOSDPGPush(8.0 872/855 [PushOp(8:0fb31b37:::benchmark_data_smithi179_148205_object121612:head, version: 821'7779, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:0fb31b37:::benchmark_data_smithi179_148205_object121612:head@821'7779, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-02-09T18:00:44.806 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:59:41.295744+0000", 2024-02-09T18:00:44.806 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 63.432468854, 2024-02-09T18:00:44.806 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.19777230200000001, 2024-02-09T18:00:44.806 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:00:44.806 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T18:00:44.806 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:00:44.806 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.806 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:00:44.806 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:41.295744+0000", 2024-02-09T18:00:44.806 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:00:44.806 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.807 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.807 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:00:44.807 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:41.295744+0000", 2024-02-09T18:00:44.807 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:00:44.807 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.807 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.807 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:00:44.807 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:41.295745+0000", 2024-02-09T18:00:44.807 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.6200000000000006e-07 2024-02-09T18:00:44.807 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.807 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.807 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:00:44.807 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:41.295797+0000", 2024-02-09T18:00:44.807 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1659000000000001e-05 2024-02-09T18:00:44.807 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.807 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.808 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:00:44.808 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:41.295817+0000", 2024-02-09T18:00:44.808 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.9978000000000001e-05 2024-02-09T18:00:44.808 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.808 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.808 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:00:44.808 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:41.295830+0000", 2024-02-09T18:00:44.808 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2751e-05 2024-02-09T18:00:44.808 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.808 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.808 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:00:44.808 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:41.490367+0000", 2024-02-09T18:00:44.808 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.19453764700000001 2024-02-09T18:00:44.808 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.808 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.808 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:00:44.809 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:41.490382+0000", 2024-02-09T18:00:44.809 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4139e-05 2024-02-09T18:00:44.809 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.809 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.809 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:00:44.809 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:41.493517+0000", 2024-02-09T18:00:44.809 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.003135166 2024-02-09T18:00:44.809 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:00:44.809 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:00:44.809 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:00:44.809 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.809 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.809 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "MOSDPGRecoveryDelete(8.8 e872,855 8:19fb00fa:::benchmark_data_smithi179_148205_object958:head,821'7767)", 2024-02-09T18:00:44.809 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:59:41.329503+0000", 2024-02-09T18:00:44.809 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 63.398710600999998, 2024-02-09T18:00:44.810 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.16418971600000001, 2024-02-09T18:00:44.810 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:00:44.810 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T18:00:44.810 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:00:44.810 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.810 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:00:44.810 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:41.329503+0000", 2024-02-09T18:00:44.810 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:00:44.810 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.810 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.810 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:00:44.810 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:41.329503+0000", 2024-02-09T18:00:44.810 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:00:44.810 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.810 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.810 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:00:44.810 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:41.329504+0000", 2024-02-09T18:00:44.811 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.8999999999999995e-07 2024-02-09T18:00:44.811 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.811 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.811 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:00:44.811 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:41.329510+0000", 2024-02-09T18:00:44.811 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.409e-06 2024-02-09T18:00:44.811 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.811 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.811 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:00:44.811 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:41.329523+0000", 2024-02-09T18:00:44.811 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2850000000000001e-05 2024-02-09T18:00:44.811 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.811 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.811 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:00:44.811 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:41.329530+0000", 2024-02-09T18:00:44.811 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.2099999999999996e-06 2024-02-09T18:00:44.812 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.812 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.812 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:00:44.812 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:41.493569+0000", 2024-02-09T18:00:44.812 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.164039292 2024-02-09T18:00:44.812 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.812 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.812 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:00:44.812 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:41.493579+0000", 2024-02-09T18:00:44.812 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.9799999999999993e-06 2024-02-09T18:00:44.812 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:00:44.812 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:00:44.812 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:00:44.812 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:41.493692+0000", 2024-02-09T18:00:44.812 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00011308500000000001 2024-02-09T18:00:44.813 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:00:44.813 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:00:44.813 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:00:44.813 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:00:44.813 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:00:44.813 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T18:00:44.813 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:44 smithi099 ceph-mon[39420]: pgmap v2216: 47 pgs: 47 active+clean; 3.2 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 134 MiB/s wr, 2.14k op/s 2024-02-09T18:00:44.814 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:44 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2008050835' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-02-09T18:00:44.814 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:44 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T18:00:44.814 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:44 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T18:00:44.814 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:44 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T18:00:45.505 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:00:45.509 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:45.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:00:45.585 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 26 16 60991 60975 146.515 111.312 0.0112001 0.00682124 2024-02-09T18:00:45.611 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:00:45.648 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:45.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:45 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2008050835' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-02-09T18:00:45.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:45 smithi179 ceph-mon[43975]: osdmap e898: 12 total, 12 up, 11 in 2024-02-09T18:00:45.675 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:00:45.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:45 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2008050835' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-02-09T18:00:45.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:45 smithi162 ceph-mon[44304]: osdmap e898: 12 total, 12 up, 11 in 2024-02-09T18:00:45.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:45.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:00:45.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:45 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2008050835' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-02-09T18:00:45.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:45 smithi099 ceph-mon[39420]: osdmap e898: 12 total, 12 up, 11 in 2024-02-09T18:00:45.779 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T18:00:45.817 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:45.843 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T18:00:45.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:45.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T18:00:45.948 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:00:45.994 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:46.028 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:00:46.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:46.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:00:46.134 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T18:00:46.168 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:46.195 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T18:00:46.199 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:46.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T18:00:46.301 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:00:46.341 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:46.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:46 smithi162 ceph-mon[44304]: pgmap v2218: 47 pgs: 47 active+clean; 3.3 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-02-09T18:00:46.586 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 27 16 62707 62691 145.058 107.25 0.0101267 0.00688941 2024-02-09T18:00:46.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:46 smithi179 ceph-mon[43975]: pgmap v2218: 47 pgs: 47 active+clean; 3.3 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-02-09T18:00:46.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:46 smithi099 ceph-mon[39420]: pgmap v2218: 47 pgs: 47 active+clean; 3.3 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-02-09T18:00:47.415 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:47 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:00:47.415 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:47 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:00:47.415 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:47 smithi099 ceph-mon[39420]: osdmap e899: 12 total, 12 up, 11 in 2024-02-09T18:00:47.415 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:47 smithi099 ceph-mon[39420]: 1.1a scrub starts 2024-02-09T18:00:47.415 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:47 smithi099 ceph-mon[39420]: 1.1a scrub ok 2024-02-09T18:00:47.586 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 28 16 64250 64234 143.322 96.4375 0.00847442 0.00697355 2024-02-09T18:00:47.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:47 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:00:47.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:47 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:00:47.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:47 smithi179 ceph-mon[43975]: osdmap e899: 12 total, 12 up, 11 in 2024-02-09T18:00:47.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:47 smithi179 ceph-mon[43975]: 1.1a scrub starts 2024-02-09T18:00:47.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:47 smithi179 ceph-mon[43975]: 1.1a scrub ok 2024-02-09T18:00:47.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:47 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:00:47.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:47 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:00:47.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:47 smithi162 ceph-mon[44304]: osdmap e899: 12 total, 12 up, 11 in 2024-02-09T18:00:47.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:47 smithi162 ceph-mon[44304]: 1.1a scrub starts 2024-02-09T18:00:47.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:47 smithi162 ceph-mon[44304]: 1.1a scrub ok 2024-02-09T18:00:48.576 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:48 smithi099 ceph-mon[39420]: 1.1 deep-scrub starts 2024-02-09T18:00:48.576 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:48 smithi099 ceph-mon[39420]: 1.1 deep-scrub ok 2024-02-09T18:00:48.576 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:48 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:00:48.576 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:48 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:00:48.576 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:48 smithi099 ceph-mon[39420]: pgmap v2220: 47 pgs: 47 active+clean; 3.6 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-02-09T18:00:48.576 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:48 smithi099 ceph-mon[39420]: osdmap e900: 12 total, 12 up, 11 in 2024-02-09T18:00:48.586 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 29 16 65790 65774 141.699 96.25 0.00747227 0.00705303 2024-02-09T18:00:48.604 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:00:48.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:48.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:00:48.614 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 9 1 2024-02-09T18:00:48.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:48 smithi162 ceph-mon[44304]: 1.1 deep-scrub starts 2024-02-09T18:00:48.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:48 smithi162 ceph-mon[44304]: 1.1 deep-scrub ok 2024-02-09T18:00:48.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:48 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:00:48.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:48 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:00:48.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:48 smithi162 ceph-mon[44304]: pgmap v2220: 47 pgs: 47 active+clean; 3.6 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-02-09T18:00:48.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:48 smithi162 ceph-mon[44304]: osdmap e900: 12 total, 12 up, 11 in 2024-02-09T18:00:48.711 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:00:48.746 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:48.770 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:00:48.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:48.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:00:48.876 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:00:48.915 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:48.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:48 smithi179 ceph-mon[43975]: 1.1 deep-scrub starts 2024-02-09T18:00:48.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:48 smithi179 ceph-mon[43975]: 1.1 deep-scrub ok 2024-02-09T18:00:48.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:48 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:00:48.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:48 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:00:48.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:48 smithi179 ceph-mon[43975]: pgmap v2220: 47 pgs: 47 active+clean; 3.6 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-02-09T18:00:48.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:48 smithi179 ceph-mon[43975]: osdmap e900: 12 total, 12 up, 11 in 2024-02-09T18:00:48.997 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:00:49.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:49.008 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:00:49.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:00:49.047 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-09T18:00:49.110 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T18:00:49.152 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:49.183 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T18:00:49.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:49.189 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T18:00:49.290 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:00:49.350 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:49.437 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:00:49.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:49.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:00:49.545 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:00:49.586 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 30 16 67220 67204 139.954 89.375 0.0125633 0.00714057 2024-02-09T18:00:49.589 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:49.773 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:00:49.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:49.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:00:49.877 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:00:49.919 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:50.451 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is set 2024-02-09T18:00:50.587 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 31 16 68518 68502 138.055 81.125 0.00982378 0.00723927 2024-02-09T18:00:50.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:50 smithi099 ceph-mon[39420]: pgmap v2222: 47 pgs: 47 active+clean; 3.9 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 112 MiB/s wr, 1.78k op/s 2024-02-09T18:00:50.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:50 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1562469548' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T18:00:50.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:50 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T18:00:50.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:50 smithi179 ceph-mon[43975]: pgmap v2222: 47 pgs: 47 active+clean; 3.9 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 112 MiB/s wr, 1.78k op/s 2024-02-09T18:00:50.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:50 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1562469548' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T18:00:50.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:50 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T18:00:50.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:50 smithi162 ceph-mon[44304]: pgmap v2222: 47 pgs: 47 active+clean; 3.9 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 112 MiB/s wr, 1.78k op/s 2024-02-09T18:00:50.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:50 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1562469548' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T18:00:50.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:50 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T18:00:51.588 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 32 16 69981 69965 136.595 91.4375 0.00945113 0.00731666 2024-02-09T18:00:51.695 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:51 smithi099 ceph-mon[39420]: 1.d deep-scrub starts 2024-02-09T18:00:51.695 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:51 smithi099 ceph-mon[39420]: 1.d deep-scrub ok 2024-02-09T18:00:51.696 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:51 smithi099 ceph-mon[39420]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:00:51.696 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:51 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T18:00:51.696 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:51 smithi099 ceph-mon[39420]: osdmap e901: 12 total, 12 up, 11 in 2024-02-09T18:00:51.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:51 smithi179 ceph-mon[43975]: 1.d deep-scrub starts 2024-02-09T18:00:51.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:51 smithi179 ceph-mon[43975]: 1.d deep-scrub ok 2024-02-09T18:00:51.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:51 smithi179 ceph-mon[43975]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:00:51.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:51 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T18:00:51.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:51 smithi179 ceph-mon[43975]: osdmap e901: 12 total, 12 up, 11 in 2024-02-09T18:00:51.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:51 smithi162 ceph-mon[44304]: 1.d deep-scrub starts 2024-02-09T18:00:51.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:51 smithi162 ceph-mon[44304]: 1.d deep-scrub ok 2024-02-09T18:00:51.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:51 smithi162 ceph-mon[44304]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:00:51.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:51 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T18:00:51.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:51 smithi162 ceph-mon[44304]: osdmap e901: 12 total, 12 up, 11 in 2024-02-09T18:00:52.444 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:52 smithi099 ceph-mon[39420]: pgmap v2224: 47 pgs: 47 active+clean; 4.1 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 121 MiB/s wr, 1.94k op/s 2024-02-09T18:00:52.585 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:00:52.588 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 33 15 71486 71471 135.308 94.125 0.0143773 0.0073865 2024-02-09T18:00:52.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:52.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:00:52.690 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:00:52.733 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:52.900 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:00:52.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:52.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:00:52.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:52 smithi179 ceph-mon[43975]: pgmap v2224: 47 pgs: 47 active+clean; 4.1 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 121 MiB/s wr, 1.94k op/s 2024-02-09T18:00:52.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:52 smithi162 ceph-mon[44304]: pgmap v2224: 47 pgs: 47 active+clean; 4.1 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 121 MiB/s wr, 1.94k op/s 2024-02-09T18:00:52.962 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:00:52.970 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:00:53.004 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T18:00:53.053 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:53.081 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T18:00:53.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:53.087 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T18:00:53.188 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T18:00:53.227 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:53.253 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T18:00:53.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:53.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T18:00:53.360 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T18:00:53.410 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:53.440 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T18:00:53.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:53.443 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T18:00:53.476 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T18:00:53.502 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:53 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2992919210' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:00:53.511 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T18:00:53.544 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:00:53.546 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T18:00:53.571 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T18:00:53.585 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:53.588 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 34 16 72994 72978 134.099 94.1875 0.0119628 0.00745216 2024-02-09T18:00:53.601 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T18:00:53.612 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:00:53.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:53.616 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:00:53.632 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T18:00:53.663 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T18:00:53.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:53 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2992919210' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:00:53.691 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T18:00:53.717 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:00:53.733 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T18:00:53.755 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:53.766 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T18:00:53.782 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:00:53.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:53.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:00:53.803 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T18:00:53.837 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T18:00:53.842 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T18:00:53.842 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T18:00:53.842 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T18:00:53.842 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T18:00:53.842 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T18:00:53.842 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T18:00:53.842 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T18:00:53.842 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T18:00:53.843 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T18:00:53.843 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T18:00:53.843 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T18:00:53.843 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T18:00:53.865 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:53 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2992919210' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:00:53.890 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:00:53.953 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:54.588 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 35 16 74465 74449 132.894 91.9375 0.0098709 0.00752045 2024-02-09T18:00:54.603 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-09T18:00:54.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:54 smithi179 ceph-mon[43975]: pgmap v2225: 47 pgs: 47 active+clean; 4.2 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-02-09T18:00:54.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:54 smithi162 ceph-mon[44304]: pgmap v2225: 47 pgs: 47 active+clean; 4.2 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-02-09T18:00:54.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:54 smithi099 ceph-mon[39420]: pgmap v2225: 47 pgs: 47 active+clean; 4.2 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-02-09T18:00:55.588 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 36 16 75991 75975 131.852 95.375 0.00491106 0.0075803 2024-02-09T18:00:56.591 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 37 16 77482 77466 130.796 93.1875 0.0142662 0.00764155 2024-02-09T18:00:56.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:56 smithi099 ceph-mon[39420]: pgmap v2226: 47 pgs: 47 active+clean; 4.3 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 90 MiB/s wr, 1.43k op/s 2024-02-09T18:00:56.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:56 smithi179 ceph-mon[43975]: pgmap v2226: 47 pgs: 47 active+clean; 4.3 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 90 MiB/s wr, 1.43k op/s 2024-02-09T18:00:56.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:56 smithi162 ceph-mon[44304]: pgmap v2226: 47 pgs: 47 active+clean; 4.3 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 90 MiB/s wr, 1.43k op/s 2024-02-09T18:00:57.592 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 38 16 79003 78987 129.854 95.0625 0.00876763 0.00769629 2024-02-09T18:00:57.813 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:00:57.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:57.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:00:57.920 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T18:00:57.960 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:57.994 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T18:00:57.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:57.999 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T18:00:58.100 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:00:58.144 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:58.176 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:00:58.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:58.183 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:00:58.284 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:00:58.320 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:58.350 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:00:58.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:00:58.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:00:58.462 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:00:58.506 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:00:58.592 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 39 16 80433 80417 128.816 89.375 0.00889237 0.0077589 2024-02-09T18:00:58.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:00:58 smithi162 ceph-mon[44304]: pgmap v2227: 47 pgs: 47 active+clean; 4.5 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 97 MiB/s wr, 1.54k op/s 2024-02-09T18:00:58.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:00:58 smithi179 ceph-mon[43975]: pgmap v2227: 47 pgs: 47 active+clean; 4.5 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 97 MiB/s wr, 1.54k op/s 2024-02-09T18:00:58.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:00:58 smithi099 ceph-mon[39420]: pgmap v2227: 47 pgs: 47 active+clean; 4.5 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 97 MiB/s wr, 1.54k op/s 2024-02-09T18:00:59.266 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T18:00:59.266 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T18:00:59.583 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T18:00:59.592 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:2024-02-09T18:00:59.592090+0000 min lat: 0.00108495 max lat: 0.218355 avg lat: 0.0078157 2024-02-09T18:00:59.592 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-09T18:00:59.592 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 40 16 81889 81873 127.871 91 0.00739651 0.0078157 2024-02-09T18:01:00.593 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 41 15 83190 83175 126.737 81.375 0.00591199 0.00788628 2024-02-09T18:01:00.711 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:01:00.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:00.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:01:00.816 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T18:01:00.844 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:00 smithi099 ceph-mon[39420]: pgmap v2228: 47 pgs: 47 active+clean; 4.8 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 92 MiB/s wr, 1.48k op/s 2024-02-09T18:01:00.850 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:00 smithi162 ceph-mon[44304]: pgmap v2228: 47 pgs: 47 active+clean; 4.8 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 92 MiB/s wr, 1.48k op/s 2024-02-09T18:01:00.856 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:00.882 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T18:01:00.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:00.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T18:01:00.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:00 smithi179 ceph-mon[43975]: pgmap v2228: 47 pgs: 47 active+clean; 4.8 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 92 MiB/s wr, 1.48k op/s 2024-02-09T18:01:00.988 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:01:01.036 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:01.593 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 42 16 84581 84565 125.787 86.875 0.017299 0.00794575 2024-02-09T18:01:01.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:01 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:01:01.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:01 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:01:01.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:01 smithi179 ceph-mon[43975]: osdmap e902: 12 total, 12 up, 11 in 2024-02-09T18:01:01.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:01 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:01:01.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:01 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:01:01.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:01 smithi162 ceph-mon[44304]: osdmap e902: 12 total, 12 up, 11 in 2024-02-09T18:01:01.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:01 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:01:01.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:01 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:01:01.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:01 smithi099 ceph-mon[39420]: osdmap e902: 12 total, 12 up, 11 in 2024-02-09T18:01:02.400 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:01:02.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:02.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:01:02.416 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-09T18:01:02.428 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:01:02.461 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:01:02.507 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:01:02.551 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:02.592 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.9 to 1 (10000) 2024-02-09T18:01:02.593 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 43 15 85793 85778 124.625 75.8125 0.0057538 0.00801944 2024-02-09T18:01:02.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:02 smithi099 ceph-mon[39420]: pgmap v2230: 47 pgs: 47 active+clean; 5.0 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-02-09T18:01:02.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:02 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:01:02.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:02 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:01:02.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:02 smithi099 ceph-mon[39420]: osdmap e903: 12 total, 12 up, 11 in 2024-02-09T18:01:02.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:02 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1859397683' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-02-09T18:01:02.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:02 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-02-09T18:01:02.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:02 smithi179 ceph-mon[43975]: pgmap v2230: 47 pgs: 47 active+clean; 5.0 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-02-09T18:01:02.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:02 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:01:02.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:02 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:01:02.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:02 smithi179 ceph-mon[43975]: osdmap e903: 12 total, 12 up, 11 in 2024-02-09T18:01:02.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:02 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1859397683' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-02-09T18:01:02.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:02 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-02-09T18:01:02.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:02 smithi162 ceph-mon[44304]: pgmap v2230: 47 pgs: 47 active+clean; 5.0 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-02-09T18:01:02.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:02 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:01:02.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:02 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:01:02.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:02 smithi162 ceph-mon[44304]: osdmap e903: 12 total, 12 up, 11 in 2024-02-09T18:01:02.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:02 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1859397683' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-02-09T18:01:02.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:02 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-02-09T18:01:03.465 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:01:03.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:03.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:01:03.576 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T18:01:03.593 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 44 16 87155 87139 123.726 85.0625 0.00593161 0.00807808 2024-02-09T18:01:03.615 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:03.648 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T18:01:03.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:03.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T18:01:03.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:03 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-02-09T18:01:03.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:03 smithi099 ceph-mon[39420]: osdmap e904: 12 total, 12 up, 11 in 2024-02-09T18:01:03.753 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:01:03.795 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:03.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:03 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-02-09T18:01:03.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:03 smithi179 ceph-mon[43975]: osdmap e904: 12 total, 12 up, 11 in 2024-02-09T18:01:03.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:03 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-02-09T18:01:03.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:03 smithi162 ceph-mon[44304]: osdmap e904: 12 total, 12 up, 11 in 2024-02-09T18:01:04.593 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 45 16 88443 88427 122.766 80.5 0.0164168 0.00814121 2024-02-09T18:01:04.905 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:04 smithi099 ceph-mon[39420]: pgmap v2233: 47 pgs: 47 active+clean; 5.1 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-02-09T18:01:04.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:04 smithi179 ceph-mon[43975]: pgmap v2233: 47 pgs: 47 active+clean; 5.1 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-02-09T18:01:04.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:04 smithi162 ceph-mon[44304]: pgmap v2233: 47 pgs: 47 active+clean; 5.1 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-02-09T18:01:04.943 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 10 1 2024-02-09T18:01:05.056 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:01:05.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:05.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:01:05.161 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:01:05.204 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:05.240 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:01:05.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:05.243 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:01:05.344 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:01:05.394 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:05.593 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 46 16 89813 89797 121.959 85.625 0.00493823 0.00819592 2024-02-09T18:01:05.756 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:01:05.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:05.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:01:05.798 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:01:05.865 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:01:05.905 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:05.937 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:01:05.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:05.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:01:06.043 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:01:06.097 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:06.128 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:01:06.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:06.135 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:01:06.236 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:01:06.252 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:01:06.267 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:06.291 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:01:06.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:06.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:01:06.395 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T18:01:06.445 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:06.469 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T18:01:06.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:06.473 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T18:01:06.574 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T18:01:06.581 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:06 smithi099 ceph-mon[39420]: pgmap v2234: 47 pgs: 47 active+clean; 5.2 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-02-09T18:01:06.593 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 47 16 91112 91096 121.091 81.1875 0.0178824 0.00825387 2024-02-09T18:01:06.611 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:06.641 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T18:01:06.647 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:06.648 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T18:01:06.749 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:01:06.798 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:06.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:06 smithi179 ceph-mon[43975]: pgmap v2234: 47 pgs: 47 active+clean; 5.2 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-02-09T18:01:06.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:06 smithi162 ceph-mon[44304]: pgmap v2234: 47 pgs: 47 active+clean; 5.2 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-02-09T18:01:07.595 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 48 16 92374 92358 120.207 78.875 0.00700858 0.00831409 2024-02-09T18:01:08.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:08 smithi179 ceph-mon[43975]: osdmap e905: 12 total, 12 up, 11 in 2024-02-09T18:01:08.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:08 smithi179 ceph-mon[43975]: pgmap v2236: 47 pgs: 47 active+clean; 5.4 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-02-09T18:01:08.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:08 smithi162 ceph-mon[44304]: osdmap e905: 12 total, 12 up, 11 in 2024-02-09T18:01:08.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:08 smithi162 ceph-mon[44304]: pgmap v2236: 47 pgs: 47 active+clean; 5.4 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-02-09T18:01:08.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:08 smithi099 ceph-mon[39420]: osdmap e905: 12 total, 12 up, 11 in 2024-02-09T18:01:08.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:08 smithi099 ceph-mon[39420]: pgmap v2236: 47 pgs: 47 active+clean; 5.4 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-02-09T18:01:08.595 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 49 16 93662 93646 119.397 80.5 0.0113618 0.00837182 2024-02-09T18:01:09.477 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T18:01:09.509 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T18:01:09.540 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T18:01:09.578 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T18:01:09.595 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 50 16 94846 94830 118.489 74 0.00886413 0.00843534 2024-02-09T18:01:09.609 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T18:01:09.630 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T18:01:09.650 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T18:01:09.676 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:01:09.677 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T18:01:09.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:09.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:01:09.702 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T18:01:09.722 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T18:01:09.741 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T18:01:09.761 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T18:01:09.762 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T18:01:09.762 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T18:01:09.763 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T18:01:09.763 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T18:01:09.763 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T18:01:09.763 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T18:01:09.763 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T18:01:09.763 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T18:01:09.763 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T18:01:09.763 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T18:01:09.763 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T18:01:09.763 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T18:01:09.782 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:01:09.817 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:09.843 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:01:09.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:09.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:01:09.948 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:01:10.010 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:10.045 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:01:10.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:10.051 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:01:10.152 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T18:01:10.195 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:10.220 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T18:01:10.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:10.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T18:01:10.325 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:01:10.353 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:10 smithi099 ceph-mon[39420]: pgmap v2237: 47 pgs: 47 active+clean; 5.6 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-02-09T18:01:10.353 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:10 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2722662052' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:01:10.353 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:10 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/770624712' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T18:01:10.360 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is set 2024-02-09T18:01:10.379 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:10.595 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 51 16 95884 95868 117.438 64.875 0.0151319 0.00851026 2024-02-09T18:01:10.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:10 smithi179 ceph-mon[43975]: pgmap v2237: 47 pgs: 47 active+clean; 5.6 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-02-09T18:01:10.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:10 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2722662052' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:01:10.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:10 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/770624712' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T18:01:10.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:10 smithi162 ceph-mon[44304]: pgmap v2237: 47 pgs: 47 active+clean; 5.6 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-02-09T18:01:10.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:10 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2722662052' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:01:10.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:10 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/770624712' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T18:01:11.546 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:01:11.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:11.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:01:11.598 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 52 16 97033 97017 116.555 71.8125 0.00947225 0.00857576 2024-02-09T18:01:11.656 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:01:11.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:11 smithi179 ceph-mon[43975]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:01:11.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:11 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/770624712' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T18:01:11.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:11 smithi179 ceph-mon[43975]: osdmap e906: 12 total, 12 up, 11 in 2024-02-09T18:01:11.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:11 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T18:01:11.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:11 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T18:01:11.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:11 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T18:01:11.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:11 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T18:01:11.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:11 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T18:01:11.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:11 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T18:01:11.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:11 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T18:01:11.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:11 smithi162 ceph-mon[44304]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:01:11.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:11 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/770624712' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T18:01:11.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:11 smithi162 ceph-mon[44304]: osdmap e906: 12 total, 12 up, 11 in 2024-02-09T18:01:11.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:11 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T18:01:11.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:11 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T18:01:11.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:11 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T18:01:11.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:11 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T18:01:11.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:11 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T18:01:11.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:11 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T18:01:11.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:11 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T18:01:11.704 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:11.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:11 smithi099 ceph-mon[39420]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:01:11.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:11 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/770624712' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T18:01:11.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:11 smithi099 ceph-mon[39420]: osdmap e906: 12 total, 12 up, 11 in 2024-02-09T18:01:11.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:11 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T18:01:11.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:11 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T18:01:11.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:11 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T18:01:11.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:11 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T18:01:11.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:11 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T18:01:11.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:11 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T18:01:11.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:11 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T18:01:11.746 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:01:11.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:11.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:01:11.852 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T18:01:11.893 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:11.923 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T18:01:11.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:11.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T18:01:12.028 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:01:12.067 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:12.249 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:01:12.253 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:12.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:01:12.354 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:01:12.402 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:12.598 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 53 16 98314 98298 115.867 80.0625 0.0105822 0.00862625 2024-02-09T18:01:12.615 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:01:12.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:12.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:01:12.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:12 smithi162 ceph-mon[44304]: pgmap v2239: 47 pgs: 47 active+clean; 5.8 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-02-09T18:01:12.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:12 smithi162 ceph-mon[44304]: osdmap e907: 12 total, 12 up, 11 in 2024-02-09T18:01:12.719 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:01:12.724 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:12 smithi099 ceph-mon[39420]: pgmap v2239: 47 pgs: 47 active+clean; 5.8 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-02-09T18:01:12.725 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:12 smithi099 ceph-mon[39420]: osdmap e907: 12 total, 12 up, 11 in 2024-02-09T18:01:12.765 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:12 smithi179 ceph-mon[43975]: pgmap v2239: 47 pgs: 47 active+clean; 5.8 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-02-09T18:01:12.765 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:12 smithi179 ceph-mon[43975]: osdmap e907: 12 total, 12 up, 11 in 2024-02-09T18:01:12.775 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:12.810 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:01:12.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:12.823 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:01:12.924 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:01:12.965 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:13.598 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 54 16 99559 99543 115.162 77.8125 0.00506928 0.00867915 2024-02-09T18:01:14.037 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:01:14.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:14.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:01:14.148 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:01:14.192 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T18:01:14.193 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [], 2024-02-09T18:01:14.193 INFO:teuthology.orchestra.run.smithi099.stdout: "complaint_time": 30, 2024-02-09T18:01:14.193 INFO:teuthology.orchestra.run.smithi099.stdout: "num_blocked_ops": 0 2024-02-09T18:01:14.193 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T18:01:14.204 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:14.230 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:01:14.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:14.236 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:01:14.337 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:01:14.395 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:14.444 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.10 to 1 (10000) 2024-02-09T18:01:14.598 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 55 16 100764 100748 114.438 75.3125 0.00621126 0.00873351 2024-02-09T18:01:14.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:14 smithi179 ceph-mon[43975]: pgmap v2241: 47 pgs: 47 active+clean; 5.9 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 83 MiB/s wr, 1.32k op/s 2024-02-09T18:01:14.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:14 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/200537283' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-02-09T18:01:14.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:14 smithi099 ceph-mon[39420]: pgmap v2241: 47 pgs: 47 active+clean; 5.9 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 83 MiB/s wr, 1.32k op/s 2024-02-09T18:01:14.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:14 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/200537283' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-02-09T18:01:14.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:14 smithi162 ceph-mon[44304]: pgmap v2241: 47 pgs: 47 active+clean; 5.9 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 83 MiB/s wr, 1.32k op/s 2024-02-09T18:01:14.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:14 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/200537283' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-02-09T18:01:15.140 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T18:01:15.598 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 56 16 101866 101850 113.624 68.875 0.0111283 0.00879686 2024-02-09T18:01:15.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:15 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/200537283' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-02-09T18:01:15.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:15 smithi099 ceph-mon[39420]: osdmap e908: 12 total, 12 up, 11 in 2024-02-09T18:01:15.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:15 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/200537283' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-02-09T18:01:15.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:15 smithi179 ceph-mon[43975]: osdmap e908: 12 total, 12 up, 11 in 2024-02-09T18:01:15.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:15 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/200537283' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-02-09T18:01:15.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:15 smithi162 ceph-mon[44304]: osdmap e908: 12 total, 12 up, 11 in 2024-02-09T18:01:16.602 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 57 15 102920 102905 112.78 65.9375 0.00436566 0.00886256 2024-02-09T18:01:16.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:16 smithi099 ceph-mon[39420]: pgmap v2243: 47 pgs: 47 active+clean; 6.0 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-02-09T18:01:16.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:16 smithi179 ceph-mon[43975]: pgmap v2243: 47 pgs: 47 active+clean; 6.0 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-02-09T18:01:16.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:16 smithi162 ceph-mon[44304]: pgmap v2243: 47 pgs: 47 active+clean; 6.0 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-02-09T18:01:17.082 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:01:17.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:17.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:01:17.124 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 11 1 2024-02-09T18:01:17.188 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:01:17.225 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:17.252 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:01:17.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:17.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:01:17.361 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:01:17.408 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:17.441 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset noscrub 2024-02-09T18:01:17.444 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:01:17.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:17.451 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:01:17.552 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:01:17.580 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-09T18:01:17.582 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:17.602 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 58 16 104119 104103 112.127 74.875 0.00874718 0.00891502 2024-02-09T18:01:17.607 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:01:17.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:17.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:01:17.697 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:01:17.712 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:01:17.761 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:01:17.802 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:18.016 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:01:18.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:18.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:01:18.055 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:18 smithi099 ceph-mon[39420]: osdmap e909: 12 total, 12 up, 11 in 2024-02-09T18:01:18.055 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:18 smithi099 ceph-mon[39420]: pgmap v2245: 47 pgs: 47 active+clean; 6.1 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-02-09T18:01:18.128 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T18:01:18.168 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:18.200 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T18:01:18.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:18.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T18:01:18.305 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:01:18.356 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:18 smithi179 ceph-mon[43975]: osdmap e909: 12 total, 12 up, 11 in 2024-02-09T18:01:18.356 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:18 smithi179 ceph-mon[43975]: pgmap v2245: 47 pgs: 47 active+clean; 6.1 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-02-09T18:01:18.359 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:18.394 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:01:18.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:18.401 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:01:18.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:18 smithi162 ceph-mon[44304]: osdmap e909: 12 total, 12 up, 11 in 2024-02-09T18:01:18.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:18 smithi162 ceph-mon[44304]: pgmap v2245: 47 pgs: 47 active+clean; 6.1 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-02-09T18:01:18.502 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T18:01:18.544 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:18.573 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T18:01:18.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:18.576 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T18:01:18.602 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 59 16 105204 105188 111.376 67.8125 0.0151609 0.00897457 2024-02-09T18:01:18.676 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:01:18.707 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:19.603 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:2024-02-09T18:01:19.602393+0000 min lat: 0.00108495 max lat: 0.222372 avg lat: 0.00902211 2024-02-09T18:01:19.603 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-09T18:01:19.603 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 60 16 106418 106402 110.784 75.875 0.00847972 0.00902211 2024-02-09T18:01:20.374 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:01:20.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:20.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:01:20.480 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:01:20.535 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:20.603 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 61 16 107636 107620 110.216 76.125 0.0107681 0.00906946 2024-02-09T18:01:20.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:20 smithi179 ceph-mon[43975]: pgmap v2246: 47 pgs: 47 active+clean; 6.3 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-02-09T18:01:20.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:20 smithi162 ceph-mon[44304]: pgmap v2246: 47 pgs: 47 active+clean; 6.3 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-02-09T18:01:20.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:20 smithi099 ceph-mon[39420]: pgmap v2246: 47 pgs: 47 active+clean; 6.3 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-02-09T18:01:20.898 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:01:20.909 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:20.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:01:21.010 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T18:01:21.052 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:21.084 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T18:01:21.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:21.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T18:01:21.189 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:01:21.224 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:21.254 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:01:21.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:21.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:01:21.358 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:01:21.403 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:21.443 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:01:21.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:21.446 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:01:21.547 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:01:21.582 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:21.603 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 62 16 108782 108766 109.594 71.625 0.01892 0.00912077 2024-02-09T18:01:21.612 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:01:21.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:21.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:01:21.719 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:01:21.754 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:21.808 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:01:21.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:21.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:01:21.914 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T18:01:21.975 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:22.011 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T18:01:22.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:22.015 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T18:01:22.116 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:01:22.142 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:22.173 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:01:22.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:22.177 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:01:22.278 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:01:22.320 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:22.350 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:01:22.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:22.353 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:01:22.454 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:01:22.495 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:22.529 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:01:22.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:22.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:01:22.556 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:01:22.558 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:01:22.603 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 63 16 109648 109632 108.714 54.125 0.0128635 0.0091934 2024-02-09T18:01:22.621 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T18:01:22.643 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T18:01:22.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:22 smithi179 ceph-mon[43975]: pgmap v2247: 47 pgs: 47 active+clean; 6.5 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-02-09T18:01:22.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:22 smithi162 ceph-mon[44304]: pgmap v2247: 47 pgs: 47 active+clean; 6.5 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-02-09T18:01:22.678 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:22.698 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T18:01:22.704 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T18:01:22.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:22.717 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T18:01:22.734 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T18:01:22.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:22 smithi099 ceph-mon[39420]: pgmap v2247: 47 pgs: 47 active+clean; 6.5 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-02-09T18:01:22.784 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T18:01:22.818 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:01:22.828 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T18:01:22.859 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:22.867 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T18:01:22.889 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:01:22.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:22.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:01:22.905 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T18:01:22.937 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T18:01:22.971 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T18:01:22.998 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:01:23.001 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T18:01:23.035 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T18:01:23.058 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:23.064 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T18:01:23.066 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T18:01:23.066 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T18:01:23.066 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T18:01:23.066 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T18:01:23.066 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T18:01:23.066 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T18:01:23.066 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T18:01:23.066 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T18:01:23.066 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T18:01:23.066 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T18:01:23.066 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T18:01:23.066 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T18:01:23.091 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:01:23.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:23.094 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:01:23.195 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:01:23.243 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:23.269 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:01:23.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:23.276 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:01:23.370 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.11 to 1 (10000) 2024-02-09T18:01:23.377 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:01:23.408 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:23.437 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:01:23.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:23.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:01:23.544 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T18:01:23.590 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:23.603 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 64 16 110693 110677 108.036 65.3125 0.0215842 0.00925236 2024-02-09T18:01:23.614 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T18:01:23.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:23.617 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T18:01:23.618 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:23 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1762951950' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:01:23.618 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:23 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1002741058' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-02-09T18:01:23.618 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:23 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-02-09T18:01:23.618 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:23 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1762951950' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:01:23.618 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:23 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1002741058' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-02-09T18:01:23.618 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:23 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-02-09T18:01:23.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:23 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1762951950' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:01:23.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:23 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1002741058' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-02-09T18:01:23.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:23 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-02-09T18:01:23.718 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:01:23.767 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:23.798 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:01:23.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:23.803 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:01:23.904 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:01:23.949 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:23.975 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:01:23.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:23.980 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:01:24.080 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:01:24.132 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:24.364 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:24 smithi179 ceph-mon[43975]: pgmap v2248: 47 pgs: 47 active+clean; 6.6 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-02-09T18:01:24.364 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:24 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-02-09T18:01:24.364 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:24 smithi179 ceph-mon[43975]: osdmap e910: 12 total, 12 up, 11 in 2024-02-09T18:01:24.603 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 65 16 111837 111821 107.474 71.5 0.006816 0.00930098 2024-02-09T18:01:24.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:24 smithi162 ceph-mon[44304]: pgmap v2248: 47 pgs: 47 active+clean; 6.6 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-02-09T18:01:24.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:24 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-02-09T18:01:24.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:24 smithi162 ceph-mon[44304]: osdmap e910: 12 total, 12 up, 11 in 2024-02-09T18:01:24.692 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:01:24.697 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:24 smithi099 ceph-mon[39420]: pgmap v2248: 47 pgs: 47 active+clean; 6.6 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-02-09T18:01:24.697 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:24 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-02-09T18:01:24.697 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:24 smithi099 ceph-mon[39420]: osdmap e910: 12 total, 12 up, 11 in 2024-02-09T18:01:24.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:24.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:01:24.799 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:01:24.838 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:24.947 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:01:24.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:24.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:01:25.052 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T18:01:25.125 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:25.156 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T18:01:25.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:25.161 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T18:01:25.262 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:01:25.305 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:25.603 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 66 16 112916 112900 106.867 67.4375 0.0140008 0.00935319 2024-02-09T18:01:26.262 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:01:26.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:26.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:01:26.369 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T18:01:26.420 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:26.449 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T18:01:26.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:26.455 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T18:01:26.556 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T18:01:26.603 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 67 16 113904 113888 106.194 61.75 0.0089911 0.00941319 2024-02-09T18:01:26.609 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:26.634 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T18:01:26.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:26.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T18:01:26.640 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:26 smithi162 ceph-mon[44304]: pgmap v2250: 47 pgs: 47 active+clean; 6.7 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-02-09T18:01:26.646 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:26 smithi099 ceph-mon[39420]: pgmap v2250: 47 pgs: 47 active+clean; 6.7 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-02-09T18:01:26.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:26 smithi179 ceph-mon[43975]: pgmap v2250: 47 pgs: 47 active+clean; 6.7 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-02-09T18:01:26.678 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 0 1 2024-02-09T18:01:26.740 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:01:26.779 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:26.809 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:01:26.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:26.818 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:01:26.920 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:01:26.970 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:27.000 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:01:27.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:27.006 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:01:27.107 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:01:27.148 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:27.237 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:01:27.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:27.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:01:27.347 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:01:27.400 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:27.450 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:01:27.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:27.456 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:01:27.557 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T18:01:27.603 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 68 16 114995 114979 105.635 68.1875 0.00670349 0.00946253 2024-02-09T18:01:27.605 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:27.634 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T18:01:27.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:27.642 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T18:01:27.743 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T18:01:27.775 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:27.814 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T18:01:27.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:27.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T18:01:27.919 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T18:01:27.960 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:27.987 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T18:01:27.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:27.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T18:01:28.095 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:01:28.149 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:28.284 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:01:28.286 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:01:28.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:28.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:01:28.392 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:01:28.431 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:28 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:01:28.432 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:28 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:01:28.432 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:28 smithi099 ceph-mon[39420]: pgmap v2251: 47 pgs: 47 active+clean; 6.8 GiB data, 29 GiB used, 954 GiB / 983 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-02-09T18:01:28.432 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:28 smithi099 ceph-mon[39420]: osdmap e911: 12 total, 12 up, 11 in 2024-02-09T18:01:28.441 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:28.607 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 69 16 116103 116087 105.102 69.25 0.01408 0.00951048 2024-02-09T18:01:28.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:28 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:01:28.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:28 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:01:28.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:28 smithi179 ceph-mon[43975]: pgmap v2251: 47 pgs: 47 active+clean; 6.8 GiB data, 29 GiB used, 954 GiB / 983 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-02-09T18:01:28.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:28 smithi179 ceph-mon[43975]: osdmap e911: 12 total, 12 up, 11 in 2024-02-09T18:01:28.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:28 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:01:28.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:28 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:01:28.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:28 smithi162 ceph-mon[44304]: pgmap v2251: 47 pgs: 47 active+clean; 6.8 GiB data, 29 GiB used, 954 GiB / 983 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-02-09T18:01:28.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:28 smithi162 ceph-mon[44304]: osdmap e911: 12 total, 12 up, 11 in 2024-02-09T18:01:29.113 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T18:01:29.208 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:01:29.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:29.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:01:29.317 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:01:29.347 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:29.373 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:01:29.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:29.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:01:29.443 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T18:01:29.443 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T18:01:29.480 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T18:01:29.512 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:29.544 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T18:01:29.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:29.551 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T18:01:29.607 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 70 16 117131 117115 104.519 64.25 0.00488634 0.00956333 2024-02-09T18:01:29.652 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T18:01:29.699 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:29.725 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T18:01:29.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:29.731 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T18:01:29.832 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:01:29.868 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:29.903 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:01:29.909 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:29.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:01:30.010 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T18:01:30.057 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:30.099 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T18:01:30.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:30.105 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T18:01:30.206 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:01:30.248 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:30.396 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is unset 2024-02-09T18:01:30.607 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 71 16 118213 118197 103.999 67.625 0.0131708 0.00961132 2024-02-09T18:01:30.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:30 smithi179 ceph-mon[43975]: pgmap v2253: 47 pgs: 47 active+clean; 7.0 GiB data, 29 GiB used, 954 GiB / 983 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-02-09T18:01:30.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:30 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1172470279' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T18:01:30.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:30 smithi162 ceph-mon[44304]: pgmap v2253: 47 pgs: 47 active+clean; 7.0 GiB data, 29 GiB used, 954 GiB / 983 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-02-09T18:01:30.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:30 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1172470279' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T18:01:30.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:30 smithi099 ceph-mon[39420]: pgmap v2253: 47 pgs: 47 active+clean; 7.0 GiB data, 29 GiB used, 954 GiB / 983 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-02-09T18:01:30.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:30 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1172470279' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T18:01:31.607 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 72 16 119291 119275 103.491 67.375 0.016852 0.00965857 2024-02-09T18:01:31.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:31 smithi179 ceph-mon[43975]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:01:31.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:31 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1172470279' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T18:01:31.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:31 smithi179 ceph-mon[43975]: osdmap e912: 12 total, 12 up, 11 in 2024-02-09T18:01:31.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:31 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:01:31.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:31 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:01:31.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:31 smithi162 ceph-mon[44304]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:01:31.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:31 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1172470279' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T18:01:31.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:31 smithi162 ceph-mon[44304]: osdmap e912: 12 total, 12 up, 11 in 2024-02-09T18:01:31.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:31 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:01:31.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:31 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:01:31.689 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:31 smithi099 ceph-mon[39420]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:01:31.689 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:31 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1172470279' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T18:01:31.689 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:31 smithi099 ceph-mon[39420]: osdmap e912: 12 total, 12 up, 11 in 2024-02-09T18:01:31.689 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:31 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:01:31.689 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:31 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:01:32.196 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-09T18:01:32.607 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 73 16 120216 120200 102.865 57.8125 0.00621694 0.00971697 2024-02-09T18:01:32.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:32 smithi179 ceph-mon[43975]: pgmap v2255: 47 pgs: 47 active+clean; 7.1 GiB data, 30 GiB used, 954 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-02-09T18:01:32.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:32 smithi179 ceph-mon[43975]: osdmap e913: 12 total, 12 up, 11 in 2024-02-09T18:01:32.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:32 smithi179 ceph-mon[43975]: 1.18 scrub starts 2024-02-09T18:01:32.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:32 smithi162 ceph-mon[44304]: pgmap v2255: 47 pgs: 47 active+clean; 7.1 GiB data, 30 GiB used, 954 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-02-09T18:01:32.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:32 smithi162 ceph-mon[44304]: osdmap e913: 12 total, 12 up, 11 in 2024-02-09T18:01:32.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:32 smithi162 ceph-mon[44304]: 1.18 scrub starts 2024-02-09T18:01:32.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:32 smithi099 ceph-mon[39420]: pgmap v2255: 47 pgs: 47 active+clean; 7.1 GiB data, 30 GiB used, 954 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-02-09T18:01:32.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:32 smithi099 ceph-mon[39420]: osdmap e913: 12 total, 12 up, 11 in 2024-02-09T18:01:32.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:32 smithi099 ceph-mon[39420]: 1.18 scrub starts 2024-02-09T18:01:33.288 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:01:33.324 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:01:33.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:33.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:01:33.432 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T18:01:33.475 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:33.505 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T18:01:33.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:33.509 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T18:01:33.608 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 74 16 121166 121150 102.277 59.375 0.0147228 0.00977303 2024-02-09T18:01:33.610 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:01:33.623 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:33 smithi099 ceph-mon[39420]: 1.18 scrub ok 2024-02-09T18:01:33.623 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:33 smithi099 ceph-mon[39420]: 9.c scrub starts 2024-02-09T18:01:33.623 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:33 smithi099 ceph-mon[39420]: 9.8 scrub starts 2024-02-09T18:01:33.664 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:33.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:33 smithi179 ceph-mon[43975]: 1.18 scrub ok 2024-02-09T18:01:33.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:33 smithi179 ceph-mon[43975]: 9.c scrub starts 2024-02-09T18:01:33.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:33 smithi179 ceph-mon[43975]: 9.8 scrub starts 2024-02-09T18:01:33.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:33 smithi162 ceph-mon[44304]: 1.18 scrub ok 2024-02-09T18:01:33.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:33 smithi162 ceph-mon[44304]: 9.c scrub starts 2024-02-09T18:01:33.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:33 smithi162 ceph-mon[44304]: 9.8 scrub starts 2024-02-09T18:01:34.608 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 75 16 121912 121896 101.536 46.625 0.0113123 0.00984499 2024-02-09T18:01:34.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:34 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:01:34.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:34 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:01:34.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:34 smithi099 ceph-mon[39420]: 1.f scrub starts 2024-02-09T18:01:34.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:34 smithi099 ceph-mon[39420]: pgmap v2257: 47 pgs: 47 active+clean; 7.2 GiB data, 30 GiB used, 953 GiB / 983 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-02-09T18:01:34.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:34 smithi099 ceph-mon[39420]: 1.f scrub ok 2024-02-09T18:01:34.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:34 smithi099 ceph-mon[39420]: osdmap e914: 12 total, 12 up, 11 in 2024-02-09T18:01:34.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:34 smithi099 ceph-mon[39420]: 9.2 scrub starts 2024-02-09T18:01:34.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:34 smithi099 ceph-mon[39420]: 1.16 scrub starts 2024-02-09T18:01:34.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:34 smithi099 ceph-mon[39420]: 1.16 scrub ok 2024-02-09T18:01:34.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:34 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:01:34.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:34 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:01:34.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:34 smithi179 ceph-mon[43975]: 1.f scrub starts 2024-02-09T18:01:34.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:34 smithi179 ceph-mon[43975]: pgmap v2257: 47 pgs: 47 active+clean; 7.2 GiB data, 30 GiB used, 953 GiB / 983 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-02-09T18:01:34.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:34 smithi179 ceph-mon[43975]: 1.f scrub ok 2024-02-09T18:01:34.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:34 smithi179 ceph-mon[43975]: osdmap e914: 12 total, 12 up, 11 in 2024-02-09T18:01:34.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:34 smithi179 ceph-mon[43975]: 9.2 scrub starts 2024-02-09T18:01:34.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:34 smithi179 ceph-mon[43975]: 1.16 scrub starts 2024-02-09T18:01:34.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:34 smithi179 ceph-mon[43975]: 1.16 scrub ok 2024-02-09T18:01:34.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:34 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:01:34.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:34 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:01:34.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:34 smithi162 ceph-mon[44304]: 1.f scrub starts 2024-02-09T18:01:34.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:34 smithi162 ceph-mon[44304]: pgmap v2257: 47 pgs: 47 active+clean; 7.2 GiB data, 30 GiB used, 953 GiB / 983 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-02-09T18:01:34.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:34 smithi162 ceph-mon[44304]: 1.f scrub ok 2024-02-09T18:01:34.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:34 smithi162 ceph-mon[44304]: osdmap e914: 12 total, 12 up, 11 in 2024-02-09T18:01:34.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:34 smithi162 ceph-mon[44304]: 9.2 scrub starts 2024-02-09T18:01:34.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:34 smithi162 ceph-mon[44304]: 1.16 scrub starts 2024-02-09T18:01:34.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:34 smithi162 ceph-mon[44304]: 1.16 scrub ok 2024-02-09T18:01:35.010 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-09T18:01:35.455 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:35 smithi099 ceph-mon[39420]: 1.b scrub starts 2024-02-09T18:01:35.455 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:35 smithi099 ceph-mon[39420]: 9.8 scrub ok 2024-02-09T18:01:35.455 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:35 smithi099 ceph-mon[39420]: 9.c scrub ok 2024-02-09T18:01:35.455 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:35 smithi099 ceph-mon[39420]: 1.b scrub ok 2024-02-09T18:01:35.455 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:35 smithi099 ceph-mon[39420]: 1.3 scrub starts 2024-02-09T18:01:35.608 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 76 16 122848 122832 100.97 58.5 0.0162077 0.00989977 2024-02-09T18:01:35.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:35 smithi179 ceph-mon[43975]: 1.b scrub starts 2024-02-09T18:01:35.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:35 smithi179 ceph-mon[43975]: 9.8 scrub ok 2024-02-09T18:01:35.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:35 smithi179 ceph-mon[43975]: 9.c scrub ok 2024-02-09T18:01:35.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:35 smithi179 ceph-mon[43975]: 1.b scrub ok 2024-02-09T18:01:35.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:35 smithi179 ceph-mon[43975]: 1.3 scrub starts 2024-02-09T18:01:35.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:35 smithi162 ceph-mon[44304]: 1.b scrub starts 2024-02-09T18:01:35.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:35 smithi162 ceph-mon[44304]: 9.8 scrub ok 2024-02-09T18:01:35.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:35 smithi162 ceph-mon[44304]: 9.c scrub ok 2024-02-09T18:01:35.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:35 smithi162 ceph-mon[44304]: 1.b scrub ok 2024-02-09T18:01:35.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:35 smithi162 ceph-mon[44304]: 1.3 scrub starts 2024-02-09T18:01:35.957 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:01:35.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:35.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:01:36.064 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:01:36.092 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:36.223 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:01:36.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:36.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:01:36.331 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:01:36.373 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:36.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:36 smithi099 ceph-mon[39420]: 1.6 scrub starts 2024-02-09T18:01:36.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:36 smithi099 ceph-mon[39420]: 1.6 scrub ok 2024-02-09T18:01:36.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:36 smithi099 ceph-mon[39420]: 1.3 scrub ok 2024-02-09T18:01:36.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:36 smithi099 ceph-mon[39420]: pgmap v2259: 47 pgs: 47 active+clean; 7.3 GiB data, 30 GiB used, 953 GiB / 983 GiB avail; 54 MiB/s wr, 862 op/s 2024-02-09T18:01:36.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:36 smithi099 ceph-mon[39420]: 1.14 scrub starts 2024-02-09T18:01:36.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:36 smithi099 ceph-mon[39420]: 9.2 scrub ok 2024-02-09T18:01:36.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:36 smithi099 ceph-mon[39420]: 1.14 scrub ok 2024-02-09T18:01:36.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:36 smithi099 ceph-mon[39420]: 1.19 scrub starts 2024-02-09T18:01:36.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:36 smithi099 ceph-mon[39420]: 1.19 scrub ok 2024-02-09T18:01:36.608 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 77 16 123833 123817 100.458 61.5625 0.0148427 0.00995015 2024-02-09T18:01:36.637 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:01:36.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:36.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:01:36.743 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T18:01:36.781 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:36 smithi162 ceph-mon[44304]: 1.6 scrub starts 2024-02-09T18:01:36.781 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:36 smithi162 ceph-mon[44304]: 1.6 scrub ok 2024-02-09T18:01:36.782 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:36 smithi162 ceph-mon[44304]: 1.3 scrub ok 2024-02-09T18:01:36.782 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:36 smithi162 ceph-mon[44304]: pgmap v2259: 47 pgs: 47 active+clean; 7.3 GiB data, 30 GiB used, 953 GiB / 983 GiB avail; 54 MiB/s wr, 862 op/s 2024-02-09T18:01:36.782 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:36 smithi162 ceph-mon[44304]: 1.14 scrub starts 2024-02-09T18:01:36.782 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:36 smithi162 ceph-mon[44304]: 9.2 scrub ok 2024-02-09T18:01:36.782 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:36 smithi162 ceph-mon[44304]: 1.14 scrub ok 2024-02-09T18:01:36.782 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:36 smithi162 ceph-mon[44304]: 1.19 scrub starts 2024-02-09T18:01:36.782 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:36 smithi162 ceph-mon[44304]: 1.19 scrub ok 2024-02-09T18:01:36.787 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:36.820 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T18:01:36.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:36.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T18:01:36.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:36 smithi179 ceph-mon[43975]: 1.6 scrub starts 2024-02-09T18:01:36.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:36 smithi179 ceph-mon[43975]: 1.6 scrub ok 2024-02-09T18:01:36.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:36 smithi179 ceph-mon[43975]: 1.3 scrub ok 2024-02-09T18:01:36.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:36 smithi179 ceph-mon[43975]: pgmap v2259: 47 pgs: 47 active+clean; 7.3 GiB data, 30 GiB used, 953 GiB / 983 GiB avail; 54 MiB/s wr, 862 op/s 2024-02-09T18:01:36.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:36 smithi179 ceph-mon[43975]: 1.14 scrub starts 2024-02-09T18:01:36.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:36 smithi179 ceph-mon[43975]: 9.2 scrub ok 2024-02-09T18:01:36.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:36 smithi179 ceph-mon[43975]: 1.14 scrub ok 2024-02-09T18:01:36.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:36 smithi179 ceph-mon[43975]: 1.19 scrub starts 2024-02-09T18:01:36.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:36 smithi179 ceph-mon[43975]: 1.19 scrub ok 2024-02-09T18:01:36.926 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T18:01:36.958 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:36.991 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T18:01:36.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:36.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T18:01:37.098 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:01:37.155 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:37.477 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:37 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:01:37.477 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:37 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:01:37.477 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:37 smithi099 ceph-mon[39420]: 1.1b scrub starts 2024-02-09T18:01:37.478 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:37 smithi099 ceph-mon[39420]: osdmap e915: 12 total, 12 up, 11 in 2024-02-09T18:01:37.478 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:37 smithi099 ceph-mon[39420]: 1.1b scrub ok 2024-02-09T18:01:37.478 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:37 smithi099 ceph-mon[39420]: 1.5 scrub starts 2024-02-09T18:01:37.478 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:37 smithi099 ceph-mon[39420]: 1.5 scrub ok 2024-02-09T18:01:37.478 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:37 smithi099 ceph-mon[39420]: 1.7 scrub starts 2024-02-09T18:01:37.608 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 78 16 124788 124772 99.9353 59.6875 0.00389145 0.00999049 2024-02-09T18:01:37.807 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:01:37.812 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:01:37.845 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:01:37.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:37.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:01:37.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:37 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:01:37.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:37 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:01:37.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:37 smithi179 ceph-mon[43975]: 1.1b scrub starts 2024-02-09T18:01:37.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:37 smithi179 ceph-mon[43975]: osdmap e915: 12 total, 12 up, 11 in 2024-02-09T18:01:37.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:37 smithi179 ceph-mon[43975]: 1.1b scrub ok 2024-02-09T18:01:37.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:37 smithi179 ceph-mon[43975]: 1.5 scrub starts 2024-02-09T18:01:37.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:37 smithi179 ceph-mon[43975]: 1.5 scrub ok 2024-02-09T18:01:37.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:37 smithi179 ceph-mon[43975]: 1.7 scrub starts 2024-02-09T18:01:37.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:37 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:01:37.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:37 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:01:37.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:37 smithi162 ceph-mon[44304]: 1.1b scrub starts 2024-02-09T18:01:37.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:37 smithi162 ceph-mon[44304]: osdmap e915: 12 total, 12 up, 11 in 2024-02-09T18:01:37.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:37 smithi162 ceph-mon[44304]: 1.1b scrub ok 2024-02-09T18:01:37.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:37 smithi162 ceph-mon[44304]: 1.5 scrub starts 2024-02-09T18:01:37.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:37 smithi162 ceph-mon[44304]: 1.5 scrub ok 2024-02-09T18:01:37.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:37 smithi162 ceph-mon[44304]: 1.7 scrub starts 2024-02-09T18:01:37.950 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T18:01:37.987 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:38.020 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T18:01:38.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:38.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T18:01:38.127 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:01:38.173 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:38.384 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:01:38.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:38.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:01:38.490 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:01:38.530 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:38.556 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:01:38.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:38.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:01:38.608 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 79 16 125761 125745 99.4402 60.8125 0.00863957 0.010052 2024-02-09T18:01:38.663 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:01:38.714 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:38 smithi099 ceph-mon[39420]: pgmap v2261: 47 pgs: 47 active+clean; 7.4 GiB data, 31 GiB used, 953 GiB / 983 GiB avail; 54 MiB/s wr, 861 op/s 2024-02-09T18:01:38.714 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:38 smithi099 ceph-mon[39420]: 1.7 scrub ok 2024-02-09T18:01:38.720 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:38.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:38 smithi179 ceph-mon[43975]: pgmap v2261: 47 pgs: 47 active+clean; 7.4 GiB data, 31 GiB used, 953 GiB / 983 GiB avail; 54 MiB/s wr, 861 op/s 2024-02-09T18:01:38.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:38 smithi179 ceph-mon[43975]: 1.7 scrub ok 2024-02-09T18:01:38.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:38 smithi162 ceph-mon[44304]: pgmap v2261: 47 pgs: 47 active+clean; 7.4 GiB data, 31 GiB used, 953 GiB / 983 GiB avail; 54 MiB/s wr, 861 op/s 2024-02-09T18:01:38.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:38 smithi162 ceph-mon[44304]: 1.7 scrub ok 2024-02-09T18:01:39.608 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:2024-02-09T18:01:39.607780+0000 min lat: 0.00108495 max lat: 0.245758 avg lat: 0.0101011 2024-02-09T18:01:39.608 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-09T18:01:39.608 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 80 16 126729 126713 98.9535 60.5 0.0189995 0.0101011 2024-02-09T18:01:39.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:39 smithi099 ceph-mon[39420]: 1.1e scrub starts 2024-02-09T18:01:39.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:39 smithi099 ceph-mon[39420]: 1.1e scrub ok 2024-02-09T18:01:39.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:39 smithi099 ceph-mon[39420]: 9.6 scrub starts 2024-02-09T18:01:39.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:39 smithi179 ceph-mon[43975]: 1.1e scrub starts 2024-02-09T18:01:39.926 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:39 smithi179 ceph-mon[43975]: 1.1e scrub ok 2024-02-09T18:01:39.926 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:39 smithi179 ceph-mon[43975]: 9.6 scrub starts 2024-02-09T18:01:39.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:39 smithi162 ceph-mon[44304]: 1.1e scrub starts 2024-02-09T18:01:39.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:39 smithi162 ceph-mon[44304]: 1.1e scrub ok 2024-02-09T18:01:39.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:39 smithi162 ceph-mon[44304]: 9.6 scrub starts 2024-02-09T18:01:40.498 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.0 to 1 (10000) 2024-02-09T18:01:40.608 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 81 16 127644 127628 98.438 57.1875 0.00763473 0.0101544 2024-02-09T18:01:40.801 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:40 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:01:40.801 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:40 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:01:40.801 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:40 smithi099 ceph-mon[39420]: pgmap v2262: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 7.6 GiB data, 31 GiB used, 952 GiB / 983 GiB avail; 59 MiB/s wr, 945 op/s 2024-02-09T18:01:40.801 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:40 smithi099 ceph-mon[39420]: osdmap e916: 12 total, 12 up, 11 in 2024-02-09T18:01:40.801 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:40 smithi099 ceph-mon[39420]: 1.c scrub starts 2024-02-09T18:01:40.801 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:40 smithi099 ceph-mon[39420]: 1.c scrub ok 2024-02-09T18:01:40.802 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:40 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/669731603' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-09T18:01:40.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:40 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:01:40.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:40 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:01:40.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:40 smithi179 ceph-mon[43975]: pgmap v2262: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 7.6 GiB data, 31 GiB used, 952 GiB / 983 GiB avail; 59 MiB/s wr, 945 op/s 2024-02-09T18:01:40.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:40 smithi179 ceph-mon[43975]: osdmap e916: 12 total, 12 up, 11 in 2024-02-09T18:01:40.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:40 smithi179 ceph-mon[43975]: 1.c scrub starts 2024-02-09T18:01:40.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:40 smithi179 ceph-mon[43975]: 1.c scrub ok 2024-02-09T18:01:40.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:40 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/669731603' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-09T18:01:40.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:40 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:01:40.996 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:40 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:01:40.996 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:40 smithi162 ceph-mon[44304]: pgmap v2262: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 7.6 GiB data, 31 GiB used, 952 GiB / 983 GiB avail; 59 MiB/s wr, 945 op/s 2024-02-09T18:01:40.996 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:40 smithi162 ceph-mon[44304]: osdmap e916: 12 total, 12 up, 11 in 2024-02-09T18:01:40.996 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:40 smithi162 ceph-mon[44304]: 1.c scrub starts 2024-02-09T18:01:40.996 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:40 smithi162 ceph-mon[44304]: 1.c scrub ok 2024-02-09T18:01:40.997 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:40 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/669731603' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-09T18:01:41.609 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 82 16 128514 128498 97.8994 54.375 0.0282336 0.0102102 2024-02-09T18:01:41.867 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:41 smithi099 ceph-mon[39420]: 1.4 scrub starts 2024-02-09T18:01:41.867 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:41 smithi099 ceph-mon[39420]: 1.4 scrub ok 2024-02-09T18:01:41.867 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:41 smithi099 ceph-mon[39420]: 1.1c scrub starts 2024-02-09T18:01:41.867 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:41 smithi099 ceph-mon[39420]: 1.1c scrub ok 2024-02-09T18:01:41.867 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:41 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/669731603' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-09T18:01:41.867 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:41 smithi099 ceph-mon[39420]: osdmap e917: 12 total, 12 up, 11 in 2024-02-09T18:01:41.867 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:41 smithi099 ceph-mon[39420]: 9.6 scrub ok 2024-02-09T18:01:41.867 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:41 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1406986156' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:01:41.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:41 smithi179 ceph-mon[43975]: 1.4 scrub starts 2024-02-09T18:01:41.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:41 smithi179 ceph-mon[43975]: 1.4 scrub ok 2024-02-09T18:01:41.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:41 smithi179 ceph-mon[43975]: 1.1c scrub starts 2024-02-09T18:01:41.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:41 smithi179 ceph-mon[43975]: 1.1c scrub ok 2024-02-09T18:01:41.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:41 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/669731603' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-09T18:01:41.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:41 smithi179 ceph-mon[43975]: osdmap e917: 12 total, 12 up, 11 in 2024-02-09T18:01:41.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:41 smithi179 ceph-mon[43975]: 9.6 scrub ok 2024-02-09T18:01:41.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:41 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1406986156' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:01:41.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:41 smithi162 ceph-mon[44304]: 1.4 scrub starts 2024-02-09T18:01:41.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:41 smithi162 ceph-mon[44304]: 1.4 scrub ok 2024-02-09T18:01:41.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:41 smithi162 ceph-mon[44304]: 1.1c scrub starts 2024-02-09T18:01:41.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:41 smithi162 ceph-mon[44304]: 1.1c scrub ok 2024-02-09T18:01:41.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:41 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/669731603' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-09T18:01:41.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:41 smithi162 ceph-mon[44304]: osdmap e917: 12 total, 12 up, 11 in 2024-02-09T18:01:41.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:41 smithi162 ceph-mon[44304]: 9.6 scrub ok 2024-02-09T18:01:41.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:41 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1406986156' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:01:42.230 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T18:01:42.230 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T18:01:42.230 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T18:01:42.230 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T18:01:42.231 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T18:01:42.231 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T18:01:42.231 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T18:01:42.231 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T18:01:42.231 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T18:01:42.231 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T18:01:42.231 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T18:01:42.231 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T18:01:42.231 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T18:01:42.231 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T18:01:42.231 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T18:01:42.231 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T18:01:42.231 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T18:01:42.231 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T18:01:42.232 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T18:01:42.232 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T18:01:42.232 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T18:01:42.232 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T18:01:42.232 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T18:01:42.232 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T18:01:42.610 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 83 16 129336 129320 97.339 51.375 0.0278158 0.0102681 2024-02-09T18:01:42.882 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:42 smithi099 ceph-mon[39420]: 9.5 scrub starts 2024-02-09T18:01:42.882 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:42 smithi099 ceph-mon[39420]: pgmap v2265: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 7.7 GiB data, 32 GiB used, 952 GiB / 983 GiB avail; 70 MiB/s wr, 1.13k op/s 2024-02-09T18:01:42.883 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:42 smithi099 ceph-mon[39420]: 1.11 scrub starts 2024-02-09T18:01:42.883 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:42 smithi099 ceph-mon[39420]: 1.11 scrub ok 2024-02-09T18:01:42.883 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:42 smithi099 ceph-mon[39420]: 9.9 scrub starts 2024-02-09T18:01:42.883 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:42 smithi099 ceph-mon[39420]: 1.1d scrub starts 2024-02-09T18:01:42.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:42 smithi179 ceph-mon[43975]: 9.5 scrub starts 2024-02-09T18:01:42.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:42 smithi179 ceph-mon[43975]: pgmap v2265: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 7.7 GiB data, 32 GiB used, 952 GiB / 983 GiB avail; 70 MiB/s wr, 1.13k op/s 2024-02-09T18:01:42.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:42 smithi179 ceph-mon[43975]: 1.11 scrub starts 2024-02-09T18:01:42.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:42 smithi179 ceph-mon[43975]: 1.11 scrub ok 2024-02-09T18:01:42.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:42 smithi179 ceph-mon[43975]: 9.9 scrub starts 2024-02-09T18:01:42.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:42 smithi179 ceph-mon[43975]: 1.1d scrub starts 2024-02-09T18:01:42.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:42 smithi162 ceph-mon[44304]: 9.5 scrub starts 2024-02-09T18:01:42.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:42 smithi162 ceph-mon[44304]: pgmap v2265: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 7.7 GiB data, 32 GiB used, 952 GiB / 983 GiB avail; 70 MiB/s wr, 1.13k op/s 2024-02-09T18:01:42.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:42 smithi162 ceph-mon[44304]: 1.11 scrub starts 2024-02-09T18:01:42.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:42 smithi162 ceph-mon[44304]: 1.11 scrub ok 2024-02-09T18:01:42.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:42 smithi162 ceph-mon[44304]: 9.9 scrub starts 2024-02-09T18:01:42.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:42 smithi162 ceph-mon[44304]: 1.1d scrub starts 2024-02-09T18:01:43.321 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:01:43.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:43.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:01:43.433 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T18:01:43.477 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:43.522 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T18:01:43.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:43.528 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T18:01:43.610 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 84 16 130051 130035 96.7114 44.6875 0.00911858 0.0103353 2024-02-09T18:01:43.629 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:01:43.685 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:43.724 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:01:43.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:43.734 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:01:43.739 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:43 smithi179 ceph-mon[43975]: 1.1d scrub ok 2024-02-09T18:01:43.739 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:43 smithi179 ceph-mon[43975]: 9.b scrub starts 2024-02-09T18:01:43.739 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:43 smithi179 ceph-mon[43975]: 9.1 scrub starts 2024-02-09T18:01:43.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:43 smithi099 ceph-mon[39420]: 1.1d scrub ok 2024-02-09T18:01:43.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:43 smithi099 ceph-mon[39420]: 9.b scrub starts 2024-02-09T18:01:43.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:43 smithi099 ceph-mon[39420]: 9.1 scrub starts 2024-02-09T18:01:43.835 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:01:43.864 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:43 smithi162 ceph-mon[44304]: 1.1d scrub ok 2024-02-09T18:01:43.865 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:43 smithi162 ceph-mon[44304]: 9.b scrub starts 2024-02-09T18:01:43.865 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:43 smithi162 ceph-mon[44304]: 9.1 scrub starts 2024-02-09T18:01:43.867 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:43.896 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:01:43.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:43.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:01:44.003 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:01:44.044 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:44.152 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:01:44.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:44.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:01:44.258 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:01:44.356 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:44.403 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:01:44.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:44.409 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:01:44.510 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:01:44.561 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:44.615 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 85 16 130753 130737 96.0888 43.875 0.0307258 0.0104027 2024-02-09T18:01:44.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:44 smithi179 ceph-mon[43975]: 9.5 scrub ok 2024-02-09T18:01:44.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:44 smithi179 ceph-mon[43975]: 1.e scrub starts 2024-02-09T18:01:44.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:44 smithi179 ceph-mon[43975]: 1.e scrub ok 2024-02-09T18:01:44.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:44 smithi179 ceph-mon[43975]: pgmap v2266: 47 pgs: 3 active+clean+scrubbing, 44 active+clean; 7.8 GiB data, 32 GiB used, 952 GiB / 983 GiB avail; 56 MiB/s wr, 893 op/s 2024-02-09T18:01:44.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:44 smithi179 ceph-mon[43975]: 9.3 scrub starts 2024-02-09T18:01:44.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:44 smithi179 ceph-mon[43975]: 9.9 scrub ok 2024-02-09T18:01:44.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:44 smithi179 ceph-mon[43975]: 9.0 scrub starts 2024-02-09T18:01:44.800 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:44 smithi099 ceph-mon[39420]: 9.5 scrub ok 2024-02-09T18:01:44.800 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:44 smithi099 ceph-mon[39420]: 1.e scrub starts 2024-02-09T18:01:44.800 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:44 smithi099 ceph-mon[39420]: 1.e scrub ok 2024-02-09T18:01:44.800 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:44 smithi099 ceph-mon[39420]: pgmap v2266: 47 pgs: 3 active+clean+scrubbing, 44 active+clean; 7.8 GiB data, 32 GiB used, 952 GiB / 983 GiB avail; 56 MiB/s wr, 893 op/s 2024-02-09T18:01:44.800 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:44 smithi099 ceph-mon[39420]: 9.3 scrub starts 2024-02-09T18:01:44.800 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:44 smithi099 ceph-mon[39420]: 9.9 scrub ok 2024-02-09T18:01:44.800 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:44 smithi099 ceph-mon[39420]: 9.0 scrub starts 2024-02-09T18:01:44.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:44 smithi162 ceph-mon[44304]: 9.5 scrub ok 2024-02-09T18:01:44.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:44 smithi162 ceph-mon[44304]: 1.e scrub starts 2024-02-09T18:01:44.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:44 smithi162 ceph-mon[44304]: 1.e scrub ok 2024-02-09T18:01:44.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:44 smithi162 ceph-mon[44304]: pgmap v2266: 47 pgs: 3 active+clean+scrubbing, 44 active+clean; 7.8 GiB data, 32 GiB used, 952 GiB / 983 GiB avail; 56 MiB/s wr, 893 op/s 2024-02-09T18:01:44.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:44 smithi162 ceph-mon[44304]: 9.3 scrub starts 2024-02-09T18:01:44.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:44 smithi162 ceph-mon[44304]: 9.9 scrub ok 2024-02-09T18:01:44.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:44 smithi162 ceph-mon[44304]: 9.0 scrub starts 2024-02-09T18:01:44.979 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 2 1 2024-02-09T18:01:45.025 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:01:45.031 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:45.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:01:45.132 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:01:45.138 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T18:01:45.139 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [ 2024-02-09T18:01:45.139 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.139 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26563.0:130881 9.0 9:07dd3dbc:::benchmark_data_smithi179_161915_object130880:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e917)", 2024-02-09T18:01:45.139 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:01:44.767092+0000", 2024-02-09T18:01:45.139 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.0077702939999999996, 2024-02-09T18:01:45.139 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0078062909999999999, 2024-02-09T18:01:45.139 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:01:45.139 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "waiting for sub ops", 2024-02-09T18:01:45.139 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:01:45.139 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26563", 2024-02-09T18:01:45.139 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:01:45.139 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 130881 2024-02-09T18:01:45.139 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.140 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:01:45.140 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.140 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:01:45.140 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.767092+0000", 2024-02-09T18:01:45.140 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:01:45.140 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.140 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.140 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:01:45.140 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.767092+0000", 2024-02-09T18:01:45.140 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:01:45.140 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.140 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.140 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:01:45.141 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.767094+0000", 2024-02-09T18:01:45.141 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.209e-06 2024-02-09T18:01:45.141 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.141 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.141 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:01:45.141 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.767151+0000", 2024-02-09T18:01:45.141 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7297999999999999e-05 2024-02-09T18:01:45.141 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.141 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.141 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:01:45.141 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.767159+0000", 2024-02-09T18:01:45.141 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.3899999999999993e-06 2024-02-09T18:01:45.141 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.141 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.142 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:01:45.142 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.767166+0000", 2024-02-09T18:01:45.142 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.0520000000000004e-06 2024-02-09T18:01:45.142 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.142 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.142 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:01:45.142 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.772501+0000", 2024-02-09T18:01:45.142 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0053344899999999999 2024-02-09T18:01:45.142 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.142 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.142 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:01:45.142 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.772631+0000", 2024-02-09T18:01:45.142 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012970200000000001 2024-02-09T18:01:45.143 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.143 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.143 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 6,11", 2024-02-09T18:01:45.143 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.772782+0000", 2024-02-09T18:01:45.143 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000151154 2024-02-09T18:01:45.143 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:01:45.143 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:01:45.143 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:01:45.143 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.143 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.143 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26563.0:130883 9.0 9.c47d5b10 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e917)", 2024-02-09T18:01:45.143 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:01:44.767192+0000", 2024-02-09T18:01:45.143 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.0076709769999999998, 2024-02-09T18:01:45.143 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0077863100000000003, 2024-02-09T18:01:45.144 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:01:45.144 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "queued for pg", 2024-02-09T18:01:45.144 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:01:45.144 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26563", 2024-02-09T18:01:45.144 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:01:45.144 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 130883 2024-02-09T18:01:45.144 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.144 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:01:45.144 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.144 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:01:45.144 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.767192+0000", 2024-02-09T18:01:45.144 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:01:45.144 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.144 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.145 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:01:45.145 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.767192+0000", 2024-02-09T18:01:45.145 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:01:45.145 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.145 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.145 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:01:45.145 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.767192+0000", 2024-02-09T18:01:45.145 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7999999999999995e-07 2024-02-09T18:01:45.145 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.145 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.145 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:01:45.145 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.767225+0000", 2024-02-09T18:01:45.145 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.2759000000000003e-05 2024-02-09T18:01:45.145 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.146 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.146 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:01:45.146 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.767234+0000", 2024-02-09T18:01:45.146 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.6209999999999993e-06 2024-02-09T18:01:45.146 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.146 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.146 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:01:45.146 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.767240+0000", 2024-02-09T18:01:45.146 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.9789999999999998e-06 2024-02-09T18:01:45.146 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:01:45.146 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:01:45.146 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:01:45.146 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.146 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.146 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26563.0:130885 9.f 9:f3fe4878:::benchmark_data_smithi179_161915_object130884:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e917)", 2024-02-09T18:01:45.147 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:01:44.769430+0000", 2024-02-09T18:01:45.147 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.0054324229999999996, 2024-02-09T18:01:45.147 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0056077059999999996, 2024-02-09T18:01:45.147 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:01:45.147 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "waiting for sub ops", 2024-02-09T18:01:45.147 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:01:45.147 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26563", 2024-02-09T18:01:45.147 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:01:45.147 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 130885 2024-02-09T18:01:45.147 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.147 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:01:45.147 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.147 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:01:45.147 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.769430+0000", 2024-02-09T18:01:45.147 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:01:45.148 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.148 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.148 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:01:45.148 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.769430+0000", 2024-02-09T18:01:45.148 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:01:45.148 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.148 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.148 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:01:45.148 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.769431+0000", 2024-02-09T18:01:45.148 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.004e-06 2024-02-09T18:01:45.148 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.148 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.148 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:01:45.148 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.769484+0000", 2024-02-09T18:01:45.148 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2818999999999997e-05 2024-02-09T18:01:45.149 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.149 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.149 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:01:45.149 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.769497+0000", 2024-02-09T18:01:45.149 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2462e-05 2024-02-09T18:01:45.149 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.149 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.149 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:01:45.149 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.769505+0000", 2024-02-09T18:01:45.149 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.8400000000000001e-06 2024-02-09T18:01:45.149 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.149 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.149 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:01:45.149 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.769558+0000", 2024-02-09T18:01:45.149 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2209000000000001e-05 2024-02-09T18:01:45.150 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.150 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.150 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:01:45.150 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.769744+0000", 2024-02-09T18:01:45.150 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00018681599999999999 2024-02-09T18:01:45.150 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.150 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.150 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 5,9", 2024-02-09T18:01:45.150 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.769878+0000", 2024-02-09T18:01:45.150 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013338900000000001 2024-02-09T18:01:45.150 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.150 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.150 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:01:45.150 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.774894+0000", 2024-02-09T18:01:45.150 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0050161340000000002 2024-02-09T18:01:45.151 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:01:45.151 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:01:45.151 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:01:45.151 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.151 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.151 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26563.0:130888 9.a 9:50aaa043:::benchmark_data_smithi179_161915_object130887:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e917)", 2024-02-09T18:01:45.151 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:01:44.769525+0000", 2024-02-09T18:01:45.151 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.0053380700000000003, 2024-02-09T18:01:45.151 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0055969909999999999, 2024-02-09T18:01:45.151 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:01:45.151 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "waiting for sub ops", 2024-02-09T18:01:45.151 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:01:45.151 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26563", 2024-02-09T18:01:45.151 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:01:45.151 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 130888 2024-02-09T18:01:45.151 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.151 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:01:45.152 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.152 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:01:45.152 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.769525+0000", 2024-02-09T18:01:45.152 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:01:45.152 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.152 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.152 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:01:45.152 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.769525+0000", 2024-02-09T18:01:45.152 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:01:45.152 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.152 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.152 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:01:45.152 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.769525+0000", 2024-02-09T18:01:45.152 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3300000000000002e-07 2024-02-09T18:01:45.152 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.153 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.153 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:01:45.153 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.769554+0000", 2024-02-09T18:01:45.153 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.8379000000000001e-05 2024-02-09T18:01:45.153 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.153 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.153 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:01:45.153 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.769561+0000", 2024-02-09T18:01:45.153 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.2679999999999997e-06 2024-02-09T18:01:45.153 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.153 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.153 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:01:45.153 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.769567+0000", 2024-02-09T18:01:45.153 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6640000000000003e-06 2024-02-09T18:01:45.153 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.153 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.153 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:01:45.154 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.769618+0000", 2024-02-09T18:01:45.154 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1298000000000003e-05 2024-02-09T18:01:45.154 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.154 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.154 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:01:45.154 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.769752+0000", 2024-02-09T18:01:45.154 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000133776 2024-02-09T18:01:45.154 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.154 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.154 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 5,7", 2024-02-09T18:01:45.154 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.769889+0000", 2024-02-09T18:01:45.154 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013756799999999999 2024-02-09T18:01:45.154 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.154 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.154 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:01:45.154 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.774876+0000", 2024-02-09T18:01:45.155 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0049870640000000003 2024-02-09T18:01:45.155 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:01:45.155 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:01:45.155 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:01:45.155 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.155 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.155 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.26563.0:130886 9.6 e917/882 9:69203a4b:::benchmark_data_smithi179_161915_object130885:head v 917'8082, mlcod=917'8080)", 2024-02-09T18:01:45.155 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:01:44.772237+0000", 2024-02-09T18:01:45.155 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 0.0026258539999999999, 2024-02-09T18:01:45.155 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0029672930000000002, 2024-02-09T18:01:45.155 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:01:45.155 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T18:01:45.155 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:01:45.155 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.155 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:01:45.155 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.772237+0000", 2024-02-09T18:01:45.156 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:01:45.156 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.156 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.156 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:01:45.156 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.772237+0000", 2024-02-09T18:01:45.156 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:01:45.156 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.156 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.156 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:01:45.156 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.772238+0000", 2024-02-09T18:01:45.156 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.2099999999999995e-07 2024-02-09T18:01:45.156 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.156 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.156 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:01:45.157 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.772315+0000", 2024-02-09T18:01:45.157 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.7657000000000005e-05 2024-02-09T18:01:45.157 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.157 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.157 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:01:45.157 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.772326+0000", 2024-02-09T18:01:45.157 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0577999999999999e-05 2024-02-09T18:01:45.157 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.157 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.157 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:01:45.157 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.772333+0000", 2024-02-09T18:01:45.157 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.9609999999999999e-06 2024-02-09T18:01:45.157 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.157 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.157 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:01:45.157 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.772398+0000", 2024-02-09T18:01:45.158 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.5500000000000006e-05 2024-02-09T18:01:45.158 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.158 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.158 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:01:45.158 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.772422+0000", 2024-02-09T18:01:45.158 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.3459e-05 2024-02-09T18:01:45.158 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:01:45.158 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:01:45.158 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:01:45.158 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.158 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.158 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop_reply(client.26563.0:130888 9.a e917/882)", 2024-02-09T18:01:45.158 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:01:44.775174+0000", 2024-02-09T18:01:45.158 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 4294967295.9996886, 2024-02-09T18:01:45.158 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.8907999999999996e-05, 2024-02-09T18:01:45.159 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:01:45.159 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T18:01:45.159 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:01:45.159 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.159 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:01:45.159 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.775174+0000", 2024-02-09T18:01:45.159 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:01:45.159 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.159 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.159 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:01:45.159 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.775174+0000", 2024-02-09T18:01:45.159 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:01:45.159 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.159 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.159 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:01:45.159 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.775175+0000", 2024-02-09T18:01:45.160 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.6700000000000003e-07 2024-02-09T18:01:45.160 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.160 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.160 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:01:45.160 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.775179+0000", 2024-02-09T18:01:45.160 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.4530000000000004e-06 2024-02-09T18:01:45.160 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.160 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.160 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:01:45.160 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.775190+0000", 2024-02-09T18:01:45.160 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0298e-05 2024-02-09T18:01:45.160 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.160 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.160 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:01:45.160 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.775195+0000", 2024-02-09T18:01:45.161 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6010000000000002e-06 2024-02-09T18:01:45.161 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.161 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.161 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:01:45.161 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.775242+0000", 2024-02-09T18:01:45.161 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.6736000000000001e-05 2024-02-09T18:01:45.161 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:01:45.161 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:01:45.161 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:01:45.161 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:44.775258+0000", 2024-02-09T18:01:45.161 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.6110999999999999e-05 2024-02-09T18:01:45.161 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:01:45.161 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:01:45.161 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:01:45.161 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:01:45.162 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T18:01:45.162 INFO:teuthology.orchestra.run.smithi099.stdout: "num_ops": 6 2024-02-09T18:01:45.162 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T18:01:45.211 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:45.360 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:01:45.361 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T18:01:45.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:45.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:01:45.467 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:01:45.526 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:45.558 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:01:45.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:45.562 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:01:45.611 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 86 16 131533 131517 95.5385 48.75 0.0056109 0.0104626 2024-02-09T18:01:45.662 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T18:01:45.703 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:45.742 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T18:01:45.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:45.746 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T18:01:45.835 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:45 smithi099 ceph-mon[39420]: 1.9 scrub starts 2024-02-09T18:01:45.835 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:45 smithi099 ceph-mon[39420]: 1.9 scrub ok 2024-02-09T18:01:45.835 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:45 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:01:45.835 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:45 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:01:45.835 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:45 smithi099 ceph-mon[39420]: 9.1 scrub ok 2024-02-09T18:01:45.835 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:45 smithi099 ceph-mon[39420]: 9.d scrub starts 2024-02-09T18:01:45.846 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:01:45.893 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:45.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:45 smithi179 ceph-mon[43975]: 1.9 scrub starts 2024-02-09T18:01:45.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:45 smithi179 ceph-mon[43975]: 1.9 scrub ok 2024-02-09T18:01:45.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:45 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:01:45.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:45 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:01:45.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:45 smithi179 ceph-mon[43975]: 9.1 scrub ok 2024-02-09T18:01:45.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:45 smithi179 ceph-mon[43975]: 9.d scrub starts 2024-02-09T18:01:45.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:45 smithi162 ceph-mon[44304]: 1.9 scrub starts 2024-02-09T18:01:45.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:45 smithi162 ceph-mon[44304]: 1.9 scrub ok 2024-02-09T18:01:45.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:45 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:01:45.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:45 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:01:45.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:45 smithi162 ceph-mon[44304]: 9.1 scrub ok 2024-02-09T18:01:45.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:45 smithi162 ceph-mon[44304]: 9.d scrub starts 2024-02-09T18:01:46.030 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:01:46.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:46.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:01:46.139 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T18:01:46.196 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:46.242 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T18:01:46.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:46.246 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T18:01:46.347 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T18:01:46.385 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:46.432 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T18:01:46.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:46.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T18:01:46.539 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:01:46.573 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:46.612 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 87 15 132311 132296 95.0001 48.6875 0.0245942 0.0105212 2024-02-09T18:01:46.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:46 smithi162 ceph-mon[44304]: 9.b scrub ok 2024-02-09T18:01:46.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:46 smithi162 ceph-mon[44304]: pgmap v2267: 47 pgs: 3 active+clean+scrubbing, 44 active+clean; 7.9 GiB data, 32 GiB used, 951 GiB / 983 GiB avail; 54 MiB/s wr, 867 op/s 2024-02-09T18:01:46.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:46 smithi162 ceph-mon[44304]: 1.17 scrub starts 2024-02-09T18:01:46.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:46 smithi162 ceph-mon[44304]: 9.e scrub starts 2024-02-09T18:01:46.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:46 smithi162 ceph-mon[44304]: 1.17 scrub ok 2024-02-09T18:01:46.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:46 smithi162 ceph-mon[44304]: osdmap e918: 12 total, 12 up, 11 in 2024-02-09T18:01:46.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:46 smithi162 ceph-mon[44304]: 9.3 scrub ok 2024-02-09T18:01:46.682 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:46 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2064703975' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T18:01:46.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:46 smithi179 ceph-mon[43975]: 9.b scrub ok 2024-02-09T18:01:46.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:46 smithi179 ceph-mon[43975]: pgmap v2267: 47 pgs: 3 active+clean+scrubbing, 44 active+clean; 7.9 GiB data, 32 GiB used, 951 GiB / 983 GiB avail; 54 MiB/s wr, 867 op/s 2024-02-09T18:01:46.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:46 smithi179 ceph-mon[43975]: 1.17 scrub starts 2024-02-09T18:01:46.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:46 smithi179 ceph-mon[43975]: 9.e scrub starts 2024-02-09T18:01:46.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:46 smithi179 ceph-mon[43975]: 1.17 scrub ok 2024-02-09T18:01:46.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:46 smithi179 ceph-mon[43975]: osdmap e918: 12 total, 12 up, 11 in 2024-02-09T18:01:46.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:46 smithi179 ceph-mon[43975]: 9.3 scrub ok 2024-02-09T18:01:46.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:46 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2064703975' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T18:01:46.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:46 smithi099 ceph-mon[39420]: 9.b scrub ok 2024-02-09T18:01:46.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:46 smithi099 ceph-mon[39420]: pgmap v2267: 47 pgs: 3 active+clean+scrubbing, 44 active+clean; 7.9 GiB data, 32 GiB used, 951 GiB / 983 GiB avail; 54 MiB/s wr, 867 op/s 2024-02-09T18:01:46.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:46 smithi099 ceph-mon[39420]: 1.17 scrub starts 2024-02-09T18:01:46.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:46 smithi099 ceph-mon[39420]: 9.e scrub starts 2024-02-09T18:01:46.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:46 smithi099 ceph-mon[39420]: 1.17 scrub ok 2024-02-09T18:01:46.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:46 smithi099 ceph-mon[39420]: osdmap e918: 12 total, 12 up, 11 in 2024-02-09T18:01:46.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:46 smithi099 ceph-mon[39420]: 9.3 scrub ok 2024-02-09T18:01:46.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:46 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2064703975' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T18:01:47.612 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 88 16 133059 133043 94.4512 46.6875 0.0155322 0.010584 2024-02-09T18:01:47.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:47 smithi179 ceph-mon[43975]: 9.0 scrub ok 2024-02-09T18:01:47.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:47 smithi179 ceph-mon[43975]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T18:01:47.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:47 smithi179 ceph-mon[43975]: Cluster is now healthy 2024-02-09T18:01:47.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:47 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2064703975' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T18:01:47.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:47 smithi179 ceph-mon[43975]: osdmap e919: 12 total, 12 up, 11 in 2024-02-09T18:01:47.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:47 smithi179 ceph-mon[43975]: 1.a scrub starts 2024-02-09T18:01:47.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:47 smithi179 ceph-mon[43975]: 1.a scrub ok 2024-02-09T18:01:47.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:47 smithi179 ceph-mon[43975]: 9.d scrub ok 2024-02-09T18:01:47.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:47 smithi162 ceph-mon[44304]: 9.0 scrub ok 2024-02-09T18:01:47.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:47 smithi162 ceph-mon[44304]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T18:01:47.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:47 smithi162 ceph-mon[44304]: Cluster is now healthy 2024-02-09T18:01:47.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:47 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2064703975' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T18:01:47.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:47 smithi162 ceph-mon[44304]: osdmap e919: 12 total, 12 up, 11 in 2024-02-09T18:01:47.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:47 smithi162 ceph-mon[44304]: 1.a scrub starts 2024-02-09T18:01:47.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:47 smithi162 ceph-mon[44304]: 1.a scrub ok 2024-02-09T18:01:47.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:47 smithi162 ceph-mon[44304]: 9.d scrub ok 2024-02-09T18:01:47.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:47 smithi099 ceph-mon[39420]: 9.0 scrub ok 2024-02-09T18:01:47.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:47 smithi099 ceph-mon[39420]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T18:01:47.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:47 smithi099 ceph-mon[39420]: Cluster is now healthy 2024-02-09T18:01:47.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:47 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2064703975' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T18:01:47.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:47 smithi099 ceph-mon[39420]: osdmap e919: 12 total, 12 up, 11 in 2024-02-09T18:01:47.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:47 smithi099 ceph-mon[39420]: 1.a scrub starts 2024-02-09T18:01:47.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:47 smithi099 ceph-mon[39420]: 1.a scrub ok 2024-02-09T18:01:47.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:47 smithi099 ceph-mon[39420]: 9.d scrub ok 2024-02-09T18:01:48.528 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is unset 2024-02-09T18:01:48.612 INFO:tasks.radosbench.radosbench.2.smithi179.stdout: 89 16 134003 133987 94.053 59 0.0100576 0.0106288 2024-02-09T18:01:48.883 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:48 smithi099 ceph-mon[39420]: pgmap v2270: 47 pgs: 4 active+clean+scrubbing, 43 active+clean; 8.0 GiB data, 32 GiB used, 951 GiB / 983 GiB avail; 47 MiB/s wr, 751 op/s 2024-02-09T18:01:48.883 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:48 smithi099 ceph-mon[39420]: 9.7 scrub starts 2024-02-09T18:01:48.883 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:48 smithi099 ceph-mon[39420]: 9.e scrub ok 2024-02-09T18:01:48.883 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:48 smithi099 ceph-mon[39420]: 1.8 scrub starts 2024-02-09T18:01:48.883 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:48 smithi099 ceph-mon[39420]: 1.8 scrub ok 2024-02-09T18:01:48.915 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-09T18:01:48.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:48 smithi179 ceph-mon[43975]: pgmap v2270: 47 pgs: 4 active+clean+scrubbing, 43 active+clean; 8.0 GiB data, 32 GiB used, 951 GiB / 983 GiB avail; 47 MiB/s wr, 751 op/s 2024-02-09T18:01:48.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:48 smithi179 ceph-mon[43975]: 9.7 scrub starts 2024-02-09T18:01:48.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:48 smithi179 ceph-mon[43975]: 9.e scrub ok 2024-02-09T18:01:48.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:48 smithi179 ceph-mon[43975]: 1.8 scrub starts 2024-02-09T18:01:48.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:48 smithi179 ceph-mon[43975]: 1.8 scrub ok 2024-02-09T18:01:48.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:48 smithi162 ceph-mon[44304]: pgmap v2270: 47 pgs: 4 active+clean+scrubbing, 43 active+clean; 8.0 GiB data, 32 GiB used, 951 GiB / 983 GiB avail; 47 MiB/s wr, 751 op/s 2024-02-09T18:01:48.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:48 smithi162 ceph-mon[44304]: 9.7 scrub starts 2024-02-09T18:01:48.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:48 smithi162 ceph-mon[44304]: 9.e scrub ok 2024-02-09T18:01:48.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:48 smithi162 ceph-mon[44304]: 1.8 scrub starts 2024-02-09T18:01:48.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:48 smithi162 ceph-mon[44304]: 1.8 scrub ok 2024-02-09T18:01:49.439 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:01:49.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:49.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:01:49.488 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:01:49.496 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:01:49.545 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:01:49.599 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:49.612 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Total time run: 90.0125 2024-02-09T18:01:49.612 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Total writes made: 134879 2024-02-09T18:01:49.612 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Write size: 65536 2024-02-09T18:01:49.612 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Object size: 65536 2024-02-09T18:01:49.612 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Bandwidth (MB/sec): 93.6529 2024-02-09T18:01:49.612 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Stddev Bandwidth: 42.2434 2024-02-09T18:01:49.612 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Max bandwidth (MB/sec): 245.875 2024-02-09T18:01:49.613 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Min bandwidth (MB/sec): 43.875 2024-02-09T18:01:49.613 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Average IOPS: 1498 2024-02-09T18:01:49.613 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Stddev IOPS: 675.895 2024-02-09T18:01:49.613 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Max IOPS: 3934 2024-02-09T18:01:49.613 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Min IOPS: 702 2024-02-09T18:01:49.613 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Average Latency(s): 0.0106745 2024-02-09T18:01:49.613 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Stddev Latency(s): 0.0116244 2024-02-09T18:01:49.613 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Max latency(s): 0.245758 2024-02-09T18:01:49.613 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Min latency(s): 0.00108495 2024-02-09T18:01:49.618 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Cleaning up (deleting benchmark objects) 2024-02-09T18:01:49.724 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:01:49.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:49.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:01:49.729 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:49 smithi099 ceph-mon[39420]: 1.0 scrub starts 2024-02-09T18:01:49.729 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:49 smithi099 ceph-mon[39420]: 1.0 scrub ok 2024-02-09T18:01:49.729 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:49 smithi099 ceph-mon[39420]: 9.7 scrub ok 2024-02-09T18:01:49.830 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:01:49.874 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:49.921 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:01:49.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:49 smithi179 ceph-mon[43975]: 1.0 scrub starts 2024-02-09T18:01:49.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:49 smithi179 ceph-mon[43975]: 1.0 scrub ok 2024-02-09T18:01:49.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:49 smithi179 ceph-mon[43975]: 9.7 scrub ok 2024-02-09T18:01:49.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:49.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:01:49.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:49 smithi162 ceph-mon[44304]: 1.0 scrub starts 2024-02-09T18:01:49.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:49 smithi162 ceph-mon[44304]: 1.0 scrub ok 2024-02-09T18:01:49.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:49 smithi162 ceph-mon[44304]: 9.7 scrub ok 2024-02-09T18:01:50.026 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:01:50.063 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:50.148 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:01:50.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:50.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:01:50.255 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T18:01:50.309 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:50.346 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T18:01:50.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:50.353 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T18:01:50.454 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T18:01:50.498 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:50.537 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T18:01:50.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:50.542 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T18:01:50.592 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:50 smithi179 ceph-mon[43975]: pgmap v2271: 47 pgs: 3 active+clean+scrubbing, 44 active+clean; 8.1 GiB data, 33 GiB used, 951 GiB / 983 GiB avail; 49 MiB/s wr, 788 op/s 2024-02-09T18:01:50.643 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:01:50.674 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:01:50.691 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:50 smithi099 ceph-mon[39420]: pgmap v2271: 47 pgs: 3 active+clean+scrubbing, 44 active+clean; 8.1 GiB data, 33 GiB used, 951 GiB / 983 GiB avail; 49 MiB/s wr, 788 op/s 2024-02-09T18:01:50.695 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:50.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:50 smithi162 ceph-mon[44304]: pgmap v2271: 47 pgs: 3 active+clean+scrubbing, 44 active+clean; 8.1 GiB data, 33 GiB used, 951 GiB / 983 GiB avail; 49 MiB/s wr, 788 op/s 2024-02-09T18:01:51.252 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:01:51.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:51.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:01:51.363 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T18:01:51.401 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:51.426 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T18:01:51.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:51.430 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T18:01:51.458 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set noscrub 2024-02-09T18:01:51.531 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:01:51.573 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:51.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:51 smithi179 ceph-mon[43975]: 1.2 scrub starts 2024-02-09T18:01:51.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:51 smithi179 ceph-mon[43975]: 1.2 scrub ok 2024-02-09T18:01:51.698 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:01:51.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:51.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:01:51.803 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:01:51.832 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:51 smithi099 ceph-mon[39420]: 1.2 scrub starts 2024-02-09T18:01:51.832 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:51 smithi099 ceph-mon[39420]: 1.2 scrub ok 2024-02-09T18:01:51.847 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:51.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:51 smithi162 ceph-mon[44304]: 1.2 scrub starts 2024-02-09T18:01:51.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:51 smithi162 ceph-mon[44304]: 1.2 scrub ok 2024-02-09T18:01:52.202 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:01:52.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:52.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:01:52.213 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:01:52.308 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:01:52.337 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:52.367 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:01:52.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:52.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:01:52.471 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T18:01:52.512 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:52.539 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T18:01:52.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:52.547 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T18:01:52.648 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:01:52.683 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:52.722 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:01:52.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:52.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:01:52.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:52 smithi099 ceph-mon[39420]: pgmap v2272: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 8.2 GiB data, 33 GiB used, 951 GiB / 983 GiB avail; 127 B/s rd, 50 MiB/s wr, 860 op/s 2024-02-09T18:01:52.828 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:01:52.861 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:52 smithi162 ceph-mon[44304]: pgmap v2272: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 8.2 GiB data, 33 GiB used, 951 GiB / 983 GiB avail; 127 B/s rd, 50 MiB/s wr, 860 op/s 2024-02-09T18:01:52.863 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:52.895 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:01:52.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:52.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:01:52.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:52 smithi179 ceph-mon[43975]: pgmap v2272: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 8.2 GiB data, 33 GiB used, 951 GiB / 983 GiB avail; 127 B/s rd, 50 MiB/s wr, 860 op/s 2024-02-09T18:01:53.001 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:01:53.060 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:53.089 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:01:53.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:53.094 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:01:53.194 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T18:01:53.234 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:53.267 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T18:01:53.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:53.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T18:01:53.377 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:01:53.419 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:53.446 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:01:53.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:53.452 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:01:53.553 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T18:01:53.586 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:53.601 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:53 smithi162 ceph-mon[44304]: 1.15 scrub starts 2024-02-09T18:01:53.601 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:53 smithi162 ceph-mon[44304]: 9.f deep-scrub starts 2024-02-09T18:01:53.601 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:53 smithi162 ceph-mon[44304]: 1.15 scrub ok 2024-02-09T18:01:53.601 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:53 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2989971130' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-09T18:01:53.610 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.2 to 1 (10000) 2024-02-09T18:01:53.614 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T18:01:53.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:53.620 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T18:01:53.632 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:53 smithi099 ceph-mon[39420]: 1.15 scrub starts 2024-02-09T18:01:53.632 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:53 smithi099 ceph-mon[39420]: 9.f deep-scrub starts 2024-02-09T18:01:53.632 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:53 smithi099 ceph-mon[39420]: 1.15 scrub ok 2024-02-09T18:01:53.632 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:53 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2989971130' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-09T18:01:53.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:53 smithi179 ceph-mon[43975]: 1.15 scrub starts 2024-02-09T18:01:53.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:53 smithi179 ceph-mon[43975]: 9.f deep-scrub starts 2024-02-09T18:01:53.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:53 smithi179 ceph-mon[43975]: 1.15 scrub ok 2024-02-09T18:01:53.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:53 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2989971130' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-09T18:01:53.721 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:01:53.780 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:54.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:54 smithi179 ceph-mon[43975]: pgmap v2273: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 8.1 GiB data, 33 GiB used, 950 GiB / 983 GiB avail; 127 B/s rd, 46 MiB/s wr, 904 op/s 2024-02-09T18:01:54.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:54 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2989971130' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-09T18:01:54.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:54 smithi179 ceph-mon[43975]: osdmap e920: 12 total, 12 up, 11 in 2024-02-09T18:01:54.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:54 smithi179 ceph-mon[43975]: 9.a scrub starts 2024-02-09T18:01:54.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:54 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:01:54.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:54 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:01:54.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:54 smithi162 ceph-mon[44304]: pgmap v2273: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 8.1 GiB data, 33 GiB used, 950 GiB / 983 GiB avail; 127 B/s rd, 46 MiB/s wr, 904 op/s 2024-02-09T18:01:54.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:54 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2989971130' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-09T18:01:54.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:54 smithi162 ceph-mon[44304]: osdmap e920: 12 total, 12 up, 11 in 2024-02-09T18:01:54.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:54 smithi162 ceph-mon[44304]: 9.a scrub starts 2024-02-09T18:01:54.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:54 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:01:54.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:54 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:01:54.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:54 smithi099 ceph-mon[39420]: pgmap v2273: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 8.1 GiB data, 33 GiB used, 950 GiB / 983 GiB avail; 127 B/s rd, 46 MiB/s wr, 904 op/s 2024-02-09T18:01:54.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:54 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2989971130' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-09T18:01:54.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:54 smithi099 ceph-mon[39420]: osdmap e920: 12 total, 12 up, 11 in 2024-02-09T18:01:54.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:54 smithi099 ceph-mon[39420]: 9.a scrub starts 2024-02-09T18:01:54.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:54 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:01:54.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:54 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:01:55.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:55 smithi099 ceph-mon[39420]: osdmap e921: 12 total, 12 up, 11 in 2024-02-09T18:01:55.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:55 smithi162 ceph-mon[44304]: osdmap e921: 12 total, 12 up, 11 in 2024-02-09T18:01:55.931 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:55 smithi179 ceph-mon[43975]: osdmap e921: 12 total, 12 up, 11 in 2024-02-09T18:01:56.443 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:01:56.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:56.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:01:56.553 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:01:56.609 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:56.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:56 smithi099 ceph-mon[39420]: pgmap v2276: 47 pgs: 47 active+clean; 8.1 GiB data, 33 GiB used, 951 GiB / 983 GiB avail; 127 B/s rd, 32 MiB/s wr, 805 op/s 2024-02-09T18:01:56.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:56 smithi099 ceph-mon[39420]: 1.1 scrub starts 2024-02-09T18:01:56.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:56 smithi099 ceph-mon[39420]: 1.1 scrub ok 2024-02-09T18:01:56.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:56 smithi179 ceph-mon[43975]: pgmap v2276: 47 pgs: 47 active+clean; 8.1 GiB data, 33 GiB used, 951 GiB / 983 GiB avail; 127 B/s rd, 32 MiB/s wr, 805 op/s 2024-02-09T18:01:56.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:56 smithi179 ceph-mon[43975]: 1.1 scrub starts 2024-02-09T18:01:56.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:56 smithi179 ceph-mon[43975]: 1.1 scrub ok 2024-02-09T18:01:56.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:56 smithi162 ceph-mon[44304]: pgmap v2276: 47 pgs: 47 active+clean; 8.1 GiB data, 33 GiB used, 951 GiB / 983 GiB avail; 127 B/s rd, 32 MiB/s wr, 805 op/s 2024-02-09T18:01:56.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:56 smithi162 ceph-mon[44304]: 1.1 scrub starts 2024-02-09T18:01:56.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:56 smithi162 ceph-mon[44304]: 1.1 scrub ok 2024-02-09T18:01:57.469 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:01:57.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:57.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:01:57.587 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:01:57.620 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:57.655 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:01:57.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:57.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:01:57.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:57 smithi099 ceph-mon[39420]: 9.4 scrub starts 2024-02-09T18:01:57.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:57 smithi099 ceph-mon[39420]: 9.a scrub ok 2024-02-09T18:01:57.763 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:01:57.814 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:57.912 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:01:57.916 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:57.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:01:57.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:57 smithi179 ceph-mon[43975]: 9.4 scrub starts 2024-02-09T18:01:57.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:57 smithi179 ceph-mon[43975]: 9.a scrub ok 2024-02-09T18:01:57.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:57 smithi162 ceph-mon[44304]: 9.4 scrub starts 2024-02-09T18:01:57.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:57 smithi162 ceph-mon[44304]: 9.a scrub ok 2024-02-09T18:01:57.955 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-02-09T18:01:57.955 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph pg dump --format=json 2024-02-09T18:01:58.018 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T18:01:58.055 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:58.079 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T18:01:58.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:58.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T18:01:58.186 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:01:58.219 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:58.253 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:01:58.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:58.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:01:58.360 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T18:01:58.401 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:58.427 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T18:01:58.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:58.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T18:01:58.534 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:01:58.563 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:58.597 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:01:58.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:58.607 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:01:58.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:58 smithi162 ceph-mon[44304]: pgmap v2277: 47 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 45 active+clean; 8.0 GiB data, 32 GiB used, 951 GiB / 983 GiB avail; 127 B/s rd, 18 MiB/s wr, 835 op/s 2024-02-09T18:01:58.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:58 smithi162 ceph-mon[44304]: 9.4 scrub ok 2024-02-09T18:01:58.707 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T18:01:58.746 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:58.777 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T18:01:58.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:58.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T18:01:58.886 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:01:58.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:58 smithi179 ceph-mon[43975]: pgmap v2277: 47 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 45 active+clean; 8.0 GiB data, 32 GiB used, 951 GiB / 983 GiB avail; 127 B/s rd, 18 MiB/s wr, 835 op/s 2024-02-09T18:01:58.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:58 smithi179 ceph-mon[43975]: 9.4 scrub ok 2024-02-09T18:01:58.934 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:58.935 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:58 smithi099 ceph-mon[39420]: pgmap v2277: 47 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 45 active+clean; 8.0 GiB data, 32 GiB used, 951 GiB / 983 GiB avail; 127 B/s rd, 18 MiB/s wr, 835 op/s 2024-02-09T18:01:58.935 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:58 smithi099 ceph-mon[39420]: 9.4 scrub ok 2024-02-09T18:01:58.957 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:01:58.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:58.963 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:01:59.064 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:01:59.102 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:59.181 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:01:59.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:01:59.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:01:59.276 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T18:01:59.276 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T18:01:59.276 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T18:01:59.277 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T18:01:59.290 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:01:59.307 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T18:01:59.358 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T18:01:59.361 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:01:59.416 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T18:01:59.465 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T18:01:59.515 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T18:01:59.557 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T18:01:59.594 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T18:01:59.626 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T18:01:59.626 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T18:01:59.626 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T18:01:59.626 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T18:01:59.626 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T18:01:59.626 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T18:01:59.626 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T18:01:59.626 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T18:01:59.626 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T18:01:59.626 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T18:01:59.626 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T18:01:59.627 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T18:01:59.627 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T18:01:59.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:59 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:01:59.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:59 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:01:59.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:59 smithi179 ceph-mon[43975]: osdmap e922: 12 total, 12 up, 11 in 2024-02-09T18:01:59.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:01:59 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2107241620' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:01:59.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:59 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:01:59.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:59 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:01:59.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:59 smithi162 ceph-mon[44304]: osdmap e922: 12 total, 12 up, 11 in 2024-02-09T18:01:59.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:01:59 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2107241620' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:01:59.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:59 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:01:59.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:59 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:01:59.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:59 smithi099 ceph-mon[39420]: osdmap e922: 12 total, 12 up, 11 in 2024-02-09T18:01:59.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:01:59 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2107241620' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:02:00.716 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T18:02:00.716 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T18:02:00.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:00 smithi162 ceph-mon[44304]: pgmap v2279: 47 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 45 active+clean; 7.8 GiB data, 32 GiB used, 951 GiB / 983 GiB avail; 1.6 MiB/s wr, 875 op/s 2024-02-09T18:02:00.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:00 smithi099 ceph-mon[39420]: pgmap v2279: 47 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 45 active+clean; 7.8 GiB data, 32 GiB used, 951 GiB / 983 GiB avail; 1.6 MiB/s wr, 875 op/s 2024-02-09T18:02:01.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:00 smithi179 ceph-mon[43975]: pgmap v2279: 47 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 45 active+clean; 7.8 GiB data, 32 GiB used, 951 GiB / 983 GiB avail; 1.6 MiB/s wr, 875 op/s 2024-02-09T18:02:01.379 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:02:01.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:01.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:02:01.488 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:02:01.534 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:01.765 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:02:01.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:01.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:02:01.869 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:02:01.921 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:02.001 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:01 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:02:02.001 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:01 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:02:02.001 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:01 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:02:02.002 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:01 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:02:02.002 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:01 smithi099 ceph-mon[39420]: osdmap e923: 12 total, 12 up, 11 in 2024-02-09T18:02:02.002 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:01 smithi099 ceph-mon[39420]: 1.13 scrub starts 2024-02-09T18:02:02.060 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:02:02.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:02.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:02:02.171 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T18:02:02.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:01 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:02:02.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:01 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:02:02.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:01 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:02:02.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:01 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:02:02.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:01 smithi179 ceph-mon[43975]: osdmap e923: 12 total, 12 up, 11 in 2024-02-09T18:02:02.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:01 smithi179 ceph-mon[43975]: 1.13 scrub starts 2024-02-09T18:02:02.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:01 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:02:02.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:01 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:02:02.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:01 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:02:02.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:01 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:02:02.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:01 smithi162 ceph-mon[44304]: osdmap e923: 12 total, 12 up, 11 in 2024-02-09T18:02:02.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:01 smithi162 ceph-mon[44304]: 1.13 scrub starts 2024-02-09T18:02:02.202 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:02.226 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T18:02:02.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:02.234 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T18:02:02.335 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:02:02.397 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:02.635 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:02:02.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:02.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:02:02.687 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-09T18:02:02.740 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:02:02.741 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:02:02.791 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:02.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:02 smithi099 ceph-mon[39420]: pgmap v2281: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 7.7 GiB data, 32 GiB used, 952 GiB / 983 GiB avail; 0 B/s wr, 877 op/s 2024-02-09T18:02:02.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:02 smithi099 ceph-mon[39420]: 1.13 scrub ok 2024-02-09T18:02:02.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:02 smithi099 ceph-mon[39420]: osdmap e924: 12 total, 12 up, 11 in 2024-02-09T18:02:02.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:02 smithi099 ceph-mon[39420]: 1.d scrub starts 2024-02-09T18:02:02.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:02 smithi099 ceph-mon[39420]: 1.d scrub ok 2024-02-09T18:02:02.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:02 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4155047812' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T18:02:02.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:02 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T18:02:03.083 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is set 2024-02-09T18:02:03.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:02 smithi179 ceph-mon[43975]: pgmap v2281: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 7.7 GiB data, 32 GiB used, 952 GiB / 983 GiB avail; 0 B/s wr, 877 op/s 2024-02-09T18:02:03.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:02 smithi179 ceph-mon[43975]: 1.13 scrub ok 2024-02-09T18:02:03.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:02 smithi179 ceph-mon[43975]: osdmap e924: 12 total, 12 up, 11 in 2024-02-09T18:02:03.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:02 smithi179 ceph-mon[43975]: 1.d scrub starts 2024-02-09T18:02:03.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:02 smithi179 ceph-mon[43975]: 1.d scrub ok 2024-02-09T18:02:03.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:02 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4155047812' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T18:02:03.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:02 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T18:02:03.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:02 smithi162 ceph-mon[44304]: pgmap v2281: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 7.7 GiB data, 32 GiB used, 952 GiB / 983 GiB avail; 0 B/s wr, 877 op/s 2024-02-09T18:02:03.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:02 smithi162 ceph-mon[44304]: 1.13 scrub ok 2024-02-09T18:02:03.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:02 smithi162 ceph-mon[44304]: osdmap e924: 12 total, 12 up, 11 in 2024-02-09T18:02:03.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:02 smithi162 ceph-mon[44304]: 1.d scrub starts 2024-02-09T18:02:03.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:02 smithi162 ceph-mon[44304]: 1.d scrub ok 2024-02-09T18:02:03.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:02 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4155047812' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T18:02:03.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:02 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T18:02:03.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:03 smithi099 ceph-mon[39420]: 1.12 scrub starts 2024-02-09T18:02:03.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:03 smithi099 ceph-mon[39420]: 1.12 scrub ok 2024-02-09T18:02:03.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:03 smithi099 ceph-mon[39420]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:02:03.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:03 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T18:02:03.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:03 smithi099 ceph-mon[39420]: osdmap e925: 12 total, 12 up, 11 in 2024-02-09T18:02:04.045 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:02:04.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:04.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:02:04.154 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T18:02:04.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:03 smithi179 ceph-mon[43975]: 1.12 scrub starts 2024-02-09T18:02:04.175 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:03 smithi179 ceph-mon[43975]: 1.12 scrub ok 2024-02-09T18:02:04.175 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:03 smithi179 ceph-mon[43975]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:02:04.175 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:03 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T18:02:04.175 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:03 smithi179 ceph-mon[43975]: osdmap e925: 12 total, 12 up, 11 in 2024-02-09T18:02:04.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:03 smithi162 ceph-mon[44304]: 1.12 scrub starts 2024-02-09T18:02:04.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:03 smithi162 ceph-mon[44304]: 1.12 scrub ok 2024-02-09T18:02:04.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:03 smithi162 ceph-mon[44304]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:02:04.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:03 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T18:02:04.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:03 smithi162 ceph-mon[44304]: osdmap e925: 12 total, 12 up, 11 in 2024-02-09T18:02:04.185 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:02:04.215 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:04.250 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T18:02:04.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:04.254 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T18:02:04.355 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:02:04.387 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:04.410 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:02:04.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:04.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:02:04.516 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:02:04.551 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:04.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:04 smithi099 ceph-mon[39420]: pgmap v2284: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 7.7 GiB data, 31 GiB used, 952 GiB / 983 GiB avail; 0 B/s wr, 856 op/s 2024-02-09T18:02:05.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:04 smithi179 ceph-mon[43975]: pgmap v2284: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 7.7 GiB data, 31 GiB used, 952 GiB / 983 GiB avail; 0 B/s wr, 856 op/s 2024-02-09T18:02:05.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:04 smithi162 ceph-mon[44304]: pgmap v2284: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 7.7 GiB data, 31 GiB used, 952 GiB / 983 GiB avail; 0 B/s wr, 856 op/s 2024-02-09T18:02:05.211 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:02:05.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:05.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:02:05.256 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T18:02:05.317 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:02:05.390 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:05.868 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:02:05.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:05.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:02:05.975 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:02:06.011 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:06.084 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-09T18:02:06.490 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:02:06.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:06.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:02:06.602 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:02:06.640 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:06.682 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:02:06.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:06.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:02:06.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:06 smithi099 ceph-mon[39420]: pgmap v2285: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 7.6 GiB data, 31 GiB used, 952 GiB / 983 GiB avail; 0 B/s wr, 666 op/s 2024-02-09T18:02:06.791 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:02:06.850 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:06.880 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:02:06.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:06.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:02:06.931 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:06 smithi162 ceph-mon[44304]: pgmap v2285: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 7.6 GiB data, 31 GiB used, 952 GiB / 983 GiB avail; 0 B/s wr, 666 op/s 2024-02-09T18:02:06.987 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:02:07.038 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:07.062 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:02:07.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:07.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:02:07.170 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:02:07.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:06 smithi179 ceph-mon[43975]: pgmap v2285: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 7.6 GiB data, 31 GiB used, 952 GiB / 983 GiB avail; 0 B/s wr, 666 op/s 2024-02-09T18:02:07.202 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:07.231 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:02:07.234 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:07.234 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:02:07.335 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T18:02:07.369 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:07.395 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T18:02:07.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:07.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T18:02:07.504 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:02:07.545 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:07.571 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:02:07.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:07.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:02:07.677 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:02:07.794 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:08.069 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:08 smithi099 ceph-mon[39420]: pgmap v2286: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 7.5 GiB data, 31 GiB used, 953 GiB / 983 GiB avail; 0 B/s wr, 715 op/s 2024-02-09T18:02:08.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:08 smithi179 ceph-mon[43975]: pgmap v2286: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 7.5 GiB data, 31 GiB used, 953 GiB / 983 GiB avail; 0 B/s wr, 715 op/s 2024-02-09T18:02:08.431 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:08 smithi162 ceph-mon[44304]: pgmap v2286: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 7.5 GiB data, 31 GiB used, 953 GiB / 983 GiB avail; 0 B/s wr, 715 op/s 2024-02-09T18:02:08.569 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T18:02:08.585 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-02-09T18:02:08.774 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T18:02:08.774 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T18:02:09.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:09 smithi179 ceph-mon[43975]: from='client.17154 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T18:02:09.431 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:09 smithi162 ceph-mon[44304]: from='client.17154 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T18:02:09.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:09 smithi099 ceph-mon[39420]: from='client.17154 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T18:02:10.368 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:02:10.376 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:10 smithi099 ceph-mon[39420]: pgmap v2287: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 7.4 GiB data, 31 GiB used, 953 GiB / 983 GiB avail; 0 B/s wr, 778 op/s 2024-02-09T18:02:10.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:10.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:02:10.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:10 smithi179 ceph-mon[43975]: pgmap v2287: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 7.4 GiB data, 31 GiB used, 953 GiB / 983 GiB avail; 0 B/s wr, 778 op/s 2024-02-09T18:02:10.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:10 smithi162 ceph-mon[44304]: pgmap v2287: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 7.4 GiB data, 31 GiB used, 953 GiB / 983 GiB avail; 0 B/s wr, 778 op/s 2024-02-09T18:02:10.489 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:02:10.526 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:11.276 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:02:11.356 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:11 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T18:02:11.402 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-09T18:02:11.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:11 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T18:02:11.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:11 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T18:02:11.887 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:02:11.887 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:02:11.896 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":2286,"stamp":"2024-02-09T18:02:07.294920+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8025473592,"num_objects":122453,"num_object_clones":0,"num_object_copies":367359,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":122453,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147309,"num_write_kb":8632257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152489,"ondisk_log_size":152489,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":144,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":32328752,"kb_used_data":23767164,"kb_used_omap":429,"kb_used_meta":8560722,"kb_avail":998867920,"statfs":{"total":1055945392128,"available":1022840750080,"internally_reserved":0,"allocated":24337575936,"data_stored":24334429629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":439572,"internal_metadata":8766180076},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-310116352,"num_objects":-4732,"num_object_clones":0,"num_object_copies":-14196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4732,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.612162"},"pg_stats":[{"pgid":"1.1e","version":"63'139","reported_seq":1172,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:03.062191+0000","last_change":"2024-02-09T18:01:38.358617+0000","last_active":"2024-02-09T18:02:03.062191+0000","last_peered":"2024-02-09T18:02:03.062191+0000","last_clean":"2024-02-09T18:02:03.062191+0000","last_became_active":"2024-02-09T17:59:12.002465+0000","last_became_peered":"2024-02-09T17:59:12.002465+0000","last_unstale":"2024-02-09T18:02:03.062191+0000","last_undegraded":"2024-02-09T18:02:03.062191+0000","last_fullsized":"2024-02-09T18:02:03.062191+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:38.358497+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:01:38.358497+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:03:07.994234+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,7],"acting":[11,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"63'139","reported_seq":1191,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:03.074878+0000","last_change":"2024-02-09T18:01:42.251012+0000","last_active":"2024-02-09T18:02:03.074878+0000","last_peered":"2024-02-09T18:02:03.074878+0000","last_clean":"2024-02-09T18:02:03.074878+0000","last_became_active":"2024-02-09T17:43:28.594296+0000","last_became_peered":"2024-02-09T17:43:28.594296+0000","last_unstale":"2024-02-09T18:02:03.074878+0000","last_undegraded":"2024-02-09T18:02:03.074878+0000","last_fullsized":"2024-02-09T18:02:03.074878+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:42.250926+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:48:24.634092+0000","last_clean_scrub_stamp":"2024-02-09T18:01:42.250926+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:49.813369+0000","scrub_duration":153,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"63'139","reported_seq":1709,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-02-09T18:02:01.701703+0000","last_change":"2024-02-09T18:01:40.309486+0000","last_active":"2024-02-09T18:02:01.701703+0000","last_peered":"2024-02-09T18:02:01.701703+0000","last_clean":"2024-02-09T18:02:01.701703+0000","last_became_active":"2024-02-09T17:43:19.103584+0000","last_became_peered":"2024-02-09T17:43:19.103584+0000","last_unstale":"2024-02-09T18:02:01.701703+0000","last_undegraded":"2024-02-09T18:02:01.701703+0000","last_fullsized":"2024-02-09T18:02:01.701703+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:40.309363+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:23.912504+0000","last_clean_scrub_stamp":"2024-02-09T18:01:40.309363+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:48.235694+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"63'139","reported_seq":1507,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-02-09T18:02:01.701550+0000","last_change":"2024-02-09T18:01:36.490300+0000","last_active":"2024-02-09T18:02:01.701550+0000","last_peered":"2024-02-09T18:02:01.701550+0000","last_clean":"2024-02-09T18:02:01.701550+0000","last_became_active":"2024-02-09T17:43:19.102729+0000","last_became_peered":"2024-02-09T17:43:19.102729+0000","last_unstale":"2024-02-09T18:02:01.701550+0000","last_undegraded":"2024-02-09T18:02:01.701550+0000","last_fullsized":"2024-02-09T18:02:01.701550+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:36.490178+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:45:44.988633+0000","last_clean_scrub_stamp":"2024-02-09T18:01:36.490178+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:47.318325+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"63'139","reported_seq":403,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:03.078985+0000","last_change":"2024-02-09T18:00:46.991880+0000","last_active":"2024-02-09T18:02:03.078985+0000","last_peered":"2024-02-09T18:02:03.078985+0000","last_clean":"2024-02-09T18:02:03.078985+0000","last_became_active":"2024-02-09T17:59:30.061943+0000","last_became_peered":"2024-02-09T17:59:30.061943+0000","last_unstale":"2024-02-09T18:02:03.078985+0000","last_undegraded":"2024-02-09T18:02:03.078985+0000","last_fullsized":"2024-02-09T18:02:03.078985+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:00:46.991759+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:00:46.991759+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,11],"acting":[1,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.19","version":"63'139","reported_seq":2018,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:03.070154+0000","last_change":"2024-02-09T18:01:35.688095+0000","last_active":"2024-02-09T18:02:03.070154+0000","last_peered":"2024-02-09T18:02:03.070154+0000","last_clean":"2024-02-09T18:02:03.070154+0000","last_became_active":"2024-02-09T17:55:21.231291+0000","last_became_peered":"2024-02-09T17:55:21.231291+0000","last_unstale":"2024-02-09T18:02:03.070154+0000","last_undegraded":"2024-02-09T18:02:03.070154+0000","last_fullsized":"2024-02-09T18:02:03.070154+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":791,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:35.687991+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:30.193194+0000","last_clean_scrub_stamp":"2024-02-09T18:01:35.687991+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:52.363506+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"63'139","reported_seq":1855,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:03.066542+0000","last_change":"2024-02-09T18:01:31.621577+0000","last_active":"2024-02-09T18:02:03.066542+0000","last_peered":"2024-02-09T18:02:03.066542+0000","last_clean":"2024-02-09T18:02:03.066542+0000","last_became_active":"2024-02-09T17:43:28.585573+0000","last_became_peered":"2024-02-09T17:43:28.585573+0000","last_unstale":"2024-02-09T18:02:03.066542+0000","last_undegraded":"2024-02-09T18:02:03.066542+0000","last_fullsized":"2024-02-09T18:02:03.066542+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:31.621498+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:01:31.621498+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:51.160170+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"63'139","reported_seq":1299,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-02-09T18:02:01.701956+0000","last_change":"2024-02-09T18:01:45.544759+0000","last_active":"2024-02-09T18:02:01.701956+0000","last_peered":"2024-02-09T18:02:01.701956+0000","last_clean":"2024-02-09T18:02:01.701956+0000","last_became_active":"2024-02-09T17:48:46.184419+0000","last_became_peered":"2024-02-09T17:48:46.184419+0000","last_unstale":"2024-02-09T18:02:01.701956+0000","last_undegraded":"2024-02-09T18:02:01.701956+0000","last_fullsized":"2024-02-09T18:02:01.701956+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:45.544660+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:44:50.214686+0000","last_clean_scrub_stamp":"2024-02-09T18:01:45.544660+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:58.555199+0000","scrub_duration":211,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"63'139","reported_seq":1185,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:03.074946+0000","last_change":"2024-02-09T18:01:34.123704+0000","last_active":"2024-02-09T18:02:03.074946+0000","last_peered":"2024-02-09T18:02:03.074946+0000","last_clean":"2024-02-09T18:02:03.074946+0000","last_became_active":"2024-02-09T17:59:12.003987+0000","last_became_peered":"2024-02-09T17:59:12.003987+0000","last_unstale":"2024-02-09T18:02:03.074946+0000","last_undegraded":"2024-02-09T18:02:03.074946+0000","last_fullsized":"2024-02-09T18:02:03.074946+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:34.123585+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:11.970338+0000","last_clean_scrub_stamp":"2024-02-09T18:01:34.123585+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:58.496407+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"63'139","reported_seq":1612,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:03.060537+0000","last_change":"2024-02-09T18:01:53.216105+0000","last_active":"2024-02-09T18:02:03.060537+0000","last_peered":"2024-02-09T18:02:03.060537+0000","last_clean":"2024-02-09T18:02:03.060537+0000","last_became_active":"2024-02-09T17:43:19.138398+0000","last_became_peered":"2024-02-09T17:43:19.138398+0000","last_unstale":"2024-02-09T18:02:03.060537+0000","last_undegraded":"2024-02-09T18:02:03.060537+0000","last_fullsized":"2024-02-09T18:02:03.060537+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:53.215991+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:01:53.215991+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:03:22.751872+0000","scrub_duration":218,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"63'139","reported_seq":1683,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:03.061590+0000","last_change":"2024-02-09T18:01:35.453210+0000","last_active":"2024-02-09T18:02:03.061590+0000","last_peered":"2024-02-09T18:02:03.061590+0000","last_clean":"2024-02-09T18:02:03.061590+0000","last_became_active":"2024-02-09T17:43:28.589307+0000","last_became_peered":"2024-02-09T17:43:28.589307+0000","last_unstale":"2024-02-09T18:02:03.061590+0000","last_undegraded":"2024-02-09T18:02:03.061590+0000","last_fullsized":"2024-02-09T18:02:03.061590+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:35.453077+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:38:26.968892+0000","last_clean_scrub_stamp":"2024-02-09T18:01:35.453077+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:53.479776+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"924'9298","reported_seq":9477,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-02-09T18:02:02.563762+0000","last_change":"2024-02-09T18:01:34.244695+0000","last_active":"2024-02-09T18:02:02.563762+0000","last_peered":"2024-02-09T18:02:02.563762+0000","last_clean":"2024-02-09T18:02:02.563762+0000","last_became_active":"2024-02-09T18:00:07.530069+0000","last_became_peered":"2024-02-09T18:00:07.530069+0000","last_unstale":"2024-02-09T18:02:02.563762+0000","last_undegraded":"2024-02-09T18:02:02.563762+0000","last_fullsized":"2024-02-09T18:02:02.563762+0000","mapping_epoch":882,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"914'7732","last_scrub_stamp":"2024-02-09T18:01:34.244573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_clean_scrub_stamp":"2024-02-09T18:01:34.244573+0000","objects_scrubbed":7702,"log_size":9298,"log_dups_size":0,"ondisk_log_size":9298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:43.273517+0000","scrub_duration":1550,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":514850816,"num_objects":7856,"num_object_clones":0,"num_object_copies":23568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9298,"num_write_kb":548928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,5],"acting":[9,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"63'139","reported_seq":2262,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:03.060283+0000","last_change":"2024-02-09T18:01:48.080986+0000","last_active":"2024-02-09T18:02:03.060283+0000","last_peered":"2024-02-09T18:02:03.060283+0000","last_clean":"2024-02-09T18:02:03.060283+0000","last_became_active":"2024-02-09T17:41:14.521079+0000","last_became_peered":"2024-02-09T17:41:14.521079+0000","last_unstale":"2024-02-09T18:02:03.060283+0000","last_undegraded":"2024-02-09T18:02:03.060283+0000","last_fullsized":"2024-02-09T18:02:03.060283+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:48.080862+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:01:48.080862+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:03:14.657015+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.a","version":"925'9250","reported_seq":9437,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:05.546101+0000","last_change":"2024-02-09T18:01:56.341368+0000","last_active":"2024-02-09T18:02:05.546101+0000","last_peered":"2024-02-09T18:02:05.546101+0000","last_clean":"2024-02-09T18:02:05.546101+0000","last_became_active":"2024-02-09T18:00:07.530662+0000","last_became_peered":"2024-02-09T18:00:07.530662+0000","last_unstale":"2024-02-09T18:02:05.546101+0000","last_undegraded":"2024-02-09T18:02:05.546101+0000","last_fullsized":"2024-02-09T18:02:05.546101+0000","mapping_epoch":882,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"921'8782","last_scrub_stamp":"2024-02-09T18:01:56.341243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_clean_scrub_stamp":"2024-02-09T18:01:56.341243+0000","objects_scrubbed":8111,"log_size":9250,"log_dups_size":0,"ondisk_log_size":9250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:03:18.609818+0000","scrub_duration":2244,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":496762880,"num_objects":7580,"num_object_clones":0,"num_object_copies":22740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9250,"num_write_kb":538560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"63'139","reported_seq":1845,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:03.081824+0000","last_change":"2024-02-09T18:01:50.478494+0000","last_active":"2024-02-09T18:02:03.081824+0000","last_peered":"2024-02-09T18:02:03.081824+0000","last_clean":"2024-02-09T18:02:03.081824+0000","last_became_active":"2024-02-09T17:59:30.054441+0000","last_became_peered":"2024-02-09T17:59:30.054441+0000","last_unstale":"2024-02-09T18:02:03.081824+0000","last_undegraded":"2024-02-09T18:02:03.081824+0000","last_fullsized":"2024-02-09T18:02:03.081824+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:50.478373+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:01:50.478373+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:03:15.818287+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,8],"acting":[0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"924'9108","reported_seq":9293,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-02-09T18:02:02.574757+0000","last_change":"2024-02-09T18:01:44.676180+0000","last_active":"2024-02-09T18:02:02.574757+0000","last_peered":"2024-02-09T18:02:02.574757+0000","last_clean":"2024-02-09T18:02:02.574757+0000","last_became_active":"2024-02-09T18:00:07.529897+0000","last_became_peered":"2024-02-09T18:00:07.529897+0000","last_unstale":"2024-02-09T18:02:02.574757+0000","last_undegraded":"2024-02-09T18:02:02.574757+0000","last_fullsized":"2024-02-09T18:02:02.574757+0000","mapping_epoch":882,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"917'8181","last_scrub_stamp":"2024-02-09T18:01:44.676020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_clean_scrub_stamp":"2024-02-09T18:01:44.676020+0000","objects_scrubbed":8120,"log_size":9108,"log_dups_size":0,"ondisk_log_size":9108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:45.871299+0000","scrub_duration":2392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":512229376,"num_objects":7816,"num_object_clones":0,"num_object_copies":23448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9108,"num_write_kb":541568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"63'139","reported_seq":1469,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:03.074642+0000","last_change":"2024-02-09T18:01:35.235026+0000","last_active":"2024-02-09T18:02:03.074642+0000","last_peered":"2024-02-09T18:02:03.074642+0000","last_clean":"2024-02-09T18:02:03.074642+0000","last_became_active":"2024-02-09T17:59:30.073778+0000","last_became_peered":"2024-02-09T17:59:30.073778+0000","last_unstale":"2024-02-09T18:02:03.074642+0000","last_undegraded":"2024-02-09T18:02:03.074642+0000","last_fullsized":"2024-02-09T18:02:03.074642+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:35.234907+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:46:40.684762+0000","last_clean_scrub_stamp":"2024-02-09T18:01:35.234907+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:53.437281+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,8],"acting":[1,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"925'9236","reported_seq":9422,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:03.920228+0000","last_change":"2024-02-09T18:01:43.964751+0000","last_active":"2024-02-09T18:02:03.920228+0000","last_peered":"2024-02-09T18:02:03.920228+0000","last_clean":"2024-02-09T18:02:03.920228+0000","last_became_active":"2024-02-09T18:00:07.523197+0000","last_became_peered":"2024-02-09T18:00:07.523197+0000","last_unstale":"2024-02-09T18:02:03.920228+0000","last_undegraded":"2024-02-09T18:02:03.920228+0000","last_fullsized":"2024-02-09T18:02:03.920228+0000","mapping_epoch":882,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"917'8180","last_scrub_stamp":"2024-02-09T18:01:43.964525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_clean_scrub_stamp":"2024-02-09T18:01:43.964525+0000","objects_scrubbed":8109,"log_size":9236,"log_dups_size":0,"ondisk_log_size":9236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:52.470537+0000","scrub_duration":2493,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":506593280,"num_objects":7730,"num_object_clones":0,"num_object_copies":23190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9236,"num_write_kb":542912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,5],"acting":[10,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"63'139","reported_seq":1004,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:03.070302+0000","last_change":"2024-02-09T18:01:55.578635+0000","last_active":"2024-02-09T18:02:03.070302+0000","last_peered":"2024-02-09T18:02:03.070302+0000","last_clean":"2024-02-09T18:02:03.070302+0000","last_became_active":"2024-02-09T17:54:32.415032+0000","last_became_peered":"2024-02-09T17:54:32.415032+0000","last_unstale":"2024-02-09T18:02:03.070302+0000","last_undegraded":"2024-02-09T18:02:03.070302+0000","last_fullsized":"2024-02-09T18:02:03.070302+0000","mapping_epoch":766,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":767,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:55.578511+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T18:00:46.701010+0000","last_clean_scrub_stamp":"2024-02-09T18:01:55.578511+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:03:05.676030+0000","scrub_duration":102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.e","version":"925'9171","reported_seq":9356,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:03.921036+0000","last_change":"2024-02-09T18:01:47.601326+0000","last_active":"2024-02-09T18:02:03.921036+0000","last_peered":"2024-02-09T18:02:03.921036+0000","last_clean":"2024-02-09T18:02:03.921036+0000","last_became_active":"2024-02-09T18:00:07.522722+0000","last_became_peered":"2024-02-09T18:00:07.522722+0000","last_unstale":"2024-02-09T18:02:03.921036+0000","last_undegraded":"2024-02-09T18:02:03.921036+0000","last_fullsized":"2024-02-09T18:02:03.921036+0000","mapping_epoch":882,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"919'8328","last_scrub_stamp":"2024-02-09T18:01:47.601038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_clean_scrub_stamp":"2024-02-09T18:01:47.601038+0000","objects_scrubbed":8281,"log_size":9171,"log_dups_size":0,"ondisk_log_size":9171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:50.286460+0000","scrub_duration":2081,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":504430592,"num_objects":7697,"num_object_clones":0,"num_object_copies":23091,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9171,"num_write_kb":539776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,1],"acting":[10,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"63'139","reported_seq":1763,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:03.072701+0000","last_change":"2024-02-09T18:01:34.702740+0000","last_active":"2024-02-09T18:02:03.072701+0000","last_peered":"2024-02-09T18:02:03.072701+0000","last_clean":"2024-02-09T18:02:03.072701+0000","last_became_active":"2024-02-09T17:43:14.024894+0000","last_became_peered":"2024-02-09T17:43:14.024894+0000","last_unstale":"2024-02-09T18:02:03.072701+0000","last_undegraded":"2024-02-09T18:02:03.072701+0000","last_fullsized":"2024-02-09T18:02:03.072701+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:34.702627+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:11.970338+0000","last_clean_scrub_stamp":"2024-02-09T18:01:34.702627+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:38.122355+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.f","version":"925'10874","reported_seq":11049,"reported_epoch":925,"state":"active+clean+scrubbing+deep","last_fresh":"2024-02-09T18:02:05.575079+0000","last_change":"2024-02-09T18:01:53.142466+0000","last_active":"2024-02-09T18:02:05.575079+0000","last_peered":"2024-02-09T18:02:05.575079+0000","last_clean":"2024-02-09T18:02:05.575079+0000","last_became_active":"2024-02-09T18:00:07.523115+0000","last_became_peered":"2024-02-09T18:00:07.523115+0000","last_unstale":"2024-02-09T18:02:05.575079+0000","last_undegraded":"2024-02-09T18:02:05.575079+0000","last_fullsized":"2024-02-09T18:02:05.575079+0000","mapping_epoch":882,"log_start":"889'800","ondisk_log_start":"889'800","created":882,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_clean_scrub_stamp":"2024-02-09T18:00:06.282957+0000","objects_scrubbed":1949,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 12s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":500498432,"num_objects":7637,"num_object_clones":0,"num_object_copies":22911,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9203,"num_write_kb":538880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,9],"acting":[1,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"63'139","reported_seq":1388,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:03.084470+0000","last_change":"2024-02-09T18:01:37.427302+0000","last_active":"2024-02-09T18:02:03.084470+0000","last_peered":"2024-02-09T18:02:03.084470+0000","last_clean":"2024-02-09T18:02:03.084470+0000","last_became_active":"2024-02-09T17:52:32.292519+0000","last_became_peered":"2024-02-09T17:52:32.292519+0000","last_unstale":"2024-02-09T18:02:03.084470+0000","last_undegraded":"2024-02-09T18:02:03.084470+0000","last_fullsized":"2024-02-09T18:02:03.084470+0000","mapping_epoch":716,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":717,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:37.427192+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:01:37.427192+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:59.188404+0000","scrub_duration":384,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.c","version":"925'9329","reported_seq":9509,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:06.054466+0000","last_change":"2024-02-09T18:01:34.347659+0000","last_active":"2024-02-09T18:02:06.054466+0000","last_peered":"2024-02-09T18:02:06.054466+0000","last_clean":"2024-02-09T18:02:06.054466+0000","last_became_active":"2024-02-09T18:00:07.372788+0000","last_became_peered":"2024-02-09T18:00:07.372788+0000","last_unstale":"2024-02-09T18:02:06.054466+0000","last_undegraded":"2024-02-09T18:02:06.054466+0000","last_fullsized":"2024-02-09T18:02:06.054466+0000","mapping_epoch":882,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"914'7638","last_scrub_stamp":"2024-02-09T18:01:34.347419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_clean_scrub_stamp":"2024-02-09T18:01:34.347419+0000","objects_scrubbed":7603,"log_size":9329,"log_dups_size":0,"ondisk_log_size":9329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:42.673871+0000","scrub_duration":1789,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":500105216,"num_objects":7631,"num_object_clones":0,"num_object_copies":22893,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9329,"num_write_kb":542720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,0],"acting":[3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"63'139","reported_seq":1842,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:03.062082+0000","last_change":"2024-02-09T18:01:40.226412+0000","last_active":"2024-02-09T18:02:03.062082+0000","last_peered":"2024-02-09T18:02:03.062082+0000","last_clean":"2024-02-09T18:02:03.062082+0000","last_became_active":"2024-02-09T17:33:00.284924+0000","last_became_peered":"2024-02-09T17:33:00.284924+0000","last_unstale":"2024-02-09T18:02:03.062082+0000","last_undegraded":"2024-02-09T18:02:03.062082+0000","last_fullsized":"2024-02-09T18:02:03.062082+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:40.226335+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:56:45.406767+0000","last_clean_scrub_stamp":"2024-02-09T18:01:40.226335+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:03:07.722142+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.d","version":"925'9219","reported_seq":9407,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:05.558097+0000","last_change":"2024-02-09T18:01:47.090054+0000","last_active":"2024-02-09T18:02:05.558097+0000","last_peered":"2024-02-09T18:02:05.558097+0000","last_clean":"2024-02-09T18:02:05.558097+0000","last_became_active":"2024-02-09T18:00:07.530751+0000","last_became_peered":"2024-02-09T18:00:07.530751+0000","last_unstale":"2024-02-09T18:02:05.558097+0000","last_undegraded":"2024-02-09T18:02:05.558097+0000","last_fullsized":"2024-02-09T18:02:05.558097+0000","mapping_epoch":882,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"919'8257","last_scrub_stamp":"2024-02-09T18:01:47.089900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_clean_scrub_stamp":"2024-02-09T18:01:47.089900+0000","objects_scrubbed":8211,"log_size":9219,"log_dups_size":0,"ondisk_log_size":9219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:53.209083+0000","scrub_duration":2004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":495976472,"num_objects":7569,"num_object_clones":0,"num_object_copies":22707,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7569,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9219,"num_write_kb":537153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"63'139","reported_seq":1692,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:03.069834+0000","last_change":"2024-02-09T18:01:36.708824+0000","last_active":"2024-02-09T18:02:03.069834+0000","last_peered":"2024-02-09T18:02:03.069834+0000","last_clean":"2024-02-09T18:02:03.069834+0000","last_became_active":"2024-02-09T17:58:10.037010+0000","last_became_peered":"2024-02-09T17:58:10.037010+0000","last_unstale":"2024-02-09T18:02:03.069834+0000","last_undegraded":"2024-02-09T18:02:03.069834+0000","last_fullsized":"2024-02-09T18:02:03.069834+0000","mapping_epoch":839,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":840,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:36.708692+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:01:36.708692+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:03:03.478015+0000","scrub_duration":171,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,8],"acting":[10,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.2","version":"925'9292","reported_seq":9471,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:06.013393+0000","last_change":"2024-02-09T18:01:35.388996+0000","last_active":"2024-02-09T18:02:06.013393+0000","last_peered":"2024-02-09T18:02:06.013393+0000","last_clean":"2024-02-09T18:02:06.013393+0000","last_became_active":"2024-02-09T18:00:07.021665+0000","last_became_peered":"2024-02-09T18:00:07.021665+0000","last_unstale":"2024-02-09T18:02:06.013393+0000","last_undegraded":"2024-02-09T18:02:06.013393+0000","last_fullsized":"2024-02-09T18:02:06.013393+0000","mapping_epoch":882,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"914'7575","last_scrub_stamp":"2024-02-09T18:01:35.388781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_clean_scrub_stamp":"2024-02-09T18:01:35.388781+0000","objects_scrubbed":7520,"log_size":9292,"log_dups_size":0,"ondisk_log_size":9292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:42.842479+0000","scrub_duration":1798,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":493355008,"num_objects":7528,"num_object_clones":0,"num_object_copies":22584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9292,"num_write_kb":538240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,9],"acting":[3,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"63'139","reported_seq":1771,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-02-09T18:02:01.702085+0000","last_change":"2024-02-09T18:01:46.756318+0000","last_active":"2024-02-09T18:02:01.702085+0000","last_peered":"2024-02-09T18:02:01.702085+0000","last_clean":"2024-02-09T18:02:01.702085+0000","last_became_active":"2024-02-09T17:59:30.057223+0000","last_became_peered":"2024-02-09T17:59:30.057223+0000","last_unstale":"2024-02-09T18:02:01.702085+0000","last_undegraded":"2024-02-09T18:02:01.702085+0000","last_fullsized":"2024-02-09T18:02:01.702085+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:46.756218+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:56:44.962386+0000","last_clean_scrub_stamp":"2024-02-09T18:01:46.756218+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:03:11.667820+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,0],"acting":[9,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.3","version":"924'9135","reported_seq":9319,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-02-09T18:02:02.599557+0000","last_change":"2024-02-09T18:01:45.906378+0000","last_active":"2024-02-09T18:02:02.599557+0000","last_peered":"2024-02-09T18:02:02.599557+0000","last_clean":"2024-02-09T18:02:02.599557+0000","last_became_active":"2024-02-09T18:00:07.523241+0000","last_became_peered":"2024-02-09T18:00:07.523241+0000","last_unstale":"2024-02-09T18:02:02.599557+0000","last_undegraded":"2024-02-09T18:02:02.599557+0000","last_fullsized":"2024-02-09T18:02:02.599557+0000","mapping_epoch":882,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"918'8296","last_scrub_stamp":"2024-02-09T18:01:45.904138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_clean_scrub_stamp":"2024-02-09T18:01:45.904138+0000","objects_scrubbed":8245,"log_size":9135,"log_dups_size":0,"ondisk_log_size":9135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:47.771934+0000","scrub_duration":2159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":509804544,"num_objects":7779,"num_object_clones":0,"num_object_copies":23337,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9135,"num_write_kb":541248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,5],"acting":[9,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.b","version":"63'139","reported_seq":1131,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:03.061906+0000","last_change":"2024-02-09T18:01:34.507016+0000","last_active":"2024-02-09T18:02:03.061906+0000","last_peered":"2024-02-09T18:02:03.061906+0000","last_clean":"2024-02-09T18:02:03.061906+0000","last_became_active":"2024-02-09T17:55:08.938203+0000","last_became_peered":"2024-02-09T17:55:08.938203+0000","last_unstale":"2024-02-09T18:02:03.061906+0000","last_undegraded":"2024-02-09T18:02:03.061906+0000","last_fullsized":"2024-02-09T18:02:03.061906+0000","mapping_epoch":781,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":782,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:34.506894+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:01:34.506894+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:39.439332+0000","scrub_duration":295,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.0","version":"925'9306","reported_seq":9494,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:05.554676+0000","last_change":"2024-02-09T18:01:46.422565+0000","last_active":"2024-02-09T18:02:05.554676+0000","last_peered":"2024-02-09T18:02:05.554676+0000","last_clean":"2024-02-09T18:02:05.554676+0000","last_became_active":"2024-02-09T18:00:07.131105+0000","last_became_peered":"2024-02-09T18:00:07.131105+0000","last_unstale":"2024-02-09T18:02:05.554676+0000","last_undegraded":"2024-02-09T18:02:05.554676+0000","last_fullsized":"2024-02-09T18:02:05.554676+0000","mapping_epoch":882,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"918'8233","last_scrub_stamp":"2024-02-09T18:01:46.422346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_clean_scrub_stamp":"2024-02-09T18:01:46.422346+0000","objects_scrubbed":8168,"log_size":9306,"log_dups_size":0,"ondisk_log_size":9306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:47.555306+0000","scrub_duration":2345,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":495321088,"num_objects":7558,"num_object_clones":0,"num_object_copies":22674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9306,"num_write_kb":539648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"63'139","reported_seq":1261,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:03.077669+0000","last_change":"2024-02-09T18:01:47.651147+0000","last_active":"2024-02-09T18:02:03.077669+0000","last_peered":"2024-02-09T18:02:03.077669+0000","last_clean":"2024-02-09T18:02:03.077669+0000","last_became_active":"2024-02-09T17:59:12.022685+0000","last_became_peered":"2024-02-09T17:59:12.022685+0000","last_unstale":"2024-02-09T18:02:03.077669+0000","last_undegraded":"2024-02-09T18:02:03.077669+0000","last_fullsized":"2024-02-09T18:02:03.077669+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:47.651055+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:01:47.651055+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:03:03.122903+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1","version":"924'9049","reported_seq":9231,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-02-09T18:02:02.587312+0000","last_change":"2024-02-09T18:01:44.919616+0000","last_active":"2024-02-09T18:02:02.587312+0000","last_peered":"2024-02-09T18:02:02.587312+0000","last_clean":"2024-02-09T18:02:02.587312+0000","last_became_active":"2024-02-09T18:00:07.131217+0000","last_became_peered":"2024-02-09T18:00:07.131217+0000","last_unstale":"2024-02-09T18:02:02.587312+0000","last_undegraded":"2024-02-09T18:02:02.587312+0000","last_fullsized":"2024-02-09T18:02:02.587312+0000","mapping_epoch":882,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"917'8098","last_scrub_stamp":"2024-02-09T18:01:44.919443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_clean_scrub_stamp":"2024-02-09T18:01:44.919443+0000","objects_scrubbed":8057,"log_size":9049,"log_dups_size":0,"ondisk_log_size":9049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:03:13.502087+0000","scrub_duration":2210,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":501284864,"num_objects":7649,"num_object_clones":0,"num_object_copies":22947,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9049,"num_write_kb":534336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,6],"acting":[9,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.9","version":"63'139","reported_seq":2268,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:03.060247+0000","last_change":"2024-02-09T18:01:44.388634+0000","last_active":"2024-02-09T18:02:03.060247+0000","last_peered":"2024-02-09T18:02:03.060247+0000","last_clean":"2024-02-09T18:02:03.060247+0000","last_became_active":"2024-02-09T17:43:28.590237+0000","last_became_peered":"2024-02-09T17:43:28.590237+0000","last_unstale":"2024-02-09T18:02:03.060247+0000","last_undegraded":"2024-02-09T18:02:03.060247+0000","last_fullsized":"2024-02-09T18:02:03.060247+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:44.388511+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:59:49.716259+0000","last_clean_scrub_stamp":"2024-02-09T18:01:44.388511+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:59.402237+0000","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.6","version":"925'9100","reported_seq":9283,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:03.891052+0000","last_change":"2024-02-09T18:01:40.622569+0000","last_active":"2024-02-09T18:02:03.891052+0000","last_peered":"2024-02-09T18:02:03.891052+0000","last_clean":"2024-02-09T18:02:03.891052+0000","last_became_active":"2024-02-09T18:00:07.523268+0000","last_became_peered":"2024-02-09T18:00:07.523268+0000","last_unstale":"2024-02-09T18:02:03.891052+0000","last_undegraded":"2024-02-09T18:02:03.891052+0000","last_fullsized":"2024-02-09T18:02:03.891052+0000","mapping_epoch":882,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"917'7893","last_scrub_stamp":"2024-02-09T18:01:40.622417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_clean_scrub_stamp":"2024-02-09T18:01:40.622417+0000","objects_scrubbed":7824,"log_size":9100,"log_dups_size":0,"ondisk_log_size":9100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:03:03.921720+0000","scrub_duration":2093,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":494927872,"num_objects":7552,"num_object_clones":0,"num_object_copies":22656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9100,"num_write_kb":532864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,1],"acting":[10,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.e","version":"63'139","reported_seq":400,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:03.060389+0000","last_change":"2024-02-09T18:01:43.193091+0000","last_active":"2024-02-09T18:02:03.060389+0000","last_peered":"2024-02-09T18:02:03.060389+0000","last_clean":"2024-02-09T18:02:03.060389+0000","last_became_active":"2024-02-09T17:59:30.050934+0000","last_became_peered":"2024-02-09T17:59:30.050934+0000","last_unstale":"2024-02-09T18:02:03.060389+0000","last_undegraded":"2024-02-09T18:02:03.060389+0000","last_fullsized":"2024-02-09T18:02:03.060389+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:43.192960+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:14.981453+0000","last_clean_scrub_stamp":"2024-02-09T18:01:43.192960+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:03:12.502112+0000","scrub_duration":73,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,8],"acting":[7,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"925'9102","reported_seq":9289,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:03.912480+0000","last_change":"2024-02-09T18:01:49.345309+0000","last_active":"2024-02-09T18:02:03.912480+0000","last_peered":"2024-02-09T18:02:03.912480+0000","last_clean":"2024-02-09T18:02:03.912480+0000","last_became_active":"2024-02-09T18:00:07.554448+0000","last_became_peered":"2024-02-09T18:00:07.554448+0000","last_unstale":"2024-02-09T18:02:03.912480+0000","last_undegraded":"2024-02-09T18:02:03.912480+0000","last_fullsized":"2024-02-09T18:02:03.912480+0000","mapping_epoch":882,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"919'8379","last_scrub_stamp":"2024-02-09T18:01:49.345117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_clean_scrub_stamp":"2024-02-09T18:01:49.345117+0000","objects_scrubbed":8341,"log_size":9102,"log_dups_size":0,"ondisk_log_size":9102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:03:09.342813+0000","scrub_duration":1866,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503316480,"num_objects":7680,"num_object_clones":0,"num_object_copies":23040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9102,"num_write_kb":537024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,3],"acting":[10,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"63'139","reported_seq":1793,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:03.090754+0000","last_change":"2024-02-09T18:01:33.327365+0000","last_active":"2024-02-09T18:02:03.090754+0000","last_peered":"2024-02-09T18:02:03.090754+0000","last_clean":"2024-02-09T18:02:03.090754+0000","last_became_active":"2024-02-09T17:59:30.045852+0000","last_became_peered":"2024-02-09T17:59:30.045852+0000","last_unstale":"2024-02-09T18:02:03.090754+0000","last_undegraded":"2024-02-09T18:02:03.090754+0000","last_fullsized":"2024-02-09T18:02:03.090754+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:33.327269+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:48:34.651647+0000","last_clean_scrub_stamp":"2024-02-09T18:01:33.327269+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:44.381596+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"925'9191","reported_seq":9375,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:05.557749+0000","last_change":"2024-02-09T18:01:58.054003+0000","last_active":"2024-02-09T18:02:05.557749+0000","last_peered":"2024-02-09T18:02:05.557749+0000","last_clean":"2024-02-09T18:02:05.557749+0000","last_became_active":"2024-02-09T18:00:07.024203+0000","last_became_peered":"2024-02-09T18:00:07.024203+0000","last_unstale":"2024-02-09T18:02:05.557749+0000","last_undegraded":"2024-02-09T18:02:05.557749+0000","last_fullsized":"2024-02-09T18:02:05.557749+0000","mapping_epoch":882,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"921'8840","last_scrub_stamp":"2024-02-09T18:01:58.053829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_clean_scrub_stamp":"2024-02-09T18:01:58.053829+0000","objects_scrubbed":7986,"log_size":9191,"log_dups_size":0,"ondisk_log_size":9191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:03:20.575579+0000","scrub_duration":1808,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":497876992,"num_objects":7597,"num_object_clones":0,"num_object_copies":22791,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9191,"num_write_kb":537216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"63'139","reported_seq":1756,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:03.077582+0000","last_change":"2024-02-09T18:01:39.607321+0000","last_active":"2024-02-09T18:02:03.077582+0000","last_peered":"2024-02-09T18:02:03.077582+0000","last_clean":"2024-02-09T18:02:03.077582+0000","last_became_active":"2024-02-09T17:59:12.208604+0000","last_became_peered":"2024-02-09T17:59:12.208604+0000","last_unstale":"2024-02-09T18:02:03.077582+0000","last_undegraded":"2024-02-09T18:02:03.077582+0000","last_fullsized":"2024-02-09T18:02:03.077582+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:39.607215+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:14.135321+0000","last_clean_scrub_stamp":"2024-02-09T18:01:39.607215+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:57.277819+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"925'9320","reported_seq":9505,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:07.000249+0000","last_change":"2024-02-09T18:01:43.124506+0000","last_active":"2024-02-09T18:02:07.000249+0000","last_peered":"2024-02-09T18:02:07.000249+0000","last_clean":"2024-02-09T18:02:07.000249+0000","last_became_active":"2024-02-09T18:00:07.530724+0000","last_became_peered":"2024-02-09T18:00:07.530724+0000","last_unstale":"2024-02-09T18:02:07.000249+0000","last_undegraded":"2024-02-09T18:02:07.000249+0000","last_fullsized":"2024-02-09T18:02:07.000249+0000","mapping_epoch":882,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"917'8139","last_scrub_stamp":"2024-02-09T18:01:43.124250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_clean_scrub_stamp":"2024-02-09T18:01:43.124250+0000","objects_scrubbed":8088,"log_size":9320,"log_dups_size":0,"ondisk_log_size":9320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:03:12.471464+0000","scrub_duration":1914,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":497549312,"num_objects":7592,"num_object_clones":0,"num_object_copies":22776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9320,"num_write_kb":541184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"63'139","reported_seq":1334,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:03.060615+0000","last_change":"2024-02-09T18:02:02.290371+0000","last_active":"2024-02-09T18:02:03.060615+0000","last_peered":"2024-02-09T18:02:03.060615+0000","last_clean":"2024-02-09T18:02:03.060615+0000","last_became_active":"2024-02-09T17:59:30.053570+0000","last_became_peered":"2024-02-09T17:59:30.053570+0000","last_unstale":"2024-02-09T18:02:03.060615+0000","last_undegraded":"2024-02-09T18:02:03.060615+0000","last_fullsized":"2024-02-09T18:02:03.060615+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:02:02.290244+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T18:00:49.890794+0000","last_clean_scrub_stamp":"2024-02-09T18:02:02.290244+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:03:14.747875+0000","scrub_duration":245,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,8],"acting":[7,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"63'139","reported_seq":1488,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:03.060219+0000","last_change":"2024-02-09T18:00:40.745321+0000","last_active":"2024-02-09T18:02:03.060219+0000","last_peered":"2024-02-09T18:02:03.060219+0000","last_clean":"2024-02-09T18:02:03.060219+0000","last_became_active":"2024-02-09T17:46:47.013920+0000","last_became_peered":"2024-02-09T17:46:47.013920+0000","last_unstale":"2024-02-09T18:02:03.060219+0000","last_undegraded":"2024-02-09T18:02:03.060219+0000","last_fullsized":"2024-02-09T18:02:03.060219+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:00:40.745190+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:48:35.170638+0000","last_clean_scrub_stamp":"2024-02-09T18:00:40.745190+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:09.398468+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"63'139","reported_seq":1757,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-02-09T18:02:01.701377+0000","last_change":"2024-02-09T18:01:41.456117+0000","last_active":"2024-02-09T18:02:01.701377+0000","last_peered":"2024-02-09T18:02:01.701377+0000","last_clean":"2024-02-09T18:02:01.701377+0000","last_became_active":"2024-02-09T17:55:21.231902+0000","last_became_peered":"2024-02-09T17:55:21.231902+0000","last_unstale":"2024-02-09T18:02:01.701377+0000","last_undegraded":"2024-02-09T18:02:01.701377+0000","last_fullsized":"2024-02-09T18:02:01.701377+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":791,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:41.456012+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:44:51.190789+0000","last_clean_scrub_stamp":"2024-02-09T18:01:41.456012+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:03:04.012313+0000","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.12","version":"63'139","reported_seq":1842,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:03.078893+0000","last_change":"2024-02-09T18:02:02.359151+0000","last_active":"2024-02-09T18:02:03.078893+0000","last_peered":"2024-02-09T18:02:03.078893+0000","last_clean":"2024-02-09T18:02:03.078893+0000","last_became_active":"2024-02-09T17:46:47.015622+0000","last_became_peered":"2024-02-09T17:46:47.015622+0000","last_unstale":"2024-02-09T18:02:03.078893+0000","last_undegraded":"2024-02-09T18:02:03.078893+0000","last_fullsized":"2024-02-09T18:02:03.078893+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:02:02.359039+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:02:02.359039+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:03:02.704420+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.13","version":"63'139","reported_seq":1461,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:03.074698+0000","last_change":"2024-02-09T18:02:01.517991+0000","last_active":"2024-02-09T18:02:03.074698+0000","last_peered":"2024-02-09T18:02:03.074698+0000","last_clean":"2024-02-09T18:02:03.074698+0000","last_became_active":"2024-02-09T17:41:14.521672+0000","last_became_peered":"2024-02-09T17:41:14.521672+0000","last_unstale":"2024-02-09T18:02:03.074698+0000","last_undegraded":"2024-02-09T18:02:03.074698+0000","last_fullsized":"2024-02-09T18:02:03.074698+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:02:01.517864+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:02:01.517864+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:03:09.590936+0000","scrub_duration":293,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":8024883224,"num_objects":122451,"num_object_clones":0,"num_object_copies":367353,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":122451,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147309,"num_write_kb":8632257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":23991496704,"data_stored":23991484488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148180,"ondisk_log_size":148180,"up":48,"acting":48,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547057119,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2888536,"kb_used_data":1941072,"kb_used_omap":1,"kb_used_meta":947454,"kb_avail":90856616,"statfs":{"total":95995035648,"available":93037174784,"internally_reserved":0,"allocated":1987657728,"data_stored":1987372969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":970193348},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 18:01:13 2024","interfaces":[{"interface":"back","average":{"1min":1.5660000000000001,"5min":1.5409999999999999,"15min":1.6799999999999999},"min":{"1min":0.26100000000000001,"5min":0.22,"15min":0.21199999999999999},"max":{"1min":6.0030000000000001,"5min":13.307,"15min":20.003},"last":2.9940000000000002},{"interface":"front","average":{"1min":1.635,"5min":1.978,"15min":1.6639999999999999},"min":{"1min":0.34200000000000003,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":5.2519999999999998,"5min":9.0790000000000006,"15min":9.6259999999999994},"last":0.32900000000000001}]},{"osd":1,"last update":"Fri Feb 9 18:01:13 2024","interfaces":[{"interface":"back","average":{"1min":1.5209999999999999,"5min":2.1190000000000002,"15min":1.823},"min":{"1min":0.29999999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":5.1669999999999998,"5min":11.795999999999999,"15min":11.795999999999999},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.69599999999999995,"5min":2.1070000000000002,"15min":1.929},"min":{"1min":0.32300000000000001,"5min":0.26400000000000001,"15min":0.25},"max":{"1min":2.6240000000000001,"5min":14.81,"15min":14.81},"last":3.1859999999999999}]},{"osd":2,"last update":"Fri Feb 9 18:01:16 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":2.105,"15min":1.6919999999999999},"min":{"1min":0.34300000000000003,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.3999999999999999,"5min":14.766999999999999,"15min":14.766999999999999},"last":3.2650000000000001},{"interface":"front","average":{"1min":1.087,"5min":1.923,"15min":1.429},"min":{"1min":0.316,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":7.3499999999999996,"5min":13.337,"15min":13.337},"last":3.214}]},{"osd":3,"last update":"Fri Feb 9 18:01:16 2024","interfaces":[{"interface":"back","average":{"1min":1.6240000000000001,"5min":1.617,"15min":1.6160000000000001},"min":{"1min":0.41499999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":5.3250000000000002,"5min":7.5259999999999998,"15min":7.5259999999999998},"last":0.42099999999999999},{"interface":"front","average":{"1min":1.206,"5min":1.8420000000000001,"15min":1.948},"min":{"1min":0.435,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":5.2889999999999997,"5min":8.4770000000000003,"15min":8.4770000000000003},"last":3.1560000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:58:44 2024","interfaces":[{"interface":"back","average":{"1min":3.1709999999999998,"5min":2.5779999999999998,"15min":2.4790000000000001},"min":{"1min":0.40999999999999998,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":14.866,"5min":14.866,"15min":14.866},"last":0.45700000000000002},{"interface":"front","average":{"1min":3.2349999999999999,"5min":2.5640000000000001,"15min":2.452},"min":{"1min":0.496,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":14.939,"5min":14.939,"15min":14.939},"last":0.42299999999999999}]},{"osd":5,"last update":"Fri Feb 9 18:01:13 2024","interfaces":[{"interface":"back","average":{"1min":1.611,"5min":1.5780000000000001,"15min":1.524},"min":{"1min":0.30499999999999999,"5min":0.27000000000000002,"15min":0.252},"max":{"1min":5.2229999999999999,"5min":13.263999999999999,"15min":18.878},"last":3.0840000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":1.988,"15min":1.681},"min":{"1min":0.34899999999999998,"5min":0.30099999999999999,"15min":0.28999999999999998},"max":{"1min":1.3380000000000001,"5min":14.914999999999999,"15min":20.038},"last":3.3370000000000002}]},{"osd":6,"last update":"Fri Feb 9 18:01:13 2024","interfaces":[{"interface":"back","average":{"1min":1.4530000000000001,"5min":2.0489999999999999,"15min":1.6100000000000001},"min":{"1min":0.38300000000000001,"5min":0.36899999999999999,"15min":0.30399999999999999},"max":{"1min":5.2080000000000002,"5min":11.885,"15min":11.885},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.65200000000000002,"5min":2.024,"15min":1.6499999999999999},"min":{"1min":0.38100000000000001,"5min":0.36899999999999999,"15min":0.32400000000000001},"max":{"1min":1.4810000000000001,"5min":14.962,"15min":15.409000000000001},"last":3.121}]},{"osd":7,"last update":"Fri Feb 9 18:01:13 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":2.0579999999999998,"15min":1.7250000000000001},"min":{"1min":0.40300000000000002,"5min":0.32800000000000001,"15min":0.317},"max":{"1min":3.621,"5min":14.890000000000001,"15min":18.846},"last":3.2400000000000002},{"interface":"front","average":{"1min":1.825,"5min":1.921,"15min":1.9419999999999999},"min":{"1min":0.52700000000000002,"5min":0.33400000000000002,"15min":0.318},"max":{"1min":5.3369999999999997,"5min":9.1029999999999998,"15min":24.373999999999999},"last":0.39600000000000002}]},{"osd":8,"last update":"Fri Feb 9 18:01:24 2024","interfaces":[{"interface":"back","average":{"1min":1.9259999999999999,"5min":2.7799999999999998,"15min":1.931},"min":{"1min":0.46000000000000002,"5min":0.307,"15min":0.27900000000000003},"max":{"1min":5.3730000000000002,"5min":29.233000000000001,"15min":29.233000000000001},"last":0.747},{"interface":"front","average":{"1min":1.9770000000000001,"5min":2.976,"15min":2.0350000000000001},"min":{"1min":0.55100000000000005,"5min":0.34499999999999997,"15min":0.32700000000000001},"max":{"1min":5.3959999999999999,"5min":29.091999999999999,"15min":29.091999999999999},"last":0.77700000000000002}]},{"osd":9,"last update":"Fri Feb 9 18:01:55 2024","interfaces":[{"interface":"back","average":{"1min":2.2810000000000001,"5min":2.3900000000000001,"15min":2.0350000000000001},"min":{"1min":0.36699999999999999,"5min":0.32600000000000001,"15min":0.29399999999999998},"max":{"1min":6.0810000000000004,"5min":19.469000000000001,"15min":22.613},"last":3.0310000000000001},{"interface":"front","average":{"1min":2.343,"5min":2.456,"15min":1.8759999999999999},"min":{"1min":0.47199999999999998,"5min":0.35899999999999999,"15min":0.34399999999999997},"max":{"1min":6.1120000000000001,"5min":19.555,"15min":19.555},"last":3.056}]},{"osd":10,"last update":"Fri Feb 9 18:01:13 2024","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":3.1230000000000002,"15min":2.214},"min":{"1min":0.45000000000000001,"5min":0.42899999999999999,"15min":0.32700000000000001},"max":{"1min":5.2220000000000004,"5min":25.093,"15min":25.093},"last":3.2890000000000001},{"interface":"front","average":{"1min":1.244,"5min":2.6659999999999999,"15min":1.9159999999999999},"min":{"1min":0.52800000000000002,"5min":0.437,"15min":0.35999999999999999},"max":{"1min":5.2549999999999999,"5min":14.836,"15min":20.233000000000001},"last":3.3130000000000002}]}]},{"osd":10,"up_from":84,"seq":360777253350,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3330460,"kb_used_data":2464912,"kb_used_omap":1,"kb_used_meta":865534,"kb_avail":90414692,"statfs":{"total":95995035648,"available":92584644608,"internally_reserved":0,"allocated":2524069888,"data_stored":2523785129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":886307264},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 18:01:01 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":1.909,"15min":1.5329999999999999},"min":{"1min":0.26800000000000002,"5min":0.25600000000000001,"15min":0.23999999999999999},"max":{"1min":2.5350000000000001,"5min":12.029999999999999,"15min":12.029999999999999},"last":0.314},{"interface":"front","average":{"1min":0.94499999999999995,"5min":2.105,"15min":1.5940000000000001},"min":{"1min":0.34799999999999998,"5min":0.254,"15min":0.254},"max":{"1min":2.4889999999999999,"5min":15.802,"15min":15.802},"last":0.52400000000000002}]},{"osd":1,"last update":"Fri Feb 9 18:01:25 2024","interfaces":[{"interface":"back","average":{"1min":4.4770000000000003,"5min":2.6190000000000002,"15min":1.917},"min":{"1min":0.33000000000000002,"5min":0.254,"15min":0.254},"max":{"1min":52.210999999999999,"5min":52.210999999999999,"15min":52.210999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":4.5640000000000001,"5min":2.8700000000000001,"15min":2.0019999999999998},"min":{"1min":0.29299999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":51.537999999999997,"5min":51.537999999999997,"15min":51.537999999999997},"last":0.501}]},{"osd":2,"last update":"Fri Feb 9 18:01:25 2024","interfaces":[{"interface":"back","average":{"1min":1.5529999999999999,"5min":2.3599999999999999,"15min":2.0030000000000001},"min":{"1min":0.38500000000000001,"5min":0.30099999999999999,"15min":0.26600000000000001},"max":{"1min":5.2729999999999997,"5min":22.652999999999999,"15min":22.652999999999999},"last":0.40699999999999997},{"interface":"front","average":{"1min":1.4750000000000001,"5min":2.1389999999999998,"15min":2.0899999999999999},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.25600000000000001},"max":{"1min":5.4210000000000003,"5min":12.180999999999999,"15min":38.868000000000002},"last":0.434}]},{"osd":3,"last update":"Fri Feb 9 18:01:07 2024","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.421,"15min":1.421},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":4.383,"5min":4.383,"15min":4.383},"last":0.38300000000000001},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":3.7349999999999999,"5min":3.7349999999999999,"15min":3.7349999999999999},"last":0.67500000000000004}]},{"osd":4,"last update":"Fri Feb 9 18:01:55 2024","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.448,"15min":1.2490000000000001},"min":{"1min":0.35999999999999999,"5min":0.253,"15min":0.253},"max":{"1min":3.859,"5min":13.446,"15min":13.446},"last":0.57799999999999996},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.494,"15min":1.345},"min":{"1min":0.44900000000000001,"5min":0.34200000000000003,"15min":0.33600000000000002},"max":{"1min":5.367,"5min":12.097,"15min":12.097},"last":0.61099999999999999}]},{"osd":5,"last update":"Fri Feb 9 18:01:25 2024","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.5069999999999999,"15min":1.3440000000000001},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":3.806,"5min":13.4,"15min":13.4},"last":2.266},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.6839999999999999,"15min":1.448},"min":{"1min":0.374,"5min":0.315,"15min":0.315},"max":{"1min":5.3140000000000001,"5min":12.157999999999999,"15min":12.157999999999999},"last":2.2770000000000001}]},{"osd":6,"last update":"Fri Feb 9 18:01:39 2024","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.6180000000000001,"15min":1.4119999999999999},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.32300000000000001},"max":{"1min":3.8330000000000002,"5min":13.507,"15min":13.507},"last":0.622},{"interface":"front","average":{"1min":1.7310000000000001,"5min":1.8580000000000001,"15min":1.5820000000000001},"min":{"1min":0.499,"5min":0.38200000000000001,"15min":0.32600000000000001},"max":{"1min":6.7350000000000003,"5min":12.205,"15min":12.414},"last":0.55200000000000005}]},{"osd":7,"last update":"Fri Feb 9 18:01:25 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.623,"15min":2.218},"min":{"1min":0.37,"5min":0.35399999999999998,"15min":0.34000000000000002},"max":{"1min":3.8860000000000001,"5min":13.481,"15min":13.481},"last":0.69899999999999995},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.865,"15min":2.3170000000000002},"min":{"1min":0.38200000000000001,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":5.883,"5min":13.388999999999999,"15min":13.388999999999999},"last":0.59699999999999998}]},{"osd":8,"last update":"Fri Feb 9 18:01:25 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":3.0950000000000002,"15min":1.8899999999999999},"min":{"1min":0.39100000000000001,"5min":0.38100000000000001,"15min":0.31},"max":{"1min":5.21,"5min":22.559000000000001,"15min":22.559000000000001},"last":0.78600000000000003},{"interface":"front","average":{"1min":1.3200000000000001,"5min":2.4239999999999999,"15min":1.736},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.30599999999999999},"max":{"1min":3.7759999999999998,"5min":22.824999999999999,"15min":22.824999999999999},"last":0.65100000000000002}]},{"osd":9,"last update":"Fri Feb 9 18:01:27 2024","interfaces":[{"interface":"back","average":{"1min":1.841,"5min":2.9590000000000001,"15min":2.1200000000000001},"min":{"1min":0.36299999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":6.4050000000000002,"5min":23.201000000000001,"15min":23.201000000000001},"last":2.0390000000000001},{"interface":"front","average":{"1min":1.98,"5min":2.802,"15min":2.0830000000000002},"min":{"1min":0.45800000000000002,"5min":0.39800000000000002,"15min":0.36099999999999999},"max":{"1min":6.2460000000000004,"5min":17.498999999999999,"15min":17.498999999999999},"last":0.68799999999999994}]},{"osd":11,"last update":"Fri Feb 9 18:01:01 2024","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":2.536,"15min":1.7230000000000001},"min":{"1min":0.49299999999999999,"5min":0.45800000000000002,"15min":0.35399999999999998},"max":{"1min":2.677,"5min":21.18,"15min":23.579000000000001},"last":0.63800000000000001},{"interface":"front","average":{"1min":1.4590000000000001,"5min":2.8260000000000001,"15min":1.859},"min":{"1min":0.61599999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":4.4699999999999998,"5min":32.707000000000001,"15min":32.707000000000001},"last":0.66400000000000003}]}]},{"osd":9,"up_from":77,"seq":330712482280,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3908948,"kb_used_data":2993220,"kb_used_omap":1,"kb_used_meta":915518,"kb_avail":89836204,"statfs":{"total":95995035648,"available":91992272896,"internally_reserved":0,"allocated":3065057280,"data_stored":3064776978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":937490880},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 18:01:31 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.198,"15min":1.1879999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":5.5389999999999997,"5min":11.515000000000001,"15min":11.515000000000001},"last":3.383},{"interface":"front","average":{"1min":1.474,"5min":1.0149999999999999,"15min":0.93899999999999995},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":3.5899999999999999,"5min":6.7400000000000002,"15min":6.7400000000000002},"last":0.26500000000000001}]},{"osd":1,"last update":"Fri Feb 9 18:01:13 2024","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.595,"15min":1.712},"min":{"1min":0.26700000000000002,"5min":0.247,"15min":0.23599999999999999},"max":{"1min":3.4399999999999999,"5min":8.3190000000000008,"15min":61.031999999999996},"last":0.39200000000000002},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.7929999999999999,"15min":2.0049999999999999},"min":{"1min":0.316,"5min":0.25600000000000001,"15min":0.25},"max":{"1min":5.8019999999999996,"5min":8.4359999999999999,"15min":61.970999999999997},"last":0.32300000000000001}]},{"osd":2,"last update":"Fri Feb 9 17:58:45 2024","interfaces":[{"interface":"back","average":{"1min":2.286,"5min":1.71,"15min":1.8200000000000001},"min":{"1min":0.42499999999999999,"5min":0.27000000000000002,"15min":0.26900000000000002},"max":{"1min":8.4410000000000007,"5min":9.8689999999999998,"15min":10.657999999999999},"last":1.0329999999999999},{"interface":"front","average":{"1min":2.7839999999999998,"5min":2.2269999999999999,"15min":1.806},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.248},"max":{"1min":7.1449999999999996,"5min":11.35,"15min":11.35},"last":1.1240000000000001}]},{"osd":3,"last update":"Fri Feb 9 18:01:13 2024","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":0.96099999999999997,"15min":0.94699999999999995},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":2.8340000000000001,"5min":3.5710000000000002,"15min":3.5710000000000002},"last":1.4910000000000001},{"interface":"front","average":{"1min":1.1299999999999999,"5min":0.94899999999999995,"15min":0.91900000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":2.855,"5min":4.7850000000000001,"15min":4.7850000000000001},"last":0.36099999999999999}]},{"osd":4,"last update":"Fri Feb 9 18:01:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.6459999999999999,"15min":1.518},"min":{"1min":0.313,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":3.9900000000000002,"5min":10.725,"15min":19.414000000000001},"last":1.53},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.27,"15min":1.327},"min":{"1min":0.38200000000000001,"5min":0.29999999999999999,"15min":0.26900000000000002},"max":{"1min":2.714,"5min":8.3599999999999994,"15min":19.224},"last":0.42499999999999999}]},{"osd":5,"last update":"Fri Feb 9 18:01:24 2024","interfaces":[{"interface":"back","average":{"1min":1.5069999999999999,"5min":1.6579999999999999,"15min":1.752},"min":{"1min":0.40400000000000003,"5min":0.33200000000000002,"15min":0.30299999999999999},"max":{"1min":6.0940000000000003,"5min":10.763999999999999,"15min":77.924000000000007},"last":0.49099999999999999},{"interface":"front","average":{"1min":1.393,"5min":1.8380000000000001,"15min":1.7010000000000001},"min":{"1min":0.36099999999999999,"5min":0.316,"15min":0.28000000000000003},"max":{"1min":3.0979999999999999,"5min":10.851000000000001,"15min":77.899000000000001},"last":0.52600000000000002}]},{"osd":6,"last update":"Fri Feb 9 18:01:13 2024","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":1.5920000000000001,"15min":1.425},"min":{"1min":0.432,"5min":0.36599999999999999,"15min":0.314},"max":{"1min":2.7229999999999999,"5min":10.071,"15min":19.375},"last":0.71599999999999997},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.681,"15min":1.5409999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33500000000000002},"max":{"1min":2.778,"5min":9.9900000000000002,"15min":19.178000000000001},"last":0.64500000000000002}]},{"osd":7,"last update":"Fri Feb 9 18:01:53 2024","interfaces":[{"interface":"back","average":{"1min":2.1819999999999999,"5min":1.762,"15min":2.0190000000000001},"min":{"1min":0.378,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":8.8529999999999998,"5min":10.021000000000001,"15min":10.021000000000001},"last":0.68700000000000006},{"interface":"front","average":{"1min":1.79,"5min":2.0750000000000002,"15min":1.996},"min":{"1min":0.39900000000000002,"5min":0.379,"15min":0.379},"max":{"1min":7.008,"5min":11.138,"15min":11.138},"last":0.56299999999999994}]},{"osd":8,"last update":"Fri Feb 9 18:01:31 2024","interfaces":[{"interface":"back","average":{"1min":1.9990000000000001,"5min":2.0899999999999999,"15min":1.52},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.29599999999999999},"max":{"1min":8.9350000000000005,"5min":10.210000000000001,"15min":10.744},"last":0.79500000000000004},{"interface":"front","average":{"1min":2.044,"5min":2.1699999999999999,"15min":1.571},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.32200000000000001},"max":{"1min":8.9550000000000001,"5min":10.237,"15min":10.768000000000001},"last":5.2009999999999996}]},{"osd":10,"last update":"Fri Feb 9 18:01:41 2024","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":2.5880000000000001,"15min":2.0049999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":7.093,"5min":8.6969999999999992,"15min":20.773},"last":3.3540000000000001},{"interface":"front","average":{"1min":1.899,"5min":3.1160000000000001,"15min":1.9890000000000001},"min":{"1min":0.54600000000000004,"5min":0.40000000000000002,"15min":0.34699999999999998},"max":{"1min":5.673,"5min":18.579999999999998,"15min":20.637},"last":0.60099999999999998}]},{"osd":11,"last update":"Fri Feb 9 18:01:41 2024","interfaces":[{"interface":"back","average":{"1min":1.464,"5min":2.6869999999999998,"15min":1.9430000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.33000000000000002},"max":{"1min":3.7309999999999999,"5min":40.259999999999998,"15min":40.259999999999998},"last":0.81999999999999995},{"interface":"front","average":{"1min":1.5149999999999999,"5min":3.1469999999999998,"15min":1.9910000000000001},"min":{"1min":0.55400000000000005,"5min":0.40400000000000003,"15min":0.36499999999999999},"max":{"1min":3.7829999999999999,"5min":39.683,"15min":39.683},"last":0.85399999999999998}]}]},{"osd":8,"up_from":171,"seq":734439408023,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1517076,"kb_used_data":998276,"kb_used_omap":212,"kb_used_meta":518571,"kb_avail":92228076,"statfs":{"total":95995035648,"available":94441549824,"internally_reserved":0,"allocated":1022234624,"data_stored":1021954322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217579,"internal_metadata":531017237},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 18:01:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":1.8440000000000001,"15min":1.6579999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.22600000000000001},"max":{"1min":2.077,"5min":10.099,"15min":12.712999999999999},"last":0.26700000000000002},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.4910000000000001,"15min":1.516},"min":{"1min":0.29699999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":4.3129999999999997,"5min":12.178000000000001,"15min":12.725},"last":0.504}]},{"osd":1,"last update":"Fri Feb 9 18:01:22 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.52,"15min":1.5209999999999999},"min":{"1min":0.29199999999999998,"5min":0.248,"15min":0.23799999999999999},"max":{"1min":3.7890000000000001,"5min":6.8970000000000002,"15min":8.2550000000000008},"last":0.28299999999999997},{"interface":"front","average":{"1min":1.526,"5min":1.6850000000000001,"15min":1.7490000000000001},"min":{"1min":0.30199999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":3.9790000000000001,"5min":12.262,"15min":21.399000000000001},"last":0.46200000000000002}]},{"osd":2,"last update":"Fri Feb 9 18:01:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4990000000000001,"5min":1.851,"15min":1.349},"min":{"1min":0.34899999999999998,"5min":0.25,"15min":0.25},"max":{"1min":4.0890000000000004,"5min":6.9260000000000002,"15min":6.9260000000000002},"last":0.69599999999999995},{"interface":"front","average":{"1min":1.8660000000000001,"5min":1.873,"15min":1.7909999999999999},"min":{"1min":0.34300000000000003,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":4.1479999999999997,"5min":12.465999999999999,"15min":12.465999999999999},"last":1.282}]},{"osd":3,"last update":"Fri Feb 9 17:56:20 2024","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":2.552,"15min":2.8039999999999998},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":5.2560000000000002,"5min":15.654,"15min":15.654},"last":6.5350000000000001},{"interface":"front","average":{"1min":1.306,"5min":2.3149999999999999,"15min":2.0150000000000001},"min":{"1min":0.34300000000000003,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":5.2759999999999998,"5min":15.609999999999999,"15min":15.609999999999999},"last":6.5599999999999996}]},{"osd":4,"last update":"Fri Feb 9 18:01:22 2024","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.593,"15min":1.2509999999999999},"min":{"1min":0.32500000000000001,"5min":0.29499999999999998,"15min":0.29399999999999998},"max":{"1min":4,"5min":12.297000000000001,"15min":12.297000000000001},"last":1.349},{"interface":"front","average":{"1min":0.89000000000000001,"5min":1.5369999999999999,"15min":1.3759999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28000000000000003},"max":{"1min":3.9060000000000001,"5min":12.227,"15min":12.227},"last":0.36399999999999999}]},{"osd":5,"last update":"Fri Feb 9 18:01:53 2024","interfaces":[{"interface":"back","average":{"1min":1.696,"5min":1.661,"15min":1.526},"min":{"1min":0.376,"5min":0.307,"15min":0.27400000000000002},"max":{"1min":4.0179999999999998,"5min":12.4,"15min":12.4},"last":0.54900000000000004},{"interface":"front","average":{"1min":1.734,"5min":1.6739999999999999,"15min":1.4430000000000001},"min":{"1min":0.46500000000000002,"5min":0.35499999999999998,"15min":0.30299999999999999},"max":{"1min":4.0549999999999997,"5min":12.368,"15min":12.368},"last":0.66700000000000004}]},{"osd":6,"last update":"Fri Feb 9 18:01:22 2024","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.4690000000000001,"15min":1.5620000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.29899999999999999},"max":{"1min":3.8730000000000002,"5min":6.7599999999999998,"15min":9.7539999999999996},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.66000000000000003,"5min":1.7929999999999999,"15min":1.512},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.9379999999999999,"5min":10.178000000000001,"15min":10.178000000000001},"last":0.76000000000000001}]},{"osd":7,"last update":"Fri Feb 9 18:01:22 2024","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.9390000000000001,"15min":1.7110000000000001},"min":{"1min":0.42599999999999999,"5min":0.35999999999999999,"15min":0.32900000000000001},"max":{"1min":4.1050000000000004,"5min":12.380000000000001,"15min":12.380000000000001},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.80800000000000005,"5min":1.7450000000000001,"15min":1.591},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.309},"max":{"1min":2.036,"5min":10.148999999999999,"15min":10.148999999999999},"last":0.56499999999999995}]},{"osd":9,"last update":"Fri Feb 9 18:01:22 2024","interfaces":[{"interface":"back","average":{"1min":2.8140000000000001,"5min":3.4159999999999999,"15min":2.1989999999999998},"min":{"1min":0.45700000000000002,"5min":0.34999999999999998,"15min":0.26800000000000002},"max":{"1min":17.623000000000001,"5min":46.213999999999999,"15min":46.213999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":2.0299999999999998,"5min":3.496,"15min":2.286},"min":{"1min":0.41299999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":13.67,"5min":41.177,"15min":41.177},"last":0.53000000000000003}]},{"osd":10,"last update":"Fri Feb 9 18:01:22 2024","interfaces":[{"interface":"back","average":{"1min":1.9710000000000001,"5min":3.0270000000000001,"15min":1.869},"min":{"1min":0.432,"5min":0.39600000000000002,"15min":0.32300000000000001},"max":{"1min":18.164000000000001,"5min":18.164000000000001,"15min":18.164000000000001},"last":2.0449999999999999},{"interface":"front","average":{"1min":2.1269999999999998,"5min":2.6970000000000001,"15min":1.9770000000000001},"min":{"1min":0.45100000000000001,"5min":0.40200000000000002,"15min":0.34300000000000003},"max":{"1min":18.111000000000001,"5min":18.111000000000001,"15min":18.111000000000001},"last":2.0590000000000002}]},{"osd":11,"last update":"Fri Feb 9 18:01:53 2024","interfaces":[{"interface":"back","average":{"1min":2.089,"5min":2.3559999999999999,"15min":2.085},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":5.8380000000000001,"5min":10.231999999999999,"15min":10.231999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":2.1309999999999998,"5min":2.3980000000000001,"15min":2.1240000000000001},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":5.8540000000000001,"5min":10.256,"15min":10.256},"last":0.60399999999999998}]}]},{"osd":7,"up_from":57,"seq":244813136370,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3293732,"kb_used_data":2428188,"kb_used_omap":1,"kb_used_meta":865534,"kb_avail":90451420,"statfs":{"total":95995035648,"available":92622254080,"internally_reserved":0,"allocated":2486464512,"data_stored":2486172129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":886307274},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 18:02:00 2024","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.754,"15min":1.4590000000000001},"min":{"1min":0.316,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":5.3550000000000004,"5min":6.1420000000000003,"15min":101.167},"last":4.21},{"interface":"front","average":{"1min":1.2230000000000001,"5min":0.96499999999999997,"15min":1.4570000000000001},"min":{"1min":0.35299999999999998,"5min":0.28499999999999998,"15min":0.26800000000000002},"max":{"1min":5.3840000000000003,"5min":15.890000000000001,"15min":101.07599999999999},"last":4.194}]},{"osd":1,"last update":"Fri Feb 9 18:02:00 2024","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.04,"15min":1.375},"min":{"1min":0.29399999999999998,"5min":0.28499999999999998,"15min":0.28299999999999997},"max":{"1min":3.3159999999999998,"5min":8.7050000000000001,"15min":17.314},"last":0.35299999999999998},{"interface":"front","average":{"1min":1.387,"5min":1.1180000000000001,"15min":1.3400000000000001},"min":{"1min":0.30299999999999999,"5min":0.246,"15min":0.246},"max":{"1min":4.0330000000000004,"5min":15.935,"15min":15.935},"last":1.6859999999999999}]},{"osd":2,"last update":"Fri Feb 9 18:02:00 2024","interfaces":[{"interface":"back","average":{"1min":2.0190000000000001,"5min":1.4430000000000001,"15min":1.4510000000000001},"min":{"1min":0.42099999999999999,"5min":0.27300000000000002,"15min":0.25},"max":{"1min":10.602,"5min":16.190000000000001,"15min":16.190000000000001},"last":2.5859999999999999},{"interface":"front","average":{"1min":1.5049999999999999,"5min":1.3280000000000001,"15min":1.4990000000000001},"min":{"1min":0.35999999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":4.0019999999999998,"5min":15.965999999999999,"15min":15.965999999999999},"last":17.596}]},{"osd":3,"last update":"Fri Feb 9 18:01:14 2024","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":1.175,"15min":1.2050000000000001},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":3.4449999999999998,"5min":7.0270000000000001,"15min":7.0270000000000001},"last":2.5609999999999999},{"interface":"front","average":{"1min":1.1539999999999999,"5min":2.0230000000000001,"15min":2.1680000000000001},"min":{"1min":0.498,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":3.3260000000000001,"5min":16.029,"15min":16.029},"last":3.96}]},{"osd":4,"last update":"Fri Feb 9 18:02:00 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.63700000000000001,"15min":0.83299999999999996},"min":{"1min":0.34000000000000002,"5min":0.27100000000000002,"15min":0.25600000000000001},"max":{"1min":3.7210000000000001,"5min":4.3330000000000002,"15min":7.2999999999999998},"last":0.39500000000000002},{"interface":"front","average":{"1min":1.0920000000000001,"5min":0.88200000000000001,"15min":1.0409999999999999},"min":{"1min":0.379,"5min":0.26600000000000001,"15min":0.23899999999999999},"max":{"1min":5.6539999999999999,"5min":17.814,"15min":17.814},"last":0.33800000000000002}]},{"osd":5,"last update":"Fri Feb 9 18:02:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":0.72599999999999998,"15min":1.0049999999999999},"min":{"1min":0.315,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":3.4889999999999999,"5min":6.944,"15min":8.0060000000000002},"last":0.442},{"interface":"front","average":{"1min":1.145,"5min":0.82799999999999996,"15min":1.0569999999999999},"min":{"1min":0.35099999999999998,"5min":0.32900000000000001,"15min":0.32100000000000001},"max":{"1min":4.1980000000000004,"5min":6.9080000000000004,"15min":7.867},"last":0.56499999999999995}]},{"osd":6,"last update":"Fri Feb 9 18:01:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":0.995,"15min":1.1559999999999999},"min":{"1min":0.51300000000000001,"5min":0.36099999999999999,"15min":0.311},"max":{"1min":3.52,"5min":16.158000000000001,"15min":16.158000000000001},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.97899999999999998,"5min":0.79900000000000004,"15min":1.0349999999999999},"min":{"1min":0.42199999999999999,"5min":0.39500000000000002,"15min":0.32000000000000001},"max":{"1min":3.5190000000000001,"5min":7.0810000000000004,"15min":7.1669999999999998},"last":0.59799999999999998}]},{"osd":8,"last update":"Fri Feb 9 18:01:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.7310000000000001,"15min":1.482},"min":{"1min":0.44400000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":2.9740000000000002,"5min":11.648999999999999,"15min":11.648999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.79,"15min":1.4430000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.37},"max":{"1min":6.4589999999999996,"5min":11.629,"15min":11.629},"last":0.629}]},{"osd":9,"last update":"Fri Feb 9 18:01:06 2024","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.5980000000000001,"15min":1.5960000000000001},"min":{"1min":0.496,"5min":0.43099999999999999,"15min":0.375},"max":{"1min":3.605,"5min":8.0169999999999995,"15min":11.542999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":2.0289999999999999,"5min":1.8799999999999999,"15min":1.6910000000000001},"min":{"1min":0.53100000000000003,"5min":0.40100000000000002,"15min":0.40000000000000002},"max":{"1min":7.665,"5min":13.161,"15min":15.852},"last":0.71099999999999997}]},{"osd":10,"last update":"Fri Feb 9 18:01:44 2024","interfaces":[{"interface":"back","average":{"1min":2.4950000000000001,"5min":2.0179999999999998,"15min":1.54},"min":{"1min":0.60299999999999998,"5min":0.44900000000000001,"15min":0.32800000000000001},"max":{"1min":8.7669999999999995,"5min":15.999000000000001,"15min":15.999000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":2.0289999999999999,"5min":1.764,"15min":1.7569999999999999},"min":{"1min":0.55800000000000005,"5min":0.53700000000000003,"15min":0.40400000000000003},"max":{"1min":4.9400000000000004,"5min":8.5020000000000007,"15min":11.571999999999999},"last":0.61799999999999999}]},{"osd":11,"last update":"Fri Feb 9 18:02:00 2024","interfaces":[{"interface":"back","average":{"1min":1.6359999999999999,"5min":1.964,"15min":1.5469999999999999},"min":{"1min":0.54800000000000004,"5min":0.47299999999999998,"15min":0.35999999999999999},"max":{"1min":4.0679999999999996,"5min":16.111999999999998,"15min":16.111999999999998},"last":0.60899999999999999},{"interface":"front","average":{"1min":1.996,"5min":1.857,"15min":1.651},"min":{"1min":0.57099999999999995,"5min":0.53200000000000003,"15min":0.438},"max":{"1min":4.3819999999999997,"5min":7.0030000000000001,"15min":12.243},"last":1.284}]}]},{"osd":6,"up_from":49,"seq":210453398008,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1675160,"kb_used_data":967888,"kb_used_omap":1,"kb_used_meta":707262,"kb_avail":92069992,"statfs":{"total":95995035648,"available":94279671808,"internally_reserved":0,"allocated":991117312,"data_stored":990832553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":724236742},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 18:01:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69199999999999995,"15min":1.133},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":3.4260000000000002,"5min":4.5270000000000001,"15min":10.481999999999999},"last":0.433},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.66300000000000003,"15min":1.1890000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.24399999999999999},"max":{"1min":3.5539999999999998,"5min":3.5539999999999998,"15min":10.436},"last":1.506}]},{"osd":1,"last update":"Fri Feb 9 18:01:48 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.95099999999999996,"15min":1.7549999999999999},"min":{"1min":0.32000000000000001,"5min":0.29699999999999999,"15min":0.25},"max":{"1min":3.5710000000000002,"5min":4.7069999999999999,"15min":79.945999999999998},"last":3.1909999999999998},{"interface":"front","average":{"1min":0.90700000000000003,"5min":1.0069999999999999,"15min":1.806},"min":{"1min":0.309,"5min":0.309,"15min":0.28599999999999998},"max":{"1min":3.5880000000000001,"5min":4.7300000000000004,"15min":80.003},"last":3.2240000000000002}]},{"osd":2,"last update":"Fri Feb 9 18:01:48 2024","interfaces":[{"interface":"back","average":{"1min":1.5700000000000001,"5min":1.147,"15min":1.8200000000000001},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.28100000000000003},"max":{"1min":8.141,"5min":9.2970000000000006,"15min":84.912999999999997},"last":1.6579999999999999},{"interface":"front","average":{"1min":1.6120000000000001,"5min":1.204,"15min":1.8700000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.34000000000000002},"max":{"1min":8.1709999999999994,"5min":9.3260000000000005,"15min":84.942999999999998},"last":1.6930000000000001}]},{"osd":3,"last update":"Fri Feb 9 18:01:32 2024","interfaces":[{"interface":"back","average":{"1min":1.629,"5min":0.89300000000000002,"15min":0.77100000000000002},"min":{"1min":0.40000000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":5.6479999999999997,"5min":5.6479999999999997,"15min":5.6479999999999997},"last":1.595},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.153,"15min":1.173},"min":{"1min":0.34300000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":4.0490000000000004,"5min":9.0909999999999993,"15min":9.0909999999999993},"last":1.5209999999999999}]},{"osd":4,"last update":"Fri Feb 9 18:01:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1279999999999999,"5min":0.64600000000000002,"15min":1.0960000000000001},"min":{"1min":0.29199999999999998,"5min":0.27300000000000002,"15min":0.26200000000000001},"max":{"1min":5.6840000000000002,"5min":5.6840000000000002,"15min":14.182},"last":0.27600000000000002},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.70899999999999996,"15min":1.1579999999999999},"min":{"1min":0.35499999999999998,"5min":0.34100000000000003,"15min":0.314},"max":{"1min":5.7030000000000003,"5min":5.7030000000000003,"15min":14.210000000000001},"last":0.33100000000000002}]},{"osd":5,"last update":"Fri Feb 9 18:01:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.57199999999999995,"15min":1.1259999999999999},"min":{"1min":0.30599999999999999,"5min":0.30299999999999999,"15min":0.26600000000000001},"max":{"1min":4.4130000000000003,"5min":4.4130000000000003,"15min":14.151999999999999},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.66200000000000003,"15min":1.129},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.26100000000000001},"max":{"1min":3.468,"5min":3.5569999999999999,"15min":15.509},"last":0.47099999999999997}]},{"osd":7,"last update":"Fri Feb 9 18:01:24 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.65100000000000002,"15min":1.2110000000000001},"min":{"1min":0.32600000000000001,"5min":0.31,"15min":0.28699999999999998},"max":{"1min":3.5150000000000001,"5min":3.5150000000000001,"15min":23.265999999999998},"last":1.4850000000000001},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69899999999999995,"15min":1.0940000000000001},"min":{"1min":0.44,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.5209999999999999,"5min":5.4969999999999999,"15min":10.599},"last":0.64000000000000001}]},{"osd":8,"last update":"Fri Feb 9 17:58:34 2024","interfaces":[{"interface":"back","average":{"1min":2.806,"5min":1.635,"15min":1.27},"min":{"1min":0.41599999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":9.2940000000000005,"5min":9.2940000000000005,"15min":9.9480000000000004},"last":0.40300000000000002},{"interface":"front","average":{"1min":2.3780000000000001,"5min":1.623,"15min":1.3109999999999999},"min":{"1min":0.53400000000000003,"5min":0.41399999999999998,"15min":0.40400000000000003},"max":{"1min":9.3469999999999995,"5min":9.3469999999999995,"15min":11.316000000000001},"last":0.57899999999999996}]},{"osd":9,"last update":"Fri Feb 9 18:01:55 2024","interfaces":[{"interface":"back","average":{"1min":1.6220000000000001,"5min":1.5489999999999999,"15min":1.6060000000000001},"min":{"1min":0.53800000000000003,"5min":0.47799999999999998,"15min":0.32600000000000001},"max":{"1min":5.1820000000000004,"5min":7.6669999999999998,"15min":36.606000000000002},"last":0.68000000000000005},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.8200000000000001,"15min":1.843},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.34300000000000003},"max":{"1min":3.5329999999999999,"5min":8.0269999999999992,"15min":34.408000000000001},"last":1.788}]},{"osd":10,"last update":"Fri Feb 9 18:01:24 2024","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.829,"15min":1.728},"min":{"1min":0.37,"5min":0.37,"15min":0.34399999999999997},"max":{"1min":5.8920000000000003,"5min":9.8460000000000001,"15min":21.364999999999998},"last":0.59599999999999997},{"interface":"front","average":{"1min":1.298,"5min":1.861,"15min":1.8440000000000001},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.40100000000000002},"max":{"1min":5.9189999999999996,"5min":9.7989999999999995,"15min":21.408999999999999},"last":0.621}]},{"osd":11,"last update":"Fri Feb 9 18:01:48 2024","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.73,"15min":1.766},"min":{"1min":0.48899999999999999,"5min":0.48399999999999999,"15min":0.372},"max":{"1min":4.0979999999999999,"5min":6.9809999999999999,"15min":35.912999999999997},"last":0.83599999999999997},{"interface":"front","average":{"1min":1.278,"5min":1.7689999999999999,"15min":1.827},"min":{"1min":0.53500000000000003,"5min":0.51900000000000002,"15min":0.39700000000000002},"max":{"1min":4.1779999999999999,"5min":7.0099999999999998,"15min":36.156999999999996},"last":0.85799999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575376,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5004124,"kb_used_data":3881620,"kb_used_omap":1,"kb_used_meta":1122366,"kb_avail":88741028,"statfs":{"total":95995035648,"available":90870812672,"internally_reserved":0,"allocated":3974778880,"data_stored":3974490049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1149303237},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 18:01:03 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":1.264,"15min":1.6080000000000001},"min":{"1min":0.28100000000000003,"5min":0.246,"15min":0.19},"max":{"1min":1.6439999999999999,"5min":13.068,"15min":14.885999999999999},"last":1.361},{"interface":"front","average":{"1min":0.92900000000000005,"5min":1.2789999999999999,"15min":1.8500000000000001},"min":{"1min":0.32200000000000001,"5min":0.27400000000000002,"15min":0.214},"max":{"1min":4.2839999999999998,"5min":7.5780000000000003,"15min":33.389000000000003},"last":1.4650000000000001}]},{"osd":2,"last update":"Fri Feb 9 18:01:25 2024","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.3859999999999999,"15min":1.7749999999999999},"min":{"1min":0.35799999999999998,"5min":0.28799999999999998,"15min":0.222},"max":{"1min":6.5060000000000002,"5min":10.510999999999999,"15min":14.83},"last":1.4319999999999999},{"interface":"front","average":{"1min":1.3260000000000001,"5min":1.234,"15min":1.5760000000000001},"min":{"1min":0.38700000000000001,"5min":0.28100000000000003,"15min":0.222},"max":{"1min":3.9060000000000001,"5min":7.6070000000000002,"15min":11.103},"last":2.6469999999999998}]},{"osd":3,"last update":"Fri Feb 9 18:01:15 2024","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.71799999999999997,"15min":0.69399999999999995},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":2.9630000000000001,"5min":2.9630000000000001,"15min":2.9630000000000001},"last":1.637},{"interface":"front","average":{"1min":0.89200000000000002,"5min":1.1279999999999999,"15min":1.1679999999999999},"min":{"1min":0.32500000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":3.1539999999999999,"5min":4.4279999999999999,"15min":4.4279999999999999},"last":5.3940000000000001}]},{"osd":4,"last update":"Fri Feb 9 18:01:10 2024","interfaces":[{"interface":"back","average":{"1min":1.0600000000000001,"5min":1.0429999999999999,"15min":1.3859999999999999},"min":{"1min":0.42599999999999999,"5min":0.27600000000000002,"15min":0.26900000000000002},"max":{"1min":3.2810000000000001,"5min":5.4249999999999998,"15min":45.137999999999998},"last":1.48},{"interface":"front","average":{"1min":0.93300000000000005,"5min":1.1180000000000001,"15min":1.5600000000000001},"min":{"1min":0.42499999999999999,"5min":0.27400000000000002,"15min":0.26400000000000001},"max":{"1min":3.0390000000000001,"5min":7.7370000000000001,"15min":44.950000000000003},"last":1.556}]},{"osd":5,"last update":"Fri Feb 9 18:01:37 2024","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.0580000000000001,"15min":1.448},"min":{"1min":0.34499999999999997,"5min":0.29399999999999998,"15min":0.26000000000000001},"max":{"1min":5.1219999999999999,"5min":7.7839999999999998,"15min":15.106999999999999},"last":4.1159999999999997},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.077,"15min":1.7989999999999999},"min":{"1min":0.34100000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":5.1500000000000004,"5min":5.665,"15min":33.247},"last":1.4510000000000001}]},{"osd":6,"last update":"Fri Feb 9 18:01:54 2024","interfaces":[{"interface":"back","average":{"1min":1.5980000000000001,"5min":1.2450000000000001,"15min":1.77},"min":{"1min":0.375,"5min":0.33100000000000002,"15min":0.24299999999999999},"max":{"1min":5.4610000000000003,"5min":7.8380000000000001,"15min":33.338000000000001},"last":4.2130000000000001},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.2350000000000001,"15min":1.8129999999999999},"min":{"1min":0.39000000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":5.4119999999999999,"5min":7.9199999999999999,"15min":33.286999999999999},"last":4.2670000000000003}]},{"osd":7,"last update":"Fri Feb 9 18:01:55 2024","interfaces":[{"interface":"back","average":{"1min":2.02,"5min":1.4119999999999999,"15min":1.8799999999999999},"min":{"1min":0.40899999999999997,"5min":0.38400000000000001,"15min":0.316},"max":{"1min":6.298,"5min":7.8689999999999998,"15min":33.314999999999998},"last":4.1929999999999996},{"interface":"front","average":{"1min":1.8120000000000001,"5min":1.413,"15min":1.907},"min":{"1min":0.52800000000000002,"5min":0.432,"15min":0.40200000000000002},"max":{"1min":5.4539999999999997,"5min":7.8940000000000001,"15min":33.363999999999997},"last":1.302}]},{"osd":8,"last update":"Fri Feb 9 18:01:03 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.9390000000000001,"15min":1.7370000000000001},"min":{"1min":0.48799999999999999,"5min":0.46600000000000003,"15min":0.35599999999999998},"max":{"1min":3.2519999999999998,"5min":10.423,"15min":10.423},"last":4.1630000000000003},{"interface":"front","average":{"1min":1.258,"5min":2.21,"15min":1.982},"min":{"1min":0.52400000000000002,"5min":0.36399999999999999,"15min":0.35599999999999998},"max":{"1min":4.923,"5min":10.663,"15min":12.69},"last":1.3360000000000001}]},{"osd":9,"last update":"Fri Feb 9 18:01:03 2024","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":2.2090000000000001,"15min":1.968},"min":{"1min":0.503,"5min":0.503,"15min":0.39300000000000002},"max":{"1min":3.302,"5min":10.475,"15min":20.789000000000001},"last":1.387},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.8240000000000001,"15min":1.855},"min":{"1min":0.5,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":3.3820000000000001,"5min":7.2110000000000003,"15min":9.5239999999999991},"last":1.409}]},{"osd":10,"last update":"Fri Feb 9 18:01:37 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":2.1120000000000001,"15min":2.214},"min":{"1min":0.57899999999999996,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":5.1630000000000003,"5min":12.151,"15min":33.213999999999999},"last":4.25},{"interface":"front","average":{"1min":1.6879999999999999,"5min":1.958,"15min":2.0779999999999998},"min":{"1min":0.47999999999999998,"5min":0.372,"15min":0.372},"max":{"1min":5.2439999999999998,"5min":9.1859999999999999,"15min":15.271000000000001},"last":1.51}]},{"osd":11,"last update":"Fri Feb 9 18:01:54 2024","interfaces":[{"interface":"back","average":{"1min":2.2650000000000001,"5min":2.6349999999999998,"15min":2.5249999999999999},"min":{"1min":0.51000000000000001,"5min":0.378,"15min":0.378},"max":{"1min":7.6260000000000003,"5min":16.881,"15min":19.648},"last":1.5329999999999999},{"interface":"front","average":{"1min":1.552,"5min":2.3399999999999999,"15min":2.5129999999999999},"min":{"1min":0.54500000000000004,"5min":0.372,"15min":0.372},"max":{"1min":5.8049999999999997,"5min":15.209,"15min":33.161000000000001},"last":4.2309999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738901,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1363288,"kb_used_data":497744,"kb_used_omap":1,"kb_used_meta":865534,"kb_avail":92381864,"statfs":{"total":95995035648,"available":94599028736,"internally_reserved":0,"allocated":509689856,"data_stored":509405097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":886307269},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 9 18:01:45 2024","interfaces":[{"interface":"back","average":{"1min":2.9009999999999998,"5min":1.681,"15min":1.7030000000000001},"min":{"1min":0.317,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":23.282,"5min":23.282,"15min":23.282},"last":1.0580000000000001},{"interface":"front","average":{"1min":2.9590000000000001,"5min":1.754,"15min":1.7749999999999999},"min":{"1min":0.39500000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":23.32,"5min":23.32,"15min":23.32},"last":1.244}]},{"osd":2,"last update":"Fri Feb 9 18:01:16 2024","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.3120000000000001,"15min":1.681},"min":{"1min":0.379,"5min":0.253,"15min":0.245},"max":{"1min":8.1799999999999997,"5min":13.457000000000001,"15min":20.013999999999999},"last":1.2150000000000001},{"interface":"front","average":{"1min":1.417,"5min":1.29,"15min":1.6040000000000001},"min":{"1min":0.39900000000000002,"5min":0.219,"15min":0.219},"max":{"1min":4.9379999999999997,"5min":16.085000000000001,"15min":20.103999999999999},"last":0.31}]},{"osd":3,"last update":"Fri Feb 9 18:01:16 2024","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.248,"15min":1.236},"min":{"1min":0.35699999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":4.8460000000000001,"5min":4.8460000000000001,"15min":4.8460000000000001},"last":2.2000000000000002},{"interface":"front","average":{"1min":0.997,"5min":1.71,"15min":1.829},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":3.4540000000000002,"5min":11.714,"15min":11.714},"last":1.833}]},{"osd":4,"last update":"Fri Feb 9 18:01:16 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.94099999999999995,"15min":1.3280000000000001},"min":{"1min":0.307,"5min":0.29999999999999999,"15min":0.28000000000000003},"max":{"1min":1.478,"5min":9.9760000000000009,"15min":12.757},"last":1.1539999999999999},{"interface":"front","average":{"1min":0.76300000000000001,"5min":1.3700000000000001,"15min":1.7490000000000001},"min":{"1min":0.39100000000000001,"5min":0.25800000000000001,"15min":0.249},"max":{"1min":2.1659999999999999,"5min":11.635,"15min":17.091999999999999},"last":1.75}]},{"osd":5,"last update":"Fri Feb 9 18:01:28 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.056,"15min":1.605},"min":{"1min":0.36699999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":5.415,"5min":10.031000000000001,"15min":48.924999999999997},"last":1.2789999999999999},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.1339999999999999,"15min":1.6639999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.35299999999999998},"max":{"1min":5.4710000000000001,"5min":10.06,"15min":48.969999999999999},"last":1.2969999999999999}]},{"osd":6,"last update":"Fri Feb 9 18:01:57 2024","interfaces":[{"interface":"back","average":{"1min":2.613,"5min":1.2509999999999999,"15min":1.5860000000000001},"min":{"1min":0.36199999999999999,"5min":0.35399999999999998,"15min":0.307},"max":{"1min":8.702,"5min":8.702,"15min":15.667},"last":1.9430000000000001},{"interface":"front","average":{"1min":1.411,"5min":1.1140000000000001,"15min":1.6100000000000001},"min":{"1min":0.38,"5min":0.36399999999999999,"15min":0.34799999999999998},"max":{"1min":6.3380000000000001,"5min":6.3380000000000001,"15min":15.702999999999999},"last":1.7809999999999999}]},{"osd":7,"last update":"Fri Feb 9 18:01:28 2024","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.343,"15min":1.6459999999999999},"min":{"1min":0.48899999999999999,"5min":0.307,"15min":0.29199999999999998},"max":{"1min":2.7959999999999998,"5min":11.682,"15min":15.271000000000001},"last":1.712},{"interface":"front","average":{"1min":1.306,"5min":1.4810000000000001,"15min":1.786},"min":{"1min":0.49399999999999999,"5min":0.35899999999999999,"15min":0.28299999999999997},"max":{"1min":4.5880000000000001,"5min":11.773999999999999,"15min":17.152999999999999},"last":1.8069999999999999}]},{"osd":8,"last update":"Fri Feb 9 18:01:28 2024","interfaces":[{"interface":"back","average":{"1min":1.657,"5min":1.7809999999999999,"15min":1.9430000000000001},"min":{"1min":0.45600000000000002,"5min":0.33800000000000002,"15min":0.313},"max":{"1min":4.9109999999999996,"5min":11.227,"15min":17.189},"last":1.992},{"interface":"front","average":{"1min":1.256,"5min":2.1059999999999999,"15min":1.911},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.36499999999999999},"max":{"1min":3.5910000000000002,"5min":18.600000000000001,"15min":18.600000000000001},"last":2.7050000000000001}]},{"osd":9,"last update":"Fri Feb 9 18:01:16 2024","interfaces":[{"interface":"back","average":{"1min":1.4370000000000001,"5min":2.6539999999999999,"15min":2.3620000000000001},"min":{"1min":0.51700000000000002,"5min":0.40999999999999998,"15min":0.39000000000000001},"max":{"1min":7.29,"5min":11.801,"15min":24.757000000000001},"last":1.909},{"interface":"front","average":{"1min":1.6339999999999999,"5min":2.1219999999999999,"15min":2.1509999999999998},"min":{"1min":0.60399999999999998,"5min":0.41599999999999998,"15min":0.40899999999999997},"max":{"1min":7.1689999999999996,"5min":12.759,"15min":24.681999999999999},"last":1.8839999999999999}]},{"osd":10,"last update":"Fri Feb 9 18:01:28 2024","interfaces":[{"interface":"back","average":{"1min":1.78,"5min":1.831,"15min":2.1949999999999998},"min":{"1min":0.49399999999999999,"5min":0.438,"15min":0.41999999999999998},"max":{"1min":4.8789999999999996,"5min":7.8029999999999999,"15min":57.979999999999997},"last":1.9690000000000001},{"interface":"front","average":{"1min":1.9119999999999999,"5min":2.4940000000000002,"15min":2.4489999999999998},"min":{"1min":0.63700000000000001,"5min":0.42099999999999999,"15min":0.40400000000000003},"max":{"1min":9.202,"5min":12.689,"15min":57.606999999999999},"last":1.8580000000000001}]},{"osd":11,"last update":"Fri Feb 9 18:01:57 2024","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":1.9770000000000001,"15min":2.2959999999999998},"min":{"1min":0.49199999999999999,"5min":0.44800000000000001,"15min":0.437},"max":{"1min":7.6609999999999996,"5min":15.698,"15min":58.351999999999997},"last":1.3300000000000001},{"interface":"front","average":{"1min":1.427,"5min":2.028,"15min":2.3380000000000001},"min":{"1min":0.54800000000000004,"5min":0.51800000000000002,"15min":0.441},"max":{"1min":7.7370000000000001,"5min":15.666,"15min":57.314},"last":1.3600000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411850,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":663784,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":93081368,"statfs":{"total":95995035648,"available":95315320832,"internally_reserved":0,"allocated":942080,"data_stored":653769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:58:48 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.53200000000000003,"15min":0.48699999999999999},"min":{"1min":0.23699999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.9619999999999997,"5min":4.9619999999999997,"15min":4.9619999999999997},"last":0.33800000000000002},{"interface":"front","average":{"1min":1.292,"5min":0.81899999999999995,"15min":0.73999999999999999},"min":{"1min":0.33000000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":4.5430000000000001,"5min":4.5430000000000001,"15min":4.5430000000000001},"last":0.47799999999999998}]},{"osd":1,"last update":"Fri Feb 9 18:01:26 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.96499999999999997,"15min":1.9590000000000001},"min":{"1min":0.255,"5min":0.23300000000000001,"15min":0.20799999999999999},"max":{"1min":2.0640000000000001,"5min":7.1600000000000001,"15min":52.646000000000001},"last":3.2639999999999998},{"interface":"front","average":{"1min":0.67500000000000004,"5min":1.042,"15min":1.9610000000000001},"min":{"1min":0.32900000000000001,"5min":0.30099999999999999,"15min":0.248},"max":{"1min":1.9590000000000001,"5min":7.1950000000000003,"15min":52.683999999999997},"last":3.0249999999999999}]},{"osd":3,"last update":"Fri Feb 9 18:01:15 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":2.7930000000000001,"15min":3.181},"min":{"1min":0.26700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":1.266,"5min":29.228000000000002,"15min":29.228000000000002},"last":3.0659999999999998},{"interface":"front","average":{"1min":0.82399999999999995,"5min":1.4039999999999999,"15min":1.5},"min":{"1min":0.26500000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":3.044,"5min":10.492000000000001,"15min":10.492000000000001},"last":5.5839999999999996}]},{"osd":4,"last update":"Fri Feb 9 18:01:56 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.73299999999999998,"15min":1.2849999999999999},"min":{"1min":0.35499999999999998,"5min":0.33600000000000002,"15min":0.245},"max":{"1min":4.633,"5min":4.633,"15min":12.398},"last":3.1509999999999998},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.73999999999999999,"15min":1.321},"min":{"1min":0.42399999999999999,"5min":0.38200000000000001,"15min":0.25700000000000001},"max":{"1min":4.7000000000000002,"5min":4.7000000000000002,"15min":10.888999999999999},"last":3.3039999999999998}]},{"osd":5,"last update":"Fri Feb 9 18:01:48 2024","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":0.83999999999999997,"15min":1.6719999999999999},"min":{"1min":0.34599999999999997,"5min":0.28100000000000003,"15min":0.245},"max":{"1min":5.0049999999999999,"5min":5.0049999999999999,"15min":35.436999999999998},"last":0.51800000000000002},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.1299999999999999,"15min":1.7150000000000001},"min":{"1min":0.32100000000000001,"5min":0.27000000000000002,"15min":0.26900000000000002},"max":{"1min":5.4459999999999997,"5min":11.414,"15min":35.484000000000002},"last":1.387}]},{"osd":6,"last update":"Fri Feb 9 18:01:56 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":1.1279999999999999,"15min":1.6890000000000001},"min":{"1min":0.42399999999999999,"5min":0.32300000000000001,"15min":0.28399999999999997},"max":{"1min":2.6819999999999999,"5min":11.465999999999999,"15min":18.518999999999998},"last":3.2509999999999999},{"interface":"front","average":{"1min":0.72699999999999998,"5min":1.1899999999999999,"15min":1.6819999999999999},"min":{"1min":0.38500000000000001,"5min":0.317,"15min":0.28199999999999997},"max":{"1min":1.9430000000000001,"5min":11.598000000000001,"15min":18.481999999999999},"last":3.2090000000000001}]},{"osd":7,"last update":"Fri Feb 9 18:01:26 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.91700000000000004,"15min":1.5509999999999999},"min":{"1min":0.40000000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.897,"5min":7.1109999999999998,"15min":16.666},"last":3.7770000000000001},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.3640000000000001,"15min":1.7250000000000001},"min":{"1min":0.36599999999999999,"5min":0.309,"15min":0.309},"max":{"1min":3.9199999999999999,"5min":11.574,"15min":16.638999999999999},"last":3.5419999999999998}]},{"osd":8,"last update":"Fri Feb 9 18:01:56 2024","interfaces":[{"interface":"back","average":{"1min":1.5549999999999999,"5min":1.4410000000000001,"15min":1.8660000000000001},"min":{"1min":0.47799999999999998,"5min":0.39600000000000002,"15min":0.35499999999999998},"max":{"1min":5.2430000000000003,"5min":10.095000000000001,"15min":12.500999999999999},"last":3.1160000000000001},{"interface":"front","average":{"1min":1.355,"5min":1.73,"15min":2.0489999999999999},"min":{"1min":0.54900000000000004,"5min":0.378,"15min":0.35699999999999998},"max":{"1min":5.2779999999999996,"5min":11.492000000000001,"15min":54.759999999999998},"last":0.55900000000000005}]},{"osd":9,"last update":"Fri Feb 9 18:01:58 2024","interfaces":[{"interface":"back","average":{"1min":1.4850000000000001,"5min":1.5900000000000001,"15min":2.23},"min":{"1min":0.36499999999999999,"5min":0.35799999999999998,"15min":0.318},"max":{"1min":5.4039999999999999,"5min":9.7669999999999995,"15min":53.935000000000002},"last":1.421},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.7669999999999999,"15min":2.1970000000000001},"min":{"1min":0.52600000000000002,"5min":0.36399999999999999,"15min":0.34899999999999998},"max":{"1min":2.8399999999999999,"5min":11.522,"15min":54.670999999999999},"last":0.58599999999999997}]},{"osd":10,"last update":"Fri Feb 9 18:01:35 2024","interfaces":[{"interface":"back","average":{"1min":1.968,"5min":1.8999999999999999,"15min":2.04},"min":{"1min":0.5,"5min":0.48799999999999999,"15min":0.33500000000000002},"max":{"1min":7.266,"5min":8.5570000000000004,"15min":54.039999999999999},"last":3.181},{"interface":"front","average":{"1min":2.0049999999999999,"5min":1.929,"15min":2.0720000000000001},"min":{"1min":0.52200000000000002,"5min":0.48799999999999999,"15min":0.42599999999999999},"max":{"1min":7.319,"5min":8.6150000000000002,"15min":54.118000000000002},"last":3.3359999999999999}]},{"osd":11,"last update":"Fri Feb 9 18:01:56 2024","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":2.0790000000000002,"15min":2.3079999999999998},"min":{"1min":0.56200000000000006,"5min":0.44,"15min":0.41999999999999998},"max":{"1min":5.032,"5min":11.548,"15min":54.799999999999997},"last":0.61499999999999999},{"interface":"front","average":{"1min":1.6419999999999999,"5min":1.792,"15min":2.3260000000000001},"min":{"1min":0.46000000000000002,"5min":0.40600000000000003,"15min":0.35499999999999998},"max":{"1min":7.335,"5min":8.8770000000000007,"15min":54.103000000000002},"last":6.1070000000000002}]}]},{"osd":3,"up_from":854,"seq":3667902070821,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3360792,"kb_used_data":3172112,"kb_used_omap":202,"kb_used_meta":188469,"kb_avail":90384360,"statfs":{"total":95995035648,"available":92553584640,"internally_reserved":0,"allocated":3248242688,"data_stored":3247957929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207633,"internal_metadata":192992495},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 18:01:16 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.93300000000000005,"15min":0.95399999999999996},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.7969999999999997,"5min":6.9080000000000004,"15min":6.9080000000000004},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.501,"5min":1.5860000000000001,"15min":1.7669999999999999},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.611,"5min":17.077999999999999,"15min":17.077999999999999},"last":2.738}]},{"osd":1,"last update":"Fri Feb 9 18:01:07 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":2.3010000000000002,"5min":2.3010000000000002,"15min":2.3010000000000002},"last":1.8520000000000001},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":1.8300000000000001,"5min":1.8300000000000001,"15min":1.8300000000000001},"last":0.41499999999999998}]},{"osd":2,"last update":"Fri Feb 9 18:01:16 2024","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":2.177,"15min":2.3359999999999999},"min":{"1min":0.317,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":3.5129999999999999,"5min":11.712,"15min":11.712},"last":4.6630000000000003},{"interface":"front","average":{"1min":1.002,"5min":1.1699999999999999,"15min":1.198},"min":{"1min":0.38800000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":2.2810000000000001,"5min":6.9539999999999997,"15min":6.9539999999999997},"last":0.80900000000000005}]},{"osd":4,"last update":"Fri Feb 9 18:01:16 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.89600000000000002,"15min":0.90600000000000003},"min":{"1min":0.38400000000000001,"5min":0.317,"15min":0.317},"max":{"1min":5.3760000000000003,"5min":5.9100000000000001,"15min":5.9100000000000001},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.79100000000000004,"5min":1.3420000000000001,"15min":1.4339999999999999},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":2.1349999999999998,"5min":10.755000000000001,"15min":10.755000000000001},"last":4.4119999999999999}]},{"osd":5,"last update":"Fri Feb 9 18:01:16 2024","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":1.0429999999999999,"15min":1.0640000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":5.3280000000000003,"5min":5.9539999999999997,"15min":5.9539999999999997},"last":0.501},{"interface":"front","average":{"1min":0.76200000000000001,"5min":1.494,"15min":1.6160000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":2.1549999999999998,"5min":10.707000000000001,"15min":10.707000000000001},"last":4.3499999999999996}]},{"osd":6,"last update":"Fri Feb 9 18:01:16 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":1.8,"15min":1.952},"min":{"1min":0.373,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":2.6259999999999999,"5min":10.808,"15min":10.808},"last":4.3869999999999996},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.69599999999999995,"15min":0.69399999999999995},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.962,"5min":2.766,"15min":2.766},"last":0.73699999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70299999999999996}]},{"osd":8,"last update":"Fri Feb 9 18:01:16 2024","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.3680000000000001,"15min":1.3640000000000001},"min":{"1min":0.52300000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":5.4900000000000002,"5min":5.9820000000000002,"15min":5.9820000000000002},"last":0.47599999999999998},{"interface":"front","average":{"1min":1.113,"5min":1.772,"15min":1.8819999999999999},"min":{"1min":0.434,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":4.351,"5min":10.782,"15min":10.782},"last":5.7670000000000003}]},{"osd":9,"last update":"Fri Feb 9 18:01:16 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.70899999999999996,"15min":0.67100000000000004},"min":{"1min":0.502,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":2.2469999999999999,"5min":2.2469999999999999,"15min":2.2469999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":1.48,"5min":1.373,"15min":1.355},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":5.4480000000000004,"5min":6.008,"15min":6.008},"last":0.45100000000000001}]},{"osd":10,"last update":"Fri Feb 9 18:01:16 2024","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.839,"15min":1.9670000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":2.242,"5min":10.885999999999999,"15min":10.885999999999999},"last":4.4420000000000002},{"interface":"front","average":{"1min":1.1719999999999999,"5min":1.3540000000000001,"15min":1.3839999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":2.883,"5min":8.8970000000000002,"15min":8.8970000000000002},"last":4.4630000000000001}]},{"osd":11,"last update":"Fri Feb 9 18:01:16 2024","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":2.1339999999999999,"15min":2.278},"min":{"1min":0.56399999999999995,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":5.524,"5min":17.126000000000001,"15min":17.126000000000001},"last":9.4390000000000001},{"interface":"front","average":{"1min":1.087,"5min":0.96099999999999997,"15min":0.93999999999999995},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":3.3740000000000001,"5min":3.4940000000000002,"15min":3.4940000000000002},"last":0.78500000000000003}]}]},{"osd":4,"up_from":0,"seq":137438953987,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":40,"seq":171798692349,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5322852,"kb_used_data":4421212,"kb_used_omap":1,"kb_used_meta":901630,"kb_avail":88422300,"statfs":{"total":95995035648,"available":90544435200,"internally_reserved":0,"allocated":4527321088,"data_stored":4527028705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":923269573},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:58:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.82599999999999996,"15min":0.874},"min":{"1min":0.27800000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":1.341,"5min":5.399,"15min":5.399},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.622,"5min":0.92100000000000004,"15min":0.97099999999999997},"min":{"1min":0.40400000000000003,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.409,"5min":5.4329999999999998,"15min":5.4329999999999998},"last":0.52100000000000002}]},{"osd":1,"last update":"Fri Feb 9 18:01:23 2024","interfaces":[{"interface":"back","average":{"1min":1.9990000000000001,"5min":0.96299999999999997,"15min":1.3899999999999999},"min":{"1min":0.317,"5min":0.27200000000000002,"15min":0.254},"max":{"1min":16.271000000000001,"5min":16.271000000000001,"15min":19.079999999999998},"last":2.3759999999999999},{"interface":"front","average":{"1min":1.861,"5min":0.91500000000000004,"15min":1.2030000000000001},"min":{"1min":0.29399999999999998,"5min":0.26900000000000002,"15min":0.23499999999999999},"max":{"1min":16.344999999999999,"5min":16.344999999999999,"15min":19.065999999999999},"last":2.7610000000000001}]},{"osd":2,"last update":"Fri Feb 9 18:02:02 2024","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.68000000000000005,"15min":1.3460000000000001},"min":{"1min":0.27500000000000002,"5min":0.27400000000000002,"15min":0.254},"max":{"1min":6.0069999999999997,"5min":6.0069999999999997,"15min":9.673},"last":0.27500000000000002},{"interface":"front","average":{"1min":1.478,"5min":0.99299999999999999,"15min":1.5309999999999999},"min":{"1min":0.315,"5min":0.25800000000000001,"15min":0.24399999999999999},"max":{"1min":4.0679999999999996,"5min":4.3289999999999997,"15min":14.978},"last":2.3330000000000002}]},{"osd":3,"last update":"Fri Feb 9 18:01:18 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.68799999999999994,"15min":0.71599999999999997},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.5029999999999999,"5min":2.625,"15min":2.625},"last":0.98899999999999999},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.92200000000000004,"15min":0.96399999999999997},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.8120000000000001,"5min":4.1379999999999999,"15min":4.1379999999999999},"last":2.085}]},{"osd":4,"last update":"Fri Feb 9 18:01:23 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.56299999999999994,"15min":0.95799999999999996},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":3.383,"5min":3.383,"15min":28.481999999999999},"last":2.5019999999999998},{"interface":"front","average":{"1min":1.079,"5min":0.74199999999999999,"15min":1.1579999999999999},"min":{"1min":0.35999999999999999,"5min":0.35199999999999998,"15min":0.22500000000000001},"max":{"1min":6.1239999999999997,"5min":6.1239999999999997,"15min":15.843},"last":2.4140000000000001}]},{"osd":6,"last update":"Fri Feb 9 18:02:02 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.63800000000000001,"15min":0.97799999999999998},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.891,"5min":2.8700000000000001,"15min":15.026},"last":0.38},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.66100000000000003,"15min":1.129},"min":{"1min":0.36899999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.8680000000000001,"5min":2.278,"15min":20.192},"last":0.45400000000000001}]},{"osd":7,"last update":"Fri Feb 9 18:01:34 2024","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":0.75700000000000001,"15min":1.2250000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":4.6130000000000004,"5min":4.6130000000000004,"15min":18.134},"last":2.431},{"interface":"front","average":{"1min":1.052,"5min":0.749,"15min":1.3340000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.31},"max":{"1min":3.504,"5min":3.504,"15min":19.300000000000001},"last":2.4470000000000001}]},{"osd":8,"last update":"Fri Feb 9 18:01:48 2024","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.8600000000000001,"15min":1.669},"min":{"1min":0.377,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":3.4780000000000002,"5min":12.227,"15min":12.227},"last":2.395},{"interface":"front","average":{"1min":1.141,"5min":1.2709999999999999,"15min":1.5169999999999999},"min":{"1min":0.39600000000000002,"5min":0.372,"15min":0.372},"max":{"1min":4.6200000000000001,"5min":8.3770000000000007,"15min":8.3770000000000007},"last":0.47199999999999998}]},{"osd":9,"last update":"Fri Feb 9 18:02:02 2024","interfaces":[{"interface":"back","average":{"1min":1.655,"5min":1.819,"15min":1.677},"min":{"1min":0.39500000000000002,"5min":0.315,"15min":0.315},"max":{"1min":6.4729999999999999,"5min":11.167999999999999,"15min":15.124000000000001},"last":0.55400000000000005},{"interface":"front","average":{"1min":1.835,"5min":1.627,"15min":1.571},"min":{"1min":0.38,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":6.4589999999999996,"5min":13.462999999999999,"15min":13.462999999999999},"last":0.63}]},{"osd":10,"last update":"Fri Feb 9 18:01:23 2024","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":1.3700000000000001,"15min":1.752},"min":{"1min":0.47699999999999998,"5min":0.46700000000000003,"15min":0.373},"max":{"1min":2.2280000000000002,"5min":11.506,"15min":80.042000000000002},"last":0.50700000000000001},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.663,"15min":1.875},"min":{"1min":0.52800000000000002,"5min":0.39900000000000002,"15min":0.38200000000000001},"max":{"1min":6.2869999999999999,"5min":11.638,"15min":79.807000000000002},"last":0.56599999999999995}]},{"osd":11,"last update":"Fri Feb 9 18:01:48 2024","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":1.6399999999999999,"15min":1.6890000000000001},"min":{"1min":0.495,"5min":0.434,"15min":0.39400000000000002},"max":{"1min":2.3610000000000002,"5min":6.9240000000000004,"15min":9.7680000000000007},"last":0.58299999999999996},{"interface":"front","average":{"1min":1.173,"5min":1.569,"15min":1.7210000000000001},"min":{"1min":0.52400000000000002,"5min":0.48699999999999999,"15min":0.42199999999999999},"max":{"1min":3.2610000000000001,"5min":7.2709999999999999,"15min":20.981000000000002},"last":0.59799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":507314176,"data_stored":507314176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3969323008,"data_stored":3969318936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2975006720,"data_stored":2975006720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4507635712,"data_stored":4507631640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":984809472,"data_stored":984809472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2474840064,"data_stored":2474835992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1019084800,"data_stored":1019084800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":3052273664,"data_stored":3052273664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2516189184,"data_stored":2516189184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1985019904,"data_stored":1985019904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-09T18:02:11.902 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph pg dump --format=json 2024-02-09T18:02:11.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:11.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:02:12.004 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:02:12.046 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:12.081 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:02:12.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:12.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:02:12.168 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:12 smithi099 ceph-mon[39420]: pgmap v2288: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 7.2 GiB data, 30 GiB used, 953 GiB / 983 GiB avail; 0 B/s wr, 826 op/s 2024-02-09T18:02:12.189 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:02:12.235 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:12.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:12 smithi179 ceph-mon[43975]: pgmap v2288: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 7.2 GiB data, 30 GiB used, 953 GiB / 983 GiB avail; 0 B/s wr, 826 op/s 2024-02-09T18:02:12.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:12 smithi162 ceph-mon[44304]: pgmap v2288: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 7.2 GiB data, 30 GiB used, 953 GiB / 983 GiB avail; 0 B/s wr, 826 op/s 2024-02-09T18:02:12.523 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:02:12.529 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:12.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:02:12.630 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:02:12.695 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:12.911 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:02:14.185 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:02:14.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:14.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:02:14.290 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T18:02:14.333 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:14.376 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T18:02:14.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:14.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T18:02:14.484 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:02:14.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:14 smithi099 ceph-mon[39420]: pgmap v2289: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 7.2 GiB data, 30 GiB used, 953 GiB / 983 GiB avail; 0 B/s wr, 777 op/s 2024-02-09T18:02:14.517 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:14.551 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:02:14.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:14.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:02:14.659 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:02:14.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:14 smithi179 ceph-mon[43975]: pgmap v2289: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 7.2 GiB data, 30 GiB used, 953 GiB / 983 GiB avail; 0 B/s wr, 777 op/s 2024-02-09T18:02:14.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:14 smithi162 ceph-mon[44304]: pgmap v2289: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 7.2 GiB data, 30 GiB used, 953 GiB / 983 GiB avail; 0 B/s wr, 777 op/s 2024-02-09T18:02:14.711 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:14.746 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:02:14.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:14.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:02:14.855 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:02:14.890 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:15.654 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:02:15.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:15.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:02:15.762 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:02:15.790 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:15.958 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:02:15.995 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:02:16.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:16.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:02:16.101 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:02:16.152 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:16.238 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:02:16.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:16.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:02:16.344 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:02:16.386 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:16.468 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:02:16.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:16.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:02:16.572 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T18:02:16.620 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:16.625 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:16 smithi179 ceph-mon[43975]: pgmap v2290: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 7.1 GiB data, 30 GiB used, 954 GiB / 983 GiB avail; 0 B/s wr, 742 op/s 2024-02-09T18:02:16.656 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T18:02:16.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:16.666 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T18:02:16.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:16 smithi162 ceph-mon[44304]: pgmap v2290: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 7.1 GiB data, 30 GiB used, 954 GiB / 983 GiB avail; 0 B/s wr, 742 op/s 2024-02-09T18:02:16.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:16 smithi099 ceph-mon[39420]: pgmap v2290: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 7.1 GiB data, 30 GiB used, 954 GiB / 983 GiB avail; 0 B/s wr, 742 op/s 2024-02-09T18:02:16.766 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:02:16.800 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:16.825 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:02:16.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:16.829 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:02:16.930 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T18:02:16.968 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:16.992 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T18:02:16.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:16.996 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T18:02:17.097 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:02:17.130 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:17.163 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:02:17.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:17.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:02:17.268 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:02:17.315 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:17.341 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:02:17.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:17.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:02:17.449 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:02:17.496 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:17.528 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:02:17.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:17.531 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:02:17.632 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:02:17.641 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T18:02:17.675 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T18:02:17.682 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:17.713 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T18:02:17.717 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:02:17.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:17.725 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:02:17.756 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T18:02:17.811 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T18:02:17.824 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T18:02:17.826 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:02:17.866 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T18:02:17.879 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:17.898 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T18:02:17.937 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T18:02:18.161 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T18:02:18.186 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T18:02:18.230 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T18:02:18.236 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T18:02:18.236 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T18:02:18.236 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T18:02:18.236 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T18:02:18.236 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T18:02:18.236 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T18:02:18.236 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T18:02:18.237 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T18:02:18.237 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T18:02:18.237 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T18:02:18.237 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T18:02:18.237 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T18:02:18.270 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is set 2024-02-09T18:02:18.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:18 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2378762349' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T18:02:18.426 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:18 smithi179 ceph-mon[43975]: pgmap v2291: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 7.0 GiB data, 29 GiB used, 954 GiB / 983 GiB avail; 0 B/s wr, 819 op/s 2024-02-09T18:02:18.426 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:18 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/56217199' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:02:18.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:18 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2378762349' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T18:02:18.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:18 smithi162 ceph-mon[44304]: pgmap v2291: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 7.0 GiB data, 29 GiB used, 954 GiB / 983 GiB avail; 0 B/s wr, 819 op/s 2024-02-09T18:02:18.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:18 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/56217199' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:02:18.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:18 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2378762349' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T18:02:18.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:18 smithi099 ceph-mon[39420]: pgmap v2291: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 7.0 GiB data, 29 GiB used, 954 GiB / 983 GiB avail; 0 B/s wr, 819 op/s 2024-02-09T18:02:18.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:18 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/56217199' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:02:19.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:19 smithi179 ceph-mon[43975]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:02:19.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:19 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2378762349' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T18:02:19.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:19 smithi179 ceph-mon[43975]: osdmap e926: 12 total, 12 up, 11 in 2024-02-09T18:02:19.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:19 smithi162 ceph-mon[44304]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:02:19.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:19 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2378762349' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T18:02:19.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:19 smithi162 ceph-mon[44304]: osdmap e926: 12 total, 12 up, 11 in 2024-02-09T18:02:19.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:19 smithi099 ceph-mon[39420]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:02:19.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:19 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2378762349' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T18:02:19.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:19 smithi099 ceph-mon[39420]: osdmap e926: 12 total, 12 up, 11 in 2024-02-09T18:02:20.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:20 smithi179 ceph-mon[43975]: pgmap v2293: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.9 GiB data, 29 GiB used, 954 GiB / 983 GiB avail; 0 B/s wr, 776 op/s 2024-02-09T18:02:20.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:20 smithi162 ceph-mon[44304]: pgmap v2293: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.9 GiB data, 29 GiB used, 954 GiB / 983 GiB avail; 0 B/s wr, 776 op/s 2024-02-09T18:02:20.460 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:20 smithi099 ceph-mon[39420]: pgmap v2293: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.9 GiB data, 29 GiB used, 954 GiB / 983 GiB avail; 0 B/s wr, 776 op/s 2024-02-09T18:02:20.471 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:02:20.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:20.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:02:20.579 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:02:20.619 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:20.646 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:02:20.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:20.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:02:20.755 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T18:02:20.795 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:20.822 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T18:02:20.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:20.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T18:02:20.929 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:02:20.969 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:21.011 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:02:21.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:21.020 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:02:21.121 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:02:21.160 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:21.188 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:02:21.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:21.199 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:02:21.300 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:02:21.334 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:21.356 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:02:21.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:21.359 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:02:21.460 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:02:21.486 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:21.512 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:02:21.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:21.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:02:21.617 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T18:02:21.652 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:21.689 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T18:02:21.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:21.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T18:02:21.797 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:02:21.836 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:21.871 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:02:21.876 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:21.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:02:21.978 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:02:22.013 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:22.044 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:02:22.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:22.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:02:22.148 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T18:02:22.195 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:22.232 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T18:02:22.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:22.239 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T18:02:22.340 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:02:22.357 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:22 smithi162 ceph-mon[44304]: pgmap v2294: 47 pgs: 47 active+clean; 6.8 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 0 B/s wr, 780 op/s 2024-02-09T18:02:22.389 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:22.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:22 smithi099 ceph-mon[39420]: pgmap v2294: 47 pgs: 47 active+clean; 6.8 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 0 B/s wr, 780 op/s 2024-02-09T18:02:22.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:22 smithi179 ceph-mon[43975]: pgmap v2294: 47 pgs: 47 active+clean; 6.8 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 0 B/s wr, 780 op/s 2024-02-09T18:02:22.856 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:02:22.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:22.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:02:22.962 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T18:02:22.976 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T18:02:22.976 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [], 2024-02-09T18:02:22.976 INFO:teuthology.orchestra.run.smithi099.stdout: "num_ops": 0 2024-02-09T18:02:22.976 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T18:02:22.999 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T18:02:23.011 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:23.040 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T18:02:23.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:23.047 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T18:02:23.148 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:02:23.179 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:23.205 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:02:23.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:23.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:02:23.249 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T18:02:23.281 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-02-09T18:02:23.314 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T18:02:23.352 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:23.377 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:23 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T18:02:23.377 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:23 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T18:02:23.377 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:23 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T18:02:23.377 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:23 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T18:02:23.377 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:23 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T18:02:23.381 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T18:02:23.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:23.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T18:02:23.486 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T18:02:23.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:23 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T18:02:23.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:23 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T18:02:23.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:23 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T18:02:23.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:23 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T18:02:23.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:23 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T18:02:23.534 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:23.573 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T18:02:23.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:23.580 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T18:02:23.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:23 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T18:02:23.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:23 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T18:02:23.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:23 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T18:02:23.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:23 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T18:02:23.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:23 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T18:02:23.681 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T18:02:23.742 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:23.780 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-09T18:02:23.799 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T18:02:23.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:23.806 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T18:02:23.892 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset noscrub 2024-02-09T18:02:23.907 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:02:24.019 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:24.058 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:02:24.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:24.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:02:24.170 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:02:24.240 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:24.274 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:02:24.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:24.277 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:02:24.378 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:02:24.386 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:24 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:02:24.386 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:24 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:02:24.386 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:24 smithi099 ceph-mon[39420]: from='client.26756 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T18:02:24.386 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:24 smithi099 ceph-mon[39420]: pgmap v2295: 47 pgs: 47 active+clean; 6.7 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 0 B/s wr, 791 op/s 2024-02-09T18:02:24.386 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:24 smithi099 ceph-mon[39420]: osdmap e927: 12 total, 12 up, 11 in 2024-02-09T18:02:24.432 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:24.477 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:02:24.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:24.486 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:02:24.586 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:02:24.639 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:24 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:02:24.639 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:24 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:02:24.639 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:24 smithi179 ceph-mon[43975]: from='client.26756 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T18:02:24.639 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:24 smithi179 ceph-mon[43975]: pgmap v2295: 47 pgs: 47 active+clean; 6.7 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 0 B/s wr, 791 op/s 2024-02-09T18:02:24.639 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:24 smithi179 ceph-mon[43975]: osdmap e927: 12 total, 12 up, 11 in 2024-02-09T18:02:24.647 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:24.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:24 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:02:24.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:24 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:02:24.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:24 smithi162 ceph-mon[44304]: from='client.26756 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T18:02:24.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:24 smithi162 ceph-mon[44304]: pgmap v2295: 47 pgs: 47 active+clean; 6.7 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 0 B/s wr, 791 op/s 2024-02-09T18:02:24.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:24 smithi162 ceph-mon[44304]: osdmap e927: 12 total, 12 up, 11 in 2024-02-09T18:02:24.677 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:02:24.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:24.684 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:02:24.715 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:02:24.766 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":2294,"stamp":"2024-02-09T18:02:21.299315+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7271416376,"num_objects":110947,"num_object_clones":0,"num_object_copies":332841,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":110947,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158815,"num_write_kb":8632257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":163267,"ondisk_log_size":163267,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":144,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":30175560,"kb_used_data":21580776,"kb_used_omap":429,"kb_used_meta":8594066,"kb_avail":1001021112,"statfs":{"total":1055945392128,"available":1025045618688,"internally_reserved":0,"allocated":22098714624,"data_stored":22095528204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":439572,"internal_metadata":8800324332},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-511508480,"num_objects":-7805,"num_object_clones":0,"num_object_copies":-23415,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7805,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003612"},"pg_stats":[{"pgid":"1.1e","version":"63'139","reported_seq":1172,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:03.062191+0000","last_change":"2024-02-09T18:01:38.358617+0000","last_active":"2024-02-09T18:02:03.062191+0000","last_peered":"2024-02-09T18:02:03.062191+0000","last_clean":"2024-02-09T18:02:03.062191+0000","last_became_active":"2024-02-09T17:59:12.002465+0000","last_became_peered":"2024-02-09T17:59:12.002465+0000","last_unstale":"2024-02-09T18:02:03.062191+0000","last_undegraded":"2024-02-09T18:02:03.062191+0000","last_fullsized":"2024-02-09T18:02:03.062191+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:38.358497+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:01:38.358497+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:03:07.994234+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,7],"acting":[11,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"63'139","reported_seq":1194,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-02-09T18:02:18.088905+0000","last_change":"2024-02-09T18:01:42.251012+0000","last_active":"2024-02-09T18:02:18.088905+0000","last_peered":"2024-02-09T18:02:18.088905+0000","last_clean":"2024-02-09T18:02:18.088905+0000","last_became_active":"2024-02-09T17:43:28.594296+0000","last_became_peered":"2024-02-09T17:43:28.594296+0000","last_unstale":"2024-02-09T18:02:18.088905+0000","last_undegraded":"2024-02-09T18:02:18.088905+0000","last_fullsized":"2024-02-09T18:02:18.088905+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:42.250926+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:48:24.634092+0000","last_clean_scrub_stamp":"2024-02-09T18:01:42.250926+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:49.813369+0000","scrub_duration":153,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"63'139","reported_seq":1711,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:03.068795+0000","last_change":"2024-02-09T18:01:40.309486+0000","last_active":"2024-02-09T18:02:03.068795+0000","last_peered":"2024-02-09T18:02:03.068795+0000","last_clean":"2024-02-09T18:02:03.068795+0000","last_became_active":"2024-02-09T17:43:19.103584+0000","last_became_peered":"2024-02-09T17:43:19.103584+0000","last_unstale":"2024-02-09T18:02:03.068795+0000","last_undegraded":"2024-02-09T18:02:03.068795+0000","last_fullsized":"2024-02-09T18:02:03.068795+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:40.309363+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:23.912504+0000","last_clean_scrub_stamp":"2024-02-09T18:01:40.309363+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:48.235694+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"63'139","reported_seq":1509,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:03.068774+0000","last_change":"2024-02-09T18:01:36.490300+0000","last_active":"2024-02-09T18:02:03.068774+0000","last_peered":"2024-02-09T18:02:03.068774+0000","last_clean":"2024-02-09T18:02:03.068774+0000","last_became_active":"2024-02-09T17:43:19.102729+0000","last_became_peered":"2024-02-09T17:43:19.102729+0000","last_unstale":"2024-02-09T18:02:03.068774+0000","last_undegraded":"2024-02-09T18:02:03.068774+0000","last_fullsized":"2024-02-09T18:02:03.068774+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:36.490178+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:45:44.988633+0000","last_clean_scrub_stamp":"2024-02-09T18:01:36.490178+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:47.318325+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"63'139","reported_seq":405,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-02-09T18:02:18.088595+0000","last_change":"2024-02-09T18:00:46.991880+0000","last_active":"2024-02-09T18:02:18.088595+0000","last_peered":"2024-02-09T18:02:18.088595+0000","last_clean":"2024-02-09T18:02:18.088595+0000","last_became_active":"2024-02-09T17:59:30.061943+0000","last_became_peered":"2024-02-09T17:59:30.061943+0000","last_unstale":"2024-02-09T18:02:18.088595+0000","last_undegraded":"2024-02-09T18:02:18.088595+0000","last_fullsized":"2024-02-09T18:02:18.088595+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:00:46.991759+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:00:46.991759+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,11],"acting":[1,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.19","version":"63'139","reported_seq":2020,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-02-09T18:02:18.098405+0000","last_change":"2024-02-09T18:01:35.688095+0000","last_active":"2024-02-09T18:02:18.098405+0000","last_peered":"2024-02-09T18:02:18.098405+0000","last_clean":"2024-02-09T18:02:18.098405+0000","last_became_active":"2024-02-09T17:55:21.231291+0000","last_became_peered":"2024-02-09T17:55:21.231291+0000","last_unstale":"2024-02-09T18:02:18.098405+0000","last_undegraded":"2024-02-09T18:02:18.098405+0000","last_fullsized":"2024-02-09T18:02:18.098405+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":791,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:35.687991+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:30.193194+0000","last_clean_scrub_stamp":"2024-02-09T18:01:35.687991+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:52.363506+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"63'139","reported_seq":1857,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-02-09T18:02:18.093118+0000","last_change":"2024-02-09T18:01:31.621577+0000","last_active":"2024-02-09T18:02:18.093118+0000","last_peered":"2024-02-09T18:02:18.093118+0000","last_clean":"2024-02-09T18:02:18.093118+0000","last_became_active":"2024-02-09T17:43:28.585573+0000","last_became_peered":"2024-02-09T17:43:28.585573+0000","last_unstale":"2024-02-09T18:02:18.093118+0000","last_undegraded":"2024-02-09T18:02:18.093118+0000","last_fullsized":"2024-02-09T18:02:18.093118+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:31.621498+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:01:31.621498+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:51.160170+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"63'139","reported_seq":1301,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:03.068637+0000","last_change":"2024-02-09T18:01:45.544759+0000","last_active":"2024-02-09T18:02:03.068637+0000","last_peered":"2024-02-09T18:02:03.068637+0000","last_clean":"2024-02-09T18:02:03.068637+0000","last_became_active":"2024-02-09T17:48:46.184419+0000","last_became_peered":"2024-02-09T17:48:46.184419+0000","last_unstale":"2024-02-09T18:02:03.068637+0000","last_undegraded":"2024-02-09T18:02:03.068637+0000","last_fullsized":"2024-02-09T18:02:03.068637+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:45.544660+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:44:50.214686+0000","last_clean_scrub_stamp":"2024-02-09T18:01:45.544660+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:58.555199+0000","scrub_duration":211,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"63'139","reported_seq":1187,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-02-09T18:02:18.089031+0000","last_change":"2024-02-09T18:01:34.123704+0000","last_active":"2024-02-09T18:02:18.089031+0000","last_peered":"2024-02-09T18:02:18.089031+0000","last_clean":"2024-02-09T18:02:18.089031+0000","last_became_active":"2024-02-09T17:59:12.003987+0000","last_became_peered":"2024-02-09T17:59:12.003987+0000","last_unstale":"2024-02-09T18:02:18.089031+0000","last_undegraded":"2024-02-09T18:02:18.089031+0000","last_fullsized":"2024-02-09T18:02:18.089031+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:34.123585+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:11.970338+0000","last_clean_scrub_stamp":"2024-02-09T18:01:34.123585+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:58.496407+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"63'139","reported_seq":1612,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:03.060537+0000","last_change":"2024-02-09T18:01:53.216105+0000","last_active":"2024-02-09T18:02:03.060537+0000","last_peered":"2024-02-09T18:02:03.060537+0000","last_clean":"2024-02-09T18:02:03.060537+0000","last_became_active":"2024-02-09T17:43:19.138398+0000","last_became_peered":"2024-02-09T17:43:19.138398+0000","last_unstale":"2024-02-09T18:02:03.060537+0000","last_undegraded":"2024-02-09T18:02:03.060537+0000","last_fullsized":"2024-02-09T18:02:03.060537+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:53.215991+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:01:53.215991+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:03:22.751872+0000","scrub_duration":218,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"63'139","reported_seq":1686,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-02-09T18:02:18.088405+0000","last_change":"2024-02-09T18:01:35.453210+0000","last_active":"2024-02-09T18:02:18.088405+0000","last_peered":"2024-02-09T18:02:18.088405+0000","last_clean":"2024-02-09T18:02:18.088405+0000","last_became_active":"2024-02-09T17:43:28.589307+0000","last_became_peered":"2024-02-09T17:43:28.589307+0000","last_unstale":"2024-02-09T18:02:18.088405+0000","last_undegraded":"2024-02-09T18:02:18.088405+0000","last_fullsized":"2024-02-09T18:02:18.088405+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:35.453077+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:38:26.968892+0000","last_clean_scrub_stamp":"2024-02-09T18:01:35.453077+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:53.479776+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"925'10018","reported_seq":10200,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:17.648517+0000","last_change":"2024-02-09T18:01:34.244695+0000","last_active":"2024-02-09T18:02:17.648517+0000","last_peered":"2024-02-09T18:02:17.648517+0000","last_clean":"2024-02-09T18:02:17.648517+0000","last_became_active":"2024-02-09T18:00:07.530069+0000","last_became_peered":"2024-02-09T18:00:07.530069+0000","last_unstale":"2024-02-09T18:02:17.648517+0000","last_undegraded":"2024-02-09T18:02:17.648517+0000","last_fullsized":"2024-02-09T18:02:17.648517+0000","mapping_epoch":882,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"914'7732","last_scrub_stamp":"2024-02-09T18:01:34.244573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_clean_scrub_stamp":"2024-02-09T18:01:34.244573+0000","objects_scrubbed":7702,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:43.273517+0000","scrub_duration":1550,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":467664896,"num_objects":7136,"num_object_clones":0,"num_object_copies":21408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10018,"num_write_kb":548928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,5],"acting":[9,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"63'139","reported_seq":2262,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:03.060283+0000","last_change":"2024-02-09T18:01:48.080986+0000","last_active":"2024-02-09T18:02:03.060283+0000","last_peered":"2024-02-09T18:02:03.060283+0000","last_clean":"2024-02-09T18:02:03.060283+0000","last_became_active":"2024-02-09T17:41:14.521079+0000","last_became_peered":"2024-02-09T17:41:14.521079+0000","last_unstale":"2024-02-09T18:02:03.060283+0000","last_undegraded":"2024-02-09T18:02:03.060283+0000","last_fullsized":"2024-02-09T18:02:03.060283+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:48.080862+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:01:48.080862+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:03:14.657015+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.a","version":"926'9990","reported_seq":10177,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-02-09T18:02:20.611325+0000","last_change":"2024-02-09T18:01:56.341368+0000","last_active":"2024-02-09T18:02:20.611325+0000","last_peered":"2024-02-09T18:02:20.611325+0000","last_clean":"2024-02-09T18:02:20.611325+0000","last_became_active":"2024-02-09T18:00:07.530662+0000","last_became_peered":"2024-02-09T18:00:07.530662+0000","last_unstale":"2024-02-09T18:02:20.611325+0000","last_undegraded":"2024-02-09T18:02:20.611325+0000","last_fullsized":"2024-02-09T18:02:20.611325+0000","mapping_epoch":882,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"921'8782","last_scrub_stamp":"2024-02-09T18:01:56.341243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_clean_scrub_stamp":"2024-02-09T18:01:56.341243+0000","objects_scrubbed":8111,"log_size":9990,"log_dups_size":0,"ondisk_log_size":9990,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:03:18.609818+0000","scrub_duration":2244,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448266240,"num_objects":6840,"num_object_clones":0,"num_object_copies":20520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9990,"num_write_kb":538560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"63'139","reported_seq":1847,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-02-09T18:02:18.134235+0000","last_change":"2024-02-09T18:01:50.478494+0000","last_active":"2024-02-09T18:02:18.134235+0000","last_peered":"2024-02-09T18:02:18.134235+0000","last_clean":"2024-02-09T18:02:18.134235+0000","last_became_active":"2024-02-09T17:59:30.054441+0000","last_became_peered":"2024-02-09T17:59:30.054441+0000","last_unstale":"2024-02-09T18:02:18.134235+0000","last_undegraded":"2024-02-09T18:02:18.134235+0000","last_fullsized":"2024-02-09T18:02:18.134235+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:50.478373+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:01:50.478373+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:03:15.818287+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,8],"acting":[0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"925'9944","reported_seq":10131,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:17.608505+0000","last_change":"2024-02-09T18:01:44.676180+0000","last_active":"2024-02-09T18:02:17.608505+0000","last_peered":"2024-02-09T18:02:17.608505+0000","last_clean":"2024-02-09T18:02:17.608505+0000","last_became_active":"2024-02-09T18:00:07.529897+0000","last_became_peered":"2024-02-09T18:00:07.529897+0000","last_unstale":"2024-02-09T18:02:17.608505+0000","last_undegraded":"2024-02-09T18:02:17.608505+0000","last_fullsized":"2024-02-09T18:02:17.608505+0000","mapping_epoch":882,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"917'8181","last_scrub_stamp":"2024-02-09T18:01:44.676020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_clean_scrub_stamp":"2024-02-09T18:01:44.676020+0000","objects_scrubbed":8120,"log_size":9944,"log_dups_size":0,"ondisk_log_size":9944,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:45.871299+0000","scrub_duration":2392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":457441280,"num_objects":6980,"num_object_clones":0,"num_object_copies":20940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9944,"num_write_kb":541568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"63'139","reported_seq":1471,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-02-09T18:02:18.088621+0000","last_change":"2024-02-09T18:01:35.235026+0000","last_active":"2024-02-09T18:02:18.088621+0000","last_peered":"2024-02-09T18:02:18.088621+0000","last_clean":"2024-02-09T18:02:18.088621+0000","last_became_active":"2024-02-09T17:59:30.073778+0000","last_became_peered":"2024-02-09T17:59:30.073778+0000","last_unstale":"2024-02-09T18:02:18.088621+0000","last_undegraded":"2024-02-09T18:02:18.088621+0000","last_fullsized":"2024-02-09T18:02:18.088621+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:35.234907+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:46:40.684762+0000","last_clean_scrub_stamp":"2024-02-09T18:01:35.234907+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:53.437281+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,8],"acting":[1,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"926'9959","reported_seq":10148,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-02-09T18:02:18.926380+0000","last_change":"2024-02-09T18:01:43.964751+0000","last_active":"2024-02-09T18:02:18.926380+0000","last_peered":"2024-02-09T18:02:18.926380+0000","last_clean":"2024-02-09T18:02:18.926380+0000","last_became_active":"2024-02-09T18:00:07.523197+0000","last_became_peered":"2024-02-09T18:00:07.523197+0000","last_unstale":"2024-02-09T18:02:18.926380+0000","last_undegraded":"2024-02-09T18:02:18.926380+0000","last_fullsized":"2024-02-09T18:02:18.926380+0000","mapping_epoch":882,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"917'8180","last_scrub_stamp":"2024-02-09T18:01:43.964525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_clean_scrub_stamp":"2024-02-09T18:01:43.964525+0000","objects_scrubbed":8109,"log_size":9959,"log_dups_size":0,"ondisk_log_size":9959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:52.470537+0000","scrub_duration":2493,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459210752,"num_objects":7007,"num_object_clones":0,"num_object_copies":21021,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9959,"num_write_kb":542912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,5],"acting":[10,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"63'139","reported_seq":1006,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-02-09T18:02:18.098600+0000","last_change":"2024-02-09T18:01:55.578635+0000","last_active":"2024-02-09T18:02:18.098600+0000","last_peered":"2024-02-09T18:02:18.098600+0000","last_clean":"2024-02-09T18:02:18.098600+0000","last_became_active":"2024-02-09T17:54:32.415032+0000","last_became_peered":"2024-02-09T17:54:32.415032+0000","last_unstale":"2024-02-09T18:02:18.098600+0000","last_undegraded":"2024-02-09T18:02:18.098600+0000","last_fullsized":"2024-02-09T18:02:18.098600+0000","mapping_epoch":766,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":767,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:55.578511+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T18:00:46.701010+0000","last_clean_scrub_stamp":"2024-02-09T18:01:55.578511+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:03:05.676030+0000","scrub_duration":102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.e","version":"926'9910","reported_seq":10095,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-02-09T18:02:18.939990+0000","last_change":"2024-02-09T18:01:47.601326+0000","last_active":"2024-02-09T18:02:18.939990+0000","last_peered":"2024-02-09T18:02:18.939990+0000","last_clean":"2024-02-09T18:02:18.939990+0000","last_became_active":"2024-02-09T18:00:07.522722+0000","last_became_peered":"2024-02-09T18:00:07.522722+0000","last_unstale":"2024-02-09T18:02:18.939990+0000","last_undegraded":"2024-02-09T18:02:18.939990+0000","last_fullsized":"2024-02-09T18:02:18.939990+0000","mapping_epoch":882,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"919'8328","last_scrub_stamp":"2024-02-09T18:01:47.601038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_clean_scrub_stamp":"2024-02-09T18:01:47.601038+0000","objects_scrubbed":8281,"log_size":9910,"log_dups_size":0,"ondisk_log_size":9910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:50.286460+0000","scrub_duration":2081,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":455999488,"num_objects":6958,"num_object_clones":0,"num_object_copies":20874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9910,"num_write_kb":539776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,1],"acting":[10,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"63'139","reported_seq":1765,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-02-09T18:02:18.096583+0000","last_change":"2024-02-09T18:01:34.702740+0000","last_active":"2024-02-09T18:02:18.096583+0000","last_peered":"2024-02-09T18:02:18.096583+0000","last_clean":"2024-02-09T18:02:18.096583+0000","last_became_active":"2024-02-09T17:43:14.024894+0000","last_became_peered":"2024-02-09T17:43:14.024894+0000","last_unstale":"2024-02-09T18:02:18.096583+0000","last_undegraded":"2024-02-09T18:02:18.096583+0000","last_fullsized":"2024-02-09T18:02:18.096583+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:34.702627+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:11.970338+0000","last_clean_scrub_stamp":"2024-02-09T18:01:34.702627+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:38.122355+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.f","version":"926'13369","reported_seq":13625,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-02-09T18:02:20.649588+0000","last_change":"2024-02-09T18:02:18.182853+0000","last_active":"2024-02-09T18:02:20.649588+0000","last_peered":"2024-02-09T18:02:20.649588+0000","last_clean":"2024-02-09T18:02:20.649588+0000","last_became_active":"2024-02-09T18:00:07.523115+0000","last_became_peered":"2024-02-09T18:00:07.523115+0000","last_unstale":"2024-02-09T18:02:20.649588+0000","last_undegraded":"2024-02-09T18:02:20.649588+0000","last_fullsized":"2024-02-09T18:02:20.649588+0000","mapping_epoch":882,"log_start":"896'3300","ondisk_log_start":"896'3300","created":882,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_clean_scrub_stamp":"2024-02-09T18:00:06.282957+0000","objects_scrubbed":3824,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:23.141577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453115904,"num_objects":6914,"num_object_clones":0,"num_object_copies":20742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9926,"num_write_kb":538880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,9],"acting":[1,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"63'139","reported_seq":1390,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-02-09T18:02:18.117841+0000","last_change":"2024-02-09T18:01:37.427302+0000","last_active":"2024-02-09T18:02:18.117841+0000","last_peered":"2024-02-09T18:02:18.117841+0000","last_clean":"2024-02-09T18:02:18.117841+0000","last_became_active":"2024-02-09T17:52:32.292519+0000","last_became_peered":"2024-02-09T17:52:32.292519+0000","last_unstale":"2024-02-09T18:02:18.117841+0000","last_undegraded":"2024-02-09T18:02:18.117841+0000","last_fullsized":"2024-02-09T18:02:18.117841+0000","mapping_epoch":716,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":717,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:37.427192+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:01:37.427192+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:59.188404+0000","scrub_duration":384,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.c","version":"926'10038","reported_seq":10220,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-02-09T18:02:21.069718+0000","last_change":"2024-02-09T18:01:34.347659+0000","last_active":"2024-02-09T18:02:21.069718+0000","last_peered":"2024-02-09T18:02:21.069718+0000","last_clean":"2024-02-09T18:02:21.069718+0000","last_became_active":"2024-02-09T18:00:07.372788+0000","last_became_peered":"2024-02-09T18:00:07.372788+0000","last_unstale":"2024-02-09T18:02:21.069718+0000","last_undegraded":"2024-02-09T18:02:21.069718+0000","last_fullsized":"2024-02-09T18:02:21.069718+0000","mapping_epoch":882,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"914'7638","last_scrub_stamp":"2024-02-09T18:01:34.347419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_clean_scrub_stamp":"2024-02-09T18:01:34.347419+0000","objects_scrubbed":7603,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:42.673871+0000","scrub_duration":1789,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453640192,"num_objects":6922,"num_object_clones":0,"num_object_copies":20766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10038,"num_write_kb":542720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,0],"acting":[3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"63'139","reported_seq":1842,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:03.062082+0000","last_change":"2024-02-09T18:01:40.226412+0000","last_active":"2024-02-09T18:02:03.062082+0000","last_peered":"2024-02-09T18:02:03.062082+0000","last_clean":"2024-02-09T18:02:03.062082+0000","last_became_active":"2024-02-09T17:33:00.284924+0000","last_became_peered":"2024-02-09T17:33:00.284924+0000","last_unstale":"2024-02-09T18:02:03.062082+0000","last_undegraded":"2024-02-09T18:02:03.062082+0000","last_fullsized":"2024-02-09T18:02:03.062082+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:40.226335+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:56:45.406767+0000","last_clean_scrub_stamp":"2024-02-09T18:01:40.226335+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:03:07.722142+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.d","version":"926'9932","reported_seq":10122,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-02-09T18:02:20.670893+0000","last_change":"2024-02-09T18:01:47.090054+0000","last_active":"2024-02-09T18:02:20.670893+0000","last_peered":"2024-02-09T18:02:20.670893+0000","last_clean":"2024-02-09T18:02:20.670893+0000","last_became_active":"2024-02-09T18:00:07.530751+0000","last_became_peered":"2024-02-09T18:00:07.530751+0000","last_unstale":"2024-02-09T18:02:20.670893+0000","last_undegraded":"2024-02-09T18:02:20.670893+0000","last_fullsized":"2024-02-09T18:02:20.670893+0000","mapping_epoch":882,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"919'8257","last_scrub_stamp":"2024-02-09T18:01:47.089900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_clean_scrub_stamp":"2024-02-09T18:01:47.089900+0000","objects_scrubbed":8211,"log_size":9932,"log_dups_size":0,"ondisk_log_size":9932,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:53.209083+0000","scrub_duration":2004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":449249304,"num_objects":6856,"num_object_clones":0,"num_object_copies":20568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6856,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9932,"num_write_kb":537153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"63'139","reported_seq":1694,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-02-09T18:02:18.096402+0000","last_change":"2024-02-09T18:01:36.708824+0000","last_active":"2024-02-09T18:02:18.096402+0000","last_peered":"2024-02-09T18:02:18.096402+0000","last_clean":"2024-02-09T18:02:18.096402+0000","last_became_active":"2024-02-09T17:58:10.037010+0000","last_became_peered":"2024-02-09T17:58:10.037010+0000","last_unstale":"2024-02-09T18:02:18.096402+0000","last_undegraded":"2024-02-09T18:02:18.096402+0000","last_fullsized":"2024-02-09T18:02:18.096402+0000","mapping_epoch":839,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":840,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:36.708692+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:01:36.708692+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:03:03.478015+0000","scrub_duration":171,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,8],"acting":[10,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.2","version":"926'10061","reported_seq":10243,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-02-09T18:02:21.068189+0000","last_change":"2024-02-09T18:01:35.388996+0000","last_active":"2024-02-09T18:02:21.068189+0000","last_peered":"2024-02-09T18:02:21.068189+0000","last_clean":"2024-02-09T18:02:21.068189+0000","last_became_active":"2024-02-09T18:00:07.021665+0000","last_became_peered":"2024-02-09T18:00:07.021665+0000","last_unstale":"2024-02-09T18:02:21.068189+0000","last_undegraded":"2024-02-09T18:02:21.068189+0000","last_fullsized":"2024-02-09T18:02:21.068189+0000","mapping_epoch":882,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"914'7575","last_scrub_stamp":"2024-02-09T18:01:35.388781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_clean_scrub_stamp":"2024-02-09T18:01:35.388781+0000","objects_scrubbed":7520,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:42.842479+0000","scrub_duration":1798,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":442957824,"num_objects":6759,"num_object_clones":0,"num_object_copies":20277,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10061,"num_write_kb":538240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,9],"acting":[3,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"63'139","reported_seq":1773,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:03.074465+0000","last_change":"2024-02-09T18:01:46.756318+0000","last_active":"2024-02-09T18:02:03.074465+0000","last_peered":"2024-02-09T18:02:03.074465+0000","last_clean":"2024-02-09T18:02:03.074465+0000","last_became_active":"2024-02-09T17:59:30.057223+0000","last_became_peered":"2024-02-09T17:59:30.057223+0000","last_unstale":"2024-02-09T18:02:03.074465+0000","last_undegraded":"2024-02-09T18:02:03.074465+0000","last_fullsized":"2024-02-09T18:02:03.074465+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:46.756218+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:56:44.962386+0000","last_clean_scrub_stamp":"2024-02-09T18:01:46.756218+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:03:11.667820+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,0],"acting":[9,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.3","version":"925'9864","reported_seq":10052,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:17.657649+0000","last_change":"2024-02-09T18:01:45.906378+0000","last_active":"2024-02-09T18:02:17.657649+0000","last_peered":"2024-02-09T18:02:17.657649+0000","last_clean":"2024-02-09T18:02:17.657649+0000","last_became_active":"2024-02-09T18:00:07.523241+0000","last_became_peered":"2024-02-09T18:00:07.523241+0000","last_unstale":"2024-02-09T18:02:17.657649+0000","last_undegraded":"2024-02-09T18:02:17.657649+0000","last_fullsized":"2024-02-09T18:02:17.657649+0000","mapping_epoch":882,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"918'8296","last_scrub_stamp":"2024-02-09T18:01:45.904138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_clean_scrub_stamp":"2024-02-09T18:01:45.904138+0000","objects_scrubbed":8245,"log_size":9864,"log_dups_size":0,"ondisk_log_size":9864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:47.771934+0000","scrub_duration":2159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":462028800,"num_objects":7050,"num_object_clones":0,"num_object_copies":21150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9864,"num_write_kb":541248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,5],"acting":[9,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.b","version":"63'139","reported_seq":1131,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:03.061906+0000","last_change":"2024-02-09T18:01:34.507016+0000","last_active":"2024-02-09T18:02:03.061906+0000","last_peered":"2024-02-09T18:02:03.061906+0000","last_clean":"2024-02-09T18:02:03.061906+0000","last_became_active":"2024-02-09T17:55:08.938203+0000","last_became_peered":"2024-02-09T17:55:08.938203+0000","last_unstale":"2024-02-09T18:02:03.061906+0000","last_undegraded":"2024-02-09T18:02:03.061906+0000","last_fullsized":"2024-02-09T18:02:03.061906+0000","mapping_epoch":781,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":782,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:34.506894+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:01:34.506894+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:39.439332+0000","scrub_duration":295,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.0","version":"926'10067","reported_seq":10257,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-02-09T18:02:20.615149+0000","last_change":"2024-02-09T18:01:46.422565+0000","last_active":"2024-02-09T18:02:20.615149+0000","last_peered":"2024-02-09T18:02:20.615149+0000","last_clean":"2024-02-09T18:02:20.615149+0000","last_became_active":"2024-02-09T18:00:07.131105+0000","last_became_peered":"2024-02-09T18:00:07.131105+0000","last_unstale":"2024-02-09T18:02:20.615149+0000","last_undegraded":"2024-02-09T18:02:20.615149+0000","last_fullsized":"2024-02-09T18:02:20.615149+0000","mapping_epoch":882,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"918'8233","last_scrub_stamp":"2024-02-09T18:01:46.422346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_clean_scrub_stamp":"2024-02-09T18:01:46.422346+0000","objects_scrubbed":8168,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:47.555306+0000","scrub_duration":2345,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":445448192,"num_objects":6797,"num_object_clones":0,"num_object_copies":20391,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10067,"num_write_kb":539648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"63'139","reported_seq":1263,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-02-09T18:02:18.087242+0000","last_change":"2024-02-09T18:01:47.651147+0000","last_active":"2024-02-09T18:02:18.087242+0000","last_peered":"2024-02-09T18:02:18.087242+0000","last_clean":"2024-02-09T18:02:18.087242+0000","last_became_active":"2024-02-09T17:59:12.022685+0000","last_became_peered":"2024-02-09T17:59:12.022685+0000","last_unstale":"2024-02-09T18:02:18.087242+0000","last_undegraded":"2024-02-09T18:02:18.087242+0000","last_fullsized":"2024-02-09T18:02:18.087242+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:47.651055+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:01:47.651055+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:03:03.122903+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1","version":"925'9772","reported_seq":9957,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:17.607330+0000","last_change":"2024-02-09T18:01:44.919616+0000","last_active":"2024-02-09T18:02:17.607330+0000","last_peered":"2024-02-09T18:02:17.607330+0000","last_clean":"2024-02-09T18:02:17.607330+0000","last_became_active":"2024-02-09T18:00:07.131217+0000","last_became_peered":"2024-02-09T18:00:07.131217+0000","last_unstale":"2024-02-09T18:02:17.607330+0000","last_undegraded":"2024-02-09T18:02:17.607330+0000","last_fullsized":"2024-02-09T18:02:17.607330+0000","mapping_epoch":882,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"917'8098","last_scrub_stamp":"2024-02-09T18:01:44.919443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_clean_scrub_stamp":"2024-02-09T18:01:44.919443+0000","objects_scrubbed":8057,"log_size":9772,"log_dups_size":0,"ondisk_log_size":9772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:03:13.502087+0000","scrub_duration":2210,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453902336,"num_objects":6926,"num_object_clones":0,"num_object_copies":20778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9772,"num_write_kb":534336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,6],"acting":[9,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.9","version":"63'139","reported_seq":2268,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:03.060247+0000","last_change":"2024-02-09T18:01:44.388634+0000","last_active":"2024-02-09T18:02:03.060247+0000","last_peered":"2024-02-09T18:02:03.060247+0000","last_clean":"2024-02-09T18:02:03.060247+0000","last_became_active":"2024-02-09T17:43:28.590237+0000","last_became_peered":"2024-02-09T17:43:28.590237+0000","last_unstale":"2024-02-09T18:02:03.060247+0000","last_undegraded":"2024-02-09T18:02:03.060247+0000","last_fullsized":"2024-02-09T18:02:03.060247+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:44.388511+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:59:49.716259+0000","last_clean_scrub_stamp":"2024-02-09T18:01:44.388511+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:59.402237+0000","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.6","version":"926'9789","reported_seq":9974,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-02-09T18:02:18.914297+0000","last_change":"2024-02-09T18:01:40.622569+0000","last_active":"2024-02-09T18:02:18.914297+0000","last_peered":"2024-02-09T18:02:18.914297+0000","last_clean":"2024-02-09T18:02:18.914297+0000","last_became_active":"2024-02-09T18:00:07.523268+0000","last_became_peered":"2024-02-09T18:00:07.523268+0000","last_unstale":"2024-02-09T18:02:18.914297+0000","last_undegraded":"2024-02-09T18:02:18.914297+0000","last_fullsized":"2024-02-09T18:02:18.914297+0000","mapping_epoch":882,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"917'7893","last_scrub_stamp":"2024-02-09T18:01:40.622417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_clean_scrub_stamp":"2024-02-09T18:01:40.622417+0000","objects_scrubbed":7824,"log_size":9789,"log_dups_size":0,"ondisk_log_size":9789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:03:03.921720+0000","scrub_duration":2093,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":449773568,"num_objects":6863,"num_object_clones":0,"num_object_copies":20589,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9789,"num_write_kb":532864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,1],"acting":[10,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.e","version":"63'139","reported_seq":400,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:03.060389+0000","last_change":"2024-02-09T18:01:43.193091+0000","last_active":"2024-02-09T18:02:03.060389+0000","last_peered":"2024-02-09T18:02:03.060389+0000","last_clean":"2024-02-09T18:02:03.060389+0000","last_became_active":"2024-02-09T17:59:30.050934+0000","last_became_peered":"2024-02-09T17:59:30.050934+0000","last_unstale":"2024-02-09T18:02:03.060389+0000","last_undegraded":"2024-02-09T18:02:03.060389+0000","last_fullsized":"2024-02-09T18:02:03.060389+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:43.192960+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:14.981453+0000","last_clean_scrub_stamp":"2024-02-09T18:01:43.192960+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:03:12.502112+0000","scrub_duration":73,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,8],"acting":[7,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"926'9795","reported_seq":9985,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-02-09T18:02:18.863423+0000","last_change":"2024-02-09T18:01:49.345309+0000","last_active":"2024-02-09T18:02:18.863423+0000","last_peered":"2024-02-09T18:02:18.863423+0000","last_clean":"2024-02-09T18:02:18.863423+0000","last_became_active":"2024-02-09T18:00:07.554448+0000","last_became_peered":"2024-02-09T18:00:07.554448+0000","last_unstale":"2024-02-09T18:02:18.863423+0000","last_undegraded":"2024-02-09T18:02:18.863423+0000","last_fullsized":"2024-02-09T18:02:18.863423+0000","mapping_epoch":882,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"919'8379","last_scrub_stamp":"2024-02-09T18:01:49.345117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_clean_scrub_stamp":"2024-02-09T18:01:49.345117+0000","objects_scrubbed":8341,"log_size":9795,"log_dups_size":0,"ondisk_log_size":9795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:03:09.342813+0000","scrub_duration":1866,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":457900032,"num_objects":6987,"num_object_clones":0,"num_object_copies":20961,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9795,"num_write_kb":537024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,3],"acting":[10,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"63'139","reported_seq":1795,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-02-09T18:02:18.099249+0000","last_change":"2024-02-09T18:01:33.327365+0000","last_active":"2024-02-09T18:02:18.099249+0000","last_peered":"2024-02-09T18:02:18.099249+0000","last_clean":"2024-02-09T18:02:18.099249+0000","last_became_active":"2024-02-09T17:59:30.045852+0000","last_became_peered":"2024-02-09T17:59:30.045852+0000","last_unstale":"2024-02-09T18:02:18.099249+0000","last_undegraded":"2024-02-09T18:02:18.099249+0000","last_fullsized":"2024-02-09T18:02:18.099249+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:33.327269+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:48:34.651647+0000","last_clean_scrub_stamp":"2024-02-09T18:01:33.327269+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:44.381596+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"926'9904","reported_seq":10091,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-02-09T18:02:20.657302+0000","last_change":"2024-02-09T18:01:58.054003+0000","last_active":"2024-02-09T18:02:20.657302+0000","last_peered":"2024-02-09T18:02:20.657302+0000","last_clean":"2024-02-09T18:02:20.657302+0000","last_became_active":"2024-02-09T18:00:07.024203+0000","last_became_peered":"2024-02-09T18:00:07.024203+0000","last_unstale":"2024-02-09T18:02:20.657302+0000","last_undegraded":"2024-02-09T18:02:20.657302+0000","last_fullsized":"2024-02-09T18:02:20.657302+0000","mapping_epoch":882,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"921'8840","last_scrub_stamp":"2024-02-09T18:01:58.053829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_clean_scrub_stamp":"2024-02-09T18:01:58.053829+0000","objects_scrubbed":7986,"log_size":9904,"log_dups_size":0,"ondisk_log_size":9904,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:03:20.575579+0000","scrub_duration":1808,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":451149824,"num_objects":6884,"num_object_clones":0,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9904,"num_write_kb":537216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"63'139","reported_seq":1758,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-02-09T18:02:18.087357+0000","last_change":"2024-02-09T18:01:39.607321+0000","last_active":"2024-02-09T18:02:18.087357+0000","last_peered":"2024-02-09T18:02:18.087357+0000","last_clean":"2024-02-09T18:02:18.087357+0000","last_became_active":"2024-02-09T17:59:12.208604+0000","last_became_peered":"2024-02-09T17:59:12.208604+0000","last_unstale":"2024-02-09T18:02:18.087357+0000","last_undegraded":"2024-02-09T18:02:18.087357+0000","last_fullsized":"2024-02-09T18:02:18.087357+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:39.607215+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:14.135321+0000","last_clean_scrub_stamp":"2024-02-09T18:01:39.607215+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:57.277819+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"925'9846","reported_seq":10031,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:17.083648+0000","last_change":"2024-02-09T18:01:43.124506+0000","last_active":"2024-02-09T18:02:17.083648+0000","last_peered":"2024-02-09T18:02:17.083648+0000","last_clean":"2024-02-09T18:02:17.083648+0000","last_became_active":"2024-02-09T18:00:07.530724+0000","last_became_peered":"2024-02-09T18:00:07.530724+0000","last_unstale":"2024-02-09T18:02:17.083648+0000","last_undegraded":"2024-02-09T18:02:17.083648+0000","last_fullsized":"2024-02-09T18:02:17.083648+0000","mapping_epoch":882,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"917'8139","last_scrub_stamp":"2024-02-09T18:01:43.124250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_clean_scrub_stamp":"2024-02-09T18:01:43.124250+0000","objects_scrubbed":8088,"log_size":9846,"log_dups_size":0,"ondisk_log_size":9846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:03:12.471464+0000","scrub_duration":1914,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":463077376,"num_objects":7066,"num_object_clones":0,"num_object_copies":21198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9846,"num_write_kb":541184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"63'139","reported_seq":1334,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:03.060615+0000","last_change":"2024-02-09T18:02:02.290371+0000","last_active":"2024-02-09T18:02:03.060615+0000","last_peered":"2024-02-09T18:02:03.060615+0000","last_clean":"2024-02-09T18:02:03.060615+0000","last_became_active":"2024-02-09T17:59:30.053570+0000","last_became_peered":"2024-02-09T17:59:30.053570+0000","last_unstale":"2024-02-09T18:02:03.060615+0000","last_undegraded":"2024-02-09T18:02:03.060615+0000","last_fullsized":"2024-02-09T18:02:03.060615+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:02:02.290244+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T18:00:49.890794+0000","last_clean_scrub_stamp":"2024-02-09T18:02:02.290244+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:03:14.747875+0000","scrub_duration":245,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,8],"acting":[7,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"63'139","reported_seq":1488,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:03.060219+0000","last_change":"2024-02-09T18:00:40.745321+0000","last_active":"2024-02-09T18:02:03.060219+0000","last_peered":"2024-02-09T18:02:03.060219+0000","last_clean":"2024-02-09T18:02:03.060219+0000","last_became_active":"2024-02-09T17:46:47.013920+0000","last_became_peered":"2024-02-09T17:46:47.013920+0000","last_unstale":"2024-02-09T18:02:03.060219+0000","last_undegraded":"2024-02-09T18:02:03.060219+0000","last_fullsized":"2024-02-09T18:02:03.060219+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:00:40.745190+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:48:35.170638+0000","last_clean_scrub_stamp":"2024-02-09T18:00:40.745190+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:02:09.398468+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"63'139","reported_seq":1759,"reported_epoch":925,"state":"active+clean","last_fresh":"2024-02-09T18:02:03.068380+0000","last_change":"2024-02-09T18:01:41.456117+0000","last_active":"2024-02-09T18:02:03.068380+0000","last_peered":"2024-02-09T18:02:03.068380+0000","last_clean":"2024-02-09T18:02:03.068380+0000","last_became_active":"2024-02-09T17:55:21.231902+0000","last_became_peered":"2024-02-09T17:55:21.231902+0000","last_unstale":"2024-02-09T18:02:03.068380+0000","last_undegraded":"2024-02-09T18:02:03.068380+0000","last_fullsized":"2024-02-09T18:02:03.068380+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":791,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:41.456012+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:44:51.190789+0000","last_clean_scrub_stamp":"2024-02-09T18:01:41.456012+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:03:04.012313+0000","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.12","version":"63'139","reported_seq":1844,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-02-09T18:02:18.088246+0000","last_change":"2024-02-09T18:02:02.359151+0000","last_active":"2024-02-09T18:02:18.088246+0000","last_peered":"2024-02-09T18:02:18.088246+0000","last_clean":"2024-02-09T18:02:18.088246+0000","last_became_active":"2024-02-09T17:46:47.015622+0000","last_became_peered":"2024-02-09T17:46:47.015622+0000","last_unstale":"2024-02-09T18:02:18.088246+0000","last_undegraded":"2024-02-09T18:02:18.088246+0000","last_fullsized":"2024-02-09T18:02:18.088246+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:02:02.359039+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:02:02.359039+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:03:02.704420+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.13","version":"63'139","reported_seq":1463,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-02-09T18:02:18.088747+0000","last_change":"2024-02-09T18:02:01.517991+0000","last_active":"2024-02-09T18:02:18.088747+0000","last_peered":"2024-02-09T18:02:18.088747+0000","last_clean":"2024-02-09T18:02:18.088747+0000","last_became_active":"2024-02-09T17:41:14.521672+0000","last_became_peered":"2024-02-09T17:41:14.521672+0000","last_unstale":"2024-02-09T18:02:18.088747+0000","last_undegraded":"2024-02-09T18:02:18.088747+0000","last_fullsized":"2024-02-09T18:02:18.088747+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:02:01.517864+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:02:01.517864+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:03:09.590936+0000","scrub_duration":293,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":7270826008,"num_objects":110945,"num_object_clones":0,"num_object_copies":332835,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":110945,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158815,"num_write_kb":8632257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21825794048,"data_stored":21825781832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158958,"ondisk_log_size":158958,"up":48,"acting":48,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547057121,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2768152,"kb_used_data":1812368,"kb_used_omap":1,"kb_used_meta":955774,"kb_avail":90977000,"statfs":{"total":95995035648,"available":93160448000,"internally_reserved":0,"allocated":1855864832,"data_stored":1855580073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":978713028},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 18:02:16 2024","interfaces":[{"interface":"back","average":{"1min":1.5469999999999999,"5min":1.583,"15min":1.7430000000000001},"min":{"1min":0.22800000000000001,"5min":0.22,"15min":0.21199999999999999},"max":{"1min":6.0460000000000003,"5min":13.307,"15min":20.003},"last":0.46200000000000002},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.8919999999999999,"15min":1.6779999999999999},"min":{"1min":0.23899999999999999,"5min":0.23300000000000001,"15min":0.23100000000000001},"max":{"1min":4.8230000000000004,"5min":9.0790000000000006,"15min":9.6259999999999994},"last":0.27700000000000002}]},{"osd":1,"last update":"Fri Feb 9 18:02:16 2024","interfaces":[{"interface":"back","average":{"1min":1.706,"5min":2.1779999999999999,"15min":1.8839999999999999},"min":{"1min":0.28100000000000003,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":7.0199999999999996,"5min":11.795999999999999,"15min":11.795999999999999},"last":3.8860000000000001},{"interface":"front","average":{"1min":1.6160000000000001,"5min":2.012,"15min":1.976},"min":{"1min":0.316,"5min":0.29999999999999999,"15min":0.25},"max":{"1min":4.2649999999999997,"5min":14.81,"15min":14.81},"last":0.316}]},{"osd":2,"last update":"Fri Feb 9 18:02:16 2024","interfaces":[{"interface":"back","average":{"1min":2.2930000000000001,"5min":2.0840000000000001,"15min":1.746},"min":{"1min":0.32400000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":7.4130000000000003,"5min":14.766999999999999,"15min":14.766999999999999},"last":2.9260000000000002},{"interface":"front","average":{"1min":1.7629999999999999,"5min":1.879,"15min":1.468},"min":{"1min":0.316,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":5.5199999999999996,"5min":13.337,"15min":13.337},"last":3.7509999999999999}]},{"osd":3,"last update":"Fri Feb 9 18:02:16 2024","interfaces":[{"interface":"back","average":{"1min":1.889,"5min":1.6719999999999999,"15min":1.6339999999999999},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":10.458,"5min":10.458,"15min":10.458},"last":0.35299999999999998},{"interface":"front","average":{"1min":1.704,"5min":1.7829999999999999,"15min":1.929},"min":{"1min":0.36499999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":4.4290000000000003,"5min":8.4770000000000003,"15min":8.4770000000000003},"last":4.2549999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:58:44 2024","interfaces":[{"interface":"back","average":{"1min":3.1709999999999998,"5min":2.5779999999999998,"15min":2.4790000000000001},"min":{"1min":0.40999999999999998,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":14.866,"5min":14.866,"15min":14.866},"last":0.45700000000000002},{"interface":"front","average":{"1min":3.2349999999999999,"5min":2.5640000000000001,"15min":2.452},"min":{"1min":0.496,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":14.939,"5min":14.939,"15min":14.939},"last":0.42299999999999999}]},{"osd":5,"last update":"Fri Feb 9 18:02:16 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.641,"15min":1.4830000000000001},"min":{"1min":0.29099999999999998,"5min":0.27000000000000002,"15min":0.252},"max":{"1min":6.0039999999999996,"5min":13.263999999999999,"15min":13.263999999999999},"last":0.433},{"interface":"front","average":{"1min":1.264,"5min":1.8500000000000001,"15min":1.6299999999999999},"min":{"1min":0.40899999999999997,"5min":0.34899999999999998,"15min":0.29199999999999998},"max":{"1min":4.3380000000000001,"5min":14.914999999999999,"15min":20.038},"last":0.54700000000000004}]},{"osd":6,"last update":"Fri Feb 9 18:02:16 2024","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":1.8899999999999999,"15min":1.6220000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.308},"max":{"1min":4.9059999999999997,"5min":9.1300000000000008,"15min":11.885},"last":0.40100000000000002},{"interface":"front","average":{"1min":1.286,"5min":1.8600000000000001,"15min":1.675},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.32400000000000001},"max":{"1min":4.3710000000000004,"5min":14.962,"15min":15.409000000000001},"last":0.57599999999999996}]},{"osd":7,"last update":"Fri Feb 9 18:02:16 2024","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.9019999999999999,"15min":1.6850000000000001},"min":{"1min":0.38500000000000001,"5min":0.38100000000000001,"15min":0.317},"max":{"1min":4.4000000000000004,"5min":14.890000000000001,"15min":15.51},"last":0.51300000000000001},{"interface":"front","average":{"1min":1.006,"5min":1.925,"15min":1.881},"min":{"1min":0.39100000000000001,"5min":0.34899999999999998,"15min":0.318},"max":{"1min":4.8819999999999997,"5min":9.1029999999999998,"15min":24.373999999999999},"last":0.44800000000000001}]},{"osd":8,"last update":"Fri Feb 9 18:01:24 2024","interfaces":[{"interface":"back","average":{"1min":1.9259999999999999,"5min":2.7799999999999998,"15min":1.931},"min":{"1min":0.46000000000000002,"5min":0.307,"15min":0.27900000000000003},"max":{"1min":5.3730000000000002,"5min":29.233000000000001,"15min":29.233000000000001},"last":0.86199999999999999},{"interface":"front","average":{"1min":1.9770000000000001,"5min":2.976,"15min":2.0350000000000001},"min":{"1min":0.55100000000000005,"5min":0.34499999999999997,"15min":0.32700000000000001},"max":{"1min":5.3959999999999999,"5min":29.091999999999999,"15min":29.091999999999999},"last":0.89000000000000001}]},{"osd":9,"last update":"Fri Feb 9 18:01:55 2024","interfaces":[{"interface":"back","average":{"1min":2.2810000000000001,"5min":2.3900000000000001,"15min":2.0350000000000001},"min":{"1min":0.36699999999999999,"5min":0.32600000000000001,"15min":0.29399999999999998},"max":{"1min":6.0810000000000004,"5min":19.469000000000001,"15min":22.613},"last":0.64100000000000001},{"interface":"front","average":{"1min":2.343,"5min":2.456,"15min":1.8759999999999999},"min":{"1min":0.47199999999999998,"5min":0.35899999999999999,"15min":0.34399999999999997},"max":{"1min":6.1120000000000001,"5min":19.555,"15min":19.555},"last":0.67000000000000004}]},{"osd":10,"last update":"Fri Feb 9 18:02:16 2024","interfaces":[{"interface":"back","average":{"1min":1.718,"5min":2.9359999999999999,"15min":2.2749999999999999},"min":{"1min":0.46600000000000003,"5min":0.45000000000000001,"15min":0.32700000000000001},"max":{"1min":4.6980000000000004,"5min":25.093,"15min":25.093},"last":0.60899999999999999},{"interface":"front","average":{"1min":1.7789999999999999,"5min":2.508,"15min":1.984},"min":{"1min":0.51700000000000002,"5min":0.48999999999999999,"15min":0.35999999999999999},"max":{"1min":4.7249999999999996,"5min":14.836,"15min":20.233000000000001},"last":0.69499999999999995}]}]},{"osd":10,"up_from":84,"seq":360777253353,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3092136,"kb_used_data":2230556,"kb_used_omap":1,"kb_used_meta":861374,"kb_avail":90653016,"statfs":{"total":95995035648,"available":92828688384,"internally_reserved":0,"allocated":2284089344,"data_stored":2283800128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":882047424},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 18:02:05 2024","interfaces":[{"interface":"back","average":{"1min":1.804,"5min":1.8300000000000001,"15min":1.6100000000000001},"min":{"1min":0.29099999999999998,"5min":0.26800000000000002,"15min":0.23999999999999999},"max":{"1min":5.5460000000000003,"5min":12.029999999999999,"15min":12.029999999999999},"last":2.1509999999999998},{"interface":"front","average":{"1min":1.8049999999999999,"5min":1.9530000000000001,"15min":1.667},"min":{"1min":0.27200000000000002,"5min":0.254,"15min":0.254},"max":{"1min":6.6710000000000003,"5min":15.802,"15min":15.802},"last":2.2210000000000001}]},{"osd":1,"last update":"Fri Feb 9 18:01:25 2024","interfaces":[{"interface":"back","average":{"1min":4.4770000000000003,"5min":2.6190000000000002,"15min":1.917},"min":{"1min":0.33000000000000002,"5min":0.254,"15min":0.254},"max":{"1min":52.210999999999999,"5min":52.210999999999999,"15min":52.210999999999999},"last":0.247},{"interface":"front","average":{"1min":4.5640000000000001,"5min":2.8700000000000001,"15min":2.0019999999999998},"min":{"1min":0.29299999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":51.537999999999997,"5min":51.537999999999997,"15min":51.537999999999997},"last":0.34499999999999997}]},{"osd":2,"last update":"Fri Feb 9 18:01:25 2024","interfaces":[{"interface":"back","average":{"1min":1.5529999999999999,"5min":2.3599999999999999,"15min":2.0030000000000001},"min":{"1min":0.38500000000000001,"5min":0.30099999999999999,"15min":0.26600000000000001},"max":{"1min":5.2729999999999997,"5min":22.652999999999999,"15min":22.652999999999999},"last":2.1749999999999998},{"interface":"front","average":{"1min":1.4750000000000001,"5min":2.1389999999999998,"15min":2.0899999999999999},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.25600000000000001},"max":{"1min":5.4210000000000003,"5min":12.180999999999999,"15min":38.868000000000002},"last":2.2970000000000002}]},{"osd":3,"last update":"Fri Feb 9 18:02:08 2024","interfaces":[{"interface":"back","average":{"1min":1.8340000000000001,"5min":1.5029999999999999,"15min":1.448},"min":{"1min":0.36799999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":6.6390000000000002,"5min":6.6390000000000002,"15min":6.6390000000000002},"last":0.311},{"interface":"front","average":{"1min":1.139,"5min":1.204,"15min":1.2150000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":6.1159999999999997,"5min":6.1159999999999997,"15min":6.1159999999999997},"last":1.931}]},{"osd":4,"last update":"Fri Feb 9 18:01:55 2024","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.448,"15min":1.2490000000000001},"min":{"1min":0.35999999999999999,"5min":0.253,"15min":0.253},"max":{"1min":3.859,"5min":13.446,"15min":13.446},"last":2.04},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.494,"15min":1.345},"min":{"1min":0.44900000000000001,"5min":0.34200000000000003,"15min":0.33600000000000002},"max":{"1min":5.367,"5min":12.097,"15min":12.097},"last":2.2370000000000001}]},{"osd":5,"last update":"Fri Feb 9 18:01:25 2024","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.5069999999999999,"15min":1.3440000000000001},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":3.806,"5min":13.4,"15min":13.4},"last":2.0960000000000001},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.6839999999999999,"15min":1.448},"min":{"1min":0.374,"5min":0.315,"15min":0.315},"max":{"1min":5.3140000000000001,"5min":12.157999999999999,"15min":12.157999999999999},"last":2.262}]},{"osd":6,"last update":"Fri Feb 9 18:01:39 2024","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.6180000000000001,"15min":1.4119999999999999},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.32300000000000001},"max":{"1min":3.8330000000000002,"5min":13.507,"15min":13.507},"last":2.2010000000000001},{"interface":"front","average":{"1min":1.7310000000000001,"5min":1.8580000000000001,"15min":1.5820000000000001},"min":{"1min":0.499,"5min":0.38200000000000001,"15min":0.32600000000000001},"max":{"1min":6.7350000000000003,"5min":12.205,"15min":12.414},"last":1.9770000000000001}]},{"osd":7,"last update":"Fri Feb 9 18:01:25 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.623,"15min":2.218},"min":{"1min":0.37,"5min":0.35399999999999998,"15min":0.34000000000000002},"max":{"1min":3.8860000000000001,"5min":13.481,"15min":13.481},"last":2.0089999999999999},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.865,"15min":2.3170000000000002},"min":{"1min":0.38200000000000001,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":5.883,"5min":13.388999999999999,"15min":13.388999999999999},"last":0.38100000000000001}]},{"osd":8,"last update":"Fri Feb 9 18:01:25 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":3.0950000000000002,"15min":1.8899999999999999},"min":{"1min":0.39100000000000001,"5min":0.38100000000000001,"15min":0.31},"max":{"1min":5.21,"5min":22.559000000000001,"15min":22.559000000000001},"last":0.40999999999999998},{"interface":"front","average":{"1min":1.3200000000000001,"5min":2.4239999999999999,"15min":1.736},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.30599999999999999},"max":{"1min":3.7759999999999998,"5min":22.824999999999999,"15min":22.824999999999999},"last":2.0670000000000002}]},{"osd":9,"last update":"Fri Feb 9 18:01:27 2024","interfaces":[{"interface":"back","average":{"1min":1.841,"5min":2.9590000000000001,"15min":2.1200000000000001},"min":{"1min":0.36299999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":6.4050000000000002,"5min":23.201000000000001,"15min":23.201000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":1.98,"5min":2.802,"15min":2.0830000000000002},"min":{"1min":0.45800000000000002,"5min":0.39800000000000002,"15min":0.36099999999999999},"max":{"1min":6.2460000000000004,"5min":17.498999999999999,"15min":17.498999999999999},"last":2.2799999999999998}]},{"osd":11,"last update":"Fri Feb 9 18:02:05 2024","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":2.153,"15min":1.764},"min":{"1min":0.53900000000000003,"5min":0.46300000000000002,"15min":0.35399999999999998},"max":{"1min":5.5860000000000003,"5min":21.18,"15min":23.579000000000001},"last":5.4340000000000002},{"interface":"front","average":{"1min":1.6200000000000001,"5min":2.4390000000000001,"15min":1.903},"min":{"1min":0.56999999999999995,"5min":0.48499999999999999,"15min":0.35899999999999999},"max":{"1min":5.617,"5min":32.707000000000001,"15min":32.707000000000001},"last":5.4649999999999999}]}]},{"osd":9,"up_from":77,"seq":330712482283,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3659416,"kb_used_data":2717520,"kb_used_omap":1,"kb_used_meta":941886,"kb_avail":90085736,"statfs":{"total":95995035648,"available":92247793664,"internally_reserved":0,"allocated":2782740480,"data_stored":2782455721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":964491712},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 18:01:31 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.198,"15min":1.1879999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":5.5389999999999997,"5min":11.515000000000001,"15min":11.515000000000001},"last":4.9619999999999997},{"interface":"front","average":{"1min":1.474,"5min":1.0149999999999999,"15min":0.93899999999999995},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":3.5899999999999999,"5min":6.7400000000000002,"15min":6.7400000000000002},"last":0.70899999999999996}]},{"osd":1,"last update":"Fri Feb 9 18:01:13 2024","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.595,"15min":1.712},"min":{"1min":0.26700000000000002,"5min":0.247,"15min":0.23599999999999999},"max":{"1min":3.4399999999999999,"5min":8.3190000000000008,"15min":61.031999999999996},"last":0.52100000000000002},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.7929999999999999,"15min":2.0049999999999999},"min":{"1min":0.316,"5min":0.25600000000000001,"15min":0.25},"max":{"1min":5.8019999999999996,"5min":8.4359999999999999,"15min":61.970999999999997},"last":0.47099999999999997}]},{"osd":2,"last update":"Fri Feb 9 17:58:45 2024","interfaces":[{"interface":"back","average":{"1min":2.286,"5min":1.71,"15min":1.8200000000000001},"min":{"1min":0.42499999999999999,"5min":0.27000000000000002,"15min":0.26900000000000002},"max":{"1min":8.4410000000000007,"5min":9.8689999999999998,"15min":10.657999999999999},"last":1.0329999999999999},{"interface":"front","average":{"1min":2.7839999999999998,"5min":2.2269999999999999,"15min":1.806},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.248},"max":{"1min":7.1449999999999996,"5min":11.35,"15min":11.35},"last":1.1240000000000001}]},{"osd":3,"last update":"Fri Feb 9 18:01:13 2024","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":0.96099999999999997,"15min":0.94699999999999995},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":2.8340000000000001,"5min":3.5710000000000002,"15min":3.5710000000000002},"last":5.0369999999999999},{"interface":"front","average":{"1min":1.1299999999999999,"5min":0.94899999999999995,"15min":0.91900000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":2.855,"5min":4.7850000000000001,"15min":4.7850000000000001},"last":2.6539999999999999}]},{"osd":4,"last update":"Fri Feb 9 18:01:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.6459999999999999,"15min":1.518},"min":{"1min":0.313,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":3.9900000000000002,"5min":10.725,"15min":19.414000000000001},"last":4.9189999999999996},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.27,"15min":1.327},"min":{"1min":0.38200000000000001,"5min":0.29999999999999999,"15min":0.26900000000000002},"max":{"1min":2.714,"5min":8.3599999999999994,"15min":19.224},"last":0.49099999999999999}]},{"osd":5,"last update":"Fri Feb 9 18:01:24 2024","interfaces":[{"interface":"back","average":{"1min":1.5069999999999999,"5min":1.6579999999999999,"15min":1.752},"min":{"1min":0.40400000000000003,"5min":0.33200000000000002,"15min":0.30299999999999999},"max":{"1min":6.0940000000000003,"5min":10.763999999999999,"15min":77.924000000000007},"last":0.36299999999999999},{"interface":"front","average":{"1min":1.393,"5min":1.8380000000000001,"15min":1.7010000000000001},"min":{"1min":0.36099999999999999,"5min":0.316,"15min":0.28000000000000003},"max":{"1min":3.0979999999999999,"5min":10.851000000000001,"15min":77.899000000000001},"last":0.442}]},{"osd":6,"last update":"Fri Feb 9 18:01:13 2024","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":1.5920000000000001,"15min":1.425},"min":{"1min":0.432,"5min":0.36599999999999999,"15min":0.314},"max":{"1min":2.7229999999999999,"5min":10.071,"15min":19.375},"last":0.34499999999999997},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.681,"15min":1.5409999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33500000000000002},"max":{"1min":2.778,"5min":9.9900000000000002,"15min":19.178000000000001},"last":0.42299999999999999}]},{"osd":7,"last update":"Fri Feb 9 18:01:53 2024","interfaces":[{"interface":"back","average":{"1min":2.1819999999999999,"5min":1.762,"15min":2.0190000000000001},"min":{"1min":0.378,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":8.8529999999999998,"5min":10.021000000000001,"15min":10.021000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":1.79,"5min":2.0750000000000002,"15min":1.996},"min":{"1min":0.39900000000000002,"5min":0.379,"15min":0.379},"max":{"1min":7.008,"5min":11.138,"15min":11.138},"last":0.59999999999999998}]},{"osd":8,"last update":"Fri Feb 9 18:01:31 2024","interfaces":[{"interface":"back","average":{"1min":1.9990000000000001,"5min":2.0899999999999999,"15min":1.52},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.29599999999999999},"max":{"1min":8.9350000000000005,"5min":10.210000000000001,"15min":10.744},"last":0.50900000000000001},{"interface":"front","average":{"1min":2.044,"5min":2.1699999999999999,"15min":1.571},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.32200000000000001},"max":{"1min":8.9550000000000001,"5min":10.237,"15min":10.768000000000001},"last":0.56200000000000006}]},{"osd":10,"last update":"Fri Feb 9 18:01:41 2024","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":2.5880000000000001,"15min":2.0049999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":7.093,"5min":8.6969999999999992,"15min":20.773},"last":5.0700000000000003},{"interface":"front","average":{"1min":1.899,"5min":3.1160000000000001,"15min":1.9890000000000001},"min":{"1min":0.54600000000000004,"5min":0.40000000000000002,"15min":0.34699999999999998},"max":{"1min":5.673,"5min":18.579999999999998,"15min":20.637},"last":4.3929999999999998}]},{"osd":11,"last update":"Fri Feb 9 18:01:41 2024","interfaces":[{"interface":"back","average":{"1min":1.464,"5min":2.6869999999999998,"15min":1.9430000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.33000000000000002},"max":{"1min":3.7309999999999999,"5min":40.259999999999998,"15min":40.259999999999998},"last":0.55000000000000004},{"interface":"front","average":{"1min":1.5149999999999999,"5min":3.1469999999999998,"15min":1.9910000000000001},"min":{"1min":0.55400000000000005,"5min":0.40400000000000003,"15min":0.36499999999999999},"max":{"1min":3.7829999999999999,"5min":39.683,"15min":39.683},"last":0.57399999999999995}]}]},{"osd":8,"up_from":171,"seq":734439408026,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1410776,"kb_used_data":900304,"kb_used_omap":212,"kb_used_meta":510251,"kb_avail":92334376,"statfs":{"total":95995035648,"available":94550401024,"internally_reserved":0,"allocated":921911296,"data_stored":921626537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217579,"internal_metadata":522497557},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 18:01:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":1.8440000000000001,"15min":1.6579999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.22600000000000001},"max":{"1min":2.077,"5min":10.099,"15min":12.712999999999999},"last":0.74299999999999999},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.4910000000000001,"15min":1.516},"min":{"1min":0.29699999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":4.3129999999999997,"5min":12.178000000000001,"15min":12.725},"last":5.3250000000000002}]},{"osd":1,"last update":"Fri Feb 9 18:01:22 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.52,"15min":1.5209999999999999},"min":{"1min":0.29199999999999998,"5min":0.248,"15min":0.23799999999999999},"max":{"1min":3.7890000000000001,"5min":6.8970000000000002,"15min":8.2550000000000008},"last":1.5049999999999999},{"interface":"front","average":{"1min":1.526,"5min":1.6850000000000001,"15min":1.7490000000000001},"min":{"1min":0.30199999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":3.9790000000000001,"5min":12.262,"15min":21.399000000000001},"last":0.66500000000000004}]},{"osd":2,"last update":"Fri Feb 9 18:01:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4990000000000001,"5min":1.851,"15min":1.349},"min":{"1min":0.34899999999999998,"5min":0.25,"15min":0.25},"max":{"1min":4.0890000000000004,"5min":6.9260000000000002,"15min":6.9260000000000002},"last":0.379},{"interface":"front","average":{"1min":1.8660000000000001,"5min":1.873,"15min":1.7909999999999999},"min":{"1min":0.34300000000000003,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":4.1479999999999997,"5min":12.465999999999999,"15min":12.465999999999999},"last":1.2829999999999999}]},{"osd":3,"last update":"Fri Feb 9 17:56:20 2024","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":2.552,"15min":2.8039999999999998},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":5.2560000000000002,"5min":15.654,"15min":15.654},"last":6.5350000000000001},{"interface":"front","average":{"1min":1.306,"5min":2.3149999999999999,"15min":2.0150000000000001},"min":{"1min":0.34300000000000003,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":5.2759999999999998,"5min":15.609999999999999,"15min":15.609999999999999},"last":6.5599999999999996}]},{"osd":4,"last update":"Fri Feb 9 18:01:22 2024","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.593,"15min":1.2509999999999999},"min":{"1min":0.32500000000000001,"5min":0.29499999999999998,"15min":0.29399999999999998},"max":{"1min":4,"5min":12.297000000000001,"15min":12.297000000000001},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.89000000000000001,"5min":1.5369999999999999,"15min":1.3759999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28000000000000003},"max":{"1min":3.9060000000000001,"5min":12.227,"15min":12.227},"last":0.39700000000000002}]},{"osd":5,"last update":"Fri Feb 9 18:01:53 2024","interfaces":[{"interface":"back","average":{"1min":1.696,"5min":1.661,"15min":1.526},"min":{"1min":0.376,"5min":0.307,"15min":0.27400000000000002},"max":{"1min":4.0179999999999998,"5min":12.4,"15min":12.4},"last":0.41999999999999998},{"interface":"front","average":{"1min":1.734,"5min":1.6739999999999999,"15min":1.4430000000000001},"min":{"1min":0.46500000000000002,"5min":0.35499999999999998,"15min":0.30299999999999999},"max":{"1min":4.0549999999999997,"5min":12.368,"15min":12.368},"last":0.61099999999999999}]},{"osd":6,"last update":"Fri Feb 9 18:01:22 2024","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.4690000000000001,"15min":1.5620000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.29899999999999999},"max":{"1min":3.8730000000000002,"5min":6.7599999999999998,"15min":9.7539999999999996},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.66000000000000003,"5min":1.7929999999999999,"15min":1.512},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.9379999999999999,"5min":10.178000000000001,"15min":10.178000000000001},"last":0.44}]},{"osd":7,"last update":"Fri Feb 9 18:01:22 2024","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.9390000000000001,"15min":1.7110000000000001},"min":{"1min":0.42599999999999999,"5min":0.35999999999999999,"15min":0.32900000000000001},"max":{"1min":4.1050000000000004,"5min":12.380000000000001,"15min":12.380000000000001},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.80800000000000005,"5min":1.7450000000000001,"15min":1.591},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.309},"max":{"1min":2.036,"5min":10.148999999999999,"15min":10.148999999999999},"last":0.47899999999999998}]},{"osd":9,"last update":"Fri Feb 9 18:01:22 2024","interfaces":[{"interface":"back","average":{"1min":2.8140000000000001,"5min":3.4159999999999999,"15min":2.1989999999999998},"min":{"1min":0.45700000000000002,"5min":0.34999999999999998,"15min":0.26800000000000002},"max":{"1min":17.623000000000001,"5min":46.213999999999999,"15min":46.213999999999999},"last":0.45700000000000002},{"interface":"front","average":{"1min":2.0299999999999998,"5min":3.496,"15min":2.286},"min":{"1min":0.41299999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":13.67,"5min":41.177,"15min":41.177},"last":0.53600000000000003}]},{"osd":10,"last update":"Fri Feb 9 18:01:22 2024","interfaces":[{"interface":"back","average":{"1min":1.9710000000000001,"5min":3.0270000000000001,"15min":1.869},"min":{"1min":0.432,"5min":0.39600000000000002,"15min":0.32300000000000001},"max":{"1min":18.164000000000001,"5min":18.164000000000001,"15min":18.164000000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":2.1269999999999998,"5min":2.6970000000000001,"15min":1.9770000000000001},"min":{"1min":0.45100000000000001,"5min":0.40200000000000002,"15min":0.34300000000000003},"max":{"1min":18.111000000000001,"5min":18.111000000000001,"15min":18.111000000000001},"last":0.54900000000000004}]},{"osd":11,"last update":"Fri Feb 9 18:01:53 2024","interfaces":[{"interface":"back","average":{"1min":2.089,"5min":2.3559999999999999,"15min":2.085},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":5.8380000000000001,"5min":10.231999999999999,"15min":10.231999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":2.1309999999999998,"5min":2.3980000000000001,"15min":2.1240000000000001},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":5.8540000000000001,"5min":10.256,"15min":10.256},"last":0.59599999999999997}]}]},{"osd":7,"up_from":57,"seq":244813136372,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3131428,"kb_used_data":2274204,"kb_used_omap":1,"kb_used_meta":857214,"kb_avail":90613724,"statfs":{"total":95995035648,"available":92788453376,"internally_reserved":0,"allocated":2328784896,"data_stored":2328492513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":877787594},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 18:02:00 2024","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.754,"15min":1.4590000000000001},"min":{"1min":0.316,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":5.3550000000000004,"5min":6.1420000000000003,"15min":101.167},"last":0.56299999999999994},{"interface":"front","average":{"1min":1.2230000000000001,"5min":0.96499999999999997,"15min":1.4570000000000001},"min":{"1min":0.35299999999999998,"5min":0.28499999999999998,"15min":0.26800000000000002},"max":{"1min":5.3840000000000003,"5min":15.890000000000001,"15min":101.07599999999999},"last":3.5569999999999999}]},{"osd":1,"last update":"Fri Feb 9 18:02:00 2024","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.04,"15min":1.375},"min":{"1min":0.29399999999999998,"5min":0.28499999999999998,"15min":0.28299999999999997},"max":{"1min":3.3159999999999998,"5min":8.7050000000000001,"15min":17.314},"last":3.0569999999999999},{"interface":"front","average":{"1min":1.387,"5min":1.1180000000000001,"15min":1.3400000000000001},"min":{"1min":0.30299999999999999,"5min":0.246,"15min":0.246},"max":{"1min":4.0330000000000004,"5min":15.935,"15min":15.935},"last":3.593}]},{"osd":2,"last update":"Fri Feb 9 18:02:00 2024","interfaces":[{"interface":"back","average":{"1min":2.0190000000000001,"5min":1.4430000000000001,"15min":1.4510000000000001},"min":{"1min":0.42099999999999999,"5min":0.27300000000000002,"15min":0.25},"max":{"1min":10.602,"5min":16.190000000000001,"15min":16.190000000000001},"last":3.6219999999999999},{"interface":"front","average":{"1min":1.5049999999999999,"5min":1.3280000000000001,"15min":1.4990000000000001},"min":{"1min":0.35999999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":4.0019999999999998,"5min":15.965999999999999,"15min":15.965999999999999},"last":3.73}]},{"osd":3,"last update":"Fri Feb 9 18:01:14 2024","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":1.175,"15min":1.2050000000000001},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":3.4449999999999998,"5min":7.0270000000000001,"15min":7.0270000000000001},"last":3.02},{"interface":"front","average":{"1min":1.1539999999999999,"5min":2.0230000000000001,"15min":2.1680000000000001},"min":{"1min":0.498,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":3.3260000000000001,"5min":16.029,"15min":16.029},"last":6.8099999999999996}]},{"osd":4,"last update":"Fri Feb 9 18:02:00 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.63700000000000001,"15min":0.83299999999999996},"min":{"1min":0.34000000000000002,"5min":0.27100000000000002,"15min":0.25600000000000001},"max":{"1min":3.7210000000000001,"5min":4.3330000000000002,"15min":7.2999999999999998},"last":0.52700000000000002},{"interface":"front","average":{"1min":1.0920000000000001,"5min":0.88200000000000001,"15min":1.0409999999999999},"min":{"1min":0.379,"5min":0.26600000000000001,"15min":0.23899999999999999},"max":{"1min":5.6539999999999999,"5min":17.814,"15min":17.814},"last":0.64700000000000002}]},{"osd":5,"last update":"Fri Feb 9 18:02:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":0.72599999999999998,"15min":1.0049999999999999},"min":{"1min":0.315,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":3.4889999999999999,"5min":6.944,"15min":8.0060000000000002},"last":0.80600000000000005},{"interface":"front","average":{"1min":1.145,"5min":0.82799999999999996,"15min":1.0569999999999999},"min":{"1min":0.35099999999999998,"5min":0.32900000000000001,"15min":0.32100000000000001},"max":{"1min":4.1980000000000004,"5min":6.9080000000000004,"15min":7.867},"last":2.9660000000000002}]},{"osd":6,"last update":"Fri Feb 9 18:01:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":0.995,"15min":1.1559999999999999},"min":{"1min":0.51300000000000001,"5min":0.36099999999999999,"15min":0.311},"max":{"1min":3.52,"5min":16.158000000000001,"15min":16.158000000000001},"last":3.649},{"interface":"front","average":{"1min":0.97899999999999998,"5min":0.79900000000000004,"15min":1.0349999999999999},"min":{"1min":0.42199999999999999,"5min":0.39500000000000002,"15min":0.32000000000000001},"max":{"1min":3.5190000000000001,"5min":7.0810000000000004,"15min":7.1669999999999998},"last":3.0979999999999999}]},{"osd":8,"last update":"Fri Feb 9 18:01:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.7310000000000001,"15min":1.482},"min":{"1min":0.44400000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":2.9740000000000002,"5min":11.648999999999999,"15min":11.648999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.79,"15min":1.4430000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.37},"max":{"1min":6.4589999999999996,"5min":11.629,"15min":11.629},"last":10.818}]},{"osd":9,"last update":"Fri Feb 9 18:02:10 2024","interfaces":[{"interface":"back","average":{"1min":1.5369999999999999,"5min":1.7090000000000001,"15min":1.631},"min":{"1min":0.46300000000000002,"5min":0.43099999999999999,"15min":0.375},"max":{"1min":5.6269999999999998,"5min":8.0169999999999995,"15min":11.542999999999999},"last":3.069},{"interface":"front","average":{"1min":2.4910000000000001,"5min":2.1600000000000001,"15min":1.788},"min":{"1min":0.49199999999999999,"5min":0.46500000000000002,"15min":0.40000000000000002},"max":{"1min":7.3040000000000003,"5min":13.161,"15min":15.852},"last":0.61899999999999999}]},{"osd":10,"last update":"Fri Feb 9 18:01:44 2024","interfaces":[{"interface":"back","average":{"1min":2.4950000000000001,"5min":2.0179999999999998,"15min":1.54},"min":{"1min":0.60299999999999998,"5min":0.44900000000000001,"15min":0.32800000000000001},"max":{"1min":8.7669999999999995,"5min":15.999000000000001,"15min":15.999000000000001},"last":5.8399999999999999},{"interface":"front","average":{"1min":2.0289999999999999,"5min":1.764,"15min":1.7569999999999999},"min":{"1min":0.55800000000000005,"5min":0.53700000000000003,"15min":0.40400000000000003},"max":{"1min":4.9400000000000004,"5min":8.5020000000000007,"15min":11.571999999999999},"last":0.68100000000000005}]},{"osd":11,"last update":"Fri Feb 9 18:02:00 2024","interfaces":[{"interface":"back","average":{"1min":1.6359999999999999,"5min":1.964,"15min":1.5469999999999999},"min":{"1min":0.54800000000000004,"5min":0.47299999999999998,"15min":0.35999999999999999},"max":{"1min":4.0679999999999996,"5min":16.111999999999998,"15min":16.111999999999998},"last":6.5140000000000002},{"interface":"front","average":{"1min":1.996,"5min":1.857,"15min":1.651},"min":{"1min":0.57099999999999995,"5min":0.53200000000000003,"15min":0.438},"max":{"1min":4.3819999999999997,"5min":7.0030000000000001,"15min":12.243},"last":3.21}]}]},{"osd":6,"up_from":49,"seq":210453398011,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1586468,"kb_used_data":873628,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":92158684,"statfs":{"total":95995035648,"available":94370492416,"internally_reserved":0,"allocated":894595072,"data_stored":894305856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":729938374},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 18:01:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69199999999999995,"15min":1.133},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":3.4260000000000002,"5min":4.5270000000000001,"15min":10.481999999999999},"last":1.476},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.66300000000000003,"15min":1.1890000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.24399999999999999},"max":{"1min":3.5539999999999998,"5min":3.5539999999999998,"15min":10.436},"last":4.9400000000000004}]},{"osd":1,"last update":"Fri Feb 9 18:01:48 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.95099999999999996,"15min":1.7549999999999999},"min":{"1min":0.32000000000000001,"5min":0.29699999999999999,"15min":0.25},"max":{"1min":3.5710000000000002,"5min":4.7069999999999999,"15min":79.945999999999998},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.90700000000000003,"5min":1.0069999999999999,"15min":1.806},"min":{"1min":0.309,"5min":0.309,"15min":0.28599999999999998},"max":{"1min":3.5880000000000001,"5min":4.7300000000000004,"15min":80.003},"last":0.374}]},{"osd":2,"last update":"Fri Feb 9 18:01:48 2024","interfaces":[{"interface":"back","average":{"1min":1.5700000000000001,"5min":1.147,"15min":1.8200000000000001},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.28100000000000003},"max":{"1min":8.141,"5min":9.2970000000000006,"15min":84.912999999999997},"last":4.8579999999999997},{"interface":"front","average":{"1min":1.6120000000000001,"5min":1.204,"15min":1.8700000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.34000000000000002},"max":{"1min":8.1709999999999994,"5min":9.3260000000000005,"15min":84.942999999999998},"last":4.8819999999999997}]},{"osd":3,"last update":"Fri Feb 9 18:01:32 2024","interfaces":[{"interface":"back","average":{"1min":1.629,"5min":0.89300000000000002,"15min":0.77100000000000002},"min":{"1min":0.40000000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":5.6479999999999997,"5min":5.6479999999999997,"15min":5.6479999999999997},"last":5.181},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.153,"15min":1.173},"min":{"1min":0.34300000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":4.0490000000000004,"5min":9.0909999999999993,"15min":9.0909999999999993},"last":0.38700000000000001}]},{"osd":4,"last update":"Fri Feb 9 18:01:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1279999999999999,"5min":0.64600000000000002,"15min":1.0960000000000001},"min":{"1min":0.29199999999999998,"5min":0.27300000000000002,"15min":0.26200000000000001},"max":{"1min":5.6840000000000002,"5min":5.6840000000000002,"15min":14.182},"last":0.36099999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.70899999999999996,"15min":1.1579999999999999},"min":{"1min":0.35499999999999998,"5min":0.34100000000000003,"15min":0.314},"max":{"1min":5.7030000000000003,"5min":5.7030000000000003,"15min":14.210000000000001},"last":0.40100000000000002}]},{"osd":5,"last update":"Fri Feb 9 18:01:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.57199999999999995,"15min":1.1259999999999999},"min":{"1min":0.30599999999999999,"5min":0.30299999999999999,"15min":0.26600000000000001},"max":{"1min":4.4130000000000003,"5min":4.4130000000000003,"15min":14.151999999999999},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.66200000000000003,"15min":1.129},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.26100000000000001},"max":{"1min":3.468,"5min":3.5569999999999999,"15min":15.509},"last":0.41699999999999998}]},{"osd":7,"last update":"Fri Feb 9 18:01:24 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.65100000000000002,"15min":1.2110000000000001},"min":{"1min":0.32600000000000001,"5min":0.31,"15min":0.28699999999999998},"max":{"1min":3.5150000000000001,"5min":3.5150000000000001,"15min":23.265999999999998},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69899999999999995,"15min":1.0940000000000001},"min":{"1min":0.44,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.5209999999999999,"5min":5.4969999999999999,"15min":10.599},"last":0.437}]},{"osd":8,"last update":"Fri Feb 9 17:58:34 2024","interfaces":[{"interface":"back","average":{"1min":2.806,"5min":1.635,"15min":1.27},"min":{"1min":0.41599999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":9.2940000000000005,"5min":9.2940000000000005,"15min":9.9480000000000004},"last":0.40300000000000002},{"interface":"front","average":{"1min":2.3780000000000001,"5min":1.623,"15min":1.3109999999999999},"min":{"1min":0.53400000000000003,"5min":0.41399999999999998,"15min":0.40400000000000003},"max":{"1min":9.3469999999999995,"5min":9.3469999999999995,"15min":11.316000000000001},"last":0.57899999999999996}]},{"osd":9,"last update":"Fri Feb 9 18:01:55 2024","interfaces":[{"interface":"back","average":{"1min":1.6220000000000001,"5min":1.5489999999999999,"15min":1.6060000000000001},"min":{"1min":0.53800000000000003,"5min":0.47799999999999998,"15min":0.32600000000000001},"max":{"1min":5.1820000000000004,"5min":7.6669999999999998,"15min":36.606000000000002},"last":0.46100000000000002},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.8200000000000001,"15min":1.843},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.34300000000000003},"max":{"1min":3.5329999999999999,"5min":8.0269999999999992,"15min":34.408000000000001},"last":2.5699999999999998}]},{"osd":10,"last update":"Fri Feb 9 18:01:24 2024","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.829,"15min":1.728},"min":{"1min":0.37,"5min":0.37,"15min":0.34399999999999997},"max":{"1min":5.8920000000000003,"5min":9.8460000000000001,"15min":21.364999999999998},"last":0.48199999999999998},{"interface":"front","average":{"1min":1.298,"5min":1.861,"15min":1.8440000000000001},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.40100000000000002},"max":{"1min":5.9189999999999996,"5min":9.7989999999999995,"15min":21.408999999999999},"last":0.50800000000000001}]},{"osd":11,"last update":"Fri Feb 9 18:01:48 2024","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.73,"15min":1.766},"min":{"1min":0.48899999999999999,"5min":0.48399999999999999,"15min":0.372},"max":{"1min":4.0979999999999999,"5min":6.9809999999999999,"15min":35.912999999999997},"last":0.52600000000000002},{"interface":"front","average":{"1min":1.278,"5min":1.7689999999999999,"15min":1.827},"min":{"1min":0.53500000000000003,"5min":0.51900000000000002,"15min":0.39700000000000002},"max":{"1min":4.1779999999999999,"5min":7.0099999999999998,"15min":36.156999999999996},"last":0.54900000000000004}]}]},{"osd":1,"up_from":13,"seq":55834575379,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4645800,"kb_used_data":3509536,"kb_used_omap":1,"kb_used_meta":1136254,"kb_avail":89099352,"statfs":{"total":95995035648,"available":91237736448,"internally_reserved":0,"allocated":3593764864,"data_stored":3593471576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1163524549},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 18:02:08 2024","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.23,"15min":1.6439999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.19},"max":{"1min":5.218,"5min":13.068,"15min":14.885999999999999},"last":0.35499999999999998},{"interface":"front","average":{"1min":1.5369999999999999,"5min":1.3149999999999999,"15min":1.8879999999999999},"min":{"1min":0.32400000000000001,"5min":0.30199999999999999,"15min":0.214},"max":{"1min":5.2460000000000004,"5min":7.5780000000000003,"15min":33.389000000000003},"last":0.46899999999999997}]},{"osd":2,"last update":"Fri Feb 9 18:01:25 2024","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.3859999999999999,"15min":1.7749999999999999},"min":{"1min":0.35799999999999998,"5min":0.28799999999999998,"15min":0.222},"max":{"1min":6.5060000000000002,"5min":10.510999999999999,"15min":14.83},"last":2.6669999999999998},{"interface":"front","average":{"1min":1.3260000000000001,"5min":1.234,"15min":1.5760000000000001},"min":{"1min":0.38700000000000001,"5min":0.28100000000000003,"15min":0.222},"max":{"1min":3.9060000000000001,"5min":7.6070000000000002,"15min":11.103},"last":0.39800000000000002}]},{"osd":3,"last update":"Fri Feb 9 18:02:17 2024","interfaces":[{"interface":"back","average":{"1min":2.0089999999999999,"5min":0.98299999999999998,"15min":0.78200000000000003},"min":{"1min":0.32500000000000001,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":7.5819999999999999,"5min":7.5819999999999999,"15min":7.5819999999999999},"last":0.70699999999999996},{"interface":"front","average":{"1min":2.5379999999999998,"5min":1.3979999999999999,"15min":1.258},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":10.494999999999999,"5min":10.494999999999999,"15min":10.494999999999999},"last":2.7829999999999999}]},{"osd":4,"last update":"Fri Feb 9 18:02:10 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.0569999999999999,"15min":1.2669999999999999},"min":{"1min":0.32500000000000001,"5min":0.27600000000000002,"15min":0.26900000000000002},"max":{"1min":5.0890000000000004,"5min":5.0890000000000004,"15min":9.032},"last":0.48799999999999999},{"interface":"front","average":{"1min":1.6879999999999999,"5min":1.1970000000000001,"15min":1.508},"min":{"1min":0.38100000000000001,"5min":0.28799999999999998,"15min":0.26400000000000001},"max":{"1min":7.4690000000000003,"5min":7.7370000000000001,"15min":15.331},"last":0.60999999999999999}]},{"osd":5,"last update":"Fri Feb 9 18:01:37 2024","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.0580000000000001,"15min":1.448},"min":{"1min":0.34499999999999997,"5min":0.29399999999999998,"15min":0.26000000000000001},"max":{"1min":5.1219999999999999,"5min":7.7839999999999998,"15min":15.106999999999999},"last":4.6459999999999999},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.077,"15min":1.7989999999999999},"min":{"1min":0.34100000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":5.1500000000000004,"5min":5.665,"15min":33.247},"last":4.6299999999999999}]},{"osd":6,"last update":"Fri Feb 9 18:01:54 2024","interfaces":[{"interface":"back","average":{"1min":1.5980000000000001,"5min":1.2450000000000001,"15min":1.77},"min":{"1min":0.375,"5min":0.33100000000000002,"15min":0.24299999999999999},"max":{"1min":5.4610000000000003,"5min":7.8380000000000001,"15min":33.338000000000001},"last":4.5149999999999997},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.2350000000000001,"15min":1.8129999999999999},"min":{"1min":0.39000000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":5.4119999999999999,"5min":7.9199999999999999,"15min":33.286999999999999},"last":1.4750000000000001}]},{"osd":7,"last update":"Fri Feb 9 18:01:55 2024","interfaces":[{"interface":"back","average":{"1min":2.02,"5min":1.4119999999999999,"15min":1.8799999999999999},"min":{"1min":0.40899999999999997,"5min":0.38400000000000001,"15min":0.316},"max":{"1min":6.298,"5min":7.8689999999999998,"15min":33.314999999999998},"last":4.9509999999999996},{"interface":"front","average":{"1min":1.8120000000000001,"5min":1.413,"15min":1.907},"min":{"1min":0.52800000000000002,"5min":0.432,"15min":0.40200000000000002},"max":{"1min":5.4539999999999997,"5min":7.8940000000000001,"15min":33.363999999999997},"last":0.59499999999999997}]},{"osd":8,"last update":"Fri Feb 9 18:02:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6479999999999999,"5min":1.8600000000000001,"15min":1.7849999999999999},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.35599999999999998},"max":{"1min":6.1840000000000002,"5min":10.423,"15min":10.423},"last":0.89100000000000001},{"interface":"front","average":{"1min":2.2149999999999999,"5min":2.2610000000000001,"15min":2.081},"min":{"1min":0.48899999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":7.54,"5min":10.663,"15min":12.69},"last":0.78800000000000003}]},{"osd":9,"last update":"Fri Feb 9 18:02:08 2024","interfaces":[{"interface":"back","average":{"1min":2.3540000000000001,"5min":2.1280000000000001,"15min":2.0720000000000001},"min":{"1min":0.52300000000000002,"5min":0.503,"15min":0.39500000000000002},"max":{"1min":10.378,"5min":10.475,"15min":20.789000000000001},"last":0.66100000000000003},{"interface":"front","average":{"1min":2.339,"5min":1.843,"15min":1.915},"min":{"1min":0.42799999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":10.439,"5min":10.439,"15min":10.439},"last":0.53700000000000003}]},{"osd":10,"last update":"Fri Feb 9 18:01:37 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":2.1120000000000001,"15min":2.214},"min":{"1min":0.57899999999999996,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":5.1630000000000003,"5min":12.151,"15min":33.213999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":1.6879999999999999,"5min":1.958,"15min":2.0779999999999998},"min":{"1min":0.47999999999999998,"5min":0.372,"15min":0.372},"max":{"1min":5.2439999999999998,"5min":9.1859999999999999,"15min":15.271000000000001},"last":0.64300000000000002}]},{"osd":11,"last update":"Fri Feb 9 18:01:54 2024","interfaces":[{"interface":"back","average":{"1min":2.2650000000000001,"5min":2.6349999999999998,"15min":2.5249999999999999},"min":{"1min":0.51000000000000001,"5min":0.378,"15min":0.378},"max":{"1min":7.6260000000000003,"5min":16.881,"15min":19.648},"last":0.63},{"interface":"front","average":{"1min":1.552,"5min":2.3399999999999999,"15min":2.5129999999999999},"min":{"1min":0.54500000000000004,"5min":0.372,"15min":0.372},"max":{"1min":5.8049999999999997,"5min":15.209,"15min":33.161000000000001},"last":0.67500000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738904,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1322020,"kb_used_data":452316,"kb_used_omap":1,"kb_used_meta":869694,"kb_avail":92423132,"statfs":{"total":95995035648,"available":94641287168,"internally_reserved":0,"allocated":463171584,"data_stored":462882368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":890567109},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 9 18:01:45 2024","interfaces":[{"interface":"back","average":{"1min":2.9009999999999998,"5min":1.681,"15min":1.7030000000000001},"min":{"1min":0.317,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":23.282,"5min":23.282,"15min":23.282},"last":6.3339999999999996},{"interface":"front","average":{"1min":2.9590000000000001,"5min":1.754,"15min":1.7749999999999999},"min":{"1min":0.39500000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":23.32,"5min":23.32,"15min":23.32},"last":6.3609999999999998}]},{"osd":2,"last update":"Fri Feb 9 18:02:17 2024","interfaces":[{"interface":"back","average":{"1min":2.177,"5min":1.399,"15min":1.6899999999999999},"min":{"1min":0.308,"5min":0.253,"15min":0.245},"max":{"1min":7.5780000000000003,"5min":10.131,"15min":20.013999999999999},"last":6.2569999999999997},{"interface":"front","average":{"1min":2.6110000000000002,"5min":1.522,"15min":1.677},"min":{"1min":0.29499999999999998,"5min":0.219,"15min":0.219},"max":{"1min":9.5579999999999998,"5min":9.5579999999999998,"15min":20.103999999999999},"last":0.64200000000000002}]},{"osd":3,"last update":"Fri Feb 9 18:02:17 2024","interfaces":[{"interface":"back","average":{"1min":2.6970000000000001,"5min":1.5409999999999999,"15min":1.3340000000000001},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":9.6379999999999999,"5min":9.6379999999999999,"15min":9.6379999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":2.1429999999999998,"5min":1.7609999999999999,"15min":1.8460000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":5.4660000000000002,"5min":11.714,"15min":11.714},"last":0.624}]},{"osd":4,"last update":"Fri Feb 9 18:02:17 2024","interfaces":[{"interface":"back","average":{"1min":2.1749999999999998,"5min":1.196,"15min":1.413},"min":{"1min":0.33600000000000002,"5min":0.29999999999999999,"15min":0.28100000000000003},"max":{"1min":8.9239999999999995,"5min":9.9760000000000009,"15min":12.757},"last":6.3029999999999999},{"interface":"front","average":{"1min":2.0089999999999999,"5min":1.3480000000000001,"15min":1.8149999999999999},"min":{"1min":0.38,"5min":0.28499999999999998,"15min":0.249},"max":{"1min":8.9469999999999992,"5min":11.635,"15min":17.091999999999999},"last":0.74299999999999999}]},{"osd":5,"last update":"Fri Feb 9 18:01:28 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.056,"15min":1.605},"min":{"1min":0.36699999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":5.415,"5min":10.031000000000001,"15min":48.924999999999997},"last":6.4729999999999999},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.1339999999999999,"15min":1.6639999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.35299999999999998},"max":{"1min":5.4710000000000001,"5min":10.06,"15min":48.969999999999999},"last":6.4199999999999999}]},{"osd":6,"last update":"Fri Feb 9 18:01:57 2024","interfaces":[{"interface":"back","average":{"1min":2.613,"5min":1.2509999999999999,"15min":1.5860000000000001},"min":{"1min":0.36199999999999999,"5min":0.35399999999999998,"15min":0.307},"max":{"1min":8.702,"5min":8.702,"15min":15.667},"last":0.56200000000000006},{"interface":"front","average":{"1min":1.411,"5min":1.1140000000000001,"15min":1.6100000000000001},"min":{"1min":0.38,"5min":0.36399999999999999,"15min":0.34799999999999998},"max":{"1min":6.3380000000000001,"5min":6.3380000000000001,"15min":15.702999999999999},"last":0.70499999999999996}]},{"osd":7,"last update":"Fri Feb 9 18:01:28 2024","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.343,"15min":1.6459999999999999},"min":{"1min":0.48899999999999999,"5min":0.307,"15min":0.29199999999999998},"max":{"1min":2.7959999999999998,"5min":11.682,"15min":15.271000000000001},"last":0.76400000000000001},{"interface":"front","average":{"1min":1.306,"5min":1.4810000000000001,"15min":1.786},"min":{"1min":0.49399999999999999,"5min":0.35899999999999999,"15min":0.28299999999999997},"max":{"1min":4.5880000000000001,"5min":11.773999999999999,"15min":17.152999999999999},"last":0.81200000000000006}]},{"osd":8,"last update":"Fri Feb 9 18:01:28 2024","interfaces":[{"interface":"back","average":{"1min":1.657,"5min":1.7809999999999999,"15min":1.9430000000000001},"min":{"1min":0.45600000000000002,"5min":0.33800000000000002,"15min":0.313},"max":{"1min":4.9109999999999996,"5min":11.227,"15min":17.189},"last":0.59899999999999998},{"interface":"front","average":{"1min":1.256,"5min":2.1059999999999999,"15min":1.911},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.36499999999999999},"max":{"1min":3.5910000000000002,"5min":18.600000000000001,"15min":18.600000000000001},"last":0.78500000000000003}]},{"osd":9,"last update":"Fri Feb 9 18:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.7989999999999999,"5min":2.2970000000000002,"15min":2.4199999999999999},"min":{"1min":0.46700000000000003,"5min":0.40999999999999998,"15min":0.39000000000000001},"max":{"1min":4.7380000000000004,"5min":11.801,"15min":24.757000000000001},"last":0.83299999999999996},{"interface":"front","average":{"1min":2.1899999999999999,"5min":2.097,"15min":2.2330000000000001},"min":{"1min":0.53800000000000003,"5min":0.41599999999999998,"15min":0.40899999999999997},"max":{"1min":5.125,"5min":12.759,"15min":24.681999999999999},"last":0.86499999999999999}]},{"osd":10,"last update":"Fri Feb 9 18:01:28 2024","interfaces":[{"interface":"back","average":{"1min":1.78,"5min":1.831,"15min":2.1949999999999998},"min":{"1min":0.49399999999999999,"5min":0.438,"15min":0.41999999999999998},"max":{"1min":4.8789999999999996,"5min":7.8029999999999999,"15min":57.979999999999997},"last":0.89800000000000002},{"interface":"front","average":{"1min":1.9119999999999999,"5min":2.4940000000000002,"15min":2.4489999999999998},"min":{"1min":0.63700000000000001,"5min":0.42099999999999999,"15min":0.40400000000000003},"max":{"1min":9.202,"5min":12.689,"15min":57.606999999999999},"last":0.84899999999999998}]},{"osd":11,"last update":"Fri Feb 9 18:01:57 2024","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":1.9770000000000001,"15min":2.2959999999999998},"min":{"1min":0.49199999999999999,"5min":0.44800000000000001,"15min":0.437},"max":{"1min":7.6609999999999996,"5min":15.698,"15min":58.351999999999997},"last":6.3899999999999997},{"interface":"front","average":{"1min":1.427,"5min":2.028,"15min":2.3380000000000001},"min":{"1min":0.54800000000000004,"5min":0.51800000000000002,"15min":0.441},"max":{"1min":7.7370000000000001,"5min":15.666,"15min":57.314},"last":6.4459999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411853,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":663792,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":93081360,"statfs":{"total":95995035648,"available":95315312640,"internally_reserved":0,"allocated":954368,"data_stored":661600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:58:48 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.53200000000000003,"15min":0.48699999999999999},"min":{"1min":0.23699999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.9619999999999997,"5min":4.9619999999999997,"15min":4.9619999999999997},"last":0.33800000000000002},{"interface":"front","average":{"1min":1.292,"5min":0.81899999999999995,"15min":0.73999999999999999},"min":{"1min":0.33000000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":4.5430000000000001,"5min":4.5430000000000001,"15min":4.5430000000000001},"last":0.47799999999999998}]},{"osd":1,"last update":"Fri Feb 9 18:01:26 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.96499999999999997,"15min":1.9590000000000001},"min":{"1min":0.255,"5min":0.23300000000000001,"15min":0.20799999999999999},"max":{"1min":2.0640000000000001,"5min":7.1600000000000001,"15min":52.646000000000001},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.67500000000000004,"5min":1.042,"15min":1.9610000000000001},"min":{"1min":0.32900000000000001,"5min":0.30099999999999999,"15min":0.248},"max":{"1min":1.9590000000000001,"5min":7.1950000000000003,"15min":52.683999999999997},"last":0.33600000000000002}]},{"osd":3,"last update":"Fri Feb 9 18:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":2.3759999999999999,"15min":3.0419999999999998},"min":{"1min":0.315,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":4.6669999999999998,"5min":29.228000000000002,"15min":29.228000000000002},"last":3.0409999999999999},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.353,"15min":1.4830000000000001},"min":{"1min":0.28499999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":5.5839999999999996,"5min":10.492000000000001,"15min":10.492000000000001},"last":0.624}]},{"osd":4,"last update":"Fri Feb 9 18:01:56 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.73299999999999998,"15min":1.2849999999999999},"min":{"1min":0.35499999999999998,"5min":0.33600000000000002,"15min":0.245},"max":{"1min":4.633,"5min":4.633,"15min":12.398},"last":0.377},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.73999999999999999,"15min":1.321},"min":{"1min":0.42399999999999999,"5min":0.38200000000000001,"15min":0.25700000000000001},"max":{"1min":4.7000000000000002,"5min":4.7000000000000002,"15min":10.888999999999999},"last":0.40300000000000002}]},{"osd":5,"last update":"Fri Feb 9 18:01:48 2024","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":0.83999999999999997,"15min":1.6719999999999999},"min":{"1min":0.34599999999999997,"5min":0.28100000000000003,"15min":0.245},"max":{"1min":5.0049999999999999,"5min":5.0049999999999999,"15min":35.436999999999998},"last":0.49099999999999999},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.1299999999999999,"15min":1.7150000000000001},"min":{"1min":0.32100000000000001,"5min":0.27000000000000002,"15min":0.26900000000000002},"max":{"1min":5.4459999999999997,"5min":11.414,"15min":35.484000000000002},"last":0.68100000000000005}]},{"osd":6,"last update":"Fri Feb 9 18:01:56 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":1.1279999999999999,"15min":1.6890000000000001},"min":{"1min":0.42399999999999999,"5min":0.32300000000000001,"15min":0.28399999999999997},"max":{"1min":2.6819999999999999,"5min":11.465999999999999,"15min":18.518999999999998},"last":2.0059999999999998},{"interface":"front","average":{"1min":0.72699999999999998,"5min":1.1899999999999999,"15min":1.6819999999999999},"min":{"1min":0.38500000000000001,"5min":0.317,"15min":0.28199999999999997},"max":{"1min":1.9430000000000001,"5min":11.598000000000001,"15min":18.481999999999999},"last":2.4359999999999999}]},{"osd":7,"last update":"Fri Feb 9 18:01:26 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.91700000000000004,"15min":1.5509999999999999},"min":{"1min":0.40000000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.897,"5min":7.1109999999999998,"15min":16.666},"last":0.53400000000000003},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.3640000000000001,"15min":1.7250000000000001},"min":{"1min":0.36599999999999999,"5min":0.309,"15min":0.309},"max":{"1min":3.9199999999999999,"5min":11.574,"15min":16.638999999999999},"last":0.55200000000000005}]},{"osd":8,"last update":"Fri Feb 9 18:01:56 2024","interfaces":[{"interface":"back","average":{"1min":1.5549999999999999,"5min":1.4410000000000001,"15min":1.8660000000000001},"min":{"1min":0.47799999999999998,"5min":0.39600000000000002,"15min":0.35499999999999998},"max":{"1min":5.2430000000000003,"5min":10.095000000000001,"15min":12.500999999999999},"last":0.50700000000000001},{"interface":"front","average":{"1min":1.355,"5min":1.73,"15min":2.0489999999999999},"min":{"1min":0.54900000000000004,"5min":0.378,"15min":0.35699999999999998},"max":{"1min":5.2779999999999996,"5min":11.492000000000001,"15min":54.759999999999998},"last":0.58499999999999996}]},{"osd":9,"last update":"Fri Feb 9 18:01:58 2024","interfaces":[{"interface":"back","average":{"1min":1.4850000000000001,"5min":1.5900000000000001,"15min":2.23},"min":{"1min":0.36499999999999999,"5min":0.35799999999999998,"15min":0.318},"max":{"1min":5.4039999999999999,"5min":9.7669999999999995,"15min":53.935000000000002},"last":0.65400000000000003},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.7669999999999999,"15min":2.1970000000000001},"min":{"1min":0.52600000000000002,"5min":0.36399999999999999,"15min":0.34899999999999998},"max":{"1min":2.8399999999999999,"5min":11.522,"15min":54.670999999999999},"last":0.52400000000000002}]},{"osd":10,"last update":"Fri Feb 9 18:01:35 2024","interfaces":[{"interface":"back","average":{"1min":1.968,"5min":1.8999999999999999,"15min":2.04},"min":{"1min":0.5,"5min":0.48799999999999999,"15min":0.33500000000000002},"max":{"1min":7.266,"5min":8.5570000000000004,"15min":54.039999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":2.0049999999999999,"5min":1.929,"15min":2.0720000000000001},"min":{"1min":0.52200000000000002,"5min":0.48799999999999999,"15min":0.42599999999999999},"max":{"1min":7.319,"5min":8.6150000000000002,"15min":54.118000000000002},"last":0.59799999999999998}]},{"osd":11,"last update":"Fri Feb 9 18:01:56 2024","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":2.0790000000000002,"15min":2.3079999999999998},"min":{"1min":0.56200000000000006,"5min":0.44,"15min":0.41999999999999998},"max":{"1min":5.032,"5min":11.548,"15min":54.799999999999997},"last":4.2649999999999997},{"interface":"front","average":{"1min":1.6419999999999999,"5min":1.792,"15min":2.3260000000000001},"min":{"1min":0.46000000000000002,"5min":0.40600000000000003,"15min":0.35499999999999998},"max":{"1min":7.335,"5min":8.8770000000000007,"15min":54.103000000000002},"last":4.3559999999999999}]}]},{"osd":3,"up_from":854,"seq":3667902070824,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2995364,"kb_used_data":2806684,"kb_used_omap":202,"kb_used_meta":188469,"kb_avail":90749788,"statfs":{"total":95995035648,"available":92927782912,"internally_reserved":0,"allocated":2874044416,"data_stored":2873755200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207633,"internal_metadata":192992495},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 18:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.7929999999999999,"5min":1.099,"15min":1.0089999999999999},"min":{"1min":0.253,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":8.4079999999999995,"5min":8.4079999999999995,"15min":8.4079999999999995},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.91800000000000004,"5min":1.3979999999999999,"15min":1.704},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":4.5339999999999998,"5min":17.077999999999999,"15min":17.077999999999999},"last":0.376}]},{"osd":1,"last update":"Fri Feb 9 18:02:10 2024","interfaces":[{"interface":"back","average":{"1min":2.3730000000000002,"5min":1.077,"15min":0.86099999999999999},"min":{"1min":0.27600000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":8.5649999999999995,"5min":8.5649999999999995,"15min":8.5649999999999995},"last":0.41099999999999998},{"interface":"front","average":{"1min":1.734,"5min":0.82599999999999996,"15min":0.67500000000000004},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":8.1579999999999995,"5min":8.1579999999999995,"15min":8.1579999999999995},"last":0.34300000000000003}]},{"osd":2,"last update":"Fri Feb 9 18:02:17 2024","interfaces":[{"interface":"back","average":{"1min":2.5619999999999998,"5min":2.2069999999999999,"15min":2.3460000000000001},"min":{"1min":0.28199999999999997,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":13.098000000000001,"5min":13.098000000000001,"15min":13.098000000000001},"last":0.42999999999999999},{"interface":"front","average":{"1min":2.1669999999999998,"5min":1.361,"15min":1.2609999999999999},"min":{"1min":0.378,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":9.3260000000000005,"5min":9.3260000000000005,"15min":9.3260000000000005},"last":0.45100000000000001}]},{"osd":4,"last update":"Fri Feb 9 18:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":0.92500000000000004,"15min":0.91500000000000004},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":5.2750000000000004,"5min":5.9100000000000001,"15min":5.9100000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":1.286,"5min":1.3029999999999999,"15min":1.421},"min":{"1min":0.33900000000000002,"5min":0.318,"15min":0.318},"max":{"1min":5.3410000000000002,"5min":10.755000000000001,"15min":10.755000000000001},"last":0.35799999999999998}]},{"osd":5,"last update":"Fri Feb 9 18:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.0349999999999999,"15min":1.0609999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":5.1210000000000004,"5min":5.9539999999999997,"15min":5.9539999999999997},"last":0.441},{"interface":"front","average":{"1min":1.5800000000000001,"5min":1.4750000000000001,"15min":1.6100000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":6.7190000000000003,"5min":10.707000000000001,"15min":10.707000000000001},"last":0.39800000000000002}]},{"osd":6,"last update":"Fri Feb 9 18:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.974,"5min":1.7889999999999999,"15min":1.948},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":6.6790000000000003,"5min":10.808,"15min":10.808},"last":1.3859999999999999},{"interface":"front","average":{"1min":1.5780000000000001,"5min":0.873,"15min":0.753},"min":{"1min":0.37,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":8.3550000000000004,"5min":8.3550000000000004,"15min":8.3550000000000004},"last":0.499}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70299999999999996}]},{"osd":8,"last update":"Fri Feb 9 18:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":1.3220000000000001,"15min":1.349},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":5.056,"5min":5.9820000000000002,"15min":5.9820000000000002},"last":0.51100000000000001},{"interface":"front","average":{"1min":1.732,"5min":1.7310000000000001,"15min":1.8680000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":5.7670000000000003,"5min":10.782,"15min":10.782},"last":0.47099999999999997}]},{"osd":9,"last update":"Fri Feb 9 18:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.643,"5min":0.90700000000000003,"15min":0.73699999999999999},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":8.3840000000000003,"5min":8.3840000000000003,"15min":8.3840000000000003},"last":0.56399999999999995},{"interface":"front","average":{"1min":1.248,"5min":1.353,"15min":1.349},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":5.2009999999999996,"5min":6.008,"15min":6.008},"last":0.622}]},{"osd":10,"last update":"Fri Feb 9 18:02:17 2024","interfaces":[{"interface":"back","average":{"1min":2.069,"5min":1.847,"15min":1.97},"min":{"1min":0.42499999999999999,"5min":0.377,"15min":0.377},"max":{"1min":6.867,"5min":10.885999999999999,"15min":10.885999999999999},"last":0.88800000000000001},{"interface":"front","average":{"1min":2.1000000000000001,"5min":1.494,"15min":1.431},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":6.9109999999999996,"5min":8.8970000000000002,"15min":8.8970000000000002},"last":0.90600000000000003}]},{"osd":11,"last update":"Fri Feb 9 18:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.694,"5min":2.0030000000000001,"15min":2.234},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":9.4390000000000001,"5min":17.126000000000001,"15min":17.126000000000001},"last":0.73599999999999999},{"interface":"front","average":{"1min":2.2839999999999998,"5min":1.232,"15min":1.03},"min":{"1min":0.54600000000000004,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":8.798,"5min":8.798,"15min":8.798},"last":3.1539999999999999}]}]},{"osd":4,"up_from":0,"seq":137438953989,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":40,"seq":171798692352,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4900208,"kb_used_data":4002728,"kb_used_omap":1,"kb_used_meta":897470,"kb_avail":88844944,"statfs":{"total":95995035648,"available":90977222656,"internally_reserved":0,"allocated":4098793472,"data_stored":4098496632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":919009733},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:58:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.82599999999999996,"15min":0.874},"min":{"1min":0.27800000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":1.341,"5min":5.399,"15min":5.399},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.622,"5min":0.92100000000000004,"15min":0.97099999999999997},"min":{"1min":0.40400000000000003,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.409,"5min":5.4329999999999998,"15min":5.4329999999999998},"last":0.52100000000000002}]},{"osd":1,"last update":"Fri Feb 9 18:01:23 2024","interfaces":[{"interface":"back","average":{"1min":1.9990000000000001,"5min":0.96299999999999997,"15min":1.3899999999999999},"min":{"1min":0.317,"5min":0.27200000000000002,"15min":0.254},"max":{"1min":16.271000000000001,"5min":16.271000000000001,"15min":19.079999999999998},"last":0.61699999999999999},{"interface":"front","average":{"1min":1.861,"5min":0.91500000000000004,"15min":1.2030000000000001},"min":{"1min":0.29399999999999998,"5min":0.26900000000000002,"15min":0.23499999999999999},"max":{"1min":16.344999999999999,"5min":16.344999999999999,"15min":19.065999999999999},"last":4.7629999999999999}]},{"osd":2,"last update":"Fri Feb 9 18:02:02 2024","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.68000000000000005,"15min":1.3460000000000001},"min":{"1min":0.27500000000000002,"5min":0.27400000000000002,"15min":0.254},"max":{"1min":6.0069999999999997,"5min":6.0069999999999997,"15min":9.673},"last":0.58399999999999996},{"interface":"front","average":{"1min":1.478,"5min":0.99299999999999999,"15min":1.5309999999999999},"min":{"1min":0.315,"5min":0.25800000000000001,"15min":0.24399999999999999},"max":{"1min":4.0679999999999996,"5min":4.3289999999999997,"15min":14.978},"last":4.8019999999999996}]},{"osd":3,"last update":"Fri Feb 9 18:01:18 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.68799999999999994,"15min":0.71599999999999997},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.5029999999999999,"5min":2.625,"15min":2.625},"last":0.438},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.92200000000000004,"15min":0.96399999999999997},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.8120000000000001,"5min":4.1379999999999999,"15min":4.1379999999999999},"last":4.8499999999999996}]},{"osd":4,"last update":"Fri Feb 9 18:01:23 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.56299999999999994,"15min":0.95799999999999996},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":3.383,"5min":3.383,"15min":28.481999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":1.079,"5min":0.74199999999999999,"15min":1.1579999999999999},"min":{"1min":0.35999999999999999,"5min":0.35199999999999998,"15min":0.22500000000000001},"max":{"1min":6.1239999999999997,"5min":6.1239999999999997,"15min":15.843},"last":0.68600000000000005}]},{"osd":6,"last update":"Fri Feb 9 18:02:02 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.63800000000000001,"15min":0.97799999999999998},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.891,"5min":2.8700000000000001,"15min":15.026},"last":4.8280000000000003},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.66100000000000003,"15min":1.129},"min":{"1min":0.36899999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.8680000000000001,"5min":2.278,"15min":20.192},"last":4.8979999999999997}]},{"osd":7,"last update":"Fri Feb 9 18:01:34 2024","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":0.75700000000000001,"15min":1.2250000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":4.6130000000000004,"5min":4.6130000000000004,"15min":18.134},"last":0.88400000000000001},{"interface":"front","average":{"1min":1.052,"5min":0.749,"15min":1.3340000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.31},"max":{"1min":3.504,"5min":3.504,"15min":19.300000000000001},"last":0.90900000000000003}]},{"osd":8,"last update":"Fri Feb 9 18:01:48 2024","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.8600000000000001,"15min":1.669},"min":{"1min":0.377,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":3.4780000000000002,"5min":12.227,"15min":12.227},"last":0.66100000000000003},{"interface":"front","average":{"1min":1.141,"5min":1.2709999999999999,"15min":1.5169999999999999},"min":{"1min":0.39600000000000002,"5min":0.372,"15min":0.372},"max":{"1min":4.6200000000000001,"5min":8.3770000000000007,"15min":8.3770000000000007},"last":0.51200000000000001}]},{"osd":9,"last update":"Fri Feb 9 18:02:02 2024","interfaces":[{"interface":"back","average":{"1min":1.655,"5min":1.819,"15min":1.677},"min":{"1min":0.39500000000000002,"5min":0.315,"15min":0.315},"max":{"1min":6.4729999999999999,"5min":11.167999999999999,"15min":15.124000000000001},"last":0.48499999999999999},{"interface":"front","average":{"1min":1.835,"5min":1.627,"15min":1.571},"min":{"1min":0.38,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":6.4589999999999996,"5min":13.462999999999999,"15min":13.462999999999999},"last":4.875}]},{"osd":10,"last update":"Fri Feb 9 18:01:23 2024","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":1.3700000000000001,"15min":1.752},"min":{"1min":0.47699999999999998,"5min":0.46700000000000003,"15min":0.373},"max":{"1min":2.2280000000000002,"5min":11.506,"15min":80.042000000000002},"last":0.56000000000000005},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.663,"15min":1.875},"min":{"1min":0.52800000000000002,"5min":0.39900000000000002,"15min":0.38200000000000001},"max":{"1min":6.2869999999999999,"5min":11.638,"15min":79.807000000000002},"last":4.9450000000000003}]},{"osd":11,"last update":"Fri Feb 9 18:01:48 2024","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":1.6399999999999999,"15min":1.6890000000000001},"min":{"1min":0.495,"5min":0.434,"15min":0.39400000000000002},"max":{"1min":2.3610000000000002,"5min":6.9240000000000004,"15min":9.7680000000000007},"last":0.53700000000000003},{"interface":"front","average":{"1min":1.173,"5min":1.569,"15min":1.7210000000000001},"min":{"1min":0.52400000000000002,"5min":0.48699999999999999,"15min":0.42199999999999999},"max":{"1min":3.2610000000000001,"5min":7.2709999999999999,"15min":20.981000000000002},"last":4.9219999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":460718080,"data_stored":460718080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3585806336,"data_stored":3585802264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2688876544,"data_stored":2688876544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4068478976,"data_stored":4068474904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":887357440,"data_stored":887357440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2318471168,"data_stored":2318467096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":918421504,"data_stored":918421504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2768175104,"data_stored":2768175104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2275475456,"data_stored":2275475456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1854013440,"data_stored":1854013440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-09T18:02:24.770 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-02-09T18:02:24.771 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T18:02:24.771 INFO:tasks.thrashosds.thrasher:inject_pause on osd.9 2024-02-09T18:02:24.771 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-09T18:02:24.771 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-09T18:02:24.771 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.9 config set filestore_inject_stall 3 2024-02-09T18:02:24.785 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T18:02:24.812 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:24.839 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T18:02:24.848 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:24.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T18:02:24.949 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:02:24.985 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:25.026 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:02:25.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:25.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:02:25.132 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:02:25.158 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T18:02:25.200 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:25.465 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:02:25.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:25.469 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:02:25.570 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:02:25.616 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:02:25.619 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:02:25.621 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:26.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:26 smithi179 ceph-mon[43975]: pgmap v2297: 47 pgs: 47 active+clean; 6.6 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 0 B/s wr, 756 op/s 2024-02-09T18:02:26.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:26 smithi162 ceph-mon[44304]: pgmap v2297: 47 pgs: 47 active+clean; 6.6 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 0 B/s wr, 756 op/s 2024-02-09T18:02:26.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:26 smithi099 ceph-mon[39420]: pgmap v2297: 47 pgs: 47 active+clean; 6.6 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 0 B/s wr, 756 op/s 2024-02-09T18:02:26.805 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:02:26.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:26.811 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:02:26.912 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:02:26.945 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:26.974 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:02:26.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:26.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:02:27.078 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T18:02:27.109 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:27.133 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T18:02:27.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:27.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T18:02:27.239 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:02:27.294 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:27.943 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T18:02:27.943 INFO:teuthology.orchestra.run.smithi179.stdout: "success": "" 2024-02-09T18:02:27.943 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T18:02:27.962 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:02:27.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:27.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:02:28.084 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:02:28.118 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:28.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:28 smithi099 ceph-mon[39420]: pgmap v2298: 47 pgs: 47 active+clean; 6.5 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 780 op/s 2024-02-09T18:02:28.362 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:28 smithi179 ceph-mon[43975]: pgmap v2298: 47 pgs: 47 active+clean; 6.5 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 780 op/s 2024-02-09T18:02:28.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:28 smithi162 ceph-mon[44304]: pgmap v2298: 47 pgs: 47 active+clean; 6.5 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 780 op/s 2024-02-09T18:02:28.977 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:02:28.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:28.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:02:29.087 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:02:29.162 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:29.194 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:02:29.199 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:29.200 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:02:29.301 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:02:29.393 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:29.912 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T18:02:29.912 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T18:02:30.426 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:02:30.426 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:02:30.426 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: osdmap e928: 12 total, 12 up, 11 in 2024-02-09T18:02:30.426 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {auth=28,config=2,health=12,kv=58} crc {auth=1792813228,config=2679774077,health=179246142,kv=2444497438}) 2024-02-09T18:02:30.426 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=4138030616,logm=332343796}) 2024-02-09T18:02:30.426 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1727195888}) 2024-02-09T18:02:30.426 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1257956056}) 2024-02-09T18:02:30.426 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=500254099}) 2024-02-09T18:02:30.426 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=365449423}) 2024-02-09T18:02:30.427 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2792877541}) 2024-02-09T18:02:30.427 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3635600428}) 2024-02-09T18:02:30.427 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2411805209}) 2024-02-09T18:02:30.427 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2156228893}) 2024-02-09T18:02:30.427 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1535274803}) 2024-02-09T18:02:30.427 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3552332210}) 2024-02-09T18:02:30.427 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3841274669}) 2024-02-09T18:02:30.427 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4028159809}) 2024-02-09T18:02:30.427 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=809242284}) 2024-02-09T18:02:30.427 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2837973831}) 2024-02-09T18:02:30.427 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3537637704}) 2024-02-09T18:02:30.427 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2239055411}) 2024-02-09T18:02:30.428 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=384501206}) 2024-02-09T18:02:30.428 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1044840052}) 2024-02-09T18:02:30.428 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3979854366}) 2024-02-09T18:02:30.428 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2355146118}) 2024-02-09T18:02:30.428 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3476602836}) 2024-02-09T18:02:30.428 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2003095735}) 2024-02-09T18:02:30.428 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2800928888}) 2024-02-09T18:02:30.428 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=280593289}) 2024-02-09T18:02:30.428 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1467132434}) 2024-02-09T18:02:30.428 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1104176080}) 2024-02-09T18:02:30.428 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2610730830}) 2024-02-09T18:02:30.428 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=852819282}) 2024-02-09T18:02:30.428 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2954262465}) 2024-02-09T18:02:30.431 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2065753459}) 2024-02-09T18:02:30.431 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3473733430}) 2024-02-09T18:02:30.431 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4152336871}) 2024-02-09T18:02:30.431 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4055568037}) 2024-02-09T18:02:30.431 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=937086783}) 2024-02-09T18:02:30.431 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=669166534}) 2024-02-09T18:02:30.431 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3632903486}) 2024-02-09T18:02:30.431 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1925026490}) 2024-02-09T18:02:30.431 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3569819108}) 2024-02-09T18:02:30.431 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3559099869}) 2024-02-09T18:02:30.431 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3237186833}) 2024-02-09T18:02:30.431 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2733924555}) 2024-02-09T18:02:30.432 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=319198364}) 2024-02-09T18:02:30.432 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=207038696}) 2024-02-09T18:02:30.432 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3492216922}) 2024-02-09T18:02:30.432 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=274917097}) 2024-02-09T18:02:30.432 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=831157579}) 2024-02-09T18:02:30.432 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2469546415}) 2024-02-09T18:02:30.432 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3954900589}) 2024-02-09T18:02:30.432 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1825971682}) 2024-02-09T18:02:30.432 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1698208980}) 2024-02-09T18:02:30.432 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2201760209}) 2024-02-09T18:02:30.432 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3151513253}) 2024-02-09T18:02:30.432 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2305847853}) 2024-02-09T18:02:30.432 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=722471450}) 2024-02-09T18:02:30.432 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2181583845}) 2024-02-09T18:02:30.433 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2155681733}) 2024-02-09T18:02:30.433 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: pgmap v2300: 47 pgs: 47 active+clean; 6.4 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 834 op/s 2024-02-09T18:02:30.433 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4132127095}) 2024-02-09T18:02:30.433 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1184146327}) 2024-02-09T18:02:30.433 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2427357777}) 2024-02-09T18:02:30.433 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2867939250}) 2024-02-09T18:02:30.433 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3661106200}) 2024-02-09T18:02:30.433 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4276909332' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:02:30.433 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2543232901}) 2024-02-09T18:02:30.433 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1893748319}) 2024-02-09T18:02:30.433 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3108683825}) 2024-02-09T18:02:30.433 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1033298209}) 2024-02-09T18:02:30.434 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1567983429}) 2024-02-09T18:02:30.434 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3999996799}) 2024-02-09T18:02:30.434 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3659905492}) 2024-02-09T18:02:30.434 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=808258309}) 2024-02-09T18:02:30.434 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1321690855}) 2024-02-09T18:02:30.434 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1652674042}) 2024-02-09T18:02:30.434 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2601941443}) 2024-02-09T18:02:30.434 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=598342682}) 2024-02-09T18:02:30.434 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=562239962}) 2024-02-09T18:02:30.434 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1304561750}) 2024-02-09T18:02:30.434 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1443748464}) 2024-02-09T18:02:30.434 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1303359339}) 2024-02-09T18:02:30.435 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3237093459}) 2024-02-09T18:02:30.435 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=850356225}) 2024-02-09T18:02:30.435 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=892099117}) 2024-02-09T18:02:30.435 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=69,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=7} crc {logm=1279227212,mdsmap=3984425268,mgr=1708625883,mgr_command_descs=708342470,mgr_metadata=670763949,mgrstat=1780723156}) 2024-02-09T18:02:30.435 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=2,mon_config_key=98} crc {mgrstat=2779027833,mon_config_key=3391493961}) 2024-02-09T18:02:30.435 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2233069991}) 2024-02-09T18:02:30.435 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=83,monmap=5,osd_metadata=12} crc {mon_config_key=329105446,monmap=423865542,osd_metadata=2412076312}) 2024-02-09T18:02:30.435 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {osd_pg_creating=1,osdmap=99} crc {osd_pg_creating=479701159,osdmap=3543007771}) 2024-02-09T18:02:30.435 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=24} crc {osdmap=293858379}) 2024-02-09T18:02:30.436 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:02:30.436 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:02:30.436 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: osdmap e928: 12 total, 12 up, 11 in 2024-02-09T18:02:30.436 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {auth=28,config=2,health=12,kv=58} crc {auth=1792813228,config=2679774077,health=179246142,kv=2444497438}) 2024-02-09T18:02:30.436 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=4138030616,logm=332343796}) 2024-02-09T18:02:30.436 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1727195888}) 2024-02-09T18:02:30.436 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1257956056}) 2024-02-09T18:02:30.436 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=500254099}) 2024-02-09T18:02:30.437 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=365449423}) 2024-02-09T18:02:30.437 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2792877541}) 2024-02-09T18:02:30.437 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3635600428}) 2024-02-09T18:02:30.437 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2411805209}) 2024-02-09T18:02:30.437 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2156228893}) 2024-02-09T18:02:30.437 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1535274803}) 2024-02-09T18:02:30.437 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3552332210}) 2024-02-09T18:02:30.437 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3841274669}) 2024-02-09T18:02:30.437 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4028159809}) 2024-02-09T18:02:30.437 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=809242284}) 2024-02-09T18:02:30.437 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2837973831}) 2024-02-09T18:02:30.437 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3537637704}) 2024-02-09T18:02:30.437 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2239055411}) 2024-02-09T18:02:30.438 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=384501206}) 2024-02-09T18:02:30.438 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1044840052}) 2024-02-09T18:02:30.438 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3979854366}) 2024-02-09T18:02:30.438 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2355146118}) 2024-02-09T18:02:30.438 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3476602836}) 2024-02-09T18:02:30.438 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2003095735}) 2024-02-09T18:02:30.438 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2800928888}) 2024-02-09T18:02:30.438 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=280593289}) 2024-02-09T18:02:30.438 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1467132434}) 2024-02-09T18:02:30.438 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1104176080}) 2024-02-09T18:02:30.438 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2610730830}) 2024-02-09T18:02:30.438 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=852819282}) 2024-02-09T18:02:30.439 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2954262465}) 2024-02-09T18:02:30.439 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2065753459}) 2024-02-09T18:02:30.439 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3473733430}) 2024-02-09T18:02:30.439 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4152336871}) 2024-02-09T18:02:30.439 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4055568037}) 2024-02-09T18:02:30.439 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=937086783}) 2024-02-09T18:02:30.439 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=669166534}) 2024-02-09T18:02:30.439 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3632903486}) 2024-02-09T18:02:30.439 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1925026490}) 2024-02-09T18:02:30.439 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3569819108}) 2024-02-09T18:02:30.439 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3559099869}) 2024-02-09T18:02:30.439 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3237186833}) 2024-02-09T18:02:30.439 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2733924555}) 2024-02-09T18:02:30.439 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=319198364}) 2024-02-09T18:02:30.439 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=207038696}) 2024-02-09T18:02:30.440 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3492216922}) 2024-02-09T18:02:30.440 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=274917097}) 2024-02-09T18:02:30.440 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=831157579}) 2024-02-09T18:02:30.440 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2469546415}) 2024-02-09T18:02:30.440 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3954900589}) 2024-02-09T18:02:30.440 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1825971682}) 2024-02-09T18:02:30.440 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1698208980}) 2024-02-09T18:02:30.440 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2201760209}) 2024-02-09T18:02:30.440 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3151513253}) 2024-02-09T18:02:30.440 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2305847853}) 2024-02-09T18:02:30.440 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=722471450}) 2024-02-09T18:02:30.440 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2181583845}) 2024-02-09T18:02:30.440 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2155681733}) 2024-02-09T18:02:30.441 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: pgmap v2300: 47 pgs: 47 active+clean; 6.4 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 834 op/s 2024-02-09T18:02:30.441 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4132127095}) 2024-02-09T18:02:30.441 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1184146327}) 2024-02-09T18:02:30.441 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2427357777}) 2024-02-09T18:02:30.441 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2867939250}) 2024-02-09T18:02:30.441 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3661106200}) 2024-02-09T18:02:30.441 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4276909332' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:02:30.441 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2543232901}) 2024-02-09T18:02:30.441 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1893748319}) 2024-02-09T18:02:30.441 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3108683825}) 2024-02-09T18:02:30.441 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1033298209}) 2024-02-09T18:02:30.441 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1567983429}) 2024-02-09T18:02:30.441 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3999996799}) 2024-02-09T18:02:30.441 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3659905492}) 2024-02-09T18:02:30.442 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=808258309}) 2024-02-09T18:02:30.442 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1321690855}) 2024-02-09T18:02:30.442 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1652674042}) 2024-02-09T18:02:30.442 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2601941443}) 2024-02-09T18:02:30.442 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=598342682}) 2024-02-09T18:02:30.442 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=562239962}) 2024-02-09T18:02:30.442 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1304561750}) 2024-02-09T18:02:30.442 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1443748464}) 2024-02-09T18:02:30.442 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1303359339}) 2024-02-09T18:02:30.442 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3237093459}) 2024-02-09T18:02:30.442 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=850356225}) 2024-02-09T18:02:30.442 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=892099117}) 2024-02-09T18:02:30.442 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=69,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=7} crc {logm=1279227212,mdsmap=3984425268,mgr=1708625883,mgr_command_descs=708342470,mgr_metadata=670763949,mgrstat=1780723156}) 2024-02-09T18:02:30.442 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=2,mon_config_key=98} crc {mgrstat=2779027833,mon_config_key=3391493961}) 2024-02-09T18:02:30.443 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2233069991}) 2024-02-09T18:02:30.443 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=83,monmap=5,osd_metadata=12} crc {mon_config_key=329105446,monmap=423865542,osd_metadata=2412076312}) 2024-02-09T18:02:30.443 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {osd_pg_creating=1,osdmap=99} crc {osd_pg_creating=479701159,osdmap=3543007771}) 2024-02-09T18:02:30.443 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=24} crc {osdmap=293858379}) 2024-02-09T18:02:30.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:02:30.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:02:30.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: osdmap e928: 12 total, 12 up, 11 in 2024-02-09T18:02:30.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {auth=28,config=2,health=12,kv=58} crc {auth=1792813228,config=2679774077,health=179246142,kv=2444497438}) 2024-02-09T18:02:30.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=4138030616,logm=332343796}) 2024-02-09T18:02:30.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1727195888}) 2024-02-09T18:02:30.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1257956056}) 2024-02-09T18:02:30.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=500254099}) 2024-02-09T18:02:30.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=365449423}) 2024-02-09T18:02:30.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2792877541}) 2024-02-09T18:02:30.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3635600428}) 2024-02-09T18:02:30.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2411805209}) 2024-02-09T18:02:30.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2156228893}) 2024-02-09T18:02:30.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1535274803}) 2024-02-09T18:02:30.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3552332210}) 2024-02-09T18:02:30.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3841274669}) 2024-02-09T18:02:30.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4028159809}) 2024-02-09T18:02:30.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=809242284}) 2024-02-09T18:02:30.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2837973831}) 2024-02-09T18:02:30.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3537637704}) 2024-02-09T18:02:30.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2239055411}) 2024-02-09T18:02:30.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=384501206}) 2024-02-09T18:02:30.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1044840052}) 2024-02-09T18:02:30.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3979854366}) 2024-02-09T18:02:30.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2355146118}) 2024-02-09T18:02:30.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3476602836}) 2024-02-09T18:02:30.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2003095735}) 2024-02-09T18:02:30.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2800928888}) 2024-02-09T18:02:30.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=280593289}) 2024-02-09T18:02:30.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1467132434}) 2024-02-09T18:02:30.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1104176080}) 2024-02-09T18:02:30.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2610730830}) 2024-02-09T18:02:30.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=852819282}) 2024-02-09T18:02:30.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2954262465}) 2024-02-09T18:02:30.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2065753459}) 2024-02-09T18:02:30.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3473733430}) 2024-02-09T18:02:30.498 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4152336871}) 2024-02-09T18:02:30.498 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4055568037}) 2024-02-09T18:02:30.498 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=937086783}) 2024-02-09T18:02:30.498 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=669166534}) 2024-02-09T18:02:30.498 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3632903486}) 2024-02-09T18:02:30.498 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1925026490}) 2024-02-09T18:02:30.499 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3569819108}) 2024-02-09T18:02:30.499 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3559099869}) 2024-02-09T18:02:30.499 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3237186833}) 2024-02-09T18:02:30.499 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2733924555}) 2024-02-09T18:02:30.499 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=319198364}) 2024-02-09T18:02:30.499 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=207038696}) 2024-02-09T18:02:30.499 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3492216922}) 2024-02-09T18:02:30.499 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=274917097}) 2024-02-09T18:02:30.499 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=831157579}) 2024-02-09T18:02:30.499 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2469546415}) 2024-02-09T18:02:30.499 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3954900589}) 2024-02-09T18:02:30.499 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1825971682}) 2024-02-09T18:02:30.500 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1698208980}) 2024-02-09T18:02:30.500 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2201760209}) 2024-02-09T18:02:30.500 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3151513253}) 2024-02-09T18:02:30.500 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2305847853}) 2024-02-09T18:02:30.500 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=722471450}) 2024-02-09T18:02:30.500 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2181583845}) 2024-02-09T18:02:30.500 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2155681733}) 2024-02-09T18:02:30.500 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: pgmap v2300: 47 pgs: 47 active+clean; 6.4 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 834 op/s 2024-02-09T18:02:30.500 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4132127095}) 2024-02-09T18:02:30.500 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1184146327}) 2024-02-09T18:02:30.500 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2427357777}) 2024-02-09T18:02:30.500 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2867939250}) 2024-02-09T18:02:30.501 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3661106200}) 2024-02-09T18:02:30.501 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4276909332' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:02:30.501 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2543232901}) 2024-02-09T18:02:30.501 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1893748319}) 2024-02-09T18:02:30.501 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3108683825}) 2024-02-09T18:02:30.501 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1033298209}) 2024-02-09T18:02:30.501 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1567983429}) 2024-02-09T18:02:30.501 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3999996799}) 2024-02-09T18:02:30.501 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3659905492}) 2024-02-09T18:02:30.501 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=808258309}) 2024-02-09T18:02:30.501 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1321690855}) 2024-02-09T18:02:30.501 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1652674042}) 2024-02-09T18:02:30.501 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2601941443}) 2024-02-09T18:02:30.502 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=598342682}) 2024-02-09T18:02:30.502 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=562239962}) 2024-02-09T18:02:30.502 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1304561750}) 2024-02-09T18:02:30.502 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1443748464}) 2024-02-09T18:02:30.502 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1303359339}) 2024-02-09T18:02:30.502 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3237093459}) 2024-02-09T18:02:30.502 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=850356225}) 2024-02-09T18:02:30.502 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=892099117}) 2024-02-09T18:02:30.502 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=69,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=7} crc {logm=1279227212,mdsmap=3984425268,mgr=1708625883,mgr_command_descs=708342470,mgr_metadata=670763949,mgrstat=1780723156}) 2024-02-09T18:02:30.502 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=2,mon_config_key=98} crc {mgrstat=2779027833,mon_config_key=3391493961}) 2024-02-09T18:02:30.502 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2233069991}) 2024-02-09T18:02:30.503 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=83,monmap=5,osd_metadata=12} crc {mon_config_key=329105446,monmap=423865542,osd_metadata=2412076312}) 2024-02-09T18:02:30.503 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {osd_pg_creating=1,osdmap=99} crc {osd_pg_creating=479701159,osdmap=3543007771}) 2024-02-09T18:02:30.503 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=24} crc {osdmap=293858379}) 2024-02-09T18:02:30.559 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T18:02:30.560 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T18:02:30.560 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T18:02:30.560 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T18:02:30.560 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T18:02:30.560 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T18:02:30.560 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T18:02:30.560 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T18:02:30.560 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T18:02:30.560 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T18:02:30.560 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T18:02:30.560 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T18:02:30.561 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T18:02:30.561 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T18:02:30.561 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T18:02:30.561 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T18:02:30.561 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T18:02:30.561 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T18:02:30.561 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T18:02:30.561 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T18:02:30.561 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T18:02:30.561 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T18:02:30.561 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T18:02:30.562 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T18:02:32.082 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:02:32.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:32.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:02:32.193 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:02:32.240 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:32.351 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:02:32.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:32.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:02:32.362 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:32 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:02:32.362 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:32 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:02:32.362 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:32 smithi099 ceph-mon[39420]: osdmap e929: 12 total, 12 up, 11 in 2024-02-09T18:02:32.362 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:32 smithi099 ceph-mon[39420]: pgmap v2302: 47 pgs: 47 active+clean; 6.2 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 915 op/s 2024-02-09T18:02:32.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:32 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:02:32.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:32 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:02:32.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:32 smithi179 ceph-mon[43975]: osdmap e929: 12 total, 12 up, 11 in 2024-02-09T18:02:32.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:32 smithi179 ceph-mon[43975]: pgmap v2302: 47 pgs: 47 active+clean; 6.2 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 915 op/s 2024-02-09T18:02:32.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:32 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:02:32.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:32 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:02:32.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:32 smithi162 ceph-mon[44304]: osdmap e929: 12 total, 12 up, 11 in 2024-02-09T18:02:32.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:32 smithi162 ceph-mon[44304]: pgmap v2302: 47 pgs: 47 active+clean; 6.2 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 915 op/s 2024-02-09T18:02:32.462 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T18:02:32.520 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:32.551 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T18:02:32.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:32.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T18:02:32.655 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:02:32.711 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:33.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:33 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3332960261' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T18:02:33.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:33 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3332960261' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T18:02:33.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:33 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3332960261' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T18:02:33.763 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 5, 6, 7, 8, 9, 10, 11, 0, 2] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 7, 9, 10, 11, 8, 3] 2024-02-09T18:02:33.763 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T18:02:33.763 INFO:tasks.thrashosds.thrasher:inject_pause on osd.10 2024-02-09T18:02:33.763 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-09T18:02:33.763 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-09T18:02:33.764 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.10 config set filestore_inject_stall 3 2024-02-09T18:02:33.836 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is unset 2024-02-09T18:02:34.195 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T18:02:34.237 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-09T18:02:34.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:34 smithi099 ceph-mon[39420]: pgmap v2303: 47 pgs: 47 active+clean; 6.2 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 955 op/s 2024-02-09T18:02:34.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:34 smithi099 ceph-mon[39420]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:02:34.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:34 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3332960261' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T18:02:34.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:34 smithi099 ceph-mon[39420]: osdmap e930: 12 total, 12 up, 11 in 2024-02-09T18:02:34.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:34 smithi179 ceph-mon[43975]: pgmap v2303: 47 pgs: 47 active+clean; 6.2 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 955 op/s 2024-02-09T18:02:34.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:34 smithi179 ceph-mon[43975]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:02:34.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:34 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3332960261' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T18:02:34.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:34 smithi179 ceph-mon[43975]: osdmap e930: 12 total, 12 up, 11 in 2024-02-09T18:02:34.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:34 smithi162 ceph-mon[44304]: pgmap v2303: 47 pgs: 47 active+clean; 6.2 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 955 op/s 2024-02-09T18:02:34.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:34 smithi162 ceph-mon[44304]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:02:34.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:34 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3332960261' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T18:02:34.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:34 smithi162 ceph-mon[44304]: osdmap e930: 12 total, 12 up, 11 in 2024-02-09T18:02:35.097 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:02:35.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:35.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:02:35.212 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T18:02:35.251 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:35.288 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T18:02:35.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:35.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T18:02:35.383 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:35 smithi099 ceph-mon[39420]: 9.f scrub starts 2024-02-09T18:02:35.392 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:02:35.395 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:02:35.424 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:35.448 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:02:35.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:35.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:02:35.555 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T18:02:35.589 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:35 smithi162 ceph-mon[44304]: 9.f scrub starts 2024-02-09T18:02:35.592 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:35.625 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T18:02:35.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:35.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T18:02:35.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:35 smithi179 ceph-mon[43975]: 9.f scrub starts 2024-02-09T18:02:35.732 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T18:02:35.785 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:35.818 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T18:02:35.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:35.826 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T18:02:35.927 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:02:35.976 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:36.017 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:02:36.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:36.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:02:36.122 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T18:02:36.170 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:36.380 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:36 smithi099 ceph-mon[39420]: pgmap v2305: 47 pgs: 47 active+clean; 6.1 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 721 op/s 2024-02-09T18:02:36.381 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:36 smithi099 ceph-mon[39420]: 9.f scrub ok 2024-02-09T18:02:36.381 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:36 smithi162 ceph-mon[44304]: pgmap v2305: 47 pgs: 47 active+clean; 6.1 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 721 op/s 2024-02-09T18:02:36.381 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:36 smithi162 ceph-mon[44304]: 9.f scrub ok 2024-02-09T18:02:36.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:36 smithi179 ceph-mon[43975]: pgmap v2305: 47 pgs: 47 active+clean; 6.1 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 721 op/s 2024-02-09T18:02:36.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:36 smithi179 ceph-mon[43975]: 9.f scrub ok 2024-02-09T18:02:36.904 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-09T18:02:37.247 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T18:02:37.257 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T18:02:37.257 INFO:teuthology.orchestra.run.smithi179.stdout: "success": "" 2024-02-09T18:02:37.258 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T18:02:37.343 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T18:02:37.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:37.349 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T18:02:37.450 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:02:37.503 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:37.532 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:02:37.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:37.539 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:02:37.541 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:37 smithi179 ceph-mon[43975]: 1.10 scrub starts 2024-02-09T18:02:37.541 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:37 smithi179 ceph-mon[43975]: 1.10 scrub ok 2024-02-09T18:02:37.640 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:02:37.670 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:37 smithi162 ceph-mon[44304]: 1.10 scrub starts 2024-02-09T18:02:37.670 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:37 smithi162 ceph-mon[44304]: 1.10 scrub ok 2024-02-09T18:02:37.674 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:37.713 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:02:37.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:37.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:02:37.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:37 smithi099 ceph-mon[39420]: 1.10 scrub starts 2024-02-09T18:02:37.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:37 smithi099 ceph-mon[39420]: 1.10 scrub ok 2024-02-09T18:02:37.821 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:02:37.868 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:38.072 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:02:38.143 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T18:02:38.143 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T18:02:38.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:38 smithi099 ceph-mon[39420]: pgmap v2306: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.0 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 842 op/s 2024-02-09T18:02:38.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:38 smithi179 ceph-mon[43975]: pgmap v2306: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.0 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 842 op/s 2024-02-09T18:02:38.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:38 smithi162 ceph-mon[44304]: pgmap v2306: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.0 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 842 op/s 2024-02-09T18:02:39.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:39 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:02:39.675 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:39 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:02:39.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:39 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:02:39.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:39 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:02:39.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:39 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:02:39.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:39 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:02:40.032 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:02:40.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:40.035 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:02:40.136 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:02:40.171 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:40.394 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:40 smithi099 ceph-mon[39420]: pgmap v2307: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.8 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 913 op/s 2024-02-09T18:02:40.394 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:40 smithi099 ceph-mon[39420]: osdmap e931: 12 total, 12 up, 11 in 2024-02-09T18:02:40.394 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:40 smithi099 ceph-mon[39420]: 1.1a scrub starts 2024-02-09T18:02:40.394 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:40 smithi099 ceph-mon[39420]: 1.1a scrub ok 2024-02-09T18:02:40.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:40 smithi179 ceph-mon[43975]: pgmap v2307: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.8 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 913 op/s 2024-02-09T18:02:40.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:40 smithi179 ceph-mon[43975]: osdmap e931: 12 total, 12 up, 11 in 2024-02-09T18:02:40.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:40 smithi179 ceph-mon[43975]: 1.1a scrub starts 2024-02-09T18:02:40.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:40 smithi179 ceph-mon[43975]: 1.1a scrub ok 2024-02-09T18:02:40.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:40 smithi162 ceph-mon[44304]: pgmap v2307: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.8 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 913 op/s 2024-02-09T18:02:40.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:40 smithi162 ceph-mon[44304]: osdmap e931: 12 total, 12 up, 11 in 2024-02-09T18:02:40.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:40 smithi162 ceph-mon[44304]: 1.1a scrub starts 2024-02-09T18:02:40.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:40 smithi162 ceph-mon[44304]: 1.1a scrub ok 2024-02-09T18:02:40.873 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:02:40.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:40.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:02:40.917 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-09T18:02:40.984 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:02:41.034 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:41.071 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:02:41.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:41.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:02:41.175 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:02:41.177 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:02:41.219 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:41.301 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T18:02:41.317 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:02:41.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:41.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:02:41.414 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:41 smithi162 ceph-mon[44304]: 1.b scrub starts 2024-02-09T18:02:41.414 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:41 smithi162 ceph-mon[44304]: 1.b scrub ok 2024-02-09T18:02:41.438 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:02:41.482 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:41.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:41 smithi099 ceph-mon[39420]: 1.b scrub starts 2024-02-09T18:02:41.497 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:41 smithi099 ceph-mon[39420]: 1.b scrub ok 2024-02-09T18:02:41.572 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:02:41.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:41.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:02:41.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:41 smithi179 ceph-mon[43975]: 1.b scrub starts 2024-02-09T18:02:41.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:41 smithi179 ceph-mon[43975]: 1.b scrub ok 2024-02-09T18:02:41.682 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:02:41.726 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:41.811 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:02:41.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:41.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:02:41.918 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:02:41.957 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:42.133 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:02:42.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:42.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:02:42.239 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:02:42.293 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:42.418 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:42 smithi162 ceph-mon[44304]: pgmap v2309: 47 pgs: 47 active+clean; 5.7 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 930 op/s 2024-02-09T18:02:42.460 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:42 smithi162 ceph-mon[44304]: 1.6 scrub starts 2024-02-09T18:02:42.460 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:42 smithi162 ceph-mon[44304]: 1.6 scrub ok 2024-02-09T18:02:42.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:42 smithi179 ceph-mon[43975]: pgmap v2309: 47 pgs: 47 active+clean; 5.7 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 930 op/s 2024-02-09T18:02:42.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:42 smithi179 ceph-mon[43975]: 1.6 scrub starts 2024-02-09T18:02:42.674 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:42 smithi179 ceph-mon[43975]: 1.6 scrub ok 2024-02-09T18:02:42.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:42 smithi099 ceph-mon[39420]: pgmap v2309: 47 pgs: 47 active+clean; 5.7 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 930 op/s 2024-02-09T18:02:42.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:42 smithi099 ceph-mon[39420]: 1.6 scrub starts 2024-02-09T18:02:42.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:42 smithi099 ceph-mon[39420]: 1.6 scrub ok 2024-02-09T18:02:42.868 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:02:42.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:42.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:02:42.974 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T18:02:43.044 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:43.264 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T18:02:43.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:43.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T18:02:43.369 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:02:43.460 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:43.635 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:43 smithi099 ceph-mon[39420]: 9.c scrub starts 2024-02-09T18:02:43.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:43 smithi162 ceph-mon[44304]: 9.c scrub starts 2024-02-09T18:02:43.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:43 smithi179 ceph-mon[43975]: 9.c scrub starts 2024-02-09T18:02:44.031 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T18:02:44.031 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T18:02:44.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:44 smithi162 ceph-mon[44304]: pgmap v2310: 47 pgs: 47 active+clean; 5.6 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 884 op/s 2024-02-09T18:02:44.724 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:02:44.725 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:44 smithi099 ceph-mon[39420]: pgmap v2310: 47 pgs: 47 active+clean; 5.6 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 884 op/s 2024-02-09T18:02:44.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:44.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:02:44.750 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-09T18:02:44.829 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:02:44.872 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:44.911 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:02:44.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:44.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:02:44.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:44 smithi179 ceph-mon[43975]: pgmap v2310: 47 pgs: 47 active+clean; 5.6 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 884 op/s 2024-02-09T18:02:45.016 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:02:45.077 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:45.122 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T18:02:45.433 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:45 smithi099 ceph-mon[39420]: 9.c scrub ok 2024-02-09T18:02:45.434 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:45 smithi099 ceph-mon[39420]: 9.2 scrub starts 2024-02-09T18:02:45.543 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:02:45.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:45.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:02:45.581 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T18:02:45.623 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T18:02:45.649 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T18:02:45.660 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T18:02:45.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:45 smithi162 ceph-mon[44304]: 9.c scrub ok 2024-02-09T18:02:45.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:45 smithi162 ceph-mon[44304]: 9.2 scrub starts 2024-02-09T18:02:45.682 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:45.685 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T18:02:45.713 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T18:02:45.716 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T18:02:45.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:45.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T18:02:45.762 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T18:02:45.815 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T18:02:45.826 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:02:45.848 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T18:02:45.881 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T18:02:45.892 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:45.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:45 smithi179 ceph-mon[43975]: 9.c scrub ok 2024-02-09T18:02:45.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:45 smithi179 ceph-mon[43975]: 9.2 scrub starts 2024-02-09T18:02:45.928 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T18:02:45.935 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:02:45.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:45.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:02:45.980 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T18:02:46.040 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:02:46.041 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T18:02:46.053 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T18:02:46.053 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T18:02:46.053 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T18:02:46.053 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T18:02:46.053 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T18:02:46.053 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T18:02:46.053 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T18:02:46.053 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T18:02:46.053 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T18:02:46.053 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T18:02:46.053 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T18:02:46.053 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T18:02:46.089 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:46.114 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:02:46.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:46.118 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:02:46.219 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:02:46.266 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:46.312 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:02:46.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:46.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:02:46.416 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:02:46.448 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is unset 2024-02-09T18:02:46.466 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:46.516 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:02:46.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:46.546 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:02:46.647 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T18:02:46.702 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:46 smithi179 ceph-mon[43975]: pgmap v2311: 47 pgs: 47 active+clean; 5.6 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 877 op/s 2024-02-09T18:02:46.703 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:46 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2409165448' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:02:46.703 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:46 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1719624011' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T18:02:46.703 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:46 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T18:02:46.709 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:46.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:46 smithi099 ceph-mon[39420]: pgmap v2311: 47 pgs: 47 active+clean; 5.6 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 877 op/s 2024-02-09T18:02:46.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:46 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2409165448' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:02:46.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:46 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1719624011' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T18:02:46.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:46 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T18:02:46.751 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T18:02:46.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:46.760 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T18:02:46.857 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:46 smithi162 ceph-mon[44304]: pgmap v2311: 47 pgs: 47 active+clean; 5.6 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 877 op/s 2024-02-09T18:02:46.857 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:46 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2409165448' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:02:46.857 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:46 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1719624011' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T18:02:46.857 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:46 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T18:02:46.860 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T18:02:46.921 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:46.957 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T18:02:46.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:46.962 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T18:02:47.063 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:02:47.099 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:47.127 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:02:47.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:47.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:02:47.203 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T18:02:47.203 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-09T18:02:47.203 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-02-09T18:02:47.203 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-02-09T18:02:47.203 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T18:02:47.230 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:02:47.272 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:47.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:47.317 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:02:47.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:02:47.417 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:02:47.472 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:47.589 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:47 smithi162 ceph-mon[44304]: 1.f scrub starts 2024-02-09T18:02:47.589 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:47 smithi162 ceph-mon[44304]: 9.2 scrub ok 2024-02-09T18:02:47.589 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:47 smithi162 ceph-mon[44304]: 1.f scrub ok 2024-02-09T18:02:47.589 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:47 smithi162 ceph-mon[44304]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T18:02:47.589 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:47 smithi162 ceph-mon[44304]: Cluster is now healthy 2024-02-09T18:02:47.589 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:47 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T18:02:47.589 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:47 smithi162 ceph-mon[44304]: osdmap e932: 12 total, 12 up, 11 in 2024-02-09T18:02:47.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:47 smithi099 ceph-mon[39420]: 1.f scrub starts 2024-02-09T18:02:47.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:47 smithi099 ceph-mon[39420]: 9.2 scrub ok 2024-02-09T18:02:47.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:47 smithi099 ceph-mon[39420]: 1.f scrub ok 2024-02-09T18:02:47.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:47 smithi099 ceph-mon[39420]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T18:02:47.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:47 smithi099 ceph-mon[39420]: Cluster is now healthy 2024-02-09T18:02:47.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:47 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T18:02:47.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:47 smithi099 ceph-mon[39420]: osdmap e932: 12 total, 12 up, 11 in 2024-02-09T18:02:47.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:47 smithi179 ceph-mon[43975]: 1.f scrub starts 2024-02-09T18:02:47.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:47 smithi179 ceph-mon[43975]: 9.2 scrub ok 2024-02-09T18:02:47.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:47 smithi179 ceph-mon[43975]: 1.f scrub ok 2024-02-09T18:02:47.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:47 smithi179 ceph-mon[43975]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T18:02:47.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:47 smithi179 ceph-mon[43975]: Cluster is now healthy 2024-02-09T18:02:47.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:47 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T18:02:47.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:47 smithi179 ceph-mon[43975]: osdmap e932: 12 total, 12 up, 11 in 2024-02-09T18:02:47.951 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-09T18:02:48.028 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:02:48.031 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:48.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:02:48.132 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:02:48.195 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:48.234 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:02:48.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:48.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:02:48.340 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:02:48.374 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T18:02:48.378 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:48.405 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:02:48.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:48.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:02:48.464 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:48 smithi162 ceph-mon[44304]: 9.b scrub starts 2024-02-09T18:02:48.465 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:48 smithi162 ceph-mon[44304]: pgmap v2313: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.4 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 800 op/s 2024-02-09T18:02:48.513 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:02:48.548 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:48.572 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:02:48.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:48.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:02:48.680 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:02:48.714 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:48 smithi179 ceph-mon[43975]: 9.b scrub starts 2024-02-09T18:02:48.714 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:48 smithi179 ceph-mon[43975]: pgmap v2313: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.4 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 800 op/s 2024-02-09T18:02:48.716 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:48.743 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:02:48.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:48 smithi099 ceph-mon[39420]: 9.b scrub starts 2024-02-09T18:02:48.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:48 smithi099 ceph-mon[39420]: pgmap v2313: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.4 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 800 op/s 2024-02-09T18:02:48.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:48.751 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:02:48.852 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T18:02:48.900 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:48.946 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T18:02:48.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:48.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T18:02:48.958 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 5, 6, 7, 8, 9, 10, 11, 0, 2] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 7, 9, 10, 11, 8, 3] 2024-02-09T18:02:48.959 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T18:02:48.959 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-02-09T18:02:48.959 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd in 4 2024-02-09T18:02:49.050 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T18:02:49.100 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:49.133 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T18:02:49.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:49.136 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T18:02:49.237 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:02:49.273 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:49.302 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:02:49.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:49.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:02:49.340 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T18:02:49.354 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:02:49.410 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:02:49.459 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:49.616 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:49 smithi099 ceph-mon[39420]: 9.b scrub ok 2024-02-09T18:02:49.819 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:49 smithi162 ceph-mon[44304]: 9.b scrub ok 2024-02-09T18:02:49.912 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:02:49.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:49.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:02:49.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:49 smithi179 ceph-mon[43975]: 9.b scrub ok 2024-02-09T18:02:50.016 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set noscrub 2024-02-09T18:02:50.021 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:02:50.123 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:50.300 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:02:50.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:50.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:02:50.405 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T18:02:50.438 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:50.484 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T18:02:50.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:50.496 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T18:02:50.597 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:02:50.631 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:50 smithi099 ceph-mon[39420]: pgmap v2314: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 5.3 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 851 op/s 2024-02-09T18:02:50.633 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:50.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:50 smithi162 ceph-mon[44304]: pgmap v2314: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 5.3 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 851 op/s 2024-02-09T18:02:50.772 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T18:02:50.773 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-02-09T18:02:50.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-02-09T18:02:50.773 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-02-09T18:02:50.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.773 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.16440.0:141248 7.4 7:29871cf3:::benchmark_data_smithi179_135804_object33457:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e723)", 2024-02-09T18:02:50.773 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:52:50.763245+0000", 2024-02-09T18:02:50.773 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 599.99602761300002, 2024-02-09T18:02:50.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.73688722799999995, 2024-02-09T18:02:50.773 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:02:50.773 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:02:50.773 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T18:02:50.773 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.16440", 2024-02-09T18:02:50.773 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T18:02:50.774 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 141248 2024-02-09T18:02:50.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.774 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:02:50.774 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.774 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:02:50.774 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:50.763245+0000", 2024-02-09T18:02:50.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:02:50.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.774 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.774 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:02:50.774 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:50.763245+0000", 2024-02-09T18:02:50.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:02:50.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.774 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.775 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:02:50.775 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:50.763246+0000", 2024-02-09T18:02:50.775 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4559999999999999e-06 2024-02-09T18:02:50.775 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.775 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.775 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:02:50.775 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:50.763252+0000", 2024-02-09T18:02:50.775 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3730000000000001e-06 2024-02-09T18:02:50.775 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.775 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.775 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:02:50.775 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:50.763265+0000", 2024-02-09T18:02:50.775 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3726999999999999e-05 2024-02-09T18:02:50.775 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.775 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.776 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:02:50.776 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:50.763274+0000", 2024-02-09T18:02:50.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9090000000000006e-06 2024-02-09T18:02:50.776 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.776 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.776 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:02:50.776 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:50.763327+0000", 2024-02-09T18:02:50.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3238000000000001e-05 2024-02-09T18:02:50.776 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.776 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.776 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T18:02:50.776 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:50.763334+0000", 2024-02-09T18:02:50.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8159999999999996e-06 2024-02-09T18:02:50.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:02:50.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:51.471656+0000", 2024-02-09T18:02:50.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.70832204399999998 2024-02-09T18:02:50.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:02:50.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:51.471737+0000", 2024-02-09T18:02:50.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0729999999999994e-05 2024-02-09T18:02:50.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,9", 2024-02-09T18:02:50.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:51.471801+0000", 2024-02-09T18:02:50.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4272999999999999e-05 2024-02-09T18:02:50.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T18:02:50.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:51.474217+0000", 2024-02-09T18:02:50.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002415351 2024-02-09T18:02:50.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:02:50.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:51.496732+0000", 2024-02-09T18:02:50.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.022515545000000001 2024-02-09T18:02:50.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:02:50.779 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:51.500035+0000", 2024-02-09T18:02:50.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0033027030000000001 2024-02-09T18:02:50.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.779 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:02:50.779 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:51.500091+0000", 2024-02-09T18:02:50.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5875000000000003e-05 2024-02-09T18:02:50.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.779 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:02:50.779 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:51.500132+0000", 2024-02-09T18:02:50.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1187999999999999e-05 2024-02-09T18:02:50.779 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:02:50.779 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:02:50.779 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:02:50.780 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.780 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.780 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.16440.0:141264 7.4 7:26eb256d:::benchmark_data_smithi179_135804_object33473:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e723)", 2024-02-09T18:02:50.780 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:52:50.797302+0000", 2024-02-09T18:02:50.780 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 599.961969995, 2024-02-09T18:02:50.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.702938854, 2024-02-09T18:02:50.780 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:02:50.780 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:02:50.780 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T18:02:50.780 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.16440", 2024-02-09T18:02:50.780 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T18:02:50.780 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 141264 2024-02-09T18:02:50.780 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.781 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:02:50.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.781 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:02:50.781 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:50.797302+0000", 2024-02-09T18:02:50.781 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:02:50.781 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.781 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:02:50.781 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:50.797302+0000", 2024-02-09T18:02:50.781 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:02:50.781 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.781 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:02:50.782 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:50.797304+0000", 2024-02-09T18:02:50.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0159999999999998e-06 2024-02-09T18:02:50.782 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.782 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.782 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:02:50.782 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:50.797311+0000", 2024-02-09T18:02:50.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0929999999999997e-06 2024-02-09T18:02:50.782 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.782 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.782 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:02:50.782 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:50.797332+0000", 2024-02-09T18:02:50.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0506999999999999e-05 2024-02-09T18:02:50.782 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.782 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.782 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:02:50.782 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:50.797341+0000", 2024-02-09T18:02:50.783 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4190000000000008e-06 2024-02-09T18:02:50.783 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.783 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:02:50.783 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:50.797404+0000", 2024-02-09T18:02:50.783 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2890000000000003e-05 2024-02-09T18:02:50.783 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.783 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T18:02:50.783 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:50.797416+0000", 2024-02-09T18:02:50.783 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1687e-05 2024-02-09T18:02:50.783 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.783 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:02:50.783 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:51.474253+0000", 2024-02-09T18:02:50.783 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.67683691700000004 2024-02-09T18:02:50.784 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.784 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.784 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:02:50.784 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:51.474324+0000", 2024-02-09T18:02:50.784 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0618999999999996e-05 2024-02-09T18:02:50.784 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.784 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.784 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,9", 2024-02-09T18:02:50.784 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:51.474422+0000", 2024-02-09T18:02:50.784 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8517999999999994e-05 2024-02-09T18:02:50.784 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.784 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.784 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T18:02:50.784 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:51.481255+0000", 2024-02-09T18:02:50.784 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0068330939999999996 2024-02-09T18:02:50.784 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.785 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.785 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:02:50.785 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:51.496772+0000", 2024-02-09T18:02:50.785 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015516501 2024-02-09T18:02:50.785 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.785 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.785 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:02:50.785 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:51.500177+0000", 2024-02-09T18:02:50.785 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00340523 2024-02-09T18:02:50.785 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.785 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.785 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:02:50.785 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:51.500211+0000", 2024-02-09T18:02:50.785 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3665999999999999e-05 2024-02-09T18:02:50.785 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.785 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.785 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:02:50.786 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:52:51.500241+0000", 2024-02-09T18:02:50.786 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0697000000000001e-05 2024-02-09T18:02:50.786 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:02:50.786 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:02:50.786 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:02:50.786 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.786 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.786 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.16440.0:151795 7.1 7:8f72a001:::benchmark_data_smithi179_135804_object44004:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e738)", 2024-02-09T18:02:50.786 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:53:11.603516+0000", 2024-02-09T18:02:50.786 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 579.15575591899994, 2024-02-09T18:02:50.786 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0709706040000002, 2024-02-09T18:02:50.786 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:02:50.786 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-09T18:02:50.786 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T18:02:50.786 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.16440", 2024-02-09T18:02:50.787 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T18:02:50.787 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 151795 2024-02-09T18:02:50.787 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.787 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:02:50.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.787 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:02:50.787 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:11.603516+0000", 2024-02-09T18:02:50.787 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:02:50.787 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.787 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:02:50.787 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:11.603516+0000", 2024-02-09T18:02:50.787 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:02:50.787 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.787 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:02:50.787 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:11.603517+0000", 2024-02-09T18:02:50.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0900000000000001e-07 2024-02-09T18:02:50.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.788 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:02:50.788 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:11.603518+0000", 2024-02-09T18:02:50.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3489999999999999e-06 2024-02-09T18:02:50.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.788 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:02:50.788 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:11.603521+0000", 2024-02-09T18:02:50.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.202e-06 2024-02-09T18:02:50.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.788 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:02:50.789 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:11.603525+0000", 2024-02-09T18:02:50.789 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7179999999999998e-06 2024-02-09T18:02:50.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.789 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:02:50.789 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:11.608746+0000", 2024-02-09T18:02:50.789 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0052211439999999996 2024-02-09T18:02:50.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.789 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for missing object", 2024-02-09T18:02:50.789 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:11.608797+0000", 2024-02-09T18:02:50.789 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1087000000000001e-05 2024-02-09T18:02:50.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.790 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:02:50.790 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:14.674472+0000", 2024-02-09T18:02:50.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0656748600000001 2024-02-09T18:02:50.790 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.790 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.790 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:02:50.790 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:14.674487+0000", 2024-02-09T18:02:50.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5034999999999999e-05 2024-02-09T18:02:50.790 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:02:50.790 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:02:50.790 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:02:50.790 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.790 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.790 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.16440.0:151800 7.1 7:8df5fb9d:::benchmark_data_smithi179_135804_object44009:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e738)", 2024-02-09T18:02:50.791 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:53:11.603528+0000", 2024-02-09T18:02:50.791 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 579.15574398700005, 2024-02-09T18:02:50.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0879878120000002, 2024-02-09T18:02:50.791 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:02:50.791 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-09T18:02:50.791 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T18:02:50.791 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.16440", 2024-02-09T18:02:50.791 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T18:02:50.791 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 151800 2024-02-09T18:02:50.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.791 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:02:50.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:02:50.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:11.603528+0000", 2024-02-09T18:02:50.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:02:50.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.792 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:02:50.792 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:11.603528+0000", 2024-02-09T18:02:50.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:02:50.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.792 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:02:50.792 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:11.603529+0000", 2024-02-09T18:02:50.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.05e-07 2024-02-09T18:02:50.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.792 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:02:50.792 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:11.603530+0000", 2024-02-09T18:02:50.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0899999999999999e-06 2024-02-09T18:02:50.793 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.793 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.793 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:02:50.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:11.603535+0000", 2024-02-09T18:02:50.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2519999999999999e-06 2024-02-09T18:02:50.793 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.793 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.793 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:02:50.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:11.603538+0000", 2024-02-09T18:02:50.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8880000000000001e-06 2024-02-09T18:02:50.793 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.793 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.793 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:02:50.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:11.608830+0000", 2024-02-09T18:02:50.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0052918560000000002 2024-02-09T18:02:50.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.794 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for missing object", 2024-02-09T18:02:50.794 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:11.608885+0000", 2024-02-09T18:02:50.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5036e-05 2024-02-09T18:02:50.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.794 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:02:50.794 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:14.691507+0000", 2024-02-09T18:02:50.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0826226220000001 2024-02-09T18:02:50.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.794 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:02:50.794 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:14.691516+0000", 2024-02-09T18:02:50.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8629999999999997e-06 2024-02-09T18:02:50.794 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:02:50.795 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:02:50.795 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:02:50.795 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.795 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.795 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.16440.0:163103 7.1 7:8cf911dd:::benchmark_data_smithi179_135804_object55312:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e746)", 2024-02-09T18:02:50.795 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:53:29.217038+0000", 2024-02-09T18:02:50.795 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 561.54223462100003, 2024-02-09T18:02:50.795 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.82073820500000005, 2024-02-09T18:02:50.795 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:02:50.795 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-09T18:02:50.795 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T18:02:50.795 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.16440", 2024-02-09T18:02:50.795 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T18:02:50.795 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 163103 2024-02-09T18:02:50.795 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.796 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:02:50.796 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.796 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:02:50.796 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:29.217038+0000", 2024-02-09T18:02:50.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:02:50.796 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.796 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.796 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:02:50.796 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:29.217038+0000", 2024-02-09T18:02:50.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:02:50.796 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.796 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.796 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:02:50.796 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:29.217038+0000", 2024-02-09T18:02:50.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4499999999999997e-07 2024-02-09T18:02:50.797 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.797 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.797 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:02:50.797 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:29.217040+0000", 2024-02-09T18:02:50.797 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0789999999999999e-06 2024-02-09T18:02:50.797 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.797 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.797 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:02:50.797 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:29.217046+0000", 2024-02-09T18:02:50.797 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8699999999999997e-06 2024-02-09T18:02:50.797 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.797 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.797 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:02:50.797 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:29.217052+0000", 2024-02-09T18:02:50.797 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1680000000000001e-06 2024-02-09T18:02:50.797 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.798 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.798 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:02:50.798 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:29.228805+0000", 2024-02-09T18:02:50.798 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011752291 2024-02-09T18:02:50.798 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.798 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.798 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for missing object", 2024-02-09T18:02:50.798 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:29.228904+0000", 2024-02-09T18:02:50.798 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9148999999999995e-05 2024-02-09T18:02:50.798 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.798 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.798 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:02:50.798 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:30.037764+0000", 2024-02-09T18:02:50.798 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.80886064099999999 2024-02-09T18:02:50.798 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.799 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:02:50.799 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:30.037776+0000", 2024-02-09T18:02:50.799 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1562e-05 2024-02-09T18:02:50.799 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:02:50.799 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:02:50.799 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:02:50.799 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.799 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.16440.0:163111 7.1 7:87b5931b:::benchmark_data_smithi179_135804_object55320:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e746)", 2024-02-09T18:02:50.799 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:53:29.217058+0000", 2024-02-09T18:02:50.799 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 561.54221427899995, 2024-02-09T18:02:50.799 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.82342750300000001, 2024-02-09T18:02:50.799 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:02:50.799 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-09T18:02:50.800 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T18:02:50.800 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.16440", 2024-02-09T18:02:50.800 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T18:02:50.800 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 163111 2024-02-09T18:02:50.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.800 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:02:50.800 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.800 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:02:50.800 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:29.217058+0000", 2024-02-09T18:02:50.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:02:50.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.800 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.800 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:02:50.800 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:29.217058+0000", 2024-02-09T18:02:50.801 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:02:50.801 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.801 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.801 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:02:50.801 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:29.217058+0000", 2024-02-09T18:02:50.801 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.39e-07 2024-02-09T18:02:50.801 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.801 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.801 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:02:50.801 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:29.217061+0000", 2024-02-09T18:02:50.801 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0600000000000002e-06 2024-02-09T18:02:50.801 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.801 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.801 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:02:50.801 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:29.217067+0000", 2024-02-09T18:02:50.802 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3130000000000004e-06 2024-02-09T18:02:50.802 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.802 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.802 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:02:50.802 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:29.217072+0000", 2024-02-09T18:02:50.802 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7269999999999998e-06 2024-02-09T18:02:50.802 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.802 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.802 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:02:50.802 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:29.228948+0000", 2024-02-09T18:02:50.802 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011876640000000001 2024-02-09T18:02:50.802 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.802 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.803 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for missing object", 2024-02-09T18:02:50.803 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:29.229036+0000", 2024-02-09T18:02:50.803 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7712000000000003e-05 2024-02-09T18:02:50.803 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.803 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.803 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:02:50.803 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:30.040471+0000", 2024-02-09T18:02:50.803 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.81143475799999998 2024-02-09T18:02:50.803 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:50.803 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:50.803 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:02:50.803 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:30.040486+0000", 2024-02-09T18:02:50.803 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4953999999999999e-05 2024-02-09T18:02:50.803 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:02:50.803 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:02:50.803 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:02:50.804 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:02:50.804 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:02:50.804 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T18:02:50.916 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:02:50.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:50 smithi179 ceph-mon[43975]: pgmap v2314: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 5.3 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 851 op/s 2024-02-09T18:02:50.969 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:02:50.986 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:02:50.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:50.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:02:51.092 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:02:51.128 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:51.502 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:02:51.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:51.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:02:51.519 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:51 smithi099 ceph-mon[39420]: 1.1b scrub starts 2024-02-09T18:02:51.519 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:51 smithi099 ceph-mon[39420]: 1.1b scrub ok 2024-02-09T18:02:51.539 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-09T18:02:51.608 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T18:02:51.678 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:51.712 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T18:02:51.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:51.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T18:02:51.820 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:02:51.877 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:51 smithi162 ceph-mon[44304]: 1.1b scrub starts 2024-02-09T18:02:51.877 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:51 smithi162 ceph-mon[44304]: 1.1b scrub ok 2024-02-09T18:02:51.877 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:51.902 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:02:51.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:51.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:02:51.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:51 smithi179 ceph-mon[43975]: 1.1b scrub starts 2024-02-09T18:02:51.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:51 smithi179 ceph-mon[43975]: 1.1b scrub ok 2024-02-09T18:02:51.983 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T18:02:52.007 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:02:52.061 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:52.095 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:02:52.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:52.102 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:02:52.203 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:02:52.253 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:52.303 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:02:52.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:52.310 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:02:52.411 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:02:52.481 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:52.517 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:02:52.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:52.524 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:02:52.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:52 smithi162 ceph-mon[44304]: 9.e scrub starts 2024-02-09T18:02:52.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:52 smithi162 ceph-mon[44304]: pgmap v2315: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.2 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 839 op/s 2024-02-09T18:02:52.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:52 smithi162 ceph-mon[44304]: 1.18 scrub starts 2024-02-09T18:02:52.550 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:52 smithi162 ceph-mon[44304]: 1.18 scrub ok 2024-02-09T18:02:52.625 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T18:02:52.670 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:52.877 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T18:02:52.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:52.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T18:02:52.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:52 smithi179 ceph-mon[43975]: 9.e scrub starts 2024-02-09T18:02:52.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:52 smithi179 ceph-mon[43975]: pgmap v2315: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.2 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 839 op/s 2024-02-09T18:02:52.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:52 smithi179 ceph-mon[43975]: 1.18 scrub starts 2024-02-09T18:02:52.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:52 smithi179 ceph-mon[43975]: 1.18 scrub ok 2024-02-09T18:02:52.983 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:02:52.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:52 smithi099 ceph-mon[39420]: 9.e scrub starts 2024-02-09T18:02:52.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:52 smithi099 ceph-mon[39420]: pgmap v2315: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.2 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 839 op/s 2024-02-09T18:02:52.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:52 smithi099 ceph-mon[39420]: 1.18 scrub starts 2024-02-09T18:02:52.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:52 smithi099 ceph-mon[39420]: 1.18 scrub ok 2024-02-09T18:02:53.031 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:53.646 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:53 smithi162 ceph-mon[44304]: 1.1c scrub starts 2024-02-09T18:02:53.646 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:53 smithi162 ceph-mon[44304]: 1.1c scrub ok 2024-02-09T18:02:53.646 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:53 smithi162 ceph-mon[44304]: 9.0 scrub starts 2024-02-09T18:02:53.646 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:53 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:02:53.646 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:53 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:02:53.646 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:53 smithi162 ceph-mon[44304]: 9.8 scrub starts 2024-02-09T18:02:53.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:53 smithi179 ceph-mon[43975]: 1.1c scrub starts 2024-02-09T18:02:53.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:53 smithi179 ceph-mon[43975]: 1.1c scrub ok 2024-02-09T18:02:53.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:53 smithi179 ceph-mon[43975]: 9.0 scrub starts 2024-02-09T18:02:53.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:53 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:02:53.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:53 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:02:53.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:53 smithi179 ceph-mon[43975]: 9.8 scrub starts 2024-02-09T18:02:53.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:53 smithi099 ceph-mon[39420]: 1.1c scrub starts 2024-02-09T18:02:53.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:53 smithi099 ceph-mon[39420]: 1.1c scrub ok 2024-02-09T18:02:53.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:53 smithi099 ceph-mon[39420]: 9.0 scrub starts 2024-02-09T18:02:53.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:53 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:02:53.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:53 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:02:53.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:53 smithi099 ceph-mon[39420]: 9.8 scrub starts 2024-02-09T18:02:54.003 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T18:02:54.003 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-02-09T18:02:54.003 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:54.003 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:188583 9.d e933/882 9:b2c811ab:::benchmark_data_smithi179_161915_object53701:head v 933'11685, mlcod=933'11683)", 2024-02-09T18:02:54.003 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:02:53.986720+0000", 2024-02-09T18:02:54.003 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.012926148, 2024-02-09T18:02:54.003 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012956387999999999, 2024-02-09T18:02:54.003 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:02:54.004 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-09T18:02:54.004 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:02:54.004 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:54.004 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:02:54.004 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:53.986720+0000", 2024-02-09T18:02:54.004 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:02:54.004 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:54.004 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:54.004 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:02:54.004 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:53.986720+0000", 2024-02-09T18:02:54.004 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:02:54.004 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:54.004 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:54.005 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:02:54.005 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:53.986720+0000", 2024-02-09T18:02:54.005 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.06e-07 2024-02-09T18:02:54.005 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:54.005 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:54.005 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:02:54.005 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:53.986726+0000", 2024-02-09T18:02:54.005 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3959999999999997e-06 2024-02-09T18:02:54.005 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:54.005 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:54.005 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:02:54.005 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:53.986737+0000", 2024-02-09T18:02:54.005 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0829000000000001e-05 2024-02-09T18:02:54.005 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:54.006 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:54.006 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:02:54.006 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:53.986743+0000", 2024-02-09T18:02:54.006 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2879999999999998e-06 2024-02-09T18:02:54.006 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:54.006 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:54.006 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:02:54.006 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:53.986793+0000", 2024-02-09T18:02:54.006 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0346999999999998e-05 2024-02-09T18:02:54.006 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:54.006 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:54.006 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:02:54.006 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:53.986811+0000", 2024-02-09T18:02:54.007 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7544999999999999e-05 2024-02-09T18:02:54.007 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:02:54.007 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:02:54.007 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:02:54.007 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:54.007 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:54.008 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:188577 9.8 e933/882 9:123e66d9:::benchmark_data_smithi179_161915_object53695:head v 933'12013, mlcod=933'12009)", 2024-02-09T18:02:54.008 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:02:53.989824+0000", 2024-02-09T18:02:54.008 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0098219350000000004, 2024-02-09T18:02:54.008 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0099224129999999997, 2024-02-09T18:02:54.008 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:02:54.008 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-09T18:02:54.008 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:02:54.008 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:54.008 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:02:54.008 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:53.989824+0000", 2024-02-09T18:02:54.008 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:02:54.008 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:54.008 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:54.009 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:02:54.009 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:53.989824+0000", 2024-02-09T18:02:54.009 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:02:54.009 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:54.009 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:54.009 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:02:54.009 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:53.989825+0000", 2024-02-09T18:02:54.010 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1480000000000001e-06 2024-02-09T18:02:54.010 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:54.010 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:54.010 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:02:54.010 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:53.989832+0000", 2024-02-09T18:02:54.011 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3289999999999996e-06 2024-02-09T18:02:54.011 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:54.011 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:54.011 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:02:54.011 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:53.989847+0000", 2024-02-09T18:02:54.011 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4532000000000001e-05 2024-02-09T18:02:54.012 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:54.012 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:54.012 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:02:54.012 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:53.989856+0000", 2024-02-09T18:02:54.012 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0399999999999998e-06 2024-02-09T18:02:54.012 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:54.012 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:54.012 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:02:54.012 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:53.989944+0000", 2024-02-09T18:02:54.012 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7628999999999997e-05 2024-02-09T18:02:54.012 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:54.013 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:54.013 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:02:54.013 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:53.989971+0000", 2024-02-09T18:02:54.013 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7430999999999999e-05 2024-02-09T18:02:54.013 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:02:54.013 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:02:54.014 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:02:54.014 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:54.014 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:54.014 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:188584 9.5 e933/882 9:a4f3fc20:::benchmark_data_smithi179_161915_object53702:head v 933'11842, mlcod=933'11840)", 2024-02-09T18:02:54.014 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:02:53.995241+0000", 2024-02-09T18:02:54.014 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0044052049999999997, 2024-02-09T18:02:54.014 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0045677959999999998, 2024-02-09T18:02:54.014 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:02:54.014 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-09T18:02:54.014 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:02:54.014 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:54.014 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:02:54.014 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:53.995241+0000", 2024-02-09T18:02:54.015 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:02:54.015 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:54.015 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:54.015 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:02:54.015 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:53.995241+0000", 2024-02-09T18:02:54.015 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:02:54.015 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:54.015 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:54.015 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:02:54.015 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:53.995242+0000", 2024-02-09T18:02:54.015 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4389999999999999e-06 2024-02-09T18:02:54.015 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:54.015 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:54.015 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:02:54.015 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:53.995250+0000", 2024-02-09T18:02:54.016 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8359999999999992e-06 2024-02-09T18:02:54.016 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:54.016 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:54.016 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:02:54.016 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:53.995263+0000", 2024-02-09T18:02:54.016 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3144e-05 2024-02-09T18:02:54.016 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:54.016 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:54.016 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:02:54.016 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:53.995272+0000", 2024-02-09T18:02:54.016 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7169999999999997e-06 2024-02-09T18:02:54.016 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:54.016 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:54.017 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:02:54.017 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:53.995343+0000", 2024-02-09T18:02:54.017 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0743999999999999e-05 2024-02-09T18:02:54.017 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:02:54.017 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:02:54.017 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:02:54.017 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:53.995362+0000", 2024-02-09T18:02:54.017 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9833000000000001e-05 2024-02-09T18:02:54.017 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:02:54.017 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:02:54.017 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:02:54.017 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:02:54.017 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-02-09T18:02:54.017 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 3 2024-02-09T18:02:54.018 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T18:02:54.264 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:02:54.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:54.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:02:54.374 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:02:54.411 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:54.527 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:54 smithi099 ceph-mon[39420]: 9.e scrub ok 2024-02-09T18:02:54.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:54 smithi162 ceph-mon[44304]: 9.e scrub ok 2024-02-09T18:02:54.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:54 smithi162 ceph-mon[44304]: 1.19 scrub starts 2024-02-09T18:02:54.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:54 smithi162 ceph-mon[44304]: pgmap v2316: 47 pgs: 47 active+clean; 5.1 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 839 op/s 2024-02-09T18:02:54.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:54 smithi162 ceph-mon[44304]: 1.19 scrub ok 2024-02-09T18:02:54.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:54 smithi162 ceph-mon[44304]: osdmap e933: 12 total, 12 up, 11 in 2024-02-09T18:02:54.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:54 smithi162 ceph-mon[44304]: 9.0 scrub ok 2024-02-09T18:02:54.749 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-09T18:02:54.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:54 smithi179 ceph-mon[43975]: 9.e scrub ok 2024-02-09T18:02:54.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:54 smithi179 ceph-mon[43975]: 1.19 scrub starts 2024-02-09T18:02:54.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:54 smithi179 ceph-mon[43975]: pgmap v2316: 47 pgs: 47 active+clean; 5.1 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 839 op/s 2024-02-09T18:02:54.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:54 smithi179 ceph-mon[43975]: 1.19 scrub ok 2024-02-09T18:02:54.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:54 smithi179 ceph-mon[43975]: osdmap e933: 12 total, 12 up, 11 in 2024-02-09T18:02:54.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:54 smithi179 ceph-mon[43975]: 9.0 scrub ok 2024-02-09T18:02:54.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:54 smithi099 ceph-mon[39420]: 1.19 scrub starts 2024-02-09T18:02:54.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:54 smithi099 ceph-mon[39420]: pgmap v2316: 47 pgs: 47 active+clean; 5.1 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 839 op/s 2024-02-09T18:02:54.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:54 smithi099 ceph-mon[39420]: 1.19 scrub ok 2024-02-09T18:02:54.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:54 smithi099 ceph-mon[39420]: osdmap e933: 12 total, 12 up, 11 in 2024-02-09T18:02:54.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:54 smithi099 ceph-mon[39420]: 9.0 scrub ok 2024-02-09T18:02:55.185 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T18:02:55.803 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:02:55.805 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:55 smithi099 ceph-mon[39420]: 9.8 scrub ok 2024-02-09T18:02:55.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:55.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:02:55.909 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T18:02:55.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:55 smithi179 ceph-mon[43975]: 9.8 scrub ok 2024-02-09T18:02:55.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:55 smithi162 ceph-mon[44304]: 9.8 scrub ok 2024-02-09T18:02:55.937 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:56.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:56 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:02:56.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:56 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:02:56.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:56 smithi099 ceph-mon[39420]: pgmap v2318: 47 pgs: 47 active+clean; 5.0 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 739 op/s 2024-02-09T18:02:56.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:56 smithi099 ceph-mon[39420]: osdmap e934: 12 total, 12 up, 11 in 2024-02-09T18:02:56.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:56 smithi099 ceph-mon[39420]: 9.9 scrub starts 2024-02-09T18:02:56.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:56 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:02:56.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:56 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:02:56.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:56 smithi179 ceph-mon[43975]: pgmap v2318: 47 pgs: 47 active+clean; 5.0 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 739 op/s 2024-02-09T18:02:56.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:56 smithi179 ceph-mon[43975]: osdmap e934: 12 total, 12 up, 11 in 2024-02-09T18:02:56.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:56 smithi179 ceph-mon[43975]: 9.9 scrub starts 2024-02-09T18:02:56.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:56 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:02:56.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:56 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:02:56.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:56 smithi162 ceph-mon[44304]: pgmap v2318: 47 pgs: 47 active+clean; 5.0 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 739 op/s 2024-02-09T18:02:56.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:56 smithi162 ceph-mon[44304]: osdmap e934: 12 total, 12 up, 11 in 2024-02-09T18:02:56.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:56 smithi162 ceph-mon[44304]: 9.9 scrub starts 2024-02-09T18:02:57.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:57 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mon-a[39397]: 2024-02-09T18:02:57.377+0000 7feb57590700 -1 mon.a@0(leader).osd e934 definitely_dead 0 2024-02-09T18:02:57.882 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T18:02:57.919 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T18:02:57.949 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T18:02:57.978 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T18:02:58.009 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T18:02:58.036 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T18:02:58.073 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T18:02:58.110 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T18:02:58.113 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is set 2024-02-09T18:02:58.127 INFO:teuthology.orchestra.run.smithi099.stderr:marked in osd.4. 2024-02-09T18:02:58.141 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T18:02:58.169 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T18:02:58.195 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T18:02:58.218 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T18:02:58.220 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T18:02:58.220 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T18:02:58.220 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T18:02:58.220 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T18:02:58.220 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T18:02:58.220 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T18:02:58.220 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T18:02:58.220 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T18:02:58.220 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T18:02:58.220 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T18:02:58.220 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T18:02:58.220 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T18:02:58.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:58 smithi179 ceph-mon[43975]: pgmap v2320: 47 pgs: 47 active+clean; 4.9 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 786 op/s 2024-02-09T18:02:58.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:58 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3039984134' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-02-09T18:02:58.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:58 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-02-09T18:02:58.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:58 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1395735922' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T18:02:58.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:58 smithi179 ceph-mon[43975]: 9.9 scrub ok 2024-02-09T18:02:58.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:58 smithi179 ceph-mon[43975]: 1.c scrub starts 2024-02-09T18:02:58.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:58 smithi179 ceph-mon[43975]: 1.c scrub ok 2024-02-09T18:02:58.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:58 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1544002092' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:02:58.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:58 smithi162 ceph-mon[44304]: pgmap v2320: 47 pgs: 47 active+clean; 4.9 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 786 op/s 2024-02-09T18:02:58.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:58 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3039984134' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-02-09T18:02:58.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:58 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-02-09T18:02:58.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:58 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1395735922' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T18:02:58.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:58 smithi162 ceph-mon[44304]: 9.9 scrub ok 2024-02-09T18:02:58.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:58 smithi162 ceph-mon[44304]: 1.c scrub starts 2024-02-09T18:02:58.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:58 smithi162 ceph-mon[44304]: 1.c scrub ok 2024-02-09T18:02:58.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:58 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1544002092' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:02:58.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:58 smithi099 ceph-mon[39420]: pgmap v2320: 47 pgs: 47 active+clean; 4.9 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 786 op/s 2024-02-09T18:02:58.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:58 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3039984134' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-02-09T18:02:58.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:58 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-02-09T18:02:58.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:58 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1395735922' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T18:02:58.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:58 smithi099 ceph-mon[39420]: 9.9 scrub ok 2024-02-09T18:02:58.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:58 smithi099 ceph-mon[39420]: 1.c scrub starts 2024-02-09T18:02:58.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:58 smithi099 ceph-mon[39420]: 1.c scrub ok 2024-02-09T18:02:58.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:58 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1544002092' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:02:58.589 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T18:02:58.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:58.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T18:02:58.693 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:02:58.720 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:58.740 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:02:58.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:58.743 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:02:58.844 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:02:58.875 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:59.075 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:02:59.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:59.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:02:59.105 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-02-09T18:02:59.179 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T18:02:59.207 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:59.332 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T18:02:59.332 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:59 smithi162 ceph-mon[44304]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:02:59.332 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:59 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-02-09T18:02:59.333 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:59 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1395735922' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T18:02:59.333 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:59 smithi162 ceph-mon[44304]: osdmap e935: 12 total, 12 up, 12 in 2024-02-09T18:02:59.333 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:02:59 smithi162 ceph-mon[44304]: osdmap e936: 12 total, 12 up, 12 in 2024-02-09T18:02:59.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:59.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T18:02:59.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:59 smithi179 ceph-mon[43975]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:02:59.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:59 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-02-09T18:02:59.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:59 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1395735922' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T18:02:59.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:59 smithi179 ceph-mon[43975]: osdmap e935: 12 total, 12 up, 12 in 2024-02-09T18:02:59.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:02:59 smithi179 ceph-mon[43975]: osdmap e936: 12 total, 12 up, 12 in 2024-02-09T18:02:59.436 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T18:02:59.465 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:59.484 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T18:02:59.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:59.487 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T18:02:59.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:59 smithi099 ceph-mon[39420]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:02:59.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:59 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-02-09T18:02:59.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:59 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1395735922' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T18:02:59.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:59 smithi099 ceph-mon[39420]: osdmap e935: 12 total, 12 up, 12 in 2024-02-09T18:02:59.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:02:59 smithi099 ceph-mon[39420]: osdmap e936: 12 total, 12 up, 12 in 2024-02-09T18:02:59.587 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T18:02:59.617 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:59.637 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T18:02:59.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:59.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T18:02:59.676 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T18:02:59.677 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T18:02:59.741 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:02:59.769 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:02:59.788 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:02:59.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:02:59.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:02:59.892 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T18:02:59.920 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:00.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:00 smithi162 ceph-mon[44304]: pgmap v2323: 47 pgs: 1 peering, 46 active+clean; 4.8 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 861 op/s 2024-02-09T18:03:00.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:00 smithi162 ceph-mon[44304]: osdmap e937: 12 total, 12 up, 12 in 2024-02-09T18:03:00.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:00 smithi099 ceph-mon[39420]: pgmap v2323: 47 pgs: 1 peering, 46 active+clean; 4.8 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 861 op/s 2024-02-09T18:03:00.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:00 smithi099 ceph-mon[39420]: osdmap e937: 12 total, 12 up, 12 in 2024-02-09T18:03:00.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:00 smithi179 ceph-mon[43975]: pgmap v2323: 47 pgs: 1 peering, 46 active+clean; 4.8 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 861 op/s 2024-02-09T18:03:00.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:00 smithi179 ceph-mon[43975]: osdmap e937: 12 total, 12 up, 12 in 2024-02-09T18:03:00.674 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T18:03:00.677 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-09T18:03:00.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:00.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T18:03:00.781 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:03:00.843 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:00.888 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:03:00.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:00.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:03:00.996 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T18:03:01.049 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:01.079 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T18:03:01.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:01.084 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T18:03:01.166 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T18:03:01.183 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:01 smithi162 ceph-mon[44304]: 9.c starting backfill to osd.4 from (0'0,0'0] MAX to 934'12070 2024-02-09T18:03:01.183 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:01 smithi162 ceph-mon[44304]: 9.4 starting backfill to osd.4 from (0'0,0'0] MAX to 934'11863 2024-02-09T18:03:01.184 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:03:01.230 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:01.256 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:03:01.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:01.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:03:01.364 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:03:01.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:01 smithi179 ceph-mon[43975]: 9.c starting backfill to osd.4 from (0'0,0'0] MAX to 934'12070 2024-02-09T18:03:01.423 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:01 smithi179 ceph-mon[43975]: 9.4 starting backfill to osd.4 from (0'0,0'0] MAX to 934'11863 2024-02-09T18:03:01.435 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:01.471 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:03:01.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:01.478 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:03:01.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:01 smithi099 ceph-mon[39420]: 9.c starting backfill to osd.4 from (0'0,0'0] MAX to 934'12070 2024-02-09T18:03:01.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:01 smithi099 ceph-mon[39420]: 9.4 starting backfill to osd.4 from (0'0,0'0] MAX to 934'11863 2024-02-09T18:03:01.578 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:03:01.629 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-09T18:03:01.630 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:01.653 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:03:01.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:01.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:03:01.759 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T18:03:01.803 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:01.842 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T18:03:01.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:01.850 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T18:03:01.951 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T18:03:02.004 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:02.048 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:03:02.127 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:02 smithi179 ceph-mon[43975]: osdmap e938: 12 total, 12 up, 12 in 2024-02-09T18:03:02.144 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T18:03:02.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:02.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T18:03:02.151 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:02 smithi162 ceph-mon[44304]: osdmap e938: 12 total, 12 up, 12 in 2024-02-09T18:03:02.151 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:02 smithi162 ceph-mon[44304]: pgmap v2326: 47 pgs: 2 active+remapped+backfilling, 3 active+undersized+remapped, 1 peering, 41 active+clean; 4.7 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 745 op/s; 9613/232848 objects misplaced (4.128%); 2.0 MiB/s, 32 objects/s recovering 2024-02-09T18:03:02.151 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:02 smithi162 ceph-mon[44304]: osdmap e939: 12 total, 12 up, 12 in 2024-02-09T18:03:02.251 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T18:03:02.282 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:02.317 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T18:03:02.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:02.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T18:03:02.374 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T18:03:02.420 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:02 smithi099 ceph-mon[39420]: osdmap e938: 12 total, 12 up, 12 in 2024-02-09T18:03:02.420 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:02 smithi099 ceph-mon[39420]: pgmap v2326: 47 pgs: 2 active+remapped+backfilling, 3 active+undersized+remapped, 1 peering, 41 active+clean; 4.7 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 745 op/s; 9613/232848 objects misplaced (4.128%); 2.0 MiB/s, 32 objects/s recovering 2024-02-09T18:03:02.420 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:02 smithi099 ceph-mon[39420]: osdmap e939: 12 total, 12 up, 12 in 2024-02-09T18:03:02.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:02 smithi179 ceph-mon[43975]: pgmap v2326: 47 pgs: 2 active+remapped+backfilling, 3 active+undersized+remapped, 1 peering, 41 active+clean; 4.7 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 745 op/s; 9613/232848 objects misplaced (4.128%); 2.0 MiB/s, 32 objects/s recovering 2024-02-09T18:03:02.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:02 smithi179 ceph-mon[43975]: osdmap e939: 12 total, 12 up, 12 in 2024-02-09T18:03:02.430 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:03:02.473 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:02.637 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:03:02.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:02.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:03:02.744 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T18:03:02.796 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:02.970 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T18:03:02.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:02.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T18:03:03.076 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:03:03.120 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:03.156 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:03:03.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:03.160 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:03:03.261 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T18:03:03.304 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:03.336 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T18:03:03.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:03.341 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T18:03:03.442 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T18:03:03.500 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:03.536 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T18:03:03.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:03.540 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T18:03:03.641 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:03:03.652 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T18:03:03.652 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T18:03:03.681 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:04.040 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:03:04.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:04.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:03:04.106 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 5, 6, 7, 8, 9, 10, 11, 0, 2, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 7, 9, 10, 11, 8, 3] 2024-02-09T18:03:04.106 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T18:03:04.106 INFO:tasks.thrashosds.thrasher:Removing osd 10, in_osds are: [1, 3, 5, 6, 7, 8, 9, 10, 11, 0, 2, 4] 2024-02-09T18:03:04.107 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd out 10 2024-02-09T18:03:04.170 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:03:04.207 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:04.234 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:03:04.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:04.238 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:03:04.338 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:03:04.378 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:04 smithi099 ceph-mon[39420]: pgmap v2328: 47 pgs: 2 active+remapped+backfilling, 2 active+undersized+remapped, 4 peering, 39 active+clean; 4.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 484 B/s rd, 16 KiB/s wr, 433 op/s; 9613/230613 objects misplaced (4.168%); 1.9 MiB/s, 30 objects/s recovering 2024-02-09T18:03:04.382 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:04 smithi162 ceph-mon[44304]: pgmap v2328: 47 pgs: 2 active+remapped+backfilling, 2 active+undersized+remapped, 4 peering, 39 active+clean; 4.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 484 B/s rd, 16 KiB/s wr, 433 op/s; 9613/230613 objects misplaced (4.168%); 1.9 MiB/s, 30 objects/s recovering 2024-02-09T18:03:04.384 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:04.428 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-09T18:03:04.676 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:04 smithi179 ceph-mon[43975]: pgmap v2328: 47 pgs: 2 active+remapped+backfilling, 2 active+undersized+remapped, 4 peering, 39 active+clean; 4.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 484 B/s rd, 16 KiB/s wr, 433 op/s; 9613/230613 objects misplaced (4.168%); 1.9 MiB/s, 30 objects/s recovering 2024-02-09T18:03:04.844 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T18:03:05.154 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:03:05.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:05.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:03:05.260 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T18:03:05.309 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:05.356 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T18:03:05.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:05.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T18:03:05.461 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:03:05.507 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:03:05.508 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:05.738 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:03:05.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:05.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:03:05.844 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:03:05.894 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:05.974 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:03:05.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:05.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:03:06.082 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:03:06.148 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:06.271 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:03:06.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:06.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:03:06.275 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:03:06.376 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:03:06.395 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:06 smithi162 ceph-mon[44304]: pgmap v2329: 47 pgs: 2 active+remapped+backfilling, 2 active+undersized+remapped, 3 peering, 40 active+clean; 4.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 11 KiB/s wr, 438 op/s; 9613/228216 objects misplaced (4.212%); 1.3 MiB/s, 21 objects/s recovering 2024-02-09T18:03:06.396 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:06 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:03:06.396 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:06 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:03:06.407 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:06 smithi099 ceph-mon[39420]: pgmap v2329: 47 pgs: 2 active+remapped+backfilling, 2 active+undersized+remapped, 3 peering, 40 active+clean; 4.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 11 KiB/s wr, 438 op/s; 9613/228216 objects misplaced (4.212%); 1.3 MiB/s, 21 objects/s recovering 2024-02-09T18:03:06.407 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:06 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:03:06.407 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:06 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:03:06.442 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:06.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:06 smithi179 ceph-mon[43975]: pgmap v2329: 47 pgs: 2 active+remapped+backfilling, 2 active+undersized+remapped, 3 peering, 40 active+clean; 4.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 11 KiB/s wr, 438 op/s; 9613/228216 objects misplaced (4.212%); 1.3 MiB/s, 21 objects/s recovering 2024-02-09T18:03:06.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:06 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:03:06.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:06 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:03:06.837 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T18:03:06.838 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T18:03:07.409 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:03:07.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:07.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:03:07.442 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is set 2024-02-09T18:03:07.517 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:03:07.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:07 smithi162 ceph-mon[44304]: osdmap e940: 12 total, 12 up, 12 in 2024-02-09T18:03:07.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:07 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:03:07.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:07 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:03:07.560 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:07 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/554493313' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T18:03:07.569 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:07.599 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-09T18:03:07.614 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:03:07.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:07.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:03:07.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:07 smithi179 ceph-mon[43975]: osdmap e940: 12 total, 12 up, 12 in 2024-02-09T18:03:07.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:07 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:03:07.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:07 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:03:07.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:07 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/554493313' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T18:03:07.718 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:03:07.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:07 smithi099 ceph-mon[39420]: osdmap e940: 12 total, 12 up, 12 in 2024-02-09T18:03:07.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:07 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:03:07.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:07 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:03:07.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:07 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/554493313' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T18:03:07.779 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:08.046 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T18:03:08.655 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:08 smithi162 ceph-mon[44304]: pgmap v2331: 47 pgs: 2 active+remapped+backfilling, 45 active+clean; 4.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 329 B/s rd, 11 KiB/s wr, 630 op/s; 7850/222072 objects misplaced (3.535%); 14 MiB/s, 219 objects/s recovering 2024-02-09T18:03:08.655 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:08 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:03:08.655 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:08 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:03:08.655 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:08 smithi162 ceph-mon[44304]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:03:08.655 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:08 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/554493313' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T18:03:08.655 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:08 smithi162 ceph-mon[44304]: osdmap e941: 12 total, 12 up, 12 in 2024-02-09T18:03:08.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:08 smithi179 ceph-mon[43975]: pgmap v2331: 47 pgs: 2 active+remapped+backfilling, 45 active+clean; 4.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 329 B/s rd, 11 KiB/s wr, 630 op/s; 7850/222072 objects misplaced (3.535%); 14 MiB/s, 219 objects/s recovering 2024-02-09T18:03:08.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:08 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:03:08.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:08 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:03:08.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:08 smithi179 ceph-mon[43975]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:03:08.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:08 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/554493313' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T18:03:08.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:08 smithi179 ceph-mon[43975]: osdmap e941: 12 total, 12 up, 12 in 2024-02-09T18:03:08.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:08 smithi099 ceph-mon[39420]: pgmap v2331: 47 pgs: 2 active+remapped+backfilling, 45 active+clean; 4.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 329 B/s rd, 11 KiB/s wr, 630 op/s; 7850/222072 objects misplaced (3.535%); 14 MiB/s, 219 objects/s recovering 2024-02-09T18:03:08.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:08 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:03:08.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:08 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:03:08.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:08 smithi099 ceph-mon[39420]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:03:08.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:08 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/554493313' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T18:03:08.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:08 smithi099 ceph-mon[39420]: osdmap e941: 12 total, 12 up, 12 in 2024-02-09T18:03:09.955 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:03:09.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:09.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:03:10.063 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:03:10.099 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:10.259 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T18:03:10.259 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T18:03:10.458 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:03:10.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:10.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:03:10.569 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:03:10.623 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:10.642 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:10 smithi162 ceph-mon[44304]: pgmap v2333: 47 pgs: 2 active+remapped+backfilling, 45 active+clean; 4.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 798 op/s; 7850/215583 objects misplaced (3.641%); 11 MiB/s, 177 objects/s recovering 2024-02-09T18:03:10.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:10 smithi099 ceph-mon[39420]: pgmap v2333: 47 pgs: 2 active+remapped+backfilling, 45 active+clean; 4.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 798 op/s; 7850/215583 objects misplaced (3.641%); 11 MiB/s, 177 objects/s recovering 2024-02-09T18:03:10.763 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:03:10.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:10.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:03:10.870 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:03:10.903 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:10.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:10 smithi179 ceph-mon[43975]: pgmap v2333: 47 pgs: 2 active+remapped+backfilling, 45 active+clean; 4.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 798 op/s; 7850/215583 objects misplaced (3.641%); 11 MiB/s, 177 objects/s recovering 2024-02-09T18:03:10.978 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:03:10.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:10.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:03:11.015 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-09T18:03:11.085 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:03:11.180 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:11.237 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:03:11.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:11.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:03:11.345 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:03:11.393 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:11.416 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:03:11.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:11.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:03:11.517 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T18:03:11.526 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:03:11.578 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:11.830 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T18:03:11.834 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:03:11.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:11.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:03:11.865 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T18:03:11.918 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T18:03:11.942 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:03:11.955 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T18:03:11.989 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T18:03:12.003 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:12.024 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T18:03:12.061 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T18:03:12.114 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T18:03:12.148 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:03:12.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:12.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:03:12.157 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T18:03:12.210 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T18:03:12.246 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T18:03:12.257 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:03:12.289 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T18:03:12.289 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T18:03:12.289 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T18:03:12.289 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T18:03:12.289 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T18:03:12.289 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T18:03:12.290 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T18:03:12.290 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T18:03:12.290 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T18:03:12.290 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T18:03:12.290 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T18:03:12.290 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T18:03:12.290 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T18:03:12.349 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:12.501 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset noscrub 2024-02-09T18:03:12.561 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:12 smithi099 ceph-mon[39420]: pgmap v2334: 47 pgs: 2 active+remapped+backfilling, 45 active+clean; 4.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 848 op/s; 6134/210237 objects misplaced (2.918%); 20 MiB/s, 322 objects/s recovering 2024-02-09T18:03:12.561 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:12 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2079800886' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:03:12.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:12 smithi179 ceph-mon[43975]: pgmap v2334: 47 pgs: 2 active+remapped+backfilling, 45 active+clean; 4.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 848 op/s; 6134/210237 objects misplaced (2.918%); 20 MiB/s, 322 objects/s recovering 2024-02-09T18:03:12.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:12 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2079800886' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:03:12.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:12 smithi162 ceph-mon[44304]: pgmap v2334: 47 pgs: 2 active+remapped+backfilling, 45 active+clean; 4.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 848 op/s; 6134/210237 objects misplaced (2.918%); 20 MiB/s, 322 objects/s recovering 2024-02-09T18:03:12.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:12 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2079800886' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:03:12.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:12 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mon-a[39397]: 2024-02-09T18:03:12.590+0000 7feb57590700 -1 mon.a@0(leader).osd e941 definitely_dead 0 2024-02-09T18:03:13.438 INFO:teuthology.orchestra.run.smithi099.stderr:marked out osd.10. 2024-02-09T18:03:13.599 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T18:03:13.599 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-02-09T18:03:13.599 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-02-09T18:03:13.599 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-02-09T18:03:13.599 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.599 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.16440.0:156824 7.c 7:321e6228:::benchmark_data_smithi179_135804_object49033:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e742)", 2024-02-09T18:03:13.600 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:53:19.256314+0000", 2024-02-09T18:03:13.600 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 594.33868899100003, 2024-02-09T18:03:13.600 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9669922509999997, 2024-02-09T18:03:13.600 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:03:13.600 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-09T18:03:13.600 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T18:03:13.600 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.16440", 2024-02-09T18:03:13.600 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T18:03:13.600 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 156824 2024-02-09T18:03:13.601 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.601 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:03:13.601 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.601 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:03:13.601 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:19.256314+0000", 2024-02-09T18:03:13.601 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:03:13.601 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.601 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.601 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:03:13.601 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:19.256314+0000", 2024-02-09T18:03:13.602 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:03:13.602 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.602 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.602 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:03:13.602 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:19.256315+0000", 2024-02-09T18:03:13.602 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7000000000000003e-07 2024-02-09T18:03:13.602 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.602 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.603 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:03:13.603 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:19.256319+0000", 2024-02-09T18:03:13.603 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7390000000000001e-06 2024-02-09T18:03:13.603 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.603 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.603 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:03:13.603 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:19.256327+0000", 2024-02-09T18:03:13.603 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6490000000000008e-06 2024-02-09T18:03:13.603 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.604 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.604 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:03:13.604 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:19.256336+0000", 2024-02-09T18:03:13.604 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1449999999999997e-06 2024-02-09T18:03:13.604 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.604 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.604 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:03:13.605 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:19.256389+0000", 2024-02-09T18:03:13.605 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3430999999999997e-05 2024-02-09T18:03:13.605 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.605 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.605 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for missing object", 2024-02-09T18:03:13.605 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:19.256452+0000", 2024-02-09T18:03:13.605 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3037e-05 2024-02-09T18:03:13.605 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.606 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.606 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:03:13.606 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:23.223291+0000", 2024-02-09T18:03:13.606 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9668390609999999 2024-02-09T18:03:13.606 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.606 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.607 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:03:13.607 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:23.223307+0000", 2024-02-09T18:03:13.607 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5319e-05 2024-02-09T18:03:13.607 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:03:13.607 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:03:13.607 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:03:13.607 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.607 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.608 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.16440.0:158824 7.c 7:3cb4d6c0:::benchmark_data_smithi179_135804_object51033:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e743)", 2024-02-09T18:03:13.608 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:53:22.174549+0000", 2024-02-09T18:03:13.608 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 591.42045425399999, 2024-02-09T18:03:13.608 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22753790300000001, 2024-02-09T18:03:13.608 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:03:13.608 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:03:13.608 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T18:03:13.609 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.16440", 2024-02-09T18:03:13.609 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T18:03:13.609 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 158824 2024-02-09T18:03:13.609 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.609 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:03:13.609 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.609 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:03:13.609 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:22.174549+0000", 2024-02-09T18:03:13.610 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:03:13.610 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.610 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.610 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:03:13.610 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:22.174549+0000", 2024-02-09T18:03:13.610 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:03:13.610 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.610 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.610 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:03:13.611 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:22.174551+0000", 2024-02-09T18:03:13.611 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.658e-06 2024-02-09T18:03:13.611 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.611 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.611 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:03:13.611 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:22.174556+0000", 2024-02-09T18:03:13.611 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0780000000000003e-06 2024-02-09T18:03:13.611 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.611 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.611 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:03:13.612 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:22.174567+0000", 2024-02-09T18:03:13.612 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1173e-05 2024-02-09T18:03:13.612 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.612 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.612 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:03:13.612 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:22.174575+0000", 2024-02-09T18:03:13.612 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9300000000000003e-06 2024-02-09T18:03:13.612 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.612 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.613 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:03:13.613 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:22.174658+0000", 2024-02-09T18:03:13.613 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3374999999999994e-05 2024-02-09T18:03:13.613 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.613 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.613 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:03:13.613 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:22.174749+0000", 2024-02-09T18:03:13.613 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0161000000000006e-05 2024-02-09T18:03:13.613 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.613 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.613 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,11", 2024-02-09T18:03:13.613 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:22.174893+0000", 2024-02-09T18:03:13.613 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014465600000000001 2024-02-09T18:03:13.614 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.614 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.614 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T18:03:13.614 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:22.179028+0000", 2024-02-09T18:03:13.614 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041342949999999996 2024-02-09T18:03:13.614 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.614 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.614 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:03:13.614 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:22.192467+0000", 2024-02-09T18:03:13.614 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013439536 2024-02-09T18:03:13.614 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.614 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.614 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:03:13.614 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:22.398661+0000", 2024-02-09T18:03:13.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.206193554 2024-02-09T18:03:13.615 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.615 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:03:13.615 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:22.398728+0000", 2024-02-09T18:03:13.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7787999999999995e-05 2024-02-09T18:03:13.615 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.615 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:03:13.615 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:22.402087+0000", 2024-02-09T18:03:13.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0033586990000000001 2024-02-09T18:03:13.615 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:03:13.615 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:03:13.615 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:03:13.616 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.616 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.616 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.16440.0:180042 7.c 7:3ed72c98:::benchmark_data_smithi179_135804_object72251:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e749)", 2024-02-09T18:03:13.616 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:53:54.742640+0000", 2024-02-09T18:03:13.616 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 558.85236299799999, 2024-02-09T18:03:13.616 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23305893699999999, 2024-02-09T18:03:13.616 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:03:13.616 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:03:13.616 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T18:03:13.616 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.16440", 2024-02-09T18:03:13.616 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T18:03:13.616 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 180042 2024-02-09T18:03:13.616 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.616 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:03:13.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.617 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:03:13.617 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.742640+0000", 2024-02-09T18:03:13.617 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:03:13.617 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.617 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:03:13.617 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.742640+0000", 2024-02-09T18:03:13.617 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:03:13.617 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.617 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:03:13.617 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.742641+0000", 2024-02-09T18:03:13.617 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4900000000000005e-07 2024-02-09T18:03:13.617 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.618 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:03:13.618 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.742645+0000", 2024-02-09T18:03:13.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.49e-06 2024-02-09T18:03:13.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.618 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:03:13.618 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.742653+0000", 2024-02-09T18:03:13.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2050000000000002e-06 2024-02-09T18:03:13.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.618 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:03:13.618 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.742660+0000", 2024-02-09T18:03:13.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8290000000000002e-06 2024-02-09T18:03:13.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.619 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:03:13.619 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.742721+0000", 2024-02-09T18:03:13.619 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1445999999999994e-05 2024-02-09T18:03:13.619 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.619 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.619 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:03:13.619 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.742800+0000", 2024-02-09T18:03:13.619 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9288000000000004e-05 2024-02-09T18:03:13.619 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.619 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.619 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,11", 2024-02-09T18:03:13.619 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.742910+0000", 2024-02-09T18:03:13.619 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010961199999999999 2024-02-09T18:03:13.619 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.619 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.620 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:03:13.620 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.748191+0000", 2024-02-09T18:03:13.620 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0052813269999999997 2024-02-09T18:03:13.620 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.620 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.620 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T18:03:13.620 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.749269+0000", 2024-02-09T18:03:13.620 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0010771050000000001 2024-02-09T18:03:13.620 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.620 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.620 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:03:13.620 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.973364+0000", 2024-02-09T18:03:13.620 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.224095869 2024-02-09T18:03:13.620 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.621 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.621 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:03:13.621 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.973414+0000", 2024-02-09T18:03:13.621 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.994e-05 2024-02-09T18:03:13.621 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.621 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.621 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:03:13.621 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.975699+0000", 2024-02-09T18:03:13.621 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0022850769999999999 2024-02-09T18:03:13.621 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:03:13.621 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:03:13.621 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:03:13.621 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.621 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.621 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.16440.0:180054 7.c 7:3547ab8c:::benchmark_data_smithi179_135804_object72263:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e749)", 2024-02-09T18:03:13.622 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:53:54.764783+0000", 2024-02-09T18:03:13.622 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 558.83022024399997, 2024-02-09T18:03:13.622 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22504629000000001, 2024-02-09T18:03:13.622 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:03:13.622 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:03:13.622 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T18:03:13.622 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.16440", 2024-02-09T18:03:13.622 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T18:03:13.622 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 180054 2024-02-09T18:03:13.622 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.622 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:03:13.622 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.623 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:03:13.623 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.764783+0000", 2024-02-09T18:03:13.623 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:03:13.623 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.623 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:03:13.623 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.764783+0000", 2024-02-09T18:03:13.623 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:03:13.623 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.624 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:03:13.624 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.764784+0000", 2024-02-09T18:03:13.624 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4099999999999997e-07 2024-02-09T18:03:13.624 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.624 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.624 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:03:13.624 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.764788+0000", 2024-02-09T18:03:13.624 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3710000000000002e-06 2024-02-09T18:03:13.624 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.624 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.625 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:03:13.625 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.764800+0000", 2024-02-09T18:03:13.625 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1493e-05 2024-02-09T18:03:13.625 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.625 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.625 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:03:13.625 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.764807+0000", 2024-02-09T18:03:13.625 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2690000000000002e-06 2024-02-09T18:03:13.625 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.625 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.625 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:03:13.626 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.764871+0000", 2024-02-09T18:03:13.626 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3392000000000006e-05 2024-02-09T18:03:13.626 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.626 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.626 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:03:13.626 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.764983+0000", 2024-02-09T18:03:13.626 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011234 2024-02-09T18:03:13.626 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.626 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.626 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,11", 2024-02-09T18:03:13.626 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.765154+0000", 2024-02-09T18:03:13.627 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000170951 2024-02-09T18:03:13.627 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.627 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.627 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:03:13.627 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.770510+0000", 2024-02-09T18:03:13.627 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0053557479999999996 2024-02-09T18:03:13.627 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.627 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.627 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T18:03:13.627 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.770818+0000", 2024-02-09T18:03:13.627 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00030799700000000001 2024-02-09T18:03:13.627 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.628 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.628 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:03:13.628 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.987184+0000", 2024-02-09T18:03:13.628 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.216366849 2024-02-09T18:03:13.628 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.628 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.628 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:03:13.628 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.987255+0000", 2024-02-09T18:03:13.628 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0888000000000003e-05 2024-02-09T18:03:13.628 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.628 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.628 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:03:13.628 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.989830+0000", 2024-02-09T18:03:13.628 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0025741510000000002 2024-02-09T18:03:13.629 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:03:13.629 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:03:13.629 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:03:13.629 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.629 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.629 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.16440.0:180063 7.c 7:34f65bea:::benchmark_data_smithi179_135804_object72272:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e749)", 2024-02-09T18:03:13.629 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:53:54.783269+0000", 2024-02-09T18:03:13.629 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 558.81173475799994, 2024-02-09T18:03:13.629 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.221613017, 2024-02-09T18:03:13.629 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:03:13.629 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:03:13.630 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T18:03:13.630 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.16440", 2024-02-09T18:03:13.630 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T18:03:13.630 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 180063 2024-02-09T18:03:13.630 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.630 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:03:13.630 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.630 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:03:13.630 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.783269+0000", 2024-02-09T18:03:13.630 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:03:13.630 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.630 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.631 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:03:13.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.783269+0000", 2024-02-09T18:03:13.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:03:13.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.631 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:03:13.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.783270+0000", 2024-02-09T18:03:13.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.562e-06 2024-02-09T18:03:13.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.631 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:03:13.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.783277+0000", 2024-02-09T18:03:13.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8920000000000003e-06 2024-02-09T18:03:13.632 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.632 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.632 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:03:13.632 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.783292+0000", 2024-02-09T18:03:13.632 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4965e-05 2024-02-09T18:03:13.632 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.632 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.632 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:03:13.633 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.783300+0000", 2024-02-09T18:03:13.633 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4340000000000003e-06 2024-02-09T18:03:13.633 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.633 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.633 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:03:13.633 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.783364+0000", 2024-02-09T18:03:13.633 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4844000000000005e-05 2024-02-09T18:03:13.633 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.634 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.634 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:03:13.634 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.783487+0000", 2024-02-09T18:03:13.634 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012261999999999999 2024-02-09T18:03:13.634 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.634 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.634 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,11", 2024-02-09T18:03:13.634 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.783646+0000", 2024-02-09T18:03:13.634 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015908099999999999 2024-02-09T18:03:13.634 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.635 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.635 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:03:13.635 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.788961+0000", 2024-02-09T18:03:13.635 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0053144259999999997 2024-02-09T18:03:13.635 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.635 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.635 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T18:03:13.635 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:54.791274+0000", 2024-02-09T18:03:13.635 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023135260000000002 2024-02-09T18:03:13.636 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.636 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.636 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:03:13.636 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:55.002510+0000", 2024-02-09T18:03:13.636 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21123641500000001 2024-02-09T18:03:13.636 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.636 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.636 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:03:13.636 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:55.002548+0000", 2024-02-09T18:03:13.636 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7036000000000003e-05 2024-02-09T18:03:13.637 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.637 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.637 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:03:13.637 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:55.004882+0000", 2024-02-09T18:03:13.637 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023342160000000001 2024-02-09T18:03:13.637 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:03:13.637 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:03:13.637 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:03:13.637 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.637 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.638 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.16440.0:180215 7.c 7:3d493c52:::benchmark_data_smithi179_135804_object72424:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e749)", 2024-02-09T18:03:13.638 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:53:55.124898+0000", 2024-02-09T18:03:13.638 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 558.47010583400004, 2024-02-09T18:03:13.638 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0195489210000002, 2024-02-09T18:03:13.638 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:03:13.638 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-09T18:03:13.638 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T18:03:13.639 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.16440", 2024-02-09T18:03:13.639 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T18:03:13.639 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 180215 2024-02-09T18:03:13.639 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.639 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:03:13.639 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.639 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:03:13.639 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:55.124898+0000", 2024-02-09T18:03:13.639 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:03:13.639 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.640 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.640 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:03:13.640 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:55.124898+0000", 2024-02-09T18:03:13.640 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:03:13.640 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.640 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.640 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:03:13.640 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:55.124898+0000", 2024-02-09T18:03:13.641 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9900000000000002e-07 2024-02-09T18:03:13.641 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.641 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.641 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:03:13.641 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:55.124900+0000", 2024-02-09T18:03:13.641 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8610000000000001e-06 2024-02-09T18:03:13.641 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.641 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.641 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:03:13.641 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:55.124905+0000", 2024-02-09T18:03:13.642 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6859999999999997e-06 2024-02-09T18:03:13.642 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.642 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.642 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:03:13.642 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:55.124908+0000", 2024-02-09T18:03:13.642 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1729999999999999e-06 2024-02-09T18:03:13.642 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.642 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.642 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:03:13.643 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:55.131925+0000", 2024-02-09T18:03:13.643 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0070172129999999996 2024-02-09T18:03:13.643 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.643 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.643 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for missing object", 2024-02-09T18:03:13.643 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:55.131983+0000", 2024-02-09T18:03:13.643 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8168000000000003e-05 2024-02-09T18:03:13.643 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.644 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.644 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:03:13.644 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:57.144431+0000", 2024-02-09T18:03:13.644 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0124483209999999 2024-02-09T18:03:13.644 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.644 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.644 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:03:13.644 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:57.144447+0000", 2024-02-09T18:03:13.645 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.52e-05 2024-02-09T18:03:13.645 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:03:13.645 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:03:13.645 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:03:13.645 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.645 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.645 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.16440.0:180223 7.c 7:3058aa28:::benchmark_data_smithi179_135804_object72432:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e749)", 2024-02-09T18:03:13.645 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:53:55.124924+0000", 2024-02-09T18:03:13.645 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 558.47007931799999, 2024-02-09T18:03:13.645 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0243867280000001, 2024-02-09T18:03:13.646 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:03:13.646 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-09T18:03:13.646 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T18:03:13.646 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.16440", 2024-02-09T18:03:13.646 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T18:03:13.646 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 180223 2024-02-09T18:03:13.646 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.646 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:03:13.646 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.647 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:03:13.647 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:55.124924+0000", 2024-02-09T18:03:13.647 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:03:13.647 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.647 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.647 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:03:13.647 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:55.124924+0000", 2024-02-09T18:03:13.647 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:03:13.648 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.648 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.648 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:03:13.648 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:55.124924+0000", 2024-02-09T18:03:13.648 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2399999999999999e-07 2024-02-09T18:03:13.648 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.648 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.648 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:03:13.648 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:55.124926+0000", 2024-02-09T18:03:13.648 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.232e-06 2024-02-09T18:03:13.648 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.649 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.649 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:03:13.649 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:55.124930+0000", 2024-02-09T18:03:13.649 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.617e-06 2024-02-09T18:03:13.649 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.649 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.649 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:03:13.649 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:55.124933+0000", 2024-02-09T18:03:13.649 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9189999999999999e-06 2024-02-09T18:03:13.649 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.650 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.650 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:03:13.650 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:55.132004+0000", 2024-02-09T18:03:13.650 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0070709099999999997 2024-02-09T18:03:13.650 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.650 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.650 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for missing object", 2024-02-09T18:03:13.650 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:55.132062+0000", 2024-02-09T18:03:13.651 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7766000000000002e-05 2024-02-09T18:03:13.651 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.651 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.651 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:03:13.651 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:57.149294+0000", 2024-02-09T18:03:13.651 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0172325710000001 2024-02-09T18:03:13.651 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.651 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.651 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:03:13.651 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:53:57.149311+0000", 2024-02-09T18:03:13.652 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6489000000000001e-05 2024-02-09T18:03:13.652 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:03:13.652 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:03:13.652 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:03:13.652 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.652 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.652 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.16440.0:188793 7.c 7:3c4b8ada:::benchmark_data_smithi179_135804_object81002:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e752)", 2024-02-09T18:03:13.652 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:54:08.154901+0000", 2024-02-09T18:03:13.652 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 545.44010205799998, 2024-02-09T18:03:13.653 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.31438855300000002, 2024-02-09T18:03:13.653 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:03:13.653 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-09T18:03:13.653 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T18:03:13.653 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.16440", 2024-02-09T18:03:13.653 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T18:03:13.653 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 188793 2024-02-09T18:03:13.653 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.653 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:03:13.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.653 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:03:13.654 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:08.154901+0000", 2024-02-09T18:03:13.654 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:03:13.654 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.654 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.654 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:03:13.654 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:08.154901+0000", 2024-02-09T18:03:13.654 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:03:13.654 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.655 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.655 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:03:13.655 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:08.154902+0000", 2024-02-09T18:03:13.655 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7199999999999999e-07 2024-02-09T18:03:13.655 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.655 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.655 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:03:13.655 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:08.154905+0000", 2024-02-09T18:03:13.655 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2590000000000001e-06 2024-02-09T18:03:13.655 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.655 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.655 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:03:13.656 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:08.154911+0000", 2024-02-09T18:03:13.656 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5090000000000001e-06 2024-02-09T18:03:13.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.656 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:03:13.656 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:08.154917+0000", 2024-02-09T18:03:13.656 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6749999999999996e-06 2024-02-09T18:03:13.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.657 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.657 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:03:13.657 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:08.155056+0000", 2024-02-09T18:03:13.657 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013835200000000001 2024-02-09T18:03:13.657 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.657 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.657 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for missing object", 2024-02-09T18:03:13.657 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:08.155116+0000", 2024-02-09T18:03:13.658 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0544000000000002e-05 2024-02-09T18:03:13.658 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.658 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.658 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:03:13.658 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:08.469274+0000", 2024-02-09T18:03:13.658 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.31415796899999998 2024-02-09T18:03:13.658 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.658 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.658 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:03:13.659 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:08.469290+0000", 2024-02-09T18:03:13.659 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5773e-05 2024-02-09T18:03:13.659 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:03:13.659 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:03:13.659 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:03:13.659 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.659 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.659 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.16440.0:188801 7.c 7:33962c26:::benchmark_data_smithi179_135804_object81010:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e752)", 2024-02-09T18:03:13.660 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:54:08.154928+0000", 2024-02-09T18:03:13.660 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 545.44007547199999, 2024-02-09T18:03:13.660 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.31720087699999999, 2024-02-09T18:03:13.660 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:03:13.660 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-09T18:03:13.660 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T18:03:13.660 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.16440", 2024-02-09T18:03:13.660 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T18:03:13.660 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 188801 2024-02-09T18:03:13.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.661 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:03:13.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.661 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:03:13.661 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:08.154928+0000", 2024-02-09T18:03:13.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:03:13.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.662 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:03:13.662 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:08.154928+0000", 2024-02-09T18:03:13.662 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:03:13.662 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.662 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.662 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:03:13.662 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:08.154928+0000", 2024-02-09T18:03:13.662 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.89e-07 2024-02-09T18:03:13.662 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.663 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.663 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:03:13.663 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:08.154931+0000", 2024-02-09T18:03:13.663 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5600000000000001e-06 2024-02-09T18:03:13.663 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.663 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.663 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:03:13.663 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:08.154938+0000", 2024-02-09T18:03:13.663 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5549999999999999e-06 2024-02-09T18:03:13.664 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.664 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.664 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:03:13.664 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:08.154943+0000", 2024-02-09T18:03:13.664 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7599999999999999e-06 2024-02-09T18:03:13.664 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.664 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.664 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:03:13.664 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:08.155149+0000", 2024-02-09T18:03:13.665 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020600300000000001 2024-02-09T18:03:13.665 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.665 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.665 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for missing object", 2024-02-09T18:03:13.665 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:08.155201+0000", 2024-02-09T18:03:13.665 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1898999999999999e-05 2024-02-09T18:03:13.665 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.665 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.665 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:03:13.666 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:08.472113+0000", 2024-02-09T18:03:13.666 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.31691208900000001 2024-02-09T18:03:13.666 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.666 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.666 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:03:13.666 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:08.472129+0000", 2024-02-09T18:03:13.666 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5622000000000001e-05 2024-02-09T18:03:13.666 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:03:13.666 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:03:13.666 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:03:13.667 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.667 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.667 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.16440.0:191308 7.c 7:3bb83098:::benchmark_data_smithi179_135804_object83517:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e754)", 2024-02-09T18:03:13.667 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:54:11.757068+0000", 2024-02-09T18:03:13.667 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 541.83793545399999, 2024-02-09T18:03:13.667 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38417522799999998, 2024-02-09T18:03:13.667 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:03:13.667 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-02-09T18:03:13.667 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T18:03:13.668 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.16440", 2024-02-09T18:03:13.668 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/2813719955", 2024-02-09T18:03:13.668 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 191308 2024-02-09T18:03:13.668 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.668 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:03:13.668 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.668 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:03:13.669 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:11.757068+0000", 2024-02-09T18:03:13.669 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:03:13.669 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.669 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.669 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:03:13.669 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:11.757068+0000", 2024-02-09T18:03:13.669 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:03:13.669 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.669 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.670 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:03:13.670 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:11.757068+0000", 2024-02-09T18:03:13.670 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2100000000000002e-07 2024-02-09T18:03:13.670 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.670 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.670 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:03:13.670 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:11.757071+0000", 2024-02-09T18:03:13.670 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.238e-06 2024-02-09T18:03:13.670 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.671 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.671 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:03:13.671 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:11.757077+0000", 2024-02-09T18:03:13.671 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3520000000000003e-06 2024-02-09T18:03:13.671 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.671 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.671 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:03:13.671 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:11.757082+0000", 2024-02-09T18:03:13.671 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8280000000000002e-06 2024-02-09T18:03:13.672 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.672 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.672 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:03:13.672 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:11.758782+0000", 2024-02-09T18:03:13.672 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001699629 2024-02-09T18:03:13.672 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.672 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.672 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for missing object", 2024-02-09T18:03:13.672 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:11.758839+0000", 2024-02-09T18:03:13.673 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7797000000000002e-05 2024-02-09T18:03:13.673 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.673 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.673 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:03:13.673 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.141219+0000", 2024-02-09T18:03:13.673 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38237959700000002 2024-02-09T18:03:13.673 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.673 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.673 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:03:13.673 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:54:12.141243+0000", 2024-02-09T18:03:13.674 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4366e-05 2024-02-09T18:03:13.674 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:03:13.674 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:03:13.674 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:03:13.674 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.674 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.674 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "rep_scrubmap(1.f e866 from shard 3)", 2024-02-09T18:03:13.674 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:59:30.394793+0000", 2024-02-09T18:03:13.674 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 223.20021092100001, 2024-02-09T18:03:13.675 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.089808922999999999, 2024-02-09T18:03:13.675 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:03:13.675 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-09T18:03:13.675 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:03:13.675 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.675 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:03:13.675 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:30.394793+0000", 2024-02-09T18:03:13.675 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:03:13.675 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.675 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.676 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:03:13.676 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:30.394793+0000", 2024-02-09T18:03:13.676 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:03:13.676 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.676 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.676 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:03:13.676 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:30.394794+0000", 2024-02-09T18:03:13.676 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.569999999999999e-07 2024-02-09T18:03:13.677 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.677 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.677 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:03:13.677 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:30.394798+0000", 2024-02-09T18:03:13.677 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.476e-06 2024-02-09T18:03:13.677 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.677 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.677 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:03:13.677 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:30.394808+0000", 2024-02-09T18:03:13.677 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0057e-05 2024-02-09T18:03:13.678 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.678 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.678 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:03:13.678 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:30.394815+0000", 2024-02-09T18:03:13.678 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9639999999999997e-06 2024-02-09T18:03:13.678 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.678 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.678 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:03:13.678 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:30.484570+0000", 2024-02-09T18:03:13.679 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.089755366000000003 2024-02-09T18:03:13.679 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.679 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.679 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:03:13.679 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:30.484584+0000", 2024-02-09T18:03:13.679 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3692999999999999e-05 2024-02-09T18:03:13.679 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.679 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.679 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:03:13.680 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:30.484601+0000", 2024-02-09T18:03:13.680 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7410000000000001e-05 2024-02-09T18:03:13.680 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:03:13.680 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:03:13.680 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:03:13.680 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.680 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.681 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:159576 9.1 e926/882 9:8d6a5b32:::benchmark_data_smithi179_161915_object24694:head v 926'9896, mlcod=926'9893)", 2024-02-09T18:03:13.681 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:02:20.188714+0000", 2024-02-09T18:03:13.681 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 53.406289344999998, 2024-02-09T18:03:13.681 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019579346000000001, 2024-02-09T18:03:13.681 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:03:13.681 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:03:13.681 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:03:13.681 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.682 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:03:13.682 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.188714+0000", 2024-02-09T18:03:13.682 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:03:13.682 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.682 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.682 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:03:13.682 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.188714+0000", 2024-02-09T18:03:13.682 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:03:13.683 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.683 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.683 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:03:13.683 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.188715+0000", 2024-02-09T18:03:13.683 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.54e-07 2024-02-09T18:03:13.683 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.683 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.683 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:03:13.683 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.188721+0000", 2024-02-09T18:03:13.683 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0519999999999998e-06 2024-02-09T18:03:13.684 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.684 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.684 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:03:13.684 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.188733+0000", 2024-02-09T18:03:13.684 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2099999999999999e-05 2024-02-09T18:03:13.684 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.684 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.685 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:03:13.685 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.188740+0000", 2024-02-09T18:03:13.685 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7719999999999997e-06 2024-02-09T18:03:13.685 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.685 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.685 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:03:13.685 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.190962+0000", 2024-02-09T18:03:13.685 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0022218310000000001 2024-02-09T18:03:13.685 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.685 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.686 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:03:13.686 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.190985+0000", 2024-02-09T18:03:13.686 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3637000000000001e-05 2024-02-09T18:03:13.686 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.686 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.686 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:03:13.686 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.208259+0000", 2024-02-09T18:03:13.686 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.017273279999999998 2024-02-09T18:03:13.686 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.686 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.687 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:03:13.687 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.208293+0000", 2024-02-09T18:03:13.687 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4919999999999998e-05 2024-02-09T18:03:13.687 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:03:13.687 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:03:13.687 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:03:13.687 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.687 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.687 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:159929 9.1 e926/882 9:889430ee:::benchmark_data_smithi179_161915_object25047:head v 926'9923, mlcod=926'9921)", 2024-02-09T18:03:13.687 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:02:20.586831+0000", 2024-02-09T18:03:13.688 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 53.008172242000001, 2024-02-09T18:03:13.688 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.021476810999999998, 2024-02-09T18:03:13.688 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:03:13.688 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:03:13.688 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:03:13.688 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.688 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:03:13.688 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.586831+0000", 2024-02-09T18:03:13.689 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:03:13.689 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.689 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.689 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:03:13.689 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.586831+0000", 2024-02-09T18:03:13.689 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:03:13.689 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.689 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.689 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:03:13.689 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.586833+0000", 2024-02-09T18:03:13.690 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5859999999999999e-06 2024-02-09T18:03:13.690 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.690 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.690 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:03:13.690 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.586843+0000", 2024-02-09T18:03:13.690 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7750000000000004e-06 2024-02-09T18:03:13.690 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.690 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.690 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:03:13.691 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.586861+0000", 2024-02-09T18:03:13.691 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8411e-05 2024-02-09T18:03:13.691 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.691 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.691 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:03:13.691 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.586872+0000", 2024-02-09T18:03:13.691 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.062e-05 2024-02-09T18:03:13.691 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.691 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.691 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:03:13.692 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.586936+0000", 2024-02-09T18:03:13.692 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4778999999999998e-05 2024-02-09T18:03:13.692 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.692 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.692 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:03:13.692 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.586953+0000", 2024-02-09T18:03:13.692 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.662e-05 2024-02-09T18:03:13.692 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.692 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.692 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:03:13.693 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.608262+0000", 2024-02-09T18:03:13.693 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.02130895 2024-02-09T18:03:13.693 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.693 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.693 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:03:13.693 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.608308+0000", 2024-02-09T18:03:13.693 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6069999999999998e-05 2024-02-09T18:03:13.693 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:03:13.693 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:03:13.694 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:03:13.694 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.694 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.694 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:159930 9.1 e926/882 9:8c309051:::benchmark_data_smithi179_161915_object25048:head v 926'9924, mlcod=926'9921)", 2024-02-09T18:03:13.694 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:02:20.586894+0000", 2024-02-09T18:03:13.694 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 53.008109781000002, 2024-02-09T18:03:13.694 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.021444139000000001, 2024-02-09T18:03:13.694 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:03:13.694 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:03:13.694 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:03:13.695 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.695 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:03:13.695 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.586894+0000", 2024-02-09T18:03:13.695 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:03:13.695 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.695 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.695 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:03:13.695 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.586894+0000", 2024-02-09T18:03:13.695 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:03:13.695 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.696 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.696 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:03:13.696 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.586894+0000", 2024-02-09T18:03:13.696 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7000000000000001e-07 2024-02-09T18:03:13.696 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.696 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.696 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:03:13.696 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.586898+0000", 2024-02-09T18:03:13.696 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1489999999999996e-06 2024-02-09T18:03:13.696 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.697 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.697 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:03:13.697 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.586906+0000", 2024-02-09T18:03:13.697 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5050000000000001e-06 2024-02-09T18:03:13.697 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.697 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.697 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:03:13.697 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.586910+0000", 2024-02-09T18:03:13.698 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4399999999999998e-06 2024-02-09T18:03:13.698 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.698 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.698 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:03:13.698 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.591042+0000", 2024-02-09T18:03:13.698 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041318070000000004 2024-02-09T18:03:13.698 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.698 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.698 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:03:13.699 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.591063+0000", 2024-02-09T18:03:13.699 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0721e-05 2024-02-09T18:03:13.699 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.699 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.699 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:03:13.699 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.608312+0000", 2024-02-09T18:03:13.699 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.017249536999999999 2024-02-09T18:03:13.699 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.699 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.699 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:03:13.700 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.608338+0000", 2024-02-09T18:03:13.700 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5709999999999999e-05 2024-02-09T18:03:13.700 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:03:13.700 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:03:13.700 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:03:13.700 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.700 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.700 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:159940 9.0 e926/882 9:05237317:::benchmark_data_smithi179_161915_object25058:head v 926'10063, mlcod=926'10061)", 2024-02-09T18:03:13.701 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:02:20.594718+0000", 2024-02-09T18:03:13.701 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 53.000285806000001, 2024-02-09T18:03:13.701 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019689294999999999, 2024-02-09T18:03:13.701 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:03:13.701 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:03:13.701 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:03:13.701 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.701 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:03:13.702 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.594718+0000", 2024-02-09T18:03:13.702 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:03:13.702 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.702 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.702 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:03:13.702 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.594718+0000", 2024-02-09T18:03:13.702 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:03:13.702 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.702 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.702 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:03:13.702 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.594719+0000", 2024-02-09T18:03:13.703 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.47e-07 2024-02-09T18:03:13.703 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.703 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.703 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:03:13.703 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.594729+0000", 2024-02-09T18:03:13.703 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0821e-05 2024-02-09T18:03:13.703 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.703 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.704 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:03:13.704 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.594748+0000", 2024-02-09T18:03:13.704 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9003000000000001e-05 2024-02-09T18:03:13.704 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.704 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.704 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:03:13.704 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.594758+0000", 2024-02-09T18:03:13.704 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0710000000000001e-06 2024-02-09T18:03:13.705 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.705 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.705 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:03:13.705 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.594833+0000", 2024-02-09T18:03:13.705 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5457000000000006e-05 2024-02-09T18:03:13.705 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.705 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.706 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:03:13.706 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.594853+0000", 2024-02-09T18:03:13.706 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9650999999999999e-05 2024-02-09T18:03:13.706 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.706 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.706 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:03:13.706 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.614354+0000", 2024-02-09T18:03:13.706 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019501322000000001 2024-02-09T18:03:13.706 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.706 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.707 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:03:13.707 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.614407+0000", 2024-02-09T18:03:13.707 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3022999999999997e-05 2024-02-09T18:03:13.707 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:03:13.707 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:03:13.707 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:03:13.707 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.708 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.708 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "rep_scrubmap(1.f e866 from shard 3)", 2024-02-09T18:03:13.708 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:02:46.279015+0000", 2024-02-09T18:03:13.708 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 27.315988351000001, 2024-02-09T18:03:13.708 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.079179055999999998, 2024-02-09T18:03:13.708 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:03:13.708 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-09T18:03:13.708 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:03:13.708 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.708 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:03:13.709 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:46.279015+0000", 2024-02-09T18:03:13.709 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:03:13.709 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.709 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.709 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:03:13.709 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:46.279015+0000", 2024-02-09T18:03:13.709 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:03:13.709 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.709 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.710 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:03:13.710 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:46.279016+0000", 2024-02-09T18:03:13.710 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1800000000000005e-07 2024-02-09T18:03:13.710 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.710 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.710 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:03:13.710 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:46.279021+0000", 2024-02-09T18:03:13.710 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4190000000000001e-06 2024-02-09T18:03:13.710 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.710 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.710 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:03:13.711 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:46.279032+0000", 2024-02-09T18:03:13.711 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0434999999999999e-05 2024-02-09T18:03:13.711 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.711 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.711 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:03:13.711 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:46.279039+0000", 2024-02-09T18:03:13.711 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4649999999999997e-06 2024-02-09T18:03:13.711 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.711 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.711 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:03:13.712 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:46.358165+0000", 2024-02-09T18:03:13.712 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.079126009999999997 2024-02-09T18:03:13.712 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.712 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.712 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:03:13.712 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:46.358178+0000", 2024-02-09T18:03:13.712 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.313e-05 2024-02-09T18:03:13.712 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.713 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.713 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:03:13.713 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:46.358194+0000", 2024-02-09T18:03:13.713 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5778999999999999e-05 2024-02-09T18:03:13.713 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:03:13.713 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:03:13.713 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:03:13.713 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.713 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.713 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:182392 9.0 e932/882 9:0ff9e6b3:::benchmark_data_smithi179_161915_object47510:head v 932'11381, mlcod=932'11378)", 2024-02-09T18:03:13.714 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:02:46.582874+0000", 2024-02-09T18:03:13.714 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 27.012129142999999, 2024-02-09T18:03:13.714 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020231804999999999, 2024-02-09T18:03:13.714 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:03:13.714 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:03:13.714 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:03:13.714 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.714 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:03:13.714 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:46.582874+0000", 2024-02-09T18:03:13.715 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:03:13.715 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.715 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.715 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:03:13.715 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:46.582874+0000", 2024-02-09T18:03:13.715 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:03:13.715 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.715 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.715 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:03:13.715 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:46.582875+0000", 2024-02-09T18:03:13.716 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3200000000000003e-07 2024-02-09T18:03:13.716 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.716 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.716 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:03:13.716 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:46.582883+0000", 2024-02-09T18:03:13.716 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9500000000000001e-06 2024-02-09T18:03:13.716 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.716 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.716 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:03:13.717 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:46.582898+0000", 2024-02-09T18:03:13.717 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5077e-05 2024-02-09T18:03:13.717 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.717 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.717 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:03:13.717 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:46.582907+0000", 2024-02-09T18:03:13.717 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2309999999999996e-06 2024-02-09T18:03:13.717 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.717 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.717 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:03:13.718 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:46.589287+0000", 2024-02-09T18:03:13.718 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0063808449999999996 2024-02-09T18:03:13.718 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.718 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.718 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:03:13.718 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:46.589312+0000", 2024-02-09T18:03:13.718 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4967999999999999e-05 2024-02-09T18:03:13.718 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.718 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.718 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:03:13.718 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:46.603074+0000", 2024-02-09T18:03:13.719 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013762046 2024-02-09T18:03:13.719 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.719 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.719 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:03:13.719 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:46.603106+0000", 2024-02-09T18:03:13.719 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1755999999999999e-05 2024-02-09T18:03:13.719 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:03:13.719 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:03:13.719 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:03:13.720 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.720 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.720 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:186791 9.0 e932/882 9:06eeaffb:::benchmark_data_smithi179_161915_object51909:head v 932'11646, mlcod=932'11643)", 2024-02-09T18:03:13.720 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:02:51.779991+0000", 2024-02-09T18:03:13.720 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 21.815012271000001, 2024-02-09T18:03:13.720 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019321132000000001, 2024-02-09T18:03:13.720 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:03:13.720 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:03:13.720 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:03:13.720 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.721 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:03:13.721 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:51.779991+0000", 2024-02-09T18:03:13.721 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:03:13.721 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.721 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.721 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:03:13.721 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:51.779991+0000", 2024-02-09T18:03:13.721 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:03:13.721 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:03:13.722 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:51.779992+0000", 2024-02-09T18:03:13.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1500000000000003e-07 2024-02-09T18:03:13.722 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:03:13.722 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:51.779999+0000", 2024-02-09T18:03:13.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7460000000000003e-06 2024-02-09T18:03:13.723 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.723 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.723 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:03:13.723 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:51.780013+0000", 2024-02-09T18:03:13.723 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.376e-05 2024-02-09T18:03:13.723 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.723 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.723 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:03:13.724 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:51.780021+0000", 2024-02-09T18:03:13.724 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1170000000000004e-06 2024-02-09T18:03:13.724 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.724 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.724 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:03:13.724 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:51.780073+0000", 2024-02-09T18:03:13.724 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1959999999999997e-05 2024-02-09T18:03:13.724 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.724 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.724 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:03:13.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:51.780098+0000", 2024-02-09T18:03:13.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5174999999999999e-05 2024-02-09T18:03:13.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:03:13.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:51.799252+0000", 2024-02-09T18:03:13.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019154417999999999 2024-02-09T18:03:13.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.726 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.726 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:03:13.726 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:51.799312+0000", 2024-02-09T18:03:13.726 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0041000000000003e-05 2024-02-09T18:03:13.726 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:03:13.726 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:03:13.726 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:03:13.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.726 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.726 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:186972 9.1 e932/882 9:843e711c:::benchmark_data_smithi179_161915_object52090:head v 932'11596, mlcod=932'11592)", 2024-02-09T18:03:13.726 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:02:51.998851+0000", 2024-02-09T18:03:13.727 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 21.596152373999999, 2024-02-09T18:03:13.727 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019476911999999999, 2024-02-09T18:03:13.727 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:03:13.727 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:03:13.727 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:03:13.727 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.727 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:03:13.727 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:51.998851+0000", 2024-02-09T18:03:13.727 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:03:13.727 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.728 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.728 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:03:13.728 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:51.998851+0000", 2024-02-09T18:03:13.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:03:13.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.728 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.728 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:03:13.728 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:51.998852+0000", 2024-02-09T18:03:13.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2900000000000003e-07 2024-02-09T18:03:13.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:03:13.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:51.998858+0000", 2024-02-09T18:03:13.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9000000000000003e-06 2024-02-09T18:03:13.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:03:13.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:51.998868+0000", 2024-02-09T18:03:13.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0030000000000001e-05 2024-02-09T18:03:13.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.730 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:03:13.730 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:51.998874+0000", 2024-02-09T18:03:13.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5690000000000001e-06 2024-02-09T18:03:13.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.730 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:03:13.731 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:52.001980+0000", 2024-02-09T18:03:13.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0031054889999999999 2024-02-09T18:03:13.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:03:13.731 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:52.002001+0000", 2024-02-09T18:03:13.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0781999999999999e-05 2024-02-09T18:03:13.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.732 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:03:13.732 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:52.018260+0000", 2024-02-09T18:03:13.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.016259191999999999 2024-02-09T18:03:13.732 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.732 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:03:13.732 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:52.018328+0000", 2024-02-09T18:03:13.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8220999999999996e-05 2024-02-09T18:03:13.733 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:03:13.733 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:03:13.733 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:03:13.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.733 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:190493 9.0 e934/882 9:00bc0427:::benchmark_data_smithi179_161915_object55611:head v 934'11895, mlcod=934'11891)", 2024-02-09T18:03:13.733 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:02:56.324259+0000", 2024-02-09T18:03:13.733 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 17.270744729, 2024-02-09T18:03:13.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019601338999999999, 2024-02-09T18:03:13.734 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:03:13.734 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:03:13.734 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:03:13.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:03:13.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:56.324259+0000", 2024-02-09T18:03:13.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:03:13.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.735 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:03:13.735 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:56.324259+0000", 2024-02-09T18:03:13.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:03:13.735 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.735 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.735 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:03:13.735 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:56.324260+0000", 2024-02-09T18:03:13.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.034e-06 2024-02-09T18:03:13.735 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.736 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.736 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:03:13.736 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:56.324267+0000", 2024-02-09T18:03:13.736 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1300000000000003e-06 2024-02-09T18:03:13.736 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.736 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.736 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:03:13.736 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:56.324278+0000", 2024-02-09T18:03:13.737 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0664e-05 2024-02-09T18:03:13.737 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.737 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.737 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:03:13.737 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:56.324285+0000", 2024-02-09T18:03:13.737 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4449999999999999e-06 2024-02-09T18:03:13.737 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.737 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.737 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:03:13.737 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:56.327267+0000", 2024-02-09T18:03:13.738 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0029821700000000001 2024-02-09T18:03:13.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:03:13.738 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:56.327287+0000", 2024-02-09T18:03:13.738 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.963e-05 2024-02-09T18:03:13.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:03:13.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:56.343811+0000", 2024-02-09T18:03:13.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.016523975999999999 2024-02-09T18:03:13.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:03:13.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:03:13.739 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:03:13.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:56.343860+0000", 2024-02-09T18:03:13.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9289999999999997e-05 2024-02-09T18:03:13.739 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:03:13.739 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:03:13.740 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:03:13.740 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:03:13.740 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:03:13.740 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T18:03:13.741 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:13 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4045573189' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["10"]}]: dispatch 2024-02-09T18:03:13.741 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:13 smithi162 ceph-mon[44304]: Client client.admin marked osd.10 out, while it was still marked up 2024-02-09T18:03:13.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:13 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4045573189' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["10"]}]: dispatch 2024-02-09T18:03:13.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:13 smithi099 ceph-mon[39420]: Client client.admin marked osd.10 out, while it was still marked up 2024-02-09T18:03:13.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:13 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4045573189' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["10"]}]: dispatch 2024-02-09T18:03:13.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:13 smithi179 ceph-mon[43975]: Client client.admin marked osd.10 out, while it was still marked up 2024-02-09T18:03:14.345 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-09T18:03:14.377 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:03:14.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:14.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:03:14.486 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:03:14.532 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:14.607 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:03:14.610 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:14 smithi099 ceph-mon[39420]: pgmap v2335: 47 pgs: 2 active+remapped+backfilling, 45 active+clean; 4.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 908 op/s; 6134/206397 objects misplaced (2.972%); 20 MiB/s, 322 objects/s recovering 2024-02-09T18:03:14.610 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:14 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4045573189' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["10"]}]': finished 2024-02-09T18:03:14.610 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:14 smithi099 ceph-mon[39420]: osdmap e942: 12 total, 12 up, 11 in 2024-02-09T18:03:14.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:14.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:03:14.651 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T18:03:14.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:14 smithi162 ceph-mon[44304]: pgmap v2335: 47 pgs: 2 active+remapped+backfilling, 45 active+clean; 4.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 908 op/s; 6134/206397 objects misplaced (2.972%); 20 MiB/s, 322 objects/s recovering 2024-02-09T18:03:14.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:14 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4045573189' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["10"]}]': finished 2024-02-09T18:03:14.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:14 smithi162 ceph-mon[44304]: osdmap e942: 12 total, 12 up, 11 in 2024-02-09T18:03:14.713 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:03:14.743 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:14.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:14 smithi179 ceph-mon[43975]: pgmap v2335: 47 pgs: 2 active+remapped+backfilling, 45 active+clean; 4.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 908 op/s; 6134/206397 objects misplaced (2.972%); 20 MiB/s, 322 objects/s recovering 2024-02-09T18:03:14.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:14 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4045573189' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["10"]}]': finished 2024-02-09T18:03:14.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:14 smithi179 ceph-mon[43975]: osdmap e942: 12 total, 12 up, 11 in 2024-02-09T18:03:15.092 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:03:15.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:15.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:03:15.124 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:03:15.199 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T18:03:15.231 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:15.583 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T18:03:15.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:15.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T18:03:15.688 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:03:15.720 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:15 smithi162 ceph-mon[44304]: osdmap e943: 12 total, 12 up, 11 in 2024-02-09T18:03:15.720 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:15 smithi162 ceph-mon[44304]: osdmap e944: 12 total, 12 up, 11 in 2024-02-09T18:03:15.721 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:15.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:15 smithi099 ceph-mon[39420]: osdmap e943: 12 total, 12 up, 11 in 2024-02-09T18:03:15.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:15 smithi099 ceph-mon[39420]: osdmap e944: 12 total, 12 up, 11 in 2024-02-09T18:03:15.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:15 smithi179 ceph-mon[43975]: osdmap e943: 12 total, 12 up, 11 in 2024-02-09T18:03:15.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:15 smithi179 ceph-mon[43975]: osdmap e944: 12 total, 12 up, 11 in 2024-02-09T18:03:16.175 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:03:16.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:16.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:03:16.284 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:03:16.335 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:16.392 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T18:03:16.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:16 smithi099 ceph-mon[39420]: pgmap v2338: 47 pgs: 2 active+remapped+backfilling, 45 active+clean; 4.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 660 op/s; 6134/206397 objects misplaced (2.972%); 10 MiB/s, 164 objects/s recovering 2024-02-09T18:03:16.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:16 smithi099 ceph-mon[39420]: 9.b starting backfill to osd.1 from (0'0,0'0] MAX to 941'12751 2024-02-09T18:03:16.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:16 smithi099 ceph-mon[39420]: 9.9 starting backfill to osd.3 from (0'0,0'0] MAX to 941'12752 2024-02-09T18:03:16.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:16 smithi099 ceph-mon[39420]: 9.7 starting backfill to osd.2 from (0'0,0'0] MAX to 941'12620 2024-02-09T18:03:16.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:16 smithi099 ceph-mon[39420]: 9.6 starting backfill to osd.9 from (0'0,0'0] MAX to 941'12623 2024-02-09T18:03:16.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:16 smithi099 ceph-mon[39420]: 9.e starting backfill to osd.9 from (0'0,0'0] MAX to 941'12721 2024-02-09T18:03:16.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:16 smithi099 ceph-mon[39420]: osdmap e945: 12 total, 12 up, 11 in 2024-02-09T18:03:16.824 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:03:16.836 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:16.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:03:16.841 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T18:03:16.841 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-09T18:03:16.841 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-02-09T18:03:16.841 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T18:03:16.869 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:16 smithi162 ceph-mon[44304]: pgmap v2338: 47 pgs: 2 active+remapped+backfilling, 45 active+clean; 4.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 660 op/s; 6134/206397 objects misplaced (2.972%); 10 MiB/s, 164 objects/s recovering 2024-02-09T18:03:16.869 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:16 smithi162 ceph-mon[44304]: 9.b starting backfill to osd.1 from (0'0,0'0] MAX to 941'12751 2024-02-09T18:03:16.869 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:16 smithi162 ceph-mon[44304]: 9.9 starting backfill to osd.3 from (0'0,0'0] MAX to 941'12752 2024-02-09T18:03:16.869 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:16 smithi162 ceph-mon[44304]: 9.7 starting backfill to osd.2 from (0'0,0'0] MAX to 941'12620 2024-02-09T18:03:16.869 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:16 smithi162 ceph-mon[44304]: 9.6 starting backfill to osd.9 from (0'0,0'0] MAX to 941'12623 2024-02-09T18:03:16.870 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:16 smithi162 ceph-mon[44304]: 9.e starting backfill to osd.9 from (0'0,0'0] MAX to 941'12721 2024-02-09T18:03:16.870 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:16 smithi162 ceph-mon[44304]: osdmap e945: 12 total, 12 up, 11 in 2024-02-09T18:03:16.882 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:03:16.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:16 smithi179 ceph-mon[43975]: pgmap v2338: 47 pgs: 2 active+remapped+backfilling, 45 active+clean; 4.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 660 op/s; 6134/206397 objects misplaced (2.972%); 10 MiB/s, 164 objects/s recovering 2024-02-09T18:03:16.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:16 smithi179 ceph-mon[43975]: 9.b starting backfill to osd.1 from (0'0,0'0] MAX to 941'12751 2024-02-09T18:03:16.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:16 smithi179 ceph-mon[43975]: 9.9 starting backfill to osd.3 from (0'0,0'0] MAX to 941'12752 2024-02-09T18:03:16.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:16 smithi179 ceph-mon[43975]: 9.7 starting backfill to osd.2 from (0'0,0'0] MAX to 941'12620 2024-02-09T18:03:16.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:16 smithi179 ceph-mon[43975]: 9.6 starting backfill to osd.9 from (0'0,0'0] MAX to 941'12623 2024-02-09T18:03:16.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:16 smithi179 ceph-mon[43975]: 9.e starting backfill to osd.9 from (0'0,0'0] MAX to 941'12721 2024-02-09T18:03:16.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:16 smithi179 ceph-mon[43975]: osdmap e945: 12 total, 12 up, 11 in 2024-02-09T18:03:16.937 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:03:16.987 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:17.094 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:03:17.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:17.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:03:17.198 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T18:03:17.246 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:17.705 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-09T18:03:17.712 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T18:03:17.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:17.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T18:03:17.818 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:03:17.847 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:17.872 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:03:17.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:17.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:03:17.977 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T18:03:18.009 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:18.017 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T18:03:18.041 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T18:03:18.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:18.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T18:03:18.149 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T18:03:18.187 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:18.295 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T18:03:18.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:18.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T18:03:18.401 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:03:18.465 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:18.495 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:03:18.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:18.505 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:03:18.518 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is unset 2024-02-09T18:03:18.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:18 smithi162 ceph-mon[44304]: pgmap v2341: 47 pgs: 1 peering, 1 active+remapped+backfill_toofull, 3 active+undersized+remapped, 5 active+remapped+backfilling, 37 active+clean; 4.1 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 531 op/s; 18973/200019 objects misplaced (9.486%); 53 MiB/s, 854 objects/s recovering 2024-02-09T18:03:18.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:18 smithi162 ceph-mon[44304]: osdmap e946: 12 total, 12 up, 11 in 2024-02-09T18:03:18.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:18 smithi162 ceph-mon[44304]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T18:03:18.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:18 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1700273575' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T18:03:18.606 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:03:18.652 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:18.751 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:18 smithi099 ceph-mon[39420]: pgmap v2341: 47 pgs: 1 peering, 1 active+remapped+backfill_toofull, 3 active+undersized+remapped, 5 active+remapped+backfilling, 37 active+clean; 4.1 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 531 op/s; 18973/200019 objects misplaced (9.486%); 53 MiB/s, 854 objects/s recovering 2024-02-09T18:03:18.751 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:18 smithi099 ceph-mon[39420]: osdmap e946: 12 total, 12 up, 11 in 2024-02-09T18:03:18.751 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:18 smithi099 ceph-mon[39420]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T18:03:18.751 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:18 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1700273575' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T18:03:18.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:18 smithi179 ceph-mon[43975]: pgmap v2341: 47 pgs: 1 peering, 1 active+remapped+backfill_toofull, 3 active+undersized+remapped, 5 active+remapped+backfilling, 37 active+clean; 4.1 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 531 op/s; 18973/200019 objects misplaced (9.486%); 53 MiB/s, 854 objects/s recovering 2024-02-09T18:03:18.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:18 smithi179 ceph-mon[43975]: osdmap e946: 12 total, 12 up, 11 in 2024-02-09T18:03:18.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:18 smithi179 ceph-mon[43975]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-09T18:03:18.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:18 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1700273575' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T18:03:19.037 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:03:19.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:19.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:03:19.142 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T18:03:19.193 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:19.226 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T18:03:19.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:19.238 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T18:03:19.339 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T18:03:19.381 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:19.549 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T18:03:19.552 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:19.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T18:03:19.653 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:03:19.681 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:19 smithi162 ceph-mon[44304]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:03:19.681 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:19 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1700273575' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T18:03:19.681 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:19 smithi162 ceph-mon[44304]: osdmap e947: 12 total, 12 up, 11 in 2024-02-09T18:03:19.714 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:19 smithi179 ceph-mon[43975]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:03:19.714 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:19 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1700273575' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T18:03:19.714 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:19 smithi179 ceph-mon[43975]: osdmap e947: 12 total, 12 up, 11 in 2024-02-09T18:03:19.721 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:19.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:19 smithi099 ceph-mon[39420]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:03:19.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:19 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1700273575' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T18:03:19.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:19 smithi099 ceph-mon[39420]: osdmap e947: 12 total, 12 up, 11 in 2024-02-09T18:03:19.759 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:03:19.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:19.770 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:03:19.871 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T18:03:19.911 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:19.952 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T18:03:19.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:19.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T18:03:20.058 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:03:20.074 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 5, 6, 7, 8, 9, 11, 0, 2, 4] out_osds: [10] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 7, 9, 10, 11, 8, 3] 2024-02-09T18:03:20.075 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T18:03:20.075 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-02-09T18:03:20.075 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-09T18:03:20.094 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:20.115 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:03:20.119 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:20.119 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:03:20.220 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:03:20.256 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:20.283 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:03:20.288 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T18:03:20.288 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-09T18:03:20.288 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-02-09T18:03:20.289 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-02-09T18:03:20.289 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T18:03:20.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:20.290 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:03:20.316 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T18:03:20.376 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T18:03:20.390 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:03:20.421 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:20.424 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T18:03:20.457 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:03:20.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:20.463 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:03:20.502 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T18:03:20.543 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T18:03:20.562 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:03:20.565 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:03:20.586 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T18:03:20.638 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:20.638 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T18:03:20.666 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:20 smithi162 ceph-mon[44304]: pgmap v2344: 47 pgs: 1 peering, 1 active+remapped+backfill_toofull, 3 active+undersized+remapped, 6 active+remapped+backfilling, 36 active+clean; 4.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 511 B/s rd, 16 KiB/s wr, 666 op/s; 22787/198417 objects misplaced (11.484%); 58 MiB/s, 920 objects/s recovering 2024-02-09T18:03:20.666 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:20 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/966011510' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:03:20.683 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T18:03:20.714 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T18:03:20.757 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T18:03:20.794 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T18:03:20.813 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:03:20.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:20.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:03:20.826 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:20 smithi099 ceph-mon[39420]: pgmap v2344: 47 pgs: 1 peering, 1 active+remapped+backfill_toofull, 3 active+undersized+remapped, 6 active+remapped+backfilling, 36 active+clean; 4.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 511 B/s rd, 16 KiB/s wr, 666 op/s; 22787/198417 objects misplaced (11.484%); 58 MiB/s, 920 objects/s recovering 2024-02-09T18:03:20.826 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:20 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/966011510' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:03:20.842 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T18:03:20.880 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T18:03:20.880 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T18:03:20.880 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T18:03:20.880 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T18:03:20.880 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T18:03:20.880 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T18:03:20.880 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T18:03:20.881 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T18:03:20.881 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T18:03:20.881 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T18:03:20.881 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T18:03:20.881 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T18:03:20.921 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:03:20.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:20 smithi179 ceph-mon[43975]: pgmap v2344: 47 pgs: 1 peering, 1 active+remapped+backfill_toofull, 3 active+undersized+remapped, 6 active+remapped+backfilling, 36 active+clean; 4.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 511 B/s rd, 16 KiB/s wr, 666 op/s; 22787/198417 objects misplaced (11.484%); 58 MiB/s, 920 objects/s recovering 2024-02-09T18:03:20.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:20 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/966011510' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:03:20.983 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:21.141 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-09T18:03:21.512 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T18:03:21.654 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:21 smithi099 ceph-mon[39420]: 1.2 scrub starts 2024-02-09T18:03:21.654 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:21 smithi099 ceph-mon[39420]: 1.2 scrub ok 2024-02-09T18:03:21.654 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:21 smithi099 ceph-mon[39420]: 1.9 scrub starts 2024-02-09T18:03:21.654 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:21 smithi099 ceph-mon[39420]: 1.9 scrub ok 2024-02-09T18:03:21.755 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:03:21.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:21.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:03:21.836 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:21 smithi162 ceph-mon[44304]: 1.2 scrub starts 2024-02-09T18:03:21.836 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:21 smithi162 ceph-mon[44304]: 1.2 scrub ok 2024-02-09T18:03:21.836 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:21 smithi162 ceph-mon[44304]: 1.9 scrub starts 2024-02-09T18:03:21.836 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:21 smithi162 ceph-mon[44304]: 1.9 scrub ok 2024-02-09T18:03:21.858 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:03:21.915 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:21.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:21 smithi179 ceph-mon[43975]: 1.2 scrub starts 2024-02-09T18:03:21.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:21 smithi179 ceph-mon[43975]: 1.2 scrub ok 2024-02-09T18:03:21.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:21 smithi179 ceph-mon[43975]: 1.9 scrub starts 2024-02-09T18:03:21.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:21 smithi179 ceph-mon[43975]: 1.9 scrub ok 2024-02-09T18:03:21.963 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-09T18:03:22.181 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:03:22.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:22.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:03:22.289 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:03:22.329 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:22.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:22 smithi179 ceph-mon[43975]: pgmap v2345: 47 pgs: 1 active+remapped+backfill_toofull, 1 peering, 6 active+remapped+backfilling, 39 active+clean; 3.9 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 347 B/s rd, 11 KiB/s wr, 795 op/s; 19432/192348 objects misplaced (10.103%); 68 MiB/s, 1.08k objects/s recovering 2024-02-09T18:03:22.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:22 smithi162 ceph-mon[44304]: pgmap v2345: 47 pgs: 1 active+remapped+backfill_toofull, 1 peering, 6 active+remapped+backfilling, 39 active+clean; 3.9 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 347 B/s rd, 11 KiB/s wr, 795 op/s; 19432/192348 objects misplaced (10.103%); 68 MiB/s, 1.08k objects/s recovering 2024-02-09T18:03:22.998 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:22 smithi099 ceph-mon[39420]: pgmap v2345: 47 pgs: 1 active+remapped+backfill_toofull, 1 peering, 6 active+remapped+backfilling, 39 active+clean; 3.9 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 347 B/s rd, 11 KiB/s wr, 795 op/s; 19432/192348 objects misplaced (10.103%); 68 MiB/s, 1.08k objects/s recovering 2024-02-09T18:03:23.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:23 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:03:23.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:23 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:03:23.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:23 smithi162 ceph-mon[44304]: 1.0 scrub starts 2024-02-09T18:03:23.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:23 smithi162 ceph-mon[44304]: 1.0 scrub ok 2024-02-09T18:03:23.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:23 smithi162 ceph-mon[44304]: osdmap e948: 12 total, 12 up, 11 in 2024-02-09T18:03:23.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:23 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T18:03:23.679 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:23 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T18:03:23.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:23 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T18:03:23.680 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:23 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T18:03:23.687 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T18:03:23.733 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-02-09T18:03:23.733 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T18:03:23.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:23 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:03:23.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:23 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:03:23.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:23 smithi179 ceph-mon[43975]: 1.0 scrub starts 2024-02-09T18:03:23.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:23 smithi179 ceph-mon[43975]: 1.0 scrub ok 2024-02-09T18:03:23.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:23 smithi179 ceph-mon[43975]: osdmap e948: 12 total, 12 up, 11 in 2024-02-09T18:03:23.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:23 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T18:03:23.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:23 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T18:03:23.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:23 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T18:03:23.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:23 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T18:03:23.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:23 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:03:23.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:23 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:03:23.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:23 smithi099 ceph-mon[39420]: 1.0 scrub starts 2024-02-09T18:03:23.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:23 smithi099 ceph-mon[39420]: 1.0 scrub ok 2024-02-09T18:03:23.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:23 smithi099 ceph-mon[39420]: osdmap e948: 12 total, 12 up, 11 in 2024-02-09T18:03:23.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:23 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T18:03:23.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:23 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T18:03:23.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:23 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T18:03:23.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:23 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T18:03:24.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:24 smithi162 ceph-mon[44304]: pgmap v2347: 47 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 40 active+clean; 3.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 341 B/s rd, 11 KiB/s wr, 623 op/s; 17547/188805 objects misplaced (9.294%); 47 MiB/s, 754 objects/s recovering 2024-02-09T18:03:24.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:24 smithi162 ceph-mon[44304]: 1.d scrub starts 2024-02-09T18:03:24.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:24 smithi162 ceph-mon[44304]: 1.d scrub ok 2024-02-09T18:03:24.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:24 smithi179 ceph-mon[43975]: pgmap v2347: 47 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 40 active+clean; 3.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 341 B/s rd, 11 KiB/s wr, 623 op/s; 17547/188805 objects misplaced (9.294%); 47 MiB/s, 754 objects/s recovering 2024-02-09T18:03:24.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:24 smithi179 ceph-mon[43975]: 1.d scrub starts 2024-02-09T18:03:24.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:24 smithi179 ceph-mon[43975]: 1.d scrub ok 2024-02-09T18:03:24.941 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.9 dump_ops_in_flight 2024-02-09T18:03:24.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:24 smithi099 ceph-mon[39420]: pgmap v2347: 47 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 40 active+clean; 3.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 341 B/s rd, 11 KiB/s wr, 623 op/s; 17547/188805 objects misplaced (9.294%); 47 MiB/s, 754 objects/s recovering 2024-02-09T18:03:24.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:24 smithi099 ceph-mon[39420]: 1.d scrub starts 2024-02-09T18:03:24.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:24 smithi099 ceph-mon[39420]: 1.d scrub ok 2024-02-09T18:03:25.270 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T18:03:25.828 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:25 smithi179 ceph-mon[43975]: 1.19 scrub starts 2024-02-09T18:03:25.828 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:25 smithi179 ceph-mon[43975]: 1.19 scrub ok 2024-02-09T18:03:25.828 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:25 smithi179 ceph-mon[43975]: 1.5 scrub starts 2024-02-09T18:03:25.828 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:25 smithi179 ceph-mon[43975]: osdmap e949: 12 total, 12 up, 11 in 2024-02-09T18:03:25.828 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:25 smithi179 ceph-mon[43975]: 1.5 scrub ok 2024-02-09T18:03:25.828 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:25 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T18:03:25.828 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:25 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T18:03:25.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:25 smithi162 ceph-mon[44304]: 1.19 scrub starts 2024-02-09T18:03:25.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:25 smithi162 ceph-mon[44304]: 1.19 scrub ok 2024-02-09T18:03:25.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:25 smithi162 ceph-mon[44304]: 1.5 scrub starts 2024-02-09T18:03:25.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:25 smithi162 ceph-mon[44304]: osdmap e949: 12 total, 12 up, 11 in 2024-02-09T18:03:25.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:25 smithi162 ceph-mon[44304]: 1.5 scrub ok 2024-02-09T18:03:25.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:25 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T18:03:25.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:25 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T18:03:25.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:25 smithi099 ceph-mon[39420]: 1.19 scrub starts 2024-02-09T18:03:25.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:25 smithi099 ceph-mon[39420]: 1.19 scrub ok 2024-02-09T18:03:25.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:25 smithi099 ceph-mon[39420]: 1.5 scrub starts 2024-02-09T18:03:25.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:25 smithi099 ceph-mon[39420]: osdmap e949: 12 total, 12 up, 11 in 2024-02-09T18:03:25.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:25 smithi099 ceph-mon[39420]: 1.5 scrub ok 2024-02-09T18:03:25.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:25 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T18:03:25.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:25 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T18:03:26.010 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:03:26.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:26.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:03:26.331 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T18:03:26.368 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:26.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:26 smithi179 ceph-mon[43975]: pgmap v2349: 47 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 40 active+clean; 3.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 469 op/s; 17547/188805 objects misplaced (9.294%); 39 MiB/s, 624 objects/s recovering 2024-02-09T18:03:26.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:26 smithi179 ceph-mon[43975]: osdmap e950: 12 total, 12 up, 11 in 2024-02-09T18:03:26.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:26 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T18:03:26.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:26 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T18:03:26.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:26 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T18:03:26.827 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T18:03:26.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:26.844 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T18:03:26.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:26 smithi162 ceph-mon[44304]: pgmap v2349: 47 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 40 active+clean; 3.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 469 op/s; 17547/188805 objects misplaced (9.294%); 39 MiB/s, 624 objects/s recovering 2024-02-09T18:03:26.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:26 smithi162 ceph-mon[44304]: osdmap e950: 12 total, 12 up, 11 in 2024-02-09T18:03:26.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:26 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T18:03:26.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:26 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T18:03:26.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:26 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T18:03:26.945 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:03:26.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:26 smithi099 ceph-mon[39420]: pgmap v2349: 47 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 40 active+clean; 3.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 469 op/s; 17547/188805 objects misplaced (9.294%); 39 MiB/s, 624 objects/s recovering 2024-02-09T18:03:26.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:26 smithi099 ceph-mon[39420]: osdmap e950: 12 total, 12 up, 11 in 2024-02-09T18:03:26.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:26 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T18:03:26.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:26 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T18:03:26.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:26 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T18:03:27.020 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:27.282 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:03:27.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:27.286 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:03:27.352 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:03:27.386 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:03:27.447 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:27.470 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:03:27.594 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:27 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:03:27.594 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:27 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:03:27.594 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:27 smithi099 ceph-mon[39420]: 1.15 scrub starts 2024-02-09T18:03:27.594 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:27 smithi099 ceph-mon[39420]: 1.15 scrub ok 2024-02-09T18:03:27.594 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:27 smithi099 ceph-mon[39420]: osdmap e951: 12 total, 12 up, 11 in 2024-02-09T18:03:27.715 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T18:03:27.715 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T18:03:27.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:27 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:03:27.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:27 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:03:27.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:27 smithi179 ceph-mon[43975]: 1.15 scrub starts 2024-02-09T18:03:27.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:27 smithi179 ceph-mon[43975]: 1.15 scrub ok 2024-02-09T18:03:27.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:27 smithi179 ceph-mon[43975]: osdmap e951: 12 total, 12 up, 11 in 2024-02-09T18:03:27.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:27 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:03:27.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:27 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:03:27.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:27 smithi162 ceph-mon[44304]: 1.15 scrub starts 2024-02-09T18:03:27.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:27 smithi162 ceph-mon[44304]: 1.15 scrub ok 2024-02-09T18:03:27.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:27 smithi162 ceph-mon[44304]: osdmap e951: 12 total, 12 up, 11 in 2024-02-09T18:03:28.459 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.9 dump_blocked_ops 2024-02-09T18:03:28.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:28 smithi179 ceph-mon[43975]: pgmap v2352: 47 pgs: 1 peering, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 41 active+clean; 3.7 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 583 op/s; 13399/182760 objects misplaced (7.331%); 56 MiB/s, 901 objects/s recovering 2024-02-09T18:03:28.891 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T18:03:28.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:28 smithi162 ceph-mon[44304]: pgmap v2352: 47 pgs: 1 peering, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 41 active+clean; 3.7 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 583 op/s; 13399/182760 objects misplaced (7.331%); 56 MiB/s, 901 objects/s recovering 2024-02-09T18:03:28.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:28 smithi099 ceph-mon[39420]: pgmap v2352: 47 pgs: 1 peering, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 41 active+clean; 3.7 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 583 op/s; 13399/182760 objects misplaced (7.331%); 56 MiB/s, 901 objects/s recovering 2024-02-09T18:03:29.519 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:03:29.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:29.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:03:29.629 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:03:29.680 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:29.722 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:03:29.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:29.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:03:29.829 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T18:03:29.894 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:30.376 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T18:03:30.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:30.384 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T18:03:30.485 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:03:30.532 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:30.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:30 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:03:30.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:30 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:03:30.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:30 smithi179 ceph-mon[43975]: 1.e scrub starts 2024-02-09T18:03:30.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:30 smithi179 ceph-mon[43975]: 1.e scrub ok 2024-02-09T18:03:30.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:30 smithi179 ceph-mon[43975]: pgmap v2353: 47 pgs: 1 peering, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 41 active+clean; 3.7 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 433 op/s; 12202/181005 objects misplaced (6.741%); 48 MiB/s, 773 objects/s recovering 2024-02-09T18:03:30.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:30 smithi179 ceph-mon[43975]: osdmap e952: 12 total, 12 up, 11 in 2024-02-09T18:03:30.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:30 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:03:30.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:30 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:03:30.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:30 smithi162 ceph-mon[44304]: 1.e scrub starts 2024-02-09T18:03:30.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:30 smithi162 ceph-mon[44304]: 1.e scrub ok 2024-02-09T18:03:30.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:30 smithi162 ceph-mon[44304]: pgmap v2353: 47 pgs: 1 peering, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 41 active+clean; 3.7 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 433 op/s; 12202/181005 objects misplaced (6.741%); 48 MiB/s, 773 objects/s recovering 2024-02-09T18:03:30.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:30 smithi162 ceph-mon[44304]: osdmap e952: 12 total, 12 up, 11 in 2024-02-09T18:03:30.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:30 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:03:30.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:30 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:03:30.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:30 smithi099 ceph-mon[39420]: 1.e scrub starts 2024-02-09T18:03:30.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:30 smithi099 ceph-mon[39420]: 1.e scrub ok 2024-02-09T18:03:30.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:30 smithi099 ceph-mon[39420]: pgmap v2353: 47 pgs: 1 peering, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 41 active+clean; 3.7 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 433 op/s; 12202/181005 objects misplaced (6.741%); 48 MiB/s, 773 objects/s recovering 2024-02-09T18:03:30.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:30 smithi099 ceph-mon[39420]: osdmap e952: 12 total, 12 up, 11 in 2024-02-09T18:03:31.077 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T18:03:31.077 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T18:03:31.863 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.9 dump_historic_ops 2024-02-09T18:03:32.293 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T18:03:32.614 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:03:32.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:32.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:03:32.725 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:03:32.768 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:32.846 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:03:32.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:32.855 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:03:32.855 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:32 smithi179 ceph-mon[43975]: pgmap v2355: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 42 active+clean; 3.6 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 783 op/s; 10514/174696 objects misplaced (6.018%); 61 MiB/s, 981 objects/s recovering 2024-02-09T18:03:32.856 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:32 smithi179 ceph-mon[43975]: 1.14 scrub starts 2024-02-09T18:03:32.856 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:32 smithi179 ceph-mon[43975]: 1.14 scrub ok 2024-02-09T18:03:32.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:32 smithi162 ceph-mon[44304]: pgmap v2355: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 42 active+clean; 3.6 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 783 op/s; 10514/174696 objects misplaced (6.018%); 61 MiB/s, 981 objects/s recovering 2024-02-09T18:03:32.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:32 smithi162 ceph-mon[44304]: 1.14 scrub starts 2024-02-09T18:03:32.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:32 smithi162 ceph-mon[44304]: 1.14 scrub ok 2024-02-09T18:03:32.956 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:03:32.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:32 smithi099 ceph-mon[39420]: pgmap v2355: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 42 active+clean; 3.6 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 783 op/s; 10514/174696 objects misplaced (6.018%); 61 MiB/s, 981 objects/s recovering 2024-02-09T18:03:32.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:32 smithi099 ceph-mon[39420]: 1.14 scrub starts 2024-02-09T18:03:32.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:32 smithi099 ceph-mon[39420]: 1.14 scrub ok 2024-02-09T18:03:33.015 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:33.360 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T18:03:33.360 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T18:03:33.360 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 952, 2024-02-09T18:03:33.360 INFO:teuthology.orchestra.run.smithi099.stdout: "fsid": "a69c870a-c76e-11ee-95b7-87774f69a715", 2024-02-09T18:03:33.360 INFO:teuthology.orchestra.run.smithi099.stdout: "created": "2024-02-09T17:15:26.113606+0000", 2024-02-09T18:03:33.360 INFO:teuthology.orchestra.run.smithi099.stdout: "modified": "2024-02-09T18:03:29.596726+0000", 2024-02-09T18:03:33.360 INFO:teuthology.orchestra.run.smithi099.stdout: "last_up_change": "2024-02-09T17:59:11.889493+0000", 2024-02-09T18:03:33.360 INFO:teuthology.orchestra.run.smithi099.stdout: "last_in_change": "2024-02-09T18:03:13.429500+0000", 2024-02-09T18:03:33.360 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-09T18:03:33.360 INFO:teuthology.orchestra.run.smithi099.stdout: "flags_num": 5804032, 2024-02-09T18:03:33.360 INFO:teuthology.orchestra.run.smithi099.stdout: "flags_set": [ 2024-02-09T18:03:33.361 INFO:teuthology.orchestra.run.smithi099.stdout: "nodeep-scrub", 2024-02-09T18:03:33.361 INFO:teuthology.orchestra.run.smithi099.stdout: "pglog_hardlimit", 2024-02-09T18:03:33.361 INFO:teuthology.orchestra.run.smithi099.stdout: "purged_snapdirs", 2024-02-09T18:03:33.361 INFO:teuthology.orchestra.run.smithi099.stdout: "recovery_deletes", 2024-02-09T18:03:33.361 INFO:teuthology.orchestra.run.smithi099.stdout: "sortbitwise" 2024-02-09T18:03:33.361 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T18:03:33.361 INFO:teuthology.orchestra.run.smithi099.stdout: "crush_version": 28, 2024-02-09T18:03:33.361 INFO:teuthology.orchestra.run.smithi099.stdout: "full_ratio": 0.94999998807907104, 2024-02-09T18:03:33.361 INFO:teuthology.orchestra.run.smithi099.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-09T18:03:33.361 INFO:teuthology.orchestra.run.smithi099.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-09T18:03:33.361 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_snapshot": "", 2024-02-09T18:03:33.361 INFO:teuthology.orchestra.run.smithi099.stdout: "pool_max": 9, 2024-02-09T18:03:33.361 INFO:teuthology.orchestra.run.smithi099.stdout: "max_osd": 12, 2024-02-09T18:03:33.361 INFO:teuthology.orchestra.run.smithi099.stdout: "require_min_compat_client": "luminous", 2024-02-09T18:03:33.362 INFO:teuthology.orchestra.run.smithi099.stdout: "min_compat_client": "luminous", 2024-02-09T18:03:33.362 INFO:teuthology.orchestra.run.smithi099.stdout: "require_osd_release": "squid", 2024-02-09T18:03:33.362 INFO:teuthology.orchestra.run.smithi099.stdout: "allow_crimson": false, 2024-02-09T18:03:33.362 INFO:teuthology.orchestra.run.smithi099.stdout: "pools": [ 2024-02-09T18:03:33.362 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.362 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 1, 2024-02-09T18:03:33.362 INFO:teuthology.orchestra.run.smithi099.stdout: "pool_name": ".mgr", 2024-02-09T18:03:33.362 INFO:teuthology.orchestra.run.smithi099.stdout: "create_time": "2024-02-09T17:18:56.552286+0000", 2024-02-09T18:03:33.362 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 1, 2024-02-09T18:03:33.362 INFO:teuthology.orchestra.run.smithi099.stdout: "flags_names": "hashpspool", 2024-02-09T18:03:33.362 INFO:teuthology.orchestra.run.smithi099.stdout: "type": 1, 2024-02-09T18:03:33.362 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-02-09T18:03:33.362 INFO:teuthology.orchestra.run.smithi099.stdout: "min_size": 2, 2024-02-09T18:03:33.362 INFO:teuthology.orchestra.run.smithi099.stdout: "crush_rule": 0, 2024-02-09T18:03:33.362 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_count": 0, 2024-02-09T18:03:33.362 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_target": 0, 2024-02-09T18:03:33.363 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_barrier": 0, 2024-02-09T18:03:33.363 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-09T18:03:33.363 INFO:teuthology.orchestra.run.smithi099.stdout: "object_hash": 2, 2024-02-09T18:03:33.363 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_autoscale_mode": "off", 2024-02-09T18:03:33.363 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num": 31, 2024-02-09T18:03:33.363 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_placement_num": 31, 2024-02-09T18:03:33.363 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_placement_num_target": 31, 2024-02-09T18:03:33.363 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num_target": 31, 2024-02-09T18:03:33.363 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num_pending": 31, 2024-02-09T18:03:33.363 INFO:teuthology.orchestra.run.smithi099.stdout: "last_pg_merge_meta": { 2024-02-09T18:03:33.363 INFO:teuthology.orchestra.run.smithi099.stdout: "source_pgid": "0.0", 2024-02-09T18:03:33.363 INFO:teuthology.orchestra.run.smithi099.stdout: "ready_epoch": 0, 2024-02-09T18:03:33.363 INFO:teuthology.orchestra.run.smithi099.stdout: "last_epoch_started": 0, 2024-02-09T18:03:33.363 INFO:teuthology.orchestra.run.smithi099.stdout: "last_epoch_clean": 0, 2024-02-09T18:03:33.363 INFO:teuthology.orchestra.run.smithi099.stdout: "source_version": "0'0", 2024-02-09T18:03:33.364 INFO:teuthology.orchestra.run.smithi099.stdout: "target_version": "0'0" 2024-02-09T18:03:33.364 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.364 INFO:teuthology.orchestra.run.smithi099.stdout: "last_change": "544", 2024-02-09T18:03:33.364 INFO:teuthology.orchestra.run.smithi099.stdout: "last_force_op_resend": "0", 2024-02-09T18:03:33.364 INFO:teuthology.orchestra.run.smithi099.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-09T18:03:33.364 INFO:teuthology.orchestra.run.smithi099.stdout: "last_force_op_resend_preluminous": "514", 2024-02-09T18:03:33.364 INFO:teuthology.orchestra.run.smithi099.stdout: "auid": 0, 2024-02-09T18:03:33.364 INFO:teuthology.orchestra.run.smithi099.stdout: "snap_mode": "selfmanaged", 2024-02-09T18:03:33.364 INFO:teuthology.orchestra.run.smithi099.stdout: "snap_seq": 0, 2024-02-09T18:03:33.364 INFO:teuthology.orchestra.run.smithi099.stdout: "snap_epoch": 0, 2024-02-09T18:03:33.364 INFO:teuthology.orchestra.run.smithi099.stdout: "pool_snaps": [], 2024-02-09T18:03:33.364 INFO:teuthology.orchestra.run.smithi099.stdout: "removed_snaps": "[]", 2024-02-09T18:03:33.364 INFO:teuthology.orchestra.run.smithi099.stdout: "quota_max_bytes": 0, 2024-02-09T18:03:33.364 INFO:teuthology.orchestra.run.smithi099.stdout: "quota_max_objects": 0, 2024-02-09T18:03:33.365 INFO:teuthology.orchestra.run.smithi099.stdout: "tiers": [], 2024-02-09T18:03:33.365 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_of": -1, 2024-02-09T18:03:33.365 INFO:teuthology.orchestra.run.smithi099.stdout: "read_tier": -1, 2024-02-09T18:03:33.365 INFO:teuthology.orchestra.run.smithi099.stdout: "write_tier": -1, 2024-02-09T18:03:33.365 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_mode": "none", 2024-02-09T18:03:33.365 INFO:teuthology.orchestra.run.smithi099.stdout: "target_max_bytes": 0, 2024-02-09T18:03:33.365 INFO:teuthology.orchestra.run.smithi099.stdout: "target_max_objects": 0, 2024-02-09T18:03:33.365 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-09T18:03:33.365 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-09T18:03:33.365 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-09T18:03:33.365 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_min_flush_age": 0, 2024-02-09T18:03:33.365 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_min_evict_age": 0, 2024-02-09T18:03:33.365 INFO:teuthology.orchestra.run.smithi099.stdout: "erasure_code_profile": "", 2024-02-09T18:03:33.365 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_params": { 2024-02-09T18:03:33.365 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "none" 2024-02-09T18:03:33.366 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.366 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_period": 0, 2024-02-09T18:03:33.366 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_count": 0, 2024-02-09T18:03:33.366 INFO:teuthology.orchestra.run.smithi099.stdout: "use_gmt_hitset": true, 2024-02-09T18:03:33.366 INFO:teuthology.orchestra.run.smithi099.stdout: "min_read_recency_for_promote": 0, 2024-02-09T18:03:33.366 INFO:teuthology.orchestra.run.smithi099.stdout: "min_write_recency_for_promote": 0, 2024-02-09T18:03:33.366 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_grade_decay_rate": 0, 2024-02-09T18:03:33.366 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_search_last_n": 0, 2024-02-09T18:03:33.366 INFO:teuthology.orchestra.run.smithi099.stdout: "grade_table": [], 2024-02-09T18:03:33.366 INFO:teuthology.orchestra.run.smithi099.stdout: "stripe_width": 0, 2024-02-09T18:03:33.366 INFO:teuthology.orchestra.run.smithi099.stdout: "expected_num_objects": 0, 2024-02-09T18:03:33.366 INFO:teuthology.orchestra.run.smithi099.stdout: "fast_read": false, 2024-02-09T18:03:33.366 INFO:teuthology.orchestra.run.smithi099.stdout: "options": { 2024-02-09T18:03:33.366 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num_max": 32, 2024-02-09T18:03:33.366 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num_min": 1 2024-02-09T18:03:33.367 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.367 INFO:teuthology.orchestra.run.smithi099.stdout: "application_metadata": { 2024-02-09T18:03:33.367 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": {} 2024-02-09T18:03:33.367 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.367 INFO:teuthology.orchestra.run.smithi099.stdout: "read_balance": { 2024-02-09T18:03:33.367 INFO:teuthology.orchestra.run.smithi099.stdout: "score_type": "Fair distribution", 2024-02-09T18:03:33.367 INFO:teuthology.orchestra.run.smithi099.stdout: "score_acting": 7.119999885559082, 2024-02-09T18:03:33.367 INFO:teuthology.orchestra.run.smithi099.stdout: "score_stable": 2.130000114440918, 2024-02-09T18:03:33.367 INFO:teuthology.orchestra.run.smithi099.stdout: "optimal_score": 1, 2024-02-09T18:03:33.367 INFO:teuthology.orchestra.run.smithi099.stdout: "raw_score_acting": 7.119999885559082, 2024-02-09T18:03:33.367 INFO:teuthology.orchestra.run.smithi099.stdout: "raw_score_stable": 2.130000114440918, 2024-02-09T18:03:33.367 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity_weighted": 0.78079599142074585, 2024-02-09T18:03:33.367 INFO:teuthology.orchestra.run.smithi099.stdout: "average_primary_affinity": 0.85000002384185791, 2024-02-09T18:03:33.367 INFO:teuthology.orchestra.run.smithi099.stdout: "average_primary_affinity_weighted": 0.77999997138977051, 2024-02-09T18:03:33.368 INFO:teuthology.orchestra.run.smithi099.stdout: "error_message": "pool 1 has acting primaries on OSD(s) with primary affinity 0, read balance score is not accurate" 2024-02-09T18:03:33.368 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.368 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.368 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.368 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-02-09T18:03:33.368 INFO:teuthology.orchestra.run.smithi099.stdout: "pool_name": "unique_pool_7", 2024-02-09T18:03:33.368 INFO:teuthology.orchestra.run.smithi099.stdout: "create_time": "2024-02-09T18:00:06.232818+0000", 2024-02-09T18:03:33.368 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 1, 2024-02-09T18:03:33.368 INFO:teuthology.orchestra.run.smithi099.stdout: "flags_names": "hashpspool", 2024-02-09T18:03:33.368 INFO:teuthology.orchestra.run.smithi099.stdout: "type": 1, 2024-02-09T18:03:33.368 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-02-09T18:03:33.368 INFO:teuthology.orchestra.run.smithi099.stdout: "min_size": 2, 2024-02-09T18:03:33.368 INFO:teuthology.orchestra.run.smithi099.stdout: "crush_rule": 0, 2024-02-09T18:03:33.368 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_count": 0, 2024-02-09T18:03:33.368 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_target": 0, 2024-02-09T18:03:33.369 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_barrier": 0, 2024-02-09T18:03:33.369 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-09T18:03:33.369 INFO:teuthology.orchestra.run.smithi099.stdout: "object_hash": 2, 2024-02-09T18:03:33.369 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_autoscale_mode": "off", 2024-02-09T18:03:33.369 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num": 16, 2024-02-09T18:03:33.369 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_placement_num": 16, 2024-02-09T18:03:33.369 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_placement_num_target": 16, 2024-02-09T18:03:33.369 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num_target": 16, 2024-02-09T18:03:33.369 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num_pending": 16, 2024-02-09T18:03:33.369 INFO:teuthology.orchestra.run.smithi099.stdout: "last_pg_merge_meta": { 2024-02-09T18:03:33.369 INFO:teuthology.orchestra.run.smithi099.stdout: "source_pgid": "0.0", 2024-02-09T18:03:33.369 INFO:teuthology.orchestra.run.smithi099.stdout: "ready_epoch": 0, 2024-02-09T18:03:33.369 INFO:teuthology.orchestra.run.smithi099.stdout: "last_epoch_started": 0, 2024-02-09T18:03:33.369 INFO:teuthology.orchestra.run.smithi099.stdout: "last_epoch_clean": 0, 2024-02-09T18:03:33.370 INFO:teuthology.orchestra.run.smithi099.stdout: "source_version": "0'0", 2024-02-09T18:03:33.370 INFO:teuthology.orchestra.run.smithi099.stdout: "target_version": "0'0" 2024-02-09T18:03:33.370 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.370 INFO:teuthology.orchestra.run.smithi099.stdout: "last_change": "886", 2024-02-09T18:03:33.370 INFO:teuthology.orchestra.run.smithi099.stdout: "last_force_op_resend": "0", 2024-02-09T18:03:33.370 INFO:teuthology.orchestra.run.smithi099.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-09T18:03:33.370 INFO:teuthology.orchestra.run.smithi099.stdout: "last_force_op_resend_preluminous": "0", 2024-02-09T18:03:33.370 INFO:teuthology.orchestra.run.smithi099.stdout: "auid": 0, 2024-02-09T18:03:33.370 INFO:teuthology.orchestra.run.smithi099.stdout: "snap_mode": "selfmanaged", 2024-02-09T18:03:33.370 INFO:teuthology.orchestra.run.smithi099.stdout: "snap_seq": 0, 2024-02-09T18:03:33.370 INFO:teuthology.orchestra.run.smithi099.stdout: "snap_epoch": 0, 2024-02-09T18:03:33.370 INFO:teuthology.orchestra.run.smithi099.stdout: "pool_snaps": [], 2024-02-09T18:03:33.370 INFO:teuthology.orchestra.run.smithi099.stdout: "removed_snaps": "[]", 2024-02-09T18:03:33.370 INFO:teuthology.orchestra.run.smithi099.stdout: "quota_max_bytes": 0, 2024-02-09T18:03:33.370 INFO:teuthology.orchestra.run.smithi099.stdout: "quota_max_objects": 0, 2024-02-09T18:03:33.371 INFO:teuthology.orchestra.run.smithi099.stdout: "tiers": [], 2024-02-09T18:03:33.371 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_of": -1, 2024-02-09T18:03:33.371 INFO:teuthology.orchestra.run.smithi099.stdout: "read_tier": -1, 2024-02-09T18:03:33.371 INFO:teuthology.orchestra.run.smithi099.stdout: "write_tier": -1, 2024-02-09T18:03:33.371 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_mode": "none", 2024-02-09T18:03:33.371 INFO:teuthology.orchestra.run.smithi099.stdout: "target_max_bytes": 0, 2024-02-09T18:03:33.371 INFO:teuthology.orchestra.run.smithi099.stdout: "target_max_objects": 0, 2024-02-09T18:03:33.371 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-09T18:03:33.371 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-09T18:03:33.371 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-09T18:03:33.371 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_min_flush_age": 0, 2024-02-09T18:03:33.371 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_min_evict_age": 0, 2024-02-09T18:03:33.371 INFO:teuthology.orchestra.run.smithi099.stdout: "erasure_code_profile": "", 2024-02-09T18:03:33.371 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_params": { 2024-02-09T18:03:33.371 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "none" 2024-02-09T18:03:33.372 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.372 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_period": 0, 2024-02-09T18:03:33.372 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_count": 0, 2024-02-09T18:03:33.372 INFO:teuthology.orchestra.run.smithi099.stdout: "use_gmt_hitset": true, 2024-02-09T18:03:33.372 INFO:teuthology.orchestra.run.smithi099.stdout: "min_read_recency_for_promote": 0, 2024-02-09T18:03:33.372 INFO:teuthology.orchestra.run.smithi099.stdout: "min_write_recency_for_promote": 0, 2024-02-09T18:03:33.372 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_grade_decay_rate": 0, 2024-02-09T18:03:33.372 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_search_last_n": 0, 2024-02-09T18:03:33.372 INFO:teuthology.orchestra.run.smithi099.stdout: "grade_table": [], 2024-02-09T18:03:33.372 INFO:teuthology.orchestra.run.smithi099.stdout: "stripe_width": 0, 2024-02-09T18:03:33.372 INFO:teuthology.orchestra.run.smithi099.stdout: "expected_num_objects": 0, 2024-02-09T18:03:33.372 INFO:teuthology.orchestra.run.smithi099.stdout: "fast_read": false, 2024-02-09T18:03:33.372 INFO:teuthology.orchestra.run.smithi099.stdout: "options": {}, 2024-02-09T18:03:33.372 INFO:teuthology.orchestra.run.smithi099.stdout: "application_metadata": { 2024-02-09T18:03:33.372 INFO:teuthology.orchestra.run.smithi099.stdout: "rados": {} 2024-02-09T18:03:33.373 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.373 INFO:teuthology.orchestra.run.smithi099.stdout: "read_balance": { 2024-02-09T18:03:33.373 INFO:teuthology.orchestra.run.smithi099.stdout: "score_type": "Fair distribution", 2024-02-09T18:03:33.373 INFO:teuthology.orchestra.run.smithi099.stdout: "score_acting": 4.809999942779541, 2024-02-09T18:03:33.373 INFO:teuthology.orchestra.run.smithi099.stdout: "score_stable": 4.809999942779541, 2024-02-09T18:03:33.373 INFO:teuthology.orchestra.run.smithi099.stdout: "optimal_score": 0.9100000262260437, 2024-02-09T18:03:33.373 INFO:teuthology.orchestra.run.smithi099.stdout: "raw_score_acting": 4.380000114440918, 2024-02-09T18:03:33.373 INFO:teuthology.orchestra.run.smithi099.stdout: "raw_score_stable": 4.380000114440918, 2024-02-09T18:03:33.373 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity_weighted": 0.78079599142074585, 2024-02-09T18:03:33.373 INFO:teuthology.orchestra.run.smithi099.stdout: "average_primary_affinity": 0.85000002384185791, 2024-02-09T18:03:33.373 INFO:teuthology.orchestra.run.smithi099.stdout: "average_primary_affinity_weighted": 0.77999997138977051 2024-02-09T18:03:33.373 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.373 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.373 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T18:03:33.373 INFO:teuthology.orchestra.run.smithi099.stdout: "osds": [ 2024-02-09T18:03:33.373 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.374 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 0, 2024-02-09T18:03:33.374 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "21739aa7-e278-45da-9a3f-867385f01baf", 2024-02-09T18:03:33.374 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T18:03:33.374 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T18:03:33.374 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T18:03:33.374 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 0.609619140625, 2024-02-09T18:03:33.374 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T18:03:33.374 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T18:03:33.374 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 8, 2024-02-09T18:03:33.374 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 938, 2024-02-09T18:03:33.374 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T18:03:33.374 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T18:03:33.374 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T18:03:33.374 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:03:33.374 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.375 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:03:33.375 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6802", 2024-02-09T18:03:33.375 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T18:03:33.375 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.375 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.375 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:03:33.375 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6803", 2024-02-09T18:03:33.375 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T18:03:33.375 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.375 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.375 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.375 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T18:03:33.375 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:03:33.375 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.376 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:03:33.376 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6804", 2024-02-09T18:03:33.376 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T18:03:33.376 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.376 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.376 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:03:33.376 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6805", 2024-02-09T18:03:33.376 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T18:03:33.376 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.376 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.376 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.376 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T18:03:33.376 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:03:33.376 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.376 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:03:33.377 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6808", 2024-02-09T18:03:33.377 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T18:03:33.377 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.377 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.377 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:03:33.377 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6809", 2024-02-09T18:03:33.377 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T18:03:33.377 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.377 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.377 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.377 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T18:03:33.377 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:03:33.377 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.377 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:03:33.377 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6806", 2024-02-09T18:03:33.378 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T18:03:33.378 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.378 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.378 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:03:33.378 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6807", 2024-02-09T18:03:33.378 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T18:03:33.378 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.378 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.378 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.378 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.99:6803/3859868923", 2024-02-09T18:03:33.378 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.99:6805/3859868923", 2024-02-09T18:03:33.378 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.99:6809/3859868923", 2024-02-09T18:03:33.378 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.99:6807/3859868923", 2024-02-09T18:03:33.378 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T18:03:33.378 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T18:03:33.379 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T18:03:33.379 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.379 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.379 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.379 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 1, 2024-02-09T18:03:33.379 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "8eab347c-9065-4de1-93ae-e2eb7e11cab7", 2024-02-09T18:03:33.379 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T18:03:33.379 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T18:03:33.379 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T18:03:33.379 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T18:03:33.379 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T18:03:33.379 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T18:03:33.379 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 13, 2024-02-09T18:03:33.379 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 949, 2024-02-09T18:03:33.379 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T18:03:33.380 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T18:03:33.380 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T18:03:33.380 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:03:33.380 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.380 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:03:33.380 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6810", 2024-02-09T18:03:33.380 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T18:03:33.380 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.380 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.380 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:03:33.380 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6811", 2024-02-09T18:03:33.380 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T18:03:33.380 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.380 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.381 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.381 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T18:03:33.381 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:03:33.381 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.381 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:03:33.381 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6812", 2024-02-09T18:03:33.381 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T18:03:33.381 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.381 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.381 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:03:33.381 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6813", 2024-02-09T18:03:33.381 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T18:03:33.381 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.381 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.381 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.382 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T18:03:33.382 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:03:33.382 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.382 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:03:33.382 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6816", 2024-02-09T18:03:33.382 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T18:03:33.382 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.382 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.382 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:03:33.382 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6817", 2024-02-09T18:03:33.382 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T18:03:33.382 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.382 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.382 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.383 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T18:03:33.383 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:03:33.383 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.383 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:03:33.383 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6814", 2024-02-09T18:03:33.383 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T18:03:33.383 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.383 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.383 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:03:33.383 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6815", 2024-02-09T18:03:33.383 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T18:03:33.383 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.383 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.383 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.384 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.99:6811/3868328452", 2024-02-09T18:03:33.384 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.99:6813/3868328452", 2024-02-09T18:03:33.384 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.99:6817/3868328452", 2024-02-09T18:03:33.384 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.99:6815/3868328452", 2024-02-09T18:03:33.384 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T18:03:33.384 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T18:03:33.384 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T18:03:33.384 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.384 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.384 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.384 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 2, 2024-02-09T18:03:33.384 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "575f4257-7e34-485e-963c-2607b69e1220", 2024-02-09T18:03:33.384 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T18:03:33.384 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T18:03:33.384 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T18:03:33.385 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 0.3601837158203125, 2024-02-09T18:03:33.385 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T18:03:33.385 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T18:03:33.385 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 18, 2024-02-09T18:03:33.385 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 945, 2024-02-09T18:03:33.385 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T18:03:33.385 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T18:03:33.385 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T18:03:33.385 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:03:33.385 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.385 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:03:33.385 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6818", 2024-02-09T18:03:33.385 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T18:03:33.385 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.385 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.386 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:03:33.386 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6819", 2024-02-09T18:03:33.386 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T18:03:33.386 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.386 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.386 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.386 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T18:03:33.386 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:03:33.386 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.386 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:03:33.386 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6820", 2024-02-09T18:03:33.386 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T18:03:33.386 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.386 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.386 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:03:33.387 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6821", 2024-02-09T18:03:33.387 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T18:03:33.387 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.387 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.387 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.387 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T18:03:33.387 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:03:33.387 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.387 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:03:33.387 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6824", 2024-02-09T18:03:33.387 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T18:03:33.387 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.387 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.387 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:03:33.387 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6825", 2024-02-09T18:03:33.387 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T18:03:33.388 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.388 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.388 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.388 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T18:03:33.388 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:03:33.388 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.388 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:03:33.388 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6822", 2024-02-09T18:03:33.388 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T18:03:33.388 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.389 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.389 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:03:33.389 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6823", 2024-02-09T18:03:33.389 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T18:03:33.389 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.389 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.389 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.389 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.99:6819/537506237", 2024-02-09T18:03:33.389 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.99:6821/537506237", 2024-02-09T18:03:33.389 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.99:6825/537506237", 2024-02-09T18:03:33.389 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.99:6823/537506237", 2024-02-09T18:03:33.389 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T18:03:33.389 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T18:03:33.389 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T18:03:33.389 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.390 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.390 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.390 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 3, 2024-02-09T18:03:33.390 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "1a520bcf-81bb-4f1d-80fb-11e98008e429", 2024-02-09T18:03:33.390 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T18:03:33.390 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T18:03:33.390 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T18:03:33.390 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 0.8014984130859375, 2024-02-09T18:03:33.390 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 27, 2024-02-09T18:03:33.390 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 814, 2024-02-09T18:03:33.390 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 854, 2024-02-09T18:03:33.390 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 950, 2024-02-09T18:03:33.390 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 815, 2024-02-09T18:03:33.390 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T18:03:33.390 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T18:03:33.391 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:03:33.391 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.391 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:03:33.391 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6826", 2024-02-09T18:03:33.391 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4191781533 2024-02-09T18:03:33.391 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.391 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.391 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:03:33.391 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6827", 2024-02-09T18:03:33.391 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4191781533 2024-02-09T18:03:33.391 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.391 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.391 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.391 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T18:03:33.391 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:03:33.392 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.392 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:03:33.392 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6828", 2024-02-09T18:03:33.392 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4191781533 2024-02-09T18:03:33.392 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.392 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.392 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:03:33.392 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6829", 2024-02-09T18:03:33.392 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4191781533 2024-02-09T18:03:33.392 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.392 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.392 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.392 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T18:03:33.392 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:03:33.393 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.393 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:03:33.393 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6832", 2024-02-09T18:03:33.393 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4191781533 2024-02-09T18:03:33.393 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.393 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.393 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:03:33.393 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6833", 2024-02-09T18:03:33.393 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4191781533 2024-02-09T18:03:33.393 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.393 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.393 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.393 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T18:03:33.393 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:03:33.393 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.394 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:03:33.394 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6830", 2024-02-09T18:03:33.394 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4191781533 2024-02-09T18:03:33.394 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.394 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.394 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:03:33.394 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6831", 2024-02-09T18:03:33.394 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4191781533 2024-02-09T18:03:33.394 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.394 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.394 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.394 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.99:6827/4191781533", 2024-02-09T18:03:33.394 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.99:6829/4191781533", 2024-02-09T18:03:33.394 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.99:6833/4191781533", 2024-02-09T18:03:33.394 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.99:6831/4191781533", 2024-02-09T18:03:33.394 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T18:03:33.395 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T18:03:33.395 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T18:03:33.395 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.395 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.395 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.395 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 4, 2024-02-09T18:03:33.395 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "4617c2fe-c9d8-40cd-887b-303e3215bdf8", 2024-02-09T18:03:33.395 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T18:03:33.395 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T18:03:33.395 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T18:03:33.396 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T18:03:33.396 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T18:03:33.396 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T18:03:33.396 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 32, 2024-02-09T18:03:33.396 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 938, 2024-02-09T18:03:33.396 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T18:03:33.396 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T18:03:33.396 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T18:03:33.396 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:03:33.397 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.397 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:03:33.397 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6800", 2024-02-09T18:03:33.397 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T18:03:33.397 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.397 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.397 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:03:33.397 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6801", 2024-02-09T18:03:33.397 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T18:03:33.397 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.397 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.397 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.397 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T18:03:33.397 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:03:33.397 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.398 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:03:33.398 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6802", 2024-02-09T18:03:33.398 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T18:03:33.398 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.398 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.398 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:03:33.398 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6803", 2024-02-09T18:03:33.398 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T18:03:33.398 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.398 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.398 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.398 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T18:03:33.398 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:03:33.398 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.398 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:03:33.398 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6806", 2024-02-09T18:03:33.399 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T18:03:33.399 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.399 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.399 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:03:33.399 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6807", 2024-02-09T18:03:33.399 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T18:03:33.399 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.399 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.399 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.399 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T18:03:33.399 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:03:33.399 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.399 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:03:33.399 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6804", 2024-02-09T18:03:33.400 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T18:03:33.400 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.400 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.400 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:03:33.400 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6805", 2024-02-09T18:03:33.400 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T18:03:33.400 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.400 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.400 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.400 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.162:6801/4178676504", 2024-02-09T18:03:33.400 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.162:6803/4178676504", 2024-02-09T18:03:33.400 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.162:6807/4178676504", 2024-02-09T18:03:33.400 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.162:6805/4178676504", 2024-02-09T18:03:33.400 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T18:03:33.400 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T18:03:33.401 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T18:03:33.401 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.401 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.401 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.401 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 5, 2024-02-09T18:03:33.401 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "da98e3f3-173e-46fe-be8b-1ccbe8b0dabc", 2024-02-09T18:03:33.401 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T18:03:33.401 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T18:03:33.401 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T18:03:33.401 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 0, 2024-02-09T18:03:33.401 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T18:03:33.401 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T18:03:33.401 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 40, 2024-02-09T18:03:33.401 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 855, 2024-02-09T18:03:33.401 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T18:03:33.401 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T18:03:33.402 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T18:03:33.402 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:03:33.402 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.402 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:03:33.402 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6808", 2024-02-09T18:03:33.402 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T18:03:33.402 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.402 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.402 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:03:33.402 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6809", 2024-02-09T18:03:33.402 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T18:03:33.402 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.402 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.402 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.403 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T18:03:33.403 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:03:33.403 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.403 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:03:33.403 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6810", 2024-02-09T18:03:33.403 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T18:03:33.403 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.403 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.403 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:03:33.403 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6811", 2024-02-09T18:03:33.403 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T18:03:33.403 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.403 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.403 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.403 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T18:03:33.404 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:03:33.404 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.404 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:03:33.404 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6814", 2024-02-09T18:03:33.404 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T18:03:33.404 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.404 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.404 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:03:33.404 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6815", 2024-02-09T18:03:33.404 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T18:03:33.404 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.404 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.404 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.404 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T18:03:33.405 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:03:33.405 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.405 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:03:33.405 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6812", 2024-02-09T18:03:33.405 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T18:03:33.405 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.405 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.405 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:03:33.405 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6813", 2024-02-09T18:03:33.405 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T18:03:33.405 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.405 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.405 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.405 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.162:6809/1489371339", 2024-02-09T18:03:33.405 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.162:6811/1489371339", 2024-02-09T18:03:33.406 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.162:6815/1489371339", 2024-02-09T18:03:33.406 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.162:6813/1489371339", 2024-02-09T18:03:33.406 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T18:03:33.406 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T18:03:33.406 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T18:03:33.406 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.406 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.406 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.406 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 6, 2024-02-09T18:03:33.406 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "ff50382d-5ddf-4cda-bb0c-ff055feeaa40", 2024-02-09T18:03:33.406 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T18:03:33.406 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T18:03:33.406 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T18:03:33.406 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T18:03:33.406 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T18:03:33.406 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T18:03:33.407 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 49, 2024-02-09T18:03:33.407 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 945, 2024-02-09T18:03:33.407 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T18:03:33.407 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T18:03:33.407 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T18:03:33.407 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:03:33.407 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.407 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:03:33.407 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6816", 2024-02-09T18:03:33.407 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T18:03:33.407 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.407 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.407 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:03:33.407 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6817", 2024-02-09T18:03:33.407 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T18:03:33.408 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.408 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.408 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.408 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T18:03:33.408 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:03:33.408 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.408 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:03:33.408 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6818", 2024-02-09T18:03:33.408 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T18:03:33.408 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.408 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.408 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:03:33.408 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6819", 2024-02-09T18:03:33.408 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T18:03:33.408 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.408 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.409 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.409 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T18:03:33.409 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:03:33.409 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.409 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:03:33.409 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6822", 2024-02-09T18:03:33.409 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T18:03:33.409 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.409 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.409 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:03:33.409 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6823", 2024-02-09T18:03:33.409 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T18:03:33.409 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.409 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.409 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.410 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T18:03:33.410 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:03:33.410 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.410 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:03:33.410 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6820", 2024-02-09T18:03:33.410 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T18:03:33.410 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.410 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.410 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:03:33.410 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6821", 2024-02-09T18:03:33.410 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T18:03:33.410 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.410 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.410 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.410 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.162:6817/821169663", 2024-02-09T18:03:33.411 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.162:6819/821169663", 2024-02-09T18:03:33.411 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.162:6823/821169663", 2024-02-09T18:03:33.411 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.162:6821/821169663", 2024-02-09T18:03:33.411 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T18:03:33.411 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T18:03:33.411 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T18:03:33.411 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.411 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.411 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.411 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 7, 2024-02-09T18:03:33.411 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "a82984dc-5e22-424b-a446-ae2ee71eee19", 2024-02-09T18:03:33.411 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T18:03:33.411 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T18:03:33.411 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T18:03:33.411 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 0.5982513427734375, 2024-02-09T18:03:33.411 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T18:03:33.412 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T18:03:33.412 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 57, 2024-02-09T18:03:33.412 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 945, 2024-02-09T18:03:33.412 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T18:03:33.412 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T18:03:33.412 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T18:03:33.412 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:03:33.412 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.412 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:03:33.412 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6824", 2024-02-09T18:03:33.412 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T18:03:33.412 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.412 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.412 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:03:33.412 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6825", 2024-02-09T18:03:33.413 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T18:03:33.413 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.413 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.413 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.413 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T18:03:33.413 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:03:33.413 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.413 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:03:33.413 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6826", 2024-02-09T18:03:33.413 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T18:03:33.413 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.413 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.413 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:03:33.413 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6827", 2024-02-09T18:03:33.413 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T18:03:33.414 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.414 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.414 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.414 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T18:03:33.414 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:03:33.414 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.414 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:03:33.414 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6830", 2024-02-09T18:03:33.414 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T18:03:33.414 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.414 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.414 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:03:33.414 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6831", 2024-02-09T18:03:33.414 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T18:03:33.414 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.414 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.415 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.415 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T18:03:33.415 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:03:33.415 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.415 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:03:33.415 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6828", 2024-02-09T18:03:33.415 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T18:03:33.415 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.415 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.415 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:03:33.415 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6829", 2024-02-09T18:03:33.415 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T18:03:33.415 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.415 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.415 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.415 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.162:6825/965264369", 2024-02-09T18:03:33.416 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.162:6827/965264369", 2024-02-09T18:03:33.416 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.162:6831/965264369", 2024-02-09T18:03:33.416 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.162:6829/965264369", 2024-02-09T18:03:33.416 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T18:03:33.416 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T18:03:33.416 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T18:03:33.416 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.416 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.416 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.416 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 8, 2024-02-09T18:03:33.416 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "63f9d79f-0400-44dc-b388-df00fefb5820", 2024-02-09T18:03:33.417 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T18:03:33.417 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T18:03:33.417 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T18:03:33.417 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T18:03:33.417 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 66, 2024-02-09T18:03:33.417 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 145, 2024-02-09T18:03:33.417 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 171, 2024-02-09T18:03:33.417 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 945, 2024-02-09T18:03:33.417 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 146, 2024-02-09T18:03:33.417 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T18:03:33.417 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T18:03:33.417 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:03:33.417 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.418 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:03:33.418 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6800", 2024-02-09T18:03:33.418 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T18:03:33.418 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.418 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.418 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:03:33.418 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6801", 2024-02-09T18:03:33.418 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T18:03:33.418 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.418 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.418 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.418 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T18:03:33.418 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:03:33.418 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.418 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:03:33.418 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6802", 2024-02-09T18:03:33.419 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T18:03:33.419 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.419 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.419 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:03:33.419 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6803", 2024-02-09T18:03:33.419 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T18:03:33.419 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.419 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.419 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.419 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T18:03:33.419 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:03:33.419 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.419 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:03:33.419 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6806", 2024-02-09T18:03:33.419 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T18:03:33.420 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.420 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.420 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:03:33.420 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6807", 2024-02-09T18:03:33.420 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T18:03:33.420 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.420 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.420 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.420 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T18:03:33.420 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:03:33.420 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.420 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:03:33.420 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6804", 2024-02-09T18:03:33.420 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T18:03:33.420 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.421 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.421 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:03:33.421 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6805", 2024-02-09T18:03:33.421 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T18:03:33.421 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.421 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.421 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.421 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.179:6801/2558294495", 2024-02-09T18:03:33.421 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.179:6803/2558294495", 2024-02-09T18:03:33.421 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.179:6807/2558294495", 2024-02-09T18:03:33.421 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.179:6805/2558294495", 2024-02-09T18:03:33.421 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T18:03:33.421 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T18:03:33.421 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T18:03:33.421 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.421 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.422 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.422 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 9, 2024-02-09T18:03:33.422 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "7d995854-7328-4dfc-b1a9-0edca8d74f46", 2024-02-09T18:03:33.422 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T18:03:33.422 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T18:03:33.422 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T18:03:33.422 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T18:03:33.422 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T18:03:33.422 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T18:03:33.422 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 77, 2024-02-09T18:03:33.422 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 945, 2024-02-09T18:03:33.422 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T18:03:33.422 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T18:03:33.422 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T18:03:33.422 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:03:33.422 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.423 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:03:33.423 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6808", 2024-02-09T18:03:33.423 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T18:03:33.423 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.423 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.423 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:03:33.423 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6809", 2024-02-09T18:03:33.423 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T18:03:33.423 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.423 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.423 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.423 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T18:03:33.423 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:03:33.423 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.423 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:03:33.423 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6810", 2024-02-09T18:03:33.424 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T18:03:33.424 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.424 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.424 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:03:33.424 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6811", 2024-02-09T18:03:33.424 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T18:03:33.424 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.424 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.424 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.424 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T18:03:33.424 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:03:33.424 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.424 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:03:33.424 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6814", 2024-02-09T18:03:33.424 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T18:03:33.424 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.425 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.425 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:03:33.425 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6815", 2024-02-09T18:03:33.425 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T18:03:33.425 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.425 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.425 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.425 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T18:03:33.425 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:03:33.425 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.425 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:03:33.425 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6812", 2024-02-09T18:03:33.425 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T18:03:33.425 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.425 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.425 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:03:33.426 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6813", 2024-02-09T18:03:33.426 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T18:03:33.426 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.426 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.426 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.426 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.179:6809/504427183", 2024-02-09T18:03:33.426 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.179:6811/504427183", 2024-02-09T18:03:33.426 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.179:6815/504427183", 2024-02-09T18:03:33.426 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.179:6813/504427183", 2024-02-09T18:03:33.426 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T18:03:33.426 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T18:03:33.426 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T18:03:33.426 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.426 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.426 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.427 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 10, 2024-02-09T18:03:33.427 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "151fbedd-08e6-46e6-a4e5-a14248399f61", 2024-02-09T18:03:33.427 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T18:03:33.427 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 0, 2024-02-09T18:03:33.427 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 0, 2024-02-09T18:03:33.427 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T18:03:33.427 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T18:03:33.427 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T18:03:33.427 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 84, 2024-02-09T18:03:33.427 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 882, 2024-02-09T18:03:33.427 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T18:03:33.427 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T18:03:33.427 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T18:03:33.428 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:03:33.428 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.428 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:03:33.428 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6816", 2024-02-09T18:03:33.428 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T18:03:33.428 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.428 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.428 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:03:33.428 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6817", 2024-02-09T18:03:33.429 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T18:03:33.429 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.429 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.429 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.429 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T18:03:33.429 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:03:33.429 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.429 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:03:33.429 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6818", 2024-02-09T18:03:33.429 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T18:03:33.429 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.429 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.430 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:03:33.430 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6819", 2024-02-09T18:03:33.430 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T18:03:33.430 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.430 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.430 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.430 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T18:03:33.430 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:03:33.430 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.430 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:03:33.430 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6822", 2024-02-09T18:03:33.430 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T18:03:33.431 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.431 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.431 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:03:33.431 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6823", 2024-02-09T18:03:33.431 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T18:03:33.431 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.431 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.431 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.431 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T18:03:33.431 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:03:33.431 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.432 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:03:33.432 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6820", 2024-02-09T18:03:33.432 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T18:03:33.432 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.432 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.432 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:03:33.432 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6821", 2024-02-09T18:03:33.432 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T18:03:33.432 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.432 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.432 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.433 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.179:6817/2403345164", 2024-02-09T18:03:33.433 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.179:6819/2403345164", 2024-02-09T18:03:33.433 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.179:6823/2403345164", 2024-02-09T18:03:33.433 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.179:6821/2403345164", 2024-02-09T18:03:33.433 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T18:03:33.433 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T18:03:33.433 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T18:03:33.433 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.433 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.433 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.433 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 11, 2024-02-09T18:03:33.434 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "6d3a3ad8-417a-483f-991c-52549dbe7d73", 2024-02-09T18:03:33.434 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T18:03:33.434 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T18:03:33.434 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T18:03:33.434 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T18:03:33.434 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T18:03:33.434 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T18:03:33.434 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 90, 2024-02-09T18:03:33.434 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 943, 2024-02-09T18:03:33.434 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T18:03:33.434 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T18:03:33.434 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T18:03:33.435 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:03:33.435 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.435 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:03:33.435 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6824", 2024-02-09T18:03:33.435 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T18:03:33.435 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.435 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.435 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:03:33.435 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6825", 2024-02-09T18:03:33.435 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T18:03:33.435 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.435 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.436 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.436 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T18:03:33.436 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:03:33.436 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.436 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:03:33.436 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6826", 2024-02-09T18:03:33.436 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T18:03:33.436 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.436 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.436 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:03:33.436 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6827", 2024-02-09T18:03:33.436 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T18:03:33.437 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.437 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.437 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.437 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T18:03:33.437 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:03:33.437 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.437 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:03:33.437 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6830", 2024-02-09T18:03:33.437 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T18:03:33.437 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.437 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.437 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:03:33.438 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6831", 2024-02-09T18:03:33.438 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T18:03:33.438 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.438 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.438 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.438 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T18:03:33.438 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:03:33.438 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.438 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:03:33.438 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6828", 2024-02-09T18:03:33.438 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T18:03:33.438 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.439 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.439 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:03:33.439 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6829", 2024-02-09T18:03:33.439 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T18:03:33.439 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.439 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.439 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.439 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.179:6825/2899402373", 2024-02-09T18:03:33.439 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.179:6827/2899402373", 2024-02-09T18:03:33.439 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.179:6831/2899402373", 2024-02-09T18:03:33.439 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.179:6829/2899402373", 2024-02-09T18:03:33.439 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T18:03:33.439 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T18:03:33.440 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T18:03:33.440 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.440 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.440 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T18:03:33.440 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_xinfo": [ 2024-02-09T18:03:33.440 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.440 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 0, 2024-02-09T18:03:33.440 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T18:03:33.440 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T18:03:33.440 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T18:03:33.440 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T18:03:33.440 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T18:03:33.440 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T18:03:25.851815+0000", 2024-02-09T18:03:33.441 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T18:03:33.441 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.441 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.441 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 1, 2024-02-09T18:03:33.441 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T18:03:33.441 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T18:03:33.441 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T18:03:33.441 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T18:03:33.441 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T18:03:33.441 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T18:03:07.352057+0000", 2024-02-09T18:03:33.441 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T18:03:33.441 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.442 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.442 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 2, 2024-02-09T18:03:33.442 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T18:03:33.442 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T18:03:33.442 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T18:03:33.442 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T18:03:33.442 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T18:03:33.442 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T18:02:38.612967+0000", 2024-02-09T18:03:33.442 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T18:03:33.442 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.442 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.442 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 3, 2024-02-09T18:03:33.443 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T18:03:33.443 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T18:03:33.443 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T18:03:33.443 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T18:03:33.443 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T18:03:33.443 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T18:03:06.473727+0000", 2024-02-09T18:03:33.443 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 814 2024-02-09T18:03:33.443 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.443 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.443 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 4, 2024-02-09T18:03:33.443 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T18:03:33.444 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T18:03:33.444 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T18:03:33.444 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T18:03:33.444 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T18:03:33.444 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T18:03:28.714945+0000", 2024-02-09T18:03:33.444 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T18:03:33.444 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.444 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.444 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 5, 2024-02-09T18:03:33.444 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T18:03:33.444 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T18:03:33.445 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T18:03:33.445 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T18:03:33.445 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T18:03:33.445 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T18:02:30.725309+0000", 2024-02-09T18:03:33.445 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T18:03:33.445 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.445 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.445 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 6, 2024-02-09T18:03:33.445 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T18:03:33.445 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T18:03:33.445 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T18:03:33.445 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T18:03:33.446 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T18:03:33.446 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T18:03:05.756967+0000", 2024-02-09T18:03:33.446 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T18:03:33.446 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.446 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.446 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 7, 2024-02-09T18:03:33.446 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T18:03:33.446 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T18:03:33.446 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T18:03:33.446 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T18:03:33.446 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T18:03:33.446 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T18:02:54.758608+0000", 2024-02-09T18:03:33.446 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T18:03:33.446 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.447 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.447 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 8, 2024-02-09T18:03:33.447 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T18:03:33.447 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T18:03:33.447 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T18:03:33.447 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T18:03:33.447 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T18:03:33.447 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T18:02:28.805072+0000", 2024-02-09T18:03:33.447 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 145 2024-02-09T18:03:33.447 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.447 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.447 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 9, 2024-02-09T18:03:33.448 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T18:03:33.448 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T18:03:33.448 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T18:03:33.448 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T18:03:33.448 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T18:03:33.448 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T18:02:52.741251+0000", 2024-02-09T18:03:33.448 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T18:03:33.448 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.448 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.448 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 10, 2024-02-09T18:03:33.448 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T18:03:33.448 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T18:03:33.449 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T18:03:33.449 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T18:03:33.449 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 65536, 2024-02-09T18:03:33.449 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T18:03:21.800213+0000", 2024-02-09T18:03:33.449 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T18:03:33.449 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.449 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.449 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 11, 2024-02-09T18:03:33.449 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T18:03:33.449 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T18:03:33.449 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T18:03:33.449 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T18:03:33.449 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T18:03:33.450 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T18:02:22.636628+0000", 2024-02-09T18:03:33.450 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T18:03:33.450 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.450 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T18:03:33.450 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_upmap": [ 2024-02-09T18:03:33.450 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.450 INFO:teuthology.orchestra.run.smithi099.stdout: "pgid": "1.7", 2024-02-09T18:03:33.450 INFO:teuthology.orchestra.run.smithi099.stdout: "osds": [ 2024-02-09T18:03:33.450 INFO:teuthology.orchestra.run.smithi099.stdout: 1, 2024-02-09T18:03:33.450 INFO:teuthology.orchestra.run.smithi099.stdout: 8, 2024-02-09T18:03:33.450 INFO:teuthology.orchestra.run.smithi099.stdout: 6 2024-02-09T18:03:33.450 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.451 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.451 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T18:03:33.451 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_upmap_items": [ 2024-02-09T18:03:33.451 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.451 INFO:teuthology.orchestra.run.smithi099.stdout: "pgid": "1.5", 2024-02-09T18:03:33.451 INFO:teuthology.orchestra.run.smithi099.stdout: "mappings": [ 2024-02-09T18:03:33.451 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.451 INFO:teuthology.orchestra.run.smithi099.stdout: "from": 6, 2024-02-09T18:03:33.451 INFO:teuthology.orchestra.run.smithi099.stdout: "to": 8 2024-02-09T18:03:33.451 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.451 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.452 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.452 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.452 INFO:teuthology.orchestra.run.smithi099.stdout: "pgid": "1.1e", 2024-02-09T18:03:33.452 INFO:teuthology.orchestra.run.smithi099.stdout: "mappings": [ 2024-02-09T18:03:33.452 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.452 INFO:teuthology.orchestra.run.smithi099.stdout: "from": 8, 2024-02-09T18:03:33.452 INFO:teuthology.orchestra.run.smithi099.stdout: "to": 3 2024-02-09T18:03:33.452 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.452 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.452 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.452 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T18:03:33.452 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_upmap_primaries": [], 2024-02-09T18:03:33.453 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_temp": [ 2024-02-09T18:03:33.453 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.453 INFO:teuthology.orchestra.run.smithi099.stdout: "pgid": "9.6", 2024-02-09T18:03:33.453 INFO:teuthology.orchestra.run.smithi099.stdout: "osds": [ 2024-02-09T18:03:33.453 INFO:teuthology.orchestra.run.smithi099.stdout: 1, 2024-02-09T18:03:33.453 INFO:teuthology.orchestra.run.smithi099.stdout: 5, 2024-02-09T18:03:33.453 INFO:teuthology.orchestra.run.smithi099.stdout: 10 2024-02-09T18:03:33.453 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.453 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.453 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.453 INFO:teuthology.orchestra.run.smithi099.stdout: "pgid": "9.7", 2024-02-09T18:03:33.454 INFO:teuthology.orchestra.run.smithi099.stdout: "osds": [ 2024-02-09T18:03:33.454 INFO:teuthology.orchestra.run.smithi099.stdout: 3, 2024-02-09T18:03:33.454 INFO:teuthology.orchestra.run.smithi099.stdout: 7, 2024-02-09T18:03:33.454 INFO:teuthology.orchestra.run.smithi099.stdout: 10 2024-02-09T18:03:33.454 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.454 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.454 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.454 INFO:teuthology.orchestra.run.smithi099.stdout: "pgid": "9.9", 2024-02-09T18:03:33.454 INFO:teuthology.orchestra.run.smithi099.stdout: "osds": [ 2024-02-09T18:03:33.454 INFO:teuthology.orchestra.run.smithi099.stdout: 11, 2024-02-09T18:03:33.454 INFO:teuthology.orchestra.run.smithi099.stdout: 5, 2024-02-09T18:03:33.455 INFO:teuthology.orchestra.run.smithi099.stdout: 10 2024-02-09T18:03:33.455 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.455 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.455 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.455 INFO:teuthology.orchestra.run.smithi099.stdout: "pgid": "9.b", 2024-02-09T18:03:33.455 INFO:teuthology.orchestra.run.smithi099.stdout: "osds": [ 2024-02-09T18:03:33.455 INFO:teuthology.orchestra.run.smithi099.stdout: 8, 2024-02-09T18:03:33.455 INFO:teuthology.orchestra.run.smithi099.stdout: 5, 2024-02-09T18:03:33.455 INFO:teuthology.orchestra.run.smithi099.stdout: 10 2024-02-09T18:03:33.456 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.456 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.456 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:03:33.456 INFO:teuthology.orchestra.run.smithi099.stdout: "pgid": "9.e", 2024-02-09T18:03:33.456 INFO:teuthology.orchestra.run.smithi099.stdout: "osds": [ 2024-02-09T18:03:33.456 INFO:teuthology.orchestra.run.smithi099.stdout: 1, 2024-02-09T18:03:33.456 INFO:teuthology.orchestra.run.smithi099.stdout: 7, 2024-02-09T18:03:33.456 INFO:teuthology.orchestra.run.smithi099.stdout: 10 2024-02-09T18:03:33.456 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:03:33.456 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.457 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T18:03:33.457 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_temp": [], 2024-02-09T18:03:33.457 INFO:teuthology.orchestra.run.smithi099.stdout: "blocklist": { 2024-02-09T18:03:33.457 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:0/3904172782": "2024-02-10T17:16:24.229365+0000", 2024-02-09T18:03:33.457 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:6801/1586034576": "2024-02-10T17:16:24.229365+0000", 2024-02-09T18:03:33.457 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:6800/746248609": "2024-02-10T17:15:54.399201+0000", 2024-02-09T18:03:33.457 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:0/4092784197": "2024-02-10T17:16:24.229365+0000", 2024-02-09T18:03:33.457 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:0/2844598101": "2024-02-10T17:15:54.399201+0000", 2024-02-09T18:03:33.457 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:6801/746248609": "2024-02-10T17:15:54.399201+0000", 2024-02-09T18:03:33.457 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:0/2528750050": "2024-02-10T17:15:54.399201+0000", 2024-02-09T18:03:33.457 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:6800/1586034576": "2024-02-10T17:16:24.229365+0000", 2024-02-09T18:03:33.457 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:0/2214140078": "2024-02-10T17:15:54.399201+0000", 2024-02-09T18:03:33.457 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:0/4217821965": "2024-02-10T17:16:24.229365+0000" 2024-02-09T18:03:33.458 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.458 INFO:teuthology.orchestra.run.smithi099.stdout: "range_blocklist": {}, 2024-02-09T18:03:33.458 INFO:teuthology.orchestra.run.smithi099.stdout: "erasure_code_profiles": { 2024-02-09T18:03:33.458 INFO:teuthology.orchestra.run.smithi099.stdout: "default": { 2024-02-09T18:03:33.458 INFO:teuthology.orchestra.run.smithi099.stdout: "crush-failure-domain": "osd", 2024-02-09T18:03:33.458 INFO:teuthology.orchestra.run.smithi099.stdout: "k": "2", 2024-02-09T18:03:33.458 INFO:teuthology.orchestra.run.smithi099.stdout: "m": "1", 2024-02-09T18:03:33.458 INFO:teuthology.orchestra.run.smithi099.stdout: "plugin": "jerasure", 2024-02-09T18:03:33.458 INFO:teuthology.orchestra.run.smithi099.stdout: "technique": "reed_sol_van" 2024-02-09T18:03:33.458 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.458 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:03:33.458 INFO:teuthology.orchestra.run.smithi099.stdout: "removed_snaps_queue": [], 2024-02-09T18:03:33.458 INFO:teuthology.orchestra.run.smithi099.stdout: "new_removed_snaps": [], 2024-02-09T18:03:33.459 INFO:teuthology.orchestra.run.smithi099.stdout: "new_purged_snaps": [], 2024-02-09T18:03:33.459 INFO:teuthology.orchestra.run.smithi099.stdout: "crush_node_flags": {}, 2024-02-09T18:03:33.459 INFO:teuthology.orchestra.run.smithi099.stdout: "device_class_flags": {}, 2024-02-09T18:03:33.459 INFO:teuthology.orchestra.run.smithi099.stdout: "stretch_mode": { 2024-02-09T18:03:33.459 INFO:teuthology.orchestra.run.smithi099.stdout: "stretch_mode_enabled": false, 2024-02-09T18:03:33.459 INFO:teuthology.orchestra.run.smithi099.stdout: "stretch_bucket_count": 0, 2024-02-09T18:03:33.459 INFO:teuthology.orchestra.run.smithi099.stdout: "degraded_stretch_mode": 0, 2024-02-09T18:03:33.459 INFO:teuthology.orchestra.run.smithi099.stdout: "recovering_stretch_mode": 0, 2024-02-09T18:03:33.459 INFO:teuthology.orchestra.run.smithi099.stdout: "stretch_mode_bucket": 0 2024-02-09T18:03:33.459 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:03:33.459 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T18:03:33.643 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:33 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1617309274' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-09T18:03:33.805 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:03:33.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:33.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:03:33.910 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:03:33.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:33 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1617309274' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-09T18:03:33.959 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:33.999 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:33 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1617309274' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-09T18:03:34.078 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:03:34.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:34.087 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:03:34.188 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:03:34.246 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:34.284 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:03:34.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:34.289 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:03:34.390 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:03:34.428 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:34.470 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T18:03:34.470 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T18:03:34.649 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:34 smithi099 ceph-mon[39420]: pgmap v2356: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 42 active+clean; 3.5 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 788 op/s; 9032/170433 objects misplaced (5.299%); 58 MiB/s, 925 objects/s recovering 2024-02-09T18:03:34.698 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:03:34.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:34.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:03:34.746 INFO:tasks.thrashosds.thrasher:j is {'epoch': 952, 'fsid': 'a69c870a-c76e-11ee-95b7-87774f69a715', 'created': '2024-02-09T17:15:26.113606+0000', 'modified': '2024-02-09T18:03:29.596726+0000', 'last_up_change': '2024-02-09T17:59:11.889493+0000', 'last_in_change': '2024-02-09T18:03:13.429500+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 28, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-09T17:18:56.552286+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '544', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '514', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.119999885559082, 'score_stable': 2.130000114440918, 'optimal_score': 1, 'raw_score_acting': 7.119999885559082, 'raw_score_stable': 2.130000114440918, 'primary_affinity_weighted': 0.7807959914207458, 'average_primary_affinity': 0.8500000238418579, 'average_primary_affinity_weighted': 0.7799999713897705, 'error_message': 'pool 1 has acting primaries on OSD(s) with primary affinity 0, read balance score is not accurate'}}, {'pool': 9, 'pool_name': 'unique_pool_7', 'create_time': '2024-02-09T18:00:06.232818+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '886', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 4.809999942779541, 'score_stable': 4.809999942779541, 'optimal_score': 0.9100000262260437, 'raw_score_acting': 4.380000114440918, 'raw_score_stable': 4.380000114440918, 'primary_affinity_weighted': 0.7807959914207458, 'average_primary_affinity': 0.8500000238418579, 'average_primary_affinity_weighted': 0.7799999713897705}}], 'osds': [{'osd': 0, 'uuid': '21739aa7-e278-45da-9a3f-867385f01baf', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.609619140625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 938, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6802', 'nonce': 3859868923}, {'type': 'v1', 'addr': '172.21.15.99:6803', 'nonce': 3859868923}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6804', 'nonce': 3859868923}, {'type': 'v1', 'addr': '172.21.15.99:6805', 'nonce': 3859868923}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6808', 'nonce': 3859868923}, {'type': 'v1', 'addr': '172.21.15.99:6809', 'nonce': 3859868923}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6806', 'nonce': 3859868923}, {'type': 'v1', 'addr': '172.21.15.99:6807', 'nonce': 3859868923}]}, 'public_addr': '172.21.15.99:6803/3859868923', 'cluster_addr': '172.21.15.99:6805/3859868923', 'heartbeat_back_addr': '172.21.15.99:6809/3859868923', 'heartbeat_front_addr': '172.21.15.99:6807/3859868923', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '8eab347c-9065-4de1-93ae-e2eb7e11cab7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 949, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6810', 'nonce': 3868328452}, {'type': 'v1', 'addr': '172.21.15.99:6811', 'nonce': 3868328452}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6812', 'nonce': 3868328452}, {'type': 'v1', 'addr': '172.21.15.99:6813', 'nonce': 3868328452}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6816', 'nonce': 3868328452}, {'type': 'v1', 'addr': '172.21.15.99:6817', 'nonce': 3868328452}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6814', 'nonce': 3868328452}, {'type': 'v1', 'addr': '172.21.15.99:6815', 'nonce': 3868328452}]}, 'public_addr': '172.21.15.99:6811/3868328452', 'cluster_addr': '172.21.15.99:6813/3868328452', 'heartbeat_back_addr': '172.21.15.99:6817/3868328452', 'heartbeat_front_addr': '172.21.15.99:6815/3868328452', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '575f4257-7e34-485e-963c-2607b69e1220', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.3601837158203125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 945, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6818', 'nonce': 537506237}, {'type': 'v1', 'addr': '172.21.15.99:6819', 'nonce': 537506237}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6820', 'nonce': 537506237}, {'type': 'v1', 'addr': '172.21.15.99:6821', 'nonce': 537506237}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6824', 'nonce': 537506237}, {'type': 'v1', 'addr': '172.21.15.99:6825', 'nonce': 537506237}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6822', 'nonce': 537506237}, {'type': 'v1', 'addr': '172.21.15.99:6823', 'nonce': 537506237}]}, 'public_addr': '172.21.15.99:6819/537506237', 'cluster_addr': '172.21.15.99:6821/537506237', 'heartbeat_back_addr': '172.21.15.99:6825/537506237', 'heartbeat_front_addr': '172.21.15.99:6823/537506237', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '1a520bcf-81bb-4f1d-80fb-11e98008e429', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.8014984130859375, 'last_clean_begin': 27, 'last_clean_end': 814, 'up_from': 854, 'up_thru': 950, 'down_at': 815, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6826', 'nonce': 4191781533}, {'type': 'v1', 'addr': '172.21.15.99:6827', 'nonce': 4191781533}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6828', 'nonce': 4191781533}, {'type': 'v1', 'addr': '172.21.15.99:6829', 'nonce': 4191781533}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6832', 'nonce': 4191781533}, {'type': 'v1', 'addr': '172.21.15.99:6833', 'nonce': 4191781533}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6830', 'nonce': 4191781533}, {'type': 'v1', 'addr': '172.21.15.99:6831', 'nonce': 4191781533}]}, 'public_addr': '172.21.15.99:6827/4191781533', 'cluster_addr': '172.21.15.99:6829/4191781533', 'heartbeat_back_addr': '172.21.15.99:6833/4191781533', 'heartbeat_front_addr': '172.21.15.99:6831/4191781533', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '4617c2fe-c9d8-40cd-887b-303e3215bdf8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 938, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6800', 'nonce': 4178676504}, {'type': 'v1', 'addr': '172.21.15.162:6801', 'nonce': 4178676504}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6802', 'nonce': 4178676504}, {'type': 'v1', 'addr': '172.21.15.162:6803', 'nonce': 4178676504}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6806', 'nonce': 4178676504}, {'type': 'v1', 'addr': '172.21.15.162:6807', 'nonce': 4178676504}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6804', 'nonce': 4178676504}, {'type': 'v1', 'addr': '172.21.15.162:6805', 'nonce': 4178676504}]}, 'public_addr': '172.21.15.162:6801/4178676504', 'cluster_addr': '172.21.15.162:6803/4178676504', 'heartbeat_back_addr': '172.21.15.162:6807/4178676504', 'heartbeat_front_addr': '172.21.15.162:6805/4178676504', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'da98e3f3-173e-46fe-be8b-1ccbe8b0dabc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 855, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6808', 'nonce': 1489371339}, {'type': 'v1', 'addr': '172.21.15.162:6809', 'nonce': 1489371339}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6810', 'nonce': 1489371339}, {'type': 'v1', 'addr': '172.21.15.162:6811', 'nonce': 1489371339}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6814', 'nonce': 1489371339}, {'type': 'v1', 'addr': '172.21.15.162:6815', 'nonce': 1489371339}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6812', 'nonce': 1489371339}, {'type': 'v1', 'addr': '172.21.15.162:6813', 'nonce': 1489371339}]}, 'public_addr': '172.21.15.162:6809/1489371339', 'cluster_addr': '172.21.15.162:6811/1489371339', 'heartbeat_back_addr': '172.21.15.162:6815/1489371339', 'heartbeat_front_addr': '172.21.15.162:6813/1489371339', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'ff50382d-5ddf-4cda-bb0c-ff055feeaa40', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 49, 'up_thru': 945, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6816', 'nonce': 821169663}, {'type': 'v1', 'addr': '172.21.15.162:6817', 'nonce': 821169663}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6818', 'nonce': 821169663}, {'type': 'v1', 'addr': '172.21.15.162:6819', 'nonce': 821169663}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6822', 'nonce': 821169663}, {'type': 'v1', 'addr': '172.21.15.162:6823', 'nonce': 821169663}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6820', 'nonce': 821169663}, {'type': 'v1', 'addr': '172.21.15.162:6821', 'nonce': 821169663}]}, 'public_addr': '172.21.15.162:6817/821169663', 'cluster_addr': '172.21.15.162:6819/821169663', 'heartbeat_back_addr': '172.21.15.162:6823/821169663', 'heartbeat_front_addr': '172.21.15.162:6821/821169663', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'a82984dc-5e22-424b-a446-ae2ee71eee19', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.5982513427734375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 57, 'up_thru': 945, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6824', 'nonce': 965264369}, {'type': 'v1', 'addr': '172.21.15.162:6825', 'nonce': 965264369}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6826', 'nonce': 965264369}, {'type': 'v1', 'addr': '172.21.15.162:6827', 'nonce': 965264369}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6830', 'nonce': 965264369}, {'type': 'v1', 'addr': '172.21.15.162:6831', 'nonce': 965264369}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6828', 'nonce': 965264369}, {'type': 'v1', 'addr': '172.21.15.162:6829', 'nonce': 965264369}]}, 'public_addr': '172.21.15.162:6825/965264369', 'cluster_addr': '172.21.15.162:6827/965264369', 'heartbeat_back_addr': '172.21.15.162:6831/965264369', 'heartbeat_front_addr': '172.21.15.162:6829/965264369', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '63f9d79f-0400-44dc-b388-df00fefb5820', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 66, 'last_clean_end': 145, 'up_from': 171, 'up_thru': 945, 'down_at': 146, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6800', 'nonce': 2558294495}, {'type': 'v1', 'addr': '172.21.15.179:6801', 'nonce': 2558294495}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6802', 'nonce': 2558294495}, {'type': 'v1', 'addr': '172.21.15.179:6803', 'nonce': 2558294495}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6806', 'nonce': 2558294495}, {'type': 'v1', 'addr': '172.21.15.179:6807', 'nonce': 2558294495}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6804', 'nonce': 2558294495}, {'type': 'v1', 'addr': '172.21.15.179:6805', 'nonce': 2558294495}]}, 'public_addr': '172.21.15.179:6801/2558294495', 'cluster_addr': '172.21.15.179:6803/2558294495', 'heartbeat_back_addr': '172.21.15.179:6807/2558294495', 'heartbeat_front_addr': '172.21.15.179:6805/2558294495', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '7d995854-7328-4dfc-b1a9-0edca8d74f46', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 77, 'up_thru': 945, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6808', 'nonce': 504427183}, {'type': 'v1', 'addr': '172.21.15.179:6809', 'nonce': 504427183}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6810', 'nonce': 504427183}, {'type': 'v1', 'addr': '172.21.15.179:6811', 'nonce': 504427183}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6814', 'nonce': 504427183}, {'type': 'v1', 'addr': '172.21.15.179:6815', 'nonce': 504427183}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6812', 'nonce': 504427183}, {'type': 'v1', 'addr': '172.21.15.179:6813', 'nonce': 504427183}]}, 'public_addr': '172.21.15.179:6809/504427183', 'cluster_addr': '172.21.15.179:6811/504427183', 'heartbeat_back_addr': '172.21.15.179:6815/504427183', 'heartbeat_front_addr': '172.21.15.179:6813/504427183', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '151fbedd-08e6-46e6-a4e5-a14248399f61', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 84, 'up_thru': 882, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6816', 'nonce': 2403345164}, {'type': 'v1', 'addr': '172.21.15.179:6817', 'nonce': 2403345164}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6818', 'nonce': 2403345164}, {'type': 'v1', 'addr': '172.21.15.179:6819', 'nonce': 2403345164}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6822', 'nonce': 2403345164}, {'type': 'v1', 'addr': '172.21.15.179:6823', 'nonce': 2403345164}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6820', 'nonce': 2403345164}, {'type': 'v1', 'addr': '172.21.15.179:6821', 'nonce': 2403345164}]}, 'public_addr': '172.21.15.179:6817/2403345164', 'cluster_addr': '172.21.15.179:6819/2403345164', 'heartbeat_back_addr': '172.21.15.179:6823/2403345164', 'heartbeat_front_addr': '172.21.15.179:6821/2403345164', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '6d3a3ad8-417a-483f-991c-52549dbe7d73', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 90, 'up_thru': 943, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6824', 'nonce': 2899402373}, {'type': 'v1', 'addr': '172.21.15.179:6825', 'nonce': 2899402373}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6826', 'nonce': 2899402373}, {'type': 'v1', 'addr': '172.21.15.179:6827', 'nonce': 2899402373}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6830', 'nonce': 2899402373}, {'type': 'v1', 'addr': '172.21.15.179:6831', 'nonce': 2899402373}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6828', 'nonce': 2899402373}, {'type': 'v1', 'addr': '172.21.15.179:6829', 'nonce': 2899402373}]}, 'public_addr': '172.21.15.179:6825/2899402373', 'cluster_addr': '172.21.15.179:6827/2899402373', 'heartbeat_back_addr': '172.21.15.179:6831/2899402373', 'heartbeat_front_addr': '172.21.15.179:6829/2899402373', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T18:03:25.851815+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T18:03:07.352057+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T18:02:38.612967+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T18:03:06.473727+0000', 'dead_epoch': 814}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T18:03:28.714945+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T18:02:30.725309+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T18:03:05.756967+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T18:02:54.758608+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T18:02:28.805072+0000', 'dead_epoch': 145}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T18:02:52.741251+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-09T18:03:21.800213+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T18:02:22.636628+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.7', 'osds': [1, 8, 6]}], 'pg_upmap_items': [{'pgid': '1.5', 'mappings': [{'from': 6, 'to': 8}]}, {'pgid': '1.1e', 'mappings': [{'from': 8, 'to': 3}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '9.6', 'osds': [1, 5, 10]}, {'pgid': '9.7', 'osds': [3, 7, 10]}, {'pgid': '9.9', 'osds': [11, 5, 10]}, {'pgid': '9.b', 'osds': [8, 5, 10]}, {'pgid': '9.e', 'osds': [1, 7, 10]}], 'primary_temp': [], 'blocklist': {'172.21.15.99:0/3904172782': '2024-02-10T17:16:24.229365+0000', '172.21.15.99:6801/1586034576': '2024-02-10T17:16:24.229365+0000', '172.21.15.99:6800/746248609': '2024-02-10T17:15:54.399201+0000', '172.21.15.99:0/4092784197': '2024-02-10T17:16:24.229365+0000', '172.21.15.99:0/2844598101': '2024-02-10T17:15:54.399201+0000', '172.21.15.99:6801/746248609': '2024-02-10T17:15:54.399201+0000', '172.21.15.99:0/2528750050': '2024-02-10T17:15:54.399201+0000', '172.21.15.99:6800/1586034576': '2024-02-10T17:16:24.229365+0000', '172.21.15.99:0/2214140078': '2024-02-10T17:15:54.399201+0000', '172.21.15.99:0/4217821965': '2024-02-10T17:16:24.229365+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-09T18:03:34.746 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 1.1e 2024-02-09T18:03:34.746 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd rm-pg-upmap-items 1.1e 2024-02-09T18:03:34.803 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:03:34.862 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:34.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:34 smithi179 ceph-mon[43975]: pgmap v2356: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 42 active+clean; 3.5 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 788 op/s; 9032/170433 objects misplaced (5.299%); 58 MiB/s, 925 objects/s recovering 2024-02-09T18:03:34.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:34 smithi162 ceph-mon[44304]: pgmap v2356: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 42 active+clean; 3.5 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 788 op/s; 9032/170433 objects misplaced (5.299%); 58 MiB/s, 925 objects/s recovering 2024-02-09T18:03:34.977 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:03:34.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:34.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:03:35.087 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T18:03:35.141 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:35.298 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T18:03:35.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:35.304 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T18:03:35.326 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.10 dump_ops_in_flight 2024-02-09T18:03:35.404 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:03:35.434 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:35.460 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:03:35.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:35.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:03:35.565 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:03:35.596 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:35.625 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:03:35.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:35.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:03:35.731 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:03:35.765 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:35.787 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T18:03:35.788 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:03:35.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:35.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:03:35.897 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:03:35.941 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:36.085 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:03:36.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:36.088 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:03:36.189 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:03:36.239 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:36.668 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is unset 2024-02-09T18:03:36.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:36 smithi179 ceph-mon[43975]: pgmap v2357: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 42 active+clean; 3.5 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 505 op/s; 9032/170433 objects misplaced (5.299%); 35 MiB/s, 552 objects/s recovering 2024-02-09T18:03:36.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:36 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2936866874' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T18:03:36.804 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:03:36.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:36.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:03:36.913 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:03:36.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:36 smithi162 ceph-mon[44304]: pgmap v2357: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 42 active+clean; 3.5 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 505 op/s; 9032/170433 objects misplaced (5.299%); 35 MiB/s, 552 objects/s recovering 2024-02-09T18:03:36.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:36 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2936866874' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T18:03:36.955 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:36 smithi099 ceph-mon[39420]: pgmap v2357: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 42 active+clean; 3.5 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 505 op/s; 9032/170433 objects misplaced (5.299%); 35 MiB/s, 552 objects/s recovering 2024-02-09T18:03:36.955 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:36 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2936866874' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T18:03:36.963 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:37.702 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T18:03:37.760 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T18:03:37.768 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T18:03:37.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:37 smithi179 ceph-mon[43975]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T18:03:37.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:37 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2936866874' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T18:03:37.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:37 smithi179 ceph-mon[43975]: osdmap e953: 12 total, 12 up, 11 in 2024-02-09T18:03:37.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:37 smithi162 ceph-mon[44304]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T18:03:37.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:37 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2936866874' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T18:03:37.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:37 smithi162 ceph-mon[44304]: osdmap e953: 12 total, 12 up, 11 in 2024-02-09T18:03:37.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:37 smithi099 ceph-mon[39420]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T18:03:37.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:37 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2936866874' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T18:03:37.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:37 smithi099 ceph-mon[39420]: osdmap e953: 12 total, 12 up, 11 in 2024-02-09T18:03:37.998 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T18:03:38.039 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T18:03:38.061 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T18:03:38.062 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T18:03:38.068 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T18:03:38.104 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T18:03:38.140 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T18:03:38.189 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T18:03:38.217 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T18:03:38.248 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T18:03:38.248 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:03:38.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:38.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:03:38.277 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T18:03:38.283 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T18:03:38.283 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T18:03:38.283 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T18:03:38.283 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T18:03:38.283 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T18:03:38.283 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T18:03:38.283 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T18:03:38.283 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T18:03:38.283 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T18:03:38.283 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T18:03:38.283 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T18:03:38.284 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T18:03:38.358 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:03:38.393 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:38.427 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:03:38.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:38.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:03:38.537 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:03:38.582 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:38.610 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:03:38.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:38.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:03:38.671 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:38 smithi162 ceph-mon[44304]: pgmap v2359: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 42 active+clean; 3.3 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 773 op/s; 7134/162423 objects misplaced (4.392%); 31 MiB/s, 492 objects/s recovering 2024-02-09T18:03:38.720 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T18:03:38.804 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:38 smithi179 ceph-mon[43975]: pgmap v2359: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 42 active+clean; 3.3 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 773 op/s; 7134/162423 objects misplaced (4.392%); 31 MiB/s, 492 objects/s recovering 2024-02-09T18:03:38.804 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:38 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2249543076' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:03:38.804 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:38 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:03:38.804 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:38 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:03:38.830 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:38.842 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.10 dump_blocked_ops 2024-02-09T18:03:38.872 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T18:03:38.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:38.879 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T18:03:38.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:38 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2249543076' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:03:38.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:38 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:03:38.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:38 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:03:38.979 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T18:03:38.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:38 smithi099 ceph-mon[39420]: pgmap v2359: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 42 active+clean; 3.3 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 773 op/s; 7134/162423 objects misplaced (4.392%); 31 MiB/s, 492 objects/s recovering 2024-02-09T18:03:38.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:38 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2249543076' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:03:38.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:38 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:03:38.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:38 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:03:39.060 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:39.102 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T18:03:39.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:39.109 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T18:03:39.210 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:03:39.241 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:39.293 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T18:03:39.796 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:03:39.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:39.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:03:39.811 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:03:39.878 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:39 smithi179 ceph-mon[43975]: osdmap e954: 12 total, 12 up, 11 in 2024-02-09T18:03:39.900 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:03:39.927 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:39 smithi099 ceph-mon[39420]: osdmap e954: 12 total, 12 up, 11 in 2024-02-09T18:03:39.929 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:40.131 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:03:40.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:40.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:03:40.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:39 smithi162 ceph-mon[44304]: osdmap e954: 12 total, 12 up, 11 in 2024-02-09T18:03:40.235 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:03:40.267 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:40.820 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:03:40.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:40.826 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:03:40.928 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T18:03:40.992 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:40 smithi179 ceph-mon[43975]: pgmap v2361: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 42 active+clean; 3.2 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 624 op/s; 6472/159681 objects misplaced (4.053%); 26 MiB/s, 414 objects/s recovering 2024-02-09T18:03:40.992 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:40 smithi179 ceph-mon[43975]: 1.8 scrub starts 2024-02-09T18:03:40.992 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:40 smithi179 ceph-mon[43975]: 1.8 scrub ok 2024-02-09T18:03:40.992 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:40 smithi179 ceph-mon[43975]: osdmap e955: 12 total, 12 up, 11 in 2024-02-09T18:03:40.992 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:40 smithi179 ceph-mon[43975]: 1.17 deep-scrub starts 2024-02-09T18:03:40.992 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:40 smithi179 ceph-mon[43975]: 1.17 deep-scrub ok 2024-02-09T18:03:40.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:40 smithi099 ceph-mon[39420]: pgmap v2361: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 42 active+clean; 3.2 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 624 op/s; 6472/159681 objects misplaced (4.053%); 26 MiB/s, 414 objects/s recovering 2024-02-09T18:03:40.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:40 smithi099 ceph-mon[39420]: 1.8 scrub starts 2024-02-09T18:03:40.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:40 smithi099 ceph-mon[39420]: 1.8 scrub ok 2024-02-09T18:03:40.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:40 smithi099 ceph-mon[39420]: osdmap e955: 12 total, 12 up, 11 in 2024-02-09T18:03:40.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:40 smithi099 ceph-mon[39420]: 1.17 deep-scrub starts 2024-02-09T18:03:40.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:40 smithi099 ceph-mon[39420]: 1.17 deep-scrub ok 2024-02-09T18:03:40.998 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:41.089 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T18:03:41.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:41.097 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T18:03:41.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:40 smithi162 ceph-mon[44304]: pgmap v2361: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 42 active+clean; 3.2 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 624 op/s; 6472/159681 objects misplaced (4.053%); 26 MiB/s, 414 objects/s recovering 2024-02-09T18:03:41.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:40 smithi162 ceph-mon[44304]: 1.8 scrub starts 2024-02-09T18:03:41.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:40 smithi162 ceph-mon[44304]: 1.8 scrub ok 2024-02-09T18:03:41.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:40 smithi162 ceph-mon[44304]: osdmap e955: 12 total, 12 up, 11 in 2024-02-09T18:03:41.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:40 smithi162 ceph-mon[44304]: 1.17 deep-scrub starts 2024-02-09T18:03:41.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:40 smithi162 ceph-mon[44304]: 1.17 deep-scrub ok 2024-02-09T18:03:41.197 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T18:03:41.244 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:41.288 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T18:03:41.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:41.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T18:03:41.396 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:03:41.435 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:41.497 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T18:03:41.497 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-02-09T18:03:41.497 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-02-09T18:03:41.497 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-02-09T18:03:41.497 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T18:03:41.698 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T18:03:41.828 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set noscrub 2024-02-09T18:03:41.869 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:41 smithi179 ceph-mon[43975]: osdmap e956: 12 total, 12 up, 11 in 2024-02-09T18:03:41.870 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:41 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:03:41.870 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:41 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:03:41.870 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:41 smithi179 ceph-mon[43975]: 1.11 scrub starts 2024-02-09T18:03:41.870 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:41 smithi179 ceph-mon[43975]: 1.11 scrub ok 2024-02-09T18:03:41.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:41 smithi099 ceph-mon[39420]: osdmap e956: 12 total, 12 up, 11 in 2024-02-09T18:03:41.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:41 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:03:41.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:41 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:03:41.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:41 smithi099 ceph-mon[39420]: 1.11 scrub starts 2024-02-09T18:03:41.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:41 smithi099 ceph-mon[39420]: 1.11 scrub ok 2024-02-09T18:03:42.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:41 smithi162 ceph-mon[44304]: osdmap e956: 12 total, 12 up, 11 in 2024-02-09T18:03:42.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:41 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:03:42.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:41 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:03:42.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:41 smithi162 ceph-mon[44304]: 1.11 scrub starts 2024-02-09T18:03:42.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:41 smithi162 ceph-mon[44304]: 1.11 scrub ok 2024-02-09T18:03:42.263 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.10 dump_historic_ops 2024-02-09T18:03:42.684 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T18:03:42.697 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:03:42.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:42.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:03:42.804 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:03:42.865 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:42.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:42 smithi099 ceph-mon[39420]: pgmap v2364: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 42 active+clean; 3.1 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 689 op/s; 4465/154050 objects misplaced (2.898%); 40 MiB/s, 637 objects/s recovering 2024-02-09T18:03:42.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:42 smithi099 ceph-mon[39420]: osdmap e957: 12 total, 12 up, 11 in 2024-02-09T18:03:42.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:42 smithi099 ceph-mon[39420]: 1.4 scrub starts 2024-02-09T18:03:42.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:42 smithi099 ceph-mon[39420]: 1.4 scrub ok 2024-02-09T18:03:43.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:42 smithi179 ceph-mon[43975]: pgmap v2364: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 42 active+clean; 3.1 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 689 op/s; 4465/154050 objects misplaced (2.898%); 40 MiB/s, 637 objects/s recovering 2024-02-09T18:03:43.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:42 smithi179 ceph-mon[43975]: osdmap e957: 12 total, 12 up, 11 in 2024-02-09T18:03:43.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:42 smithi179 ceph-mon[43975]: 1.4 scrub starts 2024-02-09T18:03:43.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:42 smithi179 ceph-mon[43975]: 1.4 scrub ok 2024-02-09T18:03:43.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:42 smithi162 ceph-mon[44304]: pgmap v2364: 47 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 42 active+clean; 3.1 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 689 op/s; 4465/154050 objects misplaced (2.898%); 40 MiB/s, 637 objects/s recovering 2024-02-09T18:03:43.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:42 smithi162 ceph-mon[44304]: osdmap e957: 12 total, 12 up, 11 in 2024-02-09T18:03:43.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:42 smithi162 ceph-mon[44304]: 1.4 scrub starts 2024-02-09T18:03:43.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:42 smithi162 ceph-mon[44304]: 1.4 scrub ok 2024-02-09T18:03:43.234 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:03:43.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:43.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:03:43.264 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:03:43.275 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:03:43.344 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:03:43.400 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:43.550 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:03:43.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:43.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:03:43.659 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:03:43.703 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:43.735 INFO:teuthology.orchestra.run.smithi099.stderr:clear 1.1e pg_upmap_items mapping 2024-02-09T18:03:43.752 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:03:43.761 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:43.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:03:43.856 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:43 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1514604277' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "1.1e"}]: dispatch 2024-02-09T18:03:43.856 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:43 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "1.1e"}]: dispatch 2024-02-09T18:03:43.863 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T18:03:43.909 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:43.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:43 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1514604277' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "1.1e"}]: dispatch 2024-02-09T18:03:43.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:43 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "1.1e"}]: dispatch 2024-02-09T18:03:43.938 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T18:03:43.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:43.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T18:03:44.047 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:03:44.111 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:44.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:43 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1514604277' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "1.1e"}]: dispatch 2024-02-09T18:03:44.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:43 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "1.1e"}]: dispatch 2024-02-09T18:03:44.492 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:03:44.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:44.495 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:03:44.596 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T18:03:44.641 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:44.695 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T18:03:44.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:44.706 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T18:03:44.741 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:44 smithi179 ceph-mon[43975]: pgmap v2366: 47 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 44 active+clean; 3.1 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 659 op/s; 3965/150483 objects misplaced (2.635%); 29 MiB/s, 464 objects/s recovering 2024-02-09T18:03:44.741 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:44 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "1.1e"}]': finished 2024-02-09T18:03:44.742 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:44 smithi179 ceph-mon[43975]: osdmap e958: 12 total, 12 up, 11 in 2024-02-09T18:03:44.742 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:44 smithi179 ceph-mon[43975]: 9.5 scrub starts 2024-02-09T18:03:44.808 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T18:03:44.850 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:44.889 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T18:03:44.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:44.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T18:03:44.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:44 smithi099 ceph-mon[39420]: pgmap v2366: 47 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 44 active+clean; 3.1 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 659 op/s; 3965/150483 objects misplaced (2.635%); 29 MiB/s, 464 objects/s recovering 2024-02-09T18:03:44.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:44 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "1.1e"}]': finished 2024-02-09T18:03:44.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:44 smithi099 ceph-mon[39420]: osdmap e958: 12 total, 12 up, 11 in 2024-02-09T18:03:44.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:44 smithi099 ceph-mon[39420]: 9.5 scrub starts 2024-02-09T18:03:44.999 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:03:45.047 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:45.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:44 smithi162 ceph-mon[44304]: pgmap v2366: 47 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 44 active+clean; 3.1 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 659 op/s; 3965/150483 objects misplaced (2.635%); 29 MiB/s, 464 objects/s recovering 2024-02-09T18:03:45.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:44 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "1.1e"}]': finished 2024-02-09T18:03:45.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:44 smithi162 ceph-mon[44304]: osdmap e958: 12 total, 12 up, 11 in 2024-02-09T18:03:45.179 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:44 smithi162 ceph-mon[44304]: 9.5 scrub starts 2024-02-09T18:03:45.254 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T18:03:45.254 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-02-09T18:03:45.254 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-02-09T18:03:45.254 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-02-09T18:03:45.254 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.254 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:236004 8.2 8:40ef1fe3:::benchmark_data_smithi179_148205_object112932:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e862)", 2024-02-09T18:03:45.254 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:59:26.375049+0000", 2024-02-09T18:03:45.254 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 258.87209123399998, 2024-02-09T18:03:45.255 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8983950969999999, 2024-02-09T18:03:45.255 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:03:45.255 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-02-09T18:03:45.255 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:03:45.255 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:03:45.255 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:03:45.255 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 236004 2024-02-09T18:03:45.255 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.255 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:03:45.255 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.255 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:03:45.255 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.375049+0000", 2024-02-09T18:03:45.255 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:03:45.255 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.256 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.256 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:03:45.256 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.375049+0000", 2024-02-09T18:03:45.256 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:03:45.256 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.256 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.256 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:03:45.256 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.375049+0000", 2024-02-09T18:03:45.256 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8799999999999998e-07 2024-02-09T18:03:45.256 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.256 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.256 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:03:45.256 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.375052+0000", 2024-02-09T18:03:45.256 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2220000000000001e-06 2024-02-09T18:03:45.257 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.257 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.257 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:03:45.257 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.375058+0000", 2024-02-09T18:03:45.257 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1879999999999999e-06 2024-02-09T18:03:45.257 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.257 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.257 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:03:45.257 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.375064+0000", 2024-02-09T18:03:45.257 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7049999999999996e-06 2024-02-09T18:03:45.257 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.257 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.257 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:03:45.257 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.386004+0000", 2024-02-09T18:03:45.257 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010940818 2024-02-09T18:03:45.258 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.258 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.258 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:03:45.258 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.386116+0000", 2024-02-09T18:03:45.258 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000111963 2024-02-09T18:03:45.258 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.258 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.258 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,4", 2024-02-09T18:03:45.258 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.386236+0000", 2024-02-09T18:03:45.258 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000119412 2024-02-09T18:03:45.258 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.258 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.258 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:03:45.258 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.393404+0000", 2024-02-09T18:03:45.258 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0071677149999999999 2024-02-09T18:03:45.259 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.259 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.259 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:03:45.259 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.393430+0000", 2024-02-09T18:03:45.259 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6991e-05 2024-02-09T18:03:45.259 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.259 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.259 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:03:45.259 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.407175+0000", 2024-02-09T18:03:45.259 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013744605999999999 2024-02-09T18:03:45.259 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.259 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.259 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:03:45.259 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.407189+0000", 2024-02-09T18:03:45.259 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4334e-05 2024-02-09T18:03:45.260 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.260 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.260 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:03:45.260 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.006142+0000", 2024-02-09T18:03:45.260 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.59895221300000001 2024-02-09T18:03:45.260 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.260 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.260 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:03:45.260 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.006155+0000", 2024-02-09T18:03:45.260 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3543000000000001e-05 2024-02-09T18:03:45.260 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.260 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.260 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:03:45.260 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.273348+0000", 2024-02-09T18:03:45.261 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2671923569999999 2024-02-09T18:03:45.261 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.261 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.261 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:03:45.261 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.273444+0000", 2024-02-09T18:03:45.261 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6642000000000001e-05 2024-02-09T18:03:45.261 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:03:45.261 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:03:45.261 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:03:45.261 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.261 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.261 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:236026 8.5 8:ad941cae:::benchmark_data_smithi179_148205_object112954:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e863)", 2024-02-09T18:03:45.261 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:59:26.414625+0000", 2024-02-09T18:03:45.261 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 258.83251529799998, 2024-02-09T18:03:45.261 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.834063314, 2024-02-09T18:03:45.262 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:03:45.262 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:03:45.262 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:03:45.262 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:03:45.262 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:03:45.262 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 236026 2024-02-09T18:03:45.262 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.262 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:03:45.262 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.262 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:03:45.262 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.414625+0000", 2024-02-09T18:03:45.262 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:03:45.262 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.262 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.262 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:03:45.262 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.414625+0000", 2024-02-09T18:03:45.263 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:03:45.263 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.263 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.263 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:03:45.263 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.414626+0000", 2024-02-09T18:03:45.263 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4109999999999999e-06 2024-02-09T18:03:45.263 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.263 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.263 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:03:45.263 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.414632+0000", 2024-02-09T18:03:45.263 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3550000000000004e-06 2024-02-09T18:03:45.263 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.263 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.263 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:03:45.263 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.414644+0000", 2024-02-09T18:03:45.263 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1849e-05 2024-02-09T18:03:45.264 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.264 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.264 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:03:45.264 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.414652+0000", 2024-02-09T18:03:45.264 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.9349999999999994e-06 2024-02-09T18:03:45.264 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.264 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.264 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:03:45.264 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.414722+0000", 2024-02-09T18:03:45.264 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9941e-05 2024-02-09T18:03:45.264 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.264 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.264 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:03:45.264 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.414732+0000", 2024-02-09T18:03:45.264 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0932e-05 2024-02-09T18:03:45.265 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.265 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.265 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:03:45.265 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.007379+0000", 2024-02-09T18:03:45.265 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.59264656999999998 2024-02-09T18:03:45.265 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.265 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.265 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:03:45.265 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.007390+0000", 2024-02-09T18:03:45.265 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1085e-05 2024-02-09T18:03:45.265 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.265 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.265 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:03:45.265 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.241566+0000", 2024-02-09T18:03:45.265 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2341757929999999 2024-02-09T18:03:45.265 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.266 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.266 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:03:45.266 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.241697+0000", 2024-02-09T18:03:45.266 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013109800000000001 2024-02-09T18:03:45.266 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.266 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.266 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4,11", 2024-02-09T18:03:45.266 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.241780+0000", 2024-02-09T18:03:45.266 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3157000000000004e-05 2024-02-09T18:03:45.266 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.266 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.266 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:03:45.266 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.248365+0000", 2024-02-09T18:03:45.266 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0065845749999999996 2024-02-09T18:03:45.266 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.266 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.267 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:03:45.267 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.248386+0000", 2024-02-09T18:03:45.267 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1569999999999998e-05 2024-02-09T18:03:45.267 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.267 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.267 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:03:45.267 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.248446+0000", 2024-02-09T18:03:45.267 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9531999999999999e-05 2024-02-09T18:03:45.267 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.267 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.267 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:03:45.267 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.248513+0000", 2024-02-09T18:03:45.267 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7459000000000006e-05 2024-02-09T18:03:45.267 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.268 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.268 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:03:45.268 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.248580+0000", 2024-02-09T18:03:45.268 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6608999999999999e-05 2024-02-09T18:03:45.268 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.268 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.268 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:03:45.268 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.248688+0000", 2024-02-09T18:03:45.268 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010844300000000001 2024-02-09T18:03:45.268 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:03:45.269 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:03:45.269 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:03:45.269 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.269 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.269 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:236033 8.5 8:ad8e1f4f:::benchmark_data_smithi179_148205_object112961:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e863)", 2024-02-09T18:03:45.269 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:59:26.428459+0000", 2024-02-09T18:03:45.269 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 258.81868107700001, 2024-02-09T18:03:45.269 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8368791660000001, 2024-02-09T18:03:45.269 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:03:45.269 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:03:45.269 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:03:45.269 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:03:45.269 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:03:45.269 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 236033 2024-02-09T18:03:45.269 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.270 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:03:45.270 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.270 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:03:45.270 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.428459+0000", 2024-02-09T18:03:45.270 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:03:45.270 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.270 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.270 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:03:45.270 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.428459+0000", 2024-02-09T18:03:45.270 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:03:45.270 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.270 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.270 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:03:45.270 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.428461+0000", 2024-02-09T18:03:45.271 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0080000000000001e-06 2024-02-09T18:03:45.271 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.271 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.271 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:03:45.271 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.428469+0000", 2024-02-09T18:03:45.271 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2199999999999992e-06 2024-02-09T18:03:45.271 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.271 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.271 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:03:45.271 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.455561+0000", 2024-02-09T18:03:45.271 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.027091944 2024-02-09T18:03:45.271 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.271 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.271 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:03:45.271 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.455575+0000", 2024-02-09T18:03:45.272 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3230999999999999e-05 2024-02-09T18:03:45.272 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.272 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.272 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:03:45.272 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.455665+0000", 2024-02-09T18:03:45.272 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.0321000000000005e-05 2024-02-09T18:03:45.272 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.272 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.272 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:03:45.272 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.455678+0000", 2024-02-09T18:03:45.272 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3125999999999999e-05 2024-02-09T18:03:45.272 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.272 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.272 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:03:45.273 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.007516+0000", 2024-02-09T18:03:45.273 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.55183836200000003 2024-02-09T18:03:45.273 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.273 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.273 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:03:45.273 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.007529+0000", 2024-02-09T18:03:45.273 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2636000000000001e-05 2024-02-09T18:03:45.273 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.273 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.273 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:03:45.273 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.248729+0000", 2024-02-09T18:03:45.273 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.241199443 2024-02-09T18:03:45.273 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.273 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.273 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:03:45.273 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.248845+0000", 2024-02-09T18:03:45.274 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000116072 2024-02-09T18:03:45.274 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.274 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.274 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4,11", 2024-02-09T18:03:45.274 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.248974+0000", 2024-02-09T18:03:45.274 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012956299999999999 2024-02-09T18:03:45.274 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.274 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.274 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:03:45.274 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.253563+0000", 2024-02-09T18:03:45.274 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.004589007 2024-02-09T18:03:45.274 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.275 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.275 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:03:45.275 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.253610+0000", 2024-02-09T18:03:45.275 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6653999999999998e-05 2024-02-09T18:03:45.275 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.275 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.275 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:03:45.275 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.265079+0000", 2024-02-09T18:03:45.275 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011468715000000001 2024-02-09T18:03:45.275 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.275 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.275 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:03:45.275 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.265176+0000", 2024-02-09T18:03:45.275 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7295999999999998e-05 2024-02-09T18:03:45.275 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.276 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.276 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:03:45.276 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.265233+0000", 2024-02-09T18:03:45.276 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6900000000000001e-05 2024-02-09T18:03:45.276 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.276 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.276 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:03:45.276 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.265338+0000", 2024-02-09T18:03:45.276 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010566800000000001 2024-02-09T18:03:45.276 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:03:45.276 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:03:45.276 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:03:45.276 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.276 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.277 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:236034 8.2 8:418c03b5:::benchmark_data_smithi179_148205_object112962:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e863)", 2024-02-09T18:03:45.277 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:59:26.455616+0000", 2024-02-09T18:03:45.277 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 258.79152462799999, 2024-02-09T18:03:45.277 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8388413300000002, 2024-02-09T18:03:45.277 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:03:45.277 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:03:45.277 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:03:45.277 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:03:45.277 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:03:45.277 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 236034 2024-02-09T18:03:45.277 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.277 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:03:45.277 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.277 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:03:45.277 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.455616+0000", 2024-02-09T18:03:45.278 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:03:45.278 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.278 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.278 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:03:45.278 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.455616+0000", 2024-02-09T18:03:45.278 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:03:45.278 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.278 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.278 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:03:45.278 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.455617+0000", 2024-02-09T18:03:45.278 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5349999999999999e-06 2024-02-09T18:03:45.278 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.278 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.278 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:03:45.278 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.455623+0000", 2024-02-09T18:03:45.279 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4539999999999998e-06 2024-02-09T18:03:45.279 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.279 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.279 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:03:45.279 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.455633+0000", 2024-02-09T18:03:45.279 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0592e-05 2024-02-09T18:03:45.279 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.279 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.279 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:03:45.279 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.455639+0000", 2024-02-09T18:03:45.279 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9710000000000001e-06 2024-02-09T18:03:45.279 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.279 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.279 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:03:45.279 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.455721+0000", 2024-02-09T18:03:45.279 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1332e-05 2024-02-09T18:03:45.280 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.280 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.280 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:03:45.280 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.455731+0000", 2024-02-09T18:03:45.280 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0346999999999999e-05 2024-02-09T18:03:45.280 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.280 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.280 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:03:45.280 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.006221+0000", 2024-02-09T18:03:45.280 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.55049037899999997 2024-02-09T18:03:45.280 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.280 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.280 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:03:45.280 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.006230+0000", 2024-02-09T18:03:45.281 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.1549999999999996e-06 2024-02-09T18:03:45.281 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.281 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.281 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:03:45.281 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.273690+0000", 2024-02-09T18:03:45.281 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.267459924 2024-02-09T18:03:45.281 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.281 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.281 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:03:45.281 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.273764+0000", 2024-02-09T18:03:45.281 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.3337000000000006e-05 2024-02-09T18:03:45.281 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.281 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.281 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,4,9", 2024-02-09T18:03:45.281 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.273854+0000", 2024-02-09T18:03:45.282 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.0600999999999995e-05 2024-02-09T18:03:45.282 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.282 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.282 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:03:45.282 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.294163+0000", 2024-02-09T18:03:45.282 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.020309026000000001 2024-02-09T18:03:45.282 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.282 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.282 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:03:45.282 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.294180+0000", 2024-02-09T18:03:45.282 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7122999999999999e-05 2024-02-09T18:03:45.282 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.282 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.282 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:03:45.282 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.294239+0000", 2024-02-09T18:03:45.283 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8445e-05 2024-02-09T18:03:45.283 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.283 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.283 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:03:45.283 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.294281+0000", 2024-02-09T18:03:45.283 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2203999999999998e-05 2024-02-09T18:03:45.283 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.283 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.283 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:03:45.283 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.294340+0000", 2024-02-09T18:03:45.283 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9023000000000001e-05 2024-02-09T18:03:45.283 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.283 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.283 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:03:45.284 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.294457+0000", 2024-02-09T18:03:45.284 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000116882 2024-02-09T18:03:45.284 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:03:45.284 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:03:45.284 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:03:45.284 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.284 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.284 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:236036 8.2 8:4fcb3643:::benchmark_data_smithi179_148205_object112964:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e863)", 2024-02-09T18:03:45.284 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:59:26.455656+0000", 2024-02-09T18:03:45.284 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 258.79148406000002, 2024-02-09T18:03:45.284 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8499204420000002, 2024-02-09T18:03:45.284 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:03:45.284 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:03:45.284 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:03:45.285 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:03:45.285 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:03:45.285 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 236036 2024-02-09T18:03:45.285 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.285 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:03:45.285 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.285 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:03:45.285 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.455656+0000", 2024-02-09T18:03:45.285 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:03:45.285 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.285 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.285 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:03:45.286 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.455656+0000", 2024-02-09T18:03:45.286 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:03:45.286 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.286 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.286 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:03:45.286 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.455657+0000", 2024-02-09T18:03:45.287 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.27e-07 2024-02-09T18:03:45.287 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.287 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.287 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:03:45.287 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.455659+0000", 2024-02-09T18:03:45.287 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.554e-06 2024-02-09T18:03:45.287 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.287 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.287 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:03:45.287 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.455665+0000", 2024-02-09T18:03:45.287 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2190000000000001e-06 2024-02-09T18:03:45.287 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.287 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.287 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:03:45.288 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.455670+0000", 2024-02-09T18:03:45.288 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9219999999999996e-06 2024-02-09T18:03:45.288 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.288 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.288 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:03:45.288 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.455937+0000", 2024-02-09T18:03:45.288 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00026710699999999999 2024-02-09T18:03:45.288 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.289 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.289 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:03:45.289 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.455947+0000", 2024-02-09T18:03:45.289 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5400000000000001e-06 2024-02-09T18:03:45.289 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.289 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.289 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:03:45.289 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.006257+0000", 2024-02-09T18:03:45.289 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.55031018899999995 2024-02-09T18:03:45.289 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.289 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.289 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:03:45.289 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.006265+0000", 2024-02-09T18:03:45.289 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4340000000000003e-06 2024-02-09T18:03:45.290 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.290 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.290 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:03:45.290 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.278585+0000", 2024-02-09T18:03:45.290 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.272320501 2024-02-09T18:03:45.290 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.290 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.290 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:03:45.290 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.278733+0000", 2024-02-09T18:03:45.290 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000147627 2024-02-09T18:03:45.290 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.290 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.290 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,4,9", 2024-02-09T18:03:45.290 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.278889+0000", 2024-02-09T18:03:45.291 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015604700000000001 2024-02-09T18:03:45.291 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.291 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.291 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:03:45.291 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.294187+0000", 2024-02-09T18:03:45.291 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.015298262999999999 2024-02-09T18:03:45.291 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.291 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.291 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:03:45.291 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.294504+0000", 2024-02-09T18:03:45.291 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00031670299999999999 2024-02-09T18:03:45.291 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.291 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.291 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:03:45.291 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.305250+0000", 2024-02-09T18:03:45.292 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.01074599 2024-02-09T18:03:45.292 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.292 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.292 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:03:45.292 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.305422+0000", 2024-02-09T18:03:45.292 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00017232700000000001 2024-02-09T18:03:45.292 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.292 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.292 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:03:45.292 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.305473+0000", 2024-02-09T18:03:45.292 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1119000000000003e-05 2024-02-09T18:03:45.292 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.292 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.292 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:03:45.292 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.305577+0000", 2024-02-09T18:03:45.293 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000103473 2024-02-09T18:03:45.293 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:03:45.293 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:03:45.293 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:03:45.293 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.293 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.293 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:236039 8.5 8:a407574c:::benchmark_data_smithi179_148205_object112967:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e863)", 2024-02-09T18:03:45.293 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:59:26.456976+0000", 2024-02-09T18:03:45.293 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 258.79016411200001, 2024-02-09T18:03:45.293 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.808632196, 2024-02-09T18:03:45.293 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:03:45.293 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:03:45.293 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:03:45.293 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:03:45.294 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:03:45.294 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 236039 2024-02-09T18:03:45.294 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.294 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:03:45.294 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.294 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:03:45.294 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.456976+0000", 2024-02-09T18:03:45.294 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:03:45.294 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.294 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.294 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:03:45.294 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.456976+0000", 2024-02-09T18:03:45.294 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:03:45.294 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.294 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.295 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:03:45.295 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.456979+0000", 2024-02-09T18:03:45.295 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3080000000000002e-06 2024-02-09T18:03:45.295 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.295 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.295 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:03:45.295 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.456985+0000", 2024-02-09T18:03:45.295 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.63e-06 2024-02-09T18:03:45.295 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.295 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.295 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:03:45.295 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.456999+0000", 2024-02-09T18:03:45.295 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4132e-05 2024-02-09T18:03:45.295 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.296 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.296 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:03:45.296 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.457008+0000", 2024-02-09T18:03:45.296 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.1179999999999998e-06 2024-02-09T18:03:45.296 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.296 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.296 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:03:45.296 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.457082+0000", 2024-02-09T18:03:45.296 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.3135999999999999e-05 2024-02-09T18:03:45.296 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.296 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.296 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:03:45.296 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.457091+0000", 2024-02-09T18:03:45.297 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5959999999999993e-06 2024-02-09T18:03:45.297 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.297 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.297 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:03:45.297 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.007556+0000", 2024-02-09T18:03:45.297 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.55046519400000005 2024-02-09T18:03:45.297 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.297 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.297 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:03:45.297 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.007564+0000", 2024-02-09T18:03:45.297 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.3339999999999996e-06 2024-02-09T18:03:45.297 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.297 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.297 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:03:45.297 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.253643+0000", 2024-02-09T18:03:45.298 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.246079443 2024-02-09T18:03:45.298 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.298 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.298 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:03:45.298 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.254504+0000", 2024-02-09T18:03:45.298 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00086107999999999996 2024-02-09T18:03:45.298 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.298 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.298 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4,11", 2024-02-09T18:03:45.298 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.254658+0000", 2024-02-09T18:03:45.298 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015400200000000001 2024-02-09T18:03:45.298 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.298 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.299 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:03:45.299 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.265158+0000", 2024-02-09T18:03:45.299 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010500083 2024-02-09T18:03:45.299 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.299 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.299 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:03:45.299 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.265347+0000", 2024-02-09T18:03:45.299 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00018832699999999999 2024-02-09T18:03:45.299 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.299 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.299 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:03:45.299 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.265386+0000", 2024-02-09T18:03:45.299 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9870000000000003e-05 2024-02-09T18:03:45.299 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.299 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.300 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:03:45.300 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.265434+0000", 2024-02-09T18:03:45.300 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7627999999999999e-05 2024-02-09T18:03:45.300 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.300 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.300 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:03:45.300 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.265490+0000", 2024-02-09T18:03:45.300 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5448999999999997e-05 2024-02-09T18:03:45.300 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.300 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.300 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:03:45.300 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.265608+0000", 2024-02-09T18:03:45.300 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000118866 2024-02-09T18:03:45.301 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:03:45.301 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:03:45.301 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:03:45.301 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.301 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.301 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:236056 8.5 8:afbd21ac:::benchmark_data_smithi179_148205_object112984:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e863)", 2024-02-09T18:03:45.301 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:59:26.494045+0000", 2024-02-09T18:03:45.301 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 258.75309557600002, 2024-02-09T18:03:45.301 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7864225920000001, 2024-02-09T18:03:45.301 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:03:45.301 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:03:45.301 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:03:45.301 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:03:45.301 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:03:45.301 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 236056 2024-02-09T18:03:45.302 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.302 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:03:45.302 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.302 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:03:45.302 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.494045+0000", 2024-02-09T18:03:45.302 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:03:45.302 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.302 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.302 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:03:45.302 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.494045+0000", 2024-02-09T18:03:45.302 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:03:45.302 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.302 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.302 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:03:45.303 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.494046+0000", 2024-02-09T18:03:45.303 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.359e-06 2024-02-09T18:03:45.303 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.303 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.303 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:03:45.303 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.494053+0000", 2024-02-09T18:03:45.303 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8240000000000002e-06 2024-02-09T18:03:45.303 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.303 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.303 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:03:45.303 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.494065+0000", 2024-02-09T18:03:45.303 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2089999999999999e-05 2024-02-09T18:03:45.303 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.303 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.303 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:03:45.304 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.494072+0000", 2024-02-09T18:03:45.304 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1389999999999998e-06 2024-02-09T18:03:45.304 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.304 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.304 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:03:45.304 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.494156+0000", 2024-02-09T18:03:45.304 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4151000000000001e-05 2024-02-09T18:03:45.304 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.304 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.304 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:03:45.304 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.494167+0000", 2024-02-09T18:03:45.304 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0929e-05 2024-02-09T18:03:45.304 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.304 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.304 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:03:45.305 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.007579+0000", 2024-02-09T18:03:45.305 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.513412176 2024-02-09T18:03:45.305 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.305 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.305 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:03:45.305 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.007584+0000", 2024-02-09T18:03:45.305 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3889999999999998e-06 2024-02-09T18:03:45.305 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.305 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.305 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:03:45.306 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.259989+0000", 2024-02-09T18:03:45.306 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.252405336 2024-02-09T18:03:45.306 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.306 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.306 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:03:45.306 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.260127+0000", 2024-02-09T18:03:45.306 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000137992 2024-02-09T18:03:45.306 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.306 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.306 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4,11", 2024-02-09T18:03:45.306 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.260278+0000", 2024-02-09T18:03:45.306 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015093600000000001 2024-02-09T18:03:45.306 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.306 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.306 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:03:45.307 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.265399+0000", 2024-02-09T18:03:45.307 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0051205479999999999 2024-02-09T18:03:45.307 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.307 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.307 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:03:45.307 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.265660+0000", 2024-02-09T18:03:45.307 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00026186900000000002 2024-02-09T18:03:45.307 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.307 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.307 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:03:45.307 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.280217+0000", 2024-02-09T18:03:45.307 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014556098999999999 2024-02-09T18:03:45.307 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.307 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.307 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:03:45.308 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.280275+0000", 2024-02-09T18:03:45.308 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.808e-05 2024-02-09T18:03:45.308 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.308 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.308 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:03:45.308 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.280341+0000", 2024-02-09T18:03:45.308 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5888000000000003e-05 2024-02-09T18:03:45.308 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.308 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.308 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:03:45.308 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.280467+0000", 2024-02-09T18:03:45.308 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000126787 2024-02-09T18:03:45.308 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:03:45.308 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:03:45.309 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:03:45.309 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.309 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.309 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:236057 8.2 8:461a0d56:::benchmark_data_smithi179_148205_object112985:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e863)", 2024-02-09T18:03:45.310 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:59:26.494215+0000", 2024-02-09T18:03:45.310 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 258.75292502000002, 2024-02-09T18:03:45.310 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.811620604, 2024-02-09T18:03:45.310 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:03:45.310 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:03:45.310 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:03:45.310 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:03:45.310 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:03:45.310 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 236057 2024-02-09T18:03:45.310 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.310 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:03:45.310 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.310 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:03:45.311 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.494215+0000", 2024-02-09T18:03:45.311 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:03:45.311 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.311 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.311 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:03:45.311 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.494215+0000", 2024-02-09T18:03:45.311 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:03:45.311 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.311 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.311 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:03:45.311 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.494216+0000", 2024-02-09T18:03:45.311 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.0599999999999999e-07 2024-02-09T18:03:45.311 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.311 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.312 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:03:45.312 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.494220+0000", 2024-02-09T18:03:45.312 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7469999999999999e-06 2024-02-09T18:03:45.312 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.312 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.312 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:03:45.312 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.494229+0000", 2024-02-09T18:03:45.312 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2220000000000008e-06 2024-02-09T18:03:45.312 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.312 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.312 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:03:45.312 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.494234+0000", 2024-02-09T18:03:45.312 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1159999999999998e-06 2024-02-09T18:03:45.312 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.312 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.313 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:03:45.313 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.494289+0000", 2024-02-09T18:03:45.313 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5121000000000003e-05 2024-02-09T18:03:45.313 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.313 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.313 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:03:45.313 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.494302+0000", 2024-02-09T18:03:45.313 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2221e-05 2024-02-09T18:03:45.313 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.313 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.313 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:03:45.313 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.006367+0000", 2024-02-09T18:03:45.313 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.512065722 2024-02-09T18:03:45.313 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.313 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.314 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:03:45.314 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.006379+0000", 2024-02-09T18:03:45.314 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1562e-05 2024-02-09T18:03:45.314 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.314 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.314 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:03:45.314 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.284221+0000", 2024-02-09T18:03:45.314 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2778420449999999 2024-02-09T18:03:45.314 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.314 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.314 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:03:45.314 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.284358+0000", 2024-02-09T18:03:45.314 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013716600000000001 2024-02-09T18:03:45.315 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.315 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.315 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,4,9", 2024-02-09T18:03:45.315 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.284496+0000", 2024-02-09T18:03:45.315 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013806100000000001 2024-02-09T18:03:45.315 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.315 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.315 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:03:45.315 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.294194+0000", 2024-02-09T18:03:45.315 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0096976349999999996 2024-02-09T18:03:45.315 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.315 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.315 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:03:45.315 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.305305+0000", 2024-02-09T18:03:45.315 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011110890999999999 2024-02-09T18:03:45.316 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.316 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.316 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:03:45.316 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.305614+0000", 2024-02-09T18:03:45.316 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00030973200000000001 2024-02-09T18:03:45.316 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.316 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.316 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:03:45.316 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.305690+0000", 2024-02-09T18:03:45.316 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5555000000000004e-05 2024-02-09T18:03:45.316 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.316 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.316 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:03:45.316 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.305738+0000", 2024-02-09T18:03:45.317 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8018e-05 2024-02-09T18:03:45.317 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.317 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.317 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:03:45.317 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.305836+0000", 2024-02-09T18:03:45.317 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7884e-05 2024-02-09T18:03:45.317 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:03:45.317 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:03:45.317 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:03:45.317 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.317 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.317 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:236058 8.5 8:a5607057:::benchmark_data_smithi179_148205_object112986:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e863)", 2024-02-09T18:03:45.317 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:59:26.498098+0000", 2024-02-09T18:03:45.317 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 258.74904259499999, 2024-02-09T18:03:45.317 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7935283659999999, 2024-02-09T18:03:45.318 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:03:45.318 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:03:45.318 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:03:45.318 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:03:45.318 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:03:45.318 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 236058 2024-02-09T18:03:45.318 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.318 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:03:45.318 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.318 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:03:45.318 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.498098+0000", 2024-02-09T18:03:45.318 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:03:45.318 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.318 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.319 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:03:45.319 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.498098+0000", 2024-02-09T18:03:45.319 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:03:45.319 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.319 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.319 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:03:45.319 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.498099+0000", 2024-02-09T18:03:45.319 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3400000000000001e-06 2024-02-09T18:03:45.319 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.319 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.319 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:03:45.319 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.498104+0000", 2024-02-09T18:03:45.319 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0490000000000003e-06 2024-02-09T18:03:45.319 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.319 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.320 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:03:45.320 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.498116+0000", 2024-02-09T18:03:45.320 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2021000000000001e-05 2024-02-09T18:03:45.320 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.320 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.320 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:03:45.320 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.498124+0000", 2024-02-09T18:03:45.320 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.892e-06 2024-02-09T18:03:45.320 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.320 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.320 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:03:45.320 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.498184+0000", 2024-02-09T18:03:45.320 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9481999999999997e-05 2024-02-09T18:03:45.320 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.321 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.321 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:03:45.321 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.498192+0000", 2024-02-09T18:03:45.321 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.011e-06 2024-02-09T18:03:45.321 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.321 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.321 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:03:45.321 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.007624+0000", 2024-02-09T18:03:45.321 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.50943232199999999 2024-02-09T18:03:45.321 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.321 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.321 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:03:45.321 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.007628+0000", 2024-02-09T18:03:45.321 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4129999999999999e-06 2024-02-09T18:03:45.321 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.322 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.322 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:03:45.322 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.265695+0000", 2024-02-09T18:03:45.322 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.258067217 2024-02-09T18:03:45.322 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.322 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.322 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:03:45.322 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.265848+0000", 2024-02-09T18:03:45.322 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015218799999999999 2024-02-09T18:03:45.322 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.322 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.322 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4,11", 2024-02-09T18:03:45.322 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.265936+0000", 2024-02-09T18:03:45.322 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8640000000000005e-05 2024-02-09T18:03:45.323 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.323 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.323 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:03:45.323 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.280082+0000", 2024-02-09T18:03:45.323 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014145985 2024-02-09T18:03:45.323 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.323 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.323 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:03:45.323 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.280157+0000", 2024-02-09T18:03:45.323 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4746000000000006e-05 2024-02-09T18:03:45.323 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.323 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.323 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:03:45.323 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.291374+0000", 2024-02-09T18:03:45.323 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011217122 2024-02-09T18:03:45.324 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.324 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.324 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:03:45.324 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.291441+0000", 2024-02-09T18:03:45.324 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6841999999999995e-05 2024-02-09T18:03:45.324 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.324 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.324 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:03:45.324 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.291510+0000", 2024-02-09T18:03:45.324 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9494999999999998e-05 2024-02-09T18:03:45.324 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.324 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.324 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:03:45.324 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.291626+0000", 2024-02-09T18:03:45.325 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011560100000000001 2024-02-09T18:03:45.325 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:03:45.325 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:03:45.325 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:03:45.325 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.325 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.325 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:236065 8.5 8:a06c5d66:::benchmark_data_smithi179_148205_object112993:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e863)", 2024-02-09T18:03:45.325 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:59:26.528647+0000", 2024-02-09T18:03:45.325 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 258.71849333599999, 2024-02-09T18:03:45.325 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7632569490000001, 2024-02-09T18:03:45.325 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:03:45.325 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:03:45.325 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:03:45.325 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:03:45.325 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:03:45.326 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 236065 2024-02-09T18:03:45.326 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.326 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:03:45.326 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.326 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:03:45.326 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.528647+0000", 2024-02-09T18:03:45.326 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:03:45.326 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.326 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.326 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:03:45.326 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.528647+0000", 2024-02-09T18:03:45.326 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:03:45.326 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.326 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.327 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:03:45.327 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.528648+0000", 2024-02-09T18:03:45.327 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.271e-06 2024-02-09T18:03:45.327 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.327 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.327 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:03:45.327 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.528653+0000", 2024-02-09T18:03:45.327 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1030000000000001e-06 2024-02-09T18:03:45.327 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.327 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.327 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:03:45.327 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.528664+0000", 2024-02-09T18:03:45.328 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0728e-05 2024-02-09T18:03:45.328 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.328 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.328 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:03:45.328 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.528672+0000", 2024-02-09T18:03:45.328 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.9209999999999992e-06 2024-02-09T18:03:45.328 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.329 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.329 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:03:45.329 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.528743+0000", 2024-02-09T18:03:45.329 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1292999999999997e-05 2024-02-09T18:03:45.329 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.329 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.329 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:03:45.329 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.528757+0000", 2024-02-09T18:03:45.329 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3213999999999999e-05 2024-02-09T18:03:45.329 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.329 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.329 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:03:45.329 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.007659+0000", 2024-02-09T18:03:45.329 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.47890293 2024-02-09T18:03:45.329 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.330 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.330 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:03:45.330 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.007664+0000", 2024-02-09T18:03:45.330 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3590000000000001e-06 2024-02-09T18:03:45.330 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.330 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.330 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:03:45.330 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.270147+0000", 2024-02-09T18:03:45.330 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.262483529 2024-02-09T18:03:45.330 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.330 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.330 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:03:45.330 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.270275+0000", 2024-02-09T18:03:45.330 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012748800000000001 2024-02-09T18:03:45.331 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.331 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.331 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4,11", 2024-02-09T18:03:45.331 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.270390+0000", 2024-02-09T18:03:45.331 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000114889 2024-02-09T18:03:45.331 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.331 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.331 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:03:45.331 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.280226+0000", 2024-02-09T18:03:45.331 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0098362129999999999 2024-02-09T18:03:45.331 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.331 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.331 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:03:45.331 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.280523+0000", 2024-02-09T18:03:45.331 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00029722800000000002 2024-02-09T18:03:45.331 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.332 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.332 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:03:45.332 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.291675+0000", 2024-02-09T18:03:45.332 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011151797 2024-02-09T18:03:45.332 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.332 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.332 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:03:45.332 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.291733+0000", 2024-02-09T18:03:45.332 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8140000000000002e-05 2024-02-09T18:03:45.332 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.332 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.332 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:03:45.332 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.291794+0000", 2024-02-09T18:03:45.332 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0979e-05 2024-02-09T18:03:45.333 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.333 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.333 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:03:45.333 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.291904+0000", 2024-02-09T18:03:45.333 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000109867 2024-02-09T18:03:45.333 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:03:45.333 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:03:45.333 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:03:45.333 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.333 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.334 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:236067 8.5 8:af1cfb54:::benchmark_data_smithi179_148205_object112995:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e863)", 2024-02-09T18:03:45.334 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:59:26.539658+0000", 2024-02-09T18:03:45.334 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 258.70748185600002, 2024-02-09T18:03:45.334 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.761787164, 2024-02-09T18:03:45.334 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:03:45.334 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:03:45.334 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:03:45.334 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:03:45.334 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:03:45.334 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 236067 2024-02-09T18:03:45.334 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.334 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:03:45.334 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.334 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:03:45.335 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.539658+0000", 2024-02-09T18:03:45.335 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:03:45.335 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.335 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.335 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:03:45.335 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.539658+0000", 2024-02-09T18:03:45.335 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:03:45.335 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.335 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.335 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:03:45.335 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.539660+0000", 2024-02-09T18:03:45.335 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.531e-06 2024-02-09T18:03:45.335 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.335 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.335 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:03:45.336 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.539667+0000", 2024-02-09T18:03:45.336 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5799999999999997e-06 2024-02-09T18:03:45.336 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.336 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.336 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:03:45.336 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.539683+0000", 2024-02-09T18:03:45.336 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6577000000000001e-05 2024-02-09T18:03:45.336 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.336 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.336 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:03:45.336 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.539693+0000", 2024-02-09T18:03:45.336 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0161e-05 2024-02-09T18:03:45.336 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.336 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.337 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:03:45.337 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.539771+0000", 2024-02-09T18:03:45.337 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7669999999999993e-05 2024-02-09T18:03:45.337 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.337 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.337 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:03:45.338 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.539785+0000", 2024-02-09T18:03:45.338 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3665999999999999e-05 2024-02-09T18:03:45.338 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.338 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.338 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:03:45.338 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.008228+0000", 2024-02-09T18:03:45.338 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.46844367100000001 2024-02-09T18:03:45.338 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.338 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.338 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:03:45.338 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.008235+0000", 2024-02-09T18:03:45.338 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.0160000000000003e-06 2024-02-09T18:03:45.338 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.339 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.339 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:03:45.339 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.275373+0000", 2024-02-09T18:03:45.339 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.267137822 2024-02-09T18:03:45.339 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.339 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.339 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:03:45.339 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.275491+0000", 2024-02-09T18:03:45.339 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011803000000000001 2024-02-09T18:03:45.339 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.339 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.339 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4,11", 2024-02-09T18:03:45.339 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.275624+0000", 2024-02-09T18:03:45.339 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013240800000000001 2024-02-09T18:03:45.340 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.340 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.340 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:03:45.340 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.285950+0000", 2024-02-09T18:03:45.340 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010325968 2024-02-09T18:03:45.340 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.340 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.340 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:03:45.340 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.291315+0000", 2024-02-09T18:03:45.340 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0053651439999999996 2024-02-09T18:03:45.340 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.340 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.340 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:03:45.340 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.291954+0000", 2024-02-09T18:03:45.341 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00063973399999999998 2024-02-09T18:03:45.341 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.341 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.341 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:03:45.341 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.301266+0000", 2024-02-09T18:03:45.341 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0093119099999999996 2024-02-09T18:03:45.341 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.341 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.341 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:03:45.341 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.301328+0000", 2024-02-09T18:03:45.341 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1575999999999995e-05 2024-02-09T18:03:45.341 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.341 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.341 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:03:45.342 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.301446+0000", 2024-02-09T18:03:45.342 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0001177 2024-02-09T18:03:45.342 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:03:45.342 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:03:45.342 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:03:45.342 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.342 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.342 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.26563.0:204326 9.e e946/943 9:75e9ee8a:::benchmark_data_smithi179_161915_object69444:head v 946'12815, mlcod=945'12812)", 2024-02-09T18:03:45.342 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:17.475438+0000", 2024-02-09T18:03:45.342 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 27.771702292000001, 2024-02-09T18:03:45.342 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.43709178599999998, 2024-02-09T18:03:45.342 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:03:45.342 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:03:45.342 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:03:45.342 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.343 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:03:45.343 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.475438+0000", 2024-02-09T18:03:45.343 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:03:45.343 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.343 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.343 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:03:45.343 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.475438+0000", 2024-02-09T18:03:45.343 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:03:45.343 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.343 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.343 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:03:45.343 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.475439+0000", 2024-02-09T18:03:45.343 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.47e-07 2024-02-09T18:03:45.343 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.344 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.344 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:03:45.344 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.475447+0000", 2024-02-09T18:03:45.344 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.9610000000000005e-06 2024-02-09T18:03:45.344 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.344 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.344 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:03:45.344 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.475464+0000", 2024-02-09T18:03:45.344 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6929999999999999e-05 2024-02-09T18:03:45.344 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.344 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.344 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:03:45.344 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.475473+0000", 2024-02-09T18:03:45.344 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.6719999999999993e-06 2024-02-09T18:03:45.344 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.345 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.345 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:03:45.345 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.484457+0000", 2024-02-09T18:03:45.345 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0089841810000000008 2024-02-09T18:03:45.345 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.345 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.345 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:03:45.345 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.484485+0000", 2024-02-09T18:03:45.345 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7871000000000001e-05 2024-02-09T18:03:45.345 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.345 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.345 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:03:45.345 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.912479+0000", 2024-02-09T18:03:45.345 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.42799490299999998 2024-02-09T18:03:45.345 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.346 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.346 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:03:45.346 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.912530+0000", 2024-02-09T18:03:45.346 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0321000000000002e-05 2024-02-09T18:03:45.346 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:03:45.346 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:03:45.346 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:03:45.346 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.346 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.346 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.26563.0:204332 9.6 e946/943 9:647e4d95:::benchmark_data_smithi179_161915_object69450:head v 946'12703, mlcod=945'12701)", 2024-02-09T18:03:45.346 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:17.481214+0000", 2024-02-09T18:03:45.346 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 27.765926242999999, 2024-02-09T18:03:45.346 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.52768333700000003, 2024-02-09T18:03:45.347 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:03:45.347 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:03:45.347 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:03:45.347 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.347 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:03:45.347 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.481214+0000", 2024-02-09T18:03:45.347 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:03:45.347 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.347 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.347 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:03:45.347 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.481214+0000", 2024-02-09T18:03:45.347 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:03:45.347 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.347 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.347 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:03:45.347 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.481214+0000", 2024-02-09T18:03:45.347 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1e-07 2024-02-09T18:03:45.348 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.348 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.348 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:03:45.348 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.481220+0000", 2024-02-09T18:03:45.348 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4709999999999998e-06 2024-02-09T18:03:45.348 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.348 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.348 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:03:45.348 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.481226+0000", 2024-02-09T18:03:45.348 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2489999999999998e-06 2024-02-09T18:03:45.348 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.348 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.348 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:03:45.348 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.481234+0000", 2024-02-09T18:03:45.348 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5129999999999999e-06 2024-02-09T18:03:45.349 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.349 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.349 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:03:45.349 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.489666+0000", 2024-02-09T18:03:45.349 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0084323179999999994 2024-02-09T18:03:45.349 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.349 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.349 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:03:45.349 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.489689+0000", 2024-02-09T18:03:45.349 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3507999999999999e-05 2024-02-09T18:03:45.349 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.349 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.349 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:03:45.349 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:18.008848+0000", 2024-02-09T18:03:45.350 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.51915874900000003 2024-02-09T18:03:45.350 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.350 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.350 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:03:45.350 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:18.008897+0000", 2024-02-09T18:03:45.350 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9218999999999998e-05 2024-02-09T18:03:45.350 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:03:45.350 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:03:45.350 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:03:45.350 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.350 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.350 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.26563.0:204346 9.e e946/943 9:7b6ff32e:::benchmark_data_smithi179_161915_object69464:head v 946'12816, mlcod=945'12813)", 2024-02-09T18:03:45.350 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:17.515163+0000", 2024-02-09T18:03:45.351 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 27.731977236999999, 2024-02-09T18:03:45.351 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.50301004100000002, 2024-02-09T18:03:45.351 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:03:45.351 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:03:45.351 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:03:45.351 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.351 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:03:45.351 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.515163+0000", 2024-02-09T18:03:45.351 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:03:45.351 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.351 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.351 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:03:45.352 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.515163+0000", 2024-02-09T18:03:45.352 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:03:45.352 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.352 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.352 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:03:45.352 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.515164+0000", 2024-02-09T18:03:45.352 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1519999999999999e-06 2024-02-09T18:03:45.352 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.352 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.352 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:03:45.352 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.515174+0000", 2024-02-09T18:03:45.352 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6940000000000007e-06 2024-02-09T18:03:45.352 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.353 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.353 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:03:45.353 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.515195+0000", 2024-02-09T18:03:45.353 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0971e-05 2024-02-09T18:03:45.353 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.353 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.353 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:03:45.353 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.515204+0000", 2024-02-09T18:03:45.353 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.4620000000000002e-06 2024-02-09T18:03:45.353 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.353 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.353 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:03:45.353 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.516391+0000", 2024-02-09T18:03:45.353 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0011864639999999999 2024-02-09T18:03:45.354 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.354 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.354 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:03:45.354 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.516416+0000", 2024-02-09T18:03:45.354 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5077999999999999e-05 2024-02-09T18:03:45.354 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.354 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.354 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:03:45.354 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:18.018118+0000", 2024-02-09T18:03:45.354 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.50170174199999995 2024-02-09T18:03:45.354 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.354 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.354 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:03:45.354 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:18.018173+0000", 2024-02-09T18:03:45.355 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5478e-05 2024-02-09T18:03:45.355 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:03:45.355 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:03:45.355 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:03:45.355 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.355 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.355 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.26563.0:204342 9.7 e946/943 9:ef4e912e:::benchmark_data_smithi179_161915_object69460:head v 946'12698, mlcod=945'12696)", 2024-02-09T18:03:45.355 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:17.521143+0000", 2024-02-09T18:03:45.355 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 27.725997489000001, 2024-02-09T18:03:45.355 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.39707365500000003, 2024-02-09T18:03:45.355 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:03:45.355 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:03:45.355 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:03:45.355 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.355 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:03:45.355 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.521143+0000", 2024-02-09T18:03:45.356 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:03:45.356 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.356 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.356 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:03:45.356 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.521143+0000", 2024-02-09T18:03:45.356 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:03:45.356 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.356 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.356 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:03:45.356 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.521144+0000", 2024-02-09T18:03:45.356 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3799999999999999e-06 2024-02-09T18:03:45.356 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.357 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.357 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:03:45.357 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.521153+0000", 2024-02-09T18:03:45.357 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5569999999999996e-06 2024-02-09T18:03:45.357 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.357 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.357 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:03:45.357 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.521171+0000", 2024-02-09T18:03:45.357 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8012999999999998e-05 2024-02-09T18:03:45.357 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.357 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.357 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:03:45.357 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.521179+0000", 2024-02-09T18:03:45.357 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.0919999999999998e-06 2024-02-09T18:03:45.358 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.358 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.358 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:03:45.358 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.523195+0000", 2024-02-09T18:03:45.358 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0020156369999999998 2024-02-09T18:03:45.358 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.358 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.358 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:03:45.358 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.523219+0000", 2024-02-09T18:03:45.358 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4045000000000001e-05 2024-02-09T18:03:45.358 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.358 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.358 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:03:45.358 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.918163+0000", 2024-02-09T18:03:45.358 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.39494473699999999 2024-02-09T18:03:45.359 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.359 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.359 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:03:45.359 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.918216+0000", 2024-02-09T18:03:45.359 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3193999999999999e-05 2024-02-09T18:03:45.359 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:03:45.359 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:03:45.359 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:03:45.359 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.359 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.359 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.26563.0:204356 9.e e946/943 9:7ed8cb3e:::benchmark_data_smithi179_161915_object69474:head v 946'12817, mlcod=945'12813)", 2024-02-09T18:03:45.359 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:17.529235+0000", 2024-02-09T18:03:45.359 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 27.717905146, 2024-02-09T18:03:45.359 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.488972974, 2024-02-09T18:03:45.360 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:03:45.360 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:03:45.360 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:03:45.360 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.360 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:03:45.360 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.529235+0000", 2024-02-09T18:03:45.360 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:03:45.360 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.360 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.360 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:03:45.360 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.529235+0000", 2024-02-09T18:03:45.360 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:03:45.360 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.361 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.361 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:03:45.361 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.529236+0000", 2024-02-09T18:03:45.361 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.22e-06 2024-02-09T18:03:45.361 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.361 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.361 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:03:45.361 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.529245+0000", 2024-02-09T18:03:45.361 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.6729999999999998e-06 2024-02-09T18:03:45.361 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.361 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.361 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:03:45.361 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.529258+0000", 2024-02-09T18:03:45.361 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2928000000000001e-05 2024-02-09T18:03:45.362 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.362 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.362 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:03:45.362 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.529267+0000", 2024-02-09T18:03:45.362 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5220000000000007e-06 2024-02-09T18:03:45.362 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.362 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.362 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:03:45.362 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.550372+0000", 2024-02-09T18:03:45.362 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.021105426 2024-02-09T18:03:45.362 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.362 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.362 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:03:45.362 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.550395+0000", 2024-02-09T18:03:45.363 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2614000000000001e-05 2024-02-09T18:03:45.363 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.363 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.363 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:03:45.363 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:18.018178+0000", 2024-02-09T18:03:45.363 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.467782965 2024-02-09T18:03:45.363 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.363 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.363 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:03:45.363 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:18.018208+0000", 2024-02-09T18:03:45.363 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0626000000000001e-05 2024-02-09T18:03:45.363 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:03:45.363 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:03:45.363 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:03:45.363 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.364 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.364 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.26563.0:204359 9.b e946/943 9:d380bded:::benchmark_data_smithi179_161915_object69477:head v 946'12851, mlcod=945'12849)", 2024-02-09T18:03:45.364 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:17.544167+0000", 2024-02-09T18:03:45.364 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 27.702973234000002, 2024-02-09T18:03:45.364 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.46437108700000002, 2024-02-09T18:03:45.364 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:03:45.364 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:03:45.364 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:03:45.364 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.364 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:03:45.364 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.544167+0000", 2024-02-09T18:03:45.364 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:03:45.364 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.365 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.365 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:03:45.365 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.544167+0000", 2024-02-09T18:03:45.365 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:03:45.365 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.365 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.365 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:03:45.365 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.544167+0000", 2024-02-09T18:03:45.365 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7099999999999998e-07 2024-02-09T18:03:45.365 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.365 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.365 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:03:45.365 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.544174+0000", 2024-02-09T18:03:45.365 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3589999999999996e-06 2024-02-09T18:03:45.365 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.366 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.366 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:03:45.366 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.544189+0000", 2024-02-09T18:03:45.366 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4907e-05 2024-02-09T18:03:45.366 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.366 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.366 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:03:45.366 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.544197+0000", 2024-02-09T18:03:45.366 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7690000000000002e-06 2024-02-09T18:03:45.366 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.366 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.366 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:03:45.366 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.544281+0000", 2024-02-09T18:03:45.366 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3199000000000001e-05 2024-02-09T18:03:45.366 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.367 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.367 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:03:45.367 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.544308+0000", 2024-02-09T18:03:45.367 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7350000000000001e-05 2024-02-09T18:03:45.367 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.367 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.367 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:03:45.367 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:18.008486+0000", 2024-02-09T18:03:45.367 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.46417840199999999 2024-02-09T18:03:45.367 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.367 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.367 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:03:45.367 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:18.008538+0000", 2024-02-09T18:03:45.367 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1829999999999997e-05 2024-02-09T18:03:45.367 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:03:45.368 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:03:45.368 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:03:45.368 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.368 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.368 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.26563.0:204365 9.b e946/943 9:d4e000a7:::benchmark_data_smithi179_161915_object69483:head v 946'12852, mlcod=945'12849)", 2024-02-09T18:03:45.368 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:17.554395+0000", 2024-02-09T18:03:45.368 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 27.692744935, 2024-02-09T18:03:45.368 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.45417522300000002, 2024-02-09T18:03:45.368 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:03:45.368 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:03:45.368 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:03:45.368 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.368 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:03:45.369 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.554395+0000", 2024-02-09T18:03:45.369 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:03:45.369 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.369 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.369 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:03:45.369 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.554395+0000", 2024-02-09T18:03:45.369 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:03:45.369 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.369 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.369 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:03:45.369 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.554397+0000", 2024-02-09T18:03:45.369 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.331e-06 2024-02-09T18:03:45.369 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.369 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.369 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:03:45.370 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.554406+0000", 2024-02-09T18:03:45.370 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5980000000000003e-06 2024-02-09T18:03:45.370 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.370 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.370 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:03:45.370 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.554419+0000", 2024-02-09T18:03:45.370 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2516e-05 2024-02-09T18:03:45.370 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.370 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.370 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:03:45.370 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.554428+0000", 2024-02-09T18:03:45.370 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7050000000000005e-06 2024-02-09T18:03:45.370 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.370 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.371 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:03:45.371 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.554484+0000", 2024-02-09T18:03:45.371 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6069999999999997e-05 2024-02-09T18:03:45.371 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.371 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.371 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:03:45.371 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.554509+0000", 2024-02-09T18:03:45.371 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4967999999999999e-05 2024-02-09T18:03:45.371 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.371 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.371 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:03:45.371 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:18.008542+0000", 2024-02-09T18:03:45.371 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.45403305500000002 2024-02-09T18:03:45.371 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.371 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.372 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:03:45.372 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:18.008571+0000", 2024-02-09T18:03:45.372 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.898e-05 2024-02-09T18:03:45.372 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:03:45.372 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:03:45.372 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:03:45.372 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.372 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.372 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.26563.0:204373 9.b e946/943 9:d1b4954d:::benchmark_data_smithi179_161915_object69491:head v 946'12853, mlcod=945'12849)", 2024-02-09T18:03:45.372 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:17.570961+0000", 2024-02-09T18:03:45.372 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 27.676179768000001, 2024-02-09T18:03:45.372 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.43764235699999998, 2024-02-09T18:03:45.372 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:03:45.372 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:03:45.373 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:03:45.373 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.373 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:03:45.373 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.570961+0000", 2024-02-09T18:03:45.373 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:03:45.373 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.373 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.373 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:03:45.373 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.570961+0000", 2024-02-09T18:03:45.373 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:03:45.373 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.373 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.373 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:03:45.373 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.570962+0000", 2024-02-09T18:03:45.373 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.111e-06 2024-02-09T18:03:45.374 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.374 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.374 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:03:45.374 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.570970+0000", 2024-02-09T18:03:45.374 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.9109999999999993e-06 2024-02-09T18:03:45.374 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.374 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.374 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:03:45.374 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.570992+0000", 2024-02-09T18:03:45.374 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2815000000000001e-05 2024-02-09T18:03:45.374 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.374 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.374 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:03:45.374 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.571001+0000", 2024-02-09T18:03:45.374 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1419999999999993e-06 2024-02-09T18:03:45.375 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.375 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.375 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:03:45.375 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.653730+0000", 2024-02-09T18:03:45.375 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.082729840999999998 2024-02-09T18:03:45.375 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.375 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.375 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:03:45.375 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.653746+0000", 2024-02-09T18:03:45.375 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5817000000000001e-05 2024-02-09T18:03:45.375 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.375 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.375 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:03:45.376 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:18.008573+0000", 2024-02-09T18:03:45.376 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.35482643699999999 2024-02-09T18:03:45.376 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.376 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.376 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:03:45.376 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:18.008603+0000", 2024-02-09T18:03:45.376 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0283000000000001e-05 2024-02-09T18:03:45.376 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:03:45.376 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:03:45.376 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:03:45.376 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.376 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.376 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.26563.0:204384 9.e e946/943 9:7332cb43:::benchmark_data_smithi179_161915_object69502:head v 946'12818, mlcod=945'12813)", 2024-02-09T18:03:45.376 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:17.615161+0000", 2024-02-09T18:03:45.376 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 27.631979464, 2024-02-09T18:03:45.377 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.40307902800000001, 2024-02-09T18:03:45.377 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:03:45.377 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:03:45.377 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:03:45.377 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.377 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:03:45.377 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.615161+0000", 2024-02-09T18:03:45.377 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:03:45.377 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.377 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.377 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:03:45.378 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.615161+0000", 2024-02-09T18:03:45.378 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:03:45.378 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.378 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.378 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:03:45.378 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.615162+0000", 2024-02-09T18:03:45.378 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.302e-06 2024-02-09T18:03:45.378 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.378 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.378 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:03:45.378 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.615172+0000", 2024-02-09T18:03:45.378 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5599999999999999e-06 2024-02-09T18:03:45.378 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.378 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.378 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:03:45.379 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.615191+0000", 2024-02-09T18:03:45.379 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9538000000000002e-05 2024-02-09T18:03:45.379 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.379 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.379 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:03:45.379 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.615200+0000", 2024-02-09T18:03:45.379 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7029999999999995e-06 2024-02-09T18:03:45.379 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.379 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.379 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:03:45.379 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.653688+0000", 2024-02-09T18:03:45.379 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.038487747000000003 2024-02-09T18:03:45.379 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.379 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.380 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:03:45.380 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.653709+0000", 2024-02-09T18:03:45.380 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0843e-05 2024-02-09T18:03:45.380 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.380 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.380 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:03:45.380 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:18.018211+0000", 2024-02-09T18:03:45.380 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.36450210300000002 2024-02-09T18:03:45.380 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:03:45.380 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:03:45.380 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:03:45.380 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:18.018240+0000", 2024-02-09T18:03:45.380 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.9232000000000001e-05 2024-02-09T18:03:45.380 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:03:45.381 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:03:45.381 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:03:45.381 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:03:45.381 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:03:45.381 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T18:03:45.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:45 smithi179 ceph-mon[43975]: osdmap e959: 12 total, 12 up, 11 in 2024-02-09T18:03:45.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:45 smithi179 ceph-mon[43975]: 9.5 scrub ok 2024-02-09T18:03:45.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:45 smithi099 ceph-mon[39420]: osdmap e959: 12 total, 12 up, 11 in 2024-02-09T18:03:45.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:45 smithi099 ceph-mon[39420]: 9.5 scrub ok 2024-02-09T18:03:45.998 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.11 dump_ops_in_flight 2024-02-09T18:03:46.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:45 smithi162 ceph-mon[44304]: osdmap e959: 12 total, 12 up, 11 in 2024-02-09T18:03:46.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:45 smithi162 ceph-mon[44304]: 9.5 scrub ok 2024-02-09T18:03:46.305 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T18:03:46.509 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:03:46.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:46.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:03:46.614 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:03:46.642 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:46.784 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:46 smithi179 ceph-mon[43975]: pgmap v2369: 47 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 44 active+clean; 3.1 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 661 op/s; 3965/150483 objects misplaced (2.635%); 29 MiB/s, 466 objects/s recovering 2024-02-09T18:03:46.784 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:46 smithi179 ceph-mon[43975]: osdmap e960: 12 total, 12 up, 11 in 2024-02-09T18:03:46.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:46 smithi099 ceph-mon[39420]: pgmap v2369: 47 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 44 active+clean; 3.1 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 661 op/s; 3965/150483 objects misplaced (2.635%); 29 MiB/s, 466 objects/s recovering 2024-02-09T18:03:46.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:46 smithi099 ceph-mon[39420]: osdmap e960: 12 total, 12 up, 11 in 2024-02-09T18:03:47.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:46 smithi162 ceph-mon[44304]: pgmap v2369: 47 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 44 active+clean; 3.1 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 661 op/s; 3965/150483 objects misplaced (2.635%); 29 MiB/s, 466 objects/s recovering 2024-02-09T18:03:47.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:46 smithi162 ceph-mon[44304]: osdmap e960: 12 total, 12 up, 11 in 2024-02-09T18:03:47.198 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:03:47.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:47.201 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:03:47.302 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T18:03:47.329 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:47.577 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T18:03:47.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:47.580 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T18:03:47.681 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T18:03:47.708 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:47.727 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T18:03:47.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:47.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T18:03:47.791 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:47 smithi179 ceph-mon[43975]: 1.3 scrub starts 2024-02-09T18:03:47.791 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:47 smithi179 ceph-mon[43975]: osdmap e961: 12 total, 12 up, 11 in 2024-02-09T18:03:47.791 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:47 smithi179 ceph-mon[43975]: 1.3 scrub ok 2024-02-09T18:03:47.833 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:03:47.864 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:47.899 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:03:47.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:47.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:03:48.005 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T18:03:48.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:47 smithi162 ceph-mon[44304]: 1.3 scrub starts 2024-02-09T18:03:48.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:47 smithi162 ceph-mon[44304]: osdmap e961: 12 total, 12 up, 11 in 2024-02-09T18:03:48.058 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:47 smithi162 ceph-mon[44304]: 1.3 scrub ok 2024-02-09T18:03:48.074 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:48.109 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T18:03:48.119 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:48.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T18:03:48.145 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:47 smithi099 ceph-mon[39420]: 1.3 scrub starts 2024-02-09T18:03:48.145 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:47 smithi099 ceph-mon[39420]: osdmap e961: 12 total, 12 up, 11 in 2024-02-09T18:03:48.145 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:47 smithi099 ceph-mon[39420]: 1.3 scrub ok 2024-02-09T18:03:48.178 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T18:03:48.178 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-02-09T18:03:48.178 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 0 2024-02-09T18:03:48.178 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T18:03:48.220 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:03:48.265 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:48.815 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T18:03:48.846 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T18:03:48.887 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T18:03:48.923 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T18:03:48.947 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T18:03:49.016 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T18:03:49.016 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T18:03:49.017 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:48 smithi099 ceph-mon[39420]: pgmap v2372: 47 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 2 peering, 43 active+clean; 2.9 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 724 op/s; 2878/141789 objects misplaced (2.030%); 13 MiB/s, 206 objects/s recovering 2024-02-09T18:03:49.017 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:48 smithi099 ceph-mon[39420]: osdmap e962: 12 total, 12 up, 11 in 2024-02-09T18:03:49.017 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T18:03:49.035 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T18:03:49.040 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:03:49.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:49.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:03:49.055 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T18:03:49.076 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T18:03:49.097 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T18:03:49.098 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T18:03:49.098 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T18:03:49.098 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T18:03:49.099 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T18:03:49.099 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T18:03:49.099 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T18:03:49.099 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T18:03:49.099 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T18:03:49.099 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T18:03:49.099 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T18:03:49.099 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T18:03:49.099 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T18:03:49.144 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T18:03:49.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:48 smithi179 ceph-mon[43975]: pgmap v2372: 47 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 2 peering, 43 active+clean; 2.9 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 724 op/s; 2878/141789 objects misplaced (2.030%); 13 MiB/s, 206 objects/s recovering 2024-02-09T18:03:49.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:48 smithi179 ceph-mon[43975]: osdmap e962: 12 total, 12 up, 11 in 2024-02-09T18:03:49.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:48 smithi162 ceph-mon[44304]: pgmap v2372: 47 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 2 peering, 43 active+clean; 2.9 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 724 op/s; 2878/141789 objects misplaced (2.030%); 13 MiB/s, 206 objects/s recovering 2024-02-09T18:03:49.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:48 smithi162 ceph-mon[44304]: osdmap e962: 12 total, 12 up, 11 in 2024-02-09T18:03:49.193 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:49.223 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T18:03:49.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:49.228 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T18:03:49.265 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.11 dump_blocked_ops 2024-02-09T18:03:49.330 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:03:49.375 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:49.406 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:03:49.410 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:49.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:03:49.511 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T18:03:49.555 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:49.586 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T18:03:49.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:49.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T18:03:49.694 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T18:03:49.708 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T18:03:49.756 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:49.791 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is set 2024-02-09T18:03:49.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:49 smithi162 ceph-mon[44304]: 9.3 scrub starts 2024-02-09T18:03:49.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:49 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1623151852' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:03:49.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:49 smithi162 ceph-mon[44304]: 9.3 scrub ok 2024-02-09T18:03:49.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:49 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3028693220' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T18:03:49.971 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T18:03:49.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:49.977 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T18:03:50.017 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:49 smithi179 ceph-mon[43975]: 9.3 scrub starts 2024-02-09T18:03:50.018 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:49 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1623151852' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:03:50.018 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:49 smithi179 ceph-mon[43975]: 9.3 scrub ok 2024-02-09T18:03:50.018 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:49 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3028693220' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T18:03:50.079 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:03:50.111 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:50.141 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:03:50.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:50.144 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:03:50.207 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:49 smithi099 ceph-mon[39420]: 9.3 scrub starts 2024-02-09T18:03:50.207 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:49 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1623151852' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:03:50.207 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:49 smithi099 ceph-mon[39420]: 9.3 scrub ok 2024-02-09T18:03:50.207 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:49 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3028693220' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T18:03:50.245 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:03:50.295 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:50.343 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:03:50.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:50.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:03:50.449 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:03:50.488 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:50.685 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:03:50.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:50.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:03:50.793 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:03:50.829 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:50.861 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:03:50.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:50.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:03:50.973 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:03:51.022 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:51.055 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:50 smithi179 ceph-mon[43975]: pgmap v2374: 47 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 2 peering, 43 active+clean; 2.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 776 op/s; 2878/139752 objects misplaced (2.059%); 11 MiB/s, 179 objects/s recovering 2024-02-09T18:03:51.055 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:50 smithi179 ceph-mon[43975]: 1.a deep-scrub starts 2024-02-09T18:03:51.055 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:50 smithi179 ceph-mon[43975]: 1.a deep-scrub ok 2024-02-09T18:03:51.056 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:50 smithi179 ceph-mon[43975]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:03:51.056 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:50 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3028693220' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T18:03:51.056 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:50 smithi179 ceph-mon[43975]: osdmap e963: 12 total, 12 up, 11 in 2024-02-09T18:03:51.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:50 smithi162 ceph-mon[44304]: pgmap v2374: 47 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 2 peering, 43 active+clean; 2.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 776 op/s; 2878/139752 objects misplaced (2.059%); 11 MiB/s, 179 objects/s recovering 2024-02-09T18:03:51.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:50 smithi162 ceph-mon[44304]: 1.a deep-scrub starts 2024-02-09T18:03:51.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:50 smithi162 ceph-mon[44304]: 1.a deep-scrub ok 2024-02-09T18:03:51.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:50 smithi162 ceph-mon[44304]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:03:51.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:50 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3028693220' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T18:03:51.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:50 smithi162 ceph-mon[44304]: osdmap e963: 12 total, 12 up, 11 in 2024-02-09T18:03:51.219 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:50 smithi099 ceph-mon[39420]: pgmap v2374: 47 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 2 peering, 43 active+clean; 2.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 776 op/s; 2878/139752 objects misplaced (2.059%); 11 MiB/s, 179 objects/s recovering 2024-02-09T18:03:51.219 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:50 smithi099 ceph-mon[39420]: 1.a deep-scrub starts 2024-02-09T18:03:51.219 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:50 smithi099 ceph-mon[39420]: 1.a deep-scrub ok 2024-02-09T18:03:51.219 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:50 smithi099 ceph-mon[39420]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:03:51.219 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:50 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3028693220' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T18:03:51.220 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:50 smithi099 ceph-mon[39420]: osdmap e963: 12 total, 12 up, 11 in 2024-02-09T18:03:51.232 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:03:51.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:51.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:03:51.340 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:03:51.383 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:51.412 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:03:51.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:51.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:03:51.518 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:03:51.586 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:51.620 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:03:51.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:51.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:03:51.728 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:03:51.771 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:51.787 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T18:03:51.787 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T18:03:51.809 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:03:51.814 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:51 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:03:51.814 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:51 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:03:51.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:51.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:03:51.916 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T18:03:51.953 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:51.987 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T18:03:51.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:51.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T18:03:51.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:51 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:03:51.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:51 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:03:52.096 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:03:52.159 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:52.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:51 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:03:52.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:51 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:03:52.201 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:03:52.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:52.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:03:52.306 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:03:52.351 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:52.383 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:03:52.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:52.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:03:52.488 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:03:52.536 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:52.567 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:03:52.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:52.574 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:03:52.616 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.11 dump_historic_ops 2024-02-09T18:03:52.675 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:03:52.707 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:52.720 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-09T18:03:52.736 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:03:52.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:52.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:03:52.843 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T18:03:52.893 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:52.915 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T18:03:52.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:52.920 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T18:03:52.958 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 5, 6, 7, 8, 9, 11, 0, 2, 4] out_osds: [10] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 7, 9, 10, 11, 8, 3] 2024-02-09T18:03:52.958 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T18:03:52.958 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-02-09T18:03:52.958 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-09T18:03:52.958 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-09T18:03:52.959 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-02-09T18:03:52.993 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:52 smithi099 ceph-mon[39420]: pgmap v2376: 47 pgs: 1 active+remapped+backfill_toofull, 1 peering, 45 active+clean; 2.7 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 955 op/s; 2685/134454 objects misplaced (1.997%); 9.2 MiB/s, 147 objects/s recovering 2024-02-09T18:03:52.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:52 smithi099 ceph-mon[39420]: osdmap e964: 12 total, 12 up, 11 in 2024-02-09T18:03:53.013 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T18:03:53.022 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:03:53.079 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:53.091 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:03:53.108 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:03:53.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:53.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:03:53.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:52 smithi179 ceph-mon[43975]: pgmap v2376: 47 pgs: 1 active+remapped+backfill_toofull, 1 peering, 45 active+clean; 2.7 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 955 op/s; 2685/134454 objects misplaced (1.997%); 9.2 MiB/s, 147 objects/s recovering 2024-02-09T18:03:53.174 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:52 smithi179 ceph-mon[43975]: osdmap e964: 12 total, 12 up, 11 in 2024-02-09T18:03:53.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:52 smithi162 ceph-mon[44304]: pgmap v2376: 47 pgs: 1 active+remapped+backfill_toofull, 1 peering, 45 active+clean; 2.7 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 955 op/s; 2685/134454 objects misplaced (1.997%); 9.2 MiB/s, 147 objects/s recovering 2024-02-09T18:03:53.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:52 smithi162 ceph-mon[44304]: osdmap e964: 12 total, 12 up, 11 in 2024-02-09T18:03:53.214 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:03:53.258 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T18:03:53.259 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:53.289 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:03:53.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:53.295 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:03:53.372 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T18:03:53.395 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:03:53.468 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:53.504 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:03:53.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:53.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:03:53.612 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T18:03:53.671 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:53.875 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T18:03:53.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:53.877 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T18:03:53.978 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T18:03:54.036 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:54.996 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:03:55.090 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:54 smithi162 ceph-mon[44304]: pgmap v2378: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 2.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 661 op/s; 2685/129882 objects misplaced (2.067%) 2024-02-09T18:03:55.090 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:54 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:03:55.090 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:54 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:03:55.090 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:54 smithi162 ceph-mon[44304]: osdmap e965: 12 total, 12 up, 11 in 2024-02-09T18:03:55.116 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T18:03:55.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:55.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T18:03:55.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:54 smithi179 ceph-mon[43975]: pgmap v2378: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 2.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 661 op/s; 2685/129882 objects misplaced (2.067%) 2024-02-09T18:03:55.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:54 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:03:55.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:54 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:03:55.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:54 smithi179 ceph-mon[43975]: osdmap e965: 12 total, 12 up, 11 in 2024-02-09T18:03:55.225 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T18:03:55.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:54 smithi099 ceph-mon[39420]: pgmap v2378: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 2.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 661 op/s; 2685/129882 objects misplaced (2.067%) 2024-02-09T18:03:55.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:54 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:03:55.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:54 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:03:55.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:54 smithi099 ceph-mon[39420]: osdmap e965: 12 total, 12 up, 11 in 2024-02-09T18:03:55.272 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:55.493 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T18:03:55.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:55.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T18:03:55.601 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:03:55.647 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:55.935 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T18:03:55.935 INFO:teuthology.orchestra.run.smithi162.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-02-09T18:03:55.935 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T18:03:56.196 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:03:56.199 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:56.199 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:03:56.300 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T18:03:56.345 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:56.705 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T18:03:56.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:56.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T18:03:56.814 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T18:03:56.857 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:56.858 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is set 2024-02-09T18:03:56.900 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T18:03:56.906 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:56.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T18:03:57.008 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T18:03:57.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:56 smithi162 ceph-mon[44304]: pgmap v2380: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 2.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 548 op/s; 2685/129882 objects misplaced (2.067%) 2024-02-09T18:03:57.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:56 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/217682458' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T18:03:57.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:56 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T18:03:57.049 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:57.082 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T18:03:57.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:57.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T18:03:57.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:56 smithi179 ceph-mon[43975]: pgmap v2380: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 2.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 548 op/s; 2685/129882 objects misplaced (2.067%) 2024-02-09T18:03:57.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:56 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/217682458' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T18:03:57.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:56 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T18:03:57.188 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:03:57.228 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:57.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:56 smithi099 ceph-mon[39420]: pgmap v2380: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 2.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 548 op/s; 2685/129882 objects misplaced (2.067%) 2024-02-09T18:03:57.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:56 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/217682458' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T18:03:57.246 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:56 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T18:03:58.175 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:57 smithi179 ceph-mon[43975]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:03:58.175 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:57 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T18:03:58.175 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:57 smithi179 ceph-mon[43975]: osdmap e966: 12 total, 12 up, 11 in 2024-02-09T18:03:58.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:57 smithi162 ceph-mon[44304]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:03:58.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:57 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T18:03:58.180 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:57 smithi162 ceph-mon[44304]: osdmap e966: 12 total, 12 up, 11 in 2024-02-09T18:03:58.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:57 smithi099 ceph-mon[39420]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:03:58.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:57 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T18:03:58.247 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:57 smithi099 ceph-mon[39420]: osdmap e966: 12 total, 12 up, 11 in 2024-02-09T18:03:58.585 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:03:58.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:03:58.589 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:03:58.690 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:03:58.747 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:03:58.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:58 smithi099 ceph-mon[39420]: pgmap v2382: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 2.4 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 827 op/s; 2374/119556 objects misplaced (1.986%) 2024-02-09T18:03:58.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:03:58 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3449124735' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:03:58.998 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-02-09T18:03:58.998 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T18:03:59.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:58 smithi179 ceph-mon[43975]: pgmap v2382: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 2.4 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 827 op/s; 2374/119556 objects misplaced (1.986%) 2024-02-09T18:03:59.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:03:58 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3449124735' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:03:59.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:58 smithi162 ceph-mon[44304]: pgmap v2382: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 2.4 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 827 op/s; 2374/119556 objects misplaced (1.986%) 2024-02-09T18:03:59.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:03:58 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3449124735' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:03:59.881 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.8 dump_ops_in_flight 2024-02-09T18:04:00.277 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T18:04:00.430 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:04:00.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:00.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:04:00.536 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T18:04:00.556 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T18:04:00.556 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T18:04:00.557 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T18:04:00.557 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T18:04:00.557 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T18:04:00.557 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T18:04:00.557 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T18:04:00.557 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T18:04:00.557 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T18:04:00.557 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T18:04:00.557 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T18:04:00.557 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T18:04:00.557 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T18:04:00.557 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T18:04:00.558 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T18:04:00.558 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T18:04:00.558 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T18:04:00.558 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T18:04:00.558 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T18:04:00.558 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T18:04:00.558 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T18:04:00.558 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T18:04:00.558 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T18:04:00.558 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T18:04:00.582 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:00.618 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T18:04:00.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:00.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T18:04:00.723 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T18:04:00.777 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:00.821 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T18:04:00.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:00.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T18:04:00.928 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T18:04:00.968 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:01.001 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T18:04:01.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:01.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T18:04:01.106 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:04:01.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:00 smithi179 ceph-mon[43975]: pgmap v2383: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 2.4 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 800 op/s; 2374/116355 objects misplaced (2.040%) 2024-02-09T18:04:01.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:00 smithi162 ceph-mon[44304]: pgmap v2383: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 2.4 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 800 op/s; 2374/116355 objects misplaced (2.040%) 2024-02-09T18:04:01.178 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:01.230 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:04:01.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:01.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:04:01.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:00 smithi099 ceph-mon[39420]: pgmap v2383: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 2.4 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 800 op/s; 2374/116355 objects misplaced (2.040%) 2024-02-09T18:04:01.338 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T18:04:01.399 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:01.729 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 5, 6, 7, 8, 9, 11, 0, 2, 4] out_osds: [10] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 7, 9, 10, 11, 8, 3] 2024-02-09T18:04:01.729 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T18:04:01.729 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-02-09T18:04:01.729 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-09T18:04:01.729 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-09T18:04:01.730 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.3 config set filestore_inject_stall 3 2024-02-09T18:04:01.862 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T18:04:01.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:01.868 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T18:04:01.969 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:04:02.006 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:02.039 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:04:02.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:02.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:04:02.141 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:04:02.149 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:04:02.209 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:02.386 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset noscrub 2024-02-09T18:04:02.423 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:04:02.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:02.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:04:02.526 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T18:04:02.527 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-02-09T18:04:02.527 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:02.527 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:236281 9.b 9:df0141ba:::benchmark_data_smithi179_161915_object101399:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e966)", 2024-02-09T18:04:02.527 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:04:02.502963+0000", 2024-02-09T18:04:02.527 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.022232501000000002, 2024-02-09T18:04:02.527 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.022258764, 2024-02-09T18:04:02.527 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:04:02.527 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "waiting for sub ops", 2024-02-09T18:04:02.527 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:04:02.527 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:04:02.527 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:04:02.527 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 236281 2024-02-09T18:04:02.527 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:02.527 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:04:02.528 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:02.528 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:04:02.528 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:02.502963+0000", 2024-02-09T18:04:02.528 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:04:02.528 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:02.528 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:02.528 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:04:02.528 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:02.502963+0000", 2024-02-09T18:04:02.528 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:04:02.528 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:02.528 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:02.528 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:04:02.528 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:02.502965+0000", 2024-02-09T18:04:02.528 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.497e-06 2024-02-09T18:04:02.529 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:02.529 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:02.529 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:04:02.529 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:02.502970+0000", 2024-02-09T18:04:02.529 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7520000000000004e-06 2024-02-09T18:04:02.529 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:02.529 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:02.529 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:04:02.529 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:02.502997+0000", 2024-02-09T18:04:02.529 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.756e-05 2024-02-09T18:04:02.529 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:02.529 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:02.529 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:04:02.529 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:02.503007+0000", 2024-02-09T18:04:02.529 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7510000000000003e-06 2024-02-09T18:04:02.530 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:02.530 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:02.530 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:04:02.530 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:02.510832+0000", 2024-02-09T18:04:02.530 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0078247170000000001 2024-02-09T18:04:02.530 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:02.530 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:02.530 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:04:02.530 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:02.510916+0000", 2024-02-09T18:04:02.530 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4269000000000002e-05 2024-02-09T18:04:02.530 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:02.530 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:02.531 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5", 2024-02-09T18:04:02.531 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:02.510991+0000", 2024-02-09T18:04:02.531 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4795999999999993e-05 2024-02-09T18:04:02.531 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:02.531 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:02.531 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:04:02.531 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:02.519624+0000", 2024-02-09T18:04:02.531 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0086334700000000007 2024-02-09T18:04:02.531 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:02.531 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:02.531 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:02.531 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:02.522721+0000", 2024-02-09T18:04:02.531 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0030972030000000002 2024-02-09T18:04:02.531 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:04:02.532 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:04:02.532 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:04:02.532 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:02.532 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:02.532 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:236284 9.b 9:d6e573d7:::benchmark_data_smithi179_161915_object101402:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e966)", 2024-02-09T18:04:02.532 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:04:02.510502+0000", 2024-02-09T18:04:02.532 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.014693487, 2024-02-09T18:04:02.532 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014794450000000001, 2024-02-09T18:04:02.532 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:04:02.532 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "waiting for sub ops", 2024-02-09T18:04:02.532 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:04:02.532 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:04:02.532 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:04:02.532 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 236284 2024-02-09T18:04:02.533 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:02.533 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:04:02.533 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:02.533 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:04:02.533 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:02.510502+0000", 2024-02-09T18:04:02.533 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:04:02.533 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:02.533 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:02.533 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:04:02.533 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:02.510502+0000", 2024-02-09T18:04:02.533 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:04:02.533 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:02.533 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:02.533 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:04:02.533 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:02.510504+0000", 2024-02-09T18:04:02.534 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1319999999999999e-06 2024-02-09T18:04:02.534 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:02.534 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:02.534 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:04:02.534 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:02.510509+0000", 2024-02-09T18:04:02.534 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2569999999999998e-06 2024-02-09T18:04:02.534 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:02.534 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:02.534 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:04:02.534 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:02.510525+0000", 2024-02-09T18:04:02.534 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6473999999999998e-05 2024-02-09T18:04:02.534 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:02.534 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:02.534 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:04:02.535 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:02.510534+0000", 2024-02-09T18:04:02.535 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.0690000000000008e-06 2024-02-09T18:04:02.535 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:02.535 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:02.535 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:04:02.535 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:02.515128+0000", 2024-02-09T18:04:02.535 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0045933459999999999 2024-02-09T18:04:02.535 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:02.535 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:02.535 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:04:02.535 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:02.515212+0000", 2024-02-09T18:04:02.535 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4126000000000001e-05 2024-02-09T18:04:02.535 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:02.535 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:02.535 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5", 2024-02-09T18:04:02.536 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:02.515280+0000", 2024-02-09T18:04:02.536 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8701999999999994e-05 2024-02-09T18:04:02.536 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:02.536 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:02.536 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:04:02.536 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:02.519742+0000", 2024-02-09T18:04:02.536 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0044610379999999996 2024-02-09T18:04:02.536 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:02.536 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:02.536 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:02.536 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:02.522816+0000", 2024-02-09T18:04:02.536 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.003074635 2024-02-09T18:04:02.536 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:04:02.536 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:04:02.536 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:04:02.537 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:04:02.537 INFO:teuthology.orchestra.run.smithi179.stdout: ], 2024-02-09T18:04:02.537 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 2 2024-02-09T18:04:02.537 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T18:04:02.537 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:04:02.583 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:02.796 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:04:02.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:02.799 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:04:02.901 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:04:02.963 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:02.990 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:04:02.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:02.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:04:03.099 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T18:04:03.132 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:02 smithi179 ceph-mon[43975]: pgmap v2384: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 2.2 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 905 op/s; 2094/108132 objects misplaced (1.937%) 2024-02-09T18:04:03.135 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:03.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:02 smithi162 ceph-mon[44304]: pgmap v2384: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 2.2 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 905 op/s; 2094/108132 objects misplaced (1.937%) 2024-02-09T18:04:03.184 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:02 smithi099 ceph-mon[39420]: pgmap v2384: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 2.2 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 905 op/s; 2094/108132 objects misplaced (1.937%) 2024-02-09T18:04:03.296 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T18:04:03.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:03.302 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T18:04:03.322 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.8 dump_blocked_ops 2024-02-09T18:04:03.403 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:04:03.431 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T18:04:03.441 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:03.651 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:04:03.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:03.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:04:03.726 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T18:04:03.757 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:04:03.811 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:03.899 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:04:03.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:03.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:04:04.007 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:04:04.039 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:04.072 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:04:04.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:04.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:04:04.111 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:04:04.177 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:04:04.222 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:04.255 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:04:04.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:04.260 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:04:04.362 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:04:04.406 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:04.752 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T18:04:04.752 INFO:teuthology.orchestra.run.smithi099.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-02-09T18:04:04.752 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T18:04:05.015 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:04 smithi179 ceph-mon[43975]: pgmap v2385: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 2.1 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 939 op/s; 2094/103020 objects misplaced (2.033%) 2024-02-09T18:04:05.185 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:04 smithi162 ceph-mon[44304]: pgmap v2385: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 2.1 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 939 op/s; 2094/103020 objects misplaced (2.033%) 2024-02-09T18:04:05.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:04 smithi099 ceph-mon[39420]: pgmap v2385: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 2.1 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 939 op/s; 2094/103020 objects misplaced (2.033%) 2024-02-09T18:04:05.774 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T18:04:05.774 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-02-09T18:04:05.774 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-02-09T18:04:05.774 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-02-09T18:04:05.774 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T18:04:06.166 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:04:06.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:06.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:04:06.277 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:04:06.332 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:06.595 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:04:06.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:06.602 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:04:06.621 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.8 dump_historic_ops 2024-02-09T18:04:06.703 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:04:06.745 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:06.817 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:04:06.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:06.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:04:06.922 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T18:04:06.958 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:06.992 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T18:04:06.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:06 smithi099 ceph-mon[39420]: pgmap v2386: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 2.1 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 894 op/s; 2094/103020 objects misplaced (2.033%) 2024-02-09T18:04:06.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:06.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T18:04:07.041 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T18:04:07.098 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T18:04:07.161 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:06 smithi162 ceph-mon[44304]: pgmap v2386: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 2.1 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 894 op/s; 2094/103020 objects misplaced (2.033%) 2024-02-09T18:04:07.171 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:07.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:06 smithi179 ceph-mon[43975]: pgmap v2386: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 2.1 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 894 op/s; 2094/103020 objects misplaced (2.033%) 2024-02-09T18:04:07.200 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T18:04:07.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:07.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T18:04:07.310 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:04:07.357 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:07.387 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:04:07.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:07.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:04:07.495 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:04:07.526 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:07.562 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:04:07.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:07.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:04:07.674 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:04:07.713 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:04:07.720 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:07.747 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:04:07.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:07.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:04:07.851 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:04:07.885 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:07.911 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:04:07.918 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:07.919 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:04:08.020 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T18:04:08.066 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:08.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:08 smithi179 ceph-mon[43975]: pgmap v2387: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 1.9 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 893 op/s; 1829/93060 objects misplaced (1.965%) 2024-02-09T18:04:08.429 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:08 smithi162 ceph-mon[44304]: pgmap v2387: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 1.9 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 893 op/s; 1829/93060 objects misplaced (1.965%) 2024-02-09T18:04:08.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:08 smithi099 ceph-mon[39420]: pgmap v2387: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 1.9 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 893 op/s; 1829/93060 objects misplaced (1.965%) 2024-02-09T18:04:08.518 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T18:04:08.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:08.521 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T18:04:08.623 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T18:04:08.681 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:08.722 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T18:04:08.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:08.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T18:04:08.827 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T18:04:08.881 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:08.907 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T18:04:08.918 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:08.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T18:04:09.020 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:04:09.074 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:09.235 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T18:04:09.235 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-02-09T18:04:09.235 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-02-09T18:04:09.235 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-02-09T18:04:09.235 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.235 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:236077 8.b 8:d02b57f2:::benchmark_data_smithi179_148205_object113005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e863)", 2024-02-09T18:04:09.235 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:59:26.606600+0000", 2024-02-09T18:04:09.235 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 282.62367169300001, 2024-02-09T18:04:09.235 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6539503770000001, 2024-02-09T18:04:09.236 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:04:09.236 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:04:09.236 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:04:09.236 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:04:09.236 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:04:09.236 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 236077 2024-02-09T18:04:09.236 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.236 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:04:09.236 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.236 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:04:09.237 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.606600+0000", 2024-02-09T18:04:09.237 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:04:09.237 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.237 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.237 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:04:09.237 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.606600+0000", 2024-02-09T18:04:09.237 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:04:09.237 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.237 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.237 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:04:09.237 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.606602+0000", 2024-02-09T18:04:09.237 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4389999999999999e-06 2024-02-09T18:04:09.237 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.237 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.238 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:04:09.238 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.606609+0000", 2024-02-09T18:04:09.238 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7139999999999996e-06 2024-02-09T18:04:09.238 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.238 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.238 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:04:09.238 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.606625+0000", 2024-02-09T18:04:09.238 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6365999999999999e-05 2024-02-09T18:04:09.238 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.238 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.238 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:04:09.239 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.606634+0000", 2024-02-09T18:04:09.239 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.3370000000000006e-06 2024-02-09T18:04:09.239 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.239 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.239 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:04:09.239 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.606693+0000", 2024-02-09T18:04:09.239 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9222000000000003e-05 2024-02-09T18:04:09.239 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.239 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.239 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:04:09.239 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.606703+0000", 2024-02-09T18:04:09.240 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.8060000000000006e-06 2024-02-09T18:04:09.240 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.240 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.240 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:04:09.240 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.006088+0000", 2024-02-09T18:04:09.240 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.399384348 2024-02-09T18:04:09.240 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.240 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.240 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:04:09.240 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.006102+0000", 2024-02-09T18:04:09.240 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4442e-05 2024-02-09T18:04:09.240 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.240 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.241 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:04:09.241 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.238871+0000", 2024-02-09T18:04:09.241 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2327688619999999 2024-02-09T18:04:09.241 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.241 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.241 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:04:09.241 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.239029+0000", 2024-02-09T18:04:09.241 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000158049 2024-02-09T18:04:09.241 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.241 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.241 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4,11", 2024-02-09T18:04:09.241 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.239206+0000", 2024-02-09T18:04:09.241 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00017735599999999999 2024-02-09T18:04:09.242 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.242 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.242 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:09.242 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.244222+0000", 2024-02-09T18:04:09.242 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00501565 2024-02-09T18:04:09.242 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.242 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.242 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:09.242 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.244302+0000", 2024-02-09T18:04:09.243 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.0358000000000005e-05 2024-02-09T18:04:09.243 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.243 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.243 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:04:09.243 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.249476+0000", 2024-02-09T18:04:09.243 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0051733550000000001 2024-02-09T18:04:09.243 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.243 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.243 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:09.243 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.260367+0000", 2024-02-09T18:04:09.243 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010891171999999999 2024-02-09T18:04:09.243 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.243 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.243 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:04:09.244 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.260433+0000", 2024-02-09T18:04:09.244 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6156999999999997e-05 2024-02-09T18:04:09.244 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.244 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.244 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:04:09.244 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.260551+0000", 2024-02-09T18:04:09.244 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000117744 2024-02-09T18:04:09.244 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:04:09.244 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:04:09.244 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:04:09.244 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.244 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.244 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:237144 8.b 8:d410fa72:::benchmark_data_smithi179_148205_object114072:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e867)", 2024-02-09T18:04:09.245 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:59:30.150092+0000", 2024-02-09T18:04:09.245 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 279.08018033600001, 2024-02-09T18:04:09.245 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.31223010400000001, 2024-02-09T18:04:09.245 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:04:09.245 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:04:09.245 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:04:09.245 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:04:09.245 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:04:09.245 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 237144 2024-02-09T18:04:09.245 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.245 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:04:09.245 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.245 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:04:09.245 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.150092+0000", 2024-02-09T18:04:09.246 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:04:09.246 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.246 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.246 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:04:09.246 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.150092+0000", 2024-02-09T18:04:09.246 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:04:09.246 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.246 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.246 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:04:09.246 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.150094+0000", 2024-02-09T18:04:09.246 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9230000000000001e-06 2024-02-09T18:04:09.246 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.246 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.247 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:04:09.247 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.150102+0000", 2024-02-09T18:04:09.247 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2060000000000007e-06 2024-02-09T18:04:09.247 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.247 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.247 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:04:09.247 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.150122+0000", 2024-02-09T18:04:09.247 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9930999999999999e-05 2024-02-09T18:04:09.247 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.247 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.247 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:04:09.247 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.150141+0000", 2024-02-09T18:04:09.247 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9142000000000001e-05 2024-02-09T18:04:09.247 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.247 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.248 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:04:09.248 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.150214+0000", 2024-02-09T18:04:09.248 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2846e-05 2024-02-09T18:04:09.248 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.248 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.248 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:04:09.248 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.150364+0000", 2024-02-09T18:04:09.248 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015035600000000001 2024-02-09T18:04:09.248 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.248 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.248 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4,11", 2024-02-09T18:04:09.248 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.151273+0000", 2024-02-09T18:04:09.248 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00090894200000000004 2024-02-09T18:04:09.248 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.249 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.249 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:09.249 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.156327+0000", 2024-02-09T18:04:09.249 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00505376 2024-02-09T18:04:09.249 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.249 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.249 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:04:09.249 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.156343+0000", 2024-02-09T18:04:09.249 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6002e-05 2024-02-09T18:04:09.249 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.249 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.249 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:09.249 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.162202+0000", 2024-02-09T18:04:09.249 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0058588900000000003 2024-02-09T18:04:09.249 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.250 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.250 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:09.250 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.462198+0000", 2024-02-09T18:04:09.250 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.29999659200000001 2024-02-09T18:04:09.250 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.250 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.250 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:04:09.250 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.462246+0000", 2024-02-09T18:04:09.250 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8016999999999998e-05 2024-02-09T18:04:09.250 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.250 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.250 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:04:09.250 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.462322+0000", 2024-02-09T18:04:09.251 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5496999999999999e-05 2024-02-09T18:04:09.251 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:04:09.251 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:04:09.251 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:04:09.251 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.251 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.251 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:202997 9.b 9:dc8fd382:::benchmark_data_smithi179_161915_object68115:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e942)", 2024-02-09T18:04:09.251 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:13.445860+0000", 2024-02-09T18:04:09.251 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 55.784411657, 2024-02-09T18:04:09.251 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.223805365, 2024-02-09T18:04:09.251 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:04:09.252 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:04:09.252 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:04:09.252 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:04:09.252 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:04:09.252 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 202997 2024-02-09T18:04:09.252 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.252 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:04:09.252 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.252 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:04:09.252 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.445860+0000", 2024-02-09T18:04:09.252 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:04:09.252 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.252 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.252 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:04:09.253 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.445860+0000", 2024-02-09T18:04:09.253 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:04:09.253 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.253 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.253 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:04:09.253 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.445862+0000", 2024-02-09T18:04:09.253 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0499999999999999e-06 2024-02-09T18:04:09.253 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.253 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.253 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:04:09.253 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.445869+0000", 2024-02-09T18:04:09.253 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.866e-06 2024-02-09T18:04:09.253 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.253 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.254 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:04:09.254 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.445882+0000", 2024-02-09T18:04:09.254 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2989999999999999e-05 2024-02-09T18:04:09.254 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.254 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.254 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:04:09.254 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.445891+0000", 2024-02-09T18:04:09.254 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.1179999999999998e-06 2024-02-09T18:04:09.254 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.254 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.254 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:04:09.254 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.445947+0000", 2024-02-09T18:04:09.254 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5736999999999998e-05 2024-02-09T18:04:09.254 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.255 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.255 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op must wait for map", 2024-02-09T18:04:09.255 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.445952+0000", 2024-02-09T18:04:09.255 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3510000000000004e-06 2024-02-09T18:04:09.255 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.255 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.255 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:04:09.255 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.453790+0000", 2024-02-09T18:04:09.255 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0078388480000000007 2024-02-09T18:04:09.255 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.255 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.255 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:04:09.255 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.453800+0000", 2024-02-09T18:04:09.255 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5000000000000005e-06 2024-02-09T18:04:09.255 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.256 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.256 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:04:09.256 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:14.441943+0000", 2024-02-09T18:04:09.256 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.98814301900000001 2024-02-09T18:04:09.256 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.256 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.256 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:04:09.256 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:14.441959+0000", 2024-02-09T18:04:09.256 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6325999999999999e-05 2024-02-09T18:04:09.256 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.256 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.256 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:04:09.256 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.652434+0000", 2024-02-09T18:04:09.257 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.210474933 2024-02-09T18:04:09.257 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.257 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.257 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:04:09.257 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.652554+0000", 2024-02-09T18:04:09.257 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000120025 2024-02-09T18:04:09.257 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.257 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.257 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5,10", 2024-02-09T18:04:09.257 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.652642+0000", 2024-02-09T18:04:09.257 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7954999999999994e-05 2024-02-09T18:04:09.257 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.257 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.257 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:09.258 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.658212+0000", 2024-02-09T18:04:09.258 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0055703460000000003 2024-02-09T18:04:09.258 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.258 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.258 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:04:09.258 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.663862+0000", 2024-02-09T18:04:09.258 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0056495490000000002 2024-02-09T18:04:09.258 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.258 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.258 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:09.258 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.669378+0000", 2024-02-09T18:04:09.258 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0055155359999999997 2024-02-09T18:04:09.258 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.258 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.258 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:09.259 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.669475+0000", 2024-02-09T18:04:09.259 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7275000000000006e-05 2024-02-09T18:04:09.259 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.259 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.259 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:04:09.259 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.669544+0000", 2024-02-09T18:04:09.259 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8911000000000005e-05 2024-02-09T18:04:09.259 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.259 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.259 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:04:09.259 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.669666+0000", 2024-02-09T18:04:09.259 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012203 2024-02-09T18:04:09.259 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:04:09.259 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:04:09.260 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:04:09.260 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.260 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.260 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:203024 9.b 9:d448a439:::benchmark_data_smithi179_161915_object68142:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e942)", 2024-02-09T18:04:09.260 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:13.495413+0000", 2024-02-09T18:04:09.260 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 55.734858682999999, 2024-02-09T18:04:09.260 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1746194750000001, 2024-02-09T18:04:09.260 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:04:09.260 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:04:09.260 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:04:09.260 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:04:09.260 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:04:09.260 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 203024 2024-02-09T18:04:09.261 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.261 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:04:09.261 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.261 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:04:09.261 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.495413+0000", 2024-02-09T18:04:09.261 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:04:09.261 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.261 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.261 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:04:09.261 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.495413+0000", 2024-02-09T18:04:09.261 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:04:09.261 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.261 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.262 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:04:09.262 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.495415+0000", 2024-02-09T18:04:09.262 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.333e-06 2024-02-09T18:04:09.262 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.262 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.262 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:04:09.262 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.495421+0000", 2024-02-09T18:04:09.262 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8760000000000002e-06 2024-02-09T18:04:09.262 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.262 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.262 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:04:09.262 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.495434+0000", 2024-02-09T18:04:09.262 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3061000000000001e-05 2024-02-09T18:04:09.262 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.263 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.263 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:04:09.263 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.495441+0000", 2024-02-09T18:04:09.263 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.0639999999999996e-06 2024-02-09T18:04:09.263 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.263 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.263 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:04:09.263 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.495496+0000", 2024-02-09T18:04:09.263 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4954000000000003e-05 2024-02-09T18:04:09.263 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.263 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.263 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:04:09.263 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.495505+0000", 2024-02-09T18:04:09.264 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.4690000000000003e-06 2024-02-09T18:04:09.264 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.264 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.264 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:04:09.264 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:14.442055+0000", 2024-02-09T18:04:09.264 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.94654996499999999 2024-02-09T18:04:09.264 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.264 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.264 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:04:09.264 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:14.442064+0000", 2024-02-09T18:04:09.264 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.3219999999999999e-06 2024-02-09T18:04:09.264 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.264 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.265 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:04:09.265 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.658266+0000", 2024-02-09T18:04:09.265 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2162012870000001 2024-02-09T18:04:09.265 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.265 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.265 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:04:09.265 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.658392+0000", 2024-02-09T18:04:09.265 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012630900000000001 2024-02-09T18:04:09.265 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.265 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.265 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5,10", 2024-02-09T18:04:09.265 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.658528+0000", 2024-02-09T18:04:09.265 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013552200000000001 2024-02-09T18:04:09.266 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.266 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.266 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:04:09.266 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.669681+0000", 2024-02-09T18:04:09.266 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011153758999999999 2024-02-09T18:04:09.266 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.266 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.266 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:09.266 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.669737+0000", 2024-02-09T18:04:09.266 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6150000000000003e-05 2024-02-09T18:04:09.266 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.267 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.267 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:09.267 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.669796+0000", 2024-02-09T18:04:09.267 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8425999999999999e-05 2024-02-09T18:04:09.267 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.267 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.267 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:09.267 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.669860+0000", 2024-02-09T18:04:09.267 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3965000000000002e-05 2024-02-09T18:04:09.267 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.267 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.267 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:04:09.267 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.669918+0000", 2024-02-09T18:04:09.268 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8461000000000002e-05 2024-02-09T18:04:09.268 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.268 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.268 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:04:09.268 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.670033+0000", 2024-02-09T18:04:09.268 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000114552 2024-02-09T18:04:09.268 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:04:09.268 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:04:09.268 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:04:09.268 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.268 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.268 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:203038 9.b 9:d56ac938:::benchmark_data_smithi179_161915_object68156:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e942)", 2024-02-09T18:04:09.268 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:13.535999+0000", 2024-02-09T18:04:09.269 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 55.694272660999999, 2024-02-09T18:04:09.269 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1349075289999999, 2024-02-09T18:04:09.269 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:04:09.269 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:04:09.269 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:04:09.269 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:04:09.269 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:04:09.269 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 203038 2024-02-09T18:04:09.269 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.269 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:04:09.269 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.269 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:04:09.269 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.535999+0000", 2024-02-09T18:04:09.270 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:04:09.270 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.270 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.270 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:04:09.270 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.535999+0000", 2024-02-09T18:04:09.270 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:04:09.270 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.270 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.270 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:04:09.270 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.536001+0000", 2024-02-09T18:04:09.270 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5659999999999999e-06 2024-02-09T18:04:09.270 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.270 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.271 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:04:09.271 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.536007+0000", 2024-02-09T18:04:09.271 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2009999999999996e-06 2024-02-09T18:04:09.271 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.271 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.271 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:04:09.271 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.536025+0000", 2024-02-09T18:04:09.271 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7379000000000001e-05 2024-02-09T18:04:09.271 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.271 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.271 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:04:09.271 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.536035+0000", 2024-02-09T18:04:09.271 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0114e-05 2024-02-09T18:04:09.272 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.272 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.272 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:04:09.272 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.536101+0000", 2024-02-09T18:04:09.272 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5939999999999995e-05 2024-02-09T18:04:09.272 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.272 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.272 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:04:09.272 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.536110+0000", 2024-02-09T18:04:09.272 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.1649999999999995e-06 2024-02-09T18:04:09.272 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.272 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.272 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:04:09.272 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:14.442112+0000", 2024-02-09T18:04:09.273 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.90600196700000002 2024-02-09T18:04:09.273 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.273 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.273 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:04:09.273 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:14.442121+0000", 2024-02-09T18:04:09.273 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.9530000000000005e-06 2024-02-09T18:04:09.273 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.273 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.273 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:04:09.273 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.663892+0000", 2024-02-09T18:04:09.273 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.22177107 2024-02-09T18:04:09.273 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.273 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.273 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:04:09.273 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.664014+0000", 2024-02-09T18:04:09.273 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012196700000000001 2024-02-09T18:04:09.274 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.274 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.274 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5,10", 2024-02-09T18:04:09.274 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.664152+0000", 2024-02-09T18:04:09.274 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013860899999999999 2024-02-09T18:04:09.274 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.274 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.274 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:04:09.274 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.669813+0000", 2024-02-09T18:04:09.274 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0056608930000000002 2024-02-09T18:04:09.274 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.274 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.274 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:09.274 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.670079+0000", 2024-02-09T18:04:09.275 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00026547300000000001 2024-02-09T18:04:09.275 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.275 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.275 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:09.275 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.670116+0000", 2024-02-09T18:04:09.275 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7001e-05 2024-02-09T18:04:09.275 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.275 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.275 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:09.275 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.670746+0000", 2024-02-09T18:04:09.275 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00063032400000000003 2024-02-09T18:04:09.275 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.275 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.275 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:04:09.276 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.670801+0000", 2024-02-09T18:04:09.276 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4648999999999998e-05 2024-02-09T18:04:09.276 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.276 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.276 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:04:09.276 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.670907+0000", 2024-02-09T18:04:09.276 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000106258 2024-02-09T18:04:09.276 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:04:09.276 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:04:09.276 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:04:09.276 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.276 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.276 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:204359 9.b 9:d380bded:::benchmark_data_smithi179_161915_object69477:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e946)", 2024-02-09T18:04:09.276 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:17.542455+0000", 2024-02-09T18:04:09.277 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 51.687816865000002, 2024-02-09T18:04:09.277 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.467034742, 2024-02-09T18:04:09.277 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:04:09.277 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:04:09.277 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:04:09.277 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:04:09.277 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:04:09.277 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 204359 2024-02-09T18:04:09.277 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.277 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:04:09.277 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.277 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:04:09.277 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.542455+0000", 2024-02-09T18:04:09.277 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:04:09.277 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.278 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.278 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:04:09.278 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.542455+0000", 2024-02-09T18:04:09.278 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:04:09.278 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.278 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.278 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:04:09.278 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.542456+0000", 2024-02-09T18:04:09.278 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2359999999999999e-06 2024-02-09T18:04:09.278 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.278 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.278 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:04:09.279 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.542461+0000", 2024-02-09T18:04:09.279 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7350000000000004e-06 2024-02-09T18:04:09.279 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.279 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.279 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:04:09.279 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.542474+0000", 2024-02-09T18:04:09.279 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2418999999999999e-05 2024-02-09T18:04:09.279 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.279 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.279 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:04:09.279 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.542481+0000", 2024-02-09T18:04:09.279 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7919999999999993e-06 2024-02-09T18:04:09.280 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.280 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.280 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:04:09.280 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.542530+0000", 2024-02-09T18:04:09.280 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9060999999999997e-05 2024-02-09T18:04:09.280 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.280 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.280 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:04:09.280 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.542662+0000", 2024-02-09T18:04:09.280 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013146699999999999 2024-02-09T18:04:09.280 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.280 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.281 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5,10", 2024-02-09T18:04:09.281 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.542797+0000", 2024-02-09T18:04:09.281 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000135043 2024-02-09T18:04:09.281 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.281 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.281 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:09.281 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.548368+0000", 2024-02-09T18:04:09.281 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.005571276 2024-02-09T18:04:09.281 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.281 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.281 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:09.281 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.548412+0000", 2024-02-09T18:04:09.281 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3412999999999999e-05 2024-02-09T18:04:09.281 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.282 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.282 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:04:09.282 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.554027+0000", 2024-02-09T18:04:09.282 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0056148750000000001 2024-02-09T18:04:09.282 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.282 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.282 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:09.282 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:18.009355+0000", 2024-02-09T18:04:09.282 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.45532827799999998 2024-02-09T18:04:09.282 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.282 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.282 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:04:09.282 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:18.009398+0000", 2024-02-09T18:04:09.282 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3517999999999999e-05 2024-02-09T18:04:09.283 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.283 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.283 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:04:09.283 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:18.009490+0000", 2024-02-09T18:04:09.283 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.1629e-05 2024-02-09T18:04:09.283 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:04:09.283 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:04:09.283 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:04:09.283 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.283 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.283 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:204365 9.b 9:d4e000a7:::benchmark_data_smithi179_161915_object69483:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e946)", 2024-02-09T18:04:09.283 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:17.548483+0000", 2024-02-09T18:04:09.283 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 51.681789295000002, 2024-02-09T18:04:09.283 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.46123535100000002, 2024-02-09T18:04:09.283 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:04:09.284 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:04:09.284 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:04:09.284 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:04:09.284 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:04:09.284 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 204365 2024-02-09T18:04:09.284 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.284 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:04:09.284 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.284 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:04:09.284 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.548483+0000", 2024-02-09T18:04:09.284 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:04:09.284 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.284 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.284 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:04:09.285 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.548483+0000", 2024-02-09T18:04:09.285 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:04:09.285 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.285 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.285 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:04:09.285 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.548484+0000", 2024-02-09T18:04:09.285 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4810000000000001e-06 2024-02-09T18:04:09.285 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.285 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.285 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:04:09.285 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.548490+0000", 2024-02-09T18:04:09.285 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7880000000000003e-06 2024-02-09T18:04:09.285 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.285 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.286 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:04:09.286 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.548500+0000", 2024-02-09T18:04:09.286 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.8730000000000002e-06 2024-02-09T18:04:09.286 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.286 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.286 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:04:09.286 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.548507+0000", 2024-02-09T18:04:09.286 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7390000000000002e-06 2024-02-09T18:04:09.286 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.286 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.286 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:04:09.286 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.548866+0000", 2024-02-09T18:04:09.287 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00035978099999999999 2024-02-09T18:04:09.287 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.287 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.287 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:04:09.287 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.548994+0000", 2024-02-09T18:04:09.287 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000127849 2024-02-09T18:04:09.287 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.287 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.287 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5,10", 2024-02-09T18:04:09.287 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.549124+0000", 2024-02-09T18:04:09.287 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000129579 2024-02-09T18:04:09.288 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.288 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.288 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:04:09.288 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.556168+0000", 2024-02-09T18:04:09.288 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0070443579999999997 2024-02-09T18:04:09.288 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.288 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.288 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:09.288 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.559407+0000", 2024-02-09T18:04:09.289 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0032383350000000002 2024-02-09T18:04:09.289 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.289 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.289 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:09.289 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.566273+0000", 2024-02-09T18:04:09.289 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0068665050000000002 2024-02-09T18:04:09.289 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.289 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.289 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:09.289 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:18.009548+0000", 2024-02-09T18:04:09.289 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.44327513499999999 2024-02-09T18:04:09.289 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.289 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.289 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:04:09.289 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:18.009590+0000", 2024-02-09T18:04:09.290 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1805999999999999e-05 2024-02-09T18:04:09.290 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.290 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.290 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:04:09.290 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:18.009718+0000", 2024-02-09T18:04:09.290 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012812199999999999 2024-02-09T18:04:09.290 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:04:09.290 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:04:09.290 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:04:09.290 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.290 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.290 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:204373 9.b 9:d1b4954d:::benchmark_data_smithi179_161915_object69491:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e946)", 2024-02-09T18:04:09.290 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:17.570467+0000", 2024-02-09T18:04:09.291 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 51.659805181000003, 2024-02-09T18:04:09.291 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.43951240600000002, 2024-02-09T18:04:09.291 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:04:09.291 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:04:09.291 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:04:09.291 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:04:09.291 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:04:09.291 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 204373 2024-02-09T18:04:09.291 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.291 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:04:09.291 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.291 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:04:09.291 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.570467+0000", 2024-02-09T18:04:09.291 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:04:09.292 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.292 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.292 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:04:09.292 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.570467+0000", 2024-02-09T18:04:09.292 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:04:09.292 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.292 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.292 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:04:09.292 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.570468+0000", 2024-02-09T18:04:09.292 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1379999999999999e-06 2024-02-09T18:04:09.292 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.292 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.292 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:04:09.292 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.570473+0000", 2024-02-09T18:04:09.293 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8369999999999996e-06 2024-02-09T18:04:09.293 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.293 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.293 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:04:09.293 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.570484+0000", 2024-02-09T18:04:09.293 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1558e-05 2024-02-09T18:04:09.293 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.293 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.293 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:04:09.293 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.570490+0000", 2024-02-09T18:04:09.293 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.818e-06 2024-02-09T18:04:09.293 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.293 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.294 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:04:09.294 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.570539+0000", 2024-02-09T18:04:09.294 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8300999999999997e-05 2024-02-09T18:04:09.294 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.294 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.294 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:04:09.294 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.570680+0000", 2024-02-09T18:04:09.294 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014110500000000001 2024-02-09T18:04:09.294 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.294 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.294 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5,10", 2024-02-09T18:04:09.294 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.570832+0000", 2024-02-09T18:04:09.294 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015264099999999999 2024-02-09T18:04:09.294 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.294 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.295 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:09.295 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.576150+0000", 2024-02-09T18:04:09.295 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0053176559999999996 2024-02-09T18:04:09.295 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.295 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.295 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:09.295 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.576235+0000", 2024-02-09T18:04:09.295 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4844000000000003e-05 2024-02-09T18:04:09.295 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.295 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.295 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:04:09.295 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.576449+0000", 2024-02-09T18:04:09.295 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00021409900000000001 2024-02-09T18:04:09.295 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.295 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.296 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:09.296 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:18.009782+0000", 2024-02-09T18:04:09.296 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.43333312699999998 2024-02-09T18:04:09.296 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.296 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.296 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:04:09.296 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:18.009845+0000", 2024-02-09T18:04:09.296 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2781999999999996e-05 2024-02-09T18:04:09.296 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.296 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.296 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:04:09.296 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:18.009979+0000", 2024-02-09T18:04:09.296 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013449999999999999 2024-02-09T18:04:09.296 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:04:09.296 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:04:09.296 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:04:09.297 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.297 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.297 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:218845 9.b 9:d7630bba:::benchmark_data_smithi179_161915_object83963:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e955)", 2024-02-09T18:04:09.297 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:39.728479+0000", 2024-02-09T18:04:09.297 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 29.501792963, 2024-02-09T18:04:09.297 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0303738099999999, 2024-02-09T18:04:09.297 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:04:09.297 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:04:09.297 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:04:09.297 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:04:09.297 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:04:09.297 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 218845 2024-02-09T18:04:09.297 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.297 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:04:09.297 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.297 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:04:09.298 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.728479+0000", 2024-02-09T18:04:09.298 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:04:09.298 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.298 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.298 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:04:09.298 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.728479+0000", 2024-02-09T18:04:09.298 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:04:09.298 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.298 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.298 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:04:09.298 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.728481+0000", 2024-02-09T18:04:09.298 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3259999999999998e-06 2024-02-09T18:04:09.298 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.298 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.298 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:04:09.298 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.728492+0000", 2024-02-09T18:04:09.298 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0499000000000001e-05 2024-02-09T18:04:09.299 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.299 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.299 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:04:09.299 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.728513+0000", 2024-02-09T18:04:09.299 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1019999999999999e-05 2024-02-09T18:04:09.299 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.299 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.299 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:04:09.299 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.728526+0000", 2024-02-09T18:04:09.299 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2808e-05 2024-02-09T18:04:09.299 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.299 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.299 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:04:09.299 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.728640+0000", 2024-02-09T18:04:09.299 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011443299999999999 2024-02-09T18:04:09.299 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.299 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.300 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:04:09.300 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.728652+0000", 2024-02-09T18:04:09.300 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1966e-05 2024-02-09T18:04:09.300 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.300 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.300 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:04:09.300 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.744074+0000", 2024-02-09T18:04:09.300 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.015421535 2024-02-09T18:04:09.300 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.300 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.300 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:04:09.300 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.744231+0000", 2024-02-09T18:04:09.300 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015734700000000001 2024-02-09T18:04:09.300 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.300 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.300 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5", 2024-02-09T18:04:09.301 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.744312+0000", 2024-02-09T18:04:09.301 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1019000000000005e-05 2024-02-09T18:04:09.301 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.301 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.301 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:09.301 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.748870+0000", 2024-02-09T18:04:09.301 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0045577200000000003 2024-02-09T18:04:09.301 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.301 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.301 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:04:09.301 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.753192+0000", 2024-02-09T18:04:09.301 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0043224830000000002 2024-02-09T18:04:09.301 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.302 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.302 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:09.302 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.758753+0000", 2024-02-09T18:04:09.302 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0055606500000000003 2024-02-09T18:04:09.302 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.302 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.302 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:04:09.302 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.758809+0000", 2024-02-09T18:04:09.302 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6573999999999997e-05 2024-02-09T18:04:09.302 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.302 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.302 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:04:09.302 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.758853+0000", 2024-02-09T18:04:09.302 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3430000000000003e-05 2024-02-09T18:04:09.302 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:04:09.302 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:04:09.303 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:04:09.303 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.303 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.303 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:218848 9.b 9:dcfc0c50:::benchmark_data_smithi179_161915_object83966:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e955)", 2024-02-09T18:04:09.303 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:39.728563+0000", 2024-02-09T18:04:09.303 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 29.501709119000001, 2024-02-09T18:04:09.303 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.03056269, 2024-02-09T18:04:09.303 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:04:09.303 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:04:09.303 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:04:09.303 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:04:09.303 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:04:09.303 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 218848 2024-02-09T18:04:09.303 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.303 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:04:09.304 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.304 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:04:09.304 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.728563+0000", 2024-02-09T18:04:09.304 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:04:09.304 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.304 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.304 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:04:09.304 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.728563+0000", 2024-02-09T18:04:09.304 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:04:09.304 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.304 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.304 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:04:09.304 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.728563+0000", 2024-02-09T18:04:09.304 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0100000000000005e-07 2024-02-09T18:04:09.304 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.305 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.305 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:04:09.305 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.728567+0000", 2024-02-09T18:04:09.305 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.174e-06 2024-02-09T18:04:09.305 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.305 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.305 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:04:09.305 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.728574+0000", 2024-02-09T18:04:09.305 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5959999999999998e-06 2024-02-09T18:04:09.305 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.305 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.305 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:04:09.305 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.728580+0000", 2024-02-09T18:04:09.305 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1419999999999998e-06 2024-02-09T18:04:09.305 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.305 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.306 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:04:09.306 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.728680+0000", 2024-02-09T18:04:09.306 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010011800000000001 2024-02-09T18:04:09.306 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.306 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.306 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:04:09.306 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.728689+0000", 2024-02-09T18:04:09.306 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5890000000000003e-06 2024-02-09T18:04:09.306 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.306 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.306 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:04:09.306 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.748905+0000", 2024-02-09T18:04:09.306 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.02021641 2024-02-09T18:04:09.306 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.306 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.306 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:04:09.307 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.748987+0000", 2024-02-09T18:04:09.307 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1627999999999998e-05 2024-02-09T18:04:09.307 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.307 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.307 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5", 2024-02-09T18:04:09.307 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.749061+0000", 2024-02-09T18:04:09.307 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.3549999999999999e-05 2024-02-09T18:04:09.307 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.307 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.307 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:09.307 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.759006+0000", 2024-02-09T18:04:09.307 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0099452959999999993 2024-02-09T18:04:09.307 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.307 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.307 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:04:09.307 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.759020+0000", 2024-02-09T18:04:09.308 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3767e-05 2024-02-09T18:04:09.308 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.308 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.308 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:09.308 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.759069+0000", 2024-02-09T18:04:09.308 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9521999999999998e-05 2024-02-09T18:04:09.308 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.308 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.308 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:04:09.308 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.759102+0000", 2024-02-09T18:04:09.308 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2608999999999999e-05 2024-02-09T18:04:09.308 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.308 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.308 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:04:09.308 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.759126+0000", 2024-02-09T18:04:09.309 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3788e-05 2024-02-09T18:04:09.309 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:04:09.309 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:04:09.309 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:04:09.309 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.309 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.309 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:218853 9.b 9:d58a0ea1:::benchmark_data_smithi179_161915_object83971:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e955)", 2024-02-09T18:04:09.309 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:39.734016+0000", 2024-02-09T18:04:09.309 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 29.496256358, 2024-02-09T18:04:09.309 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0403554850000001, 2024-02-09T18:04:09.309 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:04:09.310 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:04:09.310 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:04:09.310 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:04:09.310 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:04:09.310 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 218853 2024-02-09T18:04:09.310 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.310 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:04:09.310 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.310 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:04:09.310 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.734016+0000", 2024-02-09T18:04:09.310 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:04:09.310 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.310 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.310 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:04:09.311 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.734016+0000", 2024-02-09T18:04:09.311 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:04:09.311 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.311 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.311 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:04:09.311 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.734017+0000", 2024-02-09T18:04:09.311 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7069999999999999e-06 2024-02-09T18:04:09.311 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.311 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.311 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:04:09.311 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.734024+0000", 2024-02-09T18:04:09.311 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6329999999999999e-06 2024-02-09T18:04:09.311 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.311 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.312 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:04:09.312 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.734039+0000", 2024-02-09T18:04:09.312 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4496e-05 2024-02-09T18:04:09.312 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.312 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.312 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:04:09.312 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.734047+0000", 2024-02-09T18:04:09.312 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8310000000000007e-06 2024-02-09T18:04:09.312 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.312 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.312 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:04:09.313 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.734104+0000", 2024-02-09T18:04:09.313 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6239000000000001e-05 2024-02-09T18:04:09.313 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.313 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.313 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:04:09.313 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.734111+0000", 2024-02-09T18:04:09.313 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7549999999999994e-06 2024-02-09T18:04:09.313 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.313 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.313 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:04:09.313 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.753228+0000", 2024-02-09T18:04:09.313 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.019117042 2024-02-09T18:04:09.313 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.314 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.314 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:04:09.314 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.753314+0000", 2024-02-09T18:04:09.314 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5922999999999997e-05 2024-02-09T18:04:09.314 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.314 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.314 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5", 2024-02-09T18:04:09.314 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.753430+0000", 2024-02-09T18:04:09.314 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011583599999999999 2024-02-09T18:04:09.314 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.314 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.314 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:04:09.314 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.759027+0000", 2024-02-09T18:04:09.314 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0055966399999999999 2024-02-09T18:04:09.315 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.315 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.315 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:09.315 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.759189+0000", 2024-02-09T18:04:09.315 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000162194 2024-02-09T18:04:09.315 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.315 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.315 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:09.315 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.774272+0000", 2024-02-09T18:04:09.315 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.015083088999999999 2024-02-09T18:04:09.315 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.315 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.315 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:04:09.315 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.774328+0000", 2024-02-09T18:04:09.316 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6230000000000002e-05 2024-02-09T18:04:09.316 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.316 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.316 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:04:09.316 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.774371+0000", 2024-02-09T18:04:09.316 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2870000000000001e-05 2024-02-09T18:04:09.316 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:04:09.316 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:04:09.316 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:04:09.316 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.316 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.316 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:218911 9.b 9:d44071a9:::benchmark_data_smithi179_161915_object84029:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e955)", 2024-02-09T18:04:09.316 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:39.804466+0000", 2024-02-09T18:04:09.316 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 29.425806213000001, 2024-02-09T18:04:09.316 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.97004875499999998, 2024-02-09T18:04:09.317 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:04:09.317 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:04:09.317 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:04:09.317 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:04:09.317 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:04:09.317 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 218911 2024-02-09T18:04:09.317 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.317 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:04:09.317 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.317 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:04:09.317 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.804466+0000", 2024-02-09T18:04:09.317 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:04:09.317 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.317 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.317 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:04:09.317 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.804466+0000", 2024-02-09T18:04:09.318 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:04:09.318 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.318 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.318 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:04:09.318 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.804468+0000", 2024-02-09T18:04:09.318 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6789999999999999e-06 2024-02-09T18:04:09.318 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.318 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.318 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:04:09.318 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.804475+0000", 2024-02-09T18:04:09.318 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2629999999999997e-06 2024-02-09T18:04:09.318 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.318 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.318 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:04:09.319 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.804492+0000", 2024-02-09T18:04:09.319 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7235e-05 2024-02-09T18:04:09.319 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.319 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.319 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:04:09.319 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.804502+0000", 2024-02-09T18:04:09.319 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0315e-05 2024-02-09T18:04:09.319 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.319 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.319 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:04:09.319 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.804575+0000", 2024-02-09T18:04:09.319 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2276999999999999e-05 2024-02-09T18:04:09.319 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.320 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.320 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:04:09.320 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.804586+0000", 2024-02-09T18:04:09.320 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1156e-05 2024-02-09T18:04:09.320 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.320 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.320 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:04:09.320 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.759242+0000", 2024-02-09T18:04:09.320 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.95465582900000001 2024-02-09T18:04:09.320 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.320 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.320 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:04:09.320 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.759352+0000", 2024-02-09T18:04:09.320 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000110634 2024-02-09T18:04:09.320 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.321 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.321 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5", 2024-02-09T18:04:09.321 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.759463+0000", 2024-02-09T18:04:09.321 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000111072 2024-02-09T18:04:09.321 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.321 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.321 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:09.321 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.774195+0000", 2024-02-09T18:04:09.321 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014731259 2024-02-09T18:04:09.321 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.321 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.321 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:04:09.322 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.774226+0000", 2024-02-09T18:04:09.322 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1853999999999997e-05 2024-02-09T18:04:09.322 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.322 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.322 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:09.322 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.774428+0000", 2024-02-09T18:04:09.322 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00020128400000000001 2024-02-09T18:04:09.322 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.322 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.322 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:04:09.322 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.774479+0000", 2024-02-09T18:04:09.322 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1266e-05 2024-02-09T18:04:09.322 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.322 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.323 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:04:09.323 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.774515+0000", 2024-02-09T18:04:09.323 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.5632e-05 2024-02-09T18:04:09.323 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:04:09.323 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:04:09.323 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:04:09.323 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.323 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.323 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:218913 9.b 9:d0ead2ca:::benchmark_data_smithi179_161915_object84031:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e955)", 2024-02-09T18:04:09.323 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:39.807005+0000", 2024-02-09T18:04:09.323 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 29.423267107000001, 2024-02-09T18:04:09.323 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.96770659599999997, 2024-02-09T18:04:09.323 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:04:09.323 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:04:09.323 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:04:09.324 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:04:09.324 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:04:09.324 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 218913 2024-02-09T18:04:09.324 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.324 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:04:09.324 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.324 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:04:09.324 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.807005+0000", 2024-02-09T18:04:09.324 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:04:09.324 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.324 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.324 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:04:09.324 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.807005+0000", 2024-02-09T18:04:09.325 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:04:09.325 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.325 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.325 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:04:09.325 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.807006+0000", 2024-02-09T18:04:09.325 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.8200000000000008e-07 2024-02-09T18:04:09.325 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.325 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.325 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:04:09.325 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.807011+0000", 2024-02-09T18:04:09.325 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9679999999999997e-06 2024-02-09T18:04:09.325 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.325 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.325 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:04:09.325 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.807023+0000", 2024-02-09T18:04:09.325 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2104e-05 2024-02-09T18:04:09.325 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.326 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.326 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:04:09.326 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.807030+0000", 2024-02-09T18:04:09.326 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4499999999999998e-06 2024-02-09T18:04:09.326 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.326 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.326 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:04:09.326 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.807083+0000", 2024-02-09T18:04:09.326 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2369e-05 2024-02-09T18:04:09.326 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.326 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.326 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:04:09.327 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.807090+0000", 2024-02-09T18:04:09.327 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.0859999999999996e-06 2024-02-09T18:04:09.327 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.327 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.327 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:04:09.327 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.764674+0000", 2024-02-09T18:04:09.327 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.95758389700000002 2024-02-09T18:04:09.327 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.327 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.327 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:04:09.327 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.764787+0000", 2024-02-09T18:04:09.327 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000113126 2024-02-09T18:04:09.327 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.327 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.328 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5", 2024-02-09T18:04:09.328 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.764903+0000", 2024-02-09T18:04:09.328 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000116243 2024-02-09T18:04:09.328 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.328 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.328 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:04:09.328 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.774233+0000", 2024-02-09T18:04:09.328 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0093295749999999997 2024-02-09T18:04:09.328 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.328 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.328 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:09.328 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.774573+0000", 2024-02-09T18:04:09.329 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00034025700000000001 2024-02-09T18:04:09.329 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.329 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.329 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:09.329 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.774627+0000", 2024-02-09T18:04:09.329 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4042000000000002e-05 2024-02-09T18:04:09.329 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.329 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.329 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:04:09.329 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.774677+0000", 2024-02-09T18:04:09.329 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0052000000000001e-05 2024-02-09T18:04:09.329 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.329 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.330 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:04:09.330 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.774712+0000", 2024-02-09T18:04:09.330 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4445e-05 2024-02-09T18:04:09.330 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:04:09.330 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:04:09.330 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:04:09.330 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.330 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.330 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:218917 9.b 9:da2ad27a:::benchmark_data_smithi179_161915_object84035:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e955)", 2024-02-09T18:04:09.330 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:39.814791+0000", 2024-02-09T18:04:09.330 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 29.415480611, 2024-02-09T18:04:09.331 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.97067265400000002, 2024-02-09T18:04:09.331 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:04:09.331 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:04:09.331 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:04:09.331 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:04:09.331 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:04:09.331 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 218917 2024-02-09T18:04:09.331 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.331 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:04:09.331 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.331 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:04:09.331 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.814791+0000", 2024-02-09T18:04:09.331 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:04:09.331 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.331 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.332 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:04:09.332 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.814791+0000", 2024-02-09T18:04:09.332 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:04:09.332 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.332 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.332 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:04:09.332 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.814793+0000", 2024-02-09T18:04:09.332 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7549999999999999e-06 2024-02-09T18:04:09.332 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.332 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.332 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:04:09.333 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.814800+0000", 2024-02-09T18:04:09.333 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2069999999999998e-06 2024-02-09T18:04:09.333 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.333 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.333 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:04:09.333 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.814827+0000", 2024-02-09T18:04:09.333 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6673999999999999e-05 2024-02-09T18:04:09.333 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.333 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.333 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:04:09.333 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.814836+0000", 2024-02-09T18:04:09.333 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.3940000000000002e-06 2024-02-09T18:04:09.333 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.333 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.334 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:04:09.334 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.814902+0000", 2024-02-09T18:04:09.334 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5582999999999999e-05 2024-02-09T18:04:09.334 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.334 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.334 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:04:09.334 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.814912+0000", 2024-02-09T18:04:09.334 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0122999999999999e-05 2024-02-09T18:04:09.334 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.334 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.334 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:04:09.334 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.769797+0000", 2024-02-09T18:04:09.334 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.95488499900000001 2024-02-09T18:04:09.335 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.335 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.335 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:04:09.335 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.769886+0000", 2024-02-09T18:04:09.335 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8718e-05 2024-02-09T18:04:09.335 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.335 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.335 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5", 2024-02-09T18:04:09.335 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.769994+0000", 2024-02-09T18:04:09.335 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000108558 2024-02-09T18:04:09.335 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.335 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.335 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:09.336 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.774762+0000", 2024-02-09T18:04:09.336 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.004767327 2024-02-09T18:04:09.336 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.336 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.336 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:09.336 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.774813+0000", 2024-02-09T18:04:09.336 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0794999999999997e-05 2024-02-09T18:04:09.336 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.336 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.336 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:04:09.336 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.785363+0000", 2024-02-09T18:04:09.336 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010550878 2024-02-09T18:04:09.337 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.337 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.337 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:04:09.337 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.785419+0000", 2024-02-09T18:04:09.337 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5288999999999999e-05 2024-02-09T18:04:09.337 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.337 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.337 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:04:09.337 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.785464+0000", 2024-02-09T18:04:09.337 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5354e-05 2024-02-09T18:04:09.337 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:04:09.337 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:04:09.337 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:04:09.337 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.338 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.338 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:218926 9.b 9:d6680828:::benchmark_data_smithi179_161915_object84044:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e955)", 2024-02-09T18:04:09.338 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:39.829041+0000", 2024-02-09T18:04:09.338 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 29.401231029000002, 2024-02-09T18:04:09.338 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.95663194100000004, 2024-02-09T18:04:09.338 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:04:09.338 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:04:09.338 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:04:09.338 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:04:09.338 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:04:09.338 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 218926 2024-02-09T18:04:09.338 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.338 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:04:09.338 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.339 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:04:09.339 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.829041+0000", 2024-02-09T18:04:09.339 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:04:09.339 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.339 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.339 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:04:09.339 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.829041+0000", 2024-02-09T18:04:09.339 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:04:09.339 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.339 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.339 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:04:09.339 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.829043+0000", 2024-02-09T18:04:09.339 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.517e-06 2024-02-09T18:04:09.339 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.340 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.340 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:04:09.340 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.829049+0000", 2024-02-09T18:04:09.340 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0039999999999996e-06 2024-02-09T18:04:09.340 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.340 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.340 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:04:09.340 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.829062+0000", 2024-02-09T18:04:09.340 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.361e-05 2024-02-09T18:04:09.340 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.340 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.340 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:04:09.340 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.829071+0000", 2024-02-09T18:04:09.340 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3399999999999998e-06 2024-02-09T18:04:09.341 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.341 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.341 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:04:09.341 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.829122+0000", 2024-02-09T18:04:09.341 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1651999999999999e-05 2024-02-09T18:04:09.341 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.341 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.341 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:04:09.341 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.829129+0000", 2024-02-09T18:04:09.341 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9009999999999997e-06 2024-02-09T18:04:09.341 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.341 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.341 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:04:09.341 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.774852+0000", 2024-02-09T18:04:09.342 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.94572300600000003 2024-02-09T18:04:09.342 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.342 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.342 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:04:09.342 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.774981+0000", 2024-02-09T18:04:09.342 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000128868 2024-02-09T18:04:09.342 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.342 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.342 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5", 2024-02-09T18:04:09.342 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.775118+0000", 2024-02-09T18:04:09.342 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013751900000000001 2024-02-09T18:04:09.342 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.342 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.342 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:09.342 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.785528+0000", 2024-02-09T18:04:09.342 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010409932 2024-02-09T18:04:09.343 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.343 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.343 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:09.343 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.785583+0000", 2024-02-09T18:04:09.343 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4208e-05 2024-02-09T18:04:09.343 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.343 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.343 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:04:09.343 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.785596+0000", 2024-02-09T18:04:09.343 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3855e-05 2024-02-09T18:04:09.343 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.343 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.343 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:04:09.343 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.785647+0000", 2024-02-09T18:04:09.344 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0902999999999997e-05 2024-02-09T18:04:09.344 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.344 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.344 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:04:09.344 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.785673+0000", 2024-02-09T18:04:09.344 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5626000000000002e-05 2024-02-09T18:04:09.344 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:04:09.344 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:04:09.344 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:04:09.344 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.344 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.344 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:218931 9.b 9:de156c95:::benchmark_data_smithi179_161915_object84049:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e955)", 2024-02-09T18:04:09.344 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:39.835073+0000", 2024-02-09T18:04:09.345 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 29.395199195, 2024-02-09T18:04:09.345 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.965848398, 2024-02-09T18:04:09.345 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:04:09.345 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:04:09.345 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:04:09.345 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:04:09.345 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:04:09.345 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 218931 2024-02-09T18:04:09.345 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.345 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:04:09.345 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.345 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:04:09.345 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.835073+0000", 2024-02-09T18:04:09.345 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:04:09.345 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.346 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.346 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:04:09.346 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.835073+0000", 2024-02-09T18:04:09.346 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:04:09.346 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.346 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.346 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:04:09.346 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.835075+0000", 2024-02-09T18:04:09.346 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.649e-06 2024-02-09T18:04:09.346 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.346 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.346 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:04:09.346 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.835083+0000", 2024-02-09T18:04:09.346 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3950000000000001e-06 2024-02-09T18:04:09.347 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.347 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.347 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:04:09.347 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.835099+0000", 2024-02-09T18:04:09.347 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6314e-05 2024-02-09T18:04:09.347 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.347 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.347 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:04:09.347 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.835109+0000", 2024-02-09T18:04:09.347 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.4059999999999994e-06 2024-02-09T18:04:09.347 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.347 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.347 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:04:09.347 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.835181+0000", 2024-02-09T18:04:09.347 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2168000000000004e-05 2024-02-09T18:04:09.348 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.348 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.348 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:04:09.348 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.835193+0000", 2024-02-09T18:04:09.348 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1765e-05 2024-02-09T18:04:09.348 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.348 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.348 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:04:09.348 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.780183+0000", 2024-02-09T18:04:09.348 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.94499091800000001 2024-02-09T18:04:09.348 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.348 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.348 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:04:09.348 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.780285+0000", 2024-02-09T18:04:09.349 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000101152 2024-02-09T18:04:09.349 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.349 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.349 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5", 2024-02-09T18:04:09.349 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.780394+0000", 2024-02-09T18:04:09.349 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010912999999999999 2024-02-09T18:04:09.349 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.349 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.349 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:09.349 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.800715+0000", 2024-02-09T18:04:09.349 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.020320794999999999 2024-02-09T18:04:09.349 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.349 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.349 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:04:09.350 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.800731+0000", 2024-02-09T18:04:09.350 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6047000000000001e-05 2024-02-09T18:04:09.350 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.350 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.350 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:09.350 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.800819+0000", 2024-02-09T18:04:09.350 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7986999999999996e-05 2024-02-09T18:04:09.350 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.350 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.350 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:04:09.350 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.800881+0000", 2024-02-09T18:04:09.350 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2718999999999994e-05 2024-02-09T18:04:09.350 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.350 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.350 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:04:09.351 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.800921+0000", 2024-02-09T18:04:09.351 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9953000000000002e-05 2024-02-09T18:04:09.351 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:04:09.351 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:04:09.351 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:04:09.351 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.351 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.351 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:218934 9.b 9:d03e3383:::benchmark_data_smithi179_161915_object84052:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e955)", 2024-02-09T18:04:09.351 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:39.836243+0000", 2024-02-09T18:04:09.351 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 29.39402917, 2024-02-09T18:04:09.351 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.96491841199999995, 2024-02-09T18:04:09.351 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:04:09.351 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:04:09.351 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:04:09.352 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:04:09.352 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:04:09.352 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 218934 2024-02-09T18:04:09.352 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.352 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:04:09.352 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.352 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:04:09.352 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.836243+0000", 2024-02-09T18:04:09.352 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:04:09.352 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.352 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.352 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:04:09.352 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.836243+0000", 2024-02-09T18:04:09.352 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:04:09.352 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.353 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.353 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:04:09.353 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.836244+0000", 2024-02-09T18:04:09.353 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8199999999999999e-07 2024-02-09T18:04:09.353 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.353 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.353 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:04:09.353 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.836247+0000", 2024-02-09T18:04:09.353 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.732e-06 2024-02-09T18:04:09.353 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.353 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.353 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:04:09.353 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.836257+0000", 2024-02-09T18:04:09.353 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0097e-05 2024-02-09T18:04:09.353 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.354 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.354 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:04:09.354 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.836263+0000", 2024-02-09T18:04:09.354 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3550000000000004e-06 2024-02-09T18:04:09.354 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.354 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.354 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:04:09.354 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.836313+0000", 2024-02-09T18:04:09.354 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.045e-05 2024-02-09T18:04:09.354 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.354 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.354 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:04:09.354 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.836322+0000", 2024-02-09T18:04:09.354 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.1689999999999998e-06 2024-02-09T18:04:09.354 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.354 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.355 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:04:09.355 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.785706+0000", 2024-02-09T18:04:09.355 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.949383317 2024-02-09T18:04:09.355 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.355 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.355 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:04:09.355 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.785822+0000", 2024-02-09T18:04:09.355 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000115832 2024-02-09T18:04:09.355 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.355 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.355 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5", 2024-02-09T18:04:09.355 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.785896+0000", 2024-02-09T18:04:09.355 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4567000000000006e-05 2024-02-09T18:04:09.355 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.356 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.356 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:09.356 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.800765+0000", 2024-02-09T18:04:09.356 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014868947 2024-02-09T18:04:09.356 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.356 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.356 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:04:09.356 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.800776+0000", 2024-02-09T18:04:09.356 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1321000000000001e-05 2024-02-09T18:04:09.356 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.356 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.356 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:09.356 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.801049+0000", 2024-02-09T18:04:09.357 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00027226699999999997 2024-02-09T18:04:09.357 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.357 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.357 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:04:09.357 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.801102+0000", 2024-02-09T18:04:09.357 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.287e-05 2024-02-09T18:04:09.357 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.357 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.357 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:04:09.357 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.801161+0000", 2024-02-09T18:04:09.357 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9806000000000003e-05 2024-02-09T18:04:09.357 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:04:09.357 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:04:09.357 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:04:09.358 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.358 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.358 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:218949 9.b 9:d4992918:::benchmark_data_smithi179_161915_object84067:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e955)", 2024-02-09T18:04:09.358 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:39.868952+0000", 2024-02-09T18:04:09.358 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 29.361320056, 2024-02-09T18:04:09.358 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.93233731200000003, 2024-02-09T18:04:09.358 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:04:09.358 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:04:09.358 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:04:09.358 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:04:09.358 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:04:09.358 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 218949 2024-02-09T18:04:09.358 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.358 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:04:09.359 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.359 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:04:09.359 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.868952+0000", 2024-02-09T18:04:09.359 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:04:09.359 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.359 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.359 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:04:09.359 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.868952+0000", 2024-02-09T18:04:09.359 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:04:09.359 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.359 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.359 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:04:09.359 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.868953+0000", 2024-02-09T18:04:09.360 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3859999999999999e-06 2024-02-09T18:04:09.360 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.360 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.360 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:04:09.360 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.868960+0000", 2024-02-09T18:04:09.360 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9110000000000004e-06 2024-02-09T18:04:09.360 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.360 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.360 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:04:09.360 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.868975+0000", 2024-02-09T18:04:09.360 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4568e-05 2024-02-09T18:04:09.360 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.361 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.361 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:04:09.361 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.868983+0000", 2024-02-09T18:04:09.361 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.0739999999999993e-06 2024-02-09T18:04:09.361 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.361 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.361 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:04:09.361 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.869048+0000", 2024-02-09T18:04:09.361 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4838999999999994e-05 2024-02-09T18:04:09.361 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.361 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.361 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:04:09.361 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.869062+0000", 2024-02-09T18:04:09.361 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4154e-05 2024-02-09T18:04:09.362 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.362 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.362 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:04:09.362 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.790566+0000", 2024-02-09T18:04:09.362 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.921504133 2024-02-09T18:04:09.362 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.362 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.362 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:04:09.362 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.790694+0000", 2024-02-09T18:04:09.362 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012811699999999999 2024-02-09T18:04:09.362 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.362 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.362 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5", 2024-02-09T18:04:09.362 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.790816+0000", 2024-02-09T18:04:09.363 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012227399999999999 2024-02-09T18:04:09.363 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.363 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.363 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:04:09.363 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.800781+0000", 2024-02-09T18:04:09.363 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0099647710000000007 2024-02-09T18:04:09.363 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.363 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.363 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:09.363 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.800961+0000", 2024-02-09T18:04:09.363 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00018003600000000001 2024-02-09T18:04:09.363 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.363 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.363 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:09.363 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.801204+0000", 2024-02-09T18:04:09.363 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00024293200000000001 2024-02-09T18:04:09.364 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.364 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.364 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:04:09.364 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.801253+0000", 2024-02-09T18:04:09.364 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.884e-05 2024-02-09T18:04:09.364 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.364 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.364 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:04:09.364 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.801289+0000", 2024-02-09T18:04:09.364 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6276999999999998e-05 2024-02-09T18:04:09.364 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:04:09.364 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:04:09.365 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:04:09.365 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.365 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.365 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:218959 9.b 9:dd8bf74e:::benchmark_data_smithi179_161915_object84077:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e955)", 2024-02-09T18:04:09.365 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:39.893098+0000", 2024-02-09T18:04:09.365 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 29.337174572999999, 2024-02-09T18:04:09.365 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.90831333999999997, 2024-02-09T18:04:09.365 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:04:09.365 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:04:09.365 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:04:09.365 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:04:09.365 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:04:09.365 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 218959 2024-02-09T18:04:09.365 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.366 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:04:09.366 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.366 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:04:09.366 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.893098+0000", 2024-02-09T18:04:09.366 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:04:09.366 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.366 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.366 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:04:09.366 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.893098+0000", 2024-02-09T18:04:09.366 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:04:09.366 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.366 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.366 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:04:09.366 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.893099+0000", 2024-02-09T18:04:09.366 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.677e-06 2024-02-09T18:04:09.367 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.367 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.367 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:04:09.367 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.893105+0000", 2024-02-09T18:04:09.367 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0929999999999999e-06 2024-02-09T18:04:09.367 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.367 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.367 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:04:09.367 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.893122+0000", 2024-02-09T18:04:09.367 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6328999999999999e-05 2024-02-09T18:04:09.367 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.367 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.367 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:04:09.367 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.893129+0000", 2024-02-09T18:04:09.367 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5390000000000002e-06 2024-02-09T18:04:09.368 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.368 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.368 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:04:09.368 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.893198+0000", 2024-02-09T18:04:09.368 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9344999999999994e-05 2024-02-09T18:04:09.368 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.368 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.368 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:04:09.368 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.893207+0000", 2024-02-09T18:04:09.368 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5029999999999997e-06 2024-02-09T18:04:09.368 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.368 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.368 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:04:09.368 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.795918+0000", 2024-02-09T18:04:09.368 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.90271066700000002 2024-02-09T18:04:09.369 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.369 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.369 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:04:09.369 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.796043+0000", 2024-02-09T18:04:09.369 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000125253 2024-02-09T18:04:09.369 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.369 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.369 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5", 2024-02-09T18:04:09.369 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.796173+0000", 2024-02-09T18:04:09.369 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000130288 2024-02-09T18:04:09.369 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.369 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.369 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:04:09.369 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.800977+0000", 2024-02-09T18:04:09.370 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0048040649999999997 2024-02-09T18:04:09.370 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.370 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.370 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:09.370 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.801011+0000", 2024-02-09T18:04:09.370 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.3549e-05 2024-02-09T18:04:09.370 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.370 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.370 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:09.370 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.801331+0000", 2024-02-09T18:04:09.370 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00031982100000000003 2024-02-09T18:04:09.370 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.370 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.371 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:04:09.371 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.801376+0000", 2024-02-09T18:04:09.371 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5825000000000003e-05 2024-02-09T18:04:09.371 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.371 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.371 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:04:09.371 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.801411+0000", 2024-02-09T18:04:09.371 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4385999999999999e-05 2024-02-09T18:04:09.371 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:04:09.371 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:04:09.371 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:04:09.371 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.371 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.371 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:218960 9.b 9:df61268f:::benchmark_data_smithi179_161915_object84078:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e955)", 2024-02-09T18:04:09.372 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:39.897308+0000", 2024-02-09T18:04:09.372 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 29.332963835000001, 2024-02-09T18:04:09.372 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.92915402400000002, 2024-02-09T18:04:09.372 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:04:09.372 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:04:09.372 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:04:09.372 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:04:09.372 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:04:09.372 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 218960 2024-02-09T18:04:09.372 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.372 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:04:09.372 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.372 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:04:09.372 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.897308+0000", 2024-02-09T18:04:09.373 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:04:09.373 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.373 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.373 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:04:09.373 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.897308+0000", 2024-02-09T18:04:09.373 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:04:09.373 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.373 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.373 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:04:09.373 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.897309+0000", 2024-02-09T18:04:09.373 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2200000000000002e-07 2024-02-09T18:04:09.373 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.373 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.374 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:04:09.374 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.897314+0000", 2024-02-09T18:04:09.374 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0919999999999997e-06 2024-02-09T18:04:09.374 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.374 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.374 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:04:09.374 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.897325+0000", 2024-02-09T18:04:09.374 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1097999999999999e-05 2024-02-09T18:04:09.374 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.374 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.374 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:04:09.374 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.897332+0000", 2024-02-09T18:04:09.374 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8770000000000004e-06 2024-02-09T18:04:09.374 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.375 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.375 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:04:09.375 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.897384+0000", 2024-02-09T18:04:09.375 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1799999999999999e-05 2024-02-09T18:04:09.375 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.375 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.375 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:04:09.375 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.897391+0000", 2024-02-09T18:04:09.375 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4499999999999998e-06 2024-02-09T18:04:09.375 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.375 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.375 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:04:09.375 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.801446+0000", 2024-02-09T18:04:09.375 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.90405485100000005 2024-02-09T18:04:09.376 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.376 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.376 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:04:09.376 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.801555+0000", 2024-02-09T18:04:09.376 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000108349 2024-02-09T18:04:09.376 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.376 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.376 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5", 2024-02-09T18:04:09.376 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.801621+0000", 2024-02-09T18:04:09.376 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6285000000000007e-05 2024-02-09T18:04:09.376 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.376 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.376 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:09.376 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.812163+0000", 2024-02-09T18:04:09.376 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010541903 2024-02-09T18:04:09.377 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.377 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.377 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:04:09.377 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.812177+0000", 2024-02-09T18:04:09.377 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4188e-05 2024-02-09T18:04:09.377 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.377 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.377 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:04:09.377 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.826369+0000", 2024-02-09T18:04:09.377 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014192421 2024-02-09T18:04:09.377 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.377 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.377 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:04:09.377 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.826421+0000", 2024-02-09T18:04:09.377 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1959999999999997e-05 2024-02-09T18:04:09.377 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:04:09.378 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:04:09.378 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:04:09.378 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.826462+0000", 2024-02-09T18:04:09.378 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0828000000000003e-05 2024-02-09T18:04:09.378 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:04:09.378 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:04:09.378 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:04:09.378 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:04:09.378 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:04:09.378 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T18:04:10.019 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-09T18:04:10.417 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:10 smithi099 ceph-mon[39420]: pgmap v2388: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 1.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 823 op/s; 1829/89880 objects misplaced (2.035%) 2024-02-09T18:04:10.419 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:10 smithi179 ceph-mon[43975]: pgmap v2388: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 1.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 823 op/s; 1829/89880 objects misplaced (2.035%) 2024-02-09T18:04:10.446 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:04:10.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:10.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:04:10.556 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:04:10.619 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:10.648 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:04:10.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:10.654 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:04:10.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:10 smithi162 ceph-mon[44304]: pgmap v2388: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 1.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 823 op/s; 1829/89880 objects misplaced (2.035%) 2024-02-09T18:04:10.755 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T18:04:10.822 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:10.860 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T18:04:10.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:10.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T18:04:10.964 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:04:10.999 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:11.033 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:04:11.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:11.038 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:04:11.139 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T18:04:11.172 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:11.205 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T18:04:11.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:11.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T18:04:11.311 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T18:04:11.362 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:11.395 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T18:04:11.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:11.401 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T18:04:11.436 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:11 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4279007563' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T18:04:11.436 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:11 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T18:04:11.502 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:04:11.509 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is unset 2024-02-09T18:04:11.555 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:11.587 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:04:11.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:11.591 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:04:11.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:11 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4279007563' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T18:04:11.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:11 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T18:04:11.691 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:04:11.747 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:11.856 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 5, 6, 7, 8, 9, 11, 0, 2, 4] out_osds: [10] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 7, 9, 10, 11, 8, 3] 2024-02-09T18:04:11.856 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T18:04:11.857 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-02-09T18:04:11.857 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-09T18:04:11.915 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T18:04:11.915 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T18:04:11.915 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T18:04:11.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:11 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4279007563' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T18:04:11.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:11 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T18:04:11.938 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T18:04:11.992 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T18:04:12.035 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T18:04:12.069 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T18:04:12.105 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T18:04:12.131 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T18:04:12.163 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T18:04:12.209 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T18:04:12.233 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T18:04:12.238 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T18:04:12.238 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T18:04:12.238 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T18:04:12.238 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T18:04:12.238 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T18:04:12.238 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T18:04:12.238 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T18:04:12.238 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T18:04:12.238 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T18:04:12.238 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T18:04:12.239 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T18:04:12.239 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T18:04:12.460 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:04:12.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:12.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:04:12.565 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:04:12.603 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:12 smithi099 ceph-mon[39420]: pgmap v2389: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 1.7 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 958 op/s; 1547/81852 objects misplaced (1.890%) 2024-02-09T18:04:12.603 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:12 smithi099 ceph-mon[39420]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:04:12.603 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:12 smithi099 ceph-mon[39420]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T18:04:12.603 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:12 smithi099 ceph-mon[39420]: osdmap e967: 12 total, 12 up, 11 in 2024-02-09T18:04:12.603 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:12 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1509923867' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:04:12.605 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:12.760 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:04:12.870 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:04:12.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:12.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:04:12.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:12 smithi179 ceph-mon[43975]: pgmap v2389: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 1.7 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 958 op/s; 1547/81852 objects misplaced (1.890%) 2024-02-09T18:04:12.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:12 smithi179 ceph-mon[43975]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:04:12.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:12 smithi179 ceph-mon[43975]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T18:04:12.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:12 smithi179 ceph-mon[43975]: osdmap e967: 12 total, 12 up, 11 in 2024-02-09T18:04:12.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:12 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1509923867' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:04:12.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:12 smithi162 ceph-mon[44304]: pgmap v2389: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 1.7 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 958 op/s; 1547/81852 objects misplaced (1.890%) 2024-02-09T18:04:12.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:12 smithi162 ceph-mon[44304]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:04:12.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:12 smithi162 ceph-mon[44304]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T18:04:12.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:12 smithi162 ceph-mon[44304]: osdmap e967: 12 total, 12 up, 11 in 2024-02-09T18:04:12.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:12 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1509923867' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:04:12.981 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:04:13.025 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:13.057 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:04:13.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:13.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:04:13.166 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T18:04:13.208 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:13.242 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T18:04:13.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:13.252 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T18:04:13.327 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:04:13.353 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:04:13.398 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:13.436 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:04:13.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:13.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:04:13.542 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:04:13.577 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:13 smithi099 ceph-mon[39420]: 1.f scrub starts 2024-02-09T18:04:13.577 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:13 smithi099 ceph-mon[39420]: 1.1 scrub starts 2024-02-09T18:04:13.577 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:13 smithi099 ceph-mon[39420]: 1.1 scrub ok 2024-02-09T18:04:13.577 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:13 smithi099 ceph-mon[39420]: 1.f scrub ok 2024-02-09T18:04:13.580 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:13.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:13 smithi179 ceph-mon[43975]: 1.f scrub starts 2024-02-09T18:04:13.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:13 smithi179 ceph-mon[43975]: 1.1 scrub starts 2024-02-09T18:04:13.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:13 smithi179 ceph-mon[43975]: 1.1 scrub ok 2024-02-09T18:04:13.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:13 smithi179 ceph-mon[43975]: 1.f scrub ok 2024-02-09T18:04:13.717 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:04:13.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:13.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:04:13.825 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T18:04:13.861 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:13 smithi162 ceph-mon[44304]: 1.f scrub starts 2024-02-09T18:04:13.861 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:13 smithi162 ceph-mon[44304]: 1.1 scrub starts 2024-02-09T18:04:13.862 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:13 smithi162 ceph-mon[44304]: 1.1 scrub ok 2024-02-09T18:04:13.862 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:13 smithi162 ceph-mon[44304]: 1.f scrub ok 2024-02-09T18:04:13.864 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:13.888 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T18:04:13.891 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:13.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T18:04:13.993 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:04:14.019 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:14.500 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-09T18:04:14.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:14 smithi099 ceph-mon[39420]: pgmap v2391: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 1.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 871 op/s; 1547/76875 objects misplaced (2.012%) 2024-02-09T18:04:14.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:14 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:04:14.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:14 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:04:14.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:14 smithi179 ceph-mon[43975]: pgmap v2391: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 1.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 871 op/s; 1547/76875 objects misplaced (2.012%) 2024-02-09T18:04:14.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:14 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:04:14.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:14 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:04:14.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:14 smithi162 ceph-mon[44304]: pgmap v2391: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 1.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 871 op/s; 1547/76875 objects misplaced (2.012%) 2024-02-09T18:04:14.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:14 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:04:14.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:14 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:04:15.302 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T18:04:15.392 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:04:15.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:15.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:04:15.499 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:04:15.537 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:15.569 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:04:15.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:15.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:04:15.681 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T18:04:15.726 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:15.751 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:15 smithi099 ceph-mon[39420]: 9.1 scrub starts 2024-02-09T18:04:15.751 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:15 smithi099 ceph-mon[39420]: osdmap e968: 12 total, 12 up, 11 in 2024-02-09T18:04:15.751 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:15 smithi099 ceph-mon[39420]: 9.1 scrub ok 2024-02-09T18:04:15.767 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T18:04:15.775 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:15 smithi162 ceph-mon[44304]: 9.1 scrub starts 2024-02-09T18:04:15.775 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:15 smithi162 ceph-mon[44304]: osdmap e968: 12 total, 12 up, 11 in 2024-02-09T18:04:15.775 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:15 smithi162 ceph-mon[44304]: 9.1 scrub ok 2024-02-09T18:04:15.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:15.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T18:04:15.878 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:04:15.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:15 smithi179 ceph-mon[43975]: 9.1 scrub starts 2024-02-09T18:04:15.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:15 smithi179 ceph-mon[43975]: osdmap e968: 12 total, 12 up, 11 in 2024-02-09T18:04:15.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:15 smithi179 ceph-mon[43975]: 9.1 scrub ok 2024-02-09T18:04:15.930 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:15.970 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:04:15.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:15.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:04:16.077 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T18:04:16.124 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:16.154 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T18:04:16.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:16.161 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T18:04:16.262 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:04:16.309 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:16.340 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:04:16.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:16.347 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:04:16.448 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:04:16.507 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:16.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:16 smithi179 ceph-mon[43975]: 1.1b scrub starts 2024-02-09T18:04:16.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:16 smithi179 ceph-mon[43975]: 1.1b scrub ok 2024-02-09T18:04:16.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:16 smithi179 ceph-mon[43975]: pgmap v2393: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 1.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 673 op/s; 1547/76875 objects misplaced (2.012%) 2024-02-09T18:04:16.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:16 smithi179 ceph-mon[43975]: 1.10 scrub starts 2024-02-09T18:04:16.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:16 smithi179 ceph-mon[43975]: 1.10 scrub ok 2024-02-09T18:04:16.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:16 smithi179 ceph-mon[43975]: 9.0 scrub starts 2024-02-09T18:04:16.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:16 smithi162 ceph-mon[44304]: 1.1b scrub starts 2024-02-09T18:04:16.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:16 smithi162 ceph-mon[44304]: 1.1b scrub ok 2024-02-09T18:04:16.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:16 smithi162 ceph-mon[44304]: pgmap v2393: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 1.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 673 op/s; 1547/76875 objects misplaced (2.012%) 2024-02-09T18:04:16.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:16 smithi162 ceph-mon[44304]: 1.10 scrub starts 2024-02-09T18:04:16.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:16 smithi162 ceph-mon[44304]: 1.10 scrub ok 2024-02-09T18:04:16.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:16 smithi162 ceph-mon[44304]: 9.0 scrub starts 2024-02-09T18:04:16.950 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:16 smithi099 ceph-mon[39420]: 1.1b scrub starts 2024-02-09T18:04:16.950 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:16 smithi099 ceph-mon[39420]: 1.1b scrub ok 2024-02-09T18:04:16.950 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:16 smithi099 ceph-mon[39420]: pgmap v2393: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 1.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 673 op/s; 1547/76875 objects misplaced (2.012%) 2024-02-09T18:04:16.950 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:16 smithi099 ceph-mon[39420]: 1.10 scrub starts 2024-02-09T18:04:16.950 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:16 smithi099 ceph-mon[39420]: 1.10 scrub ok 2024-02-09T18:04:16.950 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:16 smithi099 ceph-mon[39420]: 9.0 scrub starts 2024-02-09T18:04:17.409 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T18:04:17.409 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T18:04:17.696 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:04:17.701 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:17 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:04:17.701 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:17 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:04:17.701 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:17 smithi099 ceph-mon[39420]: 9.b scrub starts 2024-02-09T18:04:17.701 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:17 smithi099 ceph-mon[39420]: 9.b scrub ok 2024-02-09T18:04:17.701 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:17 smithi099 ceph-mon[39420]: 1.18 scrub starts 2024-02-09T18:04:17.701 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:17 smithi099 ceph-mon[39420]: 1.18 scrub ok 2024-02-09T18:04:17.701 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:17 smithi099 ceph-mon[39420]: 1.6 scrub starts 2024-02-09T18:04:17.701 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:17 smithi099 ceph-mon[39420]: 1.6 scrub ok 2024-02-09T18:04:17.701 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:17 smithi099 ceph-mon[39420]: osdmap e969: 12 total, 12 up, 11 in 2024-02-09T18:04:17.701 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:17 smithi099 ceph-mon[39420]: 9.0 scrub ok 2024-02-09T18:04:17.701 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:17 smithi099 ceph-mon[39420]: 9.6 scrub starts 2024-02-09T18:04:17.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:17.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:04:17.718 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:04:17.733 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T18:04:17.733 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T18:04:17.733 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 969, 2024-02-09T18:04:17.733 INFO:teuthology.orchestra.run.smithi099.stdout: "fsid": "a69c870a-c76e-11ee-95b7-87774f69a715", 2024-02-09T18:04:17.733 INFO:teuthology.orchestra.run.smithi099.stdout: "created": "2024-02-09T17:15:26.113606+0000", 2024-02-09T18:04:17.734 INFO:teuthology.orchestra.run.smithi099.stdout: "modified": "2024-02-09T18:04:16.483760+0000", 2024-02-09T18:04:17.734 INFO:teuthology.orchestra.run.smithi099.stdout: "last_up_change": "2024-02-09T17:59:11.889493+0000", 2024-02-09T18:04:17.734 INFO:teuthology.orchestra.run.smithi099.stdout: "last_in_change": "2024-02-09T18:03:13.429500+0000", 2024-02-09T18:04:17.734 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-09T18:04:17.734 INFO:teuthology.orchestra.run.smithi099.stdout: "flags_num": 5804032, 2024-02-09T18:04:17.734 INFO:teuthology.orchestra.run.smithi099.stdout: "flags_set": [ 2024-02-09T18:04:17.734 INFO:teuthology.orchestra.run.smithi099.stdout: "nodeep-scrub", 2024-02-09T18:04:17.734 INFO:teuthology.orchestra.run.smithi099.stdout: "pglog_hardlimit", 2024-02-09T18:04:17.734 INFO:teuthology.orchestra.run.smithi099.stdout: "purged_snapdirs", 2024-02-09T18:04:17.734 INFO:teuthology.orchestra.run.smithi099.stdout: "recovery_deletes", 2024-02-09T18:04:17.734 INFO:teuthology.orchestra.run.smithi099.stdout: "sortbitwise" 2024-02-09T18:04:17.734 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T18:04:17.734 INFO:teuthology.orchestra.run.smithi099.stdout: "crush_version": 28, 2024-02-09T18:04:17.735 INFO:teuthology.orchestra.run.smithi099.stdout: "full_ratio": 0.94999998807907104, 2024-02-09T18:04:17.735 INFO:teuthology.orchestra.run.smithi099.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-09T18:04:17.735 INFO:teuthology.orchestra.run.smithi099.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-09T18:04:17.735 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_snapshot": "", 2024-02-09T18:04:17.735 INFO:teuthology.orchestra.run.smithi099.stdout: "pool_max": 9, 2024-02-09T18:04:17.735 INFO:teuthology.orchestra.run.smithi099.stdout: "max_osd": 12, 2024-02-09T18:04:17.735 INFO:teuthology.orchestra.run.smithi099.stdout: "require_min_compat_client": "luminous", 2024-02-09T18:04:17.735 INFO:teuthology.orchestra.run.smithi099.stdout: "min_compat_client": "luminous", 2024-02-09T18:04:17.735 INFO:teuthology.orchestra.run.smithi099.stdout: "require_osd_release": "squid", 2024-02-09T18:04:17.735 INFO:teuthology.orchestra.run.smithi099.stdout: "allow_crimson": false, 2024-02-09T18:04:17.735 INFO:teuthology.orchestra.run.smithi099.stdout: "pools": [ 2024-02-09T18:04:17.735 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.736 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 1, 2024-02-09T18:04:17.736 INFO:teuthology.orchestra.run.smithi099.stdout: "pool_name": ".mgr", 2024-02-09T18:04:17.736 INFO:teuthology.orchestra.run.smithi099.stdout: "create_time": "2024-02-09T17:18:56.552286+0000", 2024-02-09T18:04:17.736 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 1, 2024-02-09T18:04:17.736 INFO:teuthology.orchestra.run.smithi099.stdout: "flags_names": "hashpspool", 2024-02-09T18:04:17.736 INFO:teuthology.orchestra.run.smithi099.stdout: "type": 1, 2024-02-09T18:04:17.736 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-02-09T18:04:17.736 INFO:teuthology.orchestra.run.smithi099.stdout: "min_size": 2, 2024-02-09T18:04:17.736 INFO:teuthology.orchestra.run.smithi099.stdout: "crush_rule": 0, 2024-02-09T18:04:17.736 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_count": 0, 2024-02-09T18:04:17.736 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_target": 0, 2024-02-09T18:04:17.737 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_barrier": 0, 2024-02-09T18:04:17.737 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-09T18:04:17.737 INFO:teuthology.orchestra.run.smithi099.stdout: "object_hash": 2, 2024-02-09T18:04:17.737 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_autoscale_mode": "off", 2024-02-09T18:04:17.737 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num": 31, 2024-02-09T18:04:17.737 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_placement_num": 31, 2024-02-09T18:04:17.737 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_placement_num_target": 31, 2024-02-09T18:04:17.737 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num_target": 31, 2024-02-09T18:04:17.737 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num_pending": 31, 2024-02-09T18:04:17.737 INFO:teuthology.orchestra.run.smithi099.stdout: "last_pg_merge_meta": { 2024-02-09T18:04:17.737 INFO:teuthology.orchestra.run.smithi099.stdout: "source_pgid": "0.0", 2024-02-09T18:04:17.737 INFO:teuthology.orchestra.run.smithi099.stdout: "ready_epoch": 0, 2024-02-09T18:04:17.737 INFO:teuthology.orchestra.run.smithi099.stdout: "last_epoch_started": 0, 2024-02-09T18:04:17.738 INFO:teuthology.orchestra.run.smithi099.stdout: "last_epoch_clean": 0, 2024-02-09T18:04:17.738 INFO:teuthology.orchestra.run.smithi099.stdout: "source_version": "0'0", 2024-02-09T18:04:17.738 INFO:teuthology.orchestra.run.smithi099.stdout: "target_version": "0'0" 2024-02-09T18:04:17.738 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.738 INFO:teuthology.orchestra.run.smithi099.stdout: "last_change": "544", 2024-02-09T18:04:17.738 INFO:teuthology.orchestra.run.smithi099.stdout: "last_force_op_resend": "0", 2024-02-09T18:04:17.738 INFO:teuthology.orchestra.run.smithi099.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-09T18:04:17.738 INFO:teuthology.orchestra.run.smithi099.stdout: "last_force_op_resend_preluminous": "514", 2024-02-09T18:04:17.738 INFO:teuthology.orchestra.run.smithi099.stdout: "auid": 0, 2024-02-09T18:04:17.738 INFO:teuthology.orchestra.run.smithi099.stdout: "snap_mode": "selfmanaged", 2024-02-09T18:04:17.738 INFO:teuthology.orchestra.run.smithi099.stdout: "snap_seq": 0, 2024-02-09T18:04:17.738 INFO:teuthology.orchestra.run.smithi099.stdout: "snap_epoch": 0, 2024-02-09T18:04:17.739 INFO:teuthology.orchestra.run.smithi099.stdout: "pool_snaps": [], 2024-02-09T18:04:17.739 INFO:teuthology.orchestra.run.smithi099.stdout: "removed_snaps": "[]", 2024-02-09T18:04:17.739 INFO:teuthology.orchestra.run.smithi099.stdout: "quota_max_bytes": 0, 2024-02-09T18:04:17.739 INFO:teuthology.orchestra.run.smithi099.stdout: "quota_max_objects": 0, 2024-02-09T18:04:17.739 INFO:teuthology.orchestra.run.smithi099.stdout: "tiers": [], 2024-02-09T18:04:17.739 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_of": -1, 2024-02-09T18:04:17.739 INFO:teuthology.orchestra.run.smithi099.stdout: "read_tier": -1, 2024-02-09T18:04:17.739 INFO:teuthology.orchestra.run.smithi099.stdout: "write_tier": -1, 2024-02-09T18:04:17.739 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_mode": "none", 2024-02-09T18:04:17.739 INFO:teuthology.orchestra.run.smithi099.stdout: "target_max_bytes": 0, 2024-02-09T18:04:17.739 INFO:teuthology.orchestra.run.smithi099.stdout: "target_max_objects": 0, 2024-02-09T18:04:17.739 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-09T18:04:17.739 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-09T18:04:17.739 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-09T18:04:17.740 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_min_flush_age": 0, 2024-02-09T18:04:17.740 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_min_evict_age": 0, 2024-02-09T18:04:17.740 INFO:teuthology.orchestra.run.smithi099.stdout: "erasure_code_profile": "", 2024-02-09T18:04:17.740 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_params": { 2024-02-09T18:04:17.740 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "none" 2024-02-09T18:04:17.740 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.740 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_period": 0, 2024-02-09T18:04:17.740 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_count": 0, 2024-02-09T18:04:17.740 INFO:teuthology.orchestra.run.smithi099.stdout: "use_gmt_hitset": true, 2024-02-09T18:04:17.740 INFO:teuthology.orchestra.run.smithi099.stdout: "min_read_recency_for_promote": 0, 2024-02-09T18:04:17.740 INFO:teuthology.orchestra.run.smithi099.stdout: "min_write_recency_for_promote": 0, 2024-02-09T18:04:17.740 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_grade_decay_rate": 0, 2024-02-09T18:04:17.740 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_search_last_n": 0, 2024-02-09T18:04:17.741 INFO:teuthology.orchestra.run.smithi099.stdout: "grade_table": [], 2024-02-09T18:04:17.741 INFO:teuthology.orchestra.run.smithi099.stdout: "stripe_width": 0, 2024-02-09T18:04:17.741 INFO:teuthology.orchestra.run.smithi099.stdout: "expected_num_objects": 0, 2024-02-09T18:04:17.741 INFO:teuthology.orchestra.run.smithi099.stdout: "fast_read": false, 2024-02-09T18:04:17.741 INFO:teuthology.orchestra.run.smithi099.stdout: "options": { 2024-02-09T18:04:17.741 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num_max": 32, 2024-02-09T18:04:17.741 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num_min": 1 2024-02-09T18:04:17.741 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.741 INFO:teuthology.orchestra.run.smithi099.stdout: "application_metadata": { 2024-02-09T18:04:17.741 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": {} 2024-02-09T18:04:17.741 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.741 INFO:teuthology.orchestra.run.smithi099.stdout: "read_balance": { 2024-02-09T18:04:17.741 INFO:teuthology.orchestra.run.smithi099.stdout: "score_type": "Fair distribution", 2024-02-09T18:04:17.741 INFO:teuthology.orchestra.run.smithi099.stdout: "score_acting": 7.119999885559082, 2024-02-09T18:04:17.742 INFO:teuthology.orchestra.run.smithi099.stdout: "score_stable": 2.130000114440918, 2024-02-09T18:04:17.742 INFO:teuthology.orchestra.run.smithi099.stdout: "optimal_score": 1, 2024-02-09T18:04:17.742 INFO:teuthology.orchestra.run.smithi099.stdout: "raw_score_acting": 7.119999885559082, 2024-02-09T18:04:17.742 INFO:teuthology.orchestra.run.smithi099.stdout: "raw_score_stable": 2.130000114440918, 2024-02-09T18:04:17.742 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity_weighted": 0.78079599142074585, 2024-02-09T18:04:17.742 INFO:teuthology.orchestra.run.smithi099.stdout: "average_primary_affinity": 0.85000002384185791, 2024-02-09T18:04:17.742 INFO:teuthology.orchestra.run.smithi099.stdout: "average_primary_affinity_weighted": 0.77999997138977051, 2024-02-09T18:04:17.742 INFO:teuthology.orchestra.run.smithi099.stdout: "error_message": "pool 1 has acting primaries on OSD(s) with primary affinity 0, read balance score is not accurate" 2024-02-09T18:04:17.742 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.742 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.742 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.742 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2024-02-09T18:04:17.742 INFO:teuthology.orchestra.run.smithi099.stdout: "pool_name": "unique_pool_7", 2024-02-09T18:04:17.743 INFO:teuthology.orchestra.run.smithi099.stdout: "create_time": "2024-02-09T18:00:06.232818+0000", 2024-02-09T18:04:17.743 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 1, 2024-02-09T18:04:17.743 INFO:teuthology.orchestra.run.smithi099.stdout: "flags_names": "hashpspool", 2024-02-09T18:04:17.743 INFO:teuthology.orchestra.run.smithi099.stdout: "type": 1, 2024-02-09T18:04:17.743 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2024-02-09T18:04:17.743 INFO:teuthology.orchestra.run.smithi099.stdout: "min_size": 2, 2024-02-09T18:04:17.743 INFO:teuthology.orchestra.run.smithi099.stdout: "crush_rule": 0, 2024-02-09T18:04:17.743 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_count": 0, 2024-02-09T18:04:17.743 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_target": 0, 2024-02-09T18:04:17.743 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_barrier": 0, 2024-02-09T18:04:17.743 INFO:teuthology.orchestra.run.smithi099.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-09T18:04:17.743 INFO:teuthology.orchestra.run.smithi099.stdout: "object_hash": 2, 2024-02-09T18:04:17.743 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_autoscale_mode": "off", 2024-02-09T18:04:17.744 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num": 16, 2024-02-09T18:04:17.744 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_placement_num": 16, 2024-02-09T18:04:17.744 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_placement_num_target": 16, 2024-02-09T18:04:17.744 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num_target": 16, 2024-02-09T18:04:17.744 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_num_pending": 16, 2024-02-09T18:04:17.744 INFO:teuthology.orchestra.run.smithi099.stdout: "last_pg_merge_meta": { 2024-02-09T18:04:17.744 INFO:teuthology.orchestra.run.smithi099.stdout: "source_pgid": "0.0", 2024-02-09T18:04:17.744 INFO:teuthology.orchestra.run.smithi099.stdout: "ready_epoch": 0, 2024-02-09T18:04:17.744 INFO:teuthology.orchestra.run.smithi099.stdout: "last_epoch_started": 0, 2024-02-09T18:04:17.744 INFO:teuthology.orchestra.run.smithi099.stdout: "last_epoch_clean": 0, 2024-02-09T18:04:17.744 INFO:teuthology.orchestra.run.smithi099.stdout: "source_version": "0'0", 2024-02-09T18:04:17.744 INFO:teuthology.orchestra.run.smithi099.stdout: "target_version": "0'0" 2024-02-09T18:04:17.744 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.744 INFO:teuthology.orchestra.run.smithi099.stdout: "last_change": "886", 2024-02-09T18:04:17.745 INFO:teuthology.orchestra.run.smithi099.stdout: "last_force_op_resend": "0", 2024-02-09T18:04:17.745 INFO:teuthology.orchestra.run.smithi099.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-09T18:04:17.745 INFO:teuthology.orchestra.run.smithi099.stdout: "last_force_op_resend_preluminous": "0", 2024-02-09T18:04:17.745 INFO:teuthology.orchestra.run.smithi099.stdout: "auid": 0, 2024-02-09T18:04:17.745 INFO:teuthology.orchestra.run.smithi099.stdout: "snap_mode": "selfmanaged", 2024-02-09T18:04:17.745 INFO:teuthology.orchestra.run.smithi099.stdout: "snap_seq": 0, 2024-02-09T18:04:17.745 INFO:teuthology.orchestra.run.smithi099.stdout: "snap_epoch": 0, 2024-02-09T18:04:17.745 INFO:teuthology.orchestra.run.smithi099.stdout: "pool_snaps": [], 2024-02-09T18:04:17.745 INFO:teuthology.orchestra.run.smithi099.stdout: "removed_snaps": "[]", 2024-02-09T18:04:17.745 INFO:teuthology.orchestra.run.smithi099.stdout: "quota_max_bytes": 0, 2024-02-09T18:04:17.745 INFO:teuthology.orchestra.run.smithi099.stdout: "quota_max_objects": 0, 2024-02-09T18:04:17.745 INFO:teuthology.orchestra.run.smithi099.stdout: "tiers": [], 2024-02-09T18:04:17.745 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_of": -1, 2024-02-09T18:04:17.745 INFO:teuthology.orchestra.run.smithi099.stdout: "read_tier": -1, 2024-02-09T18:04:17.745 INFO:teuthology.orchestra.run.smithi099.stdout: "write_tier": -1, 2024-02-09T18:04:17.746 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_mode": "none", 2024-02-09T18:04:17.746 INFO:teuthology.orchestra.run.smithi099.stdout: "target_max_bytes": 0, 2024-02-09T18:04:17.746 INFO:teuthology.orchestra.run.smithi099.stdout: "target_max_objects": 0, 2024-02-09T18:04:17.746 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-09T18:04:17.746 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-09T18:04:17.746 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-09T18:04:17.746 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_min_flush_age": 0, 2024-02-09T18:04:17.746 INFO:teuthology.orchestra.run.smithi099.stdout: "cache_min_evict_age": 0, 2024-02-09T18:04:17.746 INFO:teuthology.orchestra.run.smithi099.stdout: "erasure_code_profile": "", 2024-02-09T18:04:17.746 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_params": { 2024-02-09T18:04:17.746 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "none" 2024-02-09T18:04:17.746 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.746 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_period": 0, 2024-02-09T18:04:17.746 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_count": 0, 2024-02-09T18:04:17.746 INFO:teuthology.orchestra.run.smithi099.stdout: "use_gmt_hitset": true, 2024-02-09T18:04:17.746 INFO:teuthology.orchestra.run.smithi099.stdout: "min_read_recency_for_promote": 0, 2024-02-09T18:04:17.746 INFO:teuthology.orchestra.run.smithi099.stdout: "min_write_recency_for_promote": 0, 2024-02-09T18:04:17.747 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_grade_decay_rate": 0, 2024-02-09T18:04:17.747 INFO:teuthology.orchestra.run.smithi099.stdout: "hit_set_search_last_n": 0, 2024-02-09T18:04:17.747 INFO:teuthology.orchestra.run.smithi099.stdout: "grade_table": [], 2024-02-09T18:04:17.747 INFO:teuthology.orchestra.run.smithi099.stdout: "stripe_width": 0, 2024-02-09T18:04:17.747 INFO:teuthology.orchestra.run.smithi099.stdout: "expected_num_objects": 0, 2024-02-09T18:04:17.747 INFO:teuthology.orchestra.run.smithi099.stdout: "fast_read": false, 2024-02-09T18:04:17.747 INFO:teuthology.orchestra.run.smithi099.stdout: "options": {}, 2024-02-09T18:04:17.747 INFO:teuthology.orchestra.run.smithi099.stdout: "application_metadata": { 2024-02-09T18:04:17.747 INFO:teuthology.orchestra.run.smithi099.stdout: "rados": {} 2024-02-09T18:04:17.747 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.747 INFO:teuthology.orchestra.run.smithi099.stdout: "read_balance": { 2024-02-09T18:04:17.747 INFO:teuthology.orchestra.run.smithi099.stdout: "score_type": "Fair distribution", 2024-02-09T18:04:17.747 INFO:teuthology.orchestra.run.smithi099.stdout: "score_acting": 4.809999942779541, 2024-02-09T18:04:17.747 INFO:teuthology.orchestra.run.smithi099.stdout: "score_stable": 4.809999942779541, 2024-02-09T18:04:17.747 INFO:teuthology.orchestra.run.smithi099.stdout: "optimal_score": 0.9100000262260437, 2024-02-09T18:04:17.747 INFO:teuthology.orchestra.run.smithi099.stdout: "raw_score_acting": 4.380000114440918, 2024-02-09T18:04:17.748 INFO:teuthology.orchestra.run.smithi099.stdout: "raw_score_stable": 4.380000114440918, 2024-02-09T18:04:17.748 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity_weighted": 0.78079599142074585, 2024-02-09T18:04:17.748 INFO:teuthology.orchestra.run.smithi099.stdout: "average_primary_affinity": 0.85000002384185791, 2024-02-09T18:04:17.748 INFO:teuthology.orchestra.run.smithi099.stdout: "average_primary_affinity_weighted": 0.77999997138977051 2024-02-09T18:04:17.748 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.748 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.748 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T18:04:17.748 INFO:teuthology.orchestra.run.smithi099.stdout: "osds": [ 2024-02-09T18:04:17.748 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.748 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 0, 2024-02-09T18:04:17.748 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "21739aa7-e278-45da-9a3f-867385f01baf", 2024-02-09T18:04:17.748 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T18:04:17.748 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T18:04:17.748 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T18:04:17.748 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 0.609619140625, 2024-02-09T18:04:17.748 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T18:04:17.749 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T18:04:17.749 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 8, 2024-02-09T18:04:17.749 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 938, 2024-02-09T18:04:17.749 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T18:04:17.749 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T18:04:17.749 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T18:04:17.749 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:04:17.749 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.749 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:04:17.749 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6802", 2024-02-09T18:04:17.749 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T18:04:17.749 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.749 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.749 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:04:17.749 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6803", 2024-02-09T18:04:17.749 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T18:04:17.750 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.750 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.750 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.750 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T18:04:17.750 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:04:17.750 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.750 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:04:17.750 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6804", 2024-02-09T18:04:17.750 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T18:04:17.750 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.750 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.750 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:04:17.750 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6805", 2024-02-09T18:04:17.750 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T18:04:17.751 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.751 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.751 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.751 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T18:04:17.751 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:04:17.751 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.751 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:04:17.751 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6808", 2024-02-09T18:04:17.751 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T18:04:17.751 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.751 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.751 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:04:17.751 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6809", 2024-02-09T18:04:17.751 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T18:04:17.751 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.752 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.752 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.752 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T18:04:17.752 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:04:17.752 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.752 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:04:17.752 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6806", 2024-02-09T18:04:17.752 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T18:04:17.752 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.752 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.752 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:04:17.752 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6807", 2024-02-09T18:04:17.752 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3859868923 2024-02-09T18:04:17.752 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.752 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.752 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.753 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.99:6803/3859868923", 2024-02-09T18:04:17.753 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.99:6805/3859868923", 2024-02-09T18:04:17.753 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.99:6809/3859868923", 2024-02-09T18:04:17.753 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.99:6807/3859868923", 2024-02-09T18:04:17.753 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T18:04:17.753 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T18:04:17.753 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T18:04:17.753 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.753 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.753 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.753 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 1, 2024-02-09T18:04:17.753 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "8eab347c-9065-4de1-93ae-e2eb7e11cab7", 2024-02-09T18:04:17.753 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T18:04:17.753 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T18:04:17.753 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T18:04:17.754 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T18:04:17.754 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T18:04:17.754 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T18:04:17.754 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 13, 2024-02-09T18:04:17.754 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 961, 2024-02-09T18:04:17.754 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T18:04:17.754 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T18:04:17.754 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T18:04:17.754 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:04:17.754 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.754 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:04:17.754 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6810", 2024-02-09T18:04:17.754 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T18:04:17.754 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.755 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.755 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:04:17.755 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6811", 2024-02-09T18:04:17.755 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T18:04:17.755 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.755 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.755 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.755 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T18:04:17.755 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:04:17.755 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.755 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:04:17.755 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6812", 2024-02-09T18:04:17.755 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T18:04:17.755 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.755 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.756 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:04:17.756 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6813", 2024-02-09T18:04:17.756 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T18:04:17.756 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.756 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.756 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.756 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T18:04:17.756 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:04:17.756 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.756 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:04:17.756 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6816", 2024-02-09T18:04:17.756 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T18:04:17.756 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.756 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.756 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:04:17.757 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6817", 2024-02-09T18:04:17.757 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T18:04:17.757 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.757 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.757 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.757 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T18:04:17.757 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:04:17.757 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.757 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:04:17.757 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6814", 2024-02-09T18:04:17.757 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T18:04:17.757 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.757 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.757 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:04:17.757 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6815", 2024-02-09T18:04:17.758 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 3868328452 2024-02-09T18:04:17.758 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.758 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.758 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.758 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.99:6811/3868328452", 2024-02-09T18:04:17.758 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.99:6813/3868328452", 2024-02-09T18:04:17.758 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.99:6817/3868328452", 2024-02-09T18:04:17.758 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.99:6815/3868328452", 2024-02-09T18:04:17.758 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T18:04:17.758 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T18:04:17.758 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T18:04:17.758 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.758 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.758 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.759 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 2, 2024-02-09T18:04:17.759 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "575f4257-7e34-485e-963c-2607b69e1220", 2024-02-09T18:04:17.759 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T18:04:17.759 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T18:04:17.759 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T18:04:17.759 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 0.3601837158203125, 2024-02-09T18:04:17.759 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T18:04:17.759 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T18:04:17.759 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 18, 2024-02-09T18:04:17.759 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 945, 2024-02-09T18:04:17.759 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T18:04:17.759 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T18:04:17.759 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T18:04:17.760 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:04:17.760 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.760 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:04:17.760 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6818", 2024-02-09T18:04:17.760 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T18:04:17.760 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.760 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.760 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:04:17.760 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6819", 2024-02-09T18:04:17.760 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T18:04:17.760 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.760 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.760 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.760 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T18:04:17.761 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:04:17.761 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.761 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:04:17.761 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6820", 2024-02-09T18:04:17.761 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T18:04:17.761 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.761 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.761 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:04:17.761 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6821", 2024-02-09T18:04:17.761 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T18:04:17.761 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.761 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.762 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.762 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T18:04:17.762 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:04:17.762 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.762 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:04:17.762 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6824", 2024-02-09T18:04:17.762 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T18:04:17.762 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.762 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.762 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:04:17.762 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6825", 2024-02-09T18:04:17.762 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T18:04:17.763 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.763 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.763 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.763 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T18:04:17.763 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:04:17.763 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.763 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:04:17.763 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6822", 2024-02-09T18:04:17.763 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T18:04:17.763 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.763 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.763 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:04:17.763 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6823", 2024-02-09T18:04:17.764 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 537506237 2024-02-09T18:04:17.764 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.764 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.764 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.764 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.99:6819/537506237", 2024-02-09T18:04:17.764 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.99:6821/537506237", 2024-02-09T18:04:17.764 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.99:6825/537506237", 2024-02-09T18:04:17.764 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.99:6823/537506237", 2024-02-09T18:04:17.764 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T18:04:17.764 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T18:04:17.764 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T18:04:17.764 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.764 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.765 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.765 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 3, 2024-02-09T18:04:17.765 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "1a520bcf-81bb-4f1d-80fb-11e98008e429", 2024-02-09T18:04:17.765 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T18:04:17.765 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T18:04:17.765 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T18:04:17.765 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 0.8014984130859375, 2024-02-09T18:04:17.765 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 27, 2024-02-09T18:04:17.765 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 814, 2024-02-09T18:04:17.765 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 854, 2024-02-09T18:04:17.765 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 950, 2024-02-09T18:04:17.765 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 815, 2024-02-09T18:04:17.765 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T18:04:17.765 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T18:04:17.765 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:04:17.766 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.766 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:04:17.766 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6826", 2024-02-09T18:04:17.766 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4191781533 2024-02-09T18:04:17.766 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.766 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.766 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:04:17.766 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6827", 2024-02-09T18:04:17.766 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4191781533 2024-02-09T18:04:17.766 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.766 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.766 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.766 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T18:04:17.766 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:04:17.767 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.767 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:04:17.767 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6828", 2024-02-09T18:04:17.767 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4191781533 2024-02-09T18:04:17.767 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.767 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.767 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:04:17.767 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6829", 2024-02-09T18:04:17.767 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4191781533 2024-02-09T18:04:17.767 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.767 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.767 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.767 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T18:04:17.767 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:04:17.768 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.768 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:04:17.768 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6832", 2024-02-09T18:04:17.768 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4191781533 2024-02-09T18:04:17.768 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.768 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.768 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:04:17.768 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6833", 2024-02-09T18:04:17.768 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4191781533 2024-02-09T18:04:17.768 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.768 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.768 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.769 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T18:04:17.769 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:04:17.769 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.769 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:04:17.769 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6830", 2024-02-09T18:04:17.769 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4191781533 2024-02-09T18:04:17.769 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.769 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.769 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:04:17.769 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.99:6831", 2024-02-09T18:04:17.769 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4191781533 2024-02-09T18:04:17.769 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.769 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.769 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.769 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.99:6827/4191781533", 2024-02-09T18:04:17.770 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.99:6829/4191781533", 2024-02-09T18:04:17.770 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.99:6833/4191781533", 2024-02-09T18:04:17.770 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.99:6831/4191781533", 2024-02-09T18:04:17.770 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T18:04:17.770 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T18:04:17.770 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T18:04:17.770 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.770 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.770 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.770 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 4, 2024-02-09T18:04:17.770 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "4617c2fe-c9d8-40cd-887b-303e3215bdf8", 2024-02-09T18:04:17.770 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T18:04:17.770 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T18:04:17.770 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T18:04:17.771 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T18:04:17.771 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T18:04:17.771 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T18:04:17.771 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 32, 2024-02-09T18:04:17.771 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 938, 2024-02-09T18:04:17.771 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T18:04:17.771 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T18:04:17.771 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T18:04:17.771 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:04:17.771 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.771 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:04:17.771 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6800", 2024-02-09T18:04:17.771 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T18:04:17.771 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.772 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.772 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:04:17.772 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6801", 2024-02-09T18:04:17.772 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T18:04:17.772 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.772 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.772 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.772 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T18:04:17.772 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:04:17.772 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.772 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:04:17.772 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6802", 2024-02-09T18:04:17.772 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T18:04:17.773 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.773 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.773 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:04:17.773 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6803", 2024-02-09T18:04:17.773 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T18:04:17.773 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.773 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.773 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.773 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T18:04:17.773 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:04:17.773 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.773 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:04:17.773 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6806", 2024-02-09T18:04:17.773 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T18:04:17.774 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.774 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.774 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:04:17.774 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6807", 2024-02-09T18:04:17.774 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T18:04:17.774 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.774 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.774 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.774 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T18:04:17.774 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:04:17.774 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.774 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:04:17.775 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6804", 2024-02-09T18:04:17.775 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T18:04:17.775 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.775 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.775 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:04:17.775 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6805", 2024-02-09T18:04:17.775 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4178676504 2024-02-09T18:04:17.775 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.775 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.775 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.775 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.162:6801/4178676504", 2024-02-09T18:04:17.775 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.162:6803/4178676504", 2024-02-09T18:04:17.775 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.162:6807/4178676504", 2024-02-09T18:04:17.776 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.162:6805/4178676504", 2024-02-09T18:04:17.776 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T18:04:17.776 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T18:04:17.776 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T18:04:17.776 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.776 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.776 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.776 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 5, 2024-02-09T18:04:17.776 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "da98e3f3-173e-46fe-be8b-1ccbe8b0dabc", 2024-02-09T18:04:17.776 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T18:04:17.776 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T18:04:17.776 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T18:04:17.777 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 0, 2024-02-09T18:04:17.777 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T18:04:17.777 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T18:04:17.777 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 40, 2024-02-09T18:04:17.777 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 855, 2024-02-09T18:04:17.777 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T18:04:17.777 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T18:04:17.777 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T18:04:17.777 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:04:17.777 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.777 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:04:17.777 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6808", 2024-02-09T18:04:17.777 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T18:04:17.777 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.777 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.778 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:04:17.778 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6809", 2024-02-09T18:04:17.778 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T18:04:17.778 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.778 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.778 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.778 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T18:04:17.778 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:04:17.778 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.778 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:04:17.778 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6810", 2024-02-09T18:04:17.778 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T18:04:17.778 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.779 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.779 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:04:17.779 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6811", 2024-02-09T18:04:17.779 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T18:04:17.779 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.779 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.779 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.779 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T18:04:17.779 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:04:17.779 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.779 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:04:17.779 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6814", 2024-02-09T18:04:17.779 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T18:04:17.779 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.779 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.780 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:04:17.780 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6815", 2024-02-09T18:04:17.780 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T18:04:17.780 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.780 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.780 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.780 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T18:04:17.780 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:04:17.780 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.780 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:04:17.780 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6812", 2024-02-09T18:04:17.780 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T18:04:17.780 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.780 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.781 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:04:17.781 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6813", 2024-02-09T18:04:17.781 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 1489371339 2024-02-09T18:04:17.781 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.781 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.781 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.781 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.162:6809/1489371339", 2024-02-09T18:04:17.781 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.162:6811/1489371339", 2024-02-09T18:04:17.781 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.162:6815/1489371339", 2024-02-09T18:04:17.781 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.162:6813/1489371339", 2024-02-09T18:04:17.781 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T18:04:17.781 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T18:04:17.781 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T18:04:17.781 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.782 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.782 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.782 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 6, 2024-02-09T18:04:17.782 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "ff50382d-5ddf-4cda-bb0c-ff055feeaa40", 2024-02-09T18:04:17.782 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T18:04:17.782 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T18:04:17.782 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T18:04:17.782 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T18:04:17.782 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T18:04:17.782 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T18:04:17.782 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 49, 2024-02-09T18:04:17.782 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 945, 2024-02-09T18:04:17.782 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T18:04:17.782 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T18:04:17.783 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T18:04:17.783 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:04:17.783 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.783 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:04:17.783 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6816", 2024-02-09T18:04:17.783 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T18:04:17.783 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.783 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.783 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:04:17.783 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6817", 2024-02-09T18:04:17.783 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T18:04:17.783 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.783 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.783 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.783 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T18:04:17.784 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:04:17.784 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.784 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:04:17.784 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6818", 2024-02-09T18:04:17.784 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T18:04:17.784 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.784 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.784 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:04:17.784 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6819", 2024-02-09T18:04:17.784 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T18:04:17.784 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.784 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.785 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.785 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T18:04:17.785 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:04:17.785 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.785 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:04:17.785 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6822", 2024-02-09T18:04:17.785 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T18:04:17.785 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.785 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.785 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:04:17.785 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6823", 2024-02-09T18:04:17.785 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T18:04:17.785 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.785 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.785 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.786 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T18:04:17.786 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:04:17.786 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.786 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:04:17.786 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6820", 2024-02-09T18:04:17.786 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T18:04:17.786 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.786 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.786 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:04:17.786 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6821", 2024-02-09T18:04:17.786 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 821169663 2024-02-09T18:04:17.786 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.786 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.786 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.786 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.162:6817/821169663", 2024-02-09T18:04:17.787 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.162:6819/821169663", 2024-02-09T18:04:17.787 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.162:6823/821169663", 2024-02-09T18:04:17.787 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.162:6821/821169663", 2024-02-09T18:04:17.787 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T18:04:17.787 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T18:04:17.787 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T18:04:17.787 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.787 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.787 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.787 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 7, 2024-02-09T18:04:17.787 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "a82984dc-5e22-424b-a446-ae2ee71eee19", 2024-02-09T18:04:17.787 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T18:04:17.787 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T18:04:17.787 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T18:04:17.787 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 0.5982513427734375, 2024-02-09T18:04:17.787 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T18:04:17.788 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T18:04:17.788 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 57, 2024-02-09T18:04:17.788 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 945, 2024-02-09T18:04:17.788 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T18:04:17.788 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T18:04:17.788 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T18:04:17.788 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:04:17.788 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.788 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:04:17.788 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6824", 2024-02-09T18:04:17.788 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T18:04:17.788 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.788 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.788 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:04:17.788 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6825", 2024-02-09T18:04:17.789 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T18:04:17.789 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.789 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.789 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.789 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T18:04:17.789 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:04:17.789 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.789 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:04:17.789 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6826", 2024-02-09T18:04:17.789 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T18:04:17.789 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.789 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.789 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:04:17.789 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6827", 2024-02-09T18:04:17.789 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T18:04:17.789 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.790 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.790 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.790 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T18:04:17.790 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:04:17.790 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.790 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:04:17.790 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6830", 2024-02-09T18:04:17.790 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T18:04:17.790 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.790 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.790 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:04:17.790 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6831", 2024-02-09T18:04:17.790 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T18:04:17.790 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.790 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.791 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.791 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T18:04:17.791 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:04:17.791 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.791 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:04:17.791 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6828", 2024-02-09T18:04:17.791 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T18:04:17.791 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.791 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.791 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:04:17.791 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.162:6829", 2024-02-09T18:04:17.791 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 965264369 2024-02-09T18:04:17.791 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.792 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.792 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.792 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.162:6825/965264369", 2024-02-09T18:04:17.792 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.162:6827/965264369", 2024-02-09T18:04:17.792 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.162:6831/965264369", 2024-02-09T18:04:17.792 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.162:6829/965264369", 2024-02-09T18:04:17.792 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T18:04:17.792 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T18:04:17.792 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T18:04:17.792 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.792 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.792 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.792 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 8, 2024-02-09T18:04:17.792 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "63f9d79f-0400-44dc-b388-df00fefb5820", 2024-02-09T18:04:17.793 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T18:04:17.793 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T18:04:17.793 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T18:04:17.793 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T18:04:17.793 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 66, 2024-02-09T18:04:17.793 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 145, 2024-02-09T18:04:17.793 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 171, 2024-02-09T18:04:17.793 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 955, 2024-02-09T18:04:17.793 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 146, 2024-02-09T18:04:17.793 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T18:04:17.793 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T18:04:17.793 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:04:17.793 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.793 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:04:17.794 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6800", 2024-02-09T18:04:17.794 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T18:04:17.794 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.794 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.794 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:04:17.794 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6801", 2024-02-09T18:04:17.794 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T18:04:17.794 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.794 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.794 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.794 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T18:04:17.794 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:04:17.794 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.794 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:04:17.795 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6802", 2024-02-09T18:04:17.795 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T18:04:17.795 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.795 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.795 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:04:17.795 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6803", 2024-02-09T18:04:17.795 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T18:04:17.795 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.795 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.795 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.795 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T18:04:17.795 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:04:17.795 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.795 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:04:17.796 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6806", 2024-02-09T18:04:17.796 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T18:04:17.796 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.796 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.796 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:04:17.796 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6807", 2024-02-09T18:04:17.796 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T18:04:17.796 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.796 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.796 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.796 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T18:04:17.796 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:04:17.796 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.796 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:04:17.797 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6804", 2024-02-09T18:04:17.797 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T18:04:17.797 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.797 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.797 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:04:17.797 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6805", 2024-02-09T18:04:17.797 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2558294495 2024-02-09T18:04:17.797 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.797 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.797 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.797 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.179:6801/2558294495", 2024-02-09T18:04:17.797 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.179:6803/2558294495", 2024-02-09T18:04:17.797 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.179:6807/2558294495", 2024-02-09T18:04:17.798 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.179:6805/2558294495", 2024-02-09T18:04:17.798 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T18:04:17.798 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T18:04:17.798 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T18:04:17.798 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.798 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.798 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.798 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 9, 2024-02-09T18:04:17.798 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "7d995854-7328-4dfc-b1a9-0edca8d74f46", 2024-02-09T18:04:17.798 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T18:04:17.798 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T18:04:17.798 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T18:04:17.798 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T18:04:17.799 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T18:04:17.799 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T18:04:17.799 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 77, 2024-02-09T18:04:17.799 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 945, 2024-02-09T18:04:17.799 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T18:04:17.799 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T18:04:17.799 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T18:04:17.799 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:04:17.799 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.799 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:04:17.799 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6808", 2024-02-09T18:04:17.799 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T18:04:17.799 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.799 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.800 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:04:17.800 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6809", 2024-02-09T18:04:17.800 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T18:04:17.800 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.800 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.800 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.800 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T18:04:17.800 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:04:17.800 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.800 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:04:17.800 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6810", 2024-02-09T18:04:17.800 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T18:04:17.800 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.801 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.801 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:04:17.801 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6811", 2024-02-09T18:04:17.801 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T18:04:17.801 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.801 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.801 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.801 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T18:04:17.801 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:04:17.801 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.801 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:04:17.801 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6814", 2024-02-09T18:04:17.801 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T18:04:17.802 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.802 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.802 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:04:17.802 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6815", 2024-02-09T18:04:17.802 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T18:04:17.802 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.802 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.802 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.802 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T18:04:17.802 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:04:17.802 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.802 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:04:17.802 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6812", 2024-02-09T18:04:17.802 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T18:04:17.802 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.802 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.803 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:04:17.803 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6813", 2024-02-09T18:04:17.803 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 504427183 2024-02-09T18:04:17.803 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.803 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.803 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.803 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.179:6809/504427183", 2024-02-09T18:04:17.803 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.179:6811/504427183", 2024-02-09T18:04:17.803 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.179:6815/504427183", 2024-02-09T18:04:17.803 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.179:6813/504427183", 2024-02-09T18:04:17.803 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T18:04:17.803 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T18:04:17.803 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T18:04:17.804 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.804 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.804 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.804 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 10, 2024-02-09T18:04:17.804 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "151fbedd-08e6-46e6-a4e5-a14248399f61", 2024-02-09T18:04:17.804 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T18:04:17.804 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 0, 2024-02-09T18:04:17.804 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 0, 2024-02-09T18:04:17.804 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T18:04:17.804 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T18:04:17.804 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T18:04:17.804 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 84, 2024-02-09T18:04:17.804 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 882, 2024-02-09T18:04:17.805 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T18:04:17.805 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T18:04:17.805 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T18:04:17.805 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:04:17.805 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.805 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:04:17.805 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6816", 2024-02-09T18:04:17.805 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T18:04:17.805 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.805 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.805 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:04:17.806 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6817", 2024-02-09T18:04:17.806 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T18:04:17.806 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.806 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.806 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.806 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T18:04:17.806 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:04:17.806 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.806 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:04:17.806 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6818", 2024-02-09T18:04:17.806 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T18:04:17.806 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.807 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.807 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:04:17.807 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6819", 2024-02-09T18:04:17.807 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T18:04:17.807 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.807 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.807 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.807 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T18:04:17.807 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:04:17.807 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.807 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:04:17.807 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6822", 2024-02-09T18:04:17.807 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T18:04:17.808 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.808 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.808 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:04:17.808 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6823", 2024-02-09T18:04:17.808 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T18:04:17.808 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.808 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.808 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.808 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T18:04:17.808 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:04:17.808 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.811 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:04:17.811 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6820", 2024-02-09T18:04:17.811 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T18:04:17.811 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.811 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.811 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:04:17.811 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6821", 2024-02-09T18:04:17.811 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2403345164 2024-02-09T18:04:17.811 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.811 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.812 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.812 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.179:6817/2403345164", 2024-02-09T18:04:17.812 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.179:6819/2403345164", 2024-02-09T18:04:17.812 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.179:6823/2403345164", 2024-02-09T18:04:17.812 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.179:6821/2403345164", 2024-02-09T18:04:17.812 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T18:04:17.812 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T18:04:17.812 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T18:04:17.812 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.812 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.812 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.812 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 11, 2024-02-09T18:04:17.812 INFO:teuthology.orchestra.run.smithi099.stdout: "uuid": "6d3a3ad8-417a-483f-991c-52549dbe7d73", 2024-02-09T18:04:17.813 INFO:teuthology.orchestra.run.smithi099.stdout: "up": 1, 2024-02-09T18:04:17.813 INFO:teuthology.orchestra.run.smithi099.stdout: "in": 1, 2024-02-09T18:04:17.813 INFO:teuthology.orchestra.run.smithi099.stdout: "weight": 1, 2024-02-09T18:04:17.813 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_affinity": 1, 2024-02-09T18:04:17.813 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_begin": 0, 2024-02-09T18:04:17.813 INFO:teuthology.orchestra.run.smithi099.stdout: "last_clean_end": 0, 2024-02-09T18:04:17.813 INFO:teuthology.orchestra.run.smithi099.stdout: "up_from": 90, 2024-02-09T18:04:17.813 INFO:teuthology.orchestra.run.smithi099.stdout: "up_thru": 961, 2024-02-09T18:04:17.813 INFO:teuthology.orchestra.run.smithi099.stdout: "down_at": 0, 2024-02-09T18:04:17.813 INFO:teuthology.orchestra.run.smithi099.stdout: "lost_at": 0, 2024-02-09T18:04:17.813 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addrs": { 2024-02-09T18:04:17.814 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:04:17.814 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.814 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:04:17.814 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6824", 2024-02-09T18:04:17.814 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T18:04:17.814 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.814 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.814 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:04:17.814 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6825", 2024-02-09T18:04:17.814 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T18:04:17.814 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.814 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.814 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.815 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addrs": { 2024-02-09T18:04:17.815 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:04:17.815 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.815 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:04:17.815 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6826", 2024-02-09T18:04:17.815 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T18:04:17.815 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.815 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.815 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:04:17.815 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6827", 2024-02-09T18:04:17.815 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T18:04:17.815 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.815 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.816 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.816 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addrs": { 2024-02-09T18:04:17.816 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:04:17.816 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.816 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:04:17.816 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6830", 2024-02-09T18:04:17.816 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T18:04:17.816 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.816 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.816 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:04:17.816 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6831", 2024-02-09T18:04:17.816 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T18:04:17.816 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.816 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.817 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.817 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addrs": { 2024-02-09T18:04:17.817 INFO:teuthology.orchestra.run.smithi099.stdout: "addrvec": [ 2024-02-09T18:04:17.817 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.817 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v2", 2024-02-09T18:04:17.817 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6828", 2024-02-09T18:04:17.817 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T18:04:17.817 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.817 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.817 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-02-09T18:04:17.817 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "172.21.15.179:6829", 2024-02-09T18:04:17.817 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 2899402373 2024-02-09T18:04:17.817 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.817 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.818 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.818 INFO:teuthology.orchestra.run.smithi099.stdout: "public_addr": "172.21.15.179:6825/2899402373", 2024-02-09T18:04:17.818 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_addr": "172.21.15.179:6827/2899402373", 2024-02-09T18:04:17.818 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_back_addr": "172.21.15.179:6831/2899402373", 2024-02-09T18:04:17.818 INFO:teuthology.orchestra.run.smithi099.stdout: "heartbeat_front_addr": "172.21.15.179:6829/2899402373", 2024-02-09T18:04:17.818 INFO:teuthology.orchestra.run.smithi099.stdout: "state": [ 2024-02-09T18:04:17.818 INFO:teuthology.orchestra.run.smithi099.stdout: "exists", 2024-02-09T18:04:17.818 INFO:teuthology.orchestra.run.smithi099.stdout: "up" 2024-02-09T18:04:17.818 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.818 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.818 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T18:04:17.818 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_xinfo": [ 2024-02-09T18:04:17.818 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.818 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 0, 2024-02-09T18:04:17.819 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T18:04:17.819 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T18:04:17.819 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T18:04:17.819 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T18:04:17.819 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T18:04:17.819 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T18:03:25.851815+0000", 2024-02-09T18:04:17.819 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T18:04:17.819 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.819 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.819 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 1, 2024-02-09T18:04:17.819 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T18:04:17.819 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T18:04:17.819 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T18:04:17.820 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T18:04:17.820 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T18:04:17.820 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T18:03:07.352057+0000", 2024-02-09T18:04:17.820 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T18:04:17.820 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.820 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.820 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 2, 2024-02-09T18:04:17.820 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T18:04:17.820 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T18:04:17.820 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T18:04:17.820 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T18:04:17.820 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T18:04:17.821 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T18:03:53.750446+0000", 2024-02-09T18:04:17.821 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T18:04:17.821 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.821 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.821 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 3, 2024-02-09T18:04:17.821 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T18:04:17.821 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T18:04:17.821 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T18:04:17.821 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T18:04:17.821 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T18:04:17.821 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T18:03:06.473727+0000", 2024-02-09T18:04:17.821 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 814 2024-02-09T18:04:17.821 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.822 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.822 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 4, 2024-02-09T18:04:17.822 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T18:04:17.822 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T18:04:17.822 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T18:04:17.822 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T18:04:17.822 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T18:04:17.822 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T18:03:28.714945+0000", 2024-02-09T18:04:17.822 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T18:04:17.822 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.822 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.822 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 5, 2024-02-09T18:04:17.822 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T18:04:17.823 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T18:04:17.823 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T18:04:17.823 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T18:04:17.823 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T18:04:17.823 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T18:03:37.910096+0000", 2024-02-09T18:04:17.823 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T18:04:17.823 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.823 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.823 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 6, 2024-02-09T18:04:17.823 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T18:04:17.823 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T18:04:17.823 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T18:04:17.823 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T18:04:17.823 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T18:04:17.824 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T18:03:05.756967+0000", 2024-02-09T18:04:17.824 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T18:04:17.824 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.824 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.824 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 7, 2024-02-09T18:04:17.824 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T18:04:17.824 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T18:04:17.824 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T18:04:17.824 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T18:04:17.824 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T18:04:17.824 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T18:04:15.618566+0000", 2024-02-09T18:04:17.824 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T18:04:17.824 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.824 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.825 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 8, 2024-02-09T18:04:17.825 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T18:04:17.825 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T18:04:17.825 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T18:04:17.825 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T18:04:17.825 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T18:04:17.825 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T18:03:50.783888+0000", 2024-02-09T18:04:17.825 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 145 2024-02-09T18:04:17.825 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.825 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.825 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 9, 2024-02-09T18:04:17.825 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T18:04:17.825 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T18:04:17.825 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T18:04:17.826 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T18:04:17.826 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T18:04:17.826 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T18:04:13.878100+0000", 2024-02-09T18:04:17.826 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T18:04:17.826 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.826 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.826 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 10, 2024-02-09T18:04:17.826 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T18:04:17.826 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T18:04:17.826 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T18:04:17.826 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T18:04:17.826 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 65536, 2024-02-09T18:04:17.826 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T18:03:21.800213+0000", 2024-02-09T18:04:17.826 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T18:04:17.827 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.827 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.827 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": 11, 2024-02-09T18:04:17.827 INFO:teuthology.orchestra.run.smithi099.stdout: "down_stamp": "0.000000", 2024-02-09T18:04:17.827 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_probability": 0, 2024-02-09T18:04:17.827 INFO:teuthology.orchestra.run.smithi099.stdout: "laggy_interval": 0, 2024-02-09T18:04:17.827 INFO:teuthology.orchestra.run.smithi099.stdout: "features": 4540701272860131327, 2024-02-09T18:04:17.827 INFO:teuthology.orchestra.run.smithi099.stdout: "old_weight": 0, 2024-02-09T18:04:17.827 INFO:teuthology.orchestra.run.smithi099.stdout: "last_purged_snaps_scrub": "2024-02-09T18:03:40.705030+0000", 2024-02-09T18:04:17.827 INFO:teuthology.orchestra.run.smithi099.stdout: "dead_epoch": 0 2024-02-09T18:04:17.827 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.827 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T18:04:17.828 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_upmap": [ 2024-02-09T18:04:17.828 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.828 INFO:teuthology.orchestra.run.smithi099.stdout: "pgid": "1.7", 2024-02-09T18:04:17.828 INFO:teuthology.orchestra.run.smithi099.stdout: "osds": [ 2024-02-09T18:04:17.828 INFO:teuthology.orchestra.run.smithi099.stdout: 1, 2024-02-09T18:04:17.828 INFO:teuthology.orchestra.run.smithi099.stdout: 8, 2024-02-09T18:04:17.828 INFO:teuthology.orchestra.run.smithi099.stdout: 6 2024-02-09T18:04:17.828 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.828 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.828 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T18:04:17.828 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_upmap_items": [ 2024-02-09T18:04:17.828 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.829 INFO:teuthology.orchestra.run.smithi099.stdout: "pgid": "1.5", 2024-02-09T18:04:17.829 INFO:teuthology.orchestra.run.smithi099.stdout: "mappings": [ 2024-02-09T18:04:17.829 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.829 INFO:teuthology.orchestra.run.smithi099.stdout: "from": 6, 2024-02-09T18:04:17.829 INFO:teuthology.orchestra.run.smithi099.stdout: "to": 8 2024-02-09T18:04:17.829 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.829 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.829 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.829 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T18:04:17.829 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_upmap_primaries": [], 2024-02-09T18:04:17.829 INFO:teuthology.orchestra.run.smithi099.stdout: "pg_temp": [ 2024-02-09T18:04:17.829 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:04:17.829 INFO:teuthology.orchestra.run.smithi099.stdout: "pgid": "9.7", 2024-02-09T18:04:17.829 INFO:teuthology.orchestra.run.smithi099.stdout: "osds": [ 2024-02-09T18:04:17.830 INFO:teuthology.orchestra.run.smithi099.stdout: 3, 2024-02-09T18:04:17.830 INFO:teuthology.orchestra.run.smithi099.stdout: 7, 2024-02-09T18:04:17.830 INFO:teuthology.orchestra.run.smithi099.stdout: 10 2024-02-09T18:04:17.830 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:04:17.830 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.830 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2024-02-09T18:04:17.830 INFO:teuthology.orchestra.run.smithi099.stdout: "primary_temp": [], 2024-02-09T18:04:17.830 INFO:teuthology.orchestra.run.smithi099.stdout: "blocklist": { 2024-02-09T18:04:17.830 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:0/3904172782": "2024-02-10T17:16:24.229365+0000", 2024-02-09T18:04:17.830 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:6801/1586034576": "2024-02-10T17:16:24.229365+0000", 2024-02-09T18:04:17.830 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:6800/746248609": "2024-02-10T17:15:54.399201+0000", 2024-02-09T18:04:17.830 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:0/4092784197": "2024-02-10T17:16:24.229365+0000", 2024-02-09T18:04:17.830 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:0/2844598101": "2024-02-10T17:15:54.399201+0000", 2024-02-09T18:04:17.831 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:6801/746248609": "2024-02-10T17:15:54.399201+0000", 2024-02-09T18:04:17.831 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:0/2528750050": "2024-02-10T17:15:54.399201+0000", 2024-02-09T18:04:17.831 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:6800/1586034576": "2024-02-10T17:16:24.229365+0000", 2024-02-09T18:04:17.831 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:0/2214140078": "2024-02-10T17:15:54.399201+0000", 2024-02-09T18:04:17.831 INFO:teuthology.orchestra.run.smithi099.stdout: "172.21.15.99:0/4217821965": "2024-02-10T17:16:24.229365+0000" 2024-02-09T18:04:17.831 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.831 INFO:teuthology.orchestra.run.smithi099.stdout: "range_blocklist": {}, 2024-02-09T18:04:17.831 INFO:teuthology.orchestra.run.smithi099.stdout: "erasure_code_profiles": { 2024-02-09T18:04:17.831 INFO:teuthology.orchestra.run.smithi099.stdout: "default": { 2024-02-09T18:04:17.831 INFO:teuthology.orchestra.run.smithi099.stdout: "crush-failure-domain": "osd", 2024-02-09T18:04:17.831 INFO:teuthology.orchestra.run.smithi099.stdout: "k": "2", 2024-02-09T18:04:17.831 INFO:teuthology.orchestra.run.smithi099.stdout: "m": "1", 2024-02-09T18:04:17.831 INFO:teuthology.orchestra.run.smithi099.stdout: "plugin": "jerasure", 2024-02-09T18:04:17.831 INFO:teuthology.orchestra.run.smithi099.stdout: "technique": "reed_sol_van" 2024-02-09T18:04:17.831 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.832 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:04:17.832 INFO:teuthology.orchestra.run.smithi099.stdout: "removed_snaps_queue": [], 2024-02-09T18:04:17.832 INFO:teuthology.orchestra.run.smithi099.stdout: "new_removed_snaps": [], 2024-02-09T18:04:17.832 INFO:teuthology.orchestra.run.smithi099.stdout: "new_purged_snaps": [], 2024-02-09T18:04:17.832 INFO:teuthology.orchestra.run.smithi099.stdout: "crush_node_flags": {}, 2024-02-09T18:04:17.832 INFO:teuthology.orchestra.run.smithi099.stdout: "device_class_flags": {}, 2024-02-09T18:04:17.832 INFO:teuthology.orchestra.run.smithi099.stdout: "stretch_mode": { 2024-02-09T18:04:17.832 INFO:teuthology.orchestra.run.smithi099.stdout: "stretch_mode_enabled": false, 2024-02-09T18:04:17.832 INFO:teuthology.orchestra.run.smithi099.stdout: "stretch_bucket_count": 0, 2024-02-09T18:04:17.832 INFO:teuthology.orchestra.run.smithi099.stdout: "degraded_stretch_mode": 0, 2024-02-09T18:04:17.832 INFO:teuthology.orchestra.run.smithi099.stdout: "recovering_stretch_mode": 0, 2024-02-09T18:04:17.832 INFO:teuthology.orchestra.run.smithi099.stdout: "stretch_mode_bucket": 0 2024-02-09T18:04:17.833 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:04:17.833 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T18:04:17.833 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T18:04:17.868 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:17 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:04:17.868 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:17 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:04:17.868 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:17 smithi162 ceph-mon[44304]: 9.b scrub starts 2024-02-09T18:04:17.868 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:17 smithi162 ceph-mon[44304]: 9.b scrub ok 2024-02-09T18:04:17.868 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:17 smithi162 ceph-mon[44304]: 1.18 scrub starts 2024-02-09T18:04:17.868 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:17 smithi162 ceph-mon[44304]: 1.18 scrub ok 2024-02-09T18:04:17.868 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:17 smithi162 ceph-mon[44304]: 1.6 scrub starts 2024-02-09T18:04:17.868 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:17 smithi162 ceph-mon[44304]: 1.6 scrub ok 2024-02-09T18:04:17.868 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:17 smithi162 ceph-mon[44304]: osdmap e969: 12 total, 12 up, 11 in 2024-02-09T18:04:17.869 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:17 smithi162 ceph-mon[44304]: 9.0 scrub ok 2024-02-09T18:04:17.869 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:17 smithi162 ceph-mon[44304]: 9.6 scrub starts 2024-02-09T18:04:17.869 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:17.906 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T18:04:17.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:17.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T18:04:17.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:17 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:04:17.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:17 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:04:17.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:17 smithi179 ceph-mon[43975]: 9.b scrub starts 2024-02-09T18:04:17.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:17 smithi179 ceph-mon[43975]: 9.b scrub ok 2024-02-09T18:04:17.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:17 smithi179 ceph-mon[43975]: 1.18 scrub starts 2024-02-09T18:04:17.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:17 smithi179 ceph-mon[43975]: 1.18 scrub ok 2024-02-09T18:04:17.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:17 smithi179 ceph-mon[43975]: 1.6 scrub starts 2024-02-09T18:04:17.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:17 smithi179 ceph-mon[43975]: 1.6 scrub ok 2024-02-09T18:04:17.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:17 smithi179 ceph-mon[43975]: osdmap e969: 12 total, 12 up, 11 in 2024-02-09T18:04:17.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:17 smithi179 ceph-mon[43975]: 9.0 scrub ok 2024-02-09T18:04:17.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:17 smithi179 ceph-mon[43975]: 9.6 scrub starts 2024-02-09T18:04:18.015 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T18:04:18.059 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:18.084 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T18:04:18.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:18.088 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T18:04:18.189 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:04:18.236 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:18.701 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:04:18.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:18.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:04:18.766 INFO:tasks.thrashosds.thrasher:j is {'epoch': 969, 'fsid': 'a69c870a-c76e-11ee-95b7-87774f69a715', 'created': '2024-02-09T17:15:26.113606+0000', 'modified': '2024-02-09T18:04:16.483760+0000', 'last_up_change': '2024-02-09T17:59:11.889493+0000', 'last_in_change': '2024-02-09T18:03:13.429500+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 28, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-09T17:18:56.552286+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '544', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '514', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.119999885559082, 'score_stable': 2.130000114440918, 'optimal_score': 1, 'raw_score_acting': 7.119999885559082, 'raw_score_stable': 2.130000114440918, 'primary_affinity_weighted': 0.7807959914207458, 'average_primary_affinity': 0.8500000238418579, 'average_primary_affinity_weighted': 0.7799999713897705, 'error_message': 'pool 1 has acting primaries on OSD(s) with primary affinity 0, read balance score is not accurate'}}, {'pool': 9, 'pool_name': 'unique_pool_7', 'create_time': '2024-02-09T18:00:06.232818+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '886', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 4.809999942779541, 'score_stable': 4.809999942779541, 'optimal_score': 0.9100000262260437, 'raw_score_acting': 4.380000114440918, 'raw_score_stable': 4.380000114440918, 'primary_affinity_weighted': 0.7807959914207458, 'average_primary_affinity': 0.8500000238418579, 'average_primary_affinity_weighted': 0.7799999713897705}}], 'osds': [{'osd': 0, 'uuid': '21739aa7-e278-45da-9a3f-867385f01baf', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.609619140625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 938, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6802', 'nonce': 3859868923}, {'type': 'v1', 'addr': '172.21.15.99:6803', 'nonce': 3859868923}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6804', 'nonce': 3859868923}, {'type': 'v1', 'addr': '172.21.15.99:6805', 'nonce': 3859868923}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6808', 'nonce': 3859868923}, {'type': 'v1', 'addr': '172.21.15.99:6809', 'nonce': 3859868923}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6806', 'nonce': 3859868923}, {'type': 'v1', 'addr': '172.21.15.99:6807', 'nonce': 3859868923}]}, 'public_addr': '172.21.15.99:6803/3859868923', 'cluster_addr': '172.21.15.99:6805/3859868923', 'heartbeat_back_addr': '172.21.15.99:6809/3859868923', 'heartbeat_front_addr': '172.21.15.99:6807/3859868923', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '8eab347c-9065-4de1-93ae-e2eb7e11cab7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 961, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6810', 'nonce': 3868328452}, {'type': 'v1', 'addr': '172.21.15.99:6811', 'nonce': 3868328452}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6812', 'nonce': 3868328452}, {'type': 'v1', 'addr': '172.21.15.99:6813', 'nonce': 3868328452}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6816', 'nonce': 3868328452}, {'type': 'v1', 'addr': '172.21.15.99:6817', 'nonce': 3868328452}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6814', 'nonce': 3868328452}, {'type': 'v1', 'addr': '172.21.15.99:6815', 'nonce': 3868328452}]}, 'public_addr': '172.21.15.99:6811/3868328452', 'cluster_addr': '172.21.15.99:6813/3868328452', 'heartbeat_back_addr': '172.21.15.99:6817/3868328452', 'heartbeat_front_addr': '172.21.15.99:6815/3868328452', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '575f4257-7e34-485e-963c-2607b69e1220', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.3601837158203125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 945, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6818', 'nonce': 537506237}, {'type': 'v1', 'addr': '172.21.15.99:6819', 'nonce': 537506237}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6820', 'nonce': 537506237}, {'type': 'v1', 'addr': '172.21.15.99:6821', 'nonce': 537506237}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6824', 'nonce': 537506237}, {'type': 'v1', 'addr': '172.21.15.99:6825', 'nonce': 537506237}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6822', 'nonce': 537506237}, {'type': 'v1', 'addr': '172.21.15.99:6823', 'nonce': 537506237}]}, 'public_addr': '172.21.15.99:6819/537506237', 'cluster_addr': '172.21.15.99:6821/537506237', 'heartbeat_back_addr': '172.21.15.99:6825/537506237', 'heartbeat_front_addr': '172.21.15.99:6823/537506237', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '1a520bcf-81bb-4f1d-80fb-11e98008e429', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.8014984130859375, 'last_clean_begin': 27, 'last_clean_end': 814, 'up_from': 854, 'up_thru': 950, 'down_at': 815, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6826', 'nonce': 4191781533}, {'type': 'v1', 'addr': '172.21.15.99:6827', 'nonce': 4191781533}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6828', 'nonce': 4191781533}, {'type': 'v1', 'addr': '172.21.15.99:6829', 'nonce': 4191781533}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6832', 'nonce': 4191781533}, {'type': 'v1', 'addr': '172.21.15.99:6833', 'nonce': 4191781533}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6830', 'nonce': 4191781533}, {'type': 'v1', 'addr': '172.21.15.99:6831', 'nonce': 4191781533}]}, 'public_addr': '172.21.15.99:6827/4191781533', 'cluster_addr': '172.21.15.99:6829/4191781533', 'heartbeat_back_addr': '172.21.15.99:6833/4191781533', 'heartbeat_front_addr': '172.21.15.99:6831/4191781533', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '4617c2fe-c9d8-40cd-887b-303e3215bdf8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 938, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6800', 'nonce': 4178676504}, {'type': 'v1', 'addr': '172.21.15.162:6801', 'nonce': 4178676504}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6802', 'nonce': 4178676504}, {'type': 'v1', 'addr': '172.21.15.162:6803', 'nonce': 4178676504}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6806', 'nonce': 4178676504}, {'type': 'v1', 'addr': '172.21.15.162:6807', 'nonce': 4178676504}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6804', 'nonce': 4178676504}, {'type': 'v1', 'addr': '172.21.15.162:6805', 'nonce': 4178676504}]}, 'public_addr': '172.21.15.162:6801/4178676504', 'cluster_addr': '172.21.15.162:6803/4178676504', 'heartbeat_back_addr': '172.21.15.162:6807/4178676504', 'heartbeat_front_addr': '172.21.15.162:6805/4178676504', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'da98e3f3-173e-46fe-be8b-1ccbe8b0dabc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 855, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6808', 'nonce': 1489371339}, {'type': 'v1', 'addr': '172.21.15.162:6809', 'nonce': 1489371339}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6810', 'nonce': 1489371339}, {'type': 'v1', 'addr': '172.21.15.162:6811', 'nonce': 1489371339}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6814', 'nonce': 1489371339}, {'type': 'v1', 'addr': '172.21.15.162:6815', 'nonce': 1489371339}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6812', 'nonce': 1489371339}, {'type': 'v1', 'addr': '172.21.15.162:6813', 'nonce': 1489371339}]}, 'public_addr': '172.21.15.162:6809/1489371339', 'cluster_addr': '172.21.15.162:6811/1489371339', 'heartbeat_back_addr': '172.21.15.162:6815/1489371339', 'heartbeat_front_addr': '172.21.15.162:6813/1489371339', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'ff50382d-5ddf-4cda-bb0c-ff055feeaa40', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 49, 'up_thru': 945, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6816', 'nonce': 821169663}, {'type': 'v1', 'addr': '172.21.15.162:6817', 'nonce': 821169663}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6818', 'nonce': 821169663}, {'type': 'v1', 'addr': '172.21.15.162:6819', 'nonce': 821169663}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6822', 'nonce': 821169663}, {'type': 'v1', 'addr': '172.21.15.162:6823', 'nonce': 821169663}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6820', 'nonce': 821169663}, {'type': 'v1', 'addr': '172.21.15.162:6821', 'nonce': 821169663}]}, 'public_addr': '172.21.15.162:6817/821169663', 'cluster_addr': '172.21.15.162:6819/821169663', 'heartbeat_back_addr': '172.21.15.162:6823/821169663', 'heartbeat_front_addr': '172.21.15.162:6821/821169663', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'a82984dc-5e22-424b-a446-ae2ee71eee19', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.5982513427734375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 57, 'up_thru': 945, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6824', 'nonce': 965264369}, {'type': 'v1', 'addr': '172.21.15.162:6825', 'nonce': 965264369}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6826', 'nonce': 965264369}, {'type': 'v1', 'addr': '172.21.15.162:6827', 'nonce': 965264369}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6830', 'nonce': 965264369}, {'type': 'v1', 'addr': '172.21.15.162:6831', 'nonce': 965264369}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6828', 'nonce': 965264369}, {'type': 'v1', 'addr': '172.21.15.162:6829', 'nonce': 965264369}]}, 'public_addr': '172.21.15.162:6825/965264369', 'cluster_addr': '172.21.15.162:6827/965264369', 'heartbeat_back_addr': '172.21.15.162:6831/965264369', 'heartbeat_front_addr': '172.21.15.162:6829/965264369', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '63f9d79f-0400-44dc-b388-df00fefb5820', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 66, 'last_clean_end': 145, 'up_from': 171, 'up_thru': 955, 'down_at': 146, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6800', 'nonce': 2558294495}, {'type': 'v1', 'addr': '172.21.15.179:6801', 'nonce': 2558294495}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6802', 'nonce': 2558294495}, {'type': 'v1', 'addr': '172.21.15.179:6803', 'nonce': 2558294495}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6806', 'nonce': 2558294495}, {'type': 'v1', 'addr': '172.21.15.179:6807', 'nonce': 2558294495}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6804', 'nonce': 2558294495}, {'type': 'v1', 'addr': '172.21.15.179:6805', 'nonce': 2558294495}]}, 'public_addr': '172.21.15.179:6801/2558294495', 'cluster_addr': '172.21.15.179:6803/2558294495', 'heartbeat_back_addr': '172.21.15.179:6807/2558294495', 'heartbeat_front_addr': '172.21.15.179:6805/2558294495', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '7d995854-7328-4dfc-b1a9-0edca8d74f46', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 77, 'up_thru': 945, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6808', 'nonce': 504427183}, {'type': 'v1', 'addr': '172.21.15.179:6809', 'nonce': 504427183}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6810', 'nonce': 504427183}, {'type': 'v1', 'addr': '172.21.15.179:6811', 'nonce': 504427183}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6814', 'nonce': 504427183}, {'type': 'v1', 'addr': '172.21.15.179:6815', 'nonce': 504427183}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6812', 'nonce': 504427183}, {'type': 'v1', 'addr': '172.21.15.179:6813', 'nonce': 504427183}]}, 'public_addr': '172.21.15.179:6809/504427183', 'cluster_addr': '172.21.15.179:6811/504427183', 'heartbeat_back_addr': '172.21.15.179:6815/504427183', 'heartbeat_front_addr': '172.21.15.179:6813/504427183', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '151fbedd-08e6-46e6-a4e5-a14248399f61', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 84, 'up_thru': 882, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6816', 'nonce': 2403345164}, {'type': 'v1', 'addr': '172.21.15.179:6817', 'nonce': 2403345164}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6818', 'nonce': 2403345164}, {'type': 'v1', 'addr': '172.21.15.179:6819', 'nonce': 2403345164}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6822', 'nonce': 2403345164}, {'type': 'v1', 'addr': '172.21.15.179:6823', 'nonce': 2403345164}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6820', 'nonce': 2403345164}, {'type': 'v1', 'addr': '172.21.15.179:6821', 'nonce': 2403345164}]}, 'public_addr': '172.21.15.179:6817/2403345164', 'cluster_addr': '172.21.15.179:6819/2403345164', 'heartbeat_back_addr': '172.21.15.179:6823/2403345164', 'heartbeat_front_addr': '172.21.15.179:6821/2403345164', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '6d3a3ad8-417a-483f-991c-52549dbe7d73', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 90, 'up_thru': 961, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6824', 'nonce': 2899402373}, {'type': 'v1', 'addr': '172.21.15.179:6825', 'nonce': 2899402373}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6826', 'nonce': 2899402373}, {'type': 'v1', 'addr': '172.21.15.179:6827', 'nonce': 2899402373}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6830', 'nonce': 2899402373}, {'type': 'v1', 'addr': '172.21.15.179:6831', 'nonce': 2899402373}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6828', 'nonce': 2899402373}, {'type': 'v1', 'addr': '172.21.15.179:6829', 'nonce': 2899402373}]}, 'public_addr': '172.21.15.179:6825/2899402373', 'cluster_addr': '172.21.15.179:6827/2899402373', 'heartbeat_back_addr': '172.21.15.179:6831/2899402373', 'heartbeat_front_addr': '172.21.15.179:6829/2899402373', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T18:03:25.851815+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T18:03:07.352057+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T18:03:53.750446+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T18:03:06.473727+0000', 'dead_epoch': 814}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T18:03:28.714945+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T18:03:37.910096+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T18:03:05.756967+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T18:04:15.618566+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T18:03:50.783888+0000', 'dead_epoch': 145}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T18:04:13.878100+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-09T18:03:21.800213+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-09T18:03:40.705030+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.7', 'osds': [1, 8, 6]}], 'pg_upmap_items': [{'pgid': '1.5', 'mappings': [{'from': 6, 'to': 8}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '9.7', 'osds': [3, 7, 10]}], 'primary_temp': [], 'blocklist': {'172.21.15.99:0/3904172782': '2024-02-10T17:16:24.229365+0000', '172.21.15.99:6801/1586034576': '2024-02-10T17:16:24.229365+0000', '172.21.15.99:6800/746248609': '2024-02-10T17:15:54.399201+0000', '172.21.15.99:0/4092784197': '2024-02-10T17:16:24.229365+0000', '172.21.15.99:0/2844598101': '2024-02-10T17:15:54.399201+0000', '172.21.15.99:6801/746248609': '2024-02-10T17:15:54.399201+0000', '172.21.15.99:0/2528750050': '2024-02-10T17:15:54.399201+0000', '172.21.15.99:6800/1586034576': '2024-02-10T17:16:24.229365+0000', '172.21.15.99:0/2214140078': '2024-02-10T17:15:54.399201+0000', '172.21.15.99:0/4217821965': '2024-02-10T17:16:24.229365+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-09T18:04:18.766 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph pg dump --format=json 2024-02-09T18:04:18.806 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:04:18.831 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:18 smithi099 ceph-mon[39420]: 1.1c scrub starts 2024-02-09T18:04:18.831 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:18 smithi099 ceph-mon[39420]: 1.1c scrub ok 2024-02-09T18:04:18.832 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:18 smithi099 ceph-mon[39420]: pgmap v2395: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 1.4 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 804 op/s; 1201/67359 objects misplaced (1.783%); 1.0 MiB/s, 16 objects/s recovering 2024-02-09T18:04:18.832 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:18 smithi099 ceph-mon[39420]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-09T18:04:18.832 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:18 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2270651347' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-09T18:04:18.832 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:18 smithi099 ceph-mon[39420]: 9.6 scrub ok 2024-02-09T18:04:18.832 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:18 smithi099 ceph-mon[39420]: 1.1d scrub starts 2024-02-09T18:04:18.837 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:18 smithi162 ceph-mon[44304]: 1.1c scrub starts 2024-02-09T18:04:18.837 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:18 smithi162 ceph-mon[44304]: 1.1c scrub ok 2024-02-09T18:04:18.837 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:18 smithi162 ceph-mon[44304]: pgmap v2395: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 1.4 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 804 op/s; 1201/67359 objects misplaced (1.783%); 1.0 MiB/s, 16 objects/s recovering 2024-02-09T18:04:18.837 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:18 smithi162 ceph-mon[44304]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-09T18:04:18.837 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:18 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2270651347' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-09T18:04:18.837 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:18 smithi162 ceph-mon[44304]: 9.6 scrub ok 2024-02-09T18:04:18.837 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:18 smithi162 ceph-mon[44304]: 1.1d scrub starts 2024-02-09T18:04:18.838 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:18.877 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:04:18.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:18.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:04:18.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:18 smithi179 ceph-mon[43975]: 1.1c scrub starts 2024-02-09T18:04:18.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:18 smithi179 ceph-mon[43975]: 1.1c scrub ok 2024-02-09T18:04:18.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:18 smithi179 ceph-mon[43975]: pgmap v2395: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 1.4 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 804 op/s; 1201/67359 objects misplaced (1.783%); 1.0 MiB/s, 16 objects/s recovering 2024-02-09T18:04:18.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:18 smithi179 ceph-mon[43975]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-09T18:04:18.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:18 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2270651347' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-09T18:04:18.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:18 smithi179 ceph-mon[43975]: 9.6 scrub ok 2024-02-09T18:04:18.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:18 smithi179 ceph-mon[43975]: 1.1d scrub starts 2024-02-09T18:04:18.983 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:04:19.031 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:19.085 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:04:19.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:19.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:04:19.192 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T18:04:19.225 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:19.253 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T18:04:19.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:19.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T18:04:19.360 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:04:19.406 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:04:19.444 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:19.450 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-09T18:04:19.531 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:19 smithi162 ceph-mon[44304]: 1.1e scrub starts 2024-02-09T18:04:19.531 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:19 smithi162 ceph-mon[44304]: 1.1e scrub ok 2024-02-09T18:04:19.532 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:19 smithi162 ceph-mon[44304]: 1.1d scrub ok 2024-02-09T18:04:19.559 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:04:19.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:19.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:04:19.663 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T18:04:19.706 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:19.743 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T18:04:19.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:19.746 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T18:04:19.748 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:19 smithi099 ceph-mon[39420]: 1.1e scrub starts 2024-02-09T18:04:19.748 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:19 smithi099 ceph-mon[39420]: 1.1e scrub ok 2024-02-09T18:04:19.748 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:19 smithi099 ceph-mon[39420]: 1.1d scrub ok 2024-02-09T18:04:19.847 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5 2024-02-09T18:04:19.896 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:19.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:19 smithi179 ceph-mon[43975]: 1.1e scrub starts 2024-02-09T18:04:19.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:19 smithi179 ceph-mon[43975]: 1.1e scrub ok 2024-02-09T18:04:19.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:19 smithi179 ceph-mon[43975]: 1.1d scrub ok 2024-02-09T18:04:19.934 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5" found: no such container 2024-02-09T18:04:19.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:19.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.5' 2024-02-09T18:04:20.039 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T18:04:20.092 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:20.122 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T18:04:20.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:20.126 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T18:04:20.227 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:04:20.277 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:20.309 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:04:20.317 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:20.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:04:20.419 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:04:20.466 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:20.536 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:20 smithi162 ceph-mon[44304]: pgmap v2396: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 1.3 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 744 op/s; 1201/64170 objects misplaced (1.872%); 788 KiB/s, 12 objects/s recovering 2024-02-09T18:04:20.536 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:20 smithi162 ceph-mon[44304]: 9.8 scrub starts 2024-02-09T18:04:20.536 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:20 smithi162 ceph-mon[44304]: 9.8 scrub ok 2024-02-09T18:04:20.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:20 smithi099 ceph-mon[39420]: pgmap v2396: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 1.3 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 744 op/s; 1201/64170 objects misplaced (1.872%); 788 KiB/s, 12 objects/s recovering 2024-02-09T18:04:20.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:20 smithi099 ceph-mon[39420]: 9.8 scrub starts 2024-02-09T18:04:20.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:20 smithi099 ceph-mon[39420]: 9.8 scrub ok 2024-02-09T18:04:20.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:20 smithi179 ceph-mon[43975]: pgmap v2396: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 1.3 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 744 op/s; 1201/64170 objects misplaced (1.872%); 788 KiB/s, 12 objects/s recovering 2024-02-09T18:04:20.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:20 smithi179 ceph-mon[43975]: 9.8 scrub starts 2024-02-09T18:04:20.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:20 smithi179 ceph-mon[43975]: 9.8 scrub ok 2024-02-09T18:04:21.998 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:04:22.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:22.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:04:22.106 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:04:22.139 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:22.167 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:04:22.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:22.171 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:04:22.272 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:04:22.317 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:22.362 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:04:22.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:22.365 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:04:22.466 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T18:04:22.492 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:22.512 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T18:04:22.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:22.518 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T18:04:22.619 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:04:22.680 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:22 smithi099 ceph-mon[39420]: pgmap v2397: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 1.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 864 op/s; 328/56097 objects misplaced (0.585%); 6.3 MiB/s, 101 objects/s recovering 2024-02-09T18:04:22.691 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:22.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:22 smithi179 ceph-mon[43975]: pgmap v2397: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 1.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 864 op/s; 328/56097 objects misplaced (0.585%); 6.3 MiB/s, 101 objects/s recovering 2024-02-09T18:04:22.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:22 smithi162 ceph-mon[44304]: pgmap v2397: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 1.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 864 op/s; 328/56097 objects misplaced (0.585%); 6.3 MiB/s, 101 objects/s recovering 2024-02-09T18:04:23.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:23 smithi179 ceph-mon[43975]: 1.1a scrub starts 2024-02-09T18:04:23.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:23 smithi179 ceph-mon[43975]: 1.1a scrub ok 2024-02-09T18:04:23.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:23 smithi162 ceph-mon[44304]: 1.1a scrub starts 2024-02-09T18:04:23.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:23 smithi162 ceph-mon[44304]: 1.1a scrub ok 2024-02-09T18:04:23.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:23 smithi099 ceph-mon[39420]: 1.1a scrub starts 2024-02-09T18:04:23.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:23 smithi099 ceph-mon[39420]: 1.1a scrub ok 2024-02-09T18:04:24.595 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:04:24.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:24.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:04:24.700 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:04:24.751 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:24.779 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:04:24.782 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:24.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:04:24.816 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:04:24.828 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:24 smithi099 ceph-mon[39420]: 1.b scrub starts 2024-02-09T18:04:24.828 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:24 smithi099 ceph-mon[39420]: 1.b scrub ok 2024-02-09T18:04:24.828 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:24 smithi099 ceph-mon[39420]: pgmap v2398: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 1.0 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 963 op/s; 328/51141 objects misplaced (0.641%); 5.7 MiB/s, 91 objects/s recovering 2024-02-09T18:04:24.828 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:24 smithi099 ceph-mon[39420]: osdmap e970: 12 total, 12 up, 11 in 2024-02-09T18:04:24.828 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:24 smithi099 ceph-mon[39420]: 9.d scrub starts 2024-02-09T18:04:24.884 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T18:04:24.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:24 smithi179 ceph-mon[43975]: 1.b scrub starts 2024-02-09T18:04:24.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:24 smithi179 ceph-mon[43975]: 1.b scrub ok 2024-02-09T18:04:24.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:24 smithi179 ceph-mon[43975]: pgmap v2398: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 1.0 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 963 op/s; 328/51141 objects misplaced (0.641%); 5.7 MiB/s, 91 objects/s recovering 2024-02-09T18:04:24.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:24 smithi179 ceph-mon[43975]: osdmap e970: 12 total, 12 up, 11 in 2024-02-09T18:04:24.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:24 smithi179 ceph-mon[43975]: 9.d scrub starts 2024-02-09T18:04:24.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:24 smithi162 ceph-mon[44304]: 1.b scrub starts 2024-02-09T18:04:24.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:24 smithi162 ceph-mon[44304]: 1.b scrub ok 2024-02-09T18:04:24.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:24 smithi162 ceph-mon[44304]: pgmap v2398: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 1.0 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 963 op/s; 328/51141 objects misplaced (0.641%); 5.7 MiB/s, 91 objects/s recovering 2024-02-09T18:04:24.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:24 smithi162 ceph-mon[44304]: osdmap e970: 12 total, 12 up, 11 in 2024-02-09T18:04:24.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:24 smithi162 ceph-mon[44304]: 9.d scrub starts 2024-02-09T18:04:24.931 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:24.961 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T18:04:24.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:24.966 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T18:04:25.067 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:04:25.102 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:25.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:25 smithi179 ceph-mon[43975]: 9.d scrub ok 2024-02-09T18:04:25.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:25 smithi179 ceph-mon[43975]: osdmap e971: 12 total, 12 up, 11 in 2024-02-09T18:04:25.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:25 smithi179 ceph-mon[43975]: 1.16 scrub starts 2024-02-09T18:04:25.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:25 smithi179 ceph-mon[43975]: 1.16 scrub ok 2024-02-09T18:04:25.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:25 smithi162 ceph-mon[44304]: 9.d scrub ok 2024-02-09T18:04:25.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:25 smithi162 ceph-mon[44304]: osdmap e971: 12 total, 12 up, 11 in 2024-02-09T18:04:25.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:25 smithi162 ceph-mon[44304]: 1.16 scrub starts 2024-02-09T18:04:25.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:25 smithi162 ceph-mon[44304]: 1.16 scrub ok 2024-02-09T18:04:25.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:25 smithi099 ceph-mon[39420]: 9.d scrub ok 2024-02-09T18:04:25.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:25 smithi099 ceph-mon[39420]: osdmap e971: 12 total, 12 up, 11 in 2024-02-09T18:04:25.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:25 smithi099 ceph-mon[39420]: 1.16 scrub starts 2024-02-09T18:04:25.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:25 smithi099 ceph-mon[39420]: 1.16 scrub ok 2024-02-09T18:04:26.467 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:04:26.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:26.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:04:26.587 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T18:04:26.600 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is unset 2024-02-09T18:04:26.645 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:26.679 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T18:04:26.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:26.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T18:04:26.737 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:26 smithi099 ceph-mon[39420]: 9.9 scrub starts 2024-02-09T18:04:26.737 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:26 smithi099 ceph-mon[39420]: 9.9 scrub ok 2024-02-09T18:04:26.737 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:26 smithi099 ceph-mon[39420]: pgmap v2401: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 1.0 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 675 op/s; 328/51141 objects misplaced (0.641%); 5.6 MiB/s, 89 objects/s recovering 2024-02-09T18:04:26.737 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:26 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2588173667' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T18:04:26.737 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:26 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T18:04:26.790 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:04:26.827 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:26.852 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:04:26.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:26.856 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:04:26.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:26 smithi179 ceph-mon[43975]: 9.9 scrub starts 2024-02-09T18:04:26.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:26 smithi179 ceph-mon[43975]: 9.9 scrub ok 2024-02-09T18:04:26.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:26 smithi179 ceph-mon[43975]: pgmap v2401: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 1.0 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 675 op/s; 328/51141 objects misplaced (0.641%); 5.6 MiB/s, 89 objects/s recovering 2024-02-09T18:04:26.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:26 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2588173667' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T18:04:26.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:26 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T18:04:26.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:26 smithi162 ceph-mon[44304]: 9.9 scrub starts 2024-02-09T18:04:26.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:26 smithi162 ceph-mon[44304]: 9.9 scrub ok 2024-02-09T18:04:26.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:26 smithi162 ceph-mon[44304]: pgmap v2401: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 1.0 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 675 op/s; 328/51141 objects misplaced (0.641%); 5.6 MiB/s, 89 objects/s recovering 2024-02-09T18:04:26.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:26 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2588173667' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T18:04:26.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:26 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T18:04:26.957 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T18:04:26.992 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:27.024 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T18:04:27.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:27.027 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T18:04:27.127 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T18:04:27.180 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:27.208 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T18:04:27.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:27.212 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T18:04:27.314 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:04:27.354 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:27.389 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:04:27.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:27.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:04:27.493 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:04:27.527 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:27.563 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:04:27.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:27.567 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:04:27.668 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:04:27.726 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:27 smithi099 ceph-mon[39420]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T18:04:27.727 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:27 smithi099 ceph-mon[39420]: Cluster is now healthy 2024-02-09T18:04:27.727 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:27 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2588173667' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T18:04:27.727 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:27 smithi099 ceph-mon[39420]: osdmap e972: 12 total, 12 up, 11 in 2024-02-09T18:04:27.729 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:27.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:27 smithi179 ceph-mon[43975]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T18:04:27.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:27 smithi179 ceph-mon[43975]: Cluster is now healthy 2024-02-09T18:04:27.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:27 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2588173667' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T18:04:27.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:27 smithi179 ceph-mon[43975]: osdmap e972: 12 total, 12 up, 11 in 2024-02-09T18:04:27.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:27 smithi162 ceph-mon[44304]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T18:04:27.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:27 smithi162 ceph-mon[44304]: Cluster is now healthy 2024-02-09T18:04:27.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:27 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2588173667' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T18:04:27.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:27 smithi162 ceph-mon[44304]: osdmap e972: 12 total, 12 up, 11 in 2024-02-09T18:04:28.484 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:04:28.514 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:04:28.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:28.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:04:28.618 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:04:28.664 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:28.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:28 smithi099 ceph-mon[39420]: 1.2 scrub starts 2024-02-09T18:04:28.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:28 smithi099 ceph-mon[39420]: 1.2 scrub ok 2024-02-09T18:04:28.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:28 smithi099 ceph-mon[39420]: pgmap v2403: 47 pgs: 47 active+clean; 879 MiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 772 op/s; 3.3 MiB/s, 52 objects/s recovering 2024-02-09T18:04:28.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:28 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T18:04:28.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:28 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T18:04:28.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:28 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T18:04:28.774 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:04:28.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:28.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:04:28.885 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:04:28.910 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:28 smithi162 ceph-mon[44304]: 1.2 scrub starts 2024-02-09T18:04:28.910 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:28 smithi162 ceph-mon[44304]: 1.2 scrub ok 2024-02-09T18:04:28.910 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:28 smithi162 ceph-mon[44304]: pgmap v2403: 47 pgs: 47 active+clean; 879 MiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 772 op/s; 3.3 MiB/s, 52 objects/s recovering 2024-02-09T18:04:28.910 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:28 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T18:04:28.910 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:28 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T18:04:28.910 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:28 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T18:04:28.912 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:28.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:28 smithi179 ceph-mon[43975]: 1.2 scrub starts 2024-02-09T18:04:28.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:28 smithi179 ceph-mon[43975]: 1.2 scrub ok 2024-02-09T18:04:28.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:28 smithi179 ceph-mon[43975]: pgmap v2403: 47 pgs: 47 active+clean; 879 MiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 772 op/s; 3.3 MiB/s, 52 objects/s recovering 2024-02-09T18:04:28.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:28 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T18:04:28.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:28 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T18:04:28.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:28 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T18:04:28.931 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:04:28.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:28.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:04:29.035 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:04:29.121 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:29.210 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:04:29.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:29.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:04:29.317 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:04:29.353 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:29.384 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:04:29.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:29.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:04:29.477 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T18:04:29.490 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T18:04:29.490 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:04:29.528 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:29.537 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T18:04:29.551 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:04:29.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:29.554 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:04:29.567 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T18:04:29.605 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T18:04:29.652 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:29 smithi162 ceph-mon[44304]: 9.7 scrub starts 2024-02-09T18:04:29.653 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:29 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2800424182' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:04:29.655 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T18:04:29.657 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T18:04:29.690 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T18:04:29.697 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:29.726 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T18:04:29.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:29.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T18:04:29.749 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T18:04:29.809 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T18:04:29.825 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T18:04:29.835 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:04:29.848 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T18:04:29.851 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-02-09T18:04:29.894 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T18:04:29.946 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:29 smithi179 ceph-mon[43975]: 9.7 scrub starts 2024-02-09T18:04:29.947 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:29 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2800424182' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:04:29.947 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:29 smithi099 ceph-mon[39420]: 9.7 scrub starts 2024-02-09T18:04:29.947 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:29 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2800424182' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:04:29.947 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:29.947 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T18:04:29.966 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T18:04:29.966 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T18:04:29.966 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T18:04:29.966 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T18:04:29.966 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T18:04:29.966 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T18:04:29.966 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T18:04:29.966 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T18:04:29.966 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T18:04:29.966 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T18:04:29.966 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T18:04:29.967 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T18:04:30.394 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:04:30.399 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:30.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:04:30.500 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:04:30.504 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set noscrub 2024-02-09T18:04:30.563 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:30.592 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:04:30.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:30.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:04:30.703 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:04:30.750 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:30.785 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:04:30.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:30.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:04:30.894 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3 2024-02-09T18:04:30.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:30 smithi179 ceph-mon[43975]: 9.7 scrub ok 2024-02-09T18:04:30.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:30 smithi179 ceph-mon[43975]: pgmap v2404: 47 pgs: 47 active+clean; 837 MiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 609 op/s; 3.3 MiB/s, 52 objects/s recovering 2024-02-09T18:04:30.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:30 smithi179 ceph-mon[43975]: from='client.17283 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T18:04:30.931 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:30 smithi162 ceph-mon[44304]: 9.7 scrub ok 2024-02-09T18:04:30.931 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:30 smithi162 ceph-mon[44304]: pgmap v2404: 47 pgs: 47 active+clean; 837 MiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 609 op/s; 3.3 MiB/s, 52 objects/s recovering 2024-02-09T18:04:30.932 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:30 smithi162 ceph-mon[44304]: from='client.17283 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T18:04:30.939 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:30 smithi099 ceph-mon[39420]: 9.7 scrub ok 2024-02-09T18:04:30.939 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:30 smithi099 ceph-mon[39420]: pgmap v2404: 47 pgs: 47 active+clean; 837 MiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 609 op/s; 3.3 MiB/s, 52 objects/s recovering 2024-02-09T18:04:30.939 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:30 smithi099 ceph-mon[39420]: from='client.17283 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T18:04:30.941 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:31.931 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:31 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:04:31.931 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:31 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:04:31.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:31 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:04:31.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:31 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:04:32.098 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3" found: no such container 2024-02-09T18:04:32.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:32.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.3' 2024-02-09T18:04:32.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:31 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:04:32.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:31 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:04:32.204 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T18:04:32.243 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:32.268 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T18:04:32.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:32.271 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T18:04:32.372 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T18:04:32.414 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:32.451 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T18:04:32.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:32.455 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T18:04:32.556 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:04:32.605 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:32.633 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:04:32.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:32.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:04:32.739 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:04:32.775 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:32.801 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:04:32.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:32.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:04:32.907 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T18:04:32.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:32 smithi162 ceph-mon[44304]: pgmap v2405: 47 pgs: 47 active+clean; 667 MiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 816 op/s; 2.5 MiB/s, 40 objects/s recovering 2024-02-09T18:04:32.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:32 smithi162 ceph-mon[44304]: osdmap e973: 12 total, 12 up, 11 in 2024-02-09T18:04:32.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:32 smithi162 ceph-mon[44304]: 9.c scrub starts 2024-02-09T18:04:32.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:32 smithi162 ceph-mon[44304]: 9.4 scrub starts 2024-02-09T18:04:32.945 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:32 smithi099 ceph-mon[39420]: pgmap v2405: 47 pgs: 47 active+clean; 667 MiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 816 op/s; 2.5 MiB/s, 40 objects/s recovering 2024-02-09T18:04:32.945 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:32 smithi099 ceph-mon[39420]: osdmap e973: 12 total, 12 up, 11 in 2024-02-09T18:04:32.945 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:32 smithi099 ceph-mon[39420]: 9.c scrub starts 2024-02-09T18:04:32.945 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:32 smithi099 ceph-mon[39420]: 9.4 scrub starts 2024-02-09T18:04:32.950 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:32 smithi179 ceph-mon[43975]: pgmap v2405: 47 pgs: 47 active+clean; 667 MiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 816 op/s; 2.5 MiB/s, 40 objects/s recovering 2024-02-09T18:04:32.950 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:32 smithi179 ceph-mon[43975]: osdmap e973: 12 total, 12 up, 11 in 2024-02-09T18:04:32.950 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:32 smithi179 ceph-mon[43975]: 9.c scrub starts 2024-02-09T18:04:32.950 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:32 smithi179 ceph-mon[43975]: 9.4 scrub starts 2024-02-09T18:04:32.960 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:32.994 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T18:04:32.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:33.000 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T18:04:33.101 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:04:33.141 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:33.312 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:04:33.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:33.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:04:33.417 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:04:33.453 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:33.487 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:04:33.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:33.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:04:33.600 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:04:33.628 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:33.661 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:04:33.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:33.665 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:04:33.766 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T18:04:33.805 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:33.832 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T18:04:33.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:33.838 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T18:04:33.902 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:33 smithi099 ceph-mon[39420]: 9.c scrub ok 2024-02-09T18:04:33.903 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:33 smithi099 ceph-mon[39420]: 9.4 scrub ok 2024-02-09T18:04:33.903 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:33 smithi099 ceph-mon[39420]: 1.7 scrub starts 2024-02-09T18:04:33.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:33 smithi179 ceph-mon[43975]: 9.c scrub ok 2024-02-09T18:04:33.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:33 smithi179 ceph-mon[43975]: 9.4 scrub ok 2024-02-09T18:04:33.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:33 smithi179 ceph-mon[43975]: 1.7 scrub starts 2024-02-09T18:04:33.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:33 smithi162 ceph-mon[44304]: 9.c scrub ok 2024-02-09T18:04:33.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:33 smithi162 ceph-mon[44304]: 9.4 scrub ok 2024-02-09T18:04:33.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:33 smithi162 ceph-mon[44304]: 1.7 scrub starts 2024-02-09T18:04:33.939 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:04:33.971 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:34.003 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:04:34.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:34.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:04:34.011 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:04:34.039 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":2404,"stamp":"2024-02-09T18:04:29.361176+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":878051896,"num_objects":13392,"num_object_clones":0,"num_object_copies":40176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13392,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":256370,"num_write_kb":8632257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23980,"num_bytes_recovered":1571487744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":165022,"ondisk_log_size":165022,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":143,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":11991372,"kb_used_data":2942792,"kb_used_omap":429,"kb_used_meta":9048018,"kb_avail":1019205300,"statfs":{"total":1055945392128,"available":1043666227200,"internally_reserved":0,"allocated":3013419008,"data_stored":3010107585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":439567,"internal_metadata":9265171185},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-239534080,"num_objects":-3655,"num_object_clones":0,"num_object_copies":-10965,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-328,"num_objects_unfound":0,"num_objects_dirty":-3655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3655,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":316,"num_bytes_recovered":20709376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001470"},"pg_stats":[{"pgid":"1.1e","version":"63'139","reported_seq":1284,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-02-09T18:04:26.597871+0000","last_change":"2024-02-09T18:04:17.979500+0000","last_active":"2024-02-09T18:04:26.597871+0000","last_peered":"2024-02-09T18:04:26.597871+0000","last_clean":"2024-02-09T18:04:26.597871+0000","last_became_active":"2024-02-09T18:03:47.759332+0000","last_became_peered":"2024-02-09T18:03:47.759332+0000","last_unstale":"2024-02-09T18:04:26.597871+0000","last_undegraded":"2024-02-09T18:04:26.597871+0000","last_fullsized":"2024-02-09T18:04:26.597871+0000","mapping_epoch":961,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":962,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:04:17.979359+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:04:17.979359+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:05:21.403225+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"63'139","reported_seq":1292,"reported_epoch":971,"state":"active+clean","last_fresh":"2024-02-09T18:04:24.621829+0000","last_change":"2024-02-09T18:04:18.487803+0000","last_active":"2024-02-09T18:04:24.621829+0000","last_peered":"2024-02-09T18:04:24.621829+0000","last_clean":"2024-02-09T18:04:24.621829+0000","last_became_active":"2024-02-09T17:43:28.594296+0000","last_became_peered":"2024-02-09T17:43:28.594296+0000","last_unstale":"2024-02-09T18:04:24.621829+0000","last_undegraded":"2024-02-09T18:04:24.621829+0000","last_fullsized":"2024-02-09T18:04:24.621829+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:04:18.487685+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:48:24.634092+0000","last_clean_scrub_stamp":"2024-02-09T18:04:18.487685+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:05:26.347834+0000","scrub_duration":158,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"63'139","reported_seq":1822,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-02-09T18:04:26.620735+0000","last_change":"2024-02-09T18:04:16.722975+0000","last_active":"2024-02-09T18:04:26.620735+0000","last_peered":"2024-02-09T18:04:26.620735+0000","last_clean":"2024-02-09T18:04:26.620735+0000","last_became_active":"2024-02-09T17:43:19.103584+0000","last_became_peered":"2024-02-09T17:43:19.103584+0000","last_unstale":"2024-02-09T18:04:26.620735+0000","last_undegraded":"2024-02-09T18:04:26.620735+0000","last_fullsized":"2024-02-09T18:04:26.620735+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:04:16.722822+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:23.912504+0000","last_clean_scrub_stamp":"2024-02-09T18:04:16.722822+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:05:44.573787+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"63'139","reported_seq":1620,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-02-09T18:04:26.620993+0000","last_change":"2024-02-09T18:04:14.994622+0000","last_active":"2024-02-09T18:04:26.620993+0000","last_peered":"2024-02-09T18:04:26.620993+0000","last_clean":"2024-02-09T18:04:26.620993+0000","last_became_active":"2024-02-09T17:43:19.102729+0000","last_became_peered":"2024-02-09T17:43:19.102729+0000","last_unstale":"2024-02-09T18:04:26.620993+0000","last_undegraded":"2024-02-09T18:04:26.620993+0000","last_fullsized":"2024-02-09T18:04:26.620993+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:04:14.994498+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:45:44.988633+0000","last_clean_scrub_stamp":"2024-02-09T18:04:14.994498+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:05:35.490809+0000","scrub_duration":190,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"63'139","reported_seq":517,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-02-09T18:04:26.616759+0000","last_change":"2024-02-09T18:04:21.826722+0000","last_active":"2024-02-09T18:04:26.616759+0000","last_peered":"2024-02-09T18:04:26.616759+0000","last_clean":"2024-02-09T18:04:26.616759+0000","last_became_active":"2024-02-09T18:03:02.127510+0000","last_became_peered":"2024-02-09T18:03:02.127510+0000","last_unstale":"2024-02-09T18:04:26.616759+0000","last_undegraded":"2024-02-09T18:04:26.616759+0000","last_fullsized":"2024-02-09T18:04:26.616759+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:04:21.826597+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:04:21.826597+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:05:35.813109+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"63'139","reported_seq":1160,"reported_epoch":971,"state":"active+clean","last_fresh":"2024-02-09T18:04:24.622709+0000","last_change":"2024-02-09T18:03:24.313241+0000","last_active":"2024-02-09T18:04:24.622709+0000","last_peered":"2024-02-09T18:04:24.622709+0000","last_clean":"2024-02-09T18:04:24.622709+0000","last_became_active":"2024-02-09T18:03:17.456437+0000","last_became_peered":"2024-02-09T18:03:17.456437+0000","last_unstale":"2024-02-09T18:04:24.622709+0000","last_undegraded":"2024-02-09T18:04:24.622709+0000","last_fullsized":"2024-02-09T18:04:24.622709+0000","mapping_epoch":945,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":946,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:03:24.313107+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:35:30.193194+0000","last_clean_scrub_stamp":"2024-02-09T18:03:24.313107+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:04:37.050846+0000","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.18","version":"63'139","reported_seq":1964,"reported_epoch":971,"state":"active+clean","last_fresh":"2024-02-09T18:04:24.616062+0000","last_change":"2024-02-09T18:04:16.454940+0000","last_active":"2024-02-09T18:04:24.616062+0000","last_peered":"2024-02-09T18:04:24.616062+0000","last_clean":"2024-02-09T18:04:24.616062+0000","last_became_active":"2024-02-09T17:43:28.585573+0000","last_became_peered":"2024-02-09T17:43:28.585573+0000","last_unstale":"2024-02-09T18:04:24.616062+0000","last_undegraded":"2024-02-09T18:04:24.616062+0000","last_fullsized":"2024-02-09T18:04:24.616062+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:04:16.454828+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:04:16.454828+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:05:17.710560+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"63'139","reported_seq":1413,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-02-09T18:04:26.621848+0000","last_change":"2024-02-09T18:03:40.548995+0000","last_active":"2024-02-09T18:04:26.621848+0000","last_peered":"2024-02-09T18:04:26.621848+0000","last_clean":"2024-02-09T18:04:26.621848+0000","last_became_active":"2024-02-09T18:03:17.461024+0000","last_became_peered":"2024-02-09T18:03:17.461024+0000","last_unstale":"2024-02-09T18:04:26.621848+0000","last_undegraded":"2024-02-09T18:04:26.621848+0000","last_fullsized":"2024-02-09T18:04:26.621848+0000","mapping_epoch":945,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":946,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:03:40.548833+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T18:03:40.548833+0000","last_clean_scrub_stamp":"2024-02-09T18:03:40.548833+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:05:03.819330+0000","scrub_duration":171,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"63'139","reported_seq":1286,"reported_epoch":971,"state":"active+clean","last_fresh":"2024-02-09T18:04:25.366605+0000","last_change":"2024-02-09T18:04:25.366646+0000","last_active":"2024-02-09T18:04:25.366605+0000","last_peered":"2024-02-09T18:04:25.366605+0000","last_clean":"2024-02-09T18:04:25.366605+0000","last_became_active":"2024-02-09T17:59:12.003987+0000","last_became_peered":"2024-02-09T17:59:12.003987+0000","last_unstale":"2024-02-09T18:04:25.366605+0000","last_undegraded":"2024-02-09T18:04:25.366605+0000","last_fullsized":"2024-02-09T18:04:25.366605+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:04:25.366536+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:11.970338+0000","last_clean_scrub_stamp":"2024-02-09T18:04:25.366536+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:05:49.186227+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"63'139","reported_seq":1715,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-02-09T18:04:26.613115+0000","last_change":"2024-02-09T18:03:26.434329+0000","last_active":"2024-02-09T18:04:26.613115+0000","last_peered":"2024-02-09T18:04:26.613115+0000","last_clean":"2024-02-09T18:04:26.613115+0000","last_became_active":"2024-02-09T17:43:19.138398+0000","last_became_peered":"2024-02-09T17:43:19.138398+0000","last_unstale":"2024-02-09T18:04:26.613115+0000","last_undegraded":"2024-02-09T18:04:26.613115+0000","last_fullsized":"2024-02-09T18:04:26.613115+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:03:26.434203+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:03:26.434203+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:04:41.385640+0000","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"63'139","reported_seq":1784,"reported_epoch":971,"state":"active+clean","last_fresh":"2024-02-09T18:04:24.563724+0000","last_change":"2024-02-09T18:03:31.476192+0000","last_active":"2024-02-09T18:04:24.563724+0000","last_peered":"2024-02-09T18:04:24.563724+0000","last_clean":"2024-02-09T18:04:24.563724+0000","last_became_active":"2024-02-09T17:43:28.589307+0000","last_became_peered":"2024-02-09T17:43:28.589307+0000","last_unstale":"2024-02-09T18:04:24.563724+0000","last_undegraded":"2024-02-09T18:04:24.563724+0000","last_fullsized":"2024-02-09T18:04:24.563724+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:03:31.476058+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:38:26.968892+0000","last_clean_scrub_stamp":"2024-02-09T18:03:31.476058+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:04:52.570657+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"972'16364","reported_seq":16719,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-02-09T18:04:27.798243+0000","last_change":"2024-02-09T18:04:19.778192+0000","last_active":"2024-02-09T18:04:27.798243+0000","last_peered":"2024-02-09T18:04:27.798243+0000","last_clean":"2024-02-09T18:04:27.798243+0000","last_became_active":"2024-02-09T18:00:07.530069+0000","last_became_peered":"2024-02-09T18:00:07.530069+0000","last_unstale":"2024-02-09T18:04:27.798243+0000","last_undegraded":"2024-02-09T18:04:27.798243+0000","last_fullsized":"2024-02-09T18:04:27.798243+0000","mapping_epoch":882,"log_start":"907'6300","ondisk_log_start":"907'6300","created":882,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"969'15965","last_scrub_stamp":"2024-02-09T18:04:19.778007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_clean_scrub_stamp":"2024-02-09T18:04:19.778007+0000","objects_scrubbed":1198,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:05:36.951115+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51773440,"num_objects":790,"num_object_clones":0,"num_object_copies":2370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16364,"num_write_kb":548928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,5],"acting":[9,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"63'139","reported_seq":2365,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-02-09T18:04:26.608715+0000","last_change":"2024-02-09T18:03:22.321914+0000","last_active":"2024-02-09T18:04:26.608715+0000","last_peered":"2024-02-09T18:04:26.608715+0000","last_clean":"2024-02-09T18:04:26.608715+0000","last_became_active":"2024-02-09T17:41:14.521079+0000","last_became_peered":"2024-02-09T17:41:14.521079+0000","last_unstale":"2024-02-09T18:04:26.608715+0000","last_undegraded":"2024-02-09T18:04:26.608715+0000","last_fullsized":"2024-02-09T18:04:26.608715+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:03:22.321795+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:03:22.321795+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:04:39.882557+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.a","version":"971'15964","reported_seq":16244,"reported_epoch":971,"state":"active+clean","last_fresh":"2024-02-09T18:04:26.014294+0000","last_change":"2024-02-09T18:01:56.341368+0000","last_active":"2024-02-09T18:04:26.014294+0000","last_peered":"2024-02-09T18:04:26.014294+0000","last_clean":"2024-02-09T18:04:26.014294+0000","last_became_active":"2024-02-09T18:00:07.530662+0000","last_became_peered":"2024-02-09T18:00:07.530662+0000","last_unstale":"2024-02-09T18:04:26.014294+0000","last_undegraded":"2024-02-09T18:04:26.014294+0000","last_fullsized":"2024-02-09T18:04:26.014294+0000","mapping_epoch":882,"log_start":"905'5900","ondisk_log_start":"905'5900","created":882,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"921'8782","last_scrub_stamp":"2024-02-09T18:01:56.341243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_clean_scrub_stamp":"2024-02-09T18:01:56.341243+0000","objects_scrubbed":8111,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2244,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56754176,"num_objects":866,"num_object_clones":0,"num_object_copies":2598,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15964,"num_write_kb":538560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"63'139","reported_seq":1964,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-02-09T18:04:27.080556+0000","last_change":"2024-02-09T18:04:27.080608+0000","last_active":"2024-02-09T18:04:27.080556+0000","last_peered":"2024-02-09T18:04:27.080556+0000","last_clean":"2024-02-09T18:04:27.080556+0000","last_became_active":"2024-02-09T18:03:02.123767+0000","last_became_peered":"2024-02-09T18:03:02.123767+0000","last_unstale":"2024-02-09T18:04:27.080556+0000","last_undegraded":"2024-02-09T18:04:27.080556+0000","last_fullsized":"2024-02-09T18:04:27.080556+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:04:27.080461+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:04:27.080461+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:05:35.497832+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"972'16100","reported_seq":23873,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-02-09T18:04:27.075300+0000","last_change":"2024-02-09T18:04:16.142389+0000","last_active":"2024-02-09T18:04:27.075300+0000","last_peered":"2024-02-09T18:04:27.075300+0000","last_clean":"2024-02-09T18:04:27.075300+0000","last_became_active":"2024-02-09T18:03:40.707173+0000","last_became_peered":"2024-02-09T18:03:40.707173+0000","last_unstale":"2024-02-09T18:04:27.075300+0000","last_undegraded":"2024-02-09T18:04:27.075300+0000","last_fullsized":"2024-02-09T18:04:27.075300+0000","mapping_epoch":955,"log_start":"905'6000","ondisk_log_start":"905'6000","created":882,"last_epoch_clean":956,"parent":"0.0","parent_split_bits":0,"last_scrub":"968'15555","last_scrub_stamp":"2024-02-09T18:04:16.142211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_clean_scrub_stamp":"2024-02-09T18:04:16.142211+0000","objects_scrubbed":1379,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:05:17.598133+0000","scrub_duration":382,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54001664,"num_objects":824,"num_object_clones":0,"num_object_copies":2472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16100,"num_write_kb":541568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3696,"num_bytes_recovered":242221056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,1],"acting":[8,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"63'139","reported_seq":1579,"reported_epoch":971,"state":"active+clean","last_fresh":"2024-02-09T18:04:24.621578+0000","last_change":"2024-02-09T18:03:46.781098+0000","last_active":"2024-02-09T18:04:24.621578+0000","last_peered":"2024-02-09T18:04:24.621578+0000","last_clean":"2024-02-09T18:04:24.621578+0000","last_became_active":"2024-02-09T18:03:02.134791+0000","last_became_peered":"2024-02-09T18:03:02.134791+0000","last_unstale":"2024-02-09T18:04:24.621578+0000","last_undegraded":"2024-02-09T18:04:24.621578+0000","last_fullsized":"2024-02-09T18:04:24.621578+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:03:46.780958+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:46:40.684762+0000","last_clean_scrub_stamp":"2024-02-09T18:03:46.780958+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:05:06.328911+0000","scrub_duration":180,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"972'16147","reported_seq":23870,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-02-09T18:04:27.142124+0000","last_change":"2024-02-09T18:04:25.095830+0000","last_active":"2024-02-09T18:04:27.142124+0000","last_peered":"2024-02-09T18:04:27.142124+0000","last_clean":"2024-02-09T18:04:27.142124+0000","last_became_active":"2024-02-09T18:03:40.709331+0000","last_became_peered":"2024-02-09T18:03:40.709331+0000","last_unstale":"2024-02-09T18:04:27.142124+0000","last_undegraded":"2024-02-09T18:04:27.142124+0000","last_fullsized":"2024-02-09T18:04:27.142124+0000","mapping_epoch":955,"log_start":"906'6100","ondisk_log_start":"906'6100","created":882,"last_epoch_clean":956,"parent":"0.0","parent_split_bits":0,"last_scrub":"971'16027","last_scrub_stamp":"2024-02-09T18:04:25.095548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_clean_scrub_stamp":"2024-02-09T18:04:25.095548+0000","objects_scrubbed":948,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:05:36.495432+0000","scrub_duration":238,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53673984,"num_objects":819,"num_object_clones":0,"num_object_copies":2457,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16147,"num_write_kb":542912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3676,"num_bytes_recovered":240910336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,3],"acting":[11,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1","version":"63'139","reported_seq":700,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-02-09T18:04:26.603547+0000","last_change":"2024-02-09T18:04:12.558192+0000","last_active":"2024-02-09T18:04:26.603547+0000","last_peered":"2024-02-09T18:04:26.603547+0000","last_clean":"2024-02-09T18:04:26.603547+0000","last_became_active":"2024-02-09T18:03:17.460826+0000","last_became_peered":"2024-02-09T18:03:17.460826+0000","last_unstale":"2024-02-09T18:04:26.603547+0000","last_undegraded":"2024-02-09T18:04:26.603547+0000","last_fullsized":"2024-02-09T18:04:26.603547+0000","mapping_epoch":945,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":946,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:04:12.558054+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T18:00:46.701010+0000","last_clean_scrub_stamp":"2024-02-09T18:04:12.558054+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:05:23.117630+0000","scrub_duration":88,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2],"acting":[9,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.e","version":"971'16014","reported_seq":23384,"reported_epoch":971,"state":"active+clean","last_fresh":"2024-02-09T18:04:26.013980+0000","last_change":"2024-02-09T18:03:47.809954+0000","last_active":"2024-02-09T18:04:26.013980+0000","last_peered":"2024-02-09T18:04:26.013980+0000","last_clean":"2024-02-09T18:04:26.013980+0000","last_became_active":"2024-02-09T18:03:47.762571+0000","last_became_peered":"2024-02-09T18:03:47.762571+0000","last_unstale":"2024-02-09T18:04:26.013980+0000","last_undegraded":"2024-02-09T18:04:26.013980+0000","last_fullsized":"2024-02-09T18:04:26.013980+0000","mapping_epoch":961,"log_start":"906'6000","ondisk_log_start":"906'6000","created":882,"last_epoch_clean":962,"parent":"0.0","parent_split_bits":0,"last_scrub":"932'11675","last_scrub_stamp":"2024-02-09T18:02:52.747271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_clean_scrub_stamp":"2024-02-09T18:02:52.747271+0000","objects_scrubbed":5228,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1483,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55967744,"num_objects":854,"num_object_clones":0,"num_object_copies":2562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16014,"num_write_kb":539776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3509,"num_bytes_recovered":229965824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,9],"acting":[1,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"63'139","reported_seq":326,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-02-09T18:04:26.599256+0000","last_change":"2024-02-09T18:04:16.472671+0000","last_active":"2024-02-09T18:04:26.599256+0000","last_peered":"2024-02-09T18:04:26.599256+0000","last_clean":"2024-02-09T18:04:26.599256+0000","last_became_active":"2024-02-09T18:03:17.455504+0000","last_became_peered":"2024-02-09T18:03:17.455504+0000","last_unstale":"2024-02-09T18:04:26.599256+0000","last_undegraded":"2024-02-09T18:04:26.599256+0000","last_fullsized":"2024-02-09T18:04:26.599256+0000","mapping_epoch":945,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":946,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:04:16.472595+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:11.970338+0000","last_clean_scrub_stamp":"2024-02-09T18:04:16.472595+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:05:31.081281+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.f","version":"971'19418","reported_seq":19834,"reported_epoch":971,"state":"active+clean","last_fresh":"2024-02-09T18:04:26.023370+0000","last_change":"2024-02-09T18:02:36.054089+0000","last_active":"2024-02-09T18:04:26.023370+0000","last_peered":"2024-02-09T18:04:26.023370+0000","last_clean":"2024-02-09T18:04:26.023370+0000","last_became_active":"2024-02-09T18:00:07.523115+0000","last_became_peered":"2024-02-09T18:00:07.523115+0000","last_unstale":"2024-02-09T18:04:26.023370+0000","last_undegraded":"2024-02-09T18:04:26.023370+0000","last_fullsized":"2024-02-09T18:04:26.023370+0000","mapping_epoch":882,"log_start":"921'9400","ondisk_log_start":"921'9400","created":882,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'14212","last_scrub_stamp":"2024-02-09T18:02:36.053845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_clean_scrub_stamp":"2024-02-09T18:02:36.053845+0000","objects_scrubbed":6123,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1613,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56688640,"num_objects":865,"num_object_clones":0,"num_object_copies":2595,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15975,"num_write_kb":538880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,9],"acting":[1,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"63'139","reported_seq":1481,"reported_epoch":971,"state":"active+clean","last_fresh":"2024-02-09T18:04:24.621945+0000","last_change":"2024-02-09T18:01:37.427302+0000","last_active":"2024-02-09T18:04:24.621945+0000","last_peered":"2024-02-09T18:04:24.621945+0000","last_clean":"2024-02-09T18:04:24.621945+0000","last_became_active":"2024-02-09T17:52:32.292519+0000","last_became_peered":"2024-02-09T17:52:32.292519+0000","last_unstale":"2024-02-09T18:04:24.621945+0000","last_undegraded":"2024-02-09T18:04:24.621945+0000","last_fullsized":"2024-02-09T18:04:24.621945+0000","mapping_epoch":716,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":717,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:01:37.427192+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:01:37.427192+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":384,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.c","version":"971'16091","reported_seq":24972,"reported_epoch":971,"state":"active+clean","last_fresh":"2024-02-09T18:04:26.201702+0000","last_change":"2024-02-09T18:03:26.619973+0000","last_active":"2024-02-09T18:04:26.201702+0000","last_peered":"2024-02-09T18:04:26.201702+0000","last_clean":"2024-02-09T18:04:26.201702+0000","last_became_active":"2024-02-09T18:03:26.582534+0000","last_became_peered":"2024-02-09T18:03:26.582534+0000","last_unstale":"2024-02-09T18:04:26.201702+0000","last_undegraded":"2024-02-09T18:04:26.201702+0000","last_fullsized":"2024-02-09T18:04:26.201702+0000","mapping_epoch":950,"log_start":"905'6000","ondisk_log_start":"905'6000","created":882,"last_epoch_clean":951,"parent":"0.0","parent_split_bits":0,"last_scrub":"931'11348","last_scrub_stamp":"2024-02-09T18:02:44.082539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_clean_scrub_stamp":"2024-02-09T18:02:44.082539+0000","objects_scrubbed":5643,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56950784,"num_objects":869,"num_object_clones":0,"num_object_copies":2607,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16091,"num_write_kb":542720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4263,"num_bytes_recovered":279314432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,9],"acting":[3,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"63'139","reported_seq":1945,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-02-09T18:04:26.597948+0000","last_change":"2024-02-09T18:03:41.972866+0000","last_active":"2024-02-09T18:04:26.597948+0000","last_peered":"2024-02-09T18:04:26.597948+0000","last_clean":"2024-02-09T18:04:26.597948+0000","last_became_active":"2024-02-09T17:33:00.284924+0000","last_became_peered":"2024-02-09T17:33:00.284924+0000","last_unstale":"2024-02-09T18:04:26.597948+0000","last_undegraded":"2024-02-09T18:04:26.597948+0000","last_fullsized":"2024-02-09T18:04:26.597948+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:03:41.972744+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:56:45.406767+0000","last_clean_scrub_stamp":"2024-02-09T18:03:41.972744+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:05:01.285475+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.d","version":"971'15918","reported_seq":16215,"reported_epoch":971,"state":"active+clean","last_fresh":"2024-02-09T18:04:26.030758+0000","last_change":"2024-02-09T18:04:24.418929+0000","last_active":"2024-02-09T18:04:26.030758+0000","last_peered":"2024-02-09T18:04:26.030758+0000","last_clean":"2024-02-09T18:04:26.030758+0000","last_became_active":"2024-02-09T18:00:07.530751+0000","last_became_peered":"2024-02-09T18:00:07.530751+0000","last_unstale":"2024-02-09T18:04:26.030758+0000","last_undegraded":"2024-02-09T18:04:26.030758+0000","last_fullsized":"2024-02-09T18:04:26.030758+0000","mapping_epoch":882,"log_start":"906'5900","ondisk_log_start":"906'5900","created":882,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"970'15841","last_scrub_stamp":"2024-02-09T18:04:24.418818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_clean_scrub_stamp":"2024-02-09T18:04:24.418818+0000","objects_scrubbed":947,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:05:42.721057+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56950808,"num_objects":870,"num_object_clones":0,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":870,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15918,"num_write_kb":537153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"63'139","reported_seq":1127,"reported_epoch":971,"state":"active+clean","last_fresh":"2024-02-09T18:04:24.563811+0000","last_change":"2024-02-09T18:03:24.584338+0000","last_active":"2024-02-09T18:04:24.563811+0000","last_peered":"2024-02-09T18:04:24.563811+0000","last_clean":"2024-02-09T18:04:24.563811+0000","last_became_active":"2024-02-09T18:03:17.466944+0000","last_became_peered":"2024-02-09T18:03:17.466944+0000","last_unstale":"2024-02-09T18:04:24.563811+0000","last_undegraded":"2024-02-09T18:04:24.563811+0000","last_fullsized":"2024-02-09T18:04:24.563811+0000","mapping_epoch":945,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":946,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:03:24.584215+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:03:24.584215+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:04:53.243489+0000","scrub_duration":190,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,11],"acting":[2,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2","version":"971'15904","reported_seq":16238,"reported_epoch":971,"state":"active+clean","last_fresh":"2024-02-09T18:04:26.242662+0000","last_change":"2024-02-09T18:02:46.243504+0000","last_active":"2024-02-09T18:04:26.242662+0000","last_peered":"2024-02-09T18:04:26.242662+0000","last_clean":"2024-02-09T18:04:26.242662+0000","last_became_active":"2024-02-09T18:00:07.021665+0000","last_became_peered":"2024-02-09T18:00:07.021665+0000","last_unstale":"2024-02-09T18:04:26.242662+0000","last_undegraded":"2024-02-09T18:04:26.242662+0000","last_fullsized":"2024-02-09T18:04:26.242662+0000","mapping_epoch":882,"log_start":"906'5900","ondisk_log_start":"906'5900","created":882,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"931'11351","last_scrub_stamp":"2024-02-09T18:02:46.243255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_clean_scrub_stamp":"2024-02-09T18:02:46.243255+0000","objects_scrubbed":5525,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1541,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60030976,"num_objects":916,"num_object_clones":0,"num_object_copies":2748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15904,"num_write_kb":538240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,9],"acting":[3,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"63'139","reported_seq":1885,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-02-09T18:04:26.598850+0000","last_change":"2024-02-09T18:03:49.362613+0000","last_active":"2024-02-09T18:04:26.598850+0000","last_peered":"2024-02-09T18:04:26.598850+0000","last_clean":"2024-02-09T18:04:26.598850+0000","last_became_active":"2024-02-09T18:03:02.134238+0000","last_became_peered":"2024-02-09T18:03:02.134238+0000","last_unstale":"2024-02-09T18:04:26.598850+0000","last_undegraded":"2024-02-09T18:04:26.598850+0000","last_fullsized":"2024-02-09T18:04:26.598850+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:03:49.362497+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T18:03:49.362497+0000","last_clean_scrub_stamp":"2024-02-09T18:03:49.362497+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:04:51.747751+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.3","version":"972'16219","reported_seq":16538,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-02-09T18:04:27.775489+0000","last_change":"2024-02-09T18:03:49.005419+0000","last_active":"2024-02-09T18:04:27.775489+0000","last_peered":"2024-02-09T18:04:27.775489+0000","last_clean":"2024-02-09T18:04:27.775489+0000","last_became_active":"2024-02-09T18:00:07.523241+0000","last_became_peered":"2024-02-09T18:00:07.523241+0000","last_unstale":"2024-02-09T18:04:27.775489+0000","last_undegraded":"2024-02-09T18:04:27.775489+0000","last_fullsized":"2024-02-09T18:04:27.775489+0000","mapping_epoch":882,"log_start":"907'6200","ondisk_log_start":"907'6200","created":882,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"962'14090","last_scrub_stamp":"2024-02-09T18:03:49.005324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_clean_scrub_stamp":"2024-02-09T18:03:49.005324+0000","objects_scrubbed":2833,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:05:12.565757+0000","scrub_duration":601,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45547520,"num_objects":695,"num_object_clones":0,"num_object_copies":2085,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16219,"num_write_kb":541248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,5],"acting":[9,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.b","version":"63'139","reported_seq":1242,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-02-09T18:04:26.597780+0000","last_change":"2024-02-09T18:04:23.112481+0000","last_active":"2024-02-09T18:04:26.597780+0000","last_peered":"2024-02-09T18:04:26.597780+0000","last_clean":"2024-02-09T18:04:26.597780+0000","last_became_active":"2024-02-09T17:55:08.938203+0000","last_became_peered":"2024-02-09T17:55:08.938203+0000","last_unstale":"2024-02-09T18:04:26.597780+0000","last_undegraded":"2024-02-09T18:04:26.597780+0000","last_fullsized":"2024-02-09T18:04:26.597780+0000","mapping_epoch":781,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":782,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:04:23.112393+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:04:23.112393+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:05:24.877526+0000","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.0","version":"971'15947","reported_seq":16310,"reported_epoch":971,"state":"active+clean","last_fresh":"2024-02-09T18:04:26.016902+0000","last_change":"2024-02-09T18:04:16.803011+0000","last_active":"2024-02-09T18:04:26.016902+0000","last_peered":"2024-02-09T18:04:26.016902+0000","last_clean":"2024-02-09T18:04:26.016902+0000","last_became_active":"2024-02-09T18:00:07.131105+0000","last_became_peered":"2024-02-09T18:00:07.131105+0000","last_unstale":"2024-02-09T18:04:26.016902+0000","last_undegraded":"2024-02-09T18:04:26.016902+0000","last_fullsized":"2024-02-09T18:04:26.016902+0000","mapping_epoch":882,"log_start":"905'5900","ondisk_log_start":"905'5900","created":882,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"969'15473","last_scrub_stamp":"2024-02-09T18:04:16.802880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_clean_scrub_stamp":"2024-02-09T18:04:16.802880+0000","objects_scrubbed":1403,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:05:43.011818+0000","scrub_duration":448,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60096512,"num_objects":917,"num_object_clones":0,"num_object_copies":2751,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15947,"num_write_kb":539648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"63'139","reported_seq":1362,"reported_epoch":971,"state":"active+clean","last_fresh":"2024-02-09T18:04:24.564614+0000","last_change":"2024-02-09T18:03:39.430261+0000","last_active":"2024-02-09T18:04:24.564614+0000","last_peered":"2024-02-09T18:04:24.564614+0000","last_clean":"2024-02-09T18:04:24.564614+0000","last_became_active":"2024-02-09T17:59:12.022685+0000","last_became_peered":"2024-02-09T17:59:12.022685+0000","last_unstale":"2024-02-09T18:04:24.564614+0000","last_undegraded":"2024-02-09T18:04:24.564614+0000","last_fullsized":"2024-02-09T18:04:24.564614+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:03:39.430150+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:03:39.430150+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:04:55.852760+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1","version":"972'15928","reported_seq":16228,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-02-09T18:04:27.825119+0000","last_change":"2024-02-09T18:04:14.783859+0000","last_active":"2024-02-09T18:04:27.825119+0000","last_peered":"2024-02-09T18:04:27.825119+0000","last_clean":"2024-02-09T18:04:27.825119+0000","last_became_active":"2024-02-09T18:00:07.131217+0000","last_became_peered":"2024-02-09T18:00:07.131217+0000","last_unstale":"2024-02-09T18:04:27.825119+0000","last_undegraded":"2024-02-09T18:04:27.825119+0000","last_fullsized":"2024-02-09T18:04:27.825119+0000","mapping_epoch":882,"log_start":"905'5900","ondisk_log_start":"905'5900","created":882,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"968'15298","last_scrub_stamp":"2024-02-09T18:04:14.783763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_clean_scrub_stamp":"2024-02-09T18:04:14.783763+0000","objects_scrubbed":1410,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:05:37.558184+0000","scrub_duration":365,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50462720,"num_objects":770,"num_object_clones":0,"num_object_copies":2310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15928,"num_write_kb":534336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,6],"acting":[9,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.9","version":"63'139","reported_seq":2371,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-02-09T18:04:26.608854+0000","last_change":"2024-02-09T18:03:20.494993+0000","last_active":"2024-02-09T18:04:26.608854+0000","last_peered":"2024-02-09T18:04:26.608854+0000","last_clean":"2024-02-09T18:04:26.608854+0000","last_became_active":"2024-02-09T17:43:28.590237+0000","last_became_peered":"2024-02-09T17:43:28.590237+0000","last_unstale":"2024-02-09T18:04:26.608854+0000","last_undegraded":"2024-02-09T18:04:26.608854+0000","last_fullsized":"2024-02-09T18:04:26.608854+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:03:20.494858+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:59:49.716259+0000","last_clean_scrub_stamp":"2024-02-09T18:03:20.494858+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:04:48.560533+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.6","version":"971'15819","reported_seq":22952,"reported_epoch":971,"state":"active+clean","last_fresh":"2024-02-09T18:04:25.999462+0000","last_change":"2024-02-09T18:04:17.741530+0000","last_active":"2024-02-09T18:04:25.999462+0000","last_peered":"2024-02-09T18:04:25.999462+0000","last_clean":"2024-02-09T18:04:25.999462+0000","last_became_active":"2024-02-09T18:03:46.757252+0000","last_became_peered":"2024-02-09T18:03:46.757252+0000","last_unstale":"2024-02-09T18:04:25.999462+0000","last_undegraded":"2024-02-09T18:04:25.999462+0000","last_fullsized":"2024-02-09T18:04:25.999462+0000","mapping_epoch":960,"log_start":"905'5800","ondisk_log_start":"905'5800","created":882,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":0,"last_scrub":"969'15437","last_scrub_stamp":"2024-02-09T18:04:17.741377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_clean_scrub_stamp":"2024-02-09T18:04:17.741377+0000","objects_scrubbed":1220,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:05:21.740731+0000","scrub_duration":423,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54591488,"num_objects":833,"num_object_clones":0,"num_object_copies":2499,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15819,"num_write_kb":532864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3412,"num_bytes_recovered":223608832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,9],"acting":[1,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.e","version":"63'139","reported_seq":336,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-02-09T18:04:26.616986+0000","last_change":"2024-02-09T18:03:29.110512+0000","last_active":"2024-02-09T18:04:26.616986+0000","last_peered":"2024-02-09T18:04:26.616986+0000","last_clean":"2024-02-09T18:04:26.616986+0000","last_became_active":"2024-02-09T18:03:02.137032+0000","last_became_peered":"2024-02-09T18:03:02.137032+0000","last_unstale":"2024-02-09T18:04:26.616986+0000","last_undegraded":"2024-02-09T18:04:26.616986+0000","last_fullsized":"2024-02-09T18:04:26.616986+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:03:29.110395+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:14.981453+0000","last_clean_scrub_stamp":"2024-02-09T18:03:29.110395+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:04:48.323839+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"971'15978","reported_seq":18539,"reported_epoch":971,"state":"active+clean","last_fresh":"2024-02-09T18:04:26.253353+0000","last_change":"2024-02-09T18:04:24.606320+0000","last_active":"2024-02-09T18:04:26.253353+0000","last_peered":"2024-02-09T18:04:26.253353+0000","last_clean":"2024-02-09T18:04:26.253353+0000","last_became_active":"2024-02-09T18:04:24.566629+0000","last_became_peered":"2024-02-09T18:04:24.566629+0000","last_unstale":"2024-02-09T18:04:26.253353+0000","last_undegraded":"2024-02-09T18:04:26.253353+0000","last_fullsized":"2024-02-09T18:04:26.253353+0000","mapping_epoch":970,"log_start":"905'5900","ondisk_log_start":"905'5900","created":882,"last_epoch_clean":971,"parent":"0.0","parent_split_bits":0,"last_scrub":"919'8379","last_scrub_stamp":"2024-02-09T18:01:49.345117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_clean_scrub_stamp":"2024-02-09T18:01:49.345117+0000","objects_scrubbed":8341,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1866,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52690944,"num_objects":804,"num_object_clones":0,"num_object_copies":2412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":537024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1129,"num_bytes_recovered":73990144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.f","version":"63'139","reported_seq":1911,"reported_epoch":971,"state":"active+clean","last_fresh":"2024-02-09T18:04:24.623296+0000","last_change":"2024-02-09T18:04:12.643328+0000","last_active":"2024-02-09T18:04:24.623296+0000","last_peered":"2024-02-09T18:04:24.623296+0000","last_clean":"2024-02-09T18:04:24.623296+0000","last_became_active":"2024-02-09T18:03:02.138442+0000","last_became_peered":"2024-02-09T18:03:02.138442+0000","last_unstale":"2024-02-09T18:04:24.623296+0000","last_undegraded":"2024-02-09T18:04:24.623296+0000","last_fullsized":"2024-02-09T18:04:24.623296+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:04:12.643179+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:48:34.651647+0000","last_clean_scrub_stamp":"2024-02-09T18:04:12.643179+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:05:19.274253+0000","scrub_duration":267,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"971'15903","reported_seq":24783,"reported_epoch":971,"state":"active+clean","last_fresh":"2024-02-09T18:04:25.996402+0000","last_change":"2024-02-09T18:03:25.608758+0000","last_active":"2024-02-09T18:04:25.996402+0000","last_peered":"2024-02-09T18:04:25.996402+0000","last_clean":"2024-02-09T18:04:25.996402+0000","last_became_active":"2024-02-09T18:03:25.577029+0000","last_became_peered":"2024-02-09T18:03:25.577029+0000","last_unstale":"2024-02-09T18:04:25.996402+0000","last_undegraded":"2024-02-09T18:04:25.996402+0000","last_fullsized":"2024-02-09T18:04:25.996402+0000","mapping_epoch":949,"log_start":"905'5900","ondisk_log_start":"905'5900","created":882,"last_epoch_clean":950,"parent":"0.0","parent_split_bits":0,"last_scrub":"921'8840","last_scrub_stamp":"2024-02-09T18:01:58.053829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_clean_scrub_stamp":"2024-02-09T18:01:58.053829+0000","objects_scrubbed":7986,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1808,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57999360,"num_objects":885,"num_object_clones":0,"num_object_copies":2655,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15903,"num_write_kb":537216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4295,"num_bytes_recovered":281477120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"63'139","reported_seq":1857,"reported_epoch":971,"state":"active+clean","last_fresh":"2024-02-09T18:04:24.564920+0000","last_change":"2024-02-09T18:02:57.599920+0000","last_active":"2024-02-09T18:04:24.564920+0000","last_peered":"2024-02-09T18:04:24.564920+0000","last_clean":"2024-02-09T18:04:24.564920+0000","last_became_active":"2024-02-09T17:59:12.208604+0000","last_became_peered":"2024-02-09T17:59:12.208604+0000","last_unstale":"2024-02-09T18:04:24.564920+0000","last_undegraded":"2024-02-09T18:04:24.564920+0000","last_fullsized":"2024-02-09T18:04:24.564920+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:02:57.599801+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:14.135321+0000","last_clean_scrub_stamp":"2024-02-09T18:02:57.599801+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"972'16099","reported_seq":16416,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-02-09T18:04:27.127479+0000","last_change":"2024-02-09T18:03:44.836700+0000","last_active":"2024-02-09T18:04:27.127479+0000","last_peered":"2024-02-09T18:04:27.127479+0000","last_clean":"2024-02-09T18:04:27.127479+0000","last_became_active":"2024-02-09T18:00:07.530724+0000","last_became_peered":"2024-02-09T18:00:07.530724+0000","last_unstale":"2024-02-09T18:04:27.127479+0000","last_undegraded":"2024-02-09T18:04:27.127479+0000","last_fullsized":"2024-02-09T18:04:27.127479+0000","mapping_epoch":882,"log_start":"905'6000","ondisk_log_start":"905'6000","created":882,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"959'13965","last_scrub_stamp":"2024-02-09T18:03:44.836459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T18:00:06.282957+0000","last_clean_scrub_stamp":"2024-02-09T18:03:44.836459+0000","objects_scrubbed":2976,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:04:52.712515+0000","scrub_duration":803,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53280768,"num_objects":813,"num_object_clones":0,"num_object_copies":2439,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16099,"num_write_kb":541184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"63'139","reported_seq":1446,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-02-09T18:04:26.613668+0000","last_change":"2024-02-09T18:03:23.550457+0000","last_active":"2024-02-09T18:04:26.613668+0000","last_peered":"2024-02-09T18:04:26.613668+0000","last_clean":"2024-02-09T18:04:26.613668+0000","last_became_active":"2024-02-09T18:03:02.132990+0000","last_became_peered":"2024-02-09T18:03:02.132990+0000","last_unstale":"2024-02-09T18:04:26.613668+0000","last_undegraded":"2024-02-09T18:04:26.613668+0000","last_fullsized":"2024-02-09T18:04:26.613668+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:03:23.550293+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T18:00:49.890794+0000","last_clean_scrub_stamp":"2024-02-09T18:03:23.550293+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:04:46.320553+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"63'139","reported_seq":1608,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-02-09T18:04:26.608553+0000","last_change":"2024-02-09T18:04:15.439255+0000","last_active":"2024-02-09T18:04:26.608553+0000","last_peered":"2024-02-09T18:04:26.608553+0000","last_clean":"2024-02-09T18:04:26.608553+0000","last_became_active":"2024-02-09T18:03:17.456089+0000","last_became_peered":"2024-02-09T18:03:17.456089+0000","last_unstale":"2024-02-09T18:04:26.608553+0000","last_undegraded":"2024-02-09T18:04:26.608553+0000","last_fullsized":"2024-02-09T18:04:26.608553+0000","mapping_epoch":945,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":946,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:04:15.439120+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:48:35.170638+0000","last_clean_scrub_stamp":"2024-02-09T18:04:15.439120+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:05:22.356551+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"63'139","reported_seq":1862,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-02-09T18:04:26.620552+0000","last_change":"2024-02-09T18:03:41.553376+0000","last_active":"2024-02-09T18:04:26.620552+0000","last_peered":"2024-02-09T18:04:26.620552+0000","last_clean":"2024-02-09T18:04:26.620552+0000","last_became_active":"2024-02-09T17:55:21.231902+0000","last_became_peered":"2024-02-09T17:55:21.231902+0000","last_unstale":"2024-02-09T18:04:26.620552+0000","last_undegraded":"2024-02-09T18:04:26.620552+0000","last_fullsized":"2024-02-09T18:04:26.620552+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":791,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:03:41.553250+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:44:51.190789+0000","last_clean_scrub_stamp":"2024-02-09T18:03:41.553250+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:05:04.872775+0000","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.12","version":"63'139","reported_seq":1935,"reported_epoch":971,"state":"active+clean","last_fresh":"2024-02-09T18:04:24.621479+0000","last_change":"2024-02-09T18:02:02.359151+0000","last_active":"2024-02-09T18:04:24.621479+0000","last_peered":"2024-02-09T18:04:24.621479+0000","last_clean":"2024-02-09T18:04:24.621479+0000","last_became_active":"2024-02-09T17:46:47.015622+0000","last_became_peered":"2024-02-09T17:46:47.015622+0000","last_unstale":"2024-02-09T18:04:24.621479+0000","last_undegraded":"2024-02-09T18:04:24.621479+0000","last_fullsized":"2024-02-09T18:04:24.621479+0000","mapping_epoch":600,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:02:02.359039+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:02:02.359039+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.13","version":"63'139","reported_seq":1563,"reported_epoch":971,"state":"active+clean","last_fresh":"2024-02-09T18:04:24.621684+0000","last_change":"2024-02-09T18:03:17.470154+0000","last_active":"2024-02-09T18:04:24.621684+0000","last_peered":"2024-02-09T18:04:24.621684+0000","last_clean":"2024-02-09T18:04:24.621684+0000","last_became_active":"2024-02-09T18:03:17.469790+0000","last_became_peered":"2024-02-09T18:03:17.469790+0000","last_unstale":"2024-02-09T18:04:24.621684+0000","last_undegraded":"2024-02-09T18:04:24.621684+0000","last_fullsized":"2024-02-09T18:04:24.621684+0000","mapping_epoch":945,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":946,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:02:01.517864+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:02:01.517864+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":293,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":877461528,"num_objects":13390,"num_object_clones":0,"num_object_copies":40170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13390,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":256370,"num_write_kb":8632257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23980,"num_bytes_recovered":1571487744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2977968128,"data_stored":2977955912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160713,"ondisk_log_size":160713,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547057147,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1187096,"kb_used_data":217420,"kb_used_omap":1,"kb_used_meta":969662,"kb_avail":92558056,"statfs":{"total":95995035648,"available":94779449344,"internally_reserved":0,"allocated":222638080,"data_stored":222335859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":992934340},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 18:04:25 2024","interfaces":[{"interface":"back","average":{"1min":1.732,"5min":1.597,"15min":1.752},"min":{"1min":0.248,"5min":0.22800000000000001,"15min":0.21199999999999999},"max":{"1min":9.1669999999999998,"5min":13.307,"15min":20.003},"last":0.48499999999999999},{"interface":"front","average":{"1min":1.915,"5min":1.5429999999999999,"15min":1.6970000000000001},"min":{"1min":0.26600000000000001,"5min":0.23899999999999999,"15min":0.23100000000000001},"max":{"1min":8.6539999999999999,"5min":8.6539999999999999,"15min":9.6259999999999994},"last":0.28599999999999998}]},{"osd":1,"last update":"Fri Feb 9 18:04:25 2024","interfaces":[{"interface":"back","average":{"1min":2.2789999999999999,"5min":1.9750000000000001,"15min":1.9650000000000001},"min":{"1min":0.28399999999999997,"5min":0.23100000000000001,"15min":0.22600000000000001},"max":{"1min":8.6140000000000008,"5min":21.616,"15min":21.616},"last":2.4140000000000001},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.5589999999999999,"15min":1.958},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.25},"max":{"1min":6.21,"5min":22.456,"15min":22.456},"last":3.4060000000000001}]},{"osd":2,"last update":"Fri Feb 9 18:04:25 2024","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.4399999999999999,"15min":1.7310000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.25600000000000001},"max":{"1min":6.7359999999999998,"5min":8.3290000000000006,"15min":14.766999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.5660000000000001,"15min":1.47},"min":{"1min":0.35399999999999998,"5min":0.316,"15min":0.27800000000000002},"max":{"1min":4.0430000000000001,"5min":13.337,"15min":13.337},"last":2.673}]},{"osd":3,"last update":"Fri Feb 9 18:04:25 2024","interfaces":[{"interface":"back","average":{"1min":2.4020000000000001,"5min":2.4569999999999999,"15min":1.8959999999999999},"min":{"1min":0.34899999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":8.7080000000000002,"5min":41.395000000000003,"15min":41.395000000000003},"last":2.1269999999999998},{"interface":"front","average":{"1min":1.8140000000000001,"5min":2.2290000000000001,"15min":2.077},"min":{"1min":0.39400000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":6.0350000000000001,"5min":41.276000000000003,"15min":41.276000000000003},"last":2.2890000000000001}]},{"osd":4,"last update":"Fri Feb 9 17:58:44 2024","interfaces":[{"interface":"back","average":{"1min":3.1709999999999998,"5min":2.5779999999999998,"15min":2.4790000000000001},"min":{"1min":0.40999999999999998,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":14.866,"5min":14.866,"15min":14.866},"last":0.45700000000000002},{"interface":"front","average":{"1min":3.2349999999999999,"5min":2.5640000000000001,"15min":2.452},"min":{"1min":0.496,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":14.939,"5min":14.939,"15min":14.939},"last":0.42299999999999999}]},{"osd":5,"last update":"Fri Feb 9 18:04:25 2024","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":1.46,"15min":1.516},"min":{"1min":0.30199999999999999,"5min":0.29099999999999998,"15min":0.27000000000000002},"max":{"1min":5.6319999999999997,"5min":13.263999999999999,"15min":13.263999999999999},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.88900000000000001,"5min":1.115,"15min":1.615},"min":{"1min":0.33900000000000002,"5min":0.317,"15min":0.29199999999999998},"max":{"1min":3.7730000000000001,"5min":8.3030000000000008,"15min":20.038},"last":0.46800000000000003}]},{"osd":6,"last update":"Fri Feb 9 18:04:25 2024","interfaces":[{"interface":"back","average":{"1min":2.355,"5min":1.6479999999999999,"15min":1.7509999999999999},"min":{"1min":0.35499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":8.7370000000000001,"5min":8.7370000000000001,"15min":11.885},"last":0.35499999999999998},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.264,"15min":1.714},"min":{"1min":0.38900000000000001,"5min":0.35699999999999998,"15min":0.32400000000000001},"max":{"1min":6.4569999999999999,"5min":8.2699999999999996,"15min":15.409000000000001},"last":0.42299999999999999}]},{"osd":7,"last update":"Fri Feb 9 18:04:25 2024","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":1.365,"15min":1.716},"min":{"1min":0.36699999999999999,"5min":0.32600000000000001,"15min":0.317},"max":{"1min":6.069,"5min":8.2390000000000008,"15min":15.51},"last":0.499},{"interface":"front","average":{"1min":2.1160000000000001,"5min":1.7230000000000001,"15min":1.931},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.318},"max":{"1min":8.6829999999999998,"5min":8.6829999999999998,"15min":24.373999999999999},"last":0.38600000000000001}]},{"osd":8,"last update":"Fri Feb 9 18:03:30 2024","interfaces":[{"interface":"back","average":{"1min":2.2080000000000002,"5min":2.1150000000000002,"15min":2.0790000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.27900000000000003},"max":{"1min":7.266,"5min":9.3710000000000004,"15min":29.233000000000001},"last":0.63500000000000001},{"interface":"front","average":{"1min":2.2639999999999998,"5min":2.3010000000000002,"15min":2.1819999999999999},"min":{"1min":0.46500000000000002,"5min":0.35899999999999999,"15min":0.32700000000000001},"max":{"1min":7.2990000000000004,"5min":14.115,"15min":29.091999999999999},"last":0.65800000000000003}]},{"osd":9,"last update":"Fri Feb 9 18:03:55 2024","interfaces":[{"interface":"back","average":{"1min":2.0760000000000001,"5min":2.024,"15min":2.2010000000000001},"min":{"1min":0.34499999999999997,"5min":0.32000000000000001,"15min":0.313},"max":{"1min":12.68,"5min":13.363,"15min":22.613},"last":0.39700000000000002},{"interface":"front","average":{"1min":2.1240000000000001,"5min":2.081,"15min":2.0310000000000001},"min":{"1min":0.38,"5min":0.35899999999999999,"15min":0.35299999999999998},"max":{"1min":12.707000000000001,"5min":13.390000000000001,"15min":19.555},"last":0.44600000000000001}]},{"osd":10,"last update":"Fri Feb 9 18:04:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.653,"15min":2.363},"min":{"1min":0.42899999999999999,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":3.7370000000000001,"5min":13.254,"15min":25.093},"last":0.61699999999999999},{"interface":"front","average":{"1min":1.071,"5min":1.71,"15min":2.0550000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.35999999999999999},"max":{"1min":3.831,"5min":13.295,"15min":20.233000000000001},"last":0.67400000000000004}]}]},{"osd":10,"up_from":0,"seq":360777253379,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":9,"up_from":77,"seq":330712482309,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1454616,"kb_used_data":400268,"kb_used_omap":1,"kb_used_meta":1054334,"kb_avail":92290536,"statfs":{"total":95995035648,"available":94505508864,"internally_reserved":0,"allocated":409874432,"data_stored":409572211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":1079638464},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 18:03:33 2024","interfaces":[{"interface":"back","average":{"1min":1.792,"5min":1.393,"15min":1.2529999999999999},"min":{"1min":0.254,"5min":0.219,"15min":0.219},"max":{"1min":8.4469999999999992,"5min":11.515000000000001,"15min":11.515000000000001},"last":0.71499999999999997},{"interface":"front","average":{"1min":1.6850000000000001,"5min":1.2669999999999999,"15min":1.0229999999999999},"min":{"1min":0.314,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":12.521000000000001,"5min":12.521000000000001,"15min":12.521000000000001},"last":0.82999999999999996}]},{"osd":1,"last update":"Fri Feb 9 18:04:24 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.4279999999999999,"15min":1.728},"min":{"1min":0.252,"5min":0.252,"15min":0.23599999999999999},"max":{"1min":3.972,"5min":7.4790000000000001,"15min":61.031999999999996},"last":0.60999999999999999},{"interface":"front","average":{"1min":1.607,"5min":1.6419999999999999,"15min":2.101},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25600000000000001},"max":{"1min":7.2149999999999999,"5min":9.0670000000000002,"15min":61.970999999999997},"last":0.70399999999999996}]},{"osd":2,"last update":"Fri Feb 9 18:04:17 2024","interfaces":[{"interface":"back","average":{"1min":1.8320000000000001,"5min":1.8320000000000001,"15min":1.8320000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":7.6820000000000004,"5min":7.6820000000000004,"15min":7.6820000000000004},"last":0.74099999999999999},{"interface":"front","average":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":5.0300000000000002,"5min":5.0300000000000002,"15min":5.0300000000000002},"last":0.75900000000000001}]},{"osd":3,"last update":"Fri Feb 9 18:04:24 2024","interfaces":[{"interface":"back","average":{"1min":2.085,"5min":1.548,"15min":1.1419999999999999},"min":{"1min":0.28499999999999998,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":6.2590000000000003,"5min":6.3399999999999999,"15min":6.3399999999999999},"last":0.748},{"interface":"front","average":{"1min":2.218,"5min":1.3839999999999999,"15min":1.0640000000000001},"min":{"1min":0.32500000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":7.7290000000000001,"5min":7.7290000000000001,"15min":7.7290000000000001},"last":0.79400000000000004}]},{"osd":4,"last update":"Fri Feb 9 18:03:24 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":1.609,"15min":1.5369999999999999},"min":{"1min":0.29999999999999999,"5min":0.28899999999999998,"15min":0.26900000000000002},"max":{"1min":5.6020000000000003,"5min":10.725,"15min":19.414000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.2569999999999999,"15min":1.3440000000000001},"min":{"1min":0.39700000000000002,"5min":0.30099999999999999,"15min":0.29999999999999999},"max":{"1min":7.7169999999999996,"5min":7.7169999999999996,"15min":19.224},"last":0.77800000000000002}]},{"osd":5,"last update":"Fri Feb 9 18:03:24 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.5740000000000001,"15min":1.4910000000000001},"min":{"1min":0.35899999999999999,"5min":0.33200000000000002,"15min":0.32100000000000001},"max":{"1min":7.8120000000000003,"5min":10.763999999999999,"15min":10.763999999999999},"last":0.72999999999999998},{"interface":"front","average":{"1min":1.9690000000000001,"5min":1.8169999999999999,"15min":1.478},"min":{"1min":0.42599999999999999,"5min":0.35999999999999999,"15min":0.28000000000000003},"max":{"1min":7.7569999999999997,"5min":10.851000000000001,"15min":10.851000000000001},"last":0.69299999999999995}]},{"osd":6,"last update":"Fri Feb 9 18:04:24 2024","interfaces":[{"interface":"back","average":{"1min":1.6779999999999999,"5min":1.663,"15min":1.5860000000000001},"min":{"1min":0.35899999999999999,"5min":0.34499999999999997,"15min":0.33400000000000002},"max":{"1min":7.2910000000000004,"5min":9.1980000000000004,"15min":19.375},"last":0.629},{"interface":"front","average":{"1min":1.8109999999999999,"5min":1.7150000000000001,"15min":1.6359999999999999},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.33500000000000002},"max":{"1min":7.3869999999999996,"5min":9.1449999999999996,"15min":19.178000000000001},"last":0.68300000000000005}]},{"osd":7,"last update":"Fri Feb 9 18:02:55 2024","interfaces":[{"interface":"back","average":{"1min":1.9570000000000001,"5min":1.724,"15min":2.0059999999999998},"min":{"1min":0.35699999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":8.1899999999999995,"5min":8.8529999999999998,"15min":10.021000000000001},"last":0.623},{"interface":"front","average":{"1min":1.605,"5min":2.004,"15min":1.972},"min":{"1min":0.46000000000000002,"5min":0.38100000000000001,"15min":0.379},"max":{"1min":7.7839999999999998,"5min":10.802,"15min":11.138},"last":0.748}]},{"osd":8,"last update":"Fri Feb 9 18:03:33 2024","interfaces":[{"interface":"back","average":{"1min":2.8999999999999999,"5min":1.927,"15min":1.6950000000000001},"min":{"1min":0.34599999999999997,"5min":0.30599999999999999,"15min":0.29599999999999999},"max":{"1min":10.569000000000001,"5min":10.569000000000001,"15min":10.744},"last":0.51700000000000002},{"interface":"front","average":{"1min":2.9630000000000001,"5min":2.0449999999999999,"15min":1.762},"min":{"1min":0.44,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":10.601000000000001,"5min":10.601000000000001,"15min":10.768000000000001},"last":0.64800000000000002}]},{"osd":10,"last update":"Fri Feb 9 18:03:48 2024","interfaces":[{"interface":"back","average":{"1min":2.5699999999999998,"5min":2.218,"15min":2.234},"min":{"1min":0.42599999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":7.3380000000000001,"5min":8.6349999999999998,"15min":20.773},"last":0.54100000000000004},{"interface":"front","average":{"1min":2.6949999999999998,"5min":2.274,"15min":2.194},"min":{"1min":0.42999999999999999,"5min":0.40000000000000002,"15min":0.34699999999999998},"max":{"1min":8.3870000000000005,"5min":14.528,"15min":20.637},"last":0.57699999999999996}]},{"osd":11,"last update":"Fri Feb 9 18:03:48 2024","interfaces":[{"interface":"back","average":{"1min":2.121,"5min":2.2440000000000002,"15min":2.0819999999999999},"min":{"1min":0.51900000000000002,"5min":0.35299999999999998,"15min":0.33000000000000002},"max":{"1min":9.7149999999999999,"5min":40.259999999999998,"15min":40.259999999999998},"last":0.63600000000000001},{"interface":"front","average":{"1min":2.1600000000000001,"5min":2.4220000000000002,"15min":2.1360000000000001},"min":{"1min":0.59299999999999997,"5min":0.49099999999999999,"15min":0.39000000000000001},"max":{"1min":9.7400000000000002,"5min":39.683,"15min":39.683},"last":0.81299999999999994}]}]},{"osd":8,"up_from":171,"seq":734439408052,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":618264,"kb_used_data":107788,"kb_used_omap":212,"kb_used_meta":510251,"kb_avail":93126888,"statfs":{"total":95995035648,"available":95361933312,"internally_reserved":0,"allocated":110374912,"data_stored":110072691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217579,"internal_metadata":522497557},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 18:03:26 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.5169999999999999,"15min":1.7509999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22600000000000001},"max":{"1min":4.1500000000000004,"5min":10.099,"15min":12.712999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":1.706,"5min":1.8080000000000001,"15min":1.6699999999999999},"min":{"1min":0.26300000000000001,"5min":0.25600000000000001,"15min":0.23400000000000001},"max":{"1min":7.1020000000000003,"5min":18.081,"15min":18.081},"last":0.57299999999999995}]},{"osd":1,"last update":"Fri Feb 9 18:03:26 2024","interfaces":[{"interface":"back","average":{"1min":2.6629999999999998,"5min":1.5409999999999999,"15min":1.609},"min":{"1min":0.26900000000000002,"5min":0.248,"15min":0.23799999999999999},"max":{"1min":7.1070000000000002,"5min":7.1070000000000002,"15min":8.2550000000000008},"last":39.734000000000002},{"interface":"front","average":{"1min":2.335,"5min":2.1309999999999998,"15min":1.9750000000000001},"min":{"1min":0.23599999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":17.035,"5min":17.988,"15min":21.399000000000001},"last":39.097000000000001}]},{"osd":2,"last update":"Fri Feb 9 18:03:26 2024","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.6679999999999999,"15min":1.4279999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.25},"max":{"1min":4.4699999999999998,"5min":6.7119999999999997,"15min":6.9260000000000002},"last":0.50600000000000001},{"interface":"front","average":{"1min":2.02,"5min":2.1680000000000001,"15min":1.901},"min":{"1min":0.316,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":7.6239999999999997,"5min":18.045000000000002,"15min":18.045000000000002},"last":0.47399999999999998}]},{"osd":3,"last update":"Fri Feb 9 17:56:20 2024","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":2.552,"15min":2.8039999999999998},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":5.2560000000000002,"5min":15.654,"15min":15.654},"last":6.5350000000000001},{"interface":"front","average":{"1min":1.306,"5min":2.3149999999999999,"15min":2.0150000000000001},"min":{"1min":0.34300000000000003,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":5.2759999999999998,"5min":15.609999999999999,"15min":15.609999999999999},"last":6.5599999999999996}]},{"osd":4,"last update":"Fri Feb 9 18:03:26 2024","interfaces":[{"interface":"back","average":{"1min":2.552,"5min":1.6120000000000001,"15min":1.357},"min":{"1min":0.35499999999999998,"5min":0.309,"15min":0.29399999999999998},"max":{"1min":7.5149999999999997,"5min":12.297000000000001,"15min":12.297000000000001},"last":0.59499999999999997},{"interface":"front","average":{"1min":2.3959999999999999,"5min":1.5329999999999999,"15min":1.3819999999999999},"min":{"1min":0.33100000000000002,"5min":0.28499999999999998,"15min":0.28000000000000003},"max":{"1min":7.1760000000000002,"5min":12.227,"15min":12.227},"last":0.71299999999999997}]},{"osd":5,"last update":"Fri Feb 9 18:03:53 2024","interfaces":[{"interface":"back","average":{"1min":2.0830000000000002,"5min":1.6779999999999999,"15min":1.607},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.27400000000000002},"max":{"1min":13.68,"5min":13.68,"15min":13.68},"last":0.753},{"interface":"front","average":{"1min":2.1509999999999998,"5min":1.744,"15min":1.5},"min":{"1min":0.377,"5min":0.377,"15min":0.30299999999999999},"max":{"1min":13.773,"5min":13.773,"15min":13.773},"last":0.77900000000000003}]},{"osd":6,"last update":"Fri Feb 9 18:03:26 2024","interfaces":[{"interface":"back","average":{"1min":2.758,"5min":1.5549999999999999,"15min":1.633},"min":{"1min":0.36099999999999999,"5min":0.33300000000000002,"15min":0.313},"max":{"1min":7.2000000000000002,"5min":7.2000000000000002,"15min":9.7539999999999996},"last":0.74199999999999999},{"interface":"front","average":{"1min":1.6930000000000001,"5min":1.7390000000000001,"15min":1.625},"min":{"1min":0.34000000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":5.0940000000000003,"5min":10.178000000000001,"15min":10.178000000000001},"last":0.73099999999999998}]},{"osd":7,"last update":"Fri Feb 9 18:03:26 2024","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.754,"15min":1.7210000000000001},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.32900000000000001},"max":{"1min":4.5229999999999997,"5min":12.380000000000001,"15min":12.380000000000001},"last":0.68000000000000005},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.726,"15min":1.6419999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.309},"max":{"1min":4.2069999999999999,"5min":10.148999999999999,"15min":10.148999999999999},"last":0.69899999999999995}]},{"osd":9,"last update":"Fri Feb 9 18:03:26 2024","interfaces":[{"interface":"back","average":{"1min":1.9139999999999999,"5min":2.9780000000000002,"15min":2.4009999999999998},"min":{"1min":0.371,"5min":0.34999999999999998,"15min":0.32700000000000001},"max":{"1min":5.4740000000000002,"5min":46.213999999999999,"15min":46.213999999999999},"last":0.432},{"interface":"front","average":{"1min":1.7310000000000001,"5min":2.8809999999999998,"15min":2.427},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":9.9120000000000008,"5min":41.177,"15min":41.177},"last":0.55400000000000005}]},{"osd":10,"last update":"Fri Feb 9 18:03:26 2024","interfaces":[{"interface":"back","average":{"1min":2.5449999999999999,"5min":2.4620000000000002,"15min":2.0840000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.32300000000000001},"max":{"1min":9.1639999999999997,"5min":18.164000000000001,"15min":18.164000000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":2.843,"5min":2.0129999999999999,"15min":2.1499999999999999},"min":{"1min":0.442,"5min":0.41099999999999998,"15min":0.38700000000000001},"max":{"1min":7.1479999999999997,"5min":18.111000000000001,"15min":18.111000000000001},"last":0.52900000000000003}]},{"osd":11,"last update":"Fri Feb 9 18:03:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3600000000000001,"5min":2.044,"15min":2.0339999999999998},"min":{"1min":0.48699999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":5.9269999999999996,"5min":10.231999999999999,"15min":10.231999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":1.4139999999999999,"5min":2.093,"15min":2.0750000000000002},"min":{"1min":0.55100000000000005,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":5.9550000000000001,"5min":10.256,"15min":10.256},"last":0.66800000000000004}]}]},{"osd":7,"up_from":57,"seq":244813136398,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1159444,"kb_used_data":266124,"kb_used_omap":1,"kb_used_meta":893310,"kb_avail":92585708,"statfs":{"total":95995035648,"available":94807764992,"internally_reserved":0,"allocated":272510976,"data_stored":272205617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":914749898},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 18:04:04 2024","interfaces":[{"interface":"back","average":{"1min":1.8859999999999999,"5min":1.01,"15min":1.3859999999999999},"min":{"1min":0.318,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":11.489000000000001,"5min":11.489000000000001,"15min":101.167},"last":4.601},{"interface":"front","average":{"1min":2.609,"5min":1.5349999999999999,"15min":1.482},"min":{"1min":0.35399999999999998,"5min":0.33200000000000002,"15min":0.27400000000000002},"max":{"1min":7.3700000000000001,"5min":15.890000000000001,"15min":101.07599999999999},"last":4.6150000000000002}]},{"osd":1,"last update":"Fri Feb 9 18:04:04 2024","interfaces":[{"interface":"back","average":{"1min":3.8140000000000001,"5min":1.8899999999999999,"15min":1.544},"min":{"1min":0.31,"5min":0.29399999999999998,"15min":0.28299999999999997},"max":{"1min":18.062999999999999,"5min":18.062999999999999,"15min":18.062999999999999},"last":1.5700000000000001},{"interface":"front","average":{"1min":3.8900000000000001,"5min":2.0299999999999998,"15min":1.5149999999999999},"min":{"1min":0.36099999999999999,"5min":0.25700000000000001,"15min":0.246},"max":{"1min":18.315000000000001,"5min":18.315000000000001,"15min":18.315000000000001},"last":3.577}]},{"osd":2,"last update":"Fri Feb 9 18:04:04 2024","interfaces":[{"interface":"back","average":{"1min":3.0190000000000001,"5min":1.915,"15min":1.5149999999999999},"min":{"1min":0.33400000000000002,"5min":0.27300000000000002,"15min":0.25},"max":{"1min":7.4189999999999996,"5min":16.190000000000001,"15min":16.190000000000001},"last":7.6159999999999997},{"interface":"front","average":{"1min":3.0779999999999998,"5min":2.1190000000000002,"15min":1.647},"min":{"1min":0.44800000000000001,"5min":0.35999999999999999,"15min":0.28999999999999998},"max":{"1min":7.4909999999999997,"5min":17.596,"15min":17.596},"last":7.6429999999999998}]},{"osd":3,"last update":"Fri Feb 9 18:04:23 2024","interfaces":[{"interface":"back","average":{"1min":2.633,"5min":1.8089999999999999,"15min":1.4159999999999999},"min":{"1min":0.379,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":7.6340000000000003,"5min":7.6340000000000003,"15min":7.6340000000000003},"last":0.46100000000000002},{"interface":"front","average":{"1min":2.8300000000000001,"5min":2.3540000000000001,"15min":2.278},"min":{"1min":0.40500000000000003,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":10.089,"5min":16.029,"15min":16.029},"last":2.3170000000000002}]},{"osd":4,"last update":"Fri Feb 9 18:04:04 2024","interfaces":[{"interface":"back","average":{"1min":1.97,"5min":0.98499999999999999,"15min":0.80700000000000005},"min":{"1min":0.312,"5min":0.28599999999999998,"15min":0.25900000000000001},"max":{"1min":5.9740000000000002,"5min":5.9740000000000002,"15min":5.9740000000000002},"last":0.376},{"interface":"front","average":{"1min":2.6739999999999999,"5min":1.4710000000000001,"15min":1.099},"min":{"1min":0.28000000000000003,"5min":0.27600000000000002,"15min":0.26300000000000001},"max":{"1min":11.433999999999999,"5min":17.814,"15min":17.814},"last":0.53000000000000003}]},{"osd":5,"last update":"Fri Feb 9 18:04:04 2024","interfaces":[{"interface":"back","average":{"1min":2.2829999999999999,"5min":1.165,"15min":0.95099999999999996},"min":{"1min":0.45300000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":11.406000000000001,"5min":11.406000000000001,"15min":11.406000000000001},"last":0.44400000000000001},{"interface":"front","average":{"1min":2.3170000000000002,"5min":1.3420000000000001,"15min":1.1020000000000001},"min":{"1min":0.34499999999999997,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":7.6719999999999997,"5min":7.6719999999999997,"15min":7.867},"last":0.58699999999999997}]},{"osd":6,"last update":"Fri Feb 9 18:04:23 2024","interfaces":[{"interface":"back","average":{"1min":1.992,"5min":1.742,"15min":1.2170000000000001},"min":{"1min":0.38,"5min":0.35499999999999998,"15min":0.311},"max":{"1min":5.8769999999999998,"5min":16.158000000000001,"15min":16.158000000000001},"last":0.504},{"interface":"front","average":{"1min":1.5720000000000001,"5min":1.5580000000000001,"15min":1.081},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35899999999999999},"max":{"1min":6.5259999999999998,"5min":8.1259999999999994,"15min":8.1259999999999994},"last":0.39300000000000002}]},{"osd":8,"last update":"Fri Feb 9 18:04:23 2024","interfaces":[{"interface":"back","average":{"1min":1.986,"5min":1.403,"15min":1.5089999999999999},"min":{"1min":0.39300000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":7.9000000000000004,"5min":11.358000000000001,"15min":11.648999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":2.0859999999999999,"5min":1.8280000000000001,"15min":1.623},"min":{"1min":0.51700000000000002,"5min":0.379,"15min":0.37},"max":{"1min":5.8499999999999996,"5min":10.818,"15min":11.629},"last":0.68600000000000005}]},{"osd":9,"last update":"Fri Feb 9 18:04:12 2024","interfaces":[{"interface":"back","average":{"1min":2.2749999999999999,"5min":1.8,"15min":1.698},"min":{"1min":0.41799999999999998,"5min":0.41499999999999998,"15min":0.38900000000000001},"max":{"1min":7.9900000000000002,"5min":11.465,"15min":11.542999999999999},"last":0.48399999999999999},{"interface":"front","average":{"1min":2.4020000000000001,"5min":2.1259999999999999,"15min":1.841},"min":{"1min":0.497,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":7.9390000000000001,"5min":13.161,"15min":15.852},"last":2.3319999999999999}]},{"osd":10,"last update":"Fri Feb 9 18:03:46 2024","interfaces":[{"interface":"back","average":{"1min":2.6760000000000002,"5min":2.3149999999999999,"15min":1.728},"min":{"1min":0.53700000000000003,"5min":0.44900000000000001,"15min":0.32800000000000001},"max":{"1min":7.694,"5min":15.999000000000001,"15min":15.999000000000001},"last":0.54700000000000004},{"interface":"front","average":{"1min":2.3610000000000002,"5min":1.8819999999999999,"15min":1.8380000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.40400000000000003},"max":{"1min":11.515000000000001,"5min":11.515000000000001,"15min":11.571999999999999},"last":5.2720000000000002}]},{"osd":11,"last update":"Fri Feb 9 18:04:04 2024","interfaces":[{"interface":"back","average":{"1min":3.1659999999999999,"5min":2.1080000000000001,"15min":1.7450000000000001},"min":{"1min":0.48999999999999999,"5min":0.47299999999999998,"15min":0.375},"max":{"1min":9.1890000000000001,"5min":16.111999999999998,"15min":16.111999999999998},"last":0.67300000000000004},{"interface":"front","average":{"1min":2.73,"5min":2.0870000000000002,"15min":1.833},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.44},"max":{"1min":10.215999999999999,"5min":10.215999999999999,"15min":12.243},"last":0.76600000000000001}]}]},{"osd":6,"up_from":49,"seq":210453398037,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":849040,"kb_used_data":119552,"kb_used_omap":1,"kb_used_meta":729470,"kb_avail":92896112,"statfs":{"total":95995035648,"available":95125618688,"internally_reserved":0,"allocated":122421248,"data_stored":122123513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":746977734},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 18:03:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":0.70299999999999996,"15min":1.0569999999999999},"min":{"1min":0.33100000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":7.3630000000000004,"5min":7.3630000000000004,"15min":10.481999999999999},"last":1.143},{"interface":"front","average":{"1min":1.254,"5min":0.872,"15min":1.121},"min":{"1min":0.27200000000000002,"5min":0.26700000000000002,"15min":0.24399999999999999},"max":{"1min":7.407,"5min":7.407,"15min":10.436},"last":1.2250000000000001}]},{"osd":1,"last update":"Fri Feb 9 18:03:49 2024","interfaces":[{"interface":"back","average":{"1min":3.3319999999999999,"5min":1.3460000000000001,"15min":1.423},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.25},"max":{"1min":32.064,"5min":32.064,"15min":32.064},"last":0.99099999999999999},{"interface":"front","average":{"1min":3.4039999999999999,"5min":1.4079999999999999,"15min":1.478},"min":{"1min":0.36699999999999999,"5min":0.309,"15min":0.28599999999999998},"max":{"1min":32.027000000000001,"5min":32.027000000000001,"15min":32.027000000000001},"last":1.026}]},{"osd":2,"last update":"Fri Feb 9 18:03:49 2024","interfaces":[{"interface":"back","average":{"1min":1.7509999999999999,"5min":1.3580000000000001,"15min":1.3720000000000001},"min":{"1min":0.38500000000000001,"5min":0.31900000000000001,"15min":0.28100000000000003},"max":{"1min":7.4779999999999998,"5min":9.2970000000000006,"15min":9.2970000000000006},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.792,"5min":1.417,"15min":1.423},"min":{"1min":0.45500000000000002,"5min":0.39000000000000001,"15min":0.34599999999999997},"max":{"1min":7.5659999999999998,"5min":9.3260000000000005,"15min":9.3260000000000005},"last":1.1950000000000001}]},{"osd":3,"last update":"Fri Feb 9 18:03:36 2024","interfaces":[{"interface":"back","average":{"1min":1.7330000000000001,"5min":1.2130000000000001,"15min":0.877},"min":{"1min":0.33700000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":7.5030000000000001,"5min":7.5030000000000001,"15min":7.5030000000000001},"last":2.9660000000000002},{"interface":"front","average":{"1min":1.2310000000000001,"5min":1.1160000000000001,"15min":1.1599999999999999},"min":{"1min":0.36299999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":8.1760000000000002,"5min":9.0909999999999993,"15min":9.0909999999999993},"last":0.93000000000000005}]},{"osd":4,"last update":"Fri Feb 9 18:03:49 2024","interfaces":[{"interface":"back","average":{"1min":1.6870000000000001,"5min":0.94899999999999995,"15min":1.032},"min":{"1min":0.29499999999999998,"5min":0.27600000000000002,"15min":0.26200000000000001},"max":{"1min":10.215,"5min":10.215,"15min":14.182},"last":0.38800000000000001},{"interface":"front","average":{"1min":1.7430000000000001,"5min":1.004,"15min":1.093},"min":{"1min":0.37,"5min":0.33100000000000002,"15min":0.314},"max":{"1min":10.256,"5min":10.256,"15min":14.210000000000001},"last":0.42499999999999999}]},{"osd":5,"last update":"Fri Feb 9 18:03:49 2024","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":0.71899999999999997,"15min":1.0409999999999999},"min":{"1min":0.36099999999999999,"5min":0.29399999999999998,"15min":0.26600000000000001},"max":{"1min":8.1389999999999993,"5min":8.1389999999999993,"15min":14.151999999999999},"last":0.88500000000000001},{"interface":"front","average":{"1min":1.6619999999999999,"5min":0.85499999999999998,"15min":0.98899999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.26100000000000001},"max":{"1min":7.4470000000000001,"5min":7.4470000000000001,"15min":8.9629999999999992},"last":1.113}]},{"osd":7,"last update":"Fri Feb 9 18:03:28 2024","interfaces":[{"interface":"back","average":{"1min":1.9550000000000001,"5min":1.032,"15min":1.095},"min":{"1min":0.437,"5min":0.314,"15min":0.28699999999999998},"max":{"1min":13.991,"5min":13.991,"15min":13.991},"last":0.82199999999999995},{"interface":"front","average":{"1min":1.569,"5min":1.002,"15min":0.96799999999999997},"min":{"1min":0.40400000000000003,"5min":0.32200000000000001,"15min":0.30299999999999999},"max":{"1min":8.4870000000000001,"5min":8.4870000000000001,"15min":8.4870000000000001},"last":0.85099999999999998}]},{"osd":8,"last update":"Fri Feb 9 17:58:34 2024","interfaces":[{"interface":"back","average":{"1min":2.806,"5min":1.635,"15min":1.27},"min":{"1min":0.41599999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":9.2940000000000005,"5min":9.2940000000000005,"15min":9.9480000000000004},"last":0.40300000000000002},{"interface":"front","average":{"1min":2.3780000000000001,"5min":1.623,"15min":1.3109999999999999},"min":{"1min":0.53400000000000003,"5min":0.41399999999999998,"15min":0.40400000000000003},"max":{"1min":9.3469999999999995,"5min":9.3469999999999995,"15min":11.316000000000001},"last":0.57899999999999996}]},{"osd":9,"last update":"Fri Feb 9 18:03:57 2024","interfaces":[{"interface":"back","average":{"1min":1.903,"5min":1.361,"15min":1.655},"min":{"1min":0.51800000000000002,"5min":0.36299999999999999,"15min":0.32600000000000001},"max":{"1min":7.5090000000000003,"5min":7.5090000000000003,"15min":36.606000000000002},"last":1.0580000000000001},{"interface":"front","average":{"1min":1.5960000000000001,"5min":1.3580000000000001,"15min":1.871},"min":{"1min":0.503,"5min":0.46000000000000002,"15min":0.34300000000000003},"max":{"1min":7.5380000000000003,"5min":8.0269999999999992,"15min":34.408000000000001},"last":1.0860000000000001}]},{"osd":10,"last update":"Fri Feb 9 18:03:28 2024","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.488,"15min":1.6950000000000001},"min":{"1min":0.496,"5min":0.37,"15min":0.34399999999999997},"max":{"1min":8.4309999999999992,"5min":8.4309999999999992,"15min":21.364999999999998},"last":0.49299999999999999},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.516,"15min":1.768},"min":{"1min":0.502,"5min":0.42499999999999999,"15min":0.40100000000000002},"max":{"1min":8.5380000000000003,"5min":8.5380000000000003,"15min":21.408999999999999},"last":0.72299999999999998}]},{"osd":11,"last update":"Fri Feb 9 18:03:49 2024","interfaces":[{"interface":"back","average":{"1min":2.214,"5min":1.6459999999999999,"15min":1.835},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.372},"max":{"1min":9.6859999999999999,"5min":9.6859999999999999,"15min":35.912999999999997},"last":0.76000000000000001},{"interface":"front","average":{"1min":2.2530000000000001,"5min":1.6850000000000001,"15min":1.8959999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.39700000000000002},"max":{"1min":9.7300000000000004,"5min":9.7300000000000004,"15min":36.156999999999996},"last":0.78900000000000003}]}]},{"osd":1,"up_from":13,"seq":55834575404,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1735508,"kb_used_data":527044,"kb_used_omap":1,"kb_used_meta":1208446,"kb_avail":92009644,"statfs":{"total":95995035648,"available":94217875456,"internally_reserved":0,"allocated":539693056,"data_stored":539391249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1237449157},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 18:04:11 2024","interfaces":[{"interface":"back","average":{"1min":1.643,"5min":1.1990000000000001,"15min":1.5529999999999999},"min":{"1min":0.26300000000000001,"5min":0.245,"15min":0.19},"max":{"1min":6.6079999999999997,"5min":13.068,"15min":14.885999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.393,"15min":1.7250000000000001},"min":{"1min":0.39200000000000002,"5min":0.28399999999999997,"15min":0.214},"max":{"1min":4.1319999999999997,"5min":9.6969999999999992,"15min":33.389000000000003},"last":0.26900000000000002}]},{"osd":2,"last update":"Fri Feb 9 18:03:31 2024","interfaces":[{"interface":"back","average":{"1min":1.857,"5min":1.605,"15min":1.7649999999999999},"min":{"1min":0.34499999999999997,"5min":0.28299999999999997,"15min":0.222},"max":{"1min":9.6649999999999991,"5min":9.7210000000000001,"15min":14.83},"last":0.39600000000000002},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.327,"15min":1.5389999999999999},"min":{"1min":0.36199999999999999,"5min":0.34699999999999998,"15min":0.28100000000000003},"max":{"1min":4.7549999999999999,"5min":5.7140000000000004,"15min":11.103},"last":0.32200000000000001}]},{"osd":3,"last update":"Fri Feb 9 18:04:22 2024","interfaces":[{"interface":"back","average":{"1min":2.7349999999999999,"5min":1.694,"15min":1.0189999999999999},"min":{"1min":0.371,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":13.292999999999999,"5min":13.292999999999999,"15min":13.292999999999999},"last":0.56999999999999995},{"interface":"front","average":{"1min":2.948,"5min":1.9830000000000001,"15min":1.4530000000000001},"min":{"1min":0.29099999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":13.265000000000001,"5min":13.265000000000001,"15min":13.265000000000001},"last":0.65500000000000003}]},{"osd":4,"last update":"Fri Feb 9 18:04:15 2024","interfaces":[{"interface":"back","average":{"1min":2.532,"5min":1.552,"15min":1.3069999999999999},"min":{"1min":0.39000000000000001,"5min":0.27600000000000002,"15min":0.26900000000000002},"max":{"1min":10.121,"5min":10.121,"15min":10.121},"last":0.624},{"interface":"front","average":{"1min":3.044,"5min":1.72,"15min":1.617},"min":{"1min":0.34399999999999997,"5min":0.33800000000000002,"15min":0.26400000000000001},"max":{"1min":10.157999999999999,"5min":10.157999999999999,"15min":15.331},"last":1.982}]},{"osd":5,"last update":"Fri Feb 9 18:03:47 2024","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.278,"15min":1.3959999999999999},"min":{"1min":0.371,"5min":0.32200000000000001,"15min":0.26000000000000001},"max":{"1min":4.9859999999999998,"5min":10.553000000000001,"15min":15.106999999999999},"last":2.79},{"interface":"front","average":{"1min":2.165,"5min":1.401,"15min":1.6910000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.28000000000000003},"max":{"1min":10.196,"5min":10.196,"15min":33.247},"last":13.106}]},{"osd":6,"last update":"Fri Feb 9 18:03:55 2024","interfaces":[{"interface":"back","average":{"1min":1.659,"5min":1.5660000000000001,"15min":1.724},"min":{"1min":0.48399999999999999,"5min":0.33100000000000002,"15min":0.24299999999999999},"max":{"1min":7.0529999999999999,"5min":10.605,"15min":33.338000000000001},"last":5.1420000000000003},{"interface":"front","average":{"1min":1.294,"5min":1.427,"15min":1.6839999999999999},"min":{"1min":0.45500000000000002,"5min":0.376,"15min":0.32800000000000001},"max":{"1min":4.0419999999999998,"5min":10.720000000000001,"15min":33.286999999999999},"last":5.1100000000000003}]},{"osd":7,"last update":"Fri Feb 9 18:04:00 2024","interfaces":[{"interface":"back","average":{"1min":1.528,"5min":1.6899999999999999,"15min":1.764},"min":{"1min":0.40799999999999997,"5min":0.40699999999999997,"15min":0.316},"max":{"1min":4.3490000000000002,"5min":10.58,"15min":33.314999999999998},"last":0.49399999999999999},{"interface":"front","average":{"1min":1.29,"5min":1.4279999999999999,"15min":1.7090000000000001},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":3.718,"5min":5.4539999999999997,"15min":33.363999999999997},"last":1.758}]},{"osd":8,"last update":"Fri Feb 9 18:04:11 2024","interfaces":[{"interface":"back","average":{"1min":1.528,"5min":1.633,"15min":1.73},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":4.4749999999999996,"5min":10.627000000000001,"15min":10.627000000000001},"last":0.52900000000000003},{"interface":"front","average":{"1min":1.512,"5min":1.843,"15min":2.0230000000000001},"min":{"1min":0.47199999999999998,"5min":0.44900000000000001,"15min":0.36399999999999999},"max":{"1min":4.1630000000000003,"5min":10.663,"15min":12.69},"last":0.44500000000000001}]},{"osd":9,"last update":"Fri Feb 9 18:04:11 2024","interfaces":[{"interface":"back","average":{"1min":2.5569999999999999,"5min":1.8839999999999999,"15min":2.0659999999999998},"min":{"1min":0.56399999999999995,"5min":0.46300000000000002,"15min":0.42799999999999999},"max":{"1min":10.337999999999999,"5min":10.378,"15min":20.789000000000001},"last":1.089},{"interface":"front","average":{"1min":3.2469999999999999,"5min":2.0710000000000002,"15min":1.996},"min":{"1min":0.47599999999999998,"5min":0.375,"15min":0.34200000000000003},"max":{"1min":10.522,"5min":10.522,"15min":10.522},"last":13.231999999999999}]},{"osd":10,"last update":"Fri Feb 9 18:03:47 2024","interfaces":[{"interface":"back","average":{"1min":2.0880000000000001,"5min":2.0329999999999999,"15min":2.1890000000000001},"min":{"1min":0.45700000000000002,"5min":0.42199999999999999,"15min":0.40999999999999998},"max":{"1min":9.4079999999999995,"5min":12.151,"15min":33.213999999999999},"last":0.66500000000000004},{"interface":"front","average":{"1min":2.766,"5min":1.986,"15min":2.0870000000000002},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.372},"max":{"1min":12.382,"5min":12.382,"15min":15.271000000000001},"last":0.751}]},{"osd":11,"last update":"Fri Feb 9 18:03:55 2024","interfaces":[{"interface":"back","average":{"1min":1.768,"5min":2.3109999999999999,"15min":2.5649999999999999},"min":{"1min":0.51100000000000001,"5min":0.51000000000000001,"15min":0.378},"max":{"1min":4.2130000000000001,"5min":16.881,"15min":19.648},"last":0.58699999999999997},{"interface":"front","average":{"1min":3.4089999999999998,"5min":2.4399999999999999,"15min":2.6480000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.372},"max":{"1min":12.423,"5min":12.423,"15min":33.161000000000001},"last":0.93799999999999994}]}]},{"osd":0,"up_from":8,"seq":34359738930,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1009048,"kb_used_data":125452,"kb_used_omap":1,"kb_used_meta":883582,"kb_avail":92736104,"statfs":{"total":95995035648,"available":94961770496,"internally_reserved":0,"allocated":128462848,"data_stored":128160627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":904788421},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 9 18:03:46 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":1.627,"15min":1.524},"min":{"1min":0.26700000000000002,"5min":0.25800000000000001,"15min":0.23100000000000001},"max":{"1min":1.8360000000000001,"5min":23.282,"15min":23.282},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.67200000000000004,"5min":1.6990000000000001,"15min":1.5980000000000001},"min":{"1min":0.371,"5min":0.32700000000000001,"15min":0.29199999999999998},"max":{"1min":1.873,"5min":23.32,"15min":23.32},"last":0.53400000000000003}]},{"osd":2,"last update":"Fri Feb 9 18:04:21 2024","interfaces":[{"interface":"back","average":{"1min":1.5680000000000001,"5min":1.6020000000000001,"15min":1.617},"min":{"1min":0.33700000000000002,"5min":0.27200000000000002,"15min":0.245},"max":{"1min":8.2750000000000004,"5min":11.161,"15min":20.013999999999999},"last":1.6830000000000001},{"interface":"front","average":{"1min":1.619,"5min":1.5229999999999999,"15min":1.627},"min":{"1min":0.33400000000000002,"5min":0.29199999999999998,"15min":0.219},"max":{"1min":6.8730000000000002,"5min":9.5579999999999998,"15min":20.103999999999999},"last":1.6479999999999999}]},{"osd":3,"last update":"Fri Feb 9 18:04:21 2024","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":2.1000000000000001,"15min":1.52},"min":{"1min":0.30199999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":4.492,"5min":45.688000000000002,"15min":45.688000000000002},"last":5.6929999999999996},{"interface":"front","average":{"1min":1.752,"5min":2.23,"15min":2.0019999999999998},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":8.3089999999999993,"5min":45.551000000000002,"15min":45.551000000000002},"last":5.7460000000000004}]},{"osd":4,"last update":"Fri Feb 9 18:04:21 2024","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.4159999999999999,"15min":1.345},"min":{"1min":0.33900000000000002,"5min":0.307,"15min":0.28100000000000003},"max":{"1min":6.234,"5min":11.393000000000001,"15min":12.757},"last":0.57199999999999995},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.413,"15min":1.649},"min":{"1min":0.48699999999999999,"5min":0.33900000000000002,"15min":0.249},"max":{"1min":6.2779999999999996,"5min":11.635,"15min":17.091999999999999},"last":0.626}]},{"osd":5,"last update":"Fri Feb 9 18:03:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7769999999999999,"5min":1.4470000000000001,"15min":1.3819999999999999},"min":{"1min":0.42099999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":11.435,"5min":11.435,"15min":12.949999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":1.8140000000000001,"5min":1.49,"15min":1.4399999999999999},"min":{"1min":0.50700000000000001,"5min":0.38300000000000001,"15min":0.35299999999999998},"max":{"1min":11.462999999999999,"5min":11.462999999999999,"15min":12.981999999999999},"last":0.71199999999999997}]},{"osd":6,"last update":"Fri Feb 9 18:04:05 2024","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.4319999999999999,"15min":1.522},"min":{"1min":0.313,"5min":0.313,"15min":0.307},"max":{"1min":4.3869999999999996,"5min":8.702,"15min":15.667},"last":0.745},{"interface":"front","average":{"1min":1.161,"5min":1.1919999999999999,"15min":1.5229999999999999},"min":{"1min":0.432,"5min":0.38,"15min":0.34799999999999998},"max":{"1min":4.2060000000000004,"5min":6.3380000000000001,"15min":15.702999999999999},"last":1.2350000000000001}]},{"osd":7,"last update":"Fri Feb 9 18:03:33 2024","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.365,"15min":1.6080000000000001},"min":{"1min":0.39400000000000002,"5min":0.33900000000000002,"15min":0.29199999999999998},"max":{"1min":3.6429999999999998,"5min":11.682,"15min":11.682},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.4970000000000001,"15min":1.708},"min":{"1min":0.49199999999999999,"5min":0.373,"15min":0.28299999999999997},"max":{"1min":4.7729999999999997,"5min":11.773999999999999,"15min":17.152999999999999},"last":1.2569999999999999}]},{"osd":8,"last update":"Fri Feb 9 18:03:33 2024","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.6399999999999999,"15min":1.984},"min":{"1min":0.40600000000000003,"5min":0.33800000000000002,"15min":0.313},"max":{"1min":4.3109999999999999,"5min":8.7490000000000006,"15min":17.189},"last":0.76800000000000002},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.538,"15min":1.909},"min":{"1min":0.47699999999999998,"5min":0.41399999999999998,"15min":0.36499999999999999},"max":{"1min":5.5609999999999999,"5min":11.747999999999999,"15min":18.600000000000001},"last":1.212}]},{"osd":9,"last update":"Fri Feb 9 18:04:21 2024","interfaces":[{"interface":"back","average":{"1min":2.4089999999999998,"5min":1.8460000000000001,"15min":2.4009999999999998},"min":{"1min":0.66700000000000004,"5min":0.443,"15min":0.39000000000000001},"max":{"1min":6.6070000000000002,"5min":11.801,"15min":24.757000000000001},"last":1.1619999999999999},{"interface":"front","average":{"1min":1.712,"5min":1.6539999999999999,"15min":2.149},"min":{"1min":0.51700000000000002,"5min":0.47799999999999998,"15min":0.40899999999999997},"max":{"1min":5.8159999999999998,"5min":12.759,"15min":24.681999999999999},"last":1.278}]},{"osd":10,"last update":"Fri Feb 9 18:03:39 2024","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.9199999999999999,"15min":2.2730000000000001},"min":{"1min":0.432,"5min":0.432,"15min":0.41999999999999998},"max":{"1min":4.1399999999999997,"5min":8.7840000000000007,"15min":57.979999999999997},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.879,"15min":2.4169999999999998},"min":{"1min":0.63800000000000001,"5min":0.44,"15min":0.40400000000000003},"max":{"1min":2.9590000000000001,"5min":12.689,"15min":57.606999999999999},"last":0.79000000000000004}]},{"osd":11,"last update":"Fri Feb 9 18:04:05 2024","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.5249999999999999,"15min":2.173},"min":{"1min":0.57099999999999995,"5min":0.47599999999999998,"15min":0.44800000000000001},"max":{"1min":5.8650000000000002,"5min":10.157999999999999,"15min":58.351999999999997},"last":0.65300000000000002},{"interface":"front","average":{"1min":1.337,"5min":1.5600000000000001,"15min":2.1930000000000001},"min":{"1min":0.623,"5min":0.499,"15min":0.441},"max":{"1min":5.8929999999999998,"5min":10.176,"15min":57.314},"last":0.67800000000000005}]}]},{"osd":2,"up_from":18,"seq":77309411878,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":741076,"kb_used_data":56008,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":93004076,"statfs":{"total":95995035648,"available":95236173824,"internally_reserved":0,"allocated":57352192,"data_stored":57050905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:58:48 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.53200000000000003,"15min":0.48699999999999999},"min":{"1min":0.23699999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.9619999999999997,"5min":4.9619999999999997,"15min":4.9619999999999997},"last":0.33800000000000002},{"interface":"front","average":{"1min":1.292,"5min":0.81899999999999995,"15min":0.73999999999999999},"min":{"1min":0.33000000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":4.5430000000000001,"5min":4.5430000000000001,"15min":4.5430000000000001},"last":0.47799999999999998}]},{"osd":1,"last update":"Fri Feb 9 18:03:30 2024","interfaces":[{"interface":"back","average":{"1min":2.133,"5min":1.216,"15min":1.8},"min":{"1min":0.251,"5min":0.23300000000000001,"15min":0.20799999999999999},"max":{"1min":12.814,"5min":12.814,"15min":46.698999999999998},"last":0.28899999999999998},{"interface":"front","average":{"1min":1.798,"5min":1.1839999999999999,"15min":1.784},"min":{"1min":0.32000000000000001,"5min":0.313,"15min":0.248},"max":{"1min":10.066000000000001,"5min":10.066000000000001,"15min":46.728999999999999},"last":0.35899999999999999}]},{"osd":3,"last update":"Fri Feb 9 18:04:19 2024","interfaces":[{"interface":"back","average":{"1min":2,"5min":1.6779999999999999,"15min":2.8090000000000002},"min":{"1min":0.312,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":5.7519999999999998,"5min":29.228000000000002,"15min":29.228000000000002},"last":0.39600000000000002},{"interface":"front","average":{"1min":1.478,"5min":1.28,"15min":1.4590000000000001},"min":{"1min":0.27400000000000002,"5min":0.251,"15min":0.251},"max":{"1min":6.0629999999999997,"5min":10.492000000000001,"15min":10.492000000000001},"last":0.309}]},{"osd":4,"last update":"Fri Feb 9 18:04:01 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.95699999999999996,"15min":1.153},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.245},"max":{"1min":3.8119999999999998,"5min":9.609,"15min":10.419},"last":0.438},{"interface":"front","average":{"1min":0.997,"5min":1.004,"15min":1.129},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.25700000000000001},"max":{"1min":3.8519999999999999,"5min":9.6500000000000004,"15min":10.529},"last":0.46600000000000003}]},{"osd":5,"last update":"Fri Feb 9 18:03:49 2024","interfaces":[{"interface":"back","average":{"1min":2.2210000000000001,"5min":1.21,"15min":1.421},"min":{"1min":0.29199999999999998,"5min":0.28100000000000003,"15min":0.245},"max":{"1min":11.536,"5min":11.536,"15min":15.455},"last":0.374},{"interface":"front","average":{"1min":2.2930000000000001,"5min":1.347,"15min":1.421},"min":{"1min":0.41699999999999998,"5min":0.27000000000000002,"15min":0.26900000000000002},"max":{"1min":11.571,"5min":11.571,"15min":15.487},"last":0.42799999999999999}]},{"osd":6,"last update":"Fri Feb 9 18:04:01 2024","interfaces":[{"interface":"back","average":{"1min":1.5369999999999999,"5min":1.363,"15min":1.4890000000000001},"min":{"1min":0.39400000000000002,"5min":0.32300000000000001,"15min":0.28399999999999997},"max":{"1min":6.0140000000000002,"5min":11.465999999999999,"15min":18.518999999999998},"last":0.40799999999999997},{"interface":"front","average":{"1min":1.6579999999999999,"5min":1.409,"15min":1.52},"min":{"1min":0.35599999999999998,"5min":0.32900000000000001,"15min":0.28199999999999997},"max":{"1min":4.8899999999999997,"5min":11.598000000000001,"15min":18.481999999999999},"last":0.47599999999999998}]},{"osd":7,"last update":"Fri Feb 9 18:03:30 2024","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":0.94999999999999996,"15min":1.383},"min":{"1min":0.40999999999999998,"5min":0.37,"15min":0.33100000000000002},"max":{"1min":3.9020000000000001,"5min":5.6139999999999999,"15min":12.422000000000001},"last":0.41999999999999998},{"interface":"front","average":{"1min":1.9279999999999999,"5min":1.5209999999999999,"15min":1.5269999999999999},"min":{"1min":0.38800000000000001,"5min":0.35699999999999998,"15min":0.309},"max":{"1min":11.603,"5min":11.603,"15min":11.603},"last":0.50900000000000001}]},{"osd":8,"last update":"Fri Feb 9 18:04:01 2024","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.2450000000000001,"15min":1.8},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":5.1050000000000004,"5min":7.5060000000000002,"15min":12.500999999999999},"last":0.56999999999999995},{"interface":"front","average":{"1min":1.9019999999999999,"5min":1.8660000000000001,"15min":2.0009999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":7.399,"5min":11.492000000000001,"15min":54.759999999999998},"last":0.55700000000000005}]},{"osd":9,"last update":"Fri Feb 9 18:04:01 2024","interfaces":[{"interface":"back","average":{"1min":1.7889999999999999,"5min":1.486,"15min":2.1259999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.318},"max":{"1min":5.9299999999999997,"5min":9.7669999999999995,"15min":53.935000000000002},"last":0.53300000000000003},{"interface":"front","average":{"1min":1.7889999999999999,"5min":1.758,"15min":2.1000000000000001},"min":{"1min":0.45200000000000001,"5min":0.36399999999999999,"15min":0.34899999999999998},"max":{"1min":4.6779999999999999,"5min":11.522,"15min":54.670999999999999},"last":0.59999999999999998}]},{"osd":10,"last update":"Fri Feb 9 18:03:38 2024","interfaces":[{"interface":"back","average":{"1min":2.1709999999999998,"5min":1.8500000000000001,"15min":2.1419999999999999},"min":{"1min":0.498,"5min":0.46999999999999997,"15min":0.33500000000000002},"max":{"1min":6.3609999999999998,"5min":8.5570000000000004,"15min":54.039999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":2.1840000000000002,"5min":1.7989999999999999,"15min":2.1469999999999998},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42599999999999999},"max":{"1min":6.2199999999999998,"5min":8.6150000000000002,"15min":54.118000000000002},"last":0.58999999999999997}]},{"osd":11,"last update":"Fri Feb 9 18:04:01 2024","interfaces":[{"interface":"back","average":{"1min":2.3399999999999999,"5min":2.1960000000000002,"15min":2.2610000000000001},"min":{"1min":0.5,"5min":0.5,"15min":0.42099999999999999},"max":{"1min":12.212999999999999,"5min":12.212999999999999,"15min":54.799999999999997},"last":0.58099999999999996},{"interface":"front","average":{"1min":1.956,"5min":1.7909999999999999,"15min":2.274},"min":{"1min":0.44,"5min":0.44,"15min":0.35499999999999998},"max":{"1min":11.922000000000001,"5min":11.922000000000001,"15min":54.103000000000002},"last":0.48699999999999999}]}]},{"osd":3,"up_from":854,"seq":3667902070850,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":739208,"kb_used_data":470016,"kb_used_omap":202,"kb_used_meta":268981,"kb_avail":93005944,"statfs":{"total":95995035648,"available":95238086656,"internally_reserved":0,"allocated":481296384,"data_stored":480998649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207633,"internal_metadata":275436783},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 18:04:23 2024","interfaces":[{"interface":"back","average":{"1min":1.4259999999999999,"5min":1.2649999999999999,"15min":1.0649999999999999},"min":{"1min":0.309,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":5.2039999999999997,"5min":8.4079999999999995,"15min":8.4079999999999995},"last":4.8979999999999997},{"interface":"front","average":{"1min":2.5489999999999999,"5min":1.333,"15min":1.6830000000000001},"min":{"1min":0.28699999999999998,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":11.287000000000001,"5min":17.077999999999999,"15min":17.077999999999999},"last":5.3460000000000001}]},{"osd":1,"last update":"Fri Feb 9 18:04:12 2024","interfaces":[{"interface":"back","average":{"1min":3.581,"5min":1.863,"15min":1.123},"min":{"1min":0.25900000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":21.73,"5min":21.73,"15min":21.73},"last":4.8209999999999997},{"interface":"front","average":{"1min":4.298,"5min":1.7969999999999999,"15min":0.999},"min":{"1min":0.30399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":19.268000000000001,"5min":19.268000000000001,"15min":19.268000000000001},"last":12.356999999999999}]},{"osd":2,"last update":"Fri Feb 9 18:04:23 2024","interfaces":[{"interface":"back","average":{"1min":2.649,"5min":2.0019999999999998,"15min":2.278},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":7.3330000000000002,"5min":13.098000000000001,"15min":13.098000000000001},"last":2.3279999999999998},{"interface":"front","average":{"1min":2.4710000000000001,"5min":1.597,"15min":1.3400000000000001},"min":{"1min":0.441,"5min":0.309,"15min":0.309},"max":{"1min":8.8539999999999992,"5min":9.3260000000000005,"15min":9.3260000000000005},"last":4.8600000000000003}]},{"osd":4,"last update":"Fri Feb 9 18:04:23 2024","interfaces":[{"interface":"back","average":{"1min":2.9239999999999999,"5min":1.3839999999999999,"15min":1.0680000000000001},"min":{"1min":0.32700000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":16.329000000000001,"5min":16.329000000000001,"15min":16.329000000000001},"last":5.3869999999999996},{"interface":"front","average":{"1min":1.7430000000000001,"5min":1.3180000000000001,"15min":1.4259999999999999},"min":{"1min":0.32800000000000001,"5min":0.318,"15min":0.318},"max":{"1min":6.5679999999999996,"5min":10.755000000000001,"15min":10.755000000000001},"last":2.4670000000000001}]},{"osd":5,"last update":"Fri Feb 9 18:04:23 2024","interfaces":[{"interface":"back","average":{"1min":2.8140000000000001,"5min":1.379,"15min":1.1759999999999999},"min":{"1min":0.34999999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":16.405000000000001,"5min":16.405000000000001,"15min":16.405000000000001},"last":5.4370000000000003},{"interface":"front","average":{"1min":1.7330000000000001,"5min":1.3879999999999999,"15min":1.581},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":6.6440000000000001,"5min":10.707000000000001,"15min":10.707000000000001},"last":2.395}]},{"osd":6,"last update":"Fri Feb 9 18:04:23 2024","interfaces":[{"interface":"back","average":{"1min":1.8799999999999999,"5min":1.6240000000000001,"15min":1.893},"min":{"1min":0.42299999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":6.4050000000000002,"5min":10.808,"15min":10.808},"last":2.3610000000000002},{"interface":"front","average":{"1min":1.6830000000000001,"5min":1.1830000000000001,"15min":0.85699999999999998},"min":{"1min":0.505,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":5.5499999999999998,"5min":8.3550000000000004,"15min":8.3550000000000004},"last":0.58399999999999996}]},{"osd":7,"last update":"Fri Feb 9 18:04:02 2024","interfaces":[{"interface":"back","average":{"1min":2.0800000000000001,"5min":2.0800000000000001,"15min":2.0800000000000001},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":9.423,"5min":9.423,"15min":9.423},"last":0.55500000000000005},{"interface":"front","average":{"1min":2.2530000000000001,"5min":2.2530000000000001,"15min":2.2530000000000001},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":12.074,"5min":12.074,"15min":12.074},"last":5.516}]},{"osd":8,"last update":"Fri Feb 9 18:04:23 2024","interfaces":[{"interface":"back","average":{"1min":2.5259999999999998,"5min":1.605,"15min":1.4430000000000001},"min":{"1min":0.41799999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":16.228000000000002,"5min":16.228000000000002,"15min":16.228000000000002},"last":5.4139999999999997},{"interface":"front","average":{"1min":1.8600000000000001,"5min":1.6359999999999999,"15min":1.8360000000000001},"min":{"1min":0.46000000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":6.6040000000000001,"5min":10.782,"15min":10.782},"last":2.419}]},{"osd":9,"last update":"Fri Feb 9 18:04:23 2024","interfaces":[{"interface":"back","average":{"1min":3.2400000000000002,"5min":1.6659999999999999,"15min":0.98999999999999999},"min":{"1min":0.39600000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":16.172999999999998,"5min":16.172999999999998,"15min":16.172999999999998},"last":5.4660000000000002},{"interface":"front","average":{"1min":2.339,"5min":1.702,"15min":1.4650000000000001},"min":{"1min":0.50900000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":7.7080000000000002,"5min":9.8350000000000009,"15min":9.8350000000000009},"last":2.4430000000000001}]},{"osd":10,"last update":"Fri Feb 9 18:04:15 2024","interfaces":[{"interface":"back","average":{"1min":2.206,"5min":2.206,"15min":2.206},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":8.7870000000000008,"5min":8.7870000000000008,"15min":8.7870000000000008},"last":0.73099999999999998},{"interface":"front","average":{"1min":3.2040000000000002,"5min":3.2040000000000002,"15min":3.2040000000000002},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":16.263000000000002,"5min":16.263000000000002,"15min":16.263000000000002},"last":0.83199999999999996}]},{"osd":11,"last update":"Fri Feb 9 18:04:23 2024","interfaces":[{"interface":"back","average":{"1min":2.79,"5min":1.853,"15min":2.1840000000000002},"min":{"1min":0.497,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":16.295000000000002,"5min":17.126000000000001,"15min":17.126000000000001},"last":5.4900000000000002},{"interface":"front","average":{"1min":2.2109999999999999,"5min":1.758,"15min":1.206},"min":{"1min":0.53900000000000003,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":8.827,"5min":9.375,"15min":9.375},"last":0.51300000000000001}]}]},{"osd":4,"up_from":32,"seq":137438954015,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1011340,"kb_used_data":113856,"kb_used_omap":1,"kb_used_meta":897470,"kb_avail":92733812,"statfs":{"total":95995035648,"available":94959423488,"internally_reserved":0,"allocated":116588544,"data_stored":116290809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":919009733},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:58:54 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.749,"15min":1.2709999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.22800000000000001},"max":{"1min":1.734,"5min":7.7359999999999998,"15min":10.301},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.81799999999999995,"15min":1.3400000000000001},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.768,"5min":7.7729999999999997,"15min":10.407999999999999},"last":0.35599999999999998}]},{"osd":1,"last update":"Fri Feb 9 18:04:06 2024","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.3939999999999999,"15min":1.411},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":5.5229999999999997,"5min":39.810000000000002,"15min":50.255000000000003},"last":6.1340000000000003},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.4590000000000001,"15min":1.6619999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.23100000000000001},"max":{"1min":5.5499999999999998,"5min":39.853999999999999,"15min":49.991999999999997},"last":6.2999999999999998}]},{"osd":2,"last update":"Fri Feb 9 18:04:00 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":1.071,"15min":0.97599999999999998},"min":{"1min":0.28299999999999997,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":4.8239999999999998,"5min":4.8239999999999998,"15min":4.8239999999999998},"last":6.1799999999999997},{"interface":"front","average":{"1min":1.0029999999999999,"5min":1.141,"15min":1.0049999999999999},"min":{"1min":0.31900000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":4.8499999999999996,"5min":5.0570000000000004,"15min":5.0570000000000004},"last":6.2220000000000004}]},{"osd":3,"last update":"Fri Feb 9 18:04:18 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.115,"15min":1.097},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":3.8879999999999999,"5min":5.9290000000000003,"15min":5.9290000000000003},"last":1.1519999999999999},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.0740000000000001,"15min":1.111},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":3.8730000000000002,"5min":4.4169999999999998,"15min":4.4169999999999998},"last":6.0970000000000004}]},{"osd":5,"last update":"Fri Feb 9 18:03:26 2024","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":0.67600000000000005,"15min":0.83499999999999996},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.24399999999999999},"max":{"1min":3.6459999999999999,"5min":3.6459999999999999,"15min":8.3480000000000008},"last":1.194},{"interface":"front","average":{"1min":1.044,"5min":0.755,"15min":0.90600000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.31},"max":{"1min":3.7349999999999999,"5min":3.7349999999999999,"15min":8.3800000000000008},"last":1.2230000000000001}]},{"osd":6,"last update":"Fri Feb 9 18:04:06 2024","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":0.93600000000000005,"15min":0.92700000000000005},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.24299999999999999},"max":{"1min":6.4009999999999998,"5min":6.9530000000000003,"15min":23.343},"last":6.048},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.89700000000000002,"15min":0.87},"min":{"1min":0.38900000000000001,"5min":0.29899999999999999,"15min":0.28799999999999998},"max":{"1min":1.8500000000000001,"5min":7.1980000000000004,"15min":21.861000000000001},"last":6.0679999999999996}]},{"osd":7,"last update":"Fri Feb 9 18:03:51 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":0.91000000000000003,"15min":0.98299999999999998},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.32900000000000001},"max":{"1min":4.202,"5min":4.202,"15min":7.3890000000000002},"last":5.2030000000000003},{"interface":"front","average":{"1min":1.0229999999999999,"5min":0.90800000000000003,"15min":1.0049999999999999},"min":{"1min":0.432,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":3.5699999999999998,"5min":6.7919999999999998,"15min":9.5519999999999996},"last":5.2549999999999999}]},{"osd":8,"last update":"Fri Feb 9 18:03:26 2024","interfaces":[{"interface":"back","average":{"1min":2.0310000000000001,"5min":1.266,"15min":1.4370000000000001},"min":{"1min":0.45900000000000002,"5min":0.34799999999999998,"15min":0.33100000000000002},"max":{"1min":6.4619999999999997,"5min":6.8890000000000002,"15min":9.7059999999999995},"last":5.3250000000000002},{"interface":"front","average":{"1min":2.0720000000000001,"5min":1.4299999999999999,"15min":1.304},"min":{"1min":0.47799999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":6.4329999999999998,"5min":9.0109999999999992,"15min":9.0109999999999992},"last":5.3819999999999997}]},{"osd":9,"last update":"Fri Feb 9 18:03:26 2024","interfaces":[{"interface":"back","average":{"1min":2.5550000000000002,"5min":1.641,"15min":1.865},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":7.2030000000000003,"5min":10.563000000000001,"15min":10.563000000000001},"last":3.8029999999999999},{"interface":"front","average":{"1min":2.1739999999999999,"5min":1.605,"15min":1.571},"min":{"1min":0.49399999999999999,"5min":0.373,"15min":0.373},"max":{"1min":4.649,"5min":6.3799999999999999,"15min":7.9850000000000003},"last":5.2960000000000003}]},{"osd":10,"last update":"Fri Feb 9 18:03:51 2024","interfaces":[{"interface":"back","average":{"1min":1.754,"5min":1.8819999999999999,"15min":1.4299999999999999},"min":{"1min":0.53800000000000003,"5min":0.44400000000000001,"15min":0.40200000000000002},"max":{"1min":6.2169999999999996,"5min":8.2829999999999995,"15min":8.2829999999999995},"last":5.3570000000000002},{"interface":"front","average":{"1min":1.698,"5min":1.476,"15min":1.48},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.38},"max":{"1min":5.6760000000000002,"5min":6.181,"15min":7.9189999999999996},"last":6.0199999999999996}]},{"osd":11,"last update":"Fri Feb 9 18:04:18 2024","interfaces":[{"interface":"back","average":{"1min":1.597,"5min":1.5149999999999999,"15min":1.5629999999999999},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":7.8620000000000001,"5min":7.8620000000000001,"15min":7.8620000000000001},"last":1.9730000000000001},{"interface":"front","average":{"1min":1.337,"5min":1.486,"15min":1.6040000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46000000000000002},"max":{"1min":7.4619999999999997,"5min":8.2219999999999995,"15min":9.5960000000000001},"last":6.3559999999999999}]}]},{"osd":5,"up_from":40,"seq":171798692377,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1486732,"kb_used_data":539264,"kb_used_omap":1,"kb_used_meta":947454,"kb_avail":92258420,"statfs":{"total":95995035648,"available":94472622080,"internally_reserved":0,"allocated":552206336,"data_stored":551905455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":970193349},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:58:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.82599999999999996,"15min":0.874},"min":{"1min":0.27800000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":1.341,"5min":5.399,"15min":5.399},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.622,"5min":0.92100000000000004,"15min":0.97099999999999997},"min":{"1min":0.40400000000000003,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.409,"5min":5.4329999999999998,"15min":5.4329999999999998},"last":0.52100000000000002}]},{"osd":1,"last update":"Fri Feb 9 18:03:25 2024","interfaces":[{"interface":"back","average":{"1min":1.498,"5min":1.2,"15min":1.2989999999999999},"min":{"1min":0.29299999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":4.8109999999999999,"5min":16.271000000000001,"15min":19.079999999999998},"last":0.56699999999999995},{"interface":"front","average":{"1min":1.9790000000000001,"5min":1.3420000000000001,"15min":1.2370000000000001},"min":{"1min":0.28000000000000003,"5min":0.27900000000000003,"15min":0.23499999999999999},"max":{"1min":7.2839999999999998,"5min":16.344999999999999,"15min":19.065999999999999},"last":0.36699999999999999}]},{"osd":2,"last update":"Fri Feb 9 18:04:04 2024","interfaces":[{"interface":"back","average":{"1min":1.7310000000000001,"5min":1.1100000000000001,"15min":1.147},"min":{"1min":0.32400000000000001,"5min":0.27400000000000002,"15min":0.254},"max":{"1min":7.71,"5min":7.71,"15min":7.71},"last":0.80200000000000005},{"interface":"front","average":{"1min":1.4530000000000001,"5min":1.262,"15min":1.361},"min":{"1min":0.38600000000000001,"5min":0.25800000000000001,"15min":0.24399999999999999},"max":{"1min":5.1799999999999997,"5min":6.859,"15min":14.978},"last":0.45100000000000001}]},{"osd":3,"last update":"Fri Feb 9 18:03:23 2024","interfaces":[{"interface":"back","average":{"1min":1.571,"5min":0.91200000000000003,"15min":0.79000000000000004},"min":{"1min":0.28499999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":7.0069999999999997,"5min":7.0069999999999997,"15min":7.0069999999999997},"last":10.896000000000001},{"interface":"front","average":{"1min":1.827,"5min":1.3680000000000001,"15min":1.113},"min":{"1min":0.34300000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":6.9630000000000001,"5min":7.218,"15min":7.218},"last":0.66100000000000003}]},{"osd":4,"last update":"Fri Feb 9 18:03:25 2024","interfaces":[{"interface":"back","average":{"1min":1.512,"5min":0.86799999999999999,"15min":0.92100000000000004},"min":{"1min":0.32500000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":5.7990000000000004,"5min":5.7990000000000004,"15min":28.481999999999999},"last":0.27400000000000002},{"interface":"front","average":{"1min":1.6899999999999999,"5min":0.94899999999999995,"15min":1.0389999999999999},"min":{"1min":0.378,"5min":0.34799999999999998,"15min":0.22500000000000001},"max":{"1min":4.7779999999999996,"5min":6.1239999999999997,"15min":15.843},"last":0.308}]},{"osd":6,"last update":"Fri Feb 9 18:04:04 2024","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":0.90900000000000003,"15min":0.94299999999999995},"min":{"1min":0.33800000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":5.5419999999999998,"5min":6.8230000000000004,"15min":15.026},"last":0.39900000000000002},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.034,"15min":1.093},"min":{"1min":0.379,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":5.8650000000000002,"5min":6.9290000000000003,"15min":20.192},"last":0.83099999999999996}]},{"osd":7,"last update":"Fri Feb 9 18:03:35 2024","interfaces":[{"interface":"back","average":{"1min":2.2440000000000002,"5min":1.083,"15min":1.137},"min":{"1min":0.376,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":12.798999999999999,"5min":12.798999999999999,"15min":18.134},"last":1.018},{"interface":"front","average":{"1min":2.3149999999999999,"5min":1.1399999999999999,"15min":1.2549999999999999},"min":{"1min":0.505,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":12.885999999999999,"5min":12.885999999999999,"15min":19.300000000000001},"last":1.05}]},{"osd":8,"last update":"Fri Feb 9 18:03:50 2024","interfaces":[{"interface":"back","average":{"1min":2.343,"5min":1.5760000000000001,"15min":1.645},"min":{"1min":0.41899999999999998,"5min":0.35399999999999998,"15min":0.34200000000000003},"max":{"1min":7.71,"5min":7.71,"15min":12.227},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.79800000000000004,"5min":1.1990000000000001,"15min":1.4159999999999999},"min":{"1min":0.376,"5min":0.376,"15min":0.372},"max":{"1min":2.3530000000000002,"5min":7.3680000000000003,"15min":8.3770000000000007},"last":0.57899999999999996}]},{"osd":9,"last update":"Fri Feb 9 18:04:04 2024","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.6479999999999999,"15min":1.8149999999999999},"min":{"1min":0.39800000000000002,"5min":0.315,"15min":0.315},"max":{"1min":6.7889999999999997,"5min":21.686,"15min":21.686},"last":0.53500000000000003},{"interface":"front","average":{"1min":1.849,"5min":1.8999999999999999,"15min":1.645},"min":{"1min":0.42499999999999999,"5min":0.38,"15min":0.36799999999999999},"max":{"1min":6.7130000000000001,"5min":21.734999999999999,"15min":21.734999999999999},"last":0.55500000000000005}]},{"osd":10,"last update":"Fri Feb 9 18:03:25 2024","interfaces":[{"interface":"back","average":{"1min":3.5099999999999998,"5min":1.7889999999999999,"15min":1.5129999999999999},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.373},"max":{"1min":21.027000000000001,"5min":21.027000000000001,"15min":21.027000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":3.48,"5min":1.9430000000000001,"15min":1.71},"min":{"1min":0.443,"5min":0.443,"15min":0.39900000000000002},"max":{"1min":20.859999999999999,"5min":20.859999999999999,"15min":20.859999999999999},"last":0.74299999999999999}]},{"osd":11,"last update":"Fri Feb 9 18:03:50 2024","interfaces":[{"interface":"back","average":{"1min":2.0920000000000001,"5min":1.506,"15min":1.611},"min":{"1min":0.622,"5min":0.495,"15min":0.434},"max":{"1min":9.7959999999999994,"5min":9.7959999999999994,"15min":9.7959999999999994},"last":0.64800000000000002},{"interface":"front","average":{"1min":1.9610000000000001,"5min":1.9059999999999999,"15min":1.7909999999999999},"min":{"1min":0.60199999999999998,"5min":0.48699999999999999,"15min":0.434},"max":{"1min":6.899,"5min":9.8819999999999997,"15min":20.981000000000002},"last":0.59499999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":127336448,"data_stored":127336448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":517279744,"data_stored":517275672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":55115776,"data_stored":55115776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":464191488,"data_stored":464191488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":111673344,"data_stored":111673344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":531632128,"data_stored":531628056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":117243904,"data_stored":117243904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":269160448,"data_stored":269156376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":108462080,"data_stored":108462080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":400097280,"data_stored":400097280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":59441152,"data_stored":59441152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":216334336,"data_stored":216334336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-09T18:04:34.045 INFO:tasks.thrashosds.thrasher:Setting 1.19 to [6, 0, 8, 11, 7, 5] 2024-02-09T18:04:34.045 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '1.19', '6', '0', '8', '11', '7', '5'] 2024-02-09T18:04:34.045 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd pg-upmap-items 1.19 6 0 8 11 7 5 2024-02-09T18:04:34.109 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:04:34.187 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:34.673 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T18:04:34.674 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [], 2024-02-09T18:04:34.674 INFO:teuthology.orchestra.run.smithi099.stdout: "complaint_time": 30, 2024-02-09T18:04:34.674 INFO:teuthology.orchestra.run.smithi099.stdout: "num_blocked_ops": 0 2024-02-09T18:04:34.674 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T18:04:34.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:34 smithi099 ceph-mon[39420]: pgmap v2407: 47 pgs: 47 active+clean; 568 MiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 996 op/s; 2.5 MiB/s, 39 objects/s recovering 2024-02-09T18:04:34.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:34 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:04:34.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:34 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:04:34.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:34 smithi099 ceph-mon[39420]: 1.7 scrub ok 2024-02-09T18:04:34.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:34 smithi099 ceph-mon[39420]: osdmap e974: 12 total, 12 up, 11 in 2024-02-09T18:04:34.995 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:34 smithi099 ceph-mon[39420]: 9.2 deep-scrub starts 2024-02-09T18:04:35.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:34 smithi179 ceph-mon[43975]: pgmap v2407: 47 pgs: 47 active+clean; 568 MiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 996 op/s; 2.5 MiB/s, 39 objects/s recovering 2024-02-09T18:04:35.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:34 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:04:35.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:34 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:04:35.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:34 smithi179 ceph-mon[43975]: 1.7 scrub ok 2024-02-09T18:04:35.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:34 smithi179 ceph-mon[43975]: osdmap e974: 12 total, 12 up, 11 in 2024-02-09T18:04:35.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:34 smithi179 ceph-mon[43975]: 9.2 deep-scrub starts 2024-02-09T18:04:35.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:34 smithi162 ceph-mon[44304]: pgmap v2407: 47 pgs: 47 active+clean; 568 MiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 996 op/s; 2.5 MiB/s, 39 objects/s recovering 2024-02-09T18:04:35.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:34 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:04:35.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:34 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:04:35.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:34 smithi162 ceph-mon[44304]: 1.7 scrub ok 2024-02-09T18:04:35.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:34 smithi162 ceph-mon[44304]: osdmap e974: 12 total, 12 up, 11 in 2024-02-09T18:04:35.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:34 smithi162 ceph-mon[44304]: 9.2 deep-scrub starts 2024-02-09T18:04:35.351 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-09T18:04:36.607 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:04:36.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:36.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:04:36.713 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:04:36.756 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:36.896 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:36 smithi099 ceph-mon[39420]: pgmap v2409: 47 pgs: 47 active+clean; 568 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 623 op/s 2024-02-09T18:04:37.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:36 smithi179 ceph-mon[43975]: pgmap v2409: 47 pgs: 47 active+clean; 568 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 623 op/s 2024-02-09T18:04:37.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:36 smithi162 ceph-mon[44304]: pgmap v2409: 47 pgs: 47 active+clean; 568 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 623 op/s 2024-02-09T18:04:37.722 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:04:37.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:37.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:04:37.831 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:04:37.875 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:37.896 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:04:37.897 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:04:37.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:37.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:04:37.931 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-09T18:04:37.973 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:37 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:04:37.973 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:37 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:04:37.973 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:37 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:04:37.973 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:37 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:04:37.974 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:37 smithi099 ceph-mon[39420]: osdmap e975: 12 total, 12 up, 11 in 2024-02-09T18:04:37.974 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:37 smithi099 ceph-mon[39420]: 1.12 scrub starts 2024-02-09T18:04:37.974 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:37 smithi099 ceph-mon[39420]: 1.12 scrub ok 2024-02-09T18:04:38.002 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:04:38.043 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:37 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:04:38.043 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:37 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:04:38.043 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:37 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:04:38.043 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:37 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:04:38.043 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:37 smithi179 ceph-mon[43975]: osdmap e975: 12 total, 12 up, 11 in 2024-02-09T18:04:38.043 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:37 smithi179 ceph-mon[43975]: 1.12 scrub starts 2024-02-09T18:04:38.043 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:37 smithi179 ceph-mon[43975]: 1.12 scrub ok 2024-02-09T18:04:38.045 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:38.067 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:04:38.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:38.071 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:04:38.172 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:04:38.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:37 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:04:38.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:37 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:04:38.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:37 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:04:38.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:37 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:04:38.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:37 smithi162 ceph-mon[44304]: osdmap e975: 12 total, 12 up, 11 in 2024-02-09T18:04:38.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:37 smithi162 ceph-mon[44304]: 1.12 scrub starts 2024-02-09T18:04:38.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:37 smithi162 ceph-mon[44304]: 1.12 scrub ok 2024-02-09T18:04:38.252 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:38.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:38 smithi099 ceph-mon[39420]: 1.19 deep-scrub starts 2024-02-09T18:04:38.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:38 smithi099 ceph-mon[39420]: pgmap v2411: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 359 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 820 op/s 2024-02-09T18:04:38.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:38 smithi099 ceph-mon[39420]: 9.2 deep-scrub ok 2024-02-09T18:04:38.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:38 smithi099 ceph-mon[39420]: 1.19 deep-scrub ok 2024-02-09T18:04:38.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:38 smithi099 ceph-mon[39420]: 1.c scrub starts 2024-02-09T18:04:38.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:38 smithi099 ceph-mon[39420]: 1.c scrub ok 2024-02-09T18:04:39.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:38 smithi179 ceph-mon[43975]: 1.19 deep-scrub starts 2024-02-09T18:04:39.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:38 smithi179 ceph-mon[43975]: pgmap v2411: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 359 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 820 op/s 2024-02-09T18:04:39.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:38 smithi179 ceph-mon[43975]: 9.2 deep-scrub ok 2024-02-09T18:04:39.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:38 smithi179 ceph-mon[43975]: 1.19 deep-scrub ok 2024-02-09T18:04:39.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:38 smithi179 ceph-mon[43975]: 1.c scrub starts 2024-02-09T18:04:39.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:38 smithi179 ceph-mon[43975]: 1.c scrub ok 2024-02-09T18:04:39.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:38 smithi162 ceph-mon[44304]: 1.19 deep-scrub starts 2024-02-09T18:04:39.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:38 smithi162 ceph-mon[44304]: pgmap v2411: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 359 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 820 op/s 2024-02-09T18:04:39.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:38 smithi162 ceph-mon[44304]: 9.2 deep-scrub ok 2024-02-09T18:04:39.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:38 smithi162 ceph-mon[44304]: 1.19 deep-scrub ok 2024-02-09T18:04:39.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:38 smithi162 ceph-mon[44304]: 1.c scrub starts 2024-02-09T18:04:39.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:38 smithi162 ceph-mon[44304]: 1.c scrub ok 2024-02-09T18:04:39.322 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:04:39.325 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:04:39.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:39.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:04:39.431 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T18:04:39.465 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:39.484 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T18:04:39.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:39.489 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T18:04:39.590 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:04:39.693 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:39.757 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:39 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:04:39.758 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:39 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:04:39.758 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:39 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2958374125' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T18:04:39.774 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is set 2024-02-09T18:04:39.996 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:39 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:04:39.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:39 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:04:39.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:39 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2958374125' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T18:04:40.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:39 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:04:40.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:39 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:04:40.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:39 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2958374125' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-09T18:04:40.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:40 smithi099 ceph-mon[39420]: pgmap v2412: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 315 MiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 731 op/s 2024-02-09T18:04:40.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:40 smithi099 ceph-mon[39420]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:04:40.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:40 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2958374125' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T18:04:40.997 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:40 smithi099 ceph-mon[39420]: osdmap e976: 12 total, 12 up, 11 in 2024-02-09T18:04:41.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:40 smithi179 ceph-mon[43975]: pgmap v2412: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 315 MiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 731 op/s 2024-02-09T18:04:41.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:40 smithi179 ceph-mon[43975]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:04:41.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:40 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2958374125' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T18:04:41.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:40 smithi179 ceph-mon[43975]: osdmap e976: 12 total, 12 up, 11 in 2024-02-09T18:04:41.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:40 smithi162 ceph-mon[44304]: pgmap v2412: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 315 MiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 731 op/s 2024-02-09T18:04:41.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:40 smithi162 ceph-mon[44304]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:04:41.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:40 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2958374125' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-09T18:04:41.178 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:40 smithi162 ceph-mon[44304]: osdmap e976: 12 total, 12 up, 11 in 2024-02-09T18:04:41.283 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:04:41.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:41.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:04:41.389 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T18:04:41.443 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:41.478 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T18:04:41.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:41.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T18:04:41.587 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:04:41.644 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:41.737 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:04:41.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:41.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:04:41.832 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:41 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:04:41.832 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:41 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:04:41.843 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:04:41.907 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:41.941 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:04:41.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:41.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:04:42.045 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T18:04:42.083 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:41 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:04:42.083 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:41 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:04:42.084 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:42.112 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T18:04:42.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:42.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T18:04:42.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:41 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:04:42.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:41 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:04:42.217 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:04:42.248 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:42.336 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:04:42.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:42.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:04:42.435 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Removed 134879 objects 2024-02-09T18:04:42.435 INFO:tasks.radosbench.radosbench.2.smithi179.stdout:Clean up completed and total clean up time :172.817 2024-02-09T18:04:42.447 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:04:42.473 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:42.922 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:04:42.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:42.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:04:43.026 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:04:43.054 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:42 smithi179 ceph-mon[43975]: pgmap v2414: 47 pgs: 47 active+clean; 145 MiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 880 op/s 2024-02-09T18:04:43.054 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:42 smithi179 ceph-mon[43975]: osdmap e977: 12 total, 12 up, 11 in 2024-02-09T18:04:43.055 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:42 smithi179 ceph-mon[43975]: osdmap e978: 12 total, 12 up, 11 in 2024-02-09T18:04:43.055 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:42 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/269133489' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.19", "id": [6, 0, 8, 11, 7, 5]}]: dispatch 2024-02-09T18:04:43.057 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:43.077 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:04:43.079 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:43.079 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:04:43.142 INFO:teuthology.orchestra.run.smithi099.stderr:set 1.19 pg_upmap_items mapping to [6->0,8->11,7->5] 2024-02-09T18:04:43.167 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:42 smithi099 ceph-mon[39420]: pgmap v2414: 47 pgs: 47 active+clean; 145 MiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 880 op/s 2024-02-09T18:04:43.167 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:42 smithi099 ceph-mon[39420]: osdmap e977: 12 total, 12 up, 11 in 2024-02-09T18:04:43.167 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:42 smithi099 ceph-mon[39420]: osdmap e978: 12 total, 12 up, 11 in 2024-02-09T18:04:43.167 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:42 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/269133489' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.19", "id": [6, 0, 8, 11, 7, 5]}]: dispatch 2024-02-09T18:04:43.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:42 smithi162 ceph-mon[44304]: pgmap v2414: 47 pgs: 47 active+clean; 145 MiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 880 op/s 2024-02-09T18:04:43.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:42 smithi162 ceph-mon[44304]: osdmap e977: 12 total, 12 up, 11 in 2024-02-09T18:04:43.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:42 smithi162 ceph-mon[44304]: osdmap e978: 12 total, 12 up, 11 in 2024-02-09T18:04:43.177 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:42 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/269133489' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.19", "id": [6, 0, 8, 11, 7, 5]}]: dispatch 2024-02-09T18:04:43.180 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:04:43.212 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:43.233 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:04:43.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:43.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:04:43.337 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:04:43.365 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:43.572 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:04:44.315 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:04:44.317 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:44.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:04:44.419 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:04:44.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:44 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/269133489' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.19", "id": [6, 0, 8, 11, 7, 5]}]': finished 2024-02-09T18:04:44.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:44 smithi179 ceph-mon[43975]: osdmap e979: 12 total, 12 up, 11 in 2024-02-09T18:04:44.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:44 smithi179 ceph-mon[43975]: pgmap v2418: 47 pgs: 47 active+clean; 45 MiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 1.08k op/s 2024-02-09T18:04:44.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:44 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/269133489' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.19", "id": [6, 0, 8, 11, 7, 5]}]': finished 2024-02-09T18:04:44.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:44 smithi162 ceph-mon[44304]: osdmap e979: 12 total, 12 up, 11 in 2024-02-09T18:04:44.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:44 smithi162 ceph-mon[44304]: pgmap v2418: 47 pgs: 47 active+clean; 45 MiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 1.08k op/s 2024-02-09T18:04:44.444 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:44 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/269133489' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.19", "id": [6, 0, 8, 11, 7, 5]}]': finished 2024-02-09T18:04:44.444 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:44 smithi099 ceph-mon[39420]: osdmap e979: 12 total, 12 up, 11 in 2024-02-09T18:04:44.444 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:44 smithi099 ceph-mon[39420]: pgmap v2418: 47 pgs: 47 active+clean; 45 MiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 1.08k op/s 2024-02-09T18:04:44.447 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:44.466 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:04:44.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:44.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:04:44.570 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:04:44.598 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:44.721 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:04:44.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:44.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:04:44.825 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:04:44.858 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:44.879 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:04:44.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:44.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:04:44.919 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-09T18:04:44.977 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T18:04:44.984 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T18:04:44.999 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T18:04:45.016 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:45.017 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T18:04:45.037 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T18:04:45.037 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T18:04:45.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:45.041 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T18:04:45.057 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T18:04:45.077 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T18:04:45.142 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:04:45.170 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:45.268 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:04:45.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:45.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:04:45.299 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T18:04:45.320 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T18:04:45.340 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T18:04:45.360 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T18:04:45.372 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:04:45.380 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T18:04:45.400 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T18:04:45.401 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T18:04:45.402 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T18:04:45.402 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T18:04:45.402 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T18:04:45.402 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T18:04:45.402 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T18:04:45.402 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T18:04:45.402 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T18:04:45.402 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T18:04:45.402 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T18:04:45.402 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T18:04:45.402 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T18:04:45.403 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:45 smithi162 ceph-mon[44304]: osdmap e980: 12 total, 12 up, 11 in 2024-02-09T18:04:45.403 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:45 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1936386061' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:04:45.407 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:45.407 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:45 smithi099 ceph-mon[39420]: osdmap e980: 12 total, 12 up, 11 in 2024-02-09T18:04:45.407 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:45 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1936386061' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:04:45.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:45 smithi179 ceph-mon[43975]: osdmap e980: 12 total, 12 up, 11 in 2024-02-09T18:04:45.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:45 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1936386061' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:04:45.426 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:04:45.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:45.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:04:45.531 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:04:45.560 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:45.582 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:04:45.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:45.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:04:45.686 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:04:45.714 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:46.063 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:04:46.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:46.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:04:46.167 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:04:46.199 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:46.220 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:04:46.223 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:46.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:04:46.325 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:04:46.328 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:46 smithi099 ceph-mon[39420]: osdmap e981: 12 total, 12 up, 11 in 2024-02-09T18:04:46.328 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:46 smithi099 ceph-mon[39420]: pgmap v2421: 47 pgs: 47 active+clean; 45 MiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 447 op/s 2024-02-09T18:04:46.354 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:46.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:46 smithi179 ceph-mon[43975]: osdmap e981: 12 total, 12 up, 11 in 2024-02-09T18:04:46.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:46 smithi179 ceph-mon[43975]: pgmap v2421: 47 pgs: 47 active+clean; 45 MiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 447 op/s 2024-02-09T18:04:46.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:46 smithi162 ceph-mon[44304]: osdmap e981: 12 total, 12 up, 11 in 2024-02-09T18:04:46.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:46 smithi162 ceph-mon[44304]: pgmap v2421: 47 pgs: 47 active+clean; 45 MiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 447 op/s 2024-02-09T18:04:46.427 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:04:46.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:46.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:04:46.541 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T18:04:46.573 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:46.595 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T18:04:46.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:46.598 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T18:04:46.699 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:04:46.731 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:46.752 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:04:46.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:46.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:04:46.856 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:04:46.888 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:46.910 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:04:46.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:46.913 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:04:47.014 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:04:47.039 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:04:47.042 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:47.063 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:04:47.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:47.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:04:47.073 INFO:teuthology.orchestra.run.smithi099.stdout:ERROR: (22) Invalid argument 2024-02-09T18:04:47.073 INFO:teuthology.orchestra.run.smithi099.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-09T18:04:47.162 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:47 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:04:47.162 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:47 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:04:47.162 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:47 smithi162 ceph-mon[44304]: osdmap e982: 12 total, 12 up, 11 in 2024-02-09T18:04:47.168 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:04:47.195 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:47.343 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:47 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:04:47.343 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:47 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:04:47.343 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:47 smithi099 ceph-mon[39420]: osdmap e982: 12 total, 12 up, 11 in 2024-02-09T18:04:47.343 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:04:47.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:47.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:04:47.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:47 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:04:47.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:47 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:04:47.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:47 smithi179 ceph-mon[43975]: osdmap e982: 12 total, 12 up, 11 in 2024-02-09T18:04:47.449 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8 2024-02-09T18:04:47.480 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:47.501 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8" found: no such container 2024-02-09T18:04:47.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:47.504 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.8' 2024-02-09T18:04:47.605 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:04:47.635 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:47.673 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:04:47.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:47.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:04:47.786 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T18:04:47.818 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:47.839 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T18:04:47.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:47.842 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T18:04:47.943 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:04:47.970 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:47.982 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_7 2024-02-09T18:04:47.982 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd pool rm unique_pool_7 unique_pool_7 --yes-i-really-really-mean-it 2024-02-09T18:04:47.998 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:04:48.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:48.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:04:48.102 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:04:48.128 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:48.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:48 smithi099 ceph-mon[39420]: osdmap e983: 12 total, 12 up, 11 in 2024-02-09T18:04:48.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:48 smithi099 ceph-mon[39420]: pgmap v2424: 47 pgs: 1 active+undersized+remapped, 46 active+clean; 577 KiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 176 op/s 2024-02-09T18:04:48.377 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:04:48.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:48.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:04:48.404 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-09T18:04:48.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:48 smithi179 ceph-mon[43975]: osdmap e983: 12 total, 12 up, 11 in 2024-02-09T18:04:48.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:48 smithi179 ceph-mon[43975]: pgmap v2424: 47 pgs: 1 active+undersized+remapped, 46 active+clean; 577 KiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 176 op/s 2024-02-09T18:04:48.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:48 smithi162 ceph-mon[44304]: osdmap e983: 12 total, 12 up, 11 in 2024-02-09T18:04:48.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:48 smithi162 ceph-mon[44304]: pgmap v2424: 47 pgs: 1 active+undersized+remapped, 46 active+clean; 577 KiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 176 op/s 2024-02-09T18:04:48.435 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:04:48.481 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:04:48.509 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:48.610 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:04:48.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:48.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:04:48.715 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T18:04:48.747 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:48.769 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T18:04:48.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:48.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T18:04:48.793 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-09T18:04:48.872 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T18:04:48.905 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:48.927 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T18:04:48.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:48.930 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T18:04:49.031 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:04:49.057 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:49.111 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:04:49.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:49.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:04:49.215 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10 2024-02-09T18:04:49.247 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:49.269 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10" found: no such container 2024-02-09T18:04:49.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:49.272 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.10' 2024-02-09T18:04:49.294 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 5, 6, 7, 8, 9, 11, 0, 2, 4] out_osds: [10] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 7, 9, 10, 11, 8, 3] 2024-02-09T18:04:49.294 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-09T18:04:49.295 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-02-09T18:04:49.295 INFO:tasks.thrashosds.thrasher:Setting osd 11 primary_affinity to 0.982899 2024-02-09T18:04:49.295 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd primary-affinity 11 0.9828994659789425 2024-02-09T18:04:49.309 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:49 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:04:49.309 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:49 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:04:49.309 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:49 smithi099 ceph-mon[39420]: osdmap e984: 12 total, 12 up, 11 in 2024-02-09T18:04:49.372 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:04:49.405 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:49.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:49 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:04:49.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:49 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:04:49.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:49 smithi179 ceph-mon[43975]: osdmap e984: 12 total, 12 up, 11 in 2024-02-09T18:04:49.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:49 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:04:49.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:49 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:04:49.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:49 smithi162 ceph-mon[44304]: osdmap e984: 12 total, 12 up, 11 in 2024-02-09T18:04:49.427 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:04:49.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:49.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:04:49.531 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11 2024-02-09T18:04:49.562 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:49.582 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11" found: no such container 2024-02-09T18:04:49.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:49.585 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.11' 2024-02-09T18:04:49.686 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:04:49.715 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:49.964 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:04:49.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:49.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:04:50.068 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1 2024-02-09T18:04:50.095 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:50.193 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1" found: no such container 2024-02-09T18:04:50.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:50.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.1' 2024-02-09T18:04:50.243 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:50 smithi099 ceph-mon[39420]: pgmap v2426: 47 pgs: 1 active+undersized+remapped, 46 active+clean; 577 KiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 166 op/s 2024-02-09T18:04:50.297 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:04:50.306 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:04:50.306 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:04:50.333 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:50.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:50 smithi179 ceph-mon[43975]: pgmap v2426: 47 pgs: 1 active+undersized+remapped, 46 active+clean; 577 KiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 166 op/s 2024-02-09T18:04:50.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:50 smithi162 ceph-mon[44304]: pgmap v2426: 47 pgs: 1 active+undersized+remapped, 46 active+clean; 577 KiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 166 op/s 2024-02-09T18:04:51.185 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is set 2024-02-09T18:04:51.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:51 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1445169412' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T18:04:51.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:51 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1445169412' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T18:04:51.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:51 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1445169412' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-09T18:04:51.882 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:04:51.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:51.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:04:51.989 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:04:52.017 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:52.039 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:04:52.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:52.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:04:52.146 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6 2024-02-09T18:04:52.179 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:52.201 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6" found: no such container 2024-02-09T18:04:52.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:52.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.6' 2024-02-09T18:04:52.294 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:52 smithi099 ceph-mon[39420]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:04:52.294 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:52 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1445169412' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T18:04:52.294 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:52 smithi099 ceph-mon[39420]: osdmap e985: 12 total, 12 up, 11 in 2024-02-09T18:04:52.294 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:52 smithi099 ceph-mon[39420]: pgmap v2428: 47 pgs: 47 active+clean; 577 KiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 40 op/s 2024-02-09T18:04:52.304 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:04:52.334 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:52.533 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:04:52.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:52.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:04:52.637 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4 2024-02-09T18:04:52.667 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:52 smithi162 ceph-mon[44304]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:04:52.667 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:52 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1445169412' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T18:04:52.667 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:52 smithi162 ceph-mon[44304]: osdmap e985: 12 total, 12 up, 11 in 2024-02-09T18:04:52.667 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:52 smithi162 ceph-mon[44304]: pgmap v2428: 47 pgs: 47 active+clean; 577 KiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 40 op/s 2024-02-09T18:04:52.669 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:52.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:52 smithi179 ceph-mon[43975]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:04:52.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:52 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1445169412' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-09T18:04:52.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:52 smithi179 ceph-mon[43975]: osdmap e985: 12 total, 12 up, 11 in 2024-02-09T18:04:52.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:52 smithi179 ceph-mon[43975]: pgmap v2428: 47 pgs: 47 active+clean; 577 KiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 40 op/s 2024-02-09T18:04:52.691 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4" found: no such container 2024-02-09T18:04:52.694 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:52.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.4' 2024-02-09T18:04:52.795 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9 2024-02-09T18:04:52.828 INFO:teuthology.orchestra.run.smithi179.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:52.849 INFO:teuthology.orchestra.run.smithi179.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9" found: no such container 2024-02-09T18:04:52.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:52.853 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi179 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.9' 2024-02-09T18:04:52.888 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:04:52.954 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:04:52.985 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:53.007 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:04:53.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:53.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:04:53.111 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2 2024-02-09T18:04:53.140 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:53.196 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'unique_pool_7' removed 2024-02-09T18:04:53.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:53 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/829089169' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-09T18:04:53.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:53 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/829089169' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-09T18:04:53.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:53 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/829089169' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-09T18:04:54.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:54 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/829089169' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-02-09T18:04:54.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:54 smithi099 ceph-mon[39420]: osdmap e986: 12 total, 12 up, 11 in 2024-02-09T18:04:54.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:54 smithi099 ceph-mon[39420]: pgmap v2430: 31 pgs: 31 active+clean; 577 KiB data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:04:54.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:54 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/829089169' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-02-09T18:04:54.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:54 smithi179 ceph-mon[43975]: osdmap e986: 12 total, 12 up, 11 in 2024-02-09T18:04:54.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:54 smithi179 ceph-mon[43975]: pgmap v2430: 31 pgs: 31 active+clean; 577 KiB data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:04:54.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:54 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/829089169' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-02-09T18:04:54.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:54 smithi162 ceph-mon[44304]: osdmap e986: 12 total, 12 up, 11 in 2024-02-09T18:04:54.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:54 smithi162 ceph-mon[44304]: pgmap v2430: 31 pgs: 31 active+clean; 577 KiB data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:04:54.691 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2" found: no such container 2024-02-09T18:04:54.694 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:54.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.2' 2024-02-09T18:04:54.795 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7 2024-02-09T18:04:54.827 INFO:teuthology.orchestra.run.smithi162.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:54.849 INFO:teuthology.orchestra.run.smithi162.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7" found: no such container 2024-02-09T18:04:54.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:54.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.7' 2024-02-09T18:04:54.953 DEBUG:teuthology.orchestra.run.smithi099:> sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0 2024-02-09T18:04:54.981 INFO:teuthology.orchestra.run.smithi099.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-09T18:04:55.418 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-02-09T18:04:55.427 INFO:tasks.thrashosds:joining thrashosds 2024-02-09T18:04:55.532 INFO:teuthology.orchestra.run.smithi099.stderr:Error: no container with name or ID "ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0" found: no such container 2024-02-09T18:04:55.542 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-09T18:04:55.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi099 with status 125: 'sudo docker kill -s 1 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd.0' 2024-02-09T18:04:56.480 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset noscrub 2024-02-09T18:04:56.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:56 smithi099 ceph-mon[39420]: pgmap v2431: 31 pgs: 31 active+clean; 577 KiB data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:04:56.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:56 smithi179 ceph-mon[43975]: pgmap v2431: 31 pgs: 31 active+clean; 577 KiB data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:04:56.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:56 smithi162 ceph-mon[44304]: pgmap v2431: 31 pgs: 31 active+clean; 577 KiB data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:04:57.946 INFO:teuthology.orchestra.run.smithi099.stdout:osd.0: {} 2024-02-09T18:04:57.947 INFO:teuthology.orchestra.run.smithi099.stdout:osd.1: {} 2024-02-09T18:04:57.947 INFO:teuthology.orchestra.run.smithi099.stdout:osd.2: {} 2024-02-09T18:04:57.947 INFO:teuthology.orchestra.run.smithi099.stdout:osd.3: {} 2024-02-09T18:04:57.948 INFO:teuthology.orchestra.run.smithi099.stdout:osd.4: {} 2024-02-09T18:04:57.972 INFO:teuthology.orchestra.run.smithi099.stdout:osd.5: {} 2024-02-09T18:04:57.999 INFO:teuthology.orchestra.run.smithi099.stdout:osd.6: {} 2024-02-09T18:04:58.019 INFO:teuthology.orchestra.run.smithi099.stdout:osd.7: {} 2024-02-09T18:04:58.039 INFO:teuthology.orchestra.run.smithi099.stdout:osd.8: {} 2024-02-09T18:04:58.059 INFO:teuthology.orchestra.run.smithi099.stdout:osd.9: {} 2024-02-09T18:04:58.079 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:04:58.080 INFO:teuthology.orchestra.run.smithi099.stdout:osd.10: {} 2024-02-09T18:04:58.118 INFO:teuthology.orchestra.run.smithi099.stdout:osd.11: {} 2024-02-09T18:04:58.119 INFO:teuthology.orchestra.run.smithi099.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-09T18:04:58.119 INFO:teuthology.orchestra.run.smithi099.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-09T18:04:58.119 INFO:teuthology.orchestra.run.smithi099.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-09T18:04:58.119 INFO:teuthology.orchestra.run.smithi099.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-09T18:04:58.119 INFO:teuthology.orchestra.run.smithi099.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-09T18:04:58.119 INFO:teuthology.orchestra.run.smithi099.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-09T18:04:58.119 INFO:teuthology.orchestra.run.smithi099.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-09T18:04:58.120 INFO:teuthology.orchestra.run.smithi099.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-09T18:04:58.120 INFO:teuthology.orchestra.run.smithi099.stderr:osd.8: osd_enable_op_tracker = '' 2024-02-09T18:04:58.120 INFO:teuthology.orchestra.run.smithi099.stderr:osd.9: osd_enable_op_tracker = '' 2024-02-09T18:04:58.120 INFO:teuthology.orchestra.run.smithi099.stderr:osd.10: osd_enable_op_tracker = '' 2024-02-09T18:04:58.120 INFO:teuthology.orchestra.run.smithi099.stderr:osd.11: osd_enable_op_tracker = '' 2024-02-09T18:04:58.396 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:58 smithi099 ceph-mon[39420]: pgmap v2432: 31 pgs: 31 active+clean; 577 KiB data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:04:58.396 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:58 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1386240078' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:04:58.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:58 smithi179 ceph-mon[43975]: pgmap v2432: 31 pgs: 31 active+clean; 577 KiB data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:04:58.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:58 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1386240078' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:04:58.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:58 smithi162 ceph-mon[44304]: pgmap v2432: 31 pgs: 31 active+clean; 577 KiB data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:04:58.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:58 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1386240078' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-09T18:04:59.148 INFO:teuthology.orchestra.run.smithi099.stderr:set osd.11 primary-affinity to 0.982899 (8644152) 2024-02-09T18:04:59.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:04:59 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4198972202' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 11, "weight": 0.9828994659789425}]: dispatch 2024-02-09T18:04:59.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:04:59 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4198972202' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 11, "weight": 0.9828994659789425}]: dispatch 2024-02-09T18:04:59.492 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:04:59 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4198972202' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 11, "weight": 0.9828994659789425}]: dispatch 2024-02-09T18:05:00.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:00 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4198972202' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 0.9828994659789425}]': finished 2024-02-09T18:05:00.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:00 smithi179 ceph-mon[43975]: osdmap e987: 12 total, 12 up, 11 in 2024-02-09T18:05:00.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:00 smithi179 ceph-mon[43975]: pgmap v2434: 31 pgs: 31 active+clean; 577 KiB data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:05:00.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:00 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4198972202' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 0.9828994659789425}]': finished 2024-02-09T18:05:00.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:00 smithi162 ceph-mon[44304]: osdmap e987: 12 total, 12 up, 11 in 2024-02-09T18:05:00.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:00 smithi162 ceph-mon[44304]: pgmap v2434: 31 pgs: 31 active+clean; 577 KiB data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:05:00.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:00 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4198972202' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 0.9828994659789425}]': finished 2024-02-09T18:05:00.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:00 smithi099 ceph-mon[39420]: osdmap e987: 12 total, 12 up, 11 in 2024-02-09T18:05:00.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:00 smithi099 ceph-mon[39420]: pgmap v2434: 31 pgs: 31 active+clean; 577 KiB data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:05:01.270 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T18:05:01.270 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [], 2024-02-09T18:05:01.270 INFO:teuthology.orchestra.run.smithi099.stdout: "num_ops": 0 2024-02-09T18:05:01.270 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T18:05:02.245 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-09T18:05:02.437 INFO:teuthology.orchestra.run.smithi099.stderr:noscrub is unset 2024-02-09T18:05:02.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:02 smithi099 ceph-mon[39420]: pgmap v2435: 31 pgs: 31 active+clean; 577 KiB data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:05:02.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:02 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3607720182' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T18:05:02.740 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:05:02.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:02 smithi179 ceph-mon[43975]: pgmap v2435: 31 pgs: 31 active+clean; 577 KiB data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:05:02.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:02 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3607720182' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T18:05:02.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:02 smithi162 ceph-mon[44304]: pgmap v2435: 31 pgs: 31 active+clean; 577 KiB data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:05:02.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:02 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3607720182' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-09T18:05:03.196 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-09T18:05:03.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:03 smithi099 ceph-mon[39420]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:05:03.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:03 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3607720182' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T18:05:03.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:03 smithi099 ceph-mon[39420]: osdmap e988: 12 total, 12 up, 11 in 2024-02-09T18:05:03.868 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:05:03.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:03 smithi179 ceph-mon[43975]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:05:03.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:03 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3607720182' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T18:05:03.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:03 smithi179 ceph-mon[43975]: osdmap e988: 12 total, 12 up, 11 in 2024-02-09T18:05:03.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:03 smithi162 ceph-mon[44304]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-09T18:05:03.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:03 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3607720182' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-09T18:05:03.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:03 smithi162 ceph-mon[44304]: osdmap e988: 12 total, 12 up, 11 in 2024-02-09T18:05:04.646 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:04 smithi099 ceph-mon[39420]: pgmap v2437: 31 pgs: 31 active+clean; 577 KiB data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:05:04.647 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:04 smithi099 ceph-mon[39420]: 1.8 scrub starts 2024-02-09T18:05:04.647 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:04 smithi099 ceph-mon[39420]: 1.8 scrub ok 2024-02-09T18:05:04.853 INFO:tasks.thrashosds.thrasher:inning osd 2024-02-09T18:05:04.853 INFO:tasks.thrashosds.thrasher:Adding osd 10 2024-02-09T18:05:04.853 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd in 10 2024-02-09T18:05:04.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:04 smithi179 ceph-mon[43975]: pgmap v2437: 31 pgs: 31 active+clean; 577 KiB data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:05:04.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:04 smithi179 ceph-mon[43975]: 1.8 scrub starts 2024-02-09T18:05:04.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:04 smithi179 ceph-mon[43975]: 1.8 scrub ok 2024-02-09T18:05:04.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:04 smithi162 ceph-mon[44304]: pgmap v2437: 31 pgs: 31 active+clean; 577 KiB data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:05:04.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:04 smithi162 ceph-mon[44304]: 1.8 scrub starts 2024-02-09T18:05:04.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:04 smithi162 ceph-mon[44304]: 1.8 scrub ok 2024-02-09T18:05:05.620 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T18:05:05.620 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [], 2024-02-09T18:05:05.620 INFO:teuthology.orchestra.run.smithi099.stdout: "complaint_time": 30, 2024-02-09T18:05:05.620 INFO:teuthology.orchestra.run.smithi099.stdout: "num_blocked_ops": 0 2024-02-09T18:05:05.620 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T18:05:06.388 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-09T18:05:06.699 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:06 smithi099 ceph-mon[39420]: 1.4 scrub starts 2024-02-09T18:05:06.699 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:06 smithi099 ceph-mon[39420]: 1.4 scrub ok 2024-02-09T18:05:06.700 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:06 smithi099 ceph-mon[39420]: pgmap v2438: 31 pgs: 31 active+clean; 577 KiB data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:05:06.700 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:06 smithi099 ceph-mon[39420]: 1.a scrub starts 2024-02-09T18:05:06.700 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:06 smithi099 ceph-mon[39420]: 1.a scrub ok 2024-02-09T18:05:06.767 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:05:06.771 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:05:06.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:06 smithi179 ceph-mon[43975]: 1.4 scrub starts 2024-02-09T18:05:06.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:06 smithi179 ceph-mon[43975]: 1.4 scrub ok 2024-02-09T18:05:06.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:06 smithi179 ceph-mon[43975]: pgmap v2438: 31 pgs: 31 active+clean; 577 KiB data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:05:06.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:06 smithi179 ceph-mon[43975]: 1.a scrub starts 2024-02-09T18:05:06.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:06 smithi179 ceph-mon[43975]: 1.a scrub ok 2024-02-09T18:05:06.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:06 smithi162 ceph-mon[44304]: 1.4 scrub starts 2024-02-09T18:05:06.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:06 smithi162 ceph-mon[44304]: 1.4 scrub ok 2024-02-09T18:05:06.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:06 smithi162 ceph-mon[44304]: pgmap v2438: 31 pgs: 31 active+clean; 577 KiB data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:05:06.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:06 smithi162 ceph-mon[44304]: 1.a scrub starts 2024-02-09T18:05:06.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:06 smithi162 ceph-mon[44304]: 1.a scrub ok 2024-02-09T18:05:08.146 INFO:teuthology.orchestra.run.smithi099.stderr:nodeep-scrub is unset 2024-02-09T18:05:08.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:08 smithi179 ceph-mon[43975]: 1.13 scrub starts 2024-02-09T18:05:08.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:08 smithi179 ceph-mon[43975]: pgmap v2439: 31 pgs: 31 active+clean; 577 KiB data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:05:08.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:08 smithi179 ceph-mon[43975]: 1.14 scrub starts 2024-02-09T18:05:08.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:08 smithi179 ceph-mon[43975]: 1.14 scrub ok 2024-02-09T18:05:08.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:08 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2764843310' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T18:05:08.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:08 smithi162 ceph-mon[44304]: 1.13 scrub starts 2024-02-09T18:05:08.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:08 smithi162 ceph-mon[44304]: pgmap v2439: 31 pgs: 31 active+clean; 577 KiB data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:05:08.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:08 smithi162 ceph-mon[44304]: 1.14 scrub starts 2024-02-09T18:05:08.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:08 smithi162 ceph-mon[44304]: 1.14 scrub ok 2024-02-09T18:05:08.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:08 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2764843310' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T18:05:08.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:08 smithi099 ceph-mon[39420]: 1.13 scrub starts 2024-02-09T18:05:08.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:08 smithi099 ceph-mon[39420]: pgmap v2439: 31 pgs: 31 active+clean; 577 KiB data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:05:08.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:08 smithi099 ceph-mon[39420]: 1.14 scrub starts 2024-02-09T18:05:08.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:08 smithi099 ceph-mon[39420]: 1.14 scrub ok 2024-02-09T18:05:08.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:08 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2764843310' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-09T18:05:09.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:09 smithi179 ceph-mon[43975]: 1.13 scrub ok 2024-02-09T18:05:09.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:09 smithi179 ceph-mon[43975]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T18:05:09.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:09 smithi179 ceph-mon[43975]: Cluster is now healthy 2024-02-09T18:05:09.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:09 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2764843310' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T18:05:09.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:09 smithi179 ceph-mon[43975]: osdmap e989: 12 total, 12 up, 11 in 2024-02-09T18:05:09.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:09 smithi162 ceph-mon[44304]: 1.13 scrub ok 2024-02-09T18:05:09.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:09 smithi162 ceph-mon[44304]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T18:05:09.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:09 smithi162 ceph-mon[44304]: Cluster is now healthy 2024-02-09T18:05:09.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:09 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2764843310' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T18:05:09.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:09 smithi162 ceph-mon[44304]: osdmap e989: 12 total, 12 up, 11 in 2024-02-09T18:05:09.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:09 smithi099 ceph-mon[39420]: 1.13 scrub ok 2024-02-09T18:05:09.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:09 smithi099 ceph-mon[39420]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-09T18:05:09.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:09 smithi099 ceph-mon[39420]: Cluster is now healthy 2024-02-09T18:05:09.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:09 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2764843310' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-09T18:05:09.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:09 smithi099 ceph-mon[39420]: osdmap e989: 12 total, 12 up, 11 in 2024-02-09T18:05:10.201 INFO:teuthology.orchestra.run.smithi099.stderr:marked in osd.10. 2024-02-09T18:05:10.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:10 smithi179 ceph-mon[43975]: 1.0 scrub starts 2024-02-09T18:05:10.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:10 smithi179 ceph-mon[43975]: 1.0 scrub ok 2024-02-09T18:05:10.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:10 smithi179 ceph-mon[43975]: 1.17 scrub starts 2024-02-09T18:05:10.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:10 smithi179 ceph-mon[43975]: 1.17 scrub ok 2024-02-09T18:05:10.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:10 smithi179 ceph-mon[43975]: pgmap v2441: 31 pgs: 31 active+clean; 577 KiB data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:05:10.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:10 smithi179 ceph-mon[43975]: 1.5 scrub starts 2024-02-09T18:05:10.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:10 smithi179 ceph-mon[43975]: 1.5 scrub ok 2024-02-09T18:05:10.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:10 smithi162 ceph-mon[44304]: 1.0 scrub starts 2024-02-09T18:05:10.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:10 smithi162 ceph-mon[44304]: 1.0 scrub ok 2024-02-09T18:05:10.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:10 smithi162 ceph-mon[44304]: 1.17 scrub starts 2024-02-09T18:05:10.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:10 smithi162 ceph-mon[44304]: 1.17 scrub ok 2024-02-09T18:05:10.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:10 smithi162 ceph-mon[44304]: pgmap v2441: 31 pgs: 31 active+clean; 577 KiB data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:05:10.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:10 smithi162 ceph-mon[44304]: 1.5 scrub starts 2024-02-09T18:05:10.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:10 smithi162 ceph-mon[44304]: 1.5 scrub ok 2024-02-09T18:05:10.447 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:10 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mon-a[39397]: 2024-02-09T18:05:10.142+0000 7feb57590700 -1 mon.a@0(leader).osd e989 definitely_dead 0 2024-02-09T18:05:10.447 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:10 smithi099 ceph-mon[39420]: 1.0 scrub starts 2024-02-09T18:05:10.447 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:10 smithi099 ceph-mon[39420]: 1.0 scrub ok 2024-02-09T18:05:10.447 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:10 smithi099 ceph-mon[39420]: 1.17 scrub starts 2024-02-09T18:05:10.447 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:10 smithi099 ceph-mon[39420]: 1.17 scrub ok 2024-02-09T18:05:10.447 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:10 smithi099 ceph-mon[39420]: pgmap v2441: 31 pgs: 31 active+clean; 577 KiB data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:05:10.447 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:10 smithi099 ceph-mon[39420]: 1.5 scrub starts 2024-02-09T18:05:10.447 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:10 smithi099 ceph-mon[39420]: 1.5 scrub ok 2024-02-09T18:05:11.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:11 smithi179 ceph-mon[43975]: 1.e scrub starts 2024-02-09T18:05:11.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:11 smithi179 ceph-mon[43975]: 1.e scrub ok 2024-02-09T18:05:11.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:11 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3200402392' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["10"]}]: dispatch 2024-02-09T18:05:11.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:11 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3200402392' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["10"]}]': finished 2024-02-09T18:05:11.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:11 smithi179 ceph-mon[43975]: osdmap e990: 12 total, 12 up, 12 in 2024-02-09T18:05:11.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:11 smithi162 ceph-mon[44304]: 1.e scrub starts 2024-02-09T18:05:11.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:11 smithi162 ceph-mon[44304]: 1.e scrub ok 2024-02-09T18:05:11.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:11 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3200402392' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["10"]}]: dispatch 2024-02-09T18:05:11.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:11 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3200402392' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["10"]}]': finished 2024-02-09T18:05:11.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:11 smithi162 ceph-mon[44304]: osdmap e990: 12 total, 12 up, 12 in 2024-02-09T18:05:11.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:11 smithi099 ceph-mon[39420]: 1.e scrub starts 2024-02-09T18:05:11.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:11 smithi099 ceph-mon[39420]: 1.e scrub ok 2024-02-09T18:05:11.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:11 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3200402392' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["10"]}]: dispatch 2024-02-09T18:05:11.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:11 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3200402392' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["10"]}]': finished 2024-02-09T18:05:11.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:11 smithi099 ceph-mon[39420]: osdmap e990: 12 total, 12 up, 12 in 2024-02-09T18:05:11.632 INFO:tasks.thrashosds.thrasher:Added osd 10 2024-02-09T18:05:11.632 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-02-09T18:05:11.915 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T18:05:11.915 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 20, 2024-02-09T18:05:11.915 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 600, 2024-02-09T18:05:11.915 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [ 2024-02-09T18:05:11.915 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.915 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26305.0:116097 8.e 8:74ccbe31:::benchmark_data_smithi179_148205_object116096:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e819)", 2024-02-09T18:05:11.915 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:56:47.748905+0000", 2024-02-09T18:05:11.916 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 504.162145329, 2024-02-09T18:05:11.916 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.217008865, 2024-02-09T18:05:11.916 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:11.916 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:11.916 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:05:11.916 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26305", 2024-02-09T18:05:11.916 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:05:11.916 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 116097 2024-02-09T18:05:11.916 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.916 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:11.916 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.916 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:11.917 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.748905+0000", 2024-02-09T18:05:11.917 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:11.917 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.917 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.917 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:11.917 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.748905+0000", 2024-02-09T18:05:11.917 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:11.917 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.917 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.917 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:11.917 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.748907+0000", 2024-02-09T18:05:11.917 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.217e-06 2024-02-09T18:05:11.917 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.917 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.917 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:11.918 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.748962+0000", 2024-02-09T18:05:11.918 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.5415999999999999e-05 2024-02-09T18:05:11.918 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.918 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.918 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:11.918 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.748975+0000", 2024-02-09T18:05:11.918 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.326e-05 2024-02-09T18:05:11.918 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.918 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.918 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:11.918 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.748984+0000", 2024-02-09T18:05:11.918 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.7280000000000001e-06 2024-02-09T18:05:11.918 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.918 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.918 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:11.919 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.749029+0000", 2024-02-09T18:05:11.919 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.5045e-05 2024-02-09T18:05:11.919 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.919 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.919 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:11.919 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.749174+0000", 2024-02-09T18:05:11.919 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014466199999999999 2024-02-09T18:05:11.919 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.919 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.919 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 2,4", 2024-02-09T18:05:11.919 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.749323+0000", 2024-02-09T18:05:11.919 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014889999999999999 2024-02-09T18:05:11.919 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.919 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.920 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:11.920 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.756474+0000", 2024-02-09T18:05:11.920 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0071517459999999996 2024-02-09T18:05:11.920 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.920 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.920 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:05:11.920 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.756500+0000", 2024-02-09T18:05:11.920 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.5803999999999999e-05 2024-02-09T18:05:11.920 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.920 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.920 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:11.920 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.965826+0000", 2024-02-09T18:05:11.921 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20932557500000001 2024-02-09T18:05:11.921 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.921 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.921 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:11.921 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.965872+0000", 2024-02-09T18:05:11.921 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.5720999999999998e-05 2024-02-09T18:05:11.921 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.921 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.921 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:11.921 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:56:47.965914+0000", 2024-02-09T18:05:11.921 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2790999999999997e-05 2024-02-09T18:05:11.921 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:11.921 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:11.921 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:11.922 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.922 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.922 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26305.0:168443 8.e 8:75047d9b:::benchmark_data_smithi179_148205_object45371:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e831)", 2024-02-09T18:05:11.922 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:57:49.594515+0000", 2024-02-09T18:05:11.922 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 442.31653618799999, 2024-02-09T18:05:11.922 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.21924401700000001, 2024-02-09T18:05:11.922 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:11.922 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:11.922 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:05:11.922 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26305", 2024-02-09T18:05:11.922 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:05:11.922 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 168443 2024-02-09T18:05:11.922 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.922 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:11.923 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.923 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:11.923 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.594515+0000", 2024-02-09T18:05:11.923 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:11.923 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.923 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.923 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:11.923 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.594515+0000", 2024-02-09T18:05:11.923 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:11.923 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.923 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.923 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:11.923 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.594516+0000", 2024-02-09T18:05:11.924 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.556e-06 2024-02-09T18:05:11.924 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.924 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.924 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:11.924 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.594522+0000", 2024-02-09T18:05:11.924 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.9390000000000002e-06 2024-02-09T18:05:11.924 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.924 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.924 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:11.924 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.594537+0000", 2024-02-09T18:05:11.924 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4640999999999999e-05 2024-02-09T18:05:11.924 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.924 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.924 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:11.925 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.594545+0000", 2024-02-09T18:05:11.925 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.7559999999999999e-06 2024-02-09T18:05:11.925 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.925 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.925 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:11.925 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.594605+0000", 2024-02-09T18:05:11.925 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.0772e-05 2024-02-09T18:05:11.925 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.925 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.925 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:11.925 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.594719+0000", 2024-02-09T18:05:11.925 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000113268 2024-02-09T18:05:11.925 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.925 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.926 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 2,4", 2024-02-09T18:05:11.926 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.594795+0000", 2024-02-09T18:05:11.926 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.5986000000000001e-05 2024-02-09T18:05:11.926 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.926 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.926 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:05:11.926 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.599971+0000", 2024-02-09T18:05:11.926 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.005176117 2024-02-09T18:05:11.926 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.926 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.926 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:11.926 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.603138+0000", 2024-02-09T18:05:11.926 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.003167156 2024-02-09T18:05:11.926 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.926 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.927 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:11.927 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.813646+0000", 2024-02-09T18:05:11.927 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.21050845000000001 2024-02-09T18:05:11.927 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.927 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.927 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:11.927 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.813712+0000", 2024-02-09T18:05:11.927 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.6223999999999995e-05 2024-02-09T18:05:11.927 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.927 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.927 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:11.927 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.813759+0000", 2024-02-09T18:05:11.927 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.6152000000000001e-05 2024-02-09T18:05:11.928 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:11.928 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:11.928 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:11.928 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.928 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.928 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26305.0:168459 8.e 8:7608c4d4:::benchmark_data_smithi179_148205_object45387:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e831)", 2024-02-09T18:05:11.928 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:57:49.614584+0000", 2024-02-09T18:05:11.928 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 442.29646718599997, 2024-02-09T18:05:11.928 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20803569199999999, 2024-02-09T18:05:11.928 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:11.928 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:11.928 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:05:11.928 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26305", 2024-02-09T18:05:11.928 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:05:11.929 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 168459 2024-02-09T18:05:11.929 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.929 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:11.929 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.929 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:11.929 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.614584+0000", 2024-02-09T18:05:11.929 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:11.929 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.929 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.929 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:11.929 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.614584+0000", 2024-02-09T18:05:11.929 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:11.929 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.929 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.929 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:11.930 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.614585+0000", 2024-02-09T18:05:11.930 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.8360000000000001e-06 2024-02-09T18:05:11.930 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.930 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.930 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:11.931 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.614594+0000", 2024-02-09T18:05:11.931 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.1580000000000005e-06 2024-02-09T18:05:11.931 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.931 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.931 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:11.931 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.614608+0000", 2024-02-09T18:05:11.931 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4688000000000001e-05 2024-02-09T18:05:11.931 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.931 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.931 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:11.931 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.614619+0000", 2024-02-09T18:05:11.931 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0849999999999999e-05 2024-02-09T18:05:11.931 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.931 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.931 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:11.932 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.614696+0000", 2024-02-09T18:05:11.932 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.6582000000000006e-05 2024-02-09T18:05:11.932 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.932 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.932 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:11.932 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.614858+0000", 2024-02-09T18:05:11.932 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000161786 2024-02-09T18:05:11.932 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.932 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.932 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 2,4", 2024-02-09T18:05:11.932 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.614984+0000", 2024-02-09T18:05:11.932 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012610400000000001 2024-02-09T18:05:11.932 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.933 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.933 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:05:11.933 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.619184+0000", 2024-02-09T18:05:11.933 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0041998790000000001 2024-02-09T18:05:11.933 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.933 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.933 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:11.933 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.619502+0000", 2024-02-09T18:05:11.933 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000318955 2024-02-09T18:05:11.933 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.933 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.933 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:11.933 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.822529+0000", 2024-02-09T18:05:11.933 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20302652600000001 2024-02-09T18:05:11.934 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.934 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.934 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:11.934 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.822585+0000", 2024-02-09T18:05:11.934 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.5522999999999997e-05 2024-02-09T18:05:11.934 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.934 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.934 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:11.934 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.822619+0000", 2024-02-09T18:05:11.934 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.4805000000000003e-05 2024-02-09T18:05:11.934 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:11.934 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:11.934 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:11.934 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.935 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.935 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26305.0:236009 8.e 8:7e48ee93:::benchmark_data_smithi179_148205_object112937:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e862)", 2024-02-09T18:05:11.935 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:59:26.390644+0000", 2024-02-09T18:05:11.935 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 345.52040670000002, 2024-02-09T18:05:11.935 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.814926925, 2024-02-09T18:05:11.935 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:11.935 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "reached pg", 2024-02-09T18:05:11.935 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:05:11.936 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26305", 2024-02-09T18:05:11.936 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:05:11.936 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 236009 2024-02-09T18:05:11.936 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.936 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:11.936 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.936 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:11.936 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.390644+0000", 2024-02-09T18:05:11.936 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:11.936 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.936 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.936 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:11.936 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.390644+0000", 2024-02-09T18:05:11.937 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:11.937 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.937 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.937 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:11.937 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.390646+0000", 2024-02-09T18:05:11.937 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.6950000000000001e-06 2024-02-09T18:05:11.937 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.937 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.937 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:11.937 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.390652+0000", 2024-02-09T18:05:11.937 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.55e-06 2024-02-09T18:05:11.937 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.937 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.937 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:11.937 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.390669+0000", 2024-02-09T18:05:11.938 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.6460000000000002e-05 2024-02-09T18:05:11.938 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.938 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.938 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:11.938 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.390679+0000", 2024-02-09T18:05:11.938 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.7510000000000003e-06 2024-02-09T18:05:11.938 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.938 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.938 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:11.938 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.393354+0000", 2024-02-09T18:05:11.938 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0026755770000000002 2024-02-09T18:05:11.938 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.938 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.938 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:11.939 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.393498+0000", 2024-02-09T18:05:11.939 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014355100000000001 2024-02-09T18:05:11.939 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.939 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.939 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 2,4", 2024-02-09T18:05:11.939 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.393620+0000", 2024-02-09T18:05:11.939 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000122414 2024-02-09T18:05:11.939 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.939 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.939 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:05:11.939 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.399157+0000", 2024-02-09T18:05:11.939 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.005536862 2024-02-09T18:05:11.939 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.939 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.939 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:11.940 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.403553+0000", 2024-02-09T18:05:11.940 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.004395866 2024-02-09T18:05:11.940 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.940 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.940 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:11.940 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.412176+0000", 2024-02-09T18:05:11.940 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.008623337 2024-02-09T18:05:11.940 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.940 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.940 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T18:05:11.940 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.412189+0000", 2024-02-09T18:05:11.940 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2448e-05 2024-02-09T18:05:11.940 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.940 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.941 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:11.941 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:27.004007+0000", 2024-02-09T18:05:11.941 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.59181836399999999 2024-02-09T18:05:11.941 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.941 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.941 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T18:05:11.941 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:27.004019+0000", 2024-02-09T18:05:11.941 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2288e-05 2024-02-09T18:05:11.941 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.941 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.941 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:11.941 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:28.205498+0000", 2024-02-09T18:05:11.941 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.201478711 2024-02-09T18:05:11.941 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.942 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.942 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:11.942 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:28.205571+0000", 2024-02-09T18:05:11.942 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.3051000000000002e-05 2024-02-09T18:05:11.942 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:11.942 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:11.942 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:11.942 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.942 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.942 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26305.0:236047 8.e 8:73a6a57f:::benchmark_data_smithi179_148205_object112975:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e863)", 2024-02-09T18:05:11.942 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:59:26.471358+0000", 2024-02-09T18:05:11.942 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 345.43969270999997, 2024-02-09T18:05:11.942 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.747920285, 2024-02-09T18:05:11.942 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:11.943 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:11.943 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:05:11.943 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26305", 2024-02-09T18:05:11.943 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:05:11.943 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 236047 2024-02-09T18:05:11.943 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.943 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:11.943 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.943 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:11.944 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.471358+0000", 2024-02-09T18:05:11.944 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:11.944 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.944 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.944 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:11.944 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.471358+0000", 2024-02-09T18:05:11.944 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:11.944 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.944 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.944 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:11.944 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.471360+0000", 2024-02-09T18:05:11.945 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.9069999999999999e-06 2024-02-09T18:05:11.945 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.945 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.945 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:11.945 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.471367+0000", 2024-02-09T18:05:11.945 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.4440000000000002e-06 2024-02-09T18:05:11.945 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.945 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.945 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:11.945 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.471387+0000", 2024-02-09T18:05:11.945 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.944e-05 2024-02-09T18:05:11.945 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.945 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.945 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:11.945 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.471403+0000", 2024-02-09T18:05:11.946 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.6359e-05 2024-02-09T18:05:11.946 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.946 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.946 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:11.946 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.471478+0000", 2024-02-09T18:05:11.946 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.4571000000000001e-05 2024-02-09T18:05:11.946 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.946 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.946 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T18:05:11.946 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.471490+0000", 2024-02-09T18:05:11.946 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1836e-05 2024-02-09T18:05:11.946 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.946 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.947 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:11.947 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:27.004143+0000", 2024-02-09T18:05:11.947 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.53265329800000005 2024-02-09T18:05:11.947 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.947 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.947 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T18:05:11.947 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:27.004151+0000", 2024-02-09T18:05:11.947 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.3469999999999999e-06 2024-02-09T18:05:11.947 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.947 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.947 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:11.947 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:28.205748+0000", 2024-02-09T18:05:11.947 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.201596828 2024-02-09T18:05:11.947 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.947 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.947 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:11.948 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:28.205830+0000", 2024-02-09T18:05:11.948 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.1379999999999997e-05 2024-02-09T18:05:11.948 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.948 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.948 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 2,4,7", 2024-02-09T18:05:11.948 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:28.205921+0000", 2024-02-09T18:05:11.948 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.1828000000000002e-05 2024-02-09T18:05:11.948 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.948 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.948 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:11.948 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:28.218891+0000", 2024-02-09T18:05:11.948 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.012970014 2024-02-09T18:05:11.948 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.949 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.949 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:05:11.949 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:28.218906+0000", 2024-02-09T18:05:11.949 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4978e-05 2024-02-09T18:05:11.949 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.949 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.949 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:11.949 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:28.218948+0000", 2024-02-09T18:05:11.949 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.1356000000000002e-05 2024-02-09T18:05:11.949 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.949 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.949 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:11.949 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:28.219146+0000", 2024-02-09T18:05:11.949 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000197901 2024-02-09T18:05:11.949 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.950 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.950 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:11.950 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:28.219187+0000", 2024-02-09T18:05:11.950 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.1520000000000002e-05 2024-02-09T18:05:11.950 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.950 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.950 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:11.950 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:28.219278+0000", 2024-02-09T18:05:11.950 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.1278000000000003e-05 2024-02-09T18:05:11.950 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:11.950 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:11.950 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:11.950 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.950 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.950 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26305.0:236049 8.e 8:7a26c75e:::benchmark_data_smithi179_148205_object112977:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e863)", 2024-02-09T18:05:11.950 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:59:26.475692+0000", 2024-02-09T18:05:11.951 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 345.43535907400002, 2024-02-09T18:05:11.951 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.750529687, 2024-02-09T18:05:11.951 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:11.951 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:11.951 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:05:11.951 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26305", 2024-02-09T18:05:11.951 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:05:11.951 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 236049 2024-02-09T18:05:11.951 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.951 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:11.951 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.951 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:11.951 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.475692+0000", 2024-02-09T18:05:11.951 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:11.951 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.951 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.952 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:11.952 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.475692+0000", 2024-02-09T18:05:11.952 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:11.952 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.952 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.952 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:11.952 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.475694+0000", 2024-02-09T18:05:11.952 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.125e-06 2024-02-09T18:05:11.952 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.952 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.952 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:11.952 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.475702+0000", 2024-02-09T18:05:11.953 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.0870000000000007e-06 2024-02-09T18:05:11.953 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.953 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.953 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:11.953 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.475724+0000", 2024-02-09T18:05:11.953 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2050999999999999e-05 2024-02-09T18:05:11.953 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.953 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.953 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:11.953 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.475735+0000", 2024-02-09T18:05:11.953 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0891e-05 2024-02-09T18:05:11.953 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.953 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.953 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:11.954 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.475821+0000", 2024-02-09T18:05:11.954 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.6254999999999993e-05 2024-02-09T18:05:11.954 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.954 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.954 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T18:05:11.954 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.475832+0000", 2024-02-09T18:05:11.954 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1202e-05 2024-02-09T18:05:11.954 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.954 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.954 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:11.954 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:27.004177+0000", 2024-02-09T18:05:11.954 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.52834420699999995 2024-02-09T18:05:11.954 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.954 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.954 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T18:05:11.955 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:27.004184+0000", 2024-02-09T18:05:11.955 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.3339999999999996e-06 2024-02-09T18:05:11.955 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.955 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.955 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:11.955 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:28.210029+0000", 2024-02-09T18:05:11.955 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2058448690000001 2024-02-09T18:05:11.955 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.955 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.955 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:11.955 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:28.210149+0000", 2024-02-09T18:05:11.955 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012064 2024-02-09T18:05:11.955 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.955 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.956 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 2,4,7", 2024-02-09T18:05:11.956 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:28.210266+0000", 2024-02-09T18:05:11.956 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00011641099999999999 2024-02-09T18:05:11.956 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.956 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.956 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:05:11.956 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:28.219096+0000", 2024-02-09T18:05:11.956 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0088306919999999994 2024-02-09T18:05:11.956 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.956 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.956 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:11.956 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:28.219318+0000", 2024-02-09T18:05:11.956 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000221284 2024-02-09T18:05:11.956 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.957 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.957 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:11.957 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:28.219352+0000", 2024-02-09T18:05:11.957 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.4134000000000002e-05 2024-02-09T18:05:11.957 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.957 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.957 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:11.957 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:28.226103+0000", 2024-02-09T18:05:11.957 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0067508510000000004 2024-02-09T18:05:11.957 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.957 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.957 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:11.957 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:28.226148+0000", 2024-02-09T18:05:11.957 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.4948999999999999e-05 2024-02-09T18:05:11.957 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.958 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.958 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:11.958 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:28.226221+0000", 2024-02-09T18:05:11.958 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.3705e-05 2024-02-09T18:05:11.958 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:11.958 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:11.958 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:11.958 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.958 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.958 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26305.0:236060 8.e 8:70d5ad2e:::benchmark_data_smithi179_148205_object112988:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e863)", 2024-02-09T18:05:11.958 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:59:26.511750+0000", 2024-02-09T18:05:11.958 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 345.399300762, 2024-02-09T18:05:11.958 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.714634473, 2024-02-09T18:05:11.958 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:11.958 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:11.959 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:05:11.959 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26305", 2024-02-09T18:05:11.959 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:05:11.959 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 236060 2024-02-09T18:05:11.959 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.959 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:11.959 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.959 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:11.959 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.511750+0000", 2024-02-09T18:05:11.959 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:11.959 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.959 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.959 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:11.959 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.511750+0000", 2024-02-09T18:05:11.959 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:11.960 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.960 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.960 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:11.960 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.511752+0000", 2024-02-09T18:05:11.960 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.351e-06 2024-02-09T18:05:11.960 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.960 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.960 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:11.960 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.511761+0000", 2024-02-09T18:05:11.960 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.8310000000000007e-06 2024-02-09T18:05:11.960 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.960 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.960 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:11.960 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.511779+0000", 2024-02-09T18:05:11.961 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.7927999999999998e-05 2024-02-09T18:05:11.961 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.961 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.961 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:11.961 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.511789+0000", 2024-02-09T18:05:11.961 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0312000000000001e-05 2024-02-09T18:05:11.961 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.961 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.961 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:11.961 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.511849+0000", 2024-02-09T18:05:11.961 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.9129000000000003e-05 2024-02-09T18:05:11.961 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.961 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.961 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T18:05:11.962 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:26.511858+0000", 2024-02-09T18:05:11.962 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.5140000000000007e-06 2024-02-09T18:05:11.962 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.962 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.962 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:11.962 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:27.004210+0000", 2024-02-09T18:05:11.962 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.49235142100000001 2024-02-09T18:05:11.962 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.962 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.962 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T18:05:11.962 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:27.004221+0000", 2024-02-09T18:05:11.962 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1627999999999999e-05 2024-02-09T18:05:11.962 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.962 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.962 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:11.962 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:28.214469+0000", 2024-02-09T18:05:11.963 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.210247514 2024-02-09T18:05:11.963 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.963 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.963 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:11.963 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:28.214599+0000", 2024-02-09T18:05:11.963 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013047300000000001 2024-02-09T18:05:11.963 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.963 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.963 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 2,4,7", 2024-02-09T18:05:11.963 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:28.214741+0000", 2024-02-09T18:05:11.963 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000141404 2024-02-09T18:05:11.963 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.963 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.963 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:05:11.963 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:28.219103+0000", 2024-02-09T18:05:11.964 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0043628499999999997 2024-02-09T18:05:11.964 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.964 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.964 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:11.964 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:28.219382+0000", 2024-02-09T18:05:11.964 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00027818599999999998 2024-02-09T18:05:11.964 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.964 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.964 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:11.964 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:28.226263+0000", 2024-02-09T18:05:11.964 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0068816440000000001 2024-02-09T18:05:11.964 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.964 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.964 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:11.964 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:28.226293+0000", 2024-02-09T18:05:11.965 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.9751999999999999e-05 2024-02-09T18:05:11.965 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.965 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.965 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:11.965 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:28.226326+0000", 2024-02-09T18:05:11.965 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.2586000000000003e-05 2024-02-09T18:05:11.965 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.965 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.965 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:11.965 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:28.226385+0000", 2024-02-09T18:05:11.965 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.8950000000000003e-05 2024-02-09T18:05:11.966 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:11.966 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:11.966 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:11.966 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.966 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.966 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26305.0:237101 8.e 8:7150a798:::benchmark_data_smithi179_148205_object114029:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e866)", 2024-02-09T18:05:11.966 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:59:30.044294+0000", 2024-02-09T18:05:11.967 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 341.86675685199998, 2024-02-09T18:05:11.967 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.418291619, 2024-02-09T18:05:11.967 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:11.967 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:11.967 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:05:11.967 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26305", 2024-02-09T18:05:11.967 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:05:11.967 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 237101 2024-02-09T18:05:11.967 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.968 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:11.968 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.968 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:11.968 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.044294+0000", 2024-02-09T18:05:11.968 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:11.968 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.968 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.968 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:11.968 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.044294+0000", 2024-02-09T18:05:11.968 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:11.968 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.968 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.968 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:11.968 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.044296+0000", 2024-02-09T18:05:11.969 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.813e-06 2024-02-09T18:05:11.969 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.969 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.969 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:11.969 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.044299+0000", 2024-02-09T18:05:11.969 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.54e-06 2024-02-09T18:05:11.969 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.969 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.969 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:11.969 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.044311+0000", 2024-02-09T18:05:11.969 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1238e-05 2024-02-09T18:05:11.969 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.969 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.969 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:11.969 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.044321+0000", 2024-02-09T18:05:11.970 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0376e-05 2024-02-09T18:05:11.970 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.970 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.970 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:11.970 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.047567+0000", 2024-02-09T18:05:11.970 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0032459139999999999 2024-02-09T18:05:11.970 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.970 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.970 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:11.970 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.047726+0000", 2024-02-09T18:05:11.970 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00015911500000000001 2024-02-09T18:05:11.970 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.970 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.970 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 2,4,7", 2024-02-09T18:05:11.971 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.047877+0000", 2024-02-09T18:05:11.971 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00015116000000000001 2024-02-09T18:05:11.971 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.971 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.971 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:05:11.971 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.053791+0000", 2024-02-09T18:05:11.972 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0059139090000000002 2024-02-09T18:05:11.972 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.972 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.972 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:11.972 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.070142+0000", 2024-02-09T18:05:11.972 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.016351415000000001 2024-02-09T18:05:11.972 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.972 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.972 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:11.972 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.076728+0000", 2024-02-09T18:05:11.973 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0065859600000000001 2024-02-09T18:05:11.973 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.973 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.973 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:11.973 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.462407+0000", 2024-02-09T18:05:11.973 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.38567898900000003 2024-02-09T18:05:11.973 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.973 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.973 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:11.973 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.462470+0000", 2024-02-09T18:05:11.973 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.2561e-05 2024-02-09T18:05:11.973 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.973 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.973 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:11.974 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.462586+0000", 2024-02-09T18:05:11.974 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000115629 2024-02-09T18:05:11.974 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:11.974 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:11.974 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:11.974 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.974 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.974 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26305.0:237107 8.e 8:7fbacb68:::benchmark_data_smithi179_148205_object114035:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e867)", 2024-02-09T18:05:11.974 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:59:30.059642+0000", 2024-02-09T18:05:11.974 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 341.85140881199999, 2024-02-09T18:05:11.974 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.40309223599999999, 2024-02-09T18:05:11.974 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:11.974 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:11.974 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:05:11.974 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26305", 2024-02-09T18:05:11.975 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:05:11.975 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 237107 2024-02-09T18:05:11.975 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.975 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:11.975 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.975 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:11.975 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.059642+0000", 2024-02-09T18:05:11.975 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:11.975 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.975 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.975 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:11.975 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.059642+0000", 2024-02-09T18:05:11.976 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:11.976 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.976 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.976 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:11.976 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.059644+0000", 2024-02-09T18:05:11.976 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5230000000000001e-06 2024-02-09T18:05:11.976 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.976 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.976 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:11.976 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.059648+0000", 2024-02-09T18:05:11.976 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.0559999999999998e-06 2024-02-09T18:05:11.976 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.977 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.977 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:11.977 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.059657+0000", 2024-02-09T18:05:11.977 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.6760000000000002e-06 2024-02-09T18:05:11.977 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.977 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.977 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:11.977 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.059666+0000", 2024-02-09T18:05:11.977 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.5890000000000003e-06 2024-02-09T18:05:11.977 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.977 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.977 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:11.977 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.064399+0000", 2024-02-09T18:05:11.977 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0047331910000000003 2024-02-09T18:05:11.977 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.978 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.978 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:11.978 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.064553+0000", 2024-02-09T18:05:11.978 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000154365 2024-02-09T18:05:11.978 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.978 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.978 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 2,4,7", 2024-02-09T18:05:11.978 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.064701+0000", 2024-02-09T18:05:11.978 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000148002 2024-02-09T18:05:11.978 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.978 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.978 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:11.978 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.076773+0000", 2024-02-09T18:05:11.978 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.012071544 2024-02-09T18:05:11.979 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.979 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.979 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:05:11.979 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.077220+0000", 2024-02-09T18:05:11.979 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00044704100000000002 2024-02-09T18:05:11.979 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.979 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.979 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:11.979 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.078428+0000", 2024-02-09T18:05:11.979 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0012076229999999999 2024-02-09T18:05:11.980 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.980 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.980 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:11.980 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.462631+0000", 2024-02-09T18:05:11.980 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.38420335999999999 2024-02-09T18:05:11.980 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.980 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.980 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:11.980 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.462670+0000", 2024-02-09T18:05:11.980 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9382999999999999e-05 2024-02-09T18:05:11.981 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.981 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.981 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:11.981 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.462734+0000", 2024-02-09T18:05:11.981 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.3882999999999998e-05 2024-02-09T18:05:11.981 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:11.981 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:11.981 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:11.981 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.982 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.982 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26305.0:237110 8.e 8:7715ece3:::benchmark_data_smithi179_148205_object114038:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e867)", 2024-02-09T18:05:11.982 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:59:30.066141+0000", 2024-02-09T18:05:11.982 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 341.84491023999999, 2024-02-09T18:05:11.982 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.39673577700000001, 2024-02-09T18:05:11.982 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:11.982 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:11.982 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:05:11.982 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26305", 2024-02-09T18:05:11.982 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:05:11.982 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 237110 2024-02-09T18:05:11.982 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.982 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:11.982 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.983 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:11.983 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.066141+0000", 2024-02-09T18:05:11.983 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:11.983 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.983 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.983 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:11.983 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.066141+0000", 2024-02-09T18:05:11.984 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:11.984 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.984 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.984 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:11.984 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.066143+0000", 2024-02-09T18:05:11.985 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.052e-06 2024-02-09T18:05:11.985 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.986 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.986 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:11.986 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.066148+0000", 2024-02-09T18:05:11.986 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7869999999999998e-06 2024-02-09T18:05:11.986 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.986 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.986 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:11.986 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.066163+0000", 2024-02-09T18:05:11.986 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4192999999999999e-05 2024-02-09T18:05:11.986 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.986 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.986 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:11.986 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.066172+0000", 2024-02-09T18:05:11.987 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.9020000000000005e-06 2024-02-09T18:05:11.987 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.987 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.988 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:11.988 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.070175+0000", 2024-02-09T18:05:11.988 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0040034349999999996 2024-02-09T18:05:11.988 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.988 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.988 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:11.988 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.070311+0000", 2024-02-09T18:05:11.988 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000136174 2024-02-09T18:05:11.988 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.988 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.988 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 2,4,7", 2024-02-09T18:05:11.988 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.070465+0000", 2024-02-09T18:05:11.989 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00015384400000000001 2024-02-09T18:05:11.989 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.989 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.989 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:05:11.989 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.078375+0000", 2024-02-09T18:05:11.989 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0079100639999999996 2024-02-09T18:05:11.989 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.989 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.989 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:11.989 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.083582+0000", 2024-02-09T18:05:11.989 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0052073559999999998 2024-02-09T18:05:11.989 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.990 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.990 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:11.990 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.083623+0000", 2024-02-09T18:05:11.990 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.0924999999999999e-05 2024-02-09T18:05:11.990 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.990 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.990 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:11.990 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.462774+0000", 2024-02-09T18:05:11.990 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.37915115100000002 2024-02-09T18:05:11.990 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.990 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.990 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:11.991 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.462808+0000", 2024-02-09T18:05:11.991 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.3878000000000003e-05 2024-02-09T18:05:11.991 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.991 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.991 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:11.991 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.462876+0000", 2024-02-09T18:05:11.991 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.8015999999999994e-05 2024-02-09T18:05:11.991 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:11.991 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:11.991 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:11.991 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.991 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.991 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26305.0:237119 8.e 8:7c20ff81:::benchmark_data_smithi179_148205_object114047:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e867)", 2024-02-09T18:05:11.991 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:59:30.081666+0000", 2024-02-09T18:05:11.992 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 341.82938464099999, 2024-02-09T18:05:11.992 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.38135496699999999, 2024-02-09T18:05:11.992 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:11.992 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:11.992 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:05:11.992 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26305", 2024-02-09T18:05:11.992 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:05:11.992 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 237119 2024-02-09T18:05:11.992 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.992 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:11.992 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.992 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:11.992 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.081666+0000", 2024-02-09T18:05:11.993 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:11.993 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.993 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.993 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:11.994 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.081666+0000", 2024-02-09T18:05:11.994 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:11.994 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.994 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.994 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:11.994 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.081667+0000", 2024-02-09T18:05:11.994 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.065e-06 2024-02-09T18:05:11.994 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.994 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.994 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:11.994 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.081671+0000", 2024-02-09T18:05:11.994 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.3469999999999999e-06 2024-02-09T18:05:11.994 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.995 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.995 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:11.995 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.081679+0000", 2024-02-09T18:05:11.995 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.1559999999999995e-06 2024-02-09T18:05:11.995 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.995 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.995 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:11.995 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.081684+0000", 2024-02-09T18:05:11.995 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.8609999999999997e-06 2024-02-09T18:05:11.995 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.995 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.995 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:11.995 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.083655+0000", 2024-02-09T18:05:11.995 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0019717580000000001 2024-02-09T18:05:11.996 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.996 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.996 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:11.996 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.083804+0000", 2024-02-09T18:05:11.996 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000148782 2024-02-09T18:05:11.996 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.996 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.996 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 2,4,7", 2024-02-09T18:05:11.996 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.083956+0000", 2024-02-09T18:05:11.996 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00015187600000000001 2024-02-09T18:05:11.996 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.996 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.996 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:11.996 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.091988+0000", 2024-02-09T18:05:11.996 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.008032038 2024-02-09T18:05:11.997 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.997 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.997 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:05:11.997 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.092222+0000", 2024-02-09T18:05:11.997 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000234047 2024-02-09T18:05:11.997 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.997 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.997 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:11.997 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.100861+0000", 2024-02-09T18:05:11.997 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0086388130000000004 2024-02-09T18:05:11.997 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.997 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.997 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:11.997 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.462924+0000", 2024-02-09T18:05:11.997 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.36206300499999999 2024-02-09T18:05:11.998 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.998 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.998 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:11.998 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.462958+0000", 2024-02-09T18:05:11.998 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.3779999999999998e-05 2024-02-09T18:05:11.998 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.998 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.998 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:11.998 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.463021+0000", 2024-02-09T18:05:11.998 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.3439e-05 2024-02-09T18:05:11.998 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:11.998 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:11.998 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:11.998 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.999 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:11.999 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26305.0:237128 8.e 8:7078c95e:::benchmark_data_smithi179_148205_object114056:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e867)", 2024-02-09T18:05:11.999 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:59:30.093189+0000", 2024-02-09T18:05:11.999 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 341.81786204700001, 2024-02-09T18:05:11.999 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.37002378600000002, 2024-02-09T18:05:11.999 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:11.999 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:11.999 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:05:11.999 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26305", 2024-02-09T18:05:11.999 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:05:11.999 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 237128 2024-02-09T18:05:11.999 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:11.999 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:12.000 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.000 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:12.000 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.093189+0000", 2024-02-09T18:05:12.000 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:12.000 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.000 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.000 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:12.000 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.093189+0000", 2024-02-09T18:05:12.000 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:12.000 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.000 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.000 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:12.000 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.093190+0000", 2024-02-09T18:05:12.001 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.203e-06 2024-02-09T18:05:12.001 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.001 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.001 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:12.001 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.093195+0000", 2024-02-09T18:05:12.001 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.7310000000000001e-06 2024-02-09T18:05:12.001 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.001 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.001 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:12.001 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.093208+0000", 2024-02-09T18:05:12.001 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3166000000000001e-05 2024-02-09T18:05:12.001 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.001 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.001 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:12.001 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.093214+0000", 2024-02-09T18:05:12.002 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.3709999999999996e-06 2024-02-09T18:05:12.002 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.002 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.002 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:12.002 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.093264+0000", 2024-02-09T18:05:12.002 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.0207999999999998e-05 2024-02-09T18:05:12.002 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.002 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.002 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:12.002 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.093408+0000", 2024-02-09T18:05:12.002 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014369599999999999 2024-02-09T18:05:12.002 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.003 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.003 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 2,4,7", 2024-02-09T18:05:12.003 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.093557+0000", 2024-02-09T18:05:12.003 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000148381 2024-02-09T18:05:12.003 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.003 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.003 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:05:12.003 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.099322+0000", 2024-02-09T18:05:12.003 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0057659529999999999 2024-02-09T18:05:12.003 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.003 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.004 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:12.004 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.104290+0000", 2024-02-09T18:05:12.004 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0049675630000000004 2024-02-09T18:05:12.004 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.004 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.004 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:12.004 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.106119+0000", 2024-02-09T18:05:12.004 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.001828971 2024-02-09T18:05:12.004 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.004 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.004 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:12.004 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.463065+0000", 2024-02-09T18:05:12.004 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.35694569700000001 2024-02-09T18:05:12.004 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.005 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.005 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:12.005 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.463095+0000", 2024-02-09T18:05:12.005 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.0091e-05 2024-02-09T18:05:12.005 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.005 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.005 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:12.005 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.463213+0000", 2024-02-09T18:05:12.005 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000117755 2024-02-09T18:05:12.005 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:12.005 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:12.005 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:12.005 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.005 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.006 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26305.0:237131 8.e 8:7aa87fd1:::benchmark_data_smithi179_148205_object114059:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e867)", 2024-02-09T18:05:12.006 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:59:30.110468+0000", 2024-02-09T18:05:12.006 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 341.80058300799999, 2024-02-09T18:05:12.006 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.35289621399999999, 2024-02-09T18:05:12.006 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:12.006 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:12.006 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:05:12.006 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26305", 2024-02-09T18:05:12.006 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:05:12.006 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 237131 2024-02-09T18:05:12.006 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.006 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:12.006 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.007 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:12.007 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.110468+0000", 2024-02-09T18:05:12.007 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:12.007 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.007 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.007 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:12.007 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.110468+0000", 2024-02-09T18:05:12.007 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:12.007 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.007 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.007 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:12.007 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.110470+0000", 2024-02-09T18:05:12.007 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.1100000000000001e-06 2024-02-09T18:05:12.007 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.008 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.008 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:12.008 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.110479+0000", 2024-02-09T18:05:12.008 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.0729999999999994e-06 2024-02-09T18:05:12.008 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.008 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.008 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:12.008 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.110508+0000", 2024-02-09T18:05:12.008 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.9251000000000002e-05 2024-02-09T18:05:12.008 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.008 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.008 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:12.008 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.110520+0000", 2024-02-09T18:05:12.008 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.128e-05 2024-02-09T18:05:12.008 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.009 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.009 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:12.009 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.110580+0000", 2024-02-09T18:05:12.009 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.9985999999999998e-05 2024-02-09T18:05:12.009 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.009 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.009 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:12.009 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.110679+0000", 2024-02-09T18:05:12.009 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.9937000000000002e-05 2024-02-09T18:05:12.009 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.009 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.009 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 2,4,7", 2024-02-09T18:05:12.009 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.110770+0000", 2024-02-09T18:05:12.009 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.0309000000000006e-05 2024-02-09T18:05:12.009 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.010 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.010 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:12.010 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.116865+0000", 2024-02-09T18:05:12.010 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0060954750000000004 2024-02-09T18:05:12.010 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.010 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.010 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:05:12.010 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.117796+0000", 2024-02-09T18:05:12.010 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00093073600000000004 2024-02-09T18:05:12.010 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.010 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.010 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:12.010 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.117866+0000", 2024-02-09T18:05:12.011 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.9911000000000002e-05 2024-02-09T18:05:12.011 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.011 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.011 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:12.011 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.463255+0000", 2024-02-09T18:05:12.011 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.345389481 2024-02-09T18:05:12.011 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.011 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.011 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:12.011 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.463289+0000", 2024-02-09T18:05:12.011 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.3327999999999997e-05 2024-02-09T18:05:12.011 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.011 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.012 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:12.012 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.463364+0000", 2024-02-09T18:05:12.012 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.5337000000000001e-05 2024-02-09T18:05:12.012 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:12.012 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:12.012 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:12.012 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.012 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.012 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26305.0:237133 8.e 8:7ae1aeb8:::benchmark_data_smithi179_148205_object114061:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e867)", 2024-02-09T18:05:12.012 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:59:30.117917+0000", 2024-02-09T18:05:12.012 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 341.79313353399999, 2024-02-09T18:05:12.012 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.34567557799999998, 2024-02-09T18:05:12.012 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:12.012 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:12.013 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:05:12.013 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26305", 2024-02-09T18:05:12.013 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:05:12.013 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 237133 2024-02-09T18:05:12.013 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.013 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:12.013 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.013 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:12.013 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.117917+0000", 2024-02-09T18:05:12.013 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:12.013 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.013 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.013 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:12.014 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.117917+0000", 2024-02-09T18:05:12.014 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:12.014 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.014 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.014 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:12.014 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.117919+0000", 2024-02-09T18:05:12.014 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.652e-06 2024-02-09T18:05:12.014 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.014 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.014 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:12.014 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.117923+0000", 2024-02-09T18:05:12.014 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.0149999999999997e-06 2024-02-09T18:05:12.014 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.014 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.015 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:12.015 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.117934+0000", 2024-02-09T18:05:12.015 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0882e-05 2024-02-09T18:05:12.015 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.015 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.015 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:12.015 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.117940+0000", 2024-02-09T18:05:12.015 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.0580000000000003e-06 2024-02-09T18:05:12.015 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.015 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.015 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:12.015 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.117981+0000", 2024-02-09T18:05:12.015 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.1244e-05 2024-02-09T18:05:12.015 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.015 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.016 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:12.016 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.118078+0000", 2024-02-09T18:05:12.016 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.6896000000000002e-05 2024-02-09T18:05:12.016 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.016 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.016 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 2,4,7", 2024-02-09T18:05:12.016 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.118170+0000", 2024-02-09T18:05:12.016 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.2047999999999997e-05 2024-02-09T18:05:12.016 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.016 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.016 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:12.016 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.124115+0000", 2024-02-09T18:05:12.016 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.005945016 2024-02-09T18:05:12.017 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.017 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.017 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:05:12.017 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.124218+0000", 2024-02-09T18:05:12.017 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00010336399999999999 2024-02-09T18:05:12.017 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.017 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.017 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:12.017 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.124825+0000", 2024-02-09T18:05:12.017 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00060685800000000005 2024-02-09T18:05:12.017 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.017 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.017 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:12.017 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.463420+0000", 2024-02-09T18:05:12.017 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.33859503899999999 2024-02-09T18:05:12.018 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.018 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.018 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:12.018 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.463486+0000", 2024-02-09T18:05:12.018 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.5622000000000003e-05 2024-02-09T18:05:12.018 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.018 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.018 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:12.018 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.463593+0000", 2024-02-09T18:05:12.018 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000106884 2024-02-09T18:05:12.018 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:12.018 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:12.018 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:12.018 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.019 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.019 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26305.0:237153 8.e 8:7d91e821:::benchmark_data_smithi179_148205_object114081:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e867)", 2024-02-09T18:05:12.019 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:59:30.201005+0000", 2024-02-09T18:05:12.019 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 341.71004586999999, 2024-02-09T18:05:12.019 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.26281026299999999, 2024-02-09T18:05:12.019 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:12.019 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:12.019 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:05:12.019 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26305", 2024-02-09T18:05:12.019 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:05:12.019 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 237153 2024-02-09T18:05:12.019 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.019 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:12.019 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.020 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:12.020 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.201005+0000", 2024-02-09T18:05:12.020 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:12.020 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.020 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.020 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:12.020 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.201005+0000", 2024-02-09T18:05:12.020 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:12.020 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.020 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.020 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:12.020 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.201006+0000", 2024-02-09T18:05:12.020 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.235e-06 2024-02-09T18:05:12.020 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.021 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.021 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:12.021 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.201010+0000", 2024-02-09T18:05:12.021 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.394e-06 2024-02-09T18:05:12.021 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.021 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.021 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:12.021 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.201022+0000", 2024-02-09T18:05:12.021 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2053e-05 2024-02-09T18:05:12.021 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.021 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.021 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:12.021 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.201028+0000", 2024-02-09T18:05:12.021 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.0100000000000001e-06 2024-02-09T18:05:12.022 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.022 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.022 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:12.022 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.201062+0000", 2024-02-09T18:05:12.022 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.4180000000000001e-05 2024-02-09T18:05:12.022 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.022 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.022 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:12.022 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.201130+0000", 2024-02-09T18:05:12.022 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.7906999999999999e-05 2024-02-09T18:05:12.022 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.022 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.022 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 2,4,7", 2024-02-09T18:05:12.022 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.201217+0000", 2024-02-09T18:05:12.023 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.7398999999999994e-05 2024-02-09T18:05:12.023 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.023 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.023 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:12.023 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.205386+0000", 2024-02-09T18:05:12.023 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.004169262 2024-02-09T18:05:12.023 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.023 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.023 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:05:12.023 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.206313+0000", 2024-02-09T18:05:12.023 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00092634200000000003 2024-02-09T18:05:12.023 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.023 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.023 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:12.023 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.206343+0000", 2024-02-09T18:05:12.023 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.0318e-05 2024-02-09T18:05:12.024 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.024 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.024 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:12.024 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.463649+0000", 2024-02-09T18:05:12.024 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.25730603699999999 2024-02-09T18:05:12.024 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.024 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.024 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:12.024 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.463712+0000", 2024-02-09T18:05:12.024 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.3323000000000003e-05 2024-02-09T18:05:12.024 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.024 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.024 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:12.025 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:30.463815+0000", 2024-02-09T18:05:12.025 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000102803 2024-02-09T18:05:12.025 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:12.025 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:12.025 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:12.025 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.025 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.025 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.26563.0:162050 9.c e926/882 9:360464c7:::benchmark_data_smithi179_161915_object27168:head v 926'10145, mlcod=926'10143)", 2024-02-09T18:05:12.025 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:02:23.163516+0000", 2024-02-09T18:05:12.025 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 168.74753478900001, 2024-02-09T18:05:12.025 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.031757301000000002, 2024-02-09T18:05:12.025 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:12.025 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:12.025 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:12.026 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.026 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:12.026 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:02:23.163516+0000", 2024-02-09T18:05:12.026 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:12.026 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.026 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.026 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:12.026 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:02:23.163516+0000", 2024-02-09T18:05:12.026 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:12.026 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.026 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.026 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:12.026 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:02:23.163518+0000", 2024-02-09T18:05:12.026 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.8449999999999999e-06 2024-02-09T18:05:12.027 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.027 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.027 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:12.027 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:02:23.163529+0000", 2024-02-09T18:05:12.027 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0664e-05 2024-02-09T18:05:12.027 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.027 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.027 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:12.027 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:02:23.163547+0000", 2024-02-09T18:05:12.027 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.8910999999999999e-05 2024-02-09T18:05:12.027 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.027 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.027 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:12.027 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:02:23.163558+0000", 2024-02-09T18:05:12.027 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0445000000000001e-05 2024-02-09T18:05:12.028 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.028 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.028 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:12.028 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:02:23.163645+0000", 2024-02-09T18:05:12.028 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.7138999999999994e-05 2024-02-09T18:05:12.028 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.028 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.028 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:12.028 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:02:23.163670+0000", 2024-02-09T18:05:12.028 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.5422000000000002e-05 2024-02-09T18:05:12.028 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.028 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.028 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:12.029 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:02:23.195209+0000", 2024-02-09T18:05:12.029 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.031538433999999997 2024-02-09T18:05:12.029 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.029 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.029 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:12.029 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:02:23.195273+0000", 2024-02-09T18:05:12.029 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.4441000000000002e-05 2024-02-09T18:05:12.029 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:12.029 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:12.029 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:12.029 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.029 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.029 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.26563.0:162685 9.c e927/882 9:30595625:::benchmark_data_smithi179_161915_object27803:head v 927'10191, mlcod=927'10189)", 2024-02-09T18:05:12.030 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:02:23.918342+0000", 2024-02-09T18:05:12.030 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 167.99270929799999, 2024-02-09T18:05:12.030 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.027948067, 2024-02-09T18:05:12.030 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:12.030 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:12.030 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:12.030 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.030 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:12.030 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:02:23.918342+0000", 2024-02-09T18:05:12.030 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:12.030 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.030 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.030 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:12.031 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:02:23.918342+0000", 2024-02-09T18:05:12.031 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:12.031 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.031 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.031 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:12.031 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:02:23.918343+0000", 2024-02-09T18:05:12.031 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1969999999999999e-06 2024-02-09T18:05:12.031 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.031 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.031 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:12.031 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:02:23.918350+0000", 2024-02-09T18:05:12.031 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.4989999999999997e-06 2024-02-09T18:05:12.031 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.031 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.032 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:12.032 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:02:23.918366+0000", 2024-02-09T18:05:12.032 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.607e-05 2024-02-09T18:05:12.032 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.032 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.032 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:12.032 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:02:23.918374+0000", 2024-02-09T18:05:12.032 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.1859999999999992e-06 2024-02-09T18:05:12.032 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.032 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.032 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:12.032 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:02:23.918448+0000", 2024-02-09T18:05:12.032 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.4003000000000003e-05 2024-02-09T18:05:12.033 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.033 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.033 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:12.033 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:02:23.918464+0000", 2024-02-09T18:05:12.033 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5778999999999999e-05 2024-02-09T18:05:12.033 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.033 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.033 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:12.033 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:02:23.946241+0000", 2024-02-09T18:05:12.033 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.027776354 2024-02-09T18:05:12.033 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.033 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.033 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:12.033 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:02:23.946290+0000", 2024-02-09T18:05:12.033 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.8979e-05 2024-02-09T18:05:12.034 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:12.034 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:12.034 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:12.034 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.034 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.034 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.26563.0:190181 9.c e934/882 9:3e76a6fb:::benchmark_data_smithi179_161915_object55299:head v 934'11944, mlcod=934'11938)", 2024-02-09T18:05:12.034 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:02:56.007548+0000", 2024-02-09T18:05:12.034 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 135.90350277300001, 2024-02-09T18:05:12.034 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.026034113000000001, 2024-02-09T18:05:12.034 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:12.034 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:12.034 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:12.034 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.034 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:12.035 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:02:56.007548+0000", 2024-02-09T18:05:12.035 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:12.035 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.035 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.035 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:12.035 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:02:56.007548+0000", 2024-02-09T18:05:12.035 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:12.035 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.035 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.035 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:12.035 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:02:56.007549+0000", 2024-02-09T18:05:12.035 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.850000000000001e-07 2024-02-09T18:05:12.035 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.035 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.035 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:12.035 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:02:56.007557+0000", 2024-02-09T18:05:12.036 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.9610000000000005e-06 2024-02-09T18:05:12.036 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.036 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.036 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:12.036 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:02:56.007568+0000", 2024-02-09T18:05:12.036 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0710000000000001e-05 2024-02-09T18:05:12.036 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.036 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.036 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:12.036 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:02:56.007575+0000", 2024-02-09T18:05:12.036 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.8770000000000004e-06 2024-02-09T18:05:12.036 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.036 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.037 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:12.037 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:02:56.007625+0000", 2024-02-09T18:05:12.037 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.0111000000000002e-05 2024-02-09T18:05:12.037 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.037 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.037 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:12.037 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:02:56.007645+0000", 2024-02-09T18:05:12.037 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0758e-05 2024-02-09T18:05:12.037 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.037 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.037 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:12.037 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:02:56.033546+0000", 2024-02-09T18:05:12.037 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.025900579999999999 2024-02-09T18:05:12.037 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.037 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.037 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:12.038 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:02:56.033582+0000", 2024-02-09T18:05:12.038 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.6130999999999997e-05 2024-02-09T18:05:12.038 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:12.038 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:12.038 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:12.038 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.038 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.038 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "MOSDScrubReserve(1.19,RELEASE,e:975,from:6,reservation_nonce:0)", 2024-02-09T18:05:12.038 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:04:37.474447+0000", 2024-02-09T18:05:12.038 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 34.436603998000002, 2024-02-09T18:05:12.038 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00021813900000000001, 2024-02-09T18:05:12.038 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:12.038 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T18:05:12.038 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:12.038 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.038 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:12.039 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:37.474447+0000", 2024-02-09T18:05:12.039 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:12.039 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.039 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.039 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:12.039 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:37.474447+0000", 2024-02-09T18:05:12.039 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:12.039 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.039 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.039 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:12.039 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:37.474447+0000", 2024-02-09T18:05:12.039 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.8599999999999999e-07 2024-02-09T18:05:12.039 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.039 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.039 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:12.040 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:37.474450+0000", 2024-02-09T18:05:12.040 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.7539999999999998e-06 2024-02-09T18:05:12.040 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.040 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.040 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:12.040 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:37.474456+0000", 2024-02-09T18:05:12.040 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7799999999999997e-06 2024-02-09T18:05:12.040 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.040 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.040 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:12.040 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:37.474463+0000", 2024-02-09T18:05:12.040 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.2960000000000001e-06 2024-02-09T18:05:12.040 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.040 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.040 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:12.041 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:37.474613+0000", 2024-02-09T18:05:12.041 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000149617 2024-02-09T18:05:12.041 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.041 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.041 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:12.041 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:37.474632+0000", 2024-02-09T18:05:12.041 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.9681999999999999e-05 2024-02-09T18:05:12.041 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.041 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.041 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:12.041 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:37.474665+0000", 2024-02-09T18:05:12.041 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.2623999999999998e-05 2024-02-09T18:05:12.041 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:12.041 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:12.042 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:12.042 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.042 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.042 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "MOSDScrubReserve(1.c,RELEASE,e:975,from:3,reservation_nonce:0)", 2024-02-09T18:05:12.042 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:04:37.746960+0000", 2024-02-09T18:05:12.042 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 34.164090928, 2024-02-09T18:05:12.042 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00021495499999999999, 2024-02-09T18:05:12.042 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:12.042 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T18:05:12.042 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:12.042 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.042 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:12.042 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:37.746960+0000", 2024-02-09T18:05:12.042 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:12.042 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.042 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.043 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:12.043 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:37.746960+0000", 2024-02-09T18:05:12.043 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:12.043 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.043 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.043 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:12.043 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:37.746960+0000", 2024-02-09T18:05:12.043 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.0100000000000001e-07 2024-02-09T18:05:12.043 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.043 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.043 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:12.043 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:37.746963+0000", 2024-02-09T18:05:12.043 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.1659999999999998e-06 2024-02-09T18:05:12.043 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.043 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.044 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:12.044 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:37.746970+0000", 2024-02-09T18:05:12.044 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.5830000000000003e-06 2024-02-09T18:05:12.044 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.044 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.044 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:12.044 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:37.746978+0000", 2024-02-09T18:05:12.044 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.5140000000000001e-06 2024-02-09T18:05:12.044 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.044 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.044 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:12.044 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:37.747131+0000", 2024-02-09T18:05:12.044 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00015227400000000001 2024-02-09T18:05:12.044 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.044 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.045 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:12.045 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:37.747147+0000", 2024-02-09T18:05:12.045 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.6228000000000001e-05 2024-02-09T18:05:12.045 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:12.045 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:12.045 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:12.045 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:37.747175+0000", 2024-02-09T18:05:12.045 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.7889e-05 2024-02-09T18:05:12.045 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:12.045 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:12.045 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:12.045 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:12.045 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:12.045 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T18:05:12.159 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:05:12.203 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:12 smithi099 ceph-mon[39420]: 1.15 scrub starts 2024-02-09T18:05:12.203 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:12 smithi099 ceph-mon[39420]: 1.15 scrub ok 2024-02-09T18:05:12.203 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:12 smithi099 ceph-mon[39420]: 1.11 deep-scrub starts 2024-02-09T18:05:12.203 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:12 smithi099 ceph-mon[39420]: osdmap e991: 12 total, 12 up, 12 in 2024-02-09T18:05:12.203 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:12 smithi099 ceph-mon[39420]: 1.11 deep-scrub ok 2024-02-09T18:05:12.203 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:12 smithi099 ceph-mon[39420]: 1.3 scrub starts 2024-02-09T18:05:12.203 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:12 smithi099 ceph-mon[39420]: pgmap v2444: 31 pgs: 1 peering, 30 active+clean; 577 KiB data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:05:12.204 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:12 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T18:05:12.204 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:12 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T18:05:12.204 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:12 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T18:05:12.204 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:12 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T18:05:12.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:12 smithi179 ceph-mon[43975]: 1.15 scrub starts 2024-02-09T18:05:12.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:12 smithi179 ceph-mon[43975]: 1.15 scrub ok 2024-02-09T18:05:12.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:12 smithi179 ceph-mon[43975]: 1.11 deep-scrub starts 2024-02-09T18:05:12.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:12 smithi179 ceph-mon[43975]: osdmap e991: 12 total, 12 up, 12 in 2024-02-09T18:05:12.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:12 smithi179 ceph-mon[43975]: 1.11 deep-scrub ok 2024-02-09T18:05:12.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:12 smithi179 ceph-mon[43975]: 1.3 scrub starts 2024-02-09T18:05:12.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:12 smithi179 ceph-mon[43975]: pgmap v2444: 31 pgs: 1 peering, 30 active+clean; 577 KiB data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:05:12.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:12 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T18:05:12.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:12 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T18:05:12.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:12 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T18:05:12.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:12 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T18:05:12.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:12 smithi162 ceph-mon[44304]: 1.15 scrub starts 2024-02-09T18:05:12.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:12 smithi162 ceph-mon[44304]: 1.15 scrub ok 2024-02-09T18:05:12.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:12 smithi162 ceph-mon[44304]: 1.11 deep-scrub starts 2024-02-09T18:05:12.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:12 smithi162 ceph-mon[44304]: osdmap e991: 12 total, 12 up, 12 in 2024-02-09T18:05:12.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:12 smithi162 ceph-mon[44304]: 1.11 deep-scrub ok 2024-02-09T18:05:12.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:12 smithi162 ceph-mon[44304]: 1.3 scrub starts 2024-02-09T18:05:12.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:12 smithi162 ceph-mon[44304]: pgmap v2444: 31 pgs: 1 peering, 30 active+clean; 577 KiB data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-09T18:05:12.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:12 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T18:05:12.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:12 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T18:05:12.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:12 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T18:05:12.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:12 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T18:05:13.395 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:13 smithi099 ceph-mon[39420]: 1.3 scrub ok 2024-02-09T18:05:13.396 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:13 smithi099 ceph-mon[39420]: 1.d scrub starts 2024-02-09T18:05:13.396 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:13 smithi099 ceph-mon[39420]: 1.d scrub ok 2024-02-09T18:05:13.396 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:13 smithi099 ceph-mon[39420]: osdmap e992: 12 total, 12 up, 12 in 2024-02-09T18:05:13.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:13 smithi179 ceph-mon[43975]: 1.3 scrub ok 2024-02-09T18:05:13.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:13 smithi179 ceph-mon[43975]: 1.d scrub starts 2024-02-09T18:05:13.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:13 smithi179 ceph-mon[43975]: 1.d scrub ok 2024-02-09T18:05:13.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:13 smithi179 ceph-mon[43975]: osdmap e992: 12 total, 12 up, 12 in 2024-02-09T18:05:13.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:13 smithi162 ceph-mon[44304]: 1.3 scrub ok 2024-02-09T18:05:13.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:13 smithi162 ceph-mon[44304]: 1.d scrub starts 2024-02-09T18:05:13.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:13 smithi162 ceph-mon[44304]: 1.d scrub ok 2024-02-09T18:05:13.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:13 smithi162 ceph-mon[44304]: osdmap e992: 12 total, 12 up, 12 in 2024-02-09T18:05:14.194 INFO:teuthology.orchestra.run.smithi099.stdout:mon.a: {} 2024-02-09T18:05:14.205 INFO:teuthology.orchestra.run.smithi099.stdout:mon.c: {} 2024-02-09T18:05:14.218 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:14 smithi099 ceph-mon[39420]: 1.9 scrub starts 2024-02-09T18:05:14.219 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:14 smithi099 ceph-mon[39420]: 1.9 scrub ok 2024-02-09T18:05:14.219 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:14 smithi099 ceph-mon[39420]: osdmap e993: 12 total, 12 up, 12 in 2024-02-09T18:05:14.219 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:14 smithi099 ceph-mon[39420]: pgmap v2447: 31 pgs: 3 remapped+peering, 1 peering, 27 active+clean; 590 KiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 511 B/s rd, 16 KiB/s wr, 1 op/s; 8.8 KiB/s, 0 objects/s recovering 2024-02-09T18:05:14.219 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:14 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T18:05:14.219 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:14 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T18:05:14.219 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:14 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T18:05:14.219 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:14 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T18:05:14.219 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:14 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T18:05:14.219 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:14 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1811363656' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-09T18:05:14.223 INFO:teuthology.orchestra.run.smithi099.stdout:mon.b: {} 2024-02-09T18:05:14.232 INFO:teuthology.orchestra.run.smithi099.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-02-09T18:05:14.233 INFO:teuthology.orchestra.run.smithi099.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-02-09T18:05:14.233 INFO:teuthology.orchestra.run.smithi099.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-02-09T18:05:14.359 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-09T18:05:14.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:14 smithi179 ceph-mon[43975]: 1.9 scrub starts 2024-02-09T18:05:14.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:14 smithi179 ceph-mon[43975]: 1.9 scrub ok 2024-02-09T18:05:14.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:14 smithi179 ceph-mon[43975]: osdmap e993: 12 total, 12 up, 12 in 2024-02-09T18:05:14.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:14 smithi179 ceph-mon[43975]: pgmap v2447: 31 pgs: 3 remapped+peering, 1 peering, 27 active+clean; 590 KiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 511 B/s rd, 16 KiB/s wr, 1 op/s; 8.8 KiB/s, 0 objects/s recovering 2024-02-09T18:05:14.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:14 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T18:05:14.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:14 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T18:05:14.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:14 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T18:05:14.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:14 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T18:05:14.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:14 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T18:05:14.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:14 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1811363656' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-09T18:05:14.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:14 smithi162 ceph-mon[44304]: 1.9 scrub starts 2024-02-09T18:05:14.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:14 smithi162 ceph-mon[44304]: 1.9 scrub ok 2024-02-09T18:05:14.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:14 smithi162 ceph-mon[44304]: osdmap e993: 12 total, 12 up, 12 in 2024-02-09T18:05:14.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:14 smithi162 ceph-mon[44304]: pgmap v2447: 31 pgs: 3 remapped+peering, 1 peering, 27 active+clean; 590 KiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 511 B/s rd, 16 KiB/s wr, 1 op/s; 8.8 KiB/s, 0 objects/s recovering 2024-02-09T18:05:14.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:14 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T18:05:14.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:14 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T18:05:14.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:14 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T18:05:14.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:14 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T18:05:14.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:14 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T18:05:14.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:14 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1811363656' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-09T18:05:14.978 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:05:14.980 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 0 1 2024-02-09T18:05:15.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:15 smithi099 ceph-mon[39420]: osdmap e994: 12 total, 12 up, 12 in 2024-02-09T18:05:15.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:15 smithi099 ceph-mon[39420]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-09T18:05:15.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:15 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T18:05:15.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:15 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T18:05:15.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:15 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T18:05:15.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:15 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T18:05:15.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:15 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T18:05:15.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:15 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T18:05:15.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:15 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T18:05:15.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:15 smithi099 ceph-mon[39420]: 1.6 scrub starts 2024-02-09T18:05:15.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:15 smithi099 ceph-mon[39420]: 1.6 scrub ok 2024-02-09T18:05:15.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:15 smithi179 ceph-mon[43975]: osdmap e994: 12 total, 12 up, 12 in 2024-02-09T18:05:15.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:15 smithi179 ceph-mon[43975]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-09T18:05:15.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:15 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T18:05:15.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:15 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T18:05:15.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:15 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T18:05:15.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:15 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T18:05:15.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:15 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T18:05:15.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:15 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T18:05:15.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:15 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T18:05:15.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:15 smithi179 ceph-mon[43975]: 1.6 scrub starts 2024-02-09T18:05:15.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:15 smithi179 ceph-mon[43975]: 1.6 scrub ok 2024-02-09T18:05:15.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:15 smithi162 ceph-mon[44304]: osdmap e994: 12 total, 12 up, 12 in 2024-02-09T18:05:15.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:15 smithi162 ceph-mon[44304]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-09T18:05:15.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:15 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T18:05:15.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:15 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T18:05:15.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:15 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T18:05:15.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:15 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T18:05:15.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:15 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T18:05:15.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:15 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T18:05:15.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:15 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T18:05:15.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:15 smithi162 ceph-mon[44304]: 1.6 scrub starts 2024-02-09T18:05:15.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:15 smithi162 ceph-mon[44304]: 1.6 scrub ok 2024-02-09T18:05:16.369 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:05:16.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:16 smithi099 ceph-mon[39420]: pgmap v2449: 31 pgs: 2 remapped+peering, 5 peering, 24 active+clean; 616 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 48 KiB/s wr, 4 op/s; 25 KiB/s, 0 objects/s recovering 2024-02-09T18:05:16.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:16 smithi099 ceph-mon[39420]: 1.1 scrub starts 2024-02-09T18:05:16.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:16 smithi099 ceph-mon[39420]: 1.1 scrub ok 2024-02-09T18:05:16.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:16 smithi179 ceph-mon[43975]: pgmap v2449: 31 pgs: 2 remapped+peering, 5 peering, 24 active+clean; 616 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 48 KiB/s wr, 4 op/s; 25 KiB/s, 0 objects/s recovering 2024-02-09T18:05:16.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:16 smithi179 ceph-mon[43975]: 1.1 scrub starts 2024-02-09T18:05:16.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:16 smithi179 ceph-mon[43975]: 1.1 scrub ok 2024-02-09T18:05:16.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:16 smithi162 ceph-mon[44304]: pgmap v2449: 31 pgs: 2 remapped+peering, 5 peering, 24 active+clean; 616 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 48 KiB/s wr, 4 op/s; 25 KiB/s, 0 objects/s recovering 2024-02-09T18:05:16.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:16 smithi162 ceph-mon[44304]: 1.1 scrub starts 2024-02-09T18:05:16.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:16 smithi162 ceph-mon[44304]: 1.1 scrub ok 2024-02-09T18:05:17.001 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T18:05:17.001 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [], 2024-02-09T18:05:17.001 INFO:teuthology.orchestra.run.smithi099.stdout: "num_ops": 0 2024-02-09T18:05:17.001 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T18:05:17.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:17 smithi099 ceph-mon[39420]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-09T18:05:17.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:17 smithi099 ceph-mon[39420]: Cluster is now healthy 2024-02-09T18:05:17.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:17 smithi179 ceph-mon[43975]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-09T18:05:17.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:17 smithi179 ceph-mon[43975]: Cluster is now healthy 2024-02-09T18:05:17.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:17 smithi162 ceph-mon[44304]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-09T18:05:17.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:17 smithi162 ceph-mon[44304]: Cluster is now healthy 2024-02-09T18:05:17.809 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-09T18:05:18.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:18 smithi099 ceph-mon[39420]: pgmap v2450: 31 pgs: 4 peering, 27 active+clean; 616 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s; 18 KiB/s, 0 objects/s recovering 2024-02-09T18:05:18.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:18 smithi179 ceph-mon[43975]: pgmap v2450: 31 pgs: 4 peering, 27 active+clean; 616 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s; 18 KiB/s, 0 objects/s recovering 2024-02-09T18:05:18.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:18 smithi162 ceph-mon[44304]: pgmap v2450: 31 pgs: 4 peering, 27 active+clean; 616 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s; 18 KiB/s, 0 objects/s recovering 2024-02-09T18:05:19.191 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:05:19.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:19 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:05:19.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:19 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:05:19.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:19 smithi099 ceph-mon[39420]: osdmap e995: 12 total, 12 up, 12 in 2024-02-09T18:05:19.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:19 smithi099 ceph-mon[39420]: 1.18 scrub starts 2024-02-09T18:05:19.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:19 smithi099 ceph-mon[39420]: 1.18 scrub ok 2024-02-09T18:05:19.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:19 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:05:19.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:19 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:05:19.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:19 smithi179 ceph-mon[43975]: osdmap e995: 12 total, 12 up, 12 in 2024-02-09T18:05:19.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:19 smithi179 ceph-mon[43975]: 1.18 scrub starts 2024-02-09T18:05:19.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:19 smithi179 ceph-mon[43975]: 1.18 scrub ok 2024-02-09T18:05:19.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:19 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:05:19.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:19 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:05:19.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:19 smithi162 ceph-mon[44304]: osdmap e995: 12 total, 12 up, 12 in 2024-02-09T18:05:19.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:19 smithi162 ceph-mon[44304]: 1.18 scrub starts 2024-02-09T18:05:19.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:19 smithi162 ceph-mon[44304]: 1.18 scrub ok 2024-02-09T18:05:20.236 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.0 to 1 (10000) 2024-02-09T18:05:20.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:20 smithi099 ceph-mon[39420]: pgmap v2452: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:20.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:20 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3744528935' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-09T18:05:20.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:20 smithi179 ceph-mon[43975]: pgmap v2452: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:20.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:20 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3744528935' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-09T18:05:20.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:20 smithi162 ceph-mon[44304]: pgmap v2452: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:20.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:20 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3744528935' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-09T18:05:21.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:21 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3744528935' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-09T18:05:21.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:21 smithi099 ceph-mon[39420]: osdmap e996: 12 total, 12 up, 12 in 2024-02-09T18:05:21.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:21 smithi099 ceph-mon[39420]: 1.f scrub starts 2024-02-09T18:05:21.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:21 smithi099 ceph-mon[39420]: 1.f scrub ok 2024-02-09T18:05:21.507 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd primary-affinity 0 1 2024-02-09T18:05:21.641 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T18:05:21.642 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [], 2024-02-09T18:05:21.642 INFO:teuthology.orchestra.run.smithi099.stdout: "complaint_time": 30, 2024-02-09T18:05:21.642 INFO:teuthology.orchestra.run.smithi099.stdout: "num_blocked_ops": 0 2024-02-09T18:05:21.642 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T18:05:21.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:21 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3744528935' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-09T18:05:21.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:21 smithi179 ceph-mon[43975]: osdmap e996: 12 total, 12 up, 12 in 2024-02-09T18:05:21.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:21 smithi179 ceph-mon[43975]: 1.f scrub starts 2024-02-09T18:05:21.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:21 smithi179 ceph-mon[43975]: 1.f scrub ok 2024-02-09T18:05:21.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:21 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3744528935' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-09T18:05:21.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:21 smithi162 ceph-mon[44304]: osdmap e996: 12 total, 12 up, 12 in 2024-02-09T18:05:21.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:21 smithi162 ceph-mon[44304]: 1.f scrub starts 2024-02-09T18:05:21.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:21 smithi162 ceph-mon[44304]: 1.f scrub ok 2024-02-09T18:05:22.227 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-09T18:05:22.312 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:05:22.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:22 smithi099 ceph-mon[39420]: pgmap v2454: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:22.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:22 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:05:22.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:22 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:05:22.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:22 smithi179 ceph-mon[43975]: pgmap v2454: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:22.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:22 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:05:22.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:22 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:05:22.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:22 smithi162 ceph-mon[44304]: pgmap v2454: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:22.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:22 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:05:22.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:22 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:05:23.427 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:05:23.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:23 smithi099 ceph-mon[39420]: osdmap e997: 12 total, 12 up, 12 in 2024-02-09T18:05:23.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:23 smithi179 ceph-mon[43975]: osdmap e997: 12 total, 12 up, 12 in 2024-02-09T18:05:23.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:23 smithi162 ceph-mon[44304]: osdmap e997: 12 total, 12 up, 12 in 2024-02-09T18:05:24.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:24 smithi099 ceph-mon[39420]: 1.1e scrub starts 2024-02-09T18:05:24.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:24 smithi099 ceph-mon[39420]: pgmap v2456: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:24.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:24 smithi099 ceph-mon[39420]: 1.1e scrub ok 2024-02-09T18:05:24.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:24 smithi179 ceph-mon[43975]: 1.1e scrub starts 2024-02-09T18:05:24.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:24 smithi179 ceph-mon[43975]: pgmap v2456: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:24.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:24 smithi179 ceph-mon[43975]: 1.1e scrub ok 2024-02-09T18:05:24.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:24 smithi162 ceph-mon[44304]: 1.1e scrub starts 2024-02-09T18:05:24.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:24 smithi162 ceph-mon[44304]: pgmap v2456: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:24.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:24 smithi162 ceph-mon[44304]: 1.1e scrub ok 2024-02-09T18:05:26.094 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T18:05:26.094 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 20, 2024-02-09T18:05:26.094 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 600, 2024-02-09T18:05:26.094 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [ 2024-02-09T18:05:26.094 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.094 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26563.0:203004 9.6 9:642ebd36:::benchmark_data_smithi179_161915_object68122:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e942)", 2024-02-09T18:05:26.094 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:03:13.470566+0000", 2024-02-09T18:05:26.095 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 132.615492907, 2024-02-09T18:05:26.095 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.3025122279999999, 2024-02-09T18:05:26.095 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:26.095 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:26.095 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:05:26.095 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26563", 2024-02-09T18:05:26.095 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:05:26.095 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 203004 2024-02-09T18:05:26.095 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.096 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:26.096 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.096 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:26.096 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.470566+0000", 2024-02-09T18:05:26.096 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:26.096 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.096 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.097 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:26.097 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.470566+0000", 2024-02-09T18:05:26.097 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:26.097 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.097 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.097 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:26.097 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.470567+0000", 2024-02-09T18:05:26.097 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.0399999999999996e-07 2024-02-09T18:05:26.097 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.098 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.098 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:26.098 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.470570+0000", 2024-02-09T18:05:26.098 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.6430000000000001e-06 2024-02-09T18:05:26.098 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.098 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.098 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:26.098 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.470580+0000", 2024-02-09T18:05:26.099 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.4280000000000002e-06 2024-02-09T18:05:26.099 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.099 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.100 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:26.100 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.470586+0000", 2024-02-09T18:05:26.100 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.4910000000000002e-06 2024-02-09T18:05:26.100 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.100 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.100 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:26.100 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.470686+0000", 2024-02-09T18:05:26.101 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.9790000000000005e-05 2024-02-09T18:05:26.101 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.101 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.101 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op must wait for map", 2024-02-09T18:05:26.101 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.470693+0000", 2024-02-09T18:05:26.101 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.7410000000000003e-06 2024-02-09T18:05:26.101 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.101 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.102 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:26.102 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.473961+0000", 2024-02-09T18:05:26.102 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0032677959999999999 2024-02-09T18:05:26.102 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.102 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.102 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T18:05:26.102 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.473969+0000", 2024-02-09T18:05:26.102 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.4290000000000001e-06 2024-02-09T18:05:26.102 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.103 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.103 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:26.103 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:14.442459+0000", 2024-02-09T18:05:26.103 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.96849018899999995 2024-02-09T18:05:26.103 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.103 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.103 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T18:05:26.103 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:14.442473+0000", 2024-02-09T18:05:26.103 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3604999999999999e-05 2024-02-09T18:05:26.104 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.104 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.104 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:26.104 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.723258+0000", 2024-02-09T18:05:26.104 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.280785031 2024-02-09T18:05:26.104 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.104 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.104 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:26.105 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.726957+0000", 2024-02-09T18:05:26.105 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0036994910000000001 2024-02-09T18:05:26.105 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.105 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.105 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 5,9,10", 2024-02-09T18:05:26.105 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.727062+0000", 2024-02-09T18:05:26.105 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000105042 2024-02-09T18:05:26.105 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.105 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.106 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:05:26.106 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.731288+0000", 2024-02-09T18:05:26.106 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0042255540000000003 2024-02-09T18:05:26.106 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.106 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.106 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:26.106 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.731323+0000", 2024-02-09T18:05:26.106 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.4585999999999997e-05 2024-02-09T18:05:26.106 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.107 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.107 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:26.107 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.772747+0000", 2024-02-09T18:05:26.107 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.041424743 2024-02-09T18:05:26.107 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.107 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.107 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:26.107 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.772890+0000", 2024-02-09T18:05:26.108 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014235300000000001 2024-02-09T18:05:26.108 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.108 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.108 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:26.108 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.772962+0000", 2024-02-09T18:05:26.108 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.2180000000000003e-05 2024-02-09T18:05:26.108 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.108 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.109 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:26.109 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.773078+0000", 2024-02-09T18:05:26.109 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000116532 2024-02-09T18:05:26.109 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:26.109 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:26.109 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:26.109 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.109 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.109 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26563.0:203007 9.6 9:6ddaaad7:::benchmark_data_smithi179_161915_object68125:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e942)", 2024-02-09T18:05:26.109 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:03:13.478316+0000", 2024-02-09T18:05:26.110 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 132.607742734, 2024-02-09T18:05:26.110 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.295137843, 2024-02-09T18:05:26.110 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:26.110 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:26.110 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:05:26.110 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26563", 2024-02-09T18:05:26.110 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:05:26.110 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 203007 2024-02-09T18:05:26.110 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.110 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:26.110 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.111 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:26.111 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.478316+0000", 2024-02-09T18:05:26.111 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:26.111 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.111 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.111 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:26.111 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.478316+0000", 2024-02-09T18:05:26.111 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:26.111 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.111 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.111 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:26.111 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.478318+0000", 2024-02-09T18:05:26.111 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3599999999999999e-06 2024-02-09T18:05:26.111 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.111 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.112 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:26.112 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.478322+0000", 2024-02-09T18:05:26.112 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.4279999999999998e-06 2024-02-09T18:05:26.112 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.112 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.112 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:26.112 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.478332+0000", 2024-02-09T18:05:26.112 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.5419999999999994e-06 2024-02-09T18:05:26.112 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.112 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.112 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:26.112 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.478338+0000", 2024-02-09T18:05:26.112 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.1140000000000002e-06 2024-02-09T18:05:26.113 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.113 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.113 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:26.113 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.478373+0000", 2024-02-09T18:05:26.113 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.5531999999999998e-05 2024-02-09T18:05:26.113 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.113 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.113 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T18:05:26.113 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.478380+0000", 2024-02-09T18:05:26.113 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.2339999999999999e-06 2024-02-09T18:05:26.113 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.113 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.113 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:26.113 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:14.442826+0000", 2024-02-09T18:05:26.113 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.96444693599999998 2024-02-09T18:05:26.114 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.114 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.114 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T18:05:26.114 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:14.442836+0000", 2024-02-09T18:05:26.114 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.8060000000000006e-06 2024-02-09T18:05:26.114 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.114 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.114 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:26.114 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.731524+0000", 2024-02-09T18:05:26.114 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.288687836 2024-02-09T18:05:26.115 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.115 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.115 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:26.115 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.731624+0000", 2024-02-09T18:05:26.115 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000100048 2024-02-09T18:05:26.115 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.115 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.115 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 5,9,10", 2024-02-09T18:05:26.115 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.731759+0000", 2024-02-09T18:05:26.115 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013495 2024-02-09T18:05:26.115 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.115 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.115 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:05:26.115 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.772656+0000", 2024-02-09T18:05:26.116 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.040897368000000003 2024-02-09T18:05:26.116 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.117 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.117 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:26.117 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.772803+0000", 2024-02-09T18:05:26.117 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014618199999999999 2024-02-09T18:05:26.117 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.117 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.117 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:26.117 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.772847+0000", 2024-02-09T18:05:26.117 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.3928999999999999e-05 2024-02-09T18:05:26.117 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.117 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.117 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:26.117 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.773258+0000", 2024-02-09T18:05:26.117 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00041180299999999997 2024-02-09T18:05:26.118 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.118 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.118 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:26.118 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.773330+0000", 2024-02-09T18:05:26.118 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.1334999999999994e-05 2024-02-09T18:05:26.118 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.118 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.118 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:26.118 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.773454+0000", 2024-02-09T18:05:26.118 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012443999999999999 2024-02-09T18:05:26.118 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:26.118 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:26.118 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:26.118 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.118 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.119 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26563.0:203008 9.e 9:7f8d1f4f:::benchmark_data_smithi179_161915_object68126:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e942)", 2024-02-09T18:05:26.119 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:03:13.478357+0000", 2024-02-09T18:05:26.119 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 132.60770155700001, 2024-02-09T18:05:26.119 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2952857890000002, 2024-02-09T18:05:26.119 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:26.119 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:26.119 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:05:26.119 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26563", 2024-02-09T18:05:26.119 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:05:26.119 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 203008 2024-02-09T18:05:26.119 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.119 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:26.119 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.119 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:26.120 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.478357+0000", 2024-02-09T18:05:26.120 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:26.120 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.120 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.120 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:26.120 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.478357+0000", 2024-02-09T18:05:26.120 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:26.120 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.120 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.120 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:26.120 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.478358+0000", 2024-02-09T18:05:26.120 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.0800000000000005e-07 2024-02-09T18:05:26.120 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.121 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.121 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:26.121 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.478361+0000", 2024-02-09T18:05:26.121 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.7999999999999999e-06 2024-02-09T18:05:26.121 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.121 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.121 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:26.121 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.478368+0000", 2024-02-09T18:05:26.121 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.0620000000000003e-06 2024-02-09T18:05:26.121 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.121 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.121 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:26.121 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.478373+0000", 2024-02-09T18:05:26.121 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.6929999999999998e-06 2024-02-09T18:05:26.122 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.122 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.122 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:26.122 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.478420+0000", 2024-02-09T18:05:26.122 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.7244999999999999e-05 2024-02-09T18:05:26.122 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.122 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.122 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T18:05:26.122 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.478430+0000", 2024-02-09T18:05:26.122 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.7489999999999993e-06 2024-02-09T18:05:26.122 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.122 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.122 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:26.122 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:14.442696+0000", 2024-02-09T18:05:26.122 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.96426599000000002 2024-02-09T18:05:26.123 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.123 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.123 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T18:05:26.123 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:14.442705+0000", 2024-02-09T18:05:26.123 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.7240000000000004e-06 2024-02-09T18:05:26.123 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.123 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.123 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:26.123 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.764374+0000", 2024-02-09T18:05:26.123 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3216690609999999 2024-02-09T18:05:26.123 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.123 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.123 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:26.123 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.767753+0000", 2024-02-09T18:05:26.124 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.003378935 2024-02-09T18:05:26.124 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.124 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.124 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 7,9,10", 2024-02-09T18:05:26.124 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.767896+0000", 2024-02-09T18:05:26.124 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000142471 2024-02-09T18:05:26.124 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.124 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.124 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:05:26.124 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.772762+0000", 2024-02-09T18:05:26.125 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0048660999999999999 2024-02-09T18:05:26.125 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.125 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.125 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:26.125 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.773288+0000", 2024-02-09T18:05:26.125 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00052633499999999995 2024-02-09T18:05:26.125 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.125 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.125 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:26.125 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.773323+0000", 2024-02-09T18:05:26.125 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.4984999999999998e-05 2024-02-09T18:05:26.125 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.126 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.126 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:26.126 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.773510+0000", 2024-02-09T18:05:26.126 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00018670899999999999 2024-02-09T18:05:26.126 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.126 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.126 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:26.126 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.773549+0000", 2024-02-09T18:05:26.126 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9104999999999999e-05 2024-02-09T18:05:26.126 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.126 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.126 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:26.126 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.773643+0000", 2024-02-09T18:05:26.126 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.4316999999999998e-05 2024-02-09T18:05:26.127 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:26.127 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:26.127 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:26.127 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.127 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.127 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26563.0:203010 9.6 9:62999e77:::benchmark_data_smithi179_161915_object68128:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e942)", 2024-02-09T18:05:26.127 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:03:13.479864+0000", 2024-02-09T18:05:26.127 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 132.60619506, 2024-02-09T18:05:26.127 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.3162811379999999, 2024-02-09T18:05:26.127 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:26.127 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:26.127 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:05:26.127 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26563", 2024-02-09T18:05:26.127 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:05:26.127 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 203010 2024-02-09T18:05:26.128 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.128 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:26.128 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.128 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:26.128 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.479864+0000", 2024-02-09T18:05:26.128 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:26.128 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.128 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.128 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:26.128 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.479864+0000", 2024-02-09T18:05:26.128 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:26.128 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.129 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.129 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:26.129 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.479865+0000", 2024-02-09T18:05:26.129 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.6000000000000002e-07 2024-02-09T18:05:26.129 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.129 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.129 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:26.129 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.479869+0000", 2024-02-09T18:05:26.129 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.5800000000000002e-06 2024-02-09T18:05:26.129 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.129 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.129 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:26.129 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.479881+0000", 2024-02-09T18:05:26.129 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1873e-05 2024-02-09T18:05:26.129 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.130 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.130 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:26.130 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.479891+0000", 2024-02-09T18:05:26.130 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.5040000000000008e-06 2024-02-09T18:05:26.130 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.130 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.130 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:26.130 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.479938+0000", 2024-02-09T18:05:26.130 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.7392000000000003e-05 2024-02-09T18:05:26.130 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.130 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.130 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T18:05:26.130 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.479947+0000", 2024-02-09T18:05:26.130 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.251e-06 2024-02-09T18:05:26.131 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.131 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.131 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:26.131 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:14.442865+0000", 2024-02-09T18:05:26.131 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.96291792899999995 2024-02-09T18:05:26.131 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.131 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.131 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T18:05:26.131 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:14.442878+0000", 2024-02-09T18:05:26.131 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2329e-05 2024-02-09T18:05:26.131 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.132 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.132 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:26.132 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.785692+0000", 2024-02-09T18:05:26.132 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.342814419 2024-02-09T18:05:26.132 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.132 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.132 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:26.132 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.785820+0000", 2024-02-09T18:05:26.132 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012762599999999999 2024-02-09T18:05:26.132 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.132 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.132 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 5,9,10", 2024-02-09T18:05:26.132 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.785978+0000", 2024-02-09T18:05:26.132 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00015850700000000001 2024-02-09T18:05:26.133 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.133 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.133 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:26.133 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.795852+0000", 2024-02-09T18:05:26.133 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0098741420000000007 2024-02-09T18:05:26.133 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.133 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.133 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:26.133 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.795904+0000", 2024-02-09T18:05:26.133 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1261000000000003e-05 2024-02-09T18:05:26.133 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.133 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.134 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:05:26.134 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.795916+0000", 2024-02-09T18:05:26.134 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2235000000000001e-05 2024-02-09T18:05:26.134 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.134 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.134 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:26.134 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.795953+0000", 2024-02-09T18:05:26.134 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.6884000000000001e-05 2024-02-09T18:05:26.134 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.134 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.134 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:26.134 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.796013+0000", 2024-02-09T18:05:26.134 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.0633e-05 2024-02-09T18:05:26.134 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.134 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.134 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:26.135 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.796145+0000", 2024-02-09T18:05:26.135 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013171299999999999 2024-02-09T18:05:26.135 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:26.135 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:26.135 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:26.135 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.135 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.135 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26563.0:203017 9.e 9:79480bca:::benchmark_data_smithi179_161915_object68135:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e942)", 2024-02-09T18:05:26.135 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:03:13.486772+0000", 2024-02-09T18:05:26.135 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 132.59928717700001, 2024-02-09T18:05:26.135 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.298778386, 2024-02-09T18:05:26.135 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:26.135 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:26.135 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:05:26.136 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26563", 2024-02-09T18:05:26.136 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:05:26.136 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 203017 2024-02-09T18:05:26.136 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.136 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:26.136 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.136 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:26.136 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.486772+0000", 2024-02-09T18:05:26.136 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:26.136 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.137 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.137 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:26.137 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.486772+0000", 2024-02-09T18:05:26.137 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:26.137 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.137 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.137 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:26.137 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.486773+0000", 2024-02-09T18:05:26.137 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.6099999999999999e-07 2024-02-09T18:05:26.137 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.137 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.137 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:26.137 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.486777+0000", 2024-02-09T18:05:26.137 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2189999999999998e-06 2024-02-09T18:05:26.137 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.137 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.138 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:26.138 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.486787+0000", 2024-02-09T18:05:26.138 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.6539999999999994e-06 2024-02-09T18:05:26.138 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.138 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.138 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:26.138 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.486792+0000", 2024-02-09T18:05:26.138 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7649999999999999e-06 2024-02-09T18:05:26.138 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.138 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.138 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:26.138 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.486841+0000", 2024-02-09T18:05:26.138 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.8430000000000002e-05 2024-02-09T18:05:26.138 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.138 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.139 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T18:05:26.139 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.486848+0000", 2024-02-09T18:05:26.139 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.7100000000000001e-06 2024-02-09T18:05:26.139 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.139 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.139 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:26.139 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:14.442730+0000", 2024-02-09T18:05:26.139 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.95588244099999997 2024-02-09T18:05:26.139 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.139 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.139 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T18:05:26.139 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:14.442739+0000", 2024-02-09T18:05:26.139 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.5690000000000005e-06 2024-02-09T18:05:26.139 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.139 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.140 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:26.140 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.773715+0000", 2024-02-09T18:05:26.140 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3309760449999999 2024-02-09T18:05:26.140 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.140 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.140 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:26.140 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.773830+0000", 2024-02-09T18:05:26.140 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00011531699999999999 2024-02-09T18:05:26.140 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.140 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.140 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 7,9,10", 2024-02-09T18:05:26.140 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.773951+0000", 2024-02-09T18:05:26.140 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000120665 2024-02-09T18:05:26.141 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.141 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.141 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:26.141 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.785271+0000", 2024-02-09T18:05:26.141 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.011320289000000001 2024-02-09T18:05:26.141 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.141 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.141 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:05:26.141 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.785286+0000", 2024-02-09T18:05:26.141 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5245999999999999e-05 2024-02-09T18:05:26.141 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.141 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.141 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:26.141 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.785330+0000", 2024-02-09T18:05:26.142 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.4192e-05 2024-02-09T18:05:26.142 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.142 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.142 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:26.142 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.785373+0000", 2024-02-09T18:05:26.142 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2524999999999997e-05 2024-02-09T18:05:26.142 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.142 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.142 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:26.142 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.785446+0000", 2024-02-09T18:05:26.142 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.3421000000000001e-05 2024-02-09T18:05:26.142 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.142 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.142 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:26.143 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.785550+0000", 2024-02-09T18:05:26.143 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00010403699999999999 2024-02-09T18:05:26.143 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:26.143 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:26.143 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:26.143 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.143 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.143 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26563.0:203020 9.6 9:618d631f:::benchmark_data_smithi179_161915_object68138:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e942)", 2024-02-09T18:05:26.143 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:03:13.493786+0000", 2024-02-09T18:05:26.143 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 132.59227297300001, 2024-02-09T18:05:26.143 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.3123387969999998, 2024-02-09T18:05:26.143 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:26.143 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:26.144 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:05:26.144 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26563", 2024-02-09T18:05:26.144 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:05:26.144 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 203020 2024-02-09T18:05:26.144 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.144 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:26.144 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.144 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:26.144 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.493786+0000", 2024-02-09T18:05:26.144 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:26.144 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.144 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.144 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:26.145 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.493786+0000", 2024-02-09T18:05:26.145 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:26.145 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.145 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.145 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:26.145 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.493787+0000", 2024-02-09T18:05:26.146 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1599999999999999e-06 2024-02-09T18:05:26.146 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.146 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.146 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:26.146 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.493793+0000", 2024-02-09T18:05:26.146 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.4500000000000003e-06 2024-02-09T18:05:26.146 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.146 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.146 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:26.146 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.493804+0000", 2024-02-09T18:05:26.146 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1174e-05 2024-02-09T18:05:26.146 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.146 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.147 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:26.147 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.493812+0000", 2024-02-09T18:05:26.147 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.9589999999999995e-06 2024-02-09T18:05:26.147 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.147 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.147 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:26.147 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.493865+0000", 2024-02-09T18:05:26.148 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3309000000000001e-05 2024-02-09T18:05:26.148 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.148 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.148 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T18:05:26.148 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.493872+0000", 2024-02-09T18:05:26.149 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.5950000000000004e-06 2024-02-09T18:05:26.149 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.149 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.149 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:26.149 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:14.442909+0000", 2024-02-09T18:05:26.149 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.94903729400000003 2024-02-09T18:05:26.150 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.150 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.150 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T18:05:26.150 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:14.442917+0000", 2024-02-09T18:05:26.150 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.3829999999999992e-06 2024-02-09T18:05:26.150 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.151 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.151 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:26.151 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.790970+0000", 2024-02-09T18:05:26.151 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.348052601 2024-02-09T18:05:26.151 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.151 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.151 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:26.151 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.791079+0000", 2024-02-09T18:05:26.151 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000109104 2024-02-09T18:05:26.151 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.152 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.152 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 5,9,10", 2024-02-09T18:05:26.152 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.791198+0000", 2024-02-09T18:05:26.152 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000119255 2024-02-09T18:05:26.152 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.152 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.152 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:05:26.152 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.796156+0000", 2024-02-09T18:05:26.152 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0049577409999999999 2024-02-09T18:05:26.153 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.153 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.153 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:26.153 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.796194+0000", 2024-02-09T18:05:26.153 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.7815000000000003e-05 2024-02-09T18:05:26.153 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.153 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.153 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:26.153 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.796243+0000", 2024-02-09T18:05:26.153 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.8887999999999997e-05 2024-02-09T18:05:26.153 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.153 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.153 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:26.153 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.805986+0000", 2024-02-09T18:05:26.154 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0097428359999999995 2024-02-09T18:05:26.154 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.154 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.154 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:26.154 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.806052+0000", 2024-02-09T18:05:26.154 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.6680999999999994e-05 2024-02-09T18:05:26.154 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.154 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.154 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:26.154 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.806125+0000", 2024-02-09T18:05:26.154 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.2552000000000006e-05 2024-02-09T18:05:26.154 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:26.154 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:26.154 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:26.155 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.155 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.155 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26563.0:203029 9.e 9:7c08cbdc:::benchmark_data_smithi179_161915_object68147:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e942)", 2024-02-09T18:05:26.155 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:03:13.509723+0000", 2024-02-09T18:05:26.155 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 132.576336064, 2024-02-09T18:05:26.155 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.276171046, 2024-02-09T18:05:26.155 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:26.155 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:26.155 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:05:26.155 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26563", 2024-02-09T18:05:26.155 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:05:26.155 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 203029 2024-02-09T18:05:26.155 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.155 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:26.155 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.156 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:26.156 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.509723+0000", 2024-02-09T18:05:26.156 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:26.156 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.156 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.156 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:26.156 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.509723+0000", 2024-02-09T18:05:26.156 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:26.156 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.156 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.156 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:26.157 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.509724+0000", 2024-02-09T18:05:26.157 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1170000000000001e-06 2024-02-09T18:05:26.157 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.157 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.157 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:26.157 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.509728+0000", 2024-02-09T18:05:26.157 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.1829999999999996e-06 2024-02-09T18:05:26.157 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.157 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.157 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:26.157 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.509737+0000", 2024-02-09T18:05:26.157 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.1079999999999999e-06 2024-02-09T18:05:26.157 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.157 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.157 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:26.158 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.509744+0000", 2024-02-09T18:05:26.158 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.759e-06 2024-02-09T18:05:26.158 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.158 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.158 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:26.158 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.509806+0000", 2024-02-09T18:05:26.158 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.1970000000000005e-05 2024-02-09T18:05:26.158 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.158 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.158 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T18:05:26.158 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.509817+0000", 2024-02-09T18:05:26.158 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0665999999999999e-05 2024-02-09T18:05:26.158 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.158 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.158 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:26.159 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:14.442763+0000", 2024-02-09T18:05:26.159 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.93294627100000005 2024-02-09T18:05:26.159 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.159 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.159 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T18:05:26.159 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:14.442772+0000", 2024-02-09T18:05:26.159 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.4589999999999998e-06 2024-02-09T18:05:26.159 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.159 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.159 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:26.159 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.779412+0000", 2024-02-09T18:05:26.159 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3366402260000001 2024-02-09T18:05:26.160 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.160 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.160 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:26.160 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.779568+0000", 2024-02-09T18:05:26.160 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000156706 2024-02-09T18:05:26.160 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.160 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.161 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 7,9,10", 2024-02-09T18:05:26.161 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.779729+0000", 2024-02-09T18:05:26.161 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00016017399999999999 2024-02-09T18:05:26.161 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.161 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.161 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:05:26.162 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.785293+0000", 2024-02-09T18:05:26.162 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0055640999999999998 2024-02-09T18:05:26.162 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.162 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.162 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:26.162 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.785607+0000", 2024-02-09T18:05:26.163 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000314166 2024-02-09T18:05:26.163 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.163 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.163 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:26.163 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.785650+0000", 2024-02-09T18:05:26.163 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.3609000000000002e-05 2024-02-09T18:05:26.163 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.164 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.164 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:26.164 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.785739+0000", 2024-02-09T18:05:26.164 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.8027000000000003e-05 2024-02-09T18:05:26.164 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.164 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.165 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:26.165 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.785791+0000", 2024-02-09T18:05:26.165 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1990000000000002e-05 2024-02-09T18:05:26.165 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.165 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.165 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:26.165 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.785894+0000", 2024-02-09T18:05:26.165 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000103515 2024-02-09T18:05:26.165 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:26.165 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:26.166 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:26.166 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.166 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.166 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26563.0:203046 9.6 9:6250f7ae:::benchmark_data_smithi179_161915_object68164:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e942)", 2024-02-09T18:05:26.166 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:03:13.574880+0000", 2024-02-09T18:05:26.166 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 132.51117943700001, 2024-02-09T18:05:26.166 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2368195399999999, 2024-02-09T18:05:26.166 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:26.166 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:26.166 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:05:26.166 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26563", 2024-02-09T18:05:26.166 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:05:26.166 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 203046 2024-02-09T18:05:26.166 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.167 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:26.167 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.167 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:26.167 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.574880+0000", 2024-02-09T18:05:26.167 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:26.167 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.167 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.167 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:26.167 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.574880+0000", 2024-02-09T18:05:26.167 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:26.167 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.167 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.167 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:26.167 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.574881+0000", 2024-02-09T18:05:26.168 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3689999999999999e-06 2024-02-09T18:05:26.168 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.168 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.168 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:26.168 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.574889+0000", 2024-02-09T18:05:26.168 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.6969999999999993e-06 2024-02-09T18:05:26.168 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.168 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.168 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:26.168 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.574902+0000", 2024-02-09T18:05:26.168 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3124e-05 2024-02-09T18:05:26.168 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.168 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.169 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:26.169 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.574910+0000", 2024-02-09T18:05:26.169 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.4970000000000001e-06 2024-02-09T18:05:26.169 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.169 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.169 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:26.169 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.574956+0000", 2024-02-09T18:05:26.169 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.6156000000000003e-05 2024-02-09T18:05:26.169 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.169 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.169 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T18:05:26.169 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.574964+0000", 2024-02-09T18:05:26.170 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.6629999999999993e-06 2024-02-09T18:05:26.170 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.170 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.170 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:26.170 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:14.443003+0000", 2024-02-09T18:05:26.170 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.86803885800000002 2024-02-09T18:05:26.170 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.170 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.170 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T18:05:26.170 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:14.443009+0000", 2024-02-09T18:05:26.171 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.835e-06 2024-02-09T18:05:26.171 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.171 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.171 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:26.171 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.796280+0000", 2024-02-09T18:05:26.171 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.353271546 2024-02-09T18:05:26.171 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.171 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.171 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:26.171 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.796408+0000", 2024-02-09T18:05:26.171 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000127823 2024-02-09T18:05:26.171 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.171 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.171 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 5,9,10", 2024-02-09T18:05:26.171 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.796541+0000", 2024-02-09T18:05:26.172 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013288699999999999 2024-02-09T18:05:26.172 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.172 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.172 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:26.172 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.806189+0000", 2024-02-09T18:05:26.172 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0096475799999999994 2024-02-09T18:05:26.172 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.172 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.172 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:26.172 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.806246+0000", 2024-02-09T18:05:26.172 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7843999999999997e-05 2024-02-09T18:05:26.172 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.172 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.173 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:05:26.173 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.806259+0000", 2024-02-09T18:05:26.173 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2619000000000001e-05 2024-02-09T18:05:26.173 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.173 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.173 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:26.173 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.811535+0000", 2024-02-09T18:05:26.173 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0052763940000000002 2024-02-09T18:05:26.173 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.173 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.173 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:26.173 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.811589+0000", 2024-02-09T18:05:26.174 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3656000000000003e-05 2024-02-09T18:05:26.174 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.174 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.174 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:26.174 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.811699+0000", 2024-02-09T18:05:26.174 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000109992 2024-02-09T18:05:26.174 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:26.174 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:26.174 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:26.174 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.174 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.174 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26563.0:222994 9.6 9:6e1b0cf6:::benchmark_data_smithi179_161915_object88112:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e960)", 2024-02-09T18:05:26.174 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:03:45.754288+0000", 2024-02-09T18:05:26.174 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 100.33177055, 2024-02-09T18:05:26.174 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.036498148, 2024-02-09T18:05:26.174 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:26.175 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:26.175 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:05:26.175 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26563", 2024-02-09T18:05:26.175 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:05:26.175 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 222994 2024-02-09T18:05:26.175 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.175 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:26.175 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.175 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:26.175 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.754288+0000", 2024-02-09T18:05:26.176 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:26.176 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.176 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.176 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:26.176 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.754288+0000", 2024-02-09T18:05:26.176 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:26.176 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.176 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.176 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:26.176 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.754290+0000", 2024-02-09T18:05:26.176 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.257e-06 2024-02-09T18:05:26.177 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.177 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.177 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:26.177 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.754295+0000", 2024-02-09T18:05:26.177 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.9359999999999998e-06 2024-02-09T18:05:26.177 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.177 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.178 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:26.178 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.754305+0000", 2024-02-09T18:05:26.178 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0001e-05 2024-02-09T18:05:26.178 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.178 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.178 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:26.178 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.754312+0000", 2024-02-09T18:05:26.178 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.5669999999999997e-06 2024-02-09T18:05:26.178 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.178 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.178 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:26.178 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.760889+0000", 2024-02-09T18:05:26.178 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0065772360000000002 2024-02-09T18:05:26.178 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.179 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.179 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T18:05:26.179 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.760899+0000", 2024-02-09T18:05:26.179 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.2860000000000005e-06 2024-02-09T18:05:26.179 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.179 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.179 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:26.179 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.786182+0000", 2024-02-09T18:05:26.179 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0252835659999999 2024-02-09T18:05:26.179 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.179 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.179 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:26.179 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.786273+0000", 2024-02-09T18:05:26.179 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.0446999999999996e-05 2024-02-09T18:05:26.179 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.180 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.180 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 5,9", 2024-02-09T18:05:26.180 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.786353+0000", 2024-02-09T18:05:26.180 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.9837000000000001e-05 2024-02-09T18:05:26.180 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.180 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.180 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:26.180 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.790643+0000", 2024-02-09T18:05:26.180 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0042901739999999999 2024-02-09T18:05:26.180 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.180 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.180 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:26.180 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.790698+0000", 2024-02-09T18:05:26.180 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.5099999999999998e-05 2024-02-09T18:05:26.181 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.181 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.181 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:05:26.181 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.790708+0000", 2024-02-09T18:05:26.181 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0419e-05 2024-02-09T18:05:26.181 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.181 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.181 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:26.181 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.790757+0000", 2024-02-09T18:05:26.181 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.8903000000000003e-05 2024-02-09T18:05:26.181 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.181 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.181 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:26.181 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.790787+0000", 2024-02-09T18:05:26.181 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.9419000000000001e-05 2024-02-09T18:05:26.181 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:26.182 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:26.182 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:26.182 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.182 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.182 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26563.0:222995 9.6 9:6c24e58a:::benchmark_data_smithi179_161915_object88113:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e960)", 2024-02-09T18:05:26.182 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:03:45.754328+0000", 2024-02-09T18:05:26.182 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 100.33173082899999, 2024-02-09T18:05:26.182 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.04459309, 2024-02-09T18:05:26.182 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:26.182 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:26.182 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:05:26.182 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26563", 2024-02-09T18:05:26.182 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:05:26.182 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 222995 2024-02-09T18:05:26.183 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.183 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:26.183 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.183 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:26.183 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.754328+0000", 2024-02-09T18:05:26.183 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:26.183 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.183 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.183 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:26.183 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.754328+0000", 2024-02-09T18:05:26.183 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:26.183 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.183 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.184 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:26.184 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.754329+0000", 2024-02-09T18:05:26.184 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.4999999999999998e-07 2024-02-09T18:05:26.184 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.184 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.184 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:26.184 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.754331+0000", 2024-02-09T18:05:26.184 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.4159999999999998e-06 2024-02-09T18:05:26.184 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.185 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.185 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:26.185 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.754337+0000", 2024-02-09T18:05:26.185 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.0700000000000003e-06 2024-02-09T18:05:26.185 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.185 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.185 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:26.186 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.754342+0000", 2024-02-09T18:05:26.186 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.8910000000000003e-06 2024-02-09T18:05:26.186 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.186 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.186 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:26.186 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.760918+0000", 2024-02-09T18:05:26.186 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0065759290000000003 2024-02-09T18:05:26.186 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.186 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.186 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T18:05:26.186 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.760922+0000", 2024-02-09T18:05:26.187 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.5420000000000001e-06 2024-02-09T18:05:26.187 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.187 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.187 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:26.187 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.790805+0000", 2024-02-09T18:05:26.187 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0298835209999999 2024-02-09T18:05:26.187 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.187 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.187 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:26.187 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.790885+0000", 2024-02-09T18:05:26.187 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.9810999999999999e-05 2024-02-09T18:05:26.187 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.187 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.187 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 5,9", 2024-02-09T18:05:26.187 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.790954+0000", 2024-02-09T18:05:26.187 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.8862999999999994e-05 2024-02-09T18:05:26.188 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.188 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.188 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:26.188 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.798777+0000", 2024-02-09T18:05:26.188 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0078234800000000007 2024-02-09T18:05:26.188 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.188 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.188 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:26.188 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.798822+0000", 2024-02-09T18:05:26.188 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.4693000000000001e-05 2024-02-09T18:05:26.188 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.188 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.188 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:05:26.189 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.798836+0000", 2024-02-09T18:05:26.189 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4110999999999999e-05 2024-02-09T18:05:26.189 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.189 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.189 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:26.189 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.798882+0000", 2024-02-09T18:05:26.189 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.6269e-05 2024-02-09T18:05:26.189 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.189 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.189 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:26.189 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.798921+0000", 2024-02-09T18:05:26.189 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9044000000000001e-05 2024-02-09T18:05:26.189 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:26.189 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:26.189 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:26.190 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.190 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.190 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26563.0:223024 9.6 9:6e7bd9f7:::benchmark_data_smithi179_161915_object88142:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e960)", 2024-02-09T18:05:26.190 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:03:45.790579+0000", 2024-02-09T18:05:26.190 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 100.295480243, 2024-02-09T18:05:26.190 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0084922000000001, 2024-02-09T18:05:26.190 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:26.191 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:26.191 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:05:26.191 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26563", 2024-02-09T18:05:26.191 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:05:26.191 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 223024 2024-02-09T18:05:26.191 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.191 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:26.191 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.191 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:26.191 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.790579+0000", 2024-02-09T18:05:26.191 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:26.191 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.191 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.191 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:26.192 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.790579+0000", 2024-02-09T18:05:26.192 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:26.192 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.192 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.192 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:26.192 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.790580+0000", 2024-02-09T18:05:26.192 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.0699999999999996e-07 2024-02-09T18:05:26.192 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.192 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.192 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:26.192 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.790583+0000", 2024-02-09T18:05:26.192 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.4230000000000001e-06 2024-02-09T18:05:26.192 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.193 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.193 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:26.193 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.790591+0000", 2024-02-09T18:05:26.193 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.0269999999999995e-06 2024-02-09T18:05:26.193 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.193 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.193 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:26.193 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.790597+0000", 2024-02-09T18:05:26.193 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.0440000000000001e-06 2024-02-09T18:05:26.193 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.193 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.193 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:26.194 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.791099+0000", 2024-02-09T18:05:26.194 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00050220799999999997 2024-02-09T18:05:26.194 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.194 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.194 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T18:05:26.194 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.791107+0000", 2024-02-09T18:05:26.195 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.9070000000000007e-06 2024-02-09T18:05:26.195 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.195 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.195 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:26.195 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.794601+0000", 2024-02-09T18:05:26.195 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.003493652 2024-02-09T18:05:26.196 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.196 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.196 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:26.196 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.794694+0000", 2024-02-09T18:05:26.196 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.3598999999999996e-05 2024-02-09T18:05:26.196 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.197 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.197 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 5,9", 2024-02-09T18:05:26.197 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.794774+0000", 2024-02-09T18:05:26.197 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.9684000000000005e-05 2024-02-09T18:05:26.197 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.197 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.197 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:26.197 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.798943+0000", 2024-02-09T18:05:26.197 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0041691719999999996 2024-02-09T18:05:26.198 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.198 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.198 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:26.198 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.798987+0000", 2024-02-09T18:05:26.198 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.3414999999999997e-05 2024-02-09T18:05:26.198 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.198 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.198 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:05:26.198 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.798999+0000", 2024-02-09T18:05:26.198 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1827e-05 2024-02-09T18:05:26.198 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.199 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.199 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:26.199 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.799040+0000", 2024-02-09T18:05:26.199 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.1520999999999998e-05 2024-02-09T18:05:26.199 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.199 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.199 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:26.200 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.799071+0000", 2024-02-09T18:05:26.200 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.0914000000000003e-05 2024-02-09T18:05:26.200 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:26.200 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:26.200 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:26.200 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.200 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.200 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26563.0:223034 9.6 9:6c443062:::benchmark_data_smithi179_161915_object88152:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e960)", 2024-02-09T18:05:26.200 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:03:45.800365+0000", 2024-02-09T18:05:26.200 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 100.285694527, 2024-02-09T18:05:26.200 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0109718620000001, 2024-02-09T18:05:26.200 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:26.201 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:26.201 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:05:26.201 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26563", 2024-02-09T18:05:26.201 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:05:26.201 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 223034 2024-02-09T18:05:26.201 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.201 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:26.201 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.201 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:26.201 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.800365+0000", 2024-02-09T18:05:26.201 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:26.201 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.201 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.202 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:26.202 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.800365+0000", 2024-02-09T18:05:26.202 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:26.202 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.202 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.202 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:26.202 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.800366+0000", 2024-02-09T18:05:26.202 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.31e-06 2024-02-09T18:05:26.202 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.202 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.202 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:26.202 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.800372+0000", 2024-02-09T18:05:26.202 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.9390000000000002e-06 2024-02-09T18:05:26.202 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.203 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.203 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:26.203 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.800384+0000", 2024-02-09T18:05:26.203 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.271e-05 2024-02-09T18:05:26.203 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.203 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.203 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:26.203 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.800398+0000", 2024-02-09T18:05:26.203 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3885e-05 2024-02-09T18:05:26.203 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.203 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.203 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:26.203 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.800450+0000", 2024-02-09T18:05:26.204 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1598999999999998e-05 2024-02-09T18:05:26.204 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.204 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.204 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T18:05:26.204 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.800459+0000", 2024-02-09T18:05:26.204 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.5380000000000008e-06 2024-02-09T18:05:26.204 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.204 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.204 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:26.204 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.799108+0000", 2024-02-09T18:05:26.204 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.99864802399999997 2024-02-09T18:05:26.204 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.204 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.204 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:26.205 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.799193+0000", 2024-02-09T18:05:26.205 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.5748999999999995e-05 2024-02-09T18:05:26.205 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.205 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.205 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 5,9", 2024-02-09T18:05:26.205 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.799258+0000", 2024-02-09T18:05:26.205 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.436e-05 2024-02-09T18:05:26.205 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.205 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.205 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:26.205 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.807790+0000", 2024-02-09T18:05:26.205 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0085318800000000004 2024-02-09T18:05:26.205 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.205 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.205 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:26.205 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.807829+0000", 2024-02-09T18:05:26.206 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9388999999999998e-05 2024-02-09T18:05:26.206 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.206 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.206 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:05:26.206 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.811283+0000", 2024-02-09T18:05:26.206 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0034545410000000002 2024-02-09T18:05:26.206 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.206 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.206 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:26.206 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.811315+0000", 2024-02-09T18:05:26.206 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.1148000000000001e-05 2024-02-09T18:05:26.206 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.207 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.207 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:26.207 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.811336+0000", 2024-02-09T18:05:26.207 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.179e-05 2024-02-09T18:05:26.207 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:26.207 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:26.207 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:26.207 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.207 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.207 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26563.0:223037 9.6 9:608df3a4:::benchmark_data_smithi179_161915_object88155:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e960)", 2024-02-09T18:05:26.207 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:03:45.802367+0000", 2024-02-09T18:05:26.207 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 100.28369167699999, 2024-02-09T18:05:26.207 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.009106705, 2024-02-09T18:05:26.207 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:26.208 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:26.208 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:05:26.208 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26563", 2024-02-09T18:05:26.208 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:05:26.208 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 223037 2024-02-09T18:05:26.208 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.208 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:26.208 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.208 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:26.208 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.802367+0000", 2024-02-09T18:05:26.208 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:26.208 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.208 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.208 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:26.209 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.802367+0000", 2024-02-09T18:05:26.209 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:26.209 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.209 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.209 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:26.209 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.802368+0000", 2024-02-09T18:05:26.209 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.3099999999999997e-07 2024-02-09T18:05:26.209 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.209 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.209 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:26.209 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.802371+0000", 2024-02-09T18:05:26.209 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.3579999999999999e-06 2024-02-09T18:05:26.209 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.209 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.209 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:26.210 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.802385+0000", 2024-02-09T18:05:26.210 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3498e-05 2024-02-09T18:05:26.210 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.210 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.210 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:26.210 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.802401+0000", 2024-02-09T18:05:26.210 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.6138999999999999e-05 2024-02-09T18:05:26.210 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.210 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.210 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:26.210 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.802441+0000", 2024-02-09T18:05:26.210 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9753999999999999e-05 2024-02-09T18:05:26.210 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.210 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.210 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T18:05:26.211 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.802446+0000", 2024-02-09T18:05:26.211 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1699999999999996e-06 2024-02-09T18:05:26.211 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.211 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.211 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:26.211 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.803918+0000", 2024-02-09T18:05:26.211 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0014721499999999 2024-02-09T18:05:26.211 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.211 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.211 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:26.211 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.804015+0000", 2024-02-09T18:05:26.211 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.7063000000000002e-05 2024-02-09T18:05:26.211 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.211 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.211 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 5,9", 2024-02-09T18:05:26.212 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.804123+0000", 2024-02-09T18:05:26.212 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000107713 2024-02-09T18:05:26.212 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.212 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.212 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:26.212 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.807879+0000", 2024-02-09T18:05:26.212 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0037564640000000002 2024-02-09T18:05:26.212 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.212 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.212 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:26.212 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.811372+0000", 2024-02-09T18:05:26.213 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0034922719999999998 2024-02-09T18:05:26.213 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.213 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.213 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:05:26.213 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.811428+0000", 2024-02-09T18:05:26.213 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6447999999999999e-05 2024-02-09T18:05:26.214 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.214 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.214 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:26.214 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.811453+0000", 2024-02-09T18:05:26.214 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.5089999999999999e-05 2024-02-09T18:05:26.215 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.215 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.215 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:26.215 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.811474+0000", 2024-02-09T18:05:26.215 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0854999999999999e-05 2024-02-09T18:05:26.215 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:26.215 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:26.216 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:26.216 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.216 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.216 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26563.0:223044 9.6 9:6feecccb:::benchmark_data_smithi179_161915_object88162:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e960)", 2024-02-09T18:05:26.216 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:03:45.809214+0000", 2024-02-09T18:05:26.216 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 100.276844986, 2024-02-09T18:05:26.216 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.010393484, 2024-02-09T18:05:26.216 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:26.217 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:26.217 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:05:26.217 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26563", 2024-02-09T18:05:26.217 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:05:26.217 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 223044 2024-02-09T18:05:26.217 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.218 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:26.218 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.218 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:26.218 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.809214+0000", 2024-02-09T18:05:26.218 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:26.218 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.218 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.218 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:26.218 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.809214+0000", 2024-02-09T18:05:26.218 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:26.218 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.218 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.218 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:26.218 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.809214+0000", 2024-02-09T18:05:26.218 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.1899999999999998e-07 2024-02-09T18:05:26.219 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.219 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.219 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:26.219 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.809217+0000", 2024-02-09T18:05:26.219 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.683e-06 2024-02-09T18:05:26.219 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.219 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.219 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:26.219 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.809225+0000", 2024-02-09T18:05:26.219 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.2519999999999999e-06 2024-02-09T18:05:26.219 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.219 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.219 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:26.219 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.809231+0000", 2024-02-09T18:05:26.220 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.4659999999999998e-06 2024-02-09T18:05:26.220 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.220 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.220 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:26.220 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.809267+0000", 2024-02-09T18:05:26.220 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.5811e-05 2024-02-09T18:05:26.220 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.220 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.220 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T18:05:26.220 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.809272+0000", 2024-02-09T18:05:26.220 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6629999999999998e-06 2024-02-09T18:05:26.220 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.220 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.221 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:26.221 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.807909+0000", 2024-02-09T18:05:26.221 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.99863655600000001 2024-02-09T18:05:26.221 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.221 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.221 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:26.221 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.808003+0000", 2024-02-09T18:05:26.221 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.3945999999999998e-05 2024-02-09T18:05:26.221 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.221 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.221 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 5,9", 2024-02-09T18:05:26.221 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.808071+0000", 2024-02-09T18:05:26.221 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.8287999999999994e-05 2024-02-09T18:05:26.221 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.221 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.222 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:26.222 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.811419+0000", 2024-02-09T18:05:26.222 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0033482059999999998 2024-02-09T18:05:26.222 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.222 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.222 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:05:26.222 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.819515+0000", 2024-02-09T18:05:26.222 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0080954119999999997 2024-02-09T18:05:26.222 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.222 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.222 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:26.222 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.819555+0000", 2024-02-09T18:05:26.222 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9965000000000001e-05 2024-02-09T18:05:26.222 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.223 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.223 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:26.223 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.819587+0000", 2024-02-09T18:05:26.223 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.2329000000000002e-05 2024-02-09T18:05:26.223 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.223 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.223 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:26.223 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.819608+0000", 2024-02-09T18:05:26.223 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0488000000000001e-05 2024-02-09T18:05:26.223 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:26.223 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:26.223 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:26.224 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.224 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.224 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26563.0:223054 9.6 9:6735df74:::benchmark_data_smithi179_161915_object88172:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e960)", 2024-02-09T18:05:26.224 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:03:45.825580+0000", 2024-02-09T18:05:26.224 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 100.26047937, 2024-02-09T18:05:26.224 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0017285760000001, 2024-02-09T18:05:26.224 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:26.224 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:26.224 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:05:26.225 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26563", 2024-02-09T18:05:26.225 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:05:26.225 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 223054 2024-02-09T18:05:26.225 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.225 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:26.225 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.225 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:26.225 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.825580+0000", 2024-02-09T18:05:26.225 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:26.225 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.225 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.226 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:26.226 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.825580+0000", 2024-02-09T18:05:26.226 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:26.226 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.226 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.226 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:26.226 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.825581+0000", 2024-02-09T18:05:26.226 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1149999999999999e-06 2024-02-09T18:05:26.226 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.226 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.226 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:26.226 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.825585+0000", 2024-02-09T18:05:26.226 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.6789999999999996e-06 2024-02-09T18:05:26.226 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.226 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.227 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:26.227 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.825596+0000", 2024-02-09T18:05:26.227 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0849999999999999e-05 2024-02-09T18:05:26.227 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.227 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.227 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:26.227 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.825602+0000", 2024-02-09T18:05:26.227 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7039999999999999e-06 2024-02-09T18:05:26.227 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.227 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.227 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:26.227 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.825650+0000", 2024-02-09T18:05:26.227 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.7494999999999998e-05 2024-02-09T18:05:26.227 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.228 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.228 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T18:05:26.228 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.825659+0000", 2024-02-09T18:05:26.228 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.3840000000000003e-06 2024-02-09T18:05:26.228 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.228 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.228 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:26.228 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.811499+0000", 2024-02-09T18:05:26.228 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.98583990700000002 2024-02-09T18:05:26.228 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.228 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.228 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:26.228 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.811560+0000", 2024-02-09T18:05:26.228 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.1272000000000006e-05 2024-02-09T18:05:26.229 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.229 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.229 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 5,9", 2024-02-09T18:05:26.229 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.811672+0000", 2024-02-09T18:05:26.229 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000112148 2024-02-09T18:05:26.229 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.229 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.229 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:26.229 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.819642+0000", 2024-02-09T18:05:26.229 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0079701870000000001 2024-02-09T18:05:26.229 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.229 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.229 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:05:26.229 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.819654+0000", 2024-02-09T18:05:26.229 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1702000000000001e-05 2024-02-09T18:05:26.230 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.230 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.230 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:26.230 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.827251+0000", 2024-02-09T18:05:26.230 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0075965770000000002 2024-02-09T18:05:26.230 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.230 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.230 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:26.230 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.827284+0000", 2024-02-09T18:05:26.230 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.2984000000000001e-05 2024-02-09T18:05:26.230 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.230 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.230 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:26.230 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.827308+0000", 2024-02-09T18:05:26.230 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.4572000000000001e-05 2024-02-09T18:05:26.231 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:26.231 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:26.231 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:26.231 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.231 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.231 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26563.0:223055 9.6 9:68302ade:::benchmark_data_smithi179_161915_object88173:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e960)", 2024-02-09T18:05:26.231 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:03:45.825614+0000", 2024-02-09T18:05:26.231 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 100.260445476, 2024-02-09T18:05:26.231 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.001841888, 2024-02-09T18:05:26.231 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:26.231 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:26.231 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:05:26.231 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26563", 2024-02-09T18:05:26.231 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:05:26.232 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 223055 2024-02-09T18:05:26.232 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.232 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:26.232 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.232 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:26.232 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.825614+0000", 2024-02-09T18:05:26.232 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:26.232 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.232 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.232 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:26.232 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.825614+0000", 2024-02-09T18:05:26.232 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:26.232 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.233 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.233 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:26.233 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.825614+0000", 2024-02-09T18:05:26.233 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.4499999999999998e-07 2024-02-09T18:05:26.233 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.233 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.233 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:26.233 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.825616+0000", 2024-02-09T18:05:26.233 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.6729999999999999e-06 2024-02-09T18:05:26.233 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.233 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.233 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:26.233 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.825622+0000", 2024-02-09T18:05:26.233 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.9619999999999998e-06 2024-02-09T18:05:26.234 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.234 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.234 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:26.234 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.825625+0000", 2024-02-09T18:05:26.234 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9129999999999996e-06 2024-02-09T18:05:26.234 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.234 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.234 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:26.234 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.825685+0000", 2024-02-09T18:05:26.234 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.9163999999999998e-05 2024-02-09T18:05:26.234 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.234 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.234 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T18:05:26.234 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:45.825695+0000", 2024-02-09T18:05:26.235 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0427e-05 2024-02-09T18:05:26.235 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.235 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.235 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:26.235 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.815751+0000", 2024-02-09T18:05:26.235 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.99005614500000005 2024-02-09T18:05:26.235 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.235 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.235 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:26.235 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.815861+0000", 2024-02-09T18:05:26.235 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000109506 2024-02-09T18:05:26.235 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.235 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.235 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 5,9", 2024-02-09T18:05:26.235 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.815946+0000", 2024-02-09T18:05:26.236 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.5006999999999994e-05 2024-02-09T18:05:26.236 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.236 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.236 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:26.236 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.819683+0000", 2024-02-09T18:05:26.236 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00373763 2024-02-09T18:05:26.236 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.236 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.236 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:05:26.236 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.823438+0000", 2024-02-09T18:05:26.236 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.003754594 2024-02-09T18:05:26.236 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.236 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.237 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:26.237 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.827385+0000", 2024-02-09T18:05:26.237 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0039471860000000001 2024-02-09T18:05:26.237 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.237 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.237 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:26.237 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.827433+0000", 2024-02-09T18:05:26.237 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.7657000000000001e-05 2024-02-09T18:05:26.237 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.237 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.237 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:26.237 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.827455+0000", 2024-02-09T18:05:26.237 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2679000000000001e-05 2024-02-09T18:05:26.238 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:26.238 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:26.238 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:26.238 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.238 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.238 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26563.0:223209 9.e 9:7b9751ac:::benchmark_data_smithi179_161915_object88327:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e961)", 2024-02-09T18:05:26.238 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:03:46.805111+0000", 2024-02-09T18:05:26.238 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 99.280948488000007, 2024-02-09T18:05:26.238 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.005190757, 2024-02-09T18:05:26.238 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:26.238 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:26.238 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:05:26.238 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26563", 2024-02-09T18:05:26.238 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:05:26.238 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 223209 2024-02-09T18:05:26.239 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.239 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:26.239 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.239 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:26.239 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.805111+0000", 2024-02-09T18:05:26.239 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:26.239 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.239 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.239 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:26.239 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.805111+0000", 2024-02-09T18:05:26.239 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:26.240 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.240 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.240 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:26.240 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.805111+0000", 2024-02-09T18:05:26.240 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.1800000000000004e-07 2024-02-09T18:05:26.240 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.240 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.240 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:26.240 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.805116+0000", 2024-02-09T18:05:26.241 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.1690000000000002e-06 2024-02-09T18:05:26.241 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.241 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.241 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:26.241 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.805126+0000", 2024-02-09T18:05:26.241 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0203999999999999e-05 2024-02-09T18:05:26.241 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.241 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.241 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:26.241 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.805132+0000", 2024-02-09T18:05:26.241 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6849999999999998e-06 2024-02-09T18:05:26.241 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.242 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.242 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:26.242 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.849290+0000", 2024-02-09T18:05:26.242 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.044158240000000001 2024-02-09T18:05:26.242 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.243 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.243 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T18:05:26.243 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.849297+0000", 2024-02-09T18:05:26.243 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.7279999999999995e-06 2024-02-09T18:05:26.243 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.243 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.244 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:26.244 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:47.803854+0000", 2024-02-09T18:05:26.244 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.95455696700000003 2024-02-09T18:05:26.244 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.244 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.244 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:26.245 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:47.804005+0000", 2024-02-09T18:05:26.245 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000150607 2024-02-09T18:05:26.245 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.245 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.245 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 7,9", 2024-02-09T18:05:26.245 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:47.804157+0000", 2024-02-09T18:05:26.245 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000151965 2024-02-09T18:05:26.245 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.245 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.245 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:26.245 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:47.810132+0000", 2024-02-09T18:05:26.245 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0059754229999999997 2024-02-09T18:05:26.246 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.246 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.246 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:26.246 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:47.810193+0000", 2024-02-09T18:05:26.246 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.0328000000000002e-05 2024-02-09T18:05:26.246 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.246 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.247 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:05:26.247 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:47.810209+0000", 2024-02-09T18:05:26.247 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.6120000000000002e-05 2024-02-09T18:05:26.247 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.247 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.247 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:26.247 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:47.810259+0000", 2024-02-09T18:05:26.247 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.9809999999999999e-05 2024-02-09T18:05:26.247 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.247 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.247 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:26.248 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:47.810301+0000", 2024-02-09T18:05:26.248 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2592999999999997e-05 2024-02-09T18:05:26.248 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:26.248 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:26.248 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:26.248 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.248 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.248 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26563.0:223210 9.e 9:71d1f332:::benchmark_data_smithi179_161915_object88328:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e961)", 2024-02-09T18:05:26.248 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:03:46.808156+0000", 2024-02-09T18:05:26.248 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 99.277903527000007, 2024-02-09T18:05:26.248 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.010215431, 2024-02-09T18:05:26.248 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:26.248 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:26.248 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:05:26.248 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26563", 2024-02-09T18:05:26.249 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:05:26.249 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 223210 2024-02-09T18:05:26.249 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.249 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:26.249 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.249 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:26.249 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.808156+0000", 2024-02-09T18:05:26.249 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:26.249 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.249 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.249 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:26.249 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.808156+0000", 2024-02-09T18:05:26.249 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:26.250 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.250 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.250 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:26.250 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.808156+0000", 2024-02-09T18:05:26.250 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.9999999999999997e-07 2024-02-09T18:05:26.250 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.250 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.250 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:26.250 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.808160+0000", 2024-02-09T18:05:26.250 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.6789999999999998e-06 2024-02-09T18:05:26.250 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.250 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.251 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:26.251 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.808169+0000", 2024-02-09T18:05:26.251 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.5510000000000005e-06 2024-02-09T18:05:26.251 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.251 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.251 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:26.251 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.808176+0000", 2024-02-09T18:05:26.251 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.3729999999999998e-06 2024-02-09T18:05:26.251 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.251 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.251 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:26.251 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.849319+0000", 2024-02-09T18:05:26.252 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.041143122999999997 2024-02-09T18:05:26.252 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.252 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.252 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T18:05:26.252 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.849325+0000", 2024-02-09T18:05:26.252 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.3350000000000003e-06 2024-02-09T18:05:26.252 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.252 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.252 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:26.252 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:47.810341+0000", 2024-02-09T18:05:26.252 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.96101615900000004 2024-02-09T18:05:26.252 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.252 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.253 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:26.253 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:47.810476+0000", 2024-02-09T18:05:26.253 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000134618 2024-02-09T18:05:26.253 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.253 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.253 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 7,9", 2024-02-09T18:05:26.253 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:47.810614+0000", 2024-02-09T18:05:26.253 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000137866 2024-02-09T18:05:26.253 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.253 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.253 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:26.253 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:47.818265+0000", 2024-02-09T18:05:26.254 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0076507659999999998 2024-02-09T18:05:26.254 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.254 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.254 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:26.254 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:47.818298+0000", 2024-02-09T18:05:26.254 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.2833999999999998e-05 2024-02-09T18:05:26.254 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.254 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.254 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:05:26.254 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:47.818305+0000", 2024-02-09T18:05:26.254 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.7570000000000004e-06 2024-02-09T18:05:26.254 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.255 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.255 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:26.255 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:47.818338+0000", 2024-02-09T18:05:26.255 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.2811999999999997e-05 2024-02-09T18:05:26.255 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.255 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.255 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:26.255 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:47.818371+0000", 2024-02-09T18:05:26.255 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.2858000000000003e-05 2024-02-09T18:05:26.255 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:26.255 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:26.255 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:26.255 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.255 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.255 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26563.0:223255 9.e 9:7147753c:::benchmark_data_smithi179_161915_object88373:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e961)", 2024-02-09T18:05:26.256 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:03:46.869313+0000", 2024-02-09T18:05:26.256 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 99.216745932999999, 2024-02-09T18:05:26.256 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.99142572399999995, 2024-02-09T18:05:26.256 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:26.256 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:26.256 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:05:26.256 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26563", 2024-02-09T18:05:26.256 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:05:26.256 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 223255 2024-02-09T18:05:26.256 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.256 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:26.256 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.256 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:26.257 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.869313+0000", 2024-02-09T18:05:26.257 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:26.257 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.257 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.257 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:26.257 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.869313+0000", 2024-02-09T18:05:26.257 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:26.257 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.257 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.257 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:26.257 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.869314+0000", 2024-02-09T18:05:26.257 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.3099999999999997e-07 2024-02-09T18:05:26.257 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.258 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.258 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:26.258 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.869316+0000", 2024-02-09T18:05:26.258 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.5129999999999999e-06 2024-02-09T18:05:26.258 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.258 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.258 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:26.258 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.869322+0000", 2024-02-09T18:05:26.258 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7309999999999998e-06 2024-02-09T18:05:26.258 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.258 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.258 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:26.258 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.869326+0000", 2024-02-09T18:05:26.258 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.1450000000000001e-06 2024-02-09T18:05:26.259 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.259 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.259 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:26.259 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.869368+0000", 2024-02-09T18:05:26.259 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.1699999999999997e-05 2024-02-09T18:05:26.259 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.259 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.259 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T18:05:26.259 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.869376+0000", 2024-02-09T18:05:26.259 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.3089999999999996e-06 2024-02-09T18:05:26.259 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.260 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.260 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:26.260 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:47.836228+0000", 2024-02-09T18:05:26.260 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.96685190499999996 2024-02-09T18:05:26.260 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.260 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.260 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:26.260 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:47.836290+0000", 2024-02-09T18:05:26.260 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.1503999999999999e-05 2024-02-09T18:05:26.260 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.260 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.260 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 7,9", 2024-02-09T18:05:26.260 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:47.836367+0000", 2024-02-09T18:05:26.261 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.7584999999999997e-05 2024-02-09T18:05:26.261 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.261 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.261 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:05:26.261 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:47.841039+0000", 2024-02-09T18:05:26.261 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0046714369999999996 2024-02-09T18:05:26.261 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.261 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.261 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:26.261 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:47.849111+0000", 2024-02-09T18:05:26.261 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0080724539999999997 2024-02-09T18:05:26.261 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.261 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.262 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:26.262 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:47.860645+0000", 2024-02-09T18:05:26.262 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.011534301 2024-02-09T18:05:26.262 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.262 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.262 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:26.262 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:47.860702+0000", 2024-02-09T18:05:26.262 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6637e-05 2024-02-09T18:05:26.262 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.262 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.262 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:26.262 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:47.860739+0000", 2024-02-09T18:05:26.262 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.6872000000000002e-05 2024-02-09T18:05:26.262 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:26.262 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:26.263 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:26.263 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.263 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.263 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26563.0:223256 9.e 9:7da50d47:::benchmark_data_smithi179_161915_object88374:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e961)", 2024-02-09T18:05:26.263 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:03:46.869338+0000", 2024-02-09T18:05:26.263 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 99.216720840999997, 2024-02-09T18:05:26.263 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.99178391200000005, 2024-02-09T18:05:26.263 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:26.263 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:26.263 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:05:26.263 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26563", 2024-02-09T18:05:26.263 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:05:26.263 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 223256 2024-02-09T18:05:26.264 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.264 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:26.264 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.264 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:26.264 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.869338+0000", 2024-02-09T18:05:26.264 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:26.264 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.264 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.264 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:26.264 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.869338+0000", 2024-02-09T18:05:26.264 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:26.265 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.265 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.265 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:26.265 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.869339+0000", 2024-02-09T18:05:26.265 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.7099999999999997e-07 2024-02-09T18:05:26.265 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.265 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.265 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:26.265 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.869341+0000", 2024-02-09T18:05:26.265 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.1950000000000002e-06 2024-02-09T18:05:26.265 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.265 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.265 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:26.265 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.869348+0000", 2024-02-09T18:05:26.265 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.0160000000000003e-06 2024-02-09T18:05:26.266 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.266 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.266 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:26.266 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.869353+0000", 2024-02-09T18:05:26.266 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.9849999999999997e-06 2024-02-09T18:05:26.266 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.266 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.266 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:26.266 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.869412+0000", 2024-02-09T18:05:26.266 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.9181999999999997e-05 2024-02-09T18:05:26.266 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.266 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.266 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T18:05:26.266 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:46.869419+0000", 2024-02-09T18:05:26.266 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.3669999999999999e-06 2024-02-09T18:05:26.266 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.266 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.267 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:26.267 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:47.838598+0000", 2024-02-09T18:05:26.267 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.969178294 2024-02-09T18:05:26.267 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.267 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.267 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:26.267 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:47.838711+0000", 2024-02-09T18:05:26.267 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000113732 2024-02-09T18:05:26.267 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.267 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.267 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 7,9", 2024-02-09T18:05:26.267 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:47.838839+0000", 2024-02-09T18:05:26.267 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000127358 2024-02-09T18:05:26.267 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.267 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.268 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:05:26.268 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:47.849125+0000", 2024-02-09T18:05:26.268 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.010286336 2024-02-09T18:05:26.268 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.268 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.268 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:26.268 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:47.849312+0000", 2024-02-09T18:05:26.268 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00018704 2024-02-09T18:05:26.268 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.268 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.268 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:26.268 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:47.861023+0000", 2024-02-09T18:05:26.268 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.011711216 2024-02-09T18:05:26.268 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.269 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.269 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:26.269 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:47.861079+0000", 2024-02-09T18:05:26.269 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.5362000000000003e-05 2024-02-09T18:05:26.269 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:26.269 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:26.269 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:26.269 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:47.861122+0000", 2024-02-09T18:05:26.269 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.3458000000000003e-05 2024-02-09T18:05:26.269 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:26.269 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:26.269 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:26.269 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:26.269 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:26.269 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T18:05:26.438 INFO:teuthology.orchestra.run.smithi099.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-02-09T18:05:26.694 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:26 smithi099 ceph-mon[39420]: pgmap v2457: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:26.695 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:26 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/795471023' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-09T18:05:26.695 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:26 smithi099 ceph-mon[39420]: 1.1d scrub starts 2024-02-09T18:05:26.720 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-09T18:05:26.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:26 smithi179 ceph-mon[43975]: pgmap v2457: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:26.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:26 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/795471023' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-09T18:05:26.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:26 smithi179 ceph-mon[43975]: 1.1d scrub starts 2024-02-09T18:05:26.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:26 smithi162 ceph-mon[44304]: pgmap v2457: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:26.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:26 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/795471023' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-09T18:05:26.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:26 smithi162 ceph-mon[44304]: 1.1d scrub starts 2024-02-09T18:05:27.466 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 1 1 2024-02-09T18:05:27.516 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:05:27.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:27 smithi099 ceph-mon[39420]: 1.b deep-scrub starts 2024-02-09T18:05:27.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:27 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/795471023' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-02-09T18:05:27.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:27 smithi099 ceph-mon[39420]: osdmap e998: 12 total, 12 up, 12 in 2024-02-09T18:05:27.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:27 smithi099 ceph-mon[39420]: 1.1d scrub ok 2024-02-09T18:05:27.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:27 smithi099 ceph-mon[39420]: 1.b deep-scrub ok 2024-02-09T18:05:27.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:27 smithi179 ceph-mon[43975]: 1.b deep-scrub starts 2024-02-09T18:05:27.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:27 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/795471023' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-02-09T18:05:27.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:27 smithi179 ceph-mon[43975]: osdmap e998: 12 total, 12 up, 12 in 2024-02-09T18:05:27.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:27 smithi179 ceph-mon[43975]: 1.1d scrub ok 2024-02-09T18:05:27.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:27 smithi179 ceph-mon[43975]: 1.b deep-scrub ok 2024-02-09T18:05:27.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:27 smithi162 ceph-mon[44304]: 1.b deep-scrub starts 2024-02-09T18:05:27.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:27 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/795471023' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-02-09T18:05:27.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:27 smithi162 ceph-mon[44304]: osdmap e998: 12 total, 12 up, 12 in 2024-02-09T18:05:27.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:27 smithi162 ceph-mon[44304]: 1.1d scrub ok 2024-02-09T18:05:27.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:27 smithi162 ceph-mon[44304]: 1.b deep-scrub ok 2024-02-09T18:05:28.350 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:05:28.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:28 smithi099 ceph-mon[39420]: pgmap v2459: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:28.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:28 smithi099 ceph-mon[39420]: osdmap e999: 12 total, 12 up, 12 in 2024-02-09T18:05:28.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:28 smithi179 ceph-mon[43975]: pgmap v2459: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:28.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:28 smithi179 ceph-mon[43975]: osdmap e999: 12 total, 12 up, 12 in 2024-02-09T18:05:28.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:28 smithi162 ceph-mon[44304]: pgmap v2459: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:28.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:28 smithi162 ceph-mon[44304]: osdmap e999: 12 total, 12 up, 12 in 2024-02-09T18:05:30.715 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:30 smithi099 ceph-mon[39420]: pgmap v2461: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:30.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:30 smithi179 ceph-mon[43975]: pgmap v2461: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:30.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:30 smithi162 ceph-mon[44304]: pgmap v2461: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:31.083 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T18:05:31.083 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [], 2024-02-09T18:05:31.083 INFO:teuthology.orchestra.run.smithi099.stdout: "num_ops": 0 2024-02-09T18:05:31.083 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T18:05:31.456 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.1 to 1 (10000) 2024-02-09T18:05:31.725 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:31 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/629168158' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-09T18:05:31.751 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-09T18:05:31.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:31 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/629168158' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-09T18:05:31.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:31 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/629168158' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-09T18:05:32.481 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd primary-affinity 1 1 2024-02-09T18:05:32.489 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:05:32.727 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:32 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:05:32.727 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:32 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:05:32.727 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:32 smithi099 ceph-mon[39420]: pgmap v2462: 31 pgs: 31 active+clean; 577 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:32.727 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:32 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/629168158' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-09T18:05:32.727 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:32 smithi099 ceph-mon[39420]: osdmap e1000: 12 total, 12 up, 12 in 2024-02-09T18:05:32.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:32 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:05:32.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:32 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:05:32.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:32 smithi179 ceph-mon[43975]: pgmap v2462: 31 pgs: 31 active+clean; 577 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:32.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:32 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/629168158' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-09T18:05:32.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:32 smithi179 ceph-mon[43975]: osdmap e1000: 12 total, 12 up, 12 in 2024-02-09T18:05:32.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:32 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:05:32.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:32 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:05:32.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:32 smithi162 ceph-mon[44304]: pgmap v2462: 31 pgs: 31 active+clean; 577 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:32.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:32 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/629168158' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-09T18:05:32.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:32 smithi162 ceph-mon[44304]: osdmap e1000: 12 total, 12 up, 12 in 2024-02-09T18:05:33.254 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:05:34.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:34 smithi099 ceph-mon[39420]: pgmap v2464: 31 pgs: 31 active+clean; 577 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:34.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:34 smithi179 ceph-mon[43975]: pgmap v2464: 31 pgs: 31 active+clean; 577 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:34.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:34 smithi162 ceph-mon[44304]: pgmap v2464: 31 pgs: 31 active+clean; 577 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:35.081 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T18:05:35.082 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [], 2024-02-09T18:05:35.082 INFO:teuthology.orchestra.run.smithi099.stdout: "complaint_time": 30, 2024-02-09T18:05:35.082 INFO:teuthology.orchestra.run.smithi099.stdout: "num_blocked_ops": 0 2024-02-09T18:05:35.082 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T18:05:35.711 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:35 smithi099 ceph-mon[39420]: 1.10 deep-scrub starts 2024-02-09T18:05:35.712 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:35 smithi099 ceph-mon[39420]: 1.10 deep-scrub ok 2024-02-09T18:05:35.736 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-09T18:05:35.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:35 smithi179 ceph-mon[43975]: 1.10 deep-scrub starts 2024-02-09T18:05:35.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:35 smithi179 ceph-mon[43975]: 1.10 deep-scrub ok 2024-02-09T18:05:35.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:35 smithi162 ceph-mon[44304]: 1.10 deep-scrub starts 2024-02-09T18:05:35.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:35 smithi162 ceph-mon[44304]: 1.10 deep-scrub ok 2024-02-09T18:05:36.222 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:05:36.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:36 smithi099 ceph-mon[39420]: pgmap v2465: 31 pgs: 31 active+clean; 577 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:36.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:36 smithi179 ceph-mon[43975]: pgmap v2465: 31 pgs: 31 active+clean; 577 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:36.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:36 smithi162 ceph-mon[44304]: pgmap v2465: 31 pgs: 31 active+clean; 577 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:37.468 INFO:teuthology.orchestra.run.smithi099.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-02-09T18:05:37.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:37 smithi099 ceph-mon[39420]: 1.1a scrub starts 2024-02-09T18:05:37.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:37 smithi099 ceph-mon[39420]: 1.1a scrub ok 2024-02-09T18:05:37.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:37 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1618607076' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-09T18:05:37.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:37 smithi179 ceph-mon[43975]: 1.1a scrub starts 2024-02-09T18:05:37.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:37 smithi179 ceph-mon[43975]: 1.1a scrub ok 2024-02-09T18:05:37.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:37 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1618607076' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-09T18:05:37.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:37 smithi162 ceph-mon[44304]: 1.1a scrub starts 2024-02-09T18:05:37.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:37 smithi162 ceph-mon[44304]: 1.1a scrub ok 2024-02-09T18:05:37.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:37 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1618607076' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-09T18:05:38.568 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:38 smithi099 ceph-mon[39420]: 1.1b scrub starts 2024-02-09T18:05:38.568 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:38 smithi099 ceph-mon[39420]: 1.1b scrub ok 2024-02-09T18:05:38.568 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:38 smithi099 ceph-mon[39420]: pgmap v2466: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:38.568 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:38 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1618607076' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-02-09T18:05:38.568 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:38 smithi099 ceph-mon[39420]: osdmap e1001: 12 total, 12 up, 12 in 2024-02-09T18:05:38.595 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 2 1 2024-02-09T18:05:38.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:38 smithi179 ceph-mon[43975]: 1.1b scrub starts 2024-02-09T18:05:38.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:38 smithi179 ceph-mon[43975]: 1.1b scrub ok 2024-02-09T18:05:38.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:38 smithi179 ceph-mon[43975]: pgmap v2466: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:38.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:38 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1618607076' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-02-09T18:05:38.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:38 smithi179 ceph-mon[43975]: osdmap e1001: 12 total, 12 up, 12 in 2024-02-09T18:05:38.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:38 smithi162 ceph-mon[44304]: 1.1b scrub starts 2024-02-09T18:05:38.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:38 smithi162 ceph-mon[44304]: 1.1b scrub ok 2024-02-09T18:05:38.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:38 smithi162 ceph-mon[44304]: pgmap v2466: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:38.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:38 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1618607076' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-02-09T18:05:38.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:38 smithi162 ceph-mon[44304]: osdmap e1001: 12 total, 12 up, 12 in 2024-02-09T18:05:39.027 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T18:05:39.027 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 20, 2024-02-09T18:05:39.027 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 600, 2024-02-09T18:05:39.028 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [ 2024-02-09T18:05:39.028 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.028 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.26305.0:168514 8.e e831/785 8:7ea3477d:::benchmark_data_smithi179_148205_object45442:head v 831'10557, mlcod=831'10548)", 2024-02-09T18:05:39.028 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:57:49.796134+0000", 2024-02-09T18:05:39.028 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 469.22822458600001, 2024-02-09T18:05:39.028 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.045335991999999999, 2024-02-09T18:05:39.028 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:39.028 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:39.028 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:39.028 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.028 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:39.028 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796134+0000", 2024-02-09T18:05:39.028 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:39.028 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.029 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.029 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:39.029 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796134+0000", 2024-02-09T18:05:39.029 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:39.029 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.029 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.029 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:39.029 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796135+0000", 2024-02-09T18:05:39.029 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.34e-07 2024-02-09T18:05:39.029 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.029 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.029 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:39.029 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796141+0000", 2024-02-09T18:05:39.030 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.049e-06 2024-02-09T18:05:39.030 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.030 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.030 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:39.030 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796146+0000", 2024-02-09T18:05:39.030 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6910000000000002e-06 2024-02-09T18:05:39.030 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.030 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.030 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:39.030 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796151+0000", 2024-02-09T18:05:39.030 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2930000000000002e-06 2024-02-09T18:05:39.030 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.030 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.030 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:39.030 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.820112+0000", 2024-02-09T18:05:39.031 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.023961469999999999 2024-02-09T18:05:39.031 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.031 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.031 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:39.031 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.820133+0000", 2024-02-09T18:05:39.031 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.1214000000000001e-05 2024-02-09T18:05:39.031 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.031 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.031 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:39.031 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.841426+0000", 2024-02-09T18:05:39.031 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.021292789999999999 2024-02-09T18:05:39.031 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.032 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.032 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:39.032 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.841470+0000", 2024-02-09T18:05:39.032 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.4251e-05 2024-02-09T18:05:39.032 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:39.032 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:39.032 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:39.032 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.032 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.032 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.26305.0:168516 8.e e831/785 8:7a9c00b8:::benchmark_data_smithi179_148205_object45444:head v 831'10558, mlcod=831'10548)", 2024-02-09T18:05:39.032 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:57:49.796156+0000", 2024-02-09T18:05:39.032 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 469.22820332700002, 2024-02-09T18:05:39.032 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.045346616999999999, 2024-02-09T18:05:39.032 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:39.033 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:39.033 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:39.033 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.033 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:39.033 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796156+0000", 2024-02-09T18:05:39.033 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:39.033 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.033 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.033 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:39.033 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796156+0000", 2024-02-09T18:05:39.033 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:39.033 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.033 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.033 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:39.034 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796156+0000", 2024-02-09T18:05:39.034 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0599999999999999e-07 2024-02-09T18:05:39.034 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.034 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.034 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:39.034 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796158+0000", 2024-02-09T18:05:39.034 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.593e-06 2024-02-09T18:05:39.034 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.034 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.034 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:39.034 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796164+0000", 2024-02-09T18:05:39.034 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.037e-06 2024-02-09T18:05:39.034 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.034 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.034 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:39.035 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796168+0000", 2024-02-09T18:05:39.035 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.6880000000000001e-06 2024-02-09T18:05:39.035 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.035 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.035 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:39.035 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.823414+0000", 2024-02-09T18:05:39.035 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.027245933 2024-02-09T18:05:39.035 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.035 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.035 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:39.035 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.823428+0000", 2024-02-09T18:05:39.035 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4035e-05 2024-02-09T18:05:39.035 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.035 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.036 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:39.036 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.841475+0000", 2024-02-09T18:05:39.036 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.018046627999999999 2024-02-09T18:05:39.036 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.036 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.036 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:39.036 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.841502+0000", 2024-02-09T18:05:39.036 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.7497000000000001e-05 2024-02-09T18:05:39.036 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:39.036 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:39.036 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:39.036 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.036 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.036 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.26305.0:168535 8.e e831/785 8:738351cd:::benchmark_data_smithi179_148205_object45463:head v 831'10559, mlcod=831'10548)", 2024-02-09T18:05:39.036 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:57:49.796173+0000", 2024-02-09T18:05:39.037 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 469.22818626999998, 2024-02-09T18:05:39.037 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.045364935000000002, 2024-02-09T18:05:39.037 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:39.037 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:39.037 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:39.037 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.037 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:39.037 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796173+0000", 2024-02-09T18:05:39.037 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:39.037 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.037 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.037 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:39.037 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796173+0000", 2024-02-09T18:05:39.037 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:39.038 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.038 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.038 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:39.038 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796173+0000", 2024-02-09T18:05:39.038 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.31e-07 2024-02-09T18:05:39.038 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.038 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.038 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:39.038 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796176+0000", 2024-02-09T18:05:39.038 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.0189999999999998e-06 2024-02-09T18:05:39.038 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.038 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.038 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:39.038 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796181+0000", 2024-02-09T18:05:39.038 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6280000000000002e-06 2024-02-09T18:05:39.039 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.039 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.039 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:39.039 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796185+0000", 2024-02-09T18:05:39.039 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.6899999999999998e-06 2024-02-09T18:05:39.039 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.039 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.039 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:39.039 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.826413+0000", 2024-02-09T18:05:39.039 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.030228034000000001 2024-02-09T18:05:39.039 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.039 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.039 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:39.039 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.826427+0000", 2024-02-09T18:05:39.040 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4259e-05 2024-02-09T18:05:39.040 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.040 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.040 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:39.040 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.841509+0000", 2024-02-09T18:05:39.040 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.015081762 2024-02-09T18:05:39.040 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.040 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.040 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:39.040 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.841538+0000", 2024-02-09T18:05:39.040 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.8411999999999998e-05 2024-02-09T18:05:39.040 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:39.040 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:39.040 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:39.041 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.041 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.041 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.26305.0:168570 8.e e831/785 8:7ca3aecf:::benchmark_data_smithi179_148205_object45498:head v 831'10560, mlcod=831'10548)", 2024-02-09T18:05:39.041 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:57:49.796190+0000", 2024-02-09T18:05:39.041 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 469.22816922200002, 2024-02-09T18:05:39.041 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.045370932000000003, 2024-02-09T18:05:39.041 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:39.041 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:39.041 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:39.041 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.041 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:39.041 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796190+0000", 2024-02-09T18:05:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796190+0000", 2024-02-09T18:05:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796190+0000", 2024-02-09T18:05:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4999999999999999e-07 2024-02-09T18:05:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.042 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:39.043 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796195+0000", 2024-02-09T18:05:39.043 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.203e-06 2024-02-09T18:05:39.043 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.043 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.043 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:39.043 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796201+0000", 2024-02-09T18:05:39.044 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6919999999999999e-06 2024-02-09T18:05:39.044 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.044 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.044 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:39.044 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.796204+0000", 2024-02-09T18:05:39.045 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.7400000000000002e-06 2024-02-09T18:05:39.045 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.045 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.045 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:39.045 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.829598+0000", 2024-02-09T18:05:39.046 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.033393316999999999 2024-02-09T18:05:39.046 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.046 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.046 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:39.046 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.829619+0000", 2024-02-09T18:05:39.046 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.1075999999999999e-05 2024-02-09T18:05:39.046 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.046 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.046 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:39.047 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.841542+0000", 2024-02-09T18:05:39.047 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.011923023 2024-02-09T18:05:39.047 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.047 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.047 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:39.047 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:57:49.841561+0000", 2024-02-09T18:05:39.047 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.8731000000000001e-05 2024-02-09T18:05:39.047 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:39.047 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:39.047 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:39.047 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.047 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.047 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(osd.1.0:209346 8.1 e844/785 8:80b6b8ae:::benchmark_data_smithi179_148205_object82787:head v 844'12275, mlcod=844'12260)", 2024-02-09T18:05:39.048 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:58:26.848196+0000", 2024-02-09T18:05:39.048 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 432.17616299500003, 2024-02-09T18:05:39.048 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.041168392999999998, 2024-02-09T18:05:39.048 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:39.048 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:39.049 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:39.049 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.049 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:39.049 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.848196+0000", 2024-02-09T18:05:39.049 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:39.049 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.049 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.049 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:39.049 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.848196+0000", 2024-02-09T18:05:39.049 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:39.049 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.049 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.049 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:39.049 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.848197+0000", 2024-02-09T18:05:39.050 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.031e-06 2024-02-09T18:05:39.050 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.050 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.050 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:39.050 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.848205+0000", 2024-02-09T18:05:39.050 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.3119999999999994e-06 2024-02-09T18:05:39.050 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.050 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.050 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:39.050 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.848216+0000", 2024-02-09T18:05:39.050 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0903e-05 2024-02-09T18:05:39.050 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.050 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.050 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:39.051 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.848224+0000", 2024-02-09T18:05:39.051 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.7069999999999992e-06 2024-02-09T18:05:39.051 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.051 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.051 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:39.051 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.851875+0000", 2024-02-09T18:05:39.051 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0036507929999999998 2024-02-09T18:05:39.051 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.051 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.051 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:39.051 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.851900+0000", 2024-02-09T18:05:39.051 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.5401e-05 2024-02-09T18:05:39.051 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.051 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.051 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:39.052 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.889318+0000", 2024-02-09T18:05:39.052 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.037417589000000001 2024-02-09T18:05:39.052 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.052 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.052 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:39.052 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.889364+0000", 2024-02-09T18:05:39.052 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.6656999999999997e-05 2024-02-09T18:05:39.052 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:39.052 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:39.052 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:39.052 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.053 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.053 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(osd.1.0:209356 8.1 e844/785 8:80c2393c:::benchmark_data_smithi179_148205_object79337:head v 844'12285, mlcod=844'12260)", 2024-02-09T18:05:39.053 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:58:26.893137+0000", 2024-02-09T18:05:39.053 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 432.13122190199999, 2024-02-09T18:05:39.053 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.062734074000000001, 2024-02-09T18:05:39.053 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:39.054 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:39.054 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:39.054 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.054 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:39.054 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.893137+0000", 2024-02-09T18:05:39.054 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:39.055 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.055 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.055 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:39.055 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.893137+0000", 2024-02-09T18:05:39.055 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:39.055 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.055 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.056 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:39.056 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.893139+0000", 2024-02-09T18:05:39.056 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2239999999999998e-06 2024-02-09T18:05:39.056 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.056 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.056 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:39.056 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.893151+0000", 2024-02-09T18:05:39.056 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1552999999999999e-05 2024-02-09T18:05:39.056 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.056 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.056 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:39.057 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.893171+0000", 2024-02-09T18:05:39.057 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0230000000000001e-05 2024-02-09T18:05:39.057 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.057 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.057 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:39.057 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.893182+0000", 2024-02-09T18:05:39.058 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1276e-05 2024-02-09T18:05:39.058 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.058 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.058 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:39.058 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.903906+0000", 2024-02-09T18:05:39.058 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.010723514 2024-02-09T18:05:39.058 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.058 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.058 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:39.058 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.903929+0000", 2024-02-09T18:05:39.058 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.3560000000000001e-05 2024-02-09T18:05:39.059 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.059 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.059 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:39.059 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.955838+0000", 2024-02-09T18:05:39.059 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.051908724000000003 2024-02-09T18:05:39.059 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.059 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.059 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:39.059 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.955871+0000", 2024-02-09T18:05:39.060 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.2993000000000001e-05 2024-02-09T18:05:39.060 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:39.060 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:39.060 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:39.060 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.060 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.060 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(osd.1.0:209357 8.1 e844/785 8:80c3ca64:::benchmark_data_smithi179_148205_object73485:head v 844'12286, mlcod=844'12260)", 2024-02-09T18:05:39.060 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:58:26.898575+0000", 2024-02-09T18:05:39.060 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 432.125784318, 2024-02-09T18:05:39.060 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.057326736000000003, 2024-02-09T18:05:39.060 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:39.060 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:39.061 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:39.061 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.061 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:39.061 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.898575+0000", 2024-02-09T18:05:39.061 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:39.061 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.061 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.061 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:39.061 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.898575+0000", 2024-02-09T18:05:39.061 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:39.061 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.061 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.061 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:39.061 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.898576+0000", 2024-02-09T18:05:39.062 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.127e-06 2024-02-09T18:05:39.062 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.062 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.062 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:39.062 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.898584+0000", 2024-02-09T18:05:39.062 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.7620000000000001e-06 2024-02-09T18:05:39.062 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.062 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.062 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:39.062 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.898596+0000", 2024-02-09T18:05:39.062 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1214e-05 2024-02-09T18:05:39.062 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.062 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.062 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:39.063 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.898603+0000", 2024-02-09T18:05:39.063 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.8959999999999997e-06 2024-02-09T18:05:39.063 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.063 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.063 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:39.063 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.908768+0000", 2024-02-09T18:05:39.063 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.010165406 2024-02-09T18:05:39.063 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.063 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.063 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:39.063 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.908793+0000", 2024-02-09T18:05:39.063 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.4651000000000002e-05 2024-02-09T18:05:39.064 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.064 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.064 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:39.064 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.955874+0000", 2024-02-09T18:05:39.064 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.047081537999999999 2024-02-09T18:05:39.064 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.064 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.064 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:39.064 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.955901+0000", 2024-02-09T18:05:39.064 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.7141999999999999e-05 2024-02-09T18:05:39.064 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:39.064 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:39.064 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:39.065 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.065 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.065 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(osd.1.0:209358 8.1 e844/785 8:80c51ccd:::benchmark_data_smithi179_148205_object109835:head v 844'12287, mlcod=844'12260)", 2024-02-09T18:05:39.065 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:58:26.902989+0000", 2024-02-09T18:05:39.065 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 432.12136969099998, 2024-02-09T18:05:39.065 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.052941015000000001, 2024-02-09T18:05:39.065 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:39.065 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:39.065 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:39.065 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.065 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:39.065 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.902989+0000", 2024-02-09T18:05:39.065 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:39.065 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.066 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.066 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:39.066 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.902989+0000", 2024-02-09T18:05:39.066 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:39.066 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.066 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.066 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:39.066 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.902991+0000", 2024-02-09T18:05:39.066 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.606e-06 2024-02-09T18:05:39.066 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.066 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.066 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:39.067 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.903001+0000", 2024-02-09T18:05:39.067 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0539e-05 2024-02-09T18:05:39.067 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.067 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.067 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:39.067 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.903016+0000", 2024-02-09T18:05:39.067 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4980000000000001e-05 2024-02-09T18:05:39.067 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.067 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.067 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:39.067 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.903026+0000", 2024-02-09T18:05:39.067 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.55e-06 2024-02-09T18:05:39.067 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.067 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.068 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:39.068 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.912525+0000", 2024-02-09T18:05:39.068 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0094991769999999993 2024-02-09T18:05:39.068 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.068 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.068 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:39.068 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.912546+0000", 2024-02-09T18:05:39.068 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0781e-05 2024-02-09T18:05:39.068 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.068 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.068 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:39.068 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.955905+0000", 2024-02-09T18:05:39.068 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.043359645000000002 2024-02-09T18:05:39.068 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.068 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.069 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:39.069 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.955930+0000", 2024-02-09T18:05:39.069 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.4737e-05 2024-02-09T18:05:39.069 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:39.069 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:39.069 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:39.069 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.069 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.069 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.26305.0:196480 8.1 e844/785 8:852f53cb:::benchmark_data_smithi179_148205_object73408:head v 844'12288, mlcod=844'12281)", 2024-02-09T18:05:39.069 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:58:26.910901+0000", 2024-02-09T18:05:39.069 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 432.11345781199998, 2024-02-09T18:05:39.069 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.045055407999999998, 2024-02-09T18:05:39.069 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:39.069 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:39.069 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:39.069 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.070 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:39.070 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.910901+0000", 2024-02-09T18:05:39.070 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:39.070 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.070 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.070 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:39.070 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.910901+0000", 2024-02-09T18:05:39.070 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:39.070 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.070 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.070 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:39.070 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.910902+0000", 2024-02-09T18:05:39.070 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.5099999999999999e-07 2024-02-09T18:05:39.070 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.070 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.071 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:39.071 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.910908+0000", 2024-02-09T18:05:39.071 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.8880000000000002e-06 2024-02-09T18:05:39.071 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.071 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.071 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:39.071 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.910920+0000", 2024-02-09T18:05:39.071 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2469e-05 2024-02-09T18:05:39.071 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.071 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.071 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:39.071 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.910927+0000", 2024-02-09T18:05:39.071 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.6490000000000002e-06 2024-02-09T18:05:39.071 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.072 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.072 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:39.072 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.916446+0000", 2024-02-09T18:05:39.072 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.005519637 2024-02-09T18:05:39.072 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.072 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.072 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:39.072 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.916460+0000", 2024-02-09T18:05:39.072 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3365e-05 2024-02-09T18:05:39.072 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.072 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.072 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:39.072 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.955932+0000", 2024-02-09T18:05:39.072 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.039472410999999999 2024-02-09T18:05:39.073 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.073 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.073 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:39.073 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:26.955956+0000", 2024-02-09T18:05:39.073 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.4238000000000001e-05 2024-02-09T18:05:39.073 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:39.073 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:39.073 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:39.073 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.073 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.073 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(osd.1.0:210078 8.1 e845/785 8:8353a4c3:::benchmark_data_smithi179_148205_object95143:head v 845'12885, mlcod=845'12875)", 2024-02-09T18:05:39.073 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:58:30.647538+0000", 2024-02-09T18:05:39.073 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 428.37682113199998, 2024-02-09T18:05:39.073 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.040794214000000002, 2024-02-09T18:05:39.074 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:39.074 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:39.074 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:39.074 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.074 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:39.074 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:30.647538+0000", 2024-02-09T18:05:39.074 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:39.074 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.075 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.075 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:39.075 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:30.647538+0000", 2024-02-09T18:05:39.075 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:39.075 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.075 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.075 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:39.075 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:30.647538+0000", 2024-02-09T18:05:39.075 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.2900000000000003e-07 2024-02-09T18:05:39.075 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.075 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.075 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:39.075 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:30.647544+0000", 2024-02-09T18:05:39.075 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1399999999999999e-06 2024-02-09T18:05:39.076 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.076 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.076 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:39.076 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:30.647553+0000", 2024-02-09T18:05:39.076 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.7100000000000002e-06 2024-02-09T18:05:39.076 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.076 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.076 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:39.076 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:30.647562+0000", 2024-02-09T18:05:39.076 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.9239999999999996e-06 2024-02-09T18:05:39.076 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.076 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.076 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:39.077 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:30.647609+0000", 2024-02-09T18:05:39.077 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.7144999999999997e-05 2024-02-09T18:05:39.077 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.077 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.077 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:39.077 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:30.647630+0000", 2024-02-09T18:05:39.078 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0247000000000001e-05 2024-02-09T18:05:39.078 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.078 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.078 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:39.078 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:30.688292+0000", 2024-02-09T18:05:39.078 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.040662500999999997 2024-02-09T18:05:39.078 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.078 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.078 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:39.078 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:58:30.688332+0000", 2024-02-09T18:05:39.078 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9817999999999997e-05 2024-02-09T18:05:39.078 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:39.078 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:39.078 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:39.079 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.079 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.079 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.26563.0:223602 9.7 e962/943 9:ea3ce47f:::benchmark_data_smithi179_161915_object88720:head v 962'13927, mlcod=962'13924)", 2024-02-09T18:05:39.079 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:03:48.203500+0000", 2024-02-09T18:05:39.079 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 110.82085868, 2024-02-09T18:05:39.079 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.021987789000000001, 2024-02-09T18:05:39.079 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:39.079 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:39.080 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:39.080 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.080 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:39.080 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:48.203500+0000", 2024-02-09T18:05:39.080 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:39.080 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.081 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.081 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:39.081 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:48.203500+0000", 2024-02-09T18:05:39.081 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:39.081 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.082 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.082 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:39.082 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:48.203502+0000", 2024-02-09T18:05:39.082 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.575e-06 2024-02-09T18:05:39.082 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.082 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.082 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:39.082 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:48.203512+0000", 2024-02-09T18:05:39.082 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0515e-05 2024-02-09T18:05:39.083 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.083 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.083 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:39.083 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:48.203533+0000", 2024-02-09T18:05:39.083 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0217999999999999e-05 2024-02-09T18:05:39.083 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.083 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.083 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:39.084 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:48.203546+0000", 2024-02-09T18:05:39.084 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3417999999999999e-05 2024-02-09T18:05:39.084 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.084 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.084 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:39.084 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:48.206407+0000", 2024-02-09T18:05:39.084 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.002861429 2024-02-09T18:05:39.084 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.084 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.085 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:39.085 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:48.206429+0000", 2024-02-09T18:05:39.085 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2101000000000001e-05 2024-02-09T18:05:39.085 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.085 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.085 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:39.085 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:48.225422+0000", 2024-02-09T18:05:39.085 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.018992810999999998 2024-02-09T18:05:39.085 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.085 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.085 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:39.085 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:48.225488+0000", 2024-02-09T18:05:39.085 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.5722000000000005e-05 2024-02-09T18:05:39.085 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:39.086 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:39.086 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:39.086 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.086 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.086 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.26563.0:261631 9.7 e973/970 9:e4a3e80c:::benchmark_data_smithi179_161915_object126749:head v 973'16308, mlcod=973'16306)", 2024-02-09T18:05:39.086 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:04:32.715422+0000", 2024-02-09T18:05:39.086 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 66.308936837000005, 2024-02-09T18:05:39.086 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.031019008000000001, 2024-02-09T18:05:39.086 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:39.086 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:39.086 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:39.086 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.086 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:39.086 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:32.715422+0000", 2024-02-09T18:05:39.086 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:39.087 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.087 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.087 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:39.087 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:32.715422+0000", 2024-02-09T18:05:39.087 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:39.087 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.087 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.087 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:39.087 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:32.715424+0000", 2024-02-09T18:05:39.087 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.734e-06 2024-02-09T18:05:39.087 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.087 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.087 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:39.087 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:32.715434+0000", 2024-02-09T18:05:39.087 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.005e-05 2024-02-09T18:05:39.088 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.088 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.088 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:39.088 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:32.715453+0000", 2024-02-09T18:05:39.088 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.8995e-05 2024-02-09T18:05:39.088 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.088 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.088 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:39.088 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:32.715463+0000", 2024-02-09T18:05:39.088 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.8910000000000007e-06 2024-02-09T18:05:39.088 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.088 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.089 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:39.089 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:32.729055+0000", 2024-02-09T18:05:39.089 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.013592408 2024-02-09T18:05:39.089 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.089 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.089 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:39.089 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:32.729082+0000", 2024-02-09T18:05:39.089 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.6483e-05 2024-02-09T18:05:39.089 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.089 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.089 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:39.089 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:32.746400+0000", 2024-02-09T18:05:39.089 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.017317895 2024-02-09T18:05:39.089 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.090 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.090 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:39.090 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:32.746441+0000", 2024-02-09T18:05:39.090 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.1551999999999998e-05 2024-02-09T18:05:39.090 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:39.090 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:39.090 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:39.090 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.090 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.090 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.26563.0:261632 9.7 e973/970 9:ece3f67d:::benchmark_data_smithi179_161915_object126750:head v 973'16309, mlcod=973'16306)", 2024-02-09T18:05:39.090 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:04:32.715488+0000", 2024-02-09T18:05:39.090 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 66.308870498999994, 2024-02-09T18:05:39.090 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.035558346999999997, 2024-02-09T18:05:39.090 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:39.090 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:39.090 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:39.091 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.091 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:39.091 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:32.715488+0000", 2024-02-09T18:05:39.091 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:39.091 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.091 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.091 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:39.091 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:32.715488+0000", 2024-02-09T18:05:39.091 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:39.091 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.091 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.092 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:39.092 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:32.715489+0000", 2024-02-09T18:05:39.092 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9999999999999998e-07 2024-02-09T18:05:39.092 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.092 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.092 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:39.092 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:32.715493+0000", 2024-02-09T18:05:39.092 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.4989999999999996e-06 2024-02-09T18:05:39.092 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.092 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.092 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:39.092 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:32.715500+0000", 2024-02-09T18:05:39.092 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.7719999999999997e-06 2024-02-09T18:05:39.092 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.092 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.093 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:39.093 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:32.715505+0000", 2024-02-09T18:05:39.093 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.8620000000000002e-06 2024-02-09T18:05:39.093 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.093 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.093 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:39.093 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:32.733547+0000", 2024-02-09T18:05:39.093 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.018042319000000001 2024-02-09T18:05:39.093 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.093 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.093 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:39.093 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:32.733568+0000", 2024-02-09T18:05:39.094 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0715000000000001e-05 2024-02-09T18:05:39.094 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.094 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.094 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:39.094 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:32.750993+0000", 2024-02-09T18:05:39.094 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.017424832000000001 2024-02-09T18:05:39.094 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.094 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.094 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:39.094 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:32.751047+0000", 2024-02-09T18:05:39.094 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3947999999999999e-05 2024-02-09T18:05:39.094 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:39.094 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:39.094 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:39.094 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.095 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.095 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.26563.0:261635 9.7 e973/970 9:ea3152f6:::benchmark_data_smithi179_161915_object126753:head v 973'16310, mlcod=973'16306)", 2024-02-09T18:05:39.095 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:04:32.721495+0000", 2024-02-09T18:05:39.095 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 66.302864013999994, 2024-02-09T18:05:39.095 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.029577928, 2024-02-09T18:05:39.095 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:39.095 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:39.095 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:39.095 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.095 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:39.095 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:32.721495+0000", 2024-02-09T18:05:39.095 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:39.095 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.095 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.096 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:39.096 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:32.721495+0000", 2024-02-09T18:05:39.096 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:39.096 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.096 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.096 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:39.096 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:32.721496+0000", 2024-02-09T18:05:39.096 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.341e-06 2024-02-09T18:05:39.096 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.096 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.096 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:39.096 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:32.721504+0000", 2024-02-09T18:05:39.096 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.4590000000000001e-06 2024-02-09T18:05:39.096 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.097 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.097 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:39.097 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:32.721519+0000", 2024-02-09T18:05:39.097 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5444999999999999e-05 2024-02-09T18:05:39.097 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.097 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.097 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:39.097 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:32.721529+0000", 2024-02-09T18:05:39.097 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.7440000000000002e-06 2024-02-09T18:05:39.097 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.097 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.097 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:39.097 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:32.737668+0000", 2024-02-09T18:05:39.097 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.016139263000000001 2024-02-09T18:05:39.097 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.098 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.098 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:39.098 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:32.737689+0000", 2024-02-09T18:05:39.098 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0452999999999999e-05 2024-02-09T18:05:39.098 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.098 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.098 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:39.098 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:32.751050+0000", 2024-02-09T18:05:39.098 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.013361148999999999 2024-02-09T18:05:39.098 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.098 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.098 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:39.098 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:32.751073+0000", 2024-02-09T18:05:39.098 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.3074e-05 2024-02-09T18:05:39.098 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:39.099 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:39.099 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:39.099 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.099 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.099 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.26563.0:261639 9.7 e973/970 9:e5700736:::benchmark_data_smithi179_161915_object126757:head v 973'16311, mlcod=973'16306)", 2024-02-09T18:05:39.099 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:04:32.725409+0000", 2024-02-09T18:05:39.099 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 66.298950089000002, 2024-02-09T18:05:39.099 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.025686482, 2024-02-09T18:05:39.099 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:39.099 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:39.099 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:39.099 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.099 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:39.099 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:32.725409+0000", 2024-02-09T18:05:39.100 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:39.100 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.100 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.100 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:39.100 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:32.725409+0000", 2024-02-09T18:05:39.100 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:39.100 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.100 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.100 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:39.100 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:32.725411+0000", 2024-02-09T18:05:39.100 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.731e-06 2024-02-09T18:05:39.100 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.100 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.100 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:39.101 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:32.725422+0000", 2024-02-09T18:05:39.101 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1657999999999999e-05 2024-02-09T18:05:39.101 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.101 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.101 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:39.101 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:32.725439+0000", 2024-02-09T18:05:39.101 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.7056999999999999e-05 2024-02-09T18:05:39.101 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.101 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.101 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:39.101 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:32.725449+0000", 2024-02-09T18:05:39.101 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.5300000000000002e-06 2024-02-09T18:05:39.102 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.102 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.102 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:39.102 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:32.742089+0000", 2024-02-09T18:05:39.102 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.016639943000000001 2024-02-09T18:05:39.102 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.102 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.102 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:39.102 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:32.742110+0000", 2024-02-09T18:05:39.102 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.1382999999999998e-05 2024-02-09T18:05:39.102 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.102 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.102 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:39.102 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:32.751074+0000", 2024-02-09T18:05:39.103 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0089643299999999995 2024-02-09T18:05:39.103 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.103 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.103 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:39.103 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:32.751095+0000", 2024-02-09T18:05:39.103 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.0849999999999999e-05 2024-02-09T18:05:39.103 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:39.103 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:39.103 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:39.103 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.103 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.103 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.26563.0:264129 9.7 e974/970 9:e6fb8f68:::benchmark_data_smithi179_161915_object129247:head v 974'16464, mlcod=974'16462)", 2024-02-09T18:05:39.104 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:04:35.893350+0000", 2024-02-09T18:05:39.104 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 63.131008629, 2024-02-09T18:05:39.104 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.024416139999999999, 2024-02-09T18:05:39.104 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:39.104 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:39.104 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:39.104 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.104 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:39.104 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:35.893350+0000", 2024-02-09T18:05:39.104 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:39.104 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.104 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.105 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:39.105 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:35.893350+0000", 2024-02-09T18:05:39.105 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:39.105 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.105 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.105 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:39.105 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:35.893352+0000", 2024-02-09T18:05:39.105 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.28e-06 2024-02-09T18:05:39.105 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.105 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.105 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:39.105 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:35.893360+0000", 2024-02-09T18:05:39.105 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.5059999999999995e-06 2024-02-09T18:05:39.105 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.106 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.106 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:39.106 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:35.893375+0000", 2024-02-09T18:05:39.106 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5048999999999999e-05 2024-02-09T18:05:39.106 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.106 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.106 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:39.106 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:35.893383+0000", 2024-02-09T18:05:39.106 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.8050000000000007e-06 2024-02-09T18:05:39.106 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.106 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.106 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:39.106 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:35.893462+0000", 2024-02-09T18:05:39.106 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.9363000000000006e-05 2024-02-09T18:05:39.106 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.107 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.107 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:39.107 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:35.893488+0000", 2024-02-09T18:05:39.107 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.6001999999999999e-05 2024-02-09T18:05:39.107 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.107 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.107 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:39.107 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:35.917733+0000", 2024-02-09T18:05:39.107 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.024244735 2024-02-09T18:05:39.107 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.107 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.107 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:39.107 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:35.917766+0000", 2024-02-09T18:05:39.107 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.3399999999999999e-05 2024-02-09T18:05:39.108 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:39.108 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:39.108 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:39.108 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.108 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.108 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.26563.0:265973 9.7 e975/970 9:edb578f7:::benchmark_data_smithi179_161915_object131091:head v 975'16584, mlcod=975'16582)", 2024-02-09T18:05:39.108 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:04:38.173431+0000", 2024-02-09T18:05:39.108 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 60.85092796, 2024-02-09T18:05:39.108 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.021675327000000001, 2024-02-09T18:05:39.108 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:39.108 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:39.109 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:39.109 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.109 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:39.109 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:38.173431+0000", 2024-02-09T18:05:39.109 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:39.109 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.109 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.109 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:39.109 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:38.173431+0000", 2024-02-09T18:05:39.109 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:39.109 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.109 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.109 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:39.109 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:38.173432+0000", 2024-02-09T18:05:39.109 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4330000000000001e-06 2024-02-09T18:05:39.110 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.110 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.110 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:39.110 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:38.173444+0000", 2024-02-09T18:05:39.110 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1204e-05 2024-02-09T18:05:39.110 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.110 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.110 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:39.110 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:38.173463+0000", 2024-02-09T18:05:39.110 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.9559999999999999e-05 2024-02-09T18:05:39.111 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.111 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.111 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:39.111 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:38.173474+0000", 2024-02-09T18:05:39.111 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0645999999999999e-05 2024-02-09T18:05:39.111 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.111 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.111 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:39.112 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:38.174429+0000", 2024-02-09T18:05:39.112 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00095473900000000002 2024-02-09T18:05:39.112 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.112 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.112 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:39.112 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:38.174444+0000", 2024-02-09T18:05:39.113 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.573e-05 2024-02-09T18:05:39.113 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.113 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.113 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:39.113 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:38.195062+0000", 2024-02-09T18:05:39.113 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.020617927000000001 2024-02-09T18:05:39.113 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.113 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.113 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:39.113 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:38.195106+0000", 2024-02-09T18:05:39.113 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.4088000000000002e-05 2024-02-09T18:05:39.113 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:39.113 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:39.113 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:39.114 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.114 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.114 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_repop(client.26563.0:269077 9.7 e976/970 9:e51c2645:::benchmark_data_smithi179_161915_object134195:head v 976'16750, mlcod=976'16747)", 2024-02-09T18:05:39.114 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:04:41.554456+0000", 2024-02-09T18:05:39.114 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 57.469902689999998, 2024-02-09T18:05:39.114 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.023639095999999998, 2024-02-09T18:05:39.114 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:39.115 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:39.115 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:39.115 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.115 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:39.115 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:41.554456+0000", 2024-02-09T18:05:39.115 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:39.116 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.116 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.116 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:39.116 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:41.554456+0000", 2024-02-09T18:05:39.116 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:39.117 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.117 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.117 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:39.117 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:41.554457+0000", 2024-02-09T18:05:39.117 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.9999999999999996e-07 2024-02-09T18:05:39.117 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.117 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.118 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:39.118 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:41.554465+0000", 2024-02-09T18:05:39.118 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.8639999999999996e-06 2024-02-09T18:05:39.118 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.118 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.118 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:39.118 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:41.554477+0000", 2024-02-09T18:05:39.119 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1668000000000001e-05 2024-02-09T18:05:39.119 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.119 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.119 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:39.119 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:41.554485+0000", 2024-02-09T18:05:39.119 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.3159999999999997e-06 2024-02-09T18:05:39.119 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.120 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.120 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:39.120 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:41.557063+0000", 2024-02-09T18:05:39.120 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0025783999999999998 2024-02-09T18:05:39.120 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.120 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.120 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:39.120 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:41.557086+0000", 2024-02-09T18:05:39.120 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2464999999999999e-05 2024-02-09T18:05:39.120 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.120 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.120 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:05:39.120 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:41.578046+0000", 2024-02-09T18:05:39.120 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.020960097 2024-02-09T18:05:39.121 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.121 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.121 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:39.121 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:41.578095+0000", 2024-02-09T18:05:39.121 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.9385999999999998e-05 2024-02-09T18:05:39.121 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:39.121 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:39.121 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:39.121 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.121 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.121 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "rep_scrubmap(1.5 e945 from shard 8)", 2024-02-09T18:05:39.121 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:05:09.585811+0000", 2024-02-09T18:05:39.121 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 29.438547637999999, 2024-02-09T18:05:39.121 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.051302986000000002, 2024-02-09T18:05:39.121 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:39.122 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T18:05:39.122 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:39.122 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.122 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:39.122 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:05:09.585811+0000", 2024-02-09T18:05:39.122 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:39.122 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.122 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.122 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:39.122 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:05:09.585811+0000", 2024-02-09T18:05:39.122 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:39.122 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.122 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.122 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:39.123 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:05:09.585813+0000", 2024-02-09T18:05:39.123 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.8250000000000001e-06 2024-02-09T18:05:39.123 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.123 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.123 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:39.123 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:05:09.585824+0000", 2024-02-09T18:05:39.123 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1250000000000001e-05 2024-02-09T18:05:39.123 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.123 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.123 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:39.123 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:05:09.585853+0000", 2024-02-09T18:05:39.123 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.8187999999999998e-05 2024-02-09T18:05:39.123 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.124 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.124 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:39.124 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:05:09.585867+0000", 2024-02-09T18:05:39.124 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.453e-05 2024-02-09T18:05:39.124 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.124 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.124 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:39.124 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:05:09.637091+0000", 2024-02-09T18:05:39.124 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.051224108999999997 2024-02-09T18:05:39.124 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.124 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.124 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:39.124 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:05:09.637100+0000", 2024-02-09T18:05:39.124 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.0659999999999993e-06 2024-02-09T18:05:39.125 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.125 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.125 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:39.125 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:05:09.637114+0000", 2024-02-09T18:05:39.125 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4018e-05 2024-02-09T18:05:39.125 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:39.125 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:39.125 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:39.125 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.125 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.125 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "rep_scrubmap(1.5 e945 from shard 11)", 2024-02-09T18:05:39.125 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:05:09.601899+0000", 2024-02-09T18:05:39.126 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 29.422459766999999, 2024-02-09T18:05:39.126 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.035266377000000002, 2024-02-09T18:05:39.126 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:05:39.126 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T18:05:39.126 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:05:39.126 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.126 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:05:39.126 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:05:09.601899+0000", 2024-02-09T18:05:39.126 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:39.126 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.126 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.126 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:05:39.126 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:05:09.601899+0000", 2024-02-09T18:05:39.126 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:05:39.126 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.127 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.127 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:05:39.127 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:05:09.601901+0000", 2024-02-09T18:05:39.127 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.9e-06 2024-02-09T18:05:39.127 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.127 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.127 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:05:39.127 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:05:09.601913+0000", 2024-02-09T18:05:39.127 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1841999999999999e-05 2024-02-09T18:05:39.127 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.127 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.127 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:05:39.127 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:05:09.601942+0000", 2024-02-09T18:05:39.127 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.9023999999999999e-05 2024-02-09T18:05:39.127 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.128 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.128 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:05:39.128 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:05:09.601956+0000", 2024-02-09T18:05:39.128 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3971999999999999e-05 2024-02-09T18:05:39.128 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.128 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.128 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:05:39.128 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:05:09.637134+0000", 2024-02-09T18:05:39.128 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.035178330000000001 2024-02-09T18:05:39.128 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.128 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.128 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:05:39.128 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:05:09.637139+0000", 2024-02-09T18:05:39.128 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2769999999999996e-06 2024-02-09T18:05:39.129 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:05:39.129 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:05:39.129 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:05:39.129 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:05:09.637165+0000", 2024-02-09T18:05:39.129 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.6032000000000001e-05 2024-02-09T18:05:39.129 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:39.129 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:39.129 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:39.129 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:05:39.129 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:05:39.130 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T18:05:39.272 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:05:39.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:39 smithi099 ceph-mon[39420]: 1.2 scrub starts 2024-02-09T18:05:39.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:39 smithi099 ceph-mon[39420]: 1.2 scrub ok 2024-02-09T18:05:39.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:39 smithi179 ceph-mon[43975]: 1.2 scrub starts 2024-02-09T18:05:39.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:39 smithi179 ceph-mon[43975]: 1.2 scrub ok 2024-02-09T18:05:39.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:39 smithi162 ceph-mon[44304]: 1.2 scrub starts 2024-02-09T18:05:39.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:39 smithi162 ceph-mon[44304]: 1.2 scrub ok 2024-02-09T18:05:39.961 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-09T18:05:40.237 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T18:05:40.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:40 smithi099 ceph-mon[39420]: pgmap v2468: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:40.875 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:40 smithi162 ceph-mon[44304]: pgmap v2468: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:40.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:40 smithi179 ceph-mon[43975]: pgmap v2468: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:41.486 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:41 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:05:41.486 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:41 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:05:41.773 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:41 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:05:41.774 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:41 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:05:41.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:41 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:05:41.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:41 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:05:42.149 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T18:05:42.149 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-09T18:05:42.149 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-02-09T18:05:42.149 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T18:05:42.492 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.2 to 1 (10000) 2024-02-09T18:05:42.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:42 smithi099 ceph-mon[39420]: pgmap v2469: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:42.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:42 smithi099 ceph-mon[39420]: osdmap e1002: 12 total, 12 up, 12 in 2024-02-09T18:05:42.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:42 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2404430439' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-09T18:05:42.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:42 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:05:42.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:42 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:05:42.781 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:42 smithi162 ceph-mon[44304]: pgmap v2469: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:42.781 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:42 smithi162 ceph-mon[44304]: osdmap e1002: 12 total, 12 up, 12 in 2024-02-09T18:05:42.781 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:42 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2404430439' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-09T18:05:42.782 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:42 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:05:42.782 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:42 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:05:42.806 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-09T18:05:42.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:42 smithi179 ceph-mon[43975]: pgmap v2469: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:42.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:42 smithi179 ceph-mon[43975]: osdmap e1002: 12 total, 12 up, 12 in 2024-02-09T18:05:42.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:42 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2404430439' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-09T18:05:42.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:42 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:05:42.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:42 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:05:43.071 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T18:05:43.281 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd primary-affinity 2 1 2024-02-09T18:05:43.550 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:05:43.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:43 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2404430439' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-09T18:05:43.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:43 smithi162 ceph-mon[44304]: osdmap e1003: 12 total, 12 up, 12 in 2024-02-09T18:05:43.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:43 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2404430439' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-09T18:05:43.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:43 smithi099 ceph-mon[39420]: osdmap e1003: 12 total, 12 up, 12 in 2024-02-09T18:05:43.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:43 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2404430439' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-09T18:05:43.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:43 smithi179 ceph-mon[43975]: osdmap e1003: 12 total, 12 up, 12 in 2024-02-09T18:05:44.788 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T18:05:44.788 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-09T18:05:44.788 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-02-09T18:05:44.788 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-02-09T18:05:44.788 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T18:05:44.807 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:44 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:05:44.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:44 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:05:44.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:44 smithi162 ceph-mon[44304]: pgmap v2472: 31 pgs: 31 active+clean; 577 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:44.808 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:44 smithi162 ceph-mon[44304]: osdmap e1004: 12 total, 12 up, 12 in 2024-02-09T18:05:44.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:44 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:05:44.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:44 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:05:44.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:44 smithi179 ceph-mon[43975]: pgmap v2472: 31 pgs: 31 active+clean; 577 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:44.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:44 smithi179 ceph-mon[43975]: osdmap e1004: 12 total, 12 up, 12 in 2024-02-09T18:05:44.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:44 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:05:44.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:44 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:05:44.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:44 smithi099 ceph-mon[39420]: pgmap v2472: 31 pgs: 31 active+clean; 577 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:44.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:44 smithi099 ceph-mon[39420]: osdmap e1004: 12 total, 12 up, 12 in 2024-02-09T18:05:45.405 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-09T18:05:45.673 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T18:05:46.511 INFO:teuthology.orchestra.run.smithi099.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-02-09T18:05:46.622 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:46 smithi162 ceph-mon[44304]: pgmap v2474: 31 pgs: 31 active+clean; 577 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:46.622 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:46 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3699188761' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-02-09T18:05:46.916 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:46 smithi099 ceph-mon[39420]: pgmap v2474: 31 pgs: 31 active+clean; 577 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:46.916 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:46 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3699188761' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-02-09T18:05:46.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:46 smithi179 ceph-mon[43975]: pgmap v2474: 31 pgs: 31 active+clean; 577 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:46.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:46 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3699188761' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-02-09T18:05:47.337 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 4 1 2024-02-09T18:05:47.558 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T18:05:47.558 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-02-09T18:05:47.559 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-02-09T18:05:47.559 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-02-09T18:05:47.559 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.559 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:203946 9.4 e945/936 9:20967b11:::benchmark_data_smithi179_161915_object69064:head v 945'12646, mlcod=945'12643)", 2024-02-09T18:05:47.559 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:03:16.957433+0000", 2024-02-09T18:05:47.559 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 150.597749459, 2024-02-09T18:05:47.559 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.022360977000000001, 2024-02-09T18:05:47.559 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:05:47.559 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:47.559 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:05:47.559 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.560 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:05:47.560 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:16.957433+0000", 2024-02-09T18:05:47.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:47.560 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.560 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.560 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:05:47.560 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:16.957433+0000", 2024-02-09T18:05:47.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:47.560 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.560 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.560 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:05:47.561 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:16.957434+0000", 2024-02-09T18:05:47.561 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.58e-07 2024-02-09T18:05:47.561 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.561 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.561 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:05:47.561 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:16.957438+0000", 2024-02-09T18:05:47.561 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8330000000000001e-06 2024-02-09T18:05:47.562 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.562 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.562 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:05:47.562 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:16.957444+0000", 2024-02-09T18:05:47.562 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8200000000000002e-06 2024-02-09T18:05:47.562 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.562 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.562 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:05:47.562 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:16.957449+0000", 2024-02-09T18:05:47.562 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6110000000000004e-06 2024-02-09T18:05:47.562 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.563 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.563 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:47.563 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:16.961764+0000", 2024-02-09T18:05:47.563 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0043151880000000002 2024-02-09T18:05:47.563 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.563 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.563 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:05:47.563 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:16.961787+0000", 2024-02-09T18:05:47.563 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3377999999999999e-05 2024-02-09T18:05:47.564 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.564 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.564 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:05:47.564 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:16.979747+0000", 2024-02-09T18:05:47.564 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.017959649000000001 2024-02-09T18:05:47.564 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.564 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.564 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:05:47.564 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:16.979794+0000", 2024-02-09T18:05:47.565 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7139999999999999e-05 2024-02-09T18:05:47.566 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:47.566 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:05:47.566 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:47.566 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.566 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.566 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:203951 9.c e945/936 9:381d7018:::benchmark_data_smithi179_161915_object69069:head v 945'12821, mlcod=945'12819)", 2024-02-09T18:05:47.566 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:03:16.960865+0000", 2024-02-09T18:05:47.566 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 150.594317421, 2024-02-09T18:05:47.566 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018971024999999999, 2024-02-09T18:05:47.566 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:05:47.567 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:47.567 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:05:47.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.568 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:05:47.568 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:16.960865+0000", 2024-02-09T18:05:47.568 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:47.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.568 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.568 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:05:47.568 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:16.960865+0000", 2024-02-09T18:05:47.568 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:47.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.568 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.569 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:05:47.569 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:16.960867+0000", 2024-02-09T18:05:47.569 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8560000000000001e-06 2024-02-09T18:05:47.569 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.569 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.570 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:05:47.570 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:16.960876+0000", 2024-02-09T18:05:47.570 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2890000000000003e-06 2024-02-09T18:05:47.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.570 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.571 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:05:47.571 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:16.960892+0000", 2024-02-09T18:05:47.571 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5608999999999999e-05 2024-02-09T18:05:47.571 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.571 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.571 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:05:47.572 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:16.960902+0000", 2024-02-09T18:05:47.572 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9820000000000003e-06 2024-02-09T18:05:47.572 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.572 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.572 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:47.572 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:16.962671+0000", 2024-02-09T18:05:47.573 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017691040000000001 2024-02-09T18:05:47.573 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.573 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.573 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:05:47.573 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:16.962692+0000", 2024-02-09T18:05:47.574 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0876000000000001e-05 2024-02-09T18:05:47.574 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.574 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.574 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:05:47.574 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:16.979798+0000", 2024-02-09T18:05:47.574 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.017105967999999999 2024-02-09T18:05:47.575 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.575 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.575 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:05:47.575 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:16.979836+0000", 2024-02-09T18:05:47.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8340999999999998e-05 2024-02-09T18:05:47.575 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:47.576 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:05:47.576 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:47.576 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.576 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.576 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:204895 9.4 e947/936 9:2a8deffb:::benchmark_data_smithi179_161915_object70013:head v 947'12695, mlcod=947'12692)", 2024-02-09T18:05:47.576 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:03:18.657722+0000", 2024-02-09T18:05:47.577 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 148.89746065700001, 2024-02-09T18:05:47.577 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.027015747999999999, 2024-02-09T18:05:47.577 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:05:47.577 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:47.577 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:05:47.577 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.578 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:05:47.578 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:18.657722+0000", 2024-02-09T18:05:47.578 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:47.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.578 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.578 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:05:47.579 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:18.657722+0000", 2024-02-09T18:05:47.579 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:47.579 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.579 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.579 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:05:47.579 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:18.657723+0000", 2024-02-09T18:05:47.580 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9500000000000001e-07 2024-02-09T18:05:47.580 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.580 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.580 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:05:47.580 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:18.657730+0000", 2024-02-09T18:05:47.581 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5970000000000003e-06 2024-02-09T18:05:47.581 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.581 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.581 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:05:47.581 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:18.657744+0000", 2024-02-09T18:05:47.581 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.329e-05 2024-02-09T18:05:47.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.582 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.582 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:05:47.582 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:18.657752+0000", 2024-02-09T18:05:47.582 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7279999999999995e-06 2024-02-09T18:05:47.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.583 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.583 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:47.583 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:18.664697+0000", 2024-02-09T18:05:47.583 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0069450859999999996 2024-02-09T18:05:47.583 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.583 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.584 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:05:47.584 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:18.664720+0000", 2024-02-09T18:05:47.584 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3283999999999999e-05 2024-02-09T18:05:47.584 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.584 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.584 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:05:47.585 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:18.684676+0000", 2024-02-09T18:05:47.585 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019956538999999999 2024-02-09T18:05:47.585 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.585 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.585 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:05:47.585 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:18.684738+0000", 2024-02-09T18:05:47.586 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1328999999999995e-05 2024-02-09T18:05:47.586 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:47.586 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:05:47.586 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:47.586 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.586 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.587 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:205277 9.c e947/936 9:33934be5:::benchmark_data_smithi179_161915_object70395:head v 947'12920, mlcod=947'12914)", 2024-02-09T18:05:47.587 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:03:19.148851+0000", 2024-02-09T18:05:47.587 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 148.40633152500001, 2024-02-09T18:05:47.587 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.021001125999999998, 2024-02-09T18:05:47.587 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:05:47.588 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:47.588 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:05:47.588 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.588 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:05:47.588 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:19.148851+0000", 2024-02-09T18:05:47.588 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:47.589 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.589 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.589 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:05:47.589 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:19.148851+0000", 2024-02-09T18:05:47.589 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:47.590 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.590 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.590 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:05:47.590 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:19.148852+0000", 2024-02-09T18:05:47.590 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9999999999999996e-07 2024-02-09T18:05:47.590 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.591 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.591 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:05:47.591 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:19.148858+0000", 2024-02-09T18:05:47.591 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5370000000000003e-06 2024-02-09T18:05:47.591 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.591 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.592 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:05:47.592 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:19.148874+0000", 2024-02-09T18:05:47.592 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5942000000000001e-05 2024-02-09T18:05:47.592 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.592 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.593 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:05:47.593 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:19.148881+0000", 2024-02-09T18:05:47.593 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7499999999999997e-06 2024-02-09T18:05:47.593 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.593 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.593 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:47.594 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:19.155017+0000", 2024-02-09T18:05:47.594 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0061361369999999998 2024-02-09T18:05:47.594 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.594 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.594 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:05:47.594 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:19.155039+0000", 2024-02-09T18:05:47.595 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1644000000000001e-05 2024-02-09T18:05:47.595 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.595 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.595 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:05:47.595 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:19.169795+0000", 2024-02-09T18:05:47.595 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014756030999999999 2024-02-09T18:05:47.596 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.596 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.596 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:05:47.596 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:19.169852+0000", 2024-02-09T18:05:47.596 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7284999999999998e-05 2024-02-09T18:05:47.596 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:47.597 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:05:47.597 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:47.597 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.597 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.598 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(9.4 947/936 [PushOp(9:2cfebe7a:::benchmark_data_smithi179_161915_object105674:head, version: 909'6583, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(9:2cfebe7a:::benchmark_data_smithi179_161915_object105674:head@909'6583, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-02-09T18:05:47.598 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:03:19.149266+0000", 2024-02-09T18:05:47.598 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 148.405916201, 2024-02-09T18:05:47.598 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020590573000000001, 2024-02-09T18:05:47.598 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:05:47.598 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-09T18:05:47.599 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:05:47.599 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.599 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:05:47.599 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:19.149266+0000", 2024-02-09T18:05:47.599 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:47.599 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.600 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.600 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:05:47.600 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:19.149266+0000", 2024-02-09T18:05:47.600 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:47.600 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.601 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.601 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:05:47.601 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:19.149267+0000", 2024-02-09T18:05:47.601 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8999999999999996e-07 2024-02-09T18:05:47.601 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.602 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.602 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:05:47.602 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:19.149327+0000", 2024-02-09T18:05:47.602 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9404000000000002e-05 2024-02-09T18:05:47.602 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.602 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.603 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:05:47.603 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:19.149346+0000", 2024-02-09T18:05:47.603 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9400999999999999e-05 2024-02-09T18:05:47.603 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.603 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.603 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:05:47.604 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:19.149361+0000", 2024-02-09T18:05:47.604 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4799e-05 2024-02-09T18:05:47.604 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.604 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.604 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:47.604 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:19.163459+0000", 2024-02-09T18:05:47.605 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014098682 2024-02-09T18:05:47.605 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.605 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.605 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:05:47.605 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:19.163469+0000", 2024-02-09T18:05:47.606 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2229999999999996e-06 2024-02-09T18:05:47.606 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.606 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.606 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:05:47.606 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:19.169857+0000", 2024-02-09T18:05:47.606 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0063883739999999996 2024-02-09T18:05:47.607 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:47.607 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:05:47.607 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:47.607 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.607 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.607 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:205476 9.4 e947/936 9:241627de:::benchmark_data_smithi179_161915_object70594:head v 947'12732, mlcod=947'12728)", 2024-02-09T18:05:47.608 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:03:19.402331+0000", 2024-02-09T18:05:47.608 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 148.15285189700001, 2024-02-09T18:05:47.608 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.021966551000000001, 2024-02-09T18:05:47.608 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:05:47.608 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:47.608 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:05:47.609 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.609 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:05:47.609 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:19.402331+0000", 2024-02-09T18:05:47.609 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:47.609 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.609 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.610 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:05:47.610 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:19.402331+0000", 2024-02-09T18:05:47.610 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:47.610 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.610 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.611 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:05:47.611 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:19.402331+0000", 2024-02-09T18:05:47.611 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9400000000000001e-07 2024-02-09T18:05:47.611 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.611 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.611 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:05:47.612 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:19.402335+0000", 2024-02-09T18:05:47.612 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0439999999999997e-06 2024-02-09T18:05:47.612 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.612 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.612 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:05:47.612 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:19.402341+0000", 2024-02-09T18:05:47.613 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7969999999999997e-06 2024-02-09T18:05:47.613 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.613 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.613 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:05:47.613 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:19.402345+0000", 2024-02-09T18:05:47.613 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4309999999999996e-06 2024-02-09T18:05:47.614 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.614 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.614 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:47.614 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:19.405818+0000", 2024-02-09T18:05:47.614 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034726380000000001 2024-02-09T18:05:47.614 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.615 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:05:47.615 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:19.405845+0000", 2024-02-09T18:05:47.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7058000000000001e-05 2024-02-09T18:05:47.615 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.616 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:05:47.616 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:19.424247+0000", 2024-02-09T18:05:47.616 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018402146000000001 2024-02-09T18:05:47.616 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.616 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.617 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:05:47.617 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:19.424297+0000", 2024-02-09T18:05:47.617 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0142999999999997e-05 2024-02-09T18:05:47.617 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:47.617 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:05:47.617 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:47.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.618 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:219160 9.c e956/950 9:3270be20:::benchmark_data_smithi179_161915_object84278:head v 956'13797, mlcod=956'13795)", 2024-02-09T18:05:47.618 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:03:41.045555+0000", 2024-02-09T18:05:47.618 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 126.509627979, 2024-02-09T18:05:47.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020791958999999999, 2024-02-09T18:05:47.619 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:05:47.619 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:47.619 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:05:47.619 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.619 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:05:47.619 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:41.045555+0000", 2024-02-09T18:05:47.620 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:47.620 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.620 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.620 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:05:47.620 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:41.045555+0000", 2024-02-09T18:05:47.621 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:47.621 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.621 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.621 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:05:47.621 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:41.045555+0000", 2024-02-09T18:05:47.621 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1500000000000005e-07 2024-02-09T18:05:47.622 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.622 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.622 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:05:47.622 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:41.045560+0000", 2024-02-09T18:05:47.622 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9439999999999996e-06 2024-02-09T18:05:47.622 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.623 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:05:47.623 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:41.045581+0000", 2024-02-09T18:05:47.623 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1151000000000001e-05 2024-02-09T18:05:47.623 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.624 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:05:47.624 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:41.045587+0000", 2024-02-09T18:05:47.624 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7289999999999997e-06 2024-02-09T18:05:47.624 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.624 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.625 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:47.625 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:41.049958+0000", 2024-02-09T18:05:47.625 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0043708050000000002 2024-02-09T18:05:47.625 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.625 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.625 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:05:47.626 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:41.049980+0000", 2024-02-09T18:05:47.626 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2189e-05 2024-02-09T18:05:47.626 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.626 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.626 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:05:47.626 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:41.066290+0000", 2024-02-09T18:05:47.627 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.016310130999999999 2024-02-09T18:05:47.627 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.627 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.627 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:05:47.628 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:41.066347+0000", 2024-02-09T18:05:47.628 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6495e-05 2024-02-09T18:05:47.628 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:47.628 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:05:47.628 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:47.628 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.629 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.629 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:221848 9.4 e958/949 9:2a2f7bdf:::benchmark_data_smithi179_161915_object86966:head v 958'13808, mlcod=958'13806)", 2024-02-09T18:05:47.629 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:03:44.393959+0000", 2024-02-09T18:05:47.629 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 123.161223904, 2024-02-09T18:05:47.629 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020762971000000002, 2024-02-09T18:05:47.629 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:05:47.630 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:47.630 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:05:47.630 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.630 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:05:47.630 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:44.393959+0000", 2024-02-09T18:05:47.630 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:47.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.631 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:05:47.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:44.393959+0000", 2024-02-09T18:05:47.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:47.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.631 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:05:47.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:44.393959+0000", 2024-02-09T18:05:47.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6100000000000001e-07 2024-02-09T18:05:47.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.631 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:05:47.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:44.393964+0000", 2024-02-09T18:05:47.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5260000000000004e-06 2024-02-09T18:05:47.632 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.632 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.632 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:05:47.632 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:44.393971+0000", 2024-02-09T18:05:47.632 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5519999999999999e-06 2024-02-09T18:05:47.632 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.632 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.632 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:05:47.632 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:44.393977+0000", 2024-02-09T18:05:47.632 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3990000000000003e-06 2024-02-09T18:05:47.632 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.632 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.632 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:47.632 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:44.398307+0000", 2024-02-09T18:05:47.633 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0043299180000000003 2024-02-09T18:05:47.633 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.633 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.633 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:05:47.633 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:44.398328+0000", 2024-02-09T18:05:47.633 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1495e-05 2024-02-09T18:05:47.633 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.633 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.633 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:05:47.633 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:44.414678+0000", 2024-02-09T18:05:47.633 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.016350133999999999 2024-02-09T18:05:47.633 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.633 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.633 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:05:47.633 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:44.414722+0000", 2024-02-09T18:05:47.634 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3485999999999997e-05 2024-02-09T18:05:47.634 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:47.634 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:05:47.634 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:47.634 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.634 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.634 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:223748 9.4 e962/949 9:2856971a:::benchmark_data_smithi179_161915_object88866:head v 962'13924, mlcod=962'13922)", 2024-02-09T18:05:47.634 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:03:48.369608+0000", 2024-02-09T18:05:47.634 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 119.185575067, 2024-02-09T18:05:47.634 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019377798000000002, 2024-02-09T18:05:47.634 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:05:47.634 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:47.634 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:05:47.634 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.634 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:05:47.635 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:48.369608+0000", 2024-02-09T18:05:47.635 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:47.635 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.635 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.635 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:05:47.635 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:48.369608+0000", 2024-02-09T18:05:47.635 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:47.635 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.635 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.635 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:05:47.635 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:48.369609+0000", 2024-02-09T18:05:47.635 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2470000000000001e-06 2024-02-09T18:05:47.635 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.635 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.635 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:05:47.636 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:48.369617+0000", 2024-02-09T18:05:47.636 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3639999999999999e-06 2024-02-09T18:05:47.636 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.636 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.636 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:05:47.636 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:48.369633+0000", 2024-02-09T18:05:47.636 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6079999999999999e-05 2024-02-09T18:05:47.636 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.636 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.636 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:05:47.636 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:48.369643+0000", 2024-02-09T18:05:47.636 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0164e-05 2024-02-09T18:05:47.636 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.636 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.636 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:47.637 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:48.369726+0000", 2024-02-09T18:05:47.637 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2238999999999996e-05 2024-02-09T18:05:47.637 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.637 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.637 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:05:47.637 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:48.369752+0000", 2024-02-09T18:05:47.637 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5995000000000001e-05 2024-02-09T18:05:47.637 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.637 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.637 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:05:47.637 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:48.388937+0000", 2024-02-09T18:05:47.637 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019185443 2024-02-09T18:05:47.637 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.637 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.637 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:05:47.638 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:48.388985+0000", 2024-02-09T18:05:47.638 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8266000000000002e-05 2024-02-09T18:05:47.638 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:47.638 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:05:47.638 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:47.638 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.638 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.638 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:233573 9.c e966/950 9:3111b377:::benchmark_data_smithi179_161915_object98691:head v 966'14697, mlcod=966'14694)", 2024-02-09T18:05:47.638 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:03:59.536242+0000", 2024-02-09T18:05:47.638 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 108.01894027, 2024-02-09T18:05:47.638 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020902422, 2024-02-09T18:05:47.638 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:05:47.638 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:47.638 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:05:47.638 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.639 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:05:47.639 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:59.536242+0000", 2024-02-09T18:05:47.639 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:47.639 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.639 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.639 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:05:47.639 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:59.536242+0000", 2024-02-09T18:05:47.639 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:47.639 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.639 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.639 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:05:47.639 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:59.536244+0000", 2024-02-09T18:05:47.639 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.13e-06 2024-02-09T18:05:47.639 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.640 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.640 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:05:47.640 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:59.536254+0000", 2024-02-09T18:05:47.640 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0550999999999999e-05 2024-02-09T18:05:47.640 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.640 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.640 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:05:47.640 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:59.536268+0000", 2024-02-09T18:05:47.640 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3923999999999999e-05 2024-02-09T18:05:47.640 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.640 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.640 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:05:47.640 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:59.536276+0000", 2024-02-09T18:05:47.640 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3550000000000005e-06 2024-02-09T18:05:47.640 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.641 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.641 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:47.641 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:59.538544+0000", 2024-02-09T18:05:47.641 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0022680019999999999 2024-02-09T18:05:47.641 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.641 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.641 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:05:47.641 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:59.538568+0000", 2024-02-09T18:05:47.641 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4046e-05 2024-02-09T18:05:47.641 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.641 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.641 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:05:47.641 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:59.557106+0000", 2024-02-09T18:05:47.641 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018537475000000001 2024-02-09T18:05:47.641 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.642 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.642 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:05:47.642 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:59.557145+0000", 2024-02-09T18:05:47.642 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8939000000000001e-05 2024-02-09T18:05:47.642 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:47.642 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:05:47.642 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:47.642 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.642 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.642 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:234503 9.4 e966/949 9:2f85dca1:::benchmark_data_smithi179_161915_object99621:head v 966'14600, mlcod=966'14597)", 2024-02-09T18:05:47.642 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:04:00.561320+0000", 2024-02-09T18:05:47.642 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 106.99386238699999, 2024-02-09T18:05:47.642 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019198045, 2024-02-09T18:05:47.642 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:05:47.643 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:47.643 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:05:47.643 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.643 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:05:47.643 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:00.561320+0000", 2024-02-09T18:05:47.643 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:47.643 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.643 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.643 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:05:47.643 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:00.561320+0000", 2024-02-09T18:05:47.643 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:47.643 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.643 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.643 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:05:47.643 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:00.561321+0000", 2024-02-09T18:05:47.643 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1429999999999999e-06 2024-02-09T18:05:47.644 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.644 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.644 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:05:47.644 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:00.561331+0000", 2024-02-09T18:05:47.644 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2380000000000003e-06 2024-02-09T18:05:47.644 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.644 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.644 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:05:47.644 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:00.561343+0000", 2024-02-09T18:05:47.644 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2578999999999999e-05 2024-02-09T18:05:47.644 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.644 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.644 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:05:47.644 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:00.561352+0000", 2024-02-09T18:05:47.645 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5269999999999998e-06 2024-02-09T18:05:47.645 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.645 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.645 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:47.645 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:00.562531+0000", 2024-02-09T18:05:47.645 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0011792809999999999 2024-02-09T18:05:47.645 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.645 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.645 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:05:47.645 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:00.562547+0000", 2024-02-09T18:05:47.645 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5838999999999999e-05 2024-02-09T18:05:47.645 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.645 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.645 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:05:47.645 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:00.580453+0000", 2024-02-09T18:05:47.645 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0179063 2024-02-09T18:05:47.646 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.646 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.646 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:05:47.646 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:00.580518+0000", 2024-02-09T18:05:47.646 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5137999999999999e-05 2024-02-09T18:05:47.646 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:47.646 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:05:47.646 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:47.646 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.646 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.646 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:237623 9.c e966/950 9:32b688ba:::benchmark_data_smithi179_161915_object102741:head v 966'14934, mlcod=966'14932)", 2024-02-09T18:05:47.646 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:04:04.021519+0000", 2024-02-09T18:05:47.646 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 103.53366352099999, 2024-02-09T18:05:47.646 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019581734999999999, 2024-02-09T18:05:47.646 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:05:47.647 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:47.647 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:05:47.647 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.647 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:05:47.647 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:04.021519+0000", 2024-02-09T18:05:47.647 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:47.647 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.647 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.647 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:05:47.647 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:04.021519+0000", 2024-02-09T18:05:47.647 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:47.647 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.647 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.647 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:05:47.647 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:04.021521+0000", 2024-02-09T18:05:47.648 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5600000000000001e-06 2024-02-09T18:05:47.648 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.648 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.648 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:05:47.648 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:04.021530+0000", 2024-02-09T18:05:47.648 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6930000000000002e-06 2024-02-09T18:05:47.648 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.648 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.648 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:05:47.648 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:04.021550+0000", 2024-02-09T18:05:47.648 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9253e-05 2024-02-09T18:05:47.648 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.648 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.648 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:05:47.648 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:04.021560+0000", 2024-02-09T18:05:47.649 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0637e-05 2024-02-09T18:05:47.649 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.649 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.649 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:47.649 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:04.021628+0000", 2024-02-09T18:05:47.649 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7898999999999995e-05 2024-02-09T18:05:47.649 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.649 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.649 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:05:47.649 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:04.021649+0000", 2024-02-09T18:05:47.649 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1121e-05 2024-02-09T18:05:47.649 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.649 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.649 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:05:47.649 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:04.041048+0000", 2024-02-09T18:05:47.650 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019398569000000001 2024-02-09T18:05:47.650 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.650 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.650 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:05:47.650 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:04.041101+0000", 2024-02-09T18:05:47.650 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3003000000000001e-05 2024-02-09T18:05:47.650 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:47.650 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:05:47.650 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:47.650 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.650 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.650 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:243371 9.c e966/950 9:374de1c3:::benchmark_data_smithi179_161915_object108489:head v 966'15271, mlcod=966'15268)", 2024-02-09T18:05:47.650 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:04:10.768947+0000", 2024-02-09T18:05:47.650 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 96.786235535000003, 2024-02-09T18:05:47.650 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020516627999999999, 2024-02-09T18:05:47.650 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:05:47.651 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:47.651 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:05:47.651 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.651 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:05:47.651 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:10.768947+0000", 2024-02-09T18:05:47.651 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:47.651 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.651 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.651 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:05:47.651 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:10.768947+0000", 2024-02-09T18:05:47.651 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:47.651 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.651 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.651 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:05:47.651 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:10.768948+0000", 2024-02-09T18:05:47.652 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.018e-06 2024-02-09T18:05:47.652 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.652 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.652 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:05:47.652 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:10.768956+0000", 2024-02-09T18:05:47.652 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1319999999999994e-06 2024-02-09T18:05:47.652 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.652 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.652 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:05:47.652 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:10.768977+0000", 2024-02-09T18:05:47.652 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1180999999999999e-05 2024-02-09T18:05:47.652 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.652 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.652 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:05:47.653 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:10.768986+0000", 2024-02-09T18:05:47.653 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2390000000000002e-06 2024-02-09T18:05:47.653 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.653 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:47.653 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:10.769036+0000", 2024-02-09T18:05:47.653 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0726999999999997e-05 2024-02-09T18:05:47.653 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.653 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:05:47.653 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:10.769051+0000", 2024-02-09T18:05:47.653 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4715000000000001e-05 2024-02-09T18:05:47.653 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.654 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:05:47.654 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:10.789407+0000", 2024-02-09T18:05:47.654 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020355518999999999 2024-02-09T18:05:47.654 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.654 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.654 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:05:47.654 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:10.789464+0000", 2024-02-09T18:05:47.654 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7096999999999999e-05 2024-02-09T18:05:47.654 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:47.654 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:05:47.654 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:47.654 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.654 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.654 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:260592 9.c e972/950 9:3a4646a6:::benchmark_data_smithi179_161915_object125710:head v 972'16399, mlcod=972'16397)", 2024-02-09T18:05:47.654 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:04:31.465067+0000", 2024-02-09T18:05:47.654 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 76.090115406999999, 2024-02-09T18:05:47.655 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019138470000000001, 2024-02-09T18:05:47.655 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:05:47.655 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:47.655 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:05:47.655 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.655 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:05:47.655 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:31.465067+0000", 2024-02-09T18:05:47.655 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:47.655 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.655 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.655 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:05:47.655 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:31.465067+0000", 2024-02-09T18:05:47.655 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:47.655 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.656 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:05:47.656 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:31.465068+0000", 2024-02-09T18:05:47.656 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6500000000000008e-07 2024-02-09T18:05:47.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.656 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:05:47.656 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:31.465076+0000", 2024-02-09T18:05:47.656 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9740000000000002e-06 2024-02-09T18:05:47.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.656 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:05:47.656 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:31.465092+0000", 2024-02-09T18:05:47.656 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5903999999999999e-05 2024-02-09T18:05:47.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.657 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.657 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:05:47.657 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:31.465100+0000", 2024-02-09T18:05:47.657 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1189999999999997e-06 2024-02-09T18:05:47.657 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.657 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.657 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:47.657 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:31.470568+0000", 2024-02-09T18:05:47.657 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0054682769999999997 2024-02-09T18:05:47.657 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.657 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.657 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:05:47.657 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:31.470590+0000", 2024-02-09T18:05:47.657 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.194e-05 2024-02-09T18:05:47.657 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.658 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.658 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:05:47.658 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:31.484154+0000", 2024-02-09T18:05:47.658 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013563950999999999 2024-02-09T18:05:47.658 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.658 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.658 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:05:47.658 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:31.484206+0000", 2024-02-09T18:05:47.658 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.134e-05 2024-02-09T18:05:47.658 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:47.658 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:05:47.658 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:47.658 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.658 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.658 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:262314 9.4 e973/949 9:296b01c6:::benchmark_data_smithi179_161915_object127432:head v 973'16279, mlcod=973'16276)", 2024-02-09T18:05:47.659 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:04:33.465204+0000", 2024-02-09T18:05:47.659 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 74.089978158999998, 2024-02-09T18:05:47.659 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019458262, 2024-02-09T18:05:47.659 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:05:47.659 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:47.659 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:05:47.659 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.659 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:05:47.659 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:33.465204+0000", 2024-02-09T18:05:47.659 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:47.659 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.659 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.659 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:05:47.659 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:33.465204+0000", 2024-02-09T18:05:47.659 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:47.660 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.660 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.660 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:05:47.660 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:33.465205+0000", 2024-02-09T18:05:47.660 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9800000000000004e-07 2024-02-09T18:05:47.660 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.660 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.660 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:05:47.660 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:33.465212+0000", 2024-02-09T18:05:47.660 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5690000000000001e-06 2024-02-09T18:05:47.660 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.660 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.661 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:05:47.661 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:33.465220+0000", 2024-02-09T18:05:47.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7460000000000006e-06 2024-02-09T18:05:47.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.661 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:05:47.661 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:33.465226+0000", 2024-02-09T18:05:47.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9209999999999997e-06 2024-02-09T18:05:47.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.662 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:47.662 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:33.469392+0000", 2024-02-09T18:05:47.662 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041658260000000001 2024-02-09T18:05:47.662 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.662 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.662 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:05:47.662 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:33.469417+0000", 2024-02-09T18:05:47.662 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4768000000000001e-05 2024-02-09T18:05:47.662 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.662 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.662 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:05:47.663 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:33.484603+0000", 2024-02-09T18:05:47.663 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015185865 2024-02-09T18:05:47.663 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.663 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.663 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:05:47.663 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:33.484663+0000", 2024-02-09T18:05:47.663 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0068999999999997e-05 2024-02-09T18:05:47.663 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:47.663 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:05:47.663 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:47.663 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.663 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.664 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:265365 9.4 e975/949 9:2c166aec:::benchmark_data_smithi179_161915_object130483:head v 975'16481, mlcod=975'16478)", 2024-02-09T18:05:47.664 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:04:37.487618+0000", 2024-02-09T18:05:47.664 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 70.067564681999997, 2024-02-09T18:05:47.664 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.021574613999999999, 2024-02-09T18:05:47.664 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:05:47.664 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:47.664 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:05:47.664 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.664 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:05:47.664 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:37.487618+0000", 2024-02-09T18:05:47.665 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:47.665 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.665 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.665 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:05:47.665 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:37.487618+0000", 2024-02-09T18:05:47.665 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:47.665 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.665 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.665 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:05:47.665 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:37.487619+0000", 2024-02-09T18:05:47.665 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0640000000000001e-06 2024-02-09T18:05:47.665 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.666 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.666 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:05:47.666 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:37.487627+0000", 2024-02-09T18:05:47.666 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2770000000000005e-06 2024-02-09T18:05:47.666 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.666 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.666 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:05:47.666 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:37.487644+0000", 2024-02-09T18:05:47.666 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6512999999999999e-05 2024-02-09T18:05:47.666 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.666 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.666 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:05:47.666 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:37.487652+0000", 2024-02-09T18:05:47.667 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4800000000000001e-06 2024-02-09T18:05:47.667 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.667 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.667 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:47.667 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:37.489240+0000", 2024-02-09T18:05:47.667 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0015878000000000001 2024-02-09T18:05:47.667 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.667 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.667 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:05:47.667 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:37.489263+0000", 2024-02-09T18:05:47.667 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3331000000000001e-05 2024-02-09T18:05:47.667 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.667 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.668 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:05:47.668 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:37.509135+0000", 2024-02-09T18:05:47.668 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019871166999999999 2024-02-09T18:05:47.668 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.668 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.668 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:05:47.668 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:37.509193+0000", 2024-02-09T18:05:47.668 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7982000000000002e-05 2024-02-09T18:05:47.668 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:47.668 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:05:47.668 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:47.669 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.669 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.669 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:265645 9.4 e975/949 9:235046d6:::benchmark_data_smithi179_161915_object130763:head v 975'16501, mlcod=975'16497)", 2024-02-09T18:05:47.669 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:04:37.785487+0000", 2024-02-09T18:05:47.669 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 69.769695831000007, 2024-02-09T18:05:47.669 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019025454000000001, 2024-02-09T18:05:47.669 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:05:47.669 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:47.669 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:05:47.669 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.669 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:05:47.669 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:37.785487+0000", 2024-02-09T18:05:47.669 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:47.670 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.670 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.670 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:05:47.670 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:37.785487+0000", 2024-02-09T18:05:47.670 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:47.670 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.670 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.670 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:05:47.670 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:37.785488+0000", 2024-02-09T18:05:47.670 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2800000000000005e-07 2024-02-09T18:05:47.670 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.670 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.671 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:05:47.671 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:37.785496+0000", 2024-02-09T18:05:47.671 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0569999999999993e-06 2024-02-09T18:05:47.671 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.671 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.671 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:05:47.671 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:37.785508+0000", 2024-02-09T18:05:47.671 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1858000000000001e-05 2024-02-09T18:05:47.671 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.671 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.671 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:05:47.672 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:37.785516+0000", 2024-02-09T18:05:47.672 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9519999999999994e-06 2024-02-09T18:05:47.672 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.672 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.672 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:47.672 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:37.785754+0000", 2024-02-09T18:05:47.672 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023883899999999999 2024-02-09T18:05:47.672 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.672 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.672 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:05:47.673 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:37.785773+0000", 2024-02-09T18:05:47.673 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8855000000000002e-05 2024-02-09T18:05:47.673 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.673 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.673 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:05:47.673 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:37.804465+0000", 2024-02-09T18:05:47.673 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018691888 2024-02-09T18:05:47.673 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.673 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.673 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:05:47.673 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:37.804512+0000", 2024-02-09T18:05:47.673 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7076999999999997e-05 2024-02-09T18:05:47.673 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:47.674 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:05:47.674 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:47.674 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.674 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.674 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:267626 9.c e976/950 9:3a2ed158:::benchmark_data_smithi179_161915_object132744:head v 976'16818, mlcod=976'16814)", 2024-02-09T18:05:47.674 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:04:40.049911+0000", 2024-02-09T18:05:47.676 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 67.505271351000005, 2024-02-09T18:05:47.676 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019692004999999999, 2024-02-09T18:05:47.676 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:05:47.676 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:47.676 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:05:47.676 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.676 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:05:47.676 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:40.049911+0000", 2024-02-09T18:05:47.677 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:47.677 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.677 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.677 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:05:47.677 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:40.049911+0000", 2024-02-09T18:05:47.677 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:47.677 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.677 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.677 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:05:47.677 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:40.049912+0000", 2024-02-09T18:05:47.677 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7800000000000002e-07 2024-02-09T18:05:47.677 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.678 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.678 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:05:47.678 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:40.049916+0000", 2024-02-09T18:05:47.678 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4179999999999999e-06 2024-02-09T18:05:47.678 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.678 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.678 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:05:47.678 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:40.049922+0000", 2024-02-09T18:05:47.678 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0430000000000004e-06 2024-02-09T18:05:47.678 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.678 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.678 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:05:47.679 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:40.049926+0000", 2024-02-09T18:05:47.679 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0269999999999997e-06 2024-02-09T18:05:47.679 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.679 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.679 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:47.679 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:40.056173+0000", 2024-02-09T18:05:47.679 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0062466730000000003 2024-02-09T18:05:47.679 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.679 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.679 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:05:47.679 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:40.056191+0000", 2024-02-09T18:05:47.679 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8331000000000001e-05 2024-02-09T18:05:47.679 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.680 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.680 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:05:47.680 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:40.069550+0000", 2024-02-09T18:05:47.680 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013358636 2024-02-09T18:05:47.680 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.680 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.680 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:05:47.680 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:40.069603+0000", 2024-02-09T18:05:47.680 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3498999999999997e-05 2024-02-09T18:05:47.680 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:47.680 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:05:47.681 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:47.681 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.681 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.681 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:268192 9.c e976/950 9:33146611:::benchmark_data_smithi179_161915_object133310:head v 976'16855, mlcod=976'16850)", 2024-02-09T18:05:47.681 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:04:40.646531+0000", 2024-02-09T18:05:47.681 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 66.908651762999995, 2024-02-09T18:05:47.681 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018893849000000001, 2024-02-09T18:05:47.681 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:05:47.681 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:47.681 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:05:47.681 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.681 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:05:47.682 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:40.646531+0000", 2024-02-09T18:05:47.682 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:47.682 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.682 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.682 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:05:47.682 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:40.646531+0000", 2024-02-09T18:05:47.682 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:47.682 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.682 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.682 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:05:47.682 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:40.646532+0000", 2024-02-09T18:05:47.682 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3200000000000004e-07 2024-02-09T18:05:47.683 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.683 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.683 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:05:47.683 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:40.646538+0000", 2024-02-09T18:05:47.683 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0009999999999998e-06 2024-02-09T18:05:47.683 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.683 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.683 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:05:47.683 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:40.646549+0000", 2024-02-09T18:05:47.683 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1616e-05 2024-02-09T18:05:47.683 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.683 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.683 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:05:47.684 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:40.646557+0000", 2024-02-09T18:05:47.684 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4019999999999996e-06 2024-02-09T18:05:47.684 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.684 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.684 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:47.684 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:40.653270+0000", 2024-02-09T18:05:47.684 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0067137580000000002 2024-02-09T18:05:47.684 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.684 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.684 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:05:47.684 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:40.653292+0000", 2024-02-09T18:05:47.685 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1114000000000001e-05 2024-02-09T18:05:47.685 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.685 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.685 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:05:47.685 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:40.665393+0000", 2024-02-09T18:05:47.685 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012101859 2024-02-09T18:05:47.685 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.685 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.685 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:05:47.685 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:40.665425+0000", 2024-02-09T18:05:47.685 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1367e-05 2024-02-09T18:05:47.685 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:47.685 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:05:47.686 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:47.686 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.686 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.686 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "rep_scrubmap(1.e e938 from shard 3)", 2024-02-09T18:05:47.686 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:05:10.067304+0000", 2024-02-09T18:05:47.686 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 37.487879079000002, 2024-02-09T18:05:47.686 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020020283, 2024-02-09T18:05:47.686 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:05:47.686 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-09T18:05:47.686 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:05:47.686 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.686 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:05:47.687 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:05:10.067304+0000", 2024-02-09T18:05:47.687 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:47.687 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.687 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.687 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:05:47.687 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:05:10.067304+0000", 2024-02-09T18:05:47.687 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:47.687 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.687 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.687 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:05:47.687 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:05:10.067305+0000", 2024-02-09T18:05:47.687 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9010000000000001e-06 2024-02-09T18:05:47.688 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.688 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.688 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:05:47.688 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:05:10.067316+0000", 2024-02-09T18:05:47.688 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0611999999999999e-05 2024-02-09T18:05:47.688 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.688 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.688 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:05:47.688 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:05:10.067338+0000", 2024-02-09T18:05:47.688 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1478e-05 2024-02-09T18:05:47.688 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.689 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.689 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:05:47.689 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:05:10.067351+0000", 2024-02-09T18:05:47.689 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3705e-05 2024-02-09T18:05:47.689 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.689 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.689 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:47.689 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:05:10.087303+0000", 2024-02-09T18:05:47.689 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019951488999999999 2024-02-09T18:05:47.689 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.689 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.689 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:05:47.690 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:05:10.087311+0000", 2024-02-09T18:05:47.690 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6950000000000006e-06 2024-02-09T18:05:47.690 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:47.690 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:47.690 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:05:47.690 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:05:10.087324+0000", 2024-02-09T18:05:47.690 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2403e-05 2024-02-09T18:05:47.690 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:47.690 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:05:47.690 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:47.691 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:47.691 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:05:47.691 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T18:05:47.692 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:05:47.692 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:47 smithi162 ceph-mon[44304]: 1.1c scrub starts 2024-02-09T18:05:47.692 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:47 smithi162 ceph-mon[44304]: 1.1c scrub ok 2024-02-09T18:05:47.692 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:47 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3699188761' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-02-09T18:05:47.692 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:47 smithi162 ceph-mon[44304]: osdmap e1005: 12 total, 12 up, 12 in 2024-02-09T18:05:47.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:47 smithi099 ceph-mon[39420]: 1.1c scrub starts 2024-02-09T18:05:47.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:47 smithi099 ceph-mon[39420]: 1.1c scrub ok 2024-02-09T18:05:47.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:47 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3699188761' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-02-09T18:05:47.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:47 smithi099 ceph-mon[39420]: osdmap e1005: 12 total, 12 up, 12 in 2024-02-09T18:05:47.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:47 smithi179 ceph-mon[43975]: 1.1c scrub starts 2024-02-09T18:05:47.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:47 smithi179 ceph-mon[43975]: 1.1c scrub ok 2024-02-09T18:05:47.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:47 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3699188761' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-02-09T18:05:47.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:47 smithi179 ceph-mon[43975]: osdmap e1005: 12 total, 12 up, 12 in 2024-02-09T18:05:48.273 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-09T18:05:48.544 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T18:05:48.789 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:48 smithi162 ceph-mon[44304]: pgmap v2476: 31 pgs: 31 active+clean; 577 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:48.789 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:48 smithi162 ceph-mon[44304]: osdmap e1006: 12 total, 12 up, 12 in 2024-02-09T18:05:48.813 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:48 smithi099 ceph-mon[39420]: pgmap v2476: 31 pgs: 31 active+clean; 577 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:48.813 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:48 smithi099 ceph-mon[39420]: osdmap e1006: 12 total, 12 up, 12 in 2024-02-09T18:05:48.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:48 smithi179 ceph-mon[43975]: pgmap v2476: 31 pgs: 31 active+clean; 577 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:48.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:48 smithi179 ceph-mon[43975]: osdmap e1006: 12 total, 12 up, 12 in 2024-02-09T18:05:49.692 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:49 smithi162 ceph-mon[44304]: 1.18 deep-scrub starts 2024-02-09T18:05:49.692 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:49 smithi162 ceph-mon[44304]: 1.18 deep-scrub ok 2024-02-09T18:05:49.692 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:49 smithi162 ceph-mon[44304]: 1.c scrub starts 2024-02-09T18:05:49.692 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:49 smithi162 ceph-mon[44304]: 1.c scrub ok 2024-02-09T18:05:49.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:49 smithi179 ceph-mon[43975]: 1.18 deep-scrub starts 2024-02-09T18:05:49.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:49 smithi179 ceph-mon[43975]: 1.18 deep-scrub ok 2024-02-09T18:05:49.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:49 smithi179 ceph-mon[43975]: 1.c scrub starts 2024-02-09T18:05:49.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:49 smithi179 ceph-mon[43975]: 1.c scrub ok 2024-02-09T18:05:49.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:49 smithi099 ceph-mon[39420]: 1.18 deep-scrub starts 2024-02-09T18:05:49.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:49 smithi099 ceph-mon[39420]: 1.18 deep-scrub ok 2024-02-09T18:05:49.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:49 smithi099 ceph-mon[39420]: 1.c scrub starts 2024-02-09T18:05:49.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:49 smithi099 ceph-mon[39420]: 1.c scrub ok 2024-02-09T18:05:50.248 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T18:05:50.249 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-09T18:05:50.249 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-02-09T18:05:50.249 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T18:05:50.521 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:50 smithi162 ceph-mon[44304]: pgmap v2478: 31 pgs: 31 active+clean; 577 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:50.521 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:50 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/248624455' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-09T18:05:50.528 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.4 to 1 (10000) 2024-02-09T18:05:50.918 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-09T18:05:50.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:50 smithi179 ceph-mon[43975]: pgmap v2478: 31 pgs: 31 active+clean; 577 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:50.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:50 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/248624455' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-09T18:05:50.948 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:50 smithi099 ceph-mon[39420]: pgmap v2478: 31 pgs: 31 active+clean; 577 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:50.948 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:50 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/248624455' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-09T18:05:51.185 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T18:05:51.397 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd primary-affinity 4 1 2024-02-09T18:05:51.672 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:05:51.733 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:51 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/248624455' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-09T18:05:51.734 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:51 smithi162 ceph-mon[44304]: osdmap e1007: 12 total, 12 up, 12 in 2024-02-09T18:05:51.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:51 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/248624455' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-09T18:05:51.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:51 smithi099 ceph-mon[39420]: osdmap e1007: 12 total, 12 up, 12 in 2024-02-09T18:05:51.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:51 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/248624455' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-09T18:05:51.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:51 smithi179 ceph-mon[43975]: osdmap e1007: 12 total, 12 up, 12 in 2024-02-09T18:05:52.641 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:52 smithi162 ceph-mon[44304]: pgmap v2480: 31 pgs: 31 active+clean; 577 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:52.641 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:52 smithi162 ceph-mon[44304]: 1.12 scrub starts 2024-02-09T18:05:52.641 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:52 smithi162 ceph-mon[44304]: 1.12 scrub ok 2024-02-09T18:05:52.641 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:52 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:05:52.641 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:52 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:05:52.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:52 smithi179 ceph-mon[43975]: pgmap v2480: 31 pgs: 31 active+clean; 577 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:52.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:52 smithi179 ceph-mon[43975]: 1.12 scrub starts 2024-02-09T18:05:52.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:52 smithi179 ceph-mon[43975]: 1.12 scrub ok 2024-02-09T18:05:52.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:52 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:05:52.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:52 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:05:52.923 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T18:05:52.924 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-09T18:05:52.924 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-02-09T18:05:52.924 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-02-09T18:05:52.924 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T18:05:52.962 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:52 smithi099 ceph-mon[39420]: pgmap v2480: 31 pgs: 31 active+clean; 577 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:52.962 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:52 smithi099 ceph-mon[39420]: 1.12 scrub starts 2024-02-09T18:05:52.962 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:52 smithi099 ceph-mon[39420]: 1.12 scrub ok 2024-02-09T18:05:52.962 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:52 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:05:52.963 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:52 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:05:53.544 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-09T18:05:53.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:53 smithi099 ceph-mon[39420]: osdmap e1008: 12 total, 12 up, 12 in 2024-02-09T18:05:53.811 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T18:05:53.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:53 smithi179 ceph-mon[43975]: osdmap e1008: 12 total, 12 up, 12 in 2024-02-09T18:05:53.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:53 smithi162 ceph-mon[44304]: osdmap e1008: 12 total, 12 up, 12 in 2024-02-09T18:05:54.550 INFO:teuthology.orchestra.run.smithi099.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-02-09T18:05:54.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:54 smithi162 ceph-mon[44304]: pgmap v2482: 31 pgs: 31 active+clean; 577 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:54.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:54 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/763620765' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-02-09T18:05:54.914 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:54 smithi099 ceph-mon[39420]: pgmap v2482: 31 pgs: 31 active+clean; 577 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:54.914 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:54 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/763620765' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-02-09T18:05:54.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:54 smithi179 ceph-mon[43975]: pgmap v2482: 31 pgs: 31 active+clean; 577 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:54.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:54 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/763620765' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-02-09T18:05:55.275 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 5 1 2024-02-09T18:05:55.545 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:05:55.554 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:55 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/763620765' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-02-09T18:05:55.554 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:55 smithi099 ceph-mon[39420]: osdmap e1009: 12 total, 12 up, 12 in 2024-02-09T18:05:55.555 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T18:05:55.556 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-02-09T18:05:55.556 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-02-09T18:05:55.556 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-02-09T18:05:55.556 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.556 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26305.0:227494 8.0 8:01b71254:::benchmark_data_smithi179_148205_object104422:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e855)", 2024-02-09T18:05:55.556 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:59:11.998116+0000", 2024-02-09T18:05:55.556 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 403.553359344, 2024-02-09T18:05:55.556 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.34592274, 2024-02-09T18:05:55.556 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:05:55.556 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:55.557 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T18:05:55.557 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26305", 2024-02-09T18:05:55.557 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:05:55.557 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 227494 2024-02-09T18:05:55.557 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.557 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:05:55.557 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.557 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:05:55.557 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998116+0000", 2024-02-09T18:05:55.557 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:55.557 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.557 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.557 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:05:55.558 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998116+0000", 2024-02-09T18:05:55.558 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:55.558 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.558 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.558 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:05:55.558 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998117+0000", 2024-02-09T18:05:55.558 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5149999999999999e-06 2024-02-09T18:05:55.558 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.558 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.558 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:05:55.558 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998122+0000", 2024-02-09T18:05:55.558 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7809999999999996e-06 2024-02-09T18:05:55.559 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.559 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.559 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:05:55.559 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998137+0000", 2024-02-09T18:05:55.559 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4949e-05 2024-02-09T18:05:55.559 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.559 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.559 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:05:55.559 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998146+0000", 2024-02-09T18:05:55.559 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2310000000000002e-06 2024-02-09T18:05:55.559 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.559 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.559 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:55.559 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998290+0000", 2024-02-09T18:05:55.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014393899999999999 2024-02-09T18:05:55.560 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.560 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.560 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T18:05:55.560 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998303+0000", 2024-02-09T18:05:55.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2330000000000001e-05 2024-02-09T18:05:55.560 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.560 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.560 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:55.560 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.279571+0000", 2024-02-09T18:05:55.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2812685070000001 2024-02-09T18:05:55.560 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.560 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.561 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:05:55.561 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.283751+0000", 2024-02-09T18:05:55.561 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041799860000000001 2024-02-09T18:05:55.561 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.561 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.561 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-02-09T18:05:55.561 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.283934+0000", 2024-02-09T18:05:55.561 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018285600000000001 2024-02-09T18:05:55.561 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.561 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.561 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:55.561 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.289705+0000", 2024-02-09T18:05:55.561 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0057705450000000002 2024-02-09T18:05:55.562 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.562 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.562 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T18:05:55.562 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.342954+0000", 2024-02-09T18:05:55.562 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.053249059000000001 2024-02-09T18:05:55.562 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.562 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.562 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:55.562 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.343089+0000", 2024-02-09T18:05:55.562 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013504599999999999 2024-02-09T18:05:55.562 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.562 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.562 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:05:55.562 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.343134+0000", 2024-02-09T18:05:55.563 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4849999999999999e-05 2024-02-09T18:05:55.563 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.563 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.563 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:05:55.563 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.344039+0000", 2024-02-09T18:05:55.563 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00090514600000000001 2024-02-09T18:05:55.563 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:55.563 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:05:55.563 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:55.563 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.563 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.563 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26305.0:227497 8.8 8:17781b6a:::benchmark_data_smithi179_148205_object104425:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e855)", 2024-02-09T18:05:55.563 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:59:11.998163+0000", 2024-02-09T18:05:55.564 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 403.553312049, 2024-02-09T18:05:55.564 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3606899770000001, 2024-02-09T18:05:55.564 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:05:55.564 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:55.564 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T18:05:55.564 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26305", 2024-02-09T18:05:55.564 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:05:55.564 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 227497 2024-02-09T18:05:55.564 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.564 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:05:55.564 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.564 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:05:55.564 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998163+0000", 2024-02-09T18:05:55.564 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:55.565 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.565 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.565 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:05:55.565 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998163+0000", 2024-02-09T18:05:55.565 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:55.565 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.565 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.565 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:05:55.565 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998164+0000", 2024-02-09T18:05:55.565 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3900000000000005e-07 2024-02-09T18:05:55.565 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.565 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.565 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:05:55.565 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998167+0000", 2024-02-09T18:05:55.565 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8030000000000001e-06 2024-02-09T18:05:55.566 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.566 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.566 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:05:55.566 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998175+0000", 2024-02-09T18:05:55.566 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4980000000000006e-06 2024-02-09T18:05:55.566 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.566 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.566 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:05:55.566 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998182+0000", 2024-02-09T18:05:55.566 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4819999999999999e-06 2024-02-09T18:05:55.566 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.566 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.566 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:55.566 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998334+0000", 2024-02-09T18:05:55.566 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015232700000000001 2024-02-09T18:05:55.567 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.567 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T18:05:55.567 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998342+0000", 2024-02-09T18:05:55.567 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3960000000000006e-06 2024-02-09T18:05:55.567 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.567 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:55.567 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.334857+0000", 2024-02-09T18:05:55.567 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3365149760000001 2024-02-09T18:05:55.567 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.567 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:05:55.567 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.338705+0000", 2024-02-09T18:05:55.567 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0038474719999999998 2024-02-09T18:05:55.567 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.568 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-02-09T18:05:55.568 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.338874+0000", 2024-02-09T18:05:55.568 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016929600000000001 2024-02-09T18:05:55.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.568 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.568 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T18:05:55.568 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.344097+0000", 2024-02-09T18:05:55.568 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0052226929999999996 2024-02-09T18:05:55.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.568 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.568 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:55.568 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.357933+0000", 2024-02-09T18:05:55.568 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013836372 2024-02-09T18:05:55.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.568 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.568 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:55.569 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.357982+0000", 2024-02-09T18:05:55.569 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9208e-05 2024-02-09T18:05:55.569 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.569 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.569 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:05:55.569 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.358027+0000", 2024-02-09T18:05:55.569 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5120999999999997e-05 2024-02-09T18:05:55.569 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.569 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.569 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:05:55.569 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.358853+0000", 2024-02-09T18:05:55.569 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00082579400000000003 2024-02-09T18:05:55.569 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:55.569 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:05:55.570 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:55.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.570 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.570 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26305.0:227498 8.8 8:110b63aa:::benchmark_data_smithi179_148205_object104426:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e855)", 2024-02-09T18:05:55.570 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:59:11.998189+0000", 2024-02-09T18:05:55.570 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 403.55328655400001, 2024-02-09T18:05:55.570 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3787320059999999, 2024-02-09T18:05:55.570 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:05:55.570 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:55.570 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T18:05:55.570 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26305", 2024-02-09T18:05:55.570 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:05:55.570 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 227498 2024-02-09T18:05:55.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.571 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:05:55.571 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.571 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:05:55.571 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998189+0000", 2024-02-09T18:05:55.571 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:55.571 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.571 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.571 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:05:55.571 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998189+0000", 2024-02-09T18:05:55.571 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:55.571 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.571 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.571 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:05:55.571 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998189+0000", 2024-02-09T18:05:55.572 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.46e-07 2024-02-09T18:05:55.572 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.572 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.572 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:05:55.572 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998191+0000", 2024-02-09T18:05:55.572 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0540000000000002e-06 2024-02-09T18:05:55.572 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.572 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.572 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:05:55.572 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998201+0000", 2024-02-09T18:05:55.572 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4639999999999995e-06 2024-02-09T18:05:55.572 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.572 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.572 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:05:55.572 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998206+0000", 2024-02-09T18:05:55.573 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3850000000000001e-06 2024-02-09T18:05:55.573 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.573 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.573 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:55.573 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998381+0000", 2024-02-09T18:05:55.573 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017492699999999999 2024-02-09T18:05:55.573 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.573 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.573 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T18:05:55.573 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998390+0000", 2024-02-09T18:05:55.573 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2730000000000008e-06 2024-02-09T18:05:55.573 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.573 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.574 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:55.574 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.344127+0000", 2024-02-09T18:05:55.574 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.345736383 2024-02-09T18:05:55.574 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.574 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.574 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:05:55.574 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.344223+0000", 2024-02-09T18:05:55.574 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6124000000000003e-05 2024-02-09T18:05:55.574 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.574 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.574 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-02-09T18:05:55.574 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.344334+0000", 2024-02-09T18:05:55.574 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000110812 2024-02-09T18:05:55.574 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.575 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.575 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T18:05:55.575 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.358863+0000", 2024-02-09T18:05:55.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014529013 2024-02-09T18:05:55.575 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.575 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.575 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:55.575 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.358931+0000", 2024-02-09T18:05:55.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8344999999999997e-05 2024-02-09T18:05:55.575 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.575 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.575 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:55.575 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.376027+0000", 2024-02-09T18:05:55.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.017095986000000001 2024-02-09T18:05:55.576 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.576 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.576 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:05:55.576 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.376064+0000", 2024-02-09T18:05:55.576 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7546999999999998e-05 2024-02-09T18:05:55.576 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.576 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.576 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:05:55.576 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.376921+0000", 2024-02-09T18:05:55.576 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00085624699999999998 2024-02-09T18:05:55.576 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:55.576 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:05:55.576 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:55.576 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.577 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.577 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26305.0:227499 8.8 8:1fe79160:::benchmark_data_smithi179_148205_object104427:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e855)", 2024-02-09T18:05:55.577 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:59:11.998213+0000", 2024-02-09T18:05:55.577 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 403.55326249500001, 2024-02-09T18:05:55.577 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.391633219, 2024-02-09T18:05:55.577 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:05:55.577 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:55.577 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T18:05:55.577 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26305", 2024-02-09T18:05:55.577 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:05:55.577 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 227499 2024-02-09T18:05:55.577 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.577 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:05:55.577 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.577 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:05:55.578 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998213+0000", 2024-02-09T18:05:55.578 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:55.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.578 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.578 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:05:55.578 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998213+0000", 2024-02-09T18:05:55.578 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:55.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.578 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.578 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:05:55.578 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998213+0000", 2024-02-09T18:05:55.578 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.15e-07 2024-02-09T18:05:55.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.578 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.578 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:05:55.579 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998215+0000", 2024-02-09T18:05:55.579 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9709999999999998e-06 2024-02-09T18:05:55.579 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.579 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.579 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:05:55.579 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998221+0000", 2024-02-09T18:05:55.579 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7740000000000001e-06 2024-02-09T18:05:55.579 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.579 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.579 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:05:55.579 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998226+0000", 2024-02-09T18:05:55.579 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3929999999999999e-06 2024-02-09T18:05:55.579 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.579 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.579 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:55.579 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998426+0000", 2024-02-09T18:05:55.579 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019980700000000001 2024-02-09T18:05:55.580 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.580 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.580 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T18:05:55.580 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998431+0000", 2024-02-09T18:05:55.580 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2599999999999996e-06 2024-02-09T18:05:55.580 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.580 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.580 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:55.580 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.347880+0000", 2024-02-09T18:05:55.580 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.34944884 2024-02-09T18:05:55.580 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.580 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.580 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:05:55.580 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.347974+0000", 2024-02-09T18:05:55.581 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3292000000000001e-05 2024-02-09T18:05:55.581 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.581 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.581 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-02-09T18:05:55.581 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.348073+0000", 2024-02-09T18:05:55.581 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9600999999999997e-05 2024-02-09T18:05:55.581 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.581 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.581 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T18:05:55.581 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.358883+0000", 2024-02-09T18:05:55.581 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010810257 2024-02-09T18:05:55.581 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.581 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.582 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:55.582 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.368946+0000", 2024-02-09T18:05:55.582 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010062176000000001 2024-02-09T18:05:55.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.582 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.582 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:55.582 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.388904+0000", 2024-02-09T18:05:55.582 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019958554 2024-02-09T18:05:55.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.582 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.582 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:05:55.582 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.388951+0000", 2024-02-09T18:05:55.582 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7045000000000001e-05 2024-02-09T18:05:55.583 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.583 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.583 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:05:55.583 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.389846+0000", 2024-02-09T18:05:55.583 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00089483399999999995 2024-02-09T18:05:55.583 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:55.583 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:05:55.583 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:55.583 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.583 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.583 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26305.0:227502 8.8 8:11f9033f:::benchmark_data_smithi179_148205_object104430:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e855)", 2024-02-09T18:05:55.583 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:59:11.998243+0000", 2024-02-09T18:05:55.583 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 403.55323242499998, 2024-02-09T18:05:55.583 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3920856269999999, 2024-02-09T18:05:55.584 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:05:55.584 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:55.584 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T18:05:55.584 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26305", 2024-02-09T18:05:55.584 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:05:55.584 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 227502 2024-02-09T18:05:55.584 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.584 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:05:55.584 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.584 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:05:55.584 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998243+0000", 2024-02-09T18:05:55.584 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:55.584 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.584 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.585 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:05:55.585 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998243+0000", 2024-02-09T18:05:55.585 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:55.585 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.585 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.585 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:05:55.585 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998243+0000", 2024-02-09T18:05:55.585 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7599999999999997e-07 2024-02-09T18:05:55.585 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.585 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.585 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:05:55.585 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998253+0000", 2024-02-09T18:05:55.585 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0111e-05 2024-02-09T18:05:55.585 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.585 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.586 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:05:55.586 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998261+0000", 2024-02-09T18:05:55.586 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5020000000000003e-06 2024-02-09T18:05:55.586 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.586 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.586 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:05:55.586 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998267+0000", 2024-02-09T18:05:55.586 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1550000000000004e-06 2024-02-09T18:05:55.586 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.586 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.586 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:55.586 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998464+0000", 2024-02-09T18:05:55.586 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000196417 2024-02-09T18:05:55.586 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.587 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.587 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T18:05:55.587 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998468+0000", 2024-02-09T18:05:55.587 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0799999999999999e-06 2024-02-09T18:05:55.587 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.587 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.587 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:55.587 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.351348+0000", 2024-02-09T18:05:55.587 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.352880117 2024-02-09T18:05:55.587 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.587 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.587 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:05:55.587 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.351468+0000", 2024-02-09T18:05:55.588 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000119954 2024-02-09T18:05:55.588 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.588 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.588 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-02-09T18:05:55.588 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.351557+0000", 2024-02-09T18:05:55.588 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9628000000000003e-05 2024-02-09T18:05:55.588 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.588 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.588 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T18:05:55.588 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.362708+0000", 2024-02-09T18:05:55.588 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011151173 2024-02-09T18:05:55.588 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.588 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.588 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:55.589 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.388864+0000", 2024-02-09T18:05:55.589 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.026155549 2024-02-09T18:05:55.589 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.589 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.589 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:55.589 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.389892+0000", 2024-02-09T18:05:55.589 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0010278010000000001 2024-02-09T18:05:55.589 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.589 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.589 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:05:55.589 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.389931+0000", 2024-02-09T18:05:55.589 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9555000000000003e-05 2024-02-09T18:05:55.589 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.589 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.589 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:05:55.590 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.390328+0000", 2024-02-09T18:05:55.590 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00039710899999999998 2024-02-09T18:05:55.590 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:55.590 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:05:55.590 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:55.590 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.590 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.590 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26305.0:227504 8.0 8:0869e94d:::benchmark_data_smithi179_148205_object104432:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e855)", 2024-02-09T18:05:55.590 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:59:11.998274+0000", 2024-02-09T18:05:55.590 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 403.553201441, 2024-02-09T18:05:55.590 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3462828920000001, 2024-02-09T18:05:55.590 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:05:55.590 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:55.591 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T18:05:55.591 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26305", 2024-02-09T18:05:55.591 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:05:55.591 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 227504 2024-02-09T18:05:55.591 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.591 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:05:55.591 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.591 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:05:55.591 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998274+0000", 2024-02-09T18:05:55.591 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:55.591 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.591 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.591 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:05:55.592 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998274+0000", 2024-02-09T18:05:55.592 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:55.592 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.592 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.592 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:05:55.592 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998274+0000", 2024-02-09T18:05:55.592 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4499999999999997e-07 2024-02-09T18:05:55.592 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.592 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.592 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:05:55.592 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998276+0000", 2024-02-09T18:05:55.592 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1830000000000001e-06 2024-02-09T18:05:55.592 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.592 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.593 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:05:55.593 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998283+0000", 2024-02-09T18:05:55.593 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2319999999999998e-06 2024-02-09T18:05:55.593 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.593 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.593 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:05:55.593 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998288+0000", 2024-02-09T18:05:55.593 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4890000000000003e-06 2024-02-09T18:05:55.593 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.593 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.593 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:55.593 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998445+0000", 2024-02-09T18:05:55.593 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015720299999999999 2024-02-09T18:05:55.593 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.594 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.594 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T18:05:55.594 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998450+0000", 2024-02-09T18:05:55.594 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1570000000000002e-06 2024-02-09T18:05:55.594 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.594 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.594 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:55.594 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.289743+0000", 2024-02-09T18:05:55.594 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.291293096 2024-02-09T18:05:55.594 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.594 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.594 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:05:55.594 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.289902+0000", 2024-02-09T18:05:55.594 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000159829 2024-02-09T18:05:55.595 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.595 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.595 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-02-09T18:05:55.595 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.290008+0000", 2024-02-09T18:05:55.595 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010593499999999999 2024-02-09T18:05:55.595 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.595 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.595 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T18:05:55.595 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.342959+0000", 2024-02-09T18:05:55.595 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.052950127 2024-02-09T18:05:55.595 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.595 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.595 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:55.595 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.343061+0000", 2024-02-09T18:05:55.596 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000102858 2024-02-09T18:05:55.596 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.596 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.596 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:55.596 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.344127+0000", 2024-02-09T18:05:55.596 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001065131 2024-02-09T18:05:55.596 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.596 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.596 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:05:55.596 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.344164+0000", 2024-02-09T18:05:55.596 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7465999999999997e-05 2024-02-09T18:05:55.596 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.596 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.597 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:05:55.597 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.344557+0000", 2024-02-09T18:05:55.597 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000392741 2024-02-09T18:05:55.597 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:55.597 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:05:55.597 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:55.597 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.597 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.597 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26305.0:227518 8.8 8:19d954b1:::benchmark_data_smithi179_148205_object104446:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e855)", 2024-02-09T18:05:55.597 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:59:11.998294+0000", 2024-02-09T18:05:55.597 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 403.55318120700002, 2024-02-09T18:05:55.597 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4160738020000001, 2024-02-09T18:05:55.597 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:05:55.597 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:55.598 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T18:05:55.598 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26305", 2024-02-09T18:05:55.598 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:05:55.598 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 227518 2024-02-09T18:05:55.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.598 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:05:55.598 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.598 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:05:55.598 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998294+0000", 2024-02-09T18:05:55.598 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:55.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.598 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.598 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:05:55.598 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998294+0000", 2024-02-09T18:05:55.598 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:55.599 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.599 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.599 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:05:55.599 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998294+0000", 2024-02-09T18:05:55.599 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0699999999999998e-07 2024-02-09T18:05:55.599 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.599 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.599 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:05:55.599 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998296+0000", 2024-02-09T18:05:55.599 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9740000000000001e-06 2024-02-09T18:05:55.599 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.599 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.599 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:05:55.600 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998303+0000", 2024-02-09T18:05:55.600 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8379999999999996e-06 2024-02-09T18:05:55.600 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.600 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.600 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:05:55.600 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998310+0000", 2024-02-09T18:05:55.600 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4620000000000001e-06 2024-02-09T18:05:55.600 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.600 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.600 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:55.600 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998516+0000", 2024-02-09T18:05:55.600 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020658499999999999 2024-02-09T18:05:55.600 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.601 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.601 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T18:05:55.601 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998521+0000", 2024-02-09T18:05:55.601 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1989999999999999e-06 2024-02-09T18:05:55.601 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.601 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.601 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:55.601 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.354905+0000", 2024-02-09T18:05:55.601 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3563849160000001 2024-02-09T18:05:55.601 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.601 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.601 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:05:55.601 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.354991+0000", 2024-02-09T18:05:55.602 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6057000000000006e-05 2024-02-09T18:05:55.602 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.602 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.602 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-02-09T18:05:55.602 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.355088+0000", 2024-02-09T18:05:55.602 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6674000000000003e-05 2024-02-09T18:05:55.602 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.602 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.602 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T18:05:55.602 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.362712+0000", 2024-02-09T18:05:55.602 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0076242139999999998 2024-02-09T18:05:55.602 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.602 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.603 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:55.603 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.394473+0000", 2024-02-09T18:05:55.603 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.031760268000000001 2024-02-09T18:05:55.603 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.603 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.603 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:55.603 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.413352+0000", 2024-02-09T18:05:55.603 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.01887902 2024-02-09T18:05:55.603 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.603 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.603 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:05:55.603 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.413396+0000", 2024-02-09T18:05:55.603 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4638000000000002e-05 2024-02-09T18:05:55.604 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.604 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.604 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:05:55.604 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.414368+0000", 2024-02-09T18:05:55.604 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00097154999999999995 2024-02-09T18:05:55.604 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:55.604 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:05:55.604 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:55.604 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.604 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.604 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26305.0:227520 8.0 8:04c0cad8:::benchmark_data_smithi179_148205_object104448:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e855)", 2024-02-09T18:05:55.604 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:59:11.998331+0000", 2024-02-09T18:05:55.604 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 403.55314464899999, 2024-02-09T18:05:55.604 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4170492729999999, 2024-02-09T18:05:55.605 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:05:55.605 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:55.605 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T18:05:55.605 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26305", 2024-02-09T18:05:55.605 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:05:55.605 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 227520 2024-02-09T18:05:55.605 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.605 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:05:55.605 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.605 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:05:55.605 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998331+0000", 2024-02-09T18:05:55.605 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:55.605 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.605 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.605 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:05:55.605 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998331+0000", 2024-02-09T18:05:55.606 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:55.606 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.606 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.606 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:05:55.606 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998331+0000", 2024-02-09T18:05:55.606 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2899999999999999e-07 2024-02-09T18:05:55.606 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.606 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.606 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:05:55.606 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998333+0000", 2024-02-09T18:05:55.606 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.407e-06 2024-02-09T18:05:55.606 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.606 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.606 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:05:55.606 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998340+0000", 2024-02-09T18:05:55.607 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6760000000000001e-06 2024-02-09T18:05:55.607 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.607 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.607 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:05:55.607 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998345+0000", 2024-02-09T18:05:55.607 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3070000000000002e-06 2024-02-09T18:05:55.607 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.607 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.607 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:55.607 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998480+0000", 2024-02-09T18:05:55.607 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013433999999999999 2024-02-09T18:05:55.607 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.607 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.607 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T18:05:55.608 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998484+0000", 2024-02-09T18:05:55.608 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0239999999999999e-06 2024-02-09T18:05:55.608 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.608 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.608 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:55.608 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.358906+0000", 2024-02-09T18:05:55.608 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3604217140000001 2024-02-09T18:05:55.608 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.608 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.608 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:05:55.608 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.358995+0000", 2024-02-09T18:05:55.608 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9765000000000005e-05 2024-02-09T18:05:55.608 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.609 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.609 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-02-09T18:05:55.609 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.359083+0000", 2024-02-09T18:05:55.609 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8020000000000001e-05 2024-02-09T18:05:55.609 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.609 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.609 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T18:05:55.609 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.375958+0000", 2024-02-09T18:05:55.609 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.016874364999999999 2024-02-09T18:05:55.609 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.609 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.609 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:55.609 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.397324+0000", 2024-02-09T18:05:55.610 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.021366517000000002 2024-02-09T18:05:55.610 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.610 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.610 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:55.610 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.414315+0000", 2024-02-09T18:05:55.610 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.016990577 2024-02-09T18:05:55.610 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.610 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.611 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:05:55.611 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.414374+0000", 2024-02-09T18:05:55.611 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8862e-05 2024-02-09T18:05:55.611 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.611 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.611 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:05:55.611 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.415380+0000", 2024-02-09T18:05:55.611 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0010062700000000001 2024-02-09T18:05:55.611 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:55.611 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:05:55.611 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:55.611 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.611 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.612 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26305.0:227523 8.0 8:0e3dcc86:::benchmark_data_smithi179_148205_object104451:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e855)", 2024-02-09T18:05:55.612 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:59:11.998373+0000", 2024-02-09T18:05:55.612 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 403.55310202800001, 2024-02-09T18:05:55.612 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.428534019, 2024-02-09T18:05:55.612 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:05:55.612 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:55.612 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T18:05:55.612 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26305", 2024-02-09T18:05:55.612 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:05:55.612 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 227523 2024-02-09T18:05:55.612 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.613 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:05:55.613 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.613 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:05:55.613 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998373+0000", 2024-02-09T18:05:55.613 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:55.613 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.613 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.613 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:05:55.613 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998373+0000", 2024-02-09T18:05:55.613 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:55.613 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.613 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.614 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:05:55.614 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998374+0000", 2024-02-09T18:05:55.614 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4399999999999999e-07 2024-02-09T18:05:55.614 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.614 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.614 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:05:55.614 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998378+0000", 2024-02-09T18:05:55.614 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2910000000000001e-06 2024-02-09T18:05:55.614 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.614 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.614 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:05:55.615 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998386+0000", 2024-02-09T18:05:55.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7309999999999993e-06 2024-02-09T18:05:55.615 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.615 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:05:55.615 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998394+0000", 2024-02-09T18:05:55.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9929999999999995e-06 2024-02-09T18:05:55.615 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.615 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:55.615 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998496+0000", 2024-02-09T18:05:55.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010235500000000001 2024-02-09T18:05:55.615 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.616 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.616 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T18:05:55.616 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:11.998501+0000", 2024-02-09T18:05:55.616 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1740000000000002e-06 2024-02-09T18:05:55.616 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.616 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.616 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:55.616 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.362739+0000", 2024-02-09T18:05:55.616 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.364238831 2024-02-09T18:05:55.616 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.616 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.616 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:05:55.616 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.362859+0000", 2024-02-09T18:05:55.617 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000119603 2024-02-09T18:05:55.617 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.617 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-02-09T18:05:55.617 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.362956+0000", 2024-02-09T18:05:55.617 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6475e-05 2024-02-09T18:05:55.617 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.617 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T18:05:55.617 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.375962+0000", 2024-02-09T18:05:55.617 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013006807 2024-02-09T18:05:55.617 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.617 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:55.617 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.401246+0000", 2024-02-09T18:05:55.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.025283903999999999 2024-02-09T18:05:55.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.618 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:55.618 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.426034+0000", 2024-02-09T18:05:55.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.024787618000000001 2024-02-09T18:05:55.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.618 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:05:55.618 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.426077+0000", 2024-02-09T18:05:55.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2895999999999997e-05 2024-02-09T18:05:55.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.618 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:05:55.619 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.426907+0000", 2024-02-09T18:05:55.619 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00083049700000000003 2024-02-09T18:05:55.619 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:55.619 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:05:55.619 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:55.619 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.619 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.619 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26305.0:227547 8.0 8:013c8224:::benchmark_data_smithi179_148205_object104475:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e855)", 2024-02-09T18:05:55.619 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:59:12.090265+0000", 2024-02-09T18:05:55.619 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 403.46120977200002, 2024-02-09T18:05:55.619 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.337111073, 2024-02-09T18:05:55.619 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:05:55.619 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:55.620 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T18:05:55.620 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26305", 2024-02-09T18:05:55.620 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:05:55.620 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 227547 2024-02-09T18:05:55.620 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.620 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:05:55.620 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.620 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:05:55.620 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:12.090265+0000", 2024-02-09T18:05:55.620 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:55.620 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.620 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.620 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:05:55.620 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:12.090265+0000", 2024-02-09T18:05:55.621 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:55.621 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.621 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.621 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:05:55.621 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:12.090267+0000", 2024-02-09T18:05:55.621 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.415e-06 2024-02-09T18:05:55.621 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.621 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.621 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:05:55.621 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:12.090272+0000", 2024-02-09T18:05:55.621 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1100000000000002e-06 2024-02-09T18:05:55.621 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.621 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.621 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:05:55.621 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:12.090285+0000", 2024-02-09T18:05:55.621 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2544000000000001e-05 2024-02-09T18:05:55.622 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.622 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.622 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:05:55.622 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:12.090291+0000", 2024-02-09T18:05:55.622 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4230000000000002e-06 2024-02-09T18:05:55.622 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.622 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.622 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:55.622 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:12.090352+0000", 2024-02-09T18:05:55.622 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0618000000000001e-05 2024-02-09T18:05:55.622 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.622 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.622 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T18:05:55.622 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:12.090362+0000", 2024-02-09T18:05:55.623 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0407999999999999e-05 2024-02-09T18:05:55.623 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.623 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:55.623 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.365713+0000", 2024-02-09T18:05:55.623 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.275351372 2024-02-09T18:05:55.623 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.623 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:05:55.623 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.365822+0000", 2024-02-09T18:05:55.623 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010898700000000001 2024-02-09T18:05:55.623 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.624 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.624 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-02-09T18:05:55.624 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.365904+0000", 2024-02-09T18:05:55.624 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1748000000000004e-05 2024-02-09T18:05:55.624 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.624 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.624 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T18:05:55.624 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.375966+0000", 2024-02-09T18:05:55.624 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010061965000000001 2024-02-09T18:05:55.624 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.624 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.624 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:55.624 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.401302+0000", 2024-02-09T18:05:55.624 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.025335658 2024-02-09T18:05:55.625 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.625 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.625 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:55.625 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.426937+0000", 2024-02-09T18:05:55.625 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.025635656 2024-02-09T18:05:55.625 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.625 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.625 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:05:55.625 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.426971+0000", 2024-02-09T18:05:55.625 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3812000000000001e-05 2024-02-09T18:05:55.625 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.625 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.626 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:05:55.626 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.427377+0000", 2024-02-09T18:05:55.626 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00040535700000000002 2024-02-09T18:05:55.626 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:55.626 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:05:55.626 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:55.626 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.626 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.626 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26305.0:227549 8.0 8:0c64e541:::benchmark_data_smithi179_148205_object104477:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e855)", 2024-02-09T18:05:55.626 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:59:12.095341+0000", 2024-02-09T18:05:55.626 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 403.456133921, 2024-02-09T18:05:55.626 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3324133790000001, 2024-02-09T18:05:55.627 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:05:55.627 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:55.627 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T18:05:55.627 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26305", 2024-02-09T18:05:55.627 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:05:55.627 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 227549 2024-02-09T18:05:55.627 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.627 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:05:55.627 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.627 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:05:55.627 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:12.095341+0000", 2024-02-09T18:05:55.627 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:55.627 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.628 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.628 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:05:55.628 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:12.095341+0000", 2024-02-09T18:05:55.628 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:55.628 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.628 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.628 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:05:55.628 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:12.095343+0000", 2024-02-09T18:05:55.628 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.268e-06 2024-02-09T18:05:55.628 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.628 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.628 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:05:55.628 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:12.095348+0000", 2024-02-09T18:05:55.628 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0950000000000003e-06 2024-02-09T18:05:55.628 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.629 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.629 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:05:55.629 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:12.095360+0000", 2024-02-09T18:05:55.629 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2500999999999999e-05 2024-02-09T18:05:55.629 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.629 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.629 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:05:55.629 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:12.095366+0000", 2024-02-09T18:05:55.629 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2489999999999998e-06 2024-02-09T18:05:55.629 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.629 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.629 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:55.629 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:12.095424+0000", 2024-02-09T18:05:55.629 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7120000000000002e-05 2024-02-09T18:05:55.630 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.630 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.630 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-02-09T18:05:55.630 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:12.095432+0000", 2024-02-09T18:05:55.630 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2519999999999999e-06 2024-02-09T18:05:55.630 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.630 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.630 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:55.630 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.368942+0000", 2024-02-09T18:05:55.630 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2735106270000001 2024-02-09T18:05:55.630 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.630 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.630 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:05:55.630 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.369044+0000", 2024-02-09T18:05:55.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000101147 2024-02-09T18:05:55.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.631 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-02-09T18:05:55.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.369131+0000", 2024-02-09T18:05:55.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7236000000000003e-05 2024-02-09T18:05:55.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.631 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T18:05:55.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.388857+0000", 2024-02-09T18:05:55.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019726554 2024-02-09T18:05:55.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.631 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:55.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.401330+0000", 2024-02-09T18:05:55.632 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.01247239 2024-02-09T18:05:55.632 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.632 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.632 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:55.632 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.427406+0000", 2024-02-09T18:05:55.632 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.026076535000000001 2024-02-09T18:05:55.632 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.632 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.632 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:05:55.632 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.427440+0000", 2024-02-09T18:05:55.632 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3976000000000001e-05 2024-02-09T18:05:55.632 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.632 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.632 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:05:55.633 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.427755+0000", 2024-02-09T18:05:55.633 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00031442900000000002 2024-02-09T18:05:55.633 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:55.633 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:05:55.633 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:55.633 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.633 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.633 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26305.0:227570 8.0 8:0e211069:::benchmark_data_smithi179_148205_object104498:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e856)", 2024-02-09T18:05:55.633 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:59:13.278761+0000", 2024-02-09T18:05:55.633 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 402.272714366, 2024-02-09T18:05:55.633 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17806154799999999, 2024-02-09T18:05:55.633 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:05:55.633 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:55.633 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T18:05:55.634 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26305", 2024-02-09T18:05:55.634 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:05:55.634 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 227570 2024-02-09T18:05:55.634 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.634 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:05:55.634 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.634 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:05:55.634 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.278761+0000", 2024-02-09T18:05:55.634 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:55.634 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.634 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.634 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:05:55.634 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.278761+0000", 2024-02-09T18:05:55.634 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:55.635 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.635 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.635 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:05:55.635 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.278764+0000", 2024-02-09T18:05:55.635 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6010000000000002e-06 2024-02-09T18:05:55.635 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.635 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.635 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:05:55.635 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.278774+0000", 2024-02-09T18:05:55.635 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0159999999999999e-05 2024-02-09T18:05:55.635 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.635 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.635 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:05:55.635 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.278791+0000", 2024-02-09T18:05:55.635 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7640999999999999e-05 2024-02-09T18:05:55.636 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.636 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.636 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:05:55.636 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.278805+0000", 2024-02-09T18:05:55.636 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3200000000000001e-05 2024-02-09T18:05:55.636 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.636 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.636 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:55.636 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.372129+0000", 2024-02-09T18:05:55.636 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.093324628000000007 2024-02-09T18:05:55.636 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.636 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.636 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:05:55.636 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.372257+0000", 2024-02-09T18:05:55.637 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000128026 2024-02-09T18:05:55.637 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.637 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.637 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-02-09T18:05:55.637 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.372399+0000", 2024-02-09T18:05:55.637 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014223400000000001 2024-02-09T18:05:55.637 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.637 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.637 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T18:05:55.637 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.389856+0000", 2024-02-09T18:05:55.637 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.017457073 2024-02-09T18:05:55.637 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.637 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.637 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:55.638 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.401353+0000", 2024-02-09T18:05:55.638 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011496773 2024-02-09T18:05:55.638 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.638 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.638 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:55.638 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.455813+0000", 2024-02-09T18:05:55.638 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.054460000000000001 2024-02-09T18:05:55.638 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.638 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.638 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:05:55.638 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.455884+0000", 2024-02-09T18:05:55.638 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1088999999999996e-05 2024-02-09T18:05:55.638 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.638 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.639 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:05:55.639 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.456822+0000", 2024-02-09T18:05:55.639 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00093812300000000002 2024-02-09T18:05:55.639 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:55.639 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:05:55.639 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:55.639 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.639 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.639 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26305.0:227586 8.8 8:1d4c5532:::benchmark_data_smithi179_148205_object104514:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e856)", 2024-02-09T18:05:55.639 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:59:13.314490+0000", 2024-02-09T18:05:55.639 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 402.23698484400001, 2024-02-09T18:05:55.639 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.145055872, 2024-02-09T18:05:55.639 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:05:55.639 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:55.640 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T18:05:55.640 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26305", 2024-02-09T18:05:55.640 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:05:55.640 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 227586 2024-02-09T18:05:55.640 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.640 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:05:55.640 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.640 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:05:55.640 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.314490+0000", 2024-02-09T18:05:55.640 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:55.640 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.640 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.640 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:05:55.641 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.314490+0000", 2024-02-09T18:05:55.641 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:55.641 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.641 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.641 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:05:55.641 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.314492+0000", 2024-02-09T18:05:55.641 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4929999999999999e-06 2024-02-09T18:05:55.641 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.641 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.641 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:05:55.641 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.314498+0000", 2024-02-09T18:05:55.641 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7760000000000003e-06 2024-02-09T18:05:55.641 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.641 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.641 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:05:55.641 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.314511+0000", 2024-02-09T18:05:55.642 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3658e-05 2024-02-09T18:05:55.642 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.642 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.642 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:05:55.642 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.314519+0000", 2024-02-09T18:05:55.642 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1149999999999994e-06 2024-02-09T18:05:55.642 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.642 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.642 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:55.642 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.372115+0000", 2024-02-09T18:05:55.642 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.057595814000000002 2024-02-09T18:05:55.642 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.642 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.642 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:05:55.643 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.372221+0000", 2024-02-09T18:05:55.643 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000105466 2024-02-09T18:05:55.643 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.643 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.643 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-02-09T18:05:55.643 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.372323+0000", 2024-02-09T18:05:55.643 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000102568 2024-02-09T18:05:55.643 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.643 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.643 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T18:05:55.643 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.390376+0000", 2024-02-09T18:05:55.643 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018052770999999999 2024-02-09T18:05:55.643 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.643 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.643 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:55.644 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.401293+0000", 2024-02-09T18:05:55.644 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010917296999999999 2024-02-09T18:05:55.644 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.644 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.644 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:55.644 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.458576+0000", 2024-02-09T18:05:55.644 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.057282680000000002 2024-02-09T18:05:55.644 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.644 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.644 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:05:55.644 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.458648+0000", 2024-02-09T18:05:55.644 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2404000000000007e-05 2024-02-09T18:05:55.644 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.644 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.645 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:05:55.645 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.459546+0000", 2024-02-09T18:05:55.645 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00089782999999999996 2024-02-09T18:05:55.645 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:55.645 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:05:55.645 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:55.645 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.645 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.645 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26305.0:227943 8.0 8:08b9b718:::benchmark_data_smithi179_148205_object104871:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e856)", 2024-02-09T18:05:55.645 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:59:13.896905+0000", 2024-02-09T18:05:55.645 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 401.654570466, 2024-02-09T18:05:55.645 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.1365845, 2024-02-09T18:05:55.645 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:05:55.645 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:55.645 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T18:05:55.645 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26305", 2024-02-09T18:05:55.645 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:05:55.646 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 227943 2024-02-09T18:05:55.646 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.646 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:05:55.646 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.646 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:05:55.646 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.896905+0000", 2024-02-09T18:05:55.646 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:55.646 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.646 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.646 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:05:55.646 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.896905+0000", 2024-02-09T18:05:55.646 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:55.646 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.646 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.646 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:05:55.646 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.896906+0000", 2024-02-09T18:05:55.647 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4300000000000002e-07 2024-02-09T18:05:55.647 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.647 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.647 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:05:55.647 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.896909+0000", 2024-02-09T18:05:55.647 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.224e-06 2024-02-09T18:05:55.647 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.647 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.647 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:05:55.647 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.896918+0000", 2024-02-09T18:05:55.647 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1430000000000004e-06 2024-02-09T18:05:55.647 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.647 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.647 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:05:55.647 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.896923+0000", 2024-02-09T18:05:55.647 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2190000000000003e-06 2024-02-09T18:05:55.648 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.648 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.648 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:55.648 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.899842+0000", 2024-02-09T18:05:55.648 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0029192189999999998 2024-02-09T18:05:55.648 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.648 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.648 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:05:55.648 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.899928+0000", 2024-02-09T18:05:55.648 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5589999999999999e-05 2024-02-09T18:05:55.648 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.648 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.648 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-02-09T18:05:55.649 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.900016+0000", 2024-02-09T18:05:55.649 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8403e-05 2024-02-09T18:05:55.649 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.649 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.649 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T18:05:55.649 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.903811+0000", 2024-02-09T18:05:55.649 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0037944570000000002 2024-02-09T18:05:55.649 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.649 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.649 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:55.649 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.905875+0000", 2024-02-09T18:05:55.649 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002063765 2024-02-09T18:05:55.649 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.649 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.650 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:55.650 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:14.032464+0000", 2024-02-09T18:05:55.650 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.126589431 2024-02-09T18:05:55.650 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.650 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.650 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:05:55.650 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:14.032530+0000", 2024-02-09T18:05:55.650 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6026999999999997e-05 2024-02-09T18:05:55.650 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.650 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.650 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:05:55.650 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:14.033489+0000", 2024-02-09T18:05:55.650 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000959179 2024-02-09T18:05:55.650 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:55.651 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:05:55.651 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:55.651 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.651 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.651 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26305.0:227949 8.0 8:09a6da4e:::benchmark_data_smithi179_148205_object104877:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e856)", 2024-02-09T18:05:55.651 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:59:13.906426+0000", 2024-02-09T18:05:55.651 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 401.64504950600002, 2024-02-09T18:05:55.651 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.12770536399999999, 2024-02-09T18:05:55.651 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:05:55.651 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:55.651 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T18:05:55.651 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26305", 2024-02-09T18:05:55.651 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:05:55.652 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 227949 2024-02-09T18:05:55.652 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.652 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:05:55.652 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.652 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:05:55.652 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.906426+0000", 2024-02-09T18:05:55.652 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:55.652 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.652 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.652 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:05:55.652 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.906426+0000", 2024-02-09T18:05:55.652 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:55.652 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.653 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:05:55.653 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.906427+0000", 2024-02-09T18:05:55.653 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8999999999999995e-07 2024-02-09T18:05:55.653 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.653 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:05:55.653 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.906431+0000", 2024-02-09T18:05:55.653 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3989999999999997e-06 2024-02-09T18:05:55.653 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.653 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:05:55.653 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.906440+0000", 2024-02-09T18:05:55.653 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8079999999999994e-06 2024-02-09T18:05:55.654 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.654 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.654 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:05:55.654 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.906445+0000", 2024-02-09T18:05:55.654 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.293e-06 2024-02-09T18:05:55.654 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.654 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.654 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:55.654 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.906683+0000", 2024-02-09T18:05:55.654 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023812099999999999 2024-02-09T18:05:55.654 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.654 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.654 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:05:55.654 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.906779+0000", 2024-02-09T18:05:55.655 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5495000000000006e-05 2024-02-09T18:05:55.655 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.655 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.655 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-02-09T18:05:55.655 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.906876+0000", 2024-02-09T18:05:55.655 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6884000000000003e-05 2024-02-09T18:05:55.655 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.655 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.655 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T18:05:55.655 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.925596+0000", 2024-02-09T18:05:55.655 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018720744000000001 2024-02-09T18:05:55.655 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.655 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.655 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:55.656 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.926609+0000", 2024-02-09T18:05:55.656 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001012317 2024-02-09T18:05:55.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.656 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:55.656 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:14.033593+0000", 2024-02-09T18:05:55.656 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.10698395400000001 2024-02-09T18:05:55.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.656 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:05:55.656 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:14.033652+0000", 2024-02-09T18:05:55.656 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9528999999999999e-05 2024-02-09T18:05:55.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.657 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:05:55.657 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:14.034131+0000", 2024-02-09T18:05:55.657 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00047893000000000002 2024-02-09T18:05:55.657 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:55.657 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:05:55.657 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:55.657 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.657 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.657 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26305.0:227950 8.0 8:0cf3d7e8:::benchmark_data_smithi179_148205_object104878:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e856)", 2024-02-09T18:05:55.657 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:59:13.906456+0000", 2024-02-09T18:05:55.657 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 401.64501886599999, 2024-02-09T18:05:55.657 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.13474096999999999, 2024-02-09T18:05:55.657 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:05:55.657 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:55.658 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T18:05:55.658 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26305", 2024-02-09T18:05:55.658 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:05:55.658 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 227950 2024-02-09T18:05:55.658 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.658 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:05:55.658 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.658 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:05:55.658 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.906456+0000", 2024-02-09T18:05:55.658 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:55.658 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.658 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.658 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:05:55.658 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.906456+0000", 2024-02-09T18:05:55.659 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:55.659 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.659 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.659 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:05:55.659 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.906457+0000", 2024-02-09T18:05:55.659 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5899999999999996e-07 2024-02-09T18:05:55.659 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.659 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.659 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:05:55.659 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.906459+0000", 2024-02-09T18:05:55.659 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3319999999999999e-06 2024-02-09T18:05:55.659 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.659 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.660 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:05:55.660 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.906466+0000", 2024-02-09T18:05:55.660 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2890000000000003e-06 2024-02-09T18:05:55.660 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.660 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.660 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:05:55.660 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.906471+0000", 2024-02-09T18:05:55.660 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3669999999999996e-06 2024-02-09T18:05:55.660 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.660 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.660 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:55.660 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.910602+0000", 2024-02-09T18:05:55.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041307380000000001 2024-02-09T18:05:55.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.661 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:05:55.661 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.910703+0000", 2024-02-09T18:05:55.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010165700000000001 2024-02-09T18:05:55.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.661 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-02-09T18:05:55.661 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.910801+0000", 2024-02-09T18:05:55.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7428000000000003e-05 2024-02-09T18:05:55.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.661 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T18:05:55.661 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.926629+0000", 2024-02-09T18:05:55.662 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015827964 2024-02-09T18:05:55.662 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.662 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.662 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:55.662 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.926649+0000", 2024-02-09T18:05:55.662 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0571999999999999e-05 2024-02-09T18:05:55.662 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.662 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.662 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:55.662 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:14.040043+0000", 2024-02-09T18:05:55.662 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.113393962 2024-02-09T18:05:55.662 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.662 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.662 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:05:55.663 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:14.040099+0000", 2024-02-09T18:05:55.663 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5903999999999999e-05 2024-02-09T18:05:55.663 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.663 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.663 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:05:55.663 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:14.041197+0000", 2024-02-09T18:05:55.663 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0010981980000000001 2024-02-09T18:05:55.663 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:55.663 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:05:55.663 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:55.663 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.663 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.663 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26305.0:227952 8.8 8:101d55b4:::benchmark_data_smithi179_148205_object104880:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e856)", 2024-02-09T18:05:55.663 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:59:13.906476+0000", 2024-02-09T18:05:55.663 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 401.64499924699999, 2024-02-09T18:05:55.663 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.13472445499999999, 2024-02-09T18:05:55.664 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:05:55.664 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:55.664 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T18:05:55.664 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26305", 2024-02-09T18:05:55.664 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:05:55.664 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 227952 2024-02-09T18:05:55.664 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.664 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:05:55.664 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.664 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:05:55.664 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.906476+0000", 2024-02-09T18:05:55.664 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:55.664 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.664 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.664 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:05:55.665 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.906476+0000", 2024-02-09T18:05:55.665 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:55.665 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.665 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.665 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:05:55.665 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.906476+0000", 2024-02-09T18:05:55.665 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1e-07 2024-02-09T18:05:55.665 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.665 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.665 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:05:55.665 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.906478+0000", 2024-02-09T18:05:55.665 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.421e-06 2024-02-09T18:05:55.665 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.665 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.666 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:05:55.666 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.906482+0000", 2024-02-09T18:05:55.666 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.228e-06 2024-02-09T18:05:55.666 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.666 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.666 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:05:55.666 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.906485+0000", 2024-02-09T18:05:55.666 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9189999999999999e-06 2024-02-09T18:05:55.666 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.666 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.666 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:55.666 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.910571+0000", 2024-02-09T18:05:55.666 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0040860740000000003 2024-02-09T18:05:55.666 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.666 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.667 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:05:55.667 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.910679+0000", 2024-02-09T18:05:55.667 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010851799999999999 2024-02-09T18:05:55.667 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.667 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.667 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-02-09T18:05:55.667 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.910788+0000", 2024-02-09T18:05:55.667 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000108999 2024-02-09T18:05:55.667 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.667 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.667 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:55.667 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.926579+0000", 2024-02-09T18:05:55.667 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015790379 2024-02-09T18:05:55.667 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.667 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.668 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T18:05:55.668 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:13.926638+0000", 2024-02-09T18:05:55.668 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8916999999999999e-05 2024-02-09T18:05:55.668 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.668 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.668 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:55.668 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:14.040063+0000", 2024-02-09T18:05:55.668 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.11342529 2024-02-09T18:05:55.668 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.668 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.668 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:05:55.668 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:14.040131+0000", 2024-02-09T18:05:55.668 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8147999999999999e-05 2024-02-09T18:05:55.668 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.669 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.669 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:05:55.669 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:14.041200+0000", 2024-02-09T18:05:55.669 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0010692519999999999 2024-02-09T18:05:55.669 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:55.669 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:05:55.669 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:55.669 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.669 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.669 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26305.0:236421 8.8 8:1bb471e3:::benchmark_data_smithi179_148205_object113349:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e865)", 2024-02-09T18:05:55.669 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:59:28.869314+0000", 2024-02-09T18:05:55.669 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 386.68216088200001, 2024-02-09T18:05:55.669 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.120203579, 2024-02-09T18:05:55.669 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:05:55.669 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:55.670 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T18:05:55.670 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26305", 2024-02-09T18:05:55.670 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:05:55.670 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 236421 2024-02-09T18:05:55.670 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.670 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:05:55.670 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.670 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:05:55.670 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:28.869314+0000", 2024-02-09T18:05:55.670 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:55.670 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.670 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.670 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:05:55.670 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:28.869314+0000", 2024-02-09T18:05:55.670 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:55.671 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.671 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.671 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:05:55.671 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:28.869316+0000", 2024-02-09T18:05:55.671 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.474e-06 2024-02-09T18:05:55.671 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.671 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.671 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:05:55.671 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:28.869322+0000", 2024-02-09T18:05:55.671 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0650000000000004e-06 2024-02-09T18:05:55.671 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.671 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.671 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:05:55.672 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:28.869337+0000", 2024-02-09T18:05:55.672 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4585e-05 2024-02-09T18:05:55.672 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.672 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.672 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:05:55.672 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:28.869345+0000", 2024-02-09T18:05:55.672 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0709999999999995e-06 2024-02-09T18:05:55.672 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.672 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.672 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:55.672 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:28.877798+0000", 2024-02-09T18:05:55.672 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0084532649999999997 2024-02-09T18:05:55.672 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.673 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.673 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:05:55.673 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:28.877910+0000", 2024-02-09T18:05:55.673 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000111686 2024-02-09T18:05:55.673 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.673 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.673 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-02-09T18:05:55.673 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:28.878014+0000", 2024-02-09T18:05:55.673 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010464899999999999 2024-02-09T18:05:55.673 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.673 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.673 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T18:05:55.673 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:28.895472+0000", 2024-02-09T18:05:55.673 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.017457534 2024-02-09T18:05:55.673 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.674 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.674 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:55.674 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:28.898746+0000", 2024-02-09T18:05:55.674 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0032738580000000001 2024-02-09T18:05:55.674 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.674 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.674 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:55.674 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:28.988701+0000", 2024-02-09T18:05:55.674 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.089955607000000007 2024-02-09T18:05:55.674 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.674 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.674 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:05:55.674 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:28.988771+0000", 2024-02-09T18:05:55.674 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0055999999999995e-05 2024-02-09T18:05:55.674 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.675 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.675 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:05:55.675 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:28.989518+0000", 2024-02-09T18:05:55.675 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00074672900000000001 2024-02-09T18:05:55.675 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:55.675 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:05:55.675 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:55.675 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.675 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.675 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26305.0:236423 8.8 8:1f059922:::benchmark_data_smithi179_148205_object113351:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e865)", 2024-02-09T18:05:55.675 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:59:28.869360+0000", 2024-02-09T18:05:55.675 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 386.68211519300002, 2024-02-09T18:05:55.676 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.12081482, 2024-02-09T18:05:55.676 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:05:55.676 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:55.676 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T18:05:55.676 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26305", 2024-02-09T18:05:55.676 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:05:55.676 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 236423 2024-02-09T18:05:55.676 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.676 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:05:55.676 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.676 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:05:55.676 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:28.869360+0000", 2024-02-09T18:05:55.676 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:55.676 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.676 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.677 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:05:55.677 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:28.869360+0000", 2024-02-09T18:05:55.677 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:55.677 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.677 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.677 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:05:55.677 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:28.869361+0000", 2024-02-09T18:05:55.677 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.51e-07 2024-02-09T18:05:55.677 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.677 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.677 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:05:55.677 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:28.869363+0000", 2024-02-09T18:05:55.677 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5100000000000001e-06 2024-02-09T18:05:55.678 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.678 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.678 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:05:55.678 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:28.869371+0000", 2024-02-09T18:05:55.678 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6799999999999993e-06 2024-02-09T18:05:55.678 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.678 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.678 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:05:55.678 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:28.869376+0000", 2024-02-09T18:05:55.678 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5929999999999996e-06 2024-02-09T18:05:55.678 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.678 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.678 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:55.678 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:28.882152+0000", 2024-02-09T18:05:55.679 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012775715999999999 2024-02-09T18:05:55.679 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.679 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.679 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:05:55.679 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:28.882316+0000", 2024-02-09T18:05:55.679 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016431599999999999 2024-02-09T18:05:55.679 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.679 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.679 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-02-09T18:05:55.679 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:28.882479+0000", 2024-02-09T18:05:55.679 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000162721 2024-02-09T18:05:55.679 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.679 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.679 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T18:05:55.680 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:28.898756+0000", 2024-02-09T18:05:55.680 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.016276848 2024-02-09T18:05:55.680 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.680 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.680 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:55.680 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:28.898789+0000", 2024-02-09T18:05:55.680 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3492999999999999e-05 2024-02-09T18:05:55.680 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.680 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.680 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:55.680 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:28.989570+0000", 2024-02-09T18:05:55.680 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.090780195999999994 2024-02-09T18:05:55.680 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.681 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.681 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:05:55.681 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:28.989634+0000", 2024-02-09T18:05:55.681 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4428e-05 2024-02-09T18:05:55.681 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.681 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.681 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:05:55.681 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:28.990175+0000", 2024-02-09T18:05:55.681 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00054086800000000001 2024-02-09T18:05:55.681 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:55.681 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:05:55.681 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:55.681 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.681 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.682 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26305.0:236436 8.8 8:1e75a49b:::benchmark_data_smithi179_148205_object113364:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e865)", 2024-02-09T18:05:55.682 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:59:28.882883+0000", 2024-02-09T18:05:55.682 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 386.66859199999999, 2024-02-09T18:05:55.682 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.122588117, 2024-02-09T18:05:55.682 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:05:55.682 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:05:55.682 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-02-09T18:05:55.682 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26305", 2024-02-09T18:05:55.682 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:05:55.682 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 236436 2024-02-09T18:05:55.682 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.682 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:05:55.682 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.682 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:05:55.682 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:28.882883+0000", 2024-02-09T18:05:55.683 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:55.683 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.683 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.683 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:05:55.683 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:28.882883+0000", 2024-02-09T18:05:55.683 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:05:55.683 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.683 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.683 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:05:55.683 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:28.882884+0000", 2024-02-09T18:05:55.683 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0400000000000002e-07 2024-02-09T18:05:55.683 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.683 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.683 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:05:55.684 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:28.882886+0000", 2024-02-09T18:05:55.684 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2929999999999999e-06 2024-02-09T18:05:55.684 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.684 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.684 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:05:55.684 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:28.882893+0000", 2024-02-09T18:05:55.684 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5390000000000004e-06 2024-02-09T18:05:55.684 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.684 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.684 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:05:55.684 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:28.882898+0000", 2024-02-09T18:05:55.684 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1939999999999997e-06 2024-02-09T18:05:55.684 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.684 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.685 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:05:55.685 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:28.898819+0000", 2024-02-09T18:05:55.685 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015921083999999999 2024-02-09T18:05:55.685 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.685 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.685 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:05:55.685 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:28.898910+0000", 2024-02-09T18:05:55.685 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1528999999999997e-05 2024-02-09T18:05:55.685 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.685 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.685 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-02-09T18:05:55.685 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:28.899008+0000", 2024-02-09T18:05:55.685 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7707000000000005e-05 2024-02-09T18:05:55.685 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.686 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.686 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-02-09T18:05:55.686 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:28.906004+0000", 2024-02-09T18:05:55.686 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0069960159999999999 2024-02-09T18:05:55.686 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.686 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.686 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:55.686 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:28.916816+0000", 2024-02-09T18:05:55.686 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010811955999999999 2024-02-09T18:05:55.686 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.686 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.686 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:05:55.686 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:29.004545+0000", 2024-02-09T18:05:55.686 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.087729381999999995 2024-02-09T18:05:55.687 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.687 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.687 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:05:55.687 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:29.004600+0000", 2024-02-09T18:05:55.687 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4802999999999997e-05 2024-02-09T18:05:55.687 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:05:55.687 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:05:55.687 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:05:55.687 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:29.005471+0000", 2024-02-09T18:05:55.687 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00087120999999999997 2024-02-09T18:05:55.687 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:55.687 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:05:55.687 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:55.687 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:05:55.687 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:05:55.688 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T18:05:55.917 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:55 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/763620765' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-02-09T18:05:55.917 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:55 smithi162 ceph-mon[44304]: osdmap e1009: 12 total, 12 up, 12 in 2024-02-09T18:05:55.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:55 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/763620765' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-02-09T18:05:55.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:55 smithi179 ceph-mon[43975]: osdmap e1009: 12 total, 12 up, 12 in 2024-02-09T18:05:56.217 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-09T18:05:56.487 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T18:05:56.860 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:56 smithi099 ceph-mon[39420]: pgmap v2484: 31 pgs: 31 active+clean; 577 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:56.860 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:56 smithi099 ceph-mon[39420]: 1.16 scrub starts 2024-02-09T18:05:56.861 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:56 smithi099 ceph-mon[39420]: 1.16 scrub ok 2024-02-09T18:05:56.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:56 smithi179 ceph-mon[43975]: pgmap v2484: 31 pgs: 31 active+clean; 577 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:56.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:56 smithi179 ceph-mon[43975]: 1.16 scrub starts 2024-02-09T18:05:56.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:56 smithi179 ceph-mon[43975]: 1.16 scrub ok 2024-02-09T18:05:56.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:56 smithi162 ceph-mon[44304]: pgmap v2484: 31 pgs: 31 active+clean; 577 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:56.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:56 smithi162 ceph-mon[44304]: 1.16 scrub starts 2024-02-09T18:05:56.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:56 smithi162 ceph-mon[44304]: 1.16 scrub ok 2024-02-09T18:05:58.257 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T18:05:58.257 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-09T18:05:58.257 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-02-09T18:05:58.257 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T18:05:58.566 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.5 to 1 (10000) 2024-02-09T18:05:58.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:58 smithi179 ceph-mon[43975]: pgmap v2485: 31 pgs: 31 active+clean; 577 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:58.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:58 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3245370628' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-09T18:05:58.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:58 smithi162 ceph-mon[44304]: pgmap v2485: 31 pgs: 31 active+clean; 577 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:58.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:58 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3245370628' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-09T18:05:58.961 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-09T18:05:58.978 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:58 smithi099 ceph-mon[39420]: pgmap v2485: 31 pgs: 31 active+clean; 577 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:05:58.978 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:58 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3245370628' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-09T18:05:59.228 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T18:05:59.381 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd primary-affinity 5 1 2024-02-09T18:05:59.650 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:05:59.740 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:59 smithi162 ceph-mon[44304]: 1.7 scrub starts 2024-02-09T18:05:59.740 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:59 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3245370628' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-09T18:05:59.740 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:59 smithi162 ceph-mon[44304]: osdmap e1010: 12 total, 12 up, 12 in 2024-02-09T18:05:59.741 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:05:59 smithi162 ceph-mon[44304]: 1.7 scrub ok 2024-02-09T18:05:59.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:59 smithi099 ceph-mon[39420]: 1.7 scrub starts 2024-02-09T18:05:59.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:59 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3245370628' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-09T18:05:59.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:59 smithi099 ceph-mon[39420]: osdmap e1010: 12 total, 12 up, 12 in 2024-02-09T18:05:59.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:05:59 smithi099 ceph-mon[39420]: 1.7 scrub ok 2024-02-09T18:05:59.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:59 smithi179 ceph-mon[43975]: 1.7 scrub starts 2024-02-09T18:05:59.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:59 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3245370628' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-09T18:05:59.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:59 smithi179 ceph-mon[43975]: osdmap e1010: 12 total, 12 up, 12 in 2024-02-09T18:05:59.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:05:59 smithi179 ceph-mon[43975]: 1.7 scrub ok 2024-02-09T18:06:00.674 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:00 smithi162 ceph-mon[44304]: pgmap v2487: 31 pgs: 31 active+clean; 577 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:00.674 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:00 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:06:00.674 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:00 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:06:00.674 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:00 smithi162 ceph-mon[44304]: osdmap e1011: 12 total, 12 up, 12 in 2024-02-09T18:06:00.836 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:00 smithi099 ceph-mon[39420]: pgmap v2487: 31 pgs: 31 active+clean; 577 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:00.836 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:00 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:06:00.836 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:00 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:06:00.837 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:00 smithi099 ceph-mon[39420]: osdmap e1011: 12 total, 12 up, 12 in 2024-02-09T18:06:00.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:00 smithi179 ceph-mon[43975]: pgmap v2487: 31 pgs: 31 active+clean; 577 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:00.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:00 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:06:00.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:00 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:06:00.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:00 smithi179 ceph-mon[43975]: osdmap e1011: 12 total, 12 up, 12 in 2024-02-09T18:06:00.979 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T18:06:00.980 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-09T18:06:00.980 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-02-09T18:06:00.980 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-02-09T18:06:00.980 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T18:06:01.579 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-09T18:06:01.844 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T18:06:02.580 INFO:teuthology.orchestra.run.smithi099.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-02-09T18:06:02.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:02 smithi179 ceph-mon[43975]: pgmap v2489: 31 pgs: 31 active+clean; 577 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:02.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:02 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:06:02.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:02 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:06:02.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:02 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:06:02.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:02 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:06:02.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:02 smithi179 ceph-mon[43975]: osdmap e1012: 12 total, 12 up, 12 in 2024-02-09T18:06:02.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:02 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4082134127' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-02-09T18:06:02.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:02 smithi162 ceph-mon[44304]: pgmap v2489: 31 pgs: 31 active+clean; 577 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:02.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:02 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:06:02.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:02 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:06:02.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:02 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:06:02.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:02 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:06:02.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:02 smithi162 ceph-mon[44304]: osdmap e1012: 12 total, 12 up, 12 in 2024-02-09T18:06:02.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:02 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4082134127' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-02-09T18:06:02.945 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:02 smithi099 ceph-mon[39420]: pgmap v2489: 31 pgs: 31 active+clean; 577 KiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:02.945 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:02 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:06:02.945 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:02 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:06:02.945 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:02 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:06:02.945 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:02 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:06:02.946 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:02 smithi099 ceph-mon[39420]: osdmap e1012: 12 total, 12 up, 12 in 2024-02-09T18:06:02.946 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:02 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4082134127' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-02-09T18:06:03.359 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 6 1 2024-02-09T18:06:03.582 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:03 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4082134127' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-02-09T18:06:03.582 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:03 smithi099 ceph-mon[39420]: osdmap e1013: 12 total, 12 up, 12 in 2024-02-09T18:06:03.582 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:03 smithi099 ceph-mon[39420]: osdmap e1014: 12 total, 12 up, 12 in 2024-02-09T18:06:03.629 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:06:03.719 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T18:06:03.720 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-02-09T18:06:03.720 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-02-09T18:06:03.720 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-02-09T18:06:03.720 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.720 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "rep_scrubmap(1.f e866 from shard 3)", 2024-02-09T18:06:03.720 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T17:59:30.394793+0000", 2024-02-09T18:06:03.720 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 393.32209363800001, 2024-02-09T18:06:03.720 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.089808922999999999, 2024-02-09T18:06:03.720 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:06:03.720 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-09T18:06:03.720 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:06:03.721 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.721 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:06:03.721 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:30.394793+0000", 2024-02-09T18:06:03.721 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:03.721 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.721 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.721 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:06:03.721 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:30.394793+0000", 2024-02-09T18:06:03.721 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:03.721 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.721 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.721 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:06:03.721 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:30.394794+0000", 2024-02-09T18:06:03.721 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.569999999999999e-07 2024-02-09T18:06:03.722 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:06:03.722 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:30.394798+0000", 2024-02-09T18:06:03.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.476e-06 2024-02-09T18:06:03.722 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:06:03.722 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:30.394808+0000", 2024-02-09T18:06:03.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0057e-05 2024-02-09T18:06:03.722 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:06:03.722 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:30.394815+0000", 2024-02-09T18:06:03.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9639999999999997e-06 2024-02-09T18:06:03.723 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.723 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.723 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:06:03.723 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:30.484570+0000", 2024-02-09T18:06:03.723 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.089755366000000003 2024-02-09T18:06:03.723 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.723 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.723 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:06:03.723 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:30.484584+0000", 2024-02-09T18:06:03.723 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3692999999999999e-05 2024-02-09T18:06:03.723 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.723 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.723 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:06:03.723 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T17:59:30.484601+0000", 2024-02-09T18:06:03.724 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7410000000000001e-05 2024-02-09T18:06:03.724 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:03.724 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:06:03.724 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:03.724 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.724 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.724 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:159929 9.1 e926/882 9:889430ee:::benchmark_data_smithi179_161915_object25047:head v 926'9923, mlcod=926'9921)", 2024-02-09T18:06:03.724 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:02:20.586831+0000", 2024-02-09T18:06:03.724 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 223.13005495900001, 2024-02-09T18:06:03.724 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.021476810999999998, 2024-02-09T18:06:03.724 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:06:03.724 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:03.724 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:06:03.724 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:06:03.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.586831+0000", 2024-02-09T18:06:03.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:03.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:06:03.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.586831+0000", 2024-02-09T18:06:03.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:03.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:06:03.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.586833+0000", 2024-02-09T18:06:03.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5859999999999999e-06 2024-02-09T18:06:03.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.726 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.726 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:06:03.726 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.586843+0000", 2024-02-09T18:06:03.726 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7750000000000004e-06 2024-02-09T18:06:03.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.726 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.726 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:06:03.726 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.586861+0000", 2024-02-09T18:06:03.726 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8411e-05 2024-02-09T18:06:03.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.726 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.726 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:06:03.726 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.586872+0000", 2024-02-09T18:06:03.727 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.062e-05 2024-02-09T18:06:03.727 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.727 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.727 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:06:03.727 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.586936+0000", 2024-02-09T18:06:03.727 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4778999999999998e-05 2024-02-09T18:06:03.727 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.727 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.727 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:06:03.727 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.586953+0000", 2024-02-09T18:06:03.727 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.662e-05 2024-02-09T18:06:03.727 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.727 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.727 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:06:03.727 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.608262+0000", 2024-02-09T18:06:03.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.02130895 2024-02-09T18:06:03.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.728 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.728 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:06:03.728 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.608308+0000", 2024-02-09T18:06:03.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6069999999999998e-05 2024-02-09T18:06:03.728 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:03.728 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:06:03.728 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:03.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.728 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.728 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:159930 9.1 e926/882 9:8c309051:::benchmark_data_smithi179_161915_object25048:head v 926'9924, mlcod=926'9921)", 2024-02-09T18:06:03.728 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:02:20.586894+0000", 2024-02-09T18:06:03.728 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 223.12999249800001, 2024-02-09T18:06:03.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.021444139000000001, 2024-02-09T18:06:03.729 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:06:03.729 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:03.729 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:06:03.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:06:03.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.586894+0000", 2024-02-09T18:06:03.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:03.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:06:03.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.586894+0000", 2024-02-09T18:06:03.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:03.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:06:03.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.586894+0000", 2024-02-09T18:06:03.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7000000000000001e-07 2024-02-09T18:06:03.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.730 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:06:03.730 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.586898+0000", 2024-02-09T18:06:03.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1489999999999996e-06 2024-02-09T18:06:03.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.730 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:06:03.730 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.586906+0000", 2024-02-09T18:06:03.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5050000000000001e-06 2024-02-09T18:06:03.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.730 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:06:03.730 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.586910+0000", 2024-02-09T18:06:03.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4399999999999998e-06 2024-02-09T18:06:03.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:06:03.731 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.591042+0000", 2024-02-09T18:06:03.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041318070000000004 2024-02-09T18:06:03.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:06:03.731 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.591063+0000", 2024-02-09T18:06:03.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0721e-05 2024-02-09T18:06:03.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:06:03.731 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.608312+0000", 2024-02-09T18:06:03.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.017249536999999999 2024-02-09T18:06:03.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:06:03.732 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:20.608338+0000", 2024-02-09T18:06:03.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5709999999999999e-05 2024-02-09T18:06:03.732 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:03.732 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:06:03.732 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:03.732 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.732 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "rep_scrubmap(1.f e866 from shard 3)", 2024-02-09T18:06:03.732 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:02:46.279015+0000", 2024-02-09T18:06:03.732 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 197.43787106799999, 2024-02-09T18:06:03.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.079179055999999998, 2024-02-09T18:06:03.732 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:06:03.732 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-09T18:06:03.732 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:06:03.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:06:03.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:46.279015+0000", 2024-02-09T18:06:03.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:03.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:06:03.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:46.279015+0000", 2024-02-09T18:06:03.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:03.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:06:03.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:46.279016+0000", 2024-02-09T18:06:03.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1800000000000005e-07 2024-02-09T18:06:03.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:06:03.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:46.279021+0000", 2024-02-09T18:06:03.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4190000000000001e-06 2024-02-09T18:06:03.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:06:03.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:46.279032+0000", 2024-02-09T18:06:03.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0434999999999999e-05 2024-02-09T18:06:03.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:06:03.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:46.279039+0000", 2024-02-09T18:06:03.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4649999999999997e-06 2024-02-09T18:06:03.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:06:03.735 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:46.358165+0000", 2024-02-09T18:06:03.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.079126009999999997 2024-02-09T18:06:03.735 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.735 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.735 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:06:03.735 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:46.358178+0000", 2024-02-09T18:06:03.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.313e-05 2024-02-09T18:06:03.735 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.735 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.735 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:06:03.735 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:02:46.358194+0000", 2024-02-09T18:06:03.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5778999999999999e-05 2024-02-09T18:06:03.735 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:03.735 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:06:03.735 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:03.735 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.736 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.736 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:203400 9.0 e944/882 9:00923123:::benchmark_data_smithi179_161915_object68518:head v 944'12686, mlcod=944'12684)", 2024-02-09T18:06:03.736 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:03:16.200019+0000", 2024-02-09T18:06:03.736 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 167.51686694599999, 2024-02-09T18:06:03.736 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020633133000000001, 2024-02-09T18:06:03.736 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:06:03.736 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:03.736 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:06:03.736 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.736 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:06:03.736 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:16.200019+0000", 2024-02-09T18:06:03.736 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:03.736 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.736 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.737 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:06:03.737 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:16.200019+0000", 2024-02-09T18:06:03.737 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:03.737 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.737 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.737 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:06:03.737 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:16.200021+0000", 2024-02-09T18:06:03.737 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7659999999999999e-06 2024-02-09T18:06:03.737 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.737 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.737 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:06:03.737 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:16.200031+0000", 2024-02-09T18:06:03.737 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6700000000000006e-06 2024-02-09T18:06:03.737 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.737 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:06:03.738 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:16.200057+0000", 2024-02-09T18:06:03.738 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5884000000000002e-05 2024-02-09T18:06:03.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:06:03.738 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:16.200066+0000", 2024-02-09T18:06:03.738 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5889999999999992e-06 2024-02-09T18:06:03.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:06:03.738 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:16.202694+0000", 2024-02-09T18:06:03.738 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002627912 2024-02-09T18:06:03.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.739 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:06:03.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:16.202717+0000", 2024-02-09T18:06:03.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2606e-05 2024-02-09T18:06:03.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.739 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:06:03.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:16.220607+0000", 2024-02-09T18:06:03.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.017890321000000001 2024-02-09T18:06:03.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.739 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:06:03.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:16.220652+0000", 2024-02-09T18:06:03.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5385e-05 2024-02-09T18:06:03.739 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:03.739 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:06:03.740 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:03.740 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.740 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.740 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:205928 9.0 e947/882 9:0b7699e7:::benchmark_data_smithi179_161915_object71046:head v 947'12844, mlcod=947'12841)", 2024-02-09T18:06:03.740 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:03:20.002336+0000", 2024-02-09T18:06:03.740 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 163.714550395, 2024-02-09T18:06:03.740 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.022585653000000001, 2024-02-09T18:06:03.740 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:06:03.740 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:03.740 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:06:03.740 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.740 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:06:03.740 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:20.002336+0000", 2024-02-09T18:06:03.740 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:03.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:06:03.741 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:20.002336+0000", 2024-02-09T18:06:03.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:03.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:06:03.741 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:20.002336+0000", 2024-02-09T18:06:03.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7700000000000001e-07 2024-02-09T18:06:03.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:06:03.741 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:20.002339+0000", 2024-02-09T18:06:03.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0149999999999999e-06 2024-02-09T18:06:03.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:06:03.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:20.002343+0000", 2024-02-09T18:06:03.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6540000000000001e-06 2024-02-09T18:06:03.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:06:03.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:20.002346+0000", 2024-02-09T18:06:03.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8040000000000002e-06 2024-02-09T18:06:03.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:06:03.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:20.009581+0000", 2024-02-09T18:06:03.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0072349950000000001 2024-02-09T18:06:03.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.743 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:06:03.743 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:20.009607+0000", 2024-02-09T18:06:03.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6241000000000002e-05 2024-02-09T18:06:03.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.743 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:06:03.743 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:20.024873+0000", 2024-02-09T18:06:03.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015266221 2024-02-09T18:06:03.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.743 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:06:03.743 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:03:20.024921+0000", 2024-02-09T18:06:03.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8446000000000003e-05 2024-02-09T18:06:03.743 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:03.744 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:06:03.744 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:03.744 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.744 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.744 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:238015 9.0 e966/882 9:0a983c2a:::benchmark_data_smithi179_161915_object103133:head v 966'14811, mlcod=966'14807)", 2024-02-09T18:06:03.744 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:04:04.488521+0000", 2024-02-09T18:06:03.744 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 119.22836568699999, 2024-02-09T18:06:03.744 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.023487556, 2024-02-09T18:06:03.744 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:06:03.744 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:03.744 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:06:03.744 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.744 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:06:03.744 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:04.488521+0000", 2024-02-09T18:06:03.744 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:03.745 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.745 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.745 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:06:03.745 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:04.488521+0000", 2024-02-09T18:06:03.745 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:03.745 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.745 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.745 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:06:03.745 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:04.488521+0000", 2024-02-09T18:06:03.745 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2699999999999998e-07 2024-02-09T18:06:03.745 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.745 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.745 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:06:03.745 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:04.488531+0000", 2024-02-09T18:06:03.745 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4849999999999998e-06 2024-02-09T18:06:03.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.746 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:06:03.746 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:04.488545+0000", 2024-02-09T18:06:03.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.436e-05 2024-02-09T18:06:03.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.746 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:06:03.746 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:04.488553+0000", 2024-02-09T18:06:03.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3560000000000004e-06 2024-02-09T18:06:03.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.746 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:06:03.746 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:04.488606+0000", 2024-02-09T18:06:03.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.363e-05 2024-02-09T18:06:03.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.747 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:06:03.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:04.488627+0000", 2024-02-09T18:06:03.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0293e-05 2024-02-09T18:06:03.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.747 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:06:03.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:04.511975+0000", 2024-02-09T18:06:03.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.023348145000000001 2024-02-09T18:06:03.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.747 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:06:03.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:04.512008+0000", 2024-02-09T18:06:03.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3359999999999999e-05 2024-02-09T18:06:03.748 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:03.748 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:06:03.748 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:03.748 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.748 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.748 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:238016 9.0 e966/882 9:0e38db32:::benchmark_data_smithi179_161915_object103134:head v 966'14812, mlcod=966'14807)", 2024-02-09T18:06:03.748 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:04:04.488574+0000", 2024-02-09T18:06:03.748 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 119.22831259100001, 2024-02-09T18:06:03.748 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.023468826000000002, 2024-02-09T18:06:03.748 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:06:03.748 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:03.748 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:06:03.748 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.748 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:06:03.748 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:04.488574+0000", 2024-02-09T18:06:03.749 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:03.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.749 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.749 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:06:03.749 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:04.488574+0000", 2024-02-09T18:06:03.749 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:03.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.749 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.749 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:06:03.749 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:04.488574+0000", 2024-02-09T18:06:03.749 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2399999999999999e-07 2024-02-09T18:06:03.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.749 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.749 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:06:03.749 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:04.488578+0000", 2024-02-09T18:06:03.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6519999999999999e-06 2024-02-09T18:06:03.750 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.750 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:06:03.750 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:04.488584+0000", 2024-02-09T18:06:03.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9739999999999999e-06 2024-02-09T18:06:03.750 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.750 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:06:03.750 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:04.488588+0000", 2024-02-09T18:06:03.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2220000000000004e-06 2024-02-09T18:06:03.750 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.750 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:06:03.750 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:04.493532+0000", 2024-02-09T18:06:03.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0049446860000000002 2024-02-09T18:06:03.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:06:03.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:04.493551+0000", 2024-02-09T18:06:03.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8915000000000001e-05 2024-02-09T18:06:03.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:06:03.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:04.512015+0000", 2024-02-09T18:06:03.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018463403 2024-02-09T18:06:03.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:06:03.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:04.512042+0000", 2024-02-09T18:06:03.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7650000000000001e-05 2024-02-09T18:06:03.752 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:03.752 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:06:03.752 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:03.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.752 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:238019 9.0 e966/882 9:0e0b3ea2:::benchmark_data_smithi179_161915_object103137:head v 966'14813, mlcod=966'14807)", 2024-02-09T18:06:03.752 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:04:04.490943+0000", 2024-02-09T18:06:03.752 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 119.225943512, 2024-02-09T18:06:03.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.021130339000000001, 2024-02-09T18:06:03.752 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:06:03.752 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:03.752 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:06:03.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:06:03.753 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:04.490943+0000", 2024-02-09T18:06:03.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:03.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.753 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:06:03.753 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:04.490943+0000", 2024-02-09T18:06:03.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:03.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.753 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:06:03.753 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:04.490944+0000", 2024-02-09T18:06:03.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4900000000000005e-07 2024-02-09T18:06:03.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.753 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:06:03.754 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:04.490952+0000", 2024-02-09T18:06:03.754 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7979999999999995e-06 2024-02-09T18:06:03.754 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.754 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.754 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:06:03.754 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:04.490963+0000", 2024-02-09T18:06:03.754 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1113e-05 2024-02-09T18:06:03.754 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.754 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.754 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:06:03.754 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:04.490970+0000", 2024-02-09T18:06:03.754 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5409999999999997e-06 2024-02-09T18:06:03.754 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.754 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.754 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:06:03.754 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:04.498447+0000", 2024-02-09T18:06:03.755 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0074773089999999997 2024-02-09T18:06:03.755 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.755 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.755 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:06:03.755 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:04.498468+0000", 2024-02-09T18:06:03.755 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0627000000000001e-05 2024-02-09T18:06:03.755 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.755 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.755 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:06:03.755 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:04.512046+0000", 2024-02-09T18:06:03.755 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.01357849 2024-02-09T18:06:03.755 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.755 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.755 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:06:03.755 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:04.512073+0000", 2024-02-09T18:06:03.755 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6611999999999999e-05 2024-02-09T18:06:03.756 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:03.756 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:06:03.756 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:03.756 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.756 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.756 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:239274 9.1 e966/882 9:8e4035ee:::benchmark_data_smithi179_161915_object104392:head v 966'14840, mlcod=966'14837)", 2024-02-09T18:06:03.756 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:04:05.994035+0000", 2024-02-09T18:06:03.756 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 117.722851627, 2024-02-09T18:06:03.756 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.021505494999999999, 2024-02-09T18:06:03.756 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:06:03.756 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:03.756 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:06:03.756 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.756 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:06:03.757 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:05.994035+0000", 2024-02-09T18:06:03.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:03.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:06:03.757 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:05.994035+0000", 2024-02-09T18:06:03.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:03.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:06:03.757 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:05.994035+0000", 2024-02-09T18:06:03.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5700000000000002e-07 2024-02-09T18:06:03.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:06:03.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:05.994041+0000", 2024-02-09T18:06:03.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7270000000000004e-06 2024-02-09T18:06:03.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.758 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:06:03.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:05.994052+0000", 2024-02-09T18:06:03.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0964999999999999e-05 2024-02-09T18:06:03.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.758 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:06:03.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:05.994059+0000", 2024-02-09T18:06:03.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2699999999999999e-06 2024-02-09T18:06:03.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.758 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:06:03.759 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:06.001533+0000", 2024-02-09T18:06:03.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0074735870000000003 2024-02-09T18:06:03.759 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.759 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:06:03.759 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:06.001552+0000", 2024-02-09T18:06:03.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9607999999999999e-05 2024-02-09T18:06:03.759 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.759 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:06:03.759 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:06.015493+0000", 2024-02-09T18:06:03.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013940576999999999 2024-02-09T18:06:03.759 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.760 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:06:03.760 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:06.015540+0000", 2024-02-09T18:06:03.760 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7104000000000002e-05 2024-02-09T18:06:03.760 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:03.760 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:06:03.760 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:03.760 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.760 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.760 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:241135 9.0 e966/882 9:0edd9c51:::benchmark_data_smithi179_161915_object106253:head v 966'15011, mlcod=966'15008)", 2024-02-09T18:06:03.760 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:04:08.202414+0000", 2024-02-09T18:06:03.760 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 115.51447260400001, 2024-02-09T18:06:03.760 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.023479771999999999, 2024-02-09T18:06:03.760 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:06:03.760 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:03.761 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:06:03.761 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.761 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:06:03.761 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:08.202414+0000", 2024-02-09T18:06:03.761 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:03.761 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.761 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.761 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:06:03.761 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:08.202414+0000", 2024-02-09T18:06:03.761 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:03.761 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.761 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.761 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:06:03.761 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:08.202414+0000", 2024-02-09T18:06:03.761 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9400000000000004e-07 2024-02-09T18:06:03.762 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.762 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.762 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:06:03.762 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:08.202422+0000", 2024-02-09T18:06:03.762 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0589999999999997e-06 2024-02-09T18:06:03.762 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.762 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.762 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:06:03.762 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:08.202437+0000", 2024-02-09T18:06:03.762 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5594999999999999e-05 2024-02-09T18:06:03.762 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.762 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.762 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:06:03.762 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:08.202445+0000", 2024-02-09T18:06:03.762 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0069999999999997e-06 2024-02-09T18:06:03.763 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.763 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:06:03.763 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:08.202514+0000", 2024-02-09T18:06:03.763 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9055999999999997e-05 2024-02-09T18:06:03.763 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.763 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:06:03.763 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:08.202533+0000", 2024-02-09T18:06:03.763 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9100999999999999e-05 2024-02-09T18:06:03.763 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.763 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:06:03.763 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:08.225858+0000", 2024-02-09T18:06:03.763 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.023324685000000001 2024-02-09T18:06:03.763 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.764 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:06:03.764 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:08.225893+0000", 2024-02-09T18:06:03.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5374999999999999e-05 2024-02-09T18:06:03.764 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:03.764 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:06:03.764 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:03.764 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.764 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.764 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:241146 9.0 e966/882 9:0cf0a79a:::benchmark_data_smithi179_161915_object106264:head v 966'15012, mlcod=966'15009)", 2024-02-09T18:06:03.764 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:04:08.202707+0000", 2024-02-09T18:06:03.764 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 115.514179067, 2024-02-09T18:06:03.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.023206004999999998, 2024-02-09T18:06:03.764 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:06:03.764 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:03.765 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:06:03.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:06:03.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:08.202707+0000", 2024-02-09T18:06:03.765 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:03.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:06:03.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:08.202707+0000", 2024-02-09T18:06:03.765 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:03.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:06:03.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:08.202708+0000", 2024-02-09T18:06:03.765 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5199999999999996e-07 2024-02-09T18:06:03.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.766 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:06:03.766 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:08.202717+0000", 2024-02-09T18:06:03.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2399999999999996e-06 2024-02-09T18:06:03.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.766 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:06:03.766 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:08.202728+0000", 2024-02-09T18:06:03.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0659000000000001e-05 2024-02-09T18:06:03.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.766 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:06:03.766 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:08.202734+0000", 2024-02-09T18:06:03.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5679999999999996e-06 2024-02-09T18:06:03.767 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.767 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.767 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:06:03.767 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:08.206981+0000", 2024-02-09T18:06:03.767 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042470490000000001 2024-02-09T18:06:03.767 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.767 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.767 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:06:03.767 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:08.207004+0000", 2024-02-09T18:06:03.767 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2524999999999999e-05 2024-02-09T18:06:03.767 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.767 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.767 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:06:03.767 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:08.225896+0000", 2024-02-09T18:06:03.768 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018891702 2024-02-09T18:06:03.768 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.768 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.768 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:06:03.768 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:08.225913+0000", 2024-02-09T18:06:03.768 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.751e-05 2024-02-09T18:06:03.768 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:03.768 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:06:03.768 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:03.768 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.768 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.768 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:242560 9.1 e966/882 9:8e41da34:::benchmark_data_smithi179_161915_object107678:head v 966'15031, mlcod=966'15028)", 2024-02-09T18:06:03.768 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:04:09.877685+0000", 2024-02-09T18:06:03.768 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 113.839200971, 2024-02-09T18:06:03.769 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.022413642000000001, 2024-02-09T18:06:03.769 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:06:03.769 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:03.769 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:06:03.769 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.769 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:06:03.769 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:09.877685+0000", 2024-02-09T18:06:03.769 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:03.769 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.769 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.769 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:06:03.769 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:09.877685+0000", 2024-02-09T18:06:03.769 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:03.769 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.769 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:06:03.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:09.877686+0000", 2024-02-09T18:06:03.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1099999999999995e-07 2024-02-09T18:06:03.770 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:06:03.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:09.877691+0000", 2024-02-09T18:06:03.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.429e-06 2024-02-09T18:06:03.770 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:06:03.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:09.877702+0000", 2024-02-09T18:06:03.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0651e-05 2024-02-09T18:06:03.770 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:06:03.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:09.877708+0000", 2024-02-09T18:06:03.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1290000000000001e-06 2024-02-09T18:06:03.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.771 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:06:03.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:09.884278+0000", 2024-02-09T18:06:03.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0065700209999999997 2024-02-09T18:06:03.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.771 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:06:03.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:09.884298+0000", 2024-02-09T18:06:03.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0188999999999999e-05 2024-02-09T18:06:03.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.771 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:06:03.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:09.900059+0000", 2024-02-09T18:06:03.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015760374000000001 2024-02-09T18:06:03.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.772 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.772 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:06:03.772 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:09.900099+0000", 2024-02-09T18:06:03.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0238000000000003e-05 2024-02-09T18:06:03.772 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:03.772 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:06:03.772 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:03.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.772 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.772 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:260591 9.0 e972/882 9:0e9dbf6a:::benchmark_data_smithi179_161915_object125709:head v 972'16241, mlcod=972'16238)", 2024-02-09T18:06:03.772 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:04:31.465032+0000", 2024-02-09T18:06:03.772 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 92.251854175000005, 2024-02-09T18:06:03.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020742837, 2024-02-09T18:06:03.773 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:06:03.773 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:03.773 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:06:03.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.773 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:06:03.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:31.465032+0000", 2024-02-09T18:06:03.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:03.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.773 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:06:03.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:31.465032+0000", 2024-02-09T18:06:03.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:03.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.774 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:06:03.774 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:31.465033+0000", 2024-02-09T18:06:03.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4e-07 2024-02-09T18:06:03.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.774 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.774 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:06:03.774 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:31.465039+0000", 2024-02-09T18:06:03.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.224e-06 2024-02-09T18:06:03.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.774 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.774 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:06:03.774 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:31.465049+0000", 2024-02-09T18:06:03.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0198e-05 2024-02-09T18:06:03.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.774 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.775 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:06:03.775 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:31.465057+0000", 2024-02-09T18:06:03.775 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9719999999999992e-06 2024-02-09T18:06:03.775 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.775 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.775 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:06:03.775 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:31.468591+0000", 2024-02-09T18:06:03.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0035339899999999999 2024-02-09T18:06:03.776 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.776 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.776 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:06:03.776 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:31.468616+0000", 2024-02-09T18:06:03.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4258999999999999e-05 2024-02-09T18:06:03.776 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.776 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.776 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:06:03.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:31.485730+0000", 2024-02-09T18:06:03.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.017114111000000001 2024-02-09T18:06:03.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:06:03.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:31.485775+0000", 2024-02-09T18:06:03.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5243e-05 2024-02-09T18:06:03.777 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:03.777 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:06:03.777 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:03.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.777 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:260620 9.0 e972/882 9:0c6c42fd:::benchmark_data_smithi179_161915_object125738:head v 972'16244, mlcod=972'16240)", 2024-02-09T18:06:03.777 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:04:31.499399+0000", 2024-02-09T18:06:03.778 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 92.217486789000006, 2024-02-09T18:06:03.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.026584535999999999, 2024-02-09T18:06:03.778 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:06:03.778 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:03.778 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:06:03.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:06:03.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:31.499399+0000", 2024-02-09T18:06:03.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:03.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:06:03.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:31.499399+0000", 2024-02-09T18:06:03.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:03.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.779 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:06:03.779 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:31.499401+0000", 2024-02-09T18:06:03.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4500000000000001e-06 2024-02-09T18:06:03.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.779 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:06:03.779 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:31.499408+0000", 2024-02-09T18:06:03.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0380000000000002e-06 2024-02-09T18:06:03.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.779 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:06:03.779 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:31.499419+0000", 2024-02-09T18:06:03.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.129e-05 2024-02-09T18:06:03.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.780 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:06:03.780 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:31.499427+0000", 2024-02-09T18:06:03.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9149999999999996e-06 2024-02-09T18:06:03.780 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.780 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.780 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:06:03.780 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:31.502280+0000", 2024-02-09T18:06:03.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0028525540000000002 2024-02-09T18:06:03.780 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.780 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.780 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:06:03.780 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:31.502296+0000", 2024-02-09T18:06:03.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5845000000000002e-05 2024-02-09T18:06:03.780 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.780 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.781 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:06:03.781 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:31.525936+0000", 2024-02-09T18:06:03.781 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.023640960999999999 2024-02-09T18:06:03.781 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.781 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:06:03.781 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:31.525984+0000", 2024-02-09T18:06:03.781 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7482999999999999e-05 2024-02-09T18:06:03.781 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:03.781 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:06:03.781 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:03.781 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.781 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:260621 9.0 e972/882 9:070c7f36:::benchmark_data_smithi179_161915_object125739:head v 972'16245, mlcod=972'16240)", 2024-02-09T18:06:03.781 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:04:31.503858+0000", 2024-02-09T18:06:03.781 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 92.213027987999993, 2024-02-09T18:06:03.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.022156420999999999, 2024-02-09T18:06:03.782 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:06:03.782 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:03.782 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:06:03.782 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.782 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:06:03.782 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:31.503858+0000", 2024-02-09T18:06:03.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:03.782 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.782 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.782 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:06:03.782 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:31.503858+0000", 2024-02-09T18:06:03.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:03.782 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.782 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.783 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:06:03.783 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:31.503859+0000", 2024-02-09T18:06:03.783 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.08e-06 2024-02-09T18:06:03.783 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.783 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:06:03.783 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:31.503868+0000", 2024-02-09T18:06:03.783 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4149999999999999e-06 2024-02-09T18:06:03.783 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.783 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:06:03.783 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:31.503881+0000", 2024-02-09T18:06:03.783 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3754000000000001e-05 2024-02-09T18:06:03.783 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.783 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:06:03.783 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:31.503891+0000", 2024-02-09T18:06:03.784 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9150000000000008e-06 2024-02-09T18:06:03.784 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.784 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.784 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:06:03.784 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:31.506162+0000", 2024-02-09T18:06:03.784 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0022708289999999998 2024-02-09T18:06:03.784 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.784 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.784 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:06:03.784 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:31.506175+0000", 2024-02-09T18:06:03.784 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2746e-05 2024-02-09T18:06:03.784 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.784 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.784 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:06:03.784 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:31.525989+0000", 2024-02-09T18:06:03.784 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019814050999999999 2024-02-09T18:06:03.785 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.785 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.785 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:06:03.785 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:31.526015+0000", 2024-02-09T18:06:03.785 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5630999999999999e-05 2024-02-09T18:06:03.785 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:03.785 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:06:03.785 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:03.785 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.785 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.785 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:261777 9.0 e973/882 9:0244f94e:::benchmark_data_smithi179_161915_object126895:head v 973'16312, mlcod=973'16309)", 2024-02-09T18:06:03.785 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:04:32.894109+0000", 2024-02-09T18:06:03.785 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 90.822777236999997, 2024-02-09T18:06:03.785 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.023563245999999999, 2024-02-09T18:06:03.785 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:06:03.786 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:03.786 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:06:03.786 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.786 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:06:03.786 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:32.894109+0000", 2024-02-09T18:06:03.786 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:03.786 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.786 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.786 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:06:03.786 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:32.894109+0000", 2024-02-09T18:06:03.786 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:03.786 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.786 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.786 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:06:03.786 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:32.894110+0000", 2024-02-09T18:06:03.786 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7499999999999999e-07 2024-02-09T18:06:03.787 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.787 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:06:03.787 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:32.894118+0000", 2024-02-09T18:06:03.787 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8089999999999993e-06 2024-02-09T18:06:03.787 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.787 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:06:03.787 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:32.894130+0000", 2024-02-09T18:06:03.787 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2503999999999999e-05 2024-02-09T18:06:03.787 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.787 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:06:03.787 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:32.894138+0000", 2024-02-09T18:06:03.787 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3819999999999998e-06 2024-02-09T18:06:03.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.788 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:06:03.788 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:32.894188+0000", 2024-02-09T18:06:03.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0661999999999997e-05 2024-02-09T18:06:03.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.788 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:06:03.788 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:32.894210+0000", 2024-02-09T18:06:03.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1486000000000001e-05 2024-02-09T18:06:03.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.788 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:06:03.788 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:32.917623+0000", 2024-02-09T18:06:03.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.023413521999999999 2024-02-09T18:06:03.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.789 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:06:03.789 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:32.917672+0000", 2024-02-09T18:06:03.789 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9005999999999998e-05 2024-02-09T18:06:03.789 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:03.789 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:06:03.789 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:03.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.789 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:261781 9.0 e973/882 9:07dac638:::benchmark_data_smithi179_161915_object126899:head v 973'16313, mlcod=973'16309)", 2024-02-09T18:06:03.789 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:04:32.894293+0000", 2024-02-09T18:06:03.789 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 90.822592940000007, 2024-02-09T18:06:03.789 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.023414043999999998, 2024-02-09T18:06:03.789 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:06:03.789 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:03.790 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:06:03.790 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.790 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:06:03.790 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:32.894293+0000", 2024-02-09T18:06:03.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:03.790 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.790 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.790 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:06:03.790 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:32.894293+0000", 2024-02-09T18:06:03.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:03.790 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.790 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.790 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:06:03.790 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:32.894294+0000", 2024-02-09T18:06:03.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.89e-07 2024-02-09T18:06:03.790 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:06:03.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:32.894299+0000", 2024-02-09T18:06:03.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1970000000000004e-06 2024-02-09T18:06:03.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:06:03.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:32.894308+0000", 2024-02-09T18:06:03.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0229999999999998e-06 2024-02-09T18:06:03.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:06:03.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:32.894314+0000", 2024-02-09T18:06:03.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9370000000000001e-06 2024-02-09T18:06:03.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.792 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:06:03.792 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:32.902829+0000", 2024-02-09T18:06:03.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0085145900000000007 2024-02-09T18:06:03.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.792 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:06:03.792 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:32.902852+0000", 2024-02-09T18:06:03.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3862e-05 2024-02-09T18:06:03.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.792 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:06:03.792 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:32.917678+0000", 2024-02-09T18:06:03.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014826097999999999 2024-02-09T18:06:03.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.793 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:06:03.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:32.917707+0000", 2024-02-09T18:06:03.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8847999999999999e-05 2024-02-09T18:06:03.793 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:03.793 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:06:03.793 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:03.793 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.793 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.793 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:263776 9.1 e974/882 9:8649d415:::benchmark_data_smithi179_161915_object128894:head v 974'16326, mlcod=974'16322)", 2024-02-09T18:06:03.793 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:04:35.470309+0000", 2024-02-09T18:06:03.793 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 88.246577442000003, 2024-02-09T18:06:03.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.022347972000000001, 2024-02-09T18:06:03.793 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:06:03.793 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:03.793 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:06:03.793 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.794 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:06:03.794 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:35.470309+0000", 2024-02-09T18:06:03.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:03.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.794 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:06:03.794 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:35.470309+0000", 2024-02-09T18:06:03.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:03.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.794 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:06:03.794 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:35.470309+0000", 2024-02-09T18:06:03.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2099999999999998e-07 2024-02-09T18:06:03.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.795 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:06:03.795 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:35.470313+0000", 2024-02-09T18:06:03.795 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3209999999999998e-06 2024-02-09T18:06:03.795 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.795 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.795 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:06:03.795 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:35.470319+0000", 2024-02-09T18:06:03.795 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7450000000000001e-06 2024-02-09T18:06:03.795 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.795 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.795 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:06:03.795 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:35.470324+0000", 2024-02-09T18:06:03.795 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6110000000000004e-06 2024-02-09T18:06:03.795 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.795 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.796 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:06:03.796 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:35.474161+0000", 2024-02-09T18:06:03.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0038374989999999999 2024-02-09T18:06:03.796 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.796 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.796 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:06:03.796 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:35.474183+0000", 2024-02-09T18:06:03.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2061000000000001e-05 2024-02-09T18:06:03.796 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.796 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.796 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:06:03.796 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:35.492610+0000", 2024-02-09T18:06:03.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018426221 2024-02-09T18:06:03.796 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.796 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.797 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:06:03.797 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:35.492657+0000", 2024-02-09T18:06:03.797 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7193e-05 2024-02-09T18:06:03.797 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:03.797 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:06:03.797 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:03.797 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.797 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.797 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:268255 9.1 e976/882 9:8e1bfb4e:::benchmark_data_smithi179_161915_object133373:head v 976'16588, mlcod=976'16585)", 2024-02-09T18:06:03.797 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:04:40.699545+0000", 2024-02-09T18:06:03.797 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 83.017341684000002, 2024-02-09T18:06:03.797 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.022087895999999999, 2024-02-09T18:06:03.797 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:06:03.797 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:03.797 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:06:03.798 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.798 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:06:03.798 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:40.699545+0000", 2024-02-09T18:06:03.798 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:03.798 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.798 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.798 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:06:03.798 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:40.699545+0000", 2024-02-09T18:06:03.798 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:03.798 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.798 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.798 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:06:03.798 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:40.699545+0000", 2024-02-09T18:06:03.798 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4700000000000001e-07 2024-02-09T18:06:03.798 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.799 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:06:03.799 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:40.699553+0000", 2024-02-09T18:06:03.799 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3810000000000001e-06 2024-02-09T18:06:03.799 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.799 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:06:03.799 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:40.699565+0000", 2024-02-09T18:06:03.799 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2156e-05 2024-02-09T18:06:03.799 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.799 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:06:03.799 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:40.699572+0000", 2024-02-09T18:06:03.799 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1319999999999996e-06 2024-02-09T18:06:03.799 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.800 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.800 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:06:03.800 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:40.703311+0000", 2024-02-09T18:06:03.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0037390539999999999 2024-02-09T18:06:03.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.800 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.800 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:06:03.800 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:40.703336+0000", 2024-02-09T18:06:03.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4598000000000001e-05 2024-02-09T18:06:03.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.800 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.800 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:06:03.800 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:40.721589+0000", 2024-02-09T18:06:03.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018253469000000001 2024-02-09T18:06:03.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:03.801 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:03.801 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:06:03.801 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:04:40.721632+0000", 2024-02-09T18:06:03.801 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3359000000000003e-05 2024-02-09T18:06:03.801 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:03.801 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:06:03.801 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:03.801 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:03.801 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:06:03.801 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T18:06:03.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:03 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4082134127' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-02-09T18:06:03.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:03 smithi162 ceph-mon[44304]: osdmap e1013: 12 total, 12 up, 12 in 2024-02-09T18:06:03.802 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:03 smithi162 ceph-mon[44304]: osdmap e1014: 12 total, 12 up, 12 in 2024-02-09T18:06:03.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:03 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4082134127' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-02-09T18:06:03.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:03 smithi179 ceph-mon[43975]: osdmap e1013: 12 total, 12 up, 12 in 2024-02-09T18:06:03.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:03 smithi179 ceph-mon[43975]: osdmap e1014: 12 total, 12 up, 12 in 2024-02-09T18:06:04.449 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-09T18:06:04.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:04 smithi162 ceph-mon[44304]: pgmap v2492: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:04.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:04 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:06:04.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:04 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:06:04.720 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T18:06:04.876 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:04 smithi099 ceph-mon[39420]: pgmap v2492: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:04.876 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:04 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:06:04.876 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:04 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:06:04.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:04 smithi179 ceph-mon[43975]: pgmap v2492: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:04.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:04 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:06:04.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:04 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:06:05.598 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.6 to 1 (10000) 2024-02-09T18:06:05.912 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:05 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:06:05.912 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:05 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:06:05.912 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:05 smithi162 ceph-mon[44304]: osdmap e1015: 12 total, 12 up, 12 in 2024-02-09T18:06:05.912 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:05 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1301094972' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-09T18:06:05.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:05 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:06:05.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:05 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:06:05.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:05 smithi179 ceph-mon[43975]: osdmap e1015: 12 total, 12 up, 12 in 2024-02-09T18:06:05.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:05 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1301094972' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-09T18:06:05.935 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:05 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:06:05.935 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:05 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:06:05.935 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:05 smithi099 ceph-mon[39420]: osdmap e1015: 12 total, 12 up, 12 in 2024-02-09T18:06:05.935 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:05 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1301094972' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-09T18:06:06.379 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd primary-affinity 6 1 2024-02-09T18:06:06.535 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T18:06:06.536 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-09T18:06:06.536 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-02-09T18:06:06.536 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T18:06:06.646 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:06:06.896 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:06 smithi162 ceph-mon[44304]: pgmap v2495: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:06.897 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:06 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1301094972' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-09T18:06:06.897 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:06 smithi162 ceph-mon[44304]: osdmap e1016: 12 total, 12 up, 12 in 2024-02-09T18:06:06.905 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:06 smithi099 ceph-mon[39420]: pgmap v2495: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:06.905 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:06 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1301094972' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-09T18:06:06.905 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:06 smithi099 ceph-mon[39420]: osdmap e1016: 12 total, 12 up, 12 in 2024-02-09T18:06:06.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:06 smithi179 ceph-mon[43975]: pgmap v2495: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:06.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:06 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1301094972' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-09T18:06:06.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:06 smithi179 ceph-mon[43975]: osdmap e1016: 12 total, 12 up, 12 in 2024-02-09T18:06:07.159 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-09T18:06:07.430 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T18:06:08.198 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:08 smithi099 ceph-mon[39420]: pgmap v2497: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:08.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:08 smithi179 ceph-mon[43975]: pgmap v2497: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:08.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:08 smithi162 ceph-mon[44304]: pgmap v2497: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:09.168 INFO:teuthology.orchestra.run.smithi099.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-02-09T18:06:09.176 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T18:06:09.176 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-02-09T18:06:09.176 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-02-09T18:06:09.176 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-02-09T18:06:09.176 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T18:06:09.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:09 smithi179 ceph-mon[43975]: 1.a scrub starts 2024-02-09T18:06:09.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:09 smithi179 ceph-mon[43975]: 1.a scrub ok 2024-02-09T18:06:09.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:09 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3521969324' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-02-09T18:06:09.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:09 smithi162 ceph-mon[44304]: 1.a scrub starts 2024-02-09T18:06:09.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:09 smithi162 ceph-mon[44304]: 1.a scrub ok 2024-02-09T18:06:09.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:09 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3521969324' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-02-09T18:06:09.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:09 smithi099 ceph-mon[39420]: 1.a scrub starts 2024-02-09T18:06:09.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:09 smithi099 ceph-mon[39420]: 1.a scrub ok 2024-02-09T18:06:09.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:09 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3521969324' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-02-09T18:06:09.885 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-09T18:06:10.032 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 7 1 2024-02-09T18:06:10.153 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.b/config 2024-02-09T18:06:10.175 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:10 smithi162 ceph-mon[44304]: 1.19 scrub starts 2024-02-09T18:06:10.175 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:10 smithi162 ceph-mon[44304]: 1.19 scrub ok 2024-02-09T18:06:10.175 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:10 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3521969324' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-02-09T18:06:10.302 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:06:10.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:10 smithi162 ceph-mon[44304]: osdmap e1017: 12 total, 12 up, 12 in 2024-02-09T18:06:10.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:10 smithi162 ceph-mon[44304]: pgmap v2499: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:10.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:10 smithi099 ceph-mon[39420]: 1.19 scrub starts 2024-02-09T18:06:10.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:10 smithi099 ceph-mon[39420]: 1.19 scrub ok 2024-02-09T18:06:10.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:10 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3521969324' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-02-09T18:06:10.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:10 smithi099 ceph-mon[39420]: osdmap e1017: 12 total, 12 up, 12 in 2024-02-09T18:06:10.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:10 smithi099 ceph-mon[39420]: pgmap v2499: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:10.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:10 smithi179 ceph-mon[43975]: 1.19 scrub starts 2024-02-09T18:06:10.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:10 smithi179 ceph-mon[43975]: 1.19 scrub ok 2024-02-09T18:06:10.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:10 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3521969324' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-02-09T18:06:10.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:10 smithi179 ceph-mon[43975]: osdmap e1017: 12 total, 12 up, 12 in 2024-02-09T18:06:10.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:10 smithi179 ceph-mon[43975]: pgmap v2499: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:12.096 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-02-09T18:06:12.096 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-02-09T18:06:12.096 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-02-09T18:06:12.096 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-02-09T18:06:12.096 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.096 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "rep_scrubmap(1.d e866 from shard 1)", 2024-02-09T18:06:12.096 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:00:49.780916+0000", 2024-02-09T18:06:12.096 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 322.31192892600001, 2024-02-09T18:06:12.096 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.094998210999999999, 2024-02-09T18:06:12.096 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:06:12.096 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-09T18:06:12.097 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:06:12.097 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.097 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:06:12.097 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:00:49.780916+0000", 2024-02-09T18:06:12.097 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:12.097 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.097 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.097 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:06:12.097 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:00:49.780916+0000", 2024-02-09T18:06:12.097 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:12.097 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.097 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.097 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:06:12.097 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:00:49.780917+0000", 2024-02-09T18:06:12.097 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.78e-07 2024-02-09T18:06:12.098 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.098 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.098 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:06:12.098 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:00:49.780920+0000", 2024-02-09T18:06:12.098 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8589999999999998e-06 2024-02-09T18:06:12.098 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.098 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.098 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:06:12.098 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:00:49.780930+0000", 2024-02-09T18:06:12.098 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8849999999999994e-06 2024-02-09T18:06:12.098 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.098 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.098 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:06:12.098 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:00:49.780936+0000", 2024-02-09T18:06:12.099 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2140000000000004e-06 2024-02-09T18:06:12.099 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.099 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.099 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:06:12.099 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:00:49.875881+0000", 2024-02-09T18:06:12.099 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.094945422000000002 2024-02-09T18:06:12.099 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.099 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.099 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:06:12.099 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:00:49.875895+0000", 2024-02-09T18:06:12.099 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4153e-05 2024-02-09T18:06:12.099 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.099 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.099 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:06:12.100 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:00:49.875914+0000", 2024-02-09T18:06:12.100 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8700000000000001e-05 2024-02-09T18:06:12.100 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:12.100 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:06:12.100 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:12.100 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.100 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.100 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:67570 9.5 e900/882 9:ac315c6c:::benchmark_data_smithi179_161915_object67569:head v 900'4301, mlcod=900'4299)", 2024-02-09T18:06:12.100 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:00:49.815945+0000", 2024-02-09T18:06:12.100 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 322.27689983599998, 2024-02-09T18:06:12.100 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.066651540999999995, 2024-02-09T18:06:12.100 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:06:12.100 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:12.100 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:06:12.101 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.101 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:06:12.101 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:00:49.815945+0000", 2024-02-09T18:06:12.101 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:12.101 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.101 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.101 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:06:12.101 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:00:49.815945+0000", 2024-02-09T18:06:12.101 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:12.101 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.101 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.101 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:06:12.101 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:00:49.815946+0000", 2024-02-09T18:06:12.101 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6199999999999996e-07 2024-02-09T18:06:12.101 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.102 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.102 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:06:12.102 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:00:49.816026+0000", 2024-02-09T18:06:12.102 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0327999999999994e-05 2024-02-09T18:06:12.102 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.102 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.102 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:06:12.102 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:00:49.816039+0000", 2024-02-09T18:06:12.102 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.349e-05 2024-02-09T18:06:12.102 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.102 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.102 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:06:12.102 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:00:49.816046+0000", 2024-02-09T18:06:12.102 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0700000000000001e-06 2024-02-09T18:06:12.103 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.103 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.103 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:06:12.103 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:00:49.875945+0000", 2024-02-09T18:06:12.103 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.059899023000000003 2024-02-09T18:06:12.103 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.103 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.103 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:06:12.103 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:00:49.875966+0000", 2024-02-09T18:06:12.103 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0678000000000001e-05 2024-02-09T18:06:12.103 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.103 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.103 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:06:12.104 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:00:49.882564+0000", 2024-02-09T18:06:12.104 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0065977229999999998 2024-02-09T18:06:12.104 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.104 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.104 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:06:12.104 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:00:49.882596+0000", 2024-02-09T18:06:12.104 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2366999999999997e-05 2024-02-09T18:06:12.104 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:12.104 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:06:12.104 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:12.104 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.104 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.104 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:138171 9.5 e919/882 9:a9fdacd0:::benchmark_data_smithi179_161915_object3289:head v 919'8661, mlcod=919'8659)", 2024-02-09T18:06:12.104 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:01:53.001713+0000", 2024-02-09T18:06:12.105 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 259.09113114199999, 2024-02-09T18:06:12.105 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19039196999999999, 2024-02-09T18:06:12.105 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:06:12.105 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:12.105 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:06:12.105 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.105 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:06:12.105 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.001713+0000", 2024-02-09T18:06:12.105 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:12.105 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.105 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.105 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:06:12.105 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.001713+0000", 2024-02-09T18:06:12.105 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:12.105 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.105 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.106 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:06:12.106 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.001714+0000", 2024-02-09T18:06:12.106 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5300000000000003e-07 2024-02-09T18:06:12.106 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.106 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.106 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:06:12.106 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.001724+0000", 2024-02-09T18:06:12.106 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0076e-05 2024-02-09T18:06:12.106 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.106 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.106 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:06:12.106 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.001739+0000", 2024-02-09T18:06:12.106 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4440999999999999e-05 2024-02-09T18:06:12.106 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.106 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.106 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:06:12.107 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.001745+0000", 2024-02-09T18:06:12.107 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7050000000000002e-06 2024-02-09T18:06:12.107 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.107 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.107 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:06:12.107 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.003281+0000", 2024-02-09T18:06:12.107 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001536078 2024-02-09T18:06:12.107 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.107 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.107 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:06:12.107 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.003301+0000", 2024-02-09T18:06:12.107 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9474e-05 2024-02-09T18:06:12.107 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.107 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.107 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:06:12.107 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.192052+0000", 2024-02-09T18:06:12.108 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18875141600000001 2024-02-09T18:06:12.108 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.108 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.108 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:06:12.108 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.192105+0000", 2024-02-09T18:06:12.108 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3026999999999999e-05 2024-02-09T18:06:12.108 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:12.108 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:06:12.108 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:12.108 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.108 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.108 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:138174 9.5 e919/882 9:a85e62f7:::benchmark_data_smithi179_161915_object3292:head v 919'8662, mlcod=919'8659)", 2024-02-09T18:06:12.108 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:01:53.009746+0000", 2024-02-09T18:06:12.108 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 259.08309893900002, 2024-02-09T18:06:12.109 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.182391837, 2024-02-09T18:06:12.109 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:06:12.109 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:12.109 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:06:12.109 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.109 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:06:12.109 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.009746+0000", 2024-02-09T18:06:12.109 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:12.109 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.109 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.109 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:06:12.109 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.009746+0000", 2024-02-09T18:06:12.109 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:12.109 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.109 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.110 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:06:12.110 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.009747+0000", 2024-02-09T18:06:12.110 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7699999999999992e-07 2024-02-09T18:06:12.110 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.110 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.110 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:06:12.110 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.009756+0000", 2024-02-09T18:06:12.110 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6269999999999995e-06 2024-02-09T18:06:12.110 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.110 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.110 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:06:12.110 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.009771+0000", 2024-02-09T18:06:12.110 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4684e-05 2024-02-09T18:06:12.110 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.110 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.111 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:06:12.111 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.009779+0000", 2024-02-09T18:06:12.111 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7249999999999997e-06 2024-02-09T18:06:12.111 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.111 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.111 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:06:12.111 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.009846+0000", 2024-02-09T18:06:12.111 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7246e-05 2024-02-09T18:06:12.111 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.111 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.111 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:06:12.111 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.009863+0000", 2024-02-09T18:06:12.111 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7606000000000001e-05 2024-02-09T18:06:12.111 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.111 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.112 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:06:12.112 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.192109+0000", 2024-02-09T18:06:12.112 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18224596800000001 2024-02-09T18:06:12.112 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.112 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.112 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:06:12.112 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.192137+0000", 2024-02-09T18:06:12.112 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8004000000000002e-05 2024-02-09T18:06:12.112 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:12.112 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:06:12.112 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:12.112 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.112 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.112 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:138179 9.d e919/882 9:bebb6f93:::benchmark_data_smithi179_161915_object3297:head v 919'8620, mlcod=919'8618)", 2024-02-09T18:06:12.112 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:01:53.009875+0000", 2024-02-09T18:06:12.113 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 259.08296950099998, 2024-02-09T18:06:12.113 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18565922600000001, 2024-02-09T18:06:12.113 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:06:12.113 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:12.113 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:06:12.113 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.113 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:06:12.113 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.009875+0000", 2024-02-09T18:06:12.113 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:12.113 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.113 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.113 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:06:12.113 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.009875+0000", 2024-02-09T18:06:12.113 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:12.113 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.114 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.114 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:06:12.114 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.009876+0000", 2024-02-09T18:06:12.114 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5300000000000003e-07 2024-02-09T18:06:12.114 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.114 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.114 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:06:12.114 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.009884+0000", 2024-02-09T18:06:12.114 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8709999999999997e-06 2024-02-09T18:06:12.114 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.114 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.114 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:06:12.114 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.009905+0000", 2024-02-09T18:06:12.114 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1129e-05 2024-02-09T18:06:12.114 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.115 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.115 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:06:12.115 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.009912+0000", 2024-02-09T18:06:12.115 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9140000000000002e-06 2024-02-09T18:06:12.115 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.115 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.115 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:06:12.115 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.014020+0000", 2024-02-09T18:06:12.115 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041087199999999997 2024-02-09T18:06:12.115 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.115 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.115 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:06:12.115 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.014041+0000", 2024-02-09T18:06:12.115 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0594999999999998e-05 2024-02-09T18:06:12.115 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.115 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.116 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:06:12.116 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.195484+0000", 2024-02-09T18:06:12.116 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.181442673 2024-02-09T18:06:12.116 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.116 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.116 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:06:12.116 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.195534+0000", 2024-02-09T18:06:12.116 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0571000000000001e-05 2024-02-09T18:06:12.116 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:12.116 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:06:12.116 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:12.116 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.116 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.116 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:138198 9.d e919/882 9:b9e4c9f9:::benchmark_data_smithi179_161915_object3316:head v 919'8621, mlcod=919'8618)", 2024-02-09T18:06:12.116 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:01:53.034031+0000", 2024-02-09T18:06:12.117 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 259.058813504, 2024-02-09T18:06:12.117 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.161532536, 2024-02-09T18:06:12.117 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:06:12.117 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:12.117 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:06:12.117 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.117 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:06:12.117 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.034031+0000", 2024-02-09T18:06:12.117 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:12.117 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.117 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.117 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:06:12.117 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.034031+0000", 2024-02-09T18:06:12.117 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:12.117 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.118 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.118 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:06:12.118 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.034033+0000", 2024-02-09T18:06:12.118 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6980000000000001e-06 2024-02-09T18:06:12.118 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.118 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.118 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:06:12.118 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.034042+0000", 2024-02-09T18:06:12.118 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1430000000000004e-06 2024-02-09T18:06:12.118 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.118 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.118 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:06:12.118 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.034062+0000", 2024-02-09T18:06:12.118 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0095999999999999e-05 2024-02-09T18:06:12.118 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.119 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.119 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:06:12.119 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.034072+0000", 2024-02-09T18:06:12.119 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0142e-05 2024-02-09T18:06:12.119 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.119 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.119 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:06:12.119 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.034149+0000", 2024-02-09T18:06:12.119 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6439000000000005e-05 2024-02-09T18:06:12.119 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.119 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.119 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:06:12.119 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.034169+0000", 2024-02-09T18:06:12.119 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0126999999999999e-05 2024-02-09T18:06:12.120 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.120 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.120 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:06:12.120 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.195538+0000", 2024-02-09T18:06:12.120 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16136936099999999 2024-02-09T18:06:12.120 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.120 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.120 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:06:12.120 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.195564+0000", 2024-02-09T18:06:12.120 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5530000000000001e-05 2024-02-09T18:06:12.120 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:12.120 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:06:12.120 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:12.120 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.120 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.121 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:138201 9.5 e919/882 9:a7dcdf66:::benchmark_data_smithi179_161915_object3319:head v 919'8663, mlcod=919'8659)", 2024-02-09T18:06:12.121 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:01:53.043069+0000", 2024-02-09T18:06:12.121 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 259.04977545999998, 2024-02-09T18:06:12.121 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.152518976, 2024-02-09T18:06:12.121 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:06:12.121 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:12.121 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:06:12.121 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.121 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:06:12.121 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.043069+0000", 2024-02-09T18:06:12.121 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:12.121 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.121 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.121 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:06:12.121 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.043069+0000", 2024-02-09T18:06:12.121 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:12.122 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.122 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.122 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:06:12.122 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.043070+0000", 2024-02-09T18:06:12.122 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0589999999999999e-06 2024-02-09T18:06:12.122 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.122 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.122 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:06:12.122 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.043078+0000", 2024-02-09T18:06:12.122 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9929999999999995e-06 2024-02-09T18:06:12.122 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.122 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.122 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:06:12.122 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.043093+0000", 2024-02-09T18:06:12.122 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5158e-05 2024-02-09T18:06:12.123 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.123 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.123 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:06:12.123 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.043102+0000", 2024-02-09T18:06:12.123 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7099999999999996e-06 2024-02-09T18:06:12.123 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.123 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.123 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:06:12.123 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.043192+0000", 2024-02-09T18:06:12.123 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0033999999999999e-05 2024-02-09T18:06:12.123 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.123 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.123 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:06:12.123 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.043215+0000", 2024-02-09T18:06:12.124 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2980999999999999e-05 2024-02-09T18:06:12.124 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.124 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.124 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:06:12.124 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.195566+0000", 2024-02-09T18:06:12.124 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.15235079600000001 2024-02-09T18:06:12.124 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.124 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.124 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:06:12.124 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.195588+0000", 2024-02-09T18:06:12.124 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2245000000000001e-05 2024-02-09T18:06:12.124 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:12.124 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:06:12.124 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:12.125 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.125 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.125 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:138208 9.d e919/882 9:b674a445:::benchmark_data_smithi179_161915_object3326:head v 919'8622, mlcod=919'8618)", 2024-02-09T18:06:12.125 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:01:53.048648+0000", 2024-02-09T18:06:12.125 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 259.04419691499999, 2024-02-09T18:06:12.125 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.14697751000000001, 2024-02-09T18:06:12.125 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:06:12.125 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:12.125 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:06:12.125 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.125 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:06:12.125 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.048648+0000", 2024-02-09T18:06:12.125 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:12.125 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.125 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.126 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:06:12.126 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.048648+0000", 2024-02-09T18:06:12.126 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:12.126 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.126 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.126 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:06:12.126 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.048649+0000", 2024-02-09T18:06:12.126 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3799999999999996e-07 2024-02-09T18:06:12.126 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.126 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.126 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:06:12.126 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.048655+0000", 2024-02-09T18:06:12.126 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6880000000000002e-06 2024-02-09T18:06:12.126 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.126 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.126 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:06:12.127 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.048667+0000", 2024-02-09T18:06:12.127 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1552000000000001e-05 2024-02-09T18:06:12.127 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.127 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.127 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:06:12.127 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.048674+0000", 2024-02-09T18:06:12.127 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6490000000000008e-06 2024-02-09T18:06:12.127 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.127 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.127 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:06:12.127 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.048727+0000", 2024-02-09T18:06:12.127 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3069999999999998e-05 2024-02-09T18:06:12.127 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.127 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.127 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:06:12.128 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.048741+0000", 2024-02-09T18:06:12.128 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3499000000000001e-05 2024-02-09T18:06:12.128 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.128 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.128 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:06:12.128 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.195591+0000", 2024-02-09T18:06:12.128 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.14684998799999999 2024-02-09T18:06:12.128 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.128 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.128 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:06:12.128 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.195625+0000", 2024-02-09T18:06:12.128 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4125999999999998e-05 2024-02-09T18:06:12.128 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:12.128 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:06:12.128 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:12.129 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.129 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.129 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:138229 9.d e919/882 9:be7e413e:::benchmark_data_smithi179_161915_object3347:head v 919'8623, mlcod=919'8618)", 2024-02-09T18:06:12.129 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:01:53.075013+0000", 2024-02-09T18:06:12.129 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 259.01783163300001, 2024-02-09T18:06:12.129 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.120647305, 2024-02-09T18:06:12.129 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:06:12.129 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:12.129 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:06:12.129 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.129 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:06:12.129 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.075013+0000", 2024-02-09T18:06:12.129 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:12.129 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.129 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.130 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:06:12.130 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.075013+0000", 2024-02-09T18:06:12.130 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:12.130 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.130 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.130 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:06:12.130 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.075014+0000", 2024-02-09T18:06:12.130 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0279999999999999e-06 2024-02-09T18:06:12.130 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.130 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.130 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:06:12.130 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.075022+0000", 2024-02-09T18:06:12.130 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8420000000000005e-06 2024-02-09T18:06:12.130 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.130 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.130 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:06:12.131 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.075039+0000", 2024-02-09T18:06:12.131 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7516999999999998e-05 2024-02-09T18:06:12.131 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.131 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.131 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:06:12.131 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.075048+0000", 2024-02-09T18:06:12.131 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1120000000000002e-06 2024-02-09T18:06:12.131 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.131 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.131 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:06:12.131 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.075101+0000", 2024-02-09T18:06:12.131 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2997000000000001e-05 2024-02-09T18:06:12.131 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.131 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.131 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:06:12.132 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.075124+0000", 2024-02-09T18:06:12.132 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2572999999999999e-05 2024-02-09T18:06:12.132 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.132 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.132 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:06:12.132 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.195627+0000", 2024-02-09T18:06:12.132 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.12050343400000001 2024-02-09T18:06:12.132 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.132 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.132 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:06:12.132 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.195660+0000", 2024-02-09T18:06:12.132 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2802000000000002e-05 2024-02-09T18:06:12.132 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:12.132 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:06:12.132 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:12.133 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.133 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.133 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:138230 9.5 e919/882 9:af004162:::benchmark_data_smithi179_161915_object3348:head v 919'8664, mlcod=919'8659)", 2024-02-09T18:06:12.133 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:01:53.077752+0000", 2024-02-09T18:06:12.133 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 259.01509264600003, 2024-02-09T18:06:12.133 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.121983631, 2024-02-09T18:06:12.133 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:06:12.133 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:12.133 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:06:12.133 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.133 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:06:12.133 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.077752+0000", 2024-02-09T18:06:12.133 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:12.133 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.133 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.134 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:06:12.134 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.077752+0000", 2024-02-09T18:06:12.134 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:12.134 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.134 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.134 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:06:12.134 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.077753+0000", 2024-02-09T18:06:12.134 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.108e-06 2024-02-09T18:06:12.134 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.134 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.134 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:06:12.134 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.077761+0000", 2024-02-09T18:06:12.134 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0250000000000002e-06 2024-02-09T18:06:12.134 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.135 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.135 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:06:12.135 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.077778+0000", 2024-02-09T18:06:12.135 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7337e-05 2024-02-09T18:06:12.135 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.135 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.135 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:06:12.135 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.077787+0000", 2024-02-09T18:06:12.135 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2320000000000001e-06 2024-02-09T18:06:12.135 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.135 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.135 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:06:12.135 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.078992+0000", 2024-02-09T18:06:12.135 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001205051 2024-02-09T18:06:12.135 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.136 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.136 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:06:12.136 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.079011+0000", 2024-02-09T18:06:12.136 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9420999999999999e-05 2024-02-09T18:06:12.136 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.136 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.136 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:06:12.136 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.199676+0000", 2024-02-09T18:06:12.136 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.120665245 2024-02-09T18:06:12.136 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.136 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.136 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:06:12.136 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.199735+0000", 2024-02-09T18:06:12.136 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9212000000000002e-05 2024-02-09T18:06:12.137 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:12.137 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:06:12.137 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:12.137 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.137 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.137 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:138239 9.5 e919/882 9:abfa7cde:::benchmark_data_smithi179_161915_object3357:head v 919'8665, mlcod=919'8659)", 2024-02-09T18:06:12.137 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:01:53.086686+0000", 2024-02-09T18:06:12.137 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 259.00615842100001, 2024-02-09T18:06:12.137 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.113085006, 2024-02-09T18:06:12.137 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:06:12.137 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:12.137 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:06:12.137 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.137 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:06:12.137 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.086686+0000", 2024-02-09T18:06:12.138 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:12.138 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.138 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.138 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:06:12.138 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.086686+0000", 2024-02-09T18:06:12.138 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:12.138 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.138 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.138 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:06:12.138 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.086687+0000", 2024-02-09T18:06:12.138 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.09e-07 2024-02-09T18:06:12.138 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.138 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.138 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:06:12.138 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.086694+0000", 2024-02-09T18:06:12.139 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0269999999999998e-06 2024-02-09T18:06:12.139 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.139 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.139 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:06:12.139 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.086707+0000", 2024-02-09T18:06:12.139 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2981e-05 2024-02-09T18:06:12.139 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.139 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.139 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:06:12.139 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.086715+0000", 2024-02-09T18:06:12.139 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.588e-06 2024-02-09T18:06:12.139 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.139 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.139 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:06:12.139 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.086759+0000", 2024-02-09T18:06:12.140 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4709999999999997e-05 2024-02-09T18:06:12.140 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.140 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.140 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:06:12.140 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.086774+0000", 2024-02-09T18:06:12.140 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4409e-05 2024-02-09T18:06:12.140 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.140 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.140 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:06:12.140 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.199743+0000", 2024-02-09T18:06:12.140 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.112969714 2024-02-09T18:06:12.140 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.140 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.140 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:06:12.141 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.199771+0000", 2024-02-09T18:06:12.141 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7668e-05 2024-02-09T18:06:12.141 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:12.141 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:06:12.141 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:12.141 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.141 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.141 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:138243 9.d e919/882 9:b78e9a4f:::benchmark_data_smithi179_161915_object3361:head v 919'8624, mlcod=919'8618)", 2024-02-09T18:06:12.141 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:01:53.095689+0000", 2024-02-09T18:06:12.141 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 258.99715574800001, 2024-02-09T18:06:12.141 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.107911805, 2024-02-09T18:06:12.141 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:06:12.141 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:12.141 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:06:12.141 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.141 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:06:12.142 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.095689+0000", 2024-02-09T18:06:12.142 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:12.142 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.142 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.142 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:06:12.142 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.095689+0000", 2024-02-09T18:06:12.142 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:12.142 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.142 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.142 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:06:12.142 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.095690+0000", 2024-02-09T18:06:12.142 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.017e-06 2024-02-09T18:06:12.142 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.142 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.143 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:06:12.143 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.095697+0000", 2024-02-09T18:06:12.143 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.002e-06 2024-02-09T18:06:12.143 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.143 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.143 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:06:12.143 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.095710+0000", 2024-02-09T18:06:12.143 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.307e-05 2024-02-09T18:06:12.143 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.143 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.143 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:06:12.143 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.095718+0000", 2024-02-09T18:06:12.143 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2290000000000003e-06 2024-02-09T18:06:12.143 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.143 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.144 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:06:12.144 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.095802+0000", 2024-02-09T18:06:12.144 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4175e-05 2024-02-09T18:06:12.144 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.144 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.144 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:06:12.144 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.095824+0000", 2024-02-09T18:06:12.144 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1270000000000001e-05 2024-02-09T18:06:12.144 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.144 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.144 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:06:12.144 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.203555+0000", 2024-02-09T18:06:12.144 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.107731321 2024-02-09T18:06:12.144 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.144 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.145 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:06:12.145 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.203601+0000", 2024-02-09T18:06:12.145 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5720999999999998e-05 2024-02-09T18:06:12.145 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:12.145 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:06:12.145 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:12.145 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.145 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.145 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:138251 9.d e919/882 9:b8f70d5d:::benchmark_data_smithi179_161915_object3369:head v 919'8625, mlcod=919'8618)", 2024-02-09T18:06:12.145 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:01:53.104469+0000", 2024-02-09T18:06:12.145 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 258.98837568099998, 2024-02-09T18:06:12.145 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.099161527999999999, 2024-02-09T18:06:12.145 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:06:12.145 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:12.145 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:06:12.145 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.146 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:06:12.146 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.104469+0000", 2024-02-09T18:06:12.146 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:12.146 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.146 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.146 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:06:12.146 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.104469+0000", 2024-02-09T18:06:12.146 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:12.146 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.146 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.146 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:06:12.146 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.104470+0000", 2024-02-09T18:06:12.146 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6099999999999999e-07 2024-02-09T18:06:12.146 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.146 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.147 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:06:12.147 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.104475+0000", 2024-02-09T18:06:12.147 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.259e-06 2024-02-09T18:06:12.147 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.147 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.147 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:06:12.147 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.104487+0000", 2024-02-09T18:06:12.147 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2051e-05 2024-02-09T18:06:12.147 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.147 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.147 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:06:12.147 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.104494+0000", 2024-02-09T18:06:12.147 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3069999999999997e-06 2024-02-09T18:06:12.147 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.147 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.147 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:06:12.147 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.104560+0000", 2024-02-09T18:06:12.148 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5936e-05 2024-02-09T18:06:12.148 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.148 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.148 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:06:12.148 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.104583+0000", 2024-02-09T18:06:12.148 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2470999999999999e-05 2024-02-09T18:06:12.148 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.148 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.148 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:06:12.148 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.203604+0000", 2024-02-09T18:06:12.148 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.099021667999999993 2024-02-09T18:06:12.148 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.148 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.149 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:06:12.149 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.203630+0000", 2024-02-09T18:06:12.149 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5975000000000001e-05 2024-02-09T18:06:12.149 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:12.149 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:06:12.149 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:12.149 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.149 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.149 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:138254 9.d e919/882 9:bdf952ff:::benchmark_data_smithi179_161915_object3372:head v 919'8626, mlcod=919'8618)", 2024-02-09T18:06:12.149 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:01:53.110269+0000", 2024-02-09T18:06:12.149 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 258.982575209, 2024-02-09T18:06:12.149 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.093388179000000002, 2024-02-09T18:06:12.149 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:06:12.149 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:12.149 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:06:12.149 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.150 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:06:12.150 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.110269+0000", 2024-02-09T18:06:12.150 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:12.150 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.150 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.150 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:06:12.150 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.110269+0000", 2024-02-09T18:06:12.150 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:12.150 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.150 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.150 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:06:12.150 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.110270+0000", 2024-02-09T18:06:12.150 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7000000000000004e-07 2024-02-09T18:06:12.150 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.150 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.151 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:06:12.151 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.110275+0000", 2024-02-09T18:06:12.151 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3340000000000001e-06 2024-02-09T18:06:12.151 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.151 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.151 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:06:12.151 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.110287+0000", 2024-02-09T18:06:12.151 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1445e-05 2024-02-09T18:06:12.151 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.151 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.151 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:06:12.151 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.110295+0000", 2024-02-09T18:06:12.151 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.729e-06 2024-02-09T18:06:12.151 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.151 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.152 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:06:12.152 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.110365+0000", 2024-02-09T18:06:12.152 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0826000000000003e-05 2024-02-09T18:06:12.152 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.152 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.152 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:06:12.152 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.110385+0000", 2024-02-09T18:06:12.152 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9283000000000002e-05 2024-02-09T18:06:12.152 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.152 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.152 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:06:12.152 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.203633+0000", 2024-02-09T18:06:12.152 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.093248287999999999 2024-02-09T18:06:12.152 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.153 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.153 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:06:12.153 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.203657+0000", 2024-02-09T18:06:12.153 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4504000000000001e-05 2024-02-09T18:06:12.153 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:12.153 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:06:12.153 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:12.153 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.153 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.153 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:138264 9.d e919/882 9:b5917361:::benchmark_data_smithi179_161915_object3382:head v 919'8627, mlcod=919'8618)", 2024-02-09T18:06:12.153 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:01:53.127206+0000", 2024-02-09T18:06:12.153 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 258.965638136, 2024-02-09T18:06:12.153 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.076473997000000002, 2024-02-09T18:06:12.153 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:06:12.153 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:12.154 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:06:12.154 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.154 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:06:12.154 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.127206+0000", 2024-02-09T18:06:12.154 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:12.154 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.154 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.154 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:06:12.154 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.127206+0000", 2024-02-09T18:06:12.154 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:12.154 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.154 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.154 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:06:12.154 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.127207+0000", 2024-02-09T18:06:12.154 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4900000000000005e-07 2024-02-09T18:06:12.155 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.155 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.155 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:06:12.155 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.127214+0000", 2024-02-09T18:06:12.155 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7039999999999997e-06 2024-02-09T18:06:12.155 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.155 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.155 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:06:12.155 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.127226+0000", 2024-02-09T18:06:12.155 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2636000000000001e-05 2024-02-09T18:06:12.155 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.155 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.155 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:06:12.155 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.127243+0000", 2024-02-09T18:06:12.155 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6892999999999999e-05 2024-02-09T18:06:12.155 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.156 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.156 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:06:12.156 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.192140+0000", 2024-02-09T18:06:12.156 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.064896477999999994 2024-02-09T18:06:12.156 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.156 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.156 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:06:12.156 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.192155+0000", 2024-02-09T18:06:12.156 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4666e-05 2024-02-09T18:06:12.156 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.156 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.156 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:06:12.156 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.203659+0000", 2024-02-09T18:06:12.156 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011504904 2024-02-09T18:06:12.156 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.157 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.157 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:06:12.157 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.203680+0000", 2024-02-09T18:06:12.157 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0967000000000001e-05 2024-02-09T18:06:12.157 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:12.157 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:06:12.157 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:12.157 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.157 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.157 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:138269 9.5 e919/882 9:a364bd26:::benchmark_data_smithi179_161915_object3387:head v 919'8666, mlcod=919'8659)", 2024-02-09T18:06:12.157 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:01:53.135939+0000", 2024-02-09T18:06:12.157 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 258.95690584599998, 2024-02-09T18:06:12.157 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.072003169000000006, 2024-02-09T18:06:12.157 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:06:12.157 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:12.157 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:06:12.158 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.158 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:06:12.158 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.135939+0000", 2024-02-09T18:06:12.158 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:12.158 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.158 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.158 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:06:12.158 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.135939+0000", 2024-02-09T18:06:12.158 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:12.158 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.158 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.158 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:06:12.158 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.135940+0000", 2024-02-09T18:06:12.158 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0249999999999999e-06 2024-02-09T18:06:12.158 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.159 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.159 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:06:12.159 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.135947+0000", 2024-02-09T18:06:12.159 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7849999999999992e-06 2024-02-09T18:06:12.159 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.159 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.159 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:06:12.159 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.135963+0000", 2024-02-09T18:06:12.159 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6016999999999999e-05 2024-02-09T18:06:12.159 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.159 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.159 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:06:12.159 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.135971+0000", 2024-02-09T18:06:12.159 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5089999999999996e-06 2024-02-09T18:06:12.159 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.159 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.160 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:06:12.160 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.195615+0000", 2024-02-09T18:06:12.160 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.059644219999999998 2024-02-09T18:06:12.160 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.160 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.160 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:06:12.160 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.195634+0000", 2024-02-09T18:06:12.160 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8453000000000001e-05 2024-02-09T18:06:12.160 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.160 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.160 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:06:12.160 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.207901+0000", 2024-02-09T18:06:12.160 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012267162 2024-02-09T18:06:12.160 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.160 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.160 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:06:12.161 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.207942+0000", 2024-02-09T18:06:12.161 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0998000000000003e-05 2024-02-09T18:06:12.161 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:12.161 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:06:12.161 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:12.161 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.161 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.161 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26563.0:138271 9.5 e919/882 9:a5190b7c:::benchmark_data_smithi179_161915_object3389:head v 919'8667, mlcod=919'8659)", 2024-02-09T18:06:12.161 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:01:53.141918+0000", 2024-02-09T18:06:12.161 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 258.95092677000002, 2024-02-09T18:06:12.161 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.076447451, 2024-02-09T18:06:12.161 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:06:12.161 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:12.161 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:06:12.161 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.162 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:06:12.162 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.141918+0000", 2024-02-09T18:06:12.162 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:12.162 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.162 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.162 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:06:12.162 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.141918+0000", 2024-02-09T18:06:12.162 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:12.162 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.162 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.162 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:06:12.162 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.141919+0000", 2024-02-09T18:06:12.162 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4399999999999998e-07 2024-02-09T18:06:12.162 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.162 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.163 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:06:12.163 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.141925+0000", 2024-02-09T18:06:12.163 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0639999999999999e-06 2024-02-09T18:06:12.163 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.163 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.163 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:06:12.163 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.141937+0000", 2024-02-09T18:06:12.163 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2602000000000001e-05 2024-02-09T18:06:12.163 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.163 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.163 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:06:12.163 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.141945+0000", 2024-02-09T18:06:12.163 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8210000000000002e-06 2024-02-09T18:06:12.163 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.163 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.164 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:06:12.164 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.199800+0000", 2024-02-09T18:06:12.164 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.057854712000000003 2024-02-09T18:06:12.164 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.164 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.164 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:06:12.164 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.199829+0000", 2024-02-09T18:06:12.164 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8782e-05 2024-02-09T18:06:12.164 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.164 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.164 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-02-09T18:06:12.164 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.218317+0000", 2024-02-09T18:06:12.164 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018488121999999999 2024-02-09T18:06:12.164 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.165 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.165 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:06:12.165 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:01:53.218365+0000", 2024-02-09T18:06:12.165 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8403999999999999e-05 2024-02-09T18:06:12.165 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:12.165 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:06:12.165 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:12.165 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.165 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.165 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "rep_scrubmap(1.d e938 from shard 1)", 2024-02-09T18:06:12.165 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:05:11.940845+0000", 2024-02-09T18:06:12.165 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 60.151999044, 2024-02-09T18:06:12.165 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.091569949999999997, 2024-02-09T18:06:12.165 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:06:12.165 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-09T18:06:12.166 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:06:12.166 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.166 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:06:12.166 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:05:11.940845+0000", 2024-02-09T18:06:12.166 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:12.166 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.166 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.166 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:06:12.166 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:05:11.940845+0000", 2024-02-09T18:06:12.166 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:12.166 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.166 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.166 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:06:12.166 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:05:11.940846+0000", 2024-02-09T18:06:12.166 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2800000000000005e-07 2024-02-09T18:06:12.167 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.167 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.167 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:06:12.167 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:05:11.940852+0000", 2024-02-09T18:06:12.167 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4870000000000002e-06 2024-02-09T18:06:12.167 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.167 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.167 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:06:12.167 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:05:11.940864+0000", 2024-02-09T18:06:12.167 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1829e-05 2024-02-09T18:06:12.167 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.167 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.167 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:06:12.167 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:05:11.940872+0000", 2024-02-09T18:06:12.167 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.174e-06 2024-02-09T18:06:12.167 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.168 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.168 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:06:12.168 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:05:12.032393+0000", 2024-02-09T18:06:12.168 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.091521216000000002 2024-02-09T18:06:12.168 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.168 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.168 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:06:12.168 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:05:12.032403+0000", 2024-02-09T18:06:12.168 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8719999999999997e-06 2024-02-09T18:06:12.168 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.168 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.168 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:06:12.168 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:05:12.032415+0000", 2024-02-09T18:06:12.168 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2444e-05 2024-02-09T18:06:12.168 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:12.169 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:06:12.169 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:12.169 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.169 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.169 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDScrubReserve(1.4,QREQUEST,e:1017,from:11,reservation_nonce:36)", 2024-02-09T18:06:12.169 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:06:11.530451+0000", 2024-02-09T18:06:12.169 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.56239373599999998, 2024-02-09T18:06:12.169 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022866999999999999, 2024-02-09T18:06:12.169 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:06:12.169 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-09T18:06:12.169 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:06:12.169 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.169 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:06:12.169 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:06:11.530451+0000", 2024-02-09T18:06:12.169 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:12.170 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.170 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.170 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:06:12.170 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:06:11.530451+0000", 2024-02-09T18:06:12.170 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:12.170 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.170 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.170 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:06:12.170 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:06:11.530453+0000", 2024-02-09T18:06:12.170 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1399999999999998e-06 2024-02-09T18:06:12.170 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.170 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.170 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:06:12.170 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:06:11.530466+0000", 2024-02-09T18:06:12.170 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3236e-05 2024-02-09T18:06:12.171 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.171 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.171 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:06:12.171 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:06:11.530508+0000", 2024-02-09T18:06:12.171 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1578000000000001e-05 2024-02-09T18:06:12.171 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.171 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.171 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:06:12.171 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:06:11.530519+0000", 2024-02-09T18:06:12.171 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0883999999999999e-05 2024-02-09T18:06:12.171 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.171 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.171 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:06:12.171 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:06:11.530622+0000", 2024-02-09T18:06:12.171 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000103105 2024-02-09T18:06:12.171 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.172 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.172 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:06:12.172 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:06:11.530640+0000", 2024-02-09T18:06:12.172 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8641e-05 2024-02-09T18:06:12.172 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.172 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.172 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:06:12.172 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:06:11.530679+0000", 2024-02-09T18:06:12.172 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9085999999999998e-05 2024-02-09T18:06:12.172 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:12.172 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:06:12.172 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:12.172 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.172 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.172 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "replica_scrub(pg: 1.4,from:0'0,to:0'0,epoch:1017/261,start:1:20000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-02-09T18:06:12.173 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-02-09T18:06:11.531902+0000", 2024-02-09T18:06:12.173 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.56094213000000004, 2024-02-09T18:06:12.173 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00026041900000000002, 2024-02-09T18:06:12.173 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-02-09T18:06:12.173 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-02-09T18:06:12.173 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-02-09T18:06:12.173 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.173 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-02-09T18:06:12.173 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:06:11.531902+0000", 2024-02-09T18:06:12.173 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:12.173 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.173 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.173 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-02-09T18:06:12.173 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:06:11.531902+0000", 2024-02-09T18:06:12.173 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-02-09T18:06:12.173 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.174 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.174 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-02-09T18:06:12.174 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:06:11.531904+0000", 2024-02-09T18:06:12.174 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.584e-06 2024-02-09T18:06:12.174 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.174 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.174 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-02-09T18:06:12.174 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:06:11.531915+0000", 2024-02-09T18:06:12.174 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1049e-05 2024-02-09T18:06:12.174 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.174 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.174 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-02-09T18:06:12.174 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:06:11.531944+0000", 2024-02-09T18:06:12.174 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9272999999999999e-05 2024-02-09T18:06:12.174 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.175 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.175 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-02-09T18:06:12.175 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:06:11.531962+0000", 2024-02-09T18:06:12.175 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7385999999999999e-05 2024-02-09T18:06:12.175 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.175 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.175 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-02-09T18:06:12.175 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:06:11.532089+0000", 2024-02-09T18:06:12.175 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000127661 2024-02-09T18:06:12.175 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.175 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.175 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-02-09T18:06:12.175 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:06:11.532114+0000", 2024-02-09T18:06:12.175 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4862999999999999e-05 2024-02-09T18:06:12.175 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-02-09T18:06:12.175 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-02-09T18:06:12.176 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-02-09T18:06:12.176 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-02-09T18:06:11.532163+0000", 2024-02-09T18:06:12.176 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8603000000000002e-05 2024-02-09T18:06:12.176 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:12.176 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:06:12.176 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:12.176 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-02-09T18:06:12.176 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-02-09T18:06:12.176 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-02-09T18:06:12.446 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.7 to 1 (10000) 2024-02-09T18:06:12.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:12 smithi099 ceph-mon[39420]: pgmap v2500: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:12.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:12 smithi099 ceph-mon[39420]: 1.4 scrub starts 2024-02-09T18:06:12.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:12 smithi099 ceph-mon[39420]: 1.4 scrub ok 2024-02-09T18:06:12.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:12 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1914253150' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-09T18:06:12.781 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:12 smithi162 ceph-mon[44304]: pgmap v2500: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:12.781 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:12 smithi162 ceph-mon[44304]: 1.4 scrub starts 2024-02-09T18:06:12.781 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:12 smithi162 ceph-mon[44304]: 1.4 scrub ok 2024-02-09T18:06:12.781 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:12 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1914253150' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-09T18:06:12.808 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.9 dump_ops_in_flight 2024-02-09T18:06:12.843 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:12 smithi179 ceph-mon[43975]: pgmap v2500: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:12.843 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:12 smithi179 ceph-mon[43975]: 1.4 scrub starts 2024-02-09T18:06:12.844 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:12 smithi179 ceph-mon[43975]: 1.4 scrub ok 2024-02-09T18:06:12.844 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:12 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1914253150' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-09T18:06:13.087 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T18:06:13.183 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd primary-affinity 7 1 2024-02-09T18:06:13.446 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:06:13.574 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:13 smithi179 ceph-mon[43975]: 1.14 scrub starts 2024-02-09T18:06:13.574 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:13 smithi179 ceph-mon[43975]: 1.14 scrub ok 2024-02-09T18:06:13.574 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:13 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1914253150' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-09T18:06:13.574 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:13 smithi179 ceph-mon[43975]: osdmap e1018: 12 total, 12 up, 12 in 2024-02-09T18:06:13.700 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:13 smithi099 ceph-mon[39420]: 1.14 scrub starts 2024-02-09T18:06:13.700 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:13 smithi099 ceph-mon[39420]: 1.14 scrub ok 2024-02-09T18:06:13.700 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:13 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1914253150' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-09T18:06:13.700 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:13 smithi099 ceph-mon[39420]: osdmap e1018: 12 total, 12 up, 12 in 2024-02-09T18:06:13.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:13 smithi162 ceph-mon[44304]: 1.14 scrub starts 2024-02-09T18:06:13.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:13 smithi162 ceph-mon[44304]: 1.14 scrub ok 2024-02-09T18:06:13.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:13 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1914253150' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-09T18:06:13.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:13 smithi162 ceph-mon[44304]: osdmap e1018: 12 total, 12 up, 12 in 2024-02-09T18:06:14.615 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:14 smithi179 ceph-mon[43975]: pgmap v2502: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:14.722 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:14 smithi099 ceph-mon[39420]: pgmap v2502: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:14.896 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T18:06:14.896 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-02-09T18:06:14.896 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 0 2024-02-09T18:06:14.896 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T18:06:14.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:14 smithi162 ceph-mon[44304]: pgmap v2502: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:15.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:15 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T18:06:15.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:15 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T18:06:15.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:15 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T18:06:15.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:15 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T18:06:15.698 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.9 dump_blocked_ops 2024-02-09T18:06:15.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:15 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T18:06:15.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:15 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T18:06:15.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:15 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T18:06:15.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:15 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T18:06:15.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:15 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T18:06:15.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:15 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T18:06:15.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:15 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T18:06:15.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:15 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T18:06:15.999 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T18:06:16.468 INFO:teuthology.orchestra.run.smithi099.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-02-09T18:06:16.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:16 smithi179 ceph-mon[43975]: 1.17 scrub starts 2024-02-09T18:06:16.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:16 smithi179 ceph-mon[43975]: 1.17 scrub ok 2024-02-09T18:06:16.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:16 smithi179 ceph-mon[43975]: pgmap v2503: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:16.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:16 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2588031629' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-09T18:06:16.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:16 smithi099 ceph-mon[39420]: 1.17 scrub starts 2024-02-09T18:06:16.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:16 smithi099 ceph-mon[39420]: 1.17 scrub ok 2024-02-09T18:06:16.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:16 smithi099 ceph-mon[39420]: pgmap v2503: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:16.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:16 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2588031629' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-09T18:06:16.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:16 smithi162 ceph-mon[44304]: 1.17 scrub starts 2024-02-09T18:06:16.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:16 smithi162 ceph-mon[44304]: 1.17 scrub ok 2024-02-09T18:06:16.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:16 smithi162 ceph-mon[44304]: pgmap v2503: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:16.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:16 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2588031629' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-09T18:06:17.227 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 9 1 2024-02-09T18:06:17.495 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:06:17.687 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T18:06:17.687 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-02-09T18:06:17.688 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-02-09T18:06:17.688 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-02-09T18:06:17.688 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T18:06:17.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:17 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2588031629' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-02-09T18:06:17.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:17 smithi099 ceph-mon[39420]: osdmap e1019: 12 total, 12 up, 12 in 2024-02-09T18:06:17.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:17 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2588031629' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-02-09T18:06:17.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:17 smithi179 ceph-mon[43975]: osdmap e1019: 12 total, 12 up, 12 in 2024-02-09T18:06:17.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:17 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2588031629' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-02-09T18:06:17.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:17 smithi162 ceph-mon[44304]: osdmap e1019: 12 total, 12 up, 12 in 2024-02-09T18:06:18.456 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.9 dump_historic_ops 2024-02-09T18:06:18.718 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T18:06:18.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:18 smithi099 ceph-mon[39420]: pgmap v2505: 31 pgs: 1 peering, 30 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:18.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:18 smithi099 ceph-mon[39420]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-09T18:06:18.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:18 smithi099 ceph-mon[39420]: osdmap e1020: 12 total, 12 up, 12 in 2024-02-09T18:06:18.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:18 smithi179 ceph-mon[43975]: pgmap v2505: 31 pgs: 1 peering, 30 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:18.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:18 smithi179 ceph-mon[43975]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-09T18:06:18.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:18 smithi179 ceph-mon[43975]: osdmap e1020: 12 total, 12 up, 12 in 2024-02-09T18:06:18.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:18 smithi162 ceph-mon[44304]: pgmap v2505: 31 pgs: 1 peering, 30 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:18.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:18 smithi162 ceph-mon[44304]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-09T18:06:18.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:18 smithi162 ceph-mon[44304]: osdmap e1020: 12 total, 12 up, 12 in 2024-02-09T18:06:20.411 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T18:06:20.411 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-02-09T18:06:20.412 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-02-09T18:06:20.412 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-02-09T18:06:20.412 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.412 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:149659 8.8 8:110929c2:::benchmark_data_smithi179_148205_object26587:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e826)", 2024-02-09T18:06:20.412 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:57:26.089697+0000", 2024-02-09T18:06:20.412 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 534.31789268299997, 2024-02-09T18:06:20.412 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.22273084000000001, 2024-02-09T18:06:20.412 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:20.412 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:20.412 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:20.412 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:06:20.412 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:06:20.412 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 149659 2024-02-09T18:06:20.413 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.413 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:20.413 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.413 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:20.413 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.089697+0000", 2024-02-09T18:06:20.413 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:20.413 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.413 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.413 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:20.413 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.089697+0000", 2024-02-09T18:06:20.413 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:20.413 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.413 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.413 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:20.413 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.089697+0000", 2024-02-09T18:06:20.414 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.3200000000000001e-07 2024-02-09T18:06:20.414 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.414 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.414 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:20.414 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.089700+0000", 2024-02-09T18:06:20.414 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1679999999999998e-06 2024-02-09T18:06:20.414 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.414 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.414 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:20.414 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.089706+0000", 2024-02-09T18:06:20.414 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8310000000000003e-06 2024-02-09T18:06:20.414 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.414 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.414 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:20.415 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.089712+0000", 2024-02-09T18:06:20.415 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5840000000000002e-06 2024-02-09T18:06:20.415 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.415 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.415 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:20.415 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.090056+0000", 2024-02-09T18:06:20.415 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00034346799999999999 2024-02-09T18:06:20.415 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.415 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.415 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:20.415 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.090176+0000", 2024-02-09T18:06:20.415 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012014 2024-02-09T18:06:20.415 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.415 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.416 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-02-09T18:06:20.416 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.090294+0000", 2024-02-09T18:06:20.416 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011880200000000001 2024-02-09T18:06:20.416 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.416 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.416 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:20.416 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.117377+0000", 2024-02-09T18:06:20.416 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.027082812000000001 2024-02-09T18:06:20.416 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.416 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.416 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:20.416 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.312327+0000", 2024-02-09T18:06:20.416 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.194950186 2024-02-09T18:06:20.417 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.417 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.417 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:20.417 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.312377+0000", 2024-02-09T18:06:20.417 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9329999999999997e-05 2024-02-09T18:06:20.417 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.417 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.417 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:20.417 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.312428+0000", 2024-02-09T18:06:20.417 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1187000000000003e-05 2024-02-09T18:06:20.417 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:20.417 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:20.417 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:20.417 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.418 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.418 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:149670 8.4 8:2f78b43d:::benchmark_data_smithi179_148205_object26598:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e826)", 2024-02-09T18:06:20.418 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:57:26.089894+0000", 2024-02-09T18:06:20.418 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 534.31769582100003, 2024-02-09T18:06:20.418 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.22246386800000001, 2024-02-09T18:06:20.418 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:20.418 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:20.418 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:20.418 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:06:20.418 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:06:20.418 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 149670 2024-02-09T18:06:20.418 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.418 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:20.418 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.418 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:20.419 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.089894+0000", 2024-02-09T18:06:20.419 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:20.419 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.419 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.419 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:20.419 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.089894+0000", 2024-02-09T18:06:20.419 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:20.419 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.419 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.419 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:20.419 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.089894+0000", 2024-02-09T18:06:20.419 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6600000000000002e-07 2024-02-09T18:06:20.419 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.419 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.419 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:20.419 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.089897+0000", 2024-02-09T18:06:20.420 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5179999999999999e-06 2024-02-09T18:06:20.420 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.420 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.420 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:20.420 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.089903+0000", 2024-02-09T18:06:20.420 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1480000000000003e-06 2024-02-09T18:06:20.420 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.420 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.420 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:20.420 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.089908+0000", 2024-02-09T18:06:20.420 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3070000000000002e-06 2024-02-09T18:06:20.420 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.421 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.421 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:20.421 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.097177+0000", 2024-02-09T18:06:20.421 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0072688199999999996 2024-02-09T18:06:20.421 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.421 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.421 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:20.421 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.097255+0000", 2024-02-09T18:06:20.421 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7460999999999996e-05 2024-02-09T18:06:20.421 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.421 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.421 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,10", 2024-02-09T18:06:20.421 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.097316+0000", 2024-02-09T18:06:20.421 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1753000000000003e-05 2024-02-09T18:06:20.422 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.422 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.422 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:20.422 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.110956+0000", 2024-02-09T18:06:20.422 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013639347 2024-02-09T18:06:20.422 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.422 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.422 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:20.422 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.111147+0000", 2024-02-09T18:06:20.422 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00019126200000000001 2024-02-09T18:06:20.422 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.422 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.422 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:20.422 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.312274+0000", 2024-02-09T18:06:20.422 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.201127057 2024-02-09T18:06:20.422 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.423 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.423 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:20.423 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.312328+0000", 2024-02-09T18:06:20.423 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.376e-05 2024-02-09T18:06:20.423 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.423 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.423 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:20.423 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.312358+0000", 2024-02-09T18:06:20.423 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0069e-05 2024-02-09T18:06:20.423 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:20.423 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:20.423 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:20.423 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.423 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.423 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:149671 8.0 8:0e2f9c24:::benchmark_data_smithi179_148205_object26599:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e826)", 2024-02-09T18:06:20.424 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:57:26.089915+0000", 2024-02-09T18:06:20.424 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 534.31767499800003, 2024-02-09T18:06:20.424 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.22250051400000001, 2024-02-09T18:06:20.424 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:20.424 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:20.424 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:20.424 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:06:20.424 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:06:20.424 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 149671 2024-02-09T18:06:20.424 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.424 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:20.424 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.424 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:20.424 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.089915+0000", 2024-02-09T18:06:20.425 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:20.425 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.425 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.425 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:20.425 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.089915+0000", 2024-02-09T18:06:20.425 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:20.425 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.425 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.425 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:20.425 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.089915+0000", 2024-02-09T18:06:20.425 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7300000000000002e-07 2024-02-09T18:06:20.425 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.425 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.426 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:20.426 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.089917+0000", 2024-02-09T18:06:20.426 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.12e-06 2024-02-09T18:06:20.426 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.426 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.426 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:20.426 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.089923+0000", 2024-02-09T18:06:20.426 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8649999999999998e-06 2024-02-09T18:06:20.426 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.426 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.426 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:20.426 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.089928+0000", 2024-02-09T18:06:20.426 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1540000000000001e-06 2024-02-09T18:06:20.426 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.427 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.427 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:20.427 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.090406+0000", 2024-02-09T18:06:20.427 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00047733699999999999 2024-02-09T18:06:20.427 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.427 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.427 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:20.427 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.090515+0000", 2024-02-09T18:06:20.427 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000108859 2024-02-09T18:06:20.427 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.427 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.427 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-02-09T18:06:20.427 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.090613+0000", 2024-02-09T18:06:20.427 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.8345999999999997e-05 2024-02-09T18:06:20.427 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.428 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.428 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:20.428 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.117389+0000", 2024-02-09T18:06:20.428 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.026776399999999999 2024-02-09T18:06:20.428 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.428 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.428 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:20.428 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.312290+0000", 2024-02-09T18:06:20.428 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.19490034000000001 2024-02-09T18:06:20.428 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.428 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.428 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:20.428 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.312331+0000", 2024-02-09T18:06:20.428 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1572000000000001e-05 2024-02-09T18:06:20.429 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.429 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.429 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:20.429 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.312415+0000", 2024-02-09T18:06:20.429 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4147999999999995e-05 2024-02-09T18:06:20.429 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:20.429 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:20.429 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:20.429 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.429 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.429 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:149672 8.0 8:029caa0a:::benchmark_data_smithi179_148205_object26600:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e826)", 2024-02-09T18:06:20.429 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:57:26.092158+0000", 2024-02-09T18:06:20.429 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 534.31543198600002, 2024-02-09T18:06:20.429 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.22041646100000001, 2024-02-09T18:06:20.430 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:20.430 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:20.430 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:20.430 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:06:20.430 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:06:20.430 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 149672 2024-02-09T18:06:20.430 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.430 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:20.430 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.430 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:20.430 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.092158+0000", 2024-02-09T18:06:20.430 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:20.430 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.431 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.431 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:20.431 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.092158+0000", 2024-02-09T18:06:20.431 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:20.431 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.431 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.431 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:20.431 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.092159+0000", 2024-02-09T18:06:20.431 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.8299999999999995e-07 2024-02-09T18:06:20.431 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.431 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.431 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:20.431 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.092164+0000", 2024-02-09T18:06:20.431 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6619999999999996e-06 2024-02-09T18:06:20.432 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.432 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.432 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:20.432 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.092174+0000", 2024-02-09T18:06:20.432 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0008e-05 2024-02-09T18:06:20.432 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.432 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.432 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:20.432 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.092181+0000", 2024-02-09T18:06:20.432 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6459999999999993e-06 2024-02-09T18:06:20.432 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.432 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.432 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:20.432 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.099362+0000", 2024-02-09T18:06:20.433 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0071812400000000002 2024-02-09T18:06:20.433 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.433 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.433 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:20.433 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.099478+0000", 2024-02-09T18:06:20.433 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000115896 2024-02-09T18:06:20.433 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.433 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.433 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-02-09T18:06:20.433 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.099581+0000", 2024-02-09T18:06:20.433 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0001031 2024-02-09T18:06:20.433 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.433 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.433 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:20.433 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.117395+0000", 2024-02-09T18:06:20.434 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017813711999999999 2024-02-09T18:06:20.434 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.434 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.434 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:20.434 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.312450+0000", 2024-02-09T18:06:20.434 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.19505461199999999 2024-02-09T18:06:20.434 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.434 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.434 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:20.434 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.312496+0000", 2024-02-09T18:06:20.434 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6569000000000001e-05 2024-02-09T18:06:20.434 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.434 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.434 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:20.435 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.312574+0000", 2024-02-09T18:06:20.435 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8033000000000003e-05 2024-02-09T18:06:20.435 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:20.435 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:20.435 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:20.435 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.435 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.435 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:149675 8.4 8:274fa1cc:::benchmark_data_smithi179_148205_object26603:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e826)", 2024-02-09T18:06:20.435 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:57:26.099085+0000", 2024-02-09T18:06:20.435 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 534.30850531800002, 2024-02-09T18:06:20.435 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.22750145699999999, 2024-02-09T18:06:20.435 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:20.435 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:20.435 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:20.435 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:06:20.436 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:06:20.436 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 149675 2024-02-09T18:06:20.436 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.436 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:20.436 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.436 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:20.436 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.099085+0000", 2024-02-09T18:06:20.436 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:20.436 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.436 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.436 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:20.436 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.099085+0000", 2024-02-09T18:06:20.436 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:20.436 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.436 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.437 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:20.437 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.099086+0000", 2024-02-09T18:06:20.437 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.156e-06 2024-02-09T18:06:20.437 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.437 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.437 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:20.437 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.099090+0000", 2024-02-09T18:06:20.437 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7949999999999998e-06 2024-02-09T18:06:20.437 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.437 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.437 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:20.437 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.099101+0000", 2024-02-09T18:06:20.437 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0448999999999999e-05 2024-02-09T18:06:20.437 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.437 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.438 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:20.438 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.099108+0000", 2024-02-09T18:06:20.438 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.345e-06 2024-02-09T18:06:20.438 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.438 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.438 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:20.438 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.101512+0000", 2024-02-09T18:06:20.438 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0024040490000000001 2024-02-09T18:06:20.438 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.438 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.438 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:20.438 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.101623+0000", 2024-02-09T18:06:20.438 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000110485 2024-02-09T18:06:20.438 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.439 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.439 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,10", 2024-02-09T18:06:20.439 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.101729+0000", 2024-02-09T18:06:20.439 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000105788 2024-02-09T18:06:20.439 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.439 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.439 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:20.439 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.111034+0000", 2024-02-09T18:06:20.439 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0093058329999999995 2024-02-09T18:06:20.439 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.439 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.439 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:20.439 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.111185+0000", 2024-02-09T18:06:20.439 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015010199999999999 2024-02-09T18:06:20.439 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.440 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.440 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:20.440 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.326493+0000", 2024-02-09T18:06:20.440 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.215308465 2024-02-09T18:06:20.440 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.440 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.440 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:20.440 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.326548+0000", 2024-02-09T18:06:20.440 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4759999999999997e-05 2024-02-09T18:06:20.440 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.440 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.440 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:20.440 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.326586+0000", 2024-02-09T18:06:20.440 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8229999999999998e-05 2024-02-09T18:06:20.441 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:20.441 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:20.441 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:20.441 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.441 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.441 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:149677 8.0 8:0d59ad39:::benchmark_data_smithi179_148205_object26605:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e826)", 2024-02-09T18:06:20.441 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:57:26.099122+0000", 2024-02-09T18:06:20.441 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 534.30846804700002, 2024-02-09T18:06:20.441 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.224356316, 2024-02-09T18:06:20.441 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:20.441 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:20.441 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:20.441 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:06:20.441 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:06:20.442 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 149677 2024-02-09T18:06:20.442 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.442 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:20.442 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.442 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:20.442 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.099122+0000", 2024-02-09T18:06:20.442 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:20.442 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.442 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.442 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:20.442 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.099122+0000", 2024-02-09T18:06:20.442 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:20.442 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.442 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.442 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:20.443 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.099122+0000", 2024-02-09T18:06:20.443 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1399999999999997e-07 2024-02-09T18:06:20.443 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.443 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.443 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:20.443 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.099124+0000", 2024-02-09T18:06:20.443 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1900000000000002e-06 2024-02-09T18:06:20.443 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.443 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.443 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:20.443 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.099136+0000", 2024-02-09T18:06:20.443 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1308000000000001e-05 2024-02-09T18:06:20.443 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.443 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.444 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:20.444 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.099141+0000", 2024-02-09T18:06:20.444 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.446e-06 2024-02-09T18:06:20.444 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.444 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.444 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:20.444 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.104437+0000", 2024-02-09T18:06:20.444 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0052956260000000003 2024-02-09T18:06:20.444 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.444 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.444 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:20.444 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.104549+0000", 2024-02-09T18:06:20.444 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000111869 2024-02-09T18:06:20.444 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.445 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.445 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-02-09T18:06:20.445 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.104663+0000", 2024-02-09T18:06:20.445 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000114103 2024-02-09T18:06:20.445 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.445 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.445 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:20.445 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.117403+0000", 2024-02-09T18:06:20.445 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.012739968000000001 2024-02-09T18:06:20.445 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.445 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.445 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:20.445 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.323314+0000", 2024-02-09T18:06:20.445 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.205910818 2024-02-09T18:06:20.446 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.446 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.446 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:20.446 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.323374+0000", 2024-02-09T18:06:20.446 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0745000000000002e-05 2024-02-09T18:06:20.446 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.446 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.446 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:20.446 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.323478+0000", 2024-02-09T18:06:20.446 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000103829 2024-02-09T18:06:20.446 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:20.446 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:20.446 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:20.446 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.447 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.447 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:149679 8.8 8:12d0fffd:::benchmark_data_smithi179_148205_object26607:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e826)", 2024-02-09T18:06:20.447 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:57:26.099148+0000", 2024-02-09T18:06:20.447 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 534.308441507, 2024-02-09T18:06:20.447 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.22577888700000001, 2024-02-09T18:06:20.447 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:20.447 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:20.447 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:20.447 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:06:20.447 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:06:20.447 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 149679 2024-02-09T18:06:20.447 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.447 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:20.447 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.448 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:20.448 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.099148+0000", 2024-02-09T18:06:20.448 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:20.448 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.448 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.448 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:20.448 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.099148+0000", 2024-02-09T18:06:20.448 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:20.448 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.448 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.448 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:20.448 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.099149+0000", 2024-02-09T18:06:20.448 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1399999999999997e-07 2024-02-09T18:06:20.449 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.449 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.449 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:20.449 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.099151+0000", 2024-02-09T18:06:20.449 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5000000000000002e-06 2024-02-09T18:06:20.449 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.449 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.449 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:20.449 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.099157+0000", 2024-02-09T18:06:20.449 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4890000000000003e-06 2024-02-09T18:06:20.449 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.449 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.449 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:20.449 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.099161+0000", 2024-02-09T18:06:20.450 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4349999999999999e-06 2024-02-09T18:06:20.450 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.450 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.450 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:20.450 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.104420+0000", 2024-02-09T18:06:20.450 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0052591169999999998 2024-02-09T18:06:20.450 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.450 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.450 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:20.450 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.104559+0000", 2024-02-09T18:06:20.450 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013904999999999999 2024-02-09T18:06:20.450 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.450 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.451 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-02-09T18:06:20.451 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.104667+0000", 2024-02-09T18:06:20.451 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010785099999999999 2024-02-09T18:06:20.451 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.451 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.451 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:20.451 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.117746+0000", 2024-02-09T18:06:20.451 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013078601 2024-02-09T18:06:20.451 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.451 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.451 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:20.451 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.324784+0000", 2024-02-09T18:06:20.451 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.207038051 2024-02-09T18:06:20.451 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.452 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.452 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:20.452 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.324840+0000", 2024-02-09T18:06:20.452 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5949999999999998e-05 2024-02-09T18:06:20.452 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.452 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.452 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:20.452 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.324927+0000", 2024-02-09T18:06:20.452 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7429000000000006e-05 2024-02-09T18:06:20.452 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:20.452 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:20.452 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:20.452 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.453 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.453 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:149683 8.0 8:0a05b423:::benchmark_data_smithi179_148205_object26611:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e826)", 2024-02-09T18:06:20.453 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:57:26.100174+0000", 2024-02-09T18:06:20.453 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 534.30741631399997, 2024-02-09T18:06:20.453 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.232663442, 2024-02-09T18:06:20.453 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:20.453 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:20.453 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:20.453 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:06:20.453 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:06:20.453 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 149683 2024-02-09T18:06:20.453 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.453 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:20.453 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.454 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:20.454 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.100174+0000", 2024-02-09T18:06:20.454 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:20.454 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.454 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.454 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:20.454 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.100174+0000", 2024-02-09T18:06:20.454 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:20.454 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.454 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.454 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:20.454 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.100174+0000", 2024-02-09T18:06:20.454 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.6799999999999999e-07 2024-02-09T18:06:20.454 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.455 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.455 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:20.455 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.100179+0000", 2024-02-09T18:06:20.455 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6519999999999997e-06 2024-02-09T18:06:20.455 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.455 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.455 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:20.455 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.100193+0000", 2024-02-09T18:06:20.455 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4124000000000001e-05 2024-02-09T18:06:20.455 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.455 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.455 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:20.455 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.100201+0000", 2024-02-09T18:06:20.455 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.525e-06 2024-02-09T18:06:20.455 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.456 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.456 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:20.456 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.109190+0000", 2024-02-09T18:06:20.456 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0089890479999999995 2024-02-09T18:06:20.456 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.456 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.456 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:20.456 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.109299+0000", 2024-02-09T18:06:20.456 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000109423 2024-02-09T18:06:20.456 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.456 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.456 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-02-09T18:06:20.456 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.109403+0000", 2024-02-09T18:06:20.456 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000103639 2024-02-09T18:06:20.457 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.457 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.457 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:20.457 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.117761+0000", 2024-02-09T18:06:20.457 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0083584420000000006 2024-02-09T18:06:20.457 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.457 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.457 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:20.457 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.332703+0000", 2024-02-09T18:06:20.457 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.214942037 2024-02-09T18:06:20.457 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.457 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.457 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:20.457 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.332754+0000", 2024-02-09T18:06:20.457 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1082000000000003e-05 2024-02-09T18:06:20.458 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.458 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.458 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:20.458 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.332837+0000", 2024-02-09T18:06:20.458 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2602000000000006e-05 2024-02-09T18:06:20.458 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:20.458 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:20.458 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:20.458 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.458 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.458 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:149687 8.4 8:23f63abb:::benchmark_data_smithi179_148205_object26615:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e826)", 2024-02-09T18:06:20.458 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:57:26.106520+0000", 2024-02-09T18:06:20.458 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 534.30106945600005, 2024-02-09T18:06:20.458 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.22018114899999999, 2024-02-09T18:06:20.459 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:20.459 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:20.459 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:20.459 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:06:20.459 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:06:20.459 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 149687 2024-02-09T18:06:20.459 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.459 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:20.459 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.459 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:20.459 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.106520+0000", 2024-02-09T18:06:20.459 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:20.459 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.459 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.459 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:20.460 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.106520+0000", 2024-02-09T18:06:20.460 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:20.460 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.460 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.460 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:20.460 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.106522+0000", 2024-02-09T18:06:20.460 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.511e-06 2024-02-09T18:06:20.460 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.460 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.460 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:20.460 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.106527+0000", 2024-02-09T18:06:20.460 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5790000000000003e-06 2024-02-09T18:06:20.460 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.460 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.460 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:20.461 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.106539+0000", 2024-02-09T18:06:20.461 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1412e-05 2024-02-09T18:06:20.461 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.461 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.461 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:20.461 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.106548+0000", 2024-02-09T18:06:20.461 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.1190000000000003e-06 2024-02-09T18:06:20.461 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.461 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.461 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:20.461 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.106962+0000", 2024-02-09T18:06:20.461 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00041418999999999999 2024-02-09T18:06:20.461 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.461 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.461 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:20.462 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.107087+0000", 2024-02-09T18:06:20.462 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012501499999999999 2024-02-09T18:06:20.462 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.462 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.462 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,10", 2024-02-09T18:06:20.462 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.107204+0000", 2024-02-09T18:06:20.462 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011710299999999999 2024-02-09T18:06:20.462 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.462 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.462 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:20.462 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.111255+0000", 2024-02-09T18:06:20.462 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0040507959999999997 2024-02-09T18:06:20.462 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.462 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.462 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:20.462 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.117993+0000", 2024-02-09T18:06:20.463 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0067382750000000002 2024-02-09T18:06:20.463 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.463 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.463 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:20.463 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.326622+0000", 2024-02-09T18:06:20.463 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.20862879500000001 2024-02-09T18:06:20.463 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.463 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.463 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:20.463 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.326669+0000", 2024-02-09T18:06:20.463 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6864999999999999e-05 2024-02-09T18:06:20.463 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.463 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.463 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:20.463 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.326702+0000", 2024-02-09T18:06:20.464 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2489e-05 2024-02-09T18:06:20.464 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:20.464 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:20.464 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:20.464 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.464 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.464 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:149688 8.8 8:16d18b7b:::benchmark_data_smithi179_148205_object26616:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e826)", 2024-02-09T18:06:20.464 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:57:26.106564+0000", 2024-02-09T18:06:20.464 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 534.30102582699999, 2024-02-09T18:06:20.464 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.21854638300000001, 2024-02-09T18:06:20.464 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:20.464 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:20.464 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:20.464 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:06:20.465 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:06:20.465 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 149688 2024-02-09T18:06:20.465 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.465 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:20.465 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.465 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:20.465 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.106564+0000", 2024-02-09T18:06:20.465 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:20.465 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.465 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.465 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:20.465 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.106564+0000", 2024-02-09T18:06:20.465 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:20.465 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.466 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.466 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:20.466 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.106565+0000", 2024-02-09T18:06:20.466 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3799999999999997e-07 2024-02-09T18:06:20.466 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.466 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.466 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:20.466 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.106567+0000", 2024-02-09T18:06:20.466 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.666e-06 2024-02-09T18:06:20.466 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.466 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.466 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:20.466 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.106574+0000", 2024-02-09T18:06:20.466 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9450000000000004e-06 2024-02-09T18:06:20.467 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.467 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.467 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:20.467 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.106580+0000", 2024-02-09T18:06:20.467 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7049999999999996e-06 2024-02-09T18:06:20.467 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.467 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.467 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:20.467 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.112909+0000", 2024-02-09T18:06:20.467 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0063293580000000002 2024-02-09T18:06:20.467 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.467 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.467 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:20.467 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.113015+0000", 2024-02-09T18:06:20.468 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000105809 2024-02-09T18:06:20.468 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.468 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.468 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-02-09T18:06:20.468 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.113126+0000", 2024-02-09T18:06:20.468 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000110796 2024-02-09T18:06:20.468 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.468 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.468 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:20.468 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.117767+0000", 2024-02-09T18:06:20.468 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0046414200000000003 2024-02-09T18:06:20.468 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.468 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.468 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:20.469 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.324976+0000", 2024-02-09T18:06:20.469 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.207208697 2024-02-09T18:06:20.469 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.469 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.469 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:20.469 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.325029+0000", 2024-02-09T18:06:20.469 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2589000000000001e-05 2024-02-09T18:06:20.469 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.469 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.469 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:20.469 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.325110+0000", 2024-02-09T18:06:20.469 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1860000000000006e-05 2024-02-09T18:06:20.469 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:20.469 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:20.470 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:20.470 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.470 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.470 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:149694 8.0 8:03ca8ca8:::benchmark_data_smithi179_148205_object26622:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e826)", 2024-02-09T18:06:20.470 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:57:26.118252+0000", 2024-02-09T18:06:20.470 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 534.28933764999999, 2024-02-09T18:06:20.470 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.21477158800000001, 2024-02-09T18:06:20.470 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:20.470 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:20.470 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:20.470 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:06:20.470 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:06:20.470 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 149694 2024-02-09T18:06:20.470 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.471 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:20.471 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.471 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:20.471 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.118252+0000", 2024-02-09T18:06:20.471 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:20.471 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.471 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.471 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:20.471 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.118252+0000", 2024-02-09T18:06:20.471 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:20.471 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.471 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.471 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:20.471 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.118254+0000", 2024-02-09T18:06:20.472 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.404e-06 2024-02-09T18:06:20.472 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.472 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.472 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:20.472 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.118258+0000", 2024-02-09T18:06:20.472 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2760000000000002e-06 2024-02-09T18:06:20.472 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.472 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.472 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:20.472 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.118270+0000", 2024-02-09T18:06:20.472 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1809e-05 2024-02-09T18:06:20.472 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.472 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.472 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:20.473 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.118277+0000", 2024-02-09T18:06:20.473 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.3749999999999997e-06 2024-02-09T18:06:20.473 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.473 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.473 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:20.473 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.118338+0000", 2024-02-09T18:06:20.473 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0668000000000003e-05 2024-02-09T18:06:20.473 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.473 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.473 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:20.473 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.118472+0000", 2024-02-09T18:06:20.473 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013422099999999999 2024-02-09T18:06:20.473 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.473 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.473 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5", 2024-02-09T18:06:20.474 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.118542+0000", 2024-02-09T18:06:20.474 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.0389000000000007e-05 2024-02-09T18:06:20.474 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.474 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.474 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:20.474 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.122532+0000", 2024-02-09T18:06:20.474 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0039895210000000002 2024-02-09T18:06:20.474 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.474 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.474 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:20.474 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.332892+0000", 2024-02-09T18:06:20.474 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.21036048099999999 2024-02-09T18:06:20.474 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.474 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.474 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:20.475 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.332942+0000", 2024-02-09T18:06:20.475 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9916000000000001e-05 2024-02-09T18:06:20.475 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.475 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.475 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:20.475 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:57:26.333024+0000", 2024-02-09T18:06:20.475 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1527999999999996e-05 2024-02-09T18:06:20.475 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:20.475 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:20.475 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:20.475 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.475 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.475 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:227511 8.c 8:3becadfa:::benchmark_data_smithi179_148205_object104439:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e854)", 2024-02-09T18:06:20.475 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:59:11.946267+0000", 2024-02-09T18:06:20.475 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 428.46132309400002, 2024-02-09T18:06:20.476 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3075633039999999, 2024-02-09T18:06:20.476 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:20.476 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:20.476 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:20.476 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:06:20.476 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:06:20.476 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 227511 2024-02-09T18:06:20.476 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.476 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:20.476 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.476 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:20.476 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:11.946267+0000", 2024-02-09T18:06:20.476 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:20.476 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.476 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.477 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:20.477 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:11.946267+0000", 2024-02-09T18:06:20.477 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:20.477 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.477 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.477 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:20.477 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:11.946268+0000", 2024-02-09T18:06:20.477 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.519e-06 2024-02-09T18:06:20.477 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.477 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.477 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:20.477 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:11.946275+0000", 2024-02-09T18:06:20.477 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.409e-06 2024-02-09T18:06:20.477 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.478 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.478 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:20.478 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:11.946291+0000", 2024-02-09T18:06:20.478 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6192e-05 2024-02-09T18:06:20.478 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.478 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.478 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:20.478 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:11.946299+0000", 2024-02-09T18:06:20.478 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6890000000000004e-06 2024-02-09T18:06:20.478 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.478 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.478 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:20.478 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:11.946349+0000", 2024-02-09T18:06:20.478 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0858e-05 2024-02-09T18:06:20.479 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.479 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.479 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:20.479 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:11.946359+0000", 2024-02-09T18:06:20.479 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6530000000000006e-06 2024-02-09T18:06:20.479 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.479 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.479 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:20.479 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:11.996463+0000", 2024-02-09T18:06:20.479 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.050103901999999999 2024-02-09T18:06:20.479 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.479 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.479 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:20.479 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:11.996479+0000", 2024-02-09T18:06:20.480 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5543e-05 2024-02-09T18:06:20.480 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.480 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.480 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:20.480 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:13.243564+0000", 2024-02-09T18:06:20.480 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.247085767 2024-02-09T18:06:20.480 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.480 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.480 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:20.480 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:13.243672+0000", 2024-02-09T18:06:20.480 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000108046 2024-02-09T18:06:20.480 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.480 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.480 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,8", 2024-02-09T18:06:20.481 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:13.243778+0000", 2024-02-09T18:06:20.481 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000105598 2024-02-09T18:06:20.481 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.481 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.481 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:20.481 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:13.252865+0000", 2024-02-09T18:06:20.481 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0090875220000000007 2024-02-09T18:06:20.481 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.481 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.481 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:20.481 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:13.252881+0000", 2024-02-09T18:06:20.481 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5645e-05 2024-02-09T18:06:20.481 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.481 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.481 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:20.482 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:13.252912+0000", 2024-02-09T18:06:20.482 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0595000000000001e-05 2024-02-09T18:06:20.482 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.482 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.482 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:20.482 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:13.252974+0000", 2024-02-09T18:06:20.482 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2650999999999994e-05 2024-02-09T18:06:20.482 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.482 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.482 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:20.482 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:13.253830+0000", 2024-02-09T18:06:20.482 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00085571499999999995 2024-02-09T18:06:20.482 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:20.482 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:20.482 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:20.483 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.483 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.483 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:227524 8.c 8:35cb0d7d:::benchmark_data_smithi179_148205_object104452:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e855)", 2024-02-09T18:06:20.483 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:59:12.029962+0000", 2024-02-09T18:06:20.483 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 428.37762766700001, 2024-02-09T18:06:20.483 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2525532400000001, 2024-02-09T18:06:20.483 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:20.483 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:20.483 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:20.483 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:06:20.483 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:06:20.483 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 227524 2024-02-09T18:06:20.483 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.484 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:20.484 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.484 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:20.484 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:12.029962+0000", 2024-02-09T18:06:20.484 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:20.484 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.484 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.484 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:20.484 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:12.029962+0000", 2024-02-09T18:06:20.484 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:20.484 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.484 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.484 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:20.484 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:12.029964+0000", 2024-02-09T18:06:20.484 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.717e-06 2024-02-09T18:06:20.485 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.485 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.485 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:20.485 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:12.029972+0000", 2024-02-09T18:06:20.485 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1899999999999995e-06 2024-02-09T18:06:20.485 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.485 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.485 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:20.485 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:12.029991+0000", 2024-02-09T18:06:20.485 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9222e-05 2024-02-09T18:06:20.485 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.485 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.485 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:20.486 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:12.030003+0000", 2024-02-09T18:06:20.486 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1588e-05 2024-02-09T18:06:20.486 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.486 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.486 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:20.486 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:12.030073+0000", 2024-02-09T18:06:20.486 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9873e-05 2024-02-09T18:06:20.486 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.486 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.486 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:20.486 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:12.030089+0000", 2024-02-09T18:06:20.486 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.609e-05 2024-02-09T18:06:20.486 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.486 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.487 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:20.487 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:13.248789+0000", 2024-02-09T18:06:20.487 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2186999789999999 2024-02-09T18:06:20.487 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.487 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.487 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:20.487 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:13.248882+0000", 2024-02-09T18:06:20.487 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.3426999999999998e-05 2024-02-09T18:06:20.487 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.487 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.487 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,8", 2024-02-09T18:06:20.487 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:13.248976+0000", 2024-02-09T18:06:20.487 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.3539e-05 2024-02-09T18:06:20.487 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.488 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.488 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:20.488 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:13.253963+0000", 2024-02-09T18:06:20.488 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0049873740000000001 2024-02-09T18:06:20.488 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.488 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.488 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:20.488 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:13.258135+0000", 2024-02-09T18:06:20.488 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0041720459999999996 2024-02-09T18:06:20.488 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.488 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.488 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:20.488 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:13.281480+0000", 2024-02-09T18:06:20.489 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.023345267999999999 2024-02-09T18:06:20.489 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.489 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.489 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:20.489 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:13.281517+0000", 2024-02-09T18:06:20.489 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6962000000000003e-05 2024-02-09T18:06:20.489 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.489 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.489 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:20.489 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:13.282515+0000", 2024-02-09T18:06:20.489 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00099796499999999996 2024-02-09T18:06:20.490 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:20.490 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:20.490 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:20.490 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.490 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.490 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:227546 8.c 8:32c04115:::benchmark_data_smithi179_148205_object104474:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e855)", 2024-02-09T18:06:20.490 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:59:12.089341+0000", 2024-02-09T18:06:20.490 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 428.318249337, 2024-02-09T18:06:20.490 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.193855152, 2024-02-09T18:06:20.490 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:20.490 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:20.490 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:20.490 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:06:20.491 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:06:20.491 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 227546 2024-02-09T18:06:20.491 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.491 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:20.491 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.491 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:20.491 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:12.089341+0000", 2024-02-09T18:06:20.491 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:20.491 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.491 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.491 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:20.491 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:12.089341+0000", 2024-02-09T18:06:20.491 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:20.491 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.492 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.492 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:20.492 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:12.089342+0000", 2024-02-09T18:06:20.492 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4899999999999999e-06 2024-02-09T18:06:20.492 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.492 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.492 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:20.492 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:12.089348+0000", 2024-02-09T18:06:20.492 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6250000000000004e-06 2024-02-09T18:06:20.492 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.492 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.492 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:20.493 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:12.089361+0000", 2024-02-09T18:06:20.493 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2954e-05 2024-02-09T18:06:20.493 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.493 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.493 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:20.493 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:12.089368+0000", 2024-02-09T18:06:20.493 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.678e-06 2024-02-09T18:06:20.493 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.493 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.493 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:20.493 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:12.089429+0000", 2024-02-09T18:06:20.493 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1147000000000003e-05 2024-02-09T18:06:20.493 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.493 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.494 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:20.494 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:12.089439+0000", 2024-02-09T18:06:20.494 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7340000000000003e-06 2024-02-09T18:06:20.494 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.494 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.494 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:20.494 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:13.254000+0000", 2024-02-09T18:06:20.494 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1645605130000001 2024-02-09T18:06:20.494 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.494 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.494 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:20.494 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:13.254095+0000", 2024-02-09T18:06:20.494 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5802000000000002e-05 2024-02-09T18:06:20.494 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.495 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.495 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,8", 2024-02-09T18:06:20.495 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:13.254193+0000", 2024-02-09T18:06:20.495 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7171999999999997e-05 2024-02-09T18:06:20.495 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.495 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.495 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:20.495 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:13.265819+0000", 2024-02-09T18:06:20.495 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011626299999999999 2024-02-09T18:06:20.495 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.495 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.495 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:20.495 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:13.265885+0000", 2024-02-09T18:06:20.496 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6344e-05 2024-02-09T18:06:20.496 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.496 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.496 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:20.496 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:13.282563+0000", 2024-02-09T18:06:20.496 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.016677821999999998 2024-02-09T18:06:20.496 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.496 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.496 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:20.496 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:13.282600+0000", 2024-02-09T18:06:20.496 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6534e-05 2024-02-09T18:06:20.496 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.496 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.496 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:20.497 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:13.283196+0000", 2024-02-09T18:06:20.497 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000596037 2024-02-09T18:06:20.497 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:20.497 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:20.497 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:20.497 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.497 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.497 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:227550 8.c 8:368c29c6:::benchmark_data_smithi179_148205_object104478:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e855)", 2024-02-09T18:06:20.497 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:59:12.096470+0000", 2024-02-09T18:06:20.497 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 428.31111984500001, 2024-02-09T18:06:20.497 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1875665529999999, 2024-02-09T18:06:20.497 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:20.497 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:20.497 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:20.498 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:06:20.498 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:06:20.498 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 227550 2024-02-09T18:06:20.498 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.498 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:20.498 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.498 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:20.498 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:12.096470+0000", 2024-02-09T18:06:20.498 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:20.498 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.498 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.498 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:20.498 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:12.096470+0000", 2024-02-09T18:06:20.498 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:20.499 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.499 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.499 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:20.499 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:12.096474+0000", 2024-02-09T18:06:20.499 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.42e-06 2024-02-09T18:06:20.499 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.499 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.499 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:20.499 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:12.096481+0000", 2024-02-09T18:06:20.500 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.0589999999999997e-06 2024-02-09T18:06:20.500 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.500 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.500 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:20.500 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:12.096499+0000", 2024-02-09T18:06:20.500 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7408999999999999e-05 2024-02-09T18:06:20.500 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.500 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.500 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:20.500 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:12.096507+0000", 2024-02-09T18:06:20.500 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1810000000000001e-06 2024-02-09T18:06:20.500 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.500 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.500 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:20.501 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:12.096578+0000", 2024-02-09T18:06:20.501 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1169999999999998e-05 2024-02-09T18:06:20.501 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.501 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.501 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:20.501 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:12.096592+0000", 2024-02-09T18:06:20.501 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3436e-05 2024-02-09T18:06:20.501 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.501 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.501 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:20.501 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:13.258182+0000", 2024-02-09T18:06:20.501 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.161590481 2024-02-09T18:06:20.501 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.501 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.502 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:20.502 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:13.258330+0000", 2024-02-09T18:06:20.502 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014806699999999999 2024-02-09T18:06:20.502 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.502 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.502 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,8", 2024-02-09T18:06:20.502 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:13.258490+0000", 2024-02-09T18:06:20.502 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015984199999999999 2024-02-09T18:06:20.502 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.502 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.502 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:20.502 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:13.265876+0000", 2024-02-09T18:06:20.502 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0073859260000000001 2024-02-09T18:06:20.502 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.503 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.503 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:20.503 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:13.265889+0000", 2024-02-09T18:06:20.503 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2809999999999999e-05 2024-02-09T18:06:20.503 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.503 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.503 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:20.503 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:13.283317+0000", 2024-02-09T18:06:20.503 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017428484000000001 2024-02-09T18:06:20.503 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.503 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.503 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:20.503 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:13.283376+0000", 2024-02-09T18:06:20.503 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8761999999999997e-05 2024-02-09T18:06:20.504 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.504 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.504 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:20.504 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:13.284037+0000", 2024-02-09T18:06:20.504 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00066050600000000005 2024-02-09T18:06:20.504 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:20.504 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:20.504 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:20.504 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.504 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.504 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:227567 8.c 8:3c765ccb:::benchmark_data_smithi179_148205_object104495:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e855)", 2024-02-09T18:06:20.504 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:59:12.299756+0000", 2024-02-09T18:06:20.504 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 428.10783404699998, 2024-02-09T18:06:20.504 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.984905531, 2024-02-09T18:06:20.504 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:20.505 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:20.505 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:20.505 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:06:20.505 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:06:20.505 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 227567 2024-02-09T18:06:20.505 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.505 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:20.505 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.505 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:20.505 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:12.299756+0000", 2024-02-09T18:06:20.505 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:20.505 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.505 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.505 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:20.505 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:12.299756+0000", 2024-02-09T18:06:20.506 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:20.506 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.506 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.506 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:20.506 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:12.299757+0000", 2024-02-09T18:06:20.506 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.685e-06 2024-02-09T18:06:20.506 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.506 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.506 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:20.506 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:12.299765+0000", 2024-02-09T18:06:20.506 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5660000000000001e-06 2024-02-09T18:06:20.506 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.506 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.506 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:20.506 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:12.299783+0000", 2024-02-09T18:06:20.507 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8369e-05 2024-02-09T18:06:20.507 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.507 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.507 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:20.507 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:12.299792+0000", 2024-02-09T18:06:20.507 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.0499999999999997e-06 2024-02-09T18:06:20.507 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.507 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.507 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:20.507 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:12.299869+0000", 2024-02-09T18:06:20.507 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6328999999999994e-05 2024-02-09T18:06:20.507 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.507 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.507 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:20.508 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:12.299879+0000", 2024-02-09T18:06:20.508 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0242e-05 2024-02-09T18:06:20.508 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.508 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.508 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:20.508 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:13.262023+0000", 2024-02-09T18:06:20.508 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.96214388100000003 2024-02-09T18:06:20.508 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.508 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.508 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:20.508 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:13.262164+0000", 2024-02-09T18:06:20.508 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000141116 2024-02-09T18:06:20.508 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.508 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.509 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,8", 2024-02-09T18:06:20.509 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:13.262284+0000", 2024-02-09T18:06:20.509 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000119634 2024-02-09T18:06:20.509 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.509 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.509 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:20.509 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:13.265957+0000", 2024-02-09T18:06:20.509 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0036734139999999998 2024-02-09T18:06:20.509 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.509 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.509 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:20.509 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:13.266304+0000", 2024-02-09T18:06:20.509 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00034735300000000001 2024-02-09T18:06:20.509 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.509 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.510 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:20.510 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:13.284102+0000", 2024-02-09T18:06:20.510 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017798054000000001 2024-02-09T18:06:20.510 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.510 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.510 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:20.510 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:13.284165+0000", 2024-02-09T18:06:20.510 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2199000000000005e-05 2024-02-09T18:06:20.510 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.510 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.510 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:20.510 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:13.284661+0000", 2024-02-09T18:06:20.510 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000496639 2024-02-09T18:06:20.510 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:20.511 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:20.511 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:20.511 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.511 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.511 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:240787 9.3 9:c89671ad:::benchmark_data_smithi179_161915_object105905:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e966)", 2024-02-09T18:06:20.511 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:04:07.661791+0000", 2024-02-09T18:06:20.511 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 132.74579850200001, 2024-02-09T18:06:20.511 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.20862383100000001, 2024-02-09T18:06:20.511 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:20.511 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:20.511 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:20.511 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:06:20.511 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:20.511 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 240787 2024-02-09T18:06:20.512 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.512 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:20.512 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.512 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:20.512 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:07.661791+0000", 2024-02-09T18:06:20.512 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:20.512 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.512 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.512 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:20.512 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:07.661791+0000", 2024-02-09T18:06:20.512 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:20.512 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.512 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.513 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:20.513 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:07.661793+0000", 2024-02-09T18:06:20.513 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.592e-06 2024-02-09T18:06:20.513 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.513 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.513 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:20.513 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:07.661799+0000", 2024-02-09T18:06:20.513 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1240000000000002e-06 2024-02-09T18:06:20.513 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.513 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.513 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:20.513 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:07.661813+0000", 2024-02-09T18:06:20.513 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3628999999999999e-05 2024-02-09T18:06:20.514 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.514 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.514 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:20.514 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:07.661823+0000", 2024-02-09T18:06:20.514 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0376e-05 2024-02-09T18:06:20.514 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.514 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.514 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:20.514 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:07.661882+0000", 2024-02-09T18:06:20.514 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9237999999999998e-05 2024-02-09T18:06:20.514 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.514 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.514 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:20.514 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:07.662011+0000", 2024-02-09T18:06:20.515 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000129188 2024-02-09T18:06:20.515 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.515 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.515 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,5", 2024-02-09T18:06:20.515 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:07.662148+0000", 2024-02-09T18:06:20.515 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013698200000000001 2024-02-09T18:06:20.515 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.515 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.515 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:20.515 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:07.666292+0000", 2024-02-09T18:06:20.515 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0041434130000000003 2024-02-09T18:06:20.515 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.515 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.515 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:20.515 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:07.667697+0000", 2024-02-09T18:06:20.516 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0014051280000000001 2024-02-09T18:06:20.516 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.516 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.516 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:20.516 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:07.870343+0000", 2024-02-09T18:06:20.516 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.202646043 2024-02-09T18:06:20.516 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.516 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.516 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:20.516 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:07.870380+0000", 2024-02-09T18:06:20.516 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6937000000000002e-05 2024-02-09T18:06:20.516 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.517 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.517 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:20.517 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:07.870415+0000", 2024-02-09T18:06:20.517 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.5181000000000001e-05 2024-02-09T18:06:20.517 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:20.517 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:20.517 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:20.517 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.517 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.517 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:241987 9.3 9:c19ba1d3:::benchmark_data_smithi179_161915_object107105:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e966)", 2024-02-09T18:06:20.517 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:04:09.127975+0000", 2024-02-09T18:06:20.517 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 131.27961481599999, 2024-02-09T18:06:20.517 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.223194105, 2024-02-09T18:06:20.517 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:20.517 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:20.518 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:20.518 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:06:20.518 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:20.518 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 241987 2024-02-09T18:06:20.518 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.518 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:20.518 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.518 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:20.518 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:09.127975+0000", 2024-02-09T18:06:20.518 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:20.518 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.518 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.518 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:20.519 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:09.127975+0000", 2024-02-09T18:06:20.519 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:20.519 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.519 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.519 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:20.519 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:09.127977+0000", 2024-02-09T18:06:20.519 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.528e-06 2024-02-09T18:06:20.519 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.519 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.519 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:20.519 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:09.127981+0000", 2024-02-09T18:06:20.519 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9420000000000003e-06 2024-02-09T18:06:20.519 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.519 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.519 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:20.520 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:09.127996+0000", 2024-02-09T18:06:20.520 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4093e-05 2024-02-09T18:06:20.520 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.520 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.520 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:20.520 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:09.128004+0000", 2024-02-09T18:06:20.520 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4370000000000007e-06 2024-02-09T18:06:20.520 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.520 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.520 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:20.520 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:09.133264+0000", 2024-02-09T18:06:20.520 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00526027 2024-02-09T18:06:20.520 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.521 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.521 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:20.521 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:09.133408+0000", 2024-02-09T18:06:20.521 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000143478 2024-02-09T18:06:20.521 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.521 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.521 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,5", 2024-02-09T18:06:20.521 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:09.134228+0000", 2024-02-09T18:06:20.521 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00082060800000000001 2024-02-09T18:06:20.521 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.521 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.521 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:20.521 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:09.140331+0000", 2024-02-09T18:06:20.521 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0061022960000000001 2024-02-09T18:06:20.522 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.522 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.522 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:20.522 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:09.152252+0000", 2024-02-09T18:06:20.522 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.01192092 2024-02-09T18:06:20.522 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.522 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.522 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:20.522 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:09.351063+0000", 2024-02-09T18:06:20.522 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.198811345 2024-02-09T18:06:20.522 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.522 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.522 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:20.522 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:09.351122+0000", 2024-02-09T18:06:20.523 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9076999999999997e-05 2024-02-09T18:06:20.523 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.523 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.523 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:20.523 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:09.351169+0000", 2024-02-09T18:06:20.523 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7110999999999997e-05 2024-02-09T18:06:20.523 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:20.523 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:20.523 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:20.523 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.523 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.523 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:241991 9.3 9:c94e008d:::benchmark_data_smithi179_161915_object107109:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e966)", 2024-02-09T18:06:20.523 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:04:09.128602+0000", 2024-02-09T18:06:20.523 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 131.27898812699999, 2024-02-09T18:06:20.524 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.22268842999999999, 2024-02-09T18:06:20.524 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:20.524 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:20.524 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:20.524 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:06:20.524 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:20.524 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 241991 2024-02-09T18:06:20.524 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.524 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:20.524 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.524 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:20.524 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:09.128602+0000", 2024-02-09T18:06:20.524 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:20.525 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.525 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.525 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:20.525 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:09.128602+0000", 2024-02-09T18:06:20.525 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:20.525 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.525 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.525 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:20.525 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:09.128603+0000", 2024-02-09T18:06:20.525 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4099999999999997e-07 2024-02-09T18:06:20.525 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.525 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.525 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:20.525 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:09.128607+0000", 2024-02-09T18:06:20.525 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.899e-06 2024-02-09T18:06:20.526 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.526 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.526 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:20.526 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:09.128619+0000", 2024-02-09T18:06:20.526 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1642e-05 2024-02-09T18:06:20.526 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.526 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.526 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:20.526 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:09.128626+0000", 2024-02-09T18:06:20.526 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1239999999999999e-06 2024-02-09T18:06:20.526 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.526 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.526 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:20.526 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:09.140351+0000", 2024-02-09T18:06:20.527 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011724887 2024-02-09T18:06:20.527 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.527 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.527 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:20.527 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:09.140468+0000", 2024-02-09T18:06:20.527 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011735699999999999 2024-02-09T18:06:20.527 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.527 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.527 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,5", 2024-02-09T18:06:20.527 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:09.140586+0000", 2024-02-09T18:06:20.527 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000117403 2024-02-09T18:06:20.527 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.527 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.527 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:20.527 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:09.150655+0000", 2024-02-09T18:06:20.528 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010068641 2024-02-09T18:06:20.528 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.528 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.528 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:20.528 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:09.152305+0000", 2024-02-09T18:06:20.528 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0016509129999999999 2024-02-09T18:06:20.528 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.528 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.528 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:20.528 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:09.351214+0000", 2024-02-09T18:06:20.528 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.198908999 2024-02-09T18:06:20.528 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.528 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.528 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:20.529 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:09.351253+0000", 2024-02-09T18:06:20.529 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8538999999999998e-05 2024-02-09T18:06:20.529 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.529 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.529 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:20.529 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:09.351290+0000", 2024-02-09T18:06:20.529 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7184999999999997e-05 2024-02-09T18:06:20.529 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:20.529 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:20.529 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:20.529 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.529 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.529 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:242000 9.3 9:c48756f7:::benchmark_data_smithi179_161915_object107118:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e966)", 2024-02-09T18:06:20.529 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:04:09.144592+0000", 2024-02-09T18:06:20.530 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 131.26299736000001, 2024-02-09T18:06:20.530 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.21127868399999999, 2024-02-09T18:06:20.530 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:20.530 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:20.530 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:20.530 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:06:20.530 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:20.530 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 242000 2024-02-09T18:06:20.530 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.530 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:20.530 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.530 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:20.530 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:09.144592+0000", 2024-02-09T18:06:20.530 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:20.530 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.531 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.531 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:20.531 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:09.144592+0000", 2024-02-09T18:06:20.531 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:20.531 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.531 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.531 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:20.531 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:09.144594+0000", 2024-02-09T18:06:20.531 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5880000000000001e-06 2024-02-09T18:06:20.531 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.531 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.531 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:20.531 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:09.144599+0000", 2024-02-09T18:06:20.531 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2730000000000002e-06 2024-02-09T18:06:20.531 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.532 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.532 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:20.532 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:09.144611+0000", 2024-02-09T18:06:20.532 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.183e-05 2024-02-09T18:06:20.532 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.532 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.532 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:20.532 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:09.144619+0000", 2024-02-09T18:06:20.532 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8930000000000005e-06 2024-02-09T18:06:20.532 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.532 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.532 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:20.532 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:09.145075+0000", 2024-02-09T18:06:20.532 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00045599699999999998 2024-02-09T18:06:20.533 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.533 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.533 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:20.533 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:09.145238+0000", 2024-02-09T18:06:20.533 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000162962 2024-02-09T18:06:20.533 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.533 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.533 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,5", 2024-02-09T18:06:20.533 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:09.145386+0000", 2024-02-09T18:06:20.533 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014821699999999999 2024-02-09T18:06:20.533 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.533 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.533 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:20.533 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:09.150852+0000", 2024-02-09T18:06:20.534 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0054658399999999996 2024-02-09T18:06:20.534 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.534 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.534 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:20.534 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:09.152586+0000", 2024-02-09T18:06:20.534 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0017340050000000001 2024-02-09T18:06:20.534 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.534 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.534 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:20.534 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:09.355775+0000", 2024-02-09T18:06:20.534 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.203188483 2024-02-09T18:06:20.534 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.534 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.534 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:20.534 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:09.355830+0000", 2024-02-09T18:06:20.535 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5590000000000001e-05 2024-02-09T18:06:20.535 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:20.535 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:20.535 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:20.535 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:09.355871+0000", 2024-02-09T18:06:20.535 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1006e-05 2024-02-09T18:06:20.535 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:20.535 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:20.535 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:20.535 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:20.535 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:20.535 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T18:06:20.536 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.9 to 1 (10000) 2024-02-09T18:06:20.701 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:20 smithi179 ceph-mon[43975]: pgmap v2507: 31 pgs: 1 peering, 30 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:20.701 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:20 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2331091226' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-02-09T18:06:20.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:20 smithi099 ceph-mon[39420]: pgmap v2507: 31 pgs: 1 peering, 30 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:20.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:20 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2331091226' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-02-09T18:06:20.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:20 smithi162 ceph-mon[44304]: pgmap v2507: 31 pgs: 1 peering, 30 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:20.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:20 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2331091226' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-02-09T18:06:21.059 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.10 dump_ops_in_flight 2024-02-09T18:06:21.293 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd primary-affinity 9 1 2024-02-09T18:06:21.320 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T18:06:21.559 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:06:21.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:21 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2331091226' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-02-09T18:06:21.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:21 smithi099 ceph-mon[39420]: osdmap e1021: 12 total, 12 up, 12 in 2024-02-09T18:06:21.838 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:21 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2331091226' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-02-09T18:06:21.838 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:21 smithi179 ceph-mon[43975]: osdmap e1021: 12 total, 12 up, 12 in 2024-02-09T18:06:21.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:21 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2331091226' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-02-09T18:06:21.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:21 smithi162 ceph-mon[44304]: osdmap e1021: 12 total, 12 up, 12 in 2024-02-09T18:06:22.643 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:22 smithi179 ceph-mon[43975]: pgmap v2509: 31 pgs: 1 peering, 30 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:22.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:22 smithi099 ceph-mon[39420]: pgmap v2509: 31 pgs: 1 peering, 30 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:22.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:22 smithi162 ceph-mon[44304]: pgmap v2509: 31 pgs: 1 peering, 30 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:23.269 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T18:06:23.270 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-02-09T18:06:23.270 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 0 2024-02-09T18:06:23.270 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T18:06:23.598 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:23 smithi179 ceph-mon[43975]: 1.9 scrub starts 2024-02-09T18:06:23.598 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:23 smithi179 ceph-mon[43975]: 1.9 scrub ok 2024-02-09T18:06:23.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:23 smithi099 ceph-mon[39420]: 1.9 scrub starts 2024-02-09T18:06:23.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:23 smithi099 ceph-mon[39420]: 1.9 scrub ok 2024-02-09T18:06:23.904 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.10 dump_blocked_ops 2024-02-09T18:06:23.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:23 smithi162 ceph-mon[44304]: 1.9 scrub starts 2024-02-09T18:06:23.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:23 smithi162 ceph-mon[44304]: 1.9 scrub ok 2024-02-09T18:06:24.174 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T18:06:24.504 INFO:teuthology.orchestra.run.smithi099.stderr:set osd.9 primary-affinity to 1 (8655362) 2024-02-09T18:06:24.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:24 smithi099 ceph-mon[39420]: 1.8 scrub starts 2024-02-09T18:06:24.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:24 smithi099 ceph-mon[39420]: 1.6 scrub starts 2024-02-09T18:06:24.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:24 smithi099 ceph-mon[39420]: 1.15 deep-scrub starts 2024-02-09T18:06:24.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:24 smithi099 ceph-mon[39420]: 1.8 scrub ok 2024-02-09T18:06:24.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:24 smithi099 ceph-mon[39420]: 1.6 scrub ok 2024-02-09T18:06:24.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:24 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:06:24.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:24 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:06:24.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:24 smithi099 ceph-mon[39420]: 1.15 deep-scrub ok 2024-02-09T18:06:24.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:24 smithi099 ceph-mon[39420]: pgmap v2510: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:24.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:24 smithi099 ceph-mon[39420]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-09T18:06:24.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:24 smithi099 ceph-mon[39420]: Cluster is now healthy 2024-02-09T18:06:24.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:24 smithi099 ceph-mon[39420]: osdmap e1022: 12 total, 12 up, 12 in 2024-02-09T18:06:24.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:24 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/540730297' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]: dispatch 2024-02-09T18:06:24.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:24 smithi179 ceph-mon[43975]: 1.8 scrub starts 2024-02-09T18:06:24.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:24 smithi179 ceph-mon[43975]: 1.6 scrub starts 2024-02-09T18:06:24.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:24 smithi179 ceph-mon[43975]: 1.15 deep-scrub starts 2024-02-09T18:06:24.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:24 smithi179 ceph-mon[43975]: 1.8 scrub ok 2024-02-09T18:06:24.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:24 smithi179 ceph-mon[43975]: 1.6 scrub ok 2024-02-09T18:06:24.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:24 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:06:24.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:24 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:06:24.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:24 smithi179 ceph-mon[43975]: 1.15 deep-scrub ok 2024-02-09T18:06:24.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:24 smithi179 ceph-mon[43975]: pgmap v2510: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:24.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:24 smithi179 ceph-mon[43975]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-09T18:06:24.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:24 smithi179 ceph-mon[43975]: Cluster is now healthy 2024-02-09T18:06:24.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:24 smithi179 ceph-mon[43975]: osdmap e1022: 12 total, 12 up, 12 in 2024-02-09T18:06:24.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:24 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/540730297' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]: dispatch 2024-02-09T18:06:24.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:24 smithi162 ceph-mon[44304]: 1.8 scrub starts 2024-02-09T18:06:24.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:24 smithi162 ceph-mon[44304]: 1.6 scrub starts 2024-02-09T18:06:24.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:24 smithi162 ceph-mon[44304]: 1.15 deep-scrub starts 2024-02-09T18:06:24.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:24 smithi162 ceph-mon[44304]: 1.8 scrub ok 2024-02-09T18:06:24.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:24 smithi162 ceph-mon[44304]: 1.6 scrub ok 2024-02-09T18:06:24.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:24 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:06:24.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:24 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:06:24.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:24 smithi162 ceph-mon[44304]: 1.15 deep-scrub ok 2024-02-09T18:06:24.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:24 smithi162 ceph-mon[44304]: pgmap v2510: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:24.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:24 smithi162 ceph-mon[44304]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-09T18:06:24.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:24 smithi162 ceph-mon[44304]: Cluster is now healthy 2024-02-09T18:06:24.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:24 smithi162 ceph-mon[44304]: osdmap e1022: 12 total, 12 up, 12 in 2024-02-09T18:06:24.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:24 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/540730297' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]: dispatch 2024-02-09T18:06:25.430 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 10 1 2024-02-09T18:06:25.695 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:06:25.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:25 smithi099 ceph-mon[39420]: 1.11 scrub starts 2024-02-09T18:06:25.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:25 smithi099 ceph-mon[39420]: 1.11 scrub ok 2024-02-09T18:06:25.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:25 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/540730297' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-02-09T18:06:25.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:25 smithi099 ceph-mon[39420]: osdmap e1023: 12 total, 12 up, 12 in 2024-02-09T18:06:25.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:25 smithi099 ceph-mon[39420]: 1.13 scrub starts 2024-02-09T18:06:25.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:25 smithi099 ceph-mon[39420]: 1.13 scrub ok 2024-02-09T18:06:25.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:25 smithi179 ceph-mon[43975]: 1.11 scrub starts 2024-02-09T18:06:25.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:25 smithi179 ceph-mon[43975]: 1.11 scrub ok 2024-02-09T18:06:25.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:25 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/540730297' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-02-09T18:06:25.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:25 smithi179 ceph-mon[43975]: osdmap e1023: 12 total, 12 up, 12 in 2024-02-09T18:06:25.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:25 smithi179 ceph-mon[43975]: 1.13 scrub starts 2024-02-09T18:06:25.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:25 smithi179 ceph-mon[43975]: 1.13 scrub ok 2024-02-09T18:06:25.924 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T18:06:25.924 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-02-09T18:06:25.924 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-02-09T18:06:25.924 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-02-09T18:06:25.924 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T18:06:25.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:25 smithi162 ceph-mon[44304]: 1.11 scrub starts 2024-02-09T18:06:25.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:25 smithi162 ceph-mon[44304]: 1.11 scrub ok 2024-02-09T18:06:25.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:25 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/540730297' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-02-09T18:06:25.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:25 smithi162 ceph-mon[44304]: osdmap e1023: 12 total, 12 up, 12 in 2024-02-09T18:06:25.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:25 smithi162 ceph-mon[44304]: 1.13 scrub starts 2024-02-09T18:06:25.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:25 smithi162 ceph-mon[44304]: 1.13 scrub ok 2024-02-09T18:06:26.600 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:26 smithi179 ceph-mon[43975]: pgmap v2513: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:26.629 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.10 dump_historic_ops 2024-02-09T18:06:26.890 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T18:06:26.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:26 smithi162 ceph-mon[44304]: pgmap v2513: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:26.936 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:26 smithi099 ceph-mon[39420]: pgmap v2513: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:27.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:27 smithi179 ceph-mon[43975]: 1.0 scrub starts 2024-02-09T18:06:27.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:27 smithi179 ceph-mon[43975]: 1.0 scrub ok 2024-02-09T18:06:27.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:27 smithi179 ceph-mon[43975]: 1.1 scrub starts 2024-02-09T18:06:27.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:27 smithi179 ceph-mon[43975]: 1.1 scrub ok 2024-02-09T18:06:27.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:27 smithi162 ceph-mon[44304]: 1.0 scrub starts 2024-02-09T18:06:27.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:27 smithi162 ceph-mon[44304]: 1.0 scrub ok 2024-02-09T18:06:27.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:27 smithi162 ceph-mon[44304]: 1.1 scrub starts 2024-02-09T18:06:27.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:27 smithi162 ceph-mon[44304]: 1.1 scrub ok 2024-02-09T18:06:27.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:27 smithi099 ceph-mon[39420]: 1.0 scrub starts 2024-02-09T18:06:27.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:27 smithi099 ceph-mon[39420]: 1.0 scrub ok 2024-02-09T18:06:27.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:27 smithi099 ceph-mon[39420]: 1.1 scrub starts 2024-02-09T18:06:27.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:27 smithi099 ceph-mon[39420]: 1.1 scrub ok 2024-02-09T18:06:28.516 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.10 to 1 (10000) 2024-02-09T18:06:28.768 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T18:06:28.768 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-02-09T18:06:28.768 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-02-09T18:06:28.769 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-02-09T18:06:28.769 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.769 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:236004 8.2 8:40ef1fe3:::benchmark_data_smithi179_148205_object112932:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e862)", 2024-02-09T18:06:28.769 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:59:26.375049+0000", 2024-02-09T18:06:28.769 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 422.38982006399999, 2024-02-09T18:06:28.769 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8983950969999999, 2024-02-09T18:06:28.769 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:28.769 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-02-09T18:06:28.769 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:28.769 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:06:28.769 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:06:28.769 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 236004 2024-02-09T18:06:28.769 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.769 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:28.770 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.770 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:28.770 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.375049+0000", 2024-02-09T18:06:28.770 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:28.770 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.770 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.770 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:28.770 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.375049+0000", 2024-02-09T18:06:28.770 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:28.770 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.770 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.770 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:28.770 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.375049+0000", 2024-02-09T18:06:28.770 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8799999999999998e-07 2024-02-09T18:06:28.771 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.771 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.771 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:28.771 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.375052+0000", 2024-02-09T18:06:28.771 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2220000000000001e-06 2024-02-09T18:06:28.771 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.771 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.771 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:28.771 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.375058+0000", 2024-02-09T18:06:28.771 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1879999999999999e-06 2024-02-09T18:06:28.771 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.771 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.771 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:28.771 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.375064+0000", 2024-02-09T18:06:28.771 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7049999999999996e-06 2024-02-09T18:06:28.772 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.772 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.772 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:28.772 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.386004+0000", 2024-02-09T18:06:28.772 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010940818 2024-02-09T18:06:28.772 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.772 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.772 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:28.772 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.386116+0000", 2024-02-09T18:06:28.772 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000111963 2024-02-09T18:06:28.772 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.772 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.772 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,4", 2024-02-09T18:06:28.773 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.386236+0000", 2024-02-09T18:06:28.773 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000119412 2024-02-09T18:06:28.773 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.773 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.773 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:28.773 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.393404+0000", 2024-02-09T18:06:28.773 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0071677149999999999 2024-02-09T18:06:28.773 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.773 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.773 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:28.773 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.393430+0000", 2024-02-09T18:06:28.773 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6991e-05 2024-02-09T18:06:28.773 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.773 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.774 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:28.774 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.407175+0000", 2024-02-09T18:06:28.774 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013744605999999999 2024-02-09T18:06:28.774 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.774 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.774 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:28.774 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.407189+0000", 2024-02-09T18:06:28.774 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4334e-05 2024-02-09T18:06:28.774 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.774 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.774 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:28.774 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.006142+0000", 2024-02-09T18:06:28.774 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.59895221300000001 2024-02-09T18:06:28.774 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.774 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.775 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:28.775 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.006155+0000", 2024-02-09T18:06:28.775 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3543000000000001e-05 2024-02-09T18:06:28.775 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.775 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.775 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:28.775 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.273348+0000", 2024-02-09T18:06:28.775 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2671923569999999 2024-02-09T18:06:28.775 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.775 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.775 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:28.775 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.273444+0000", 2024-02-09T18:06:28.775 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6642000000000001e-05 2024-02-09T18:06:28.775 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:28.776 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:28.776 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:28.776 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.776 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.776 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:236026 8.5 8:ad941cae:::benchmark_data_smithi179_148205_object112954:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e863)", 2024-02-09T18:06:28.776 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:59:26.414625+0000", 2024-02-09T18:06:28.776 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 422.35024412799999, 2024-02-09T18:06:28.776 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.834063314, 2024-02-09T18:06:28.776 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:28.776 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:28.776 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:28.776 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:06:28.776 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:06:28.776 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 236026 2024-02-09T18:06:28.777 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.777 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:28.777 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.777 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:28.777 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.414625+0000", 2024-02-09T18:06:28.777 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:28.777 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.777 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.777 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:28.777 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.414625+0000", 2024-02-09T18:06:28.777 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:28.777 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.777 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.777 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:28.777 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.414626+0000", 2024-02-09T18:06:28.778 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4109999999999999e-06 2024-02-09T18:06:28.778 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.778 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.778 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:28.778 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.414632+0000", 2024-02-09T18:06:28.778 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3550000000000004e-06 2024-02-09T18:06:28.778 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.778 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.778 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:28.778 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.414644+0000", 2024-02-09T18:06:28.778 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1849e-05 2024-02-09T18:06:28.778 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.778 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.778 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:28.778 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.414652+0000", 2024-02-09T18:06:28.779 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.9349999999999994e-06 2024-02-09T18:06:28.779 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.779 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.779 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:28.779 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.414722+0000", 2024-02-09T18:06:28.779 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9941e-05 2024-02-09T18:06:28.779 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.779 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.779 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:28.779 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.414732+0000", 2024-02-09T18:06:28.779 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0932e-05 2024-02-09T18:06:28.779 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.779 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.779 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:28.779 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.007379+0000", 2024-02-09T18:06:28.780 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.59264656999999998 2024-02-09T18:06:28.780 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.780 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.780 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:28.780 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.007390+0000", 2024-02-09T18:06:28.780 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1085e-05 2024-02-09T18:06:28.780 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.780 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.780 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:28.780 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.241566+0000", 2024-02-09T18:06:28.780 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2341757929999999 2024-02-09T18:06:28.780 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.780 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.780 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:28.781 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.241697+0000", 2024-02-09T18:06:28.781 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013109800000000001 2024-02-09T18:06:28.781 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.781 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.781 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4,11", 2024-02-09T18:06:28.781 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.241780+0000", 2024-02-09T18:06:28.781 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3157000000000004e-05 2024-02-09T18:06:28.781 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.781 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.781 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:28.781 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.248365+0000", 2024-02-09T18:06:28.781 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0065845749999999996 2024-02-09T18:06:28.781 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.781 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.781 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:28.782 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.248386+0000", 2024-02-09T18:06:28.782 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1569999999999998e-05 2024-02-09T18:06:28.782 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.782 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.782 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:28.782 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.248446+0000", 2024-02-09T18:06:28.782 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9531999999999999e-05 2024-02-09T18:06:28.782 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.782 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.782 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:28.782 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.248513+0000", 2024-02-09T18:06:28.782 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7459000000000006e-05 2024-02-09T18:06:28.782 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.782 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.783 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:28.783 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.248580+0000", 2024-02-09T18:06:28.783 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6608999999999999e-05 2024-02-09T18:06:28.783 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.783 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.783 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:28.783 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.248688+0000", 2024-02-09T18:06:28.783 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010844300000000001 2024-02-09T18:06:28.783 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:28.783 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:28.783 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:28.783 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.783 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.783 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:236033 8.5 8:ad8e1f4f:::benchmark_data_smithi179_148205_object112961:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e863)", 2024-02-09T18:06:28.784 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:59:26.428459+0000", 2024-02-09T18:06:28.784 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 422.33640990700002, 2024-02-09T18:06:28.784 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8368791660000001, 2024-02-09T18:06:28.784 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:28.784 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:28.784 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:28.784 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:06:28.784 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:06:28.784 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 236033 2024-02-09T18:06:28.784 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.784 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:28.784 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.784 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:28.784 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.428459+0000", 2024-02-09T18:06:28.785 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:28.785 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.785 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.785 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:28.785 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.428459+0000", 2024-02-09T18:06:28.785 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:28.785 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.785 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.785 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:28.785 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.428461+0000", 2024-02-09T18:06:28.785 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0080000000000001e-06 2024-02-09T18:06:28.785 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.785 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.785 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:28.785 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.428469+0000", 2024-02-09T18:06:28.786 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2199999999999992e-06 2024-02-09T18:06:28.786 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.786 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.786 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:28.786 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.455561+0000", 2024-02-09T18:06:28.786 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.027091944 2024-02-09T18:06:28.786 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.786 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.786 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:28.786 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.455575+0000", 2024-02-09T18:06:28.786 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3230999999999999e-05 2024-02-09T18:06:28.786 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.786 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.786 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:28.786 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.455665+0000", 2024-02-09T18:06:28.786 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.0321000000000005e-05 2024-02-09T18:06:28.787 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.787 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.787 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:28.787 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.455678+0000", 2024-02-09T18:06:28.787 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3125999999999999e-05 2024-02-09T18:06:28.787 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.787 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.787 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:28.787 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.007516+0000", 2024-02-09T18:06:28.787 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.55183836200000003 2024-02-09T18:06:28.787 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.787 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.787 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:28.787 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.007529+0000", 2024-02-09T18:06:28.787 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2636000000000001e-05 2024-02-09T18:06:28.788 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.788 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.788 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:28.788 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.248729+0000", 2024-02-09T18:06:28.788 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.241199443 2024-02-09T18:06:28.788 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.788 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.788 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:28.788 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.248845+0000", 2024-02-09T18:06:28.788 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000116072 2024-02-09T18:06:28.788 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.788 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.788 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4,11", 2024-02-09T18:06:28.788 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.248974+0000", 2024-02-09T18:06:28.789 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012956299999999999 2024-02-09T18:06:28.789 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.789 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.789 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:28.789 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.253563+0000", 2024-02-09T18:06:28.789 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.004589007 2024-02-09T18:06:28.789 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.789 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.789 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:28.789 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.253610+0000", 2024-02-09T18:06:28.789 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6653999999999998e-05 2024-02-09T18:06:28.789 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.789 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.789 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:28.789 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.265079+0000", 2024-02-09T18:06:28.790 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011468715000000001 2024-02-09T18:06:28.790 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.790 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.790 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:28.790 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.265176+0000", 2024-02-09T18:06:28.790 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7295999999999998e-05 2024-02-09T18:06:28.790 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.790 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.790 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:28.790 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.265233+0000", 2024-02-09T18:06:28.790 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6900000000000001e-05 2024-02-09T18:06:28.790 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.790 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.790 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:28.790 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.265338+0000", 2024-02-09T18:06:28.790 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010566800000000001 2024-02-09T18:06:28.791 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:28.791 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:28.791 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:28.791 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.791 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.791 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:236034 8.2 8:418c03b5:::benchmark_data_smithi179_148205_object112962:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e863)", 2024-02-09T18:06:28.791 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:59:26.455616+0000", 2024-02-09T18:06:28.791 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 422.309253458, 2024-02-09T18:06:28.791 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8388413300000002, 2024-02-09T18:06:28.791 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:28.791 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:28.791 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:28.791 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:06:28.791 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:06:28.791 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 236034 2024-02-09T18:06:28.792 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.792 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:28.792 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.792 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:28.792 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.455616+0000", 2024-02-09T18:06:28.792 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:28.792 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.792 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.792 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:28.792 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.455616+0000", 2024-02-09T18:06:28.792 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:28.792 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.792 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.793 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:28.793 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.455617+0000", 2024-02-09T18:06:28.793 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5349999999999999e-06 2024-02-09T18:06:28.793 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.793 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.793 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:28.793 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.455623+0000", 2024-02-09T18:06:28.793 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4539999999999998e-06 2024-02-09T18:06:28.793 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.793 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.793 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:28.793 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.455633+0000", 2024-02-09T18:06:28.793 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0592e-05 2024-02-09T18:06:28.793 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.793 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.794 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:28.794 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.455639+0000", 2024-02-09T18:06:28.794 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9710000000000001e-06 2024-02-09T18:06:28.794 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.794 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.794 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:28.794 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.455721+0000", 2024-02-09T18:06:28.794 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1332e-05 2024-02-09T18:06:28.794 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.794 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.794 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:28.794 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.455731+0000", 2024-02-09T18:06:28.794 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0346999999999999e-05 2024-02-09T18:06:28.794 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.794 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.795 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:28.795 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.006221+0000", 2024-02-09T18:06:28.795 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.55049037899999997 2024-02-09T18:06:28.795 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.795 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.795 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:28.795 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.006230+0000", 2024-02-09T18:06:28.795 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.1549999999999996e-06 2024-02-09T18:06:28.795 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.795 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.795 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:28.795 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.273690+0000", 2024-02-09T18:06:28.795 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.267459924 2024-02-09T18:06:28.795 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.795 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.796 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:28.796 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.273764+0000", 2024-02-09T18:06:28.796 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.3337000000000006e-05 2024-02-09T18:06:28.796 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.796 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.796 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,4,9", 2024-02-09T18:06:28.796 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.273854+0000", 2024-02-09T18:06:28.796 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.0600999999999995e-05 2024-02-09T18:06:28.796 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.796 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.796 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:28.796 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.294163+0000", 2024-02-09T18:06:28.796 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.020309026000000001 2024-02-09T18:06:28.796 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.796 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.797 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:28.797 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.294180+0000", 2024-02-09T18:06:28.797 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7122999999999999e-05 2024-02-09T18:06:28.797 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.797 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.797 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:28.797 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.294239+0000", 2024-02-09T18:06:28.797 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8445e-05 2024-02-09T18:06:28.797 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.797 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.797 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:28.797 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.294281+0000", 2024-02-09T18:06:28.797 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2203999999999998e-05 2024-02-09T18:06:28.797 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.797 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.798 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:28.798 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.294340+0000", 2024-02-09T18:06:28.798 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9023000000000001e-05 2024-02-09T18:06:28.798 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.798 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.798 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:28.798 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.294457+0000", 2024-02-09T18:06:28.798 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000116882 2024-02-09T18:06:28.798 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:28.798 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:28.798 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:28.798 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.798 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.798 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:236036 8.2 8:4fcb3643:::benchmark_data_smithi179_148205_object112964:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e863)", 2024-02-09T18:06:28.798 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:59:26.455656+0000", 2024-02-09T18:06:28.799 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 422.30921289000003, 2024-02-09T18:06:28.799 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8499204420000002, 2024-02-09T18:06:28.799 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:28.799 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:28.799 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:28.799 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:06:28.799 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:06:28.799 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 236036 2024-02-09T18:06:28.799 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.799 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:28.799 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.799 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:28.799 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.455656+0000", 2024-02-09T18:06:28.799 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:28.799 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.799 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.800 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:28.800 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.455656+0000", 2024-02-09T18:06:28.800 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:28.800 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.800 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.800 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:28.800 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.455657+0000", 2024-02-09T18:06:28.800 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.27e-07 2024-02-09T18:06:28.800 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.800 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.800 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:28.800 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.455659+0000", 2024-02-09T18:06:28.800 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.554e-06 2024-02-09T18:06:28.800 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.801 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.801 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:28.801 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.455665+0000", 2024-02-09T18:06:28.801 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2190000000000001e-06 2024-02-09T18:06:28.801 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.801 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.801 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:28.801 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.455670+0000", 2024-02-09T18:06:28.801 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9219999999999996e-06 2024-02-09T18:06:28.801 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.801 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.801 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:28.801 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.455937+0000", 2024-02-09T18:06:28.801 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00026710699999999999 2024-02-09T18:06:28.801 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.801 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.802 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:28.802 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.455947+0000", 2024-02-09T18:06:28.802 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5400000000000001e-06 2024-02-09T18:06:28.802 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.802 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.802 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:28.802 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.006257+0000", 2024-02-09T18:06:28.802 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.55031018899999995 2024-02-09T18:06:28.802 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.802 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.802 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:28.802 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.006265+0000", 2024-02-09T18:06:28.802 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4340000000000003e-06 2024-02-09T18:06:28.802 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.802 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.803 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:28.803 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.278585+0000", 2024-02-09T18:06:28.803 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.272320501 2024-02-09T18:06:28.803 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.803 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.803 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:28.803 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.278733+0000", 2024-02-09T18:06:28.803 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000147627 2024-02-09T18:06:28.803 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.803 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.803 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,4,9", 2024-02-09T18:06:28.803 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.278889+0000", 2024-02-09T18:06:28.803 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015604700000000001 2024-02-09T18:06:28.803 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.803 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.804 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:28.804 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.294187+0000", 2024-02-09T18:06:28.804 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.015298262999999999 2024-02-09T18:06:28.804 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.804 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.804 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:28.804 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.294504+0000", 2024-02-09T18:06:28.804 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00031670299999999999 2024-02-09T18:06:28.804 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.804 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.804 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:28.804 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.305250+0000", 2024-02-09T18:06:28.804 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.01074599 2024-02-09T18:06:28.804 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.804 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.805 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:28.805 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.305422+0000", 2024-02-09T18:06:28.805 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00017232700000000001 2024-02-09T18:06:28.805 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.805 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.805 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:28.805 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.305473+0000", 2024-02-09T18:06:28.805 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1119000000000003e-05 2024-02-09T18:06:28.805 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.805 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.805 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:28.805 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.305577+0000", 2024-02-09T18:06:28.805 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000103473 2024-02-09T18:06:28.805 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:28.805 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:28.806 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:28.806 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.806 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.806 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:236039 8.5 8:a407574c:::benchmark_data_smithi179_148205_object112967:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e863)", 2024-02-09T18:06:28.806 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:59:26.456976+0000", 2024-02-09T18:06:28.806 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 422.30789294200002, 2024-02-09T18:06:28.806 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.808632196, 2024-02-09T18:06:28.806 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:28.806 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:28.806 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:28.806 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:06:28.806 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:06:28.806 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 236039 2024-02-09T18:06:28.806 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.806 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:28.806 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.807 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:28.807 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.456976+0000", 2024-02-09T18:06:28.807 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:28.807 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.807 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.807 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:28.807 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.456976+0000", 2024-02-09T18:06:28.807 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:28.807 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.807 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.807 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:28.807 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.456979+0000", 2024-02-09T18:06:28.807 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3080000000000002e-06 2024-02-09T18:06:28.807 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.807 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.808 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:28.808 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.456985+0000", 2024-02-09T18:06:28.808 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.63e-06 2024-02-09T18:06:28.808 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.808 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.808 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:28.808 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.456999+0000", 2024-02-09T18:06:28.808 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4132e-05 2024-02-09T18:06:28.808 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.808 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.808 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:28.808 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.457008+0000", 2024-02-09T18:06:28.808 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.1179999999999998e-06 2024-02-09T18:06:28.808 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.808 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.809 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:28.809 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.457082+0000", 2024-02-09T18:06:28.809 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.3135999999999999e-05 2024-02-09T18:06:28.809 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.809 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.809 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:28.809 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.457091+0000", 2024-02-09T18:06:28.809 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5959999999999993e-06 2024-02-09T18:06:28.809 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.809 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.809 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:28.809 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.007556+0000", 2024-02-09T18:06:28.809 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.55046519400000005 2024-02-09T18:06:28.809 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.809 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.810 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:28.810 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.007564+0000", 2024-02-09T18:06:28.810 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.3339999999999996e-06 2024-02-09T18:06:28.810 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.810 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.810 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:28.810 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.253643+0000", 2024-02-09T18:06:28.810 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.246079443 2024-02-09T18:06:28.810 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.810 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.810 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:28.810 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.254504+0000", 2024-02-09T18:06:28.810 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00086107999999999996 2024-02-09T18:06:28.810 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.810 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.810 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4,11", 2024-02-09T18:06:28.811 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.254658+0000", 2024-02-09T18:06:28.811 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015400200000000001 2024-02-09T18:06:28.811 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.811 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.811 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:28.811 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.265158+0000", 2024-02-09T18:06:28.811 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010500083 2024-02-09T18:06:28.811 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.811 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.811 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:28.811 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.265347+0000", 2024-02-09T18:06:28.811 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00018832699999999999 2024-02-09T18:06:28.811 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.811 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.811 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:28.812 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.265386+0000", 2024-02-09T18:06:28.812 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9870000000000003e-05 2024-02-09T18:06:28.812 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.812 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.812 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:28.812 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.265434+0000", 2024-02-09T18:06:28.812 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7627999999999999e-05 2024-02-09T18:06:28.812 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.812 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.812 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:28.812 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.265490+0000", 2024-02-09T18:06:28.812 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5448999999999997e-05 2024-02-09T18:06:28.812 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.812 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.813 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:28.813 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.265608+0000", 2024-02-09T18:06:28.813 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000118866 2024-02-09T18:06:28.813 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:28.813 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:28.813 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:28.813 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.813 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.813 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:236056 8.5 8:afbd21ac:::benchmark_data_smithi179_148205_object112984:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e863)", 2024-02-09T18:06:28.813 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:59:26.494045+0000", 2024-02-09T18:06:28.813 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 422.27082440599997, 2024-02-09T18:06:28.813 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7864225920000001, 2024-02-09T18:06:28.813 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:28.813 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:28.813 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:28.814 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:06:28.814 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:06:28.814 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 236056 2024-02-09T18:06:28.814 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.814 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:28.814 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.814 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:28.814 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.494045+0000", 2024-02-09T18:06:28.814 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:28.814 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.814 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.814 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:28.814 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.494045+0000", 2024-02-09T18:06:28.814 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:28.814 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.815 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.815 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:28.815 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.494046+0000", 2024-02-09T18:06:28.815 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.359e-06 2024-02-09T18:06:28.815 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.815 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.815 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:28.815 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.494053+0000", 2024-02-09T18:06:28.815 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8240000000000002e-06 2024-02-09T18:06:28.815 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.815 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.815 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:28.815 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.494065+0000", 2024-02-09T18:06:28.815 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2089999999999999e-05 2024-02-09T18:06:28.815 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.816 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.816 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:28.816 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.494072+0000", 2024-02-09T18:06:28.816 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1389999999999998e-06 2024-02-09T18:06:28.816 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.816 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.816 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:28.816 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.494156+0000", 2024-02-09T18:06:28.816 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4151000000000001e-05 2024-02-09T18:06:28.816 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.816 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.816 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:28.816 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.494167+0000", 2024-02-09T18:06:28.816 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0929e-05 2024-02-09T18:06:28.816 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.817 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.817 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:28.817 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.007579+0000", 2024-02-09T18:06:28.817 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.513412176 2024-02-09T18:06:28.817 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.817 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.817 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:28.817 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.007584+0000", 2024-02-09T18:06:28.817 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3889999999999998e-06 2024-02-09T18:06:28.817 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.817 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.817 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:28.817 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.259989+0000", 2024-02-09T18:06:28.817 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.252405336 2024-02-09T18:06:28.817 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.817 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.818 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:28.818 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.260127+0000", 2024-02-09T18:06:28.818 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000137992 2024-02-09T18:06:28.818 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.818 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.818 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4,11", 2024-02-09T18:06:28.818 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.260278+0000", 2024-02-09T18:06:28.818 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015093600000000001 2024-02-09T18:06:28.818 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.818 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.818 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:28.818 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.265399+0000", 2024-02-09T18:06:28.818 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0051205479999999999 2024-02-09T18:06:28.818 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.819 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.819 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:28.819 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.265660+0000", 2024-02-09T18:06:28.819 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00026186900000000002 2024-02-09T18:06:28.819 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.819 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.819 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:28.819 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.280217+0000", 2024-02-09T18:06:28.819 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014556098999999999 2024-02-09T18:06:28.819 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.819 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.819 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:28.819 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.280275+0000", 2024-02-09T18:06:28.819 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.808e-05 2024-02-09T18:06:28.819 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.819 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.820 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:28.820 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.280341+0000", 2024-02-09T18:06:28.820 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5888000000000003e-05 2024-02-09T18:06:28.820 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.820 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.820 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:28.820 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.280467+0000", 2024-02-09T18:06:28.820 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000126787 2024-02-09T18:06:28.820 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:28.820 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:28.820 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:28.820 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.820 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.821 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:236057 8.2 8:461a0d56:::benchmark_data_smithi179_148205_object112985:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e863)", 2024-02-09T18:06:28.821 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:59:26.494215+0000", 2024-02-09T18:06:28.821 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 422.27065384999997, 2024-02-09T18:06:28.821 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.811620604, 2024-02-09T18:06:28.821 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:28.821 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:28.821 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:28.821 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:06:28.821 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:06:28.821 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 236057 2024-02-09T18:06:28.821 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.821 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:28.821 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.821 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:28.822 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.494215+0000", 2024-02-09T18:06:28.822 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:28.822 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.822 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.822 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:28.822 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.494215+0000", 2024-02-09T18:06:28.822 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:28.822 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.822 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.822 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:28.822 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.494216+0000", 2024-02-09T18:06:28.822 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.0599999999999999e-07 2024-02-09T18:06:28.822 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.822 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.822 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:28.823 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.494220+0000", 2024-02-09T18:06:28.823 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7469999999999999e-06 2024-02-09T18:06:28.823 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.823 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.823 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:28.823 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.494229+0000", 2024-02-09T18:06:28.823 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2220000000000008e-06 2024-02-09T18:06:28.823 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.823 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.823 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:28.823 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.494234+0000", 2024-02-09T18:06:28.823 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1159999999999998e-06 2024-02-09T18:06:28.823 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.823 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.823 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:28.824 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.494289+0000", 2024-02-09T18:06:28.824 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5121000000000003e-05 2024-02-09T18:06:28.824 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.824 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.824 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:28.824 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.494302+0000", 2024-02-09T18:06:28.824 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2221e-05 2024-02-09T18:06:28.824 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.824 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.824 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:28.824 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.006367+0000", 2024-02-09T18:06:28.824 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.512065722 2024-02-09T18:06:28.824 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.825 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.825 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:28.825 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.006379+0000", 2024-02-09T18:06:28.825 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1562e-05 2024-02-09T18:06:28.825 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.825 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.825 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:28.825 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.284221+0000", 2024-02-09T18:06:28.825 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2778420449999999 2024-02-09T18:06:28.825 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.825 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.825 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:28.825 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.284358+0000", 2024-02-09T18:06:28.825 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013716600000000001 2024-02-09T18:06:28.825 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.826 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.826 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,4,9", 2024-02-09T18:06:28.826 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.284496+0000", 2024-02-09T18:06:28.826 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013806100000000001 2024-02-09T18:06:28.826 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.826 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.826 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:28.826 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.294194+0000", 2024-02-09T18:06:28.826 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0096976349999999996 2024-02-09T18:06:28.826 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.826 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.826 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:28.826 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.305305+0000", 2024-02-09T18:06:28.826 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011110890999999999 2024-02-09T18:06:28.826 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.826 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.827 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:28.827 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.305614+0000", 2024-02-09T18:06:28.827 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00030973200000000001 2024-02-09T18:06:28.827 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.827 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.827 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:28.827 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.305690+0000", 2024-02-09T18:06:28.827 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5555000000000004e-05 2024-02-09T18:06:28.827 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.827 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.827 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:28.827 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.305738+0000", 2024-02-09T18:06:28.827 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8018e-05 2024-02-09T18:06:28.827 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.827 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.827 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:28.828 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.305836+0000", 2024-02-09T18:06:28.828 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7884e-05 2024-02-09T18:06:28.828 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:28.828 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:28.828 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:28.828 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.828 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.828 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:236058 8.5 8:a5607057:::benchmark_data_smithi179_148205_object112986:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e863)", 2024-02-09T18:06:28.828 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:59:26.498098+0000", 2024-02-09T18:06:28.828 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 422.266771425, 2024-02-09T18:06:28.828 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7935283659999999, 2024-02-09T18:06:28.828 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:28.828 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:28.828 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:28.829 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:06:28.829 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:06:28.829 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 236058 2024-02-09T18:06:28.829 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.829 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:28.829 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.829 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:28.829 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.498098+0000", 2024-02-09T18:06:28.829 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:28.829 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.829 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.829 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:28.829 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.498098+0000", 2024-02-09T18:06:28.829 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:28.830 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.830 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.830 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:28.830 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.498099+0000", 2024-02-09T18:06:28.830 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3400000000000001e-06 2024-02-09T18:06:28.830 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.830 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.830 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:28.830 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.498104+0000", 2024-02-09T18:06:28.830 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0490000000000003e-06 2024-02-09T18:06:28.830 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.830 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.830 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:28.830 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.498116+0000", 2024-02-09T18:06:28.830 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2021000000000001e-05 2024-02-09T18:06:28.831 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.831 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.831 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:28.831 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.498124+0000", 2024-02-09T18:06:28.831 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.892e-06 2024-02-09T18:06:28.831 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.831 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.831 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:28.831 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.498184+0000", 2024-02-09T18:06:28.831 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9481999999999997e-05 2024-02-09T18:06:28.831 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.831 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.831 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:28.831 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.498192+0000", 2024-02-09T18:06:28.831 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.011e-06 2024-02-09T18:06:28.831 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.831 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.832 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:28.832 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.007624+0000", 2024-02-09T18:06:28.832 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.50943232199999999 2024-02-09T18:06:28.832 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.832 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.832 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:28.832 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.007628+0000", 2024-02-09T18:06:28.832 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4129999999999999e-06 2024-02-09T18:06:28.832 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.832 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.832 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:28.832 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.265695+0000", 2024-02-09T18:06:28.832 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.258067217 2024-02-09T18:06:28.832 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.832 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.833 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:28.833 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.265848+0000", 2024-02-09T18:06:28.833 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015218799999999999 2024-02-09T18:06:28.833 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.833 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.833 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4,11", 2024-02-09T18:06:28.833 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.265936+0000", 2024-02-09T18:06:28.833 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8640000000000005e-05 2024-02-09T18:06:28.833 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.833 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.833 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:28.833 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.280082+0000", 2024-02-09T18:06:28.833 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014145985 2024-02-09T18:06:28.834 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.834 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.834 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:28.834 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.280157+0000", 2024-02-09T18:06:28.834 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4746000000000006e-05 2024-02-09T18:06:28.834 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.834 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.834 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:28.834 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.291374+0000", 2024-02-09T18:06:28.834 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011217122 2024-02-09T18:06:28.834 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.834 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.834 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:28.834 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.291441+0000", 2024-02-09T18:06:28.834 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6841999999999995e-05 2024-02-09T18:06:28.834 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.835 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.835 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:28.835 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.291510+0000", 2024-02-09T18:06:28.835 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9494999999999998e-05 2024-02-09T18:06:28.835 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.835 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.835 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:28.835 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.291626+0000", 2024-02-09T18:06:28.835 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011560100000000001 2024-02-09T18:06:28.835 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:28.835 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:28.835 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:28.835 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.835 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.836 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:236065 8.5 8:a06c5d66:::benchmark_data_smithi179_148205_object112993:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e863)", 2024-02-09T18:06:28.836 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:59:26.528647+0000", 2024-02-09T18:06:28.836 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 422.236222166, 2024-02-09T18:06:28.836 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7632569490000001, 2024-02-09T18:06:28.836 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:28.836 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:28.836 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:28.836 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:06:28.836 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:06:28.836 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 236065 2024-02-09T18:06:28.836 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.836 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:28.836 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.836 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:28.836 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.528647+0000", 2024-02-09T18:06:28.837 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:28.837 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.837 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.837 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:28.837 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.528647+0000", 2024-02-09T18:06:28.837 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:28.837 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.837 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.837 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:28.837 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.528648+0000", 2024-02-09T18:06:28.837 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.271e-06 2024-02-09T18:06:28.837 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.837 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.837 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:28.837 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.528653+0000", 2024-02-09T18:06:28.838 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1030000000000001e-06 2024-02-09T18:06:28.838 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.838 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.838 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:28.838 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.528664+0000", 2024-02-09T18:06:28.838 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0728e-05 2024-02-09T18:06:28.838 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.838 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.838 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:28.838 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.528672+0000", 2024-02-09T18:06:28.838 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.9209999999999992e-06 2024-02-09T18:06:28.838 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.838 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.838 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:28.838 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.528743+0000", 2024-02-09T18:06:28.838 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1292999999999997e-05 2024-02-09T18:06:28.839 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.839 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.839 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:28.839 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.528757+0000", 2024-02-09T18:06:28.839 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3213999999999999e-05 2024-02-09T18:06:28.839 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.839 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.839 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:28.839 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.007659+0000", 2024-02-09T18:06:28.839 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.47890293 2024-02-09T18:06:28.839 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.839 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.839 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:28.839 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.007664+0000", 2024-02-09T18:06:28.839 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3590000000000001e-06 2024-02-09T18:06:28.840 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.840 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.840 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:28.840 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.270147+0000", 2024-02-09T18:06:28.840 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.262483529 2024-02-09T18:06:28.840 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.840 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.840 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:28.840 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.270275+0000", 2024-02-09T18:06:28.840 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012748800000000001 2024-02-09T18:06:28.840 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.840 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.840 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4,11", 2024-02-09T18:06:28.840 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.270390+0000", 2024-02-09T18:06:28.840 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000114889 2024-02-09T18:06:28.841 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.841 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.841 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:28.841 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.280226+0000", 2024-02-09T18:06:28.841 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0098362129999999999 2024-02-09T18:06:28.841 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.841 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.841 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:28.841 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.280523+0000", 2024-02-09T18:06:28.841 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00029722800000000002 2024-02-09T18:06:28.841 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.841 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.841 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:28.841 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.291675+0000", 2024-02-09T18:06:28.841 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011151797 2024-02-09T18:06:28.842 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.842 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.842 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:28.842 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.291733+0000", 2024-02-09T18:06:28.842 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8140000000000002e-05 2024-02-09T18:06:28.842 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.842 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.842 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:28.842 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.291794+0000", 2024-02-09T18:06:28.842 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0979e-05 2024-02-09T18:06:28.842 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.842 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.842 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:28.842 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.291904+0000", 2024-02-09T18:06:28.842 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000109867 2024-02-09T18:06:28.842 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:28.843 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:28.843 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:28.843 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.843 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.843 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:236067 8.5 8:af1cfb54:::benchmark_data_smithi179_148205_object112995:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e863)", 2024-02-09T18:06:28.843 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:59:26.539658+0000", 2024-02-09T18:06:28.843 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 422.22521068600003, 2024-02-09T18:06:28.843 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.761787164, 2024-02-09T18:06:28.843 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:28.843 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:28.843 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:28.843 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:06:28.843 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:06:28.843 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 236067 2024-02-09T18:06:28.843 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.843 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:28.844 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.844 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:28.844 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.539658+0000", 2024-02-09T18:06:28.844 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:28.844 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.844 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.844 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:28.844 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.539658+0000", 2024-02-09T18:06:28.844 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:28.844 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.844 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.844 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:28.844 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.539660+0000", 2024-02-09T18:06:28.844 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.531e-06 2024-02-09T18:06:28.844 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.845 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.845 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:28.845 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.539667+0000", 2024-02-09T18:06:28.845 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5799999999999997e-06 2024-02-09T18:06:28.845 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.845 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.845 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:28.845 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.539683+0000", 2024-02-09T18:06:28.845 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6577000000000001e-05 2024-02-09T18:06:28.845 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.845 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.845 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:28.845 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.539693+0000", 2024-02-09T18:06:28.845 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0161e-05 2024-02-09T18:06:28.845 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.846 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.846 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:28.846 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.539771+0000", 2024-02-09T18:06:28.846 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7669999999999993e-05 2024-02-09T18:06:28.846 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.846 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.846 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:28.846 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.539785+0000", 2024-02-09T18:06:28.846 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3665999999999999e-05 2024-02-09T18:06:28.846 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.846 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.846 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:28.846 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.008228+0000", 2024-02-09T18:06:28.846 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.46844367100000001 2024-02-09T18:06:28.846 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.846 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.846 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:28.847 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.008235+0000", 2024-02-09T18:06:28.847 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.0160000000000003e-06 2024-02-09T18:06:28.847 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.847 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.847 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:28.847 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.275373+0000", 2024-02-09T18:06:28.847 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.267137822 2024-02-09T18:06:28.847 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.847 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.847 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:28.847 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.275491+0000", 2024-02-09T18:06:28.847 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011803000000000001 2024-02-09T18:06:28.847 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.847 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.847 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4,11", 2024-02-09T18:06:28.847 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.275624+0000", 2024-02-09T18:06:28.848 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013240800000000001 2024-02-09T18:06:28.848 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.848 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.848 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:28.848 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.285950+0000", 2024-02-09T18:06:28.848 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010325968 2024-02-09T18:06:28.848 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.848 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.848 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:28.848 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.291315+0000", 2024-02-09T18:06:28.848 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0053651439999999996 2024-02-09T18:06:28.848 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.848 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.848 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:28.848 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.291954+0000", 2024-02-09T18:06:28.849 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00063973399999999998 2024-02-09T18:06:28.849 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.849 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.849 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:28.849 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.301266+0000", 2024-02-09T18:06:28.849 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0093119099999999996 2024-02-09T18:06:28.849 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.849 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.849 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:28.849 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.301328+0000", 2024-02-09T18:06:28.849 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1575999999999995e-05 2024-02-09T18:06:28.849 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.849 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.849 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:28.849 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.301446+0000", 2024-02-09T18:06:28.850 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0001177 2024-02-09T18:06:28.850 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:28.850 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:28.850 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:28.850 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.850 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.850 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.26563.0:204326 9.e e946/943 9:75e9ee8a:::benchmark_data_smithi179_161915_object69444:head v 946'12815, mlcod=945'12812)", 2024-02-09T18:06:28.850 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:17.475438+0000", 2024-02-09T18:06:28.850 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 191.289431122, 2024-02-09T18:06:28.850 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.43709178599999998, 2024-02-09T18:06:28.850 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:28.850 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:28.850 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:28.850 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.850 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:28.850 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.475438+0000", 2024-02-09T18:06:28.850 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:28.851 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.851 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.851 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:28.851 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.475438+0000", 2024-02-09T18:06:28.851 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:28.851 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.851 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.851 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:28.851 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.475439+0000", 2024-02-09T18:06:28.851 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.47e-07 2024-02-09T18:06:28.851 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.851 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.851 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:28.851 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.475447+0000", 2024-02-09T18:06:28.851 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.9610000000000005e-06 2024-02-09T18:06:28.851 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.852 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.852 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:28.852 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.475464+0000", 2024-02-09T18:06:28.852 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6929999999999999e-05 2024-02-09T18:06:28.852 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.852 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.852 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:28.852 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.475473+0000", 2024-02-09T18:06:28.852 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.6719999999999993e-06 2024-02-09T18:06:28.852 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.852 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.852 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:28.852 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.484457+0000", 2024-02-09T18:06:28.852 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0089841810000000008 2024-02-09T18:06:28.853 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.853 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.853 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:28.853 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.484485+0000", 2024-02-09T18:06:28.853 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7871000000000001e-05 2024-02-09T18:06:28.853 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.853 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.853 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:28.853 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.912479+0000", 2024-02-09T18:06:28.853 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.42799490299999998 2024-02-09T18:06:28.853 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.853 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.853 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:28.853 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.912530+0000", 2024-02-09T18:06:28.853 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0321000000000002e-05 2024-02-09T18:06:28.854 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:28.854 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:28.854 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:28.854 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.854 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.854 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.26563.0:204332 9.6 e946/943 9:647e4d95:::benchmark_data_smithi179_161915_object69450:head v 946'12703, mlcod=945'12701)", 2024-02-09T18:06:28.854 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:17.481214+0000", 2024-02-09T18:06:28.854 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 191.28365507300001, 2024-02-09T18:06:28.854 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.52768333700000003, 2024-02-09T18:06:28.854 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:28.854 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:28.854 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:28.854 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.854 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:28.855 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.481214+0000", 2024-02-09T18:06:28.855 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:28.855 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.855 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.855 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:28.855 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.481214+0000", 2024-02-09T18:06:28.855 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:28.855 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.855 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.855 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:28.855 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.481214+0000", 2024-02-09T18:06:28.855 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1e-07 2024-02-09T18:06:28.855 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.855 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.856 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:28.856 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.481220+0000", 2024-02-09T18:06:28.856 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4709999999999998e-06 2024-02-09T18:06:28.856 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.856 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.856 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:28.856 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.481226+0000", 2024-02-09T18:06:28.856 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2489999999999998e-06 2024-02-09T18:06:28.856 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.856 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.856 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:28.856 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.481234+0000", 2024-02-09T18:06:28.856 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5129999999999999e-06 2024-02-09T18:06:28.856 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.857 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.857 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:28.857 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.489666+0000", 2024-02-09T18:06:28.857 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0084323179999999994 2024-02-09T18:06:28.857 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.857 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.857 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:28.857 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.489689+0000", 2024-02-09T18:06:28.857 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3507999999999999e-05 2024-02-09T18:06:28.857 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.857 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.857 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:28.857 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:18.008848+0000", 2024-02-09T18:06:28.857 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.51915874900000003 2024-02-09T18:06:28.858 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.858 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.858 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:28.858 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:18.008897+0000", 2024-02-09T18:06:28.858 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9218999999999998e-05 2024-02-09T18:06:28.858 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:28.858 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:28.858 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:28.858 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.858 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.858 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.26563.0:204346 9.e e946/943 9:7b6ff32e:::benchmark_data_smithi179_161915_object69464:head v 946'12816, mlcod=945'12813)", 2024-02-09T18:06:28.858 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:17.515163+0000", 2024-02-09T18:06:28.858 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 191.24970606700001, 2024-02-09T18:06:28.858 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.50301004100000002, 2024-02-09T18:06:28.859 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:28.859 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:28.859 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:28.859 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.859 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:28.859 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.515163+0000", 2024-02-09T18:06:28.859 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:28.859 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.859 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.859 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:28.859 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.515163+0000", 2024-02-09T18:06:28.859 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:28.859 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.859 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.859 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:28.859 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.515164+0000", 2024-02-09T18:06:28.860 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1519999999999999e-06 2024-02-09T18:06:28.860 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.860 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.860 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:28.860 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.515174+0000", 2024-02-09T18:06:28.860 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6940000000000007e-06 2024-02-09T18:06:28.860 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.860 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.860 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:28.860 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.515195+0000", 2024-02-09T18:06:28.860 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0971e-05 2024-02-09T18:06:28.860 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.860 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.860 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:28.861 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.515204+0000", 2024-02-09T18:06:28.861 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.4620000000000002e-06 2024-02-09T18:06:28.861 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.861 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.861 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:28.861 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.516391+0000", 2024-02-09T18:06:28.861 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0011864639999999999 2024-02-09T18:06:28.861 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.861 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.861 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:28.861 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.516416+0000", 2024-02-09T18:06:28.861 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5077999999999999e-05 2024-02-09T18:06:28.861 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.861 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.862 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:28.862 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:18.018118+0000", 2024-02-09T18:06:28.862 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.50170174199999995 2024-02-09T18:06:28.862 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.862 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.862 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:28.862 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:18.018173+0000", 2024-02-09T18:06:28.862 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5478e-05 2024-02-09T18:06:28.862 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:28.862 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:28.862 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:28.862 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.862 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.862 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.26563.0:204342 9.7 e946/943 9:ef4e912e:::benchmark_data_smithi179_161915_object69460:head v 946'12698, mlcod=945'12696)", 2024-02-09T18:06:28.862 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:17.521143+0000", 2024-02-09T18:06:28.862 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 191.24372631899999, 2024-02-09T18:06:28.862 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.39707365500000003, 2024-02-09T18:06:28.863 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:28.863 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:28.863 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:28.863 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.863 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:28.863 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.521143+0000", 2024-02-09T18:06:28.863 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:28.863 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.863 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.863 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:28.863 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.521143+0000", 2024-02-09T18:06:28.863 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:28.863 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.863 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.863 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:28.863 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.521144+0000", 2024-02-09T18:06:28.864 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3799999999999999e-06 2024-02-09T18:06:28.864 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.864 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.864 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:28.864 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.521153+0000", 2024-02-09T18:06:28.864 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5569999999999996e-06 2024-02-09T18:06:28.864 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.864 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.864 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:28.864 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.521171+0000", 2024-02-09T18:06:28.864 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8012999999999998e-05 2024-02-09T18:06:28.864 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.864 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.864 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:28.865 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.521179+0000", 2024-02-09T18:06:28.865 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.0919999999999998e-06 2024-02-09T18:06:28.865 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.865 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.865 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:28.865 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.523195+0000", 2024-02-09T18:06:28.865 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0020156369999999998 2024-02-09T18:06:28.865 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.865 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.865 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:28.865 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.523219+0000", 2024-02-09T18:06:28.865 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4045000000000001e-05 2024-02-09T18:06:28.865 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.865 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.865 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:28.866 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.918163+0000", 2024-02-09T18:06:28.866 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.39494473699999999 2024-02-09T18:06:28.866 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.866 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.866 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:28.866 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.918216+0000", 2024-02-09T18:06:28.866 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3193999999999999e-05 2024-02-09T18:06:28.866 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:28.866 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:28.866 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:28.866 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.866 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.866 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.26563.0:204356 9.e e946/943 9:7ed8cb3e:::benchmark_data_smithi179_161915_object69474:head v 946'12817, mlcod=945'12813)", 2024-02-09T18:06:28.866 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:17.529235+0000", 2024-02-09T18:06:28.866 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 191.235633976, 2024-02-09T18:06:28.866 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.488972974, 2024-02-09T18:06:28.867 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:28.867 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:28.867 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:28.867 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.867 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:28.867 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.529235+0000", 2024-02-09T18:06:28.867 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:28.867 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.867 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.867 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:28.867 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.529235+0000", 2024-02-09T18:06:28.867 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:28.867 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.867 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.867 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:28.867 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.529236+0000", 2024-02-09T18:06:28.867 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.22e-06 2024-02-09T18:06:28.868 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.868 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.868 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:28.868 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.529245+0000", 2024-02-09T18:06:28.868 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.6729999999999998e-06 2024-02-09T18:06:28.868 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.868 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.868 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:28.868 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.529258+0000", 2024-02-09T18:06:28.868 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2928000000000001e-05 2024-02-09T18:06:28.868 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.868 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.868 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:28.868 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.529267+0000", 2024-02-09T18:06:28.868 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5220000000000007e-06 2024-02-09T18:06:28.869 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.869 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.869 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:28.869 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.550372+0000", 2024-02-09T18:06:28.869 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.021105426 2024-02-09T18:06:28.869 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.869 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.869 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:28.869 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.550395+0000", 2024-02-09T18:06:28.869 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2614000000000001e-05 2024-02-09T18:06:28.869 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.869 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.869 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:28.869 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:18.018178+0000", 2024-02-09T18:06:28.869 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.467782965 2024-02-09T18:06:28.869 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.870 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.870 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:28.870 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:18.018208+0000", 2024-02-09T18:06:28.870 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0626000000000001e-05 2024-02-09T18:06:28.870 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:28.870 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:28.870 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:28.870 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.870 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.870 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.26563.0:204359 9.b e946/943 9:d380bded:::benchmark_data_smithi179_161915_object69477:head v 946'12851, mlcod=945'12849)", 2024-02-09T18:06:28.870 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:17.544167+0000", 2024-02-09T18:06:28.870 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 191.22070206399999, 2024-02-09T18:06:28.870 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.46437108700000002, 2024-02-09T18:06:28.870 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:28.870 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:28.871 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:28.871 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.871 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:28.871 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.544167+0000", 2024-02-09T18:06:28.871 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:28.871 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.871 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.871 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:28.871 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.544167+0000", 2024-02-09T18:06:28.871 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:28.871 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.871 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.871 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:28.871 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.544167+0000", 2024-02-09T18:06:28.871 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7099999999999998e-07 2024-02-09T18:06:28.871 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.871 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.872 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:28.872 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.544174+0000", 2024-02-09T18:06:28.872 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3589999999999996e-06 2024-02-09T18:06:28.872 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.872 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.872 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:28.872 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.544189+0000", 2024-02-09T18:06:28.872 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4907e-05 2024-02-09T18:06:28.872 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.872 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.872 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:28.872 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.544197+0000", 2024-02-09T18:06:28.872 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7690000000000002e-06 2024-02-09T18:06:28.872 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.872 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.873 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:28.873 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.544281+0000", 2024-02-09T18:06:28.873 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3199000000000001e-05 2024-02-09T18:06:28.873 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.873 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.873 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:28.873 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.544308+0000", 2024-02-09T18:06:28.873 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7350000000000001e-05 2024-02-09T18:06:28.873 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.873 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.873 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:28.873 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:18.008486+0000", 2024-02-09T18:06:28.873 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.46417840199999999 2024-02-09T18:06:28.873 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.873 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.874 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:28.874 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:18.008538+0000", 2024-02-09T18:06:28.874 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1829999999999997e-05 2024-02-09T18:06:28.874 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:28.874 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:28.874 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:28.874 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.874 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.874 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.26563.0:204365 9.b e946/943 9:d4e000a7:::benchmark_data_smithi179_161915_object69483:head v 946'12852, mlcod=945'12849)", 2024-02-09T18:06:28.874 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:17.554395+0000", 2024-02-09T18:06:28.874 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 191.21047376499999, 2024-02-09T18:06:28.874 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.45417522300000002, 2024-02-09T18:06:28.874 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:28.874 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:28.874 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:28.875 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.875 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:28.875 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.554395+0000", 2024-02-09T18:06:28.875 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:28.875 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.875 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.875 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:28.875 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.554395+0000", 2024-02-09T18:06:28.875 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:28.875 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.875 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.875 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:28.875 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.554397+0000", 2024-02-09T18:06:28.875 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.331e-06 2024-02-09T18:06:28.875 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.876 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.876 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:28.876 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.554406+0000", 2024-02-09T18:06:28.876 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5980000000000003e-06 2024-02-09T18:06:28.876 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.876 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.876 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:28.876 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.554419+0000", 2024-02-09T18:06:28.876 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2516e-05 2024-02-09T18:06:28.876 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.876 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.876 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:28.876 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.554428+0000", 2024-02-09T18:06:28.876 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7050000000000005e-06 2024-02-09T18:06:28.877 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.877 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.877 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:28.877 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.554484+0000", 2024-02-09T18:06:28.877 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6069999999999997e-05 2024-02-09T18:06:28.877 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.877 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.877 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:28.877 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.554509+0000", 2024-02-09T18:06:28.877 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4967999999999999e-05 2024-02-09T18:06:28.877 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.877 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.877 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:28.877 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:18.008542+0000", 2024-02-09T18:06:28.877 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.45403305500000002 2024-02-09T18:06:28.877 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.878 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.878 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:28.878 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:18.008571+0000", 2024-02-09T18:06:28.878 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.898e-05 2024-02-09T18:06:28.878 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:28.878 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:28.878 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:28.878 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.878 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.878 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.26563.0:204373 9.b e946/943 9:d1b4954d:::benchmark_data_smithi179_161915_object69491:head v 946'12853, mlcod=945'12849)", 2024-02-09T18:06:28.878 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:17.570961+0000", 2024-02-09T18:06:28.878 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 191.19390859800001, 2024-02-09T18:06:28.878 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.43764235699999998, 2024-02-09T18:06:28.878 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:28.878 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:28.878 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:28.879 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.879 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:28.879 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.570961+0000", 2024-02-09T18:06:28.879 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:28.879 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.879 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.879 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:28.879 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.570961+0000", 2024-02-09T18:06:28.879 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:28.879 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.879 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.879 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:28.879 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.570962+0000", 2024-02-09T18:06:28.879 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.111e-06 2024-02-09T18:06:28.879 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.879 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.879 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:28.880 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.570970+0000", 2024-02-09T18:06:28.880 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.9109999999999993e-06 2024-02-09T18:06:28.880 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.880 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.880 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:28.880 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.570992+0000", 2024-02-09T18:06:28.880 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2815000000000001e-05 2024-02-09T18:06:28.880 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.880 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.880 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:28.880 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.571001+0000", 2024-02-09T18:06:28.880 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1419999999999993e-06 2024-02-09T18:06:28.880 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.880 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.881 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:28.881 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.653730+0000", 2024-02-09T18:06:28.881 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.082729840999999998 2024-02-09T18:06:28.881 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.881 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.881 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:28.881 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.653746+0000", 2024-02-09T18:06:28.881 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5817000000000001e-05 2024-02-09T18:06:28.881 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.881 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.881 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:28.881 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:18.008573+0000", 2024-02-09T18:06:28.881 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.35482643699999999 2024-02-09T18:06:28.881 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.881 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.881 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:28.882 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:18.008603+0000", 2024-02-09T18:06:28.882 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0283000000000001e-05 2024-02-09T18:06:28.882 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:28.882 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:28.882 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:28.882 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.882 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.882 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.26563.0:204384 9.e e946/943 9:7332cb43:::benchmark_data_smithi179_161915_object69502:head v 946'12818, mlcod=945'12813)", 2024-02-09T18:06:28.882 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:17.615161+0000", 2024-02-09T18:06:28.882 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 191.14970829399999, 2024-02-09T18:06:28.882 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.40307902800000001, 2024-02-09T18:06:28.882 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:28.882 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:28.882 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:28.882 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.882 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:28.883 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.615161+0000", 2024-02-09T18:06:28.883 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:28.883 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.883 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.883 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:28.883 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.615161+0000", 2024-02-09T18:06:28.883 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:28.883 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.883 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.883 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:28.883 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.615162+0000", 2024-02-09T18:06:28.883 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.302e-06 2024-02-09T18:06:28.883 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.883 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.883 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:28.883 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.615172+0000", 2024-02-09T18:06:28.884 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5599999999999999e-06 2024-02-09T18:06:28.884 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.884 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.884 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:28.884 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.615191+0000", 2024-02-09T18:06:28.884 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9538000000000002e-05 2024-02-09T18:06:28.884 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.884 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.884 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:28.884 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.615200+0000", 2024-02-09T18:06:28.884 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7029999999999995e-06 2024-02-09T18:06:28.884 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.884 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.884 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:28.884 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.653688+0000", 2024-02-09T18:06:28.884 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.038487747000000003 2024-02-09T18:06:28.885 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.885 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.885 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:28.885 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.653709+0000", 2024-02-09T18:06:28.885 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0843e-05 2024-02-09T18:06:28.885 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.885 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.885 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:28.885 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:18.018211+0000", 2024-02-09T18:06:28.885 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.36450210300000002 2024-02-09T18:06:28.885 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:28.885 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:28.885 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:28.885 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:18.018240+0000", 2024-02-09T18:06:28.885 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.9232000000000001e-05 2024-02-09T18:06:28.885 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:28.885 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:28.886 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:28.886 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:28.886 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:28.886 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T18:06:28.886 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:28 smithi099 ceph-mon[39420]: pgmap v2514: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:28.886 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:28 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4233410580' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-02-09T18:06:28.887 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:28 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:06:28.887 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:28 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:06:28.887 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:28 smithi179 ceph-mon[43975]: pgmap v2514: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:28.887 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:28 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4233410580' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-02-09T18:06:28.887 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:28 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:06:28.887 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:28 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:06:28.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:28 smithi162 ceph-mon[44304]: pgmap v2514: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:28.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:28 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4233410580' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-02-09T18:06:28.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:28 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:06:28.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:28 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:06:29.261 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd primary-affinity 10 1 2024-02-09T18:06:29.457 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.11 dump_ops_in_flight 2024-02-09T18:06:29.532 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:06:29.720 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T18:06:29.914 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:29 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4233410580' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-02-09T18:06:29.915 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:29 smithi099 ceph-mon[39420]: osdmap e1024: 12 total, 12 up, 12 in 2024-02-09T18:06:29.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:29 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4233410580' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-02-09T18:06:29.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:29 smithi179 ceph-mon[43975]: osdmap e1024: 12 total, 12 up, 12 in 2024-02-09T18:06:29.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:29 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4233410580' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-02-09T18:06:29.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:29 smithi162 ceph-mon[44304]: osdmap e1024: 12 total, 12 up, 12 in 2024-02-09T18:06:30.912 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:30 smithi179 ceph-mon[43975]: pgmap v2516: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:30.919 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:30 smithi099 ceph-mon[39420]: pgmap v2516: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:30.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:30 smithi162 ceph-mon[44304]: pgmap v2516: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:31.496 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T18:06:31.497 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-02-09T18:06:31.497 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 0 2024-02-09T18:06:31.497 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T18:06:31.537 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:31 smithi099 ceph-mon[39420]: 1.1e scrub starts 2024-02-09T18:06:31.537 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:31 smithi099 ceph-mon[39420]: 1.1e scrub ok 2024-02-09T18:06:31.537 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:31 smithi099 ceph-mon[39420]: 1.f scrub starts 2024-02-09T18:06:31.875 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:31 smithi179 ceph-mon[43975]: 1.1e scrub starts 2024-02-09T18:06:31.875 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:31 smithi179 ceph-mon[43975]: 1.1e scrub ok 2024-02-09T18:06:31.875 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:31 smithi179 ceph-mon[43975]: 1.f scrub starts 2024-02-09T18:06:31.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:31 smithi162 ceph-mon[44304]: 1.1e scrub starts 2024-02-09T18:06:31.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:31 smithi162 ceph-mon[44304]: 1.1e scrub ok 2024-02-09T18:06:31.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:31 smithi162 ceph-mon[44304]: 1.f scrub starts 2024-02-09T18:06:32.186 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.11 dump_blocked_ops 2024-02-09T18:06:32.448 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T18:06:32.547 INFO:teuthology.orchestra.run.smithi099.stderr:set osd.10 primary-affinity to 1 (8655362) 2024-02-09T18:06:32.902 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:32 smithi099 ceph-mon[39420]: 1.f scrub ok 2024-02-09T18:06:32.902 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:32 smithi099 ceph-mon[39420]: pgmap v2517: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:32.902 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:32 smithi099 ceph-mon[39420]: 1.1d scrub starts 2024-02-09T18:06:32.902 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:32 smithi099 ceph-mon[39420]: 1.1d scrub ok 2024-02-09T18:06:32.902 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:32 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1930740336' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]: dispatch 2024-02-09T18:06:32.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:32 smithi179 ceph-mon[43975]: 1.f scrub ok 2024-02-09T18:06:32.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:32 smithi179 ceph-mon[43975]: pgmap v2517: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:32.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:32 smithi179 ceph-mon[43975]: 1.1d scrub starts 2024-02-09T18:06:32.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:32 smithi179 ceph-mon[43975]: 1.1d scrub ok 2024-02-09T18:06:32.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:32 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1930740336' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]: dispatch 2024-02-09T18:06:32.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:32 smithi162 ceph-mon[44304]: 1.f scrub ok 2024-02-09T18:06:32.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:32 smithi162 ceph-mon[44304]: pgmap v2517: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:32.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:32 smithi162 ceph-mon[44304]: 1.1d scrub starts 2024-02-09T18:06:32.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:32 smithi162 ceph-mon[44304]: 1.1d scrub ok 2024-02-09T18:06:32.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:32 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1930740336' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]: dispatch 2024-02-09T18:06:33.307 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 11 1 2024-02-09T18:06:33.573 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:06:33.767 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:33 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1930740336' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-02-09T18:06:33.767 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:33 smithi179 ceph-mon[43975]: osdmap e1025: 12 total, 12 up, 12 in 2024-02-09T18:06:33.831 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:33 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1930740336' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-02-09T18:06:33.831 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:33 smithi099 ceph-mon[39420]: osdmap e1025: 12 total, 12 up, 12 in 2024-02-09T18:06:33.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:33 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1930740336' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-02-09T18:06:33.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:33 smithi162 ceph-mon[44304]: osdmap e1025: 12 total, 12 up, 12 in 2024-02-09T18:06:34.352 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T18:06:34.352 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-02-09T18:06:34.353 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-02-09T18:06:34.353 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-02-09T18:06:34.353 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T18:06:34.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:34 smithi179 ceph-mon[43975]: pgmap v2519: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:34.866 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:34 smithi099 ceph-mon[39420]: pgmap v2519: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:34.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:34 smithi162 ceph-mon[44304]: pgmap v2519: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:35.109 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.11 dump_historic_ops 2024-02-09T18:06:35.372 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T18:06:36.562 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.11 to 1 (10000) 2024-02-09T18:06:36.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:36 smithi179 ceph-mon[43975]: pgmap v2520: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:36.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:36 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/580068274' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-02-09T18:06:36.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:36 smithi162 ceph-mon[44304]: pgmap v2520: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:36.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:36 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/580068274' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-02-09T18:06:36.932 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:36 smithi099 ceph-mon[39420]: pgmap v2520: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:36.932 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:36 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/580068274' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-02-09T18:06:37.245 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T18:06:37.245 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-02-09T18:06:37.245 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-02-09T18:06:37.246 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-02-09T18:06:37.246 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.246 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:118924 8.f 8:f9ce6499:::benchmark_data_smithi179_148205_object118923:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e820)", 2024-02-09T18:06:37.246 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:56:51.499652+0000", 2024-02-09T18:06:37.246 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 585.74181710200003, 2024-02-09T18:06:37.246 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.22277385199999999, 2024-02-09T18:06:37.246 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:37.246 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:37.246 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:37.246 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:06:37.246 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:06:37.246 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 118924 2024-02-09T18:06:37.246 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.246 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:37.247 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.247 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:37.247 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:51.499652+0000", 2024-02-09T18:06:37.247 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:37.247 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.247 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.247 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:37.247 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:51.499652+0000", 2024-02-09T18:06:37.247 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:37.247 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.247 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.247 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:37.247 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:51.499652+0000", 2024-02-09T18:06:37.247 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4400000000000001e-07 2024-02-09T18:06:37.248 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.248 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.248 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:37.248 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:51.499685+0000", 2024-02-09T18:06:37.248 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2614999999999999e-05 2024-02-09T18:06:37.248 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.248 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.248 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:37.248 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:51.499689+0000", 2024-02-09T18:06:37.248 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5539999999999999e-06 2024-02-09T18:06:37.248 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.248 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.248 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:37.248 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:51.499692+0000", 2024-02-09T18:06:37.249 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.9969999999999999e-06 2024-02-09T18:06:37.249 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.249 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.249 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:37.249 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:51.508068+0000", 2024-02-09T18:06:37.249 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0083757680000000004 2024-02-09T18:06:37.249 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.249 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.249 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:37.249 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:51.508204+0000", 2024-02-09T18:06:37.249 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013621099999999999 2024-02-09T18:06:37.249 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.249 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.249 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,2", 2024-02-09T18:06:37.249 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:51.508334+0000", 2024-02-09T18:06:37.250 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000129744 2024-02-09T18:06:37.250 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.250 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.250 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:37.250 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:51.519699+0000", 2024-02-09T18:06:37.250 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011364509 2024-02-09T18:06:37.250 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.250 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.250 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:37.250 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:51.529382+0000", 2024-02-09T18:06:37.250 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0096831190000000004 2024-02-09T18:06:37.250 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.250 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.250 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:37.250 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:51.722352+0000", 2024-02-09T18:06:37.251 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.19296983600000001 2024-02-09T18:06:37.251 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.251 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.251 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:37.251 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:51.722399+0000", 2024-02-09T18:06:37.251 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.706e-05 2024-02-09T18:06:37.251 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.251 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.251 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:37.251 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:51.722426+0000", 2024-02-09T18:06:37.251 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7095000000000001e-05 2024-02-09T18:06:37.251 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:37.251 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:37.251 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:37.251 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.252 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.252 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:118934 8.d 8:b1f047fc:::benchmark_data_smithi179_148205_object118933:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e820)", 2024-02-09T18:06:37.252 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:56:51.505801+0000", 2024-02-09T18:06:37.252 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 585.73566807700001, 2024-02-09T18:06:37.252 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.22089035700000001, 2024-02-09T18:06:37.252 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:37.252 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:37.252 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:37.252 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:06:37.252 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:06:37.252 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 118934 2024-02-09T18:06:37.252 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.252 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:37.253 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.253 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:37.253 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:51.505801+0000", 2024-02-09T18:06:37.253 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:37.253 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.253 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.253 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:37.253 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:51.505801+0000", 2024-02-09T18:06:37.253 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:37.253 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.253 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.253 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:37.253 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:51.505803+0000", 2024-02-09T18:06:37.253 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.641e-06 2024-02-09T18:06:37.253 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.254 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.254 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:37.254 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:51.505851+0000", 2024-02-09T18:06:37.254 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8560999999999998e-05 2024-02-09T18:06:37.254 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.254 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.254 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:37.254 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:51.505864+0000", 2024-02-09T18:06:37.254 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2748000000000001e-05 2024-02-09T18:06:37.254 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.254 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.254 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:37.254 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:51.505872+0000", 2024-02-09T18:06:37.254 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8499999999999994e-06 2024-02-09T18:06:37.254 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.255 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.255 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:37.255 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:51.513610+0000", 2024-02-09T18:06:37.255 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0077382900000000001 2024-02-09T18:06:37.255 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.255 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.255 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:37.255 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:51.513753+0000", 2024-02-09T18:06:37.255 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014270900000000001 2024-02-09T18:06:37.255 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.255 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.255 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,7", 2024-02-09T18:06:37.255 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:51.513898+0000", 2024-02-09T18:06:37.255 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014506799999999999 2024-02-09T18:06:37.255 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.255 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.256 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:37.256 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:51.522361+0000", 2024-02-09T18:06:37.256 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0084634380000000002 2024-02-09T18:06:37.256 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.256 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.256 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:37.256 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:51.524344+0000", 2024-02-09T18:06:37.256 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001983149 2024-02-09T18:06:37.256 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.256 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.256 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:37.256 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:51.726601+0000", 2024-02-09T18:06:37.256 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.20225710699999999 2024-02-09T18:06:37.256 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.257 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.257 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:37.257 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:51.726662+0000", 2024-02-09T18:06:37.257 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0608e-05 2024-02-09T18:06:37.257 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.257 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.257 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:37.257 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:51.726691+0000", 2024-02-09T18:06:37.257 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.9187999999999999e-05 2024-02-09T18:06:37.257 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:37.257 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:37.257 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:37.257 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.257 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.257 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:121161 8.d 8:b9cd4525:::benchmark_data_smithi179_148205_object121160:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e821)", 2024-02-09T18:06:37.258 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:56:54.374685+0000", 2024-02-09T18:06:37.258 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 582.86678350299997, 2024-02-09T18:06:37.258 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.22104886000000001, 2024-02-09T18:06:37.258 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:37.258 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:37.258 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:37.258 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:06:37.258 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:06:37.258 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 121161 2024-02-09T18:06:37.258 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.258 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:37.258 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.258 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:37.259 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:54.374685+0000", 2024-02-09T18:06:37.259 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:37.259 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.259 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.259 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:37.259 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:54.374685+0000", 2024-02-09T18:06:37.259 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:37.259 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.259 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.259 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:37.259 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:54.374687+0000", 2024-02-09T18:06:37.259 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3289999999999999e-06 2024-02-09T18:06:37.259 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.259 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.259 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:37.260 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:54.374710+0000", 2024-02-09T18:06:37.260 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2955999999999998e-05 2024-02-09T18:06:37.260 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.260 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.260 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:37.260 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:54.374723+0000", 2024-02-09T18:06:37.260 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3227999999999999e-05 2024-02-09T18:06:37.260 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.260 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.260 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:37.260 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:54.374732+0000", 2024-02-09T18:06:37.260 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.6100000000000006e-06 2024-02-09T18:06:37.260 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.260 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.261 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:37.261 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:54.374960+0000", 2024-02-09T18:06:37.261 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000228641 2024-02-09T18:06:37.261 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.261 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.261 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:37.261 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:54.375091+0000", 2024-02-09T18:06:37.261 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013061600000000001 2024-02-09T18:06:37.261 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.261 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.261 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,7", 2024-02-09T18:06:37.261 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:54.375225+0000", 2024-02-09T18:06:37.261 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000134394 2024-02-09T18:06:37.261 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.262 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.262 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:37.262 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:54.379405+0000", 2024-02-09T18:06:37.262 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0041795000000000001 2024-02-09T18:06:37.262 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.262 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.262 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:37.262 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:54.379890+0000", 2024-02-09T18:06:37.262 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00048558599999999998 2024-02-09T18:06:37.262 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.262 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.262 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:37.262 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:54.595661+0000", 2024-02-09T18:06:37.262 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.21577057199999999 2024-02-09T18:06:37.263 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.263 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.263 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:37.263 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:54.595694+0000", 2024-02-09T18:06:37.263 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2985999999999999e-05 2024-02-09T18:06:37.263 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.263 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.263 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:37.263 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:56:54.595734+0000", 2024-02-09T18:06:37.263 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0441999999999997e-05 2024-02-09T18:06:37.263 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:37.263 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:37.263 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:37.263 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.263 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.264 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:197719 8.6 8:68f66482:::benchmark_data_smithi179_148205_object74647:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e845)", 2024-02-09T18:06:37.264 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:58:29.314106+0000", 2024-02-09T18:06:37.264 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 487.92736296999999, 2024-02-09T18:06:37.264 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.23611070100000001, 2024-02-09T18:06:37.264 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:37.264 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:37.264 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:37.264 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:06:37.264 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:06:37.264 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 197719 2024-02-09T18:06:37.264 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.264 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:37.264 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.265 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:37.265 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:29.314106+0000", 2024-02-09T18:06:37.265 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:37.265 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.265 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.265 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:37.265 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:29.314106+0000", 2024-02-09T18:06:37.265 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:37.265 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.265 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.265 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:37.265 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:29.314108+0000", 2024-02-09T18:06:37.265 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5659999999999999e-06 2024-02-09T18:06:37.265 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.266 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.266 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:37.266 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:29.314113+0000", 2024-02-09T18:06:37.266 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3569999999999997e-06 2024-02-09T18:06:37.266 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.266 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.266 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:37.266 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:29.314129+0000", 2024-02-09T18:06:37.266 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5934e-05 2024-02-09T18:06:37.266 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.267 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.267 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:37.267 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:29.314147+0000", 2024-02-09T18:06:37.267 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8162e-05 2024-02-09T18:06:37.267 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.267 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.267 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:37.267 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:29.314205+0000", 2024-02-09T18:06:37.267 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7546000000000001e-05 2024-02-09T18:06:37.267 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.267 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.267 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:37.267 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:29.314304+0000", 2024-02-09T18:06:37.268 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.8974000000000005e-05 2024-02-09T18:06:37.268 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.268 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.268 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 7,10", 2024-02-09T18:06:37.268 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:29.314376+0000", 2024-02-09T18:06:37.268 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2480999999999999e-05 2024-02-09T18:06:37.268 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.268 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.268 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:37.268 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:29.323147+0000", 2024-02-09T18:06:37.268 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.008770936 2024-02-09T18:06:37.268 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.268 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.268 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:37.268 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:29.329202+0000", 2024-02-09T18:06:37.269 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0060551270000000004 2024-02-09T18:06:37.269 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.269 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.269 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:37.269 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:29.550115+0000", 2024-02-09T18:06:37.269 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.220912632 2024-02-09T18:06:37.269 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.269 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.269 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:37.269 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:29.550162+0000", 2024-02-09T18:06:37.269 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7790000000000002e-05 2024-02-09T18:06:37.269 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.269 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.269 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:37.270 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:29.550217+0000", 2024-02-09T18:06:37.270 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4196000000000001e-05 2024-02-09T18:06:37.270 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:37.270 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:37.270 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:37.270 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.270 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.270 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:197724 8.6 8:6d6b545a:::benchmark_data_smithi179_148205_object74652:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e845)", 2024-02-09T18:06:37.270 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:58:29.318732+0000", 2024-02-09T18:06:37.270 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 487.92273734899999, 2024-02-09T18:06:37.270 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.24356633799999999, 2024-02-09T18:06:37.270 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:37.270 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:37.270 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:37.270 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:06:37.271 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:06:37.271 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 197724 2024-02-09T18:06:37.271 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.271 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:37.271 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.271 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:37.271 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:29.318732+0000", 2024-02-09T18:06:37.271 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:37.271 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.271 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.271 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:37.271 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:29.318732+0000", 2024-02-09T18:06:37.271 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:37.271 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.271 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.272 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:37.272 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:29.318733+0000", 2024-02-09T18:06:37.272 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1429999999999999e-06 2024-02-09T18:06:37.272 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.272 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.272 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:37.272 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:29.318738+0000", 2024-02-09T18:06:37.272 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5890000000000002e-06 2024-02-09T18:06:37.272 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.272 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.272 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:37.272 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:29.318751+0000", 2024-02-09T18:06:37.272 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2656999999999999e-05 2024-02-09T18:06:37.273 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.273 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.273 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:37.273 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:29.318759+0000", 2024-02-09T18:06:37.273 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.0399999999999993e-06 2024-02-09T18:06:37.273 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.273 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.273 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:37.273 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:29.318829+0000", 2024-02-09T18:06:37.273 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9480000000000006e-05 2024-02-09T18:06:37.273 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.273 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.273 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:37.273 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:29.318924+0000", 2024-02-09T18:06:37.273 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5969000000000002e-05 2024-02-09T18:06:37.274 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.274 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.274 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 7,10", 2024-02-09T18:06:37.274 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:29.318999+0000", 2024-02-09T18:06:37.274 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4275000000000003e-05 2024-02-09T18:06:37.274 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.274 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.274 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:37.274 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:29.323328+0000", 2024-02-09T18:06:37.274 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0043287760000000003 2024-02-09T18:06:37.274 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.274 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.274 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:37.274 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:29.329241+0000", 2024-02-09T18:06:37.275 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0059137360000000002 2024-02-09T18:06:37.275 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.275 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.275 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:37.275 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:29.562179+0000", 2024-02-09T18:06:37.275 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.23293786399999999 2024-02-09T18:06:37.275 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.275 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.275 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:37.275 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:29.562236+0000", 2024-02-09T18:06:37.275 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6699e-05 2024-02-09T18:06:37.275 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.275 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.275 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:37.276 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:29.562298+0000", 2024-02-09T18:06:37.276 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.211e-05 2024-02-09T18:06:37.276 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:37.276 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:37.276 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:37.276 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.276 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.276 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:197813 8.6 8:6c26cafb:::benchmark_data_smithi179_148205_object74741:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e845)", 2024-02-09T18:06:37.276 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:58:29.448480+0000", 2024-02-09T18:06:37.276 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 487.79298872300001, 2024-02-09T18:06:37.276 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.246263336, 2024-02-09T18:06:37.276 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:37.276 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:37.276 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:37.276 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:06:37.277 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:06:37.277 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 197813 2024-02-09T18:06:37.277 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.277 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:37.277 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.278 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:37.278 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:29.448480+0000", 2024-02-09T18:06:37.278 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:37.278 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.278 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.278 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:37.278 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:29.448480+0000", 2024-02-09T18:06:37.278 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:37.278 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.278 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.278 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:37.278 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:29.448482+0000", 2024-02-09T18:06:37.278 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.297e-06 2024-02-09T18:06:37.279 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.279 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.279 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:37.279 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:29.448487+0000", 2024-02-09T18:06:37.279 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1789999999999999e-06 2024-02-09T18:06:37.279 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.279 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.279 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:37.279 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:29.448503+0000", 2024-02-09T18:06:37.279 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.613e-05 2024-02-09T18:06:37.279 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.279 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.279 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:37.279 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:29.448510+0000", 2024-02-09T18:06:37.280 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9720000000000003e-06 2024-02-09T18:06:37.280 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.280 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.280 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:37.280 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:29.448555+0000", 2024-02-09T18:06:37.280 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4700999999999998e-05 2024-02-09T18:06:37.280 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.280 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.280 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:37.280 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:29.448678+0000", 2024-02-09T18:06:37.280 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012394700000000001 2024-02-09T18:06:37.280 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.280 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.280 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 7,10", 2024-02-09T18:06:37.281 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:29.448757+0000", 2024-02-09T18:06:37.281 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8343999999999994e-05 2024-02-09T18:06:37.281 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.281 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.281 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:37.281 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:29.459817+0000", 2024-02-09T18:06:37.281 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011060396 2024-02-09T18:06:37.281 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.281 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.281 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:37.281 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:29.460678+0000", 2024-02-09T18:06:37.281 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00086114299999999998 2024-02-09T18:06:37.281 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.281 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.281 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:37.282 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:29.694636+0000", 2024-02-09T18:06:37.282 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.23395796399999999 2024-02-09T18:06:37.282 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.282 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.282 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:37.282 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:29.694694+0000", 2024-02-09T18:06:37.282 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7655999999999998e-05 2024-02-09T18:06:37.282 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.282 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.282 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:37.282 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:29.694744+0000", 2024-02-09T18:06:37.282 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9607000000000001e-05 2024-02-09T18:06:37.282 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:37.282 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:37.282 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:37.283 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.283 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.283 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:198410 8.6 8:6c8704ab:::benchmark_data_smithi179_148205_object75338:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e845)", 2024-02-09T18:06:37.283 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:58:30.597628+0000", 2024-02-09T18:06:37.283 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 486.64384142900002, 2024-02-09T18:06:37.283 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.23764317700000001, 2024-02-09T18:06:37.283 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:37.283 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:37.283 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:37.283 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:06:37.283 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:06:37.283 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 198410 2024-02-09T18:06:37.283 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.283 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:37.283 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.284 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:37.284 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:30.597628+0000", 2024-02-09T18:06:37.284 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:37.284 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.284 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.284 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:37.284 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:30.597628+0000", 2024-02-09T18:06:37.284 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:37.284 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.284 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.284 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:37.284 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:30.597629+0000", 2024-02-09T18:06:37.284 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4249999999999999e-06 2024-02-09T18:06:37.284 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.284 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.285 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:37.285 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:30.597634+0000", 2024-02-09T18:06:37.285 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.254e-06 2024-02-09T18:06:37.285 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.285 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.285 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:37.285 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:30.597646+0000", 2024-02-09T18:06:37.285 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2269000000000001e-05 2024-02-09T18:06:37.285 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.285 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.285 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:37.285 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:30.597654+0000", 2024-02-09T18:06:37.285 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4109999999999999e-06 2024-02-09T18:06:37.285 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.286 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.286 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:37.286 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:30.667137+0000", 2024-02-09T18:06:37.286 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.069483090999999997 2024-02-09T18:06:37.286 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.286 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.286 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:37.286 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:30.667253+0000", 2024-02-09T18:06:37.286 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000116085 2024-02-09T18:06:37.286 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.286 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.286 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 7,10", 2024-02-09T18:06:37.286 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:30.667367+0000", 2024-02-09T18:06:37.286 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000113455 2024-02-09T18:06:37.286 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.287 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.287 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:37.287 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:30.678016+0000", 2024-02-09T18:06:37.287 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010649034999999999 2024-02-09T18:06:37.287 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.287 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.287 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:37.287 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:30.678099+0000", 2024-02-09T18:06:37.287 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3834999999999999e-05 2024-02-09T18:06:37.287 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.287 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.287 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:37.287 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:30.835179+0000", 2024-02-09T18:06:37.287 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15708006899999999 2024-02-09T18:06:37.288 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.288 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.288 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:37.288 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:30.835217+0000", 2024-02-09T18:06:37.288 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7972000000000002e-05 2024-02-09T18:06:37.288 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.288 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.288 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:37.288 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:30.835271+0000", 2024-02-09T18:06:37.288 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3276000000000003e-05 2024-02-09T18:06:37.288 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:37.288 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:37.288 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:37.288 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.289 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.289 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:198695 8.6 8:60eac4b4:::benchmark_data_smithi179_148205_object75623:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e845)", 2024-02-09T18:06:37.289 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:58:31.140807+0000", 2024-02-09T18:06:37.289 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 486.10066174399998, 2024-02-09T18:06:37.289 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.27579035699999999, 2024-02-09T18:06:37.289 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:37.289 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:37.289 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:37.289 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:06:37.289 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:06:37.289 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 198695 2024-02-09T18:06:37.289 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.289 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:37.289 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.289 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:37.290 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:31.140807+0000", 2024-02-09T18:06:37.290 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:37.290 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.290 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.290 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:37.290 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:31.140807+0000", 2024-02-09T18:06:37.290 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:37.290 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.290 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.290 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:37.290 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:31.140808+0000", 2024-02-09T18:06:37.290 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.0100000000000004e-07 2024-02-09T18:06:37.290 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.290 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.291 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:37.291 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:31.140812+0000", 2024-02-09T18:06:37.291 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9140000000000001e-06 2024-02-09T18:06:37.291 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.291 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.291 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:37.291 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:31.140823+0000", 2024-02-09T18:06:37.291 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0784e-05 2024-02-09T18:06:37.291 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.291 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.291 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:37.291 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:31.140829+0000", 2024-02-09T18:06:37.291 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4760000000000003e-06 2024-02-09T18:06:37.291 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.292 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.292 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:37.292 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:31.246071+0000", 2024-02-09T18:06:37.292 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.105242137 2024-02-09T18:06:37.292 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.292 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.292 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:37.292 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:31.246202+0000", 2024-02-09T18:06:37.292 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013021500000000001 2024-02-09T18:06:37.292 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.292 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.292 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 7,10", 2024-02-09T18:06:37.292 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:31.246330+0000", 2024-02-09T18:06:37.292 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000128236 2024-02-09T18:06:37.293 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.293 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.293 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:37.293 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:31.257870+0000", 2024-02-09T18:06:37.293 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011540668 2024-02-09T18:06:37.293 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.293 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.293 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:37.293 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:31.272358+0000", 2024-02-09T18:06:37.293 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014487778999999999 2024-02-09T18:06:37.293 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.293 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.293 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:37.293 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:31.416479+0000", 2024-02-09T18:06:37.293 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.144120321 2024-02-09T18:06:37.294 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.294 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.294 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:37.294 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:31.416536+0000", 2024-02-09T18:06:37.294 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7345000000000001e-05 2024-02-09T18:06:37.294 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.294 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.294 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:37.294 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:31.416598+0000", 2024-02-09T18:06:37.294 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1680999999999994e-05 2024-02-09T18:06:37.294 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:37.294 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:37.294 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:37.294 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.294 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.295 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:198719 8.6 8:6148d776:::benchmark_data_smithi179_148205_object75647:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e845)", 2024-02-09T18:06:37.295 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:58:31.192406+0000", 2024-02-09T18:06:37.295 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 486.04906304399998, 2024-02-09T18:06:37.295 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.22432825100000001, 2024-02-09T18:06:37.295 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:37.295 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:37.295 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:37.295 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:06:37.295 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:06:37.295 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 198719 2024-02-09T18:06:37.295 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.295 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:37.295 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.295 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:37.295 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:31.192406+0000", 2024-02-09T18:06:37.296 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:37.296 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.296 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.296 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:37.296 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:31.192406+0000", 2024-02-09T18:06:37.296 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:37.296 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.296 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.296 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:37.296 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:31.192407+0000", 2024-02-09T18:06:37.296 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9400000000000005e-07 2024-02-09T18:06:37.296 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.296 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.296 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:37.296 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:31.192410+0000", 2024-02-09T18:06:37.297 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.044e-06 2024-02-09T18:06:37.297 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.297 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.297 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:37.297 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:31.192418+0000", 2024-02-09T18:06:37.297 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.276e-06 2024-02-09T18:06:37.297 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.297 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.297 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:37.297 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:31.192424+0000", 2024-02-09T18:06:37.297 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3430000000000001e-06 2024-02-09T18:06:37.297 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.297 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.297 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:37.298 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:31.251587+0000", 2024-02-09T18:06:37.298 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.059162679000000003 2024-02-09T18:06:37.298 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.298 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.298 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:37.298 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:31.251702+0000", 2024-02-09T18:06:37.298 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000114701 2024-02-09T18:06:37.298 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.298 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.298 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 7,10", 2024-02-09T18:06:37.298 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:31.251832+0000", 2024-02-09T18:06:37.298 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013071299999999999 2024-02-09T18:06:37.298 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.298 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.298 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:37.299 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:31.257876+0000", 2024-02-09T18:06:37.299 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0060441499999999999 2024-02-09T18:06:37.299 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.299 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.299 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:37.299 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:31.272397+0000", 2024-02-09T18:06:37.299 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014521076000000001 2024-02-09T18:06:37.299 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.299 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.299 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:37.299 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:31.416641+0000", 2024-02-09T18:06:37.299 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14424366 2024-02-09T18:06:37.299 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.299 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.300 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:37.300 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:31.416690+0000", 2024-02-09T18:06:37.300 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9115e-05 2024-02-09T18:06:37.300 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.300 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.300 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:37.300 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:58:31.416734+0000", 2024-02-09T18:06:37.300 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3900000000000003e-05 2024-02-09T18:06:37.300 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:37.300 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:37.300 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:37.300 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.300 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.300 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:203000 9.9 9:95585f2b:::benchmark_data_smithi179_161915_object68118:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e942)", 2024-02-09T18:06:37.301 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:13.464725+0000", 2024-02-09T18:06:37.301 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 203.776743993, 2024-02-09T18:06:37.301 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2018209820000001, 2024-02-09T18:06:37.301 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:37.301 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:37.301 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:37.301 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:06:37.301 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:37.301 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 203000 2024-02-09T18:06:37.301 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.301 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:37.301 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.301 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:37.301 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.464725+0000", 2024-02-09T18:06:37.301 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:37.302 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.302 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.302 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:37.302 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.464725+0000", 2024-02-09T18:06:37.302 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:37.302 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.302 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.302 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:37.302 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.464726+0000", 2024-02-09T18:06:37.302 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.192e-06 2024-02-09T18:06:37.302 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.302 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.302 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:37.302 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.464730+0000", 2024-02-09T18:06:37.303 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3359999999999997e-06 2024-02-09T18:06:37.303 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.303 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.303 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:37.303 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.464743+0000", 2024-02-09T18:06:37.303 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2103000000000001e-05 2024-02-09T18:06:37.303 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.303 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.303 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:37.303 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.464749+0000", 2024-02-09T18:06:37.303 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.336e-06 2024-02-09T18:06:37.303 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.303 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.303 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:37.303 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.464796+0000", 2024-02-09T18:06:37.304 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7413999999999997e-05 2024-02-09T18:06:37.304 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.304 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.304 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:37.304 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.464804+0000", 2024-02-09T18:06:37.304 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2350000000000002e-06 2024-02-09T18:06:37.304 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.304 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.304 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:37.304 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:14.439558+0000", 2024-02-09T18:06:37.304 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.97475451300000004 2024-02-09T18:06:37.304 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.304 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.305 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:37.305 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:14.439575+0000", 2024-02-09T18:06:37.305 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6426000000000002e-05 2024-02-09T18:06:37.305 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.305 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.305 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:37.305 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.649193+0000", 2024-02-09T18:06:37.305 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.209618643 2024-02-09T18:06:37.305 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.305 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.305 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:37.305 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.652658+0000", 2024-02-09T18:06:37.305 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0034649189999999999 2024-02-09T18:06:37.305 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.305 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.306 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,5,10", 2024-02-09T18:06:37.306 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.652797+0000", 2024-02-09T18:06:37.306 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000139276 2024-02-09T18:06:37.306 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.306 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.306 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:37.306 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.657238+0000", 2024-02-09T18:06:37.306 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0044404370000000002 2024-02-09T18:06:37.306 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.306 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.306 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:37.306 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.666204+0000", 2024-02-09T18:06:37.306 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0089665979999999992 2024-02-09T18:06:37.306 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.307 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.307 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:37.307 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.666381+0000", 2024-02-09T18:06:37.307 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000177092 2024-02-09T18:06:37.307 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.307 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.307 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:37.307 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.666415+0000", 2024-02-09T18:06:37.307 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.3497000000000001e-05 2024-02-09T18:06:37.307 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.307 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.307 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:37.307 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.666451+0000", 2024-02-09T18:06:37.307 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6146999999999998e-05 2024-02-09T18:06:37.308 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.308 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.308 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:37.308 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.666546+0000", 2024-02-09T18:06:37.308 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.4817999999999999e-05 2024-02-09T18:06:37.308 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:37.308 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:37.308 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:37.308 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.308 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.308 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:203006 9.9 9:960b5c54:::benchmark_data_smithi179_161915_object68124:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e942)", 2024-02-09T18:06:37.308 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:13.478261+0000", 2024-02-09T18:06:37.308 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 203.76320835499999, 2024-02-09T18:06:37.308 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1982893250000002, 2024-02-09T18:06:37.309 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:37.309 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:37.309 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:37.309 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:06:37.309 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:37.309 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 203006 2024-02-09T18:06:37.309 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.309 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:37.309 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.309 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:37.309 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.478261+0000", 2024-02-09T18:06:37.309 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:37.309 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.309 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.309 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:37.310 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.478261+0000", 2024-02-09T18:06:37.310 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:37.310 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.310 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.310 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:37.310 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.478262+0000", 2024-02-09T18:06:37.310 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6610000000000001e-06 2024-02-09T18:06:37.310 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.310 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.310 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:37.310 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.478268+0000", 2024-02-09T18:06:37.310 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7790000000000001e-06 2024-02-09T18:06:37.310 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.310 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.310 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:37.311 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.478281+0000", 2024-02-09T18:06:37.311 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.256e-05 2024-02-09T18:06:37.311 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.311 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.311 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:37.311 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.478289+0000", 2024-02-09T18:06:37.311 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5210000000000002e-06 2024-02-09T18:06:37.311 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.311 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.311 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:37.311 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.478352+0000", 2024-02-09T18:06:37.311 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2510999999999998e-05 2024-02-09T18:06:37.311 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.311 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.312 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:37.312 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.478364+0000", 2024-02-09T18:06:37.312 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.224e-05 2024-02-09T18:06:37.312 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.312 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.312 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:37.312 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:14.439608+0000", 2024-02-09T18:06:37.312 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.96124383999999996 2024-02-09T18:06:37.312 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.312 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.312 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:37.312 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:14.439617+0000", 2024-02-09T18:06:37.312 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.4879999999999996e-06 2024-02-09T18:06:37.312 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.313 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.313 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:37.313 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.657283+0000", 2024-02-09T18:06:37.313 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2176659919999999 2024-02-09T18:06:37.313 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.313 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.313 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:37.313 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.657377+0000", 2024-02-09T18:06:37.313 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.3628000000000005e-05 2024-02-09T18:06:37.313 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.313 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.313 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,5,10", 2024-02-09T18:06:37.313 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.657480+0000", 2024-02-09T18:06:37.313 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000103057 2024-02-09T18:06:37.314 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.314 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.314 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:37.314 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.666391+0000", 2024-02-09T18:06:37.314 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0089111910000000006 2024-02-09T18:06:37.314 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.314 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.314 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:37.314 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.666600+0000", 2024-02-09T18:06:37.314 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00020844900000000001 2024-02-09T18:06:37.314 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.314 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.314 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:37.314 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.666664+0000", 2024-02-09T18:06:37.314 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4585999999999995e-05 2024-02-09T18:06:37.315 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.315 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.315 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:37.315 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.676375+0000", 2024-02-09T18:06:37.315 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0097106750000000002 2024-02-09T18:06:37.315 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.315 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.315 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:37.315 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.676434+0000", 2024-02-09T18:06:37.315 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8765999999999999e-05 2024-02-09T18:06:37.315 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.315 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.315 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:37.315 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.676550+0000", 2024-02-09T18:06:37.315 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000116381 2024-02-09T18:06:37.316 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:37.316 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:37.316 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:37.316 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.316 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.316 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:203013 9.9 9:9e25c461:::benchmark_data_smithi179_161915_object68131:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e942)", 2024-02-09T18:06:37.316 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:13.481430+0000", 2024-02-09T18:06:37.316 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 203.76003928099999, 2024-02-09T18:06:37.316 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1954094309999999, 2024-02-09T18:06:37.316 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:37.316 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:37.316 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:37.316 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:06:37.316 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:37.317 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 203013 2024-02-09T18:06:37.317 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.317 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:37.317 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.317 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:37.317 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.481430+0000", 2024-02-09T18:06:37.317 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:37.317 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.317 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.317 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:37.317 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.481430+0000", 2024-02-09T18:06:37.317 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:37.317 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.317 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.318 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:37.318 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.481431+0000", 2024-02-09T18:06:37.318 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.083e-06 2024-02-09T18:06:37.318 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.318 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.318 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:37.318 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.481436+0000", 2024-02-09T18:06:37.318 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8690000000000003e-06 2024-02-09T18:06:37.318 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.318 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.318 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:37.318 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.481448+0000", 2024-02-09T18:06:37.318 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2598e-05 2024-02-09T18:06:37.318 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.319 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.319 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:37.319 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.481456+0000", 2024-02-09T18:06:37.319 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7170000000000008e-06 2024-02-09T18:06:37.319 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.319 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.319 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:37.319 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.481516+0000", 2024-02-09T18:06:37.319 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9859999999999999e-05 2024-02-09T18:06:37.319 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.319 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.319 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:37.319 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.481528+0000", 2024-02-09T18:06:37.319 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1764e-05 2024-02-09T18:06:37.319 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.320 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.320 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:37.320 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:14.439641+0000", 2024-02-09T18:06:37.320 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.95811385199999999 2024-02-09T18:06:37.320 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.320 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.320 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:37.320 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:14.439651+0000", 2024-02-09T18:06:37.320 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.38e-06 2024-02-09T18:06:37.320 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.320 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.320 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:37.320 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.661806+0000", 2024-02-09T18:06:37.320 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2221547349999999 2024-02-09T18:06:37.321 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.321 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.321 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:37.321 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.661929+0000", 2024-02-09T18:06:37.321 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000123089 2024-02-09T18:06:37.321 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.321 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.321 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,5,10", 2024-02-09T18:06:37.321 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.662024+0000", 2024-02-09T18:06:37.321 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5810000000000006e-05 2024-02-09T18:06:37.321 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.321 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.321 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:37.321 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.666616+0000", 2024-02-09T18:06:37.322 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0045914010000000002 2024-02-09T18:06:37.322 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.322 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.322 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:37.322 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.666717+0000", 2024-02-09T18:06:37.322 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010158999999999999 2024-02-09T18:06:37.322 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.322 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.322 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:37.322 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.666839+0000", 2024-02-09T18:06:37.322 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012170700000000001 2024-02-09T18:06:37.322 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.322 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.322 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:37.322 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.676686+0000", 2024-02-09T18:06:37.323 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0098467799999999994 2024-02-09T18:06:37.323 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.323 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.323 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:37.323 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.676737+0000", 2024-02-09T18:06:37.323 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0878999999999999e-05 2024-02-09T18:06:37.323 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.323 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.323 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:37.323 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.676839+0000", 2024-02-09T18:06:37.323 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000102317 2024-02-09T18:06:37.323 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:37.323 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:37.323 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:37.323 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.323 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.324 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:203014 9.9 9:9ed7597e:::benchmark_data_smithi179_161915_object68132:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e942)", 2024-02-09T18:06:37.324 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:13.483816+0000", 2024-02-09T18:06:37.324 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 203.757652608, 2024-02-09T18:06:37.324 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2060873989999998, 2024-02-09T18:06:37.324 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:37.324 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:37.324 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:37.324 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:06:37.324 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:37.324 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 203014 2024-02-09T18:06:37.324 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.324 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:37.324 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.324 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:37.325 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.483816+0000", 2024-02-09T18:06:37.325 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:37.325 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.325 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.325 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:37.325 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.483816+0000", 2024-02-09T18:06:37.325 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:37.325 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.325 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.325 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:37.325 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.483817+0000", 2024-02-09T18:06:37.325 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.4799999999999997e-07 2024-02-09T18:06:37.325 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.325 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.325 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:37.326 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.483822+0000", 2024-02-09T18:06:37.326 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6979999999999997e-06 2024-02-09T18:06:37.326 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.326 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.326 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:37.326 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.483832+0000", 2024-02-09T18:06:37.326 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.9799999999999993e-06 2024-02-09T18:06:37.326 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.326 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.326 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:37.326 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.483838+0000", 2024-02-09T18:06:37.326 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5419999999999996e-06 2024-02-09T18:06:37.326 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.326 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.326 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:37.327 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.483891+0000", 2024-02-09T18:06:37.327 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3609999999999997e-05 2024-02-09T18:06:37.327 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.327 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.327 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:37.327 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.483904+0000", 2024-02-09T18:06:37.327 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2789000000000001e-05 2024-02-09T18:06:37.327 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.327 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.327 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:37.327 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:14.439676+0000", 2024-02-09T18:06:37.327 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.95577193299999996 2024-02-09T18:06:37.327 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.327 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.327 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:37.328 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:14.439685+0000", 2024-02-09T18:06:37.328 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.833e-06 2024-02-09T18:06:37.328 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.328 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.328 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:37.328 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.666865+0000", 2024-02-09T18:06:37.328 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2271803320000001 2024-02-09T18:06:37.328 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.328 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.328 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:37.328 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.666946+0000", 2024-02-09T18:06:37.328 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1073000000000001e-05 2024-02-09T18:06:37.328 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.328 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.329 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,5,10", 2024-02-09T18:06:37.329 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.667048+0000", 2024-02-09T18:06:37.329 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010150899999999999 2024-02-09T18:06:37.329 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.329 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.329 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:37.329 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.676602+0000", 2024-02-09T18:06:37.329 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0095542260000000007 2024-02-09T18:06:37.329 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.329 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.329 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:37.329 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.676648+0000", 2024-02-09T18:06:37.329 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5739999999999999e-05 2024-02-09T18:06:37.329 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.329 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.330 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:37.330 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.689675+0000", 2024-02-09T18:06:37.330 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013027232 2024-02-09T18:06:37.330 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.330 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.330 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:37.330 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.689733+0000", 2024-02-09T18:06:37.330 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7988000000000001e-05 2024-02-09T18:06:37.330 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.330 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.330 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:37.330 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.689790+0000", 2024-02-09T18:06:37.330 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6903000000000001e-05 2024-02-09T18:06:37.330 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.330 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.331 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:37.331 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.689904+0000", 2024-02-09T18:06:37.331 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000114063 2024-02-09T18:06:37.331 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:37.331 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:37.331 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:37.331 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.331 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.331 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:218837 9.9 9:94d5c2ba:::benchmark_data_smithi179_161915_object83955:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e955)", 2024-02-09T18:06:37.331 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:39.717899+0000", 2024-02-09T18:06:37.331 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 177.52357014200001, 2024-02-09T18:06:37.331 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0320935040000001, 2024-02-09T18:06:37.331 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:37.331 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:37.331 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:37.332 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:06:37.332 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:37.332 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 218837 2024-02-09T18:06:37.332 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.332 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:37.332 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.332 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:37.332 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.717899+0000", 2024-02-09T18:06:37.332 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:37.332 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.332 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.333 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:37.333 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.717899+0000", 2024-02-09T18:06:37.333 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:37.333 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.333 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.333 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:37.333 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.717912+0000", 2024-02-09T18:06:37.333 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.31e-05 2024-02-09T18:06:37.333 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.333 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.333 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:37.333 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.717917+0000", 2024-02-09T18:06:37.333 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9540000000000003e-06 2024-02-09T18:06:37.333 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.333 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.333 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:37.334 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.717929+0000", 2024-02-09T18:06:37.334 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1671000000000001e-05 2024-02-09T18:06:37.334 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.334 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.334 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:37.334 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.717938+0000", 2024-02-09T18:06:37.334 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7999999999999993e-06 2024-02-09T18:06:37.334 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.334 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.334 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:37.334 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.718347+0000", 2024-02-09T18:06:37.334 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00040833900000000002 2024-02-09T18:06:37.334 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.334 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.334 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:37.335 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.718359+0000", 2024-02-09T18:06:37.335 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2781e-05 2024-02-09T18:06:37.335 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.335 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.335 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:37.335 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.744162+0000", 2024-02-09T18:06:37.335 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0258021429999999 2024-02-09T18:06:37.335 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.335 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.335 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:37.335 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.744287+0000", 2024-02-09T18:06:37.335 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000125182 2024-02-09T18:06:37.335 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.335 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.336 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,5", 2024-02-09T18:06:37.336 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.744405+0000", 2024-02-09T18:06:37.336 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000118467 2024-02-09T18:06:37.336 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.336 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.336 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:37.336 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.749890+0000", 2024-02-09T18:06:37.336 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0054846590000000002 2024-02-09T18:06:37.336 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.336 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.336 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:37.336 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.749905+0000", 2024-02-09T18:06:37.336 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4737e-05 2024-02-09T18:06:37.336 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.336 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.337 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:37.337 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.749928+0000", 2024-02-09T18:06:37.337 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3482999999999999e-05 2024-02-09T18:06:37.337 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.337 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.337 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:37.337 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.749965+0000", 2024-02-09T18:06:37.337 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6423e-05 2024-02-09T18:06:37.337 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.337 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.337 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:37.337 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.749992+0000", 2024-02-09T18:06:37.337 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7764999999999999e-05 2024-02-09T18:06:37.337 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:37.337 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:37.338 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:37.338 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.338 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.338 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:218876 9.9 9:95e61885:::benchmark_data_smithi179_161915_object83994:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e955)", 2024-02-09T18:06:37.338 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:39.758823+0000", 2024-02-09T18:06:37.338 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 177.48264591099999, 2024-02-09T18:06:37.338 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.001186785, 2024-02-09T18:06:37.338 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:37.338 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:37.338 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:37.338 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:06:37.338 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:37.338 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 218876 2024-02-09T18:06:37.338 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.339 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:37.339 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.339 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:37.339 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.758823+0000", 2024-02-09T18:06:37.339 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:37.339 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.339 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.339 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:37.339 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.758823+0000", 2024-02-09T18:06:37.339 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:37.339 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.339 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.339 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:37.339 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.758825+0000", 2024-02-09T18:06:37.339 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7859999999999999e-06 2024-02-09T18:06:37.340 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.340 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.340 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:37.340 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.758830+0000", 2024-02-09T18:06:37.340 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6579999999999999e-06 2024-02-09T18:06:37.340 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.340 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.340 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:37.340 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.758842+0000", 2024-02-09T18:06:37.340 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1576000000000001e-05 2024-02-09T18:06:37.340 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.340 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.340 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:37.340 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.758851+0000", 2024-02-09T18:06:37.340 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.0529999999999996e-06 2024-02-09T18:06:37.341 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.341 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.341 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:37.341 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.758911+0000", 2024-02-09T18:06:37.341 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0059000000000002e-05 2024-02-09T18:06:37.341 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.341 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.341 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:37.341 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.758918+0000", 2024-02-09T18:06:37.341 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.0090000000000001e-06 2024-02-09T18:06:37.341 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.341 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.341 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:37.341 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.750036+0000", 2024-02-09T18:06:37.342 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.99111734799999995 2024-02-09T18:06:37.342 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.342 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.342 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:37.342 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.750159+0000", 2024-02-09T18:06:37.342 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012359799999999999 2024-02-09T18:06:37.342 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.342 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.342 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,5", 2024-02-09T18:06:37.342 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.750275+0000", 2024-02-09T18:06:37.342 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000115798 2024-02-09T18:06:37.342 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.342 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.342 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:37.342 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.759851+0000", 2024-02-09T18:06:37.343 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0095757789999999995 2024-02-09T18:06:37.343 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.343 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.343 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:37.343 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.759900+0000", 2024-02-09T18:06:37.343 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9329000000000002e-05 2024-02-09T18:06:37.343 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.343 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.343 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:37.343 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.759914+0000", 2024-02-09T18:06:37.343 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3696999999999999e-05 2024-02-09T18:06:37.343 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.343 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.343 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:37.344 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.759971+0000", 2024-02-09T18:06:37.344 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7253999999999998e-05 2024-02-09T18:06:37.344 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.344 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.344 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:37.344 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.760010+0000", 2024-02-09T18:06:37.344 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8841000000000003e-05 2024-02-09T18:06:37.344 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:37.344 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:37.344 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:37.344 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.344 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.344 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:218932 9.9 9:94d28bc6:::benchmark_data_smithi179_161915_object84050:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e955)", 2024-02-09T18:06:37.344 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:39.836014+0000", 2024-02-09T18:06:37.345 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 177.40545512, 2024-02-09T18:06:37.345 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.92410255600000002, 2024-02-09T18:06:37.345 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:37.345 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:37.345 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:37.345 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:06:37.345 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:37.345 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 218932 2024-02-09T18:06:37.345 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.345 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:37.345 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.345 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:37.345 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.836014+0000", 2024-02-09T18:06:37.345 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:37.345 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.346 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.346 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:37.346 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.836014+0000", 2024-02-09T18:06:37.346 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:37.346 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.346 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.346 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:37.346 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.836015+0000", 2024-02-09T18:06:37.346 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4839999999999999e-06 2024-02-09T18:06:37.346 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.346 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.346 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:37.346 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.836022+0000", 2024-02-09T18:06:37.346 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.319e-06 2024-02-09T18:06:37.347 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.347 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.347 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:37.347 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.836036+0000", 2024-02-09T18:06:37.347 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4409e-05 2024-02-09T18:06:37.347 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.347 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.347 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:37.347 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.836045+0000", 2024-02-09T18:06:37.347 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7860000000000002e-06 2024-02-09T18:06:37.347 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.347 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.347 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:37.347 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.836094+0000", 2024-02-09T18:06:37.347 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8962999999999999e-05 2024-02-09T18:06:37.348 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.348 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.348 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:37.348 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.836101+0000", 2024-02-09T18:06:37.348 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4340000000000003e-06 2024-02-09T18:06:37.348 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.348 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.348 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:37.348 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.755295+0000", 2024-02-09T18:06:37.348 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.91919386000000003 2024-02-09T18:06:37.348 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.348 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.348 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:37.348 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.755388+0000", 2024-02-09T18:06:37.348 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.3094e-05 2024-02-09T18:06:37.349 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.349 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.349 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,5", 2024-02-09T18:06:37.349 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.755461+0000", 2024-02-09T18:06:37.349 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.3140999999999996e-05 2024-02-09T18:06:37.349 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.349 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.349 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:37.349 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.760020+0000", 2024-02-09T18:06:37.349 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0045591219999999997 2024-02-09T18:06:37.349 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.349 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.349 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:37.349 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.760051+0000", 2024-02-09T18:06:37.349 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0275999999999999e-05 2024-02-09T18:06:37.350 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.350 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.350 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:37.350 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.760074+0000", 2024-02-09T18:06:37.350 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3413000000000001e-05 2024-02-09T18:06:37.350 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.350 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.350 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:37.350 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.760098+0000", 2024-02-09T18:06:37.350 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3921e-05 2024-02-09T18:06:37.350 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.350 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.350 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:37.350 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.760116+0000", 2024-02-09T18:06:37.350 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8334000000000001e-05 2024-02-09T18:06:37.351 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:37.351 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:37.351 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:37.351 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.351 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.351 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:218955 9.9 9:91ed5d15:::benchmark_data_smithi179_161915_object84073:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e955)", 2024-02-09T18:06:37.351 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:39.883952+0000", 2024-02-09T18:06:37.351 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 177.35751706100001, 2024-02-09T18:06:37.351 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.88369091899999996, 2024-02-09T18:06:37.351 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:37.351 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:37.351 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:37.351 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:06:37.351 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:37.352 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 218955 2024-02-09T18:06:37.352 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.352 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:37.352 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.352 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:37.352 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.883952+0000", 2024-02-09T18:06:37.352 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:37.352 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.352 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.352 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:37.352 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.883952+0000", 2024-02-09T18:06:37.352 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:37.352 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.352 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.352 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:37.353 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.883953+0000", 2024-02-09T18:06:37.353 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3209999999999999e-06 2024-02-09T18:06:37.353 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.353 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.353 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:37.353 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.883959+0000", 2024-02-09T18:06:37.353 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5110000000000003e-06 2024-02-09T18:06:37.353 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.353 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.353 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:37.353 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.883968+0000", 2024-02-09T18:06:37.353 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7389999999999994e-06 2024-02-09T18:06:37.353 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.353 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.353 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:37.354 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.883975+0000", 2024-02-09T18:06:37.354 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5230000000000001e-06 2024-02-09T18:06:37.354 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.354 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.354 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:37.354 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.884030+0000", 2024-02-09T18:06:37.354 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5445999999999997e-05 2024-02-09T18:06:37.354 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.354 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.354 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:37.354 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.884038+0000", 2024-02-09T18:06:37.354 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.0409999999999998e-06 2024-02-09T18:06:37.354 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.354 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.354 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:37.355 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.760177+0000", 2024-02-09T18:06:37.355 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.87613875299999999 2024-02-09T18:06:37.355 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.355 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.355 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:37.355 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.760285+0000", 2024-02-09T18:06:37.355 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000107898 2024-02-09T18:06:37.355 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.355 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.355 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,5", 2024-02-09T18:06:37.355 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.760386+0000", 2024-02-09T18:06:37.355 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010107699999999999 2024-02-09T18:06:37.355 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.355 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.355 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:37.356 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.764870+0000", 2024-02-09T18:06:37.356 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.004483351 2024-02-09T18:06:37.356 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.356 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.356 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:37.356 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.767120+0000", 2024-02-09T18:06:37.356 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0022507529999999999 2024-02-09T18:06:37.356 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.356 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.356 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:37.356 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.767543+0000", 2024-02-09T18:06:37.356 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00042264100000000002 2024-02-09T18:06:37.356 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.356 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.357 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:37.357 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.767598+0000", 2024-02-09T18:06:37.357 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5387999999999999e-05 2024-02-09T18:06:37.357 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.357 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.357 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:37.357 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.767643+0000", 2024-02-09T18:06:37.357 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4477000000000001e-05 2024-02-09T18:06:37.357 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:37.357 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:37.357 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:37.357 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.357 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.357 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:261006 9.5 9:ac266569:::benchmark_data_smithi179_161915_object126124:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e973)", 2024-02-09T18:06:37.357 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:04:31.962376+0000", 2024-02-09T18:06:37.358 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 125.279093301, 2024-02-09T18:06:37.358 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.234158806, 2024-02-09T18:06:37.358 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:37.358 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:37.358 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:37.358 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:06:37.358 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:37.358 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 261006 2024-02-09T18:06:37.358 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.358 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:37.358 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.358 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:37.358 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:31.962376+0000", 2024-02-09T18:06:37.358 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:37.358 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.359 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.359 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:37.359 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:31.962376+0000", 2024-02-09T18:06:37.359 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:37.359 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.359 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.359 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:37.359 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:31.962377+0000", 2024-02-09T18:06:37.359 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.339e-06 2024-02-09T18:06:37.359 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.359 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.359 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:37.359 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:31.962382+0000", 2024-02-09T18:06:37.359 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2469999999999999e-06 2024-02-09T18:06:37.359 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.360 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.360 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:37.360 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:31.962392+0000", 2024-02-09T18:06:37.360 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5240000000000006e-06 2024-02-09T18:06:37.360 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.360 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.360 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:37.360 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:31.962398+0000", 2024-02-09T18:06:37.360 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9329999999999998e-06 2024-02-09T18:06:37.360 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.360 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.360 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:37.360 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:31.963321+0000", 2024-02-09T18:06:37.360 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000923758 2024-02-09T18:06:37.360 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.361 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.361 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:37.361 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:31.963443+0000", 2024-02-09T18:06:37.361 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012139800000000001 2024-02-09T18:06:37.361 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.361 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.361 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,7", 2024-02-09T18:06:37.361 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:31.963558+0000", 2024-02-09T18:06:37.361 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000114962 2024-02-09T18:06:37.361 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.361 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.361 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:37.361 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:31.968668+0000", 2024-02-09T18:06:37.361 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0051103420000000004 2024-02-09T18:06:37.361 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.362 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.362 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:37.362 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:31.979143+0000", 2024-02-09T18:06:37.362 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010475104000000001 2024-02-09T18:06:37.362 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.362 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.362 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:37.362 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:32.196428+0000", 2024-02-09T18:06:37.362 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.21728467000000001 2024-02-09T18:06:37.362 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.362 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.362 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:37.362 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:32.196492+0000", 2024-02-09T18:06:37.362 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3949999999999996e-05 2024-02-09T18:06:37.362 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.362 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.363 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:37.363 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:32.196534+0000", 2024-02-09T18:06:37.363 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2579e-05 2024-02-09T18:06:37.363 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:37.363 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:37.363 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:37.363 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.363 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.363 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:261016 9.5 9:a6cc0f03:::benchmark_data_smithi179_161915_object126134:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e973)", 2024-02-09T18:06:37.363 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:04:31.969340+0000", 2024-02-09T18:06:37.363 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 125.27212851199999, 2024-02-09T18:06:37.363 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.22733603699999999, 2024-02-09T18:06:37.363 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:37.363 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:37.363 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:37.364 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:06:37.364 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:37.364 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 261016 2024-02-09T18:06:37.364 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.364 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:37.364 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.364 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:37.364 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:31.969340+0000", 2024-02-09T18:06:37.364 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:37.364 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.364 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.364 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:37.364 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:31.969340+0000", 2024-02-09T18:06:37.364 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:37.365 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.365 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.365 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:37.365 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:31.969342+0000", 2024-02-09T18:06:37.365 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4449999999999999e-06 2024-02-09T18:06:37.365 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.365 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.365 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:37.365 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:31.969347+0000", 2024-02-09T18:06:37.365 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8770000000000001e-06 2024-02-09T18:06:37.365 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.365 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.365 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:37.365 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:31.969357+0000", 2024-02-09T18:06:37.365 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0577999999999999e-05 2024-02-09T18:06:37.366 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.366 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.366 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:37.366 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:31.969363+0000", 2024-02-09T18:06:37.366 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0950000000000001e-06 2024-02-09T18:06:37.366 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.366 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.366 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:37.366 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:31.969415+0000", 2024-02-09T18:06:37.366 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1227999999999998e-05 2024-02-09T18:06:37.366 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.366 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.366 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:37.366 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:31.969528+0000", 2024-02-09T18:06:37.366 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000112879 2024-02-09T18:06:37.367 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.367 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.367 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,7", 2024-02-09T18:06:37.367 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:31.969606+0000", 2024-02-09T18:06:37.367 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8937000000000007e-05 2024-02-09T18:06:37.367 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.367 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.367 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:37.367 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:31.973716+0000", 2024-02-09T18:06:37.367 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0041097429999999999 2024-02-09T18:06:37.367 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.367 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.367 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:37.367 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:31.979181+0000", 2024-02-09T18:06:37.367 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0054646240000000004 2024-02-09T18:06:37.368 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.368 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.368 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:37.368 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:32.196586+0000", 2024-02-09T18:06:37.368 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.21740524999999999 2024-02-09T18:06:37.368 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.368 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.368 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:37.368 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:32.196639+0000", 2024-02-09T18:06:37.368 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3077e-05 2024-02-09T18:06:37.368 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.368 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.368 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:37.368 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:32.196676+0000", 2024-02-09T18:06:37.368 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7304000000000001e-05 2024-02-09T18:06:37.369 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:37.369 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:37.369 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:37.369 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.369 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.369 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:261026 9.5 9:af1186e3:::benchmark_data_smithi179_161915_object126144:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e973)", 2024-02-09T18:06:37.369 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:04:31.983965+0000", 2024-02-09T18:06:37.369 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 125.257504002, 2024-02-09T18:06:37.369 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.223311168, 2024-02-09T18:06:37.369 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:37.369 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:37.369 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:37.369 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:06:37.369 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:37.369 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 261026 2024-02-09T18:06:37.370 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.370 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:37.370 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.370 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:37.370 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:31.983965+0000", 2024-02-09T18:06:37.370 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:37.370 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.370 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.370 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:37.370 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:31.983965+0000", 2024-02-09T18:06:37.370 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:37.370 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.370 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.370 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:37.370 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:31.983966+0000", 2024-02-09T18:06:37.371 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5260000000000001e-06 2024-02-09T18:06:37.371 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.371 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.371 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:37.371 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:31.983973+0000", 2024-02-09T18:06:37.371 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8569999999999997e-06 2024-02-09T18:06:37.371 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.371 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.371 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:37.371 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:31.983990+0000", 2024-02-09T18:06:37.371 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6280999999999999e-05 2024-02-09T18:06:37.371 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.371 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.371 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:37.371 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:31.983999+0000", 2024-02-09T18:06:37.372 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.1740000000000006e-06 2024-02-09T18:06:37.372 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.372 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.372 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:37.372 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:31.984049+0000", 2024-02-09T18:06:37.372 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9997999999999998e-05 2024-02-09T18:06:37.372 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.372 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.372 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:37.372 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:31.984166+0000", 2024-02-09T18:06:37.372 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011686800000000001 2024-02-09T18:06:37.372 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.372 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.372 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,7", 2024-02-09T18:06:37.372 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:31.984289+0000", 2024-02-09T18:06:37.373 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000123576 2024-02-09T18:06:37.373 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.373 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.373 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:37.373 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:31.988473+0000", 2024-02-09T18:06:37.373 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0041839590000000001 2024-02-09T18:06:37.373 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.373 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.373 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:37.373 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:31.994797+0000", 2024-02-09T18:06:37.373 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0063234019999999997 2024-02-09T18:06:37.373 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.373 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.373 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:37.373 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:32.207171+0000", 2024-02-09T18:06:37.373 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.212374432 2024-02-09T18:06:37.374 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.374 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.374 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:37.374 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:32.207232+0000", 2024-02-09T18:06:37.374 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1288e-05 2024-02-09T18:06:37.374 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:37.374 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:37.374 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:37.374 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:04:32.207276+0000", 2024-02-09T18:06:37.374 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3807000000000003e-05 2024-02-09T18:06:37.374 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:37.374 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:37.374 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:37.374 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:37.374 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:37.375 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T18:06:37.376 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd primary-affinity 11 1 2024-02-09T18:06:37.650 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:06:37.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:37 smithi099 ceph-mon[39420]: 1.5 scrub starts 2024-02-09T18:06:37.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:37 smithi099 ceph-mon[39420]: 1.5 scrub ok 2024-02-09T18:06:37.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:37 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/580068274' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-02-09T18:06:37.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:37 smithi099 ceph-mon[39420]: osdmap e1026: 12 total, 12 up, 12 in 2024-02-09T18:06:37.855 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:37 smithi179 ceph-mon[43975]: 1.5 scrub starts 2024-02-09T18:06:37.855 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:37 smithi179 ceph-mon[43975]: 1.5 scrub ok 2024-02-09T18:06:37.855 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:37 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/580068274' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-02-09T18:06:37.855 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:37 smithi179 ceph-mon[43975]: osdmap e1026: 12 total, 12 up, 12 in 2024-02-09T18:06:37.881 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.8 dump_ops_in_flight 2024-02-09T18:06:37.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:37 smithi162 ceph-mon[44304]: 1.5 scrub starts 2024-02-09T18:06:37.982 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:37 smithi162 ceph-mon[44304]: 1.5 scrub ok 2024-02-09T18:06:37.982 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:37 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/580068274' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-02-09T18:06:37.982 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:37 smithi162 ceph-mon[44304]: osdmap e1026: 12 total, 12 up, 12 in 2024-02-09T18:06:38.142 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T18:06:38.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:38 smithi179 ceph-mon[43975]: pgmap v2522: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:38.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:38 smithi179 ceph-mon[43975]: 1.3 scrub starts 2024-02-09T18:06:38.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:38 smithi179 ceph-mon[43975]: 1.3 scrub ok 2024-02-09T18:06:38.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:38 smithi162 ceph-mon[44304]: pgmap v2522: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:38.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:38 smithi162 ceph-mon[44304]: 1.3 scrub starts 2024-02-09T18:06:38.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:38 smithi162 ceph-mon[44304]: 1.3 scrub ok 2024-02-09T18:06:38.956 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:38 smithi099 ceph-mon[39420]: pgmap v2522: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:38.956 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:38 smithi099 ceph-mon[39420]: 1.3 scrub starts 2024-02-09T18:06:38.956 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:38 smithi099 ceph-mon[39420]: 1.3 scrub ok 2024-02-09T18:06:40.119 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T18:06:40.119 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-02-09T18:06:40.120 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 0 2024-02-09T18:06:40.120 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T18:06:40.576 INFO:teuthology.orchestra.run.smithi099.stderr:set osd.11 primary-affinity to 1 (8655362) 2024-02-09T18:06:40.766 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:40 smithi179 ceph-mon[43975]: pgmap v2523: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:40.767 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:40 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2929467273' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]: dispatch 2024-02-09T18:06:40.796 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.8 dump_blocked_ops 2024-02-09T18:06:40.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:40 smithi162 ceph-mon[44304]: pgmap v2523: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:40.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:40 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2929467273' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]: dispatch 2024-02-09T18:06:40.983 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:40 smithi099 ceph-mon[39420]: pgmap v2523: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:40.983 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:40 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2929467273' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]: dispatch 2024-02-09T18:06:41.056 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T18:06:41.346 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 8 1 2024-02-09T18:06:41.613 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:06:41.836 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:41 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2929467273' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-02-09T18:06:41.836 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:41 smithi099 ceph-mon[39420]: osdmap e1027: 12 total, 12 up, 12 in 2024-02-09T18:06:41.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:41 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2929467273' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-02-09T18:06:41.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:41 smithi179 ceph-mon[43975]: osdmap e1027: 12 total, 12 up, 12 in 2024-02-09T18:06:41.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:41 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2929467273' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-02-09T18:06:41.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:41 smithi162 ceph-mon[44304]: osdmap e1027: 12 total, 12 up, 12 in 2024-02-09T18:06:42.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:42 smithi099 ceph-mon[39420]: pgmap v2525: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:42.814 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T18:06:42.814 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-02-09T18:06:42.814 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-02-09T18:06:42.814 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-02-09T18:06:42.814 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T18:06:42.834 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:42 smithi179 ceph-mon[43975]: pgmap v2525: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:42.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:42 smithi162 ceph-mon[44304]: pgmap v2525: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:43.590 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.8 dump_historic_ops 2024-02-09T18:06:43.851 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.c/config 2024-02-09T18:06:43.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:43 smithi179 ceph-mon[43975]: 1.d scrub starts 2024-02-09T18:06:43.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:43 smithi179 ceph-mon[43975]: 1.d scrub ok 2024-02-09T18:06:43.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:43 smithi162 ceph-mon[44304]: 1.d scrub starts 2024-02-09T18:06:43.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:43 smithi162 ceph-mon[44304]: 1.d scrub ok 2024-02-09T18:06:43.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:43 smithi099 ceph-mon[39420]: 1.d scrub starts 2024-02-09T18:06:43.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:43 smithi099 ceph-mon[39420]: 1.d scrub ok 2024-02-09T18:06:44.589 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.8 to 1 (10000) 2024-02-09T18:06:44.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:44 smithi179 ceph-mon[43975]: 1.1b scrub starts 2024-02-09T18:06:44.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:44 smithi179 ceph-mon[43975]: 1.1b scrub ok 2024-02-09T18:06:44.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:44 smithi179 ceph-mon[43975]: pgmap v2526: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:44.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:44 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3780593899' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-02-09T18:06:44.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:44 smithi162 ceph-mon[44304]: 1.1b scrub starts 2024-02-09T18:06:44.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:44 smithi162 ceph-mon[44304]: 1.1b scrub ok 2024-02-09T18:06:44.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:44 smithi162 ceph-mon[44304]: pgmap v2526: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:44.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:44 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3780593899' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-02-09T18:06:44.981 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:44 smithi099 ceph-mon[39420]: 1.1b scrub starts 2024-02-09T18:06:44.981 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:44 smithi099 ceph-mon[39420]: 1.1b scrub ok 2024-02-09T18:06:44.981 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:44 smithi099 ceph-mon[39420]: pgmap v2526: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:44.981 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:44 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3780593899' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-02-09T18:06:45.403 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd primary-affinity 8 1 2024-02-09T18:06:45.671 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:06:45.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:45 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3780593899' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-02-09T18:06:45.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:45 smithi099 ceph-mon[39420]: osdmap e1028: 12 total, 12 up, 12 in 2024-02-09T18:06:45.852 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-02-09T18:06:45.853 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-02-09T18:06:45.853 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-02-09T18:06:45.853 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-02-09T18:06:45.853 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.853 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:236077 8.b 8:d02b57f2:::benchmark_data_smithi179_148205_object113005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e863)", 2024-02-09T18:06:45.853 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:59:26.606600+0000", 2024-02-09T18:06:45.853 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 439.24183892000002, 2024-02-09T18:06:45.853 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6539503770000001, 2024-02-09T18:06:45.853 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:45.853 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:45.854 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:45.854 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:06:45.854 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:06:45.854 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 236077 2024-02-09T18:06:45.854 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.854 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:45.854 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.854 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:45.854 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.606600+0000", 2024-02-09T18:06:45.854 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:45.854 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.854 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.854 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:45.855 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.606600+0000", 2024-02-09T18:06:45.855 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:45.855 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.855 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.855 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:45.855 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.606602+0000", 2024-02-09T18:06:45.855 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4389999999999999e-06 2024-02-09T18:06:45.855 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.855 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.855 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:45.855 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.606609+0000", 2024-02-09T18:06:45.855 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7139999999999996e-06 2024-02-09T18:06:45.855 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.855 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.856 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:45.856 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.606625+0000", 2024-02-09T18:06:45.856 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6365999999999999e-05 2024-02-09T18:06:45.856 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.856 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.856 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:45.856 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.606634+0000", 2024-02-09T18:06:45.856 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.3370000000000006e-06 2024-02-09T18:06:45.856 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.856 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.856 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:45.856 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.606693+0000", 2024-02-09T18:06:45.856 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9222000000000003e-05 2024-02-09T18:06:45.856 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.857 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.857 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:45.857 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:26.606703+0000", 2024-02-09T18:06:45.857 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.8060000000000006e-06 2024-02-09T18:06:45.857 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.857 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.857 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:45.857 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.006088+0000", 2024-02-09T18:06:45.857 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.399384348 2024-02-09T18:06:45.857 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.859 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.859 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:45.859 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:27.006102+0000", 2024-02-09T18:06:45.859 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4442e-05 2024-02-09T18:06:45.859 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.859 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.859 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:45.859 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.238871+0000", 2024-02-09T18:06:45.859 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2327688619999999 2024-02-09T18:06:45.859 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.859 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.860 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:45.860 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.239029+0000", 2024-02-09T18:06:45.860 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000158049 2024-02-09T18:06:45.860 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.860 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.860 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4,11", 2024-02-09T18:06:45.860 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.239206+0000", 2024-02-09T18:06:45.860 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00017735599999999999 2024-02-09T18:06:45.860 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.860 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.860 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:45.860 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.244222+0000", 2024-02-09T18:06:45.860 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00501565 2024-02-09T18:06:45.861 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.861 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.861 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:45.861 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.244302+0000", 2024-02-09T18:06:45.861 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.0358000000000005e-05 2024-02-09T18:06:45.861 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.861 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.861 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:45.861 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.249476+0000", 2024-02-09T18:06:45.861 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0051733550000000001 2024-02-09T18:06:45.861 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.861 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.861 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:45.861 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.260367+0000", 2024-02-09T18:06:45.861 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010891171999999999 2024-02-09T18:06:45.862 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.862 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.862 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:45.862 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.260433+0000", 2024-02-09T18:06:45.862 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6156999999999997e-05 2024-02-09T18:06:45.862 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.862 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.862 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:45.862 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:28.260551+0000", 2024-02-09T18:06:45.862 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000117744 2024-02-09T18:06:45.862 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:45.862 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:45.862 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:45.863 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.863 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.863 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26305.0:237144 8.b 8:d410fa72:::benchmark_data_smithi179_148205_object114072:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e867)", 2024-02-09T18:06:45.863 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T17:59:30.150092+0000", 2024-02-09T18:06:45.863 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 435.69834756300003, 2024-02-09T18:06:45.863 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.31223010400000001, 2024-02-09T18:06:45.863 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:45.863 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:45.863 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:45.863 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26305", 2024-02-09T18:06:45.863 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/968150614", 2024-02-09T18:06:45.863 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 237144 2024-02-09T18:06:45.863 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.863 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:45.863 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.864 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:45.864 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.150092+0000", 2024-02-09T18:06:45.864 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:45.864 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.864 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.864 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:45.864 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.150092+0000", 2024-02-09T18:06:45.864 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:45.864 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.864 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.864 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:45.864 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.150094+0000", 2024-02-09T18:06:45.864 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9230000000000001e-06 2024-02-09T18:06:45.864 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.865 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.865 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:45.865 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.150102+0000", 2024-02-09T18:06:45.865 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2060000000000007e-06 2024-02-09T18:06:45.865 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.865 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.865 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:45.865 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.150122+0000", 2024-02-09T18:06:45.865 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9930999999999999e-05 2024-02-09T18:06:45.865 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.865 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.865 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:45.865 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.150141+0000", 2024-02-09T18:06:45.865 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9142000000000001e-05 2024-02-09T18:06:45.865 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.866 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.866 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:45.866 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.150214+0000", 2024-02-09T18:06:45.866 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2846e-05 2024-02-09T18:06:45.866 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.866 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.866 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:45.866 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.150364+0000", 2024-02-09T18:06:45.866 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015035600000000001 2024-02-09T18:06:45.866 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.866 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.866 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4,11", 2024-02-09T18:06:45.866 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.151273+0000", 2024-02-09T18:06:45.866 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00090894200000000004 2024-02-09T18:06:45.866 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.867 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.867 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:45.867 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.156327+0000", 2024-02-09T18:06:45.867 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00505376 2024-02-09T18:06:45.867 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.867 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.867 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:45.867 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.156343+0000", 2024-02-09T18:06:45.867 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6002e-05 2024-02-09T18:06:45.867 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.867 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.867 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:45.867 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.162202+0000", 2024-02-09T18:06:45.867 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0058588900000000003 2024-02-09T18:06:45.867 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.868 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.868 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:45.868 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.462198+0000", 2024-02-09T18:06:45.868 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.29999659200000001 2024-02-09T18:06:45.868 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.868 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.868 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:45.868 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.462246+0000", 2024-02-09T18:06:45.869 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8016999999999998e-05 2024-02-09T18:06:45.869 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.869 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.869 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:45.869 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T17:59:30.462322+0000", 2024-02-09T18:06:45.869 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5496999999999999e-05 2024-02-09T18:06:45.869 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:45.869 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:45.869 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:45.869 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.869 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.869 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:202997 9.b 9:dc8fd382:::benchmark_data_smithi179_161915_object68115:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e942)", 2024-02-09T18:06:45.869 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:13.445860+0000", 2024-02-09T18:06:45.870 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 212.40257888400001, 2024-02-09T18:06:45.870 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.223805365, 2024-02-09T18:06:45.870 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:45.870 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:45.870 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:45.870 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:06:45.870 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:45.871 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 202997 2024-02-09T18:06:45.871 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.871 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:45.871 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.871 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:45.871 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.445860+0000", 2024-02-09T18:06:45.871 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:45.871 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.871 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.871 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:45.871 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.445860+0000", 2024-02-09T18:06:45.872 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:45.872 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.872 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.872 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:45.872 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.445862+0000", 2024-02-09T18:06:45.872 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0499999999999999e-06 2024-02-09T18:06:45.872 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.872 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.872 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:45.872 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.445869+0000", 2024-02-09T18:06:45.872 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.866e-06 2024-02-09T18:06:45.872 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.872 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.872 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:45.873 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.445882+0000", 2024-02-09T18:06:45.873 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2989999999999999e-05 2024-02-09T18:06:45.873 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.873 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.873 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:45.873 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.445891+0000", 2024-02-09T18:06:45.873 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.1179999999999998e-06 2024-02-09T18:06:45.873 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.873 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.873 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:45.873 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.445947+0000", 2024-02-09T18:06:45.873 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5736999999999998e-05 2024-02-09T18:06:45.873 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.873 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.874 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op must wait for map", 2024-02-09T18:06:45.874 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.445952+0000", 2024-02-09T18:06:45.874 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3510000000000004e-06 2024-02-09T18:06:45.874 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.874 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.874 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:45.874 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.453790+0000", 2024-02-09T18:06:45.874 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0078388480000000007 2024-02-09T18:06:45.874 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.874 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.874 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:45.875 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.453800+0000", 2024-02-09T18:06:45.875 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5000000000000005e-06 2024-02-09T18:06:45.875 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.875 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.875 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:45.875 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:14.441943+0000", 2024-02-09T18:06:45.875 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.98814301900000001 2024-02-09T18:06:45.875 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.875 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.875 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:45.875 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:14.441959+0000", 2024-02-09T18:06:45.875 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6325999999999999e-05 2024-02-09T18:06:45.875 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.875 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.875 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:45.876 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.652434+0000", 2024-02-09T18:06:45.876 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.210474933 2024-02-09T18:06:45.876 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.876 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.876 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:45.876 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.652554+0000", 2024-02-09T18:06:45.876 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000120025 2024-02-09T18:06:45.876 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.876 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.876 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5,10", 2024-02-09T18:06:45.876 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.652642+0000", 2024-02-09T18:06:45.876 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7954999999999994e-05 2024-02-09T18:06:45.876 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.876 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.877 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:45.877 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.658212+0000", 2024-02-09T18:06:45.877 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0055703460000000003 2024-02-09T18:06:45.877 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.877 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.877 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:45.877 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.663862+0000", 2024-02-09T18:06:45.877 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0056495490000000002 2024-02-09T18:06:45.877 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.877 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.877 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:45.877 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.669378+0000", 2024-02-09T18:06:45.877 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0055155359999999997 2024-02-09T18:06:45.877 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.877 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.878 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:45.878 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.669475+0000", 2024-02-09T18:06:45.878 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7275000000000006e-05 2024-02-09T18:06:45.878 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.878 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.879 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:45.879 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.669544+0000", 2024-02-09T18:06:45.879 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8911000000000005e-05 2024-02-09T18:06:45.879 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.879 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.879 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:45.879 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.669666+0000", 2024-02-09T18:06:45.879 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012203 2024-02-09T18:06:45.879 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:45.880 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:45.880 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:45.880 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.880 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.880 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:203024 9.b 9:d448a439:::benchmark_data_smithi179_161915_object68142:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e942)", 2024-02-09T18:06:45.880 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:13.495413+0000", 2024-02-09T18:06:45.880 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 212.35302591000001, 2024-02-09T18:06:45.880 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1746194750000001, 2024-02-09T18:06:45.880 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:45.880 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:45.880 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:45.880 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:06:45.880 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:45.880 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 203024 2024-02-09T18:06:45.881 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.881 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:45.881 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.881 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:45.881 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.495413+0000", 2024-02-09T18:06:45.881 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:45.881 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.881 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.881 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:45.881 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.495413+0000", 2024-02-09T18:06:45.881 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:45.881 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.881 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.881 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:45.881 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.495415+0000", 2024-02-09T18:06:45.882 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.333e-06 2024-02-09T18:06:45.882 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.882 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.882 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:45.882 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.495421+0000", 2024-02-09T18:06:45.882 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8760000000000002e-06 2024-02-09T18:06:45.882 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.882 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.882 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:45.882 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.495434+0000", 2024-02-09T18:06:45.882 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3061000000000001e-05 2024-02-09T18:06:45.882 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.882 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.882 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:45.882 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.495441+0000", 2024-02-09T18:06:45.883 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.0639999999999996e-06 2024-02-09T18:06:45.883 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.883 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.883 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:45.883 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.495496+0000", 2024-02-09T18:06:45.883 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4954000000000003e-05 2024-02-09T18:06:45.883 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.883 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.883 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:45.883 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.495505+0000", 2024-02-09T18:06:45.883 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.4690000000000003e-06 2024-02-09T18:06:45.883 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.883 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.883 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:45.884 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:14.442055+0000", 2024-02-09T18:06:45.884 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.94654996499999999 2024-02-09T18:06:45.884 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.884 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.884 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:45.884 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:14.442064+0000", 2024-02-09T18:06:45.884 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.3219999999999999e-06 2024-02-09T18:06:45.884 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.884 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.884 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:45.884 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.658266+0000", 2024-02-09T18:06:45.884 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2162012870000001 2024-02-09T18:06:45.884 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.884 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.885 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:45.885 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.658392+0000", 2024-02-09T18:06:45.885 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012630900000000001 2024-02-09T18:06:45.885 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.885 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.885 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5,10", 2024-02-09T18:06:45.885 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.658528+0000", 2024-02-09T18:06:45.885 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013552200000000001 2024-02-09T18:06:45.885 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.885 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.885 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:45.885 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.669681+0000", 2024-02-09T18:06:45.885 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011153758999999999 2024-02-09T18:06:45.885 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.886 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.886 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:45.886 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.669737+0000", 2024-02-09T18:06:45.886 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6150000000000003e-05 2024-02-09T18:06:45.886 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.886 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.886 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:45.886 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.669796+0000", 2024-02-09T18:06:45.886 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8425999999999999e-05 2024-02-09T18:06:45.886 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.886 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.886 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:45.886 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.669860+0000", 2024-02-09T18:06:45.886 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3965000000000002e-05 2024-02-09T18:06:45.886 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.887 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.887 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:45.887 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.669918+0000", 2024-02-09T18:06:45.887 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8461000000000002e-05 2024-02-09T18:06:45.887 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.887 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.887 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:45.887 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.670033+0000", 2024-02-09T18:06:45.887 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000114552 2024-02-09T18:06:45.887 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:45.887 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:45.887 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:45.887 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.887 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.887 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:203038 9.b 9:d56ac938:::benchmark_data_smithi179_161915_object68156:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e942)", 2024-02-09T18:06:45.888 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:13.535999+0000", 2024-02-09T18:06:45.888 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 212.312439888, 2024-02-09T18:06:45.888 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1349075289999999, 2024-02-09T18:06:45.888 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:45.888 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:45.888 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:45.888 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:06:45.888 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:45.888 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 203038 2024-02-09T18:06:45.888 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.888 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:45.888 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.888 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:45.888 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.535999+0000", 2024-02-09T18:06:45.889 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:45.889 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.889 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.889 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:45.889 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.535999+0000", 2024-02-09T18:06:45.889 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:45.889 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.889 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.889 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:45.889 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.536001+0000", 2024-02-09T18:06:45.889 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5659999999999999e-06 2024-02-09T18:06:45.889 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.889 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.890 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:45.890 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.536007+0000", 2024-02-09T18:06:45.890 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2009999999999996e-06 2024-02-09T18:06:45.890 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.890 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.890 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:45.890 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.536025+0000", 2024-02-09T18:06:45.890 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7379000000000001e-05 2024-02-09T18:06:45.890 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.890 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.890 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:45.890 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.536035+0000", 2024-02-09T18:06:45.890 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0114e-05 2024-02-09T18:06:45.890 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.890 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.890 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:45.891 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.536101+0000", 2024-02-09T18:06:45.891 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5939999999999995e-05 2024-02-09T18:06:45.891 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.891 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.891 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:45.891 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:13.536110+0000", 2024-02-09T18:06:45.891 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.1649999999999995e-06 2024-02-09T18:06:45.891 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.891 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.891 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:45.891 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:14.442112+0000", 2024-02-09T18:06:45.891 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.90600196700000002 2024-02-09T18:06:45.891 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.891 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.891 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:45.892 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:14.442121+0000", 2024-02-09T18:06:45.892 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.9530000000000005e-06 2024-02-09T18:06:45.892 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.892 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.892 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:45.892 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.663892+0000", 2024-02-09T18:06:45.892 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.22177107 2024-02-09T18:06:45.892 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.892 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.892 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:45.892 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.664014+0000", 2024-02-09T18:06:45.892 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012196700000000001 2024-02-09T18:06:45.892 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.892 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.893 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5,10", 2024-02-09T18:06:45.893 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.664152+0000", 2024-02-09T18:06:45.893 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013860899999999999 2024-02-09T18:06:45.893 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.893 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.893 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:45.893 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.669813+0000", 2024-02-09T18:06:45.893 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0056608930000000002 2024-02-09T18:06:45.893 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.893 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.893 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:45.893 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.670079+0000", 2024-02-09T18:06:45.893 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00026547300000000001 2024-02-09T18:06:45.893 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.893 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.894 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:45.894 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.670116+0000", 2024-02-09T18:06:45.894 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7001e-05 2024-02-09T18:06:45.894 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.894 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.894 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:45.894 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.670746+0000", 2024-02-09T18:06:45.894 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00063032400000000003 2024-02-09T18:06:45.894 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.894 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.894 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:45.894 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.670801+0000", 2024-02-09T18:06:45.894 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4648999999999998e-05 2024-02-09T18:06:45.894 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.894 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.895 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:45.895 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:15.670907+0000", 2024-02-09T18:06:45.895 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000106258 2024-02-09T18:06:45.895 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:45.895 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:45.895 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:45.895 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.895 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.895 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:204359 9.b 9:d380bded:::benchmark_data_smithi179_161915_object69477:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e946)", 2024-02-09T18:06:45.895 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:17.542455+0000", 2024-02-09T18:06:45.895 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 208.30598409199999, 2024-02-09T18:06:45.895 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.467034742, 2024-02-09T18:06:45.895 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:45.895 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:45.896 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:45.896 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:06:45.896 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:45.896 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 204359 2024-02-09T18:06:45.896 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.896 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:45.896 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.896 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:45.896 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.542455+0000", 2024-02-09T18:06:45.896 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:45.896 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.896 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.896 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:45.896 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.542455+0000", 2024-02-09T18:06:45.897 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:45.897 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.897 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.897 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:45.897 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.542456+0000", 2024-02-09T18:06:45.897 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2359999999999999e-06 2024-02-09T18:06:45.897 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.897 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.897 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:45.897 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.542461+0000", 2024-02-09T18:06:45.897 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7350000000000004e-06 2024-02-09T18:06:45.897 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.897 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.897 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:45.897 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.542474+0000", 2024-02-09T18:06:45.898 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2418999999999999e-05 2024-02-09T18:06:45.898 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.898 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.898 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:45.898 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.542481+0000", 2024-02-09T18:06:45.898 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7919999999999993e-06 2024-02-09T18:06:45.898 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.898 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.898 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:45.899 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.542530+0000", 2024-02-09T18:06:45.899 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9060999999999997e-05 2024-02-09T18:06:45.899 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.899 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.899 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:45.899 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.542662+0000", 2024-02-09T18:06:45.899 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013146699999999999 2024-02-09T18:06:45.899 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.899 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.899 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5,10", 2024-02-09T18:06:45.899 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.542797+0000", 2024-02-09T18:06:45.899 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000135043 2024-02-09T18:06:45.899 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.899 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.900 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:45.900 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.548368+0000", 2024-02-09T18:06:45.900 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.005571276 2024-02-09T18:06:45.900 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.900 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.900 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:45.900 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.548412+0000", 2024-02-09T18:06:45.900 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3412999999999999e-05 2024-02-09T18:06:45.900 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.900 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.900 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:45.900 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.554027+0000", 2024-02-09T18:06:45.900 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0056148750000000001 2024-02-09T18:06:45.900 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.901 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.901 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:45.901 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:18.009355+0000", 2024-02-09T18:06:45.901 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.45532827799999998 2024-02-09T18:06:45.901 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.901 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.901 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:45.901 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:18.009398+0000", 2024-02-09T18:06:45.901 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3517999999999999e-05 2024-02-09T18:06:45.901 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.901 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.901 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:45.901 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:18.009490+0000", 2024-02-09T18:06:45.901 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.1629e-05 2024-02-09T18:06:45.901 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:45.902 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:45.902 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:45.902 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.902 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.902 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:204365 9.b 9:d4e000a7:::benchmark_data_smithi179_161915_object69483:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e946)", 2024-02-09T18:06:45.902 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:17.548483+0000", 2024-02-09T18:06:45.902 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 208.299956522, 2024-02-09T18:06:45.902 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.46123535100000002, 2024-02-09T18:06:45.902 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:45.902 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:45.902 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:45.902 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:06:45.902 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:45.902 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 204365 2024-02-09T18:06:45.903 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.903 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:45.903 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.903 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:45.903 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.548483+0000", 2024-02-09T18:06:45.903 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:45.903 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.903 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.903 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:45.903 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.548483+0000", 2024-02-09T18:06:45.903 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:45.903 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.903 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.903 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:45.903 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.548484+0000", 2024-02-09T18:06:45.904 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4810000000000001e-06 2024-02-09T18:06:45.904 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.904 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.904 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:45.904 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.548490+0000", 2024-02-09T18:06:45.904 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7880000000000003e-06 2024-02-09T18:06:45.904 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.904 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.904 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:45.904 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.548500+0000", 2024-02-09T18:06:45.904 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.8730000000000002e-06 2024-02-09T18:06:45.904 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.905 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.905 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:45.905 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.548507+0000", 2024-02-09T18:06:45.905 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7390000000000002e-06 2024-02-09T18:06:45.905 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.905 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.905 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:45.905 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.548866+0000", 2024-02-09T18:06:45.905 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00035978099999999999 2024-02-09T18:06:45.905 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.905 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.905 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:45.905 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.548994+0000", 2024-02-09T18:06:45.905 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000127849 2024-02-09T18:06:45.905 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.905 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.906 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5,10", 2024-02-09T18:06:45.906 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.549124+0000", 2024-02-09T18:06:45.906 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000129579 2024-02-09T18:06:45.906 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.906 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.906 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:45.906 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.556168+0000", 2024-02-09T18:06:45.906 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0070443579999999997 2024-02-09T18:06:45.906 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.906 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.906 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:45.906 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.559407+0000", 2024-02-09T18:06:45.906 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0032383350000000002 2024-02-09T18:06:45.907 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.907 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.907 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:45.907 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.566273+0000", 2024-02-09T18:06:45.907 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0068665050000000002 2024-02-09T18:06:45.907 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.907 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.907 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:45.907 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:18.009548+0000", 2024-02-09T18:06:45.907 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.44327513499999999 2024-02-09T18:06:45.907 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.907 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.907 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:45.907 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:18.009590+0000", 2024-02-09T18:06:45.908 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1805999999999999e-05 2024-02-09T18:06:45.908 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.908 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.908 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:45.908 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:18.009718+0000", 2024-02-09T18:06:45.908 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012812199999999999 2024-02-09T18:06:45.908 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:45.908 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:45.908 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:45.908 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.908 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.908 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:204373 9.b 9:d1b4954d:::benchmark_data_smithi179_161915_object69491:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e946)", 2024-02-09T18:06:45.908 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:17.570467+0000", 2024-02-09T18:06:45.908 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 208.27797240800001, 2024-02-09T18:06:45.909 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.43951240600000002, 2024-02-09T18:06:45.909 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:45.909 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:45.909 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:45.909 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:06:45.909 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:45.909 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 204373 2024-02-09T18:06:45.909 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.909 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:45.909 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.909 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:45.909 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.570467+0000", 2024-02-09T18:06:45.909 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:45.909 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.910 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.910 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:45.910 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.570467+0000", 2024-02-09T18:06:45.910 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:45.910 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.910 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.910 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:45.910 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.570468+0000", 2024-02-09T18:06:45.910 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1379999999999999e-06 2024-02-09T18:06:45.910 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.910 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.910 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:45.910 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.570473+0000", 2024-02-09T18:06:45.910 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8369999999999996e-06 2024-02-09T18:06:45.911 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.911 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.911 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:45.911 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.570484+0000", 2024-02-09T18:06:45.911 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1558e-05 2024-02-09T18:06:45.911 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.911 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.911 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:45.911 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.570490+0000", 2024-02-09T18:06:45.911 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.818e-06 2024-02-09T18:06:45.911 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.911 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.911 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:45.911 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.570539+0000", 2024-02-09T18:06:45.912 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8300999999999997e-05 2024-02-09T18:06:45.912 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.912 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.912 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:45.912 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.570680+0000", 2024-02-09T18:06:45.912 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014110500000000001 2024-02-09T18:06:45.912 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.912 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.912 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5,10", 2024-02-09T18:06:45.912 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.570832+0000", 2024-02-09T18:06:45.912 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015264099999999999 2024-02-09T18:06:45.912 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.912 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.912 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:45.912 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.576150+0000", 2024-02-09T18:06:45.913 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0053176559999999996 2024-02-09T18:06:45.913 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.913 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.913 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:45.913 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.576235+0000", 2024-02-09T18:06:45.913 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4844000000000003e-05 2024-02-09T18:06:45.913 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.913 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.913 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:45.913 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:17.576449+0000", 2024-02-09T18:06:45.913 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00021409900000000001 2024-02-09T18:06:45.913 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.913 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.913 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:45.914 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:18.009782+0000", 2024-02-09T18:06:45.914 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.43333312699999998 2024-02-09T18:06:45.914 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.914 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.914 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:45.914 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:18.009845+0000", 2024-02-09T18:06:45.914 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2781999999999996e-05 2024-02-09T18:06:45.914 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.914 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.914 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:45.914 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:18.009979+0000", 2024-02-09T18:06:45.914 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013449999999999999 2024-02-09T18:06:45.914 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:45.914 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:45.915 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:45.915 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.915 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.915 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:218845 9.b 9:d7630bba:::benchmark_data_smithi179_161915_object83963:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e955)", 2024-02-09T18:06:45.915 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:39.728479+0000", 2024-02-09T18:06:45.915 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 186.11996019, 2024-02-09T18:06:45.915 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0303738099999999, 2024-02-09T18:06:45.915 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:45.915 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:45.915 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:45.915 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:06:45.915 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:45.915 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 218845 2024-02-09T18:06:45.915 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.916 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:45.916 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.916 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:45.916 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.728479+0000", 2024-02-09T18:06:45.916 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:45.916 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.916 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.916 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:45.916 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.728479+0000", 2024-02-09T18:06:45.916 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:45.916 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.916 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.916 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:45.916 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.728481+0000", 2024-02-09T18:06:45.916 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3259999999999998e-06 2024-02-09T18:06:45.917 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.917 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.917 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:45.917 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.728492+0000", 2024-02-09T18:06:45.917 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0499000000000001e-05 2024-02-09T18:06:45.917 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.917 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.917 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:45.917 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.728513+0000", 2024-02-09T18:06:45.917 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1019999999999999e-05 2024-02-09T18:06:45.917 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.917 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.917 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:45.918 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.728526+0000", 2024-02-09T18:06:45.918 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2808e-05 2024-02-09T18:06:45.918 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.918 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.918 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:45.918 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.728640+0000", 2024-02-09T18:06:45.918 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011443299999999999 2024-02-09T18:06:45.918 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.918 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.918 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:45.918 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.728652+0000", 2024-02-09T18:06:45.918 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1966e-05 2024-02-09T18:06:45.918 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.918 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.919 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:45.919 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.744074+0000", 2024-02-09T18:06:45.919 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.015421535 2024-02-09T18:06:45.919 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.919 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.919 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:45.919 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.744231+0000", 2024-02-09T18:06:45.919 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015734700000000001 2024-02-09T18:06:45.919 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.919 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.919 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5", 2024-02-09T18:06:45.919 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.744312+0000", 2024-02-09T18:06:45.919 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1019000000000005e-05 2024-02-09T18:06:45.919 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.919 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.920 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:45.920 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.748870+0000", 2024-02-09T18:06:45.920 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0045577200000000003 2024-02-09T18:06:45.920 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.920 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.920 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:45.920 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.753192+0000", 2024-02-09T18:06:45.920 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0043224830000000002 2024-02-09T18:06:45.920 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.920 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.920 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:45.920 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.758753+0000", 2024-02-09T18:06:45.920 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0055606500000000003 2024-02-09T18:06:45.920 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.921 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.921 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:45.921 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.758809+0000", 2024-02-09T18:06:45.921 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6573999999999997e-05 2024-02-09T18:06:45.921 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.921 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.921 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:45.921 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.758853+0000", 2024-02-09T18:06:45.921 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3430000000000003e-05 2024-02-09T18:06:45.921 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:45.921 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:45.921 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:45.921 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.921 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.921 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:218848 9.b 9:dcfc0c50:::benchmark_data_smithi179_161915_object83966:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e955)", 2024-02-09T18:06:45.922 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:39.728563+0000", 2024-02-09T18:06:45.922 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 186.11987634600001, 2024-02-09T18:06:45.922 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.03056269, 2024-02-09T18:06:45.922 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:45.922 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:45.922 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:45.922 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:06:45.922 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:45.922 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 218848 2024-02-09T18:06:45.922 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.922 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:45.922 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.922 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:45.922 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.728563+0000", 2024-02-09T18:06:45.922 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:45.923 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.923 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.923 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:45.923 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.728563+0000", 2024-02-09T18:06:45.923 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:45.923 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.923 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.923 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:45.923 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.728563+0000", 2024-02-09T18:06:45.923 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0100000000000005e-07 2024-02-09T18:06:45.923 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.923 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.923 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:45.923 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.728567+0000", 2024-02-09T18:06:45.923 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.174e-06 2024-02-09T18:06:45.924 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.924 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.924 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:45.924 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.728574+0000", 2024-02-09T18:06:45.924 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5959999999999998e-06 2024-02-09T18:06:45.924 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.924 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.924 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:45.924 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.728580+0000", 2024-02-09T18:06:45.924 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1419999999999998e-06 2024-02-09T18:06:45.924 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.924 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.924 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:45.924 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.728680+0000", 2024-02-09T18:06:45.925 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010011800000000001 2024-02-09T18:06:45.925 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.925 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.925 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:45.925 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.728689+0000", 2024-02-09T18:06:45.925 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5890000000000003e-06 2024-02-09T18:06:45.925 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.925 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.925 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:45.925 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.748905+0000", 2024-02-09T18:06:45.925 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.02021641 2024-02-09T18:06:45.925 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.925 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.925 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:45.925 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.748987+0000", 2024-02-09T18:06:45.926 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1627999999999998e-05 2024-02-09T18:06:45.926 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.926 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.926 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5", 2024-02-09T18:06:45.926 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.749061+0000", 2024-02-09T18:06:45.926 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.3549999999999999e-05 2024-02-09T18:06:45.926 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.926 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.926 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:45.926 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.759006+0000", 2024-02-09T18:06:45.926 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0099452959999999993 2024-02-09T18:06:45.926 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.926 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.926 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:45.927 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.759020+0000", 2024-02-09T18:06:45.927 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3767e-05 2024-02-09T18:06:45.927 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.927 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.927 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:45.927 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.759069+0000", 2024-02-09T18:06:45.927 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9521999999999998e-05 2024-02-09T18:06:45.927 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.927 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.927 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:45.927 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.759102+0000", 2024-02-09T18:06:45.927 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2608999999999999e-05 2024-02-09T18:06:45.927 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.927 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.927 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:45.928 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.759126+0000", 2024-02-09T18:06:45.928 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3788e-05 2024-02-09T18:06:45.928 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:45.928 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:45.928 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:45.928 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.928 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.928 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:218853 9.b 9:d58a0ea1:::benchmark_data_smithi179_161915_object83971:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e955)", 2024-02-09T18:06:45.928 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:39.734016+0000", 2024-02-09T18:06:45.928 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 186.114423585, 2024-02-09T18:06:45.928 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0403554850000001, 2024-02-09T18:06:45.928 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:45.928 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:45.928 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:45.928 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:06:45.929 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:45.929 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 218853 2024-02-09T18:06:45.929 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.929 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:45.929 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.929 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:45.929 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.734016+0000", 2024-02-09T18:06:45.929 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:45.929 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.929 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.929 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:45.929 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.734016+0000", 2024-02-09T18:06:45.929 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:45.929 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.930 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.930 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:45.930 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.734017+0000", 2024-02-09T18:06:45.930 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7069999999999999e-06 2024-02-09T18:06:45.930 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.930 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.930 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:45.930 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.734024+0000", 2024-02-09T18:06:45.930 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6329999999999999e-06 2024-02-09T18:06:45.930 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.930 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.930 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:45.930 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.734039+0000", 2024-02-09T18:06:45.930 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4496e-05 2024-02-09T18:06:45.931 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.931 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.931 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:45.931 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.734047+0000", 2024-02-09T18:06:45.931 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8310000000000007e-06 2024-02-09T18:06:45.931 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.931 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.931 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:45.931 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.734104+0000", 2024-02-09T18:06:45.931 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6239000000000001e-05 2024-02-09T18:06:45.931 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.931 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.931 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:45.931 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.734111+0000", 2024-02-09T18:06:45.931 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7549999999999994e-06 2024-02-09T18:06:45.932 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.932 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.932 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:45.932 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.753228+0000", 2024-02-09T18:06:45.932 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.019117042 2024-02-09T18:06:45.932 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.932 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.932 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:45.932 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.753314+0000", 2024-02-09T18:06:45.932 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5922999999999997e-05 2024-02-09T18:06:45.932 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.932 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.932 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5", 2024-02-09T18:06:45.932 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.753430+0000", 2024-02-09T18:06:45.933 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011583599999999999 2024-02-09T18:06:45.933 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.933 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.933 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:45.933 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.759027+0000", 2024-02-09T18:06:45.933 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0055966399999999999 2024-02-09T18:06:45.933 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.933 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.933 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:45.933 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.759189+0000", 2024-02-09T18:06:45.933 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000162194 2024-02-09T18:06:45.933 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.933 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.933 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:45.933 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.774272+0000", 2024-02-09T18:06:45.934 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.015083088999999999 2024-02-09T18:06:45.934 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.934 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.934 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:45.934 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.774328+0000", 2024-02-09T18:06:45.934 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6230000000000002e-05 2024-02-09T18:06:45.934 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.934 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.934 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:45.934 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.774371+0000", 2024-02-09T18:06:45.934 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2870000000000001e-05 2024-02-09T18:06:45.934 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:45.934 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:45.934 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:45.935 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.935 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.935 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:218911 9.b 9:d44071a9:::benchmark_data_smithi179_161915_object84029:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e955)", 2024-02-09T18:06:45.935 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:39.804466+0000", 2024-02-09T18:06:45.935 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 186.04397344, 2024-02-09T18:06:45.935 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.97004875499999998, 2024-02-09T18:06:45.935 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:45.935 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:45.935 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:45.935 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:06:45.935 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:45.935 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 218911 2024-02-09T18:06:45.935 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.935 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:45.936 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.936 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:45.936 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.804466+0000", 2024-02-09T18:06:45.936 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:45.936 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.936 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.936 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:45.936 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.804466+0000", 2024-02-09T18:06:45.936 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:45.936 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.936 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.936 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:45.936 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.804468+0000", 2024-02-09T18:06:45.936 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6789999999999999e-06 2024-02-09T18:06:45.937 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.937 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.937 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:45.937 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.804475+0000", 2024-02-09T18:06:45.937 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2629999999999997e-06 2024-02-09T18:06:45.937 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.937 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.937 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:45.937 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.804492+0000", 2024-02-09T18:06:45.937 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7235e-05 2024-02-09T18:06:45.937 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.937 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.937 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:45.937 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.804502+0000", 2024-02-09T18:06:45.937 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0315e-05 2024-02-09T18:06:45.938 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.938 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.938 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:45.938 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.804575+0000", 2024-02-09T18:06:45.938 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2276999999999999e-05 2024-02-09T18:06:45.938 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.938 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.938 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:45.938 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.804586+0000", 2024-02-09T18:06:45.938 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1156e-05 2024-02-09T18:06:45.938 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.938 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.938 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:45.938 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.759242+0000", 2024-02-09T18:06:45.939 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.95465582900000001 2024-02-09T18:06:45.939 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.939 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.939 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:45.939 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.759352+0000", 2024-02-09T18:06:45.939 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000110634 2024-02-09T18:06:45.939 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.939 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.939 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5", 2024-02-09T18:06:45.939 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.759463+0000", 2024-02-09T18:06:45.939 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000111072 2024-02-09T18:06:45.939 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.939 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.939 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:45.939 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.774195+0000", 2024-02-09T18:06:45.940 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014731259 2024-02-09T18:06:45.940 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.940 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.940 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:45.940 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.774226+0000", 2024-02-09T18:06:45.940 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1853999999999997e-05 2024-02-09T18:06:45.940 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.940 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.940 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:45.940 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.774428+0000", 2024-02-09T18:06:45.940 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00020128400000000001 2024-02-09T18:06:45.940 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.940 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.940 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:45.941 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.774479+0000", 2024-02-09T18:06:45.941 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1266e-05 2024-02-09T18:06:45.941 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.941 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.941 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:45.941 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.774515+0000", 2024-02-09T18:06:45.941 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.5632e-05 2024-02-09T18:06:45.941 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:45.941 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:45.941 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:45.941 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.941 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.941 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:218913 9.b 9:d0ead2ca:::benchmark_data_smithi179_161915_object84031:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e955)", 2024-02-09T18:06:45.941 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:39.807005+0000", 2024-02-09T18:06:45.941 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 186.041434334, 2024-02-09T18:06:45.941 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.96770659599999997, 2024-02-09T18:06:45.942 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:45.942 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:45.942 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:45.942 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:06:45.942 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:45.942 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 218913 2024-02-09T18:06:45.942 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.942 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:45.942 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.942 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:45.942 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.807005+0000", 2024-02-09T18:06:45.942 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:45.942 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.942 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.942 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:45.943 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.807005+0000", 2024-02-09T18:06:45.943 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:45.943 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.943 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.943 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:45.943 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.807006+0000", 2024-02-09T18:06:45.943 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.8200000000000008e-07 2024-02-09T18:06:45.943 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.943 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.943 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:45.943 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.807011+0000", 2024-02-09T18:06:45.943 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9679999999999997e-06 2024-02-09T18:06:45.943 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.943 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.944 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:45.944 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.807023+0000", 2024-02-09T18:06:45.944 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2104e-05 2024-02-09T18:06:45.944 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.944 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.944 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:45.944 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.807030+0000", 2024-02-09T18:06:45.944 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4499999999999998e-06 2024-02-09T18:06:45.944 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.944 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.944 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:45.944 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.807083+0000", 2024-02-09T18:06:45.944 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2369e-05 2024-02-09T18:06:45.944 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.944 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.945 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:45.945 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.807090+0000", 2024-02-09T18:06:45.945 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.0859999999999996e-06 2024-02-09T18:06:45.945 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.945 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.945 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:45.945 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.764674+0000", 2024-02-09T18:06:45.945 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.95758389700000002 2024-02-09T18:06:45.945 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.945 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.945 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:45.945 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.764787+0000", 2024-02-09T18:06:45.945 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000113126 2024-02-09T18:06:45.945 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.945 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.945 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5", 2024-02-09T18:06:45.946 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.764903+0000", 2024-02-09T18:06:45.946 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000116243 2024-02-09T18:06:45.946 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.946 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.946 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:45.946 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.774233+0000", 2024-02-09T18:06:45.946 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0093295749999999997 2024-02-09T18:06:45.946 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.946 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.946 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:45.946 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.774573+0000", 2024-02-09T18:06:45.946 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00034025700000000001 2024-02-09T18:06:45.946 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.946 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.947 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:45.947 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.774627+0000", 2024-02-09T18:06:45.947 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4042000000000002e-05 2024-02-09T18:06:45.947 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.947 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.947 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:45.947 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.774677+0000", 2024-02-09T18:06:45.947 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0052000000000001e-05 2024-02-09T18:06:45.947 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.947 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.947 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:45.947 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.774712+0000", 2024-02-09T18:06:45.947 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4445e-05 2024-02-09T18:06:45.947 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:45.948 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:45.948 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:45.948 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.948 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.948 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:218917 9.b 9:da2ad27a:::benchmark_data_smithi179_161915_object84035:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e955)", 2024-02-09T18:06:45.948 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:39.814791+0000", 2024-02-09T18:06:45.948 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 186.03364783800001, 2024-02-09T18:06:45.948 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.97067265400000002, 2024-02-09T18:06:45.948 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:45.948 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:45.948 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:45.948 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:06:45.948 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:45.948 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 218917 2024-02-09T18:06:45.949 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.949 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:45.949 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.949 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:45.949 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.814791+0000", 2024-02-09T18:06:45.949 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:45.949 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.949 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.949 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:45.949 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.814791+0000", 2024-02-09T18:06:45.949 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:45.949 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.949 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.949 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:45.949 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.814793+0000", 2024-02-09T18:06:45.950 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7549999999999999e-06 2024-02-09T18:06:45.950 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.950 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.950 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:45.950 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.814800+0000", 2024-02-09T18:06:45.950 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2069999999999998e-06 2024-02-09T18:06:45.950 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.950 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.950 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:45.950 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.814827+0000", 2024-02-09T18:06:45.950 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6673999999999999e-05 2024-02-09T18:06:45.950 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.950 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.950 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:45.950 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.814836+0000", 2024-02-09T18:06:45.951 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.3940000000000002e-06 2024-02-09T18:06:45.951 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.951 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.951 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:45.951 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.814902+0000", 2024-02-09T18:06:45.951 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5582999999999999e-05 2024-02-09T18:06:45.951 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.951 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.951 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:45.951 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.814912+0000", 2024-02-09T18:06:45.951 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0122999999999999e-05 2024-02-09T18:06:45.951 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.951 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.951 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:45.951 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.769797+0000", 2024-02-09T18:06:45.951 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.95488499900000001 2024-02-09T18:06:45.952 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.952 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.952 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:45.952 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.769886+0000", 2024-02-09T18:06:45.952 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8718e-05 2024-02-09T18:06:45.952 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.952 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.952 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5", 2024-02-09T18:06:45.952 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.769994+0000", 2024-02-09T18:06:45.952 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000108558 2024-02-09T18:06:45.952 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.952 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.952 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:45.952 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.774762+0000", 2024-02-09T18:06:45.953 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.004767327 2024-02-09T18:06:45.953 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.953 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.953 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:45.953 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.774813+0000", 2024-02-09T18:06:45.953 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0794999999999997e-05 2024-02-09T18:06:45.953 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.953 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.953 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:45.953 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.785363+0000", 2024-02-09T18:06:45.953 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010550878 2024-02-09T18:06:45.953 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.953 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.953 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:45.953 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.785419+0000", 2024-02-09T18:06:45.954 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5288999999999999e-05 2024-02-09T18:06:45.954 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.954 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.954 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:45.954 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.785464+0000", 2024-02-09T18:06:45.954 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5354e-05 2024-02-09T18:06:45.954 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:45.954 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:45.954 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:45.954 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.954 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.954 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:218926 9.b 9:d6680828:::benchmark_data_smithi179_161915_object84044:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e955)", 2024-02-09T18:06:45.954 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:39.829041+0000", 2024-02-09T18:06:45.954 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 186.01939825599999, 2024-02-09T18:06:45.955 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.95663194100000004, 2024-02-09T18:06:45.955 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:45.955 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:45.955 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:45.955 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:06:45.955 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:45.955 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 218926 2024-02-09T18:06:45.955 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.955 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:45.955 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.955 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:45.955 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.829041+0000", 2024-02-09T18:06:45.955 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:45.955 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.955 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.956 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:45.956 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.829041+0000", 2024-02-09T18:06:45.956 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:45.956 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.956 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.956 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:45.956 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.829043+0000", 2024-02-09T18:06:45.956 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.517e-06 2024-02-09T18:06:45.956 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.956 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.956 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:45.956 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.829049+0000", 2024-02-09T18:06:45.956 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0039999999999996e-06 2024-02-09T18:06:45.956 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.957 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.957 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:45.957 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.829062+0000", 2024-02-09T18:06:45.957 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.361e-05 2024-02-09T18:06:45.957 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.957 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.957 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:45.957 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.829071+0000", 2024-02-09T18:06:45.957 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3399999999999998e-06 2024-02-09T18:06:45.957 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.957 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.957 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:45.957 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.829122+0000", 2024-02-09T18:06:45.957 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1651999999999999e-05 2024-02-09T18:06:45.957 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.958 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.958 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:45.958 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.829129+0000", 2024-02-09T18:06:45.958 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9009999999999997e-06 2024-02-09T18:06:45.958 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.958 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.958 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:45.958 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.774852+0000", 2024-02-09T18:06:45.958 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.94572300600000003 2024-02-09T18:06:45.958 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.958 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.958 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:45.958 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.774981+0000", 2024-02-09T18:06:45.958 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000128868 2024-02-09T18:06:45.958 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.959 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.959 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5", 2024-02-09T18:06:45.959 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.775118+0000", 2024-02-09T18:06:45.959 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013751900000000001 2024-02-09T18:06:45.959 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.959 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.959 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:45.959 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.785528+0000", 2024-02-09T18:06:45.959 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010409932 2024-02-09T18:06:45.959 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.959 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.959 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:45.959 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.785583+0000", 2024-02-09T18:06:45.959 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4208e-05 2024-02-09T18:06:45.959 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.959 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.959 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:45.960 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.785596+0000", 2024-02-09T18:06:45.960 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3855e-05 2024-02-09T18:06:45.960 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.960 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.960 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:45.960 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.785647+0000", 2024-02-09T18:06:45.960 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0902999999999997e-05 2024-02-09T18:06:45.960 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.960 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.960 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:45.960 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.785673+0000", 2024-02-09T18:06:45.960 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5626000000000002e-05 2024-02-09T18:06:45.960 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:45.960 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:45.961 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:45.961 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.961 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.961 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:218931 9.b 9:de156c95:::benchmark_data_smithi179_161915_object84049:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e955)", 2024-02-09T18:06:45.961 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:39.835073+0000", 2024-02-09T18:06:45.961 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 186.01336642199999, 2024-02-09T18:06:45.961 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.965848398, 2024-02-09T18:06:45.961 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:45.961 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:45.961 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:45.961 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:06:45.961 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:45.961 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 218931 2024-02-09T18:06:45.961 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.961 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:45.962 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.962 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:45.962 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.835073+0000", 2024-02-09T18:06:45.962 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:45.962 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.962 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.962 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:45.962 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.835073+0000", 2024-02-09T18:06:45.962 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:45.962 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.962 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.962 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:45.962 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.835075+0000", 2024-02-09T18:06:45.962 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.649e-06 2024-02-09T18:06:45.962 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.963 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.963 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:45.963 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.835083+0000", 2024-02-09T18:06:45.963 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3950000000000001e-06 2024-02-09T18:06:45.963 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.963 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.963 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:45.963 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.835099+0000", 2024-02-09T18:06:45.963 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6314e-05 2024-02-09T18:06:45.963 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.963 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.963 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:45.963 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.835109+0000", 2024-02-09T18:06:45.963 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.4059999999999994e-06 2024-02-09T18:06:45.963 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.964 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.964 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:45.964 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.835181+0000", 2024-02-09T18:06:45.964 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2168000000000004e-05 2024-02-09T18:06:45.964 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.964 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.964 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:45.964 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.835193+0000", 2024-02-09T18:06:45.964 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1765e-05 2024-02-09T18:06:45.964 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.964 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.964 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:45.964 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.780183+0000", 2024-02-09T18:06:45.964 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.94499091800000001 2024-02-09T18:06:45.965 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.965 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.965 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:45.965 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.780285+0000", 2024-02-09T18:06:45.965 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000101152 2024-02-09T18:06:45.965 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.965 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.965 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5", 2024-02-09T18:06:45.965 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.780394+0000", 2024-02-09T18:06:45.965 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010912999999999999 2024-02-09T18:06:45.965 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.965 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.965 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:45.965 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.800715+0000", 2024-02-09T18:06:45.966 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.020320794999999999 2024-02-09T18:06:45.966 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.966 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.966 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:45.966 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.800731+0000", 2024-02-09T18:06:45.966 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6047000000000001e-05 2024-02-09T18:06:45.966 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.966 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.966 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:45.966 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.800819+0000", 2024-02-09T18:06:45.966 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7986999999999996e-05 2024-02-09T18:06:45.966 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.966 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.966 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:45.967 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.800881+0000", 2024-02-09T18:06:45.967 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2718999999999994e-05 2024-02-09T18:06:45.967 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.967 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.967 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:45.967 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.800921+0000", 2024-02-09T18:06:45.967 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9953000000000002e-05 2024-02-09T18:06:45.967 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:45.967 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:45.967 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:45.967 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.967 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.967 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:218934 9.b 9:d03e3383:::benchmark_data_smithi179_161915_object84052:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e955)", 2024-02-09T18:06:45.967 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:39.836243+0000", 2024-02-09T18:06:45.967 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 186.012196397, 2024-02-09T18:06:45.968 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.96491841199999995, 2024-02-09T18:06:45.968 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:45.968 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:45.968 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:45.968 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:06:45.968 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:45.968 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 218934 2024-02-09T18:06:45.968 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.968 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:45.968 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.968 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:45.968 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.836243+0000", 2024-02-09T18:06:45.968 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:45.968 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.968 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.969 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:45.969 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.836243+0000", 2024-02-09T18:06:45.969 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:45.969 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.969 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.969 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:45.969 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.836244+0000", 2024-02-09T18:06:45.969 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8199999999999999e-07 2024-02-09T18:06:45.969 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.969 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.969 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:45.969 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.836247+0000", 2024-02-09T18:06:45.969 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.732e-06 2024-02-09T18:06:45.969 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.970 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.970 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:45.970 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.836257+0000", 2024-02-09T18:06:45.970 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0097e-05 2024-02-09T18:06:45.970 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.970 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.970 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:45.970 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.836263+0000", 2024-02-09T18:06:45.970 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3550000000000004e-06 2024-02-09T18:06:45.970 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.970 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.970 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:45.970 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.836313+0000", 2024-02-09T18:06:45.970 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.045e-05 2024-02-09T18:06:45.970 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.970 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.971 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:45.971 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.836322+0000", 2024-02-09T18:06:45.971 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.1689999999999998e-06 2024-02-09T18:06:45.971 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.971 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.971 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:45.971 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.785706+0000", 2024-02-09T18:06:45.971 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.949383317 2024-02-09T18:06:45.971 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.971 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.971 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:45.971 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.785822+0000", 2024-02-09T18:06:45.971 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000115832 2024-02-09T18:06:45.971 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.972 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.972 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5", 2024-02-09T18:06:45.972 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.785896+0000", 2024-02-09T18:06:45.972 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4567000000000006e-05 2024-02-09T18:06:45.972 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.972 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.972 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:45.972 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.800765+0000", 2024-02-09T18:06:45.972 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014868947 2024-02-09T18:06:45.972 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.972 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.972 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:45.972 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.800776+0000", 2024-02-09T18:06:45.972 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1321000000000001e-05 2024-02-09T18:06:45.972 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.973 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.973 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:45.973 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.801049+0000", 2024-02-09T18:06:45.973 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00027226699999999997 2024-02-09T18:06:45.973 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.973 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.973 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:45.973 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.801102+0000", 2024-02-09T18:06:45.973 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.287e-05 2024-02-09T18:06:45.973 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.973 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.973 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:45.973 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.801161+0000", 2024-02-09T18:06:45.973 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9806000000000003e-05 2024-02-09T18:06:45.973 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:45.974 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:45.974 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:45.974 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.974 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.974 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:218949 9.b 9:d4992918:::benchmark_data_smithi179_161915_object84067:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e955)", 2024-02-09T18:06:45.974 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:39.868952+0000", 2024-02-09T18:06:45.974 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 185.979487283, 2024-02-09T18:06:45.974 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.93233731200000003, 2024-02-09T18:06:45.974 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:45.974 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:45.974 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:45.974 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:06:45.974 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:45.974 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 218949 2024-02-09T18:06:45.975 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.975 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:45.975 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.975 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:45.975 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.868952+0000", 2024-02-09T18:06:45.975 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:45.975 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.975 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.975 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:45.975 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.868952+0000", 2024-02-09T18:06:45.975 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:45.975 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.975 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.975 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:45.976 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.868953+0000", 2024-02-09T18:06:45.976 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3859999999999999e-06 2024-02-09T18:06:45.976 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.976 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.976 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:45.976 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.868960+0000", 2024-02-09T18:06:45.976 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9110000000000004e-06 2024-02-09T18:06:45.976 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.976 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.976 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:45.976 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.868975+0000", 2024-02-09T18:06:45.976 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4568e-05 2024-02-09T18:06:45.976 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.976 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.977 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:45.977 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.868983+0000", 2024-02-09T18:06:45.977 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.0739999999999993e-06 2024-02-09T18:06:45.977 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.977 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.977 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:45.977 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.869048+0000", 2024-02-09T18:06:45.977 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4838999999999994e-05 2024-02-09T18:06:45.977 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.977 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.977 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:45.977 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.869062+0000", 2024-02-09T18:06:45.977 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4154e-05 2024-02-09T18:06:45.977 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.977 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.978 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:45.978 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.790566+0000", 2024-02-09T18:06:45.978 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.921504133 2024-02-09T18:06:45.978 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.978 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.978 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:45.978 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.790694+0000", 2024-02-09T18:06:45.978 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012811699999999999 2024-02-09T18:06:45.978 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.978 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.978 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5", 2024-02-09T18:06:45.978 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.790816+0000", 2024-02-09T18:06:45.978 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012227399999999999 2024-02-09T18:06:45.978 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.978 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.979 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:45.979 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.800781+0000", 2024-02-09T18:06:45.979 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0099647710000000007 2024-02-09T18:06:45.979 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.979 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.979 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:45.979 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.800961+0000", 2024-02-09T18:06:45.979 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00018003600000000001 2024-02-09T18:06:45.979 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.979 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.979 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:45.979 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.801204+0000", 2024-02-09T18:06:45.979 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00024293200000000001 2024-02-09T18:06:45.979 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.980 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.980 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:45.980 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.801253+0000", 2024-02-09T18:06:45.980 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.884e-05 2024-02-09T18:06:45.980 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.980 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.980 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:45.980 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.801289+0000", 2024-02-09T18:06:45.980 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6276999999999998e-05 2024-02-09T18:06:45.980 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:45.980 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:45.980 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:45.980 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.980 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.981 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:218959 9.b 9:dd8bf74e:::benchmark_data_smithi179_161915_object84077:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e955)", 2024-02-09T18:06:45.981 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:39.893098+0000", 2024-02-09T18:06:45.981 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 185.95534180000001, 2024-02-09T18:06:45.981 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.90831333999999997, 2024-02-09T18:06:45.981 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:45.981 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:45.981 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:45.981 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:06:45.981 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:45.981 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 218959 2024-02-09T18:06:45.981 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.981 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:45.981 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.981 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:45.981 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.893098+0000", 2024-02-09T18:06:45.982 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:45.982 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.982 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.982 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:45.982 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.893098+0000", 2024-02-09T18:06:45.982 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:45.982 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.982 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.982 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:45.982 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.893099+0000", 2024-02-09T18:06:45.982 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.677e-06 2024-02-09T18:06:45.982 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.982 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.982 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:45.983 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.893105+0000", 2024-02-09T18:06:45.983 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0929999999999999e-06 2024-02-09T18:06:45.983 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.983 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.983 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:45.983 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.893122+0000", 2024-02-09T18:06:45.983 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6328999999999999e-05 2024-02-09T18:06:45.983 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.983 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.983 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:45.983 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.893129+0000", 2024-02-09T18:06:45.983 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5390000000000002e-06 2024-02-09T18:06:45.983 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.983 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.983 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:45.984 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.893198+0000", 2024-02-09T18:06:45.984 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9344999999999994e-05 2024-02-09T18:06:45.984 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.984 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.984 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:45.984 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.893207+0000", 2024-02-09T18:06:45.984 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5029999999999997e-06 2024-02-09T18:06:45.984 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.984 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.984 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:45.984 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.795918+0000", 2024-02-09T18:06:45.984 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.90271066700000002 2024-02-09T18:06:45.984 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.984 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.984 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:45.985 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.796043+0000", 2024-02-09T18:06:45.985 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000125253 2024-02-09T18:06:45.985 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.985 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.985 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5", 2024-02-09T18:06:45.985 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.796173+0000", 2024-02-09T18:06:45.985 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000130288 2024-02-09T18:06:45.985 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.985 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.985 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:45.985 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.800977+0000", 2024-02-09T18:06:45.985 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0048040649999999997 2024-02-09T18:06:45.985 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.985 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.986 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:45.986 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.801011+0000", 2024-02-09T18:06:45.986 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.3549e-05 2024-02-09T18:06:45.986 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.986 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.986 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:45.986 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.801331+0000", 2024-02-09T18:06:45.986 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00031982100000000003 2024-02-09T18:06:45.986 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.986 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.986 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:45.986 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.801376+0000", 2024-02-09T18:06:45.986 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5825000000000003e-05 2024-02-09T18:06:45.986 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.986 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.987 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:45.987 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.801411+0000", 2024-02-09T18:06:45.987 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4385999999999999e-05 2024-02-09T18:06:45.987 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:45.987 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:45.987 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:45.987 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.987 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.987 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.26563.0:218960 9.b 9:df61268f:::benchmark_data_smithi179_161915_object84078:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e955)", 2024-02-09T18:06:45.987 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-02-09T18:03:39.897308+0000", 2024-02-09T18:06:45.987 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 185.951131062, 2024-02-09T18:06:45.987 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.92915402400000002, 2024-02-09T18:06:45.987 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-02-09T18:06:45.987 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:45.988 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-02-09T18:06:45.988 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.26563", 2024-02-09T18:06:45.988 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:45.988 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 218960 2024-02-09T18:06:45.988 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.988 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-02-09T18:06:45.988 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.988 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-02-09T18:06:45.988 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.897308+0000", 2024-02-09T18:06:45.988 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:45.988 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.988 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.988 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-02-09T18:06:45.988 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.897308+0000", 2024-02-09T18:06:45.989 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-02-09T18:06:45.989 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.989 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.989 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-02-09T18:06:45.989 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.897309+0000", 2024-02-09T18:06:45.989 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2200000000000002e-07 2024-02-09T18:06:45.989 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.989 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.989 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-02-09T18:06:45.989 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.897314+0000", 2024-02-09T18:06:45.989 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0919999999999997e-06 2024-02-09T18:06:45.989 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.989 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.990 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-02-09T18:06:45.990 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.897325+0000", 2024-02-09T18:06:45.990 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1097999999999999e-05 2024-02-09T18:06:45.990 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.990 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.990 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-02-09T18:06:45.990 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.897332+0000", 2024-02-09T18:06:45.990 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8770000000000004e-06 2024-02-09T18:06:45.990 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.990 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.990 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:45.990 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.897384+0000", 2024-02-09T18:06:45.990 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1799999999999999e-05 2024-02-09T18:06:45.990 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.990 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.991 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-02-09T18:06:45.991 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:39.897391+0000", 2024-02-09T18:06:45.991 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4499999999999998e-06 2024-02-09T18:06:45.991 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.991 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.991 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-02-09T18:06:45.991 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.801446+0000", 2024-02-09T18:06:45.991 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.90405485100000005 2024-02-09T18:06:45.991 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.991 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.991 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-02-09T18:06:45.991 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.801555+0000", 2024-02-09T18:06:45.991 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000108349 2024-02-09T18:06:45.991 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.991 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.992 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5", 2024-02-09T18:06:45.992 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.801621+0000", 2024-02-09T18:06:45.992 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6285000000000007e-05 2024-02-09T18:06:45.992 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.992 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.992 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:45.992 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.812163+0000", 2024-02-09T18:06:45.992 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010541903 2024-02-09T18:06:45.992 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.992 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.992 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-02-09T18:06:45.992 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.812177+0000", 2024-02-09T18:06:45.992 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4188e-05 2024-02-09T18:06:45.992 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.993 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.993 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:45.993 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.826369+0000", 2024-02-09T18:06:45.993 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.014192421 2024-02-09T18:06:45.993 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.993 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.993 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-02-09T18:06:45.993 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.826421+0000", 2024-02-09T18:06:45.993 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1959999999999997e-05 2024-02-09T18:06:45.993 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-02-09T18:06:45.993 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-02-09T18:06:45.993 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-02-09T18:06:45.993 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-02-09T18:03:40.826462+0000", 2024-02-09T18:06:45.993 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0828000000000003e-05 2024-02-09T18:06:45.993 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:45.994 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:45.994 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:45.994 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-02-09T18:06:45.994 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-02-09T18:06:45.994 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-02-09T18:06:45.995 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:45 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3780593899' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-02-09T18:06:45.995 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:45 smithi162 ceph-mon[44304]: osdmap e1028: 12 total, 12 up, 12 in 2024-02-09T18:06:45.995 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:45 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3780593899' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-02-09T18:06:45.995 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:45 smithi179 ceph-mon[43975]: osdmap e1028: 12 total, 12 up, 12 in 2024-02-09T18:06:46.520 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-09T18:06:46.861 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:46 smithi099 ceph-mon[39420]: pgmap v2528: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:46.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:46 smithi179 ceph-mon[43975]: pgmap v2528: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:46.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:46 smithi162 ceph-mon[44304]: pgmap v2528: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:47.431 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:06:48.175 INFO:teuthology.orchestra.run.smithi099.stderr:set osd.8 primary-affinity to 1 (8655362) 2024-02-09T18:06:48.197 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:48 smithi099 ceph-mon[39420]: pgmap v2529: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:48.197 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:48 smithi099 ceph-mon[39420]: 1.e deep-scrub starts 2024-02-09T18:06:48.197 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:48 smithi099 ceph-mon[39420]: 1.e deep-scrub ok 2024-02-09T18:06:48.197 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:48 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2086827234' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-02-09T18:06:48.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:48 smithi179 ceph-mon[43975]: pgmap v2529: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:48.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:48 smithi179 ceph-mon[43975]: 1.e deep-scrub starts 2024-02-09T18:06:48.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:48 smithi179 ceph-mon[43975]: 1.e deep-scrub ok 2024-02-09T18:06:48.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:48 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2086827234' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-02-09T18:06:48.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:48 smithi162 ceph-mon[44304]: pgmap v2529: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:48.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:48 smithi162 ceph-mon[44304]: 1.e deep-scrub starts 2024-02-09T18:06:48.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:48 smithi162 ceph-mon[44304]: 1.e deep-scrub ok 2024-02-09T18:06:48.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:48 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2086827234' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-02-09T18:06:49.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:49 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2086827234' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-02-09T18:06:49.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:49 smithi099 ceph-mon[39420]: osdmap e1029: 12 total, 12 up, 12 in 2024-02-09T18:06:49.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:49 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2086827234' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-02-09T18:06:49.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:49 smithi162 ceph-mon[44304]: osdmap e1029: 12 total, 12 up, 12 in 2024-02-09T18:06:49.435 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd reweight 3 1 2024-02-09T18:06:49.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:49 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2086827234' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-02-09T18:06:49.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:49 smithi179 ceph-mon[43975]: osdmap e1029: 12 total, 12 up, 12 in 2024-02-09T18:06:50.004 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:06:50.016 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T18:06:50.016 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [], 2024-02-09T18:06:50.017 INFO:teuthology.orchestra.run.smithi099.stdout: "num_ops": 0 2024-02-09T18:06:50.017 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T18:06:50.495 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:50 smithi099 ceph-mon[39420]: 1.b scrub starts 2024-02-09T18:06:50.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:50 smithi099 ceph-mon[39420]: 1.b scrub ok 2024-02-09T18:06:50.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:50 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:06:50.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:50 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:06:50.496 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:50 smithi099 ceph-mon[39420]: pgmap v2531: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:50.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:50 smithi179 ceph-mon[43975]: 1.b scrub starts 2024-02-09T18:06:50.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:50 smithi179 ceph-mon[43975]: 1.b scrub ok 2024-02-09T18:06:50.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:50 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:06:50.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:50 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:06:50.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:50 smithi179 ceph-mon[43975]: pgmap v2531: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:50.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:50 smithi162 ceph-mon[44304]: 1.b scrub starts 2024-02-09T18:06:50.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:50 smithi162 ceph-mon[44304]: 1.b scrub ok 2024-02-09T18:06:50.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:50 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:06:50.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:50 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:06:50.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:50 smithi162 ceph-mon[44304]: pgmap v2531: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:51.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:51 smithi099 ceph-mon[39420]: osdmap e1030: 12 total, 12 up, 12 in 2024-02-09T18:06:51.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:51 smithi179 ceph-mon[43975]: osdmap e1030: 12 total, 12 up, 12 in 2024-02-09T18:06:51.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:51 smithi162 ceph-mon[44304]: osdmap e1030: 12 total, 12 up, 12 in 2024-02-09T18:06:51.847 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-09T18:06:52.417 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:06:52.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:52 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:06:52.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:52 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:06:52.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:52 smithi099 ceph-mon[39420]: osdmap e1031: 12 total, 12 up, 12 in 2024-02-09T18:06:52.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:52 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:06:52.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:52 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:06:52.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:52 smithi099 ceph-mon[39420]: pgmap v2534: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:52.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:52 smithi099 ceph-mon[39420]: osdmap e1032: 12 total, 12 up, 12 in 2024-02-09T18:06:52.570 INFO:teuthology.orchestra.run.smithi099.stderr:2024-02-09T18:06:52.569+0000 7fe0e52cf700 0 --2- 172.21.15.99:0/3865188825 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe0e0100c20 0x7fe0e0101080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-02-09T18:06:52.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:52 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:06:52.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:52 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:06:52.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:52 smithi179 ceph-mon[43975]: osdmap e1031: 12 total, 12 up, 12 in 2024-02-09T18:06:52.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:52 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:06:52.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:52 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:06:52.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:52 smithi179 ceph-mon[43975]: pgmap v2534: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:52.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:52 smithi179 ceph-mon[43975]: osdmap e1032: 12 total, 12 up, 12 in 2024-02-09T18:06:52.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:52 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:06:52.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:52 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:06:52.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:52 smithi162 ceph-mon[44304]: osdmap e1031: 12 total, 12 up, 12 in 2024-02-09T18:06:52.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:52 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:06:52.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:52 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:06:52.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:52 smithi162 ceph-mon[44304]: pgmap v2534: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:52.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:52 smithi162 ceph-mon[44304]: osdmap e1032: 12 total, 12 up, 12 in 2024-02-09T18:06:53.203 INFO:teuthology.orchestra.run.smithi099.stderr:reweighted osd.3 to 1 (10000) 2024-02-09T18:06:53.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:53 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3036379375' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-09T18:06:53.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:53 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3036379375' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-09T18:06:53.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:53 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3036379375' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-09T18:06:54.218 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd primary-affinity 3 1 2024-02-09T18:06:54.446 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:54 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3036379375' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-09T18:06:54.446 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:54 smithi099 ceph-mon[39420]: osdmap e1033: 12 total, 12 up, 12 in 2024-02-09T18:06:54.446 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:54 smithi099 ceph-mon[39420]: pgmap v2537: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:54.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:54 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3036379375' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-09T18:06:54.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:54 smithi179 ceph-mon[43975]: osdmap e1033: 12 total, 12 up, 12 in 2024-02-09T18:06:54.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:54 smithi179 ceph-mon[43975]: pgmap v2537: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:54.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:54 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3036379375' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-09T18:06:54.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:54 smithi162 ceph-mon[44304]: osdmap e1033: 12 total, 12 up, 12 in 2024-02-09T18:06:54.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:54 smithi162 ceph-mon[44304]: pgmap v2537: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:54.732 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T18:06:54.732 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [], 2024-02-09T18:06:54.733 INFO:teuthology.orchestra.run.smithi099.stdout: "complaint_time": 30, 2024-02-09T18:06:54.733 INFO:teuthology.orchestra.run.smithi099.stdout: "num_blocked_ops": 0 2024-02-09T18:06:54.733 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T18:06:54.741 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:06:55.428 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:55 smithi099 ceph-mon[39420]: 1.18 scrub starts 2024-02-09T18:06:55.428 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:55 smithi099 ceph-mon[39420]: 1.18 scrub ok 2024-02-09T18:06:55.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:55 smithi179 ceph-mon[43975]: 1.18 scrub starts 2024-02-09T18:06:55.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:55 smithi179 ceph-mon[43975]: 1.18 scrub ok 2024-02-09T18:06:55.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:55 smithi162 ceph-mon[44304]: 1.18 scrub starts 2024-02-09T18:06:55.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:55 smithi162 ceph-mon[44304]: 1.18 scrub ok 2024-02-09T18:06:56.326 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-09T18:06:56.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:56 smithi099 ceph-mon[39420]: pgmap v2538: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:56.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:56 smithi179 ceph-mon[43975]: pgmap v2538: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:56.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:56 smithi162 ceph-mon[44304]: pgmap v2538: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:57.058 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:06:58.170 INFO:teuthology.orchestra.run.smithi099.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-02-09T18:06:58.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:58 smithi179 ceph-mon[43975]: osdmap e1034: 12 total, 12 up, 12 in 2024-02-09T18:06:58.439 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:58 smithi179 ceph-mon[43975]: pgmap v2540: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:58.439 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:58 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2170433448' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-09T18:06:58.439 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:58 smithi162 ceph-mon[44304]: osdmap e1034: 12 total, 12 up, 12 in 2024-02-09T18:06:58.440 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:58 smithi162 ceph-mon[44304]: pgmap v2540: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:58.440 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:58 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2170433448' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-09T18:06:58.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:58 smithi099 ceph-mon[39420]: osdmap e1034: 12 total, 12 up, 12 in 2024-02-09T18:06:58.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:58 smithi099 ceph-mon[39420]: pgmap v2540: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:06:58.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:58 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2170433448' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-09T18:06:59.255 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-02-09T18:06:59.256 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-02-09T18:06:59.256 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-02-09T18:06:59.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:59 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2170433448' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-02-09T18:06:59.427 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:06:59 smithi162 ceph-mon[44304]: osdmap e1035: 12 total, 12 up, 12 in 2024-02-09T18:06:59.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:59 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2170433448' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-02-09T18:06:59.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:06:59 smithi099 ceph-mon[39420]: osdmap e1035: 12 total, 12 up, 12 in 2024-02-09T18:06:59.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:59 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2170433448' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-02-09T18:06:59.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:06:59 smithi179 ceph-mon[43975]: osdmap e1035: 12 total, 12 up, 12 in 2024-02-09T18:06:59.819 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-02-09T18:06:59.819 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 20, 2024-02-09T18:06:59.820 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 600, 2024-02-09T18:06:59.820 INFO:teuthology.orchestra.run.smithi099.stdout: "ops": [ 2024-02-09T18:06:59.820 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.820 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "MOSDPGPush(8.8 862/855 [PushOp(8:1051e066:::benchmark_data_smithi179_148205_object114077:head, version: 818'7173, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:1051e066:::benchmark_data_smithi179_148205_object114077:head@818'7173, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-02-09T18:06:59.820 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:59:25.526864+0000", 2024-02-09T18:06:59.820 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 454.28893536200002, 2024-02-09T18:06:59.820 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.32089669799999998, 2024-02-09T18:06:59.820 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:06:59.820 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T18:06:59.820 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:06:59.820 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.820 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:06:59.820 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:25.526864+0000", 2024-02-09T18:06:59.821 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:06:59.821 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.821 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.821 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:06:59.821 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:25.526864+0000", 2024-02-09T18:06:59.821 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:06:59.821 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.821 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.821 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:06:59.821 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:25.526865+0000", 2024-02-09T18:06:59.821 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.705e-06 2024-02-09T18:06:59.821 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.821 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.821 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:06:59.822 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:25.526908+0000", 2024-02-09T18:06:59.822 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.2385999999999997e-05 2024-02-09T18:06:59.822 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.822 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.822 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:06:59.822 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:25.526943+0000", 2024-02-09T18:06:59.822 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.5564e-05 2024-02-09T18:06:59.822 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.822 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.822 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:06:59.822 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:25.526963+0000", 2024-02-09T18:06:59.822 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.9371000000000001e-05 2024-02-09T18:06:59.822 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.822 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.823 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:06:59.823 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:25.838423+0000", 2024-02-09T18:06:59.823 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.31145991000000001 2024-02-09T18:06:59.823 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.823 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.823 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:06:59.823 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:25.838436+0000", 2024-02-09T18:06:59.823 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.394e-05 2024-02-09T18:06:59.823 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.823 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.823 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:06:59.823 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:25.847760+0000", 2024-02-09T18:06:59.823 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0093238220000000007 2024-02-09T18:06:59.823 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:06:59.824 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:06:59.824 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:06:59.824 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.824 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.824 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "MOSDPGPush(8.0 862/855 [PushOp(8:0ca54e91:::benchmark_data_smithi179_148205_object113481:head, version: 817'7260, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:0ca54e91:::benchmark_data_smithi179_148205_object113481:head@817'7260, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-02-09T18:06:59.825 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T17:59:25.543883+0000", 2024-02-09T18:06:59.825 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 454.27191552699998, 2024-02-09T18:06:59.825 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.366177213, 2024-02-09T18:06:59.825 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:06:59.825 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "started", 2024-02-09T18:06:59.825 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:06:59.825 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.825 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:06:59.825 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:25.543883+0000", 2024-02-09T18:06:59.825 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:06:59.825 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.825 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.825 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:06:59.825 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:25.543883+0000", 2024-02-09T18:06:59.825 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:06:59.826 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.826 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.826 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:06:59.826 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:25.543885+0000", 2024-02-09T18:06:59.826 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1260000000000001e-06 2024-02-09T18:06:59.826 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.826 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.826 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:06:59.826 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:25.543943+0000", 2024-02-09T18:06:59.826 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.8764000000000002e-05 2024-02-09T18:06:59.826 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.826 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.826 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:06:59.826 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:25.543971+0000", 2024-02-09T18:06:59.826 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.7291e-05 2024-02-09T18:06:59.827 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.827 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.827 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:06:59.827 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:25.543993+0000", 2024-02-09T18:06:59.827 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2283999999999999e-05 2024-02-09T18:06:59.827 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.827 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.827 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:06:59.827 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:25.906614+0000", 2024-02-09T18:06:59.827 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.36262160799999998 2024-02-09T18:06:59.827 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.827 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.827 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:06:59.827 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:25.906627+0000", 2024-02-09T18:06:59.827 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2627e-05 2024-02-09T18:06:59.828 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.828 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.828 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:06:59.828 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T17:59:25.910061+0000", 2024-02-09T18:06:59.828 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.003433513 2024-02-09T18:06:59.829 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:06:59.829 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:06:59.829 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:06:59.829 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.829 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.829 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26563.0:102123 9.c 9:31005b1a:::benchmark_data_smithi179_161915_object102122:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e908)", 2024-02-09T18:06:59.829 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:01:15.801564+0000", 2024-02-09T18:06:59.829 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 344.01423506600003, 2024-02-09T18:06:59.829 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.22203677899999999, 2024-02-09T18:06:59.829 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:06:59.829 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:59.829 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:06:59.829 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26563", 2024-02-09T18:06:59.829 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:59.830 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 102123 2024-02-09T18:06:59.830 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.830 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:06:59.830 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.830 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:06:59.830 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.801564+0000", 2024-02-09T18:06:59.830 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:06:59.830 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.830 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.830 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:06:59.830 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.801564+0000", 2024-02-09T18:06:59.830 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:06:59.830 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.830 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.831 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:06:59.831 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.801566+0000", 2024-02-09T18:06:59.831 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.6390000000000001e-06 2024-02-09T18:06:59.831 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.831 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.831 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:06:59.831 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.801621+0000", 2024-02-09T18:06:59.831 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.5532999999999998e-05 2024-02-09T18:06:59.831 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.831 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.831 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:06:59.831 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.801632+0000", 2024-02-09T18:06:59.831 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0822000000000001e-05 2024-02-09T18:06:59.831 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.831 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.832 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:06:59.832 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.801638+0000", 2024-02-09T18:06:59.832 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6960000000000002e-06 2024-02-09T18:06:59.832 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.832 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.832 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:06:59.832 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.801698+0000", 2024-02-09T18:06:59.832 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.0013000000000003e-05 2024-02-09T18:06:59.832 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.832 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.832 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:06:59.832 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.801828+0000", 2024-02-09T18:06:59.832 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000130901 2024-02-09T18:06:59.833 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.833 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.833 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,9", 2024-02-09T18:06:59.833 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.801933+0000", 2024-02-09T18:06:59.833 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000104603 2024-02-09T18:06:59.833 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.833 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.833 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:59.833 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.807655+0000", 2024-02-09T18:06:59.833 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0057223070000000003 2024-02-09T18:06:59.833 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.833 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.833 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:06:59.833 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.812421+0000", 2024-02-09T18:06:59.834 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0047657300000000001 2024-02-09T18:06:59.834 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.834 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.834 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:59.834 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:16.023451+0000", 2024-02-09T18:06:59.834 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.211029622 2024-02-09T18:06:59.834 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.834 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.834 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:06:59.834 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:16.023570+0000", 2024-02-09T18:06:59.834 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000119101 2024-02-09T18:06:59.834 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.835 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.835 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:06:59.835 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:16.023601+0000", 2024-02-09T18:06:59.835 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.0812000000000002e-05 2024-02-09T18:06:59.835 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:06:59.835 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:06:59.835 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:06:59.835 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.835 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.836 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26563.0:102150 9.2 9:43bb530d:::benchmark_data_smithi179_161915_object102149:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e908)", 2024-02-09T18:06:59.836 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:01:15.819292+0000", 2024-02-09T18:06:59.836 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 343.99650728400002, 2024-02-09T18:06:59.836 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.21020171400000001, 2024-02-09T18:06:59.836 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:06:59.836 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:59.836 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:06:59.837 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26563", 2024-02-09T18:06:59.837 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:59.837 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 102150 2024-02-09T18:06:59.837 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.837 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:06:59.837 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.837 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:06:59.837 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.819292+0000", 2024-02-09T18:06:59.837 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:06:59.837 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.837 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.837 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:06:59.837 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.819292+0000", 2024-02-09T18:06:59.837 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:06:59.837 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.837 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.838 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:06:59.838 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.819293+0000", 2024-02-09T18:06:59.838 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5710000000000001e-06 2024-02-09T18:06:59.838 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.838 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.838 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:06:59.838 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.819337+0000", 2024-02-09T18:06:59.838 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.4091999999999997e-05 2024-02-09T18:06:59.838 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.838 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.838 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:06:59.838 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.819349+0000", 2024-02-09T18:06:59.838 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2055999999999999e-05 2024-02-09T18:06:59.838 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.838 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.839 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:06:59.839 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.819357+0000", 2024-02-09T18:06:59.839 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.0229999999999992e-06 2024-02-09T18:06:59.839 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.839 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.839 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:06:59.839 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.819426+0000", 2024-02-09T18:06:59.839 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.8314999999999999e-05 2024-02-09T18:06:59.839 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.839 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.839 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:06:59.839 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.819563+0000", 2024-02-09T18:06:59.839 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013752799999999999 2024-02-09T18:06:59.839 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.839 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.840 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,9", 2024-02-09T18:06:59.840 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.819709+0000", 2024-02-09T18:06:59.840 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014563 2024-02-09T18:06:59.840 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.840 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.840 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:06:59.840 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.822488+0000", 2024-02-09T18:06:59.840 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0027790689999999999 2024-02-09T18:06:59.840 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.840 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.840 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:59.840 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.822924+0000", 2024-02-09T18:06:59.840 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00043558699999999998 2024-02-09T18:06:59.841 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.841 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.841 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:59.841 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:16.029425+0000", 2024-02-09T18:06:59.841 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20650153099999999 2024-02-09T18:06:59.841 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.841 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.841 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:06:59.841 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:16.029466+0000", 2024-02-09T18:06:59.841 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.1100000000000003e-05 2024-02-09T18:06:59.841 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.841 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.841 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:06:59.841 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:16.029493+0000", 2024-02-09T18:06:59.841 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.7212e-05 2024-02-09T18:06:59.841 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:06:59.842 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:06:59.842 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:06:59.842 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.842 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.842 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26563.0:102153 9.c 9:33a88231:::benchmark_data_smithi179_161915_object102152:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e908)", 2024-02-09T18:06:59.842 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:01:15.823126+0000", 2024-02-09T18:06:59.842 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 343.99267279600002, 2024-02-09T18:06:59.842 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.214794611, 2024-02-09T18:06:59.842 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:06:59.842 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:59.842 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:06:59.842 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26563", 2024-02-09T18:06:59.842 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:59.842 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 102153 2024-02-09T18:06:59.842 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.843 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:06:59.843 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.843 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:06:59.843 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.823126+0000", 2024-02-09T18:06:59.843 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:06:59.843 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.843 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.843 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:06:59.843 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.823126+0000", 2024-02-09T18:06:59.843 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:06:59.843 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.843 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.844 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:06:59.844 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.823127+0000", 2024-02-09T18:06:59.844 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.4e-07 2024-02-09T18:06:59.844 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.844 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.844 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:06:59.844 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.823184+0000", 2024-02-09T18:06:59.844 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7184e-05 2024-02-09T18:06:59.844 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.844 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.844 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:06:59.844 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.823193+0000", 2024-02-09T18:06:59.844 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.0140000000000004e-06 2024-02-09T18:06:59.844 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.844 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.844 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:06:59.845 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.823199+0000", 2024-02-09T18:06:59.845 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.5430000000000001e-06 2024-02-09T18:06:59.845 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.845 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.845 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:06:59.845 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.823244+0000", 2024-02-09T18:06:59.845 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.5008e-05 2024-02-09T18:06:59.845 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.845 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.845 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:06:59.845 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.823384+0000", 2024-02-09T18:06:59.845 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000139715 2024-02-09T18:06:59.845 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.845 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.846 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,9", 2024-02-09T18:06:59.846 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.823505+0000", 2024-02-09T18:06:59.846 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000121358 2024-02-09T18:06:59.846 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.846 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.846 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:06:59.846 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.830435+0000", 2024-02-09T18:06:59.846 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0069299110000000004 2024-02-09T18:06:59.846 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.846 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.846 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:59.846 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.831723+0000", 2024-02-09T18:06:59.846 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0012878259999999999 2024-02-09T18:06:59.846 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.846 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.847 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:59.847 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:16.037829+0000", 2024-02-09T18:06:59.847 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.206106659 2024-02-09T18:06:59.847 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.847 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.847 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:06:59.847 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:16.037884+0000", 2024-02-09T18:06:59.847 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.4416000000000002e-05 2024-02-09T18:06:59.847 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.847 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.847 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:06:59.847 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:16.037921+0000", 2024-02-09T18:06:59.847 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.7036999999999998e-05 2024-02-09T18:06:59.847 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:06:59.847 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:06:59.848 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:06:59.848 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.848 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.848 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26563.0:102155 9.2 9:41bfd2a4:::benchmark_data_smithi179_161915_object102154:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e908)", 2024-02-09T18:06:59.848 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:01:15.823239+0000", 2024-02-09T18:06:59.848 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 343.99255962500001, 2024-02-09T18:06:59.848 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20882051400000001, 2024-02-09T18:06:59.848 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:06:59.848 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:59.848 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:06:59.848 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26563", 2024-02-09T18:06:59.848 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:59.848 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 102155 2024-02-09T18:06:59.848 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.849 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:06:59.849 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.849 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:06:59.849 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.823239+0000", 2024-02-09T18:06:59.849 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:06:59.849 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.849 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.849 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:06:59.849 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.823239+0000", 2024-02-09T18:06:59.849 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:06:59.849 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.849 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.849 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:06:59.849 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.823240+0000", 2024-02-09T18:06:59.849 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7999999999999995e-07 2024-02-09T18:06:59.850 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.850 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.850 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:06:59.850 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.823338+0000", 2024-02-09T18:06:59.850 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.8484000000000001e-05 2024-02-09T18:06:59.850 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.850 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.850 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:06:59.850 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.823347+0000", 2024-02-09T18:06:59.850 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.2020000000000004e-06 2024-02-09T18:06:59.850 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.850 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.850 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:06:59.850 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.823353+0000", 2024-02-09T18:06:59.850 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.6189999999999996e-06 2024-02-09T18:06:59.851 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.851 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.851 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:06:59.851 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.823406+0000", 2024-02-09T18:06:59.851 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2457000000000003e-05 2024-02-09T18:06:59.851 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.851 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.851 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:06:59.851 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.823538+0000", 2024-02-09T18:06:59.851 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000132557 2024-02-09T18:06:59.851 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.851 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.851 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,9", 2024-02-09T18:06:59.851 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.823692+0000", 2024-02-09T18:06:59.851 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00015398300000000001 2024-02-09T18:06:59.851 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.852 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.852 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:06:59.852 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.826758+0000", 2024-02-09T18:06:59.852 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0030653630000000002 2024-02-09T18:06:59.852 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.852 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.852 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:59.852 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.829858+0000", 2024-02-09T18:06:59.852 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.003100659 2024-02-09T18:06:59.852 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.852 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.852 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:59.852 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:16.031957+0000", 2024-02-09T18:06:59.852 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20209832799999999 2024-02-09T18:06:59.853 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.853 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.853 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:06:59.853 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:16.032021+0000", 2024-02-09T18:06:59.853 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.3973000000000006e-05 2024-02-09T18:06:59.853 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.853 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.853 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:06:59.853 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:16.032060+0000", 2024-02-09T18:06:59.853 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9308999999999999e-05 2024-02-09T18:06:59.853 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:06:59.853 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:06:59.853 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:06:59.853 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.853 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.854 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26563.0:102157 9.c 9:38079185:::benchmark_data_smithi179_161915_object102156:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e908)", 2024-02-09T18:06:59.854 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:01:15.823665+0000", 2024-02-09T18:06:59.854 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 343.99213392199999, 2024-02-09T18:06:59.854 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.21436031699999999, 2024-02-09T18:06:59.854 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:06:59.854 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:59.854 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:06:59.854 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26563", 2024-02-09T18:06:59.854 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:59.854 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 102157 2024-02-09T18:06:59.854 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.854 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:06:59.854 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.854 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:06:59.854 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.823665+0000", 2024-02-09T18:06:59.855 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:06:59.855 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.855 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.855 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:06:59.855 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.823665+0000", 2024-02-09T18:06:59.855 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:06:59.855 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.855 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.855 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:06:59.855 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.823666+0000", 2024-02-09T18:06:59.855 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.5499999999999997e-07 2024-02-09T18:06:59.855 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.855 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.855 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:06:59.855 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.823733+0000", 2024-02-09T18:06:59.855 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.7019000000000003e-05 2024-02-09T18:06:59.856 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.856 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.856 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:06:59.856 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.823744+0000", 2024-02-09T18:06:59.856 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1389000000000001e-05 2024-02-09T18:06:59.856 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.856 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.856 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:06:59.856 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.823752+0000", 2024-02-09T18:06:59.856 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.7600000000000002e-06 2024-02-09T18:06:59.856 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.856 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.856 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:06:59.856 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.826795+0000", 2024-02-09T18:06:59.856 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0030426960000000001 2024-02-09T18:06:59.857 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.857 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.857 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:06:59.857 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.826939+0000", 2024-02-09T18:06:59.857 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014479900000000001 2024-02-09T18:06:59.857 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.857 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.857 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,9", 2024-02-09T18:06:59.857 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.827093+0000", 2024-02-09T18:06:59.857 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000153232 2024-02-09T18:06:59.857 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.857 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.857 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:06:59.857 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.830456+0000", 2024-02-09T18:06:59.857 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.003362898 2024-02-09T18:06:59.858 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.858 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.858 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:59.858 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.831855+0000", 2024-02-09T18:06:59.858 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0013997269999999999 2024-02-09T18:06:59.858 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.858 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.858 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:59.858 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:16.037966+0000", 2024-02-09T18:06:59.858 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20611041199999999 2024-02-09T18:06:59.858 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.858 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.858 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:06:59.858 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:16.038001+0000", 2024-02-09T18:06:59.858 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.5491999999999998e-05 2024-02-09T18:06:59.859 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.859 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.859 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:06:59.859 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:16.038025+0000", 2024-02-09T18:06:59.859 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.4138000000000002e-05 2024-02-09T18:06:59.859 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:06:59.859 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:06:59.859 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:06:59.859 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.859 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.859 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26563.0:102175 9.c 9:36f2471f:::benchmark_data_smithi179_161915_object102174:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e908)", 2024-02-09T18:06:59.859 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:01:15.837929+0000", 2024-02-09T18:06:59.859 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 343.97787022799997, 2024-02-09T18:06:59.859 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20884376499999999, 2024-02-09T18:06:59.860 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:06:59.860 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:59.860 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:06:59.860 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26563", 2024-02-09T18:06:59.860 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:59.860 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 102175 2024-02-09T18:06:59.860 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.860 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:06:59.860 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.860 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:06:59.860 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.837929+0000", 2024-02-09T18:06:59.860 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:06:59.860 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.860 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.860 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:06:59.861 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.837929+0000", 2024-02-09T18:06:59.861 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:06:59.861 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.861 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.861 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:06:59.861 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.837930+0000", 2024-02-09T18:06:59.861 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.7689999999999999e-06 2024-02-09T18:06:59.861 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.861 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.861 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:06:59.861 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.838001+0000", 2024-02-09T18:06:59.861 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.0166999999999994e-05 2024-02-09T18:06:59.861 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.861 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.861 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:06:59.862 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.838014+0000", 2024-02-09T18:06:59.862 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3478999999999999e-05 2024-02-09T18:06:59.862 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.862 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.862 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:06:59.862 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.838022+0000", 2024-02-09T18:06:59.862 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.7600000000000002e-06 2024-02-09T18:06:59.862 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.862 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.862 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:06:59.862 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.838075+0000", 2024-02-09T18:06:59.862 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3049e-05 2024-02-09T18:06:59.862 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.862 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.862 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:06:59.863 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.838171+0000", 2024-02-09T18:06:59.863 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.6260000000000003e-05 2024-02-09T18:06:59.863 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.863 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.863 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 0,9", 2024-02-09T18:06:59.863 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.838261+0000", 2024-02-09T18:06:59.863 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.0012999999999994e-05 2024-02-09T18:06:59.863 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.863 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.863 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:06:59.863 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.841033+0000", 2024-02-09T18:06:59.863 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0027719239999999998 2024-02-09T18:06:59.863 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.863 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.863 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:59.863 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:15.842032+0000", 2024-02-09T18:06:59.864 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00099858600000000009 2024-02-09T18:06:59.864 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.864 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.864 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:59.864 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:16.046689+0000", 2024-02-09T18:06:59.864 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20465778200000001 2024-02-09T18:06:59.864 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.864 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.864 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:06:59.864 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:16.046731+0000", 2024-02-09T18:06:59.864 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.1307000000000003e-05 2024-02-09T18:06:59.864 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.864 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.864 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:06:59.864 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:01:16.046772+0000", 2024-02-09T18:06:59.865 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.1668999999999997e-05 2024-02-09T18:06:59.865 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:06:59.865 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:06:59.865 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:06:59.865 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.865 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.865 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26563.0:202995 9.7 9:e7890342:::benchmark_data_smithi179_161915_object68113:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e942)", 2024-02-09T18:06:59.865 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:03:13.440542+0000", 2024-02-09T18:06:59.865 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 226.37525737300001, 2024-02-09T18:06:59.865 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.2145433790000002, 2024-02-09T18:06:59.865 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:06:59.865 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "reached pg", 2024-02-09T18:06:59.865 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:06:59.865 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26563", 2024-02-09T18:06:59.865 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:59.866 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 202995 2024-02-09T18:06:59.866 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.866 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:06:59.866 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.866 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:06:59.866 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.440542+0000", 2024-02-09T18:06:59.866 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:06:59.866 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.866 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.866 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:06:59.866 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.440542+0000", 2024-02-09T18:06:59.866 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:06:59.866 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.866 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.866 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:06:59.867 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.440543+0000", 2024-02-09T18:06:59.867 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.874e-06 2024-02-09T18:06:59.867 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.867 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.867 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:06:59.867 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.440551+0000", 2024-02-09T18:06:59.867 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.0190000000000006e-06 2024-02-09T18:06:59.867 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.867 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.867 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:06:59.867 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.462902+0000", 2024-02-09T18:06:59.867 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.022350722999999999 2024-02-09T18:06:59.867 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.867 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.867 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:06:59.868 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.462919+0000", 2024-02-09T18:06:59.868 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.6901e-05 2024-02-09T18:06:59.868 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.868 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.868 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:06:59.868 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.463199+0000", 2024-02-09T18:06:59.868 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00028011900000000001 2024-02-09T18:06:59.868 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.868 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.868 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op must wait for map", 2024-02-09T18:06:59.868 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.463207+0000", 2024-02-09T18:06:59.868 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.3420000000000002e-06 2024-02-09T18:06:59.868 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.868 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.868 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:06:59.868 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.469139+0000", 2024-02-09T18:06:59.869 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0059323939999999997 2024-02-09T18:06:59.869 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.869 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.869 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T18:06:59.869 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:13.469149+0000", 2024-02-09T18:06:59.869 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.9760000000000007e-06 2024-02-09T18:06:59.869 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.869 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.869 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:06:59.869 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:14.439912+0000", 2024-02-09T18:06:59.869 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.97076341499999996 2024-02-09T18:06:59.869 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.869 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.869 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for peered", 2024-02-09T18:06:59.869 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:14.439929+0000", 2024-02-09T18:06:59.870 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.6429000000000001e-05 2024-02-09T18:06:59.870 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.870 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.870 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:06:59.870 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.652439+0000", 2024-02-09T18:06:59.870 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2125101629999999 2024-02-09T18:06:59.870 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.870 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.870 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:06:59.870 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:15.655085+0000", 2024-02-09T18:06:59.870 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0026460239999999999 2024-02-09T18:06:59.870 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:06:59.870 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:06:59.870 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:06:59.870 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.870 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.871 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26563.0:204342 9.7 9:ef4e912e:::benchmark_data_smithi179_161915_object69460:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e946)", 2024-02-09T18:06:59.871 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:03:17.504442+0000", 2024-02-09T18:06:59.871 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 222.31135722900001, 2024-02-09T18:06:59.871 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.41440982199999998, 2024-02-09T18:06:59.871 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:06:59.871 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:59.871 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:06:59.871 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26563", 2024-02-09T18:06:59.871 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:59.871 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 204342 2024-02-09T18:06:59.871 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.871 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:06:59.871 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.871 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:06:59.871 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.504442+0000", 2024-02-09T18:06:59.872 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:06:59.872 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.872 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.872 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:06:59.872 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.504442+0000", 2024-02-09T18:06:59.872 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:06:59.872 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.872 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.872 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:06:59.872 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.504443+0000", 2024-02-09T18:06:59.872 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0780000000000001e-06 2024-02-09T18:06:59.872 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.872 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.872 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:06:59.872 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.504448+0000", 2024-02-09T18:06:59.873 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.8149999999999996e-06 2024-02-09T18:06:59.873 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.873 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.873 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:06:59.873 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.504456+0000", 2024-02-09T18:06:59.873 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.2470000000000008e-06 2024-02-09T18:06:59.873 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.873 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.873 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:06:59.873 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.504461+0000", 2024-02-09T18:06:59.873 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.0019999999999997e-06 2024-02-09T18:06:59.873 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.873 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.873 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:06:59.874 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.504526+0000", 2024-02-09T18:06:59.874 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.5363000000000004e-05 2024-02-09T18:06:59.874 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.874 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.874 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:06:59.874 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.504637+0000", 2024-02-09T18:06:59.874 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000111202 2024-02-09T18:06:59.874 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.874 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.874 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 2,7,10", 2024-02-09T18:06:59.874 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.504759+0000", 2024-02-09T18:06:59.874 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000121278 2024-02-09T18:06:59.874 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.874 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.874 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:59.875 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.512509+0000", 2024-02-09T18:06:59.875 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0077498489999999996 2024-02-09T18:06:59.875 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.875 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.875 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:06:59.875 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.512525+0000", 2024-02-09T18:06:59.875 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.6084000000000001e-05 2024-02-09T18:06:59.875 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.875 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.875 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:59.875 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.513258+0000", 2024-02-09T18:06:59.875 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00073342399999999999 2024-02-09T18:06:59.875 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.875 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.875 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:59.876 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.918738+0000", 2024-02-09T18:06:59.876 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.40547958200000001 2024-02-09T18:06:59.876 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.876 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.876 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:06:59.876 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.918781+0000", 2024-02-09T18:06:59.876 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.3683000000000002e-05 2024-02-09T18:06:59.876 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.876 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.876 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:06:59.876 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.918852+0000", 2024-02-09T18:06:59.876 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.0215000000000005e-05 2024-02-09T18:06:59.876 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:06:59.876 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:06:59.876 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:06:59.877 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.877 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.877 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26563.0:204349 9.7 9:eab1c346:::benchmark_data_smithi179_161915_object69467:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e946)", 2024-02-09T18:06:59.877 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:03:17.521709+0000", 2024-02-09T18:06:59.877 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 222.29408994900001, 2024-02-09T18:06:59.877 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.39727006799999998, 2024-02-09T18:06:59.877 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:06:59.877 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:59.877 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:06:59.877 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26563", 2024-02-09T18:06:59.877 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:59.877 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 204349 2024-02-09T18:06:59.877 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.877 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:06:59.877 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.878 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:06:59.878 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.521709+0000", 2024-02-09T18:06:59.878 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:06:59.878 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.878 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.878 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:06:59.878 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.521709+0000", 2024-02-09T18:06:59.878 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:06:59.878 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.878 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.878 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:06:59.878 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.521710+0000", 2024-02-09T18:06:59.878 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3939999999999999e-06 2024-02-09T18:06:59.878 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.878 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.879 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:06:59.879 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.521716+0000", 2024-02-09T18:06:59.879 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6389999999999997e-06 2024-02-09T18:06:59.879 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.879 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.879 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:06:59.879 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.521725+0000", 2024-02-09T18:06:59.879 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.7110000000000001e-06 2024-02-09T18:06:59.879 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.879 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.879 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:06:59.879 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.521730+0000", 2024-02-09T18:06:59.879 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.7640000000000002e-06 2024-02-09T18:06:59.879 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.880 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.880 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:06:59.880 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.521770+0000", 2024-02-09T18:06:59.880 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9298999999999998e-05 2024-02-09T18:06:59.880 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.880 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.880 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:06:59.880 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.521861+0000", 2024-02-09T18:06:59.880 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.0948999999999999e-05 2024-02-09T18:06:59.880 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.880 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.880 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 2,7,10", 2024-02-09T18:06:59.880 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.521947+0000", 2024-02-09T18:06:59.880 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.6656999999999994e-05 2024-02-09T18:06:59.880 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.881 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.881 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:59.881 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.527044+0000", 2024-02-09T18:06:59.881 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0050968130000000004 2024-02-09T18:06:59.881 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.881 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.881 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:06:59.881 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.527054+0000", 2024-02-09T18:06:59.881 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.9680000000000001e-06 2024-02-09T18:06:59.881 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.881 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.881 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:59.881 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.534066+0000", 2024-02-09T18:06:59.881 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0070115519999999999 2024-02-09T18:06:59.881 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.881 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.882 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:59.882 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.918886+0000", 2024-02-09T18:06:59.882 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.38482064799999999 2024-02-09T18:06:59.882 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.882 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.882 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:06:59.882 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.918916+0000", 2024-02-09T18:06:59.882 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.9935999999999999e-05 2024-02-09T18:06:59.882 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.882 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.882 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:06:59.882 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.918979+0000", 2024-02-09T18:06:59.882 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.2737999999999995e-05 2024-02-09T18:06:59.882 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:06:59.882 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:06:59.883 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:06:59.883 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.883 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.883 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26563.0:204372 9.7 9:e3aa8f15:::benchmark_data_smithi179_161915_object69490:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e946)", 2024-02-09T18:06:59.883 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:03:17.561476+0000", 2024-02-09T18:06:59.883 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 222.254322836, 2024-02-09T18:06:59.883 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.35761279099999999, 2024-02-09T18:06:59.883 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:06:59.883 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:59.883 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:06:59.883 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26563", 2024-02-09T18:06:59.883 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:59.884 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 204372 2024-02-09T18:06:59.884 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.884 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:06:59.884 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.884 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:06:59.884 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.561476+0000", 2024-02-09T18:06:59.884 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:06:59.884 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.884 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.884 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:06:59.884 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.561476+0000", 2024-02-09T18:06:59.884 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:06:59.884 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.884 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.885 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:06:59.885 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.561478+0000", 2024-02-09T18:06:59.885 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.587e-06 2024-02-09T18:06:59.885 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.885 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.885 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:06:59.885 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.561483+0000", 2024-02-09T18:06:59.885 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.0520000000000001e-06 2024-02-09T18:06:59.885 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.885 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.885 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:06:59.885 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.561498+0000", 2024-02-09T18:06:59.885 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5226e-05 2024-02-09T18:06:59.885 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.885 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.886 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:06:59.886 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.561507+0000", 2024-02-09T18:06:59.886 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.02e-06 2024-02-09T18:06:59.886 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.886 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.886 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:06:59.886 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.561566+0000", 2024-02-09T18:06:59.886 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.9187000000000001e-05 2024-02-09T18:06:59.886 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.886 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.886 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:06:59.886 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.561706+0000", 2024-02-09T18:06:59.886 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014025999999999999 2024-02-09T18:06:59.886 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.886 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.887 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 2,7,10", 2024-02-09T18:06:59.887 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.561854+0000", 2024-02-09T18:06:59.887 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00014786899999999999 2024-02-09T18:06:59.887 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.887 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.887 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:59.887 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.570581+0000", 2024-02-09T18:06:59.887 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0087267419999999991 2024-02-09T18:06:59.887 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.887 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.887 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:06:59.887 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.570612+0000", 2024-02-09T18:06:59.887 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.1239999999999999e-05 2024-02-09T18:06:59.887 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.887 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.887 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:59.888 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.571153+0000", 2024-02-09T18:06:59.888 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00054108100000000001 2024-02-09T18:06:59.888 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.888 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.888 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:59.888 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.919001+0000", 2024-02-09T18:06:59.888 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.34784789399999999 2024-02-09T18:06:59.888 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.888 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.888 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:06:59.888 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.919030+0000", 2024-02-09T18:06:59.888 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.8575e-05 2024-02-09T18:06:59.888 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.888 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.889 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:06:59.889 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.919089+0000", 2024-02-09T18:06:59.889 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.9058000000000003e-05 2024-02-09T18:06:59.889 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:06:59.889 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:06:59.889 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:06:59.889 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.889 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.889 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26563.0:204374 9.7 9:eccec283:::benchmark_data_smithi179_161915_object69492:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e946)", 2024-02-09T18:06:59.889 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:03:17.577812+0000", 2024-02-09T18:06:59.889 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 222.237987076, 2024-02-09T18:06:59.889 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.34138302100000001, 2024-02-09T18:06:59.889 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:06:59.889 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:59.889 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:06:59.890 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26563", 2024-02-09T18:06:59.890 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:59.890 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 204374 2024-02-09T18:06:59.890 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.890 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:06:59.890 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.890 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:06:59.890 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.577812+0000", 2024-02-09T18:06:59.890 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:06:59.890 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.890 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.890 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:06:59.890 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.577812+0000", 2024-02-09T18:06:59.890 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:06:59.890 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.891 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.891 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:06:59.891 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.577813+0000", 2024-02-09T18:06:59.891 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.297e-06 2024-02-09T18:06:59.891 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.891 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.891 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:06:59.891 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.577818+0000", 2024-02-09T18:06:59.891 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.8149999999999996e-06 2024-02-09T18:06:59.891 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.891 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.891 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:06:59.891 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.577832+0000", 2024-02-09T18:06:59.891 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3699e-05 2024-02-09T18:06:59.891 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.892 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.892 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:06:59.892 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.577840+0000", 2024-02-09T18:06:59.892 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.0499999999999992e-06 2024-02-09T18:06:59.892 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.892 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.892 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:06:59.892 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.578858+0000", 2024-02-09T18:06:59.892 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.001018052 2024-02-09T18:06:59.892 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.892 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.892 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:06:59.892 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.578993+0000", 2024-02-09T18:06:59.892 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00013555699999999999 2024-02-09T18:06:59.892 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.893 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.893 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 2,7,10", 2024-02-09T18:06:59.893 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.579819+0000", 2024-02-09T18:06:59.893 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00082564400000000005 2024-02-09T18:06:59.893 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.893 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.893 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:59.893 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.585906+0000", 2024-02-09T18:06:59.893 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.006086724 2024-02-09T18:06:59.893 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.893 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.893 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:59.893 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.586057+0000", 2024-02-09T18:06:59.893 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00015102599999999999 2024-02-09T18:06:59.893 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.894 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.894 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:06:59.894 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.587704+0000", 2024-02-09T18:06:59.894 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0016472310000000001 2024-02-09T18:06:59.894 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.894 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.894 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:59.894 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.919112+0000", 2024-02-09T18:06:59.894 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.33140793600000001 2024-02-09T18:06:59.894 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.894 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.894 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:06:59.894 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.919140+0000", 2024-02-09T18:06:59.894 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.7693999999999999e-05 2024-02-09T18:06:59.894 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.895 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.895 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:06:59.895 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:03:17.919195+0000", 2024-02-09T18:06:59.895 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.5296e-05 2024-02-09T18:06:59.895 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:06:59.895 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:06:59.895 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:06:59.895 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.895 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.895 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26563.0:240778 9.2 9:453d68af:::benchmark_data_smithi179_161915_object105896:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e966)", 2024-02-09T18:06:59.895 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:04:07.651027+0000", 2024-02-09T18:06:59.895 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 172.16477241999999, 2024-02-09T18:06:59.895 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.21941550600000001, 2024-02-09T18:06:59.895 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:06:59.895 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:59.896 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:06:59.896 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26563", 2024-02-09T18:06:59.896 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:59.896 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 240778 2024-02-09T18:06:59.896 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.896 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:06:59.896 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.896 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:06:59.896 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.651027+0000", 2024-02-09T18:06:59.896 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:06:59.896 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.896 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.896 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:06:59.896 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.651027+0000", 2024-02-09T18:06:59.896 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:06:59.897 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.897 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.897 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:06:59.897 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.651028+0000", 2024-02-09T18:06:59.897 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5e-06 2024-02-09T18:06:59.897 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.897 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.897 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:06:59.897 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.651033+0000", 2024-02-09T18:06:59.897 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.1599999999999997e-06 2024-02-09T18:06:59.897 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.897 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.897 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:06:59.897 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.651049+0000", 2024-02-09T18:06:59.897 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.5519000000000002e-05 2024-02-09T18:06:59.898 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.898 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.898 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:06:59.898 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.651062+0000", 2024-02-09T18:06:59.898 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3026e-05 2024-02-09T18:06:59.898 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.898 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.898 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:06:59.898 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.651108+0000", 2024-02-09T18:06:59.898 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.6607999999999998e-05 2024-02-09T18:06:59.898 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.898 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.898 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:06:59.898 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.651230+0000", 2024-02-09T18:06:59.898 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000121259 2024-02-09T18:06:59.899 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.899 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.899 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,9", 2024-02-09T18:06:59.899 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.651350+0000", 2024-02-09T18:06:59.899 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000120906 2024-02-09T18:06:59.899 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.899 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.899 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:06:59.899 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.661800+0000", 2024-02-09T18:06:59.899 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.010449293 2024-02-09T18:06:59.899 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.899 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.899 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:59.899 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.668017+0000", 2024-02-09T18:06:59.899 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0062171809999999996 2024-02-09T18:06:59.900 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.900 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.900 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:59.900 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.870361+0000", 2024-02-09T18:06:59.900 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20234414100000001 2024-02-09T18:06:59.900 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.900 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.900 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:06:59.900 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.870411+0000", 2024-02-09T18:06:59.900 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.0164000000000003e-05 2024-02-09T18:06:59.900 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.900 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.900 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:06:59.900 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.870442+0000", 2024-02-09T18:06:59.900 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.0749e-05 2024-02-09T18:06:59.901 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:06:59.901 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:06:59.901 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:06:59.901 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.901 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.901 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26563.0:240781 9.2 9:4b238f3a:::benchmark_data_smithi179_161915_object105899:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e966)", 2024-02-09T18:06:59.901 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:04:07.654543+0000", 2024-02-09T18:06:59.901 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 172.161256345, 2024-02-09T18:06:59.901 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.21603850799999999, 2024-02-09T18:06:59.901 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:06:59.901 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:59.901 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:06:59.901 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26563", 2024-02-09T18:06:59.901 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:59.901 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 240781 2024-02-09T18:06:59.901 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.902 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:06:59.902 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.902 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:06:59.902 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.654543+0000", 2024-02-09T18:06:59.902 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:06:59.902 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.902 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.902 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:06:59.902 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.654543+0000", 2024-02-09T18:06:59.902 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:06:59.902 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.902 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.902 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:06:59.902 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.654544+0000", 2024-02-09T18:06:59.902 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.854e-06 2024-02-09T18:06:59.903 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.903 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.903 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:06:59.903 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.654550+0000", 2024-02-09T18:06:59.903 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.8869999999999997e-06 2024-02-09T18:06:59.903 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.903 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.903 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:06:59.903 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.654564+0000", 2024-02-09T18:06:59.903 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.3631999999999999e-05 2024-02-09T18:06:59.903 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.903 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.903 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:06:59.903 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.654573+0000", 2024-02-09T18:06:59.903 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 9.0510000000000002e-06 2024-02-09T18:06:59.904 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.904 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.904 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:06:59.904 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.656273+0000", 2024-02-09T18:06:59.904 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0017003210000000001 2024-02-09T18:06:59.904 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.904 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.904 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:06:59.904 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.656400+0000", 2024-02-09T18:06:59.904 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00012641599999999999 2024-02-09T18:06:59.904 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.904 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.904 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,9", 2024-02-09T18:06:59.904 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.656519+0000", 2024-02-09T18:06:59.904 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000119577 2024-02-09T18:06:59.905 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.905 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.905 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:06:59.905 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.661807+0000", 2024-02-09T18:06:59.905 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0052875730000000003 2024-02-09T18:06:59.905 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.905 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.905 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:59.905 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.668056+0000", 2024-02-09T18:06:59.905 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0062491150000000004 2024-02-09T18:06:59.905 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.905 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.905 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:59.905 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.870492+0000", 2024-02-09T18:06:59.905 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20243635200000001 2024-02-09T18:06:59.905 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.906 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.906 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:06:59.906 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.870548+0000", 2024-02-09T18:06:59.906 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.5278000000000002e-05 2024-02-09T18:06:59.906 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.906 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.906 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:06:59.906 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.870581+0000", 2024-02-09T18:06:59.906 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.3451999999999998e-05 2024-02-09T18:06:59.906 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:06:59.906 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:06:59.906 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:06:59.906 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.906 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.906 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26563.0:240788 9.c 9:320f0ae1:::benchmark_data_smithi179_161915_object105906:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e966)", 2024-02-09T18:06:59.906 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:04:07.661970+0000", 2024-02-09T18:06:59.907 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 172.15382900700001, 2024-02-09T18:06:59.907 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20853450600000001, 2024-02-09T18:06:59.907 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:06:59.907 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:59.907 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:06:59.907 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26563", 2024-02-09T18:06:59.907 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:59.907 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 240788 2024-02-09T18:06:59.907 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.907 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:06:59.907 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.907 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:06:59.907 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.661970+0000", 2024-02-09T18:06:59.907 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:06:59.907 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.907 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.908 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:06:59.908 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.661970+0000", 2024-02-09T18:06:59.908 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:06:59.908 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.908 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.908 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:06:59.908 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.661971+0000", 2024-02-09T18:06:59.908 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2300000000000001e-06 2024-02-09T18:06:59.908 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.908 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.908 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:06:59.908 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.661977+0000", 2024-02-09T18:06:59.908 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3759999999999999e-06 2024-02-09T18:06:59.908 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.909 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.909 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:06:59.909 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.661993+0000", 2024-02-09T18:06:59.909 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.6248000000000001e-05 2024-02-09T18:06:59.909 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.909 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.909 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:06:59.909 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.662001+0000", 2024-02-09T18:06:59.909 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.5830000000000007e-06 2024-02-09T18:06:59.909 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.909 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.909 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:06:59.909 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.662062+0000", 2024-02-09T18:06:59.909 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.1049000000000005e-05 2024-02-09T18:06:59.909 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.910 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.910 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:06:59.910 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.662144+0000", 2024-02-09T18:06:59.910 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 8.1519000000000003e-05 2024-02-09T18:06:59.910 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.910 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.910 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 4,9", 2024-02-09T18:06:59.910 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.662222+0000", 2024-02-09T18:06:59.910 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.7638000000000004e-05 2024-02-09T18:06:59.910 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.910 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.910 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:06:59.910 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.667540+0000", 2024-02-09T18:06:59.910 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0053188649999999999 2024-02-09T18:06:59.911 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.911 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.911 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:59.911 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.667937+0000", 2024-02-09T18:06:59.911 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00039689 2024-02-09T18:06:59.911 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.911 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.911 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:59.911 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.870407+0000", 2024-02-09T18:06:59.911 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.202469701 2024-02-09T18:06:59.911 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.911 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.911 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:06:59.911 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.870464+0000", 2024-02-09T18:06:59.911 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6691000000000003e-05 2024-02-09T18:06:59.912 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.912 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.912 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:06:59.912 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.870504+0000", 2024-02-09T18:06:59.912 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.0716000000000001e-05 2024-02-09T18:06:59.912 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:06:59.912 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:06:59.912 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:06:59.912 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.912 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.912 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26563.0:240795 9.2 9:4e8f1d6c:::benchmark_data_smithi179_161915_object105913:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e966)", 2024-02-09T18:06:59.912 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:04:07.668971+0000", 2024-02-09T18:06:59.912 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 172.14682776800001, 2024-02-09T18:06:59.912 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.219270193, 2024-02-09T18:06:59.913 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:06:59.913 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:59.913 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:06:59.913 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26563", 2024-02-09T18:06:59.913 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:59.913 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 240795 2024-02-09T18:06:59.913 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.913 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:06:59.913 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.913 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:06:59.913 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.668971+0000", 2024-02-09T18:06:59.913 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:06:59.913 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.913 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.914 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:06:59.914 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.668971+0000", 2024-02-09T18:06:59.914 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:06:59.914 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.914 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.914 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:06:59.914 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.668972+0000", 2024-02-09T18:06:59.914 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1480000000000001e-06 2024-02-09T18:06:59.914 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.914 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.914 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:06:59.914 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.668976+0000", 2024-02-09T18:06:59.914 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.031e-06 2024-02-09T18:06:59.914 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.914 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.914 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:06:59.915 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.668987+0000", 2024-02-09T18:06:59.915 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.0519e-05 2024-02-09T18:06:59.915 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.915 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.915 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:06:59.915 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.668992+0000", 2024-02-09T18:06:59.915 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.5570000000000003e-06 2024-02-09T18:06:59.915 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.915 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.915 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:06:59.915 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.669037+0000", 2024-02-09T18:06:59.915 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.4190999999999998e-05 2024-02-09T18:06:59.915 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.915 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.915 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:06:59.916 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.669154+0000", 2024-02-09T18:06:59.916 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000116996 2024-02-09T18:06:59.916 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.916 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.916 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,9", 2024-02-09T18:06:59.916 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.669486+0000", 2024-02-09T18:06:59.916 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00033244399999999999 2024-02-09T18:06:59.916 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.916 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.916 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:06:59.916 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.674520+0000", 2024-02-09T18:06:59.916 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0050341040000000002 2024-02-09T18:06:59.916 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.916 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.917 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:59.917 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.680295+0000", 2024-02-09T18:06:59.917 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0057752890000000003 2024-02-09T18:06:59.917 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.917 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.917 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:59.917 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.888145+0000", 2024-02-09T18:06:59.917 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.207849907 2024-02-09T18:06:59.917 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.917 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.917 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:06:59.917 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.888202+0000", 2024-02-09T18:06:59.917 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.66e-05 2024-02-09T18:06:59.917 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.917 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.918 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:06:59.918 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.888241+0000", 2024-02-09T18:06:59.918 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.9406999999999997e-05 2024-02-09T18:06:59.918 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:06:59.918 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:06:59.918 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:06:59.918 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.918 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.918 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26563.0:240801 9.2 9:45ea3801:::benchmark_data_smithi179_161915_object105919:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e966)", 2024-02-09T18:06:59.918 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:04:07.675002+0000", 2024-02-09T18:06:59.918 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 172.14079725900001, 2024-02-09T18:06:59.918 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.21339767900000001, 2024-02-09T18:06:59.918 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:06:59.918 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:59.918 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:06:59.918 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26563", 2024-02-09T18:06:59.919 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:59.919 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 240801 2024-02-09T18:06:59.919 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.919 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:06:59.919 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.919 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:06:59.919 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.675002+0000", 2024-02-09T18:06:59.919 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:06:59.919 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.919 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.919 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:06:59.919 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.675002+0000", 2024-02-09T18:06:59.919 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:06:59.919 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.919 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.919 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:06:59.920 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.675004+0000", 2024-02-09T18:06:59.920 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.8419999999999999e-06 2024-02-09T18:06:59.920 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.920 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.920 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:06:59.920 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.675009+0000", 2024-02-09T18:06:59.920 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.2159999999999997e-06 2024-02-09T18:06:59.920 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.920 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.920 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:06:59.920 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.675021+0000", 2024-02-09T18:06:59.920 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.1943000000000001e-05 2024-02-09T18:06:59.920 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.920 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.921 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:06:59.921 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.675028+0000", 2024-02-09T18:06:59.921 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.695e-06 2024-02-09T18:06:59.921 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.921 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.921 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:06:59.921 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.675082+0000", 2024-02-09T18:06:59.921 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.3686000000000001e-05 2024-02-09T18:06:59.921 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.921 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.921 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:06:59.921 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.675195+0000", 2024-02-09T18:06:59.921 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000112451 2024-02-09T18:06:59.921 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.921 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.921 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 1,9", 2024-02-09T18:06:59.922 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.675303+0000", 2024-02-09T18:06:59.922 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000108063 2024-02-09T18:06:59.922 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.922 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.922 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:06:59.922 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.680484+0000", 2024-02-09T18:06:59.922 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0051811959999999999 2024-02-09T18:06:59.922 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.922 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.922 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:59.922 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.680971+0000", 2024-02-09T18:06:59.922 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000486909 2024-02-09T18:06:59.922 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.922 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.922 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:59.923 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.888299+0000", 2024-02-09T18:06:59.923 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20732788399999999 2024-02-09T18:06:59.923 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.923 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.923 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:06:59.923 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.888354+0000", 2024-02-09T18:06:59.923 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.5714999999999998e-05 2024-02-09T18:06:59.923 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.923 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.923 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:06:59.923 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:07.888399+0000", 2024-02-09T18:06:59.923 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.5079e-05 2024-02-09T18:06:59.923 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:06:59.923 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:06:59.923 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:06:59.924 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.924 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.924 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26563.0:241999 9.c 9:37a9bc32:::benchmark_data_smithi179_161915_object107117:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e966)", 2024-02-09T18:06:59.924 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:04:09.144265+0000", 2024-02-09T18:06:59.924 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 170.67153430900001, 2024-02-09T18:06:59.924 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.21141717600000001, 2024-02-09T18:06:59.924 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:06:59.924 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:59.924 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:06:59.924 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26563", 2024-02-09T18:06:59.924 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:59.924 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 241999 2024-02-09T18:06:59.924 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.924 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:06:59.924 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.925 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:06:59.925 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:09.144265+0000", 2024-02-09T18:06:59.925 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:06:59.925 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.925 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.925 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:06:59.925 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:09.144265+0000", 2024-02-09T18:06:59.925 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:06:59.925 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.925 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.925 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:06:59.925 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:09.144266+0000", 2024-02-09T18:06:59.925 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.116e-06 2024-02-09T18:06:59.925 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.925 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.926 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:06:59.926 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:09.144270+0000", 2024-02-09T18:06:59.926 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 4.7330000000000003e-06 2024-02-09T18:06:59.926 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.926 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.926 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:06:59.926 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:09.144285+0000", 2024-02-09T18:06:59.926 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4268e-05 2024-02-09T18:06:59.926 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.926 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.926 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:06:59.926 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:09.144292+0000", 2024-02-09T18:06:59.926 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.7209999999999994e-06 2024-02-09T18:06:59.926 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.926 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.926 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:06:59.927 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:09.144356+0000", 2024-02-09T18:06:59.927 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 6.3849000000000005e-05 2024-02-09T18:06:59.927 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.927 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.927 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:06:59.927 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:09.144484+0000", 2024-02-09T18:06:59.927 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000128035 2024-02-09T18:06:59.927 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.927 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.927 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 4,9", 2024-02-09T18:06:59.927 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:09.144612+0000", 2024-02-09T18:06:59.927 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000127945 2024-02-09T18:06:59.927 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.927 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.927 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:06:59.928 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:09.149210+0000", 2024-02-09T18:06:59.928 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.0045976439999999997 2024-02-09T18:06:59.928 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.928 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.928 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:59.928 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:09.149579+0000", 2024-02-09T18:06:59.928 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00036889099999999999 2024-02-09T18:06:59.928 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.928 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.928 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:59.928 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:09.355620+0000", 2024-02-09T18:06:59.928 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.206040746 2024-02-09T18:06:59.928 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.928 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.928 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:06:59.929 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:09.355657+0000", 2024-02-09T18:06:59.929 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.7292000000000001e-05 2024-02-09T18:06:59.929 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.929 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.929 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:06:59.929 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:09.355682+0000", 2024-02-09T18:06:59.929 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 2.4936e-05 2024-02-09T18:06:59.929 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:06:59.929 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:06:59.929 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:06:59.929 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.929 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.929 INFO:teuthology.orchestra.run.smithi099.stdout: "description": "osd_op(client.26563.0:242013 9.c 9:3c937cef:::benchmark_data_smithi179_161915_object107131:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e966)", 2024-02-09T18:06:59.929 INFO:teuthology.orchestra.run.smithi099.stdout: "initiated_at": "2024-02-09T18:04:09.158384+0000", 2024-02-09T18:06:59.929 INFO:teuthology.orchestra.run.smithi099.stdout: "age": 170.657415415, 2024-02-09T18:06:59.930 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.21036733099999999, 2024-02-09T18:06:59.930 INFO:teuthology.orchestra.run.smithi099.stdout: "type_data": { 2024-02-09T18:06:59.930 INFO:teuthology.orchestra.run.smithi099.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-09T18:06:59.930 INFO:teuthology.orchestra.run.smithi099.stdout: "client_info": { 2024-02-09T18:06:59.930 INFO:teuthology.orchestra.run.smithi099.stdout: "client": "client.26563", 2024-02-09T18:06:59.930 INFO:teuthology.orchestra.run.smithi099.stdout: "client_addr": "172.21.15.179:0/3563010471", 2024-02-09T18:06:59.930 INFO:teuthology.orchestra.run.smithi099.stdout: "tid": 242013 2024-02-09T18:06:59.930 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.930 INFO:teuthology.orchestra.run.smithi099.stdout: "events": [ 2024-02-09T18:06:59.930 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.930 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "initiated", 2024-02-09T18:06:59.930 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:09.158384+0000", 2024-02-09T18:06:59.930 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:06:59.930 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.930 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.930 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "header_read", 2024-02-09T18:06:59.931 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:09.158384+0000", 2024-02-09T18:06:59.931 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0 2024-02-09T18:06:59.931 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.931 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.931 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "throttled", 2024-02-09T18:06:59.931 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:09.158385+0000", 2024-02-09T18:06:59.931 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.409e-06 2024-02-09T18:06:59.931 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.931 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.931 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "all_read", 2024-02-09T18:06:59.931 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:09.158399+0000", 2024-02-09T18:06:59.931 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.4531e-05 2024-02-09T18:06:59.931 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.931 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.931 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "dispatched", 2024-02-09T18:06:59.932 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:09.158412+0000", 2024-02-09T18:06:59.932 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 1.2967e-05 2024-02-09T18:06:59.932 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.932 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.932 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "queued_for_pg", 2024-02-09T18:06:59.932 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:09.158420+0000", 2024-02-09T18:06:59.932 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 7.1849999999999998e-06 2024-02-09T18:06:59.932 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.932 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.932 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "reached_pg", 2024-02-09T18:06:59.932 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:09.158476+0000", 2024-02-09T18:06:59.932 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 5.6671e-05 2024-02-09T18:06:59.932 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.932 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.932 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "started", 2024-02-09T18:06:59.932 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:09.158590+0000", 2024-02-09T18:06:59.933 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.00011349300000000001 2024-02-09T18:06:59.933 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.933 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.933 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "waiting for subops from 4,9", 2024-02-09T18:06:59.933 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:09.158714+0000", 2024-02-09T18:06:59.933 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.000124702 2024-02-09T18:06:59.933 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.933 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.933 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:59.933 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:09.163897+0000", 2024-02-09T18:06:59.933 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.005182786 2024-02-09T18:06:59.933 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.933 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.933 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "op_commit", 2024-02-09T18:06:59.933 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:09.168104+0000", 2024-02-09T18:06:59.933 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.004206937 2024-02-09T18:06:59.934 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.934 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.934 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "sub_op_commit_rec", 2024-02-09T18:06:59.934 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:09.368682+0000", 2024-02-09T18:06:59.934 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 0.20057767700000001 2024-02-09T18:06:59.934 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.934 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.934 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "commit_sent", 2024-02-09T18:06:59.934 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:09.368717+0000", 2024-02-09T18:06:59.934 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.5247000000000003e-05 2024-02-09T18:06:59.934 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-02-09T18:06:59.934 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-02-09T18:06:59.934 INFO:teuthology.orchestra.run.smithi099.stdout: "event": "done", 2024-02-09T18:06:59.934 INFO:teuthology.orchestra.run.smithi099.stdout: "time": "2024-02-09T18:04:09.368751+0000", 2024-02-09T18:06:59.934 INFO:teuthology.orchestra.run.smithi099.stdout: "duration": 3.3726000000000002e-05 2024-02-09T18:06:59.935 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:06:59.935 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:06:59.935 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:06:59.935 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-02-09T18:06:59.935 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-02-09T18:06:59.935 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-02-09T18:07:00.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:00 smithi099 ceph-mon[39420]: 1.10 scrub starts 2024-02-09T18:07:00.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:00 smithi099 ceph-mon[39420]: 1.10 scrub ok 2024-02-09T18:07:00.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:00 smithi099 ceph-mon[39420]: pgmap v2542: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:00.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:00 smithi179 ceph-mon[43975]: 1.10 scrub starts 2024-02-09T18:07:00.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:00 smithi179 ceph-mon[43975]: 1.10 scrub ok 2024-02-09T18:07:00.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:00 smithi179 ceph-mon[43975]: pgmap v2542: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:00.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:00 smithi162 ceph-mon[44304]: 1.10 scrub starts 2024-02-09T18:07:00.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:00 smithi162 ceph-mon[44304]: 1.10 scrub ok 2024-02-09T18:07:00.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:00 smithi162 ceph-mon[44304]: pgmap v2542: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:00.796 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-02-09T18:07:00.796 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-02-09T18:07:00.797 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd dump --format=json 2024-02-09T18:07:01.064 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:07:01.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:01 smithi099 ceph-mon[39420]: 1.1a scrub starts 2024-02-09T18:07:01.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:01 smithi099 ceph-mon[39420]: 1.1a scrub ok 2024-02-09T18:07:01.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:01 smithi179 ceph-mon[43975]: 1.1a scrub starts 2024-02-09T18:07:01.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:01 smithi179 ceph-mon[43975]: 1.1a scrub ok 2024-02-09T18:07:01.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:01 smithi162 ceph-mon[44304]: 1.1a scrub starts 2024-02-09T18:07:01.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:01 smithi162 ceph-mon[44304]: 1.1a scrub ok 2024-02-09T18:07:02.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:02 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:07:02.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:02 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:07:02.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:02 smithi099 ceph-mon[39420]: osdmap e1036: 12 total, 12 up, 12 in 2024-02-09T18:07:02.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:02 smithi099 ceph-mon[39420]: pgmap v2544: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:02.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:02 smithi099 ceph-mon[39420]: osdmap e1037: 12 total, 12 up, 12 in 2024-02-09T18:07:02.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:02 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:07:02.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:02 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:07:02.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:02 smithi179 ceph-mon[43975]: osdmap e1036: 12 total, 12 up, 12 in 2024-02-09T18:07:02.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:02 smithi179 ceph-mon[43975]: pgmap v2544: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:02.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:02 smithi179 ceph-mon[43975]: osdmap e1037: 12 total, 12 up, 12 in 2024-02-09T18:07:02.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:02 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:07:02.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:02 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:07:02.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:02 smithi162 ceph-mon[44304]: osdmap e1036: 12 total, 12 up, 12 in 2024-02-09T18:07:02.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:02 smithi162 ceph-mon[44304]: pgmap v2544: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:02.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:02 smithi162 ceph-mon[44304]: osdmap e1037: 12 total, 12 up, 12 in 2024-02-09T18:07:03.027 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T18:07:03.027 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":1037,"fsid":"a69c870a-c76e-11ee-95b7-87774f69a715","created":"2024-02-09T17:15:26.113606+0000","modified":"2024-02-09T18:07:02.163530+0000","last_up_change":"2024-02-09T17:59:11.889493+0000","last_in_change":"2024-02-09T18:05:10.193989+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-09T17:18:56.552286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"544","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"514","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.940000057220459,"score_stable":1.940000057220459,"optimal_score":1,"raw_score_acting":1.940000057220459,"raw_score_stable":1.940000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"21739aa7-e278-45da-9a3f-867385f01baf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":998,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":3859868923},{"type":"v1","addr":"172.21.15.99:6803","nonce":3859868923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":3859868923},{"type":"v1","addr":"172.21.15.99:6805","nonce":3859868923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":3859868923},{"type":"v1","addr":"172.21.15.99:6809","nonce":3859868923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":3859868923},{"type":"v1","addr":"172.21.15.99:6807","nonce":3859868923}]},"public_addr":"172.21.15.99:6803/3859868923","cluster_addr":"172.21.15.99:6805/3859868923","heartbeat_back_addr":"172.21.15.99:6809/3859868923","heartbeat_front_addr":"172.21.15.99:6807/3859868923","state":["exists","up"]},{"osd":1,"uuid":"8eab347c-9065-4de1-93ae-e2eb7e11cab7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":993,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":3868328452},{"type":"v1","addr":"172.21.15.99:6811","nonce":3868328452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":3868328452},{"type":"v1","addr":"172.21.15.99:6813","nonce":3868328452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":3868328452},{"type":"v1","addr":"172.21.15.99:6817","nonce":3868328452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":3868328452},{"type":"v1","addr":"172.21.15.99:6815","nonce":3868328452}]},"public_addr":"172.21.15.99:6811/3868328452","cluster_addr":"172.21.15.99:6813/3868328452","heartbeat_back_addr":"172.21.15.99:6817/3868328452","heartbeat_front_addr":"172.21.15.99:6815/3868328452","state":["exists","up"]},{"osd":2,"uuid":"575f4257-7e34-485e-963c-2607b69e1220","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":1005,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":537506237},{"type":"v1","addr":"172.21.15.99:6819","nonce":537506237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":537506237},{"type":"v1","addr":"172.21.15.99:6821","nonce":537506237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":537506237},{"type":"v1","addr":"172.21.15.99:6825","nonce":537506237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":537506237},{"type":"v1","addr":"172.21.15.99:6823","nonce":537506237}]},"public_addr":"172.21.15.99:6819/537506237","cluster_addr":"172.21.15.99:6821/537506237","heartbeat_back_addr":"172.21.15.99:6825/537506237","heartbeat_front_addr":"172.21.15.99:6823/537506237","state":["exists","up"]},{"osd":3,"uuid":"1a520bcf-81bb-4f1d-80fb-11e98008e429","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":27,"last_clean_end":814,"up_from":854,"up_thru":970,"down_at":815,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":4191781533},{"type":"v1","addr":"172.21.15.99:6827","nonce":4191781533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":4191781533},{"type":"v1","addr":"172.21.15.99:6829","nonce":4191781533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":4191781533},{"type":"v1","addr":"172.21.15.99:6833","nonce":4191781533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":4191781533},{"type":"v1","addr":"172.21.15.99:6831","nonce":4191781533}]},"public_addr":"172.21.15.99:6827/4191781533","cluster_addr":"172.21.15.99:6829/4191781533","heartbeat_back_addr":"172.21.15.99:6833/4191781533","heartbeat_front_addr":"172.21.15.99:6831/4191781533","state":["exists","up"]},{"osd":4,"uuid":"4617c2fe-c9d8-40cd-887b-303e3215bdf8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":992,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":4178676504},{"type":"v1","addr":"172.21.15.162:6801","nonce":4178676504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":4178676504},{"type":"v1","addr":"172.21.15.162:6803","nonce":4178676504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":4178676504},{"type":"v1","addr":"172.21.15.162:6807","nonce":4178676504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":4178676504},{"type":"v1","addr":"172.21.15.162:6805","nonce":4178676504}]},"public_addr":"172.21.15.162:6801/4178676504","cluster_addr":"172.21.15.162:6803/4178676504","heartbeat_back_addr":"172.21.15.162:6807/4178676504","heartbeat_front_addr":"172.21.15.162:6805/4178676504","state":["exists","up"]},{"osd":5,"uuid":"da98e3f3-173e-46fe-be8b-1ccbe8b0dabc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":1013,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1489371339},{"type":"v1","addr":"172.21.15.162:6809","nonce":1489371339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1489371339},{"type":"v1","addr":"172.21.15.162:6811","nonce":1489371339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1489371339},{"type":"v1","addr":"172.21.15.162:6815","nonce":1489371339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1489371339},{"type":"v1","addr":"172.21.15.162:6813","nonce":1489371339}]},"public_addr":"172.21.15.162:6809/1489371339","cluster_addr":"172.21.15.162:6811/1489371339","heartbeat_back_addr":"172.21.15.162:6815/1489371339","heartbeat_front_addr":"172.21.15.162:6813/1489371339","state":["exists","up"]},{"osd":6,"uuid":"ff50382d-5ddf-4cda-bb0c-ff055feeaa40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":982,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":821169663},{"type":"v1","addr":"172.21.15.162:6817","nonce":821169663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":821169663},{"type":"v1","addr":"172.21.15.162:6819","nonce":821169663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":821169663},{"type":"v1","addr":"172.21.15.162:6823","nonce":821169663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":821169663},{"type":"v1","addr":"172.21.15.162:6821","nonce":821169663}]},"public_addr":"172.21.15.162:6817/821169663","cluster_addr":"172.21.15.162:6819/821169663","heartbeat_back_addr":"172.21.15.162:6823/821169663","heartbeat_front_addr":"172.21.15.162:6821/821169663","state":["exists","up"]},{"osd":7,"uuid":"a82984dc-5e22-424b-a446-ae2ee71eee19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":1019,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":965264369},{"type":"v1","addr":"172.21.15.162:6825","nonce":965264369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":965264369},{"type":"v1","addr":"172.21.15.162:6827","nonce":965264369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":965264369},{"type":"v1","addr":"172.21.15.162:6831","nonce":965264369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":965264369},{"type":"v1","addr":"172.21.15.162:6829","nonce":965264369}]},"public_addr":"172.21.15.162:6825/965264369","cluster_addr":"172.21.15.162:6827/965264369","heartbeat_back_addr":"172.21.15.162:6831/965264369","heartbeat_front_addr":"172.21.15.162:6829/965264369","state":["exists","up"]},{"osd":8,"uuid":"63f9d79f-0400-44dc-b388-df00fefb5820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":66,"last_clean_end":145,"up_from":171,"up_thru":993,"down_at":146,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2558294495},{"type":"v1","addr":"172.21.15.179:6801","nonce":2558294495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2558294495},{"type":"v1","addr":"172.21.15.179:6803","nonce":2558294495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2558294495},{"type":"v1","addr":"172.21.15.179:6807","nonce":2558294495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2558294495},{"type":"v1","addr":"172.21.15.179:6805","nonce":2558294495}]},"public_addr":"172.21.15.179:6801/2558294495","cluster_addr":"172.21.15.179:6803/2558294495","heartbeat_back_addr":"172.21.15.179:6807/2558294495","heartbeat_front_addr":"172.21.15.179:6805/2558294495","state":["exists","up"]},{"osd":9,"uuid":"7d995854-7328-4dfc-b1a9-0edca8d74f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":945,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":504427183},{"type":"v1","addr":"172.21.15.179:6809","nonce":504427183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":504427183},{"type":"v1","addr":"172.21.15.179:6811","nonce":504427183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":504427183},{"type":"v1","addr":"172.21.15.179:6815","nonce":504427183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":504427183},{"type":"v1","addr":"172.21.15.179:6813","nonce":504427183}]},"public_addr":"172.21.15.179:6809/504427183","cluster_addr":"172.21.15.179:6811/504427183","heartbeat_back_addr":"172.21.15.179:6815/504427183","heartbeat_front_addr":"172.21.15.179:6813/504427183","state":["exists","up"]},{"osd":10,"uuid":"151fbedd-08e6-46e6-a4e5-a14248399f61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":993,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":2403345164},{"type":"v1","addr":"172.21.15.179:6817","nonce":2403345164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":2403345164},{"type":"v1","addr":"172.21.15.179:6819","nonce":2403345164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":2403345164},{"type":"v1","addr":"172.21.15.179:6823","nonce":2403345164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":2403345164},{"type":"v1","addr":"172.21.15.179:6821","nonce":2403345164}]},"public_addr":"172.21.15.179:6817/2403345164","cluster_addr":"172.21.15.179:6819/2403345164","heartbeat_back_addr":"172.21.15.179:6823/2403345164","heartbeat_front_addr":"172.21.15.179:6821/2403345164","state":["exists","up"]},{"osd":11,"uuid":"6d3a3ad8-417a-483f-991c-52549dbe7d73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":90,"up_thru":961,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2899402373},{"type":"v1","addr":"172.21.15.179:6825","nonce":2899402373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2899402373},{"type":"v1","addr":"172.21.15.179:6827","nonce":2899402373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2899402373},{"type":"v1","addr":"172.21.15.179:6831","nonce":2899402373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2899402373},{"type":"v1","addr":"172.21.15.179:6829","nonce":2899402373}]},"public_addr":"172.21.15.179:6825/2899402373","cluster_addr":"172.21.15.179:6827/2899402373","heartbeat_back_addr":"172.21.15.179:6831/2899402373","heartbeat_front_addr":"172.21.15.179:6829/2899402373","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T18:06:04.022574+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T18:07:00.659032+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T18:06:27.748132+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T18:05:59.396849+0000","dead_epoch":814},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T18:06:01.425134+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T18:06:01.397427+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T18:06:49.199556+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T18:06:50.875145+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T18:06:22.786122+0000","dead_epoch":145},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T18:06:51.217247+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T18:06:03.521991+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T18:05:51.597857+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"1.7","osds":[1,8,6]}],"pg_upmap_items":[{"pgid":"1.5","mappings":[{"from":6,"to":8}]},{"pgid":"1.19","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.99:0/3904172782":"2024-02-10T17:16:24.229365+0000","172.21.15.99:6801/1586034576":"2024-02-10T17:16:24.229365+0000","172.21.15.99:6800/746248609":"2024-02-10T17:15:54.399201+0000","172.21.15.99:0/4092784197":"2024-02-10T17:16:24.229365+0000","172.21.15.99:0/2844598101":"2024-02-10T17:15:54.399201+0000","172.21.15.99:6801/746248609":"2024-02-10T17:15:54.399201+0000","172.21.15.99:0/2528750050":"2024-02-10T17:15:54.399201+0000","172.21.15.99:6800/1586034576":"2024-02-10T17:16:24.229365+0000","172.21.15.99:0/2214140078":"2024-02-10T17:15:54.399201+0000","172.21.15.99:0/4217821965":"2024-02-10T17:16:24.229365+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-09T18:07:03.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:03 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1389675208' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T18:07:03.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:03 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1389675208' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T18:07:03.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:03 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1389675208' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T18:07:04.000 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-02-09T18:07:04.001 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd dump --format=json 2024-02-09T18:07:04.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:04 smithi099 ceph-mon[39420]: pgmap v2546: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:04.269 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:07:04.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:04 smithi179 ceph-mon[43975]: pgmap v2546: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:04.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:04 smithi162 ceph-mon[44304]: pgmap v2546: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:05.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:05 smithi099 ceph-mon[39420]: 1.12 scrub starts 2024-02-09T18:07:05.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:05 smithi099 ceph-mon[39420]: 1.16 scrub starts 2024-02-09T18:07:05.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:05 smithi099 ceph-mon[39420]: 1.16 scrub ok 2024-02-09T18:07:05.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:05 smithi099 ceph-mon[39420]: 1.12 scrub ok 2024-02-09T18:07:05.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:05 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:07:05.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:05 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:07:05.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:05 smithi179 ceph-mon[43975]: 1.12 scrub starts 2024-02-09T18:07:05.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:05 smithi179 ceph-mon[43975]: 1.16 scrub starts 2024-02-09T18:07:05.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:05 smithi179 ceph-mon[43975]: 1.16 scrub ok 2024-02-09T18:07:05.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:05 smithi179 ceph-mon[43975]: 1.12 scrub ok 2024-02-09T18:07:05.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:05 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:07:05.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:05 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:07:05.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:05 smithi162 ceph-mon[44304]: 1.12 scrub starts 2024-02-09T18:07:05.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:05 smithi162 ceph-mon[44304]: 1.16 scrub starts 2024-02-09T18:07:05.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:05 smithi162 ceph-mon[44304]: 1.16 scrub ok 2024-02-09T18:07:05.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:05 smithi162 ceph-mon[44304]: 1.12 scrub ok 2024-02-09T18:07:05.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:05 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:07:05.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:05 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:07:06.485 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T18:07:06.485 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":1038,"fsid":"a69c870a-c76e-11ee-95b7-87774f69a715","created":"2024-02-09T17:15:26.113606+0000","modified":"2024-02-09T18:07:05.197990+0000","last_up_change":"2024-02-09T17:59:11.889493+0000","last_in_change":"2024-02-09T18:05:10.193989+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-09T17:18:56.552286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"544","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"514","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.940000057220459,"score_stable":1.940000057220459,"optimal_score":1,"raw_score_acting":1.940000057220459,"raw_score_stable":1.940000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"21739aa7-e278-45da-9a3f-867385f01baf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":998,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":3859868923},{"type":"v1","addr":"172.21.15.99:6803","nonce":3859868923}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":3859868923},{"type":"v1","addr":"172.21.15.99:6805","nonce":3859868923}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":3859868923},{"type":"v1","addr":"172.21.15.99:6809","nonce":3859868923}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":3859868923},{"type":"v1","addr":"172.21.15.99:6807","nonce":3859868923}]},"public_addr":"172.21.15.99:6803/3859868923","cluster_addr":"172.21.15.99:6805/3859868923","heartbeat_back_addr":"172.21.15.99:6809/3859868923","heartbeat_front_addr":"172.21.15.99:6807/3859868923","state":["exists","up"]},{"osd":1,"uuid":"8eab347c-9065-4de1-93ae-e2eb7e11cab7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":993,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":3868328452},{"type":"v1","addr":"172.21.15.99:6811","nonce":3868328452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":3868328452},{"type":"v1","addr":"172.21.15.99:6813","nonce":3868328452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":3868328452},{"type":"v1","addr":"172.21.15.99:6817","nonce":3868328452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":3868328452},{"type":"v1","addr":"172.21.15.99:6815","nonce":3868328452}]},"public_addr":"172.21.15.99:6811/3868328452","cluster_addr":"172.21.15.99:6813/3868328452","heartbeat_back_addr":"172.21.15.99:6817/3868328452","heartbeat_front_addr":"172.21.15.99:6815/3868328452","state":["exists","up"]},{"osd":2,"uuid":"575f4257-7e34-485e-963c-2607b69e1220","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":1005,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":537506237},{"type":"v1","addr":"172.21.15.99:6819","nonce":537506237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":537506237},{"type":"v1","addr":"172.21.15.99:6821","nonce":537506237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":537506237},{"type":"v1","addr":"172.21.15.99:6825","nonce":537506237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":537506237},{"type":"v1","addr":"172.21.15.99:6823","nonce":537506237}]},"public_addr":"172.21.15.99:6819/537506237","cluster_addr":"172.21.15.99:6821/537506237","heartbeat_back_addr":"172.21.15.99:6825/537506237","heartbeat_front_addr":"172.21.15.99:6823/537506237","state":["exists","up"]},{"osd":3,"uuid":"1a520bcf-81bb-4f1d-80fb-11e98008e429","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":27,"last_clean_end":814,"up_from":854,"up_thru":970,"down_at":815,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":4191781533},{"type":"v1","addr":"172.21.15.99:6827","nonce":4191781533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":4191781533},{"type":"v1","addr":"172.21.15.99:6829","nonce":4191781533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":4191781533},{"type":"v1","addr":"172.21.15.99:6833","nonce":4191781533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":4191781533},{"type":"v1","addr":"172.21.15.99:6831","nonce":4191781533}]},"public_addr":"172.21.15.99:6827/4191781533","cluster_addr":"172.21.15.99:6829/4191781533","heartbeat_back_addr":"172.21.15.99:6833/4191781533","heartbeat_front_addr":"172.21.15.99:6831/4191781533","state":["exists","up"]},{"osd":4,"uuid":"4617c2fe-c9d8-40cd-887b-303e3215bdf8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":992,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":4178676504},{"type":"v1","addr":"172.21.15.162:6801","nonce":4178676504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":4178676504},{"type":"v1","addr":"172.21.15.162:6803","nonce":4178676504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":4178676504},{"type":"v1","addr":"172.21.15.162:6807","nonce":4178676504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":4178676504},{"type":"v1","addr":"172.21.15.162:6805","nonce":4178676504}]},"public_addr":"172.21.15.162:6801/4178676504","cluster_addr":"172.21.15.162:6803/4178676504","heartbeat_back_addr":"172.21.15.162:6807/4178676504","heartbeat_front_addr":"172.21.15.162:6805/4178676504","state":["exists","up"]},{"osd":5,"uuid":"da98e3f3-173e-46fe-be8b-1ccbe8b0dabc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":1013,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1489371339},{"type":"v1","addr":"172.21.15.162:6809","nonce":1489371339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1489371339},{"type":"v1","addr":"172.21.15.162:6811","nonce":1489371339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1489371339},{"type":"v1","addr":"172.21.15.162:6815","nonce":1489371339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1489371339},{"type":"v1","addr":"172.21.15.162:6813","nonce":1489371339}]},"public_addr":"172.21.15.162:6809/1489371339","cluster_addr":"172.21.15.162:6811/1489371339","heartbeat_back_addr":"172.21.15.162:6815/1489371339","heartbeat_front_addr":"172.21.15.162:6813/1489371339","state":["exists","up"]},{"osd":6,"uuid":"ff50382d-5ddf-4cda-bb0c-ff055feeaa40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":982,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":821169663},{"type":"v1","addr":"172.21.15.162:6817","nonce":821169663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":821169663},{"type":"v1","addr":"172.21.15.162:6819","nonce":821169663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":821169663},{"type":"v1","addr":"172.21.15.162:6823","nonce":821169663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":821169663},{"type":"v1","addr":"172.21.15.162:6821","nonce":821169663}]},"public_addr":"172.21.15.162:6817/821169663","cluster_addr":"172.21.15.162:6819/821169663","heartbeat_back_addr":"172.21.15.162:6823/821169663","heartbeat_front_addr":"172.21.15.162:6821/821169663","state":["exists","up"]},{"osd":7,"uuid":"a82984dc-5e22-424b-a446-ae2ee71eee19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":1019,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":965264369},{"type":"v1","addr":"172.21.15.162:6825","nonce":965264369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":965264369},{"type":"v1","addr":"172.21.15.162:6827","nonce":965264369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":965264369},{"type":"v1","addr":"172.21.15.162:6831","nonce":965264369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":965264369},{"type":"v1","addr":"172.21.15.162:6829","nonce":965264369}]},"public_addr":"172.21.15.162:6825/965264369","cluster_addr":"172.21.15.162:6827/965264369","heartbeat_back_addr":"172.21.15.162:6831/965264369","heartbeat_front_addr":"172.21.15.162:6829/965264369","state":["exists","up"]},{"osd":8,"uuid":"63f9d79f-0400-44dc-b388-df00fefb5820","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":66,"last_clean_end":145,"up_from":171,"up_thru":993,"down_at":146,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2558294495},{"type":"v1","addr":"172.21.15.179:6801","nonce":2558294495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2558294495},{"type":"v1","addr":"172.21.15.179:6803","nonce":2558294495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2558294495},{"type":"v1","addr":"172.21.15.179:6807","nonce":2558294495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2558294495},{"type":"v1","addr":"172.21.15.179:6805","nonce":2558294495}]},"public_addr":"172.21.15.179:6801/2558294495","cluster_addr":"172.21.15.179:6803/2558294495","heartbeat_back_addr":"172.21.15.179:6807/2558294495","heartbeat_front_addr":"172.21.15.179:6805/2558294495","state":["exists","up"]},{"osd":9,"uuid":"7d995854-7328-4dfc-b1a9-0edca8d74f46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":945,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":504427183},{"type":"v1","addr":"172.21.15.179:6809","nonce":504427183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":504427183},{"type":"v1","addr":"172.21.15.179:6811","nonce":504427183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":504427183},{"type":"v1","addr":"172.21.15.179:6815","nonce":504427183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":504427183},{"type":"v1","addr":"172.21.15.179:6813","nonce":504427183}]},"public_addr":"172.21.15.179:6809/504427183","cluster_addr":"172.21.15.179:6811/504427183","heartbeat_back_addr":"172.21.15.179:6815/504427183","heartbeat_front_addr":"172.21.15.179:6813/504427183","state":["exists","up"]},{"osd":10,"uuid":"151fbedd-08e6-46e6-a4e5-a14248399f61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":993,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":2403345164},{"type":"v1","addr":"172.21.15.179:6817","nonce":2403345164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":2403345164},{"type":"v1","addr":"172.21.15.179:6819","nonce":2403345164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":2403345164},{"type":"v1","addr":"172.21.15.179:6823","nonce":2403345164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":2403345164},{"type":"v1","addr":"172.21.15.179:6821","nonce":2403345164}]},"public_addr":"172.21.15.179:6817/2403345164","cluster_addr":"172.21.15.179:6819/2403345164","heartbeat_back_addr":"172.21.15.179:6823/2403345164","heartbeat_front_addr":"172.21.15.179:6821/2403345164","state":["exists","up"]},{"osd":11,"uuid":"6d3a3ad8-417a-483f-991c-52549dbe7d73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":90,"up_thru":961,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2899402373},{"type":"v1","addr":"172.21.15.179:6825","nonce":2899402373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2899402373},{"type":"v1","addr":"172.21.15.179:6827","nonce":2899402373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2899402373},{"type":"v1","addr":"172.21.15.179:6831","nonce":2899402373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2899402373},{"type":"v1","addr":"172.21.15.179:6829","nonce":2899402373}]},"public_addr":"172.21.15.179:6825/2899402373","cluster_addr":"172.21.15.179:6827/2899402373","heartbeat_back_addr":"172.21.15.179:6831/2899402373","heartbeat_front_addr":"172.21.15.179:6829/2899402373","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T18:06:04.022574+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T18:07:00.659032+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T18:06:27.748132+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T18:05:59.396849+0000","dead_epoch":814},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T18:06:01.425134+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T18:07:04.326783+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T18:06:49.199556+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T18:06:50.875145+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T18:06:22.786122+0000","dead_epoch":145},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T18:06:51.217247+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T18:06:03.521991+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T18:05:51.597857+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"1.7","osds":[1,8,6]}],"pg_upmap_items":[{"pgid":"1.5","mappings":[{"from":6,"to":8}]},{"pgid":"1.19","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.99:0/3904172782":"2024-02-10T17:16:24.229365+0000","172.21.15.99:6801/1586034576":"2024-02-10T17:16:24.229365+0000","172.21.15.99:6800/746248609":"2024-02-10T17:15:54.399201+0000","172.21.15.99:0/4092784197":"2024-02-10T17:16:24.229365+0000","172.21.15.99:0/2844598101":"2024-02-10T17:15:54.399201+0000","172.21.15.99:6801/746248609":"2024-02-10T17:15:54.399201+0000","172.21.15.99:0/2528750050":"2024-02-10T17:15:54.399201+0000","172.21.15.99:6800/1586034576":"2024-02-10T17:16:24.229365+0000","172.21.15.99:0/2214140078":"2024-02-10T17:15:54.399201+0000","172.21.15.99:0/4217821965":"2024-02-10T17:16:24.229365+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-09T18:07:06.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:06 smithi099 ceph-mon[39420]: osdmap e1038: 12 total, 12 up, 12 in 2024-02-09T18:07:06.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:06 smithi099 ceph-mon[39420]: pgmap v2548: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:06.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:06 smithi179 ceph-mon[43975]: osdmap e1038: 12 total, 12 up, 12 in 2024-02-09T18:07:06.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:06 smithi179 ceph-mon[43975]: pgmap v2548: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:06.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:06 smithi162 ceph-mon[44304]: osdmap e1038: 12 total, 12 up, 12 in 2024-02-09T18:07:06.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:06 smithi162 ceph-mon[44304]: pgmap v2548: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:07.254 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-02-09T18:07:07.255 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-02-09T18:07:07.255 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-02-09T18:07:07.255 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-02-09T18:07:07.256 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-02-09T18:07:07.256 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-02-09T18:07:07.256 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-02-09T18:07:07.257 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-02-09T18:07:07.257 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-02-09T18:07:07.258 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-02-09T18:07:07.258 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-02-09T18:07:07.259 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-02-09T18:07:07.502 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:07 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1999272591' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T18:07:07.502 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:07 smithi099 ceph-mon[39420]: osdmap e1039: 12 total, 12 up, 12 in 2024-02-09T18:07:07.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:07 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1999272591' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T18:07:07.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:07 smithi179 ceph-mon[43975]: osdmap e1039: 12 total, 12 up, 12 in 2024-02-09T18:07:07.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:07 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1999272591' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T18:07:07.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:07 smithi162 ceph-mon[44304]: osdmap e1039: 12 total, 12 up, 12 in 2024-02-09T18:07:07.982 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:07:07.987 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:07:08.000 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:07:08.001 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:07:08.274 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:07:08.282 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:07:08.283 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:07:08.285 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:07:08.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:08 smithi099 ceph-mon[39420]: pgmap v2550: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:08.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:08 smithi179 ceph-mon[43975]: pgmap v2550: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:08.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:08 smithi162 ceph-mon[44304]: pgmap v2550: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:09.544 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:07:09.545 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:07:09.545 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:07:09.545 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:07:10.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:10 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:07:10.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:10 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:07:10.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:10 smithi099 ceph-mon[39420]: osdmap e1040: 12 total, 12 up, 12 in 2024-02-09T18:07:10.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:10 smithi099 ceph-mon[39420]: pgmap v2552: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:10.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:10 smithi099 ceph-mon[39420]: 1.2 scrub starts 2024-02-09T18:07:10.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:10 smithi099 ceph-mon[39420]: 1.2 scrub ok 2024-02-09T18:07:10.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:10 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:07:10.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:10 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:07:10.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:10 smithi179 ceph-mon[43975]: osdmap e1040: 12 total, 12 up, 12 in 2024-02-09T18:07:10.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:10 smithi179 ceph-mon[43975]: pgmap v2552: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:10.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:10 smithi179 ceph-mon[43975]: 1.2 scrub starts 2024-02-09T18:07:10.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:10 smithi179 ceph-mon[43975]: 1.2 scrub ok 2024-02-09T18:07:10.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:10 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:07:10.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:10 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:07:10.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:10 smithi162 ceph-mon[44304]: osdmap e1040: 12 total, 12 up, 12 in 2024-02-09T18:07:10.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:10 smithi162 ceph-mon[44304]: pgmap v2552: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:10.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:10 smithi162 ceph-mon[44304]: 1.2 scrub starts 2024-02-09T18:07:10.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:10 smithi162 ceph-mon[44304]: 1.2 scrub ok 2024-02-09T18:07:11.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:11 smithi099 ceph-mon[39420]: 1.7 scrub starts 2024-02-09T18:07:11.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:11 smithi099 ceph-mon[39420]: 1.7 scrub ok 2024-02-09T18:07:11.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:11 smithi179 ceph-mon[43975]: 1.7 scrub starts 2024-02-09T18:07:11.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:11 smithi179 ceph-mon[43975]: 1.7 scrub ok 2024-02-09T18:07:11.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:11 smithi162 ceph-mon[44304]: 1.7 scrub starts 2024-02-09T18:07:11.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:11 smithi162 ceph-mon[44304]: 1.7 scrub ok 2024-02-09T18:07:12.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:12 smithi099 ceph-mon[39420]: 1.19 scrub starts 2024-02-09T18:07:12.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:12 smithi099 ceph-mon[39420]: 1.19 scrub ok 2024-02-09T18:07:12.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:12 smithi099 ceph-mon[39420]: pgmap v2553: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:12.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:12 smithi099 ceph-mon[39420]: osdmap e1041: 12 total, 12 up, 12 in 2024-02-09T18:07:12.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:12 smithi179 ceph-mon[43975]: 1.19 scrub starts 2024-02-09T18:07:12.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:12 smithi179 ceph-mon[43975]: 1.19 scrub ok 2024-02-09T18:07:12.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:12 smithi179 ceph-mon[43975]: pgmap v2553: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:12.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:12 smithi179 ceph-mon[43975]: osdmap e1041: 12 total, 12 up, 12 in 2024-02-09T18:07:12.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:12 smithi162 ceph-mon[44304]: 1.19 scrub starts 2024-02-09T18:07:12.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:12 smithi162 ceph-mon[44304]: 1.19 scrub ok 2024-02-09T18:07:12.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:12 smithi162 ceph-mon[44304]: pgmap v2553: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:12.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:12 smithi162 ceph-mon[44304]: osdmap e1041: 12 total, 12 up, 12 in 2024-02-09T18:07:13.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:13 smithi099 ceph-mon[39420]: 1.c scrub starts 2024-02-09T18:07:13.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:13 smithi179 ceph-mon[43975]: 1.c scrub starts 2024-02-09T18:07:13.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:13 smithi162 ceph-mon[44304]: 1.c scrub starts 2024-02-09T18:07:14.410 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:14 smithi099 ceph-mon[39420]: 1.c scrub ok 2024-02-09T18:07:14.410 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:14 smithi099 ceph-mon[39420]: pgmap v2555: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:14.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:14 smithi179 ceph-mon[43975]: 1.c scrub ok 2024-02-09T18:07:14.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:14 smithi179 ceph-mon[43975]: pgmap v2555: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:14.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:14 smithi162 ceph-mon[44304]: 1.c scrub ok 2024-02-09T18:07:14.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:14 smithi162 ceph-mon[44304]: pgmap v2555: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:15.350 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:15 smithi099 ceph-mon[39420]: 1.1c deep-scrub starts 2024-02-09T18:07:15.350 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:15 smithi099 ceph-mon[39420]: 1.1c deep-scrub ok 2024-02-09T18:07:15.351 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:15 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T18:07:15.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:15 smithi179 ceph-mon[43975]: 1.1c deep-scrub starts 2024-02-09T18:07:15.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:15 smithi179 ceph-mon[43975]: 1.1c deep-scrub ok 2024-02-09T18:07:15.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:15 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T18:07:15.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:15 smithi162 ceph-mon[44304]: 1.1c deep-scrub starts 2024-02-09T18:07:15.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:15 smithi162 ceph-mon[44304]: 1.1c deep-scrub ok 2024-02-09T18:07:15.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:15 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T18:07:16.563 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:16 smithi099 ceph-mon[39420]: pgmap v2556: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:16.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:16 smithi179 ceph-mon[43975]: pgmap v2556: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:16.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:16 smithi162 ceph-mon[44304]: pgmap v2556: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:18.522 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:18 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:07:18.522 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:18 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:07:18.522 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:18 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:07:18.522 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:18 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:07:18.522 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:18 smithi099 ceph-mon[39420]: osdmap e1042: 12 total, 12 up, 12 in 2024-02-09T18:07:18.523 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:18 smithi099 ceph-mon[39420]: pgmap v2558: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:18.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:18 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:07:18.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:18 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:07:18.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:18 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:07:18.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:18 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:07:18.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:18 smithi179 ceph-mon[43975]: osdmap e1042: 12 total, 12 up, 12 in 2024-02-09T18:07:18.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:18 smithi179 ceph-mon[43975]: pgmap v2558: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:18.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:18 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:07:18.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:18 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:07:18.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:18 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:07:18.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:18 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:07:18.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:18 smithi162 ceph-mon[44304]: osdmap e1042: 12 total, 12 up, 12 in 2024-02-09T18:07:18.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:18 smithi162 ceph-mon[44304]: pgmap v2558: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:19.493 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:19 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T18:07:19.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:19 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T18:07:19.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:19 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T18:07:19.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:19 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T18:07:19.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:19 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T18:07:19.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:19 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T18:07:19.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:19 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T18:07:19.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:19 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T18:07:19.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:19 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T18:07:20.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:20 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:07:20.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:20 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:07:20.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:20 smithi099 ceph-mon[39420]: pgmap v2559: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:20.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:20 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:07:20.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:20 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:07:20.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:20 smithi179 ceph-mon[43975]: pgmap v2559: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:20.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:20 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:07:20.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:20 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:07:20.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:20 smithi162 ceph-mon[44304]: pgmap v2559: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:21.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:21 smithi099 ceph-mon[39420]: osdmap e1043: 12 total, 12 up, 12 in 2024-02-09T18:07:21.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:21 smithi179 ceph-mon[43975]: osdmap e1043: 12 total, 12 up, 12 in 2024-02-09T18:07:21.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:21 smithi162 ceph-mon[44304]: osdmap e1043: 12 total, 12 up, 12 in 2024-02-09T18:07:22.480 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:22 smithi099 ceph-mon[39420]: pgmap v2561: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:22.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:22 smithi179 ceph-mon[43975]: pgmap v2561: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:22.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:22 smithi162 ceph-mon[44304]: pgmap v2561: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:23.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:23 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:07:23.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:23 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:07:23.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:23 smithi099 ceph-mon[39420]: osdmap e1044: 12 total, 12 up, 12 in 2024-02-09T18:07:23.526 INFO:teuthology.orchestra.run.smithi099.stdout:3667902070886 2024-02-09T18:07:23.527 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-02-09T18:07:23.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:23 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:07:23.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:23 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:07:23.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:23 smithi179 ceph-mon[43975]: osdmap e1044: 12 total, 12 up, 12 in 2024-02-09T18:07:23.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:23 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:07:23.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:23 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:07:23.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:23 smithi162 ceph-mon[44304]: osdmap e1044: 12 total, 12 up, 12 in 2024-02-09T18:07:24.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:24 smithi099 ceph-mon[39420]: pgmap v2563: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:24.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:24 smithi099 ceph-mon[39420]: 1.a scrub starts 2024-02-09T18:07:24.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:24 smithi099 ceph-mon[39420]: 1.a scrub ok 2024-02-09T18:07:24.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:24 smithi179 ceph-mon[43975]: pgmap v2563: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:24.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:24 smithi179 ceph-mon[43975]: 1.a scrub starts 2024-02-09T18:07:24.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:24 smithi179 ceph-mon[43975]: 1.a scrub ok 2024-02-09T18:07:24.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:24 smithi162 ceph-mon[44304]: pgmap v2563: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:24.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:24 smithi162 ceph-mon[44304]: 1.a scrub starts 2024-02-09T18:07:24.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:24 smithi162 ceph-mon[44304]: 1.a scrub ok 2024-02-09T18:07:26.142 INFO:teuthology.orchestra.run.smithi099.stdout:330712482343 2024-02-09T18:07:26.143 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-02-09T18:07:26.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:26 smithi099 ceph-mon[39420]: pgmap v2564: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:26.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:26 smithi179 ceph-mon[43975]: pgmap v2564: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:26.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:26 smithi162 ceph-mon[44304]: pgmap v2564: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:27.735 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:27 smithi099 ceph-mon[39420]: 1.14 scrub starts 2024-02-09T18:07:27.735 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:27 smithi099 ceph-mon[39420]: 1.14 scrub ok 2024-02-09T18:07:27.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:27 smithi179 ceph-mon[43975]: 1.14 scrub starts 2024-02-09T18:07:27.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:27 smithi179 ceph-mon[43975]: 1.14 scrub ok 2024-02-09T18:07:27.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:27 smithi162 ceph-mon[44304]: 1.14 scrub starts 2024-02-09T18:07:27.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:27 smithi162 ceph-mon[44304]: 1.14 scrub ok 2024-02-09T18:07:28.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:28 smithi099 ceph-mon[39420]: 1.17 scrub starts 2024-02-09T18:07:28.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:28 smithi099 ceph-mon[39420]: pgmap v2565: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:28.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:28 smithi099 ceph-mon[39420]: 1.17 scrub ok 2024-02-09T18:07:28.787 INFO:teuthology.orchestra.run.smithi099.stdout:210453398075 2024-02-09T18:07:28.788 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-02-09T18:07:28.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:28 smithi179 ceph-mon[43975]: 1.17 scrub starts 2024-02-09T18:07:28.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:28 smithi179 ceph-mon[43975]: pgmap v2565: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:28.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:28 smithi179 ceph-mon[43975]: 1.17 scrub ok 2024-02-09T18:07:28.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:28 smithi162 ceph-mon[44304]: 1.17 scrub starts 2024-02-09T18:07:28.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:28 smithi162 ceph-mon[44304]: pgmap v2565: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:28.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:28 smithi162 ceph-mon[44304]: 1.17 scrub ok 2024-02-09T18:07:29.530 INFO:teuthology.orchestra.run.smithi099.stdout:360777253415 2024-02-09T18:07:29.531 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-02-09T18:07:29.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: 1.0 scrub starts 2024-02-09T18:07:29.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: 1.0 scrub ok 2024-02-09T18:07:29.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=1120856390,config=2679774077,health=3897446751,kv=744579444}) 2024-02-09T18:07:29.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=1301102820,logm=3662421561}) 2024-02-09T18:07:29.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2670705399}) 2024-02-09T18:07:29.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1682127979}) 2024-02-09T18:07:29.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=403881442}) 2024-02-09T18:07:29.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=588881377}) 2024-02-09T18:07:29.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2695294666}) 2024-02-09T18:07:29.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2668322858}) 2024-02-09T18:07:29.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3342851147}) 2024-02-09T18:07:29.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3929975127}) 2024-02-09T18:07:29.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4008246186}) 2024-02-09T18:07:29.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1671367544}) 2024-02-09T18:07:29.745 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3728969341}) 2024-02-09T18:07:29.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3736510291}) 2024-02-09T18:07:29.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4003430310}) 2024-02-09T18:07:29.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1118891243}) 2024-02-09T18:07:29.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=914491748}) 2024-02-09T18:07:29.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2429726709}) 2024-02-09T18:07:29.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3445657699}) 2024-02-09T18:07:29.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=650119449}) 2024-02-09T18:07:29.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2101688677}) 2024-02-09T18:07:29.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=615213512}) 2024-02-09T18:07:29.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3150055218}) 2024-02-09T18:07:29.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1164083796}) 2024-02-09T18:07:29.746 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2658585886}) 2024-02-09T18:07:29.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1894006670}) 2024-02-09T18:07:29.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3773520614}) 2024-02-09T18:07:29.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=161241513}) 2024-02-09T18:07:29.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=506387269}) 2024-02-09T18:07:29.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2122281499}) 2024-02-09T18:07:29.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=83504242}) 2024-02-09T18:07:29.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2262368085}) 2024-02-09T18:07:29.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1234098306}) 2024-02-09T18:07:29.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1584843408}) 2024-02-09T18:07:29.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1095058217}) 2024-02-09T18:07:29.747 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3394947753}) 2024-02-09T18:07:29.748 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3341477036}) 2024-02-09T18:07:29.748 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3802029157}) 2024-02-09T18:07:29.748 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=949366715}) 2024-02-09T18:07:29.748 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=364371109}) 2024-02-09T18:07:29.748 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2062225411}) 2024-02-09T18:07:29.748 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3863082259}) 2024-02-09T18:07:29.748 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2670650895}) 2024-02-09T18:07:29.748 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2489698623}) 2024-02-09T18:07:29.748 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3342138685}) 2024-02-09T18:07:29.748 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3877326483}) 2024-02-09T18:07:29.748 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2312062082}) 2024-02-09T18:07:29.749 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3245592922}) 2024-02-09T18:07:29.749 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=11957636}) 2024-02-09T18:07:29.749 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2990792194}) 2024-02-09T18:07:29.749 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4247085149}) 2024-02-09T18:07:29.749 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3659410868}) 2024-02-09T18:07:29.749 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1610312443}) 2024-02-09T18:07:29.749 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2461616993}) 2024-02-09T18:07:29.749 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2705267823}) 2024-02-09T18:07:29.749 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4139551108}) 2024-02-09T18:07:29.749 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=722859643}) 2024-02-09T18:07:29.749 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2619416388}) 2024-02-09T18:07:29.749 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2906346008}) 2024-02-09T18:07:29.750 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=807276385}) 2024-02-09T18:07:29.750 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:29 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1573305806}) 2024-02-09T18:07:29.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: 1.0 scrub starts 2024-02-09T18:07:29.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: 1.0 scrub ok 2024-02-09T18:07:29.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=1120856390,config=2679774077,health=3897446751,kv=744579444}) 2024-02-09T18:07:29.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=1301102820,logm=3662421561}) 2024-02-09T18:07:29.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2670705399}) 2024-02-09T18:07:29.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1682127979}) 2024-02-09T18:07:29.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=403881442}) 2024-02-09T18:07:29.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=588881377}) 2024-02-09T18:07:29.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2695294666}) 2024-02-09T18:07:29.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2668322858}) 2024-02-09T18:07:29.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3342851147}) 2024-02-09T18:07:29.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3929975127}) 2024-02-09T18:07:29.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4008246186}) 2024-02-09T18:07:29.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1671367544}) 2024-02-09T18:07:29.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3728969341}) 2024-02-09T18:07:29.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3736510291}) 2024-02-09T18:07:29.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4003430310}) 2024-02-09T18:07:29.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1118891243}) 2024-02-09T18:07:29.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=914491748}) 2024-02-09T18:07:29.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2429726709}) 2024-02-09T18:07:29.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3445657699}) 2024-02-09T18:07:29.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=650119449}) 2024-02-09T18:07:29.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2101688677}) 2024-02-09T18:07:29.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=615213512}) 2024-02-09T18:07:29.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3150055218}) 2024-02-09T18:07:29.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1164083796}) 2024-02-09T18:07:29.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2658585886}) 2024-02-09T18:07:29.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1894006670}) 2024-02-09T18:07:29.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3773520614}) 2024-02-09T18:07:29.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=161241513}) 2024-02-09T18:07:29.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=506387269}) 2024-02-09T18:07:29.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2122281499}) 2024-02-09T18:07:29.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=83504242}) 2024-02-09T18:07:29.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2262368085}) 2024-02-09T18:07:29.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1234098306}) 2024-02-09T18:07:29.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1584843408}) 2024-02-09T18:07:29.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1095058217}) 2024-02-09T18:07:29.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3394947753}) 2024-02-09T18:07:29.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3341477036}) 2024-02-09T18:07:29.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3802029157}) 2024-02-09T18:07:29.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=949366715}) 2024-02-09T18:07:29.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=364371109}) 2024-02-09T18:07:29.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2062225411}) 2024-02-09T18:07:29.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3863082259}) 2024-02-09T18:07:29.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2670650895}) 2024-02-09T18:07:29.926 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2489698623}) 2024-02-09T18:07:29.926 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3342138685}) 2024-02-09T18:07:29.926 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3877326483}) 2024-02-09T18:07:29.926 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2312062082}) 2024-02-09T18:07:29.926 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3245592922}) 2024-02-09T18:07:29.926 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=11957636}) 2024-02-09T18:07:29.926 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2990792194}) 2024-02-09T18:07:29.926 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4247085149}) 2024-02-09T18:07:29.926 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3659410868}) 2024-02-09T18:07:29.926 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1610312443}) 2024-02-09T18:07:29.926 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2461616993}) 2024-02-09T18:07:29.926 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2705267823}) 2024-02-09T18:07:29.926 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4139551108}) 2024-02-09T18:07:29.926 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=722859643}) 2024-02-09T18:07:29.926 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2619416388}) 2024-02-09T18:07:29.927 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2906346008}) 2024-02-09T18:07:29.927 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=807276385}) 2024-02-09T18:07:29.927 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:29 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1573305806}) 2024-02-09T18:07:29.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: 1.0 scrub starts 2024-02-09T18:07:29.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: 1.0 scrub ok 2024-02-09T18:07:29.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=1120856390,config=2679774077,health=3897446751,kv=744579444}) 2024-02-09T18:07:29.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=1301102820,logm=3662421561}) 2024-02-09T18:07:29.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2670705399}) 2024-02-09T18:07:29.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1682127979}) 2024-02-09T18:07:29.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=403881442}) 2024-02-09T18:07:29.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=588881377}) 2024-02-09T18:07:29.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2695294666}) 2024-02-09T18:07:29.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2668322858}) 2024-02-09T18:07:29.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3342851147}) 2024-02-09T18:07:29.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3929975127}) 2024-02-09T18:07:29.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4008246186}) 2024-02-09T18:07:29.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1671367544}) 2024-02-09T18:07:29.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3728969341}) 2024-02-09T18:07:29.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3736510291}) 2024-02-09T18:07:29.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4003430310}) 2024-02-09T18:07:29.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1118891243}) 2024-02-09T18:07:29.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=914491748}) 2024-02-09T18:07:29.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2429726709}) 2024-02-09T18:07:29.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3445657699}) 2024-02-09T18:07:29.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=650119449}) 2024-02-09T18:07:29.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2101688677}) 2024-02-09T18:07:29.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=615213512}) 2024-02-09T18:07:29.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3150055218}) 2024-02-09T18:07:29.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1164083796}) 2024-02-09T18:07:29.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2658585886}) 2024-02-09T18:07:29.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1894006670}) 2024-02-09T18:07:29.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3773520614}) 2024-02-09T18:07:29.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=161241513}) 2024-02-09T18:07:29.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=506387269}) 2024-02-09T18:07:29.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2122281499}) 2024-02-09T18:07:29.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=83504242}) 2024-02-09T18:07:29.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2262368085}) 2024-02-09T18:07:29.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1234098306}) 2024-02-09T18:07:29.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1584843408}) 2024-02-09T18:07:29.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1095058217}) 2024-02-09T18:07:29.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3394947753}) 2024-02-09T18:07:29.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3341477036}) 2024-02-09T18:07:29.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3802029157}) 2024-02-09T18:07:29.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=949366715}) 2024-02-09T18:07:29.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=364371109}) 2024-02-09T18:07:29.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2062225411}) 2024-02-09T18:07:29.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3863082259}) 2024-02-09T18:07:29.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2670650895}) 2024-02-09T18:07:29.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2489698623}) 2024-02-09T18:07:29.930 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3342138685}) 2024-02-09T18:07:29.931 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3877326483}) 2024-02-09T18:07:29.931 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2312062082}) 2024-02-09T18:07:29.931 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3245592922}) 2024-02-09T18:07:29.931 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=11957636}) 2024-02-09T18:07:29.931 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2990792194}) 2024-02-09T18:07:29.931 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4247085149}) 2024-02-09T18:07:29.931 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3659410868}) 2024-02-09T18:07:29.931 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1610312443}) 2024-02-09T18:07:29.931 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2461616993}) 2024-02-09T18:07:29.931 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2705267823}) 2024-02-09T18:07:29.931 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4139551108}) 2024-02-09T18:07:29.931 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=722859643}) 2024-02-09T18:07:29.931 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2619416388}) 2024-02-09T18:07:29.931 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2906346008}) 2024-02-09T18:07:29.931 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=807276385}) 2024-02-09T18:07:29.932 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:29 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1573305806}) 2024-02-09T18:07:29.962 INFO:teuthology.orchestra.run.smithi099.stdout:734439408087 2024-02-09T18:07:29.962 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-02-09T18:07:30.497 INFO:teuthology.orchestra.run.smithi099.stdout:386547057182 2024-02-09T18:07:30.498 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-02-09T18:07:30.735 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:30 smithi099 ceph-mon[39420]: pgmap v2566: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:30.735 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3533739409}) 2024-02-09T18:07:30.735 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3877105148}) 2024-02-09T18:07:30.735 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2772297204}) 2024-02-09T18:07:30.735 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1526309652}) 2024-02-09T18:07:30.735 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2389163954}) 2024-02-09T18:07:30.735 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2213854622}) 2024-02-09T18:07:30.736 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2110841605}) 2024-02-09T18:07:30.736 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3765664337}) 2024-02-09T18:07:30.736 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2248187072}) 2024-02-09T18:07:30.736 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1283158612}) 2024-02-09T18:07:30.736 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1304053059}) 2024-02-09T18:07:30.736 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3328767971}) 2024-02-09T18:07:30.736 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3220400150}) 2024-02-09T18:07:30.736 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1498205227}) 2024-02-09T18:07:30.736 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1386154074}) 2024-02-09T18:07:30.736 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3038495048}) 2024-02-09T18:07:30.736 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=574720214}) 2024-02-09T18:07:30.736 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=94934694}) 2024-02-09T18:07:30.736 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1310419313}) 2024-02-09T18:07:30.736 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2100302674}) 2024-02-09T18:07:30.737 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2615486846}) 2024-02-09T18:07:30.737 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1955506112}) 2024-02-09T18:07:30.737 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1761947135}) 2024-02-09T18:07:30.737 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1951496974}) 2024-02-09T18:07:30.737 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1456445625}) 2024-02-09T18:07:30.737 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=378599450}) 2024-02-09T18:07:30.737 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2209894190}) 2024-02-09T18:07:30.737 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1298446187}) 2024-02-09T18:07:30.737 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4246635080}) 2024-02-09T18:07:30.737 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1811988602}) 2024-02-09T18:07:30.737 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1683551695}) 2024-02-09T18:07:30.737 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {logm=28,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=38} crc {logm=2714735023,mdsmap=3984425268,mgr=1708625883,mgr_command_descs=708342470,mgr_metadata=670763949,mgrstat=1254251221,mon_config_key=1417369807}) 2024-02-09T18:07:30.737 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1025931032}) 2024-02-09T18:07:30.737 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2286369753}) 2024-02-09T18:07:30.738 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=43,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=39} crc {mon_config_key=1648439176,monmap=423865542,osd_metadata=2412076312,osd_pg_creating=2437762886,osdmap=3574106916}) 2024-02-09T18:07:30.738 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:30 smithi099 ceph-mon[39420]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=72} crc {osdmap=2816553493}) 2024-02-09T18:07:30.738 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:30 smithi099 ceph-mon[39420]: 1.4 scrub starts 2024-02-09T18:07:30.738 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:30 smithi099 ceph-mon[39420]: 1.4 scrub ok 2024-02-09T18:07:30.793 INFO:teuthology.orchestra.run.smithi099.stderr:2024-02-09T18:07:28.164+0000 7f604affd700 0 --2- 172.21.15.99:0/3361610391 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f604c0699d0 0x7f604c069e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-02-09T18:07:30.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:30 smithi179 ceph-mon[43975]: pgmap v2566: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:30.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3533739409}) 2024-02-09T18:07:30.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3877105148}) 2024-02-09T18:07:30.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2772297204}) 2024-02-09T18:07:30.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1526309652}) 2024-02-09T18:07:30.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2389163954}) 2024-02-09T18:07:30.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2213854622}) 2024-02-09T18:07:30.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2110841605}) 2024-02-09T18:07:30.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3765664337}) 2024-02-09T18:07:30.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2248187072}) 2024-02-09T18:07:30.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1283158612}) 2024-02-09T18:07:30.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1304053059}) 2024-02-09T18:07:30.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3328767971}) 2024-02-09T18:07:30.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3220400150}) 2024-02-09T18:07:30.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1498205227}) 2024-02-09T18:07:30.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1386154074}) 2024-02-09T18:07:30.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3038495048}) 2024-02-09T18:07:30.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=574720214}) 2024-02-09T18:07:30.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=94934694}) 2024-02-09T18:07:30.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1310419313}) 2024-02-09T18:07:30.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2100302674}) 2024-02-09T18:07:30.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2615486846}) 2024-02-09T18:07:30.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1955506112}) 2024-02-09T18:07:30.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1761947135}) 2024-02-09T18:07:30.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1951496974}) 2024-02-09T18:07:30.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1456445625}) 2024-02-09T18:07:30.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=378599450}) 2024-02-09T18:07:30.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2209894190}) 2024-02-09T18:07:30.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1298446187}) 2024-02-09T18:07:30.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4246635080}) 2024-02-09T18:07:30.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1811988602}) 2024-02-09T18:07:30.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1683551695}) 2024-02-09T18:07:30.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {logm=28,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=38} crc {logm=2714735023,mdsmap=3984425268,mgr=1708625883,mgr_command_descs=708342470,mgr_metadata=670763949,mgrstat=1254251221,mon_config_key=1417369807}) 2024-02-09T18:07:30.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1025931032}) 2024-02-09T18:07:30.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2286369753}) 2024-02-09T18:07:30.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=43,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=39} crc {mon_config_key=1648439176,monmap=423865542,osd_metadata=2412076312,osd_pg_creating=2437762886,osdmap=3574106916}) 2024-02-09T18:07:30.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:30 smithi179 ceph-mon[43975]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=72} crc {osdmap=2816553493}) 2024-02-09T18:07:30.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:30 smithi179 ceph-mon[43975]: 1.4 scrub starts 2024-02-09T18:07:30.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:30 smithi179 ceph-mon[43975]: 1.4 scrub ok 2024-02-09T18:07:30.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:30 smithi162 ceph-mon[44304]: pgmap v2566: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:30.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3533739409}) 2024-02-09T18:07:30.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3877105148}) 2024-02-09T18:07:30.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2772297204}) 2024-02-09T18:07:30.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1526309652}) 2024-02-09T18:07:30.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2389163954}) 2024-02-09T18:07:30.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2213854622}) 2024-02-09T18:07:30.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2110841605}) 2024-02-09T18:07:30.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3765664337}) 2024-02-09T18:07:30.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2248187072}) 2024-02-09T18:07:30.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1283158612}) 2024-02-09T18:07:30.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1304053059}) 2024-02-09T18:07:30.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3328767971}) 2024-02-09T18:07:30.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3220400150}) 2024-02-09T18:07:30.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1498205227}) 2024-02-09T18:07:30.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1386154074}) 2024-02-09T18:07:30.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3038495048}) 2024-02-09T18:07:30.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=574720214}) 2024-02-09T18:07:30.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=94934694}) 2024-02-09T18:07:30.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1310419313}) 2024-02-09T18:07:30.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2100302674}) 2024-02-09T18:07:30.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2615486846}) 2024-02-09T18:07:30.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1955506112}) 2024-02-09T18:07:30.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1761947135}) 2024-02-09T18:07:30.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1951496974}) 2024-02-09T18:07:30.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1456445625}) 2024-02-09T18:07:30.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=378599450}) 2024-02-09T18:07:30.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2209894190}) 2024-02-09T18:07:30.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1298446187}) 2024-02-09T18:07:30.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4246635080}) 2024-02-09T18:07:30.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1811988602}) 2024-02-09T18:07:30.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1683551695}) 2024-02-09T18:07:30.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {logm=28,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=38} crc {logm=2714735023,mdsmap=3984425268,mgr=1708625883,mgr_command_descs=708342470,mgr_metadata=670763949,mgrstat=1254251221,mon_config_key=1417369807}) 2024-02-09T18:07:30.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1025931032}) 2024-02-09T18:07:30.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2286369753}) 2024-02-09T18:07:30.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=43,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=39} crc {mon_config_key=1648439176,monmap=423865542,osd_metadata=2412076312,osd_pg_creating=2437762886,osdmap=3574106916}) 2024-02-09T18:07:30.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:30 smithi162 ceph-mon[44304]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=72} crc {osdmap=2816553493}) 2024-02-09T18:07:30.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:30 smithi162 ceph-mon[44304]: 1.4 scrub starts 2024-02-09T18:07:30.929 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:30 smithi162 ceph-mon[44304]: 1.4 scrub ok 2024-02-09T18:07:32.188 INFO:teuthology.orchestra.run.smithi099.stdout:137438954053 2024-02-09T18:07:32.189 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-02-09T18:07:32.712 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:32 smithi099 ceph-mon[39420]: pgmap v2567: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:32.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:32 smithi179 ceph-mon[43975]: pgmap v2567: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:32.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:32 smithi162 ceph-mon[44304]: pgmap v2567: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:33.095 INFO:teuthology.orchestra.run.smithi099.stdout:34359738967 2024-02-09T18:07:33.096 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-02-09T18:07:33.718 INFO:teuthology.orchestra.run.smithi099.stdout:244813136435 2024-02-09T18:07:33.719 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-02-09T18:07:34.418 INFO:teuthology.orchestra.run.smithi099.stdout:77309411915 2024-02-09T18:07:34.419 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-02-09T18:07:34.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:34 smithi099 ceph-mon[39420]: pgmap v2568: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:34.796 INFO:teuthology.orchestra.run.smithi099.stdout:171798692413 2024-02-09T18:07:34.797 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-02-09T18:07:34.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:34 smithi179 ceph-mon[43975]: pgmap v2568: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:34.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:34 smithi162 ceph-mon[44304]: pgmap v2568: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:35.315 INFO:teuthology.orchestra.run.smithi099.stdout:55834575443 2024-02-09T18:07:35.315 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-02-09T18:07:35.331 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:07:35.341 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:07:35.348 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:07:35.349 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:07:35.356 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:07:35.369 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:07:35.369 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:07:35.705 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:35 smithi099 ceph-mon[39420]: 1.11 scrub starts 2024-02-09T18:07:35.705 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:35 smithi099 ceph-mon[39420]: 1.11 scrub ok 2024-02-09T18:07:35.855 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:07:35.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:35 smithi179 ceph-mon[43975]: 1.11 scrub starts 2024-02-09T18:07:35.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:35 smithi179 ceph-mon[43975]: 1.11 scrub ok 2024-02-09T18:07:35.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:35 smithi162 ceph-mon[44304]: 1.11 scrub starts 2024-02-09T18:07:35.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:35 smithi162 ceph-mon[44304]: 1.11 scrub ok 2024-02-09T18:07:36.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:36 smithi099 ceph-mon[39420]: 1.1 scrub starts 2024-02-09T18:07:36.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:36 smithi099 ceph-mon[39420]: 1.1 scrub ok 2024-02-09T18:07:36.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:36 smithi099 ceph-mon[39420]: 1.9 scrub starts 2024-02-09T18:07:36.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:36 smithi099 ceph-mon[39420]: pgmap v2569: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:36.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:36 smithi099 ceph-mon[39420]: 1.9 scrub ok 2024-02-09T18:07:36.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:36 smithi179 ceph-mon[43975]: 1.1 scrub starts 2024-02-09T18:07:36.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:36 smithi179 ceph-mon[43975]: 1.1 scrub ok 2024-02-09T18:07:36.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:36 smithi179 ceph-mon[43975]: 1.9 scrub starts 2024-02-09T18:07:36.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:36 smithi179 ceph-mon[43975]: pgmap v2569: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:36.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:36 smithi179 ceph-mon[43975]: 1.9 scrub ok 2024-02-09T18:07:36.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:36 smithi162 ceph-mon[44304]: 1.1 scrub starts 2024-02-09T18:07:36.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:36 smithi162 ceph-mon[44304]: 1.1 scrub ok 2024-02-09T18:07:36.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:36 smithi162 ceph-mon[44304]: 1.9 scrub starts 2024-02-09T18:07:36.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:36 smithi162 ceph-mon[44304]: pgmap v2569: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:36.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:36 smithi162 ceph-mon[44304]: 1.9 scrub ok 2024-02-09T18:07:38.742 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:38 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:07:38.742 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:38 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:07:38.742 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:38 smithi099 ceph-mon[39420]: 1.8 scrub starts 2024-02-09T18:07:38.742 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:38 smithi099 ceph-mon[39420]: 1.8 scrub ok 2024-02-09T18:07:38.742 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:38 smithi099 ceph-mon[39420]: pgmap v2570: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:38.742 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:38 smithi099 ceph-mon[39420]: osdmap e1045: 12 total, 12 up, 12 in 2024-02-09T18:07:38.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:38 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:07:38.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:38 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:07:38.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:38 smithi179 ceph-mon[43975]: 1.8 scrub starts 2024-02-09T18:07:38.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:38 smithi179 ceph-mon[43975]: 1.8 scrub ok 2024-02-09T18:07:38.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:38 smithi179 ceph-mon[43975]: pgmap v2570: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:38.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:38 smithi179 ceph-mon[43975]: osdmap e1045: 12 total, 12 up, 12 in 2024-02-09T18:07:38.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:38 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:07:38.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:38 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:07:38.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:38 smithi162 ceph-mon[44304]: 1.8 scrub starts 2024-02-09T18:07:38.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:38 smithi162 ceph-mon[44304]: 1.8 scrub ok 2024-02-09T18:07:38.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:38 smithi162 ceph-mon[44304]: pgmap v2570: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:38.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:38 smithi162 ceph-mon[44304]: osdmap e1045: 12 total, 12 up, 12 in 2024-02-09T18:07:40.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:40 smithi099 ceph-mon[39420]: pgmap v2572: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:40.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:40 smithi179 ceph-mon[43975]: pgmap v2572: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:40.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:40 smithi162 ceph-mon[44304]: pgmap v2572: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:42.704 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:42 smithi099 ceph-mon[39420]: pgmap v2573: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:42.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:42 smithi179 ceph-mon[43975]: pgmap v2573: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:42.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:42 smithi162 ceph-mon[44304]: pgmap v2573: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:43.958 INFO:teuthology.orchestra.run.smithi099.stdout:734439408092 2024-02-09T18:07:44.398 INFO:teuthology.orchestra.run.smithi099.stdout:210453398078 2024-02-09T18:07:44.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:44 smithi099 ceph-mon[39420]: pgmap v2574: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:44.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:44 smithi099 ceph-mon[39420]: 1.15 scrub starts 2024-02-09T18:07:44.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:44 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/4292961577' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-02-09T18:07:44.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:44 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/3298652273' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-09T18:07:44.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:44 smithi179 ceph-mon[43975]: pgmap v2574: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:44.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:44 smithi179 ceph-mon[43975]: 1.15 scrub starts 2024-02-09T18:07:44.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:44 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/4292961577' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-02-09T18:07:44.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:44 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/3298652273' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-09T18:07:44.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:44 smithi162 ceph-mon[44304]: pgmap v2574: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:44.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:44 smithi162 ceph-mon[44304]: 1.15 scrub starts 2024-02-09T18:07:44.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:44 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/4292961577' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-02-09T18:07:44.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:44 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/3298652273' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-09T18:07:45.700 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:45 smithi099 ceph-mon[39420]: 1.15 scrub ok 2024-02-09T18:07:45.700 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:45 smithi099 ceph-mon[39420]: 1.13 scrub starts 2024-02-09T18:07:45.700 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:45 smithi099 ceph-mon[39420]: 1.13 scrub ok 2024-02-09T18:07:45.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:45 smithi179 ceph-mon[43975]: 1.15 scrub ok 2024-02-09T18:07:45.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:45 smithi179 ceph-mon[43975]: 1.13 scrub starts 2024-02-09T18:07:45.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:45 smithi179 ceph-mon[43975]: 1.13 scrub ok 2024-02-09T18:07:45.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:45 smithi162 ceph-mon[44304]: 1.15 scrub ok 2024-02-09T18:07:45.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:45 smithi162 ceph-mon[44304]: 1.13 scrub starts 2024-02-09T18:07:45.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:45 smithi162 ceph-mon[44304]: 1.13 scrub ok 2024-02-09T18:07:46.291 INFO:tasks.cephadm.ceph_manager.ceph:need seq 734439408087 got 734439408092 for osd.8 2024-02-09T18:07:46.291 DEBUG:teuthology.parallel:result is None 2024-02-09T18:07:46.847 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:46 smithi099 ceph-mon[39420]: 1.5 scrub starts 2024-02-09T18:07:46.847 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:46 smithi099 ceph-mon[39420]: 1.5 scrub ok 2024-02-09T18:07:46.848 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:46 smithi099 ceph-mon[39420]: pgmap v2575: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:46.848 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:46 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/969770747' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-02-09T18:07:46.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:46 smithi179 ceph-mon[43975]: 1.5 scrub starts 2024-02-09T18:07:46.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:46 smithi179 ceph-mon[43975]: 1.5 scrub ok 2024-02-09T18:07:46.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:46 smithi179 ceph-mon[43975]: pgmap v2575: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:46.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:46 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/969770747' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-02-09T18:07:46.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:46 smithi162 ceph-mon[44304]: 1.5 scrub starts 2024-02-09T18:07:46.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:46 smithi162 ceph-mon[44304]: 1.5 scrub ok 2024-02-09T18:07:46.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:46 smithi162 ceph-mon[44304]: pgmap v2575: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:46.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:46 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/969770747' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-02-09T18:07:47.275 INFO:teuthology.orchestra.run.smithi099.stdout:3667902070890 2024-02-09T18:07:47.590 INFO:tasks.cephadm.ceph_manager.ceph:need seq 210453398075 got 210453398078 for osd.6 2024-02-09T18:07:47.591 DEBUG:teuthology.parallel:result is None 2024-02-09T18:07:47.627 INFO:teuthology.orchestra.run.smithi099.stdout:360777253419 2024-02-09T18:07:47.912 INFO:teuthology.orchestra.run.smithi099.stdout:330712482349 2024-02-09T18:07:47.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:47 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1204323203' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-09T18:07:47.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:47 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/333121453' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-02-09T18:07:47.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:47 smithi179 ceph-mon[43975]: 1.1d deep-scrub starts 2024-02-09T18:07:47.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:47 smithi179 ceph-mon[43975]: 1.1d deep-scrub ok 2024-02-09T18:07:47.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:47 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1204323203' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-09T18:07:47.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:47 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/333121453' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-02-09T18:07:47.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:47 smithi162 ceph-mon[44304]: 1.1d deep-scrub starts 2024-02-09T18:07:47.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:47 smithi162 ceph-mon[44304]: 1.1d deep-scrub ok 2024-02-09T18:07:47.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:47 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1204323203' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-09T18:07:47.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:47 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/333121453' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-02-09T18:07:47.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:47 smithi099 ceph-mon[39420]: 1.1d deep-scrub starts 2024-02-09T18:07:47.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:47 smithi099 ceph-mon[39420]: 1.1d deep-scrub ok 2024-02-09T18:07:48.421 INFO:tasks.cephadm.ceph_manager.ceph:need seq 330712482343 got 330712482349 for osd.9 2024-02-09T18:07:48.421 DEBUG:teuthology.parallel:result is None 2024-02-09T18:07:48.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:48 smithi099 ceph-mon[39420]: pgmap v2576: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:48.832 INFO:tasks.cephadm.ceph_manager.ceph:need seq 360777253415 got 360777253419 for osd.10 2024-02-09T18:07:48.832 DEBUG:teuthology.parallel:result is None 2024-02-09T18:07:48.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:48 smithi179 ceph-mon[43975]: pgmap v2576: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:48.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:48 smithi162 ceph-mon[44304]: pgmap v2576: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:49.722 INFO:teuthology.orchestra.run.smithi099.stdout:77309411920 2024-02-09T18:07:49.729 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:07:49.730 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:07:49.730 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:07:49.939 INFO:tasks.cephadm.ceph_manager.ceph:need seq 3667902070886 got 3667902070890 for osd.3 2024-02-09T18:07:49.939 DEBUG:teuthology.parallel:result is None 2024-02-09T18:07:49.997 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:07:50.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:50 smithi099 ceph-mon[39420]: pgmap v2577: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:50.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:50 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2768729292' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-09T18:07:50.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:50 smithi179 ceph-mon[43975]: pgmap v2577: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:50.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:50 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2768729292' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-09T18:07:50.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:50 smithi162 ceph-mon[44304]: pgmap v2577: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:50.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:50 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2768729292' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-09T18:07:51.420 INFO:teuthology.orchestra.run.smithi099.stdout:386547057188 2024-02-09T18:07:51.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:51 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1459738208' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-02-09T18:07:51.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:51 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1459738208' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-02-09T18:07:51.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:51 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1459738208' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-02-09T18:07:52.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:52 smithi179 ceph-mon[43975]: 1.d scrub starts 2024-02-09T18:07:52.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:52 smithi179 ceph-mon[43975]: pgmap v2578: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:52.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:52 smithi179 ceph-mon[43975]: 1.d scrub ok 2024-02-09T18:07:52.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:52 smithi162 ceph-mon[44304]: 1.d scrub starts 2024-02-09T18:07:52.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:52 smithi162 ceph-mon[44304]: pgmap v2578: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:52.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:52 smithi162 ceph-mon[44304]: 1.d scrub ok 2024-02-09T18:07:52.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:52 smithi099 ceph-mon[39420]: 1.d scrub starts 2024-02-09T18:07:52.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:52 smithi099 ceph-mon[39420]: pgmap v2578: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:52.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:52 smithi099 ceph-mon[39420]: 1.d scrub ok 2024-02-09T18:07:53.808 INFO:teuthology.orchestra.run.smithi099.stdout:137438954058 2024-02-09T18:07:53.815 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:53 smithi099 ceph-mon[39420]: 1.6 deep-scrub starts 2024-02-09T18:07:53.815 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:53 smithi099 ceph-mon[39420]: 1.6 deep-scrub ok 2024-02-09T18:07:53.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:53 smithi179 ceph-mon[43975]: 1.6 deep-scrub starts 2024-02-09T18:07:53.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:53 smithi179 ceph-mon[43975]: 1.6 deep-scrub ok 2024-02-09T18:07:53.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:53 smithi162 ceph-mon[44304]: 1.6 deep-scrub starts 2024-02-09T18:07:53.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:53 smithi162 ceph-mon[44304]: 1.6 deep-scrub ok 2024-02-09T18:07:54.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:54 smithi179 ceph-mon[43975]: pgmap v2579: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:54.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:54 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/275812798' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-09T18:07:54.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:54 smithi162 ceph-mon[44304]: pgmap v2579: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:54.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:54 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/275812798' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-09T18:07:54.972 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:54 smithi099 ceph-mon[39420]: pgmap v2579: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:54.972 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:54 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/275812798' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-09T18:07:55.410 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411915 got 77309411920 for osd.2 2024-02-09T18:07:55.410 DEBUG:teuthology.parallel:result is None 2024-02-09T18:07:56.312 INFO:tasks.cephadm.ceph_manager.ceph:need seq 386547057182 got 386547057188 for osd.11 2024-02-09T18:07:56.313 DEBUG:teuthology.parallel:result is None 2024-02-09T18:07:56.547 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:56 smithi099 ceph-mon[39420]: pgmap v2580: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:56.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:56 smithi179 ceph-mon[43975]: pgmap v2580: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:56.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:56 smithi162 ceph-mon[44304]: pgmap v2580: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:57.007 INFO:teuthology.orchestra.run.smithi099.stdout:171798692420 2024-02-09T18:07:57.235 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438954053 got 137438954058 for osd.4 2024-02-09T18:07:57.235 DEBUG:teuthology.parallel:result is None 2024-02-09T18:07:57.806 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:57 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1500421972' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-09T18:07:57.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:57 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1500421972' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-09T18:07:57.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:57 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1500421972' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-09T18:07:58.151 INFO:teuthology.orchestra.run.smithi099.stdout:55834575448 2024-02-09T18:07:58.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:58 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:07:58.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:58 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:07:58.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:58 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:07:58.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:58 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:07:58.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:58 smithi099 ceph-mon[39420]: pgmap v2581: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:58.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:58 smithi099 ceph-mon[39420]: osdmap e1046: 12 total, 12 up, 12 in 2024-02-09T18:07:58.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:58 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1126306574' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-09T18:07:58.910 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575443 got 55834575448 for osd.1 2024-02-09T18:07:58.910 DEBUG:teuthology.parallel:result is None 2024-02-09T18:07:58.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:58 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:07:58.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:58 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:07:58.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:58 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:07:58.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:58 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:07:58.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:58 smithi179 ceph-mon[43975]: pgmap v2581: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:58.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:58 smithi179 ceph-mon[43975]: osdmap e1046: 12 total, 12 up, 12 in 2024-02-09T18:07:58.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:58 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1126306574' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-09T18:07:58.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:58 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:07:58.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:58 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:07:58.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:58 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:07:58.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:58 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:07:58.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:58 smithi162 ceph-mon[44304]: pgmap v2581: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:07:58.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:58 smithi162 ceph-mon[44304]: osdmap e1046: 12 total, 12 up, 12 in 2024-02-09T18:07:58.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:58 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1126306574' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-09T18:07:59.545 INFO:teuthology.orchestra.run.smithi099.stdout:34359738973 2024-02-09T18:07:59.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:59 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:07:59.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:07:59 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:07:59.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:59 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:07:59.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:07:59 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:07:59.950 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:59 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:07:59.950 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:07:59 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:07:59.976 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798692413 got 171798692420 for osd.5 2024-02-09T18:07:59.977 DEBUG:teuthology.parallel:result is None 2024-02-09T18:08:00.743 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:08:00 smithi099 ceph-mon[39420]: 1.1e scrub starts 2024-02-09T18:08:00.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:08:00 smithi099 ceph-mon[39420]: 1.1e scrub ok 2024-02-09T18:08:00.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:08:00 smithi099 ceph-mon[39420]: pgmap v2583: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:08:00.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:08:00 smithi099 ceph-mon[39420]: osdmap e1047: 12 total, 12 up, 12 in 2024-02-09T18:08:00.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:08:00 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2835395008' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-09T18:08:00.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:08:00 smithi099 ceph-mon[39420]: 1.f scrub starts 2024-02-09T18:08:00.822 INFO:teuthology.orchestra.run.smithi099.stdout:244813136441 2024-02-09T18:08:00.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:08:00 smithi179 ceph-mon[43975]: 1.1e scrub starts 2024-02-09T18:08:00.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:08:00 smithi179 ceph-mon[43975]: 1.1e scrub ok 2024-02-09T18:08:00.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:08:00 smithi179 ceph-mon[43975]: pgmap v2583: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:08:00.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:08:00 smithi179 ceph-mon[43975]: osdmap e1047: 12 total, 12 up, 12 in 2024-02-09T18:08:00.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:08:00 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2835395008' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-09T18:08:00.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:08:00 smithi179 ceph-mon[43975]: 1.f scrub starts 2024-02-09T18:08:00.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:08:00 smithi162 ceph-mon[44304]: 1.1e scrub starts 2024-02-09T18:08:00.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:08:00 smithi162 ceph-mon[44304]: 1.1e scrub ok 2024-02-09T18:08:00.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:08:00 smithi162 ceph-mon[44304]: pgmap v2583: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:08:00.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:08:00 smithi162 ceph-mon[44304]: osdmap e1047: 12 total, 12 up, 12 in 2024-02-09T18:08:00.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:08:00 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2835395008' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-09T18:08:00.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:08:00 smithi162 ceph-mon[44304]: 1.f scrub starts 2024-02-09T18:08:01.047 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738967 got 34359738973 for osd.0 2024-02-09T18:08:01.047 DEBUG:teuthology.parallel:result is None 2024-02-09T18:08:01.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:08:01 smithi099 ceph-mon[39420]: 1.f scrub ok 2024-02-09T18:08:01.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:08:01 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/1938669515' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-09T18:08:01.844 INFO:tasks.cephadm.ceph_manager.ceph:need seq 244813136435 got 244813136441 for osd.7 2024-02-09T18:08:01.844 DEBUG:teuthology.parallel:result is None 2024-02-09T18:08:01.845 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-02-09T18:08:01.845 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph pg dump --format=json 2024-02-09T18:08:01.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:08:01 smithi179 ceph-mon[43975]: 1.f scrub ok 2024-02-09T18:08:01.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:08:01 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/1938669515' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-09T18:08:01.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:08:01 smithi162 ceph-mon[44304]: 1.f scrub ok 2024-02-09T18:08:01.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:08:01 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/1938669515' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-09T18:08:02.109 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:08:02.685 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:08:02 smithi099 ceph-mon[39420]: pgmap v2585: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:08:02.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:08:02 smithi179 ceph-mon[43975]: pgmap v2585: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:08:02.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:08:02 smithi162 ceph-mon[44304]: pgmap v2585: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:08:04.261 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T18:08:04.262 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-02-09T18:08:04.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:08:04 smithi179 ceph-mon[43975]: pgmap v2586: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:08:04.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:08:04 smithi162 ceph-mon[44304]: pgmap v2586: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:08:04.941 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:08:04 smithi099 ceph-mon[39420]: pgmap v2586: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:08:04.967 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":2586,"stamp":"2024-02-09T18:08:03.414175+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":93,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":10119680,"kb_used_data":10212,"kb_used_omap":430,"kb_used_meta":10108881,"kb_avail":1114822144,"statfs":{"total":1151940427776,"available":1141577875456,"internally_reserved":0,"allocated":10457088,"data_stored":6743170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":441167,"internal_metadata":10351494321},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001736"},"pg_stats":[{"pgid":"1.1e","version":"63'139","reported_seq":1460,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-02-09T18:07:59.936721+0000","last_change":"2024-02-09T18:07:58.910839+0000","last_active":"2024-02-09T18:07:59.936721+0000","last_peered":"2024-02-09T18:07:59.936721+0000","last_clean":"2024-02-09T18:07:59.936721+0000","last_became_active":"2024-02-09T18:03:47.759332+0000","last_became_peered":"2024-02-09T18:03:47.759332+0000","last_unstale":"2024-02-09T18:07:59.936721+0000","last_undegraded":"2024-02-09T18:07:59.936721+0000","last_fullsized":"2024-02-09T18:07:59.936721+0000","mapping_epoch":961,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":962,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:58.910679+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:07:58.910679+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:09:08.763696+0000","scrub_duration":117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"63'139","reported_seq":1469,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-02-09T18:07:59.543958+0000","last_change":"2024-02-09T18:07:47.252312+0000","last_active":"2024-02-09T18:07:59.543958+0000","last_peered":"2024-02-09T18:07:59.543958+0000","last_clean":"2024-02-09T18:07:59.543958+0000","last_became_active":"2024-02-09T17:43:28.594296+0000","last_became_peered":"2024-02-09T17:43:28.594296+0000","last_unstale":"2024-02-09T18:07:59.543958+0000","last_undegraded":"2024-02-09T18:07:59.543958+0000","last_fullsized":"2024-02-09T18:07:59.543958+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:47.252184+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T18:07:47.252184+0000","last_clean_scrub_stamp":"2024-02-09T18:07:47.252184+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:09:13.542304+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"63'139","reported_seq":1989,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-02-09T18:07:59.936470+0000","last_change":"2024-02-09T18:07:14.310594+0000","last_active":"2024-02-09T18:07:59.936470+0000","last_peered":"2024-02-09T18:07:59.936470+0000","last_clean":"2024-02-09T18:07:59.936470+0000","last_became_active":"2024-02-09T17:43:19.103584+0000","last_became_peered":"2024-02-09T17:43:19.103584+0000","last_unstale":"2024-02-09T18:07:59.936470+0000","last_undegraded":"2024-02-09T18:07:59.936470+0000","last_fullsized":"2024-02-09T18:07:59.936470+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:14.310481+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T18:07:14.310481+0000","last_clean_scrub_stamp":"2024-02-09T18:07:14.310481+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:08:24.380102+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"63'139","reported_seq":1787,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-02-09T18:07:59.936317+0000","last_change":"2024-02-09T18:06:43.197838+0000","last_active":"2024-02-09T18:07:59.936317+0000","last_peered":"2024-02-09T18:07:59.936317+0000","last_clean":"2024-02-09T18:07:59.936317+0000","last_became_active":"2024-02-09T17:43:19.102729+0000","last_became_peered":"2024-02-09T17:43:19.102729+0000","last_unstale":"2024-02-09T18:07:59.936317+0000","last_undegraded":"2024-02-09T18:07:59.936317+0000","last_fullsized":"2024-02-09T18:07:59.936317+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:06:43.197734+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:45:44.988633+0000","last_clean_scrub_stamp":"2024-02-09T18:06:43.197734+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:08:09.428876+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"63'139","reported_seq":683,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-02-09T18:07:59.936179+0000","last_change":"2024-02-09T18:07:00.433777+0000","last_active":"2024-02-09T18:07:59.936179+0000","last_peered":"2024-02-09T18:07:59.936179+0000","last_clean":"2024-02-09T18:07:59.936179+0000","last_became_active":"2024-02-09T18:03:02.127510+0000","last_became_peered":"2024-02-09T18:03:02.127510+0000","last_unstale":"2024-02-09T18:07:59.936179+0000","last_undegraded":"2024-02-09T18:07:59.936179+0000","last_fullsized":"2024-02-09T18:07:59.936179+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:00.433674+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:07:00.433674+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:08:26.949531+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"63'139","reported_seq":1835,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-02-09T18:07:59.936378+0000","last_change":"2024-02-09T18:07:11.021282+0000","last_active":"2024-02-09T18:07:59.936378+0000","last_peered":"2024-02-09T18:07:59.936378+0000","last_clean":"2024-02-09T18:07:59.936378+0000","last_became_active":"2024-02-09T18:05:14.211564+0000","last_became_peered":"2024-02-09T18:05:14.211564+0000","last_unstale":"2024-02-09T18:07:59.936378+0000","last_undegraded":"2024-02-09T18:07:59.936378+0000","last_fullsized":"2024-02-09T18:07:59.936378+0000","mapping_epoch":993,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":994,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:11.021172+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T18:04:37.468994+0000","last_clean_scrub_stamp":"2024-02-09T18:07:11.021172+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:08:26.060867+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"63'139","reported_seq":2124,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-02-09T18:07:59.935799+0000","last_change":"2024-02-09T18:06:53.873646+0000","last_active":"2024-02-09T18:07:59.935799+0000","last_peered":"2024-02-09T18:07:59.935799+0000","last_clean":"2024-02-09T18:07:59.935799+0000","last_became_active":"2024-02-09T18:06:03.580532+0000","last_became_peered":"2024-02-09T18:06:03.580532+0000","last_unstale":"2024-02-09T18:07:59.935799+0000","last_undegraded":"2024-02-09T18:07:59.935799+0000","last_fullsized":"2024-02-09T18:07:59.935799+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:06:53.873522+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T18:05:48.428084+0000","last_clean_scrub_stamp":"2024-02-09T18:06:53.873522+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:08:18.043167+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"63'139","reported_seq":1598,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-02-09T18:07:59.936546+0000","last_change":"2024-02-09T18:07:27.587234+0000","last_active":"2024-02-09T18:07:59.936546+0000","last_peered":"2024-02-09T18:07:59.936546+0000","last_clean":"2024-02-09T18:07:59.936546+0000","last_became_active":"2024-02-09T18:05:14.211197+0000","last_became_peered":"2024-02-09T18:05:14.211197+0000","last_unstale":"2024-02-09T18:07:59.936546+0000","last_undegraded":"2024-02-09T18:07:59.936546+0000","last_fullsized":"2024-02-09T18:07:59.936546+0000","mapping_epoch":993,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":994,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:27.587110+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T18:03:40.548833+0000","last_clean_scrub_stamp":"2024-02-09T18:07:27.587110+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:08:44.420949+0000","scrub_duration":188,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"63'139","reported_seq":1455,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-02-09T18:07:59.543760+0000","last_change":"2024-02-09T18:07:03.801662+0000","last_active":"2024-02-09T18:07:59.543760+0000","last_peered":"2024-02-09T18:07:59.543760+0000","last_clean":"2024-02-09T18:07:59.543760+0000","last_became_active":"2024-02-09T17:59:12.003987+0000","last_became_peered":"2024-02-09T17:59:12.003987+0000","last_unstale":"2024-02-09T18:07:59.543760+0000","last_undegraded":"2024-02-09T18:07:59.543760+0000","last_fullsized":"2024-02-09T18:07:59.543760+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:03.801550+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:11.970338+0000","last_clean_scrub_stamp":"2024-02-09T18:07:03.801550+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:08:13.429137+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"63'139","reported_seq":1890,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-02-09T18:07:59.936457+0000","last_change":"2024-02-09T18:07:43.654399+0000","last_active":"2024-02-09T18:07:59.936457+0000","last_peered":"2024-02-09T18:07:59.936457+0000","last_clean":"2024-02-09T18:07:59.936457+0000","last_became_active":"2024-02-09T17:43:19.138398+0000","last_became_peered":"2024-02-09T17:43:19.138398+0000","last_unstale":"2024-02-09T18:07:59.936457+0000","last_undegraded":"2024-02-09T18:07:59.936457+0000","last_fullsized":"2024-02-09T18:07:59.936457+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:43.654293+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T18:06:22.898046+0000","last_clean_scrub_stamp":"2024-02-09T18:07:43.654293+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:08:58.977702+0000","scrub_duration":196,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"63'139","reported_seq":1962,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-02-09T18:07:59.542565+0000","last_change":"2024-02-09T18:07:26.376103+0000","last_active":"2024-02-09T18:07:59.542565+0000","last_peered":"2024-02-09T18:07:59.542565+0000","last_clean":"2024-02-09T18:07:59.542565+0000","last_became_active":"2024-02-09T17:43:28.589307+0000","last_became_peered":"2024-02-09T17:43:28.589307+0000","last_unstale":"2024-02-09T18:07:59.542565+0000","last_undegraded":"2024-02-09T18:07:59.542565+0000","last_fullsized":"2024-02-09T18:07:59.542565+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:26.376000+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:38:26.968892+0000","last_clean_scrub_stamp":"2024-02-09T18:07:26.376000+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:08:42.343492+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"63'139","reported_seq":2540,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-02-09T18:07:59.935977+0000","last_change":"2024-02-09T18:07:28.365226+0000","last_active":"2024-02-09T18:07:59.935977+0000","last_peered":"2024-02-09T18:07:59.935977+0000","last_clean":"2024-02-09T18:07:59.935977+0000","last_became_active":"2024-02-09T17:41:14.521079+0000","last_became_peered":"2024-02-09T17:41:14.521079+0000","last_unstale":"2024-02-09T18:07:59.935977+0000","last_undegraded":"2024-02-09T18:07:59.935977+0000","last_fullsized":"2024-02-09T18:07:59.935977+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:28.365133+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:07:28.365133+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:08:57.339359+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"63'139","reported_seq":1192,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-02-09T18:07:59.936369+0000","last_change":"2024-02-09T18:07:09.415201+0000","last_active":"2024-02-09T18:07:59.936369+0000","last_peered":"2024-02-09T18:07:59.936369+0000","last_clean":"2024-02-09T18:07:59.936369+0000","last_became_active":"2024-02-09T18:06:17.478373+0000","last_became_peered":"2024-02-09T18:06:17.478373+0000","last_unstale":"2024-02-09T18:07:59.936369+0000","last_undegraded":"2024-02-09T18:07:59.936369+0000","last_fullsized":"2024-02-09T18:07:59.936369+0000","mapping_epoch":1019,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":1020,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:09.415099+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:07:09.415099+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:08:17.563124+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"63'139","reported_seq":1747,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-02-09T18:07:59.543573+0000","last_change":"2024-02-09T18:06:37.782212+0000","last_active":"2024-02-09T18:07:59.543573+0000","last_peered":"2024-02-09T18:07:59.543573+0000","last_clean":"2024-02-09T18:07:59.543573+0000","last_became_active":"2024-02-09T18:03:02.134791+0000","last_became_peered":"2024-02-09T18:03:02.134791+0000","last_unstale":"2024-02-09T18:07:59.543573+0000","last_undegraded":"2024-02-09T18:07:59.543573+0000","last_fullsized":"2024-02-09T18:07:59.543573+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:06:37.782111+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:46:40.684762+0000","last_clean_scrub_stamp":"2024-02-09T18:06:37.782111+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:08:04.224692+0000","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"63'139","reported_seq":480,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-02-09T18:07:59.936474+0000","last_change":"2024-02-09T18:07:34.985801+0000","last_active":"2024-02-09T18:07:59.936474+0000","last_peered":"2024-02-09T18:07:59.936474+0000","last_clean":"2024-02-09T18:07:59.936474+0000","last_became_active":"2024-02-09T18:05:14.211371+0000","last_became_peered":"2024-02-09T18:05:14.211371+0000","last_unstale":"2024-02-09T18:07:59.936474+0000","last_undegraded":"2024-02-09T18:07:59.936474+0000","last_fullsized":"2024-02-09T18:07:59.936474+0000","mapping_epoch":993,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":994,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:34.985693+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T18:00:46.701010+0000","last_clean_scrub_stamp":"2024-02-09T18:07:34.985693+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:08:39.774211+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"63'139","reported_seq":394,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-02-09T18:07:59.936384+0000","last_change":"2024-02-09T18:07:51.912625+0000","last_active":"2024-02-09T18:07:59.936384+0000","last_peered":"2024-02-09T18:07:59.936384+0000","last_clean":"2024-02-09T18:07:59.936384+0000","last_became_active":"2024-02-09T18:05:14.211331+0000","last_became_peered":"2024-02-09T18:05:14.211331+0000","last_unstale":"2024-02-09T18:07:59.936384+0000","last_undegraded":"2024-02-09T18:07:59.936384+0000","last_fullsized":"2024-02-09T18:07:59.936384+0000","mapping_epoch":993,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":994,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:51.912498+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T18:07:51.912498+0000","last_clean_scrub_stamp":"2024-02-09T18:07:51.912498+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:09:19.321922+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.7","version":"63'139","reported_seq":1657,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-02-09T18:07:59.543865+0000","last_change":"2024-02-09T18:07:10.067566+0000","last_active":"2024-02-09T18:07:59.543865+0000","last_peered":"2024-02-09T18:07:59.543865+0000","last_clean":"2024-02-09T18:07:59.543865+0000","last_became_active":"2024-02-09T17:52:32.292519+0000","last_became_peered":"2024-02-09T17:52:32.292519+0000","last_unstale":"2024-02-09T18:07:59.543865+0000","last_undegraded":"2024-02-09T18:07:59.543865+0000","last_fullsized":"2024-02-09T18:07:59.543865+0000","mapping_epoch":716,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":717,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:10.067465+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:07:10.067465+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:08:17.504558+0000","scrub_duration":278,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"63'139","reported_seq":2121,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-02-09T18:07:59.936628+0000","last_change":"2024-02-09T18:07:29.604329+0000","last_active":"2024-02-09T18:07:59.936628+0000","last_peered":"2024-02-09T18:07:59.936628+0000","last_clean":"2024-02-09T18:07:59.936628+0000","last_became_active":"2024-02-09T17:33:00.284924+0000","last_became_peered":"2024-02-09T17:33:00.284924+0000","last_unstale":"2024-02-09T18:07:59.936628+0000","last_undegraded":"2024-02-09T18:07:59.936628+0000","last_fullsized":"2024-02-09T18:07:59.936628+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:29.604237+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:56:45.406767+0000","last_clean_scrub_stamp":"2024-02-09T18:07:29.604237+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:08:42.253371+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.5","version":"63'139","reported_seq":1307,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-02-09T18:07:59.936721+0000","last_change":"2024-02-09T18:07:44.949102+0000","last_active":"2024-02-09T18:07:59.936721+0000","last_peered":"2024-02-09T18:07:59.936721+0000","last_clean":"2024-02-09T18:07:59.936721+0000","last_became_active":"2024-02-09T18:05:14.210449+0000","last_became_peered":"2024-02-09T18:05:14.210449+0000","last_unstale":"2024-02-09T18:07:59.936721+0000","last_undegraded":"2024-02-09T18:07:59.936721+0000","last_fullsized":"2024-02-09T18:07:59.936721+0000","mapping_epoch":993,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":994,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:44.948971+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:07:44.948971+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:09:12.382631+0000","scrub_duration":141,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,8],"acting":[10,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.a","version":"63'139","reported_seq":2060,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-02-09T18:07:59.936297+0000","last_change":"2024-02-09T18:07:23.484093+0000","last_active":"2024-02-09T18:07:59.936297+0000","last_peered":"2024-02-09T18:07:59.936297+0000","last_clean":"2024-02-09T18:07:59.936297+0000","last_became_active":"2024-02-09T18:03:02.134238+0000","last_became_peered":"2024-02-09T18:03:02.134238+0000","last_unstale":"2024-02-09T18:07:59.936297+0000","last_undegraded":"2024-02-09T18:07:59.936297+0000","last_fullsized":"2024-02-09T18:07:59.936297+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:23.483995+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T18:03:49.362497+0000","last_clean_scrub_stamp":"2024-02-09T18:07:23.483995+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:08:27.179567+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.b","version":"63'139","reported_seq":1409,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-02-09T18:07:59.936397+0000","last_change":"2024-02-09T18:06:49.074743+0000","last_active":"2024-02-09T18:07:59.936397+0000","last_peered":"2024-02-09T18:07:59.936397+0000","last_clean":"2024-02-09T18:07:59.936397+0000","last_became_active":"2024-02-09T17:55:08.938203+0000","last_became_peered":"2024-02-09T17:55:08.938203+0000","last_unstale":"2024-02-09T18:07:59.936397+0000","last_undegraded":"2024-02-09T18:07:59.936397+0000","last_fullsized":"2024-02-09T18:07:59.936397+0000","mapping_epoch":781,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":782,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:06:49.074627+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T18:05:26.531050+0000","last_clean_scrub_stamp":"2024-02-09T18:06:49.074627+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:08:17.069529+0000","scrub_duration":230,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.8","version":"63'139","reported_seq":1276,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-02-09T18:07:59.935905+0000","last_change":"2024-02-09T18:07:36.768760+0000","last_active":"2024-02-09T18:07:59.935905+0000","last_peered":"2024-02-09T18:07:59.935905+0000","last_clean":"2024-02-09T18:07:59.935905+0000","last_became_active":"2024-02-09T18:06:03.579088+0000","last_became_peered":"2024-02-09T18:06:03.579088+0000","last_unstale":"2024-02-09T18:07:59.935905+0000","last_undegraded":"2024-02-09T18:07:59.935905+0000","last_fullsized":"2024-02-09T18:07:59.935905+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:36.768659+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:07:36.768659+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:08:45.132904+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"63'139","reported_seq":2546,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-02-09T18:07:59.936012+0000","last_change":"2024-02-09T18:07:35.563141+0000","last_active":"2024-02-09T18:07:59.936012+0000","last_peered":"2024-02-09T18:07:59.936012+0000","last_clean":"2024-02-09T18:07:59.936012+0000","last_became_active":"2024-02-09T17:43:28.590237+0000","last_became_peered":"2024-02-09T17:43:28.590237+0000","last_unstale":"2024-02-09T18:07:59.936012+0000","last_undegraded":"2024-02-09T18:07:59.936012+0000","last_fullsized":"2024-02-09T18:07:59.936012+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:35.563036+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:59:49.716259+0000","last_clean_scrub_stamp":"2024-02-09T18:07:35.563036+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:08:38.471263+0000","scrub_duration":197,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"63'139","reported_seq":502,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-02-09T18:07:59.935956+0000","last_change":"2024-02-09T18:06:47.498203+0000","last_active":"2024-02-09T18:07:59.935956+0000","last_peered":"2024-02-09T18:07:59.935956+0000","last_clean":"2024-02-09T18:07:59.935956+0000","last_became_active":"2024-02-09T18:03:02.137032+0000","last_became_peered":"2024-02-09T18:03:02.137032+0000","last_unstale":"2024-02-09T18:07:59.935956+0000","last_undegraded":"2024-02-09T18:07:59.935956+0000","last_fullsized":"2024-02-09T18:07:59.935956+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:06:47.498092+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T18:06:47.498092+0000","last_clean_scrub_stamp":"2024-02-09T18:06:47.498092+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.f","version":"63'139","reported_seq":2088,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-02-09T18:07:59.958075+0000","last_change":"2024-02-09T18:07:59.958123+0000","last_active":"2024-02-09T18:07:59.958075+0000","last_peered":"2024-02-09T18:07:59.958075+0000","last_clean":"2024-02-09T18:07:59.958075+0000","last_became_active":"2024-02-09T18:03:02.138442+0000","last_became_peered":"2024-02-09T18:03:02.138442+0000","last_unstale":"2024-02-09T18:07:59.958075+0000","last_undegraded":"2024-02-09T18:07:59.958075+0000","last_fullsized":"2024-02-09T18:07:59.958075+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:59.957992+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:48:34.651647+0000","last_clean_scrub_stamp":"2024-02-09T18:07:59.957992+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:09:00.606103+0000","scrub_duration":196,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"63'139","reported_seq":2033,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-02-09T18:07:59.935498+0000","last_change":"2024-02-09T18:07:12.841278+0000","last_active":"2024-02-09T18:07:59.935498+0000","last_peered":"2024-02-09T18:07:59.935498+0000","last_clean":"2024-02-09T18:07:59.935498+0000","last_became_active":"2024-02-09T17:59:12.208604+0000","last_became_peered":"2024-02-09T17:59:12.208604+0000","last_unstale":"2024-02-09T18:07:59.935498+0000","last_undegraded":"2024-02-09T18:07:59.935498+0000","last_fullsized":"2024-02-09T18:07:59.935498+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:12.841153+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:14.135321+0000","last_clean_scrub_stamp":"2024-02-09T18:07:12.841153+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:08:38.766542+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"63'139","reported_seq":1620,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-02-09T18:07:59.936543+0000","last_change":"2024-02-09T18:07:51.498713+0000","last_active":"2024-02-09T18:07:59.936543+0000","last_peered":"2024-02-09T18:07:59.936543+0000","last_clean":"2024-02-09T18:07:59.936543+0000","last_became_active":"2024-02-09T18:03:02.132990+0000","last_became_peered":"2024-02-09T18:03:02.132990+0000","last_unstale":"2024-02-09T18:07:59.936543+0000","last_undegraded":"2024-02-09T18:07:59.936543+0000","last_fullsized":"2024-02-09T18:07:59.936543+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:51.498614+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T18:00:49.890794+0000","last_clean_scrub_stamp":"2024-02-09T18:07:51.498614+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:09:07.875541+0000","scrub_duration":197,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"63'139","reported_seq":1090,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-02-09T18:07:59.936001+0000","last_change":"2024-02-09T18:06:58.795872+0000","last_active":"2024-02-09T18:07:59.936001+0000","last_peered":"2024-02-09T18:07:59.936001+0000","last_clean":"2024-02-09T18:07:59.936001+0000","last_became_active":"2024-02-09T18:06:03.580581+0000","last_became_peered":"2024-02-09T18:06:03.580581+0000","last_unstale":"2024-02-09T18:07:59.936001+0000","last_undegraded":"2024-02-09T18:07:59.936001+0000","last_fullsized":"2024-02-09T18:07:59.936001+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:06:58.795739+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T18:05:33.857425+0000","last_clean_scrub_stamp":"2024-02-09T18:06:58.795739+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:08:02.202312+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"63'139","reported_seq":1560,"reported_epoch":1046,"state":"active+clean","last_fresh":"2024-02-09T18:07:58.318477+0000","last_change":"2024-02-09T18:07:34.454093+0000","last_active":"2024-02-09T18:07:58.318477+0000","last_peered":"2024-02-09T18:07:58.318477+0000","last_clean":"2024-02-09T18:07:58.318477+0000","last_became_active":"2024-02-09T18:05:27.447245+0000","last_became_peered":"2024-02-09T18:05:27.447245+0000","last_unstale":"2024-02-09T18:07:58.318477+0000","last_undegraded":"2024-02-09T18:07:58.318477+0000","last_fullsized":"2024-02-09T18:07:58.318477+0000","mapping_epoch":998,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":999,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:34.453939+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T18:05:11.269341+0000","last_clean_scrub_stamp":"2024-02-09T18:07:34.453939+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:08:45.889300+0000","scrub_duration":118,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"63'139","reported_seq":1609,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-02-09T18:07:59.936105+0000","last_change":"2024-02-09T18:07:03.824695+0000","last_active":"2024-02-09T18:07:59.936105+0000","last_peered":"2024-02-09T18:07:59.936105+0000","last_clean":"2024-02-09T18:07:59.936105+0000","last_became_active":"2024-02-09T18:06:03.580019+0000","last_became_peered":"2024-02-09T18:06:03.580019+0000","last_unstale":"2024-02-09T18:07:59.936105+0000","last_undegraded":"2024-02-09T18:07:59.936105+0000","last_fullsized":"2024-02-09T18:07:59.936105+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:03.824553+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:07:03.824553+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:08:05.828313+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"63'139","reported_seq":1749,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-02-09T18:07:59.543680+0000","last_change":"2024-02-09T18:07:45.389122+0000","last_active":"2024-02-09T18:07:59.543680+0000","last_peered":"2024-02-09T18:07:59.543680+0000","last_clean":"2024-02-09T18:07:59.543680+0000","last_became_active":"2024-02-09T18:05:14.212995+0000","last_became_peered":"2024-02-09T18:05:14.212995+0000","last_unstale":"2024-02-09T18:07:59.543680+0000","last_undegraded":"2024-02-09T18:07:59.543680+0000","last_fullsized":"2024-02-09T18:07:59.543680+0000","mapping_epoch":993,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":994,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:45.389019+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:07:45.389019+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:08:59.448319+0000","scrub_duration":274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547057191,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1006496,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":1005758,"kb_avail":92738656,"statfs":{"total":95995035648,"available":94964383744,"internally_reserved":0,"allocated":741376,"data_stored":431582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":1029896644},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 18:07:30 2024","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":1.05,"15min":1.5049999999999999},"min":{"1min":0.32700000000000001,"5min":0.24399999999999999,"15min":0.22},"max":{"1min":11.006,"5min":11.006,"15min":13.307},"last":0.33600000000000002},{"interface":"front","average":{"1min":1.25,"5min":1.1910000000000001,"15min":1.6439999999999999},"min":{"1min":0.41199999999999998,"5min":0.26200000000000001,"15min":0.23100000000000001},"max":{"1min":13.771000000000001,"5min":13.771000000000001,"15min":13.771000000000001},"last":0.40799999999999997}]},{"osd":1,"last update":"Fri Feb 9 18:07:30 2024","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.5009999999999999,"15min":1.8089999999999999},"min":{"1min":0.38500000000000001,"5min":0.23100000000000001,"15min":0.22600000000000001},"max":{"1min":10.757,"5min":21.616,"15min":21.616},"last":0.35499999999999998},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.3069999999999999,"15min":1.724},"min":{"1min":0.38800000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":13.667,"5min":22.456,"15min":22.456},"last":0.58999999999999997}]},{"osd":2,"last update":"Fri Feb 9 18:07:30 2024","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.1539999999999999,"15min":1.635},"min":{"1min":0.34999999999999998,"5min":0.27200000000000002,"15min":0.25600000000000001},"max":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":14.766999999999999},"last":0.379},{"interface":"front","average":{"1min":1.351,"5min":1.1160000000000001,"15min":1.446},"min":{"1min":0.41099999999999998,"5min":0.32400000000000001,"15min":0.27800000000000002},"max":{"1min":13.837999999999999,"5min":13.837999999999999,"15min":13.837999999999999},"last":0.53800000000000003}]},{"osd":3,"last update":"Fri Feb 9 18:07:30 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":1.8740000000000001,"15min":1.72},"min":{"1min":0.40100000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":2.3319999999999999,"5min":41.395000000000003,"15min":41.395000000000003},"last":0.45400000000000001},{"interface":"front","average":{"1min":1.593,"5min":1.857,"15min":1.8799999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.32300000000000001},"max":{"1min":16.914999999999999,"5min":41.276000000000003,"15min":41.276000000000003},"last":0.47999999999999998}]},{"osd":4,"last update":"Fri Feb 9 17:58:44 2024","interfaces":[{"interface":"back","average":{"1min":3.1709999999999998,"5min":2.5779999999999998,"15min":2.4790000000000001},"min":{"1min":0.40999999999999998,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":14.866,"5min":14.866,"15min":14.866},"last":0.45700000000000002},{"interface":"front","average":{"1min":3.2349999999999999,"5min":2.5640000000000001,"15min":2.452},"min":{"1min":0.496,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":14.939,"5min":14.939,"15min":14.939},"last":0.42299999999999999}]},{"osd":5,"last update":"Fri Feb 9 18:07:30 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.995,"15min":1.3620000000000001},"min":{"1min":0.46800000000000003,"5min":0.30199999999999999,"15min":0.27000000000000002},"max":{"1min":0.93700000000000006,"5min":6.0839999999999996,"15min":13.263999999999999},"last":0.629},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.91700000000000004,"15min":1.4019999999999999},"min":{"1min":0.59399999999999997,"5min":0.317,"15min":0.30099999999999999},"max":{"1min":1.1359999999999999,"5min":6.5350000000000001,"15min":14.914999999999999},"last":0.68200000000000005}]},{"osd":6,"last update":"Fri Feb 9 18:07:30 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":1.28,"15min":1.6020000000000001},"min":{"1min":0.57599999999999996,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.1000000000000001,"5min":8.7370000000000001,"15min":11.885},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.86199999999999999,"5min":1.0609999999999999,"15min":1.518},"min":{"1min":0.60199999999999998,"5min":0.38900000000000001,"15min":0.32400000000000001},"max":{"1min":1.0960000000000001,"5min":7.0830000000000002,"15min":15.409000000000001},"last":0.64200000000000002}]},{"osd":7,"last update":"Fri Feb 9 18:07:30 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":1.1579999999999999,"15min":1.548},"min":{"1min":0.51700000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.0489999999999999,"5min":7.0410000000000004,"15min":15.51},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.80500000000000005,"5min":1.359,"15min":1.6319999999999999},"min":{"1min":0.56699999999999995,"5min":0.38600000000000001,"15min":0.33400000000000002},"max":{"1min":1.0920000000000001,"5min":8.6829999999999998,"15min":10.911},"last":0.72999999999999998}]},{"osd":8,"last update":"Fri Feb 9 18:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":1.246,"15min":1.9830000000000001},"min":{"1min":0.45500000000000002,"5min":0.314,"15min":0.307},"max":{"1min":0.84199999999999997,"5min":8.7629999999999999,"15min":29.233000000000001},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.66700000000000004,"5min":1.284,"15min":2.0800000000000001},"min":{"1min":0.51300000000000001,"5min":0.40000000000000002,"15min":0.34499999999999997},"max":{"1min":0.96699999999999997,"5min":8.7889999999999997,"15min":29.091999999999999},"last":0.501}]},{"osd":9,"last update":"Fri Feb 9 18:07:02 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":1.395,"15min":2.0830000000000002},"min":{"1min":0.39500000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.3029999999999999,"5min":12.68,"15min":22.613},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.69899999999999995,"5min":1.4690000000000001,"15min":1.9810000000000001},"min":{"1min":0.51700000000000002,"5min":0.38,"15min":0.35899999999999999},"max":{"1min":1.3500000000000001,"5min":12.707000000000001,"15min":19.555},"last":0.66500000000000004}]},{"osd":10,"last update":"Fri Feb 9 18:07:30 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.98899999999999999,"15min":2.1659999999999999},"min":{"1min":0.45900000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.135,"5min":6.5350000000000001,"15min":25.093},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.871,"5min":1.075,"15min":1.871},"min":{"1min":0.59799999999999998,"5min":0.46000000000000002,"15min":0.35999999999999999},"max":{"1min":1.173,"5min":6.5640000000000001,"15min":20.233000000000001},"last":0.754}]}]},{"osd":10,"up_from":84,"seq":360777253423,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":898212,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":897470,"kb_avail":92846940,"statfs":{"total":95995035648,"available":95075266560,"internally_reserved":0,"allocated":741376,"data_stored":431582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":919009728},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 18:07:17 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.623,"15min":0.627},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1,"5min":1,"15min":1},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.622,"15min":0.63},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.91700000000000004,"5min":1.014,"15min":1.014},"last":0.57199999999999995}]},{"osd":1,"last update":"Fri Feb 9 18:04:30 2024","interfaces":[{"interface":"back","average":{"1min":3.3650000000000002,"5min":2.8839999999999999,"15min":2.1760000000000002},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":20.363,"5min":52.210999999999999,"15min":52.210999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":3.4540000000000002,"5min":3.0070000000000001,"15min":2.2919999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":20.422999999999998,"5min":51.537999999999997,"15min":51.537999999999997},"last":0.67900000000000005}]},{"osd":2,"last update":"Fri Feb 9 18:07:36 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.93600000000000005,"15min":1.6990000000000001},"min":{"1min":0.39900000000000002,"5min":0.247,"15min":0.247},"max":{"1min":1.048,"5min":11.327999999999999,"15min":22.652999999999999},"last":0.436},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.89900000000000002,"15min":1.6359999999999999},"min":{"1min":0.40000000000000002,"5min":0.29599999999999999,"15min":0.28899999999999998},"max":{"1min":1.1839999999999999,"5min":11.298999999999999,"15min":12.180999999999999},"last":0.63800000000000001}]},{"osd":3,"last update":"Fri Feb 9 18:07:17 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":1.3979999999999999,"15min":1.4410000000000001},"min":{"1min":0.35999999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.80300000000000005,"5min":15.117000000000001,"15min":15.117000000000001},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.876,"15min":1.1000000000000001},"min":{"1min":0.33600000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":1.3660000000000001,"5min":5.569,"15min":6.1159999999999997},"last":0.55200000000000005}]},{"osd":4,"last update":"Fri Feb 9 18:07:05 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":1.109,"15min":1.272},"min":{"1min":0.46899999999999997,"5min":0.33300000000000002,"15min":0.253},"max":{"1min":0.90900000000000003,"5min":11.127000000000001,"15min":13.446},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.68999999999999995,"5min":1.069,"15min":1.319},"min":{"1min":0.504,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.264,"5min":7.8029999999999999,"15min":12.097},"last":0.76800000000000002}]},{"osd":5,"last update":"Fri Feb 9 18:07:36 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":1.2070000000000001,"15min":1.1819999999999999},"min":{"1min":0.38800000000000001,"5min":0.34999999999999998,"15min":0.28100000000000003},"max":{"1min":0.98399999999999999,"5min":7.1459999999999999,"15min":13.4},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.82599999999999996,"5min":1.3560000000000001,"15min":1.353},"min":{"1min":0.56899999999999995,"5min":0.36099999999999999,"15min":0.315},"max":{"1min":1.212,"5min":11.068,"15min":12.157999999999999},"last":0.69299999999999995}]},{"osd":6,"last update":"Fri Feb 9 18:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":1.3169999999999999,"15min":1.319},"min":{"1min":0.49399999999999999,"5min":0.40899999999999997,"15min":0.32300000000000001},"max":{"1min":1.2390000000000001,"5min":11.164,"15min":13.507},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.83599999999999997,"5min":1.0569999999999999,"15min":1.369},"min":{"1min":0.56999999999999995,"5min":0.372,"15min":0.35099999999999998},"max":{"1min":1.28,"5min":4.9009999999999998,"15min":12.205},"last":0.73899999999999999}]},{"osd":7,"last update":"Fri Feb 9 18:07:36 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":1.236,"15min":1.3149999999999999},"min":{"1min":0.52000000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.1220000000000001,"5min":5.5970000000000004,"15min":13.481},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.77900000000000003,"5min":1.5349999999999999,"15min":1.5760000000000001},"min":{"1min":0.46600000000000003,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.3939999999999999,"5min":15.074,"15min":15.074},"last":0.67800000000000005}]},{"osd":8,"last update":"Fri Feb 9 18:07:36 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.94299999999999995,"15min":1.95},"min":{"1min":0.39100000000000001,"5min":0.31,"15min":0.31},"max":{"1min":1.2529999999999999,"5min":8.4510000000000005,"15min":22.559000000000001},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.749,"5min":1.0389999999999999,"15min":1.617},"min":{"1min":0.45500000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.198,"5min":7.0919999999999996,"15min":22.824999999999999},"last":0.47899999999999998}]},{"osd":9,"last update":"Fri Feb 9 18:07:36 2024","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":1.363,"15min":2.1749999999999998},"min":{"1min":0.42899999999999999,"5min":0.377,"15min":0.32400000000000001},"max":{"1min":1.304,"5min":15.163,"15min":23.201000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.75800000000000001,"5min":1.2889999999999999,"15min":2.0099999999999998},"min":{"1min":0.42499999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.3220000000000001,"5min":11.196,"15min":17.498999999999999},"last":0.622}]},{"osd":11,"last update":"Fri Feb 9 18:07:17 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":1.4370000000000001,"15min":1.859},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.36799999999999999},"max":{"1min":1.3580000000000001,"5min":11.228,"15min":21.18},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.85499999999999998,"5min":1.5089999999999999,"15min":2.0030000000000001},"min":{"1min":0.57399999999999995,"5min":0.51900000000000002,"15min":0.35899999999999999},"max":{"1min":1.413,"5min":11.266,"15min":32.707000000000001},"last":0.752}]}]},{"osd":9,"up_from":77,"seq":330712482353,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1064800,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":1064062,"kb_avail":92680352,"statfs":{"total":95995035648,"available":94904680448,"internally_reserved":0,"allocated":741376,"data_stored":431582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":1089599936},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 18:07:50 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":1.0309999999999999,"15min":1.1619999999999999},"min":{"1min":0.29899999999999999,"5min":0.254,"15min":0.219},"max":{"1min":0.71299999999999997,"5min":8.4469999999999992,"15min":11.515000000000001},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.67500000000000004,"5min":1.115,"15min":1.042},"min":{"1min":0.36899999999999999,"5min":0.314,"15min":0.23200000000000001},"max":{"1min":0.83599999999999997,"5min":12.521000000000001,"15min":12.521000000000001},"last":0.66800000000000004}]},{"osd":1,"last update":"Fri Feb 9 18:07:29 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":1.1639999999999999,"15min":1.6870000000000001},"min":{"1min":0.38100000000000001,"5min":0.252,"15min":0.23599999999999999},"max":{"1min":0.90900000000000003,"5min":7.4790000000000001,"15min":61.031999999999996},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.66800000000000004,"5min":1.399,"15min":1.921},"min":{"1min":0.499,"5min":0.25900000000000001,"15min":0.25600000000000001},"max":{"1min":1.0720000000000001,"5min":9.0670000000000002,"15min":61.970999999999997},"last":0.71899999999999997}]},{"osd":2,"last update":"Fri Feb 9 18:07:24 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":1.1519999999999999,"15min":1.605},"min":{"1min":0.40200000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":1.169,"5min":7.6820000000000004,"15min":7.6820000000000004},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.67000000000000004,"5min":1.288,"15min":1.5920000000000001},"min":{"1min":0.35199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.3380000000000001,"5min":7.8620000000000001,"15min":7.8620000000000001},"last":0.42299999999999999}]},{"osd":3,"last update":"Fri Feb 9 18:07:29 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":1.2769999999999999,"15min":1.1160000000000001},"min":{"1min":0.35699999999999998,"5min":0.28499999999999998,"15min":0.22500000000000001},"max":{"1min":1.216,"5min":6.2590000000000003,"15min":6.3399999999999999},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.65400000000000003,"5min":1.329,"15min":1.095},"min":{"1min":0.44,"5min":0.32500000000000001,"15min":0.28100000000000003},"max":{"1min":0.89600000000000002,"5min":7.9429999999999996,"15min":7.9429999999999996},"last":0.48899999999999999}]},{"osd":4,"last update":"Fri Feb 9 18:07:36 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.998,"15min":1.4079999999999999},"min":{"1min":0.57499999999999996,"5min":0.29999999999999999,"15min":0.26900000000000002},"max":{"1min":0.96399999999999997,"5min":5.6020000000000003,"15min":19.414000000000001},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.82099999999999995,"5min":1.0580000000000001,"15min":1.2549999999999999},"min":{"1min":0.54900000000000004,"5min":0.39700000000000002,"15min":0.29999999999999999},"max":{"1min":1.038,"5min":7.7169999999999996,"15min":19.224},"last":0.89500000000000002}]},{"osd":5,"last update":"Fri Feb 9 18:07:36 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":1.0660000000000001,"15min":1.349},"min":{"1min":0.505,"5min":0.33600000000000002,"15min":0.33200000000000002},"max":{"1min":1.1299999999999999,"5min":7.8899999999999997,"15min":10.763999999999999},"last":0.93600000000000005},{"interface":"front","average":{"1min":0.78600000000000003,"5min":1.1930000000000001,"15min":1.373},"min":{"1min":0.54900000000000004,"5min":0.32700000000000001,"15min":0.316},"max":{"1min":1.087,"5min":7.7969999999999997,"15min":10.851000000000001},"last":0.998}]},{"osd":6,"last update":"Fri Feb 9 18:07:29 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":1.385,"15min":1.5009999999999999},"min":{"1min":0.46200000000000002,"5min":0.318,"15min":0.318},"max":{"1min":0.99199999999999999,"5min":9.1980000000000004,"15min":19.375},"last":0.874},{"interface":"front","average":{"1min":0.77300000000000002,"5min":1.462,"15min":1.534},"min":{"1min":0.51400000000000001,"5min":0.38200000000000001,"15min":0.34999999999999998},"max":{"1min":1.1279999999999999,"5min":9.1449999999999996,"15min":19.178000000000001},"last":0.91800000000000004}]},{"osd":7,"last update":"Fri Feb 9 18:02:55 2024","interfaces":[{"interface":"back","average":{"1min":1.9570000000000001,"5min":1.724,"15min":2.0059999999999998},"min":{"1min":0.35699999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":8.1899999999999995,"5min":8.8529999999999998,"15min":10.021000000000001},"last":0.623},{"interface":"front","average":{"1min":1.605,"5min":2.004,"15min":1.972},"min":{"1min":0.46000000000000002,"5min":0.38100000000000001,"15min":0.379},"max":{"1min":7.7839999999999998,"5min":10.802,"15min":11.138},"last":0.748}]},{"osd":8,"last update":"Fri Feb 9 18:07:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":1.343,"15min":1.5780000000000001},"min":{"1min":0.40300000000000002,"5min":0.31900000000000001,"15min":0.30599999999999999},"max":{"1min":1.036,"5min":14.042,"15min":14.042},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.75800000000000001,"5min":1.4490000000000001,"15min":1.6830000000000001},"min":{"1min":0.53800000000000003,"5min":0.39500000000000002,"15min":0.35499999999999998},"max":{"1min":1.242,"5min":14.087,"15min":14.087},"last":0.80800000000000005}]},{"osd":10,"last update":"Fri Feb 9 18:07:53 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":1.27,"15min":1.881},"min":{"1min":0.45100000000000001,"5min":0.35399999999999998,"15min":0.28100000000000003},"max":{"1min":1.0009999999999999,"5min":7.8300000000000001,"15min":10.333},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.252,"15min":1.972},"min":{"1min":0.46899999999999997,"5min":0.371,"15min":0.35999999999999999},"max":{"1min":1.3600000000000001,"5min":8.3870000000000005,"15min":18.579999999999998},"last":0.97399999999999998}]},{"osd":11,"last update":"Fri Feb 9 18:07:53 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":1.0920000000000001,"15min":1.7969999999999999},"min":{"1min":0.51200000000000001,"5min":0.42299999999999999,"15min":0.35299999999999998},"max":{"1min":1.1639999999999999,"5min":9.7149999999999999,"15min":40.259999999999998},"last":1.024},{"interface":"front","average":{"1min":0.86799999999999999,"5min":1.163,"15min":2.012},"min":{"1min":0.58699999999999997,"5min":0.46400000000000002,"15min":0.40400000000000003},"max":{"1min":1.2090000000000001,"5min":9.7400000000000002,"15min":39.683},"last":1.0700000000000001}]}]},{"osd":8,"up_from":171,"seq":734439408096,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":537568,"kb_used_data":724,"kb_used_omap":212,"kb_used_meta":536619,"kb_avail":93207584,"statfs":{"total":95995035648,"available":95444566016,"internally_reserved":0,"allocated":741376,"data_stored":431582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217579,"internal_metadata":549498389},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 18:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.81100000000000005,"15min":1.5049999999999999},"min":{"1min":0.34699999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.4490000000000001,"5min":6.0190000000000001,"15min":10.832000000000001},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.65600000000000003,"5min":1.077,"15min":1.462},"min":{"1min":0.40500000000000003,"5min":0.26300000000000001,"15min":0.23400000000000001},"max":{"1min":1.645,"5min":13.590999999999999,"15min":18.081},"last":0.55800000000000005}]},{"osd":1,"last update":"Fri Feb 9 18:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":1.6919999999999999,"15min":1.663},"min":{"1min":0.35399999999999998,"5min":0.26900000000000002,"15min":0.23799999999999999},"max":{"1min":2.1000000000000001,"5min":39.734000000000002,"15min":39.734000000000002},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.63500000000000001,"5min":1.7729999999999999,"15min":1.9890000000000001},"min":{"1min":0.35399999999999998,"5min":0.23599999999999999,"15min":0.23200000000000001},"max":{"1min":1.867,"5min":39.097000000000001,"15min":39.097000000000001},"last":0.41499999999999998}]},{"osd":2,"last update":"Fri Feb 9 18:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":1.1850000000000001,"15min":1.4379999999999999},"min":{"1min":0.313,"5min":0.28100000000000003,"15min":0.25},"max":{"1min":1.5009999999999999,"5min":13.741,"15min":13.741},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.66300000000000003,"5min":1.3939999999999999,"15min":1.764},"min":{"1min":0.33300000000000002,"5min":0.316,"15min":0.27300000000000002},"max":{"1min":1.55,"5min":13.803000000000001,"15min":18.045000000000002},"last":0.52900000000000003}]},{"osd":3,"last update":"Fri Feb 9 17:56:20 2024","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":2.552,"15min":2.8039999999999998},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":5.2560000000000002,"5min":15.654,"15min":15.654},"last":6.5350000000000001},{"interface":"front","average":{"1min":1.306,"5min":2.3149999999999999,"15min":2.0150000000000001},"min":{"1min":0.34300000000000003,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":5.2759999999999998,"5min":15.609999999999999,"15min":15.609999999999999},"last":6.5599999999999996}]},{"osd":4,"last update":"Fri Feb 9 18:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":1.234,"15min":1.3},"min":{"1min":0.35299999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.03,"5min":9.8610000000000007,"15min":12.297000000000001},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.68200000000000005,"5min":1.2769999999999999,"15min":1.304},"min":{"1min":0.40400000000000003,"5min":0.33100000000000002,"15min":0.28499999999999998},"max":{"1min":1.145,"5min":9.9160000000000004,"15min":12.227},"last":0.73099999999999998}]},{"osd":5,"last update":"Fri Feb 9 18:08:01 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":1.1479999999999999,"15min":1.3919999999999999},"min":{"1min":0.51000000000000001,"5min":0.29599999999999999,"15min":0.28399999999999997},"max":{"1min":0.94499999999999995,"5min":13.68,"15min":13.68},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.79500000000000004,"5min":1.181,"15min":1.3859999999999999},"min":{"1min":0.62,"5min":0.377,"15min":0.35499999999999998},"max":{"1min":0.95399999999999996,"5min":13.773,"15min":13.773},"last":0.80600000000000005}]},{"osd":6,"last update":"Fri Feb 9 18:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":1.532,"15min":1.4830000000000001},"min":{"1min":0.52900000000000003,"5min":0.35099999999999998,"15min":0.32700000000000001},"max":{"1min":1.0680000000000001,"5min":9.8879999999999999,"15min":9.8879999999999999},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.77000000000000002,"5min":1.0249999999999999,"15min":1.4450000000000001},"min":{"1min":0.54300000000000004,"5min":0.30299999999999999,"15min":0.28599999999999998},"max":{"1min":1.036,"5min":5.8440000000000003,"15min":10.178000000000001},"last":0.77200000000000002}]},{"osd":7,"last update":"Fri Feb 9 18:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":1.165,"15min":1.518},"min":{"1min":0.56799999999999995,"5min":0.34799999999999998,"15min":0.32900000000000001},"max":{"1min":1.0169999999999999,"5min":13.709,"15min":13.709},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.76800000000000002,"5min":1.081,"15min":1.4550000000000001},"min":{"1min":0.55800000000000005,"5min":0.313,"15min":0.309},"max":{"1min":1.0680000000000001,"5min":5.9870000000000001,"15min":10.148999999999999},"last":0.79300000000000004}]},{"osd":9,"last update":"Fri Feb 9 18:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":1.1739999999999999,"15min":2.2330000000000001},"min":{"1min":0.38500000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.0269999999999999,"5min":14.186,"15min":46.213999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.65800000000000003,"5min":1.111,"15min":2.2280000000000002},"min":{"1min":0.44700000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.89600000000000002,"5min":15.712999999999999,"15min":41.177},"last":0.65000000000000002}]},{"osd":10,"last update":"Fri Feb 9 18:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":1.2609999999999999,"15min":1.992},"min":{"1min":0.432,"5min":0.379,"15min":0.34599999999999997},"max":{"1min":1.2769999999999999,"5min":9.1639999999999997,"15min":18.164000000000001},"last":0.747},{"interface":"front","average":{"1min":0.74399999999999999,"5min":1.351,"15min":1.944},"min":{"1min":0.48899999999999999,"5min":0.40699999999999997,"15min":0.40200000000000002},"max":{"1min":1.296,"5min":9.8309999999999995,"15min":18.111000000000001},"last":0.58199999999999996}]},{"osd":11,"last update":"Fri Feb 9 18:08:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.89900000000000002,"15min":1.7230000000000001},"min":{"1min":0.42699999999999999,"5min":0.36299999999999999,"15min":0.33300000000000002},"max":{"1min":0.97699999999999998,"5min":5.9269999999999996,"15min":10.231999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.98999999999999999,"15min":1.7809999999999999},"min":{"1min":0.56899999999999995,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.1319999999999999,"5min":5.9550000000000001,"15min":10.256},"last":0.81999999999999995}]}]},{"osd":7,"up_from":57,"seq":244813136442,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":912296,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":911358,"kb_avail":92832856,"statfs":{"total":95995035648,"available":95060844544,"internally_reserved":0,"allocated":946176,"data_stored":632830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":933231050},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 18:07:07 2024","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.1839999999999999,"15min":1.018},"min":{"1min":0.34699999999999998,"5min":0.23899999999999999,"15min":0.23100000000000001},"max":{"1min":12.617000000000001,"5min":12.617000000000001,"15min":12.617000000000001},"last":0.78000000000000003},{"interface":"front","average":{"1min":1.3260000000000001,"5min":1.6639999999999999,"15min":1.2390000000000001},"min":{"1min":0.439,"5min":0.34200000000000003,"15min":0.28499999999999998},"max":{"1min":12.605,"5min":12.605,"15min":15.890000000000001},"last":0.63300000000000001}]},{"osd":1,"last update":"Fri Feb 9 18:07:07 2024","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.952,"15min":1.4870000000000001},"min":{"1min":0.37,"5min":0.309,"15min":0.28299999999999997},"max":{"1min":12.132,"5min":18.062999999999999,"15min":18.062999999999999},"last":0.85899999999999999},{"interface":"front","average":{"1min":1.3100000000000001,"5min":2.0099999999999998,"15min":1.4610000000000001},"min":{"1min":0.41199999999999998,"5min":0.26600000000000001,"15min":0.246},"max":{"1min":12.115,"5min":18.315000000000001,"15min":18.315000000000001},"last":0.79100000000000004}]},{"osd":2,"last update":"Fri Feb 9 18:07:07 2024","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.821,"15min":1.524},"min":{"1min":0.55000000000000004,"5min":0.33000000000000002,"15min":0.27300000000000002},"max":{"1min":13.055,"5min":13.055,"15min":16.190000000000001},"last":0.59799999999999998},{"interface":"front","average":{"1min":1.48,"5min":2.1299999999999999,"15min":1.6479999999999999},"min":{"1min":0.54100000000000004,"5min":0.36099999999999999,"15min":0.28999999999999998},"max":{"1min":13.101000000000001,"5min":17.596,"15min":17.596},"last":0.82899999999999996}]},{"osd":3,"last update":"Fri Feb 9 18:07:32 2024","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.72,"15min":1.4039999999999999},"min":{"1min":0.47199999999999998,"5min":0.307,"15min":0.307},"max":{"1min":7.0469999999999997,"5min":10.881,"15min":10.881},"last":0.68300000000000005},{"interface":"front","average":{"1min":1.373,"5min":1.804,"15min":2.0579999999999998},"min":{"1min":0.42799999999999999,"5min":0.40300000000000002,"15min":0.39500000000000002},"max":{"1min":12.179,"5min":14.202,"15min":16.029},"last":0.66900000000000004}]},{"osd":4,"last update":"Fri Feb 9 18:07:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":1.117,"15min":0.81899999999999995},"min":{"1min":0.34300000000000003,"5min":0.312,"15min":0.27100000000000002},"max":{"1min":0.83399999999999996,"5min":6.0999999999999996,"15min":6.0999999999999996},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.69099999999999995,"5min":1.371,"15min":0.98099999999999998},"min":{"1min":0.44,"5min":0.28000000000000003,"15min":0.26300000000000001},"max":{"1min":0.89700000000000002,"5min":11.433999999999999,"15min":17.814},"last":0.80600000000000005}]},{"osd":5,"last update":"Fri Feb 9 18:07:07 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":1.296,"15min":0.93700000000000006},"min":{"1min":0.34499999999999997,"5min":0.29799999999999999,"15min":0.27500000000000002},"max":{"1min":0.873,"5min":11.406000000000001,"15min":11.406000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.69599999999999995,"5min":1.3169999999999999,"15min":1.014},"min":{"1min":0.45400000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.93000000000000005,"5min":7.6719999999999997,"15min":7.6719999999999997},"last":0.72399999999999998}]},{"osd":6,"last update":"Fri Feb 9 18:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":1.262,"15min":1.0780000000000001},"min":{"1min":0.50900000000000001,"5min":0.34599999999999997,"15min":0.311},"max":{"1min":0.94599999999999995,"5min":8.0999999999999996,"15min":16.158000000000001},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.74299999999999999,"5min":1.29,"15min":0.999},"min":{"1min":0.40799999999999997,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.0169999999999999,"5min":8.1259999999999994,"15min":8.1259999999999994},"last":0.92200000000000004}]},{"osd":8,"last update":"Fri Feb 9 18:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":1.2949999999999999,"15min":1.3839999999999999},"min":{"1min":0.63700000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.099,"5min":11.358000000000001,"15min":11.648999999999999},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.83199999999999996,"5min":1.393,"15min":1.494},"min":{"1min":0.55800000000000005,"5min":0.41699999999999998,"15min":0.375},"max":{"1min":1.0700000000000001,"5min":7.6539999999999999,"15min":11.629},"last":0.98899999999999999}]},{"osd":9,"last update":"Fri Feb 9 18:07:19 2024","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":1.3819999999999999,"15min":1.5449999999999999},"min":{"1min":0.626,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.1299999999999999,"5min":11.465,"15min":11.465},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.93600000000000005,"5min":1.464,"15min":1.7070000000000001},"min":{"1min":0.58799999999999997,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.1339999999999999,"5min":11.548999999999999,"15min":13.161},"last":1.0229999999999999}]},{"osd":10,"last update":"Fri Feb 9 18:07:58 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":1.319,"15min":1.599},"min":{"1min":0.63200000000000001,"5min":0.42499999999999999,"15min":0.32800000000000001},"max":{"1min":1.0700000000000001,"5min":7.694,"15min":15.999000000000001},"last":1.008},{"interface":"front","average":{"1min":0.85099999999999998,"5min":1.472,"15min":1.5980000000000001},"min":{"1min":0.54700000000000004,"5min":0.48199999999999998,"15min":0.40400000000000003},"max":{"1min":1.0660000000000001,"5min":11.515000000000001,"15min":11.515000000000001},"last":1.0660000000000001}]},{"osd":11,"last update":"Fri Feb 9 18:07:07 2024","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":1.593,"15min":1.704},"min":{"1min":0.625,"5min":0.44400000000000001,"15min":0.375},"max":{"1min":1.1299999999999999,"5min":9.1890000000000001,"15min":16.111999999999998},"last":0.95199999999999996},{"interface":"front","average":{"1min":0.89700000000000002,"5min":1.639,"15min":1.7350000000000001},"min":{"1min":0.69599999999999995,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.095,"5min":10.215999999999999,"15min":12.243},"last":1.0409999999999999}]}]},{"osd":6,"up_from":49,"seq":210453398082,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":718048,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":716990,"kb_avail":93027104,"statfs":{"total":95995035648,"available":95259754496,"internally_reserved":0,"allocated":1069056,"data_stored":759262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":734198214},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 18:07:53 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.84899999999999998,"15min":0.88100000000000001},"min":{"1min":0.36099999999999999,"5min":0.27600000000000002,"15min":0.23599999999999999},"max":{"1min":1.234,"5min":7.3630000000000004,"15min":10.356},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.84599999999999997,"15min":0.85399999999999998},"min":{"1min":0.441,"5min":0.27200000000000002,"15min":0.26700000000000002},"max":{"1min":0.98699999999999999,"5min":7.407,"15min":8.9459999999999997},"last":0.74299999999999999}]},{"osd":1,"last update":"Fri Feb 9 18:07:53 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":1.3759999999999999,"15min":1.1240000000000001},"min":{"1min":0.51300000000000001,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.0509999999999999,"5min":32.064,"15min":32.064},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.78300000000000003,"5min":1.4430000000000001,"15min":1.1850000000000001},"min":{"1min":0.51600000000000001,"5min":0.34100000000000003,"15min":0.309},"max":{"1min":1.298,"5min":32.027000000000001,"15min":32.027000000000001},"last":0.69699999999999995}]},{"osd":2,"last update":"Fri Feb 9 18:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":1.032,"15min":1.1359999999999999},"min":{"1min":0.45800000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.038,"5min":7.4779999999999998,"15min":9.2970000000000006},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.79700000000000004,"5min":1.0980000000000001,"15min":1.1899999999999999},"min":{"1min":0.48999999999999999,"5min":0.35699999999999998,"15min":0.34999999999999998},"max":{"1min":1.3620000000000001,"5min":7.5659999999999998,"15min":9.3260000000000005},"last":0.88700000000000001}]},{"osd":3,"last update":"Fri Feb 9 18:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":1.002,"15min":0.90700000000000003},"min":{"1min":0.33300000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.91600000000000004,"5min":8.6880000000000006,"15min":8.6880000000000006},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.627,"5min":0.81799999999999995,"15min":1.036},"min":{"1min":0.41099999999999998,"5min":0.36199999999999999,"15min":0.33500000000000002},"max":{"1min":1.0940000000000001,"5min":8.1760000000000002,"15min":9.0909999999999993},"last":0.63400000000000001}]},{"osd":4,"last update":"Fri Feb 9 18:07:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.80200000000000005,"15min":0.71099999999999997},"min":{"1min":0.28699999999999998,"5min":0.28100000000000003,"15min":0.26600000000000001},"max":{"1min":0.99199999999999999,"5min":10.215,"15min":10.215},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.89100000000000001,"15min":0.78200000000000003},"min":{"1min":0.378,"5min":0.36599999999999999,"15min":0.33100000000000002},"max":{"1min":1.1299999999999999,"5min":10.256,"15min":10.256},"last":0.89700000000000002}]},{"osd":5,"last update":"Fri Feb 9 18:07:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.82399999999999995,"15min":0.69499999999999995},"min":{"1min":0.35699999999999998,"5min":0.34599999999999997,"15min":0.26600000000000001},"max":{"1min":0.78800000000000003,"5min":8.1389999999999993,"15min":8.1389999999999993},"last":0.871},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.95199999999999996,"15min":0.749},"min":{"1min":0.35799999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.4079999999999999,"5min":7.4470000000000001,"15min":7.4470000000000001},"last":0.91700000000000004}]},{"osd":7,"last update":"Fri Feb 9 18:07:36 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.96499999999999997,"15min":0.83099999999999996},"min":{"1min":0.376,"5min":0.30199999999999999,"15min":0.29499999999999998},"max":{"1min":0.92900000000000005,"5min":13.991,"15min":13.991},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.96899999999999997,"15min":0.82999999999999996},"min":{"1min":0.39900000000000002,"5min":0.38500000000000001,"15min":0.30299999999999999},"max":{"1min":1.0049999999999999,"5min":8.4870000000000001,"15min":8.4870000000000001},"last":0.83499999999999996}]},{"osd":8,"last update":"Fri Feb 9 17:58:34 2024","interfaces":[{"interface":"back","average":{"1min":2.806,"5min":1.635,"15min":1.27},"min":{"1min":0.41599999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":9.2940000000000005,"5min":9.2940000000000005,"15min":9.9480000000000004},"last":0.40300000000000002},{"interface":"front","average":{"1min":2.3780000000000001,"5min":1.623,"15min":1.3109999999999999},"min":{"1min":0.53400000000000003,"5min":0.41399999999999998,"15min":0.40400000000000003},"max":{"1min":9.3469999999999995,"5min":9.3469999999999995,"15min":11.316000000000001},"last":0.57899999999999996}]},{"osd":9,"last update":"Fri Feb 9 18:07:04 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":1.099,"15min":1.5780000000000001},"min":{"1min":0.52800000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.075,"5min":7.5090000000000003,"15min":36.606000000000002},"last":1.089},{"interface":"front","average":{"1min":0.85899999999999999,"5min":1.071,"15min":1.7969999999999999},"min":{"1min":0.56999999999999995,"5min":0.437,"15min":0.34300000000000003},"max":{"1min":1.119,"5min":7.5380000000000003,"15min":34.408000000000001},"last":0.98399999999999999}]},{"osd":10,"last update":"Fri Feb 9 18:07:36 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.996,"15min":1.4430000000000001},"min":{"1min":0.54500000000000004,"5min":0.48399999999999999,"15min":0.36599999999999999},"max":{"1min":1.2689999999999999,"5min":8.4309999999999992,"15min":21.364999999999998},"last":1.1080000000000001},{"interface":"front","average":{"1min":0.91300000000000003,"5min":1.028,"15min":1.458},"min":{"1min":0.58199999999999996,"5min":0.502,"15min":0.40100000000000002},"max":{"1min":1.3280000000000001,"5min":8.5380000000000003,"15min":21.408999999999999},"last":1.0780000000000001}]},{"osd":11,"last update":"Fri Feb 9 18:07:53 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":1.171,"15min":1.6120000000000001},"min":{"1min":0.505,"5min":0.435,"15min":0.435},"max":{"1min":1.085,"5min":9.6859999999999999,"15min":35.912999999999997},"last":0.97099999999999997},{"interface":"front","average":{"1min":0.89000000000000001,"5min":1.214,"15min":1.6579999999999999},"min":{"1min":0.56399999999999995,"5min":0.51200000000000001,"15min":0.48699999999999999},"max":{"1min":1.214,"5min":9.7300000000000004,"15min":36.156999999999996},"last":1.008}]}]},{"osd":1,"up_from":13,"seq":55834575449,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1233764,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":1233022,"kb_avail":92511388,"statfs":{"total":95995035648,"available":94731661312,"internally_reserved":0,"allocated":741376,"data_stored":431582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1262614981},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 18:07:16 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":1.046,"15min":1.375},"min":{"1min":0.31,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":0.63600000000000001,"5min":6.6079999999999997,"15min":14.885999999999999},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.51400000000000001,"5min":1.204,"15min":1.52},"min":{"1min":0.36599999999999999,"5min":0.26900000000000002,"15min":0.246},"max":{"1min":0.78100000000000003,"5min":9.6969999999999992,"15min":15.411},"last":0.44700000000000001}]},{"osd":2,"last update":"Fri Feb 9 18:07:35 2024","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.262,"15min":1.4950000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.28299999999999997},"max":{"1min":11.683999999999999,"5min":11.683999999999999,"15min":13.173},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.88700000000000001,"5min":1.1779999999999999,"15min":1.349},"min":{"1min":0.40200000000000002,"5min":0.32000000000000001,"15min":0.28100000000000003},"max":{"1min":6.7149999999999999,"5min":7.8090000000000002,"15min":11.103},"last":0.56999999999999995}]},{"osd":3,"last update":"Fri Feb 9 18:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":1.2969999999999999,"15min":0.98699999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.55500000000000005,"5min":13.292999999999999,"15min":13.292999999999999},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.46200000000000002,"5min":1.3640000000000001,"15min":1.3169999999999999},"min":{"1min":0.34300000000000003,"5min":0.29099999999999998,"15min":0.28999999999999998},"max":{"1min":0.69399999999999995,"5min":13.265000000000001,"15min":13.265000000000001},"last":0.432}]},{"osd":4,"last update":"Fri Feb 9 18:07:19 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":1.4299999999999999,"15min":1.2210000000000001},"min":{"1min":0.46999999999999997,"5min":0.38600000000000001,"15min":0.27600000000000002},"max":{"1min":0.84099999999999997,"5min":13.151,"15min":13.151},"last":0.5},{"interface":"front","average":{"1min":0.629,"5min":1.6679999999999999,"15min":1.5609999999999999},"min":{"1min":0.46800000000000003,"5min":0.33800000000000002,"15min":0.26400000000000001},"max":{"1min":0.80100000000000005,"5min":13.177,"15min":15.331},"last":0.55600000000000005}]},{"osd":5,"last update":"Fri Feb 9 18:07:54 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":1.0720000000000001,"15min":1.3100000000000001},"min":{"1min":0.46300000000000002,"5min":0.371,"15min":0.29399999999999998},"max":{"1min":1.0109999999999999,"5min":4.9859999999999998,"15min":15.106999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.73399999999999999,"5min":1.4330000000000001,"15min":1.478},"min":{"1min":0.57499999999999996,"5min":0.318,"15min":0.28000000000000003},"max":{"1min":1.0680000000000001,"5min":13.106,"15min":15.355},"last":0.61899999999999999}]},{"osd":6,"last update":"Fri Feb 9 18:07:07 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":1.4450000000000001,"15min":1.4379999999999999},"min":{"1min":0.48499999999999999,"5min":0.39300000000000002,"15min":0.24299999999999999},"max":{"1min":0.83499999999999996,"5min":10.605,"15min":15.154999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.64600000000000002,"5min":1.371,"15min":1.446},"min":{"1min":0.45200000000000001,"5min":0.376,"15min":0.32800000000000001},"max":{"1min":0.92600000000000005,"5min":10.720000000000001,"15min":15.253},"last":0.69899999999999995}]},{"osd":7,"last update":"Fri Feb 9 18:07:07 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":1.542,"15min":1.5609999999999999},"min":{"1min":0.60299999999999998,"5min":0.40799999999999997,"15min":0.316},"max":{"1min":0.98799999999999999,"5min":10.58,"15min":15.287000000000001},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.71599999999999997,"5min":1.1579999999999999,"15min":1.4490000000000001},"min":{"1min":0.49099999999999999,"5min":0.379,"15min":0.379},"max":{"1min":0.86099999999999999,"5min":5.2919999999999998,"15min":15.311},"last":0.58399999999999996}]},{"osd":8,"last update":"Fri Feb 9 18:07:16 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":1.24,"15min":1.6379999999999999},"min":{"1min":0.504,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.089,"5min":10.627000000000001,"15min":10.627000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.82399999999999995,"5min":1.2050000000000001,"15min":1.867},"min":{"1min":0.51400000000000001,"5min":0.44500000000000001,"15min":0.36399999999999999},"max":{"1min":1.2250000000000001,"5min":5.7800000000000002,"15min":12.464},"last":0.67900000000000005}]},{"osd":9,"last update":"Fri Feb 9 18:07:16 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":1.363,"15min":1.9470000000000001},"min":{"1min":0.46300000000000002,"5min":0.42999999999999999,"15min":0.42799999999999999},"max":{"1min":1.125,"5min":10.337999999999999,"15min":20.789000000000001},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.80600000000000005,"5min":1.804,"15min":1.8779999999999999},"min":{"1min":0.51400000000000001,"5min":0.375,"15min":0.34200000000000003},"max":{"1min":1.1020000000000001,"5min":13.231999999999999,"15min":13.231999999999999},"last":0.68999999999999995}]},{"osd":10,"last update":"Fri Feb 9 18:07:16 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.81599999999999995,"15min":0.81399999999999995},"min":{"1min":0.63100000000000001,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.871,"15min":0.86799999999999999},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.78800000000000003}]},{"osd":11,"last update":"Fri Feb 9 18:07:07 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":1.5449999999999999,"15min":2.2650000000000001},"min":{"1min":0.63400000000000001,"5min":0.441,"15min":0.378},"max":{"1min":1.175,"5min":10.593999999999999,"15min":16.881},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.82899999999999996,"5min":2.1080000000000001,"15min":2.3759999999999999},"min":{"1min":0.61899999999999999,"5min":0.41799999999999998,"15min":0.372},"max":{"1min":1.147,"5min":13.205,"15min":15.209},"last":0.67000000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738973,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":902664,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":901630,"kb_avail":92842488,"statfs":{"total":95995035648,"available":95070707712,"internally_reserved":0,"allocated":1044480,"data_stored":744180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":923269573},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 9 18:07:56 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.76300000000000001,"15min":1.417},"min":{"1min":0.30399999999999999,"5min":0.25900000000000001,"15min":0.23100000000000001},"max":{"1min":0.54700000000000004,"5min":6.8070000000000004,"15min":23.282},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.84999999999999998,"15min":1.496},"min":{"1min":0.38600000000000001,"5min":0.35499999999999998,"15min":0.29199999999999998},"max":{"1min":0.63600000000000001,"5min":6.8410000000000002,"15min":23.32},"last":0.50800000000000001}]},{"osd":2,"last update":"Fri Feb 9 18:07:30 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.95999999999999996,"15min":1.4690000000000001},"min":{"1min":0.29499999999999998,"5min":0.27400000000000002,"15min":0.253},"max":{"1min":0.68899999999999995,"5min":11.161,"15min":20.013999999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.88100000000000001,"15min":1.462},"min":{"1min":0.39800000000000002,"5min":0.27100000000000002,"15min":0.219},"max":{"1min":0.67200000000000004,"5min":6.8730000000000002,"15min":20.103999999999999},"last":0.54300000000000004}]},{"osd":3,"last update":"Fri Feb 9 18:07:30 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":1.599,"15min":1.4570000000000001},"min":{"1min":0.318,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.59899999999999998,"5min":45.688000000000002,"15min":45.688000000000002},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.498,"5min":1.6160000000000001,"15min":1.7549999999999999},"min":{"1min":0.38200000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.69799999999999995,"5min":45.551000000000002,"15min":45.551000000000002},"last":0.47099999999999997}]},{"osd":4,"last update":"Fri Feb 9 18:07:30 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":1.0109999999999999,"15min":1.2490000000000001},"min":{"1min":0.439,"5min":0.33900000000000002,"15min":0.28100000000000003},"max":{"1min":0.99399999999999999,"5min":11.393000000000001,"15min":12.757},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.752,"5min":1.1100000000000001,"15min":1.4470000000000001},"min":{"1min":0.53700000000000003,"5min":0.39000000000000001,"15min":0.25800000000000001},"max":{"1min":1.0429999999999999,"5min":11.114000000000001,"15min":17.091999999999999},"last":0.68400000000000005}]},{"osd":5,"last update":"Fri Feb 9 18:07:42 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":1.1080000000000001,"15min":1.2230000000000001},"min":{"1min":0.47799999999999998,"5min":0.38500000000000001,"15min":0.30199999999999999},"max":{"1min":1.1850000000000001,"5min":11.435,"15min":11.435},"last":0.628},{"interface":"front","average":{"1min":0.76500000000000001,"5min":1.1879999999999999,"15min":1.292},"min":{"1min":0.51100000000000001,"5min":0.44,"15min":0.38300000000000001},"max":{"1min":1.153,"5min":11.462999999999999,"15min":11.462999999999999},"last":0.73899999999999999}]},{"osd":6,"last update":"Fri Feb 9 18:07:10 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":1.01,"15min":1.357},"min":{"1min":0.48299999999999998,"5min":0.313,"15min":0.307},"max":{"1min":1.069,"5min":6.069,"15min":15.667},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.90400000000000003,"15min":1.2949999999999999},"min":{"1min":0.46100000000000002,"5min":0.432,"15min":0.34799999999999998},"max":{"1min":0.86199999999999999,"5min":4.5529999999999999,"15min":15.702999999999999},"last":0.751}]},{"osd":7,"last update":"Fri Feb 9 18:07:42 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.98899999999999999,"15min":1.409},"min":{"1min":0.42299999999999999,"5min":0.39400000000000002,"15min":0.307},"max":{"1min":0.876,"5min":5.9729999999999999,"15min":11.682},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.90400000000000003,"15min":1.3779999999999999},"min":{"1min":0.495,"5min":0.46000000000000002,"15min":0.32600000000000001},"max":{"1min":0.92800000000000005,"5min":4.7729999999999997,"15min":17.152999999999999},"last":0.64100000000000001}]},{"osd":8,"last update":"Fri Feb 9 18:07:42 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.97899999999999998,"15min":1.738},"min":{"1min":0.57999999999999996,"5min":0.40600000000000003,"15min":0.33800000000000002},"max":{"1min":1.2150000000000001,"5min":4.4290000000000003,"15min":17.189},"last":0.91800000000000004},{"interface":"front","average":{"1min":0.86099999999999999,"5min":1.046,"15min":1.6950000000000001},"min":{"1min":0.623,"5min":0.47499999999999998,"15min":0.36499999999999999},"max":{"1min":1.1439999999999999,"5min":5.7939999999999996,"15min":18.600000000000001},"last":0.84099999999999997}]},{"osd":9,"last update":"Fri Feb 9 18:07:30 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":1.252,"15min":2.0430000000000001},"min":{"1min":0.67700000000000005,"5min":0.443,"15min":0.39000000000000001},"max":{"1min":1.1739999999999999,"5min":6.6070000000000002,"15min":17.216000000000001},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.88900000000000001,"5min":1.0900000000000001,"15min":1.7949999999999999},"min":{"1min":0.66100000000000003,"5min":0.50700000000000001,"15min":0.40899999999999997},"max":{"1min":1.0800000000000001,"5min":5.8159999999999998,"15min":19.986999999999998},"last":0.85999999999999999}]},{"osd":10,"last update":"Fri Feb 9 18:07:56 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.97299999999999998,"15min":1.996},"min":{"1min":0.66000000000000003,"5min":0.432,"15min":0.42499999999999999},"max":{"1min":1.077,"5min":4.4630000000000001,"15min":57.979999999999997},"last":0.94799999999999995},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.96399999999999997,"15min":2.153},"min":{"1min":0.58899999999999997,"5min":0.46200000000000002,"15min":0.41299999999999998},"max":{"1min":1.1579999999999999,"5min":4.5880000000000001,"15min":57.606999999999999},"last":0.88600000000000001}]},{"osd":11,"last update":"Fri Feb 9 18:07:10 2024","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":1.0740000000000001,"15min":1.9690000000000001},"min":{"1min":0.55200000000000005,"5min":0.47599999999999998,"15min":0.44800000000000001},"max":{"1min":1.1859999999999999,"5min":6.3899999999999997,"15min":58.351999999999997},"last":0.97099999999999997},{"interface":"front","average":{"1min":0.90000000000000002,"5min":1.145,"15min":1.9750000000000001},"min":{"1min":0.64000000000000001,"5min":0.499,"15min":0.441},"max":{"1min":1.1299999999999999,"5min":6.4459999999999997,"15min":57.314},"last":0.99299999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411923,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":690152,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93055000,"statfs":{"total":95995035648,"available":95288320000,"internally_reserved":0,"allocated":946176,"data_stored":632830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:58:48 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.53200000000000003,"15min":0.48699999999999999},"min":{"1min":0.23699999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.9619999999999997,"5min":4.9619999999999997,"15min":4.9619999999999997},"last":0.33800000000000002},{"interface":"front","average":{"1min":1.292,"5min":0.81899999999999995,"15min":0.73999999999999999},"min":{"1min":0.33000000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":4.5430000000000001,"5min":4.5430000000000001,"15min":4.5430000000000001},"last":0.47799999999999998}]},{"osd":1,"last update":"Fri Feb 9 18:07:38 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":1.1399999999999999,"15min":1.367},"min":{"1min":0.25900000000000001,"5min":0.23699999999999999,"15min":0.20799999999999999},"max":{"1min":0.621,"5min":25.167999999999999,"15min":25.167999999999999},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.46200000000000002,"5min":1.2390000000000001,"15min":1.395},"min":{"1min":0.28899999999999998,"5min":0.249,"15min":0.248},"max":{"1min":0.72399999999999998,"5min":25.143000000000001,"15min":25.143000000000001},"last":0.40100000000000002}]},{"osd":3,"last update":"Fri Feb 9 18:07:24 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":1.2889999999999999,"15min":2.347},"min":{"1min":0.308,"5min":0.25,"15min":0.19900000000000001},"max":{"1min":0.66700000000000004,"5min":25.093,"15min":29.228000000000002},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.38700000000000001,"5min":1.1359999999999999,"15min":1.3460000000000001},"min":{"1min":0.27100000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.64300000000000002,"5min":20.093,"15min":20.093},"last":0.35299999999999998}]},{"osd":4,"last update":"Fri Feb 9 18:07:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":1.0009999999999999,"15min":0.999},"min":{"1min":0.47199999999999998,"5min":0.30199999999999999,"15min":0.245},"max":{"1min":0.90200000000000002,"5min":9.609,"15min":10.365},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.69999999999999996,"5min":1.042,"15min":1.073},"min":{"1min":0.442,"5min":0.36699999999999999,"15min":0.26600000000000001},"max":{"1min":0.86099999999999999,"5min":9.6500000000000004,"15min":10.529},"last":0.60799999999999998}]},{"osd":5,"last update":"Fri Feb 9 18:07:55 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":1.232,"15min":1.214},"min":{"1min":0.39700000000000002,"5min":0.29199999999999998,"15min":0.245},"max":{"1min":0.88400000000000001,"5min":11.536,"15min":12.305999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.65300000000000002,"5min":1.2689999999999999,"15min":1.2330000000000001},"min":{"1min":0.42599999999999999,"5min":0.40500000000000003,"15min":0.26900000000000002},"max":{"1min":0.93999999999999995,"5min":11.571,"15min":11.571},"last":0.61899999999999999}]},{"osd":6,"last update":"Fri Feb 9 18:07:10 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":1.2549999999999999,"15min":1.3049999999999999},"min":{"1min":0.377,"5min":0.316,"15min":0.28399999999999997},"max":{"1min":0.94699999999999995,"5min":7.8460000000000001,"15min":11.465999999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.624,"5min":1.278,"15min":1.341},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.28199999999999997},"max":{"1min":0.88300000000000001,"5min":7.9489999999999998,"15min":11.598000000000001},"last":0.46500000000000002}]},{"osd":7,"last update":"Fri Feb 9 18:07:38 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.95899999999999996,"15min":1.0980000000000001},"min":{"1min":0.438,"5min":0.33300000000000002,"15min":0.33100000000000002},"max":{"1min":0.90400000000000003,"5min":4.9770000000000003,"15min":10.403},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.67000000000000004,"5min":1.26,"15min":1.3180000000000001},"min":{"1min":0.47799999999999998,"5min":0.38800000000000001,"15min":0.309},"max":{"1min":0.89100000000000001,"5min":11.603,"15min":11.603},"last":0.48899999999999999}]},{"osd":8,"last update":"Fri Feb 9 18:07:10 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.98999999999999999,"15min":1.5920000000000001},"min":{"1min":0.47299999999999998,"5min":0.314,"15min":0.314},"max":{"1min":1.073,"5min":7.5060000000000002,"15min":12.500999999999999},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.752,"5min":1.484,"15min":1.7689999999999999},"min":{"1min":0.41199999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.0189999999999999,"5min":8.1890000000000001,"15min":11.634},"last":0.68899999999999995}]},{"osd":9,"last update":"Fri Feb 9 18:07:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":1.0720000000000001,"15min":1.7310000000000001},"min":{"1min":0.44700000000000001,"5min":0.34499999999999997,"15min":0.318},"max":{"1min":1.0289999999999999,"5min":5.9299999999999997,"15min":12.276},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.77400000000000002,"5min":1.4590000000000001,"15min":1.9019999999999999},"min":{"1min":0.48299999999999998,"5min":0.45200000000000001,"15min":0.36199999999999999},"max":{"1min":1.04,"5min":7.9169999999999998,"15min":12.148},"last":0.64600000000000002}]},{"osd":10,"last update":"Fri Feb 9 18:07:47 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":1.1220000000000001,"15min":1.6819999999999999},"min":{"1min":0.55400000000000005,"5min":0.47199999999999998,"15min":0.40799999999999997},"max":{"1min":1.0509999999999999,"5min":6.3609999999999998,"15min":12.313000000000001},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.79000000000000004,"5min":1.24,"15min":1.716},"min":{"1min":0.60599999999999998,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.97999999999999998,"5min":10.968999999999999,"15min":12.343999999999999},"last":0.73699999999999999}]},{"osd":11,"last update":"Fri Feb 9 18:07:10 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":1.6160000000000001,"15min":1.9279999999999999},"min":{"1min":0.48499999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.0629999999999999,"5min":12.212999999999999,"15min":12.212999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.77700000000000002,"5min":1.3120000000000001,"15min":1.843},"min":{"1min":0.58699999999999997,"5min":0.40200000000000002,"15min":0.35499999999999998},"max":{"1min":1.0449999999999999,"5min":11.922000000000001,"15min":12.243},"last":0.67100000000000004}]}]},{"osd":3,"up_from":854,"seq":3667902070894,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":270240,"kb_used_data":1044,"kb_used_omap":202,"kb_used_meta":268981,"kb_avail":93474912,"statfs":{"total":95995035648,"available":95718309888,"internally_reserved":0,"allocated":1069056,"data_stored":759262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207633,"internal_metadata":275436783},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 18:07:25 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.93100000000000005,"15min":0.998},"min":{"1min":0.246,"5min":0.246,"15min":0.22500000000000001},"max":{"1min":0.56699999999999995,"5min":9.6189999999999998,"15min":9.6189999999999998},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.45400000000000001,"5min":1.0209999999999999,"15min":1.425},"min":{"1min":0.30499999999999999,"5min":0.25800000000000001,"15min":0.20599999999999999},"max":{"1min":0.626,"5min":11.287000000000001,"15min":17.077999999999999},"last":0.34699999999999998}]},{"osd":1,"last update":"Fri Feb 9 18:07:22 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":1.53,"15min":1.1200000000000001},"min":{"1min":0.28199999999999997,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.60799999999999998,"5min":21.73,"15min":21.73},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.45600000000000002,"5min":1.639,"15min":1.022},"min":{"1min":0.28199999999999997,"5min":0.27300000000000002,"15min":0.23899999999999999},"max":{"1min":0.65100000000000002,"5min":19.268000000000001,"15min":19.268000000000001},"last":0.28000000000000003}]},{"osd":2,"last update":"Fri Feb 9 18:07:25 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":1.2230000000000001,"15min":1.948},"min":{"1min":0.28199999999999997,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.72099999999999997,"5min":9.5899999999999999,"15min":13.098000000000001},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.5,"5min":1.2,"15min":1.2569999999999999},"min":{"1min":0.35499999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.74099999999999999,"5min":8.8539999999999992,"15min":9.3260000000000005},"last":0.40799999999999997}]},{"osd":4,"last update":"Fri Feb 9 18:07:25 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":1.252,"15min":1.0289999999999999},"min":{"1min":0.51000000000000001,"5min":0.32700000000000001,"15min":0.28100000000000003},"max":{"1min":0.79800000000000004,"5min":16.329000000000001,"15min":16.329000000000001},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.65500000000000003,"5min":1.0629999999999999,"15min":1.282},"min":{"1min":0.46500000000000002,"5min":0.32800000000000001,"15min":0.318},"max":{"1min":0.97199999999999998,"5min":6.5679999999999996,"15min":10.755000000000001},"last":0.46800000000000003}]},{"osd":5,"last update":"Fri Feb 9 18:07:25 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":1.2450000000000001,"15min":1.1180000000000001},"min":{"1min":0.45700000000000002,"5min":0.34999999999999998,"15min":0.28499999999999998},"max":{"1min":0.97099999999999997,"5min":16.405000000000001,"15min":16.405000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.64600000000000002,"5min":1.0349999999999999,"15min":1.3959999999999999},"min":{"1min":0.50700000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.82599999999999996,"5min":6.6440000000000001,"15min":10.707000000000001},"last":0.48799999999999999}]},{"osd":6,"last update":"Fri Feb 9 18:07:25 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":1.155,"15min":1.657},"min":{"1min":0.49299999999999999,"5min":0.41599999999999998,"15min":0.35899999999999999},"max":{"1min":0.997,"5min":6.4050000000000002,"15min":10.808},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.68999999999999995,"5min":1.075,"15min":0.88},"min":{"1min":0.441,"5min":0.32200000000000001,"15min":0.29799999999999999},"max":{"1min":0.86899999999999999,"5min":5.8920000000000003,"15min":8.3550000000000004},"last":0.45700000000000002}]},{"osd":7,"last update":"Fri Feb 9 18:07:04 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":1.4159999999999999,"15min":1.8580000000000001},"min":{"1min":0.48399999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.88300000000000001,"5min":9.423,"15min":9.423},"last":0.437},{"interface":"front","average":{"1min":0.73099999999999998,"5min":1.5680000000000001,"15min":2.024},"min":{"1min":0.54700000000000004,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.92100000000000004,"5min":16.439,"15min":16.439},"last":0.50800000000000001}]},{"osd":8,"last update":"Fri Feb 9 18:07:25 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":1.3460000000000001,"15min":1.343},"min":{"1min":0.50700000000000001,"5min":0.40400000000000003,"15min":0.34499999999999997},"max":{"1min":1.1359999999999999,"5min":16.228000000000002,"15min":16.228000000000002},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.80100000000000005,"5min":1.21,"15min":1.6259999999999999},"min":{"1min":0.63400000000000001,"5min":0.41399999999999998,"15min":0.38500000000000001},"max":{"1min":1.085,"5min":6.6040000000000001,"15min":10.782},"last":0.58499999999999996}]},{"osd":9,"last update":"Fri Feb 9 18:07:25 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":1.526,"15min":1.028},"min":{"1min":0.51500000000000001,"5min":0.39600000000000002,"15min":0.38200000000000001},"max":{"1min":0.995,"5min":16.172999999999998,"15min":16.172999999999998},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.77500000000000002,"5min":1.462,"15min":1.387},"min":{"1min":0.56899999999999995,"5min":0.42499999999999999,"15min":0.32400000000000001},"max":{"1min":1.119,"5min":9.8350000000000009,"15min":9.8350000000000009},"last":0.56899999999999995}]},{"osd":10,"last update":"Fri Feb 9 18:04:15 2024","interfaces":[{"interface":"back","average":{"1min":2.206,"5min":2.206,"15min":2.206},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":8.7870000000000008,"5min":8.7870000000000008,"15min":8.7870000000000008},"last":0.73099999999999998},{"interface":"front","average":{"1min":3.2040000000000002,"5min":3.2040000000000002,"15min":3.2040000000000002},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":16.263000000000002,"5min":16.263000000000002,"15min":16.263000000000002},"last":0.83199999999999996}]},{"osd":11,"last update":"Fri Feb 9 18:07:25 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":1.3,"15min":1.8839999999999999},"min":{"1min":0.59599999999999997,"5min":0.497,"15min":0.38800000000000001},"max":{"1min":1.0209999999999999,"5min":16.295000000000002,"15min":17.126000000000001},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.79100000000000004,"5min":1.4590000000000001,"15min":1.2070000000000001},"min":{"1min":0.60499999999999998,"5min":0.36199999999999999,"15min":0.34799999999999998},"max":{"1min":1.097,"5min":9.375,"15min":9.375},"last":0.52000000000000002}]}]},{"osd":4,"up_from":32,"seq":137438954060,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":906532,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":905790,"kb_avail":92838620,"statfs":{"total":95995035648,"available":95066746880,"internally_reserved":0,"allocated":741376,"data_stored":431582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":927529413},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:58:54 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.749,"15min":1.2709999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.22800000000000001},"max":{"1min":1.734,"5min":7.7359999999999998,"15min":10.301},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.81799999999999995,"15min":1.3400000000000001},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.768,"5min":7.7729999999999997,"15min":10.407999999999999},"last":0.35599999999999998}]},{"osd":1,"last update":"Fri Feb 9 18:07:14 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.96099999999999997,"15min":1.3620000000000001},"min":{"1min":0.46600000000000003,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.68999999999999995,"5min":6.1340000000000003,"15min":50.255000000000003},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.67700000000000005,"5min":1.034,"15min":1.5600000000000001},"min":{"1min":0.51400000000000001,"5min":0.30099999999999999,"15min":0.27800000000000002},"max":{"1min":0.79400000000000004,"5min":6.2999999999999998,"15min":49.991999999999997},"last":0.71299999999999997}]},{"osd":2,"last update":"Fri Feb 9 18:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":1.0449999999999999,"15min":0.98199999999999998},"min":{"1min":0.40000000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.90600000000000003,"5min":6.1799999999999997,"15min":6.1799999999999997},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.66100000000000003,"5min":1.0940000000000001,"15min":1.016},"min":{"1min":0.42299999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.83899999999999997,"5min":6.2220000000000004,"15min":6.2220000000000004},"last":0.73099999999999998}]},{"osd":3,"last update":"Fri Feb 9 18:07:27 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.90800000000000003,"15min":1.0369999999999999},"min":{"1min":0.52900000000000003,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.877,"5min":7,"15min":7},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.89500000000000002,"15min":1.046},"min":{"1min":0.49399999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.82799999999999996,"5min":6.0970000000000004,"15min":6.0970000000000004},"last":0.63500000000000001}]},{"osd":5,"last update":"Fri Feb 9 18:07:37 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.81599999999999995,"15min":0.65700000000000003},"min":{"1min":0.35399999999999998,"5min":0.27200000000000002,"15min":0.24399999999999999},"max":{"1min":0.69099999999999995,"5min":7.0469999999999997,"15min":7.0469999999999997},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.90000000000000002,"15min":0.73599999999999999},"min":{"1min":0.45000000000000001,"5min":0.32200000000000001,"15min":0.314},"max":{"1min":0.78000000000000003,"5min":7.0780000000000003,"15min":7.0780000000000003},"last":0.48199999999999998}]},{"osd":6,"last update":"Fri Feb 9 18:07:14 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":1.016,"15min":0.91500000000000004},"min":{"1min":0.32500000000000001,"5min":0.28999999999999998,"15min":0.27600000000000002},"max":{"1min":0.93200000000000005,"5min":6.4009999999999998,"15min":23.343},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.67000000000000004,"5min":1.111,"15min":0.95199999999999996},"min":{"1min":0.45200000000000001,"5min":0.29899999999999999,"15min":0.28999999999999998},"max":{"1min":0.97799999999999998,"5min":9.2260000000000009,"15min":21.861000000000001},"last":0.56699999999999995}]},{"osd":7,"last update":"Fri Feb 9 18:07:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.79700000000000004,"15min":0.76600000000000001},"min":{"1min":0.40999999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.84599999999999997,"5min":5.2030000000000003,"15min":5.2030000000000003},"last":0.503},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.80000000000000004,"15min":0.77600000000000002},"min":{"1min":0.51700000000000002,"5min":0.36099999999999999,"15min":0.30599999999999999},"max":{"1min":0.91700000000000004,"5min":5.2549999999999999,"15min":6.7919999999999998},"last":0.60199999999999998}]},{"osd":8,"last update":"Fri Feb 9 18:07:37 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":1.1479999999999999,"15min":1.2430000000000001},"min":{"1min":0.59699999999999998,"5min":0.38900000000000001,"15min":0.33100000000000002},"max":{"1min":1.03,"5min":6.4619999999999997,"15min":9.7059999999999995},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.84599999999999997,"5min":1.196,"15min":1.2729999999999999},"min":{"1min":0.65700000000000003,"5min":0.42899999999999999,"15min":0.35899999999999999},"max":{"1min":1.179,"5min":6.4329999999999998,"15min":9.0109999999999992},"last":0.77100000000000002}]},{"osd":9,"last update":"Fri Feb 9 18:07:37 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":1.4199999999999999,"15min":1.427},"min":{"1min":0.55000000000000004,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.97699999999999998,"5min":12.583,"15min":12.583},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.79500000000000004,"5min":1.4079999999999999,"15min":1.4350000000000001},"min":{"1min":0.65700000000000003,"5min":0.47399999999999998,"15min":0.373},"max":{"1min":1.004,"5min":10.679,"15min":10.679},"last":0.88300000000000001}]},{"osd":10,"last update":"Fri Feb 9 18:07:54 2024","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":1.026,"15min":1.421},"min":{"1min":0.61199999999999999,"5min":0.47699999999999998,"15min":0.41299999999999998},"max":{"1min":1.2090000000000001,"5min":6.2169999999999996,"15min":8.2829999999999995},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.84499999999999997,"5min":1.083,"15min":1.321},"min":{"1min":0.63,"5min":0.42799999999999999,"15min":0.41299999999999998},"max":{"1min":1.091,"5min":6.0199999999999996,"15min":7.9189999999999996},"last":0.79200000000000004}]},{"osd":11,"last update":"Fri Feb 9 18:07:27 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":1.1779999999999999,"15min":1.4079999999999999},"min":{"1min":0.64200000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.232,"5min":7.8620000000000001,"15min":7.8620000000000001},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.85999999999999999,"5min":1.282,"15min":1.502},"min":{"1min":0.57499999999999996,"5min":0.46500000000000002,"15min":0.46400000000000002},"max":{"1min":1.1870000000000001,"5min":8.2219999999999995,"15min":9.5960000000000001},"last":0.82599999999999996}]}]},{"osd":5,"up_from":40,"seq":171798692421,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":978908,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":977982,"kb_avail":92766244,"statfs":{"total":95995035648,"available":94992633856,"internally_reserved":0,"allocated":933888,"data_stored":625314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1001454021},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:58:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.82599999999999996,"15min":0.874},"min":{"1min":0.27800000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":1.341,"5min":5.399,"15min":5.399},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.622,"5min":0.92100000000000004,"15min":0.97099999999999997},"min":{"1min":0.40400000000000003,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.409,"5min":5.4329999999999998,"15min":5.4329999999999998},"last":0.52100000000000002}]},{"osd":1,"last update":"Fri Feb 9 18:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.89200000000000002,"15min":0.98099999999999998},"min":{"1min":0.48599999999999999,"5min":0.247,"15min":0.247},"max":{"1min":0.80200000000000005,"5min":4.8109999999999999,"15min":16.271000000000001},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.98699999999999999,"15min":0.997},"min":{"1min":0.46100000000000002,"5min":0.27600000000000002,"15min":0.23999999999999999},"max":{"1min":0.89000000000000001,"5min":7.2839999999999998,"15min":16.344999999999999},"last":0.92300000000000004}]},{"osd":2,"last update":"Fri Feb 9 18:07:13 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":1.175,"15min":0.97899999999999998},"min":{"1min":0.47299999999999998,"5min":0.307,"15min":0.254},"max":{"1min":0.78400000000000003,"5min":7.71,"15min":7.71},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.72299999999999998,"5min":1.1399999999999999,"15min":1.1359999999999999},"min":{"1min":0.504,"5min":0.29199999999999998,"15min":0.24399999999999999},"max":{"1min":1.004,"5min":8.0030000000000001,"15min":8.0030000000000001},"last":0.80000000000000004}]},{"osd":3,"last update":"Fri Feb 9 18:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":1.014,"15min":0.82899999999999996},"min":{"1min":0.51700000000000002,"5min":0.28000000000000003,"15min":0.26700000000000002},"max":{"1min":0.74399999999999999,"5min":10.896000000000001,"15min":10.896000000000001},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.67800000000000005,"5min":1.115,"15min":1.1000000000000001},"min":{"1min":0.504,"5min":0.34300000000000003,"15min":0.28000000000000003},"max":{"1min":0.83499999999999996,"5min":6.9630000000000001,"15min":7.218},"last":1.026}]},{"osd":4,"last update":"Fri Feb 9 18:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.82099999999999995,"15min":0.71499999999999997},"min":{"1min":0.41899999999999998,"5min":0.27400000000000002,"15min":0.25900000000000001},"max":{"1min":0.72299999999999998,"5min":5.7990000000000004,"15min":5.7990000000000004},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.997,"15min":0.82899999999999996},"min":{"1min":0.54600000000000004,"5min":0.29599999999999999,"15min":0.22500000000000001},"max":{"1min":0.79800000000000004,"5min":6.1040000000000001,"15min":6.1239999999999997},"last":0.752}]},{"osd":6,"last update":"Fri Feb 9 18:07:13 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.88200000000000001,"15min":0.78800000000000003},"min":{"1min":0.39200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.86099999999999999,"5min":6.8230000000000004,"15min":6.8230000000000004},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.081,"15min":0.89500000000000002},"min":{"1min":0.42599999999999999,"5min":0.34899999999999998,"15min":0.33100000000000002},"max":{"1min":0.90600000000000003,"5min":6.9290000000000003,"15min":6.9290000000000003},"last":0.98199999999999998}]},{"osd":7,"last update":"Fri Feb 9 18:07:46 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":1.117,"15min":0.93500000000000005},"min":{"1min":0.432,"5min":0.36499999999999999,"15min":0.28199999999999997},"max":{"1min":0.76500000000000001,"5min":12.798999999999999,"15min":12.798999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.746,"5min":1.1910000000000001,"15min":0.92100000000000004},"min":{"1min":0.61199999999999999,"5min":0.48399999999999999,"15min":0.32400000000000001},"max":{"1min":0.88800000000000001,"5min":12.885999999999999,"15min":12.885999999999999},"last":0.82499999999999996}]},{"osd":8,"last update":"Fri Feb 9 18:06:59 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":1.238,"15min":1.454},"min":{"1min":0.63700000000000001,"5min":0.35299999999999998,"15min":0.34200000000000003},"max":{"1min":1.3,"5min":7.71,"15min":12.227},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.80800000000000005,"5min":1.014,"15min":1.206},"min":{"1min":0.626,"5min":0.376,"15min":0.372},"max":{"1min":1.0960000000000001,"5min":6.3739999999999997,"15min":8.3770000000000007},"last":0.77600000000000002}]},{"osd":9,"last update":"Fri Feb 9 18:07:13 2024","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":1.4359999999999999,"15min":1.6519999999999999},"min":{"1min":0.63900000000000001,"5min":0.39800000000000002,"15min":0.315},"max":{"1min":1.139,"5min":21.686,"15min":21.686},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.92000000000000004,"5min":1.5740000000000001,"15min":1.494},"min":{"1min":0.65400000000000003,"5min":0.42499999999999999,"15min":0.36799999999999999},"max":{"1min":1.2849999999999999,"5min":21.734999999999999,"15min":21.734999999999999},"last":1.0680000000000001}]},{"osd":10,"last update":"Fri Feb 9 18:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":1.5700000000000001,"15min":1.4219999999999999},"min":{"1min":0.66500000000000004,"5min":0.41599999999999998,"15min":0.39300000000000002},"max":{"1min":1.046,"5min":21.027000000000001,"15min":21.027000000000001},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.94099999999999995,"5min":1.5609999999999999,"15min":1.571},"min":{"1min":0.78800000000000003,"5min":0.443,"15min":0.39900000000000002},"max":{"1min":1.367,"5min":20.859999999999999,"15min":20.859999999999999},"last":1.113}]},{"osd":11,"last update":"Fri Feb 9 18:06:59 2024","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":1.3140000000000001,"15min":1.486},"min":{"1min":0.68700000000000006,"5min":0.505,"15min":0.434},"max":{"1min":1.04,"5min":9.7959999999999994,"15min":9.7959999999999994},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.92900000000000005,"5min":1.5640000000000001,"15min":1.6890000000000001},"min":{"1min":0.71299999999999997,"5min":0.505,"15min":0.434},"max":{"1min":1.1859999999999999,"5min":9.8819999999999997,"15min":20.981000000000002},"last":1.1599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-09T18:08:04.971 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph pg dump --format=json 2024-02-09T18:08:05.237 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:08:05.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:08:05 smithi099 ceph-mon[39420]: from='client.17658 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T18:08:05.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:08:05 smithi099 ceph-mon[39420]: 1.10 scrub starts 2024-02-09T18:08:05.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:08:05 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:08:05.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:08:05 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:08:05.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:08:05 smithi099 ceph-mon[39420]: 1.10 scrub ok 2024-02-09T18:08:05.744 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:08:05 smithi099 ceph-mon[39420]: osdmap e1048: 12 total, 12 up, 12 in 2024-02-09T18:08:05.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:08:05 smithi179 ceph-mon[43975]: from='client.17658 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T18:08:05.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:08:05 smithi179 ceph-mon[43975]: 1.10 scrub starts 2024-02-09T18:08:05.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:08:05 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:08:05.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:08:05 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:08:05.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:08:05 smithi179 ceph-mon[43975]: 1.10 scrub ok 2024-02-09T18:08:05.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:08:05 smithi179 ceph-mon[43975]: osdmap e1048: 12 total, 12 up, 12 in 2024-02-09T18:08:05.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:08:05 smithi162 ceph-mon[44304]: from='client.17658 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T18:08:05.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:08:05 smithi162 ceph-mon[44304]: 1.10 scrub starts 2024-02-09T18:08:05.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:08:05 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:08:05.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:08:05 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:08:05.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:08:05 smithi162 ceph-mon[44304]: 1.10 scrub ok 2024-02-09T18:08:05.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:08:05 smithi162 ceph-mon[44304]: osdmap e1048: 12 total, 12 up, 12 in 2024-02-09T18:08:06.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:08:06 smithi179 ceph-mon[43975]: pgmap v2588: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:08:06.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:08:06 smithi162 ceph-mon[44304]: pgmap v2588: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:08:06.939 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:08:06 smithi099 ceph-mon[39420]: pgmap v2588: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:08:07.479 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-02-09T18:08:07.480 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-02-09T18:08:08.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:08:08 smithi099 ceph-mon[39420]: 1.e scrub starts 2024-02-09T18:08:08.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:08:08 smithi099 ceph-mon[39420]: 1.e scrub ok 2024-02-09T18:08:08.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:08:08 smithi099 ceph-mon[39420]: pgmap v2589: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:08:08.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:08:08 smithi099 ceph-mon[39420]: from='client.17664 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T18:08:08.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:08:08 smithi099 ceph-mon[39420]: purged_snaps scrub starts 2024-02-09T18:08:08.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:08:08 smithi099 ceph-mon[39420]: purged_snaps scrub ok 2024-02-09T18:08:08.465 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":2589,"stamp":"2024-02-09T18:08:07.415040+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":93,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":10119812,"kb_used_data":10356,"kb_used_omap":430,"kb_used_meta":10108881,"kb_avail":1114822012,"statfs":{"total":1151940427776,"available":1141577740288,"internally_reserved":0,"allocated":10604544,"data_stored":6833412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":441167,"internal_metadata":10351494321},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001850"},"pg_stats":[{"pgid":"1.1e","version":"63'139","reported_seq":1462,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-02-09T18:08:05.680803+0000","last_change":"2024-02-09T18:07:58.910839+0000","last_active":"2024-02-09T18:08:05.680803+0000","last_peered":"2024-02-09T18:08:05.680803+0000","last_clean":"2024-02-09T18:08:05.680803+0000","last_became_active":"2024-02-09T18:03:47.759332+0000","last_became_peered":"2024-02-09T18:03:47.759332+0000","last_unstale":"2024-02-09T18:08:05.680803+0000","last_undegraded":"2024-02-09T18:08:05.680803+0000","last_fullsized":"2024-02-09T18:08:05.680803+0000","mapping_epoch":961,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":962,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:58.910679+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:07:58.910679+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:09:08.763696+0000","scrub_duration":117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"63'139","reported_seq":1471,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-02-09T18:08:04.560510+0000","last_change":"2024-02-09T18:07:47.252312+0000","last_active":"2024-02-09T18:08:04.560510+0000","last_peered":"2024-02-09T18:08:04.560510+0000","last_clean":"2024-02-09T18:08:04.560510+0000","last_became_active":"2024-02-09T17:43:28.594296+0000","last_became_peered":"2024-02-09T17:43:28.594296+0000","last_unstale":"2024-02-09T18:08:04.560510+0000","last_undegraded":"2024-02-09T18:08:04.560510+0000","last_fullsized":"2024-02-09T18:08:04.560510+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:47.252184+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T18:07:47.252184+0000","last_clean_scrub_stamp":"2024-02-09T18:07:47.252184+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:09:13.542304+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"63'139","reported_seq":1991,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-02-09T18:08:04.554780+0000","last_change":"2024-02-09T18:07:14.310594+0000","last_active":"2024-02-09T18:08:04.554780+0000","last_peered":"2024-02-09T18:08:04.554780+0000","last_clean":"2024-02-09T18:08:04.554780+0000","last_became_active":"2024-02-09T17:43:19.103584+0000","last_became_peered":"2024-02-09T17:43:19.103584+0000","last_unstale":"2024-02-09T18:08:04.554780+0000","last_undegraded":"2024-02-09T18:08:04.554780+0000","last_fullsized":"2024-02-09T18:08:04.554780+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:14.310481+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T18:07:14.310481+0000","last_clean_scrub_stamp":"2024-02-09T18:07:14.310481+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:08:24.380102+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"63'139","reported_seq":1790,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-02-09T18:08:04.554724+0000","last_change":"2024-02-09T18:06:43.197838+0000","last_active":"2024-02-09T18:08:04.554724+0000","last_peered":"2024-02-09T18:08:04.554724+0000","last_clean":"2024-02-09T18:08:04.554724+0000","last_became_active":"2024-02-09T17:43:19.102729+0000","last_became_peered":"2024-02-09T17:43:19.102729+0000","last_unstale":"2024-02-09T18:08:04.554724+0000","last_undegraded":"2024-02-09T18:08:04.554724+0000","last_fullsized":"2024-02-09T18:08:04.554724+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:06:43.197734+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:45:44.988633+0000","last_clean_scrub_stamp":"2024-02-09T18:06:43.197734+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:08:09.428876+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"63'139","reported_seq":685,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-02-09T18:08:05.680888+0000","last_change":"2024-02-09T18:07:00.433777+0000","last_active":"2024-02-09T18:08:05.680888+0000","last_peered":"2024-02-09T18:08:05.680888+0000","last_clean":"2024-02-09T18:08:05.680888+0000","last_became_active":"2024-02-09T18:03:02.127510+0000","last_became_peered":"2024-02-09T18:03:02.127510+0000","last_unstale":"2024-02-09T18:08:05.680888+0000","last_undegraded":"2024-02-09T18:08:05.680888+0000","last_fullsized":"2024-02-09T18:08:05.680888+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:00.433674+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:07:00.433674+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:08:26.949531+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"63'139","reported_seq":1837,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-02-09T18:08:05.680635+0000","last_change":"2024-02-09T18:07:11.021282+0000","last_active":"2024-02-09T18:08:05.680635+0000","last_peered":"2024-02-09T18:08:05.680635+0000","last_clean":"2024-02-09T18:08:05.680635+0000","last_became_active":"2024-02-09T18:05:14.211564+0000","last_became_peered":"2024-02-09T18:05:14.211564+0000","last_unstale":"2024-02-09T18:08:05.680635+0000","last_undegraded":"2024-02-09T18:08:05.680635+0000","last_fullsized":"2024-02-09T18:08:05.680635+0000","mapping_epoch":993,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":994,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:11.021172+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T18:04:37.468994+0000","last_clean_scrub_stamp":"2024-02-09T18:07:11.021172+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:08:26.060867+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"63'139","reported_seq":2124,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-02-09T18:07:59.935799+0000","last_change":"2024-02-09T18:06:53.873646+0000","last_active":"2024-02-09T18:07:59.935799+0000","last_peered":"2024-02-09T18:07:59.935799+0000","last_clean":"2024-02-09T18:07:59.935799+0000","last_became_active":"2024-02-09T18:06:03.580532+0000","last_became_peered":"2024-02-09T18:06:03.580532+0000","last_unstale":"2024-02-09T18:07:59.935799+0000","last_undegraded":"2024-02-09T18:07:59.935799+0000","last_fullsized":"2024-02-09T18:07:59.935799+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:06:53.873522+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T18:05:48.428084+0000","last_clean_scrub_stamp":"2024-02-09T18:06:53.873522+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:08:18.043167+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"63'139","reported_seq":1600,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-02-09T18:08:04.554827+0000","last_change":"2024-02-09T18:07:27.587234+0000","last_active":"2024-02-09T18:08:04.554827+0000","last_peered":"2024-02-09T18:08:04.554827+0000","last_clean":"2024-02-09T18:08:04.554827+0000","last_became_active":"2024-02-09T18:05:14.211197+0000","last_became_peered":"2024-02-09T18:05:14.211197+0000","last_unstale":"2024-02-09T18:08:04.554827+0000","last_undegraded":"2024-02-09T18:08:04.554827+0000","last_fullsized":"2024-02-09T18:08:04.554827+0000","mapping_epoch":993,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":994,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:27.587110+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T18:03:40.548833+0000","last_clean_scrub_stamp":"2024-02-09T18:07:27.587110+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:08:44.420949+0000","scrub_duration":188,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"63'139","reported_seq":1457,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-02-09T18:08:04.560424+0000","last_change":"2024-02-09T18:07:03.801662+0000","last_active":"2024-02-09T18:08:04.560424+0000","last_peered":"2024-02-09T18:08:04.560424+0000","last_clean":"2024-02-09T18:08:04.560424+0000","last_became_active":"2024-02-09T17:59:12.003987+0000","last_became_peered":"2024-02-09T17:59:12.003987+0000","last_unstale":"2024-02-09T18:08:04.560424+0000","last_undegraded":"2024-02-09T18:08:04.560424+0000","last_fullsized":"2024-02-09T18:08:04.560424+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:03.801550+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:11.970338+0000","last_clean_scrub_stamp":"2024-02-09T18:07:03.801550+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:08:13.429137+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"63'139","reported_seq":1892,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-02-09T18:08:05.179288+0000","last_change":"2024-02-09T18:07:43.654399+0000","last_active":"2024-02-09T18:08:05.179288+0000","last_peered":"2024-02-09T18:08:05.179288+0000","last_clean":"2024-02-09T18:08:05.179288+0000","last_became_active":"2024-02-09T17:43:19.138398+0000","last_became_peered":"2024-02-09T17:43:19.138398+0000","last_unstale":"2024-02-09T18:08:05.179288+0000","last_undegraded":"2024-02-09T18:08:05.179288+0000","last_fullsized":"2024-02-09T18:08:05.179288+0000","mapping_epoch":519,"log_start":"0'0","ondisk_log_start":"0'0","created":514,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:43.654293+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T18:06:22.898046+0000","last_clean_scrub_stamp":"2024-02-09T18:07:43.654293+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:08:58.977702+0000","scrub_duration":196,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"63'139","reported_seq":1964,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-02-09T18:08:04.553576+0000","last_change":"2024-02-09T18:07:26.376103+0000","last_active":"2024-02-09T18:08:04.553576+0000","last_peered":"2024-02-09T18:08:04.553576+0000","last_clean":"2024-02-09T18:08:04.553576+0000","last_became_active":"2024-02-09T17:43:28.589307+0000","last_became_peered":"2024-02-09T17:43:28.589307+0000","last_unstale":"2024-02-09T18:08:04.553576+0000","last_undegraded":"2024-02-09T18:08:04.553576+0000","last_fullsized":"2024-02-09T18:08:04.553576+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:26.376000+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:38:26.968892+0000","last_clean_scrub_stamp":"2024-02-09T18:07:26.376000+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:08:42.343492+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"63'139","reported_seq":2542,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-02-09T18:08:05.179299+0000","last_change":"2024-02-09T18:07:28.365226+0000","last_active":"2024-02-09T18:08:05.179299+0000","last_peered":"2024-02-09T18:08:05.179299+0000","last_clean":"2024-02-09T18:08:05.179299+0000","last_became_active":"2024-02-09T17:41:14.521079+0000","last_became_peered":"2024-02-09T17:41:14.521079+0000","last_unstale":"2024-02-09T18:08:05.179299+0000","last_undegraded":"2024-02-09T18:08:05.179299+0000","last_fullsized":"2024-02-09T18:08:05.179299+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:28.365133+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:07:28.365133+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:08:57.339359+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"63'139","reported_seq":1194,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-02-09T18:08:05.179223+0000","last_change":"2024-02-09T18:07:09.415201+0000","last_active":"2024-02-09T18:08:05.179223+0000","last_peered":"2024-02-09T18:08:05.179223+0000","last_clean":"2024-02-09T18:08:05.179223+0000","last_became_active":"2024-02-09T18:06:17.478373+0000","last_became_peered":"2024-02-09T18:06:17.478373+0000","last_unstale":"2024-02-09T18:08:05.179223+0000","last_undegraded":"2024-02-09T18:08:05.179223+0000","last_fullsized":"2024-02-09T18:08:05.179223+0000","mapping_epoch":1019,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":1020,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:09.415099+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:07:09.415099+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:08:17.563124+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"63'139","reported_seq":1749,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-02-09T18:08:04.560367+0000","last_change":"2024-02-09T18:06:37.782212+0000","last_active":"2024-02-09T18:08:04.560367+0000","last_peered":"2024-02-09T18:08:04.560367+0000","last_clean":"2024-02-09T18:08:04.560367+0000","last_became_active":"2024-02-09T18:03:02.134791+0000","last_became_peered":"2024-02-09T18:03:02.134791+0000","last_unstale":"2024-02-09T18:08:04.560367+0000","last_undegraded":"2024-02-09T18:08:04.560367+0000","last_fullsized":"2024-02-09T18:08:04.560367+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:06:37.782111+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:46:40.684762+0000","last_clean_scrub_stamp":"2024-02-09T18:06:37.782111+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"63'139","reported_seq":482,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-02-09T18:08:05.680735+0000","last_change":"2024-02-09T18:07:34.985801+0000","last_active":"2024-02-09T18:08:05.680735+0000","last_peered":"2024-02-09T18:08:05.680735+0000","last_clean":"2024-02-09T18:08:05.680735+0000","last_became_active":"2024-02-09T18:05:14.211371+0000","last_became_peered":"2024-02-09T18:05:14.211371+0000","last_unstale":"2024-02-09T18:08:05.680735+0000","last_undegraded":"2024-02-09T18:08:05.680735+0000","last_fullsized":"2024-02-09T18:08:05.680735+0000","mapping_epoch":993,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":994,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:34.985693+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T18:00:46.701010+0000","last_clean_scrub_stamp":"2024-02-09T18:07:34.985693+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:08:39.774211+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"63'139","reported_seq":396,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-02-09T18:08:05.680786+0000","last_change":"2024-02-09T18:07:51.912625+0000","last_active":"2024-02-09T18:08:05.680786+0000","last_peered":"2024-02-09T18:08:05.680786+0000","last_clean":"2024-02-09T18:08:05.680786+0000","last_became_active":"2024-02-09T18:05:14.211331+0000","last_became_peered":"2024-02-09T18:05:14.211331+0000","last_unstale":"2024-02-09T18:08:05.680786+0000","last_undegraded":"2024-02-09T18:08:05.680786+0000","last_fullsized":"2024-02-09T18:08:05.680786+0000","mapping_epoch":993,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":994,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:51.912498+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T18:07:51.912498+0000","last_clean_scrub_stamp":"2024-02-09T18:07:51.912498+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:09:19.321922+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.7","version":"63'139","reported_seq":1659,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-02-09T18:08:04.560439+0000","last_change":"2024-02-09T18:07:10.067566+0000","last_active":"2024-02-09T18:08:04.560439+0000","last_peered":"2024-02-09T18:08:04.560439+0000","last_clean":"2024-02-09T18:08:04.560439+0000","last_became_active":"2024-02-09T17:52:32.292519+0000","last_became_peered":"2024-02-09T17:52:32.292519+0000","last_unstale":"2024-02-09T18:08:04.560439+0000","last_undegraded":"2024-02-09T18:08:04.560439+0000","last_fullsized":"2024-02-09T18:08:04.560439+0000","mapping_epoch":716,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":717,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:10.067465+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:07:10.067465+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:08:17.504558+0000","scrub_duration":278,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"63'139","reported_seq":2123,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-02-09T18:08:05.680884+0000","last_change":"2024-02-09T18:07:29.604329+0000","last_active":"2024-02-09T18:08:05.680884+0000","last_peered":"2024-02-09T18:08:05.680884+0000","last_clean":"2024-02-09T18:08:05.680884+0000","last_became_active":"2024-02-09T17:33:00.284924+0000","last_became_peered":"2024-02-09T17:33:00.284924+0000","last_unstale":"2024-02-09T18:08:05.680884+0000","last_undegraded":"2024-02-09T18:08:05.680884+0000","last_fullsized":"2024-02-09T18:08:05.680884+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:29.604237+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:56:45.406767+0000","last_clean_scrub_stamp":"2024-02-09T18:07:29.604237+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:08:42.253371+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.5","version":"63'139","reported_seq":1309,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-02-09T18:08:05.680677+0000","last_change":"2024-02-09T18:07:44.949102+0000","last_active":"2024-02-09T18:08:05.680677+0000","last_peered":"2024-02-09T18:08:05.680677+0000","last_clean":"2024-02-09T18:08:05.680677+0000","last_became_active":"2024-02-09T18:05:14.210449+0000","last_became_peered":"2024-02-09T18:05:14.210449+0000","last_unstale":"2024-02-09T18:08:05.680677+0000","last_undegraded":"2024-02-09T18:08:05.680677+0000","last_fullsized":"2024-02-09T18:08:05.680677+0000","mapping_epoch":993,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":994,"parent":"0.0","parent_split_bits":3,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:44.948971+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:07:44.948971+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:09:12.382631+0000","scrub_duration":141,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,8],"acting":[10,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.a","version":"63'139","reported_seq":2060,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-02-09T18:07:59.936297+0000","last_change":"2024-02-09T18:07:23.484093+0000","last_active":"2024-02-09T18:07:59.936297+0000","last_peered":"2024-02-09T18:07:59.936297+0000","last_clean":"2024-02-09T18:07:59.936297+0000","last_became_active":"2024-02-09T18:03:02.134238+0000","last_became_peered":"2024-02-09T18:03:02.134238+0000","last_unstale":"2024-02-09T18:07:59.936297+0000","last_undegraded":"2024-02-09T18:07:59.936297+0000","last_fullsized":"2024-02-09T18:07:59.936297+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:23.483995+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T18:03:49.362497+0000","last_clean_scrub_stamp":"2024-02-09T18:07:23.483995+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:08:27.179567+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.b","version":"63'139","reported_seq":1411,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-02-09T18:08:05.680763+0000","last_change":"2024-02-09T18:06:49.074743+0000","last_active":"2024-02-09T18:08:05.680763+0000","last_peered":"2024-02-09T18:08:05.680763+0000","last_clean":"2024-02-09T18:08:05.680763+0000","last_became_active":"2024-02-09T17:55:08.938203+0000","last_became_peered":"2024-02-09T17:55:08.938203+0000","last_unstale":"2024-02-09T18:08:05.680763+0000","last_undegraded":"2024-02-09T18:08:05.680763+0000","last_fullsized":"2024-02-09T18:08:05.680763+0000","mapping_epoch":781,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":782,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:06:49.074627+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T18:05:26.531050+0000","last_clean_scrub_stamp":"2024-02-09T18:06:49.074627+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:08:17.069529+0000","scrub_duration":230,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.8","version":"63'139","reported_seq":1276,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-02-09T18:07:59.935905+0000","last_change":"2024-02-09T18:07:36.768760+0000","last_active":"2024-02-09T18:07:59.935905+0000","last_peered":"2024-02-09T18:07:59.935905+0000","last_clean":"2024-02-09T18:07:59.935905+0000","last_became_active":"2024-02-09T18:06:03.579088+0000","last_became_peered":"2024-02-09T18:06:03.579088+0000","last_unstale":"2024-02-09T18:07:59.935905+0000","last_undegraded":"2024-02-09T18:07:59.935905+0000","last_fullsized":"2024-02-09T18:07:59.935905+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:36.768659+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:07:36.768659+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:08:45.132904+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"63'139","reported_seq":2548,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-02-09T18:08:05.179167+0000","last_change":"2024-02-09T18:07:35.563141+0000","last_active":"2024-02-09T18:08:05.179167+0000","last_peered":"2024-02-09T18:08:05.179167+0000","last_clean":"2024-02-09T18:08:05.179167+0000","last_became_active":"2024-02-09T17:43:28.590237+0000","last_became_peered":"2024-02-09T17:43:28.590237+0000","last_unstale":"2024-02-09T18:08:05.179167+0000","last_undegraded":"2024-02-09T18:08:05.179167+0000","last_fullsized":"2024-02-09T18:08:05.179167+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:35.563036+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:59:49.716259+0000","last_clean_scrub_stamp":"2024-02-09T18:07:35.563036+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:08:38.471263+0000","scrub_duration":197,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"63'139","reported_seq":504,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-02-09T18:08:05.680627+0000","last_change":"2024-02-09T18:06:47.498203+0000","last_active":"2024-02-09T18:08:05.680627+0000","last_peered":"2024-02-09T18:08:05.680627+0000","last_clean":"2024-02-09T18:08:05.680627+0000","last_became_active":"2024-02-09T18:03:02.137032+0000","last_became_peered":"2024-02-09T18:03:02.137032+0000","last_unstale":"2024-02-09T18:08:05.680627+0000","last_undegraded":"2024-02-09T18:08:05.680627+0000","last_fullsized":"2024-02-09T18:08:05.680627+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:06:47.498092+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T18:06:47.498092+0000","last_clean_scrub_stamp":"2024-02-09T18:06:47.498092+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.f","version":"63'139","reported_seq":2090,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-02-09T18:08:05.680630+0000","last_change":"2024-02-09T18:07:59.958123+0000","last_active":"2024-02-09T18:08:05.680630+0000","last_peered":"2024-02-09T18:08:05.680630+0000","last_clean":"2024-02-09T18:08:05.680630+0000","last_became_active":"2024-02-09T18:03:02.138442+0000","last_became_peered":"2024-02-09T18:03:02.138442+0000","last_unstale":"2024-02-09T18:08:05.680630+0000","last_undegraded":"2024-02-09T18:08:05.680630+0000","last_fullsized":"2024-02-09T18:08:05.680630+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:59.957992+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:48:34.651647+0000","last_clean_scrub_stamp":"2024-02-09T18:07:59.957992+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:09:00.606103+0000","scrub_duration":196,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"63'139","reported_seq":2035,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-02-09T18:08:04.554156+0000","last_change":"2024-02-09T18:07:12.841278+0000","last_active":"2024-02-09T18:08:04.554156+0000","last_peered":"2024-02-09T18:08:04.554156+0000","last_clean":"2024-02-09T18:08:04.554156+0000","last_became_active":"2024-02-09T17:59:12.208604+0000","last_became_peered":"2024-02-09T17:59:12.208604+0000","last_unstale":"2024-02-09T18:08:04.554156+0000","last_undegraded":"2024-02-09T18:08:04.554156+0000","last_fullsized":"2024-02-09T18:08:04.554156+0000","mapping_epoch":854,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":855,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:12.841153+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:43:14.135321+0000","last_clean_scrub_stamp":"2024-02-09T18:07:12.841153+0000","objects_scrubbed":1,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:08:38.766542+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"63'139","reported_seq":1622,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-02-09T18:08:05.179350+0000","last_change":"2024-02-09T18:07:51.498713+0000","last_active":"2024-02-09T18:08:05.179350+0000","last_peered":"2024-02-09T18:08:05.179350+0000","last_clean":"2024-02-09T18:08:05.179350+0000","last_became_active":"2024-02-09T18:03:02.132990+0000","last_became_peered":"2024-02-09T18:03:02.132990+0000","last_unstale":"2024-02-09T18:08:05.179350+0000","last_undegraded":"2024-02-09T18:08:05.179350+0000","last_fullsized":"2024-02-09T18:08:05.179350+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":4,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:51.498614+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T18:00:49.890794+0000","last_clean_scrub_stamp":"2024-02-09T18:07:51.498614+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:09:07.875541+0000","scrub_duration":197,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"63'139","reported_seq":1098,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-02-09T18:08:04.497658+0000","last_change":"2024-02-09T18:08:04.497688+0000","last_active":"2024-02-09T18:08:04.497658+0000","last_peered":"2024-02-09T18:08:04.497658+0000","last_clean":"2024-02-09T18:08:04.497658+0000","last_became_active":"2024-02-09T18:06:03.580581+0000","last_became_peered":"2024-02-09T18:06:03.580581+0000","last_unstale":"2024-02-09T18:08:04.497658+0000","last_undegraded":"2024-02-09T18:08:04.497658+0000","last_fullsized":"2024-02-09T18:08:04.497658+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:08:04.497585+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T18:05:33.857425+0000","last_clean_scrub_stamp":"2024-02-09T18:08:04.497585+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:09:27.025055+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"63'139","reported_seq":1562,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-02-09T18:07:59.543702+0000","last_change":"2024-02-09T18:07:34.454093+0000","last_active":"2024-02-09T18:07:59.543702+0000","last_peered":"2024-02-09T18:07:59.543702+0000","last_clean":"2024-02-09T18:07:59.543702+0000","last_became_active":"2024-02-09T18:05:27.447245+0000","last_became_peered":"2024-02-09T18:05:27.447245+0000","last_unstale":"2024-02-09T18:07:59.543702+0000","last_undegraded":"2024-02-09T18:07:59.543702+0000","last_fullsized":"2024-02-09T18:07:59.543702+0000","mapping_epoch":998,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":999,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:34.453939+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T18:05:11.269341+0000","last_clean_scrub_stamp":"2024-02-09T18:07:34.453939+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:08:45.889300+0000","scrub_duration":118,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"63'139","reported_seq":1609,"reported_epoch":1047,"state":"active+clean","last_fresh":"2024-02-09T18:07:59.936105+0000","last_change":"2024-02-09T18:07:03.824695+0000","last_active":"2024-02-09T18:07:59.936105+0000","last_peered":"2024-02-09T18:07:59.936105+0000","last_clean":"2024-02-09T18:07:59.936105+0000","last_became_active":"2024-02-09T18:06:03.580019+0000","last_became_peered":"2024-02-09T18:06:03.580019+0000","last_unstale":"2024-02-09T18:07:59.936105+0000","last_undegraded":"2024-02-09T18:07:59.936105+0000","last_fullsized":"2024-02-09T18:07:59.936105+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:03.824553+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:07:03.824553+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:08:05.828313+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"63'139","reported_seq":1751,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-02-09T18:08:04.560370+0000","last_change":"2024-02-09T18:07:45.389122+0000","last_active":"2024-02-09T18:08:04.560370+0000","last_peered":"2024-02-09T18:08:04.560370+0000","last_clean":"2024-02-09T18:08:04.560370+0000","last_became_active":"2024-02-09T18:05:14.212995+0000","last_became_peered":"2024-02-09T18:05:14.212995+0000","last_unstale":"2024-02-09T18:08:04.560370+0000","last_undegraded":"2024-02-09T18:08:04.560370+0000","last_fullsized":"2024-02-09T18:08:04.560370+0000","mapping_epoch":993,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":994,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'139","last_scrub_stamp":"2024-02-09T18:07:45.389019+0000","last_deep_scrub":"63'139","last_deep_scrub_stamp":"2024-02-09T17:21:06.804958+0000","last_clean_scrub_stamp":"2024-02-09T18:07:45.389019+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-09T18:08:59.448319+0000","scrub_duration":274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4309,"ondisk_log_size":4309,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":90,"seq":386547057192,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1006508,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":1005758,"kb_avail":92738644,"statfs":{"total":95995035648,"available":94964371456,"internally_reserved":0,"allocated":753664,"data_stored":439098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":1029896644},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 18:07:30 2024","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":1.05,"15min":1.5049999999999999},"min":{"1min":0.32700000000000001,"5min":0.24399999999999999,"15min":0.22},"max":{"1min":11.006,"5min":11.006,"15min":13.307},"last":0.76600000000000001},{"interface":"front","average":{"1min":1.25,"5min":1.1910000000000001,"15min":1.6439999999999999},"min":{"1min":0.41199999999999998,"5min":0.26200000000000001,"15min":0.23100000000000001},"max":{"1min":13.771000000000001,"5min":13.771000000000001,"15min":13.771000000000001},"last":0.55200000000000005}]},{"osd":1,"last update":"Fri Feb 9 18:07:30 2024","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.5009999999999999,"15min":1.8089999999999999},"min":{"1min":0.38500000000000001,"5min":0.23100000000000001,"15min":0.22600000000000001},"max":{"1min":10.757,"5min":21.616,"15min":21.616},"last":0.79000000000000004},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.3069999999999999,"15min":1.724},"min":{"1min":0.38800000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":13.667,"5min":22.456,"15min":22.456},"last":0.622}]},{"osd":2,"last update":"Fri Feb 9 18:07:30 2024","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.1539999999999999,"15min":1.635},"min":{"1min":0.34999999999999998,"5min":0.27200000000000002,"15min":0.25600000000000001},"max":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":14.766999999999999},"last":0.83399999999999996},{"interface":"front","average":{"1min":1.351,"5min":1.1160000000000001,"15min":1.446},"min":{"1min":0.41099999999999998,"5min":0.32400000000000001,"15min":0.27800000000000002},"max":{"1min":13.837999999999999,"5min":13.837999999999999,"15min":13.837999999999999},"last":0.53000000000000003}]},{"osd":3,"last update":"Fri Feb 9 18:07:30 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":1.8740000000000001,"15min":1.72},"min":{"1min":0.40100000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":2.3319999999999999,"5min":41.395000000000003,"15min":41.395000000000003},"last":0.75},{"interface":"front","average":{"1min":1.593,"5min":1.857,"15min":1.8799999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.32300000000000001},"max":{"1min":16.914999999999999,"5min":41.276000000000003,"15min":41.276000000000003},"last":0.85999999999999999}]},{"osd":4,"last update":"Fri Feb 9 17:58:44 2024","interfaces":[{"interface":"back","average":{"1min":3.1709999999999998,"5min":2.5779999999999998,"15min":2.4790000000000001},"min":{"1min":0.40999999999999998,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":14.866,"5min":14.866,"15min":14.866},"last":0.45700000000000002},{"interface":"front","average":{"1min":3.2349999999999999,"5min":2.5640000000000001,"15min":2.452},"min":{"1min":0.496,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":14.939,"5min":14.939,"15min":14.939},"last":0.42299999999999999}]},{"osd":5,"last update":"Fri Feb 9 18:07:30 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.995,"15min":1.3620000000000001},"min":{"1min":0.46800000000000003,"5min":0.30199999999999999,"15min":0.27000000000000002},"max":{"1min":0.93700000000000006,"5min":6.0839999999999996,"15min":13.263999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.91700000000000004,"15min":1.4019999999999999},"min":{"1min":0.59399999999999997,"5min":0.317,"15min":0.30099999999999999},"max":{"1min":1.1359999999999999,"5min":6.5350000000000001,"15min":14.914999999999999},"last":0.77800000000000002}]},{"osd":6,"last update":"Fri Feb 9 18:07:30 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":1.28,"15min":1.6020000000000001},"min":{"1min":0.57599999999999996,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.1000000000000001,"5min":8.7370000000000001,"15min":11.885},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.86199999999999999,"5min":1.0609999999999999,"15min":1.518},"min":{"1min":0.60199999999999998,"5min":0.38900000000000001,"15min":0.32400000000000001},"max":{"1min":1.0960000000000001,"5min":7.0830000000000002,"15min":15.409000000000001},"last":0.73599999999999999}]},{"osd":7,"last update":"Fri Feb 9 18:07:30 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":1.1579999999999999,"15min":1.548},"min":{"1min":0.51700000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.0489999999999999,"5min":7.0410000000000004,"15min":15.51},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.80500000000000005,"5min":1.359,"15min":1.6319999999999999},"min":{"1min":0.56699999999999995,"5min":0.38600000000000001,"15min":0.33400000000000002},"max":{"1min":1.0920000000000001,"5min":8.6829999999999998,"15min":10.911},"last":0.67600000000000005}]},{"osd":8,"last update":"Fri Feb 9 18:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":1.246,"15min":1.9830000000000001},"min":{"1min":0.45500000000000002,"5min":0.314,"15min":0.307},"max":{"1min":0.84199999999999997,"5min":8.7629999999999999,"15min":29.233000000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.66700000000000004,"5min":1.284,"15min":2.0800000000000001},"min":{"1min":0.51300000000000001,"5min":0.40000000000000002,"15min":0.34499999999999997},"max":{"1min":0.96699999999999997,"5min":8.7889999999999997,"15min":29.091999999999999},"last":0.66300000000000003}]},{"osd":9,"last update":"Fri Feb 9 18:08:03 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":1.171,"15min":1.796},"min":{"1min":0.443,"5min":0.34499999999999997,"15min":0.32000000000000001},"max":{"1min":0.92200000000000004,"5min":12.68,"15min":22.613},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.745,"5min":1.254,"15min":1.792},"min":{"1min":0.54000000000000004,"5min":0.38,"15min":0.35899999999999999},"max":{"1min":1.0169999999999999,"5min":12.707000000000001,"15min":19.555},"last":0.72399999999999998}]},{"osd":10,"last update":"Fri Feb 9 18:07:30 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.98899999999999999,"15min":2.1659999999999999},"min":{"1min":0.45900000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.135,"5min":6.5350000000000001,"15min":25.093},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.871,"5min":1.075,"15min":1.871},"min":{"1min":0.59799999999999998,"5min":0.46000000000000002,"15min":0.35999999999999999},"max":{"1min":1.173,"5min":6.5640000000000001,"15min":20.233000000000001},"last":0.88400000000000001}]}]},{"osd":10,"up_from":84,"seq":360777253424,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":898220,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":897470,"kb_avail":92846932,"statfs":{"total":95995035648,"available":95075258368,"internally_reserved":0,"allocated":753664,"data_stored":439098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":919009728},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 18:07:17 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.623,"15min":0.627},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1,"5min":1,"15min":1},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.622,"15min":0.63},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.91700000000000004,"5min":1.014,"15min":1.014},"last":0.66100000000000003}]},{"osd":1,"last update":"Fri Feb 9 18:04:30 2024","interfaces":[{"interface":"back","average":{"1min":3.3650000000000002,"5min":2.8839999999999999,"15min":2.1760000000000002},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":20.363,"5min":52.210999999999999,"15min":52.210999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":3.4540000000000002,"5min":3.0070000000000001,"15min":2.2919999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":20.422999999999998,"5min":51.537999999999997,"15min":51.537999999999997},"last":0.67900000000000005}]},{"osd":2,"last update":"Fri Feb 9 18:07:36 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.93600000000000005,"15min":1.6990000000000001},"min":{"1min":0.39900000000000002,"5min":0.247,"15min":0.247},"max":{"1min":1.048,"5min":11.327999999999999,"15min":22.652999999999999},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.89900000000000002,"15min":1.6359999999999999},"min":{"1min":0.40000000000000002,"5min":0.29599999999999999,"15min":0.28899999999999998},"max":{"1min":1.1839999999999999,"5min":11.298999999999999,"15min":12.180999999999999},"last":0.68200000000000005}]},{"osd":3,"last update":"Fri Feb 9 18:07:17 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":1.3979999999999999,"15min":1.4410000000000001},"min":{"1min":0.35999999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.80300000000000005,"5min":15.117000000000001,"15min":15.117000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.876,"15min":1.1000000000000001},"min":{"1min":0.33600000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":1.3660000000000001,"5min":5.569,"15min":6.1159999999999997},"last":0.63400000000000001}]},{"osd":4,"last update":"Fri Feb 9 18:07:05 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":1.109,"15min":1.272},"min":{"1min":0.46899999999999997,"5min":0.33300000000000002,"15min":0.253},"max":{"1min":0.90900000000000003,"5min":11.127000000000001,"15min":13.446},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.68999999999999995,"5min":1.069,"15min":1.319},"min":{"1min":0.504,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.264,"5min":7.8029999999999999,"15min":12.097},"last":0.78200000000000003}]},{"osd":5,"last update":"Fri Feb 9 18:07:36 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":1.2070000000000001,"15min":1.1819999999999999},"min":{"1min":0.38800000000000001,"5min":0.34999999999999998,"15min":0.28100000000000003},"max":{"1min":0.98399999999999999,"5min":7.1459999999999999,"15min":13.4},"last":0.75},{"interface":"front","average":{"1min":0.82599999999999996,"5min":1.3560000000000001,"15min":1.353},"min":{"1min":0.56899999999999995,"5min":0.36099999999999999,"15min":0.315},"max":{"1min":1.212,"5min":11.068,"15min":12.157999999999999},"last":0.90100000000000002}]},{"osd":6,"last update":"Fri Feb 9 18:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":1.3169999999999999,"15min":1.319},"min":{"1min":0.49399999999999999,"5min":0.40899999999999997,"15min":0.32300000000000001},"max":{"1min":1.2390000000000001,"5min":11.164,"15min":13.507},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.83599999999999997,"5min":1.0569999999999999,"15min":1.369},"min":{"1min":0.56999999999999995,"5min":0.372,"15min":0.35099999999999998},"max":{"1min":1.28,"5min":4.9009999999999998,"15min":12.205},"last":0.88200000000000001}]},{"osd":7,"last update":"Fri Feb 9 18:07:36 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":1.236,"15min":1.3149999999999999},"min":{"1min":0.52000000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.1220000000000001,"5min":5.5970000000000004,"15min":13.481},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.77900000000000003,"5min":1.5349999999999999,"15min":1.5760000000000001},"min":{"1min":0.46600000000000003,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.3939999999999999,"5min":15.074,"15min":15.074},"last":0.76900000000000002}]},{"osd":8,"last update":"Fri Feb 9 18:07:36 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.94299999999999995,"15min":1.95},"min":{"1min":0.39100000000000001,"5min":0.31,"15min":0.31},"max":{"1min":1.2529999999999999,"5min":8.4510000000000005,"15min":22.559000000000001},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.749,"5min":1.0389999999999999,"15min":1.617},"min":{"1min":0.45500000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.198,"5min":7.0919999999999996,"15min":22.824999999999999},"last":0.69799999999999995}]},{"osd":9,"last update":"Fri Feb 9 18:07:36 2024","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":1.363,"15min":2.1749999999999998},"min":{"1min":0.42899999999999999,"5min":0.377,"15min":0.32400000000000001},"max":{"1min":1.304,"5min":15.163,"15min":23.201000000000001},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.75800000000000001,"5min":1.2889999999999999,"15min":2.0099999999999998},"min":{"1min":0.42499999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.3220000000000001,"5min":11.196,"15min":17.498999999999999},"last":0.71699999999999997}]},{"osd":11,"last update":"Fri Feb 9 18:07:17 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":1.4370000000000001,"15min":1.859},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.36799999999999999},"max":{"1min":1.3580000000000001,"5min":11.228,"15min":21.18},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.85499999999999998,"5min":1.5089999999999999,"15min":2.0030000000000001},"min":{"1min":0.57399999999999995,"5min":0.51900000000000002,"15min":0.35899999999999999},"max":{"1min":1.413,"5min":11.266,"15min":32.707000000000001},"last":0.83399999999999996}]}]},{"osd":9,"up_from":77,"seq":330712482353,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1064800,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":1064062,"kb_avail":92680352,"statfs":{"total":95995035648,"available":94904680448,"internally_reserved":0,"allocated":741376,"data_stored":431582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":1089599936},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 18:07:50 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":1.0309999999999999,"15min":1.1619999999999999},"min":{"1min":0.29899999999999999,"5min":0.254,"15min":0.219},"max":{"1min":0.71299999999999997,"5min":8.4469999999999992,"15min":11.515000000000001},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.67500000000000004,"5min":1.115,"15min":1.042},"min":{"1min":0.36899999999999999,"5min":0.314,"15min":0.23200000000000001},"max":{"1min":0.83599999999999997,"5min":12.521000000000001,"15min":12.521000000000001},"last":0.66800000000000004}]},{"osd":1,"last update":"Fri Feb 9 18:07:29 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":1.1639999999999999,"15min":1.6870000000000001},"min":{"1min":0.38100000000000001,"5min":0.252,"15min":0.23599999999999999},"max":{"1min":0.90900000000000003,"5min":7.4790000000000001,"15min":61.031999999999996},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.66800000000000004,"5min":1.399,"15min":1.921},"min":{"1min":0.499,"5min":0.25900000000000001,"15min":0.25600000000000001},"max":{"1min":1.0720000000000001,"5min":9.0670000000000002,"15min":61.970999999999997},"last":0.71899999999999997}]},{"osd":2,"last update":"Fri Feb 9 18:07:24 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":1.1519999999999999,"15min":1.605},"min":{"1min":0.40200000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":1.169,"5min":7.6820000000000004,"15min":7.6820000000000004},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.67000000000000004,"5min":1.288,"15min":1.5920000000000001},"min":{"1min":0.35199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.3380000000000001,"5min":7.8620000000000001,"15min":7.8620000000000001},"last":0.42299999999999999}]},{"osd":3,"last update":"Fri Feb 9 18:07:29 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":1.2769999999999999,"15min":1.1160000000000001},"min":{"1min":0.35699999999999998,"5min":0.28499999999999998,"15min":0.22500000000000001},"max":{"1min":1.216,"5min":6.2590000000000003,"15min":6.3399999999999999},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.65400000000000003,"5min":1.329,"15min":1.095},"min":{"1min":0.44,"5min":0.32500000000000001,"15min":0.28100000000000003},"max":{"1min":0.89600000000000002,"5min":7.9429999999999996,"15min":7.9429999999999996},"last":0.48899999999999999}]},{"osd":4,"last update":"Fri Feb 9 18:07:36 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.998,"15min":1.4079999999999999},"min":{"1min":0.57499999999999996,"5min":0.29999999999999999,"15min":0.26900000000000002},"max":{"1min":0.96399999999999997,"5min":5.6020000000000003,"15min":19.414000000000001},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.82099999999999995,"5min":1.0580000000000001,"15min":1.2549999999999999},"min":{"1min":0.54900000000000004,"5min":0.39700000000000002,"15min":0.29999999999999999},"max":{"1min":1.038,"5min":7.7169999999999996,"15min":19.224},"last":0.89500000000000002}]},{"osd":5,"last update":"Fri Feb 9 18:07:36 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":1.0660000000000001,"15min":1.349},"min":{"1min":0.505,"5min":0.33600000000000002,"15min":0.33200000000000002},"max":{"1min":1.1299999999999999,"5min":7.8899999999999997,"15min":10.763999999999999},"last":0.93600000000000005},{"interface":"front","average":{"1min":0.78600000000000003,"5min":1.1930000000000001,"15min":1.373},"min":{"1min":0.54900000000000004,"5min":0.32700000000000001,"15min":0.316},"max":{"1min":1.087,"5min":7.7969999999999997,"15min":10.851000000000001},"last":0.998}]},{"osd":6,"last update":"Fri Feb 9 18:07:29 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":1.385,"15min":1.5009999999999999},"min":{"1min":0.46200000000000002,"5min":0.318,"15min":0.318},"max":{"1min":0.99199999999999999,"5min":9.1980000000000004,"15min":19.375},"last":0.874},{"interface":"front","average":{"1min":0.77300000000000002,"5min":1.462,"15min":1.534},"min":{"1min":0.51400000000000001,"5min":0.38200000000000001,"15min":0.34999999999999998},"max":{"1min":1.1279999999999999,"5min":9.1449999999999996,"15min":19.178000000000001},"last":0.91800000000000004}]},{"osd":7,"last update":"Fri Feb 9 18:02:55 2024","interfaces":[{"interface":"back","average":{"1min":1.9570000000000001,"5min":1.724,"15min":2.0059999999999998},"min":{"1min":0.35699999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":8.1899999999999995,"5min":8.8529999999999998,"15min":10.021000000000001},"last":0.623},{"interface":"front","average":{"1min":1.605,"5min":2.004,"15min":1.972},"min":{"1min":0.46000000000000002,"5min":0.38100000000000001,"15min":0.379},"max":{"1min":7.7839999999999998,"5min":10.802,"15min":11.138},"last":0.748}]},{"osd":8,"last update":"Fri Feb 9 18:07:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":1.343,"15min":1.5780000000000001},"min":{"1min":0.40300000000000002,"5min":0.31900000000000001,"15min":0.30599999999999999},"max":{"1min":1.036,"5min":14.042,"15min":14.042},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.75800000000000001,"5min":1.4490000000000001,"15min":1.6830000000000001},"min":{"1min":0.53800000000000003,"5min":0.39500000000000002,"15min":0.35499999999999998},"max":{"1min":1.242,"5min":14.087,"15min":14.087},"last":0.80800000000000005}]},{"osd":10,"last update":"Fri Feb 9 18:07:53 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":1.27,"15min":1.881},"min":{"1min":0.45100000000000001,"5min":0.35399999999999998,"15min":0.28100000000000003},"max":{"1min":1.0009999999999999,"5min":7.8300000000000001,"15min":10.333},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.252,"15min":1.972},"min":{"1min":0.46899999999999997,"5min":0.371,"15min":0.35999999999999999},"max":{"1min":1.3600000000000001,"5min":8.3870000000000005,"15min":18.579999999999998},"last":0.97399999999999998}]},{"osd":11,"last update":"Fri Feb 9 18:07:53 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":1.0920000000000001,"15min":1.7969999999999999},"min":{"1min":0.51200000000000001,"5min":0.42299999999999999,"15min":0.35299999999999998},"max":{"1min":1.1639999999999999,"5min":9.7149999999999999,"15min":40.259999999999998},"last":1.024},{"interface":"front","average":{"1min":0.86799999999999999,"5min":1.163,"15min":2.012},"min":{"1min":0.58699999999999997,"5min":0.46400000000000002,"15min":0.40400000000000003},"max":{"1min":1.2090000000000001,"5min":9.7400000000000002,"15min":39.683},"last":1.0700000000000001}]}]},{"osd":8,"up_from":171,"seq":734439408097,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":537580,"kb_used_data":736,"kb_used_omap":212,"kb_used_meta":536619,"kb_avail":93207572,"statfs":{"total":95995035648,"available":95444553728,"internally_reserved":0,"allocated":753664,"data_stored":439098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217579,"internal_metadata":549498389},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 18:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.81100000000000005,"15min":1.5049999999999999},"min":{"1min":0.34699999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.4490000000000001,"5min":6.0190000000000001,"15min":10.832000000000001},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.65600000000000003,"5min":1.077,"15min":1.462},"min":{"1min":0.40500000000000003,"5min":0.26300000000000001,"15min":0.23400000000000001},"max":{"1min":1.645,"5min":13.590999999999999,"15min":18.081},"last":0.59699999999999998}]},{"osd":1,"last update":"Fri Feb 9 18:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":1.6919999999999999,"15min":1.663},"min":{"1min":0.35399999999999998,"5min":0.26900000000000002,"15min":0.23799999999999999},"max":{"1min":2.1000000000000001,"5min":39.734000000000002,"15min":39.734000000000002},"last":0.628},{"interface":"front","average":{"1min":0.63500000000000001,"5min":1.7729999999999999,"15min":1.9890000000000001},"min":{"1min":0.35399999999999998,"5min":0.23599999999999999,"15min":0.23200000000000001},"max":{"1min":1.867,"5min":39.097000000000001,"15min":39.097000000000001},"last":0.73699999999999999}]},{"osd":2,"last update":"Fri Feb 9 18:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":1.1850000000000001,"15min":1.4379999999999999},"min":{"1min":0.313,"5min":0.28100000000000003,"15min":0.25},"max":{"1min":1.5009999999999999,"5min":13.741,"15min":13.741},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.66300000000000003,"5min":1.3939999999999999,"15min":1.764},"min":{"1min":0.33300000000000002,"5min":0.316,"15min":0.27300000000000002},"max":{"1min":1.55,"5min":13.803000000000001,"15min":18.045000000000002},"last":0.82399999999999995}]},{"osd":3,"last update":"Fri Feb 9 17:56:20 2024","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":2.552,"15min":2.8039999999999998},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":5.2560000000000002,"5min":15.654,"15min":15.654},"last":6.5350000000000001},{"interface":"front","average":{"1min":1.306,"5min":2.3149999999999999,"15min":2.0150000000000001},"min":{"1min":0.34300000000000003,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":5.2759999999999998,"5min":15.609999999999999,"15min":15.609999999999999},"last":6.5599999999999996}]},{"osd":4,"last update":"Fri Feb 9 18:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":1.234,"15min":1.3},"min":{"1min":0.35299999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.03,"5min":9.8610000000000007,"15min":12.297000000000001},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.68200000000000005,"5min":1.2769999999999999,"15min":1.304},"min":{"1min":0.40400000000000003,"5min":0.33100000000000002,"15min":0.28499999999999998},"max":{"1min":1.145,"5min":9.9160000000000004,"15min":12.227},"last":0.96799999999999997}]},{"osd":5,"last update":"Fri Feb 9 18:08:01 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":1.1479999999999999,"15min":1.3919999999999999},"min":{"1min":0.51000000000000001,"5min":0.29599999999999999,"15min":0.28399999999999997},"max":{"1min":0.94499999999999995,"5min":13.68,"15min":13.68},"last":1.165},{"interface":"front","average":{"1min":0.79500000000000004,"5min":1.181,"15min":1.3859999999999999},"min":{"1min":0.62,"5min":0.377,"15min":0.35499999999999998},"max":{"1min":0.95399999999999996,"5min":13.773,"15min":13.773},"last":0.87}]},{"osd":6,"last update":"Fri Feb 9 18:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":1.532,"15min":1.4830000000000001},"min":{"1min":0.52900000000000003,"5min":0.35099999999999998,"15min":0.32700000000000001},"max":{"1min":1.0680000000000001,"5min":9.8879999999999999,"15min":9.8879999999999999},"last":1.2589999999999999},{"interface":"front","average":{"1min":0.77000000000000002,"5min":1.0249999999999999,"15min":1.4450000000000001},"min":{"1min":0.54300000000000004,"5min":0.30299999999999999,"15min":0.28599999999999998},"max":{"1min":1.036,"5min":5.8440000000000003,"15min":10.178000000000001},"last":1.0389999999999999}]},{"osd":7,"last update":"Fri Feb 9 18:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":1.165,"15min":1.518},"min":{"1min":0.56799999999999995,"5min":0.34799999999999998,"15min":0.32900000000000001},"max":{"1min":1.0169999999999999,"5min":13.709,"15min":13.709},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.76800000000000002,"5min":1.081,"15min":1.4550000000000001},"min":{"1min":0.55800000000000005,"5min":0.313,"15min":0.309},"max":{"1min":1.0680000000000001,"5min":5.9870000000000001,"15min":10.148999999999999},"last":0.79300000000000004}]},{"osd":9,"last update":"Fri Feb 9 18:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":1.1739999999999999,"15min":2.2330000000000001},"min":{"1min":0.38500000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.0269999999999999,"5min":14.186,"15min":46.213999999999999},"last":1.008},{"interface":"front","average":{"1min":0.65800000000000003,"5min":1.111,"15min":2.2280000000000002},"min":{"1min":0.44700000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.89600000000000002,"5min":15.712999999999999,"15min":41.177},"last":1.1279999999999999}]},{"osd":10,"last update":"Fri Feb 9 18:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":1.2609999999999999,"15min":1.992},"min":{"1min":0.432,"5min":0.379,"15min":0.34599999999999997},"max":{"1min":1.2769999999999999,"5min":9.1639999999999997,"15min":18.164000000000001},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.74399999999999999,"5min":1.351,"15min":1.944},"min":{"1min":0.48899999999999999,"5min":0.40699999999999997,"15min":0.40200000000000002},"max":{"1min":1.296,"5min":9.8309999999999995,"15min":18.111000000000001},"last":1.02}]},{"osd":11,"last update":"Fri Feb 9 18:08:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.89900000000000002,"15min":1.7230000000000001},"min":{"1min":0.42699999999999999,"5min":0.36299999999999999,"15min":0.33300000000000002},"max":{"1min":0.97699999999999998,"5min":5.9269999999999996,"15min":10.231999999999999},"last":1.1120000000000001},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.98999999999999999,"15min":1.7809999999999999},"min":{"1min":0.56899999999999995,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.1319999999999999,"5min":5.9550000000000001,"15min":10.256},"last":1.3049999999999999}]}]},{"osd":7,"up_from":57,"seq":244813136443,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":912308,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":911358,"kb_avail":92832844,"statfs":{"total":95995035648,"available":95060832256,"internally_reserved":0,"allocated":958464,"data_stored":640346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":933231050},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 18:07:07 2024","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.1839999999999999,"15min":1.018},"min":{"1min":0.34699999999999998,"5min":0.23899999999999999,"15min":0.23100000000000001},"max":{"1min":12.617000000000001,"5min":12.617000000000001,"15min":12.617000000000001},"last":0.68700000000000006},{"interface":"front","average":{"1min":1.3260000000000001,"5min":1.6639999999999999,"15min":1.2390000000000001},"min":{"1min":0.439,"5min":0.34200000000000003,"15min":0.28499999999999998},"max":{"1min":12.605,"5min":12.605,"15min":15.890000000000001},"last":0.57699999999999996}]},{"osd":1,"last update":"Fri Feb 9 18:07:07 2024","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.952,"15min":1.4870000000000001},"min":{"1min":0.37,"5min":0.309,"15min":0.28299999999999997},"max":{"1min":12.132,"5min":18.062999999999999,"15min":18.062999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":1.3100000000000001,"5min":2.0099999999999998,"15min":1.4610000000000001},"min":{"1min":0.41199999999999998,"5min":0.26600000000000001,"15min":0.246},"max":{"1min":12.115,"5min":18.315000000000001,"15min":18.315000000000001},"last":0.65900000000000003}]},{"osd":2,"last update":"Fri Feb 9 18:07:07 2024","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.821,"15min":1.524},"min":{"1min":0.55000000000000004,"5min":0.33000000000000002,"15min":0.27300000000000002},"max":{"1min":13.055,"5min":13.055,"15min":16.190000000000001},"last":0.73399999999999999},{"interface":"front","average":{"1min":1.48,"5min":2.1299999999999999,"15min":1.6479999999999999},"min":{"1min":0.54100000000000004,"5min":0.36099999999999999,"15min":0.28999999999999998},"max":{"1min":13.101000000000001,"5min":17.596,"15min":17.596},"last":0.98799999999999999}]},{"osd":3,"last update":"Fri Feb 9 18:07:32 2024","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.72,"15min":1.4039999999999999},"min":{"1min":0.47199999999999998,"5min":0.307,"15min":0.307},"max":{"1min":7.0469999999999997,"5min":10.881,"15min":10.881},"last":0.85599999999999998},{"interface":"front","average":{"1min":1.373,"5min":1.804,"15min":2.0579999999999998},"min":{"1min":0.42799999999999999,"5min":0.40300000000000002,"15min":0.39500000000000002},"max":{"1min":12.179,"5min":14.202,"15min":16.029},"last":0.83999999999999997}]},{"osd":4,"last update":"Fri Feb 9 18:07:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":1.117,"15min":0.81899999999999995},"min":{"1min":0.34300000000000003,"5min":0.312,"15min":0.27100000000000002},"max":{"1min":0.83399999999999996,"5min":6.0999999999999996,"15min":6.0999999999999996},"last":0.96399999999999997},{"interface":"front","average":{"1min":0.69099999999999995,"5min":1.371,"15min":0.98099999999999998},"min":{"1min":0.44,"5min":0.28000000000000003,"15min":0.26300000000000001},"max":{"1min":0.89700000000000002,"5min":11.433999999999999,"15min":17.814},"last":0.80000000000000004}]},{"osd":5,"last update":"Fri Feb 9 18:07:07 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":1.296,"15min":0.93700000000000006},"min":{"1min":0.34499999999999997,"5min":0.29799999999999999,"15min":0.27500000000000002},"max":{"1min":0.873,"5min":11.406000000000001,"15min":11.406000000000001},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.69599999999999995,"5min":1.3169999999999999,"15min":1.014},"min":{"1min":0.45400000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.93000000000000005,"5min":7.6719999999999997,"15min":7.6719999999999997},"last":0.89200000000000002}]},{"osd":6,"last update":"Fri Feb 9 18:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":1.262,"15min":1.0780000000000001},"min":{"1min":0.50900000000000001,"5min":0.34599999999999997,"15min":0.311},"max":{"1min":0.94599999999999995,"5min":8.0999999999999996,"15min":16.158000000000001},"last":0.93899999999999995},{"interface":"front","average":{"1min":0.74299999999999999,"5min":1.29,"15min":0.999},"min":{"1min":0.40799999999999997,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.0169999999999999,"5min":8.1259999999999994,"15min":8.1259999999999994},"last":1.159}]},{"osd":8,"last update":"Fri Feb 9 18:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":1.2949999999999999,"15min":1.3839999999999999},"min":{"1min":0.63700000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.099,"5min":11.358000000000001,"15min":11.648999999999999},"last":1.026},{"interface":"front","average":{"1min":0.83199999999999996,"5min":1.393,"15min":1.494},"min":{"1min":0.55800000000000005,"5min":0.41699999999999998,"15min":0.375},"max":{"1min":1.0700000000000001,"5min":7.6539999999999999,"15min":11.629},"last":1.008}]},{"osd":9,"last update":"Fri Feb 9 18:07:19 2024","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":1.3819999999999999,"15min":1.5449999999999999},"min":{"1min":0.626,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.1299999999999999,"5min":11.465,"15min":11.465},"last":1.1859999999999999},{"interface":"front","average":{"1min":0.93600000000000005,"5min":1.464,"15min":1.7070000000000001},"min":{"1min":0.58799999999999997,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.1339999999999999,"5min":11.548999999999999,"15min":13.161},"last":1.268}]},{"osd":10,"last update":"Fri Feb 9 18:07:58 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":1.319,"15min":1.599},"min":{"1min":0.63200000000000001,"5min":0.42499999999999999,"15min":0.32800000000000001},"max":{"1min":1.0700000000000001,"5min":7.694,"15min":15.999000000000001},"last":1.2330000000000001},{"interface":"front","average":{"1min":0.85099999999999998,"5min":1.472,"15min":1.5980000000000001},"min":{"1min":0.54700000000000004,"5min":0.48199999999999998,"15min":0.40400000000000003},"max":{"1min":1.0660000000000001,"5min":11.515000000000001,"15min":11.515000000000001},"last":1.069}]},{"osd":11,"last update":"Fri Feb 9 18:07:07 2024","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":1.593,"15min":1.704},"min":{"1min":0.625,"5min":0.44400000000000001,"15min":0.375},"max":{"1min":1.1299999999999999,"5min":9.1890000000000001,"15min":16.111999999999998},"last":1.288},{"interface":"front","average":{"1min":0.89700000000000002,"5min":1.639,"15min":1.7350000000000001},"min":{"1min":0.69599999999999995,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.095,"5min":10.215999999999999,"15min":12.243},"last":1.1100000000000001}]}]},{"osd":6,"up_from":49,"seq":210453398083,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":718060,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":716990,"kb_avail":93027092,"statfs":{"total":95995035648,"available":95259742208,"internally_reserved":0,"allocated":1081344,"data_stored":766778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":734198214},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 18:07:53 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.84899999999999998,"15min":0.88100000000000001},"min":{"1min":0.36099999999999999,"5min":0.27600000000000002,"15min":0.23599999999999999},"max":{"1min":1.234,"5min":7.3630000000000004,"15min":10.356},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.84599999999999997,"15min":0.85399999999999998},"min":{"1min":0.441,"5min":0.27200000000000002,"15min":0.26700000000000002},"max":{"1min":0.98699999999999999,"5min":7.407,"15min":8.9459999999999997},"last":0.89700000000000002}]},{"osd":1,"last update":"Fri Feb 9 18:07:53 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":1.3759999999999999,"15min":1.1240000000000001},"min":{"1min":0.51300000000000001,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.0509999999999999,"5min":32.064,"15min":32.064},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.78300000000000003,"5min":1.4430000000000001,"15min":1.1850000000000001},"min":{"1min":0.51600000000000001,"5min":0.34100000000000003,"15min":0.309},"max":{"1min":1.298,"5min":32.027000000000001,"15min":32.027000000000001},"last":0.76000000000000001}]},{"osd":2,"last update":"Fri Feb 9 18:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":1.032,"15min":1.1359999999999999},"min":{"1min":0.45800000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.038,"5min":7.4779999999999998,"15min":9.2970000000000006},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.79700000000000004,"5min":1.0980000000000001,"15min":1.1899999999999999},"min":{"1min":0.48999999999999999,"5min":0.35699999999999998,"15min":0.34999999999999998},"max":{"1min":1.3620000000000001,"5min":7.5659999999999998,"15min":9.3260000000000005},"last":0.81000000000000005}]},{"osd":3,"last update":"Fri Feb 9 18:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":1.002,"15min":0.90700000000000003},"min":{"1min":0.33300000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.91600000000000004,"5min":8.6880000000000006,"15min":8.6880000000000006},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.627,"5min":0.81799999999999995,"15min":1.036},"min":{"1min":0.41099999999999998,"5min":0.36199999999999999,"15min":0.33500000000000002},"max":{"1min":1.0940000000000001,"5min":8.1760000000000002,"15min":9.0909999999999993},"last":0.745}]},{"osd":4,"last update":"Fri Feb 9 18:07:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.80200000000000005,"15min":0.71099999999999997},"min":{"1min":0.28699999999999998,"5min":0.28100000000000003,"15min":0.26600000000000001},"max":{"1min":0.99199999999999999,"5min":10.215,"15min":10.215},"last":0.497},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.89100000000000001,"15min":0.78200000000000003},"min":{"1min":0.378,"5min":0.36599999999999999,"15min":0.33100000000000002},"max":{"1min":1.1299999999999999,"5min":10.256,"15min":10.256},"last":0.61799999999999999}]},{"osd":5,"last update":"Fri Feb 9 18:07:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.82399999999999995,"15min":0.69499999999999995},"min":{"1min":0.35699999999999998,"5min":0.34599999999999997,"15min":0.26600000000000001},"max":{"1min":0.78800000000000003,"5min":8.1389999999999993,"15min":8.1389999999999993},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.95199999999999996,"15min":0.749},"min":{"1min":0.35799999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.4079999999999999,"5min":7.4470000000000001,"15min":7.4470000000000001},"last":0.85599999999999998}]},{"osd":7,"last update":"Fri Feb 9 18:07:36 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.96499999999999997,"15min":0.83099999999999996},"min":{"1min":0.376,"5min":0.30199999999999999,"15min":0.29499999999999998},"max":{"1min":0.92900000000000005,"5min":13.991,"15min":13.991},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.96899999999999997,"15min":0.82999999999999996},"min":{"1min":0.39900000000000002,"5min":0.38500000000000001,"15min":0.30299999999999999},"max":{"1min":1.0049999999999999,"5min":8.4870000000000001,"15min":8.4870000000000001},"last":0.70899999999999996}]},{"osd":8,"last update":"Fri Feb 9 17:58:34 2024","interfaces":[{"interface":"back","average":{"1min":2.806,"5min":1.635,"15min":1.27},"min":{"1min":0.41599999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":9.2940000000000005,"5min":9.2940000000000005,"15min":9.9480000000000004},"last":0.40300000000000002},{"interface":"front","average":{"1min":2.3780000000000001,"5min":1.623,"15min":1.3109999999999999},"min":{"1min":0.53400000000000003,"5min":0.41399999999999998,"15min":0.40400000000000003},"max":{"1min":9.3469999999999995,"5min":9.3469999999999995,"15min":11.316000000000001},"last":0.57899999999999996}]},{"osd":9,"last update":"Fri Feb 9 18:08:05 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":1.0569999999999999,"15min":1.4019999999999999},"min":{"1min":0.59899999999999998,"5min":0.39900000000000002,"15min":0.36299999999999999},"max":{"1min":1.504,"5min":7.5090000000000003,"15min":36.606000000000002},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.90200000000000002,"5min":1.0189999999999999,"15min":1.546},"min":{"1min":0.626,"5min":0.437,"15min":0.34300000000000003},"max":{"1min":1.458,"5min":7.5380000000000003,"15min":34.408000000000001},"last":0.92800000000000005}]},{"osd":10,"last update":"Fri Feb 9 18:07:36 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.996,"15min":1.4430000000000001},"min":{"1min":0.54500000000000004,"5min":0.48399999999999999,"15min":0.36599999999999999},"max":{"1min":1.2689999999999999,"5min":8.4309999999999992,"15min":21.364999999999998},"last":0.94299999999999995},{"interface":"front","average":{"1min":0.91300000000000003,"5min":1.028,"15min":1.458},"min":{"1min":0.58199999999999996,"5min":0.502,"15min":0.40100000000000002},"max":{"1min":1.3280000000000001,"5min":8.5380000000000003,"15min":21.408999999999999},"last":0.83999999999999997}]},{"osd":11,"last update":"Fri Feb 9 18:07:53 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":1.171,"15min":1.6120000000000001},"min":{"1min":0.505,"5min":0.435,"15min":0.435},"max":{"1min":1.085,"5min":9.6859999999999999,"15min":35.912999999999997},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.89000000000000001,"5min":1.214,"15min":1.6579999999999999},"min":{"1min":0.56399999999999995,"5min":0.51200000000000001,"15min":0.48699999999999999},"max":{"1min":1.214,"5min":9.7300000000000004,"15min":36.156999999999996},"last":0.88800000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575450,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1233772,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":1233022,"kb_avail":92511380,"statfs":{"total":95995035648,"available":94731653120,"internally_reserved":0,"allocated":753664,"data_stored":439098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1262614981},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 18:07:16 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":1.046,"15min":1.375},"min":{"1min":0.31,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":0.63600000000000001,"5min":6.6079999999999997,"15min":14.885999999999999},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.51400000000000001,"5min":1.204,"15min":1.52},"min":{"1min":0.36599999999999999,"5min":0.26900000000000002,"15min":0.246},"max":{"1min":0.78100000000000003,"5min":9.6969999999999992,"15min":15.411},"last":0.438}]},{"osd":2,"last update":"Fri Feb 9 18:07:35 2024","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.262,"15min":1.4950000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.28299999999999997},"max":{"1min":11.683999999999999,"5min":11.683999999999999,"15min":13.173},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.88700000000000001,"5min":1.1779999999999999,"15min":1.349},"min":{"1min":0.40200000000000002,"5min":0.32000000000000001,"15min":0.28100000000000003},"max":{"1min":6.7149999999999999,"5min":7.8090000000000002,"15min":11.103},"last":0.48699999999999999}]},{"osd":3,"last update":"Fri Feb 9 18:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":1.2969999999999999,"15min":0.98699999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.55500000000000005,"5min":13.292999999999999,"15min":13.292999999999999},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.46200000000000002,"5min":1.3640000000000001,"15min":1.3169999999999999},"min":{"1min":0.34300000000000003,"5min":0.29099999999999998,"15min":0.28999999999999998},"max":{"1min":0.69399999999999995,"5min":13.265000000000001,"15min":13.265000000000001},"last":0.39400000000000002}]},{"osd":4,"last update":"Fri Feb 9 18:07:19 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":1.4299999999999999,"15min":1.2210000000000001},"min":{"1min":0.46999999999999997,"5min":0.38600000000000001,"15min":0.27600000000000002},"max":{"1min":0.84099999999999997,"5min":13.151,"15min":13.151},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.629,"5min":1.6679999999999999,"15min":1.5609999999999999},"min":{"1min":0.46800000000000003,"5min":0.33800000000000002,"15min":0.26400000000000001},"max":{"1min":0.80100000000000005,"5min":13.177,"15min":15.331},"last":0.495}]},{"osd":5,"last update":"Fri Feb 9 18:07:54 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":1.0720000000000001,"15min":1.3100000000000001},"min":{"1min":0.46300000000000002,"5min":0.371,"15min":0.29399999999999998},"max":{"1min":1.0109999999999999,"5min":4.9859999999999998,"15min":15.106999999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.73399999999999999,"5min":1.4330000000000001,"15min":1.478},"min":{"1min":0.57499999999999996,"5min":0.318,"15min":0.28000000000000003},"max":{"1min":1.0680000000000001,"5min":13.106,"15min":15.355},"last":0.54700000000000004}]},{"osd":6,"last update":"Fri Feb 9 18:07:07 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":1.4450000000000001,"15min":1.4379999999999999},"min":{"1min":0.48499999999999999,"5min":0.39300000000000002,"15min":0.24299999999999999},"max":{"1min":0.83499999999999996,"5min":10.605,"15min":15.154999999999999},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.64600000000000002,"5min":1.371,"15min":1.446},"min":{"1min":0.45200000000000001,"5min":0.376,"15min":0.32800000000000001},"max":{"1min":0.92600000000000005,"5min":10.720000000000001,"15min":15.253},"last":0.93500000000000005}]},{"osd":7,"last update":"Fri Feb 9 18:07:07 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":1.542,"15min":1.5609999999999999},"min":{"1min":0.60299999999999998,"5min":0.40799999999999997,"15min":0.316},"max":{"1min":0.98799999999999999,"5min":10.58,"15min":15.287000000000001},"last":0.75},{"interface":"front","average":{"1min":0.71599999999999997,"5min":1.1579999999999999,"15min":1.4490000000000001},"min":{"1min":0.49099999999999999,"5min":0.379,"15min":0.379},"max":{"1min":0.86099999999999999,"5min":5.2919999999999998,"15min":15.311},"last":0.63600000000000001}]},{"osd":8,"last update":"Fri Feb 9 18:07:16 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":1.24,"15min":1.6379999999999999},"min":{"1min":0.504,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.089,"5min":10.627000000000001,"15min":10.627000000000001},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.82399999999999995,"5min":1.2050000000000001,"15min":1.867},"min":{"1min":0.51400000000000001,"5min":0.44500000000000001,"15min":0.36399999999999999},"max":{"1min":1.2250000000000001,"5min":5.7800000000000002,"15min":12.464},"last":0.79600000000000004}]},{"osd":9,"last update":"Fri Feb 9 18:07:16 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":1.363,"15min":1.9470000000000001},"min":{"1min":0.46300000000000002,"5min":0.42999999999999999,"15min":0.42799999999999999},"max":{"1min":1.125,"5min":10.337999999999999,"15min":20.789000000000001},"last":0.97499999999999998},{"interface":"front","average":{"1min":0.80600000000000005,"5min":1.804,"15min":1.8779999999999999},"min":{"1min":0.51400000000000001,"5min":0.375,"15min":0.34200000000000003},"max":{"1min":1.1020000000000001,"5min":13.231999999999999,"15min":13.231999999999999},"last":0.72699999999999998}]},{"osd":10,"last update":"Fri Feb 9 18:07:16 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.81599999999999995,"15min":0.81399999999999995},"min":{"1min":0.63100000000000001,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.871,"15min":0.86799999999999999},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.91800000000000004}]},{"osd":11,"last update":"Fri Feb 9 18:07:07 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":1.5449999999999999,"15min":2.2650000000000001},"min":{"1min":0.63400000000000001,"5min":0.441,"15min":0.378},"max":{"1min":1.175,"5min":10.593999999999999,"15min":16.881},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.82899999999999996,"5min":2.1080000000000001,"15min":2.3759999999999999},"min":{"1min":0.61899999999999999,"5min":0.41799999999999998,"15min":0.372},"max":{"1min":1.147,"5min":13.205,"15min":15.209},"last":0.82399999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738974,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":902688,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":901630,"kb_avail":92842464,"statfs":{"total":95995035648,"available":95070683136,"internally_reserved":0,"allocated":1069056,"data_stored":759262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":923269573},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 9 18:07:56 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.76300000000000001,"15min":1.417},"min":{"1min":0.30399999999999999,"5min":0.25900000000000001,"15min":0.23100000000000001},"max":{"1min":0.54700000000000004,"5min":6.8070000000000004,"15min":23.282},"last":0.378},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.84999999999999998,"15min":1.496},"min":{"1min":0.38600000000000001,"5min":0.35499999999999998,"15min":0.29199999999999998},"max":{"1min":0.63600000000000001,"5min":6.8410000000000002,"15min":23.32},"last":0.46800000000000003}]},{"osd":2,"last update":"Fri Feb 9 18:07:30 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.95999999999999996,"15min":1.4690000000000001},"min":{"1min":0.29499999999999998,"5min":0.27400000000000002,"15min":0.253},"max":{"1min":0.68899999999999995,"5min":11.161,"15min":20.013999999999999},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.88100000000000001,"15min":1.462},"min":{"1min":0.39800000000000002,"5min":0.27100000000000002,"15min":0.219},"max":{"1min":0.67200000000000004,"5min":6.8730000000000002,"15min":20.103999999999999},"last":0.441}]},{"osd":3,"last update":"Fri Feb 9 18:07:30 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":1.599,"15min":1.4570000000000001},"min":{"1min":0.318,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.59899999999999998,"5min":45.688000000000002,"15min":45.688000000000002},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.498,"5min":1.6160000000000001,"15min":1.7549999999999999},"min":{"1min":0.38200000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.69799999999999995,"5min":45.551000000000002,"15min":45.551000000000002},"last":0.54400000000000004}]},{"osd":4,"last update":"Fri Feb 9 18:07:30 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":1.0109999999999999,"15min":1.2490000000000001},"min":{"1min":0.439,"5min":0.33900000000000002,"15min":0.28100000000000003},"max":{"1min":0.99399999999999999,"5min":11.393000000000001,"15min":12.757},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.752,"5min":1.1100000000000001,"15min":1.4470000000000001},"min":{"1min":0.53700000000000003,"5min":0.39000000000000001,"15min":0.25800000000000001},"max":{"1min":1.0429999999999999,"5min":11.114000000000001,"15min":17.091999999999999},"last":0.56200000000000006}]},{"osd":5,"last update":"Fri Feb 9 18:07:42 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":1.1080000000000001,"15min":1.2230000000000001},"min":{"1min":0.47799999999999998,"5min":0.38500000000000001,"15min":0.30199999999999999},"max":{"1min":1.1850000000000001,"5min":11.435,"15min":11.435},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.76500000000000001,"5min":1.1879999999999999,"15min":1.292},"min":{"1min":0.51100000000000001,"5min":0.44,"15min":0.38300000000000001},"max":{"1min":1.153,"5min":11.462999999999999,"15min":11.462999999999999},"last":0.53300000000000003}]},{"osd":6,"last update":"Fri Feb 9 18:07:10 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":1.01,"15min":1.357},"min":{"1min":0.48299999999999998,"5min":0.313,"15min":0.307},"max":{"1min":1.069,"5min":6.069,"15min":15.667},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.90400000000000003,"15min":1.2949999999999999},"min":{"1min":0.46100000000000002,"5min":0.432,"15min":0.34799999999999998},"max":{"1min":0.86199999999999999,"5min":4.5529999999999999,"15min":15.702999999999999},"last":0.62}]},{"osd":7,"last update":"Fri Feb 9 18:07:42 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.98899999999999999,"15min":1.409},"min":{"1min":0.42299999999999999,"5min":0.39400000000000002,"15min":0.307},"max":{"1min":0.876,"5min":5.9729999999999999,"15min":11.682},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.90400000000000003,"15min":1.3779999999999999},"min":{"1min":0.495,"5min":0.46000000000000002,"15min":0.32600000000000001},"max":{"1min":0.92800000000000005,"5min":4.7729999999999997,"15min":17.152999999999999},"last":0.67900000000000005}]},{"osd":8,"last update":"Fri Feb 9 18:07:42 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.97899999999999998,"15min":1.738},"min":{"1min":0.57999999999999996,"5min":0.40600000000000003,"15min":0.33800000000000002},"max":{"1min":1.2150000000000001,"5min":4.4290000000000003,"15min":17.189},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.86099999999999999,"5min":1.046,"15min":1.6950000000000001},"min":{"1min":0.623,"5min":0.47499999999999998,"15min":0.36499999999999999},"max":{"1min":1.1439999999999999,"5min":5.7939999999999996,"15min":18.600000000000001},"last":0.746}]},{"osd":9,"last update":"Fri Feb 9 18:07:30 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":1.252,"15min":2.0430000000000001},"min":{"1min":0.67700000000000005,"5min":0.443,"15min":0.39000000000000001},"max":{"1min":1.1739999999999999,"5min":6.6070000000000002,"15min":17.216000000000001},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.88900000000000001,"5min":1.0900000000000001,"15min":1.7949999999999999},"min":{"1min":0.66100000000000003,"5min":0.50700000000000001,"15min":0.40899999999999997},"max":{"1min":1.0800000000000001,"5min":5.8159999999999998,"15min":19.986999999999998},"last":0.84499999999999997}]},{"osd":10,"last update":"Fri Feb 9 18:07:56 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.97299999999999998,"15min":1.996},"min":{"1min":0.66000000000000003,"5min":0.432,"15min":0.42499999999999999},"max":{"1min":1.077,"5min":4.4630000000000001,"15min":57.979999999999997},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.96399999999999997,"15min":2.153},"min":{"1min":0.58899999999999997,"5min":0.46200000000000002,"15min":0.41299999999999998},"max":{"1min":1.1579999999999999,"5min":4.5880000000000001,"15min":57.606999999999999},"last":0.80600000000000005}]},{"osd":11,"last update":"Fri Feb 9 18:07:10 2024","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":1.0740000000000001,"15min":1.9690000000000001},"min":{"1min":0.55200000000000005,"5min":0.47599999999999998,"15min":0.44800000000000001},"max":{"1min":1.1859999999999999,"5min":6.3899999999999997,"15min":58.351999999999997},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.90000000000000002,"5min":1.145,"15min":1.9750000000000001},"min":{"1min":0.64000000000000001,"5min":0.499,"15min":0.441},"max":{"1min":1.1299999999999999,"5min":6.4459999999999997,"15min":57.314},"last":0.80000000000000004}]}]},{"osd":2,"up_from":18,"seq":77309411924,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":690164,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93054988,"statfs":{"total":95995035648,"available":95288307712,"internally_reserved":0,"allocated":958464,"data_stored":640346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:58:48 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.53200000000000003,"15min":0.48699999999999999},"min":{"1min":0.23699999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.9619999999999997,"5min":4.9619999999999997,"15min":4.9619999999999997},"last":0.33800000000000002},{"interface":"front","average":{"1min":1.292,"5min":0.81899999999999995,"15min":0.73999999999999999},"min":{"1min":0.33000000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":4.5430000000000001,"5min":4.5430000000000001,"15min":4.5430000000000001},"last":0.47799999999999998}]},{"osd":1,"last update":"Fri Feb 9 18:07:38 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":1.1399999999999999,"15min":1.367},"min":{"1min":0.25900000000000001,"5min":0.23699999999999999,"15min":0.20799999999999999},"max":{"1min":0.621,"5min":25.167999999999999,"15min":25.167999999999999},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.46200000000000002,"5min":1.2390000000000001,"15min":1.395},"min":{"1min":0.28899999999999998,"5min":0.249,"15min":0.248},"max":{"1min":0.72399999999999998,"5min":25.143000000000001,"15min":25.143000000000001},"last":0.42299999999999999}]},{"osd":3,"last update":"Fri Feb 9 18:07:24 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":1.2889999999999999,"15min":2.347},"min":{"1min":0.308,"5min":0.25,"15min":0.19900000000000001},"max":{"1min":0.66700000000000004,"5min":25.093,"15min":29.228000000000002},"last":0.38},{"interface":"front","average":{"1min":0.38700000000000001,"5min":1.1359999999999999,"15min":1.3460000000000001},"min":{"1min":0.27100000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.64300000000000002,"5min":20.093,"15min":20.093},"last":0.32300000000000001}]},{"osd":4,"last update":"Fri Feb 9 18:07:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":1.0009999999999999,"15min":0.999},"min":{"1min":0.47199999999999998,"5min":0.30199999999999999,"15min":0.245},"max":{"1min":0.90200000000000002,"5min":9.609,"15min":10.365},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.69999999999999996,"5min":1.042,"15min":1.073},"min":{"1min":0.442,"5min":0.36699999999999999,"15min":0.26600000000000001},"max":{"1min":0.86099999999999999,"5min":9.6500000000000004,"15min":10.529},"last":0.502}]},{"osd":5,"last update":"Fri Feb 9 18:07:55 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":1.232,"15min":1.214},"min":{"1min":0.39700000000000002,"5min":0.29199999999999998,"15min":0.245},"max":{"1min":0.88400000000000001,"5min":11.536,"15min":12.305999999999999},"last":0.439},{"interface":"front","average":{"1min":0.65300000000000002,"5min":1.2689999999999999,"15min":1.2330000000000001},"min":{"1min":0.42599999999999999,"5min":0.40500000000000003,"15min":0.26900000000000002},"max":{"1min":0.93999999999999995,"5min":11.571,"15min":11.571},"last":0.51400000000000001}]},{"osd":6,"last update":"Fri Feb 9 18:07:10 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":1.2549999999999999,"15min":1.3049999999999999},"min":{"1min":0.377,"5min":0.316,"15min":0.28399999999999997},"max":{"1min":0.94699999999999995,"5min":7.8460000000000001,"15min":11.465999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.624,"5min":1.278,"15min":1.341},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.28199999999999997},"max":{"1min":0.88300000000000001,"5min":7.9489999999999998,"15min":11.598000000000001},"last":0.39900000000000002}]},{"osd":7,"last update":"Fri Feb 9 18:07:38 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.95899999999999996,"15min":1.0980000000000001},"min":{"1min":0.438,"5min":0.33300000000000002,"15min":0.33100000000000002},"max":{"1min":0.90400000000000003,"5min":4.9770000000000003,"15min":10.403},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.67000000000000004,"5min":1.26,"15min":1.3180000000000001},"min":{"1min":0.47799999999999998,"5min":0.38800000000000001,"15min":0.309},"max":{"1min":0.89100000000000001,"5min":11.603,"15min":11.603},"last":0.47299999999999998}]},{"osd":8,"last update":"Fri Feb 9 18:07:10 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.98999999999999999,"15min":1.5920000000000001},"min":{"1min":0.47299999999999998,"5min":0.314,"15min":0.314},"max":{"1min":1.073,"5min":7.5060000000000002,"15min":12.500999999999999},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.752,"5min":1.484,"15min":1.7689999999999999},"min":{"1min":0.41199999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.0189999999999999,"5min":8.1890000000000001,"15min":11.634},"last":0.753}]},{"osd":9,"last update":"Fri Feb 9 18:07:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":1.0720000000000001,"15min":1.7310000000000001},"min":{"1min":0.44700000000000001,"5min":0.34499999999999997,"15min":0.318},"max":{"1min":1.0289999999999999,"5min":5.9299999999999997,"15min":12.276},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.77400000000000002,"5min":1.4590000000000001,"15min":1.9019999999999999},"min":{"1min":0.48299999999999998,"5min":0.45200000000000001,"15min":0.36199999999999999},"max":{"1min":1.04,"5min":7.9169999999999998,"15min":12.148},"last":0.70899999999999996}]},{"osd":10,"last update":"Fri Feb 9 18:07:47 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":1.1220000000000001,"15min":1.6819999999999999},"min":{"1min":0.55400000000000005,"5min":0.47199999999999998,"15min":0.40799999999999997},"max":{"1min":1.0509999999999999,"5min":6.3609999999999998,"15min":12.313000000000001},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.79000000000000004,"5min":1.24,"15min":1.716},"min":{"1min":0.60599999999999998,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.97999999999999998,"5min":10.968999999999999,"15min":12.343999999999999},"last":0.65900000000000003}]},{"osd":11,"last update":"Fri Feb 9 18:07:10 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":1.6160000000000001,"15min":1.9279999999999999},"min":{"1min":0.48499999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.0629999999999999,"5min":12.212999999999999,"15min":12.212999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.77700000000000002,"5min":1.3120000000000001,"15min":1.843},"min":{"1min":0.58699999999999997,"5min":0.40200000000000002,"15min":0.35499999999999998},"max":{"1min":1.0449999999999999,"5min":11.922000000000001,"15min":12.243},"last":0.77600000000000002}]}]},{"osd":3,"up_from":854,"seq":3667902070895,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":270252,"kb_used_data":1056,"kb_used_omap":202,"kb_used_meta":268981,"kb_avail":93474900,"statfs":{"total":95995035648,"available":95718297600,"internally_reserved":0,"allocated":1081344,"data_stored":766778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207633,"internal_metadata":275436783},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 18:07:25 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.93100000000000005,"15min":0.998},"min":{"1min":0.246,"5min":0.246,"15min":0.22500000000000001},"max":{"1min":0.56699999999999995,"5min":9.6189999999999998,"15min":9.6189999999999998},"last":0.372},{"interface":"front","average":{"1min":0.45400000000000001,"5min":1.0209999999999999,"15min":1.425},"min":{"1min":0.30499999999999999,"5min":0.25800000000000001,"15min":0.20599999999999999},"max":{"1min":0.626,"5min":11.287000000000001,"15min":17.077999999999999},"last":0.46700000000000003}]},{"osd":1,"last update":"Fri Feb 9 18:07:22 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":1.53,"15min":1.1200000000000001},"min":{"1min":0.28199999999999997,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.60799999999999998,"5min":21.73,"15min":21.73},"last":0.504},{"interface":"front","average":{"1min":0.45600000000000002,"5min":1.639,"15min":1.022},"min":{"1min":0.28199999999999997,"5min":0.27300000000000002,"15min":0.23899999999999999},"max":{"1min":0.65100000000000002,"5min":19.268000000000001,"15min":19.268000000000001},"last":0.48499999999999999}]},{"osd":2,"last update":"Fri Feb 9 18:07:25 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":1.2230000000000001,"15min":1.948},"min":{"1min":0.28199999999999997,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.72099999999999997,"5min":9.5899999999999999,"15min":13.098000000000001},"last":0.437},{"interface":"front","average":{"1min":0.5,"5min":1.2,"15min":1.2569999999999999},"min":{"1min":0.35499999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.74099999999999999,"5min":8.8539999999999992,"15min":9.3260000000000005},"last":0.51800000000000002}]},{"osd":4,"last update":"Fri Feb 9 18:07:25 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":1.252,"15min":1.0289999999999999},"min":{"1min":0.51000000000000001,"5min":0.32700000000000001,"15min":0.28100000000000003},"max":{"1min":0.79800000000000004,"5min":16.329000000000001,"15min":16.329000000000001},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.65500000000000003,"5min":1.0629999999999999,"15min":1.282},"min":{"1min":0.46500000000000002,"5min":0.32800000000000001,"15min":0.318},"max":{"1min":0.97199999999999998,"5min":6.5679999999999996,"15min":10.755000000000001},"last":0.61499999999999999}]},{"osd":5,"last update":"Fri Feb 9 18:07:25 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":1.2450000000000001,"15min":1.1180000000000001},"min":{"1min":0.45700000000000002,"5min":0.34999999999999998,"15min":0.28499999999999998},"max":{"1min":0.97099999999999997,"5min":16.405000000000001,"15min":16.405000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.64600000000000002,"5min":1.0349999999999999,"15min":1.3959999999999999},"min":{"1min":0.50700000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.82599999999999996,"5min":6.6440000000000001,"15min":10.707000000000001},"last":0.56100000000000005}]},{"osd":6,"last update":"Fri Feb 9 18:07:25 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":1.155,"15min":1.657},"min":{"1min":0.49299999999999999,"5min":0.41599999999999998,"15min":0.35899999999999999},"max":{"1min":0.997,"5min":6.4050000000000002,"15min":10.808},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.68999999999999995,"5min":1.075,"15min":0.88},"min":{"1min":0.441,"5min":0.32200000000000001,"15min":0.29799999999999999},"max":{"1min":0.86899999999999999,"5min":5.8920000000000003,"15min":8.3550000000000004},"last":0.64200000000000002}]},{"osd":7,"last update":"Fri Feb 9 18:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":1.129,"15min":1.7629999999999999},"min":{"1min":0.437,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.82799999999999996,"5min":9.423,"15min":9.423},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.68700000000000006,"5min":1.2549999999999999,"15min":1.9199999999999999},"min":{"1min":0.50800000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.88300000000000001,"5min":16.439,"15min":16.439},"last":0.55200000000000005}]},{"osd":8,"last update":"Fri Feb 9 18:07:25 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":1.3460000000000001,"15min":1.343},"min":{"1min":0.50700000000000001,"5min":0.40400000000000003,"15min":0.34499999999999997},"max":{"1min":1.1359999999999999,"5min":16.228000000000002,"15min":16.228000000000002},"last":0.871},{"interface":"front","average":{"1min":0.80100000000000005,"5min":1.21,"15min":1.6259999999999999},"min":{"1min":0.63400000000000001,"5min":0.41399999999999998,"15min":0.38500000000000001},"max":{"1min":1.085,"5min":6.6040000000000001,"15min":10.782},"last":0.89700000000000002}]},{"osd":9,"last update":"Fri Feb 9 18:07:25 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":1.526,"15min":1.028},"min":{"1min":0.51500000000000001,"5min":0.39600000000000002,"15min":0.38200000000000001},"max":{"1min":0.995,"5min":16.172999999999998,"15min":16.172999999999998},"last":0.751},{"interface":"front","average":{"1min":0.77500000000000002,"5min":1.462,"15min":1.387},"min":{"1min":0.56899999999999995,"5min":0.42499999999999999,"15min":0.32400000000000001},"max":{"1min":1.119,"5min":9.8350000000000009,"15min":9.8350000000000009},"last":0.69299999999999995}]},{"osd":10,"last update":"Fri Feb 9 18:04:15 2024","interfaces":[{"interface":"back","average":{"1min":2.206,"5min":2.206,"15min":2.206},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":8.7870000000000008,"5min":8.7870000000000008,"15min":8.7870000000000008},"last":0.73099999999999998},{"interface":"front","average":{"1min":3.2040000000000002,"5min":3.2040000000000002,"15min":3.2040000000000002},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":16.263000000000002,"5min":16.263000000000002,"15min":16.263000000000002},"last":0.83199999999999996}]},{"osd":11,"last update":"Fri Feb 9 18:07:25 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":1.3,"15min":1.8839999999999999},"min":{"1min":0.59599999999999997,"5min":0.497,"15min":0.38800000000000001},"max":{"1min":1.0209999999999999,"5min":16.295000000000002,"15min":17.126000000000001},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.79100000000000004,"5min":1.4590000000000001,"15min":1.2070000000000001},"min":{"1min":0.60499999999999998,"5min":0.36199999999999999,"15min":0.34799999999999998},"max":{"1min":1.097,"5min":9.375,"15min":9.375},"last":0.88200000000000001}]}]},{"osd":4,"up_from":32,"seq":137438954061,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":906540,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":905790,"kb_avail":92838612,"statfs":{"total":95995035648,"available":95066738688,"internally_reserved":0,"allocated":753664,"data_stored":439098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":927529413},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:58:54 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.749,"15min":1.2709999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.22800000000000001},"max":{"1min":1.734,"5min":7.7359999999999998,"15min":10.301},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.81799999999999995,"15min":1.3400000000000001},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.768,"5min":7.7729999999999997,"15min":10.407999999999999},"last":0.35599999999999998}]},{"osd":1,"last update":"Fri Feb 9 18:07:14 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.96099999999999997,"15min":1.3620000000000001},"min":{"1min":0.46600000000000003,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.68999999999999995,"5min":6.1340000000000003,"15min":50.255000000000003},"last":0.627},{"interface":"front","average":{"1min":0.67700000000000005,"5min":1.034,"15min":1.5600000000000001},"min":{"1min":0.51400000000000001,"5min":0.30099999999999999,"15min":0.27800000000000002},"max":{"1min":0.79400000000000004,"5min":6.2999999999999998,"15min":49.991999999999997},"last":0.71599999999999997}]},{"osd":2,"last update":"Fri Feb 9 18:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":1.0449999999999999,"15min":0.98199999999999998},"min":{"1min":0.40000000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.90600000000000003,"5min":6.1799999999999997,"15min":6.1799999999999997},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.66100000000000003,"5min":1.0940000000000001,"15min":1.016},"min":{"1min":0.42299999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.83899999999999997,"5min":6.2220000000000004,"15min":6.2220000000000004},"last":0.69899999999999995}]},{"osd":3,"last update":"Fri Feb 9 18:07:27 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.90800000000000003,"15min":1.0369999999999999},"min":{"1min":0.52900000000000003,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.877,"5min":7,"15min":7},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.89500000000000002,"15min":1.046},"min":{"1min":0.49399999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.82799999999999996,"5min":6.0970000000000004,"15min":6.0970000000000004},"last":0.77100000000000002}]},{"osd":5,"last update":"Fri Feb 9 18:07:37 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.81599999999999995,"15min":0.65700000000000003},"min":{"1min":0.35399999999999998,"5min":0.27200000000000002,"15min":0.24399999999999999},"max":{"1min":0.69099999999999995,"5min":7.0469999999999997,"15min":7.0469999999999997},"last":0.502},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.90000000000000002,"15min":0.73599999999999999},"min":{"1min":0.45000000000000001,"5min":0.32200000000000001,"15min":0.314},"max":{"1min":0.78000000000000003,"5min":7.0780000000000003,"15min":7.0780000000000003},"last":0.58099999999999996}]},{"osd":6,"last update":"Fri Feb 9 18:07:14 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":1.016,"15min":0.91500000000000004},"min":{"1min":0.32500000000000001,"5min":0.28999999999999998,"15min":0.27600000000000002},"max":{"1min":0.93200000000000005,"5min":6.4009999999999998,"15min":23.343},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.67000000000000004,"5min":1.111,"15min":0.95199999999999996},"min":{"1min":0.45200000000000001,"5min":0.29899999999999999,"15min":0.28999999999999998},"max":{"1min":0.97799999999999998,"5min":9.2260000000000009,"15min":21.861000000000001},"last":0.68400000000000005}]},{"osd":7,"last update":"Fri Feb 9 18:07:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.79700000000000004,"15min":0.76600000000000001},"min":{"1min":0.40999999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.84599999999999997,"5min":5.2030000000000003,"15min":5.2030000000000003},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.80000000000000004,"15min":0.77600000000000002},"min":{"1min":0.51700000000000002,"5min":0.36099999999999999,"15min":0.30599999999999999},"max":{"1min":0.91700000000000004,"5min":5.2549999999999999,"15min":6.7919999999999998},"last":0.65500000000000003}]},{"osd":8,"last update":"Fri Feb 9 18:07:37 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":1.1479999999999999,"15min":1.2430000000000001},"min":{"1min":0.59699999999999998,"5min":0.38900000000000001,"15min":0.33100000000000002},"max":{"1min":1.03,"5min":6.4619999999999997,"15min":9.7059999999999995},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.84599999999999997,"5min":1.196,"15min":1.2729999999999999},"min":{"1min":0.65700000000000003,"5min":0.42899999999999999,"15min":0.35899999999999999},"max":{"1min":1.179,"5min":6.4329999999999998,"15min":9.0109999999999992},"last":0.75700000000000001}]},{"osd":9,"last update":"Fri Feb 9 18:07:37 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":1.4199999999999999,"15min":1.427},"min":{"1min":0.55000000000000004,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.97699999999999998,"5min":12.583,"15min":12.583},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.79500000000000004,"5min":1.4079999999999999,"15min":1.4350000000000001},"min":{"1min":0.65700000000000003,"5min":0.47399999999999998,"15min":0.373},"max":{"1min":1.004,"5min":10.679,"15min":10.679},"last":0.88200000000000001}]},{"osd":10,"last update":"Fri Feb 9 18:07:54 2024","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":1.026,"15min":1.421},"min":{"1min":0.61199999999999999,"5min":0.47699999999999998,"15min":0.41299999999999998},"max":{"1min":1.2090000000000001,"5min":6.2169999999999996,"15min":8.2829999999999995},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.84499999999999997,"5min":1.083,"15min":1.321},"min":{"1min":0.63,"5min":0.42799999999999999,"15min":0.41299999999999998},"max":{"1min":1.091,"5min":6.0199999999999996,"15min":7.9189999999999996},"last":0.83599999999999997}]},{"osd":11,"last update":"Fri Feb 9 18:07:27 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":1.1779999999999999,"15min":1.4079999999999999},"min":{"1min":0.64200000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.232,"5min":7.8620000000000001,"15min":7.8620000000000001},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.85999999999999999,"5min":1.282,"15min":1.502},"min":{"1min":0.57499999999999996,"5min":0.46500000000000002,"15min":0.46400000000000002},"max":{"1min":1.1870000000000001,"5min":8.2219999999999995,"15min":9.5960000000000001},"last":0.86099999999999999}]}]},{"osd":5,"up_from":40,"seq":171798692422,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":978920,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":977982,"kb_avail":92766232,"statfs":{"total":95995035648,"available":94992621568,"internally_reserved":0,"allocated":946176,"data_stored":632830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1001454021},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 17:58:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.82599999999999996,"15min":0.874},"min":{"1min":0.27800000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":1.341,"5min":5.399,"15min":5.399},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.622,"5min":0.92100000000000004,"15min":0.97099999999999997},"min":{"1min":0.40400000000000003,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.409,"5min":5.4329999999999998,"15min":5.4329999999999998},"last":0.52100000000000002}]},{"osd":1,"last update":"Fri Feb 9 18:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.89200000000000002,"15min":0.98099999999999998},"min":{"1min":0.48599999999999999,"5min":0.247,"15min":0.247},"max":{"1min":0.80200000000000005,"5min":4.8109999999999999,"15min":16.271000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.98699999999999999,"15min":0.997},"min":{"1min":0.46100000000000002,"5min":0.27600000000000002,"15min":0.23999999999999999},"max":{"1min":0.89000000000000001,"5min":7.2839999999999998,"15min":16.344999999999999},"last":0.67000000000000004}]},{"osd":2,"last update":"Fri Feb 9 18:07:13 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":1.175,"15min":0.97899999999999998},"min":{"1min":0.47299999999999998,"5min":0.307,"15min":0.254},"max":{"1min":0.78400000000000003,"5min":7.71,"15min":7.71},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.72299999999999998,"5min":1.1399999999999999,"15min":1.1359999999999999},"min":{"1min":0.504,"5min":0.29199999999999998,"15min":0.24399999999999999},"max":{"1min":1.004,"5min":8.0030000000000001,"15min":8.0030000000000001},"last":0.58599999999999997}]},{"osd":3,"last update":"Fri Feb 9 18:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":1.014,"15min":0.82899999999999996},"min":{"1min":0.51700000000000002,"5min":0.28000000000000003,"15min":0.26700000000000002},"max":{"1min":0.74399999999999999,"5min":10.896000000000001,"15min":10.896000000000001},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.67800000000000005,"5min":1.115,"15min":1.1000000000000001},"min":{"1min":0.504,"5min":0.34300000000000003,"15min":0.28000000000000003},"max":{"1min":0.83499999999999996,"5min":6.9630000000000001,"15min":7.218},"last":0.63500000000000001}]},{"osd":4,"last update":"Fri Feb 9 18:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.82099999999999995,"15min":0.71499999999999997},"min":{"1min":0.41899999999999998,"5min":0.27400000000000002,"15min":0.25900000000000001},"max":{"1min":0.72299999999999998,"5min":5.7990000000000004,"15min":5.7990000000000004},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.997,"15min":0.82899999999999996},"min":{"1min":0.54600000000000004,"5min":0.29599999999999999,"15min":0.22500000000000001},"max":{"1min":0.79800000000000004,"5min":6.1040000000000001,"15min":6.1239999999999997},"last":0.54200000000000004}]},{"osd":6,"last update":"Fri Feb 9 18:07:13 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.88200000000000001,"15min":0.78800000000000003},"min":{"1min":0.39200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.86099999999999999,"5min":6.8230000000000004,"15min":6.8230000000000004},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.081,"15min":0.89500000000000002},"min":{"1min":0.42599999999999999,"5min":0.34899999999999998,"15min":0.33100000000000002},"max":{"1min":0.90600000000000003,"5min":6.9290000000000003,"15min":6.9290000000000003},"last":0.52000000000000002}]},{"osd":7,"last update":"Fri Feb 9 18:07:46 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":1.117,"15min":0.93500000000000005},"min":{"1min":0.432,"5min":0.36499999999999999,"15min":0.28199999999999997},"max":{"1min":0.76500000000000001,"5min":12.798999999999999,"15min":12.798999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.746,"5min":1.1910000000000001,"15min":0.92100000000000004},"min":{"1min":0.61199999999999999,"5min":0.48399999999999999,"15min":0.32400000000000001},"max":{"1min":0.88800000000000001,"5min":12.885999999999999,"15min":12.885999999999999},"last":0.60099999999999998}]},{"osd":8,"last update":"Fri Feb 9 18:08:01 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":1.1859999999999999,"15min":1.383},"min":{"1min":0.67500000000000004,"5min":0.35299999999999998,"15min":0.34200000000000003},"max":{"1min":1.042,"5min":7.71,"15min":12.227},"last":0.751},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.86099999999999999,"15min":1.1299999999999999},"min":{"1min":0.62,"5min":0.376,"15min":0.372},"max":{"1min":1,"5min":4.641,"15min":8.3770000000000007},"last":0.77600000000000002}]},{"osd":9,"last update":"Fri Feb 9 18:07:13 2024","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":1.4359999999999999,"15min":1.6519999999999999},"min":{"1min":0.63900000000000001,"5min":0.39800000000000002,"15min":0.315},"max":{"1min":1.139,"5min":21.686,"15min":21.686},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.92000000000000004,"5min":1.5740000000000001,"15min":1.494},"min":{"1min":0.65400000000000003,"5min":0.42499999999999999,"15min":0.36799999999999999},"max":{"1min":1.2849999999999999,"5min":21.734999999999999,"15min":21.734999999999999},"last":0.69299999999999995}]},{"osd":10,"last update":"Fri Feb 9 18:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":1.5700000000000001,"15min":1.4219999999999999},"min":{"1min":0.66500000000000004,"5min":0.41599999999999998,"15min":0.39300000000000002},"max":{"1min":1.046,"5min":21.027000000000001,"15min":21.027000000000001},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.94099999999999995,"5min":1.5609999999999999,"15min":1.571},"min":{"1min":0.78800000000000003,"5min":0.443,"15min":0.39900000000000002},"max":{"1min":1.367,"5min":20.859999999999999,"15min":20.859999999999999},"last":0.71799999999999997}]},{"osd":11,"last update":"Fri Feb 9 18:08:01 2024","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":1.2,"15min":1.4179999999999999},"min":{"1min":0.59899999999999998,"5min":0.50900000000000001,"15min":0.434},"max":{"1min":1.101,"5min":9.7959999999999994,"15min":9.7959999999999994},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.93700000000000006,"5min":1.1839999999999999,"15min":1.4299999999999999},"min":{"1min":0.77800000000000002,"5min":0.51900000000000002,"15min":0.434},"max":{"1min":1.1599999999999999,"5min":6.899,"15min":9.8819999999999997},"last":0.74099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-09T18:08:08.469 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-02-09T18:08:08.469 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:720625d38f7f95e6fa31efce87133247d9d28517 shell --fsid a69c870a-c76e-11ee-95b7-87774f69a715 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-02-09T18:08:08.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:08:08 smithi179 ceph-mon[43975]: 1.e scrub starts 2024-02-09T18:08:08.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:08:08 smithi179 ceph-mon[43975]: 1.e scrub ok 2024-02-09T18:08:08.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:08:08 smithi179 ceph-mon[43975]: pgmap v2589: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:08:08.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:08:08 smithi179 ceph-mon[43975]: from='client.17664 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T18:08:08.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:08:08 smithi179 ceph-mon[43975]: purged_snaps scrub starts 2024-02-09T18:08:08.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:08:08 smithi179 ceph-mon[43975]: purged_snaps scrub ok 2024-02-09T18:08:08.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:08:08 smithi162 ceph-mon[44304]: 1.e scrub starts 2024-02-09T18:08:08.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:08:08 smithi162 ceph-mon[44304]: 1.e scrub ok 2024-02-09T18:08:08.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:08:08 smithi162 ceph-mon[44304]: pgmap v2589: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:08:08.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:08:08 smithi162 ceph-mon[44304]: from='client.17664 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T18:08:08.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:08:08 smithi162 ceph-mon[44304]: purged_snaps scrub starts 2024-02-09T18:08:08.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:08:08 smithi162 ceph-mon[44304]: purged_snaps scrub ok 2024-02-09T18:08:08.735 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/mon.a/config 2024-02-09T18:08:09.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:08:09 smithi099 ceph-mon[39420]: osdmap e1049: 12 total, 12 up, 12 in 2024-02-09T18:08:09.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:08:09 smithi179 ceph-mon[43975]: osdmap e1049: 12 total, 12 up, 12 in 2024-02-09T18:08:09.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:08:09 smithi162 ceph-mon[44304]: osdmap e1049: 12 total, 12 up, 12 in 2024-02-09T18:08:10.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:08:10 smithi099 ceph-mon[39420]: pgmap v2591: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:08:10.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:08:10 smithi179 ceph-mon[43975]: pgmap v2591: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:08:10.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:08:10 smithi162 ceph-mon[44304]: pgmap v2591: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:08:11.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:08:11 smithi099 ceph-mon[39420]: 1.3 scrub starts 2024-02-09T18:08:11.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:08:11 smithi099 ceph-mon[39420]: 1.3 scrub ok 2024-02-09T18:08:11.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:08:11 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2219497766' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-02-09T18:08:11.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:08:11 smithi099 ceph-mon[39420]: from='client.? 172.21.15.99:0/2219497766' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-02-09T18:08:11.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:08:11 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T18:08:11.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:08:11 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T18:08:11.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:08:11 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T18:08:11.494 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:08:11 smithi099 ceph-mon[39420]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T18:08:11.661 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-09T18:08:11.671 INFO:tasks.cephadm:Teardown begin 2024-02-09T18:08:11.671 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-09T18:08:11.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:08:11 smithi179 ceph-mon[43975]: 1.3 scrub starts 2024-02-09T18:08:11.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:08:11 smithi179 ceph-mon[43975]: 1.3 scrub ok 2024-02-09T18:08:11.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:08:11 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2219497766' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-02-09T18:08:11.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:08:11 smithi179 ceph-mon[43975]: from='client.? 172.21.15.99:0/2219497766' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-02-09T18:08:11.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:08:11 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T18:08:11.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:08:11 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T18:08:11.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:08:11 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T18:08:11.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:08:11 smithi179 ceph-mon[43975]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T18:08:11.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:08:11 smithi162 ceph-mon[44304]: 1.3 scrub starts 2024-02-09T18:08:11.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:08:11 smithi162 ceph-mon[44304]: 1.3 scrub ok 2024-02-09T18:08:11.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:08:11 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2219497766' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-02-09T18:08:11.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:08:11 smithi162 ceph-mon[44304]: from='client.? 172.21.15.99:0/2219497766' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-02-09T18:08:11.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:08:11 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T18:08:11.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:08:11 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T18:08:11.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:08:11 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T18:08:11.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:08:11 smithi162 ceph-mon[44304]: from='mgr.14148 172.21.15.99:0/1635220709' entity='mgr.y' 2024-02-09T18:08:11.749 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-09T18:08:11.805 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-09T18:08:11.858 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-09T18:08:11.859 DEBUG:teuthology.orchestra.run.smithi099:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-09T18:08:11.895 DEBUG:teuthology.orchestra.run.smithi162:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-09T18:08:11.916 DEBUG:teuthology.orchestra.run.smithi179:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-09T18:08:11.940 INFO:tasks.cephadm:Stopping all daemons... 2024-02-09T18:08:11.940 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-02-09T18:08:11.940 DEBUG:teuthology.orchestra.run.smithi099:> sudo systemctl stop ceph-a69c870a-c76e-11ee-95b7-87774f69a715@mon.a 2024-02-09T18:08:12.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:08:11 smithi099 systemd[1]: Stopping Ceph mon.a for a69c870a-c76e-11ee-95b7-87774f69a715... 2024-02-09T18:08:12.244 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:08:12 smithi099 ceph-mon[39420]: pgmap v2592: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:08:12.670 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:08:12 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mon-a[39397]: 2024-02-09T18:08:12.253+0000 7feb5cd9b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T18:08:12.671 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:08:12 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mon-a[39397]: 2024-02-09T18:08:12.253+0000 7feb5cd9b700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-02-09T18:08:12.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:08:12 smithi179 ceph-mon[43975]: pgmap v2592: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:08:12.676 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:08:12 smithi162 ceph-mon[44304]: pgmap v2592: 31 pgs: 31 active+clean; 577 KiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T18:08:12.994 INFO:journalctl@ceph.mon.a.smithi099.stdout:Feb 09 18:08:12 smithi099 bash[325584]: ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mon-a 2024-02-09T18:08:13.248 DEBUG:teuthology.orchestra.run.smithi099:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a69c870a-c76e-11ee-95b7-87774f69a715@mon.a.service' 2024-02-09T18:08:13.287 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T18:08:13.287 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-02-09T18:08:13.287 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-02-09T18:08:13.288 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-a69c870a-c76e-11ee-95b7-87774f69a715@mon.b 2024-02-09T18:08:13.639 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:08:13 smithi162 systemd[1]: Stopping Ceph mon.b for a69c870a-c76e-11ee-95b7-87774f69a715... 2024-02-09T18:08:13.926 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:08:13 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mon-b[44281]: 2024-02-09T18:08:13.635+0000 7f3b2e54c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T18:08:13.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:08:13 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mon-b[44281]: 2024-02-09T18:08:13.635+0000 7f3b2e54c700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-02-09T18:08:14.426 INFO:journalctl@ceph.mon.b.smithi162.stdout:Feb 09 18:08:14 smithi162 bash[174994]: ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mon-b 2024-02-09T18:08:14.710 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a69c870a-c76e-11ee-95b7-87774f69a715@mon.b.service' 2024-02-09T18:08:14.758 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T18:08:14.758 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-02-09T18:08:14.758 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-02-09T18:08:14.758 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-a69c870a-c76e-11ee-95b7-87774f69a715@mon.c 2024-02-09T18:08:15.067 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:08:14 smithi179 systemd[1]: Stopping Ceph mon.c for a69c870a-c76e-11ee-95b7-87774f69a715... 2024-02-09T18:08:15.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:08:15 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mon-c[43952]: 2024-02-09T18:08:15.063+0000 7f53881e4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T18:08:15.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:08:15 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mon-c[43952]: 2024-02-09T18:08:15.063+0000 7f53881e4700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-02-09T18:08:15.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Feb 09 18:08:15 smithi179 bash[174262]: ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mon-c 2024-02-09T18:08:16.081 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a69c870a-c76e-11ee-95b7-87774f69a715@mon.c.service' 2024-02-09T18:08:16.138 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T18:08:16.139 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-02-09T18:08:16.139 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-02-09T18:08:16.139 DEBUG:teuthology.orchestra.run.smithi099:> sudo systemctl stop ceph-a69c870a-c76e-11ee-95b7-87774f69a715@mgr.y 2024-02-09T18:08:16.452 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 18:08:16 smithi099 systemd[1]: Stopping Ceph mgr.y for a69c870a-c76e-11ee-95b7-87774f69a715... 2024-02-09T18:08:17.244 INFO:journalctl@ceph.mgr.y.smithi099.stdout:Feb 09 18:08:16 smithi099 bash[325729]: ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-y 2024-02-09T18:08:17.261 DEBUG:teuthology.orchestra.run.smithi099:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a69c870a-c76e-11ee-95b7-87774f69a715@mgr.y.service' 2024-02-09T18:08:17.296 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T18:08:17.297 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-02-09T18:08:17.297 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-02-09T18:08:17.297 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-a69c870a-c76e-11ee-95b7-87774f69a715@mgr.x 2024-02-09T18:08:17.677 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Feb 09 18:08:17 smithi162 systemd[1]: Stopping Ceph mgr.x for a69c870a-c76e-11ee-95b7-87774f69a715... 2024-02-09T18:08:18.177 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Feb 09 18:08:18 smithi162 bash[175152]: ceph-a69c870a-c76e-11ee-95b7-87774f69a715-mgr-x 2024-02-09T18:08:18.526 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Feb 09 18:08:18 smithi162 systemd[1]: ceph-a69c870a-c76e-11ee-95b7-87774f69a715@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-02-09T18:08:18.526 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Feb 09 18:08:18 smithi162 systemd[1]: ceph-a69c870a-c76e-11ee-95b7-87774f69a715@mgr.x.service: Failed with result 'exit-code'. 2024-02-09T18:08:18.528 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a69c870a-c76e-11ee-95b7-87774f69a715@mgr.x.service' 2024-02-09T18:08:18.563 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T18:08:18.563 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-02-09T18:08:18.564 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-02-09T18:08:18.564 DEBUG:teuthology.orchestra.run.smithi099:> sudo systemctl stop ceph-a69c870a-c76e-11ee-95b7-87774f69a715@osd.0 2024-02-09T18:08:18.994 INFO:journalctl@ceph.osd.0.smithi099.stdout:Feb 09 18:08:18 smithi099 systemd[1]: Stopping Ceph osd.0 for a69c870a-c76e-11ee-95b7-87774f69a715... 2024-02-09T18:08:18.994 INFO:journalctl@ceph.osd.0.smithi099.stdout:Feb 09 18:08:18 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-0[58275]: 2024-02-09T18:08:18.821+0000 7f0dcf73e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T18:08:18.994 INFO:journalctl@ceph.osd.0.smithi099.stdout:Feb 09 18:08:18 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-0[58275]: 2024-02-09T18:08:18.821+0000 7f0dcf73e700 -1 osd.0 1050 *** Got signal Terminated *** 2024-02-09T18:08:18.994 INFO:journalctl@ceph.osd.0.smithi099.stdout:Feb 09 18:08:18 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-0[58275]: 2024-02-09T18:08:18.821+0000 7f0dcf73e700 -1 osd.0 1050 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-09T18:08:24.994 INFO:journalctl@ceph.osd.0.smithi099.stdout:Feb 09 18:08:24 smithi099 bash[325875]: ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-0 2024-02-09T18:08:26.933 DEBUG:teuthology.orchestra.run.smithi099:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a69c870a-c76e-11ee-95b7-87774f69a715@osd.0.service' 2024-02-09T18:08:26.973 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T18:08:26.973 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-02-09T18:08:26.973 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-02-09T18:08:26.974 DEBUG:teuthology.orchestra.run.smithi099:> sudo systemctl stop ceph-a69c870a-c76e-11ee-95b7-87774f69a715@osd.1 2024-02-09T18:08:27.244 INFO:journalctl@ceph.osd.1.smithi099.stdout:Feb 09 18:08:27 smithi099 systemd[1]: Stopping Ceph osd.1 for a69c870a-c76e-11ee-95b7-87774f69a715... 2024-02-09T18:08:27.743 INFO:journalctl@ceph.osd.1.smithi099.stdout:Feb 09 18:08:27 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-1[65435]: 2024-02-09T18:08:27.282+0000 7fdf64910700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T18:08:27.744 INFO:journalctl@ceph.osd.1.smithi099.stdout:Feb 09 18:08:27 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-1[65435]: 2024-02-09T18:08:27.282+0000 7fdf64910700 -1 osd.1 1050 *** Got signal Terminated *** 2024-02-09T18:08:27.744 INFO:journalctl@ceph.osd.1.smithi099.stdout:Feb 09 18:08:27 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-1[65435]: 2024-02-09T18:08:27.282+0000 7fdf64910700 -1 osd.1 1050 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-09T18:08:33.494 INFO:journalctl@ceph.osd.1.smithi099.stdout:Feb 09 18:08:33 smithi099 bash[326147]: ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-1 2024-02-09T18:08:35.227 DEBUG:teuthology.orchestra.run.smithi099:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a69c870a-c76e-11ee-95b7-87774f69a715@osd.1.service' 2024-02-09T18:08:35.267 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T18:08:35.267 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-02-09T18:08:35.267 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-02-09T18:08:35.267 DEBUG:teuthology.orchestra.run.smithi099:> sudo systemctl stop ceph-a69c870a-c76e-11ee-95b7-87774f69a715@osd.2 2024-02-09T18:08:35.494 INFO:journalctl@ceph.osd.2.smithi099.stdout:Feb 09 18:08:35 smithi099 systemd[1]: Stopping Ceph osd.2 for a69c870a-c76e-11ee-95b7-87774f69a715... 2024-02-09T18:08:35.994 INFO:journalctl@ceph.osd.2.smithi099.stdout:Feb 09 18:08:35 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-2[73177]: 2024-02-09T18:08:35.598+0000 7f3830a5e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T18:08:35.994 INFO:journalctl@ceph.osd.2.smithi099.stdout:Feb 09 18:08:35 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-2[73177]: 2024-02-09T18:08:35.598+0000 7f3830a5e700 -1 osd.2 1050 *** Got signal Terminated *** 2024-02-09T18:08:35.994 INFO:journalctl@ceph.osd.2.smithi099.stdout:Feb 09 18:08:35 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-2[73177]: 2024-02-09T18:08:35.598+0000 7f3830a5e700 -1 osd.2 1050 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-09T18:08:41.744 INFO:journalctl@ceph.osd.2.smithi099.stdout:Feb 09 18:08:41 smithi099 bash[326411]: ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-2 2024-02-09T18:08:43.460 DEBUG:teuthology.orchestra.run.smithi099:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a69c870a-c76e-11ee-95b7-87774f69a715@osd.2.service' 2024-02-09T18:08:43.500 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T18:08:43.501 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-02-09T18:08:43.501 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-02-09T18:08:43.501 DEBUG:teuthology.orchestra.run.smithi099:> sudo systemctl stop ceph-a69c870a-c76e-11ee-95b7-87774f69a715@osd.3 2024-02-09T18:08:43.744 INFO:journalctl@ceph.osd.3.smithi099.stdout:Feb 09 18:08:43 smithi099 systemd[1]: Stopping Ceph osd.3 for a69c870a-c76e-11ee-95b7-87774f69a715... 2024-02-09T18:08:44.244 INFO:journalctl@ceph.osd.3.smithi099.stdout:Feb 09 18:08:43 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-3[280202]: 2024-02-09T18:08:43.807+0000 7fb8fa2c9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T18:08:44.244 INFO:journalctl@ceph.osd.3.smithi099.stdout:Feb 09 18:08:43 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-3[280202]: 2024-02-09T18:08:43.807+0000 7fb8fa2c9700 -1 osd.3 1050 *** Got signal Terminated *** 2024-02-09T18:08:44.244 INFO:journalctl@ceph.osd.3.smithi099.stdout:Feb 09 18:08:43 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-3[280202]: 2024-02-09T18:08:43.807+0000 7fb8fa2c9700 -1 osd.3 1050 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-09T18:08:45.172 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:08:44 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:08:44.919+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:08:45.177 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:08:44 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:08:44.867+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.294439+0000 front 2024-02-09T18:08:23.294609+0000 (oldest deadline 2024-02-09T18:08:44.394323+0000) 2024-02-09T18:08:46.177 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:08:45 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:08:45.837+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.294439+0000 front 2024-02-09T18:08:23.294609+0000 (oldest deadline 2024-02-09T18:08:44.394323+0000) 2024-02-09T18:08:46.248 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:08:45 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:08:45.947+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:08:46.672 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:08:46 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:08:46.247+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:20.919791+0000 front 2024-02-09T18:08:20.919903+0000 (oldest deadline 2024-02-09T18:08:46.219692+0000) 2024-02-09T18:08:46.672 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:08:46 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:08:46.439+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:08:47.176 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:08:46 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:08:46.788+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.294439+0000 front 2024-02-09T18:08:23.294609+0000 (oldest deadline 2024-02-09T18:08:44.394323+0000) 2024-02-09T18:08:47.177 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:08:46 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:08:46.961+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.082620+0000 front 2024-02-09T18:08:23.082482+0000 (oldest deadline 2024-02-09T18:08:46.582507+0000) 2024-02-09T18:08:47.221 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:08:46 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:08:46.941+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:08:47.472 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:08:47 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:08:47.221+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:20.919791+0000 front 2024-02-09T18:08:20.919903+0000 (oldest deadline 2024-02-09T18:08:46.219692+0000) 2024-02-09T18:08:47.472 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:08:47 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:08:47.220+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:08:47.494 INFO:journalctl@ceph.osd.3.smithi099.stdout:Feb 09 18:08:47 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-3[280202]: 2024-02-09T18:08:47.081+0000 7fb8f68c4700 -1 osd.3 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.930735+0000 front 2024-02-09T18:08:22.930865+0000 (oldest deadline 2024-02-09T18:08:46.430944+0000) 2024-02-09T18:08:47.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:08:47 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:08:47.471+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:08:48.176 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:08:47 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:08:47.833+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.294439+0000 front 2024-02-09T18:08:23.294609+0000 (oldest deadline 2024-02-09T18:08:44.394323+0000) 2024-02-09T18:08:48.177 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:08:47 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:08:47.978+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.082620+0000 front 2024-02-09T18:08:23.082482+0000 (oldest deadline 2024-02-09T18:08:46.582507+0000) 2024-02-09T18:08:48.243 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:08:47 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:08:47.989+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:08:48.243 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:08:48 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:08:48.208+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:08:48.493 INFO:journalctl@ceph.osd.3.smithi099.stdout:Feb 09 18:08:48 smithi099 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-3[280202]: 2024-02-09T18:08:48.120+0000 7fb8f68c4700 -1 osd.3 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.930735+0000 front 2024-02-09T18:08:22.930865+0000 (oldest deadline 2024-02-09T18:08:46.430944+0000) 2024-02-09T18:08:48.511 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:08:48 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:08:48.241+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:20.919791+0000 front 2024-02-09T18:08:20.919903+0000 (oldest deadline 2024-02-09T18:08:46.219692+0000) 2024-02-09T18:08:48.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:08:48 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:08:48.510+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:08:49.176 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:08:48 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:08:48.871+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.294439+0000 front 2024-02-09T18:08:23.294609+0000 (oldest deadline 2024-02-09T18:08:44.394323+0000) 2024-02-09T18:08:49.177 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:08:48 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:08:48.977+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.082620+0000 front 2024-02-09T18:08:23.082482+0000 (oldest deadline 2024-02-09T18:08:46.582507+0000) 2024-02-09T18:08:49.281 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:08:49 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:08:49.233+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:08:49.281 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:08:49 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:08:49.002+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:08:49.671 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:08:49 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:08:49.471+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:08:49.672 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:08:49 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:08:49.280+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:20.919791+0000 front 2024-02-09T18:08:20.919903+0000 (oldest deadline 2024-02-09T18:08:46.219692+0000) 2024-02-09T18:08:49.979 INFO:journalctl@ceph.osd.3.smithi099.stdout:Feb 09 18:08:49 smithi099 bash[326674]: ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-3 2024-02-09T18:08:50.177 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:08:49 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:08:49.903+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.294439+0000 front 2024-02-09T18:08:23.294609+0000 (oldest deadline 2024-02-09T18:08:44.394323+0000) 2024-02-09T18:08:50.177 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:08:49 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:08:49.998+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.082620+0000 front 2024-02-09T18:08:23.082482+0000 (oldest deadline 2024-02-09T18:08:46.582507+0000) 2024-02-09T18:08:50.240 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:08:49 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:08:49.981+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:08:50.496 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:08:50 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:08:50.238+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:20.919791+0000 front 2024-02-09T18:08:20.919903+0000 (oldest deadline 2024-02-09T18:08:46.219692+0000) 2024-02-09T18:08:50.497 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:08:50 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:08:50.256+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:08:50.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:08:50 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:08:50.495+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:08:51.177 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:08:50 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:08:50.952+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.082620+0000 front 2024-02-09T18:08:23.082482+0000 (oldest deadline 2024-02-09T18:08:46.582507+0000) 2024-02-09T18:08:51.177 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:08:50 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:08:50.865+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.294439+0000 front 2024-02-09T18:08:23.294609+0000 (oldest deadline 2024-02-09T18:08:44.394323+0000) 2024-02-09T18:08:51.228 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:08:50 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:08:50.953+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:08:51.539 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:08:51 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:08:51.269+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:20.919791+0000 front 2024-02-09T18:08:20.919903+0000 (oldest deadline 2024-02-09T18:08:46.219692+0000) 2024-02-09T18:08:51.539 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:08:51 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:08:51.226+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:08:51.552 DEBUG:teuthology.orchestra.run.smithi099:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a69c870a-c76e-11ee-95b7-87774f69a715@osd.3.service' 2024-02-09T18:08:51.591 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T18:08:51.591 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-02-09T18:08:51.591 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-02-09T18:08:51.591 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-a69c870a-c76e-11ee-95b7-87774f69a715@osd.4 2024-02-09T18:08:51.895 INFO:journalctl@ceph.osd.4.smithi162.stdout:Feb 09 18:08:51 smithi162 systemd[1]: Stopping Ceph osd.4 for a69c870a-c76e-11ee-95b7-87774f69a715... 2024-02-09T18:08:51.923 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:08:51 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:08:51.537+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:08:52.176 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:08:51 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:08:51.906+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.294439+0000 front 2024-02-09T18:08:23.294609+0000 (oldest deadline 2024-02-09T18:08:44.394323+0000) 2024-02-09T18:08:52.177 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:08:51 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:08:51.987+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.082620+0000 front 2024-02-09T18:08:23.082482+0000 (oldest deadline 2024-02-09T18:08:46.582507+0000) 2024-02-09T18:08:52.177 INFO:journalctl@ceph.osd.4.smithi162.stdout:Feb 09 18:08:51 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-4[49654]: 2024-02-09T18:08:51.893+0000 7f1a23e0a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T18:08:52.177 INFO:journalctl@ceph.osd.4.smithi162.stdout:Feb 09 18:08:51 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-4[49654]: 2024-02-09T18:08:51.893+0000 7f1a23e0a700 -1 osd.4 1050 *** Got signal Terminated *** 2024-02-09T18:08:52.177 INFO:journalctl@ceph.osd.4.smithi162.stdout:Feb 09 18:08:51 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-4[49654]: 2024-02-09T18:08:51.893+0000 7f1a23e0a700 -1 osd.4 1050 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-09T18:08:52.422 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:08:52 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:08:52.233+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:20.919791+0000 front 2024-02-09T18:08:20.919903+0000 (oldest deadline 2024-02-09T18:08:46.219692+0000) 2024-02-09T18:08:52.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:08:52 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:08:52.181+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:08:52.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:08:51 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:08:51.983+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:08:52.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:08:52 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:08:52.535+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:08:53.176 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:08:52 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:08:52.889+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.294439+0000 front 2024-02-09T18:08:23.294609+0000 (oldest deadline 2024-02-09T18:08:44.394323+0000) 2024-02-09T18:08:53.177 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:08:53 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:08:53.028+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.082620+0000 front 2024-02-09T18:08:23.082482+0000 (oldest deadline 2024-02-09T18:08:46.582507+0000) 2024-02-09T18:08:53.422 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:08:53 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:08:53.185+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:20.919791+0000 front 2024-02-09T18:08:20.919903+0000 (oldest deadline 2024-02-09T18:08:46.219692+0000) 2024-02-09T18:08:53.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:08:53 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:08:53.174+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:08:53.423 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:08:53 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:08:53.011+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:08:53.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:08:53 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:08:53.502+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:08:54.177 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:08:53 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:08:53.902+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.294439+0000 front 2024-02-09T18:08:23.294609+0000 (oldest deadline 2024-02-09T18:08:44.394323+0000) 2024-02-09T18:08:54.177 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:08:53 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:08:53.902+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.195929+0000 front 2024-02-09T18:08:30.195975+0000 (oldest deadline 2024-02-09T18:08:53.695778+0000) 2024-02-09T18:08:54.177 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:08:54 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:08:54.078+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.082620+0000 front 2024-02-09T18:08:23.082482+0000 (oldest deadline 2024-02-09T18:08:46.582507+0000) 2024-02-09T18:08:54.422 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:08:54 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:08:54.202+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:20.919791+0000 front 2024-02-09T18:08:20.919903+0000 (oldest deadline 2024-02-09T18:08:46.219692+0000) 2024-02-09T18:08:54.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:08:54 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:08:54.161+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:08:54.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:08:53 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:08:53.981+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:08:54.677 INFO:journalctl@ceph.osd.4.smithi162.stdout:Feb 09 18:08:54 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-4[49654]: 2024-02-09T18:08:54.246+0000 7f1a20405700 -1 osd.4 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:28.119828+0000 front 2024-02-09T18:08:28.119799+0000 (oldest deadline 2024-02-09T18:08:54.019765+0000) 2024-02-09T18:08:54.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:08:54 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:08:54.517+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:08:55.211 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:08:54 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:08:54.945+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.294439+0000 front 2024-02-09T18:08:23.294609+0000 (oldest deadline 2024-02-09T18:08:44.394323+0000) 2024-02-09T18:08:55.212 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:08:54 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:08:54.945+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.195929+0000 front 2024-02-09T18:08:30.195975+0000 (oldest deadline 2024-02-09T18:08:53.695778+0000) 2024-02-09T18:08:55.212 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:08:55 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:08:55.063+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.082620+0000 front 2024-02-09T18:08:23.082482+0000 (oldest deadline 2024-02-09T18:08:46.582507+0000) 2024-02-09T18:08:55.212 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:08:55 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:08:55.063+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.883837+0000 front 2024-02-09T18:08:31.883818+0000 (oldest deadline 2024-02-09T18:08:54.783844+0000) 2024-02-09T18:08:55.234 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:08:54 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:08:54.966+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:08:55.235 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:08:55 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:08:55.135+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:08:55.508 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:08:55 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:08:55.232+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:20.919791+0000 front 2024-02-09T18:08:20.919903+0000 (oldest deadline 2024-02-09T18:08:46.219692+0000) 2024-02-09T18:08:55.676 INFO:journalctl@ceph.osd.4.smithi162.stdout:Feb 09 18:08:55 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-4[49654]: 2024-02-09T18:08:55.210+0000 7f1a20405700 -1 osd.4 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:28.119828+0000 front 2024-02-09T18:08:28.119799+0000 (oldest deadline 2024-02-09T18:08:54.019765+0000) 2024-02-09T18:08:55.677 INFO:journalctl@ceph.osd.5.smithi162.stdout:Feb 09 18:08:55 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-5[56859]: 2024-02-09T18:08:55.371+0000 7f8e2122c700 -1 osd.5 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:29.061449+0000 front 2024-02-09T18:08:29.061391+0000 (oldest deadline 2024-02-09T18:08:54.361172+0000) 2024-02-09T18:08:55.921 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:08:55 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:08:55.506+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:08:56.177 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:08:55 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:08:55.911+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.294439+0000 front 2024-02-09T18:08:23.294609+0000 (oldest deadline 2024-02-09T18:08:44.394323+0000) 2024-02-09T18:08:56.177 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:08:55 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:08:55.911+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.195929+0000 front 2024-02-09T18:08:30.195975+0000 (oldest deadline 2024-02-09T18:08:53.695778+0000) 2024-02-09T18:08:56.177 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:08:56 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:08:56.109+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.082620+0000 front 2024-02-09T18:08:23.082482+0000 (oldest deadline 2024-02-09T18:08:46.582507+0000) 2024-02-09T18:08:56.177 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:08:56 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:08:56.109+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.883837+0000 front 2024-02-09T18:08:31.883818+0000 (oldest deadline 2024-02-09T18:08:54.783844+0000) 2024-02-09T18:08:56.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:08:55 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:08:55.984+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:08:56.422 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:08:56 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:08:56.219+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:20.919791+0000 front 2024-02-09T18:08:20.919903+0000 (oldest deadline 2024-02-09T18:08:46.219692+0000) 2024-02-09T18:08:56.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:08:56 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:08:56.149+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:08:56.676 INFO:journalctl@ceph.osd.4.smithi162.stdout:Feb 09 18:08:56 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-4[49654]: 2024-02-09T18:08:56.180+0000 7f1a20405700 -1 osd.4 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:28.119828+0000 front 2024-02-09T18:08:28.119799+0000 (oldest deadline 2024-02-09T18:08:54.019765+0000) 2024-02-09T18:08:56.677 INFO:journalctl@ceph.osd.5.smithi162.stdout:Feb 09 18:08:56 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-5[56859]: 2024-02-09T18:08:56.330+0000 7f8e2122c700 -1 osd.5 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:29.061449+0000 front 2024-02-09T18:08:29.061391+0000 (oldest deadline 2024-02-09T18:08:54.361172+0000) 2024-02-09T18:08:56.921 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:08:56 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:08:56.492+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:08:57.155 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:08:56 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:08:56.875+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.294439+0000 front 2024-02-09T18:08:23.294609+0000 (oldest deadline 2024-02-09T18:08:44.394323+0000) 2024-02-09T18:08:57.155 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:08:56 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:08:56.875+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.195929+0000 front 2024-02-09T18:08:30.195975+0000 (oldest deadline 2024-02-09T18:08:53.695778+0000) 2024-02-09T18:08:57.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:08:57 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:08:57.013+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:08:57.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:08:57 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:08:57.125+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:08:57.422 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:08:57 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:08:57.202+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:20.919791+0000 front 2024-02-09T18:08:20.919903+0000 (oldest deadline 2024-02-09T18:08:46.219692+0000) 2024-02-09T18:08:57.422 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:08:57 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:08:57.202+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.520735+0000 front 2024-02-09T18:08:31.520846+0000 (oldest deadline 2024-02-09T18:08:56.820686+0000) 2024-02-09T18:08:57.426 INFO:journalctl@ceph.osd.5.smithi162.stdout:Feb 09 18:08:57 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-5[56859]: 2024-02-09T18:08:57.328+0000 7f8e2122c700 -1 osd.5 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:29.061449+0000 front 2024-02-09T18:08:29.061391+0000 (oldest deadline 2024-02-09T18:08:54.361172+0000) 2024-02-09T18:08:57.427 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:08:57 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:08:57.153+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.082620+0000 front 2024-02-09T18:08:23.082482+0000 (oldest deadline 2024-02-09T18:08:46.582507+0000) 2024-02-09T18:08:57.427 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:08:57 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:08:57.153+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.883837+0000 front 2024-02-09T18:08:31.883818+0000 (oldest deadline 2024-02-09T18:08:54.783844+0000) 2024-02-09T18:08:57.921 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:08:57 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:08:57.505+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:08:57.926 INFO:journalctl@ceph.osd.4.smithi162.stdout:Feb 09 18:08:57 smithi162 bash[175313]: ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-4 2024-02-09T18:08:57.927 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:08:57 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:08:57.903+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.294439+0000 front 2024-02-09T18:08:23.294609+0000 (oldest deadline 2024-02-09T18:08:44.394323+0000) 2024-02-09T18:08:57.927 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:08:57 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:08:57.903+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.195929+0000 front 2024-02-09T18:08:30.195975+0000 (oldest deadline 2024-02-09T18:08:53.695778+0000) 2024-02-09T18:08:58.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:08:57 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:08:57.970+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:08:58.422 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:08:58 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:08:58.189+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:20.919791+0000 front 2024-02-09T18:08:20.919903+0000 (oldest deadline 2024-02-09T18:08:46.219692+0000) 2024-02-09T18:08:58.422 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:08:58 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:08:58.189+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.520735+0000 front 2024-02-09T18:08:31.520846+0000 (oldest deadline 2024-02-09T18:08:56.820686+0000) 2024-02-09T18:08:58.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:08:58 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:08:58.089+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:08:58.423 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:08:58 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:08:58.089+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:27.021731+0000 front 2024-02-09T18:08:27.021779+0000 (oldest deadline 2024-02-09T18:08:57.523295+0000) 2024-02-09T18:08:58.557 INFO:journalctl@ceph.osd.5.smithi162.stdout:Feb 09 18:08:58 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-5[56859]: 2024-02-09T18:08:58.325+0000 7f8e2122c700 -1 osd.5 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:29.061449+0000 front 2024-02-09T18:08:29.061391+0000 (oldest deadline 2024-02-09T18:08:54.361172+0000) 2024-02-09T18:08:58.558 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:08:58 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:08:58.195+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.082620+0000 front 2024-02-09T18:08:23.082482+0000 (oldest deadline 2024-02-09T18:08:46.582507+0000) 2024-02-09T18:08:58.558 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:08:58 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:08:58.195+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.883837+0000 front 2024-02-09T18:08:31.883818+0000 (oldest deadline 2024-02-09T18:08:54.783844+0000) 2024-02-09T18:08:58.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:08:58 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:08:58.555+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:08:58.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:08:58 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:08:58.555+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:08:59.176 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:08:58 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:08:58.900+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.294439+0000 front 2024-02-09T18:08:23.294609+0000 (oldest deadline 2024-02-09T18:08:44.394323+0000) 2024-02-09T18:08:59.177 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:08:58 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:08:58.900+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.195929+0000 front 2024-02-09T18:08:30.195975+0000 (oldest deadline 2024-02-09T18:08:53.695778+0000) 2024-02-09T18:08:59.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:08:58 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:08:58.989+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:08:59.422 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:08:59 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:08:59.238+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:20.919791+0000 front 2024-02-09T18:08:20.919903+0000 (oldest deadline 2024-02-09T18:08:46.219692+0000) 2024-02-09T18:08:59.422 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:08:59 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:08:59.238+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.520735+0000 front 2024-02-09T18:08:31.520846+0000 (oldest deadline 2024-02-09T18:08:56.820686+0000) 2024-02-09T18:08:59.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:08:59 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:08:59.131+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:08:59.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:08:59 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:08:59.131+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:27.021731+0000 front 2024-02-09T18:08:27.021779+0000 (oldest deadline 2024-02-09T18:08:57.523295+0000) 2024-02-09T18:08:59.638 INFO:journalctl@ceph.osd.5.smithi162.stdout:Feb 09 18:08:59 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-5[56859]: 2024-02-09T18:08:59.362+0000 7f8e2122c700 -1 osd.5 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:29.061449+0000 front 2024-02-09T18:08:29.061391+0000 (oldest deadline 2024-02-09T18:08:54.361172+0000) 2024-02-09T18:08:59.639 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:08:59 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:08:59.226+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.082620+0000 front 2024-02-09T18:08:23.082482+0000 (oldest deadline 2024-02-09T18:08:46.582507+0000) 2024-02-09T18:08:59.639 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:08:59 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:08:59.226+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.883837+0000 front 2024-02-09T18:08:31.883818+0000 (oldest deadline 2024-02-09T18:08:54.783844+0000) 2024-02-09T18:08:59.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:08:59 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:08:59.570+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:08:59.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:08:59 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:08:59.570+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:08:59.974 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a69c870a-c76e-11ee-95b7-87774f69a715@osd.4.service' 2024-02-09T18:09:00.015 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T18:09:00.015 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-02-09T18:09:00.016 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-02-09T18:09:00.016 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-a69c870a-c76e-11ee-95b7-87774f69a715@osd.5 2024-02-09T18:09:00.176 INFO:journalctl@ceph.osd.5.smithi162.stdout:Feb 09 18:09:00 smithi162 systemd[1]: Stopping Ceph osd.5 for a69c870a-c76e-11ee-95b7-87774f69a715... 2024-02-09T18:09:00.177 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:08:59 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:08:59.905+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.294439+0000 front 2024-02-09T18:08:23.294609+0000 (oldest deadline 2024-02-09T18:08:44.394323+0000) 2024-02-09T18:09:00.177 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:08:59 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:08:59.905+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.195929+0000 front 2024-02-09T18:08:30.195975+0000 (oldest deadline 2024-02-09T18:08:53.695778+0000) 2024-02-09T18:09:00.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:08:59 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:08:59.998+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:09:00.422 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:00 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:00.196+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:20.919791+0000 front 2024-02-09T18:08:20.919903+0000 (oldest deadline 2024-02-09T18:08:46.219692+0000) 2024-02-09T18:09:00.422 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:00 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:00.196+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.520735+0000 front 2024-02-09T18:08:31.520846+0000 (oldest deadline 2024-02-09T18:08:56.820686+0000) 2024-02-09T18:09:00.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:00 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:00.135+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:09:00.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:00 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:00.135+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:27.021731+0000 front 2024-02-09T18:08:27.021779+0000 (oldest deadline 2024-02-09T18:08:57.523295+0000) 2024-02-09T18:09:00.677 INFO:journalctl@ceph.osd.5.smithi162.stdout:Feb 09 18:09:00 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-5[56859]: 2024-02-09T18:09:00.337+0000 7f8e24c31700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T18:09:00.677 INFO:journalctl@ceph.osd.5.smithi162.stdout:Feb 09 18:09:00 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-5[56859]: 2024-02-09T18:09:00.337+0000 7f8e24c31700 -1 osd.5 1050 *** Got signal Terminated *** 2024-02-09T18:09:00.677 INFO:journalctl@ceph.osd.5.smithi162.stdout:Feb 09 18:09:00 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-5[56859]: 2024-02-09T18:09:00.337+0000 7f8e24c31700 -1 osd.5 1050 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-09T18:09:00.677 INFO:journalctl@ceph.osd.5.smithi162.stdout:Feb 09 18:09:00 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-5[56859]: 2024-02-09T18:09:00.368+0000 7f8e2122c700 -1 osd.5 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:29.061449+0000 front 2024-02-09T18:08:29.061391+0000 (oldest deadline 2024-02-09T18:08:54.361172+0000) 2024-02-09T18:09:00.677 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:00 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:00.221+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.082620+0000 front 2024-02-09T18:08:23.082482+0000 (oldest deadline 2024-02-09T18:08:46.582507+0000) 2024-02-09T18:09:00.677 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:00 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:00.221+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.883837+0000 front 2024-02-09T18:08:31.883818+0000 (oldest deadline 2024-02-09T18:08:54.783844+0000) 2024-02-09T18:09:00.921 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:00 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:00.523+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:00.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:00 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:00.523+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:01.221 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:09:00 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:09:00.952+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.294439+0000 front 2024-02-09T18:08:23.294609+0000 (oldest deadline 2024-02-09T18:08:44.394323+0000) 2024-02-09T18:09:01.222 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:09:00 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:09:00.952+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.195929+0000 front 2024-02-09T18:08:30.195975+0000 (oldest deadline 2024-02-09T18:08:53.695778+0000) 2024-02-09T18:09:01.421 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:00 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:00.978+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:09:01.422 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:01 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:01.181+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:20.919791+0000 front 2024-02-09T18:08:20.919903+0000 (oldest deadline 2024-02-09T18:08:46.219692+0000) 2024-02-09T18:09:01.422 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:01 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:01.181+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.520735+0000 front 2024-02-09T18:08:31.520846+0000 (oldest deadline 2024-02-09T18:08:56.820686+0000) 2024-02-09T18:09:01.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:01 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:01.147+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:09:01.423 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:01 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:01.147+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:27.021731+0000 front 2024-02-09T18:08:27.021779+0000 (oldest deadline 2024-02-09T18:08:57.523295+0000) 2024-02-09T18:09:01.676 INFO:journalctl@ceph.osd.5.smithi162.stdout:Feb 09 18:09:01 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-5[56859]: 2024-02-09T18:09:01.325+0000 7f8e2122c700 -1 osd.5 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:29.061449+0000 front 2024-02-09T18:08:29.061391+0000 (oldest deadline 2024-02-09T18:08:54.361172+0000) 2024-02-09T18:09:01.677 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:01 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:01.220+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.082620+0000 front 2024-02-09T18:08:23.082482+0000 (oldest deadline 2024-02-09T18:08:46.582507+0000) 2024-02-09T18:09:01.677 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:01 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:01.220+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.883837+0000 front 2024-02-09T18:08:31.883818+0000 (oldest deadline 2024-02-09T18:08:54.783844+0000) 2024-02-09T18:09:01.921 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:01 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:01.533+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:01.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:01 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:01.533+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:02.351 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:09:01 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:09:01.992+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.294439+0000 front 2024-02-09T18:08:23.294609+0000 (oldest deadline 2024-02-09T18:08:44.394323+0000) 2024-02-09T18:09:02.351 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:09:01 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:09:01.992+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.195929+0000 front 2024-02-09T18:08:30.195975+0000 (oldest deadline 2024-02-09T18:08:53.695778+0000) 2024-02-09T18:09:02.351 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:09:01 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:09:01.992+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:37.697405+0000 front 2024-02-09T18:08:37.697439+0000 (oldest deadline 2024-02-09T18:09:01.797221+0000) 2024-02-09T18:09:02.351 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:02 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:02.204+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.082620+0000 front 2024-02-09T18:08:23.082482+0000 (oldest deadline 2024-02-09T18:08:46.582507+0000) 2024-02-09T18:09:02.351 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:02 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:02.204+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.883837+0000 front 2024-02-09T18:08:31.883818+0000 (oldest deadline 2024-02-09T18:08:54.783844+0000) 2024-02-09T18:09:02.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:01 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:01.984+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:09:02.422 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:02 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:02.229+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:20.919791+0000 front 2024-02-09T18:08:20.919903+0000 (oldest deadline 2024-02-09T18:08:46.219692+0000) 2024-02-09T18:09:02.422 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:02 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:02.229+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.520735+0000 front 2024-02-09T18:08:31.520846+0000 (oldest deadline 2024-02-09T18:08:56.820686+0000) 2024-02-09T18:09:02.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:02 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:02.171+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:09:02.423 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:02 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:02.171+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:27.021731+0000 front 2024-02-09T18:08:27.021779+0000 (oldest deadline 2024-02-09T18:08:57.523295+0000) 2024-02-09T18:09:02.677 INFO:journalctl@ceph.osd.5.smithi162.stdout:Feb 09 18:09:02 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-5[56859]: 2024-02-09T18:09:02.348+0000 7f8e2122c700 -1 osd.5 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:29.061449+0000 front 2024-02-09T18:08:29.061391+0000 (oldest deadline 2024-02-09T18:08:54.361172+0000) 2024-02-09T18:09:02.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:02 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:02.535+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:02.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:02 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:02.535+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:03.304 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:09:03 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:09:03.009+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.294439+0000 front 2024-02-09T18:08:23.294609+0000 (oldest deadline 2024-02-09T18:08:44.394323+0000) 2024-02-09T18:09:03.304 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:09:03 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:09:03.009+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.195929+0000 front 2024-02-09T18:08:30.195975+0000 (oldest deadline 2024-02-09T18:08:53.695778+0000) 2024-02-09T18:09:03.304 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:09:03 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:09:03.009+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:37.697405+0000 front 2024-02-09T18:08:37.697439+0000 (oldest deadline 2024-02-09T18:09:01.797221+0000) 2024-02-09T18:09:03.305 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:03 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:03.213+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.082620+0000 front 2024-02-09T18:08:23.082482+0000 (oldest deadline 2024-02-09T18:08:46.582507+0000) 2024-02-09T18:09:03.305 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:03 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:03.213+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.883837+0000 front 2024-02-09T18:08:31.883818+0000 (oldest deadline 2024-02-09T18:08:54.783844+0000) 2024-02-09T18:09:03.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:03 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:03.003+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:09:03.422 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:03 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:03.210+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:20.919791+0000 front 2024-02-09T18:08:20.919903+0000 (oldest deadline 2024-02-09T18:08:46.219692+0000) 2024-02-09T18:09:03.422 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:03 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:03.210+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.520735+0000 front 2024-02-09T18:08:31.520846+0000 (oldest deadline 2024-02-09T18:08:56.820686+0000) 2024-02-09T18:09:03.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:03 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:03.218+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:09:03.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:03 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:03.218+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:27.021731+0000 front 2024-02-09T18:08:27.021779+0000 (oldest deadline 2024-02-09T18:08:57.523295+0000) 2024-02-09T18:09:03.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:03 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:03.218+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.624366+0000 front 2024-02-09T18:08:38.624336+0000 (oldest deadline 2024-02-09T18:09:02.724311+0000) 2024-02-09T18:09:03.676 INFO:journalctl@ceph.osd.5.smithi162.stdout:Feb 09 18:09:03 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-5[56859]: 2024-02-09T18:09:03.302+0000 7f8e2122c700 -1 osd.5 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:29.061449+0000 front 2024-02-09T18:08:29.061391+0000 (oldest deadline 2024-02-09T18:08:54.361172+0000) 2024-02-09T18:09:03.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:03 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:03.577+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:03.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:03 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:03.577+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:04.249 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:03 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:03.986+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:09:04.249 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:03 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:03.986+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.694773+0000 front 2024-02-09T18:08:38.694887+0000 (oldest deadline 2024-02-09T18:09:03.394749+0000) 2024-02-09T18:09:04.249 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:04 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:04.171+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:20.919791+0000 front 2024-02-09T18:08:20.919903+0000 (oldest deadline 2024-02-09T18:08:46.219692+0000) 2024-02-09T18:09:04.250 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:04 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:04.171+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.520735+0000 front 2024-02-09T18:08:31.520846+0000 (oldest deadline 2024-02-09T18:08:56.820686+0000) 2024-02-09T18:09:04.331 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:09:04 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:09:04.010+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.294439+0000 front 2024-02-09T18:08:23.294609+0000 (oldest deadline 2024-02-09T18:08:44.394323+0000) 2024-02-09T18:09:04.332 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:09:04 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:09:04.010+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.195929+0000 front 2024-02-09T18:08:30.195975+0000 (oldest deadline 2024-02-09T18:08:53.695778+0000) 2024-02-09T18:09:04.332 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:09:04 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:09:04.010+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:37.697405+0000 front 2024-02-09T18:08:37.697439+0000 (oldest deadline 2024-02-09T18:09:01.797221+0000) 2024-02-09T18:09:04.332 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:04 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:04.235+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.082620+0000 front 2024-02-09T18:08:23.082482+0000 (oldest deadline 2024-02-09T18:08:46.582507+0000) 2024-02-09T18:09:04.332 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:04 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:04.235+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.883837+0000 front 2024-02-09T18:08:31.883818+0000 (oldest deadline 2024-02-09T18:08:54.783844+0000) 2024-02-09T18:09:04.332 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:04 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:04.235+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:39.985697+0000 front 2024-02-09T18:08:39.985861+0000 (oldest deadline 2024-02-09T18:09:04.085883+0000) 2024-02-09T18:09:04.601 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:04 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:04.246+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:09:04.602 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:04 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:04.246+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:27.021731+0000 front 2024-02-09T18:08:27.021779+0000 (oldest deadline 2024-02-09T18:08:57.523295+0000) 2024-02-09T18:09:04.602 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:04 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:04.246+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.624366+0000 front 2024-02-09T18:08:38.624336+0000 (oldest deadline 2024-02-09T18:09:02.724311+0000) 2024-02-09T18:09:04.676 INFO:journalctl@ceph.osd.5.smithi162.stdout:Feb 09 18:09:04 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-5[56859]: 2024-02-09T18:09:04.330+0000 7f8e2122c700 -1 osd.5 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:29.061449+0000 front 2024-02-09T18:08:29.061391+0000 (oldest deadline 2024-02-09T18:08:54.361172+0000) 2024-02-09T18:09:04.677 INFO:journalctl@ceph.osd.5.smithi162.stdout:Feb 09 18:09:04 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-5[56859]: 2024-02-09T18:09:04.330+0000 7f8e2122c700 -1 osd.5 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:39.563065+0000 front 2024-02-09T18:08:39.562981+0000 (oldest deadline 2024-02-09T18:09:04.262965+0000) 2024-02-09T18:09:04.921 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:04 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:04.600+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:04.921 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:04 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:04.600+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:05.261 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:09:04 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:09:04.964+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.294439+0000 front 2024-02-09T18:08:23.294609+0000 (oldest deadline 2024-02-09T18:08:44.394323+0000) 2024-02-09T18:09:05.261 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:09:04 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:09:04.964+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.195929+0000 front 2024-02-09T18:08:30.195975+0000 (oldest deadline 2024-02-09T18:08:53.695778+0000) 2024-02-09T18:09:05.261 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:09:04 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:09:04.964+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:37.697405+0000 front 2024-02-09T18:08:37.697439+0000 (oldest deadline 2024-02-09T18:09:01.797221+0000) 2024-02-09T18:09:05.421 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:05 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:05.225+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:09:05.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:05 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:05.225+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:27.021731+0000 front 2024-02-09T18:08:27.021779+0000 (oldest deadline 2024-02-09T18:08:57.523295+0000) 2024-02-09T18:09:05.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:05 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:05.225+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.624366+0000 front 2024-02-09T18:08:38.624336+0000 (oldest deadline 2024-02-09T18:09:02.724311+0000) 2024-02-09T18:09:05.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:04 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:04.991+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:09:05.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:04 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:04.991+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.694773+0000 front 2024-02-09T18:08:38.694887+0000 (oldest deadline 2024-02-09T18:09:03.394749+0000) 2024-02-09T18:09:05.422 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:05 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:05.168+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:20.919791+0000 front 2024-02-09T18:08:20.919903+0000 (oldest deadline 2024-02-09T18:08:46.219692+0000) 2024-02-09T18:09:05.422 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:05 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:05.168+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.520735+0000 front 2024-02-09T18:08:31.520846+0000 (oldest deadline 2024-02-09T18:08:56.820686+0000) 2024-02-09T18:09:05.677 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:05 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:05.260+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.082620+0000 front 2024-02-09T18:08:23.082482+0000 (oldest deadline 2024-02-09T18:08:46.582507+0000) 2024-02-09T18:09:05.677 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:05 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:05.260+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.883837+0000 front 2024-02-09T18:08:31.883818+0000 (oldest deadline 2024-02-09T18:08:54.783844+0000) 2024-02-09T18:09:05.677 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:05 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:05.260+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:39.985697+0000 front 2024-02-09T18:08:39.985861+0000 (oldest deadline 2024-02-09T18:09:04.085883+0000) 2024-02-09T18:09:05.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:05 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:05.611+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:05.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:05 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:05.611+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:06.177 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:09:05 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:09:05.916+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.294439+0000 front 2024-02-09T18:08:23.294609+0000 (oldest deadline 2024-02-09T18:08:44.394323+0000) 2024-02-09T18:09:06.177 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:09:05 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:09:05.916+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.195929+0000 front 2024-02-09T18:08:30.195975+0000 (oldest deadline 2024-02-09T18:08:53.695778+0000) 2024-02-09T18:09:06.177 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:09:05 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:09:05.916+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:37.697405+0000 front 2024-02-09T18:08:37.697439+0000 (oldest deadline 2024-02-09T18:09:01.797221+0000) 2024-02-09T18:09:06.250 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:05 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:05.980+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:09:06.250 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:05 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:05.980+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.694773+0000 front 2024-02-09T18:08:38.694887+0000 (oldest deadline 2024-02-09T18:09:03.394749+0000) 2024-02-09T18:09:06.250 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:06 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:06.146+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:20.919791+0000 front 2024-02-09T18:08:20.919903+0000 (oldest deadline 2024-02-09T18:08:46.219692+0000) 2024-02-09T18:09:06.250 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:06 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:06.146+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.520735+0000 front 2024-02-09T18:08:31.520846+0000 (oldest deadline 2024-02-09T18:08:56.820686+0000) 2024-02-09T18:09:06.250 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:06 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:06.146+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:39.622765+0000 front 2024-02-09T18:08:39.622813+0000 (oldest deadline 2024-02-09T18:09:05.522706+0000) 2024-02-09T18:09:06.627 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:06 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:06.248+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:09:06.627 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:06 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:06.248+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:27.021731+0000 front 2024-02-09T18:08:27.021779+0000 (oldest deadline 2024-02-09T18:08:57.523295+0000) 2024-02-09T18:09:06.627 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:06 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:06.248+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.624366+0000 front 2024-02-09T18:08:38.624336+0000 (oldest deadline 2024-02-09T18:09:02.724311+0000) 2024-02-09T18:09:06.676 INFO:journalctl@ceph.osd.5.smithi162.stdout:Feb 09 18:09:06 smithi162 bash[175576]: ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-5 2024-02-09T18:09:06.677 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:06 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:06.256+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.082620+0000 front 2024-02-09T18:08:23.082482+0000 (oldest deadline 2024-02-09T18:08:46.582507+0000) 2024-02-09T18:09:06.677 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:06 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:06.256+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.883837+0000 front 2024-02-09T18:08:31.883818+0000 (oldest deadline 2024-02-09T18:08:54.783844+0000) 2024-02-09T18:09:06.677 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:06 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:06.256+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:39.985697+0000 front 2024-02-09T18:08:39.985861+0000 (oldest deadline 2024-02-09T18:09:04.085883+0000) 2024-02-09T18:09:06.921 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:06 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:06.625+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:06.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:06 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:06.625+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:06.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:06 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:06.625+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.425025+0000 front 2024-02-09T18:08:38.425098+0000 (oldest deadline 2024-02-09T18:09:06.025618+0000) 2024-02-09T18:09:07.163 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:09:06 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:09:06.901+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.294439+0000 front 2024-02-09T18:08:23.294609+0000 (oldest deadline 2024-02-09T18:08:44.394323+0000) 2024-02-09T18:09:07.163 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:09:06 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:09:06.901+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.195929+0000 front 2024-02-09T18:08:30.195975+0000 (oldest deadline 2024-02-09T18:08:53.695778+0000) 2024-02-09T18:09:07.163 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:09:06 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:09:06.901+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:37.697405+0000 front 2024-02-09T18:08:37.697439+0000 (oldest deadline 2024-02-09T18:09:01.797221+0000) 2024-02-09T18:09:07.196 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:06 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:06.944+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:09:07.196 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:06 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:06.944+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.694773+0000 front 2024-02-09T18:08:38.694887+0000 (oldest deadline 2024-02-09T18:09:03.394749+0000) 2024-02-09T18:09:07.567 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:07 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:07.249+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.082620+0000 front 2024-02-09T18:08:23.082482+0000 (oldest deadline 2024-02-09T18:08:46.582507+0000) 2024-02-09T18:09:07.567 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:07 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:07.249+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.883837+0000 front 2024-02-09T18:08:31.883818+0000 (oldest deadline 2024-02-09T18:08:54.783844+0000) 2024-02-09T18:09:07.567 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:07 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:07.249+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:39.985697+0000 front 2024-02-09T18:08:39.985861+0000 (oldest deadline 2024-02-09T18:09:04.085883+0000) 2024-02-09T18:09:07.583 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:07 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:07.195+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:20.919791+0000 front 2024-02-09T18:08:20.919903+0000 (oldest deadline 2024-02-09T18:08:46.219692+0000) 2024-02-09T18:09:07.583 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:07 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:07.195+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.520735+0000 front 2024-02-09T18:08:31.520846+0000 (oldest deadline 2024-02-09T18:08:56.820686+0000) 2024-02-09T18:09:07.583 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:07 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:07.195+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:39.622765+0000 front 2024-02-09T18:08:39.622813+0000 (oldest deadline 2024-02-09T18:09:05.522706+0000) 2024-02-09T18:09:07.583 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:07 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:07.283+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:09:07.583 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:07 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:07.283+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:27.021731+0000 front 2024-02-09T18:08:27.021779+0000 (oldest deadline 2024-02-09T18:08:57.523295+0000) 2024-02-09T18:09:07.583 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:07 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:07.283+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.624366+0000 front 2024-02-09T18:08:38.624336+0000 (oldest deadline 2024-02-09T18:09:02.724311+0000) 2024-02-09T18:09:07.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:07 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:07.580+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:07.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:07 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:07.580+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:07.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:07 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:07.580+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.425025+0000 front 2024-02-09T18:08:38.425098+0000 (oldest deadline 2024-02-09T18:09:06.025618+0000) 2024-02-09T18:09:08.177 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:09:07 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:09:07.878+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.294439+0000 front 2024-02-09T18:08:23.294609+0000 (oldest deadline 2024-02-09T18:08:44.394323+0000) 2024-02-09T18:09:08.177 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:09:07 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:09:07.878+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.195929+0000 front 2024-02-09T18:08:30.195975+0000 (oldest deadline 2024-02-09T18:08:53.695778+0000) 2024-02-09T18:09:08.177 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:09:07 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:09:07.878+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:37.697405+0000 front 2024-02-09T18:08:37.697439+0000 (oldest deadline 2024-02-09T18:09:01.797221+0000) 2024-02-09T18:09:08.193 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a69c870a-c76e-11ee-95b7-87774f69a715@osd.5.service' 2024-02-09T18:09:08.236 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T18:09:08.236 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-02-09T18:09:08.236 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-02-09T18:09:08.236 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-a69c870a-c76e-11ee-95b7-87774f69a715@osd.6 2024-02-09T18:09:08.237 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:07 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:07.978+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:09:08.237 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:07 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:07.978+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.694773+0000 front 2024-02-09T18:08:38.694887+0000 (oldest deadline 2024-02-09T18:09:03.394749+0000) 2024-02-09T18:09:08.541 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:08 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:08.270+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.082620+0000 front 2024-02-09T18:08:23.082482+0000 (oldest deadline 2024-02-09T18:08:46.582507+0000) 2024-02-09T18:09:08.541 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:08 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:08.270+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.883837+0000 front 2024-02-09T18:08:31.883818+0000 (oldest deadline 2024-02-09T18:08:54.783844+0000) 2024-02-09T18:09:08.541 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:08 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:08.270+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:39.985697+0000 front 2024-02-09T18:08:39.985861+0000 (oldest deadline 2024-02-09T18:09:04.085883+0000) 2024-02-09T18:09:08.542 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:09:08 smithi162 systemd[1]: Stopping Ceph osd.6 for a69c870a-c76e-11ee-95b7-87774f69a715... 2024-02-09T18:09:08.542 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:09:08 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:09:08.538+0000 7f2030aa2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T18:09:08.596 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:08 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:08.235+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:20.919791+0000 front 2024-02-09T18:08:20.919903+0000 (oldest deadline 2024-02-09T18:08:46.219692+0000) 2024-02-09T18:09:08.596 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:08 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:08.235+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.520735+0000 front 2024-02-09T18:08:31.520846+0000 (oldest deadline 2024-02-09T18:08:56.820686+0000) 2024-02-09T18:09:08.597 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:08 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:08.235+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:39.622765+0000 front 2024-02-09T18:08:39.622813+0000 (oldest deadline 2024-02-09T18:09:05.522706+0000) 2024-02-09T18:09:08.597 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:08 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:08.245+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:09:08.597 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:08 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:08.245+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:27.021731+0000 front 2024-02-09T18:08:27.021779+0000 (oldest deadline 2024-02-09T18:08:57.523295+0000) 2024-02-09T18:09:08.597 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:08 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:08.245+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.624366+0000 front 2024-02-09T18:08:38.624336+0000 (oldest deadline 2024-02-09T18:09:02.724311+0000) 2024-02-09T18:09:08.914 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:09:08 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:09:08.538+0000 7f2030aa2700 -1 osd.6 1050 *** Got signal Terminated *** 2024-02-09T18:09:08.915 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:09:08 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:09:08.538+0000 7f2030aa2700 -1 osd.6 1050 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-09T18:09:08.921 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:08 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:08.595+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:08.921 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:08 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:08.595+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:08.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:08 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:08.595+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.425025+0000 front 2024-02-09T18:08:38.425098+0000 (oldest deadline 2024-02-09T18:09:06.025618+0000) 2024-02-09T18:09:09.176 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:09:08 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:09:08.913+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.294439+0000 front 2024-02-09T18:08:23.294609+0000 (oldest deadline 2024-02-09T18:08:44.394323+0000) 2024-02-09T18:09:09.177 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:09:08 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:09:08.913+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.195929+0000 front 2024-02-09T18:08:30.195975+0000 (oldest deadline 2024-02-09T18:08:53.695778+0000) 2024-02-09T18:09:09.177 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:09:08 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:09:08.913+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:37.697405+0000 front 2024-02-09T18:08:37.697439+0000 (oldest deadline 2024-02-09T18:09:01.797221+0000) 2024-02-09T18:09:09.260 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:08 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:08.994+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:09:09.260 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:08 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:08.994+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.694773+0000 front 2024-02-09T18:08:38.694887+0000 (oldest deadline 2024-02-09T18:09:03.394749+0000) 2024-02-09T18:09:09.623 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:09 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:09.283+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:20.919791+0000 front 2024-02-09T18:08:20.919903+0000 (oldest deadline 2024-02-09T18:08:46.219692+0000) 2024-02-09T18:09:09.623 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:09 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:09.283+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.520735+0000 front 2024-02-09T18:08:31.520846+0000 (oldest deadline 2024-02-09T18:08:56.820686+0000) 2024-02-09T18:09:09.623 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:09 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:09.283+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:39.622765+0000 front 2024-02-09T18:08:39.622813+0000 (oldest deadline 2024-02-09T18:09:05.522706+0000) 2024-02-09T18:09:09.623 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:09 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:09.258+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:09:09.623 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:09 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:09.258+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:27.021731+0000 front 2024-02-09T18:08:27.021779+0000 (oldest deadline 2024-02-09T18:08:57.523295+0000) 2024-02-09T18:09:09.624 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:09 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:09.258+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.624366+0000 front 2024-02-09T18:08:38.624336+0000 (oldest deadline 2024-02-09T18:09:02.724311+0000) 2024-02-09T18:09:09.676 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:09 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:09.224+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.082620+0000 front 2024-02-09T18:08:23.082482+0000 (oldest deadline 2024-02-09T18:08:46.582507+0000) 2024-02-09T18:09:09.677 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:09 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:09.224+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.883837+0000 front 2024-02-09T18:08:31.883818+0000 (oldest deadline 2024-02-09T18:08:54.783844+0000) 2024-02-09T18:09:09.677 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:09 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:09.224+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:39.985697+0000 front 2024-02-09T18:08:39.985861+0000 (oldest deadline 2024-02-09T18:09:04.085883+0000) 2024-02-09T18:09:09.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:09 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:09.621+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:09.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:09 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:09.621+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:09.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:09 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:09.621+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.425025+0000 front 2024-02-09T18:08:38.425098+0000 (oldest deadline 2024-02-09T18:09:06.025618+0000) 2024-02-09T18:09:10.177 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:09:09 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:09:09.894+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.294439+0000 front 2024-02-09T18:08:23.294609+0000 (oldest deadline 2024-02-09T18:08:44.394323+0000) 2024-02-09T18:09:10.177 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:09:09 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:09:09.894+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.195929+0000 front 2024-02-09T18:08:30.195975+0000 (oldest deadline 2024-02-09T18:08:53.695778+0000) 2024-02-09T18:09:10.177 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:09:09 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:09:09.894+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:37.697405+0000 front 2024-02-09T18:08:37.697439+0000 (oldest deadline 2024-02-09T18:09:01.797221+0000) 2024-02-09T18:09:10.237 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:09 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:09.971+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:09:10.237 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:09 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:09.971+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.694773+0000 front 2024-02-09T18:08:38.694887+0000 (oldest deadline 2024-02-09T18:09:03.394749+0000) 2024-02-09T18:09:10.587 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:10 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:10.268+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:20.919791+0000 front 2024-02-09T18:08:20.919903+0000 (oldest deadline 2024-02-09T18:08:46.219692+0000) 2024-02-09T18:09:10.587 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:10 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:10.268+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.520735+0000 front 2024-02-09T18:08:31.520846+0000 (oldest deadline 2024-02-09T18:08:56.820686+0000) 2024-02-09T18:09:10.587 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:10 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:10.268+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:39.622765+0000 front 2024-02-09T18:08:39.622813+0000 (oldest deadline 2024-02-09T18:09:05.522706+0000) 2024-02-09T18:09:10.587 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:10 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:10.235+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:09:10.587 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:10 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:10.235+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:27.021731+0000 front 2024-02-09T18:08:27.021779+0000 (oldest deadline 2024-02-09T18:08:57.523295+0000) 2024-02-09T18:09:10.587 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:10 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:10.235+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.624366+0000 front 2024-02-09T18:08:38.624336+0000 (oldest deadline 2024-02-09T18:09:02.724311+0000) 2024-02-09T18:09:10.677 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:10 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:10.226+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.082620+0000 front 2024-02-09T18:08:23.082482+0000 (oldest deadline 2024-02-09T18:08:46.582507+0000) 2024-02-09T18:09:10.677 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:10 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:10.226+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.883837+0000 front 2024-02-09T18:08:31.883818+0000 (oldest deadline 2024-02-09T18:08:54.783844+0000) 2024-02-09T18:09:10.677 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:10 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:10.226+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:39.985697+0000 front 2024-02-09T18:08:39.985861+0000 (oldest deadline 2024-02-09T18:09:04.085883+0000) 2024-02-09T18:09:10.921 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:10 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:10.584+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:10.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:10 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:10.584+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:10.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:10 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:10.584+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.425025+0000 front 2024-02-09T18:08:38.425098+0000 (oldest deadline 2024-02-09T18:09:06.025618+0000) 2024-02-09T18:09:11.176 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:09:10 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:09:10.924+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.294439+0000 front 2024-02-09T18:08:23.294609+0000 (oldest deadline 2024-02-09T18:08:44.394323+0000) 2024-02-09T18:09:11.176 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:09:10 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:09:10.924+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.195929+0000 front 2024-02-09T18:08:30.195975+0000 (oldest deadline 2024-02-09T18:08:53.695778+0000) 2024-02-09T18:09:11.177 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:09:10 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:09:10.924+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:37.697405+0000 front 2024-02-09T18:08:37.697439+0000 (oldest deadline 2024-02-09T18:09:01.797221+0000) 2024-02-09T18:09:11.177 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:09:10 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:09:10.924+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:47.598768+0000 front 2024-02-09T18:08:47.598816+0000 (oldest deadline 2024-02-09T18:09:09.898686+0000) 2024-02-09T18:09:11.276 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:11 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:11.007+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:09:11.276 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:11 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:11.007+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.694773+0000 front 2024-02-09T18:08:38.694887+0000 (oldest deadline 2024-02-09T18:09:03.394749+0000) 2024-02-09T18:09:11.277 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:11 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:11.200+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:09:11.277 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:11 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:11.200+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:27.021731+0000 front 2024-02-09T18:08:27.021779+0000 (oldest deadline 2024-02-09T18:08:57.523295+0000) 2024-02-09T18:09:11.277 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:11 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:11.200+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.624366+0000 front 2024-02-09T18:08:38.624336+0000 (oldest deadline 2024-02-09T18:09:02.724311+0000) 2024-02-09T18:09:11.609 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:11 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:11.275+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:20.919791+0000 front 2024-02-09T18:08:20.919903+0000 (oldest deadline 2024-02-09T18:08:46.219692+0000) 2024-02-09T18:09:11.609 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:11 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:11.275+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.520735+0000 front 2024-02-09T18:08:31.520846+0000 (oldest deadline 2024-02-09T18:08:56.820686+0000) 2024-02-09T18:09:11.609 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:11 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:11.275+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:39.622765+0000 front 2024-02-09T18:08:39.622813+0000 (oldest deadline 2024-02-09T18:09:05.522706+0000) 2024-02-09T18:09:11.676 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:11 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:11.267+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.082620+0000 front 2024-02-09T18:08:23.082482+0000 (oldest deadline 2024-02-09T18:08:46.582507+0000) 2024-02-09T18:09:11.677 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:11 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:11.267+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.883837+0000 front 2024-02-09T18:08:31.883818+0000 (oldest deadline 2024-02-09T18:08:54.783844+0000) 2024-02-09T18:09:11.677 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:11 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:11.267+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:39.985697+0000 front 2024-02-09T18:08:39.985861+0000 (oldest deadline 2024-02-09T18:09:04.085883+0000) 2024-02-09T18:09:11.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:11 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:11.607+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:11.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:11 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:11.607+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:11.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:11 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:11.607+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.425025+0000 front 2024-02-09T18:08:38.425098+0000 (oldest deadline 2024-02-09T18:09:06.025618+0000) 2024-02-09T18:09:12.232 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:09:11 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:09:11.940+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.294439+0000 front 2024-02-09T18:08:23.294609+0000 (oldest deadline 2024-02-09T18:08:44.394323+0000) 2024-02-09T18:09:12.233 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:09:11 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:09:11.940+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.195929+0000 front 2024-02-09T18:08:30.195975+0000 (oldest deadline 2024-02-09T18:08:53.695778+0000) 2024-02-09T18:09:12.233 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:09:11 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:09:11.940+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:37.697405+0000 front 2024-02-09T18:08:37.697439+0000 (oldest deadline 2024-02-09T18:09:01.797221+0000) 2024-02-09T18:09:12.233 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:09:11 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:09:11.940+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:47.598768+0000 front 2024-02-09T18:08:47.598816+0000 (oldest deadline 2024-02-09T18:09:09.898686+0000) 2024-02-09T18:09:12.277 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:12 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:12.011+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:09:12.278 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:12 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:12.011+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.694773+0000 front 2024-02-09T18:08:38.694887+0000 (oldest deadline 2024-02-09T18:09:03.394749+0000) 2024-02-09T18:09:12.278 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:12 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:12.169+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:09:12.278 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:12 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:12.169+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:27.021731+0000 front 2024-02-09T18:08:27.021779+0000 (oldest deadline 2024-02-09T18:08:57.523295+0000) 2024-02-09T18:09:12.278 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:12 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:12.169+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.624366+0000 front 2024-02-09T18:08:38.624336+0000 (oldest deadline 2024-02-09T18:09:02.724311+0000) 2024-02-09T18:09:12.612 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:12 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:12.275+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:20.919791+0000 front 2024-02-09T18:08:20.919903+0000 (oldest deadline 2024-02-09T18:08:46.219692+0000) 2024-02-09T18:09:12.613 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:12 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:12.275+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.520735+0000 front 2024-02-09T18:08:31.520846+0000 (oldest deadline 2024-02-09T18:08:56.820686+0000) 2024-02-09T18:09:12.613 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:12 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:12.275+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:39.622765+0000 front 2024-02-09T18:08:39.622813+0000 (oldest deadline 2024-02-09T18:09:05.522706+0000) 2024-02-09T18:09:12.676 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:12 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:12.231+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.082620+0000 front 2024-02-09T18:08:23.082482+0000 (oldest deadline 2024-02-09T18:08:46.582507+0000) 2024-02-09T18:09:12.676 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:12 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:12.231+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.883837+0000 front 2024-02-09T18:08:31.883818+0000 (oldest deadline 2024-02-09T18:08:54.783844+0000) 2024-02-09T18:09:12.677 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:12 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:12.231+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:39.985697+0000 front 2024-02-09T18:08:39.985861+0000 (oldest deadline 2024-02-09T18:09:04.085883+0000) 2024-02-09T18:09:12.677 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:12 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:12.231+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:47.586927+0000 front 2024-02-09T18:08:47.586965+0000 (oldest deadline 2024-02-09T18:09:11.686882+0000) 2024-02-09T18:09:12.921 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:12 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:12.611+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:12.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:12 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:12.611+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:12.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:12 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:12.611+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.425025+0000 front 2024-02-09T18:08:38.425098+0000 (oldest deadline 2024-02-09T18:09:06.025618+0000) 2024-02-09T18:09:12.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:12 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:12.611+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:46.026169+0000 front 2024-02-09T18:08:46.026259+0000 (oldest deadline 2024-02-09T18:09:12.426582+0000) 2024-02-09T18:09:13.291 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:12 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:12.987+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:09:13.292 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:12 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:12.987+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.694773+0000 front 2024-02-09T18:08:38.694887+0000 (oldest deadline 2024-02-09T18:09:03.394749+0000) 2024-02-09T18:09:13.292 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:12 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:12.987+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:47.495766+0000 front 2024-02-09T18:08:47.495922+0000 (oldest deadline 2024-02-09T18:09:12.195909+0000) 2024-02-09T18:09:13.292 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:13 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:13.198+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:09:13.292 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:13 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:13.198+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:27.021731+0000 front 2024-02-09T18:08:27.021779+0000 (oldest deadline 2024-02-09T18:08:57.523295+0000) 2024-02-09T18:09:13.292 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:13 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:13.198+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.624366+0000 front 2024-02-09T18:08:38.624336+0000 (oldest deadline 2024-02-09T18:09:02.724311+0000) 2024-02-09T18:09:13.427 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:13 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:13.186+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.082620+0000 front 2024-02-09T18:08:23.082482+0000 (oldest deadline 2024-02-09T18:08:46.582507+0000) 2024-02-09T18:09:13.427 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:13 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:13.186+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.883837+0000 front 2024-02-09T18:08:31.883818+0000 (oldest deadline 2024-02-09T18:08:54.783844+0000) 2024-02-09T18:09:13.427 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:13 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:13.186+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:39.985697+0000 front 2024-02-09T18:08:39.985861+0000 (oldest deadline 2024-02-09T18:09:04.085883+0000) 2024-02-09T18:09:13.427 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:13 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:13.186+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:47.586927+0000 front 2024-02-09T18:08:47.586965+0000 (oldest deadline 2024-02-09T18:09:11.686882+0000) 2024-02-09T18:09:13.427 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:09:12 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:09:12.963+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.294439+0000 front 2024-02-09T18:08:23.294609+0000 (oldest deadline 2024-02-09T18:08:44.394323+0000) 2024-02-09T18:09:13.427 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:09:12 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:09:12.963+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.195929+0000 front 2024-02-09T18:08:30.195975+0000 (oldest deadline 2024-02-09T18:08:53.695778+0000) 2024-02-09T18:09:13.427 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:09:12 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:09:12.963+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:37.697405+0000 front 2024-02-09T18:08:37.697439+0000 (oldest deadline 2024-02-09T18:09:01.797221+0000) 2024-02-09T18:09:13.427 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:09:12 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6[64718]: 2024-02-09T18:09:12.963+0000 7f2029095700 -1 osd.6 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:47.598768+0000 front 2024-02-09T18:08:47.598816+0000 (oldest deadline 2024-02-09T18:09:09.898686+0000) 2024-02-09T18:09:13.627 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:13 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:13.290+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:20.919791+0000 front 2024-02-09T18:08:20.919903+0000 (oldest deadline 2024-02-09T18:08:46.219692+0000) 2024-02-09T18:09:13.628 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:13 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:13.290+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.520735+0000 front 2024-02-09T18:08:31.520846+0000 (oldest deadline 2024-02-09T18:08:56.820686+0000) 2024-02-09T18:09:13.628 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:13 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:13.290+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:39.622765+0000 front 2024-02-09T18:08:39.622813+0000 (oldest deadline 2024-02-09T18:09:05.522706+0000) 2024-02-09T18:09:13.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:13 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:13.626+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:13.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:13 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:13.626+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:13.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:13 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:13.626+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.425025+0000 front 2024-02-09T18:08:38.425098+0000 (oldest deadline 2024-02-09T18:09:06.025618+0000) 2024-02-09T18:09:13.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:13 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:13.626+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:46.026169+0000 front 2024-02-09T18:08:46.026259+0000 (oldest deadline 2024-02-09T18:09:12.426582+0000) 2024-02-09T18:09:14.339 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:14 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:14.011+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:09:14.339 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:14 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:14.011+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.694773+0000 front 2024-02-09T18:08:38.694887+0000 (oldest deadline 2024-02-09T18:09:03.394749+0000) 2024-02-09T18:09:14.339 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:14 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:14.011+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:47.495766+0000 front 2024-02-09T18:08:47.495922+0000 (oldest deadline 2024-02-09T18:09:12.195909+0000) 2024-02-09T18:09:14.340 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:14 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:14.232+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:09:14.340 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:14 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:14.232+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:27.021731+0000 front 2024-02-09T18:08:27.021779+0000 (oldest deadline 2024-02-09T18:08:57.523295+0000) 2024-02-09T18:09:14.340 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:14 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:14.232+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.624366+0000 front 2024-02-09T18:08:38.624336+0000 (oldest deadline 2024-02-09T18:09:02.724311+0000) 2024-02-09T18:09:14.340 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:14 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:14.232+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:48.025152+0000 front 2024-02-09T18:08:48.025186+0000 (oldest deadline 2024-02-09T18:09:13.925314+0000) 2024-02-09T18:09:14.427 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:14 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:14.169+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.082620+0000 front 2024-02-09T18:08:23.082482+0000 (oldest deadline 2024-02-09T18:08:46.582507+0000) 2024-02-09T18:09:14.427 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:14 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:14.169+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.883837+0000 front 2024-02-09T18:08:31.883818+0000 (oldest deadline 2024-02-09T18:08:54.783844+0000) 2024-02-09T18:09:14.427 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:14 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:14.169+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:39.985697+0000 front 2024-02-09T18:08:39.985861+0000 (oldest deadline 2024-02-09T18:09:04.085883+0000) 2024-02-09T18:09:14.427 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:14 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:14.169+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:47.586927+0000 front 2024-02-09T18:08:47.586965+0000 (oldest deadline 2024-02-09T18:09:11.686882+0000) 2024-02-09T18:09:14.653 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:14 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:14.338+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:20.919791+0000 front 2024-02-09T18:08:20.919903+0000 (oldest deadline 2024-02-09T18:08:46.219692+0000) 2024-02-09T18:09:14.653 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:14 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:14.338+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.520735+0000 front 2024-02-09T18:08:31.520846+0000 (oldest deadline 2024-02-09T18:08:56.820686+0000) 2024-02-09T18:09:14.653 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:14 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:14.338+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:39.622765+0000 front 2024-02-09T18:08:39.622813+0000 (oldest deadline 2024-02-09T18:09:05.522706+0000) 2024-02-09T18:09:14.921 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:14 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:14.651+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:14.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:14 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:14.651+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:14.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:14 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:14.651+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.425025+0000 front 2024-02-09T18:08:38.425098+0000 (oldest deadline 2024-02-09T18:09:06.025618+0000) 2024-02-09T18:09:14.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:14 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:14.651+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:46.026169+0000 front 2024-02-09T18:08:46.026259+0000 (oldest deadline 2024-02-09T18:09:12.426582+0000) 2024-02-09T18:09:14.926 INFO:journalctl@ceph.osd.6.smithi162.stdout:Feb 09 18:09:14 smithi162 bash[175841]: ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-6 2024-02-09T18:09:15.385 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:15 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:15.055+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:09:15.385 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:15 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:15.055+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.694773+0000 front 2024-02-09T18:08:38.694887+0000 (oldest deadline 2024-02-09T18:09:03.394749+0000) 2024-02-09T18:09:15.385 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:15 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:15.055+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:47.495766+0000 front 2024-02-09T18:08:47.495922+0000 (oldest deadline 2024-02-09T18:09:12.195909+0000) 2024-02-09T18:09:15.385 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:15 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:15.262+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:09:15.385 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:15 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:15.262+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:27.021731+0000 front 2024-02-09T18:08:27.021779+0000 (oldest deadline 2024-02-09T18:08:57.523295+0000) 2024-02-09T18:09:15.385 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:15 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:15.262+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.624366+0000 front 2024-02-09T18:08:38.624336+0000 (oldest deadline 2024-02-09T18:09:02.724311+0000) 2024-02-09T18:09:15.386 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:15 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:15.262+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:48.025152+0000 front 2024-02-09T18:08:48.025186+0000 (oldest deadline 2024-02-09T18:09:13.925314+0000) 2024-02-09T18:09:15.641 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:15 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:15.383+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:20.919791+0000 front 2024-02-09T18:08:20.919903+0000 (oldest deadline 2024-02-09T18:08:46.219692+0000) 2024-02-09T18:09:15.641 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:15 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:15.383+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.520735+0000 front 2024-02-09T18:08:31.520846+0000 (oldest deadline 2024-02-09T18:08:56.820686+0000) 2024-02-09T18:09:15.641 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:15 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:15.383+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:39.622765+0000 front 2024-02-09T18:08:39.622813+0000 (oldest deadline 2024-02-09T18:09:05.522706+0000) 2024-02-09T18:09:15.667 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:15 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:15.177+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.082620+0000 front 2024-02-09T18:08:23.082482+0000 (oldest deadline 2024-02-09T18:08:46.582507+0000) 2024-02-09T18:09:15.667 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:15 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:15.177+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.883837+0000 front 2024-02-09T18:08:31.883818+0000 (oldest deadline 2024-02-09T18:08:54.783844+0000) 2024-02-09T18:09:15.667 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:15 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:15.177+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:39.985697+0000 front 2024-02-09T18:08:39.985861+0000 (oldest deadline 2024-02-09T18:09:04.085883+0000) 2024-02-09T18:09:15.667 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:15 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:15.177+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:47.586927+0000 front 2024-02-09T18:08:47.586965+0000 (oldest deadline 2024-02-09T18:09:11.686882+0000) 2024-02-09T18:09:15.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:15 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:15.639+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:15.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:15 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:15.639+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:15.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:15 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:15.639+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.425025+0000 front 2024-02-09T18:08:38.425098+0000 (oldest deadline 2024-02-09T18:09:06.025618+0000) 2024-02-09T18:09:15.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:15 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:15.639+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:46.026169+0000 front 2024-02-09T18:08:46.026259+0000 (oldest deadline 2024-02-09T18:09:12.426582+0000) 2024-02-09T18:09:16.295 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:16 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:16.215+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.082620+0000 front 2024-02-09T18:08:23.082482+0000 (oldest deadline 2024-02-09T18:08:46.582507+0000) 2024-02-09T18:09:16.295 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:16 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:16.216+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.883837+0000 front 2024-02-09T18:08:31.883818+0000 (oldest deadline 2024-02-09T18:08:54.783844+0000) 2024-02-09T18:09:16.295 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:16 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:16.216+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:39.985697+0000 front 2024-02-09T18:08:39.985861+0000 (oldest deadline 2024-02-09T18:09:04.085883+0000) 2024-02-09T18:09:16.295 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:16 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:16.216+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:47.586927+0000 front 2024-02-09T18:08:47.586965+0000 (oldest deadline 2024-02-09T18:09:11.686882+0000) 2024-02-09T18:09:16.302 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a69c870a-c76e-11ee-95b7-87774f69a715@osd.6.service' 2024-02-09T18:09:16.342 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T18:09:16.342 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-02-09T18:09:16.342 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-02-09T18:09:16.342 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-a69c870a-c76e-11ee-95b7-87774f69a715@osd.7 2024-02-09T18:09:16.381 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:16 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:16.096+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:09:16.381 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:16 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:16.096+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.694773+0000 front 2024-02-09T18:08:38.694887+0000 (oldest deadline 2024-02-09T18:09:03.394749+0000) 2024-02-09T18:09:16.381 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:16 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:16.096+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:47.495766+0000 front 2024-02-09T18:08:47.495922+0000 (oldest deadline 2024-02-09T18:09:12.195909+0000) 2024-02-09T18:09:16.382 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:16 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:16.261+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:09:16.382 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:16 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:16.261+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:27.021731+0000 front 2024-02-09T18:08:27.021779+0000 (oldest deadline 2024-02-09T18:08:57.523295+0000) 2024-02-09T18:09:16.382 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:16 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:16.261+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.624366+0000 front 2024-02-09T18:08:38.624336+0000 (oldest deadline 2024-02-09T18:09:02.724311+0000) 2024-02-09T18:09:16.382 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:16 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:16.261+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:48.025152+0000 front 2024-02-09T18:08:48.025186+0000 (oldest deadline 2024-02-09T18:09:13.925314+0000) 2024-02-09T18:09:16.660 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:16 smithi162 systemd[1]: Stopping Ceph osd.7 for a69c870a-c76e-11ee-95b7-87774f69a715... 2024-02-09T18:09:16.661 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:16 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:16.659+0000 7f570b6d3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T18:09:16.671 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:16 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:16.379+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:20.919791+0000 front 2024-02-09T18:08:20.919903+0000 (oldest deadline 2024-02-09T18:08:46.219692+0000) 2024-02-09T18:09:16.672 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:16 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:16.379+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.520735+0000 front 2024-02-09T18:08:31.520846+0000 (oldest deadline 2024-02-09T18:08:56.820686+0000) 2024-02-09T18:09:16.672 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:16 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:16.379+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:39.622765+0000 front 2024-02-09T18:08:39.622813+0000 (oldest deadline 2024-02-09T18:09:05.522706+0000) 2024-02-09T18:09:16.672 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:16 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:16.611+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:16.672 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:16 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:16.611+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:16.672 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:16 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:16.611+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.425025+0000 front 2024-02-09T18:08:38.425098+0000 (oldest deadline 2024-02-09T18:09:06.025618+0000) 2024-02-09T18:09:16.672 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:16 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:16.611+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:46.026169+0000 front 2024-02-09T18:08:46.026259+0000 (oldest deadline 2024-02-09T18:09:12.426582+0000) 2024-02-09T18:09:16.926 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:16 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:16.659+0000 7f570b6d3700 -1 osd.7 1050 *** Got signal Terminated *** 2024-02-09T18:09:16.926 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:16 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:16.659+0000 7f570b6d3700 -1 osd.7 1050 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-09T18:09:17.403 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:17 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:17.103+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:09:17.403 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:17 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:17.103+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.694773+0000 front 2024-02-09T18:08:38.694887+0000 (oldest deadline 2024-02-09T18:09:03.394749+0000) 2024-02-09T18:09:17.403 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:17 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:17.103+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:47.495766+0000 front 2024-02-09T18:08:47.495922+0000 (oldest deadline 2024-02-09T18:09:12.195909+0000) 2024-02-09T18:09:17.404 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:17 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:17.277+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:09:17.404 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:17 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:17.277+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:27.021731+0000 front 2024-02-09T18:08:27.021779+0000 (oldest deadline 2024-02-09T18:08:57.523295+0000) 2024-02-09T18:09:17.404 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:17 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:17.277+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.624366+0000 front 2024-02-09T18:08:38.624336+0000 (oldest deadline 2024-02-09T18:09:02.724311+0000) 2024-02-09T18:09:17.404 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:17 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:17.277+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:48.025152+0000 front 2024-02-09T18:08:48.025186+0000 (oldest deadline 2024-02-09T18:09:13.925314+0000) 2024-02-09T18:09:17.672 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:17 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:17.402+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:20.919791+0000 front 2024-02-09T18:08:20.919903+0000 (oldest deadline 2024-02-09T18:08:46.219692+0000) 2024-02-09T18:09:17.672 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:17 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:17.402+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.520735+0000 front 2024-02-09T18:08:31.520846+0000 (oldest deadline 2024-02-09T18:08:56.820686+0000) 2024-02-09T18:09:17.672 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:17 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:17.402+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:39.622765+0000 front 2024-02-09T18:08:39.622813+0000 (oldest deadline 2024-02-09T18:09:05.522706+0000) 2024-02-09T18:09:17.672 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:17 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:17.579+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:17.672 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:17 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:17.579+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:17.672 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:17 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:17.579+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.425025+0000 front 2024-02-09T18:08:38.425098+0000 (oldest deadline 2024-02-09T18:09:06.025618+0000) 2024-02-09T18:09:17.672 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:17 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:17.579+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:46.026169+0000 front 2024-02-09T18:08:46.026259+0000 (oldest deadline 2024-02-09T18:09:12.426582+0000) 2024-02-09T18:09:17.677 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:17 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:17.197+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.082620+0000 front 2024-02-09T18:08:23.082482+0000 (oldest deadline 2024-02-09T18:08:46.582507+0000) 2024-02-09T18:09:17.677 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:17 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:17.197+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.883837+0000 front 2024-02-09T18:08:31.883818+0000 (oldest deadline 2024-02-09T18:08:54.783844+0000) 2024-02-09T18:09:17.677 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:17 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:17.197+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:39.985697+0000 front 2024-02-09T18:08:39.985861+0000 (oldest deadline 2024-02-09T18:09:04.085883+0000) 2024-02-09T18:09:17.677 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:17 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:17.197+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:47.586927+0000 front 2024-02-09T18:08:47.586965+0000 (oldest deadline 2024-02-09T18:09:11.686882+0000) 2024-02-09T18:09:18.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:18 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:18.097+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:09:18.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:18 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:18.097+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.694773+0000 front 2024-02-09T18:08:38.694887+0000 (oldest deadline 2024-02-09T18:09:03.394749+0000) 2024-02-09T18:09:18.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:18 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:18.097+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:47.495766+0000 front 2024-02-09T18:08:47.495922+0000 (oldest deadline 2024-02-09T18:09:12.195909+0000) 2024-02-09T18:09:18.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:18 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:18.236+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:09:18.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:18 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:18.236+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:27.021731+0000 front 2024-02-09T18:08:27.021779+0000 (oldest deadline 2024-02-09T18:08:57.523295+0000) 2024-02-09T18:09:18.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:18 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:18.236+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.624366+0000 front 2024-02-09T18:08:38.624336+0000 (oldest deadline 2024-02-09T18:09:02.724311+0000) 2024-02-09T18:09:18.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:18 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:18.236+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:48.025152+0000 front 2024-02-09T18:08:48.025186+0000 (oldest deadline 2024-02-09T18:09:13.925314+0000) 2024-02-09T18:09:18.423 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:18 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:18.236+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.826611+0000 front 2024-02-09T18:08:56.826637+0000 (oldest deadline 2024-02-09T18:09:17.926388+0000) 2024-02-09T18:09:18.426 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:18 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:18.154+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.082620+0000 front 2024-02-09T18:08:23.082482+0000 (oldest deadline 2024-02-09T18:08:46.582507+0000) 2024-02-09T18:09:18.427 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:18 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:18.154+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.883837+0000 front 2024-02-09T18:08:31.883818+0000 (oldest deadline 2024-02-09T18:08:54.783844+0000) 2024-02-09T18:09:18.427 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:18 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:18.154+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:39.985697+0000 front 2024-02-09T18:08:39.985861+0000 (oldest deadline 2024-02-09T18:09:04.085883+0000) 2024-02-09T18:09:18.427 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:18 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:18.154+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:47.586927+0000 front 2024-02-09T18:08:47.586965+0000 (oldest deadline 2024-02-09T18:09:11.686882+0000) 2024-02-09T18:09:18.427 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:18 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:18.154+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:53.387879+0000 front 2024-02-09T18:08:53.387999+0000 (oldest deadline 2024-02-09T18:09:18.087983+0000) 2024-02-09T18:09:18.922 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:18 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:18.422+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:20.919791+0000 front 2024-02-09T18:08:20.919903+0000 (oldest deadline 2024-02-09T18:08:46.219692+0000) 2024-02-09T18:09:18.922 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:18 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:18.422+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.520735+0000 front 2024-02-09T18:08:31.520846+0000 (oldest deadline 2024-02-09T18:08:56.820686+0000) 2024-02-09T18:09:18.922 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:18 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:18.422+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:39.622765+0000 front 2024-02-09T18:08:39.622813+0000 (oldest deadline 2024-02-09T18:09:05.522706+0000) 2024-02-09T18:09:18.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:18 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:18.628+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:18.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:18 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:18.628+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:18.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:18 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:18.628+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.425025+0000 front 2024-02-09T18:08:38.425098+0000 (oldest deadline 2024-02-09T18:09:06.025618+0000) 2024-02-09T18:09:18.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:18 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:18.628+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:46.026169+0000 front 2024-02-09T18:08:46.026259+0000 (oldest deadline 2024-02-09T18:09:12.426582+0000) 2024-02-09T18:09:19.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:19 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:19.073+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:09:19.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:19 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:19.073+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.694773+0000 front 2024-02-09T18:08:38.694887+0000 (oldest deadline 2024-02-09T18:09:03.394749+0000) 2024-02-09T18:09:19.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:19 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:19.073+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:47.495766+0000 front 2024-02-09T18:08:47.495922+0000 (oldest deadline 2024-02-09T18:09:12.195909+0000) 2024-02-09T18:09:19.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:19 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:19.222+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:09:19.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:19 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:19.222+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:27.021731+0000 front 2024-02-09T18:08:27.021779+0000 (oldest deadline 2024-02-09T18:08:57.523295+0000) 2024-02-09T18:09:19.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:19 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:19.222+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.624366+0000 front 2024-02-09T18:08:38.624336+0000 (oldest deadline 2024-02-09T18:09:02.724311+0000) 2024-02-09T18:09:19.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:19 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:19.222+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:48.025152+0000 front 2024-02-09T18:08:48.025186+0000 (oldest deadline 2024-02-09T18:09:13.925314+0000) 2024-02-09T18:09:19.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:19 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:19.222+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.826611+0000 front 2024-02-09T18:08:56.826637+0000 (oldest deadline 2024-02-09T18:09:17.926388+0000) 2024-02-09T18:09:19.677 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:19 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:19.187+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.082620+0000 front 2024-02-09T18:08:23.082482+0000 (oldest deadline 2024-02-09T18:08:46.582507+0000) 2024-02-09T18:09:19.677 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:19 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:19.187+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.883837+0000 front 2024-02-09T18:08:31.883818+0000 (oldest deadline 2024-02-09T18:08:54.783844+0000) 2024-02-09T18:09:19.677 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:19 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:19.187+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:39.985697+0000 front 2024-02-09T18:08:39.985861+0000 (oldest deadline 2024-02-09T18:09:04.085883+0000) 2024-02-09T18:09:19.677 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:19 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:19.187+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:47.586927+0000 front 2024-02-09T18:08:47.586965+0000 (oldest deadline 2024-02-09T18:09:11.686882+0000) 2024-02-09T18:09:19.677 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:19 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:19.187+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:53.387879+0000 front 2024-02-09T18:08:53.387999+0000 (oldest deadline 2024-02-09T18:09:18.087983+0000) 2024-02-09T18:09:19.922 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:19 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:19.425+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:20.919791+0000 front 2024-02-09T18:08:20.919903+0000 (oldest deadline 2024-02-09T18:08:46.219692+0000) 2024-02-09T18:09:19.922 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:19 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:19.425+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.520735+0000 front 2024-02-09T18:08:31.520846+0000 (oldest deadline 2024-02-09T18:08:56.820686+0000) 2024-02-09T18:09:19.922 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:19 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:19.425+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:39.622765+0000 front 2024-02-09T18:08:39.622813+0000 (oldest deadline 2024-02-09T18:09:05.522706+0000) 2024-02-09T18:09:19.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:19 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:19.639+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:19.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:19 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:19.639+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:19.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:19 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:19.639+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.425025+0000 front 2024-02-09T18:08:38.425098+0000 (oldest deadline 2024-02-09T18:09:06.025618+0000) 2024-02-09T18:09:19.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:19 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:19.639+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:46.026169+0000 front 2024-02-09T18:08:46.026259+0000 (oldest deadline 2024-02-09T18:09:12.426582+0000) 2024-02-09T18:09:20.402 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:20 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:20.042+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:09:20.402 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:20 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:20.042+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.694773+0000 front 2024-02-09T18:08:38.694887+0000 (oldest deadline 2024-02-09T18:09:03.394749+0000) 2024-02-09T18:09:20.402 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:20 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:20.042+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:47.495766+0000 front 2024-02-09T18:08:47.495922+0000 (oldest deadline 2024-02-09T18:09:12.195909+0000) 2024-02-09T18:09:20.402 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:20 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:20.225+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:09:20.402 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:20 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:20.225+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:27.021731+0000 front 2024-02-09T18:08:27.021779+0000 (oldest deadline 2024-02-09T18:08:57.523295+0000) 2024-02-09T18:09:20.402 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:20 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:20.225+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.624366+0000 front 2024-02-09T18:08:38.624336+0000 (oldest deadline 2024-02-09T18:09:02.724311+0000) 2024-02-09T18:09:20.403 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:20 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:20.225+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:48.025152+0000 front 2024-02-09T18:08:48.025186+0000 (oldest deadline 2024-02-09T18:09:13.925314+0000) 2024-02-09T18:09:20.403 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:20 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:20.225+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.826611+0000 front 2024-02-09T18:08:56.826637+0000 (oldest deadline 2024-02-09T18:09:17.926388+0000) 2024-02-09T18:09:20.672 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:20 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:20.400+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:20.919791+0000 front 2024-02-09T18:08:20.919903+0000 (oldest deadline 2024-02-09T18:08:46.219692+0000) 2024-02-09T18:09:20.672 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:20 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:20.400+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.520735+0000 front 2024-02-09T18:08:31.520846+0000 (oldest deadline 2024-02-09T18:08:56.820686+0000) 2024-02-09T18:09:20.672 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:20 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:20.400+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:39.622765+0000 front 2024-02-09T18:08:39.622813+0000 (oldest deadline 2024-02-09T18:09:05.522706+0000) 2024-02-09T18:09:20.672 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:20 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:20.641+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:20.672 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:20 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:20.641+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:20.672 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:20 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:20.641+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.425025+0000 front 2024-02-09T18:08:38.425098+0000 (oldest deadline 2024-02-09T18:09:06.025618+0000) 2024-02-09T18:09:20.672 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:20 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:20.641+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:46.026169+0000 front 2024-02-09T18:08:46.026259+0000 (oldest deadline 2024-02-09T18:09:12.426582+0000) 2024-02-09T18:09:20.676 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:20 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:20.178+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.082620+0000 front 2024-02-09T18:08:23.082482+0000 (oldest deadline 2024-02-09T18:08:46.582507+0000) 2024-02-09T18:09:20.676 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:20 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:20.178+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.883837+0000 front 2024-02-09T18:08:31.883818+0000 (oldest deadline 2024-02-09T18:08:54.783844+0000) 2024-02-09T18:09:20.676 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:20 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:20.178+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:39.985697+0000 front 2024-02-09T18:08:39.985861+0000 (oldest deadline 2024-02-09T18:09:04.085883+0000) 2024-02-09T18:09:20.677 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:20 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:20.178+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:47.586927+0000 front 2024-02-09T18:08:47.586965+0000 (oldest deadline 2024-02-09T18:09:11.686882+0000) 2024-02-09T18:09:20.677 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:20 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:20.178+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:53.387879+0000 front 2024-02-09T18:08:53.387999+0000 (oldest deadline 2024-02-09T18:09:18.087983+0000) 2024-02-09T18:09:21.353 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:21 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:21.050+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:09:21.354 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:21 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:21.050+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.694773+0000 front 2024-02-09T18:08:38.694887+0000 (oldest deadline 2024-02-09T18:09:03.394749+0000) 2024-02-09T18:09:21.354 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:21 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:21.050+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:47.495766+0000 front 2024-02-09T18:08:47.495922+0000 (oldest deadline 2024-02-09T18:09:12.195909+0000) 2024-02-09T18:09:21.354 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:21 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:21.050+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.197757+0000 front 2024-02-09T18:08:56.197854+0000 (oldest deadline 2024-02-09T18:09:20.897605+0000) 2024-02-09T18:09:21.354 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:21 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:21.180+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:09:21.354 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:21 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:21.180+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:27.021731+0000 front 2024-02-09T18:08:27.021779+0000 (oldest deadline 2024-02-09T18:08:57.523295+0000) 2024-02-09T18:09:21.354 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:21 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:21.180+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.624366+0000 front 2024-02-09T18:08:38.624336+0000 (oldest deadline 2024-02-09T18:09:02.724311+0000) 2024-02-09T18:09:21.354 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:21 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:21.180+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:48.025152+0000 front 2024-02-09T18:08:48.025186+0000 (oldest deadline 2024-02-09T18:09:13.925314+0000) 2024-02-09T18:09:21.354 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:21 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:21.180+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.826611+0000 front 2024-02-09T18:08:56.826637+0000 (oldest deadline 2024-02-09T18:09:17.926388+0000) 2024-02-09T18:09:21.427 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:21 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:21.173+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.082620+0000 front 2024-02-09T18:08:23.082482+0000 (oldest deadline 2024-02-09T18:08:46.582507+0000) 2024-02-09T18:09:21.427 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:21 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:21.173+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.883837+0000 front 2024-02-09T18:08:31.883818+0000 (oldest deadline 2024-02-09T18:08:54.783844+0000) 2024-02-09T18:09:21.427 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:21 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:21.173+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:39.985697+0000 front 2024-02-09T18:08:39.985861+0000 (oldest deadline 2024-02-09T18:09:04.085883+0000) 2024-02-09T18:09:21.427 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:21 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:21.173+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:47.586927+0000 front 2024-02-09T18:08:47.586965+0000 (oldest deadline 2024-02-09T18:09:11.686882+0000) 2024-02-09T18:09:21.427 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:21 smithi162 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7[71983]: 2024-02-09T18:09:21.173+0000 7f5707cce700 -1 osd.7 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:53.387879+0000 front 2024-02-09T18:08:53.387999+0000 (oldest deadline 2024-02-09T18:09:18.087983+0000) 2024-02-09T18:09:21.672 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:21 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:21.352+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:20.919791+0000 front 2024-02-09T18:08:20.919903+0000 (oldest deadline 2024-02-09T18:08:46.219692+0000) 2024-02-09T18:09:21.672 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:21 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:21.352+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.520735+0000 front 2024-02-09T18:08:31.520846+0000 (oldest deadline 2024-02-09T18:08:56.820686+0000) 2024-02-09T18:09:21.672 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:21 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:21.352+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:39.622765+0000 front 2024-02-09T18:08:39.622813+0000 (oldest deadline 2024-02-09T18:09:05.522706+0000) 2024-02-09T18:09:21.672 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:21 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:21.600+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:21.672 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:21 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:21.600+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:21.672 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:21 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:21.600+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.425025+0000 front 2024-02-09T18:08:38.425098+0000 (oldest deadline 2024-02-09T18:09:06.025618+0000) 2024-02-09T18:09:21.673 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:21 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:21.600+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:46.026169+0000 front 2024-02-09T18:08:46.026259+0000 (oldest deadline 2024-02-09T18:09:12.426582+0000) 2024-02-09T18:09:22.392 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:22 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:22.070+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:09:22.393 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:22 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:22.070+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.694773+0000 front 2024-02-09T18:08:38.694887+0000 (oldest deadline 2024-02-09T18:09:03.394749+0000) 2024-02-09T18:09:22.393 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:22 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:22.070+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:47.495766+0000 front 2024-02-09T18:08:47.495922+0000 (oldest deadline 2024-02-09T18:09:12.195909+0000) 2024-02-09T18:09:22.393 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:22 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:22.070+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.197757+0000 front 2024-02-09T18:08:56.197854+0000 (oldest deadline 2024-02-09T18:09:20.897605+0000) 2024-02-09T18:09:22.393 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:22 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:22.188+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:09:22.393 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:22 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:22.188+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:27.021731+0000 front 2024-02-09T18:08:27.021779+0000 (oldest deadline 2024-02-09T18:08:57.523295+0000) 2024-02-09T18:09:22.393 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:22 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:22.188+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.624366+0000 front 2024-02-09T18:08:38.624336+0000 (oldest deadline 2024-02-09T18:09:02.724311+0000) 2024-02-09T18:09:22.394 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:22 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:22.188+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:48.025152+0000 front 2024-02-09T18:08:48.025186+0000 (oldest deadline 2024-02-09T18:09:13.925314+0000) 2024-02-09T18:09:22.394 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:22 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:22.188+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.826611+0000 front 2024-02-09T18:08:56.826637+0000 (oldest deadline 2024-02-09T18:09:17.926388+0000) 2024-02-09T18:09:22.655 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:22 smithi162 bash[176108]: ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-7 2024-02-09T18:09:22.672 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:22 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:22.618+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:22.672 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:22 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:22.618+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:22.672 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:22 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:22.618+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.425025+0000 front 2024-02-09T18:08:38.425098+0000 (oldest deadline 2024-02-09T18:09:06.025618+0000) 2024-02-09T18:09:22.672 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:22 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:22.618+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:46.026169+0000 front 2024-02-09T18:08:46.026259+0000 (oldest deadline 2024-02-09T18:09:12.426582+0000) 2024-02-09T18:09:22.672 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:22 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:22.390+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:20.919791+0000 front 2024-02-09T18:08:20.919903+0000 (oldest deadline 2024-02-09T18:08:46.219692+0000) 2024-02-09T18:09:22.672 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:22 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:22.390+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.520735+0000 front 2024-02-09T18:08:31.520846+0000 (oldest deadline 2024-02-09T18:08:56.820686+0000) 2024-02-09T18:09:22.673 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:22 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:22.390+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:39.622765+0000 front 2024-02-09T18:08:39.622813+0000 (oldest deadline 2024-02-09T18:09:05.522706+0000) 2024-02-09T18:09:23.412 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:23 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:23.036+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:09:23.413 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:23 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:23.036+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.694773+0000 front 2024-02-09T18:08:38.694887+0000 (oldest deadline 2024-02-09T18:09:03.394749+0000) 2024-02-09T18:09:23.413 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:23 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:23.036+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:47.495766+0000 front 2024-02-09T18:08:47.495922+0000 (oldest deadline 2024-02-09T18:09:12.195909+0000) 2024-02-09T18:09:23.413 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:23 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:23.036+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.197757+0000 front 2024-02-09T18:08:56.197854+0000 (oldest deadline 2024-02-09T18:09:20.897605+0000) 2024-02-09T18:09:23.413 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:23 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:23.209+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:09:23.413 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:23 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:23.209+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:27.021731+0000 front 2024-02-09T18:08:27.021779+0000 (oldest deadline 2024-02-09T18:08:57.523295+0000) 2024-02-09T18:09:23.413 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:23 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:23.209+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.624366+0000 front 2024-02-09T18:08:38.624336+0000 (oldest deadline 2024-02-09T18:09:02.724311+0000) 2024-02-09T18:09:23.413 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:23 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:23.209+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:48.025152+0000 front 2024-02-09T18:08:48.025186+0000 (oldest deadline 2024-02-09T18:09:13.925314+0000) 2024-02-09T18:09:23.413 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:23 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:23.209+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.826611+0000 front 2024-02-09T18:08:56.826637+0000 (oldest deadline 2024-02-09T18:09:17.926388+0000) 2024-02-09T18:09:23.672 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:23 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:23.582+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:23.672 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:23 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:23.582+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:23.672 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:23 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:23.582+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.425025+0000 front 2024-02-09T18:08:38.425098+0000 (oldest deadline 2024-02-09T18:09:06.025618+0000) 2024-02-09T18:09:23.672 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:23 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:23.582+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:46.026169+0000 front 2024-02-09T18:08:46.026259+0000 (oldest deadline 2024-02-09T18:09:12.426582+0000) 2024-02-09T18:09:23.672 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:23 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:23.411+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:20.919791+0000 front 2024-02-09T18:08:20.919903+0000 (oldest deadline 2024-02-09T18:08:46.219692+0000) 2024-02-09T18:09:23.672 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:23 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:23.411+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.520735+0000 front 2024-02-09T18:08:31.520846+0000 (oldest deadline 2024-02-09T18:08:56.820686+0000) 2024-02-09T18:09:23.672 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:23 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:23.411+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:39.622765+0000 front 2024-02-09T18:08:39.622813+0000 (oldest deadline 2024-02-09T18:09:05.522706+0000) 2024-02-09T18:09:23.673 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:23 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:23.411+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.624743+0000 front 2024-02-09T18:08:56.624808+0000 (oldest deadline 2024-02-09T18:09:22.524707+0000) 2024-02-09T18:09:24.149 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a69c870a-c76e-11ee-95b7-87774f69a715@osd.7.service' 2024-02-09T18:09:24.176 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:24 smithi162 systemd[1]: ceph-a69c870a-c76e-11ee-95b7-87774f69a715@osd.7.service: Succeeded. 2024-02-09T18:09:24.176 INFO:journalctl@ceph.osd.7.smithi162.stdout:Feb 09 18:09:24 smithi162 systemd[1]: Stopped Ceph osd.7 for a69c870a-c76e-11ee-95b7-87774f69a715. 2024-02-09T18:09:24.188 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T18:09:24.188 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-02-09T18:09:24.188 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-02-09T18:09:24.189 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-a69c870a-c76e-11ee-95b7-87774f69a715@osd.8 2024-02-09T18:09:24.422 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:24 smithi179 systemd[1]: Stopping Ceph osd.8 for a69c870a-c76e-11ee-95b7-87774f69a715... 2024-02-09T18:09:24.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:24 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:24.205+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:09:24.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:24 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:24.205+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:27.021731+0000 front 2024-02-09T18:08:27.021779+0000 (oldest deadline 2024-02-09T18:08:57.523295+0000) 2024-02-09T18:09:24.423 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:24 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:24.205+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.624366+0000 front 2024-02-09T18:08:38.624336+0000 (oldest deadline 2024-02-09T18:09:02.724311+0000) 2024-02-09T18:09:24.423 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:24 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:24.205+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:48.025152+0000 front 2024-02-09T18:08:48.025186+0000 (oldest deadline 2024-02-09T18:09:13.925314+0000) 2024-02-09T18:09:24.423 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:24 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:24.205+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.826611+0000 front 2024-02-09T18:08:56.826637+0000 (oldest deadline 2024-02-09T18:09:17.926388+0000) 2024-02-09T18:09:24.423 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:24 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:24.050+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:09:24.423 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:24 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:24.050+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.694773+0000 front 2024-02-09T18:08:38.694887+0000 (oldest deadline 2024-02-09T18:09:03.394749+0000) 2024-02-09T18:09:24.423 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:24 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:24.050+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:47.495766+0000 front 2024-02-09T18:08:47.495922+0000 (oldest deadline 2024-02-09T18:09:12.195909+0000) 2024-02-09T18:09:24.423 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:24 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:24.050+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.197757+0000 front 2024-02-09T18:08:56.197854+0000 (oldest deadline 2024-02-09T18:09:20.897605+0000) 2024-02-09T18:09:24.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:24 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:24.624+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:24.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:24 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:24.624+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:24.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:24 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:24.624+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.425025+0000 front 2024-02-09T18:08:38.425098+0000 (oldest deadline 2024-02-09T18:09:06.025618+0000) 2024-02-09T18:09:24.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:24 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:24.624+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:46.026169+0000 front 2024-02-09T18:08:46.026259+0000 (oldest deadline 2024-02-09T18:09:12.426582+0000) 2024-02-09T18:09:24.923 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:24 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:24.440+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:20.919791+0000 front 2024-02-09T18:08:20.919903+0000 (oldest deadline 2024-02-09T18:08:46.219692+0000) 2024-02-09T18:09:24.923 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:24 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:24.440+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.520735+0000 front 2024-02-09T18:08:31.520846+0000 (oldest deadline 2024-02-09T18:08:56.820686+0000) 2024-02-09T18:09:24.923 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:24 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:24.440+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:39.622765+0000 front 2024-02-09T18:08:39.622813+0000 (oldest deadline 2024-02-09T18:09:05.522706+0000) 2024-02-09T18:09:24.923 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:24 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:24.440+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.624743+0000 front 2024-02-09T18:08:56.624808+0000 (oldest deadline 2024-02-09T18:09:22.524707+0000) 2024-02-09T18:09:24.923 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:24 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:24.472+0000 7f2f14de7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T18:09:24.923 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:24 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:24.472+0000 7f2f14de7700 -1 osd.8 1050 *** Got signal Terminated *** 2024-02-09T18:09:24.923 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:24 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:24.472+0000 7f2f14de7700 -1 osd.8 1050 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-09T18:09:25.407 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:25 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:25.156+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:09:25.407 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:25 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:25.157+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:27.021731+0000 front 2024-02-09T18:08:27.021779+0000 (oldest deadline 2024-02-09T18:08:57.523295+0000) 2024-02-09T18:09:25.407 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:25 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:25.157+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.624366+0000 front 2024-02-09T18:08:38.624336+0000 (oldest deadline 2024-02-09T18:09:02.724311+0000) 2024-02-09T18:09:25.407 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:25 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:25.157+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:48.025152+0000 front 2024-02-09T18:08:48.025186+0000 (oldest deadline 2024-02-09T18:09:13.925314+0000) 2024-02-09T18:09:25.407 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:25 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:25.157+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.826611+0000 front 2024-02-09T18:08:56.826637+0000 (oldest deadline 2024-02-09T18:09:17.926388+0000) 2024-02-09T18:09:25.408 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:25 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:25.020+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:09:25.408 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:25 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:25.020+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.694773+0000 front 2024-02-09T18:08:38.694887+0000 (oldest deadline 2024-02-09T18:09:03.394749+0000) 2024-02-09T18:09:25.408 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:25 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:25.020+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:47.495766+0000 front 2024-02-09T18:08:47.495922+0000 (oldest deadline 2024-02-09T18:09:12.195909+0000) 2024-02-09T18:09:25.408 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:25 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:25.020+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.197757+0000 front 2024-02-09T18:08:56.197854+0000 (oldest deadline 2024-02-09T18:09:20.897605+0000) 2024-02-09T18:09:25.672 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:25 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:25.615+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:25.672 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:25 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:25.615+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:25.672 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:25 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:25.615+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.425025+0000 front 2024-02-09T18:08:38.425098+0000 (oldest deadline 2024-02-09T18:09:06.025618+0000) 2024-02-09T18:09:25.672 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:25 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:25.615+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:46.026169+0000 front 2024-02-09T18:08:46.026259+0000 (oldest deadline 2024-02-09T18:09:12.426582+0000) 2024-02-09T18:09:25.672 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:25 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:25.406+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:20.919791+0000 front 2024-02-09T18:08:20.919903+0000 (oldest deadline 2024-02-09T18:08:46.219692+0000) 2024-02-09T18:09:25.673 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:25 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:25.406+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.520735+0000 front 2024-02-09T18:08:31.520846+0000 (oldest deadline 2024-02-09T18:08:56.820686+0000) 2024-02-09T18:09:25.673 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:25 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:25.406+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:39.622765+0000 front 2024-02-09T18:08:39.622813+0000 (oldest deadline 2024-02-09T18:09:05.522706+0000) 2024-02-09T18:09:25.673 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:25 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:25.406+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.624743+0000 front 2024-02-09T18:08:56.624808+0000 (oldest deadline 2024-02-09T18:09:22.524707+0000) 2024-02-09T18:09:26.389 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:26 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:26.139+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:09:26.389 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:26 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:26.139+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:27.021731+0000 front 2024-02-09T18:08:27.021779+0000 (oldest deadline 2024-02-09T18:08:57.523295+0000) 2024-02-09T18:09:26.389 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:26 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:26.139+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.624366+0000 front 2024-02-09T18:08:38.624336+0000 (oldest deadline 2024-02-09T18:09:02.724311+0000) 2024-02-09T18:09:26.390 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:26 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:26.139+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:48.025152+0000 front 2024-02-09T18:08:48.025186+0000 (oldest deadline 2024-02-09T18:09:13.925314+0000) 2024-02-09T18:09:26.390 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:26 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:26.139+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.826611+0000 front 2024-02-09T18:08:56.826637+0000 (oldest deadline 2024-02-09T18:09:17.926388+0000) 2024-02-09T18:09:26.390 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:25 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:25.993+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:09:26.390 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:25 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:25.993+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.694773+0000 front 2024-02-09T18:08:38.694887+0000 (oldest deadline 2024-02-09T18:09:03.394749+0000) 2024-02-09T18:09:26.390 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:25 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:25.993+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:47.495766+0000 front 2024-02-09T18:08:47.495922+0000 (oldest deadline 2024-02-09T18:09:12.195909+0000) 2024-02-09T18:09:26.390 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:25 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:25.993+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.197757+0000 front 2024-02-09T18:08:56.197854+0000 (oldest deadline 2024-02-09T18:09:20.897605+0000) 2024-02-09T18:09:26.672 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:26 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:26.388+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:20.919791+0000 front 2024-02-09T18:08:20.919903+0000 (oldest deadline 2024-02-09T18:08:46.219692+0000) 2024-02-09T18:09:26.672 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:26 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:26.388+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.520735+0000 front 2024-02-09T18:08:31.520846+0000 (oldest deadline 2024-02-09T18:08:56.820686+0000) 2024-02-09T18:09:26.672 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:26 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:26.388+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:39.622765+0000 front 2024-02-09T18:08:39.622813+0000 (oldest deadline 2024-02-09T18:09:05.522706+0000) 2024-02-09T18:09:26.672 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:26 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:26.388+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.624743+0000 front 2024-02-09T18:08:56.624808+0000 (oldest deadline 2024-02-09T18:09:22.524707+0000) 2024-02-09T18:09:26.672 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:26 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:26.636+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:26.672 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:26 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:26.636+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:26.672 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:26 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:26.636+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.425025+0000 front 2024-02-09T18:08:38.425098+0000 (oldest deadline 2024-02-09T18:09:06.025618+0000) 2024-02-09T18:09:26.672 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:26 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:26.636+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:46.026169+0000 front 2024-02-09T18:08:46.026259+0000 (oldest deadline 2024-02-09T18:09:12.426582+0000) 2024-02-09T18:09:27.371 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:27 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:27.112+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:09:27.371 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:27 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:27.112+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:27.021731+0000 front 2024-02-09T18:08:27.021779+0000 (oldest deadline 2024-02-09T18:08:57.523295+0000) 2024-02-09T18:09:27.371 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:27 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:27.112+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.624366+0000 front 2024-02-09T18:08:38.624336+0000 (oldest deadline 2024-02-09T18:09:02.724311+0000) 2024-02-09T18:09:27.372 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:27 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:27.112+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:48.025152+0000 front 2024-02-09T18:08:48.025186+0000 (oldest deadline 2024-02-09T18:09:13.925314+0000) 2024-02-09T18:09:27.372 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:27 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:27.112+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.826611+0000 front 2024-02-09T18:08:56.826637+0000 (oldest deadline 2024-02-09T18:09:17.926388+0000) 2024-02-09T18:09:27.372 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:27 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:27.014+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:09:27.372 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:27 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:27.014+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.694773+0000 front 2024-02-09T18:08:38.694887+0000 (oldest deadline 2024-02-09T18:09:03.394749+0000) 2024-02-09T18:09:27.372 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:27 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:27.014+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:47.495766+0000 front 2024-02-09T18:08:47.495922+0000 (oldest deadline 2024-02-09T18:09:12.195909+0000) 2024-02-09T18:09:27.372 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:27 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:27.014+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.197757+0000 front 2024-02-09T18:08:56.197854+0000 (oldest deadline 2024-02-09T18:09:20.897605+0000) 2024-02-09T18:09:27.372 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:27 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:27.014+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:00.898361+0000 front 2024-02-09T18:09:00.898273+0000 (oldest deadline 2024-02-09T18:09:26.198031+0000) 2024-02-09T18:09:27.672 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:27 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:27.592+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:27.672 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:27 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:27.592+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:27.672 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:27 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:27.592+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.425025+0000 front 2024-02-09T18:08:38.425098+0000 (oldest deadline 2024-02-09T18:09:06.025618+0000) 2024-02-09T18:09:27.672 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:27 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:27.592+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:46.026169+0000 front 2024-02-09T18:08:46.026259+0000 (oldest deadline 2024-02-09T18:09:12.426582+0000) 2024-02-09T18:09:27.672 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:27 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:27.370+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:20.919791+0000 front 2024-02-09T18:08:20.919903+0000 (oldest deadline 2024-02-09T18:08:46.219692+0000) 2024-02-09T18:09:27.672 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:27 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:27.370+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.520735+0000 front 2024-02-09T18:08:31.520846+0000 (oldest deadline 2024-02-09T18:08:56.820686+0000) 2024-02-09T18:09:27.673 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:27 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:27.370+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:39.622765+0000 front 2024-02-09T18:08:39.622813+0000 (oldest deadline 2024-02-09T18:09:05.522706+0000) 2024-02-09T18:09:27.673 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:27 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:27.370+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.624743+0000 front 2024-02-09T18:08:56.624808+0000 (oldest deadline 2024-02-09T18:09:22.524707+0000) 2024-02-09T18:09:27.673 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:27 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:27.370+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:02.525242+0000 front 2024-02-09T18:09:02.525391+0000 (oldest deadline 2024-02-09T18:09:27.125856+0000) 2024-02-09T18:09:28.369 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:28 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:28.100+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:09:28.369 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:28 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:28.100+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:27.021731+0000 front 2024-02-09T18:08:27.021779+0000 (oldest deadline 2024-02-09T18:08:57.523295+0000) 2024-02-09T18:09:28.369 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:28 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:28.100+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.624366+0000 front 2024-02-09T18:08:38.624336+0000 (oldest deadline 2024-02-09T18:09:02.724311+0000) 2024-02-09T18:09:28.369 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:28 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:28.100+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:48.025152+0000 front 2024-02-09T18:08:48.025186+0000 (oldest deadline 2024-02-09T18:09:13.925314+0000) 2024-02-09T18:09:28.369 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:28 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:28.100+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.826611+0000 front 2024-02-09T18:08:56.826637+0000 (oldest deadline 2024-02-09T18:09:17.926388+0000) 2024-02-09T18:09:28.370 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:27 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:27.971+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:09:28.370 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:27 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:27.971+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.694773+0000 front 2024-02-09T18:08:38.694887+0000 (oldest deadline 2024-02-09T18:09:03.394749+0000) 2024-02-09T18:09:28.370 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:27 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:27.971+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:47.495766+0000 front 2024-02-09T18:08:47.495922+0000 (oldest deadline 2024-02-09T18:09:12.195909+0000) 2024-02-09T18:09:28.370 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:27 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:27.971+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.197757+0000 front 2024-02-09T18:08:56.197854+0000 (oldest deadline 2024-02-09T18:09:20.897605+0000) 2024-02-09T18:09:28.370 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:27 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:27.971+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:00.898361+0000 front 2024-02-09T18:09:00.898273+0000 (oldest deadline 2024-02-09T18:09:26.198031+0000) 2024-02-09T18:09:28.641 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:28 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:28.367+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:20.919791+0000 front 2024-02-09T18:08:20.919903+0000 (oldest deadline 2024-02-09T18:08:46.219692+0000) 2024-02-09T18:09:28.641 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:28 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:28.367+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.520735+0000 front 2024-02-09T18:08:31.520846+0000 (oldest deadline 2024-02-09T18:08:56.820686+0000) 2024-02-09T18:09:28.641 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:28 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:28.367+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:39.622765+0000 front 2024-02-09T18:08:39.622813+0000 (oldest deadline 2024-02-09T18:09:05.522706+0000) 2024-02-09T18:09:28.642 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:28 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:28.367+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.624743+0000 front 2024-02-09T18:08:56.624808+0000 (oldest deadline 2024-02-09T18:09:22.524707+0000) 2024-02-09T18:09:28.642 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:28 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:28.367+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:02.525242+0000 front 2024-02-09T18:09:02.525391+0000 (oldest deadline 2024-02-09T18:09:27.125856+0000) 2024-02-09T18:09:28.921 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:28 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:28.640+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:28.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:28 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:28.640+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:28.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:28 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:28.640+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.425025+0000 front 2024-02-09T18:08:38.425098+0000 (oldest deadline 2024-02-09T18:09:06.025618+0000) 2024-02-09T18:09:28.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:28 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:28.640+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:46.026169+0000 front 2024-02-09T18:08:46.026259+0000 (oldest deadline 2024-02-09T18:09:12.426582+0000) 2024-02-09T18:09:29.342 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:29 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:29.092+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:09:29.342 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:29 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:29.092+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:27.021731+0000 front 2024-02-09T18:08:27.021779+0000 (oldest deadline 2024-02-09T18:08:57.523295+0000) 2024-02-09T18:09:29.342 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:29 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:29.092+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.624366+0000 front 2024-02-09T18:08:38.624336+0000 (oldest deadline 2024-02-09T18:09:02.724311+0000) 2024-02-09T18:09:29.342 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:29 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:29.092+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:48.025152+0000 front 2024-02-09T18:08:48.025186+0000 (oldest deadline 2024-02-09T18:09:13.925314+0000) 2024-02-09T18:09:29.342 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:29 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:29.092+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.826611+0000 front 2024-02-09T18:08:56.826637+0000 (oldest deadline 2024-02-09T18:09:17.926388+0000) 2024-02-09T18:09:29.343 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:28 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:28.982+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:09:29.343 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:28 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:28.982+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.694773+0000 front 2024-02-09T18:08:38.694887+0000 (oldest deadline 2024-02-09T18:09:03.394749+0000) 2024-02-09T18:09:29.343 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:28 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:28.982+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:47.495766+0000 front 2024-02-09T18:08:47.495922+0000 (oldest deadline 2024-02-09T18:09:12.195909+0000) 2024-02-09T18:09:29.343 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:28 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:28.982+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.197757+0000 front 2024-02-09T18:08:56.197854+0000 (oldest deadline 2024-02-09T18:09:20.897605+0000) 2024-02-09T18:09:29.343 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:28 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:28.982+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:00.898361+0000 front 2024-02-09T18:09:00.898273+0000 (oldest deadline 2024-02-09T18:09:26.198031+0000) 2024-02-09T18:09:29.608 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:29 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:29.340+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:20.919791+0000 front 2024-02-09T18:08:20.919903+0000 (oldest deadline 2024-02-09T18:08:46.219692+0000) 2024-02-09T18:09:29.608 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:29 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:29.340+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:31.520735+0000 front 2024-02-09T18:08:31.520846+0000 (oldest deadline 2024-02-09T18:08:56.820686+0000) 2024-02-09T18:09:29.608 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:29 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:29.340+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:39.622765+0000 front 2024-02-09T18:08:39.622813+0000 (oldest deadline 2024-02-09T18:09:05.522706+0000) 2024-02-09T18:09:29.608 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:29 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:29.340+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.624743+0000 front 2024-02-09T18:08:56.624808+0000 (oldest deadline 2024-02-09T18:09:22.524707+0000) 2024-02-09T18:09:29.608 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:29 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8[87257]: 2024-02-09T18:09:29.340+0000 7f2f0d3da700 -1 osd.8 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:02.525242+0000 front 2024-02-09T18:09:02.525391+0000 (oldest deadline 2024-02-09T18:09:27.125856+0000) 2024-02-09T18:09:29.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:29 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:29.607+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:29.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:29 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:29.607+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:29.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:29 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:29.607+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.425025+0000 front 2024-02-09T18:08:38.425098+0000 (oldest deadline 2024-02-09T18:09:06.025618+0000) 2024-02-09T18:09:29.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:29 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:29.607+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:46.026169+0000 front 2024-02-09T18:08:46.026259+0000 (oldest deadline 2024-02-09T18:09:12.426582+0000) 2024-02-09T18:09:30.351 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:30 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:30.072+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:09:30.351 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:30 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:30.072+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:27.021731+0000 front 2024-02-09T18:08:27.021779+0000 (oldest deadline 2024-02-09T18:08:57.523295+0000) 2024-02-09T18:09:30.351 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:30 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:30.072+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.624366+0000 front 2024-02-09T18:08:38.624336+0000 (oldest deadline 2024-02-09T18:09:02.724311+0000) 2024-02-09T18:09:30.351 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:30 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:30.072+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:48.025152+0000 front 2024-02-09T18:08:48.025186+0000 (oldest deadline 2024-02-09T18:09:13.925314+0000) 2024-02-09T18:09:30.351 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:30 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:30.072+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.826611+0000 front 2024-02-09T18:08:56.826637+0000 (oldest deadline 2024-02-09T18:09:17.926388+0000) 2024-02-09T18:09:30.351 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:29 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:29.972+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:09:30.351 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:29 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:29.972+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.694773+0000 front 2024-02-09T18:08:38.694887+0000 (oldest deadline 2024-02-09T18:09:03.394749+0000) 2024-02-09T18:09:30.351 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:29 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:29.972+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:47.495766+0000 front 2024-02-09T18:08:47.495922+0000 (oldest deadline 2024-02-09T18:09:12.195909+0000) 2024-02-09T18:09:30.352 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:29 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:29.972+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.197757+0000 front 2024-02-09T18:08:56.197854+0000 (oldest deadline 2024-02-09T18:09:20.897605+0000) 2024-02-09T18:09:30.352 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:29 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:29.972+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:00.898361+0000 front 2024-02-09T18:09:00.898273+0000 (oldest deadline 2024-02-09T18:09:26.198031+0000) 2024-02-09T18:09:30.600 INFO:journalctl@ceph.osd.8.smithi179.stdout:Feb 09 18:09:30 smithi179 bash[174441]: ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-8 2024-02-09T18:09:30.601 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:30 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:30.599+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:30.601 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:30 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:30.599+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:30.601 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:30 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:30.599+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.425025+0000 front 2024-02-09T18:08:38.425098+0000 (oldest deadline 2024-02-09T18:09:06.025618+0000) 2024-02-09T18:09:30.921 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:30 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:30.599+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:46.026169+0000 front 2024-02-09T18:08:46.026259+0000 (oldest deadline 2024-02-09T18:09:12.426582+0000) 2024-02-09T18:09:30.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:30 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:30.599+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:04.529946+0000 front 2024-02-09T18:09:04.529983+0000 (oldest deadline 2024-02-09T18:09:30.429896+0000) 2024-02-09T18:09:31.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:30 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:30.989+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:09:31.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:30 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:30.989+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.694773+0000 front 2024-02-09T18:08:38.694887+0000 (oldest deadline 2024-02-09T18:09:03.394749+0000) 2024-02-09T18:09:31.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:30 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:30.989+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:47.495766+0000 front 2024-02-09T18:08:47.495922+0000 (oldest deadline 2024-02-09T18:09:12.195909+0000) 2024-02-09T18:09:31.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:30 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:30.989+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.197757+0000 front 2024-02-09T18:08:56.197854+0000 (oldest deadline 2024-02-09T18:09:20.897605+0000) 2024-02-09T18:09:31.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:30 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:30.989+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:00.898361+0000 front 2024-02-09T18:09:00.898273+0000 (oldest deadline 2024-02-09T18:09:26.198031+0000) 2024-02-09T18:09:31.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:31 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:31.102+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:09:31.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:31 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:31.102+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:27.021731+0000 front 2024-02-09T18:08:27.021779+0000 (oldest deadline 2024-02-09T18:08:57.523295+0000) 2024-02-09T18:09:31.423 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:31 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:31.102+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.624366+0000 front 2024-02-09T18:08:38.624336+0000 (oldest deadline 2024-02-09T18:09:02.724311+0000) 2024-02-09T18:09:31.423 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:31 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:31.102+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:48.025152+0000 front 2024-02-09T18:08:48.025186+0000 (oldest deadline 2024-02-09T18:09:13.925314+0000) 2024-02-09T18:09:31.423 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:31 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:31.102+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.826611+0000 front 2024-02-09T18:08:56.826637+0000 (oldest deadline 2024-02-09T18:09:17.926388+0000) 2024-02-09T18:09:31.423 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:31 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:31.102+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:00.227496+0000 front 2024-02-09T18:09:00.227536+0000 (oldest deadline 2024-02-09T18:09:30.827901+0000) 2024-02-09T18:09:31.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:31 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:31.591+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:31.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:31 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:31.591+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:31.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:31 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:31.591+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.425025+0000 front 2024-02-09T18:08:38.425098+0000 (oldest deadline 2024-02-09T18:09:06.025618+0000) 2024-02-09T18:09:31.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:31 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:31.591+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:46.026169+0000 front 2024-02-09T18:08:46.026259+0000 (oldest deadline 2024-02-09T18:09:12.426582+0000) 2024-02-09T18:09:31.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:31 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:31.591+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:04.529946+0000 front 2024-02-09T18:09:04.529983+0000 (oldest deadline 2024-02-09T18:09:30.429896+0000) 2024-02-09T18:09:32.411 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:31 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:31.967+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:09:32.411 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:31 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:31.967+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.694773+0000 front 2024-02-09T18:08:38.694887+0000 (oldest deadline 2024-02-09T18:09:03.394749+0000) 2024-02-09T18:09:32.411 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:31 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:31.968+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:47.495766+0000 front 2024-02-09T18:08:47.495922+0000 (oldest deadline 2024-02-09T18:09:12.195909+0000) 2024-02-09T18:09:32.411 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:31 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:31.968+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.197757+0000 front 2024-02-09T18:08:56.197854+0000 (oldest deadline 2024-02-09T18:09:20.897605+0000) 2024-02-09T18:09:32.411 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:31 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:31.968+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:00.898361+0000 front 2024-02-09T18:09:00.898273+0000 (oldest deadline 2024-02-09T18:09:26.198031+0000) 2024-02-09T18:09:32.411 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:32 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:32.110+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:09:32.412 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:32 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:32.110+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:27.021731+0000 front 2024-02-09T18:08:27.021779+0000 (oldest deadline 2024-02-09T18:08:57.523295+0000) 2024-02-09T18:09:32.412 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:32 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:32.110+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.624366+0000 front 2024-02-09T18:08:38.624336+0000 (oldest deadline 2024-02-09T18:09:02.724311+0000) 2024-02-09T18:09:32.412 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:32 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:32.110+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:48.025152+0000 front 2024-02-09T18:08:48.025186+0000 (oldest deadline 2024-02-09T18:09:13.925314+0000) 2024-02-09T18:09:32.412 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:32 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:32.110+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.826611+0000 front 2024-02-09T18:08:56.826637+0000 (oldest deadline 2024-02-09T18:09:17.926388+0000) 2024-02-09T18:09:32.412 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:32 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:32.110+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:00.227496+0000 front 2024-02-09T18:09:00.227536+0000 (oldest deadline 2024-02-09T18:09:30.827901+0000) 2024-02-09T18:09:32.419 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a69c870a-c76e-11ee-95b7-87774f69a715@osd.8.service' 2024-02-09T18:09:32.462 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T18:09:32.462 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-02-09T18:09:32.462 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-02-09T18:09:32.462 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-a69c870a-c76e-11ee-95b7-87774f69a715@osd.9 2024-02-09T18:09:32.672 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:32 smithi179 systemd[1]: Stopping Ceph osd.9 for a69c870a-c76e-11ee-95b7-87774f69a715... 2024-02-09T18:09:32.672 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:32 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:32.583+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:32.672 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:32 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:32.583+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:32.672 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:32 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:32.583+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.425025+0000 front 2024-02-09T18:08:38.425098+0000 (oldest deadline 2024-02-09T18:09:06.025618+0000) 2024-02-09T18:09:32.672 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:32 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:32.583+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:46.026169+0000 front 2024-02-09T18:08:46.026259+0000 (oldest deadline 2024-02-09T18:09:12.426582+0000) 2024-02-09T18:09:32.672 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:32 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:32.583+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:04.529946+0000 front 2024-02-09T18:09:04.529983+0000 (oldest deadline 2024-02-09T18:09:30.429896+0000) 2024-02-09T18:09:33.112 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:32 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:32.825+0000 7f4facb33700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T18:09:33.112 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:32 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:32.825+0000 7f4facb33700 -1 osd.9 1050 *** Got signal Terminated *** 2024-02-09T18:09:33.112 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:32 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:32.825+0000 7f4facb33700 -1 osd.9 1050 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-09T18:09:33.112 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:33 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:33.009+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:09:33.113 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:33 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:33.009+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.694773+0000 front 2024-02-09T18:08:38.694887+0000 (oldest deadline 2024-02-09T18:09:03.394749+0000) 2024-02-09T18:09:33.113 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:33 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:33.009+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:47.495766+0000 front 2024-02-09T18:08:47.495922+0000 (oldest deadline 2024-02-09T18:09:12.195909+0000) 2024-02-09T18:09:33.113 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:33 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:33.009+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.197757+0000 front 2024-02-09T18:08:56.197854+0000 (oldest deadline 2024-02-09T18:09:20.897605+0000) 2024-02-09T18:09:33.113 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:33 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:33.009+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:00.898361+0000 front 2024-02-09T18:09:00.898273+0000 (oldest deadline 2024-02-09T18:09:26.198031+0000) 2024-02-09T18:09:33.421 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:33 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:33.111+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:09:33.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:33 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:33.111+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:27.021731+0000 front 2024-02-09T18:08:27.021779+0000 (oldest deadline 2024-02-09T18:08:57.523295+0000) 2024-02-09T18:09:33.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:33 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:33.111+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.624366+0000 front 2024-02-09T18:08:38.624336+0000 (oldest deadline 2024-02-09T18:09:02.724311+0000) 2024-02-09T18:09:33.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:33 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:33.111+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:48.025152+0000 front 2024-02-09T18:08:48.025186+0000 (oldest deadline 2024-02-09T18:09:13.925314+0000) 2024-02-09T18:09:33.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:33 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:33.111+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.826611+0000 front 2024-02-09T18:08:56.826637+0000 (oldest deadline 2024-02-09T18:09:17.926388+0000) 2024-02-09T18:09:33.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:33 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:33.111+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:00.227496+0000 front 2024-02-09T18:09:00.227536+0000 (oldest deadline 2024-02-09T18:09:30.827901+0000) 2024-02-09T18:09:33.921 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:33 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:33.609+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:33.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:33 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:33.609+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:33.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:33 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:33.609+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.425025+0000 front 2024-02-09T18:08:38.425098+0000 (oldest deadline 2024-02-09T18:09:06.025618+0000) 2024-02-09T18:09:33.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:33 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:33.609+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:46.026169+0000 front 2024-02-09T18:08:46.026259+0000 (oldest deadline 2024-02-09T18:09:12.426582+0000) 2024-02-09T18:09:33.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:33 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:33.609+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:04.529946+0000 front 2024-02-09T18:09:04.529983+0000 (oldest deadline 2024-02-09T18:09:30.429896+0000) 2024-02-09T18:09:34.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:33 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:33.971+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:09:34.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:33 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:33.971+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.694773+0000 front 2024-02-09T18:08:38.694887+0000 (oldest deadline 2024-02-09T18:09:03.394749+0000) 2024-02-09T18:09:34.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:33 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:33.971+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:47.495766+0000 front 2024-02-09T18:08:47.495922+0000 (oldest deadline 2024-02-09T18:09:12.195909+0000) 2024-02-09T18:09:34.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:33 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:33.971+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.197757+0000 front 2024-02-09T18:08:56.197854+0000 (oldest deadline 2024-02-09T18:09:20.897605+0000) 2024-02-09T18:09:34.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:33 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:33.971+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:00.898361+0000 front 2024-02-09T18:09:00.898273+0000 (oldest deadline 2024-02-09T18:09:26.198031+0000) 2024-02-09T18:09:34.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:34 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:34.105+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:09:34.423 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:34 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:34.105+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:27.021731+0000 front 2024-02-09T18:08:27.021779+0000 (oldest deadline 2024-02-09T18:08:57.523295+0000) 2024-02-09T18:09:34.423 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:34 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:34.105+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.624366+0000 front 2024-02-09T18:08:38.624336+0000 (oldest deadline 2024-02-09T18:09:02.724311+0000) 2024-02-09T18:09:34.423 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:34 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:34.105+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:48.025152+0000 front 2024-02-09T18:08:48.025186+0000 (oldest deadline 2024-02-09T18:09:13.925314+0000) 2024-02-09T18:09:34.423 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:34 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:34.105+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.826611+0000 front 2024-02-09T18:08:56.826637+0000 (oldest deadline 2024-02-09T18:09:17.926388+0000) 2024-02-09T18:09:34.423 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:34 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:34.105+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:00.227496+0000 front 2024-02-09T18:09:00.227536+0000 (oldest deadline 2024-02-09T18:09:30.827901+0000) 2024-02-09T18:09:34.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:34 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:34.594+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:34.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:34 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:34.594+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:34.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:34 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:34.594+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.425025+0000 front 2024-02-09T18:08:38.425098+0000 (oldest deadline 2024-02-09T18:09:06.025618+0000) 2024-02-09T18:09:34.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:34 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:34.594+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:46.026169+0000 front 2024-02-09T18:08:46.026259+0000 (oldest deadline 2024-02-09T18:09:12.426582+0000) 2024-02-09T18:09:34.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:34 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:34.594+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:04.529946+0000 front 2024-02-09T18:09:04.529983+0000 (oldest deadline 2024-02-09T18:09:30.429896+0000) 2024-02-09T18:09:35.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:35 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:35.124+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:09:35.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:35 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:35.124+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:27.021731+0000 front 2024-02-09T18:08:27.021779+0000 (oldest deadline 2024-02-09T18:08:57.523295+0000) 2024-02-09T18:09:35.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:35 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:35.124+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.624366+0000 front 2024-02-09T18:08:38.624336+0000 (oldest deadline 2024-02-09T18:09:02.724311+0000) 2024-02-09T18:09:35.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:35 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:35.124+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:48.025152+0000 front 2024-02-09T18:08:48.025186+0000 (oldest deadline 2024-02-09T18:09:13.925314+0000) 2024-02-09T18:09:35.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:35 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:35.124+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.826611+0000 front 2024-02-09T18:08:56.826637+0000 (oldest deadline 2024-02-09T18:09:17.926388+0000) 2024-02-09T18:09:35.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:35 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:35.124+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:00.227496+0000 front 2024-02-09T18:09:00.227536+0000 (oldest deadline 2024-02-09T18:09:30.827901+0000) 2024-02-09T18:09:35.423 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:34 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:34.944+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:09:35.423 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:34 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:34.944+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.694773+0000 front 2024-02-09T18:08:38.694887+0000 (oldest deadline 2024-02-09T18:09:03.394749+0000) 2024-02-09T18:09:35.423 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:34 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:34.944+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:47.495766+0000 front 2024-02-09T18:08:47.495922+0000 (oldest deadline 2024-02-09T18:09:12.195909+0000) 2024-02-09T18:09:35.423 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:34 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:34.944+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.197757+0000 front 2024-02-09T18:08:56.197854+0000 (oldest deadline 2024-02-09T18:09:20.897605+0000) 2024-02-09T18:09:35.423 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:34 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:34.944+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:00.898361+0000 front 2024-02-09T18:09:00.898273+0000 (oldest deadline 2024-02-09T18:09:26.198031+0000) 2024-02-09T18:09:35.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:35 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:35.562+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:35.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:35 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:35.562+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:35.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:35 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:35.562+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.425025+0000 front 2024-02-09T18:08:38.425098+0000 (oldest deadline 2024-02-09T18:09:06.025618+0000) 2024-02-09T18:09:35.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:35 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:35.562+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:46.026169+0000 front 2024-02-09T18:08:46.026259+0000 (oldest deadline 2024-02-09T18:09:12.426582+0000) 2024-02-09T18:09:35.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:35 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:35.562+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:04.529946+0000 front 2024-02-09T18:09:04.529983+0000 (oldest deadline 2024-02-09T18:09:30.429896+0000) 2024-02-09T18:09:35.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:35 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:35.562+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-02-09T18:09:10.430677+0000 front 2024-02-09T18:09:10.430649+0000 (oldest deadline 2024-02-09T18:09:35.130523+0000) 2024-02-09T18:09:36.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:36 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:36.105+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:09:36.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:36 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:36.105+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:27.021731+0000 front 2024-02-09T18:08:27.021779+0000 (oldest deadline 2024-02-09T18:08:57.523295+0000) 2024-02-09T18:09:36.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:36 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:36.105+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.624366+0000 front 2024-02-09T18:08:38.624336+0000 (oldest deadline 2024-02-09T18:09:02.724311+0000) 2024-02-09T18:09:36.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:36 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:36.105+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:48.025152+0000 front 2024-02-09T18:08:48.025186+0000 (oldest deadline 2024-02-09T18:09:13.925314+0000) 2024-02-09T18:09:36.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:36 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:36.105+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.826611+0000 front 2024-02-09T18:08:56.826637+0000 (oldest deadline 2024-02-09T18:09:17.926388+0000) 2024-02-09T18:09:36.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:36 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:36.105+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:00.227496+0000 front 2024-02-09T18:09:00.227536+0000 (oldest deadline 2024-02-09T18:09:30.827901+0000) 2024-02-09T18:09:36.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:35 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:35.968+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:09:36.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:35 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:35.968+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.694773+0000 front 2024-02-09T18:08:38.694887+0000 (oldest deadline 2024-02-09T18:09:03.394749+0000) 2024-02-09T18:09:36.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:35 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:35.968+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:47.495766+0000 front 2024-02-09T18:08:47.495922+0000 (oldest deadline 2024-02-09T18:09:12.195909+0000) 2024-02-09T18:09:36.423 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:35 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:35.968+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.197757+0000 front 2024-02-09T18:08:56.197854+0000 (oldest deadline 2024-02-09T18:09:20.897605+0000) 2024-02-09T18:09:36.423 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:35 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:35.968+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:00.898361+0000 front 2024-02-09T18:09:00.898273+0000 (oldest deadline 2024-02-09T18:09:26.198031+0000) 2024-02-09T18:09:36.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:36 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:36.581+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:36.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:36 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:36.581+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:36.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:36 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:36.581+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.425025+0000 front 2024-02-09T18:08:38.425098+0000 (oldest deadline 2024-02-09T18:09:06.025618+0000) 2024-02-09T18:09:36.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:36 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:36.581+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:46.026169+0000 front 2024-02-09T18:08:46.026259+0000 (oldest deadline 2024-02-09T18:09:12.426582+0000) 2024-02-09T18:09:36.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:36 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:36.581+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:04.529946+0000 front 2024-02-09T18:09:04.529983+0000 (oldest deadline 2024-02-09T18:09:30.429896+0000) 2024-02-09T18:09:36.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:36 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:36.581+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-02-09T18:09:10.430677+0000 front 2024-02-09T18:09:10.430649+0000 (oldest deadline 2024-02-09T18:09:35.130523+0000) 2024-02-09T18:09:37.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:37 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:37.100+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:23.521392+0000 front 2024-02-09T18:08:23.521531+0000 (oldest deadline 2024-02-09T18:08:47.021339+0000) 2024-02-09T18:09:37.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:37 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:37.100+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:27.021731+0000 front 2024-02-09T18:08:27.021779+0000 (oldest deadline 2024-02-09T18:08:57.523295+0000) 2024-02-09T18:09:37.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:37 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:37.100+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.624366+0000 front 2024-02-09T18:08:38.624336+0000 (oldest deadline 2024-02-09T18:09:02.724311+0000) 2024-02-09T18:09:37.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:37 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:37.100+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:48.025152+0000 front 2024-02-09T18:08:48.025186+0000 (oldest deadline 2024-02-09T18:09:13.925314+0000) 2024-02-09T18:09:37.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:37 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:37.100+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.826611+0000 front 2024-02-09T18:08:56.826637+0000 (oldest deadline 2024-02-09T18:09:17.926388+0000) 2024-02-09T18:09:37.422 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:37 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9[57402]: 2024-02-09T18:09:37.100+0000 7f4fa5126700 -1 osd.9 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:00.227496+0000 front 2024-02-09T18:09:00.227536+0000 (oldest deadline 2024-02-09T18:09:30.827901+0000) 2024-02-09T18:09:37.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:36 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:36.941+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:09:37.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:36 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:36.941+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.694773+0000 front 2024-02-09T18:08:38.694887+0000 (oldest deadline 2024-02-09T18:09:03.394749+0000) 2024-02-09T18:09:37.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:36 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:36.941+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:47.495766+0000 front 2024-02-09T18:08:47.495922+0000 (oldest deadline 2024-02-09T18:09:12.195909+0000) 2024-02-09T18:09:37.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:36 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:36.941+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.197757+0000 front 2024-02-09T18:08:56.197854+0000 (oldest deadline 2024-02-09T18:09:20.897605+0000) 2024-02-09T18:09:37.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:36 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:36.941+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:00.898361+0000 front 2024-02-09T18:09:00.898273+0000 (oldest deadline 2024-02-09T18:09:26.198031+0000) 2024-02-09T18:09:37.423 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:36 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:36.941+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-02-09T18:09:11.999842+0000 front 2024-02-09T18:09:11.999868+0000 (oldest deadline 2024-02-09T18:09:36.699581+0000) 2024-02-09T18:09:37.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:37 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:37.542+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:37.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:37 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:37.542+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:37.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:37 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:37.542+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.425025+0000 front 2024-02-09T18:08:38.425098+0000 (oldest deadline 2024-02-09T18:09:06.025618+0000) 2024-02-09T18:09:37.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:37 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:37.542+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:46.026169+0000 front 2024-02-09T18:08:46.026259+0000 (oldest deadline 2024-02-09T18:09:12.426582+0000) 2024-02-09T18:09:37.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:37 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:37.542+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:04.529946+0000 front 2024-02-09T18:09:04.529983+0000 (oldest deadline 2024-02-09T18:09:30.429896+0000) 2024-02-09T18:09:37.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:37 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:37.542+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-02-09T18:09:10.430677+0000 front 2024-02-09T18:09:10.430649+0000 (oldest deadline 2024-02-09T18:09:35.130523+0000) 2024-02-09T18:09:38.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:37 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:37.952+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:09:38.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:37 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:37.952+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.694773+0000 front 2024-02-09T18:08:38.694887+0000 (oldest deadline 2024-02-09T18:09:03.394749+0000) 2024-02-09T18:09:38.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:37 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:37.952+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:47.495766+0000 front 2024-02-09T18:08:47.495922+0000 (oldest deadline 2024-02-09T18:09:12.195909+0000) 2024-02-09T18:09:38.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:37 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:37.952+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.197757+0000 front 2024-02-09T18:08:56.197854+0000 (oldest deadline 2024-02-09T18:09:20.897605+0000) 2024-02-09T18:09:38.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:37 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:37.952+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:00.898361+0000 front 2024-02-09T18:09:00.898273+0000 (oldest deadline 2024-02-09T18:09:26.198031+0000) 2024-02-09T18:09:38.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:37 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:37.952+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-02-09T18:09:11.999842+0000 front 2024-02-09T18:09:11.999868+0000 (oldest deadline 2024-02-09T18:09:36.699581+0000) 2024-02-09T18:09:38.922 INFO:journalctl@ceph.osd.9.smithi179.stdout:Feb 09 18:09:38 smithi179 bash[174704]: ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-9 2024-02-09T18:09:38.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:38 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:38.552+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:38.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:38 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:38.552+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:38.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:38 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:38.552+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.425025+0000 front 2024-02-09T18:08:38.425098+0000 (oldest deadline 2024-02-09T18:09:06.025618+0000) 2024-02-09T18:09:38.923 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:38 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:38.552+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:46.026169+0000 front 2024-02-09T18:08:46.026259+0000 (oldest deadline 2024-02-09T18:09:12.426582+0000) 2024-02-09T18:09:38.923 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:38 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:38.552+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:04.529946+0000 front 2024-02-09T18:09:04.529983+0000 (oldest deadline 2024-02-09T18:09:30.429896+0000) 2024-02-09T18:09:38.923 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:38 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:38.552+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-02-09T18:09:10.430677+0000 front 2024-02-09T18:09:10.430649+0000 (oldest deadline 2024-02-09T18:09:35.130523+0000) 2024-02-09T18:09:39.421 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:38 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:38.935+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:09:39.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:38 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:38.935+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.694773+0000 front 2024-02-09T18:08:38.694887+0000 (oldest deadline 2024-02-09T18:09:03.394749+0000) 2024-02-09T18:09:39.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:38 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:38.935+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:47.495766+0000 front 2024-02-09T18:08:47.495922+0000 (oldest deadline 2024-02-09T18:09:12.195909+0000) 2024-02-09T18:09:39.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:38 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:38.935+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.197757+0000 front 2024-02-09T18:08:56.197854+0000 (oldest deadline 2024-02-09T18:09:20.897605+0000) 2024-02-09T18:09:39.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:38 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:38.935+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:00.898361+0000 front 2024-02-09T18:09:00.898273+0000 (oldest deadline 2024-02-09T18:09:26.198031+0000) 2024-02-09T18:09:39.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:38 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:38.935+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-02-09T18:09:11.999842+0000 front 2024-02-09T18:09:11.999868+0000 (oldest deadline 2024-02-09T18:09:36.699581+0000) 2024-02-09T18:09:39.905 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:39 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:39.527+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:39.906 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:39 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:39.527+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:39.906 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:39 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:39.527+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.425025+0000 front 2024-02-09T18:08:38.425098+0000 (oldest deadline 2024-02-09T18:09:06.025618+0000) 2024-02-09T18:09:39.906 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:39 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:39.527+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:46.026169+0000 front 2024-02-09T18:08:46.026259+0000 (oldest deadline 2024-02-09T18:09:12.426582+0000) 2024-02-09T18:09:39.906 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:39 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:39.527+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:04.529946+0000 front 2024-02-09T18:09:04.529983+0000 (oldest deadline 2024-02-09T18:09:30.429896+0000) 2024-02-09T18:09:39.906 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:39 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:39.527+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-02-09T18:09:10.430677+0000 front 2024-02-09T18:09:10.430649+0000 (oldest deadline 2024-02-09T18:09:35.130523+0000) 2024-02-09T18:09:40.172 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:39 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:39.903+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:09:40.172 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:39 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:39.903+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.694773+0000 front 2024-02-09T18:08:38.694887+0000 (oldest deadline 2024-02-09T18:09:03.394749+0000) 2024-02-09T18:09:40.172 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:39 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:39.903+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:47.495766+0000 front 2024-02-09T18:08:47.495922+0000 (oldest deadline 2024-02-09T18:09:12.195909+0000) 2024-02-09T18:09:40.172 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:39 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:39.903+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.197757+0000 front 2024-02-09T18:08:56.197854+0000 (oldest deadline 2024-02-09T18:09:20.897605+0000) 2024-02-09T18:09:40.172 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:39 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:39.903+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:00.898361+0000 front 2024-02-09T18:09:00.898273+0000 (oldest deadline 2024-02-09T18:09:26.198031+0000) 2024-02-09T18:09:40.172 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:39 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:39.903+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-02-09T18:09:11.999842+0000 front 2024-02-09T18:09:11.999868+0000 (oldest deadline 2024-02-09T18:09:36.699581+0000) 2024-02-09T18:09:40.646 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:40 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:40.537+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:40.647 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:40 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:40.537+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:40.647 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:40 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:40.537+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.425025+0000 front 2024-02-09T18:08:38.425098+0000 (oldest deadline 2024-02-09T18:09:06.025618+0000) 2024-02-09T18:09:40.647 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:40 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:40.537+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:46.026169+0000 front 2024-02-09T18:08:46.026259+0000 (oldest deadline 2024-02-09T18:09:12.426582+0000) 2024-02-09T18:09:40.647 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:40 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:40.537+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:04.529946+0000 front 2024-02-09T18:09:04.529983+0000 (oldest deadline 2024-02-09T18:09:30.429896+0000) 2024-02-09T18:09:40.647 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:40 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:40.537+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-02-09T18:09:10.430677+0000 front 2024-02-09T18:09:10.430649+0000 (oldest deadline 2024-02-09T18:09:35.130523+0000) 2024-02-09T18:09:40.654 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a69c870a-c76e-11ee-95b7-87774f69a715@osd.9.service' 2024-02-09T18:09:40.694 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T18:09:40.695 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-02-09T18:09:40.695 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-02-09T18:09:40.695 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-a69c870a-c76e-11ee-95b7-87774f69a715@osd.10 2024-02-09T18:09:40.921 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:40 smithi179 systemd[1]: Stopping Ceph osd.10 for a69c870a-c76e-11ee-95b7-87774f69a715... 2024-02-09T18:09:41.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:40 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:40.930+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:09:41.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:40 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:40.930+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.694773+0000 front 2024-02-09T18:08:38.694887+0000 (oldest deadline 2024-02-09T18:09:03.394749+0000) 2024-02-09T18:09:41.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:40 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:40.930+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:47.495766+0000 front 2024-02-09T18:08:47.495922+0000 (oldest deadline 2024-02-09T18:09:12.195909+0000) 2024-02-09T18:09:41.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:40 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:40.930+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.197757+0000 front 2024-02-09T18:08:56.197854+0000 (oldest deadline 2024-02-09T18:09:20.897605+0000) 2024-02-09T18:09:41.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:40 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:40.930+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:00.898361+0000 front 2024-02-09T18:09:00.898273+0000 (oldest deadline 2024-02-09T18:09:26.198031+0000) 2024-02-09T18:09:41.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:40 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:40.930+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-02-09T18:09:11.999842+0000 front 2024-02-09T18:09:11.999868+0000 (oldest deadline 2024-02-09T18:09:36.699581+0000) 2024-02-09T18:09:41.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:41 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:41.007+0000 7f4350621700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T18:09:41.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:41 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:41.007+0000 7f4350621700 -1 osd.10 1050 *** Got signal Terminated *** 2024-02-09T18:09:41.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:41 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:41.007+0000 7f4350621700 -1 osd.10 1050 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-09T18:09:41.921 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:41 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:41.567+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:41.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:41 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:41.567+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:41.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:41 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:41.567+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.425025+0000 front 2024-02-09T18:08:38.425098+0000 (oldest deadline 2024-02-09T18:09:06.025618+0000) 2024-02-09T18:09:41.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:41 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:41.567+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:46.026169+0000 front 2024-02-09T18:08:46.026259+0000 (oldest deadline 2024-02-09T18:09:12.426582+0000) 2024-02-09T18:09:41.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:41 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:41.567+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:04.529946+0000 front 2024-02-09T18:09:04.529983+0000 (oldest deadline 2024-02-09T18:09:30.429896+0000) 2024-02-09T18:09:41.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:41 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:41.567+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-02-09T18:09:10.430677+0000 front 2024-02-09T18:09:10.430649+0000 (oldest deadline 2024-02-09T18:09:35.130523+0000) 2024-02-09T18:09:42.421 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:41 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:41.966+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:09:42.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:41 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:41.966+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.694773+0000 front 2024-02-09T18:08:38.694887+0000 (oldest deadline 2024-02-09T18:09:03.394749+0000) 2024-02-09T18:09:42.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:41 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:41.966+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:47.495766+0000 front 2024-02-09T18:08:47.495922+0000 (oldest deadline 2024-02-09T18:09:12.195909+0000) 2024-02-09T18:09:42.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:41 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:41.966+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.197757+0000 front 2024-02-09T18:08:56.197854+0000 (oldest deadline 2024-02-09T18:09:20.897605+0000) 2024-02-09T18:09:42.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:41 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:41.966+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:00.898361+0000 front 2024-02-09T18:09:00.898273+0000 (oldest deadline 2024-02-09T18:09:26.198031+0000) 2024-02-09T18:09:42.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:41 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:41.966+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-02-09T18:09:11.999842+0000 front 2024-02-09T18:09:11.999868+0000 (oldest deadline 2024-02-09T18:09:36.699581+0000) 2024-02-09T18:09:42.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:42 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:42.613+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:42.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:42 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:42.613+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:42.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:42 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:42.614+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.425025+0000 front 2024-02-09T18:08:38.425098+0000 (oldest deadline 2024-02-09T18:09:06.025618+0000) 2024-02-09T18:09:42.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:42 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:42.614+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:46.026169+0000 front 2024-02-09T18:08:46.026259+0000 (oldest deadline 2024-02-09T18:09:12.426582+0000) 2024-02-09T18:09:42.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:42 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:42.614+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:04.529946+0000 front 2024-02-09T18:09:04.529983+0000 (oldest deadline 2024-02-09T18:09:30.429896+0000) 2024-02-09T18:09:42.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:42 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:42.614+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-02-09T18:09:10.430677+0000 front 2024-02-09T18:09:10.430649+0000 (oldest deadline 2024-02-09T18:09:35.130523+0000) 2024-02-09T18:09:42.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:42 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:42.614+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6828 osd.7 since back 2024-02-09T18:09:16.831538+0000 front 2024-02-09T18:09:16.831519+0000 (oldest deadline 2024-02-09T18:09:42.131451+0000) 2024-02-09T18:09:43.421 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:42 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:42.923+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:09:43.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:42 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:42.923+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.694773+0000 front 2024-02-09T18:08:38.694887+0000 (oldest deadline 2024-02-09T18:09:03.394749+0000) 2024-02-09T18:09:43.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:42 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:42.923+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:47.495766+0000 front 2024-02-09T18:08:47.495922+0000 (oldest deadline 2024-02-09T18:09:12.195909+0000) 2024-02-09T18:09:43.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:42 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:42.923+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.197757+0000 front 2024-02-09T18:08:56.197854+0000 (oldest deadline 2024-02-09T18:09:20.897605+0000) 2024-02-09T18:09:43.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:42 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:42.923+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:00.898361+0000 front 2024-02-09T18:09:00.898273+0000 (oldest deadline 2024-02-09T18:09:26.198031+0000) 2024-02-09T18:09:43.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:42 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:42.923+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-02-09T18:09:11.999842+0000 front 2024-02-09T18:09:11.999868+0000 (oldest deadline 2024-02-09T18:09:36.699581+0000) 2024-02-09T18:09:43.909 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:43 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:43.626+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:43.910 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:43 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:43.626+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:43.910 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:43 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:43.626+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.425025+0000 front 2024-02-09T18:08:38.425098+0000 (oldest deadline 2024-02-09T18:09:06.025618+0000) 2024-02-09T18:09:43.910 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:43 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:43.626+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:46.026169+0000 front 2024-02-09T18:08:46.026259+0000 (oldest deadline 2024-02-09T18:09:12.426582+0000) 2024-02-09T18:09:43.910 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:43 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:43.626+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:04.529946+0000 front 2024-02-09T18:09:04.529983+0000 (oldest deadline 2024-02-09T18:09:30.429896+0000) 2024-02-09T18:09:43.910 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:43 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:43.626+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-02-09T18:09:10.430677+0000 front 2024-02-09T18:09:10.430649+0000 (oldest deadline 2024-02-09T18:09:35.130523+0000) 2024-02-09T18:09:43.910 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:43 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:43.626+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6828 osd.7 since back 2024-02-09T18:09:16.831538+0000 front 2024-02-09T18:09:16.831519+0000 (oldest deadline 2024-02-09T18:09:42.131451+0000) 2024-02-09T18:09:44.171 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:43 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:43.907+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:09:44.172 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:43 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:43.907+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.694773+0000 front 2024-02-09T18:08:38.694887+0000 (oldest deadline 2024-02-09T18:09:03.394749+0000) 2024-02-09T18:09:44.172 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:43 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:43.907+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:47.495766+0000 front 2024-02-09T18:08:47.495922+0000 (oldest deadline 2024-02-09T18:09:12.195909+0000) 2024-02-09T18:09:44.172 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:43 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:43.907+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.197757+0000 front 2024-02-09T18:08:56.197854+0000 (oldest deadline 2024-02-09T18:09:20.897605+0000) 2024-02-09T18:09:44.172 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:43 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:43.907+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:00.898361+0000 front 2024-02-09T18:09:00.898273+0000 (oldest deadline 2024-02-09T18:09:26.198031+0000) 2024-02-09T18:09:44.172 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:43 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:43.907+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-02-09T18:09:11.999842+0000 front 2024-02-09T18:09:11.999868+0000 (oldest deadline 2024-02-09T18:09:36.699581+0000) 2024-02-09T18:09:44.932 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:44 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:44.674+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:44.932 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:44 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:44.674+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:44.932 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:44 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:44.674+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.425025+0000 front 2024-02-09T18:08:38.425098+0000 (oldest deadline 2024-02-09T18:09:06.025618+0000) 2024-02-09T18:09:44.932 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:44 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:44.674+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:46.026169+0000 front 2024-02-09T18:08:46.026259+0000 (oldest deadline 2024-02-09T18:09:12.426582+0000) 2024-02-09T18:09:44.932 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:44 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:44.674+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:04.529946+0000 front 2024-02-09T18:09:04.529983+0000 (oldest deadline 2024-02-09T18:09:30.429896+0000) 2024-02-09T18:09:44.932 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:44 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:44.675+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-02-09T18:09:10.430677+0000 front 2024-02-09T18:09:10.430649+0000 (oldest deadline 2024-02-09T18:09:35.130523+0000) 2024-02-09T18:09:44.932 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:44 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:44.675+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6828 osd.7 since back 2024-02-09T18:09:16.831538+0000 front 2024-02-09T18:09:16.831519+0000 (oldest deadline 2024-02-09T18:09:42.131451+0000) 2024-02-09T18:09:45.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:44 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:44.930+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:09:45.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:44 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:44.930+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.694773+0000 front 2024-02-09T18:08:38.694887+0000 (oldest deadline 2024-02-09T18:09:03.394749+0000) 2024-02-09T18:09:45.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:44 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:44.930+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:47.495766+0000 front 2024-02-09T18:08:47.495922+0000 (oldest deadline 2024-02-09T18:09:12.195909+0000) 2024-02-09T18:09:45.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:44 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:44.930+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.197757+0000 front 2024-02-09T18:08:56.197854+0000 (oldest deadline 2024-02-09T18:09:20.897605+0000) 2024-02-09T18:09:45.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:44 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:44.930+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:00.898361+0000 front 2024-02-09T18:09:00.898273+0000 (oldest deadline 2024-02-09T18:09:26.198031+0000) 2024-02-09T18:09:45.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:44 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:44.930+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-02-09T18:09:11.999842+0000 front 2024-02-09T18:09:11.999868+0000 (oldest deadline 2024-02-09T18:09:36.699581+0000) 2024-02-09T18:09:45.422 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:44 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:44.930+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6828 osd.7 since back 2024-02-09T18:09:18.400760+0000 front 2024-02-09T18:09:18.400836+0000 (oldest deadline 2024-02-09T18:09:44.200987+0000) 2024-02-09T18:09:46.172 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:45 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:45.913+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:21.490774+0000 front 2024-02-09T18:08:21.490653+0000 (oldest deadline 2024-02-09T18:08:44.390666+0000) 2024-02-09T18:09:46.172 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:45 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:45.913+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.694773+0000 front 2024-02-09T18:08:38.694887+0000 (oldest deadline 2024-02-09T18:09:03.394749+0000) 2024-02-09T18:09:46.172 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:45 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:45.913+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:47.495766+0000 front 2024-02-09T18:08:47.495922+0000 (oldest deadline 2024-02-09T18:09:12.195909+0000) 2024-02-09T18:09:46.172 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:45 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:45.913+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-02-09T18:08:56.197757+0000 front 2024-02-09T18:08:56.197854+0000 (oldest deadline 2024-02-09T18:09:20.897605+0000) 2024-02-09T18:09:46.172 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:45 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:45.913+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:00.898361+0000 front 2024-02-09T18:09:00.898273+0000 (oldest deadline 2024-02-09T18:09:26.198031+0000) 2024-02-09T18:09:46.172 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:45 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:45.913+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-02-09T18:09:11.999842+0000 front 2024-02-09T18:09:11.999868+0000 (oldest deadline 2024-02-09T18:09:36.699581+0000) 2024-02-09T18:09:46.172 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:45 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10[64567]: 2024-02-09T18:09:45.913+0000 7f4348c14700 -1 osd.10 1050 heartbeat_check: no reply from 172.21.15.162:6828 osd.7 since back 2024-02-09T18:09:18.400760+0000 front 2024-02-09T18:09:18.400836+0000 (oldest deadline 2024-02-09T18:09:44.200987+0000) 2024-02-09T18:09:46.173 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:45 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:45.715+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:46.173 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:45 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:45.715+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:46.173 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:45 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:45.715+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.425025+0000 front 2024-02-09T18:08:38.425098+0000 (oldest deadline 2024-02-09T18:09:06.025618+0000) 2024-02-09T18:09:46.173 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:45 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:45.715+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:46.026169+0000 front 2024-02-09T18:08:46.026259+0000 (oldest deadline 2024-02-09T18:09:12.426582+0000) 2024-02-09T18:09:46.173 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:45 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:45.715+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:04.529946+0000 front 2024-02-09T18:09:04.529983+0000 (oldest deadline 2024-02-09T18:09:30.429896+0000) 2024-02-09T18:09:46.173 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:45 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:45.715+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-02-09T18:09:10.430677+0000 front 2024-02-09T18:09:10.430649+0000 (oldest deadline 2024-02-09T18:09:35.130523+0000) 2024-02-09T18:09:46.173 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:45 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:45.715+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6828 osd.7 since back 2024-02-09T18:09:16.831538+0000 front 2024-02-09T18:09:16.831519+0000 (oldest deadline 2024-02-09T18:09:42.131451+0000) 2024-02-09T18:09:47.172 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:46 smithi179 bash[174965]: ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-10 2024-02-09T18:09:47.172 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:46 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:46.764+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:47.172 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:46 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:46.764+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:47.172 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:46 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:46.764+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.425025+0000 front 2024-02-09T18:08:38.425098+0000 (oldest deadline 2024-02-09T18:09:06.025618+0000) 2024-02-09T18:09:47.173 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:46 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:46.764+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:46.026169+0000 front 2024-02-09T18:08:46.026259+0000 (oldest deadline 2024-02-09T18:09:12.426582+0000) 2024-02-09T18:09:47.173 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:46 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:46.764+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:04.529946+0000 front 2024-02-09T18:09:04.529983+0000 (oldest deadline 2024-02-09T18:09:30.429896+0000) 2024-02-09T18:09:47.173 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:46 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:46.764+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-02-09T18:09:10.430677+0000 front 2024-02-09T18:09:10.430649+0000 (oldest deadline 2024-02-09T18:09:35.130523+0000) 2024-02-09T18:09:47.173 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:46 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:46.764+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6828 osd.7 since back 2024-02-09T18:09:16.831538+0000 front 2024-02-09T18:09:16.831519+0000 (oldest deadline 2024-02-09T18:09:42.131451+0000) 2024-02-09T18:09:47.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:47 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:47.774+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:47.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:47 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:47.774+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:47.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:47 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:47.774+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.425025+0000 front 2024-02-09T18:08:38.425098+0000 (oldest deadline 2024-02-09T18:09:06.025618+0000) 2024-02-09T18:09:47.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:47 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:47.774+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:46.026169+0000 front 2024-02-09T18:08:46.026259+0000 (oldest deadline 2024-02-09T18:09:12.426582+0000) 2024-02-09T18:09:47.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:47 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:47.774+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:04.529946+0000 front 2024-02-09T18:09:04.529983+0000 (oldest deadline 2024-02-09T18:09:30.429896+0000) 2024-02-09T18:09:47.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:47 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:47.774+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-02-09T18:09:10.430677+0000 front 2024-02-09T18:09:10.430649+0000 (oldest deadline 2024-02-09T18:09:35.130523+0000) 2024-02-09T18:09:47.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:47 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:47.774+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6828 osd.7 since back 2024-02-09T18:09:16.831538+0000 front 2024-02-09T18:09:16.831519+0000 (oldest deadline 2024-02-09T18:09:42.131451+0000) 2024-02-09T18:09:48.640 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a69c870a-c76e-11ee-95b7-87774f69a715@osd.10.service' 2024-02-09T18:09:48.667 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:48 smithi179 systemd[1]: ceph-a69c870a-c76e-11ee-95b7-87774f69a715@osd.10.service: Succeeded. 2024-02-09T18:09:48.667 INFO:journalctl@ceph.osd.10.smithi179.stdout:Feb 09 18:09:48 smithi179 systemd[1]: Stopped Ceph osd.10 for a69c870a-c76e-11ee-95b7-87774f69a715. 2024-02-09T18:09:48.679 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T18:09:48.680 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-02-09T18:09:48.680 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-02-09T18:09:48.680 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-a69c870a-c76e-11ee-95b7-87774f69a715@osd.11 2024-02-09T18:09:48.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:48 smithi179 systemd[1]: Stopping Ceph osd.11 for a69c870a-c76e-11ee-95b7-87774f69a715... 2024-02-09T18:09:48.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:48 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:48.785+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:48.922 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:48 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:48.785+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:48.923 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:48 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:48.785+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.425025+0000 front 2024-02-09T18:08:38.425098+0000 (oldest deadline 2024-02-09T18:09:06.025618+0000) 2024-02-09T18:09:48.923 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:48 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:48.785+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:46.026169+0000 front 2024-02-09T18:08:46.026259+0000 (oldest deadline 2024-02-09T18:09:12.426582+0000) 2024-02-09T18:09:48.923 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:48 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:48.785+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:04.529946+0000 front 2024-02-09T18:09:04.529983+0000 (oldest deadline 2024-02-09T18:09:30.429896+0000) 2024-02-09T18:09:48.923 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:48 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:48.785+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-02-09T18:09:10.430677+0000 front 2024-02-09T18:09:10.430649+0000 (oldest deadline 2024-02-09T18:09:35.130523+0000) 2024-02-09T18:09:48.923 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:48 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:48.785+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6828 osd.7 since back 2024-02-09T18:09:16.831538+0000 front 2024-02-09T18:09:16.831519+0000 (oldest deadline 2024-02-09T18:09:42.131451+0000) 2024-02-09T18:09:49.422 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:48 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:48.979+0000 7f109ae0c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T18:09:49.422 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:48 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:48.979+0000 7f109ae0c700 -1 osd.11 1050 *** Got signal Terminated *** 2024-02-09T18:09:49.422 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:48 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:48.979+0000 7f109ae0c700 -1 osd.11 1050 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-09T18:09:50.172 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:49 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:49.827+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:50.172 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:49 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:49.827+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:50.172 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:49 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:49.827+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.425025+0000 front 2024-02-09T18:08:38.425098+0000 (oldest deadline 2024-02-09T18:09:06.025618+0000) 2024-02-09T18:09:50.172 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:49 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:49.827+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:46.026169+0000 front 2024-02-09T18:08:46.026259+0000 (oldest deadline 2024-02-09T18:09:12.426582+0000) 2024-02-09T18:09:50.172 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:49 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:49.827+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:04.529946+0000 front 2024-02-09T18:09:04.529983+0000 (oldest deadline 2024-02-09T18:09:30.429896+0000) 2024-02-09T18:09:50.172 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:49 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:49.827+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-02-09T18:09:10.430677+0000 front 2024-02-09T18:09:10.430649+0000 (oldest deadline 2024-02-09T18:09:35.130523+0000) 2024-02-09T18:09:50.172 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:49 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:49.827+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6828 osd.7 since back 2024-02-09T18:09:16.831538+0000 front 2024-02-09T18:09:16.831519+0000 (oldest deadline 2024-02-09T18:09:42.131451+0000) 2024-02-09T18:09:51.172 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:50 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:50.796+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:51.172 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:50 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:50.796+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:51.172 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:50 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:50.796+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.425025+0000 front 2024-02-09T18:08:38.425098+0000 (oldest deadline 2024-02-09T18:09:06.025618+0000) 2024-02-09T18:09:51.172 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:50 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:50.796+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:46.026169+0000 front 2024-02-09T18:08:46.026259+0000 (oldest deadline 2024-02-09T18:09:12.426582+0000) 2024-02-09T18:09:51.172 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:50 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:50.796+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:04.529946+0000 front 2024-02-09T18:09:04.529983+0000 (oldest deadline 2024-02-09T18:09:30.429896+0000) 2024-02-09T18:09:51.172 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:50 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:50.796+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-02-09T18:09:10.430677+0000 front 2024-02-09T18:09:10.430649+0000 (oldest deadline 2024-02-09T18:09:35.130523+0000) 2024-02-09T18:09:51.172 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:50 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:50.796+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6828 osd.7 since back 2024-02-09T18:09:16.831538+0000 front 2024-02-09T18:09:16.831519+0000 (oldest deadline 2024-02-09T18:09:42.131451+0000) 2024-02-09T18:09:52.172 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:51 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:51.820+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:52.172 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:51 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:51.820+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:52.172 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:51 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:51.820+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.425025+0000 front 2024-02-09T18:08:38.425098+0000 (oldest deadline 2024-02-09T18:09:06.025618+0000) 2024-02-09T18:09:52.172 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:51 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:51.820+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:46.026169+0000 front 2024-02-09T18:08:46.026259+0000 (oldest deadline 2024-02-09T18:09:12.426582+0000) 2024-02-09T18:09:52.172 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:51 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:51.820+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:04.529946+0000 front 2024-02-09T18:09:04.529983+0000 (oldest deadline 2024-02-09T18:09:30.429896+0000) 2024-02-09T18:09:52.172 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:51 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:51.820+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-02-09T18:09:10.430677+0000 front 2024-02-09T18:09:10.430649+0000 (oldest deadline 2024-02-09T18:09:35.130523+0000) 2024-02-09T18:09:52.172 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:51 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:51.820+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6828 osd.7 since back 2024-02-09T18:09:16.831538+0000 front 2024-02-09T18:09:16.831519+0000 (oldest deadline 2024-02-09T18:09:42.131451+0000) 2024-02-09T18:09:52.172 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:51 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:51.820+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.179:6804 osd.8 since back 2024-02-09T18:09:25.032280+0000 front 2024-02-09T18:09:25.032362+0000 (oldest deadline 2024-02-09T18:09:50.932297+0000) 2024-02-09T18:09:53.172 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:52 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:52.781+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:53.172 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:52 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:52.781+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:53.172 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:52 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:52.781+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.425025+0000 front 2024-02-09T18:08:38.425098+0000 (oldest deadline 2024-02-09T18:09:06.025618+0000) 2024-02-09T18:09:53.172 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:52 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:52.781+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:46.026169+0000 front 2024-02-09T18:08:46.026259+0000 (oldest deadline 2024-02-09T18:09:12.426582+0000) 2024-02-09T18:09:53.172 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:52 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:52.781+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:04.529946+0000 front 2024-02-09T18:09:04.529983+0000 (oldest deadline 2024-02-09T18:09:30.429896+0000) 2024-02-09T18:09:53.172 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:52 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:52.781+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-02-09T18:09:10.430677+0000 front 2024-02-09T18:09:10.430649+0000 (oldest deadline 2024-02-09T18:09:35.130523+0000) 2024-02-09T18:09:53.172 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:52 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:52.781+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6828 osd.7 since back 2024-02-09T18:09:16.831538+0000 front 2024-02-09T18:09:16.831519+0000 (oldest deadline 2024-02-09T18:09:42.131451+0000) 2024-02-09T18:09:53.172 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:52 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:52.781+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.179:6804 osd.8 since back 2024-02-09T18:09:25.032280+0000 front 2024-02-09T18:09:25.032362+0000 (oldest deadline 2024-02-09T18:09:50.932297+0000) 2024-02-09T18:09:54.172 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:53 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:53.766+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6806 osd.0 since back 2024-02-09T18:08:22.623016+0000 front 2024-02-09T18:08:22.623003+0000 (oldest deadline 2024-02-09T18:08:45.522845+0000) 2024-02-09T18:09:54.172 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:53 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:53.766+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6814 osd.1 since back 2024-02-09T18:08:30.824136+0000 front 2024-02-09T18:08:30.824462+0000 (oldest deadline 2024-02-09T18:08:58.424479+0000) 2024-02-09T18:09:54.172 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:53 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:53.766+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6822 osd.2 since back 2024-02-09T18:08:38.425025+0000 front 2024-02-09T18:08:38.425098+0000 (oldest deadline 2024-02-09T18:09:06.025618+0000) 2024-02-09T18:09:54.172 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:53 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:53.766+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.99:6830 osd.3 since back 2024-02-09T18:08:46.026169+0000 front 2024-02-09T18:08:46.026259+0000 (oldest deadline 2024-02-09T18:09:12.426582+0000) 2024-02-09T18:09:54.172 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:53 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:53.766+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-02-09T18:09:04.529946+0000 front 2024-02-09T18:09:04.529983+0000 (oldest deadline 2024-02-09T18:09:30.429896+0000) 2024-02-09T18:09:54.172 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:53 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:53.766+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-02-09T18:09:10.430677+0000 front 2024-02-09T18:09:10.430649+0000 (oldest deadline 2024-02-09T18:09:35.130523+0000) 2024-02-09T18:09:54.172 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:53 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:53.766+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.162:6828 osd.7 since back 2024-02-09T18:09:16.831538+0000 front 2024-02-09T18:09:16.831519+0000 (oldest deadline 2024-02-09T18:09:42.131451+0000) 2024-02-09T18:09:54.172 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:53 smithi179 ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11[71684]: 2024-02-09T18:09:53.766+0000 7f1097407700 -1 osd.11 1050 heartbeat_check: no reply from 172.21.15.179:6804 osd.8 since back 2024-02-09T18:09:25.032280+0000 front 2024-02-09T18:09:25.032362+0000 (oldest deadline 2024-02-09T18:09:50.932297+0000) 2024-02-09T18:09:55.172 INFO:journalctl@ceph.osd.11.smithi179.stdout:Feb 09 18:09:54 smithi179 bash[175229]: ceph-a69c870a-c76e-11ee-95b7-87774f69a715-osd-11 2024-02-09T18:09:57.014 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a69c870a-c76e-11ee-95b7-87774f69a715@osd.11.service' 2024-02-09T18:09:57.059 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T18:09:57.059 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-02-09T18:09:57.059 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a69c870a-c76e-11ee-95b7-87774f69a715 --force --keep-logs 2024-02-09T18:09:57.287 INFO:teuthology.orchestra.run.smithi099.stdout:Deleting cluster with fsid: a69c870a-c76e-11ee-95b7-87774f69a715 2024-02-09T18:09:58.478 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a69c870a-c76e-11ee-95b7-87774f69a715 --force --keep-logs 2024-02-09T18:09:58.708 INFO:teuthology.orchestra.run.smithi162.stdout:Deleting cluster with fsid: a69c870a-c76e-11ee-95b7-87774f69a715 2024-02-09T18:09:59.896 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a69c870a-c76e-11ee-95b7-87774f69a715 --force --keep-logs 2024-02-09T18:10:00.123 INFO:teuthology.orchestra.run.smithi179.stdout:Deleting cluster with fsid: a69c870a-c76e-11ee-95b7-87774f69a715 2024-02-09T18:10:01.147 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-09T18:10:01.182 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-09T18:10:01.220 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-09T18:10:01.250 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-09T18:10:01.253 DEBUG:teuthology.misc:Transferring archived files from smithi099:/var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/crash to /home/teuthworker/archive/lflores-2024-02-09_16:40:54-rados-wip-yuri2-testing-2024-02-06-1154-distro-default-smithi/7554109/remote/smithi099/crash 2024-02-09T18:10:01.253 DEBUG:teuthology.orchestra.run.smithi099:> sudo tar c -f - -C /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/crash -- . 2024-02-09T18:10:01.312 INFO:teuthology.orchestra.run.smithi099.stderr:tar: /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/crash: Cannot open: No such file or directory 2024-02-09T18:10:01.312 INFO:teuthology.orchestra.run.smithi099.stderr:tar: Error is not recoverable: exiting now 2024-02-09T18:10:01.314 DEBUG:teuthology.misc:Transferring archived files from smithi162:/var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/crash to /home/teuthworker/archive/lflores-2024-02-09_16:40:54-rados-wip-yuri2-testing-2024-02-06-1154-distro-default-smithi/7554109/remote/smithi162/crash 2024-02-09T18:10:01.315 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/crash -- . 2024-02-09T18:10:01.373 INFO:teuthology.orchestra.run.smithi162.stderr:tar: /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/crash: Cannot open: No such file or directory 2024-02-09T18:10:01.373 INFO:teuthology.orchestra.run.smithi162.stderr:tar: Error is not recoverable: exiting now 2024-02-09T18:10:01.375 DEBUG:teuthology.misc:Transferring archived files from smithi179:/var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/crash to /home/teuthworker/archive/lflores-2024-02-09_16:40:54-rados-wip-yuri2-testing-2024-02-06-1154-distro-default-smithi/7554109/remote/smithi179/crash 2024-02-09T18:10:01.375 DEBUG:teuthology.orchestra.run.smithi179:> sudo tar c -f - -C /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/crash -- . 2024-02-09T18:10:01.433 INFO:teuthology.orchestra.run.smithi179.stderr:tar: /var/lib/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/crash: Cannot open: No such file or directory 2024-02-09T18:10:01.433 INFO:teuthology.orchestra.run.smithi179.stderr:tar: Error is not recoverable: exiting now 2024-02-09T18:10:01.434 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-09T18:10:01.434 DEBUG:teuthology.orchestra.run.smithi099:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | head -n 1 2024-02-09T18:10:01.530 INFO:teuthology.orchestra.run.smithi099.stdout:2024-02-09T17:30:00.000207+0000 mon.a (mon.0) 1268 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-02-09T18:10:01.530 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-02-09T18:10:01.530 DEBUG:teuthology.orchestra.run.smithi099:> sudo egrep '\[SEC\]' /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | head -n 1 2024-02-09T18:10:01.566 DEBUG:teuthology.orchestra.run.smithi099:> sudo egrep '\[ERR\]' /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | head -n 1 2024-02-09T18:10:01.640 DEBUG:teuthology.orchestra.run.smithi099:> sudo egrep '\[WRN\]' /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | head -n 1 2024-02-09T18:10:01.714 INFO:teuthology.orchestra.run.smithi099.stdout:2024-02-09T17:30:00.000207+0000 mon.a (mon.0) 1268 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-02-09T18:10:01.715 INFO:tasks.cephadm:Compressing logs... 2024-02-09T18:10:01.715 DEBUG:teuthology.orchestra.run.smithi099:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-09T18:10:01.757 DEBUG:teuthology.orchestra.run.smithi162:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-09T18:10:01.760 DEBUG:teuthology.orchestra.run.smithi179:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-09T18:10:01.797 INFO:teuthology.orchestra.run.smithi162.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-02-09T18:10:01.805 INFO:teuthology.orchestra.run.smithi099.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-09T18:10:01.806 INFO:teuthology.orchestra.run.smithi099.stderr:: No such file or directory 2024-02-09T18:10:01.809 INFO:teuthology.orchestra.run.smithi179.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-02-09T18:10:01.809 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-09T18:10:01.815 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-volume.log 2024-02-09T18:10:01.817 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-osd.2.log 2024-02-09T18:10:01.817 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-mgr.y.log 2024-02-09T18:10:01.817 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph.audit.log 2024-02-09T18:10:01.818 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph.cephadm.log 2024-02-09T18:10:01.819 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-osd.3.log 2024-02-09T18:10:01.819 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-osd.0.log 2024-02-09T18:10:01.819 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/objectstore_tool.7.log 2024-02-09T18:10:01.819 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-mon.a.log 2024-02-09T18:10:01.819 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph.log 2024-02-09T18:10:01.819 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-osd.1.log 2024-02-09T18:10:01.822 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-09T18:10:01.825 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-osd.11.log 2024-02-09T18:10:01.825 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-volume.log 2024-02-09T18:10:01.825 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph.audit.log 2024-02-09T18:10:01.825 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph.cephadm.log 2024-02-09T18:10:01.825 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-osd.10.log 2024-02-09T18:10:01.826 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-osd.8.log 2024-02-09T18:10:01.827 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/objectstore_tool.7.log 2024-02-09T18:10:01.827 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-mon.c.log 2024-02-09T18:10:01.827 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/objectstore_tool.7.log: gzip -5 --verbose -- /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-osd.9.log 2024-02-09T18:10:01.827 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph.log 2024-02-09T18:10:01.835 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/objectstore_tool.7.log: /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-mon.a.log: /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph.log: /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-osd.1.log: 85.3% -- replaced with /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph.log.gz 2024-02-09T18:10:01.835 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-osd.6.log 2024-02-09T18:10:01.836 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-volume.log 2024-02-09T18:10:01.836 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph.audit.log 2024-02-09T18:10:01.837 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-osd.6.log: /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph.cephadm.log 2024-02-09T18:10:01.838 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-osd.7.log 2024-02-09T18:10:01.838 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph.audit.log: /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-mgr.x.log 2024-02-09T18:10:01.838 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-mon.b.log 2024-02-09T18:10:01.838 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph.log 2024-02-09T18:10:01.838 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-osd.4.log 2024-02-09T18:10:01.839 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-osd.5.log 2024-02-09T18:10:01.847 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-osd.4.log: /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-osd.5.log: 93.6% -- replaced with /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-mgr.x.log.gz 2024-02-09T18:10:01.872 INFO:teuthology.orchestra.run.smithi099.stderr: 82.7% -- replaced with /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph.cephadm.log.gz 2024-02-09T18:10:01.893 INFO:teuthology.orchestra.run.smithi162.stderr: 91.4% -- replaced with /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-volume.log.gz 2024-02-09T18:10:01.895 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-osd.9.log: /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph.log: 82.4% -- replaced with /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph.cephadm.log.gz 2024-02-09T18:10:01.989 INFO:teuthology.orchestra.run.smithi162.stderr: 82.4% -- replaced with /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph.cephadm.log.gz 2024-02-09T18:10:01.995 INFO:teuthology.orchestra.run.smithi099.stderr: 89.6% -- replaced with /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph.audit.log.gz 2024-02-09T18:10:02.004 INFO:teuthology.orchestra.run.smithi099.stderr: 91.4% -- replaced with /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-volume.log.gz 2024-02-09T18:10:02.043 INFO:teuthology.orchestra.run.smithi099.stderr: 94.3% -- replaced with /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/objectstore_tool.7.log.gz 2024-02-09T18:10:02.062 INFO:teuthology.orchestra.run.smithi162.stderr: 95.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-09T18:10:02.136 INFO:teuthology.orchestra.run.smithi099.stderr: 95.9% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-09T18:10:02.646 INFO:teuthology.orchestra.run.smithi179.stderr: 94.7% -- replaced with /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/objectstore_tool.7.log.gz 2024-02-09T18:10:02.653 INFO:teuthology.orchestra.run.smithi179.stderr: 91.3% -- replaced with /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-volume.log.gz 2024-02-09T18:10:02.757 INFO:teuthology.orchestra.run.smithi162.stderr: 85.3% -- replaced with /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph.log.gz 2024-02-09T18:10:02.764 INFO:teuthology.orchestra.run.smithi099.stderr: 91.2% -- replaced with /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-mgr.y.log.gz 2024-02-09T18:10:03.014 INFO:teuthology.orchestra.run.smithi162.stderr: 89.7% -- replaced with /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph.audit.log.gz 2024-02-09T18:10:03.037 INFO:teuthology.orchestra.run.smithi179.stderr: 95.4% -- replaced with /var/log/ceph/cephadm.log.gz 89.7% 2024-02-09T18:10:03.037 INFO:teuthology.orchestra.run.smithi179.stderr: -- replaced with /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph.audit.log.gz 2024-02-09T18:10:03.037 INFO:teuthology.orchestra.run.smithi179.stderr: 85.3% -- replaced with /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph.log.gz 2024-02-09T18:10:05.386 INFO:teuthology.orchestra.run.smithi179.stderr: 91.3% -- replaced with /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-mon.c.log.gz 2024-02-09T18:10:08.247 INFO:teuthology.orchestra.run.smithi162.stderr: 91.1% -- replaced with /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-mon.b.log.gz 2024-02-09T18:10:10.554 INFO:teuthology.orchestra.run.smithi099.stderr: 90.0% -- replaced with /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-mon.a.log.gz 2024-02-09T18:12:20.547 INFO:teuthology.orchestra.run.smithi099.stderr: 94.4% -- replaced with /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-osd.2.log.gz 2024-02-09T18:13:52.811 INFO:teuthology.orchestra.run.smithi179.stderr: 93.3% -- replaced with /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-osd.8.log.gz 2024-02-09T18:14:08.965 INFO:teuthology.orchestra.run.smithi179.stderr: 93.9% -- replaced with /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-osd.10.log.gz 2024-02-09T18:14:57.120 INFO:teuthology.orchestra.run.smithi099.stderr: 93.7% -- replaced with /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-osd.3.log.gz 2024-02-09T18:15:31.481 INFO:teuthology.orchestra.run.smithi099.stderr: 92.0% -- replaced with /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-osd.0.log.gz 2024-02-09T18:15:32.981 INFO:teuthology.orchestra.run.smithi179.stderr: 93.1% -- replaced with /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-osd.9.log.gz 2024-02-09T18:15:36.474 INFO:teuthology.orchestra.run.smithi162.stderr: 93.1% -- replaced with /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-osd.7.log.gz 2024-02-09T18:15:52.320 INFO:teuthology.orchestra.run.smithi162.stderr: 92.7% -- replaced with /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-osd.5.log.gz 2024-02-09T18:16:07.666 INFO:teuthology.orchestra.run.smithi162.stderr: 92.1% -- replaced with /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-osd.6.log.gz 2024-02-09T18:17:02.474 INFO:teuthology.orchestra.run.smithi179.stderr: 92.4% -- replaced with /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-osd.11.log.gz 2024-02-09T18:17:02.475 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-02-09T18:17:02.475 INFO:teuthology.orchestra.run.smithi179.stderr:real 7m0.696s 2024-02-09T18:17:02.475 INFO:teuthology.orchestra.run.smithi179.stderr:user 8m3.671s 2024-02-09T18:17:02.476 INFO:teuthology.orchestra.run.smithi179.stderr:sys 0m20.664s 2024-02-09T18:17:13.814 INFO:teuthology.orchestra.run.smithi099.stderr: 92.4% -- replaced with /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-osd.1.log.gz 2024-02-09T18:17:13.824 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-02-09T18:17:13.824 INFO:teuthology.orchestra.run.smithi099.stderr:real 7m12.052s 2024-02-09T18:17:13.824 INFO:teuthology.orchestra.run.smithi099.stderr:user 8m22.404s 2024-02-09T18:17:13.824 INFO:teuthology.orchestra.run.smithi099.stderr:sys 0m20.835s 2024-02-09T18:18:24.113 INFO:teuthology.orchestra.run.smithi162.stderr: 91.3% -- replaced with /var/log/ceph/a69c870a-c76e-11ee-95b7-87774f69a715/ceph-osd.4.log.gz 2024-02-09T18:18:24.115 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-02-09T18:18:24.115 INFO:teuthology.orchestra.run.smithi162.stderr:real 8m22.336s 2024-02-09T18:18:24.115 INFO:teuthology.orchestra.run.smithi162.stderr:user 9m46.923s 2024-02-09T18:18:24.115 INFO:teuthology.orchestra.run.smithi162.stderr:sys 0m23.979s 2024-02-09T18:18:24.115 INFO:tasks.cephadm:Archiving logs... 2024-02-09T18:18:24.116 DEBUG:teuthology.misc:Transferring archived files from smithi099:/var/log/ceph to /home/teuthworker/archive/lflores-2024-02-09_16:40:54-rados-wip-yuri2-testing-2024-02-06-1154-distro-default-smithi/7554109/remote/smithi099/log 2024-02-09T18:18:24.117 DEBUG:teuthology.orchestra.run.smithi099:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-09T18:19:53.266 DEBUG:teuthology.misc:Transferring archived files from smithi162:/var/log/ceph to /home/teuthworker/archive/lflores-2024-02-09_16:40:54-rados-wip-yuri2-testing-2024-02-06-1154-distro-default-smithi/7554109/remote/smithi162/log 2024-02-09T18:19:53.267 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-09T18:21:42.611 DEBUG:teuthology.misc:Transferring archived files from smithi179:/var/log/ceph to /home/teuthworker/archive/lflores-2024-02-09_16:40:54-rados-wip-yuri2-testing-2024-02-06-1154-distro-default-smithi/7554109/remote/smithi179/log 2024-02-09T18:21:42.612 DEBUG:teuthology.orchestra.run.smithi179:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-09T18:23:06.126 INFO:tasks.cephadm:Removing cluster... 2024-02-09T18:23:06.127 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a69c870a-c76e-11ee-95b7-87774f69a715 --force 2024-02-09T18:23:06.682 INFO:teuthology.orchestra.run.smithi099.stdout:Deleting cluster with fsid: a69c870a-c76e-11ee-95b7-87774f69a715 2024-02-09T18:23:07.520 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a69c870a-c76e-11ee-95b7-87774f69a715 --force 2024-02-09T18:23:08.064 INFO:teuthology.orchestra.run.smithi162.stdout:Deleting cluster with fsid: a69c870a-c76e-11ee-95b7-87774f69a715 2024-02-09T18:23:09.004 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a69c870a-c76e-11ee-95b7-87774f69a715 --force 2024-02-09T18:23:09.424 INFO:teuthology.orchestra.run.smithi179.stdout:Deleting cluster with fsid: a69c870a-c76e-11ee-95b7-87774f69a715 2024-02-09T18:23:10.251 INFO:tasks.cephadm:Removing cephadm ... 2024-02-09T18:23:10.251 DEBUG:teuthology.orchestra.run.smithi099:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-09T18:23:10.296 DEBUG:teuthology.orchestra.run.smithi162:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-09T18:23:10.331 DEBUG:teuthology.orchestra.run.smithi179:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-09T18:23:10.355 INFO:tasks.cephadm:Teardown complete 2024-02-09T18:23:10.356 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-09T18:23:10.365 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-09T18:23:10.366 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-09T18:23:10.368 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-09T18:23:10.373 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-09T18:23:10.517 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-09T18:23:10.527 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-09T18:23:10.527 DEBUG:teuthology.orchestra.run.smithi099:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-09T18:23:10.530 DEBUG:teuthology.orchestra.run.smithi162:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-09T18:23:10.532 DEBUG:teuthology.orchestra.run.smithi179:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-09T18:23:10.547 INFO:teuthology.orchestra.run.smithi099.stderr:bash: ntpq: command not found 2024-02-09T18:23:10.549 INFO:teuthology.orchestra.run.smithi162.stderr:bash: ntpq: command not found 2024-02-09T18:23:10.549 INFO:teuthology.orchestra.run.smithi179.stderr:bash: ntpq: command not found 2024-02-09T18:23:10.563 INFO:teuthology.orchestra.run.smithi179.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-09T18:23:10.563 INFO:teuthology.orchestra.run.smithi179.stdout:=============================================================================== 2024-02-09T18:23:10.563 INFO:teuthology.orchestra.run.smithi179.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 29 +579us[ +579us] +/- 116ms 2024-02-09T18:23:10.563 INFO:teuthology.orchestra.run.smithi179.stdout:^* hv02.front.sepia.ceph.com 3 6 377 29 -325us[ -311us] +/- 49ms 2024-02-09T18:23:10.563 INFO:teuthology.orchestra.run.smithi179.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 30 +133us[ +147us] +/- 81ms 2024-02-09T18:23:10.564 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-02-09T18:23:10.574 INFO:teuthology.orchestra.run.smithi162.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-09T18:23:10.574 INFO:teuthology.orchestra.run.smithi162.stdout:=============================================================================== 2024-02-09T18:23:10.574 INFO:teuthology.orchestra.run.smithi162.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 32 +588us[ +577us] +/- 116ms 2024-02-09T18:23:10.574 INFO:teuthology.orchestra.run.smithi162.stdout:^* hv02.front.sepia.ceph.com 3 6 377 29 -331us[ -342us] +/- 49ms 2024-02-09T18:23:10.575 INFO:teuthology.orchestra.run.smithi162.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 32 +143us[ +132us] +/- 81ms 2024-02-09T18:23:10.575 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-02-09T18:23:10.609 INFO:teuthology.orchestra.run.smithi099.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-09T18:23:10.609 INFO:teuthology.orchestra.run.smithi099.stdout:=============================================================================== 2024-02-09T18:23:10.609 INFO:teuthology.orchestra.run.smithi099.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 31 +569us[ +569us] +/- 116ms 2024-02-09T18:23:10.609 INFO:teuthology.orchestra.run.smithi099.stdout:^* hv02.front.sepia.ceph.com 3 6 377 34 -352us[ -343us] +/- 49ms 2024-02-09T18:23:10.609 INFO:teuthology.orchestra.run.smithi099.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 38 +121us[ +130us] +/- 81ms 2024-02-09T18:23:10.609 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-02-09T18:23:10.610 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-09T18:23:10.620 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-09T18:23:10.621 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-09T18:23:10.630 DEBUG:teuthology.orchestra.run.smithi099:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-09T18:23:10.735 DEBUG:teuthology.orchestra.run.smithi162:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-09T18:23:10.980 DEBUG:teuthology.orchestra.run.smithi179:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-09T18:23:11.202 DEBUG:teuthology.orchestra.run.smithi099:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-09T18:23:11.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-09T18:23:11.237 DEBUG:teuthology.orchestra.run.smithi162:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-09T18:23:11.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-09T18:23:11.274 DEBUG:teuthology.orchestra.run.smithi179:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-09T18:23:11.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-09T18:23:11.307 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-09T18:23:11.318 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-09T18:23:11.326 INFO:teuthology.task.internal:Duration was 4654.555399 seconds 2024-02-09T18:23:11.327 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-09T18:23:11.336 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-09T18:23:11.336 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-09T18:23:11.338 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-09T18:23:11.341 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-09T18:23:11.420 INFO:teuthology.orchestra.run.smithi162.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-09T18:23:11.429 INFO:teuthology.orchestra.run.smithi099.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-09T18:23:11.465 INFO:teuthology.orchestra.run.smithi179.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-09T18:23:11.723 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-09T18:23:11.724 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi099.front.sepia.ceph.com 2024-02-09T18:23:11.724 DEBUG:teuthology.orchestra.run.smithi099:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-09T18:23:11.768 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi162.front.sepia.ceph.com 2024-02-09T18:23:11.769 DEBUG:teuthology.orchestra.run.smithi162:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-09T18:23:11.808 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi179.front.sepia.ceph.com 2024-02-09T18:23:11.808 DEBUG:teuthology.orchestra.run.smithi179:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-09T18:23:11.842 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-09T18:23:11.842 DEBUG:teuthology.orchestra.run.smithi099:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-09T18:23:11.844 DEBUG:teuthology.orchestra.run.smithi162:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-09T18:23:11.850 DEBUG:teuthology.orchestra.run.smithi179:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-09T18:23:11.984 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-09T18:23:11.984 DEBUG:teuthology.orchestra.run.smithi099:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-09T18:23:11.987 DEBUG:teuthology.orchestra.run.smithi162:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-09T18:23:11.989 DEBUG:teuthology.orchestra.run.smithi179:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-09T18:23:12.228 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-09T18:23:12.238 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-09T18:23:12.238 DEBUG:teuthology.orchestra.run.smithi099:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-09T18:23:12.306 DEBUG:teuthology.orchestra.run.smithi162:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-09T18:23:12.351 DEBUG:teuthology.orchestra.run.smithi179:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-09T18:23:12.407 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-09T18:23:12.417 DEBUG:teuthology.orchestra.run.smithi099:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-09T18:23:12.419 DEBUG:teuthology.orchestra.run.smithi162:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-09T18:23:12.422 DEBUG:teuthology.orchestra.run.smithi179:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-09T18:23:12.453 INFO:teuthology.orchestra.run.smithi099.stdout:kernel.core_pattern = core 2024-02-09T18:23:12.455 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern = core 2024-02-09T18:23:12.459 INFO:teuthology.orchestra.run.smithi179.stdout:kernel.core_pattern = core 2024-02-09T18:23:12.473 DEBUG:teuthology.orchestra.run.smithi099:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-09T18:23:12.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-09T18:23:12.525 DEBUG:teuthology.orchestra.run.smithi162:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-09T18:23:12.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-09T18:23:12.542 DEBUG:teuthology.orchestra.run.smithi179:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-09T18:23:12.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-09T18:23:12.559 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-09T18:23:12.568 INFO:teuthology.task.internal:Transferring archived files... 2024-02-09T18:23:12.570 DEBUG:teuthology.misc:Transferring archived files from smithi099:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/lflores-2024-02-09_16:40:54-rados-wip-yuri2-testing-2024-02-06-1154-distro-default-smithi/7554109/remote/smithi099 2024-02-09T18:23:12.570 DEBUG:teuthology.orchestra.run.smithi099:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-09T18:23:12.629 DEBUG:teuthology.misc:Transferring archived files from smithi162:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/lflores-2024-02-09_16:40:54-rados-wip-yuri2-testing-2024-02-06-1154-distro-default-smithi/7554109/remote/smithi162 2024-02-09T18:23:12.629 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-09T18:23:12.678 DEBUG:teuthology.misc:Transferring archived files from smithi179:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/lflores-2024-02-09_16:40:54-rados-wip-yuri2-testing-2024-02-06-1154-distro-default-smithi/7554109/remote/smithi179 2024-02-09T18:23:12.678 DEBUG:teuthology.orchestra.run.smithi179:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-09T18:23:12.727 INFO:teuthology.task.internal:Removing archive directory... 2024-02-09T18:23:12.727 DEBUG:teuthology.orchestra.run.smithi099:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-09T18:23:12.729 DEBUG:teuthology.orchestra.run.smithi162:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-09T18:23:12.731 DEBUG:teuthology.orchestra.run.smithi179:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-09T18:23:12.775 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-09T18:23:12.788 INFO:teuthology.task.internal:Not uploading archives. 2024-02-09T18:23:12.788 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-09T18:23:12.797 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-09T18:23:12.797 DEBUG:teuthology.orchestra.run.smithi099:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-09T18:23:12.800 DEBUG:teuthology.orchestra.run.smithi162:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-09T18:23:12.802 DEBUG:teuthology.orchestra.run.smithi179:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-09T18:23:12.816 INFO:teuthology.orchestra.run.smithi099.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 9 18:23 /home/ubuntu/cephtest 2024-02-09T18:23:12.817 INFO:teuthology.orchestra.run.smithi162.stdout: 393225 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 9 18:23 /home/ubuntu/cephtest 2024-02-09T18:23:12.833 INFO:teuthology.orchestra.run.smithi179.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 9 18:23 /home/ubuntu/cephtest 2024-02-09T18:23:12.834 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-09T18:23:12.844 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-09T18:23:12.864 INFO:teuthology.nuke:Checking targets against current locks 2024-02-09T18:23:12.890 DEBUG:teuthology.nuke:shortname: smithi099 2024-02-09T18:23:12.890 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-09T18:23:12.912 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi099.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2024-02-09_16:40:54-rados-wip-yuri2-testing-2024-02-06-1154-distro-default-smithi/7554109', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-09 16:56:38.887306', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIOkSp3bbjLr6kK3awzep5s9kTS9xb+WUQrOAcagKtk1e1gLsEWitevL0IBcheKc8IcHpQ+27M1mIOSfzGr3Kiw='} 2024-02-09T18:23:12.916 DEBUG:teuthology.nuke:shortname: smithi162 2024-02-09T18:23:12.916 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-09T18:23:12.942 DEBUG:teuthology.nuke:shortname: smithi179 2024-02-09T18:23:12.942 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-09T18:23:12.945 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi162.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2024-02-09_16:40:54-rados-wip-yuri2-testing-2024-02-06-1154-distro-default-smithi/7554109', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-09 16:56:38.886029', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI+8p+P8LpC8YqiPlB7Q3hXsXHduf9OHu6bXQLE5hd2P8TMdB8p3b7uHVbNyvQw90d/9uZR2JZTLsaLqV4/nXlU='} 2024-02-09T18:23:12.969 INFO:teuthology.orchestra.console.smithi099:Power off 2024-02-09T18:23:12.970 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-09T18:23:12.982 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi179.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2024-02-09_16:40:54-rados-wip-yuri2-testing-2024-02-06-1154-distro-default-smithi/7554109', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-09 16:56:38.888299', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJWV+jDbi2q9qIMcY5bjzRFollCWSTk/cUhTWsadUrHWx8+sh/V62MYj07tPHtu72R/hYsR6LsLW2QbHTQDMJ+8='} 2024-02-09T18:23:12.993 DEBUG:teuthology.orchestra.console.smithi099:power off output: Chassis Power Control: Down/Off 2024-02-09T18:23:12.994 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T18:23:13.015 INFO:teuthology.orchestra.console.smithi162:Power off 2024-02-09T18:23:13.015 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-09T18:23:13.029 DEBUG:teuthology.orchestra.console.smithi099:check power output: Chassis Power is on 2024-02-09T18:23:13.042 DEBUG:teuthology.orchestra.console.smithi162:power off output: Chassis Power Control: Down/Off 2024-02-09T18:23:13.042 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T18:23:13.059 INFO:teuthology.orchestra.console.smithi179:Power off 2024-02-09T18:23:13.059 DEBUG:teuthology.orchestra.console.smithi179:pexpect command: ipmitool -H smithi179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-09T18:23:13.069 DEBUG:teuthology.orchestra.console.smithi162:check power output: Chassis Power is on 2024-02-09T18:23:13.083 DEBUG:teuthology.orchestra.console.smithi179:power off output: Chassis Power Control: Down/Off 2024-02-09T18:23:13.083 DEBUG:teuthology.orchestra.console.smithi179:pexpect command: ipmitool -H smithi179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T18:23:13.105 DEBUG:teuthology.orchestra.console.smithi179:check power output: Chassis Power is on 2024-02-09T18:23:17.031 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T18:23:17.070 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T18:23:17.106 DEBUG:teuthology.orchestra.console.smithi179:pexpect command: ipmitool -H smithi179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T18:23:17.142 DEBUG:teuthology.orchestra.console.smithi099:check power output: Chassis Power is on 2024-02-09T18:23:17.180 DEBUG:teuthology.orchestra.console.smithi162:check power output: Chassis Power is on 2024-02-09T18:23:17.215 DEBUG:teuthology.orchestra.console.smithi179:check power output: Chassis Power is on 2024-02-09T18:23:21.146 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T18:23:21.181 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T18:23:21.215 DEBUG:teuthology.orchestra.console.smithi179:pexpect command: ipmitool -H smithi179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T18:23:21.257 DEBUG:teuthology.orchestra.console.smithi099:check power output: Chassis Power is off 2024-02-09T18:23:21.291 DEBUG:teuthology.orchestra.console.smithi162:check power output: Chassis Power is on 2024-02-09T18:23:21.324 DEBUG:teuthology.orchestra.console.smithi179:check power output: Chassis Power is on 2024-02-09T18:23:21.359 INFO:teuthology.orchestra.console.smithi099:Power off completed 2024-02-09T18:23:25.296 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T18:23:25.325 DEBUG:teuthology.orchestra.console.smithi179:pexpect command: ipmitool -H smithi179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T18:23:25.407 DEBUG:teuthology.orchestra.console.smithi162:check power output: Chassis Power is off 2024-02-09T18:23:25.435 DEBUG:teuthology.orchestra.console.smithi179:check power output: Chassis Power is off 2024-02-09T18:23:25.508 INFO:teuthology.orchestra.console.smithi162:Power off completed 2024-02-09T18:23:25.536 INFO:teuthology.orchestra.console.smithi179:Power off completed 2024-02-09T18:23:25.640 INFO:teuthology.run:Summary data: description: rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/quincy backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/radosbench} duration: 4654.555399179459 failure_reason: '"2024-02-09T17:30:00.000207+0000 mon.a (mon.0) 1268 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled" in cluster log' flavor: default owner: scheduled_lflores@teuthology success: false 2024-02-09T18:23:25.640 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-09T18:23:25.717 INFO:teuthology.run:FAIL